Broadcast behavior

[puddly]

I'm exploring ways that we can improve coordinator broadcast delivery. Conceptually, regardless of what we do, we're at the mercy of other routers on the network: all stacks will reject new broadcasts once their broadcast tables are full. Almost all routers will be running SiLabs firmware but it's still nice to compare with other popular stacks with available source code:

Stack	Limits	Default capacity	Remember / window	Sustained rate	Burst tolerance	How capacity returns
Spec (R23.2)	-	-	9s per broadcast	-	-	continuous: each broadcast frees on its own 9 s timer
ZBOSS	concurrency	16 in flight	9s per broadcast	~1.8 / s	16	continuous (1s granularity)
TI Z-Stack	concurrency	9 in flight	3s per broadcast	~3 / s	9	continuous (0.1s granularity)
TI Z-Stack (Z2M)	concurrency	12 in flight	2s per broadcast	~6 / s	12	continuous (0.1s granularity)
SiLabs EmberZNet	rate + burst clamp	size = 15	~15-20s sliding window	~0.75 / s	size - 6 = 9	batched: a chunk reappears every ~5s on a global timer
SiLabs EmberZNet (size=32)	rate + burst clamp	size = 32	~15-20 s sliding window	~1.6 / s	size - 6 = 26	batched: a chunk reappears every ~5s on a global timer

The spec doesn't actually say much here as far as I can tell: every unique broadcast is to be kept track of for 9s to avoid loops. That's about it. Every stack other than SiLabs seems to take the spec literally and maintains a separate timer per entry, which lets individual entries age out gradually and for new slots to open up smoothly. SiLabs differs here and has a global 5s timer per router.

• Bursts of traffic should behave identically on every stack: if you blast 20 requests at once, all the tables will fill up immediately and the remaining requests will be dropped.
• Trickles of traffic will behave differently: stacks with independent entry aging will allow requests to trickle in at the stack's effective sustained rate as new slots open up. SiLabs, however, will lock the broadcast table completely and then only allow a new batch of broadcasts after 5s.

I haven't experimented with how this global 5s timer policy ends up working in large meshes. In theory, meshing should desynchronize the global timers of nearby routers so maybe it helps make sure routers aren't fighting one another for airtime? I wonder how a full broadcast table impacts broadcast delivery, where a broadcast is considered delivered only after it has been relayed by all known neighbors?

In Ziggurat, we currently implement the spec literally and do not do anything to limit broadcasts beyond ensuring they are not re-broadcast more than once every 9s. This isn't a "healthy" approach for an entire mesh where every node is running the same firmware but Ziggurat assumes a coordinator role so I think it's fine to break the rules in this case 😅.

Anecdotally, we've found that changing the SiLabs "broadcast table size" from 15 to 32 helps with responsiveness when dragging brightness and color sliders in Home Assistant. I think this is a complex situation:

1. If the broadcast table on the coordinator quickly fills up, no new requests can be sent for up to 20 seconds due to the global SiLabs timer. From a user's perspective, this very much makes it look like the network is broken.
2. When dragging a slider, any received broadcast (especially with transitions!) is enough to cause a color change. So if a bulb's broadcast table is full for a few seconds but a new broadcast is admitted a few seconds later (after bouncing around the network a little bit), it should be good enough feedback.
3. If group members are genuinely a few hops away from the coordinator, a sustained broadcast should be "absorbed" by nearby routers and it should stop spreading through the network pretty quickly. I think router density will determine how well these repeated commands will be heard but I think with enough nearby routers with desychronized 5s global timers, they should still effectively propagate the broadcast in such a way that the distant group members can still hear it.
4. The only time I think this will not work is if the mesh has a bottleneck of some sort: there is some single backbone router that all traffic must flow through, with the two network halves unable to hear one another. In that case, there will be global 5s delays, since the broadcast traffic is gated by that single router.
   Overall, I think increasing the Silicon Labs broadcast table size is a net positive, as their global timer approach just results in a bad user experience on the coordinator side, completely delaying all interactive traffic by over 5s after a short burst.

---

I think the practical takeaways here (in my opinion) are:

• we should rely on "stable" broadcast implementations in other routers on a Zigbee network and let the coordinator break the rules a little bit.
• we should also just switch over to source routing to avoid AODV route request storms (since we effectively can only discover routes to 9 devices at a time)
• we should implement route cache restoration on startup
• we should consider rate limiting light transition group commands to account for the conservative SiLabs behavior: 0.75/s long-term, maybe faster for an initial burst. This should really help with the group request flooding when dragging sliders in the HA frontend.

[MattWestb]

Great DEEEEEEP diving and analyze !!!

We still have one underlying problem its the broadcast storm protection in the 15.4 network layer that we cant "fix" so then the 15.4 is blocking all broadcast in the upper Zigbee network and app layer is broken.
Blocking Zigbee APP is one user problem if broadcasting addictive lighting but the real bad that its also blocking Zigbee network = all router and joining handling in the mesh network.

I think its good letting the user tuning its network on there own risk but Zigpy / ZHA shall have "safe" standard settings so then one new user is setting up the system is shall being standard = stable and safe.

And as normal G... W... D... !...

[MattWestb]

Then we is having 2 version of broadcast ack in Zigbee network layer:
Classic Ack = Repeating the frame 3 time = mission done ! and if the same frame is coming back ignore it for not looping.
New Ack: Incoming frame is transmitted and if i here (receive) all my known neighbors is broadcasting it = mission done and if not use classic method plus the filtering for not looping.

[bitcoin1192]

Is source routing really better than aodv ? zboss and zoh, that does source routing, eventually run into slow response issue in my environement...

Ziggurat, which still uses aodv doesn't have this issue, and i believe because the routing that Ziggurat currently uses. if Ziggurat start implementing source routing, will it be toggle-able option user could select ?

[puddly]

Is source routing really better than aodv ?

AODV is very slow. There are a bunch of optimizations that that scavenge next hop information from any incoming packets but it still eventually requires a ton of broadcasts to compute routes. Source routing sends one broadcast every once in a while and it computes a tree.

zboss and zoh, that does source routing, eventually run into slow response issue in my environement...

I would try to figure out what is happening. If you want to enable debug logging while running it, feel free to file an issue with the Ziggurat debug log.

Ziggurat start implementing source routing, will it be toggle-able option user could select ?

It already implements full source routing. It just is controlled by zigpy via configuration.yaml, same as every other radio library:

zha:
  zigpy_config:
    source_routing: true  # default is false