Chapter 22 / 51
Volume 3 · Protocol Development

WebSocket advanced: broadcast, limits and slow clients

Echo only proves the pipe works; real business asks: how to fan out, how to block giant messages, whether slow clients drag everyone down, and who reaps zombie connections.

Overview

Positioning: operational capabilities growing on ch. 21's registry. The quartet — broadcast + limits + backpressure + idle reclamation — is the standard chassis of chat/push services.

22.1 Intro

WS scaling problems share one root: one connection's state affects every connection — a client sending huge messages eats memory, a client that never reads piles up send buffers, a zombie connection squats in the registry. The host supplies the mechanisms (ws_message_limit, idle handling); broadcast strategy and registry governance are the application's job — this chapter is exactly those two.

22.2 Broadcast patterns

Fan-out by walking the registry (on ch21's infrastructure)
static void BroadcastText(xstrview tMsg)
{
    xmapiter it;
    xrtMutexLock(&G_WsLock);
    /* walk the registry: send each connection its own copy */
    for ( it = First(G_WsConns); !AtEnd(G_WsConns, it); Next(G_WsConns, &it) ) {
        xwsstream* pWs = KeyOf(it);
        (void)xrtWsStreamText(pWs, tMsg);  /* copying: the buffer is reusable at once */
    }
    xrtMutexUnlock(&G_WsLock);
}

What just happened — three engineering decisions inside:

  • Walk under the lock: prevents connections opening/closing mid-walk from mutating the table (trim lock granularity to your registry operations)
  • Copying sends: the message buffer is reusable immediately after broadcast — no waiting for every connection's async send to finish
  • Failure tolerance: one connection's send failure doesn't break the broadcast — failed connections leave the table naturally via their Close terminal state

Channel-based broadcast (chat rooms) just replaces the registry with a two-level channel → connection set structure — the pattern is entirely isomorphic.

22.3 Limits & subprotocols

MechanismConfigBehavior
Message capws_message_limit (server-level Custom)accumulated overflow → close 1009 (Message Too Big); overflow checks prevent bypass
Subprotocol enforcementws_protocolhandshake mismatch → 400; config validity checked at startup
OOM—server-side allocation failure → close 1011
Accumulator slimming—message accumulators >256KB are freed when the message ends, not kept resident

1009's semantics are "client, stop sending huge messages" — the limit equals your business's sane per-message ceiling (64KB for chat, 1MB for market-data pushes); don't guess with the default.

22.4 Slow clients & backpressure

  • WS-layer sends are asynchronously buffered: a slow-reading client queues data on the send side — the memory defense lines are the message cap and your broadcast-rate control
  • Governance policy is yours: connections exceeding N unsent messages get demoted (skip non-critical pushes) or closed outright — track a "demotion flag" in the registry
  • Idle reclamation: on idle_timeout expiry the driver closes via a cross-thread-safe Abort (not Close — idle scans may run on any worker; Abort is the thread-safe terminal path)
  • Your registry cooperates: removal happens in the WsClose terminal callback — every death cause (client drop, limit, idle, your own kick) flows through the same cleanup path
Layered memory: the frame layer's backpressure belongs to xrt (buffers, reentry); the policy layer's belongs to you (who gets demoted, who gets kicked); the time layer's defense is idle_timeout. Three layers, three jobs — don't pile them all into business code.

22.5 Contract

  • Broadcast: walk under the lock, copy-sending, failure-tolerant; cleanup flows uniformly through the WsClose terminal.
  • Limits: 1009/1011 are sent by the host; set and document limit values per business.
  • Reclamation: idle closes use Abort (cross-thread safe); your registry trusts only the Close terminal.

Common mistakes

Walking the registry outside the lock
Symptom
Sporadic walk crashes, missed or duplicated sends under concurrency.
Cause
WsOpen/WsClose mutate the table on other workers simultaneously — the walk must complete under the registry lock.
Close-callback semantics invoked from a non-WS thread
Symptom
A homegrown cleanup thread touches connection objects directly and crashes.
Cause
Cross-thread connection operations must go through thread-safe paths like Abort; your cleanup logic belongs in the WsClose callback, not in poking connections yourself.

Exercises

BasicChat-room broadcast

Implement multi-client chat: any client's message is broadcast to everyone else (itself excluded).

Acceptance: 3 concurrent clients see each other; one client dropping doesn't interrupt broadcasts.
AdvancedThree limit states

Set ws_message_limit=1MB; send 0.9MB, 1MB and 1.1MB messages and record behavior; then set ws_protocol="chat.v1" and verify handshake rejection.

Acceptance: the three states match the 22.3 table; mismatched-subprotocol clients get a 400 handshake.
ChallengeSlow-client demotion

Build a client that never reads; implement "3 consecutive broadcast failures → demote for 30 seconds".

Acceptance: the demoted slow client receives no non-critical pushes; resuming reads lifts the demotion automatically.

Summary

PointConclusion
Broadcastwalk under the lock + copy-sending + failure tolerance; channels are two-level registries
Limitsws_message_limit → 1009; OOM → 1011; values per business
Subprotocolws_protocol enforced, 400 at handshake
Backpressure layersframe layer xrt / policy layer application / time layer idle_timeout
ReclamationAbort is cross-thread safe; cleanup unified in the WsClose terminal
API references in this chapter
ws_* knobs · 2 items xrt · websocket_stream
xrt further reading