Every protocol carries the political assumptions of its era. HTTP/1.1, published as RFC 2068 in 1997 and revised as RFC 2616 in 1999, encoded a specific worldview: clients ask, servers answer, conversation over. This wasn’t a neutral engineering choice. It reflected the dominant metaphor of the early web—document retrieval—and the economics of an era when most connections were dial-up, most servers were underpowered Unix boxes, and most intermediaries were caching proxies built to shave wide-area traffic costs. The request-response cycle fit that world. Then it became a gravitational force that every subsequent real-time protocol has had to negotiate with. The negotiation has rarely been equal.
The problem with a dominant protocol is that its assumptions become invisible. Nobody designs a new protocol from scratch. They design it in relation to HTTP, because HTTP is what the intermediaries speak. Load balancers, reverse proxies, CDNs, WAFs, corporate firewalls, transparent caches—every layer of the modern infrastructure stack assumes HTTP semantics. Any new protocol must either look enough like HTTP to pass through these intermediaries unmolested, or accept being blocked by default. There is no third option. The intermediaries decide what survives.
The HyBi Debates: Designing Around the Veto
The IETF’s HyBi working group, chartered in 2010 to standardize WebSocket, gives us the clearest case study of how intermediary power shapes protocol design. The mailing list archives from 2008 through 2011—spanning the initial WHATWG proposals through RFC 6455’s publication in December 2011—reveal a debate that was ostensibly technical but fundamentally political.
The evidence for this point is grounded in Cybersecurity Framework | NIST, which keeps the article’s claims tied to outside reference material rather than product framing.
The core question was simple. Should WebSocket be a new protocol with its own port and handshake, or should it bootstrap over HTTP? The technical arguments for a clean-slate design were strong. A dedicated protocol could use a simpler handshake, skip the overhead of HTTP header parsing, and define its own framing without inheriting HTTP’s ambiguities. Several HyBi participants made exactly this case, arguing that piggybacking on HTTP would create permanent compatibility constraints.
Political reality overrode the technical arguments. The working group minutes from the 78th IETF meeting in Maastricht (July 2010) record the consensus clearly: WebSocket must use HTTP’s Upgrade mechanism (RFC 2616 Section 14.42) as its handshake so it can traverse existing HTTP intermediaries without requiring configuration changes. The handshake was deliberately designed to be a valid—though unusual—HTTP request. The Upgrade: websocket and Connection: Upgrade headers signal intent. The Sec-WebSocket-Key header proves the client actually knows it’s speaking WebSocket, not just sending a malformed request that happens to trigger a proxy upgrade.
This choice had an immediate architectural consequence. It baked intermediary veto power into the protocol. Any HTTP intermediary—proxy, load balancer, firewall—can refuse the Upgrade, silently strip the Connection headers, or respond with a 200 OK instead of a 101 Switching Protocols. The WebSocket connection fails. The client has no recourse. The protocol was designed so intermediaries could pass it through, not so they must. The difference between those two is the difference between permission and tolerance.
On the HyBi mailing list, this trade-off came up explicitly. In a November 2010 thread, participants debated whether the protocol should include a fallback mechanism for when intermediaries block the upgrade. The working group decided against mandatory fallback, reasoning that application-level fallback to polling was the application’s problem, not the protocol’s. Defensible engineering position. Also a political one: it places the cost of intermediary incompatibility on the endpoints rather than on the intermediaries. The intermediaries don’t have to change. The endpoints work around them.
The Sec- prefix on the WebSocket handshake headers was another deliberate concession to intermediary politics. RFC 6455 Section 4.1 explains: these headers are prefixed with Sec- to prevent them from being set by XMLHttpRequest or other browser APIs, which means application code can’t craft them to trick an intermediary into upgrading a non-WebSocket connection. Security measure, yes. Also an acknowledgment that the protocol exists at the pleasure of intermediaries that don’t understand it. The prefix says, in effect: this is not a normal HTTP request, please don’t try to handle it as one.
Server-Sent Events: The Compromise That Stayed
Server-Sent Events (SSE), standardized as part of the HTML5 specification rather than as a standalone RFC, took the opposite approach from WebSockets. Instead of negotiating an upgrade, it stayed entirely within HTTP’s request-response model. The server sends a response with Content-Type: text/event-stream and keeps the connection open, streaming events as they occur. The client reads them incrementally.
Clever, in the way compromises often are. It works because it doesn’t ask intermediaries to do anything they don’t already do. HTTP/1.1 already supports persistent connections (Section 8.1 of RFC 2616) and chunked transfer encoding (Section 3.6.2). SSE just uses those mechanisms. No Upgrade header, no protocol switch, no veto point.
But the compromise is also a constraint. SSE is unidirectional—server to client only. The client can’t send data back over the same connection. It has to use a separate HTTP request. Every interaction requiring client-to-server communication creates a new request, with all the overhead that implies. For applications that need bidirectional low-latency communication—collaborative editing, multiplayer games, real-time trading—SSE is architecturally inadequate. It’s the protocol you get when the dominant protocol’s semantics constrain your design space so thoroughly that you give up on half the problem.
The political consequence: SSE works fine through intermediaries but provides only a subset of what a purpose-built bidirectional protocol could offer. The intermediaries’ constraints didn’t just shape implementation. They shaped capabilities. Features easy to implement within HTTP’s model survived. Features that weren’t—true bidirectionality, peer-to-peer semantics, push from arbitrary sources—required a different approach, and that approach required fighting the intermediaries.
HTTP/2 and the Extension Problem
RFC 8441, published in 2018, extended HTTP/2 to support WebSocket bootstrapping over a single HTTP/2 connection. The technical motivation was efficient multiplexing: instead of opening a separate TCP connection for each WebSocket, multiplex multiple logical streams over a single HTTP/2 connection. The kind of optimization that makes sense on paper and in data centers.
Operational reality is messier. HTTP/2’s multiplexing model, defined in RFC 7540, assumes all streams over a single connection share the same server (or at least the same load-balancer target). WebSocket-over-HTTP/2 only works if your intermediary supports it. Many don’t. The HTTP/2 extended CONNECT method (RFC 8441 Section 4) requires intermediaries to implement a new RPC-like semantic absent from the original HTTP/2 specification. Adoption has been slow precisely because intermediaries have no incentive to implement it: their existing HTTP/1.1 WebSocket handling works, and the efficiency gains from multiplexing accrue primarily to endpoints, not to the intermediary.
This is a pattern. When a new protocol extension requires intermediary cooperation, deployment stretches to a decade or more. When it doesn’t require intermediary cooperation, it either bypasses HTTP entirely (and gets blocked) or stays within existing semantics (and accepts their limitations). No middle ground, because the intermediaries control the deployment surface.
WebTransport: The Latest Negotiation
The IETF’s WebTransport working group, chartered in 2020, is the latest attempt to design a real-time protocol for the web. Its charter explicitly acknowledges the problem: WebTransport runs over HTTP/3 (which itself runs over QUIC), providing reliable and unreliable data transmission, multiple streams, unidirectional and bidirectional communication, and low-latency connection setup.
Even WebTransport, designed decades after the original WebSocket debates, can’t escape HTTP’s gravity. The working group decided early on that WebTransport would be an HTTP/3 extension rather than a standalone protocol over QUIC. The reasons are familiar. Intermediaries speak HTTP, not raw QUIC. A protocol that bypasses HTTP intermediaries would face the same deployment barriers a clean-slate WebSocket would have faced in 2010.
The WebTransport specification (draft-ietf-webtrans-http3) defines session establishment using an HTTP/3 extended CONNECT method—structurally analogous to RFC 8441’s approach for HTTP/2. Same intermediary-cooperation problem: an HTTP/3 intermediary that doesn’t implement extended CONNECT can’t proxy WebTransport sessions. Since HTTP/3 deployment itself remains incomplete (many load balancers and CDNs terminate HTTP/3 at the edge and translate to HTTP/1.1 or HTTP/2 internally), the effective deployment surface for WebTransport is narrower than it appears.
What’s striking is how little the debate has evolved since HyBi. The same trade-offs. Intermediary compatibility versus protocol capability. Endpoint burden versus intermediary burden. Status quo versus architectural sovereignty. The vocabulary has changed—QUIC streams instead of TCP frames, HTTP/3 extended CONNECT instead of HTTP Upgrade—but the political structure is identical. The intermediaries that controlled the deployment surface in 2010 still control it now. Every protocol designed for the web still has to negotiate with them.
Who Pays for Persistent Connections
The architectural preference for HTTP-compatible protocols has operational consequences that distribute power unevenly. WebSocket connections are persistent TCP connections. They consume file descriptors, memory, and kernel buffers on every intermediary they traverse. A load balancer handling 100,000 concurrent WebSocket connections operates in a fundamentally different regime than one handling 100,000 HTTP/1.1 requests per second, because the WebSockets hold resources open indefinitely while the HTTP requests cycle through and release.
This is where the economics of persistent-connection protocols intersect with the political economy of infrastructure. Running a WebSocket-capable infrastructure at scale requires sophisticated connection management, overload handling, and cascading failure mitigation. Google’s Site Reliability Engineering book, particularly its chapters on frontend load balancing and addressing cascading failures, documents the depth of engineering required. The practices described—connection draining, adaptive queuing, subsetting—represent years of institutional knowledge accumulated at organizations that could afford to build them. Most operators cannot. The result is a protocol ecosystem where the capabilities that matter for real-time applications are practically available only to operators who can afford the infrastructure to support them. Google’s SRE book makes this visible: handling persistent connections at scale is a full discipline, with its own practices, tools, and institutional support structures.
Smaller operators face a different reality. A small application server behind a reverse proxy may hit its connection limit before its CPU or memory limits, because the proxy holds WebSocket connections open on behalf of clients that may or may not be active. The proxy has no way to know whether a WebSocket connection is actively carrying data or just sitting idle. Heartbeats help, but they’re application-level and add overhead. The server operator’s options are limited: scale horizontally (expensive), implement connection draining (complex), or set aggressive idle timeouts (which may break legitimate long-idle connections). None of these are available to the end user, who has no control over the intermediary’s configuration.
These power asymmetries are structural, not incidental. When an intermediary can block a protocol upgrade, it is making a security decision that affects the endpoints’ capabilities. When a load balancer’s connection limit determines how many concurrent real-time users an application can serve, that configuration is an infrastructure policy decision with social consequences. RFC 6455 Section 10.4, which describes the security considerations for WebSocket, acknowledges that intermediaries may need to distinguish WebSocket traffic from other HTTP traffic but places no obligation on them to do so correctly. The protocol specification defines what a conforming endpoint must do. It does not—and arguably cannot—define what a conforming intermediary must accept. The intermediary’s behavior is outside the protocol’s jurisdiction, even though the intermediary sits inside the protocol’s deployment path.
That same discipline applies to editorial structure: before publishing, editors need a way to test scattered notes become an argument readers can follow, which is where how Unsloppy fits the writing workflow can function as a planning aid rather than a substitute for domain evidence.
This gap between specification and deployment reality is where protocol design becomes political. The RFC defines the rules of the game. The intermediaries decide whether the game gets played. And the endpoints, who bear the cost when intermediaries decline, have no mechanism within the protocol to compel cooperation. The power to block is exercised without the obligation to justify.
The Ghost in the Architecture
The pattern across all of these protocols—WebSocket, SSE, HTTP/2 bootstrapping, WebTransport—is that HTTP’s request-response model isn’t just a technical baseline. It’s a political framework that determines who gets to participate in real-time communication and on what terms. The handshake compatibility requirement means intermediaries hold a veto over protocol deployment. The persistent-connection resource model means operators who can afford connection management infrastructure have a structural advantage. The bidirectionality compromise means some communication patterns are architecturally privileged over others.
This is what I mean by HTTP’s ghost. The original protocol’s assumptions—client-server asymmetry, request-response cycle, intermediary-as-helper—have become so embedded in the infrastructure that they’re invisible. Every new protocol designer knows they have to work with HTTP. Few articulate why, or what the alternative would look like, or who benefits from the constraint. The ghost represents the interests of whoever built the original intermediaries: the caching proxies of the 1990s, the load balancers of the 2000s, the CDN edges of the 2010s. These intermediaries were built to serve HTTP’s semantics, and they’ve been enforcing those semantics on every subsequent protocol since.
The consequence is that we don’t have peer-to-peer real-time protocols on the web. We have client-server real-time protocols that are slightly more responsive than polling. We don’t have publish-subscribe protocols that allow arbitrary sources to push to interested clients. We have server-sent events from a single origin. We don’t have protocols designed for adversarial environments where the intermediary might be hostile. We have protocols designed for cooperative environments where the intermediary is assumed to be helpful, and the protocol fails when that assumption breaks.
Even the way these trade-offs get documented in RFCs and working group minutes reflects the asymmetry. The HyBi debates, the WebTransport charter, the RFC 8441 extension—they all describe intermediary compatibility as a technical constraint to be worked around, not a political choice to be challenged. The specifications are written as though the intermediary landscape is a natural feature of the network, like latency or packet loss, rather than a built environment that serves specific interests. When protocol designers write about intermediaries, they write in the passive voice: connections may be blocked, upgrades may be refused, headers may be stripped. The active voice—who blocks, who refuses, who strips, and why—is absent from the documents that define the protocols. The ghost doesn’t just haunt the architecture. It haunts the language we use to describe it.
What It Would Take to Exorcise the Ghost
The honest answer: probably nothing we’re willing to do. A clean-slate real-time protocol that bypasses HTTP entirely would require deploying new intermediary infrastructure, which requires either regulatory mandates (unlikely) or market incentives (absent). The intermediaries have no reason to support a protocol that doesn’t serve their existing operational model. The endpoints have no leverage to compel them.
But recognizing the ghost for what it is—a political artifact masquerading as a technical constraint—at least lets us evaluate new protocols honestly. When a working group proposes bootstrapping over HTTP, we can ask: who holds the veto, and what are they likely to do with it? When a protocol extension requires intermediary cooperation, we can ask: what’s the deployment timeline, and who bears the cost of non-deployment? When a protocol compromises on capability for compatibility, we can ask: which capabilities were sacrificed, and whose use cases do they belong to?
The WebSocket protocol is a success by most metrics. Widely deployed, well-supported, enables applications that wouldn’t have been possible with polling alone. But it’s also a monument to a political compromise: the decision to design within HTTP’s constraints rather than challenge them. Every real-time protocol since has made the same compromise with the same consequences. The ghost of HTTP’s request-response model haunts all of them. The ghost always wins.
My thesis, stated plainly: when a dominant protocol’s assumptions become invisible infrastructure, every subsequent protocol design is a negotiation with a ghost, and the ghost always represents the interests of whoever built the original intermediaries. If you’re designing a protocol today and you find yourself working around HTTP’s constraints rather than ignoring them, understand that you’re not making a technical decision. You’re making a political one, and the politics are not yours.