Ask a room full of game-network engineers how to run physics on a peer-to-peer world and you will get two answers. The first is "make one node the server" — clean, safe, and exactly the wrong shape for a swarm where every peer is supposed to carry its share of the work. The second is "make it deterministic and lockstep the inputs" — clean in a different way, until you realize every client now simulates everything, which multiplies the compute instead of spreading it. For the past few weeks we have been building a third answer into the PI swarm, and this week its wire protocol shipped: every entity delta now carries a transferable, arbitrated authority stamp, and the seeder that owns a zone is the court that decides who may write it.
This is a dev-log on where the swarm stands — the fabric that got us here, the dynamic-authority model that went in as protocol v6, and the mirror-seeder relay machinery that carries it all.
The fabric that was already there
The PI swarm did not start as a game netcode — it started as a training swarm. pi_daemon nodes find each other over LAN discovery, authenticate with a pre-shared key, and pull model-shard work from a seeder that owns the authoritative trainer. Chat rooms, whispers, and presence came next, then player sessions, then entity replication. By the time physics became the question, the fabric already had three transports doing three different jobs:
SwarmEvent envelopes across seeder links.On top of that sits the zone map: a consistent-hash (jump-hash) ring over the live seeder set that answers "which seeder owns zone Z" identically on every node. Zones are the spatial shard of the world; ownership is deterministic, so it never needs an election. The leaf client that speaks this protocol today is Clowder — our network portal GUI — which connects to any seeder directly over TCP; everything below is what happens under its socket.
How mirror seeders actually work
A single seeder is a broker. Two seeders are a fabric. The thing that stitches them is the mirror relay — and it is worth describing precisely, because it is what makes authority records meaningful across a real network.
A mirror seeder opens a long-lived TCP link upstream and presents itself with two roles at once: CLIENT — so the upstream treats it as a normal authenticated peer and sends it a full welcome — and SEEDER_LINK, which marks the connection as node-to-node relay plumbing rather than a leaf:
Five details make the relay work:
discovery_port field in the mirror's hello is where it wants entity datagrams sent. The upstream learns the port once, caches the resolved address, and floods entity state there every tick — no per-datagram DNS, no endpoint guessing.event_id; chat broadcasts carry a msg_id that stays constant across hops. A message making a full circle around a cyclic topology is dropped on dedup, not amplified. Whispers route by node_id across the link to whichever seeder holds the target.Entity state crosses the same link on the UDP plane: every 50 ms tick, each seeder packs its zones into signed datagrams and sends them to every downstream link peer plus its upstream mirror. The receiving seeder merges them into its own replication table — and then its own tick re-floods them onward, which is how state crosses a multi-hop fabric without any node needing to know the topology. Despawn tombstones ride the same datagrams and are acknowledged back to their origin by node id, piggybacked on the ack trailer — so a NAT'd or multi-homed peer still gets credit for applying a removal. Bulk content rides the control plane separately: SWARM_ASSET_REQ pulls ranged byte windows of any file in the seeder's asset directory — the substrate for multi-source fetching once a piece-picking policy lands.
This relay design is what makes the next piece possible. If authority were a property of a connection — "whoever sent this packet may write" — it would die at the first relay hop. We needed authority to be a property of the entity itself.
The problem with client-authoritative physics
Until this week, the swarm's entity model was honest but trusting: a client sends GAME_UPDATE with a position and a sequence number, its host seeder checks the session owns the entity and the rate is under 60 updates a second, stamps it into the table, and floods it. It is the classic client-writes, server-forwards arrangement — and it has a hole you could fly a ship through. A modified client can write any position it likes (teleport), and because merges are latest-seq-wins, it can also write any sequence number it likes (win every merge forever). Session binding and rate limiting police the envelope, not the payload.
Pure server authority closes the hole but breaks the swarm: if only the zone owner may write, then every physics simulation — for every object, for every player — concentrates on seeders while clients become dumb terminals. For a network whose entire point is that peers carry the load, that is a contradiction.
"The question is not who owns the physics. It is who owns the write — per entity, provably, and changeably."
The literature has a name for the middle path: dynamic-authority distributed simulation. Glenn Fiedler picked exactly this model for his Oculus-sponsored networked-physics-in-VR research — players take authority over the cubes they interact with and broadcast those cubes' state; everyone else snaps that state onto their non-authority replicas. He rejected lockstep for the same reason we did: at a 90 Hz simulation rate with up to 250 ms of latency to hide, prediction would need ~25 re-simulation steps per rendered frame — a 25x compute tax on an already-CPU-heavy sim.
What shipped: authority as a field, not a role
Protocol v6 adds four bytes to every entity delta. That is the whole wire change — and it changes everything downstream:
entity_id u32 fabric-unique: (node_id & 0xFFF) << 20 | counter zone_id u32 spatial shard the entity lives in seq u32 per-entity sequence — latest wins pos[3], vel[3] f32 metres, metres/second yaw, anim f32+u16 flags u16 bit0 despawn, bit1 session-bound authority_node u32 NEW — the seeder authorized to write this entity
The rule that consumes it is short enough to hold in your head. When a delta arrives at a replica:
authority_node == 0 — drop as malformed before anything mutates.rogue_entity_writes().Because the stamp travels with the record, a relay's job stays purely mechanical: verify the datagram's HMAC, apply each delta by its own declared writer, forward verbatim. The sender of the packet is irrelevant to who may write — which is precisely what lets state cross a multi-hop fabric.
Claims, arbiters, and grants
A stamp that can never change is just a name tag. The point is that authority moves — and the movement is arbitrated:
Two of the arbiter's rules deserve a sentence each because they were found the hard way — by roasting the implementation, not by design review:
- The claim's zone must be the entity's real zone. The arbiter is selected by the zone id inside the claim — so a seeder that owned any zone could have named it, made itself the arbiter, and granted its own claim. The arbiter now requires the claim's zone to equal the entity's stored zone: you can only arbitrate entities actually standing in a zone you own.
- Grant epochs are wall-seeded. A plain per-arbiter counter restarts at zero on a seeder restart — replicas would then drop every new grant as "stale" forever. Epochs are now
max(wall_seconds, last+1): monotone through rapid grants, arbiter handoffs, and restarts.
And the shield: every delta a host seeder writes for a client-attached entity carries FLAG_SESSION. Session entities are never transferable — a player's own entity is bound to its host by definition, so the arbiter denies claims on them outright. Client-supplied reserved flag bits are masked at ingest, so a client can neither despawn itself through GAME_UPDATE nor peel off its own session mark.
What this buys: the peer-node physics integrator
Here is the payoff — the model the post title promises. With authority as a transferable, arbitrated entity property, the swarm's physics division of labor becomes:
| Actor | Role in the physics integrator |
|---|---|
| Client's host seeder | Proxies the client's authority: stamps its own node id + SESSION on client-driven deltas. The client moves; the seeder signs. |
| Claim-holding seeder | Simulates the physics for world entities it has been granted — its own rigid-body shard of the world. The integrator work lands on N seeders' attention sets, not one. |
| Zone owner | Arbitrates contested authority. Roadmap: simulates unclaimed entities as fallback + shadow-validates authority output. |
| Relays / leaves | Apply deltas whose declared writer holds the grant; forward verbatim; count rogue writes. No physics, no authority — just verification-shaped plumbing. |
Today these roles are plumbing, not computation — the authority layer is live, but no node runs a physics integrator against it yet. The division activates as 8b lands the simulation posture; what 8a guarantees is that when the sim arrives, the write rules are already enforced at the wire.
This is the "best of both worlds" in concrete form. Like lockstep, the honest nodes all agree on who is allowed to say what. Unlike lockstep, each participant only simulates the slice of the world it holds authority over — compute spreads across the swarm the same way bandwidth already did. And unlike raw client authority, a forged write is not a merge candidate, it is a counter.
The physics engine itself is ready for this division — that part was already true, and it is what made the model plausible rather than aspirational. The V Engine's ~21 K-line rigid-body system runs a fixed 60 Hz timestep with spatial-grid broadphase, warm-started manifolds, contact islands, and a PGS solver — and, critically, it is ordering-deterministic end to end: islands and manifolds are sorted deterministically, job-graph merges follow a fixed order, and pseudo-random directions are entity-seeded. Ordering determinism is the hard half of distributed simulation — the half ODE famously lacked when Fiedler had to re-seed its RNG per frame to make lockstep work at all.
The research the model stands on
None of this is novel — which is the point. The architecture is a deliberate composition of proven ideas:
| Work | What it contributes to our model |
|---|---|
| Fiedler — Networked Physics in VR (Oculus-sponsored) | The core posture: distributed simulation where players take authority over objects they interact with and broadcast state; chosen over lockstep specifically to avoid rollback re-simulation cost on a CPU-heavy sim. |
| Walsh, Ward & McLoone — physics-aware dead reckoning (ISSC 2012 + Walsh's MSc thesis, NUI Maynooth) | The formal version: a dynamic authority scheme for shared physics objects is shown to reduce overall inconsistency in peer-to-peer distributed interactive applications; physics-aware dead reckoning reduces instantaneous divergence between handoffs. |
| Bharambe et al. — Donnybrook (SIGCOMM 2008) | Attention-set-based update prioritization + multi-source dissemination; simulations demonstrated viability up to ~900 players (user study on a Quake III prototype). Their attention sets are focus-computed per player; our spatial AoI radius is a simpler instance of the same interest-management family. |
| Brown, Ushaw & Morgan — Aura Projection (ACM I3D 2019) | Physics sharded by spatial region across servers with predictive object migration — our zone_id → jump-hash owner map plus ZONE_HANDOFF is the same region→authority machinery, adapted to P2P. |
| GGPO / deterministic lockstep | The road not taken: perfect consistency at minimum bandwidth, but every peer simulates everything and the whole model demands bit-exact simulation — the gate we deliberately defer to Phase 8c. |
| Photon Quantum | The production reference for where this ends: deterministic ECS with verified vs predicted frames — our "arbiter grants + shadow validation" is a coarser-grained version of the same verified/predicted split. |
| Fiedler — Floating Point Determinism | The honest gate for Phase 8c: FP determinism is achievable — a Gas Powered Games engineer's first-hand account reports SupCom 1+2 and Demigod relying on IEEE754 across 1M+ customers — but the actual recipe is narrow: same compiler, same instruction set, enforced FPU precision/rounding mode after external-library calls. Not a posture you get by accident. |
Shipped vs roadmap — read this before quoting us
implementedauthority_node on every delta; the mint-tag first-sight gate; rogue-write rejection + counting; PHYS_AUTH_CLAIM/GRANT gossip with zone-owner arbitration; epoch-ordered grants; session-entity shields; entity_spawn/entity_write/claim_entity_authority APIs; seeder-side world entities that are claimable by construction. Verified over real mirror links: a claim crosses the gossip bus, the grant lands on claimant and displaced authority, post-grant writes replicate, and the old owner's write is rejected as rogue. Session-entity claims are denied on both nodes of a link.
roadmap Client-authority physics execution (8b): the plumbing assumes a seeder writes for its client — letting a client compute physics for objects it is granted, with the seeder shadow-validating output (dt-bounded speed, continuity, static-geometry sanity) and revoking chronic violators, is the next phase. Verifier re-simulation (8c): bit-exactness discipline plus per-frame state hashes so designated peers can re-simulate a shard and prove desync — that is when the strict-FP question gets answered, and not before there is a real sim to validate against.
Caveats and limits
- First-claim-wins is a deliberate 8a policy, not a security boundary. Any live fabric member can take any non-session entity — a malicious seeder inside the PSK perimeter could steal world objects. It is observable (grants gossip publicly; rogue writes are counted) but not prevented. Insider theft resistance belongs to 8b/8c, not to this layer.
- Claims are fire-and-forget. A claim denied while zone membership is still converging is not retried — callers re-issue. Grant ping-pong between two racing claimants is possible; each flip requires a fresh claim, so it is rate-bounded but not eliminated.
- Authority is trusted once granted. Nothing yet validates the content of an authorized write — a granted-but-compromised seeder can write legal nonsense. Shadow validation (speed/continuity clamps) is the roadmap answer.
- Datagram replay is self-limiting, not prevented. A captured HMAC-valid datagram can be replayed, but latest-wins sequence ordering plus tombstone dead-guards absorb it: a replayed packet carries equal-or-older state than what already landed, so it drops at the same gate as any stale frame. A nonce/timestamp window would tighten this further; it is documented future work.
- Byte sizes cited are the packed wire sizes (
#pragma pack(1), little-endian). Cross-architecture byte order is documented future work, same as the rest of the protocol. - The ~21 K-line physics engine and the ordering determinism are verified; bit-exact cross-machine determinism is not. The GPU compute path is render-side acceleration — any future replicated sim runs on the CPU-canonical path under strict-FP build flags.
- Physics-aware dead reckoning citation: the authority-bounds-inconsistency result is from Walsh, Ward & McLoone's ISSC 2012 paper (and Walsh's associated MSc thesis at NUI Maynooth — the thesis link is the accessible primary source). The adaptive-error-threshold dead-reckoning half of their model is cited for completeness — our replicas currently use latest-wins absorption, not predictive dead reckoning.
- The Quantum analogy is ours, not Photon's. "Arbiter grants + shadow validation" mapping to a verified/predicted split is our comparison of architectures — Photon describes Quantum strictly as deterministic predict/rollback with server-validated input, and does not frame it as authority arbitration.
- SupCom-scale FP32 lockstep is a first-hand account by a Gas Powered Games engineer quoted in Fiedler's determinism post — cited as evidence the discipline is commercially viable, not as a measurement we re-ran. Note that account reports GPG compiled with /fp:fast; "strict IEEE mode" is Fiedler's framing of the general discipline, not the specific GPG recipe.
Cross-references and further reading
Final thought
Server authority asks "which machine is in charge?" Distributed simulation asks a better question: "which machine is in charge of this thing, right now, and who agreed?" Answering it inside the state record — rather than in a connection table — is what lets a swarm replicate physics the same way it already replicates chat: hop by hop, verified at each step, honest about who wrote what. Phase 8a is four bytes and an arbitration loop. Everything it enables is much bigger.