Implement fast userspace multihop mesh routing

Replace the one-intermediate-peer relay with protocol-scoped connectivity
graphs and precomputed shortest-path/ECMP forwarding snapshots. Independent
transport readers forward opaque transit frames without a plugin or TUN
round trip. Carry source, destination, a bounded hop limit and stable flow
tags; preserve end-to-end WireGuard links across topology changes.

Classify IP flows before encryption and preserve their tags through the
WireGuard pending queue. Add offline four-agent path-change coverage, loop
and isolation tests, and an opt-in release forwarding microbenchmark. Bump
control/data ALPNs while preserving persistent network identities and state.

Also include the pending Windows Mainline idle-timeout fix and its regression
test, using a reproducible vendored dependency patch.

Validation: fmt, workspace Clippy with warnings denied, and 307 release tests
passed. Two pre-existing Windows SQLite wipe failures were excluded; public
DHT and the manual benchmark remain ignored by default. The forwarding
microbenchmark measured 103 ns (64 B) and 202 ns (1280 B) per transit packet,
excluding encryption and socket I/O.

docs/architecture.md

@@ -35,6 +35,16 @@ a `PluginContext` for re-announcements and error reports, and a bounded
 A data plane failure never stops the daemon: the control plane keeps running
 and the agent stays manageable.
 
+## Userspace routing
+
+The routing engine builds shortest paths over opaque peer identifiers, with no
+WireGuard or iroh dependency. The relay adapter binds those next hops to
+`PacketLink` handles and publishes an immutable table per network and protocol.
+One reader per raw transport forwards transit without entering the plugin or
+TUN. End-to-end plugin links survive physical link changes. The existing
+authenticated control mesh supplies first-hand topology; the data router does
+not tunnel control sessions. See [routing.md](routing.md).
+
 ## Module responsibilities
 
 | component | responsibility |
@@ -47,7 +57,9 @@ and the agent stays manageable.
 | `storage` | mandatory state and the separately recoverable cache |
 | `state` | signed records that outlive a session, merged between replicas |
 | `dataplane::transport` | authenticated datagram links to peers; where reachability lives |
-| `dataplane` | the contract a protocol implements, and nothing else |
+| `dataplane::routing` | transport-independent graph, shortest paths and opaque flow identifiers |
+| `dataplane::relay` | immutable forwarding snapshots and transport-to-transport transit |
+| `dataplane` | the contract an IP protocol implements |
 | `overlay` | the one interface an agent owns: provisioning, the TUN, whose packet is whose |
 | `dns` | the DNS view of a network, and telling the system resolver about it |
 

docs/protocol.md

@@ -9,7 +9,7 @@ Two versions exist and are independent:
 
 - **Identity scheme**, `tsunagi-network-id-v1`. Frozen. Changing it creates a
   different network space for the same name and secret.
-- **Control protocol**, ALPN `tsunagi/ctrl/1`, `PROTOCOL_VERSION = 1`.
+- **Control protocol**, ALPN `tsunagi/ctrl/2`, `PROTOCOL_VERSION = 2`.
 
 Upgrading the crate or bumping the control protocol must never change an
 existing `NetworkId`.
@@ -135,7 +135,7 @@ event does not report a network id.
 ## The data plane protocol
 
 IP plugin packets never travel on a control connection. They use their own
-ALPN, `tsunagi/data/3`, on their own iroh connection:
+ALPN, `tsunagi/data/4`, on their own iroh connection:
 
 ```text
 initiator -> responder : (the same membership handshake as above)
@@ -172,9 +172,35 @@ holding up subsequent packets. Packet IDs are scoped to a QUIC connection;
 reassembly is created only after the membership handshake. All limits live in
 `config.rs`. There is no transport-layer retransmission added by this framing.
 
-Version 3 is incompatible with previous data ALPNs. Upgrade both endpoints
-and intermediate peers together. The control protocol, network secret,
-device identity and stored network configuration are unchanged.
+Version 4 adds a routing envelope inside the transport fragmentation framing:
+
+| field | bytes | encoding |
+|---|---:|---|
+| envelope version | 1 | `1` |
+| remaining hops | 1 | initially `16`; transit decrements, drops at `1` |
+| source peer | 32 | original endpoint public key |
+| destination peer | 32 | final endpoint public key |
+| flow id | 8 | big endian opaque stable hash supplied by the plugin |
+| payload | remaining | end-to-end encrypted plugin bytes |
+
+The fixed header is 74 bytes. Zero/oversized hop limits, unknown versions,
+short frames, oversized datagrams and unknown sources are rejected. A frame
+addressed here is delivered to its source's protocol inbox; transit goes straight
+to its next transport. The network and protocol are bound to the authenticated
+transport connection, not supplied by the frame. Intermediate nodes cannot
+decrypt or authenticate the inner WireGuard payload; the destination does that.
+Flow ids are routing hints, not authorization proofs.
+
+Control ALPN 2 carries `Reachable { links: [{ peer, protocol }] }`. Each row
+belongs to the authenticated sender and is replaced atomically, expires after
+90 seconds, and is withdrawn on session closure. Only compatible authenticated
+members enter a protocol's graph; local edges always come from actual links.
+Announcements refresh on the existing control maintenance interval and are
+sent immediately on link changes. Topology stays volatile, outside signed state.
+
+Upgrade both endpoints and intermediate peers together. Older control and data
+ALPNs cannot interoperate. The network secret, device identity, identity
+derivation, transcript encoding and stored network configuration are unchanged.
 
 Separate connections mean separate congestion control, so a saturated data
 plane cannot delay control messages, and a data plane failure cannot take the
@@ -193,6 +219,8 @@ not affect other networks.
 | `Ping { seq, payload }` | small request used to verify the exchange |
 | `Pong { seq, payload }` | the echoed reply |
 | `State { records }` | a snapshot of signed records, merged into what the receiver holds |
+| `Peers { peers }` | unverified member address hints, requiring their own handshake |
+| `Reachable { links }` | sender's current direct data links, scoped by protocol |
 | `Bye { reason }` | graceful goodbye; not a revocation of anything |
 
 A `State` snapshot is merged, never substituted: an author missing from it is

docs/routing.md

@@ -0,0 +1,90 @@
+# Userspace mesh routing
+
+```text
+local OS → TUN → IP plugin (encrypt) → router → transport peer link
+transport peer link → router → transport peer link       (transit)
+transport peer link → router → IP plugin (decrypt) → TUN → local OS
+```
+
+The TUN is the boundary of the local host. The kernel sees one overlay route;
+it never forwards transit traffic. WireGuard encryption remains end to end.
+The core router sees only an opaque payload and routing metadata. iroh is the
+current transport adapter, with fragmentation below the routing envelope so
+the default 1280-byte interface MTU also works on small QUIC paths.
+
+## Graph and lifecycle
+
+Each authenticated control session advertises that peer's own direct data
+links, including the plugin protocol. Tables are separate for each network
+and protocol; only members offering a matching enabled protocol version enter
+the graph. A snapshot refreshes on the maintenance interval, expires after
+90 seconds, and is withdrawn when the session ends. Link arrival, closure and
+changed announcements rebuild routes immediately. Unchanged announcements
+refresh their age without republishing the forwarding snapshot.
+
+Breadth-first search computes the shortest directed paths and every equal-cost
+first hop in sorted order. An actual direct link always wins. The first row is
+always supplied by local transport state, never by a remote claim. Routes stop
+at 16 links. A packet also carries a decreasing hop limit, bounding temporary
+loops if different nodes have not yet received the same topology update.
+
+The existing control plane still forms authenticated pairwise sessions.
+Routing provides multihop **data** paths among these members; it does not add
+control-plane flooding or carry control sessions through the data plane.
+Announcements are volatile, not durable membership or availability guarantees.
+
+## Packet path
+
+The control loop binds next hops to transport handles in an immutable table
+and publishes it through `ArcSwap`. Readers do not acquire the topology mutex.
+The transit routine validates the fixed 74-byte envelope, looks up the source
+and destination, selects a cached next hop, decrements one byte, and calls the
+transport's synchronous datagram send. It never searches the graph, formats an
+endpoint string, validates an Ed25519 key, decrypts, or touches TUN.
+
+An exclusively owned receive buffer is reused when decrementing the hop limit.
+Shared buffers require a copy through the safe `bytes` API. There is no added
+transit queue or timer; every raw transport has its own reader. Readers yield
+after 64 ready packets to avoid starving other runtime tasks. Local delivery
+uses a bounded 256-datagram inbox and drops a new packet when full. Transport
+queues and fragment reassembly retain their own limits.
+
+`PacketLink::send_flow` accepts an opaque 64-bit flow identifier. IP plugins
+derive it before encryption from source/destination addresses, protocol and
+TCP/UDP ports. WireGuard keeps tags alongside its bounded pending queue, so
+the first application packets retain their flow identity after a handshake.
+Transit preserves the tag; it never hashes changing ciphertext. Handshake and
+keepalive frames use flow zero. Fragmented IP traffic uses a coarse address/
+protocol hash because later fragments lack ports; fragmented flows between
+the same addresses coalesce. A transition between fragmented and unfragmented
+traffic can change its path. The default overlay MTU avoids needing IP
+fragmentation for ordinary host TCP traffic.
+
+ECMP is deterministic for a source and flow while the table is unchanged.
+Closed next hops are skipped until the control loop replaces the table. A
+topology change can move a flow; datagrams remain unreliable and unordered,
+and there is no promise to preserve order across a physical path failure.
+Logical peer links and end-to-end encryption state survive these changes.
+
+## Verification and measurement
+
+The default offline suite exercises four real agents, iroh transports and
+WireGuard tunnels with memory TUNs, plus graph/forwarder unit tests. It needs
+neither administrative rights nor public relays/DHT.
+
+Run the forwarding microbenchmark explicitly:
+
+```sh
+cargo test --release -p tsunagi --lib forwarding_benchmark -- --ignored --nocapture
+```
+
+It processes one million 64-byte frames and one million 1280-byte frames with
+two equal next hops. Frame construction is outside the timed section; the
+actual ingress routine, header validation, snapshot lookup, ECMP, TTL update,
+buffer release and mock transport submission are inside. The mock transport
+counts sends without storing frames. Results are CPU forwarding cost, **not**
+end-to-end network latency or VPN throughput; encryption, fragmentation,
+sockets, congestion and scheduling contribute separately.
+
+Wire compatibility: control ALPN `tsunagi/ctrl/2`, data ALPN `tsunagi/data/4`.
+Upgrade every participant together; saved identities and network state persist.

docs/testing.md

@@ -63,15 +63,22 @@ intermediate peer. Fragment tests cover reordering, duplicates, loss, changing
 fragment sizes, malformed input, timeout and memory bounds. These are packet
 transport checks, not a claim to have run an SSH server or a host TCP stack.
 
-`crates/tsunagi/src/dataplane/relay.rs` has its own tests for the way
-through a peer in the middle: the wrapping and what a malformed one does, a
-direct path being preferred over a hop, the middle passing a datagram on
-without being handed it, what arrives through somebody reaching the peer it
-came from rather than the one that carried it, a link outliving the paths
-under it, and the datagram size not changing when the path does.
-`tests/wireguard.rs` proves it end to end — two agents that can each reach a
-third and not each other, with a real WireGuard packet crossing through the
-middle.
+Routing tests cover shortest paths, direct preference, deterministic per-flow
+ECMP, link loss, protocol isolation, malformed/unknown frames, bounded local
+queues, zero-copy transit of an owned buffer, and deliberately inconsistent
+tables whose loop terminates at the hop limit. A transit reader is exercised
+without any plugin reader, so forwarding cannot accidentally depend on one.
+WireGuard tests also cover flow tags queued before a handshake, queue overflow,
+and their preservation through encryption.
+
+`tests/wireguard.rs` includes a four-agent chain A—B—C—D with only adjacent data
+links. Real encrypted 1280-byte TCP packets travel in both directions while the
+middle TUNs remain empty. A direct A—D link is enabled, then removed; the route
+switches back to the chain without replacing end-to-end tunnels.
+
+The ignored `forwarding_benchmark` measures the synchronous transit routine in
+release mode, excluding crypto and socket I/O. Run it explicitly as described
+in [routing.md](routing.md); it has no timing threshold in the default suite.
 
 Unit tests in `crates/tsunagi/src/state/` cover the signed record model directly: tampering
 with any field breaks verification, a newer version wins while an older one
@@ -123,6 +130,10 @@ What the default suite does **not** cover is the real TUN interface, because
 that needs `CAP_NET_ADMIN`. Everything above it does run.
 
 Mainline discovery tests use an isolated loopback `mainline::Testnet`.
+`tests/mainline_socket.rs` checks that idle UDP receive timeouts do not emit
+warnings (including Windows error 10060) and that the same node still answers
+a real KRPC ping afterwards. The local dependency correction is documented in
+[`vendor/mainline/PATCHES.md`](../vendor/mainline/PATCHES.md).
 `tests/mainline.rs` restores two agents after their DHT and disposable cache
 have disappeared. `tests/discovery_lifecycle.rs` checks that publication
 continues while connected, lookup resumes after isolation, and a hung backend

docs/wireguard.md

@@ -184,7 +184,7 @@ QUIC DATAGRAM frames themselves cannot fragment.
 Reassembly is bounded and expires incomplete packets; one lost fragment loses
 one packet, without blocking unrelated traffic. The logical data payload limit
 is 64 KiB, with the relay envelope subtracted before it reaches the plugin.
-The new framing uses data ALPN `tsunagi/data/3`; both ends and intermediate
+The new framing uses data ALPN `tsunagi/data/4`; both ends and intermediate
 peers need the updated binary. Network identities and saved state do not change.
 See [protocol.md](protocol.md#the-data-plane-protocol) for the wire format.