ab
b4f3e57c8d
Replace the one-intermediate-peer relay with protocol-scoped connectivity graphs and precomputed shortest-path/ECMP forwarding snapshots. Independent transport readers forward opaque transit frames without a plugin or TUN round trip. Carry source, destination, a bounded hop limit and stable flow tags; preserve end-to-end WireGuard links across topology changes. Classify IP flows before encryption and preserve their tags through the WireGuard pending queue. Add offline four-agent path-change coverage, loop and isolation tests, and an opt-in release forwarding microbenchmark. Bump control/data ALPNs while preserving persistent network identities and state. Also include the pending Windows Mainline idle-timeout fix and its regression test, using a reproducible vendored dependency patch. Validation: fmt, workspace Clippy with warnings denied, and 307 release tests passed. Two pre-existing Windows SQLite wipe failures were excluded; public DHT and the manual benchmark remain ignored by default. The forwarding microbenchmark measured 103 ns (64 B) and 202 ns (1280 B) per transit packet, excluding encryption and socket I/O.
91 lines
4.8 KiB
Markdown
91 lines
4.8 KiB
Markdown
# Userspace mesh routing
|
|
|
|
```text
|
|
local OS → TUN → IP plugin (encrypt) → router → transport peer link
|
|
transport peer link → router → transport peer link (transit)
|
|
transport peer link → router → IP plugin (decrypt) → TUN → local OS
|
|
```
|
|
|
|
The TUN is the boundary of the local host. The kernel sees one overlay route;
|
|
it never forwards transit traffic. WireGuard encryption remains end to end.
|
|
The core router sees only an opaque payload and routing metadata. iroh is the
|
|
current transport adapter, with fragmentation below the routing envelope so
|
|
the default 1280-byte interface MTU also works on small QUIC paths.
|
|
|
|
## Graph and lifecycle
|
|
|
|
Each authenticated control session advertises that peer's own direct data
|
|
links, including the plugin protocol. Tables are separate for each network
|
|
and protocol; only members offering a matching enabled protocol version enter
|
|
the graph. A snapshot refreshes on the maintenance interval, expires after
|
|
90 seconds, and is withdrawn when the session ends. Link arrival, closure and
|
|
changed announcements rebuild routes immediately. Unchanged announcements
|
|
refresh their age without republishing the forwarding snapshot.
|
|
|
|
Breadth-first search computes the shortest directed paths and every equal-cost
|
|
first hop in sorted order. An actual direct link always wins. The first row is
|
|
always supplied by local transport state, never by a remote claim. Routes stop
|
|
at 16 links. A packet also carries a decreasing hop limit, bounding temporary
|
|
loops if different nodes have not yet received the same topology update.
|
|
|
|
The existing control plane still forms authenticated pairwise sessions.
|
|
Routing provides multihop **data** paths among these members; it does not add
|
|
control-plane flooding or carry control sessions through the data plane.
|
|
Announcements are volatile, not durable membership or availability guarantees.
|
|
|
|
## Packet path
|
|
|
|
The control loop binds next hops to transport handles in an immutable table
|
|
and publishes it through `ArcSwap`. Readers do not acquire the topology mutex.
|
|
The transit routine validates the fixed 74-byte envelope, looks up the source
|
|
and destination, selects a cached next hop, decrements one byte, and calls the
|
|
transport's synchronous datagram send. It never searches the graph, formats an
|
|
endpoint string, validates an Ed25519 key, decrypts, or touches TUN.
|
|
|
|
An exclusively owned receive buffer is reused when decrementing the hop limit.
|
|
Shared buffers require a copy through the safe `bytes` API. There is no added
|
|
transit queue or timer; every raw transport has its own reader. Readers yield
|
|
after 64 ready packets to avoid starving other runtime tasks. Local delivery
|
|
uses a bounded 256-datagram inbox and drops a new packet when full. Transport
|
|
queues and fragment reassembly retain their own limits.
|
|
|
|
`PacketLink::send_flow` accepts an opaque 64-bit flow identifier. IP plugins
|
|
derive it before encryption from source/destination addresses, protocol and
|
|
TCP/UDP ports. WireGuard keeps tags alongside its bounded pending queue, so
|
|
the first application packets retain their flow identity after a handshake.
|
|
Transit preserves the tag; it never hashes changing ciphertext. Handshake and
|
|
keepalive frames use flow zero. Fragmented IP traffic uses a coarse address/
|
|
protocol hash because later fragments lack ports; fragmented flows between
|
|
the same addresses coalesce. A transition between fragmented and unfragmented
|
|
traffic can change its path. The default overlay MTU avoids needing IP
|
|
fragmentation for ordinary host TCP traffic.
|
|
|
|
ECMP is deterministic for a source and flow while the table is unchanged.
|
|
Closed next hops are skipped until the control loop replaces the table. A
|
|
topology change can move a flow; datagrams remain unreliable and unordered,
|
|
and there is no promise to preserve order across a physical path failure.
|
|
Logical peer links and end-to-end encryption state survive these changes.
|
|
|
|
## Verification and measurement
|
|
|
|
The default offline suite exercises four real agents, iroh transports and
|
|
WireGuard tunnels with memory TUNs, plus graph/forwarder unit tests. It needs
|
|
neither administrative rights nor public relays/DHT.
|
|
|
|
Run the forwarding microbenchmark explicitly:
|
|
|
|
```sh
|
|
cargo test --release -p tsunagi --lib forwarding_benchmark -- --ignored --nocapture
|
|
```
|
|
|
|
It processes one million 64-byte frames and one million 1280-byte frames with
|
|
two equal next hops. Frame construction is outside the timed section; the
|
|
actual ingress routine, header validation, snapshot lookup, ECMP, TTL update,
|
|
buffer release and mock transport submission are inside. The mock transport
|
|
counts sends without storing frames. Results are CPU forwarding cost, **not**
|
|
end-to-end network latency or VPN throughput; encryption, fragmentation,
|
|
sockets, congestion and scheduling contribute separately.
|
|
|
|
Wire compatibility: control ALPN `tsunagi/ctrl/2`, data ALPN `tsunagi/data/4`.
|
|
Upgrade every participant together; saved identities and network state persist.
|