Files
tsunagi/docs/protocol.md
T
tsunagiandClaude Opus 5 21be7e9b44 Separate control and data logically, move WireGuard into userspace, add a CLI
Corrects the architecture on two points raised in review, while the project
is still small enough to change cheaply.

1. Control and data are separated *logically*, not physically.

The old reading — "nothing but control may ride on iroh" — threw away iroh's
whole value and would have forced the data plane to reimplement STUN, ICE and
a relay. Now both planes ride on iroh with different ALPNs and different
connections, so the data plane inherits hole punching and relay fallback,
while proto/ still knows nothing about packets and dataplane/ knows nothing
about the control protocol.

New boundary: PacketTransport / PacketLink, an authenticated unreliable
datagram channel per (network, peer, protocol). tsunagi/data/1 runs the same
membership handshake, then DataOpen/DataOpenAck, then QUIC datagrams. Only
the smaller endpoint id dials, so exactly one link exists per pair.

A plugin is handed links and never learns reachability, so the WireGuard
announcement shrank to a public key: there is no address left to lie about.

2. WireGuard now runs in userspace, on boringtun's protocol state machine.

No kernel module, no wg tool, no ip shell-out, no loopback proxy: the wgtool,
backend and bridge modules are gone. Only creating a TUN device needs
privileges, and that sits behind TunFactory, so the entire data plane —
handshake, encryption, routing, address ownership — is tested with none.

Address ownership is enforced rather than believed: outbound packets go to
the owner of the destination address, inbound packets are dropped unless
their source is the address derived for the peer that sent them.

3. A `tsunagi` binary: secret, doctor, id, up. It owns the runtime, the
logging subscriber and Ctrl-C, which the library still refuses to.

Also fixes a reference cycle where IrohTransport held Arc<Inner>, which kept
the databases open and the directory lock held after shutdown; two storage
tests caught it once the cycle existed.

81 tests pass offline with no privileges, including real IPv6 packets
crossing a real WireGuard tunnel over real iroh connections. Verified by
hand: two CLI processes forming a mesh both on loopback and via n0 discovery
using only an endpoint id.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-21 11:55:20 +01:00

207 lines
8.3 KiB
Markdown

# Protocol
Module boundaries are in [architecture.md](architecture.md); the threat model
is in [threat-model.md](threat-model.md).
## Versioning
Two versions exist and are independent:
- **Identity scheme**, `tsunagi-network-id-v1`. Frozen. Changing it creates a
different network space for the same name and secret.
- **Control protocol**, ALPN `tsunagi/ctrl/1`, `PROTOCOL_VERSION = 1`.
Upgrading the crate or bumping the control protocol must never change an
existing `NetworkId`.
## Canonical encoding
Everything that is hashed or MAC'd uses length-prefixed fields, written here as
`LP(x) = u32_be(len(x)) || x`. String concatenation is never used, so no two
different field splits can produce the same bytes.
## Network space identity
```text
salt = SHA-256( LP("tsunagi-network-id-v1") || LP(name_utf8) )
prk = HKDF-SHA256-Extract(salt, ikm = secret_bytes)
info(label) = LP("tsunagi-network-id-v1") || LP(label)
network_id = HKDF-Expand(prk, info("network-id"), 32)
discovery_key = HKDF-Expand(prk, info("discovery-key"), 32)
auth_key = HKDF-Expand(prk, info("handshake-auth"), 32)
```
HKDF's `info` parameter is what separates the three outputs (RFC 5869 §3.2).
The labels and this encoding are frozen.
**Name rules.** 1..=64 bytes of UTF-8, no control characters, no leading or
trailing ASCII whitespace — such a name is *rejected*, not trimmed. The name is
used verbatim: no case folding, no Unicode normalisation. `Home` and `home` are
different network spaces.
**Secret rules.** Used verbatim: never trimmed, case-folded, normalised or
truncated. At least 16 bytes; `NetworkSecret::generate()` produces 32 random
bytes. The canonical text form is `tsn1` followed by lowercase unpadded base32.
**Key separation.** `network_id` is public. `discovery_key` is what a discovery
backend is told; it is secret-derived but is **not** a credential — learning it
does not help pass the handshake, and it must never be used as a password or
bearer token. `auth_key` never leaves the process.
## No competing genesis
A network's description is `NetworkDescriptor { scheme, name, network_id }`. It
contains no creator identity, no creation time and no owner signature, so two
agents started independently with the same parameters produce byte-identical
descriptors. No minimum-hash rule, no vote, no leader. The secret is not part
of it.
A network's name does not change inside an existing space. A different name or
secret is a different space.
## Framing
One QUIC bidirectional stream per session. A frame is `u32_be(len) || payload`.
The announced length is checked against `Limits::max_frame_len` (64 KiB by
default) **before** any buffer of that size is allocated. Payloads are
[postcard], a compact deterministic serde format — not a general RPC framework.
No encryption is layered on top of iroh: QUIC/TLS already provides
confidentiality, integrity and endpoint authentication.
[postcard]: https://docs.rs/postcard
## Handshake
A successful iroh connection proves only *which endpoint* is on the other side,
because the endpoint id is the public key in the TLS certificate. Anyone can
dial us. Membership of a specific network is proved separately.
iroh exposes the TLS exporter (RFC 5705) through
`Connection::export_keying_material`. That yields the same secret bytes on both
ends of *this* connection, which is what stops a proof being replayed
elsewhere. **It proves nothing about the shared secret on its own**, because
both ends of any connection can compute it. The membership proof is the HMAC
keyed by `auth_key`; the exporter output is just one of its inputs.
```text
cb = TLS-Exporter(label = "tsunagi/handshake/v1", context = network_id, 32)
transcript(role) = LP("tsunagi-handshake-v1")
|| LP(role) // "initiator-proof" | "responder-proof"
|| LP(u16_be(protocol_version))
|| LP(network_id) // 32 bytes
|| LP(initiator_endpoint_id) // 32 bytes
|| LP(responder_endpoint_id) // 32 bytes
|| LP(cb) // 32 bytes
|| LP(nonce_initiator) // 16 bytes
|| LP(nonce_responder) // 16 bytes
proof(role) = HMAC-SHA256(auth_key, transcript(role))
```
What each input buys:
| input | property |
|---|---|
| `auth_key` | membership in this network space |
| `cb` | binding to this connection; a captured proof is useless on another |
| `network_id` | binding to this network space |
| both endpoint ids | binding to these two identities |
| distinct `role` labels | no reflection: a proof cannot be bounced back |
| both nonces | freshness contributed by each side |
Message order:
```text
initiator -> responder : Hello { version, network_id, nonce_i }
initiator <- responder : HelloAck { version, nonce_r }
initiator -> responder : AuthProof { proof(initiator) }
initiator <- responder : AuthProof { proof(responder) } // only after the first verified
```
The responder emits nothing derived from `auth_key` until the initiator's proof
verifies, so a caller who does not know the secret learns nothing. Comparison
is constant time. Endpoint ids always come from the TLS certificate, never from
a peer's claim. Until both steps complete, no regular control message is
accepted in either direction. The whole exchange is bounded by
`Limits::handshake_timeout`.
A `Hello` naming a network this agent does not have active is rejected with
"unknown network". Because the claim is unverified at that point, the rejection
event does not report a network id.
## The data plane protocol
IP plugin packets never travel on a control connection. They use their own
ALPN, `tsunagi/data/1`, on their own iroh connection:
```text
initiator -> responder : (the same membership handshake as above)
initiator -> responder : DataOpen { protocol }
initiator <- responder : DataOpenAck { accepted, max_datagram }
thereafter : QUIC datagrams carrying that plugin's packets
```
The membership handshake is identical and bound to the same network, so a data
channel cannot be opened by somebody who does not know the secret. `protocol`
is bounded and must name a plugin the responder actually runs; otherwise the
channel is declined, which is an ordinary outcome rather than an error.
Only one side dials — the one with the smaller endpoint id — so two agents
never open two channels for the same thing.
Packets ride as QUIC **datagrams**: unreliable and unordered, which is what a
tunnelled protocol wants, and free of the head-of-line blocking a stream would
add. The datagram limit is what caps a plugin's MTU.
Separate connections mean separate congestion control, so a saturated data
plane cannot delay control messages, and a data plane failure cannot take the
control plane down with it.
## Control messages
After authentication, every frame is an `Envelope { network_id, message }` and
the `network_id` is re-checked against the session's network on every message.
A mismatch ends that session and is counted as a protocol violation; it does
not affect other networks.
| message | meaning |
|---|---|
| `Announce { hostname, capabilities }` | this agent's hostname and IP-plugin capabilities |
| `Ping { seq, payload }` | small request used to verify the exchange |
| `Pong { seq, payload }` | the echoed reply |
| `Bye { reason }` | graceful goodbye; not a revocation of anything |
`PluginCapability { protocol, version, enabled, data }` is opaque to the core:
`data` is bounded and handed to the matching plugin unparsed. Nothing in it is
ever treated as a shell command, filesystem path or OS setting.
Every decoded message is validated against `Limits` before it reaches anything
else. A rejected message never stops a network or the agent.
## Limits
Defaults from `Limits`, all configurable:
| limit | default |
|---|---|
| frame payload | 64 KiB |
| hostname | 255 bytes |
| capabilities per announcement | 16 |
| capability payload | 4 KiB |
| echo payload | 4 KiB |
| reason string | 256 bytes |
| handshake timeout | 10 s |
| dial timeout | 10 s |
| write timeout | 30 s |
| concurrent dials per network | 8 |
| sessions per network | 64 |
| inbound handshakes in flight | 32 |
| outbound queue per session | 64 |
Liveness of an established session is delegated to QUIC: iroh configures
keep-alives and an idle timeout, so a dead peer surfaces as a read error rather
than needing a heartbeat in this protocol.