Corrects the architecture on two points raised in review, while the project is still small enough to change cheaply. 1. Control and data are separated *logically*, not physically. The old reading — "nothing but control may ride on iroh" — threw away iroh's whole value and would have forced the data plane to reimplement STUN, ICE and a relay. Now both planes ride on iroh with different ALPNs and different connections, so the data plane inherits hole punching and relay fallback, while proto/ still knows nothing about packets and dataplane/ knows nothing about the control protocol. New boundary: PacketTransport / PacketLink, an authenticated unreliable datagram channel per (network, peer, protocol). tsunagi/data/1 runs the same membership handshake, then DataOpen/DataOpenAck, then QUIC datagrams. Only the smaller endpoint id dials, so exactly one link exists per pair. A plugin is handed links and never learns reachability, so the WireGuard announcement shrank to a public key: there is no address left to lie about. 2. WireGuard now runs in userspace, on boringtun's protocol state machine. No kernel module, no wg tool, no ip shell-out, no loopback proxy: the wgtool, backend and bridge modules are gone. Only creating a TUN device needs privileges, and that sits behind TunFactory, so the entire data plane — handshake, encryption, routing, address ownership — is tested with none. Address ownership is enforced rather than believed: outbound packets go to the owner of the destination address, inbound packets are dropped unless their source is the address derived for the peer that sent them. 3. A `tsunagi` binary: secret, doctor, id, up. It owns the runtime, the logging subscriber and Ctrl-C, which the library still refuses to. Also fixes a reference cycle where IrohTransport held Arc<Inner>, which kept the databases open and the directory lock held after shutdown; two storage tests caught it once the cycle existed. 81 tests pass offline with no privileges, including real IPv6 packets crossing a real WireGuard tunnel over real iroh connections. Verified by hand: two CLI processes forming a mesh both on loopback and via n0 discovery using only an endpoint id. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
8.3 KiB
Protocol
Module boundaries are in architecture.md; the threat model is in threat-model.md.
Versioning
Two versions exist and are independent:
- Identity scheme,
tsunagi-network-id-v1. Frozen. Changing it creates a different network space for the same name and secret. - Control protocol, ALPN
tsunagi/ctrl/1,PROTOCOL_VERSION = 1.
Upgrading the crate or bumping the control protocol must never change an
existing NetworkId.
Canonical encoding
Everything that is hashed or MAC'd uses length-prefixed fields, written here as
LP(x) = u32_be(len(x)) || x. String concatenation is never used, so no two
different field splits can produce the same bytes.
Network space identity
salt = SHA-256( LP("tsunagi-network-id-v1") || LP(name_utf8) )
prk = HKDF-SHA256-Extract(salt, ikm = secret_bytes)
info(label) = LP("tsunagi-network-id-v1") || LP(label)
network_id = HKDF-Expand(prk, info("network-id"), 32)
discovery_key = HKDF-Expand(prk, info("discovery-key"), 32)
auth_key = HKDF-Expand(prk, info("handshake-auth"), 32)
HKDF's info parameter is what separates the three outputs (RFC 5869 §3.2).
The labels and this encoding are frozen.
Name rules. 1..=64 bytes of UTF-8, no control characters, no leading or
trailing ASCII whitespace — such a name is rejected, not trimmed. The name is
used verbatim: no case folding, no Unicode normalisation. Home and home are
different network spaces.
Secret rules. Used verbatim: never trimmed, case-folded, normalised or
truncated. At least 16 bytes; NetworkSecret::generate() produces 32 random
bytes. The canonical text form is tsn1 followed by lowercase unpadded base32.
Key separation. network_id is public. discovery_key is what a discovery
backend is told; it is secret-derived but is not a credential — learning it
does not help pass the handshake, and it must never be used as a password or
bearer token. auth_key never leaves the process.
No competing genesis
A network's description is NetworkDescriptor { scheme, name, network_id }. It
contains no creator identity, no creation time and no owner signature, so two
agents started independently with the same parameters produce byte-identical
descriptors. No minimum-hash rule, no vote, no leader. The secret is not part
of it.
A network's name does not change inside an existing space. A different name or secret is a different space.
Framing
One QUIC bidirectional stream per session. A frame is u32_be(len) || payload.
The announced length is checked against Limits::max_frame_len (64 KiB by
default) before any buffer of that size is allocated. Payloads are
postcard, a compact deterministic serde format — not a general RPC framework.
No encryption is layered on top of iroh: QUIC/TLS already provides confidentiality, integrity and endpoint authentication.
Handshake
A successful iroh connection proves only which endpoint is on the other side, because the endpoint id is the public key in the TLS certificate. Anyone can dial us. Membership of a specific network is proved separately.
iroh exposes the TLS exporter (RFC 5705) through
Connection::export_keying_material. That yields the same secret bytes on both
ends of this connection, which is what stops a proof being replayed
elsewhere. It proves nothing about the shared secret on its own, because
both ends of any connection can compute it. The membership proof is the HMAC
keyed by auth_key; the exporter output is just one of its inputs.
cb = TLS-Exporter(label = "tsunagi/handshake/v1", context = network_id, 32)
transcript(role) = LP("tsunagi-handshake-v1")
|| LP(role) // "initiator-proof" | "responder-proof"
|| LP(u16_be(protocol_version))
|| LP(network_id) // 32 bytes
|| LP(initiator_endpoint_id) // 32 bytes
|| LP(responder_endpoint_id) // 32 bytes
|| LP(cb) // 32 bytes
|| LP(nonce_initiator) // 16 bytes
|| LP(nonce_responder) // 16 bytes
proof(role) = HMAC-SHA256(auth_key, transcript(role))
What each input buys:
| input | property |
|---|---|
auth_key |
membership in this network space |
cb |
binding to this connection; a captured proof is useless on another |
network_id |
binding to this network space |
| both endpoint ids | binding to these two identities |
distinct role labels |
no reflection: a proof cannot be bounced back |
| both nonces | freshness contributed by each side |
Message order:
initiator -> responder : Hello { version, network_id, nonce_i }
initiator <- responder : HelloAck { version, nonce_r }
initiator -> responder : AuthProof { proof(initiator) }
initiator <- responder : AuthProof { proof(responder) } // only after the first verified
The responder emits nothing derived from auth_key until the initiator's proof
verifies, so a caller who does not know the secret learns nothing. Comparison
is constant time. Endpoint ids always come from the TLS certificate, never from
a peer's claim. Until both steps complete, no regular control message is
accepted in either direction. The whole exchange is bounded by
Limits::handshake_timeout.
A Hello naming a network this agent does not have active is rejected with
"unknown network". Because the claim is unverified at that point, the rejection
event does not report a network id.
The data plane protocol
IP plugin packets never travel on a control connection. They use their own
ALPN, tsunagi/data/1, on their own iroh connection:
initiator -> responder : (the same membership handshake as above)
initiator -> responder : DataOpen { protocol }
initiator <- responder : DataOpenAck { accepted, max_datagram }
thereafter : QUIC datagrams carrying that plugin's packets
The membership handshake is identical and bound to the same network, so a data
channel cannot be opened by somebody who does not know the secret. protocol
is bounded and must name a plugin the responder actually runs; otherwise the
channel is declined, which is an ordinary outcome rather than an error.
Only one side dials — the one with the smaller endpoint id — so two agents never open two channels for the same thing.
Packets ride as QUIC datagrams: unreliable and unordered, which is what a tunnelled protocol wants, and free of the head-of-line blocking a stream would add. The datagram limit is what caps a plugin's MTU.
Separate connections mean separate congestion control, so a saturated data plane cannot delay control messages, and a data plane failure cannot take the control plane down with it.
Control messages
After authentication, every frame is an Envelope { network_id, message } and
the network_id is re-checked against the session's network on every message.
A mismatch ends that session and is counted as a protocol violation; it does
not affect other networks.
| message | meaning |
|---|---|
Announce { hostname, capabilities } |
this agent's hostname and IP-plugin capabilities |
Ping { seq, payload } |
small request used to verify the exchange |
Pong { seq, payload } |
the echoed reply |
Bye { reason } |
graceful goodbye; not a revocation of anything |
PluginCapability { protocol, version, enabled, data } is opaque to the core:
data is bounded and handed to the matching plugin unparsed. Nothing in it is
ever treated as a shell command, filesystem path or OS setting.
Every decoded message is validated against Limits before it reaches anything
else. A rejected message never stops a network or the agent.
Limits
Defaults from Limits, all configurable:
| limit | default |
|---|---|
| frame payload | 64 KiB |
| hostname | 255 bytes |
| capabilities per announcement | 16 |
| capability payload | 4 KiB |
| echo payload | 4 KiB |
| reason string | 256 bytes |
| handshake timeout | 10 s |
| dial timeout | 10 s |
| write timeout | 30 s |
| concurrent dials per network | 8 |
| sessions per network | 64 |
| inbound handshakes in flight | 32 |
| outbound queue per session | 64 |
Liveness of an established session is delegated to QUIC: iroh configures keep-alives and an idle timeout, so a dead peer surfaces as a read error rather than needing a heartbeat in this protocol.