Aleksandr Bogomiakov
2166ba4800
AWG-only policy also selected a competing end-to-end plugin, so compatible external links could still have no compatible encrypted data channel. Carrier changes replaced tunnel state and status selected counters from different engines. Use one internal WireGuard plugin and a separate DataCarrier contract for external capabilities/lifecycle. Keep the logical link, keys, handshake, interface and counter epoch across policy, fallback and direct/relay changes. Control/discovery transport selection and membership remain independent. Share the bounded per-neighbour UDP WireGuard engine between ordinary WG and Amnezia masking. Encrypt the relay envelope inside the outer hop channel with fresh pair keys/PSK. Remove duplicate WG key/device code and the TCP/TLS inner cipher. TLS carries opaque internal WireGuard frames. Refuse missing native transports instead of letting default iroh impersonate their protocol names. Probe the actual authenticated carrier, independently of the shared IP tunnel; failed local probe submissions count as loss too. Apply the vendored iroh interface exclusion to actual direct candidates, not just public discovery records. Exclude overlay subnets from introductions and propagate the actual owned interface name, including kernel-assigned utun names, to every carrier. Production logs showed paths through the overlay itself. This removes a recursive transport path; it does not establish the historical cause of the reported 100 GB absolute counter. Exit-node traffic is also real peer IP traffic even when local applications are idle. Hide inner cipher choices/handshake clutter in CLI and tray. Report external routes, readiness and actionable policy/version failures. Count original IP packets only when their ciphertext is accepted by the logical transport, including queued packets' exact unpadded lengths. Exclude physical probes and keepalives. Export counter epochs; sum per-peer rate deltas and discard stale histories so joins/restarts cannot turn old totals into fresh traffic. Status also prints exact IP byte totals for comparison with the tray. Coordinated upgrade: data ALPN 6, internal wireguard-e2e capability v4, wg/awg v2, QUIC carrier v1, TLS v2, local IPC v23. Control ALPN 8, device/network identity derivation, signed records and saved network choices stay unchanged. Keep the historical wg-quic key store; transactionally import missing keys from wg then awg without replacing canonical keys. Forget removes legacy copies too, on the blocking pool, to prevent resurrection after restart. Preserve historic cipher options as shared settings, rejecting contradictory values. Validation on Linux in the Nix development shell: - cargo fmt --all -- --check - cargo clippy --locked --workspace --all-targets -- -D warnings - cargo test --locked --workspace --all-targets (562 passed, 3 ignored) The inherited service socket/secret/group environment overrides were removed for isolated CLI process tests. A new carrier test initially raced the first peer announcement; its deadline polling now waits for a peer row explicitly. Real loopback endpoints, membership, SQLite, WG/AWG/TLS encryption and memory TUNs cover mixed-carrier relaying and one smoltcp TCP connection surviving relay/direct/policy changes and a dropped transmission. Counter tests check exact plaintext lengths, refused sends, epochs, joins and stale histories. Public DHT, privileged kernel and manual benchmark tests remain ignored; macOS/Windows native TUNs, desktop GUI and WAN/DPI behavior were not run.
293 lines
14 KiB
Markdown
293 lines
14 KiB
Markdown
# Protocol
|
|
|
|
Module boundaries are in [architecture.md](architecture.md); the threat model
|
|
is in [threat-model.md](threat-model.md).
|
|
|
|
## Versioning
|
|
|
|
Two versions exist and are independent:
|
|
|
|
- **Identity scheme**, `tsunagi-network-id-v1`. Frozen. Changing it creates a
|
|
different network space for the same name and secret.
|
|
- **Control protocol**, ALPN `tsunagi/ctrl/8`, `PROTOCOL_VERSION = 8`.
|
|
|
|
Upgrading the crate or bumping the control protocol must never change an
|
|
existing `NetworkId`.
|
|
|
|
Control version 8 adds signed, multi-writer VIP field snapshots to version 7's
|
|
node retirements. Data ALPN is `tsunagi/data/6` for the single internal WireGuard channel. All members must upgrade
|
|
together. Network ids, derivation labels, handshake encoding and existing record
|
|
signatures are unchanged. Local CLI/tray IPC is version 23; restart the agent
|
|
with matching clients. SQLite schema 10 adds `vip_fields`, preserving identity,
|
|
network configuration, signed node records and retirement proofs.
|
|
See [virtual services](vip.md), [state sync](sync-model.md) and [AWG](amnezia.md).
|
|
|
|
## Canonical encoding
|
|
|
|
Network identity derivation and the membership transcript use length-prefixed
|
|
fields, written here as `LP(x) = u32_be(len(x)) || x`. String concatenation is
|
|
never used, so different field splits cannot produce the same bytes. VIP field
|
|
signatures use their separate domain and canonical postcard encoding described
|
|
in [virtual services](vip.md#shared-state-and-addresses).
|
|
|
|
## Network space identity
|
|
|
|
```text
|
|
salt = SHA-256( LP("tsunagi-network-id-v1") || LP(name_utf8) )
|
|
prk = HKDF-SHA256-Extract(salt, ikm = secret_bytes)
|
|
|
|
info(label) = LP("tsunagi-network-id-v1") || LP(label)
|
|
network_id = HKDF-Expand(prk, info("network-id"), 32)
|
|
discovery_key = HKDF-Expand(prk, info("discovery-key"), 32)
|
|
auth_key = HKDF-Expand(prk, info("handshake-auth"), 32)
|
|
```
|
|
|
|
HKDF's `info` parameter is what separates the three outputs (RFC 5869 §3.2).
|
|
The labels and this encoding are frozen.
|
|
|
|
**Name rules.** 1..=64 bytes of UTF-8, no control characters, no leading or
|
|
trailing ASCII whitespace — such a name is *rejected*, not trimmed. The name is
|
|
used verbatim: no case folding, no Unicode normalisation. `Home` and `home` are
|
|
different network spaces.
|
|
|
|
**Secret rules.** Used verbatim: never trimmed, case-folded, normalised or
|
|
truncated. At least 16 bytes; `NetworkSecret::generate()` produces 32 random
|
|
bytes. The canonical text form is `tsn1` followed by lowercase unpadded base32.
|
|
|
|
**Key separation.** `network_id` is public. `discovery_key` is what a discovery
|
|
backend is told; it is secret-derived but is **not** a credential — learning it
|
|
does not help pass the handshake, and it must never be used as a password or
|
|
bearer token. `auth_key` never leaves the process.
|
|
|
|
## No competing genesis
|
|
|
|
A network's description is `NetworkDescriptor { scheme, name, network_id }`. It
|
|
contains no creator identity, no creation time and no owner signature, so two
|
|
agents started independently with the same parameters produce byte-identical
|
|
descriptors. No minimum-hash rule, no vote, no leader. The secret is not part
|
|
of it.
|
|
|
|
A network's name does not change inside an existing space. A different name or
|
|
secret is a different space.
|
|
|
|
## Framing
|
|
|
|
One QUIC bidirectional stream per session. A frame is `u32_be(len) || payload`.
|
|
The announced length is checked against `Limits::max_frame_len` (64 KiB by
|
|
default) **before** any buffer of that size is allocated. Payloads are
|
|
[postcard], a compact deterministic serde format — not a general RPC framework.
|
|
|
|
No encryption is layered on top of iroh: QUIC/TLS already provides
|
|
confidentiality, integrity and endpoint authentication.
|
|
|
|
[postcard]: https://docs.rs/postcard
|
|
|
|
## Handshake
|
|
|
|
A successful iroh connection proves only *which endpoint* is on the other side,
|
|
because the endpoint id is the public key in the TLS certificate. Anyone can
|
|
dial us. Membership of a specific network is proved separately.
|
|
|
|
iroh exposes the TLS exporter (RFC 5705) through
|
|
`Connection::export_keying_material`. That yields the same secret bytes on both
|
|
ends of *this* connection, which is what stops a proof being replayed
|
|
elsewhere. **It proves nothing about the shared secret on its own**, because
|
|
both ends of any connection can compute it. The membership proof is the HMAC
|
|
keyed by `auth_key`; the exporter output is just one of its inputs.
|
|
|
|
```text
|
|
cb = TLS-Exporter(label = "tsunagi/handshake/v1", context = network_id, 32)
|
|
|
|
transcript(role) = LP("tsunagi-handshake-v1")
|
|
|| LP(role) // "initiator-proof" | "responder-proof"
|
|
|| LP(u16_be(protocol_version))
|
|
|| LP(network_id) // 32 bytes
|
|
|| LP(initiator_endpoint_id) // 32 bytes
|
|
|| LP(responder_endpoint_id) // 32 bytes
|
|
|| LP(cb) // 32 bytes
|
|
|| LP(nonce_initiator) // 16 bytes
|
|
|| LP(nonce_responder) // 16 bytes
|
|
|
|
proof(role) = HMAC-SHA256(auth_key, transcript(role))
|
|
```
|
|
|
|
What each input buys:
|
|
|
|
| input | property |
|
|
|---|---|
|
|
| `auth_key` | membership in this network space |
|
|
| `cb` | binding to this connection; a captured proof is useless on another |
|
|
| `network_id` | binding to this network space |
|
|
| both endpoint ids | binding to these two identities |
|
|
| distinct `role` labels | no reflection: a proof cannot be bounced back |
|
|
| both nonces | freshness contributed by each side |
|
|
|
|
Message order:
|
|
|
|
```text
|
|
initiator -> responder : Hello { version, network_id, nonce_i }
|
|
initiator <- responder : HelloAck { version, nonce_r }
|
|
initiator -> responder : AuthProof { proof(initiator) }
|
|
initiator <- responder : AuthProof { proof(responder) } // only after the first verified
|
|
```
|
|
|
|
The responder emits nothing derived from `auth_key` until the initiator's proof
|
|
verifies, so a caller who does not know the secret learns nothing. Comparison
|
|
is constant time. Endpoint ids always come from the TLS certificate, never from
|
|
a peer's claim. Until both steps complete, no regular control message is
|
|
accepted in either direction. The whole exchange is bounded by
|
|
`Limits::handshake_timeout`.
|
|
|
|
A `Hello` naming a network this agent does not have active is rejected with
|
|
"unknown network". Because the claim is unverified at that point, the rejection
|
|
event does not report a network id.
|
|
|
|
## The data plane protocol
|
|
|
|
IP plugin packets never travel on a control connection. They use their own
|
|
ALPN, `tsunagi/data/6`, on their own iroh connection:
|
|
|
|
```text
|
|
initiator -> responder : (the same membership handshake as above)
|
|
initiator -> responder : DataOpen { protocol, max_datagram }
|
|
initiator <- responder : DataOpenAck { accepted, max_datagram }
|
|
thereafter : QUIC datagrams carrying fragments of opaque payloads
|
|
```
|
|
|
|
The membership handshake is identical and bound to the same network, so a data
|
|
channel cannot be opened by somebody who does not know the secret. `protocol`
|
|
is bounded and must name a plugin the responder actually runs; otherwise the
|
|
channel is declined, which is an ordinary outcome rather than an error.
|
|
|
|
Only one side dials — the one with the smaller endpoint id — so two agents
|
|
never open two channels for the same thing.
|
|
|
|
Packets ride as QUIC **datagrams**: unreliable and unordered, which is what a
|
|
tunnelled protocol wants, and free of the head-of-line blocking a stream would
|
|
add. `max_datagram` negotiates the reassembled payload limit (at most 65536
|
|
bytes), independently of the current QUIC path MTU. The transport reads QUIC's
|
|
current datagram capacity for every fragment, including after migration.
|
|
|
|
Every datagram starts with a 16-byte header: packet ID (`u64`), total payload
|
|
length (`u32`), and byte offset (`u32`), all big endian. The payload is the
|
|
complete peer-relay envelope containing opaque plugin bytes. Relays reassemble
|
|
before forwarding and fragment again for their outgoing path; they never
|
|
decrypt the plugin's payload. Small packets use the same framing in one part.
|
|
|
|
Reassembly tolerates reordering and exact duplicates. Conflicting lengths,
|
|
overlapping data and excessive fragment counts discard that packet. Each link
|
|
allows at most 64 incomplete packets, 256 KiB of their payload buffers, and 128
|
|
fragments per packet. Incomplete packets expire after five seconds, without
|
|
holding up subsequent packets. Packet IDs are scoped to a QUIC connection;
|
|
reassembly is created only after the membership handshake. All limits live in
|
|
`config.rs`. There is no transport-layer retransmission added by this framing.
|
|
|
|
Version 4 added a routing envelope inside the transport fragmentation framing,
|
|
and version 5 added the end-to-end protocol to it:
|
|
|
|
| field | bytes | encoding |
|
|
|---|---:|---|
|
|
| envelope version | 1 | `2` |
|
|
| remaining hops | 1 | initially `16`; transit decrements, drops at `1` |
|
|
| source peer | 32 | original endpoint public key |
|
|
| destination peer | 32 | final endpoint public key |
|
|
| flow id | 8 | big endian opaque stable hash supplied by the plugin |
|
|
| protocol | 8 | big endian FNV-1a of the internal protocol id (`wireguard-e2e`) |
|
|
| payload | remaining | end-to-end encrypted plugin bytes |
|
|
|
|
The fixed header is 82 bytes. Zero/oversized hop limits, unknown versions,
|
|
short frames, oversized datagrams and unknown sources are rejected. A frame
|
|
addressed here is delivered to the inbox of its source and protocol; transit
|
|
goes straight to its next transport, whatever kind of transport that is. The
|
|
network is bound to the authenticated transport connection, not supplied by
|
|
the frame. The protocol names which end-to-end tunnel the payload belongs to
|
|
and is independent of the links it crosses: a datagram of one protocol can
|
|
travel over a link opened for another, and a middle member needs no protocol
|
|
in common with either end. Intermediate nodes cannot decrypt or authenticate
|
|
the inner payload; the destination's tunnel does that, so a frame naming the
|
|
wrong protocol or a forged source is simply discarded there. Flow ids are
|
|
routing hints, not authorization proofs.
|
|
|
|
`Reachable { links: [{ peer, protocol }] }` lists the sender's direct links;
|
|
`protocol` is the kind of transport the link is, for diagnostics, and routing
|
|
ignores it. Each row belongs to the authenticated sender and is replaced
|
|
atomically, expires after 90 seconds, and is withdrawn on session closure. Only
|
|
authenticated members enter the graph; local edges always come from actual
|
|
links.
|
|
Announcements refresh on the existing control maintenance interval and are
|
|
sent immediately on link changes. Topology stays volatile, outside signed state.
|
|
|
|
Version 5 also adds `hostname_since` to a signed hostname claim (signing domain
|
|
`tsunagi-signed-record-v3`): a name belongs to whoever claimed it first. Stored
|
|
records signed before are discarded on upgrade and re-published by their
|
|
authors.
|
|
|
|
Upgrade both endpoints and intermediate peers together. Older control and data
|
|
ALPNs cannot interoperate. The network secret, device identity, identity
|
|
derivation, transcript encoding and stored network configuration are unchanged.
|
|
|
|
Separate connections mean separate congestion control, so a saturated data
|
|
plane cannot delay control messages, and a data plane failure cannot take the
|
|
control plane down with it.
|
|
|
|
## Control messages
|
|
|
|
After authentication, every frame is an `Envelope { network_id, message }` and
|
|
the `network_id` is re-checked against the session's network on every message.
|
|
A mismatch ends that session and is counted as a protocol violation; it does
|
|
not affect other networks.
|
|
|
|
| message | meaning |
|
|
|---|---|
|
|
| `Announce { hostname, capabilities, broadcast }` | this agent's hostname, IP-plugin capabilities and local broadcast participation |
|
|
| `Transport { protocol, payload }` | private, bounded per-peer transport negotiation; never state or a public message event |
|
|
| `Ping { seq, payload }` | small request used to verify the exchange |
|
|
| `Pong { seq, payload }` | the echoed reply |
|
|
| `State { records }` | a snapshot of signed records, merged into what the receiver holds |
|
|
| `Retirements { markers }` | signed cleanup markers with the target's original signed claim as proof |
|
|
| `Peers { peers }` | unverified member address hints, requiring their own handshake |
|
|
| `Reachable { links }` | sender's current direct data links, by kind of transport |
|
|
| `Bye { reason }` | graceful goodbye; not a revocation of anything |
|
|
|
|
`broadcast` is a per-network local opt-in, enabled by default and treated as
|
|
false until an authenticated announcement arrives. It governs IP broadcast
|
|
fanout and local admission; encrypted transit still uses the existing envelope.
|
|
See [broadcast.md](broadcast.md) for scope and persistence.
|
|
|
|
A `State` snapshot is merged, never substituted: an author missing from it is
|
|
left untouched. Each record carries its own signature, so a peer forwarding
|
|
somebody else's record cannot alter it, and a record that fails verification
|
|
is dropped without affecting the rest of the batch. See
|
|
[sync-model.md](sync-model.md).
|
|
|
|
`PluginCapability { protocol, version, enabled, data }` is opaque to the core:
|
|
`data` is bounded and handed to the matching plugin unparsed. Nothing in it is
|
|
ever treated as a shell command, filesystem path or OS setting.
|
|
|
|
Every decoded message is validated against `Limits` before it reaches anything
|
|
else. A rejected message never stops a network or the agent.
|
|
|
|
## Limits
|
|
|
|
Defaults from `Limits`, all configurable:
|
|
|
|
| limit | default |
|
|
|---|---|
|
|
| frame payload | 64 KiB |
|
|
| hostname | 255 bytes |
|
|
| capabilities per announcement | 16 |
|
|
| capability payload | 4 KiB |
|
|
| echo payload | 4 KiB |
|
|
| reason string | 256 bytes |
|
|
| handshake timeout | 10 s |
|
|
| dial timeout | 10 s |
|
|
| write timeout | 30 s |
|
|
| concurrent dials per network | 8 |
|
|
| sessions per network | 64 |
|
|
| inbound handshakes in flight | 32 |
|
|
| outbound queue per session | 64 |
|
|
|
|
Liveness of an established session is delegated to QUIC: iroh configures
|
|
keep-alives and an idle timeout, so a dead peer surfaces as a read error rather
|
|
than needing a heartbeat in this protocol.
|