Derived IPv4 addresses could not survive anything: they changed with the range, and there was no way for a member to come back to the one it had. Addresses are now allocated and recorded as signed facts, which is the first slice of the model in docs/sync-model.md. src/state/ holds one record per author per network, carrying that author's complete current statement, signed with its persistent device key over a length-prefixed canonical encoding. Merging follows the model's rules: a higher version wins, an older one never rolls back a newer, duplicates are idempotent, absence from a snapshot is not deletion, and a same-version conflict is resolved identically on every replica and reported rather than letting replicas diverge. Records are persisted in state.sqlite, with the record and the author's version counter committed in one transaction before anything is announced, and distributed as a State control message that is merged into what the receiver already holds. No vote, deliberately, despite the request. A majority is not a trust root here — anyone with the secret can mint identities — and a quorum would stall with one peer online and diverge across a partition. Signatures plus a deterministic merge converge without either failure mode: two members claiming one address at once are resolved by the lower endpoint id, and the loser allocates again with a higher version. The range moved from the plugin to the agent, defaults to 10.13.37.0/24, and is now agreed rather than configured per member: a joining agent adopts what the network already uses, so --ipv4-range only matters for whoever starts it. The announcement went back to identity only (version 3) since the range travels in signed records now. A release tombstone exists and merges correctly, but nothing emits one yet. 116 tests. The headline ones: an address survives restarting both agents, three members get three distinct addresses, and a member started with a different range adopts the one in use. Confirmed by hand with two CLI agents restarted end to end. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
214 lines
8.7 KiB
Markdown
214 lines
8.7 KiB
Markdown
# Protocol
|
|
|
|
Module boundaries are in [architecture.md](architecture.md); the threat model
|
|
is in [threat-model.md](threat-model.md).
|
|
|
|
## Versioning
|
|
|
|
Two versions exist and are independent:
|
|
|
|
- **Identity scheme**, `tsunagi-network-id-v1`. Frozen. Changing it creates a
|
|
different network space for the same name and secret.
|
|
- **Control protocol**, ALPN `tsunagi/ctrl/1`, `PROTOCOL_VERSION = 1`.
|
|
|
|
Upgrading the crate or bumping the control protocol must never change an
|
|
existing `NetworkId`.
|
|
|
|
## Canonical encoding
|
|
|
|
Everything that is hashed or MAC'd uses length-prefixed fields, written here as
|
|
`LP(x) = u32_be(len(x)) || x`. String concatenation is never used, so no two
|
|
different field splits can produce the same bytes.
|
|
|
|
## Network space identity
|
|
|
|
```text
|
|
salt = SHA-256( LP("tsunagi-network-id-v1") || LP(name_utf8) )
|
|
prk = HKDF-SHA256-Extract(salt, ikm = secret_bytes)
|
|
|
|
info(label) = LP("tsunagi-network-id-v1") || LP(label)
|
|
network_id = HKDF-Expand(prk, info("network-id"), 32)
|
|
discovery_key = HKDF-Expand(prk, info("discovery-key"), 32)
|
|
auth_key = HKDF-Expand(prk, info("handshake-auth"), 32)
|
|
```
|
|
|
|
HKDF's `info` parameter is what separates the three outputs (RFC 5869 §3.2).
|
|
The labels and this encoding are frozen.
|
|
|
|
**Name rules.** 1..=64 bytes of UTF-8, no control characters, no leading or
|
|
trailing ASCII whitespace — such a name is *rejected*, not trimmed. The name is
|
|
used verbatim: no case folding, no Unicode normalisation. `Home` and `home` are
|
|
different network spaces.
|
|
|
|
**Secret rules.** Used verbatim: never trimmed, case-folded, normalised or
|
|
truncated. At least 16 bytes; `NetworkSecret::generate()` produces 32 random
|
|
bytes. The canonical text form is `tsn1` followed by lowercase unpadded base32.
|
|
|
|
**Key separation.** `network_id` is public. `discovery_key` is what a discovery
|
|
backend is told; it is secret-derived but is **not** a credential — learning it
|
|
does not help pass the handshake, and it must never be used as a password or
|
|
bearer token. `auth_key` never leaves the process.
|
|
|
|
## No competing genesis
|
|
|
|
A network's description is `NetworkDescriptor { scheme, name, network_id }`. It
|
|
contains no creator identity, no creation time and no owner signature, so two
|
|
agents started independently with the same parameters produce byte-identical
|
|
descriptors. No minimum-hash rule, no vote, no leader. The secret is not part
|
|
of it.
|
|
|
|
A network's name does not change inside an existing space. A different name or
|
|
secret is a different space.
|
|
|
|
## Framing
|
|
|
|
One QUIC bidirectional stream per session. A frame is `u32_be(len) || payload`.
|
|
The announced length is checked against `Limits::max_frame_len` (64 KiB by
|
|
default) **before** any buffer of that size is allocated. Payloads are
|
|
[postcard], a compact deterministic serde format — not a general RPC framework.
|
|
|
|
No encryption is layered on top of iroh: QUIC/TLS already provides
|
|
confidentiality, integrity and endpoint authentication.
|
|
|
|
[postcard]: https://docs.rs/postcard
|
|
|
|
## Handshake
|
|
|
|
A successful iroh connection proves only *which endpoint* is on the other side,
|
|
because the endpoint id is the public key in the TLS certificate. Anyone can
|
|
dial us. Membership of a specific network is proved separately.
|
|
|
|
iroh exposes the TLS exporter (RFC 5705) through
|
|
`Connection::export_keying_material`. That yields the same secret bytes on both
|
|
ends of *this* connection, which is what stops a proof being replayed
|
|
elsewhere. **It proves nothing about the shared secret on its own**, because
|
|
both ends of any connection can compute it. The membership proof is the HMAC
|
|
keyed by `auth_key`; the exporter output is just one of its inputs.
|
|
|
|
```text
|
|
cb = TLS-Exporter(label = "tsunagi/handshake/v1", context = network_id, 32)
|
|
|
|
transcript(role) = LP("tsunagi-handshake-v1")
|
|
|| LP(role) // "initiator-proof" | "responder-proof"
|
|
|| LP(u16_be(protocol_version))
|
|
|| LP(network_id) // 32 bytes
|
|
|| LP(initiator_endpoint_id) // 32 bytes
|
|
|| LP(responder_endpoint_id) // 32 bytes
|
|
|| LP(cb) // 32 bytes
|
|
|| LP(nonce_initiator) // 16 bytes
|
|
|| LP(nonce_responder) // 16 bytes
|
|
|
|
proof(role) = HMAC-SHA256(auth_key, transcript(role))
|
|
```
|
|
|
|
What each input buys:
|
|
|
|
| input | property |
|
|
|---|---|
|
|
| `auth_key` | membership in this network space |
|
|
| `cb` | binding to this connection; a captured proof is useless on another |
|
|
| `network_id` | binding to this network space |
|
|
| both endpoint ids | binding to these two identities |
|
|
| distinct `role` labels | no reflection: a proof cannot be bounced back |
|
|
| both nonces | freshness contributed by each side |
|
|
|
|
Message order:
|
|
|
|
```text
|
|
initiator -> responder : Hello { version, network_id, nonce_i }
|
|
initiator <- responder : HelloAck { version, nonce_r }
|
|
initiator -> responder : AuthProof { proof(initiator) }
|
|
initiator <- responder : AuthProof { proof(responder) } // only after the first verified
|
|
```
|
|
|
|
The responder emits nothing derived from `auth_key` until the initiator's proof
|
|
verifies, so a caller who does not know the secret learns nothing. Comparison
|
|
is constant time. Endpoint ids always come from the TLS certificate, never from
|
|
a peer's claim. Until both steps complete, no regular control message is
|
|
accepted in either direction. The whole exchange is bounded by
|
|
`Limits::handshake_timeout`.
|
|
|
|
A `Hello` naming a network this agent does not have active is rejected with
|
|
"unknown network". Because the claim is unverified at that point, the rejection
|
|
event does not report a network id.
|
|
|
|
## The data plane protocol
|
|
|
|
IP plugin packets never travel on a control connection. They use their own
|
|
ALPN, `tsunagi/data/1`, on their own iroh connection:
|
|
|
|
```text
|
|
initiator -> responder : (the same membership handshake as above)
|
|
initiator -> responder : DataOpen { protocol }
|
|
initiator <- responder : DataOpenAck { accepted, max_datagram }
|
|
thereafter : QUIC datagrams carrying that plugin's packets
|
|
```
|
|
|
|
The membership handshake is identical and bound to the same network, so a data
|
|
channel cannot be opened by somebody who does not know the secret. `protocol`
|
|
is bounded and must name a plugin the responder actually runs; otherwise the
|
|
channel is declined, which is an ordinary outcome rather than an error.
|
|
|
|
Only one side dials — the one with the smaller endpoint id — so two agents
|
|
never open two channels for the same thing.
|
|
|
|
Packets ride as QUIC **datagrams**: unreliable and unordered, which is what a
|
|
tunnelled protocol wants, and free of the head-of-line blocking a stream would
|
|
add. The datagram limit is what caps a plugin's MTU.
|
|
|
|
Separate connections mean separate congestion control, so a saturated data
|
|
plane cannot delay control messages, and a data plane failure cannot take the
|
|
control plane down with it.
|
|
|
|
## Control messages
|
|
|
|
After authentication, every frame is an `Envelope { network_id, message }` and
|
|
the `network_id` is re-checked against the session's network on every message.
|
|
A mismatch ends that session and is counted as a protocol violation; it does
|
|
not affect other networks.
|
|
|
|
| message | meaning |
|
|
|---|---|
|
|
| `Announce { hostname, capabilities }` | this agent's hostname and IP-plugin capabilities |
|
|
| `Ping { seq, payload }` | small request used to verify the exchange |
|
|
| `Pong { seq, payload }` | the echoed reply |
|
|
| `State { records }` | a snapshot of signed records, merged into what the receiver holds |
|
|
| `Bye { reason }` | graceful goodbye; not a revocation of anything |
|
|
|
|
A `State` snapshot is merged, never substituted: an author missing from it is
|
|
left untouched. Each record carries its own signature, so a peer forwarding
|
|
somebody else's record cannot alter it, and a record that fails verification
|
|
is dropped without affecting the rest of the batch. See
|
|
[sync-model.md](sync-model.md).
|
|
|
|
`PluginCapability { protocol, version, enabled, data }` is opaque to the core:
|
|
`data` is bounded and handed to the matching plugin unparsed. Nothing in it is
|
|
ever treated as a shell command, filesystem path or OS setting.
|
|
|
|
Every decoded message is validated against `Limits` before it reaches anything
|
|
else. A rejected message never stops a network or the agent.
|
|
|
|
## Limits
|
|
|
|
Defaults from `Limits`, all configurable:
|
|
|
|
| limit | default |
|
|
|---|---|
|
|
| frame payload | 64 KiB |
|
|
| hostname | 255 bytes |
|
|
| capabilities per announcement | 16 |
|
|
| capability payload | 4 KiB |
|
|
| echo payload | 4 KiB |
|
|
| reason string | 256 bytes |
|
|
| handshake timeout | 10 s |
|
|
| dial timeout | 10 s |
|
|
| write timeout | 30 s |
|
|
| concurrent dials per network | 8 |
|
|
| sessions per network | 64 |
|
|
| inbound handshakes in flight | 32 |
|
|
| outbound queue per session | 64 |
|
|
|
|
Liveness of an established session is delegated to QUIC: iroh configures
|
|
keep-alives and an idle timeout, so a dead peer surfaces as a read error rather
|
|
than needing a heartbeat in this protocol.
|