ab/tsunagi
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Proof-of-concept mesh agent library over iroh

Browse Source 
Working library with real iroh connections, not an interface sketch:

- persistent device identity in state.sqlite, stable across restarts
- deterministic network space derived from name + secret via HKDF-SHA256,
  with frozen labels and unambiguous length-prefixed encoding
- replaceable discovery returning unverified candidates only; static
  bootstrap, in-memory test backend and a composite
- real iroh connections plus an explicit mutual membership proof:
  HMAC-SHA256 over a role-separated transcript bound to the TLS exporter,
  the network id and both endpoint identities
- small versioned control protocol: handshake, announcement, ping/pong
- multiple networks per agent with enforced isolation
- automatic reconnect with bounded backoff and jitter
- mandatory state vs disposable cache, with a real directory ownership lock
- status snapshots, event stream and honest diagnostics

47 integration and unit tests cover the required scenarios offline on
loopback. Snapshots, revocations and WireGuard are designed for and
documented, not implemented.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
tsunagi
2026-09-21 00:10:07 +01:00
co-authored by Claude Opus 5
commit 7cea9afa37
44 changed files with 12916 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/architecture.md
+131
View File
@@ -0,0 +1,131 @@
# Architecture
Scope and non-scope are in [../README.md](../README.md). Rules for changing the
code are in [../AGENTS.md](../AGENTS.md). The wire format is in
[protocol.md](protocol.md).
## Two independent planes
**Control plane.** iroh provides connections between agents and carries control
messages. On top of it, this crate's protocol handles membership
authentication, participant announcements, capability exchange and — later —
state synchronisation and delivery of IP-plugin data.
**Data plane.** Separate plugins create IP connectivity. WireGuard is the first
planned one; none exists yet. Plugin keys, configuration and lifecycle are
separate from iroh identity and from the network secret. The core moves an
opaque, bounded payload and never parses it.
Only control messages travel over iroh. User IP traffic is not tunnelled
through it.
A data plane failure never stops the daemon: the control plane keeps running
and the agent stays manageable.
## Module responsibilities
| component | responsibility |
|---|---|
| `identity` | persistent device identity; network space parameters; derived identifiers and keys |
| `discovery` | obtaining and publishing address hints — not authentication, not state transfer |
| `net` | iroh connections, addresses, paths, statistics, connection events |
| `proto` | message format, handshake, membership proof, protocol limits |
| `agent` | agent and per-network lifecycle, reconnect, in-process message routing |
| `storage` | mandatory state and the separately recoverable cache |
| `dataplane` | the minimal contract future IP plugins implement |
Abstractions exist only where something is really substituted or really needs
isolating for tests: `NetworkDiscovery` and `IpPlugin`. Everything else is a
concrete type.
## Runtime shape
```text
Agent one persistent identity, one iroh endpoint,
├── EndpointAdapter one state directory, N networks
├── Storage (state.sqlite + cache.sqlite + ownership lock)
├── accept loop task ── weak ref, exits when the agent is dropped
└── NetworkRuntime per NetworkId
├── discovery + dial loop (bounded concurrency, backoff with jitter)
└── Session per peer
├── reader task ── frames in -> SessionEvent
└── writer task ── encoded frames out
```
The library starts no runtime, installs no logging subscriber, handles no
signals, never forks and never calls `process::exit`. Startup
(`Agent::spawn`) and shutdown (`Agent::shutdown`) are explicit, background
tasks are joined on shutdown, and there is no global mutable state — several
independent agents run in one test process.
`Agent::spawn` returns as soon as the local agent is ready. It never waits for
other participants or for a relay.
### Concurrency decisions
- The network runtime is a single task owning its own state, driven by a
command channel plus event channels. No shared locks on the hot path.
- A session is two tasks, because a partially consumed stream read cannot be
cancelled safely. The writer's frame write *is* cancellable, so shutdown
never waits on a peer that stopped reading.
- Outbound queues are bounded. A full queue fails that send instead of stalling
every other peer in the network.
- Envelopes are encoded in the runtime, not in the writer, so the exact number
of control bytes queued is known and reported rather than guessed.
- Simultaneous mutual dials are resolved by a deterministic rule both sides
compute identically: keep the session whose initiator has the smaller
endpoint id.
### Storage
Two physically separate SQLite files with their own schema versions and
migrations, no ORM. All access runs on the blocking pool; no database lock is
held across a network `await`. Files are owner-only where the platform supports
it. One state directory belongs to one live agent, enforced by an advisory OS
file lock — an existence check is not enough, because a crashed process must not
block a restart and two agents starting at once must not both win. A clean
shutdown releases the lock immediately.
### Observability
`AgentStatus` separates the three levels deliberately:
- **endpoint** — bound sockets, observed addresses, cache health;
- **logical network** — dial attempts and failures, handshake failures,
sessions established, disconnects, control message and byte counts, protocol
violations, plugin errors;
- **connection** — verified paths, selected path, direct/relay, RTT, QUIC
counters.
Values that cannot honestly be attributed to one network stay at the endpoint
level. A value iroh does not report is `None`.
An iroh address is an address for iroh. It is not assumed usable by any other
protocol; a future WireGuard plugin gathers its own reachability data and ships
it through the control plane as an opaque announcement.
### Relays
Standard iroh behaviour, including relay fallback, is allowed for the control
plane via `TransportPolicy::N0Defaults`. Public relays are fine for development
and carry no availability guarantee. The default is `LocalOnly`, and the test
suite never depends on a relay.
## Multiple networks
Every session, message, task and status carries an explicit `NetworkId`. Being
authenticated in network A grants nothing in network B, even over a shared
physical connection; each network gets its own sessions. Deactivating one
network does not close the agent or disturb the others.
Local deactivation is distinct from a future signed revocation of membership or
of a name. Nothing here is an irreversible global flag.
## Planned, not built
Signed per-author state, snapshots, revocations and merge rules are described
in [sync-model.md](sync-model.md). The module boundaries above are shaped so
that adding them does not require rewriting the core. The data plane locking
policy is likewise deliberately left coarse for now; the conflict domain is
described separately so it can later be narrowed to a network, peer, name or
plugin without touching the core.
docs/protocol.md
+178
View File
@@ -0,0 +1,178 @@
# Protocol
Module boundaries are in [architecture.md](architecture.md); the threat model
is in [threat-model.md](threat-model.md).
## Versioning
Two versions exist and are independent:
- **Identity scheme**, `tsunagi-network-id-v1`. Frozen. Changing it creates a
different network space for the same name and secret.
- **Control protocol**, ALPN `tsunagi/ctrl/1`, `PROTOCOL_VERSION = 1`.
Upgrading the crate or bumping the control protocol must never change an
existing `NetworkId`.
## Canonical encoding
Everything that is hashed or MAC'd uses length-prefixed fields, written here as
`LP(x) = u32_be(len(x)) || x`. String concatenation is never used, so no two
different field splits can produce the same bytes.
## Network space identity
```text
salt = SHA-256( LP("tsunagi-network-id-v1") || LP(name_utf8) )
prk = HKDF-SHA256-Extract(salt, ikm = secret_bytes)
info(label) = LP("tsunagi-network-id-v1") || LP(label)
network_id = HKDF-Expand(prk, info("network-id"), 32)
discovery_key = HKDF-Expand(prk, info("discovery-key"), 32)
auth_key = HKDF-Expand(prk, info("handshake-auth"), 32)
```
HKDF's `info` parameter is what separates the three outputs (RFC 5869 §3.2).
The labels and this encoding are frozen.
**Name rules.** 1..=64 bytes of UTF-8, no control characters, no leading or
trailing ASCII whitespace — such a name is *rejected*, not trimmed. The name is
used verbatim: no case folding, no Unicode normalisation. `Home` and `home` are
different network spaces.
**Secret rules.** Used verbatim: never trimmed, case-folded, normalised or
truncated. At least 16 bytes; `NetworkSecret::generate()` produces 32 random
bytes. The canonical text form is `tsn1` followed by lowercase unpadded base32.
**Key separation.** `network_id` is public. `discovery_key` is what a discovery
backend is told; it is secret-derived but is **not** a credential — learning it
does not help pass the handshake, and it must never be used as a password or
bearer token. `auth_key` never leaves the process.
## No competing genesis
A network's description is `NetworkDescriptor { scheme, name, network_id }`. It
contains no creator identity, no creation time and no owner signature, so two
agents started independently with the same parameters produce byte-identical
descriptors. No minimum-hash rule, no vote, no leader. The secret is not part
of it.
A network's name does not change inside an existing space. A different name or
secret is a different space.
## Framing
One QUIC bidirectional stream per session. A frame is `u32_be(len) || payload`.
The announced length is checked against `Limits::max_frame_len` (64 KiB by
default) **before** any buffer of that size is allocated. Payloads are
[postcard], a compact deterministic serde format — not a general RPC framework.
No encryption is layered on top of iroh: QUIC/TLS already provides
confidentiality, integrity and endpoint authentication.
[postcard]: https://docs.rs/postcard
## Handshake
A successful iroh connection proves only *which endpoint* is on the other side,
because the endpoint id is the public key in the TLS certificate. Anyone can
dial us. Membership of a specific network is proved separately.
iroh exposes the TLS exporter (RFC 5705) through
`Connection::export_keying_material`. That yields the same secret bytes on both
ends of *this* connection, which is what stops a proof being replayed
elsewhere. **It proves nothing about the shared secret on its own**, because
both ends of any connection can compute it. The membership proof is the HMAC
keyed by `auth_key`; the exporter output is just one of its inputs.
```text
cb = TLS-Exporter(label = "tsunagi/handshake/v1", context = network_id, 32)
transcript(role) = LP("tsunagi-handshake-v1")
|| LP(role) // "initiator-proof" | "responder-proof"
|| LP(u16_be(protocol_version))
|| LP(network_id) // 32 bytes
|| LP(initiator_endpoint_id) // 32 bytes
|| LP(responder_endpoint_id) // 32 bytes
|| LP(cb) // 32 bytes
|| LP(nonce_initiator) // 16 bytes
|| LP(nonce_responder) // 16 bytes
proof(role) = HMAC-SHA256(auth_key, transcript(role))
```
What each input buys:
| input | property |
|---|---|
| `auth_key` | membership in this network space |
| `cb` | binding to this connection; a captured proof is useless on another |
| `network_id` | binding to this network space |
| both endpoint ids | binding to these two identities |
| distinct `role` labels | no reflection: a proof cannot be bounced back |
| both nonces | freshness contributed by each side |
Message order:
```text
initiator -> responder : Hello { version, network_id, nonce_i }
initiator <- responder : HelloAck { version, nonce_r }
initiator -> responder : AuthProof { proof(initiator) }
initiator <- responder : AuthProof { proof(responder) } // only after the first verified
```
The responder emits nothing derived from `auth_key` until the initiator's proof
verifies, so a caller who does not know the secret learns nothing. Comparison
is constant time. Endpoint ids always come from the TLS certificate, never from
a peer's claim. Until both steps complete, no regular control message is
accepted in either direction. The whole exchange is bounded by
`Limits::handshake_timeout`.
A `Hello` naming a network this agent does not have active is rejected with
"unknown network". Because the claim is unverified at that point, the rejection
event does not report a network id.
## Control messages
After authentication, every frame is an `Envelope { network_id, message }` and
the `network_id` is re-checked against the session's network on every message.
A mismatch ends that session and is counted as a protocol violation; it does
not affect other networks.
| message | meaning |
|---|---|
| `Announce { hostname, capabilities }` | this agent's hostname and IP-plugin capabilities |
| `Ping { seq, payload }` | small request used to verify the exchange |
| `Pong { seq, payload }` | the echoed reply |
| `Bye { reason }` | graceful goodbye; not a revocation of anything |
`PluginCapability { protocol, version, enabled, data }` is opaque to the core:
`data` is bounded and handed to the matching plugin unparsed. Nothing in it is
ever treated as a shell command, filesystem path or OS setting.
Every decoded message is validated against `Limits` before it reaches anything
else. A rejected message never stops a network or the agent.
## Limits
Defaults from `Limits`, all configurable:
| limit | default |
|---|---|
| frame payload | 64 KiB |
| hostname | 255 bytes |
| capabilities per announcement | 16 |
| capability payload | 4 KiB |
| echo payload | 4 KiB |
| reason string | 256 bytes |
| handshake timeout | 10 s |
| dial timeout | 10 s |
| write timeout | 30 s |
| concurrent dials per network | 8 |
| sessions per network | 64 |
| inbound handshakes in flight | 32 |
| outbound queue per session | 64 |
Liveness of an established session is delegated to QUIC: iroh configures
keep-alives and an idle timeout, so a dead peer surfaces as a read error rather
than needing a heartbeat in this protocol.
docs/sync-model.md
+86
View File
@@ -0,0 +1,86 @@
# Planned state synchronisation
**Nothing in this document is implemented.** The proof of concept exchanges
hostname and capability announcements over live sessions and keeps no
replicated history. This file records the intended direction so the module
boundaries in [architecture.md](architecture.md) stay compatible with it, and so
nobody mistakes the current announcements for synchronisation.
There is no fake "ready CRDT" here and no snapshots that are not actually
verified.
## The problem
The network may be unstable. A participant can come back after months. So:
- no dependence on the author of a change being online;
- no dependence on acknowledgements from every participant ever seen;
- any available replica holding the signed data must be able to hand it to a
returning participant without the original author present.
## The model
**Signed self-contained state per author, merged between replicas.**
A record contains: the network, the author, the author's own retained version,
the full current content, and a signature. A change log may speed delivery up,
but recovery must never require the entire chain from the first event.
A network snapshot is a set of verifiable authored records plus the revocations
needed to interpret them. It is **not** a SQLite dump, and **not** a single
document trusted merely because the neighbour who forwarded it signed it.
## Merge rules
- A snapshot is merged into local state, never substituted for it wholesale.
- An older version never rolls back a newer known one.
- Absence from a snapshot does not mean deletion.
- Duplicates do not change the result.
- Two conflicting signed records at the same version from the same author need
explicit handling; they are not resolved by luck.
- Neither arrival order nor system clocks decide a winner.
- Compaction must not drop what is needed to stop revoked records being
resurrected.
## Hostnames
A hostname is a mutable binding to a persistent author, not an identity. A
rename must be a signed record that revokes the specific old binding and
announces the new one, ideally atomically in one record.
Turning a computer off is not a revocation of its hostname and does not remove
the participant. Revocations are not dropped merely because they are old, and
no acknowledgement from offline peers is required to keep working.
## Storage requirement this creates
When signed records land, writing the event and bumping the author's own
counter must happen in **one SQLite transaction, committed before the change is
published to the network**. SQLite gives atomic commit; use it instead of
separate, inconsistent writes. `state.sqlite` already has a schema version and
migrations for this.
## Limits to state honestly
- Data that every copy has lost is not recoverable from the secret.
- A signature proves authorship, not global freshness: a replica can be
behind, and you cannot tell from the signature alone.
- An isolated new client can end up with incomplete state and has no way to
know what it is missing.
- Anyone who knows the secret can author records, so a majority of records is
not evidence of anything.
## Future tests
These are **not implemented and must not be reported as passing**:
- snapshot merge against the rules above, including conflicting same-version
records;
- revocation propagation and resistance to resurrection after compaction;
- long network partitions and rejoin after an extended absence;
- hostname rename with atomic revoke-and-announce;
- recovery of a returning participant from a replica that is not the author;
- NAT traversal and hole punching between real hosts;
- relay fallback behaviour against a self-hosted relay;
- multi-process and multi-host deployment, as opposed to several library
instances inside one test process.
docs/testing.md
+66
View File
@@ -0,0 +1,66 @@
# Testing
How to run everything is in [../README.md](../README.md). Rules for writing
tests are in [../AGENTS.md](../AGENTS.md).
## Ground rules
Tests use real iroh endpoints on loopback, real handshakes, real SQLite in
per-test temporary directories, and independent agent instances. Discovery is
substitutable; **iroh, authentication, message passing and persistent storage
are not**.
The suite runs with no internet, no DHT, no public relay, no administrator
rights and no changes to OS network settings: endpoints bind `127.0.0.1:0` and
`[::1]:0`, relays are disabled, address lookup is cleared, port mapping is
disabled, and net-report probing is reduced to its minimum.
Synchronisation is always "wait for a specific event or condition under one
overall deadline" (`wait_event`, `wait_until`, 30 s). `settle()` exists only
for asserting that something did *not* happen. Ports are dynamic and
directories are isolated, so tests run in parallel.
Several library instances in one process is exactly that. It is **not** a test
of several system processes, and is not presented as one.
## What is covered
| # | scenario | file |
|---|---|---|
| 1 | deterministic identity: same name + secret ⇒ same space on different devices; a changed name or secret changes it; hostname, device key and restart do not | `tests/identity.rs` |
| 2 | four agents find each other, authenticate for real and exchange distinguishable messages; a late joiner is picked up; opaque plugin capabilities cross the control plane | `tests/multi_peer.rs` |
| 3 | an attacker who knows the address and the correct public `NetworkId` but not the secret is rejected at the handshake | `tests/authentication.rs` |
| 4 | one agent in two networks: statuses and messages do not mix; a session authenticated for one network cannot speak for the other; deactivating one leaves the other running | `tests/network_isolation.rs` |
| 5 | full stop and recreation from the same database: identity and settings survive, sessions come back automatically, a new local UDP port does not break recovery | `tests/restart.rs` |
| 6 | changing the secret through the library API: device identity survives, old sessions and messages get no access to the new space, and the retired space stays retired across a restart | `tests/restart.rs` |
| 7 | missing, corrupt and stale cache do not prevent connecting; a corrupt mandatory store is a clear error and never a fresh identity; a newer schema is refused; secrets stay out of status and `Debug` | `tests/cache_and_state.rs` |
| 8 | a dead candidate and a vanished peer do not block the others; retries are bounded and stop when the network is deactivated | `tests/resilience.rs` |
| 9 | wrong version, a message before authentication, a proof replayed on another connection, an oversized frame and a `Hello` for an inactive network are all rejected without taking the agent down | `tests/authentication.rs` |
| 10 | a second agent on the same state directory gets a clear error; after a clean stop the directory reopens; shutdown ends background tasks and refuses further work; independent agents coexist in one process | `tests/resilience.rs` |
`tests/discovery.rs` covers the discovery contract itself: a static bootstrap
candidate is enough to join, several backends compose, entries are withdrawn
when a network stops, and a forgotten network stays forgotten across a restart.
Unit tests in `src/proto/handshake.rs` cover the transcript construction
itself: role separation, channel binding, identity and network binding,
unambiguous encoding, and rejection under the wrong key.
`tests/end_to_end.rs` is the vertical slice: persistent identity → network
space → discovery → iroh → authentication → message exchange.
## Not covered, and not claimed to be
Listed in [sync-model.md](sync-model.md#future-tests): snapshots, revocations,
long partitions, hostname renames, recovery of a returning participant, NAT
traversal, relay fallback, and multi-process or multi-host deployment. None of
these are implemented, and none are marked as passing.
## Debugging a test
```bash
TSUNAGI_TEST_LOG=tsunagi=debug cargo test --test multi_peer -- --nocapture
```
The library never installs a global subscriber; the harness opts in only when
that variable is set.
docs/threat-model.md
+79
View File
@@ -0,0 +1,79 @@
# Threat model and known limits
Read this before relying on anything here. The protocol is in
[protocol.md](protocol.md).
## What is protected
- **Network membership.** A peer must prove knowledge of `auth_key`, derived
from the network name and shared secret, to get an authenticated session.
Knowing the public `NetworkId`, or an agent's address, is not enough.
- **Endpoint authenticity.** iroh's QUIC/TLS handshake authenticates the
remote endpoint id, which is its public key. Identities in the membership
transcript are taken from the certificate, never from a peer's claim.
- **Connection binding.** The membership proof includes TLS exporter output, so
a proof captured on one connection does not verify on another.
- **Role separation.** Initiator and responder proofs cover different
transcripts, so a proof cannot be reflected back at its sender.
- **Network isolation.** A session authenticated for network A cannot carry
messages for network B, even over a shared physical connection.
- **Confidentiality and integrity in transit.** Provided by QUIC/TLS. This
crate adds no encryption of its own.
- **Resource bounds.** Frame lengths are validated before allocation; strings,
lists, queues, concurrent dials and in-flight handshakes are all bounded;
handshakes, dials and writes have timeouts.
## What is not protected
- **Anyone who knows the secret is a full participant.** They can create
arbitrarily many identities, flood the network with records and collide with
other participants' names. This is why a majority is not a root of trust.
Signatures protect authorship; they do not make a participant honest.
- **Weak secrets.** This targets high-entropy secrets. There is no PAKE, so a
short human passphrase can be guessed offline by anyone who can reach the
handshake. Use `NetworkSecret::generate()`.
- **Addresses and metadata are observable.** Anyone able to watch the network
sees addresses, timing and volume. Discovery backends see the
`discovery_key` and the addresses published under it, which is enough to map
a network's participants. This library does not make a network anonymous, and
having iroh under it does not make it so.
- **A cloned state directory is a cloned identity.** `state.sqlite` holds the
device secret key and the network secrets. Copying it copies the participant.
Restoring an old backup rolls the agent's state back, which — once signed
records exist — can resurrect revoked information or replay stale versions.
- **A compromised host.** The secret is on disk to survive restarts. File
permissions are owner-only where the platform supports it, and the state
directory takes an ownership lock, but neither defends against a user who can
read the file or against malware running as that user.
- **User IP traffic.** Not carried here at all. Filtering it is the operating
system's and the user's job.
- **Denial of service.** Bounds and timeouts stop trivial resource exhaustion
from a single peer. They do not make the agent resistant to a determined
attacker who knows the secret, and no rate limiting per identity exists yet.
- **Global freshness.** A signature proves authorship, not that you have the
newest state. See [sync-model.md](sync-model.md).
- **Discovery is not trustworthy.** It returns candidates. A hostile or stale
discovery backend can waste dial attempts and learn addresses; it cannot
forge membership.
## Deliberate design consequences
- **No owner, no vote.** Nobody can evict anybody. Removing a participant means
changing the secret, which creates a different network space that the removed
participant cannot enter.
- **Rotating the secret is not revocation of past access.** Anyone who held the
old secret keeps whatever they already saw.
- **Local deactivation is not revocation.** Deactivating a network stops this
agent participating. It says nothing about anyone else.
- **Failures are contained, not escalated.** A bad proof, wrong secret,
malformed frame or unknown version rejects one message or one session. It
never stops another network and never stops the agent, and there is no
irreversible global error flag.
## Cryptographic choices
Standard primitives only, no home-made constructions: HKDF-SHA256 (RFC 5869)
for key separation, HMAC-SHA256 for the membership proof, constant-time
verification, iroh's Ed25519 endpoint keys and QUIC/TLS for the transport, and
RFC 5705 TLS exporter output for channel binding. There is no custom encryption
layer and no custom PAKE.