Separate control and data logically, move WireGuard into userspace, add a CLI
Corrects the architecture on two points raised in review, while the project is still small enough to change cheaply. 1. Control and data are separated *logically*, not physically. The old reading — "nothing but control may ride on iroh" — threw away iroh's whole value and would have forced the data plane to reimplement STUN, ICE and a relay. Now both planes ride on iroh with different ALPNs and different connections, so the data plane inherits hole punching and relay fallback, while proto/ still knows nothing about packets and dataplane/ knows nothing about the control protocol. New boundary: PacketTransport / PacketLink, an authenticated unreliable datagram channel per (network, peer, protocol). tsunagi/data/1 runs the same membership handshake, then DataOpen/DataOpenAck, then QUIC datagrams. Only the smaller endpoint id dials, so exactly one link exists per pair. A plugin is handed links and never learns reachability, so the WireGuard announcement shrank to a public key: there is no address left to lie about. 2. WireGuard now runs in userspace, on boringtun's protocol state machine. No kernel module, no wg tool, no ip shell-out, no loopback proxy: the wgtool, backend and bridge modules are gone. Only creating a TUN device needs privileges, and that sits behind TunFactory, so the entire data plane — handshake, encryption, routing, address ownership — is tested with none. Address ownership is enforced rather than believed: outbound packets go to the owner of the destination address, inbound packets are dropped unless their source is the address derived for the peer that sent them. 3. A `tsunagi` binary: secret, doctor, id, up. It owns the runtime, the logging subscriber and Ctrl-C, which the library still refuses to. Also fixes a reference cycle where IrohTransport held Arc<Inner>, which kept the databases open and the directory lock held after shutdown; two storage tests caught it once the cycle existed. 81 tests pass offline with no privileges, including real IPv6 packets crossing a real WireGuard tunnel over real iroh connections. Verified by hand: two CLI processes forming a mesh both on loopback and via n0 discovery using only an endpoint id. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
+30
-12
@@ -12,12 +12,21 @@ authentication, participant announcements, capability exchange and — later —
|
||||
state synchronisation and delivery of IP-plugin data.
|
||||
|
||||
**Data plane.** Separate plugins create IP connectivity. WireGuard is the first
|
||||
one and is implemented — see [wireguard.md](wireguard.md). Plugin keys,
|
||||
configuration and lifecycle are separate from iroh identity and from the
|
||||
one and is implemented in userspace — see [wireguard.md](wireguard.md). Plugin
|
||||
keys, configuration and lifecycle are separate from iroh identity and from the
|
||||
network secret. The core moves an opaque, bounded payload and never parses it.
|
||||
|
||||
Only control messages travel over iroh. User IP traffic is not tunnelled
|
||||
through it; WireGuard packets travel over WireGuard's own UDP sockets.
|
||||
**The transport in between.** Plugins do not open connections. They are handed
|
||||
a `PacketLink` — an authenticated, unreliable datagram channel to one peer for
|
||||
one protocol — and never learn how it is carried.
|
||||
|
||||
The separation between the two planes is **logical, not physical**. Both ride
|
||||
on iroh, on different ALPNs and different connections. That is deliberate:
|
||||
iroh's whole value is hole punching a direct path between peers behind NAT,
|
||||
with a relay as fallback, and a data plane that refused to use it would have to
|
||||
reimplement all of it. What the separation buys is that `proto` knows nothing
|
||||
about packets and `dataplane` knows nothing about the control protocol, so
|
||||
either can be replaced on its own.
|
||||
|
||||
A plugin talks to the core through three narrow hooks — `on_network_activated`,
|
||||
a `PluginContext` for re-announcements and error reports, and a bounded
|
||||
@@ -36,27 +45,36 @@ and the agent stays manageable.
|
||||
| `proto` | message format, handshake, membership proof, protocol limits |
|
||||
| `agent` | agent and per-network lifecycle, reconnect, in-process message routing |
|
||||
| `storage` | mandatory state and the separately recoverable cache |
|
||||
| `dataplane::transport` | authenticated datagram links to peers; where reachability lives |
|
||||
| `dataplane` | the contract IP plugins implement, plus the WireGuard plugin |
|
||||
|
||||
Abstractions exist only where something is really substituted or really needs
|
||||
isolating for tests: `NetworkDiscovery`, `IpPlugin`, and `WireguardBackend`
|
||||
(which is what lets the plugin be tested in full without root). Everything else
|
||||
is a concrete type.
|
||||
isolating for tests: `NetworkDiscovery`, `IpPlugin`, `PacketTransport` /
|
||||
`PacketLink` (so the data plane's carrier can change), and `TunFactory` (which
|
||||
is what lets the whole data plane be tested without privileges). Everything
|
||||
else is a concrete type.
|
||||
|
||||
## Runtime shape
|
||||
|
||||
```text
|
||||
Agent one persistent identity, one iroh endpoint,
|
||||
├── EndpointAdapter one state directory, N networks
|
||||
├── EndpointAdapter (two ALPNs) one state directory, N networks
|
||||
├── Storage (state.sqlite + cache.sqlite + ownership lock)
|
||||
├── accept loop task ── weak ref, exits when the agent is dropped
|
||||
├── IrohTransport ── data plane links, weak ref back to the agent
|
||||
├── accept loop task ── routes by ALPN; weak ref, exits when the agent drops
|
||||
├── plugin request loop ── re-announcements and plugin error reports
|
||||
└── NetworkRuntime per NetworkId
|
||||
├── discovery + dial loop (bounded concurrency, backoff with jitter)
|
||||
└── Session per peer
|
||||
├── reader task ── frames in -> SessionEvent
|
||||
└── writer task ── encoded frames out
|
||||
├── Session per peer (control)
|
||||
│ ├── reader task ── frames in -> SessionEvent
|
||||
│ └── writer task ── encoded frames out
|
||||
└── PacketLink per (peer, plugin protocol), handed to the plugin
|
||||
```
|
||||
|
||||
Every strong reference from a background task back to the agent is a `Weak`.
|
||||
A cycle there would keep the databases open and the directory lock held
|
||||
forever after shutdown.
|
||||
|
||||
The library starts no runtime, installs no logging subscriber, handles no
|
||||
signals, never forks and never calls `process::exit`. Startup
|
||||
(`Agent::spawn`) and shutdown (`Agent::shutdown`) are explicit, background
|
||||
|
||||
Reference in New Issue
Block a user