Separate control and data logically, move WireGuard into userspace, add a CLI
Corrects the architecture on two points raised in review, while the project is still small enough to change cheaply. 1. Control and data are separated *logically*, not physically. The old reading — "nothing but control may ride on iroh" — threw away iroh's whole value and would have forced the data plane to reimplement STUN, ICE and a relay. Now both planes ride on iroh with different ALPNs and different connections, so the data plane inherits hole punching and relay fallback, while proto/ still knows nothing about packets and dataplane/ knows nothing about the control protocol. New boundary: PacketTransport / PacketLink, an authenticated unreliable datagram channel per (network, peer, protocol). tsunagi/data/1 runs the same membership handshake, then DataOpen/DataOpenAck, then QUIC datagrams. Only the smaller endpoint id dials, so exactly one link exists per pair. A plugin is handed links and never learns reachability, so the WireGuard announcement shrank to a public key: there is no address left to lie about. 2. WireGuard now runs in userspace, on boringtun's protocol state machine. No kernel module, no wg tool, no ip shell-out, no loopback proxy: the wgtool, backend and bridge modules are gone. Only creating a TUN device needs privileges, and that sits behind TunFactory, so the entire data plane — handshake, encryption, routing, address ownership — is tested with none. Address ownership is enforced rather than believed: outbound packets go to the owner of the destination address, inbound packets are dropped unless their source is the address derived for the peer that sent them. 3. A `tsunagi` binary: secret, doctor, id, up. It owns the runtime, the logging subscriber and Ctrl-C, which the library still refuses to. Also fixes a reference cycle where IrohTransport held Arc<Inner>, which kept the databases open and the directory lock held after shutdown; two storage tests caught it once the cycle existed. 81 tests pass offline with no privileges, including real IPv6 packets crossing a real WireGuard tunnel over real iroh connections. Verified by hand: two CLI processes forming a mesh both on loopback and via n0 discovery using only an endpoint id. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
tsunagi
Claude Opus 5
+30
-12
@@ -12,12 +12,21 @@ authentication, participant announcements, capability exchange and — later —
|
||||
state synchronisation and delivery of IP-plugin data.
|
||||
|
||||
**Data plane.** Separate plugins create IP connectivity. WireGuard is the first
|
||||
one and is implemented — see [wireguard.md](wireguard.md). Plugin keys,
|
||||
configuration and lifecycle are separate from iroh identity and from the
|
||||
one and is implemented in userspace — see [wireguard.md](wireguard.md). Plugin
|
||||
keys, configuration and lifecycle are separate from iroh identity and from the
|
||||
network secret. The core moves an opaque, bounded payload and never parses it.
|
||||
|
||||
Only control messages travel over iroh. User IP traffic is not tunnelled
|
||||
through it; WireGuard packets travel over WireGuard's own UDP sockets.
|
||||
**The transport in between.** Plugins do not open connections. They are handed
|
||||
a `PacketLink` — an authenticated, unreliable datagram channel to one peer for
|
||||
one protocol — and never learn how it is carried.
|
||||
|
||||
The separation between the two planes is **logical, not physical**. Both ride
|
||||
on iroh, on different ALPNs and different connections. That is deliberate:
|
||||
iroh's whole value is hole punching a direct path between peers behind NAT,
|
||||
with a relay as fallback, and a data plane that refused to use it would have to
|
||||
reimplement all of it. What the separation buys is that `proto` knows nothing
|
||||
about packets and `dataplane` knows nothing about the control protocol, so
|
||||
either can be replaced on its own.
|
||||
|
||||
A plugin talks to the core through three narrow hooks — `on_network_activated`,
|
||||
a `PluginContext` for re-announcements and error reports, and a bounded
|
||||
@@ -36,27 +45,36 @@ and the agent stays manageable.
|
||||
| `proto` | message format, handshake, membership proof, protocol limits |
|
||||
| `agent` | agent and per-network lifecycle, reconnect, in-process message routing |
|
||||
| `storage` | mandatory state and the separately recoverable cache |
|
||||
| `dataplane::transport` | authenticated datagram links to peers; where reachability lives |
|
||||
| `dataplane` | the contract IP plugins implement, plus the WireGuard plugin |
|
||||
|
||||
Abstractions exist only where something is really substituted or really needs
|
||||
isolating for tests: `NetworkDiscovery`, `IpPlugin`, and `WireguardBackend`
|
||||
(which is what lets the plugin be tested in full without root). Everything else
|
||||
is a concrete type.
|
||||
isolating for tests: `NetworkDiscovery`, `IpPlugin`, `PacketTransport` /
|
||||
`PacketLink` (so the data plane's carrier can change), and `TunFactory` (which
|
||||
is what lets the whole data plane be tested without privileges). Everything
|
||||
else is a concrete type.
|
||||
|
||||
## Runtime shape
|
||||
|
||||
```text
|
||||
Agent one persistent identity, one iroh endpoint,
|
||||
├── EndpointAdapter one state directory, N networks
|
||||
├── EndpointAdapter (two ALPNs) one state directory, N networks
|
||||
├── Storage (state.sqlite + cache.sqlite + ownership lock)
|
||||
├── accept loop task ── weak ref, exits when the agent is dropped
|
||||
├── IrohTransport ── data plane links, weak ref back to the agent
|
||||
├── accept loop task ── routes by ALPN; weak ref, exits when the agent drops
|
||||
├── plugin request loop ── re-announcements and plugin error reports
|
||||
└── NetworkRuntime per NetworkId
|
||||
├── discovery + dial loop (bounded concurrency, backoff with jitter)
|
||||
└── Session per peer
|
||||
├── reader task ── frames in -> SessionEvent
|
||||
└── writer task ── encoded frames out
|
||||
├── Session per peer (control)
|
||||
│ ├── reader task ── frames in -> SessionEvent
|
||||
│ └── writer task ── encoded frames out
|
||||
└── PacketLink per (peer, plugin protocol), handed to the plugin
|
||||
```
|
||||
|
||||
Every strong reference from a background task back to the agent is a `Weak`.
|
||||
A cycle there would keep the databases open and the directory lock held
|
||||
forever after shutdown.
|
||||
|
||||
The library starts no runtime, installs no logging subscriber, handles no
|
||||
signals, never forks and never calls `process::exit`. Startup
|
||||
(`Agent::spawn`) and shutdown (`Agent::shutdown`) are explicit, background
|
||||
|
||||
@@ -132,6 +132,34 @@ A `Hello` naming a network this agent does not have active is rejected with
|
||||
"unknown network". Because the claim is unverified at that point, the rejection
|
||||
event does not report a network id.
|
||||
|
||||
## The data plane protocol
|
||||
|
||||
IP plugin packets never travel on a control connection. They use their own
|
||||
ALPN, `tsunagi/data/1`, on their own iroh connection:
|
||||
|
||||
```text
|
||||
initiator -> responder : (the same membership handshake as above)
|
||||
initiator -> responder : DataOpen { protocol }
|
||||
initiator <- responder : DataOpenAck { accepted, max_datagram }
|
||||
thereafter : QUIC datagrams carrying that plugin's packets
|
||||
```
|
||||
|
||||
The membership handshake is identical and bound to the same network, so a data
|
||||
channel cannot be opened by somebody who does not know the secret. `protocol`
|
||||
is bounded and must name a plugin the responder actually runs; otherwise the
|
||||
channel is declined, which is an ordinary outcome rather than an error.
|
||||
|
||||
Only one side dials — the one with the smaller endpoint id — so two agents
|
||||
never open two channels for the same thing.
|
||||
|
||||
Packets ride as QUIC **datagrams**: unreliable and unordered, which is what a
|
||||
tunnelled protocol wants, and free of the head-of-line blocking a stream would
|
||||
add. The datagram limit is what caps a plugin's MTU.
|
||||
|
||||
Separate connections mean separate congestion control, so a saturated data
|
||||
plane cannot delay control messages, and a data plane failure cannot take the
|
||||
control plane down with it.
|
||||
|
||||
## Control messages
|
||||
|
||||
After authentication, every frame is an `Envelope { network_id, message }` and
|
||||
|
||||
+2
-2
@@ -3,8 +3,8 @@
|
||||
**Nothing in this document is implemented.** The proof of concept exchanges
|
||||
hostname and capability announcements over live sessions and keeps no
|
||||
replicated history. That is also why WireGuard peer membership is
|
||||
session-scoped today: a peer disappears from the overlay configuration when its
|
||||
control session ends, because there is no agreed durable state to keep it. This file records the intended direction so the module
|
||||
session-scoped today: a peer leaves the overlay when its control session ends,
|
||||
because there is no agreed durable state to keep it. This file records the intended direction so the module
|
||||
boundaries in [architecture.md](architecture.md) stay compatible with it, and so
|
||||
nobody mistakes the current announcements for synchronisation.
|
||||
|
||||
|
||||
+18
-17
@@ -38,18 +38,17 @@ of several system processes, and is not presented as one.
|
||||
| 9 | wrong version, a message before authentication, a proof replayed on another connection, an oversized frame and a `Hello` for an inactive network are all rejected without taking the agent down | `tests/authentication.rs` |
|
||||
| 10 | a second agent on the same state directory gets a clear error; after a clean stop the directory reopens; shutdown ends background tasks and refuses further work; independent agents coexist in one process | `tests/resilience.rs` |
|
||||
|
||||
`tests/wireguard.rs` drives the WireGuard plugin over real iroh connections
|
||||
with the in-memory backend: a pair and a three-agent mesh converge to `N - 1`
|
||||
peers with locally derived `AllowedIPs`, a departing peer is removed, two
|
||||
networks get separate interfaces, keys and overlays, reconciliation repairs a
|
||||
configuration edited by hand, a backend failure leaves the control plane
|
||||
untouched, a restart keeps the WireGuard identity, shutdown removes every
|
||||
interface, a member claiming another member's overlay address is rejected, and
|
||||
the core carries the payload without interpreting it.
|
||||
|
||||
`tests/wireguard_system.rs` exercises the real `wg`/`ip` backend. It is
|
||||
**ignored by default** because it changes the host's network and needs Linux,
|
||||
wireguard-tools and `CAP_NET_ADMIN`.
|
||||
`tests/wireguard.rs` drives the WireGuard data plane over real iroh
|
||||
connections. Everything is real except the packet interface: real agents, real
|
||||
control plane, real data links, real WireGuard handshakes and encryption from
|
||||
boringtun, with an in-memory TUN device so none of it needs privileges. It
|
||||
covers real IPv6 packets travelling both ways through a tunnel, a three-agent
|
||||
mesh, a peer that sends from an address it does not own being dropped, packets
|
||||
for unowned addresses being counted rather than broadcast, a departing peer
|
||||
losing its tunnel, two networks keeping separate interfaces and keys, restart
|
||||
keeping the WireGuard identity, shutdown removing every interface, a forged
|
||||
overlay claim being rejected, and the core carrying the payload without
|
||||
interpreting it.
|
||||
|
||||
`tests/discovery.rs` covers the discovery contract itself: a static bootstrap
|
||||
candidate is enough to join, several backends compose, entries are withdrawn
|
||||
@@ -59,15 +58,17 @@ Unit tests in `src/proto/handshake.rs` cover the transcript construction
|
||||
itself: role separation, channel binding, identity and network binding,
|
||||
unambiguous encoding, and rejection under the wrong key.
|
||||
|
||||
Unit tests in `src/dataplane/wireguard/` cover the parts that would otherwise
|
||||
need root: key clamping against the RFC 7748 vector, overlay derivation,
|
||||
announcement validation including the address-hijack attempt, configuration
|
||||
building and rendering, the exact command plan the real backend would run, and
|
||||
parsing `wg showconf` and `ip address show` output.
|
||||
Unit tests in `src/dataplane/wireguard/` cover key clamping against the RFC
|
||||
7748 vector, overlay derivation, announcement validation including the
|
||||
address-hijack attempt, interface naming, and IP header parsing against
|
||||
truncated and nonsense input.
|
||||
|
||||
`tests/end_to_end.rs` is the vertical slice: persistent identity → network
|
||||
space → discovery → iroh → authentication → message exchange.
|
||||
|
||||
What the default suite does **not** cover is the real TUN interface, because
|
||||
that needs `CAP_NET_ADMIN`. Everything above it does run.
|
||||
|
||||
## Not covered, and not claimed to be
|
||||
|
||||
Listed in [sync-model.md](sync-model.md#future-tests): snapshots, revocations,
|
||||
|
||||
+20
-11
@@ -19,10 +19,15 @@ Read this before relying on anything here. The protocol is in
|
||||
messages for network B, even over a shared physical connection.
|
||||
- **Confidentiality and integrity in transit.** Provided by QUIC/TLS. This
|
||||
crate adds no encryption of its own.
|
||||
- **Overlay address ownership.** A WireGuard peer's `AllowedIPs` are derived
|
||||
from its public key, not taken from its announcement, so a member cannot
|
||||
claim another member's overlay address and receive its traffic. See
|
||||
[wireguard.md](wireguard.md#deterministic-overlay-addressing).
|
||||
- **Overlay address ownership.** A peer's overlay address is derived from its
|
||||
public key, not taken from its announcement. Outbound packets go to the owner
|
||||
of the destination address; inbound packets are dropped unless their source
|
||||
is the address derived for the peer that sent them. A member can therefore
|
||||
neither receive nor forge another member's traffic. See
|
||||
[wireguard.md](wireguard.md#address-ownership-is-enforced-not-announced).
|
||||
- **Tunnelled traffic is end-to-end encrypted by WireGuard**, independently of
|
||||
this crate. The transport underneath is also encrypted by iroh, but the
|
||||
tunnel's confidentiality does not depend on that.
|
||||
- **Resource bounds.** Frame lengths are validated before allocation; strings,
|
||||
lists, queues, concurrent dials and in-flight handshakes are all bounded;
|
||||
handshakes, dials and writes have timeouts.
|
||||
@@ -49,10 +54,14 @@ Read this before relying on anything here. The protocol is in
|
||||
permissions are owner-only where the platform supports it, and the state
|
||||
directory takes an ownership lock, but neither defends against a user who can
|
||||
read the file or against malware running as that user.
|
||||
- **User IP traffic.** Carried by the WireGuard plugin, not by iroh, and
|
||||
encrypted by WireGuard itself. *Filtering* it is still the operating system's
|
||||
and the user's job: the plugin creates connectivity between members and does
|
||||
not police what flows over it.
|
||||
- **User IP traffic.** Carried by the WireGuard plugin over an iroh data
|
||||
connection, and encrypted by WireGuard end to end. *Filtering* it is still
|
||||
the operating system's and the user's job: the plugin creates connectivity
|
||||
between members and does not police what flows over it.
|
||||
- **Traffic metadata reaches the relay when one is used.** If iroh cannot hole
|
||||
punch, the data connection goes through a relay, which then sees the volume
|
||||
and timing of tunnelled traffic — though not its contents, which WireGuard
|
||||
encrypted, nor the iroh layer's contents.
|
||||
- **Overlay address squatting.** A member can mint many WireGuard keys and
|
||||
therefore occupy many overlay addresses. It cannot pick which ones, but it
|
||||
can consume them and appear as many participants.
|
||||
@@ -60,9 +69,9 @@ Read this before relying on anything here. The protocol is in
|
||||
`wireguard.sqlite`, owner-only where the platform supports it. Copying that
|
||||
file copies this agent's overlay identity, exactly as copying `state.sqlite`
|
||||
copies its control plane identity.
|
||||
- **What the data plane does not police.** The plugin sets `AllowedIPs` per
|
||||
peer, which stops a member impersonating another member's overlay address.
|
||||
It does not stop a member sending whatever it likes *from its own* address.
|
||||
- **What the data plane does not police.** Address ownership stops a member
|
||||
impersonating another member. It does not stop a member sending whatever it
|
||||
likes *from its own* address.
|
||||
- **Denial of service.** Bounds and timeouts stop trivial resource exhaustion
|
||||
from a single peer. They do not make the agent resistant to a determined
|
||||
attacker who knows the secret, and no rate limiting per identity exists yet.
|
||||
|
||||
+115
-123
@@ -1,22 +1,54 @@
|
||||
# The WireGuard plugin
|
||||
# The WireGuard data plane
|
||||
|
||||
WireGuard is the first IP plugin. It creates real IP connectivity between
|
||||
WireGuard is the first IP plugin. It carries user traffic between
|
||||
participants while the control plane keeps doing its own job: deciding who is
|
||||
in the network and carrying each participant's opaque announcement.
|
||||
|
||||
Module boundaries are in [architecture.md](architecture.md), the control
|
||||
protocol in [protocol.md](protocol.md), and the security consequences in
|
||||
protocol in [protocol.md](protocol.md), the security consequences in
|
||||
[threat-model.md](threat-model.md).
|
||||
|
||||
## What stays separate
|
||||
## Userspace, not the kernel
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **User IP traffic** | Never goes through iroh. iroh carries announcements; packets travel over WireGuard's own UDP sockets. |
|
||||
| **Addresses** | An iroh address is an address for iroh. The plugin gathers and advertises its own reachability. |
|
||||
| **Payloads** | The core moves a bounded opaque blob. Only `dataplane::wireguard::announcement` interprets it. |
|
||||
| **Keys** | One WireGuard key per network, in the plugin's own store. Unrelated to the iroh device key and to the network secret. |
|
||||
| **Failures** | A data plane error is reported and retried. The control plane keeps running and the agent stays manageable. |
|
||||
WireGuard here is [boringtun]'s protocol state machine running in this
|
||||
process. There is **no kernel WireGuard module** and **no `wg` tool**: the same
|
||||
code runs everywhere, and the protocol can be exercised in tests without any
|
||||
privileges at all.
|
||||
|
||||
The only privileged step left is creating a packet interface so the operating
|
||||
system can hand us IP packets, and even that is behind a trait
|
||||
([`TunFactory`]) with an in-memory implementation.
|
||||
|
||||
| | needs privileges | what it proves |
|
||||
|---|---|---|
|
||||
| `MemoryTunFactory` | no | handshake, encryption, routing, address ownership |
|
||||
| `SystemTunFactory` | `CAP_NET_ADMIN` | traffic actually reaches the OS |
|
||||
|
||||
[boringtun]: https://docs.rs/boringtun
|
||||
[`TunFactory`]: https://docs.rs/tsunagi
|
||||
|
||||
## Where the packets go
|
||||
|
||||
The plugin does not know and does not care. It is handed a `PacketLink` per
|
||||
peer by the agent and runs a WireGuard tunnel over it:
|
||||
|
||||
```text
|
||||
TUN device (IP packets) PacketLink per peer
|
||||
| |
|
||||
v v
|
||||
destination address -> peer --Tunn.encapsulate--> ciphertext -> transport
|
||||
source address checked <--Tunn.decapsulate-- ciphertext <- transport
|
||||
```
|
||||
|
||||
Reachability — hole punching, relay fallback — belongs to the transport, which
|
||||
today is iroh. That is the whole reason the plugin's announcement says *who* it
|
||||
is and never *where* it is: there is no address for a peer to advertise, get
|
||||
wrong, or lie about.
|
||||
|
||||
**Two peers behind NAT work exactly as well as iroh does.** iroh hole punches a
|
||||
direct path when it can and falls back to a relay when it cannot; the tunnel
|
||||
rides on whichever it got. There is no separate STUN, no separate hole punching
|
||||
and no second set of NAT problems to solve for WireGuard.
|
||||
|
||||
## Deterministic overlay addressing
|
||||
|
||||
@@ -36,110 +68,73 @@ Two consequences matter:
|
||||
|
||||
* every member of a network derives the **same `/64`**, so the overlay is one
|
||||
subnet that nobody had to allocate;
|
||||
* a member's address is bound to its WireGuard public key, so
|
||||
**`AllowedIPs` are derived locally and never taken from what a peer claims**.
|
||||
* a member's address is bound to its WireGuard public key, so address
|
||||
ownership can be checked locally rather than believed.
|
||||
|
||||
That second point is the plugin's central security property. A participant who
|
||||
knows the network secret can mint as many WireGuard keys — and therefore as
|
||||
many overlay addresses — as it likes, but it cannot choose to collide with an
|
||||
existing member's address without finding a hash preimage. An announcement
|
||||
whose claimed address does not match the derivation is rejected outright.
|
||||
## Address ownership is enforced, not announced
|
||||
|
||||
## The announcement
|
||||
Kernel WireGuard enforces `AllowedIPs`. In userspace that is our job, and
|
||||
[`device`] does it on both sides:
|
||||
|
||||
Carried as the opaque `PluginCapability { protocol: "wireguard", .. }` payload,
|
||||
encoded with postcard:
|
||||
* **outbound**, a packet is routed to the peer that *owns* its destination
|
||||
address; a destination nobody owns is counted as unroutable and dropped;
|
||||
* **inbound**, a decrypted packet is dropped unless its *source* is exactly the
|
||||
address derived for the peer whose tunnel decrypted it.
|
||||
|
||||
| field | meaning |
|
||||
|---|---|
|
||||
| `version` | announcement format version, currently 1 |
|
||||
| `public_key` | the peer's X25519 WireGuard key |
|
||||
| `listen_port` | the UDP port its interface listens on |
|
||||
| `endpoints` | reachability the plugin gathered for itself, at most 8 |
|
||||
| `overlay_address` | what the peer believes its address is — cross-checked, never used |
|
||||
So a participant cannot receive traffic addressed to somebody else and cannot
|
||||
forge traffic that appears to come from somebody else. A participant who knows
|
||||
the network secret can mint many keys and therefore occupy many addresses, but
|
||||
it cannot choose to collide with an existing member without finding a hash
|
||||
preimage.
|
||||
|
||||
Validation, all before anything reaches a configuration: the version must
|
||||
match, the key must not be zero and must not be our own, the port must not be
|
||||
zero, the endpoint list must be within bounds, unusable endpoints
|
||||
(unspecified, multicast, broadcast, documentation, port zero) are dropped, and
|
||||
the claimed overlay address must equal the derived one.
|
||||
The announcement also carries the address the peer believes it has. It is never
|
||||
used — only cross-checked — so a version skew produces a clear rejection rather
|
||||
than silent non-connectivity.
|
||||
|
||||
## Building the configuration
|
||||
[`device`]: https://docs.rs/tsunagi
|
||||
|
||||
Each agent builds its **own** configuration from the agreed set of
|
||||
participants: for a full mesh of `N` members that is `N - 1` peers locally.
|
||||
Nobody hands a configuration to anybody else and no participant is
|
||||
authoritative.
|
||||
## MTU
|
||||
|
||||
* **Interface name** — `prefix + base32(network_id)`, truncated to the
|
||||
platform's 15 characters. Stable across restarts. Two agents on one host in
|
||||
the same network need different prefixes.
|
||||
* **Port** — `PortPolicy::Derived` picks a stable port from the network id
|
||||
inside a range, so a peer's cached endpoint keeps working across restarts and
|
||||
two networks on one host do not collide. `PortPolicy::Fixed` pins it.
|
||||
* **Addresses** — the agent's own `/128` plus the shared `/64`.
|
||||
* **Peer entries** — public key, derived `AllowedIPs`, the peer's first usable
|
||||
advertised endpoint, and a keepalive.
|
||||
Every packet rides in one transport datagram, and WireGuard adds 32 bytes. A
|
||||
QUIC datagram on a relayed path can be as small as roughly 1160 bytes, so the
|
||||
default interface MTU is **1100**, which leaves headroom rather than relying on
|
||||
the best case. Packets that do not fit are dropped and counted
|
||||
(`dropped_oversize`), never truncated. The observed datagram limit of each link
|
||||
is reported in the status output.
|
||||
|
||||
Nothing free-form from the network reaches a command argument or a
|
||||
configuration directive: peer keys, endpoints, prefixes and keepalives are
|
||||
typed values that the plugin re-serialises itself.
|
||||
## Lifecycle
|
||||
|
||||
## Backends
|
||||
* A network is activated → the plugin loads or creates its key for that
|
||||
network, derives the interface name, and creates the packet interface. If
|
||||
that fails — no privileges, for instance — the key and the announcement still
|
||||
work and the interface is retried on the next reconcile.
|
||||
* A peer announces its key → recorded.
|
||||
* A data link to that peer arrives → recorded.
|
||||
* Reconciliation starts a tunnel for every peer that has **both**, and removes
|
||||
tunnels for peers that lost either.
|
||||
* A network is deactivated, or the agent shuts down → the interface and every
|
||||
tunnel go away. The key stays, so coming back keeps the same overlay address.
|
||||
|
||||
The plugin computes *what* the interface should look like; a backend makes it
|
||||
so. Splitting them is what keeps every interesting decision testable without
|
||||
root.
|
||||
|
||||
* **`RecordingBackend`** — applies configurations in memory. The default test
|
||||
suite and the example use it, so neither needs privileges nor touches the
|
||||
host. It can also be told to fail, or to report drift.
|
||||
* **`WgToolBackend`** — drives the real `wg` and `ip` tools. Linux only,
|
||||
requires `CAP_NET_ADMIN`. It is split into a **pure planner** and pure
|
||||
parsers, which are unit tested on every platform, plus a thin executor. The
|
||||
WireGuard configuration is piped to `wg setconf` / `wg syncconf` on standard
|
||||
input, so the private key never reaches the filesystem.
|
||||
|
||||
Ownership is explicit: the plugin creates the interface and the plugin removes
|
||||
it. An interface that already exists and is not a WireGuard device is
|
||||
**refused, not adopted**, so the agent never takes over something it did not
|
||||
create. It changes no routing, DNS or firewall settings.
|
||||
|
||||
## Reconciliation
|
||||
|
||||
The plugin reconciles on every change — a peer announcement, a peer leaving —
|
||||
coalesced over a short debounce, and again on a timer. Each pass reads the
|
||||
interface back, compares it with the desired state, and applies only if they
|
||||
differ. A configuration edited by hand is therefore put back the way it should
|
||||
be, which is exactly what `reconciliation_repairs_a_configuration_edited_by_hand`
|
||||
in `tests/wireguard.rs` checks.
|
||||
|
||||
Deactivating a network removes its interface but **keeps its key**, so coming
|
||||
back later keeps the same overlay address. Agent shutdown removes every
|
||||
interface the plugin created.
|
||||
|
||||
## What the plugin needs from the core
|
||||
|
||||
Three small additions to the `IpPlugin` contract, all generic rather than
|
||||
WireGuard-specific:
|
||||
|
||||
* `on_network_activated` — prepare per-network state before any peer appears;
|
||||
* `attach(PluginContext)` — a handle to ask for a re-announcement when the
|
||||
plugin's own capability changes, and to report an error from its own tasks;
|
||||
* `shutdown` — remove system objects during a bounded agent shutdown.
|
||||
|
||||
Errors reported through the context are counted by the owning network's
|
||||
runtime, so `NetworkMetrics::plugin_errors` and `Event::PluginError` always
|
||||
agree.
|
||||
There is no external configuration file and no command line tool, so unlike a
|
||||
kernel-WireGuard setup there is nothing outside this process for anybody to
|
||||
edit. Reconciliation is purely "do the running tunnels match what is known".
|
||||
|
||||
## Using it
|
||||
|
||||
```bash
|
||||
# On both machines
|
||||
tsunagi up --network lab --secret "$SECRET" --wireguard
|
||||
```
|
||||
|
||||
See the two-machine walkthrough in [../README.md](../README.md#trying-it-on-two-machines).
|
||||
|
||||
From the library:
|
||||
|
||||
```rust,no_run
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
use tsunagi::config::{AgentConfig, StoragePaths, TransportPolicy};
|
||||
use tsunagi::dataplane::IpPlugin;
|
||||
use tsunagi::dataplane::wireguard::{WgToolBackend, WireguardConfig, WireguardPlugin};
|
||||
use tsunagi::dataplane::wireguard::{MemoryTunFactory, WireguardConfig, WireguardPlugin};
|
||||
use tsunagi::identity::{NetworkName, NetworkSecret};
|
||||
use tsunagi::{Agent, Result};
|
||||
|
||||
@@ -147,12 +142,14 @@ use tsunagi::{Agent, Result};
|
||||
async fn main() -> Result<()> {
|
||||
let paths = StoragePaths::user_default()?;
|
||||
|
||||
// The plugin's own state, separate from the agent's.
|
||||
let wireguard = WireguardConfig::new(paths.state_dir.join("wireguard"));
|
||||
let backend = WgToolBackend::new().expect("Linux with wg and CAP_NET_ADMIN");
|
||||
let plugin = WireguardPlugin::open(wireguard, Arc::new(backend))
|
||||
.await
|
||||
.expect("wireguard plugin");
|
||||
// MemoryTunFactory needs no privileges; swap in SystemTunFactory for a
|
||||
// real interface.
|
||||
let plugin = WireguardPlugin::open(
|
||||
WireguardConfig::new(paths.state_dir.join("wireguard")),
|
||||
Arc::new(MemoryTunFactory::new()),
|
||||
)
|
||||
.await
|
||||
.expect("wireguard plugin");
|
||||
|
||||
let agent = Agent::spawn(
|
||||
AgentConfig::new(paths)
|
||||
@@ -162,35 +159,30 @@ async fn main() -> Result<()> {
|
||||
.await?;
|
||||
|
||||
let network = agent
|
||||
.join_network(&NetworkName::new("kitchen-table")?, &NetworkSecret::generate())
|
||||
.join_network(&NetworkName::new("lab")?, &NetworkSecret::generate())
|
||||
.await?;
|
||||
|
||||
if let Some(view) = plugin.overview(network) {
|
||||
println!("{} on {} at {}", view.interface, view.overlay_prefix, view.overlay_address);
|
||||
println!("{} on {}", view.interface, view.overlay_address);
|
||||
}
|
||||
|
||||
tokio::time::sleep(Duration::from_secs(60)).await;
|
||||
agent.shutdown().await; // removes the interface
|
||||
agent.shutdown().await;
|
||||
Ok(())
|
||||
}
|
||||
```
|
||||
|
||||
There is a runnable version in `examples/wireguard_mesh.rs`, which uses the
|
||||
in-memory backend by default and the real one with `--real`.
|
||||
|
||||
## Limits and future work
|
||||
|
||||
* **Full mesh only.** Every member configures every other member. Routing
|
||||
through an intermediate participant is not implemented.
|
||||
* **No IPv4 overlay.** Addressing is IPv6 ULA, because it can be derived
|
||||
* **Full mesh only.** Every member runs a tunnel to every other member.
|
||||
Routing through an intermediate participant is not implemented.
|
||||
* **IPv6 overlay only.** Addressing is IPv6 ULA because it can be derived
|
||||
collision-free. An IPv4 overlay would need an allocator, which needs the
|
||||
agreed state described in [sync-model.md](sync-model.md).
|
||||
* **Peer membership is session-scoped.** A peer disappears from the
|
||||
configuration when its control session ends. Persisting membership across a
|
||||
long absence is part of the same future work.
|
||||
* **`WgToolBackend` is Linux only.** A netlink backend, and backends for macOS
|
||||
and Windows, are not implemented. `WgToolBackend::new()` fails with a clear
|
||||
message elsewhere.
|
||||
* **No MTU or path discovery.** The MTU is a configured constant.
|
||||
* **The real backend is not exercised by the default suite.** It needs root,
|
||||
so its tests live in `tests/wireguard_system.rs` behind `--ignored`.
|
||||
* **No routes, DNS or firewall rules.** The plugin creates its interface and
|
||||
nothing else. Anything beyond the overlay `/64` is the operator's business.
|
||||
* **Membership is session-scoped.** A peer leaves the overlay when its control
|
||||
session ends; surviving a long absence is the same future work.
|
||||
* **Userspace costs CPU.** Kernel WireGuard is faster. A kernel backend could
|
||||
return behind the same boundary, but it would give up transport-provided NAT
|
||||
traversal unless paired with a local proxy.
|
||||
* **The system interface path is barely exercised by the default suite**,
|
||||
because it needs privileges. Everything else about the data plane is.
|
||||
|
||||
Reference in New Issue
Block a user