Separate internal WireGuard encryption from data carriers
AWG-only policy also selected a competing end-to-end plugin, so compatible external links could still have no compatible encrypted data channel. Carrier changes replaced tunnel state and status selected counters from different engines. Use one internal WireGuard plugin and a separate DataCarrier contract for external capabilities/lifecycle. Keep the logical link, keys, handshake, interface and counter epoch across policy, fallback and direct/relay changes. Control/discovery transport selection and membership remain independent. Share the bounded per-neighbour UDP WireGuard engine between ordinary WG and Amnezia masking. Encrypt the relay envelope inside the outer hop channel with fresh pair keys/PSK. Remove duplicate WG key/device code and the TCP/TLS inner cipher. TLS carries opaque internal WireGuard frames. Refuse missing native transports instead of letting default iroh impersonate their protocol names. Probe the actual authenticated carrier, independently of the shared IP tunnel; failed local probe submissions count as loss too. Apply the vendored iroh interface exclusion to actual direct candidates, not just public discovery records. Exclude overlay subnets from introductions and propagate the actual owned interface name, including kernel-assigned utun names, to every carrier. Production logs showed paths through the overlay itself. This removes a recursive transport path; it does not establish the historical cause of the reported 100 GB absolute counter. Exit-node traffic is also real peer IP traffic even when local applications are idle. Hide inner cipher choices/handshake clutter in CLI and tray. Report external routes, readiness and actionable policy/version failures. Count original IP packets only when their ciphertext is accepted by the logical transport, including queued packets' exact unpadded lengths. Exclude physical probes and keepalives. Export counter epochs; sum per-peer rate deltas and discard stale histories so joins/restarts cannot turn old totals into fresh traffic. Status also prints exact IP byte totals for comparison with the tray. Coordinated upgrade: data ALPN 6, internal wireguard-e2e capability v4, wg/awg v2, QUIC carrier v1, TLS v2, local IPC v23. Control ALPN 8, device/network identity derivation, signed records and saved network choices stay unchanged. Keep the historical wg-quic key store; transactionally import missing keys from wg then awg without replacing canonical keys. Forget removes legacy copies too, on the blocking pool, to prevent resurrection after restart. Preserve historic cipher options as shared settings, rejecting contradictory values. Validation on Linux in the Nix development shell: - cargo fmt --all -- --check - cargo clippy --locked --workspace --all-targets -- -D warnings - cargo test --locked --workspace --all-targets (562 passed, 3 ignored) The inherited service socket/secret/group environment overrides were removed for isolated CLI process tests. A new carrier test initially raced the first peer announcement; its deadline polling now waits for a peer row explicitly. Real loopback endpoints, membership, SQLite, WG/AWG/TLS encryption and memory TUNs cover mixed-carrier relaying and one smoltcp TCP connection surviving relay/direct/policy changes and a dropped transmission. Counter tests check exact plaintext lengths, refused sends, epochs, joins and stale histories. Public DHT, privileged kernel and manual benchmark tests remain ignored; macOS/Windows native TUNs, desktop GUI and WAN/DPI behavior were not run.
This commit is contained in:
1 parent
8a6216bccb
commit
2166ba4800
76 files changed
+3903
-6811
No files matched your search
+29
-14
@@ -17,13 +17,16 @@ tsng protocols home awg
|
||||
```
|
||||
|
||||
Changes are local to this device, persist across restarts, and retire existing
|
||||
data links and end-to-end channels for that network before renegotiation.
|
||||
physical data links for that network before renegotiation. The single internal
|
||||
WireGuard channel, its keys, session and counters survive.
|
||||
Authenticated control sessions, membership, addresses and other networks stay
|
||||
live. Removing every protocol (`none` in the CLI, all checkboxes off in the tray)
|
||||
leaves the network control-only. This is different from automatic carrier
|
||||
fallback, which preserves the selected end-to-end channel.
|
||||
leaves the network control-only. Re-enabling a carrier resumes the existing
|
||||
internal channel, subject to ordinary WireGuard and application timeouts. The
|
||||
choice applies to both incoming and outgoing local data links; a relay may use
|
||||
a different transport on its next hop. Control and discovery are independent.
|
||||
|
||||
`awg` is Tsunagi AWG wire version 1. Its packet masking engine is adapted from
|
||||
`awg` is Tsunagi AWG wire version 2. Its packet masking engine is adapted from
|
||||
[RunaRoo/AmneziaWG-RS](https://github.com/RunaRoo/AmneziaWG-RS), with the pinned
|
||||
source and license recorded in
|
||||
[UPSTREAM.md](../crates/tsunagi-awg/UPSTREAM.md). It uses the existing boringtun
|
||||
@@ -36,7 +39,7 @@ not implemented: this carrier connects authenticated Tsunagi members.
|
||||
There is one independent UDP socket and cancellable packet-engine task per
|
||||
`(network, peer)` pair. The smaller endpoint id initiates, following the same
|
||||
rule as other data carriers. Both sides must announce enabled AWG wire version
|
||||
1. An authenticated iroh control session is required throughout; AWG supplies
|
||||
2. An authenticated iroh control session is required throughout; AWG supplies
|
||||
no alternate discovery or membership mechanism.
|
||||
|
||||
Every `open` generates new masking parameters, a header-protection key, an
|
||||
@@ -100,25 +103,37 @@ candidates. IPv6-only paths, symmetric NAT and networks blocking arbitrary UDP
|
||||
may fail AWG and fall back; there is no AWG-over-QUIC substitution or AWG TURN
|
||||
relay hidden behind its name.
|
||||
|
||||
Data delivery checks go through the actual end-to-end encrypted tunnel, once
|
||||
per second, independently of control-plane RTT. Requests and replies bind a
|
||||
Data delivery checks go through the specific authenticated physical carrier,
|
||||
once per second, independently of control-plane RTT or the shared internal
|
||||
tunnel. A successful relay path cannot mask a broken direct carrier. Requests and replies bind a
|
||||
fresh private marker/key to the sender's endpoint id. Unanswered requests expire
|
||||
after three seconds. After a 20-second warmup, at least 15 completed observations
|
||||
and at least 60% loss in the last 20 observations retire a carrier. Probes are
|
||||
consumed locally after source/destination admission and never enter the TUN.
|
||||
consumed at the carrier boundary and never enter the TUN or user IP counters.
|
||||
AWG additionally checks encrypted outer delivery throughout the pair lifetime.
|
||||
|
||||
Setup failure or persistent runtime loss advances to the next mutually supported
|
||||
carrier in the initiator's order. Earlier expired cooldowns do not jump ahead of
|
||||
untried carriers. Retry cooldowns remain bounded; reaching AWG again negotiates
|
||||
new parameters. Control sessions and the logical end-to-end tunnel survive a
|
||||
carrier change. Status names the end-to-end protocol and the actual carrier
|
||||
separately, so `wg` over `awg` is a valid result.
|
||||
carrier change. Status and the tray show the actual external transport or relay
|
||||
route and data readiness. There is no user-selectable internal cipher.
|
||||
|
||||
Control wire version 6 introduces the private transport message. All members
|
||||
must upgrade together; device identities, network ids, handshake transcript
|
||||
encoding, signed state and stored network configuration are unchanged. Data
|
||||
ALPN remains `tsunagi/data/5`; the relay envelope is unchanged.
|
||||
The internal capability is `wireguard-e2e` v4; external `wg` and `awg` use v2,
|
||||
`wg-quic` v1 and `tcp-tls` v2. All members must upgrade together. Control ALPN
|
||||
`tsunagi/ctrl/8`, discovery, membership and signed state are unchanged. The data
|
||||
ALPN is `tsunagi/data/6`; the relay envelope format is unchanged. Local IPC is v23,
|
||||
so agent, CLI and tray must match. Historical `wg-quic/wireguard.sqlite` is the
|
||||
canonical internal key store. Missing keys are imported transactionally from
|
||||
`wg` then `awg`, without replacing an existing canonical key or modifying a
|
||||
source database. Corrupt key stores fail rather than silently regenerating keys.
|
||||
Device identities, network ids, addresses and saved transport choices survive.
|
||||
|
||||
Ordinary `wg` uses the same independent pair engine with standard WireGuard
|
||||
headers, no masking/junk, fresh hop keys and PSK, and a preferred local UDP port
|
||||
(default 51820; additional pairs use dynamic ports). The whole relay envelope is
|
||||
inside the hop encryption. It is separate from the internal WireGuard channel.
|
||||
Neither UDP mode is a client for an externally configured WireGuard server.
|
||||
|
||||
## Verification
|
||||
|
||||
|
||||
+13
-8
@@ -11,14 +11,15 @@ messages. On top of it, this crate's protocol handles membership
|
||||
authentication, participant announcements, capability exchange and — later —
|
||||
state synchronisation and delivery of IP-plugin data.
|
||||
|
||||
**Data plane.** Separate plugins create IP connectivity. WireGuard is the first
|
||||
one and is implemented in userspace — see [wireguard.md](wireguard.md). Plugin
|
||||
**Data plane.** One internal WireGuard plugin creates IP connectivity in userspace — see [wireguard.md](wireguard.md). Plugin
|
||||
keys, configuration and lifecycle are separate from iroh identity and from the
|
||||
network secret. The core moves an opaque, bounded payload and never parses it.
|
||||
|
||||
**The transport in between.** Plugins do not open connections. They are handed
|
||||
a `PacketLink` — an authenticated, unreliable datagram channel to one peer for
|
||||
one protocol — and never learn how it is carried.
|
||||
the internal cipher — and never learns how it is carried. External `DataCarrier`
|
||||
adapters own their reachability capabilities and physical link lifecycle; they
|
||||
never receive plaintext overlay IP packets.
|
||||
|
||||
The separation between the two planes is **logical, not physical**. The control plane uses
|
||||
iroh; the data plane can use iroh on its own ALPN and connection, or independent
|
||||
@@ -73,8 +74,10 @@ And in crates of their own:
|
||||
|
||||
| crate | responsibility |
|
||||
|---|---|
|
||||
| `tsunagi-awg` | ephemeral pairwise Amnezia masking, private negotiation and independently encrypted UDP carriers; reuses the end-to-end WireGuard engine |
|
||||
| `tsunagi-wg-quic` | the `wg-quic` protocol: its keys, its announcement, its tunnels |
|
||||
| `tsunagi-wireguard` | independent ordinary WG and masked UDP hop engines, private pair negotiation and local candidates |
|
||||
| `tsunagi-tcp-tls` | TLS hop encryption and bounded reachability announcements; no inner IP cipher |
|
||||
| `tsunagi-awg` | ephemeral pairwise Amnezia masking, private negotiation and independently encrypted UDP carriers; shares the independent UDP hop engine with `wg` |
|
||||
| `tsunagi-wg-quic` | the single internal WireGuard cipher, keys and opaque announcement (historical crate name) |
|
||||
| `tsunagi` | the command line agent, built as the `tsng` executable: the only place that owns a runtime, a logger and signals |
|
||||
|
||||
The core library is the package `tsunagi-core`; its library target is still called `tsunagi`, so code says `use tsunagi::…`.
|
||||
@@ -85,7 +88,7 @@ they compare the *wire* version, which moves only when the bytes do, so two
|
||||
peers on different releases keep working.
|
||||
|
||||
Abstractions exist only where something is really substituted or really needs
|
||||
isolating for tests: `NetworkDiscovery`, `IpPlugin`, `PacketTransport` /
|
||||
isolating for tests: `NetworkDiscovery`, one `IpPlugin`, external `DataCarrier`, `PacketTransport` /
|
||||
`PacketLink` (so a protocol's carrier can change), `TunFactory` (which is what
|
||||
lets the whole data plane be tested without privileges), `InterfaceProvisioner`
|
||||
and `DnsPublisher` (which are where the platforms differ). Everything else is a
|
||||
@@ -161,8 +164,10 @@ Values that cannot honestly be attributed to one network stay at the endpoint
|
||||
level. A value iroh does not report is `None`.
|
||||
|
||||
An iroh address is an address for iroh. It is not assumed usable by any other
|
||||
protocol; a future WireGuard plugin gathers its own reachability data and ships
|
||||
it through the control plane as an opaque announcement.
|
||||
protocol; each external carrier gathers its own reachability data and ships it through
|
||||
the control plane as an opaque announcement or private pair negotiation.
|
||||
Local iroh candidates and introductions exclude the agent's own overlay, so
|
||||
transport traffic cannot recursively tunnel through itself.
|
||||
|
||||
### Relays
|
||||
|
||||
|
||||
+4
-4
@@ -15,9 +15,9 @@ Upgrading the crate or bumping the control protocol must never change an
|
||||
existing `NetworkId`.
|
||||
|
||||
Control version 8 adds signed, multi-writer VIP field snapshots to version 7's
|
||||
node retirements. Data ALPN remains `tsunagi/data/5`. All members must upgrade
|
||||
node retirements. Data ALPN is `tsunagi/data/6` for the single internal WireGuard channel. All members must upgrade
|
||||
together. Network ids, derivation labels, handshake encoding and existing record
|
||||
signatures are unchanged. Local CLI/tray IPC is version 22; restart the agent
|
||||
signatures are unchanged. Local CLI/tray IPC is version 23; restart the agent
|
||||
with matching clients. SQLite schema 10 adds `vip_fields`, preserving identity,
|
||||
network configuration, signed node records and retirement proofs.
|
||||
See [virtual services](vip.md), [state sync](sync-model.md) and [AWG](amnezia.md).
|
||||
@@ -145,7 +145,7 @@ event does not report a network id.
|
||||
## The data plane protocol
|
||||
|
||||
IP plugin packets never travel on a control connection. They use their own
|
||||
ALPN, `tsunagi/data/5`, on their own iroh connection:
|
||||
ALPN, `tsunagi/data/6`, on their own iroh connection:
|
||||
|
||||
```text
|
||||
initiator -> responder : (the same membership handshake as above)
|
||||
@@ -192,7 +192,7 @@ and version 5 added the end-to-end protocol to it:
|
||||
| source peer | 32 | original endpoint public key |
|
||||
| destination peer | 32 | final endpoint public key |
|
||||
| flow id | 8 | big endian opaque stable hash supplied by the plugin |
|
||||
| protocol | 8 | big endian FNV-1a of the end-to-end protocol id (`wg`, `tcp-tls`, ...) |
|
||||
| protocol | 8 | big endian FNV-1a of the internal protocol id (`wireguard-e2e`) |
|
||||
| payload | remaining | end-to-end encrypted plugin bytes |
|
||||
|
||||
The fixed header is 82 bytes. Zero/oversized hop limits, unknown versions,
|
||||
|
||||
+13
-11
@@ -16,8 +16,8 @@ the default 1280-byte interface MTU also works on small QUIC paths.
|
||||
|
||||
Each authenticated control session advertises that peer's own direct data
|
||||
links. There is one table for each network, whatever protocols run over it: a
|
||||
link is a way to reach a neighbour, and a protocol is an end-to-end tunnel to a
|
||||
peer. The envelope names the protocol, so the two are independent. A neighbour
|
||||
link is a way to reach a neighbour, while the single internal protocol is an
|
||||
end-to-end tunnel to a peer. The envelope names the protocol, so the two are independent. A neighbour
|
||||
reached over several kinds of transport (UDP, TCP, QUIC datagrams) is sent to
|
||||
over the best open one, in the order the protocols were configured, and over
|
||||
the next if that one fails or cannot carry the frame. Every authenticated
|
||||
@@ -28,14 +28,16 @@ Link arrival, closure and changed announcements rebuild routes immediately.
|
||||
Unchanged announcements refresh their age without republishing the forwarding
|
||||
snapshot.
|
||||
|
||||
Which protocol a pair of members runs their tunnel in is settled without
|
||||
asking: the member with the lower endpoint id decides, by the order it
|
||||
announces its protocols in, among those both have enabled at the same wire
|
||||
version. Both ends read the same answer from what they already hold. The
|
||||
tunnel is created as soon as the peer is reachable by any path, and it does not
|
||||
change when a direct link comes or goes underneath it: as soon as a direct
|
||||
link comes up it is shorter than the path through others, and traffic moves to
|
||||
it with nothing to redo. The user sees only a lower round-trip time.
|
||||
Every pair uses one internal WireGuard channel, matched by its fixed opaque
|
||||
capability and independent of the selected external protocols. The tunnel is
|
||||
created when a member and its key are known; its logical link survives temporary
|
||||
loss of all physical paths. Carrier selection changes, automatic fallback and
|
||||
shorter routes keep the same keys, interface and counters. The lower endpoint
|
||||
id still chooses the mutually enabled external carrier in its preference order.
|
||||
A relayed AWG first hop can lead to a QUIC next hop: the user's choice governs
|
||||
this device's adjacent links, while the inner ciphertext stays unchanged.
|
||||
An existing TCP connection continues after transient loss if its own timeout
|
||||
has not expired; a sufficiently long outage can still terminate it.
|
||||
|
||||
Breadth-first search computes the shortest directed paths and every equal-cost
|
||||
first hop in sorted order. An actual direct link always wins. The first row is
|
||||
@@ -124,5 +126,5 @@ counts sends without storing frames. Results are CPU forwarding cost, **not**
|
||||
end-to-end network latency or VPN throughput; encryption, fragmentation,
|
||||
sockets, congestion and scheduling contribute separately.
|
||||
|
||||
Wire compatibility: control ALPN `tsunagi/ctrl/8`, data ALPN `tsunagi/data/5`.
|
||||
Wire compatibility: control ALPN `tsunagi/ctrl/8`, data ALPN `tsunagi/data/6`.
|
||||
Upgrade every participant together; saved identities and network state persist.
|
||||
+23
-3
@@ -185,7 +185,8 @@ validation. Tests are offline; public STUN/DPI/NAT effectiveness needs separate
|
||||
real-network validation.
|
||||
|
||||
Runtime protocol selection tests switch real AWG and iroh/QUIC carriers and
|
||||
end-to-end WireGuard tunnels from either endpoint-id role, carry full-size
|
||||
external policies from either endpoint-id role while preserving the one internal
|
||||
WireGuard tunnel, key, handshake and counter epoch, carry full-size
|
||||
packets, keep another active network's tunnels unchanged, disable and re-enable
|
||||
the data plane without losing control sessions, and preserve selection after
|
||||
restart. CLI tests use a real agent process and control socket to exercise
|
||||
@@ -225,8 +226,8 @@ to test the entire mesh inside these namespaces.
|
||||
|
||||
Devices-window tests render the egui grid at multiple widths with its real
|
||||
scroll area. They check stretching and shrinking without an expanding first
|
||||
column, separate handshake/control RTT values, unknown measurements and short
|
||||
end-to-end hints that do not substitute the carrier protocol. This does not
|
||||
column, control RTT without internal handshake/protocol clutter, unknown measurements
|
||||
and data readiness/error hints. This does not
|
||||
establish native desktop rendering on every platform.
|
||||
|
||||
|
||||
@@ -252,3 +253,22 @@ widths, click a diagram backend, and distinguish the new-flow selection from
|
||||
traffic on an existing binding. Sampling checks cover idle open flows, stale
|
||||
observations, counter resets, recreated listeners and removal of cached
|
||||
histories. These are headless egui checks, not native desktop screenshots.
|
||||
|
||||
`tsunagi-wg-quic/tests/carriers.rs` uses the same internal WireGuard plugin over
|
||||
real ordinary WG UDP and TCP/TLS, with loopback address selection and memory
|
||||
TUNs. Exact original IP byte/packet counts exclude health probes and encryption
|
||||
overhead. The AWG suite connects AWG-only and QUIC-only endpoints through a
|
||||
mixed relay. One real smoltcp TCP connection stays established through relay →
|
||||
direct QUIC → a dropped packet with all local carriers off → AWG relay; both
|
||||
middle TUNs and endpoint keys are checked. This exercises a real userspace TCP
|
||||
stack, not a host SSH server. Legacy key migration tests cover missing keys,
|
||||
canonical precedence, idempotence and damaged source files.
|
||||
|
||||
Candidate tests exercise the vendored iroh local address filter on real bound
|
||||
endpoints and exclude overlay subnets from introductions. A renaming memory TUN checks
|
||||
that carriers receive the actual interface name, as required for macOS utun.
|
||||
A missing native WG transport is refused rather than silently carried by QUIC. Tray traffic tests
|
||||
cover a joining peer with 100 GB of prior counters, departed peers, removed
|
||||
networks and new counter epochs, including resets hidden by growth in the other
|
||||
direction. These distinguish rates from absolute volumes; they do not establish
|
||||
the cause of an earlier unrecorded high absolute counter on a real host.
|
||||
@@ -130,7 +130,7 @@ AWG profiles, PSKs and outer keys are private to one authenticated pair attempt,
|
||||
not shared network state. Address candidates are unverified until authenticated
|
||||
outer WireGuard data arrives. Header protection and structured signature packets
|
||||
reduce fixed wire markers; they do not guarantee admission by DPI or hide traffic
|
||||
analysis. iroh control availability remains a prerequisite. Encrypted end-to-end
|
||||
delivery probes detect sustained carrier loss, never confer membership and pass
|
||||
the same signed source/destination admission as ordinary IP traffic. See
|
||||
analysis. iroh control availability remains a prerequisite. Authenticated physical-link
|
||||
delivery probes detect sustained loss on that carrier, independently of a
|
||||
working end-to-end relay path. They never confer membership or enter the TUN. See
|
||||
[amnezia.md](amnezia.md) for limits and tested scope.
|
||||
+64
-95
@@ -1,6 +1,6 @@
|
||||
# The WireGuard data plane
|
||||
|
||||
WireGuard is the first IP plugin. It carries user traffic between
|
||||
WireGuard is the single internal IP cipher. It carries user traffic between
|
||||
participants while the control plane keeps doing its own job: deciding who is
|
||||
in the network and carrying each participant's opaque announcement.
|
||||
|
||||
@@ -46,20 +46,26 @@ peer by the agent and runs a WireGuard tunnel over it:
|
||||
source address checked <--Tunn.decapsulate-- ciphertext <- transport
|
||||
```
|
||||
|
||||
Reachability — hole punching, relay fallback — belongs to the transport, which
|
||||
today is iroh. That is the whole reason the plugin's announcement says *who* it
|
||||
is and never *where* it is: there is no address for a peer to advertise, get
|
||||
wrong, or lie about.
|
||||
Reachability belongs to external carriers: ordinary WireGuard UDP (`wg`),
|
||||
Amnezia-masked UDP (`awg`), iroh QUIC (`wg-quic`), or TCP/TLS (`tcp-tls`). The
|
||||
user's ordered choice applies only to those physical data links. The internal
|
||||
capability `wireguard-e2e` v4 is fixed and hidden from the CLI/tray controls.
|
||||
All carriers move the same inner encrypted bytes, including through relays.
|
||||
The encryption plugin's announcement identifies a key, never an IP endpoint.
|
||||
|
||||
**Two peers behind NAT work exactly as well as iroh does.** iroh hole punches a
|
||||
direct path when it can and falls back to a relay when it cannot; the tunnel
|
||||
rides on whichever it got. There is no separate STUN, no separate hole punching
|
||||
and no second set of NAT problems to solve for WireGuard.
|
||||
A relay removes its adjacent carrier's encryption to read routing metadata,
|
||||
then sends the inner ciphertext on another carrier. It cannot decrypt the
|
||||
endpoint IP packet. WireGuard session keys, counters and the interface survive
|
||||
carrier changes and direct/relay changes. TCP inside the overlay retransmits
|
||||
after transient loss; ordinary application and WireGuard timeouts still apply.
|
||||
The UDP modes have their own bounded STUN/pair discovery and can fail on NATs
|
||||
that iroh can traverse; selecting only AWG does not permit QUIC fallback.
|
||||
|
||||
## Checking it from outside
|
||||
|
||||
`tsng status` asks a running agent over its local control socket and prints
|
||||
what it sees, including whether each tunnel has actually handshaken. See
|
||||
the external transport or relay route, data readiness and plaintext counters.
|
||||
An unavailable channel includes a reason such as no common permitted transport. See
|
||||
[../README.md](../README.md#checking-that-it-works).
|
||||
|
||||
## Deterministic overlay addressing
|
||||
@@ -207,26 +213,40 @@ QUIC DATAGRAM frames themselves cannot fragment.
|
||||
Reassembly is bounded and expires incomplete packets; one lost fragment loses
|
||||
one packet, without blocking unrelated traffic. The logical data payload limit
|
||||
is 64 KiB, with the relay envelope subtracted before it reaches the plugin.
|
||||
The new framing uses data ALPN `tsunagi/data/5`; both ends and intermediate
|
||||
The new framing uses data ALPN `tsunagi/data/6`; both ends and intermediate
|
||||
peers need the updated binary. Network identities and saved state do not change.
|
||||
See [protocol.md](protocol.md#the-data-plane-protocol) for the wire format.
|
||||
|
||||
## Lifecycle
|
||||
## Lifecycle and counters
|
||||
|
||||
* A network is activated → the plugin loads or creates its key for that
|
||||
network, derives the interface name, and creates the packet interface. If
|
||||
that fails — no privileges, for instance — the key and the announcement still
|
||||
work and the interface is retried on the next reconcile.
|
||||
* A peer announces its key → recorded.
|
||||
* A data link to that peer arrives → recorded.
|
||||
* Reconciliation starts a tunnel for every peer that has **both**, and removes
|
||||
tunnels for peers that lost either.
|
||||
* A network is deactivated, or the agent shuts down → the interface and every
|
||||
tunnel go away. The key stays, so coming back keeps the same overlay address.
|
||||
The agent owns one interface across every network. On network activation the
|
||||
internal plugin loads a network-specific key, announces it and reconciles
|
||||
peer tunnels. It owns no interface or host route. Local IP ingress resolves
|
||||
the destination owner and sends only through this one plugin. Source admission
|
||||
is enforced after decryption by the agent, not by trusting a remote address.
|
||||
|
||||
There is no external configuration file and no command line tool, so unlike a
|
||||
kernel-WireGuard setup there is nothing outside this process for anybody to
|
||||
edit. Reconciliation is purely "do the running tunnels match what is known".
|
||||
A carrier policy change retires physical links while retaining the logical
|
||||
peer link and the internal tunnel. A peer's control membership ending removes
|
||||
its tunnel; network deactivation removes that network's channels. Its persistent
|
||||
key remains. The interface lives until agent shutdown, independently of carrier
|
||||
availability and individual network lifetimes.
|
||||
|
||||
The historical `wg-quic/wireguard.sqlite` remains the canonical key store.
|
||||
Missing keys migrate from `wg` then `awg` without replacing canonical keys or
|
||||
modifying the source files. Corrupt legacy databases are hard errors. Old
|
||||
`-o keepalive=…`, `-o wg:keepalive=…` and `mtu` settings configure the shared
|
||||
internal channel; contradictory values are rejected. Interface MTU is selected
|
||||
once with `--mtu`, independent of carrier selection.
|
||||
|
||||
User traffic counters count original IP bytes once when the encrypted frame is
|
||||
accepted by the logical transport, and once when decrypted on receipt. They do
|
||||
not promise remote delivery. They exclude carrier overhead, handshakes,
|
||||
keepalives, relay transit and carrier delivery probes. Packets queued before a
|
||||
handshake retain their exact original lengths. Each new tunnel gets an opaque
|
||||
counter epoch so tray deltas cannot cross a restart, including when its new
|
||||
counter already exceeds the old one. A carrier switch does not change that
|
||||
epoch. Network rates sum per-peer deltas; peers arriving with prior traffic
|
||||
never add that history to the current rate.
|
||||
|
||||
## Using it
|
||||
|
||||
@@ -238,71 +258,21 @@ tsng join --network lab --secret "$SECRET"
|
||||
|
||||
See the two-machine walkthrough in [../README.md](../README.md#trying-it-on-two-machines).
|
||||
|
||||
From the library:
|
||||
From the library, configure one `WireguardPlugin` with
|
||||
`AgentConfig::with_tunnel`, the agent's `with_interface`, and independent
|
||||
`with_carrier` adapters. UDP/TLS carriers also register their physical
|
||||
`PacketTransport` using `with_custom_transport`. Their own accept tasks pass
|
||||
real links to `Agent::install_inbound_link`. The `carriers.rs` integration test
|
||||
is a rootless example for ordinary WG and TLS; `tsunagi-awg/tests/pairs.rs`
|
||||
covers mixed transports and relay transitions.
|
||||
|
||||
```rust,no_run
|
||||
use std::sync::Arc;
|
||||
use tsunagi::config::{AgentConfig, StoragePaths, TransportPolicy};
|
||||
use tsunagi::dataplane::IpPlugin;
|
||||
use tsunagi::dataplane::wireguard::{MemoryTunFactory, WireguardConfig, WireguardPlugin};
|
||||
use tsunagi::identity::{NetworkName, NetworkSecret};
|
||||
use tsunagi::{Agent, Result};
|
||||
|
||||
#[tokio::main]
|
||||
async fn main() -> Result<()> {
|
||||
let paths = StoragePaths::user_default()?;
|
||||
|
||||
// MemoryTunFactory needs no privileges; swap in ManagedTunFactory for a
|
||||
// real interface.
|
||||
let plugin = WireguardPlugin::open(
|
||||
WireguardConfig::new(paths.state_dir.join("wireguard")),
|
||||
Arc::new(MemoryTunFactory::new()),
|
||||
)
|
||||
.await
|
||||
.expect("wireguard plugin");
|
||||
|
||||
let agent = Agent::spawn(
|
||||
AgentConfig::new(paths)
|
||||
.with_transport(TransportPolicy::N0Defaults)
|
||||
.with_plugin(plugin.clone() as Arc<dyn IpPlugin>),
|
||||
)
|
||||
.await?;
|
||||
|
||||
let network = agent
|
||||
.join_network(&NetworkName::new("lab")?, &NetworkSecret::generate())
|
||||
.await?;
|
||||
|
||||
if let Some(view) = plugin.overview(network) {
|
||||
println!("{} on {}", view.interface, view.overlay_address);
|
||||
}
|
||||
agent.shutdown().await;
|
||||
Ok(())
|
||||
}
|
||||
```
|
||||
|
||||
## Limits and future work
|
||||
|
||||
* **Full mesh only.** Every member runs a tunnel to every other member.
|
||||
Routing through an intermediate participant is not implemented.
|
||||
* **Nothing frees an address yet.** The release record exists and merges, but
|
||||
no command emits one.
|
||||
* **A snapshot grows with the number of members ever seen**, and is capped per
|
||||
message rather than compacted.
|
||||
* **No routes, DNS or firewall rules.** The plugin creates its interface and
|
||||
nothing else. Anything beyond the overlay `/64` is the operator's business.
|
||||
* **Membership is session-scoped.** A peer leaves the overlay when its control
|
||||
session ends; surviving a long absence is the same future work.
|
||||
* **Userspace costs CPU.** Kernel WireGuard is faster. A kernel backend could
|
||||
return behind the same boundary, but it would give up transport-provided NAT
|
||||
traversal unless paired with a local proxy.
|
||||
* **The interface is managed, not prepared.** On Linux the agent creates the
|
||||
TUN interface and configures it over netlink, in process. See
|
||||
*Provisioning* below.
|
||||
* **No provisioner exists for macOS or Windows yet.** There the agent attaches
|
||||
to an interface prepared by hand and says so.
|
||||
* **The system interface path is not exercised by the default suite**, because
|
||||
it needs privileges. Everything else about the data plane is.
|
||||
## Limits
|
||||
|
||||
The default suite exercises real cryptography, sockets, handshakes, storage and
|
||||
IP routing using memory TUNs. It does not establish public NAT traversal, DPI
|
||||
effectiveness, actual host TUN performance, or native GUI behavior on every OS.
|
||||
Userspace encryption costs CPU. Reassembly and packet queues are bounded and
|
||||
may drop packets; host TCP handles retransmission within its own timeouts.
|
||||
|
||||
## Provisioning the interface
|
||||
|
||||
@@ -313,7 +283,7 @@ should carry and no others — observes what is actually on the host, and
|
||||
applies the difference. Running it twice changes nothing the second time.
|
||||
|
||||
The decision of *what* to change is
|
||||
`dataplane::wireguard::provision::plan_changes`: pure, platform-independent
|
||||
`overlay::provision::plan_changes`: pure, platform-independent
|
||||
and unit-tested on every platform. Only the execution is behind
|
||||
`InterfaceProvisioner`, which has three implementations — netlink on Linux, a
|
||||
`MockProvisioner` over a pretend host for the tests, and one that refuses with
|
||||
@@ -342,13 +312,12 @@ addresses with them.
|
||||
|
||||
The two refusals are the interesting part:
|
||||
|
||||
* **A link that is not a TUN is never touched.** The name is derived from the
|
||||
network id, so colliding with a real device is unlikely rather than
|
||||
impossible, and deleting somebody's bridge is not a recoverable mistake.
|
||||
* **A link that is not a TUN is never touched.** The interface name is supplied locally, so colliding with a real device is
|
||||
possible, and deleting somebody's bridge is not a recoverable mistake.
|
||||
* **A TUN another process holds open is never deleted.** Carrier is the
|
||||
signal: a TUN has it exactly while something is attached. An attached one is
|
||||
a working overlay, almost certainly a second agent on this host, and it is
|
||||
told to use a different `--wg-prefix` instead.
|
||||
told to use a different `--interface` instead.
|
||||
|
||||
### Privilege
|
||||
|
||||
@@ -374,6 +343,6 @@ construction.
|
||||
### What it cannot be told to do
|
||||
|
||||
Nothing here takes a name, an address or a command from the network. The
|
||||
interface name is derived from the network id, the addresses come from the
|
||||
local plugin and the signed allocation records, and no external program is
|
||||
interface name comes from local configuration, the addresses come from the
|
||||
signed allocation records, and no external program is
|
||||
executed at any point — there is no `ip`, no shell and no `PATH` involved.
|
||||
Reference in new issue
Block a user