Aleksandr Bogomiakov
2166ba4800
AWG-only policy also selected a competing end-to-end plugin, so compatible external links could still have no compatible encrypted data channel. Carrier changes replaced tunnel state and status selected counters from different engines. Use one internal WireGuard plugin and a separate DataCarrier contract for external capabilities/lifecycle. Keep the logical link, keys, handshake, interface and counter epoch across policy, fallback and direct/relay changes. Control/discovery transport selection and membership remain independent. Share the bounded per-neighbour UDP WireGuard engine between ordinary WG and Amnezia masking. Encrypt the relay envelope inside the outer hop channel with fresh pair keys/PSK. Remove duplicate WG key/device code and the TCP/TLS inner cipher. TLS carries opaque internal WireGuard frames. Refuse missing native transports instead of letting default iroh impersonate their protocol names. Probe the actual authenticated carrier, independently of the shared IP tunnel; failed local probe submissions count as loss too. Apply the vendored iroh interface exclusion to actual direct candidates, not just public discovery records. Exclude overlay subnets from introductions and propagate the actual owned interface name, including kernel-assigned utun names, to every carrier. Production logs showed paths through the overlay itself. This removes a recursive transport path; it does not establish the historical cause of the reported 100 GB absolute counter. Exit-node traffic is also real peer IP traffic even when local applications are idle. Hide inner cipher choices/handshake clutter in CLI and tray. Report external routes, readiness and actionable policy/version failures. Count original IP packets only when their ciphertext is accepted by the logical transport, including queued packets' exact unpadded lengths. Exclude physical probes and keepalives. Export counter epochs; sum per-peer rate deltas and discard stale histories so joins/restarts cannot turn old totals into fresh traffic. Status also prints exact IP byte totals for comparison with the tray. Coordinated upgrade: data ALPN 6, internal wireguard-e2e capability v4, wg/awg v2, QUIC carrier v1, TLS v2, local IPC v23. Control ALPN 8, device/network identity derivation, signed records and saved network choices stay unchanged. Keep the historical wg-quic key store; transactionally import missing keys from wg then awg without replacing canonical keys. Forget removes legacy copies too, on the blocking pool, to prevent resurrection after restart. Preserve historic cipher options as shared settings, rejecting contradictory values. Validation on Linux in the Nix development shell: - cargo fmt --all -- --check - cargo clippy --locked --workspace --all-targets -- -D warnings - cargo test --locked --workspace --all-targets (562 passed, 3 ignored) The inherited service socket/secret/group environment overrides were removed for isolated CLI process tests. A new carrier test initially raced the first peer announcement; its deadline polling now waits for a peer row explicitly. Real loopback endpoints, membership, SQLite, WG/AWG/TLS encryption and memory TUNs cover mixed-carrier relaying and one smoltcp TCP connection surviving relay/direct/policy changes and a dropped transmission. Counter tests check exact plaintext lengths, refused sends, epochs, joins and stale histories. Public DHT, privileged kernel and manual benchmark tests remain ignored; macOS/Windows native TUNs, desktop GUI and WAN/DPI behavior were not run.
147 lines
8.4 KiB
Markdown
147 lines
8.4 KiB
Markdown
# Pairwise Amnezia transport
|
||
|
||
The CLI's default carrier preference is `wg,awg,wg-quic,tcp-tls`. To select
|
||
AWG explicitly:
|
||
|
||
```sh
|
||
tsng up --protocol awg
|
||
tsng up --protocol wg,awg,wg-quic,tcp-tls
|
||
```
|
||
|
||
The startup list is the initial per-network preference. A saved network choice
|
||
overrides it and can enable any built-in protocol at runtime:
|
||
|
||
```sh
|
||
tsng protocols home awg,wg-quic
|
||
tsng protocols home awg
|
||
```
|
||
|
||
Changes are local to this device, persist across restarts, and retire existing
|
||
physical data links for that network before renegotiation. The single internal
|
||
WireGuard channel, its keys, session and counters survive.
|
||
Authenticated control sessions, membership, addresses and other networks stay
|
||
live. Removing every protocol (`none` in the CLI, all checkboxes off in the tray)
|
||
leaves the network control-only. Re-enabling a carrier resumes the existing
|
||
internal channel, subject to ordinary WireGuard and application timeouts. The
|
||
choice applies to both incoming and outgoing local data links; a relay may use
|
||
a different transport on its next hop. Control and discovery are independent.
|
||
|
||
`awg` is Tsunagi AWG wire version 2. Its packet masking engine is adapted from
|
||
[RunaRoo/AmneziaWG-RS](https://github.com/RunaRoo/AmneziaWG-RS), with the pinned
|
||
source and license recorded in
|
||
[UPSTREAM.md](../crates/tsunagi-awg/UPSTREAM.md). It uses the existing boringtun
|
||
Noise/WireGuard implementation instead of upstream's Node, global executors
|
||
and OS interface machinery. Compatibility with ordinary Amnezia servers is
|
||
not implemented: this carrier connects authenticated Tsunagi members.
|
||
|
||
## Negotiation and lifetime
|
||
|
||
There is one independent UDP socket and cancellable packet-engine task per
|
||
`(network, peer)` pair. The smaller endpoint id initiates, following the same
|
||
rule as other data carriers. Both sides must announce enabled AWG wire version
|
||
2. An authenticated iroh control session is required throughout; AWG supplies
|
||
no alternate discovery or membership mechanism.
|
||
|
||
Every `open` generates new masking parameters, a header-protection key, an
|
||
outer WireGuard key pair and a PSK. The initiator sends a private `Offer` with
|
||
a random attempt id, its public key, candidate socket addresses and the shared
|
||
profile. The responder validates it, creates its own fresh key pair and socket,
|
||
and returns an `Answer` naming that attempt. Both send masked handshakes and
|
||
encrypted delivery probes to the candidates. Only authenticated WireGuard
|
||
**data**, not a STUN response or an incoming UDP address alone, verifies a path.
|
||
|
||
Negotiation payloads are bounded at 4096 bytes (a caller can lower the limit),
|
||
redacted in Debug, excluded from public message events and erased on drop.
|
||
They are never capabilities, signed records, cache entries or exported status.
|
||
At most 128 pairs are admitted per transport, with bounded command/data queues,
|
||
at most 32 candidates per attempt and finite establishment deadlines.
|
||
Stale answers and duplicate offers are rejected; a new attempt replaces and
|
||
closes an older pair engine. Cancellation, peer departure, network deactivation
|
||
and shutdown retire sockets without waiting for a remote reader.
|
||
|
||
A process restart, a failed setup, or a complete fallback cycle returning to AWG
|
||
starts fresh negotiation. Normal WireGuard rekeying within a live attempt keeps
|
||
that attempt's masking profile. A failed pair does not reconfigure other pairs
|
||
or other networks. A full mesh of 15 members uses 14 engines per member and 105
|
||
independent pairs in total; it does not create additional system interfaces.
|
||
|
||
## Masking
|
||
|
||
Each profile randomizes disjoint H1–H4 header ranges, a 12–48-byte S1–S4 prefix,
|
||
a ChaCha20 header-protection key, handshake trailers, junk packet count/lengths,
|
||
and extra content padding. Prefix lengths are equal within one profile so
|
||
header detection has one offset. A fresh random nonce is in every packet's
|
||
prefix; handshake headers and transport headers/counters are protected.
|
||
The standard WireGuard header is restored before boringtun authenticates a
|
||
message. This preserves boringtun's standard MAC calculations, which is another
|
||
reason this is a Tsunagi-specific AWG wire version.
|
||
|
||
Before a handshake, an I1 signature packet is generated from either a complete
|
||
DNS query or SIP OPTIONS template, with fresh identifiers and consistent length
|
||
fields, followed by random junk packets. I2–I5 are unused in this version.
|
||
The templates are structural generators, not replayed captures. QUIC imitation
|
||
is not selected. Masking can change DPI classification; it is not a promise of
|
||
indistinguishability or that any particular firewall will admit it.
|
||
|
||
## Packets, NAT and fallback
|
||
|
||
The pair engine carries opaque relay envelopes inside authenticated outer WG
|
||
packets. It fragments those envelopes into bounded chunks and expires incomplete
|
||
assemblies; a logical datagram is bounded at 64 KiB and an outer UDP packet is
|
||
at most 1200 bytes. The encapsulated IPv4 headers are internal framing only: no interface or OS address is created for a
|
||
pair. Overlay addresses, routing, ownership checks and end-to-end encryption
|
||
remain independent. A relay decrypts its own adjacent AWG carrier to forward
|
||
opaque end-to-end ciphertext, never the destination's IP packet.
|
||
|
||
Local IPv4 candidates and simultaneous encrypted probes provide direct UDP
|
||
connectivity. In `direct`/`relay` reach mode the CLI also uses
|
||
`stun.l.google.com:19302` to discover a server-reflexive address on **this pair's
|
||
socket**, with a 1.2-second bound. A STUN result is only a candidate. Local reach
|
||
mode makes no STUN request. Configure or disable it with
|
||
`-o awg:stun=HOST:PORT` or `-o awg:stun=off`. Failed discovery still permits local
|
||
candidates. IPv6-only paths, symmetric NAT and networks blocking arbitrary UDP
|
||
may fail AWG and fall back; there is no AWG-over-QUIC substitution or AWG TURN
|
||
relay hidden behind its name.
|
||
|
||
Data delivery checks go through the specific authenticated physical carrier,
|
||
once per second, independently of control-plane RTT or the shared internal
|
||
tunnel. A successful relay path cannot mask a broken direct carrier. Requests and replies bind a
|
||
fresh private marker/key to the sender's endpoint id. Unanswered requests expire
|
||
after three seconds. After a 20-second warmup, at least 15 completed observations
|
||
and at least 60% loss in the last 20 observations retire a carrier. Probes are
|
||
consumed at the carrier boundary and never enter the TUN or user IP counters.
|
||
AWG additionally checks encrypted outer delivery throughout the pair lifetime.
|
||
|
||
Setup failure or persistent runtime loss advances to the next mutually supported
|
||
carrier in the initiator's order. Earlier expired cooldowns do not jump ahead of
|
||
untried carriers. Retry cooldowns remain bounded; reaching AWG again negotiates
|
||
new parameters. Control sessions and the logical end-to-end tunnel survive a
|
||
carrier change. Status and the tray show the actual external transport or relay
|
||
route and data readiness. There is no user-selectable internal cipher.
|
||
|
||
The internal capability is `wireguard-e2e` v4; external `wg` and `awg` use v2,
|
||
`wg-quic` v1 and `tcp-tls` v2. All members must upgrade together. Control ALPN
|
||
`tsunagi/ctrl/8`, discovery, membership and signed state are unchanged. The data
|
||
ALPN is `tsunagi/data/6`; the relay envelope format is unchanged. Local IPC is v23,
|
||
so agent, CLI and tray must match. Historical `wg-quic/wireguard.sqlite` is the
|
||
canonical internal key store. Missing keys are imported transactionally from
|
||
`wg` then `awg`, without replacing an existing canonical key or modifying a
|
||
source database. Corrupt key stores fail rather than silently regenerating keys.
|
||
Device identities, network ids, addresses and saved transport choices survive.
|
||
|
||
Ordinary `wg` uses the same independent pair engine with standard WireGuard
|
||
headers, no masking/junk, fresh hop keys and PSK, and a preferred local UDP port
|
||
(default 51820; additional pairs use dynamic ports). The whole relay envelope is
|
||
inside the hop encryption. It is separate from the internal WireGuard channel.
|
||
Neither UDP mode is a client for an externally configured WireGuard server.
|
||
|
||
## Verification
|
||
|
||
The default rootless, offline tests use real iroh endpoints, membership proofs,
|
||
SQLite, UDP sockets, masked WireGuard handshakes, end-to-end encryption and
|
||
memory TUNs. They cover independent three-member pairs, full-size bidirectional
|
||
packets, restart freshness/socket release, 80% runtime datagram loss, fallback
|
||
to real iroh and fresh negotiation on retry, plus malformed profiles/messages,
|
||
role checks, probe forgery and STUN response parsing. These tests do not establish
|
||
DPI effectiveness, WAN throughput, public NAT traversal or per-pair memory cost.
|