Compare commits
7
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Aleksandr Bogomiakov
|
e701986a39 | ||
|
Aleksandr Bogomiakov
|
ed28e86748 | ||
|
Aleksandr Bogomiakov
|
781e359775 | ||
|
Aleksandr Bogomiakov
|
d5e02ec256 | ||
 Aleksandr BogomiakovandClaude
|
97377e3703 | ||
|
Aleksandr BogomiakovandClaude
|
c184ef0914 | ||
|
Aleksandr BogomiakovandClaude
|
e6ec62bcba |
No files matched your search
@@ -1 +1,2 @@
|
||||
/target
|
||||
/.claude
|
||||
@@ -8,7 +8,7 @@ before touching the code. Do not restate the full design here; follow the links.
|
||||
A proof-of-concept agent library for small private mesh networks. A user
|
||||
supplies a network name and one shared secret; agents derive the same network
|
||||
space independently, find each other, prove membership and exchange control
|
||||
messages. Scope and non-scope are in [README.md](README.md).
|
||||
messages. What it does and does not do is in [docs/architecture.md](docs/architecture.md) and [docs/threat-model.md](docs/threat-model.md).
|
||||
|
||||
There is no central server, no owner, no registration and no majority vote.
|
||||
Design accordingly: a majority is not a root of trust.
|
||||
@@ -59,8 +59,10 @@ Keep these separate. Crossing them is the main thing to review for.
|
||||
other peers goes link in, link out: it is never written to the middle's
|
||||
interface or decrypted there. Multihop routing has a bounded hop limit;
|
||||
equal-cost next hops are chosen per flow. Reachability is first-hand and
|
||||
volatile: each authenticated member advertises its own protocol-specific
|
||||
links. Build routing tables on topology changes, never per packet. Transit
|
||||
volatile: each authenticated member advertises its own direct links, of
|
||||
whatever kind. A link is a way to reach a neighbour and a protocol is an
|
||||
end-to-end tunnel to a peer; the two meet only in the envelope, so a
|
||||
tunnel never depends on which transports carry it. Build routing tables on topology changes, never per packet. Transit
|
||||
must not acquire a routing mutex or wait for a protocol/TUN reader.
|
||||
- **Broadcast fanout belongs at local IP ingress.** Participation is local to
|
||||
each network, enabled by default, persisted and advertised to authenticated
|
||||
|
||||
Generated
+21
@@ -6246,6 +6246,7 @@ dependencies = [
|
||||
"tokio",
|
||||
"tracing",
|
||||
"tracing-subscriber",
|
||||
"tsunagi-awg",
|
||||
"tsunagi-core",
|
||||
"tsunagi-tcp-tls",
|
||||
"tsunagi-wg-quic",
|
||||
@@ -6253,6 +6254,26 @@ dependencies = [
|
||||
"windows-service",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "tsunagi-awg"
|
||||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"boringtun",
|
||||
"bytes",
|
||||
"chacha20 0.9.1",
|
||||
"iroh",
|
||||
"postcard",
|
||||
"rand 0.10.3",
|
||||
"serde",
|
||||
"tempfile",
|
||||
"tokio",
|
||||
"tracing",
|
||||
"tsunagi-core",
|
||||
"tsunagi-wg-quic",
|
||||
"tsunagi-wireguard",
|
||||
"zeroize",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "tsunagi-core"
|
||||
version = "0.1.0"
|
||||
|
||||
@@ -1,766 +1,199 @@
|
||||
# tsunagi
|
||||
|
||||
A proof-of-concept Rust library for **small private mesh networks** — a handful
|
||||
of friends, home machines, a few servers. Units to dozens of participants, not
|
||||
thousands.
|
||||
|
||||
The end user configures exactly two things:
|
||||
|
||||
```text
|
||||
network_name
|
||||
secret # one shared secret; "password" and "secret" mean the same value
|
||||
```
|
||||
|
||||
From those, every agent independently derives the same network space. There is
|
||||
no central server, no network owner with special powers, no registration and no
|
||||
majority vote. Anyone who knows the parameters can join; nobody has to trust
|
||||
anybody else.
|
||||
|
||||
## What this proof of concept actually does
|
||||
|
||||
A working library with **real iroh connections** and integration tests:
|
||||
|
||||
- persistent device identity stored in SQLite, stable across restarts;
|
||||
- several independent networks at once in one agent;
|
||||
- deterministic network identity derived from name + secret;
|
||||
- candidates supplied by a replaceable discovery component;
|
||||
- automatic first contact through the public Mainline DHT (BEP44);
|
||||
- real iroh connections plus an explicit mutual proof of network membership;
|
||||
- a small versioned control protocol: handshake, hostname/capability
|
||||
announcement, ping/pong;
|
||||
- automatic reconnect with bounded exponential backoff and jitter;
|
||||
- status snapshots, an event stream and honest diagnostics;
|
||||
- configuration restored after a restart;
|
||||
- correct behaviour when the disposable cache is missing or corrupt;
|
||||
- a **WireGuard data plane**, in userspace: its own key per network, an IPv6
|
||||
overlay with deterministically derived addresses and optional IPv4, real
|
||||
tunnels carried over iroh, and address ownership enforced rather than
|
||||
believed;
|
||||
- a **command line agent**, `tsng`, with a local control socket.
|
||||
|
||||
### What it deliberately does **not** do
|
||||
|
||||
Not implemented, and not pretended to be: DNS, routing through
|
||||
intermediate participants, a full CRDT, dynamically loaded plugins, a system
|
||||
service, a complete CLI, or a local control socket. Snapshot synchronisation
|
||||
and signed revocations are designed for but not implemented — see
|
||||
[docs/sync-model.md](docs/sync-model.md). The WireGuard plugin's own limits,
|
||||
including that its system backend is Linux-only, are in
|
||||
[docs/wireguard.md](docs/wireguard.md#limits-and-future-work).
|
||||
|
||||
**Control and data are separated logically, not physically.** Both ride on
|
||||
iroh, on different ALPNs and different connections, so the data plane inherits
|
||||
iroh's hole punching and relay fallback instead of reimplementing them — while
|
||||
the control protocol still knows nothing about packets and can keep a different
|
||||
transport underneath it later. Filtering user traffic remains the operating
|
||||
system's and the user's responsibility, not this library's.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Rust 1.91 or newer (iroh 1.2 requires it), edition 2024. Pinned dependencies
|
||||
in `Cargo.lock`.
|
||||
- No internet, no public DHT, no public relay, no administrator rights and no changes
|
||||
to OS network settings are needed to build or test.
|
||||
- WireGuard runs in userspace (boringtun): **no kernel module and no `wg`
|
||||
tool**. Only creating a real network interface needs `CAP_NET_ADMIN`, and
|
||||
`--no-tun` skips even that.
|
||||
|
||||
## Trying it on two machines
|
||||
|
||||
On the first machine:
|
||||
|
||||
```bash
|
||||
cargo build --release
|
||||
./target/release/tsng status # this device, the agent, and this host
|
||||
./target/release/tsng network # the networks this device belongs to
|
||||
```
|
||||
|
||||
Two commands do the work, and they are separate on purpose: `up` runs the
|
||||
agent — one per device, for as long as you want it — and `join` decides
|
||||
what it belongs to, at any time, while it runs.
|
||||
|
||||
```bash
|
||||
# Terminal one: the agent. It prints its endpoint id and then serves.
|
||||
./target/release/tsng up
|
||||
|
||||
# Terminal two: make a network. With no --secret it invents one and prints
|
||||
# it, along with the line to send the other machine.
|
||||
./target/release/tsng join --network lab
|
||||
```
|
||||
|
||||
On the second machine, start its agent and join the same network with the
|
||||
secret that was printed:
|
||||
|
||||
```bash
|
||||
./target/release/tsng up
|
||||
./target/release/tsng join --network lab --secret tsn1...
|
||||
```
|
||||
|
||||
**Name and secret are enough for the first meeting.** Mainline DHT discovery
|
||||
is enabled by default. Each active network publishes this device's endpoint
|
||||
and searches until the first authenticated connection. While connected it
|
||||
only republishes; after 60 seconds without any authenticated connection it
|
||||
searches again. Publication is approximately every five minutes, and records
|
||||
older than 15 minutes are ignored. DHT storage nodes may retain them longer.
|
||||
|
||||
Use `--no-dht` to disable DHT, or `--peer <endpoint-id>` for optional manual
|
||||
bootstrap. `--reach local` disables public DHT regardless of the DHT flag.
|
||||
Endpoint records are public, signed, and contain no network secret. Discovery
|
||||
is not authentication: every connection must still prove network membership.
|
||||
Details and failure behavior are in [docs/mainline-dht.md](docs/mainline-dht.md).
|
||||
|
||||
**One introduction is enough for the whole network.** Members tell each
|
||||
other about the members they know, and every author of a signed record is
|
||||
somebody to try, so a device pointed at one member ends up talking to all
|
||||
of them rather than to the one that happened to be on its command line.
|
||||
What travels is a candidate — an address somebody has seen — and it is
|
||||
authenticated by the handshake like any other; being introduced grants
|
||||
nothing. It is deliberately the same shape a lookup in a distributed hash
|
||||
table would return, so that is a source to add beside this one rather than
|
||||
a redesign.
|
||||
|
||||
An agent with nobody to contact says so in `status` rather than sitting
|
||||
there looking patient.
|
||||
|
||||
Within a few seconds both print something like:
|
||||
|
||||
```text
|
||||
+ peer b47c958462 connected over direct rtt=Some(4.5ms)
|
||||
+ data link to b47c958462 for wg-quic: direct via 192.0.2.7:41234, datagram 1382
|
||||
|
||||
--- status ---
|
||||
control: 1 peer(s), 0 dial failure(s), 0 handshake failure(s)
|
||||
wg-quic: tsun0 on 10.13.37.69/24 mtu 1280, 1/1 tunnel(s) established
|
||||
4jO4kx9Z 10.13.37.237 handshake 3s ago tx=0 rx=0 dropped=0 path=direct via 192.0.2.7:41234
|
||||
```
|
||||
|
||||
`1/1 tunnel(s) established` means a real WireGuard handshake completed.
|
||||
|
||||
## Checking that it works
|
||||
|
||||
From another shell on either machine:
|
||||
|
||||
```bash
|
||||
tsng status
|
||||
```
|
||||
|
||||
```text
|
||||
endpoint 7d76ccbbc21bf30767e14422c0494740a2cecc02aa9f82d5b8d57bdae350e7fc
|
||||
hostname tsunagi-7d76ccbbc2
|
||||
bound 0.0.0.0:41641
|
||||
|
||||
network lab (z2o4qwrvnj3zb6st2aoqg4abf342j662q2ujttqrsmz22argk2ba) active
|
||||
peer b345d5271b tsunagi-b345d5271b Direct rtt 24ms
|
||||
overlay tsunz2o4qwrvnj3 fd09:…:c1c6/64 and 100.110.49.177 mtu 1280 1/1 tunnel(s) up
|
||||
SDsEb/WF fd09:…:c4b / 100.65.243.53 handshake 4s ago tx 0 rx 0 Direct via Ip(…)
|
||||
```
|
||||
|
||||
`1/1 tunnel(s) up` and a recent handshake mean the tunnel is live. Then send
|
||||
real traffic to the peer's overlay address:
|
||||
|
||||
```bash
|
||||
ping6 fd09:…:c4b # or
|
||||
ping 100.65.243.53
|
||||
```
|
||||
|
||||
`tx` and `rx` in the status should start moving.
|
||||
|
||||
**Addresses are allocated and then remembered.** The default range is
|
||||
`10.13.37.0/24`; the first member to join settles it and later members adopt
|
||||
what they find, so `--ipv4-range` only matters for whoever starts the network:
|
||||
|
||||
```bash
|
||||
tsng up --ipv4-range 10.44.0.0/16
|
||||
tsng up --ipv4-range none # no data plane at all
|
||||
```
|
||||
|
||||
One agent has one interface, so two of its networks cannot both use that
|
||||
range. The second takes the range **derived from its own network id**: not
|
||||
picked locally — every member derives the same one from something they all
|
||||
already have — so it is an agreement rather than a guess, and a device in
|
||||
several networks gets an address in each.
|
||||
|
||||
An address is claimed with a record signed by that member's persistent device
|
||||
key, stored, and merged between every replica. A member that disappears for a
|
||||
month comes back to the same address, because the claim outlived the session.
|
||||
No vote is involved — see
|
||||
[docs/wireguard.md](docs/wireguard.md#ipv4-allocated-signed-and-kept) and
|
||||
[docs/sync-model.md](docs/sync-model.md).
|
||||
|
||||
Because the address is allocated at run time rather than derived, it is not
|
||||
known until the agent has started and agreed with its peers. The agent then
|
||||
assigns it to the interface itself.
|
||||
|
||||
## Levels
|
||||
|
||||
The command line is split the way the design is. A bare `tsng up` needs
|
||||
only a network name and a secret; everything else sits under the level it
|
||||
belongs to, which `--help` shows as two sections:
|
||||
|
||||
* **System** — what the agent itself does: how it reaches peers (`--reach`),
|
||||
the one overlay interface it owns (`--interface`, `--mtu`, `--no-tun`), the
|
||||
address range (`--ipv4-range`), and the local resolver (`--no-dns` to disable).
|
||||
* **Transport** — which protocols carry packets (`--protocol`, a list) and
|
||||
their own settings (`-o key=value`, or `-o protocol:key=value`).
|
||||
|
||||
```bash
|
||||
tsng protocols # what this build can carry packets with
|
||||
```
|
||||
|
||||
```text
|
||||
wg-quic (wire version 4)
|
||||
what WireGuard's cryptography carried in iroh's QUIC datagrams
|
||||
-o keepalive=SECONDS keeps a tunnel and its link warm through a NAT
|
||||
-o mtu=BYTES largest packet a tunnel will carry, at least 576
|
||||
```
|
||||
|
||||
Each protocol declares its own settings, so the agent can list them without
|
||||
knowing anything about the protocol, and a setting no selected protocol takes
|
||||
is refused rather than ignored. `--protocol none` runs the control plane by
|
||||
itself.
|
||||
|
||||
A pair of peers uses a protocol they both have **at the same wire version**.
|
||||
That is not the software version: two peers on different builds carry traffic
|
||||
for each other for as long as the bytes between them have not changed. A peer
|
||||
with nothing in common keeps its control plane — messages and signed state
|
||||
still flow — and simply has no data plane, which `tsng status` shows as a
|
||||
session with no agreed protocol.
|
||||
|
||||
## Names
|
||||
|
||||
DNS is enabled by default and serves a local zone for **every network this device is in**,
|
||||
each named after the network, so members can be reached by name instead of
|
||||
by address:
|
||||
|
||||
```bash
|
||||
tsng up # DNS is already enabled
|
||||
dig @127.0.0.1 -p 5354 music.lab
|
||||
```
|
||||
|
||||
An explicit choice is remembered across restarts. `--no-dns` or `tsng dns off`
|
||||
disables it; `--dns` or `tsng dns on` enables it again. Existing saved opt-outs
|
||||
remain respected. The `dns on` and `dns off` commands also take effect immediately
|
||||
on a running agent. `--dns` and `--no-dns` cannot be combined.
|
||||
|
||||
The zone of a network is its name — `--dns-zone` is gone, because with
|
||||
several networks there is no single zone to name. A network name may contain
|
||||
dots, so `--network lab.internal` is how you get `music.lab.internal`.
|
||||
|
||||
Names come from signed state, which is the point: **a member that is
|
||||
switched off still resolves**, because its claim outlived the session.
|
||||
|
||||
The answers are IPv4 addresses, because that is what the overlay is. The
|
||||
*questions* are taken on `127.0.0.1` and `[::1]`, over UDP and TCP, so a
|
||||
resolver reaches it over whichever family it uses; both are published to the
|
||||
system resolver together. Loopback and nothing else: the zones are a view
|
||||
for the host running the agent, and binding an overlay address would put
|
||||
them in front of the whole mesh — an agent in two networks would then answer
|
||||
one network's questions about the other's names.
|
||||
|
||||
The zone is the network name, and it is yours to choose. A name that
|
||||
shadows a real public domain is reported and then used: a network called
|
||||
`ru` warns that every public `.ru` name becomes unreachable from this host,
|
||||
and then does it. Anything that collides with nothing — which is most names
|
||||
— is said nothing about, because a warning that fires on every private name
|
||||
anybody picks is how people learn to ignore warnings.
|
||||
|
||||
On Linux the agent tells systemd-resolved to send questions for that suffix
|
||||
here, over D-Bus, scoped to the overlay interface and as a *routing* domain
|
||||
so it never becomes the resolver for anything else. resolved drops the whole
|
||||
setting when the interface goes, and the interface goes with the agent.
|
||||
|
||||
Only an interface the agent created, though. Under `--no-tun` there is no
|
||||
host interface at all, and the agent says so and serves the zone on
|
||||
loopback rather than configuring whatever else on the host happens to share
|
||||
the name.
|
||||
|
||||
That last step needs permission that `CAP_NET_ADMIN` does not give:
|
||||
systemd-resolved asks polkit, and polkit decides by **user**, not by
|
||||
capability, so there is no way for the agent to arrange it from inside. On
|
||||
a desktop the refusal reads `Interactive authentication required`.
|
||||
|
||||
Running as a system service is enough. Otherwise the agent prints the rule
|
||||
that grants it — the four actions it calls and nothing else — ready to
|
||||
paste:
|
||||
|
||||
```bash
|
||||
sudo tee /etc/polkit-1/rules.d/50-tsunagi-resolved.rules > /dev/null <<'RULE'
|
||||
polkit.addRule(function(action, subject) {
|
||||
var allowed = [
|
||||
"org.freedesktop.resolve1.set-dns-servers",
|
||||
"org.freedesktop.resolve1.set-domains",
|
||||
"org.freedesktop.resolve1.set-default-route",
|
||||
"org.freedesktop.resolve1.revert"
|
||||
];
|
||||
if (allowed.indexOf(action.id) >= 0 && subject.user == "YOUR-USER") {
|
||||
return polkit.Result.YES;
|
||||
}
|
||||
});
|
||||
RULE
|
||||
```
|
||||
|
||||
**Without it the server still runs** — `tsng status` prints where it is
|
||||
listening and the exact `dig` line — so the automatic part is missing, not
|
||||
the feature. The refusal is said once rather than on every pass, and retried
|
||||
slowly, because nothing but a person will change it.
|
||||
|
||||
The server is authoritative for its zone and nothing else. No recursion, no
|
||||
forwarding, no cache: pointing a resolver at it can never make it a route to
|
||||
the outside.
|
||||
|
||||
## Privileges
|
||||
|
||||
On Linux the agent **manages its own overlay interface**. It creates the TUN
|
||||
interface, sets the MTU, brings it up and assigns both overlay addresses, all
|
||||
over netlink in process — no `ip` invocation, no shell, nothing that a
|
||||
remote peer could influence.
|
||||
|
||||
That needs `CAP_NET_ADMIN`, granted once:
|
||||
|
||||
```bash
|
||||
sudo setcap cap_net_admin+p /usr/local/bin/tsng
|
||||
```
|
||||
|
||||
`+p` rather than `+ep`: the capability is then *permitted* but not
|
||||
*effective*, and the agent raises it only around the handful of netlink calls
|
||||
that need it — a few milliseconds at startup, and again if its address
|
||||
allocation changes. Everything else, including every byte from the network,
|
||||
is handled with it lowered. `+ep` works too; the agent lowers it on the way
|
||||
in.
|
||||
|
||||
`tsng status` says which of these applies on the host it runs on, along
|
||||
with what the agent is doing. It grades each finding: **ok** for what works,
|
||||
**warn** for what the agent runs without and you can fix from the line it
|
||||
prints, **FAIL** for what it cannot work around. The words carry the grade as
|
||||
well as the colour, so the report reads the same piped to a file or on a
|
||||
terminal without colour, and it honours `NO_COLOR`.
|
||||
|
||||
Members are listed online first, then the ones that are away. A member that
|
||||
is away is reported plainly rather than flagged: in a mesh of laptops it is
|
||||
the ordinary condition, not a fault. The signed state is what makes that
|
||||
sayable — it remembers who belongs while they are gone, so the report can say
|
||||
"offline, 10.13.37.99 still reserved for it" instead of leaving a
|
||||
dial-failure counter to imply it. Counters are history and are never graded:
|
||||
a peer that left and came back should not leave the report looking broken.
|
||||
|
||||
The other two commands split along the line the system itself draws. `id` is
|
||||
this **device**: the key it signs with and the name it answers to. `network`
|
||||
is what it **belongs to**: which networks, their secrets, joining and
|
||||
leaving. A device outlives every network it is in, and a network outlives
|
||||
any device in it, so a command that mixed them had to be read twice.
|
||||
|
||||
Every item takes the same shape, so there is nothing to remember: name it to
|
||||
see it, name it with a value to change it.
|
||||
|
||||
```
|
||||
tsng id everything about this device
|
||||
tsng id hostname the name it answers to
|
||||
tsng id hostname mango change it
|
||||
tsng id key the key it signs with
|
||||
tsng id key rotate replace that key
|
||||
|
||||
tsng network the networks this device belongs to
|
||||
tsng network join -n lab -s tsn1… join one; adds it to a running agent
|
||||
tsng network join -n lab resume one this device has, or make it on the spot
|
||||
tsng network stop lab stop serving it, keeping everything
|
||||
tsng network start lab serve it again, from where it left off
|
||||
tsng network leave lab give up the address and name, then forget it
|
||||
tsng network secret the secret of each joined network
|
||||
tsng network secret lab just that one, for copying
|
||||
tsng network secret generate a fresh secret for a network that does not exist yet
|
||||
|
||||
tsng dns whether the local resolver is serving, and what
|
||||
tsng dns on start it, now and after every restart
|
||||
tsng dns off stop it, now and after every restart
|
||||
```
|
||||
|
||||
**A network without a secret makes one.** `tsng join --network lab`
|
||||
resolves a bare name in the obvious way: if this device is already in
|
||||
exactly one network called `lab`, that one — so the name alone resumes what
|
||||
you have; if it is in none, a fresh random secret, printed in full with the
|
||||
line to send the other machine:
|
||||
|
||||
```
|
||||
joined `lab` (k2on43wadbi5x267vp6z3ogkm7nbjedfdoxyauxhtxhwqrprylba)
|
||||
<p align="center">
|
||||
<img src="dist/linux/tsunagi.svg" width="96" alt="">
|
||||
</p>
|
||||
|
||||
<h1 align="center">tsunagi</h1>
|
||||
|
||||
<p align="center">
|
||||
<b>A mesh VPN with no server.</b><br>
|
||||
Pick a network name, share a secret, and your machines are on one private network.
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="https://github.com/house-of-vanity/tsunagi/releases/latest"><img alt="release" src="https://img.shields.io/github/v/release/house-of-vanity/tsunagi?include_prereleases"></a>
|
||||
<a href="https://github.com/house-of-vanity/tsunagi/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/house-of-vanity/tsunagi/actions/workflows/ci.yml/badge.svg"></a>
|
||||
<a href="LICENSE"><img alt="license" src="https://img.shields.io/badge/license-WTFPL-blue"></a>
|
||||
</p>
|
||||
|
||||
```console
|
||||
laptop$ tsng join -n home
|
||||
joined `home`
|
||||
secret tsn1u7c…
|
||||
|
||||
Run this on the other machine:
|
||||
server$ tsng join -n home -s tsn1u7c…
|
||||
server$ tsng id hostname server
|
||||
|
||||
tsng join --network lab --secret tsn1u7c…
|
||||
|
||||
Its agent has to be running. If it is not:
|
||||
|
||||
tsng up --peer 91e83a6e2b7a…
|
||||
laptop$ ssh server.home
|
||||
```
|
||||
|
||||
Two networks of one name is a thing that happens — a mistyped secret makes
|
||||
one — so joining says whether the network was already here, and warns when
|
||||
another configured network answers to the same name. A name is a label and
|
||||
the id is the identity.
|
||||
No account, no coordination server, no admin console, no keys to approve.
|
||||
Anyone who knows the **name** and the **secret** is in. Nobody else can find
|
||||
the network, let alone enter it.
|
||||
|
||||
That is the ad-hoc case: one person makes a network and sends the command
|
||||
round. The secret is printed *only* when the agent invented it — there is
|
||||
nowhere else to read it from — and never when it was supplied, because then
|
||||
it is already yours. Two networks of one name and no secret is the one case
|
||||
with no answer, and it says so instead of choosing.
|
||||
## Why
|
||||
|
||||
A secret is printed by `network secret` and nowhere else — not by `id`, not
|
||||
by `status`, not in a log, a `Debug` rendering or anything sent to a peer.
|
||||
Asking for it is deliberate, because these reports get pasted into chats.
|
||||
Tailscale, Headscale and ZeroTier are great, but each needs a control server:
|
||||
theirs, or one you have to run and keep alive. tsunagi doesn't. Machines find
|
||||
each other through the public BitTorrent DHT, connect directly (hole punching,
|
||||
with relay fallback), and keep the network state among themselves.
|
||||
|
||||
**`up` takes no network at all** if you would rather decide later: it
|
||||
brings up the agent and whatever it is already configured for, and waits.
|
||||
That is the shape of a daemon in one terminal and `tsng network join`
|
||||
in another.
|
||||
| | Tailscale | Headscale | ZeroTier | **tsunagi** |
|
||||
| ---------------------- | ------------------ | ---------------- | ------------------- | ------------------- |
|
||||
| Control server | hosted by vendor | you run it | hosted or self-run | **none** |
|
||||
| Account / sign-in | yes | yes | yes | **no** |
|
||||
| To join a network | sign in, get approved | pre-auth key | network ID, approval | **name + secret** |
|
||||
| Direct connections | yes | yes | yes | yes |
|
||||
| Exit nodes | yes | yes | via routes | yes |
|
||||
|
||||
`network join` is also the answer to a question `up` cannot: a state
|
||||
directory belongs to one live agent, so a second `tsng up` cannot add a
|
||||
network to the one already running. This adds it over the control socket and
|
||||
it starts at once. With no agent running it is written to the configuration
|
||||
and starts with the next `up`.
|
||||
## Features
|
||||
|
||||
The name is part of the signed state, so changing it revokes the previous
|
||||
one: there is one record per author, a new version replaces the whole claim,
|
||||
and no replica can keep the old name standing. Changing it while the agent
|
||||
runs goes through the agent, which republishes and tells its peers straight
|
||||
away.
|
||||
- **Two things to know:** a network name and a secret. A secret is generated for you when you make a network.
|
||||
- **Direct, encrypted links** between members (WireGuard cryptography), with relay fallback behind strict NATs.
|
||||
- **Names for machines:** `server.home` just works, even while `server` is switched off. A name belongs to whoever claimed it first; a later device with the same name keeps working by address and is told so.
|
||||
- **Exit nodes:** send all your internet traffic through any member that offers one.
|
||||
- **Multi-hop:** members that can't reach each other directly are routed through a member that can, over whatever kind of link each hop has. A member reached that way is online, and shown as relayed; traffic moves to a direct link by itself the moment one comes up.
|
||||
- **LAN games:** UDP broadcasts are relayed, so "LAN" lobbies show up across the mesh.
|
||||
- **Several networks at once** on one device.
|
||||
- **Tray app** for Windows, macOS and Linux; a command line for everything.
|
||||
|
||||
Replacing the signing key makes this device a different member, and it loses
|
||||
the address and name the old key held — nothing can sign on a retired key's
|
||||
behalf, and by design there is no authority that could overrule an author. So
|
||||
the outgoing key signs a release for every network on its way out, which
|
||||
frees them for whoever wants them next, and the whole thing commits at once.
|
||||
## Install
|
||||
|
||||
`status` and `id` both prefer a running agent, which is live and
|
||||
authoritative, and fall back to reading the state store when there is none.
|
||||
Reading takes no directory lock, so neither has to wait for the agent it is
|
||||
asking about — nor does either need one to be running.
|
||||
Grab a build from the [latest release](https://github.com/house-of-vanity/tsunagi/releases/latest).
|
||||
|
||||
### It cleans up after itself
|
||||
| Platform | How |
|
||||
| --------------------- | -------------------------------------------------------------------------------- |
|
||||
| **Windows** | Run `tsunagi-setup-<version>-x86_64.exe`. Installs the service, the tray and `tsng`. |
|
||||
| **macOS** | `brew tap house-of-vanity/tap && brew install --cask tsunagi-gui`<br>CLI only: `brew install tsunagi` |
|
||||
| **Debian / Ubuntu** | `sudo apt install ./tsunagi-gui_<version>_amd64.deb` (`tsunagi_…` for no tray) |
|
||||
| **Arch** | `sudo pacman -U tsunagi-gui-<version>-1-x86_64.pkg.tar.zst` |
|
||||
| **NixOS** | [Flake and module](#nixos) below |
|
||||
| **Any other Linux** | Unpack the tarball, then `sudo ./install.sh && sudo systemctl enable --now tsunagi` |
|
||||
|
||||
The interface is tied to an open file descriptor and is deliberately **not**
|
||||
made persistent, so the kernel removes it when the agent exits — on a clean
|
||||
shutdown, on a panic, on `SIGKILL`, on power loss alike. Keeping it is what
|
||||
would take an action; removing it is the default.
|
||||
Linux packages run the agent as an unprivileged service user. Add yourself to
|
||||
the `tsunagi` group (`sudo usermod -aG tsunagi $USER`, then log in again) to
|
||||
use `tsng` and the tray without `sudo`.
|
||||
|
||||
If something is left behind anyway — an interface made by an older version's
|
||||
manual recipe, or one from a run killed in the instant between creating it and
|
||||
recording it — the next start **replaces it**, along with any stale addresses
|
||||
it carried. Two things are never touched:
|
||||
## Quick start
|
||||
|
||||
* an interface that is not a TUN, because the name colliding with somebody's
|
||||
bridge is not a reason to destroy the bridge;
|
||||
* a TUN that another process is holding open, because that is a working
|
||||
overlay belonging to somebody else — most likely a second agent on this
|
||||
host, which should be given a different `--wg-prefix`.
|
||||
|
||||
Both of those refuse with an explanation rather than guessing.
|
||||
|
||||
Two settings the manual recipe used to need are gone with it.
|
||||
`keep_addr_on_down` existed only because an interface nobody held open lost
|
||||
carrier and had its IPv6 addresses flushed, and `nodad` only because duplicate
|
||||
address detection can never finish without carrier. An interface held open for
|
||||
its whole life has carrier for its whole life.
|
||||
|
||||
### The MTU is 1280
|
||||
|
||||
The default interface MTU stays at 1280 on direct paths, relays and paths
|
||||
inside another VPN. Large encrypted packets are split into smaller QUIC
|
||||
datagrams and reassembled before reaching WireGuard. SSH and full-size TCP
|
||||
segments therefore do not require a manual MTU override or MSS adjustment.
|
||||
The original IP packet, including its DF bit, is preserved. See
|
||||
[docs/wireguard.md](docs/wireguard.md#mtu) for limits and compatibility.
|
||||
|
||||
### Summary
|
||||
|
||||
| approach | agent runs as | notes |
|
||||
|---|---|---|
|
||||
| `setcap cap_net_admin+p` | ordinary user, one capability | recommended: nothing to prepare, nothing left behind. Lost on every rebuild or copy of the binary. |
|
||||
| systemd service | `User=`, `AmbientCapabilities=CAP_NET_ADMIN` | the same, for an installed service |
|
||||
| `sudo tsng up` | root | everything works, nothing is isolated |
|
||||
| macOS root LaunchDaemon | root | the native macOS deployment; see below |
|
||||
| `--no-tun` | ordinary user, no capabilities | tunnels run and handshake, traffic never reaches the OS |
|
||||
|
||||
On Windows, put `wintun.dll` beside the executable and start `tsng up`
|
||||
from PowerShell or Command Prompt opened with **Run as administrator**.
|
||||
Run commands controlling that agent (`join`, `status`, `dns`, `network`)
|
||||
as the same Windows user with the same elevation. If Windows denies access
|
||||
to the agent's control pipe, the command reports a permission error instead
|
||||
of claiming the agent is absent or trying to edit its locked state.
|
||||
Creating a TUN without sufficient privileges also explains how to restart
|
||||
the agent with the required permissions.
|
||||
|
||||
On macOS the agent creates a `utun` interface, configures it with `ifconfig`
|
||||
and `route`, and points the system resolver at the overlay by writing
|
||||
`/etc/resolver/<network>` files (which carry a port, so the DNS server needs no
|
||||
privileged socket). All of this needs **root**: macOS has no per-capability
|
||||
grant like Linux, and a `utun` plus `/etc/resolver` are root-only. The kernel
|
||||
assigns the interface its `utunN` name, so the name derived from the network id
|
||||
is only a request and the real one is adopted once the interface is up.
|
||||
|
||||
Run it as root — `sudo tsng up` for a quick start, or install it as a root
|
||||
**LaunchDaemon** for the native always-on deployment. As a daemon the agent runs
|
||||
unprivileged commands (`join`, `status`, `dns`, `network`) over its control
|
||||
socket like any other platform; the socket lives at `/var/run/tsunagi/agent.sock`
|
||||
and the stores at `/var/db/tsunagi`, both fixed system paths so they do not
|
||||
depend on whose shell started `sudo`. A ready-to-edit plist and install notes
|
||||
are in [dist/macos/](dist/macos/). The broadcast route is installed; the
|
||||
inbound-UDP firewall allowance (a `pf` anchor) is not yet, so `status` reports
|
||||
that half as incomplete and names what to allow by hand. The decision logic that
|
||||
says *what* to change is shared and tested on every platform.
|
||||
|
||||
## Checks
|
||||
|
||||
```bash
|
||||
cargo fmt --all -- --check
|
||||
cargo clippy --locked --workspace --all-targets -- -D warnings
|
||||
cargo test --locked --workspace --all-targets
|
||||
```sh
|
||||
tsng join -n home # first machine: makes the network and prints a secret
|
||||
tsng join -n home -s tsn1… # every other machine
|
||||
tsng status # who is online, over which path
|
||||
```
|
||||
|
||||
The whole suite runs offline on loopback. Set `TSUNAGI_TEST_LOG=tsunagi=debug`
|
||||
to see agent logs while a test runs.
|
||||
Every new network gets a stable pseudorandom `/24` in `10.0.0.0/8`, derived
|
||||
from its network id. Its members agree on that range even when starting offline,
|
||||
and each gets a name like `<host>.home`. Existing signed ranges are preserved.
|
||||
|
||||
There are also two runnable demos, which are demos and not substitutes for the
|
||||
tests:
|
||||
If two configured networks have overlapping ranges, the agent runs only one.
|
||||
`tsng status`, `tsng network` and the tray show which network blocks the
|
||||
other. `tsng network start <name-or-id>` or the tray’s “Switch to this network”
|
||||
button suspends the conflicting networks. They resume automatically when the
|
||||
selected network stops; the selection survives restarting the agent. Explicitly
|
||||
stopping a suspended network cancels its automatic resume.
|
||||
|
||||
```bash
|
||||
cargo run --example two_agents # control plane only
|
||||
Machines are called `tsunagi-<id>` until you rename them: `tsng id hostname laptop`.
|
||||
|
||||
### Exit nodes
|
||||
|
||||
```sh
|
||||
tsng network exit-node home on # on the machine that should be the exit
|
||||
tsng exit-node # on another one: who offers an exit
|
||||
tsng exit-node laptop # send everything through it
|
||||
tsng exit-node off
|
||||
```
|
||||
|
||||
Both run with no privileges and change nothing on the host.
|
||||
The tray does the same with a checkbox. An exit node sees everything you send
|
||||
through it, so use only members you trust.
|
||||
[Details and per-OS notes](docs/exit-node.md).
|
||||
|
||||
## Usage
|
||||
The default data transport order is `wg,awg,wg-quic,tcp-tls`. AWG negotiates a
|
||||
fresh masking profile and outer keys for each pair connection attempt, including
|
||||
reentry after fallback. Sustained data loss triggers switching independently of
|
||||
the iroh control connection. [Configuration and limits](docs/amnezia.md).
|
||||
|
||||
```rust,no_run
|
||||
use std::sync::Arc;
|
||||
use tsunagi::config::{AgentConfig, StoragePaths, TransportPolicy};
|
||||
use tsunagi::discovery::SharedMemoryDiscovery;
|
||||
use tsunagi::identity::{NetworkName, NetworkSecret};
|
||||
use tsunagi::proto::ControlMessage;
|
||||
use tsunagi::{Agent, Result};
|
||||
## Platforms
|
||||
|
||||
// The library never starts a runtime, installs a logger, handles signals,
|
||||
// forks, or calls process::exit. The binary owns all of that.
|
||||
#[tokio::main]
|
||||
async fn main() -> Result<()> {
|
||||
let config = AgentConfig::new(StoragePaths::user_default()?)
|
||||
.with_transport(TransportPolicy::N0Defaults)
|
||||
.with_discovery(Arc::new(SharedMemoryDiscovery::new()));
|
||||
| Platform | Build | CLI `tsng` | Tray | Exit node: offer / use | Packages |
|
||||
| ------------------------- | ------------ | :--------: | :--: | :--------------------: | ----------------------------------- |
|
||||
| Linux x86_64 | release | ✔ | ✔ | ✔ / ✔ | `.deb`, `.pkg.tar.zst`, tarball, Nix |
|
||||
| Linux aarch64 | release | ✔ | –¹ | ✔ / ✔ | `.deb`, `.pkg.tar.zst`, tarball, Nix |
|
||||
| Linux armv7 | release | ✔ | –¹ | ✔ / ✔ | `.deb`, `.pkg.tar.zst`, tarball |
|
||||
| macOS, Apple silicon | release | ✔ | ✔ | ✔ / ✔² | Homebrew, tarball |
|
||||
| macOS, Intel | from source | ✔³ | ✔³ | ✔ / ✔²³ | – |
|
||||
| Windows x86_64 | release | ✔ | ✔ | ✔ / ✔² | installer, zip |
|
||||
|
||||
let agent = Agent::spawn(config).await?;
|
||||
<sup>¹ Not in the release; builds from source (and through Nix).
|
||||
² Newer and less battle-tested than on Linux.
|
||||
³ Builds from source, not covered by CI.</sup>
|
||||
|
||||
let name = NetworkName::new("kitchen-table")?;
|
||||
let secret = NetworkSecret::generate(); // 32 random bytes
|
||||
println!("share this: {}", secret.encode().as_str());
|
||||
The Linux CLI is a single static binary: it runs on any distribution.
|
||||
|
||||
let network = agent.join_network(&name, &secret).await?;
|
||||
## NixOS
|
||||
|
||||
let mut events = agent.subscribe();
|
||||
tokio::spawn(async move {
|
||||
while let Ok(event) = events.recv().await {
|
||||
println!("{event:?}");
|
||||
The repository is a flake with a package and a module.
|
||||
|
||||
```nix
|
||||
# flake.nix
|
||||
{
|
||||
inputs.tsunagi.url = "github:house-of-vanity/tsunagi";
|
||||
|
||||
outputs = { nixpkgs, tsunagi, ... }: {
|
||||
nixosConfigurations.myhost = nixpkgs.lib.nixosSystem {
|
||||
modules = [
|
||||
tsunagi.nixosModules.default
|
||||
{
|
||||
services.tsunagi = {
|
||||
enable = true;
|
||||
tray.enable = true; # optional, for desktops
|
||||
users = [ "alice" ]; # may use tsng and the tray without sudo
|
||||
};
|
||||
}
|
||||
});
|
||||
|
||||
for peer in agent.network_status(network).await?.connected_peers() {
|
||||
agent
|
||||
.send(network, peer, ControlMessage::Ping { seq: 1, payload: vec![] })
|
||||
.await?;
|
||||
}
|
||||
|
||||
agent.shutdown().await;
|
||||
Ok(())
|
||||
];
|
||||
};
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
`TransportPolicy::LocalOnly` is the default, so a plain `AgentConfig::new` never
|
||||
reaches the internet by accident. Opt into `DirectOnly` or `N0Defaults`
|
||||
explicitly.
|
||||
Just trying it: `nix run github:house-of-vanity/tsunagi#tsunagi-cli -- --help`.
|
||||
`packages.<system>.tsunagi` (with the tray) and `tsunagi-cli` are also exported,
|
||||
along with `overlays.default`.
|
||||
|
||||
### Through somebody in the middle
|
||||
## Build from source
|
||||
|
||||
Tsunagi routes the overlay in userspace. A direct data link always wins;
|
||||
otherwise the shortest available path can pass through several members.
|
||||
Each authenticated member advertises only its own live, protocol-specific
|
||||
transport links. The control plane collects these into a graph and replaces
|
||||
the routing table when connectivity changes; announcements expire after
|
||||
90 seconds without refresh.
|
||||
|
||||
Transit goes transport → router → transport, without entering WireGuard or
|
||||
the intermediate host's TUN. The payload stays encrypted between the original
|
||||
endpoints. TUN connects only the local OS to Tsunagi; the kernel needs only
|
||||
the overlay route through that interface, with no IP forwarding configuration.
|
||||
|
||||
A 16-hop limit bounds loops while topology updates converge. Equal shortest
|
||||
paths use a stable flow hash derived before encryption, so packets of a TCP
|
||||
or UDP flow keep their path while topology is unchanged. Physical link changes
|
||||
do not replace end-to-end WireGuard tunnels or change the 1280-byte default MTU.
|
||||
`status` reports the hop count and first next hop for relayed paths.
|
||||
|
||||
The hot path reads an immutable routing snapshot and sends directly to a cached
|
||||
transport handle. It takes no routing mutex, walks no graph and does not parse
|
||||
the encrypted payload. See [routing.md](docs/routing.md) for the architecture,
|
||||
limits and reproducible forwarding microbenchmark.
|
||||
|
||||
This wire format requires all members to upgrade together (control ALPN 3,
|
||||
data ALPN 4); saved identities, network names, secrets and addresses survive.
|
||||
|
||||
### LAN game discovery
|
||||
|
||||
IPv4 UDP broadcast relay is enabled by default for each network. The IP router
|
||||
sends one encrypted copy to each participating peer, including through multihop
|
||||
paths. It supports `255.255.255.255` and the overlay subnet's broadcast address;
|
||||
received broadcasts never trigger another fanout.
|
||||
Rust 1.91 or newer.
|
||||
|
||||
```sh
|
||||
tsng join -n games --no-broadcast
|
||||
tsng join -n games --broadcast
|
||||
tsng network broadcast games off
|
||||
tsng network broadcast games on
|
||||
cargo build --release --bin tsng # the agent and CLI
|
||||
cargo build --release --bin tsunagi-tray # the tray (Linux needs GTK 3 and libayatana-appindicator)
|
||||
```
|
||||
|
||||
The choice persists across restarts and a plain `join`. It can be changed while
|
||||
the agent runs; `status` shows it for each network. While broadcast is on, the
|
||||
agent routes `255.255.255.255` through the Tsunagi interface and allows inbound
|
||||
UDP from the overlay range on it (Linux and Windows, needs the same privilege as
|
||||
the interface), and removes both when it is turned off or the agent stops. Games
|
||||
bound to a physical address still bypass it. Physical LAN capture/subnet sharing
|
||||
is not implemented.
|
||||
See [broadcast.md](docs/broadcast.md) for domain isolation and future LAN gateways.
|
||||
Windows needs [Wintun](https://www.wintun.net) next to `tsng.exe`.
|
||||
Per-platform notes: [Linux](dist/linux/README.md), [Arch](dist/arch/README.md),
|
||||
[macOS](dist/macos/README.md), [Windows](dist/windows/README.md).
|
||||
|
||||
## How peers find each other
|
||||
## How it works
|
||||
|
||||
Two different lookups are involved, and only one of them is this project's:
|
||||
- Your name and secret derive a network identity. Members publish signed
|
||||
contact records to the public Mainline DHT under it, and find each other there.
|
||||
- Every connection proves knowledge of the secret before it's accepted.
|
||||
Traffic between members is end-to-end encrypted, relays included.
|
||||
- Addresses and names are claimed with each device's own key and replicated
|
||||
between members. There is no one in charge, so nothing to be taken down.
|
||||
|
||||
**1. Resolving one endpoint's address — iroh's, and it works today.**
|
||||
With `--reach relay` or `--reach direct`, iroh publishes a signed
|
||||
record of this endpoint's addresses, keyed by its endpoint id, to the public
|
||||
service run by Number 0 — "n0", the company behind iroh — at `dns.iroh.link`,
|
||||
over pkarr and DNS, and resolves other endpoints the same way. That is why `--peer <endpoint-id>` works with no address attached:
|
||||
iroh looks it up. None of that code is ours.
|
||||
|
||||
**2. Finding who is in a network — Mainline DHT by default.**
|
||||
`NetworkDiscovery` supplies unverified candidates. The Mainline backend uses
|
||||
a signing key derived from the network name and secret to publish signed
|
||||
BEP44 records containing each writer's endpoint ID and addresses. Its 16
|
||||
slots provide starting candidates, not a complete membership list or a limit
|
||||
on network size. After the first authenticated connection, introductions and
|
||||
signed state reveal the other members. `StaticBootstrap` (`--peer`) remains
|
||||
available, and tests use either a local Mainline Testnet or in-memory discovery.
|
||||
|
||||
What this means in practice:
|
||||
|
||||
- With `relay` or `direct`, **your endpoint id and IP addresses are published
|
||||
to a public third-party service** (Number 0's, unless you change it). They are not secret, and the network secret is
|
||||
never published, but an observer of that service learns that your endpoint
|
||||
exists and where it is. `--reach local` publishes nothing.
|
||||
- With DHT enabled, endpoint records are also published to the public Mainline
|
||||
DHT. They are signed, not encrypted; membership still requires the secret.
|
||||
- A relay, when one is needed, sees the volume and timing of your traffic — not
|
||||
its contents. The default relays are Number 0's, in the US, EU and
|
||||
Asia-Pacific.
|
||||
|
||||
## Storage
|
||||
|
||||
Two physically separate SQLite files, placed wherever the library's
|
||||
configuration says (`StoragePaths`). A future system service supplies its own
|
||||
paths; tests always use temporary directories.
|
||||
|
||||
| file | holds | when damaged |
|
||||
|----------------|--------------------------------------------------|-------------------------|
|
||||
| `state.sqlite` | device identity, network configuration, hostname | clear error, never reset |
|
||||
| `cache.sqlite` | address hints and other recoverable data | discarded and recreated |
|
||||
|
||||
The `wg-quic` protocol keeps its own keys in its own store under `wg-quic/`,
|
||||
because a protocol's keys are neither the iroh identity nor the network
|
||||
secret.
|
||||
|
||||
The command line agent puts everything under the platform's per-user
|
||||
directories by default; `--state-dir` and `--cache-dir` override them.
|
||||
|
||||
One state directory belongs to one live agent, enforced with a real OS file
|
||||
lock rather than an existence check. That directory *is* the identity: one
|
||||
agent, one device key, one interface, and as many networks as you like on it
|
||||
— `tsng network join` adds them to the agent that is already running,
|
||||
which is why a second `tsng up` on the same directory is refused rather
|
||||
than made to work. A second agent on the same host is a second identity, and
|
||||
needs everything of its own: its own state and cache directories, its own
|
||||
interface name and its own overlay range. The library owns no globals, so
|
||||
several of them run side by side in one process as readily as in one host.
|
||||
|
||||
### Leaving, and starting over
|
||||
|
||||
Membership outlives a session, so it also has to be possible to end it.
|
||||
|
||||
```bash
|
||||
tsng network # what this device belongs to
|
||||
tsng network leave lab # give up the address and the name, then forget it
|
||||
tsng wipe --yes # remove everything and be a stranger again
|
||||
```
|
||||
|
||||
**Stopping is not leaving.** `stop` closes this network's sessions, takes
|
||||
its address off the interface and keeps it from starting again, and that is
|
||||
all: the configuration, the secret, the signed state and the protocol key
|
||||
stay exactly as they are, nothing is announced, and to the others this
|
||||
device is simply away — an ordinary condition they already handle, with its
|
||||
address and name still reserved for it. `start` picks it up where it left
|
||||
off. A network named on the `up` command line is started by that command
|
||||
whatever its stored state, and the start-up banner says so rather than
|
||||
letting a `stop` quietly come back.
|
||||
|
||||
`leave` is the other one, and it removes, locally: the configuration and
|
||||
the secret, this network's signed records, its cached address hints and the
|
||||
protocol key it used there. What it keeps is this author's version counter
|
||||
for that network — a rejoin has to continue above the release, or every
|
||||
replica would treat the new claim as stale. Everything about *other*
|
||||
networks, and the device identity itself, is untouched.
|
||||
|
||||
`leave` publishes a signed `Release` **first**, while the agent is running and
|
||||
its sessions are up, so the address and the name are freed for the others
|
||||
instead of staying reserved to a member that has gone. They pass the tombstone
|
||||
on, so a member that was away hears it from them. With no agent running,
|
||||
nothing can sign or send it: the command says so and refuses, and `--offline`
|
||||
drops the network locally while leaving the others holding the old claim. The
|
||||
protocol key for that network goes too — rejoining is joining, not resuming.
|
||||
|
||||
`leave`, `stop`, `start`, `secret` and `broadcast` accept an exact stored network
|
||||
name, its full id, or a unique id prefix (including a copied report ellipsis).
|
||||
Names are case-sensitive. If several networks share a name, or a name matches
|
||||
another network's id prefix, the command lists matching names and full ids and
|
||||
changes nothing. Use a full id to choose explicitly; a full id always takes
|
||||
precedence over a name. No secret is needed to select a configured network.
|
||||
|
||||
`wipe` removes both directories' contents: the device identity, every network,
|
||||
every signed record and everything a protocol kept beside them. It refuses
|
||||
while an agent is running, and refuses a directory with no `state.sqlite` in
|
||||
it, so a mistyped `--state-dir` cannot take somebody's documents with it.
|
||||
Without `--yes` it only says what it would remove. It is not a goodbye: nobody
|
||||
is told, because after it there is no key left to sign anything with. Leave the
|
||||
networks first if the addresses should be freed.
|
||||
The secret is the only key, so make it a good one (the generated ones are).
|
||||
The [threat model](docs/threat-model.md) says what this does and doesn't protect against.
|
||||
|
||||
## Documentation
|
||||
|
||||
- [docs/architecture.md](docs/architecture.md) — module boundaries and runtime.
|
||||
- [docs/wireguard.md](docs/wireguard.md) — the WireGuard plugin: overlay
|
||||
addressing, announcements, backends, reconciliation.
|
||||
- [docs/protocol.md](docs/protocol.md) — identity derivation, framing, handshake.
|
||||
- [docs/sync-model.md](docs/sync-model.md) — the planned signed-state model and
|
||||
what is deliberately not built yet.
|
||||
- [docs/exit-node.md](docs/exit-node.md) — exit nodes: sending all traffic through a member.
|
||||
- [docs/threat-model.md](docs/threat-model.md) — threat model and known limits.
|
||||
- [docs/testing.md](docs/testing.md) — what the suite covers and what it does not.
|
||||
- [AGENTS.md](AGENTS.md) — rules for anyone (human or agent) changing this repo.
|
||||
|
||||
## Security in one paragraph
|
||||
|
||||
Membership is proved by an HMAC over a transcript keyed by a value derived from
|
||||
the shared secret, bound to the specific iroh connection through the TLS
|
||||
exporter, to the network id, to both endpoint identities and to distinct role
|
||||
labels. This targets high-entropy secrets: there is no PAKE here, so a short
|
||||
human passphrase is guessable offline by anyone who can reach the handshake.
|
||||
Anyone who knows the secret is a full participant and can create many
|
||||
identities. Read [docs/threat-model.md](docs/threat-model.md) before relying on
|
||||
any of this.
|
||||
[Architecture](docs/architecture.md) ·
|
||||
[Protocol](docs/protocol.md) ·
|
||||
[WireGuard data plane](docs/wireguard.md) ·
|
||||
[Pairwise Amnezia transport](docs/amnezia.md) ·
|
||||
[Routing](docs/routing.md) ·
|
||||
[Exit nodes](docs/exit-node.md) ·
|
||||
[Threat model](docs/threat-model.md) ·
|
||||
[Contributing](AGENTS.md)
|
||||
|
||||
## Licence
|
||||
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
[package]
|
||||
name = "tsunagi-awg"
|
||||
version = "0.1.0"
|
||||
description = "Ephemeral pairwise Amnezia transport for Tsunagi."
|
||||
edition.workspace = true
|
||||
rust-version.workspace = true
|
||||
license = "WTFPL AND Apache-2.0"
|
||||
repository.workspace = true
|
||||
|
||||
[dependencies]
|
||||
tsunagi-core = { path = "../tsunagi-core" }
|
||||
tsunagi-wg-quic = { path = "../tsunagi-wg-quic" }
|
||||
tsunagi-wireguard = { path = "../tsunagi-wireguard" }
|
||||
iroh.workspace = true
|
||||
tokio = { workspace = true, features = ["net", "time", "sync", "macros"] }
|
||||
serde.workspace = true
|
||||
postcard.workspace = true
|
||||
bytes.workspace = true
|
||||
tracing.workspace = true
|
||||
rand = "0.10"
|
||||
zeroize = { version = "1.9", features = ["derive"] }
|
||||
boringtun = { version = "0.7.1", default-features = false }
|
||||
chacha20 = "0.9"
|
||||
|
||||
[dev-dependencies]
|
||||
tsunagi-core = { path = "../tsunagi-core", features = ["testing"] }
|
||||
tempfile.workspace = true
|
||||
|
||||
[lints]
|
||||
workspace = true
|
||||
@@ -0,0 +1,201 @@
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding those notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
APPENDIX: How to apply the Apache License to your work.
|
||||
|
||||
To apply the Apache License to your work, attach the following
|
||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
||||
replaced with your own identifying information. (Don't include
|
||||
the brackets!) The text should be enclosed in the appropriate
|
||||
comment syntax for the file format. We also recommend that a
|
||||
file or class name and description of purpose be included on the
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright 2026 Friderika Coulson (NanoBee)
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
@@ -0,0 +1,21 @@
|
||||
# Amnezia packet engine
|
||||
|
||||
`src/masking.rs` is adapted from `src/core/amnezia.rs` in
|
||||
https://github.com/RunaRoo/AmneziaWG-RS at
|
||||
`51579a7d6e8882d1d8d6d00822f09e18a9423b73`, under Apache-2.0.
|
||||
The upstream license is retained in `LICENSE-AmneziaWG-RS`.
|
||||
|
||||
The adaptation replaces configuration-file loading with bounded typed pair
|
||||
profiles, updates RNG calls to rand 0.10, removes an unnecessary unwrap, and
|
||||
does not embed upstream's Node, global executors, OS interface operations or
|
||||
detached tasks. Noise/WireGuard is supplied by boringtun, already used by
|
||||
Tsunagi. Each pair owns one socket and one cancellable packet-engine task.
|
||||
|
||||
This is Tsunagi AWG wire version 1, not a claim of compatibility with external
|
||||
Amnezia servers. The masking engine substitutes message headers outside the
|
||||
standard WireGuard state machine and restores them before Noise validation.
|
||||
There are no remote file paths, shell hooks, or persisted pair profiles.
|
||||
|
||||
The adapted masking source remains Apache-2.0. Newly written Tsunagi code
|
||||
follows the repository's WTFPL. This package declares both licenses in Cargo
|
||||
metadata; the workspace license alone does not describe its copied source.
|
||||
@@ -0,0 +1,16 @@
|
||||
//! Pairwise, ephemeral Amnezia packet transport with independent end-to-end tunnels.
|
||||
//! All negotiation travels over authenticated Tsunagi control sessions.
|
||||
mod masking;
|
||||
mod plugin;
|
||||
pub mod profile;
|
||||
mod session;
|
||||
mod stun;
|
||||
mod transport;
|
||||
|
||||
pub use plugin::AwgPlugin;
|
||||
pub use transport::{AwgTransport, PairOverview};
|
||||
|
||||
/// Protocol identifier and default second-choice direct transport.
|
||||
pub const AWG_PROTOCOL: &str = "awg";
|
||||
/// Tsunagi AWG wire version, independent of software releases.
|
||||
pub const AWG_WIRE_VERSION: u16 = 1;
|
||||
@@ -0,0 +1,336 @@
|
||||
// Adapted from RunaRoo/AmneziaWG-RS, commit 51579a7d6e8882d1d8d6d00822f09e18a9423b73.
|
||||
// Apache-2.0; see LICENSE-AmneziaWG-RS and UPSTREAM.md.
|
||||
#![allow(dead_code, missing_docs)]
|
||||
// AmneziaWG configuration, header protection, and traffic shaping.
|
||||
|
||||
use bytes::{Bytes, BytesMut};
|
||||
use chacha20::{
|
||||
ChaCha20,
|
||||
cipher::{KeyIvInit, StreamCipher},
|
||||
};
|
||||
use rand::RngExt;
|
||||
|
||||
pub const HEADER_PROTECTION_NONCE_SIZE: usize = 12;
|
||||
pub const DEFAULT_UDP_WINDOW: usize = 500;
|
||||
|
||||
#[derive(Clone, Copy, Debug, Default, Eq, PartialEq)]
|
||||
pub struct Range {
|
||||
pub min: u32,
|
||||
pub max: u32,
|
||||
}
|
||||
|
||||
impl Range {
|
||||
pub fn contains(&self, val: u32) -> bool {
|
||||
val >= self.min && val <= self.max
|
||||
}
|
||||
|
||||
pub fn random(&self) -> u32 {
|
||||
if self.min >= self.max {
|
||||
self.min
|
||||
} else {
|
||||
rand::rng().random_range(self.min..=self.max)
|
||||
}
|
||||
}
|
||||
|
||||
pub fn is_zero(&self) -> bool {
|
||||
self.min == 0 && self.max == 0
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
|
||||
pub enum AwgKind {
|
||||
Initiation,
|
||||
Response,
|
||||
Cookie,
|
||||
Transport,
|
||||
}
|
||||
|
||||
impl AwgKind {
|
||||
pub fn fixed_size(self) -> usize {
|
||||
match self {
|
||||
Self::Initiation => 148,
|
||||
Self::Response => 92,
|
||||
Self::Cookie => 64,
|
||||
Self::Transport => 32,
|
||||
}
|
||||
}
|
||||
|
||||
fn standard_type(self) -> u32 {
|
||||
match self {
|
||||
Self::Initiation => 1,
|
||||
Self::Response => 2,
|
||||
Self::Cookie => 3,
|
||||
Self::Transport => 4,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
pub struct DecodedPacket {
|
||||
pub kind: AwgKind,
|
||||
pub message: Bytes,
|
||||
pub wire_len: usize,
|
||||
}
|
||||
|
||||
pub struct AmneziaConfig {
|
||||
pub jc: u32,
|
||||
pub jmin: usize,
|
||||
pub jmax: usize,
|
||||
pub s1: usize,
|
||||
pub s2: usize,
|
||||
pub s3: usize,
|
||||
pub s4: usize,
|
||||
pub h1: Range,
|
||||
pub h2: Range,
|
||||
pub h3: Range,
|
||||
pub h4: Range,
|
||||
pub i1: String,
|
||||
pub i2: String,
|
||||
pub i3: String,
|
||||
pub i4: String,
|
||||
pub i5: String,
|
||||
pub header_protection_key: Option<[u8; 32]>,
|
||||
pub content_padding_addition: Range,
|
||||
pub random_trailers: bool,
|
||||
pub mtu: usize,
|
||||
}
|
||||
|
||||
impl AmneziaConfig {
|
||||
pub fn random_header(&self, range: &Range) -> u32 {
|
||||
range.random()
|
||||
}
|
||||
|
||||
pub fn random_padding(&self, len: usize) -> Vec<u8> {
|
||||
let mut padding = vec![0u8; len];
|
||||
if len > 0 {
|
||||
rand::fill(&mut padding);
|
||||
}
|
||||
padding
|
||||
}
|
||||
|
||||
fn shape(&self, kind: AwgKind) -> (usize, Range) {
|
||||
match kind {
|
||||
AwgKind::Initiation => (self.s1, self.h1),
|
||||
AwgKind::Response => (self.s2, self.h2),
|
||||
AwgKind::Cookie => (self.s3, self.h3),
|
||||
AwgKind::Transport => (self.s4, self.h4),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn encode(&self, kind: AwgKind, payload: Bytes, udp_window: usize) -> Bytes {
|
||||
let (padding_len, _) = self.shape(kind);
|
||||
let trailer_len = if kind != AwgKind::Transport && self.random_trailers {
|
||||
let base = padding_len.saturating_add(payload.len());
|
||||
let room = udp_window.max(DEFAULT_UDP_WINDOW).saturating_sub(base);
|
||||
if room == 0 {
|
||||
0
|
||||
} else {
|
||||
rand::rng().random_range(0..room)
|
||||
}
|
||||
} else {
|
||||
0
|
||||
};
|
||||
|
||||
let mut out = BytesMut::zeroed(padding_len + payload.len() + trailer_len);
|
||||
if padding_len > 0 {
|
||||
rand::fill(&mut out[..padding_len]);
|
||||
}
|
||||
out[padding_len..padding_len + payload.len()].copy_from_slice(&payload);
|
||||
if trailer_len > 0 {
|
||||
rand::fill(&mut out[padding_len + payload.len()..]);
|
||||
}
|
||||
|
||||
if self.header_protection_key.is_some() {
|
||||
let protected_len = if kind == AwgKind::Transport {
|
||||
16
|
||||
} else {
|
||||
kind.fixed_size()
|
||||
};
|
||||
self.apply_header_cipher(&mut out, padding_len, protected_len);
|
||||
}
|
||||
|
||||
out.freeze()
|
||||
}
|
||||
|
||||
pub fn decode(&self, datagram: &Bytes) -> Option<DecodedPacket> {
|
||||
if datagram.len() < 4 {
|
||||
return None;
|
||||
}
|
||||
|
||||
for kind in [
|
||||
AwgKind::Initiation,
|
||||
AwgKind::Response,
|
||||
AwgKind::Cookie,
|
||||
AwgKind::Transport,
|
||||
] {
|
||||
let (padding_len, headers) = self.shape(kind);
|
||||
let minimum = padding_len + kind.fixed_size();
|
||||
let size_ok = match kind {
|
||||
AwgKind::Transport | AwgKind::Initiation => datagram.len() >= minimum,
|
||||
AwgKind::Response | AwgKind::Cookie if self.random_trailers => {
|
||||
datagram.len() >= minimum
|
||||
}
|
||||
AwgKind::Response | AwgKind::Cookie => datagram.len() == minimum,
|
||||
};
|
||||
if !size_ok || datagram.len() < padding_len + 4 {
|
||||
continue;
|
||||
}
|
||||
|
||||
let protected_header =
|
||||
u32::from_le_bytes(datagram[padding_len..padding_len + 4].try_into().ok()?);
|
||||
let candidate = if self.header_protection_key.is_some() {
|
||||
protected_header ^ self.header_type_hash(datagram)?
|
||||
} else {
|
||||
protected_header
|
||||
};
|
||||
if !headers.contains(candidate) {
|
||||
continue;
|
||||
}
|
||||
|
||||
let message_len = match kind {
|
||||
AwgKind::Transport | AwgKind::Initiation => datagram.len() - padding_len,
|
||||
AwgKind::Response | AwgKind::Cookie => kind.fixed_size(),
|
||||
};
|
||||
let mut payload = BytesMut::from(&datagram[padding_len..padding_len + message_len]);
|
||||
if self.header_protection_key.is_some() {
|
||||
let protected_len = if kind == AwgKind::Transport {
|
||||
16
|
||||
} else {
|
||||
kind.fixed_size()
|
||||
};
|
||||
self.apply_header_cipher_with_nonce(
|
||||
&mut payload,
|
||||
0,
|
||||
protected_len,
|
||||
&datagram[..HEADER_PROTECTION_NONCE_SIZE],
|
||||
);
|
||||
}
|
||||
|
||||
return Some(DecodedPacket {
|
||||
kind,
|
||||
message: payload.freeze(),
|
||||
wire_len: datagram.len(),
|
||||
});
|
||||
}
|
||||
|
||||
None
|
||||
}
|
||||
|
||||
pub fn prepare_transport_plaintext(&self, packet: Bytes, udp_window: usize) -> Bytes {
|
||||
if packet.is_empty() {
|
||||
return packet;
|
||||
}
|
||||
|
||||
let requested = if !self.content_padding_addition.is_zero() {
|
||||
self.content_padding_addition.random() as usize
|
||||
} else if self.random_trailers {
|
||||
let wire_without_padding = self.s4 + AwgKind::Transport.fixed_size() + packet.len();
|
||||
let room = udp_window
|
||||
.max(DEFAULT_UDP_WINDOW)
|
||||
.saturating_sub(wire_without_padding);
|
||||
if room == 0 {
|
||||
0
|
||||
} else {
|
||||
rand::rng().random_range(0..room)
|
||||
}
|
||||
} else {
|
||||
let aligned = (packet.len() + 15) & !15;
|
||||
aligned.saturating_sub(packet.len())
|
||||
};
|
||||
|
||||
let padding = self.clamp_padding_to_mtu(packet.len(), requested);
|
||||
if padding == 0 {
|
||||
return packet;
|
||||
}
|
||||
|
||||
let mut out = BytesMut::with_capacity(packet.len() + padding);
|
||||
out.extend_from_slice(&packet);
|
||||
out.resize(packet.len() + padding, 0);
|
||||
out.freeze()
|
||||
}
|
||||
|
||||
pub fn transport_content_length(&self, plaintext: &[u8]) -> Option<usize> {
|
||||
if plaintext.is_empty() || plaintext[0] == 0 {
|
||||
return Some(0);
|
||||
}
|
||||
match plaintext[0] >> 4 {
|
||||
4 if plaintext.len() >= 20 => {
|
||||
let total = u16::from_be_bytes([plaintext[2], plaintext[3]]) as usize;
|
||||
(total >= 20 && total <= plaintext.len()).then_some(total)
|
||||
}
|
||||
6 if plaintext.len() >= 40 => {
|
||||
let payload = u16::from_be_bytes([plaintext[4], plaintext[5]]) as usize;
|
||||
let total = 40 + payload;
|
||||
(total <= plaintext.len()).then_some(total)
|
||||
}
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
fn clamp_padding_to_mtu(&self, packet_len: usize, requested: usize) -> usize {
|
||||
if self.mtu == 0 {
|
||||
return requested;
|
||||
}
|
||||
let normalized = if packet_len > self.mtu {
|
||||
packet_len % self.mtu
|
||||
} else {
|
||||
packet_len
|
||||
};
|
||||
requested.min(self.mtu.saturating_sub(normalized))
|
||||
}
|
||||
|
||||
fn header_type_hash(&self, datagram: &Bytes) -> Option<u32> {
|
||||
if datagram.len() < HEADER_PROTECTION_NONCE_SIZE {
|
||||
return None;
|
||||
}
|
||||
let mut hash = [0u8; 4];
|
||||
self.apply_keystream(&mut hash, &datagram[..HEADER_PROTECTION_NONCE_SIZE]);
|
||||
Some(u32::from_le_bytes(hash))
|
||||
}
|
||||
|
||||
fn apply_header_cipher(&self, packet: &mut [u8], offset: usize, len: usize) {
|
||||
if packet.len() < HEADER_PROTECTION_NONCE_SIZE || packet.len() < offset + len {
|
||||
return;
|
||||
}
|
||||
let nonce: [u8; HEADER_PROTECTION_NONCE_SIZE] =
|
||||
match packet[..HEADER_PROTECTION_NONCE_SIZE].try_into() {
|
||||
Ok(nonce) => nonce,
|
||||
Err(_) => return,
|
||||
};
|
||||
self.apply_header_cipher_with_nonce(packet, offset, len, &nonce);
|
||||
}
|
||||
|
||||
fn apply_header_cipher_with_nonce(
|
||||
&self,
|
||||
packet: &mut [u8],
|
||||
offset: usize,
|
||||
len: usize,
|
||||
nonce: &[u8],
|
||||
) {
|
||||
if packet.len() < offset + len {
|
||||
return;
|
||||
}
|
||||
self.apply_keystream(&mut packet[offset..offset + len], nonce);
|
||||
}
|
||||
|
||||
fn apply_keystream(&self, data: &mut [u8], nonce: &[u8]) {
|
||||
let Some(key) = self.header_protection_key.as_ref() else {
|
||||
return;
|
||||
};
|
||||
if nonce.len() != HEADER_PROTECTION_NONCE_SIZE {
|
||||
return;
|
||||
}
|
||||
let Ok(nonce) = <&[u8; HEADER_PROTECTION_NONCE_SIZE]>::try_from(nonce) else {
|
||||
return;
|
||||
};
|
||||
let mut cipher = ChaCha20::new(key.into(), nonce.into());
|
||||
cipher.apply_keystream(data);
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for AmneziaConfig {
|
||||
fn drop(&mut self) {
|
||||
use zeroize::Zeroize;
|
||||
self.header_protection_key.zeroize();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,132 @@
|
||||
//! End-to-end encryption remains independent of physical Amnezia pair links.
|
||||
use crate::{AWG_PROTOCOL, AWG_WIRE_VERSION, AwgTransport};
|
||||
use iroh::EndpointId;
|
||||
use std::sync::Arc;
|
||||
use tsunagi::dataplane::transport::SharedLink;
|
||||
use tsunagi::dataplane::{IpPlugin, PluginCapability, PluginContext, PluginError, ProtocolOption};
|
||||
use tsunagi::state::Ipv4Range;
|
||||
use tsunagi::{BoxFuture, NetworkId};
|
||||
use tsunagi_wg_quic::{NetworkOverview, WireguardConfig, WireguardPlugin};
|
||||
|
||||
/// AWG capability and an independent end-to-end WireGuard engine.
|
||||
/// Pair transport keys/profiles never enter its durable end-to-end key store.
|
||||
#[derive(Debug)]
|
||||
pub struct AwgPlugin {
|
||||
inner: Arc<WireguardPlugin>,
|
||||
transport: Arc<AwgTransport>,
|
||||
}
|
||||
impl AwgPlugin {
|
||||
/// Uses standard tunnel MTU/keepalive options; pair ports and profiles are ephemeral.
|
||||
pub const OPTIONS: &'static [ProtocolOption] = &[
|
||||
ProtocolOption {
|
||||
key: "keepalive",
|
||||
value: "SECONDS",
|
||||
help: "end-to-end tunnel keepalive; 0 disables it",
|
||||
default: Some("25"),
|
||||
},
|
||||
ProtocolOption {
|
||||
key: "mtu",
|
||||
value: "BYTES",
|
||||
help: "maximum overlay packet size",
|
||||
default: Some("1280"),
|
||||
},
|
||||
ProtocolOption {
|
||||
key: "stun",
|
||||
value: "HOST:PORT|off",
|
||||
help: "UDP public mapping discovery; disabled in local reach mode",
|
||||
default: Some("stun.l.google.com:19302"),
|
||||
},
|
||||
];
|
||||
/// Opens the end-to-end engine, retaining separate pair transport state.
|
||||
pub async fn open(
|
||||
mut config: WireguardConfig,
|
||||
transport: Arc<AwgTransport>,
|
||||
) -> Result<Arc<Self>, PluginError> {
|
||||
config.protocol = AWG_PROTOCOL;
|
||||
Ok(Arc::new(Self {
|
||||
inner: WireguardPlugin::open(config).await?,
|
||||
transport,
|
||||
}))
|
||||
}
|
||||
/// Reports end-to-end tunnel health and traffic counters.
|
||||
pub fn overview(&self, network: NetworkId) -> Option<NetworkOverview> {
|
||||
self.inner.overview(network)
|
||||
}
|
||||
}
|
||||
impl IpPlugin for AwgPlugin {
|
||||
fn protocol_id(&self) -> &str {
|
||||
AWG_PROTOCOL
|
||||
}
|
||||
fn protocol_version(&self) -> u16 {
|
||||
AWG_WIRE_VERSION
|
||||
}
|
||||
fn options(&self) -> &'static [ProtocolOption] {
|
||||
Self::OPTIONS
|
||||
}
|
||||
fn attach(&self, context: PluginContext) {
|
||||
self.inner.attach(context);
|
||||
}
|
||||
fn local_capability(
|
||||
&self,
|
||||
network: NetworkId,
|
||||
) -> Result<Option<PluginCapability>, PluginError> {
|
||||
self.inner.local_capability(network).map(|cap| {
|
||||
cap.map(|mut cap| {
|
||||
cap.protocol = AWG_PROTOCOL.into();
|
||||
cap.version = AWG_WIRE_VERSION;
|
||||
cap
|
||||
})
|
||||
})
|
||||
}
|
||||
fn on_network_activated(&self, network: NetworkId) {
|
||||
self.inner.on_network_activated(network);
|
||||
}
|
||||
fn on_peer_capability(
|
||||
&self,
|
||||
network: NetworkId,
|
||||
peer: EndpointId,
|
||||
capability: &PluginCapability,
|
||||
) -> Result<(), PluginError> {
|
||||
if capability.protocol != AWG_PROTOCOL
|
||||
|| capability.version != AWG_WIRE_VERSION
|
||||
|| !capability.enabled
|
||||
{
|
||||
return Err(PluginError::Rejected("unsupported AWG capability".into()));
|
||||
}
|
||||
self.inner.on_peer_capability(network, peer, capability)?;
|
||||
self.transport.allow_peer(network, peer);
|
||||
Ok(())
|
||||
}
|
||||
fn on_address_allocation(
|
||||
&self,
|
||||
network: NetworkId,
|
||||
range: Ipv4Range,
|
||||
allocations: &[(EndpointId, std::net::Ipv4Addr)],
|
||||
) {
|
||||
self.inner
|
||||
.on_address_allocation(network, range, allocations);
|
||||
}
|
||||
fn carry(&self, network: NetworkId, peer: EndpointId, packet: bytes::Bytes) -> bool {
|
||||
self.inner.carry(network, peer, packet)
|
||||
}
|
||||
fn on_peer_link(&self, network: NetworkId, peer: EndpointId, link: SharedLink) {
|
||||
self.inner.on_peer_link(network, peer, link);
|
||||
}
|
||||
fn on_peer_gone(&self, network: NetworkId, peer: EndpointId) {
|
||||
self.transport.forget_peer(network, peer);
|
||||
self.inner.on_peer_gone(network, peer);
|
||||
}
|
||||
fn on_network_deactivated(&self, network: NetworkId) {
|
||||
self.transport.stop_network(network);
|
||||
self.inner.on_network_deactivated(network);
|
||||
}
|
||||
fn on_network_forgotten(&self, network: NetworkId) {
|
||||
self.inner.on_network_forgotten(network);
|
||||
}
|
||||
fn shutdown(&self) -> BoxFuture<'_, ()> {
|
||||
Box::pin(async move {
|
||||
self.transport.shutdown();
|
||||
self.inner.shutdown().await;
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,226 @@
|
||||
//! Bounded profiles, regenerated for every pair connection attempt.
|
||||
use crate::masking::{AmneziaConfig, AwgKind, Range};
|
||||
use bytes::Bytes;
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
/// An ephemeral masking profile. Secret fields are redacted and erased on drop.
|
||||
#[derive(Clone, Serialize, Deserialize, zeroize::ZeroizeOnDrop)]
|
||||
pub struct PairProfile {
|
||||
id: [u8; 16],
|
||||
prefix: u16,
|
||||
headers: [[u32; 2]; 4],
|
||||
header_key: [u8; 32],
|
||||
pub(crate) psk: [u8; 32],
|
||||
junk_count: u8,
|
||||
junk_min: u16,
|
||||
junk_max: u16,
|
||||
padding: u16,
|
||||
signature: u8,
|
||||
}
|
||||
impl std::fmt::Debug for PairProfile {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
f.write_str("PairProfile([REDACTED])")
|
||||
}
|
||||
}
|
||||
impl PairProfile {
|
||||
/// Generates a new safe profile; no deterministic seed or durable state.
|
||||
pub fn generate() -> Self {
|
||||
let headers = std::array::from_fn(|i| {
|
||||
let base = (i as u32) * 0x4000_0000 + rand::random_range(1024..0x3fff_0000);
|
||||
[base, base + rand::random_range(64..1024)]
|
||||
});
|
||||
Self {
|
||||
id: rand::random(),
|
||||
prefix: rand::random_range(12..49),
|
||||
headers,
|
||||
header_key: rand::random(),
|
||||
psk: rand::random(),
|
||||
junk_count: rand::random_range(4..9),
|
||||
junk_min: rand::random_range(24..65),
|
||||
junk_max: rand::random_range(80..161),
|
||||
padding: rand::random_range(16..65),
|
||||
signature: rand::random_range(0..2),
|
||||
}
|
||||
}
|
||||
/// Public attempt identifier, useful for distinguishing retries without revealing secrets.
|
||||
pub fn id(&self) -> [u8; 16] {
|
||||
self.id
|
||||
}
|
||||
/// Header key, for packet engines and protocol tests; never log this value.
|
||||
pub fn header_key(&self) -> &[u8; 32] {
|
||||
&self.header_key
|
||||
}
|
||||
/// Rejects invalid remote profiles before allocating packets or entering the codec.
|
||||
pub fn validate(&self) -> Result<(), &'static str> {
|
||||
if !(12..=48).contains(&self.prefix)
|
||||
|| !(4..=8).contains(&self.junk_count)
|
||||
|| self.junk_min < 24
|
||||
|| self.junk_min > self.junk_max
|
||||
|| self.junk_max > 160
|
||||
|| self.padding > 64
|
||||
|| self.signature > 1
|
||||
|| self.header_key == [0; 32]
|
||||
|| self.psk == [0; 32]
|
||||
{
|
||||
return Err("invalid AWG profile bounds");
|
||||
}
|
||||
for (i, [lo, hi]) in self.headers.iter().copied().enumerate() {
|
||||
if lo < 5 || lo > hi || hi - lo > 1024 {
|
||||
return Err("invalid AWG header range");
|
||||
}
|
||||
for [other_lo, other_hi] in &self.headers[..i] {
|
||||
if lo <= *other_hi && *other_lo <= hi {
|
||||
return Err("overlapping AWG header ranges");
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
fn config(&self) -> AmneziaConfig {
|
||||
let range = |i: usize| Range {
|
||||
min: self.headers[i][0],
|
||||
max: self.headers[i][1],
|
||||
};
|
||||
AmneziaConfig {
|
||||
jc: self.junk_count.into(),
|
||||
jmin: self.junk_min.into(),
|
||||
jmax: self.junk_max.into(),
|
||||
s1: self.prefix.into(),
|
||||
s2: self.prefix.into(),
|
||||
s3: self.prefix.into(),
|
||||
s4: self.prefix.into(),
|
||||
h1: range(0),
|
||||
h2: range(1),
|
||||
h3: range(2),
|
||||
h4: range(3),
|
||||
i1: String::new(),
|
||||
i2: String::new(),
|
||||
i3: String::new(),
|
||||
i4: String::new(),
|
||||
i5: String::new(),
|
||||
header_protection_key: Some(self.header_key),
|
||||
content_padding_addition: Range {
|
||||
min: 0,
|
||||
max: self.padding.into(),
|
||||
},
|
||||
random_trailers: true,
|
||||
mtu: 1050,
|
||||
}
|
||||
}
|
||||
/// Masks one standard WireGuard message; invalid types and oversized packets are refused.
|
||||
pub fn encode(&self, raw: &[u8]) -> Option<Bytes> {
|
||||
let kind = match raw.get(..4)? {
|
||||
[1, 0, 0, 0] => AwgKind::Initiation,
|
||||
[2, 0, 0, 0] => AwgKind::Response,
|
||||
[3, 0, 0, 0] => AwgKind::Cookie,
|
||||
[4, 0, 0, 0] => AwgKind::Transport,
|
||||
_ => return None,
|
||||
};
|
||||
if raw.len() < kind.fixed_size() || raw.len() > 1050 {
|
||||
return None;
|
||||
}
|
||||
let config = self.config();
|
||||
let range = match kind {
|
||||
AwgKind::Initiation => config.h1,
|
||||
AwgKind::Response => config.h2,
|
||||
AwgKind::Cookie => config.h3,
|
||||
AwgKind::Transport => config.h4,
|
||||
};
|
||||
let mut payload = raw.to_vec();
|
||||
payload[..4].copy_from_slice(&range.random().to_le_bytes());
|
||||
Some(config.encode(kind, Bytes::from(payload), 1200))
|
||||
}
|
||||
/// Unmasks a packet and restores standard headers before WireGuard validation.
|
||||
pub fn decode(&self, wire: &[u8]) -> Option<Vec<u8>> {
|
||||
if wire.len() > 1200 {
|
||||
return None;
|
||||
}
|
||||
let decoded = self.config().decode(&Bytes::copy_from_slice(wire))?;
|
||||
let (kind, len) = match decoded.kind {
|
||||
AwgKind::Initiation => (1, 148),
|
||||
AwgKind::Response => (2, 92),
|
||||
AwgKind::Cookie => (3, 64),
|
||||
AwgKind::Transport => (4, decoded.message.len()),
|
||||
};
|
||||
let mut message = decoded.message.get(..len)?.to_vec();
|
||||
message[..4].copy_from_slice(&(kind as u32).to_le_bytes());
|
||||
Some(message)
|
||||
}
|
||||
pub(crate) fn pad(&self, packet: Bytes) -> Bytes {
|
||||
self.config().prepare_transport_plaintext(packet, 1200)
|
||||
}
|
||||
/// Generates structured signature packets with fresh fields, never static captures.
|
||||
/// These target the other Tsunagi peer, not DNS or SIP services.
|
||||
pub fn signature_packets(&self) -> Vec<Vec<u8>> {
|
||||
if self.signature == 0 {
|
||||
// A complete DNS query. Counts, label lengths and qtype remain valid.
|
||||
let label = format!("n{:016x}", rand::random::<u64>());
|
||||
let mut packet = Vec::from(rand::random::<u16>().to_be_bytes());
|
||||
packet.extend_from_slice(&[1, 0, 0, 1, 0, 0, 0, 0, 0, 0]);
|
||||
packet.push(label.len() as u8);
|
||||
packet.extend_from_slice(label.as_bytes());
|
||||
packet.extend_from_slice(b"\x07example\x03net\0\0");
|
||||
packet.push(if rand::random() { 1 } else { 28 });
|
||||
packet.extend_from_slice(&[0, 1]);
|
||||
vec![packet]
|
||||
} else {
|
||||
let id = rand::random::<u64>();
|
||||
vec![format!("OPTIONS sip:n{id:x}@example.net SIP/2.0\r\nVia: SIP/2.0/UDP example.net;branch=z9hG4bK{id:x}\r\nMax-Forwards: 70\r\nFrom: <sip:a@example.net>;tag={id:x}\r\nTo: <sip:n{id:x}@example.net>\r\nCall-ID: {id:x}@example.net\r\nCSeq: 1 OPTIONS\r\nContent-Length: 0\r\n\r\n").into_bytes()]
|
||||
}
|
||||
}
|
||||
pub(crate) fn junk_packets(&self) -> Vec<Vec<u8>> {
|
||||
(0..self.junk_count)
|
||||
.map(|_| {
|
||||
let mut packet = vec![0; rand::random_range(self.junk_min..=self.junk_max).into()];
|
||||
rand::fill(&mut packet[..]);
|
||||
packet
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
|
||||
use super::*;
|
||||
#[test]
|
||||
fn unsafe_peer_profiles_are_rejected_before_the_codec() {
|
||||
let valid = PairProfile::generate();
|
||||
let mut bad = valid.clone();
|
||||
bad.prefix = 11;
|
||||
assert!(bad.validate().is_err());
|
||||
let mut bad = valid.clone();
|
||||
bad.headers[1] = bad.headers[0];
|
||||
assert!(bad.validate().is_err());
|
||||
let mut bad = valid.clone();
|
||||
bad.junk_max = u16::MAX;
|
||||
assert!(bad.validate().is_err());
|
||||
let mut bad = valid.clone();
|
||||
bad.padding = u16::MAX;
|
||||
assert!(bad.validate().is_err());
|
||||
let mut bad = valid.clone();
|
||||
bad.psk = [0; 32];
|
||||
assert!(bad.validate().is_err());
|
||||
let mut bad = valid.clone();
|
||||
bad.signature = 2;
|
||||
assert!(bad.validate().is_err());
|
||||
}
|
||||
#[test]
|
||||
fn signature_templates_have_consistent_framing_and_fresh_fields() {
|
||||
let mut profile = PairProfile::generate();
|
||||
profile.signature = 0;
|
||||
let dns = profile.signature_packets().remove(0);
|
||||
assert_eq!(&dns[2..12], &[1, 0, 0, 1, 0, 0, 0, 0, 0, 0]);
|
||||
let mut at = 12;
|
||||
while dns[at] != 0 {
|
||||
at += 1 + usize::from(dns[at]);
|
||||
}
|
||||
assert_eq!(at + 5, dns.len());
|
||||
assert_eq!(&dns[dns.len() - 2..], &[0, 1]);
|
||||
profile.signature = 1;
|
||||
let sip = profile.signature_packets().remove(0);
|
||||
assert!(sip.starts_with(b"OPTIONS sip:"));
|
||||
assert!(sip.ends_with(b"Content-Length: 0\r\n\r\n"));
|
||||
assert_ne!(sip, profile.signature_packets().remove(0));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,402 @@
|
||||
//! One cancellable actor and UDP socket per pair. All payload and health frames
|
||||
//! are inside the Amnezia-masked authenticated WireGuard tunnel.
|
||||
use crate::profile::PairProfile;
|
||||
use boringtun::noise::{Tunn, TunnResult};
|
||||
use bytes::Bytes;
|
||||
use iroh::EndpointId;
|
||||
use std::collections::{HashMap, VecDeque};
|
||||
use std::net::{Ipv4Addr, SocketAddr};
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::sync::{Arc, Mutex};
|
||||
use std::time::{Duration, Instant};
|
||||
use tokio::net::UdpSocket;
|
||||
use tokio::sync::{mpsc, watch};
|
||||
use tsunagi::dataplane::transport::{PacketLink, TransportError};
|
||||
use tsunagi::{BoxFuture, NetworkId};
|
||||
|
||||
pub(crate) const MAX_PAYLOAD: usize = tsunagi::config::MAX_DATA_DATAGRAM;
|
||||
const CHUNK: usize = 700;
|
||||
const QUEUE: usize = 64;
|
||||
const WINDOW: Duration = Duration::from_secs(20);
|
||||
|
||||
pub(crate) struct PairLink {
|
||||
network: NetworkId,
|
||||
peer: EndpointId,
|
||||
pub(crate) attempt: [u8; 16],
|
||||
tx: mpsc::Sender<Bytes>,
|
||||
rx: tokio::sync::Mutex<mpsc::Receiver<Bytes>>,
|
||||
stop: watch::Sender<bool>,
|
||||
closed: AtomicBool,
|
||||
remote: Mutex<Option<SocketAddr>>,
|
||||
reason: Mutex<Option<String>>,
|
||||
}
|
||||
impl std::fmt::Debug for PairLink {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
f.debug_struct("AwgPair")
|
||||
.field("network", &self.network)
|
||||
.field("peer", &self.peer)
|
||||
.finish_non_exhaustive()
|
||||
}
|
||||
}
|
||||
impl PairLink {
|
||||
pub(crate) fn has_authenticated_path(&self) -> bool {
|
||||
match self.remote.lock() {
|
||||
Ok(remote) => remote.is_some(),
|
||||
Err(_) => false,
|
||||
}
|
||||
}
|
||||
}
|
||||
impl PacketLink for PairLink {
|
||||
fn network(&self) -> NetworkId {
|
||||
self.network
|
||||
}
|
||||
fn peer(&self) -> EndpointId {
|
||||
self.peer
|
||||
}
|
||||
fn max_datagram_size(&self) -> usize {
|
||||
MAX_PAYLOAD
|
||||
}
|
||||
fn send(&self, payload: Bytes) -> Result<(), TransportError> {
|
||||
if self.is_closed() {
|
||||
return Err(TransportError::Closed);
|
||||
}
|
||||
if payload.len() > MAX_PAYLOAD {
|
||||
return Err(TransportError::TooLarge {
|
||||
size: payload.len(),
|
||||
limit: MAX_PAYLOAD,
|
||||
});
|
||||
}
|
||||
self.tx
|
||||
.try_send(payload)
|
||||
.map_err(|_| TransportError::Other("AWG send queue full or closed".into()))
|
||||
}
|
||||
fn recv(&self) -> BoxFuture<'_, Option<Bytes>> {
|
||||
Box::pin(async move {
|
||||
let mut stop = self.stop.subscribe();
|
||||
if *stop.borrow() {
|
||||
return None;
|
||||
}
|
||||
let mut rx = self.rx.lock().await;
|
||||
tokio::select! {biased; _=stop.changed()=>None,packet=rx.recv()=>packet}
|
||||
})
|
||||
}
|
||||
fn closed(&self) -> BoxFuture<'_, ()> {
|
||||
Box::pin(async move {
|
||||
let mut stop = self.stop.subscribe();
|
||||
if !*stop.borrow() {
|
||||
let _ = stop.changed().await;
|
||||
}
|
||||
})
|
||||
}
|
||||
fn is_closed(&self) -> bool {
|
||||
self.closed.load(Ordering::Acquire) || *self.stop.borrow()
|
||||
}
|
||||
fn close(&self) {
|
||||
self.stop.send_replace(true);
|
||||
}
|
||||
fn failure_reason(&self) -> Option<String> {
|
||||
match self.reason.lock() {
|
||||
Ok(reason) => reason.clone(),
|
||||
Err(_) => None,
|
||||
}
|
||||
}
|
||||
fn path_description(&self) -> String {
|
||||
match self.remote.lock() {
|
||||
Ok(addr) => match *addr {
|
||||
Some(addr) => format!("awg UDP via {addr}"),
|
||||
None => "awg UDP, establishing".into(),
|
||||
},
|
||||
Err(_) => "awg UDP".into(),
|
||||
}
|
||||
}
|
||||
}
|
||||
impl Drop for PairLink {
|
||||
fn drop(&mut self) {
|
||||
self.stop.send_replace(true);
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn start(
|
||||
network: NetworkId,
|
||||
peer: EndpointId,
|
||||
socket: UdpSocket,
|
||||
tunn: Tunn,
|
||||
profile: PairProfile,
|
||||
candidates: Vec<SocketAddr>,
|
||||
parent: watch::Receiver<bool>,
|
||||
) -> Arc<PairLink> {
|
||||
let (tx, rx) = mpsc::channel(QUEUE);
|
||||
let (in_tx, in_rx) = mpsc::channel(QUEUE);
|
||||
let (stop, _) = watch::channel(false);
|
||||
let link = Arc::new(PairLink {
|
||||
network,
|
||||
peer,
|
||||
attempt: profile.id(),
|
||||
tx,
|
||||
rx: tokio::sync::Mutex::new(in_rx),
|
||||
stop: stop.clone(),
|
||||
closed: AtomicBool::new(false),
|
||||
remote: Mutex::new(None),
|
||||
reason: Mutex::new(None),
|
||||
});
|
||||
let weak = Arc::downgrade(&link);
|
||||
tokio::spawn(async move {
|
||||
let mut engine = Engine {
|
||||
stop: stop.subscribe(),
|
||||
socket,
|
||||
tunn,
|
||||
profile,
|
||||
candidates,
|
||||
remote: None,
|
||||
inbox: in_tx,
|
||||
rx,
|
||||
counter: rand::random(),
|
||||
sequence: 0,
|
||||
pending: VecDeque::new(),
|
||||
samples: VecDeque::new(),
|
||||
assemblies: HashMap::new(),
|
||||
started: Instant::now(),
|
||||
last_authenticated: Instant::now(),
|
||||
};
|
||||
engine.run(parent, &weak).await;
|
||||
if let Some(link) = weak.upgrade() {
|
||||
link.closed.store(true, Ordering::Release);
|
||||
link.stop.send_replace(true);
|
||||
}
|
||||
});
|
||||
link
|
||||
}
|
||||
|
||||
struct Assembly {
|
||||
created: Instant,
|
||||
count: u16,
|
||||
chunks: Vec<Option<Bytes>>,
|
||||
}
|
||||
struct Engine {
|
||||
stop: watch::Receiver<bool>,
|
||||
socket: UdpSocket,
|
||||
tunn: Tunn,
|
||||
profile: PairProfile,
|
||||
candidates: Vec<SocketAddr>,
|
||||
remote: Option<SocketAddr>,
|
||||
inbox: mpsc::Sender<Bytes>,
|
||||
rx: mpsc::Receiver<Bytes>,
|
||||
counter: u64,
|
||||
sequence: u64,
|
||||
pending: VecDeque<(u64, Instant)>,
|
||||
samples: VecDeque<bool>,
|
||||
assemblies: HashMap<u64, Assembly>,
|
||||
started: Instant,
|
||||
last_authenticated: Instant,
|
||||
}
|
||||
enum ResultOwned {
|
||||
Wire(Vec<u8>),
|
||||
Plain(Vec<u8>),
|
||||
Done,
|
||||
}
|
||||
fn own(result: TunnResult<'_>) -> ResultOwned {
|
||||
match result {
|
||||
TunnResult::WriteToNetwork(p) => ResultOwned::Wire(p.to_vec()),
|
||||
TunnResult::WriteToTunnelV4(p, _) => ResultOwned::Plain(p.to_vec()),
|
||||
_ => ResultOwned::Done,
|
||||
}
|
||||
}
|
||||
fn ip(payload: &[u8]) -> Bytes {
|
||||
let mut packet = vec![0; 20];
|
||||
packet[0] = 0x45;
|
||||
packet[8] = 64;
|
||||
packet[9] = 253;
|
||||
packet[12..16].copy_from_slice(&Ipv4Addr::new(192, 0, 2, 1).octets());
|
||||
packet[16..20].copy_from_slice(&Ipv4Addr::new(192, 0, 2, 2).octets());
|
||||
packet[2..4].copy_from_slice(&((20 + payload.len()) as u16).to_be_bytes());
|
||||
let sum = packet
|
||||
.as_chunks::<2>()
|
||||
.0
|
||||
.iter()
|
||||
.map(|b| u32::from(u16::from_be_bytes([b[0], b[1]])))
|
||||
.sum::<u32>();
|
||||
let sum = (sum & 0xffff) + (sum >> 16);
|
||||
packet[10..12].copy_from_slice(&(!(sum as u16)).to_be_bytes());
|
||||
packet.extend_from_slice(payload);
|
||||
Bytes::from(packet)
|
||||
}
|
||||
impl Engine {
|
||||
async fn run(&mut self, mut stop: watch::Receiver<bool>, link: &std::sync::Weak<PairLink>) {
|
||||
if *self.stop.borrow() || *stop.borrow() {
|
||||
return;
|
||||
}
|
||||
let mut buffer = [0; 1201];
|
||||
let mut scratch = [0; 4096];
|
||||
let mut timer = tokio::time::interval(Duration::from_millis(250));
|
||||
timer.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
|
||||
let mut last_probe = Instant::now() - Duration::from_secs(2);
|
||||
loop {
|
||||
tokio::select! {
|
||||
biased;
|
||||
_=stop.changed()=>break,
|
||||
_=self.stop.changed()=>break,
|
||||
_=timer.tick()=>{
|
||||
let result=own(self.tunn.update_timers(&mut scratch));self.process(result,None).await;
|
||||
if last_probe.elapsed()>=Duration::from_secs(1) {
|
||||
last_probe=Instant::now();self.sequence=self.sequence.wrapping_add(1);
|
||||
self.pending.push_back((self.sequence,Instant::now()));
|
||||
let mut probe=vec![1];probe.extend_from_slice(&self.sequence.to_be_bytes());
|
||||
probe.resize(713,0);rand::fill(&mut probe[9..]);self.encrypt(&probe).await;
|
||||
}
|
||||
while self.pending.front().is_some_and(|(_,at)|at.elapsed()>=Duration::from_secs(3)) {
|
||||
self.pending.pop_front();self.samples.push_back(false);
|
||||
}
|
||||
while self.samples.len()>20 {self.samples.pop_front();}
|
||||
self.assemblies.retain(|_,a|a.created.elapsed()<Duration::from_secs(5));
|
||||
if self.started.elapsed()>WINDOW && (self.last_authenticated.elapsed()>Duration::from_secs(10)
|
||||
|| (self.samples.len()>=15&&self.samples.iter().filter(|ok|!**ok).count()*100>=self.samples.len()*60)) {
|
||||
if let Some(link)=link.upgrade() && let Ok(mut reason)=link.reason.lock() {*reason=Some("AWG data delivery checks: persistent loss or timeout".into());}
|
||||
break;
|
||||
}
|
||||
}
|
||||
packet=self.socket.recv_from(&mut buffer)=>{
|
||||
if let Ok((len,source))=packet && let Some(message)=self.profile.decode(&buffer[..len]) {
|
||||
let result=own(self.tunn.decapsulate(None,&message,&mut scratch));
|
||||
if matches!(result,ResultOwned::Plain(_)) {
|
||||
self.remote=Some(source);self.last_authenticated=Instant::now();
|
||||
if let Some(link)=link.upgrade() && let Ok(mut remote)=link.remote.lock(){*remote=Some(source);}
|
||||
}
|
||||
self.process(result,Some(source)).await;
|
||||
for _ in 0..16 {
|
||||
let result=own(self.tunn.decapsulate(None,&[],&mut scratch));
|
||||
if matches!(result,ResultOwned::Done){break;}
|
||||
self.process(result,Some(source)).await;
|
||||
}
|
||||
}
|
||||
}
|
||||
packet=self.rx.recv()=>{
|
||||
let Some(packet)=packet else{break;};
|
||||
let count=packet.len().max(1).div_ceil(CHUNK) as u16;
|
||||
self.counter=self.counter.wrapping_add(1);
|
||||
for (index,chunk) in packet.chunks(CHUNK).enumerate() {
|
||||
let mut payload=vec![3];payload.extend_from_slice(&self.counter.to_be_bytes());
|
||||
payload.extend_from_slice(&(index as u16).to_be_bytes());payload.extend_from_slice(&count.to_be_bytes());
|
||||
payload.extend_from_slice(chunk);self.encrypt(&payload).await;
|
||||
// Large logical datagrams must let local readers and other
|
||||
// pairs drain their bounded socket/command queues.
|
||||
if (index + 1).is_multiple_of(16) {tokio::task::yield_now().await;}
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
}
|
||||
}
|
||||
async fn encrypt(&mut self, payload: &[u8]) {
|
||||
let packet = self.profile.pad(ip(payload));
|
||||
let mut scratch = [0; 4096];
|
||||
let result = own(self.tunn.encapsulate(&packet, &mut scratch));
|
||||
self.process(result, None).await;
|
||||
}
|
||||
async fn wire(&self, raw: &[u8], target: Option<SocketAddr>) {
|
||||
let Some(wire) = self.profile.encode(raw) else {
|
||||
return;
|
||||
};
|
||||
let targets: Vec<_> = target
|
||||
.or(self.remote)
|
||||
.map(|a| vec![a])
|
||||
.unwrap_or_else(|| self.candidates.clone());
|
||||
for addr in targets {
|
||||
if raw.first() == Some(&1) {
|
||||
for signature in self.profile.signature_packets() {
|
||||
let _ = self.send_wire(&signature, addr).await;
|
||||
}
|
||||
for junk in self.profile.junk_packets() {
|
||||
let _ = self.send_wire(&junk, addr).await;
|
||||
}
|
||||
}
|
||||
let _ = self.send_wire(&wire, addr).await;
|
||||
}
|
||||
}
|
||||
async fn send_wire(&self, bytes: &[u8], address: SocketAddr) {
|
||||
let mut stop = self.stop.clone();
|
||||
if *stop.borrow() {
|
||||
return;
|
||||
}
|
||||
tokio::select! {biased; _=stop.changed()=>{}, _=tokio::time::timeout(Duration::from_millis(100),self.socket.send_to(bytes,address))=>{}}
|
||||
}
|
||||
async fn process(&mut self, result: ResultOwned, target: Option<SocketAddr>) {
|
||||
match result {
|
||||
ResultOwned::Wire(raw) => self.wire(&raw, target).await,
|
||||
ResultOwned::Plain(packet) => {
|
||||
let Some(payload) = packet.get(20..) else {
|
||||
return;
|
||||
};
|
||||
let Some(seq) = payload
|
||||
.get(1..9)
|
||||
.and_then(|b| b.try_into().ok())
|
||||
.map(u64::from_be_bytes)
|
||||
else {
|
||||
return;
|
||||
};
|
||||
match payload.first() {
|
||||
Some(1) => {
|
||||
let mut pong = vec![2];
|
||||
pong.extend_from_slice(&seq.to_be_bytes());
|
||||
pong.resize(713, 0);
|
||||
let packet = self.profile.pad(ip(&pong));
|
||||
let mut scratch = [0; 4096];
|
||||
if let ResultOwned::Wire(raw) =
|
||||
own(self.tunn.encapsulate(&packet, &mut scratch))
|
||||
{
|
||||
self.wire(&raw, target).await;
|
||||
}
|
||||
}
|
||||
Some(2) => {
|
||||
if let Some(index) =
|
||||
self.pending.iter().position(|(pending, _)| *pending == seq)
|
||||
{
|
||||
self.pending.remove(index);
|
||||
self.samples.push_back(true);
|
||||
}
|
||||
}
|
||||
Some(3) => self.fragment(seq, payload),
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
ResultOwned::Done => {}
|
||||
}
|
||||
}
|
||||
fn fragment(&mut self, id: u64, payload: &[u8]) {
|
||||
let Some(header) = payload.get(9..13) else {
|
||||
return;
|
||||
};
|
||||
let index = u16::from_be_bytes([header[0], header[1]]);
|
||||
let count = u16::from_be_bytes([header[2], header[3]]);
|
||||
let chunk = &payload[13..];
|
||||
if count == 0
|
||||
|| usize::from(count) > MAX_PAYLOAD.div_ceil(CHUNK)
|
||||
|| index >= count
|
||||
|| chunk.len() > CHUNK
|
||||
{
|
||||
return;
|
||||
}
|
||||
if !self.assemblies.contains_key(&id) && self.assemblies.len() >= 8 {
|
||||
return;
|
||||
}
|
||||
let assembly = self.assemblies.entry(id).or_insert_with(|| Assembly {
|
||||
created: Instant::now(),
|
||||
count,
|
||||
chunks: vec![None; count.into()],
|
||||
});
|
||||
if assembly.count != count {
|
||||
return;
|
||||
}
|
||||
assembly.chunks[usize::from(index)] = Some(Bytes::copy_from_slice(chunk));
|
||||
if assembly.chunks.iter().all(Option::is_some)
|
||||
&& let Some(assembly) = self.assemblies.remove(&id)
|
||||
{
|
||||
let mut packet = Vec::new();
|
||||
for chunk in assembly.chunks.into_iter().flatten() {
|
||||
packet.extend_from_slice(&chunk);
|
||||
}
|
||||
if packet.len() <= MAX_PAYLOAD {
|
||||
let _ = self.inbox.try_send(Bytes::from(packet));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,101 @@
|
||||
//! Optional, bounded server-reflexive discovery on the pair's own UDP socket.
|
||||
//! A STUN answer is a candidate only; WireGuard authentication verifies a path.
|
||||
use std::net::{IpAddr, Ipv4Addr, SocketAddr};
|
||||
use std::time::Duration;
|
||||
use tokio::net::UdpSocket;
|
||||
const COOKIE: [u8; 4] = [0x21, 0x12, 0xa4, 0x42];
|
||||
pub(crate) async fn mapped(socket: &UdpSocket, server: &str) -> Option<SocketAddr> {
|
||||
tokio::time::timeout(Duration::from_millis(1200), async {
|
||||
let address = tokio::net::lookup_host(server)
|
||||
.await
|
||||
.ok()?
|
||||
.find(SocketAddr::is_ipv4)?;
|
||||
let transaction: [u8; 12] = rand::random();
|
||||
let mut request = [0; 20];
|
||||
request[..2].copy_from_slice(&[0, 1]);
|
||||
request[4..8].copy_from_slice(&COOKIE);
|
||||
request[8..].copy_from_slice(&transaction);
|
||||
socket.send_to(&request, address).await.ok()?;
|
||||
let mut bytes = [0; 1024];
|
||||
loop {
|
||||
let (len, source) = socket.recv_from(&mut bytes).await.ok()?;
|
||||
if source == address
|
||||
&& let Some(mapped) = parse(&bytes[..len], &transaction)
|
||||
{
|
||||
return Some(mapped);
|
||||
}
|
||||
}
|
||||
})
|
||||
.await
|
||||
.ok()
|
||||
.flatten()
|
||||
}
|
||||
fn parse(bytes: &[u8], transaction: &[u8; 12]) -> Option<SocketAddr> {
|
||||
if bytes.len() < 20
|
||||
|| bytes[..2] != [1, 1]
|
||||
|| bytes[4..8] != COOKIE
|
||||
|| &bytes[8..20] != transaction
|
||||
{
|
||||
return None;
|
||||
}
|
||||
let declared = u16::from_be_bytes(bytes[2..4].try_into().ok()?) as usize;
|
||||
if !declared.is_multiple_of(4) || declared != bytes.len() - 20 {
|
||||
return None;
|
||||
}
|
||||
let mut at = 20;
|
||||
while at + 4 <= bytes.len() {
|
||||
let kind = u16::from_be_bytes(bytes[at..at + 2].try_into().ok()?);
|
||||
let len = u16::from_be_bytes(bytes[at + 2..at + 4].try_into().ok()?) as usize;
|
||||
at += 4;
|
||||
let attribute = bytes.get(at..at.checked_add(len)?)?;
|
||||
if kind == 0x0020 && attribute.len() == 8 && attribute[0..2] == [0, 1] {
|
||||
let port = u16::from_be_bytes([attribute[2] ^ COOKIE[0], attribute[3] ^ COOKIE[1]]);
|
||||
let ip = Ipv4Addr::new(
|
||||
attribute[4] ^ COOKIE[0],
|
||||
attribute[5] ^ COOKIE[1],
|
||||
attribute[6] ^ COOKIE[2],
|
||||
attribute[7] ^ COOKIE[3],
|
||||
);
|
||||
if port == 0 || ip.is_unspecified() || ip.is_multicast() || ip.is_broadcast() {
|
||||
return None;
|
||||
}
|
||||
return Some(SocketAddr::new(IpAddr::V4(ip), port));
|
||||
}
|
||||
at = at.checked_add((len + 3) & !3)?;
|
||||
}
|
||||
None
|
||||
}
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
|
||||
use super::*;
|
||||
#[tokio::test]
|
||||
async fn mapping_uses_the_actual_socket_and_rejects_unrelated_or_truncated_answers() {
|
||||
let server = UdpSocket::bind((Ipv4Addr::LOCALHOST, 0)).await.unwrap();
|
||||
let address = server.local_addr().unwrap();
|
||||
let socket = UdpSocket::bind((Ipv4Addr::LOCALHOST, 0)).await.unwrap();
|
||||
let expected = socket.local_addr().unwrap();
|
||||
let task = tokio::spawn(async move {
|
||||
let mut request = [0; 128];
|
||||
let (len, source) = server.recv_from(&mut request).await.unwrap();
|
||||
assert_eq!(len, 20);
|
||||
assert_eq!(source, expected);
|
||||
let tx: [u8; 12] = request[8..20].try_into().unwrap();
|
||||
let mut reply = request[..20].to_vec();
|
||||
reply[..4].copy_from_slice(&[1, 1, 0, 12]);
|
||||
reply.extend_from_slice(&[0, 0x20, 0, 8, 0, 1]);
|
||||
let port = source.port().to_be_bytes();
|
||||
reply.extend_from_slice(&[port[0] ^ COOKIE[0], port[1] ^ COOKIE[1]]);
|
||||
for (i, b) in Ipv4Addr::LOCALHOST.octets().into_iter().enumerate() {
|
||||
reply.push(b ^ COOKIE[i]);
|
||||
}
|
||||
assert!(parse(&reply, &[0; 12]).is_none());
|
||||
for len in 0..reply.len() {
|
||||
assert!(parse(&reply[..len], &tx).is_none());
|
||||
}
|
||||
server.send_to(&reply, source).await.unwrap();
|
||||
});
|
||||
assert_eq!(mapped(&socket, &address.to_string()).await, Some(expected));
|
||||
task.await.unwrap();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,720 @@
|
||||
//! Private pair negotiation. No secret profiles in capabilities or durable state.
|
||||
use crate::{
|
||||
AWG_PROTOCOL, AWG_WIRE_VERSION,
|
||||
profile::PairProfile,
|
||||
session::{self, PairLink},
|
||||
};
|
||||
use boringtun::{
|
||||
noise::Tunn,
|
||||
x25519::{PublicKey, StaticSecret},
|
||||
};
|
||||
use iroh::EndpointId;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::collections::{HashMap, HashSet};
|
||||
use std::net::{Ipv4Addr, SocketAddr};
|
||||
use std::sync::{Arc, Mutex, OnceLock, Weak};
|
||||
use std::time::Duration;
|
||||
use tokio::net::UdpSocket;
|
||||
use tokio::sync::{mpsc, oneshot, watch};
|
||||
use tsunagi::dataplane::transport::TransportPayload;
|
||||
use tsunagi::dataplane::transport::{
|
||||
ControlSender, InboundLink, PacketLink, PacketTransport, SharedLink, TransportError,
|
||||
};
|
||||
use tsunagi::{BoxFuture, NetworkId};
|
||||
|
||||
type Pair = (NetworkId, EndpointId);
|
||||
#[derive(Serialize, Deserialize)]
|
||||
enum Negotiation {
|
||||
Offer {
|
||||
version: u16,
|
||||
profile: PairProfile,
|
||||
key: [u8; 32],
|
||||
#[serde(deserialize_with = "bounded_addresses")]
|
||||
addresses: Vec<SocketAddr>,
|
||||
},
|
||||
Answer {
|
||||
attempt: [u8; 16],
|
||||
key: [u8; 32],
|
||||
#[serde(deserialize_with = "bounded_addresses")]
|
||||
addresses: Vec<SocketAddr>,
|
||||
},
|
||||
}
|
||||
fn bounded_addresses<'de, D: serde::Deserializer<'de>>(
|
||||
deserializer: D,
|
||||
) -> Result<Vec<SocketAddr>, D::Error> {
|
||||
struct Visitor;
|
||||
impl<'de> serde::de::Visitor<'de> for Visitor {
|
||||
type Value = Vec<SocketAddr>;
|
||||
fn expecting(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
f.write_str("at most 32 UDP candidates")
|
||||
}
|
||||
fn visit_seq<A: serde::de::SeqAccess<'de>>(
|
||||
self,
|
||||
mut sequence: A,
|
||||
) -> Result<Self::Value, A::Error> {
|
||||
if sequence.size_hint().is_some_and(|n| n > 32) {
|
||||
return Err(serde::de::Error::custom("too many UDP candidates"));
|
||||
}
|
||||
let mut addresses = Vec::with_capacity(sequence.size_hint().unwrap_or(0).min(32));
|
||||
while let Some(address) = sequence.next_element()? {
|
||||
if addresses.len() == 32 {
|
||||
return Err(serde::de::Error::custom("too many UDP candidates"));
|
||||
}
|
||||
addresses.push(address);
|
||||
}
|
||||
Ok(addresses)
|
||||
}
|
||||
}
|
||||
deserializer.deserialize_seq(Visitor)
|
||||
}
|
||||
struct Pending {
|
||||
profile: PairProfile,
|
||||
key: StaticSecret,
|
||||
socket: UdpSocket,
|
||||
reply: oneshot::Sender<Result<SharedLink, TransportError>>,
|
||||
}
|
||||
struct Inner {
|
||||
stopped: watch::Sender<bool>,
|
||||
local: EndpointId,
|
||||
bind: Ipv4Addr,
|
||||
stun: Option<String>,
|
||||
busy: Mutex<HashSet<Pair>>,
|
||||
control: OnceLock<Arc<dyn ControlSender>>,
|
||||
allowed: Mutex<HashSet<Pair>>,
|
||||
pending: Mutex<HashMap<Pair, Pending>>,
|
||||
links: Mutex<HashMap<Pair, Arc<PairLink>>>,
|
||||
inbound: mpsc::Sender<InboundLink>,
|
||||
port_by_pair: Mutex<HashMap<Pair, u16>>,
|
||||
}
|
||||
/// Public diagnostics for one pair. No key or masking parameters are exposed.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct PairOverview {
|
||||
/// Other endpoint's authenticated identity.
|
||||
pub peer: EndpointId,
|
||||
/// Fresh connection-attempt identifier.
|
||||
pub attempt: [u8; 16],
|
||||
/// Last known UDP port; None if diagnostics race with pair retirement.
|
||||
pub local_port: Option<u16>,
|
||||
/// Actual transport path, if established.
|
||||
pub path: String,
|
||||
/// Whether this attempt has been retired.
|
||||
pub closed: bool,
|
||||
}
|
||||
/// A bounded set of independently configured pair engines.
|
||||
pub struct AwgTransport {
|
||||
inner: Arc<Inner>,
|
||||
commands: mpsc::Sender<(NetworkId, EndpointId, Negotiation)>,
|
||||
incoming: tokio::sync::Mutex<mpsc::Receiver<InboundLink>>,
|
||||
}
|
||||
impl std::fmt::Debug for AwgTransport {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
f.debug_struct("AwgTransport")
|
||||
.field("local", &self.inner.local)
|
||||
.finish_non_exhaustive()
|
||||
}
|
||||
}
|
||||
fn lock<T>(mutex: &Mutex<T>) -> std::sync::MutexGuard<'_, T> {
|
||||
match mutex.lock() {
|
||||
Ok(g) => g,
|
||||
Err(p) => p.into_inner(),
|
||||
}
|
||||
}
|
||||
fn error(text: &str) -> TransportError {
|
||||
TransportError::Other(text.into())
|
||||
}
|
||||
async fn addresses(inner: &Inner, socket: &UdpSocket) -> Result<Vec<SocketAddr>, TransportError> {
|
||||
let local = socket
|
||||
.local_addr()
|
||||
.map_err(|_| error("AWG socket address unavailable"))?;
|
||||
let mut candidates = if inner.bind.is_loopback() {
|
||||
vec![local]
|
||||
} else {
|
||||
tsunagi_wireguard::local_ip_candidates()
|
||||
.into_iter()
|
||||
.filter(|ip| ip.is_ipv4())
|
||||
.take(31)
|
||||
.map(|ip| SocketAddr::new(ip, local.port()))
|
||||
.collect()
|
||||
};
|
||||
if let Some(server) = &inner.stun
|
||||
&& let Some(mapped) = crate::stun::mapped(socket, server).await
|
||||
&& !candidates.contains(&mapped)
|
||||
{
|
||||
candidates.push(mapped);
|
||||
}
|
||||
if candidates.is_empty() {
|
||||
return Err(error("AWG has no UDP address candidates"));
|
||||
}
|
||||
Ok(candidates)
|
||||
}
|
||||
fn validate_endpoint(key: &[u8; 32], addresses: &[SocketAddr]) -> Result<(), TransportError> {
|
||||
if *key == [0; 32]
|
||||
|| addresses.is_empty()
|
||||
|| addresses.len() > 32
|
||||
|| addresses.iter().any(|address| {
|
||||
address.port() == 0
|
||||
|| !address.is_ipv4()
|
||||
|| address.ip().is_unspecified()
|
||||
|| address.ip().is_multicast()
|
||||
|| matches!(address.ip(), std::net::IpAddr::V4(ip) if ip.is_broadcast())
|
||||
})
|
||||
{
|
||||
return Err(error("invalid AWG endpoint"));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
impl AwgTransport {
|
||||
/// Starts the coordinator. Production binds wildcard IPv4; tests use loopback.
|
||||
pub fn new(local: EndpointId, bind: Ipv4Addr) -> Arc<Self> {
|
||||
Self::new_with_stun(local, bind, None)
|
||||
}
|
||||
/// Enables an optional STUN server (host:port) for this pair's own socket.
|
||||
/// Local-only callers pass None; failure still permits local UDP candidates.
|
||||
pub fn new_with_stun(local: EndpointId, bind: Ipv4Addr, stun: Option<String>) -> Arc<Self> {
|
||||
let (commands, mut rx) = mpsc::channel(64);
|
||||
let (inbound, incoming) = mpsc::channel(64);
|
||||
let (stopped, _) = watch::channel(false);
|
||||
let inner = Arc::new(Inner {
|
||||
stopped,
|
||||
local,
|
||||
bind,
|
||||
stun,
|
||||
busy: Mutex::default(),
|
||||
control: OnceLock::new(),
|
||||
allowed: Mutex::default(),
|
||||
pending: Mutex::default(),
|
||||
links: Mutex::default(),
|
||||
inbound,
|
||||
port_by_pair: Mutex::default(),
|
||||
});
|
||||
let weak = Arc::downgrade(&inner);
|
||||
let mut stopped = inner.stopped.subscribe();
|
||||
tokio::spawn(async move {
|
||||
while let Some((network, peer, message)) =
|
||||
tokio::select! { biased; _=stopped.changed()=>None, message=rx.recv()=>message }
|
||||
{
|
||||
let Some(inner) = weak.upgrade() else {
|
||||
break;
|
||||
};
|
||||
if !lock(&inner.allowed).contains(&(network, peer))
|
||||
|| !lock(&inner.busy).insert((network, peer))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
tokio::spawn(async move {
|
||||
let mut stop = inner.stopped.subscribe();
|
||||
tokio::select! {biased; _=stop.changed()=>{}, _=async {
|
||||
if let Err(err)=receive(&inner,(network,peer),message).await {tracing::debug!(%err,"AWG pair negotiation rejected");}
|
||||
}=>{}}
|
||||
lock(&inner.busy).remove(&(network, peer));
|
||||
});
|
||||
}
|
||||
});
|
||||
Arc::new(Self {
|
||||
inner,
|
||||
commands,
|
||||
incoming: tokio::sync::Mutex::new(incoming),
|
||||
})
|
||||
}
|
||||
/// Registers a peer after its authenticated capability has been validated.
|
||||
pub fn allow_peer(&self, network: NetworkId, peer: EndpointId) {
|
||||
let mut allowed = lock(&self.inner.allowed);
|
||||
if !*self.inner.stopped.borrow() && allowed.len() < 128 {
|
||||
allowed.insert((network, peer));
|
||||
}
|
||||
}
|
||||
/// Retires a pair when its network membership session goes away.
|
||||
pub fn forget_peer(&self, network: NetworkId, peer: EndpointId) {
|
||||
let pair = (network, peer);
|
||||
lock(&self.inner.allowed).remove(&pair);
|
||||
lock(&self.inner.pending).remove(&pair);
|
||||
lock(&self.inner.port_by_pair).remove(&pair);
|
||||
if let Some(link) = lock(&self.inner.links).remove(&pair) {
|
||||
link.close();
|
||||
}
|
||||
}
|
||||
/// Stops all pair engines in this network.
|
||||
pub fn stop_network(&self, network: NetworkId) {
|
||||
let peers: Vec<_> = lock(&self.inner.allowed)
|
||||
.iter()
|
||||
.filter(|(n, _)| *n == network)
|
||||
.map(|(_, p)| *p)
|
||||
.collect();
|
||||
for peer in peers {
|
||||
self.forget_peer(network, peer);
|
||||
}
|
||||
}
|
||||
/// Stops all sockets and coordinator work without waiting for remote peers.
|
||||
pub fn shutdown(&self) {
|
||||
self.inner.stopped.send_replace(true);
|
||||
lock(&self.inner.allowed).clear();
|
||||
lock(&self.inner.pending).clear();
|
||||
for (_, link) in lock(&self.inner.links).drain() {
|
||||
link.close();
|
||||
}
|
||||
lock(&self.inner.port_by_pair).clear();
|
||||
}
|
||||
/// Receives a successfully negotiated and authenticated inbound pair link.
|
||||
pub async fn next_inbound(&self) -> Option<InboundLink> {
|
||||
let mut stopped = self.inner.stopped.subscribe();
|
||||
if *stopped.borrow() {
|
||||
return None;
|
||||
}
|
||||
let mut incoming = self.incoming.lock().await;
|
||||
tokio::select! {biased; _=stopped.changed()=>None, link=incoming.recv()=>link}
|
||||
}
|
||||
/// Takes a public snapshot of this network's pair attempts.
|
||||
pub fn overview(&self, network: NetworkId) -> Vec<PairOverview> {
|
||||
let links = lock(&self.inner.links);
|
||||
let ports = lock(&self.inner.port_by_pair);
|
||||
links
|
||||
.iter()
|
||||
.filter(|((n, _), _)| *n == network)
|
||||
.map(|((_, peer), link)| PairOverview {
|
||||
peer: *peer,
|
||||
attempt: link.attempt,
|
||||
local_port: ports.get(&(network, *peer)).copied(),
|
||||
path: link.path_description(),
|
||||
closed: link.is_closed(),
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
}
|
||||
impl PacketTransport for AwgTransport {
|
||||
fn name(&self) -> &str {
|
||||
AWG_PROTOCOL
|
||||
}
|
||||
fn as_any(&self) -> &dyn std::any::Any {
|
||||
self
|
||||
}
|
||||
fn attach_control(&self, control: Arc<dyn ControlSender>) {
|
||||
let _ = self.inner.control.set(control);
|
||||
}
|
||||
fn on_control(
|
||||
&self,
|
||||
network: NetworkId,
|
||||
peer: EndpointId,
|
||||
protocol: &str,
|
||||
payload: &[u8],
|
||||
) -> Result<(), TransportError> {
|
||||
if protocol != AWG_PROTOCOL || !lock(&self.inner.allowed).contains(&(network, peer)) {
|
||||
return Err(TransportError::Declined(protocol.into()));
|
||||
}
|
||||
if payload.len() > 4096 {
|
||||
return Err(error("AWG negotiation too large"));
|
||||
}
|
||||
let message: Negotiation =
|
||||
postcard::from_bytes(payload).map_err(|_| error("malformed AWG negotiation"))?;
|
||||
match &message {
|
||||
Negotiation::Offer {
|
||||
version,
|
||||
profile,
|
||||
key,
|
||||
addresses,
|
||||
} => {
|
||||
if *version != AWG_WIRE_VERSION || peer.as_bytes() >= self.inner.local.as_bytes() {
|
||||
return Err(error("invalid AWG offer role/version"));
|
||||
}
|
||||
profile.validate().map_err(error)?;
|
||||
validate_endpoint(key, addresses)?;
|
||||
}
|
||||
Negotiation::Answer { key, addresses, .. } => validate_endpoint(key, addresses)?,
|
||||
}
|
||||
self.commands
|
||||
.try_send((network, peer, message))
|
||||
.map_err(|_| error("AWG negotiation queue full"))
|
||||
}
|
||||
fn open<'a>(
|
||||
&'a self,
|
||||
network: NetworkId,
|
||||
peer: EndpointId,
|
||||
protocol: &'a str,
|
||||
) -> BoxFuture<'a, Result<SharedLink, TransportError>> {
|
||||
Box::pin(async move {
|
||||
let pair = (network, peer);
|
||||
if protocol != AWG_PROTOCOL
|
||||
|| self.inner.local.as_bytes() >= peer.as_bytes()
|
||||
|| !lock(&self.inner.allowed).contains(&pair)
|
||||
{
|
||||
return Err(TransportError::Declined(protocol.into()));
|
||||
}
|
||||
let control = self
|
||||
.inner
|
||||
.control
|
||||
.get()
|
||||
.ok_or_else(|| error("AWG control plane unavailable"))?;
|
||||
let socket = UdpSocket::bind((self.inner.bind, 0))
|
||||
.await
|
||||
.map_err(|_| error("AWG UDP bind failed"))?;
|
||||
let profile = PairProfile::generate();
|
||||
let attempt = profile.id();
|
||||
let key = StaticSecret::from(rand::random::<[u8; 32]>());
|
||||
let public = PublicKey::from(&key).to_bytes();
|
||||
let message = Negotiation::Offer {
|
||||
version: AWG_WIRE_VERSION,
|
||||
profile: profile.clone(),
|
||||
key: public,
|
||||
addresses: addresses(&self.inner, &socket).await?,
|
||||
};
|
||||
let (reply, rx) = oneshot::channel();
|
||||
if lock(&self.inner.pending).len() >= 128 {
|
||||
return Err(error("too many AWG pair negotiations"));
|
||||
}
|
||||
lock(&self.inner.pending).insert(
|
||||
pair,
|
||||
Pending {
|
||||
profile,
|
||||
key,
|
||||
socket,
|
||||
reply,
|
||||
},
|
||||
);
|
||||
// A cancellation guard also retires partially started attempts.
|
||||
let mut guard = AttemptGuard {
|
||||
inner: Arc::downgrade(&self.inner),
|
||||
pair,
|
||||
attempt,
|
||||
completed: false,
|
||||
};
|
||||
let bytes =
|
||||
postcard::to_stdvec(&message).map_err(|_| error("AWG offer encoding failed"))?;
|
||||
control
|
||||
.send(network, peer, AWG_PROTOCOL, TransportPayload::new(bytes))
|
||||
.await?;
|
||||
let result = tokio::time::timeout(Duration::from_secs(15), rx)
|
||||
.await
|
||||
.map_err(|_| error("AWG negotiation/handshake timed out"))?
|
||||
.map_err(|_| error("AWG attempt cancelled"))?;
|
||||
guard.completed = result.is_ok();
|
||||
result
|
||||
})
|
||||
}
|
||||
}
|
||||
struct AttemptGuard {
|
||||
inner: Weak<Inner>,
|
||||
pair: Pair,
|
||||
attempt: [u8; 16],
|
||||
completed: bool,
|
||||
}
|
||||
impl Drop for AttemptGuard {
|
||||
fn drop(&mut self) {
|
||||
if let Some(inner) = self.inner.upgrade() {
|
||||
let mut pending = lock(&inner.pending);
|
||||
if pending
|
||||
.get(&self.pair)
|
||||
.is_some_and(|p| p.profile.id() == self.attempt)
|
||||
{
|
||||
pending.remove(&self.pair);
|
||||
}
|
||||
drop(pending);
|
||||
if !self.completed
|
||||
&& let Some(link) = lock(&inner.links).get(&self.pair)
|
||||
&& link.attempt == self.attempt
|
||||
{
|
||||
link.close();
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
async fn receive(
|
||||
inner: &Arc<Inner>,
|
||||
pair: Pair,
|
||||
message: Negotiation,
|
||||
) -> Result<(), TransportError> {
|
||||
if !lock(&inner.allowed).contains(&pair) {
|
||||
return Err(error("AWG peer no longer authenticated"));
|
||||
}
|
||||
match message {
|
||||
Negotiation::Offer {
|
||||
profile,
|
||||
key,
|
||||
addresses: candidates,
|
||||
..
|
||||
} => {
|
||||
if let Some(link) = lock(&inner.links).get(&pair) {
|
||||
if link.attempt == profile.id() {
|
||||
return Err(error("duplicate AWG offer"));
|
||||
}
|
||||
link.close();
|
||||
}
|
||||
let socket = UdpSocket::bind((inner.bind, 0))
|
||||
.await
|
||||
.map_err(|_| error("AWG UDP bind failed"))?;
|
||||
let local_port = socket
|
||||
.local_addr()
|
||||
.map_err(|_| error("AWG socket address unavailable"))?
|
||||
.port();
|
||||
let local_key = StaticSecret::from(rand::random::<[u8; 32]>());
|
||||
let public = PublicKey::from(&local_key).to_bytes();
|
||||
let answer = Negotiation::Answer {
|
||||
attempt: profile.id(),
|
||||
key: public,
|
||||
addresses: addresses(inner, &socket).await?,
|
||||
};
|
||||
let tunn = Tunn::new(
|
||||
local_key,
|
||||
PublicKey::from(key),
|
||||
Some(profile.psk),
|
||||
Some(1),
|
||||
rand::random(),
|
||||
None,
|
||||
);
|
||||
|
||||
let link = session::start(
|
||||
pair.0,
|
||||
pair.1,
|
||||
socket,
|
||||
tunn,
|
||||
profile,
|
||||
candidates,
|
||||
inner.stopped.subscribe(),
|
||||
);
|
||||
if !lock(&inner.allowed).contains(&pair) {
|
||||
link.close();
|
||||
return Err(error("AWG membership ended"));
|
||||
}
|
||||
lock(&inner.links).insert(pair, link.clone());
|
||||
lock(&inner.port_by_pair).insert(pair, local_port);
|
||||
let control = inner
|
||||
.control
|
||||
.get()
|
||||
.ok_or_else(|| error("AWG control unavailable"))?;
|
||||
if let Err(err) = control
|
||||
.send(
|
||||
pair.0,
|
||||
pair.1,
|
||||
AWG_PROTOCOL,
|
||||
TransportPayload::new(
|
||||
postcard::to_stdvec(&answer)
|
||||
.map_err(|_| error("AWG answer encoding failed"))?,
|
||||
),
|
||||
)
|
||||
.await
|
||||
{
|
||||
link.close();
|
||||
return Err(err);
|
||||
}
|
||||
publish_when_up(Arc::downgrade(inner), pair, link, None);
|
||||
}
|
||||
Negotiation::Answer {
|
||||
attempt,
|
||||
key,
|
||||
addresses: candidates,
|
||||
} => {
|
||||
if !lock(&inner.pending)
|
||||
.get(&pair)
|
||||
.is_some_and(|p| p.profile.id() == attempt)
|
||||
{
|
||||
return Err(error("stale AWG answer"));
|
||||
}
|
||||
let pending = lock(&inner.pending)
|
||||
.remove(&pair)
|
||||
.ok_or_else(|| error("AWG attempt cancelled"))?;
|
||||
let local_port = pending
|
||||
.socket
|
||||
.local_addr()
|
||||
.map_err(|_| error("AWG socket address unavailable"))?
|
||||
.port();
|
||||
let tunn = Tunn::new(
|
||||
pending.key,
|
||||
PublicKey::from(key),
|
||||
Some(pending.profile.psk),
|
||||
Some(1),
|
||||
rand::random(),
|
||||
None,
|
||||
);
|
||||
|
||||
let link = session::start(
|
||||
pair.0,
|
||||
pair.1,
|
||||
pending.socket,
|
||||
tunn,
|
||||
pending.profile,
|
||||
candidates,
|
||||
inner.stopped.subscribe(),
|
||||
);
|
||||
if !lock(&inner.allowed).contains(&pair) {
|
||||
link.close();
|
||||
return Err(error("AWG membership ended"));
|
||||
}
|
||||
lock(&inner.links).insert(pair, link.clone());
|
||||
lock(&inner.port_by_pair).insert(pair, local_port);
|
||||
publish_when_up(Arc::downgrade(inner), pair, link, Some(pending.reply));
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
fn publish_when_up(
|
||||
inner: Weak<Inner>,
|
||||
pair: Pair,
|
||||
link: Arc<PairLink>,
|
||||
reply: Option<oneshot::Sender<Result<SharedLink, TransportError>>>,
|
||||
) {
|
||||
tokio::spawn(async move {
|
||||
let mut reply = reply;
|
||||
let result = tokio::time::timeout(Duration::from_secs(14), async {
|
||||
loop {
|
||||
if link.is_closed() || reply.as_ref().is_some_and(oneshot::Sender::is_closed) {
|
||||
return Err(error("AWG pair closed during handshake"));
|
||||
}
|
||||
if link.has_authenticated_path() {
|
||||
return Ok(());
|
||||
}
|
||||
tokio::time::sleep(Duration::from_millis(25)).await;
|
||||
}
|
||||
})
|
||||
.await;
|
||||
if !matches!(result, Ok(Ok(()))) {
|
||||
link.close();
|
||||
if let Some(reply) = reply.take() {
|
||||
let _ = reply.send(Err(error("AWG authenticated data path timed out")));
|
||||
}
|
||||
return;
|
||||
}
|
||||
let Some(inner) = inner.upgrade() else {
|
||||
link.close();
|
||||
return;
|
||||
};
|
||||
if !lock(&inner.allowed).contains(&pair) {
|
||||
link.close();
|
||||
return;
|
||||
}
|
||||
if link.is_closed()
|
||||
|| !lock(&inner.links)
|
||||
.get(&pair)
|
||||
.is_some_and(|current| current.attempt == link.attempt)
|
||||
{
|
||||
link.close();
|
||||
return;
|
||||
}
|
||||
let shared: SharedLink = link.clone();
|
||||
if let Some(reply) = reply {
|
||||
if reply.send(Ok(shared)).is_err() {
|
||||
link.close();
|
||||
}
|
||||
} else if inner
|
||||
.inbound
|
||||
.try_send(InboundLink {
|
||||
network: pair.0,
|
||||
peer: pair.1,
|
||||
protocol: AWG_PROTOCOL.into(),
|
||||
link: shared,
|
||||
})
|
||||
.is_err()
|
||||
{
|
||||
link.close();
|
||||
}
|
||||
});
|
||||
}
|
||||
impl Drop for AwgTransport {
|
||||
fn drop(&mut self) {
|
||||
self.shutdown();
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
|
||||
use super::*;
|
||||
#[tokio::test]
|
||||
async fn concurrent_admission_is_bounded_across_networks_and_retirement_releases_slots() {
|
||||
let identity = tsunagi::identity::DeviceIdentity::generate();
|
||||
let transport = AwgTransport::new(identity.endpoint_id(), Ipv4Addr::LOCALHOST);
|
||||
let peer = tsunagi::identity::DeviceIdentity::generate().endpoint_id();
|
||||
std::thread::scope(|scope| {
|
||||
for thread in 0..8u8 {
|
||||
let transport = &transport;
|
||||
scope.spawn(move || {
|
||||
for item in 0..32u8 {
|
||||
let mut id = [0; 32];
|
||||
id[0] = thread;
|
||||
id[1] = item;
|
||||
transport.allow_peer(NetworkId::from_bytes(id), peer);
|
||||
}
|
||||
});
|
||||
}
|
||||
});
|
||||
assert_eq!(lock(&transport.inner.allowed).len(), 128);
|
||||
let network = lock(&transport.inner.allowed).iter().next().unwrap().0;
|
||||
transport.stop_network(network);
|
||||
let new = NetworkId::from_bytes([255; 32]);
|
||||
transport.allow_peer(new, peer);
|
||||
assert!(lock(&transport.inner.allowed).contains(&(new, peer)));
|
||||
assert_eq!(lock(&transport.inner.allowed).len(), 128);
|
||||
transport.shutdown();
|
||||
assert!(lock(&transport.inner.allowed).is_empty());
|
||||
}
|
||||
#[tokio::test]
|
||||
async fn private_negotiation_bounds_roles_and_membership_before_starting_engines() {
|
||||
let mut ids = [
|
||||
tsunagi::identity::DeviceIdentity::generate().endpoint_id(),
|
||||
tsunagi::identity::DeviceIdentity::generate().endpoint_id(),
|
||||
];
|
||||
ids.sort_by_key(|id| *id.as_bytes());
|
||||
let peer = ids[0];
|
||||
let local = ids[1];
|
||||
let network = NetworkId::from_bytes([7; 32]);
|
||||
let transport = AwgTransport::new(local, Ipv4Addr::LOCALHOST);
|
||||
let offer = |version, key, addresses| Negotiation::Offer {
|
||||
version,
|
||||
profile: PairProfile::generate(),
|
||||
key,
|
||||
addresses,
|
||||
};
|
||||
let address = SocketAddr::from((Ipv4Addr::LOCALHOST, 12345));
|
||||
let encode = |message: &Negotiation| postcard::to_stdvec(message).unwrap();
|
||||
let valid = encode(&offer(AWG_WIRE_VERSION, [1; 32], vec![address]));
|
||||
assert!(
|
||||
transport
|
||||
.on_control(network, peer, AWG_PROTOCOL, &valid)
|
||||
.is_err()
|
||||
);
|
||||
transport.allow_peer(network, peer);
|
||||
assert!(
|
||||
transport
|
||||
.on_control(network, peer, "wrong-protocol", &valid)
|
||||
.is_err()
|
||||
);
|
||||
assert!(
|
||||
transport
|
||||
.on_control(network, peer, AWG_PROTOCOL, &[0; 4097])
|
||||
.is_err()
|
||||
);
|
||||
for bytes in [
|
||||
encode(&offer(99, [1; 32], vec![address])),
|
||||
encode(&offer(AWG_WIRE_VERSION, [0; 32], vec![address])),
|
||||
encode(&offer(AWG_WIRE_VERSION, [1; 32], vec![])),
|
||||
encode(&offer(AWG_WIRE_VERSION, [1; 32], vec![address; 33])),
|
||||
] {
|
||||
assert!(
|
||||
transport
|
||||
.on_control(network, peer, AWG_PROTOCOL, &bytes)
|
||||
.is_err()
|
||||
);
|
||||
}
|
||||
for len in 0..valid.len() {
|
||||
assert!(
|
||||
transport
|
||||
.on_control(network, peer, AWG_PROTOCOL, &valid[..len])
|
||||
.is_err()
|
||||
);
|
||||
}
|
||||
let wrong_role = AwgTransport::new(peer, Ipv4Addr::LOCALHOST);
|
||||
wrong_role.allow_peer(network, local);
|
||||
assert!(
|
||||
wrong_role
|
||||
.on_control(network, local, AWG_PROTOCOL, &valid)
|
||||
.is_err()
|
||||
);
|
||||
transport.forget_peer(network, peer);
|
||||
assert!(
|
||||
transport
|
||||
.on_control(network, peer, AWG_PROTOCOL, &valid)
|
||||
.is_err()
|
||||
);
|
||||
assert!(transport.overview(network).is_empty());
|
||||
transport.shutdown();
|
||||
wrong_role.shutdown();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,530 @@
|
||||
//! Real membership, UDP sockets, pair negotiation and nested WireGuard encryption.
|
||||
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
|
||||
use bytes::Bytes;
|
||||
use std::{
|
||||
net::Ipv4Addr,
|
||||
path::Path,
|
||||
sync::{
|
||||
Arc,
|
||||
atomic::{AtomicBool, AtomicU64, Ordering},
|
||||
},
|
||||
time::Duration,
|
||||
};
|
||||
use tempfile::TempDir;
|
||||
use tsunagi::dataplane::IpPlugin;
|
||||
use tsunagi::dataplane::transport::{
|
||||
ControlSender, PacketLink, PacketTransport, SharedLink, TransportError,
|
||||
};
|
||||
use tsunagi::discovery::SharedMemoryDiscovery;
|
||||
use tsunagi::overlay::{MemoryTun, MemoryTunFactory};
|
||||
use tsunagi::testing::{config_with, network, wait_for_peers, wait_until};
|
||||
use tsunagi::{Agent, BoxFuture, NetworkId};
|
||||
use tsunagi_awg::{AWG_PROTOCOL, AwgPlugin, AwgTransport};
|
||||
use tsunagi_wg_quic::WireguardConfig;
|
||||
|
||||
struct Node {
|
||||
agent: Agent,
|
||||
plugin: Arc<AwgPlugin>,
|
||||
transport: Arc<AwgTransport>,
|
||||
tuns: MemoryTunFactory,
|
||||
loss: Arc<AtomicBool>,
|
||||
}
|
||||
impl Node {
|
||||
async fn open(root: &Path, discovery: &SharedMemoryDiscovery, interface: &str) -> Self {
|
||||
Self::open_with_fallback(root, discovery, interface, false).await
|
||||
}
|
||||
async fn open_with_fallback(
|
||||
root: &Path,
|
||||
discovery: &SharedMemoryDiscovery,
|
||||
interface: &str,
|
||||
fallback: bool,
|
||||
) -> Self {
|
||||
Self::open_config(root, discovery, interface, fallback, 1280).await
|
||||
}
|
||||
async fn open_config(
|
||||
root: &Path,
|
||||
discovery: &SharedMemoryDiscovery,
|
||||
interface: &str,
|
||||
fallback: bool,
|
||||
mtu: u32,
|
||||
) -> Self {
|
||||
let identity =
|
||||
tsunagi::storage::StateStore::open(tsunagi::StoragePaths::under(root).state_db())
|
||||
.unwrap()
|
||||
.load_or_create_device_identity()
|
||||
.unwrap();
|
||||
let transport = AwgTransport::new(identity.endpoint_id(), Ipv4Addr::LOCALHOST);
|
||||
let plugin = AwgPlugin::open(
|
||||
WireguardConfig::new(root.join("awg"))
|
||||
.with_mtu(mtu)
|
||||
.with_reconcile(Duration::from_millis(20), Duration::from_millis(250)),
|
||||
transport.clone(),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
let loss = Arc::new(AtomicBool::new(false));
|
||||
let faults = Arc::new(FaultTransport {
|
||||
inner: transport.clone(),
|
||||
loss: loss.clone(),
|
||||
});
|
||||
let tuns = MemoryTunFactory::new();
|
||||
let mut config = config_with(root, discovery)
|
||||
.with_interface(Arc::new(tuns.clone()), interface, mtu)
|
||||
.with_overlay_ipv4_range(Some(tsunagi::state::DEFAULT_IPV4_RANGE))
|
||||
.with_plugin(plugin.clone() as Arc<dyn IpPlugin>)
|
||||
.with_custom_transport(AWG_PROTOCOL, faults);
|
||||
if fallback {
|
||||
let quic = tsunagi_wg_quic::WireguardPlugin::open(
|
||||
WireguardConfig::new(root.join("wg-quic"))
|
||||
.with_mtu(mtu)
|
||||
.with_reconcile(Duration::from_millis(20), Duration::from_millis(250)),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
config = config.with_plugin(quic);
|
||||
}
|
||||
let agent = Agent::spawn(config).await.unwrap();
|
||||
let inbound_agent = agent.clone();
|
||||
let inbound_transport = transport.clone();
|
||||
let inbound_loss = loss.clone();
|
||||
tokio::spawn(async move {
|
||||
while let Some(mut link) = inbound_transport.next_inbound().await {
|
||||
link.link = impaired(link.link, inbound_loss.clone());
|
||||
let _ = inbound_agent.install_inbound_link(link).await;
|
||||
}
|
||||
});
|
||||
Self {
|
||||
agent,
|
||||
plugin,
|
||||
transport,
|
||||
tuns,
|
||||
loss,
|
||||
}
|
||||
}
|
||||
async fn ready(&self, id: NetworkId, n: usize) {
|
||||
wait_until("AWG outer paths and end-to-end handshakes", || async {
|
||||
(self
|
||||
.transport
|
||||
.overview(id)
|
||||
.iter()
|
||||
.filter(|p| !p.closed && !p.path.contains("establishing"))
|
||||
.count()
|
||||
== n
|
||||
&& self
|
||||
.plugin
|
||||
.overview(id)?
|
||||
.peers
|
||||
.iter()
|
||||
.filter(|p| {
|
||||
p.tunnel
|
||||
.as_ref()
|
||||
.is_some_and(|t| t.health.is_up() && t.path.contains("awg UDP via"))
|
||||
})
|
||||
.count()
|
||||
== n)
|
||||
.then_some(())
|
||||
})
|
||||
.await;
|
||||
}
|
||||
fn tun(&self) -> Arc<MemoryTun> {
|
||||
self.tuns
|
||||
.device(&self.agent.overlay().unwrap().interface)
|
||||
.unwrap()
|
||||
}
|
||||
fn addr(&self, id: NetworkId) -> Ipv4Addr {
|
||||
self.plugin
|
||||
.overview(id)
|
||||
.unwrap()
|
||||
.overlay_address_v4
|
||||
.unwrap()
|
||||
}
|
||||
}
|
||||
fn packet(source: Ipv4Addr, dest: Ipv4Addr, len: usize) -> Bytes {
|
||||
let mut bytes = vec![0; len];
|
||||
bytes[0] = 0x45;
|
||||
bytes[2..4].copy_from_slice(&(len as u16).to_be_bytes());
|
||||
bytes[8] = 64;
|
||||
bytes[9] = 253;
|
||||
bytes[12..16].copy_from_slice(&source.octets());
|
||||
bytes[16..20].copy_from_slice(&dest.octets());
|
||||
rand::fill(&mut bytes[20..]);
|
||||
Bytes::from(bytes)
|
||||
}
|
||||
async fn exchange(a: &Node, b: &Node, id: NetworkId) {
|
||||
for len in [64, 1097, 1280] {
|
||||
let bytes = packet(a.addr(id), b.addr(id), len);
|
||||
a.tun().push_from_os(bytes.clone());
|
||||
assert_eq!(
|
||||
tokio::time::timeout(Duration::from_secs(5), b.tun().pop_to_os())
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap(),
|
||||
bytes
|
||||
);
|
||||
}
|
||||
}
|
||||
#[tokio::test]
|
||||
async fn real_pairs_carry_full_size_packets_and_restart_with_fresh_profiles() {
|
||||
tsunagi::testing::init_tracing();
|
||||
let discovery = SharedMemoryDiscovery::new();
|
||||
let roots = [
|
||||
TempDir::new().unwrap(),
|
||||
TempDir::new().unwrap(),
|
||||
TempDir::new().unwrap(),
|
||||
];
|
||||
let a = Node::open(roots[0].path(), &discovery, "awga").await;
|
||||
let b = Node::open(roots[1].path(), &discovery, "awgb").await;
|
||||
let c = Node::open(roots[2].path(), &discovery, "awgc").await;
|
||||
let (name, secret) = network("awg-pairs");
|
||||
let id = a.agent.join_network(&name, &secret).await.unwrap();
|
||||
b.agent.join_network(&name, &secret).await.unwrap();
|
||||
c.agent.join_network(&name, &secret).await.unwrap();
|
||||
for node in [&a, &b, &c] {
|
||||
wait_for_peers(&node.agent, id, 2).await;
|
||||
node.ready(id, 2).await;
|
||||
}
|
||||
let ids: std::collections::HashSet<_> = [&a, &b, &c]
|
||||
.into_iter()
|
||||
.flat_map(|n| n.transport.overview(id).into_iter().map(|p| p.attempt))
|
||||
.collect();
|
||||
assert_eq!(
|
||||
ids.len(),
|
||||
3,
|
||||
"one independent profile for each of three pairs"
|
||||
);
|
||||
exchange(&a, &b, id).await;
|
||||
exchange(&b, &a, id).await;
|
||||
exchange(&a, &c, id).await;
|
||||
let old_ports: Vec<_> = a
|
||||
.transport
|
||||
.overview(id)
|
||||
.into_iter()
|
||||
.filter_map(|p| p.local_port)
|
||||
.collect();
|
||||
a.agent.shutdown().await;
|
||||
for port in old_ports {
|
||||
wait_until("old pair socket released", || async {
|
||||
tokio::net::UdpSocket::bind((Ipv4Addr::LOCALHOST, port))
|
||||
.await
|
||||
.ok()
|
||||
})
|
||||
.await;
|
||||
}
|
||||
let restarted = Node::open(roots[0].path(), &discovery, "awga").await;
|
||||
restarted.ready(id, 2).await;
|
||||
assert!(
|
||||
restarted
|
||||
.transport
|
||||
.overview(id)
|
||||
.iter()
|
||||
.all(|p| !ids.contains(&p.attempt))
|
||||
);
|
||||
exchange(&restarted, &b, id).await;
|
||||
exchange(&c, &restarted, id).await;
|
||||
restarted.agent.shutdown().await;
|
||||
b.agent.shutdown().await;
|
||||
c.agent.shutdown().await;
|
||||
assert!(
|
||||
tokio::time::timeout(Duration::from_secs(1), restarted.transport.next_inbound())
|
||||
.await
|
||||
.unwrap()
|
||||
.is_none()
|
||||
);
|
||||
}
|
||||
|
||||
// Drops datagrams at the link boundary. Membership, control, sockets, crypto,
|
||||
// addressing and storage remain real. This substitutes a lossy physical path.
|
||||
#[derive(Debug)]
|
||||
struct FaultTransport {
|
||||
inner: Arc<AwgTransport>,
|
||||
loss: Arc<AtomicBool>,
|
||||
}
|
||||
impl PacketTransport for FaultTransport {
|
||||
fn name(&self) -> &str {
|
||||
AWG_PROTOCOL
|
||||
}
|
||||
fn as_any(&self) -> &dyn std::any::Any {
|
||||
self
|
||||
}
|
||||
fn attach_control(&self, sender: Arc<dyn ControlSender>) {
|
||||
self.inner.attach_control(sender);
|
||||
}
|
||||
fn on_control(
|
||||
&self,
|
||||
network: NetworkId,
|
||||
peer: iroh::EndpointId,
|
||||
protocol: &str,
|
||||
bytes: &[u8],
|
||||
) -> Result<(), TransportError> {
|
||||
self.inner.on_control(network, peer, protocol, bytes)
|
||||
}
|
||||
fn open<'a>(
|
||||
&'a self,
|
||||
network: NetworkId,
|
||||
peer: iroh::EndpointId,
|
||||
protocol: &'a str,
|
||||
) -> BoxFuture<'a, Result<SharedLink, TransportError>> {
|
||||
Box::pin(async move {
|
||||
Ok(impaired(
|
||||
self.inner.open(network, peer, protocol).await?,
|
||||
self.loss.clone(),
|
||||
))
|
||||
})
|
||||
}
|
||||
}
|
||||
fn impaired(raw: SharedLink, loss: Arc<AtomicBool>) -> SharedLink {
|
||||
Arc::new(FaultLink {
|
||||
raw,
|
||||
loss,
|
||||
sequence: AtomicU64::new(0),
|
||||
})
|
||||
}
|
||||
#[derive(Debug)]
|
||||
struct FaultLink {
|
||||
raw: SharedLink,
|
||||
loss: Arc<AtomicBool>,
|
||||
sequence: AtomicU64,
|
||||
}
|
||||
impl PacketLink for FaultLink {
|
||||
fn network(&self) -> NetworkId {
|
||||
self.raw.network()
|
||||
}
|
||||
fn peer(&self) -> iroh::EndpointId {
|
||||
self.raw.peer()
|
||||
}
|
||||
fn max_datagram_size(&self) -> usize {
|
||||
self.raw.max_datagram_size()
|
||||
}
|
||||
fn send(&self, bytes: Bytes) -> Result<(), TransportError> {
|
||||
if self.loss.load(Ordering::Relaxed)
|
||||
&& !self
|
||||
.sequence
|
||||
.fetch_add(1, Ordering::Relaxed)
|
||||
.is_multiple_of(5)
|
||||
{
|
||||
Ok(())
|
||||
} else {
|
||||
self.raw.send(bytes)
|
||||
}
|
||||
}
|
||||
fn recv(&self) -> BoxFuture<'_, Option<Bytes>> {
|
||||
self.raw.recv()
|
||||
}
|
||||
fn closed(&self) -> BoxFuture<'_, ()> {
|
||||
self.raw.closed()
|
||||
}
|
||||
fn is_closed(&self) -> bool {
|
||||
self.raw.is_closed()
|
||||
}
|
||||
fn close(&self) {
|
||||
self.raw.close();
|
||||
}
|
||||
fn path_description(&self) -> String {
|
||||
self.raw.path_description()
|
||||
}
|
||||
}
|
||||
#[tokio::test]
|
||||
async fn persistent_runtime_loss_retires_the_pair_and_negotiates_fresh_parameters() {
|
||||
let discovery = SharedMemoryDiscovery::new();
|
||||
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
|
||||
let a = Node::open(roots[0].path(), &discovery, "lossa").await;
|
||||
let b = Node::open(roots[1].path(), &discovery, "lossb").await;
|
||||
let (name, secret) = network("awg-loss");
|
||||
let id = a.agent.join_network(&name, &secret).await.unwrap();
|
||||
b.agent.join_network(&name, &secret).await.unwrap();
|
||||
a.ready(id, 1).await;
|
||||
b.ready(id, 1).await;
|
||||
exchange(&a, &b, id).await;
|
||||
let old = a.transport.overview(id)[0].attempt;
|
||||
let mut events = a.agent.subscribe();
|
||||
a.loss.store(true, Ordering::Relaxed);
|
||||
tsunagi::testing::wait_event(&mut events, |event| match event {
|
||||
tsunagi::Event::DataLinkDown {
|
||||
network, reason, ..
|
||||
} if *network == id && reason.contains("delivery checks") => Some(()),
|
||||
_ => None,
|
||||
})
|
||||
.await;
|
||||
a.loss.store(false, Ordering::Relaxed);
|
||||
wait_until("fresh AWG profile after runtime loss", || async {
|
||||
a.transport
|
||||
.overview(id)
|
||||
.iter()
|
||||
.any(|p| p.attempt != old && !p.closed && !p.path.contains("establishing"))
|
||||
.then_some(())
|
||||
})
|
||||
.await;
|
||||
a.ready(id, 1).await;
|
||||
b.ready(id, 1).await;
|
||||
let initiator = if a.agent.endpoint_id().as_bytes() < b.agent.endpoint_id().as_bytes() {
|
||||
&a
|
||||
} else {
|
||||
&b
|
||||
};
|
||||
let received = initiator.plugin.overview(id).unwrap().peers[0]
|
||||
.tunnel
|
||||
.as_ref()
|
||||
.unwrap()
|
||||
.stats
|
||||
.rx_packets;
|
||||
let attempt = initiator.transport.overview(id)[0].attempt;
|
||||
wait_until(
|
||||
"fresh pair delivery checks answered after installation",
|
||||
|| async {
|
||||
let view = initiator.plugin.overview(id)?;
|
||||
(view.peers.first()?.tunnel.as_ref()?.stats.rx_packets >= received + 3).then_some(())
|
||||
},
|
||||
)
|
||||
.await;
|
||||
assert_eq!(initiator.transport.overview(id)[0].attempt, attempt);
|
||||
wait_for_peers(&a.agent, id, 1).await;
|
||||
exchange(&a, &b, id).await;
|
||||
exchange(&b, &a, id).await;
|
||||
a.agent.shutdown().await;
|
||||
b.agent.shutdown().await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn runtime_loss_falls_back_to_real_iroh_without_interrupting_control_or_ip() {
|
||||
let discovery = SharedMemoryDiscovery::new();
|
||||
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
|
||||
let a = Node::open_with_fallback(roots[0].path(), &discovery, "fallba", true).await;
|
||||
let b = Node::open_with_fallback(roots[1].path(), &discovery, "fallbb", true).await;
|
||||
let (name, secret) = network("awg-fallback");
|
||||
let id = a.agent.join_network(&name, &secret).await.unwrap();
|
||||
b.agent.join_network(&name, &secret).await.unwrap();
|
||||
a.ready(id, 1).await;
|
||||
b.ready(id, 1).await;
|
||||
exchange(&a, &b, id).await;
|
||||
let mut events = a.agent.subscribe();
|
||||
a.loss.store(true, Ordering::Relaxed);
|
||||
tsunagi::testing::wait_event(&mut events, |event| match event {
|
||||
tsunagi::Event::DataLinkDown {
|
||||
network, reason, ..
|
||||
} if *network == id && reason.contains("delivery checks") => Some(()),
|
||||
_ => None,
|
||||
})
|
||||
.await;
|
||||
for node in [&a, &b] {
|
||||
wait_until("fallback carrier is wg-quic",||async {
|
||||
let status=node.agent.status().await.ok()?;
|
||||
let network=status.networks.iter().find(|n|n.network_id==id)?;
|
||||
(network.members.iter().any(|m|matches!(&m.data_path,tsunagi::agent::DataPath::Direct{transport} if transport=="wg-quic"))).then_some(())
|
||||
}).await;
|
||||
wait_for_peers(&node.agent, id, 1).await;
|
||||
}
|
||||
exchange(&a, &b, id).await;
|
||||
exchange(&b, &a, id).await;
|
||||
a.agent.shutdown().await;
|
||||
b.agent.shutdown().await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn opaque_pair_carrier_fragments_its_full_advertised_capacity() {
|
||||
let discovery = SharedMemoryDiscovery::new();
|
||||
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
|
||||
let mut nodes = Vec::new();
|
||||
for root in &roots {
|
||||
let identity = tsunagi::storage::StateStore::open(
|
||||
tsunagi::StoragePaths::under(root.path()).state_db(),
|
||||
)
|
||||
.unwrap()
|
||||
.load_or_create_device_identity()
|
||||
.unwrap();
|
||||
let transport = AwgTransport::new(identity.endpoint_id(), Ipv4Addr::LOCALHOST);
|
||||
let agent = Agent::spawn(
|
||||
config_with(root.path(), &discovery)
|
||||
.with_custom_transport(AWG_PROTOCOL, transport.clone()),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
nodes.push((agent, transport));
|
||||
}
|
||||
let (name, secret) = network("awg-capacity");
|
||||
let id = nodes[0].0.join_network(&name, &secret).await.unwrap();
|
||||
nodes[1].0.join_network(&name, &secret).await.unwrap();
|
||||
for (i, (agent, transport)) in nodes.iter().enumerate() {
|
||||
wait_for_peers(agent, id, 1).await;
|
||||
// This fixture tests only the transport contract: register the member
|
||||
// after the real core has authenticated it, without an IP plugin/TUN.
|
||||
transport.allow_peer(id, nodes[1 - i].0.endpoint_id());
|
||||
}
|
||||
nodes.sort_by_key(|(agent, _)| *agent.endpoint_id().as_bytes());
|
||||
let outgoing = nodes[0]
|
||||
.1
|
||||
.open(id, nodes[1].0.endpoint_id(), AWG_PROTOCOL)
|
||||
.await
|
||||
.unwrap();
|
||||
let incoming = tokio::time::timeout(Duration::from_secs(5), nodes[1].1.next_inbound())
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap()
|
||||
.link;
|
||||
for (from, to) in [(&outgoing, &incoming), (&incoming, &outgoing)] {
|
||||
let mut bytes = vec![0; from.max_datagram_size()];
|
||||
rand::fill(&mut bytes[..]);
|
||||
let bytes = Bytes::from(bytes);
|
||||
from.send(bytes.clone()).unwrap();
|
||||
assert_eq!(
|
||||
tokio::time::timeout(Duration::from_secs(5), to.recv())
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap(),
|
||||
bytes
|
||||
);
|
||||
assert!(matches!(
|
||||
from.send(Bytes::from(vec![0; from.max_datagram_size() + 1])),
|
||||
Err(TransportError::TooLarge { .. })
|
||||
));
|
||||
}
|
||||
outgoing.close();
|
||||
incoming.close();
|
||||
for (agent, transport) in nodes {
|
||||
transport.shutdown();
|
||||
agent.shutdown().await;
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn encrypted_health_checks_fit_a_smaller_peers_configured_interface_mtu() {
|
||||
let discovery = SharedMemoryDiscovery::new();
|
||||
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
|
||||
let a = Node::open_config(roots[0].path(), &discovery, "smalla", false, 1280).await;
|
||||
let b = Node::open_config(roots[1].path(), &discovery, "smallb", false, 576).await;
|
||||
let (name, secret) = network("awg-small-mtu");
|
||||
let id = a.agent.join_network(&name, &secret).await.unwrap();
|
||||
b.agent.join_network(&name, &secret).await.unwrap();
|
||||
a.ready(id, 1).await;
|
||||
b.ready(id, 1).await;
|
||||
let bytes = packet(a.addr(id), b.addr(id), 576);
|
||||
a.tun().push_from_os(bytes.clone());
|
||||
assert_eq!(
|
||||
tokio::time::timeout(Duration::from_secs(5), b.tun().pop_to_os())
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap(),
|
||||
bytes
|
||||
);
|
||||
wait_until(
|
||||
"encrypted delivery checks answered with a small MTU",
|
||||
|| async {
|
||||
let view = a.plugin.overview(id)?;
|
||||
let peer = view.peers.first()?.tunnel.as_ref()?;
|
||||
(peer.stats.rx_packets >= 3).then_some(())
|
||||
},
|
||||
)
|
||||
.await;
|
||||
for node in [&a, &b] {
|
||||
assert_eq!(
|
||||
node.plugin.overview(id).unwrap().peers[0]
|
||||
.tunnel
|
||||
.as_ref()
|
||||
.unwrap()
|
||||
.stats
|
||||
.dropped_oversize,
|
||||
0
|
||||
);
|
||||
}
|
||||
a.agent.shutdown().await;
|
||||
b.agent.shutdown().await;
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
//! Fresh and bounded masking profiles, and rejection of foreign/truncated wire packets.
|
||||
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
|
||||
use tsunagi_awg::profile::PairProfile;
|
||||
|
||||
#[test]
|
||||
fn profiles_are_fresh_bounded_and_redacted() {
|
||||
let profiles: Vec<_> = (0..128).map(|_| PairProfile::generate()).collect();
|
||||
for (i, profile) in profiles.iter().enumerate() {
|
||||
profile.validate().unwrap();
|
||||
assert!(!format!("{profile:?}").contains(&format!("{:?}", profile.header_key())));
|
||||
for other in &profiles[..i] {
|
||||
assert_ne!(profile.header_key(), other.header_key());
|
||||
assert_ne!(profile.id(), other.id());
|
||||
}
|
||||
for packet in profile.signature_packets() {
|
||||
assert!(packet.len() < 1200);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn masked_packets_round_trip_and_reject_other_profiles_and_truncation() {
|
||||
let profile = PairProfile::generate();
|
||||
let other = PairProfile::generate();
|
||||
for (kind, size) in [(1, 148), (2, 92), (3, 64), (4, 800)] {
|
||||
let mut message = vec![0x7b; size];
|
||||
message[..4].copy_from_slice(&(kind as u32).to_le_bytes());
|
||||
let wire = profile.encode(&message).unwrap();
|
||||
assert_ne!(&wire[..4], &message[..4]);
|
||||
assert_eq!(profile.decode(&wire).unwrap(), message);
|
||||
assert!(other.decode(&wire).is_none());
|
||||
for length in 0..44 {
|
||||
assert!(profile.decode(&wire[..length]).is_none());
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -129,6 +129,19 @@ pub enum Event {
|
||||
/// Why it is not up.
|
||||
reason: String,
|
||||
},
|
||||
/// The name this agent claimed was claimed first by another member.
|
||||
///
|
||||
/// Nothing stops working: this agent keeps its address and every
|
||||
/// connection. It just is not found by that name until it is renamed or
|
||||
/// the member that holds the name lets it go.
|
||||
HostnameTaken {
|
||||
/// The network.
|
||||
network: NetworkId,
|
||||
/// The name that is taken.
|
||||
name: String,
|
||||
/// The member that holds it.
|
||||
holder: EndpointId,
|
||||
},
|
||||
/// An IP plugin reported an error. Never fatal.
|
||||
PluginError {
|
||||
/// The network the call was scoped to.
|
||||
|
||||
@@ -26,15 +26,15 @@ mod status;
|
||||
|
||||
pub use events::Event;
|
||||
pub use status::{
|
||||
AgentStatus, CandidateStatus, MemberStatus, NetworkMetrics, NetworkState, NetworkStatus,
|
||||
OverlayStatus, PeerStatus,
|
||||
AgentStatus, CandidateStatus, DataPath, HostnameConflict, MemberStatus, NetworkMetrics,
|
||||
NetworkState, NetworkStatus, OverlayStatus, PeerStatus, RangeConflict,
|
||||
};
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::sync::{Arc, Weak};
|
||||
|
||||
use iroh::{EndpointAddr, EndpointId};
|
||||
use tokio::sync::{RwLock, broadcast, mpsc, oneshot};
|
||||
use tokio::sync::{Mutex, RwLock, broadcast, mpsc, oneshot, watch};
|
||||
use tokio::task::JoinHandle;
|
||||
|
||||
use crate::BoxFuture;
|
||||
@@ -90,6 +90,10 @@ struct Inner {
|
||||
events: broadcast::Sender<Event>,
|
||||
networks: RwLock<HashMap<NetworkId, NetworkHandle>>,
|
||||
shutdown: Shutdown,
|
||||
/// Serialises start, stop, switching and conflict cleanup, including teardown.
|
||||
lifecycle: Mutex<()>,
|
||||
range_changed: watch::Sender<()>,
|
||||
range_task: std::sync::Mutex<Option<JoinHandle<()>>>,
|
||||
accept_task: std::sync::Mutex<Option<JoinHandle<()>>>,
|
||||
plugin_task: std::sync::Mutex<Option<JoinHandle<()>>>,
|
||||
transport: std::sync::OnceLock<Arc<dyn PacketTransport>>,
|
||||
@@ -165,6 +169,21 @@ impl PacketSink for Sink {
|
||||
let Some(inner) = self.0.upgrade() else {
|
||||
return;
|
||||
};
|
||||
if let Some(payload) = crate::dataplane::transport::health::ip_payload(&packet) {
|
||||
if let Some(header) = crate::overlay::IpHeader::parse(&packet)
|
||||
&& let (std::net::IpAddr::V4(source), std::net::IpAddr::V4(destination)) =
|
||||
(header.source(), header.destination())
|
||||
&& inner.routes.may_send_from(network, peer, source)
|
||||
&& inner.routes.is_local_destination(network, destination)
|
||||
&& let Some(handle) = inner.networks.read().await.get(&network)
|
||||
{
|
||||
let _ = handle.commands.try_send(NetCommand::HealthProbe {
|
||||
peer,
|
||||
packet: bytes::Bytes::copy_from_slice(payload),
|
||||
});
|
||||
}
|
||||
return;
|
||||
}
|
||||
let Some(interface) = inner.interface.get() else {
|
||||
return;
|
||||
};
|
||||
@@ -183,6 +202,35 @@ impl PacketSink for Sink {
|
||||
#[derive(Debug)]
|
||||
struct TransportCtx(Weak<Inner>);
|
||||
|
||||
impl crate::dataplane::transport::ControlSender for TransportCtx {
|
||||
fn send<'a>(
|
||||
&'a self,
|
||||
network: NetworkId,
|
||||
peer: EndpointId,
|
||||
protocol: &'a str,
|
||||
payload: crate::proto::message::TransportPayload,
|
||||
) -> crate::BoxFuture<'a, std::result::Result<(), crate::dataplane::transport::TransportError>>
|
||||
{
|
||||
Box::pin(async move {
|
||||
let inner = self
|
||||
.0
|
||||
.upgrade()
|
||||
.ok_or(crate::dataplane::transport::TransportError::Closed)?;
|
||||
Agent { inner }
|
||||
.send(
|
||||
network,
|
||||
peer,
|
||||
crate::proto::message::ControlMessage::Transport {
|
||||
protocol: protocol.into(),
|
||||
payload,
|
||||
},
|
||||
)
|
||||
.await
|
||||
.map_err(|err| crate::dataplane::transport::TransportError::Other(err.to_string()))
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
impl TransportContext for TransportCtx {
|
||||
fn snapshot(&self) -> crate::BoxFuture<'_, HashMap<NetworkId, NetworkKeys>> {
|
||||
Box::pin(async move {
|
||||
@@ -234,6 +282,7 @@ impl Agent {
|
||||
let (events, _) = broadcast::channel(config.limits.event_buffer);
|
||||
let limits = Arc::new(config.limits.clone());
|
||||
|
||||
let (range_changed, range_updates) = watch::channel(());
|
||||
let inner = Arc::new(Inner {
|
||||
limits,
|
||||
storage,
|
||||
@@ -243,6 +292,9 @@ impl Agent {
|
||||
events,
|
||||
networks: RwLock::new(HashMap::new()),
|
||||
shutdown: Shutdown::new(),
|
||||
lifecycle: Mutex::new(()),
|
||||
range_changed,
|
||||
range_task: std::sync::Mutex::new(None),
|
||||
accept_task: std::sync::Mutex::new(None),
|
||||
plugin_task: std::sync::Mutex::new(None),
|
||||
transport: std::sync::OnceLock::new(),
|
||||
@@ -294,6 +346,7 @@ impl Agent {
|
||||
inner.config.custom_transports.clone(),
|
||||
))
|
||||
};
|
||||
transport.attach_control(Arc::new(TransportCtx(Arc::downgrade(&inner))));
|
||||
let _ = inner.transport.set(transport);
|
||||
|
||||
if let CacheOutcome::Reset(reason) = inner.storage.cache_outcome().clone() {
|
||||
@@ -326,11 +379,16 @@ impl Agent {
|
||||
|
||||
let agent = Self { inner };
|
||||
|
||||
for stored in agent.inner.storage.list_networks().await? {
|
||||
if stored.auto_start {
|
||||
let keys = NetworkKeys::derive(&stored.name, &stored.secret);
|
||||
agent.activate_with_keys(keys).await?;
|
||||
}
|
||||
{
|
||||
let _guard = agent.inner.lifecycle.lock().await;
|
||||
agent.resume_networks_locked().await?;
|
||||
}
|
||||
let task = tokio::spawn(range_conflict_loop(
|
||||
Arc::downgrade(&agent.inner),
|
||||
range_updates,
|
||||
));
|
||||
if let Ok(mut guard) = agent.inner.range_task.lock() {
|
||||
*guard = Some(task);
|
||||
}
|
||||
|
||||
Ok(agent)
|
||||
@@ -374,100 +432,65 @@ impl Agent {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Reserves the configured overlay range for a network, if it can.
|
||||
///
|
||||
/// Done here, where activations are serialised, rather than inside the
|
||||
/// runtime: whether a range is free depends on what the other networks
|
||||
/// took, and deciding that in a task would make the answer depend on
|
||||
/// which task ran first.
|
||||
///
|
||||
/// `None` means this agent will not propose a range for that network and
|
||||
/// waits to adopt whatever it settles on. One agent has one interface, so
|
||||
/// proposing a range it could not route would be worse than having none:
|
||||
/// the lowest author's range wins, and the collision would spread.
|
||||
async fn reserve_range(&self, network: NetworkId) -> RangePlan {
|
||||
let reserve = |wanted: Ipv4Range| {
|
||||
let reservation = crate::overlay::NetworkRoutes {
|
||||
exit: Default::default(),
|
||||
broadcast: Default::default(),
|
||||
range: Some(wanted),
|
||||
local: None,
|
||||
peers: Vec::new(),
|
||||
};
|
||||
self.inner.routes.set_network(network, reservation).is_ok()
|
||||
};
|
||||
|
||||
// If this network already has signed records in storage, check if its
|
||||
// members have already agreed on a range. If so, preserve and reserve
|
||||
// THAT range rather than preemptively seizing the default configured range.
|
||||
let stored_records = self
|
||||
.inner
|
||||
.storage
|
||||
.signed_records(network)
|
||||
.await
|
||||
.unwrap_or_default();
|
||||
let mut state_set = crate::state::StateSet::new();
|
||||
state_set.merge_all(network, stored_records);
|
||||
if let Some(agreed) = state_set.agreed_range() {
|
||||
if reserve(agreed) {
|
||||
return RangePlan {
|
||||
propose: if Some(agreed) == self.inner.config.overlay_ipv4_range {
|
||||
Some(agreed)
|
||||
} else {
|
||||
None
|
||||
},
|
||||
..RangePlan::default()
|
||||
};
|
||||
/// Signed state always wins over a default or an explicit proposal.
|
||||
async fn desired_range(&self, network: NetworkId) -> Result<Option<Ipv4Range>> {
|
||||
let records = self.inner.storage.signed_records(network).await?;
|
||||
let mut state = crate::state::StateSet::new();
|
||||
state.merge_all(network, records);
|
||||
Ok(state.agreed_range().or_else(|| {
|
||||
if self.inner.config.overlay_ipv4_auto {
|
||||
Some(crate::state::derived_ipv4_range(network))
|
||||
} else {
|
||||
tracing::warn!(
|
||||
%network,
|
||||
range = %agreed,
|
||||
"agreed range for this network overlaps another network already active on this interface"
|
||||
);
|
||||
return RangePlan {
|
||||
conflict: Some(agreed),
|
||||
..RangePlan::default()
|
||||
};
|
||||
self.inner.config.overlay_ipv4_range
|
||||
}
|
||||
}))
|
||||
}
|
||||
|
||||
async fn range_conflicts(
|
||||
&self,
|
||||
network: NetworkId,
|
||||
range: Option<Ipv4Range>,
|
||||
) -> Result<Vec<RangeConflict>> {
|
||||
let Some(range) = range else {
|
||||
return Ok(Vec::new());
|
||||
};
|
||||
let overlapping = self.inner.routes.overlapping_networks(network, range);
|
||||
let stored = self.inner.storage.list_networks().await?;
|
||||
Ok(overlapping
|
||||
.into_iter()
|
||||
.filter_map(|(id, range)| {
|
||||
let name = stored.iter().find(|n| n.network_id == id)?.name.clone();
|
||||
Some(RangeConflict {
|
||||
network_id: id,
|
||||
name,
|
||||
range,
|
||||
})
|
||||
})
|
||||
.collect())
|
||||
}
|
||||
|
||||
/// Starts requested networks in saved preference order, skipping overlaps.
|
||||
/// Called with the lifecycle lock held; skipped networks remain requested.
|
||||
async fn resume_networks_locked(&self) -> Result<()> {
|
||||
let order = self.inner.storage.network_start_order().await?;
|
||||
let mut stored = self.inner.storage.list_networks().await?;
|
||||
stored.sort_by_key(|n| {
|
||||
order
|
||||
.iter()
|
||||
.position(|id| *id == n.network_id)
|
||||
.unwrap_or(usize::MAX)
|
||||
});
|
||||
for network in stored {
|
||||
if !network.auto_start || self.is_active(network.network_id).await {
|
||||
continue;
|
||||
}
|
||||
let keys = NetworkKeys::derive(&network.name, &network.secret);
|
||||
match self.activate_with_keys_locked(keys).await {
|
||||
Ok(()) | Err(Error::NetworkRangeConflict { .. }) => {}
|
||||
Err(err) => return Err(err),
|
||||
}
|
||||
}
|
||||
|
||||
let Some(configured) = self.inner.config.overlay_ipv4_range else {
|
||||
return RangePlan::default();
|
||||
};
|
||||
|
||||
// The configured range first, so a network a device has always had
|
||||
// keeps the addresses it has always had.
|
||||
if reserve(configured) {
|
||||
return RangePlan {
|
||||
propose: Some(configured),
|
||||
..RangePlan::default()
|
||||
};
|
||||
}
|
||||
|
||||
// A second network on the same agent cannot have it — one agent,
|
||||
// one interface — so it falls back to the range derived from its
|
||||
// own id, which every one of its members derives identically
|
||||
// without being told.
|
||||
let derived = crate::state::derived_ipv4_range(network);
|
||||
if reserve(derived) {
|
||||
tracing::info!(
|
||||
%network,
|
||||
range = %derived,
|
||||
"another network here holds the configured range; this one uses the \
|
||||
range derived from its id unless its members settled another"
|
||||
);
|
||||
return RangePlan {
|
||||
fallback: Some(derived),
|
||||
conflict: Some(configured),
|
||||
..RangePlan::default()
|
||||
};
|
||||
}
|
||||
|
||||
tracing::info!(%network, "not proposing a range for this network");
|
||||
RangePlan {
|
||||
conflict: Some(configured),
|
||||
..RangePlan::default()
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// The overlay interface this agent owns, when it has one.
|
||||
@@ -546,6 +569,9 @@ impl Agent {
|
||||
/// The same `(name, secret)` always produces the same [`NetworkId`], on
|
||||
/// every device.
|
||||
///
|
||||
/// An overlapping network stays configured and requested, but inactive;
|
||||
/// use [`Agent::switch_network`] to suspend the blockers explicitly.
|
||||
///
|
||||
/// This is declarative and therefore **idempotent**: joining a network
|
||||
/// that is already active succeeds and changes nothing. That matters
|
||||
/// because a configured network is activated automatically at startup, so
|
||||
@@ -599,7 +625,9 @@ impl Agent {
|
||||
}
|
||||
match self.activate_with_keys(keys).await {
|
||||
// Already a member of exactly this network space: nothing to do.
|
||||
Ok(()) | Err(Error::NetworkAlreadyActive(_)) => Ok(network_id),
|
||||
Ok(())
|
||||
| Err(Error::NetworkAlreadyActive(_))
|
||||
| Err(Error::NetworkRangeConflict { .. }) => Ok(network_id),
|
||||
Err(err) => Err(err),
|
||||
}
|
||||
}
|
||||
@@ -703,7 +731,44 @@ impl Agent {
|
||||
self.activate_with_keys(keys).await
|
||||
}
|
||||
|
||||
/// Switches to a configured network, suspending every overlapping network.
|
||||
/// Suspended networks retain their auto-start request and resume when free.
|
||||
/// Returns the networks suspended by this request.
|
||||
pub async fn switch_network(&self, network_id: NetworkId) -> Result<Vec<NetworkId>> {
|
||||
let _guard = self.inner.lifecycle.lock().await;
|
||||
if self.inner.shutdown.is_triggered() {
|
||||
return Err(Error::Stopped);
|
||||
}
|
||||
let stored = self
|
||||
.inner
|
||||
.storage
|
||||
.list_networks()
|
||||
.await?
|
||||
.into_iter()
|
||||
.find(|n| n.network_id == network_id)
|
||||
.ok_or(Error::NetworkUnknown(network_id))?;
|
||||
let range = self.desired_range(network_id).await?;
|
||||
let conflicts = self.range_conflicts(network_id, range).await?;
|
||||
self.inner.storage.prefer_network(network_id).await?;
|
||||
let mut stopped = Vec::new();
|
||||
for conflict in conflicts {
|
||||
self.stop_network_locked(conflict.network_id).await?;
|
||||
stopped.push(conflict.network_id);
|
||||
}
|
||||
if !self.is_active(network_id).await {
|
||||
self.activate_with_keys_locked(NetworkKeys::derive(&stored.name, &stored.secret))
|
||||
.await?;
|
||||
}
|
||||
self.resume_networks_locked().await?;
|
||||
Ok(stopped)
|
||||
}
|
||||
|
||||
async fn activate_with_keys(&self, keys: NetworkKeys) -> Result<()> {
|
||||
let _guard = self.inner.lifecycle.lock().await;
|
||||
self.activate_with_keys_locked(keys).await
|
||||
}
|
||||
|
||||
async fn activate_with_keys_locked(&self, keys: NetworkKeys) -> Result<()> {
|
||||
if self.inner.shutdown.is_triggered() {
|
||||
return Err(Error::Stopped);
|
||||
}
|
||||
@@ -712,8 +777,36 @@ impl Agent {
|
||||
if networks.contains_key(&network_id) {
|
||||
return Err(Error::NetworkAlreadyActive(network_id));
|
||||
}
|
||||
|
||||
let range = self.reserve_range(network_id).await;
|
||||
let stored = self
|
||||
.inner
|
||||
.storage
|
||||
.list_networks()
|
||||
.await?
|
||||
.into_iter()
|
||||
.find(|stored| stored.network_id == network_id);
|
||||
let range = self.desired_range(network_id).await?;
|
||||
if let Some(range) = range {
|
||||
let conflicts = self.inner.routes.overlapping_networks(network_id, range);
|
||||
if !conflicts.is_empty() {
|
||||
return Err(Error::NetworkRangeConflict {
|
||||
network: network_id,
|
||||
range,
|
||||
conflicts: conflicts.into_iter().map(|(id, _)| id).collect(),
|
||||
});
|
||||
}
|
||||
self.inner
|
||||
.routes
|
||||
.set_network(
|
||||
network_id,
|
||||
crate::overlay::NetworkRoutes {
|
||||
range: Some(range),
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.map_err(|err| Error::Rejected {
|
||||
reason: err.to_string(),
|
||||
})?;
|
||||
}
|
||||
let mut backends = Vec::new();
|
||||
if let Some(discovery) = &self.inner.config.discovery {
|
||||
backends.push(discovery.clone());
|
||||
@@ -729,13 +822,7 @@ impl Agent {
|
||||
as Arc<dyn crate::discovery::NetworkDiscovery>,
|
||||
)
|
||||
};
|
||||
let stored = self
|
||||
.inner
|
||||
.storage
|
||||
.list_networks()
|
||||
.await?
|
||||
.into_iter()
|
||||
.find(|stored| stored.network_id == network_id);
|
||||
|
||||
let broadcast = stored.as_ref().is_none_or(|stored| stored.broadcast);
|
||||
let exit_offer = stored.as_ref().is_some_and(|stored| stored.exit_node);
|
||||
let exit_via = stored
|
||||
@@ -763,9 +850,8 @@ impl Agent {
|
||||
device_secret: self.inner.identity.signing_key(),
|
||||
#[cfg(feature = "testing")]
|
||||
unreachable_data_peers: Arc::clone(&self.inner.config.unreachable_data_peers),
|
||||
ipv4_range: range.propose,
|
||||
ipv4_fallback: range.fallback,
|
||||
range_conflict: range.conflict,
|
||||
ipv4_range: range,
|
||||
range_changed: self.inner.range_changed.clone(),
|
||||
});
|
||||
networks.insert(network_id, handle);
|
||||
drop(networks);
|
||||
@@ -786,6 +872,30 @@ impl Agent {
|
||||
/// it does not remove this agent from anyone else's view of the network.
|
||||
/// Other networks keep running.
|
||||
pub async fn deactivate_network(&self, network_id: NetworkId) -> Result<()> {
|
||||
let _guard = self.inner.lifecycle.lock().await;
|
||||
if !self.is_active(network_id).await {
|
||||
return Err(Error::NetworkNotActive(network_id));
|
||||
}
|
||||
self.inner.storage.set_auto_start(network_id, false).await?;
|
||||
self.stop_network_locked(network_id).await?;
|
||||
self.resume_networks_locked().await
|
||||
}
|
||||
|
||||
/// Stops a configured network, including one already suspended by overlap.
|
||||
/// Clears its auto-start request so it will not resume behind the user.
|
||||
pub async fn pause_network(&self, network_id: NetworkId) -> Result<()> {
|
||||
let _guard = self.inner.lifecycle.lock().await;
|
||||
if self.inner.shutdown.is_triggered() {
|
||||
return Err(Error::Stopped);
|
||||
}
|
||||
self.inner.storage.set_auto_start(network_id, false).await?;
|
||||
if self.is_active(network_id).await {
|
||||
self.stop_network_locked(network_id).await?;
|
||||
}
|
||||
self.resume_networks_locked().await
|
||||
}
|
||||
|
||||
async fn stop_network_locked(&self, network_id: NetworkId) -> Result<()> {
|
||||
let handle = {
|
||||
let mut networks = self.inner.networks.write().await;
|
||||
networks
|
||||
@@ -814,7 +924,6 @@ impl Agent {
|
||||
interface.sync_broadcast_rules().await;
|
||||
interface.sync_exit_rules().await;
|
||||
}
|
||||
self.inner.storage.set_auto_start(network_id, false).await?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -824,17 +933,15 @@ impl Agent {
|
||||
/// Local and silent: nobody is told. [`Agent::leave_network`] is the one
|
||||
/// that says goodbye first, and is what a person means by leaving.
|
||||
pub async fn forget_network(&self, network_id: NetworkId) -> Result<()> {
|
||||
let _guard = self.inner.lifecycle.lock().await;
|
||||
if self.is_active(network_id).await {
|
||||
self.deactivate_network(network_id).await?;
|
||||
self.stop_network_locked(network_id).await?;
|
||||
}
|
||||
self.inner.storage.remove_network(network_id).await?;
|
||||
// Whatever a protocol kept for this network goes with it. The
|
||||
// agent cannot know what that is, only that there is no longer any
|
||||
// reason to hold it.
|
||||
for plugin in &self.inner.config.plugins {
|
||||
plugin.on_network_forgotten(network_id);
|
||||
}
|
||||
Ok(())
|
||||
self.resume_networks_locked().await
|
||||
}
|
||||
|
||||
/// Leaves a network: gives up what was claimed, then forgets it.
|
||||
@@ -958,13 +1065,17 @@ impl Agent {
|
||||
let mut networks = Vec::new();
|
||||
for stored in self.inner.storage.list_networks().await? {
|
||||
if active.contains(&stored.network_id)
|
||||
&& let Ok(status) = self.network_status(stored.network_id).await
|
||||
&& let Ok(mut status) = self.network_status(stored.network_id).await
|
||||
{
|
||||
status.auto_start = stored.auto_start;
|
||||
networks.push(status);
|
||||
continue;
|
||||
}
|
||||
let keys = NetworkKeys::derive(&stored.name, &stored.secret);
|
||||
let range = self.desired_range(stored.network_id).await?;
|
||||
let range_conflicts = self.range_conflicts(stored.network_id, range).await?;
|
||||
networks.push(NetworkStatus {
|
||||
auto_start: stored.auto_start,
|
||||
broadcast: stored.broadcast,
|
||||
exit_offer: stored.exit_node,
|
||||
exit_via: stored
|
||||
@@ -978,11 +1089,13 @@ impl Agent {
|
||||
state: NetworkState::Inactive,
|
||||
peers: Vec::new(),
|
||||
candidates: Vec::new(),
|
||||
// An inactive network has no runtime to ask; the roster and
|
||||
// the range come from one. Empty, not invented.
|
||||
// The range is available from signed state or the deterministic
|
||||
// default even while the network has no runtime.
|
||||
members: Vec::new(),
|
||||
range: None,
|
||||
range_conflict: None,
|
||||
hostname_conflict: None,
|
||||
range,
|
||||
range_conflict: range_conflicts.first().map(|conflict| conflict.range),
|
||||
range_conflicts,
|
||||
metrics: NetworkMetrics::default(),
|
||||
relay: Default::default(),
|
||||
});
|
||||
@@ -1038,6 +1151,7 @@ impl Agent {
|
||||
/// After this returns, the state directory can be opened by another agent.
|
||||
pub async fn shutdown(&self) {
|
||||
self.inner.shutdown.trigger();
|
||||
let lifecycle_guard = self.inner.lifecycle.lock().await;
|
||||
|
||||
let handles: Vec<NetworkHandle> = {
|
||||
let mut networks = self.inner.networks.write().await;
|
||||
@@ -1047,6 +1161,7 @@ impl Agent {
|
||||
handle.stop().await;
|
||||
}
|
||||
|
||||
drop(lifecycle_guard);
|
||||
if let Some(dht) = &self.inner.config.dht {
|
||||
dht.shutdown().await;
|
||||
}
|
||||
@@ -1063,6 +1178,7 @@ impl Agent {
|
||||
for (what, handle) in [
|
||||
("inbound connections", &self.inner.accept_task),
|
||||
("plugin requests", &self.inner.plugin_task),
|
||||
("range conflicts", &self.inner.range_task),
|
||||
] {
|
||||
let task = handle.lock().ok().and_then(|mut guard| guard.take());
|
||||
if let Some(task) = task {
|
||||
@@ -1106,16 +1222,44 @@ impl Agent {
|
||||
}
|
||||
}
|
||||
|
||||
/// What a network may propose as its overlay range, and what it cannot have.
|
||||
#[derive(Debug, Clone, Copy, Default)]
|
||||
struct RangePlan {
|
||||
/// Proposed straight away: the configured range, when it is free here.
|
||||
propose: Option<Ipv4Range>,
|
||||
/// Proposed after a moment, when the configured range is another
|
||||
/// network's: the range derived from this network's own id.
|
||||
fallback: Option<Ipv4Range>,
|
||||
/// The configured range, when this network cannot have it.
|
||||
conflict: Option<Ipv4Range>,
|
||||
/// Processes coalesced conflict notifications without blocking a network actor.
|
||||
async fn range_conflict_loop(inner: Weak<Inner>, mut updates: watch::Receiver<()>) {
|
||||
loop {
|
||||
let Some(owner) = inner.upgrade() else {
|
||||
return;
|
||||
};
|
||||
let shutdown = owner.shutdown.clone();
|
||||
drop(owner);
|
||||
tokio::select! {
|
||||
_ = shutdown.wait() => return,
|
||||
result = updates.changed() => if result.is_err() { return; },
|
||||
}
|
||||
let Some(inner) = inner.upgrade() else {
|
||||
return;
|
||||
};
|
||||
let agent = Agent { inner };
|
||||
let _guard = agent.inner.lifecycle.lock().await;
|
||||
if agent.inner.shutdown.is_triggered() {
|
||||
return;
|
||||
}
|
||||
let blocked: Vec<_> = agent
|
||||
.inner
|
||||
.networks
|
||||
.read()
|
||||
.await
|
||||
.iter()
|
||||
.filter(|(_, h)| h.range_blocked.load(std::sync::atomic::Ordering::Acquire))
|
||||
.map(|(id, _)| *id)
|
||||
.collect();
|
||||
for id in blocked {
|
||||
if let Err(err) = agent.stop_network_locked(id).await {
|
||||
tracing::warn!(%err, "cannot suspend overlapping network");
|
||||
}
|
||||
}
|
||||
if let Err(err) = agent.resume_networks_locked().await {
|
||||
tracing::warn!(%err, "cannot resume suspended networks");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// How long a release is given to reach the sessions it was queued on.
|
||||
|
||||
@@ -15,6 +15,7 @@ use tokio::sync::{broadcast, mpsc, oneshot};
|
||||
use tokio::task::JoinHandle;
|
||||
|
||||
use crate::config::{DiscoveryPolicy, Limits, ReconnectPolicy};
|
||||
use crate::dataplane::transport::health::{HealthLink, ProbeRoute, Settings as HealthSettings};
|
||||
use crate::dataplane::transport::{InboundLink, PacketLink, PacketTransport, SharedLink};
|
||||
use crate::dataplane::{PluginCapability, SharedPlugin};
|
||||
use crate::discovery::worker::DiscoveryWorker;
|
||||
@@ -32,7 +33,8 @@ use super::events::Event;
|
||||
use super::session::{self, Session, SessionEvent};
|
||||
use super::shutdown::Shutdown;
|
||||
use super::status::{
|
||||
CandidateStatus, MemberStatus, NetworkMetrics, NetworkState, NetworkStatus, PeerStatus,
|
||||
CandidateStatus, DataPath, HostnameConflict, MemberStatus, NetworkMetrics, NetworkState,
|
||||
NetworkStatus, PeerStatus,
|
||||
};
|
||||
|
||||
/// An inbound connection that already passed the handshake.
|
||||
@@ -58,6 +60,10 @@ pub(crate) enum NetCommand {
|
||||
},
|
||||
/// A peer opened a data plane link towards us.
|
||||
InboundLink(Box<InboundLink>),
|
||||
HealthProbe {
|
||||
peer: EndpointId,
|
||||
packet: bytes::Bytes,
|
||||
},
|
||||
Status {
|
||||
reply: oneshot::Sender<Box<NetworkStatus>>,
|
||||
},
|
||||
@@ -122,6 +128,7 @@ impl std::fmt::Debug for NetCommand {
|
||||
NetCommand::SetHostname(_) => f.write_str("SetHostname"),
|
||||
NetCommand::Release { .. } => f.write_str("Release"),
|
||||
NetCommand::PluginError { protocol, .. } => write!(f, "PluginError({protocol})"),
|
||||
NetCommand::HealthProbe { peer, .. } => write!(f, "HealthProbe({})", peer.fmt_short()),
|
||||
NetCommand::ProtocolFailure { peer, protocol, .. } => {
|
||||
write!(f, "ProtocolFailure({}, {protocol})", peer.fmt_short())
|
||||
}
|
||||
@@ -136,6 +143,7 @@ pub(crate) struct NetworkHandle {
|
||||
pub(crate) commands: mpsc::Sender<NetCommand>,
|
||||
shutdown: Shutdown,
|
||||
task: JoinHandle<()>,
|
||||
pub(crate) range_blocked: Arc<std::sync::atomic::AtomicBool>,
|
||||
}
|
||||
|
||||
impl NetworkHandle {
|
||||
@@ -176,17 +184,8 @@ pub(crate) struct RuntimeParams {
|
||||
/// The IPv4 overlay range this agent would use, if the network has not
|
||||
/// already settled on another one.
|
||||
pub(crate) ipv4_range: Option<Ipv4Range>,
|
||||
/// The range it was configured with but cannot have, because another
|
||||
/// network on this agent holds it.
|
||||
pub(crate) range_conflict: Option<Ipv4Range>,
|
||||
/// The range derived from this network's own id, proposed when the
|
||||
/// configured one belongs to another network here.
|
||||
///
|
||||
/// Held back at first: a network that already exists has a range of its
|
||||
/// own, and a member that proposed before hearing anything would be
|
||||
/// arguing with it instead of adopting it. Every member derives the
|
||||
/// same one, so once the wait is over there is nothing to argue about.
|
||||
pub(crate) ipv4_fallback: Option<Ipv4Range>,
|
||||
/// Coalesced notifications to the agent's lifecycle manager.
|
||||
pub(crate) range_changed: tokio::sync::watch::Sender<()>,
|
||||
/// How data plane links are opened. `None` disables the data plane.
|
||||
pub(crate) transport: Option<Arc<dyn PacketTransport>>,
|
||||
}
|
||||
@@ -235,8 +234,10 @@ pub(crate) fn spawn(params: RuntimeParams) -> NetworkHandle {
|
||||
let (commands_tx, commands_rx) = mpsc::channel(64);
|
||||
|
||||
let runtime_shutdown = shutdown.clone();
|
||||
let range_blocked = Arc::new(std::sync::atomic::AtomicBool::new(false));
|
||||
let runtime_blocked = Arc::clone(&range_blocked);
|
||||
let task = tokio::spawn(async move {
|
||||
let mut runtime = Runtime::new(params, runtime_shutdown);
|
||||
let mut runtime = Runtime::new(params, runtime_shutdown, runtime_blocked);
|
||||
runtime.run(commands_rx).await;
|
||||
});
|
||||
|
||||
@@ -245,6 +246,7 @@ pub(crate) fn spawn(params: RuntimeParams) -> NetworkHandle {
|
||||
commands: commands_tx,
|
||||
shutdown,
|
||||
task,
|
||||
range_blocked,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -256,14 +258,6 @@ pub(crate) fn spawn(params: RuntimeParams) -> NetworkHandle {
|
||||
/// itself, a few announcements later.
|
||||
const REACH_EXPIRY: Duration = Duration::from_secs(90);
|
||||
|
||||
/// How long a network waits to be told its range before proposing the one
|
||||
/// derived from its id.
|
||||
///
|
||||
/// Only for that fallback: the configured range is proposed at once, as it
|
||||
/// always was. This is the window in which an existing network's records
|
||||
/// can arrive and be adopted instead.
|
||||
const RANGE_PROPOSAL_GRACE: Duration = Duration::from_secs(3);
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
struct ProtocolDialState {
|
||||
failures: u32,
|
||||
@@ -273,6 +267,12 @@ struct ProtocolDialState {
|
||||
const DATA_LINK_BASE_BACKOFF: Duration = Duration::from_secs(3);
|
||||
const DATA_LINK_MAX_BACKOFF: Duration = Duration::from_secs(60);
|
||||
|
||||
fn next_protocol_index(supported: &[String], last_failed: Option<&str>) -> usize {
|
||||
last_failed
|
||||
.and_then(|failed| supported.iter().position(|p| p == failed))
|
||||
.map_or(0, |i| (i + 1) % supported.len().max(1))
|
||||
}
|
||||
|
||||
fn data_link_backoff(failures: u32) -> Duration {
|
||||
if failures == 0 {
|
||||
return Duration::ZERO;
|
||||
@@ -286,8 +286,9 @@ struct Runtime {
|
||||
discovery_worker: Option<DiscoveryWorker>,
|
||||
discovery_candidates: mpsc::Receiver<Candidate>,
|
||||
params: RuntimeParams,
|
||||
/// When this runtime started, for the fallback range's grace period.
|
||||
activated: std::time::Instant,
|
||||
range_blocked: Arc<std::sync::atomic::AtomicBool>,
|
||||
/// Last range successfully published to the agent's routing table.
|
||||
published_range: Option<Ipv4Range>,
|
||||
network_id: NetworkId,
|
||||
local_id: EndpointId,
|
||||
shutdown: Shutdown,
|
||||
@@ -303,7 +304,9 @@ struct Runtime {
|
||||
///
|
||||
/// The direct ones. What a protocol holds is the hub's link for that
|
||||
/// peer, which outlives any particular path.
|
||||
links: HashMap<(EndpointId, String), SharedLink>,
|
||||
links: HashMap<(EndpointId, String), Arc<HealthLink>>,
|
||||
pending_health: HashMap<(EndpointId, String), HealthSettings>,
|
||||
last_failed_protocol: HashMap<EndpointId, String>,
|
||||
/// Every data link of this network, and what is reached through what.
|
||||
hub: Arc<crate::dataplane::relay::RelayHub>,
|
||||
/// Which peers each peer says it has a live link with, and when it
|
||||
@@ -320,6 +323,10 @@ struct Runtime {
|
||||
std::time::Instant,
|
||||
),
|
||||
>,
|
||||
/// The protocols this agent last announced, in the order it announced
|
||||
/// them. The order is its preference, and the lower of two ids settles
|
||||
/// which protocol a pair runs.
|
||||
local_capabilities: Vec<PluginCapability>,
|
||||
/// What this agent last told its peers it could reach, so it is only
|
||||
/// said again when it changed.
|
||||
announced_reach: HashSet<crate::proto::message::ReachableLink>,
|
||||
@@ -338,6 +345,9 @@ struct Runtime {
|
||||
/// publish a claim afterwards — the periodic check, a peer's records
|
||||
/// arriving — would silently take it back.
|
||||
released: bool,
|
||||
/// The name this agent last said was taken, so it is said once per
|
||||
/// change rather than on every round.
|
||||
reported_name_taken: Option<HostnameConflict>,
|
||||
/// The address last reported as absent from the interface, so it is said
|
||||
/// once rather than for ever.
|
||||
reported_missing: Option<std::net::Ipv4Addr>,
|
||||
@@ -358,7 +368,11 @@ struct Runtime {
|
||||
}
|
||||
|
||||
impl Runtime {
|
||||
fn new(params: RuntimeParams, shutdown: Shutdown) -> Self {
|
||||
fn new(
|
||||
params: RuntimeParams,
|
||||
shutdown: Shutdown,
|
||||
range_blocked: Arc<std::sync::atomic::AtomicBool>,
|
||||
) -> Self {
|
||||
let network_id = params.keys.network_id();
|
||||
let local_id = params.adapter.endpoint_id();
|
||||
let (session_events_tx, session_events_rx) = mpsc::channel(256);
|
||||
@@ -366,6 +380,16 @@ impl Runtime {
|
||||
let (link_results_tx, link_results_rx) = mpsc::channel(64);
|
||||
let (candidate_tx, discovery_candidates) =
|
||||
mpsc::channel(params.limits.max_discovery_candidates.max(1));
|
||||
let hub = crate::dataplane::relay::RelayHub::new(network_id, local_id);
|
||||
// The order the protocols were configured in is also how links are
|
||||
// preferred when a neighbour can be reached over more than one.
|
||||
hub.set_priority(
|
||||
params
|
||||
.plugins
|
||||
.iter()
|
||||
.map(|plugin| plugin.protocol_id().to_string())
|
||||
.collect(),
|
||||
);
|
||||
let discovery_worker = params.discovery.as_ref().map(|backend| {
|
||||
DiscoveryWorker::spawn(
|
||||
backend.clone(),
|
||||
@@ -380,7 +404,8 @@ impl Runtime {
|
||||
discovery_worker,
|
||||
discovery_candidates,
|
||||
params,
|
||||
activated: std::time::Instant::now(),
|
||||
range_blocked,
|
||||
published_range: None,
|
||||
network_id,
|
||||
local_id,
|
||||
shutdown,
|
||||
@@ -393,8 +418,11 @@ impl Runtime {
|
||||
dial_results_tx,
|
||||
dial_results_rx,
|
||||
links: HashMap::new(),
|
||||
hub: crate::dataplane::relay::RelayHub::new(network_id, local_id),
|
||||
pending_health: HashMap::new(),
|
||||
last_failed_protocol: HashMap::new(),
|
||||
hub,
|
||||
reachable: HashMap::new(),
|
||||
local_capabilities: Vec::new(),
|
||||
announced_reach: HashSet::new(),
|
||||
opening: HashSet::new(),
|
||||
link_results_tx,
|
||||
@@ -403,6 +431,7 @@ impl Runtime {
|
||||
reported_mismatch: HashSet::new(),
|
||||
released: false,
|
||||
reported_missing: None,
|
||||
reported_name_taken: None,
|
||||
state: StateSet::new(),
|
||||
pending_state: Vec::new(),
|
||||
own_version: 0,
|
||||
@@ -478,7 +507,12 @@ impl Runtime {
|
||||
// any more, and a sender blocked on a full queue would stall shutdown.
|
||||
self.session_events_rx.close();
|
||||
self.hub.close();
|
||||
for link in self.links.values() {
|
||||
link.close();
|
||||
}
|
||||
self.links.clear();
|
||||
self.pending_health.clear();
|
||||
self.last_failed_protocol.clear();
|
||||
self.protocol_dial_states.clear();
|
||||
let peers: Vec<EndpointId> = self.sessions.keys().copied().collect();
|
||||
for peer in peers {
|
||||
@@ -582,12 +616,20 @@ impl Runtime {
|
||||
reason,
|
||||
});
|
||||
}
|
||||
NetCommand::HealthProbe { peer, packet } => {
|
||||
for ((p, _), link) in &self.links {
|
||||
if *p == peer {
|
||||
link.receive_probe(packet.clone());
|
||||
}
|
||||
}
|
||||
}
|
||||
NetCommand::ProtocolFailure {
|
||||
peer,
|
||||
protocol,
|
||||
reason,
|
||||
} => {
|
||||
if self.links.remove(&(peer, protocol.clone())).is_some() {
|
||||
if let Some(link) = self.links.remove(&(peer, protocol.clone())) {
|
||||
link.close();
|
||||
self.hub.clear_direct(peer, &protocol);
|
||||
}
|
||||
self.metrics.data_link_failures += 1;
|
||||
@@ -617,6 +659,8 @@ impl Runtime {
|
||||
tracing::debug!(%err, "could not queue re-announcement");
|
||||
}
|
||||
}
|
||||
// What was announced decides which protocol each pair runs.
|
||||
self.update_paths();
|
||||
}
|
||||
|
||||
/// Queues a message without blocking the runtime loop.
|
||||
@@ -640,7 +684,7 @@ impl Runtime {
|
||||
peer: peer.fmt_short().to_string(),
|
||||
})?;
|
||||
|
||||
match session.outbound.try_send(encoded) {
|
||||
match session.outbound.try_send(zeroize::Zeroizing::new(encoded)) {
|
||||
Ok(()) => {
|
||||
session.messages_sent += 1;
|
||||
session.bytes_sent += bytes;
|
||||
@@ -669,6 +713,7 @@ impl Runtime {
|
||||
// alone, so something has to look again; this runs on a timer and
|
||||
// the check is a comparison when nothing has changed.
|
||||
self.ensure_own_claim().await;
|
||||
self.report_hostname_taken();
|
||||
// The same timer repairs the relay picture: reachability is soft
|
||||
// state, repeated while it holds and forgotten when it stops.
|
||||
self.update_paths();
|
||||
@@ -945,35 +990,27 @@ impl Runtime {
|
||||
if let Some(agreed) = self.state.agreed_range() {
|
||||
return Some(agreed);
|
||||
}
|
||||
// Nobody has settled one yet, so this agent would be proposing its
|
||||
// own. It must not propose a range it cannot route: one agent has
|
||||
// one interface, and claiming an address in a range another network
|
||||
// already owns would spread the collision rather than contain it —
|
||||
// "the lowest author's range wins" would carry it to everybody.
|
||||
if let Some(wanted) = self.params.ipv4_range
|
||||
self.params.ipv4_range
|
||||
}
|
||||
|
||||
/// A late signed range can conflict even though activation was safe.
|
||||
/// Fail closed before touching claims, plugins or the host interface.
|
||||
fn suspend_if_overlapping(&self) -> bool {
|
||||
if let Some(range) = self.effective_range()
|
||||
&& self
|
||||
.params
|
||||
.routes
|
||||
.would_overlap(self.network_id, wanted)
|
||||
.is_none()
|
||||
.would_overlap(self.network_id, range)
|
||||
.is_some()
|
||||
{
|
||||
return Some(wanted);
|
||||
self.range_blocked
|
||||
.store(true, std::sync::atomic::Ordering::Release);
|
||||
self.params.routes.remove_network(self.network_id);
|
||||
self.shutdown.trigger();
|
||||
self.params.range_changed.send_replace(());
|
||||
return true;
|
||||
}
|
||||
|
||||
// The configured range is another network's here, so this one falls
|
||||
// back to the range derived from its id — after a moment's wait, so
|
||||
// that a network which already exists gets to say what it uses
|
||||
// first. Every member derives the same one, so members that reach
|
||||
// this point agree without negotiating.
|
||||
let fallback = self.params.ipv4_fallback?;
|
||||
if self.activated.elapsed() < RANGE_PROPOSAL_GRACE {
|
||||
return None;
|
||||
}
|
||||
self.params
|
||||
.routes
|
||||
.would_overlap(self.network_id, fallback)
|
||||
.is_none()
|
||||
.then_some(fallback)
|
||||
false
|
||||
}
|
||||
|
||||
/// Makes sure this agent holds an address, claiming one if it does not.
|
||||
@@ -983,7 +1020,7 @@ impl Runtime {
|
||||
async fn ensure_own_claim(&mut self) {
|
||||
// On the way out. Claiming again here is how a goodbye becomes a
|
||||
// hello nobody asked for.
|
||||
if self.released {
|
||||
if self.released || self.suspend_if_overlapping() {
|
||||
return;
|
||||
}
|
||||
let wanted_hostname = {
|
||||
@@ -1031,12 +1068,25 @@ impl Runtime {
|
||||
}
|
||||
};
|
||||
|
||||
// A name keeps the time it was first claimed for as long as this
|
||||
// member keeps the name, so a later claimant cannot overtake it by
|
||||
// the passage of time, and a new or changed name starts the clock
|
||||
// afresh.
|
||||
let hostname_since = wanted_hostname.as_ref().map(|name| {
|
||||
let kept = self
|
||||
.state
|
||||
.get(&self.local_id)
|
||||
.filter(|record| record.body.hostname() == Some(name.as_str()))
|
||||
.and_then(|record| record.body.hostname_since());
|
||||
kept.unwrap_or_else(unix_millis)
|
||||
});
|
||||
let body = RecordBody::Claim {
|
||||
address: wanted_address,
|
||||
// The range only travels with an address, so a member of an
|
||||
// IPv6-only network does not assert one.
|
||||
range: wanted_address.and(range),
|
||||
hostname: wanted_hostname,
|
||||
hostname_since,
|
||||
};
|
||||
|
||||
// Nothing to say is not the same as saying nothing changed: a member
|
||||
@@ -1100,6 +1150,7 @@ impl Runtime {
|
||||
for (peer, records) in std::mem::take(&mut self.pending_state) {
|
||||
self.receive_state(peer, records).await;
|
||||
}
|
||||
self.report_hostname_taken();
|
||||
}
|
||||
|
||||
/// Sends everything we know to every peer.
|
||||
@@ -1259,6 +1310,9 @@ impl Runtime {
|
||||
|
||||
/// Tells the plugins who holds which overlay address.
|
||||
async fn publish_allocations(&mut self) {
|
||||
if self.shutdown.is_triggered() || self.suspend_if_overlapping() {
|
||||
return;
|
||||
}
|
||||
let Some(range) = self.effective_range() else {
|
||||
return;
|
||||
};
|
||||
@@ -1270,10 +1324,6 @@ impl Runtime {
|
||||
.collect();
|
||||
allocations.sort_by_key(|(holder, _)| *holder.as_bytes());
|
||||
|
||||
for plugin in &self.params.plugins {
|
||||
plugin.on_address_allocation(self.network_id, range, &allocations);
|
||||
}
|
||||
|
||||
// And the system level's own view, which is what decides whose
|
||||
// packet is whose. The local address is kept out of the peer list:
|
||||
// a packet for ourselves does not go over a tunnel.
|
||||
@@ -1289,17 +1339,27 @@ impl Runtime {
|
||||
.map(|(holder, address)| (*address, *holder))
|
||||
.collect(),
|
||||
};
|
||||
if let Err(err) = self.params.routes.set_network(self.network_id, routes) {
|
||||
// Reported once per change rather than swallowed: two networks
|
||||
// wanting the same addresses is a thing the user has to settle.
|
||||
self.metrics.plugin_errors += 1;
|
||||
self.emit(Event::PluginError {
|
||||
network: self.network_id,
|
||||
protocol: "overlay".into(),
|
||||
reason: err.to_string(),
|
||||
});
|
||||
if self
|
||||
.params
|
||||
.routes
|
||||
.set_network(self.network_id, routes)
|
||||
.is_err()
|
||||
{
|
||||
self.range_blocked
|
||||
.store(true, std::sync::atomic::Ordering::Release);
|
||||
self.params.routes.remove_network(self.network_id);
|
||||
self.shutdown.trigger();
|
||||
self.params.range_changed.send_replace(());
|
||||
return;
|
||||
}
|
||||
for plugin in &self.params.plugins {
|
||||
plugin.on_address_allocation(self.network_id, range, &allocations);
|
||||
}
|
||||
if self.published_range != Some(range) {
|
||||
self.published_range = Some(range);
|
||||
self.params.range_changed.send_replace(());
|
||||
}
|
||||
|
||||
let Some(interface) = self.params.interface.clone() else {
|
||||
return;
|
||||
};
|
||||
@@ -1392,6 +1452,7 @@ impl Runtime {
|
||||
}
|
||||
|
||||
fn record_protocol_failure(&mut self, peer: EndpointId, protocol: &str) {
|
||||
self.last_failed_protocol.insert(peer, protocol.to_string());
|
||||
let key = (peer, protocol.to_string());
|
||||
let entry = self
|
||||
.protocol_dial_states
|
||||
@@ -1430,7 +1491,17 @@ impl Runtime {
|
||||
}
|
||||
}
|
||||
for (peer, protocol) in dead {
|
||||
self.links.remove(&(peer, protocol.clone()));
|
||||
let reason = self
|
||||
.links
|
||||
.remove(&(peer, protocol.clone()))
|
||||
.map(|link| {
|
||||
let reason = link
|
||||
.failure_reason()
|
||||
.unwrap_or_else(|| "link closed".into());
|
||||
link.close();
|
||||
reason
|
||||
})
|
||||
.unwrap_or_else(|| "link closed".into());
|
||||
// Only the direct path is gone. Whether the peer is still
|
||||
// reachable through somebody is decided below; the protocol's
|
||||
// link stays either way.
|
||||
@@ -1440,7 +1511,7 @@ impl Runtime {
|
||||
network: self.network_id,
|
||||
peer,
|
||||
protocol,
|
||||
reason: "link closed".into(),
|
||||
reason,
|
||||
});
|
||||
}
|
||||
|
||||
@@ -1481,12 +1552,21 @@ impl Runtime {
|
||||
continue;
|
||||
}
|
||||
|
||||
let chosen = supported.iter().find(|proto| {
|
||||
match self.protocol_dial_states.get(&(peer, (*proto).clone())) {
|
||||
Some(state) => state.cooldown_until <= now,
|
||||
None => true,
|
||||
}
|
||||
});
|
||||
let start = next_protocol_index(
|
||||
&supported,
|
||||
self.last_failed_protocol.get(&peer).map(String::as_str),
|
||||
);
|
||||
let chosen = supported
|
||||
.iter()
|
||||
.cycle()
|
||||
.skip(start)
|
||||
.take(supported.len())
|
||||
.find(
|
||||
|proto| match self.protocol_dial_states.get(&(peer, (*proto).clone())) {
|
||||
Some(state) => state.cooldown_until <= now,
|
||||
None => true,
|
||||
},
|
||||
);
|
||||
|
||||
if let Some(proto) = chosen {
|
||||
wanted.push((peer, proto.clone()));
|
||||
@@ -1658,68 +1738,93 @@ impl Runtime {
|
||||
}
|
||||
}
|
||||
|
||||
/// The protocol this agent and a member run their tunnel in, if they have
|
||||
/// one in common.
|
||||
///
|
||||
/// Both ends must name the same one without asking each other, and what
|
||||
/// each side was configured to prefer can differ. So the member with the
|
||||
/// lower id decides, by the order it announces its protocols in, and the
|
||||
/// other side reads that order from the announcement it already has.
|
||||
fn tunnel_protocol(&self, session: &Session) -> Option<String> {
|
||||
let usable = |ours: &[PluginCapability], theirs: &[PluginCapability], name: &str| {
|
||||
ours.iter().any(|capability| {
|
||||
capability.enabled
|
||||
&& capability.protocol == name
|
||||
&& theirs.iter().any(|other| {
|
||||
other.enabled
|
||||
&& other.protocol == name
|
||||
&& other.version == capability.version
|
||||
})
|
||||
})
|
||||
};
|
||||
let (decides, other) = if self.local_id.as_bytes() < session.peer.as_bytes() {
|
||||
(&self.local_capabilities, &session.capabilities)
|
||||
} else {
|
||||
(&session.capabilities, &self.local_capabilities)
|
||||
};
|
||||
decides
|
||||
.iter()
|
||||
.find(|capability| usable(decides, other, &capability.protocol))
|
||||
.map(|capability| capability.protocol.clone())
|
||||
}
|
||||
|
||||
/// Builds the graph off the packet path. Every remote row comes from that
|
||||
/// member's authenticated control session; stale or incompatible rows never
|
||||
/// enter the graph. The hub replaces an immutable table only on changes.
|
||||
/// member's authenticated control session, and a row says only who that
|
||||
/// member has a link to, never what kind of link: a member in the middle
|
||||
/// needs no protocol in common with either end. Stale rows never enter the
|
||||
/// graph. The hub replaces an immutable table only on changes.
|
||||
fn update_paths(&mut self) {
|
||||
self.reachable.retain(|peer, (_, heard)| {
|
||||
heard.elapsed() < REACH_EXPIRY && self.sessions.contains_key(peer)
|
||||
});
|
||||
for (protocol, version) in self.served_protocols() {
|
||||
let peers: Vec<_> = self
|
||||
.sessions
|
||||
.values()
|
||||
.filter(|session| {
|
||||
session.capabilities.iter().any(|capability| {
|
||||
capability.enabled
|
||||
&& capability.protocol == protocol
|
||||
&& capability.version == version
|
||||
})
|
||||
})
|
||||
.map(|session| session.peer)
|
||||
.collect();
|
||||
let mut members: HashSet<_> = peers.iter().map(|peer| *peer.as_bytes()).collect();
|
||||
members.insert(*self.local_id.as_bytes());
|
||||
let graph = self
|
||||
.reachable
|
||||
.iter()
|
||||
.filter(|(peer, _)| members.contains(peer.as_bytes()))
|
||||
.map(|(peer, (links, _))| {
|
||||
let mut neighbors: Vec<_> = links
|
||||
.iter()
|
||||
.filter(|link| {
|
||||
link.protocol == protocol
|
||||
&& members.contains(&link.peer)
|
||||
&& link.peer != *peer.as_bytes()
|
||||
})
|
||||
.map(|link| link.peer)
|
||||
.collect();
|
||||
neighbors.sort_unstable();
|
||||
(*peer.as_bytes(), neighbors)
|
||||
})
|
||||
.collect();
|
||||
self.hub.set_topology(&protocol, graph, members);
|
||||
for peer in peers {
|
||||
if self.hub.has_link(peer, &protocol) || !self.hub.reachable(peer, &protocol) {
|
||||
continue;
|
||||
}
|
||||
let Some(plugin) = self
|
||||
.params
|
||||
.plugins
|
||||
let mut members: HashSet<_> = self.sessions.keys().map(|peer| *peer.as_bytes()).collect();
|
||||
members.insert(*self.local_id.as_bytes());
|
||||
let graph = self
|
||||
.reachable
|
||||
.iter()
|
||||
.filter(|(peer, _)| members.contains(peer.as_bytes()))
|
||||
.map(|(peer, (links, _))| {
|
||||
let mut neighbors: Vec<_> = links
|
||||
.iter()
|
||||
.find(|plugin| plugin.protocol_id() == protocol)
|
||||
.cloned()
|
||||
else {
|
||||
continue;
|
||||
};
|
||||
let link = self.hub.link(peer, &protocol);
|
||||
let path = link.path_description();
|
||||
let max_datagram = link.max_datagram_size();
|
||||
plugin.on_peer_link(self.network_id, peer, link);
|
||||
.filter(|link| members.contains(&link.peer) && link.peer != *peer.as_bytes())
|
||||
.map(|link| link.peer)
|
||||
.collect();
|
||||
neighbors.sort_unstable();
|
||||
neighbors.dedup();
|
||||
(*peer.as_bytes(), neighbors)
|
||||
})
|
||||
.collect();
|
||||
self.hub.set_topology(graph, members);
|
||||
|
||||
let wanted: Vec<(EndpointId, String)> = self
|
||||
.sessions
|
||||
.values()
|
||||
.filter_map(|session| Some((session.peer, self.tunnel_protocol(session)?)))
|
||||
.collect();
|
||||
for (peer, protocol) in wanted {
|
||||
if self.hub.has_link(peer, &protocol) || !self.hub.reachable(peer) {
|
||||
continue;
|
||||
}
|
||||
let Some(plugin) = self
|
||||
.params
|
||||
.plugins
|
||||
.iter()
|
||||
.find(|plugin| plugin.protocol_id() == protocol)
|
||||
.cloned()
|
||||
else {
|
||||
continue;
|
||||
};
|
||||
let link = self.hub.link(peer, &protocol);
|
||||
let relayed = self.hub.route(peer).is_some_and(|route| route.hops > 1);
|
||||
let path = link.path_description();
|
||||
let max_datagram = link.max_datagram_size();
|
||||
plugin.on_peer_link(self.network_id, peer, link);
|
||||
// A direct link has already been reported as one.
|
||||
if relayed {
|
||||
self.emit(Event::DataLinkUp {
|
||||
network: self.network_id,
|
||||
peer,
|
||||
protocol: protocol.clone(),
|
||||
protocol,
|
||||
path,
|
||||
max_datagram,
|
||||
});
|
||||
@@ -1775,7 +1880,11 @@ impl Runtime {
|
||||
}
|
||||
|
||||
fn install_link(&mut self, inbound: InboundLink) {
|
||||
if self.no_direct_data(inbound.peer) {
|
||||
if self.no_direct_data(inbound.peer)
|
||||
|| inbound.link.is_closed()
|
||||
|| !self.sessions.contains_key(&inbound.peer)
|
||||
{
|
||||
inbound.link.close();
|
||||
return;
|
||||
}
|
||||
self.record_protocol_success(inbound.peer, &inbound.protocol);
|
||||
@@ -1786,30 +1895,65 @@ impl Runtime {
|
||||
|
||||
/// Hands a link to the plugin that owns its protocol.
|
||||
fn adopt_link(&mut self, peer: EndpointId, protocol: String, link: SharedLink) {
|
||||
let Some(plugin) = self
|
||||
if link.is_closed()
|
||||
|| link.network() != self.network_id
|
||||
|| link.peer() != peer
|
||||
|| !self.sessions.contains_key(&peer)
|
||||
{
|
||||
link.close();
|
||||
return;
|
||||
}
|
||||
if !self
|
||||
.params
|
||||
.plugins
|
||||
.iter()
|
||||
.find(|plugin| plugin.protocol_id() == protocol)
|
||||
.cloned()
|
||||
else {
|
||||
.any(|plugin| plugin.protocol_id() == protocol)
|
||||
{
|
||||
return;
|
||||
};
|
||||
|
||||
let path = link.path_description();
|
||||
self.links
|
||||
.insert((peer, protocol.clone()), Arc::clone(&link));
|
||||
// The protocol is handed the hub's link, not this one: the same
|
||||
// object for as long as the peer is a peer, whatever happens to
|
||||
// the path under it. A tunnel is not torn down because a direct
|
||||
// link came or went.
|
||||
let first_time = !self.hub.has_link(peer, &protocol);
|
||||
self.hub.set_direct(peer, &protocol, link);
|
||||
let shared = self.hub.link(peer, &protocol);
|
||||
let max_datagram = shared.max_datagram_size();
|
||||
if first_time {
|
||||
plugin.on_peer_link(self.network_id, peer, shared);
|
||||
}
|
||||
|
||||
let key = (peer, protocol.clone());
|
||||
let settings = if self.local_id.as_bytes() < peer.as_bytes() {
|
||||
let settings = HealthSettings::fresh();
|
||||
if let Ok(bytes) = postcard::to_stdvec(&(protocol.clone(), settings.clone())) {
|
||||
let _ = self.send_to(
|
||||
peer,
|
||||
ControlMessage::Transport {
|
||||
protocol: "link-health".into(),
|
||||
payload: crate::proto::message::TransportPayload::new(bytes),
|
||||
},
|
||||
);
|
||||
}
|
||||
Some(settings)
|
||||
} else {
|
||||
self.pending_health.remove(&key)
|
||||
};
|
||||
let route = self
|
||||
.sessions
|
||||
.get(&peer)
|
||||
.and_then(|s| self.tunnel_protocol(s))
|
||||
.and_then(|name| self.params.plugins.iter().find(|p| p.protocol_id() == name))
|
||||
.map(|plugin| ProbeRoute {
|
||||
plugin: plugin.clone(),
|
||||
routes: self.params.routes.clone(),
|
||||
network: self.network_id,
|
||||
peer,
|
||||
max_probe_size: self.params.interface.as_ref().map_or(540, |interface| {
|
||||
(interface.mtu() as usize).saturating_sub(36)
|
||||
}),
|
||||
});
|
||||
let link = HealthLink::new(link, self.local_id, settings, route);
|
||||
let path = link.path_description();
|
||||
let max_datagram = link.max_datagram_size();
|
||||
if let Some(old) = self.links.insert(key, Arc::clone(&link)) {
|
||||
old.close();
|
||||
}
|
||||
// What this is, is a way to reach the peer. Which protocol's
|
||||
// tunnel runs over it is decided in `update_paths`, and that tunnel
|
||||
// is handed the hub's link, not this one: the same object for as
|
||||
// long as the peer is a peer, whatever happens to the path under
|
||||
// it. A tunnel is not torn down because a direct link came or went.
|
||||
self.hub.set_direct(peer, &protocol, link);
|
||||
self.metrics.data_links_established += 1;
|
||||
self.emit(Event::DataLinkUp {
|
||||
network: self.network_id,
|
||||
@@ -1827,6 +1971,8 @@ impl Runtime {
|
||||
self.hub.remove_peer(peer);
|
||||
self.reachable.remove(&peer);
|
||||
self.protocol_dial_states.retain(|(p, _), _| *p != peer);
|
||||
self.last_failed_protocol.remove(&peer);
|
||||
self.pending_health.retain(|(p, _), _| *p != peer);
|
||||
let keys: Vec<(EndpointId, String)> = self
|
||||
.links
|
||||
.keys()
|
||||
@@ -1834,7 +1980,9 @@ impl Runtime {
|
||||
.cloned()
|
||||
.collect();
|
||||
for key in keys {
|
||||
self.links.remove(&key);
|
||||
if let Some(link) = self.links.remove(&key) {
|
||||
link.close();
|
||||
}
|
||||
self.emit(Event::DataLinkDown {
|
||||
network: self.network_id,
|
||||
peer,
|
||||
@@ -1941,6 +2089,13 @@ impl Runtime {
|
||||
});
|
||||
}
|
||||
capabilities.truncate(self.params.limits.max_capabilities);
|
||||
self.local_capabilities = capabilities
|
||||
.iter()
|
||||
.map(|capability| PluginCapability {
|
||||
data: Vec::new(),
|
||||
..capability.clone()
|
||||
})
|
||||
.collect();
|
||||
let exit_node = self.exit_ready();
|
||||
self.announced_exit = exit_node;
|
||||
Announcement {
|
||||
@@ -2076,6 +2231,50 @@ impl Runtime {
|
||||
|
||||
fn dispatch_message(&mut self, peer: EndpointId, message: ControlMessage) {
|
||||
match &message {
|
||||
ControlMessage::Transport { protocol, payload } => {
|
||||
if protocol == "link-health" {
|
||||
// Only the elected initiator supplies a profile, for an enabled,
|
||||
// mutually supported physical protocol in this authenticated session.
|
||||
if peer.as_bytes() >= self.local_id.as_bytes() {
|
||||
return;
|
||||
}
|
||||
let Ok((name, settings)) =
|
||||
postcard::from_bytes::<(String, HealthSettings)>(payload.as_bytes())
|
||||
else {
|
||||
return;
|
||||
};
|
||||
if name.len() > crate::dataplane::MAX_PROTOCOL_ID_LEN
|
||||
|| !self.served_protocols().iter().any(|(p, v)| {
|
||||
p == &name
|
||||
&& self.sessions.get(&peer).is_some_and(|s| {
|
||||
s.capabilities
|
||||
.iter()
|
||||
.any(|c| c.enabled && c.protocol == name && c.version == *v)
|
||||
})
|
||||
})
|
||||
{
|
||||
return;
|
||||
}
|
||||
let key = (peer, name);
|
||||
// Negotiation and inbound installation are separate queues.
|
||||
// Keep the newest settings even if the previous physical
|
||||
// link is still in the map when this control frame arrives.
|
||||
if let Some(link) = self.links.get(&key)
|
||||
&& !link.is_closed()
|
||||
{
|
||||
link.configure(settings.clone());
|
||||
}
|
||||
self.pending_health.insert(key, settings);
|
||||
return;
|
||||
}
|
||||
if let Some(transport) = &self.params.transport
|
||||
&& let Err(err) =
|
||||
transport.on_control(self.network_id, peer, protocol, payload.as_bytes())
|
||||
{
|
||||
tracing::debug!(%err, "transport negotiation rejected");
|
||||
}
|
||||
return;
|
||||
}
|
||||
ControlMessage::Announce(announcement) => {
|
||||
let capabilities = announcement.capabilities.clone();
|
||||
if let Some(session) = self.sessions.get_mut(&peer) {
|
||||
@@ -2175,6 +2374,69 @@ impl Runtime {
|
||||
|
||||
// ----------------------------------------------------------------- status
|
||||
|
||||
/// Whether a member can be reached now, by any path.
|
||||
///
|
||||
/// The single definition every client reports: a live control session,
|
||||
/// or a data path to it through other members. Being reached over a long
|
||||
/// path is being online; how well is `data_path`'s to say.
|
||||
fn is_online(&self, peer: EndpointId) -> bool {
|
||||
peer == self.local_id || self.sessions.contains_key(&peer) || self.hub.reachable(peer)
|
||||
}
|
||||
|
||||
/// How packets reach a member, which says nothing about whether it is
|
||||
/// there.
|
||||
fn data_path(&self, peer: EndpointId) -> DataPath {
|
||||
match self.hub.route(peer) {
|
||||
None => DataPath::None,
|
||||
Some(route) if route.hops <= 1 => DataPath::Direct {
|
||||
transport: self
|
||||
.hub
|
||||
.direct_kinds(peer)
|
||||
.into_iter()
|
||||
.next()
|
||||
.unwrap_or_default(),
|
||||
},
|
||||
Some(route) => DataPath::Relayed {
|
||||
hops: route.hops,
|
||||
via: route.via,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
/// Says so when this agent's own name is held by someone else, and
|
||||
/// again if that changes. Never anything but a statement: the agent keeps
|
||||
/// working by address either way.
|
||||
fn report_hostname_taken(&mut self) {
|
||||
let taken = self.hostname_conflict_of(&self.local_id);
|
||||
if taken == self.reported_name_taken {
|
||||
return;
|
||||
}
|
||||
if let Some(conflict) = &taken {
|
||||
tracing::warn!(
|
||||
network = %self.network_id,
|
||||
name = %conflict.name,
|
||||
holder = %conflict.holder.fmt_short(),
|
||||
"this name was claimed first by another member; it stays reachable by \
|
||||
address but is not found by this name until it is renamed"
|
||||
);
|
||||
self.emit(Event::HostnameTaken {
|
||||
network: self.network_id,
|
||||
name: conflict.name.clone(),
|
||||
holder: conflict.holder,
|
||||
});
|
||||
}
|
||||
self.reported_name_taken = taken;
|
||||
}
|
||||
|
||||
fn hostname_conflict_of(&self, author: &EndpointId) -> Option<HostnameConflict> {
|
||||
self.state
|
||||
.hostname_taken(author)
|
||||
.map(|(name, holder)| HostnameConflict {
|
||||
name: name.to_string(),
|
||||
holder,
|
||||
})
|
||||
}
|
||||
|
||||
fn status(&self) -> NetworkStatus {
|
||||
let served = self.served_protocols();
|
||||
let mut peers: Vec<PeerStatus> = self
|
||||
@@ -2212,6 +2474,7 @@ impl Runtime {
|
||||
control_messages_received: session.messages_received,
|
||||
control_bytes_sent: session.bytes_sent,
|
||||
control_bytes_received: session.bytes_received,
|
||||
data_path: self.data_path(session.peer),
|
||||
}
|
||||
})
|
||||
.collect();
|
||||
@@ -2249,6 +2512,9 @@ impl Runtime {
|
||||
overlay_address_v4: self.state.address_of(&endpoint_id),
|
||||
hostname: self.state.hostname_of(&endpoint_id).map(str::to_string),
|
||||
last_hostname: self.known_hostnames.get(&endpoint_id).cloned(),
|
||||
hostname_conflict: self.hostname_conflict_of(&endpoint_id),
|
||||
online: self.is_online(endpoint_id),
|
||||
data_path: self.data_path(endpoint_id),
|
||||
})
|
||||
})
|
||||
.collect();
|
||||
@@ -2256,6 +2522,7 @@ impl Runtime {
|
||||
members.dedup_by(|a, b| a.endpoint_id == b.endpoint_id);
|
||||
|
||||
NetworkStatus {
|
||||
auto_start: true,
|
||||
broadcast: self.params.broadcast,
|
||||
exit_offer: self.params.exit_offer,
|
||||
exit_via: self.params.exit_via,
|
||||
@@ -2276,20 +2543,25 @@ impl Runtime {
|
||||
peers,
|
||||
candidates,
|
||||
members,
|
||||
hostname_conflict: self.hostname_conflict_of(&self.local_id),
|
||||
range: self.effective_range(),
|
||||
// Only worth reporting while it is actually stuck: with a
|
||||
// fallback in play the configured range being another
|
||||
// network's is how it is meant to work, not a fault.
|
||||
range_conflict: self
|
||||
.params
|
||||
.range_conflict
|
||||
.filter(|_| self.effective_range().is_none()),
|
||||
range_conflict: None,
|
||||
range_conflicts: Vec::new(),
|
||||
metrics: self.metrics.clone(),
|
||||
relay: self.hub.counters(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Milliseconds since the Unix epoch, for the time a name was first claimed.
|
||||
fn unix_millis() -> u64 {
|
||||
std::time::SystemTime::now()
|
||||
.duration_since(std::time::UNIX_EPOCH)
|
||||
.map_or(u64::MAX, |elapsed| {
|
||||
u64::try_from(elapsed.as_millis()).unwrap_or(u64::MAX)
|
||||
})
|
||||
}
|
||||
|
||||
/// Which side dialled, given a role and the two identities.
|
||||
fn initiator_of(role: Role, local: EndpointId, peer: EndpointId) -> EndpointId {
|
||||
match role {
|
||||
@@ -2521,3 +2793,17 @@ mod relay_host_tests {
|
||||
assert_eq!(relay_host("https://bad host/").as_deref(), None);
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod fallback_order_tests {
|
||||
use super::next_protocol_index;
|
||||
#[test]
|
||||
fn degradation_advances_the_cycle_even_after_earlier_cooldowns_expire() {
|
||||
let protocols = ["wg", "awg", "wg-quic", "tcp-tls"].map(str::to_owned);
|
||||
assert_eq!(next_protocol_index(&protocols, None), 0);
|
||||
for (i, p) in protocols.iter().enumerate() {
|
||||
assert_eq!(next_protocol_index(&protocols, Some(p)), (i + 1) % 4);
|
||||
}
|
||||
assert_eq!(next_protocol_index(&protocols, Some("removed")), 0);
|
||||
}
|
||||
}
|
||||
@@ -70,7 +70,7 @@ pub(crate) struct Session {
|
||||
pub(crate) conn: Connection,
|
||||
/// Already encoded frame payloads, so the runtime can account for the exact
|
||||
/// number of control bytes it queues.
|
||||
pub(crate) outbound: mpsc::Sender<Vec<u8>>,
|
||||
pub(crate) outbound: mpsc::Sender<zeroize::Zeroizing<Vec<u8>>>,
|
||||
pub(crate) hostname: Option<String>,
|
||||
pub(crate) capabilities: Vec<PluginCapability>,
|
||||
pub(crate) broadcast: bool,
|
||||
@@ -244,7 +244,7 @@ pub(crate) fn spawn(
|
||||
|
||||
async fn writer_task(
|
||||
mut send: SendStream,
|
||||
mut outbound: mpsc::Receiver<Vec<u8>>,
|
||||
mut outbound: mpsc::Receiver<zeroize::Zeroizing<Vec<u8>>>,
|
||||
limits: Arc<Limits>,
|
||||
shutdown: Shutdown,
|
||||
parent: Shutdown,
|
||||
@@ -331,6 +331,7 @@ async fn reader_task(
|
||||
}
|
||||
};
|
||||
|
||||
let payload = zeroize::Zeroizing::new(payload);
|
||||
let bytes = payload.len();
|
||||
let envelope: Envelope = match decode(&payload) {
|
||||
Ok(envelope) => envelope,
|
||||
|
||||
@@ -59,6 +59,43 @@ pub struct OverlayStatus {
|
||||
pub broadcast_rules: Option<crate::overlay::BroadcastRulesReport>,
|
||||
}
|
||||
|
||||
/// How packets reach a member right now.
|
||||
///
|
||||
/// Being reachable is one thing and being reachable well is another, and
|
||||
/// the second is never a reason to call a member away: a member that can only
|
||||
/// be reached through somebody else is online, over a longer path.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub enum DataPath {
|
||||
/// Nothing carries packets to it yet. The member is connected for
|
||||
/// control only, which is the state of a data path being set up.
|
||||
None,
|
||||
/// A direct link, of the named kind of transport.
|
||||
Direct {
|
||||
/// The kind of transport carrying it, e.g. `wg`.
|
||||
transport: String,
|
||||
},
|
||||
/// Through other members.
|
||||
Relayed {
|
||||
/// Number of links on the way, at least two.
|
||||
hops: u8,
|
||||
/// The member the first of them goes to.
|
||||
via: EndpointId,
|
||||
},
|
||||
}
|
||||
|
||||
/// A name that a member claimed and somebody else holds.
|
||||
///
|
||||
/// A name belongs to whoever claimed it first. The member that claimed it
|
||||
/// later is still a member and still reachable by its address; it is just
|
||||
/// not found by that name, on any member of the network.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct HostnameConflict {
|
||||
/// The name that is taken.
|
||||
pub name: String,
|
||||
/// The member that holds it.
|
||||
pub holder: EndpointId,
|
||||
}
|
||||
|
||||
/// A member the signed state knows about, connected or not.
|
||||
///
|
||||
/// This is the durable roster: it comes from signed records, so a member that
|
||||
@@ -85,6 +122,15 @@ pub struct MemberStatus {
|
||||
/// Not signed, so it is a convenience for naming a member that is away
|
||||
/// and never a claim: the zone is served from `hostname`.
|
||||
pub last_hostname: Option<String>,
|
||||
/// The name it claimed, when somebody else claimed it first.
|
||||
pub hostname_conflict: Option<HostnameConflict>,
|
||||
/// Whether this agent can reach it now, by any path: a live control
|
||||
/// session, or a data path through other members. The one answer to "is
|
||||
/// it online" for every client, so that none of them has to work it out
|
||||
/// from a different part of the picture. This agent counts as online.
|
||||
pub online: bool,
|
||||
/// How packets reach it. [`DataPath::None`] while it is offline.
|
||||
pub data_path: DataPath,
|
||||
}
|
||||
|
||||
/// Status of one authenticated session.
|
||||
@@ -128,6 +174,8 @@ pub struct PeerStatus {
|
||||
pub control_bytes_sent: u64,
|
||||
/// Control payload bytes read from this session.
|
||||
pub control_bytes_received: u64,
|
||||
/// How packets reach this peer right now.
|
||||
pub data_path: DataPath,
|
||||
}
|
||||
|
||||
/// Counters scoped to one logical network.
|
||||
@@ -161,9 +209,22 @@ pub struct NetworkMetrics {
|
||||
pub data_link_failures: u64,
|
||||
}
|
||||
|
||||
/// An active network preventing another network from starting locally.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct RangeConflict {
|
||||
/// The blocking network's public identifier.
|
||||
pub network_id: NetworkId,
|
||||
/// Its configured name.
|
||||
pub name: NetworkName,
|
||||
/// Its overlapping overlay range.
|
||||
pub range: crate::state::Ipv4Range,
|
||||
}
|
||||
|
||||
/// Status of one network.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct NetworkStatus {
|
||||
/// Whether this network is requested to start or resume automatically.
|
||||
pub auto_start: bool,
|
||||
/// Local broadcast participation in this network.
|
||||
pub broadcast: bool,
|
||||
/// This agent offers itself as an exit node in this network.
|
||||
@@ -188,15 +249,16 @@ pub struct NetworkStatus {
|
||||
pub candidates: Vec<CandidateStatus>,
|
||||
/// Members the signed state knows about, whether connected or not.
|
||||
pub members: Vec<MemberStatus>,
|
||||
/// This agent's own name, when another member claimed it first and so
|
||||
/// holds it. This agent keeps working by address; nobody finds it by
|
||||
/// this name until it is renamed or the other member lets it go.
|
||||
pub hostname_conflict: Option<HostnameConflict>,
|
||||
/// The overlay range this network uses, once it has one.
|
||||
pub range: Option<crate::state::Ipv4Range>,
|
||||
/// The range it could not have, because another network on this agent
|
||||
/// already holds it.
|
||||
///
|
||||
/// One agent has one interface, so an address belongs to one network.
|
||||
/// This network waits to adopt whatever its members settle on instead of
|
||||
/// proposing something it could not route.
|
||||
/// The first overlapping active range; `range_conflicts` identifies all blockers.
|
||||
pub range_conflict: Option<crate::state::Ipv4Range>,
|
||||
/// Active networks that would be suspended by switching to this network.
|
||||
pub range_conflicts: Vec<RangeConflict>,
|
||||
/// Per-network counters.
|
||||
pub metrics: NetworkMetrics,
|
||||
/// What has gone through a peer in the middle, in both directions.
|
||||
|
||||
@@ -166,6 +166,8 @@ pub struct Limits {
|
||||
pub max_capabilities: usize,
|
||||
/// Largest opaque plugin payload, in bytes.
|
||||
pub max_capability_data_len: usize,
|
||||
/// Largest private transport negotiation payload (also hard-capped at 4096 bytes).
|
||||
pub max_transport_payload_len: usize,
|
||||
/// Largest accepted echo payload in a ping/pong exchange, in bytes.
|
||||
pub max_echo_payload_len: usize,
|
||||
/// Largest accepted free-text reason string, in bytes.
|
||||
@@ -204,6 +206,7 @@ impl Default for Limits {
|
||||
max_frame_len: 64 * 1024,
|
||||
max_hostname_len: 255,
|
||||
max_capabilities: 16,
|
||||
max_transport_payload_len: 4096,
|
||||
max_capability_data_len: 4 * 1024,
|
||||
max_echo_payload_len: 4 * 1024,
|
||||
max_reason_len: 256,
|
||||
@@ -339,7 +342,11 @@ pub struct AgentConfig {
|
||||
pub interface_name: String,
|
||||
/// The interface MTU.
|
||||
pub interface_mtu: u32,
|
||||
/// The IPv4 overlay range this agent proposes.
|
||||
/// Whether new networks derive their IPv4 range from their network id.
|
||||
///
|
||||
/// Enabled by default. Explicit ranges and IPv4 opt-out disable this.
|
||||
pub overlay_ipv4_auto: bool,
|
||||
/// The IPv4 overlay range this agent proposes when automatic selection is off.
|
||||
///
|
||||
/// Addresses are allocated from it and recorded in signed state, so a
|
||||
/// participant keeps the same one across restarts. A network that has
|
||||
@@ -373,7 +380,8 @@ impl AgentConfig {
|
||||
tun_factory: None,
|
||||
interface_name: "tsun0".to_string(),
|
||||
interface_mtu: 1280,
|
||||
overlay_ipv4_range: Some(crate::state::DEFAULT_IPV4_RANGE),
|
||||
overlay_ipv4_auto: true,
|
||||
overlay_ipv4_range: None,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -445,6 +453,7 @@ impl AgentConfig {
|
||||
|
||||
/// Sets the IPv4 overlay range this agent proposes, or disables IPv4.
|
||||
pub fn with_overlay_ipv4_range(mut self, range: Option<crate::state::Ipv4Range>) -> Self {
|
||||
self.overlay_ipv4_auto = false;
|
||||
self.overlay_ipv4_range = range;
|
||||
self
|
||||
}
|
||||
|
||||
@@ -67,10 +67,41 @@ impl Tally {
|
||||
}
|
||||
}
|
||||
|
||||
/// The links to one neighbour, best first. Which transport carries a frame
|
||||
/// is decided here and nowhere else: the end-to-end protocol is in the
|
||||
/// envelope and never depends on it.
|
||||
#[derive(Debug)]
|
||||
struct NextHop {
|
||||
peer: PeerId,
|
||||
link: SharedLink,
|
||||
links: Box<[SharedLink]>,
|
||||
}
|
||||
|
||||
impl NextHop {
|
||||
fn usable(&self) -> bool {
|
||||
self.links.iter().any(|link| !link.is_closed())
|
||||
}
|
||||
|
||||
fn first_open(&self) -> Option<&SharedLink> {
|
||||
self.links.iter().find(|link| !link.is_closed())
|
||||
}
|
||||
|
||||
/// Hands a frame to the best open link that accepts it. A link that
|
||||
/// cannot carry the frame (too large for its path, closed a moment ago)
|
||||
/// leaves it to the next one; the frame is only copied when there is
|
||||
/// another to try.
|
||||
fn send(&self, frame: Bytes) -> Result<(), TransportError> {
|
||||
let mut open = self.links.iter().filter(|link| !link.is_closed());
|
||||
let Some(mut current) = open.next() else {
|
||||
return Err(TransportError::Closed);
|
||||
};
|
||||
for next in open {
|
||||
if current.send(frame.clone()).is_ok() {
|
||||
return Ok(());
|
||||
}
|
||||
current = next;
|
||||
}
|
||||
current.send(frame)
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
@@ -82,21 +113,22 @@ struct ForwardRoute {
|
||||
impl ForwardRoute {
|
||||
fn select(&self, source: &PeerId, flow: FlowId) -> Option<&NextHop> {
|
||||
if self.next.len() == 1 {
|
||||
return self.next.first().filter(|hop| !hop.link.is_closed());
|
||||
return self.next.first().filter(|hop| hop.usable());
|
||||
}
|
||||
let seed = u64::from_le_bytes(source[..8].try_into().ok()?);
|
||||
let start = mix64(flow ^ seed) as usize % self.next.len();
|
||||
// Stable fallback if a link closed just before its table was replaced.
|
||||
(0..self.next.len())
|
||||
.map(|i| &self.next[(start + i) % self.next.len()])
|
||||
.find(|hop| !hop.link.is_closed())
|
||||
.find(|hop| hop.usable())
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Default)]
|
||||
struct ForwardingTable {
|
||||
routes: HashMap<PeerId, ForwardRoute>,
|
||||
inboxes: HashMap<PeerId, mpsc::Sender<Bytes>>,
|
||||
/// Local delivery, by who sent it and which end-to-end protocol it is.
|
||||
inboxes: HashMap<(PeerId, u64), mpsc::Sender<Bytes>>,
|
||||
members: HashSet<PeerId>,
|
||||
}
|
||||
|
||||
@@ -121,7 +153,7 @@ impl Plane {
|
||||
return;
|
||||
}
|
||||
if header.destination == self.local {
|
||||
let Some(inbox) = table.inboxes.get(&header.source) else {
|
||||
let Some(inbox) = table.inboxes.get(&(header.source, header.protocol)) else {
|
||||
self.tally.dropped_unknown.fetch_add(1, Ordering::Relaxed);
|
||||
return;
|
||||
};
|
||||
@@ -144,7 +176,7 @@ impl Plane {
|
||||
self.tally.dropped_no_link.fetch_add(1, Ordering::Relaxed);
|
||||
return;
|
||||
};
|
||||
if hop.link.send(envelope::decrement(frame)).is_ok() {
|
||||
if hop.send(envelope::decrement(frame)).is_ok() {
|
||||
self.tally.forwarded.fetch_add(1, Ordering::Relaxed);
|
||||
} else {
|
||||
self.tally.dropped_no_link.fetch_add(1, Ordering::Relaxed);
|
||||
@@ -152,27 +184,65 @@ impl Plane {
|
||||
}
|
||||
}
|
||||
|
||||
/// One physical link to a neighbour, of whatever kind of transport.
|
||||
#[derive(Debug)]
|
||||
struct Protocol {
|
||||
plane: Arc<Plane>,
|
||||
graph: HashMap<PeerId, Vec<PeerId>>,
|
||||
members: HashSet<PeerId>,
|
||||
raw: HashMap<PeerId, SharedLink>,
|
||||
readers: HashMap<PeerId, JoinHandle<()>>,
|
||||
links: HashMap<PeerId, Arc<PeerLink>>,
|
||||
struct RawLink {
|
||||
kind: String,
|
||||
link: SharedLink,
|
||||
reader: JoinHandle<()>,
|
||||
}
|
||||
|
||||
impl Protocol {
|
||||
/// How a destination is reached right now.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub struct RouteSummary {
|
||||
/// Number of transport links to it. One is a direct link.
|
||||
pub hops: u8,
|
||||
/// The neighbour the first link goes to. For a direct link that is the
|
||||
/// destination itself.
|
||||
pub via: EndpointId,
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
struct State {
|
||||
plane: Arc<Plane>,
|
||||
/// Transport kinds, best first. A neighbour with several links is
|
||||
/// reached over the first of these it has.
|
||||
priority: Vec<String>,
|
||||
graph: HashMap<PeerId, Vec<PeerId>>,
|
||||
members: HashSet<PeerId>,
|
||||
raw: HashMap<PeerId, Vec<RawLink>>,
|
||||
links: HashMap<(PeerId, String), Arc<PeerLink>>,
|
||||
}
|
||||
|
||||
impl State {
|
||||
fn rank(&self, kind: &str) -> usize {
|
||||
self.priority
|
||||
.iter()
|
||||
.position(|known| known == kind)
|
||||
.unwrap_or(usize::MAX)
|
||||
}
|
||||
|
||||
/// A neighbour's links, best kind first.
|
||||
fn ordered(&self, peer: &PeerId) -> Vec<&RawLink> {
|
||||
let mut links: Vec<&RawLink> = self
|
||||
.raw
|
||||
.get(peer)
|
||||
.map(|links| links.iter().collect())
|
||||
.unwrap_or_default();
|
||||
links.sort_by_key(|raw| (self.rank(&raw.kind), raw.kind.clone()));
|
||||
links
|
||||
}
|
||||
|
||||
fn publish(&self) {
|
||||
let mut graph = self.graph.clone();
|
||||
graph.insert(
|
||||
self.plane.local,
|
||||
self.raw
|
||||
.iter()
|
||||
.filter(|(_, link)| !link.is_closed())
|
||||
.map(|(peer, _)| *peer)
|
||||
.collect(),
|
||||
);
|
||||
let mut neighbours: Vec<PeerId> = self
|
||||
.raw
|
||||
.iter()
|
||||
.filter(|(_, links)| links.iter().any(|raw| !raw.link.is_closed()))
|
||||
.map(|(peer, _)| *peer)
|
||||
.collect();
|
||||
neighbours.sort_unstable();
|
||||
graph.insert(self.plane.local, neighbours);
|
||||
let routing = RoutingTable::build(self.plane.local, &graph, ROUTING_HOP_LIMIT);
|
||||
let routes = routing
|
||||
.iter()
|
||||
@@ -181,10 +251,12 @@ impl Protocol {
|
||||
.next_hops
|
||||
.iter()
|
||||
.filter_map(|id| {
|
||||
self.raw.get(id).map(|link| NextHop {
|
||||
peer: *id,
|
||||
link: link.clone(),
|
||||
})
|
||||
let links: Box<[_]> = self
|
||||
.ordered(id)
|
||||
.into_iter()
|
||||
.map(|raw| raw.link.clone())
|
||||
.collect();
|
||||
(!links.is_empty()).then_some(NextHop { peer: *id, links })
|
||||
})
|
||||
.collect();
|
||||
(!next.is_empty()).then_some((
|
||||
@@ -204,17 +276,21 @@ impl Protocol {
|
||||
inboxes: self
|
||||
.links
|
||||
.iter()
|
||||
.map(|(peer, link)| (*peer, link.inbox_tx.clone()))
|
||||
.map(|((peer, _), link)| ((*peer, link.tag), link.inbox_tx.clone()))
|
||||
.collect(),
|
||||
}));
|
||||
}
|
||||
|
||||
fn stop_readers(&mut self) {
|
||||
for raw in self.raw.values().flatten() {
|
||||
raw.reader.abort();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for Protocol {
|
||||
impl Drop for State {
|
||||
fn drop(&mut self) {
|
||||
for task in self.readers.values() {
|
||||
task.abort();
|
||||
}
|
||||
self.stop_readers();
|
||||
for link in self.links.values() {
|
||||
link.close();
|
||||
}
|
||||
@@ -222,12 +298,17 @@ impl Drop for Protocol {
|
||||
}
|
||||
}
|
||||
|
||||
/// One network's transport links and independently published protocol tables.
|
||||
/// One network's transport links and its published forwarding table.
|
||||
///
|
||||
/// There is one table for the whole network, whatever protocols run over it.
|
||||
/// A link is a way to reach a neighbour; a protocol is an end-to-end channel
|
||||
/// to a peer. The two meet only in the envelope, which names the protocol, so
|
||||
/// a tunnel keeps working over any mix of transports and a middle member
|
||||
/// needs no protocol in common with either end.
|
||||
#[derive(Debug)]
|
||||
pub struct RelayHub {
|
||||
network: NetworkId,
|
||||
local: PeerId,
|
||||
protocols: Mutex<HashMap<String, Protocol>>,
|
||||
state: Mutex<State>,
|
||||
tally: Arc<Tally>,
|
||||
changed: Arc<Notify>,
|
||||
}
|
||||
@@ -235,40 +316,38 @@ pub struct RelayHub {
|
||||
impl RelayHub {
|
||||
/// Creates the router at this network's local endpoint.
|
||||
pub fn new(network: NetworkId, local: EndpointId) -> Arc<Self> {
|
||||
let tally = Arc::new(Tally::default());
|
||||
Arc::new(Self {
|
||||
network,
|
||||
local: *local.as_bytes(),
|
||||
protocols: Mutex::default(),
|
||||
tally: Arc::default(),
|
||||
state: Mutex::new(State {
|
||||
plane: Arc::new(Plane {
|
||||
local: *local.as_bytes(),
|
||||
table: ArcSwap::from_pointee(ForwardingTable::default()),
|
||||
tally: tally.clone(),
|
||||
}),
|
||||
priority: Vec::new(),
|
||||
graph: HashMap::new(),
|
||||
members: HashSet::new(),
|
||||
raw: HashMap::new(),
|
||||
links: HashMap::new(),
|
||||
}),
|
||||
tally,
|
||||
changed: Arc::default(),
|
||||
})
|
||||
}
|
||||
|
||||
fn protocols(&self) -> std::sync::MutexGuard<'_, HashMap<String, Protocol>> {
|
||||
self.protocols
|
||||
.lock()
|
||||
.unwrap_or_else(|error| error.into_inner())
|
||||
fn state(&self) -> std::sync::MutexGuard<'_, State> {
|
||||
self.state.lock().unwrap_or_else(|error| error.into_inner())
|
||||
}
|
||||
|
||||
fn protocol<'a>(
|
||||
&self,
|
||||
protocols: &'a mut HashMap<String, Protocol>,
|
||||
name: &str,
|
||||
) -> &'a mut Protocol {
|
||||
protocols
|
||||
.entry(name.to_owned())
|
||||
.or_insert_with(|| Protocol {
|
||||
plane: Arc::new(Plane {
|
||||
local: self.local,
|
||||
table: ArcSwap::from_pointee(ForwardingTable::default()),
|
||||
tally: self.tally.clone(),
|
||||
}),
|
||||
graph: HashMap::new(),
|
||||
members: HashSet::new(),
|
||||
raw: HashMap::new(),
|
||||
readers: HashMap::new(),
|
||||
links: HashMap::new(),
|
||||
})
|
||||
/// Sets which kinds of transport are preferred, best first, when a
|
||||
/// neighbour can be reached over more than one.
|
||||
pub fn set_priority(&self, kinds: Vec<String>) {
|
||||
let mut state = self.state();
|
||||
if state.priority != kinds {
|
||||
state.priority = kinds;
|
||||
state.publish();
|
||||
}
|
||||
}
|
||||
|
||||
/// Wakes the control loop promptly when a transport reader ends.
|
||||
@@ -282,14 +361,11 @@ impl RelayHub {
|
||||
}
|
||||
|
||||
/// Publishes authenticated topology. Refreshing identical state is cheap.
|
||||
pub fn set_topology(
|
||||
&self,
|
||||
protocol: &str,
|
||||
graph: HashMap<PeerId, Vec<PeerId>>,
|
||||
members: HashSet<PeerId>,
|
||||
) {
|
||||
let mut protocols = self.protocols();
|
||||
let state = self.protocol(&mut protocols, protocol);
|
||||
///
|
||||
/// Every row of `graph` is one member's own account of who it has a link
|
||||
/// to; what kind of link it is does not matter here.
|
||||
pub fn set_topology(&self, graph: HashMap<PeerId, Vec<PeerId>>, members: HashSet<PeerId>) {
|
||||
let mut state = self.state();
|
||||
if state.graph != graph || state.members != members {
|
||||
state.graph = graph;
|
||||
state.members = members;
|
||||
@@ -298,22 +374,44 @@ impl RelayHub {
|
||||
}
|
||||
|
||||
/// Whether the current table has a path to this destination.
|
||||
pub fn reachable(&self, peer: EndpointId, protocol: &str) -> bool {
|
||||
self.protocols().get(protocol).is_some_and(|state| {
|
||||
state
|
||||
.plane
|
||||
.table
|
||||
.load()
|
||||
.routes
|
||||
.contains_key(peer.as_bytes())
|
||||
pub fn reachable(&self, peer: EndpointId) -> bool {
|
||||
self.state()
|
||||
.plane
|
||||
.table
|
||||
.load()
|
||||
.routes
|
||||
.contains_key(peer.as_bytes())
|
||||
}
|
||||
|
||||
/// How a destination is reached right now, if it is.
|
||||
pub fn route(&self, peer: EndpointId) -> Option<RouteSummary> {
|
||||
let table = self.state().plane.table.load_full();
|
||||
let route = table.routes.get(peer.as_bytes())?;
|
||||
let via = EndpointId::from_bytes(&route.next.first()?.peer).ok()?;
|
||||
Some(RouteSummary {
|
||||
hops: route.hops,
|
||||
via,
|
||||
})
|
||||
}
|
||||
|
||||
/// Stable logical link, preserved across topology changes.
|
||||
/// The kinds of transport with an open direct link to a peer, best
|
||||
/// first.
|
||||
pub fn direct_kinds(&self, peer: EndpointId) -> Vec<String> {
|
||||
let state = self.state();
|
||||
state
|
||||
.ordered(peer.as_bytes())
|
||||
.into_iter()
|
||||
.filter(|raw| !raw.link.is_closed())
|
||||
.map(|raw| raw.kind.clone())
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Stable end-to-end channel for a protocol, preserved across topology
|
||||
/// and transport changes.
|
||||
pub fn link(&self, peer: EndpointId, protocol: &str) -> Arc<PeerLink> {
|
||||
let mut protocols = self.protocols();
|
||||
let state = self.protocol(&mut protocols, protocol);
|
||||
if let Some(link) = state.links.get(peer.as_bytes()) {
|
||||
let mut state = self.state();
|
||||
let key = (*peer.as_bytes(), protocol.to_owned());
|
||||
if let Some(link) = state.links.get(&key) {
|
||||
return link.clone();
|
||||
}
|
||||
let (inbox_tx, inbox_rx) = mpsc::channel(INBOX);
|
||||
@@ -321,88 +419,118 @@ impl RelayHub {
|
||||
let link = Arc::new(PeerLink {
|
||||
network: self.network,
|
||||
peer,
|
||||
tag: envelope::protocol_tag(protocol),
|
||||
plane: state.plane.clone(),
|
||||
inbox_tx,
|
||||
inbox_rx: tokio::sync::Mutex::new(inbox_rx),
|
||||
closed,
|
||||
});
|
||||
state.links.insert(*peer.as_bytes(), link.clone());
|
||||
state.links.insert(key, link.clone());
|
||||
state.publish();
|
||||
link
|
||||
}
|
||||
|
||||
/// Whether the protocol already owns its logical link.
|
||||
/// Whether the protocol already owns its end-to-end channel to a peer.
|
||||
pub fn has_link(&self, peer: EndpointId, protocol: &str) -> bool {
|
||||
self.protocols()
|
||||
.get(protocol)
|
||||
.is_some_and(|state| state.links.contains_key(peer.as_bytes()))
|
||||
self.state()
|
||||
.links
|
||||
.contains_key(&(*peer.as_bytes(), protocol.to_owned()))
|
||||
}
|
||||
|
||||
/// Installs a raw transport link and starts its independent ingress reader.
|
||||
pub fn set_direct(&self, peer: EndpointId, protocol: &str, raw: SharedLink) {
|
||||
///
|
||||
/// A neighbour may have one link per kind of transport; installing a
|
||||
/// second of the same kind replaces the first.
|
||||
pub fn set_direct(&self, peer: EndpointId, kind: &str, raw: SharedLink) {
|
||||
if raw.network() != self.network || raw.peer() != peer {
|
||||
return;
|
||||
}
|
||||
let mut protocols = self.protocols();
|
||||
let state = self.protocol(&mut protocols, protocol);
|
||||
let mut state = self.state();
|
||||
let id = *peer.as_bytes();
|
||||
if let Some(old) = state.readers.remove(&id) {
|
||||
old.abort();
|
||||
}
|
||||
state.raw.insert(id, raw.clone());
|
||||
state.publish();
|
||||
let plane = state.plane.clone();
|
||||
let changed = self.changed.clone();
|
||||
state.readers.insert(
|
||||
id,
|
||||
tokio::spawn(async move {
|
||||
let mut batch = 0;
|
||||
while let Some(frame) = raw.recv().await {
|
||||
plane.receive(id, frame);
|
||||
batch += 1;
|
||||
if batch == 64 {
|
||||
tokio::task::yield_now().await;
|
||||
batch = 0;
|
||||
}
|
||||
let reader_link = raw.clone();
|
||||
let reader = tokio::spawn(async move {
|
||||
let mut batch = 0;
|
||||
while let Some(frame) = reader_link.recv().await {
|
||||
plane.receive(id, frame);
|
||||
batch += 1;
|
||||
if batch == 64 {
|
||||
tokio::task::yield_now().await;
|
||||
batch = 0;
|
||||
}
|
||||
changed.notify_one();
|
||||
}),
|
||||
);
|
||||
}
|
||||
changed.notify_one();
|
||||
});
|
||||
let links = state.raw.entry(id).or_default();
|
||||
if let Some(position) = links.iter().position(|known| known.kind == kind) {
|
||||
links.remove(position).reader.abort();
|
||||
}
|
||||
links.push(RawLink {
|
||||
kind: kind.to_owned(),
|
||||
link: raw,
|
||||
reader,
|
||||
});
|
||||
state.publish();
|
||||
}
|
||||
|
||||
/// Removes a physical link without tearing down end-to-end protocol state.
|
||||
pub fn clear_direct(&self, peer: EndpointId, protocol: &str) {
|
||||
if let Some(state) = self.protocols().get_mut(protocol) {
|
||||
state.raw.remove(peer.as_bytes());
|
||||
if let Some(task) = state.readers.remove(peer.as_bytes()) {
|
||||
task.abort();
|
||||
/// Removes one physical link without tearing down end-to-end protocol state.
|
||||
pub fn clear_direct(&self, peer: EndpointId, kind: &str) {
|
||||
let mut state = self.state();
|
||||
let mut changed = false;
|
||||
if let Some(links) = state.raw.get_mut(peer.as_bytes()) {
|
||||
if let Some(position) = links.iter().position(|known| known.kind == kind) {
|
||||
links.remove(position).reader.abort();
|
||||
changed = true;
|
||||
}
|
||||
if links.is_empty() {
|
||||
state.raw.remove(peer.as_bytes());
|
||||
}
|
||||
}
|
||||
if changed {
|
||||
state.publish();
|
||||
}
|
||||
}
|
||||
|
||||
/// Revokes all links and topology involving a departed authenticated peer.
|
||||
pub fn remove_peer(&self, peer: EndpointId) {
|
||||
for state in self.protocols().values_mut() {
|
||||
state.raw.remove(peer.as_bytes());
|
||||
if let Some(task) = state.readers.remove(peer.as_bytes()) {
|
||||
task.abort();
|
||||
let mut state = self.state();
|
||||
if let Some(links) = state.raw.remove(peer.as_bytes()) {
|
||||
for raw in links {
|
||||
raw.reader.abort();
|
||||
}
|
||||
if let Some(link) = state.links.remove(peer.as_bytes()) {
|
||||
}
|
||||
let gone: Vec<_> = state
|
||||
.links
|
||||
.keys()
|
||||
.filter(|(id, _)| id == peer.as_bytes())
|
||||
.cloned()
|
||||
.collect();
|
||||
for key in gone {
|
||||
if let Some(link) = state.links.remove(&key) {
|
||||
link.close();
|
||||
}
|
||||
state.members.remove(peer.as_bytes());
|
||||
state.graph.remove(peer.as_bytes());
|
||||
for neighbors in state.graph.values_mut() {
|
||||
neighbors.retain(|id| id != peer.as_bytes());
|
||||
}
|
||||
state.publish();
|
||||
}
|
||||
state.members.remove(peer.as_bytes());
|
||||
state.graph.remove(peer.as_bytes());
|
||||
for neighbors in state.graph.values_mut() {
|
||||
neighbors.retain(|id| id != peer.as_bytes());
|
||||
}
|
||||
state.publish();
|
||||
}
|
||||
|
||||
/// Stops every reader and releases all transport handles.
|
||||
pub fn close(&self) {
|
||||
self.protocols().clear();
|
||||
let mut state = self.state();
|
||||
state.stop_readers();
|
||||
state.raw.clear();
|
||||
for link in state.links.values() {
|
||||
link.close();
|
||||
}
|
||||
state.links.clear();
|
||||
state.graph.clear();
|
||||
state.members.clear();
|
||||
state.plane.table.store(Arc::default());
|
||||
}
|
||||
}
|
||||
|
||||
@@ -411,6 +539,7 @@ impl RelayHub {
|
||||
pub struct PeerLink {
|
||||
network: NetworkId,
|
||||
peer: EndpointId,
|
||||
tag: u64,
|
||||
plane: Arc<Plane>,
|
||||
inbox_tx: mpsc::Sender<Bytes>,
|
||||
inbox_rx: tokio::sync::Mutex<mpsc::Receiver<Bytes>>,
|
||||
@@ -454,10 +583,11 @@ impl PacketLink for PeerLink {
|
||||
let hop = route
|
||||
.select(&self.plane.local, flow)
|
||||
.ok_or(TransportError::Closed)?;
|
||||
hop.link.send(envelope::encode(
|
||||
hop.send(envelope::encode(
|
||||
self.plane.local,
|
||||
*self.peer.as_bytes(),
|
||||
flow,
|
||||
self.tag,
|
||||
&payload,
|
||||
))?;
|
||||
if route.hops > 1 {
|
||||
@@ -489,7 +619,9 @@ impl PacketLink for PeerLink {
|
||||
fn path_description(&self) -> String {
|
||||
let table = self.plane.table.load();
|
||||
match table.routes.get(self.peer.as_bytes()) {
|
||||
Some(route) if route.hops == 1 => route.next[0].link.path_description(),
|
||||
Some(route) if route.hops == 1 => route.next[0]
|
||||
.first_open()
|
||||
.map_or_else(|| "unreachable".into(), |link| link.path_description()),
|
||||
Some(route) => format!(
|
||||
"relay {} hops via {} ({} equal paths)",
|
||||
route.hops,
|
||||
|
||||
@@ -16,6 +16,9 @@ fn peer(seed: u8) -> EndpointId {
|
||||
fn id(seed: u8) -> PeerId {
|
||||
*peer(seed).as_bytes()
|
||||
}
|
||||
fn tag() -> u64 {
|
||||
envelope::protocol_tag("ip")
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
struct Wire {
|
||||
@@ -78,9 +81,8 @@ impl PacketLink for Wire {
|
||||
}
|
||||
}
|
||||
|
||||
fn topology(hub: &RelayHub, protocol: &str, edges: &[(u8, &[u8])], members: &[u8]) {
|
||||
fn topology(hub: &RelayHub, edges: &[(u8, &[u8])], members: &[u8]) {
|
||||
hub.set_topology(
|
||||
protocol,
|
||||
edges
|
||||
.iter()
|
||||
.map(|(a, bs)| (id(*a), bs.iter().map(|b| id(*b)).collect()))
|
||||
@@ -96,7 +98,7 @@ async fn ecmp_is_per_flow_direct_wins_and_existing_tunnel_survives_changes() {
|
||||
let c = Wire::new(3, true);
|
||||
hub.set_direct(peer(2), "ip", b.clone());
|
||||
hub.set_direct(peer(3), "ip", c.clone());
|
||||
topology(&hub, "ip", &[(2, &[4]), (3, &[4])], &[1, 2, 3, 4]);
|
||||
topology(&hub, &[(2, &[4]), (3, &[4])], &[1, 2, 3, 4]);
|
||||
let link = hub.link(peer(4), "ip");
|
||||
let mut chosen = HashMap::new();
|
||||
for _ in 0..8 {
|
||||
@@ -132,7 +134,7 @@ async fn ecmp_is_per_flow_direct_wins_and_existing_tunnel_survives_changes() {
|
||||
}
|
||||
assert_eq!(c.take().len(), 100);
|
||||
hub.clear_direct(peer(2), "ip");
|
||||
topology(&hub, "ip", &[(3, &[])], &[1, 2, 3, 4]);
|
||||
topology(&hub, &[(3, &[])], &[1, 2, 3, 4]);
|
||||
assert!(link.send(Bytes::new()).is_err());
|
||||
assert!(!link.is_closed());
|
||||
assert!(Arc::ptr_eq(&link, &hub.link(peer(4), "ip")));
|
||||
@@ -152,8 +154,8 @@ async fn transport_reader_forwards_without_any_plugin_reader_and_isolates_protoc
|
||||
let c = Wire::new(3, true);
|
||||
hub.set_direct(peer(1), "ip", a.clone());
|
||||
hub.set_direct(peer(3), "ip", c.clone());
|
||||
topology(&hub, "ip", &[(3, &[4])], &[1, 2, 3, 4]);
|
||||
let frame = envelope::encode(id(1), id(4), 91, b"end-to-end encrypted");
|
||||
topology(&hub, &[(3, &[4])], &[1, 2, 3, 4]);
|
||||
let frame = envelope::encode(id(1), id(4), 91, tag(), b"end-to-end encrypted");
|
||||
let pointer = frame.as_ptr();
|
||||
a.feed.send(frame).await.unwrap();
|
||||
tokio::time::timeout(std::time::Duration::from_secs(1), async {
|
||||
@@ -170,12 +172,136 @@ async fn transport_reader_forwards_without_any_plugin_reader_and_isolates_protoc
|
||||
ROUTING_HOP_LIMIT - 1
|
||||
);
|
||||
assert_eq!(&sent[0][RELAY_OVERHEAD..], b"end-to-end encrypted");
|
||||
let isolated = hub.link(peer(4), "other-ip");
|
||||
assert!(isolated.send(Bytes::new()).is_err());
|
||||
assert_eq!(hub.counters().forwarded, 1);
|
||||
hub.close();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_frame_is_delivered_only_to_the_protocol_it_names() {
|
||||
let hub = RelayHub::new(network(), peer(2));
|
||||
hub.set_direct(peer(1), "ip", Wire::new(1, true));
|
||||
let ip = hub.link(peer(1), "ip");
|
||||
let other = hub.link(peer(1), "other-ip");
|
||||
let plane = ip.plane.clone();
|
||||
plane.receive(id(1), envelope::encode(id(1), id(2), 0, tag(), b"for ip"));
|
||||
plane.receive(
|
||||
id(1),
|
||||
envelope::encode(
|
||||
id(1),
|
||||
id(2),
|
||||
0,
|
||||
envelope::protocol_tag("other-ip"),
|
||||
b"for other",
|
||||
),
|
||||
);
|
||||
plane.receive(
|
||||
id(1),
|
||||
envelope::encode(
|
||||
id(1),
|
||||
id(2),
|
||||
0,
|
||||
envelope::protocol_tag("nobody-runs-this"),
|
||||
b"for nobody",
|
||||
),
|
||||
);
|
||||
assert_eq!(ip.recv().await.unwrap(), Bytes::from_static(b"for ip"));
|
||||
assert_eq!(
|
||||
other.recv().await.unwrap(),
|
||||
Bytes::from_static(b"for other")
|
||||
);
|
||||
assert_eq!(hub.counters().dropped_unknown, 1);
|
||||
hub.close();
|
||||
}
|
||||
|
||||
/// A middle member joins two neighbours over different transports, and the
|
||||
/// two ends share a protocol the middle has never heard of.
|
||||
#[tokio::test]
|
||||
async fn a_tunnel_crosses_a_middle_member_whose_links_are_of_different_kinds() {
|
||||
let a = RelayHub::new(network(), peer(1));
|
||||
let c = RelayHub::new(network(), peer(3));
|
||||
let b = RelayHub::new(network(), peer(2));
|
||||
let a_to_c = Wire::new(3, true);
|
||||
let c_to_a = Wire::new(1, true);
|
||||
let c_to_b = Wire::new(2, true);
|
||||
let b_to_c = Wire::new(3, true);
|
||||
a.set_direct(peer(3), "wg", a_to_c.clone());
|
||||
c.set_direct(peer(1), "wg", c_to_a);
|
||||
c.set_direct(peer(2), "wg-quic", c_to_b.clone());
|
||||
b.set_direct(peer(3), "wg-quic", b_to_c.clone());
|
||||
topology(&a, &[(3, &[1, 2])], &[1, 2, 3]);
|
||||
topology(&b, &[(3, &[1, 2])], &[1, 2, 3]);
|
||||
topology(&c, &[(1, &[3]), (2, &[3])], &[1, 2, 3]);
|
||||
|
||||
let outgoing = a.link(peer(2), "tunnel");
|
||||
let incoming = b.link(peer(1), "tunnel");
|
||||
outgoing
|
||||
.send_flow(Bytes::from_static(b"sealed"), 5)
|
||||
.unwrap();
|
||||
let frame = a_to_c.take().remove(0);
|
||||
|
||||
// The middle forwards it onto its other kind of link without looking
|
||||
// inside.
|
||||
c.state().plane.receive(id(1), frame);
|
||||
let frame = c_to_b.take().remove(0);
|
||||
assert_eq!(c.counters().forwarded, 1);
|
||||
|
||||
b.state().plane.receive(id(3), frame);
|
||||
assert_eq!(
|
||||
incoming.recv().await.unwrap(),
|
||||
Bytes::from_static(b"sealed")
|
||||
);
|
||||
assert_eq!(b.counters().received_via, 1);
|
||||
assert_eq!(
|
||||
b.route(peer(1)),
|
||||
Some(RouteSummary {
|
||||
hops: 2,
|
||||
via: peer(3)
|
||||
})
|
||||
);
|
||||
a.close();
|
||||
b.close();
|
||||
c.close();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_neighbour_is_reached_over_the_preferred_kind_and_falls_back_when_it_is_lost() {
|
||||
let hub = RelayHub::new(network(), peer(1));
|
||||
hub.set_priority(vec!["wg".into(), "tcp-tls".into(), "wg-quic".into()]);
|
||||
let slow = Wire::new(2, true);
|
||||
let fast = Wire::new(2, true);
|
||||
hub.set_direct(peer(2), "wg-quic", slow.clone());
|
||||
let link = hub.link(peer(2), "tunnel");
|
||||
link.send(Bytes::from_static(b"one")).unwrap();
|
||||
assert_eq!(slow.take().len(), 1);
|
||||
assert_eq!(hub.direct_kinds(peer(2)), ["wg-quic"]);
|
||||
|
||||
// A better kind appears: traffic moves to it with nothing to redo.
|
||||
hub.set_direct(peer(2), "wg", fast.clone());
|
||||
assert_eq!(hub.direct_kinds(peer(2)), ["wg", "wg-quic"]);
|
||||
link.send(Bytes::from_static(b"two")).unwrap();
|
||||
assert_eq!(fast.take().len(), 1);
|
||||
assert!(slow.take().is_empty());
|
||||
|
||||
// It dies: the worse one carries on, and the logical link is untouched.
|
||||
fast.dead.store(true, Ordering::Relaxed);
|
||||
link.send(Bytes::from_static(b"three")).unwrap();
|
||||
assert_eq!(slow.take().len(), 1);
|
||||
assert!(!link.is_closed());
|
||||
assert_eq!(
|
||||
hub.route(peer(2)),
|
||||
Some(RouteSummary {
|
||||
hops: 1,
|
||||
via: peer(2)
|
||||
})
|
||||
);
|
||||
|
||||
hub.clear_direct(peer(2), "wg-quic");
|
||||
hub.clear_direct(peer(2), "wg");
|
||||
assert!(!hub.reachable(peer(2)));
|
||||
assert!(Arc::ptr_eq(&link, &hub.link(peer(2), "tunnel")));
|
||||
hub.close();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn local_delivery_is_bounded_and_malformed_or_unknown_sources_are_dropped() {
|
||||
let hub = RelayHub::new(network(), peer(2));
|
||||
@@ -184,18 +310,21 @@ async fn local_delivery_is_bounded_and_malformed_or_unknown_sources_are_dropped(
|
||||
let link = hub.link(peer(1), "ip");
|
||||
let plane = link.plane.clone();
|
||||
for _ in 0..INBOX + 1 {
|
||||
plane.receive(id(1), envelope::encode(id(1), id(2), 0, b"ip"));
|
||||
plane.receive(id(1), envelope::encode(id(1), id(2), 0, tag(), b"ip"));
|
||||
}
|
||||
assert_eq!(hub.counters().dropped_congested, 1);
|
||||
for _ in 0..INBOX {
|
||||
assert_eq!(link.recv().await.unwrap(), Bytes::from_static(b"ip"));
|
||||
}
|
||||
plane.receive(id(1), Bytes::from_static(b"broken"));
|
||||
plane.receive(id(1), envelope::encode(id(9), id(2), 0, b"unknown member"));
|
||||
plane.receive(
|
||||
id(1),
|
||||
envelope::encode(id(9), id(2), 0, tag(), b"unknown member"),
|
||||
);
|
||||
assert_eq!(hub.counters().dropped_unknown, 2);
|
||||
plane.receive(
|
||||
id(1),
|
||||
envelope::encode(id(1), id(9), 0, b"unknown destination"),
|
||||
envelope::encode(id(1), id(9), 0, tag(), b"unknown destination"),
|
||||
);
|
||||
assert_eq!(hub.counters().dropped_no_link, 1);
|
||||
hub.close();
|
||||
@@ -210,11 +339,11 @@ async fn inconsistent_tables_cannot_loop_beyond_the_hop_limit() {
|
||||
a.set_direct(peer(2), "ip", ab.clone());
|
||||
b.set_direct(peer(1), "ip", ba.clone());
|
||||
// Deliberately inconsistent snapshots while updates are in flight.
|
||||
topology(&a, "ip", &[(2, &[4])], &[1, 2, 3, 4]);
|
||||
topology(&b, "ip", &[(1, &[4])], &[1, 2, 3, 4]);
|
||||
topology(&a, &[(2, &[4])], &[1, 2, 3, 4]);
|
||||
topology(&b, &[(1, &[4])], &[1, 2, 3, 4]);
|
||||
let ap = a.link(peer(4), "ip").plane.clone();
|
||||
let bp = b.link(peer(4), "ip").plane.clone();
|
||||
let mut frame = envelope::encode(id(3), id(4), 7, b"transit");
|
||||
let mut frame = envelope::encode(id(3), id(4), 7, tag(), b"transit");
|
||||
for turn in 0..ROUTING_HOP_LIMIT {
|
||||
let (plane, incoming, wire) = if turn % 2 == 0 {
|
||||
(&ap, id(2), &ab)
|
||||
@@ -253,7 +382,7 @@ async fn forwarding_benchmark() {
|
||||
hub.set_direct(peer(3), "ip", wire.clone());
|
||||
let alternate = Wire::new(5, false);
|
||||
hub.set_direct(peer(5), "ip", alternate.clone());
|
||||
topology(&hub, "ip", &[(3, &[4]), (5, &[4])], &[1, 2, 3, 4, 5]);
|
||||
topology(&hub, &[(3, &[4]), (5, &[4])], &[1, 2, 3, 4, 5]);
|
||||
let plane = hub.link(peer(4), "ip").plane.clone();
|
||||
let source = id(1);
|
||||
let destination = id(4);
|
||||
@@ -264,7 +393,9 @@ async fn forwarding_benchmark() {
|
||||
const BATCHES: usize = 100;
|
||||
for _ in 0..BATCHES {
|
||||
let frames: Vec<_> = (0..COUNT)
|
||||
.map(|flow| envelope::encode(source, destination, flow as u64, &vec![42; size]))
|
||||
.map(|flow| {
|
||||
envelope::encode(source, destination, flow as u64, tag(), &vec![42; size])
|
||||
})
|
||||
.collect();
|
||||
let start = std::time::Instant::now();
|
||||
for frame in frames {
|
||||
|
||||
@@ -3,8 +3,8 @@ use super::{FlowId, PeerId};
|
||||
use crate::config::{MAX_DATA_DATAGRAM, ROUTING_HOP_LIMIT};
|
||||
use bytes::{BufMut, Bytes, BytesMut};
|
||||
|
||||
pub(crate) const HEADER: usize = 2 + 32 + 32 + 8;
|
||||
const VERSION: u8 = 1;
|
||||
pub(crate) const HEADER: usize = 2 + 32 + 32 + 8 + 8;
|
||||
const VERSION: u8 = 2;
|
||||
|
||||
#[derive(Debug, PartialEq, Eq)]
|
||||
pub(crate) struct Header {
|
||||
@@ -12,6 +12,15 @@ pub(crate) struct Header {
|
||||
pub destination: PeerId,
|
||||
pub remaining: u8,
|
||||
pub flow: FlowId,
|
||||
/// Which end-to-end protocol the payload belongs to, whatever links carry it.
|
||||
pub protocol: u64,
|
||||
}
|
||||
|
||||
/// A stable 64-bit tag for a protocol id (FNV-1a), the same on every member.
|
||||
pub(crate) fn protocol_tag(name: &str) -> u64 {
|
||||
name.bytes().fold(0xcbf2_9ce4_8422_2325, |hash, byte| {
|
||||
(hash ^ u64::from(byte)).wrapping_mul(0x0000_0100_0000_01b3)
|
||||
})
|
||||
}
|
||||
|
||||
pub(crate) fn decode(bytes: &[u8]) -> Option<Header> {
|
||||
@@ -28,16 +37,24 @@ pub(crate) fn decode(bytes: &[u8]) -> Option<Header> {
|
||||
destination: bytes[34..66].try_into().ok()?,
|
||||
remaining: bytes[1],
|
||||
flow: u64::from_be_bytes(bytes[66..74].try_into().ok()?),
|
||||
protocol: u64::from_be_bytes(bytes[74..82].try_into().ok()?),
|
||||
})
|
||||
}
|
||||
|
||||
pub(crate) fn encode(source: PeerId, destination: PeerId, flow: FlowId, payload: &[u8]) -> Bytes {
|
||||
pub(crate) fn encode(
|
||||
source: PeerId,
|
||||
destination: PeerId,
|
||||
flow: FlowId,
|
||||
protocol: u64,
|
||||
payload: &[u8],
|
||||
) -> Bytes {
|
||||
let mut bytes = BytesMut::with_capacity(HEADER + payload.len());
|
||||
bytes.put_u8(VERSION);
|
||||
bytes.put_u8(ROUTING_HOP_LIMIT);
|
||||
bytes.extend_from_slice(&source);
|
||||
bytes.extend_from_slice(&destination);
|
||||
bytes.put_u64(flow);
|
||||
bytes.put_u64(protocol);
|
||||
bytes.extend_from_slice(payload);
|
||||
bytes.freeze()
|
||||
}
|
||||
@@ -58,7 +75,7 @@ mod tests {
|
||||
use super::*;
|
||||
#[test]
|
||||
fn transit_reuses_the_buffer_and_rejects_invalid_headers() {
|
||||
let frame = encode([1; 32], [2; 32], 17, &[42; 1280]);
|
||||
let frame = encode([1; 32], [2; 32], 17, protocol_tag("wg"), &[42; 1280]);
|
||||
let address = frame.as_ptr();
|
||||
let frame = decrement(frame);
|
||||
assert_eq!(
|
||||
@@ -66,7 +83,10 @@ mod tests {
|
||||
address,
|
||||
"no payload copy for an owned buffer"
|
||||
);
|
||||
assert_eq!(decode(&frame).unwrap().remaining, ROUTING_HOP_LIMIT - 1);
|
||||
let header = decode(&frame).unwrap();
|
||||
assert_eq!(header.remaining, ROUTING_HOP_LIMIT - 1);
|
||||
assert_eq!(header.protocol, protocol_tag("wg"));
|
||||
assert_ne!(protocol_tag("wg"), protocol_tag("wg-quic"));
|
||||
assert_eq!(&frame[HEADER..], &[42; 1280]);
|
||||
for length in 0..HEADER {
|
||||
assert!(decode(&frame[..length]).is_none());
|
||||
|
||||
@@ -0,0 +1,350 @@
|
||||
//! Authenticated probes of the actual data link, independent of control RTT.
|
||||
//! Keys are negotiated privately for each physical link. No fixed wire magic.
|
||||
use super::{PacketLink, SharedLink, TransportError};
|
||||
use crate::dataplane::SharedPlugin;
|
||||
use crate::overlay::RoutingTable;
|
||||
use crate::{BoxFuture, NetworkId};
|
||||
use bytes::Bytes;
|
||||
use hmac::{Hmac, KeyInit, Mac};
|
||||
use iroh::EndpointId;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use sha2::Sha256;
|
||||
use std::collections::VecDeque;
|
||||
use std::sync::{Arc, Mutex};
|
||||
use std::time::{Duration, Instant};
|
||||
use tokio::sync::{mpsc, watch};
|
||||
|
||||
#[derive(Clone, Serialize, Deserialize, zeroize::ZeroizeOnDrop)]
|
||||
pub(crate) struct Settings {
|
||||
marker: [u8; 16],
|
||||
key: [u8; 32],
|
||||
}
|
||||
impl Settings {
|
||||
pub(crate) fn fresh() -> Self {
|
||||
Self {
|
||||
marker: rand::random(),
|
||||
key: rand::random(),
|
||||
}
|
||||
}
|
||||
}
|
||||
impl std::fmt::Debug for Settings {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
f.write_str("HealthSettings([REDACTED])")
|
||||
}
|
||||
}
|
||||
struct Samples {
|
||||
pending: VecDeque<(u64, Instant)>,
|
||||
recent: VecDeque<bool>,
|
||||
}
|
||||
impl Samples {
|
||||
fn new() -> Self {
|
||||
Self {
|
||||
pending: VecDeque::new(),
|
||||
recent: VecDeque::new(),
|
||||
}
|
||||
}
|
||||
fn sent(&mut self, seq: u64, now: Instant) {
|
||||
self.pending.push_back((seq, now));
|
||||
}
|
||||
fn acknowledge(&mut self, seq: u64) {
|
||||
if let Some(i) = self.pending.iter().position(|(n, _)| *n == seq) {
|
||||
self.pending.remove(i);
|
||||
self.recent.push_back(true);
|
||||
}
|
||||
}
|
||||
fn expire(&mut self, now: Instant) {
|
||||
while self
|
||||
.pending
|
||||
.front()
|
||||
.is_some_and(|(_, at)| now.duration_since(*at) >= Duration::from_secs(3))
|
||||
{
|
||||
self.pending.pop_front();
|
||||
self.recent.push_back(false);
|
||||
}
|
||||
while self.recent.len() > 20 {
|
||||
self.recent.pop_front();
|
||||
}
|
||||
}
|
||||
fn degraded(&self) -> bool {
|
||||
self.recent.len() >= 15
|
||||
&& self.recent.iter().filter(|ok| !**ok).count() * 100 >= self.recent.len() * 60
|
||||
}
|
||||
}
|
||||
// This prefix is inside end-to-end encryption. It never appears on the UDP wire.
|
||||
const IP_MAGIC: &[u8; 16] = b"tsunagi-health01";
|
||||
pub(crate) fn ip_payload(packet: &[u8]) -> Option<&[u8]> {
|
||||
(packet.len() >= 36 && packet[0] == 0x45 && packet[9] == 253 && &packet[20..36] == IP_MAGIC)
|
||||
.then(|| &packet[36..])
|
||||
}
|
||||
/// Inject probes through the same encrypted tunnel as application IP packets.
|
||||
/// This matters for pure WG: raw UDP probes could pass DPI while WG is blocked.
|
||||
pub(crate) struct ProbeRoute {
|
||||
pub(crate) plugin: SharedPlugin,
|
||||
pub(crate) routes: Arc<RoutingTable>,
|
||||
pub(crate) network: NetworkId,
|
||||
pub(crate) peer: EndpointId,
|
||||
pub(crate) max_probe_size: usize,
|
||||
}
|
||||
impl ProbeRoute {
|
||||
fn send(&self, payload: Bytes) -> bool {
|
||||
let Some((source, destination)) = self.routes.probe_addresses(self.network, self.peer)
|
||||
else {
|
||||
return false;
|
||||
};
|
||||
let mut packet = vec![0; 36];
|
||||
packet[0] = 0x45;
|
||||
packet[8] = 64;
|
||||
packet[9] = 253;
|
||||
packet[12..16].copy_from_slice(&source.octets());
|
||||
packet[16..20].copy_from_slice(&destination.octets());
|
||||
packet[20..36].copy_from_slice(IP_MAGIC);
|
||||
let length = (packet.len() + payload.len()) as u16;
|
||||
packet[2..4].copy_from_slice(&length.to_be_bytes());
|
||||
let sum = packet[..20]
|
||||
.as_chunks::<2>()
|
||||
.0
|
||||
.iter()
|
||||
.map(|b| u32::from(u16::from_be_bytes(*b)))
|
||||
.sum::<u32>();
|
||||
let sum = (sum & 0xffff) + (sum >> 16);
|
||||
packet[10..12].copy_from_slice(&(!(sum as u16)).to_be_bytes());
|
||||
packet.extend_from_slice(&payload);
|
||||
self.plugin
|
||||
.carry(self.network, self.peer, Bytes::from(packet))
|
||||
}
|
||||
}
|
||||
/// A monitored physical link. It forwards application bytes unchanged.
|
||||
pub(crate) struct HealthLink {
|
||||
probes: mpsc::Sender<Bytes>,
|
||||
raw: SharedLink,
|
||||
settings: Mutex<Option<Settings>>,
|
||||
rx: tokio::sync::Mutex<mpsc::Receiver<Bytes>>,
|
||||
stop: watch::Sender<bool>,
|
||||
reason: Mutex<Option<String>>,
|
||||
}
|
||||
impl std::fmt::Debug for HealthLink {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
f.debug_struct("HealthLink")
|
||||
.field("raw", &self.raw)
|
||||
.finish_non_exhaustive()
|
||||
}
|
||||
}
|
||||
fn lock<T>(m: &Mutex<T>) -> std::sync::MutexGuard<'_, T> {
|
||||
match m.lock() {
|
||||
Ok(g) => g,
|
||||
Err(p) => p.into_inner(),
|
||||
}
|
||||
}
|
||||
impl HealthLink {
|
||||
pub(crate) fn new(
|
||||
raw: SharedLink,
|
||||
local: EndpointId,
|
||||
settings: Option<Settings>,
|
||||
route: Option<ProbeRoute>,
|
||||
) -> Arc<Self> {
|
||||
let (tx, rx) = mpsc::channel(64);
|
||||
let (probe_tx, mut probe_rx) = mpsc::channel(64);
|
||||
let (stop, _) = watch::channel(false);
|
||||
let link = Arc::new(Self {
|
||||
probes: probe_tx,
|
||||
raw: raw.clone(),
|
||||
settings: Mutex::new(settings),
|
||||
rx: tokio::sync::Mutex::new(rx),
|
||||
stop,
|
||||
reason: Mutex::new(None),
|
||||
});
|
||||
let weak = Arc::downgrade(&link);
|
||||
let mut stopped = link.stop.subscribe();
|
||||
tokio::spawn(async move {
|
||||
let mut interval = tokio::time::interval(Duration::from_secs(1));
|
||||
interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
|
||||
let mut samples = Samples::new();
|
||||
let mut sequence = rand::random::<u64>();
|
||||
let mut since: Option<Instant> = None;
|
||||
let mut marker = None;
|
||||
loop {
|
||||
tokio::select! {
|
||||
biased;
|
||||
_=stopped.changed()=>break,
|
||||
_=interval.tick()=>{
|
||||
let Some(link)=weak.upgrade()else{break;};let Some(settings)=lock(&link.settings).clone()else{continue;};
|
||||
if marker != Some(settings.marker) { marker = Some(settings.marker); samples = Samples::new(); since = None; }
|
||||
let start=*since.get_or_insert_with(Instant::now);let now=Instant::now();samples.expire(now);
|
||||
if now.duration_since(start)>=Duration::from_secs(20) && samples.degraded() {
|
||||
*lock(&link.reason)=Some("data delivery checks: at least 60% loss over the last 20 probes".into());break;
|
||||
}
|
||||
sequence=sequence.wrapping_add(1);
|
||||
let size=route.as_ref().map_or(raw.max_datagram_size(),|route|route.max_probe_size);
|
||||
let packet=probe(&settings,local,1,sequence,size.min(1000));
|
||||
if let Some(route)=&route { if route.send(packet) {samples.sent(sequence,now);} }
|
||||
else { let _=raw.send(packet); samples.sent(sequence,now); }
|
||||
}
|
||||
packet=raw.recv()=>{
|
||||
let Some(packet)=packet else{break;};let Some(link)=weak.upgrade()else{break;};
|
||||
let settings=lock(&link.settings).clone();
|
||||
if route.is_none() && let Some(settings)=settings && packet.starts_with(&settings.marker) {
|
||||
if let Some((kind,seq))=verify(&settings,raw.peer(),&packet) {
|
||||
if kind==1 {let _=raw.send(probe(&settings,local,2,seq,packet.len()));}
|
||||
else if kind==2 {samples.acknowledge(seq);}
|
||||
}
|
||||
} else {let _=tx.try_send(packet);}
|
||||
}
|
||||
packet=probe_rx.recv(),if route.is_some()=>{
|
||||
let Some(packet)=packet else {break;};let Some(link)=weak.upgrade()else{break;};
|
||||
let Some(settings)=lock(&link.settings).clone()else{continue;};
|
||||
if let Some((kind,seq))=verify(&settings,raw.peer(),&packet) {
|
||||
if kind==1 { if let Some(route)=&route { route.send(probe(&settings,local,2,seq,packet.len().min(route.max_probe_size))); } }
|
||||
else if kind==2 {samples.acknowledge(seq);}
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
}
|
||||
raw.close();
|
||||
if let Some(link) = weak.upgrade() {
|
||||
link.stop.send_replace(true);
|
||||
}
|
||||
});
|
||||
link
|
||||
}
|
||||
pub(crate) fn receive_probe(&self, packet: Bytes) {
|
||||
let _ = self.probes.try_send(packet);
|
||||
}
|
||||
pub(crate) fn configure(&self, settings: Settings) {
|
||||
*lock(&self.settings) = Some(settings);
|
||||
}
|
||||
}
|
||||
fn probe(settings: &Settings, sender: EndpointId, kind: u8, seq: u64, size: usize) -> Bytes {
|
||||
let mut bytes = vec![0; size.max(57)];
|
||||
bytes[..16].copy_from_slice(&settings.marker);
|
||||
bytes[16] = kind;
|
||||
bytes[17..25].copy_from_slice(&seq.to_be_bytes());
|
||||
rand::fill(&mut bytes[25..size.max(57) - 32]);
|
||||
let tail = bytes.len() - 32;
|
||||
if let Ok(mut mac) = <Hmac<Sha256> as KeyInit>::new_from_slice(&settings.key) {
|
||||
mac.update(sender.as_bytes());
|
||||
mac.update(&bytes[..tail]);
|
||||
bytes[tail..].copy_from_slice(&mac.finalize().into_bytes());
|
||||
}
|
||||
Bytes::from(bytes)
|
||||
}
|
||||
fn verify(settings: &Settings, sender: EndpointId, bytes: &[u8]) -> Option<(u8, u64)> {
|
||||
if bytes.len() < 57 || !bytes.starts_with(&settings.marker) {
|
||||
return None;
|
||||
}
|
||||
let mut mac = <Hmac<Sha256> as KeyInit>::new_from_slice(&settings.key).ok()?;
|
||||
let tail = bytes.len() - 32;
|
||||
mac.update(sender.as_bytes());
|
||||
mac.update(&bytes[..tail]);
|
||||
mac.verify_slice(&bytes[tail..]).ok()?;
|
||||
Some((
|
||||
bytes[16],
|
||||
u64::from_be_bytes(bytes.get(17..25)?.try_into().ok()?),
|
||||
))
|
||||
}
|
||||
impl PacketLink for HealthLink {
|
||||
fn network(&self) -> NetworkId {
|
||||
self.raw.network()
|
||||
}
|
||||
fn peer(&self) -> EndpointId {
|
||||
self.raw.peer()
|
||||
}
|
||||
fn max_datagram_size(&self) -> usize {
|
||||
self.raw.max_datagram_size()
|
||||
}
|
||||
fn send(&self, payload: Bytes) -> Result<(), TransportError> {
|
||||
if self.is_closed() {
|
||||
Err(TransportError::Closed)
|
||||
} else {
|
||||
self.raw.send(payload)
|
||||
}
|
||||
}
|
||||
fn send_flow(
|
||||
&self,
|
||||
payload: Bytes,
|
||||
flow: super::super::routing::FlowId,
|
||||
) -> Result<(), TransportError> {
|
||||
if self.is_closed() {
|
||||
Err(TransportError::Closed)
|
||||
} else {
|
||||
self.raw.send_flow(payload, flow)
|
||||
}
|
||||
}
|
||||
fn recv(&self) -> BoxFuture<'_, Option<Bytes>> {
|
||||
Box::pin(async move {
|
||||
let mut stop = self.stop.subscribe();
|
||||
if *stop.borrow() {
|
||||
return None;
|
||||
}
|
||||
let mut rx = self.rx.lock().await;
|
||||
tokio::select! {biased;_=stop.changed()=>None,packet=rx.recv()=>packet}
|
||||
})
|
||||
}
|
||||
fn closed(&self) -> BoxFuture<'_, ()> {
|
||||
Box::pin(async move {
|
||||
let mut stop = self.stop.subscribe();
|
||||
if !*stop.borrow() {
|
||||
let _ = stop.changed().await;
|
||||
}
|
||||
})
|
||||
}
|
||||
fn is_closed(&self) -> bool {
|
||||
*self.stop.borrow() || self.raw.is_closed()
|
||||
}
|
||||
fn close(&self) {
|
||||
self.stop.send_replace(true);
|
||||
self.raw.close();
|
||||
}
|
||||
fn failure_reason(&self) -> Option<String> {
|
||||
lock(&self.reason)
|
||||
.clone()
|
||||
.or_else(|| self.raw.failure_reason())
|
||||
}
|
||||
fn path_description(&self) -> String {
|
||||
self.raw.path_description()
|
||||
}
|
||||
}
|
||||
impl Drop for HealthLink {
|
||||
fn drop(&mut self) {
|
||||
self.stop.send_replace(true);
|
||||
self.raw.close();
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
|
||||
use super::*;
|
||||
#[test]
|
||||
fn no_false_failure_before_window_but_persistent_loss_is_detected() {
|
||||
let mut samples = Samples::new();
|
||||
let now = Instant::now();
|
||||
for seq in 0..20 {
|
||||
samples.sent(seq, now);
|
||||
if seq % 5 == 0 {
|
||||
samples.acknowledge(seq);
|
||||
}
|
||||
}
|
||||
assert!(!samples.degraded());
|
||||
samples.expire(now + Duration::from_secs(4));
|
||||
assert!(samples.degraded());
|
||||
for seq in 20..40 {
|
||||
samples.sent(seq, now);
|
||||
samples.acknowledge(seq);
|
||||
}
|
||||
samples.expire(now + Duration::from_secs(4));
|
||||
assert!(!samples.degraded());
|
||||
}
|
||||
#[test]
|
||||
fn a_forged_or_reflected_probe_cannot_claim_delivery() {
|
||||
let a = crate::identity::DeviceIdentity::generate().endpoint_id();
|
||||
let b = crate::identity::DeviceIdentity::generate().endpoint_id();
|
||||
let settings = Settings::fresh();
|
||||
let packet = probe(&settings, a, 1, 42, 1100);
|
||||
assert_eq!(verify(&settings, a, &packet), Some((1, 42)));
|
||||
assert!(verify(&settings, b, &packet).is_none());
|
||||
let mut forged = packet.to_vec();
|
||||
forged[16] = 2;
|
||||
assert!(verify(&settings, a, &forged).is_none());
|
||||
}
|
||||
}
|
||||
@@ -97,6 +97,9 @@ impl IrohLink {
|
||||
}
|
||||
|
||||
impl PacketLink for IrohLink {
|
||||
fn close(&self) {
|
||||
self.conn.close(0u32.into(), b"data link retired");
|
||||
}
|
||||
fn network(&self) -> NetworkId {
|
||||
self.network
|
||||
}
|
||||
|
||||
@@ -30,12 +30,14 @@
|
||||
//! one plugin protocol.
|
||||
|
||||
mod fragments;
|
||||
pub(crate) mod health;
|
||||
pub mod iroh_link;
|
||||
|
||||
use std::sync::Arc;
|
||||
|
||||
use bytes::Bytes;
|
||||
use iroh::EndpointId;
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
use crate::BoxFuture;
|
||||
use crate::identity::NetworkId;
|
||||
@@ -112,6 +114,14 @@ pub trait PacketLink: Send + Sync + std::fmt::Debug + 'static {
|
||||
/// keeping a task parked on [`PacketLink::closed`].
|
||||
fn is_closed(&self) -> bool;
|
||||
|
||||
/// Explicitly retires a physical link. Implementations should wake readers.
|
||||
fn close(&self) {}
|
||||
|
||||
/// Observed reason for retiring this link, if available.
|
||||
fn failure_reason(&self) -> Option<String> {
|
||||
None
|
||||
}
|
||||
|
||||
/// A short description of the path in use, for diagnostics.
|
||||
///
|
||||
/// Reports what the transport actually knows. It must not invent a value.
|
||||
@@ -121,6 +131,76 @@ pub trait PacketLink: Send + Sync + std::fmt::Debug + 'static {
|
||||
/// A shared handle to a link.
|
||||
pub type SharedLink = std::sync::Arc<dyn PacketLink>;
|
||||
|
||||
/// Maximum opaque pair-negotiation payload.
|
||||
pub const MAX_TRANSPORT_PAYLOAD: usize = 4096;
|
||||
|
||||
/// Private transport bytes, never included in Debug output.
|
||||
#[derive(Clone, PartialEq, Eq, Serialize, zeroize::ZeroizeOnDrop)]
|
||||
pub struct TransportPayload(Vec<u8>);
|
||||
|
||||
impl<'de> Deserialize<'de> for TransportPayload {
|
||||
fn deserialize<D: serde::Deserializer<'de>>(
|
||||
deserializer: D,
|
||||
) -> std::result::Result<Self, D::Error> {
|
||||
struct Visitor;
|
||||
impl<'de> serde::de::Visitor<'de> for Visitor {
|
||||
type Value = TransportPayload;
|
||||
fn expecting(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
write!(f, "at most {MAX_TRANSPORT_PAYLOAD} private transport bytes")
|
||||
}
|
||||
fn visit_seq<A: serde::de::SeqAccess<'de>>(
|
||||
self,
|
||||
mut sequence: A,
|
||||
) -> std::result::Result<Self::Value, A::Error> {
|
||||
if sequence
|
||||
.size_hint()
|
||||
.is_some_and(|size| size > MAX_TRANSPORT_PAYLOAD)
|
||||
{
|
||||
return Err(serde::de::Error::custom("transport payload too large"));
|
||||
}
|
||||
let mut payload = TransportPayload(Vec::with_capacity(
|
||||
sequence.size_hint().unwrap_or(0).min(MAX_TRANSPORT_PAYLOAD),
|
||||
));
|
||||
while let Some(byte) = sequence.next_element::<u8>()? {
|
||||
if payload.0.len() == MAX_TRANSPORT_PAYLOAD {
|
||||
return Err(serde::de::Error::custom("transport payload too large"));
|
||||
}
|
||||
payload.0.push(byte);
|
||||
}
|
||||
Ok(payload)
|
||||
}
|
||||
}
|
||||
deserializer.deserialize_seq(Visitor)
|
||||
}
|
||||
}
|
||||
impl TransportPayload {
|
||||
/// Wraps bytes for one authenticated control session.
|
||||
pub fn new(bytes: Vec<u8>) -> Self {
|
||||
Self(bytes)
|
||||
}
|
||||
/// Borrows private bytes; callers must not log them.
|
||||
pub fn as_bytes(&self) -> &[u8] {
|
||||
&self.0
|
||||
}
|
||||
}
|
||||
impl std::fmt::Debug for TransportPayload {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
f.write_str("TransportPayload([REDACTED])")
|
||||
}
|
||||
}
|
||||
|
||||
/// Private control-plane sender for transports. Contains no network secret.
|
||||
pub trait ControlSender: Send + Sync + std::fmt::Debug + 'static {
|
||||
/// Queues a bounded message to one authenticated member of one network.
|
||||
fn send<'a>(
|
||||
&'a self,
|
||||
network: NetworkId,
|
||||
peer: EndpointId,
|
||||
protocol: &'a str,
|
||||
payload: TransportPayload,
|
||||
) -> BoxFuture<'a, Result<(), TransportError>>;
|
||||
}
|
||||
|
||||
/// An inbound link a peer opened towards us.
|
||||
#[derive(Debug)]
|
||||
pub struct InboundLink {
|
||||
@@ -139,6 +219,18 @@ pub struct InboundLink {
|
||||
/// The agent owns one of these and hands links to plugins; plugins never call
|
||||
/// it directly.
|
||||
pub trait PacketTransport: Send + Sync + std::fmt::Debug + 'static {
|
||||
/// Supplies a private sender before network activation.
|
||||
fn attach_control(&self, _control: Arc<dyn ControlSender>) {}
|
||||
/// Validates and queues private negotiation received from an authenticated member.
|
||||
fn on_control(
|
||||
&self,
|
||||
_network: NetworkId,
|
||||
_peer: EndpointId,
|
||||
_protocol: &str,
|
||||
_payload: &[u8],
|
||||
) -> Result<(), TransportError> {
|
||||
Err(TransportError::Declined(self.name().into()))
|
||||
}
|
||||
/// A short name used in diagnostics.
|
||||
fn name(&self) -> &str;
|
||||
|
||||
@@ -184,6 +276,24 @@ impl MultiPacketTransport {
|
||||
}
|
||||
|
||||
impl PacketTransport for MultiPacketTransport {
|
||||
fn attach_control(&self, control: Arc<dyn ControlSender>) {
|
||||
self.default.attach_control(control.clone());
|
||||
for transport in self.by_protocol.values() {
|
||||
transport.attach_control(control.clone());
|
||||
}
|
||||
}
|
||||
fn on_control(
|
||||
&self,
|
||||
network: NetworkId,
|
||||
peer: EndpointId,
|
||||
protocol: &str,
|
||||
payload: &[u8],
|
||||
) -> Result<(), TransportError> {
|
||||
match self.by_protocol.get(protocol) {
|
||||
Some(transport) => transport.on_control(network, peer, protocol, payload),
|
||||
None => Err(TransportError::Declined(protocol.into())),
|
||||
}
|
||||
}
|
||||
fn name(&self) -> &str {
|
||||
"multi"
|
||||
}
|
||||
|
||||
@@ -97,6 +97,19 @@ pub enum Error {
|
||||
#[error("network {0} is not configured")]
|
||||
NetworkUnknown(NetworkId),
|
||||
|
||||
/// An overlay range overlaps networks already active on this agent.
|
||||
#[error(
|
||||
"network {network} cannot start: {range} overlaps active networks {conflicts:?}; switch to it to suspend those networks"
|
||||
)]
|
||||
NetworkRangeConflict {
|
||||
/// Network that could not start.
|
||||
network: NetworkId,
|
||||
/// Its agreed or proposed range.
|
||||
range: crate::state::Ipv4Range,
|
||||
/// Active networks that must be suspended first.
|
||||
conflicts: Vec<NetworkId>,
|
||||
},
|
||||
|
||||
/// No session with that peer exists in the given network.
|
||||
#[error("no authenticated session with peer {peer} in network {network}")]
|
||||
NoSuchPeer {
|
||||
|
||||
@@ -324,6 +324,9 @@ pub struct ActiveReport {
|
||||
/// Stopping something already stopped is not an error — it is the state
|
||||
/// asked for — but it is worth saying which happened.
|
||||
pub changed: bool,
|
||||
/// Networks suspended by switching to this network.
|
||||
#[serde(default)]
|
||||
pub suspended: Vec<RangeConflictReport>,
|
||||
}
|
||||
|
||||
/// What happened when a network was joined.
|
||||
@@ -345,6 +348,13 @@ pub struct JoinedReport {
|
||||
/// Almost always a mistyped secret, and the one thing that makes two
|
||||
/// sections of a report look like one network.
|
||||
pub name_shared_with: Option<String>,
|
||||
/// Whether it started; false means it is configured but blocked by overlap.
|
||||
#[serde(default = "joined_active_default")]
|
||||
pub active: bool,
|
||||
}
|
||||
|
||||
fn joined_active_default() -> bool {
|
||||
true
|
||||
}
|
||||
|
||||
/// What happened when a network was left.
|
||||
@@ -437,9 +447,23 @@ pub struct DnsReport {
|
||||
pub names: u32,
|
||||
}
|
||||
|
||||
/// An active network with an overlapping overlay range.
|
||||
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct RangeConflictReport {
|
||||
/// Public network id, for unambiguous selection.
|
||||
pub network_id: String,
|
||||
/// Configured network name.
|
||||
pub name: String,
|
||||
/// The overlapping range.
|
||||
pub range: String,
|
||||
}
|
||||
|
||||
/// One network.
|
||||
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct NetworkReport {
|
||||
/// Whether the network starts or resumes automatically when its range is free.
|
||||
#[serde(default)]
|
||||
pub auto_start: bool,
|
||||
/// Whether this agent participates in LAN broadcasts in this network.
|
||||
#[serde(default)]
|
||||
pub broadcast: bool,
|
||||
@@ -482,13 +506,48 @@ pub struct NetworkReport {
|
||||
/// The range it could not have, because another network on this agent
|
||||
/// already holds it.
|
||||
pub range_conflict: Option<String>,
|
||||
/// Networks that switching to this network would suspend.
|
||||
#[serde(default)]
|
||||
pub range_conflicts: Vec<RangeConflictReport>,
|
||||
/// The overlay, when a protocol is running one.
|
||||
pub overlay: Option<OverlayReport>,
|
||||
/// This agent's own name, when another member claimed it first.
|
||||
pub hostname_conflict: Option<HostnameConflictReport>,
|
||||
/// Exit-node settings: whether this agent offers one, and whether it
|
||||
/// uses one.
|
||||
pub exit: ExitReport,
|
||||
}
|
||||
|
||||
/// How packets reach a member right now.
|
||||
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub enum DataPathReport {
|
||||
/// Nothing carries packets to it: it is away, or only connected for
|
||||
/// control while a data path is being set up.
|
||||
#[default]
|
||||
None,
|
||||
/// A direct link.
|
||||
Direct {
|
||||
/// The kind of transport carrying it, e.g. `wg`.
|
||||
transport: String,
|
||||
},
|
||||
/// Through other members.
|
||||
Relayed {
|
||||
/// Number of links on the way, at least two.
|
||||
hops: u8,
|
||||
/// The member the first of them goes to, as an endpoint id.
|
||||
via: String,
|
||||
},
|
||||
}
|
||||
|
||||
/// A name somebody else claimed first.
|
||||
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct HostnameConflictReport {
|
||||
/// The name that is taken.
|
||||
pub name: String,
|
||||
/// The member that holds it, as an endpoint id.
|
||||
pub holder: String,
|
||||
}
|
||||
|
||||
/// One member of the network, from signed state.
|
||||
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct MemberReport {
|
||||
@@ -500,6 +559,15 @@ pub struct MemberReport {
|
||||
pub failed_dials: u32,
|
||||
/// The last hostname this agent saw it announce, kept while it is away.
|
||||
pub hostname: Option<String>,
|
||||
/// Whether this agent can reach it now, by any path. The one answer
|
||||
/// every client gives to "is it online": a member reached through
|
||||
/// others is online, over a longer path.
|
||||
pub online: bool,
|
||||
/// How packets reach it.
|
||||
pub path: DataPathReport,
|
||||
/// The name it claimed, when somebody else claimed it first. It still
|
||||
/// works by address; it is not found by that name.
|
||||
pub hostname_conflict: Option<HostnameConflictReport>,
|
||||
}
|
||||
|
||||
/// One control plane peer.
|
||||
@@ -516,6 +584,8 @@ pub struct PeerReport {
|
||||
pub rtt_ms: Option<u64>,
|
||||
/// The peer offers itself as an exit node in this network.
|
||||
pub exit_node: bool,
|
||||
/// How packets reach it.
|
||||
pub path: DataPathReport,
|
||||
}
|
||||
|
||||
/// The WireGuard overlay of one network.
|
||||
@@ -779,7 +849,7 @@ pub const EXCHANGE_TIMEOUT: Duration = Duration::from_secs(5);
|
||||
///
|
||||
/// Bump it whenever [`Request`], [`Response`] or anything they contain
|
||||
/// changes shape.
|
||||
pub const CONTROL_PROTOCOL: u32 = u32::from_be_bytes([b'T', b'S', b'N', 17]);
|
||||
pub const CONTROL_PROTOCOL: u32 = u32::from_be_bytes([b'T', b'S', b'N', 19]);
|
||||
|
||||
/// Reads one request off an accepted stream, answers it, writes the response.
|
||||
///
|
||||
|
||||
@@ -72,7 +72,7 @@ pub enum RouteError {
|
||||
#[error(
|
||||
"network {network} uses {range}, which overlaps {other_range} already in use by \
|
||||
{other}. One agent has one interface, so an address can belong to only one of \
|
||||
them: give one network a different range."
|
||||
them: stop the other network before starting this one."
|
||||
)]
|
||||
Overlap {
|
||||
/// The network being added.
|
||||
@@ -365,6 +365,43 @@ impl RoutingTable {
|
||||
})
|
||||
}
|
||||
|
||||
/// All active networks whose reserved ranges intersect this range.
|
||||
pub fn overlapping_networks(
|
||||
&self,
|
||||
network: NetworkId,
|
||||
range: Ipv4Range,
|
||||
) -> Vec<(NetworkId, Ipv4Range)> {
|
||||
let mut conflicts: Vec<_> = self
|
||||
.read()
|
||||
.iter()
|
||||
.filter_map(|(other, routes)| {
|
||||
let other_range = routes.range?;
|
||||
(*other != network && ranges_overlap(range, other_range))
|
||||
.then_some((*other, other_range))
|
||||
})
|
||||
.collect();
|
||||
conflicts.sort_by_key(|(id, _)| *id);
|
||||
conflicts
|
||||
}
|
||||
|
||||
/// Current signed allocations used for encrypted data delivery checks.
|
||||
pub(crate) fn probe_addresses(
|
||||
&self,
|
||||
network: NetworkId,
|
||||
peer: EndpointId,
|
||||
) -> Option<(Ipv4Addr, Ipv4Addr)> {
|
||||
let table = self.read();
|
||||
let routes = table.get(&network)?;
|
||||
Some((
|
||||
routes.local?,
|
||||
routes
|
||||
.peers
|
||||
.iter()
|
||||
.find(|(_, holder)| *holder == peer)
|
||||
.map(|(address, _)| *address)?,
|
||||
))
|
||||
}
|
||||
|
||||
/// How many networks the table covers.
|
||||
pub fn len(&self) -> usize {
|
||||
self.read().len()
|
||||
@@ -510,7 +547,7 @@ mod tests {
|
||||
let err = table.set_network(second, routes()).unwrap_err();
|
||||
let RouteError::Overlap { other, .. } = err;
|
||||
assert_eq!(other, first);
|
||||
assert!(err.to_string().contains("different range"), "{err}");
|
||||
assert!(err.to_string().contains("stop the other network"), "{err}");
|
||||
assert_eq!(table.len(), 1, "the overlapping network is not added");
|
||||
|
||||
// A range of its own is fine.
|
||||
|
||||
@@ -20,7 +20,7 @@ use crate::error::ProtocolError;
|
||||
/// The version in the ALPN is the wire-compatibility version of the control
|
||||
/// protocol. It is independent of the network identity scheme version, so
|
||||
/// bumping it must not change any existing [`crate::NetworkId`].
|
||||
pub const ALPN: &[u8] = b"tsunagi/ctrl/4";
|
||||
pub const ALPN: &[u8] = b"tsunagi/ctrl/6";
|
||||
|
||||
/// ALPN of the tsunagi data plane.
|
||||
///
|
||||
@@ -30,9 +30,11 @@ pub const ALPN: &[u8] = b"tsunagi/ctrl/4";
|
||||
/// transport underneath can be replaced without touching the control protocol.
|
||||
///
|
||||
/// Version 4 adds source, destination, hop limit and stable flow id above
|
||||
/// transport fragmentation. All members must upgrade together. Persistent
|
||||
/// identities and network configuration do not change.
|
||||
pub const DATA_ALPN: &[u8] = b"tsunagi/data/4";
|
||||
/// transport fragmentation. Version 5 adds the end-to-end protocol to that
|
||||
/// envelope, so a tunnel no longer depends on the kind of link under it, and
|
||||
/// a first-claimed time to a signed hostname claim. All members must upgrade
|
||||
/// together. Persistent identities and network configuration do not change.
|
||||
pub const DATA_ALPN: &[u8] = b"tsunagi/data/5";
|
||||
|
||||
/// Largest plugin protocol identifier accepted when opening a data channel.
|
||||
pub const MAX_DATA_PROTOCOL_LEN: usize = 32;
|
||||
@@ -41,7 +43,9 @@ pub const MAX_DATA_PROTOCOL_LEN: usize = 32;
|
||||
pub const MAX_SIGNATURE_LEN: usize = 64;
|
||||
|
||||
/// Control protocol version carried inside the handshake.
|
||||
pub const PROTOCOL_VERSION: u16 = 4;
|
||||
pub const PROTOCOL_VERSION: u16 = 6;
|
||||
|
||||
pub use crate::dataplane::transport::{MAX_TRANSPORT_PAYLOAD, TransportPayload};
|
||||
|
||||
/// First message of the handshake, sent by the initiator.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
@@ -112,6 +116,13 @@ pub struct DataOpenAck {
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[non_exhaustive]
|
||||
pub enum ControlMessage {
|
||||
/// Private transport negotiation; never persisted or rebroadcast.
|
||||
Transport {
|
||||
/// Transport protocol id; the payload stays opaque to the core.
|
||||
protocol: String,
|
||||
/// Secret negotiation data, redacted in diagnostics.
|
||||
payload: TransportPayload,
|
||||
},
|
||||
/// Hostname and capability announcement.
|
||||
Announce(Announcement),
|
||||
/// A small request used to verify that the exchange works.
|
||||
@@ -262,6 +273,17 @@ pub fn validate_capability(
|
||||
/// network, the other networks or the agent.
|
||||
pub fn validate(message: &ControlMessage, limits: &Limits) -> Result<(), ProtocolError> {
|
||||
match message {
|
||||
ControlMessage::Transport { protocol, payload } => {
|
||||
if protocol.is_empty() {
|
||||
return Err(ProtocolError::Malformed("empty transport protocol"));
|
||||
}
|
||||
check_len("transport.protocol", protocol.len(), MAX_PROTOCOL_ID_LEN)?;
|
||||
check_len(
|
||||
"transport.payload",
|
||||
payload.as_bytes().len(),
|
||||
limits.max_transport_payload_len.min(MAX_TRANSPORT_PAYLOAD),
|
||||
)?;
|
||||
}
|
||||
ControlMessage::Peers { peers } => {
|
||||
check_len("peers", peers.len(), limits.max_state_records)?;
|
||||
for hint in peers {
|
||||
@@ -318,6 +340,7 @@ pub fn validate(message: &ControlMessage, limits: &Limits) -> Result<(), Protoco
|
||||
/// A short, stable label for a message kind, for metrics and diagnostics.
|
||||
pub fn kind(message: &ControlMessage) -> &'static str {
|
||||
match message {
|
||||
ControlMessage::Transport { .. } => "transport",
|
||||
ControlMessage::Announce(_) => "announce",
|
||||
ControlMessage::Ping { .. } => "ping",
|
||||
ControlMessage::Pong { .. } => "pong",
|
||||
|
||||
@@ -109,12 +109,8 @@ impl std::str::FromStr for Ipv4Range {
|
||||
}
|
||||
}
|
||||
|
||||
/// The IPv4 overlay range used unless something else is configured or agreed.
|
||||
///
|
||||
/// A small, specific `/24`: memorable, and far less likely to overlap a
|
||||
/// network the machine is already on than taking a whole `/8` or `/10` would
|
||||
/// be. Because addresses are allocated rather than derived, 254 of them is
|
||||
/// plenty for the size of network this is for.
|
||||
/// The legacy default range, retained for existing networks and explicit use.
|
||||
/// New networks use [`derived_ipv4_range`] by default.
|
||||
pub const DEFAULT_IPV4_RANGE: Ipv4Range = Ipv4Range {
|
||||
base: Ipv4Addr::new(10, 13, 37, 0),
|
||||
prefix_len: 24,
|
||||
@@ -122,15 +118,10 @@ pub const DEFAULT_IPV4_RANGE: Ipv4Range = Ipv4Range {
|
||||
|
||||
/// A range derived from a network's own identity.
|
||||
///
|
||||
/// One agent has one interface, so two of its networks cannot both use the
|
||||
/// configured range. The second needs another one — and it cannot be picked
|
||||
/// locally, because every member has to arrive at the same answer without
|
||||
/// being told. So it comes from the network id, which every member already
|
||||
/// has and nobody can choose: the same network derives the same range on
|
||||
/// every device.
|
||||
///
|
||||
/// Inside 10/8 like the default, and never equal to it, so the first
|
||||
/// network keeps what it has always had.
|
||||
/// Every member independently derives the same pseudorandom `/24` in 10/8,
|
||||
/// including isolated first members. Stored signed ranges take precedence.
|
||||
/// The derivation is kept stable for networks already using it; the legacy
|
||||
/// default is excluded. Distinct network ids can still produce collisions.
|
||||
pub fn derived_ipv4_range(network: NetworkId) -> Ipv4Range {
|
||||
let bytes = network.as_bytes();
|
||||
let second = bytes[0];
|
||||
@@ -150,7 +141,7 @@ pub fn derived_ipv4_range(network: NetworkId) -> Ipv4Range {
|
||||
}
|
||||
|
||||
/// Frozen domain separator for the bytes a record signature covers.
|
||||
pub const RECORD_DOMAIN: &str = "tsunagi-signed-record-v2";
|
||||
pub const RECORD_DOMAIN: &str = "tsunagi-signed-record-v3";
|
||||
|
||||
/// Longest hostname a record may carry.
|
||||
///
|
||||
@@ -207,6 +198,18 @@ pub enum RecordBody {
|
||||
range: Option<Ipv4Range>,
|
||||
/// The name this author answers to.
|
||||
hostname: Option<String>,
|
||||
/// When this author first claimed that name, in milliseconds since
|
||||
/// the Unix epoch, kept across every later version that keeps the
|
||||
/// name. It is what makes a name belong to whoever claimed it first
|
||||
/// rather than to whoever happens to sort lowest, so a member that
|
||||
/// arrives later with the same name cannot take it over.
|
||||
///
|
||||
/// The author's own word: a member with a wrong clock can claim an
|
||||
/// earlier time. That costs it nothing it could not do by other
|
||||
/// means, since anybody who knows the network secret is a member,
|
||||
/// and it cannot take an address or reach anything it was not
|
||||
/// already entitled to.
|
||||
hostname_since: Option<u64>,
|
||||
},
|
||||
/// This author gives up everything it claimed.
|
||||
///
|
||||
@@ -243,6 +246,14 @@ impl RecordBody {
|
||||
}
|
||||
}
|
||||
|
||||
/// When the author first claimed the hostname, if it has one.
|
||||
pub fn hostname_since(&self) -> Option<u64> {
|
||||
match self {
|
||||
RecordBody::Claim { hostname_since, .. } => *hostname_since,
|
||||
RecordBody::Release => None,
|
||||
}
|
||||
}
|
||||
|
||||
fn canonical(&self) -> Vec<u8> {
|
||||
let mut out = Vec::with_capacity(96);
|
||||
match self {
|
||||
@@ -250,6 +261,7 @@ impl RecordBody {
|
||||
address,
|
||||
range,
|
||||
hostname,
|
||||
hostname_since,
|
||||
} => {
|
||||
push_lp(&mut out, b"claim");
|
||||
push_opt(
|
||||
@@ -270,6 +282,13 @@ impl RecordBody {
|
||||
.as_deref(),
|
||||
);
|
||||
push_opt(&mut out, hostname.as_deref().map(str::as_bytes));
|
||||
push_opt(
|
||||
&mut out,
|
||||
hostname_since
|
||||
.map(|since| since.to_be_bytes())
|
||||
.as_ref()
|
||||
.map(|bytes| &bytes[..]),
|
||||
);
|
||||
}
|
||||
RecordBody::Release => {
|
||||
push_lp(&mut out, b"release");
|
||||
@@ -391,8 +410,12 @@ impl SignedRecord {
|
||||
address,
|
||||
range,
|
||||
hostname,
|
||||
hostname_since,
|
||||
} = &self.body
|
||||
{
|
||||
if hostname.is_none() && hostname_since.is_some() {
|
||||
return Err(StateError::Malformed("a time was claimed for no hostname"));
|
||||
}
|
||||
// Bounds before anything is believed, because all of this came
|
||||
// off the network.
|
||||
if let Some(range) = range {
|
||||
@@ -585,25 +608,33 @@ impl StateSet {
|
||||
|
||||
/// Who currently holds each claimed hostname.
|
||||
///
|
||||
/// The same rule as addresses, for the same reason: a name is owned, two
|
||||
/// members may claim one, and every replica has to reach the same answer
|
||||
/// about who has it without asking anybody.
|
||||
/// A name belongs to the member that claimed it first, and the others
|
||||
/// that claim it keep working by address but are not found by that name.
|
||||
/// "First" is the time each author signed into its own claim, with the
|
||||
/// lower id deciding a tie and a claim with no time counting as last, so
|
||||
/// every replica reaches the same answer about who has it without
|
||||
/// asking anybody. It is a pure function of the records: nothing depends
|
||||
/// on the order they arrived in, so there is no race to lose.
|
||||
pub fn hostname_holders(&self) -> HashMap<String, EndpointId> {
|
||||
let mut holders: HashMap<String, EndpointId> = HashMap::new();
|
||||
let mut holders: HashMap<String, (u64, EndpointId)> = HashMap::new();
|
||||
for (author, record) in &self.records {
|
||||
let Some(hostname) = record.body.hostname() else {
|
||||
continue;
|
||||
};
|
||||
let claim = (record.body.hostname_since().unwrap_or(u64::MAX), *author);
|
||||
holders
|
||||
.entry(hostname.to_string())
|
||||
.and_modify(|held| {
|
||||
if author.as_bytes() < held.as_bytes() {
|
||||
*held = *author;
|
||||
if (claim.0, claim.1.as_bytes()) < (held.0, held.1.as_bytes()) {
|
||||
*held = claim;
|
||||
}
|
||||
})
|
||||
.or_insert(*author);
|
||||
.or_insert(claim);
|
||||
}
|
||||
holders
|
||||
.into_iter()
|
||||
.map(|(hostname, (_, author))| (hostname, author))
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// The hostname an author holds, if it holds one uncontested.
|
||||
@@ -612,6 +643,19 @@ impl StateSet {
|
||||
(self.hostname_holders().get(hostname) == Some(author)).then_some(hostname)
|
||||
}
|
||||
|
||||
/// The name an author claimed and the member that holds it instead, when
|
||||
/// somebody who claimed it earlier does.
|
||||
pub fn hostname_taken(&self, author: &EndpointId) -> Option<(&str, EndpointId)> {
|
||||
let hostname = self.records.get(author)?.body.hostname()?;
|
||||
let holder = *self.hostname_holders().get(hostname)?;
|
||||
(holder != *author).then_some((hostname, holder))
|
||||
}
|
||||
|
||||
/// When an author first claimed the name it holds now, if it has one.
|
||||
pub fn hostname_since_of(&self, author: &EndpointId) -> Option<u64> {
|
||||
self.records.get(author)?.body.hostname_since()
|
||||
}
|
||||
|
||||
/// The address an author holds, if it holds one uncontested.
|
||||
pub fn address_of(&self, author: &EndpointId) -> Option<Ipv4Addr> {
|
||||
let address = self.records.get(author)?.body.claimed_address()?;
|
||||
@@ -620,15 +664,20 @@ impl StateSet {
|
||||
|
||||
/// The range the network settled on, if anybody has said.
|
||||
///
|
||||
/// When claims disagree, the one from the lowest author id wins, so every
|
||||
/// replica reaches the same answer. A participant joining later therefore
|
||||
/// adopts the range already in use instead of imposing its own.
|
||||
/// A legacy or explicit range takes precedence over the derived default,
|
||||
/// so a fresh member's default cannot renumber an established network.
|
||||
/// Other disagreements use the lowest author id, independently of arrival
|
||||
/// order. The signed record format and network identity remain unchanged.
|
||||
pub fn agreed_range(&self) -> Option<Ipv4Range> {
|
||||
let mut authors: Vec<&EndpointId> = self.records.keys().collect();
|
||||
authors.sort_by_key(|author| *author.as_bytes());
|
||||
authors
|
||||
.into_iter()
|
||||
.find_map(|author| self.records.get(author)?.body.range())
|
||||
self.records
|
||||
.iter()
|
||||
.filter_map(|(author, record)| {
|
||||
let range = record.body.range()?;
|
||||
let derived = derived_ipv4_range(NetworkId::from_bytes(record.network));
|
||||
Some(((range == derived, *author.as_bytes()), range))
|
||||
})
|
||||
.min_by_key(|(priority, _)| *priority)
|
||||
.map(|(_, range)| range)
|
||||
}
|
||||
|
||||
/// The highest version this author has published, as far as is known.
|
||||
@@ -661,6 +710,7 @@ mod tests {
|
||||
address: Some(address.parse().unwrap()),
|
||||
range: Some(range()),
|
||||
hostname: None,
|
||||
hostname_since: None,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -720,6 +770,7 @@ mod tests {
|
||||
address: Some("10.99.0.1".parse().unwrap()),
|
||||
range: Some(range()),
|
||||
hostname: None,
|
||||
hostname_since: None,
|
||||
},
|
||||
);
|
||||
assert!(matches!(outside.verify(id), Err(StateError::Malformed(_))));
|
||||
@@ -735,6 +786,7 @@ mod tests {
|
||||
prefix_len: 31,
|
||||
}),
|
||||
hostname: None,
|
||||
hostname_since: None,
|
||||
},
|
||||
);
|
||||
assert!(matches!(no_hosts.verify(id), Err(StateError::Malformed(_))));
|
||||
@@ -776,6 +828,7 @@ mod tests {
|
||||
address: None,
|
||||
range: None,
|
||||
hostname: Some(bad.to_string()),
|
||||
hostname_since: None,
|
||||
},
|
||||
);
|
||||
assert!(
|
||||
@@ -797,6 +850,7 @@ mod tests {
|
||||
address: Some("10.13.37.5".parse().unwrap()),
|
||||
range: None,
|
||||
hostname: None,
|
||||
hostname_since: None,
|
||||
},
|
||||
);
|
||||
assert!(matches!(record.verify(id), Err(StateError::Malformed(_))));
|
||||
@@ -818,6 +872,7 @@ mod tests {
|
||||
address: None,
|
||||
range: None,
|
||||
hostname: Some("music".into()),
|
||||
hostname_since: None,
|
||||
},
|
||||
)
|
||||
};
|
||||
@@ -860,6 +915,7 @@ mod tests {
|
||||
address: None,
|
||||
range: None,
|
||||
hostname: Some(name.to_string()),
|
||||
hostname_since: None,
|
||||
},
|
||||
)
|
||||
};
|
||||
@@ -879,6 +935,109 @@ mod tests {
|
||||
assert_eq!(set.hostname_of(&secret.public()), Some("new"));
|
||||
}
|
||||
|
||||
fn named_at(
|
||||
secret: &SecretKey,
|
||||
id: NetworkId,
|
||||
version: u64,
|
||||
name: &str,
|
||||
since: u64,
|
||||
) -> SignedRecord {
|
||||
SignedRecord::sign(
|
||||
secret,
|
||||
id,
|
||||
version,
|
||||
RecordBody::Claim {
|
||||
address: None,
|
||||
range: None,
|
||||
hostname: Some(name.into()),
|
||||
hostname_since: Some(since),
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_name_belongs_to_whoever_claimed_it_first_whatever_their_ids_or_arrival_order() {
|
||||
let id = network("naming");
|
||||
let mut keys = [SecretKey::generate(), SecretKey::generate()];
|
||||
keys.sort_by_key(|key| *key.public().as_bytes());
|
||||
let [low, high] = keys;
|
||||
|
||||
// The member with the higher id claimed first, so it holds the name,
|
||||
// and the lower id does not take it back by sorting first.
|
||||
let early = named_at(&high, id, 1, "music", 1_000);
|
||||
let late = named_at(&low, id, 1, "music", 2_000);
|
||||
for order in [[&early, &late], [&late, &early]] {
|
||||
let mut set = StateSet::new();
|
||||
for record in order {
|
||||
set.merge(id, (*record).clone()).unwrap();
|
||||
}
|
||||
assert_eq!(set.hostname_of(&high.public()), Some("music"));
|
||||
assert_eq!(set.hostname_of(&low.public()), None);
|
||||
assert_eq!(
|
||||
set.hostname_taken(&low.public()),
|
||||
Some(("music", high.public()))
|
||||
);
|
||||
assert_eq!(set.hostname_taken(&high.public()), None);
|
||||
}
|
||||
|
||||
// At the same instant the lower id decides, the same way everywhere.
|
||||
let mut set = StateSet::new();
|
||||
set.merge(id, named_at(&high, id, 1, "music", 5)).unwrap();
|
||||
set.merge(id, named_at(&low, id, 1, "music", 5)).unwrap();
|
||||
assert_eq!(set.hostname_of(&low.public()), Some("music"));
|
||||
assert_eq!(set.hostname_of(&high.public()), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_name_that_loses_is_found_again_once_the_holder_lets_it_go() {
|
||||
let id = network("naming");
|
||||
let first = SecretKey::generate();
|
||||
let second = SecretKey::generate();
|
||||
let mut set = StateSet::new();
|
||||
set.merge(id, named_at(&first, id, 1, "music", 1)).unwrap();
|
||||
set.merge(id, named_at(&second, id, 1, "music", 2)).unwrap();
|
||||
assert_eq!(set.hostname_of(&second.public()), None);
|
||||
|
||||
// Renaming starts the clock afresh, so it cannot be used to jump the
|
||||
// queue for a name somebody else already holds.
|
||||
set.merge(id, named_at(&first, id, 2, "other", 3)).unwrap();
|
||||
assert_eq!(set.hostname_of(&second.public()), Some("music"));
|
||||
set.merge(id, named_at(&first, id, 3, "music", 4)).unwrap();
|
||||
assert_eq!(set.hostname_of(&second.public()), Some("music"));
|
||||
assert_eq!(set.hostname_of(&first.public()), None);
|
||||
|
||||
// And a release hands it on.
|
||||
set.merge(id, SignedRecord::sign(&second, id, 2, RecordBody::Release))
|
||||
.unwrap();
|
||||
assert_eq!(set.hostname_of(&first.public()), Some("music"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_time_with_no_name_is_not_a_valid_claim() {
|
||||
let id = network("naming");
|
||||
let secret = SecretKey::generate();
|
||||
let record = SignedRecord::sign(
|
||||
&secret,
|
||||
id,
|
||||
1,
|
||||
RecordBody::Claim {
|
||||
address: None,
|
||||
range: None,
|
||||
hostname: None,
|
||||
hostname_since: Some(1),
|
||||
},
|
||||
);
|
||||
assert!(matches!(record.verify(id), Err(StateError::Malformed(_))));
|
||||
|
||||
// The time is signed: moving it breaks the signature, so nobody can
|
||||
// improve another member's place in the queue.
|
||||
let mut forged = named_at(&secret, id, 1, "music", 9_000);
|
||||
if let RecordBody::Claim { hostname_since, .. } = &mut forged.body {
|
||||
*hostname_since = Some(1);
|
||||
}
|
||||
assert_eq!(forged.verify(id).unwrap_err(), StateError::BadSignature);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_release_gives_up_the_name_as_well_as_the_address() {
|
||||
let secret = SecretKey::generate();
|
||||
@@ -894,6 +1053,7 @@ mod tests {
|
||||
address: Some("10.13.37.5".parse().unwrap()),
|
||||
range: Some(range()),
|
||||
hostname: Some("music".into()),
|
||||
hostname_since: None,
|
||||
},
|
||||
),
|
||||
)
|
||||
@@ -1057,6 +1217,7 @@ mod tests {
|
||||
address: Some("10.99.0.7".parse().unwrap()),
|
||||
range: Some(custom),
|
||||
hostname: None,
|
||||
hostname_since: None,
|
||||
},
|
||||
),
|
||||
)
|
||||
@@ -1098,7 +1259,7 @@ mod tests {
|
||||
assert_eq!(once.base.octets()[0], 10, "inside 10/8 like the default");
|
||||
assert_ne!(
|
||||
once, DEFAULT_IPV4_RANGE,
|
||||
"the default belongs to whichever network asked first"
|
||||
"the legacy default remains excluded"
|
||||
);
|
||||
|
||||
// A different network derives a different range, which is the
|
||||
@@ -1110,4 +1271,34 @@ mod tests {
|
||||
.network_id();
|
||||
assert_ne!(once, derived_ipv4_range(other));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_derived_default_cannot_displace_a_legacy_or_explicit_range() {
|
||||
let id = network("range-migration");
|
||||
let a = SecretKey::generate();
|
||||
let b = SecretKey::generate();
|
||||
let (low, high) = if a.public().as_bytes() < b.public().as_bytes() {
|
||||
(a, b)
|
||||
} else {
|
||||
(b, a)
|
||||
};
|
||||
for established in [DEFAULT_IPV4_RANGE, "10.90.0.0/16".parse().unwrap()] {
|
||||
let body = |range: Ipv4Range| RecordBody::Claim {
|
||||
address: Some(Ipv4Addr::from(u32::from(range.base) + 1)),
|
||||
range: Some(range),
|
||||
hostname: None,
|
||||
hostname_since: None,
|
||||
};
|
||||
let fresh = SignedRecord::sign(&low, id, 1, body(derived_ipv4_range(id)));
|
||||
let existing = SignedRecord::sign(&high, id, 1, body(established));
|
||||
for records in [
|
||||
vec![fresh.clone(), existing.clone()],
|
||||
vec![existing.clone(), fresh.clone()],
|
||||
] {
|
||||
let mut state = StateSet::new();
|
||||
state.merge_all(id, records);
|
||||
assert_eq!(state.agreed_range(), Some(established));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -272,6 +272,33 @@ impl Storage {
|
||||
.await
|
||||
}
|
||||
|
||||
/// Configured networks in the user's most recently selected order.
|
||||
pub async fn network_start_order(&self) -> Result<Vec<NetworkId>> {
|
||||
self.with_state(|state| {
|
||||
let configured = state.list_networks()?;
|
||||
let text = state
|
||||
.get_setting("network_start_order")?
|
||||
.unwrap_or_default();
|
||||
let mut order = Vec::new();
|
||||
for word in text.split(',') {
|
||||
if let Ok(id) = word.parse::<NetworkId>()
|
||||
&& configured.iter().any(|n| n.network_id == id)
|
||||
&& !order.contains(&id)
|
||||
{
|
||||
order.push(id);
|
||||
}
|
||||
}
|
||||
Ok(order)
|
||||
})
|
||||
.await
|
||||
}
|
||||
|
||||
/// Prefers this network at startup without disabling suspended networks.
|
||||
pub async fn prefer_network(&self, network: NetworkId) -> Result<()> {
|
||||
self.with_state(move |state| state.prefer_network(network))
|
||||
.await
|
||||
}
|
||||
|
||||
/// Removes a network configuration and its cached hints.
|
||||
pub async fn remove_network(&self, network_id: NetworkId) -> Result<()> {
|
||||
self.with_state(move |state| {
|
||||
|
||||
@@ -21,7 +21,7 @@ use crate::identity::{DeviceIdentity, NetworkId, NetworkName, NetworkSecret};
|
||||
use crate::state::{RecordBody, SignedRecord};
|
||||
|
||||
/// Schema version written by this build.
|
||||
pub const SCHEMA_VERSION: i64 = 6;
|
||||
pub const SCHEMA_VERSION: i64 = 7;
|
||||
|
||||
/// Key of the stored hostname setting.
|
||||
const SETTING_HOSTNAME: &str = "hostname";
|
||||
@@ -221,6 +221,23 @@ impl StateStore {
|
||||
)
|
||||
.map_err(|err| self.corrupt(format!("cannot migrate schema to 6: {err}")))?;
|
||||
}
|
||||
// Migration 6 -> 7: a claim gained the time its hostname was first
|
||||
// claimed, which changed the signing domain. Records signed before
|
||||
// no longer verify, so they are discarded rather than kept as rows
|
||||
// every read has to reject; each member re-publishes its own claim on
|
||||
// its next run and the others arrive again from their authors. The
|
||||
// version counter is kept, so nothing is ever published twice at one
|
||||
// version.
|
||||
if found < 7 {
|
||||
self.conn
|
||||
.execute_batch(
|
||||
"BEGIN;
|
||||
DELETE FROM signed_records;
|
||||
PRAGMA user_version = 7;
|
||||
COMMIT;",
|
||||
)
|
||||
.map_err(|err| self.corrupt(format!("cannot migrate schema to 7: {err}")))?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -431,12 +448,16 @@ impl StateStore {
|
||||
|
||||
/// Sets the auto-start flag of a configured network.
|
||||
pub fn set_auto_start(&self, network_id: NetworkId, auto_start: bool) -> Result<()> {
|
||||
self.conn
|
||||
let changed = self
|
||||
.conn
|
||||
.execute(
|
||||
"UPDATE networks SET auto_start = ?2 WHERE network_id = ?1",
|
||||
params![network_id.as_bytes().as_slice(), auto_start as i64],
|
||||
)
|
||||
.map_err(|err| Error::Storage(format!("cannot update network: {err}")))?;
|
||||
if changed == 0 {
|
||||
return Err(Error::NetworkUnknown(network_id));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -781,6 +802,44 @@ impl StateStore {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Atomically enables a network and puts it first in the startup order.
|
||||
pub fn prefer_network(&self, network: NetworkId) -> Result<()> {
|
||||
let configured = self.list_networks()?;
|
||||
if !configured.iter().any(|n| n.network_id == network) {
|
||||
return Err(Error::NetworkUnknown(network));
|
||||
}
|
||||
let existing = self.get_setting("network_start_order")?.unwrap_or_default();
|
||||
let mut order = vec![network];
|
||||
for word in existing.split(',') {
|
||||
if let Ok(id) = word.parse::<NetworkId>()
|
||||
&& configured.iter().any(|n| n.network_id == id)
|
||||
&& !order.contains(&id)
|
||||
{
|
||||
order.push(id);
|
||||
}
|
||||
}
|
||||
let value = order
|
||||
.iter()
|
||||
.map(ToString::to_string)
|
||||
.collect::<Vec<_>>()
|
||||
.join(",");
|
||||
let transaction = self
|
||||
.conn
|
||||
.unchecked_transaction()
|
||||
.map_err(|err| Error::Storage(format!("cannot begin network preference: {err}")))?;
|
||||
transaction
|
||||
.execute(
|
||||
"UPDATE networks SET auto_start = 1 WHERE network_id = ?1",
|
||||
params![network.as_bytes().as_slice()],
|
||||
)
|
||||
.map_err(|err| Error::Storage(format!("cannot enable network: {err}")))?;
|
||||
transaction.execute("INSERT INTO settings (key, value) VALUES ('network_start_order', ?1) ON CONFLICT(key) DO UPDATE SET value = excluded.value", params![value])
|
||||
.map_err(|err| Error::Storage(format!("cannot store network preference: {err}")))?;
|
||||
transaction
|
||||
.commit()
|
||||
.map_err(|err| Error::Storage(format!("cannot commit network preference: {err}")))
|
||||
}
|
||||
|
||||
/// Reads an arbitrary setting.
|
||||
pub fn get_setting(&self, key: &str) -> Result<Option<String>> {
|
||||
self.conn
|
||||
|
||||
@@ -162,7 +162,7 @@ async fn rejoining_after_leaving_is_not_taken_for_a_stale_record() {
|
||||
})
|
||||
.await;
|
||||
assert!(
|
||||
tsunagi::state::DEFAULT_IPV4_RANGE.contains(again),
|
||||
tsunagi::state::derived_ipv4_range(network_id).contains(again),
|
||||
"an address in the network's range: {again}"
|
||||
);
|
||||
|
||||
|
||||
@@ -0,0 +1,308 @@
|
||||
//! Connectivity does not depend on what kind of link each hop happens to be,
|
||||
//! and a member is online however far away it is.
|
||||
|
||||
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
|
||||
|
||||
use std::collections::{HashMap, HashSet};
|
||||
use std::sync::{Arc, Mutex};
|
||||
|
||||
use bytes::Bytes;
|
||||
use iroh::EndpointId;
|
||||
use tsunagi::NetworkId;
|
||||
use tsunagi::agent::DataPath;
|
||||
use tsunagi::dataplane::transport::SharedLink;
|
||||
use tsunagi::dataplane::{IpPlugin, PluginCapability, PluginError};
|
||||
use tsunagi::discovery::SharedMemoryDiscovery;
|
||||
use tsunagi::testing::{TestAgent, network, wait_for_peers, wait_until};
|
||||
|
||||
/// A protocol that moves bytes between two members unchanged and keeps the
|
||||
/// link it was handed, so a test can use the link directly.
|
||||
#[derive(Debug)]
|
||||
struct Carrier {
|
||||
protocol: String,
|
||||
links: Mutex<HashMap<EndpointId, SharedLink>>,
|
||||
}
|
||||
|
||||
impl Carrier {
|
||||
fn new(protocol: &str) -> Arc<Self> {
|
||||
Arc::new(Self {
|
||||
protocol: protocol.into(),
|
||||
links: Mutex::default(),
|
||||
})
|
||||
}
|
||||
|
||||
fn link(&self, peer: EndpointId) -> Option<SharedLink> {
|
||||
self.links.lock().unwrap().get(&peer).cloned()
|
||||
}
|
||||
}
|
||||
|
||||
impl IpPlugin for Carrier {
|
||||
fn protocol_id(&self) -> &str {
|
||||
&self.protocol
|
||||
}
|
||||
|
||||
fn protocol_version(&self) -> u16 {
|
||||
1
|
||||
}
|
||||
|
||||
fn local_capability(
|
||||
&self,
|
||||
_network: NetworkId,
|
||||
) -> Result<Option<PluginCapability>, PluginError> {
|
||||
Ok(Some(PluginCapability {
|
||||
protocol: self.protocol.clone(),
|
||||
version: 1,
|
||||
enabled: true,
|
||||
data: Vec::new(),
|
||||
}))
|
||||
}
|
||||
|
||||
fn on_peer_capability(
|
||||
&self,
|
||||
_network: NetworkId,
|
||||
_peer: EndpointId,
|
||||
_capability: &PluginCapability,
|
||||
) -> Result<(), PluginError> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn on_peer_link(&self, _network: NetworkId, peer: EndpointId, link: SharedLink) {
|
||||
self.links.lock().unwrap().insert(peer, link);
|
||||
}
|
||||
|
||||
fn on_peer_gone(&self, _network: NetworkId, peer: EndpointId) {
|
||||
self.links.lock().unwrap().remove(&peer);
|
||||
}
|
||||
|
||||
fn on_network_deactivated(&self, _network: NetworkId) {}
|
||||
}
|
||||
|
||||
type Blocked = Arc<Mutex<HashSet<EndpointId>>>;
|
||||
|
||||
async fn member(
|
||||
discovery: &SharedMemoryDiscovery,
|
||||
plugins: &[Arc<Carrier>],
|
||||
blocked: &Blocked,
|
||||
) -> TestAgent {
|
||||
let plugins: Vec<_> = plugins.to_vec();
|
||||
let blocked = Arc::clone(blocked);
|
||||
TestAgent::spawn_with(
|
||||
move |config| {
|
||||
let config = config.with_unreachable_data_peers(blocked);
|
||||
plugins
|
||||
.into_iter()
|
||||
.fold(config, |config, plugin| config.with_plugin(plugin))
|
||||
},
|
||||
discovery,
|
||||
)
|
||||
.await
|
||||
.unwrap()
|
||||
}
|
||||
|
||||
async fn exchange(from: &SharedLink, to: &SharedLink, payload: &'static [u8]) {
|
||||
from.send(Bytes::from_static(payload)).unwrap();
|
||||
let got = tokio::time::timeout(tsunagi::testing::DEADLINE, to.recv())
|
||||
.await
|
||||
.expect("the datagram should arrive")
|
||||
.expect("the link stays open");
|
||||
assert_eq!(got, Bytes::from_static(payload));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_tunnel_crosses_a_member_that_shares_no_protocol_with_either_end_and_moves_to_a_direct_link_by_itself()
|
||||
{
|
||||
// A and B share one protocol and nothing else. C shares a different
|
||||
// one with each, so every link here is of a different kind and the
|
||||
// middle has never heard of what A and B speak. A and B cannot reach
|
||||
// each other directly, which is the case a blocked UDP path produces.
|
||||
let discovery = SharedMemoryDiscovery::new();
|
||||
let (name, secret) = network("mixed-transports");
|
||||
|
||||
let blocked_a: Blocked = Arc::default();
|
||||
let blocked_b: Blocked = Arc::default();
|
||||
let a_shared = Carrier::new("shared");
|
||||
let b_shared = Carrier::new("shared");
|
||||
let a = member(
|
||||
&discovery,
|
||||
&[Carrier::new("a-and-c"), Arc::clone(&a_shared)],
|
||||
&blocked_a,
|
||||
)
|
||||
.await;
|
||||
let b = member(
|
||||
&discovery,
|
||||
&[Carrier::new("c-and-b"), Arc::clone(&b_shared)],
|
||||
&blocked_b,
|
||||
)
|
||||
.await;
|
||||
let c = member(
|
||||
&discovery,
|
||||
&[Carrier::new("a-and-c"), Carrier::new("c-and-b")],
|
||||
&Blocked::default(),
|
||||
)
|
||||
.await;
|
||||
let (a_id, b_id, c_id) = (
|
||||
a.agent.endpoint_id(),
|
||||
b.agent.endpoint_id(),
|
||||
c.agent.endpoint_id(),
|
||||
);
|
||||
blocked_a.lock().unwrap().insert(b_id);
|
||||
blocked_b.lock().unwrap().insert(a_id);
|
||||
|
||||
let network_id = a.agent.join_network(&name, &secret).await.unwrap();
|
||||
b.agent.join_network(&name, &secret).await.unwrap();
|
||||
c.agent.join_network(&name, &secret).await.unwrap();
|
||||
for agent in [&a, &b, &c] {
|
||||
wait_for_peers(&agent.agent, network_id, 2).await;
|
||||
}
|
||||
|
||||
// The tunnel comes up through C, and data goes both ways.
|
||||
let to_b = wait_until("a has a link to b", || async { a_shared.link(b_id) }).await;
|
||||
let to_a = wait_until("b has a link to a", || async { b_shared.link(a_id) }).await;
|
||||
exchange(&to_b, &to_a, b"a to b, sealed").await;
|
||||
exchange(&to_a, &to_b, b"b to a, sealed").await;
|
||||
|
||||
// The status says what is true: online, and through whom.
|
||||
let status = a.agent.network_status(network_id).await.unwrap();
|
||||
let b_peer = status.peers.iter().find(|p| p.endpoint_id == b_id).unwrap();
|
||||
assert_eq!(b_peer.data_path, DataPath::Relayed { hops: 2, via: c_id });
|
||||
let c_peer = status.peers.iter().find(|p| p.endpoint_id == c_id).unwrap();
|
||||
assert!(matches!(&c_peer.data_path, DataPath::Direct { transport } if transport == "a-and-c"));
|
||||
let b_member = wait_until("a lists b as a member", || async {
|
||||
let status = a.agent.network_status(network_id).await.ok()?;
|
||||
status.members.into_iter().find(|m| m.endpoint_id == b_id)
|
||||
})
|
||||
.await;
|
||||
assert!(b_member.online);
|
||||
let carried = c.agent.network_status(network_id).await.unwrap().relay;
|
||||
assert!(carried.forwarded >= 2, "{carried:?}");
|
||||
|
||||
// The direct path becomes possible: the same link carries on, now
|
||||
// without C, and nobody has to do anything about it.
|
||||
blocked_a.lock().unwrap().clear();
|
||||
blocked_b.lock().unwrap().clear();
|
||||
wait_until("a reaches b directly", || async {
|
||||
let status = a.agent.network_status(network_id).await.ok()?;
|
||||
let peer = status.peers.iter().find(|p| p.endpoint_id == b_id)?;
|
||||
matches!(&peer.data_path, DataPath::Direct { transport } if transport == "shared")
|
||||
.then_some(())
|
||||
})
|
||||
.await;
|
||||
let forwarded = c
|
||||
.agent
|
||||
.network_status(network_id)
|
||||
.await
|
||||
.unwrap()
|
||||
.relay
|
||||
.forwarded;
|
||||
exchange(&to_b, &to_a, b"direct now").await;
|
||||
assert!(
|
||||
Arc::ptr_eq(&to_b, &a_shared.link(b_id).unwrap()),
|
||||
"the tunnel's link is the one it always had"
|
||||
);
|
||||
assert_eq!(
|
||||
c.agent
|
||||
.network_status(network_id)
|
||||
.await
|
||||
.unwrap()
|
||||
.relay
|
||||
.forwarded,
|
||||
forwarded,
|
||||
"nothing goes through C any more"
|
||||
);
|
||||
|
||||
a.stop().await;
|
||||
b.stop().await;
|
||||
c.stop().await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_name_belongs_to_the_member_that_claimed_it_first_on_every_member() {
|
||||
let discovery = SharedMemoryDiscovery::new();
|
||||
let (name, secret) = network("first-claim");
|
||||
let spawn = |hostname: &'static str| {
|
||||
let discovery = discovery.clone();
|
||||
async move {
|
||||
TestAgent::spawn_with(move |config| config.with_hostname(hostname), &discovery)
|
||||
.await
|
||||
.unwrap()
|
||||
}
|
||||
};
|
||||
|
||||
let first = spawn("music").await;
|
||||
let network_id = first.agent.join_network(&name, &secret).await.unwrap();
|
||||
let first_id = first.agent.endpoint_id();
|
||||
wait_until("the first claim is signed", || async {
|
||||
let status = first.agent.network_status(network_id).await.ok()?;
|
||||
status
|
||||
.members
|
||||
.iter()
|
||||
.any(|m| m.endpoint_id == first_id && m.hostname.as_deref() == Some("music"))
|
||||
.then_some(())
|
||||
})
|
||||
.await;
|
||||
|
||||
// Whatever the ids turn out to be, the later one does not take it.
|
||||
let second = spawn("music").await;
|
||||
let third = spawn("studio").await;
|
||||
second.agent.join_network(&name, &secret).await.unwrap();
|
||||
third.agent.join_network(&name, &secret).await.unwrap();
|
||||
let second_id = second.agent.endpoint_id();
|
||||
for agent in [&first, &second, &third] {
|
||||
wait_for_peers(&agent.agent, network_id, 2).await;
|
||||
}
|
||||
|
||||
for (viewer, label) in [(&first, "first"), (&second, "second"), (&third, "third")] {
|
||||
let status = wait_until(&format!("{label} sees both claims"), || async {
|
||||
let status = viewer.agent.network_status(network_id).await.ok()?;
|
||||
// The roster is the authors of signed records, so a member
|
||||
// being in it means its claim has arrived.
|
||||
let both = [first_id, second_id]
|
||||
.iter()
|
||||
.all(|id| status.members.iter().any(|m| m.endpoint_id == *id));
|
||||
both.then_some(status)
|
||||
})
|
||||
.await;
|
||||
let holder = status
|
||||
.members
|
||||
.iter()
|
||||
.find(|m| m.endpoint_id == first_id)
|
||||
.unwrap();
|
||||
let loser = status
|
||||
.members
|
||||
.iter()
|
||||
.find(|m| m.endpoint_id == second_id)
|
||||
.unwrap();
|
||||
assert_eq!(holder.hostname.as_deref(), Some("music"), "{label}");
|
||||
assert_eq!(loser.hostname, None, "{label}: no name for the later one");
|
||||
let conflict = loser.hostname_conflict.as_ref().expect(label);
|
||||
assert_eq!(conflict.name, "music");
|
||||
assert_eq!(conflict.holder, first_id);
|
||||
assert!(holder.hostname_conflict.is_none(), "{label}");
|
||||
// Nothing about being second makes it less of a member.
|
||||
assert!(
|
||||
loser.online || viewer.agent.endpoint_id() == second_id,
|
||||
"{label}"
|
||||
);
|
||||
}
|
||||
|
||||
// The later one is told so itself, and keeps working.
|
||||
let own = second.agent.network_status(network_id).await.unwrap();
|
||||
let conflict = own.hostname_conflict.expect("told its name is taken");
|
||||
assert_eq!(
|
||||
(conflict.name.as_str(), conflict.holder),
|
||||
("music", first_id)
|
||||
);
|
||||
assert!(
|
||||
first
|
||||
.agent
|
||||
.network_status(network_id)
|
||||
.await
|
||||
.unwrap()
|
||||
.hostname_conflict
|
||||
.is_none()
|
||||
);
|
||||
|
||||
first.stop().await;
|
||||
second.stop().await;
|
||||
third.stop().await;
|
||||
}
|
||||
@@ -265,10 +265,7 @@ async fn deactivating_one_network_leaves_the_agent_and_others_running() {
|
||||
|
||||
#[tokio::test]
|
||||
async fn two_networks_on_one_agent_each_get_a_range_of_their_own() {
|
||||
// One agent, one interface: the configured range can only belong to one
|
||||
// of them. The second does not go without — it takes the range derived
|
||||
// from its own id, which every one of its members derives identically,
|
||||
// so it is an agreement and not a local invention.
|
||||
// Each network derives its range independently of activation order.
|
||||
let discovery = SharedMemoryDiscovery::new();
|
||||
let (first, first_secret) = network("two-ranges-one");
|
||||
let (second, second_secret) = network("two-ranges-two");
|
||||
@@ -309,8 +306,8 @@ async fn two_networks_on_one_agent_each_get_a_range_of_their_own() {
|
||||
"different networks, different addresses"
|
||||
);
|
||||
assert!(
|
||||
tsunagi::state::DEFAULT_IPV4_RANGE.contains(addresses.0),
|
||||
"the first keeps the configured range: {}",
|
||||
tsunagi::state::derived_ipv4_range(one).contains(addresses.0),
|
||||
"the first uses its derived range: {}",
|
||||
addresses.0
|
||||
);
|
||||
assert!(
|
||||
@@ -319,7 +316,7 @@ async fn two_networks_on_one_agent_each_get_a_range_of_their_own() {
|
||||
addresses.1
|
||||
);
|
||||
assert!(
|
||||
!tsunagi::state::DEFAULT_IPV4_RANGE.contains(addresses.1),
|
||||
!tsunagi::state::derived_ipv4_range(one).contains(addresses.1),
|
||||
"and not in the one the first network holds"
|
||||
);
|
||||
|
||||
|
||||
@@ -0,0 +1,336 @@
|
||||
//! Derived defaults and local exclusion of overlapping overlay networks.
|
||||
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
|
||||
|
||||
use tsunagi::Agent;
|
||||
use tsunagi::discovery::SharedMemoryDiscovery;
|
||||
use tsunagi::state::{DEFAULT_IPV4_RANGE, Ipv4Range, derived_ipv4_range};
|
||||
use tsunagi::testing::{TestAgent, config_with, network, wait_until};
|
||||
|
||||
// These tests need discovery to happen before a collision, so keep the initial
|
||||
// default outside the established ranges. Derived collisions have their own test.
|
||||
fn network_outside(
|
||||
name: &str,
|
||||
ranges: &[Ipv4Range],
|
||||
) -> (
|
||||
tsunagi::identity::NetworkName,
|
||||
tsunagi::identity::NetworkSecret,
|
||||
) {
|
||||
use tsunagi::identity::{NetworkKeys, NetworkName, NetworkSecret};
|
||||
let name = NetworkName::new(name).unwrap();
|
||||
let secret = (0..=255u8)
|
||||
.map(|seed| NetworkSecret::from_bytes([seed; 32]).unwrap())
|
||||
.find(|secret| {
|
||||
let derived = derived_ipv4_range(NetworkKeys::derive(&name, secret).network_id());
|
||||
ranges.iter().all(|range| !range.contains(derived.base))
|
||||
})
|
||||
.expect("a deterministic test secret with a non-overlapping default");
|
||||
(name, secret)
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn colliding_derived_defaults_are_blocked_instead_of_locally_renumbered() {
|
||||
use std::collections::HashMap;
|
||||
use tsunagi::identity::{NetworkKeys, NetworkName, NetworkSecret};
|
||||
let secret = NetworkSecret::from_bytes([19; 32]).unwrap();
|
||||
let mut seen = HashMap::new();
|
||||
let (first, second, range) = (0..=65_536)
|
||||
.find_map(|index| {
|
||||
let name = NetworkName::new(format!("derived-collision-{index}")).unwrap();
|
||||
let range = derived_ipv4_range(NetworkKeys::derive(&name, &secret).network_id());
|
||||
seen.insert(range.base, name.clone())
|
||||
.map(|first| (first, name, range))
|
||||
})
|
||||
.expect("more network names than possible /24 ranges");
|
||||
let agent = TestAgent::spawn(&SharedMemoryDiscovery::new())
|
||||
.await
|
||||
.unwrap();
|
||||
let first = agent.agent.join_network(&first, &secret).await.unwrap();
|
||||
let second = agent.agent.join_network(&second, &secret).await.unwrap();
|
||||
assert!(agent.agent.is_active(first).await);
|
||||
assert!(!agent.agent.is_active(second).await);
|
||||
let status = agent.agent.status().await.unwrap();
|
||||
let blocked = status.network(&second).unwrap();
|
||||
assert_eq!(blocked.range, Some(range));
|
||||
assert_eq!(blocked.range_conflicts[0].network_id, first);
|
||||
assert_eq!(
|
||||
agent.agent.switch_network(second).await.unwrap(),
|
||||
vec![first]
|
||||
);
|
||||
assert_eq!(
|
||||
agent.agent.network_status(second).await.unwrap().range,
|
||||
Some(range)
|
||||
);
|
||||
agent.agent.shutdown().await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn isolated_first_members_choose_the_same_range_and_keep_it_after_restart() {
|
||||
let discovery = SharedMemoryDiscovery::new();
|
||||
let (name, secret) = network("derived-first");
|
||||
let a = TestAgent::spawn(&discovery).await.unwrap();
|
||||
let b = TestAgent::spawn(&SharedMemoryDiscovery::new())
|
||||
.await
|
||||
.unwrap();
|
||||
let id = a.agent.join_network(&name, &secret).await.unwrap();
|
||||
b.agent.join_network(&name, &secret).await.unwrap();
|
||||
let expected = derived_ipv4_range(id);
|
||||
for agent in [&a.agent, &b.agent] {
|
||||
wait_until("isolated member allocates in its derived range", || async {
|
||||
let status = agent.network_status(id).await.ok()?;
|
||||
(status.range == Some(expected) && !status.members.is_empty()).then_some(())
|
||||
})
|
||||
.await;
|
||||
}
|
||||
let dir = a.stop().await;
|
||||
let a = Agent::spawn(config_with(dir.path(), &discovery))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(a.network_status(id).await.unwrap().range, Some(expected));
|
||||
a.shutdown().await;
|
||||
b.agent.shutdown().await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn overlap_blocks_activation_switches_and_resumes_across_restart() {
|
||||
let discovery = SharedMemoryDiscovery::new();
|
||||
let a = TestAgent::spawn_with(
|
||||
|c| c.with_overlay_ipv4_range(Some(DEFAULT_IPV4_RANGE)),
|
||||
&discovery,
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
let (one_name, one_secret) = network("conflict-one");
|
||||
let (two_name, two_secret) = network("conflict-two");
|
||||
let one = a.agent.join_network(&one_name, &one_secret).await.unwrap();
|
||||
a.agent.network_status(one).await.unwrap();
|
||||
let two = a.agent.join_network(&two_name, &two_secret).await.unwrap();
|
||||
assert!(a.agent.is_active(one).await);
|
||||
assert!(!a.agent.is_active(two).await);
|
||||
assert!(matches!(
|
||||
a.agent.activate_network(two).await,
|
||||
Err(tsunagi::Error::NetworkRangeConflict { .. })
|
||||
));
|
||||
let status = a.agent.status().await.unwrap();
|
||||
let blocked = status.network(&two).unwrap();
|
||||
assert_eq!(blocked.range, Some(DEFAULT_IPV4_RANGE));
|
||||
assert_eq!(blocked.range_conflicts[0].network_id, one);
|
||||
assert_eq!(blocked.range_conflicts[0].name, one_name);
|
||||
assert_eq!(a.agent.switch_network(two).await.unwrap(), vec![one]);
|
||||
assert!(a.agent.is_active(two).await);
|
||||
assert!(!a.agent.is_active(one).await);
|
||||
|
||||
a.agent.network_status(two).await.unwrap();
|
||||
let dir = a.stop().await;
|
||||
let a = Agent::spawn(config_with(dir.path(), &discovery))
|
||||
.await
|
||||
.unwrap();
|
||||
assert!(a.is_active(two).await, "the user's choice survives restart");
|
||||
assert!(!a.is_active(one).await);
|
||||
a.deactivate_network(two).await.unwrap();
|
||||
assert!(
|
||||
a.is_active(one).await,
|
||||
"suspended networks resume when the blocker stops"
|
||||
);
|
||||
assert_eq!(
|
||||
a.network_status(one).await.unwrap().range,
|
||||
Some(DEFAULT_IPV4_RANGE),
|
||||
"legacy ranges survive the new default"
|
||||
);
|
||||
a.shutdown().await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn late_authenticated_state_with_a_nested_range_suspends_only_that_network() {
|
||||
let discovery = SharedMemoryDiscovery::new();
|
||||
let broad: Ipv4Range = "10.90.0.0/16".parse().unwrap();
|
||||
let narrow: Ipv4Range = "10.90.12.0/24".parse().unwrap();
|
||||
let (name, secret) = network_outside("late-conflict", &[broad]);
|
||||
let remote = TestAgent::spawn_with(|c| c.with_overlay_ipv4_range(Some(narrow)), &discovery)
|
||||
.await
|
||||
.unwrap();
|
||||
let target = remote.agent.join_network(&name, &secret).await.unwrap();
|
||||
wait_until("remote range is persisted", || async {
|
||||
remote
|
||||
.agent
|
||||
.network_status(target)
|
||||
.await
|
||||
.ok()?
|
||||
.members
|
||||
.first()
|
||||
.map(|_| ())
|
||||
})
|
||||
.await;
|
||||
|
||||
let (other_name, other_secret) = network("existing-broad");
|
||||
// Seed an existing network using a real agent and its signed claim.
|
||||
let seeded = TestAgent::spawn_with(
|
||||
|c| c.with_overlay_ipv4_range(Some(broad)),
|
||||
&SharedMemoryDiscovery::new(),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
let other = seeded
|
||||
.agent
|
||||
.join_network(&other_name, &other_secret)
|
||||
.await
|
||||
.unwrap();
|
||||
wait_until("broad range is persisted", || async {
|
||||
seeded
|
||||
.agent
|
||||
.network_status(other)
|
||||
.await
|
||||
.ok()?
|
||||
.members
|
||||
.first()
|
||||
.map(|_| ())
|
||||
})
|
||||
.await;
|
||||
let dir = seeded.stop().await;
|
||||
let seeded = Agent::spawn(config_with(dir.path(), &discovery))
|
||||
.await
|
||||
.unwrap();
|
||||
seeded.join_network(&name, &secret).await.unwrap();
|
||||
wait_until("late overlap suspends the joining network", || async {
|
||||
let status = seeded.status().await.ok()?;
|
||||
let blocked = status.network(&target)?;
|
||||
(!seeded.is_active(target).await
|
||||
&& blocked
|
||||
.range_conflicts
|
||||
.iter()
|
||||
.any(|c| c.network_id == other))
|
||||
.then_some(())
|
||||
})
|
||||
.await;
|
||||
assert!(seeded.is_active(other).await);
|
||||
assert_eq!(seeded.switch_network(target).await.unwrap(), vec![other]);
|
||||
assert_eq!(
|
||||
seeded.network_status(target).await.unwrap().range,
|
||||
Some(narrow)
|
||||
);
|
||||
seeded.shutdown().await;
|
||||
remote.agent.shutdown().await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_wide_range_suspends_all_overlaps_and_manual_stop_cancels_resume() {
|
||||
let discovery = SharedMemoryDiscovery::new();
|
||||
let hub = TestAgent::spawn(&discovery).await.unwrap();
|
||||
let mut members = Vec::new();
|
||||
let mut ids = Vec::new();
|
||||
for (name, range) in [
|
||||
("narrow-one", "10.81.0.0/24"),
|
||||
("narrow-two", "10.81.1.0/24"),
|
||||
] {
|
||||
let peer = TestAgent::spawn_with(
|
||||
|c| c.with_overlay_ipv4_range(Some(range.parse().unwrap())),
|
||||
&discovery,
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
let (name, secret) = network_outside(
|
||||
name,
|
||||
&[
|
||||
"10.81.0.0/24".parse().unwrap(),
|
||||
"10.81.1.0/24".parse().unwrap(),
|
||||
],
|
||||
);
|
||||
let id = peer.agent.join_network(&name, &secret).await.unwrap();
|
||||
peer.agent.network_status(id).await.unwrap();
|
||||
hub.agent.join_network(&name, &secret).await.unwrap();
|
||||
wait_until("hub adopts the established narrow range", || async {
|
||||
(hub.agent.network_status(id).await.ok()?.range == Some(range.parse().unwrap()))
|
||||
.then_some(())
|
||||
})
|
||||
.await;
|
||||
members.push(peer);
|
||||
ids.push(id);
|
||||
}
|
||||
let (name, secret) = network_outside(
|
||||
"wide-overlap",
|
||||
&[
|
||||
"10.81.0.0/24".parse().unwrap(),
|
||||
"10.81.1.0/24".parse().unwrap(),
|
||||
],
|
||||
);
|
||||
let wide = TestAgent::spawn_with(
|
||||
|c| c.with_overlay_ipv4_range(Some("10.0.0.0/8".parse().unwrap())),
|
||||
&discovery,
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
let id = wide.agent.join_network(&name, &secret).await.unwrap();
|
||||
wide.agent.network_status(id).await.unwrap();
|
||||
hub.agent.join_network(&name, &secret).await.unwrap();
|
||||
wait_until("wide range is blocked by both narrow networks", || async {
|
||||
let status = hub.agent.status().await.ok()?;
|
||||
let blocked = status.network(&id)?;
|
||||
(!hub.agent.is_active(id).await && blocked.range_conflicts.len() == 2).then_some(())
|
||||
})
|
||||
.await;
|
||||
let stopped = hub.agent.switch_network(id).await.unwrap();
|
||||
assert_eq!(stopped.len(), 2);
|
||||
assert!(ids.iter().all(|id| stopped.contains(id)));
|
||||
assert_eq!(hub.agent.switch_network(ids[0]).await.unwrap(), vec![id]);
|
||||
assert!(
|
||||
hub.agent.is_active(ids[1]).await,
|
||||
"switching away from the wide range frees the other narrow range too"
|
||||
);
|
||||
assert_eq!(hub.agent.switch_network(id).await.unwrap().len(), 2);
|
||||
hub.agent.pause_network(ids[0]).await.unwrap();
|
||||
hub.agent.deactivate_network(id).await.unwrap();
|
||||
assert!(
|
||||
!hub.agent.is_active(ids[0]).await,
|
||||
"explicitly stopped networks do not resume"
|
||||
);
|
||||
assert!(hub.agent.is_active(ids[1]).await);
|
||||
assert!(
|
||||
hub.agent
|
||||
.switch_network(tsunagi::NetworkId::from_bytes([0; 32]))
|
||||
.await
|
||||
.is_err()
|
||||
);
|
||||
assert!(
|
||||
hub.agent.is_active(ids[1]).await,
|
||||
"unknown selectors do not stop any network"
|
||||
);
|
||||
hub.agent.shutdown().await;
|
||||
wide.agent.shutdown().await;
|
||||
for peer in members {
|
||||
peer.agent.shutdown().await;
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn simultaneous_switches_never_leave_overlapping_networks_active() {
|
||||
let discovery = SharedMemoryDiscovery::new();
|
||||
let agent = TestAgent::spawn_with(
|
||||
|c| c.with_overlay_ipv4_range(Some(DEFAULT_IPV4_RANGE)),
|
||||
&discovery,
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
let (a, a_secret) = network("switch-a");
|
||||
let (b, b_secret) = network("switch-b");
|
||||
let a = agent.agent.join_network(&a, &a_secret).await.unwrap();
|
||||
agent.agent.network_status(a).await.unwrap();
|
||||
let b = agent.agent.join_network(&b, &b_secret).await.unwrap();
|
||||
let (left, right) = tokio::join!(agent.agent.switch_network(b), agent.agent.switch_network(a));
|
||||
left.unwrap();
|
||||
right.unwrap();
|
||||
assert_ne!(
|
||||
agent.agent.is_active(a).await,
|
||||
agent.agent.is_active(b).await
|
||||
);
|
||||
assert_eq!(
|
||||
agent
|
||||
.agent
|
||||
.list_networks()
|
||||
.await
|
||||
.unwrap()
|
||||
.iter()
|
||||
.filter(|n| n.active)
|
||||
.count(),
|
||||
1
|
||||
);
|
||||
agent.agent.shutdown().await;
|
||||
}
|
||||
@@ -0,0 +1,130 @@
|
||||
//! Opaque transport negotiation stays inside authenticated network sessions.
|
||||
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
|
||||
|
||||
use std::sync::{Arc, Mutex};
|
||||
use tsunagi::dataplane::transport::{ControlSender, PacketTransport, SharedLink, TransportError};
|
||||
use tsunagi::discovery::SharedMemoryDiscovery;
|
||||
use tsunagi::proto::message::{ControlMessage, TransportPayload};
|
||||
use tsunagi::testing::{TestAgent, network, wait_for_peers, wait_until};
|
||||
use tsunagi::{BoxFuture, NetworkId};
|
||||
|
||||
#[derive(Debug, Default)]
|
||||
struct Negotiator {
|
||||
control: Mutex<Option<Arc<dyn ControlSender>>>,
|
||||
received: Mutex<Vec<(NetworkId, iroh::EndpointId, Vec<u8>)>>,
|
||||
}
|
||||
impl PacketTransport for Negotiator {
|
||||
fn name(&self) -> &str {
|
||||
"negotiator"
|
||||
}
|
||||
fn as_any(&self) -> &dyn std::any::Any {
|
||||
self
|
||||
}
|
||||
fn attach_control(&self, control: Arc<dyn ControlSender>) {
|
||||
*self.control.lock().unwrap() = Some(control);
|
||||
}
|
||||
fn on_control(
|
||||
&self,
|
||||
network: NetworkId,
|
||||
peer: iroh::EndpointId,
|
||||
_: &str,
|
||||
payload: &[u8],
|
||||
) -> Result<(), TransportError> {
|
||||
self.received
|
||||
.lock()
|
||||
.unwrap()
|
||||
.push((network, peer, payload.to_vec()));
|
||||
Ok(())
|
||||
}
|
||||
fn open<'a>(
|
||||
&'a self,
|
||||
_: NetworkId,
|
||||
_: iroh::EndpointId,
|
||||
_: &'a str,
|
||||
) -> BoxFuture<'a, Result<SharedLink, TransportError>> {
|
||||
Box::pin(async { Err(TransportError::Closed) })
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn negotiated_secrets_are_redacted_scoped_and_delivered_over_real_control() {
|
||||
let discovery = SharedMemoryDiscovery::new();
|
||||
let a_transport = Arc::new(Negotiator::default());
|
||||
let b_transport = Arc::new(Negotiator::default());
|
||||
let a = TestAgent::spawn_with(
|
||||
|c| c.with_custom_transport("negotiator", a_transport.clone()),
|
||||
&discovery,
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
let b = TestAgent::spawn_with(
|
||||
|c| c.with_custom_transport("negotiator", b_transport.clone()),
|
||||
&discovery,
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
let (name, secret) = network("private-negotiation");
|
||||
let id = a.agent.join_network(&name, &secret).await.unwrap();
|
||||
b.agent.join_network(&name, &secret).await.unwrap();
|
||||
wait_for_peers(&a.agent, id, 1).await;
|
||||
let message = ControlMessage::Transport {
|
||||
protocol: "negotiator".into(),
|
||||
payload: TransportPayload::new(b"never-export-this-secret".to_vec()),
|
||||
};
|
||||
assert!(!format!("{message:?}").contains("never-export"));
|
||||
let sender = a_transport.control.lock().unwrap().clone().unwrap();
|
||||
sender
|
||||
.send(
|
||||
id,
|
||||
b.agent.endpoint_id(),
|
||||
"negotiator",
|
||||
TransportPayload::new(b"never-export-this-secret".to_vec()),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
wait_until("private transport message", || async {
|
||||
(!b_transport.received.lock().unwrap().is_empty()).then_some(())
|
||||
})
|
||||
.await;
|
||||
assert_eq!(
|
||||
b_transport.received.lock().unwrap()[0],
|
||||
(
|
||||
id,
|
||||
a.agent.endpoint_id(),
|
||||
b"never-export-this-secret".to_vec()
|
||||
)
|
||||
);
|
||||
let unknown = NetworkId::from_bytes([0xa5; 32]);
|
||||
assert!(
|
||||
sender
|
||||
.send(
|
||||
unknown,
|
||||
b.agent.endpoint_id(),
|
||||
"negotiator",
|
||||
TransportPayload::new(vec![1])
|
||||
)
|
||||
.await
|
||||
.is_err()
|
||||
);
|
||||
a.agent.shutdown().await;
|
||||
b.agent.shutdown().await;
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn private_bytes_are_bounded_on_decode_and_by_local_policy() {
|
||||
let limits = tsunagi::Limits {
|
||||
max_transport_payload_len: 4,
|
||||
..Default::default()
|
||||
};
|
||||
let message = |len| ControlMessage::Transport {
|
||||
protocol: "awg".into(),
|
||||
payload: TransportPayload::new(vec![42; len]),
|
||||
};
|
||||
assert!(tsunagi::proto::message::validate(&message(4), &limits).is_ok());
|
||||
assert!(tsunagi::proto::message::validate(&message(5), &limits).is_err());
|
||||
let bytes = postcard::to_stdvec(&TransportPayload::new(vec![1; 4097])).unwrap();
|
||||
assert!(postcard::from_bytes::<TransportPayload>(&bytes).is_err());
|
||||
let bytes = postcard::to_stdvec(&message(4)).unwrap();
|
||||
let decoded: ControlMessage = postcard::from_bytes(&bytes).unwrap();
|
||||
assert_eq!(decoded, message(4));
|
||||
}
|
||||
@@ -115,6 +115,7 @@ pub struct TcpTlsLink {
|
||||
peer: EndpointId,
|
||||
tx: mpsc::Sender<Bytes>,
|
||||
rx: tokio::sync::Mutex<mpsc::Receiver<Bytes>>,
|
||||
tasks: [tokio::task::AbortHandle; 2],
|
||||
closed_notify: Arc<Notify>,
|
||||
is_closed: Arc<AtomicBool>,
|
||||
remote_addr: String,
|
||||
@@ -136,7 +137,7 @@ impl TcpTlsLink {
|
||||
|
||||
let closed_flag_w = Arc::clone(&is_closed);
|
||||
let closed_notify_w = Arc::clone(&closed_notify);
|
||||
tokio::spawn(async move {
|
||||
let writer_task = tokio::spawn(async move {
|
||||
while let Some(msg) = rx_out.recv().await {
|
||||
if msg.len() > MAX_FRAME_SIZE {
|
||||
continue;
|
||||
@@ -158,7 +159,7 @@ impl TcpTlsLink {
|
||||
|
||||
let closed_flag_r = Arc::clone(&is_closed);
|
||||
let closed_notify_r = Arc::clone(&closed_notify);
|
||||
tokio::spawn(async move {
|
||||
let reader_task = tokio::spawn(async move {
|
||||
let mut len_buf = [0u8; 2];
|
||||
loop {
|
||||
if reader.read_exact(&mut len_buf).await.is_err() {
|
||||
@@ -185,6 +186,7 @@ impl TcpTlsLink {
|
||||
peer,
|
||||
tx: tx_out,
|
||||
rx: tokio::sync::Mutex::new(rx_in),
|
||||
tasks: [writer_task.abort_handle(), reader_task.abort_handle()],
|
||||
closed_notify,
|
||||
is_closed,
|
||||
remote_addr,
|
||||
@@ -192,7 +194,19 @@ impl TcpTlsLink {
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for TcpTlsLink {
|
||||
fn drop(&mut self) {
|
||||
self.close();
|
||||
}
|
||||
}
|
||||
impl PacketLink for TcpTlsLink {
|
||||
fn close(&self) {
|
||||
self.is_closed.store(true, Ordering::Release);
|
||||
self.closed_notify.notify_waiters();
|
||||
for task in &self.tasks {
|
||||
task.abort();
|
||||
}
|
||||
}
|
||||
fn network(&self) -> NetworkId {
|
||||
self.network
|
||||
}
|
||||
|
||||
@@ -183,7 +183,21 @@ async fn run(
|
||||
.await
|
||||
.map(|report| {
|
||||
let state = if report.active { "started" } else { "stopped" };
|
||||
format!("{} {}", state, report.name)
|
||||
if report.suspended.is_empty() {
|
||||
format!("{} {}", state, report.name)
|
||||
} else {
|
||||
format!(
|
||||
"{} {}; suspended {} until it stops",
|
||||
state,
|
||||
report.name,
|
||||
report
|
||||
.suspended
|
||||
.iter()
|
||||
.map(|n| format!("{} ({})", n.name, n.network_id))
|
||||
.collect::<Vec<_>>()
|
||||
.join(", ")
|
||||
)
|
||||
}
|
||||
})
|
||||
.map_err(|err| err.to_string()),
|
||||
Command::SetBroadcast {
|
||||
@@ -211,7 +225,17 @@ async fn run(
|
||||
}
|
||||
Command::Join { name, secret } => ipc::join_network(socket, &name, &secret)
|
||||
.await
|
||||
.map(|report| format!("joined {}", report.name))
|
||||
.map(|report| {
|
||||
format!(
|
||||
"{} {}",
|
||||
if report.active {
|
||||
"joined"
|
||||
} else {
|
||||
"configured; stopped by range conflict:"
|
||||
},
|
||||
report.name
|
||||
)
|
||||
})
|
||||
.map_err(|err| err.to_string()),
|
||||
Command::Leave { network_id } => ipc::leave_network(socket, &network_id)
|
||||
.await
|
||||
|
||||
@@ -9,13 +9,17 @@ use std::collections::HashMap;
|
||||
|
||||
use eframe::egui;
|
||||
|
||||
use tsunagi::ipc::{NetworkReport, OverlayPeerReport};
|
||||
use tsunagi::ipc::{DataPathReport, NetworkReport, OverlayPeerReport};
|
||||
|
||||
use crate::agent::{AgentClient, Command};
|
||||
use crate::exit;
|
||||
use crate::format;
|
||||
use crate::stats::{Traffic, Unit};
|
||||
|
||||
/// A device that works but not as well as it could: reached through others,
|
||||
/// or with a name somebody else holds.
|
||||
const AMBER: egui::Color32 = egui::Color32::from_rgb(0xd9, 0xa4, 0x2c);
|
||||
|
||||
/// A readable title for a network's window.
|
||||
pub(crate) fn title(network: &NetworkReport) -> String {
|
||||
format!("tsunagi · {}", network.name)
|
||||
@@ -23,14 +27,23 @@ pub(crate) fn title(network: &NetworkReport) -> String {
|
||||
|
||||
/// One row of the device table, built from a tunnel and/or a signed member.
|
||||
struct Device<'a> {
|
||||
/// The agent's own answer to whether it can be reached by any path.
|
||||
online: bool,
|
||||
/// How packets reach it.
|
||||
path: Option<&'a DataPathReport>,
|
||||
/// What to call the member its traffic goes through, when it does.
|
||||
via_name: Option<String>,
|
||||
/// The name it claimed that another member holds, and what to call that
|
||||
/// member.
|
||||
name_taken: Option<(&'a str, String)>,
|
||||
/// The last hostname seen for it, kept while it is offline.
|
||||
hostname: Option<&'a str>,
|
||||
/// What clicking the hostname copies: `hostname.network`, so it resolves.
|
||||
copy: Option<String>,
|
||||
/// The full endpoint id, which clicking the short form copies.
|
||||
id: &'a str,
|
||||
proto: &'a str,
|
||||
/// End-to-end tunnel protocol, independent of the path carrying it.
|
||||
tunnel_protocol: &'a str,
|
||||
address: Option<String>,
|
||||
handshake_secs: Option<u64>,
|
||||
overlay: Option<&'a OverlayPeerReport>,
|
||||
@@ -93,6 +106,13 @@ fn name_cell(ui: &mut egui::Ui, device: &Device<'_>) {
|
||||
}
|
||||
_ => format::copy_field(ui, &short_id(device.id), device.id),
|
||||
}
|
||||
if let Some((name, holder)) = &device.name_taken {
|
||||
ui.label(egui::RichText::new(egui_phosphor::regular::WARNING).color(AMBER))
|
||||
.on_hover_text(format!(
|
||||
"`{name}` was claimed first by {holder}. This device works by address, \
|
||||
but nobody finds it by that name."
|
||||
));
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
@@ -276,6 +296,11 @@ fn known_ids<'a>(network: &'a NetworkReport, own_id: &str) -> Vec<&'a str> {
|
||||
ids
|
||||
}
|
||||
|
||||
/// How many other devices can be reached now, by any path.
|
||||
pub(crate) fn online_count(network: &NetworkReport, own_id: &str) -> usize {
|
||||
devices(network, own_id).iter().filter(|d| d.online).count()
|
||||
}
|
||||
|
||||
/// How many other devices this network knows of, connected or not.
|
||||
pub(crate) fn known_count(network: &NetworkReport, own_id: &str) -> usize {
|
||||
known_ids(network, own_id).len()
|
||||
@@ -310,14 +335,33 @@ fn devices<'a>(network: &'a NetworkReport, own_id: &str) -> Vec<Device<'a>> {
|
||||
.map(|id| {
|
||||
let overlay = overlay_peers.iter().find(|p| p.endpoint_id == id);
|
||||
let member = network.members.iter().find(|m| m.endpoint_id == id);
|
||||
let online = overlay.is_some_and(|p| p.handshake_secs_ago.is_some());
|
||||
let online = member.is_some_and(|m| m.online)
|
||||
|| network.peers.iter().any(|p| p.endpoint_id == id);
|
||||
let path = network
|
||||
.peers
|
||||
.iter()
|
||||
.find(|p| p.endpoint_id == id)
|
||||
.map(|p| &p.path)
|
||||
.or_else(|| member.map(|m| &m.path));
|
||||
let via_name = match path {
|
||||
Some(DataPathReport::Relayed { via, .. }) => Some(exit::name_of(network, via)),
|
||||
_ => None,
|
||||
};
|
||||
let name_taken = member.and_then(|m| {
|
||||
m.hostname_conflict
|
||||
.as_ref()
|
||||
.map(|c| (c.name.as_str(), exit::name_of(network, &c.holder)))
|
||||
});
|
||||
let hostname = hostnames.get(id).copied();
|
||||
Device {
|
||||
online,
|
||||
path,
|
||||
via_name,
|
||||
name_taken,
|
||||
hostname,
|
||||
copy: hostname.map(|h| format!("{h}.{}", network.name)),
|
||||
id,
|
||||
proto: overlay.map_or("—", |p| p.protocol.as_str()),
|
||||
tunnel_protocol: overlay.map_or("—", |p| p.protocol.as_str()),
|
||||
address: overlay
|
||||
.and_then(|p| p.address.clone())
|
||||
.or_else(|| member.and_then(|m| m.overlay_address_v4.clone())),
|
||||
@@ -346,9 +390,9 @@ fn devices<'a>(network: &'a NetworkReport, own_id: &str) -> Vec<Device<'a>> {
|
||||
const COLUMN_SAMPLES: [&str; 9] = [
|
||||
"",
|
||||
"exit",
|
||||
"tcp-tls",
|
||||
"wg-quic · waiting for handshake",
|
||||
"255.255.255.255",
|
||||
"23h 59m",
|
||||
"direct over tcp-tls",
|
||||
"99999/99999",
|
||||
"1023.9 MB / 1023.9 MB",
|
||||
"1023.9 kB/s",
|
||||
@@ -465,9 +509,9 @@ fn device_table(
|
||||
for (column, header) in [
|
||||
"device",
|
||||
"exit",
|
||||
"proto",
|
||||
"end-to-end",
|
||||
"address",
|
||||
"handshake",
|
||||
"data path",
|
||||
"pkts",
|
||||
"bytes",
|
||||
"rate",
|
||||
@@ -477,7 +521,22 @@ fn device_table(
|
||||
.enumerate()
|
||||
{
|
||||
cell(ui, column, &mut widths, |ui| {
|
||||
ui.weak(header);
|
||||
let response = ui.weak(header);
|
||||
match column {
|
||||
2 => {
|
||||
response.on_hover_text(
|
||||
"Protocol that encrypts traffic between this device and the \
|
||||
destination. It is independent of the route carrying it.",
|
||||
);
|
||||
}
|
||||
4 => {
|
||||
response.on_hover_text(
|
||||
"Current route carrying the encrypted tunnel traffic. `direct \
|
||||
over tcp-tls` means one Tsunagi hop with no mesh relay.",
|
||||
);
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
});
|
||||
}
|
||||
ui.end_row();
|
||||
@@ -488,7 +547,7 @@ fn device_table(
|
||||
exit_cell(ui, device, network_id, confirm);
|
||||
});
|
||||
cell(ui, 2, &mut widths, |ui| {
|
||||
ui.label(device.proto);
|
||||
tunnel_cell(ui, device);
|
||||
});
|
||||
cell(ui, 3, &mut widths, |ui| match &device.address {
|
||||
Some(address) => format::copy_field(ui, address, address),
|
||||
@@ -496,13 +555,7 @@ fn device_table(
|
||||
ui.weak("—");
|
||||
}
|
||||
});
|
||||
cell(ui, 4, &mut widths, |ui| {
|
||||
ui.label(
|
||||
device
|
||||
.handshake_secs
|
||||
.map_or_else(|| "offline".to_string(), format::age),
|
||||
);
|
||||
});
|
||||
cell(ui, 4, &mut widths, |ui| data_path_cell(ui, device));
|
||||
|
||||
match device.overlay.filter(|_| device.online) {
|
||||
Some(peer) => {
|
||||
@@ -580,6 +633,211 @@ fn device_table(
|
||||
ui.memory_mut(|m| m.data.insert_temp(widths_id, widths));
|
||||
}
|
||||
|
||||
/// A protocol name as it should appear in the table.
|
||||
fn displayed_protocol(protocol: &str) -> &str {
|
||||
if protocol.is_empty() {
|
||||
"wg-quic"
|
||||
} else {
|
||||
protocol
|
||||
}
|
||||
}
|
||||
|
||||
/// The end-to-end tunnel and its own health, without transport details.
|
||||
fn tunnel_text(protocol: &str, handshake_secs: Option<u64>) -> String {
|
||||
let protocol = displayed_protocol(protocol);
|
||||
if protocol == "—" {
|
||||
return protocol.to_string();
|
||||
}
|
||||
match handshake_secs {
|
||||
Some(secs) if protocol == "tcp-tls" => {
|
||||
format!("{protocol} · up {}", format::age(secs))
|
||||
}
|
||||
Some(secs) => format!("{protocol} · handshake {} ago", format::age(secs)),
|
||||
None => format!("{protocol} · waiting for handshake"),
|
||||
}
|
||||
}
|
||||
|
||||
fn tunnel_cell(ui: &mut egui::Ui, device: &Device<'_>) {
|
||||
let protocol = displayed_protocol(device.tunnel_protocol);
|
||||
let text = tunnel_text(protocol, device.handshake_secs);
|
||||
if protocol == "—" {
|
||||
ui.weak(text);
|
||||
return;
|
||||
}
|
||||
let hint = match device.path {
|
||||
Some(DataPathReport::Direct { transport }) => format!(
|
||||
"The end-to-end {protocol} tunnel is currently carried over a direct {transport} \
|
||||
transport."
|
||||
),
|
||||
Some(DataPathReport::Relayed { hops, .. }) => format!(
|
||||
"The end-to-end {protocol} tunnel is currently carried through other devices over \
|
||||
{hops} transport hops."
|
||||
),
|
||||
Some(DataPathReport::None) | None => {
|
||||
format!("The end-to-end {protocol} tunnel does not have a data path carrying it yet.")
|
||||
}
|
||||
};
|
||||
ui.label(text).on_hover_text(hint);
|
||||
}
|
||||
|
||||
/// Current route text and whether that route is the preferred direct form.
|
||||
fn data_path_text(
|
||||
path: Option<&DataPathReport>,
|
||||
via_name: Option<&str>,
|
||||
online: bool,
|
||||
) -> (String, bool) {
|
||||
if !online {
|
||||
return ("offline".into(), false);
|
||||
}
|
||||
match path {
|
||||
Some(DataPathReport::Direct { transport }) => (format!("direct over {transport}"), true),
|
||||
Some(DataPathReport::Relayed { hops, .. }) => (
|
||||
format!("via {} · {hops} hops", via_name.unwrap_or("another device")),
|
||||
false,
|
||||
),
|
||||
Some(DataPathReport::None) | None => ("no data path yet".into(), false),
|
||||
}
|
||||
}
|
||||
|
||||
/// How a device is reached right now.
|
||||
///
|
||||
/// A device reached through others is online, over a longer path, and says
|
||||
/// so: the colour is the only difference in how it is treated.
|
||||
fn data_path_cell(ui: &mut egui::Ui, device: &Device<'_>) {
|
||||
let (text, good) = data_path_text(device.path, device.via_name.as_deref(), device.online);
|
||||
let response = if good {
|
||||
ui.label(text)
|
||||
} else {
|
||||
ui.label(egui::RichText::new(text).color(AMBER))
|
||||
};
|
||||
response.on_hover_text(match device.path {
|
||||
Some(DataPathReport::Direct { .. }) => {
|
||||
"One Tsunagi hop with no mesh relay. The end-to-end tunnel may use a different \
|
||||
protocol from this transport."
|
||||
}
|
||||
Some(DataPathReport::Relayed { .. }) => {
|
||||
"No direct transport yet: traffic goes through another device, and moves to a \
|
||||
direct path by itself as soon as one comes up."
|
||||
}
|
||||
_ if device.online => "Connected, but no data path carries packets yet.",
|
||||
_ => "This device is not reachable now.",
|
||||
});
|
||||
}
|
||||
|
||||
fn yes_no(value: bool) -> &'static str {
|
||||
if value { "yes" } else { "no" }
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
|
||||
use super::*;
|
||||
use tsunagi::ipc::{HostnameConflictReport, MemberReport};
|
||||
|
||||
fn member(id: &str, online: bool, path: DataPathReport) -> MemberReport {
|
||||
MemberReport {
|
||||
endpoint_id: id.into(),
|
||||
online,
|
||||
path,
|
||||
..Default::default()
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_device_reached_through_another_is_online_whatever_its_tunnel_says() {
|
||||
// The tray used to call a device online only while its tunnel had
|
||||
// handshaken, which disagreed with the command line about the same
|
||||
// device. Both now read the agent's answer.
|
||||
let network = NetworkReport {
|
||||
members: vec![
|
||||
member("own", true, DataPathReport::None),
|
||||
member(
|
||||
"far",
|
||||
true,
|
||||
DataPathReport::Relayed {
|
||||
hops: 2,
|
||||
via: "near".into(),
|
||||
},
|
||||
),
|
||||
member("away", false, DataPathReport::None),
|
||||
],
|
||||
..Default::default()
|
||||
};
|
||||
let all = devices(&network, "own");
|
||||
let far = all.iter().find(|d| d.id == "far").unwrap();
|
||||
assert!(far.online);
|
||||
assert!(matches!(
|
||||
far.path,
|
||||
Some(DataPathReport::Relayed { hops: 2, .. })
|
||||
));
|
||||
assert!(!all.iter().find(|d| d.id == "away").unwrap().online);
|
||||
assert_eq!(online_count(&network, "own"), 1);
|
||||
assert_eq!(known_count(&network, "own"), 2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_name_somebody_else_holds_is_carried_to_the_row() {
|
||||
let mut taken = member("late", true, DataPathReport::None);
|
||||
taken.hostname_conflict = Some(HostnameConflictReport {
|
||||
name: "music".into(),
|
||||
holder: "early".into(),
|
||||
});
|
||||
let network = NetworkReport {
|
||||
members: vec![member("early", true, DataPathReport::None), taken],
|
||||
..Default::default()
|
||||
};
|
||||
let all = devices(&network, "own");
|
||||
let late = all.iter().find(|d| d.id == "late").unwrap();
|
||||
assert_eq!(
|
||||
late.name_taken.as_ref().map(|(name, _)| *name),
|
||||
Some("music")
|
||||
);
|
||||
assert!(
|
||||
all.iter()
|
||||
.find(|d| d.id == "early")
|
||||
.unwrap()
|
||||
.name_taken
|
||||
.is_none()
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn tunnel_and_direct_transport_are_named_as_different_layers() {
|
||||
assert_eq!(tunnel_text("wg", Some(1)), "wg · handshake 1s ago");
|
||||
assert_eq!(
|
||||
data_path_text(
|
||||
Some(&DataPathReport::Direct {
|
||||
transport: "tcp-tls".into(),
|
||||
}),
|
||||
None,
|
||||
true,
|
||||
),
|
||||
("direct over tcp-tls".into(), true)
|
||||
);
|
||||
assert_eq!(tunnel_text("tcp-tls", Some(42)), "tcp-tls · up 42s");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn incomplete_and_longer_paths_stay_explicit() {
|
||||
assert_eq!(
|
||||
tunnel_text("wg-quic", None),
|
||||
"wg-quic · waiting for handshake"
|
||||
);
|
||||
assert_eq!(
|
||||
data_path_text(
|
||||
Some(&DataPathReport::Relayed {
|
||||
hops: 2,
|
||||
via: "near".into(),
|
||||
}),
|
||||
Some("iris"),
|
||||
true,
|
||||
),
|
||||
("via iris · 2 hops".into(), false)
|
||||
);
|
||||
assert_eq!(
|
||||
data_path_text(Some(&DataPathReport::None), None, true),
|
||||
("no data path yet".into(), false)
|
||||
);
|
||||
assert_eq!(data_path_text(None, None, false), ("offline".into(), false));
|
||||
}
|
||||
}
|
||||
@@ -78,6 +78,7 @@ fn colour_of(health: Health) -> [u8; 3] {
|
||||
match health {
|
||||
Health::Disconnected => DARK_RED,
|
||||
Health::Connected => BLUE,
|
||||
Health::RangeConflict => [0xd9, 0x91, 0x20],
|
||||
Health::ExitNode => GREEN,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -243,8 +243,8 @@ fn draw_network(
|
||||
}
|
||||
ui.weak("peers");
|
||||
ui.label(format!(
|
||||
"{} connected · {} known",
|
||||
network.peers.len(),
|
||||
"{} online · {} known",
|
||||
devices::online_count(network, own_id),
|
||||
devices::known_count(network, own_id)
|
||||
));
|
||||
ui.end_row();
|
||||
@@ -252,13 +252,30 @@ fn draw_network(
|
||||
|
||||
format::sparkline(ui, series, state.unit, 34.0);
|
||||
|
||||
if !network.range_conflicts.is_empty() {
|
||||
ui.colored_label(BAD, "Stopped: overlapping network ranges");
|
||||
for conflict in &network.range_conflicts {
|
||||
ui.label(format!("{} ({}) uses {}", conflict.name, conflict.network_id, conflict.range));
|
||||
}
|
||||
ui.label("Switching suspends these networks until this network stops, then resumes them automatically.");
|
||||
if network.auto_start { ui.weak("This network will resume automatically when its range is free."); }
|
||||
}
|
||||
ui.horizontal(|ui| {
|
||||
let mut active = network.active;
|
||||
if ui.checkbox(&mut active, "active").changed() {
|
||||
agent.send(Command::SetActive {
|
||||
network_id: network.network_id.clone(),
|
||||
active,
|
||||
});
|
||||
if !network.active && !network.range_conflicts.is_empty() {
|
||||
if ui.button("Switch to this network").clicked() {
|
||||
agent.send(Command::SetActive { network_id: network.network_id.clone(), active: true });
|
||||
}
|
||||
if network.auto_start && ui.button("Keep stopped").clicked() {
|
||||
agent.send(Command::SetActive { network_id: network.network_id.clone(), active: false });
|
||||
}
|
||||
} else {
|
||||
let mut active = network.active;
|
||||
if ui.checkbox(&mut active, "active").changed() {
|
||||
agent.send(Command::SetActive {
|
||||
network_id: network.network_id.clone(),
|
||||
active,
|
||||
});
|
||||
}
|
||||
}
|
||||
let mut broadcast = network.broadcast;
|
||||
if ui.checkbox(&mut broadcast, "broadcast").changed() {
|
||||
|
||||
@@ -22,6 +22,8 @@ pub(crate) enum Health {
|
||||
Disconnected,
|
||||
/// Connected, nothing out of the ordinary.
|
||||
Connected,
|
||||
/// At least one configured network is blocked by an overlapping active range.
|
||||
RangeConflict,
|
||||
/// This device's traffic leaves through an exit node, and that works.
|
||||
ExitNode,
|
||||
}
|
||||
@@ -33,6 +35,13 @@ impl Health {
|
||||
let Some(report) = report else {
|
||||
return Self::Disconnected;
|
||||
};
|
||||
if report
|
||||
.networks
|
||||
.iter()
|
||||
.any(|network| network.auto_start && !network.range_conflicts.is_empty())
|
||||
{
|
||||
return Self::RangeConflict;
|
||||
}
|
||||
if !report
|
||||
.networks
|
||||
.iter()
|
||||
@@ -56,6 +65,7 @@ impl Health {
|
||||
match self {
|
||||
Self::Disconnected => "tsunagi: not connected",
|
||||
Self::Connected => "tsunagi: connected",
|
||||
Self::RangeConflict => "tsunagi: network range conflict; open to switch networks",
|
||||
Self::ExitNode => "tsunagi: using an exit node",
|
||||
}
|
||||
}
|
||||
@@ -108,6 +118,24 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn overlapping_networks_are_visible_even_while_other_peers_are_connected() {
|
||||
let mut blocked = NetworkReport {
|
||||
auto_start: true,
|
||||
..Default::default()
|
||||
};
|
||||
blocked
|
||||
.range_conflicts
|
||||
.push(tsunagi::ipc::RangeConflictReport {
|
||||
name: "other".into(),
|
||||
network_id: "other-id".into(),
|
||||
range: "10.90.0.0/16".into(),
|
||||
});
|
||||
let status = report(vec![network(2, ExitReport::default()), blocked]);
|
||||
assert_eq!(Health::of(Some(&status)), Health::RangeConflict);
|
||||
assert!(Health::RangeConflict.label().contains("switch networks"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn no_agent_or_nobody_connected_is_the_red_state() {
|
||||
assert_eq!(Health::of(None), Health::Disconnected);
|
||||
|
||||
@@ -79,6 +79,8 @@ pub const WIREGUARD_OVERHEAD: u32 = 32;
|
||||
/// Configuration of the WireGuard plugin.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct WireguardConfig {
|
||||
/// Diagnostic protocol id when this end-to-end engine is reused by another carrier.
|
||||
pub protocol: &'static str,
|
||||
/// Directory for the plugin's own key store. Separate from agent state.
|
||||
pub state_dir: PathBuf,
|
||||
/// WireGuard keepalive, which keeps tunnels and their links warm.
|
||||
@@ -100,6 +102,7 @@ impl WireguardConfig {
|
||||
/// Creates a configuration rooted at `state_dir` with sensible defaults.
|
||||
pub fn new(state_dir: impl Into<PathBuf>) -> Self {
|
||||
Self {
|
||||
protocol: WIREGUARD_PROTOCOL,
|
||||
state_dir: state_dir.into(),
|
||||
keepalive: Some(25),
|
||||
mtu: DEFAULT_MTU,
|
||||
@@ -394,7 +397,7 @@ impl Worker {
|
||||
fn report(&self, network: NetworkId, reason: impl std::fmt::Display) {
|
||||
tracing::warn!(network = %network.fmt_short(), %reason, "wireguard plugin error");
|
||||
if let Some(context) = self.context.get() {
|
||||
context.report_error(network, WIREGUARD_PROTOCOL, reason.to_string());
|
||||
context.report_error(network, self.config.protocol, reason.to_string());
|
||||
}
|
||||
}
|
||||
|
||||
@@ -647,7 +650,7 @@ async fn run(worker: Arc<Worker>, mut commands: mpsc::Receiver<Command>) {
|
||||
|
||||
impl IpPlugin for WireguardPlugin {
|
||||
fn protocol_id(&self) -> &str {
|
||||
WIREGUARD_PROTOCOL
|
||||
self.worker.config.protocol
|
||||
}
|
||||
|
||||
fn protocol_version(&self) -> u16 {
|
||||
@@ -686,7 +689,7 @@ impl IpPlugin for WireguardPlugin {
|
||||
// Identity only. Where to send packets is the transport's business.
|
||||
let announcement = WgAnnouncement::new(network, &state.key.public());
|
||||
Ok(Some(PluginCapability {
|
||||
protocol: WIREGUARD_PROTOCOL.to_string(),
|
||||
protocol: self.worker.config.protocol.to_string(),
|
||||
version: crate::announcement::ANNOUNCEMENT_VERSION,
|
||||
enabled: true,
|
||||
data: announcement.encode()?,
|
||||
|
||||
@@ -42,6 +42,7 @@ fn source(agent: Agent, plugin: Arc<WireguardPlugin>) -> Arc<dyn tsunagi::ipc::R
|
||||
transport: format!("{:?}", peer.transport),
|
||||
rtt_ms: peer.rtt.map(|rtt| rtt.as_millis() as u64),
|
||||
exit_node: peer.exit_node,
|
||||
path: Default::default(),
|
||||
})
|
||||
.collect(),
|
||||
overlay: plugin.overview(net.network_id).map(|view| {
|
||||
@@ -276,6 +277,7 @@ impl tsunagi::ipc::ReportSource for Control {
|
||||
network_id: network_id.to_string(),
|
||||
already_configured: already,
|
||||
name_shared_with: None,
|
||||
active: self.0.is_active(network_id).await,
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
@@ -109,6 +109,30 @@ impl WgAgent {
|
||||
.await
|
||||
}
|
||||
|
||||
async fn spawn_auto(discovery: &SharedMemoryDiscovery, tag: &str) -> Self {
|
||||
let dir = TempDir::new().unwrap();
|
||||
let tuns = MemoryTunFactory::new();
|
||||
let plugin = WireguardPlugin::open(
|
||||
WireguardConfig::new(dir.path().join("wireguard"))
|
||||
.with_reconcile(Duration::from_millis(20), Duration::from_millis(250)),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
let agent = Agent::spawn(
|
||||
config_with(dir.path(), discovery)
|
||||
.with_interface(Arc::new(tuns.clone()), tag, 1280)
|
||||
.with_plugin(plugin.clone() as Arc<dyn IpPlugin>),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
Self {
|
||||
dir,
|
||||
agent,
|
||||
plugin,
|
||||
tuns,
|
||||
}
|
||||
}
|
||||
|
||||
/// Starts an agent with an explicit overlay range, or none at all.
|
||||
async fn spawn_range(
|
||||
discovery: &SharedMemoryDiscovery,
|
||||
@@ -989,7 +1013,7 @@ async fn two_networks_share_one_interface_with_keys_and_ranges_of_their_own() {
|
||||
let (name_b, secret_b) = network("wg-right");
|
||||
let beta_range = Some("10.99.0.0/16".parse::<Ipv4Range>().unwrap());
|
||||
|
||||
let hub = WgAgent::spawn(&discovery, "th").await;
|
||||
let hub = WgAgent::spawn_auto(&discovery, "th").await;
|
||||
let left = WgAgent::spawn(&discovery, "tl").await;
|
||||
let right = WgAgent::spawn_range(&discovery, "tr", beta_range).await;
|
||||
|
||||
@@ -1661,8 +1685,10 @@ async fn lan_broadcast_fanout_is_scoped_opt_in_and_does_not_reflood() {
|
||||
bytes[28..].copy_from_slice(b"LAN GAME");
|
||||
// UDP checksum zero is valid for IPv4.
|
||||
let sum: u32 = bytes[..20]
|
||||
.chunks_exact(2)
|
||||
.map(|w| u16::from_be_bytes([w[0], w[1]]) as u32)
|
||||
.as_chunks::<2>()
|
||||
.0
|
||||
.iter()
|
||||
.map(|w| u16::from_be_bytes(*w) as u32)
|
||||
.sum();
|
||||
let checksum = !((sum & 0xffff) + (sum >> 16)) as u16;
|
||||
bytes[10..12].copy_from_slice(&checksum.to_be_bytes());
|
||||
|
||||
@@ -231,6 +231,9 @@ impl WireguardLink {
|
||||
}
|
||||
|
||||
impl PacketLink for WireguardLink {
|
||||
fn close(&self) {
|
||||
WireguardLink::close(self);
|
||||
}
|
||||
fn network(&self) -> NetworkId {
|
||||
self.network
|
||||
}
|
||||
|
||||
@@ -14,6 +14,7 @@ path = "src/main.rs"
|
||||
[dependencies]
|
||||
tsunagi-core = { path = "../tsunagi-core", version = "0.1.0", features = ["tun-device", "dns-publish"] }
|
||||
tsunagi-wireguard = { path = "../tsunagi-wireguard", version = "0.1.0" }
|
||||
tsunagi-awg = { path = "../tsunagi-awg", version = "0.1.0" }
|
||||
tsunagi-wg-quic = { path = "../tsunagi-wg-quic", version = "0.1.0" }
|
||||
tsunagi-tcp-tls = { path = "../tsunagi-tcp-tls", version = "0.1.0" }
|
||||
iroh.workspace = true
|
||||
|
||||
+518
-73
File diff suppressed because it is too large.
Load diff
@@ -6,7 +6,7 @@
|
||||
|
||||
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
|
||||
|
||||
use std::time::Duration;
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use tempfile::TempDir;
|
||||
|
||||
@@ -50,8 +50,13 @@ fn read_networks(dir: &TempDir) -> Vec<tsunagi::storage::StoredNetwork> {
|
||||
/// An agent with no network at all, the way a daemon is started before
|
||||
/// anything has been decided.
|
||||
fn start_bare(port: u16) -> Running {
|
||||
start_bare_range(port, None)
|
||||
}
|
||||
|
||||
fn start_bare_range(port: u16, range: Option<&str>) -> Running {
|
||||
let dir = TempDir::new().unwrap();
|
||||
let child = std::process::Command::new(env!("CARGO_BIN_EXE_tsng"))
|
||||
let mut command = std::process::Command::new(env!("CARGO_BIN_EXE_tsng"));
|
||||
command
|
||||
.arg("up")
|
||||
.arg("--state-dir")
|
||||
.arg(dir.path().join("state"))
|
||||
@@ -62,11 +67,31 @@ fn start_bare(port: u16) -> Running {
|
||||
.arg(format!("127.0.0.1:{port}"))
|
||||
.args(["--log", "error", "--status-interval", "0"])
|
||||
.stdout(std::process::Stdio::null())
|
||||
.stderr(std::process::Stdio::null())
|
||||
.spawn()
|
||||
.expect("the agent binary starts");
|
||||
std::thread::sleep(Duration::from_secs(2));
|
||||
Running { child, dir }
|
||||
.stderr(std::process::Stdio::null());
|
||||
if let Some(range) = range {
|
||||
command.args(["--ipv4-range", range]);
|
||||
}
|
||||
let child = command.spawn().expect("the agent binary starts");
|
||||
let mut agent = Running { child, dir };
|
||||
let runtime = tokio::runtime::Builder::new_current_thread()
|
||||
.enable_all()
|
||||
.build()
|
||||
.unwrap();
|
||||
let socket = tsunagi::ipc::control_socket_path(&agent.dir.path().join("state"));
|
||||
runtime.block_on(async {
|
||||
tokio::time::timeout(Duration::from_secs(30), async {
|
||||
while !tsunagi::ipc::is_serving(&socket).await {
|
||||
assert!(
|
||||
agent.child.try_wait().unwrap().is_none(),
|
||||
"agent exited before becoming ready"
|
||||
);
|
||||
tokio::time::sleep(Duration::from_millis(25)).await;
|
||||
}
|
||||
})
|
||||
.await
|
||||
.expect("agent becomes ready");
|
||||
});
|
||||
agent
|
||||
}
|
||||
|
||||
/// An agent already in one network, the way most of these start.
|
||||
@@ -461,3 +486,139 @@ fn an_exit_node_is_offered_per_network_only_when_asked_and_nobody_offers_by_defa
|
||||
String::from_utf8_lossy(&unknown.stderr)
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn overlapping_networks_are_reported_and_cli_start_switches_then_resumes() {
|
||||
let agent = start_bare_range(0, Some("10.90.0.0/16"));
|
||||
for name in ["resident", "games"] {
|
||||
let joined = agent.run(&["join", "-n", name, "-s", "a-secret-for-the-conflict-test"]);
|
||||
assert!(
|
||||
joined.status.success(),
|
||||
"{}",
|
||||
String::from_utf8_lossy(&joined.stderr)
|
||||
);
|
||||
if name == "games" {
|
||||
let text = String::from_utf8_lossy(&joined.stdout);
|
||||
assert!(
|
||||
text.contains("stopped because its range overlaps"),
|
||||
"{text}"
|
||||
);
|
||||
assert!(!text.contains("it is running"), "{text}");
|
||||
}
|
||||
}
|
||||
let runtime = tokio::runtime::Builder::new_current_thread()
|
||||
.enable_all()
|
||||
.build()
|
||||
.unwrap();
|
||||
let socket = tsunagi::ipc::control_socket_path(&agent.dir.path().join("state"));
|
||||
let snapshot = || {
|
||||
runtime
|
||||
.block_on(tsunagi::ipc::request_status(&socket))
|
||||
.unwrap()
|
||||
};
|
||||
let before = snapshot();
|
||||
let resident = before
|
||||
.networks
|
||||
.iter()
|
||||
.find(|n| n.name == "resident")
|
||||
.unwrap();
|
||||
let games = before.networks.iter().find(|n| n.name == "games").unwrap();
|
||||
assert!(resident.active);
|
||||
assert!(!games.active);
|
||||
assert_eq!(games.range_conflicts[0].network_id, resident.network_id);
|
||||
assert_eq!(games.range_conflicts[0].name, "resident");
|
||||
for args in [vec!["status"], vec!["network"]] {
|
||||
let result = agent.run(&args);
|
||||
let text = String::from_utf8_lossy(&result.stdout);
|
||||
assert!(text.contains("range conflict"), "{text}");
|
||||
assert!(text.contains(&resident.network_id), "{text}");
|
||||
assert!(text.contains("tsng network start"), "{text}");
|
||||
}
|
||||
let switched = agent.run(&["network", "start", "games"]);
|
||||
assert!(
|
||||
switched.status.success(),
|
||||
"{}",
|
||||
String::from_utf8_lossy(&switched.stderr)
|
||||
);
|
||||
let text = String::from_utf8_lossy(&switched.stdout);
|
||||
assert!(
|
||||
text.contains("started `games`") && text.contains("suspended `resident`"),
|
||||
"{text}"
|
||||
);
|
||||
let after = snapshot();
|
||||
assert!(
|
||||
after
|
||||
.networks
|
||||
.iter()
|
||||
.find(|n| n.name == "games")
|
||||
.unwrap()
|
||||
.active
|
||||
);
|
||||
assert!(
|
||||
!after
|
||||
.networks
|
||||
.iter()
|
||||
.find(|n| n.name == "resident")
|
||||
.unwrap()
|
||||
.active
|
||||
);
|
||||
assert!(agent.run(&["network", "stop", "games"]).status.success());
|
||||
assert!(
|
||||
snapshot()
|
||||
.networks
|
||||
.iter()
|
||||
.find(|n| n.name == "resident")
|
||||
.unwrap()
|
||||
.active
|
||||
);
|
||||
|
||||
assert!(agent.run(&["network", "start", "games"]).status.success());
|
||||
let stop_suspended = agent.run(&["network", "stop", "resident"]);
|
||||
assert!(stop_suspended.status.success());
|
||||
assert!(String::from_utf8_lossy(&stop_suspended.stdout).contains("stopped `resident`"));
|
||||
assert!(agent.run(&["network", "stop", "games"]).status.success());
|
||||
assert!(
|
||||
snapshot()
|
||||
.networks
|
||||
.iter()
|
||||
.all(|n| !n.active && !n.auto_start)
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cli_defaults_derive_each_networks_range() {
|
||||
let agent = start_bare(0);
|
||||
for name in ["derived-one", "derived-two"] {
|
||||
assert!(
|
||||
agent
|
||||
.run(&["join", "-n", name, "-s", "a-secret-for-the-derived-test"])
|
||||
.status
|
||||
.success()
|
||||
);
|
||||
}
|
||||
let runtime = tokio::runtime::Builder::new_current_thread()
|
||||
.enable_all()
|
||||
.build()
|
||||
.unwrap();
|
||||
let socket = tsunagi::ipc::control_socket_path(&agent.dir.path().join("state"));
|
||||
let deadline = Instant::now() + Duration::from_secs(30);
|
||||
loop {
|
||||
let report = runtime
|
||||
.block_on(tsunagi::ipc::request_status(&socket))
|
||||
.unwrap();
|
||||
if report.networks.iter().all(|n| n.range.is_some()) {
|
||||
assert_eq!(report.networks.len(), 2);
|
||||
for network in report.networks {
|
||||
let id = network.network_id.parse().unwrap();
|
||||
assert!(network.active);
|
||||
assert_eq!(
|
||||
network.range,
|
||||
Some(tsunagi::state::derived_ipv4_range(id).to_string())
|
||||
);
|
||||
}
|
||||
break;
|
||||
}
|
||||
assert!(Instant::now() < deadline, "networks allocate ranges");
|
||||
std::thread::sleep(Duration::from_millis(25));
|
||||
}
|
||||
}
|
||||
Vendored
+1
-1
@@ -22,7 +22,7 @@ service's socket at `/run/tsunagi/agent.sock`):
|
||||
|
||||
```sh
|
||||
sudo tsng status
|
||||
sudo tsng join <network-name> <tsn1…secret>
|
||||
sudo tsng join -n <network-name> -s <tsn1…secret>
|
||||
```
|
||||
|
||||
## How it runs
|
||||
|
||||
Vendored
+1
-1
@@ -28,7 +28,7 @@ group, so after `sudo usermod -aG tsunagi "$USER"` (and logging in again) no
|
||||
|
||||
```sh
|
||||
sudo tsng status
|
||||
sudo tsng join <network-name> <tsn1…secret>
|
||||
sudo tsng join -n <network-name> -s <tsn1…secret>
|
||||
```
|
||||
|
||||
## As your own user — one capability, no service
|
||||
|
||||
Vendored
+1
-1
@@ -48,7 +48,7 @@ Done. Start the agent with:
|
||||
Control it with sudo (the CLI finds the service's socket automatically):
|
||||
|
||||
sudo tsng status
|
||||
sudo tsng join <network-name> <tsn1…secret>
|
||||
sudo tsng join -n <network-name> -s <tsn1…secret>
|
||||
EOF
|
||||
else
|
||||
cat <<EOF
|
||||
|
||||
Vendored
+1
-1
@@ -15,7 +15,7 @@ socket and do **not** need `sudo`:
|
||||
|
||||
```sh
|
||||
tsng status
|
||||
tsng join <network-name> <tsn1…secret>
|
||||
tsng join -n <network-name> -s <tsn1…secret>
|
||||
```
|
||||
|
||||
The agent uses fixed system paths, so the commands find it regardless of whose
|
||||
|
||||
Vendored
+2
-1
@@ -55,8 +55,9 @@ Source: "{#SourceDir}\tsng.exe"; DestDir: "{app}"; Flags: ignoreversion
|
||||
Source: "{#SourceDir}\tsunagi-tray.exe"; DestDir: "{app}"; Flags: ignoreversion
|
||||
; Wintun is the unmodified, signed library from wintun.net; its licence allows
|
||||
; shipping it with an application and asks for the licence text beside it.
|
||||
; In the archive the text sits at its root, two levels above bin\amd64.
|
||||
Source: "{#WintunDir}\wintun.dll"; DestDir: "{app}"; Flags: ignoreversion
|
||||
Source: "{#WintunDir}\LICENSE.txt"; DestDir: "{app}\licenses"; DestName: "wintun-LICENSE.txt"; Flags: ignoreversion
|
||||
Source: "{#WintunDir}\..\..\LICENSE.txt"; DestDir: "{app}\licenses"; DestName: "wintun-LICENSE.txt"; Flags: ignoreversion
|
||||
|
||||
[Dirs]
|
||||
; Identity and network secrets live here; locked down below.
|
||||
|
||||
+116
@@ -0,0 +1,116 @@
|
||||
# Pairwise Amnezia transport
|
||||
|
||||
The CLI's default carrier preference is `wg,awg,wg-quic,tcp-tls`. To select
|
||||
AWG explicitly:
|
||||
|
||||
```sh
|
||||
tsng up --protocol awg
|
||||
tsng up --protocol wg,awg,wg-quic,tcp-tls
|
||||
```
|
||||
|
||||
`awg` is Tsunagi AWG wire version 1. Its packet masking engine is adapted from
|
||||
[RunaRoo/AmneziaWG-RS](https://github.com/RunaRoo/AmneziaWG-RS), with the pinned
|
||||
source and license recorded in
|
||||
[UPSTREAM.md](../crates/tsunagi-awg/UPSTREAM.md). It uses the existing boringtun
|
||||
Noise/WireGuard implementation instead of upstream's Node, global executors
|
||||
and OS interface machinery. Compatibility with ordinary Amnezia servers is
|
||||
not implemented: this carrier connects authenticated Tsunagi members.
|
||||
|
||||
## Negotiation and lifetime
|
||||
|
||||
There is one independent UDP socket and cancellable packet-engine task per
|
||||
`(network, peer)` pair. The smaller endpoint id initiates, following the same
|
||||
rule as other data carriers. Both sides must announce enabled AWG wire version
|
||||
1. An authenticated iroh control session is required throughout; AWG supplies
|
||||
no alternate discovery or membership mechanism.
|
||||
|
||||
Every `open` generates new masking parameters, a header-protection key, an
|
||||
outer WireGuard key pair and a PSK. The initiator sends a private `Offer` with
|
||||
a random attempt id, its public key, candidate socket addresses and the shared
|
||||
profile. The responder validates it, creates its own fresh key pair and socket,
|
||||
and returns an `Answer` naming that attempt. Both send masked handshakes and
|
||||
encrypted delivery probes to the candidates. Only authenticated WireGuard
|
||||
**data**, not a STUN response or an incoming UDP address alone, verifies a path.
|
||||
|
||||
Negotiation payloads are bounded at 4096 bytes (a caller can lower the limit),
|
||||
redacted in Debug, excluded from public message events and erased on drop.
|
||||
They are never capabilities, signed records, cache entries or exported status.
|
||||
At most 128 pairs are admitted per transport, with bounded command/data queues,
|
||||
at most 32 candidates per attempt and finite establishment deadlines.
|
||||
Stale answers and duplicate offers are rejected; a new attempt replaces and
|
||||
closes an older pair engine. Cancellation, peer departure, network deactivation
|
||||
and shutdown retire sockets without waiting for a remote reader.
|
||||
|
||||
A process restart, a failed setup, or a complete fallback cycle returning to AWG
|
||||
starts fresh negotiation. Normal WireGuard rekeying within a live attempt keeps
|
||||
that attempt's masking profile. A failed pair does not reconfigure other pairs
|
||||
or other networks. A full mesh of 15 members uses 14 engines per member and 105
|
||||
independent pairs in total; it does not create additional system interfaces.
|
||||
|
||||
## Masking
|
||||
|
||||
Each profile randomizes disjoint H1–H4 header ranges, a 12–48-byte S1–S4 prefix,
|
||||
a ChaCha20 header-protection key, handshake trailers, junk packet count/lengths,
|
||||
and extra content padding. Prefix lengths are equal within one profile so
|
||||
header detection has one offset. A fresh random nonce is in every packet's
|
||||
prefix; handshake headers and transport headers/counters are protected.
|
||||
The standard WireGuard header is restored before boringtun authenticates a
|
||||
message. This preserves boringtun's standard MAC calculations, which is another
|
||||
reason this is a Tsunagi-specific AWG wire version.
|
||||
|
||||
Before a handshake, an I1 signature packet is generated from either a complete
|
||||
DNS query or SIP OPTIONS template, with fresh identifiers and consistent length
|
||||
fields, followed by random junk packets. I2–I5 are unused in this version.
|
||||
The templates are structural generators, not replayed captures. QUIC imitation
|
||||
is not selected. Masking can change DPI classification; it is not a promise of
|
||||
indistinguishability or that any particular firewall will admit it.
|
||||
|
||||
## Packets, NAT and fallback
|
||||
|
||||
The pair engine carries opaque relay envelopes inside authenticated outer WG
|
||||
packets. It fragments those envelopes into bounded chunks and expires incomplete
|
||||
assemblies; a logical datagram is bounded at 64 KiB and an outer UDP packet is
|
||||
at most 1200 bytes. The encapsulated IPv4 headers are internal framing only: no interface or OS address is created for a
|
||||
pair. Overlay addresses, routing, ownership checks and end-to-end encryption
|
||||
remain independent. A relay decrypts its own adjacent AWG carrier to forward
|
||||
opaque end-to-end ciphertext, never the destination's IP packet.
|
||||
|
||||
Local IPv4 candidates and simultaneous encrypted probes provide direct UDP
|
||||
connectivity. In `direct`/`relay` reach mode the CLI also uses
|
||||
`stun.l.google.com:19302` to discover a server-reflexive address on **this pair's
|
||||
socket**, with a 1.2-second bound. A STUN result is only a candidate. Local reach
|
||||
mode makes no STUN request. Configure or disable it with
|
||||
`-o awg:stun=HOST:PORT` or `-o awg:stun=off`. Failed discovery still permits local
|
||||
candidates. IPv6-only paths, symmetric NAT and networks blocking arbitrary UDP
|
||||
may fail AWG and fall back; there is no AWG-over-QUIC substitution or AWG TURN
|
||||
relay hidden behind its name.
|
||||
|
||||
Data delivery checks go through the actual end-to-end encrypted tunnel, once
|
||||
per second, independently of control-plane RTT. Requests and replies bind a
|
||||
fresh private marker/key to the sender's endpoint id. Unanswered requests expire
|
||||
after three seconds. After a 20-second warmup, at least 15 completed observations
|
||||
and at least 60% loss in the last 20 observations retire a carrier. Probes are
|
||||
consumed locally after source/destination admission and never enter the TUN.
|
||||
AWG additionally checks encrypted outer delivery throughout the pair lifetime.
|
||||
|
||||
Setup failure or persistent runtime loss advances to the next mutually supported
|
||||
carrier in the initiator's order. Earlier expired cooldowns do not jump ahead of
|
||||
untried carriers. Retry cooldowns remain bounded; reaching AWG again negotiates
|
||||
new parameters. Control sessions and the logical end-to-end tunnel survive a
|
||||
carrier change. Status names the end-to-end protocol and the actual carrier
|
||||
separately, so `wg` over `awg` is a valid result.
|
||||
|
||||
Control wire version 6 introduces the private transport message. All members
|
||||
must upgrade together; device identities, network ids, handshake transcript
|
||||
encoding, signed state and stored network configuration are unchanged. Data
|
||||
ALPN remains `tsunagi/data/5`; the relay envelope is unchanged.
|
||||
|
||||
## Verification
|
||||
|
||||
The default rootless, offline tests use real iroh endpoints, membership proofs,
|
||||
SQLite, UDP sockets, masked WireGuard handshakes, end-to-end encryption and
|
||||
memory TUNs. They cover independent three-member pairs, full-size bidirectional
|
||||
packets, restart freshness/socket release, 80% runtime datagram loss, fallback
|
||||
to real iroh and fresh negotiation on retry, plus malformed profiles/messages,
|
||||
role checks, probe forgery and STUN response parsing. These tests do not establish
|
||||
DPI effectiveness, WAN throughput, public NAT traversal or per-pair memory cost.
|
||||
@@ -20,8 +20,9 @@ network secret. The core moves an opaque, bounded payload and never parses it.
|
||||
a `PacketLink` — an authenticated, unreliable datagram channel to one peer for
|
||||
one protocol — and never learn how it is carried.
|
||||
|
||||
The separation between the two planes is **logical, not physical**. Both ride
|
||||
on iroh, on different ALPNs and different connections. That is deliberate:
|
||||
The separation between the two planes is **logical, not physical**. The control plane uses
|
||||
iroh; the data plane can use iroh on its own ALPN and connection, or independent
|
||||
UDP/TLS carriers. That is deliberate:
|
||||
iroh's whole value is hole punching a direct path between peers behind NAT,
|
||||
with a relay as fallback, and a data plane that refused to use it would have to
|
||||
reimplement all of it. What the separation buys is that `proto` knows nothing
|
||||
@@ -32,6 +33,10 @@ A plugin talks to the core through three narrow hooks — `on_network_activated`
|
||||
a `PluginContext` for re-announcements and error reports, and a bounded
|
||||
`shutdown` — so the core never learns anything protocol-specific.
|
||||
|
||||
Private pair negotiation uses a bounded transport sender supplied by the core.
|
||||
Protocol crates do not encode or parse control messages and never receive the
|
||||
network secret. No pair profiles enter signed state.
|
||||
|
||||
A data plane failure never stops the daemon: the control plane keeps running
|
||||
and the agent stays manageable.
|
||||
|
||||
@@ -68,6 +73,7 @@ And in crates of their own:
|
||||
|
||||
| crate | responsibility |
|
||||
|---|---|
|
||||
| `tsunagi-awg` | ephemeral pairwise Amnezia masking, private negotiation and independently encrypted UDP carriers; reuses the end-to-end WireGuard engine |
|
||||
| `tsunagi-wg-quic` | the `wg-quic` protocol: its keys, its announcement, its tunnels |
|
||||
| `tsunagi` | the command line agent, built as the `tsng` executable: the only place that owns a runtime, a logger and signals |
|
||||
|
||||
|
||||
+34
-15
@@ -9,11 +9,15 @@ Two versions exist and are independent:
|
||||
|
||||
- **Identity scheme**, `tsunagi-network-id-v1`. Frozen. Changing it creates a
|
||||
different network space for the same name and secret.
|
||||
- **Control protocol**, ALPN `tsunagi/ctrl/4`, `PROTOCOL_VERSION = 4`.
|
||||
- **Control protocol**, ALPN `tsunagi/ctrl/6`, `PROTOCOL_VERSION = 6`.
|
||||
|
||||
Upgrading the crate or bumping the control protocol must never change an
|
||||
existing `NetworkId`.
|
||||
|
||||
Control version 6 adds private transport negotiation for pairwise AWG and data
|
||||
delivery checks. Data ALPN remains `tsunagi/data/5`. All members must upgrade
|
||||
together; network ids and stored state are unchanged. See [amnezia.md](amnezia.md).
|
||||
|
||||
## Canonical encoding
|
||||
|
||||
Everything that is hashed or MAC'd uses length-prefixed fields, written here as
|
||||
@@ -135,7 +139,7 @@ event does not report a network id.
|
||||
## The data plane protocol
|
||||
|
||||
IP plugin packets never travel on a control connection. They use their own
|
||||
ALPN, `tsunagi/data/4`, on their own iroh connection:
|
||||
ALPN, `tsunagi/data/5`, on their own iroh connection:
|
||||
|
||||
```text
|
||||
initiator -> responder : (the same membership handshake as above)
|
||||
@@ -172,32 +176,46 @@ holding up subsequent packets. Packet IDs are scoped to a QUIC connection;
|
||||
reassembly is created only after the membership handshake. All limits live in
|
||||
`config.rs`. There is no transport-layer retransmission added by this framing.
|
||||
|
||||
Version 4 adds a routing envelope inside the transport fragmentation framing:
|
||||
Version 4 added a routing envelope inside the transport fragmentation framing,
|
||||
and version 5 added the end-to-end protocol to it:
|
||||
|
||||
| field | bytes | encoding |
|
||||
|---|---:|---|
|
||||
| envelope version | 1 | `1` |
|
||||
| envelope version | 1 | `2` |
|
||||
| remaining hops | 1 | initially `16`; transit decrements, drops at `1` |
|
||||
| source peer | 32 | original endpoint public key |
|
||||
| destination peer | 32 | final endpoint public key |
|
||||
| flow id | 8 | big endian opaque stable hash supplied by the plugin |
|
||||
| protocol | 8 | big endian FNV-1a of the end-to-end protocol id (`wg`, `tcp-tls`, ...) |
|
||||
| payload | remaining | end-to-end encrypted plugin bytes |
|
||||
|
||||
The fixed header is 74 bytes. Zero/oversized hop limits, unknown versions,
|
||||
The fixed header is 82 bytes. Zero/oversized hop limits, unknown versions,
|
||||
short frames, oversized datagrams and unknown sources are rejected. A frame
|
||||
addressed here is delivered to its source's protocol inbox; transit goes straight
|
||||
to its next transport. The network and protocol are bound to the authenticated
|
||||
transport connection, not supplied by the frame. Intermediate nodes cannot
|
||||
decrypt or authenticate the inner WireGuard payload; the destination does that.
|
||||
Flow ids are routing hints, not authorization proofs.
|
||||
addressed here is delivered to the inbox of its source and protocol; transit
|
||||
goes straight to its next transport, whatever kind of transport that is. The
|
||||
network is bound to the authenticated transport connection, not supplied by
|
||||
the frame. The protocol names which end-to-end tunnel the payload belongs to
|
||||
and is independent of the links it crosses: a datagram of one protocol can
|
||||
travel over a link opened for another, and a middle member needs no protocol
|
||||
in common with either end. Intermediate nodes cannot decrypt or authenticate
|
||||
the inner payload; the destination's tunnel does that, so a frame naming the
|
||||
wrong protocol or a forged source is simply discarded there. Flow ids are
|
||||
routing hints, not authorization proofs.
|
||||
|
||||
Control ALPN 3 carries `Reachable { links: [{ peer, protocol }] }`. Each row
|
||||
belongs to the authenticated sender and is replaced atomically, expires after
|
||||
90 seconds, and is withdrawn on session closure. Only compatible authenticated
|
||||
members enter a protocol's graph; local edges always come from actual links.
|
||||
`Reachable { links: [{ peer, protocol }] }` lists the sender's direct links;
|
||||
`protocol` is the kind of transport the link is, for diagnostics, and routing
|
||||
ignores it. Each row belongs to the authenticated sender and is replaced
|
||||
atomically, expires after 90 seconds, and is withdrawn on session closure. Only
|
||||
authenticated members enter the graph; local edges always come from actual
|
||||
links.
|
||||
Announcements refresh on the existing control maintenance interval and are
|
||||
sent immediately on link changes. Topology stays volatile, outside signed state.
|
||||
|
||||
Version 5 also adds `hostname_since` to a signed hostname claim (signing domain
|
||||
`tsunagi-signed-record-v3`): a name belongs to whoever claimed it first. Stored
|
||||
records signed before are discarded on upgrade and re-published by their
|
||||
authors.
|
||||
|
||||
Upgrade both endpoints and intermediate peers together. Older control and data
|
||||
ALPNs cannot interoperate. The network secret, device identity, identity
|
||||
derivation, transcript encoding and stored network configuration are unchanged.
|
||||
@@ -216,11 +234,12 @@ not affect other networks.
|
||||
| message | meaning |
|
||||
|---|---|
|
||||
| `Announce { hostname, capabilities, broadcast }` | this agent's hostname, IP-plugin capabilities and local broadcast participation |
|
||||
| `Transport { protocol, payload }` | private, bounded per-peer transport negotiation; never state or a public message event |
|
||||
| `Ping { seq, payload }` | small request used to verify the exchange |
|
||||
| `Pong { seq, payload }` | the echoed reply |
|
||||
| `State { records }` | a snapshot of signed records, merged into what the receiver holds |
|
||||
| `Peers { peers }` | unverified member address hints, requiring their own handshake |
|
||||
| `Reachable { links }` | sender's current direct data links, scoped by protocol |
|
||||
| `Reachable { links }` | sender's current direct data links, by kind of transport |
|
||||
| `Bye { reason }` | graceful goodbye; not a revocation of anything |
|
||||
|
||||
`broadcast` is a per-network local opt-in, enabled by default and treated as
|
||||
|
||||
+47
-9
@@ -15,12 +15,27 @@ the default 1280-byte interface MTU also works on small QUIC paths.
|
||||
## Graph and lifecycle
|
||||
|
||||
Each authenticated control session advertises that peer's own direct data
|
||||
links, including the plugin protocol. Tables are separate for each network
|
||||
and protocol; only members offering a matching enabled protocol version enter
|
||||
the graph. A snapshot refreshes on the maintenance interval, expires after
|
||||
90 seconds, and is withdrawn when the session ends. Link arrival, closure and
|
||||
changed announcements rebuild routes immediately. Unchanged announcements
|
||||
refresh their age without republishing the forwarding snapshot.
|
||||
links. There is one table for each network, whatever protocols run over it: a
|
||||
link is a way to reach a neighbour, and a protocol is an end-to-end tunnel to a
|
||||
peer. The envelope names the protocol, so the two are independent. A neighbour
|
||||
reached over several kinds of transport (UDP, TCP, QUIC datagrams) is sent to
|
||||
over the best open one, in the order the protocols were configured, and over
|
||||
the next if that one fails or cannot carry the frame. Every authenticated
|
||||
member enters the graph; a member in the middle needs no protocol in common
|
||||
with either end, only a link to each. A snapshot refreshes on the maintenance
|
||||
interval, expires after 90 seconds, and is withdrawn when the session ends.
|
||||
Link arrival, closure and changed announcements rebuild routes immediately.
|
||||
Unchanged announcements refresh their age without republishing the forwarding
|
||||
snapshot.
|
||||
|
||||
Which protocol a pair of members runs their tunnel in is settled without
|
||||
asking: the member with the lower endpoint id decides, by the order it
|
||||
announces its protocols in, among those both have enabled at the same wire
|
||||
version. Both ends read the same answer from what they already hold. The
|
||||
tunnel is created as soon as the peer is reachable by any path, and it does not
|
||||
change when a direct link comes or goes underneath it: as soon as a direct
|
||||
link comes up it is shorter than the path through others, and traffic moves to
|
||||
it with nothing to redo. The user sees only a lower round-trip time.
|
||||
|
||||
Breadth-first search computes the shortest directed paths and every equal-cost
|
||||
first hop in sorted order. An actual direct link always wins. The first row is
|
||||
@@ -30,14 +45,29 @@ loops if different nodes have not yet received the same topology update.
|
||||
|
||||
The existing control plane still forms authenticated pairwise sessions.
|
||||
Routing provides multihop **data** paths among these members; it does not add
|
||||
control-plane flooding or carry control sessions through the data plane.
|
||||
control-plane flooding or carry control sessions through the data plane, so a
|
||||
member that has no control session with another has no data path to it either.
|
||||
Announcements are volatile, not durable membership or availability guarantees.
|
||||
|
||||
## What "online" means
|
||||
|
||||
One definition, computed by the agent and reported identically by `tsng status`
|
||||
and the tray: a member is online when this agent can reach it by any path — a
|
||||
live authenticated control session (QUIC keep-alive every 5 seconds, so a
|
||||
vanished peer is noticed within about 15), or a data path to it through other
|
||||
members. It is never inferred from ping, which a firewall may block, or from a
|
||||
tunnel's last handshake, which says only when traffic last flowed.
|
||||
|
||||
How it is reached is reported separately: `direct` over a named transport,
|
||||
`relayed` through a named member over N hops, or no data path yet while one is
|
||||
being set up. Being relayed is not being degraded in any way but distance, and
|
||||
is shown so it is not a surprise.
|
||||
|
||||
## Packet path
|
||||
|
||||
The control loop binds next hops to transport handles in an immutable table
|
||||
and publishes it through `ArcSwap`. Readers do not acquire the topology mutex.
|
||||
The transit routine validates the fixed 74-byte envelope, looks up the source
|
||||
The transit routine validates the fixed 82-byte envelope, looks up the source
|
||||
and destination, selects a cached next hop, decrements one byte, and calls the
|
||||
transport's synchronous datagram send. It never searches the graph, formats an
|
||||
endpoint string, validates an Ed25519 key, decrypts, or touches TUN.
|
||||
@@ -66,6 +96,14 @@ topology change can move a flow; datagrams remain unreliable and unordered,
|
||||
and there is no promise to preserve order across a physical path failure.
|
||||
Logical peer links and end-to-end encryption state survive these changes.
|
||||
|
||||
## Carrier health and switching
|
||||
|
||||
Carrier setup failures and persistent encrypted delivery loss advance the
|
||||
initiator's ordered fallback cycle. Logical end-to-end links survive while
|
||||
physical links are retired explicitly. The AWG carrier negotiates a fresh
|
||||
private pair profile on every reentry, without changing topology's durable
|
||||
state. See [amnezia.md](amnezia.md) for thresholds and bounds.
|
||||
|
||||
## Verification and measurement
|
||||
|
||||
The default offline suite exercises four real agents, iroh transports and
|
||||
@@ -86,5 +124,5 @@ counts sends without storing frames. Results are CPU forwarding cost, **not**
|
||||
end-to-end network latency or VPN throughput; encryption, fragmentation,
|
||||
sockets, congestion and scheduling contribute separately.
|
||||
|
||||
Wire compatibility: control ALPN `tsunagi/ctrl/4`, data ALPN `tsunagi/data/4`.
|
||||
Wire compatibility: control ALPN `tsunagi/ctrl/6`, data ALPN `tsunagi/data/5`.
|
||||
Upgrade every participant together; saved identities and network state persist.
|
||||
+19
-1
@@ -43,7 +43,9 @@ document trusted merely because the neighbour who forwarded it signed it.
|
||||
- Duplicates do not change the result.
|
||||
- Two conflicting signed records at the same version from the same author need
|
||||
explicit handling; they are not resolved by luck.
|
||||
- Neither arrival order nor system clocks decide a winner.
|
||||
- Neither arrival order nor the receiving replica's clock decides a winner. The
|
||||
one thing an author's own clock is used for is the time it first claimed a
|
||||
hostname, signed into its record (see below).
|
||||
- Compaction must not drop what is needed to stop revoked records being
|
||||
resurrected.
|
||||
|
||||
@@ -53,6 +55,22 @@ A hostname is a mutable binding to a persistent author, not an identity. A
|
||||
rename must be a signed record that revokes the specific old binding and
|
||||
announces the new one, ideally atomically in one record.
|
||||
|
||||
A hostname belongs to the member that claimed it **first**. Each claim carries
|
||||
`hostname_since`, the author's own time in milliseconds when it first claimed
|
||||
that exact name, signed with the rest of the record and kept across every later
|
||||
version that keeps the name; renaming starts it afresh. Replicas order claims
|
||||
by `(hostname_since, author id)`, a pure function of the records, so every
|
||||
replica names the same holder whatever order the records arrived in and there
|
||||
is no race to lose. A member that arrives later with a name already held keeps
|
||||
working by address and keeps its claim, but is not found by that name on any
|
||||
member; it is told so in its status and in `tsng status`, and finds the name
|
||||
again if the holder renames or releases it.
|
||||
|
||||
It is the author's word, so a member with a clock set far back can claim an
|
||||
older time. That can reorder who holds a *name* and nothing else — addresses,
|
||||
keys and everything the data plane trusts are decided without it — and anybody
|
||||
who knows the secret could take a name by other means anyway.
|
||||
|
||||
Turning a computer off is not a revocation of its hostname and does not remove
|
||||
the participant. Revocations are not dropped merely because they are old, and
|
||||
no acknowledgement from offline peers is required to keep working.
|
||||
|
||||
@@ -98,6 +98,13 @@ a batch does not stop the rest, and allocation is deterministic, spread out,
|
||||
walks past everything taken and reports a full range instead of handing out a
|
||||
duplicate.
|
||||
|
||||
`tests/range_conflicts.rs` checks isolated first members choosing the same
|
||||
range, persistence across restart, preservation of legacy ranges, exclusion
|
||||
of equal and nested ranges, late authenticated conflicts, switching all
|
||||
blockers, automatic resume, cancelling a suspended network's resume and
|
||||
concurrent switch requests. These use real agents, loopback handshakes and
|
||||
SQLite; they do not change host network settings.
|
||||
|
||||
`tests/local_control.rs` covers the local control socket end to end: a client
|
||||
asking a running agent for status over a real Unix socket, joining and
|
||||
leaving a network through it, a leftover socket file being replaced while a
|
||||
@@ -170,3 +177,10 @@ TSUNAGI_TEST_LOG=tsunagi=debug cargo test --test multi_peer -- --nocapture
|
||||
|
||||
The library never installs a global subscriber; the harness opts in only when
|
||||
that variable is set.
|
||||
|
||||
The `tsunagi-awg` suite runs real pair negotiation and masked UDP WireGuard
|
||||
engines with memory TUNs. A loss wrapper drops datagrams at the carrier boundary
|
||||
without substituting iroh, membership, encryption or SQLite. It checks runtime
|
||||
loss detection, fallback and fresh profiles, in addition to restart and input
|
||||
validation. Tests are offline; public STUN/DPI/NAT effectiveness needs separate
|
||||
real-network validation.
|
||||
+13
-2
@@ -26,8 +26,8 @@ Read this before relying on anything here. The protocol is in
|
||||
neither receive nor forge another member's traffic. See
|
||||
[wireguard.md](wireguard.md#address-ownership-is-enforced-not-announced).
|
||||
- **Tunnelled traffic is end-to-end encrypted by WireGuard**, independently of
|
||||
this crate. The transport underneath is also encrypted by iroh, but the
|
||||
tunnel's confidentiality does not depend on that.
|
||||
this crate. Carriers may be iroh, TLS, direct WG or
|
||||
pairwise AWG; confidentiality does not depend on a particular carrier.
|
||||
- **Resource bounds.** Frame lengths are validated before allocation; strings,
|
||||
lists, queues, concurrent dials and in-flight handshakes are all bounded;
|
||||
handshakes, dials and writes have timeouts.
|
||||
@@ -113,3 +113,14 @@ for key separation, HMAC-SHA256 for the membership proof, constant-time
|
||||
verification, iroh's Ed25519 endpoint keys and QUIC/TLS for the transport, and
|
||||
RFC 5705 TLS exporter output for channel binding. There is no custom encryption
|
||||
layer and no custom PAKE.
|
||||
|
||||
## Pairwise AWG masking
|
||||
|
||||
AWG profiles, PSKs and outer keys are private to one authenticated pair attempt,
|
||||
not shared network state. Address candidates are unverified until authenticated
|
||||
outer WireGuard data arrives. Header protection and structured signature packets
|
||||
reduce fixed wire markers; they do not guarantee admission by DPI or hide traffic
|
||||
analysis. iroh control availability remains a prerequisite. Encrypted end-to-end
|
||||
delivery probes detect sustained carrier loss, never confer membership and pass
|
||||
the same signed source/destination admission as ordinary IP traffic. See
|
||||
[amnezia.md](amnezia.md) for limits and tested scope.
|
||||
+33
-10
@@ -26,8 +26,8 @@ system can hand us IP packets, and even that is behind a trait
|
||||
| `ManagedTunFactory` | `CAP_NET_ADMIN` | traffic actually reaches the OS |
|
||||
|
||||
`ManagedTunFactory` creates the interface and configures it; see
|
||||
*Provisioning the interface* below and *Privileges* in
|
||||
[../README.md](../README.md#privileges). With no capability the agent runs
|
||||
*Provisioning the interface* below and Privileges in
|
||||
[../dist/linux/README.md](../dist/linux/README.md). With no capability the agent runs
|
||||
with `--no-tun`: everything but the last hop into the kernel still works.
|
||||
|
||||
[boringtun]: https://docs.rs/boringtun
|
||||
@@ -92,8 +92,8 @@ and then recorded as a signed fact**, using the model in
|
||||
[sync-model.md](sync-model.md).
|
||||
|
||||
```text
|
||||
default range 10.13.37.0/24 (override with --ipv4-range)
|
||||
who decides the first member to claim; later ones adopt what they find
|
||||
default range stable pseudorandom /24 in 10/8, derived from NetworkId
|
||||
who decides all members derive the default; signed explicit ranges win
|
||||
who signs the claiming member, with its persistent device key
|
||||
where it is kept state.sqlite, and every replica that has seen it
|
||||
what a return costs nothing: the old address is reclaimed
|
||||
@@ -120,11 +120,34 @@ the same address at the same moment are resolved by a rule both compute
|
||||
identically — the lower endpoint id keeps it — and the loser simply allocates
|
||||
again with a higher version.
|
||||
|
||||
**The range is agreed, not configured per member.** `--ipv4-range` says what
|
||||
this agent would use; a network that has already settled on something else
|
||||
wins, and the agent adopts it. So the flag matters for whoever starts the
|
||||
network and is harmless afterwards. Pass `--ipv4-range none` for an IPv6-only
|
||||
overlay.
|
||||
**The range is agreed, not configured per member.** New networks use the
|
||||
same deterministic `/24` on every device, independently of which network a
|
||||
member activates first or whether any peers are reachable. The derivation
|
||||
is unchanged from the former fallback; network identity and signed record
|
||||
encoding are unchanged. Distinct network ids can still derive the same range.
|
||||
|
||||
`--ipv4-range` proposes an explicit range; existing signed ranges take
|
||||
precedence over the new derived default, including the former `10.13.37.0/24`.
|
||||
Among disagreeing non-default ranges the lowest author id still wins. Updated
|
||||
members propagate their adopted range in their signed claims; older releases
|
||||
still apply their original selection rule until those claims converge.
|
||||
`--ipv4-range none` disables local proposals, as before.
|
||||
|
||||
An agent reserves each active network's range before starting its runtime.
|
||||
Overlapping ranges, including nested blocks such as a `/24` inside a `/16`,
|
||||
cannot run simultaneously. A late authenticated state update that introduces
|
||||
an overlap suspends that network before publishing new allocations to plugins
|
||||
or the interface. Other networks continue running.
|
||||
|
||||
Status includes every blocking network's name, id and range. `tsng network
|
||||
start <name-or-id>` and the tray's switch button stop all blockers before
|
||||
starting the selected network. Suspended networks keep their startup request
|
||||
and resume when the blocker stops or is forgotten. A manual stop cancels that
|
||||
request even for a network already suspended. The user's preference order is
|
||||
stored locally, so the same selection wins at the next startup; it is never
|
||||
advertised or treated as a remote network setting. Local IPC uses wire version
|
||||
19 for the new reports: restart the agent and tray together after upgrading.
|
||||
The peer control protocol and signed record encoding do not change.
|
||||
|
||||
The agent puts the address on the interface itself, as soon as the network has
|
||||
agreed on it — no restart, and the interface is not recreated, which would
|
||||
@@ -184,7 +207,7 @@ QUIC DATAGRAM frames themselves cannot fragment.
|
||||
Reassembly is bounded and expires incomplete packets; one lost fragment loses
|
||||
one packet, without blocking unrelated traffic. The logical data payload limit
|
||||
is 64 KiB, with the relay envelope subtracted before it reaches the plugin.
|
||||
The new framing uses data ALPN `tsunagi/data/4`; both ends and intermediate
|
||||
The new framing uses data ALPN `tsunagi/data/5`; both ends and intermediate
|
||||
peers need the updated binary. Network identities and saved state do not change.
|
||||
See [protocol.md](protocol.md#the-data-plane-protocol) for the wire format.
|
||||
|
||||
|
||||
Generated
+27
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"nodes": {
|
||||
"nixpkgs": {
|
||||
"locked": {
|
||||
"lastModified": 1790769526,
|
||||
"narHash": "sha256-5AIVFLRcx8YAwbFTVPTs7YhwJH7JyHfAQsvXmpovcXI=",
|
||||
"owner": "NixOS",
|
||||
"repo": "nixpkgs",
|
||||
"rev": "c9fe7d12cd78d1adcd12dd15e24432dde5b155a0",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
"owner": "NixOS",
|
||||
"ref": "nixpkgs-unstable",
|
||||
"repo": "nixpkgs",
|
||||
"type": "github"
|
||||
}
|
||||
},
|
||||
"root": {
|
||||
"inputs": {
|
||||
"nixpkgs": "nixpkgs"
|
||||
}
|
||||
}
|
||||
},
|
||||
"root": "root",
|
||||
"version": 7
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
{
|
||||
description = "tsunagi: serverless mesh VPN. A network name and a secret are all it takes.";
|
||||
|
||||
inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
|
||||
|
||||
outputs =
|
||||
{ self, nixpkgs }:
|
||||
let
|
||||
systems = [
|
||||
"x86_64-linux"
|
||||
"aarch64-linux"
|
||||
];
|
||||
forAllSystems = f: nixpkgs.lib.genAttrs systems (system: f nixpkgs.legacyPackages.${system});
|
||||
in
|
||||
{
|
||||
packages = forAllSystems (pkgs: rec {
|
||||
tsunagi = pkgs.callPackage ./nix/package.nix { };
|
||||
tsunagi-cli = pkgs.callPackage ./nix/package.nix { withTray = false; };
|
||||
default = tsunagi;
|
||||
});
|
||||
|
||||
overlays.default = final: _prev: {
|
||||
tsunagi = final.callPackage ./nix/package.nix { };
|
||||
tsunagi-cli = final.callPackage ./nix/package.nix { withTray = false; };
|
||||
};
|
||||
|
||||
nixosModules.default = import ./nix/module.nix self;
|
||||
|
||||
devShells = forAllSystems (pkgs: {
|
||||
default = pkgs.mkShell {
|
||||
inputsFrom = [ self.packages.${pkgs.stdenv.hostPlatform.system}.tsunagi ];
|
||||
packages = [
|
||||
pkgs.rustfmt
|
||||
pkgs.clippy
|
||||
];
|
||||
};
|
||||
});
|
||||
};
|
||||
}
|
||||
+132
@@ -0,0 +1,132 @@
|
||||
flake:
|
||||
{
|
||||
config,
|
||||
lib,
|
||||
pkgs,
|
||||
...
|
||||
}:
|
||||
|
||||
let
|
||||
cfg = config.services.tsunagi;
|
||||
socket = "/run/tsunagi/agent.sock";
|
||||
in
|
||||
{
|
||||
options.services.tsunagi = {
|
||||
enable = lib.mkEnableOption "the tsunagi mesh agent";
|
||||
|
||||
package = lib.mkOption {
|
||||
type = lib.types.package;
|
||||
default =
|
||||
if cfg.tray.enable then
|
||||
flake.packages.${pkgs.stdenv.hostPlatform.system}.tsunagi
|
||||
else
|
||||
flake.packages.${pkgs.stdenv.hostPlatform.system}.tsunagi-cli;
|
||||
defaultText = lib.literalExpression "tsunagi.packages.\${system}.tsunagi (tsunagi-cli without the tray)";
|
||||
description = "The tsunagi package to run.";
|
||||
};
|
||||
|
||||
tray.enable = lib.mkEnableOption "the tray client, started with every graphical session";
|
||||
|
||||
users = lib.mkOption {
|
||||
type = lib.types.listOf lib.types.str;
|
||||
default = [ ];
|
||||
example = [ "alice" ];
|
||||
description = ''
|
||||
Users allowed to talk to the agent without sudo (`tsng status`, `tsng join`,
|
||||
the tray). They are added to the `tsunagi` group.
|
||||
'';
|
||||
};
|
||||
|
||||
protocols = lib.mkOption {
|
||||
type = lib.types.nullOr (lib.types.listOf lib.types.str);
|
||||
default = null;
|
||||
example = [
|
||||
"wg-quic"
|
||||
"wg"
|
||||
];
|
||||
description = "Packet protocols for `tsng up --protocol`, best first. Null keeps the agent's default.";
|
||||
};
|
||||
};
|
||||
|
||||
config = lib.mkIf cfg.enable {
|
||||
users.groups.tsunagi = { };
|
||||
users.users = {
|
||||
tsunagi = {
|
||||
isSystemUser = true;
|
||||
group = "tsunagi";
|
||||
home = "/var/lib/tsunagi";
|
||||
description = "tsunagi agent";
|
||||
};
|
||||
}
|
||||
// lib.genAttrs cfg.users (_: {
|
||||
extraGroups = [ "tsunagi" ];
|
||||
});
|
||||
|
||||
systemd.services.tsunagi = {
|
||||
description = "tsunagi mesh agent";
|
||||
documentation = [ "https://github.com/house-of-vanity/tsunagi" ];
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
wants = [ "network-online.target" ];
|
||||
after = [ "network-online.target" ];
|
||||
environment = {
|
||||
TSUNAGI_STATE_DIR = "/var/lib/tsunagi";
|
||||
TSUNAGI_CACHE_DIR = "/var/cache/tsunagi";
|
||||
TSUNAGI_CONTROL_SOCKET = socket;
|
||||
TSUNAGI_CONTROL_GROUP = "tsunagi";
|
||||
};
|
||||
serviceConfig = {
|
||||
ExecStart = lib.concatStringsSep " " (
|
||||
[ "${cfg.package}/bin/tsng up" ]
|
||||
++ lib.optional (cfg.protocols != null) "--protocol ${lib.concatStringsSep "," cfg.protocols}"
|
||||
);
|
||||
User = "tsunagi";
|
||||
Group = "tsunagi";
|
||||
AmbientCapabilities = [
|
||||
"CAP_NET_ADMIN"
|
||||
"CAP_NET_BIND_SERVICE"
|
||||
];
|
||||
CapabilityBoundingSet = [
|
||||
"CAP_NET_ADMIN"
|
||||
"CAP_NET_BIND_SERVICE"
|
||||
];
|
||||
StateDirectory = "tsunagi";
|
||||
StateDirectoryMode = "0700";
|
||||
CacheDirectory = "tsunagi";
|
||||
RuntimeDirectory = "tsunagi";
|
||||
RuntimeDirectoryMode = "0755";
|
||||
Restart = "on-failure";
|
||||
RestartSec = 5;
|
||||
};
|
||||
};
|
||||
|
||||
# Exit nodes: the agent looks for iptables only in fixed FHS directories.
|
||||
systemd.tmpfiles.rules = [
|
||||
"L+ /usr/bin/iptables - - - - ${pkgs.iptables}/bin/iptables"
|
||||
];
|
||||
|
||||
# Names of mesh members are served locally and handed to the system resolver.
|
||||
services.resolved.enable = lib.mkDefault true;
|
||||
|
||||
# polkit decides by group, not capability, who may configure systemd-resolved.
|
||||
security.polkit.extraConfig = ''
|
||||
polkit.addRule(function(action, subject) {
|
||||
var allowed = [
|
||||
"org.freedesktop.resolve1.set-dns-servers",
|
||||
"org.freedesktop.resolve1.set-domains",
|
||||
"org.freedesktop.resolve1.set-default-route",
|
||||
"org.freedesktop.resolve1.revert"
|
||||
];
|
||||
if (allowed.indexOf(action.id) >= 0 && subject.isInGroup("tsunagi")) {
|
||||
return polkit.Result.YES;
|
||||
}
|
||||
});
|
||||
'';
|
||||
|
||||
environment.systemPackages = [ cfg.package ];
|
||||
environment.sessionVariables.TSUNAGI_CONTROL_SOCKET = socket;
|
||||
|
||||
environment.etc."xdg/autostart/tsunagi-tray.desktop" = lib.mkIf cfg.tray.enable {
|
||||
source = "${cfg.package}/share/applications/tsunagi-tray.desktop";
|
||||
};
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,94 @@
|
||||
{
|
||||
withTray ? true,
|
||||
lib,
|
||||
rustPlatform,
|
||||
pkg-config,
|
||||
copyDesktopItems,
|
||||
wrapGAppsHook3,
|
||||
gtk3,
|
||||
libayatana-appindicator,
|
||||
libglvnd,
|
||||
libxkbcommon,
|
||||
vulkan-loader,
|
||||
wayland,
|
||||
xdotool,
|
||||
libx11,
|
||||
libxcursor,
|
||||
libxi,
|
||||
libxrandr,
|
||||
}:
|
||||
|
||||
let
|
||||
# The tray is an egui app that loads graphics, windowing and appindicator
|
||||
# libraries at run time, so they have to be put on its library path by hand.
|
||||
trayRuntimeLibs = [
|
||||
libayatana-appindicator
|
||||
libglvnd
|
||||
libxkbcommon
|
||||
vulkan-loader
|
||||
wayland
|
||||
libx11
|
||||
libxcursor
|
||||
libxi
|
||||
libxrandr
|
||||
];
|
||||
in
|
||||
rustPlatform.buildRustPackage {
|
||||
pname = if withTray then "tsunagi" else "tsunagi-cli";
|
||||
version = "0.1.0";
|
||||
|
||||
src = lib.cleanSource ../.;
|
||||
cargoLock.lockFile = ../Cargo.lock;
|
||||
|
||||
cargoBuildFlags = [
|
||||
"--bin"
|
||||
"tsng"
|
||||
]
|
||||
++ lib.optionals withTray [
|
||||
"--bin"
|
||||
"tsunagi-tray"
|
||||
];
|
||||
|
||||
# The suite opens real sockets and interfaces; it runs in CI instead.
|
||||
doCheck = false;
|
||||
|
||||
nativeBuildInputs = [ pkg-config ] ++ lib.optionals withTray [
|
||||
copyDesktopItems
|
||||
wrapGAppsHook3
|
||||
];
|
||||
buildInputs = lib.optionals withTray [
|
||||
gtk3
|
||||
libayatana-appindicator
|
||||
xdotool
|
||||
];
|
||||
|
||||
dontWrapGApps = true;
|
||||
|
||||
postInstall = ''
|
||||
install -Dm644 dist/linux/50-tsunagi-resolved.rules \
|
||||
"$out/share/doc/tsunagi/50-tsunagi-resolved.rules"
|
||||
''
|
||||
+ lib.optionalString withTray ''
|
||||
install -Dm644 dist/linux/tsunagi.svg \
|
||||
"$out/share/icons/hicolor/scalable/apps/tsunagi.svg"
|
||||
install -Dm644 dist/linux/tsunagi-tray.desktop \
|
||||
"$out/share/applications/tsunagi-tray.desktop"
|
||||
'';
|
||||
|
||||
postFixup = lib.optionalString withTray ''
|
||||
wrapProgram "$out/bin/tsunagi-tray" \
|
||||
"''${gappsWrapperArgs[@]}" \
|
||||
--prefix LD_LIBRARY_PATH : ${lib.makeLibraryPath trayRuntimeLibs}:/run/opengl-driver/lib
|
||||
'';
|
||||
|
||||
meta = {
|
||||
description = "Serverless mesh VPN: join a network with a name and a secret";
|
||||
homepage = "https://github.com/house-of-vanity/tsunagi";
|
||||
license = lib.licenses.wtfpl;
|
||||
mainProgram = "tsng";
|
||||
platforms = [
|
||||
"x86_64-linux"
|
||||
"aarch64-linux"
|
||||
];
|
||||
};
|
||||
}
|
||||
Reference in new issue
Block a user