Compare commits
17
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
8a6216bccb | ||
|
|
2e7e238459 | ||
|
|
23af1a2d9d | ||
|
|
94318e7c16 | ||
|
|
2b0c938487 | ||
|
|
4e235cbac6 | ||
|
|
2631eb6975 | ||
|
|
90ef5cefec | ||
|
|
13bdcaf98d | ||
|
|
d941cdac7a | ||
|
|
e701986a39 | ||
|
|
ed28e86748 | ||
|
|
781e359775 | ||
|
|
d5e02ec256 | ||
|
|
97377e3703 | ||
|
|
c184ef0914 | ||
|
|
e6ec62bcba |
No files matched your search
@@ -1 +1,2 @@
|
||||
/target
|
||||
/.claude
|
||||
@@ -8,7 +8,7 @@ before touching the code. Do not restate the full design here; follow the links.
|
||||
A proof-of-concept agent library for small private mesh networks. A user
|
||||
supplies a network name and one shared secret; agents derive the same network
|
||||
space independently, find each other, prove membership and exchange control
|
||||
messages. Scope and non-scope are in [README.md](README.md).
|
||||
messages. What it does and does not do is in [docs/architecture.md](docs/architecture.md) and [docs/threat-model.md](docs/threat-model.md).
|
||||
|
||||
There is no central server, no owner, no registration and no majority vote.
|
||||
Design accordingly: a majority is not a root of trust.
|
||||
@@ -59,8 +59,10 @@ Keep these separate. Crossing them is the main thing to review for.
|
||||
other peers goes link in, link out: it is never written to the middle's
|
||||
interface or decrypted there. Multihop routing has a bounded hop limit;
|
||||
equal-cost next hops are chosen per flow. Reachability is first-hand and
|
||||
volatile: each authenticated member advertises its own protocol-specific
|
||||
links. Build routing tables on topology changes, never per packet. Transit
|
||||
volatile: each authenticated member advertises its own direct links, of
|
||||
whatever kind. A link is a way to reach a neighbour and a protocol is an
|
||||
end-to-end tunnel to a peer; the two meet only in the envelope, so a
|
||||
tunnel never depends on which transports carry it. Build routing tables on topology changes, never per packet. Transit
|
||||
must not acquire a routing mutex or wait for a protocol/TUN reader.
|
||||
- **Broadcast fanout belongs at local IP ingress.** Participation is local to
|
||||
each network, enabled by default, persisted and advertised to authenticated
|
||||
|
||||
Generated
+22
-2
@@ -2806,8 +2806,6 @@ checksum = "791930b43c0d5973160d90a8f3894509f2b273430f5c5c73b668636d0287c5c0"
|
||||
[[package]]
|
||||
name = "iroh"
|
||||
version = "1.2.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "b2f8d1cfffc83efe39a1031aab423ce09cb8048071baba550508931e9a81ce46"
|
||||
dependencies = [
|
||||
"backon",
|
||||
"blake3",
|
||||
@@ -6241,11 +6239,13 @@ dependencies = [
|
||||
"iroh",
|
||||
"netwatch",
|
||||
"nix 0.31.3",
|
||||
"serde_json",
|
||||
"simple-dns",
|
||||
"tempfile",
|
||||
"tokio",
|
||||
"tracing",
|
||||
"tracing-subscriber",
|
||||
"tsunagi-awg",
|
||||
"tsunagi-core",
|
||||
"tsunagi-tcp-tls",
|
||||
"tsunagi-wg-quic",
|
||||
@@ -6253,6 +6253,26 @@ dependencies = [
|
||||
"windows-service",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "tsunagi-awg"
|
||||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"boringtun",
|
||||
"bytes",
|
||||
"chacha20 0.9.1",
|
||||
"iroh",
|
||||
"postcard",
|
||||
"rand 0.10.3",
|
||||
"serde",
|
||||
"tempfile",
|
||||
"tokio",
|
||||
"tracing",
|
||||
"tsunagi-core",
|
||||
"tsunagi-wg-quic",
|
||||
"tsunagi-wireguard",
|
||||
"zeroize",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "tsunagi-core"
|
||||
version = "0.1.0"
|
||||
|
||||
+3
-1
@@ -4,12 +4,14 @@
|
||||
[workspace]
|
||||
resolver = "3"
|
||||
members = ["crates/*"]
|
||||
exclude = ["vendor/mainline"]
|
||||
exclude = ["vendor/mainline", "vendor/iroh"]
|
||||
|
||||
# Mainline 8.0.0 treats Windows UDP read timeouts as socket failures.
|
||||
# Keep the small receive-loop correction reproducible until an upstream fix.
|
||||
[patch.crates-io]
|
||||
mainline = { path = "vendor/mainline" }
|
||||
# Backport bounded path retries until an iroh release includes the OOM fix.
|
||||
iroh = { path = "vendor/iroh" }
|
||||
|
||||
[workspace.package]
|
||||
edition = "2024"
|
||||
|
||||
@@ -1,766 +1,253 @@
|
||||
# tsunagi
|
||||
|
||||
A proof-of-concept Rust library for **small private mesh networks** — a handful
|
||||
of friends, home machines, a few servers. Units to dozens of participants, not
|
||||
thousands.
|
||||
|
||||
The end user configures exactly two things:
|
||||
|
||||
```text
|
||||
network_name
|
||||
secret # one shared secret; "password" and "secret" mean the same value
|
||||
```
|
||||
|
||||
From those, every agent independently derives the same network space. There is
|
||||
no central server, no network owner with special powers, no registration and no
|
||||
majority vote. Anyone who knows the parameters can join; nobody has to trust
|
||||
anybody else.
|
||||
|
||||
## What this proof of concept actually does
|
||||
|
||||
A working library with **real iroh connections** and integration tests:
|
||||
|
||||
- persistent device identity stored in SQLite, stable across restarts;
|
||||
- several independent networks at once in one agent;
|
||||
- deterministic network identity derived from name + secret;
|
||||
- candidates supplied by a replaceable discovery component;
|
||||
- automatic first contact through the public Mainline DHT (BEP44);
|
||||
- real iroh connections plus an explicit mutual proof of network membership;
|
||||
- a small versioned control protocol: handshake, hostname/capability
|
||||
announcement, ping/pong;
|
||||
- automatic reconnect with bounded exponential backoff and jitter;
|
||||
- status snapshots, an event stream and honest diagnostics;
|
||||
- configuration restored after a restart;
|
||||
- correct behaviour when the disposable cache is missing or corrupt;
|
||||
- a **WireGuard data plane**, in userspace: its own key per network, an IPv6
|
||||
overlay with deterministically derived addresses and optional IPv4, real
|
||||
tunnels carried over iroh, and address ownership enforced rather than
|
||||
believed;
|
||||
- a **command line agent**, `tsng`, with a local control socket.
|
||||
|
||||
### What it deliberately does **not** do
|
||||
|
||||
Not implemented, and not pretended to be: DNS, routing through
|
||||
intermediate participants, a full CRDT, dynamically loaded plugins, a system
|
||||
service, a complete CLI, or a local control socket. Snapshot synchronisation
|
||||
and signed revocations are designed for but not implemented — see
|
||||
[docs/sync-model.md](docs/sync-model.md). The WireGuard plugin's own limits,
|
||||
including that its system backend is Linux-only, are in
|
||||
[docs/wireguard.md](docs/wireguard.md#limits-and-future-work).
|
||||
|
||||
**Control and data are separated logically, not physically.** Both ride on
|
||||
iroh, on different ALPNs and different connections, so the data plane inherits
|
||||
iroh's hole punching and relay fallback instead of reimplementing them — while
|
||||
the control protocol still knows nothing about packets and can keep a different
|
||||
transport underneath it later. Filtering user traffic remains the operating
|
||||
system's and the user's responsibility, not this library's.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Rust 1.91 or newer (iroh 1.2 requires it), edition 2024. Pinned dependencies
|
||||
in `Cargo.lock`.
|
||||
- No internet, no public DHT, no public relay, no administrator rights and no changes
|
||||
to OS network settings are needed to build or test.
|
||||
- WireGuard runs in userspace (boringtun): **no kernel module and no `wg`
|
||||
tool**. Only creating a real network interface needs `CAP_NET_ADMIN`, and
|
||||
`--no-tun` skips even that.
|
||||
|
||||
## Trying it on two machines
|
||||
|
||||
On the first machine:
|
||||
|
||||
```bash
|
||||
cargo build --release
|
||||
./target/release/tsng status # this device, the agent, and this host
|
||||
./target/release/tsng network # the networks this device belongs to
|
||||
```
|
||||
|
||||
Two commands do the work, and they are separate on purpose: `up` runs the
|
||||
agent — one per device, for as long as you want it — and `join` decides
|
||||
what it belongs to, at any time, while it runs.
|
||||
|
||||
```bash
|
||||
# Terminal one: the agent. It prints its endpoint id and then serves.
|
||||
./target/release/tsng up
|
||||
|
||||
# Terminal two: make a network. With no --secret it invents one and prints
|
||||
# it, along with the line to send the other machine.
|
||||
./target/release/tsng join --network lab
|
||||
```
|
||||
|
||||
On the second machine, start its agent and join the same network with the
|
||||
secret that was printed:
|
||||
|
||||
```bash
|
||||
./target/release/tsng up
|
||||
./target/release/tsng join --network lab --secret tsn1...
|
||||
```
|
||||
|
||||
**Name and secret are enough for the first meeting.** Mainline DHT discovery
|
||||
is enabled by default. Each active network publishes this device's endpoint
|
||||
and searches until the first authenticated connection. While connected it
|
||||
only republishes; after 60 seconds without any authenticated connection it
|
||||
searches again. Publication is approximately every five minutes, and records
|
||||
older than 15 minutes are ignored. DHT storage nodes may retain them longer.
|
||||
|
||||
Use `--no-dht` to disable DHT, or `--peer <endpoint-id>` for optional manual
|
||||
bootstrap. `--reach local` disables public DHT regardless of the DHT flag.
|
||||
Endpoint records are public, signed, and contain no network secret. Discovery
|
||||
is not authentication: every connection must still prove network membership.
|
||||
Details and failure behavior are in [docs/mainline-dht.md](docs/mainline-dht.md).
|
||||
|
||||
**One introduction is enough for the whole network.** Members tell each
|
||||
other about the members they know, and every author of a signed record is
|
||||
somebody to try, so a device pointed at one member ends up talking to all
|
||||
of them rather than to the one that happened to be on its command line.
|
||||
What travels is a candidate — an address somebody has seen — and it is
|
||||
authenticated by the handshake like any other; being introduced grants
|
||||
nothing. It is deliberately the same shape a lookup in a distributed hash
|
||||
table would return, so that is a source to add beside this one rather than
|
||||
a redesign.
|
||||
|
||||
An agent with nobody to contact says so in `status` rather than sitting
|
||||
there looking patient.
|
||||
|
||||
Within a few seconds both print something like:
|
||||
|
||||
```text
|
||||
+ peer b47c958462 connected over direct rtt=Some(4.5ms)
|
||||
+ data link to b47c958462 for wg-quic: direct via 192.0.2.7:41234, datagram 1382
|
||||
|
||||
--- status ---
|
||||
control: 1 peer(s), 0 dial failure(s), 0 handshake failure(s)
|
||||
wg-quic: tsun0 on 10.13.37.69/24 mtu 1280, 1/1 tunnel(s) established
|
||||
4jO4kx9Z 10.13.37.237 handshake 3s ago tx=0 rx=0 dropped=0 path=direct via 192.0.2.7:41234
|
||||
```
|
||||
|
||||
`1/1 tunnel(s) established` means a real WireGuard handshake completed.
|
||||
|
||||
## Checking that it works
|
||||
|
||||
From another shell on either machine:
|
||||
|
||||
```bash
|
||||
tsng status
|
||||
```
|
||||
|
||||
```text
|
||||
endpoint 7d76ccbbc21bf30767e14422c0494740a2cecc02aa9f82d5b8d57bdae350e7fc
|
||||
hostname tsunagi-7d76ccbbc2
|
||||
bound 0.0.0.0:41641
|
||||
|
||||
network lab (z2o4qwrvnj3zb6st2aoqg4abf342j662q2ujttqrsmz22argk2ba) active
|
||||
peer b345d5271b tsunagi-b345d5271b Direct rtt 24ms
|
||||
overlay tsunz2o4qwrvnj3 fd09:…:c1c6/64 and 100.110.49.177 mtu 1280 1/1 tunnel(s) up
|
||||
SDsEb/WF fd09:…:c4b / 100.65.243.53 handshake 4s ago tx 0 rx 0 Direct via Ip(…)
|
||||
```
|
||||
|
||||
`1/1 tunnel(s) up` and a recent handshake mean the tunnel is live. Then send
|
||||
real traffic to the peer's overlay address:
|
||||
|
||||
```bash
|
||||
ping6 fd09:…:c4b # or
|
||||
ping 100.65.243.53
|
||||
```
|
||||
|
||||
`tx` and `rx` in the status should start moving.
|
||||
|
||||
**Addresses are allocated and then remembered.** The default range is
|
||||
`10.13.37.0/24`; the first member to join settles it and later members adopt
|
||||
what they find, so `--ipv4-range` only matters for whoever starts the network:
|
||||
|
||||
```bash
|
||||
tsng up --ipv4-range 10.44.0.0/16
|
||||
tsng up --ipv4-range none # no data plane at all
|
||||
```
|
||||
|
||||
One agent has one interface, so two of its networks cannot both use that
|
||||
range. The second takes the range **derived from its own network id**: not
|
||||
picked locally — every member derives the same one from something they all
|
||||
already have — so it is an agreement rather than a guess, and a device in
|
||||
several networks gets an address in each.
|
||||
|
||||
An address is claimed with a record signed by that member's persistent device
|
||||
key, stored, and merged between every replica. A member that disappears for a
|
||||
month comes back to the same address, because the claim outlived the session.
|
||||
No vote is involved — see
|
||||
[docs/wireguard.md](docs/wireguard.md#ipv4-allocated-signed-and-kept) and
|
||||
[docs/sync-model.md](docs/sync-model.md).
|
||||
|
||||
Because the address is allocated at run time rather than derived, it is not
|
||||
known until the agent has started and agreed with its peers. The agent then
|
||||
assigns it to the interface itself.
|
||||
|
||||
## Levels
|
||||
|
||||
The command line is split the way the design is. A bare `tsng up` needs
|
||||
only a network name and a secret; everything else sits under the level it
|
||||
belongs to, which `--help` shows as two sections:
|
||||
|
||||
* **System** — what the agent itself does: how it reaches peers (`--reach`),
|
||||
the one overlay interface it owns (`--interface`, `--mtu`, `--no-tun`), the
|
||||
address range (`--ipv4-range`), and the local resolver (`--no-dns` to disable).
|
||||
* **Transport** — which protocols carry packets (`--protocol`, a list) and
|
||||
their own settings (`-o key=value`, or `-o protocol:key=value`).
|
||||
|
||||
```bash
|
||||
tsng protocols # what this build can carry packets with
|
||||
```
|
||||
|
||||
```text
|
||||
wg-quic (wire version 4)
|
||||
what WireGuard's cryptography carried in iroh's QUIC datagrams
|
||||
-o keepalive=SECONDS keeps a tunnel and its link warm through a NAT
|
||||
-o mtu=BYTES largest packet a tunnel will carry, at least 576
|
||||
```
|
||||
|
||||
Each protocol declares its own settings, so the agent can list them without
|
||||
knowing anything about the protocol, and a setting no selected protocol takes
|
||||
is refused rather than ignored. `--protocol none` runs the control plane by
|
||||
itself.
|
||||
|
||||
A pair of peers uses a protocol they both have **at the same wire version**.
|
||||
That is not the software version: two peers on different builds carry traffic
|
||||
for each other for as long as the bytes between them have not changed. A peer
|
||||
with nothing in common keeps its control plane — messages and signed state
|
||||
still flow — and simply has no data plane, which `tsng status` shows as a
|
||||
session with no agreed protocol.
|
||||
|
||||
## Names
|
||||
|
||||
DNS is enabled by default and serves a local zone for **every network this device is in**,
|
||||
each named after the network, so members can be reached by name instead of
|
||||
by address:
|
||||
|
||||
```bash
|
||||
tsng up # DNS is already enabled
|
||||
dig @127.0.0.1 -p 5354 music.lab
|
||||
```
|
||||
|
||||
An explicit choice is remembered across restarts. `--no-dns` or `tsng dns off`
|
||||
disables it; `--dns` or `tsng dns on` enables it again. Existing saved opt-outs
|
||||
remain respected. The `dns on` and `dns off` commands also take effect immediately
|
||||
on a running agent. `--dns` and `--no-dns` cannot be combined.
|
||||
|
||||
The zone of a network is its name — `--dns-zone` is gone, because with
|
||||
several networks there is no single zone to name. A network name may contain
|
||||
dots, so `--network lab.internal` is how you get `music.lab.internal`.
|
||||
|
||||
Names come from signed state, which is the point: **a member that is
|
||||
switched off still resolves**, because its claim outlived the session.
|
||||
|
||||
The answers are IPv4 addresses, because that is what the overlay is. The
|
||||
*questions* are taken on `127.0.0.1` and `[::1]`, over UDP and TCP, so a
|
||||
resolver reaches it over whichever family it uses; both are published to the
|
||||
system resolver together. Loopback and nothing else: the zones are a view
|
||||
for the host running the agent, and binding an overlay address would put
|
||||
them in front of the whole mesh — an agent in two networks would then answer
|
||||
one network's questions about the other's names.
|
||||
|
||||
The zone is the network name, and it is yours to choose. A name that
|
||||
shadows a real public domain is reported and then used: a network called
|
||||
`ru` warns that every public `.ru` name becomes unreachable from this host,
|
||||
and then does it. Anything that collides with nothing — which is most names
|
||||
— is said nothing about, because a warning that fires on every private name
|
||||
anybody picks is how people learn to ignore warnings.
|
||||
|
||||
On Linux the agent tells systemd-resolved to send questions for that suffix
|
||||
here, over D-Bus, scoped to the overlay interface and as a *routing* domain
|
||||
so it never becomes the resolver for anything else. resolved drops the whole
|
||||
setting when the interface goes, and the interface goes with the agent.
|
||||
|
||||
Only an interface the agent created, though. Under `--no-tun` there is no
|
||||
host interface at all, and the agent says so and serves the zone on
|
||||
loopback rather than configuring whatever else on the host happens to share
|
||||
the name.
|
||||
|
||||
That last step needs permission that `CAP_NET_ADMIN` does not give:
|
||||
systemd-resolved asks polkit, and polkit decides by **user**, not by
|
||||
capability, so there is no way for the agent to arrange it from inside. On
|
||||
a desktop the refusal reads `Interactive authentication required`.
|
||||
|
||||
Running as a system service is enough. Otherwise the agent prints the rule
|
||||
that grants it — the four actions it calls and nothing else — ready to
|
||||
paste:
|
||||
|
||||
```bash
|
||||
sudo tee /etc/polkit-1/rules.d/50-tsunagi-resolved.rules > /dev/null <<'RULE'
|
||||
polkit.addRule(function(action, subject) {
|
||||
var allowed = [
|
||||
"org.freedesktop.resolve1.set-dns-servers",
|
||||
"org.freedesktop.resolve1.set-domains",
|
||||
"org.freedesktop.resolve1.set-default-route",
|
||||
"org.freedesktop.resolve1.revert"
|
||||
];
|
||||
if (allowed.indexOf(action.id) >= 0 && subject.user == "YOUR-USER") {
|
||||
return polkit.Result.YES;
|
||||
}
|
||||
});
|
||||
RULE
|
||||
```
|
||||
|
||||
**Without it the server still runs** — `tsng status` prints where it is
|
||||
listening and the exact `dig` line — so the automatic part is missing, not
|
||||
the feature. The refusal is said once rather than on every pass, and retried
|
||||
slowly, because nothing but a person will change it.
|
||||
|
||||
The server is authoritative for its zone and nothing else. No recursion, no
|
||||
forwarding, no cache: pointing a resolver at it can never make it a route to
|
||||
the outside.
|
||||
|
||||
## Privileges
|
||||
|
||||
On Linux the agent **manages its own overlay interface**. It creates the TUN
|
||||
interface, sets the MTU, brings it up and assigns both overlay addresses, all
|
||||
over netlink in process — no `ip` invocation, no shell, nothing that a
|
||||
remote peer could influence.
|
||||
|
||||
That needs `CAP_NET_ADMIN`, granted once:
|
||||
|
||||
```bash
|
||||
sudo setcap cap_net_admin+p /usr/local/bin/tsng
|
||||
```
|
||||
|
||||
`+p` rather than `+ep`: the capability is then *permitted* but not
|
||||
*effective*, and the agent raises it only around the handful of netlink calls
|
||||
that need it — a few milliseconds at startup, and again if its address
|
||||
allocation changes. Everything else, including every byte from the network,
|
||||
is handled with it lowered. `+ep` works too; the agent lowers it on the way
|
||||
in.
|
||||
|
||||
`tsng status` says which of these applies on the host it runs on, along
|
||||
with what the agent is doing. It grades each finding: **ok** for what works,
|
||||
**warn** for what the agent runs without and you can fix from the line it
|
||||
prints, **FAIL** for what it cannot work around. The words carry the grade as
|
||||
well as the colour, so the report reads the same piped to a file or on a
|
||||
terminal without colour, and it honours `NO_COLOR`.
|
||||
|
||||
Members are listed online first, then the ones that are away. A member that
|
||||
is away is reported plainly rather than flagged: in a mesh of laptops it is
|
||||
the ordinary condition, not a fault. The signed state is what makes that
|
||||
sayable — it remembers who belongs while they are gone, so the report can say
|
||||
"offline, 10.13.37.99 still reserved for it" instead of leaving a
|
||||
dial-failure counter to imply it. Counters are history and are never graded:
|
||||
a peer that left and came back should not leave the report looking broken.
|
||||
|
||||
The other two commands split along the line the system itself draws. `id` is
|
||||
this **device**: the key it signs with and the name it answers to. `network`
|
||||
is what it **belongs to**: which networks, their secrets, joining and
|
||||
leaving. A device outlives every network it is in, and a network outlives
|
||||
any device in it, so a command that mixed them had to be read twice.
|
||||
|
||||
Every item takes the same shape, so there is nothing to remember: name it to
|
||||
see it, name it with a value to change it.
|
||||
|
||||
```
|
||||
tsng id everything about this device
|
||||
tsng id hostname the name it answers to
|
||||
tsng id hostname mango change it
|
||||
tsng id key the key it signs with
|
||||
tsng id key rotate replace that key
|
||||
|
||||
tsng network the networks this device belongs to
|
||||
tsng network join -n lab -s tsn1… join one; adds it to a running agent
|
||||
tsng network join -n lab resume one this device has, or make it on the spot
|
||||
tsng network stop lab stop serving it, keeping everything
|
||||
tsng network start lab serve it again, from where it left off
|
||||
tsng network leave lab give up the address and name, then forget it
|
||||
tsng network secret the secret of each joined network
|
||||
tsng network secret lab just that one, for copying
|
||||
tsng network secret generate a fresh secret for a network that does not exist yet
|
||||
|
||||
tsng dns whether the local resolver is serving, and what
|
||||
tsng dns on start it, now and after every restart
|
||||
tsng dns off stop it, now and after every restart
|
||||
```
|
||||
|
||||
**A network without a secret makes one.** `tsng join --network lab`
|
||||
resolves a bare name in the obvious way: if this device is already in
|
||||
exactly one network called `lab`, that one — so the name alone resumes what
|
||||
you have; if it is in none, a fresh random secret, printed in full with the
|
||||
line to send the other machine:
|
||||
|
||||
```
|
||||
joined `lab` (k2on43wadbi5x267vp6z3ogkm7nbjedfdoxyauxhtxhwqrprylba)
|
||||
<p align="center">
|
||||
<img src="dist/linux/tsunagi.svg" width="96" alt="">
|
||||
</p>
|
||||
|
||||
<h1 align="center">tsunagi</h1>
|
||||
|
||||
<p align="center">
|
||||
<b>A mesh VPN with no server.</b><br>
|
||||
Pick a network name, share a secret, and your machines are on one private network.
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="https://github.com/house-of-vanity/tsunagi/releases/latest"><img alt="release" src="https://img.shields.io/github/v/release/house-of-vanity/tsunagi?include_prereleases"></a>
|
||||
<a href="https://github.com/house-of-vanity/tsunagi/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/house-of-vanity/tsunagi/actions/workflows/ci.yml/badge.svg"></a>
|
||||
<a href="LICENSE"><img alt="license" src="https://img.shields.io/badge/license-WTFPL-blue"></a>
|
||||
</p>
|
||||
|
||||
```console
|
||||
laptop$ tsng join -n home
|
||||
joined `home`
|
||||
secret tsn1u7c…
|
||||
|
||||
Run this on the other machine:
|
||||
server$ tsng join -n home -s tsn1u7c…
|
||||
server$ tsng id hostname server
|
||||
|
||||
tsng join --network lab --secret tsn1u7c…
|
||||
|
||||
Its agent has to be running. If it is not:
|
||||
|
||||
tsng up --peer 91e83a6e2b7a…
|
||||
laptop$ ssh server.home
|
||||
```
|
||||
|
||||
Two networks of one name is a thing that happens — a mistyped secret makes
|
||||
one — so joining says whether the network was already here, and warns when
|
||||
another configured network answers to the same name. A name is a label and
|
||||
the id is the identity.
|
||||
No account, no coordination server, no admin console, no keys to approve.
|
||||
Anyone who knows the **name** and the **secret** is in. Nobody else can find
|
||||
the network, let alone enter it.
|
||||
|
||||
That is the ad-hoc case: one person makes a network and sends the command
|
||||
round. The secret is printed *only* when the agent invented it — there is
|
||||
nowhere else to read it from — and never when it was supplied, because then
|
||||
it is already yours. Two networks of one name and no secret is the one case
|
||||
with no answer, and it says so instead of choosing.
|
||||
## Why
|
||||
|
||||
A secret is printed by `network secret` and nowhere else — not by `id`, not
|
||||
by `status`, not in a log, a `Debug` rendering or anything sent to a peer.
|
||||
Asking for it is deliberate, because these reports get pasted into chats.
|
||||
Tailscale, Headscale and ZeroTier are great, but each needs a control server:
|
||||
theirs, or one you have to run and keep alive. tsunagi doesn't. Machines find
|
||||
each other through the public BitTorrent DHT, connect directly (hole punching,
|
||||
with relay fallback), and keep the network state among themselves.
|
||||
|
||||
**`up` takes no network at all** if you would rather decide later: it
|
||||
brings up the agent and whatever it is already configured for, and waits.
|
||||
That is the shape of a daemon in one terminal and `tsng network join`
|
||||
in another.
|
||||
| | Tailscale | Headscale | ZeroTier | **tsunagi** |
|
||||
| ---------------------- | ------------------ | ---------------- | ------------------- | ------------------- |
|
||||
| Control server | hosted by vendor | you run it | hosted or self-run | **none** |
|
||||
| Account / sign-in | yes | yes | yes | **no** |
|
||||
| To join a network | sign in, get approved | pre-auth key | network ID, approval | **name + secret** |
|
||||
| Direct connections | yes | yes | yes | yes |
|
||||
| Exit nodes | yes | yes | via routes | yes |
|
||||
|
||||
`network join` is also the answer to a question `up` cannot: a state
|
||||
directory belongs to one live agent, so a second `tsng up` cannot add a
|
||||
network to the one already running. This adds it over the control socket and
|
||||
it starts at once. With no agent running it is written to the configuration
|
||||
and starts with the next `up`.
|
||||
## Features
|
||||
|
||||
The name is part of the signed state, so changing it revokes the previous
|
||||
one: there is one record per author, a new version replaces the whole claim,
|
||||
and no replica can keep the old name standing. Changing it while the agent
|
||||
runs goes through the agent, which republishes and tells its peers straight
|
||||
away.
|
||||
- **Two things to know:** a network name and a secret. A secret is generated for you when you make a network.
|
||||
- **Direct, encrypted links** between members (WireGuard cryptography), with relay fallback behind strict NATs.
|
||||
- **Names for machines:** `server.home` just works, even while `server` is switched off. A name belongs to whoever claimed it first; a later device with the same name keeps working by address and is told so.
|
||||
- **Exit nodes:** send all your internet traffic through any member that offers one.
|
||||
- **Multi-hop:** members that can't reach each other directly are routed through a member that can, over whatever kind of link each hop has. A member reached that way is online, and shown as relayed; traffic moves to a direct link by itself the moment one comes up.
|
||||
- **LAN games:** UDP broadcasts are relayed, so "LAN" lobbies show up across the mesh.
|
||||
- **Several networks at once** on one device.
|
||||
- **Tray app** for Windows, macOS and Linux; a command line for everything.
|
||||
|
||||
Replacing the signing key makes this device a different member, and it loses
|
||||
the address and name the old key held — nothing can sign on a retired key's
|
||||
behalf, and by design there is no authority that could overrule an author. So
|
||||
the outgoing key signs a release for every network on its way out, which
|
||||
frees them for whoever wants them next, and the whole thing commits at once.
|
||||
## Install
|
||||
|
||||
`status` and `id` both prefer a running agent, which is live and
|
||||
authoritative, and fall back to reading the state store when there is none.
|
||||
Reading takes no directory lock, so neither has to wait for the agent it is
|
||||
asking about — nor does either need one to be running.
|
||||
Grab a build from the [latest release](https://github.com/house-of-vanity/tsunagi/releases/latest).
|
||||
|
||||
### It cleans up after itself
|
||||
| Platform | How |
|
||||
| --------------------- | -------------------------------------------------------------------------------- |
|
||||
| **Windows** | Run `tsunagi-setup-<version>-x86_64.exe`. Installs the service, the tray and `tsng`. |
|
||||
| **macOS** | `brew tap house-of-vanity/tap && brew install --cask tsunagi-gui`<br>CLI only: `brew install tsunagi` |
|
||||
| **Debian / Ubuntu** | `sudo apt install ./tsunagi-gui_<version>_amd64.deb` (`tsunagi_…` for no tray) |
|
||||
| **Arch** | `sudo pacman -U tsunagi-gui-<version>-1-x86_64.pkg.tar.zst` |
|
||||
| **NixOS** | [Flake and module](#nixos) below |
|
||||
| **Any other Linux** | Unpack the tarball, then `sudo ./install.sh && sudo systemctl enable --now tsunagi` |
|
||||
|
||||
The interface is tied to an open file descriptor and is deliberately **not**
|
||||
made persistent, so the kernel removes it when the agent exits — on a clean
|
||||
shutdown, on a panic, on `SIGKILL`, on power loss alike. Keeping it is what
|
||||
would take an action; removing it is the default.
|
||||
Linux packages run the agent as an unprivileged service user. Add yourself to
|
||||
the `tsunagi` group (`sudo usermod -aG tsunagi $USER`, then log in again) to
|
||||
use `tsng` and the tray without `sudo`.
|
||||
|
||||
If something is left behind anyway — an interface made by an older version's
|
||||
manual recipe, or one from a run killed in the instant between creating it and
|
||||
recording it — the next start **replaces it**, along with any stale addresses
|
||||
it carried. Two things are never touched:
|
||||
## Quick start
|
||||
|
||||
* an interface that is not a TUN, because the name colliding with somebody's
|
||||
bridge is not a reason to destroy the bridge;
|
||||
* a TUN that another process is holding open, because that is a working
|
||||
overlay belonging to somebody else — most likely a second agent on this
|
||||
host, which should be given a different `--wg-prefix`.
|
||||
|
||||
Both of those refuse with an explanation rather than guessing.
|
||||
|
||||
Two settings the manual recipe used to need are gone with it.
|
||||
`keep_addr_on_down` existed only because an interface nobody held open lost
|
||||
carrier and had its IPv6 addresses flushed, and `nodad` only because duplicate
|
||||
address detection can never finish without carrier. An interface held open for
|
||||
its whole life has carrier for its whole life.
|
||||
|
||||
### The MTU is 1280
|
||||
|
||||
The default interface MTU stays at 1280 on direct paths, relays and paths
|
||||
inside another VPN. Large encrypted packets are split into smaller QUIC
|
||||
datagrams and reassembled before reaching WireGuard. SSH and full-size TCP
|
||||
segments therefore do not require a manual MTU override or MSS adjustment.
|
||||
The original IP packet, including its DF bit, is preserved. See
|
||||
[docs/wireguard.md](docs/wireguard.md#mtu) for limits and compatibility.
|
||||
|
||||
### Summary
|
||||
|
||||
| approach | agent runs as | notes |
|
||||
|---|---|---|
|
||||
| `setcap cap_net_admin+p` | ordinary user, one capability | recommended: nothing to prepare, nothing left behind. Lost on every rebuild or copy of the binary. |
|
||||
| systemd service | `User=`, `AmbientCapabilities=CAP_NET_ADMIN` | the same, for an installed service |
|
||||
| `sudo tsng up` | root | everything works, nothing is isolated |
|
||||
| macOS root LaunchDaemon | root | the native macOS deployment; see below |
|
||||
| `--no-tun` | ordinary user, no capabilities | tunnels run and handshake, traffic never reaches the OS |
|
||||
|
||||
On Windows, put `wintun.dll` beside the executable and start `tsng up`
|
||||
from PowerShell or Command Prompt opened with **Run as administrator**.
|
||||
Run commands controlling that agent (`join`, `status`, `dns`, `network`)
|
||||
as the same Windows user with the same elevation. If Windows denies access
|
||||
to the agent's control pipe, the command reports a permission error instead
|
||||
of claiming the agent is absent or trying to edit its locked state.
|
||||
Creating a TUN without sufficient privileges also explains how to restart
|
||||
the agent with the required permissions.
|
||||
|
||||
On macOS the agent creates a `utun` interface, configures it with `ifconfig`
|
||||
and `route`, and points the system resolver at the overlay by writing
|
||||
`/etc/resolver/<network>` files (which carry a port, so the DNS server needs no
|
||||
privileged socket). All of this needs **root**: macOS has no per-capability
|
||||
grant like Linux, and a `utun` plus `/etc/resolver` are root-only. The kernel
|
||||
assigns the interface its `utunN` name, so the name derived from the network id
|
||||
is only a request and the real one is adopted once the interface is up.
|
||||
|
||||
Run it as root — `sudo tsng up` for a quick start, or install it as a root
|
||||
**LaunchDaemon** for the native always-on deployment. As a daemon the agent runs
|
||||
unprivileged commands (`join`, `status`, `dns`, `network`) over its control
|
||||
socket like any other platform; the socket lives at `/var/run/tsunagi/agent.sock`
|
||||
and the stores at `/var/db/tsunagi`, both fixed system paths so they do not
|
||||
depend on whose shell started `sudo`. A ready-to-edit plist and install notes
|
||||
are in [dist/macos/](dist/macos/). The broadcast route is installed; the
|
||||
inbound-UDP firewall allowance (a `pf` anchor) is not yet, so `status` reports
|
||||
that half as incomplete and names what to allow by hand. The decision logic that
|
||||
says *what* to change is shared and tested on every platform.
|
||||
|
||||
## Checks
|
||||
|
||||
```bash
|
||||
cargo fmt --all -- --check
|
||||
cargo clippy --locked --workspace --all-targets -- -D warnings
|
||||
cargo test --locked --workspace --all-targets
|
||||
```sh
|
||||
tsng join -n home # first machine: makes the network and prints a secret
|
||||
tsng join -n home -s tsn1… # every other machine
|
||||
tsng status # who is online, over which path
|
||||
```
|
||||
|
||||
The whole suite runs offline on loopback. Set `TSUNAGI_TEST_LOG=tsunagi=debug`
|
||||
to see agent logs while a test runs.
|
||||
Every new network gets a stable pseudorandom `/24` in `10.0.0.0/8`, derived
|
||||
from its network id. Its members agree on that range even when starting offline,
|
||||
and each gets a name like `<host>.home`. Existing signed ranges are preserved.
|
||||
|
||||
There are also two runnable demos, which are demos and not substitutes for the
|
||||
tests:
|
||||
If two configured networks have overlapping ranges, the agent runs only one.
|
||||
`tsng status`, `tsng network` and the tray show which network blocks the
|
||||
other. `tsng network start <name-or-id>` or the tray’s “Switch to this network”
|
||||
button suspends the conflicting networks. They resume automatically when the
|
||||
selected network stops; the selection survives restarting the agent. Explicitly
|
||||
stopping a suspended network cancels its automatic resume.
|
||||
|
||||
```bash
|
||||
cargo run --example two_agents # control plane only
|
||||
Machines are called `tsunagi-<id>` until you rename them: `tsng id hostname laptop`.
|
||||
|
||||
### Virtual services
|
||||
|
||||
```sh
|
||||
tsng vip create home api
|
||||
tsng vip port add home api tcp/443 --backend server1:6443 --backend server2:7443
|
||||
tsng vip status home api --watch
|
||||
```
|
||||
|
||||
Both run with no privileges and change nothing on the host.
|
||||
Use `api.home:443`. Each client selects the available backend with the lowest
|
||||
measured data-plane RTT. A VIP can have multiple independent TCP and UDP ports.
|
||||
Every member may edit it; the tray's **VIP** button opens a draft editor with
|
||||
explicit Save. [Commands, health, metrics and limits](docs/vip.md).
|
||||
|
||||
## Usage
|
||||
### Exit nodes
|
||||
|
||||
```rust,no_run
|
||||
use std::sync::Arc;
|
||||
use tsunagi::config::{AgentConfig, StoragePaths, TransportPolicy};
|
||||
use tsunagi::discovery::SharedMemoryDiscovery;
|
||||
use tsunagi::identity::{NetworkName, NetworkSecret};
|
||||
use tsunagi::proto::ControlMessage;
|
||||
use tsunagi::{Agent, Result};
|
||||
```sh
|
||||
tsng network exit-node home on # on the machine that should be the exit
|
||||
tsng exit-node # on another one: who offers an exit
|
||||
tsng exit-node laptop # send everything through it
|
||||
tsng exit-node off
|
||||
```
|
||||
|
||||
// The library never starts a runtime, installs a logger, handles signals,
|
||||
// forks, or calls process::exit. The binary owns all of that.
|
||||
#[tokio::main]
|
||||
async fn main() -> Result<()> {
|
||||
let config = AgentConfig::new(StoragePaths::user_default()?)
|
||||
.with_transport(TransportPolicy::N0Defaults)
|
||||
.with_discovery(Arc::new(SharedMemoryDiscovery::new()));
|
||||
The tray does the same with a checkbox. An exit node sees everything you send
|
||||
through it, so use only members you trust.
|
||||
[Details and per-OS notes](docs/exit-node.md).
|
||||
|
||||
let agent = Agent::spawn(config).await?;
|
||||
The default data transport order is `wg,awg,wg-quic,tcp-tls`. AWG negotiates a
|
||||
fresh masking profile and outer keys for each pair connection attempt, including
|
||||
reentry after fallback. Sustained data loss triggers switching independently of
|
||||
the iroh control connection. [Configuration and limits](docs/amnezia.md).
|
||||
|
||||
let name = NetworkName::new("kitchen-table")?;
|
||||
let secret = NetworkSecret::generate(); // 32 random bytes
|
||||
println!("share this: {}", secret.encode().as_str());
|
||||
Change a network's protocol selection without editing its service or restarting:
|
||||
|
||||
let network = agent.join_network(&name, &secret).await?;
|
||||
```sh
|
||||
tsng protocols # available protocols and settings
|
||||
tsng protocols home # this device's choice for home
|
||||
tsng protocols home awg,wg-quic # prefer AWG, retain QUIC as fallback
|
||||
tsng protocols home none # control plane only
|
||||
```
|
||||
|
||||
let mut events = agent.subscribe();
|
||||
tokio::spawn(async move {
|
||||
while let Ok(event) = events.recv().await {
|
||||
println!("{event:?}");
|
||||
Exact network names, full ids and unique id prefixes work here. The choice is
|
||||
local to this device, saved per network and applied immediately; an offline
|
||||
change applies on the next start. In the tray, expand **protocols** on a network
|
||||
tile to toggle protocols or change their preference order. Both endpoints need
|
||||
a common protocol at the same wire version. `tsng up --protocol` supplies the
|
||||
initial defaults for networks without a saved choice; it does not restrict
|
||||
which built-in protocols can be enabled later.
|
||||
|
||||
Remove an obsolete offline device from a network's shared history:
|
||||
|
||||
```sh
|
||||
tsng network forget-node home old-laptop
|
||||
tsng network forget-node home 10.13.37.143
|
||||
```
|
||||
|
||||
Networks accept exact names, full ids or unique id prefixes; nodes accept those
|
||||
id selectors, exact hostnames or overlay IPv4 addresses. Ambiguous selectors
|
||||
change nothing. Start the network first if it is stopped. In **Show devices**,
|
||||
click **Forget** beside an offline node and confirm its identity.
|
||||
|
||||
Every member has the same cleanup right. The signed cleanup frees the observed
|
||||
claim's name and IPv4 reservation, survives restarts and reaches offline replicas
|
||||
when they reconnect. It does not ban the device: a newer signed claim can return.
|
||||
Online nodes and the currently selected exit node cannot be forgotten.
|
||||
[State and limits](docs/sync-model.md#offline-node-cleanup).
|
||||
|
||||
## Platforms
|
||||
|
||||
| Platform | Build | CLI `tsng` | Tray | Exit node: offer / use | Packages |
|
||||
| ------------------------- | ------------ | :--------: | :--: | :--------------------: | ----------------------------------- |
|
||||
| Linux x86_64 | release | ✔ | ✔ | ✔ / ✔ | `.deb`, `.pkg.tar.zst`, tarball, Nix |
|
||||
| Linux aarch64 | release | ✔ | –¹ | ✔ / ✔ | `.deb`, `.pkg.tar.zst`, tarball, Nix |
|
||||
| Linux armv7 | release | ✔ | –¹ | ✔ / ✔ | `.deb`, `.pkg.tar.zst`, tarball |
|
||||
| macOS, Apple silicon | release | ✔ | ✔ | ✔ / ✔² | Homebrew, tarball |
|
||||
| macOS, Intel | from source | ✔³ | ✔³ | ✔ / ✔²³ | – |
|
||||
| Windows x86_64 | release | ✔ | ✔ | ✔ / ✔² | installer, zip |
|
||||
|
||||
<sup>¹ Not in the release; builds from source (and through Nix).
|
||||
² Newer and less battle-tested than on Linux.
|
||||
³ Builds from source, not covered by CI.</sup>
|
||||
|
||||
The Linux CLI is a single static binary: it runs on any distribution.
|
||||
|
||||
## NixOS
|
||||
|
||||
The repository is a flake with a package and a module.
|
||||
|
||||
```nix
|
||||
# flake.nix
|
||||
{
|
||||
inputs.tsunagi.url = "github:house-of-vanity/tsunagi";
|
||||
|
||||
outputs = { nixpkgs, tsunagi, ... }: {
|
||||
nixosConfigurations.myhost = nixpkgs.lib.nixosSystem {
|
||||
modules = [
|
||||
tsunagi.nixosModules.default
|
||||
{
|
||||
services.tsunagi = {
|
||||
enable = true;
|
||||
tray.enable = true; # optional, for desktops
|
||||
users = [ "alice" ]; # may use tsng and the tray without sudo
|
||||
};
|
||||
}
|
||||
});
|
||||
|
||||
for peer in agent.network_status(network).await?.connected_peers() {
|
||||
agent
|
||||
.send(network, peer, ControlMessage::Ping { seq: 1, payload: vec![] })
|
||||
.await?;
|
||||
}
|
||||
|
||||
agent.shutdown().await;
|
||||
Ok(())
|
||||
];
|
||||
};
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
`TransportPolicy::LocalOnly` is the default, so a plain `AgentConfig::new` never
|
||||
reaches the internet by accident. Opt into `DirectOnly` or `N0Defaults`
|
||||
explicitly.
|
||||
Just trying it: `nix run github:house-of-vanity/tsunagi#tsunagi-cli -- --help`.
|
||||
`packages.<system>.tsunagi` (with the tray) and `tsunagi-cli` are also exported,
|
||||
along with `overlays.default`.
|
||||
|
||||
### Through somebody in the middle
|
||||
For development from a checkout, `nix develop` also exposes the tray's native
|
||||
runtime libraries, including AppIndicator and graphics/windowing libraries.
|
||||
Cargo-built binaries such as `target/debug/tsunagi-tray` can run directly inside
|
||||
this shell, without installing the packaged wrapper.
|
||||
|
||||
Tsunagi routes the overlay in userspace. A direct data link always wins;
|
||||
otherwise the shortest available path can pass through several members.
|
||||
Each authenticated member advertises only its own live, protocol-specific
|
||||
transport links. The control plane collects these into a graph and replaces
|
||||
the routing table when connectivity changes; announcements expire after
|
||||
90 seconds without refresh.
|
||||
## Build from source
|
||||
|
||||
Transit goes transport → router → transport, without entering WireGuard or
|
||||
the intermediate host's TUN. The payload stays encrypted between the original
|
||||
endpoints. TUN connects only the local OS to Tsunagi; the kernel needs only
|
||||
the overlay route through that interface, with no IP forwarding configuration.
|
||||
|
||||
A 16-hop limit bounds loops while topology updates converge. Equal shortest
|
||||
paths use a stable flow hash derived before encryption, so packets of a TCP
|
||||
or UDP flow keep their path while topology is unchanged. Physical link changes
|
||||
do not replace end-to-end WireGuard tunnels or change the 1280-byte default MTU.
|
||||
`status` reports the hop count and first next hop for relayed paths.
|
||||
|
||||
The hot path reads an immutable routing snapshot and sends directly to a cached
|
||||
transport handle. It takes no routing mutex, walks no graph and does not parse
|
||||
the encrypted payload. See [routing.md](docs/routing.md) for the architecture,
|
||||
limits and reproducible forwarding microbenchmark.
|
||||
|
||||
This wire format requires all members to upgrade together (control ALPN 3,
|
||||
data ALPN 4); saved identities, network names, secrets and addresses survive.
|
||||
|
||||
### LAN game discovery
|
||||
|
||||
IPv4 UDP broadcast relay is enabled by default for each network. The IP router
|
||||
sends one encrypted copy to each participating peer, including through multihop
|
||||
paths. It supports `255.255.255.255` and the overlay subnet's broadcast address;
|
||||
received broadcasts never trigger another fanout.
|
||||
Rust 1.91 or newer.
|
||||
|
||||
```sh
|
||||
tsng join -n games --no-broadcast
|
||||
tsng join -n games --broadcast
|
||||
tsng network broadcast games off
|
||||
tsng network broadcast games on
|
||||
cargo build --release --bin tsng # the agent and CLI
|
||||
cargo build --release --bin tsunagi-tray # the tray (Linux needs GTK 3 and libayatana-appindicator)
|
||||
```
|
||||
|
||||
The choice persists across restarts and a plain `join`. It can be changed while
|
||||
the agent runs; `status` shows it for each network. While broadcast is on, the
|
||||
agent routes `255.255.255.255` through the Tsunagi interface and allows inbound
|
||||
UDP from the overlay range on it (Linux and Windows, needs the same privilege as
|
||||
the interface), and removes both when it is turned off or the agent stops. Games
|
||||
bound to a physical address still bypass it. Physical LAN capture/subnet sharing
|
||||
is not implemented.
|
||||
See [broadcast.md](docs/broadcast.md) for domain isolation and future LAN gateways.
|
||||
Windows needs [Wintun](https://www.wintun.net) next to `tsng.exe`.
|
||||
Per-platform notes: [Linux](dist/linux/README.md), [Arch](dist/arch/README.md),
|
||||
[macOS](dist/macos/README.md), [Windows](dist/windows/README.md).
|
||||
|
||||
## How peers find each other
|
||||
## How it works
|
||||
|
||||
Two different lookups are involved, and only one of them is this project's:
|
||||
- Your name and secret derive a network identity. Members publish signed
|
||||
contact records to the public Mainline DHT under it, and find each other there.
|
||||
- Every connection proves knowledge of the secret before it's accepted.
|
||||
Traffic between members is end-to-end encrypted, relays included.
|
||||
- Addresses and names are claimed with each device's own key and replicated
|
||||
between members. There is no one in charge, so nothing to be taken down.
|
||||
|
||||
**1. Resolving one endpoint's address — iroh's, and it works today.**
|
||||
With `--reach relay` or `--reach direct`, iroh publishes a signed
|
||||
record of this endpoint's addresses, keyed by its endpoint id, to the public
|
||||
service run by Number 0 — "n0", the company behind iroh — at `dns.iroh.link`,
|
||||
over pkarr and DNS, and resolves other endpoints the same way. That is why `--peer <endpoint-id>` works with no address attached:
|
||||
iroh looks it up. None of that code is ours.
|
||||
|
||||
**2. Finding who is in a network — Mainline DHT by default.**
|
||||
`NetworkDiscovery` supplies unverified candidates. The Mainline backend uses
|
||||
a signing key derived from the network name and secret to publish signed
|
||||
BEP44 records containing each writer's endpoint ID and addresses. Its 16
|
||||
slots provide starting candidates, not a complete membership list or a limit
|
||||
on network size. After the first authenticated connection, introductions and
|
||||
signed state reveal the other members. `StaticBootstrap` (`--peer`) remains
|
||||
available, and tests use either a local Mainline Testnet or in-memory discovery.
|
||||
|
||||
What this means in practice:
|
||||
|
||||
- With `relay` or `direct`, **your endpoint id and IP addresses are published
|
||||
to a public third-party service** (Number 0's, unless you change it). They are not secret, and the network secret is
|
||||
never published, but an observer of that service learns that your endpoint
|
||||
exists and where it is. `--reach local` publishes nothing.
|
||||
- With DHT enabled, endpoint records are also published to the public Mainline
|
||||
DHT. They are signed, not encrypted; membership still requires the secret.
|
||||
- A relay, when one is needed, sees the volume and timing of your traffic — not
|
||||
its contents. The default relays are Number 0's, in the US, EU and
|
||||
Asia-Pacific.
|
||||
|
||||
## Storage
|
||||
|
||||
Two physically separate SQLite files, placed wherever the library's
|
||||
configuration says (`StoragePaths`). A future system service supplies its own
|
||||
paths; tests always use temporary directories.
|
||||
|
||||
| file | holds | when damaged |
|
||||
|----------------|--------------------------------------------------|-------------------------|
|
||||
| `state.sqlite` | device identity, network configuration, hostname | clear error, never reset |
|
||||
| `cache.sqlite` | address hints and other recoverable data | discarded and recreated |
|
||||
|
||||
The `wg-quic` protocol keeps its own keys in its own store under `wg-quic/`,
|
||||
because a protocol's keys are neither the iroh identity nor the network
|
||||
secret.
|
||||
|
||||
The command line agent puts everything under the platform's per-user
|
||||
directories by default; `--state-dir` and `--cache-dir` override them.
|
||||
|
||||
One state directory belongs to one live agent, enforced with a real OS file
|
||||
lock rather than an existence check. That directory *is* the identity: one
|
||||
agent, one device key, one interface, and as many networks as you like on it
|
||||
— `tsng network join` adds them to the agent that is already running,
|
||||
which is why a second `tsng up` on the same directory is refused rather
|
||||
than made to work. A second agent on the same host is a second identity, and
|
||||
needs everything of its own: its own state and cache directories, its own
|
||||
interface name and its own overlay range. The library owns no globals, so
|
||||
several of them run side by side in one process as readily as in one host.
|
||||
|
||||
### Leaving, and starting over
|
||||
|
||||
Membership outlives a session, so it also has to be possible to end it.
|
||||
|
||||
```bash
|
||||
tsng network # what this device belongs to
|
||||
tsng network leave lab # give up the address and the name, then forget it
|
||||
tsng wipe --yes # remove everything and be a stranger again
|
||||
```
|
||||
|
||||
**Stopping is not leaving.** `stop` closes this network's sessions, takes
|
||||
its address off the interface and keeps it from starting again, and that is
|
||||
all: the configuration, the secret, the signed state and the protocol key
|
||||
stay exactly as they are, nothing is announced, and to the others this
|
||||
device is simply away — an ordinary condition they already handle, with its
|
||||
address and name still reserved for it. `start` picks it up where it left
|
||||
off. A network named on the `up` command line is started by that command
|
||||
whatever its stored state, and the start-up banner says so rather than
|
||||
letting a `stop` quietly come back.
|
||||
|
||||
`leave` is the other one, and it removes, locally: the configuration and
|
||||
the secret, this network's signed records, its cached address hints and the
|
||||
protocol key it used there. What it keeps is this author's version counter
|
||||
for that network — a rejoin has to continue above the release, or every
|
||||
replica would treat the new claim as stale. Everything about *other*
|
||||
networks, and the device identity itself, is untouched.
|
||||
|
||||
`leave` publishes a signed `Release` **first**, while the agent is running and
|
||||
its sessions are up, so the address and the name are freed for the others
|
||||
instead of staying reserved to a member that has gone. They pass the tombstone
|
||||
on, so a member that was away hears it from them. With no agent running,
|
||||
nothing can sign or send it: the command says so and refuses, and `--offline`
|
||||
drops the network locally while leaving the others holding the old claim. The
|
||||
protocol key for that network goes too — rejoining is joining, not resuming.
|
||||
|
||||
`leave`, `stop`, `start`, `secret` and `broadcast` accept an exact stored network
|
||||
name, its full id, or a unique id prefix (including a copied report ellipsis).
|
||||
Names are case-sensitive. If several networks share a name, or a name matches
|
||||
another network's id prefix, the command lists matching names and full ids and
|
||||
changes nothing. Use a full id to choose explicitly; a full id always takes
|
||||
precedence over a name. No secret is needed to select a configured network.
|
||||
|
||||
`wipe` removes both directories' contents: the device identity, every network,
|
||||
every signed record and everything a protocol kept beside them. It refuses
|
||||
while an agent is running, and refuses a directory with no `state.sqlite` in
|
||||
it, so a mistyped `--state-dir` cannot take somebody's documents with it.
|
||||
Without `--yes` it only says what it would remove. It is not a goodbye: nobody
|
||||
is told, because after it there is no key left to sign anything with. Leave the
|
||||
networks first if the addresses should be freed.
|
||||
The secret is the only key, so make it a good one (the generated ones are).
|
||||
The [threat model](docs/threat-model.md) says what this does and doesn't protect against.
|
||||
|
||||
## Documentation
|
||||
|
||||
- [docs/architecture.md](docs/architecture.md) — module boundaries and runtime.
|
||||
- [docs/wireguard.md](docs/wireguard.md) — the WireGuard plugin: overlay
|
||||
addressing, announcements, backends, reconciliation.
|
||||
- [docs/protocol.md](docs/protocol.md) — identity derivation, framing, handshake.
|
||||
- [docs/sync-model.md](docs/sync-model.md) — the planned signed-state model and
|
||||
what is deliberately not built yet.
|
||||
- [docs/exit-node.md](docs/exit-node.md) — exit nodes: sending all traffic through a member.
|
||||
- [docs/threat-model.md](docs/threat-model.md) — threat model and known limits.
|
||||
- [docs/testing.md](docs/testing.md) — what the suite covers and what it does not.
|
||||
- [AGENTS.md](AGENTS.md) — rules for anyone (human or agent) changing this repo.
|
||||
|
||||
## Security in one paragraph
|
||||
|
||||
Membership is proved by an HMAC over a transcript keyed by a value derived from
|
||||
the shared secret, bound to the specific iroh connection through the TLS
|
||||
exporter, to the network id, to both endpoint identities and to distinct role
|
||||
labels. This targets high-entropy secrets: there is no PAKE here, so a short
|
||||
human passphrase is guessable offline by anyone who can reach the handshake.
|
||||
Anyone who knows the secret is a full participant and can create many
|
||||
identities. Read [docs/threat-model.md](docs/threat-model.md) before relying on
|
||||
any of this.
|
||||
[Architecture](docs/architecture.md) ·
|
||||
[Protocol](docs/protocol.md) ·
|
||||
[WireGuard data plane](docs/wireguard.md) ·
|
||||
[Pairwise Amnezia transport](docs/amnezia.md) ·
|
||||
[Routing](docs/routing.md) ·
|
||||
[Exit nodes](docs/exit-node.md) ·
|
||||
[Virtual services](docs/vip.md) ·
|
||||
[Threat model](docs/threat-model.md) ·
|
||||
[Contributing](AGENTS.md)
|
||||
|
||||
## Licence
|
||||
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
[package]
|
||||
name = "tsunagi-awg"
|
||||
version = "0.1.0"
|
||||
description = "Ephemeral pairwise Amnezia transport for Tsunagi."
|
||||
edition.workspace = true
|
||||
rust-version.workspace = true
|
||||
license = "WTFPL AND Apache-2.0"
|
||||
repository.workspace = true
|
||||
|
||||
[dependencies]
|
||||
tsunagi-core = { path = "../tsunagi-core" }
|
||||
tsunagi-wg-quic = { path = "../tsunagi-wg-quic" }
|
||||
tsunagi-wireguard = { path = "../tsunagi-wireguard" }
|
||||
iroh.workspace = true
|
||||
tokio = { workspace = true, features = ["net", "time", "sync", "macros"] }
|
||||
serde.workspace = true
|
||||
postcard.workspace = true
|
||||
bytes.workspace = true
|
||||
tracing.workspace = true
|
||||
rand = "0.10"
|
||||
zeroize = { version = "1.9", features = ["derive"] }
|
||||
boringtun = { version = "0.7.1", default-features = false }
|
||||
chacha20 = "0.9"
|
||||
|
||||
[dev-dependencies]
|
||||
tsunagi-core = { path = "../tsunagi-core", features = ["testing"] }
|
||||
tempfile.workspace = true
|
||||
|
||||
[lints]
|
||||
workspace = true
|
||||
@@ -0,0 +1,201 @@
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding those notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
APPENDIX: How to apply the Apache License to your work.
|
||||
|
||||
To apply the Apache License to your work, attach the following
|
||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
||||
replaced with your own identifying information. (Don't include
|
||||
the brackets!) The text should be enclosed in the appropriate
|
||||
comment syntax for the file format. We also recommend that a
|
||||
file or class name and description of purpose be included on the
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright 2026 Friderika Coulson (NanoBee)
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
@@ -0,0 +1,21 @@
|
||||
# Amnezia packet engine
|
||||
|
||||
`src/masking.rs` is adapted from `src/core/amnezia.rs` in
|
||||
https://github.com/RunaRoo/AmneziaWG-RS at
|
||||
`51579a7d6e8882d1d8d6d00822f09e18a9423b73`, under Apache-2.0.
|
||||
The upstream license is retained in `LICENSE-AmneziaWG-RS`.
|
||||
|
||||
The adaptation replaces configuration-file loading with bounded typed pair
|
||||
profiles, updates RNG calls to rand 0.10, removes an unnecessary unwrap, and
|
||||
does not embed upstream's Node, global executors, OS interface operations or
|
||||
detached tasks. Noise/WireGuard is supplied by boringtun, already used by
|
||||
Tsunagi. Each pair owns one socket and one cancellable packet-engine task.
|
||||
|
||||
This is Tsunagi AWG wire version 1, not a claim of compatibility with external
|
||||
Amnezia servers. The masking engine substitutes message headers outside the
|
||||
standard WireGuard state machine and restores them before Noise validation.
|
||||
There are no remote file paths, shell hooks, or persisted pair profiles.
|
||||
|
||||
The adapted masking source remains Apache-2.0. Newly written Tsunagi code
|
||||
follows the repository's WTFPL. This package declares both licenses in Cargo
|
||||
metadata; the workspace license alone does not describe its copied source.
|
||||
@@ -0,0 +1,16 @@
|
||||
//! Pairwise, ephemeral Amnezia packet transport with independent end-to-end tunnels.
|
||||
//! All negotiation travels over authenticated Tsunagi control sessions.
|
||||
mod masking;
|
||||
mod plugin;
|
||||
pub mod profile;
|
||||
mod session;
|
||||
mod stun;
|
||||
mod transport;
|
||||
|
||||
pub use plugin::AwgPlugin;
|
||||
pub use transport::{AwgTransport, PairOverview};
|
||||
|
||||
/// Protocol identifier and default second-choice direct transport.
|
||||
pub const AWG_PROTOCOL: &str = "awg";
|
||||
/// Tsunagi AWG wire version, independent of software releases.
|
||||
pub const AWG_WIRE_VERSION: u16 = 1;
|
||||
@@ -0,0 +1,336 @@
|
||||
// Adapted from RunaRoo/AmneziaWG-RS, commit 51579a7d6e8882d1d8d6d00822f09e18a9423b73.
|
||||
// Apache-2.0; see LICENSE-AmneziaWG-RS and UPSTREAM.md.
|
||||
#![allow(dead_code, missing_docs)]
|
||||
// AmneziaWG configuration, header protection, and traffic shaping.
|
||||
|
||||
use bytes::{Bytes, BytesMut};
|
||||
use chacha20::{
|
||||
ChaCha20,
|
||||
cipher::{KeyIvInit, StreamCipher},
|
||||
};
|
||||
use rand::RngExt;
|
||||
|
||||
pub const HEADER_PROTECTION_NONCE_SIZE: usize = 12;
|
||||
pub const DEFAULT_UDP_WINDOW: usize = 500;
|
||||
|
||||
#[derive(Clone, Copy, Debug, Default, Eq, PartialEq)]
|
||||
pub struct Range {
|
||||
pub min: u32,
|
||||
pub max: u32,
|
||||
}
|
||||
|
||||
impl Range {
|
||||
pub fn contains(&self, val: u32) -> bool {
|
||||
val >= self.min && val <= self.max
|
||||
}
|
||||
|
||||
pub fn random(&self) -> u32 {
|
||||
if self.min >= self.max {
|
||||
self.min
|
||||
} else {
|
||||
rand::rng().random_range(self.min..=self.max)
|
||||
}
|
||||
}
|
||||
|
||||
pub fn is_zero(&self) -> bool {
|
||||
self.min == 0 && self.max == 0
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
|
||||
pub enum AwgKind {
|
||||
Initiation,
|
||||
Response,
|
||||
Cookie,
|
||||
Transport,
|
||||
}
|
||||
|
||||
impl AwgKind {
|
||||
pub fn fixed_size(self) -> usize {
|
||||
match self {
|
||||
Self::Initiation => 148,
|
||||
Self::Response => 92,
|
||||
Self::Cookie => 64,
|
||||
Self::Transport => 32,
|
||||
}
|
||||
}
|
||||
|
||||
fn standard_type(self) -> u32 {
|
||||
match self {
|
||||
Self::Initiation => 1,
|
||||
Self::Response => 2,
|
||||
Self::Cookie => 3,
|
||||
Self::Transport => 4,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
pub struct DecodedPacket {
|
||||
pub kind: AwgKind,
|
||||
pub message: Bytes,
|
||||
pub wire_len: usize,
|
||||
}
|
||||
|
||||
pub struct AmneziaConfig {
|
||||
pub jc: u32,
|
||||
pub jmin: usize,
|
||||
pub jmax: usize,
|
||||
pub s1: usize,
|
||||
pub s2: usize,
|
||||
pub s3: usize,
|
||||
pub s4: usize,
|
||||
pub h1: Range,
|
||||
pub h2: Range,
|
||||
pub h3: Range,
|
||||
pub h4: Range,
|
||||
pub i1: String,
|
||||
pub i2: String,
|
||||
pub i3: String,
|
||||
pub i4: String,
|
||||
pub i5: String,
|
||||
pub header_protection_key: Option<[u8; 32]>,
|
||||
pub content_padding_addition: Range,
|
||||
pub random_trailers: bool,
|
||||
pub mtu: usize,
|
||||
}
|
||||
|
||||
impl AmneziaConfig {
|
||||
pub fn random_header(&self, range: &Range) -> u32 {
|
||||
range.random()
|
||||
}
|
||||
|
||||
pub fn random_padding(&self, len: usize) -> Vec<u8> {
|
||||
let mut padding = vec![0u8; len];
|
||||
if len > 0 {
|
||||
rand::fill(&mut padding);
|
||||
}
|
||||
padding
|
||||
}
|
||||
|
||||
fn shape(&self, kind: AwgKind) -> (usize, Range) {
|
||||
match kind {
|
||||
AwgKind::Initiation => (self.s1, self.h1),
|
||||
AwgKind::Response => (self.s2, self.h2),
|
||||
AwgKind::Cookie => (self.s3, self.h3),
|
||||
AwgKind::Transport => (self.s4, self.h4),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn encode(&self, kind: AwgKind, payload: Bytes, udp_window: usize) -> Bytes {
|
||||
let (padding_len, _) = self.shape(kind);
|
||||
let trailer_len = if kind != AwgKind::Transport && self.random_trailers {
|
||||
let base = padding_len.saturating_add(payload.len());
|
||||
let room = udp_window.max(DEFAULT_UDP_WINDOW).saturating_sub(base);
|
||||
if room == 0 {
|
||||
0
|
||||
} else {
|
||||
rand::rng().random_range(0..room)
|
||||
}
|
||||
} else {
|
||||
0
|
||||
};
|
||||
|
||||
let mut out = BytesMut::zeroed(padding_len + payload.len() + trailer_len);
|
||||
if padding_len > 0 {
|
||||
rand::fill(&mut out[..padding_len]);
|
||||
}
|
||||
out[padding_len..padding_len + payload.len()].copy_from_slice(&payload);
|
||||
if trailer_len > 0 {
|
||||
rand::fill(&mut out[padding_len + payload.len()..]);
|
||||
}
|
||||
|
||||
if self.header_protection_key.is_some() {
|
||||
let protected_len = if kind == AwgKind::Transport {
|
||||
16
|
||||
} else {
|
||||
kind.fixed_size()
|
||||
};
|
||||
self.apply_header_cipher(&mut out, padding_len, protected_len);
|
||||
}
|
||||
|
||||
out.freeze()
|
||||
}
|
||||
|
||||
pub fn decode(&self, datagram: &Bytes) -> Option<DecodedPacket> {
|
||||
if datagram.len() < 4 {
|
||||
return None;
|
||||
}
|
||||
|
||||
for kind in [
|
||||
AwgKind::Initiation,
|
||||
AwgKind::Response,
|
||||
AwgKind::Cookie,
|
||||
AwgKind::Transport,
|
||||
] {
|
||||
let (padding_len, headers) = self.shape(kind);
|
||||
let minimum = padding_len + kind.fixed_size();
|
||||
let size_ok = match kind {
|
||||
AwgKind::Transport | AwgKind::Initiation => datagram.len() >= minimum,
|
||||
AwgKind::Response | AwgKind::Cookie if self.random_trailers => {
|
||||
datagram.len() >= minimum
|
||||
}
|
||||
AwgKind::Response | AwgKind::Cookie => datagram.len() == minimum,
|
||||
};
|
||||
if !size_ok || datagram.len() < padding_len + 4 {
|
||||
continue;
|
||||
}
|
||||
|
||||
let protected_header =
|
||||
u32::from_le_bytes(datagram[padding_len..padding_len + 4].try_into().ok()?);
|
||||
let candidate = if self.header_protection_key.is_some() {
|
||||
protected_header ^ self.header_type_hash(datagram)?
|
||||
} else {
|
||||
protected_header
|
||||
};
|
||||
if !headers.contains(candidate) {
|
||||
continue;
|
||||
}
|
||||
|
||||
let message_len = match kind {
|
||||
AwgKind::Transport | AwgKind::Initiation => datagram.len() - padding_len,
|
||||
AwgKind::Response | AwgKind::Cookie => kind.fixed_size(),
|
||||
};
|
||||
let mut payload = BytesMut::from(&datagram[padding_len..padding_len + message_len]);
|
||||
if self.header_protection_key.is_some() {
|
||||
let protected_len = if kind == AwgKind::Transport {
|
||||
16
|
||||
} else {
|
||||
kind.fixed_size()
|
||||
};
|
||||
self.apply_header_cipher_with_nonce(
|
||||
&mut payload,
|
||||
0,
|
||||
protected_len,
|
||||
&datagram[..HEADER_PROTECTION_NONCE_SIZE],
|
||||
);
|
||||
}
|
||||
|
||||
return Some(DecodedPacket {
|
||||
kind,
|
||||
message: payload.freeze(),
|
||||
wire_len: datagram.len(),
|
||||
});
|
||||
}
|
||||
|
||||
None
|
||||
}
|
||||
|
||||
pub fn prepare_transport_plaintext(&self, packet: Bytes, udp_window: usize) -> Bytes {
|
||||
if packet.is_empty() {
|
||||
return packet;
|
||||
}
|
||||
|
||||
let requested = if !self.content_padding_addition.is_zero() {
|
||||
self.content_padding_addition.random() as usize
|
||||
} else if self.random_trailers {
|
||||
let wire_without_padding = self.s4 + AwgKind::Transport.fixed_size() + packet.len();
|
||||
let room = udp_window
|
||||
.max(DEFAULT_UDP_WINDOW)
|
||||
.saturating_sub(wire_without_padding);
|
||||
if room == 0 {
|
||||
0
|
||||
} else {
|
||||
rand::rng().random_range(0..room)
|
||||
}
|
||||
} else {
|
||||
let aligned = (packet.len() + 15) & !15;
|
||||
aligned.saturating_sub(packet.len())
|
||||
};
|
||||
|
||||
let padding = self.clamp_padding_to_mtu(packet.len(), requested);
|
||||
if padding == 0 {
|
||||
return packet;
|
||||
}
|
||||
|
||||
let mut out = BytesMut::with_capacity(packet.len() + padding);
|
||||
out.extend_from_slice(&packet);
|
||||
out.resize(packet.len() + padding, 0);
|
||||
out.freeze()
|
||||
}
|
||||
|
||||
pub fn transport_content_length(&self, plaintext: &[u8]) -> Option<usize> {
|
||||
if plaintext.is_empty() || plaintext[0] == 0 {
|
||||
return Some(0);
|
||||
}
|
||||
match plaintext[0] >> 4 {
|
||||
4 if plaintext.len() >= 20 => {
|
||||
let total = u16::from_be_bytes([plaintext[2], plaintext[3]]) as usize;
|
||||
(total >= 20 && total <= plaintext.len()).then_some(total)
|
||||
}
|
||||
6 if plaintext.len() >= 40 => {
|
||||
let payload = u16::from_be_bytes([plaintext[4], plaintext[5]]) as usize;
|
||||
let total = 40 + payload;
|
||||
(total <= plaintext.len()).then_some(total)
|
||||
}
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
fn clamp_padding_to_mtu(&self, packet_len: usize, requested: usize) -> usize {
|
||||
if self.mtu == 0 {
|
||||
return requested;
|
||||
}
|
||||
let normalized = if packet_len > self.mtu {
|
||||
packet_len % self.mtu
|
||||
} else {
|
||||
packet_len
|
||||
};
|
||||
requested.min(self.mtu.saturating_sub(normalized))
|
||||
}
|
||||
|
||||
fn header_type_hash(&self, datagram: &Bytes) -> Option<u32> {
|
||||
if datagram.len() < HEADER_PROTECTION_NONCE_SIZE {
|
||||
return None;
|
||||
}
|
||||
let mut hash = [0u8; 4];
|
||||
self.apply_keystream(&mut hash, &datagram[..HEADER_PROTECTION_NONCE_SIZE]);
|
||||
Some(u32::from_le_bytes(hash))
|
||||
}
|
||||
|
||||
fn apply_header_cipher(&self, packet: &mut [u8], offset: usize, len: usize) {
|
||||
if packet.len() < HEADER_PROTECTION_NONCE_SIZE || packet.len() < offset + len {
|
||||
return;
|
||||
}
|
||||
let nonce: [u8; HEADER_PROTECTION_NONCE_SIZE] =
|
||||
match packet[..HEADER_PROTECTION_NONCE_SIZE].try_into() {
|
||||
Ok(nonce) => nonce,
|
||||
Err(_) => return,
|
||||
};
|
||||
self.apply_header_cipher_with_nonce(packet, offset, len, &nonce);
|
||||
}
|
||||
|
||||
fn apply_header_cipher_with_nonce(
|
||||
&self,
|
||||
packet: &mut [u8],
|
||||
offset: usize,
|
||||
len: usize,
|
||||
nonce: &[u8],
|
||||
) {
|
||||
if packet.len() < offset + len {
|
||||
return;
|
||||
}
|
||||
self.apply_keystream(&mut packet[offset..offset + len], nonce);
|
||||
}
|
||||
|
||||
fn apply_keystream(&self, data: &mut [u8], nonce: &[u8]) {
|
||||
let Some(key) = self.header_protection_key.as_ref() else {
|
||||
return;
|
||||
};
|
||||
if nonce.len() != HEADER_PROTECTION_NONCE_SIZE {
|
||||
return;
|
||||
}
|
||||
let Ok(nonce) = <&[u8; HEADER_PROTECTION_NONCE_SIZE]>::try_from(nonce) else {
|
||||
return;
|
||||
};
|
||||
let mut cipher = ChaCha20::new(key.into(), nonce.into());
|
||||
cipher.apply_keystream(data);
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for AmneziaConfig {
|
||||
fn drop(&mut self) {
|
||||
use zeroize::Zeroize;
|
||||
self.header_protection_key.zeroize();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,132 @@
|
||||
//! End-to-end encryption remains independent of physical Amnezia pair links.
|
||||
use crate::{AWG_PROTOCOL, AWG_WIRE_VERSION, AwgTransport};
|
||||
use iroh::EndpointId;
|
||||
use std::sync::Arc;
|
||||
use tsunagi::dataplane::transport::SharedLink;
|
||||
use tsunagi::dataplane::{IpPlugin, PluginCapability, PluginContext, PluginError, ProtocolOption};
|
||||
use tsunagi::state::Ipv4Range;
|
||||
use tsunagi::{BoxFuture, NetworkId};
|
||||
use tsunagi_wg_quic::{NetworkOverview, WireguardConfig, WireguardPlugin};
|
||||
|
||||
/// AWG capability and an independent end-to-end WireGuard engine.
|
||||
/// Pair transport keys/profiles never enter its durable end-to-end key store.
|
||||
#[derive(Debug)]
|
||||
pub struct AwgPlugin {
|
||||
inner: Arc<WireguardPlugin>,
|
||||
transport: Arc<AwgTransport>,
|
||||
}
|
||||
impl AwgPlugin {
|
||||
/// Uses standard tunnel MTU/keepalive options; pair ports and profiles are ephemeral.
|
||||
pub const OPTIONS: &'static [ProtocolOption] = &[
|
||||
ProtocolOption {
|
||||
key: "keepalive",
|
||||
value: "SECONDS",
|
||||
help: "end-to-end tunnel keepalive; 0 disables it",
|
||||
default: Some("25"),
|
||||
},
|
||||
ProtocolOption {
|
||||
key: "mtu",
|
||||
value: "BYTES",
|
||||
help: "maximum overlay packet size",
|
||||
default: Some("1280"),
|
||||
},
|
||||
ProtocolOption {
|
||||
key: "stun",
|
||||
value: "HOST:PORT|off",
|
||||
help: "UDP public mapping discovery; disabled in local reach mode",
|
||||
default: Some("stun.l.google.com:19302"),
|
||||
},
|
||||
];
|
||||
/// Opens the end-to-end engine, retaining separate pair transport state.
|
||||
pub async fn open(
|
||||
mut config: WireguardConfig,
|
||||
transport: Arc<AwgTransport>,
|
||||
) -> Result<Arc<Self>, PluginError> {
|
||||
config.protocol = AWG_PROTOCOL;
|
||||
Ok(Arc::new(Self {
|
||||
inner: WireguardPlugin::open(config).await?,
|
||||
transport,
|
||||
}))
|
||||
}
|
||||
/// Reports end-to-end tunnel health and traffic counters.
|
||||
pub fn overview(&self, network: NetworkId) -> Option<NetworkOverview> {
|
||||
self.inner.overview(network)
|
||||
}
|
||||
}
|
||||
impl IpPlugin for AwgPlugin {
|
||||
fn protocol_id(&self) -> &str {
|
||||
AWG_PROTOCOL
|
||||
}
|
||||
fn protocol_version(&self) -> u16 {
|
||||
AWG_WIRE_VERSION
|
||||
}
|
||||
fn options(&self) -> &'static [ProtocolOption] {
|
||||
Self::OPTIONS
|
||||
}
|
||||
fn attach(&self, context: PluginContext) {
|
||||
self.inner.attach(context);
|
||||
}
|
||||
fn local_capability(
|
||||
&self,
|
||||
network: NetworkId,
|
||||
) -> Result<Option<PluginCapability>, PluginError> {
|
||||
self.inner.local_capability(network).map(|cap| {
|
||||
cap.map(|mut cap| {
|
||||
cap.protocol = AWG_PROTOCOL.into();
|
||||
cap.version = AWG_WIRE_VERSION;
|
||||
cap
|
||||
})
|
||||
})
|
||||
}
|
||||
fn on_network_activated(&self, network: NetworkId) {
|
||||
self.inner.on_network_activated(network);
|
||||
}
|
||||
fn on_peer_capability(
|
||||
&self,
|
||||
network: NetworkId,
|
||||
peer: EndpointId,
|
||||
capability: &PluginCapability,
|
||||
) -> Result<(), PluginError> {
|
||||
if capability.protocol != AWG_PROTOCOL
|
||||
|| capability.version != AWG_WIRE_VERSION
|
||||
|| !capability.enabled
|
||||
{
|
||||
return Err(PluginError::Rejected("unsupported AWG capability".into()));
|
||||
}
|
||||
self.inner.on_peer_capability(network, peer, capability)?;
|
||||
self.transport.allow_peer(network, peer);
|
||||
Ok(())
|
||||
}
|
||||
fn on_address_allocation(
|
||||
&self,
|
||||
network: NetworkId,
|
||||
range: Ipv4Range,
|
||||
allocations: &[(EndpointId, std::net::Ipv4Addr)],
|
||||
) {
|
||||
self.inner
|
||||
.on_address_allocation(network, range, allocations);
|
||||
}
|
||||
fn carry(&self, network: NetworkId, peer: EndpointId, packet: bytes::Bytes) -> bool {
|
||||
self.inner.carry(network, peer, packet)
|
||||
}
|
||||
fn on_peer_link(&self, network: NetworkId, peer: EndpointId, link: SharedLink) {
|
||||
self.inner.on_peer_link(network, peer, link);
|
||||
}
|
||||
fn on_peer_gone(&self, network: NetworkId, peer: EndpointId) {
|
||||
self.transport.forget_peer(network, peer);
|
||||
self.inner.on_peer_gone(network, peer);
|
||||
}
|
||||
fn on_network_deactivated(&self, network: NetworkId) {
|
||||
self.transport.stop_network(network);
|
||||
self.inner.on_network_deactivated(network);
|
||||
}
|
||||
fn on_network_forgotten(&self, network: NetworkId) {
|
||||
self.inner.on_network_forgotten(network);
|
||||
}
|
||||
fn shutdown(&self) -> BoxFuture<'_, ()> {
|
||||
Box::pin(async move {
|
||||
self.transport.shutdown();
|
||||
self.inner.shutdown().await;
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,226 @@
|
||||
//! Bounded profiles, regenerated for every pair connection attempt.
|
||||
use crate::masking::{AmneziaConfig, AwgKind, Range};
|
||||
use bytes::Bytes;
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
/// An ephemeral masking profile. Secret fields are redacted and erased on drop.
|
||||
#[derive(Clone, Serialize, Deserialize, zeroize::ZeroizeOnDrop)]
|
||||
pub struct PairProfile {
|
||||
id: [u8; 16],
|
||||
prefix: u16,
|
||||
headers: [[u32; 2]; 4],
|
||||
header_key: [u8; 32],
|
||||
pub(crate) psk: [u8; 32],
|
||||
junk_count: u8,
|
||||
junk_min: u16,
|
||||
junk_max: u16,
|
||||
padding: u16,
|
||||
signature: u8,
|
||||
}
|
||||
impl std::fmt::Debug for PairProfile {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
f.write_str("PairProfile([REDACTED])")
|
||||
}
|
||||
}
|
||||
impl PairProfile {
|
||||
/// Generates a new safe profile; no deterministic seed or durable state.
|
||||
pub fn generate() -> Self {
|
||||
let headers = std::array::from_fn(|i| {
|
||||
let base = (i as u32) * 0x4000_0000 + rand::random_range(1024..0x3fff_0000);
|
||||
[base, base + rand::random_range(64..1024)]
|
||||
});
|
||||
Self {
|
||||
id: rand::random(),
|
||||
prefix: rand::random_range(12..49),
|
||||
headers,
|
||||
header_key: rand::random(),
|
||||
psk: rand::random(),
|
||||
junk_count: rand::random_range(4..9),
|
||||
junk_min: rand::random_range(24..65),
|
||||
junk_max: rand::random_range(80..161),
|
||||
padding: rand::random_range(16..65),
|
||||
signature: rand::random_range(0..2),
|
||||
}
|
||||
}
|
||||
/// Public attempt identifier, useful for distinguishing retries without revealing secrets.
|
||||
pub fn id(&self) -> [u8; 16] {
|
||||
self.id
|
||||
}
|
||||
/// Header key, for packet engines and protocol tests; never log this value.
|
||||
pub fn header_key(&self) -> &[u8; 32] {
|
||||
&self.header_key
|
||||
}
|
||||
/// Rejects invalid remote profiles before allocating packets or entering the codec.
|
||||
pub fn validate(&self) -> Result<(), &'static str> {
|
||||
if !(12..=48).contains(&self.prefix)
|
||||
|| !(4..=8).contains(&self.junk_count)
|
||||
|| self.junk_min < 24
|
||||
|| self.junk_min > self.junk_max
|
||||
|| self.junk_max > 160
|
||||
|| self.padding > 64
|
||||
|| self.signature > 1
|
||||
|| self.header_key == [0; 32]
|
||||
|| self.psk == [0; 32]
|
||||
{
|
||||
return Err("invalid AWG profile bounds");
|
||||
}
|
||||
for (i, [lo, hi]) in self.headers.iter().copied().enumerate() {
|
||||
if lo < 5 || lo > hi || hi - lo > 1024 {
|
||||
return Err("invalid AWG header range");
|
||||
}
|
||||
for [other_lo, other_hi] in &self.headers[..i] {
|
||||
if lo <= *other_hi && *other_lo <= hi {
|
||||
return Err("overlapping AWG header ranges");
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
fn config(&self) -> AmneziaConfig {
|
||||
let range = |i: usize| Range {
|
||||
min: self.headers[i][0],
|
||||
max: self.headers[i][1],
|
||||
};
|
||||
AmneziaConfig {
|
||||
jc: self.junk_count.into(),
|
||||
jmin: self.junk_min.into(),
|
||||
jmax: self.junk_max.into(),
|
||||
s1: self.prefix.into(),
|
||||
s2: self.prefix.into(),
|
||||
s3: self.prefix.into(),
|
||||
s4: self.prefix.into(),
|
||||
h1: range(0),
|
||||
h2: range(1),
|
||||
h3: range(2),
|
||||
h4: range(3),
|
||||
i1: String::new(),
|
||||
i2: String::new(),
|
||||
i3: String::new(),
|
||||
i4: String::new(),
|
||||
i5: String::new(),
|
||||
header_protection_key: Some(self.header_key),
|
||||
content_padding_addition: Range {
|
||||
min: 0,
|
||||
max: self.padding.into(),
|
||||
},
|
||||
random_trailers: true,
|
||||
mtu: 1050,
|
||||
}
|
||||
}
|
||||
/// Masks one standard WireGuard message; invalid types and oversized packets are refused.
|
||||
pub fn encode(&self, raw: &[u8]) -> Option<Bytes> {
|
||||
let kind = match raw.get(..4)? {
|
||||
[1, 0, 0, 0] => AwgKind::Initiation,
|
||||
[2, 0, 0, 0] => AwgKind::Response,
|
||||
[3, 0, 0, 0] => AwgKind::Cookie,
|
||||
[4, 0, 0, 0] => AwgKind::Transport,
|
||||
_ => return None,
|
||||
};
|
||||
if raw.len() < kind.fixed_size() || raw.len() > 1050 {
|
||||
return None;
|
||||
}
|
||||
let config = self.config();
|
||||
let range = match kind {
|
||||
AwgKind::Initiation => config.h1,
|
||||
AwgKind::Response => config.h2,
|
||||
AwgKind::Cookie => config.h3,
|
||||
AwgKind::Transport => config.h4,
|
||||
};
|
||||
let mut payload = raw.to_vec();
|
||||
payload[..4].copy_from_slice(&range.random().to_le_bytes());
|
||||
Some(config.encode(kind, Bytes::from(payload), 1200))
|
||||
}
|
||||
/// Unmasks a packet and restores standard headers before WireGuard validation.
|
||||
pub fn decode(&self, wire: &[u8]) -> Option<Vec<u8>> {
|
||||
if wire.len() > 1200 {
|
||||
return None;
|
||||
}
|
||||
let decoded = self.config().decode(&Bytes::copy_from_slice(wire))?;
|
||||
let (kind, len) = match decoded.kind {
|
||||
AwgKind::Initiation => (1, 148),
|
||||
AwgKind::Response => (2, 92),
|
||||
AwgKind::Cookie => (3, 64),
|
||||
AwgKind::Transport => (4, decoded.message.len()),
|
||||
};
|
||||
let mut message = decoded.message.get(..len)?.to_vec();
|
||||
message[..4].copy_from_slice(&(kind as u32).to_le_bytes());
|
||||
Some(message)
|
||||
}
|
||||
pub(crate) fn pad(&self, packet: Bytes) -> Bytes {
|
||||
self.config().prepare_transport_plaintext(packet, 1200)
|
||||
}
|
||||
/// Generates structured signature packets with fresh fields, never static captures.
|
||||
/// These target the other Tsunagi peer, not DNS or SIP services.
|
||||
pub fn signature_packets(&self) -> Vec<Vec<u8>> {
|
||||
if self.signature == 0 {
|
||||
// A complete DNS query. Counts, label lengths and qtype remain valid.
|
||||
let label = format!("n{:016x}", rand::random::<u64>());
|
||||
let mut packet = Vec::from(rand::random::<u16>().to_be_bytes());
|
||||
packet.extend_from_slice(&[1, 0, 0, 1, 0, 0, 0, 0, 0, 0]);
|
||||
packet.push(label.len() as u8);
|
||||
packet.extend_from_slice(label.as_bytes());
|
||||
packet.extend_from_slice(b"\x07example\x03net\0\0");
|
||||
packet.push(if rand::random() { 1 } else { 28 });
|
||||
packet.extend_from_slice(&[0, 1]);
|
||||
vec![packet]
|
||||
} else {
|
||||
let id = rand::random::<u64>();
|
||||
vec![format!("OPTIONS sip:n{id:x}@example.net SIP/2.0\r\nVia: SIP/2.0/UDP example.net;branch=z9hG4bK{id:x}\r\nMax-Forwards: 70\r\nFrom: <sip:a@example.net>;tag={id:x}\r\nTo: <sip:n{id:x}@example.net>\r\nCall-ID: {id:x}@example.net\r\nCSeq: 1 OPTIONS\r\nContent-Length: 0\r\n\r\n").into_bytes()]
|
||||
}
|
||||
}
|
||||
pub(crate) fn junk_packets(&self) -> Vec<Vec<u8>> {
|
||||
(0..self.junk_count)
|
||||
.map(|_| {
|
||||
let mut packet = vec![0; rand::random_range(self.junk_min..=self.junk_max).into()];
|
||||
rand::fill(&mut packet[..]);
|
||||
packet
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
|
||||
use super::*;
|
||||
#[test]
|
||||
fn unsafe_peer_profiles_are_rejected_before_the_codec() {
|
||||
let valid = PairProfile::generate();
|
||||
let mut bad = valid.clone();
|
||||
bad.prefix = 11;
|
||||
assert!(bad.validate().is_err());
|
||||
let mut bad = valid.clone();
|
||||
bad.headers[1] = bad.headers[0];
|
||||
assert!(bad.validate().is_err());
|
||||
let mut bad = valid.clone();
|
||||
bad.junk_max = u16::MAX;
|
||||
assert!(bad.validate().is_err());
|
||||
let mut bad = valid.clone();
|
||||
bad.padding = u16::MAX;
|
||||
assert!(bad.validate().is_err());
|
||||
let mut bad = valid.clone();
|
||||
bad.psk = [0; 32];
|
||||
assert!(bad.validate().is_err());
|
||||
let mut bad = valid.clone();
|
||||
bad.signature = 2;
|
||||
assert!(bad.validate().is_err());
|
||||
}
|
||||
#[test]
|
||||
fn signature_templates_have_consistent_framing_and_fresh_fields() {
|
||||
let mut profile = PairProfile::generate();
|
||||
profile.signature = 0;
|
||||
let dns = profile.signature_packets().remove(0);
|
||||
assert_eq!(&dns[2..12], &[1, 0, 0, 1, 0, 0, 0, 0, 0, 0]);
|
||||
let mut at = 12;
|
||||
while dns[at] != 0 {
|
||||
at += 1 + usize::from(dns[at]);
|
||||
}
|
||||
assert_eq!(at + 5, dns.len());
|
||||
assert_eq!(&dns[dns.len() - 2..], &[0, 1]);
|
||||
profile.signature = 1;
|
||||
let sip = profile.signature_packets().remove(0);
|
||||
assert!(sip.starts_with(b"OPTIONS sip:"));
|
||||
assert!(sip.ends_with(b"Content-Length: 0\r\n\r\n"));
|
||||
assert_ne!(sip, profile.signature_packets().remove(0));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,402 @@
|
||||
//! One cancellable actor and UDP socket per pair. All payload and health frames
|
||||
//! are inside the Amnezia-masked authenticated WireGuard tunnel.
|
||||
use crate::profile::PairProfile;
|
||||
use boringtun::noise::{Tunn, TunnResult};
|
||||
use bytes::Bytes;
|
||||
use iroh::EndpointId;
|
||||
use std::collections::{HashMap, VecDeque};
|
||||
use std::net::{Ipv4Addr, SocketAddr};
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::sync::{Arc, Mutex};
|
||||
use std::time::{Duration, Instant};
|
||||
use tokio::net::UdpSocket;
|
||||
use tokio::sync::{mpsc, watch};
|
||||
use tsunagi::dataplane::transport::{PacketLink, TransportError};
|
||||
use tsunagi::{BoxFuture, NetworkId};
|
||||
|
||||
pub(crate) const MAX_PAYLOAD: usize = tsunagi::config::MAX_DATA_DATAGRAM;
|
||||
const CHUNK: usize = 700;
|
||||
const QUEUE: usize = 64;
|
||||
const WINDOW: Duration = Duration::from_secs(20);
|
||||
|
||||
pub(crate) struct PairLink {
|
||||
network: NetworkId,
|
||||
peer: EndpointId,
|
||||
pub(crate) attempt: [u8; 16],
|
||||
tx: mpsc::Sender<Bytes>,
|
||||
rx: tokio::sync::Mutex<mpsc::Receiver<Bytes>>,
|
||||
stop: watch::Sender<bool>,
|
||||
closed: AtomicBool,
|
||||
remote: Mutex<Option<SocketAddr>>,
|
||||
reason: Mutex<Option<String>>,
|
||||
}
|
||||
impl std::fmt::Debug for PairLink {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
f.debug_struct("AwgPair")
|
||||
.field("network", &self.network)
|
||||
.field("peer", &self.peer)
|
||||
.finish_non_exhaustive()
|
||||
}
|
||||
}
|
||||
impl PairLink {
|
||||
pub(crate) fn has_authenticated_path(&self) -> bool {
|
||||
match self.remote.lock() {
|
||||
Ok(remote) => remote.is_some(),
|
||||
Err(_) => false,
|
||||
}
|
||||
}
|
||||
}
|
||||
impl PacketLink for PairLink {
|
||||
fn network(&self) -> NetworkId {
|
||||
self.network
|
||||
}
|
||||
fn peer(&self) -> EndpointId {
|
||||
self.peer
|
||||
}
|
||||
fn max_datagram_size(&self) -> usize {
|
||||
MAX_PAYLOAD
|
||||
}
|
||||
fn send(&self, payload: Bytes) -> Result<(), TransportError> {
|
||||
if self.is_closed() {
|
||||
return Err(TransportError::Closed);
|
||||
}
|
||||
if payload.len() > MAX_PAYLOAD {
|
||||
return Err(TransportError::TooLarge {
|
||||
size: payload.len(),
|
||||
limit: MAX_PAYLOAD,
|
||||
});
|
||||
}
|
||||
self.tx
|
||||
.try_send(payload)
|
||||
.map_err(|_| TransportError::Other("AWG send queue full or closed".into()))
|
||||
}
|
||||
fn recv(&self) -> BoxFuture<'_, Option<Bytes>> {
|
||||
Box::pin(async move {
|
||||
let mut stop = self.stop.subscribe();
|
||||
if *stop.borrow() {
|
||||
return None;
|
||||
}
|
||||
let mut rx = self.rx.lock().await;
|
||||
tokio::select! {biased; _=stop.changed()=>None,packet=rx.recv()=>packet}
|
||||
})
|
||||
}
|
||||
fn closed(&self) -> BoxFuture<'_, ()> {
|
||||
Box::pin(async move {
|
||||
let mut stop = self.stop.subscribe();
|
||||
if !*stop.borrow() {
|
||||
let _ = stop.changed().await;
|
||||
}
|
||||
})
|
||||
}
|
||||
fn is_closed(&self) -> bool {
|
||||
self.closed.load(Ordering::Acquire) || *self.stop.borrow()
|
||||
}
|
||||
fn close(&self) {
|
||||
self.stop.send_replace(true);
|
||||
}
|
||||
fn failure_reason(&self) -> Option<String> {
|
||||
match self.reason.lock() {
|
||||
Ok(reason) => reason.clone(),
|
||||
Err(_) => None,
|
||||
}
|
||||
}
|
||||
fn path_description(&self) -> String {
|
||||
match self.remote.lock() {
|
||||
Ok(addr) => match *addr {
|
||||
Some(addr) => format!("awg UDP via {addr}"),
|
||||
None => "awg UDP, establishing".into(),
|
||||
},
|
||||
Err(_) => "awg UDP".into(),
|
||||
}
|
||||
}
|
||||
}
|
||||
impl Drop for PairLink {
|
||||
fn drop(&mut self) {
|
||||
self.stop.send_replace(true);
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn start(
|
||||
network: NetworkId,
|
||||
peer: EndpointId,
|
||||
socket: UdpSocket,
|
||||
tunn: Tunn,
|
||||
profile: PairProfile,
|
||||
candidates: Vec<SocketAddr>,
|
||||
parent: watch::Receiver<bool>,
|
||||
) -> Arc<PairLink> {
|
||||
let (tx, rx) = mpsc::channel(QUEUE);
|
||||
let (in_tx, in_rx) = mpsc::channel(QUEUE);
|
||||
let (stop, _) = watch::channel(false);
|
||||
let link = Arc::new(PairLink {
|
||||
network,
|
||||
peer,
|
||||
attempt: profile.id(),
|
||||
tx,
|
||||
rx: tokio::sync::Mutex::new(in_rx),
|
||||
stop: stop.clone(),
|
||||
closed: AtomicBool::new(false),
|
||||
remote: Mutex::new(None),
|
||||
reason: Mutex::new(None),
|
||||
});
|
||||
let weak = Arc::downgrade(&link);
|
||||
tokio::spawn(async move {
|
||||
let mut engine = Engine {
|
||||
stop: stop.subscribe(),
|
||||
socket,
|
||||
tunn,
|
||||
profile,
|
||||
candidates,
|
||||
remote: None,
|
||||
inbox: in_tx,
|
||||
rx,
|
||||
counter: rand::random(),
|
||||
sequence: 0,
|
||||
pending: VecDeque::new(),
|
||||
samples: VecDeque::new(),
|
||||
assemblies: HashMap::new(),
|
||||
started: Instant::now(),
|
||||
last_authenticated: Instant::now(),
|
||||
};
|
||||
engine.run(parent, &weak).await;
|
||||
if let Some(link) = weak.upgrade() {
|
||||
link.closed.store(true, Ordering::Release);
|
||||
link.stop.send_replace(true);
|
||||
}
|
||||
});
|
||||
link
|
||||
}
|
||||
|
||||
struct Assembly {
|
||||
created: Instant,
|
||||
count: u16,
|
||||
chunks: Vec<Option<Bytes>>,
|
||||
}
|
||||
struct Engine {
|
||||
stop: watch::Receiver<bool>,
|
||||
socket: UdpSocket,
|
||||
tunn: Tunn,
|
||||
profile: PairProfile,
|
||||
candidates: Vec<SocketAddr>,
|
||||
remote: Option<SocketAddr>,
|
||||
inbox: mpsc::Sender<Bytes>,
|
||||
rx: mpsc::Receiver<Bytes>,
|
||||
counter: u64,
|
||||
sequence: u64,
|
||||
pending: VecDeque<(u64, Instant)>,
|
||||
samples: VecDeque<bool>,
|
||||
assemblies: HashMap<u64, Assembly>,
|
||||
started: Instant,
|
||||
last_authenticated: Instant,
|
||||
}
|
||||
enum ResultOwned {
|
||||
Wire(Vec<u8>),
|
||||
Plain(Vec<u8>),
|
||||
Done,
|
||||
}
|
||||
fn own(result: TunnResult<'_>) -> ResultOwned {
|
||||
match result {
|
||||
TunnResult::WriteToNetwork(p) => ResultOwned::Wire(p.to_vec()),
|
||||
TunnResult::WriteToTunnelV4(p, _) => ResultOwned::Plain(p.to_vec()),
|
||||
_ => ResultOwned::Done,
|
||||
}
|
||||
}
|
||||
fn ip(payload: &[u8]) -> Bytes {
|
||||
let mut packet = vec![0; 20];
|
||||
packet[0] = 0x45;
|
||||
packet[8] = 64;
|
||||
packet[9] = 253;
|
||||
packet[12..16].copy_from_slice(&Ipv4Addr::new(192, 0, 2, 1).octets());
|
||||
packet[16..20].copy_from_slice(&Ipv4Addr::new(192, 0, 2, 2).octets());
|
||||
packet[2..4].copy_from_slice(&((20 + payload.len()) as u16).to_be_bytes());
|
||||
let sum = packet
|
||||
.as_chunks::<2>()
|
||||
.0
|
||||
.iter()
|
||||
.map(|b| u32::from(u16::from_be_bytes([b[0], b[1]])))
|
||||
.sum::<u32>();
|
||||
let sum = (sum & 0xffff) + (sum >> 16);
|
||||
packet[10..12].copy_from_slice(&(!(sum as u16)).to_be_bytes());
|
||||
packet.extend_from_slice(payload);
|
||||
Bytes::from(packet)
|
||||
}
|
||||
impl Engine {
|
||||
async fn run(&mut self, mut stop: watch::Receiver<bool>, link: &std::sync::Weak<PairLink>) {
|
||||
if *self.stop.borrow() || *stop.borrow() {
|
||||
return;
|
||||
}
|
||||
let mut buffer = [0; 1201];
|
||||
let mut scratch = [0; 4096];
|
||||
let mut timer = tokio::time::interval(Duration::from_millis(250));
|
||||
timer.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
|
||||
let mut last_probe = Instant::now() - Duration::from_secs(2);
|
||||
loop {
|
||||
tokio::select! {
|
||||
biased;
|
||||
_=stop.changed()=>break,
|
||||
_=self.stop.changed()=>break,
|
||||
_=timer.tick()=>{
|
||||
let result=own(self.tunn.update_timers(&mut scratch));self.process(result,None).await;
|
||||
if last_probe.elapsed()>=Duration::from_secs(1) {
|
||||
last_probe=Instant::now();self.sequence=self.sequence.wrapping_add(1);
|
||||
self.pending.push_back((self.sequence,Instant::now()));
|
||||
let mut probe=vec![1];probe.extend_from_slice(&self.sequence.to_be_bytes());
|
||||
probe.resize(713,0);rand::fill(&mut probe[9..]);self.encrypt(&probe).await;
|
||||
}
|
||||
while self.pending.front().is_some_and(|(_,at)|at.elapsed()>=Duration::from_secs(3)) {
|
||||
self.pending.pop_front();self.samples.push_back(false);
|
||||
}
|
||||
while self.samples.len()>20 {self.samples.pop_front();}
|
||||
self.assemblies.retain(|_,a|a.created.elapsed()<Duration::from_secs(5));
|
||||
if self.started.elapsed()>WINDOW && (self.last_authenticated.elapsed()>Duration::from_secs(10)
|
||||
|| (self.samples.len()>=15&&self.samples.iter().filter(|ok|!**ok).count()*100>=self.samples.len()*60)) {
|
||||
if let Some(link)=link.upgrade() && let Ok(mut reason)=link.reason.lock() {*reason=Some("AWG data delivery checks: persistent loss or timeout".into());}
|
||||
break;
|
||||
}
|
||||
}
|
||||
packet=self.socket.recv_from(&mut buffer)=>{
|
||||
if let Ok((len,source))=packet && let Some(message)=self.profile.decode(&buffer[..len]) {
|
||||
let result=own(self.tunn.decapsulate(None,&message,&mut scratch));
|
||||
if matches!(result,ResultOwned::Plain(_)) {
|
||||
self.remote=Some(source);self.last_authenticated=Instant::now();
|
||||
if let Some(link)=link.upgrade() && let Ok(mut remote)=link.remote.lock(){*remote=Some(source);}
|
||||
}
|
||||
self.process(result,Some(source)).await;
|
||||
for _ in 0..16 {
|
||||
let result=own(self.tunn.decapsulate(None,&[],&mut scratch));
|
||||
if matches!(result,ResultOwned::Done){break;}
|
||||
self.process(result,Some(source)).await;
|
||||
}
|
||||
}
|
||||
}
|
||||
packet=self.rx.recv()=>{
|
||||
let Some(packet)=packet else{break;};
|
||||
let count=packet.len().max(1).div_ceil(CHUNK) as u16;
|
||||
self.counter=self.counter.wrapping_add(1);
|
||||
for (index,chunk) in packet.chunks(CHUNK).enumerate() {
|
||||
let mut payload=vec![3];payload.extend_from_slice(&self.counter.to_be_bytes());
|
||||
payload.extend_from_slice(&(index as u16).to_be_bytes());payload.extend_from_slice(&count.to_be_bytes());
|
||||
payload.extend_from_slice(chunk);self.encrypt(&payload).await;
|
||||
// Large logical datagrams must let local readers and other
|
||||
// pairs drain their bounded socket/command queues.
|
||||
if (index + 1).is_multiple_of(16) {tokio::task::yield_now().await;}
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
}
|
||||
}
|
||||
async fn encrypt(&mut self, payload: &[u8]) {
|
||||
let packet = self.profile.pad(ip(payload));
|
||||
let mut scratch = [0; 4096];
|
||||
let result = own(self.tunn.encapsulate(&packet, &mut scratch));
|
||||
self.process(result, None).await;
|
||||
}
|
||||
async fn wire(&self, raw: &[u8], target: Option<SocketAddr>) {
|
||||
let Some(wire) = self.profile.encode(raw) else {
|
||||
return;
|
||||
};
|
||||
let targets: Vec<_> = target
|
||||
.or(self.remote)
|
||||
.map(|a| vec![a])
|
||||
.unwrap_or_else(|| self.candidates.clone());
|
||||
for addr in targets {
|
||||
if raw.first() == Some(&1) {
|
||||
for signature in self.profile.signature_packets() {
|
||||
let _ = self.send_wire(&signature, addr).await;
|
||||
}
|
||||
for junk in self.profile.junk_packets() {
|
||||
let _ = self.send_wire(&junk, addr).await;
|
||||
}
|
||||
}
|
||||
let _ = self.send_wire(&wire, addr).await;
|
||||
}
|
||||
}
|
||||
async fn send_wire(&self, bytes: &[u8], address: SocketAddr) {
|
||||
let mut stop = self.stop.clone();
|
||||
if *stop.borrow() {
|
||||
return;
|
||||
}
|
||||
tokio::select! {biased; _=stop.changed()=>{}, _=tokio::time::timeout(Duration::from_millis(100),self.socket.send_to(bytes,address))=>{}}
|
||||
}
|
||||
async fn process(&mut self, result: ResultOwned, target: Option<SocketAddr>) {
|
||||
match result {
|
||||
ResultOwned::Wire(raw) => self.wire(&raw, target).await,
|
||||
ResultOwned::Plain(packet) => {
|
||||
let Some(payload) = packet.get(20..) else {
|
||||
return;
|
||||
};
|
||||
let Some(seq) = payload
|
||||
.get(1..9)
|
||||
.and_then(|b| b.try_into().ok())
|
||||
.map(u64::from_be_bytes)
|
||||
else {
|
||||
return;
|
||||
};
|
||||
match payload.first() {
|
||||
Some(1) => {
|
||||
let mut pong = vec![2];
|
||||
pong.extend_from_slice(&seq.to_be_bytes());
|
||||
pong.resize(713, 0);
|
||||
let packet = self.profile.pad(ip(&pong));
|
||||
let mut scratch = [0; 4096];
|
||||
if let ResultOwned::Wire(raw) =
|
||||
own(self.tunn.encapsulate(&packet, &mut scratch))
|
||||
{
|
||||
self.wire(&raw, target).await;
|
||||
}
|
||||
}
|
||||
Some(2) => {
|
||||
if let Some(index) =
|
||||
self.pending.iter().position(|(pending, _)| *pending == seq)
|
||||
{
|
||||
self.pending.remove(index);
|
||||
self.samples.push_back(true);
|
||||
}
|
||||
}
|
||||
Some(3) => self.fragment(seq, payload),
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
ResultOwned::Done => {}
|
||||
}
|
||||
}
|
||||
fn fragment(&mut self, id: u64, payload: &[u8]) {
|
||||
let Some(header) = payload.get(9..13) else {
|
||||
return;
|
||||
};
|
||||
let index = u16::from_be_bytes([header[0], header[1]]);
|
||||
let count = u16::from_be_bytes([header[2], header[3]]);
|
||||
let chunk = &payload[13..];
|
||||
if count == 0
|
||||
|| usize::from(count) > MAX_PAYLOAD.div_ceil(CHUNK)
|
||||
|| index >= count
|
||||
|| chunk.len() > CHUNK
|
||||
{
|
||||
return;
|
||||
}
|
||||
if !self.assemblies.contains_key(&id) && self.assemblies.len() >= 8 {
|
||||
return;
|
||||
}
|
||||
let assembly = self.assemblies.entry(id).or_insert_with(|| Assembly {
|
||||
created: Instant::now(),
|
||||
count,
|
||||
chunks: vec![None; count.into()],
|
||||
});
|
||||
if assembly.count != count {
|
||||
return;
|
||||
}
|
||||
assembly.chunks[usize::from(index)] = Some(Bytes::copy_from_slice(chunk));
|
||||
if assembly.chunks.iter().all(Option::is_some)
|
||||
&& let Some(assembly) = self.assemblies.remove(&id)
|
||||
{
|
||||
let mut packet = Vec::new();
|
||||
for chunk in assembly.chunks.into_iter().flatten() {
|
||||
packet.extend_from_slice(&chunk);
|
||||
}
|
||||
if packet.len() <= MAX_PAYLOAD {
|
||||
let _ = self.inbox.try_send(Bytes::from(packet));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,101 @@
|
||||
//! Optional, bounded server-reflexive discovery on the pair's own UDP socket.
|
||||
//! A STUN answer is a candidate only; WireGuard authentication verifies a path.
|
||||
use std::net::{IpAddr, Ipv4Addr, SocketAddr};
|
||||
use std::time::Duration;
|
||||
use tokio::net::UdpSocket;
|
||||
const COOKIE: [u8; 4] = [0x21, 0x12, 0xa4, 0x42];
|
||||
pub(crate) async fn mapped(socket: &UdpSocket, server: &str) -> Option<SocketAddr> {
|
||||
tokio::time::timeout(Duration::from_millis(1200), async {
|
||||
let address = tokio::net::lookup_host(server)
|
||||
.await
|
||||
.ok()?
|
||||
.find(SocketAddr::is_ipv4)?;
|
||||
let transaction: [u8; 12] = rand::random();
|
||||
let mut request = [0; 20];
|
||||
request[..2].copy_from_slice(&[0, 1]);
|
||||
request[4..8].copy_from_slice(&COOKIE);
|
||||
request[8..].copy_from_slice(&transaction);
|
||||
socket.send_to(&request, address).await.ok()?;
|
||||
let mut bytes = [0; 1024];
|
||||
loop {
|
||||
let (len, source) = socket.recv_from(&mut bytes).await.ok()?;
|
||||
if source == address
|
||||
&& let Some(mapped) = parse(&bytes[..len], &transaction)
|
||||
{
|
||||
return Some(mapped);
|
||||
}
|
||||
}
|
||||
})
|
||||
.await
|
||||
.ok()
|
||||
.flatten()
|
||||
}
|
||||
fn parse(bytes: &[u8], transaction: &[u8; 12]) -> Option<SocketAddr> {
|
||||
if bytes.len() < 20
|
||||
|| bytes[..2] != [1, 1]
|
||||
|| bytes[4..8] != COOKIE
|
||||
|| &bytes[8..20] != transaction
|
||||
{
|
||||
return None;
|
||||
}
|
||||
let declared = u16::from_be_bytes(bytes[2..4].try_into().ok()?) as usize;
|
||||
if !declared.is_multiple_of(4) || declared != bytes.len() - 20 {
|
||||
return None;
|
||||
}
|
||||
let mut at = 20;
|
||||
while at + 4 <= bytes.len() {
|
||||
let kind = u16::from_be_bytes(bytes[at..at + 2].try_into().ok()?);
|
||||
let len = u16::from_be_bytes(bytes[at + 2..at + 4].try_into().ok()?) as usize;
|
||||
at += 4;
|
||||
let attribute = bytes.get(at..at.checked_add(len)?)?;
|
||||
if kind == 0x0020 && attribute.len() == 8 && attribute[0..2] == [0, 1] {
|
||||
let port = u16::from_be_bytes([attribute[2] ^ COOKIE[0], attribute[3] ^ COOKIE[1]]);
|
||||
let ip = Ipv4Addr::new(
|
||||
attribute[4] ^ COOKIE[0],
|
||||
attribute[5] ^ COOKIE[1],
|
||||
attribute[6] ^ COOKIE[2],
|
||||
attribute[7] ^ COOKIE[3],
|
||||
);
|
||||
if port == 0 || ip.is_unspecified() || ip.is_multicast() || ip.is_broadcast() {
|
||||
return None;
|
||||
}
|
||||
return Some(SocketAddr::new(IpAddr::V4(ip), port));
|
||||
}
|
||||
at = at.checked_add((len + 3) & !3)?;
|
||||
}
|
||||
None
|
||||
}
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
|
||||
use super::*;
|
||||
#[tokio::test]
|
||||
async fn mapping_uses_the_actual_socket_and_rejects_unrelated_or_truncated_answers() {
|
||||
let server = UdpSocket::bind((Ipv4Addr::LOCALHOST, 0)).await.unwrap();
|
||||
let address = server.local_addr().unwrap();
|
||||
let socket = UdpSocket::bind((Ipv4Addr::LOCALHOST, 0)).await.unwrap();
|
||||
let expected = socket.local_addr().unwrap();
|
||||
let task = tokio::spawn(async move {
|
||||
let mut request = [0; 128];
|
||||
let (len, source) = server.recv_from(&mut request).await.unwrap();
|
||||
assert_eq!(len, 20);
|
||||
assert_eq!(source, expected);
|
||||
let tx: [u8; 12] = request[8..20].try_into().unwrap();
|
||||
let mut reply = request[..20].to_vec();
|
||||
reply[..4].copy_from_slice(&[1, 1, 0, 12]);
|
||||
reply.extend_from_slice(&[0, 0x20, 0, 8, 0, 1]);
|
||||
let port = source.port().to_be_bytes();
|
||||
reply.extend_from_slice(&[port[0] ^ COOKIE[0], port[1] ^ COOKIE[1]]);
|
||||
for (i, b) in Ipv4Addr::LOCALHOST.octets().into_iter().enumerate() {
|
||||
reply.push(b ^ COOKIE[i]);
|
||||
}
|
||||
assert!(parse(&reply, &[0; 12]).is_none());
|
||||
for len in 0..reply.len() {
|
||||
assert!(parse(&reply[..len], &tx).is_none());
|
||||
}
|
||||
server.send_to(&reply, source).await.unwrap();
|
||||
});
|
||||
assert_eq!(mapped(&socket, &address.to_string()).await, Some(expected));
|
||||
task.await.unwrap();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,720 @@
|
||||
//! Private pair negotiation. No secret profiles in capabilities or durable state.
|
||||
use crate::{
|
||||
AWG_PROTOCOL, AWG_WIRE_VERSION,
|
||||
profile::PairProfile,
|
||||
session::{self, PairLink},
|
||||
};
|
||||
use boringtun::{
|
||||
noise::Tunn,
|
||||
x25519::{PublicKey, StaticSecret},
|
||||
};
|
||||
use iroh::EndpointId;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::collections::{HashMap, HashSet};
|
||||
use std::net::{Ipv4Addr, SocketAddr};
|
||||
use std::sync::{Arc, Mutex, OnceLock, Weak};
|
||||
use std::time::Duration;
|
||||
use tokio::net::UdpSocket;
|
||||
use tokio::sync::{mpsc, oneshot, watch};
|
||||
use tsunagi::dataplane::transport::TransportPayload;
|
||||
use tsunagi::dataplane::transport::{
|
||||
ControlSender, InboundLink, PacketLink, PacketTransport, SharedLink, TransportError,
|
||||
};
|
||||
use tsunagi::{BoxFuture, NetworkId};
|
||||
|
||||
type Pair = (NetworkId, EndpointId);
|
||||
#[derive(Serialize, Deserialize)]
|
||||
enum Negotiation {
|
||||
Offer {
|
||||
version: u16,
|
||||
profile: PairProfile,
|
||||
key: [u8; 32],
|
||||
#[serde(deserialize_with = "bounded_addresses")]
|
||||
addresses: Vec<SocketAddr>,
|
||||
},
|
||||
Answer {
|
||||
attempt: [u8; 16],
|
||||
key: [u8; 32],
|
||||
#[serde(deserialize_with = "bounded_addresses")]
|
||||
addresses: Vec<SocketAddr>,
|
||||
},
|
||||
}
|
||||
fn bounded_addresses<'de, D: serde::Deserializer<'de>>(
|
||||
deserializer: D,
|
||||
) -> Result<Vec<SocketAddr>, D::Error> {
|
||||
struct Visitor;
|
||||
impl<'de> serde::de::Visitor<'de> for Visitor {
|
||||
type Value = Vec<SocketAddr>;
|
||||
fn expecting(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
f.write_str("at most 32 UDP candidates")
|
||||
}
|
||||
fn visit_seq<A: serde::de::SeqAccess<'de>>(
|
||||
self,
|
||||
mut sequence: A,
|
||||
) -> Result<Self::Value, A::Error> {
|
||||
if sequence.size_hint().is_some_and(|n| n > 32) {
|
||||
return Err(serde::de::Error::custom("too many UDP candidates"));
|
||||
}
|
||||
let mut addresses = Vec::with_capacity(sequence.size_hint().unwrap_or(0).min(32));
|
||||
while let Some(address) = sequence.next_element()? {
|
||||
if addresses.len() == 32 {
|
||||
return Err(serde::de::Error::custom("too many UDP candidates"));
|
||||
}
|
||||
addresses.push(address);
|
||||
}
|
||||
Ok(addresses)
|
||||
}
|
||||
}
|
||||
deserializer.deserialize_seq(Visitor)
|
||||
}
|
||||
struct Pending {
|
||||
profile: PairProfile,
|
||||
key: StaticSecret,
|
||||
socket: UdpSocket,
|
||||
reply: oneshot::Sender<Result<SharedLink, TransportError>>,
|
||||
}
|
||||
struct Inner {
|
||||
stopped: watch::Sender<bool>,
|
||||
local: EndpointId,
|
||||
bind: Ipv4Addr,
|
||||
stun: Option<String>,
|
||||
busy: Mutex<HashSet<Pair>>,
|
||||
control: OnceLock<Arc<dyn ControlSender>>,
|
||||
allowed: Mutex<HashSet<Pair>>,
|
||||
pending: Mutex<HashMap<Pair, Pending>>,
|
||||
links: Mutex<HashMap<Pair, Arc<PairLink>>>,
|
||||
inbound: mpsc::Sender<InboundLink>,
|
||||
port_by_pair: Mutex<HashMap<Pair, u16>>,
|
||||
}
|
||||
/// Public diagnostics for one pair. No key or masking parameters are exposed.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct PairOverview {
|
||||
/// Other endpoint's authenticated identity.
|
||||
pub peer: EndpointId,
|
||||
/// Fresh connection-attempt identifier.
|
||||
pub attempt: [u8; 16],
|
||||
/// Last known UDP port; None if diagnostics race with pair retirement.
|
||||
pub local_port: Option<u16>,
|
||||
/// Actual transport path, if established.
|
||||
pub path: String,
|
||||
/// Whether this attempt has been retired.
|
||||
pub closed: bool,
|
||||
}
|
||||
/// A bounded set of independently configured pair engines.
|
||||
pub struct AwgTransport {
|
||||
inner: Arc<Inner>,
|
||||
commands: mpsc::Sender<(NetworkId, EndpointId, Negotiation)>,
|
||||
incoming: tokio::sync::Mutex<mpsc::Receiver<InboundLink>>,
|
||||
}
|
||||
impl std::fmt::Debug for AwgTransport {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
f.debug_struct("AwgTransport")
|
||||
.field("local", &self.inner.local)
|
||||
.finish_non_exhaustive()
|
||||
}
|
||||
}
|
||||
fn lock<T>(mutex: &Mutex<T>) -> std::sync::MutexGuard<'_, T> {
|
||||
match mutex.lock() {
|
||||
Ok(g) => g,
|
||||
Err(p) => p.into_inner(),
|
||||
}
|
||||
}
|
||||
fn error(text: &str) -> TransportError {
|
||||
TransportError::Other(text.into())
|
||||
}
|
||||
async fn addresses(inner: &Inner, socket: &UdpSocket) -> Result<Vec<SocketAddr>, TransportError> {
|
||||
let local = socket
|
||||
.local_addr()
|
||||
.map_err(|_| error("AWG socket address unavailable"))?;
|
||||
let mut candidates = if inner.bind.is_loopback() {
|
||||
vec![local]
|
||||
} else {
|
||||
tsunagi_wireguard::local_ip_candidates()
|
||||
.into_iter()
|
||||
.filter(|ip| ip.is_ipv4())
|
||||
.take(31)
|
||||
.map(|ip| SocketAddr::new(ip, local.port()))
|
||||
.collect()
|
||||
};
|
||||
if let Some(server) = &inner.stun
|
||||
&& let Some(mapped) = crate::stun::mapped(socket, server).await
|
||||
&& !candidates.contains(&mapped)
|
||||
{
|
||||
candidates.push(mapped);
|
||||
}
|
||||
if candidates.is_empty() {
|
||||
return Err(error("AWG has no UDP address candidates"));
|
||||
}
|
||||
Ok(candidates)
|
||||
}
|
||||
fn validate_endpoint(key: &[u8; 32], addresses: &[SocketAddr]) -> Result<(), TransportError> {
|
||||
if *key == [0; 32]
|
||||
|| addresses.is_empty()
|
||||
|| addresses.len() > 32
|
||||
|| addresses.iter().any(|address| {
|
||||
address.port() == 0
|
||||
|| !address.is_ipv4()
|
||||
|| address.ip().is_unspecified()
|
||||
|| address.ip().is_multicast()
|
||||
|| matches!(address.ip(), std::net::IpAddr::V4(ip) if ip.is_broadcast())
|
||||
})
|
||||
{
|
||||
return Err(error("invalid AWG endpoint"));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
impl AwgTransport {
|
||||
/// Starts the coordinator. Production binds wildcard IPv4; tests use loopback.
|
||||
pub fn new(local: EndpointId, bind: Ipv4Addr) -> Arc<Self> {
|
||||
Self::new_with_stun(local, bind, None)
|
||||
}
|
||||
/// Enables an optional STUN server (host:port) for this pair's own socket.
|
||||
/// Local-only callers pass None; failure still permits local UDP candidates.
|
||||
pub fn new_with_stun(local: EndpointId, bind: Ipv4Addr, stun: Option<String>) -> Arc<Self> {
|
||||
let (commands, mut rx) = mpsc::channel(64);
|
||||
let (inbound, incoming) = mpsc::channel(64);
|
||||
let (stopped, _) = watch::channel(false);
|
||||
let inner = Arc::new(Inner {
|
||||
stopped,
|
||||
local,
|
||||
bind,
|
||||
stun,
|
||||
busy: Mutex::default(),
|
||||
control: OnceLock::new(),
|
||||
allowed: Mutex::default(),
|
||||
pending: Mutex::default(),
|
||||
links: Mutex::default(),
|
||||
inbound,
|
||||
port_by_pair: Mutex::default(),
|
||||
});
|
||||
let weak = Arc::downgrade(&inner);
|
||||
let mut stopped = inner.stopped.subscribe();
|
||||
tokio::spawn(async move {
|
||||
while let Some((network, peer, message)) =
|
||||
tokio::select! { biased; _=stopped.changed()=>None, message=rx.recv()=>message }
|
||||
{
|
||||
let Some(inner) = weak.upgrade() else {
|
||||
break;
|
||||
};
|
||||
if !lock(&inner.allowed).contains(&(network, peer))
|
||||
|| !lock(&inner.busy).insert((network, peer))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
tokio::spawn(async move {
|
||||
let mut stop = inner.stopped.subscribe();
|
||||
tokio::select! {biased; _=stop.changed()=>{}, _=async {
|
||||
if let Err(err)=receive(&inner,(network,peer),message).await {tracing::debug!(%err,"AWG pair negotiation rejected");}
|
||||
}=>{}}
|
||||
lock(&inner.busy).remove(&(network, peer));
|
||||
});
|
||||
}
|
||||
});
|
||||
Arc::new(Self {
|
||||
inner,
|
||||
commands,
|
||||
incoming: tokio::sync::Mutex::new(incoming),
|
||||
})
|
||||
}
|
||||
/// Registers a peer after its authenticated capability has been validated.
|
||||
pub fn allow_peer(&self, network: NetworkId, peer: EndpointId) {
|
||||
let mut allowed = lock(&self.inner.allowed);
|
||||
if !*self.inner.stopped.borrow() && allowed.len() < 128 {
|
||||
allowed.insert((network, peer));
|
||||
}
|
||||
}
|
||||
/// Retires a pair when its network membership session goes away.
|
||||
pub fn forget_peer(&self, network: NetworkId, peer: EndpointId) {
|
||||
let pair = (network, peer);
|
||||
lock(&self.inner.allowed).remove(&pair);
|
||||
lock(&self.inner.pending).remove(&pair);
|
||||
lock(&self.inner.port_by_pair).remove(&pair);
|
||||
if let Some(link) = lock(&self.inner.links).remove(&pair) {
|
||||
link.close();
|
||||
}
|
||||
}
|
||||
/// Stops all pair engines in this network.
|
||||
pub fn stop_network(&self, network: NetworkId) {
|
||||
let peers: Vec<_> = lock(&self.inner.allowed)
|
||||
.iter()
|
||||
.filter(|(n, _)| *n == network)
|
||||
.map(|(_, p)| *p)
|
||||
.collect();
|
||||
for peer in peers {
|
||||
self.forget_peer(network, peer);
|
||||
}
|
||||
}
|
||||
/// Stops all sockets and coordinator work without waiting for remote peers.
|
||||
pub fn shutdown(&self) {
|
||||
self.inner.stopped.send_replace(true);
|
||||
lock(&self.inner.allowed).clear();
|
||||
lock(&self.inner.pending).clear();
|
||||
for (_, link) in lock(&self.inner.links).drain() {
|
||||
link.close();
|
||||
}
|
||||
lock(&self.inner.port_by_pair).clear();
|
||||
}
|
||||
/// Receives a successfully negotiated and authenticated inbound pair link.
|
||||
pub async fn next_inbound(&self) -> Option<InboundLink> {
|
||||
let mut stopped = self.inner.stopped.subscribe();
|
||||
if *stopped.borrow() {
|
||||
return None;
|
||||
}
|
||||
let mut incoming = self.incoming.lock().await;
|
||||
tokio::select! {biased; _=stopped.changed()=>None, link=incoming.recv()=>link}
|
||||
}
|
||||
/// Takes a public snapshot of this network's pair attempts.
|
||||
pub fn overview(&self, network: NetworkId) -> Vec<PairOverview> {
|
||||
let links = lock(&self.inner.links);
|
||||
let ports = lock(&self.inner.port_by_pair);
|
||||
links
|
||||
.iter()
|
||||
.filter(|((n, _), _)| *n == network)
|
||||
.map(|((_, peer), link)| PairOverview {
|
||||
peer: *peer,
|
||||
attempt: link.attempt,
|
||||
local_port: ports.get(&(network, *peer)).copied(),
|
||||
path: link.path_description(),
|
||||
closed: link.is_closed(),
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
}
|
||||
impl PacketTransport for AwgTransport {
|
||||
fn name(&self) -> &str {
|
||||
AWG_PROTOCOL
|
||||
}
|
||||
fn as_any(&self) -> &dyn std::any::Any {
|
||||
self
|
||||
}
|
||||
fn attach_control(&self, control: Arc<dyn ControlSender>) {
|
||||
let _ = self.inner.control.set(control);
|
||||
}
|
||||
fn on_control(
|
||||
&self,
|
||||
network: NetworkId,
|
||||
peer: EndpointId,
|
||||
protocol: &str,
|
||||
payload: &[u8],
|
||||
) -> Result<(), TransportError> {
|
||||
if protocol != AWG_PROTOCOL || !lock(&self.inner.allowed).contains(&(network, peer)) {
|
||||
return Err(TransportError::Declined(protocol.into()));
|
||||
}
|
||||
if payload.len() > 4096 {
|
||||
return Err(error("AWG negotiation too large"));
|
||||
}
|
||||
let message: Negotiation =
|
||||
postcard::from_bytes(payload).map_err(|_| error("malformed AWG negotiation"))?;
|
||||
match &message {
|
||||
Negotiation::Offer {
|
||||
version,
|
||||
profile,
|
||||
key,
|
||||
addresses,
|
||||
} => {
|
||||
if *version != AWG_WIRE_VERSION || peer.as_bytes() >= self.inner.local.as_bytes() {
|
||||
return Err(error("invalid AWG offer role/version"));
|
||||
}
|
||||
profile.validate().map_err(error)?;
|
||||
validate_endpoint(key, addresses)?;
|
||||
}
|
||||
Negotiation::Answer { key, addresses, .. } => validate_endpoint(key, addresses)?,
|
||||
}
|
||||
self.commands
|
||||
.try_send((network, peer, message))
|
||||
.map_err(|_| error("AWG negotiation queue full"))
|
||||
}
|
||||
fn open<'a>(
|
||||
&'a self,
|
||||
network: NetworkId,
|
||||
peer: EndpointId,
|
||||
protocol: &'a str,
|
||||
) -> BoxFuture<'a, Result<SharedLink, TransportError>> {
|
||||
Box::pin(async move {
|
||||
let pair = (network, peer);
|
||||
if protocol != AWG_PROTOCOL
|
||||
|| self.inner.local.as_bytes() >= peer.as_bytes()
|
||||
|| !lock(&self.inner.allowed).contains(&pair)
|
||||
{
|
||||
return Err(TransportError::Declined(protocol.into()));
|
||||
}
|
||||
let control = self
|
||||
.inner
|
||||
.control
|
||||
.get()
|
||||
.ok_or_else(|| error("AWG control plane unavailable"))?;
|
||||
let socket = UdpSocket::bind((self.inner.bind, 0))
|
||||
.await
|
||||
.map_err(|_| error("AWG UDP bind failed"))?;
|
||||
let profile = PairProfile::generate();
|
||||
let attempt = profile.id();
|
||||
let key = StaticSecret::from(rand::random::<[u8; 32]>());
|
||||
let public = PublicKey::from(&key).to_bytes();
|
||||
let message = Negotiation::Offer {
|
||||
version: AWG_WIRE_VERSION,
|
||||
profile: profile.clone(),
|
||||
key: public,
|
||||
addresses: addresses(&self.inner, &socket).await?,
|
||||
};
|
||||
let (reply, rx) = oneshot::channel();
|
||||
if lock(&self.inner.pending).len() >= 128 {
|
||||
return Err(error("too many AWG pair negotiations"));
|
||||
}
|
||||
lock(&self.inner.pending).insert(
|
||||
pair,
|
||||
Pending {
|
||||
profile,
|
||||
key,
|
||||
socket,
|
||||
reply,
|
||||
},
|
||||
);
|
||||
// A cancellation guard also retires partially started attempts.
|
||||
let mut guard = AttemptGuard {
|
||||
inner: Arc::downgrade(&self.inner),
|
||||
pair,
|
||||
attempt,
|
||||
completed: false,
|
||||
};
|
||||
let bytes =
|
||||
postcard::to_stdvec(&message).map_err(|_| error("AWG offer encoding failed"))?;
|
||||
control
|
||||
.send(network, peer, AWG_PROTOCOL, TransportPayload::new(bytes))
|
||||
.await?;
|
||||
let result = tokio::time::timeout(Duration::from_secs(15), rx)
|
||||
.await
|
||||
.map_err(|_| error("AWG negotiation/handshake timed out"))?
|
||||
.map_err(|_| error("AWG attempt cancelled"))?;
|
||||
guard.completed = result.is_ok();
|
||||
result
|
||||
})
|
||||
}
|
||||
}
|
||||
struct AttemptGuard {
|
||||
inner: Weak<Inner>,
|
||||
pair: Pair,
|
||||
attempt: [u8; 16],
|
||||
completed: bool,
|
||||
}
|
||||
impl Drop for AttemptGuard {
|
||||
fn drop(&mut self) {
|
||||
if let Some(inner) = self.inner.upgrade() {
|
||||
let mut pending = lock(&inner.pending);
|
||||
if pending
|
||||
.get(&self.pair)
|
||||
.is_some_and(|p| p.profile.id() == self.attempt)
|
||||
{
|
||||
pending.remove(&self.pair);
|
||||
}
|
||||
drop(pending);
|
||||
if !self.completed
|
||||
&& let Some(link) = lock(&inner.links).get(&self.pair)
|
||||
&& link.attempt == self.attempt
|
||||
{
|
||||
link.close();
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
async fn receive(
|
||||
inner: &Arc<Inner>,
|
||||
pair: Pair,
|
||||
message: Negotiation,
|
||||
) -> Result<(), TransportError> {
|
||||
if !lock(&inner.allowed).contains(&pair) {
|
||||
return Err(error("AWG peer no longer authenticated"));
|
||||
}
|
||||
match message {
|
||||
Negotiation::Offer {
|
||||
profile,
|
||||
key,
|
||||
addresses: candidates,
|
||||
..
|
||||
} => {
|
||||
if let Some(link) = lock(&inner.links).get(&pair) {
|
||||
if link.attempt == profile.id() {
|
||||
return Err(error("duplicate AWG offer"));
|
||||
}
|
||||
link.close();
|
||||
}
|
||||
let socket = UdpSocket::bind((inner.bind, 0))
|
||||
.await
|
||||
.map_err(|_| error("AWG UDP bind failed"))?;
|
||||
let local_port = socket
|
||||
.local_addr()
|
||||
.map_err(|_| error("AWG socket address unavailable"))?
|
||||
.port();
|
||||
let local_key = StaticSecret::from(rand::random::<[u8; 32]>());
|
||||
let public = PublicKey::from(&local_key).to_bytes();
|
||||
let answer = Negotiation::Answer {
|
||||
attempt: profile.id(),
|
||||
key: public,
|
||||
addresses: addresses(inner, &socket).await?,
|
||||
};
|
||||
let tunn = Tunn::new(
|
||||
local_key,
|
||||
PublicKey::from(key),
|
||||
Some(profile.psk),
|
||||
Some(1),
|
||||
rand::random(),
|
||||
None,
|
||||
);
|
||||
|
||||
let link = session::start(
|
||||
pair.0,
|
||||
pair.1,
|
||||
socket,
|
||||
tunn,
|
||||
profile,
|
||||
candidates,
|
||||
inner.stopped.subscribe(),
|
||||
);
|
||||
if !lock(&inner.allowed).contains(&pair) {
|
||||
link.close();
|
||||
return Err(error("AWG membership ended"));
|
||||
}
|
||||
lock(&inner.links).insert(pair, link.clone());
|
||||
lock(&inner.port_by_pair).insert(pair, local_port);
|
||||
let control = inner
|
||||
.control
|
||||
.get()
|
||||
.ok_or_else(|| error("AWG control unavailable"))?;
|
||||
if let Err(err) = control
|
||||
.send(
|
||||
pair.0,
|
||||
pair.1,
|
||||
AWG_PROTOCOL,
|
||||
TransportPayload::new(
|
||||
postcard::to_stdvec(&answer)
|
||||
.map_err(|_| error("AWG answer encoding failed"))?,
|
||||
),
|
||||
)
|
||||
.await
|
||||
{
|
||||
link.close();
|
||||
return Err(err);
|
||||
}
|
||||
publish_when_up(Arc::downgrade(inner), pair, link, None);
|
||||
}
|
||||
Negotiation::Answer {
|
||||
attempt,
|
||||
key,
|
||||
addresses: candidates,
|
||||
} => {
|
||||
if !lock(&inner.pending)
|
||||
.get(&pair)
|
||||
.is_some_and(|p| p.profile.id() == attempt)
|
||||
{
|
||||
return Err(error("stale AWG answer"));
|
||||
}
|
||||
let pending = lock(&inner.pending)
|
||||
.remove(&pair)
|
||||
.ok_or_else(|| error("AWG attempt cancelled"))?;
|
||||
let local_port = pending
|
||||
.socket
|
||||
.local_addr()
|
||||
.map_err(|_| error("AWG socket address unavailable"))?
|
||||
.port();
|
||||
let tunn = Tunn::new(
|
||||
pending.key,
|
||||
PublicKey::from(key),
|
||||
Some(pending.profile.psk),
|
||||
Some(1),
|
||||
rand::random(),
|
||||
None,
|
||||
);
|
||||
|
||||
let link = session::start(
|
||||
pair.0,
|
||||
pair.1,
|
||||
pending.socket,
|
||||
tunn,
|
||||
pending.profile,
|
||||
candidates,
|
||||
inner.stopped.subscribe(),
|
||||
);
|
||||
if !lock(&inner.allowed).contains(&pair) {
|
||||
link.close();
|
||||
return Err(error("AWG membership ended"));
|
||||
}
|
||||
lock(&inner.links).insert(pair, link.clone());
|
||||
lock(&inner.port_by_pair).insert(pair, local_port);
|
||||
publish_when_up(Arc::downgrade(inner), pair, link, Some(pending.reply));
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
fn publish_when_up(
|
||||
inner: Weak<Inner>,
|
||||
pair: Pair,
|
||||
link: Arc<PairLink>,
|
||||
reply: Option<oneshot::Sender<Result<SharedLink, TransportError>>>,
|
||||
) {
|
||||
tokio::spawn(async move {
|
||||
let mut reply = reply;
|
||||
let result = tokio::time::timeout(Duration::from_secs(14), async {
|
||||
loop {
|
||||
if link.is_closed() || reply.as_ref().is_some_and(oneshot::Sender::is_closed) {
|
||||
return Err(error("AWG pair closed during handshake"));
|
||||
}
|
||||
if link.has_authenticated_path() {
|
||||
return Ok(());
|
||||
}
|
||||
tokio::time::sleep(Duration::from_millis(25)).await;
|
||||
}
|
||||
})
|
||||
.await;
|
||||
if !matches!(result, Ok(Ok(()))) {
|
||||
link.close();
|
||||
if let Some(reply) = reply.take() {
|
||||
let _ = reply.send(Err(error("AWG authenticated data path timed out")));
|
||||
}
|
||||
return;
|
||||
}
|
||||
let Some(inner) = inner.upgrade() else {
|
||||
link.close();
|
||||
return;
|
||||
};
|
||||
if !lock(&inner.allowed).contains(&pair) {
|
||||
link.close();
|
||||
return;
|
||||
}
|
||||
if link.is_closed()
|
||||
|| !lock(&inner.links)
|
||||
.get(&pair)
|
||||
.is_some_and(|current| current.attempt == link.attempt)
|
||||
{
|
||||
link.close();
|
||||
return;
|
||||
}
|
||||
let shared: SharedLink = link.clone();
|
||||
if let Some(reply) = reply {
|
||||
if reply.send(Ok(shared)).is_err() {
|
||||
link.close();
|
||||
}
|
||||
} else if inner
|
||||
.inbound
|
||||
.try_send(InboundLink {
|
||||
network: pair.0,
|
||||
peer: pair.1,
|
||||
protocol: AWG_PROTOCOL.into(),
|
||||
link: shared,
|
||||
})
|
||||
.is_err()
|
||||
{
|
||||
link.close();
|
||||
}
|
||||
});
|
||||
}
|
||||
impl Drop for AwgTransport {
|
||||
fn drop(&mut self) {
|
||||
self.shutdown();
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
|
||||
use super::*;
|
||||
#[tokio::test]
|
||||
async fn concurrent_admission_is_bounded_across_networks_and_retirement_releases_slots() {
|
||||
let identity = tsunagi::identity::DeviceIdentity::generate();
|
||||
let transport = AwgTransport::new(identity.endpoint_id(), Ipv4Addr::LOCALHOST);
|
||||
let peer = tsunagi::identity::DeviceIdentity::generate().endpoint_id();
|
||||
std::thread::scope(|scope| {
|
||||
for thread in 0..8u8 {
|
||||
let transport = &transport;
|
||||
scope.spawn(move || {
|
||||
for item in 0..32u8 {
|
||||
let mut id = [0; 32];
|
||||
id[0] = thread;
|
||||
id[1] = item;
|
||||
transport.allow_peer(NetworkId::from_bytes(id), peer);
|
||||
}
|
||||
});
|
||||
}
|
||||
});
|
||||
assert_eq!(lock(&transport.inner.allowed).len(), 128);
|
||||
let network = lock(&transport.inner.allowed).iter().next().unwrap().0;
|
||||
transport.stop_network(network);
|
||||
let new = NetworkId::from_bytes([255; 32]);
|
||||
transport.allow_peer(new, peer);
|
||||
assert!(lock(&transport.inner.allowed).contains(&(new, peer)));
|
||||
assert_eq!(lock(&transport.inner.allowed).len(), 128);
|
||||
transport.shutdown();
|
||||
assert!(lock(&transport.inner.allowed).is_empty());
|
||||
}
|
||||
#[tokio::test]
|
||||
async fn private_negotiation_bounds_roles_and_membership_before_starting_engines() {
|
||||
let mut ids = [
|
||||
tsunagi::identity::DeviceIdentity::generate().endpoint_id(),
|
||||
tsunagi::identity::DeviceIdentity::generate().endpoint_id(),
|
||||
];
|
||||
ids.sort_by_key(|id| *id.as_bytes());
|
||||
let peer = ids[0];
|
||||
let local = ids[1];
|
||||
let network = NetworkId::from_bytes([7; 32]);
|
||||
let transport = AwgTransport::new(local, Ipv4Addr::LOCALHOST);
|
||||
let offer = |version, key, addresses| Negotiation::Offer {
|
||||
version,
|
||||
profile: PairProfile::generate(),
|
||||
key,
|
||||
addresses,
|
||||
};
|
||||
let address = SocketAddr::from((Ipv4Addr::LOCALHOST, 12345));
|
||||
let encode = |message: &Negotiation| postcard::to_stdvec(message).unwrap();
|
||||
let valid = encode(&offer(AWG_WIRE_VERSION, [1; 32], vec![address]));
|
||||
assert!(
|
||||
transport
|
||||
.on_control(network, peer, AWG_PROTOCOL, &valid)
|
||||
.is_err()
|
||||
);
|
||||
transport.allow_peer(network, peer);
|
||||
assert!(
|
||||
transport
|
||||
.on_control(network, peer, "wrong-protocol", &valid)
|
||||
.is_err()
|
||||
);
|
||||
assert!(
|
||||
transport
|
||||
.on_control(network, peer, AWG_PROTOCOL, &[0; 4097])
|
||||
.is_err()
|
||||
);
|
||||
for bytes in [
|
||||
encode(&offer(99, [1; 32], vec![address])),
|
||||
encode(&offer(AWG_WIRE_VERSION, [0; 32], vec![address])),
|
||||
encode(&offer(AWG_WIRE_VERSION, [1; 32], vec![])),
|
||||
encode(&offer(AWG_WIRE_VERSION, [1; 32], vec![address; 33])),
|
||||
] {
|
||||
assert!(
|
||||
transport
|
||||
.on_control(network, peer, AWG_PROTOCOL, &bytes)
|
||||
.is_err()
|
||||
);
|
||||
}
|
||||
for len in 0..valid.len() {
|
||||
assert!(
|
||||
transport
|
||||
.on_control(network, peer, AWG_PROTOCOL, &valid[..len])
|
||||
.is_err()
|
||||
);
|
||||
}
|
||||
let wrong_role = AwgTransport::new(peer, Ipv4Addr::LOCALHOST);
|
||||
wrong_role.allow_peer(network, local);
|
||||
assert!(
|
||||
wrong_role
|
||||
.on_control(network, local, AWG_PROTOCOL, &valid)
|
||||
.is_err()
|
||||
);
|
||||
transport.forget_peer(network, peer);
|
||||
assert!(
|
||||
transport
|
||||
.on_control(network, peer, AWG_PROTOCOL, &valid)
|
||||
.is_err()
|
||||
);
|
||||
assert!(transport.overview(network).is_empty());
|
||||
transport.shutdown();
|
||||
wrong_role.shutdown();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,773 @@
|
||||
//! Real membership, UDP sockets, pair negotiation and nested WireGuard encryption.
|
||||
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
|
||||
use bytes::Bytes;
|
||||
use std::{
|
||||
net::Ipv4Addr,
|
||||
path::Path,
|
||||
sync::{
|
||||
Arc,
|
||||
atomic::{AtomicBool, AtomicU64, Ordering},
|
||||
},
|
||||
time::Duration,
|
||||
};
|
||||
use tempfile::TempDir;
|
||||
use tsunagi::dataplane::IpPlugin;
|
||||
use tsunagi::dataplane::transport::{
|
||||
ControlSender, PacketLink, PacketTransport, SharedLink, TransportError,
|
||||
};
|
||||
use tsunagi::discovery::SharedMemoryDiscovery;
|
||||
use tsunagi::overlay::{MemoryTun, MemoryTunFactory};
|
||||
use tsunagi::testing::{config_with, network, wait_for_peers, wait_until};
|
||||
use tsunagi::{Agent, BoxFuture, NetworkId};
|
||||
use tsunagi_awg::{AWG_PROTOCOL, AwgPlugin, AwgTransport};
|
||||
use tsunagi_wg_quic::WireguardConfig;
|
||||
|
||||
struct Node {
|
||||
agent: Agent,
|
||||
plugin: Arc<AwgPlugin>,
|
||||
quic: Option<Arc<tsunagi_wg_quic::WireguardPlugin>>,
|
||||
transport: Arc<AwgTransport>,
|
||||
tuns: MemoryTunFactory,
|
||||
loss: Arc<AtomicBool>,
|
||||
}
|
||||
impl Node {
|
||||
async fn open(root: &Path, discovery: &SharedMemoryDiscovery, interface: &str) -> Self {
|
||||
Self::open_with_fallback(root, discovery, interface, false).await
|
||||
}
|
||||
async fn open_with_fallback(
|
||||
root: &Path,
|
||||
discovery: &SharedMemoryDiscovery,
|
||||
interface: &str,
|
||||
fallback: bool,
|
||||
) -> Self {
|
||||
Self::open_config(root, discovery, interface, fallback, 1280).await
|
||||
}
|
||||
async fn open_config(
|
||||
root: &Path,
|
||||
discovery: &SharedMemoryDiscovery,
|
||||
interface: &str,
|
||||
fallback: bool,
|
||||
mtu: u32,
|
||||
) -> Self {
|
||||
Self::open_selected(root, discovery, interface, fallback, mtu, None).await
|
||||
}
|
||||
async fn open_selected(
|
||||
root: &Path,
|
||||
discovery: &SharedMemoryDiscovery,
|
||||
interface: &str,
|
||||
fallback: bool,
|
||||
mtu: u32,
|
||||
default_protocols: Option<Vec<String>>,
|
||||
) -> Self {
|
||||
let identity =
|
||||
tsunagi::storage::StateStore::open(tsunagi::StoragePaths::under(root).state_db())
|
||||
.unwrap()
|
||||
.load_or_create_device_identity()
|
||||
.unwrap();
|
||||
let transport = AwgTransport::new(identity.endpoint_id(), Ipv4Addr::LOCALHOST);
|
||||
let plugin = AwgPlugin::open(
|
||||
WireguardConfig::new(root.join("awg"))
|
||||
.with_mtu(mtu)
|
||||
.with_reconcile(Duration::from_millis(20), Duration::from_millis(250)),
|
||||
transport.clone(),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
let loss = Arc::new(AtomicBool::new(false));
|
||||
let faults = Arc::new(FaultTransport {
|
||||
inner: transport.clone(),
|
||||
loss: loss.clone(),
|
||||
});
|
||||
let tuns = MemoryTunFactory::new();
|
||||
let mut config = config_with(root, discovery)
|
||||
.with_interface(Arc::new(tuns.clone()), interface, mtu)
|
||||
.with_plugin(plugin.clone() as Arc<dyn IpPlugin>)
|
||||
.with_custom_transport(AWG_PROTOCOL, faults);
|
||||
let mut quic_plugin = None;
|
||||
if fallback {
|
||||
let quic = tsunagi_wg_quic::WireguardPlugin::open(
|
||||
WireguardConfig::new(root.join("wg-quic"))
|
||||
.with_mtu(mtu)
|
||||
.with_reconcile(Duration::from_millis(20), Duration::from_millis(250)),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
config = config.with_plugin(quic.clone());
|
||||
quic_plugin = Some(quic);
|
||||
}
|
||||
config.default_protocols = default_protocols;
|
||||
let agent = Agent::spawn(config).await.unwrap();
|
||||
let inbound_agent = agent.clone();
|
||||
let inbound_transport = transport.clone();
|
||||
let inbound_loss = loss.clone();
|
||||
tokio::spawn(async move {
|
||||
while let Some(mut link) = inbound_transport.next_inbound().await {
|
||||
link.link = impaired(link.link, inbound_loss.clone());
|
||||
let _ = inbound_agent.install_inbound_link(link).await;
|
||||
}
|
||||
});
|
||||
Self {
|
||||
agent,
|
||||
plugin,
|
||||
quic: quic_plugin,
|
||||
transport,
|
||||
tuns,
|
||||
loss,
|
||||
}
|
||||
}
|
||||
async fn ready(&self, id: NetworkId, n: usize) {
|
||||
wait_until("AWG outer paths and end-to-end handshakes", || async {
|
||||
(self
|
||||
.transport
|
||||
.overview(id)
|
||||
.iter()
|
||||
.filter(|p| !p.closed && !p.path.contains("establishing"))
|
||||
.count()
|
||||
== n
|
||||
&& self
|
||||
.plugin
|
||||
.overview(id)?
|
||||
.peers
|
||||
.iter()
|
||||
.filter(|p| {
|
||||
p.tunnel
|
||||
.as_ref()
|
||||
.is_some_and(|t| t.health.is_up() && t.path.contains("awg UDP via"))
|
||||
})
|
||||
.count()
|
||||
== n)
|
||||
.then_some(())
|
||||
})
|
||||
.await;
|
||||
}
|
||||
fn tun(&self) -> Arc<MemoryTun> {
|
||||
self.tuns
|
||||
.device(&self.agent.overlay().unwrap().interface)
|
||||
.unwrap()
|
||||
}
|
||||
fn addr(&self, id: NetworkId) -> Ipv4Addr {
|
||||
self.plugin
|
||||
.overview(id)
|
||||
.unwrap()
|
||||
.overlay_address_v4
|
||||
.unwrap()
|
||||
}
|
||||
}
|
||||
fn packet(source: Ipv4Addr, dest: Ipv4Addr, len: usize) -> Bytes {
|
||||
let mut bytes = vec![0; len];
|
||||
bytes[0] = 0x45;
|
||||
bytes[2..4].copy_from_slice(&(len as u16).to_be_bytes());
|
||||
bytes[8] = 64;
|
||||
bytes[9] = 253;
|
||||
bytes[12..16].copy_from_slice(&source.octets());
|
||||
bytes[16..20].copy_from_slice(&dest.octets());
|
||||
rand::fill(&mut bytes[20..]);
|
||||
Bytes::from(bytes)
|
||||
}
|
||||
async fn exchange(a: &Node, b: &Node, id: NetworkId) {
|
||||
for len in [64, 1097, 1280] {
|
||||
let bytes = packet(a.addr(id), b.addr(id), len);
|
||||
a.tun().push_from_os(bytes.clone());
|
||||
assert_eq!(
|
||||
tokio::time::timeout(Duration::from_secs(5), b.tun().pop_to_os())
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap(),
|
||||
bytes
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_control_only_agent_enables_real_data_without_restart_and_cancels_stale_attempts() {
|
||||
let discovery = SharedMemoryDiscovery::new();
|
||||
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
|
||||
let a = Node::open_selected(
|
||||
roots[0].path(),
|
||||
&discovery,
|
||||
"initial-none-a",
|
||||
true,
|
||||
1280,
|
||||
Some(vec![]),
|
||||
)
|
||||
.await;
|
||||
let b = Node::open_selected(
|
||||
roots[1].path(),
|
||||
&discovery,
|
||||
"initial-none-b",
|
||||
true,
|
||||
1280,
|
||||
Some(vec![]),
|
||||
)
|
||||
.await;
|
||||
let (name, secret) = network("enable-after-control-only");
|
||||
let id = a.agent.join_network(&name, &secret).await.unwrap();
|
||||
b.agent.join_network(&name, &secret).await.unwrap();
|
||||
wait_for_peers(&a.agent, id, 1).await;
|
||||
wait_for_peers(&b.agent, id, 1).await;
|
||||
assert!(a.agent.overlay().is_none());
|
||||
assert!(b.agent.overlay().is_none());
|
||||
for node in [&a, &b] {
|
||||
assert!(
|
||||
node.agent
|
||||
.network_status(id)
|
||||
.await
|
||||
.unwrap()
|
||||
.protocols
|
||||
.is_empty()
|
||||
);
|
||||
node.agent
|
||||
.set_protocols(id, vec!["awg".into()])
|
||||
.await
|
||||
.unwrap();
|
||||
assert!(node.agent.overlay().is_some());
|
||||
}
|
||||
a.ready(id, 1).await;
|
||||
b.ready(id, 1).await;
|
||||
exchange(&a, &b, id).await;
|
||||
let before = a.agent.overlay().unwrap().interface;
|
||||
// Changes race unfinished opens and prepared plugin callbacks; the last
|
||||
// selection must win even if an earlier open has completed in its queue.
|
||||
for selection in [
|
||||
vec!["wg-quic".into()],
|
||||
vec![],
|
||||
vec!["awg".into()],
|
||||
vec![],
|
||||
vec!["awg".into()],
|
||||
] {
|
||||
a.agent.set_protocols(id, selection).await.unwrap();
|
||||
}
|
||||
a.ready(id, 1).await;
|
||||
b.ready(id, 1).await;
|
||||
exchange(&a, &b, id).await;
|
||||
assert_eq!(a.agent.overlay().unwrap().interface, before);
|
||||
for node in [&a, &b] {
|
||||
let status = node.agent.network_status(id).await.unwrap();
|
||||
assert_eq!(status.protocols, ["awg"]);
|
||||
assert_eq!(status.metrics.sessions_established, 1);
|
||||
assert_eq!(status.metrics.disconnects, 0);
|
||||
}
|
||||
a.agent.shutdown().await;
|
||||
b.agent.shutdown().await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn runtime_protocol_selection_replaces_real_tunnels_and_survives_restart() {
|
||||
let discovery = SharedMemoryDiscovery::new();
|
||||
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
|
||||
let a = Node::open_with_fallback(roots[0].path(), &discovery, "runtime-a", true).await;
|
||||
let b = Node::open_with_fallback(roots[1].path(), &discovery, "runtime-b", true).await;
|
||||
let (name, secret) = network("runtime-protocols");
|
||||
let id = a.agent.join_network(&name, &secret).await.unwrap();
|
||||
b.agent.join_network(&name, &secret).await.unwrap();
|
||||
a.ready(id, 1).await;
|
||||
b.ready(id, 1).await;
|
||||
exchange(&a, &b, id).await;
|
||||
|
||||
let (other_name, other_secret, other_id) = loop {
|
||||
let (name, secret) = network("unchanged-runtime-network");
|
||||
let other_id = tsunagi::identity::NetworkKeys::derive(&name, &secret).network_id();
|
||||
if tsunagi::state::derived_ipv4_range(id) != tsunagi::state::derived_ipv4_range(other_id) {
|
||||
break (name, secret, other_id);
|
||||
}
|
||||
};
|
||||
a.agent
|
||||
.join_network(&other_name, &other_secret)
|
||||
.await
|
||||
.unwrap();
|
||||
b.agent
|
||||
.join_network(&other_name, &other_secret)
|
||||
.await
|
||||
.unwrap();
|
||||
a.ready(other_id, 1).await;
|
||||
b.ready(other_id, 1).await;
|
||||
let other_before = a
|
||||
.agent
|
||||
.network_status(other_id)
|
||||
.await
|
||||
.unwrap()
|
||||
.metrics
|
||||
.data_links_established;
|
||||
|
||||
for node in [&a, &b] {
|
||||
let before = node.agent.network_status(id).await.unwrap().protocols;
|
||||
for invalid in [vec!["unknown".into()], vec!["awg".into(), "awg".into()]] {
|
||||
assert!(node.agent.set_protocols(id, invalid).await.is_err());
|
||||
assert_eq!(
|
||||
node.agent.network_status(id).await.unwrap().protocols,
|
||||
before
|
||||
);
|
||||
}
|
||||
}
|
||||
// Exercise updates initiated by either endpoint-id role, including an
|
||||
// empty set. Membership and the control session stay alive throughout.
|
||||
for node in [&a, &b] {
|
||||
node.agent.set_protocols(id, Vec::new()).await.unwrap();
|
||||
wait_until("peer accepts control-only selection", || async {
|
||||
let other = if node.agent.endpoint_id() == a.agent.endpoint_id() {
|
||||
&b
|
||||
} else {
|
||||
&a
|
||||
};
|
||||
let status = other.agent.network_status(id).await.ok()?;
|
||||
(status.peers.len() == 1
|
||||
&& status.peers[0].protocols.is_empty()
|
||||
&& matches!(status.peers[0].data_path, tsunagi::agent::DataPath::None))
|
||||
.then_some(())
|
||||
})
|
||||
.await;
|
||||
assert_eq!(node.agent.network_status(id).await.unwrap().peers.len(), 1);
|
||||
node.agent
|
||||
.set_protocols(id, vec!["wg-quic".into(), "awg".into()])
|
||||
.await
|
||||
.unwrap();
|
||||
}
|
||||
a.agent
|
||||
.set_protocols(id, vec!["wg-quic".into()])
|
||||
.await
|
||||
.unwrap();
|
||||
wait_until("real QUIC replaces AWG", || async {
|
||||
let a_status = a.agent.network_status(id).await.ok()?;
|
||||
let b_status = b.agent.network_status(id).await.ok()?;
|
||||
(a_status.peers[0].protocols == ["wg-quic"]
|
||||
&& b_status.peers[0].protocols == ["wg-quic"]
|
||||
&& !matches!(a_status.peers[0].data_path, tsunagi::agent::DataPath::None)
|
||||
&& !matches!(b_status.peers[0].data_path, tsunagi::agent::DataPath::None)
|
||||
&& a.transport.overview(id).iter().all(|p| p.closed)
|
||||
&& b.transport.overview(id).iter().all(|p| p.closed))
|
||||
.then_some(())
|
||||
})
|
||||
.await;
|
||||
// Get addresses from the system allocation: the disabled AWG plugin no
|
||||
// longer has a live view of this network.
|
||||
for (source, dest) in [(&a, &b), (&b, &a)] {
|
||||
wait_until("QUIC end-to-end handshake", || async {
|
||||
source
|
||||
.quic
|
||||
.as_ref()?
|
||||
.overview(id)?
|
||||
.peers
|
||||
.iter()
|
||||
.any(|p| p.tunnel.as_ref().is_some_and(|t| t.health.is_up()))
|
||||
.then_some(())
|
||||
})
|
||||
.await;
|
||||
let addr = |n: &Node| {
|
||||
n.quic
|
||||
.as_ref()
|
||||
.unwrap()
|
||||
.overview(id)
|
||||
.unwrap()
|
||||
.overlay_address_v4
|
||||
.unwrap()
|
||||
};
|
||||
let bytes = packet(addr(source), addr(dest), 1280);
|
||||
source.tun().push_from_os(bytes.clone());
|
||||
assert_eq!(
|
||||
tokio::time::timeout(Duration::from_secs(5), dest.tun().pop_to_os())
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap(),
|
||||
bytes
|
||||
);
|
||||
}
|
||||
assert_eq!(
|
||||
a.agent
|
||||
.network_status(other_id)
|
||||
.await
|
||||
.unwrap()
|
||||
.metrics
|
||||
.data_links_established,
|
||||
other_before
|
||||
);
|
||||
exchange(&a, &b, other_id).await;
|
||||
a.agent.shutdown().await;
|
||||
let restarted = Node::open_with_fallback(roots[0].path(), &discovery, "runtime-a", true).await;
|
||||
wait_for_peers(&restarted.agent, id, 1).await;
|
||||
assert_eq!(
|
||||
restarted.agent.network_status(id).await.unwrap().protocols,
|
||||
["wg-quic"]
|
||||
);
|
||||
assert_eq!(
|
||||
b.agent.network_status(other_id).await.unwrap().protocols,
|
||||
["awg", "wg-quic"]
|
||||
);
|
||||
restarted.ready(other_id, 1).await;
|
||||
b.ready(other_id, 1).await;
|
||||
exchange(&restarted, &b, other_id).await;
|
||||
restarted
|
||||
.agent
|
||||
.set_protocols(id, vec!["awg".into()])
|
||||
.await
|
||||
.unwrap();
|
||||
restarted.ready(id, 1).await;
|
||||
b.ready(id, 1).await;
|
||||
exchange(&restarted, &b, id).await;
|
||||
restarted.agent.shutdown().await;
|
||||
b.agent.shutdown().await;
|
||||
}
|
||||
#[tokio::test]
|
||||
async fn real_pairs_carry_full_size_packets_and_restart_with_fresh_profiles() {
|
||||
tsunagi::testing::init_tracing();
|
||||
let discovery = SharedMemoryDiscovery::new();
|
||||
let roots = [
|
||||
TempDir::new().unwrap(),
|
||||
TempDir::new().unwrap(),
|
||||
TempDir::new().unwrap(),
|
||||
];
|
||||
let a = Node::open(roots[0].path(), &discovery, "awga").await;
|
||||
let b = Node::open(roots[1].path(), &discovery, "awgb").await;
|
||||
let c = Node::open(roots[2].path(), &discovery, "awgc").await;
|
||||
let (name, secret) = network("awg-pairs");
|
||||
let id = a.agent.join_network(&name, &secret).await.unwrap();
|
||||
b.agent.join_network(&name, &secret).await.unwrap();
|
||||
c.agent.join_network(&name, &secret).await.unwrap();
|
||||
for node in [&a, &b, &c] {
|
||||
wait_for_peers(&node.agent, id, 2).await;
|
||||
node.ready(id, 2).await;
|
||||
}
|
||||
let ids: std::collections::HashSet<_> = [&a, &b, &c]
|
||||
.into_iter()
|
||||
.flat_map(|n| n.transport.overview(id).into_iter().map(|p| p.attempt))
|
||||
.collect();
|
||||
assert_eq!(
|
||||
ids.len(),
|
||||
3,
|
||||
"one independent profile for each of three pairs"
|
||||
);
|
||||
exchange(&a, &b, id).await;
|
||||
exchange(&b, &a, id).await;
|
||||
exchange(&a, &c, id).await;
|
||||
let old_ports: Vec<_> = a
|
||||
.transport
|
||||
.overview(id)
|
||||
.into_iter()
|
||||
.filter_map(|p| p.local_port)
|
||||
.collect();
|
||||
a.agent.shutdown().await;
|
||||
for port in old_ports {
|
||||
wait_until("old pair socket released", || async {
|
||||
tokio::net::UdpSocket::bind((Ipv4Addr::LOCALHOST, port))
|
||||
.await
|
||||
.ok()
|
||||
})
|
||||
.await;
|
||||
}
|
||||
let restarted = Node::open(roots[0].path(), &discovery, "awga").await;
|
||||
restarted.ready(id, 2).await;
|
||||
assert!(
|
||||
restarted
|
||||
.transport
|
||||
.overview(id)
|
||||
.iter()
|
||||
.all(|p| !ids.contains(&p.attempt))
|
||||
);
|
||||
exchange(&restarted, &b, id).await;
|
||||
exchange(&c, &restarted, id).await;
|
||||
restarted.agent.shutdown().await;
|
||||
b.agent.shutdown().await;
|
||||
c.agent.shutdown().await;
|
||||
assert!(
|
||||
tokio::time::timeout(Duration::from_secs(1), restarted.transport.next_inbound())
|
||||
.await
|
||||
.unwrap()
|
||||
.is_none()
|
||||
);
|
||||
}
|
||||
|
||||
// Drops datagrams at the link boundary. Membership, control, sockets, crypto,
|
||||
// addressing and storage remain real. This substitutes a lossy physical path.
|
||||
#[derive(Debug)]
|
||||
struct FaultTransport {
|
||||
inner: Arc<AwgTransport>,
|
||||
loss: Arc<AtomicBool>,
|
||||
}
|
||||
impl PacketTransport for FaultTransport {
|
||||
fn name(&self) -> &str {
|
||||
AWG_PROTOCOL
|
||||
}
|
||||
fn as_any(&self) -> &dyn std::any::Any {
|
||||
self
|
||||
}
|
||||
fn attach_control(&self, sender: Arc<dyn ControlSender>) {
|
||||
self.inner.attach_control(sender);
|
||||
}
|
||||
fn on_control(
|
||||
&self,
|
||||
network: NetworkId,
|
||||
peer: iroh::EndpointId,
|
||||
protocol: &str,
|
||||
bytes: &[u8],
|
||||
) -> Result<(), TransportError> {
|
||||
self.inner.on_control(network, peer, protocol, bytes)
|
||||
}
|
||||
fn open<'a>(
|
||||
&'a self,
|
||||
network: NetworkId,
|
||||
peer: iroh::EndpointId,
|
||||
protocol: &'a str,
|
||||
) -> BoxFuture<'a, Result<SharedLink, TransportError>> {
|
||||
Box::pin(async move {
|
||||
Ok(impaired(
|
||||
self.inner.open(network, peer, protocol).await?,
|
||||
self.loss.clone(),
|
||||
))
|
||||
})
|
||||
}
|
||||
}
|
||||
fn impaired(raw: SharedLink, loss: Arc<AtomicBool>) -> SharedLink {
|
||||
Arc::new(FaultLink {
|
||||
raw,
|
||||
loss,
|
||||
sequence: AtomicU64::new(0),
|
||||
})
|
||||
}
|
||||
#[derive(Debug)]
|
||||
struct FaultLink {
|
||||
raw: SharedLink,
|
||||
loss: Arc<AtomicBool>,
|
||||
sequence: AtomicU64,
|
||||
}
|
||||
impl PacketLink for FaultLink {
|
||||
fn network(&self) -> NetworkId {
|
||||
self.raw.network()
|
||||
}
|
||||
fn peer(&self) -> iroh::EndpointId {
|
||||
self.raw.peer()
|
||||
}
|
||||
fn max_datagram_size(&self) -> usize {
|
||||
self.raw.max_datagram_size()
|
||||
}
|
||||
fn send(&self, bytes: Bytes) -> Result<(), TransportError> {
|
||||
if self.loss.load(Ordering::Relaxed)
|
||||
&& !self
|
||||
.sequence
|
||||
.fetch_add(1, Ordering::Relaxed)
|
||||
.is_multiple_of(5)
|
||||
{
|
||||
Ok(())
|
||||
} else {
|
||||
self.raw.send(bytes)
|
||||
}
|
||||
}
|
||||
fn recv(&self) -> BoxFuture<'_, Option<Bytes>> {
|
||||
self.raw.recv()
|
||||
}
|
||||
fn closed(&self) -> BoxFuture<'_, ()> {
|
||||
self.raw.closed()
|
||||
}
|
||||
fn is_closed(&self) -> bool {
|
||||
self.raw.is_closed()
|
||||
}
|
||||
fn close(&self) {
|
||||
self.raw.close();
|
||||
}
|
||||
fn path_description(&self) -> String {
|
||||
self.raw.path_description()
|
||||
}
|
||||
}
|
||||
#[tokio::test]
|
||||
async fn persistent_runtime_loss_retires_the_pair_and_negotiates_fresh_parameters() {
|
||||
let discovery = SharedMemoryDiscovery::new();
|
||||
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
|
||||
let a = Node::open(roots[0].path(), &discovery, "lossa").await;
|
||||
let b = Node::open(roots[1].path(), &discovery, "lossb").await;
|
||||
let (name, secret) = network("awg-loss");
|
||||
let id = a.agent.join_network(&name, &secret).await.unwrap();
|
||||
b.agent.join_network(&name, &secret).await.unwrap();
|
||||
a.ready(id, 1).await;
|
||||
b.ready(id, 1).await;
|
||||
exchange(&a, &b, id).await;
|
||||
let old = a.transport.overview(id)[0].attempt;
|
||||
let mut events = a.agent.subscribe();
|
||||
a.loss.store(true, Ordering::Relaxed);
|
||||
tsunagi::testing::wait_event(&mut events, |event| match event {
|
||||
tsunagi::Event::DataLinkDown {
|
||||
network, reason, ..
|
||||
} if *network == id && reason.contains("delivery checks") => Some(()),
|
||||
_ => None,
|
||||
})
|
||||
.await;
|
||||
a.loss.store(false, Ordering::Relaxed);
|
||||
wait_until("fresh AWG profile after runtime loss", || async {
|
||||
a.transport
|
||||
.overview(id)
|
||||
.iter()
|
||||
.any(|p| p.attempt != old && !p.closed && !p.path.contains("establishing"))
|
||||
.then_some(())
|
||||
})
|
||||
.await;
|
||||
a.ready(id, 1).await;
|
||||
b.ready(id, 1).await;
|
||||
let initiator = if a.agent.endpoint_id().as_bytes() < b.agent.endpoint_id().as_bytes() {
|
||||
&a
|
||||
} else {
|
||||
&b
|
||||
};
|
||||
let received = initiator.plugin.overview(id).unwrap().peers[0]
|
||||
.tunnel
|
||||
.as_ref()
|
||||
.unwrap()
|
||||
.stats
|
||||
.rx_packets;
|
||||
let attempt = initiator.transport.overview(id)[0].attempt;
|
||||
wait_until(
|
||||
"fresh pair delivery checks answered after installation",
|
||||
|| async {
|
||||
let view = initiator.plugin.overview(id)?;
|
||||
(view.peers.first()?.tunnel.as_ref()?.stats.rx_packets >= received + 3).then_some(())
|
||||
},
|
||||
)
|
||||
.await;
|
||||
assert_eq!(initiator.transport.overview(id)[0].attempt, attempt);
|
||||
wait_for_peers(&a.agent, id, 1).await;
|
||||
exchange(&a, &b, id).await;
|
||||
exchange(&b, &a, id).await;
|
||||
a.agent.shutdown().await;
|
||||
b.agent.shutdown().await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn runtime_loss_falls_back_to_real_iroh_without_interrupting_control_or_ip() {
|
||||
let discovery = SharedMemoryDiscovery::new();
|
||||
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
|
||||
let a = Node::open_with_fallback(roots[0].path(), &discovery, "fallba", true).await;
|
||||
let b = Node::open_with_fallback(roots[1].path(), &discovery, "fallbb", true).await;
|
||||
let (name, secret) = network("awg-fallback");
|
||||
let id = a.agent.join_network(&name, &secret).await.unwrap();
|
||||
b.agent.join_network(&name, &secret).await.unwrap();
|
||||
a.ready(id, 1).await;
|
||||
b.ready(id, 1).await;
|
||||
exchange(&a, &b, id).await;
|
||||
let mut events = a.agent.subscribe();
|
||||
a.loss.store(true, Ordering::Relaxed);
|
||||
tsunagi::testing::wait_event(&mut events, |event| match event {
|
||||
tsunagi::Event::DataLinkDown {
|
||||
network, reason, ..
|
||||
} if *network == id && reason.contains("delivery checks") => Some(()),
|
||||
_ => None,
|
||||
})
|
||||
.await;
|
||||
for node in [&a, &b] {
|
||||
wait_until("fallback carrier is wg-quic",||async {
|
||||
let status=node.agent.status().await.ok()?;
|
||||
let network=status.networks.iter().find(|n|n.network_id==id)?;
|
||||
(network.members.iter().any(|m|matches!(&m.data_path,tsunagi::agent::DataPath::Direct{transport} if transport=="wg-quic"))).then_some(())
|
||||
}).await;
|
||||
wait_for_peers(&node.agent, id, 1).await;
|
||||
}
|
||||
exchange(&a, &b, id).await;
|
||||
exchange(&b, &a, id).await;
|
||||
a.agent.shutdown().await;
|
||||
b.agent.shutdown().await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn opaque_pair_carrier_fragments_its_full_advertised_capacity() {
|
||||
let discovery = SharedMemoryDiscovery::new();
|
||||
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
|
||||
let mut nodes = Vec::new();
|
||||
for root in &roots {
|
||||
let identity = tsunagi::storage::StateStore::open(
|
||||
tsunagi::StoragePaths::under(root.path()).state_db(),
|
||||
)
|
||||
.unwrap()
|
||||
.load_or_create_device_identity()
|
||||
.unwrap();
|
||||
let transport = AwgTransport::new(identity.endpoint_id(), Ipv4Addr::LOCALHOST);
|
||||
let agent = Agent::spawn(
|
||||
config_with(root.path(), &discovery)
|
||||
.with_custom_transport(AWG_PROTOCOL, transport.clone()),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
nodes.push((agent, transport));
|
||||
}
|
||||
let (name, secret) = network("awg-capacity");
|
||||
let id = nodes[0].0.join_network(&name, &secret).await.unwrap();
|
||||
nodes[1].0.join_network(&name, &secret).await.unwrap();
|
||||
for (i, (agent, transport)) in nodes.iter().enumerate() {
|
||||
wait_for_peers(agent, id, 1).await;
|
||||
// This fixture tests only the transport contract: register the member
|
||||
// after the real core has authenticated it, without an IP plugin/TUN.
|
||||
transport.allow_peer(id, nodes[1 - i].0.endpoint_id());
|
||||
}
|
||||
nodes.sort_by_key(|(agent, _)| *agent.endpoint_id().as_bytes());
|
||||
let outgoing = nodes[0]
|
||||
.1
|
||||
.open(id, nodes[1].0.endpoint_id(), AWG_PROTOCOL)
|
||||
.await
|
||||
.unwrap();
|
||||
let incoming = tokio::time::timeout(Duration::from_secs(5), nodes[1].1.next_inbound())
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap()
|
||||
.link;
|
||||
for (from, to) in [(&outgoing, &incoming), (&incoming, &outgoing)] {
|
||||
let mut bytes = vec![0; from.max_datagram_size()];
|
||||
rand::fill(&mut bytes[..]);
|
||||
let bytes = Bytes::from(bytes);
|
||||
from.send(bytes.clone()).unwrap();
|
||||
assert_eq!(
|
||||
tokio::time::timeout(Duration::from_secs(5), to.recv())
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap(),
|
||||
bytes
|
||||
);
|
||||
assert!(matches!(
|
||||
from.send(Bytes::from(vec![0; from.max_datagram_size() + 1])),
|
||||
Err(TransportError::TooLarge { .. })
|
||||
));
|
||||
}
|
||||
outgoing.close();
|
||||
incoming.close();
|
||||
for (agent, transport) in nodes {
|
||||
transport.shutdown();
|
||||
agent.shutdown().await;
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn encrypted_health_checks_fit_a_smaller_peers_configured_interface_mtu() {
|
||||
let discovery = SharedMemoryDiscovery::new();
|
||||
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
|
||||
let a = Node::open_config(roots[0].path(), &discovery, "smalla", false, 1280).await;
|
||||
let b = Node::open_config(roots[1].path(), &discovery, "smallb", false, 576).await;
|
||||
let (name, secret) = network("awg-small-mtu");
|
||||
let id = a.agent.join_network(&name, &secret).await.unwrap();
|
||||
b.agent.join_network(&name, &secret).await.unwrap();
|
||||
a.ready(id, 1).await;
|
||||
b.ready(id, 1).await;
|
||||
let bytes = packet(a.addr(id), b.addr(id), 576);
|
||||
a.tun().push_from_os(bytes.clone());
|
||||
assert_eq!(
|
||||
tokio::time::timeout(Duration::from_secs(5), b.tun().pop_to_os())
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap(),
|
||||
bytes
|
||||
);
|
||||
wait_until(
|
||||
"encrypted delivery checks answered with a small MTU",
|
||||
|| async {
|
||||
let view = a.plugin.overview(id)?;
|
||||
let peer = view.peers.first()?.tunnel.as_ref()?;
|
||||
(peer.stats.rx_packets >= 3).then_some(())
|
||||
},
|
||||
)
|
||||
.await;
|
||||
for node in [&a, &b] {
|
||||
assert_eq!(
|
||||
node.plugin.overview(id).unwrap().peers[0]
|
||||
.tunnel
|
||||
.as_ref()
|
||||
.unwrap()
|
||||
.stats
|
||||
.dropped_oversize,
|
||||
0
|
||||
);
|
||||
}
|
||||
a.agent.shutdown().await;
|
||||
b.agent.shutdown().await;
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
//! Fresh and bounded masking profiles, and rejection of foreign/truncated wire packets.
|
||||
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
|
||||
use tsunagi_awg::profile::PairProfile;
|
||||
|
||||
#[test]
|
||||
fn profiles_are_fresh_bounded_and_redacted() {
|
||||
let profiles: Vec<_> = (0..128).map(|_| PairProfile::generate()).collect();
|
||||
for (i, profile) in profiles.iter().enumerate() {
|
||||
profile.validate().unwrap();
|
||||
assert!(!format!("{profile:?}").contains(&format!("{:?}", profile.header_key())));
|
||||
for other in &profiles[..i] {
|
||||
assert_ne!(profile.header_key(), other.header_key());
|
||||
assert_ne!(profile.id(), other.id());
|
||||
}
|
||||
for packet in profile.signature_packets() {
|
||||
assert!(packet.len() < 1200);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn masked_packets_round_trip_and_reject_other_profiles_and_truncation() {
|
||||
let profile = PairProfile::generate();
|
||||
let other = PairProfile::generate();
|
||||
for (kind, size) in [(1, 148), (2, 92), (3, 64), (4, 800)] {
|
||||
let mut message = vec![0x7b; size];
|
||||
message[..4].copy_from_slice(&(kind as u32).to_le_bytes());
|
||||
let wire = profile.encode(&message).unwrap();
|
||||
assert_ne!(&wire[..4], &message[..4]);
|
||||
assert_eq!(profile.decode(&wire).unwrap(), message);
|
||||
assert!(other.decode(&wire).is_none());
|
||||
for length in 0..44 {
|
||||
assert!(profile.decode(&wire[..length]).is_none());
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -129,6 +129,19 @@ pub enum Event {
|
||||
/// Why it is not up.
|
||||
reason: String,
|
||||
},
|
||||
/// The name this agent claimed was claimed first by another member.
|
||||
///
|
||||
/// Nothing stops working: this agent keeps its address and every
|
||||
/// connection. It just is not found by that name until it is renamed or
|
||||
/// the member that holds the name lets it go.
|
||||
HostnameTaken {
|
||||
/// The network.
|
||||
network: NetworkId,
|
||||
/// The name that is taken.
|
||||
name: String,
|
||||
/// The member that holds it.
|
||||
holder: EndpointId,
|
||||
},
|
||||
/// An IP plugin reported an error. Never fatal.
|
||||
PluginError {
|
||||
/// The network the call was scoped to.
|
||||
|
||||
@@ -26,15 +26,15 @@ mod status;
|
||||
|
||||
pub use events::Event;
|
||||
pub use status::{
|
||||
AgentStatus, CandidateStatus, MemberStatus, NetworkMetrics, NetworkState, NetworkStatus,
|
||||
OverlayStatus, PeerStatus,
|
||||
AgentStatus, CandidateStatus, DataPath, HostnameConflict, MemberStatus, NetworkMetrics,
|
||||
NetworkState, NetworkStatus, OverlayStatus, PeerStatus, RangeConflict,
|
||||
};
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::sync::{Arc, Weak};
|
||||
|
||||
use iroh::{EndpointAddr, EndpointId};
|
||||
use tokio::sync::{RwLock, broadcast, mpsc, oneshot};
|
||||
use tokio::sync::{Mutex, RwLock, broadcast, mpsc, oneshot, watch};
|
||||
use tokio::task::JoinHandle;
|
||||
|
||||
use crate::BoxFuture;
|
||||
@@ -58,6 +58,8 @@ use shutdown::Shutdown;
|
||||
/// Summary of a configured network, whether or not it is running.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct ConfiguredNetwork {
|
||||
/// Enabled local data-plane protocols, in preference order.
|
||||
pub protocols: Vec<String>,
|
||||
/// Saved local broadcast participation.
|
||||
pub broadcast: bool,
|
||||
/// Public network identifier.
|
||||
@@ -70,6 +72,17 @@ pub struct ConfiguredNetwork {
|
||||
pub active: bool,
|
||||
}
|
||||
|
||||
/// A durable version-scoped cleanup, not an identity ban.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct ForgetNodeOutcome {
|
||||
/// Identity whose observed claim was retired.
|
||||
pub target: EndpointId,
|
||||
/// The retired claim version; future signed claims remain admissible.
|
||||
pub record_version: u64,
|
||||
/// Authenticated peers to which cleanup was queued immediately.
|
||||
pub peers_told: usize,
|
||||
}
|
||||
|
||||
/// A running agent.
|
||||
///
|
||||
/// Cloning gives another handle to the same agent.
|
||||
@@ -90,6 +103,10 @@ struct Inner {
|
||||
events: broadcast::Sender<Event>,
|
||||
networks: RwLock<HashMap<NetworkId, NetworkHandle>>,
|
||||
shutdown: Shutdown,
|
||||
/// Serialises start, stop, switching and conflict cleanup, including teardown.
|
||||
lifecycle: Mutex<()>,
|
||||
range_changed: watch::Sender<()>,
|
||||
range_task: std::sync::Mutex<Option<JoinHandle<()>>>,
|
||||
accept_task: std::sync::Mutex<Option<JoinHandle<()>>>,
|
||||
plugin_task: std::sync::Mutex<Option<JoinHandle<()>>>,
|
||||
transport: std::sync::OnceLock<Arc<dyn PacketTransport>>,
|
||||
@@ -100,6 +117,37 @@ struct Inner {
|
||||
}
|
||||
|
||||
impl Inner {
|
||||
/// Called under the lifecycle lock after startup: an initially
|
||||
/// control-only agent creates its one interface on first enablement.
|
||||
async fn ensure_interface(self: &Arc<Self>) {
|
||||
if self.interface.get().is_some() {
|
||||
return;
|
||||
}
|
||||
if let Some(factory) = self.config.tun_factory.clone() {
|
||||
let carrier = Arc::new(Carrier(Arc::downgrade(self))) as Arc<dyn PacketCarrier>;
|
||||
match crate::overlay::Interface::start(
|
||||
factory,
|
||||
self.config.interface_name.clone(),
|
||||
self.config.interface_mtu,
|
||||
Arc::clone(&self.routes),
|
||||
carrier,
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(interface) => {
|
||||
let _ = self.interface.set(Arc::new(interface));
|
||||
}
|
||||
Err(err) => {
|
||||
let _ = self.events.send(Event::PluginError {
|
||||
network: NetworkId::from_bytes([0u8; 32]),
|
||||
protocol: "overlay".into(),
|
||||
reason: err.to_string(),
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The name this agent currently answers to.
|
||||
fn read_hostname(&self) -> String {
|
||||
match self.hostname.read() {
|
||||
@@ -165,6 +213,21 @@ impl PacketSink for Sink {
|
||||
let Some(inner) = self.0.upgrade() else {
|
||||
return;
|
||||
};
|
||||
if let Some(payload) = crate::dataplane::transport::health::ip_payload(&packet) {
|
||||
if let Some(header) = crate::overlay::IpHeader::parse(&packet)
|
||||
&& let (std::net::IpAddr::V4(source), std::net::IpAddr::V4(destination)) =
|
||||
(header.source(), header.destination())
|
||||
&& inner.routes.may_send_from(network, peer, source)
|
||||
&& inner.routes.is_local_destination(network, destination)
|
||||
&& let Some(handle) = inner.networks.read().await.get(&network)
|
||||
{
|
||||
let _ = handle.commands.try_send(NetCommand::HealthProbe {
|
||||
peer,
|
||||
packet: bytes::Bytes::copy_from_slice(payload),
|
||||
});
|
||||
}
|
||||
return;
|
||||
}
|
||||
let Some(interface) = inner.interface.get() else {
|
||||
return;
|
||||
};
|
||||
@@ -183,6 +246,35 @@ impl PacketSink for Sink {
|
||||
#[derive(Debug)]
|
||||
struct TransportCtx(Weak<Inner>);
|
||||
|
||||
impl crate::dataplane::transport::ControlSender for TransportCtx {
|
||||
fn send<'a>(
|
||||
&'a self,
|
||||
network: NetworkId,
|
||||
peer: EndpointId,
|
||||
protocol: &'a str,
|
||||
payload: crate::proto::message::TransportPayload,
|
||||
) -> crate::BoxFuture<'a, std::result::Result<(), crate::dataplane::transport::TransportError>>
|
||||
{
|
||||
Box::pin(async move {
|
||||
let inner = self
|
||||
.0
|
||||
.upgrade()
|
||||
.ok_or(crate::dataplane::transport::TransportError::Closed)?;
|
||||
Agent { inner }
|
||||
.send(
|
||||
network,
|
||||
peer,
|
||||
crate::proto::message::ControlMessage::Transport {
|
||||
protocol: protocol.into(),
|
||||
payload,
|
||||
},
|
||||
)
|
||||
.await
|
||||
.map_err(|err| crate::dataplane::transport::TransportError::Other(err.to_string()))
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
impl TransportContext for TransportCtx {
|
||||
fn snapshot(&self) -> crate::BoxFuture<'_, HashMap<NetworkId, NetworkKeys>> {
|
||||
Box::pin(async move {
|
||||
@@ -217,12 +309,133 @@ impl TransportContext for TransportCtx {
|
||||
}
|
||||
|
||||
impl Agent {
|
||||
/// Atomically saves or removes a version-checked shared VIP draft.
|
||||
/// Every authenticated network member may edit any service.
|
||||
pub async fn save_vip(
|
||||
&self,
|
||||
network: NetworkId,
|
||||
config: crate::vip::VipConfig,
|
||||
expected: Option<String>,
|
||||
delete: bool,
|
||||
) -> Result<Vec<crate::vip::VipStatus>> {
|
||||
let (reply, receive) = oneshot::channel();
|
||||
self.command(
|
||||
network,
|
||||
NetCommand::SaveVip {
|
||||
config,
|
||||
expected,
|
||||
delete,
|
||||
reply,
|
||||
},
|
||||
)
|
||||
.await?;
|
||||
receive.await.map_err(|_| Error::Stopped)?
|
||||
}
|
||||
/// Retires one offline member's observed claim without revoking membership.
|
||||
/// Every member has this right. The version must still match the selection.
|
||||
pub async fn forget_node(
|
||||
&self,
|
||||
network: NetworkId,
|
||||
peer: EndpointId,
|
||||
expected_version: u64,
|
||||
) -> Result<ForgetNodeOutcome> {
|
||||
let _lifecycle = self.inner.lifecycle.lock().await;
|
||||
let (reply, receive) = oneshot::channel();
|
||||
self.command(
|
||||
network,
|
||||
NetCommand::ForgetNode {
|
||||
peer,
|
||||
expected_version,
|
||||
reply,
|
||||
},
|
||||
)
|
||||
.await?;
|
||||
receive.await.map_err(|_| Error::Stopped)?
|
||||
}
|
||||
fn effective_protocols(&self, saved: Option<&Vec<String>>) -> Vec<String> {
|
||||
saved
|
||||
.or(self.inner.config.default_protocols.as_ref())
|
||||
.cloned()
|
||||
.unwrap_or_else(|| {
|
||||
self.inner
|
||||
.config
|
||||
.plugins
|
||||
.iter()
|
||||
.map(|p| p.protocol_id().to_owned())
|
||||
.collect()
|
||||
})
|
||||
}
|
||||
|
||||
fn selected_plugins(&self, protocols: &[String]) -> Vec<crate::dataplane::SharedPlugin> {
|
||||
protocols
|
||||
.iter()
|
||||
.filter_map(|name| {
|
||||
self.inner
|
||||
.config
|
||||
.plugins
|
||||
.iter()
|
||||
.find(|p| p.protocol_id() == name)
|
||||
.cloned()
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Saves and immediately applies a network's ordered local protocol set.
|
||||
/// An empty set disables data traffic while retaining control sessions.
|
||||
pub async fn set_protocols(&self, network_id: NetworkId, protocols: Vec<String>) -> Result<()> {
|
||||
crate::dataplane::validate_protocols(&protocols, self.inner.limits.max_capabilities)?;
|
||||
for name in &protocols {
|
||||
if !self
|
||||
.inner
|
||||
.config
|
||||
.plugins
|
||||
.iter()
|
||||
.any(|p| p.protocol_id() == name)
|
||||
{
|
||||
return Err(Error::InvalidProtocols(format!(
|
||||
"this agent has no `{name}` plugin"
|
||||
)));
|
||||
}
|
||||
}
|
||||
let _lifecycle = self.inner.lifecycle.lock().await;
|
||||
self.inner
|
||||
.storage
|
||||
.set_protocols(network_id, protocols.clone())
|
||||
.await?;
|
||||
if self.is_active(network_id).await {
|
||||
let (reply, receive) = oneshot::channel();
|
||||
if !protocols.is_empty() {
|
||||
self.inner.ensure_interface().await;
|
||||
}
|
||||
self.command(
|
||||
network_id,
|
||||
NetCommand::SetProtocols {
|
||||
protocols,
|
||||
interface: self.inner.interface.get().cloned(),
|
||||
reply,
|
||||
},
|
||||
)
|
||||
.await?;
|
||||
receive.await.map_err(|_| Error::Stopped)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
/// Starts an agent.
|
||||
///
|
||||
/// Opens the state store (taking its ownership lock), restores the
|
||||
/// persistent device identity, binds the iroh endpoint and activates every
|
||||
/// configured network whose auto-start flag is set.
|
||||
pub async fn spawn(config: AgentConfig) -> Result<Self> {
|
||||
if let Some(protocols) = &config.default_protocols {
|
||||
crate::dataplane::validate_protocols(protocols, config.limits.max_capabilities)?;
|
||||
for name in protocols {
|
||||
if !config.plugins.iter().any(|p| p.protocol_id() == name) {
|
||||
return Err(Error::InvalidProtocols(format!(
|
||||
"this agent has no `{name}` plugin"
|
||||
)));
|
||||
}
|
||||
}
|
||||
}
|
||||
let storage = Storage::open(&config.paths)?;
|
||||
let identity = storage.device_identity().await?;
|
||||
let adapter = EndpointAdapter::bind(&config, &identity).await?;
|
||||
@@ -234,6 +447,7 @@ impl Agent {
|
||||
let (events, _) = broadcast::channel(config.limits.event_buffer);
|
||||
let limits = Arc::new(config.limits.clone());
|
||||
|
||||
let (range_changed, range_updates) = watch::channel(());
|
||||
let inner = Arc::new(Inner {
|
||||
limits,
|
||||
storage,
|
||||
@@ -243,6 +457,9 @@ impl Agent {
|
||||
events,
|
||||
networks: RwLock::new(HashMap::new()),
|
||||
shutdown: Shutdown::new(),
|
||||
lifecycle: Mutex::new(()),
|
||||
range_changed,
|
||||
range_task: std::sync::Mutex::new(None),
|
||||
accept_task: std::sync::Mutex::new(None),
|
||||
plugin_task: std::sync::Mutex::new(None),
|
||||
transport: std::sync::OnceLock::new(),
|
||||
@@ -251,31 +468,13 @@ impl Agent {
|
||||
config,
|
||||
});
|
||||
|
||||
// One interface for the agent, if it was given a way to make one.
|
||||
// Failing to is reported and not fatal: the control plane works, and
|
||||
// so do the protocols, they just have nowhere to put packets.
|
||||
if let Some(factory) = inner.config.tun_factory.clone() {
|
||||
let carrier = Arc::new(Carrier(Arc::downgrade(&inner))) as Arc<dyn PacketCarrier>;
|
||||
match crate::overlay::Interface::start(
|
||||
factory,
|
||||
inner.config.interface_name.clone(),
|
||||
inner.config.interface_mtu,
|
||||
Arc::clone(&inner.routes),
|
||||
carrier,
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(interface) => {
|
||||
let _ = inner.interface.set(Arc::new(interface));
|
||||
}
|
||||
Err(err) => {
|
||||
let _ = inner.events.send(Event::PluginError {
|
||||
network: NetworkId::from_bytes([0u8; 32]),
|
||||
protocol: "overlay".into(),
|
||||
reason: err.to_string(),
|
||||
});
|
||||
}
|
||||
}
|
||||
if inner
|
||||
.config
|
||||
.default_protocols
|
||||
.as_ref()
|
||||
.is_none_or(|names| !names.is_empty())
|
||||
{
|
||||
inner.ensure_interface().await;
|
||||
}
|
||||
|
||||
// The data plane rides on iroh too, which is where it gets hole
|
||||
@@ -294,6 +493,7 @@ impl Agent {
|
||||
inner.config.custom_transports.clone(),
|
||||
))
|
||||
};
|
||||
transport.attach_control(Arc::new(TransportCtx(Arc::downgrade(&inner))));
|
||||
let _ = inner.transport.set(transport);
|
||||
|
||||
if let CacheOutcome::Reset(reason) = inner.storage.cache_outcome().clone() {
|
||||
@@ -311,8 +511,9 @@ impl Agent {
|
||||
if !inner.config.plugins.is_empty() {
|
||||
let (plugin_tx, plugin_rx) = mpsc::channel(64);
|
||||
let sink = inner
|
||||
.interface
|
||||
.get()
|
||||
.config
|
||||
.tun_factory
|
||||
.as_ref()
|
||||
.map(|_| Arc::new(Sink(Arc::downgrade(&inner))) as Arc<dyn PacketSink>);
|
||||
let context = PluginContext::new(plugin_tx, inner.identity.endpoint_id(), sink);
|
||||
for plugin in &inner.config.plugins {
|
||||
@@ -326,11 +527,16 @@ impl Agent {
|
||||
|
||||
let agent = Self { inner };
|
||||
|
||||
for stored in agent.inner.storage.list_networks().await? {
|
||||
if stored.auto_start {
|
||||
let keys = NetworkKeys::derive(&stored.name, &stored.secret);
|
||||
agent.activate_with_keys(keys).await?;
|
||||
}
|
||||
{
|
||||
let _guard = agent.inner.lifecycle.lock().await;
|
||||
agent.resume_networks_locked().await?;
|
||||
}
|
||||
let task = tokio::spawn(range_conflict_loop(
|
||||
Arc::downgrade(&agent.inner),
|
||||
range_updates,
|
||||
));
|
||||
if let Ok(mut guard) = agent.inner.range_task.lock() {
|
||||
*guard = Some(task);
|
||||
}
|
||||
|
||||
Ok(agent)
|
||||
@@ -374,100 +580,65 @@ impl Agent {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Reserves the configured overlay range for a network, if it can.
|
||||
///
|
||||
/// Done here, where activations are serialised, rather than inside the
|
||||
/// runtime: whether a range is free depends on what the other networks
|
||||
/// took, and deciding that in a task would make the answer depend on
|
||||
/// which task ran first.
|
||||
///
|
||||
/// `None` means this agent will not propose a range for that network and
|
||||
/// waits to adopt whatever it settles on. One agent has one interface, so
|
||||
/// proposing a range it could not route would be worse than having none:
|
||||
/// the lowest author's range wins, and the collision would spread.
|
||||
async fn reserve_range(&self, network: NetworkId) -> RangePlan {
|
||||
let reserve = |wanted: Ipv4Range| {
|
||||
let reservation = crate::overlay::NetworkRoutes {
|
||||
exit: Default::default(),
|
||||
broadcast: Default::default(),
|
||||
range: Some(wanted),
|
||||
local: None,
|
||||
peers: Vec::new(),
|
||||
};
|
||||
self.inner.routes.set_network(network, reservation).is_ok()
|
||||
};
|
||||
|
||||
// If this network already has signed records in storage, check if its
|
||||
// members have already agreed on a range. If so, preserve and reserve
|
||||
// THAT range rather than preemptively seizing the default configured range.
|
||||
let stored_records = self
|
||||
.inner
|
||||
.storage
|
||||
.signed_records(network)
|
||||
.await
|
||||
.unwrap_or_default();
|
||||
let mut state_set = crate::state::StateSet::new();
|
||||
state_set.merge_all(network, stored_records);
|
||||
if let Some(agreed) = state_set.agreed_range() {
|
||||
if reserve(agreed) {
|
||||
return RangePlan {
|
||||
propose: if Some(agreed) == self.inner.config.overlay_ipv4_range {
|
||||
Some(agreed)
|
||||
} else {
|
||||
None
|
||||
},
|
||||
..RangePlan::default()
|
||||
};
|
||||
/// Signed state always wins over a default or an explicit proposal.
|
||||
async fn desired_range(&self, network: NetworkId) -> Result<Option<Ipv4Range>> {
|
||||
let records = self.inner.storage.signed_records(network).await?;
|
||||
let mut state = crate::state::StateSet::new();
|
||||
state.merge_all(network, records);
|
||||
Ok(state.agreed_range().or_else(|| {
|
||||
if self.inner.config.overlay_ipv4_auto {
|
||||
Some(crate::state::derived_ipv4_range(network))
|
||||
} else {
|
||||
tracing::warn!(
|
||||
%network,
|
||||
range = %agreed,
|
||||
"agreed range for this network overlaps another network already active on this interface"
|
||||
);
|
||||
return RangePlan {
|
||||
conflict: Some(agreed),
|
||||
..RangePlan::default()
|
||||
};
|
||||
self.inner.config.overlay_ipv4_range
|
||||
}
|
||||
}))
|
||||
}
|
||||
|
||||
async fn range_conflicts(
|
||||
&self,
|
||||
network: NetworkId,
|
||||
range: Option<Ipv4Range>,
|
||||
) -> Result<Vec<RangeConflict>> {
|
||||
let Some(range) = range else {
|
||||
return Ok(Vec::new());
|
||||
};
|
||||
let overlapping = self.inner.routes.overlapping_networks(network, range);
|
||||
let stored = self.inner.storage.list_networks().await?;
|
||||
Ok(overlapping
|
||||
.into_iter()
|
||||
.filter_map(|(id, range)| {
|
||||
let name = stored.iter().find(|n| n.network_id == id)?.name.clone();
|
||||
Some(RangeConflict {
|
||||
network_id: id,
|
||||
name,
|
||||
range,
|
||||
})
|
||||
})
|
||||
.collect())
|
||||
}
|
||||
|
||||
/// Starts requested networks in saved preference order, skipping overlaps.
|
||||
/// Called with the lifecycle lock held; skipped networks remain requested.
|
||||
async fn resume_networks_locked(&self) -> Result<()> {
|
||||
let order = self.inner.storage.network_start_order().await?;
|
||||
let mut stored = self.inner.storage.list_networks().await?;
|
||||
stored.sort_by_key(|n| {
|
||||
order
|
||||
.iter()
|
||||
.position(|id| *id == n.network_id)
|
||||
.unwrap_or(usize::MAX)
|
||||
});
|
||||
for network in stored {
|
||||
if !network.auto_start || self.is_active(network.network_id).await {
|
||||
continue;
|
||||
}
|
||||
let keys = NetworkKeys::derive(&network.name, &network.secret);
|
||||
match self.activate_with_keys_locked(keys).await {
|
||||
Ok(()) | Err(Error::NetworkRangeConflict { .. }) => {}
|
||||
Err(err) => return Err(err),
|
||||
}
|
||||
}
|
||||
|
||||
let Some(configured) = self.inner.config.overlay_ipv4_range else {
|
||||
return RangePlan::default();
|
||||
};
|
||||
|
||||
// The configured range first, so a network a device has always had
|
||||
// keeps the addresses it has always had.
|
||||
if reserve(configured) {
|
||||
return RangePlan {
|
||||
propose: Some(configured),
|
||||
..RangePlan::default()
|
||||
};
|
||||
}
|
||||
|
||||
// A second network on the same agent cannot have it — one agent,
|
||||
// one interface — so it falls back to the range derived from its
|
||||
// own id, which every one of its members derives identically
|
||||
// without being told.
|
||||
let derived = crate::state::derived_ipv4_range(network);
|
||||
if reserve(derived) {
|
||||
tracing::info!(
|
||||
%network,
|
||||
range = %derived,
|
||||
"another network here holds the configured range; this one uses the \
|
||||
range derived from its id unless its members settled another"
|
||||
);
|
||||
return RangePlan {
|
||||
fallback: Some(derived),
|
||||
conflict: Some(configured),
|
||||
..RangePlan::default()
|
||||
};
|
||||
}
|
||||
|
||||
tracing::info!(%network, "not proposing a range for this network");
|
||||
RangePlan {
|
||||
conflict: Some(configured),
|
||||
..RangePlan::default()
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// The overlay interface this agent owns, when it has one.
|
||||
@@ -546,6 +717,9 @@ impl Agent {
|
||||
/// The same `(name, secret)` always produces the same [`NetworkId`], on
|
||||
/// every device.
|
||||
///
|
||||
/// An overlapping network stays configured and requested, but inactive;
|
||||
/// use [`Agent::switch_network`] to suspend the blockers explicitly.
|
||||
///
|
||||
/// This is declarative and therefore **idempotent**: joining a network
|
||||
/// that is already active succeeds and changes nothing. That matters
|
||||
/// because a configured network is activated automatically at startup, so
|
||||
@@ -599,7 +773,9 @@ impl Agent {
|
||||
}
|
||||
match self.activate_with_keys(keys).await {
|
||||
// Already a member of exactly this network space: nothing to do.
|
||||
Ok(()) | Err(Error::NetworkAlreadyActive(_)) => Ok(network_id),
|
||||
Ok(())
|
||||
| Err(Error::NetworkAlreadyActive(_))
|
||||
| Err(Error::NetworkRangeConflict { .. }) => Ok(network_id),
|
||||
Err(err) => Err(err),
|
||||
}
|
||||
}
|
||||
@@ -703,7 +879,44 @@ impl Agent {
|
||||
self.activate_with_keys(keys).await
|
||||
}
|
||||
|
||||
/// Switches to a configured network, suspending every overlapping network.
|
||||
/// Suspended networks retain their auto-start request and resume when free.
|
||||
/// Returns the networks suspended by this request.
|
||||
pub async fn switch_network(&self, network_id: NetworkId) -> Result<Vec<NetworkId>> {
|
||||
let _guard = self.inner.lifecycle.lock().await;
|
||||
if self.inner.shutdown.is_triggered() {
|
||||
return Err(Error::Stopped);
|
||||
}
|
||||
let stored = self
|
||||
.inner
|
||||
.storage
|
||||
.list_networks()
|
||||
.await?
|
||||
.into_iter()
|
||||
.find(|n| n.network_id == network_id)
|
||||
.ok_or(Error::NetworkUnknown(network_id))?;
|
||||
let range = self.desired_range(network_id).await?;
|
||||
let conflicts = self.range_conflicts(network_id, range).await?;
|
||||
self.inner.storage.prefer_network(network_id).await?;
|
||||
let mut stopped = Vec::new();
|
||||
for conflict in conflicts {
|
||||
self.stop_network_locked(conflict.network_id).await?;
|
||||
stopped.push(conflict.network_id);
|
||||
}
|
||||
if !self.is_active(network_id).await {
|
||||
self.activate_with_keys_locked(NetworkKeys::derive(&stored.name, &stored.secret))
|
||||
.await?;
|
||||
}
|
||||
self.resume_networks_locked().await?;
|
||||
Ok(stopped)
|
||||
}
|
||||
|
||||
async fn activate_with_keys(&self, keys: NetworkKeys) -> Result<()> {
|
||||
let _guard = self.inner.lifecycle.lock().await;
|
||||
self.activate_with_keys_locked(keys).await
|
||||
}
|
||||
|
||||
async fn activate_with_keys_locked(&self, keys: NetworkKeys) -> Result<()> {
|
||||
if self.inner.shutdown.is_triggered() {
|
||||
return Err(Error::Stopped);
|
||||
}
|
||||
@@ -712,8 +925,70 @@ impl Agent {
|
||||
if networks.contains_key(&network_id) {
|
||||
return Err(Error::NetworkAlreadyActive(network_id));
|
||||
}
|
||||
|
||||
let range = self.reserve_range(network_id).await;
|
||||
let stored = self
|
||||
.inner
|
||||
.storage
|
||||
.list_networks()
|
||||
.await?
|
||||
.into_iter()
|
||||
.find(|stored| stored.network_id == network_id);
|
||||
// Mandatory cleanup proofs must never be silently dropped on startup.
|
||||
let retirements = self
|
||||
.inner
|
||||
.storage
|
||||
.retirements(network_id, self.inner.limits.max_retirements)
|
||||
.await?;
|
||||
let vip_entries = self
|
||||
.inner
|
||||
.storage
|
||||
.vip_fields(
|
||||
network_id,
|
||||
self.inner
|
||||
.limits
|
||||
.max_vips
|
||||
.saturating_mul(self.inner.limits.max_vip_fields)
|
||||
.saturating_add(self.inner.limits.max_vip_tombstones),
|
||||
)
|
||||
.await?;
|
||||
let mut vips = crate::vip::VipSet::default();
|
||||
for id in vip_entries
|
||||
.iter()
|
||||
.map(|e| e.vip.clone())
|
||||
.collect::<std::collections::BTreeSet<_>>()
|
||||
{
|
||||
vips.merge(
|
||||
network_id,
|
||||
vip_entries
|
||||
.iter()
|
||||
.filter(|e| e.vip == id)
|
||||
.cloned()
|
||||
.collect(),
|
||||
&self.inner.limits,
|
||||
)?;
|
||||
}
|
||||
let range = self.desired_range(network_id).await?;
|
||||
if let Some(range) = range {
|
||||
let conflicts = self.inner.routes.overlapping_networks(network_id, range);
|
||||
if !conflicts.is_empty() {
|
||||
return Err(Error::NetworkRangeConflict {
|
||||
network: network_id,
|
||||
range,
|
||||
conflicts: conflicts.into_iter().map(|(id, _)| id).collect(),
|
||||
});
|
||||
}
|
||||
self.inner
|
||||
.routes
|
||||
.set_network(
|
||||
network_id,
|
||||
crate::overlay::NetworkRoutes {
|
||||
range: Some(range),
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.map_err(|err| Error::Rejected {
|
||||
reason: err.to_string(),
|
||||
})?;
|
||||
}
|
||||
let mut backends = Vec::new();
|
||||
if let Some(discovery) = &self.inner.config.discovery {
|
||||
backends.push(discovery.clone());
|
||||
@@ -729,20 +1004,22 @@ impl Agent {
|
||||
as Arc<dyn crate::discovery::NetworkDiscovery>,
|
||||
)
|
||||
};
|
||||
let stored = self
|
||||
.inner
|
||||
.storage
|
||||
.list_networks()
|
||||
.await?
|
||||
.into_iter()
|
||||
.find(|stored| stored.network_id == network_id);
|
||||
|
||||
let broadcast = stored.as_ref().is_none_or(|stored| stored.broadcast);
|
||||
let protocols =
|
||||
self.effective_protocols(stored.as_ref().and_then(|s| s.protocols.as_ref()));
|
||||
let plugins = self.selected_plugins(&protocols);
|
||||
if !plugins.is_empty() {
|
||||
self.inner.ensure_interface().await;
|
||||
}
|
||||
let exit_offer = stored.as_ref().is_some_and(|stored| stored.exit_node);
|
||||
let exit_via = stored
|
||||
.as_ref()
|
||||
.and_then(|stored| stored.exit_via)
|
||||
.and_then(|bytes| EndpointId::from_bytes(&bytes).ok());
|
||||
let handle = network::spawn(RuntimeParams {
|
||||
vips,
|
||||
retirements,
|
||||
broadcast,
|
||||
exit_offer,
|
||||
exit_via,
|
||||
@@ -755,7 +1032,8 @@ impl Agent {
|
||||
discovery,
|
||||
discovery_interval: self.inner.config.discovery_interval,
|
||||
discovery_policy: self.inner.config.discovery_policy.clone(),
|
||||
plugins: self.inner.config.plugins.clone(),
|
||||
available_plugins: self.inner.config.plugins.clone(),
|
||||
plugins: plugins.clone(),
|
||||
routes: Arc::clone(&self.inner.routes),
|
||||
interface: self.inner.interface.get().cloned(),
|
||||
hostname: self.inner.read_hostname(),
|
||||
@@ -763,14 +1041,13 @@ impl Agent {
|
||||
device_secret: self.inner.identity.signing_key(),
|
||||
#[cfg(feature = "testing")]
|
||||
unreachable_data_peers: Arc::clone(&self.inner.config.unreachable_data_peers),
|
||||
ipv4_range: range.propose,
|
||||
ipv4_fallback: range.fallback,
|
||||
range_conflict: range.conflict,
|
||||
ipv4_range: range,
|
||||
range_changed: self.inner.range_changed.clone(),
|
||||
});
|
||||
networks.insert(network_id, handle);
|
||||
drop(networks);
|
||||
|
||||
for plugin in &self.inner.config.plugins {
|
||||
for plugin in &plugins {
|
||||
plugin.on_network_activated(network_id);
|
||||
}
|
||||
|
||||
@@ -786,6 +1063,30 @@ impl Agent {
|
||||
/// it does not remove this agent from anyone else's view of the network.
|
||||
/// Other networks keep running.
|
||||
pub async fn deactivate_network(&self, network_id: NetworkId) -> Result<()> {
|
||||
let _guard = self.inner.lifecycle.lock().await;
|
||||
if !self.is_active(network_id).await {
|
||||
return Err(Error::NetworkNotActive(network_id));
|
||||
}
|
||||
self.inner.storage.set_auto_start(network_id, false).await?;
|
||||
self.stop_network_locked(network_id).await?;
|
||||
self.resume_networks_locked().await
|
||||
}
|
||||
|
||||
/// Stops a configured network, including one already suspended by overlap.
|
||||
/// Clears its auto-start request so it will not resume behind the user.
|
||||
pub async fn pause_network(&self, network_id: NetworkId) -> Result<()> {
|
||||
let _guard = self.inner.lifecycle.lock().await;
|
||||
if self.inner.shutdown.is_triggered() {
|
||||
return Err(Error::Stopped);
|
||||
}
|
||||
self.inner.storage.set_auto_start(network_id, false).await?;
|
||||
if self.is_active(network_id).await {
|
||||
self.stop_network_locked(network_id).await?;
|
||||
}
|
||||
self.resume_networks_locked().await
|
||||
}
|
||||
|
||||
async fn stop_network_locked(&self, network_id: NetworkId) -> Result<()> {
|
||||
let handle = {
|
||||
let mut networks = self.inner.networks.write().await;
|
||||
networks
|
||||
@@ -814,7 +1115,6 @@ impl Agent {
|
||||
interface.sync_broadcast_rules().await;
|
||||
interface.sync_exit_rules().await;
|
||||
}
|
||||
self.inner.storage.set_auto_start(network_id, false).await?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -824,17 +1124,15 @@ impl Agent {
|
||||
/// Local and silent: nobody is told. [`Agent::leave_network`] is the one
|
||||
/// that says goodbye first, and is what a person means by leaving.
|
||||
pub async fn forget_network(&self, network_id: NetworkId) -> Result<()> {
|
||||
let _guard = self.inner.lifecycle.lock().await;
|
||||
if self.is_active(network_id).await {
|
||||
self.deactivate_network(network_id).await?;
|
||||
self.stop_network_locked(network_id).await?;
|
||||
}
|
||||
self.inner.storage.remove_network(network_id).await?;
|
||||
// Whatever a protocol kept for this network goes with it. The
|
||||
// agent cannot know what that is, only that there is no longer any
|
||||
// reason to hold it.
|
||||
for plugin in &self.inner.config.plugins {
|
||||
plugin.on_network_forgotten(network_id);
|
||||
}
|
||||
Ok(())
|
||||
self.resume_networks_locked().await
|
||||
}
|
||||
|
||||
/// Leaves a network: gives up what was claimed, then forgets it.
|
||||
@@ -890,6 +1188,7 @@ impl Agent {
|
||||
.await?
|
||||
.into_iter()
|
||||
.map(|stored| ConfiguredNetwork {
|
||||
protocols: self.effective_protocols(stored.protocols.as_ref()),
|
||||
broadcast: stored.broadcast,
|
||||
network_id: stored.network_id,
|
||||
name: stored.name,
|
||||
@@ -958,13 +1257,54 @@ impl Agent {
|
||||
let mut networks = Vec::new();
|
||||
for stored in self.inner.storage.list_networks().await? {
|
||||
if active.contains(&stored.network_id)
|
||||
&& let Ok(status) = self.network_status(stored.network_id).await
|
||||
&& let Ok(mut status) = self.network_status(stored.network_id).await
|
||||
{
|
||||
status.auto_start = stored.auto_start;
|
||||
networks.push(status);
|
||||
continue;
|
||||
}
|
||||
let keys = NetworkKeys::derive(&stored.name, &stored.secret);
|
||||
let range = self.desired_range(stored.network_id).await?;
|
||||
let range_conflicts = self.range_conflicts(stored.network_id, range).await?;
|
||||
let fields = self
|
||||
.inner
|
||||
.storage
|
||||
.vip_fields(
|
||||
stored.network_id,
|
||||
self.inner
|
||||
.limits
|
||||
.max_vips
|
||||
.saturating_mul(self.inner.limits.max_vip_fields)
|
||||
.saturating_add(self.inner.limits.max_vip_tombstones),
|
||||
)
|
||||
.await?;
|
||||
let mut service_state = crate::vip::VipSet::default();
|
||||
for id in fields
|
||||
.iter()
|
||||
.map(|e| e.vip.clone())
|
||||
.collect::<std::collections::BTreeSet<_>>()
|
||||
{
|
||||
service_state.merge(
|
||||
stored.network_id,
|
||||
fields.iter().filter(|e| e.vip == id).cloned().collect(),
|
||||
&self.inner.limits,
|
||||
)?;
|
||||
}
|
||||
let vips = service_state
|
||||
.configs()
|
||||
.into_iter()
|
||||
.map(|config| crate::vip::VipStatus {
|
||||
revision: service_state.revision(&config.id),
|
||||
config,
|
||||
address: None,
|
||||
error: Some("network is inactive".into()),
|
||||
listeners: vec![],
|
||||
})
|
||||
.collect();
|
||||
networks.push(NetworkStatus {
|
||||
vips,
|
||||
protocols: self.effective_protocols(stored.protocols.as_ref()),
|
||||
auto_start: stored.auto_start,
|
||||
broadcast: stored.broadcast,
|
||||
exit_offer: stored.exit_node,
|
||||
exit_via: stored
|
||||
@@ -978,11 +1318,13 @@ impl Agent {
|
||||
state: NetworkState::Inactive,
|
||||
peers: Vec::new(),
|
||||
candidates: Vec::new(),
|
||||
// An inactive network has no runtime to ask; the roster and
|
||||
// the range come from one. Empty, not invented.
|
||||
// The range is available from signed state or the deterministic
|
||||
// default even while the network has no runtime.
|
||||
members: Vec::new(),
|
||||
range: None,
|
||||
range_conflict: None,
|
||||
hostname_conflict: None,
|
||||
range,
|
||||
range_conflict: range_conflicts.first().map(|conflict| conflict.range),
|
||||
range_conflicts,
|
||||
metrics: NetworkMetrics::default(),
|
||||
relay: Default::default(),
|
||||
});
|
||||
@@ -1038,6 +1380,7 @@ impl Agent {
|
||||
/// After this returns, the state directory can be opened by another agent.
|
||||
pub async fn shutdown(&self) {
|
||||
self.inner.shutdown.trigger();
|
||||
let lifecycle_guard = self.inner.lifecycle.lock().await;
|
||||
|
||||
let handles: Vec<NetworkHandle> = {
|
||||
let mut networks = self.inner.networks.write().await;
|
||||
@@ -1047,6 +1390,7 @@ impl Agent {
|
||||
handle.stop().await;
|
||||
}
|
||||
|
||||
drop(lifecycle_guard);
|
||||
if let Some(dht) = &self.inner.config.dht {
|
||||
dht.shutdown().await;
|
||||
}
|
||||
@@ -1063,6 +1407,7 @@ impl Agent {
|
||||
for (what, handle) in [
|
||||
("inbound connections", &self.inner.accept_task),
|
||||
("plugin requests", &self.inner.plugin_task),
|
||||
("range conflicts", &self.inner.range_task),
|
||||
] {
|
||||
let task = handle.lock().ok().and_then(|mut guard| guard.take());
|
||||
if let Some(task) = task {
|
||||
@@ -1106,16 +1451,44 @@ impl Agent {
|
||||
}
|
||||
}
|
||||
|
||||
/// What a network may propose as its overlay range, and what it cannot have.
|
||||
#[derive(Debug, Clone, Copy, Default)]
|
||||
struct RangePlan {
|
||||
/// Proposed straight away: the configured range, when it is free here.
|
||||
propose: Option<Ipv4Range>,
|
||||
/// Proposed after a moment, when the configured range is another
|
||||
/// network's: the range derived from this network's own id.
|
||||
fallback: Option<Ipv4Range>,
|
||||
/// The configured range, when this network cannot have it.
|
||||
conflict: Option<Ipv4Range>,
|
||||
/// Processes coalesced conflict notifications without blocking a network actor.
|
||||
async fn range_conflict_loop(inner: Weak<Inner>, mut updates: watch::Receiver<()>) {
|
||||
loop {
|
||||
let Some(owner) = inner.upgrade() else {
|
||||
return;
|
||||
};
|
||||
let shutdown = owner.shutdown.clone();
|
||||
drop(owner);
|
||||
tokio::select! {
|
||||
_ = shutdown.wait() => return,
|
||||
result = updates.changed() => if result.is_err() { return; },
|
||||
}
|
||||
let Some(inner) = inner.upgrade() else {
|
||||
return;
|
||||
};
|
||||
let agent = Agent { inner };
|
||||
let _guard = agent.inner.lifecycle.lock().await;
|
||||
if agent.inner.shutdown.is_triggered() {
|
||||
return;
|
||||
}
|
||||
let blocked: Vec<_> = agent
|
||||
.inner
|
||||
.networks
|
||||
.read()
|
||||
.await
|
||||
.iter()
|
||||
.filter(|(_, h)| h.range_blocked.load(std::sync::atomic::Ordering::Acquire))
|
||||
.map(|(id, _)| *id)
|
||||
.collect();
|
||||
for id in blocked {
|
||||
if let Err(err) = agent.stop_network_locked(id).await {
|
||||
tracing::warn!(%err, "cannot suspend overlapping network");
|
||||
}
|
||||
}
|
||||
if let Err(err) = agent.resume_networks_locked().await {
|
||||
tracing::warn!(%err, "cannot resume suspended networks");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// How long a release is given to reach the sessions it was queued on.
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -70,7 +70,7 @@ pub(crate) struct Session {
|
||||
pub(crate) conn: Connection,
|
||||
/// Already encoded frame payloads, so the runtime can account for the exact
|
||||
/// number of control bytes it queues.
|
||||
pub(crate) outbound: mpsc::Sender<Vec<u8>>,
|
||||
pub(crate) outbound: mpsc::Sender<zeroize::Zeroizing<Vec<u8>>>,
|
||||
pub(crate) hostname: Option<String>,
|
||||
pub(crate) capabilities: Vec<PluginCapability>,
|
||||
pub(crate) broadcast: bool,
|
||||
@@ -244,7 +244,7 @@ pub(crate) fn spawn(
|
||||
|
||||
async fn writer_task(
|
||||
mut send: SendStream,
|
||||
mut outbound: mpsc::Receiver<Vec<u8>>,
|
||||
mut outbound: mpsc::Receiver<zeroize::Zeroizing<Vec<u8>>>,
|
||||
limits: Arc<Limits>,
|
||||
shutdown: Shutdown,
|
||||
parent: Shutdown,
|
||||
@@ -331,6 +331,7 @@ async fn reader_task(
|
||||
}
|
||||
};
|
||||
|
||||
let payload = zeroize::Zeroizing::new(payload);
|
||||
let bytes = payload.len();
|
||||
let envelope: Envelope = match decode(&payload) {
|
||||
Ok(envelope) => envelope,
|
||||
|
||||
@@ -59,6 +59,43 @@ pub struct OverlayStatus {
|
||||
pub broadcast_rules: Option<crate::overlay::BroadcastRulesReport>,
|
||||
}
|
||||
|
||||
/// How packets reach a member right now.
|
||||
///
|
||||
/// Being reachable is one thing and being reachable well is another, and
|
||||
/// the second is never a reason to call a member away: a member that can only
|
||||
/// be reached through somebody else is online, over a longer path.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub enum DataPath {
|
||||
/// Nothing carries packets to it yet. The member is connected for
|
||||
/// control only, which is the state of a data path being set up.
|
||||
None,
|
||||
/// A direct link, of the named kind of transport.
|
||||
Direct {
|
||||
/// The kind of transport carrying it, e.g. `wg`.
|
||||
transport: String,
|
||||
},
|
||||
/// Through other members.
|
||||
Relayed {
|
||||
/// Number of links on the way, at least two.
|
||||
hops: u8,
|
||||
/// The member the first of them goes to.
|
||||
via: EndpointId,
|
||||
},
|
||||
}
|
||||
|
||||
/// A name that a member claimed and somebody else holds.
|
||||
///
|
||||
/// A name belongs to whoever claimed it first. The member that claimed it
|
||||
/// later is still a member and still reachable by its address; it is just
|
||||
/// not found by that name, on any member of the network.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct HostnameConflict {
|
||||
/// The name that is taken.
|
||||
pub name: String,
|
||||
/// The member that holds it.
|
||||
pub holder: EndpointId,
|
||||
}
|
||||
|
||||
/// A member the signed state knows about, connected or not.
|
||||
///
|
||||
/// This is the durable roster: it comes from signed records, so a member that
|
||||
@@ -71,6 +108,8 @@ pub struct OverlayStatus {
|
||||
/// connected.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct MemberStatus {
|
||||
/// Current signed claim version, used to bind cleanup confirmation.
|
||||
pub record_version: u64,
|
||||
/// The member's device identity.
|
||||
pub endpoint_id: EndpointId,
|
||||
/// The IPv4 overlay address it claimed and signed for.
|
||||
@@ -85,6 +124,15 @@ pub struct MemberStatus {
|
||||
/// Not signed, so it is a convenience for naming a member that is away
|
||||
/// and never a claim: the zone is served from `hostname`.
|
||||
pub last_hostname: Option<String>,
|
||||
/// The name it claimed, when somebody else claimed it first.
|
||||
pub hostname_conflict: Option<HostnameConflict>,
|
||||
/// Whether this agent can reach it now, by any path: a live control
|
||||
/// session, or a data path through other members. The one answer to "is
|
||||
/// it online" for every client, so that none of them has to work it out
|
||||
/// from a different part of the picture. This agent counts as online.
|
||||
pub online: bool,
|
||||
/// How packets reach it. [`DataPath::None`] while it is offline.
|
||||
pub data_path: DataPath,
|
||||
}
|
||||
|
||||
/// Status of one authenticated session.
|
||||
@@ -128,6 +176,8 @@ pub struct PeerStatus {
|
||||
pub control_bytes_sent: u64,
|
||||
/// Control payload bytes read from this session.
|
||||
pub control_bytes_received: u64,
|
||||
/// How packets reach this peer right now.
|
||||
pub data_path: DataPath,
|
||||
}
|
||||
|
||||
/// Counters scoped to one logical network.
|
||||
@@ -161,9 +211,26 @@ pub struct NetworkMetrics {
|
||||
pub data_link_failures: u64,
|
||||
}
|
||||
|
||||
/// An active network preventing another network from starting locally.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct RangeConflict {
|
||||
/// The blocking network's public identifier.
|
||||
pub network_id: NetworkId,
|
||||
/// Its configured name.
|
||||
pub name: NetworkName,
|
||||
/// Its overlapping overlay range.
|
||||
pub range: crate::state::Ipv4Range,
|
||||
}
|
||||
|
||||
/// Status of one network.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct NetworkStatus {
|
||||
/// Shared service configuration with this client's routing observations.
|
||||
pub vips: Vec<crate::vip::VipStatus>,
|
||||
/// Enabled local data-plane protocols, in preference order.
|
||||
pub protocols: Vec<String>,
|
||||
/// Whether this network is requested to start or resume automatically.
|
||||
pub auto_start: bool,
|
||||
/// Local broadcast participation in this network.
|
||||
pub broadcast: bool,
|
||||
/// This agent offers itself as an exit node in this network.
|
||||
@@ -188,15 +255,16 @@ pub struct NetworkStatus {
|
||||
pub candidates: Vec<CandidateStatus>,
|
||||
/// Members the signed state knows about, whether connected or not.
|
||||
pub members: Vec<MemberStatus>,
|
||||
/// This agent's own name, when another member claimed it first and so
|
||||
/// holds it. This agent keeps working by address; nobody finds it by
|
||||
/// this name until it is renamed or the other member lets it go.
|
||||
pub hostname_conflict: Option<HostnameConflict>,
|
||||
/// The overlay range this network uses, once it has one.
|
||||
pub range: Option<crate::state::Ipv4Range>,
|
||||
/// The range it could not have, because another network on this agent
|
||||
/// already holds it.
|
||||
///
|
||||
/// One agent has one interface, so an address belongs to one network.
|
||||
/// This network waits to adopt whatever its members settle on instead of
|
||||
/// proposing something it could not route.
|
||||
/// The first overlapping active range; `range_conflicts` identifies all blockers.
|
||||
pub range_conflict: Option<crate::state::Ipv4Range>,
|
||||
/// Active networks that would be suspended by switching to this network.
|
||||
pub range_conflicts: Vec<RangeConflict>,
|
||||
/// Per-network counters.
|
||||
pub metrics: NetworkMetrics,
|
||||
/// What has gone through a peer in the middle, in both directions.
|
||||
|
||||
@@ -158,6 +158,22 @@ pub enum TransportPolicy {
|
||||
/// object where that is possible (notably [`Limits::max_frame_len`]).
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct Limits {
|
||||
/// Maximum active VIP identities per network.
|
||||
pub max_vips: usize,
|
||||
/// Maximum permanent VIP deletion tombstones per network.
|
||||
pub max_vip_tombstones: usize,
|
||||
/// Maximum concurrent active TCP port checks across the agent.
|
||||
pub max_vip_checks: usize,
|
||||
/// Maximum signed fields per VIP, including removed fields.
|
||||
pub max_vip_fields: usize,
|
||||
/// Maximum listeners per VIP.
|
||||
pub max_vip_listeners: usize,
|
||||
/// Maximum backend endpoints per listener.
|
||||
pub max_vip_backends: usize,
|
||||
/// Maximum simultaneous local VIP flow bindings.
|
||||
pub max_vip_flows: usize,
|
||||
/// Maximum incomplete IPv4 datagrams per network in VIP reassembly.
|
||||
pub max_vip_fragments: usize,
|
||||
/// Largest accepted control frame payload, in bytes.
|
||||
pub max_frame_len: usize,
|
||||
/// Largest accepted hostname, in bytes.
|
||||
@@ -166,12 +182,16 @@ pub struct Limits {
|
||||
pub max_capabilities: usize,
|
||||
/// Largest opaque plugin payload, in bytes.
|
||||
pub max_capability_data_len: usize,
|
||||
/// Largest private transport negotiation payload (also hard-capped at 4096 bytes).
|
||||
pub max_transport_payload_len: usize,
|
||||
/// Largest accepted echo payload in a ping/pong exchange, in bytes.
|
||||
pub max_echo_payload_len: usize,
|
||||
/// Largest accepted free-text reason string, in bytes.
|
||||
pub max_reason_len: usize,
|
||||
/// Largest number of signed records accepted in one snapshot.
|
||||
pub max_state_records: usize,
|
||||
/// Maximum durable retirement markers retained per network.
|
||||
pub max_retirements: usize,
|
||||
/// Deadline for the whole handshake.
|
||||
pub handshake_timeout: Duration,
|
||||
/// Deadline for one outbound dial attempt.
|
||||
@@ -201,13 +221,23 @@ pub struct Limits {
|
||||
impl Default for Limits {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
max_vips: 64,
|
||||
max_vip_tombstones: 1024,
|
||||
max_vip_checks: 32,
|
||||
max_vip_fields: 192,
|
||||
max_vip_listeners: 16,
|
||||
max_vip_backends: 32,
|
||||
max_vip_flows: 4096,
|
||||
max_vip_fragments: 64,
|
||||
max_frame_len: 64 * 1024,
|
||||
max_hostname_len: 255,
|
||||
max_capabilities: 16,
|
||||
max_transport_payload_len: 4096,
|
||||
max_capability_data_len: 4 * 1024,
|
||||
max_echo_payload_len: 4 * 1024,
|
||||
max_reason_len: 256,
|
||||
max_state_records: crate::state::MAX_RECORDS_PER_MESSAGE,
|
||||
max_retirements: 1024,
|
||||
handshake_timeout: Duration::from_secs(10),
|
||||
dial_timeout: Duration::from_secs(10),
|
||||
write_timeout: Duration::from_secs(30),
|
||||
@@ -325,6 +355,9 @@ pub struct AgentConfig {
|
||||
pub reconnect: ReconnectPolicy,
|
||||
/// IP plugins whose capabilities are announced and dispatched.
|
||||
pub plugins: Vec<SharedPlugin>,
|
||||
/// Initial ordered selection for networks without a saved override.
|
||||
/// `None` enables all registered plugins; an empty list is control-only.
|
||||
pub default_protocols: Option<Vec<String>>,
|
||||
/// Custom packet transports registered per protocol.
|
||||
pub custom_transports:
|
||||
std::collections::HashMap<String, Arc<dyn crate::dataplane::PacketTransport>>,
|
||||
@@ -339,7 +372,11 @@ pub struct AgentConfig {
|
||||
pub interface_name: String,
|
||||
/// The interface MTU.
|
||||
pub interface_mtu: u32,
|
||||
/// The IPv4 overlay range this agent proposes.
|
||||
/// Whether new networks derive their IPv4 range from their network id.
|
||||
///
|
||||
/// Enabled by default. Explicit ranges and IPv4 opt-out disable this.
|
||||
pub overlay_ipv4_auto: bool,
|
||||
/// The IPv4 overlay range this agent proposes when automatic selection is off.
|
||||
///
|
||||
/// Addresses are allocated from it and recorded in signed state, so a
|
||||
/// participant keeps the same one across restarts. A network that has
|
||||
@@ -369,11 +406,13 @@ impl AgentConfig {
|
||||
limits: Limits::default(),
|
||||
reconnect: ReconnectPolicy::default(),
|
||||
plugins: Vec::new(),
|
||||
default_protocols: None,
|
||||
custom_transports: std::collections::HashMap::new(),
|
||||
tun_factory: None,
|
||||
interface_name: "tsun0".to_string(),
|
||||
interface_mtu: 1280,
|
||||
overlay_ipv4_range: Some(crate::state::DEFAULT_IPV4_RANGE),
|
||||
overlay_ipv4_auto: true,
|
||||
overlay_ipv4_range: None,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -445,6 +484,7 @@ impl AgentConfig {
|
||||
|
||||
/// Sets the IPv4 overlay range this agent proposes, or disables IPv4.
|
||||
pub fn with_overlay_ipv4_range(mut self, range: Option<crate::state::Ipv4Range>) -> Self {
|
||||
self.overlay_ipv4_auto = false;
|
||||
self.overlay_ipv4_range = range;
|
||||
self
|
||||
}
|
||||
|
||||
@@ -40,6 +40,33 @@ pub use tunnel::{GenericTunnelPlugin, TunnelCodec};
|
||||
/// Maximum length of a plugin protocol identifier.
|
||||
pub const MAX_PROTOCOL_ID_LEN: usize = 32;
|
||||
|
||||
/// Validates bounded local protocol preferences before storing or applying them.
|
||||
pub(crate) fn validate_protocols(protocols: &[String], max: usize) -> crate::error::Result<()> {
|
||||
if protocols.len() > max {
|
||||
return Err(crate::Error::InvalidProtocols(format!(
|
||||
"at most {max} protocols are allowed"
|
||||
)));
|
||||
}
|
||||
for (index, name) in protocols.iter().enumerate() {
|
||||
if name.is_empty()
|
||||
|| name.len() > MAX_PROTOCOL_ID_LEN
|
||||
|| !name
|
||||
.bytes()
|
||||
.all(|c| c.is_ascii_alphanumeric() || c == b'-' || c == b'_')
|
||||
{
|
||||
return Err(crate::Error::InvalidProtocols(
|
||||
"names must be 1–32 ASCII letters, digits, hyphens or underscores".into(),
|
||||
));
|
||||
}
|
||||
if protocols[..index].contains(name) {
|
||||
return Err(crate::Error::InvalidProtocols(format!(
|
||||
"`{name}` is listed more than once"
|
||||
)));
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// An announcement of one IP plugin's capability.
|
||||
///
|
||||
/// `data` is opaque to the core. Nothing in it may be interpreted as a shell
|
||||
|
||||
@@ -67,10 +67,41 @@ impl Tally {
|
||||
}
|
||||
}
|
||||
|
||||
/// The links to one neighbour, best first. Which transport carries a frame
|
||||
/// is decided here and nowhere else: the end-to-end protocol is in the
|
||||
/// envelope and never depends on it.
|
||||
#[derive(Debug)]
|
||||
struct NextHop {
|
||||
peer: PeerId,
|
||||
link: SharedLink,
|
||||
links: Box<[SharedLink]>,
|
||||
}
|
||||
|
||||
impl NextHop {
|
||||
fn usable(&self) -> bool {
|
||||
self.links.iter().any(|link| !link.is_closed())
|
||||
}
|
||||
|
||||
fn first_open(&self) -> Option<&SharedLink> {
|
||||
self.links.iter().find(|link| !link.is_closed())
|
||||
}
|
||||
|
||||
/// Hands a frame to the best open link that accepts it. A link that
|
||||
/// cannot carry the frame (too large for its path, closed a moment ago)
|
||||
/// leaves it to the next one; the frame is only copied when there is
|
||||
/// another to try.
|
||||
fn send(&self, frame: Bytes) -> Result<(), TransportError> {
|
||||
let mut open = self.links.iter().filter(|link| !link.is_closed());
|
||||
let Some(mut current) = open.next() else {
|
||||
return Err(TransportError::Closed);
|
||||
};
|
||||
for next in open {
|
||||
if current.send(frame.clone()).is_ok() {
|
||||
return Ok(());
|
||||
}
|
||||
current = next;
|
||||
}
|
||||
current.send(frame)
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
@@ -82,21 +113,22 @@ struct ForwardRoute {
|
||||
impl ForwardRoute {
|
||||
fn select(&self, source: &PeerId, flow: FlowId) -> Option<&NextHop> {
|
||||
if self.next.len() == 1 {
|
||||
return self.next.first().filter(|hop| !hop.link.is_closed());
|
||||
return self.next.first().filter(|hop| hop.usable());
|
||||
}
|
||||
let seed = u64::from_le_bytes(source[..8].try_into().ok()?);
|
||||
let start = mix64(flow ^ seed) as usize % self.next.len();
|
||||
// Stable fallback if a link closed just before its table was replaced.
|
||||
(0..self.next.len())
|
||||
.map(|i| &self.next[(start + i) % self.next.len()])
|
||||
.find(|hop| !hop.link.is_closed())
|
||||
.find(|hop| hop.usable())
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Default)]
|
||||
struct ForwardingTable {
|
||||
routes: HashMap<PeerId, ForwardRoute>,
|
||||
inboxes: HashMap<PeerId, mpsc::Sender<Bytes>>,
|
||||
/// Local delivery, by who sent it and which end-to-end protocol it is.
|
||||
inboxes: HashMap<(PeerId, u64), mpsc::Sender<Bytes>>,
|
||||
members: HashSet<PeerId>,
|
||||
}
|
||||
|
||||
@@ -121,7 +153,7 @@ impl Plane {
|
||||
return;
|
||||
}
|
||||
if header.destination == self.local {
|
||||
let Some(inbox) = table.inboxes.get(&header.source) else {
|
||||
let Some(inbox) = table.inboxes.get(&(header.source, header.protocol)) else {
|
||||
self.tally.dropped_unknown.fetch_add(1, Ordering::Relaxed);
|
||||
return;
|
||||
};
|
||||
@@ -144,7 +176,7 @@ impl Plane {
|
||||
self.tally.dropped_no_link.fetch_add(1, Ordering::Relaxed);
|
||||
return;
|
||||
};
|
||||
if hop.link.send(envelope::decrement(frame)).is_ok() {
|
||||
if hop.send(envelope::decrement(frame)).is_ok() {
|
||||
self.tally.forwarded.fetch_add(1, Ordering::Relaxed);
|
||||
} else {
|
||||
self.tally.dropped_no_link.fetch_add(1, Ordering::Relaxed);
|
||||
@@ -152,27 +184,65 @@ impl Plane {
|
||||
}
|
||||
}
|
||||
|
||||
/// One physical link to a neighbour, of whatever kind of transport.
|
||||
#[derive(Debug)]
|
||||
struct Protocol {
|
||||
plane: Arc<Plane>,
|
||||
graph: HashMap<PeerId, Vec<PeerId>>,
|
||||
members: HashSet<PeerId>,
|
||||
raw: HashMap<PeerId, SharedLink>,
|
||||
readers: HashMap<PeerId, JoinHandle<()>>,
|
||||
links: HashMap<PeerId, Arc<PeerLink>>,
|
||||
struct RawLink {
|
||||
kind: String,
|
||||
link: SharedLink,
|
||||
reader: JoinHandle<()>,
|
||||
}
|
||||
|
||||
impl Protocol {
|
||||
/// How a destination is reached right now.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub struct RouteSummary {
|
||||
/// Number of transport links to it. One is a direct link.
|
||||
pub hops: u8,
|
||||
/// The neighbour the first link goes to. For a direct link that is the
|
||||
/// destination itself.
|
||||
pub via: EndpointId,
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
struct State {
|
||||
plane: Arc<Plane>,
|
||||
/// Transport kinds, best first. A neighbour with several links is
|
||||
/// reached over the first of these it has.
|
||||
priority: Vec<String>,
|
||||
graph: HashMap<PeerId, Vec<PeerId>>,
|
||||
members: HashSet<PeerId>,
|
||||
raw: HashMap<PeerId, Vec<RawLink>>,
|
||||
links: HashMap<(PeerId, String), Arc<PeerLink>>,
|
||||
}
|
||||
|
||||
impl State {
|
||||
fn rank(&self, kind: &str) -> usize {
|
||||
self.priority
|
||||
.iter()
|
||||
.position(|known| known == kind)
|
||||
.unwrap_or(usize::MAX)
|
||||
}
|
||||
|
||||
/// A neighbour's links, best kind first.
|
||||
fn ordered(&self, peer: &PeerId) -> Vec<&RawLink> {
|
||||
let mut links: Vec<&RawLink> = self
|
||||
.raw
|
||||
.get(peer)
|
||||
.map(|links| links.iter().collect())
|
||||
.unwrap_or_default();
|
||||
links.sort_by_key(|raw| (self.rank(&raw.kind), raw.kind.clone()));
|
||||
links
|
||||
}
|
||||
|
||||
fn publish(&self) {
|
||||
let mut graph = self.graph.clone();
|
||||
graph.insert(
|
||||
self.plane.local,
|
||||
self.raw
|
||||
.iter()
|
||||
.filter(|(_, link)| !link.is_closed())
|
||||
.map(|(peer, _)| *peer)
|
||||
.collect(),
|
||||
);
|
||||
let mut neighbours: Vec<PeerId> = self
|
||||
.raw
|
||||
.iter()
|
||||
.filter(|(_, links)| links.iter().any(|raw| !raw.link.is_closed()))
|
||||
.map(|(peer, _)| *peer)
|
||||
.collect();
|
||||
neighbours.sort_unstable();
|
||||
graph.insert(self.plane.local, neighbours);
|
||||
let routing = RoutingTable::build(self.plane.local, &graph, ROUTING_HOP_LIMIT);
|
||||
let routes = routing
|
||||
.iter()
|
||||
@@ -181,10 +251,12 @@ impl Protocol {
|
||||
.next_hops
|
||||
.iter()
|
||||
.filter_map(|id| {
|
||||
self.raw.get(id).map(|link| NextHop {
|
||||
peer: *id,
|
||||
link: link.clone(),
|
||||
})
|
||||
let links: Box<[_]> = self
|
||||
.ordered(id)
|
||||
.into_iter()
|
||||
.map(|raw| raw.link.clone())
|
||||
.collect();
|
||||
(!links.is_empty()).then_some(NextHop { peer: *id, links })
|
||||
})
|
||||
.collect();
|
||||
(!next.is_empty()).then_some((
|
||||
@@ -204,17 +276,21 @@ impl Protocol {
|
||||
inboxes: self
|
||||
.links
|
||||
.iter()
|
||||
.map(|(peer, link)| (*peer, link.inbox_tx.clone()))
|
||||
.map(|((peer, _), link)| ((*peer, link.tag), link.inbox_tx.clone()))
|
||||
.collect(),
|
||||
}));
|
||||
}
|
||||
|
||||
fn stop_readers(&mut self) {
|
||||
for raw in self.raw.values().flatten() {
|
||||
raw.reader.abort();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for Protocol {
|
||||
impl Drop for State {
|
||||
fn drop(&mut self) {
|
||||
for task in self.readers.values() {
|
||||
task.abort();
|
||||
}
|
||||
self.stop_readers();
|
||||
for link in self.links.values() {
|
||||
link.close();
|
||||
}
|
||||
@@ -222,12 +298,17 @@ impl Drop for Protocol {
|
||||
}
|
||||
}
|
||||
|
||||
/// One network's transport links and independently published protocol tables.
|
||||
/// One network's transport links and its published forwarding table.
|
||||
///
|
||||
/// There is one table for the whole network, whatever protocols run over it.
|
||||
/// A link is a way to reach a neighbour; a protocol is an end-to-end channel
|
||||
/// to a peer. The two meet only in the envelope, which names the protocol, so
|
||||
/// a tunnel keeps working over any mix of transports and a middle member
|
||||
/// needs no protocol in common with either end.
|
||||
#[derive(Debug)]
|
||||
pub struct RelayHub {
|
||||
network: NetworkId,
|
||||
local: PeerId,
|
||||
protocols: Mutex<HashMap<String, Protocol>>,
|
||||
state: Mutex<State>,
|
||||
tally: Arc<Tally>,
|
||||
changed: Arc<Notify>,
|
||||
}
|
||||
@@ -235,40 +316,38 @@ pub struct RelayHub {
|
||||
impl RelayHub {
|
||||
/// Creates the router at this network's local endpoint.
|
||||
pub fn new(network: NetworkId, local: EndpointId) -> Arc<Self> {
|
||||
let tally = Arc::new(Tally::default());
|
||||
Arc::new(Self {
|
||||
network,
|
||||
local: *local.as_bytes(),
|
||||
protocols: Mutex::default(),
|
||||
tally: Arc::default(),
|
||||
state: Mutex::new(State {
|
||||
plane: Arc::new(Plane {
|
||||
local: *local.as_bytes(),
|
||||
table: ArcSwap::from_pointee(ForwardingTable::default()),
|
||||
tally: tally.clone(),
|
||||
}),
|
||||
priority: Vec::new(),
|
||||
graph: HashMap::new(),
|
||||
members: HashSet::new(),
|
||||
raw: HashMap::new(),
|
||||
links: HashMap::new(),
|
||||
}),
|
||||
tally,
|
||||
changed: Arc::default(),
|
||||
})
|
||||
}
|
||||
|
||||
fn protocols(&self) -> std::sync::MutexGuard<'_, HashMap<String, Protocol>> {
|
||||
self.protocols
|
||||
.lock()
|
||||
.unwrap_or_else(|error| error.into_inner())
|
||||
fn state(&self) -> std::sync::MutexGuard<'_, State> {
|
||||
self.state.lock().unwrap_or_else(|error| error.into_inner())
|
||||
}
|
||||
|
||||
fn protocol<'a>(
|
||||
&self,
|
||||
protocols: &'a mut HashMap<String, Protocol>,
|
||||
name: &str,
|
||||
) -> &'a mut Protocol {
|
||||
protocols
|
||||
.entry(name.to_owned())
|
||||
.or_insert_with(|| Protocol {
|
||||
plane: Arc::new(Plane {
|
||||
local: self.local,
|
||||
table: ArcSwap::from_pointee(ForwardingTable::default()),
|
||||
tally: self.tally.clone(),
|
||||
}),
|
||||
graph: HashMap::new(),
|
||||
members: HashSet::new(),
|
||||
raw: HashMap::new(),
|
||||
readers: HashMap::new(),
|
||||
links: HashMap::new(),
|
||||
})
|
||||
/// Sets which kinds of transport are preferred, best first, when a
|
||||
/// neighbour can be reached over more than one.
|
||||
pub fn set_priority(&self, kinds: Vec<String>) {
|
||||
let mut state = self.state();
|
||||
if state.priority != kinds {
|
||||
state.priority = kinds;
|
||||
state.publish();
|
||||
}
|
||||
}
|
||||
|
||||
/// Wakes the control loop promptly when a transport reader ends.
|
||||
@@ -282,14 +361,11 @@ impl RelayHub {
|
||||
}
|
||||
|
||||
/// Publishes authenticated topology. Refreshing identical state is cheap.
|
||||
pub fn set_topology(
|
||||
&self,
|
||||
protocol: &str,
|
||||
graph: HashMap<PeerId, Vec<PeerId>>,
|
||||
members: HashSet<PeerId>,
|
||||
) {
|
||||
let mut protocols = self.protocols();
|
||||
let state = self.protocol(&mut protocols, protocol);
|
||||
///
|
||||
/// Every row of `graph` is one member's own account of who it has a link
|
||||
/// to; what kind of link it is does not matter here.
|
||||
pub fn set_topology(&self, graph: HashMap<PeerId, Vec<PeerId>>, members: HashSet<PeerId>) {
|
||||
let mut state = self.state();
|
||||
if state.graph != graph || state.members != members {
|
||||
state.graph = graph;
|
||||
state.members = members;
|
||||
@@ -298,22 +374,44 @@ impl RelayHub {
|
||||
}
|
||||
|
||||
/// Whether the current table has a path to this destination.
|
||||
pub fn reachable(&self, peer: EndpointId, protocol: &str) -> bool {
|
||||
self.protocols().get(protocol).is_some_and(|state| {
|
||||
state
|
||||
.plane
|
||||
.table
|
||||
.load()
|
||||
.routes
|
||||
.contains_key(peer.as_bytes())
|
||||
pub fn reachable(&self, peer: EndpointId) -> bool {
|
||||
self.state()
|
||||
.plane
|
||||
.table
|
||||
.load()
|
||||
.routes
|
||||
.contains_key(peer.as_bytes())
|
||||
}
|
||||
|
||||
/// How a destination is reached right now, if it is.
|
||||
pub fn route(&self, peer: EndpointId) -> Option<RouteSummary> {
|
||||
let table = self.state().plane.table.load_full();
|
||||
let route = table.routes.get(peer.as_bytes())?;
|
||||
let via = EndpointId::from_bytes(&route.next.first()?.peer).ok()?;
|
||||
Some(RouteSummary {
|
||||
hops: route.hops,
|
||||
via,
|
||||
})
|
||||
}
|
||||
|
||||
/// Stable logical link, preserved across topology changes.
|
||||
/// The kinds of transport with an open direct link to a peer, best
|
||||
/// first.
|
||||
pub fn direct_kinds(&self, peer: EndpointId) -> Vec<String> {
|
||||
let state = self.state();
|
||||
state
|
||||
.ordered(peer.as_bytes())
|
||||
.into_iter()
|
||||
.filter(|raw| !raw.link.is_closed())
|
||||
.map(|raw| raw.kind.clone())
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Stable end-to-end channel for a protocol, preserved across topology
|
||||
/// and transport changes.
|
||||
pub fn link(&self, peer: EndpointId, protocol: &str) -> Arc<PeerLink> {
|
||||
let mut protocols = self.protocols();
|
||||
let state = self.protocol(&mut protocols, protocol);
|
||||
if let Some(link) = state.links.get(peer.as_bytes()) {
|
||||
let mut state = self.state();
|
||||
let key = (*peer.as_bytes(), protocol.to_owned());
|
||||
if let Some(link) = state.links.get(&key) {
|
||||
return link.clone();
|
||||
}
|
||||
let (inbox_tx, inbox_rx) = mpsc::channel(INBOX);
|
||||
@@ -321,88 +419,126 @@ impl RelayHub {
|
||||
let link = Arc::new(PeerLink {
|
||||
network: self.network,
|
||||
peer,
|
||||
tag: envelope::protocol_tag(protocol),
|
||||
plane: state.plane.clone(),
|
||||
inbox_tx,
|
||||
inbox_rx: tokio::sync::Mutex::new(inbox_rx),
|
||||
closed,
|
||||
});
|
||||
state.links.insert(*peer.as_bytes(), link.clone());
|
||||
state.links.insert(key, link.clone());
|
||||
state.publish();
|
||||
link
|
||||
}
|
||||
|
||||
/// Whether the protocol already owns its logical link.
|
||||
/// Whether the protocol already owns its end-to-end channel to a peer.
|
||||
pub fn has_link(&self, peer: EndpointId, protocol: &str) -> bool {
|
||||
self.protocols()
|
||||
.get(protocol)
|
||||
.is_some_and(|state| state.links.contains_key(peer.as_bytes()))
|
||||
self.state()
|
||||
.links
|
||||
.contains_key(&(*peer.as_bytes(), protocol.to_owned()))
|
||||
}
|
||||
|
||||
/// Installs a raw transport link and starts its independent ingress reader.
|
||||
pub fn set_direct(&self, peer: EndpointId, protocol: &str, raw: SharedLink) {
|
||||
///
|
||||
/// A neighbour may have one link per kind of transport; installing a
|
||||
/// second of the same kind replaces the first.
|
||||
pub fn set_direct(&self, peer: EndpointId, kind: &str, raw: SharedLink) {
|
||||
if raw.network() != self.network || raw.peer() != peer {
|
||||
return;
|
||||
}
|
||||
let mut protocols = self.protocols();
|
||||
let state = self.protocol(&mut protocols, protocol);
|
||||
let mut state = self.state();
|
||||
let id = *peer.as_bytes();
|
||||
if let Some(old) = state.readers.remove(&id) {
|
||||
old.abort();
|
||||
}
|
||||
state.raw.insert(id, raw.clone());
|
||||
state.publish();
|
||||
let plane = state.plane.clone();
|
||||
let changed = self.changed.clone();
|
||||
state.readers.insert(
|
||||
id,
|
||||
tokio::spawn(async move {
|
||||
let mut batch = 0;
|
||||
while let Some(frame) = raw.recv().await {
|
||||
plane.receive(id, frame);
|
||||
batch += 1;
|
||||
if batch == 64 {
|
||||
tokio::task::yield_now().await;
|
||||
batch = 0;
|
||||
}
|
||||
let reader_link = raw.clone();
|
||||
let reader = tokio::spawn(async move {
|
||||
let mut batch = 0;
|
||||
while let Some(frame) = reader_link.recv().await {
|
||||
plane.receive(id, frame);
|
||||
batch += 1;
|
||||
if batch == 64 {
|
||||
tokio::task::yield_now().await;
|
||||
batch = 0;
|
||||
}
|
||||
changed.notify_one();
|
||||
}),
|
||||
);
|
||||
}
|
||||
changed.notify_one();
|
||||
});
|
||||
let links = state.raw.entry(id).or_default();
|
||||
if let Some(position) = links.iter().position(|known| known.kind == kind) {
|
||||
links.remove(position).reader.abort();
|
||||
}
|
||||
links.push(RawLink {
|
||||
kind: kind.to_owned(),
|
||||
link: raw,
|
||||
reader,
|
||||
});
|
||||
state.publish();
|
||||
}
|
||||
|
||||
/// Removes a physical link without tearing down end-to-end protocol state.
|
||||
pub fn clear_direct(&self, peer: EndpointId, protocol: &str) {
|
||||
if let Some(state) = self.protocols().get_mut(protocol) {
|
||||
state.raw.remove(peer.as_bytes());
|
||||
if let Some(task) = state.readers.remove(peer.as_bytes()) {
|
||||
task.abort();
|
||||
/// Removes one physical link without tearing down end-to-end protocol state.
|
||||
pub fn clear_direct(&self, peer: EndpointId, kind: &str) {
|
||||
let mut state = self.state();
|
||||
let mut changed = false;
|
||||
if let Some(links) = state.raw.get_mut(peer.as_bytes()) {
|
||||
if let Some(position) = links.iter().position(|known| known.kind == kind) {
|
||||
links.remove(position).reader.abort();
|
||||
changed = true;
|
||||
}
|
||||
if links.is_empty() {
|
||||
state.raw.remove(peer.as_bytes());
|
||||
}
|
||||
}
|
||||
if changed {
|
||||
state.publish();
|
||||
}
|
||||
}
|
||||
|
||||
/// Retires data channels without removing an authenticated peer's topology.
|
||||
pub fn clear_peer_links(&self, peer: EndpointId) {
|
||||
let mut state = self.state();
|
||||
if let Some(links) = state.raw.remove(peer.as_bytes()) {
|
||||
for raw in links {
|
||||
raw.link.close();
|
||||
raw.reader.abort();
|
||||
}
|
||||
}
|
||||
let gone: Vec<_> = state
|
||||
.links
|
||||
.keys()
|
||||
.filter(|(id, _)| id == peer.as_bytes())
|
||||
.cloned()
|
||||
.collect();
|
||||
for key in gone {
|
||||
if let Some(link) = state.links.remove(&key) {
|
||||
link.close();
|
||||
}
|
||||
}
|
||||
state.publish();
|
||||
}
|
||||
|
||||
/// Revokes all links and topology involving a departed authenticated peer.
|
||||
pub fn remove_peer(&self, peer: EndpointId) {
|
||||
for state in self.protocols().values_mut() {
|
||||
state.raw.remove(peer.as_bytes());
|
||||
if let Some(task) = state.readers.remove(peer.as_bytes()) {
|
||||
task.abort();
|
||||
}
|
||||
if let Some(link) = state.links.remove(peer.as_bytes()) {
|
||||
link.close();
|
||||
}
|
||||
state.members.remove(peer.as_bytes());
|
||||
state.graph.remove(peer.as_bytes());
|
||||
for neighbors in state.graph.values_mut() {
|
||||
neighbors.retain(|id| id != peer.as_bytes());
|
||||
}
|
||||
state.publish();
|
||||
self.clear_peer_links(peer);
|
||||
let mut state = self.state();
|
||||
state.members.remove(peer.as_bytes());
|
||||
state.graph.remove(peer.as_bytes());
|
||||
for neighbors in state.graph.values_mut() {
|
||||
neighbors.retain(|id| id != peer.as_bytes());
|
||||
}
|
||||
state.publish();
|
||||
}
|
||||
|
||||
/// Stops every reader and releases all transport handles.
|
||||
pub fn close(&self) {
|
||||
self.protocols().clear();
|
||||
let mut state = self.state();
|
||||
state.stop_readers();
|
||||
state.raw.clear();
|
||||
for link in state.links.values() {
|
||||
link.close();
|
||||
}
|
||||
state.links.clear();
|
||||
state.graph.clear();
|
||||
state.members.clear();
|
||||
state.plane.table.store(Arc::default());
|
||||
}
|
||||
}
|
||||
|
||||
@@ -411,6 +547,7 @@ impl RelayHub {
|
||||
pub struct PeerLink {
|
||||
network: NetworkId,
|
||||
peer: EndpointId,
|
||||
tag: u64,
|
||||
plane: Arc<Plane>,
|
||||
inbox_tx: mpsc::Sender<Bytes>,
|
||||
inbox_rx: tokio::sync::Mutex<mpsc::Receiver<Bytes>>,
|
||||
@@ -454,10 +591,11 @@ impl PacketLink for PeerLink {
|
||||
let hop = route
|
||||
.select(&self.plane.local, flow)
|
||||
.ok_or(TransportError::Closed)?;
|
||||
hop.link.send(envelope::encode(
|
||||
hop.send(envelope::encode(
|
||||
self.plane.local,
|
||||
*self.peer.as_bytes(),
|
||||
flow,
|
||||
self.tag,
|
||||
&payload,
|
||||
))?;
|
||||
if route.hops > 1 {
|
||||
@@ -489,7 +627,9 @@ impl PacketLink for PeerLink {
|
||||
fn path_description(&self) -> String {
|
||||
let table = self.plane.table.load();
|
||||
match table.routes.get(self.peer.as_bytes()) {
|
||||
Some(route) if route.hops == 1 => route.next[0].link.path_description(),
|
||||
Some(route) if route.hops == 1 => route.next[0]
|
||||
.first_open()
|
||||
.map_or_else(|| "unreachable".into(), |link| link.path_description()),
|
||||
Some(route) => format!(
|
||||
"relay {} hops via {} ({} equal paths)",
|
||||
route.hops,
|
||||
|
||||
@@ -16,6 +16,9 @@ fn peer(seed: u8) -> EndpointId {
|
||||
fn id(seed: u8) -> PeerId {
|
||||
*peer(seed).as_bytes()
|
||||
}
|
||||
fn tag() -> u64 {
|
||||
envelope::protocol_tag("ip")
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
struct Wire {
|
||||
@@ -78,9 +81,8 @@ impl PacketLink for Wire {
|
||||
}
|
||||
}
|
||||
|
||||
fn topology(hub: &RelayHub, protocol: &str, edges: &[(u8, &[u8])], members: &[u8]) {
|
||||
fn topology(hub: &RelayHub, edges: &[(u8, &[u8])], members: &[u8]) {
|
||||
hub.set_topology(
|
||||
protocol,
|
||||
edges
|
||||
.iter()
|
||||
.map(|(a, bs)| (id(*a), bs.iter().map(|b| id(*b)).collect()))
|
||||
@@ -96,7 +98,7 @@ async fn ecmp_is_per_flow_direct_wins_and_existing_tunnel_survives_changes() {
|
||||
let c = Wire::new(3, true);
|
||||
hub.set_direct(peer(2), "ip", b.clone());
|
||||
hub.set_direct(peer(3), "ip", c.clone());
|
||||
topology(&hub, "ip", &[(2, &[4]), (3, &[4])], &[1, 2, 3, 4]);
|
||||
topology(&hub, &[(2, &[4]), (3, &[4])], &[1, 2, 3, 4]);
|
||||
let link = hub.link(peer(4), "ip");
|
||||
let mut chosen = HashMap::new();
|
||||
for _ in 0..8 {
|
||||
@@ -132,7 +134,7 @@ async fn ecmp_is_per_flow_direct_wins_and_existing_tunnel_survives_changes() {
|
||||
}
|
||||
assert_eq!(c.take().len(), 100);
|
||||
hub.clear_direct(peer(2), "ip");
|
||||
topology(&hub, "ip", &[(3, &[])], &[1, 2, 3, 4]);
|
||||
topology(&hub, &[(3, &[])], &[1, 2, 3, 4]);
|
||||
assert!(link.send(Bytes::new()).is_err());
|
||||
assert!(!link.is_closed());
|
||||
assert!(Arc::ptr_eq(&link, &hub.link(peer(4), "ip")));
|
||||
@@ -152,8 +154,8 @@ async fn transport_reader_forwards_without_any_plugin_reader_and_isolates_protoc
|
||||
let c = Wire::new(3, true);
|
||||
hub.set_direct(peer(1), "ip", a.clone());
|
||||
hub.set_direct(peer(3), "ip", c.clone());
|
||||
topology(&hub, "ip", &[(3, &[4])], &[1, 2, 3, 4]);
|
||||
let frame = envelope::encode(id(1), id(4), 91, b"end-to-end encrypted");
|
||||
topology(&hub, &[(3, &[4])], &[1, 2, 3, 4]);
|
||||
let frame = envelope::encode(id(1), id(4), 91, tag(), b"end-to-end encrypted");
|
||||
let pointer = frame.as_ptr();
|
||||
a.feed.send(frame).await.unwrap();
|
||||
tokio::time::timeout(std::time::Duration::from_secs(1), async {
|
||||
@@ -170,12 +172,136 @@ async fn transport_reader_forwards_without_any_plugin_reader_and_isolates_protoc
|
||||
ROUTING_HOP_LIMIT - 1
|
||||
);
|
||||
assert_eq!(&sent[0][RELAY_OVERHEAD..], b"end-to-end encrypted");
|
||||
let isolated = hub.link(peer(4), "other-ip");
|
||||
assert!(isolated.send(Bytes::new()).is_err());
|
||||
assert_eq!(hub.counters().forwarded, 1);
|
||||
hub.close();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_frame_is_delivered_only_to_the_protocol_it_names() {
|
||||
let hub = RelayHub::new(network(), peer(2));
|
||||
hub.set_direct(peer(1), "ip", Wire::new(1, true));
|
||||
let ip = hub.link(peer(1), "ip");
|
||||
let other = hub.link(peer(1), "other-ip");
|
||||
let plane = ip.plane.clone();
|
||||
plane.receive(id(1), envelope::encode(id(1), id(2), 0, tag(), b"for ip"));
|
||||
plane.receive(
|
||||
id(1),
|
||||
envelope::encode(
|
||||
id(1),
|
||||
id(2),
|
||||
0,
|
||||
envelope::protocol_tag("other-ip"),
|
||||
b"for other",
|
||||
),
|
||||
);
|
||||
plane.receive(
|
||||
id(1),
|
||||
envelope::encode(
|
||||
id(1),
|
||||
id(2),
|
||||
0,
|
||||
envelope::protocol_tag("nobody-runs-this"),
|
||||
b"for nobody",
|
||||
),
|
||||
);
|
||||
assert_eq!(ip.recv().await.unwrap(), Bytes::from_static(b"for ip"));
|
||||
assert_eq!(
|
||||
other.recv().await.unwrap(),
|
||||
Bytes::from_static(b"for other")
|
||||
);
|
||||
assert_eq!(hub.counters().dropped_unknown, 1);
|
||||
hub.close();
|
||||
}
|
||||
|
||||
/// A middle member joins two neighbours over different transports, and the
|
||||
/// two ends share a protocol the middle has never heard of.
|
||||
#[tokio::test]
|
||||
async fn a_tunnel_crosses_a_middle_member_whose_links_are_of_different_kinds() {
|
||||
let a = RelayHub::new(network(), peer(1));
|
||||
let c = RelayHub::new(network(), peer(3));
|
||||
let b = RelayHub::new(network(), peer(2));
|
||||
let a_to_c = Wire::new(3, true);
|
||||
let c_to_a = Wire::new(1, true);
|
||||
let c_to_b = Wire::new(2, true);
|
||||
let b_to_c = Wire::new(3, true);
|
||||
a.set_direct(peer(3), "wg", a_to_c.clone());
|
||||
c.set_direct(peer(1), "wg", c_to_a);
|
||||
c.set_direct(peer(2), "wg-quic", c_to_b.clone());
|
||||
b.set_direct(peer(3), "wg-quic", b_to_c.clone());
|
||||
topology(&a, &[(3, &[1, 2])], &[1, 2, 3]);
|
||||
topology(&b, &[(3, &[1, 2])], &[1, 2, 3]);
|
||||
topology(&c, &[(1, &[3]), (2, &[3])], &[1, 2, 3]);
|
||||
|
||||
let outgoing = a.link(peer(2), "tunnel");
|
||||
let incoming = b.link(peer(1), "tunnel");
|
||||
outgoing
|
||||
.send_flow(Bytes::from_static(b"sealed"), 5)
|
||||
.unwrap();
|
||||
let frame = a_to_c.take().remove(0);
|
||||
|
||||
// The middle forwards it onto its other kind of link without looking
|
||||
// inside.
|
||||
c.state().plane.receive(id(1), frame);
|
||||
let frame = c_to_b.take().remove(0);
|
||||
assert_eq!(c.counters().forwarded, 1);
|
||||
|
||||
b.state().plane.receive(id(3), frame);
|
||||
assert_eq!(
|
||||
incoming.recv().await.unwrap(),
|
||||
Bytes::from_static(b"sealed")
|
||||
);
|
||||
assert_eq!(b.counters().received_via, 1);
|
||||
assert_eq!(
|
||||
b.route(peer(1)),
|
||||
Some(RouteSummary {
|
||||
hops: 2,
|
||||
via: peer(3)
|
||||
})
|
||||
);
|
||||
a.close();
|
||||
b.close();
|
||||
c.close();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_neighbour_is_reached_over_the_preferred_kind_and_falls_back_when_it_is_lost() {
|
||||
let hub = RelayHub::new(network(), peer(1));
|
||||
hub.set_priority(vec!["wg".into(), "tcp-tls".into(), "wg-quic".into()]);
|
||||
let slow = Wire::new(2, true);
|
||||
let fast = Wire::new(2, true);
|
||||
hub.set_direct(peer(2), "wg-quic", slow.clone());
|
||||
let link = hub.link(peer(2), "tunnel");
|
||||
link.send(Bytes::from_static(b"one")).unwrap();
|
||||
assert_eq!(slow.take().len(), 1);
|
||||
assert_eq!(hub.direct_kinds(peer(2)), ["wg-quic"]);
|
||||
|
||||
// A better kind appears: traffic moves to it with nothing to redo.
|
||||
hub.set_direct(peer(2), "wg", fast.clone());
|
||||
assert_eq!(hub.direct_kinds(peer(2)), ["wg", "wg-quic"]);
|
||||
link.send(Bytes::from_static(b"two")).unwrap();
|
||||
assert_eq!(fast.take().len(), 1);
|
||||
assert!(slow.take().is_empty());
|
||||
|
||||
// It dies: the worse one carries on, and the logical link is untouched.
|
||||
fast.dead.store(true, Ordering::Relaxed);
|
||||
link.send(Bytes::from_static(b"three")).unwrap();
|
||||
assert_eq!(slow.take().len(), 1);
|
||||
assert!(!link.is_closed());
|
||||
assert_eq!(
|
||||
hub.route(peer(2)),
|
||||
Some(RouteSummary {
|
||||
hops: 1,
|
||||
via: peer(2)
|
||||
})
|
||||
);
|
||||
|
||||
hub.clear_direct(peer(2), "wg-quic");
|
||||
hub.clear_direct(peer(2), "wg");
|
||||
assert!(!hub.reachable(peer(2)));
|
||||
assert!(Arc::ptr_eq(&link, &hub.link(peer(2), "tunnel")));
|
||||
hub.close();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn local_delivery_is_bounded_and_malformed_or_unknown_sources_are_dropped() {
|
||||
let hub = RelayHub::new(network(), peer(2));
|
||||
@@ -184,18 +310,21 @@ async fn local_delivery_is_bounded_and_malformed_or_unknown_sources_are_dropped(
|
||||
let link = hub.link(peer(1), "ip");
|
||||
let plane = link.plane.clone();
|
||||
for _ in 0..INBOX + 1 {
|
||||
plane.receive(id(1), envelope::encode(id(1), id(2), 0, b"ip"));
|
||||
plane.receive(id(1), envelope::encode(id(1), id(2), 0, tag(), b"ip"));
|
||||
}
|
||||
assert_eq!(hub.counters().dropped_congested, 1);
|
||||
for _ in 0..INBOX {
|
||||
assert_eq!(link.recv().await.unwrap(), Bytes::from_static(b"ip"));
|
||||
}
|
||||
plane.receive(id(1), Bytes::from_static(b"broken"));
|
||||
plane.receive(id(1), envelope::encode(id(9), id(2), 0, b"unknown member"));
|
||||
plane.receive(
|
||||
id(1),
|
||||
envelope::encode(id(9), id(2), 0, tag(), b"unknown member"),
|
||||
);
|
||||
assert_eq!(hub.counters().dropped_unknown, 2);
|
||||
plane.receive(
|
||||
id(1),
|
||||
envelope::encode(id(1), id(9), 0, b"unknown destination"),
|
||||
envelope::encode(id(1), id(9), 0, tag(), b"unknown destination"),
|
||||
);
|
||||
assert_eq!(hub.counters().dropped_no_link, 1);
|
||||
hub.close();
|
||||
@@ -210,11 +339,11 @@ async fn inconsistent_tables_cannot_loop_beyond_the_hop_limit() {
|
||||
a.set_direct(peer(2), "ip", ab.clone());
|
||||
b.set_direct(peer(1), "ip", ba.clone());
|
||||
// Deliberately inconsistent snapshots while updates are in flight.
|
||||
topology(&a, "ip", &[(2, &[4])], &[1, 2, 3, 4]);
|
||||
topology(&b, "ip", &[(1, &[4])], &[1, 2, 3, 4]);
|
||||
topology(&a, &[(2, &[4])], &[1, 2, 3, 4]);
|
||||
topology(&b, &[(1, &[4])], &[1, 2, 3, 4]);
|
||||
let ap = a.link(peer(4), "ip").plane.clone();
|
||||
let bp = b.link(peer(4), "ip").plane.clone();
|
||||
let mut frame = envelope::encode(id(3), id(4), 7, b"transit");
|
||||
let mut frame = envelope::encode(id(3), id(4), 7, tag(), b"transit");
|
||||
for turn in 0..ROUTING_HOP_LIMIT {
|
||||
let (plane, incoming, wire) = if turn % 2 == 0 {
|
||||
(&ap, id(2), &ab)
|
||||
@@ -253,7 +382,7 @@ async fn forwarding_benchmark() {
|
||||
hub.set_direct(peer(3), "ip", wire.clone());
|
||||
let alternate = Wire::new(5, false);
|
||||
hub.set_direct(peer(5), "ip", alternate.clone());
|
||||
topology(&hub, "ip", &[(3, &[4]), (5, &[4])], &[1, 2, 3, 4, 5]);
|
||||
topology(&hub, &[(3, &[4]), (5, &[4])], &[1, 2, 3, 4, 5]);
|
||||
let plane = hub.link(peer(4), "ip").plane.clone();
|
||||
let source = id(1);
|
||||
let destination = id(4);
|
||||
@@ -264,7 +393,9 @@ async fn forwarding_benchmark() {
|
||||
const BATCHES: usize = 100;
|
||||
for _ in 0..BATCHES {
|
||||
let frames: Vec<_> = (0..COUNT)
|
||||
.map(|flow| envelope::encode(source, destination, flow as u64, &vec![42; size]))
|
||||
.map(|flow| {
|
||||
envelope::encode(source, destination, flow as u64, tag(), &vec![42; size])
|
||||
})
|
||||
.collect();
|
||||
let start = std::time::Instant::now();
|
||||
for frame in frames {
|
||||
|
||||
@@ -3,8 +3,8 @@ use super::{FlowId, PeerId};
|
||||
use crate::config::{MAX_DATA_DATAGRAM, ROUTING_HOP_LIMIT};
|
||||
use bytes::{BufMut, Bytes, BytesMut};
|
||||
|
||||
pub(crate) const HEADER: usize = 2 + 32 + 32 + 8;
|
||||
const VERSION: u8 = 1;
|
||||
pub(crate) const HEADER: usize = 2 + 32 + 32 + 8 + 8;
|
||||
const VERSION: u8 = 2;
|
||||
|
||||
#[derive(Debug, PartialEq, Eq)]
|
||||
pub(crate) struct Header {
|
||||
@@ -12,6 +12,15 @@ pub(crate) struct Header {
|
||||
pub destination: PeerId,
|
||||
pub remaining: u8,
|
||||
pub flow: FlowId,
|
||||
/// Which end-to-end protocol the payload belongs to, whatever links carry it.
|
||||
pub protocol: u64,
|
||||
}
|
||||
|
||||
/// A stable 64-bit tag for a protocol id (FNV-1a), the same on every member.
|
||||
pub(crate) fn protocol_tag(name: &str) -> u64 {
|
||||
name.bytes().fold(0xcbf2_9ce4_8422_2325, |hash, byte| {
|
||||
(hash ^ u64::from(byte)).wrapping_mul(0x0000_0100_0000_01b3)
|
||||
})
|
||||
}
|
||||
|
||||
pub(crate) fn decode(bytes: &[u8]) -> Option<Header> {
|
||||
@@ -28,16 +37,24 @@ pub(crate) fn decode(bytes: &[u8]) -> Option<Header> {
|
||||
destination: bytes[34..66].try_into().ok()?,
|
||||
remaining: bytes[1],
|
||||
flow: u64::from_be_bytes(bytes[66..74].try_into().ok()?),
|
||||
protocol: u64::from_be_bytes(bytes[74..82].try_into().ok()?),
|
||||
})
|
||||
}
|
||||
|
||||
pub(crate) fn encode(source: PeerId, destination: PeerId, flow: FlowId, payload: &[u8]) -> Bytes {
|
||||
pub(crate) fn encode(
|
||||
source: PeerId,
|
||||
destination: PeerId,
|
||||
flow: FlowId,
|
||||
protocol: u64,
|
||||
payload: &[u8],
|
||||
) -> Bytes {
|
||||
let mut bytes = BytesMut::with_capacity(HEADER + payload.len());
|
||||
bytes.put_u8(VERSION);
|
||||
bytes.put_u8(ROUTING_HOP_LIMIT);
|
||||
bytes.extend_from_slice(&source);
|
||||
bytes.extend_from_slice(&destination);
|
||||
bytes.put_u64(flow);
|
||||
bytes.put_u64(protocol);
|
||||
bytes.extend_from_slice(payload);
|
||||
bytes.freeze()
|
||||
}
|
||||
@@ -58,7 +75,7 @@ mod tests {
|
||||
use super::*;
|
||||
#[test]
|
||||
fn transit_reuses_the_buffer_and_rejects_invalid_headers() {
|
||||
let frame = encode([1; 32], [2; 32], 17, &[42; 1280]);
|
||||
let frame = encode([1; 32], [2; 32], 17, protocol_tag("wg"), &[42; 1280]);
|
||||
let address = frame.as_ptr();
|
||||
let frame = decrement(frame);
|
||||
assert_eq!(
|
||||
@@ -66,7 +83,10 @@ mod tests {
|
||||
address,
|
||||
"no payload copy for an owned buffer"
|
||||
);
|
||||
assert_eq!(decode(&frame).unwrap().remaining, ROUTING_HOP_LIMIT - 1);
|
||||
let header = decode(&frame).unwrap();
|
||||
assert_eq!(header.remaining, ROUTING_HOP_LIMIT - 1);
|
||||
assert_eq!(header.protocol, protocol_tag("wg"));
|
||||
assert_ne!(protocol_tag("wg"), protocol_tag("wg-quic"));
|
||||
assert_eq!(&frame[HEADER..], &[42; 1280]);
|
||||
for length in 0..HEADER {
|
||||
assert!(decode(&frame[..length]).is_none());
|
||||
|
||||
@@ -0,0 +1,350 @@
|
||||
//! Authenticated probes of the actual data link, independent of control RTT.
|
||||
//! Keys are negotiated privately for each physical link. No fixed wire magic.
|
||||
use super::{PacketLink, SharedLink, TransportError};
|
||||
use crate::dataplane::SharedPlugin;
|
||||
use crate::overlay::RoutingTable;
|
||||
use crate::{BoxFuture, NetworkId};
|
||||
use bytes::Bytes;
|
||||
use hmac::{Hmac, KeyInit, Mac};
|
||||
use iroh::EndpointId;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use sha2::Sha256;
|
||||
use std::collections::VecDeque;
|
||||
use std::sync::{Arc, Mutex};
|
||||
use std::time::{Duration, Instant};
|
||||
use tokio::sync::{mpsc, watch};
|
||||
|
||||
#[derive(Clone, Serialize, Deserialize, zeroize::ZeroizeOnDrop)]
|
||||
pub(crate) struct Settings {
|
||||
marker: [u8; 16],
|
||||
key: [u8; 32],
|
||||
}
|
||||
impl Settings {
|
||||
pub(crate) fn fresh() -> Self {
|
||||
Self {
|
||||
marker: rand::random(),
|
||||
key: rand::random(),
|
||||
}
|
||||
}
|
||||
}
|
||||
impl std::fmt::Debug for Settings {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
f.write_str("HealthSettings([REDACTED])")
|
||||
}
|
||||
}
|
||||
struct Samples {
|
||||
pending: VecDeque<(u64, Instant)>,
|
||||
recent: VecDeque<bool>,
|
||||
}
|
||||
impl Samples {
|
||||
fn new() -> Self {
|
||||
Self {
|
||||
pending: VecDeque::new(),
|
||||
recent: VecDeque::new(),
|
||||
}
|
||||
}
|
||||
fn sent(&mut self, seq: u64, now: Instant) {
|
||||
self.pending.push_back((seq, now));
|
||||
}
|
||||
fn acknowledge(&mut self, seq: u64) {
|
||||
if let Some(i) = self.pending.iter().position(|(n, _)| *n == seq) {
|
||||
self.pending.remove(i);
|
||||
self.recent.push_back(true);
|
||||
}
|
||||
}
|
||||
fn expire(&mut self, now: Instant) {
|
||||
while self
|
||||
.pending
|
||||
.front()
|
||||
.is_some_and(|(_, at)| now.duration_since(*at) >= Duration::from_secs(3))
|
||||
{
|
||||
self.pending.pop_front();
|
||||
self.recent.push_back(false);
|
||||
}
|
||||
while self.recent.len() > 20 {
|
||||
self.recent.pop_front();
|
||||
}
|
||||
}
|
||||
fn degraded(&self) -> bool {
|
||||
self.recent.len() >= 15
|
||||
&& self.recent.iter().filter(|ok| !**ok).count() * 100 >= self.recent.len() * 60
|
||||
}
|
||||
}
|
||||
// This prefix is inside end-to-end encryption. It never appears on the UDP wire.
|
||||
const IP_MAGIC: &[u8; 16] = b"tsunagi-health01";
|
||||
pub(crate) fn ip_payload(packet: &[u8]) -> Option<&[u8]> {
|
||||
(packet.len() >= 36 && packet[0] == 0x45 && packet[9] == 253 && &packet[20..36] == IP_MAGIC)
|
||||
.then(|| &packet[36..])
|
||||
}
|
||||
/// Inject probes through the same encrypted tunnel as application IP packets.
|
||||
/// This matters for pure WG: raw UDP probes could pass DPI while WG is blocked.
|
||||
pub(crate) struct ProbeRoute {
|
||||
pub(crate) plugin: SharedPlugin,
|
||||
pub(crate) routes: Arc<RoutingTable>,
|
||||
pub(crate) network: NetworkId,
|
||||
pub(crate) peer: EndpointId,
|
||||
pub(crate) max_probe_size: usize,
|
||||
}
|
||||
impl ProbeRoute {
|
||||
fn send(&self, payload: Bytes) -> bool {
|
||||
let Some((source, destination)) = self.routes.probe_addresses(self.network, self.peer)
|
||||
else {
|
||||
return false;
|
||||
};
|
||||
let mut packet = vec![0; 36];
|
||||
packet[0] = 0x45;
|
||||
packet[8] = 64;
|
||||
packet[9] = 253;
|
||||
packet[12..16].copy_from_slice(&source.octets());
|
||||
packet[16..20].copy_from_slice(&destination.octets());
|
||||
packet[20..36].copy_from_slice(IP_MAGIC);
|
||||
let length = (packet.len() + payload.len()) as u16;
|
||||
packet[2..4].copy_from_slice(&length.to_be_bytes());
|
||||
let sum = packet[..20]
|
||||
.as_chunks::<2>()
|
||||
.0
|
||||
.iter()
|
||||
.map(|b| u32::from(u16::from_be_bytes(*b)))
|
||||
.sum::<u32>();
|
||||
let sum = (sum & 0xffff) + (sum >> 16);
|
||||
packet[10..12].copy_from_slice(&(!(sum as u16)).to_be_bytes());
|
||||
packet.extend_from_slice(&payload);
|
||||
self.plugin
|
||||
.carry(self.network, self.peer, Bytes::from(packet))
|
||||
}
|
||||
}
|
||||
/// A monitored physical link. It forwards application bytes unchanged.
|
||||
pub(crate) struct HealthLink {
|
||||
probes: mpsc::Sender<Bytes>,
|
||||
raw: SharedLink,
|
||||
settings: Mutex<Option<Settings>>,
|
||||
rx: tokio::sync::Mutex<mpsc::Receiver<Bytes>>,
|
||||
stop: watch::Sender<bool>,
|
||||
reason: Mutex<Option<String>>,
|
||||
}
|
||||
impl std::fmt::Debug for HealthLink {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
f.debug_struct("HealthLink")
|
||||
.field("raw", &self.raw)
|
||||
.finish_non_exhaustive()
|
||||
}
|
||||
}
|
||||
fn lock<T>(m: &Mutex<T>) -> std::sync::MutexGuard<'_, T> {
|
||||
match m.lock() {
|
||||
Ok(g) => g,
|
||||
Err(p) => p.into_inner(),
|
||||
}
|
||||
}
|
||||
impl HealthLink {
|
||||
pub(crate) fn new(
|
||||
raw: SharedLink,
|
||||
local: EndpointId,
|
||||
settings: Option<Settings>,
|
||||
route: Option<ProbeRoute>,
|
||||
) -> Arc<Self> {
|
||||
let (tx, rx) = mpsc::channel(64);
|
||||
let (probe_tx, mut probe_rx) = mpsc::channel(64);
|
||||
let (stop, _) = watch::channel(false);
|
||||
let link = Arc::new(Self {
|
||||
probes: probe_tx,
|
||||
raw: raw.clone(),
|
||||
settings: Mutex::new(settings),
|
||||
rx: tokio::sync::Mutex::new(rx),
|
||||
stop,
|
||||
reason: Mutex::new(None),
|
||||
});
|
||||
let weak = Arc::downgrade(&link);
|
||||
let mut stopped = link.stop.subscribe();
|
||||
tokio::spawn(async move {
|
||||
let mut interval = tokio::time::interval(Duration::from_secs(1));
|
||||
interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
|
||||
let mut samples = Samples::new();
|
||||
let mut sequence = rand::random::<u64>();
|
||||
let mut since: Option<Instant> = None;
|
||||
let mut marker = None;
|
||||
loop {
|
||||
tokio::select! {
|
||||
biased;
|
||||
_=stopped.changed()=>break,
|
||||
_=interval.tick()=>{
|
||||
let Some(link)=weak.upgrade()else{break;};let Some(settings)=lock(&link.settings).clone()else{continue;};
|
||||
if marker != Some(settings.marker) { marker = Some(settings.marker); samples = Samples::new(); since = None; }
|
||||
let start=*since.get_or_insert_with(Instant::now);let now=Instant::now();samples.expire(now);
|
||||
if now.duration_since(start)>=Duration::from_secs(20) && samples.degraded() {
|
||||
*lock(&link.reason)=Some("data delivery checks: at least 60% loss over the last 20 probes".into());break;
|
||||
}
|
||||
sequence=sequence.wrapping_add(1);
|
||||
let size=route.as_ref().map_or(raw.max_datagram_size(),|route|route.max_probe_size);
|
||||
let packet=probe(&settings,local,1,sequence,size.min(1000));
|
||||
if let Some(route)=&route { if route.send(packet) {samples.sent(sequence,now);} }
|
||||
else { let _=raw.send(packet); samples.sent(sequence,now); }
|
||||
}
|
||||
packet=raw.recv()=>{
|
||||
let Some(packet)=packet else{break;};let Some(link)=weak.upgrade()else{break;};
|
||||
let settings=lock(&link.settings).clone();
|
||||
if route.is_none() && let Some(settings)=settings && packet.starts_with(&settings.marker) {
|
||||
if let Some((kind,seq))=verify(&settings,raw.peer(),&packet) {
|
||||
if kind==1 {let _=raw.send(probe(&settings,local,2,seq,packet.len()));}
|
||||
else if kind==2 {samples.acknowledge(seq);}
|
||||
}
|
||||
} else {let _=tx.try_send(packet);}
|
||||
}
|
||||
packet=probe_rx.recv(),if route.is_some()=>{
|
||||
let Some(packet)=packet else {break;};let Some(link)=weak.upgrade()else{break;};
|
||||
let Some(settings)=lock(&link.settings).clone()else{continue;};
|
||||
if let Some((kind,seq))=verify(&settings,raw.peer(),&packet) {
|
||||
if kind==1 { if let Some(route)=&route { route.send(probe(&settings,local,2,seq,packet.len().min(route.max_probe_size))); } }
|
||||
else if kind==2 {samples.acknowledge(seq);}
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
}
|
||||
raw.close();
|
||||
if let Some(link) = weak.upgrade() {
|
||||
link.stop.send_replace(true);
|
||||
}
|
||||
});
|
||||
link
|
||||
}
|
||||
pub(crate) fn receive_probe(&self, packet: Bytes) {
|
||||
let _ = self.probes.try_send(packet);
|
||||
}
|
||||
pub(crate) fn configure(&self, settings: Settings) {
|
||||
*lock(&self.settings) = Some(settings);
|
||||
}
|
||||
}
|
||||
fn probe(settings: &Settings, sender: EndpointId, kind: u8, seq: u64, size: usize) -> Bytes {
|
||||
let mut bytes = vec![0; size.max(57)];
|
||||
bytes[..16].copy_from_slice(&settings.marker);
|
||||
bytes[16] = kind;
|
||||
bytes[17..25].copy_from_slice(&seq.to_be_bytes());
|
||||
rand::fill(&mut bytes[25..size.max(57) - 32]);
|
||||
let tail = bytes.len() - 32;
|
||||
if let Ok(mut mac) = <Hmac<Sha256> as KeyInit>::new_from_slice(&settings.key) {
|
||||
mac.update(sender.as_bytes());
|
||||
mac.update(&bytes[..tail]);
|
||||
bytes[tail..].copy_from_slice(&mac.finalize().into_bytes());
|
||||
}
|
||||
Bytes::from(bytes)
|
||||
}
|
||||
fn verify(settings: &Settings, sender: EndpointId, bytes: &[u8]) -> Option<(u8, u64)> {
|
||||
if bytes.len() < 57 || !bytes.starts_with(&settings.marker) {
|
||||
return None;
|
||||
}
|
||||
let mut mac = <Hmac<Sha256> as KeyInit>::new_from_slice(&settings.key).ok()?;
|
||||
let tail = bytes.len() - 32;
|
||||
mac.update(sender.as_bytes());
|
||||
mac.update(&bytes[..tail]);
|
||||
mac.verify_slice(&bytes[tail..]).ok()?;
|
||||
Some((
|
||||
bytes[16],
|
||||
u64::from_be_bytes(bytes.get(17..25)?.try_into().ok()?),
|
||||
))
|
||||
}
|
||||
impl PacketLink for HealthLink {
|
||||
fn network(&self) -> NetworkId {
|
||||
self.raw.network()
|
||||
}
|
||||
fn peer(&self) -> EndpointId {
|
||||
self.raw.peer()
|
||||
}
|
||||
fn max_datagram_size(&self) -> usize {
|
||||
self.raw.max_datagram_size()
|
||||
}
|
||||
fn send(&self, payload: Bytes) -> Result<(), TransportError> {
|
||||
if self.is_closed() {
|
||||
Err(TransportError::Closed)
|
||||
} else {
|
||||
self.raw.send(payload)
|
||||
}
|
||||
}
|
||||
fn send_flow(
|
||||
&self,
|
||||
payload: Bytes,
|
||||
flow: super::super::routing::FlowId,
|
||||
) -> Result<(), TransportError> {
|
||||
if self.is_closed() {
|
||||
Err(TransportError::Closed)
|
||||
} else {
|
||||
self.raw.send_flow(payload, flow)
|
||||
}
|
||||
}
|
||||
fn recv(&self) -> BoxFuture<'_, Option<Bytes>> {
|
||||
Box::pin(async move {
|
||||
let mut stop = self.stop.subscribe();
|
||||
if *stop.borrow() {
|
||||
return None;
|
||||
}
|
||||
let mut rx = self.rx.lock().await;
|
||||
tokio::select! {biased;_=stop.changed()=>None,packet=rx.recv()=>packet}
|
||||
})
|
||||
}
|
||||
fn closed(&self) -> BoxFuture<'_, ()> {
|
||||
Box::pin(async move {
|
||||
let mut stop = self.stop.subscribe();
|
||||
if !*stop.borrow() {
|
||||
let _ = stop.changed().await;
|
||||
}
|
||||
})
|
||||
}
|
||||
fn is_closed(&self) -> bool {
|
||||
*self.stop.borrow() || self.raw.is_closed()
|
||||
}
|
||||
fn close(&self) {
|
||||
self.stop.send_replace(true);
|
||||
self.raw.close();
|
||||
}
|
||||
fn failure_reason(&self) -> Option<String> {
|
||||
lock(&self.reason)
|
||||
.clone()
|
||||
.or_else(|| self.raw.failure_reason())
|
||||
}
|
||||
fn path_description(&self) -> String {
|
||||
self.raw.path_description()
|
||||
}
|
||||
}
|
||||
impl Drop for HealthLink {
|
||||
fn drop(&mut self) {
|
||||
self.stop.send_replace(true);
|
||||
self.raw.close();
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
|
||||
use super::*;
|
||||
#[test]
|
||||
fn no_false_failure_before_window_but_persistent_loss_is_detected() {
|
||||
let mut samples = Samples::new();
|
||||
let now = Instant::now();
|
||||
for seq in 0..20 {
|
||||
samples.sent(seq, now);
|
||||
if seq % 5 == 0 {
|
||||
samples.acknowledge(seq);
|
||||
}
|
||||
}
|
||||
assert!(!samples.degraded());
|
||||
samples.expire(now + Duration::from_secs(4));
|
||||
assert!(samples.degraded());
|
||||
for seq in 20..40 {
|
||||
samples.sent(seq, now);
|
||||
samples.acknowledge(seq);
|
||||
}
|
||||
samples.expire(now + Duration::from_secs(4));
|
||||
assert!(!samples.degraded());
|
||||
}
|
||||
#[test]
|
||||
fn a_forged_or_reflected_probe_cannot_claim_delivery() {
|
||||
let a = crate::identity::DeviceIdentity::generate().endpoint_id();
|
||||
let b = crate::identity::DeviceIdentity::generate().endpoint_id();
|
||||
let settings = Settings::fresh();
|
||||
let packet = probe(&settings, a, 1, 42, 1100);
|
||||
assert_eq!(verify(&settings, a, &packet), Some((1, 42)));
|
||||
assert!(verify(&settings, b, &packet).is_none());
|
||||
let mut forged = packet.to_vec();
|
||||
forged[16] = 2;
|
||||
assert!(verify(&settings, a, &forged).is_none());
|
||||
}
|
||||
}
|
||||
@@ -97,6 +97,9 @@ impl IrohLink {
|
||||
}
|
||||
|
||||
impl PacketLink for IrohLink {
|
||||
fn close(&self) {
|
||||
self.conn.close(0u32.into(), b"data link retired");
|
||||
}
|
||||
fn network(&self) -> NetworkId {
|
||||
self.network
|
||||
}
|
||||
|
||||
@@ -30,12 +30,14 @@
|
||||
//! one plugin protocol.
|
||||
|
||||
mod fragments;
|
||||
pub(crate) mod health;
|
||||
pub mod iroh_link;
|
||||
|
||||
use std::sync::Arc;
|
||||
|
||||
use bytes::Bytes;
|
||||
use iroh::EndpointId;
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
use crate::BoxFuture;
|
||||
use crate::identity::NetworkId;
|
||||
@@ -112,6 +114,14 @@ pub trait PacketLink: Send + Sync + std::fmt::Debug + 'static {
|
||||
/// keeping a task parked on [`PacketLink::closed`].
|
||||
fn is_closed(&self) -> bool;
|
||||
|
||||
/// Explicitly retires a physical link. Implementations should wake readers.
|
||||
fn close(&self) {}
|
||||
|
||||
/// Observed reason for retiring this link, if available.
|
||||
fn failure_reason(&self) -> Option<String> {
|
||||
None
|
||||
}
|
||||
|
||||
/// A short description of the path in use, for diagnostics.
|
||||
///
|
||||
/// Reports what the transport actually knows. It must not invent a value.
|
||||
@@ -121,6 +131,76 @@ pub trait PacketLink: Send + Sync + std::fmt::Debug + 'static {
|
||||
/// A shared handle to a link.
|
||||
pub type SharedLink = std::sync::Arc<dyn PacketLink>;
|
||||
|
||||
/// Maximum opaque pair-negotiation payload.
|
||||
pub const MAX_TRANSPORT_PAYLOAD: usize = 4096;
|
||||
|
||||
/// Private transport bytes, never included in Debug output.
|
||||
#[derive(Clone, PartialEq, Eq, Serialize, zeroize::ZeroizeOnDrop)]
|
||||
pub struct TransportPayload(Vec<u8>);
|
||||
|
||||
impl<'de> Deserialize<'de> for TransportPayload {
|
||||
fn deserialize<D: serde::Deserializer<'de>>(
|
||||
deserializer: D,
|
||||
) -> std::result::Result<Self, D::Error> {
|
||||
struct Visitor;
|
||||
impl<'de> serde::de::Visitor<'de> for Visitor {
|
||||
type Value = TransportPayload;
|
||||
fn expecting(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
write!(f, "at most {MAX_TRANSPORT_PAYLOAD} private transport bytes")
|
||||
}
|
||||
fn visit_seq<A: serde::de::SeqAccess<'de>>(
|
||||
self,
|
||||
mut sequence: A,
|
||||
) -> std::result::Result<Self::Value, A::Error> {
|
||||
if sequence
|
||||
.size_hint()
|
||||
.is_some_and(|size| size > MAX_TRANSPORT_PAYLOAD)
|
||||
{
|
||||
return Err(serde::de::Error::custom("transport payload too large"));
|
||||
}
|
||||
let mut payload = TransportPayload(Vec::with_capacity(
|
||||
sequence.size_hint().unwrap_or(0).min(MAX_TRANSPORT_PAYLOAD),
|
||||
));
|
||||
while let Some(byte) = sequence.next_element::<u8>()? {
|
||||
if payload.0.len() == MAX_TRANSPORT_PAYLOAD {
|
||||
return Err(serde::de::Error::custom("transport payload too large"));
|
||||
}
|
||||
payload.0.push(byte);
|
||||
}
|
||||
Ok(payload)
|
||||
}
|
||||
}
|
||||
deserializer.deserialize_seq(Visitor)
|
||||
}
|
||||
}
|
||||
impl TransportPayload {
|
||||
/// Wraps bytes for one authenticated control session.
|
||||
pub fn new(bytes: Vec<u8>) -> Self {
|
||||
Self(bytes)
|
||||
}
|
||||
/// Borrows private bytes; callers must not log them.
|
||||
pub fn as_bytes(&self) -> &[u8] {
|
||||
&self.0
|
||||
}
|
||||
}
|
||||
impl std::fmt::Debug for TransportPayload {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
f.write_str("TransportPayload([REDACTED])")
|
||||
}
|
||||
}
|
||||
|
||||
/// Private control-plane sender for transports. Contains no network secret.
|
||||
pub trait ControlSender: Send + Sync + std::fmt::Debug + 'static {
|
||||
/// Queues a bounded message to one authenticated member of one network.
|
||||
fn send<'a>(
|
||||
&'a self,
|
||||
network: NetworkId,
|
||||
peer: EndpointId,
|
||||
protocol: &'a str,
|
||||
payload: TransportPayload,
|
||||
) -> BoxFuture<'a, Result<(), TransportError>>;
|
||||
}
|
||||
|
||||
/// An inbound link a peer opened towards us.
|
||||
#[derive(Debug)]
|
||||
pub struct InboundLink {
|
||||
@@ -139,6 +219,18 @@ pub struct InboundLink {
|
||||
/// The agent owns one of these and hands links to plugins; plugins never call
|
||||
/// it directly.
|
||||
pub trait PacketTransport: Send + Sync + std::fmt::Debug + 'static {
|
||||
/// Supplies a private sender before network activation.
|
||||
fn attach_control(&self, _control: Arc<dyn ControlSender>) {}
|
||||
/// Validates and queues private negotiation received from an authenticated member.
|
||||
fn on_control(
|
||||
&self,
|
||||
_network: NetworkId,
|
||||
_peer: EndpointId,
|
||||
_protocol: &str,
|
||||
_payload: &[u8],
|
||||
) -> Result<(), TransportError> {
|
||||
Err(TransportError::Declined(self.name().into()))
|
||||
}
|
||||
/// A short name used in diagnostics.
|
||||
fn name(&self) -> &str;
|
||||
|
||||
@@ -184,6 +276,24 @@ impl MultiPacketTransport {
|
||||
}
|
||||
|
||||
impl PacketTransport for MultiPacketTransport {
|
||||
fn attach_control(&self, control: Arc<dyn ControlSender>) {
|
||||
self.default.attach_control(control.clone());
|
||||
for transport in self.by_protocol.values() {
|
||||
transport.attach_control(control.clone());
|
||||
}
|
||||
}
|
||||
fn on_control(
|
||||
&self,
|
||||
network: NetworkId,
|
||||
peer: EndpointId,
|
||||
protocol: &str,
|
||||
payload: &[u8],
|
||||
) -> Result<(), TransportError> {
|
||||
match self.by_protocol.get(protocol) {
|
||||
Some(transport) => transport.on_control(network, peer, protocol, payload),
|
||||
None => Err(TransportError::Declined(protocol.into())),
|
||||
}
|
||||
}
|
||||
fn name(&self) -> &str {
|
||||
"multi"
|
||||
}
|
||||
|
||||
@@ -15,6 +15,12 @@ pub type Result<T, E = Error> = std::result::Result<T, E>;
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
#[non_exhaustive]
|
||||
pub enum Error {
|
||||
/// A node-history cleanup is unsafe, stale or cannot be performed.
|
||||
#[error("cannot forget node: {0}")]
|
||||
ForgetNode(String),
|
||||
/// An ordered local data-plane protocol selection is invalid.
|
||||
#[error("invalid protocol selection: {0}")]
|
||||
InvalidProtocols(String),
|
||||
/// The supplied network name does not satisfy the documented rules.
|
||||
#[error("invalid network name: {0}")]
|
||||
InvalidNetworkName(&'static str),
|
||||
@@ -97,6 +103,19 @@ pub enum Error {
|
||||
#[error("network {0} is not configured")]
|
||||
NetworkUnknown(NetworkId),
|
||||
|
||||
/// An overlay range overlaps networks already active on this agent.
|
||||
#[error(
|
||||
"network {network} cannot start: {range} overlaps active networks {conflicts:?}; switch to it to suspend those networks"
|
||||
)]
|
||||
NetworkRangeConflict {
|
||||
/// Network that could not start.
|
||||
network: NetworkId,
|
||||
/// Its agreed or proposed range.
|
||||
range: crate::state::Ipv4Range,
|
||||
/// Active networks that must be suspended first.
|
||||
conflicts: Vec<NetworkId>,
|
||||
},
|
||||
|
||||
/// No session with that peer exists in the given network.
|
||||
#[error("no authenticated session with peer {peer} in network {network}")]
|
||||
NoSuchPeer {
|
||||
|
||||
@@ -171,6 +171,33 @@ pub fn default_control_socket_path() -> PathBuf {
|
||||
#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[non_exhaustive]
|
||||
pub enum Request {
|
||||
/// Save a complete version-checked service draft or its deletion.
|
||||
SaveVip {
|
||||
/// Exact network id.
|
||||
network_id: String,
|
||||
/// Entire desired configuration.
|
||||
config: crate::vip::VipConfig,
|
||||
/// Revision originally displayed; absent for creation.
|
||||
expected: Option<String>,
|
||||
/// Remove the service permanently.
|
||||
delete: bool,
|
||||
},
|
||||
/// Retire the selected offline member's observed claim.
|
||||
ForgetNode {
|
||||
/// Exact public network id.
|
||||
network_id: String,
|
||||
/// Exact device identity selected by the client.
|
||||
peer: String,
|
||||
/// Claim version displayed when the user selected the node.
|
||||
expected_version: u64,
|
||||
},
|
||||
/// Replace one network's ordered local data-plane selection.
|
||||
SetProtocols {
|
||||
/// Public network identifier.
|
||||
network_id: String,
|
||||
/// Protocol names, best first; empty selects control only.
|
||||
protocols: Vec<String>,
|
||||
},
|
||||
/// Report what the agent is doing.
|
||||
Status,
|
||||
/// Set local broadcast participation in one configured network.
|
||||
@@ -255,6 +282,24 @@ pub enum Request {
|
||||
impl std::fmt::Debug for Request {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
match self {
|
||||
Request::SaveVip {
|
||||
network_id, config, ..
|
||||
} => write!(f, "SaveVip {{ {network_id}, {} }}", config.id),
|
||||
Request::ForgetNode {
|
||||
network_id,
|
||||
peer,
|
||||
expected_version,
|
||||
} => write!(
|
||||
f,
|
||||
"ForgetNode {{ {network_id}, peer: {peer}, version: {expected_version} }}"
|
||||
),
|
||||
Request::SetProtocols {
|
||||
network_id,
|
||||
protocols,
|
||||
} => write!(
|
||||
f,
|
||||
"SetProtocols {{ {network_id}, protocols: {protocols:?} }}"
|
||||
),
|
||||
Request::Status => f.write_str("Status"),
|
||||
Request::SetBroadcast {
|
||||
network_id,
|
||||
@@ -286,6 +331,12 @@ impl std::fmt::Debug for Request {
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[non_exhaustive]
|
||||
pub enum Response {
|
||||
/// Accepted configuration with fresh local routing observations.
|
||||
Vips(Vec<crate::vip::VipStatus>),
|
||||
/// An observed offline claim was durably retired.
|
||||
NodeForgotten(ForgottenNodeReport),
|
||||
/// Accepted ordered protocol selection.
|
||||
Protocols(Vec<String>),
|
||||
/// A status report.
|
||||
Status(Box<StatusReport>),
|
||||
/// Accepted local broadcast participation.
|
||||
@@ -312,6 +363,192 @@ pub enum Response {
|
||||
Error(String),
|
||||
}
|
||||
|
||||
/// The durable result of an offline-node cleanup.
|
||||
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct ForgottenNodeReport {
|
||||
/// Selected device identity.
|
||||
pub peer: String,
|
||||
/// The retired claim version, not a future-version ban.
|
||||
pub record_version: u64,
|
||||
/// Peers to which cleanup was immediately queued.
|
||||
pub peers_told: u32,
|
||||
}
|
||||
|
||||
/// Saves a whole VIP draft through the running agent.
|
||||
pub async fn save_vip(
|
||||
socket: &Path,
|
||||
network_id: &str,
|
||||
config: crate::vip::VipConfig,
|
||||
expected: Option<String>,
|
||||
delete: bool,
|
||||
) -> Result<Vec<crate::vip::VipStatus>> {
|
||||
match exchange(
|
||||
socket,
|
||||
&Request::SaveVip {
|
||||
network_id: network_id.into(),
|
||||
config,
|
||||
expected,
|
||||
delete,
|
||||
},
|
||||
EXCHANGE_TIMEOUT,
|
||||
)
|
||||
.await?
|
||||
{
|
||||
Response::Vips(vips) => Ok(vips),
|
||||
Response::Error(reason) => Err(Error::Rejected { reason }),
|
||||
_ => Err(Error::Rejected {
|
||||
reason: "unexpected VIP response".into(),
|
||||
}),
|
||||
}
|
||||
}
|
||||
/// Resolves a service by exact name, full id or unique id prefix.
|
||||
pub fn resolve_vip<'a>(
|
||||
network: &'a NetworkReport,
|
||||
selector: &str,
|
||||
) -> std::result::Result<&'a crate::vip::VipStatus, String> {
|
||||
let matches: Vec<_> = network
|
||||
.vips
|
||||
.iter()
|
||||
.filter(|v| v.config.name == selector || v.config.id.starts_with(selector))
|
||||
.collect();
|
||||
match matches.as_slice() {
|
||||
[vip] => Ok(vip),
|
||||
[] => Err(format!("no VIP matches `{selector}`")),
|
||||
_ => Err(format!(
|
||||
"ambiguous VIP `{selector}`: {}",
|
||||
matches
|
||||
.iter()
|
||||
.map(|v| format!("{} ({})", v.config.name, v.config.id))
|
||||
.collect::<Vec<_>>()
|
||||
.join(", ")
|
||||
)),
|
||||
}
|
||||
}
|
||||
|
||||
/// Resolves a node by exact hostname, full id, unique id prefix or overlay IPv4.
|
||||
/// Ambiguity is reported with names and full ids and never guessed.
|
||||
pub fn resolve_node<'a>(
|
||||
network: &'a NetworkReport,
|
||||
selector: &str,
|
||||
) -> std::result::Result<&'a MemberReport, String> {
|
||||
let selector = selector.trim();
|
||||
if selector.is_empty() {
|
||||
return Err("select a node by name, id or IPv4 address".into());
|
||||
}
|
||||
if let Some(member) = network.members.iter().find(|m| m.endpoint_id == selector) {
|
||||
return Ok(member);
|
||||
}
|
||||
let prefix = selector.trim_end_matches('…');
|
||||
let matches: Vec<_> = network
|
||||
.members
|
||||
.iter()
|
||||
.filter(|m| {
|
||||
m.hostname.as_deref() == Some(selector)
|
||||
|| m.overlay_address_v4.as_deref() == Some(selector)
|
||||
|| (!prefix.is_empty() && m.endpoint_id.starts_with(prefix))
|
||||
})
|
||||
.collect();
|
||||
match matches.as_slice() {
|
||||
[one] => Ok(one),
|
||||
[] => Err(format!(
|
||||
"no signed node matches `{selector}` in `{}`",
|
||||
network.name
|
||||
)),
|
||||
many => Err(format!(
|
||||
"`{selector}` is ambiguous; use a full id or unique id prefix:\n{}",
|
||||
many.iter()
|
||||
.map(|m| format!(
|
||||
" {} {} {}",
|
||||
m.hostname.as_deref().unwrap_or("unnamed"),
|
||||
m.endpoint_id,
|
||||
m.overlay_address_v4.as_deref().unwrap_or("no IPv4")
|
||||
))
|
||||
.collect::<Vec<_>>()
|
||||
.join("\n")
|
||||
)),
|
||||
}
|
||||
}
|
||||
|
||||
/// Retires an exact selected claim; an online or changed target is rejected.
|
||||
pub async fn forget_node(
|
||||
socket: &Path,
|
||||
network: &str,
|
||||
peer: &str,
|
||||
expected_version: u64,
|
||||
) -> Result<ForgottenNodeReport> {
|
||||
let request = Request::ForgetNode {
|
||||
network_id: network.into(),
|
||||
peer: peer.into(),
|
||||
expected_version,
|
||||
};
|
||||
match exchange(socket, &request, EXCHANGE_TIMEOUT).await? {
|
||||
Response::NodeForgotten(report) => Ok(report),
|
||||
Response::Error(reason) => Err(Error::ForgetNode(reason)),
|
||||
other => Err(Error::Storage(format!("unexpected response: {other:?}"))),
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod node_selector_tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn exact_names_addresses_and_unique_prefixes_select_the_same_signed_node() {
|
||||
let node = MemberReport {
|
||||
endpoint_id: "abcdef01".into(),
|
||||
hostname: Some("laptop".into()),
|
||||
overlay_address_v4: Some("10.2.0.3".into()),
|
||||
record_version: 42,
|
||||
..Default::default()
|
||||
};
|
||||
let network = NetworkReport {
|
||||
name: "home".into(),
|
||||
members: vec![node.clone()],
|
||||
..Default::default()
|
||||
};
|
||||
for selector in ["laptop", "10.2.0.3", "abcdef01", "abc…"] {
|
||||
assert_eq!(resolve_node(&network, selector), Ok(&node));
|
||||
}
|
||||
for selector in ["", "LAPTOP", "missing"] {
|
||||
assert!(resolve_node(&network, selector).is_err());
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn collisions_between_names_addresses_and_prefixes_require_an_explicit_identity() {
|
||||
let first = MemberReport {
|
||||
endpoint_id: "abcdef01".into(),
|
||||
hostname: Some("duplicate".into()),
|
||||
overlay_address_v4: Some("10.2.0.3".into()),
|
||||
..Default::default()
|
||||
};
|
||||
let second = MemberReport {
|
||||
endpoint_id: "abcdef02".into(),
|
||||
hostname: Some("duplicate".into()),
|
||||
overlay_address_v4: Some("10.2.0.3".into()),
|
||||
..Default::default()
|
||||
};
|
||||
let third = MemberReport {
|
||||
endpoint_id: "12345678".into(),
|
||||
hostname: Some("abc".into()),
|
||||
..Default::default()
|
||||
};
|
||||
let network = NetworkReport {
|
||||
name: "home".into(),
|
||||
members: vec![first.clone(), second, third],
|
||||
..Default::default()
|
||||
};
|
||||
for selector in ["duplicate", "10.2.0.3", "abc"] {
|
||||
let result = resolve_node(&network, selector);
|
||||
assert!(result.is_err_and(|error| error.contains("ambiguous")
|
||||
&& error.contains("abcdef01")
|
||||
&& error.contains("abcdef02")
|
||||
&& error.contains("duplicate")));
|
||||
}
|
||||
assert_eq!(resolve_node(&network, "abcdef01"), Ok(&first));
|
||||
}
|
||||
}
|
||||
|
||||
/// What happened when a network was stopped or started.
|
||||
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct ActiveReport {
|
||||
@@ -324,6 +561,9 @@ pub struct ActiveReport {
|
||||
/// Stopping something already stopped is not an error — it is the state
|
||||
/// asked for — but it is worth saying which happened.
|
||||
pub changed: bool,
|
||||
/// Networks suspended by switching to this network.
|
||||
#[serde(default)]
|
||||
pub suspended: Vec<RangeConflictReport>,
|
||||
}
|
||||
|
||||
/// What happened when a network was joined.
|
||||
@@ -345,6 +585,13 @@ pub struct JoinedReport {
|
||||
/// Almost always a mistyped secret, and the one thing that makes two
|
||||
/// sections of a report look like one network.
|
||||
pub name_shared_with: Option<String>,
|
||||
/// Whether it started; false means it is configured but blocked by overlap.
|
||||
#[serde(default = "joined_active_default")]
|
||||
pub active: bool,
|
||||
}
|
||||
|
||||
fn joined_active_default() -> bool {
|
||||
true
|
||||
}
|
||||
|
||||
/// What happened when a network was left.
|
||||
@@ -367,6 +614,8 @@ pub struct LeftReport {
|
||||
/// Everything the agent is doing, in one snapshot.
|
||||
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct StatusReport {
|
||||
/// Data-plane protocols this agent can enable at runtime.
|
||||
pub available_protocols: Vec<ProtocolReport>,
|
||||
/// This device's persistent endpoint id.
|
||||
pub endpoint_id: String,
|
||||
/// Hostname announced to peers.
|
||||
@@ -437,9 +686,37 @@ pub struct DnsReport {
|
||||
pub names: u32,
|
||||
}
|
||||
|
||||
/// An active network with an overlapping overlay range.
|
||||
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct RangeConflictReport {
|
||||
/// Public network id, for unambiguous selection.
|
||||
pub network_id: String,
|
||||
/// Configured network name.
|
||||
pub name: String,
|
||||
/// The overlapping range.
|
||||
pub range: String,
|
||||
}
|
||||
|
||||
/// A data-plane protocol available in the running agent.
|
||||
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct ProtocolReport {
|
||||
/// Stable protocol name used by the CLI and network negotiation.
|
||||
pub name: String,
|
||||
/// A short description of encryption and transport.
|
||||
pub summary: String,
|
||||
}
|
||||
|
||||
/// One network.
|
||||
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct NetworkReport {
|
||||
/// Shared VIPs and this client's routing observations.
|
||||
#[serde(default)]
|
||||
pub vips: Vec<crate::vip::VipStatus>,
|
||||
/// Enabled local data-plane protocols, in preference order.
|
||||
pub protocols: Vec<String>,
|
||||
/// Whether the network starts or resumes automatically when its range is free.
|
||||
#[serde(default)]
|
||||
pub auto_start: bool,
|
||||
/// Whether this agent participates in LAN broadcasts in this network.
|
||||
#[serde(default)]
|
||||
pub broadcast: bool,
|
||||
@@ -482,16 +759,53 @@ pub struct NetworkReport {
|
||||
/// The range it could not have, because another network on this agent
|
||||
/// already holds it.
|
||||
pub range_conflict: Option<String>,
|
||||
/// Networks that switching to this network would suspend.
|
||||
#[serde(default)]
|
||||
pub range_conflicts: Vec<RangeConflictReport>,
|
||||
/// The overlay, when a protocol is running one.
|
||||
pub overlay: Option<OverlayReport>,
|
||||
/// This agent's own name, when another member claimed it first.
|
||||
pub hostname_conflict: Option<HostnameConflictReport>,
|
||||
/// Exit-node settings: whether this agent offers one, and whether it
|
||||
/// uses one.
|
||||
pub exit: ExitReport,
|
||||
}
|
||||
|
||||
/// How packets reach a member right now.
|
||||
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub enum DataPathReport {
|
||||
/// Nothing carries packets to it: it is away, or only connected for
|
||||
/// control while a data path is being set up.
|
||||
#[default]
|
||||
None,
|
||||
/// A direct link.
|
||||
Direct {
|
||||
/// The kind of transport carrying it, e.g. `wg`.
|
||||
transport: String,
|
||||
},
|
||||
/// Through other members.
|
||||
Relayed {
|
||||
/// Number of links on the way, at least two.
|
||||
hops: u8,
|
||||
/// The member the first of them goes to, as an endpoint id.
|
||||
via: String,
|
||||
},
|
||||
}
|
||||
|
||||
/// A name somebody else claimed first.
|
||||
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct HostnameConflictReport {
|
||||
/// The name that is taken.
|
||||
pub name: String,
|
||||
/// The member that holds it, as an endpoint id.
|
||||
pub holder: String,
|
||||
}
|
||||
|
||||
/// One member of the network, from signed state.
|
||||
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct MemberReport {
|
||||
/// Signed claim version used to reject stale cleanup requests.
|
||||
pub record_version: u64,
|
||||
/// The member's device identity.
|
||||
pub endpoint_id: String,
|
||||
/// The IPv4 overlay address it claimed and signed for.
|
||||
@@ -500,6 +814,15 @@ pub struct MemberReport {
|
||||
pub failed_dials: u32,
|
||||
/// The last hostname this agent saw it announce, kept while it is away.
|
||||
pub hostname: Option<String>,
|
||||
/// Whether this agent can reach it now, by any path. The one answer
|
||||
/// every client gives to "is it online": a member reached through
|
||||
/// others is online, over a longer path.
|
||||
pub online: bool,
|
||||
/// How packets reach it.
|
||||
pub path: DataPathReport,
|
||||
/// The name it claimed, when somebody else claimed it first. It still
|
||||
/// works by address; it is not found by that name.
|
||||
pub hostname_conflict: Option<HostnameConflictReport>,
|
||||
}
|
||||
|
||||
/// One control plane peer.
|
||||
@@ -516,6 +839,8 @@ pub struct PeerReport {
|
||||
pub rtt_ms: Option<u64>,
|
||||
/// The peer offers itself as an exit node in this network.
|
||||
pub exit_node: bool,
|
||||
/// How packets reach it.
|
||||
pub path: DataPathReport,
|
||||
}
|
||||
|
||||
/// The WireGuard overlay of one network.
|
||||
@@ -650,6 +975,33 @@ impl OverlayPeerReport {
|
||||
/// running and what they can report or change. That is what keeps this module
|
||||
/// free of any knowledge of them.
|
||||
pub trait ReportSource: Send + Sync + 'static {
|
||||
/// Atomically saves a complete VIP draft using its original revision.
|
||||
fn save_vip(
|
||||
&self,
|
||||
_network: String,
|
||||
_config: crate::vip::VipConfig,
|
||||
_expected: Option<String>,
|
||||
_delete: bool,
|
||||
) -> BoxFuture<'_, std::result::Result<Vec<crate::vip::VipStatus>, String>> {
|
||||
Box::pin(async { Err("this agent cannot manage VIPs".into()) })
|
||||
}
|
||||
/// Persists and announces a version-scoped node cleanup.
|
||||
fn forget_node(
|
||||
&self,
|
||||
_network: String,
|
||||
_peer: String,
|
||||
_expected_version: u64,
|
||||
) -> BoxFuture<'_, std::result::Result<ForgottenNodeReport, String>> {
|
||||
Box::pin(async { Err("this agent cannot forget node history".into()) })
|
||||
}
|
||||
/// Saves and applies a local ordered protocol set for one network.
|
||||
fn set_protocols(
|
||||
&self,
|
||||
_network_id: String,
|
||||
_protocols: Vec<String>,
|
||||
) -> BoxFuture<'_, std::result::Result<Vec<String>, String>> {
|
||||
Box::pin(async { Err("this agent cannot change protocols".into()) })
|
||||
}
|
||||
/// Produces a fresh report.
|
||||
fn report(&self) -> BoxFuture<'_, StatusReport>;
|
||||
|
||||
@@ -779,7 +1131,7 @@ pub const EXCHANGE_TIMEOUT: Duration = Duration::from_secs(5);
|
||||
///
|
||||
/// Bump it whenever [`Request`], [`Response`] or anything they contain
|
||||
/// changes shape.
|
||||
pub const CONTROL_PROTOCOL: u32 = u32::from_be_bytes([b'T', b'S', b'N', 17]);
|
||||
pub const CONTROL_PROTOCOL: u32 = u32::from_be_bytes([b'T', b'S', b'N', 22]);
|
||||
|
||||
/// Reads one request off an accepted stream, answers it, writes the response.
|
||||
///
|
||||
@@ -803,6 +1155,30 @@ where
|
||||
}
|
||||
};
|
||||
let response = match request {
|
||||
Request::SaveVip {
|
||||
network_id,
|
||||
config,
|
||||
expected,
|
||||
delete,
|
||||
} => match source.save_vip(network_id, config, expected, delete).await {
|
||||
Ok(vips) => Response::Vips(vips),
|
||||
Err(reason) => Response::Error(reason),
|
||||
},
|
||||
Request::ForgetNode {
|
||||
network_id,
|
||||
peer,
|
||||
expected_version,
|
||||
} => match source.forget_node(network_id, peer, expected_version).await {
|
||||
Ok(report) => Response::NodeForgotten(report),
|
||||
Err(reason) => Response::Error(reason),
|
||||
},
|
||||
Request::SetProtocols {
|
||||
network_id,
|
||||
protocols,
|
||||
} => match source.set_protocols(network_id, protocols).await {
|
||||
Ok(protocols) => Response::Protocols(protocols),
|
||||
Err(reason) => Response::Error(reason),
|
||||
},
|
||||
Request::Status => Response::Status(Box::new(source.report().await)),
|
||||
Request::SetBroadcast {
|
||||
network_id,
|
||||
@@ -958,7 +1334,24 @@ pub async fn join_network_with_broadcast(
|
||||
}
|
||||
}
|
||||
|
||||
/// Changes one network's broadcast participation while the agent runs.
|
||||
/// Replaces the enabled data-plane protocols of one configured network.
|
||||
pub async fn set_protocols(
|
||||
socket: &Path,
|
||||
network_id: &str,
|
||||
protocols: Vec<String>,
|
||||
) -> Result<Vec<String>> {
|
||||
let request = Request::SetProtocols {
|
||||
network_id: network_id.to_owned(),
|
||||
protocols,
|
||||
};
|
||||
match exchange(socket, &request, EXCHANGE_TIMEOUT).await? {
|
||||
Response::Protocols(protocols) => Ok(protocols),
|
||||
Response::Error(reason) => Err(Error::Storage(reason)),
|
||||
other => Err(Error::Storage(format!("unexpected response: {other:?}"))),
|
||||
}
|
||||
}
|
||||
|
||||
/// Changes local broadcast participation in a network.
|
||||
pub async fn set_broadcast(
|
||||
path: impl AsRef<Path>,
|
||||
network_id: &str,
|
||||
|
||||
@@ -51,6 +51,7 @@ pub mod storage;
|
||||
pub mod task;
|
||||
#[cfg(feature = "testing")]
|
||||
pub mod testing;
|
||||
pub mod vip;
|
||||
|
||||
pub use agent::{Agent, AgentStatus, Event, NetworkStatus, PeerStatus};
|
||||
pub use config::{AgentConfig, Limits, ReconnectPolicy, StoragePaths, TransportPolicy};
|
||||
|
||||
@@ -84,6 +84,11 @@ pub trait ExitHostRules: Send + Sync + std::fmt::Debug + 'static {
|
||||
/// startup to take away what a crashed run left.
|
||||
fn clear<'a>(&'a self, interface: &'a str) -> BoxFuture<'a, ()>;
|
||||
|
||||
/// Refreshes volatile local socket protection without replacing exit routes.
|
||||
fn refresh<'a>(&'a self, _plan: &'a ExitHostPlan) -> BoxFuture<'a, Option<RuleOutcome>> {
|
||||
Box::pin(async { None })
|
||||
}
|
||||
|
||||
/// Whether using an exit node on this host needs the underlay addresses
|
||||
/// the agent talks to, because the host cannot exempt the agent's own
|
||||
/// traffic from the default route by user id (Windows). The default is no.
|
||||
|
||||
@@ -33,12 +33,13 @@ const CHAIN: &str = "INPUT";
|
||||
|
||||
/// Fixed places `iptables` is installed. Never `PATH`: the agent is
|
||||
/// privileged and a directory a user controls must not decide what it runs.
|
||||
const IPTABLES_DIRECTORIES: [&str; 5] = [
|
||||
const IPTABLES_DIRECTORIES: [&str; 6] = [
|
||||
"/usr/sbin/iptables",
|
||||
"/sbin/iptables",
|
||||
"/usr/bin/iptables",
|
||||
"/bin/iptables",
|
||||
"/usr/local/sbin/iptables",
|
||||
"/run/current-system/sw/bin/iptables",
|
||||
];
|
||||
|
||||
/// The first candidate that exists.
|
||||
@@ -328,7 +329,23 @@ fn clear_firewall(run: Run<'_>, interface: &str) {
|
||||
|
||||
/// Runs the real `iptables`.
|
||||
pub(super) fn run_iptables(args: &[String]) -> Ran {
|
||||
let Some(program) = pick_iptables(|path| std::path::Path::new(path).is_file()) else {
|
||||
run_firewall(false, args)
|
||||
}
|
||||
|
||||
pub(super) fn run_firewall(ipv6: bool, args: &[String]) -> Ran {
|
||||
let v6_paths: Vec<_> = IPTABLES_DIRECTORIES
|
||||
.iter()
|
||||
.map(|p| p.replace("iptables", "ip6tables"))
|
||||
.collect();
|
||||
let v6_program = v6_paths
|
||||
.iter()
|
||||
.find(|path| std::path::Path::new(path).is_file());
|
||||
let program = if ipv6 {
|
||||
v6_program.map(String::as_str)
|
||||
} else {
|
||||
pick_iptables(|path| std::path::Path::new(path).is_file())
|
||||
};
|
||||
let Some(program) = program else {
|
||||
return Ran::Missing;
|
||||
};
|
||||
match std::process::Command::new(program).args(args).output() {
|
||||
@@ -350,11 +367,17 @@ pub(super) fn run_iptables(args: &[String]) -> Ran {
|
||||
#[derive(Debug)]
|
||||
pub struct LinuxHostRules {
|
||||
pub(super) route: RouteHandle,
|
||||
/// Interface, sorted socket snapshot and whether exit routes committed.
|
||||
pub(super) underlay_sockets:
|
||||
std::sync::Mutex<Option<(String, Vec<super::linux_exit_mark::Listener>, bool)>>,
|
||||
}
|
||||
|
||||
impl LinuxHostRules {
|
||||
pub(crate) fn new(route: RouteHandle) -> Self {
|
||||
Self { route }
|
||||
Self {
|
||||
route,
|
||||
underlay_sockets: std::sync::Mutex::new(None),
|
||||
}
|
||||
}
|
||||
|
||||
async fn set_route(&self, interface: String, source: Ipv4Addr) -> RuleOutcome {
|
||||
|
||||
@@ -25,18 +25,19 @@
|
||||
//!
|
||||
//! ```text
|
||||
//! table 0x7473: default dev <if>
|
||||
//! 5280: from all uidrange <euid>-<euid> lookup main
|
||||
//! 5205: from all fwmark 0x40000000/0x40000000 lookup main
|
||||
//! 5206: from all uidrange <euid>-<euid> lookup main
|
||||
//! 5290: from all lookup main suppress_prefixlength 0
|
||||
//! 5300: from all lookup 0x7473
|
||||
//! ```
|
||||
//!
|
||||
//! The first keeps the agent's *own* sockets off the tunnel: its QUIC, relay,
|
||||
//! The first two keep the agent's *own* sockets and marked replies off the tunnel: its QUIC, relay,
|
||||
//! DHT and DNS traffic would otherwise be routed into the interface it is
|
||||
//! carrying, a loop. The second lets every specific route — the LAN, another
|
||||
//! carrying, a loop. The specific-route rule lets the LAN, another
|
||||
//! VPN, a tunnel to somewhere else — keep working, because only the default
|
||||
//! route is suppressed. The priorities sit after Tailscale's 5270.
|
||||
//! route is suppressed. Mark/UID bypass precedes Tailscale; application rules follow it.
|
||||
//!
|
||||
//! The first rule exempts a *user id*, so it is only right when the agent has
|
||||
//! The UID rule exempts a *user id*, so it is only right when the agent has
|
||||
//! one of its own, as the packaged service does. An agent started from a login
|
||||
//! session would take that user's own applications out of the tunnel with it,
|
||||
//! and the exit node would silently do nothing for them. That is refused, not
|
||||
@@ -48,9 +49,10 @@ use crate::state::Ipv4Range;
|
||||
use super::RuleOutcome;
|
||||
use super::exit::{ExitHostPlan, ExitHostReport, ExitHostRules, exit_tag, exit_tag_prefix};
|
||||
use super::linux::{
|
||||
LinuxHostRules, Ran, Run, comment_supported, failure, locked, run_iptables, same_rule,
|
||||
split_rule_line,
|
||||
LinuxHostRules, Ran, Run, comment_supported, failure, locked, run_firewall, run_iptables,
|
||||
same_rule, split_rule_line,
|
||||
};
|
||||
use super::linux_exit_mark;
|
||||
|
||||
/// The chains the offering rules live in: `(table, chain)`.
|
||||
const CHAINS: [(&str, &str); 2] = [("nat", "POSTROUTING"), ("filter", "FORWARD")];
|
||||
@@ -377,6 +379,32 @@ fn read_sysctl(path: &str) -> Option<String> {
|
||||
std::fs::read_to_string(path).ok()
|
||||
}
|
||||
|
||||
fn check_kernel_rpf(overlay: &str) -> Result<(), String> {
|
||||
let directory = "/proc/sys/net/ipv4/conf";
|
||||
let value = |name: &str, setting: &str| -> Result<u32, String> {
|
||||
read_sysctl(&format!("{directory}/{name}/{setting}"))
|
||||
.and_then(|s| s.trim().parse().ok())
|
||||
.ok_or_else(|| format!("cannot check {name}/{setting} before installing exit routes"))
|
||||
};
|
||||
let mut interfaces = Vec::new();
|
||||
for entry in std::fs::read_dir(directory).map_err(|err| err.to_string())? {
|
||||
let entry = entry.map_err(|err| err.to_string())?;
|
||||
let name = entry.file_name().to_string_lossy().into_owned();
|
||||
if name != "all" && name != "default" && name != "lo" && name != overlay {
|
||||
interfaces.push((
|
||||
name.clone(),
|
||||
value(&name, "rp_filter")?,
|
||||
value(&name, "src_valid_mark")?,
|
||||
));
|
||||
}
|
||||
}
|
||||
linux_exit_mark::check_kernel_rpf(
|
||||
value("all", "rp_filter")?,
|
||||
value("all", "src_valid_mark")?,
|
||||
&interfaces,
|
||||
)
|
||||
}
|
||||
|
||||
/// The effective user id from the text of `/proc/self/status`.
|
||||
fn parse_euid(status: &str) -> Option<u32> {
|
||||
status
|
||||
@@ -417,7 +445,7 @@ impl LinuxHostRules {
|
||||
let xdg = std::env::var("XDG_RUNTIME_DIR").ok();
|
||||
if let Some(reason) = client_refusal(xdg.as_deref()) {
|
||||
// Whatever an earlier run left must not stay in force.
|
||||
self.clear_client().await;
|
||||
let _ = self.clear_client(interface).await;
|
||||
return RuleOutcome::Failed(reason);
|
||||
}
|
||||
let Some(euid) = std::fs::read_to_string("/proc/self/status")
|
||||
@@ -426,24 +454,147 @@ impl LinuxHostRules {
|
||||
else {
|
||||
return RuleOutcome::Failed("cannot tell which user this agent runs as".to_string());
|
||||
};
|
||||
let installed = self
|
||||
.underlay_sockets
|
||||
.lock()
|
||||
.unwrap_or_else(|p| p.into_inner())
|
||||
.as_ref()
|
||||
.is_some_and(|(name, _, routed)| name == interface && *routed);
|
||||
if installed {
|
||||
// A failed socket refresh must never remove an active exit route.
|
||||
return self
|
||||
.refresh_sockets(interface)
|
||||
.await
|
||||
.unwrap_or(RuleOutcome::Applied);
|
||||
}
|
||||
if let Err(reason) = self.clear_client(interface).await {
|
||||
return RuleOutcome::Failed(reason);
|
||||
}
|
||||
let name = interface.to_string();
|
||||
let prepared = Self::blocking(move || {
|
||||
check_kernel_rpf(&name)?;
|
||||
linux_exit_mark::install(
|
||||
&run_firewall,
|
||||
&name,
|
||||
euid,
|
||||
std::path::Path::new("/proc/sys/net/ipv6").is_dir(),
|
||||
)
|
||||
})
|
||||
.await;
|
||||
match prepared {
|
||||
Ok(Ok(sockets)) => {
|
||||
*self
|
||||
.underlay_sockets
|
||||
.lock()
|
||||
.unwrap_or_else(|p| p.into_inner()) = Some((interface.into(), sockets, false));
|
||||
}
|
||||
Ok(Err(reason)) | Err(reason) => return RuleOutcome::Failed(reason),
|
||||
}
|
||||
let route = self.route.clone();
|
||||
let interface = interface.to_string();
|
||||
match Self::blocking(move || route.set_exit_client(&interface, euid)).await {
|
||||
let name = interface.to_string();
|
||||
let outcome = match Self::blocking(move || route.set_exit_client(&name, euid)).await {
|
||||
Ok(Ok(())) => RuleOutcome::Applied,
|
||||
Ok(Err(err)) => RuleOutcome::Failed(err.to_string()),
|
||||
Err(reason) => RuleOutcome::Failed(reason),
|
||||
};
|
||||
if let RuleOutcome::Failed(reason) = outcome {
|
||||
let reason = match self.clear_client(interface).await {
|
||||
Ok(()) => reason,
|
||||
Err(cleanup) => format!("{reason}; cannot roll back exit rules: {cleanup}"),
|
||||
};
|
||||
RuleOutcome::Failed(reason)
|
||||
} else {
|
||||
if let Some((_, _, routed)) = self
|
||||
.underlay_sockets
|
||||
.lock()
|
||||
.unwrap_or_else(|p| p.into_inner())
|
||||
.as_mut()
|
||||
{
|
||||
*routed = true;
|
||||
}
|
||||
outcome
|
||||
}
|
||||
}
|
||||
|
||||
async fn clear_client(&self) {
|
||||
async fn clear_client(&self, interface: &str) -> Result<(), String> {
|
||||
let route = self.route.clone();
|
||||
if let Ok(Err(err)) = Self::blocking(move || route.clear_exit_client()).await {
|
||||
tracing::warn!(%err, "cannot remove the exit node routing rules");
|
||||
Self::blocking(move || route.clear_exit_client())
|
||||
.await?
|
||||
.map_err(|err| err.to_string())?;
|
||||
// Route removal must precede removing reply protection.
|
||||
let name = interface.to_string();
|
||||
Self::blocking(move || {
|
||||
linux_exit_mark::clear(
|
||||
&run_firewall,
|
||||
&name,
|
||||
std::path::Path::new("/proc/sys/net/ipv6").is_dir(),
|
||||
)
|
||||
})
|
||||
.await??;
|
||||
*self
|
||||
.underlay_sockets
|
||||
.lock()
|
||||
.unwrap_or_else(|p| p.into_inner()) = None;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn refresh_sockets(&self, interface: &str) -> Option<RuleOutcome> {
|
||||
let previous = self
|
||||
.underlay_sockets
|
||||
.lock()
|
||||
.unwrap_or_else(|p| p.into_inner())
|
||||
.clone();
|
||||
if previous
|
||||
.as_ref()
|
||||
.is_none_or(|(name, _, routed)| name != interface || !routed)
|
||||
{
|
||||
return None;
|
||||
}
|
||||
let name = interface.to_string();
|
||||
let work_name = name.clone();
|
||||
let result = Self::blocking(move || {
|
||||
let uid = std::fs::read_to_string("/proc/self/status")
|
||||
.ok()
|
||||
.and_then(|s| parse_euid(&s))
|
||||
.ok_or("cannot read agent UID")?;
|
||||
check_kernel_rpf(&work_name)?;
|
||||
let v6 = std::path::Path::new("/proc/sys/net/ipv6").is_dir();
|
||||
let sockets = linux_exit_mark::local_sockets(uid, v6)?;
|
||||
if previous
|
||||
.as_ref()
|
||||
.is_some_and(|(name, old, _)| name == &work_name && old == &sockets)
|
||||
{
|
||||
return Ok(None);
|
||||
}
|
||||
linux_exit_mark::refresh(&run_firewall, &work_name, v6, &sockets)?;
|
||||
Ok(Some(sockets))
|
||||
})
|
||||
.await;
|
||||
match result {
|
||||
Ok(Ok(None)) => None,
|
||||
Ok(Ok(Some(sockets))) => {
|
||||
*self
|
||||
.underlay_sockets
|
||||
.lock()
|
||||
.unwrap_or_else(|p| p.into_inner()) = Some((name, sockets, true));
|
||||
Some(RuleOutcome::Applied)
|
||||
}
|
||||
Ok(Err(reason)) | Err(reason) => Some(RuleOutcome::Failed(reason)),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl ExitHostRules for LinuxHostRules {
|
||||
fn refresh<'a>(&'a self, plan: &'a ExitHostPlan) -> BoxFuture<'a, Option<RuleOutcome>> {
|
||||
Box::pin(async move {
|
||||
if plan.client {
|
||||
self.refresh_sockets(&plan.interface).await
|
||||
} else {
|
||||
None
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
fn apply<'a>(&'a self, plan: &'a ExitHostPlan) -> BoxFuture<'a, ExitHostReport> {
|
||||
Box::pin(async move {
|
||||
let interface = plan.interface.clone();
|
||||
@@ -467,7 +618,9 @@ impl ExitHostRules for LinuxHostRules {
|
||||
let client = if plan.client {
|
||||
Some(self.apply_client(&interface).await)
|
||||
} else {
|
||||
self.clear_client().await;
|
||||
if let Err(reason) = self.clear_client(&interface).await {
|
||||
tracing::warn!(%reason, "cannot remove exit-client rules");
|
||||
}
|
||||
None
|
||||
};
|
||||
ExitHostReport {
|
||||
@@ -482,7 +635,9 @@ impl ExitHostRules for LinuxHostRules {
|
||||
Box::pin(async move {
|
||||
let name = interface.to_string();
|
||||
let _ = Self::blocking(move || clear_offer(&run_iptables, &name)).await;
|
||||
self.clear_client().await;
|
||||
if let Err(reason) = self.clear_client(interface).await {
|
||||
tracing::warn!(%reason, "cannot remove exit-client rules");
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,439 @@
|
||||
//! Privileged routing regression, isolated from the host in named namespaces.
|
||||
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
|
||||
|
||||
use std::io::{BufRead, BufReader, Read, Write};
|
||||
use std::net::{SocketAddr, TcpListener, TcpStream, UdpSocket};
|
||||
use std::os::unix::process::CommandExt;
|
||||
use std::process::{Child, Command};
|
||||
use std::time::Duration;
|
||||
|
||||
use super::linux::run_firewall;
|
||||
use super::linux_exit_mark::{clear, install, local_sockets, refresh};
|
||||
use crate::overlay::NetlinkProvisioner;
|
||||
|
||||
const TEST: &str = "overlay::hostrules::linux_exit_kernel::strict_firewall_preserves_underlay_and_application_exit_routing";
|
||||
const ROLE: &str = "TSUNAGI_EXIT_KERNEL_TEST_ROLE";
|
||||
|
||||
fn ip(arguments: &[&str]) -> String {
|
||||
let result = Command::new("ip").args(arguments).output().unwrap();
|
||||
assert!(
|
||||
result.status.success(),
|
||||
"ip {arguments:?}: {}",
|
||||
String::from_utf8_lossy(&result.stderr)
|
||||
);
|
||||
String::from_utf8_lossy(&result.stdout).into_owned()
|
||||
}
|
||||
|
||||
struct Namespace(String);
|
||||
impl Drop for Namespace {
|
||||
fn drop(&mut self) {
|
||||
let _ = Command::new("ip").args(["netns", "del", &self.0]).status();
|
||||
}
|
||||
}
|
||||
struct Echo(Child);
|
||||
impl Drop for Echo {
|
||||
fn drop(&mut self) {
|
||||
let _ = self.0.kill();
|
||||
let _ = self.0.wait();
|
||||
}
|
||||
}
|
||||
|
||||
fn probes() {
|
||||
let should_pass = std::env::var("TSUNAGI_EXIT_KERNEL_EXPECT").unwrap() == "pass";
|
||||
for address in ["198.51.100.2:42000", "[2001:db8:2::2]:42000"] {
|
||||
let address: SocketAddr = address.parse().unwrap();
|
||||
let socket = UdpSocket::bind(if address.is_ipv4() {
|
||||
"0.0.0.0:0"
|
||||
} else {
|
||||
"[::]:0"
|
||||
})
|
||||
.unwrap();
|
||||
socket
|
||||
.set_read_timeout(Some(Duration::from_secs(1)))
|
||||
.unwrap();
|
||||
socket.send_to(b"rpf", address).unwrap();
|
||||
let mut packet = [0u8; 3];
|
||||
assert_eq!(
|
||||
socket.recv(&mut packet).is_ok(),
|
||||
should_pass,
|
||||
"UDP {address}"
|
||||
);
|
||||
if should_pass {
|
||||
assert_eq!(&packet, b"rpf");
|
||||
}
|
||||
let connected = TcpStream::connect_timeout(&address, Duration::from_secs(1));
|
||||
assert_eq!(connected.is_ok(), should_pass, "TCP {address}");
|
||||
if let Ok(mut stream) = connected {
|
||||
stream
|
||||
.set_read_timeout(Some(Duration::from_secs(1)))
|
||||
.unwrap();
|
||||
stream.write_all(b"rpf").unwrap();
|
||||
stream.read_exact(&mut packet).unwrap();
|
||||
assert_eq!(&packet, b"rpf");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn run_probes(expect: &str) {
|
||||
let result = Command::new(std::env::current_exe().unwrap())
|
||||
.args(["--ignored", "--exact", TEST, "--nocapture"])
|
||||
.env(ROLE, "probe")
|
||||
.env("TSUNAGI_EXIT_KERNEL_EXPECT", expect)
|
||||
.uid(990)
|
||||
.gid(990)
|
||||
.output()
|
||||
.unwrap();
|
||||
assert!(
|
||||
result.status.success(),
|
||||
"{}\n{}",
|
||||
String::from_utf8_lossy(&result.stdout),
|
||||
String::from_utf8_lossy(&result.stderr)
|
||||
);
|
||||
}
|
||||
|
||||
fn passive_listener() {
|
||||
let listeners: Vec<_> = ["192.0.2.1:42010", "[2001:db8:1::1]:42010"]
|
||||
.into_iter()
|
||||
.map(|address| TcpListener::bind(address).unwrap())
|
||||
.collect();
|
||||
println!("PASSIVE_READY");
|
||||
let tasks: Vec<_> = listeners
|
||||
.into_iter()
|
||||
.map(|listener| {
|
||||
std::thread::spawn(move || {
|
||||
let (mut socket, _) = listener.accept().unwrap();
|
||||
socket
|
||||
.set_read_timeout(Some(Duration::from_secs(2)))
|
||||
.unwrap();
|
||||
let mut bytes = [0; 3];
|
||||
socket.read_exact(&mut bytes).unwrap();
|
||||
socket.write_all(&bytes).unwrap();
|
||||
})
|
||||
})
|
||||
.collect();
|
||||
for task in tasks {
|
||||
task.join().unwrap();
|
||||
}
|
||||
}
|
||||
|
||||
fn udp_listener() {
|
||||
let sockets: Vec<_> = ["192.0.2.1:42011", "[2001:db8:1::1]:42011"]
|
||||
.into_iter()
|
||||
.map(|a| UdpSocket::bind(a).unwrap())
|
||||
.collect();
|
||||
println!("PASSIVE_READY");
|
||||
let tasks: Vec<_> = sockets
|
||||
.into_iter()
|
||||
.map(|socket| {
|
||||
std::thread::spawn(move || {
|
||||
socket
|
||||
.set_read_timeout(Some(Duration::from_secs(5)))
|
||||
.unwrap();
|
||||
let mut bytes = [0; 3];
|
||||
let (len, peer) = socket.recv_from(&mut bytes).unwrap();
|
||||
socket.send_to(&bytes[..len], peer).unwrap();
|
||||
})
|
||||
})
|
||||
.collect();
|
||||
for task in tasks {
|
||||
task.join().unwrap();
|
||||
}
|
||||
}
|
||||
|
||||
fn spawn_listener(role: &str) -> Echo {
|
||||
let process = Command::new(std::env::current_exe().unwrap())
|
||||
.args(["--ignored", "--exact", TEST, "--nocapture"])
|
||||
.env(ROLE, role)
|
||||
.uid(990)
|
||||
.gid(990)
|
||||
.stdout(std::process::Stdio::piped())
|
||||
.spawn()
|
||||
.unwrap();
|
||||
let mut child = Echo(process);
|
||||
let mut output = BufReader::new(child.0.stdout.take().unwrap());
|
||||
loop {
|
||||
let mut line = String::new();
|
||||
assert_ne!(
|
||||
output.read_line(&mut line).unwrap(),
|
||||
0,
|
||||
"listener exited before readiness"
|
||||
);
|
||||
if line.contains("PASSIVE_READY") {
|
||||
// Keep the reader open for libtest's final status after the socket
|
||||
// worker finishes; closing it here makes success a broken pipe.
|
||||
child.0.stdout = Some(output.into_inner());
|
||||
return child;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn client() {
|
||||
for setting in ["rp_filter", "src_valid_mark"] {
|
||||
std::fs::write(format!("/proc/sys/net/ipv4/conf/all/{setting}"), b"1").unwrap();
|
||||
}
|
||||
std::fs::write("/proc/sys/net/ipv4/conf/rpf-c/rp_filter", b"1").unwrap();
|
||||
for family in [false, true] {
|
||||
let args: Vec<_> = [
|
||||
"-t",
|
||||
"mangle",
|
||||
"-A",
|
||||
"PREROUTING",
|
||||
"-m",
|
||||
"rpfilter",
|
||||
"--invert",
|
||||
"--validmark",
|
||||
"-j",
|
||||
"DROP",
|
||||
]
|
||||
.into_iter()
|
||||
.map(str::to_string)
|
||||
.collect();
|
||||
let result = run_firewall(family, &args);
|
||||
assert!(
|
||||
matches!(result, super::linux::Ran::Exited { success: true, .. }),
|
||||
"{result:?}"
|
||||
);
|
||||
}
|
||||
let provisioner = NetlinkProvisioner::new().unwrap();
|
||||
let route = provisioner.host_rules().route;
|
||||
// Model Tailscale's earlier lookup, without running another VPN daemon.
|
||||
ip(&["-4", "rule", "add", "priority", "5270", "table", "52"]);
|
||||
ip(&[
|
||||
"-4", "route", "add", "default", "dev", "tsun0", "table", "52",
|
||||
]);
|
||||
ip(&["-6", "rule", "add", "priority", "5270", "table", "52"]);
|
||||
ip(&[
|
||||
"-6",
|
||||
"route",
|
||||
"add",
|
||||
"unreachable",
|
||||
"default",
|
||||
"table",
|
||||
"52",
|
||||
]);
|
||||
// A foreign VPN must leave neighbour discovery and connected LAN paths
|
||||
// usable; otherwise the fixture itself cannot resolve its uplink neighbour.
|
||||
for prefix in ["fe80::/64", "2001:db8:1::/64"] {
|
||||
ip(&["-6", "route", "add", "throw", prefix, "table", "52"]);
|
||||
}
|
||||
let mut passive = spawn_listener("listener");
|
||||
route.set_exit_client("tsun0", 990).unwrap();
|
||||
// uid bypass alone cannot protect reverse lookups, reproducing the bug.
|
||||
run_probes("blocked");
|
||||
install(&run_firewall, "tsun0", 990, true).unwrap();
|
||||
run_probes("pass");
|
||||
// Pair transports may bind fresh UDP ports after exit activation.
|
||||
let mut dynamic_udp = spawn_listener("udp-listener");
|
||||
refresh(
|
||||
&run_firewall,
|
||||
"tsun0",
|
||||
true,
|
||||
&local_sockets(990, true).unwrap(),
|
||||
)
|
||||
.unwrap();
|
||||
let server = std::env::var("TSUNAGI_EXIT_KERNEL_SERVER").unwrap();
|
||||
let incoming = Command::new("ip").args(["netns", "exec", &server, "python3", "-c", r#"import socket
|
||||
for family,source,target in [(socket.AF_INET,'198.51.100.2','192.0.2.1'),(socket.AF_INET6,'2001:db8:2::2','2001:db8:1::1')]:
|
||||
for kind,port in [(socket.SOCK_STREAM,42010),(socket.SOCK_DGRAM,42011)]:
|
||||
s=socket.socket(family,kind); s.settimeout(2); s.bind((source,0)); s.connect((target,port)); s.sendall(b'rpf'); assert s.recv(3)==b'rpf'; s.close()
|
||||
"#]).output().unwrap();
|
||||
assert!(
|
||||
incoming.status.success(),
|
||||
"first inbound TCP/UDP must be protected: {}",
|
||||
String::from_utf8_lossy(&incoming.stderr)
|
||||
);
|
||||
assert!(passive.0.wait().unwrap().success());
|
||||
assert!(dynamic_udp.0.wait().unwrap().success());
|
||||
// Repeated apply remains usable, and does not redirect application sockets.
|
||||
install(&run_firewall, "tsun0", 990, true).unwrap();
|
||||
run_probes("pass");
|
||||
assert!(ip(&["-4", "route", "get", "198.51.100.2", "uid", "991"]).contains("dev tsun0"));
|
||||
let blocked = Command::new("ip")
|
||||
.args(["-6", "route", "get", "2001:db8:2::2", "uid", "991"])
|
||||
.output()
|
||||
.unwrap();
|
||||
assert!(
|
||||
!blocked.status.success(),
|
||||
"application IPv6 must stay blocked"
|
||||
);
|
||||
route.clear_exit_client().unwrap();
|
||||
clear(&run_firewall, "tsun0", true).unwrap();
|
||||
for family in ["-4", "-6"] {
|
||||
assert!(
|
||||
ip(&[family, "rule", "show"]).contains("5270:"),
|
||||
"foreign routing rules survive cleanup"
|
||||
);
|
||||
ip(&[family, "rule", "del", "priority", "5270"]);
|
||||
}
|
||||
assert!(ip(&["-4", "route", "get", "198.51.100.2", "uid", "991"]).contains("dev rpf-c"));
|
||||
assert!(ip(&["-6", "route", "get", "2001:db8:2::2", "uid", "991"]).contains("dev rpf-c"));
|
||||
for family in [false, true] {
|
||||
let result = run_firewall(family, &["-t".into(), "mangle".into(), "-S".into()]);
|
||||
if let super::linux::Ran::Exited { stdout, .. } = result {
|
||||
assert!(!stdout.contains("tsunagi-exit-underlay:"));
|
||||
assert!(!stdout.contains("tsunagi-exit-sockets:"));
|
||||
assert!(
|
||||
stdout.contains("rpfilter"),
|
||||
"strict host filtering is retained"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[ignore = "needs root, iproute2, iptables/ip6tables and Python; changes only temporary namespaces"]
|
||||
fn strict_firewall_preserves_underlay_and_application_exit_routing() {
|
||||
match std::env::var(ROLE).as_deref() {
|
||||
Ok("probe") => {
|
||||
probes();
|
||||
return;
|
||||
}
|
||||
Ok("client") => {
|
||||
client();
|
||||
return;
|
||||
}
|
||||
Ok("listener") => {
|
||||
passive_listener();
|
||||
return;
|
||||
}
|
||||
Ok("udp-listener") => {
|
||||
udp_listener();
|
||||
return;
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
let suffix = std::process::id();
|
||||
let client_name = format!("tsng-rpf-c-{suffix}");
|
||||
let server_name = format!("tsng-rpf-s-{suffix}");
|
||||
ip(&["netns", "add", &client_name]);
|
||||
let _client = Namespace(client_name.clone());
|
||||
ip(&["netns", "add", &server_name]);
|
||||
let _server = Namespace(server_name.clone());
|
||||
ip(&[
|
||||
"link",
|
||||
"add",
|
||||
"rpf-c",
|
||||
"netns",
|
||||
&client_name,
|
||||
"type",
|
||||
"veth",
|
||||
"peer",
|
||||
"name",
|
||||
"rpf-s",
|
||||
"netns",
|
||||
&server_name,
|
||||
]);
|
||||
for (namespace, interface, v4, v6) in [
|
||||
(&client_name, "rpf-c", "192.0.2.1/24", "2001:db8:1::1/64"),
|
||||
(&server_name, "rpf-s", "192.0.2.2/24", "2001:db8:1::2/64"),
|
||||
] {
|
||||
ip(&["-n", namespace, "link", "set", "lo", "up"]);
|
||||
ip(&["-n", namespace, "addr", "add", v4, "dev", interface]);
|
||||
ip(&[
|
||||
"-n", namespace, "-6", "addr", "add", v6, "dev", interface, "nodad",
|
||||
]);
|
||||
ip(&["-n", namespace, "link", "set", interface, "up"]);
|
||||
}
|
||||
ip(&[
|
||||
"-n",
|
||||
&server_name,
|
||||
"addr",
|
||||
"add",
|
||||
"198.51.100.2/32",
|
||||
"dev",
|
||||
"lo",
|
||||
]);
|
||||
ip(&[
|
||||
"-n",
|
||||
&server_name,
|
||||
"-6",
|
||||
"addr",
|
||||
"add",
|
||||
"2001:db8:2::2/128",
|
||||
"dev",
|
||||
"lo",
|
||||
"nodad",
|
||||
]);
|
||||
ip(&[
|
||||
"-n",
|
||||
&client_name,
|
||||
"route",
|
||||
"add",
|
||||
"default",
|
||||
"via",
|
||||
"192.0.2.2",
|
||||
]);
|
||||
ip(&[
|
||||
"-n",
|
||||
&client_name,
|
||||
"-6",
|
||||
"route",
|
||||
"add",
|
||||
"default",
|
||||
"via",
|
||||
"2001:db8:1::2",
|
||||
]);
|
||||
ip(&["-n", &client_name, "link", "add", "tsun0", "type", "dummy"]);
|
||||
ip(&[
|
||||
"-n",
|
||||
&client_name,
|
||||
"addr",
|
||||
"add",
|
||||
"10.13.37.1/24",
|
||||
"dev",
|
||||
"tsun0",
|
||||
]);
|
||||
ip(&["-n", &client_name, "link", "set", "tsun0", "up"]);
|
||||
let echo_code = r#"import socket,selectors,sys
|
||||
selector=selectors.DefaultSelector()
|
||||
for family in [socket.AF_INET,socket.AF_INET6]:
|
||||
for kind in [socket.SOCK_DGRAM,socket.SOCK_STREAM]:
|
||||
sock=socket.socket(family,kind)
|
||||
if family==socket.AF_INET6: sock.setsockopt(socket.IPPROTO_IPV6,socket.IPV6_V6ONLY,1)
|
||||
sock.bind(('198.51.100.2' if family==socket.AF_INET else '2001:db8:2::2',42000))
|
||||
if kind==socket.SOCK_STREAM: sock.listen()
|
||||
selector.register(sock,selectors.EVENT_READ,kind)
|
||||
print('ready',flush=True)
|
||||
while True:
|
||||
for key,_ in selector.select():
|
||||
if key.data==socket.SOCK_DGRAM:
|
||||
data,address=key.fileobj.recvfrom(128); key.fileobj.sendto(data,address)
|
||||
else:
|
||||
conn,_=key.fileobj.accept(); conn.settimeout(2)
|
||||
try: conn.sendall(conn.recv(128))
|
||||
finally: conn.close()
|
||||
"#;
|
||||
let child = Command::new("ip")
|
||||
.args([
|
||||
"netns",
|
||||
"exec",
|
||||
&server_name,
|
||||
"python3",
|
||||
"-u",
|
||||
"-c",
|
||||
echo_code,
|
||||
])
|
||||
.stdout(std::process::Stdio::piped())
|
||||
.spawn()
|
||||
.unwrap();
|
||||
let mut echo = Echo(child);
|
||||
let mut ready = String::new();
|
||||
BufReader::new(echo.0.stdout.take().unwrap())
|
||||
.read_line(&mut ready)
|
||||
.unwrap();
|
||||
assert_eq!(ready.trim(), "ready");
|
||||
let result = Command::new("ip")
|
||||
.args(["netns", "exec", &client_name])
|
||||
.arg(std::env::current_exe().unwrap())
|
||||
.args(["--ignored", "--exact", TEST, "--nocapture"])
|
||||
.env(ROLE, "client")
|
||||
.env("TSUNAGI_EXIT_KERNEL_SERVER", &server_name)
|
||||
.output()
|
||||
.unwrap();
|
||||
assert!(
|
||||
result.status.success(),
|
||||
"{}\n{}",
|
||||
String::from_utf8_lossy(&result.stdout),
|
||||
String::from_utf8_lossy(&result.stderr)
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,535 @@
|
||||
//! Own-bit connection marks for exit-client underlay traffic.
|
||||
//! Reply marks are restored before host reverse-path filtering, never accepted
|
||||
//! around it. No peer address becomes a host route or firewall argument.
|
||||
|
||||
use super::linux::{Ran, failure, locked, split_rule_line};
|
||||
use std::net::{IpAddr, Ipv4Addr, Ipv6Addr};
|
||||
|
||||
/// Outside Tailscale's 0xff0000 mask and ordinary low-bit WireGuard marks.
|
||||
pub(crate) const UNDERLAY_MARK: u32 = 0x4000_0000;
|
||||
pub(crate) const UNDERLAY_MASK: u32 = UNDERLAY_MARK;
|
||||
const MARK: &str = "0x40000000/0x40000000";
|
||||
const MASK: &str = "0x40000000";
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord)]
|
||||
pub(super) struct Listener {
|
||||
address: IpAddr,
|
||||
port: u16,
|
||||
tcp: bool,
|
||||
}
|
||||
|
||||
fn parse_listeners(listing: &str, uid: u32, v6: bool, tcp: bool) -> Result<Vec<Listener>, String> {
|
||||
let mut result = Vec::new();
|
||||
for line in listing.lines().skip(1) {
|
||||
let fields: Vec<_> = line.split_whitespace().collect();
|
||||
if (tcp && fields.get(3) != Some(&"0A"))
|
||||
|| fields.get(7).and_then(|s| s.parse::<u32>().ok()) != Some(uid)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
let (address, port) = fields
|
||||
.get(1)
|
||||
.and_then(|s| s.split_once(':'))
|
||||
.ok_or("malformed local socket")?;
|
||||
let port = u16::from_str_radix(port, 16).map_err(|e| e.to_string())?;
|
||||
let address = if v6 {
|
||||
let mut bytes = [0; 16];
|
||||
for (word, chunk) in bytes.as_chunks_mut::<4>().0.iter_mut().enumerate() {
|
||||
let hex = address
|
||||
.get(word * 8..word * 8 + 8)
|
||||
.ok_or("malformed local IPv6 listener")?;
|
||||
chunk.copy_from_slice(
|
||||
&u32::from_str_radix(hex, 16)
|
||||
.map_err(|e| e.to_string())?
|
||||
.to_ne_bytes(),
|
||||
);
|
||||
}
|
||||
IpAddr::V6(Ipv6Addr::from(bytes))
|
||||
} else {
|
||||
IpAddr::V4(Ipv4Addr::from(
|
||||
u32::from_str_radix(address, 16)
|
||||
.map_err(|e| e.to_string())?
|
||||
.to_ne_bytes(),
|
||||
))
|
||||
};
|
||||
if result.len() == 128 {
|
||||
return Err("too many local sockets for exit protection".into());
|
||||
}
|
||||
if port != 0 {
|
||||
result.push(Listener { address, port, tcp });
|
||||
}
|
||||
}
|
||||
Ok(result)
|
||||
}
|
||||
|
||||
pub(super) fn local_sockets(uid: u32, v6: bool) -> Result<Vec<Listener>, String> {
|
||||
let mut listeners = Vec::new();
|
||||
for (protocol, tcp) in [("tcp", true), ("udp", false)] {
|
||||
for family in [false, true].into_iter().filter(|f| !f || v6) {
|
||||
let path = format!("/proc/net/{protocol}{}", if family { "6" } else { "" });
|
||||
listeners.extend(parse_listeners(
|
||||
&std::fs::read_to_string(path).map_err(|e| e.to_string())?,
|
||||
uid,
|
||||
family,
|
||||
tcp,
|
||||
)?);
|
||||
}
|
||||
}
|
||||
listeners.sort();
|
||||
listeners.dedup();
|
||||
Ok(listeners)
|
||||
}
|
||||
|
||||
fn listener_rules(interface: &str, listener: &Listener) -> Vec<Vec<String>> {
|
||||
let mut base = words(&[
|
||||
"!",
|
||||
"-i",
|
||||
interface,
|
||||
"-p",
|
||||
if listener.tcp { "tcp" } else { "udp" },
|
||||
"--dport",
|
||||
&listener.port.to_string(),
|
||||
]);
|
||||
if !listener.address.is_unspecified() {
|
||||
base.extend(words(&["-d", &listener.address.to_string()]));
|
||||
}
|
||||
// Match an actual local socket, not just a destination port. Listening
|
||||
// sockets need protection on the first packet, before any outbound connmark.
|
||||
base.extend(words(&[
|
||||
"-m",
|
||||
"socket",
|
||||
"--nowildcard",
|
||||
"-m",
|
||||
"comment",
|
||||
"--comment",
|
||||
&listener_tag(interface),
|
||||
]));
|
||||
let save = base
|
||||
.iter()
|
||||
.cloned()
|
||||
.chain(words(&[
|
||||
"-m",
|
||||
"mark",
|
||||
"--mark",
|
||||
MARK,
|
||||
"-j",
|
||||
"CONNMARK",
|
||||
"--save-mark",
|
||||
"--nfmask",
|
||||
MASK,
|
||||
"--ctmask",
|
||||
MASK,
|
||||
]))
|
||||
.collect();
|
||||
let mark = base
|
||||
.into_iter()
|
||||
.chain(words(&["-j", "MARK", "--set-xmark", MARK]))
|
||||
.collect();
|
||||
vec![save, mark]
|
||||
}
|
||||
pub(super) type RunFamily<'a> = &'a dyn Fn(bool, &[String]) -> Ran;
|
||||
|
||||
fn tag(interface: &str) -> String {
|
||||
format!("tsunagi-exit-underlay:{interface}")
|
||||
}
|
||||
|
||||
fn listener_tag(interface: &str) -> String {
|
||||
format!("tsunagi-exit-sockets:{interface}")
|
||||
}
|
||||
|
||||
fn words(values: &[&str]) -> Vec<String> {
|
||||
values.iter().map(|s| (*s).into()).collect()
|
||||
}
|
||||
|
||||
fn rules(interface: &str, uid: u32) -> Vec<(&'static str, Vec<String>)> {
|
||||
let tag = tag(interface);
|
||||
let uid = uid.to_string();
|
||||
// Insert at position one in this order: final OUTPUT order is MARK, SAVE.
|
||||
vec![
|
||||
(
|
||||
"PREROUTING",
|
||||
words(&[
|
||||
"!",
|
||||
"-i",
|
||||
interface,
|
||||
"-m",
|
||||
"connmark",
|
||||
"--mark",
|
||||
MARK,
|
||||
"-m",
|
||||
"comment",
|
||||
"--comment",
|
||||
&tag,
|
||||
"-j",
|
||||
"CONNMARK",
|
||||
"--restore-mark",
|
||||
"--nfmask",
|
||||
MASK,
|
||||
"--ctmask",
|
||||
MASK,
|
||||
]),
|
||||
),
|
||||
(
|
||||
"OUTPUT",
|
||||
words(&[
|
||||
"-m",
|
||||
"owner",
|
||||
"--uid-owner",
|
||||
&uid,
|
||||
"-m",
|
||||
"mark",
|
||||
"--mark",
|
||||
MARK,
|
||||
"-m",
|
||||
"comment",
|
||||
"--comment",
|
||||
&tag,
|
||||
"-j",
|
||||
"CONNMARK",
|
||||
"--save-mark",
|
||||
"--nfmask",
|
||||
MASK,
|
||||
"--ctmask",
|
||||
MASK,
|
||||
]),
|
||||
),
|
||||
(
|
||||
"OUTPUT",
|
||||
words(&[
|
||||
"-m",
|
||||
"owner",
|
||||
"--uid-owner",
|
||||
&uid,
|
||||
"-m",
|
||||
"comment",
|
||||
"--comment",
|
||||
&tag,
|
||||
"-j",
|
||||
"MARK",
|
||||
"--set-xmark",
|
||||
MARK,
|
||||
]),
|
||||
),
|
||||
]
|
||||
}
|
||||
|
||||
fn execute(run: RunFamily<'_>, v6: bool, args: Vec<String>) -> Result<String, String> {
|
||||
match run(v6, &locked(args)) {
|
||||
Ran::Exited {
|
||||
success: true,
|
||||
stdout,
|
||||
..
|
||||
} => Ok(stdout),
|
||||
Ran::Exited { stderr, .. } => Err(failure(&stderr)),
|
||||
Ran::Missing => Err(format!(
|
||||
"{} is required for exit-node connection marks",
|
||||
if v6 { "ip6tables" } else { "iptables" }
|
||||
)),
|
||||
}
|
||||
}
|
||||
|
||||
/// Removes only exact interface tags, retaining other agents and host rules.
|
||||
pub(super) fn clear(run: RunFamily<'_>, interface: &str, v6: bool) -> Result<(), String> {
|
||||
clear_tags(run, interface, v6, false)
|
||||
}
|
||||
|
||||
fn clear_tags(
|
||||
run: RunFamily<'_>,
|
||||
interface: &str,
|
||||
v6: bool,
|
||||
sockets_only: bool,
|
||||
) -> Result<(), String> {
|
||||
let wanted = tag(interface);
|
||||
let sockets = listener_tag(interface);
|
||||
for family in [false, true].into_iter().filter(|family| !family || v6) {
|
||||
let chains: &[&str] = if sockets_only {
|
||||
&["PREROUTING"]
|
||||
} else {
|
||||
&["OUTPUT", "PREROUTING"]
|
||||
};
|
||||
for chain in chains {
|
||||
let listing = execute(run, family, words(&["-t", "mangle", "-S", chain]))?;
|
||||
for line in listing.lines() {
|
||||
let tokens = split_rule_line(line);
|
||||
let [append, listed_chain, rest @ ..] = tokens.as_slice() else {
|
||||
continue;
|
||||
};
|
||||
if append == "-A"
|
||||
&& listed_chain == *chain
|
||||
&& rest.windows(2).any(|pair| {
|
||||
pair[0] == "--comment"
|
||||
&& (pair[1] == sockets || (!sockets_only && pair[1] == wanted))
|
||||
})
|
||||
{
|
||||
let args = words(&["-t", "mangle", "-D", chain])
|
||||
.into_iter()
|
||||
.chain(rest.iter().cloned())
|
||||
.collect();
|
||||
execute(run, family, args)?;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Establishes protection before routing may redirect application traffic.
|
||||
pub(super) fn install(
|
||||
run: RunFamily<'_>,
|
||||
interface: &str,
|
||||
uid: u32,
|
||||
v6: bool,
|
||||
) -> Result<Vec<Listener>, String> {
|
||||
let listeners = local_sockets(uid, v6)?;
|
||||
install_with_listeners(run, interface, uid, v6, &listeners)?;
|
||||
Ok(listeners)
|
||||
}
|
||||
|
||||
fn install_with_listeners(
|
||||
run: RunFamily<'_>,
|
||||
interface: &str,
|
||||
uid: u32,
|
||||
v6: bool,
|
||||
listeners: &[Listener],
|
||||
) -> Result<(), String> {
|
||||
clear(run, interface, v6)?;
|
||||
let result = (|| {
|
||||
for family in [false, true].into_iter().filter(|family| !family || v6) {
|
||||
for (chain, spec) in rules(interface, uid) {
|
||||
let args = words(&["-t", "mangle", "-I", chain, "1"])
|
||||
.into_iter()
|
||||
.chain(spec)
|
||||
.collect();
|
||||
execute(run, family, args)?;
|
||||
}
|
||||
for listener in listeners.iter().filter(|s| s.address.is_ipv6() == family) {
|
||||
for spec in listener_rules(interface, listener) {
|
||||
execute(
|
||||
run,
|
||||
family,
|
||||
words(&["-t", "mangle", "-I", "PREROUTING", "1"])
|
||||
.into_iter()
|
||||
.chain(spec)
|
||||
.collect(),
|
||||
)?;
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
})();
|
||||
if result.is_err()
|
||||
&& let Err(cleanup) = clear(run, interface, v6)
|
||||
{
|
||||
return Err(format!(
|
||||
"{}; cannot roll back connection marks: {cleanup}",
|
||||
result.err().unwrap_or_default()
|
||||
));
|
||||
}
|
||||
result
|
||||
}
|
||||
|
||||
/// Updates passive protection without interrupting established connmark rules.
|
||||
pub(super) fn refresh(
|
||||
run: RunFamily<'_>,
|
||||
interface: &str,
|
||||
v6: bool,
|
||||
listeners: &[Listener],
|
||||
) -> Result<(), String> {
|
||||
clear_tags(run, interface, v6, true)?;
|
||||
for family in [false, true].into_iter().filter(|f| !f || v6) {
|
||||
for listener in listeners.iter().filter(|s| s.address.is_ipv6() == family) {
|
||||
for spec in listener_rules(interface, listener) {
|
||||
execute(
|
||||
run,
|
||||
family,
|
||||
words(&["-t", "mangle", "-I", "PREROUTING", "1"])
|
||||
.into_iter()
|
||||
.chain(spec)
|
||||
.collect(),
|
||||
)?;
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Kernel strict RPF must include the restored mark, independently of iptables.
|
||||
pub(super) fn check_kernel_rpf(
|
||||
all_rpf: u32,
|
||||
all_valid: u32,
|
||||
interfaces: &[(String, u32, u32)],
|
||||
) -> Result<(), String> {
|
||||
for (interface, rpf, valid) in interfaces {
|
||||
if all_rpf.max(*rpf) == 1 && all_valid.max(*valid) == 0 {
|
||||
return Err(format!(
|
||||
"strict rp_filter on `{interface}` ignores connection marks; enable net.ipv4.conf.all.src_valid_mark=1 before using an exit node (rp_filter stays enabled)"
|
||||
));
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
|
||||
use super::*;
|
||||
use std::cell::RefCell;
|
||||
|
||||
#[test]
|
||||
fn both_families_restore_before_firewall_and_preserve_foreign_mark_bits() {
|
||||
let calls = RefCell::new(Vec::new());
|
||||
let run = |v6, args: &[String]| {
|
||||
calls.borrow_mut().push((v6, args.to_vec()));
|
||||
Ran::Exited {
|
||||
success: true,
|
||||
stdout: String::new(),
|
||||
stderr: String::new(),
|
||||
}
|
||||
};
|
||||
install_with_listeners(&run, "tsun0", 990, true, &[]).unwrap();
|
||||
let calls = calls.borrow();
|
||||
for family in [false, true] {
|
||||
let inserts: Vec<_> = calls
|
||||
.iter()
|
||||
.filter(|(v6, a)| *v6 == family && a.iter().any(|s| s == "-I"))
|
||||
.map(|(_, a)| a.join(" "))
|
||||
.collect();
|
||||
assert_eq!(inserts.len(), 3);
|
||||
assert!(inserts[0].contains("-I PREROUTING 1 ! -i tsun0 -m connmark"));
|
||||
assert!(inserts[0].contains("--restore-mark --nfmask 0x40000000 --ctmask 0x40000000"));
|
||||
assert!(inserts[1].contains("--uid-owner 990"));
|
||||
assert!(inserts[1].contains("--save-mark"));
|
||||
assert!(inserts[2].contains("--set-xmark 0x40000000/0x40000000"));
|
||||
assert!(
|
||||
!inserts
|
||||
.iter()
|
||||
.any(|s| s.contains("-j ACCEPT") || s.contains("-j RETURN"))
|
||||
);
|
||||
}
|
||||
assert_eq!(UNDERLAY_MASK & 0x00ff_0000, 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cleanup_retains_other_interfaces_and_rolls_back_partial_ipv6_installation() {
|
||||
let calls = RefCell::new(Vec::new());
|
||||
let run = |v6, args: &[String]| {
|
||||
calls.borrow_mut().push((v6, args.to_vec()));
|
||||
let listing = "-A OUTPUT -m comment --comment tsunagi-exit-underlay:tsun0 -j MARK --set-xmark 0x40000000/0x40000000\n-A OUTPUT -m comment --comment tsunagi-exit-underlay:tsun1 -j MARK --set-xmark 0x40000000/0x40000000\n-A OUTPUT -m comment --comment unrelated -j RETURN\n";
|
||||
Ran::Exited {
|
||||
success: !(v6 && args.iter().any(|s| s == "-I")),
|
||||
stdout: if args.iter().any(|s| s == "-S") {
|
||||
listing.into()
|
||||
} else {
|
||||
String::new()
|
||||
},
|
||||
stderr: "injected IPv6 failure".into(),
|
||||
}
|
||||
};
|
||||
assert!(install_with_listeners(&run, "tsun0", 990, true, &[]).is_err());
|
||||
let calls = calls.borrow();
|
||||
let deletions: Vec<_> = calls
|
||||
.iter()
|
||||
.filter(|(_, a)| a.iter().any(|s| s == "-D"))
|
||||
.collect();
|
||||
assert!(
|
||||
deletions.len() >= 4,
|
||||
"both initial and rollback cleanup run"
|
||||
);
|
||||
assert!(
|
||||
deletions
|
||||
.iter()
|
||||
.all(|(_, a)| a.iter().any(|s| s == "tsunagi-exit-underlay:tsun0"))
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn kernel_strict_validation_requires_mark_awareness_without_weakening_rpf() {
|
||||
let interfaces = vec![("eth0".into(), 1, 0)];
|
||||
assert!(check_kernel_rpf(0, 0, &interfaces).is_err());
|
||||
assert!(check_kernel_rpf(0, 1, &interfaces).is_ok());
|
||||
assert!(check_kernel_rpf(2, 0, &interfaces).is_ok());
|
||||
assert!(check_kernel_rpf(0, 0, &[("eth0".into(), 1, 1)]).is_ok());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn passive_tcp_protection_uses_only_locally_owned_listeners() {
|
||||
let listing = "header\n0: 00000000:01BB 00000000:0000 0A 0:0 0:0 0 990 0 1\n1: 0100007F:1111 00000000:0000 0A 0:0 0:0 0 1000 0 2\n2: 0100007F:2222 00000000:0000 01 0:0 0:0 0 990 0 3\n";
|
||||
let listeners = parse_listeners(listing, 990, false, true).unwrap();
|
||||
assert_eq!(
|
||||
listeners,
|
||||
vec![Listener {
|
||||
address: IpAddr::V4(Ipv4Addr::UNSPECIFIED),
|
||||
port: 443,
|
||||
tcp: true
|
||||
}]
|
||||
);
|
||||
let rules = listener_rules("tsun0", &listeners[0]);
|
||||
assert!(
|
||||
rules[0]
|
||||
.join(" ")
|
||||
.contains("-p tcp --dport 443 -m socket --nowildcard")
|
||||
);
|
||||
assert!(rules[0].join(" ").contains("--save-mark"));
|
||||
assert!(rules[1].join(" ").contains("--set-xmark"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn listener_parser_checks_family_encoding_and_bounds() {
|
||||
let loopback = Ipv6Addr::LOCALHOST.octets();
|
||||
let hex: String = loopback
|
||||
.as_chunks::<4>()
|
||||
.0
|
||||
.iter()
|
||||
.map(|bytes| format!("{:08X}", u32::from_ne_bytes(*bytes)))
|
||||
.collect();
|
||||
let listing = format!("header\n0: {hex}:01BB 0:0 0A 0:0 0:0 0 990 0 1\n");
|
||||
assert_eq!(
|
||||
parse_listeners(&listing, 990, true, true).unwrap(),
|
||||
vec![Listener {
|
||||
address: IpAddr::V6(Ipv6Addr::LOCALHOST),
|
||||
port: 443,
|
||||
tcp: true
|
||||
}]
|
||||
);
|
||||
assert!(
|
||||
parse_listeners(
|
||||
"header\n0: BAD:01BB 0:0 0A 0:0 0:0 0 990 0 1",
|
||||
990,
|
||||
true,
|
||||
true
|
||||
)
|
||||
.is_err()
|
||||
);
|
||||
let huge = format!(
|
||||
"header\n{}",
|
||||
"0: 00000000:01BB 0:0 0A 0:0 0:0 0 990 0 1\n".repeat(129)
|
||||
);
|
||||
assert!(parse_listeners(&huge, 990, false, true).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn udp_refresh_preserves_general_connmark_protection_and_other_interfaces() {
|
||||
let listing = "header\n0: 00000000:ABA0 0:0 07 0:0 0:0 0 990 0 1\n1: 00000000:0000 0:0 07 0:0 0:0 0 990 0 2\n";
|
||||
let sockets = parse_listeners(listing, 990, false, false).unwrap();
|
||||
assert_eq!(sockets.len(), 1);
|
||||
assert!(!sockets[0].tcp);
|
||||
let calls = RefCell::new(Vec::new());
|
||||
let run = |v6, args: &[String]| {
|
||||
calls.borrow_mut().push((v6, args.join(" ")));
|
||||
Ran::Exited { success: true, stdout: "-A PREROUTING -m comment --comment tsunagi-exit-underlay:tsun0 -j CONNMARK --restore-mark\n-A PREROUTING -m comment --comment tsunagi-exit-sockets:tsun0 -j MARK\n-A PREROUTING -m comment --comment tsunagi-exit-sockets:tsun1 -j MARK\n".into(), stderr: String::new() }
|
||||
};
|
||||
refresh(&run, "tsun0", false, &sockets).unwrap();
|
||||
let calls = calls.borrow();
|
||||
assert!(
|
||||
calls
|
||||
.iter()
|
||||
.any(|(_, a)| a.contains("-I PREROUTING 1") && a.contains("-p udp"))
|
||||
);
|
||||
assert!(
|
||||
calls
|
||||
.iter()
|
||||
.filter(|(_, a)| a.contains("-D"))
|
||||
.all(|(_, a)| a.contains("tsunagi-exit-sockets:tsun0"))
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -55,6 +55,10 @@ pub use exit::{
|
||||
mod linux;
|
||||
#[cfg(all(feature = "tun-device", target_os = "linux"))]
|
||||
mod linux_exit;
|
||||
#[cfg(all(test, feature = "tun-device", target_os = "linux"))]
|
||||
mod linux_exit_kernel;
|
||||
#[cfg(all(feature = "tun-device", target_os = "linux"))]
|
||||
pub(in crate::overlay) mod linux_exit_mark;
|
||||
#[cfg(all(feature = "tun-device", target_os = "linux"))]
|
||||
pub use linux::LinuxHostRules;
|
||||
|
||||
|
||||
@@ -145,18 +145,19 @@ impl Tally {
|
||||
}
|
||||
|
||||
/// The overlay interface.
|
||||
#[derive(Debug)]
|
||||
pub struct Interface {
|
||||
device: Arc<dyn TunDevice>,
|
||||
factory: Arc<dyn TunFactory>,
|
||||
name: String,
|
||||
mtu: u32,
|
||||
routes: Arc<RoutingTable>,
|
||||
carrier: Arc<dyn PacketCarrier>,
|
||||
tally: Arc<Tally>,
|
||||
/// What was last applied to the host, so an unchanged table is not
|
||||
/// re-applied on every pass.
|
||||
applied: std::sync::Mutex<Vec<Cidr>>,
|
||||
task: std::sync::Mutex<Option<JoinHandle<()>>>,
|
||||
vip_task: std::sync::Mutex<Option<JoinHandle<()>>>,
|
||||
sync_lock: Arc<tokio::sync::Mutex<()>>,
|
||||
/// The host rules for this interface, when the factory manages the host.
|
||||
host_rules: Option<Arc<dyn BroadcastHostRules>>,
|
||||
@@ -200,6 +201,14 @@ impl ExitState {
|
||||
}
|
||||
}
|
||||
|
||||
impl std::fmt::Debug for Interface {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
f.debug_struct("Interface")
|
||||
.field("name", &self.name)
|
||||
.finish_non_exhaustive()
|
||||
}
|
||||
}
|
||||
|
||||
impl Interface {
|
||||
/// Creates the interface and starts moving packets.
|
||||
pub async fn start(
|
||||
@@ -221,7 +230,29 @@ impl Interface {
|
||||
let name = device.name().to_string();
|
||||
let tally = Arc::new(Tally::default());
|
||||
|
||||
let vip_task = {
|
||||
let routes = Arc::clone(&routes);
|
||||
let carrier = Arc::clone(&carrier);
|
||||
tokio::spawn(async move {
|
||||
let mut interval = tokio::time::interval(std::time::Duration::from_millis(250));
|
||||
interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
|
||||
let mut checks = tokio::task::JoinSet::new();
|
||||
loop {
|
||||
tokio::select! {
|
||||
_=interval.tick()=>{
|
||||
let (probes,jobs)=routes.vips.tick(routes.vips.check_capacity().saturating_sub(checks.len()));
|
||||
for (route,packet) in probes {carrier.carry(route,packet);}
|
||||
for job in jobs {checks.spawn(job.run());}
|
||||
},
|
||||
Some(result)=checks.join_next(),if !checks.is_empty()=>{
|
||||
if let Ok((job,ok))=result {routes.vips.checked(job,ok);}
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
};
|
||||
let task = {
|
||||
let carrier = Arc::clone(&carrier);
|
||||
let device = Arc::clone(&device);
|
||||
let routes = Arc::clone(&routes);
|
||||
let tally = Arc::clone(&tally);
|
||||
@@ -231,6 +262,26 @@ impl Interface {
|
||||
tally.malformed.fetch_add(1, Ordering::Relaxed);
|
||||
continue;
|
||||
};
|
||||
match routes.vips.outgoing(packet.clone()) {
|
||||
crate::vip::router::Decision::Send(route, packet) => {
|
||||
for part in crate::vip::packet::fragments(packet, mtu) {
|
||||
if carrier.carry(route, part) {
|
||||
tally.sent.fetch_add(1, Ordering::Relaxed);
|
||||
} else {
|
||||
tally.undeliverable.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
continue;
|
||||
}
|
||||
crate::vip::router::Decision::Local(packet) => {
|
||||
for part in crate::vip::packet::fragments(packet, mtu) {
|
||||
let _ = device.send(part).await;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
crate::vip::router::Decision::Drop => continue,
|
||||
crate::vip::router::Decision::Pass => {}
|
||||
}
|
||||
let destination = header.destination();
|
||||
if let (IpAddr::V4(source), IpAddr::V4(destination)) =
|
||||
(header.source(), destination)
|
||||
@@ -279,9 +330,11 @@ impl Interface {
|
||||
name,
|
||||
mtu,
|
||||
routes,
|
||||
carrier,
|
||||
tally,
|
||||
applied: std::sync::Mutex::new(Vec::new()),
|
||||
task: std::sync::Mutex::new(Some(task)),
|
||||
vip_task: std::sync::Mutex::new(Some(vip_task)),
|
||||
sync_lock: Arc::new(tokio::sync::Mutex::new(())),
|
||||
host_rules,
|
||||
rules_report: std::sync::Mutex::new(None),
|
||||
@@ -397,6 +450,14 @@ impl Interface {
|
||||
return;
|
||||
}
|
||||
if planned.as_ref() == Some(&wanted) && ok {
|
||||
if let Some(outcome) = exit.refresh(&wanted).await {
|
||||
if let RuleOutcome::Failed(reason) = &outcome {
|
||||
tracing::warn!(interface = %self.name, %reason, "cannot refresh exit-client socket protection");
|
||||
}
|
||||
if let Some(report) = self.exit_state().report.as_mut() {
|
||||
report.client = Some(outcome);
|
||||
}
|
||||
}
|
||||
return;
|
||||
}
|
||||
let report = exit.apply(&wanted).await;
|
||||
@@ -708,6 +769,15 @@ impl Interface {
|
||||
task.abort();
|
||||
let _ = task.await;
|
||||
}
|
||||
let vip_task = self
|
||||
.vip_task
|
||||
.lock()
|
||||
.unwrap_or_else(|e| e.into_inner())
|
||||
.take();
|
||||
if let Some(task) = vip_task {
|
||||
task.abort();
|
||||
let _ = task.await;
|
||||
}
|
||||
self.factory.destroy(self.device.name()).await;
|
||||
}
|
||||
|
||||
@@ -762,6 +832,24 @@ impl Interface {
|
||||
return Err(Rejected::WrongDestination);
|
||||
}
|
||||
};
|
||||
// Source ownership was checked before any VIP reverse translation.
|
||||
let packet = match self.routes.vips.incoming(network, peer, packet.clone()) {
|
||||
crate::vip::router::Decision::Send(route, packet) => {
|
||||
// Probe replies use the ordinary peer carrier, never the TUN.
|
||||
self.carrier.carry(route, packet);
|
||||
return Ok(());
|
||||
}
|
||||
crate::vip::router::Decision::Local(packet) => {
|
||||
for part in crate::vip::packet::fragments(packet, self.mtu) {
|
||||
if self.device.send(part).await.is_ok() {
|
||||
self.tally.received.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
return Ok(());
|
||||
}
|
||||
crate::vip::router::Decision::Drop => return Ok(()),
|
||||
crate::vip::router::Decision::Pass => packet,
|
||||
};
|
||||
// Remote broadcasts terminate here. Only local TUN ingress can fan out.
|
||||
if self.device.send(packet).await.is_ok() {
|
||||
if broadcast {
|
||||
@@ -786,6 +874,14 @@ impl Interface {
|
||||
|
||||
impl Drop for Interface {
|
||||
fn drop(&mut self) {
|
||||
if let Some(task) = self
|
||||
.vip_task
|
||||
.lock()
|
||||
.unwrap_or_else(|e| e.into_inner())
|
||||
.take()
|
||||
{
|
||||
task.abort();
|
||||
}
|
||||
if let Ok(mut guard) = self.task.lock()
|
||||
&& let Some(task) = guard.take()
|
||||
{
|
||||
|
||||
@@ -55,6 +55,7 @@ use crate::BoxFuture;
|
||||
use crate::overlay::OverlayError;
|
||||
|
||||
use super::super::config::Cidr;
|
||||
use super::super::hostrules::linux_exit_mark::{UNDERLAY_MARK, UNDERLAY_MASK};
|
||||
use super::super::tun::{TunDevice, TunRequest};
|
||||
use super::privilege::{NetAdmin, Privilege, probe_net_admin};
|
||||
use super::{
|
||||
@@ -641,8 +642,10 @@ async fn del_broadcast_route(name: &str) -> Result<(), OverlayError> {
|
||||
const EXIT_TABLE: u32 = 0x7473;
|
||||
/// The main table.
|
||||
const MAIN_TABLE: u32 = 254;
|
||||
/// Where the exit rules sit, after Tailscale's 5270.
|
||||
const EXIT_BYPASS_PRIORITY: u32 = 5280;
|
||||
/// Protect marked underlay and initial socket routing before Tailscale's rules.
|
||||
const EXIT_MARK_PRIORITY: u32 = 5205;
|
||||
const EXIT_BYPASS_PRIORITY: u32 = 5206;
|
||||
const LEGACY_EXIT_BYPASS_PRIORITY: u32 = 5280;
|
||||
const EXIT_SPECIFIC_PRIORITY: u32 = 5290;
|
||||
const EXIT_DEFAULT_PRIORITY: u32 = 5300;
|
||||
|
||||
@@ -666,14 +669,23 @@ fn rule_priority(rule: &RuleMessage) -> Option<u32> {
|
||||
})
|
||||
}
|
||||
|
||||
/// Whether a rule is one of the three this agent adds for an exit node.
|
||||
/// Whether a rule is one this agent adds for an exit node, including old UID priority.
|
||||
///
|
||||
/// Recognised by shape as well as priority: those numbers are not reserved,
|
||||
/// and a rule somebody else put at one of them must not be removed.
|
||||
fn is_exit_rule(rule: &RuleMessage) -> bool {
|
||||
let table = rule_table(rule);
|
||||
match rule_priority(rule) {
|
||||
Some(EXIT_BYPASS_PRIORITY) => {
|
||||
Some(EXIT_MARK_PRIORITY) => {
|
||||
table == MAIN_TABLE
|
||||
&& rule
|
||||
.attributes
|
||||
.contains(&RuleAttribute::FwMark(UNDERLAY_MARK))
|
||||
&& rule
|
||||
.attributes
|
||||
.contains(&RuleAttribute::FwMask(UNDERLAY_MASK))
|
||||
}
|
||||
Some(EXIT_BYPASS_PRIORITY | LEGACY_EXIT_BYPASS_PRIORITY) => {
|
||||
table == MAIN_TABLE
|
||||
&& rule
|
||||
.attributes
|
||||
@@ -789,7 +801,8 @@ async fn remove_exit_client(handle: &rtnetlink::Handle) -> Result<(), OverlayErr
|
||||
///
|
||||
/// ```text
|
||||
/// ip route replace default dev <if> table 0x7473
|
||||
/// ip rule add priority 5280 uidrange <uid>-<uid> lookup main
|
||||
/// ip rule add priority 5205 fwmark 0x40000000/0x40000000 lookup main
|
||||
/// ip rule add priority 5206 uidrange <uid>-<uid> lookup main
|
||||
/// ip rule add priority 5290 lookup main suppress_prefixlength 0
|
||||
/// ip rule add priority 5300 lookup 0x7473
|
||||
/// ```
|
||||
@@ -819,6 +832,8 @@ async fn set_exit_client(name: &str, uid: u32) -> Result<(), OverlayError> {
|
||||
.await
|
||||
.map_err(|err| fail("default route", err))?;
|
||||
|
||||
add_mark_bypass(&handle, false).await?;
|
||||
|
||||
let mut bypass = handle
|
||||
.rule()
|
||||
.add()
|
||||
@@ -891,6 +906,7 @@ async fn set_exit_block_v6(handle: &rtnetlink::Handle, uid: u32) -> Result<(), O
|
||||
};
|
||||
}
|
||||
|
||||
add_mark_bypass(handle, true).await?;
|
||||
let mut bypass = handle
|
||||
.rule()
|
||||
.add()
|
||||
@@ -938,6 +954,27 @@ async fn set_exit_block_v6(handle: &rtnetlink::Handle, uid: u32) -> Result<(), O
|
||||
.map_err(|err| fail("rule for the default route", err))
|
||||
}
|
||||
|
||||
async fn add_mark_bypass(handle: &rtnetlink::Handle, v6: bool) -> Result<(), OverlayError> {
|
||||
let mut rule = handle
|
||||
.rule()
|
||||
.add()
|
||||
.table_id(MAIN_TABLE)
|
||||
.action(RuleAction::ToTable)
|
||||
.priority(EXIT_MARK_PRIORITY)
|
||||
.fw_mark(UNDERLAY_MARK);
|
||||
rule.message_mut().header.family = if v6 {
|
||||
rtnetlink::packet_route::AddressFamily::Inet6
|
||||
} else {
|
||||
rtnetlink::packet_route::AddressFamily::Inet
|
||||
};
|
||||
rule.message_mut()
|
||||
.attributes
|
||||
.push(RuleAttribute::FwMask(UNDERLAY_MASK));
|
||||
rule.execute().await.map_err(|err| {
|
||||
OverlayError::Unavailable(format!("cannot protect marked underlay traffic: {err}"))
|
||||
})
|
||||
}
|
||||
|
||||
async fn configure_link(configure: &Configure) -> Result<(), OverlayError> {
|
||||
let name = configure.name.as_str();
|
||||
with_netlink(|handle| async move {
|
||||
@@ -1110,6 +1147,28 @@ mod tests {
|
||||
assert!(!is_exit_rule(&rule(5280, 100, vec![uid])));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn underlay_bypass_recognises_only_our_exact_bit_and_mask() {
|
||||
let rule = |table, mark, mask| {
|
||||
let mut message = RuleMessage::default();
|
||||
message.attributes.extend([
|
||||
RuleAttribute::Table(table),
|
||||
RuleAttribute::Priority(EXIT_MARK_PRIORITY),
|
||||
RuleAttribute::FwMark(mark),
|
||||
RuleAttribute::FwMask(mask),
|
||||
]);
|
||||
message
|
||||
};
|
||||
assert!(is_exit_rule(&rule(
|
||||
MAIN_TABLE,
|
||||
UNDERLAY_MARK,
|
||||
UNDERLAY_MASK
|
||||
)));
|
||||
assert!(!is_exit_rule(&rule(MAIN_TABLE, 0x80000, 0xff0000)));
|
||||
assert!(!is_exit_rule(&rule(MAIN_TABLE, UNDERLAY_MARK, u32::MAX)));
|
||||
assert!(!is_exit_rule(&rule(52, UNDERLAY_MARK, UNDERLAY_MASK)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_ipv6_default_of_the_exit_table_refuses_instead_of_routing() {
|
||||
let route = exit_blocking_route_v6();
|
||||
|
||||
@@ -72,7 +72,7 @@ pub enum RouteError {
|
||||
#[error(
|
||||
"network {network} uses {range}, which overlaps {other_range} already in use by \
|
||||
{other}. One agent has one interface, so an address can belong to only one of \
|
||||
them: give one network a different range."
|
||||
them: stop the other network before starting this one."
|
||||
)]
|
||||
Overlap {
|
||||
/// The network being added.
|
||||
@@ -89,6 +89,7 @@ pub enum RouteError {
|
||||
/// Address ownership across every network this agent is in.
|
||||
#[derive(Debug, Default)]
|
||||
pub struct RoutingTable {
|
||||
pub(crate) vips: Arc<crate::vip::VipRouter>,
|
||||
broadcast: ArcSwap<BroadcastTable>,
|
||||
networks: RwLock<HashMap<NetworkId, NetworkRoutes>>,
|
||||
}
|
||||
@@ -146,6 +147,7 @@ impl RoutingTable {
|
||||
|
||||
/// Forgets a network.
|
||||
pub fn remove_network(&self, network: NetworkId) {
|
||||
self.vips.remove_network(network);
|
||||
let mut networks = self.write();
|
||||
networks.remove(&network);
|
||||
self.broadcast
|
||||
@@ -365,6 +367,43 @@ impl RoutingTable {
|
||||
})
|
||||
}
|
||||
|
||||
/// All active networks whose reserved ranges intersect this range.
|
||||
pub fn overlapping_networks(
|
||||
&self,
|
||||
network: NetworkId,
|
||||
range: Ipv4Range,
|
||||
) -> Vec<(NetworkId, Ipv4Range)> {
|
||||
let mut conflicts: Vec<_> = self
|
||||
.read()
|
||||
.iter()
|
||||
.filter_map(|(other, routes)| {
|
||||
let other_range = routes.range?;
|
||||
(*other != network && ranges_overlap(range, other_range))
|
||||
.then_some((*other, other_range))
|
||||
})
|
||||
.collect();
|
||||
conflicts.sort_by_key(|(id, _)| *id);
|
||||
conflicts
|
||||
}
|
||||
|
||||
/// Current signed allocations used for encrypted data delivery checks.
|
||||
pub(crate) fn probe_addresses(
|
||||
&self,
|
||||
network: NetworkId,
|
||||
peer: EndpointId,
|
||||
) -> Option<(Ipv4Addr, Ipv4Addr)> {
|
||||
let table = self.read();
|
||||
let routes = table.get(&network)?;
|
||||
Some((
|
||||
routes.local?,
|
||||
routes
|
||||
.peers
|
||||
.iter()
|
||||
.find(|(_, holder)| *holder == peer)
|
||||
.map(|(address, _)| *address)?,
|
||||
))
|
||||
}
|
||||
|
||||
/// How many networks the table covers.
|
||||
pub fn len(&self) -> usize {
|
||||
self.read().len()
|
||||
@@ -510,7 +549,7 @@ mod tests {
|
||||
let err = table.set_network(second, routes()).unwrap_err();
|
||||
let RouteError::Overlap { other, .. } = err;
|
||||
assert_eq!(other, first);
|
||||
assert!(err.to_string().contains("different range"), "{err}");
|
||||
assert!(err.to_string().contains("stop the other network"), "{err}");
|
||||
assert_eq!(table.len(), 1, "the overlapping network is not added");
|
||||
|
||||
// A range of its own is fine.
|
||||
|
||||
@@ -20,7 +20,7 @@ use crate::error::ProtocolError;
|
||||
/// The version in the ALPN is the wire-compatibility version of the control
|
||||
/// protocol. It is independent of the network identity scheme version, so
|
||||
/// bumping it must not change any existing [`crate::NetworkId`].
|
||||
pub const ALPN: &[u8] = b"tsunagi/ctrl/4";
|
||||
pub const ALPN: &[u8] = b"tsunagi/ctrl/8";
|
||||
|
||||
/// ALPN of the tsunagi data plane.
|
||||
///
|
||||
@@ -30,9 +30,11 @@ pub const ALPN: &[u8] = b"tsunagi/ctrl/4";
|
||||
/// transport underneath can be replaced without touching the control protocol.
|
||||
///
|
||||
/// Version 4 adds source, destination, hop limit and stable flow id above
|
||||
/// transport fragmentation. All members must upgrade together. Persistent
|
||||
/// identities and network configuration do not change.
|
||||
pub const DATA_ALPN: &[u8] = b"tsunagi/data/4";
|
||||
/// transport fragmentation. Version 5 adds the end-to-end protocol to that
|
||||
/// envelope, so a tunnel no longer depends on the kind of link under it, and
|
||||
/// a first-claimed time to a signed hostname claim. All members must upgrade
|
||||
/// together. Persistent identities and network configuration do not change.
|
||||
pub const DATA_ALPN: &[u8] = b"tsunagi/data/5";
|
||||
|
||||
/// Largest plugin protocol identifier accepted when opening a data channel.
|
||||
pub const MAX_DATA_PROTOCOL_LEN: usize = 32;
|
||||
@@ -41,7 +43,9 @@ pub const MAX_DATA_PROTOCOL_LEN: usize = 32;
|
||||
pub const MAX_SIGNATURE_LEN: usize = 64;
|
||||
|
||||
/// Control protocol version carried inside the handshake.
|
||||
pub const PROTOCOL_VERSION: u16 = 4;
|
||||
pub const PROTOCOL_VERSION: u16 = 8;
|
||||
|
||||
pub use crate::dataplane::transport::{MAX_TRANSPORT_PAYLOAD, TransportPayload};
|
||||
|
||||
/// First message of the handshake, sent by the initiator.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
@@ -112,6 +116,13 @@ pub struct DataOpenAck {
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[non_exhaustive]
|
||||
pub enum ControlMessage {
|
||||
/// Private transport negotiation; never persisted or rebroadcast.
|
||||
Transport {
|
||||
/// Transport protocol id; the payload stays opaque to the core.
|
||||
protocol: String,
|
||||
/// Secret negotiation data, redacted in diagnostics.
|
||||
payload: TransportPayload,
|
||||
},
|
||||
/// Hostname and capability announcement.
|
||||
Announce(Announcement),
|
||||
/// A small request used to verify that the exchange works.
|
||||
@@ -170,6 +181,16 @@ pub enum ControlMessage {
|
||||
/// Short free-text reason.
|
||||
reason: String,
|
||||
},
|
||||
/// One complete signed VIP field snapshot, merged atomically.
|
||||
Vips {
|
||||
/// Signed registers bounded by the per-VIP field limit.
|
||||
entries: Vec<crate::vip::VipEntry>,
|
||||
},
|
||||
/// Signed cleanup markers, each proving the retired claim version.
|
||||
Retirements {
|
||||
/// Bounded by the per-message signed-state limit.
|
||||
markers: Vec<crate::state::SignedRetirement>,
|
||||
},
|
||||
}
|
||||
|
||||
/// Somewhere a member has been seen.
|
||||
@@ -262,6 +283,48 @@ pub fn validate_capability(
|
||||
/// network, the other networks or the agent.
|
||||
pub fn validate(message: &ControlMessage, limits: &Limits) -> Result<(), ProtocolError> {
|
||||
match message {
|
||||
ControlMessage::Vips { entries } => {
|
||||
check_len("vip.fields", entries.len(), limits.max_vip_fields)?;
|
||||
for e in entries {
|
||||
check_len("vip.id", e.vip.len(), 32)?;
|
||||
check_len("vip.key", e.key.len(), 160)?;
|
||||
check_len("vip.value", e.value.as_ref().map_or(0, Vec::len), 256)?;
|
||||
check_len("vip.signature", e.signature.len(), 64)?;
|
||||
}
|
||||
}
|
||||
ControlMessage::Retirements { markers } => {
|
||||
check_len("retirements", markers.len(), limits.max_state_records)?;
|
||||
for marker in markers {
|
||||
check_len(
|
||||
"retirement.signature",
|
||||
marker.signature.len(),
|
||||
MAX_SIGNATURE_LEN,
|
||||
)?;
|
||||
check_len(
|
||||
"retirement.target.signature",
|
||||
marker.target.signature.len(),
|
||||
MAX_SIGNATURE_LEN,
|
||||
)?;
|
||||
if let Some(name) = marker.target.body.hostname() {
|
||||
check_len(
|
||||
"retirement.target.hostname",
|
||||
name.len(),
|
||||
limits.max_hostname_len,
|
||||
)?;
|
||||
}
|
||||
}
|
||||
}
|
||||
ControlMessage::Transport { protocol, payload } => {
|
||||
if protocol.is_empty() {
|
||||
return Err(ProtocolError::Malformed("empty transport protocol"));
|
||||
}
|
||||
check_len("transport.protocol", protocol.len(), MAX_PROTOCOL_ID_LEN)?;
|
||||
check_len(
|
||||
"transport.payload",
|
||||
payload.as_bytes().len(),
|
||||
limits.max_transport_payload_len.min(MAX_TRANSPORT_PAYLOAD),
|
||||
)?;
|
||||
}
|
||||
ControlMessage::Peers { peers } => {
|
||||
check_len("peers", peers.len(), limits.max_state_records)?;
|
||||
for hint in peers {
|
||||
@@ -318,10 +381,13 @@ pub fn validate(message: &ControlMessage, limits: &Limits) -> Result<(), Protoco
|
||||
/// A short, stable label for a message kind, for metrics and diagnostics.
|
||||
pub fn kind(message: &ControlMessage) -> &'static str {
|
||||
match message {
|
||||
ControlMessage::Transport { .. } => "transport",
|
||||
ControlMessage::Announce(_) => "announce",
|
||||
ControlMessage::Ping { .. } => "ping",
|
||||
ControlMessage::Pong { .. } => "pong",
|
||||
ControlMessage::State { .. } => "state",
|
||||
ControlMessage::Retirements { .. } => "retirements",
|
||||
ControlMessage::Vips { .. } => "vips",
|
||||
ControlMessage::Reachable { .. } => "reachable",
|
||||
ControlMessage::Peers { .. } => "peers",
|
||||
ControlMessage::Bye { .. } => "bye",
|
||||
|
||||
@@ -28,16 +28,25 @@
|
||||
//! replica that has the record needs nothing else to interpret it, and
|
||||
//! recovery never depends on replaying a chain from the beginning.
|
||||
//!
|
||||
//! Any member may also sign a separate retirement of another author's proved
|
||||
//! claim version. It suppresses stale reservations without changing that
|
||||
//! author's record, and a newer claim remains valid.
|
||||
//!
|
||||
//! # What this slice does not do yet
|
||||
//!
|
||||
//! Compaction, revocation of a whole author, and snapshots covering more than
|
||||
//! one record type. See `docs/sync-model.md` for the shape those take.
|
||||
//! Compaction and permanent revocation of a whole author. See
|
||||
//! `docs/sync-model.md` for the implemented cleanup semantics.
|
||||
|
||||
pub mod allocator;
|
||||
/// Signed shared virtual service state and its merge rules.
|
||||
pub mod vip;
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::net::Ipv4Addr;
|
||||
|
||||
mod retirement;
|
||||
pub use retirement::{MAX_RETIREMENT_LEN, SignedRetirement};
|
||||
|
||||
use iroh::{EndpointId, SecretKey, Signature};
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
@@ -109,12 +118,8 @@ impl std::str::FromStr for Ipv4Range {
|
||||
}
|
||||
}
|
||||
|
||||
/// The IPv4 overlay range used unless something else is configured or agreed.
|
||||
///
|
||||
/// A small, specific `/24`: memorable, and far less likely to overlap a
|
||||
/// network the machine is already on than taking a whole `/8` or `/10` would
|
||||
/// be. Because addresses are allocated rather than derived, 254 of them is
|
||||
/// plenty for the size of network this is for.
|
||||
/// The legacy default range, retained for existing networks and explicit use.
|
||||
/// New networks use [`derived_ipv4_range`] by default.
|
||||
pub const DEFAULT_IPV4_RANGE: Ipv4Range = Ipv4Range {
|
||||
base: Ipv4Addr::new(10, 13, 37, 0),
|
||||
prefix_len: 24,
|
||||
@@ -122,15 +127,10 @@ pub const DEFAULT_IPV4_RANGE: Ipv4Range = Ipv4Range {
|
||||
|
||||
/// A range derived from a network's own identity.
|
||||
///
|
||||
/// One agent has one interface, so two of its networks cannot both use the
|
||||
/// configured range. The second needs another one — and it cannot be picked
|
||||
/// locally, because every member has to arrive at the same answer without
|
||||
/// being told. So it comes from the network id, which every member already
|
||||
/// has and nobody can choose: the same network derives the same range on
|
||||
/// every device.
|
||||
///
|
||||
/// Inside 10/8 like the default, and never equal to it, so the first
|
||||
/// network keeps what it has always had.
|
||||
/// Every member independently derives the same pseudorandom `/24` in 10/8,
|
||||
/// including isolated first members. Stored signed ranges take precedence.
|
||||
/// The derivation is kept stable for networks already using it; the legacy
|
||||
/// default is excluded. Distinct network ids can still produce collisions.
|
||||
pub fn derived_ipv4_range(network: NetworkId) -> Ipv4Range {
|
||||
let bytes = network.as_bytes();
|
||||
let second = bytes[0];
|
||||
@@ -150,7 +150,7 @@ pub fn derived_ipv4_range(network: NetworkId) -> Ipv4Range {
|
||||
}
|
||||
|
||||
/// Frozen domain separator for the bytes a record signature covers.
|
||||
pub const RECORD_DOMAIN: &str = "tsunagi-signed-record-v2";
|
||||
pub const RECORD_DOMAIN: &str = "tsunagi-signed-record-v3";
|
||||
|
||||
/// Longest hostname a record may carry.
|
||||
///
|
||||
@@ -207,6 +207,18 @@ pub enum RecordBody {
|
||||
range: Option<Ipv4Range>,
|
||||
/// The name this author answers to.
|
||||
hostname: Option<String>,
|
||||
/// When this author first claimed that name, in milliseconds since
|
||||
/// the Unix epoch, kept across every later version that keeps the
|
||||
/// name. It is what makes a name belong to whoever claimed it first
|
||||
/// rather than to whoever happens to sort lowest, so a member that
|
||||
/// arrives later with the same name cannot take it over.
|
||||
///
|
||||
/// The author's own word: a member with a wrong clock can claim an
|
||||
/// earlier time. That costs it nothing it could not do by other
|
||||
/// means, since anybody who knows the network secret is a member,
|
||||
/// and it cannot take an address or reach anything it was not
|
||||
/// already entitled to.
|
||||
hostname_since: Option<u64>,
|
||||
},
|
||||
/// This author gives up everything it claimed.
|
||||
///
|
||||
@@ -243,6 +255,14 @@ impl RecordBody {
|
||||
}
|
||||
}
|
||||
|
||||
/// When the author first claimed the hostname, if it has one.
|
||||
pub fn hostname_since(&self) -> Option<u64> {
|
||||
match self {
|
||||
RecordBody::Claim { hostname_since, .. } => *hostname_since,
|
||||
RecordBody::Release => None,
|
||||
}
|
||||
}
|
||||
|
||||
fn canonical(&self) -> Vec<u8> {
|
||||
let mut out = Vec::with_capacity(96);
|
||||
match self {
|
||||
@@ -250,6 +270,7 @@ impl RecordBody {
|
||||
address,
|
||||
range,
|
||||
hostname,
|
||||
hostname_since,
|
||||
} => {
|
||||
push_lp(&mut out, b"claim");
|
||||
push_opt(
|
||||
@@ -270,6 +291,13 @@ impl RecordBody {
|
||||
.as_deref(),
|
||||
);
|
||||
push_opt(&mut out, hostname.as_deref().map(str::as_bytes));
|
||||
push_opt(
|
||||
&mut out,
|
||||
hostname_since
|
||||
.map(|since| since.to_be_bytes())
|
||||
.as_ref()
|
||||
.map(|bytes| &bytes[..]),
|
||||
);
|
||||
}
|
||||
RecordBody::Release => {
|
||||
push_lp(&mut out, b"release");
|
||||
@@ -391,8 +419,12 @@ impl SignedRecord {
|
||||
address,
|
||||
range,
|
||||
hostname,
|
||||
hostname_since,
|
||||
} = &self.body
|
||||
{
|
||||
if hostname.is_none() && hostname_since.is_some() {
|
||||
return Err(StateError::Malformed("a time was claimed for no hostname"));
|
||||
}
|
||||
// Bounds before anything is believed, because all of this came
|
||||
// off the network.
|
||||
if let Some(range) = range {
|
||||
@@ -465,6 +497,7 @@ pub enum Merged {
|
||||
#[derive(Debug, Clone, Default)]
|
||||
pub struct StateSet {
|
||||
records: HashMap<EndpointId, SignedRecord>,
|
||||
retirements: HashMap<EndpointId, SignedRetirement>,
|
||||
}
|
||||
|
||||
impl StateSet {
|
||||
@@ -477,6 +510,7 @@ impl StateSet {
|
||||
pub fn from_verified(records: impl IntoIterator<Item = (EndpointId, SignedRecord)>) -> Self {
|
||||
Self {
|
||||
records: records.into_iter().collect(),
|
||||
retirements: HashMap::new(),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -505,6 +539,52 @@ impl StateSet {
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Visible records, excluding versions retired by authenticated members.
|
||||
pub fn active_records(&self) -> Vec<SignedRecord> {
|
||||
self.records()
|
||||
.into_iter()
|
||||
.filter(|r| r.author_id().is_ok_and(|id| !self.is_retired(&id)))
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Whether the current known version is at or below a proved retirement.
|
||||
pub fn is_retired(&self, author: &EndpointId) -> bool {
|
||||
self.retirements
|
||||
.get(author)
|
||||
.is_some_and(|m| self.version_of(author) <= m.target.version)
|
||||
}
|
||||
|
||||
/// Strongest retained cleanup proof for an identity.
|
||||
pub fn retirement_of(&self, target: &EndpointId) -> Option<&SignedRetirement> {
|
||||
self.retirements.get(target)
|
||||
}
|
||||
|
||||
/// Durable markers in target-id order, including markers surpassed by a fresh claim.
|
||||
pub fn retirements(&self) -> Vec<SignedRetirement> {
|
||||
let mut markers: Vec<_> = self.retirements.values().cloned().collect();
|
||||
markers.sort_by_key(|m| m.target.author);
|
||||
markers
|
||||
}
|
||||
|
||||
/// Accepts any member's signed cleanup after checking both signatures.
|
||||
pub fn merge_retirement(
|
||||
&mut self,
|
||||
network: NetworkId,
|
||||
marker: SignedRetirement,
|
||||
max: usize,
|
||||
) -> Result<bool, StateError> {
|
||||
let target = marker.verify(network)?;
|
||||
if let Some(old) = self.retirements.get(&target) {
|
||||
if !marker.supersedes(old) {
|
||||
return Ok(false);
|
||||
}
|
||||
} else if self.retirements.len() >= max {
|
||||
return Err(StateError::Malformed("retirement capacity reached"));
|
||||
}
|
||||
self.retirements.insert(target, marker);
|
||||
Ok(true)
|
||||
}
|
||||
|
||||
/// Merges one record, verifying it first.
|
||||
///
|
||||
/// Merging is into the existing set, never a wholesale replacement, and an
|
||||
@@ -568,6 +648,9 @@ impl StateSet {
|
||||
pub fn address_holders(&self) -> HashMap<Ipv4Addr, EndpointId> {
|
||||
let mut holders: HashMap<Ipv4Addr, EndpointId> = HashMap::new();
|
||||
for (author, record) in &self.records {
|
||||
if self.is_retired(author) {
|
||||
continue;
|
||||
}
|
||||
let Some(address) = record.body.claimed_address() else {
|
||||
continue;
|
||||
};
|
||||
@@ -585,25 +668,36 @@ impl StateSet {
|
||||
|
||||
/// Who currently holds each claimed hostname.
|
||||
///
|
||||
/// The same rule as addresses, for the same reason: a name is owned, two
|
||||
/// members may claim one, and every replica has to reach the same answer
|
||||
/// about who has it without asking anybody.
|
||||
/// A name belongs to the member that claimed it first, and the others
|
||||
/// that claim it keep working by address but are not found by that name.
|
||||
/// "First" is the time each author signed into its own claim, with the
|
||||
/// lower id deciding a tie and a claim with no time counting as last, so
|
||||
/// every replica reaches the same answer about who has it without
|
||||
/// asking anybody. It is a pure function of the records: nothing depends
|
||||
/// on the order they arrived in, so there is no race to lose.
|
||||
pub fn hostname_holders(&self) -> HashMap<String, EndpointId> {
|
||||
let mut holders: HashMap<String, EndpointId> = HashMap::new();
|
||||
let mut holders: HashMap<String, (u64, EndpointId)> = HashMap::new();
|
||||
for (author, record) in &self.records {
|
||||
if self.is_retired(author) {
|
||||
continue;
|
||||
}
|
||||
let Some(hostname) = record.body.hostname() else {
|
||||
continue;
|
||||
};
|
||||
let claim = (record.body.hostname_since().unwrap_or(u64::MAX), *author);
|
||||
holders
|
||||
.entry(hostname.to_string())
|
||||
.and_modify(|held| {
|
||||
if author.as_bytes() < held.as_bytes() {
|
||||
*held = *author;
|
||||
if (claim.0, claim.1.as_bytes()) < (held.0, held.1.as_bytes()) {
|
||||
*held = claim;
|
||||
}
|
||||
})
|
||||
.or_insert(*author);
|
||||
.or_insert(claim);
|
||||
}
|
||||
holders
|
||||
.into_iter()
|
||||
.map(|(hostname, (_, author))| (hostname, author))
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// The hostname an author holds, if it holds one uncontested.
|
||||
@@ -612,6 +706,22 @@ impl StateSet {
|
||||
(self.hostname_holders().get(hostname) == Some(author)).then_some(hostname)
|
||||
}
|
||||
|
||||
/// The name an author claimed and the member that holds it instead, when
|
||||
/// somebody who claimed it earlier does.
|
||||
pub fn hostname_taken(&self, author: &EndpointId) -> Option<(&str, EndpointId)> {
|
||||
if self.is_retired(author) {
|
||||
return None;
|
||||
}
|
||||
let hostname = self.records.get(author)?.body.hostname()?;
|
||||
let holder = *self.hostname_holders().get(hostname)?;
|
||||
(holder != *author).then_some((hostname, holder))
|
||||
}
|
||||
|
||||
/// When an author first claimed the name it holds now, if it has one.
|
||||
pub fn hostname_since_of(&self, author: &EndpointId) -> Option<u64> {
|
||||
self.records.get(author)?.body.hostname_since()
|
||||
}
|
||||
|
||||
/// The address an author holds, if it holds one uncontested.
|
||||
pub fn address_of(&self, author: &EndpointId) -> Option<Ipv4Addr> {
|
||||
let address = self.records.get(author)?.body.claimed_address()?;
|
||||
@@ -620,21 +730,44 @@ impl StateSet {
|
||||
|
||||
/// The range the network settled on, if anybody has said.
|
||||
///
|
||||
/// When claims disagree, the one from the lowest author id wins, so every
|
||||
/// replica reaches the same answer. A participant joining later therefore
|
||||
/// adopts the range already in use instead of imposing its own.
|
||||
/// A legacy or explicit range takes precedence over the derived default,
|
||||
/// so a fresh member's default cannot renumber an established network.
|
||||
/// Other disagreements use the lowest author id, independently of arrival
|
||||
/// order. The signed record format and network identity remain unchanged.
|
||||
pub fn agreed_range(&self) -> Option<Ipv4Range> {
|
||||
let mut authors: Vec<&EndpointId> = self.records.keys().collect();
|
||||
authors.sort_by_key(|author| *author.as_bytes());
|
||||
authors
|
||||
.into_iter()
|
||||
.find_map(|author| self.records.get(author)?.body.range())
|
||||
self.records
|
||||
.iter()
|
||||
.chain(
|
||||
self.retirements
|
||||
.iter()
|
||||
.filter(|(author, _)| !self.records.contains_key(*author))
|
||||
.map(|(author, marker)| (author, &marker.target)),
|
||||
)
|
||||
.filter_map(|(author, record)| {
|
||||
let range = record.body.range()?;
|
||||
let derived = derived_ipv4_range(NetworkId::from_bytes(record.network));
|
||||
Some(((range == derived, *author.as_bytes()), range))
|
||||
})
|
||||
.min_by_key(|(priority, _)| *priority)
|
||||
.map(|(_, range)| range)
|
||||
}
|
||||
|
||||
/// The highest version this author has published, as far as is known.
|
||||
pub fn version_of(&self, author: &EndpointId) -> u64 {
|
||||
self.records.get(author).map_or(0, |record| record.version)
|
||||
}
|
||||
|
||||
/// Highest witnessed author counter across claims and authored cleanup.
|
||||
/// This differs from the claim version used to decide retirement.
|
||||
pub fn authored_version(&self, author: &EndpointId) -> u64 {
|
||||
self.retirements
|
||||
.values()
|
||||
.filter(|m| m.author == *author.as_bytes())
|
||||
.map(|m| m.version)
|
||||
.chain(std::iter::once(self.version_of(author)))
|
||||
.max()
|
||||
.unwrap_or(0)
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
@@ -661,6 +794,7 @@ mod tests {
|
||||
address: Some(address.parse().unwrap()),
|
||||
range: Some(range()),
|
||||
hostname: None,
|
||||
hostname_since: None,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -720,6 +854,7 @@ mod tests {
|
||||
address: Some("10.99.0.1".parse().unwrap()),
|
||||
range: Some(range()),
|
||||
hostname: None,
|
||||
hostname_since: None,
|
||||
},
|
||||
);
|
||||
assert!(matches!(outside.verify(id), Err(StateError::Malformed(_))));
|
||||
@@ -735,6 +870,7 @@ mod tests {
|
||||
prefix_len: 31,
|
||||
}),
|
||||
hostname: None,
|
||||
hostname_since: None,
|
||||
},
|
||||
);
|
||||
assert!(matches!(no_hosts.verify(id), Err(StateError::Malformed(_))));
|
||||
@@ -776,6 +912,7 @@ mod tests {
|
||||
address: None,
|
||||
range: None,
|
||||
hostname: Some(bad.to_string()),
|
||||
hostname_since: None,
|
||||
},
|
||||
);
|
||||
assert!(
|
||||
@@ -797,6 +934,7 @@ mod tests {
|
||||
address: Some("10.13.37.5".parse().unwrap()),
|
||||
range: None,
|
||||
hostname: None,
|
||||
hostname_since: None,
|
||||
},
|
||||
);
|
||||
assert!(matches!(record.verify(id), Err(StateError::Malformed(_))));
|
||||
@@ -818,6 +956,7 @@ mod tests {
|
||||
address: None,
|
||||
range: None,
|
||||
hostname: Some("music".into()),
|
||||
hostname_since: None,
|
||||
},
|
||||
)
|
||||
};
|
||||
@@ -860,6 +999,7 @@ mod tests {
|
||||
address: None,
|
||||
range: None,
|
||||
hostname: Some(name.to_string()),
|
||||
hostname_since: None,
|
||||
},
|
||||
)
|
||||
};
|
||||
@@ -879,6 +1019,109 @@ mod tests {
|
||||
assert_eq!(set.hostname_of(&secret.public()), Some("new"));
|
||||
}
|
||||
|
||||
fn named_at(
|
||||
secret: &SecretKey,
|
||||
id: NetworkId,
|
||||
version: u64,
|
||||
name: &str,
|
||||
since: u64,
|
||||
) -> SignedRecord {
|
||||
SignedRecord::sign(
|
||||
secret,
|
||||
id,
|
||||
version,
|
||||
RecordBody::Claim {
|
||||
address: None,
|
||||
range: None,
|
||||
hostname: Some(name.into()),
|
||||
hostname_since: Some(since),
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_name_belongs_to_whoever_claimed_it_first_whatever_their_ids_or_arrival_order() {
|
||||
let id = network("naming");
|
||||
let mut keys = [SecretKey::generate(), SecretKey::generate()];
|
||||
keys.sort_by_key(|key| *key.public().as_bytes());
|
||||
let [low, high] = keys;
|
||||
|
||||
// The member with the higher id claimed first, so it holds the name,
|
||||
// and the lower id does not take it back by sorting first.
|
||||
let early = named_at(&high, id, 1, "music", 1_000);
|
||||
let late = named_at(&low, id, 1, "music", 2_000);
|
||||
for order in [[&early, &late], [&late, &early]] {
|
||||
let mut set = StateSet::new();
|
||||
for record in order {
|
||||
set.merge(id, (*record).clone()).unwrap();
|
||||
}
|
||||
assert_eq!(set.hostname_of(&high.public()), Some("music"));
|
||||
assert_eq!(set.hostname_of(&low.public()), None);
|
||||
assert_eq!(
|
||||
set.hostname_taken(&low.public()),
|
||||
Some(("music", high.public()))
|
||||
);
|
||||
assert_eq!(set.hostname_taken(&high.public()), None);
|
||||
}
|
||||
|
||||
// At the same instant the lower id decides, the same way everywhere.
|
||||
let mut set = StateSet::new();
|
||||
set.merge(id, named_at(&high, id, 1, "music", 5)).unwrap();
|
||||
set.merge(id, named_at(&low, id, 1, "music", 5)).unwrap();
|
||||
assert_eq!(set.hostname_of(&low.public()), Some("music"));
|
||||
assert_eq!(set.hostname_of(&high.public()), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_name_that_loses_is_found_again_once_the_holder_lets_it_go() {
|
||||
let id = network("naming");
|
||||
let first = SecretKey::generate();
|
||||
let second = SecretKey::generate();
|
||||
let mut set = StateSet::new();
|
||||
set.merge(id, named_at(&first, id, 1, "music", 1)).unwrap();
|
||||
set.merge(id, named_at(&second, id, 1, "music", 2)).unwrap();
|
||||
assert_eq!(set.hostname_of(&second.public()), None);
|
||||
|
||||
// Renaming starts the clock afresh, so it cannot be used to jump the
|
||||
// queue for a name somebody else already holds.
|
||||
set.merge(id, named_at(&first, id, 2, "other", 3)).unwrap();
|
||||
assert_eq!(set.hostname_of(&second.public()), Some("music"));
|
||||
set.merge(id, named_at(&first, id, 3, "music", 4)).unwrap();
|
||||
assert_eq!(set.hostname_of(&second.public()), Some("music"));
|
||||
assert_eq!(set.hostname_of(&first.public()), None);
|
||||
|
||||
// And a release hands it on.
|
||||
set.merge(id, SignedRecord::sign(&second, id, 2, RecordBody::Release))
|
||||
.unwrap();
|
||||
assert_eq!(set.hostname_of(&first.public()), Some("music"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_time_with_no_name_is_not_a_valid_claim() {
|
||||
let id = network("naming");
|
||||
let secret = SecretKey::generate();
|
||||
let record = SignedRecord::sign(
|
||||
&secret,
|
||||
id,
|
||||
1,
|
||||
RecordBody::Claim {
|
||||
address: None,
|
||||
range: None,
|
||||
hostname: None,
|
||||
hostname_since: Some(1),
|
||||
},
|
||||
);
|
||||
assert!(matches!(record.verify(id), Err(StateError::Malformed(_))));
|
||||
|
||||
// The time is signed: moving it breaks the signature, so nobody can
|
||||
// improve another member's place in the queue.
|
||||
let mut forged = named_at(&secret, id, 1, "music", 9_000);
|
||||
if let RecordBody::Claim { hostname_since, .. } = &mut forged.body {
|
||||
*hostname_since = Some(1);
|
||||
}
|
||||
assert_eq!(forged.verify(id).unwrap_err(), StateError::BadSignature);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_release_gives_up_the_name_as_well_as_the_address() {
|
||||
let secret = SecretKey::generate();
|
||||
@@ -894,6 +1137,7 @@ mod tests {
|
||||
address: Some("10.13.37.5".parse().unwrap()),
|
||||
range: Some(range()),
|
||||
hostname: Some("music".into()),
|
||||
hostname_since: None,
|
||||
},
|
||||
),
|
||||
)
|
||||
@@ -1057,6 +1301,7 @@ mod tests {
|
||||
address: Some("10.99.0.7".parse().unwrap()),
|
||||
range: Some(custom),
|
||||
hostname: None,
|
||||
hostname_since: None,
|
||||
},
|
||||
),
|
||||
)
|
||||
@@ -1098,7 +1343,7 @@ mod tests {
|
||||
assert_eq!(once.base.octets()[0], 10, "inside 10/8 like the default");
|
||||
assert_ne!(
|
||||
once, DEFAULT_IPV4_RANGE,
|
||||
"the default belongs to whichever network asked first"
|
||||
"the legacy default remains excluded"
|
||||
);
|
||||
|
||||
// A different network derives a different range, which is the
|
||||
@@ -1110,4 +1355,34 @@ mod tests {
|
||||
.network_id();
|
||||
assert_ne!(once, derived_ipv4_range(other));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_derived_default_cannot_displace_a_legacy_or_explicit_range() {
|
||||
let id = network("range-migration");
|
||||
let a = SecretKey::generate();
|
||||
let b = SecretKey::generate();
|
||||
let (low, high) = if a.public().as_bytes() < b.public().as_bytes() {
|
||||
(a, b)
|
||||
} else {
|
||||
(b, a)
|
||||
};
|
||||
for established in [DEFAULT_IPV4_RANGE, "10.90.0.0/16".parse().unwrap()] {
|
||||
let body = |range: Ipv4Range| RecordBody::Claim {
|
||||
address: Some(Ipv4Addr::from(u32::from(range.base) + 1)),
|
||||
range: Some(range),
|
||||
hostname: None,
|
||||
hostname_since: None,
|
||||
};
|
||||
let fresh = SignedRecord::sign(&low, id, 1, body(derived_ipv4_range(id)));
|
||||
let existing = SignedRecord::sign(&high, id, 1, body(established));
|
||||
for records in [
|
||||
vec![fresh.clone(), existing.clone()],
|
||||
vec![existing.clone(), fresh.clone()],
|
||||
] {
|
||||
let mut state = StateSet::new();
|
||||
state.merge_all(id, records);
|
||||
assert_eq!(state.agreed_range(), Some(established));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,219 @@
|
||||
//! A member may retire an observed claim, never another identity's future.
|
||||
use iroh::{EndpointId, SecretKey, Signature};
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
use super::{RecordBody, SignedRecord, StateError, push_lp};
|
||||
use crate::NetworkId;
|
||||
|
||||
/// Maximum encoded retirement size, including its bounded claim proof.
|
||||
pub const MAX_RETIREMENT_LEN: usize = 1024;
|
||||
|
||||
/// A network member's signed retirement of a proved claim version.
|
||||
///
|
||||
/// The original author's signed claim proves the cutoff really existed.
|
||||
/// Newer claims remain valid. This is neither a forged Release nor a ban.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct SignedRetirement {
|
||||
/// Device identity of the member requesting cleanup.
|
||||
pub author: [u8; 32],
|
||||
/// Network in which cleanup applies.
|
||||
pub network: [u8; 32],
|
||||
/// Requester's durable author counter, committed with this marker.
|
||||
pub version: u64,
|
||||
/// The target's original signed claim; bounds the retired version.
|
||||
pub target: SignedRecord,
|
||||
/// Requester's Ed25519 signature over this marker and its proof.
|
||||
pub signature: Vec<u8>,
|
||||
}
|
||||
|
||||
impl SignedRetirement {
|
||||
fn canonical_bytes(&self) -> Vec<u8> {
|
||||
let mut out = Vec::with_capacity(768);
|
||||
push_lp(&mut out, b"tsunagi-node-retirement/v1");
|
||||
push_lp(&mut out, &self.network);
|
||||
push_lp(&mut out, &self.author);
|
||||
push_lp(&mut out, &self.version.to_be_bytes());
|
||||
// Bind every field of the original signed claim, including its signature.
|
||||
push_lp(&mut out, &self.target.network);
|
||||
push_lp(&mut out, &self.target.author);
|
||||
push_lp(&mut out, &self.target.version.to_be_bytes());
|
||||
push_lp(&mut out, &self.target.body.canonical());
|
||||
push_lp(&mut out, &self.target.signature);
|
||||
out
|
||||
}
|
||||
|
||||
/// Signs a marker with the requesting member's persistent device key.
|
||||
pub fn sign(
|
||||
secret: &SecretKey,
|
||||
network: NetworkId,
|
||||
version: u64,
|
||||
target: SignedRecord,
|
||||
) -> Self {
|
||||
let mut marker = Self {
|
||||
author: *secret.public().as_bytes(),
|
||||
network: *network.as_bytes(),
|
||||
version,
|
||||
target,
|
||||
signature: Vec::new(),
|
||||
};
|
||||
marker.signature = secret.sign(&marker.canonical_bytes()).to_bytes().to_vec();
|
||||
marker
|
||||
}
|
||||
|
||||
/// Verifies both authors, scope and a finite, genuinely signed cutoff.
|
||||
pub fn verify(&self, network: NetworkId) -> Result<EndpointId, StateError> {
|
||||
if self.network != *network.as_bytes() {
|
||||
return Err(StateError::WrongNetwork);
|
||||
}
|
||||
if self.version == 0 || self.version > i64::MAX as u64 {
|
||||
return Err(StateError::Malformed("retirement version is out of range"));
|
||||
}
|
||||
if !matches!(self.target.body, RecordBody::Claim { .. }) {
|
||||
return Err(StateError::Malformed("only a claim can be retired"));
|
||||
}
|
||||
let target = self.target.verify(network)?;
|
||||
let author = EndpointId::from_bytes(&self.author).map_err(|_| StateError::BadAuthor)?;
|
||||
if author == target {
|
||||
return Err(StateError::Malformed(
|
||||
"use Release to retire your own claim",
|
||||
));
|
||||
}
|
||||
let raw: [u8; Signature::LENGTH] =
|
||||
self.signature
|
||||
.as_slice()
|
||||
.try_into()
|
||||
.map_err(|_| StateError::BadSignatureLength {
|
||||
expected: Signature::LENGTH,
|
||||
})?;
|
||||
author
|
||||
.verify(&self.canonical_bytes(), &Signature::from_bytes(&raw))
|
||||
.map_err(|_| StateError::BadSignature)?;
|
||||
Ok(target)
|
||||
}
|
||||
|
||||
/// A deterministic strongest marker per target, independent of arrival order.
|
||||
pub(crate) fn supersedes(&self, other: &Self) -> bool {
|
||||
self.target.version > other.target.version
|
||||
|| (self.target.version == other.target.version
|
||||
&& (self.canonical_bytes(), &self.signature)
|
||||
< (other.canonical_bytes(), &other.signature))
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
|
||||
use super::*;
|
||||
use crate::state::{Ipv4Range, StateSet};
|
||||
|
||||
fn claim(key: &SecretKey, id: NetworkId, version: u64) -> SignedRecord {
|
||||
SignedRecord::sign(
|
||||
key,
|
||||
id,
|
||||
version,
|
||||
RecordBody::Claim {
|
||||
address: Some("10.23.0.2".parse().unwrap()),
|
||||
range: Some("10.23.0.0/24".parse::<Ipv4Range>().unwrap()),
|
||||
hostname: Some("retired".into()),
|
||||
hostname_since: Some(123),
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn markers_release_claims_in_any_merge_order_and_older_snapshots_cannot_restore_them() {
|
||||
let id = NetworkId::from_bytes([7; 32]);
|
||||
let target = SecretKey::generate();
|
||||
let cleaner = SecretKey::generate();
|
||||
let record = claim(&target, id, 5);
|
||||
let marker = SignedRetirement::sign(&cleaner, id, 8, record.clone());
|
||||
for first in [true, false] {
|
||||
let mut state = StateSet::new();
|
||||
if first {
|
||||
state.merge_retirement(id, marker.clone(), 16).unwrap();
|
||||
}
|
||||
state.merge(id, record.clone()).unwrap();
|
||||
if !first {
|
||||
state.merge_retirement(id, marker.clone(), 16).unwrap();
|
||||
}
|
||||
assert!(state.address_holders().is_empty());
|
||||
assert!(state.hostname_holders().is_empty());
|
||||
state.merge(id, claim(&target, id, 4)).unwrap();
|
||||
assert!(state.is_retired(&target.public()));
|
||||
assert!(
|
||||
state.agreed_range().is_some(),
|
||||
"cleanup does not renumber the network"
|
||||
);
|
||||
state.merge(id, claim(&target, id, 6)).unwrap();
|
||||
assert!(!state.is_retired(&target.public()));
|
||||
assert_eq!(state.hostname_of(&target.public()), Some("retired"));
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn proof_signature_scope_tampering_future_cutoffs_and_capacity_are_checked() {
|
||||
let id = NetworkId::from_bytes([7; 32]);
|
||||
let target = SecretKey::generate();
|
||||
let cleaner = SecretKey::generate();
|
||||
let marker = SignedRetirement::sign(&cleaner, id, 8, claim(&target, id, 5));
|
||||
assert!(marker.verify(NetworkId::from_bytes([8; 32])).is_err());
|
||||
let mut forged = marker.clone();
|
||||
forged.target.version = 99;
|
||||
assert!(forged.verify(id).is_err());
|
||||
let mut forged = marker.clone();
|
||||
forged.signature[0] ^= 1;
|
||||
assert!(forged.verify(id).is_err());
|
||||
let mut proof = marker.target.clone();
|
||||
proof.version = 99;
|
||||
assert!(
|
||||
SignedRetirement::sign(&cleaner, id, 8, proof)
|
||||
.verify(id)
|
||||
.is_err()
|
||||
);
|
||||
let mut state = StateSet::new();
|
||||
assert!(state.merge_retirement(id, marker.clone(), 0).is_err());
|
||||
assert!(state.merge_retirement(id, marker.clone(), 1).unwrap());
|
||||
assert!(!state.merge_retirement(id, marker.clone(), 1).unwrap());
|
||||
assert!(postcard::to_stdvec(&marker).unwrap().len() <= MAX_RETIREMENT_LEN);
|
||||
let another = SecretKey::generate();
|
||||
assert!(
|
||||
state
|
||||
.merge_retirement(
|
||||
id,
|
||||
SignedRetirement::sign(&cleaner, id, 9, claim(&another, id, 1)),
|
||||
1
|
||||
)
|
||||
.is_err()
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn competing_cleanup_markers_converge_and_keep_the_highest_observed_cutoff() {
|
||||
let id = NetworkId::from_bytes([7; 32]);
|
||||
let target = SecretKey::generate();
|
||||
let a = SecretKey::generate();
|
||||
let b = SecretKey::generate();
|
||||
let markers = [
|
||||
SignedRetirement::sign(&a, id, 1, claim(&target, id, 4)),
|
||||
SignedRetirement::sign(&a, id, 2, claim(&target, id, 5)),
|
||||
SignedRetirement::sign(&b, id, 3, claim(&target, id, 5)),
|
||||
];
|
||||
let mut forward = StateSet::new();
|
||||
let mut reverse = StateSet::new();
|
||||
for marker in &markers {
|
||||
forward.merge_retirement(id, marker.clone(), 16).unwrap();
|
||||
}
|
||||
for marker in markers.iter().rev() {
|
||||
reverse.merge_retirement(id, marker.clone(), 16).unwrap();
|
||||
}
|
||||
assert_eq!(forward.retirements(), reverse.retirements());
|
||||
assert_eq!(
|
||||
forward
|
||||
.retirement_of(&target.public())
|
||||
.unwrap()
|
||||
.target
|
||||
.version,
|
||||
5
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,439 @@
|
||||
//! Signed multi-writer virtual service registers and deterministic merge rules.
|
||||
use crate::error::Result;
|
||||
use crate::vip::{Backend, Listener, Protocol, VipConfig, VipStatus, invalid};
|
||||
use crate::{Limits, NetworkId};
|
||||
use iroh::{EndpointId, SecretKey, Signature};
|
||||
use serde::{Deserialize, Serialize};
|
||||
use sha2::{Digest, Sha256};
|
||||
use std::collections::{BTreeMap, BTreeSet};
|
||||
use std::net::Ipv4Addr;
|
||||
|
||||
/// A signed, independently mergeable field. A removed field keeps its tombstone.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct VipEntry {
|
||||
/// Network scope.
|
||||
pub network: [u8; 32],
|
||||
/// Stable service id.
|
||||
pub vip: String,
|
||||
/// Canonical field key.
|
||||
pub key: String,
|
||||
/// Encoded field value; absent means deletion.
|
||||
pub value: Option<Vec<u8>>,
|
||||
/// Lamport clock, independent of wall time.
|
||||
pub clock: u64,
|
||||
/// Persistent signing identity.
|
||||
pub author: [u8; 32],
|
||||
/// Author's durable network counter.
|
||||
pub version: u64,
|
||||
/// Ed25519 signature.
|
||||
pub signature: Vec<u8>,
|
||||
}
|
||||
impl VipEntry {
|
||||
fn bytes(&self) -> Vec<u8> {
|
||||
let mut e = self.clone();
|
||||
e.signature.clear();
|
||||
let mut out = b"tsunagi-vip-field/v1\0".to_vec();
|
||||
if let Ok(bytes) = postcard::to_stdvec(&e) {
|
||||
out.extend(bytes);
|
||||
}
|
||||
out
|
||||
}
|
||||
/// Verifies bounds, network and authorship before the entry is merged.
|
||||
pub fn verify(&self, network: NetworkId) -> Result<()> {
|
||||
if self.network != *network.as_bytes()
|
||||
|| self.vip.len() != 32
|
||||
|| hex::decode(&self.vip).is_err()
|
||||
|| self.key.len() > 160
|
||||
|| self.value.as_ref().is_some_and(|v| v.len() > 256)
|
||||
|| self.version == 0
|
||||
|| self.version > i64::MAX as u64
|
||||
|| self.clock == 0
|
||||
|| self.clock > i64::MAX as u64
|
||||
{
|
||||
return Err(invalid("invalid VIP field bounds or network"));
|
||||
}
|
||||
let author =
|
||||
EndpointId::from_bytes(&self.author).map_err(|_| invalid("invalid VIP author"))?;
|
||||
let raw: [u8; 64] = self
|
||||
.signature
|
||||
.as_slice()
|
||||
.try_into()
|
||||
.map_err(|_| invalid("invalid VIP signature length"))?;
|
||||
author
|
||||
.verify(&self.bytes(), &Signature::from_bytes(&raw))
|
||||
.map_err(|_| invalid("invalid VIP signature"))?;
|
||||
if self.key != "deleted"
|
||||
&& self.key != "name"
|
||||
&& !self.key.starts_with("l/")
|
||||
&& !self.key.starts_with("b/")
|
||||
{
|
||||
return Err(invalid("unknown VIP field"));
|
||||
}
|
||||
match self.key.as_str() {
|
||||
"deleted" => {
|
||||
if self.value.as_deref() != Some(&[1][..]) {
|
||||
return Err(invalid("invalid VIP deletion tombstone"));
|
||||
}
|
||||
}
|
||||
"name" => {
|
||||
let value = self
|
||||
.value
|
||||
.as_ref()
|
||||
.ok_or_else(|| invalid("VIP name cannot be absent"))?;
|
||||
let name: String =
|
||||
postcard::from_bytes(value).map_err(|_| invalid("invalid VIP name field"))?;
|
||||
VipConfig {
|
||||
id: self.vip.clone(),
|
||||
name,
|
||||
listeners: vec![],
|
||||
}
|
||||
.validate(&Limits::default())?;
|
||||
}
|
||||
key if key.starts_with("l/") => {
|
||||
let parts: Vec<_> = key.split('/').collect();
|
||||
if parts.len() != 3
|
||||
|| parts[1].parse::<Protocol>().is_err()
|
||||
|| parts[2].parse::<u16>().ok().is_none_or(|p| p == 0)
|
||||
{
|
||||
return Err(invalid("invalid listener field key"));
|
||||
}
|
||||
if let Some(value) = &self.value {
|
||||
let l: Listener = postcard::from_bytes(value)
|
||||
.map_err(|_| invalid("invalid listener field"))?;
|
||||
if !l.backends.is_empty() || key != format!("l/{}/{}", l.protocol, l.port) {
|
||||
return Err(invalid("listener key does not match its value"));
|
||||
}
|
||||
VipConfig {
|
||||
id: self.vip.clone(),
|
||||
name: "validation".into(),
|
||||
listeners: vec![l],
|
||||
}
|
||||
.validate(&Limits::default())?;
|
||||
}
|
||||
}
|
||||
key if key.starts_with("b/") => {
|
||||
let parts: Vec<_> = key.split('/').collect();
|
||||
if parts.len() != 6
|
||||
|| parts[1].parse::<Protocol>().is_err()
|
||||
|| parts[2].parse::<u16>().ok().is_none_or(|p| p == 0)
|
||||
|| parts[3].len() != 32
|
||||
|| hex::decode(parts[3]).is_err()
|
||||
|| parts[4].parse::<EndpointId>().is_err()
|
||||
|| parts[5].parse::<u16>().ok().is_none_or(|p| p == 0)
|
||||
{
|
||||
return Err(invalid("invalid backend field key"));
|
||||
}
|
||||
if let Some(value) = &self.value {
|
||||
let b: Backend = postcard::from_bytes(value)
|
||||
.map_err(|_| invalid("invalid backend field"))?;
|
||||
if b.peer != parts[4] || b.port.to_string() != parts[5] {
|
||||
return Err(invalid("backend key does not match its value"));
|
||||
}
|
||||
}
|
||||
}
|
||||
_ => return Err(invalid("unknown VIP field")),
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
fn rank(&self) -> (u64, [u8; 32], Vec<u8>) {
|
||||
(self.clock, self.author, self.bytes())
|
||||
}
|
||||
}
|
||||
/// Signed registers retained across absences, including deletion tombstones.
|
||||
#[derive(Debug, Clone, Default)]
|
||||
pub struct VipSet {
|
||||
entries: BTreeMap<(String, String), VipEntry>,
|
||||
}
|
||||
impl VipSet {
|
||||
/// Highest retained version genuinely signed by this author.
|
||||
pub fn authored_version(&self, author: EndpointId) -> u64 {
|
||||
self.entries
|
||||
.values()
|
||||
.filter(|e| e.author == *author.as_bytes())
|
||||
.map(|e| e.version)
|
||||
.max()
|
||||
.unwrap_or(0)
|
||||
}
|
||||
/// Complete independently verifiable state of one service, including tombstones.
|
||||
pub fn entries(&self, id: &str) -> Vec<VipEntry> {
|
||||
self.entries
|
||||
.values()
|
||||
.filter(|e| e.vip == id)
|
||||
.cloned()
|
||||
.collect()
|
||||
}
|
||||
/// All retained service ids, including deleted services.
|
||||
pub fn ids(&self) -> Vec<String> {
|
||||
self.entries
|
||||
.keys()
|
||||
.map(|(id, _)| id.clone())
|
||||
.collect::<BTreeSet<_>>()
|
||||
.into_iter()
|
||||
.collect()
|
||||
}
|
||||
/// Opaque optimistic concurrency token for a whole service.
|
||||
pub fn revision(&self, id: &str) -> String {
|
||||
let mut hash = Sha256::new();
|
||||
for e in self.entries(id) {
|
||||
hash.update(e.bytes());
|
||||
hash.update(&e.signature);
|
||||
}
|
||||
hex::encode(hash.finalize())
|
||||
}
|
||||
/// Atomically verifies and merges one complete service snapshot.
|
||||
pub fn merge(
|
||||
&mut self,
|
||||
network: NetworkId,
|
||||
entries: Vec<VipEntry>,
|
||||
limits: &Limits,
|
||||
) -> Result<bool> {
|
||||
if entries.len() > limits.max_vip_fields {
|
||||
return Err(invalid("too many VIP fields"));
|
||||
}
|
||||
let mut next = self.clone();
|
||||
for e in entries {
|
||||
e.verify(network)?;
|
||||
let key = (e.vip.clone(), e.key.clone());
|
||||
if next
|
||||
.entries
|
||||
.get(&key)
|
||||
.is_none_or(|old| e.rank() > old.rank())
|
||||
{
|
||||
next.entries.insert(key, e);
|
||||
}
|
||||
}
|
||||
let deleted: BTreeSet<_> = next
|
||||
.entries
|
||||
.values()
|
||||
.filter(|e| e.key == "deleted" && e.value.is_some())
|
||||
.map(|e| e.vip.clone())
|
||||
.collect();
|
||||
// A permanent service tombstone alone prevents every former field's replay.
|
||||
next.entries
|
||||
.retain(|(id, key), _| !deleted.contains(id) || key == "deleted");
|
||||
let ids = next.ids();
|
||||
if ids.len().saturating_sub(deleted.len()) > limits.max_vips
|
||||
|| deleted.len() > limits.max_vip_tombstones
|
||||
|| ids
|
||||
.iter()
|
||||
.any(|id| next.entries(id).len() > limits.max_vip_fields)
|
||||
{
|
||||
return Err(invalid(
|
||||
"VIP state capacity exceeded; deletion protection is retained",
|
||||
));
|
||||
}
|
||||
let changed = self.entries != next.entries;
|
||||
*self = next;
|
||||
Ok(changed)
|
||||
}
|
||||
/// Materialized configuration. A service tombstone permanently retires its id.
|
||||
pub fn configs(&self) -> Vec<VipConfig> {
|
||||
self.ids()
|
||||
.into_iter()
|
||||
.filter_map(|id| self.config(&id))
|
||||
.collect()
|
||||
}
|
||||
fn config(&self, id: &str) -> Option<VipConfig> {
|
||||
let fields = self.entries(id);
|
||||
if fields
|
||||
.iter()
|
||||
.any(|e| e.key == "deleted" && e.value.is_some())
|
||||
{
|
||||
return None;
|
||||
}
|
||||
let name = fields
|
||||
.iter()
|
||||
.find(|e| e.key == "name")?
|
||||
.value
|
||||
.as_ref()
|
||||
.and_then(|v| postcard::from_bytes::<String>(v).ok())?;
|
||||
let mut listeners: Vec<Listener> = fields
|
||||
.iter()
|
||||
.filter(|e| e.key.starts_with("l/"))
|
||||
.filter_map(|e| e.value.as_ref().and_then(|v| postcard::from_bytes(v).ok()))
|
||||
.collect();
|
||||
for l in &mut listeners {
|
||||
l.backends = fields
|
||||
.iter()
|
||||
.filter(|e| {
|
||||
e.key
|
||||
.starts_with(&format!("b/{}/{}/{}/", l.protocol, l.port, l.id))
|
||||
})
|
||||
.filter_map(|e| e.value.as_ref().and_then(|v| postcard::from_bytes(v).ok()))
|
||||
.collect();
|
||||
l.backends.sort();
|
||||
}
|
||||
listeners.sort_by_key(|l| (l.protocol, l.port));
|
||||
Some(VipConfig {
|
||||
id: id.into(),
|
||||
name,
|
||||
listeners,
|
||||
})
|
||||
}
|
||||
/// Builds a signed patch from the user's complete, version-checked draft.
|
||||
pub fn edit(
|
||||
&self,
|
||||
network: NetworkId,
|
||||
secret: &SecretKey,
|
||||
version: u64,
|
||||
config: &VipConfig,
|
||||
expected: Option<&str>,
|
||||
limits: &Limits,
|
||||
) -> Result<Vec<VipEntry>> {
|
||||
config.validate(limits)?;
|
||||
if let Some(revision) = expected {
|
||||
if revision != self.revision(&config.id) {
|
||||
return Err(invalid("VIP changed while editing; reload before saving"));
|
||||
}
|
||||
} else if !self.entries(&config.id).is_empty() {
|
||||
return Err(invalid("VIP id already exists"));
|
||||
}
|
||||
if self
|
||||
.entries
|
||||
.get(&(config.id.clone(), "deleted".into()))
|
||||
.is_some_and(|e| e.value.is_some())
|
||||
{
|
||||
return Err(invalid("VIP was deleted; create a new VIP"));
|
||||
}
|
||||
let before = self
|
||||
.config(&config.id)
|
||||
.map(|c| flatten(&c))
|
||||
.transpose()?
|
||||
.unwrap_or_default();
|
||||
let after = flatten(config)?;
|
||||
let keys: BTreeSet<_> = before.keys().chain(after.keys()).cloned().collect();
|
||||
let changes = keys
|
||||
.into_iter()
|
||||
.filter_map(|key| {
|
||||
let value = after.get(&key).cloned();
|
||||
(before.get(&key) != value.as_ref()).then_some((key, value))
|
||||
})
|
||||
.collect();
|
||||
self.sign(network, secret, version, &config.id, changes)
|
||||
}
|
||||
/// Builds a permanent service tombstone after checking the selected revision.
|
||||
pub fn delete(
|
||||
&self,
|
||||
network: NetworkId,
|
||||
secret: &SecretKey,
|
||||
version: u64,
|
||||
id: &str,
|
||||
expected: &str,
|
||||
) -> Result<Vec<VipEntry>> {
|
||||
if self.config(id).is_none() || self.revision(id) != expected {
|
||||
return Err(invalid("VIP missing or changed while editing"));
|
||||
}
|
||||
self.sign(
|
||||
network,
|
||||
secret,
|
||||
version,
|
||||
id,
|
||||
vec![("deleted".into(), Some(vec![1]))],
|
||||
)
|
||||
}
|
||||
fn sign(
|
||||
&self,
|
||||
network: NetworkId,
|
||||
secret: &SecretKey,
|
||||
version: u64,
|
||||
id: &str,
|
||||
changes: Vec<(String, Option<Vec<u8>>)>,
|
||||
) -> Result<Vec<VipEntry>> {
|
||||
let clock = self
|
||||
.entries
|
||||
.values()
|
||||
.map(|e| e.clock)
|
||||
.max()
|
||||
.unwrap_or(0)
|
||||
.checked_add(1)
|
||||
.filter(|n| *n <= i64::MAX as u64)
|
||||
.ok_or_else(|| invalid("VIP clock exhausted"))?;
|
||||
Ok(changes
|
||||
.into_iter()
|
||||
.map(|(key, value)| {
|
||||
let mut e = VipEntry {
|
||||
network: *network.as_bytes(),
|
||||
vip: id.into(),
|
||||
key,
|
||||
value,
|
||||
clock,
|
||||
author: *secret.public().as_bytes(),
|
||||
version,
|
||||
signature: vec![],
|
||||
};
|
||||
e.signature = secret.sign(&e.bytes()).to_bytes().to_vec();
|
||||
e
|
||||
})
|
||||
.collect())
|
||||
}
|
||||
}
|
||||
pub(crate) fn flatten(cfg: &VipConfig) -> Result<BTreeMap<String, Vec<u8>>> {
|
||||
let mut fields = BTreeMap::new();
|
||||
fields.insert(
|
||||
"name".into(),
|
||||
postcard::to_stdvec(&cfg.name).map_err(|e| invalid(e.to_string()))?,
|
||||
);
|
||||
for l in &cfg.listeners {
|
||||
let mut settings = l.clone();
|
||||
settings.backends.clear();
|
||||
fields.insert(
|
||||
format!("l/{}/{}", l.protocol, l.port),
|
||||
postcard::to_stdvec(&settings).map_err(|e| invalid(e.to_string()))?,
|
||||
);
|
||||
for b in &l.backends {
|
||||
fields.insert(
|
||||
format!("b/{}/{}/{}/{}/{}", l.protocol, l.port, l.id, b.peer, b.port),
|
||||
postcard::to_stdvec(b).map_err(|e| invalid(e.to_string()))?,
|
||||
);
|
||||
}
|
||||
}
|
||||
Ok(fields)
|
||||
}
|
||||
impl VipSet {
|
||||
/// Allocates internal addresses deterministically against durable node claims.
|
||||
pub(crate) fn views(
|
||||
&self,
|
||||
range: Option<crate::state::Ipv4Range>,
|
||||
mut taken: std::collections::HashSet<Ipv4Addr>,
|
||||
hostnames: &BTreeSet<String>,
|
||||
limits: &Limits,
|
||||
) -> Vec<VipStatus> {
|
||||
let mut names = hostnames.clone();
|
||||
let mut result = Vec::new();
|
||||
for config in self.configs() {
|
||||
let mut error = config.validate(limits).err().map(|e| e.to_string());
|
||||
if !names.insert(config.name.clone()) {
|
||||
error = Some("name conflicts with another VIP or node".into());
|
||||
}
|
||||
let address = range.and_then(|r| {
|
||||
let usable = crate::state::allocator::usable_addresses(r);
|
||||
if usable == 0 {
|
||||
return None;
|
||||
}
|
||||
let digest = Sha256::digest(config.id.as_bytes());
|
||||
let seed = u64::from_be_bytes([
|
||||
digest[0], digest[1], digest[2], digest[3], digest[4], digest[5], digest[6],
|
||||
digest[7],
|
||||
]);
|
||||
let bits = 32 - u32::from(r.prefix_len);
|
||||
let mask = if bits == 32 { 0 } else { u32::MAX << bits };
|
||||
(0..usable).find_map(|step| {
|
||||
let offset = (seed.wrapping_add(step) % usable + 1) as u32;
|
||||
let ip = Ipv4Addr::from((u32::from(r.base) & mask) | offset);
|
||||
taken.insert(ip).then_some(ip)
|
||||
})
|
||||
});
|
||||
if address.is_none() {
|
||||
error = Some("no free overlay address".into());
|
||||
}
|
||||
result.push(VipStatus {
|
||||
revision: self.revision(&config.id),
|
||||
config,
|
||||
address,
|
||||
error,
|
||||
listeners: vec![],
|
||||
});
|
||||
}
|
||||
result
|
||||
}
|
||||
}
|
||||
@@ -254,6 +254,12 @@ impl Storage {
|
||||
.await
|
||||
}
|
||||
|
||||
/// Persists the ordered protocol selection off the async executor.
|
||||
pub async fn set_protocols(&self, network_id: NetworkId, protocols: Vec<String>) -> Result<()> {
|
||||
self.with_state(move |state| state.set_protocols(network_id, &protocols))
|
||||
.await
|
||||
}
|
||||
|
||||
/// Persists whether this agent offers itself as an exit node in a network.
|
||||
pub async fn set_exit_node(&self, network_id: NetworkId, enabled: bool) -> Result<()> {
|
||||
self.with_state(move |state| state.set_exit_node(network_id, enabled))
|
||||
@@ -272,11 +278,39 @@ impl Storage {
|
||||
.await
|
||||
}
|
||||
|
||||
/// Configured networks in the user's most recently selected order.
|
||||
pub async fn network_start_order(&self) -> Result<Vec<NetworkId>> {
|
||||
self.with_state(|state| {
|
||||
let configured = state.list_networks()?;
|
||||
let text = state
|
||||
.get_setting("network_start_order")?
|
||||
.unwrap_or_default();
|
||||
let mut order = Vec::new();
|
||||
for word in text.split(',') {
|
||||
if let Ok(id) = word.parse::<NetworkId>()
|
||||
&& configured.iter().any(|n| n.network_id == id)
|
||||
&& !order.contains(&id)
|
||||
{
|
||||
order.push(id);
|
||||
}
|
||||
}
|
||||
Ok(order)
|
||||
})
|
||||
.await
|
||||
}
|
||||
|
||||
/// Prefers this network at startup without disabling suspended networks.
|
||||
pub async fn prefer_network(&self, network: NetworkId) -> Result<()> {
|
||||
self.with_state(move |state| state.prefer_network(network))
|
||||
.await
|
||||
}
|
||||
|
||||
/// Removes a network configuration and its cached hints.
|
||||
pub async fn remove_network(&self, network_id: NetworkId) -> Result<()> {
|
||||
self.with_state(move |state| {
|
||||
state.remove_network(network_id)?;
|
||||
state.forget_signed_records(network_id)?;
|
||||
state.forget_retirements(network_id)?;
|
||||
state.forget_peer_hostnames(network_id)
|
||||
})
|
||||
.await?;
|
||||
@@ -336,6 +370,50 @@ impl Storage {
|
||||
.await
|
||||
}
|
||||
|
||||
/// Loads mandatory signed VIP fields with a strict capacity bound.
|
||||
pub async fn vip_fields(
|
||||
&self,
|
||||
network: NetworkId,
|
||||
max: usize,
|
||||
) -> Result<Vec<crate::vip::VipEntry>> {
|
||||
self.with_state(move |state| state.vip_fields(network, max))
|
||||
.await
|
||||
}
|
||||
/// Stores a complete merged service snapshot and optional own counter.
|
||||
pub async fn put_vip_fields(
|
||||
&self,
|
||||
network: NetworkId,
|
||||
entries: Vec<crate::vip::VipEntry>,
|
||||
own: Option<([u8; 32], u64)>,
|
||||
) -> Result<()> {
|
||||
self.with_state(move |state| state.put_vip_fields(network, &entries, own))
|
||||
.await
|
||||
}
|
||||
/// Loads verified durable cleanup proofs off the executor.
|
||||
pub async fn retirements(
|
||||
&self,
|
||||
network: NetworkId,
|
||||
max: usize,
|
||||
) -> Result<Vec<crate::state::SignedRetirement>> {
|
||||
self.with_state(move |state| state.retirements(network, max))
|
||||
.await
|
||||
}
|
||||
|
||||
/// Persists a received cleanup proof before applying it.
|
||||
pub async fn put_retirement(&self, marker: crate::state::SignedRetirement) -> Result<()> {
|
||||
self.with_state(move |state| state.put_retirement(&marker))
|
||||
.await
|
||||
}
|
||||
|
||||
/// Commits our cleanup and author counter together before announcement.
|
||||
pub async fn publish_own_retirement(
|
||||
&self,
|
||||
marker: crate::state::SignedRetirement,
|
||||
) -> Result<()> {
|
||||
self.with_state(move |state| state.publish_own_retirement(&marker))
|
||||
.await
|
||||
}
|
||||
|
||||
/// Stores a record received from another replica.
|
||||
pub async fn put_signed_record(&self, record: SignedRecord) -> Result<()> {
|
||||
self.with_state(move |state| state.put_signed_record(&record))
|
||||
|
||||
@@ -18,10 +18,10 @@ use rusqlite::{Connection, OptionalExtension, params};
|
||||
|
||||
use crate::error::{Error, Result};
|
||||
use crate::identity::{DeviceIdentity, NetworkId, NetworkName, NetworkSecret};
|
||||
use crate::state::{RecordBody, SignedRecord};
|
||||
use crate::state::{MAX_RETIREMENT_LEN, RecordBody, SignedRecord, SignedRetirement};
|
||||
|
||||
/// Schema version written by this build.
|
||||
pub const SCHEMA_VERSION: i64 = 6;
|
||||
pub const SCHEMA_VERSION: i64 = 10;
|
||||
|
||||
/// Key of the stored hostname setting.
|
||||
const SETTING_HOSTNAME: &str = "hostname";
|
||||
@@ -32,6 +32,8 @@ const SETTING_HOSTNAME: &str = "hostname";
|
||||
/// and zeroizes on drop.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct StoredNetwork {
|
||||
/// Ordered local protocol override; absent inherits the agent's defaults.
|
||||
pub protocols: Option<Vec<String>>,
|
||||
/// Derived public network identifier.
|
||||
pub network_id: NetworkId,
|
||||
/// Network name.
|
||||
@@ -221,6 +223,52 @@ impl StateStore {
|
||||
)
|
||||
.map_err(|err| self.corrupt(format!("cannot migrate schema to 6: {err}")))?;
|
||||
}
|
||||
// Migration 6 -> 7: a claim gained the time its hostname was first
|
||||
// claimed, which changed the signing domain. Records signed before
|
||||
// no longer verify, so they are discarded rather than kept as rows
|
||||
// every read has to reject; each member re-publishes its own claim on
|
||||
// its next run and the others arrive again from their authors. The
|
||||
// version counter is kept, so nothing is ever published twice at one
|
||||
// version.
|
||||
if found < 7 {
|
||||
self.conn
|
||||
.execute_batch(
|
||||
"BEGIN;
|
||||
DELETE FROM signed_records;
|
||||
PRAGMA user_version = 7;
|
||||
COMMIT;",
|
||||
)
|
||||
.map_err(|err| self.corrupt(format!("cannot migrate schema to 7: {err}")))?;
|
||||
}
|
||||
if found < 8 {
|
||||
self.conn
|
||||
.execute_batch(
|
||||
"BEGIN;
|
||||
ALTER TABLE networks ADD COLUMN protocols TEXT;
|
||||
PRAGMA user_version = 8;
|
||||
COMMIT;",
|
||||
)
|
||||
.map_err(|err| self.corrupt(format!("cannot migrate schema to 8: {err}")))?;
|
||||
}
|
||||
if found < 9 {
|
||||
self.conn
|
||||
.execute_batch(
|
||||
"BEGIN;
|
||||
CREATE TABLE IF NOT EXISTS retirements (
|
||||
network_id BLOB NOT NULL, target BLOB NOT NULL, data BLOB NOT NULL,
|
||||
PRIMARY KEY (network_id, target)
|
||||
);
|
||||
PRAGMA user_version = 9;
|
||||
COMMIT;",
|
||||
)
|
||||
.map_err(|err| self.corrupt(format!("cannot migrate schema to 9: {err}")))?;
|
||||
}
|
||||
if found < 10 {
|
||||
self.conn.execute_batch("BEGIN; CREATE TABLE IF NOT EXISTS vip_fields (
|
||||
network_id BLOB NOT NULL, vip TEXT NOT NULL, field TEXT NOT NULL, data BLOB NOT NULL,
|
||||
PRIMARY KEY(network_id, vip, field)); PRAGMA user_version = 10; COMMIT;")
|
||||
.map_err(|err| self.corrupt(format!("cannot migrate VIP state: {err}")))?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -233,6 +281,8 @@ impl StateStore {
|
||||
"settings",
|
||||
"signed_records",
|
||||
"peer_hostnames",
|
||||
"retirements",
|
||||
"vip_fields",
|
||||
] {
|
||||
let present: Option<String> = self
|
||||
.conn
|
||||
@@ -431,12 +481,16 @@ impl StateStore {
|
||||
|
||||
/// Sets the auto-start flag of a configured network.
|
||||
pub fn set_auto_start(&self, network_id: NetworkId, auto_start: bool) -> Result<()> {
|
||||
self.conn
|
||||
let changed = self
|
||||
.conn
|
||||
.execute(
|
||||
"UPDATE networks SET auto_start = ?2 WHERE network_id = ?1",
|
||||
params![network_id.as_bytes().as_slice(), auto_start as i64],
|
||||
)
|
||||
.map_err(|err| Error::Storage(format!("cannot update network: {err}")))?;
|
||||
if changed == 0 {
|
||||
return Err(Error::NetworkUnknown(network_id));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -455,6 +509,25 @@ impl StateStore {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Saves a local ordered protocol selection without changing membership.
|
||||
pub fn set_protocols(&self, network_id: NetworkId, protocols: &[String]) -> Result<()> {
|
||||
crate::dataplane::validate_protocols(
|
||||
protocols,
|
||||
crate::config::Limits::default().max_capabilities,
|
||||
)?;
|
||||
let changed = self
|
||||
.conn
|
||||
.execute(
|
||||
"UPDATE networks SET protocols = ?2 WHERE network_id = ?1",
|
||||
params![network_id.as_bytes().as_slice(), protocols.join(",")],
|
||||
)
|
||||
.map_err(|err| Error::Storage(format!("cannot update protocols: {err}")))?;
|
||||
if changed == 0 {
|
||||
return Err(Error::NetworkUnknown(network_id));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Removes a network configuration entirely.
|
||||
pub fn remove_network(&self, network_id: NetworkId) -> Result<()> {
|
||||
self.conn
|
||||
@@ -515,7 +588,7 @@ impl StateStore {
|
||||
pub fn list_networks(&self) -> Result<Vec<StoredNetwork>> {
|
||||
let mut stmt = self
|
||||
.conn
|
||||
.prepare("SELECT network_id, name, secret, auto_start, broadcast, exit_node, exit_via FROM networks ORDER BY name")
|
||||
.prepare("SELECT network_id, name, secret, auto_start, broadcast, exit_node, exit_via, protocols FROM networks ORDER BY name")
|
||||
.map_err(|err| Error::Storage(format!("cannot list networks: {err}")))?;
|
||||
let rows = stmt
|
||||
.query_map([], |row| {
|
||||
@@ -531,19 +604,37 @@ impl StateStore {
|
||||
row.get::<_, bool>(4)?,
|
||||
row.get::<_, bool>(5)?,
|
||||
row.get::<_, Option<Vec<u8>>>(6)?,
|
||||
row.get::<_, Option<String>>(7)?,
|
||||
))
|
||||
})
|
||||
.map_err(|err| Error::Storage(format!("cannot list networks: {err}")))?;
|
||||
|
||||
let mut out = Vec::new();
|
||||
for row in rows {
|
||||
let (id, name, secret, auto_start, broadcast, exit_node, exit_via) =
|
||||
let (id, name, secret, auto_start, broadcast, exit_node, exit_via, protocols) =
|
||||
row.map_err(|err| Error::Storage(format!("cannot read network row: {err}")))?;
|
||||
let protocols = protocols
|
||||
.map(|text| {
|
||||
let max = crate::config::Limits::default().max_capabilities;
|
||||
if text.len() > max * (crate::dataplane::MAX_PROTOCOL_ID_LEN + 1) {
|
||||
return Err(self.corrupt("stored protocol selection is too long"));
|
||||
}
|
||||
let names = if text.is_empty() {
|
||||
Vec::new()
|
||||
} else {
|
||||
text.split(',').map(str::to_owned).collect()
|
||||
};
|
||||
crate::dataplane::validate_protocols(&names, max)
|
||||
.map_err(|err| self.corrupt(err))?;
|
||||
Ok(names)
|
||||
})
|
||||
.transpose()?;
|
||||
let id: [u8; 32] = id
|
||||
.as_slice()
|
||||
.try_into()
|
||||
.map_err(|_| self.corrupt("stored network id is not 32 bytes"))?;
|
||||
out.push(StoredNetwork {
|
||||
protocols,
|
||||
network_id: NetworkId::from_bytes(id),
|
||||
name: NetworkName::new(name)?,
|
||||
secret: NetworkSecret::from_bytes(secret)?,
|
||||
@@ -715,6 +806,181 @@ impl StateStore {
|
||||
Ok(version.unwrap_or(0).max(0) as u64)
|
||||
}
|
||||
|
||||
/// Loads durable cleanup markers without allocating oversized stored blobs.
|
||||
pub fn retirements(&self, network: NetworkId, max: usize) -> Result<Vec<SignedRetirement>> {
|
||||
let mut stmt = self
|
||||
.conn
|
||||
.prepare(
|
||||
"SELECT CASE WHEN length(data) <= ?2 THEN data ELSE NULL END
|
||||
FROM retirements WHERE network_id = ?1 ORDER BY target",
|
||||
)
|
||||
.map_err(|err| self.corrupt(err))?;
|
||||
let rows = stmt
|
||||
.query_map(
|
||||
params![network.as_bytes().as_slice(), MAX_RETIREMENT_LEN as i64],
|
||||
|row| row.get::<_, Option<Vec<u8>>>(0),
|
||||
)
|
||||
.map_err(|err| self.corrupt(err))?;
|
||||
let mut out = Vec::new();
|
||||
for row in rows {
|
||||
if out.len() >= max {
|
||||
return Err(Error::Storage(
|
||||
"stored retirements exceed the configured limit".into(),
|
||||
));
|
||||
}
|
||||
let bytes = row
|
||||
.map_err(|err| self.corrupt(err))?
|
||||
.ok_or_else(|| self.corrupt("oversized retirement"))?;
|
||||
let marker: SignedRetirement =
|
||||
postcard::from_bytes(&bytes).map_err(|err| self.corrupt(err))?;
|
||||
marker.verify(network).map_err(|err| self.corrupt(err))?;
|
||||
out.push(marker);
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
/// Persists a strongest verified marker received from an authenticated session.
|
||||
pub fn put_retirement(&self, marker: &SignedRetirement) -> Result<()> {
|
||||
let network = NetworkId::from_bytes(marker.network);
|
||||
marker
|
||||
.verify(network)
|
||||
.map_err(|err| Error::Storage(err.to_string()))?;
|
||||
let old: Option<Option<Vec<u8>>> = self.conn.query_row(
|
||||
"SELECT CASE WHEN length(data) <= ?3 THEN data ELSE NULL END FROM retirements WHERE network_id = ?1 AND target = ?2",
|
||||
params![marker.network.as_slice(), marker.target.author.as_slice(), MAX_RETIREMENT_LEN as i64], |row| row.get(0))
|
||||
.optional().map_err(|err| self.corrupt(err))?;
|
||||
if let Some(bytes) = old {
|
||||
let bytes = bytes.ok_or_else(|| self.corrupt("oversized retirement"))?;
|
||||
let old: SignedRetirement =
|
||||
postcard::from_bytes(&bytes).map_err(|err| self.corrupt(err))?;
|
||||
old.verify(network).map_err(|err| self.corrupt(err))?;
|
||||
if !marker.supersedes(&old) {
|
||||
return Ok(());
|
||||
}
|
||||
}
|
||||
let bytes = postcard::to_stdvec(marker).map_err(|err| Error::Storage(err.to_string()))?;
|
||||
if bytes.len() > MAX_RETIREMENT_LEN {
|
||||
return Err(Error::Storage(
|
||||
"retirement exceeds its encoded bound".into(),
|
||||
));
|
||||
}
|
||||
self.conn
|
||||
.execute(
|
||||
"INSERT INTO retirements (network_id, target, data) VALUES (?1, ?2, ?3)
|
||||
ON CONFLICT(network_id, target) DO UPDATE SET data = excluded.data",
|
||||
params![
|
||||
marker.network.as_slice(),
|
||||
marker.target.author.as_slice(),
|
||||
bytes
|
||||
],
|
||||
)
|
||||
.map_err(|err| Error::Storage(format!("cannot store retirement: {err}")))?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Commits our signed cleanup marker and author counter in one transaction.
|
||||
pub fn publish_own_retirement(&self, marker: &SignedRetirement) -> Result<()> {
|
||||
let transaction = self
|
||||
.conn
|
||||
.unchecked_transaction()
|
||||
.map_err(|err| Error::Storage(err.to_string()))?;
|
||||
self.put_retirement(marker)?;
|
||||
transaction
|
||||
.execute(
|
||||
"INSERT INTO own_record_version (network_id, author, version) VALUES (?1, ?2, ?3)
|
||||
ON CONFLICT(network_id, author) DO UPDATE SET version = max(version, excluded.version)",
|
||||
params![
|
||||
marker.network.as_slice(),
|
||||
marker.author.as_slice(),
|
||||
marker.version as i64
|
||||
],
|
||||
)
|
||||
.map_err(|err| Error::Storage(format!("cannot store retirement counter: {err}")))?;
|
||||
transaction
|
||||
.commit()
|
||||
.map_err(|err| Error::Storage(format!("cannot commit retirement: {err}")))
|
||||
}
|
||||
|
||||
/// Reads bounded mandatory VIP state; corrupt records fail activation.
|
||||
pub fn vip_fields(&self, network: NetworkId, max: usize) -> Result<Vec<crate::vip::VipEntry>> {
|
||||
let mut stmt = self.conn.prepare("SELECT CASE WHEN length(data) <= 1024 THEN data ELSE NULL END FROM vip_fields WHERE network_id=?1 ORDER BY vip, field")
|
||||
.map_err(|e| self.corrupt(e.to_string()))?;
|
||||
let mut rows = stmt
|
||||
.query(params![network.as_bytes().as_slice()])
|
||||
.map_err(|e| self.corrupt(e.to_string()))?;
|
||||
let mut entries = Vec::new();
|
||||
while let Some(row) = rows.next().map_err(|e| self.corrupt(e.to_string()))? {
|
||||
if entries.len() >= max {
|
||||
return Err(self.corrupt("stored VIP state exceeds configured limits"));
|
||||
}
|
||||
let bytes: Option<Vec<u8>> = row.get(0).map_err(|e| self.corrupt(e.to_string()))?;
|
||||
let bytes = bytes.ok_or_else(|| self.corrupt("oversized VIP field"))?;
|
||||
let entry: crate::vip::VipEntry =
|
||||
postcard::from_bytes(&bytes).map_err(|e| self.corrupt(e.to_string()))?;
|
||||
entry
|
||||
.verify(network)
|
||||
.map_err(|e| self.corrupt(e.to_string()))?;
|
||||
entries.push(entry);
|
||||
}
|
||||
Ok(entries)
|
||||
}
|
||||
|
||||
/// Commits a validated VIP snapshot and optional author counter atomically.
|
||||
pub fn put_vip_fields(
|
||||
&self,
|
||||
network: NetworkId,
|
||||
entries: &[crate::vip::VipEntry],
|
||||
own: Option<([u8; 32], u64)>,
|
||||
) -> Result<()> {
|
||||
let tx = self
|
||||
.conn
|
||||
.unchecked_transaction()
|
||||
.map_err(|e| Error::Storage(e.to_string()))?;
|
||||
if let Some(first) = entries.first() {
|
||||
if entries.iter().any(|e| e.vip != first.vip) {
|
||||
return Err(Error::Storage(
|
||||
"VIP snapshot mixes service identities".into(),
|
||||
));
|
||||
}
|
||||
tx.execute(
|
||||
"DELETE FROM vip_fields WHERE network_id=?1 AND vip=?2",
|
||||
params![network.as_bytes().as_slice(), first.vip],
|
||||
)
|
||||
.map_err(|e| Error::Storage(e.to_string()))?;
|
||||
}
|
||||
for e in entries {
|
||||
e.verify(network)?;
|
||||
let bytes = postcard::to_stdvec(e).map_err(|e| Error::Storage(e.to_string()))?;
|
||||
if bytes.len() > 1024 {
|
||||
return Err(Error::Storage("oversized VIP field".into()));
|
||||
}
|
||||
tx.execute("INSERT INTO vip_fields(network_id,vip,field,data) VALUES(?1,?2,?3,?4) ON CONFLICT(network_id,vip,field) DO UPDATE SET data=excluded.data",
|
||||
params![network.as_bytes().as_slice(),e.vip,e.key,bytes]).map_err(|e|Error::Storage(e.to_string()))?;
|
||||
}
|
||||
if let Some((author, version)) = own {
|
||||
tx.execute("INSERT INTO own_record_version(network_id,author,version) VALUES(?1,?2,?3) ON CONFLICT(network_id,author) DO UPDATE SET version=max(version,excluded.version)",
|
||||
params![network.as_bytes().as_slice(),author.as_slice(),version as i64]).map_err(|e|Error::Storage(e.to_string()))?;
|
||||
}
|
||||
tx.commit().map_err(|e| Error::Storage(e.to_string()))
|
||||
}
|
||||
|
||||
/// Drops a forgotten network's local cleanup proofs.
|
||||
pub fn forget_retirements(&self, network: NetworkId) -> Result<()> {
|
||||
self.conn
|
||||
.execute(
|
||||
"DELETE FROM vip_fields WHERE network_id = ?1",
|
||||
params![network.as_bytes().as_slice()],
|
||||
)
|
||||
.map_err(|e| Error::Storage(e.to_string()))?;
|
||||
self.conn
|
||||
.execute(
|
||||
"DELETE FROM retirements WHERE network_id = ?1",
|
||||
params![network.as_bytes().as_slice()],
|
||||
)
|
||||
.map_err(|err| Error::Storage(err.to_string()))?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Forgets every record of a network.
|
||||
pub fn forget_signed_records(&self, network_id: NetworkId) -> Result<()> {
|
||||
self.conn
|
||||
@@ -781,6 +1047,44 @@ impl StateStore {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Atomically enables a network and puts it first in the startup order.
|
||||
pub fn prefer_network(&self, network: NetworkId) -> Result<()> {
|
||||
let configured = self.list_networks()?;
|
||||
if !configured.iter().any(|n| n.network_id == network) {
|
||||
return Err(Error::NetworkUnknown(network));
|
||||
}
|
||||
let existing = self.get_setting("network_start_order")?.unwrap_or_default();
|
||||
let mut order = vec![network];
|
||||
for word in existing.split(',') {
|
||||
if let Ok(id) = word.parse::<NetworkId>()
|
||||
&& configured.iter().any(|n| n.network_id == id)
|
||||
&& !order.contains(&id)
|
||||
{
|
||||
order.push(id);
|
||||
}
|
||||
}
|
||||
let value = order
|
||||
.iter()
|
||||
.map(ToString::to_string)
|
||||
.collect::<Vec<_>>()
|
||||
.join(",");
|
||||
let transaction = self
|
||||
.conn
|
||||
.unchecked_transaction()
|
||||
.map_err(|err| Error::Storage(format!("cannot begin network preference: {err}")))?;
|
||||
transaction
|
||||
.execute(
|
||||
"UPDATE networks SET auto_start = 1 WHERE network_id = ?1",
|
||||
params![network.as_bytes().as_slice()],
|
||||
)
|
||||
.map_err(|err| Error::Storage(format!("cannot enable network: {err}")))?;
|
||||
transaction.execute("INSERT INTO settings (key, value) VALUES ('network_start_order', ?1) ON CONFLICT(key) DO UPDATE SET value = excluded.value", params![value])
|
||||
.map_err(|err| Error::Storage(format!("cannot store network preference: {err}")))?;
|
||||
transaction
|
||||
.commit()
|
||||
.map_err(|err| Error::Storage(format!("cannot commit network preference: {err}")))
|
||||
}
|
||||
|
||||
/// Reads an arbitrary setting.
|
||||
pub fn get_setting(&self, key: &str) -> Result<Option<String>> {
|
||||
self.conn
|
||||
@@ -898,6 +1202,158 @@ mod broadcast_storage_tests {
|
||||
assert!(store.peer_hostnames(id).unwrap().is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn retirement_migration_and_atomic_counter_commit_preserve_existing_state() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let path = dir.path().join("state.sqlite");
|
||||
let store = StateStore::open(&path).unwrap();
|
||||
let identity = store.load_or_create_device_identity().unwrap();
|
||||
let name = NetworkName::new("retirement-migration").unwrap();
|
||||
let secret = NetworkSecret::from_bytes([6; 32]).unwrap();
|
||||
let id = NetworkKeys::derive(&name, &secret).network_id();
|
||||
store.upsert_network(id, &name, &secret, true).unwrap();
|
||||
store.set_protocols(id, &["awg".into()]).unwrap();
|
||||
let own = SignedRecord::sign(&identity.signing_key(), id, 8, RecordBody::Release);
|
||||
store.publish_own_record(&own).unwrap();
|
||||
store
|
||||
.conn
|
||||
.execute_batch("DROP TABLE retirements; PRAGMA user_version = 8;")
|
||||
.unwrap();
|
||||
drop(store);
|
||||
let store = StateStore::open(&path).unwrap();
|
||||
assert_eq!(
|
||||
store.device_identity().unwrap().unwrap().endpoint_id(),
|
||||
identity.endpoint_id()
|
||||
);
|
||||
assert_eq!(
|
||||
store.list_networks().unwrap()[0].protocols,
|
||||
Some(vec!["awg".into()])
|
||||
);
|
||||
assert_eq!(store.signed_records(id).unwrap(), vec![own]);
|
||||
let target_key = iroh::SecretKey::generate();
|
||||
let target = SignedRecord::sign(
|
||||
&target_key,
|
||||
id,
|
||||
3,
|
||||
RecordBody::Claim {
|
||||
address: Some("10.13.37.2".parse().unwrap()),
|
||||
range: Some(crate::state::DEFAULT_IPV4_RANGE),
|
||||
hostname: Some("old-device".into()),
|
||||
hostname_since: Some(123),
|
||||
},
|
||||
);
|
||||
let marker = SignedRetirement::sign(&identity.signing_key(), id, 9, target);
|
||||
store
|
||||
.conn
|
||||
.execute_batch(
|
||||
"CREATE TRIGGER fail_retirement_counter BEFORE INSERT ON own_record_version
|
||||
WHEN NEW.version = 9 BEGIN SELECT RAISE(ABORT, 'counter write refused'); END;",
|
||||
)
|
||||
.unwrap();
|
||||
assert!(store.publish_own_retirement(&marker).is_err());
|
||||
assert!(
|
||||
store.retirements(id, 16).unwrap().is_empty(),
|
||||
"marker must roll back with counter failure"
|
||||
);
|
||||
assert_eq!(
|
||||
store
|
||||
.own_record_version(id, identity.endpoint_id())
|
||||
.unwrap(),
|
||||
8
|
||||
);
|
||||
store
|
||||
.conn
|
||||
.execute_batch("DROP TRIGGER fail_retirement_counter;")
|
||||
.unwrap();
|
||||
store.publish_own_retirement(&marker).unwrap();
|
||||
drop(store);
|
||||
let store = StateStore::open(&path).unwrap();
|
||||
assert_eq!(store.retirements(id, 16).unwrap(), vec![marker.clone()]);
|
||||
assert_eq!(
|
||||
store
|
||||
.own_record_version(id, identity.endpoint_id())
|
||||
.unwrap(),
|
||||
9
|
||||
);
|
||||
assert!(
|
||||
store.retirements(id, 0).is_err(),
|
||||
"capacity changes must not silently drop markers"
|
||||
);
|
||||
store
|
||||
.conn
|
||||
.execute(
|
||||
"UPDATE retirements SET data = ?1",
|
||||
params![vec![0u8; MAX_RETIREMENT_LEN + 1]],
|
||||
)
|
||||
.unwrap();
|
||||
assert!(matches!(
|
||||
store.retirements(id, 16),
|
||||
Err(Error::StateCorrupted { .. })
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn v7_migration_keeps_signed_state_and_protocol_overrides_are_local_and_durable() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let path = dir.path().join("state.sqlite");
|
||||
let store = StateStore::open(&path).unwrap();
|
||||
let identity = store.load_or_create_device_identity().unwrap();
|
||||
let name = NetworkName::new("protocol-migration").unwrap();
|
||||
let secret = NetworkSecret::from_bytes([4; 32]).unwrap();
|
||||
let id = NetworkKeys::derive(&name, &secret).network_id();
|
||||
store.upsert_network(id, &name, &secret, true).unwrap();
|
||||
store.set_broadcast(id, false).unwrap();
|
||||
let record = SignedRecord::sign(&identity.signing_key(), id, 17, RecordBody::Release);
|
||||
store.publish_own_record(&record).unwrap();
|
||||
store
|
||||
.conn
|
||||
.execute_batch("ALTER TABLE networks DROP COLUMN protocols; PRAGMA user_version=7;")
|
||||
.unwrap();
|
||||
drop(store);
|
||||
let store = StateStore::open(&path).unwrap();
|
||||
assert_eq!(
|
||||
store
|
||||
.load_or_create_device_identity()
|
||||
.unwrap()
|
||||
.endpoint_id(),
|
||||
identity.endpoint_id()
|
||||
);
|
||||
assert_eq!(store.signed_records(id).unwrap(), vec![record]);
|
||||
let saved = store.list_networks().unwrap();
|
||||
assert!(saved[0].protocols.is_none());
|
||||
assert!(!saved[0].broadcast);
|
||||
assert!(saved[0].auto_start);
|
||||
assert_eq!(saved[0].secret, secret);
|
||||
let chosen = vec!["awg".into(), "wg-quic".into()];
|
||||
store.set_protocols(id, &chosen).unwrap();
|
||||
assert!(
|
||||
store
|
||||
.set_protocols(id, &["awg".into(), "awg".into()])
|
||||
.is_err()
|
||||
);
|
||||
assert!(
|
||||
store
|
||||
.set_protocols(NetworkId::from_bytes([1; 32]), &[])
|
||||
.is_err()
|
||||
);
|
||||
drop(store);
|
||||
let store = StateStore::open(&path).unwrap();
|
||||
assert_eq!(store.list_networks().unwrap()[0].protocols, Some(chosen));
|
||||
store.upsert_network(id, &name, &secret, true).unwrap();
|
||||
store.set_protocols(id, &[]).unwrap();
|
||||
drop(store);
|
||||
let store = StateStore::open(&path).unwrap();
|
||||
assert_eq!(store.list_networks().unwrap()[0].protocols, Some(vec![]));
|
||||
store
|
||||
.conn
|
||||
.execute("UPDATE networks SET protocols = 'awg,awg'", [])
|
||||
.unwrap();
|
||||
assert!(matches!(
|
||||
store.list_networks(),
|
||||
Err(Error::StateCorrupted { .. })
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn v3_migration_preserves_identity_and_networks_and_defaults_broadcast_on() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
@@ -918,6 +1374,7 @@ mod broadcast_storage_tests {
|
||||
"ALTER TABLE networks DROP COLUMN broadcast; \
|
||||
ALTER TABLE networks DROP COLUMN exit_node; \
|
||||
ALTER TABLE networks DROP COLUMN exit_via; \
|
||||
ALTER TABLE networks DROP COLUMN protocols; \
|
||||
PRAGMA user_version=3;",
|
||||
)
|
||||
.unwrap();
|
||||
@@ -952,3 +1409,59 @@ mod broadcast_storage_tests {
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod vip_tests {
|
||||
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
|
||||
use super::*;
|
||||
#[test]
|
||||
fn vip_migration_and_counter_failure_preserve_identity_and_rollback_fields() {
|
||||
let dir = tempfile::TempDir::new().unwrap();
|
||||
let path = dir.path().join("state.sqlite");
|
||||
let store = StateStore::open(&path).unwrap();
|
||||
let identity = store.load_or_create_device_identity().unwrap();
|
||||
store
|
||||
.conn
|
||||
.execute_batch("DROP TABLE vip_fields; PRAGMA user_version=9;")
|
||||
.unwrap();
|
||||
drop(store);
|
||||
let store = StateStore::open(&path).unwrap();
|
||||
assert_eq!(
|
||||
store.device_identity().unwrap().unwrap().endpoint_id(),
|
||||
identity.endpoint_id()
|
||||
);
|
||||
let net = NetworkId::from_bytes([7; 32]);
|
||||
let key = identity.signing_key();
|
||||
let config = crate::vip::VipConfig::new("api");
|
||||
let fields = crate::vip::VipSet::default()
|
||||
.edit(net, &key, 7, &config, None, &crate::Limits::default())
|
||||
.unwrap();
|
||||
store.conn.execute_batch("CREATE TRIGGER fail_vip_counter BEFORE INSERT ON own_record_version BEGIN SELECT RAISE(ABORT,'counter unavailable'); END;").unwrap();
|
||||
assert!(
|
||||
store
|
||||
.put_vip_fields(net, &fields, Some((*key.public().as_bytes(), 7)))
|
||||
.is_err()
|
||||
);
|
||||
assert!(store.vip_fields(net, 192).unwrap().is_empty());
|
||||
store
|
||||
.conn
|
||||
.execute_batch("DROP TRIGGER fail_vip_counter;")
|
||||
.unwrap();
|
||||
store
|
||||
.put_vip_fields(net, &fields, Some((*key.public().as_bytes(), 7)))
|
||||
.unwrap();
|
||||
assert_eq!(store.vip_fields(net, 192).unwrap(), fields);
|
||||
assert_eq!(store.own_record_version(net, key.public()).unwrap(), 7);
|
||||
drop(store);
|
||||
let reopened = StateStore::open(path).unwrap();
|
||||
assert_eq!(reopened.vip_fields(net, 192).unwrap(), fields);
|
||||
reopened
|
||||
.conn
|
||||
.execute("UPDATE vip_fields SET data=?1", params![vec![0xffu8; 1025]])
|
||||
.unwrap();
|
||||
assert!(matches!(
|
||||
reopened.vip_fields(net, 192),
|
||||
Err(Error::StateCorrupted { .. })
|
||||
));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,239 @@
|
||||
//! Named, shared virtual services. Signed field registers converge without an
|
||||
//! owner; routing, health and connection state remain local to each client.
|
||||
use crate::Limits;
|
||||
use crate::error::{Error, Result};
|
||||
use iroh::EndpointId;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::collections::BTreeSet;
|
||||
use std::net::Ipv4Addr;
|
||||
pub(crate) mod packet;
|
||||
pub(crate) mod router;
|
||||
pub use crate::state::vip::{VipEntry, VipSet};
|
||||
pub(crate) use router::VipRouter;
|
||||
#[cfg(test)]
|
||||
mod tests;
|
||||
|
||||
/// IP transport carried by a listener.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "lowercase")]
|
||||
pub enum Protocol {
|
||||
/// TCP.
|
||||
Tcp,
|
||||
/// UDP.
|
||||
Udp,
|
||||
}
|
||||
impl std::fmt::Display for Protocol {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
f.write_str(match self {
|
||||
Self::Tcp => "tcp",
|
||||
Self::Udp => "udp",
|
||||
})
|
||||
}
|
||||
}
|
||||
impl std::str::FromStr for Protocol {
|
||||
type Err = String;
|
||||
fn from_str(s: &str) -> std::result::Result<Self, String> {
|
||||
match s {
|
||||
"tcp" => Ok(Self::Tcp),
|
||||
"udp" => Ok(Self::Udp),
|
||||
_ => Err("protocol must be tcp or udp".into()),
|
||||
}
|
||||
}
|
||||
}
|
||||
/// A node identity and its application port.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct Backend {
|
||||
/// Full device identity, resolved when editing.
|
||||
pub peer: String,
|
||||
/// Destination port.
|
||||
pub port: u16,
|
||||
}
|
||||
/// One independently routed port of a VIP.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct Listener {
|
||||
/// Stable listener generation; prevents removed backend fields resurfacing on re-creation.
|
||||
pub id: String,
|
||||
/// TCP or UDP.
|
||||
pub protocol: Protocol,
|
||||
/// Client-facing port.
|
||||
pub port: u16,
|
||||
/// Enable active TCP connect checks; UDP uses path checks and ICMP errors.
|
||||
pub port_check: bool,
|
||||
/// Period between checks in milliseconds.
|
||||
pub check_interval_ms: u32,
|
||||
/// Deadline of a port check in milliseconds.
|
||||
pub check_timeout_ms: u32,
|
||||
/// Consecutive failed checks before exclusion.
|
||||
pub failure_threshold: u8,
|
||||
/// Consecutive successes before recovery.
|
||||
pub recovery_threshold: u8,
|
||||
/// Idle expiry for this listener's flows, in seconds.
|
||||
pub idle_timeout_secs: u32,
|
||||
/// Allowed destination nodes and ports.
|
||||
pub backends: Vec<Backend>,
|
||||
}
|
||||
impl Listener {
|
||||
/// Creates a listener with automatic recovery and bounded checks.
|
||||
pub fn new(protocol: Protocol, port: u16) -> Self {
|
||||
Self {
|
||||
id: hex::encode(rand::random::<[u8; 16]>()),
|
||||
protocol,
|
||||
port,
|
||||
port_check: true,
|
||||
check_interval_ms: 3000,
|
||||
check_timeout_ms: 2000,
|
||||
failure_threshold: 2,
|
||||
recovery_threshold: 1,
|
||||
idle_timeout_secs: if protocol == Protocol::Tcp { 7200 } else { 120 },
|
||||
backends: vec![],
|
||||
}
|
||||
}
|
||||
}
|
||||
/// Entire editable service; its address is an internal derived reservation.
|
||||
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct VipConfig {
|
||||
/// Stable random 128-bit id in hexadecimal.
|
||||
pub id: String,
|
||||
/// One DNS label, unique within the network.
|
||||
pub name: String,
|
||||
/// Independent listeners.
|
||||
pub listeners: Vec<Listener>,
|
||||
}
|
||||
impl VipConfig {
|
||||
/// Creates an empty named service with a fresh stable identity.
|
||||
pub fn new(name: impl Into<String>) -> Self {
|
||||
Self {
|
||||
id: hex::encode(rand::random::<[u8; 16]>()),
|
||||
name: name.into(),
|
||||
listeners: vec![],
|
||||
}
|
||||
}
|
||||
/// Validates a complete draft before any state is changed.
|
||||
pub fn validate(&self, limits: &Limits) -> Result<()> {
|
||||
if self.id.len() != 32 || hex::decode(&self.id).is_err() {
|
||||
return Err(invalid("invalid VIP id"));
|
||||
}
|
||||
if self.name.is_empty()
|
||||
|| self.name.len() > 63
|
||||
|| self.name.starts_with('-')
|
||||
|| self.name.ends_with('-')
|
||||
|| !self
|
||||
.name
|
||||
.bytes()
|
||||
.all(|b| b.is_ascii_lowercase() || b.is_ascii_digit() || b == b'-')
|
||||
{
|
||||
return Err(invalid(
|
||||
"VIP name must be a lowercase DNS label (letters, digits and hyphens)",
|
||||
));
|
||||
}
|
||||
if self.listeners.len() > limits.max_vip_listeners {
|
||||
return Err(invalid("too many VIP listeners"));
|
||||
}
|
||||
let mut seen = BTreeSet::new();
|
||||
for l in &self.listeners {
|
||||
if l.id.len() != 32 || hex::decode(&l.id).is_err() {
|
||||
return Err(invalid("invalid listener identity"));
|
||||
}
|
||||
if l.port == 0 || !seen.insert((l.protocol, l.port)) {
|
||||
return Err(invalid(
|
||||
"listener ports must be nonzero and unique per protocol",
|
||||
));
|
||||
}
|
||||
if !(250..=3_600_000).contains(&l.check_interval_ms)
|
||||
|| !(100..=60_000).contains(&l.check_timeout_ms)
|
||||
|| l.failure_threshold == 0
|
||||
|| l.recovery_threshold == 0
|
||||
|| !(1..=604800).contains(&l.idle_timeout_secs)
|
||||
{
|
||||
return Err(invalid(
|
||||
"invalid check interval, timeout, thresholds or flow idle timeout",
|
||||
));
|
||||
}
|
||||
if l.backends.len() > limits.max_vip_backends {
|
||||
return Err(invalid("too many listener backends"));
|
||||
}
|
||||
let mut backends = BTreeSet::new();
|
||||
for b in &l.backends {
|
||||
if b.port == 0 || b.peer.parse::<EndpointId>().is_err() || !backends.insert(b) {
|
||||
return Err(invalid(
|
||||
"backend must have a unique node identity and a nonzero port",
|
||||
));
|
||||
}
|
||||
}
|
||||
}
|
||||
if crate::state::vip::flatten(self)?.len() > limits.max_vip_fields {
|
||||
return Err(invalid("VIP configuration exceeds signed field capacity"));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
pub(crate) fn invalid(reason: impl Into<String>) -> Error {
|
||||
Error::Rejected {
|
||||
reason: reason.into(),
|
||||
}
|
||||
}
|
||||
/// Cumulative IP traffic, excluding probes. Reset on agent restart.
|
||||
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct Counters {
|
||||
/// Outbound packets.
|
||||
pub tx_packets: u64,
|
||||
/// Outbound bytes.
|
||||
pub tx_bytes: u64,
|
||||
/// Inbound packets.
|
||||
pub rx_packets: u64,
|
||||
/// Inbound bytes.
|
||||
pub rx_bytes: u64,
|
||||
/// Created flow bindings.
|
||||
pub flows: u64,
|
||||
/// UDP failovers.
|
||||
pub failovers: u64,
|
||||
/// Packets rejected or lacking an eligible backend.
|
||||
pub dropped: u64,
|
||||
}
|
||||
/// One backend, as seen by this client.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct BackendStatus {
|
||||
/// Configured endpoint.
|
||||
pub backend: Backend,
|
||||
/// Eligibility and reason.
|
||||
pub state: String,
|
||||
/// Smoothed full data-plane RTT, microseconds; unknown stays absent.
|
||||
pub rtt_us: Option<u64>,
|
||||
/// Last successful path measurement age.
|
||||
pub measurement_age_ms: Option<u64>,
|
||||
/// Current bindings.
|
||||
pub active_flows: u32,
|
||||
/// Local traffic counters.
|
||||
pub counters: Counters,
|
||||
}
|
||||
/// Runtime view of one listener.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct ListenerStatus {
|
||||
/// Protocol.
|
||||
pub protocol: Protocol,
|
||||
/// Frontend port.
|
||||
pub port: u16,
|
||||
/// Best backend for a new flow, if measured and eligible.
|
||||
pub selected: Option<Backend>,
|
||||
/// Configured and draining endpoints.
|
||||
pub backends: Vec<BackendStatus>,
|
||||
/// Local counters.
|
||||
pub counters: Counters,
|
||||
}
|
||||
/// Configuration and local diagnostics returned through the same API to CLI and GUI.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct VipStatus {
|
||||
/// Persisted configuration.
|
||||
pub config: VipConfig,
|
||||
/// Opaque revision used when saving a draft.
|
||||
pub revision: String,
|
||||
/// Internal service address.
|
||||
pub address: Option<Ipv4Addr>,
|
||||
/// Conflict or allocation issue, if any.
|
||||
pub error: Option<String>,
|
||||
/// Local listener status.
|
||||
pub listeners: Vec<ListenerStatus>,
|
||||
}
|
||||
@@ -0,0 +1,566 @@
|
||||
//! Bounded IPv4 parsing, NAT checksums and fragment reassembly at local ingress.
|
||||
use super::Protocol;
|
||||
use bytes::Bytes;
|
||||
use std::collections::HashMap;
|
||||
use std::net::Ipv4Addr;
|
||||
use std::time::{Duration, Instant};
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
|
||||
pub(crate) struct Tuple {
|
||||
pub src: Ipv4Addr,
|
||||
pub dst: Ipv4Addr,
|
||||
pub sport: u16,
|
||||
pub dport: u16,
|
||||
pub protocol: Protocol,
|
||||
}
|
||||
impl Tuple {
|
||||
pub fn reverse(self) -> Self {
|
||||
Self {
|
||||
src: self.dst,
|
||||
dst: self.src,
|
||||
sport: self.dport,
|
||||
dport: self.sport,
|
||||
protocol: self.protocol,
|
||||
}
|
||||
}
|
||||
}
|
||||
pub(crate) fn header(p: &[u8]) -> Option<usize> {
|
||||
if p.len() < 20 || p[0] >> 4 != 4 {
|
||||
return None;
|
||||
}
|
||||
let n = usize::from(p[0] & 15) * 4;
|
||||
let total = usize::from(u16::from_be_bytes([p[2], p[3]]));
|
||||
(n >= 20 && n <= p.len() && total == p.len() && checksum(&p[..n]) == 0).then_some(n)
|
||||
}
|
||||
pub(crate) fn addresses(p: &[u8]) -> Option<(Ipv4Addr, Ipv4Addr)> {
|
||||
header(p)?;
|
||||
Some((
|
||||
Ipv4Addr::new(p[12], p[13], p[14], p[15]),
|
||||
Ipv4Addr::new(p[16], p[17], p[18], p[19]),
|
||||
))
|
||||
}
|
||||
pub(crate) fn tuple(p: &[u8]) -> Option<Tuple> {
|
||||
let n = header(p)?;
|
||||
if u16::from_be_bytes([p[6], p[7]]) & 0x3fff != 0 {
|
||||
return None;
|
||||
}
|
||||
let protocol = match p[9] {
|
||||
6 => Protocol::Tcp,
|
||||
17 => Protocol::Udp,
|
||||
_ => return None,
|
||||
};
|
||||
let min = if protocol == Protocol::Tcp { 20 } else { 8 };
|
||||
if p.len() < n + min {
|
||||
return None;
|
||||
}
|
||||
if protocol == Protocol::Tcp {
|
||||
let h = usize::from(p[n + 12] >> 4) * 4;
|
||||
if h < 20 || n + h > p.len() {
|
||||
return None;
|
||||
}
|
||||
} else if usize::from(u16::from_be_bytes([p[n + 4], p[n + 5]])) != p.len() - n {
|
||||
return None;
|
||||
}
|
||||
let (src, dst) = addresses(p)?;
|
||||
Some(Tuple {
|
||||
src,
|
||||
dst,
|
||||
sport: u16::from_be_bytes([p[n], p[n + 1]]),
|
||||
dport: u16::from_be_bytes([p[n + 2], p[n + 3]]),
|
||||
protocol,
|
||||
})
|
||||
}
|
||||
pub(crate) fn rewrite(p: &[u8], t: Tuple) -> Option<Bytes> {
|
||||
let n = header(p)?;
|
||||
if p.len() < n + if t.protocol == Protocol::Tcp { 20 } else { 8 } {
|
||||
return None;
|
||||
}
|
||||
let mut out = p.to_vec();
|
||||
out[12..16].copy_from_slice(&t.src.octets());
|
||||
out[16..20].copy_from_slice(&t.dst.octets());
|
||||
out[n..n + 2].copy_from_slice(&t.sport.to_be_bytes());
|
||||
out[n + 2..n + 4].copy_from_slice(&t.dport.to_be_bytes());
|
||||
finish(&mut out, n, t.protocol);
|
||||
Some(Bytes::from(out))
|
||||
}
|
||||
fn finish(p: &mut [u8], n: usize, protocol: Protocol) {
|
||||
p[10..12].fill(0);
|
||||
let c = checksum(&p[..n]);
|
||||
p[10..12].copy_from_slice(&c.to_be_bytes());
|
||||
let offset = n + if protocol == Protocol::Tcp { 16 } else { 6 };
|
||||
p[offset..offset + 2].fill(0);
|
||||
let mut sum = 0u32;
|
||||
for pair in p[12..20].as_chunks::<2>().0 {
|
||||
sum += u32::from(u16::from_be_bytes([pair[0], pair[1]]));
|
||||
}
|
||||
sum += u32::from(p[9]) + (p.len() - n) as u32;
|
||||
let c = checksum_seed(&p[n..], sum);
|
||||
let c = if c == 0 && protocol == Protocol::Udp {
|
||||
0xffff
|
||||
} else {
|
||||
c
|
||||
};
|
||||
p[offset..offset + 2].copy_from_slice(&c.to_be_bytes());
|
||||
}
|
||||
pub(crate) fn checksum(p: &[u8]) -> u16 {
|
||||
checksum_seed(p, 0)
|
||||
}
|
||||
fn checksum_seed(p: &[u8], mut sum: u32) -> u16 {
|
||||
for pair in p.chunks(2) {
|
||||
sum += u32::from(u16::from_be_bytes([pair[0], *pair.get(1).unwrap_or(&0)]));
|
||||
}
|
||||
while sum >> 16 != 0 {
|
||||
sum = (sum & 0xffff) + (sum >> 16);
|
||||
}
|
||||
!(sum as u16)
|
||||
}
|
||||
pub(crate) fn tcp_syn(p: &[u8]) -> bool {
|
||||
header(p).is_some_and(|n| p[9] == 6 && p.get(n + 13).is_some_and(|f| f & 0x12 == 2))
|
||||
}
|
||||
pub(crate) fn syn_sequence(p: &[u8]) -> Option<u32> {
|
||||
if !tcp_syn(p) {
|
||||
return None;
|
||||
}
|
||||
let n = header(p)?;
|
||||
let bytes = p.get(n + 4..n + 8)?;
|
||||
Some(u32::from_be_bytes(bytes.try_into().ok()?))
|
||||
}
|
||||
pub(crate) fn empty_syn(p: &[u8]) -> bool {
|
||||
header(p).is_some_and(|n| {
|
||||
tcp_syn(p)
|
||||
&& p.get(n + 12)
|
||||
.is_some_and(|h| n + usize::from(h >> 4) * 4 == p.len())
|
||||
})
|
||||
}
|
||||
pub(crate) fn syn_ack(p: &[u8]) -> bool {
|
||||
header(p).is_some_and(|n| p[9] == 6 && p.get(n + 13).is_some_and(|f| f & 0x12 == 0x12))
|
||||
}
|
||||
pub(crate) fn tcp_reset(p: &[u8]) -> bool {
|
||||
header(p).is_some_and(|n| p[9] == 6 && p.get(n + 13).is_some_and(|f| f & 4 != 0))
|
||||
}
|
||||
/// Rewrites an ICMP error and its quoted original header after NAT lookup.
|
||||
pub(crate) fn icmp_quote(p: &[u8]) -> Option<Tuple> {
|
||||
let n = header(p)?;
|
||||
if p[9] != 1 || p.len() < n + 8 + 28 || ![3, 11, 12].contains(&p[n]) {
|
||||
return None;
|
||||
}
|
||||
let q = &p[n + 8..];
|
||||
let h = usize::from(q[0] & 15) * 4;
|
||||
if q[0] >> 4 != 4 || h < 20 || q.len() < h + 8 {
|
||||
return None;
|
||||
}
|
||||
let protocol = match q[9] {
|
||||
6 => Protocol::Tcp,
|
||||
17 => Protocol::Udp,
|
||||
_ => return None,
|
||||
};
|
||||
Some(Tuple {
|
||||
src: Ipv4Addr::new(q[12], q[13], q[14], q[15]),
|
||||
dst: Ipv4Addr::new(q[16], q[17], q[18], q[19]),
|
||||
sport: u16::from_be_bytes([q[h], q[h + 1]]),
|
||||
dport: u16::from_be_bytes([q[h + 2], q[h + 3]]),
|
||||
protocol,
|
||||
})
|
||||
}
|
||||
pub(crate) fn rewrite_icmp(p: &[u8], original: Tuple) -> Option<Bytes> {
|
||||
let n = header(p)?;
|
||||
icmp_quote(p)?;
|
||||
let mut out = p.to_vec();
|
||||
out[12..16].copy_from_slice(&original.dst.octets());
|
||||
out[16..20].copy_from_slice(&original.src.octets());
|
||||
let q = &mut out[n + 8..];
|
||||
let h = usize::from(q[0] & 15) * 4;
|
||||
// Incrementally update the quoted transport checksum when it is present.
|
||||
let off = h + if original.protocol == Protocol::Tcp {
|
||||
16
|
||||
} else {
|
||||
6
|
||||
};
|
||||
if q.len() >= off + 2 {
|
||||
let old = u16::from_be_bytes([q[off], q[off + 1]]);
|
||||
if old != 0 || original.protocol == Protocol::Tcp {
|
||||
let mut sum = u32::from(!old);
|
||||
let mut replacement = Vec::from(original.src.octets());
|
||||
replacement.extend(original.dst.octets());
|
||||
replacement.extend(original.sport.to_be_bytes());
|
||||
replacement.extend(original.dport.to_be_bytes());
|
||||
let previous = [&q[12..20], &q[h..h + 4]].concat();
|
||||
for (a, b) in previous
|
||||
.as_chunks::<2>()
|
||||
.0
|
||||
.iter()
|
||||
.zip(replacement.as_chunks::<2>().0)
|
||||
{
|
||||
sum += u32::from(!u16::from_be_bytes([a[0], a[1]]))
|
||||
+ u32::from(u16::from_be_bytes([b[0], b[1]]));
|
||||
}
|
||||
while sum >> 16 != 0 {
|
||||
sum = (sum & 0xffff) + (sum >> 16);
|
||||
}
|
||||
let c = !(sum as u16);
|
||||
q[off..off + 2].copy_from_slice(
|
||||
&(if c == 0 && original.protocol == Protocol::Udp {
|
||||
0xffff
|
||||
} else {
|
||||
c
|
||||
})
|
||||
.to_be_bytes(),
|
||||
);
|
||||
}
|
||||
}
|
||||
q[12..16].copy_from_slice(&original.src.octets());
|
||||
q[16..20].copy_from_slice(&original.dst.octets());
|
||||
q[h..h + 2].copy_from_slice(&original.sport.to_be_bytes());
|
||||
q[h + 2..h + 4].copy_from_slice(&original.dport.to_be_bytes());
|
||||
q[10..12].fill(0);
|
||||
let c = checksum(&q[..h]);
|
||||
q[10..12].copy_from_slice(&c.to_be_bytes());
|
||||
out[n + 2..n + 4].fill(0);
|
||||
let c = checksum(&out[n..]);
|
||||
out[n + 2..n + 4].copy_from_slice(&c.to_be_bytes());
|
||||
out[10..12].fill(0);
|
||||
let c = checksum(&out[..n]);
|
||||
out[10..12].copy_from_slice(&c.to_be_bytes());
|
||||
Some(Bytes::from(out))
|
||||
}
|
||||
pub(crate) fn reject(p: &[u8]) -> Option<Bytes> {
|
||||
let t = tuple(p)?;
|
||||
let n = header(p)?;
|
||||
if t.protocol == Protocol::Tcp {
|
||||
if !tcp_syn(p) {
|
||||
return None;
|
||||
}
|
||||
let mut out = vec![0; 40];
|
||||
out[0] = 0x45;
|
||||
out[2..4].copy_from_slice(&40u16.to_be_bytes());
|
||||
out[8] = 64;
|
||||
out[9] = 6;
|
||||
out[12..16].copy_from_slice(&t.dst.octets());
|
||||
out[16..20].copy_from_slice(&t.src.octets());
|
||||
out[20..22].copy_from_slice(&t.dport.to_be_bytes());
|
||||
out[22..24].copy_from_slice(&t.sport.to_be_bytes());
|
||||
let seq = u32::from_be_bytes(p[n + 4..n + 8].try_into().ok()?);
|
||||
out[28..32].copy_from_slice(&seq.wrapping_add(1).to_be_bytes());
|
||||
out[32] = 0x50;
|
||||
out[33] = 0x14;
|
||||
finish(&mut out, 20, Protocol::Tcp);
|
||||
Some(Bytes::from(out))
|
||||
} else {
|
||||
let q = p.len().min(n + 8);
|
||||
let mut out = vec![0; 28 + q];
|
||||
out[0] = 0x45;
|
||||
let len = out.len() as u16;
|
||||
out[2..4].copy_from_slice(&len.to_be_bytes());
|
||||
out[8] = 64;
|
||||
out[9] = 1;
|
||||
out[12..16].copy_from_slice(&t.dst.octets());
|
||||
out[16..20].copy_from_slice(&t.src.octets());
|
||||
out[20] = 3;
|
||||
out[21] = 3;
|
||||
out[28..].copy_from_slice(&p[..q]);
|
||||
let c = checksum(&out[20..]);
|
||||
out[22..24].copy_from_slice(&c.to_be_bytes());
|
||||
let c = checksum(&out[..20]);
|
||||
out[10..12].copy_from_slice(&c.to_be_bytes());
|
||||
Some(Bytes::from(out))
|
||||
}
|
||||
}
|
||||
#[derive(Debug)]
|
||||
struct Datagram {
|
||||
at: Instant,
|
||||
header: Option<Vec<u8>>,
|
||||
parts: std::collections::BTreeMap<usize, Vec<u8>>,
|
||||
end: Option<usize>,
|
||||
bytes: usize,
|
||||
}
|
||||
type FragmentKey = (Option<[u8; 32]>, Ipv4Addr, Ipv4Addr, u16, u8);
|
||||
#[derive(Debug, Default)]
|
||||
pub(crate) struct Reassembly {
|
||||
pending: HashMap<FragmentKey, Datagram>,
|
||||
}
|
||||
impl Reassembly {
|
||||
pub fn expire(&mut self, now: Instant) {
|
||||
self.pending
|
||||
.retain(|_, d| now.duration_since(d.at) < Duration::from_secs(5));
|
||||
}
|
||||
pub fn accept(&mut self, p: Bytes, max: usize, scope: Option<[u8; 32]>) -> Option<Bytes> {
|
||||
let n = header(&p)?;
|
||||
let flags = u16::from_be_bytes([p[6], p[7]]);
|
||||
if flags & 0x3fff == 0 {
|
||||
return Some(p);
|
||||
}
|
||||
let now = Instant::now();
|
||||
self.pending
|
||||
.retain(|_, d| now.duration_since(d.at) < Duration::from_secs(5));
|
||||
let (src, dst) = addresses(&p)?;
|
||||
let key = (scope, src, dst, u16::from_be_bytes([p[4], p[5]]), p[9]);
|
||||
let start = usize::from(flags & 0x1fff) * 8;
|
||||
let payload = &p[n..];
|
||||
let end = start.checked_add(payload.len())?;
|
||||
if payload.is_empty()
|
||||
|| end > 65515
|
||||
|| flags & 0x2000 != 0 && !payload.len().is_multiple_of(8)
|
||||
{
|
||||
self.pending.remove(&key);
|
||||
return None;
|
||||
}
|
||||
if !self.pending.contains_key(&key) && self.pending.len() >= max {
|
||||
return None;
|
||||
}
|
||||
let d = self.pending.entry(key).or_insert_with(|| Datagram {
|
||||
at: now,
|
||||
header: None,
|
||||
parts: Default::default(),
|
||||
end: None,
|
||||
bytes: 0,
|
||||
});
|
||||
if d.parts
|
||||
.get(&start)
|
||||
.is_some_and(|bytes| bytes.as_slice() == payload)
|
||||
{
|
||||
return None;
|
||||
}
|
||||
if d.parts.iter().any(|(s, b)| *s < end && start < s + b.len()) {
|
||||
self.pending.remove(&key);
|
||||
return None;
|
||||
}
|
||||
if flags & 0x2000 == 0 {
|
||||
if d.end.is_some_and(|e| e != end) {
|
||||
self.pending.remove(&key);
|
||||
return None;
|
||||
}
|
||||
d.end = Some(end);
|
||||
}
|
||||
if d.end.is_some_and(|e| end > e) {
|
||||
self.pending.remove(&key);
|
||||
return None;
|
||||
}
|
||||
if start == 0 {
|
||||
d.header = Some(p[..n].to_vec());
|
||||
}
|
||||
d.bytes += payload.len();
|
||||
d.parts.insert(start, payload.to_vec());
|
||||
let complete = d.end.is_some_and(|e| e == d.bytes) && d.header.is_some();
|
||||
if !complete {
|
||||
return None;
|
||||
}
|
||||
let d = self.pending.remove(&key)?;
|
||||
let mut out = d.header?;
|
||||
for part in d.parts.values() {
|
||||
out.extend(part);
|
||||
}
|
||||
if out.len() > 65535 {
|
||||
return None;
|
||||
}
|
||||
let len = out.len() as u16;
|
||||
out[2..4].copy_from_slice(&len.to_be_bytes());
|
||||
out[6..8].fill(0);
|
||||
out[10..12].fill(0);
|
||||
let h = usize::from(out[0] & 15) * 4;
|
||||
let c = checksum(&out[..h]);
|
||||
out[10..12].copy_from_slice(&c.to_be_bytes());
|
||||
Some(Bytes::from(out))
|
||||
}
|
||||
}
|
||||
|
||||
/// Re-fragments an assembled NAT datagram for the unchanged overlay MTU.
|
||||
pub(crate) fn fragments(packet: Bytes, mtu: u32) -> Vec<Bytes> {
|
||||
if packet.len() <= mtu as usize {
|
||||
return vec![packet];
|
||||
}
|
||||
let Some(n) = header(&packet) else {
|
||||
return vec![];
|
||||
};
|
||||
if u16::from_be_bytes([packet[6], packet[7]]) & 0x4000 != 0 {
|
||||
return vec![];
|
||||
}
|
||||
let size = (mtu as usize).saturating_sub(n) / 8 * 8;
|
||||
if size == 0 {
|
||||
return vec![];
|
||||
}
|
||||
let mut copied = Vec::new();
|
||||
let mut offset = 20;
|
||||
while offset < n {
|
||||
let kind = packet[offset];
|
||||
if kind == 0 {
|
||||
break;
|
||||
}
|
||||
if kind == 1 {
|
||||
offset += 1;
|
||||
continue;
|
||||
}
|
||||
let Some(&len) = packet.get(offset + 1) else {
|
||||
return vec![];
|
||||
};
|
||||
let len = usize::from(len);
|
||||
if len < 2 || offset + len > n {
|
||||
return vec![];
|
||||
}
|
||||
if kind & 0x80 != 0 {
|
||||
copied.extend_from_slice(&packet[offset..offset + len]);
|
||||
}
|
||||
offset += len;
|
||||
}
|
||||
while !copied.len().is_multiple_of(4) {
|
||||
copied.push(0);
|
||||
}
|
||||
let mut result = Vec::new();
|
||||
let payload = &packet[n..];
|
||||
for (start, part) in payload.chunks(size).enumerate() {
|
||||
let offset = start * size;
|
||||
let mut out = if offset == 0 {
|
||||
packet[..n].to_vec()
|
||||
} else {
|
||||
let mut h = packet[..20].to_vec();
|
||||
h.extend_from_slice(&copied);
|
||||
h[0] = 0x40 | (h.len() / 4) as u8;
|
||||
h
|
||||
};
|
||||
let header_len = out.len();
|
||||
out.extend(part);
|
||||
let len = out.len() as u16;
|
||||
out[2..4].copy_from_slice(&len.to_be_bytes());
|
||||
let flags = (offset / 8) as u16
|
||||
| if offset + part.len() < payload.len() {
|
||||
0x2000
|
||||
} else {
|
||||
0
|
||||
};
|
||||
out[6..8].copy_from_slice(&flags.to_be_bytes());
|
||||
out[10..12].fill(0);
|
||||
let c = checksum(&out[..header_len]);
|
||||
out[10..12].copy_from_slice(&c.to_be_bytes());
|
||||
result.push(Bytes::from(out));
|
||||
}
|
||||
result
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
|
||||
use super::*;
|
||||
fn datagram() -> Bytes {
|
||||
let mut p = vec![0; 3028];
|
||||
p[0] = 0x45;
|
||||
p[2..4].copy_from_slice(&3028u16.to_be_bytes());
|
||||
p[8] = 64;
|
||||
p[9] = 17;
|
||||
p[12..16].copy_from_slice(&[10, 0, 0, 1]);
|
||||
p[16..20].copy_from_slice(&[10, 0, 0, 2]);
|
||||
p[20..22].copy_from_slice(&1000u16.to_be_bytes());
|
||||
p[22..24].copy_from_slice(&2000u16.to_be_bytes());
|
||||
p[24..26].copy_from_slice(&3008u16.to_be_bytes());
|
||||
finish(&mut p, 20, Protocol::Udp);
|
||||
Bytes::from(p)
|
||||
}
|
||||
#[test]
|
||||
fn fragmented_udp_reassembles_out_of_order_and_nat_checksums_are_valid() {
|
||||
let p = datagram();
|
||||
let parts = fragments(p.clone(), 1280);
|
||||
assert_eq!(parts.len(), 3);
|
||||
let mut reassembly = Reassembly::default();
|
||||
let mut result = None;
|
||||
for part in parts.into_iter().rev() {
|
||||
if let Some(p) = reassembly.accept(part, 8, None) {
|
||||
result = Some(p);
|
||||
}
|
||||
}
|
||||
assert_eq!(result, Some(p.clone()));
|
||||
let mut t = tuple(&p).unwrap();
|
||||
t.dst = Ipv4Addr::new(10, 0, 0, 240);
|
||||
t.dport = 53;
|
||||
let out = rewrite(&p, t).unwrap();
|
||||
assert_eq!(header(&out), Some(20));
|
||||
let mut sum = 0u32;
|
||||
for pair in out[12..20].as_chunks::<2>().0 {
|
||||
sum += u32::from(u16::from_be_bytes([pair[0], pair[1]]));
|
||||
}
|
||||
sum += 17 + (out.len() - 20) as u32;
|
||||
assert_eq!(checksum_seed(&out[20..], sum), 0);
|
||||
}
|
||||
#[test]
|
||||
fn overlaps_and_capacity_are_rejected_without_allocating_unbounded_state() {
|
||||
let parts = fragments(datagram(), 1280);
|
||||
let mut r = Reassembly::default();
|
||||
assert!(r.accept(parts[0].clone(), 0, None).is_none());
|
||||
assert!(r.pending.is_empty());
|
||||
assert!(r.accept(parts[0].clone(), 1, None).is_none());
|
||||
assert!(r.accept(parts[0].clone(), 1, None).is_none());
|
||||
assert_eq!(r.pending.len(), 1);
|
||||
let mut conflict = parts[0].to_vec();
|
||||
conflict[20] ^= 1;
|
||||
assert!(r.accept(Bytes::from(conflict), 1, None).is_none());
|
||||
assert!(r.pending.is_empty());
|
||||
for size in 0..80 {
|
||||
let p = vec![0xff; size];
|
||||
assert!(tuple(&p).is_none());
|
||||
assert!(icmp_quote(&p).is_none());
|
||||
}
|
||||
}
|
||||
#[test]
|
||||
fn fragment_assembly_is_scoped_to_authenticated_sender_and_expires_when_idle() {
|
||||
let packet = datagram();
|
||||
let parts = fragments(packet.clone(), 1280);
|
||||
let mut r = Reassembly::default();
|
||||
assert!(r.accept(parts[0].clone(), 8, Some([1; 32])).is_none());
|
||||
assert!(r.accept(parts[1].clone(), 8, Some([2; 32])).is_none());
|
||||
assert!(r.accept(parts[2].clone(), 8, Some([1; 32])).is_none());
|
||||
assert_eq!(r.accept(parts[1].clone(), 8, Some([1; 32])), Some(packet));
|
||||
assert_eq!(r.pending.len(), 1);
|
||||
r.expire(Instant::now() + Duration::from_secs(6));
|
||||
assert!(r.pending.is_empty());
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod icmp_tests {
|
||||
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
|
||||
use super::*;
|
||||
#[test]
|
||||
fn pmtu_errors_restore_quoted_addresses_ports_and_checksums() {
|
||||
let original = Tuple {
|
||||
src: Ipv4Addr::new(10, 0, 0, 1),
|
||||
dst: Ipv4Addr::new(10, 0, 0, 240),
|
||||
sport: 40000,
|
||||
dport: 53,
|
||||
protocol: Protocol::Udp,
|
||||
};
|
||||
let mut p = vec![0; 36];
|
||||
p[0] = 0x45;
|
||||
p[2..4].copy_from_slice(&36u16.to_be_bytes());
|
||||
p[8] = 64;
|
||||
p[9] = 17;
|
||||
p[24..26].copy_from_slice(&16u16.to_be_bytes());
|
||||
let c = checksum(&p[..20]);
|
||||
p[10..12].copy_from_slice(&c.to_be_bytes());
|
||||
let p = rewrite(&p, original).unwrap();
|
||||
let wire = Tuple {
|
||||
dst: Ipv4Addr::new(10, 0, 0, 2),
|
||||
sport: 50000,
|
||||
dport: 5353,
|
||||
..original
|
||||
};
|
||||
let translated = rewrite(&p, wire).unwrap();
|
||||
let mut error = vec![0; 56];
|
||||
error[0] = 0x45;
|
||||
error[2..4].copy_from_slice(&56u16.to_be_bytes());
|
||||
error[8] = 64;
|
||||
error[9] = 1;
|
||||
error[12..16].copy_from_slice(&wire.dst.octets());
|
||||
error[16..20].copy_from_slice(&wire.src.octets());
|
||||
error[20] = 3;
|
||||
error[21] = 4;
|
||||
error[26..28].copy_from_slice(&1200u16.to_be_bytes());
|
||||
error[28..].copy_from_slice(&translated[..28]);
|
||||
let c = checksum(&error[20..]);
|
||||
error[22..24].copy_from_slice(&c.to_be_bytes());
|
||||
let c = checksum(&error[..20]);
|
||||
error[10..12].copy_from_slice(&c.to_be_bytes());
|
||||
let restored = rewrite_icmp(&error, original).unwrap();
|
||||
assert_eq!(icmp_quote(&restored), Some(original));
|
||||
assert_eq!(checksum(&restored[20..]), 0);
|
||||
assert_eq!(checksum(&restored[28..48]), 0);
|
||||
assert_eq!(&restored[54..56], &p[26..28]);
|
||||
assert_eq!(&restored[26..28], &1200u16.to_be_bytes());
|
||||
assert_eq!(&restored[12..16], &original.dst.octets());
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,144 @@
|
||||
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
|
||||
use super::*;
|
||||
use crate::NetworkId;
|
||||
|
||||
#[test]
|
||||
fn concurrent_backend_edits_merge_and_deletion_cannot_resurrect() {
|
||||
let net = NetworkId::from_bytes([7; 32]);
|
||||
let a = iroh::SecretKey::generate();
|
||||
let b = iroh::SecretKey::generate();
|
||||
let limits = Limits::default();
|
||||
let mut state = VipSet::default();
|
||||
let id = hex::encode([9; 16]);
|
||||
let initial = VipConfig {
|
||||
id: id.clone(),
|
||||
name: "api".into(),
|
||||
listeners: vec![Listener::new(Protocol::Tcp, 443)],
|
||||
};
|
||||
let changes = state.edit(net, &a, 1, &initial, None, &limits).unwrap();
|
||||
state.merge(net, changes, &limits).unwrap();
|
||||
let revision = state.revision(&id);
|
||||
let mut first = initial.clone();
|
||||
first.listeners[0].backends.push(Backend {
|
||||
peer: a.public().to_string(),
|
||||
port: 6443,
|
||||
});
|
||||
let mut second = initial.clone();
|
||||
second.listeners[0].backends.push(Backend {
|
||||
peer: b.public().to_string(),
|
||||
port: 7443,
|
||||
});
|
||||
let x = state
|
||||
.edit(net, &a, 2, &first, Some(&revision), &limits)
|
||||
.unwrap();
|
||||
let y = state
|
||||
.edit(net, &b, 1, &second, Some(&revision), &limits)
|
||||
.unwrap();
|
||||
let mut reverse = state.clone();
|
||||
state.merge(net, x.clone(), &limits).unwrap();
|
||||
state.merge(net, y.clone(), &limits).unwrap();
|
||||
reverse.merge(net, y, &limits).unwrap();
|
||||
reverse.merge(net, x, &limits).unwrap();
|
||||
assert_eq!(state.configs(), reverse.configs());
|
||||
assert_eq!(state.configs()[0].listeners[0].backends.len(), 2);
|
||||
assert!(
|
||||
state
|
||||
.edit(net, &a, 3, &initial, Some(&revision), &limits)
|
||||
.is_err()
|
||||
);
|
||||
let removed = state.delete(net, &a, 3, &id, &state.revision(&id)).unwrap();
|
||||
let old = state.entries(&id);
|
||||
state.merge(net, removed, &limits).unwrap();
|
||||
state.merge(net, old, &limits).unwrap();
|
||||
assert!(state.configs().is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn signed_fields_reject_tampering_wrong_network_and_limits() {
|
||||
let net = NetworkId::from_bytes([2; 32]);
|
||||
let key = iroh::SecretKey::generate();
|
||||
let limits = Limits::default();
|
||||
let state = VipSet::default();
|
||||
let cfg = VipConfig {
|
||||
id: hex::encode([3; 16]),
|
||||
name: "dns".into(),
|
||||
..Default::default()
|
||||
};
|
||||
let entries = state.edit(net, &key, 1, &cfg, None, &limits).unwrap();
|
||||
let mut bad = entries.clone();
|
||||
bad[0].clock += 1;
|
||||
assert!(VipSet::default().merge(net, bad, &limits).is_err());
|
||||
assert!(
|
||||
VipSet::default()
|
||||
.merge(NetworkId::from_bytes([4; 32]), entries.clone(), &limits)
|
||||
.is_err()
|
||||
);
|
||||
let mut small = limits.clone();
|
||||
small.max_vips = 0;
|
||||
assert!(VipSet::default().merge(net, entries, &small).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn deletion_history_does_not_exhaust_active_capacity_or_accept_stale_fields() {
|
||||
let network = NetworkId::from_bytes([9; 32]);
|
||||
let key = iroh::SecretKey::generate();
|
||||
let limits = Limits {
|
||||
max_vips: 1,
|
||||
max_vip_tombstones: 4,
|
||||
..Default::default()
|
||||
};
|
||||
let mut state = VipSet::default();
|
||||
for index in 0..4 {
|
||||
let config = VipConfig::new("reused-name");
|
||||
let fields = state
|
||||
.edit(network, &key, index * 2 + 1, &config, None, &limits)
|
||||
.unwrap();
|
||||
state.merge(network, fields.clone(), &limits).unwrap();
|
||||
let deletion = state
|
||||
.delete(
|
||||
network,
|
||||
&key,
|
||||
index * 2 + 2,
|
||||
&config.id,
|
||||
&state.revision(&config.id),
|
||||
)
|
||||
.unwrap();
|
||||
state.merge(network, deletion, &limits).unwrap();
|
||||
assert_eq!(state.entries(&config.id).len(), 1);
|
||||
assert!(!state.merge(network, fields, &limits).unwrap());
|
||||
}
|
||||
let next = VipConfig::new("next");
|
||||
let fields = state.edit(network, &key, 9, &next, None, &limits).unwrap();
|
||||
state.merge(network, fields, &limits).unwrap();
|
||||
assert_eq!(state.configs().len(), 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn reservations_avoid_nodes_and_name_conflicts_have_a_deterministic_holder() {
|
||||
let network = NetworkId::from_bytes([4; 32]);
|
||||
let key = iroh::SecretKey::generate();
|
||||
let limits = Limits::default();
|
||||
let mut state = VipSet::default();
|
||||
for version in 1..=2 {
|
||||
let config = VipConfig::new("shared-name");
|
||||
let fields = state
|
||||
.edit(network, &key, version, &config, None, &limits)
|
||||
.unwrap();
|
||||
state.merge(network, fields, &limits).unwrap();
|
||||
}
|
||||
let range = "10.0.0.0/29".parse().unwrap();
|
||||
let taken = ["10.0.0.1".parse().unwrap()].into_iter().collect();
|
||||
let views = state.views(Some(range), taken, &Default::default(), &limits);
|
||||
assert!(views[0].error.is_none());
|
||||
assert!(
|
||||
views[1]
|
||||
.error
|
||||
.as_deref()
|
||||
.unwrap()
|
||||
.contains("name conflicts")
|
||||
);
|
||||
assert_ne!(views[0].address, views[1].address);
|
||||
for v in views {
|
||||
assert_ne!(v.address, Some("10.0.0.1".parse().unwrap()));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,358 @@
|
||||
//! Equal-rights retirement uses real membership, signatures and persistent state.
|
||||
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
|
||||
|
||||
use tsunagi::discovery::SharedMemoryDiscovery;
|
||||
use tsunagi::testing::{TestAgent, config_with, network, wait_for_peers, wait_until};
|
||||
use tsunagi::{Agent, NetworkId};
|
||||
|
||||
#[tokio::test]
|
||||
async fn bounded_marker_batches_eventually_reach_the_tail_of_a_small_send_queue() {
|
||||
use tsunagi::config::StoragePaths;
|
||||
use tsunagi::state::{RecordBody, SignedRecord, SignedRetirement};
|
||||
use tsunagi::storage::StateStore;
|
||||
|
||||
let discovery = SharedMemoryDiscovery::new();
|
||||
let cleaner = TestAgent::spawn(&discovery).await.unwrap();
|
||||
let replica = TestAgent::spawn(&discovery).await.unwrap();
|
||||
let (name, secret) = network("forget-batched-history");
|
||||
let id = cleaner.agent.join_network(&name, &secret).await.unwrap();
|
||||
replica.agent.join_network(&name, &secret).await.unwrap();
|
||||
let cleaner_dir = cleaner.stop().await;
|
||||
let replica_dir = replica.stop().await;
|
||||
let cleaner_path = StoragePaths::under(cleaner_dir.path()).state_db();
|
||||
let replica_path = StoragePaths::under(replica_dir.path()).state_db();
|
||||
// Seed genuine signed offline history in closed stores. Delivery still uses
|
||||
// real agents, authenticated loopback sessions and the persistent stores.
|
||||
let targets = tokio::task::spawn_blocking(move || {
|
||||
let cleaner = StateStore::open(&cleaner_path).unwrap();
|
||||
let replica = StateStore::open(&replica_path).unwrap();
|
||||
let author = cleaner.device_identity().unwrap().unwrap();
|
||||
let signing_key = iroh::SecretKey::from_bytes(&author.secret_bytes());
|
||||
let mut targets = Vec::new();
|
||||
for index in 0..12 {
|
||||
let target = iroh::SecretKey::generate();
|
||||
let record = SignedRecord::sign(
|
||||
&target,
|
||||
id,
|
||||
1,
|
||||
RecordBody::Claim {
|
||||
address: None,
|
||||
range: None,
|
||||
hostname: Some(format!("offline-{index}")),
|
||||
hostname_since: Some(1),
|
||||
},
|
||||
);
|
||||
cleaner.put_signed_record(&record).unwrap();
|
||||
replica.put_signed_record(&record).unwrap();
|
||||
cleaner
|
||||
.publish_own_retirement(&SignedRetirement::sign(
|
||||
&signing_key,
|
||||
id,
|
||||
100 + index,
|
||||
record,
|
||||
))
|
||||
.unwrap();
|
||||
targets.push(target.public());
|
||||
}
|
||||
targets
|
||||
})
|
||||
.await
|
||||
.unwrap();
|
||||
let replica = Agent::spawn(config_with(replica_dir.path(), &discovery))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
replica
|
||||
.network_status(id)
|
||||
.await
|
||||
.unwrap()
|
||||
.members
|
||||
.iter()
|
||||
.filter(|m| targets.contains(&m.endpoint_id))
|
||||
.count(),
|
||||
12
|
||||
);
|
||||
let mut config = config_with(cleaner_dir.path(), &discovery);
|
||||
config.limits.max_state_records = 1;
|
||||
config.limits.session_send_queue = 2;
|
||||
let cleaner = Agent::spawn(config).await.unwrap();
|
||||
wait_for_peers(&cleaner, id, 1).await;
|
||||
wait_until(
|
||||
"all cleanup batches arrive despite send queue pressure",
|
||||
|| async {
|
||||
replica
|
||||
.network_status(id)
|
||||
.await
|
||||
.ok()?
|
||||
.members
|
||||
.iter()
|
||||
.all(|m| !targets.contains(&m.endpoint_id))
|
||||
.then_some(())
|
||||
},
|
||||
)
|
||||
.await;
|
||||
cleaner.shutdown().await;
|
||||
replica.shutdown().await;
|
||||
let replica_path = StoragePaths::under(replica_dir.path()).state_db();
|
||||
tokio::task::spawn_blocking(move || {
|
||||
let replica = StateStore::open(&replica_path).unwrap();
|
||||
assert_eq!(replica.retirements(id, 16).unwrap().len(), 12);
|
||||
})
|
||||
.await
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
async fn member(
|
||||
agent: &Agent,
|
||||
network: NetworkId,
|
||||
peer: iroh::EndpointId,
|
||||
) -> tsunagi::agent::MemberStatus {
|
||||
wait_until("signed member", || async {
|
||||
agent
|
||||
.network_status(network)
|
||||
.await
|
||||
.ok()?
|
||||
.members
|
||||
.into_iter()
|
||||
.find(|m| m.endpoint_id == peer)
|
||||
})
|
||||
.await
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn retirement_survives_restart_and_stale_replicas_but_allows_a_new_claim() {
|
||||
let discovery = SharedMemoryDiscovery::new();
|
||||
let cleaner = TestAgent::spawn_with(|c| c.with_hostname("cleaner"), &discovery)
|
||||
.await
|
||||
.unwrap();
|
||||
let target = TestAgent::spawn_with(|c| c.with_hostname("old-device"), &discovery)
|
||||
.await
|
||||
.unwrap();
|
||||
let replica = TestAgent::spawn_with(|c| c.with_hostname("replica"), &discovery)
|
||||
.await
|
||||
.unwrap();
|
||||
let (name, secret) = network("forget-propagation");
|
||||
let id = cleaner.agent.join_network(&name, &secret).await.unwrap();
|
||||
target.agent.join_network(&name, &secret).await.unwrap();
|
||||
replica.agent.join_network(&name, &secret).await.unwrap();
|
||||
wait_for_peers(&cleaner.agent, id, 2).await;
|
||||
let peer = target.agent.endpoint_id();
|
||||
let original = wait_until("all three agents agree on allocated claims", || async {
|
||||
let a = cleaner.agent.network_status(id).await.ok()?;
|
||||
let b = target.agent.network_status(id).await.ok()?;
|
||||
let c = replica.agent.network_status(id).await.ok()?;
|
||||
for status in [&a, &b, &c] {
|
||||
if status.members.len() != 3 {
|
||||
return None;
|
||||
}
|
||||
let addresses: std::collections::HashSet<_> = status
|
||||
.members
|
||||
.iter()
|
||||
.filter_map(|m| m.overlay_address_v4)
|
||||
.collect();
|
||||
if addresses.len() != 3 {
|
||||
return None;
|
||||
}
|
||||
}
|
||||
let record = a.members.iter().find(|m| m.endpoint_id == peer)?;
|
||||
(b.members
|
||||
.iter()
|
||||
.find(|m| m.endpoint_id == peer)?
|
||||
.record_version
|
||||
== record.record_version
|
||||
&& c.members
|
||||
.iter()
|
||||
.find(|m| m.endpoint_id == peer)?
|
||||
.record_version
|
||||
== record.record_version)
|
||||
.then(|| record.clone())
|
||||
})
|
||||
.await;
|
||||
assert!(
|
||||
cleaner
|
||||
.agent
|
||||
.forget_node(id, peer, original.record_version)
|
||||
.await
|
||||
.is_err(),
|
||||
"online nodes are not history"
|
||||
);
|
||||
assert!(
|
||||
cleaner
|
||||
.agent
|
||||
.forget_node(id, cleaner.agent.endpoint_id(), 1)
|
||||
.await
|
||||
.is_err()
|
||||
);
|
||||
let replica_dir = replica.stop().await;
|
||||
let target_dir = target.stop().await;
|
||||
wait_until("target offline", || async {
|
||||
cleaner
|
||||
.agent
|
||||
.network_status(id)
|
||||
.await
|
||||
.ok()?
|
||||
.members
|
||||
.iter()
|
||||
.find(|m| m.endpoint_id == peer)
|
||||
.filter(|m| !m.online)
|
||||
.map(|_| ())
|
||||
})
|
||||
.await;
|
||||
assert!(
|
||||
cleaner
|
||||
.agent
|
||||
.forget_node(id, peer, original.record_version + 1)
|
||||
.await
|
||||
.is_err(),
|
||||
"stale confirmation changes nothing"
|
||||
);
|
||||
let result = cleaner
|
||||
.agent
|
||||
.forget_node(id, peer, original.record_version)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(result.target, peer);
|
||||
assert_eq!(result.record_version, original.record_version);
|
||||
assert_eq!(result.peers_told, 0);
|
||||
assert!(
|
||||
!cleaner
|
||||
.agent
|
||||
.network_status(id)
|
||||
.await
|
||||
.unwrap()
|
||||
.members
|
||||
.iter()
|
||||
.any(|m| m.endpoint_id == peer)
|
||||
);
|
||||
let cleaner_dir = cleaner.stop().await;
|
||||
let cleaner =
|
||||
Agent::spawn(config_with(cleaner_dir.path(), &discovery).with_hostname("cleaner"))
|
||||
.await
|
||||
.unwrap();
|
||||
let replica =
|
||||
Agent::spawn(config_with(replica_dir.path(), &discovery).with_hostname("replica"))
|
||||
.await
|
||||
.unwrap();
|
||||
wait_for_peers(&cleaner, id, 1).await;
|
||||
wait_until("stale replica learns durable retirement", || async {
|
||||
replica
|
||||
.network_status(id)
|
||||
.await
|
||||
.ok()?
|
||||
.members
|
||||
.iter()
|
||||
.all(|m| m.endpoint_id != peer)
|
||||
.then_some(())
|
||||
})
|
||||
.await;
|
||||
assert!(
|
||||
!cleaner
|
||||
.network_status(id)
|
||||
.await
|
||||
.unwrap()
|
||||
.members
|
||||
.iter()
|
||||
.any(|m| m.endpoint_id == peer)
|
||||
);
|
||||
replica.set_hostname("old-device").await.unwrap();
|
||||
let replica_id = replica.endpoint_id();
|
||||
wait_until("released hostname is reclaimed", || async {
|
||||
cleaner
|
||||
.network_status(id)
|
||||
.await
|
||||
.ok()?
|
||||
.members
|
||||
.iter()
|
||||
.find(|m| m.endpoint_id == replica_id)
|
||||
.filter(|m| m.hostname.as_deref() == Some("old-device"))?
|
||||
.hostname_conflict
|
||||
.is_none()
|
||||
.then_some(())
|
||||
})
|
||||
.await;
|
||||
let returned =
|
||||
Agent::spawn(config_with(target_dir.path(), &discovery).with_hostname("old-device"))
|
||||
.await
|
||||
.unwrap();
|
||||
wait_for_peers(&returned, id, 2).await;
|
||||
wait_until("returning identity publishes a fresh version", || async {
|
||||
let m = cleaner
|
||||
.network_status(id)
|
||||
.await
|
||||
.ok()?
|
||||
.members
|
||||
.into_iter()
|
||||
.find(|m| m.endpoint_id == peer)?;
|
||||
(m.online && m.record_version > original.record_version && m.hostname_conflict.is_some())
|
||||
.then_some(())
|
||||
})
|
||||
.await;
|
||||
cleaner.shutdown().await;
|
||||
replica.shutdown().await;
|
||||
returned.shutdown().await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn forgetting_is_network_scoped_and_every_member_has_the_same_right() {
|
||||
let discovery = SharedMemoryDiscovery::new();
|
||||
let a = TestAgent::spawn(&discovery).await.unwrap();
|
||||
let b = TestAgent::spawn(&discovery).await.unwrap();
|
||||
let target = TestAgent::spawn(&discovery).await.unwrap();
|
||||
let (name, secret) = network("forget-one-network");
|
||||
let (other_name, other_secret) = network("keep-other-network");
|
||||
let id = a.agent.join_network(&name, &secret).await.unwrap();
|
||||
b.agent.join_network(&name, &secret).await.unwrap();
|
||||
target.agent.join_network(&name, &secret).await.unwrap();
|
||||
let other = a
|
||||
.agent
|
||||
.join_network(&other_name, &other_secret)
|
||||
.await
|
||||
.unwrap();
|
||||
target
|
||||
.agent
|
||||
.join_network(&other_name, &other_secret)
|
||||
.await
|
||||
.unwrap();
|
||||
let peer = target.agent.endpoint_id();
|
||||
let record = member(&b.agent, id, peer).await;
|
||||
member(&a.agent, other, peer).await;
|
||||
target.agent.shutdown().await;
|
||||
wait_until("offline on B", || async {
|
||||
b.agent
|
||||
.network_status(id)
|
||||
.await
|
||||
.ok()?
|
||||
.members
|
||||
.iter()
|
||||
.find(|m| m.endpoint_id == peer)
|
||||
.filter(|m| !m.online)
|
||||
.map(|_| ())
|
||||
})
|
||||
.await;
|
||||
b.agent
|
||||
.forget_node(id, peer, record.record_version)
|
||||
.await
|
||||
.unwrap();
|
||||
wait_until("A learns B's retirement", || async {
|
||||
a.agent
|
||||
.network_status(id)
|
||||
.await
|
||||
.ok()?
|
||||
.members
|
||||
.iter()
|
||||
.all(|m| m.endpoint_id != peer)
|
||||
.then_some(())
|
||||
})
|
||||
.await;
|
||||
assert!(
|
||||
a.agent
|
||||
.network_status(other)
|
||||
.await
|
||||
.unwrap()
|
||||
.members
|
||||
.iter()
|
||||
.any(|m| m.endpoint_id == peer)
|
||||
);
|
||||
a.agent.shutdown().await;
|
||||
b.agent.shutdown().await;
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
//! Exercise the vendored retry admission in the default workspace suite.
|
||||
//! Transport integration tests still use real iroh endpoints.
|
||||
#[path = "../../../vendor/iroh/src/socket/remote_map/remote_state/pending_paths.rs"]
|
||||
mod pending_paths;
|
||||
@@ -162,7 +162,7 @@ async fn rejoining_after_leaving_is_not_taken_for_a_stale_record() {
|
||||
})
|
||||
.await;
|
||||
assert!(
|
||||
tsunagi::state::DEFAULT_IPV4_RANGE.contains(again),
|
||||
tsunagi::state::derived_ipv4_range(network_id).contains(again),
|
||||
"an address in the network's range: {again}"
|
||||
);
|
||||
|
||||
|
||||
@@ -0,0 +1,308 @@
|
||||
//! Connectivity does not depend on what kind of link each hop happens to be,
|
||||
//! and a member is online however far away it is.
|
||||
|
||||
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
|
||||
|
||||
use std::collections::{HashMap, HashSet};
|
||||
use std::sync::{Arc, Mutex};
|
||||
|
||||
use bytes::Bytes;
|
||||
use iroh::EndpointId;
|
||||
use tsunagi::NetworkId;
|
||||
use tsunagi::agent::DataPath;
|
||||
use tsunagi::dataplane::transport::SharedLink;
|
||||
use tsunagi::dataplane::{IpPlugin, PluginCapability, PluginError};
|
||||
use tsunagi::discovery::SharedMemoryDiscovery;
|
||||
use tsunagi::testing::{TestAgent, network, wait_for_peers, wait_until};
|
||||
|
||||
/// A protocol that moves bytes between two members unchanged and keeps the
|
||||
/// link it was handed, so a test can use the link directly.
|
||||
#[derive(Debug)]
|
||||
struct Carrier {
|
||||
protocol: String,
|
||||
links: Mutex<HashMap<EndpointId, SharedLink>>,
|
||||
}
|
||||
|
||||
impl Carrier {
|
||||
fn new(protocol: &str) -> Arc<Self> {
|
||||
Arc::new(Self {
|
||||
protocol: protocol.into(),
|
||||
links: Mutex::default(),
|
||||
})
|
||||
}
|
||||
|
||||
fn link(&self, peer: EndpointId) -> Option<SharedLink> {
|
||||
self.links.lock().unwrap().get(&peer).cloned()
|
||||
}
|
||||
}
|
||||
|
||||
impl IpPlugin for Carrier {
|
||||
fn protocol_id(&self) -> &str {
|
||||
&self.protocol
|
||||
}
|
||||
|
||||
fn protocol_version(&self) -> u16 {
|
||||
1
|
||||
}
|
||||
|
||||
fn local_capability(
|
||||
&self,
|
||||
_network: NetworkId,
|
||||
) -> Result<Option<PluginCapability>, PluginError> {
|
||||
Ok(Some(PluginCapability {
|
||||
protocol: self.protocol.clone(),
|
||||
version: 1,
|
||||
enabled: true,
|
||||
data: Vec::new(),
|
||||
}))
|
||||
}
|
||||
|
||||
fn on_peer_capability(
|
||||
&self,
|
||||
_network: NetworkId,
|
||||
_peer: EndpointId,
|
||||
_capability: &PluginCapability,
|
||||
) -> Result<(), PluginError> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn on_peer_link(&self, _network: NetworkId, peer: EndpointId, link: SharedLink) {
|
||||
self.links.lock().unwrap().insert(peer, link);
|
||||
}
|
||||
|
||||
fn on_peer_gone(&self, _network: NetworkId, peer: EndpointId) {
|
||||
self.links.lock().unwrap().remove(&peer);
|
||||
}
|
||||
|
||||
fn on_network_deactivated(&self, _network: NetworkId) {}
|
||||
}
|
||||
|
||||
type Blocked = Arc<Mutex<HashSet<EndpointId>>>;
|
||||
|
||||
async fn member(
|
||||
discovery: &SharedMemoryDiscovery,
|
||||
plugins: &[Arc<Carrier>],
|
||||
blocked: &Blocked,
|
||||
) -> TestAgent {
|
||||
let plugins: Vec<_> = plugins.to_vec();
|
||||
let blocked = Arc::clone(blocked);
|
||||
TestAgent::spawn_with(
|
||||
move |config| {
|
||||
let config = config.with_unreachable_data_peers(blocked);
|
||||
plugins
|
||||
.into_iter()
|
||||
.fold(config, |config, plugin| config.with_plugin(plugin))
|
||||
},
|
||||
discovery,
|
||||
)
|
||||
.await
|
||||
.unwrap()
|
||||
}
|
||||
|
||||
async fn exchange(from: &SharedLink, to: &SharedLink, payload: &'static [u8]) {
|
||||
from.send(Bytes::from_static(payload)).unwrap();
|
||||
let got = tokio::time::timeout(tsunagi::testing::DEADLINE, to.recv())
|
||||
.await
|
||||
.expect("the datagram should arrive")
|
||||
.expect("the link stays open");
|
||||
assert_eq!(got, Bytes::from_static(payload));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_tunnel_crosses_a_member_that_shares_no_protocol_with_either_end_and_moves_to_a_direct_link_by_itself()
|
||||
{
|
||||
// A and B share one protocol and nothing else. C shares a different
|
||||
// one with each, so every link here is of a different kind and the
|
||||
// middle has never heard of what A and B speak. A and B cannot reach
|
||||
// each other directly, which is the case a blocked UDP path produces.
|
||||
let discovery = SharedMemoryDiscovery::new();
|
||||
let (name, secret) = network("mixed-transports");
|
||||
|
||||
let blocked_a: Blocked = Arc::default();
|
||||
let blocked_b: Blocked = Arc::default();
|
||||
let a_shared = Carrier::new("shared");
|
||||
let b_shared = Carrier::new("shared");
|
||||
let a = member(
|
||||
&discovery,
|
||||
&[Carrier::new("a-and-c"), Arc::clone(&a_shared)],
|
||||
&blocked_a,
|
||||
)
|
||||
.await;
|
||||
let b = member(
|
||||
&discovery,
|
||||
&[Carrier::new("c-and-b"), Arc::clone(&b_shared)],
|
||||
&blocked_b,
|
||||
)
|
||||
.await;
|
||||
let c = member(
|
||||
&discovery,
|
||||
&[Carrier::new("a-and-c"), Carrier::new("c-and-b")],
|
||||
&Blocked::default(),
|
||||
)
|
||||
.await;
|
||||
let (a_id, b_id, c_id) = (
|
||||
a.agent.endpoint_id(),
|
||||
b.agent.endpoint_id(),
|
||||
c.agent.endpoint_id(),
|
||||
);
|
||||
blocked_a.lock().unwrap().insert(b_id);
|
||||
blocked_b.lock().unwrap().insert(a_id);
|
||||
|
||||
let network_id = a.agent.join_network(&name, &secret).await.unwrap();
|
||||
b.agent.join_network(&name, &secret).await.unwrap();
|
||||
c.agent.join_network(&name, &secret).await.unwrap();
|
||||
for agent in [&a, &b, &c] {
|
||||
wait_for_peers(&agent.agent, network_id, 2).await;
|
||||
}
|
||||
|
||||
// The tunnel comes up through C, and data goes both ways.
|
||||
let to_b = wait_until("a has a link to b", || async { a_shared.link(b_id) }).await;
|
||||
let to_a = wait_until("b has a link to a", || async { b_shared.link(a_id) }).await;
|
||||
exchange(&to_b, &to_a, b"a to b, sealed").await;
|
||||
exchange(&to_a, &to_b, b"b to a, sealed").await;
|
||||
|
||||
// The status says what is true: online, and through whom.
|
||||
let status = a.agent.network_status(network_id).await.unwrap();
|
||||
let b_peer = status.peers.iter().find(|p| p.endpoint_id == b_id).unwrap();
|
||||
assert_eq!(b_peer.data_path, DataPath::Relayed { hops: 2, via: c_id });
|
||||
let c_peer = status.peers.iter().find(|p| p.endpoint_id == c_id).unwrap();
|
||||
assert!(matches!(&c_peer.data_path, DataPath::Direct { transport } if transport == "a-and-c"));
|
||||
let b_member = wait_until("a lists b as a member", || async {
|
||||
let status = a.agent.network_status(network_id).await.ok()?;
|
||||
status.members.into_iter().find(|m| m.endpoint_id == b_id)
|
||||
})
|
||||
.await;
|
||||
assert!(b_member.online);
|
||||
let carried = c.agent.network_status(network_id).await.unwrap().relay;
|
||||
assert!(carried.forwarded >= 2, "{carried:?}");
|
||||
|
||||
// The direct path becomes possible: the same link carries on, now
|
||||
// without C, and nobody has to do anything about it.
|
||||
blocked_a.lock().unwrap().clear();
|
||||
blocked_b.lock().unwrap().clear();
|
||||
wait_until("a reaches b directly", || async {
|
||||
let status = a.agent.network_status(network_id).await.ok()?;
|
||||
let peer = status.peers.iter().find(|p| p.endpoint_id == b_id)?;
|
||||
matches!(&peer.data_path, DataPath::Direct { transport } if transport == "shared")
|
||||
.then_some(())
|
||||
})
|
||||
.await;
|
||||
let forwarded = c
|
||||
.agent
|
||||
.network_status(network_id)
|
||||
.await
|
||||
.unwrap()
|
||||
.relay
|
||||
.forwarded;
|
||||
exchange(&to_b, &to_a, b"direct now").await;
|
||||
assert!(
|
||||
Arc::ptr_eq(&to_b, &a_shared.link(b_id).unwrap()),
|
||||
"the tunnel's link is the one it always had"
|
||||
);
|
||||
assert_eq!(
|
||||
c.agent
|
||||
.network_status(network_id)
|
||||
.await
|
||||
.unwrap()
|
||||
.relay
|
||||
.forwarded,
|
||||
forwarded,
|
||||
"nothing goes through C any more"
|
||||
);
|
||||
|
||||
a.stop().await;
|
||||
b.stop().await;
|
||||
c.stop().await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_name_belongs_to_the_member_that_claimed_it_first_on_every_member() {
|
||||
let discovery = SharedMemoryDiscovery::new();
|
||||
let (name, secret) = network("first-claim");
|
||||
let spawn = |hostname: &'static str| {
|
||||
let discovery = discovery.clone();
|
||||
async move {
|
||||
TestAgent::spawn_with(move |config| config.with_hostname(hostname), &discovery)
|
||||
.await
|
||||
.unwrap()
|
||||
}
|
||||
};
|
||||
|
||||
let first = spawn("music").await;
|
||||
let network_id = first.agent.join_network(&name, &secret).await.unwrap();
|
||||
let first_id = first.agent.endpoint_id();
|
||||
wait_until("the first claim is signed", || async {
|
||||
let status = first.agent.network_status(network_id).await.ok()?;
|
||||
status
|
||||
.members
|
||||
.iter()
|
||||
.any(|m| m.endpoint_id == first_id && m.hostname.as_deref() == Some("music"))
|
||||
.then_some(())
|
||||
})
|
||||
.await;
|
||||
|
||||
// Whatever the ids turn out to be, the later one does not take it.
|
||||
let second = spawn("music").await;
|
||||
let third = spawn("studio").await;
|
||||
second.agent.join_network(&name, &secret).await.unwrap();
|
||||
third.agent.join_network(&name, &secret).await.unwrap();
|
||||
let second_id = second.agent.endpoint_id();
|
||||
for agent in [&first, &second, &third] {
|
||||
wait_for_peers(&agent.agent, network_id, 2).await;
|
||||
}
|
||||
|
||||
for (viewer, label) in [(&first, "first"), (&second, "second"), (&third, "third")] {
|
||||
let status = wait_until(&format!("{label} sees both claims"), || async {
|
||||
let status = viewer.agent.network_status(network_id).await.ok()?;
|
||||
// The roster is the authors of signed records, so a member
|
||||
// being in it means its claim has arrived.
|
||||
let both = [first_id, second_id]
|
||||
.iter()
|
||||
.all(|id| status.members.iter().any(|m| m.endpoint_id == *id));
|
||||
both.then_some(status)
|
||||
})
|
||||
.await;
|
||||
let holder = status
|
||||
.members
|
||||
.iter()
|
||||
.find(|m| m.endpoint_id == first_id)
|
||||
.unwrap();
|
||||
let loser = status
|
||||
.members
|
||||
.iter()
|
||||
.find(|m| m.endpoint_id == second_id)
|
||||
.unwrap();
|
||||
assert_eq!(holder.hostname.as_deref(), Some("music"), "{label}");
|
||||
assert_eq!(loser.hostname, None, "{label}: no name for the later one");
|
||||
let conflict = loser.hostname_conflict.as_ref().expect(label);
|
||||
assert_eq!(conflict.name, "music");
|
||||
assert_eq!(conflict.holder, first_id);
|
||||
assert!(holder.hostname_conflict.is_none(), "{label}");
|
||||
// Nothing about being second makes it less of a member.
|
||||
assert!(
|
||||
loser.online || viewer.agent.endpoint_id() == second_id,
|
||||
"{label}"
|
||||
);
|
||||
}
|
||||
|
||||
// The later one is told so itself, and keeps working.
|
||||
let own = second.agent.network_status(network_id).await.unwrap();
|
||||
let conflict = own.hostname_conflict.expect("told its name is taken");
|
||||
assert_eq!(
|
||||
(conflict.name.as_str(), conflict.holder),
|
||||
("music", first_id)
|
||||
);
|
||||
assert!(
|
||||
first
|
||||
.agent
|
||||
.network_status(network_id)
|
||||
.await
|
||||
.unwrap()
|
||||
.hostname_conflict
|
||||
.is_none()
|
||||
);
|
||||
|
||||
first.stop().await;
|
||||
second.stop().await;
|
||||
third.stop().await;
|
||||
}
|
||||
@@ -265,10 +265,7 @@ async fn deactivating_one_network_leaves_the_agent_and_others_running() {
|
||||
|
||||
#[tokio::test]
|
||||
async fn two_networks_on_one_agent_each_get_a_range_of_their_own() {
|
||||
// One agent, one interface: the configured range can only belong to one
|
||||
// of them. The second does not go without — it takes the range derived
|
||||
// from its own id, which every one of its members derives identically,
|
||||
// so it is an agreement and not a local invention.
|
||||
// Each network derives its range independently of activation order.
|
||||
let discovery = SharedMemoryDiscovery::new();
|
||||
let (first, first_secret) = network("two-ranges-one");
|
||||
let (second, second_secret) = network("two-ranges-two");
|
||||
@@ -309,8 +306,8 @@ async fn two_networks_on_one_agent_each_get_a_range_of_their_own() {
|
||||
"different networks, different addresses"
|
||||
);
|
||||
assert!(
|
||||
tsunagi::state::DEFAULT_IPV4_RANGE.contains(addresses.0),
|
||||
"the first keeps the configured range: {}",
|
||||
tsunagi::state::derived_ipv4_range(one).contains(addresses.0),
|
||||
"the first uses its derived range: {}",
|
||||
addresses.0
|
||||
);
|
||||
assert!(
|
||||
@@ -319,7 +316,7 @@ async fn two_networks_on_one_agent_each_get_a_range_of_their_own() {
|
||||
addresses.1
|
||||
);
|
||||
assert!(
|
||||
!tsunagi::state::DEFAULT_IPV4_RANGE.contains(addresses.1),
|
||||
!tsunagi::state::derived_ipv4_range(one).contains(addresses.1),
|
||||
"and not in the one the first network holds"
|
||||
);
|
||||
|
||||
|
||||
@@ -0,0 +1,336 @@
|
||||
//! Derived defaults and local exclusion of overlapping overlay networks.
|
||||
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
|
||||
|
||||
use tsunagi::Agent;
|
||||
use tsunagi::discovery::SharedMemoryDiscovery;
|
||||
use tsunagi::state::{DEFAULT_IPV4_RANGE, Ipv4Range, derived_ipv4_range};
|
||||
use tsunagi::testing::{TestAgent, config_with, network, wait_until};
|
||||
|
||||
// These tests need discovery to happen before a collision, so keep the initial
|
||||
// default outside the established ranges. Derived collisions have their own test.
|
||||
fn network_outside(
|
||||
name: &str,
|
||||
ranges: &[Ipv4Range],
|
||||
) -> (
|
||||
tsunagi::identity::NetworkName,
|
||||
tsunagi::identity::NetworkSecret,
|
||||
) {
|
||||
use tsunagi::identity::{NetworkKeys, NetworkName, NetworkSecret};
|
||||
let name = NetworkName::new(name).unwrap();
|
||||
let secret = (0..=255u8)
|
||||
.map(|seed| NetworkSecret::from_bytes([seed; 32]).unwrap())
|
||||
.find(|secret| {
|
||||
let derived = derived_ipv4_range(NetworkKeys::derive(&name, secret).network_id());
|
||||
ranges.iter().all(|range| !range.contains(derived.base))
|
||||
})
|
||||
.expect("a deterministic test secret with a non-overlapping default");
|
||||
(name, secret)
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn colliding_derived_defaults_are_blocked_instead_of_locally_renumbered() {
|
||||
use std::collections::HashMap;
|
||||
use tsunagi::identity::{NetworkKeys, NetworkName, NetworkSecret};
|
||||
let secret = NetworkSecret::from_bytes([19; 32]).unwrap();
|
||||
let mut seen = HashMap::new();
|
||||
let (first, second, range) = (0..=65_536)
|
||||
.find_map(|index| {
|
||||
let name = NetworkName::new(format!("derived-collision-{index}")).unwrap();
|
||||
let range = derived_ipv4_range(NetworkKeys::derive(&name, &secret).network_id());
|
||||
seen.insert(range.base, name.clone())
|
||||
.map(|first| (first, name, range))
|
||||
})
|
||||
.expect("more network names than possible /24 ranges");
|
||||
let agent = TestAgent::spawn(&SharedMemoryDiscovery::new())
|
||||
.await
|
||||
.unwrap();
|
||||
let first = agent.agent.join_network(&first, &secret).await.unwrap();
|
||||
let second = agent.agent.join_network(&second, &secret).await.unwrap();
|
||||
assert!(agent.agent.is_active(first).await);
|
||||
assert!(!agent.agent.is_active(second).await);
|
||||
let status = agent.agent.status().await.unwrap();
|
||||
let blocked = status.network(&second).unwrap();
|
||||
assert_eq!(blocked.range, Some(range));
|
||||
assert_eq!(blocked.range_conflicts[0].network_id, first);
|
||||
assert_eq!(
|
||||
agent.agent.switch_network(second).await.unwrap(),
|
||||
vec![first]
|
||||
);
|
||||
assert_eq!(
|
||||
agent.agent.network_status(second).await.unwrap().range,
|
||||
Some(range)
|
||||
);
|
||||
agent.agent.shutdown().await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn isolated_first_members_choose_the_same_range_and_keep_it_after_restart() {
|
||||
let discovery = SharedMemoryDiscovery::new();
|
||||
let (name, secret) = network("derived-first");
|
||||
let a = TestAgent::spawn(&discovery).await.unwrap();
|
||||
let b = TestAgent::spawn(&SharedMemoryDiscovery::new())
|
||||
.await
|
||||
.unwrap();
|
||||
let id = a.agent.join_network(&name, &secret).await.unwrap();
|
||||
b.agent.join_network(&name, &secret).await.unwrap();
|
||||
let expected = derived_ipv4_range(id);
|
||||
for agent in [&a.agent, &b.agent] {
|
||||
wait_until("isolated member allocates in its derived range", || async {
|
||||
let status = agent.network_status(id).await.ok()?;
|
||||
(status.range == Some(expected) && !status.members.is_empty()).then_some(())
|
||||
})
|
||||
.await;
|
||||
}
|
||||
let dir = a.stop().await;
|
||||
let a = Agent::spawn(config_with(dir.path(), &discovery))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(a.network_status(id).await.unwrap().range, Some(expected));
|
||||
a.shutdown().await;
|
||||
b.agent.shutdown().await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn overlap_blocks_activation_switches_and_resumes_across_restart() {
|
||||
let discovery = SharedMemoryDiscovery::new();
|
||||
let a = TestAgent::spawn_with(
|
||||
|c| c.with_overlay_ipv4_range(Some(DEFAULT_IPV4_RANGE)),
|
||||
&discovery,
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
let (one_name, one_secret) = network("conflict-one");
|
||||
let (two_name, two_secret) = network("conflict-two");
|
||||
let one = a.agent.join_network(&one_name, &one_secret).await.unwrap();
|
||||
a.agent.network_status(one).await.unwrap();
|
||||
let two = a.agent.join_network(&two_name, &two_secret).await.unwrap();
|
||||
assert!(a.agent.is_active(one).await);
|
||||
assert!(!a.agent.is_active(two).await);
|
||||
assert!(matches!(
|
||||
a.agent.activate_network(two).await,
|
||||
Err(tsunagi::Error::NetworkRangeConflict { .. })
|
||||
));
|
||||
let status = a.agent.status().await.unwrap();
|
||||
let blocked = status.network(&two).unwrap();
|
||||
assert_eq!(blocked.range, Some(DEFAULT_IPV4_RANGE));
|
||||
assert_eq!(blocked.range_conflicts[0].network_id, one);
|
||||
assert_eq!(blocked.range_conflicts[0].name, one_name);
|
||||
assert_eq!(a.agent.switch_network(two).await.unwrap(), vec![one]);
|
||||
assert!(a.agent.is_active(two).await);
|
||||
assert!(!a.agent.is_active(one).await);
|
||||
|
||||
a.agent.network_status(two).await.unwrap();
|
||||
let dir = a.stop().await;
|
||||
let a = Agent::spawn(config_with(dir.path(), &discovery))
|
||||
.await
|
||||
.unwrap();
|
||||
assert!(a.is_active(two).await, "the user's choice survives restart");
|
||||
assert!(!a.is_active(one).await);
|
||||
a.deactivate_network(two).await.unwrap();
|
||||
assert!(
|
||||
a.is_active(one).await,
|
||||
"suspended networks resume when the blocker stops"
|
||||
);
|
||||
assert_eq!(
|
||||
a.network_status(one).await.unwrap().range,
|
||||
Some(DEFAULT_IPV4_RANGE),
|
||||
"legacy ranges survive the new default"
|
||||
);
|
||||
a.shutdown().await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn late_authenticated_state_with_a_nested_range_suspends_only_that_network() {
|
||||
let discovery = SharedMemoryDiscovery::new();
|
||||
let broad: Ipv4Range = "10.90.0.0/16".parse().unwrap();
|
||||
let narrow: Ipv4Range = "10.90.12.0/24".parse().unwrap();
|
||||
let (name, secret) = network_outside("late-conflict", &[broad]);
|
||||
let remote = TestAgent::spawn_with(|c| c.with_overlay_ipv4_range(Some(narrow)), &discovery)
|
||||
.await
|
||||
.unwrap();
|
||||
let target = remote.agent.join_network(&name, &secret).await.unwrap();
|
||||
wait_until("remote range is persisted", || async {
|
||||
remote
|
||||
.agent
|
||||
.network_status(target)
|
||||
.await
|
||||
.ok()?
|
||||
.members
|
||||
.first()
|
||||
.map(|_| ())
|
||||
})
|
||||
.await;
|
||||
|
||||
let (other_name, other_secret) = network("existing-broad");
|
||||
// Seed an existing network using a real agent and its signed claim.
|
||||
let seeded = TestAgent::spawn_with(
|
||||
|c| c.with_overlay_ipv4_range(Some(broad)),
|
||||
&SharedMemoryDiscovery::new(),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
let other = seeded
|
||||
.agent
|
||||
.join_network(&other_name, &other_secret)
|
||||
.await
|
||||
.unwrap();
|
||||
wait_until("broad range is persisted", || async {
|
||||
seeded
|
||||
.agent
|
||||
.network_status(other)
|
||||
.await
|
||||
.ok()?
|
||||
.members
|
||||
.first()
|
||||
.map(|_| ())
|
||||
})
|
||||
.await;
|
||||
let dir = seeded.stop().await;
|
||||
let seeded = Agent::spawn(config_with(dir.path(), &discovery))
|
||||
.await
|
||||
.unwrap();
|
||||
seeded.join_network(&name, &secret).await.unwrap();
|
||||
wait_until("late overlap suspends the joining network", || async {
|
||||
let status = seeded.status().await.ok()?;
|
||||
let blocked = status.network(&target)?;
|
||||
(!seeded.is_active(target).await
|
||||
&& blocked
|
||||
.range_conflicts
|
||||
.iter()
|
||||
.any(|c| c.network_id == other))
|
||||
.then_some(())
|
||||
})
|
||||
.await;
|
||||
assert!(seeded.is_active(other).await);
|
||||
assert_eq!(seeded.switch_network(target).await.unwrap(), vec![other]);
|
||||
assert_eq!(
|
||||
seeded.network_status(target).await.unwrap().range,
|
||||
Some(narrow)
|
||||
);
|
||||
seeded.shutdown().await;
|
||||
remote.agent.shutdown().await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_wide_range_suspends_all_overlaps_and_manual_stop_cancels_resume() {
|
||||
let discovery = SharedMemoryDiscovery::new();
|
||||
let hub = TestAgent::spawn(&discovery).await.unwrap();
|
||||
let mut members = Vec::new();
|
||||
let mut ids = Vec::new();
|
||||
for (name, range) in [
|
||||
("narrow-one", "10.81.0.0/24"),
|
||||
("narrow-two", "10.81.1.0/24"),
|
||||
] {
|
||||
let peer = TestAgent::spawn_with(
|
||||
|c| c.with_overlay_ipv4_range(Some(range.parse().unwrap())),
|
||||
&discovery,
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
let (name, secret) = network_outside(
|
||||
name,
|
||||
&[
|
||||
"10.81.0.0/24".parse().unwrap(),
|
||||
"10.81.1.0/24".parse().unwrap(),
|
||||
],
|
||||
);
|
||||
let id = peer.agent.join_network(&name, &secret).await.unwrap();
|
||||
peer.agent.network_status(id).await.unwrap();
|
||||
hub.agent.join_network(&name, &secret).await.unwrap();
|
||||
wait_until("hub adopts the established narrow range", || async {
|
||||
(hub.agent.network_status(id).await.ok()?.range == Some(range.parse().unwrap()))
|
||||
.then_some(())
|
||||
})
|
||||
.await;
|
||||
members.push(peer);
|
||||
ids.push(id);
|
||||
}
|
||||
let (name, secret) = network_outside(
|
||||
"wide-overlap",
|
||||
&[
|
||||
"10.81.0.0/24".parse().unwrap(),
|
||||
"10.81.1.0/24".parse().unwrap(),
|
||||
],
|
||||
);
|
||||
let wide = TestAgent::spawn_with(
|
||||
|c| c.with_overlay_ipv4_range(Some("10.0.0.0/8".parse().unwrap())),
|
||||
&discovery,
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
let id = wide.agent.join_network(&name, &secret).await.unwrap();
|
||||
wide.agent.network_status(id).await.unwrap();
|
||||
hub.agent.join_network(&name, &secret).await.unwrap();
|
||||
wait_until("wide range is blocked by both narrow networks", || async {
|
||||
let status = hub.agent.status().await.ok()?;
|
||||
let blocked = status.network(&id)?;
|
||||
(!hub.agent.is_active(id).await && blocked.range_conflicts.len() == 2).then_some(())
|
||||
})
|
||||
.await;
|
||||
let stopped = hub.agent.switch_network(id).await.unwrap();
|
||||
assert_eq!(stopped.len(), 2);
|
||||
assert!(ids.iter().all(|id| stopped.contains(id)));
|
||||
assert_eq!(hub.agent.switch_network(ids[0]).await.unwrap(), vec![id]);
|
||||
assert!(
|
||||
hub.agent.is_active(ids[1]).await,
|
||||
"switching away from the wide range frees the other narrow range too"
|
||||
);
|
||||
assert_eq!(hub.agent.switch_network(id).await.unwrap().len(), 2);
|
||||
hub.agent.pause_network(ids[0]).await.unwrap();
|
||||
hub.agent.deactivate_network(id).await.unwrap();
|
||||
assert!(
|
||||
!hub.agent.is_active(ids[0]).await,
|
||||
"explicitly stopped networks do not resume"
|
||||
);
|
||||
assert!(hub.agent.is_active(ids[1]).await);
|
||||
assert!(
|
||||
hub.agent
|
||||
.switch_network(tsunagi::NetworkId::from_bytes([0; 32]))
|
||||
.await
|
||||
.is_err()
|
||||
);
|
||||
assert!(
|
||||
hub.agent.is_active(ids[1]).await,
|
||||
"unknown selectors do not stop any network"
|
||||
);
|
||||
hub.agent.shutdown().await;
|
||||
wide.agent.shutdown().await;
|
||||
for peer in members {
|
||||
peer.agent.shutdown().await;
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn simultaneous_switches_never_leave_overlapping_networks_active() {
|
||||
let discovery = SharedMemoryDiscovery::new();
|
||||
let agent = TestAgent::spawn_with(
|
||||
|c| c.with_overlay_ipv4_range(Some(DEFAULT_IPV4_RANGE)),
|
||||
&discovery,
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
let (a, a_secret) = network("switch-a");
|
||||
let (b, b_secret) = network("switch-b");
|
||||
let a = agent.agent.join_network(&a, &a_secret).await.unwrap();
|
||||
agent.agent.network_status(a).await.unwrap();
|
||||
let b = agent.agent.join_network(&b, &b_secret).await.unwrap();
|
||||
let (left, right) = tokio::join!(agent.agent.switch_network(b), agent.agent.switch_network(a));
|
||||
left.unwrap();
|
||||
right.unwrap();
|
||||
assert_ne!(
|
||||
agent.agent.is_active(a).await,
|
||||
agent.agent.is_active(b).await
|
||||
);
|
||||
assert_eq!(
|
||||
agent
|
||||
.agent
|
||||
.list_networks()
|
||||
.await
|
||||
.unwrap()
|
||||
.iter()
|
||||
.filter(|n| n.active)
|
||||
.count(),
|
||||
1
|
||||
);
|
||||
agent.agent.shutdown().await;
|
||||
}
|
||||
@@ -0,0 +1,130 @@
|
||||
//! Opaque transport negotiation stays inside authenticated network sessions.
|
||||
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
|
||||
|
||||
use std::sync::{Arc, Mutex};
|
||||
use tsunagi::dataplane::transport::{ControlSender, PacketTransport, SharedLink, TransportError};
|
||||
use tsunagi::discovery::SharedMemoryDiscovery;
|
||||
use tsunagi::proto::message::{ControlMessage, TransportPayload};
|
||||
use tsunagi::testing::{TestAgent, network, wait_for_peers, wait_until};
|
||||
use tsunagi::{BoxFuture, NetworkId};
|
||||
|
||||
#[derive(Debug, Default)]
|
||||
struct Negotiator {
|
||||
control: Mutex<Option<Arc<dyn ControlSender>>>,
|
||||
received: Mutex<Vec<(NetworkId, iroh::EndpointId, Vec<u8>)>>,
|
||||
}
|
||||
impl PacketTransport for Negotiator {
|
||||
fn name(&self) -> &str {
|
||||
"negotiator"
|
||||
}
|
||||
fn as_any(&self) -> &dyn std::any::Any {
|
||||
self
|
||||
}
|
||||
fn attach_control(&self, control: Arc<dyn ControlSender>) {
|
||||
*self.control.lock().unwrap() = Some(control);
|
||||
}
|
||||
fn on_control(
|
||||
&self,
|
||||
network: NetworkId,
|
||||
peer: iroh::EndpointId,
|
||||
_: &str,
|
||||
payload: &[u8],
|
||||
) -> Result<(), TransportError> {
|
||||
self.received
|
||||
.lock()
|
||||
.unwrap()
|
||||
.push((network, peer, payload.to_vec()));
|
||||
Ok(())
|
||||
}
|
||||
fn open<'a>(
|
||||
&'a self,
|
||||
_: NetworkId,
|
||||
_: iroh::EndpointId,
|
||||
_: &'a str,
|
||||
) -> BoxFuture<'a, Result<SharedLink, TransportError>> {
|
||||
Box::pin(async { Err(TransportError::Closed) })
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn negotiated_secrets_are_redacted_scoped_and_delivered_over_real_control() {
|
||||
let discovery = SharedMemoryDiscovery::new();
|
||||
let a_transport = Arc::new(Negotiator::default());
|
||||
let b_transport = Arc::new(Negotiator::default());
|
||||
let a = TestAgent::spawn_with(
|
||||
|c| c.with_custom_transport("negotiator", a_transport.clone()),
|
||||
&discovery,
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
let b = TestAgent::spawn_with(
|
||||
|c| c.with_custom_transport("negotiator", b_transport.clone()),
|
||||
&discovery,
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
let (name, secret) = network("private-negotiation");
|
||||
let id = a.agent.join_network(&name, &secret).await.unwrap();
|
||||
b.agent.join_network(&name, &secret).await.unwrap();
|
||||
wait_for_peers(&a.agent, id, 1).await;
|
||||
let message = ControlMessage::Transport {
|
||||
protocol: "negotiator".into(),
|
||||
payload: TransportPayload::new(b"never-export-this-secret".to_vec()),
|
||||
};
|
||||
assert!(!format!("{message:?}").contains("never-export"));
|
||||
let sender = a_transport.control.lock().unwrap().clone().unwrap();
|
||||
sender
|
||||
.send(
|
||||
id,
|
||||
b.agent.endpoint_id(),
|
||||
"negotiator",
|
||||
TransportPayload::new(b"never-export-this-secret".to_vec()),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
wait_until("private transport message", || async {
|
||||
(!b_transport.received.lock().unwrap().is_empty()).then_some(())
|
||||
})
|
||||
.await;
|
||||
assert_eq!(
|
||||
b_transport.received.lock().unwrap()[0],
|
||||
(
|
||||
id,
|
||||
a.agent.endpoint_id(),
|
||||
b"never-export-this-secret".to_vec()
|
||||
)
|
||||
);
|
||||
let unknown = NetworkId::from_bytes([0xa5; 32]);
|
||||
assert!(
|
||||
sender
|
||||
.send(
|
||||
unknown,
|
||||
b.agent.endpoint_id(),
|
||||
"negotiator",
|
||||
TransportPayload::new(vec![1])
|
||||
)
|
||||
.await
|
||||
.is_err()
|
||||
);
|
||||
a.agent.shutdown().await;
|
||||
b.agent.shutdown().await;
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn private_bytes_are_bounded_on_decode_and_by_local_policy() {
|
||||
let limits = tsunagi::Limits {
|
||||
max_transport_payload_len: 4,
|
||||
..Default::default()
|
||||
};
|
||||
let message = |len| ControlMessage::Transport {
|
||||
protocol: "awg".into(),
|
||||
payload: TransportPayload::new(vec![42; len]),
|
||||
};
|
||||
assert!(tsunagi::proto::message::validate(&message(4), &limits).is_ok());
|
||||
assert!(tsunagi::proto::message::validate(&message(5), &limits).is_err());
|
||||
let bytes = postcard::to_stdvec(&TransportPayload::new(vec![1; 4097])).unwrap();
|
||||
assert!(postcard::from_bytes::<TransportPayload>(&bytes).is_err());
|
||||
let bytes = postcard::to_stdvec(&message(4)).unwrap();
|
||||
let decoded: ControlMessage = postcard::from_bytes(&bytes).unwrap();
|
||||
assert_eq!(decoded, message(4));
|
||||
}
|
||||
@@ -0,0 +1,207 @@
|
||||
//! Shared services use real membership, signed state and mandatory SQLite.
|
||||
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
|
||||
use tsunagi::discovery::SharedMemoryDiscovery;
|
||||
use tsunagi::testing::{TestAgent, config_with, network, wait_for_peers, wait_until};
|
||||
use tsunagi::vip::{Backend, Listener, Protocol, VipConfig, VipStatus};
|
||||
use tsunagi::{Agent, NetworkId};
|
||||
async fn view(agent: &Agent, network: NetworkId, id: &str) -> VipStatus {
|
||||
wait_until("VIP configuration reaches the replica", || async {
|
||||
agent
|
||||
.network_status(network)
|
||||
.await
|
||||
.ok()?
|
||||
.vips
|
||||
.into_iter()
|
||||
.find(|v| v.config.id == id)
|
||||
})
|
||||
.await
|
||||
}
|
||||
#[tokio::test]
|
||||
async fn any_member_can_edit_services_and_an_offline_creator_is_not_required() {
|
||||
let discovery = SharedMemoryDiscovery::new();
|
||||
let a = TestAgent::spawn(&discovery).await.unwrap();
|
||||
let b = TestAgent::spawn(&discovery).await.unwrap();
|
||||
let (name, secret) = network("shared-vips");
|
||||
let net = a.agent.join_network(&name, &secret).await.unwrap();
|
||||
b.agent.join_network(&name, &secret).await.unwrap();
|
||||
wait_for_peers(&a.agent, net, 1).await;
|
||||
let cfg = VipConfig::new("api");
|
||||
a.agent
|
||||
.save_vip(net, cfg.clone(), None, false)
|
||||
.await
|
||||
.unwrap();
|
||||
let before = view(&b.agent, net, &cfg.id).await;
|
||||
let mut edited = before.config.clone();
|
||||
let mut listener = Listener::new(Protocol::Tcp, 443);
|
||||
listener.backends.push(Backend {
|
||||
peer: b.agent.endpoint_id().to_string(),
|
||||
port: 6443,
|
||||
});
|
||||
edited.listeners.push(listener);
|
||||
b.agent
|
||||
.save_vip(net, edited.clone(), Some(before.revision.clone()), false)
|
||||
.await
|
||||
.unwrap();
|
||||
assert!(
|
||||
b.agent
|
||||
.save_vip(net, cfg.clone(), Some(before.revision), false)
|
||||
.await
|
||||
.is_err()
|
||||
);
|
||||
wait_until("creator observes another member's edit", || async {
|
||||
(view(&a.agent, net, &cfg.id).await.config == edited).then_some(())
|
||||
})
|
||||
.await;
|
||||
let a_dir = a.stop().await;
|
||||
let c = TestAgent::spawn(&discovery).await.unwrap();
|
||||
c.agent.join_network(&name, &secret).await.unwrap();
|
||||
assert_eq!(view(&c.agent, net, &cfg.id).await.config, edited);
|
||||
let a = Agent::spawn(config_with(a_dir.path(), &discovery))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(view(&a, net, &cfg.id).await.config, edited);
|
||||
let current = view(&c.agent, net, &cfg.id).await;
|
||||
c.agent
|
||||
.save_vip(net, current.config, Some(current.revision), true)
|
||||
.await
|
||||
.unwrap();
|
||||
wait_until("deletion reaches the original creator", || async {
|
||||
a.network_status(net)
|
||||
.await
|
||||
.ok()?
|
||||
.vips
|
||||
.is_empty()
|
||||
.then_some(())
|
||||
})
|
||||
.await;
|
||||
a.shutdown().await;
|
||||
b.stop().await;
|
||||
c.stop().await;
|
||||
}
|
||||
#[tokio::test]
|
||||
async fn invalid_drafts_are_atomic_and_services_do_not_cross_networks() {
|
||||
let discovery = SharedMemoryDiscovery::new();
|
||||
let a = TestAgent::spawn(&discovery).await.unwrap();
|
||||
let (name, secret) = network("vip-one");
|
||||
let net = a.agent.join_network(&name, &secret).await.unwrap();
|
||||
let (other, secret) = network("vip-two");
|
||||
let second = a.agent.join_network(&other, &secret).await.unwrap();
|
||||
let mut cfg = VipConfig::new("api");
|
||||
cfg.listeners.push(Listener::new(Protocol::Udp, 53));
|
||||
a.agent
|
||||
.save_vip(net, cfg.clone(), None, false)
|
||||
.await
|
||||
.unwrap();
|
||||
let before = view(&a.agent, net, &cfg.id).await;
|
||||
let mut invalid = before.config.clone();
|
||||
invalid.name = "renamed".into();
|
||||
invalid.listeners.push(invalid.listeners[0].clone());
|
||||
assert!(
|
||||
a.agent
|
||||
.save_vip(net, invalid, Some(before.revision.clone()), false)
|
||||
.await
|
||||
.is_err()
|
||||
);
|
||||
let after = view(&a.agent, net, &cfg.id).await;
|
||||
assert_eq!(after.config, before.config);
|
||||
assert_eq!(after.revision, before.revision);
|
||||
assert!(
|
||||
a.agent
|
||||
.network_status(second)
|
||||
.await
|
||||
.unwrap()
|
||||
.vips
|
||||
.is_empty()
|
||||
);
|
||||
a.stop().await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_restored_author_advances_its_counter_past_its_own_updates_held_by_a_replica() {
|
||||
use tsunagi::config::StoragePaths;
|
||||
use tsunagi::storage::StateStore;
|
||||
use tsunagi::vip::VipSet;
|
||||
let discovery = SharedMemoryDiscovery::new();
|
||||
let a = TestAgent::spawn(&discovery).await.unwrap();
|
||||
let b = TestAgent::spawn(&discovery).await.unwrap();
|
||||
let (name, secret) = network("vip-author-recovery");
|
||||
let net = a.agent.join_network(&name, &secret).await.unwrap();
|
||||
b.agent.join_network(&name, &secret).await.unwrap();
|
||||
wait_for_peers(&a.agent, net, 1).await;
|
||||
let config = VipConfig::new("api");
|
||||
a.agent
|
||||
.save_vip(net, config.clone(), None, false)
|
||||
.await
|
||||
.unwrap();
|
||||
view(&b.agent, net, &config.id).await;
|
||||
let author = a.agent.endpoint_id();
|
||||
let adir = a.stop().await;
|
||||
let bdir = b.stop().await;
|
||||
let apath = StoragePaths::under(adir.path()).state_db();
|
||||
let bpath = StoragePaths::under(bdir.path()).state_db();
|
||||
// Model an old author backup with a newer, genuinely signed update retained
|
||||
// by another offline replica. Connections and recovery below remain real.
|
||||
tokio::task::spawn_blocking(move || {
|
||||
let a = StateStore::open(apath).unwrap();
|
||||
let b = StateStore::open(bpath).unwrap();
|
||||
let mut set = VipSet::default();
|
||||
set.merge(
|
||||
net,
|
||||
a.vip_fields(net, 192).unwrap(),
|
||||
&tsunagi::Limits::default(),
|
||||
)
|
||||
.unwrap();
|
||||
let mut cfg = set.configs().remove(0);
|
||||
cfg.name = "newer".into();
|
||||
let key =
|
||||
iroh::SecretKey::from_bytes(&a.device_identity().unwrap().unwrap().secret_bytes());
|
||||
let fields = set
|
||||
.edit(
|
||||
net,
|
||||
&key,
|
||||
100,
|
||||
&cfg,
|
||||
Some(&set.revision(&cfg.id)),
|
||||
&tsunagi::Limits::default(),
|
||||
)
|
||||
.unwrap();
|
||||
set.merge(net, fields, &tsunagi::Limits::default()).unwrap();
|
||||
b.put_vip_fields(net, &set.entries(&cfg.id), None).unwrap();
|
||||
})
|
||||
.await
|
||||
.unwrap();
|
||||
let b = Agent::spawn(config_with(bdir.path(), &discovery))
|
||||
.await
|
||||
.unwrap();
|
||||
let a = Agent::spawn(config_with(adir.path(), &discovery))
|
||||
.await
|
||||
.unwrap();
|
||||
let learned = wait_until(
|
||||
"the restored author learns its newer signed service",
|
||||
|| async {
|
||||
let v = view(&a, net, &config.id).await;
|
||||
(v.config.name == "newer").then_some(v)
|
||||
},
|
||||
)
|
||||
.await;
|
||||
let mut next = learned.config;
|
||||
next.name = "latest".into();
|
||||
a.save_vip(net, next, Some(learned.revision), false)
|
||||
.await
|
||||
.unwrap();
|
||||
a.shutdown().await;
|
||||
let path = StoragePaths::under(adir.path()).state_db();
|
||||
let version = tokio::task::spawn_blocking(move || {
|
||||
StateStore::open(path)
|
||||
.unwrap()
|
||||
.own_record_version(net, author)
|
||||
.unwrap()
|
||||
})
|
||||
.await
|
||||
.unwrap();
|
||||
assert!(
|
||||
version > 100,
|
||||
"the observed author counter must not be reused"
|
||||
);
|
||||
b.shutdown().await;
|
||||
}
|
||||
@@ -115,6 +115,7 @@ pub struct TcpTlsLink {
|
||||
peer: EndpointId,
|
||||
tx: mpsc::Sender<Bytes>,
|
||||
rx: tokio::sync::Mutex<mpsc::Receiver<Bytes>>,
|
||||
tasks: [tokio::task::AbortHandle; 2],
|
||||
closed_notify: Arc<Notify>,
|
||||
is_closed: Arc<AtomicBool>,
|
||||
remote_addr: String,
|
||||
@@ -136,7 +137,7 @@ impl TcpTlsLink {
|
||||
|
||||
let closed_flag_w = Arc::clone(&is_closed);
|
||||
let closed_notify_w = Arc::clone(&closed_notify);
|
||||
tokio::spawn(async move {
|
||||
let writer_task = tokio::spawn(async move {
|
||||
while let Some(msg) = rx_out.recv().await {
|
||||
if msg.len() > MAX_FRAME_SIZE {
|
||||
continue;
|
||||
@@ -158,7 +159,7 @@ impl TcpTlsLink {
|
||||
|
||||
let closed_flag_r = Arc::clone(&is_closed);
|
||||
let closed_notify_r = Arc::clone(&closed_notify);
|
||||
tokio::spawn(async move {
|
||||
let reader_task = tokio::spawn(async move {
|
||||
let mut len_buf = [0u8; 2];
|
||||
loop {
|
||||
if reader.read_exact(&mut len_buf).await.is_err() {
|
||||
@@ -185,6 +186,7 @@ impl TcpTlsLink {
|
||||
peer,
|
||||
tx: tx_out,
|
||||
rx: tokio::sync::Mutex::new(rx_in),
|
||||
tasks: [writer_task.abort_handle(), reader_task.abort_handle()],
|
||||
closed_notify,
|
||||
is_closed,
|
||||
remote_addr,
|
||||
@@ -192,7 +194,19 @@ impl TcpTlsLink {
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for TcpTlsLink {
|
||||
fn drop(&mut self) {
|
||||
self.close();
|
||||
}
|
||||
}
|
||||
impl PacketLink for TcpTlsLink {
|
||||
fn close(&self) {
|
||||
self.is_closed.store(true, Ordering::Release);
|
||||
self.closed_notify.notify_waiters();
|
||||
for task in &self.tasks {
|
||||
task.abort();
|
||||
}
|
||||
}
|
||||
fn network(&self) -> NetworkId {
|
||||
self.network
|
||||
}
|
||||
|
||||
@@ -21,6 +21,24 @@ const POLL_INTERVAL: Duration = Duration::from_secs(2);
|
||||
|
||||
/// A request from the UI to the agent.
|
||||
pub(crate) enum Command {
|
||||
SaveVip {
|
||||
token: String,
|
||||
network_id: String,
|
||||
config: tsunagi::vip::VipConfig,
|
||||
expected: Option<String>,
|
||||
delete: bool,
|
||||
},
|
||||
/// Retire the exact offline-node claim selected in the devices window.
|
||||
ForgetNode {
|
||||
network_id: String,
|
||||
peer: String,
|
||||
expected_version: u64,
|
||||
},
|
||||
/// Replace the network's enabled protocols in preference order.
|
||||
SetProtocols {
|
||||
network_id: String,
|
||||
protocols: Vec<String>,
|
||||
},
|
||||
/// Stop serving a network, or start serving it again.
|
||||
SetActive { network_id: String, active: bool },
|
||||
/// Turn local broadcast participation on or off for a network.
|
||||
@@ -44,8 +62,14 @@ pub(crate) enum Command {
|
||||
}
|
||||
|
||||
/// The latest the worker knows, read by the UI each frame.
|
||||
#[derive(Clone)]
|
||||
pub(crate) struct VipSaveResult {
|
||||
pub token: String,
|
||||
pub result: Result<Vec<tsunagi::vip::VipStatus>, String>,
|
||||
}
|
||||
#[derive(Clone, Default)]
|
||||
pub(crate) struct Snapshot {
|
||||
pub vip_result: Option<VipSaveResult>,
|
||||
/// The last status, or the reason it could not be read (agent down, no
|
||||
/// permission on the socket, …).
|
||||
pub status: Option<Result<StatusReport, String>>,
|
||||
@@ -73,6 +97,18 @@ pub(crate) struct AgentClient {
|
||||
}
|
||||
|
||||
impl AgentClient {
|
||||
#[cfg(test)]
|
||||
pub(crate) fn test_client() -> (Self, mpsc::UnboundedReceiver<Command>) {
|
||||
let (tx, rx) = mpsc::unbounded_channel();
|
||||
(
|
||||
Self {
|
||||
tx,
|
||||
shared: Arc::new(Mutex::new(Snapshot::default())),
|
||||
pending_copy: Arc::new(Mutex::new(None)),
|
||||
},
|
||||
rx,
|
||||
)
|
||||
}
|
||||
/// Starts the worker on `runtime`, reporting to `ctx` (for repaints).
|
||||
pub(crate) fn spawn(runtime: &tokio::runtime::Handle, ctx: egui::Context) -> Self {
|
||||
let (tx, rx) = mpsc::unbounded_channel();
|
||||
@@ -106,7 +142,13 @@ impl AgentClient {
|
||||
|
||||
/// Sends a command; a dropped worker simply means nothing happens.
|
||||
pub(crate) fn send(&self, command: Command) {
|
||||
let _ = self.tx.send(command);
|
||||
// Disable protocol editing until the refreshed snapshot arrives;
|
||||
// otherwise a second checkbox click could replace the first edit
|
||||
// using the same stale network selection.
|
||||
lock(&self.shared).busy = true;
|
||||
if self.tx.send(command).is_err() {
|
||||
lock(&self.shared).busy = false;
|
||||
}
|
||||
}
|
||||
|
||||
/// A copy of the latest snapshot.
|
||||
@@ -142,6 +184,8 @@ async fn worker(
|
||||
let Some(command) = command else { return };
|
||||
run(&resolve_socket(), command, &shared, &pending_copy, &ctx).await;
|
||||
refresh(&shared, &ctx).await;
|
||||
lock(&shared).busy = false;
|
||||
ctx.request_repaint();
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -179,11 +223,70 @@ async fn run(
|
||||
ctx.request_repaint();
|
||||
|
||||
let outcome: Result<String, String> = match command {
|
||||
Command::SaveVip {
|
||||
token,
|
||||
network_id,
|
||||
config,
|
||||
expected,
|
||||
delete,
|
||||
} => {
|
||||
let result = ipc::save_vip(socket, &network_id, config, expected, delete)
|
||||
.await
|
||||
.map_err(|e| e.to_string());
|
||||
lock(shared).vip_result = Some(VipSaveResult {
|
||||
token,
|
||||
result: result.clone(),
|
||||
});
|
||||
result.map(|_| "VIP configuration saved".into())
|
||||
}
|
||||
Command::ForgetNode {
|
||||
network_id,
|
||||
peer,
|
||||
expected_version,
|
||||
} => ipc::forget_node(socket, &network_id, &peer, expected_version)
|
||||
.await
|
||||
.map(|report| {
|
||||
format!(
|
||||
"forgot node claim v{}; cleanup queued to {} peers",
|
||||
report.record_version, report.peers_told
|
||||
)
|
||||
})
|
||||
.map_err(|err| err.to_string()),
|
||||
Command::SetProtocols {
|
||||
network_id,
|
||||
protocols,
|
||||
} => ipc::set_protocols(socket, &network_id, protocols)
|
||||
.await
|
||||
.map(|names| {
|
||||
format!(
|
||||
"protocols: {}",
|
||||
if names.is_empty() {
|
||||
"none".into()
|
||||
} else {
|
||||
names.join(", ")
|
||||
}
|
||||
)
|
||||
})
|
||||
.map_err(|err| err.to_string()),
|
||||
Command::SetActive { network_id, active } => ipc::set_active(socket, &network_id, active)
|
||||
.await
|
||||
.map(|report| {
|
||||
let state = if report.active { "started" } else { "stopped" };
|
||||
format!("{} {}", state, report.name)
|
||||
if report.suspended.is_empty() {
|
||||
format!("{} {}", state, report.name)
|
||||
} else {
|
||||
format!(
|
||||
"{} {}; suspended {} until it stops",
|
||||
state,
|
||||
report.name,
|
||||
report
|
||||
.suspended
|
||||
.iter()
|
||||
.map(|n| format!("{} ({})", n.name, n.network_id))
|
||||
.collect::<Vec<_>>()
|
||||
.join(", ")
|
||||
)
|
||||
}
|
||||
})
|
||||
.map_err(|err| err.to_string()),
|
||||
Command::SetBroadcast {
|
||||
@@ -211,7 +314,17 @@ async fn run(
|
||||
}
|
||||
Command::Join { name, secret } => ipc::join_network(socket, &name, &secret)
|
||||
.await
|
||||
.map(|report| format!("joined {}", report.name))
|
||||
.map(|report| {
|
||||
format!(
|
||||
"{} {}",
|
||||
if report.active {
|
||||
"joined"
|
||||
} else {
|
||||
"configured; stopped by range conflict:"
|
||||
},
|
||||
report.name
|
||||
)
|
||||
})
|
||||
.map_err(|err| err.to_string()),
|
||||
Command::Leave { network_id } => ipc::leave_network(socket, &network_id)
|
||||
.await
|
||||
@@ -236,7 +349,6 @@ async fn run(
|
||||
|
||||
{
|
||||
let mut guard = lock(shared);
|
||||
guard.busy = false;
|
||||
guard.last_action = if matches!(&outcome, Ok(message) if message.is_empty()) {
|
||||
None
|
||||
} else {
|
||||
|
||||
@@ -15,6 +15,12 @@ use crate::devices;
|
||||
use crate::stats::Traffic;
|
||||
use crate::ui::{self, UiState};
|
||||
|
||||
fn app_fonts() -> egui::FontDefinitions {
|
||||
let mut fonts = egui::FontDefinitions::default();
|
||||
egui_phosphor::add_to_fonts(&mut fonts, egui_phosphor::Variant::Regular);
|
||||
fonts
|
||||
}
|
||||
|
||||
/// The window's state.
|
||||
pub(crate) struct App {
|
||||
agent: AgentClient,
|
||||
@@ -34,9 +40,7 @@ impl App {
|
||||
let ctx = cc.egui_ctx.clone();
|
||||
// Add the Phosphor icon glyphs to the font set so the icon buttons
|
||||
// render instead of showing as tofu.
|
||||
let mut fonts = egui::FontDefinitions::default();
|
||||
egui_phosphor::add_to_fonts(&mut fonts, egui_phosphor::Variant::Regular);
|
||||
ctx.set_fonts(fonts);
|
||||
ctx.set_fonts(app_fonts());
|
||||
|
||||
let agent = AgentClient::spawn(&runtime, ctx);
|
||||
Ok(Self {
|
||||
@@ -60,6 +64,7 @@ impl eframe::App for App {
|
||||
|
||||
ui::draw(ctx, &self.agent, &mut self.state, &snapshot, &self.traffic);
|
||||
self.show_devices_windows(ctx, &snapshot);
|
||||
self.show_vip_windows(ctx, &snapshot);
|
||||
self.show_about_window(ctx, &snapshot);
|
||||
|
||||
// A secret the worker fetched for the clipboard: copy it here, on the
|
||||
@@ -74,6 +79,51 @@ impl eframe::App for App {
|
||||
}
|
||||
|
||||
impl App {
|
||||
fn show_vip_windows(&mut self, ctx: &egui::Context, snapshot: &crate::agent::Snapshot) {
|
||||
let Some(Ok(report)) = &snapshot.status else {
|
||||
return;
|
||||
};
|
||||
let ids: Vec<_> = self.state.vip_windows.keys().cloned().collect();
|
||||
for id in ids {
|
||||
let Some(network) = report.networks.iter().find(|n| n.network_id == id) else {
|
||||
self.state.vip_windows.remove(&id);
|
||||
continue;
|
||||
};
|
||||
let Some(state) = self.state.vip_windows.get_mut(&id) else {
|
||||
continue;
|
||||
};
|
||||
let builder = egui::ViewportBuilder::default()
|
||||
.with_title(format!("tsunagi · VIP · {}", network.name))
|
||||
.with_app_id(crate::icon::APP_ID)
|
||||
.with_icon(crate::icon::window())
|
||||
.with_inner_size([1280., 860.])
|
||||
.with_min_inner_size([650., 400.]);
|
||||
let keep = ctx.show_viewport_immediate(
|
||||
egui::ViewportId::from_hash_of(("vip", &id)),
|
||||
builder,
|
||||
|vctx, _| {
|
||||
let close = crate::vip::show(vctx, &self.agent, state, network, snapshot);
|
||||
if close {
|
||||
return false;
|
||||
}
|
||||
if vctx.input(|i| i.viewport().close_requested()) {
|
||||
if state.dirty() {
|
||||
state.request_close();
|
||||
vctx.send_viewport_cmd(egui::ViewportCommand::CancelClose);
|
||||
true
|
||||
} else {
|
||||
false
|
||||
}
|
||||
} else {
|
||||
true
|
||||
}
|
||||
},
|
||||
);
|
||||
if !keep {
|
||||
self.state.vip_windows.remove(&id);
|
||||
}
|
||||
}
|
||||
}
|
||||
/// Renders the About window while it is open.
|
||||
fn show_about_window(&mut self, ctx: &egui::Context, snapshot: &crate::agent::Snapshot) {
|
||||
if !self.state.about_open {
|
||||
@@ -116,20 +166,21 @@ impl App {
|
||||
.with_title(devices::title(network))
|
||||
.with_app_id(crate::icon::APP_ID)
|
||||
.with_icon(crate::icon::window())
|
||||
.with_inner_size([940.0, 540.0])
|
||||
.with_min_inner_size([480.0, 320.0]);
|
||||
.with_inner_size([1440.0, 640.0])
|
||||
.with_min_inner_size([800.0, 400.0]);
|
||||
let agent = &self.agent;
|
||||
let traffic = &self.traffic;
|
||||
// The window shares the single unit setting, so its toggle and the
|
||||
// main window's stay in step.
|
||||
let unit = &mut self.state.unit;
|
||||
let confirm = &mut self.state.exit_confirm;
|
||||
let forget_confirm = &mut self.state.forget_confirm;
|
||||
let keep = ctx.show_viewport_immediate(viewport_id, builder, |vctx, _class| {
|
||||
devices::show(
|
||||
vctx,
|
||||
agent,
|
||||
unit,
|
||||
confirm,
|
||||
(confirm, forget_confirm),
|
||||
network,
|
||||
&report.endpoint_id,
|
||||
traffic,
|
||||
@@ -142,3 +193,20 @@ impl App {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn route_icons_have_an_embedded_font_in_the_app() {
|
||||
let fonts = app_fonts();
|
||||
assert!(fonts.font_data.contains_key("phosphor"));
|
||||
assert!(
|
||||
fonts
|
||||
.families
|
||||
.get(&egui::FontFamily::Proportional)
|
||||
.is_some_and(|family| family.iter().any(|name| name == "phosphor"))
|
||||
);
|
||||
}
|
||||
}
|
||||
+907
-142
File diff suppressed because it is too large.
Load diff
@@ -78,6 +78,7 @@ fn colour_of(health: Health) -> [u8; 3] {
|
||||
match health {
|
||||
Health::Disconnected => DARK_RED,
|
||||
Health::Connected => BLUE,
|
||||
Health::RangeConflict => [0xd9, 0x91, 0x20],
|
||||
Health::ExitNode => GREEN,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -17,8 +17,11 @@ mod exit;
|
||||
mod format;
|
||||
mod icon;
|
||||
mod stats;
|
||||
mod table;
|
||||
mod tray;
|
||||
mod ui;
|
||||
mod vip;
|
||||
mod vip_status;
|
||||
mod watch;
|
||||
|
||||
use eframe::egui;
|
||||
|
||||
@@ -34,7 +34,7 @@ pub(crate) struct Series {
|
||||
}
|
||||
|
||||
impl Series {
|
||||
fn sample(&mut self, now: Instant, packets: u64, bytes: u64) {
|
||||
pub(crate) fn sample(&mut self, now: Instant, packets: u64, bytes: u64) {
|
||||
if let Some((then, prev_packets, prev_bytes)) = self.last {
|
||||
let dt = now.saturating_duration_since(then).as_secs_f32();
|
||||
if dt > 0.0 {
|
||||
|
||||
@@ -0,0 +1,88 @@
|
||||
//! Fixed row geometry for dense live tables. Children cannot resize a row or
|
||||
//! paint into its neighbours; the enclosing scroll area owns overflow.
|
||||
use eframe::egui;
|
||||
|
||||
pub(crate) const GAP: f32 = 10.0;
|
||||
|
||||
pub(crate) fn row(
|
||||
ui: &mut egui::Ui,
|
||||
id: impl std::hash::Hash,
|
||||
widths: &[f32],
|
||||
stripe: bool,
|
||||
mut cell: impl FnMut(usize, &mut egui::Ui),
|
||||
) {
|
||||
let font = egui::FontSelection::Default.resolve(ui.style());
|
||||
let text_height = ui.fonts(|fonts| fonts.row_height(&font));
|
||||
let height = ui.spacing().interact_size.y.max(text_height) + 10.0;
|
||||
let width = widths.iter().sum::<f32>() + GAP * widths.len().saturating_sub(1) as f32;
|
||||
let (rect, _) = ui.allocate_exact_size(egui::vec2(width, height), egui::Sense::hover());
|
||||
if stripe {
|
||||
ui.painter()
|
||||
.rect_filled(rect, 2.0, ui.visuals().faint_bg_color);
|
||||
}
|
||||
let mut x = rect.left();
|
||||
for (column, width) in widths.iter().enumerate() {
|
||||
let bounds = egui::Rect::from_min_size(
|
||||
egui::pos2(x, rect.top() + 3.0),
|
||||
egui::vec2(*width, height - 6.0),
|
||||
);
|
||||
let mut child = ui.new_child(
|
||||
egui::UiBuilder::new()
|
||||
.id_salt((&id, column))
|
||||
.max_rect(bounds)
|
||||
.layout(egui::Layout::left_to_right(egui::Align::Center)),
|
||||
);
|
||||
child.set_clip_rect(ui.clip_rect().intersect(bounds));
|
||||
child.style_mut().wrap_mode = Some(egui::TextWrapMode::Truncate);
|
||||
cell(column, &mut child);
|
||||
x += width + GAP;
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn oversized_content_is_clipped_to_its_cell_at_every_scale() {
|
||||
for size in [13.0, 20.0, 28.0] {
|
||||
let ctx = egui::Context::default();
|
||||
ctx.style_mut(|style| {
|
||||
style.override_font_id = Some(egui::FontId::proportional(size));
|
||||
});
|
||||
let output = ctx.run(egui::RawInput::default(), |ctx| {
|
||||
egui::CentralPanel::default().show(ctx, |ui| {
|
||||
for index in 0..4 {
|
||||
row(ui, index, &[90.0, 120.0], index % 2 == 0, |column, ui| {
|
||||
if column == 0 {
|
||||
ui.horizontal(|ui| {
|
||||
ui.label("very long device name that cannot fit");
|
||||
ui.button("Forget").on_hover_text("Forget");
|
||||
});
|
||||
} else {
|
||||
ui.label(format!("row {index}"));
|
||||
}
|
||||
});
|
||||
}
|
||||
});
|
||||
});
|
||||
let mut previous_bottom = 0.0;
|
||||
for index in 0..4 {
|
||||
let shape = output.shapes.iter().find(|shape| matches!(&shape.shape, egui::Shape::Text(t) if t.galley.text() == format!("row {index}"))).unwrap();
|
||||
let egui::Shape::Text(text) = &shape.shape else {
|
||||
unreachable!()
|
||||
};
|
||||
assert!(text.visual_bounding_rect().top() >= previous_bottom);
|
||||
assert!(
|
||||
shape.clip_rect.contains_rect(text.visual_bounding_rect()),
|
||||
"font {size}: clip {:?}, text {:?}",
|
||||
shape.clip_rect,
|
||||
text.visual_bounding_rect()
|
||||
);
|
||||
assert!(shape.clip_rect.left() >= 108.0);
|
||||
previous_bottom = shape.clip_rect.bottom();
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
+116
-11
@@ -41,10 +41,13 @@ pub(crate) struct UiState {
|
||||
pub(crate) host_edit: Option<String>,
|
||||
/// Network ids whose devices window is open.
|
||||
pub(crate) open_devices: BTreeSet<String>,
|
||||
pub(crate) vip_windows: std::collections::BTreeMap<String, crate::vip::Window>,
|
||||
/// Whether the About window is open.
|
||||
pub(crate) about_open: bool,
|
||||
/// An exit-node choice waiting for the user to confirm it.
|
||||
pub(crate) exit_confirm: Option<devices::ExitConfirm>,
|
||||
/// A selected offline claim waiting for cleanup confirmation.
|
||||
pub(crate) forget_confirm: Option<devices::ForgetConfirm>,
|
||||
}
|
||||
|
||||
impl Default for UiState {
|
||||
@@ -54,8 +57,10 @@ impl Default for UiState {
|
||||
unit: Unit::Packets,
|
||||
host_edit: None,
|
||||
open_devices: BTreeSet::new(),
|
||||
vip_windows: Default::default(),
|
||||
about_open: false,
|
||||
exit_confirm: None,
|
||||
forget_confirm: None,
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -123,7 +128,7 @@ pub(crate) fn draw(
|
||||
ui.weak("No networks yet — join one below.");
|
||||
}
|
||||
for network in &report.networks {
|
||||
draw_network(ui, agent, state, traffic, network, &report.endpoint_id);
|
||||
draw_network(ui, agent, state, traffic, network, report);
|
||||
}
|
||||
});
|
||||
}
|
||||
@@ -202,12 +207,13 @@ fn draw_network(
|
||||
state: &mut UiState,
|
||||
traffic: &Traffic,
|
||||
network: &NetworkReport,
|
||||
own_id: &str,
|
||||
report: &StatusReport,
|
||||
) {
|
||||
ui.add_space(6.0);
|
||||
egui::Frame::group(ui.style()).show(ui, |ui| {
|
||||
ui.horizontal(|ui| {
|
||||
format::copy_field(ui, network.name.as_str(), &network.name);
|
||||
if ui.button("VIP").clicked(){state.vip_windows.entry(network.network_id.clone()).or_default();}
|
||||
ui.with_layout(egui::Layout::right_to_left(egui::Align::Center), |ui| {
|
||||
let leave =
|
||||
egui::Button::new(egui::RichText::new("Leave").color(egui::Color32::WHITE))
|
||||
@@ -243,22 +249,41 @@ fn draw_network(
|
||||
}
|
||||
ui.weak("peers");
|
||||
ui.label(format!(
|
||||
"{} connected · {} known",
|
||||
network.peers.len(),
|
||||
devices::known_count(network, own_id)
|
||||
"{} online · {} known",
|
||||
devices::online_count(network, &report.endpoint_id),
|
||||
devices::known_count(network, &report.endpoint_id)
|
||||
));
|
||||
ui.end_row();
|
||||
});
|
||||
|
||||
draw_protocols(ui, agent, network, &report.available_protocols);
|
||||
|
||||
format::sparkline(ui, series, state.unit, 34.0);
|
||||
|
||||
if !network.range_conflicts.is_empty() {
|
||||
ui.colored_label(BAD, "Stopped: overlapping network ranges");
|
||||
for conflict in &network.range_conflicts {
|
||||
ui.label(format!("{} ({}) uses {}", conflict.name, conflict.network_id, conflict.range));
|
||||
}
|
||||
ui.label("Switching suspends these networks until this network stops, then resumes them automatically.");
|
||||
if network.auto_start { ui.weak("This network will resume automatically when its range is free."); }
|
||||
}
|
||||
ui.horizontal(|ui| {
|
||||
let mut active = network.active;
|
||||
if ui.checkbox(&mut active, "active").changed() {
|
||||
agent.send(Command::SetActive {
|
||||
network_id: network.network_id.clone(),
|
||||
active,
|
||||
});
|
||||
if !network.active && !network.range_conflicts.is_empty() {
|
||||
if ui.button("Switch to this network").clicked() {
|
||||
agent.send(Command::SetActive { network_id: network.network_id.clone(), active: true });
|
||||
}
|
||||
if network.auto_start && ui.button("Keep stopped").clicked() {
|
||||
agent.send(Command::SetActive { network_id: network.network_id.clone(), active: false });
|
||||
}
|
||||
} else {
|
||||
let mut active = network.active;
|
||||
if ui.checkbox(&mut active, "active").changed() {
|
||||
agent.send(Command::SetActive {
|
||||
network_id: network.network_id.clone(),
|
||||
active,
|
||||
});
|
||||
}
|
||||
}
|
||||
let mut broadcast = network.broadcast;
|
||||
if ui.checkbox(&mut broadcast, "broadcast").changed() {
|
||||
@@ -328,6 +353,86 @@ fn draw_network(
|
||||
});
|
||||
}
|
||||
|
||||
fn draw_protocols(
|
||||
ui: &mut egui::Ui,
|
||||
agent: &AgentClient,
|
||||
network: &NetworkReport,
|
||||
available: &[tsunagi::ipc::ProtocolReport],
|
||||
) {
|
||||
let summary = if network.protocols.is_empty() {
|
||||
"none".into()
|
||||
} else {
|
||||
network.protocols.join(", ")
|
||||
};
|
||||
egui::CollapsingHeader::new(format!("protocols {summary}"))
|
||||
.id_salt(("protocols", &network.network_id))
|
||||
.show(ui, |ui| {
|
||||
ui.weak("Preference order; changes apply immediately and are saved.");
|
||||
let mut selected = network.protocols.clone();
|
||||
let mut changed = false;
|
||||
ui.add_enabled_ui(!agent.snapshot().busy, |ui| {
|
||||
// Enabled items follow the user's preference; unchecked items
|
||||
// follow the agent's catalog. The worker refreshes after edits.
|
||||
let ordered: Vec<_> = network
|
||||
.protocols
|
||||
.iter()
|
||||
.filter_map(|name| available.iter().find(|p| &p.name == name))
|
||||
.chain(
|
||||
available
|
||||
.iter()
|
||||
.filter(|p| !network.protocols.contains(&p.name)),
|
||||
)
|
||||
.collect();
|
||||
for protocol in ordered {
|
||||
ui.horizontal(|ui| {
|
||||
let mut on = selected.contains(&protocol.name);
|
||||
if ui.checkbox(&mut on, &protocol.name).changed() {
|
||||
if on {
|
||||
selected.push(protocol.name.clone());
|
||||
} else {
|
||||
selected.retain(|p| p != &protocol.name);
|
||||
}
|
||||
changed = true;
|
||||
}
|
||||
if let Some(index) = selected.iter().position(|p| p == &protocol.name) {
|
||||
if ui
|
||||
.add_enabled(index > 0, egui::Button::new(icon::ARROW_UP))
|
||||
.on_hover_text("prefer earlier")
|
||||
.clicked()
|
||||
{
|
||||
selected.swap(index, index - 1);
|
||||
changed = true;
|
||||
}
|
||||
if ui
|
||||
.add_enabled(
|
||||
index + 1 < selected.len(),
|
||||
egui::Button::new(icon::ARROW_DOWN),
|
||||
)
|
||||
.on_hover_text("prefer later")
|
||||
.clicked()
|
||||
{
|
||||
selected.swap(index, index + 1);
|
||||
changed = true;
|
||||
}
|
||||
}
|
||||
});
|
||||
ui.indent(("description", &network.network_id, &protocol.name), |ui| {
|
||||
ui.weak(&protocol.summary);
|
||||
});
|
||||
}
|
||||
});
|
||||
if selected.is_empty() {
|
||||
ui.weak("Control only; no IP traffic in this network.");
|
||||
}
|
||||
if changed {
|
||||
agent.send(Command::SetProtocols {
|
||||
network_id: network.network_id.clone(),
|
||||
protocols: selected,
|
||||
});
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
/// A prominent line about an exit node. One that carries a command copies it
|
||||
/// when clicked.
|
||||
fn draw_exit_warning(ui: &mut egui::Ui, warning: &exit::Warning) {
|
||||
|
||||
@@ -0,0 +1,402 @@
|
||||
//! Per-network VIP editor. Drafts never issue commands until Save is pressed.
|
||||
use crate::agent::{AgentClient, Command, Snapshot};
|
||||
use crate::vip_status;
|
||||
use eframe::egui;
|
||||
use tsunagi::ipc::NetworkReport;
|
||||
use tsunagi::vip::{Backend, Listener, Protocol, VipConfig};
|
||||
#[derive(Clone)]
|
||||
struct Draft {
|
||||
config: VipConfig,
|
||||
original: VipConfig,
|
||||
revision: Option<String>,
|
||||
delete: bool,
|
||||
}
|
||||
impl Draft {
|
||||
fn new(config: VipConfig, revision: Option<String>) -> Self {
|
||||
Self {
|
||||
original: config.clone(),
|
||||
config,
|
||||
revision,
|
||||
delete: false,
|
||||
}
|
||||
}
|
||||
fn dirty(&self) -> bool {
|
||||
self.revision.is_none() || self.delete || self.config != self.original
|
||||
}
|
||||
}
|
||||
#[derive(Default)]
|
||||
pub(crate) struct Window {
|
||||
draft: Option<Draft>,
|
||||
pending: Option<String>,
|
||||
error: Option<String>,
|
||||
close_question: bool,
|
||||
close_after_save: bool,
|
||||
live: vip_status::View,
|
||||
configure: bool,
|
||||
generation: u64,
|
||||
acknowledged_generation: Option<u64>,
|
||||
}
|
||||
impl Window {
|
||||
pub fn request_close(&mut self) {
|
||||
self.close_question = true;
|
||||
self.close_after_save = true;
|
||||
}
|
||||
pub fn dirty(&self) -> bool {
|
||||
self.draft.as_ref().is_some_and(Draft::dirty) || self.pending.is_some()
|
||||
}
|
||||
}
|
||||
fn peer_name<'a>(n: &'a NetworkReport, id: &'a str) -> &'a str {
|
||||
n.members
|
||||
.iter()
|
||||
.find(|m| m.endpoint_id == id)
|
||||
.and_then(|m| m.hostname.as_deref())
|
||||
.unwrap_or(id)
|
||||
}
|
||||
pub(crate) fn show(
|
||||
ctx: &egui::Context,
|
||||
agent: &AgentClient,
|
||||
state: &mut Window,
|
||||
n: &NetworkReport,
|
||||
snapshot: &Snapshot,
|
||||
) -> bool {
|
||||
if state.generation != snapshot.generation {
|
||||
state.generation = snapshot.generation;
|
||||
if let Some(at) = snapshot.at {
|
||||
state.live.observe(n, at);
|
||||
}
|
||||
}
|
||||
if let Some(result) = &snapshot.vip_result
|
||||
&& state.pending.as_deref() == Some(result.token.as_str())
|
||||
{
|
||||
state.pending = None;
|
||||
state.acknowledged_generation = Some(snapshot.generation);
|
||||
match &result.result {
|
||||
Ok(vips) => {
|
||||
if let Some(draft) = &state.draft {
|
||||
state.draft = vips
|
||||
.iter()
|
||||
.find(|v| v.config.id == draft.config.id)
|
||||
.map(|v| Draft::new(v.config.clone(), Some(v.revision.clone())));
|
||||
}
|
||||
state.error = None;
|
||||
}
|
||||
Err(error) => {
|
||||
state.error = Some(error.clone());
|
||||
state.close_after_save = false;
|
||||
}
|
||||
}
|
||||
}
|
||||
let mut close = state.close_after_save && !state.dirty();
|
||||
let mut discard_new = false;
|
||||
egui::CentralPanel::default().show(ctx, |ui| {
|
||||
ui.heading(format!("VIP · {}", n.name));
|
||||
ui.weak("Configuration is shared with every member. Changes take effect only after Save.");
|
||||
if state.close_question {
|
||||
ui.horizontal(|ui| {
|
||||
ui.label("Unsaved changes.");
|
||||
if ui
|
||||
.add_enabled(
|
||||
state.pending.is_none(),
|
||||
egui::Button::new("Discard and close"),
|
||||
)
|
||||
.clicked()
|
||||
{
|
||||
state.draft = None;
|
||||
state.close_question = false;
|
||||
close = true;
|
||||
}
|
||||
if ui.button("Keep editing").clicked() {
|
||||
state.close_question = false;
|
||||
state.close_after_save = false;
|
||||
}
|
||||
});
|
||||
}
|
||||
if let Some(error) = &state.error {
|
||||
ui.colored_label(egui::Color32::LIGHT_RED, error);
|
||||
}
|
||||
ui.separator();
|
||||
ui.horizontal_wrapped(|ui| {
|
||||
for vip in &n.vips {
|
||||
let selected = state
|
||||
.draft
|
||||
.as_ref()
|
||||
.is_some_and(|d| d.config.id == vip.config.id);
|
||||
if ui.selectable_label(selected, &vip.config.name).clicked() {
|
||||
if state.dirty() {
|
||||
state.error = Some(
|
||||
"Save or discard the current draft before selecting another VIP."
|
||||
.into(),
|
||||
);
|
||||
} else {
|
||||
state.draft =
|
||||
Some(Draft::new(vip.config.clone(), Some(vip.revision.clone())));
|
||||
state.error = None;
|
||||
}
|
||||
}
|
||||
}
|
||||
if ui.button("+ New VIP").clicked() {
|
||||
if state.dirty() {
|
||||
state.error = Some("Save or discard the current draft first.".into());
|
||||
} else {
|
||||
state.draft = Some(Draft::new(VipConfig::new(""), None));
|
||||
state.configure = true;
|
||||
state.error = None;
|
||||
}
|
||||
}
|
||||
});
|
||||
let Some(draft) = &mut state.draft else {
|
||||
ui.label("Select a VIP or create one.");
|
||||
return;
|
||||
};
|
||||
let current = n.vips.iter().find(|v| v.config.id == draft.config.id);
|
||||
if state.pending.is_none()
|
||||
&& state.acknowledged_generation != Some(snapshot.generation)
|
||||
&& !draft.dirty()
|
||||
&& let Some(current) = current
|
||||
&& draft.revision.as_ref() != Some(¤t.revision)
|
||||
{
|
||||
*draft = Draft::new(current.config.clone(), Some(current.revision.clone()));
|
||||
}
|
||||
let changed =
|
||||
current.is_some_and(|v| draft.revision.as_ref().is_some_and(|r| r != &v.revision));
|
||||
if changed {
|
||||
ui.colored_label(
|
||||
egui::Color32::YELLOW,
|
||||
"This VIP changed on the network. Reload before saving; your draft is retained.",
|
||||
);
|
||||
}
|
||||
if draft.revision.is_some() && current.is_none() {
|
||||
ui.colored_label(
|
||||
egui::Color32::YELLOW,
|
||||
"This VIP was removed on the network. The draft cannot overwrite its deletion.",
|
||||
);
|
||||
}
|
||||
let busy = state.pending.is_some();
|
||||
ui.horizontal(|ui| {
|
||||
let valid = draft.config.validate(&tsunagi::Limits::default());
|
||||
if !n.active {
|
||||
ui.weak("Start the network to save changes.");
|
||||
}
|
||||
if ui
|
||||
.add_enabled(
|
||||
n.active
|
||||
&& !busy
|
||||
&& draft.dirty()
|
||||
&& !changed
|
||||
&& (valid.is_ok() || draft.delete),
|
||||
egui::Button::new("Save"),
|
||||
)
|
||||
.clicked()
|
||||
{
|
||||
let token = VipConfig::new("").id;
|
||||
state.pending = Some(token.clone());
|
||||
agent.send(Command::SaveVip {
|
||||
token,
|
||||
network_id: n.network_id.clone(),
|
||||
config: draft.config.clone(),
|
||||
expected: draft.revision.clone(),
|
||||
delete: draft.delete,
|
||||
});
|
||||
}
|
||||
if ui
|
||||
.add_enabled(!busy, egui::Button::new("Discard / reload"))
|
||||
.clicked()
|
||||
{
|
||||
if let Some(v) = current {
|
||||
*draft = Draft::new(v.config.clone(), Some(v.revision.clone()));
|
||||
} else {
|
||||
discard_new = true;
|
||||
}
|
||||
state.error = None;
|
||||
}
|
||||
if busy {
|
||||
ui.spinner();
|
||||
ui.label("Saving…");
|
||||
} else if draft.dirty() {
|
||||
ui.weak("Unsaved changes");
|
||||
}
|
||||
if let Err(error) = valid {
|
||||
ui.colored_label(egui::Color32::LIGHT_RED, error.to_string());
|
||||
}
|
||||
});
|
||||
ui.separator();
|
||||
ui.horizontal(|ui| {
|
||||
ui.selectable_value(&mut state.configure, false, "Overview");
|
||||
ui.selectable_value(&mut state.configure, true, "Configuration");
|
||||
});
|
||||
egui::ScrollArea::vertical()
|
||||
.auto_shrink([false, false])
|
||||
.show(ui, |ui| {
|
||||
if state.configure {
|
||||
ui.add_enabled_ui(!busy, |ui| {
|
||||
ui.horizontal(|ui| {
|
||||
ui.label("Name");
|
||||
ui.add(
|
||||
egui::TextEdit::singleline(&mut draft.config.name)
|
||||
.id(egui::Id::new(("vip-name", &draft.config.id))),
|
||||
);
|
||||
ui.weak(format!(".{}", n.name));
|
||||
});
|
||||
if draft.revision.is_some() {
|
||||
ui.checkbox(&mut draft.delete, "Delete this VIP when saved");
|
||||
}
|
||||
if !draft.delete {
|
||||
edit_listeners(ui, &mut draft.config, n);
|
||||
}
|
||||
});
|
||||
} else if let Some(current) = current {
|
||||
vip_status::show(ui, current, n, &mut state.live);
|
||||
} else {
|
||||
ui.weak("Save this VIP to see its live status.");
|
||||
}
|
||||
});
|
||||
});
|
||||
if discard_new {
|
||||
state.draft = None;
|
||||
state.error = None;
|
||||
}
|
||||
close
|
||||
}
|
||||
fn edit_listeners(ui: &mut egui::Ui, config: &mut VipConfig, n: &NetworkReport) {
|
||||
let mut remove = None;
|
||||
for (index, l) in config.listeners.iter_mut().enumerate() {
|
||||
ui.push_id(index,|ui|{egui::Frame::group(ui.style()).show(ui,|ui|{
|
||||
ui.horizontal(|ui|{egui::ComboBox::from_id_salt("protocol").selected_text(l.protocol.to_string()).show_ui(ui,|ui|{ui.selectable_value(&mut l.protocol,Protocol::Tcp,"TCP");ui.selectable_value(&mut l.protocol,Protocol::Udp,"UDP");});ui.label("Port");ui.add(egui::DragValue::new(&mut l.port).range(1..=65535));if ui.button("Remove listener").clicked(){remove=Some(index);}});
|
||||
if l.protocol==Protocol::Tcp{ui.checkbox(&mut l.port_check,"Active TCP port checks");}else{ui.weak("UDP health: encrypted path probes and explicit port errors; application readiness is unknown.");}
|
||||
ui.collapsing("Checks and flow expiry",|ui|{egui::Grid::new("settings").show(ui,|ui|{
|
||||
ui.label("Check interval, ms");ui.add(egui::DragValue::new(&mut l.check_interval_ms).range(250..=3_600_000));ui.end_row();
|
||||
ui.label("Check timeout, ms");ui.add(egui::DragValue::new(&mut l.check_timeout_ms).range(100..=60_000));ui.end_row();
|
||||
ui.label("Failures before exclusion");ui.add(egui::DragValue::new(&mut l.failure_threshold).range(1..=255));ui.end_row();
|
||||
ui.label("Successes before recovery");ui.add(egui::DragValue::new(&mut l.recovery_threshold).range(1..=255));ui.end_row();
|
||||
ui.label("Flow idle timeout, seconds");ui.add(egui::DragValue::new(&mut l.idle_timeout_secs).range(1..=604800));ui.end_row();
|
||||
});});
|
||||
let mut remove_backend=None;for(i,b)in l.backends.iter_mut().enumerate(){ui.push_id(i,|ui|{ui.horizontal(|ui|{
|
||||
egui::ComboBox::from_id_salt("node").width(240.).selected_text(peer_name(n,&b.peer)).show_ui(ui,|ui|{for m in &n.members{let label=format!("{} ({})",m.hostname.as_deref().unwrap_or("unnamed"),m.endpoint_id.chars().take(8).collect::<String>());ui.selectable_value(&mut b.peer,m.endpoint_id.clone(),label);}});
|
||||
ui.label("Target port");ui.add(egui::DragValue::new(&mut b.port).range(1..=65535));if ui.button("Remove").clicked(){remove_backend=Some(i);}
|
||||
});});}if let Some(i)=remove_backend{l.backends.remove(i);}
|
||||
if ui.button("+ Backend").clicked(){l.backends.push(Backend{peer:n.members.first().map(|m|m.endpoint_id.clone()).unwrap_or_default(),port:l.port});}
|
||||
});});
|
||||
}
|
||||
if let Some(index) = remove {
|
||||
config.listeners.remove(index);
|
||||
}
|
||||
if ui.button("+ Listener").clicked() {
|
||||
config.listeners.push(Listener::new(Protocol::Tcp, 443));
|
||||
}
|
||||
}
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
|
||||
use super::*;
|
||||
use tsunagi::vip::VipStatus;
|
||||
#[test]
|
||||
fn editing_the_actual_form_sends_nothing_until_save_is_clicked() {
|
||||
let (agent, mut commands) = AgentClient::test_client();
|
||||
let ctx = egui::Context::default();
|
||||
let mut config = VipConfig::new("api");
|
||||
config.listeners.push(Listener::new(Protocol::Tcp, 443));
|
||||
let mut state = Window {
|
||||
draft: Some(Draft::new(config.clone(), Some("revision".into()))),
|
||||
configure: true,
|
||||
..Default::default()
|
||||
};
|
||||
let network = NetworkReport {
|
||||
active: true,
|
||||
name: "lab".into(),
|
||||
vips: vec![VipStatus {
|
||||
config: config.clone(),
|
||||
revision: "revision".into(),
|
||||
address: None,
|
||||
error: None,
|
||||
listeners: vec![],
|
||||
}],
|
||||
..Default::default()
|
||||
};
|
||||
let snapshot = Snapshot::default();
|
||||
let frame = |events: Vec<egui::Event>, state: &mut Window| {
|
||||
ctx.run(
|
||||
egui::RawInput {
|
||||
screen_rect: Some(egui::Rect::from_min_size(
|
||||
egui::Pos2::ZERO,
|
||||
egui::vec2(1000., 800.),
|
||||
)),
|
||||
events,
|
||||
..Default::default()
|
||||
},
|
||||
|ctx| {
|
||||
show(ctx, &agent, state, &network, &snapshot);
|
||||
},
|
||||
)
|
||||
};
|
||||
frame(vec![], &mut state);
|
||||
ctx.memory_mut(|m| m.request_focus(egui::Id::new(("vip-name", &config.id))));
|
||||
frame(vec![egui::Event::Text("new".into())], &mut state);
|
||||
assert!(state.dirty());
|
||||
assert_ne!(state.draft.as_ref().unwrap().config.name, config.name);
|
||||
assert!(commands.try_recv().is_err());
|
||||
assert_eq!(network.vips[0].config, config);
|
||||
fn save_position(shape: &egui::Shape) -> Option<egui::Pos2> {
|
||||
match shape {
|
||||
egui::Shape::Text(text) if text.galley.text() == "Save" => {
|
||||
Some(text.pos + text.galley.rect.center().to_vec2())
|
||||
}
|
||||
egui::Shape::Vec(shapes) => shapes.iter().find_map(save_position),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
// Save was disabled before this edit; allow the next layout to register
|
||||
// its now-enabled hit target before dispatching the pointer press.
|
||||
let ready = frame(vec![], &mut state);
|
||||
let position = ready
|
||||
.shapes
|
||||
.iter()
|
||||
.find_map(|s| save_position(&s.shape))
|
||||
.unwrap();
|
||||
frame(
|
||||
vec![
|
||||
egui::Event::PointerMoved(position),
|
||||
egui::Event::PointerButton {
|
||||
pos: position,
|
||||
button: egui::PointerButton::Primary,
|
||||
pressed: true,
|
||||
modifiers: egui::Modifiers::NONE,
|
||||
},
|
||||
],
|
||||
&mut state,
|
||||
);
|
||||
frame(
|
||||
vec![egui::Event::PointerButton {
|
||||
pos: position,
|
||||
button: egui::PointerButton::Primary,
|
||||
pressed: false,
|
||||
modifiers: egui::Modifiers::NONE,
|
||||
}],
|
||||
&mut state,
|
||||
);
|
||||
let Command::SaveVip {
|
||||
config: saved,
|
||||
expected,
|
||||
delete,
|
||||
..
|
||||
} = commands.try_recv().unwrap()
|
||||
else {
|
||||
panic!("Save must submit one complete VIP draft");
|
||||
};
|
||||
assert_ne!(saved.name, config.name);
|
||||
assert_eq!(expected.as_deref(), Some("revision"));
|
||||
assert!(!delete);
|
||||
assert!(commands.try_recv().is_err());
|
||||
}
|
||||
#[test]
|
||||
fn all_listener_settings_render_without_a_live_agent() {
|
||||
let ctx = egui::Context::default();
|
||||
let mut cfg = VipConfig::new("api");
|
||||
cfg.listeners.push(Listener::new(Protocol::Tcp, 443));
|
||||
cfg.listeners.push(Listener::new(Protocol::Udp, 53));
|
||||
let n = NetworkReport::default();
|
||||
let _ = ctx.run(egui::RawInput::default(), |ctx| {
|
||||
egui::CentralPanel::default().show(ctx, |ui| edit_listeners(ui, &mut cfg, &n));
|
||||
});
|
||||
assert_eq!(cfg.listeners.len(), 2);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,885 @@
|
||||
//! Local VIP observations, kept apart from editable configuration drafts.
|
||||
use std::collections::BTreeMap;
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use eframe::egui;
|
||||
use tsunagi::ipc::NetworkReport;
|
||||
use tsunagi::vip::{Backend, BackendStatus, Counters, ListenerStatus, Protocol, VipStatus};
|
||||
|
||||
use crate::{
|
||||
format,
|
||||
stats::{Series, Unit},
|
||||
table,
|
||||
};
|
||||
|
||||
const ACTIVE: egui::Color32 = egui::Color32::from_rgb(73, 190, 133);
|
||||
const NEXT: egui::Color32 = egui::Color32::from_rgb(78, 157, 237);
|
||||
const WARN: egui::Color32 = egui::Color32::from_rgb(225, 170, 73);
|
||||
const BAD: egui::Color32 = egui::Color32::from_rgb(228, 112, 112);
|
||||
|
||||
#[derive(Default)]
|
||||
struct Rates {
|
||||
tx: Series,
|
||||
rx: Series,
|
||||
total: Series,
|
||||
at: Option<Instant>,
|
||||
}
|
||||
impl Rates {
|
||||
fn sample(&mut self, at: Instant, c: &Counters) {
|
||||
self.tx.sample(at, c.tx_packets, c.tx_bytes);
|
||||
self.rx.sample(at, c.rx_packets, c.rx_bytes);
|
||||
self.total.sample(
|
||||
at,
|
||||
c.tx_packets.saturating_add(c.rx_packets),
|
||||
c.tx_bytes.saturating_add(c.rx_bytes),
|
||||
);
|
||||
self.at = Some(at);
|
||||
}
|
||||
fn active(&self, now: Instant) -> bool {
|
||||
self.at
|
||||
.is_some_and(|at| now.saturating_duration_since(at) <= Duration::from_secs(5))
|
||||
&& self.total.rate(Unit::Packets) > 0.0
|
||||
}
|
||||
}
|
||||
#[derive(Clone, PartialEq, Eq, PartialOrd, Ord, Hash)]
|
||||
struct Key(String, String, u16, String, u16);
|
||||
fn key(v: &VipStatus, l: &ListenerStatus, b: &Backend) -> Key {
|
||||
// Listener identity separates counter histories when a port is recreated.
|
||||
let listener = v
|
||||
.config
|
||||
.listeners
|
||||
.iter()
|
||||
.find(|entry| entry.protocol == l.protocol && entry.port == l.port)
|
||||
.map(|entry| entry.id.clone())
|
||||
.unwrap_or_else(|| l.protocol.to_string());
|
||||
Key(
|
||||
v.config.id.clone(),
|
||||
listener,
|
||||
l.port,
|
||||
b.peer.clone(),
|
||||
b.port,
|
||||
)
|
||||
}
|
||||
struct Camera {
|
||||
rect: egui::Rect,
|
||||
members: Vec<Backend>,
|
||||
}
|
||||
impl Default for Camera {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
rect: egui::Rect::NOTHING,
|
||||
members: Vec::new(),
|
||||
}
|
||||
}
|
||||
}
|
||||
pub(crate) struct View {
|
||||
totals: BTreeMap<String, Rates>,
|
||||
backends: BTreeMap<Key, Rates>,
|
||||
cameras: BTreeMap<(String, String, u16), Camera>,
|
||||
selected: Option<Key>,
|
||||
unit: Unit,
|
||||
}
|
||||
impl Default for View {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
totals: BTreeMap::new(),
|
||||
backends: BTreeMap::new(),
|
||||
cameras: BTreeMap::new(),
|
||||
selected: None,
|
||||
unit: Unit::Bytes,
|
||||
}
|
||||
}
|
||||
}
|
||||
impl View {
|
||||
pub(crate) fn observe(&mut self, network: &NetworkReport, at: Instant) {
|
||||
self.totals
|
||||
.retain(|id, _| network.vips.iter().any(|v| &v.config.id == id));
|
||||
let mut present = std::collections::BTreeSet::new();
|
||||
for v in &network.vips {
|
||||
let mut counters = Counters::default();
|
||||
for l in &v.listeners {
|
||||
counters.tx_packets = counters.tx_packets.saturating_add(l.counters.tx_packets);
|
||||
counters.rx_packets = counters.rx_packets.saturating_add(l.counters.rx_packets);
|
||||
counters.tx_bytes = counters.tx_bytes.saturating_add(l.counters.tx_bytes);
|
||||
counters.rx_bytes = counters.rx_bytes.saturating_add(l.counters.rx_bytes);
|
||||
for b in &l.backends {
|
||||
let key = key(v, l, &b.backend);
|
||||
self.backends
|
||||
.entry(key.clone())
|
||||
.or_default()
|
||||
.sample(at, &b.counters);
|
||||
present.insert(key);
|
||||
}
|
||||
}
|
||||
self.totals
|
||||
.entry(v.config.id.clone())
|
||||
.or_default()
|
||||
.sample(at, &counters);
|
||||
}
|
||||
self.backends.retain(|key, _| present.contains(key));
|
||||
if self
|
||||
.selected
|
||||
.as_ref()
|
||||
.is_some_and(|key| !present.contains(key))
|
||||
{
|
||||
self.selected = None;
|
||||
}
|
||||
self.cameras.retain(|(id, protocol, port), _| {
|
||||
network.vips.iter().any(|v| {
|
||||
&v.config.id == id
|
||||
&& v.listeners
|
||||
.iter()
|
||||
.any(|l| l.protocol.to_string() == *protocol && l.port == *port)
|
||||
})
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
fn name<'a>(n: &'a NetworkReport, id: &'a str) -> &'a str {
|
||||
n.members
|
||||
.iter()
|
||||
.find(|m| m.endpoint_id == id)
|
||||
.and_then(|m| m.hostname.as_deref())
|
||||
.unwrap_or(id)
|
||||
}
|
||||
fn backend_name(n: &NetworkReport, b: &Backend) -> String {
|
||||
format!("{}:{}", name(n, &b.peer), b.port)
|
||||
}
|
||||
fn rtt(b: &BackendStatus) -> String {
|
||||
b.rtt_us
|
||||
.map_or_else(|| "—".into(), |us| format!("{:.2} ms", us as f64 / 1000.0))
|
||||
}
|
||||
fn health_color(state: &str) -> egui::Color32 {
|
||||
match state {
|
||||
"ready" => ACTIVE,
|
||||
"checking port" | "measuring RTT" | "draining" => WARN,
|
||||
_ => BAD,
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn show(ui: &mut egui::Ui, v: &VipStatus, n: &NetworkReport, view: &mut View) {
|
||||
ui.horizontal_wrapped(|ui| {
|
||||
ui.heading(format!("{}.{}", v.config.name, n.name));
|
||||
ui.weak("Traffic seen by this device");
|
||||
ui.selectable_value(&mut view.unit, Unit::Bytes, "bytes");
|
||||
ui.selectable_value(&mut view.unit, Unit::Packets, "pkts");
|
||||
});
|
||||
if let Some(error) = &v.error {
|
||||
ui.colored_label(BAD, error);
|
||||
}
|
||||
let total = view.totals.get(&v.config.id);
|
||||
ui.horizontal_wrapped(|ui| {
|
||||
metric(
|
||||
ui,
|
||||
"TX",
|
||||
&format::series_rate(total.map(|r| &r.tx), view.unit),
|
||||
);
|
||||
metric(
|
||||
ui,
|
||||
"RX",
|
||||
&format::series_rate(total.map(|r| &r.rx), view.unit),
|
||||
);
|
||||
let flows: u64 = v
|
||||
.listeners
|
||||
.iter()
|
||||
.flat_map(|l| &l.backends)
|
||||
.map(|b| u64::from(b.active_flows))
|
||||
.sum();
|
||||
metric(ui, "Open flows", &flows.to_string());
|
||||
});
|
||||
format::sparkline(ui, total.map(|r| &r.total), view.unit, 64.0);
|
||||
ui.add_space(8.0);
|
||||
if v.listeners.is_empty() {
|
||||
ui.weak("No listeners configured. Add one in Configuration.");
|
||||
}
|
||||
for l in &v.listeners {
|
||||
ui.push_id(
|
||||
(
|
||||
"vip live listener",
|
||||
&v.config.id,
|
||||
l.protocol.to_string(),
|
||||
l.port,
|
||||
),
|
||||
|ui| {
|
||||
egui::Frame::group(ui.style()).show(ui, |ui| {
|
||||
ui.set_min_width((ui.available_width() - 2.0).max(0.0));
|
||||
ui.horizontal_wrapped(|ui| {
|
||||
ui.heading(format!("{}/{}", l.protocol, l.port));
|
||||
ui.weak("New connections:");
|
||||
if let Some(b) = &l.selected {
|
||||
ui.colored_label(NEXT, backend_name(n, b));
|
||||
} else {
|
||||
ui.colored_label(WARN, "No eligible backend");
|
||||
}
|
||||
});
|
||||
ui.horizontal_wrapped(|ui| {
|
||||
metric(ui, "TX", &format::bytes(l.counters.tx_bytes as f64));
|
||||
metric(ui, "RX", &format::bytes(l.counters.rx_bytes as f64));
|
||||
metric(
|
||||
ui,
|
||||
"Packets TX / RX",
|
||||
&format!("{} / {}", l.counters.tx_packets, l.counters.rx_packets),
|
||||
);
|
||||
metric(ui, "Drops", &l.counters.dropped.to_string());
|
||||
if l.protocol == Protocol::Udp {
|
||||
metric(ui, "Failovers", &l.counters.failovers.to_string());
|
||||
}
|
||||
});
|
||||
ui.separator();
|
||||
diagram(ui, v, l, n, view);
|
||||
ui.add_space(6.0);
|
||||
egui::ScrollArea::horizontal()
|
||||
.id_salt("backend table scroll")
|
||||
.auto_shrink([false, true])
|
||||
.show(ui, |ui| {
|
||||
backend_table(ui, v, l, n, view);
|
||||
});
|
||||
if let Some(b) = l
|
||||
.backends
|
||||
.iter()
|
||||
.find(|b| view.selected.as_ref() == Some(&key(v, l, &b.backend)))
|
||||
{
|
||||
egui::Frame::group(ui.style()).show(ui, |ui| {
|
||||
ui.horizontal_wrapped(|ui| {
|
||||
ui.strong(backend_name(n, &b.backend));
|
||||
ui.colored_label(health_color(&b.state), &b.state);
|
||||
ui.weak(format!("{} open flows", b.active_flows));
|
||||
if let Some(age) = b.measurement_age_ms {
|
||||
ui.weak(format!(
|
||||
"RTT measured {:.1}s ago",
|
||||
age as f64 / 1000.0
|
||||
));
|
||||
} else {
|
||||
ui.weak("No RTT measurement yet");
|
||||
}
|
||||
});
|
||||
ui.horizontal_wrapped(|ui| {
|
||||
ui.weak("Node");
|
||||
format::copy_field(
|
||||
ui,
|
||||
&format::short(&b.backend.peer),
|
||||
&b.backend.peer,
|
||||
);
|
||||
if let Some(address) = n
|
||||
.members
|
||||
.iter()
|
||||
.find(|m| m.endpoint_id == b.backend.peer)
|
||||
.and_then(|m| m.overlay_address_v4.as_deref())
|
||||
{
|
||||
ui.weak("Address");
|
||||
format::copy_field(ui, address, address);
|
||||
}
|
||||
});
|
||||
});
|
||||
}
|
||||
});
|
||||
},
|
||||
);
|
||||
ui.add_space(8.0);
|
||||
}
|
||||
ui.collapsing("Internal address and identity", |ui| {
|
||||
egui::Grid::new(("vip identity", &v.config.id)).show(ui, |ui| {
|
||||
ui.weak("Address");
|
||||
if let Some(address) = v.address {
|
||||
format::copy_field(ui, &address.to_string(), &address.to_string());
|
||||
} else {
|
||||
ui.weak("Not allocated");
|
||||
}
|
||||
ui.end_row();
|
||||
ui.weak("ID");
|
||||
format::copy_field(ui, &v.config.id, &v.config.id);
|
||||
ui.end_row();
|
||||
});
|
||||
});
|
||||
}
|
||||
fn metric(ui: &mut egui::Ui, title: &str, value: &str) {
|
||||
ui.horizontal(|ui| {
|
||||
ui.weak(title);
|
||||
ui.strong(value);
|
||||
});
|
||||
ui.add_space(8.0);
|
||||
}
|
||||
|
||||
fn backend_table(
|
||||
ui: &mut egui::Ui,
|
||||
v: &VipStatus,
|
||||
l: &ListenerStatus,
|
||||
n: &NetworkReport,
|
||||
view: &mut View,
|
||||
) {
|
||||
let headers = [
|
||||
"Backend",
|
||||
"Health",
|
||||
"Route",
|
||||
"RTT",
|
||||
"Flows",
|
||||
"TX / RX",
|
||||
"TX / RX rate",
|
||||
"Traffic",
|
||||
];
|
||||
let mut widths = [170.0_f32, 110.0, 100.0, 90.0, 50.0, 160.0, 160.0, 160.0];
|
||||
// Measure every value before layout, including long identities and counters.
|
||||
let values: Vec<_> = l
|
||||
.backends
|
||||
.iter()
|
||||
.map(|b| {
|
||||
let rates = view.backends.get(&key(v, l, &b.backend));
|
||||
let route = if rates.is_some_and(|r| r.active(Instant::now())) {
|
||||
"Traffic"
|
||||
} else if l.selected.as_ref() == Some(&b.backend) {
|
||||
"Next flow"
|
||||
} else {
|
||||
"—"
|
||||
};
|
||||
[
|
||||
backend_name(n, &b.backend),
|
||||
b.state.clone(),
|
||||
route.into(),
|
||||
rtt(b),
|
||||
b.active_flows.to_string(),
|
||||
format!(
|
||||
"{} / {}",
|
||||
format::bytes(b.counters.tx_bytes as f64),
|
||||
format::bytes(b.counters.rx_bytes as f64)
|
||||
),
|
||||
format!(
|
||||
"{} / {}",
|
||||
format::series_rate(rates.map(|r| &r.tx), view.unit),
|
||||
format::series_rate(rates.map(|r| &r.rx), view.unit)
|
||||
),
|
||||
]
|
||||
})
|
||||
.collect();
|
||||
for (column, width) in widths.iter_mut().enumerate().take(7) {
|
||||
let font = egui::FontSelection::Default.resolve(ui.style());
|
||||
for text in
|
||||
std::iter::once(headers[column]).chain(values.iter().map(|row| row[column].as_str()))
|
||||
{
|
||||
*width = width.max(
|
||||
ui.fonts(|f| {
|
||||
f.layout_no_wrap(text.to_owned(), font.clone(), egui::Color32::WHITE)
|
||||
.size()
|
||||
.x
|
||||
}) + 12.0,
|
||||
);
|
||||
}
|
||||
}
|
||||
let widths_id = ui.id().with("backend column widths");
|
||||
if let Some(previous) = ui.memory(|memory| memory.data.get_temp::<[f32; 8]>(widths_id)) {
|
||||
for (width, previous) in widths.iter_mut().zip(previous) {
|
||||
*width = width.max(previous);
|
||||
}
|
||||
}
|
||||
ui.memory_mut(|memory| memory.data.insert_temp(widths_id, widths));
|
||||
let fixed = widths.iter().sum::<f32>() + table::GAP * 7.0;
|
||||
widths[7] += (ui.available_width() - fixed).max(0.0);
|
||||
table::row(ui, "backend headers", &widths, false, |column, ui| {
|
||||
ui.weak(headers[column]);
|
||||
});
|
||||
for (index, b) in l.backends.iter().enumerate() {
|
||||
let key = key(v, l, &b.backend);
|
||||
let rates = view.backends.get(&key);
|
||||
table::row(
|
||||
ui,
|
||||
&key,
|
||||
&widths,
|
||||
index % 2 == 0,
|
||||
|column, ui| match column {
|
||||
0 => {
|
||||
if ui
|
||||
.selectable_label(view.selected.as_ref() == Some(&key), &values[index][0])
|
||||
.clicked()
|
||||
{
|
||||
view.selected = Some(key.clone());
|
||||
}
|
||||
}
|
||||
1 => {
|
||||
ui.colored_label(health_color(&b.state), &values[index][1])
|
||||
.on_hover_text(&b.state);
|
||||
}
|
||||
2 => {
|
||||
ui.colored_label(
|
||||
if values[index][2] == "Traffic" {
|
||||
ACTIVE
|
||||
} else {
|
||||
NEXT
|
||||
},
|
||||
&values[index][2],
|
||||
)
|
||||
.on_hover_text(
|
||||
if l.selected.as_ref() == Some(&b.backend) {
|
||||
"Selected for new connections. Existing flows can use other backends."
|
||||
} else {
|
||||
"Traffic during the latest sample interval."
|
||||
},
|
||||
);
|
||||
}
|
||||
3..=6 => {
|
||||
ui.label(&values[index][column]);
|
||||
}
|
||||
_ => format::sparkline(ui, rates.map(|r| &r.total), view.unit, 24.0),
|
||||
},
|
||||
);
|
||||
}
|
||||
if l.backends.is_empty() {
|
||||
ui.weak("No backends. Add a node and target port in Configuration.");
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy)]
|
||||
struct Branch {
|
||||
active: bool,
|
||||
next: bool,
|
||||
}
|
||||
fn branch(l: &ListenerStatus, b: &BackendStatus, rates: Option<&Rates>, now: Instant) -> Branch {
|
||||
Branch {
|
||||
active: rates.is_some_and(|r| r.active(now)),
|
||||
next: l.selected.as_ref() == Some(&b.backend),
|
||||
}
|
||||
}
|
||||
|
||||
fn diagram(
|
||||
ui: &mut egui::Ui,
|
||||
v: &VipStatus,
|
||||
l: &ListenerStatus,
|
||||
n: &NetworkReport,
|
||||
view: &mut View,
|
||||
) {
|
||||
let camera = view
|
||||
.cameras
|
||||
.entry((v.config.id.clone(), l.protocol.to_string(), l.port))
|
||||
.or_default();
|
||||
let members: Vec<_> = l.backends.iter().map(|b| b.backend.clone()).collect();
|
||||
let height = (members.len().max(1) as f32 * 92.0 + 32.0).max(180.0);
|
||||
let bounds = egui::Rect::from_min_size(egui::Pos2::ZERO, egui::vec2(860.0, height));
|
||||
if camera.members != members || !camera.rect.is_positive() {
|
||||
camera.rect = bounds;
|
||||
camera.members = members;
|
||||
}
|
||||
ui.horizontal_wrapped(|ui| {
|
||||
legend(ui, "Traffic", ACTIVE, true);
|
||||
legend(ui, "Next new flow", NEXT, false);
|
||||
ui.weak("Click a backend · drag to pan · Ctrl+scroll to zoom");
|
||||
if ui.small_button("Fit").clicked() {
|
||||
camera.rect = bounds;
|
||||
}
|
||||
});
|
||||
let size = egui::vec2(ui.available_width(), height.clamp(180.0, 360.0));
|
||||
let (outer, _) = ui.allocate_exact_size(size, egui::Sense::hover());
|
||||
ui.painter()
|
||||
.rect_filled(outer, 6.0, ui.visuals().extreme_bg_color);
|
||||
let mut child = ui.new_child(egui::UiBuilder::new().id_salt("vip graph").max_rect(outer));
|
||||
egui::Scene::new()
|
||||
.zoom_range(0.05..=1.5)
|
||||
.max_inner_size(bounds.size())
|
||||
.show(&mut child, &mut camera.rect, |ui| {
|
||||
ui.set_min_size(bounds.size());
|
||||
let center = height / 2.0;
|
||||
let source =
|
||||
egui::Rect::from_min_size(egui::pos2(18.0, center - 32.0), egui::vec2(204.0, 64.0));
|
||||
let listener = egui::Rect::from_min_size(
|
||||
egui::pos2(290.0, center - 32.0),
|
||||
egui::vec2(146.0, 64.0),
|
||||
);
|
||||
let active = l.backends.iter().any(|b| {
|
||||
view.backends
|
||||
.get(&key(v, l, &b.backend))
|
||||
.is_some_and(|r| r.active(Instant::now()))
|
||||
});
|
||||
edge(
|
||||
ui,
|
||||
source.right_center(),
|
||||
listener.left_center(),
|
||||
if active { ACTIVE } else { NEXT },
|
||||
active,
|
||||
);
|
||||
node(
|
||||
ui,
|
||||
source,
|
||||
&format!("{}.{}", v.config.name, n.name),
|
||||
"This device",
|
||||
NEXT,
|
||||
false,
|
||||
);
|
||||
node(
|
||||
ui,
|
||||
listener,
|
||||
&format!("{}/{}", l.protocol, l.port),
|
||||
"Listener",
|
||||
NEXT,
|
||||
false,
|
||||
);
|
||||
for (index, b) in l.backends.iter().enumerate() {
|
||||
let key = key(v, l, &b.backend);
|
||||
let rates = view.backends.get(&key);
|
||||
let state = branch(l, b, rates, Instant::now());
|
||||
let rect = egui::Rect::from_min_size(
|
||||
egui::pos2(558.0, 24.0 + index as f32 * 92.0),
|
||||
egui::vec2(278.0, 68.0),
|
||||
);
|
||||
let color = if state.active {
|
||||
ACTIVE
|
||||
} else if state.next {
|
||||
NEXT
|
||||
} else if b.state == "ready" {
|
||||
ui.visuals().weak_text_color()
|
||||
} else {
|
||||
health_color(&b.state)
|
||||
};
|
||||
edge(
|
||||
ui,
|
||||
listener.right_center(),
|
||||
rect.left_center(),
|
||||
color,
|
||||
state.active,
|
||||
);
|
||||
if state.next {
|
||||
ui.painter().circle_stroke(
|
||||
rect.left_center(),
|
||||
5.0,
|
||||
egui::Stroke::new(2.0_f32, NEXT),
|
||||
);
|
||||
}
|
||||
let response = ui
|
||||
.interact(rect, ui.id().with((&key.3, key.4)), egui::Sense::click())
|
||||
.on_hover_cursor(egui::CursorIcon::PointingHand)
|
||||
.on_hover_text(format!(
|
||||
"{}\n{} · RTT {}\n{} open flows\n{}{}",
|
||||
backend_name(n, &b.backend),
|
||||
b.state,
|
||||
rtt(b),
|
||||
b.active_flows,
|
||||
if state.active {
|
||||
"Traffic in latest sample. "
|
||||
} else {
|
||||
""
|
||||
},
|
||||
if state.next {
|
||||
"Selected for new connections."
|
||||
} else {
|
||||
""
|
||||
}
|
||||
));
|
||||
if response.clicked() {
|
||||
view.selected = Some(key.clone());
|
||||
}
|
||||
let detail = format!("{} · {} · {} flows", b.state, rtt(b), b.active_flows);
|
||||
node(
|
||||
ui,
|
||||
rect,
|
||||
&backend_name(n, &b.backend),
|
||||
&detail,
|
||||
color,
|
||||
view.selected.as_ref() == Some(&key) || response.hovered(),
|
||||
);
|
||||
}
|
||||
if l.backends.is_empty() {
|
||||
ui.painter().text(
|
||||
egui::pos2(570.0, center),
|
||||
egui::Align2::LEFT_CENTER,
|
||||
"No backends",
|
||||
egui::FontId::proportional(15.0),
|
||||
ui.visuals().weak_text_color(),
|
||||
);
|
||||
}
|
||||
});
|
||||
}
|
||||
fn legend(ui: &mut egui::Ui, text: &str, color: egui::Color32, filled: bool) {
|
||||
ui.horizontal(|ui| {
|
||||
let (rect, _) = ui.allocate_exact_size(egui::vec2(12.0, 12.0), egui::Sense::hover());
|
||||
if filled {
|
||||
ui.painter().circle_filled(rect.center(), 4.0, color);
|
||||
} else {
|
||||
ui.painter()
|
||||
.circle_stroke(rect.center(), 4.0, egui::Stroke::new(1.5_f32, color));
|
||||
}
|
||||
ui.colored_label(color, text);
|
||||
});
|
||||
}
|
||||
fn node(
|
||||
ui: &egui::Ui,
|
||||
rect: egui::Rect,
|
||||
title: &str,
|
||||
detail: &str,
|
||||
color: egui::Color32,
|
||||
selected: bool,
|
||||
) {
|
||||
let painter = ui.painter_at(rect);
|
||||
painter.rect_filled(rect, 8.0, ui.visuals().widgets.inactive.bg_fill);
|
||||
painter.rect_stroke(
|
||||
rect.shrink(1.0),
|
||||
8.0,
|
||||
egui::Stroke::new(if selected { 3.0_f32 } else { 1.5_f32 }, color),
|
||||
egui::StrokeKind::Inside,
|
||||
);
|
||||
text_line(
|
||||
&painter,
|
||||
rect.min + egui::vec2(12.0, 9.0),
|
||||
title,
|
||||
16.0,
|
||||
ui.visuals().strong_text_color(),
|
||||
rect.width() - 24.0,
|
||||
);
|
||||
text_line(
|
||||
&painter,
|
||||
rect.min + egui::vec2(12.0, 34.0),
|
||||
detail,
|
||||
12.0,
|
||||
ui.visuals().text_color(),
|
||||
rect.width() - 24.0,
|
||||
);
|
||||
}
|
||||
fn text_line(
|
||||
painter: &egui::Painter,
|
||||
position: egui::Pos2,
|
||||
text: &str,
|
||||
size: f32,
|
||||
color: egui::Color32,
|
||||
width: f32,
|
||||
) {
|
||||
let mut job = egui::text::LayoutJob::simple_singleline(
|
||||
text.into(),
|
||||
egui::FontId::proportional(size),
|
||||
color,
|
||||
);
|
||||
job.wrap.max_width = width;
|
||||
job.wrap.max_rows = 1;
|
||||
job.wrap.break_anywhere = true;
|
||||
painter.galley(position, painter.layout_job(job), color);
|
||||
}
|
||||
fn edge(ui: &egui::Ui, start: egui::Pos2, end: egui::Pos2, color: egui::Color32, active: bool) {
|
||||
let bend = (end.x - start.x) * 0.5;
|
||||
let curve = egui::epaint::CubicBezierShape::from_points_stroke(
|
||||
[
|
||||
start,
|
||||
start + egui::vec2(bend, 0.0),
|
||||
end - egui::vec2(bend, 0.0),
|
||||
end,
|
||||
],
|
||||
false,
|
||||
egui::Color32::TRANSPARENT,
|
||||
egui::Stroke::new(if active { 3.0_f32 } else { 1.5_f32 }, color),
|
||||
);
|
||||
ui.painter().add(curve.clone());
|
||||
if active {
|
||||
let t = ui.input(|i| i.time) as f32 * 0.5 % 1.0;
|
||||
let position = curve.sample(t);
|
||||
ui.painter().circle_filled(position, 4.0, color);
|
||||
ui.ctx().request_repaint_after(Duration::from_millis(40));
|
||||
}
|
||||
ui.painter().arrow(
|
||||
end - egui::vec2(12.0, 0.0),
|
||||
egui::vec2(10.0, 0.0),
|
||||
egui::Stroke::new(1.5_f32, color),
|
||||
);
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
|
||||
use super::*;
|
||||
use tsunagi::vip::{Listener, VipConfig};
|
||||
fn fixture() -> (NetworkReport, VipStatus) {
|
||||
let mut config = VipConfig::new("k8s");
|
||||
config.listeners.push(Listener::new(Protocol::Tcp, 6443));
|
||||
let backends: Vec<_> = ["ai", "master", "music"]
|
||||
.into_iter()
|
||||
.map(|peer| BackendStatus {
|
||||
backend: Backend {
|
||||
peer: peer.into(),
|
||||
port: 7443,
|
||||
},
|
||||
state: "ready".into(),
|
||||
rtt_us: Some(4500),
|
||||
measurement_age_ms: Some(500),
|
||||
active_flows: 5,
|
||||
counters: Counters::default(),
|
||||
})
|
||||
.collect();
|
||||
let v = VipStatus {
|
||||
config,
|
||||
revision: "r1".into(),
|
||||
address: None,
|
||||
error: None,
|
||||
listeners: vec![ListenerStatus {
|
||||
protocol: Protocol::Tcp,
|
||||
port: 6443,
|
||||
selected: Some(backends[0].backend.clone()),
|
||||
backends,
|
||||
counters: Counters::default(),
|
||||
}],
|
||||
};
|
||||
let n = NetworkReport {
|
||||
name: "test".into(),
|
||||
vips: vec![v.clone()],
|
||||
..Default::default()
|
||||
};
|
||||
(n, v)
|
||||
}
|
||||
#[test]
|
||||
fn traffic_branches_are_sampled_independently_of_new_flow_selection_and_open_flows() {
|
||||
let (mut n, _) = fixture();
|
||||
let now = Instant::now();
|
||||
let mut view = View::default();
|
||||
view.observe(&n, now);
|
||||
n.vips[0].listeners[0].backends[2].counters.tx_packets = 20;
|
||||
n.vips[0].listeners[0].backends[2].counters.tx_bytes = 4000;
|
||||
n.vips[0].listeners[0].counters.tx_packets = 20;
|
||||
n.vips[0].listeners[0].counters.tx_bytes = 4000;
|
||||
view.observe(&n, now + Duration::from_secs(2));
|
||||
let v = &n.vips[0];
|
||||
let l = &v.listeners[0];
|
||||
let selected = branch(
|
||||
l,
|
||||
&l.backends[0],
|
||||
view.backends.get(&key(v, l, &l.backends[0].backend)),
|
||||
now + Duration::from_secs(2),
|
||||
);
|
||||
let carrying = branch(
|
||||
l,
|
||||
&l.backends[2],
|
||||
view.backends.get(&key(v, l, &l.backends[2].backend)),
|
||||
now + Duration::from_secs(2),
|
||||
);
|
||||
assert!(selected.next && !selected.active);
|
||||
assert!(carrying.active && !carrying.next);
|
||||
assert_eq!(view.totals[&v.config.id].total.rate(Unit::Bytes), 2000.0);
|
||||
assert!(
|
||||
!view.backends[&key(v, l, &l.backends[2].backend)].active(now + Duration::from_secs(8))
|
||||
);
|
||||
view.observe(&n, now + Duration::from_secs(4));
|
||||
assert!(
|
||||
!view.backends[&key(v, l, &l.backends[2].backend)].active(now + Duration::from_secs(4))
|
||||
);
|
||||
}
|
||||
#[test]
|
||||
fn removed_services_and_backends_release_history_selection_and_cameras() {
|
||||
let (mut n, v) = fixture();
|
||||
let mut view = View::default();
|
||||
view.observe(&n, Instant::now());
|
||||
view.selected = Some(key(
|
||||
&v,
|
||||
&v.listeners[0],
|
||||
&v.listeners[0].backends[0].backend,
|
||||
));
|
||||
n.vips.clear();
|
||||
view.observe(&n, Instant::now());
|
||||
assert!(
|
||||
view.backends.is_empty()
|
||||
&& view.totals.is_empty()
|
||||
&& view.selected.is_none()
|
||||
&& view.cameras.is_empty()
|
||||
);
|
||||
}
|
||||
#[test]
|
||||
fn clicking_a_scene_node_selects_its_backend_without_changing_configuration() {
|
||||
let (n, v) = fixture();
|
||||
let ctx = egui::Context::default();
|
||||
let mut view = View::default();
|
||||
let frame = |view: &mut View, events| {
|
||||
ctx.run(
|
||||
egui::RawInput {
|
||||
screen_rect: Some(egui::Rect::from_min_size(
|
||||
egui::Pos2::ZERO,
|
||||
egui::vec2(1000.0, 600.0),
|
||||
)),
|
||||
events,
|
||||
..Default::default()
|
||||
},
|
||||
|ctx| {
|
||||
egui::CentralPanel::default()
|
||||
.show(ctx, |ui| diagram(ui, &v, &v.listeners[0], &n, view));
|
||||
},
|
||||
)
|
||||
};
|
||||
frame(&mut view, vec![]);
|
||||
let output = frame(&mut view, vec![]);
|
||||
let pos = output
|
||||
.shapes
|
||||
.iter()
|
||||
.find_map(|s| match &s.shape {
|
||||
egui::Shape::Text(t) if t.galley.text() == "music:7443" => {
|
||||
Some(t.pos + egui::vec2(20.0, 10.0))
|
||||
}
|
||||
_ => None,
|
||||
})
|
||||
.expect("scene draws the backend node");
|
||||
for pressed in [true, false] {
|
||||
frame(
|
||||
&mut view,
|
||||
vec![
|
||||
egui::Event::PointerMoved(pos),
|
||||
egui::Event::PointerButton {
|
||||
pos,
|
||||
button: egui::PointerButton::Primary,
|
||||
pressed,
|
||||
modifiers: egui::Modifiers::NONE,
|
||||
},
|
||||
],
|
||||
);
|
||||
}
|
||||
assert!(
|
||||
view.selected
|
||||
== Some(key(
|
||||
&v,
|
||||
&v.listeners[0],
|
||||
&v.listeners[0].backends[2].backend
|
||||
))
|
||||
);
|
||||
assert_eq!(n.vips[0].config, v.config);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn reset_counters_and_recreated_listeners_do_not_inherit_active_traffic() {
|
||||
let (mut n, _) = fixture();
|
||||
let mut view = View::default();
|
||||
let at = Instant::now();
|
||||
n.vips[0].listeners[0].backends[0].counters.tx_packets = 100;
|
||||
view.observe(&n, at);
|
||||
n.vips[0].listeners[0].backends[0].counters.tx_packets = 0;
|
||||
view.observe(&n, at + Duration::from_secs(2));
|
||||
let v = &n.vips[0];
|
||||
let l = &v.listeners[0];
|
||||
assert!(
|
||||
!view.backends[&key(v, l, &l.backends[0].backend)].active(at + Duration::from_secs(2))
|
||||
);
|
||||
n.vips[0].config.listeners[0].id = "new listener".into();
|
||||
n.vips[0].listeners[0].backends[0].counters.tx_packets = 200;
|
||||
view.observe(&n, at + Duration::from_secs(4));
|
||||
let v = &n.vips[0];
|
||||
let l = &v.listeners[0];
|
||||
assert!(
|
||||
!view.backends[&key(v, l, &l.backends[0].backend)].active(at + Duration::from_secs(4))
|
||||
);
|
||||
assert_eq!(view.backends.len(), 3);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn live_tables_and_scenes_render_at_narrow_and_wide_sizes() {
|
||||
let (n, v) = fixture();
|
||||
let ctx = egui::Context::default();
|
||||
let mut view = View::default();
|
||||
for width in [650.0, 1000.0, 1800.0, 650.0] {
|
||||
for _ in 0..2 {
|
||||
let _ = ctx.run(
|
||||
egui::RawInput {
|
||||
screen_rect: Some(egui::Rect::from_min_size(
|
||||
egui::Pos2::ZERO,
|
||||
egui::vec2(width, 1000.0),
|
||||
)),
|
||||
..Default::default()
|
||||
},
|
||||
|ctx| {
|
||||
egui::CentralPanel::default().show(ctx, |ui| {
|
||||
egui::ScrollArea::vertical().show(ui, |ui| show(ui, &v, &n, &mut view));
|
||||
assert!(
|
||||
ui.min_rect().right() <= width + 1.0,
|
||||
"VIP content expands the window at {width}"
|
||||
);
|
||||
});
|
||||
},
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -22,6 +22,8 @@ pub(crate) enum Health {
|
||||
Disconnected,
|
||||
/// Connected, nothing out of the ordinary.
|
||||
Connected,
|
||||
/// At least one configured network is blocked by an overlapping active range.
|
||||
RangeConflict,
|
||||
/// This device's traffic leaves through an exit node, and that works.
|
||||
ExitNode,
|
||||
}
|
||||
@@ -33,6 +35,13 @@ impl Health {
|
||||
let Some(report) = report else {
|
||||
return Self::Disconnected;
|
||||
};
|
||||
if report
|
||||
.networks
|
||||
.iter()
|
||||
.any(|network| network.auto_start && !network.range_conflicts.is_empty())
|
||||
{
|
||||
return Self::RangeConflict;
|
||||
}
|
||||
if !report
|
||||
.networks
|
||||
.iter()
|
||||
@@ -56,6 +65,7 @@ impl Health {
|
||||
match self {
|
||||
Self::Disconnected => "tsunagi: not connected",
|
||||
Self::Connected => "tsunagi: connected",
|
||||
Self::RangeConflict => "tsunagi: network range conflict; open to switch networks",
|
||||
Self::ExitNode => "tsunagi: using an exit node",
|
||||
}
|
||||
}
|
||||
@@ -108,6 +118,24 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn overlapping_networks_are_visible_even_while_other_peers_are_connected() {
|
||||
let mut blocked = NetworkReport {
|
||||
auto_start: true,
|
||||
..Default::default()
|
||||
};
|
||||
blocked
|
||||
.range_conflicts
|
||||
.push(tsunagi::ipc::RangeConflictReport {
|
||||
name: "other".into(),
|
||||
network_id: "other-id".into(),
|
||||
range: "10.90.0.0/16".into(),
|
||||
});
|
||||
let status = report(vec![network(2, ExitReport::default()), blocked]);
|
||||
assert_eq!(Health::of(Some(&status)), Health::RangeConflict);
|
||||
assert!(Health::RangeConflict.label().contains("switch networks"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn no_agent_or_nobody_connected_is_the_red_state() {
|
||||
assert_eq!(Health::of(None), Health::Disconnected);
|
||||
|
||||
@@ -79,6 +79,8 @@ pub const WIREGUARD_OVERHEAD: u32 = 32;
|
||||
/// Configuration of the WireGuard plugin.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct WireguardConfig {
|
||||
/// Diagnostic protocol id when this end-to-end engine is reused by another carrier.
|
||||
pub protocol: &'static str,
|
||||
/// Directory for the plugin's own key store. Separate from agent state.
|
||||
pub state_dir: PathBuf,
|
||||
/// WireGuard keepalive, which keeps tunnels and their links warm.
|
||||
@@ -100,6 +102,7 @@ impl WireguardConfig {
|
||||
/// Creates a configuration rooted at `state_dir` with sensible defaults.
|
||||
pub fn new(state_dir: impl Into<PathBuf>) -> Self {
|
||||
Self {
|
||||
protocol: WIREGUARD_PROTOCOL,
|
||||
state_dir: state_dir.into(),
|
||||
keepalive: Some(25),
|
||||
mtu: DEFAULT_MTU,
|
||||
@@ -394,7 +397,7 @@ impl Worker {
|
||||
fn report(&self, network: NetworkId, reason: impl std::fmt::Display) {
|
||||
tracing::warn!(network = %network.fmt_short(), %reason, "wireguard plugin error");
|
||||
if let Some(context) = self.context.get() {
|
||||
context.report_error(network, WIREGUARD_PROTOCOL, reason.to_string());
|
||||
context.report_error(network, self.config.protocol, reason.to_string());
|
||||
}
|
||||
}
|
||||
|
||||
@@ -647,7 +650,7 @@ async fn run(worker: Arc<Worker>, mut commands: mpsc::Receiver<Command>) {
|
||||
|
||||
impl IpPlugin for WireguardPlugin {
|
||||
fn protocol_id(&self) -> &str {
|
||||
WIREGUARD_PROTOCOL
|
||||
self.worker.config.protocol
|
||||
}
|
||||
|
||||
fn protocol_version(&self) -> u16 {
|
||||
@@ -686,7 +689,7 @@ impl IpPlugin for WireguardPlugin {
|
||||
// Identity only. Where to send packets is the transport's business.
|
||||
let announcement = WgAnnouncement::new(network, &state.key.public());
|
||||
Ok(Some(PluginCapability {
|
||||
protocol: WIREGUARD_PROTOCOL.to_string(),
|
||||
protocol: self.worker.config.protocol.to_string(),
|
||||
version: crate::announcement::ANNOUNCEMENT_VERSION,
|
||||
enabled: true,
|
||||
data: announcement.encode()?,
|
||||
|
||||
@@ -42,6 +42,7 @@ fn source(agent: Agent, plugin: Arc<WireguardPlugin>) -> Arc<dyn tsunagi::ipc::R
|
||||
transport: format!("{:?}", peer.transport),
|
||||
rtt_ms: peer.rtt.map(|rtt| rtt.as_millis() as u64),
|
||||
exit_node: peer.exit_node,
|
||||
path: Default::default(),
|
||||
})
|
||||
.collect(),
|
||||
overlay: plugin.overview(net.network_id).map(|view| {
|
||||
@@ -276,6 +277,7 @@ impl tsunagi::ipc::ReportSource for Control {
|
||||
network_id: network_id.to_string(),
|
||||
already_configured: already,
|
||||
name_shared_with: None,
|
||||
active: self.0.is_active(network_id).await,
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
@@ -109,6 +109,30 @@ impl WgAgent {
|
||||
.await
|
||||
}
|
||||
|
||||
async fn spawn_auto(discovery: &SharedMemoryDiscovery, tag: &str) -> Self {
|
||||
let dir = TempDir::new().unwrap();
|
||||
let tuns = MemoryTunFactory::new();
|
||||
let plugin = WireguardPlugin::open(
|
||||
WireguardConfig::new(dir.path().join("wireguard"))
|
||||
.with_reconcile(Duration::from_millis(20), Duration::from_millis(250)),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
let agent = Agent::spawn(
|
||||
config_with(dir.path(), discovery)
|
||||
.with_interface(Arc::new(tuns.clone()), tag, 1280)
|
||||
.with_plugin(plugin.clone() as Arc<dyn IpPlugin>),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
Self {
|
||||
dir,
|
||||
agent,
|
||||
plugin,
|
||||
tuns,
|
||||
}
|
||||
}
|
||||
|
||||
/// Starts an agent with an explicit overlay range, or none at all.
|
||||
async fn spawn_range(
|
||||
discovery: &SharedMemoryDiscovery,
|
||||
@@ -989,7 +1013,7 @@ async fn two_networks_share_one_interface_with_keys_and_ranges_of_their_own() {
|
||||
let (name_b, secret_b) = network("wg-right");
|
||||
let beta_range = Some("10.99.0.0/16".parse::<Ipv4Range>().unwrap());
|
||||
|
||||
let hub = WgAgent::spawn(&discovery, "th").await;
|
||||
let hub = WgAgent::spawn_auto(&discovery, "th").await;
|
||||
let left = WgAgent::spawn(&discovery, "tl").await;
|
||||
let right = WgAgent::spawn_range(&discovery, "tr", beta_range).await;
|
||||
|
||||
@@ -1661,8 +1685,10 @@ async fn lan_broadcast_fanout_is_scoped_opt_in_and_does_not_reflood() {
|
||||
bytes[28..].copy_from_slice(b"LAN GAME");
|
||||
// UDP checksum zero is valid for IPv4.
|
||||
let sum: u32 = bytes[..20]
|
||||
.chunks_exact(2)
|
||||
.map(|w| u16::from_be_bytes([w[0], w[1]]) as u32)
|
||||
.as_chunks::<2>()
|
||||
.0
|
||||
.iter()
|
||||
.map(|w| u16::from_be_bytes(*w) as u32)
|
||||
.sum();
|
||||
let checksum = !((sum & 0xffff) + (sum >> 16)) as u16;
|
||||
bytes[10..12].copy_from_slice(&checksum.to_be_bytes());
|
||||
@@ -1744,3 +1770,188 @@ async fn lan_broadcast_fanout_is_scoped_opt_in_and_does_not_reflood() {
|
||||
agent.shutdown().await;
|
||||
}
|
||||
}
|
||||
|
||||
fn vip_segment(
|
||||
source: Ipv4Addr,
|
||||
destination: Ipv4Addr,
|
||||
sport: u16,
|
||||
dport: u16,
|
||||
udp: bool,
|
||||
flags: u8,
|
||||
) -> Bytes {
|
||||
fn sum(bytes: &[u8]) -> u16 {
|
||||
let mut s: u32 = bytes
|
||||
.chunks(2)
|
||||
.map(|b| u32::from(u16::from_be_bytes([b[0], *b.get(1).unwrap_or(&0)])))
|
||||
.sum();
|
||||
while s >> 16 != 0 {
|
||||
s = (s & 0xffff) + (s >> 16);
|
||||
}
|
||||
!(s as u16)
|
||||
}
|
||||
let mut p = vec![0; if udp { 36 } else { 40 }];
|
||||
p[0] = 0x45;
|
||||
let len = p.len() as u16;
|
||||
p[2..4].copy_from_slice(&len.to_be_bytes());
|
||||
p[8] = 64;
|
||||
p[9] = if udp { 17 } else { 6 };
|
||||
p[12..16].copy_from_slice(&source.octets());
|
||||
p[16..20].copy_from_slice(&destination.octets());
|
||||
p[20..22].copy_from_slice(&sport.to_be_bytes());
|
||||
p[22..24].copy_from_slice(&dport.to_be_bytes());
|
||||
if udp {
|
||||
p[24..26].copy_from_slice(&16u16.to_be_bytes());
|
||||
} else {
|
||||
p[32] = 0x50;
|
||||
p[33] = flags;
|
||||
p[24..28].copy_from_slice(&123u32.to_be_bytes());
|
||||
}
|
||||
let mut pseudo = p[12..20].to_vec();
|
||||
pseudo.extend([0, p[9]]);
|
||||
pseudo.extend(((p.len() - 20) as u16).to_be_bytes());
|
||||
pseudo.extend(&p[20..]);
|
||||
let c = sum(&pseudo);
|
||||
let offset = if udp { 26 } else { 36 };
|
||||
p[offset..offset + 2].copy_from_slice(&c.to_be_bytes());
|
||||
let c = sum(&p[..20]);
|
||||
p[10..12].copy_from_slice(&c.to_be_bytes());
|
||||
Bytes::from(p)
|
||||
}
|
||||
#[tokio::test]
|
||||
async fn vip_tcp_and_udp_remap_ports_over_real_encryption_and_preserve_source_admission() {
|
||||
use tsunagi::vip::{Backend, Listener, Protocol, VipConfig};
|
||||
let discovery = SharedMemoryDiscovery::new();
|
||||
let (name, secret) = network("vip-encrypted");
|
||||
let a = WgAgent::spawn(&discovery, "vipa").await;
|
||||
let b = WgAgent::spawn(&discovery, "vipb").await;
|
||||
let net = a.agent.join_network(&name, &secret).await.unwrap();
|
||||
b.agent.join_network(&name, &secret).await.unwrap();
|
||||
a.wait_for_tunnels(net, 1).await;
|
||||
b.wait_for_tunnels(net, 1).await;
|
||||
let aa = a.overlay(net).await;
|
||||
let ab = b.overlay(net).await;
|
||||
let ta = a.tun(net).await;
|
||||
let tb = b.tun(net).await;
|
||||
let mut cfg = VipConfig::new("service");
|
||||
for (protocol, port, backend) in [(Protocol::Tcp, 443, 6443), (Protocol::Udp, 53, 5353)] {
|
||||
let mut l = Listener::new(protocol, port);
|
||||
l.port_check = false;
|
||||
l.backends.push(Backend {
|
||||
peer: b.endpoint_id().to_string(),
|
||||
port: backend,
|
||||
});
|
||||
cfg.listeners.push(l);
|
||||
}
|
||||
a.agent
|
||||
.save_vip(net, cfg.clone(), None, false)
|
||||
.await
|
||||
.unwrap();
|
||||
let vip = wait_until("data-plane probes measure the backend RTT", || async {
|
||||
let status = a.agent.network_status(net).await.ok()?;
|
||||
let v = status.vips.iter().find(|v| v.config.id == cfg.id)?;
|
||||
(v.listeners.iter().all(|l| l.selected.is_some()))
|
||||
.then_some(v.address?)
|
||||
.filter(|_| v.listeners.len() == 2)
|
||||
})
|
||||
.await;
|
||||
for (udp, port, target) in [(false, 443, 6443), (true, 53, 5353)] {
|
||||
ta.push_from_os(vip_segment(aa, vip, 40000, port, udp, 2));
|
||||
let received = tokio::time::timeout(Duration::from_secs(5), tb.pop_to_os())
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
assert_eq!(&received[12..16], &aa.octets());
|
||||
assert_eq!(&received[16..20], &ab.octets());
|
||||
assert_eq!(u16::from_be_bytes([received[22], received[23]]), target);
|
||||
let translated = u16::from_be_bytes([received[20], received[21]]);
|
||||
tb.push_from_os(vip_segment(ab, aa, target, translated, udp, 0x12));
|
||||
let reply = tokio::time::timeout(Duration::from_secs(5), ta.pop_to_os())
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
assert_eq!(&reply[12..16], &vip.octets());
|
||||
assert_eq!(u16::from_be_bytes([reply[20], reply[21]]), port);
|
||||
assert_eq!(u16::from_be_bytes([reply[22], reply[23]]), 40000);
|
||||
// A backend cannot send a VIP source over its authenticated tunnel.
|
||||
let forged = vip_segment(vip, aa, port, 40000, udp, 0x12);
|
||||
tb.push_from_os(forged);
|
||||
}
|
||||
wait_until("forged VIP sources are rejected before NAT", || async {
|
||||
(a.agent.overlay()?.counters.wrong_source >= 2).then_some(())
|
||||
})
|
||||
.await;
|
||||
a.shutdown().await;
|
||||
b.shutdown().await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn vip_udp_reaches_another_backend_after_the_selected_agent_stops() {
|
||||
use tsunagi::vip::{Backend, Listener, Protocol, VipConfig};
|
||||
let discovery = SharedMemoryDiscovery::new();
|
||||
let (name, secret) = network("vip-failover");
|
||||
let a = WgAgent::spawn(&discovery, "vipfa").await;
|
||||
let b = WgAgent::spawn(&discovery, "vipfb").await;
|
||||
let c = WgAgent::spawn(&discovery, "vipfc").await;
|
||||
let net = a.agent.join_network(&name, &secret).await.unwrap();
|
||||
for node in [&b, &c] {
|
||||
node.agent.join_network(&name, &secret).await.unwrap();
|
||||
}
|
||||
for node in [&a, &b, &c] {
|
||||
node.wait_for_tunnels(net, 2).await;
|
||||
}
|
||||
let mut config = VipConfig::new("datagrams");
|
||||
let mut listener = Listener::new(Protocol::Udp, 8888);
|
||||
listener.backends = vec![
|
||||
Backend {
|
||||
peer: b.endpoint_id().to_string(),
|
||||
port: 9000,
|
||||
},
|
||||
Backend {
|
||||
peer: c.endpoint_id().to_string(),
|
||||
port: 9001,
|
||||
},
|
||||
];
|
||||
config.listeners.push(listener);
|
||||
a.agent
|
||||
.save_vip(net, config.clone(), None, false)
|
||||
.await
|
||||
.unwrap();
|
||||
let first = wait_until("both UDP backends have measured paths", || async {
|
||||
let s = a.agent.network_status(net).await.ok()?;
|
||||
let v = s.vips.iter().find(|v| v.config.id == config.id)?;
|
||||
let l = v.listeners.first()?;
|
||||
(l.backends.iter().all(|b| b.rtt_us.is_some())).then_some((v.address?, l.selected.clone()?))
|
||||
})
|
||||
.await;
|
||||
let (dead, alive) = if first.1.peer == b.endpoint_id().to_string() {
|
||||
(b, c)
|
||||
} else {
|
||||
(c, b)
|
||||
};
|
||||
let address = a.overlay(net).await;
|
||||
let tun = a.tun(net).await;
|
||||
let dest = dead.tun(net).await;
|
||||
tun.push_from_os(vip_segment(address, first.0, 41000, 8888, true, 0));
|
||||
tokio::time::timeout(Duration::from_secs(5), dest.pop_to_os())
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
dead.shutdown().await;
|
||||
wait_until("the remaining backend becomes the next choice", || async {
|
||||
let s = a.agent.network_status(net).await.ok()?;
|
||||
let v = s.vips.iter().find(|v| v.config.id == config.id)?;
|
||||
(v.listeners.first()?.selected.as_ref()?.peer == alive.endpoint_id().to_string())
|
||||
.then_some(())
|
||||
})
|
||||
.await;
|
||||
tun.push_from_os(vip_segment(address, first.0, 41000, 8888, true, 0));
|
||||
let packet = tokio::time::timeout(Duration::from_secs(5), alive.tun(net).await.pop_to_os())
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
assert_eq!(&packet[16..20], &alive.overlay(net).await.octets());
|
||||
let view = a.agent.network_status(net).await.unwrap();
|
||||
assert_eq!(view.vips[0].listeners[0].counters.failovers, 1);
|
||||
a.shutdown().await;
|
||||
alive.shutdown().await;
|
||||
}
|
||||
@@ -231,6 +231,9 @@ impl WireguardLink {
|
||||
}
|
||||
|
||||
impl PacketLink for WireguardLink {
|
||||
fn close(&self) {
|
||||
WireguardLink::close(self);
|
||||
}
|
||||
fn network(&self) -> NetworkId {
|
||||
self.network
|
||||
}
|
||||
|
||||
@@ -14,6 +14,7 @@ path = "src/main.rs"
|
||||
[dependencies]
|
||||
tsunagi-core = { path = "../tsunagi-core", version = "0.1.0", features = ["tun-device", "dns-publish"] }
|
||||
tsunagi-wireguard = { path = "../tsunagi-wireguard", version = "0.1.0" }
|
||||
tsunagi-awg = { path = "../tsunagi-awg", version = "0.1.0" }
|
||||
tsunagi-wg-quic = { path = "../tsunagi-wg-quic", version = "0.1.0" }
|
||||
tsunagi-tcp-tls = { path = "../tsunagi-tcp-tls", version = "0.1.0" }
|
||||
iroh.workspace = true
|
||||
@@ -26,6 +27,8 @@ clap = { version = "4.5", features = ["derive", "env"] }
|
||||
# colour is never written where it would show up as rubbish.
|
||||
anstream = "1.0"
|
||||
anstyle = "1.0"
|
||||
serde_json = "1.0"
|
||||
tempfile.workspace = true
|
||||
# Local interface addresses, for the diagnostics in `status`.
|
||||
netwatch = "0.19.3"
|
||||
|
||||
|
||||
+821
-135
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,506 @@
|
||||
//! Named virtual service administration; every mutation submits one complete draft.
|
||||
use super::{PathArgs, control_socket, resolve_network_index};
|
||||
use clap::{Args, Subcommand};
|
||||
use std::path::PathBuf;
|
||||
use tsunagi::ipc::{self, NetworkReport};
|
||||
use tsunagi::vip::{Backend, Listener, Protocol, VipConfig, VipStatus};
|
||||
type Result<T> = std::result::Result<T, Box<dyn std::error::Error>>;
|
||||
#[derive(Debug, Args)]
|
||||
#[command(
|
||||
after_help = "Examples:\n tsng vip create lab k8s\n tsng vip port add lab k8s tcp/443 --backend cp1:6443 --backend cp2:7443\n tsng vip backend add lab k8s tcp/443 cp3:6443\n tsng vip status lab k8s --watch\n tsng vip edit lab k8s\n\nNames, full ids and unique id prefixes are accepted. Changes are shared with the network.\nTCP port checks default to on; UDP uses data-path probes and explicit ICMP errors."
|
||||
)]
|
||||
pub(crate) struct VipArgs {
|
||||
#[command(flatten)]
|
||||
paths: PathArgs,
|
||||
/// Agent control socket; defaults to the running service.
|
||||
#[arg(long, global = true, env = "TSUNAGI_CONTROL_SOCKET")]
|
||||
control_socket: Option<PathBuf>,
|
||||
/// Machine-readable configuration and local observations.
|
||||
#[arg(long, global = true)]
|
||||
json: bool,
|
||||
#[command(subcommand)]
|
||||
action: Action,
|
||||
}
|
||||
#[derive(Debug, Args)]
|
||||
struct Target {
|
||||
/// Network exact name, id or unique id prefix.
|
||||
network: String,
|
||||
/// VIP exact name, id or unique id prefix.
|
||||
vip: String,
|
||||
}
|
||||
#[derive(Debug, Args)]
|
||||
struct PortTarget {
|
||||
#[command(flatten)]
|
||||
target: Target,
|
||||
/// Listener as tcp/443 or udp/53.
|
||||
listener: String,
|
||||
}
|
||||
#[derive(Debug, Subcommand)]
|
||||
enum Action {
|
||||
/// List services in one network, or every configured network.
|
||||
List { network: Option<String> },
|
||||
/// Create a named VIP with an automatically reserved internal address.
|
||||
Create { network: String, name: String },
|
||||
/// Show the complete saved configuration and local status.
|
||||
Show(Target),
|
||||
/// Show local health, routing choices and cumulative traffic counters.
|
||||
Status {
|
||||
#[command(flatten)]
|
||||
target: Target,
|
||||
/// Refresh once a second until interrupted.
|
||||
#[arg(long)]
|
||||
watch: bool,
|
||||
},
|
||||
/// Edit the entire configuration in VISUAL/EDITOR and save it atomically.
|
||||
Edit(Target),
|
||||
/// Remove a VIP; existing flows drain until completion or idle expiry.
|
||||
Remove(Target),
|
||||
/// Add, configure or remove listeners independently of the VIP.
|
||||
Port {
|
||||
#[command(subcommand)]
|
||||
action: PortAction,
|
||||
},
|
||||
/// Change a listener's destination nodes and application ports.
|
||||
Backend {
|
||||
#[command(subcommand)]
|
||||
action: BackendAction,
|
||||
},
|
||||
}
|
||||
#[derive(Debug, Subcommand)]
|
||||
enum PortAction {
|
||||
/// Add a listener, optionally with its complete initial backend set.
|
||||
Add {
|
||||
#[command(flatten)]
|
||||
target: PortTarget,
|
||||
/// Node name/id followed by :port. Repeat for multiple destinations.
|
||||
#[arg(long)]
|
||||
backend: Vec<String>,
|
||||
},
|
||||
/// Change check settings and flow idle expiry.
|
||||
Set {
|
||||
#[command(flatten)]
|
||||
target: PortTarget,
|
||||
/// Active TCP connect checks: on or off.
|
||||
#[arg(long,value_parser=["on","off"])]
|
||||
port_check: Option<String>,
|
||||
/// Milliseconds between active checks (250..3600000).
|
||||
#[arg(long)]
|
||||
check_interval_ms: Option<u32>,
|
||||
/// TCP check deadline in milliseconds (100..60000).
|
||||
#[arg(long)]
|
||||
check_timeout_ms: Option<u32>,
|
||||
/// Consecutive failures before exclusion.
|
||||
#[arg(long)]
|
||||
failures: Option<u8>,
|
||||
/// Consecutive successes before recovery.
|
||||
#[arg(long)]
|
||||
successes: Option<u8>,
|
||||
/// Flow idle expiry in seconds (1..604800).
|
||||
#[arg(long)]
|
||||
idle_timeout: Option<u32>,
|
||||
},
|
||||
/// Remove a listener from new-flow selection; existing flows drain.
|
||||
Remove(PortTarget),
|
||||
}
|
||||
#[derive(Debug, Subcommand)]
|
||||
enum BackendAction {
|
||||
/// Add a destination node and port to a listener.
|
||||
Add {
|
||||
#[command(flatten)]
|
||||
target: PortTarget,
|
||||
backend: String,
|
||||
},
|
||||
/// Remove a destination from new-flow selection.
|
||||
Remove {
|
||||
#[command(flatten)]
|
||||
target: PortTarget,
|
||||
backend: String,
|
||||
},
|
||||
}
|
||||
impl Action {
|
||||
fn network(&self) -> Option<&str> {
|
||||
match self {
|
||||
Self::List { network } => network.as_deref(),
|
||||
Self::Create { network, .. } => Some(network),
|
||||
Self::Show(t) | Self::Edit(t) | Self::Remove(t) | Self::Status { target: t, .. } => {
|
||||
Some(&t.network)
|
||||
}
|
||||
Self::Port { action } => Some(
|
||||
&match action {
|
||||
PortAction::Add { target, .. }
|
||||
| PortAction::Set { target, .. }
|
||||
| PortAction::Remove(target) => target,
|
||||
}
|
||||
.target
|
||||
.network,
|
||||
),
|
||||
Self::Backend { action } => Some(
|
||||
&match action {
|
||||
BackendAction::Add { target, .. } | BackendAction::Remove { target, .. } => {
|
||||
target
|
||||
}
|
||||
}
|
||||
.target
|
||||
.network,
|
||||
),
|
||||
}
|
||||
}
|
||||
}
|
||||
fn parse_listener(s: &str) -> Result<(Protocol, u16)> {
|
||||
let (p, n) = s
|
||||
.split_once('/')
|
||||
.ok_or("listener must be tcp/PORT or udp/PORT")?;
|
||||
let port = n.parse::<u16>()?;
|
||||
if port == 0 {
|
||||
return Err("port must not be zero".into());
|
||||
}
|
||||
Ok((p.parse()?, port))
|
||||
}
|
||||
fn backend(n: &NetworkReport, s: &str) -> Result<Backend> {
|
||||
let (node, port) = s.rsplit_once(':').ok_or("backend must be NODE:PORT")?;
|
||||
let peer = match ipc::resolve_node(n, node) {
|
||||
Ok(member) => member.endpoint_id.clone(),
|
||||
Err(error) => {
|
||||
let matches = n
|
||||
.members
|
||||
.iter()
|
||||
.any(|m| m.hostname.as_deref() == Some(node) || m.endpoint_id.starts_with(node));
|
||||
if matches {
|
||||
return Err(error.into());
|
||||
}
|
||||
node.parse::<iroh::EndpointId>()
|
||||
.map_err(|_| error)?
|
||||
.to_string()
|
||||
}
|
||||
};
|
||||
let port = port.parse()?;
|
||||
if port == 0 {
|
||||
return Err("backend port must not be zero".into());
|
||||
}
|
||||
Ok(Backend { peer, port })
|
||||
}
|
||||
fn selected(n: &NetworkReport, t: &Target) -> Result<VipStatus> {
|
||||
Ok(ipc::resolve_vip(n, &t.vip)?.clone())
|
||||
}
|
||||
fn listener<'a>(c: &'a mut VipConfig, target: &PortTarget) -> Result<&'a mut Listener> {
|
||||
let (p, n) = parse_listener(&target.listener)?;
|
||||
c.listeners
|
||||
.iter_mut()
|
||||
.find(|l| l.protocol == p && l.port == n)
|
||||
.ok_or_else(|| format!("no listener {} on {}", target.listener, c.name).into())
|
||||
}
|
||||
fn name_of<'a>(n: &'a NetworkReport, peer: &'a str) -> &'a str {
|
||||
n.members
|
||||
.iter()
|
||||
.find(|m| m.endpoint_id == peer)
|
||||
.and_then(|m| m.hostname.as_deref())
|
||||
.unwrap_or(peer)
|
||||
}
|
||||
fn show(n: &NetworkReport, v: &VipStatus, json: bool) -> Result<()> {
|
||||
if json {
|
||||
println!("{}", serde_json::to_string(v)?);
|
||||
return Ok(());
|
||||
}
|
||||
println!(
|
||||
"{}.{} {}",
|
||||
v.config.name,
|
||||
n.name,
|
||||
v.error.as_deref().unwrap_or("configured")
|
||||
);
|
||||
for l in &v.listeners {
|
||||
let choice = l
|
||||
.selected
|
||||
.as_ref()
|
||||
.map(|b| format!("{}:{}", name_of(n, &b.peer), b.port))
|
||||
.unwrap_or_else(|| "none".into());
|
||||
println!(
|
||||
" {}/{} next: {} flows: {} tx/rx: {}/{} bytes drops: {} UDP failovers: {}",
|
||||
l.protocol,
|
||||
l.port,
|
||||
choice,
|
||||
l.backends.iter().map(|b| b.active_flows).sum::<u32>(),
|
||||
l.counters.tx_bytes,
|
||||
l.counters.rx_bytes,
|
||||
l.counters.dropped,
|
||||
l.counters.failovers
|
||||
);
|
||||
for b in &l.backends {
|
||||
let rtt = b
|
||||
.rtt_us
|
||||
.map(|n| format!("{:.2} ms", n as f64 / 1000.))
|
||||
.unwrap_or_else(|| "unknown".into());
|
||||
println!(
|
||||
" {}:{} {} RTT {} active {}",
|
||||
name_of(n, &b.backend.peer),
|
||||
b.backend.port,
|
||||
b.state,
|
||||
rtt,
|
||||
b.active_flows
|
||||
);
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
pub(crate) async fn run(args: VipArgs) -> Result<()> {
|
||||
let custom = args.paths.state_dir.is_some();
|
||||
let paths = args.paths.resolve()?;
|
||||
let socket = control_socket(&paths, args.control_socket.as_ref(), custom);
|
||||
let report = ipc::request_status(&socket).await?;
|
||||
if let Action::List { network: None } = &args.action {
|
||||
for n in &report.networks {
|
||||
for v in &n.vips {
|
||||
show(n, v, args.json)?;
|
||||
}
|
||||
}
|
||||
return Ok(());
|
||||
}
|
||||
let selector = args.action.network().ok_or("network is required")?;
|
||||
let names: Vec<_> = report
|
||||
.networks
|
||||
.iter()
|
||||
.map(|n| (n.network_id.clone(), n.name.clone()))
|
||||
.collect();
|
||||
let network = &report.networks[resolve_network_index(&names, selector)?];
|
||||
if let Action::Status {
|
||||
target,
|
||||
watch: true,
|
||||
} = &args.action
|
||||
{
|
||||
let mut previous: Option<(std::time::Instant, u64, u64)> = None;
|
||||
loop {
|
||||
let report = ipc::request_status(&socket).await?;
|
||||
let n = report
|
||||
.networks
|
||||
.iter()
|
||||
.find(|n| n.network_id == network.network_id)
|
||||
.ok_or("network was removed")?;
|
||||
let vip = selected(n, target)?;
|
||||
let (tx, rx) = vip.listeners.iter().fold((0u64, 0u64), |a, l| {
|
||||
(
|
||||
a.0.saturating_add(l.counters.tx_bytes),
|
||||
a.1.saturating_add(l.counters.rx_bytes),
|
||||
)
|
||||
});
|
||||
let now = std::time::Instant::now();
|
||||
if let Some((at, old_tx, old_rx)) = previous
|
||||
&& !args.json
|
||||
{
|
||||
let secs = now.duration_since(at).as_secs_f64();
|
||||
if secs > 0. {
|
||||
println!(
|
||||
"TX {:.1} KiB/s · RX {:.1} KiB/s",
|
||||
tx.saturating_sub(old_tx) as f64 / secs / 1024.,
|
||||
rx.saturating_sub(old_rx) as f64 / secs / 1024.
|
||||
);
|
||||
}
|
||||
}
|
||||
previous = Some((now, tx, rx));
|
||||
show(n, &vip, args.json)?;
|
||||
tokio::select! {_=tokio::signal::ctrl_c()=>break,_=tokio::time::sleep(std::time::Duration::from_secs(1))=>{}}
|
||||
}
|
||||
return Ok(());
|
||||
}
|
||||
let mut edited_file = None;
|
||||
let (mut config, expected, delete) = match &args.action {
|
||||
Action::List { .. } => {
|
||||
for v in &network.vips {
|
||||
show(network, v, args.json)?;
|
||||
}
|
||||
return Ok(());
|
||||
}
|
||||
Action::Show(t) | Action::Status { target: t, .. } => {
|
||||
let v = selected(network, t)?;
|
||||
show(network, &v, args.json)?;
|
||||
if matches!(args.action, Action::Show(_)) && !args.json {
|
||||
println!("{}", serde_json::to_string_pretty(&v.config)?);
|
||||
}
|
||||
return Ok(());
|
||||
}
|
||||
Action::Create { name, .. } => (VipConfig::new(name), None, false),
|
||||
Action::Remove(t) => {
|
||||
let v = selected(network, t)?;
|
||||
(v.config, Some(v.revision), true)
|
||||
}
|
||||
Action::Edit(t) => {
|
||||
let v = selected(network, t)?;
|
||||
let mut file = tempfile::Builder::new()
|
||||
.prefix("tsunagi-vip-")
|
||||
.suffix(".json")
|
||||
.tempfile()?;
|
||||
use std::io::Write;
|
||||
file.write_all(serde_json::to_string_pretty(&v.config)?.as_bytes())?;
|
||||
file.flush()?;
|
||||
let editor = std::env::var("VISUAL")
|
||||
.or_else(|_| std::env::var("EDITOR"))
|
||||
.unwrap_or_else(|_| "vi".into());
|
||||
let mut words = editor.split_whitespace();
|
||||
let program = words.next().ok_or("empty EDITOR")?;
|
||||
let status = std::process::Command::new(program)
|
||||
.args(words)
|
||||
.arg(file.path())
|
||||
.status()?;
|
||||
if !status.success() {
|
||||
return Err("editor failed; VIP unchanged".into());
|
||||
}
|
||||
let bytes = std::fs::read(file.path())?;
|
||||
let cfg: VipConfig = match serde_json::from_slice(&bytes) {
|
||||
Ok(c) => c,
|
||||
Err(e) => {
|
||||
let (_, path) = file.keep()?;
|
||||
return Err(
|
||||
format!("invalid draft: {e}; retained at {}", path.display()).into(),
|
||||
);
|
||||
}
|
||||
};
|
||||
if cfg.id != v.config.id {
|
||||
return Err("VIP id cannot be edited".into());
|
||||
}
|
||||
edited_file = Some(file);
|
||||
(cfg, Some(v.revision), false)
|
||||
}
|
||||
Action::Port { action } => {
|
||||
let target = match action {
|
||||
PortAction::Add { target, .. }
|
||||
| PortAction::Set { target, .. }
|
||||
| PortAction::Remove(target) => target,
|
||||
};
|
||||
let v = selected(network, &target.target)?;
|
||||
let mut cfg = v.config;
|
||||
match action {
|
||||
PortAction::Add {
|
||||
target,
|
||||
backend: destinations,
|
||||
} => {
|
||||
let (p, port) = parse_listener(&target.listener)?;
|
||||
let mut l = Listener::new(p, port);
|
||||
l.backends = destinations
|
||||
.iter()
|
||||
.map(|b| backend(network, b))
|
||||
.collect::<Result<_>>()?;
|
||||
cfg.listeners.push(l);
|
||||
}
|
||||
PortAction::Remove(target) => {
|
||||
let (p, port) = parse_listener(&target.listener)?;
|
||||
listener(&mut cfg, target)?;
|
||||
cfg.listeners.retain(|l| l.protocol != p || l.port != port);
|
||||
}
|
||||
PortAction::Set {
|
||||
target,
|
||||
port_check,
|
||||
check_interval_ms,
|
||||
check_timeout_ms,
|
||||
failures,
|
||||
successes,
|
||||
idle_timeout,
|
||||
} => {
|
||||
let l = listener(&mut cfg, target)?;
|
||||
if let Some(v) = port_check {
|
||||
l.port_check = v == "on";
|
||||
}
|
||||
if let Some(v) = check_interval_ms {
|
||||
l.check_interval_ms = *v;
|
||||
}
|
||||
if let Some(v) = check_timeout_ms {
|
||||
l.check_timeout_ms = *v;
|
||||
}
|
||||
if let Some(v) = failures {
|
||||
l.failure_threshold = *v;
|
||||
}
|
||||
if let Some(v) = successes {
|
||||
l.recovery_threshold = *v;
|
||||
}
|
||||
if let Some(v) = idle_timeout {
|
||||
l.idle_timeout_secs = *v;
|
||||
}
|
||||
}
|
||||
}
|
||||
(cfg, Some(v.revision), false)
|
||||
}
|
||||
Action::Backend { action } => {
|
||||
let (target, destination, remove) = match action {
|
||||
BackendAction::Add { target, backend } => (target, backend, false),
|
||||
BackendAction::Remove { target, backend } => (target, backend, true),
|
||||
};
|
||||
let v = selected(network, &target.target)?;
|
||||
let mut cfg = v.config;
|
||||
let b = backend(network, destination)?;
|
||||
let l = listener(&mut cfg, target)?;
|
||||
if remove {
|
||||
if !l.backends.contains(&b) {
|
||||
return Err("backend is not configured".into());
|
||||
}
|
||||
l.backends.retain(|x| x != &b);
|
||||
} else {
|
||||
l.backends.push(b);
|
||||
}
|
||||
(cfg, Some(v.revision), false)
|
||||
}
|
||||
};
|
||||
// Canonical order makes displayed drafts and shared state deterministic.
|
||||
config.listeners.sort_by_key(|l| (l.protocol, l.port));
|
||||
for l in &mut config.listeners {
|
||||
l.backends.sort();
|
||||
}
|
||||
let id = config.id.clone();
|
||||
let vips = match ipc::save_vip(&socket, &network.network_id, config, expected, delete).await {
|
||||
Ok(vips) => vips,
|
||||
Err(error) => {
|
||||
if let Some(file) = edited_file {
|
||||
let (_, path) = file.keep()?;
|
||||
return Err(format!("{error}; draft retained at {}", path.display()).into());
|
||||
}
|
||||
return Err(error.into());
|
||||
}
|
||||
};
|
||||
if delete {
|
||||
if args.json {
|
||||
println!("{{\"removed\":\"{id}\"}}");
|
||||
} else {
|
||||
println!("VIP removed; existing flows drain.");
|
||||
}
|
||||
} else if let Some(v) = vips.iter().find(|v| v.config.id == id) {
|
||||
show(network, v, args.json)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use clap::Parser;
|
||||
#[test]
|
||||
fn vip_commands_parse_complete_listener_and_settings() {
|
||||
let cli = super::super::Cli::try_parse_from([
|
||||
"tsng",
|
||||
"vip",
|
||||
"port",
|
||||
"add",
|
||||
"lab",
|
||||
"k8s",
|
||||
"tcp/443",
|
||||
"--backend",
|
||||
"cp1:6443",
|
||||
"--backend",
|
||||
"cp2:7443",
|
||||
]);
|
||||
assert!(cli.is_ok(), "{cli:?}");
|
||||
assert!(
|
||||
super::super::Cli::try_parse_from([
|
||||
"tsng",
|
||||
"vip",
|
||||
"port",
|
||||
"set",
|
||||
"lab",
|
||||
"k8s",
|
||||
"tcp/443",
|
||||
"--port-check",
|
||||
"off",
|
||||
"--failures",
|
||||
"3"
|
||||
])
|
||||
.is_ok()
|
||||
);
|
||||
assert!(parse_listener("tcp/0").is_err());
|
||||
assert!(parse_listener("sctp/53").is_err());
|
||||
}
|
||||
}
|
||||
@@ -345,3 +345,95 @@ fn opposing_dns_flags_are_rejected() {
|
||||
assert!(!result.status.success());
|
||||
assert!(String::from_utf8_lossy(&result.stderr).contains("cannot be used with"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn vip_cli_creates_multiple_ports_updates_dns_and_rejects_partial_invalid_changes() {
|
||||
let agent = start("vip-test", 15367);
|
||||
let created = agent.run(&["vip", "create", "vip-test", "api", "--json"]);
|
||||
assert!(
|
||||
created.status.success(),
|
||||
"{}",
|
||||
String::from_utf8_lossy(&created.stderr)
|
||||
);
|
||||
let vip: tsunagi::vip::VipStatus = serde_json::from_slice(&created.stdout).unwrap();
|
||||
assert!(vip.config.listeners.is_empty());
|
||||
let reply = wait_for_answer("127.0.0.1:15367".parse().unwrap(), "api.vip-test");
|
||||
assert!(has_answer(&reply));
|
||||
let socket = tsunagi::ipc::control_socket_path(&agent.dir.path().join("state"));
|
||||
let runtime = tokio::runtime::Builder::new_current_thread()
|
||||
.enable_all()
|
||||
.build()
|
||||
.unwrap();
|
||||
let status = runtime
|
||||
.block_on(tsunagi::ipc::request_status(socket))
|
||||
.unwrap();
|
||||
let backend = format!("{}:6443", status.endpoint_id);
|
||||
let added = agent.run(&[
|
||||
"vip",
|
||||
"port",
|
||||
"add",
|
||||
"vip-test",
|
||||
"api",
|
||||
"tcp/443",
|
||||
"--backend",
|
||||
&backend,
|
||||
]);
|
||||
assert!(
|
||||
added.status.success(),
|
||||
"{}",
|
||||
String::from_utf8_lossy(&added.stderr)
|
||||
);
|
||||
assert!(
|
||||
agent
|
||||
.run(&["vip", "port", "add", "vip-test", "api", "udp/53"])
|
||||
.status
|
||||
.success()
|
||||
);
|
||||
assert!(
|
||||
!agent
|
||||
.run(&["vip", "port", "add", "vip-test", "api", "tcp/443"])
|
||||
.status
|
||||
.success()
|
||||
);
|
||||
assert!(
|
||||
agent
|
||||
.run(&[
|
||||
"vip",
|
||||
"port",
|
||||
"set",
|
||||
"vip-test",
|
||||
"api",
|
||||
"tcp/443",
|
||||
"--port-check",
|
||||
"off"
|
||||
])
|
||||
.status
|
||||
.success()
|
||||
);
|
||||
let shown = agent.run(&["vip", "show", "vip-test", "api", "--json"]);
|
||||
assert!(shown.status.success());
|
||||
let shown: tsunagi::vip::VipStatus = serde_json::from_slice(&shown.stdout).unwrap();
|
||||
assert_eq!(shown.config.listeners.len(), 2);
|
||||
assert!(
|
||||
!shown
|
||||
.config
|
||||
.listeners
|
||||
.iter()
|
||||
.find(|l| l.protocol == tsunagi::vip::Protocol::Tcp)
|
||||
.unwrap()
|
||||
.port_check
|
||||
);
|
||||
assert_eq!(shown.config.listeners[0].backends[0].port, 6443);
|
||||
let removed = agent.run(&["vip", "remove", "vip-test", "api"]);
|
||||
assert!(removed.status.success());
|
||||
let deadline = Instant::now() + Duration::from_secs(10);
|
||||
loop {
|
||||
if let Some(reply) = query("127.0.0.1:15367".parse().unwrap(), "api.vip-test", TYPE::A)
|
||||
&& !has_answer(&reply)
|
||||
{
|
||||
break;
|
||||
}
|
||||
assert!(Instant::now() < deadline);
|
||||
std::thread::sleep(Duration::from_millis(25));
|
||||
}
|
||||
}
|
||||
@@ -6,7 +6,7 @@
|
||||
|
||||
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
|
||||
|
||||
use std::time::Duration;
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use tempfile::TempDir;
|
||||
|
||||
@@ -50,8 +50,26 @@ fn read_networks(dir: &TempDir) -> Vec<tsunagi::storage::StoredNetwork> {
|
||||
/// An agent with no network at all, the way a daemon is started before
|
||||
/// anything has been decided.
|
||||
fn start_bare(port: u16) -> Running {
|
||||
start_bare_range(port, None)
|
||||
}
|
||||
|
||||
fn start_bare_range(port: u16, range: Option<&str>) -> Running {
|
||||
start_bare_config(port, range, None)
|
||||
}
|
||||
|
||||
fn start_bare_config(port: u16, range: Option<&str>, protocols: Option<&str>) -> Running {
|
||||
start_bare_peers(port, range, protocols, &[])
|
||||
}
|
||||
|
||||
fn start_bare_peers(
|
||||
port: u16,
|
||||
range: Option<&str>,
|
||||
protocols: Option<&str>,
|
||||
peers: &[String],
|
||||
) -> Running {
|
||||
let dir = TempDir::new().unwrap();
|
||||
let child = std::process::Command::new(env!("CARGO_BIN_EXE_tsng"))
|
||||
let mut command = std::process::Command::new(env!("CARGO_BIN_EXE_tsng"));
|
||||
command
|
||||
.arg("up")
|
||||
.arg("--state-dir")
|
||||
.arg(dir.path().join("state"))
|
||||
@@ -62,11 +80,37 @@ fn start_bare(port: u16) -> Running {
|
||||
.arg(format!("127.0.0.1:{port}"))
|
||||
.args(["--log", "error", "--status-interval", "0"])
|
||||
.stdout(std::process::Stdio::null())
|
||||
.stderr(std::process::Stdio::null())
|
||||
.spawn()
|
||||
.expect("the agent binary starts");
|
||||
std::thread::sleep(Duration::from_secs(2));
|
||||
Running { child, dir }
|
||||
.stderr(std::process::Stdio::null());
|
||||
if let Some(range) = range {
|
||||
command.args(["--ipv4-range", range]);
|
||||
}
|
||||
if let Some(protocols) = protocols {
|
||||
command.args(["--protocol", protocols]);
|
||||
}
|
||||
for peer in peers {
|
||||
command.args(["--peer", peer]);
|
||||
}
|
||||
let child = command.spawn().expect("the agent binary starts");
|
||||
let mut agent = Running { child, dir };
|
||||
let runtime = tokio::runtime::Builder::new_current_thread()
|
||||
.enable_all()
|
||||
.build()
|
||||
.unwrap();
|
||||
let socket = tsunagi::ipc::control_socket_path(&agent.dir.path().join("state"));
|
||||
runtime.block_on(async {
|
||||
tokio::time::timeout(Duration::from_secs(30), async {
|
||||
while !tsunagi::ipc::is_serving(&socket).await {
|
||||
assert!(
|
||||
agent.child.try_wait().unwrap().is_none(),
|
||||
"agent exited before becoming ready"
|
||||
);
|
||||
tokio::time::sleep(Duration::from_millis(25)).await;
|
||||
}
|
||||
})
|
||||
.await
|
||||
.expect("agent becomes ready");
|
||||
});
|
||||
agent
|
||||
}
|
||||
|
||||
/// An agent already in one network, the way most of these start.
|
||||
@@ -87,6 +131,254 @@ fn start(network: &str, port: u16) -> Running {
|
||||
agent
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cli_forgets_only_the_selected_offline_claim_over_real_control() {
|
||||
let cleaner = start("forget-cli", 0);
|
||||
let runtime = tokio::runtime::Builder::new_current_thread()
|
||||
.enable_all()
|
||||
.build()
|
||||
.unwrap();
|
||||
let socket = tsunagi::ipc::control_socket_path(&cleaner.dir.path().join("state"));
|
||||
let read = || {
|
||||
runtime
|
||||
.block_on(tsunagi::ipc::request_status(&socket))
|
||||
.unwrap()
|
||||
};
|
||||
let before = read();
|
||||
let bootstrap = format!("{}@{}", before.endpoint_id, before.bound_sockets[0]);
|
||||
let target = start_bare_peers(0, None, Some("none"), &[bootstrap]);
|
||||
assert!(
|
||||
target
|
||||
.run(&["id", "hostname", "old-cli-node"])
|
||||
.status
|
||||
.success()
|
||||
);
|
||||
assert!(
|
||||
target
|
||||
.run(&[
|
||||
"join",
|
||||
"--network",
|
||||
"forget-cli",
|
||||
"--secret",
|
||||
"a-secret-for-the-cli-test"
|
||||
])
|
||||
.status
|
||||
.success()
|
||||
);
|
||||
runtime.block_on(async {
|
||||
tokio::time::timeout(Duration::from_secs(30), async {
|
||||
loop {
|
||||
let status = tsunagi::ipc::request_status(&socket).await.unwrap();
|
||||
if status.networks[0]
|
||||
.members
|
||||
.iter()
|
||||
.any(|m| m.hostname.as_deref() == Some("old-cli-node") && m.online)
|
||||
{
|
||||
break;
|
||||
}
|
||||
tokio::time::sleep(Duration::from_millis(25)).await;
|
||||
}
|
||||
})
|
||||
.await
|
||||
.unwrap();
|
||||
});
|
||||
assert!(
|
||||
!cleaner
|
||||
.run(&["network", "forget-node", "forget-cli", "old-cli-node"])
|
||||
.status
|
||||
.success()
|
||||
);
|
||||
assert!(
|
||||
!cleaner
|
||||
.run(&["network", "forget-node", "forget-cli", &before.endpoint_id])
|
||||
.status
|
||||
.success()
|
||||
);
|
||||
assert!(
|
||||
target
|
||||
.run(&["network", "stop", "forget-cli"])
|
||||
.status
|
||||
.success()
|
||||
);
|
||||
runtime.block_on(async {
|
||||
tokio::time::timeout(Duration::from_secs(30), async {
|
||||
loop {
|
||||
let status = tsunagi::ipc::request_status(&socket).await.unwrap();
|
||||
if status.networks[0]
|
||||
.members
|
||||
.iter()
|
||||
.any(|m| m.hostname.as_deref() == Some("old-cli-node") && !m.online)
|
||||
{
|
||||
break;
|
||||
}
|
||||
tokio::time::sleep(Duration::from_millis(25)).await;
|
||||
}
|
||||
})
|
||||
.await
|
||||
.unwrap();
|
||||
});
|
||||
let forgotten = cleaner.run(&["network", "forget-node", "forget-cli", "old-cli-node"]);
|
||||
assert!(
|
||||
forgotten.status.success(),
|
||||
"{}",
|
||||
String::from_utf8_lossy(&forgotten.stderr)
|
||||
);
|
||||
assert!(String::from_utf8_lossy(&forgotten.stdout).contains("newer signed claim can return"));
|
||||
assert!(
|
||||
!read().networks[0]
|
||||
.members
|
||||
.iter()
|
||||
.any(|m| m.hostname.as_deref() == Some("old-cli-node"))
|
||||
);
|
||||
assert!(
|
||||
!cleaner
|
||||
.run(&["network", "forget-node", "forget-cli", "old-cli-node"])
|
||||
.status
|
||||
.success()
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn protocols_absent_from_startup_can_be_enabled_and_changed_offline() {
|
||||
let mut agent = start_bare_config(0, None, Some("none"));
|
||||
assert!(
|
||||
agent
|
||||
.run(&[
|
||||
"join",
|
||||
"--network",
|
||||
"from-control-only",
|
||||
"--secret",
|
||||
"a-control-only-test-secret"
|
||||
])
|
||||
.status
|
||||
.success()
|
||||
);
|
||||
let initial = agent.run(&["protocols", "from-control-only"]);
|
||||
assert!(initial.status.success());
|
||||
assert!(String::from_utf8_lossy(&initial.stdout).contains("protocols none"));
|
||||
let enabled = agent.run(&["protocols", "from-control-only", "awg"]);
|
||||
assert!(
|
||||
enabled.status.success(),
|
||||
"{}",
|
||||
String::from_utf8_lossy(&enabled.stderr)
|
||||
);
|
||||
assert_eq!(
|
||||
read_networks(&agent.dir)[0].protocols,
|
||||
Some(vec!["awg".into()])
|
||||
);
|
||||
agent.child.kill().unwrap();
|
||||
agent.child.wait().unwrap();
|
||||
let saved = agent.run(&["protocols", "from-control-only", "wg-quic,awg"]);
|
||||
assert!(
|
||||
saved.status.success(),
|
||||
"{}",
|
||||
String::from_utf8_lossy(&saved.stderr)
|
||||
);
|
||||
assert!(String::from_utf8_lossy(&saved.stdout).contains("agent stopped"));
|
||||
assert_eq!(
|
||||
read_networks(&agent.dir)[0].protocols,
|
||||
Some(vec!["wg-quic".into(), "awg".into()])
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn protocols_are_changed_live_per_network_and_saved_without_changing_the_service() {
|
||||
let agent = start("protocol-choice", 0);
|
||||
assert!(
|
||||
agent
|
||||
.run(&[
|
||||
"join",
|
||||
"--network",
|
||||
"other-choice",
|
||||
"--secret",
|
||||
"other-secret-for-cli-test"
|
||||
])
|
||||
.status
|
||||
.success()
|
||||
);
|
||||
let before = read_networks(&agent.dir);
|
||||
let other_id = before
|
||||
.iter()
|
||||
.find(|n| n.name.as_str() == "other-choice")
|
||||
.unwrap()
|
||||
.network_id;
|
||||
let changed = agent.run(&["protocols", "protocol-choice", "awg,wg-quic"]);
|
||||
assert!(
|
||||
changed.status.success(),
|
||||
"{}",
|
||||
String::from_utf8_lossy(&changed.stderr)
|
||||
);
|
||||
let shown = agent.run(&["protocols", "protocol-choice"]);
|
||||
assert!(shown.status.success());
|
||||
assert!(String::from_utf8_lossy(&shown.stdout).contains("awg,wg-quic"));
|
||||
let saved = read_networks(&agent.dir);
|
||||
assert_eq!(
|
||||
saved
|
||||
.iter()
|
||||
.find(|n| n.name.as_str() == "protocol-choice")
|
||||
.unwrap()
|
||||
.protocols,
|
||||
Some(vec!["awg".into(), "wg-quic".into()])
|
||||
);
|
||||
assert_eq!(
|
||||
saved
|
||||
.iter()
|
||||
.find(|n| n.network_id == other_id)
|
||||
.unwrap()
|
||||
.protocols,
|
||||
before
|
||||
.iter()
|
||||
.find(|n| n.network_id == other_id)
|
||||
.unwrap()
|
||||
.protocols
|
||||
);
|
||||
for invalid in ["unknown", "awg,awg", "none,awg"] {
|
||||
assert!(
|
||||
!agent
|
||||
.run(&["protocols", "protocol-choice", invalid])
|
||||
.status
|
||||
.success()
|
||||
);
|
||||
}
|
||||
assert!(!agent.run(&["protocols", "missing", "awg"]).status.success());
|
||||
let disabled = agent.run(&["protocols", "protocol-choice", "none"]);
|
||||
assert!(
|
||||
disabled.status.success(),
|
||||
"{}",
|
||||
String::from_utf8_lossy(&disabled.stderr)
|
||||
);
|
||||
assert_eq!(
|
||||
read_networks(&agent.dir)
|
||||
.iter()
|
||||
.find(|n| n.name.as_str() == "protocol-choice")
|
||||
.unwrap()
|
||||
.protocols,
|
||||
Some(Vec::new())
|
||||
);
|
||||
assert!(
|
||||
agent
|
||||
.run(&["network", "stop", "protocol-choice"])
|
||||
.status
|
||||
.success()
|
||||
);
|
||||
assert!(
|
||||
agent
|
||||
.run(&["protocols", "protocol-choice", "wg-quic"])
|
||||
.status
|
||||
.success()
|
||||
);
|
||||
assert!(
|
||||
agent
|
||||
.run(&["network", "start", "protocol-choice"])
|
||||
.status
|
||||
.success()
|
||||
);
|
||||
assert!(
|
||||
String::from_utf8_lossy(&agent.run(&["protocols", "protocol-choice"]).stdout)
|
||||
.contains("wg-quic")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn joining_with_no_secret_makes_one_and_prints_what_to_send() {
|
||||
// The hurry case: a network with somebody, for as long as it is needed,
|
||||
@@ -461,3 +753,139 @@ fn an_exit_node_is_offered_per_network_only_when_asked_and_nobody_offers_by_defa
|
||||
String::from_utf8_lossy(&unknown.stderr)
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn overlapping_networks_are_reported_and_cli_start_switches_then_resumes() {
|
||||
let agent = start_bare_range(0, Some("10.90.0.0/16"));
|
||||
for name in ["resident", "games"] {
|
||||
let joined = agent.run(&["join", "-n", name, "-s", "a-secret-for-the-conflict-test"]);
|
||||
assert!(
|
||||
joined.status.success(),
|
||||
"{}",
|
||||
String::from_utf8_lossy(&joined.stderr)
|
||||
);
|
||||
if name == "games" {
|
||||
let text = String::from_utf8_lossy(&joined.stdout);
|
||||
assert!(
|
||||
text.contains("stopped because its range overlaps"),
|
||||
"{text}"
|
||||
);
|
||||
assert!(!text.contains("it is running"), "{text}");
|
||||
}
|
||||
}
|
||||
let runtime = tokio::runtime::Builder::new_current_thread()
|
||||
.enable_all()
|
||||
.build()
|
||||
.unwrap();
|
||||
let socket = tsunagi::ipc::control_socket_path(&agent.dir.path().join("state"));
|
||||
let snapshot = || {
|
||||
runtime
|
||||
.block_on(tsunagi::ipc::request_status(&socket))
|
||||
.unwrap()
|
||||
};
|
||||
let before = snapshot();
|
||||
let resident = before
|
||||
.networks
|
||||
.iter()
|
||||
.find(|n| n.name == "resident")
|
||||
.unwrap();
|
||||
let games = before.networks.iter().find(|n| n.name == "games").unwrap();
|
||||
assert!(resident.active);
|
||||
assert!(!games.active);
|
||||
assert_eq!(games.range_conflicts[0].network_id, resident.network_id);
|
||||
assert_eq!(games.range_conflicts[0].name, "resident");
|
||||
for args in [vec!["status"], vec!["network"]] {
|
||||
let result = agent.run(&args);
|
||||
let text = String::from_utf8_lossy(&result.stdout);
|
||||
assert!(text.contains("range conflict"), "{text}");
|
||||
assert!(text.contains(&resident.network_id), "{text}");
|
||||
assert!(text.contains("tsng network start"), "{text}");
|
||||
}
|
||||
let switched = agent.run(&["network", "start", "games"]);
|
||||
assert!(
|
||||
switched.status.success(),
|
||||
"{}",
|
||||
String::from_utf8_lossy(&switched.stderr)
|
||||
);
|
||||
let text = String::from_utf8_lossy(&switched.stdout);
|
||||
assert!(
|
||||
text.contains("started `games`") && text.contains("suspended `resident`"),
|
||||
"{text}"
|
||||
);
|
||||
let after = snapshot();
|
||||
assert!(
|
||||
after
|
||||
.networks
|
||||
.iter()
|
||||
.find(|n| n.name == "games")
|
||||
.unwrap()
|
||||
.active
|
||||
);
|
||||
assert!(
|
||||
!after
|
||||
.networks
|
||||
.iter()
|
||||
.find(|n| n.name == "resident")
|
||||
.unwrap()
|
||||
.active
|
||||
);
|
||||
assert!(agent.run(&["network", "stop", "games"]).status.success());
|
||||
assert!(
|
||||
snapshot()
|
||||
.networks
|
||||
.iter()
|
||||
.find(|n| n.name == "resident")
|
||||
.unwrap()
|
||||
.active
|
||||
);
|
||||
|
||||
assert!(agent.run(&["network", "start", "games"]).status.success());
|
||||
let stop_suspended = agent.run(&["network", "stop", "resident"]);
|
||||
assert!(stop_suspended.status.success());
|
||||
assert!(String::from_utf8_lossy(&stop_suspended.stdout).contains("stopped `resident`"));
|
||||
assert!(agent.run(&["network", "stop", "games"]).status.success());
|
||||
assert!(
|
||||
snapshot()
|
||||
.networks
|
||||
.iter()
|
||||
.all(|n| !n.active && !n.auto_start)
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cli_defaults_derive_each_networks_range() {
|
||||
let agent = start_bare(0);
|
||||
for name in ["derived-one", "derived-two"] {
|
||||
assert!(
|
||||
agent
|
||||
.run(&["join", "-n", name, "-s", "a-secret-for-the-derived-test"])
|
||||
.status
|
||||
.success()
|
||||
);
|
||||
}
|
||||
let runtime = tokio::runtime::Builder::new_current_thread()
|
||||
.enable_all()
|
||||
.build()
|
||||
.unwrap();
|
||||
let socket = tsunagi::ipc::control_socket_path(&agent.dir.path().join("state"));
|
||||
let deadline = Instant::now() + Duration::from_secs(30);
|
||||
loop {
|
||||
let report = runtime
|
||||
.block_on(tsunagi::ipc::request_status(&socket))
|
||||
.unwrap();
|
||||
if report.networks.iter().all(|n| n.range.is_some()) {
|
||||
assert_eq!(report.networks.len(), 2);
|
||||
for network in report.networks {
|
||||
let id = network.network_id.parse().unwrap();
|
||||
assert!(network.active);
|
||||
assert_eq!(
|
||||
network.range,
|
||||
Some(tsunagi::state::derived_ipv4_range(id).to_string())
|
||||
);
|
||||
}
|
||||
break;
|
||||
}
|
||||
assert!(Instant::now() < deadline, "networks allocate ranges");
|
||||
std::thread::sleep(Duration::from_millis(25));
|
||||
}
|
||||
}
|
||||
Vendored
+1
-1
@@ -22,7 +22,7 @@ service's socket at `/run/tsunagi/agent.sock`):
|
||||
|
||||
```sh
|
||||
sudo tsng status
|
||||
sudo tsng join <network-name> <tsn1…secret>
|
||||
sudo tsng join -n <network-name> -s <tsn1…secret>
|
||||
```
|
||||
|
||||
## How it runs
|
||||
|
||||
Vendored
+1
-1
@@ -28,7 +28,7 @@ group, so after `sudo usermod -aG tsunagi "$USER"` (and logging in again) no
|
||||
|
||||
```sh
|
||||
sudo tsng status
|
||||
sudo tsng join <network-name> <tsn1…secret>
|
||||
sudo tsng join -n <network-name> -s <tsn1…secret>
|
||||
```
|
||||
|
||||
## As your own user — one capability, no service
|
||||
|
||||
Vendored
+1
-1
@@ -48,7 +48,7 @@ Done. Start the agent with:
|
||||
Control it with sudo (the CLI finds the service's socket automatically):
|
||||
|
||||
sudo tsng status
|
||||
sudo tsng join <network-name> <tsn1…secret>
|
||||
sudo tsng join -n <network-name> -s <tsn1…secret>
|
||||
EOF
|
||||
else
|
||||
cat <<EOF
|
||||
|
||||
Vendored
+1
-1
@@ -15,7 +15,7 @@ socket and do **not** need `sudo`:
|
||||
|
||||
```sh
|
||||
tsng status
|
||||
tsng join <network-name> <tsn1…secret>
|
||||
tsng join -n <network-name> -s <tsn1…secret>
|
||||
```
|
||||
|
||||
The agent uses fixed system paths, so the commands find it regardless of whose
|
||||
|
||||
Vendored
+2
-1
@@ -55,8 +55,9 @@ Source: "{#SourceDir}\tsng.exe"; DestDir: "{app}"; Flags: ignoreversion
|
||||
Source: "{#SourceDir}\tsunagi-tray.exe"; DestDir: "{app}"; Flags: ignoreversion
|
||||
; Wintun is the unmodified, signed library from wintun.net; its licence allows
|
||||
; shipping it with an application and asks for the licence text beside it.
|
||||
; In the archive the text sits at its root, two levels above bin\amd64.
|
||||
Source: "{#WintunDir}\wintun.dll"; DestDir: "{app}"; Flags: ignoreversion
|
||||
Source: "{#WintunDir}\LICENSE.txt"; DestDir: "{app}\licenses"; DestName: "wintun-LICENSE.txt"; Flags: ignoreversion
|
||||
Source: "{#WintunDir}\..\..\LICENSE.txt"; DestDir: "{app}\licenses"; DestName: "wintun-LICENSE.txt"; Flags: ignoreversion
|
||||
|
||||
[Dirs]
|
||||
; Identity and network secrets live here; locked down below.
|
||||
|
||||
+131
@@ -0,0 +1,131 @@
|
||||
# Pairwise Amnezia transport
|
||||
|
||||
The CLI's default carrier preference is `wg,awg,wg-quic,tcp-tls`. To select
|
||||
AWG explicitly:
|
||||
|
||||
```sh
|
||||
tsng up --protocol awg
|
||||
tsng up --protocol wg,awg,wg-quic,tcp-tls
|
||||
```
|
||||
|
||||
The startup list is the initial per-network preference. A saved network choice
|
||||
overrides it and can enable any built-in protocol at runtime:
|
||||
|
||||
```sh
|
||||
tsng protocols home awg,wg-quic
|
||||
tsng protocols home awg
|
||||
```
|
||||
|
||||
Changes are local to this device, persist across restarts, and retire existing
|
||||
data links and end-to-end channels for that network before renegotiation.
|
||||
Authenticated control sessions, membership, addresses and other networks stay
|
||||
live. Removing every protocol (`none` in the CLI, all checkboxes off in the tray)
|
||||
leaves the network control-only. This is different from automatic carrier
|
||||
fallback, which preserves the selected end-to-end channel.
|
||||
|
||||
`awg` is Tsunagi AWG wire version 1. Its packet masking engine is adapted from
|
||||
[RunaRoo/AmneziaWG-RS](https://github.com/RunaRoo/AmneziaWG-RS), with the pinned
|
||||
source and license recorded in
|
||||
[UPSTREAM.md](../crates/tsunagi-awg/UPSTREAM.md). It uses the existing boringtun
|
||||
Noise/WireGuard implementation instead of upstream's Node, global executors
|
||||
and OS interface machinery. Compatibility with ordinary Amnezia servers is
|
||||
not implemented: this carrier connects authenticated Tsunagi members.
|
||||
|
||||
## Negotiation and lifetime
|
||||
|
||||
There is one independent UDP socket and cancellable packet-engine task per
|
||||
`(network, peer)` pair. The smaller endpoint id initiates, following the same
|
||||
rule as other data carriers. Both sides must announce enabled AWG wire version
|
||||
1. An authenticated iroh control session is required throughout; AWG supplies
|
||||
no alternate discovery or membership mechanism.
|
||||
|
||||
Every `open` generates new masking parameters, a header-protection key, an
|
||||
outer WireGuard key pair and a PSK. The initiator sends a private `Offer` with
|
||||
a random attempt id, its public key, candidate socket addresses and the shared
|
||||
profile. The responder validates it, creates its own fresh key pair and socket,
|
||||
and returns an `Answer` naming that attempt. Both send masked handshakes and
|
||||
encrypted delivery probes to the candidates. Only authenticated WireGuard
|
||||
**data**, not a STUN response or an incoming UDP address alone, verifies a path.
|
||||
|
||||
Negotiation payloads are bounded at 4096 bytes (a caller can lower the limit),
|
||||
redacted in Debug, excluded from public message events and erased on drop.
|
||||
They are never capabilities, signed records, cache entries or exported status.
|
||||
At most 128 pairs are admitted per transport, with bounded command/data queues,
|
||||
at most 32 candidates per attempt and finite establishment deadlines.
|
||||
Stale answers and duplicate offers are rejected; a new attempt replaces and
|
||||
closes an older pair engine. Cancellation, peer departure, network deactivation
|
||||
and shutdown retire sockets without waiting for a remote reader.
|
||||
|
||||
A process restart, a failed setup, or a complete fallback cycle returning to AWG
|
||||
starts fresh negotiation. Normal WireGuard rekeying within a live attempt keeps
|
||||
that attempt's masking profile. A failed pair does not reconfigure other pairs
|
||||
or other networks. A full mesh of 15 members uses 14 engines per member and 105
|
||||
independent pairs in total; it does not create additional system interfaces.
|
||||
|
||||
## Masking
|
||||
|
||||
Each profile randomizes disjoint H1–H4 header ranges, a 12–48-byte S1–S4 prefix,
|
||||
a ChaCha20 header-protection key, handshake trailers, junk packet count/lengths,
|
||||
and extra content padding. Prefix lengths are equal within one profile so
|
||||
header detection has one offset. A fresh random nonce is in every packet's
|
||||
prefix; handshake headers and transport headers/counters are protected.
|
||||
The standard WireGuard header is restored before boringtun authenticates a
|
||||
message. This preserves boringtun's standard MAC calculations, which is another
|
||||
reason this is a Tsunagi-specific AWG wire version.
|
||||
|
||||
Before a handshake, an I1 signature packet is generated from either a complete
|
||||
DNS query or SIP OPTIONS template, with fresh identifiers and consistent length
|
||||
fields, followed by random junk packets. I2–I5 are unused in this version.
|
||||
The templates are structural generators, not replayed captures. QUIC imitation
|
||||
is not selected. Masking can change DPI classification; it is not a promise of
|
||||
indistinguishability or that any particular firewall will admit it.
|
||||
|
||||
## Packets, NAT and fallback
|
||||
|
||||
The pair engine carries opaque relay envelopes inside authenticated outer WG
|
||||
packets. It fragments those envelopes into bounded chunks and expires incomplete
|
||||
assemblies; a logical datagram is bounded at 64 KiB and an outer UDP packet is
|
||||
at most 1200 bytes. The encapsulated IPv4 headers are internal framing only: no interface or OS address is created for a
|
||||
pair. Overlay addresses, routing, ownership checks and end-to-end encryption
|
||||
remain independent. A relay decrypts its own adjacent AWG carrier to forward
|
||||
opaque end-to-end ciphertext, never the destination's IP packet.
|
||||
|
||||
Local IPv4 candidates and simultaneous encrypted probes provide direct UDP
|
||||
connectivity. In `direct`/`relay` reach mode the CLI also uses
|
||||
`stun.l.google.com:19302` to discover a server-reflexive address on **this pair's
|
||||
socket**, with a 1.2-second bound. A STUN result is only a candidate. Local reach
|
||||
mode makes no STUN request. Configure or disable it with
|
||||
`-o awg:stun=HOST:PORT` or `-o awg:stun=off`. Failed discovery still permits local
|
||||
candidates. IPv6-only paths, symmetric NAT and networks blocking arbitrary UDP
|
||||
may fail AWG and fall back; there is no AWG-over-QUIC substitution or AWG TURN
|
||||
relay hidden behind its name.
|
||||
|
||||
Data delivery checks go through the actual end-to-end encrypted tunnel, once
|
||||
per second, independently of control-plane RTT. Requests and replies bind a
|
||||
fresh private marker/key to the sender's endpoint id. Unanswered requests expire
|
||||
after three seconds. After a 20-second warmup, at least 15 completed observations
|
||||
and at least 60% loss in the last 20 observations retire a carrier. Probes are
|
||||
consumed locally after source/destination admission and never enter the TUN.
|
||||
AWG additionally checks encrypted outer delivery throughout the pair lifetime.
|
||||
|
||||
Setup failure or persistent runtime loss advances to the next mutually supported
|
||||
carrier in the initiator's order. Earlier expired cooldowns do not jump ahead of
|
||||
untried carriers. Retry cooldowns remain bounded; reaching AWG again negotiates
|
||||
new parameters. Control sessions and the logical end-to-end tunnel survive a
|
||||
carrier change. Status names the end-to-end protocol and the actual carrier
|
||||
separately, so `wg` over `awg` is a valid result.
|
||||
|
||||
Control wire version 6 introduces the private transport message. All members
|
||||
must upgrade together; device identities, network ids, handshake transcript
|
||||
encoding, signed state and stored network configuration are unchanged. Data
|
||||
ALPN remains `tsunagi/data/5`; the relay envelope is unchanged.
|
||||
|
||||
## Verification
|
||||
|
||||
The default rootless, offline tests use real iroh endpoints, membership proofs,
|
||||
SQLite, UDP sockets, masked WireGuard handshakes, end-to-end encryption and
|
||||
memory TUNs. They cover independent three-member pairs, full-size bidirectional
|
||||
packets, restart freshness/socket release, 80% runtime datagram loss, fallback
|
||||
to real iroh and fresh negotiation on retry, plus malformed profiles/messages,
|
||||
role checks, probe forgery and STUN response parsing. These tests do not establish
|
||||
DPI effectiveness, WAN throughput, public NAT traversal or per-pair memory cost.
|
||||
@@ -20,8 +20,9 @@ network secret. The core moves an opaque, bounded payload and never parses it.
|
||||
a `PacketLink` — an authenticated, unreliable datagram channel to one peer for
|
||||
one protocol — and never learn how it is carried.
|
||||
|
||||
The separation between the two planes is **logical, not physical**. Both ride
|
||||
on iroh, on different ALPNs and different connections. That is deliberate:
|
||||
The separation between the two planes is **logical, not physical**. The control plane uses
|
||||
iroh; the data plane can use iroh on its own ALPN and connection, or independent
|
||||
UDP/TLS carriers. That is deliberate:
|
||||
iroh's whole value is hole punching a direct path between peers behind NAT,
|
||||
with a relay as fallback, and a data plane that refused to use it would have to
|
||||
reimplement all of it. What the separation buys is that `proto` knows nothing
|
||||
@@ -32,6 +33,10 @@ A plugin talks to the core through three narrow hooks — `on_network_activated`
|
||||
a `PluginContext` for re-announcements and error reports, and a bounded
|
||||
`shutdown` — so the core never learns anything protocol-specific.
|
||||
|
||||
Private pair negotiation uses a bounded transport sender supplied by the core.
|
||||
Protocol crates do not encode or parse control messages and never receive the
|
||||
network secret. No pair profiles enter signed state.
|
||||
|
||||
A data plane failure never stops the daemon: the control plane keeps running
|
||||
and the agent stays manageable.
|
||||
|
||||
@@ -68,6 +73,7 @@ And in crates of their own:
|
||||
|
||||
| crate | responsibility |
|
||||
|---|---|
|
||||
| `tsunagi-awg` | ephemeral pairwise Amnezia masking, private negotiation and independently encrypted UDP carriers; reuses the end-to-end WireGuard engine |
|
||||
| `tsunagi-wg-quic` | the `wg-quic` protocol: its keys, its announcement, its tunnels |
|
||||
| `tsunagi` | the command line agent, built as the `tsng` executable: the only place that owns a runtime, a logger and signals |
|
||||
|
||||
|
||||
+30
-6
@@ -43,12 +43,36 @@ be an exit node in one network and not in another.
|
||||
`tsunagi-exit:<interface>:<range>` and are removed when the offer ends, the
|
||||
network stops, and when the agent exits. Only networks that offer have
|
||||
rules, so the others' traffic is not forwarded.
|
||||
- **Client:** policy routing, with no change to the main table. Table `28787`
|
||||
holds `default dev <interface>`, and three rules sit after Tailscale's:
|
||||
`5280 uidrange <agent uid> lookup main` (the agent's own connections never
|
||||
go through the tunnel, which would be a loop), `5290 lookup main
|
||||
suppress_prefixlength 0` (LAN and other specific routes keep working) and
|
||||
`5300 lookup 28787`.
|
||||
- **Client:** policy routing, with no change to the main table. Table `29811`
|
||||
holds `default dev <interface>`. Rules `5205 fwmark 0x40000000/0x40000000
|
||||
lookup main` and `5206 uidrange <agent uid> lookup main` keep the agent's
|
||||
sockets and their replies outside the tunnel, before Tailscale's rules.
|
||||
`5290 lookup main suppress_prefixlength 0` preserves LAN and other specific
|
||||
routes, and `5300 lookup 29811` directs other traffic through the exit.
|
||||
The exit table is `29811` (`0x7473`).
|
||||
- **Linux reverse-path filtering:** only the agent's dedicated UID is marked
|
||||
in mangle `OUTPUT`. Its own bit, `0x40000000`, is saved to conntrack and
|
||||
restored in `PREROUTING` before the host firewall checks reply routing.
|
||||
Other mark bits, including Tailscale's `0xff0000` mask, are preserved. The
|
||||
rules are tagged `tsunagi-exit-underlay:<interface>` and never accept packets
|
||||
around the host firewall. IPv4 and IPv6 protection is installed before exit
|
||||
routes; routes are removed before protection, including rollback on failure.
|
||||
Locally owned TCP listeners and bound UDP sockets also protect the first
|
||||
incoming packet, before any outgoing packet has created a connection mark.
|
||||
Socket addresses and ports
|
||||
come from local kernel socket tables and match an actual socket; remote
|
||||
announcements never supply these firewall arguments.
|
||||
The socket set is refreshed before private pair negotiation is sent and on
|
||||
discovery ticks, so fresh AWG ports are included. An unchanged set causes no
|
||||
firewall writes; a socket-refresh failure leaves active exit routes in place.
|
||||
Linux needs both `iptables` and `ip6tables` when IPv6 is present.
|
||||
- **Kernel strict `rp_filter`:** if an interface uses strict mode (`1`), the
|
||||
kernel must include marks in reverse lookups: set
|
||||
`net.ipv4.conf.all.src_valid_mark=1`, or enable it on those interfaces.
|
||||
The agent checks this before installing exit routes and reports an actionable
|
||||
error if marks would be ignored. It never disables `rp_filter` or changes
|
||||
host sysctls. Netfilter strict reverse checks also need their mark-aware
|
||||
form, such as `rpfilter --validmark`, which NixOS already uses.
|
||||
- Packets for addresses outside every overlay range are sent to the exit node
|
||||
by the router; the exit node's replies are accepted from any source. A member
|
||||
that is not your exit node cannot do that.
|
||||
|
||||
+44
-18
@@ -9,16 +9,26 @@ Two versions exist and are independent:
|
||||
|
||||
- **Identity scheme**, `tsunagi-network-id-v1`. Frozen. Changing it creates a
|
||||
different network space for the same name and secret.
|
||||
- **Control protocol**, ALPN `tsunagi/ctrl/4`, `PROTOCOL_VERSION = 4`.
|
||||
- **Control protocol**, ALPN `tsunagi/ctrl/8`, `PROTOCOL_VERSION = 8`.
|
||||
|
||||
Upgrading the crate or bumping the control protocol must never change an
|
||||
existing `NetworkId`.
|
||||
|
||||
Control version 8 adds signed, multi-writer VIP field snapshots to version 7's
|
||||
node retirements. Data ALPN remains `tsunagi/data/5`. All members must upgrade
|
||||
together. Network ids, derivation labels, handshake encoding and existing record
|
||||
signatures are unchanged. Local CLI/tray IPC is version 22; restart the agent
|
||||
with matching clients. SQLite schema 10 adds `vip_fields`, preserving identity,
|
||||
network configuration, signed node records and retirement proofs.
|
||||
See [virtual services](vip.md), [state sync](sync-model.md) and [AWG](amnezia.md).
|
||||
|
||||
## Canonical encoding
|
||||
|
||||
Everything that is hashed or MAC'd uses length-prefixed fields, written here as
|
||||
`LP(x) = u32_be(len(x)) || x`. String concatenation is never used, so no two
|
||||
different field splits can produce the same bytes.
|
||||
Network identity derivation and the membership transcript use length-prefixed
|
||||
fields, written here as `LP(x) = u32_be(len(x)) || x`. String concatenation is
|
||||
never used, so different field splits cannot produce the same bytes. VIP field
|
||||
signatures use their separate domain and canonical postcard encoding described
|
||||
in [virtual services](vip.md#shared-state-and-addresses).
|
||||
|
||||
## Network space identity
|
||||
|
||||
@@ -135,7 +145,7 @@ event does not report a network id.
|
||||
## The data plane protocol
|
||||
|
||||
IP plugin packets never travel on a control connection. They use their own
|
||||
ALPN, `tsunagi/data/4`, on their own iroh connection:
|
||||
ALPN, `tsunagi/data/5`, on their own iroh connection:
|
||||
|
||||
```text
|
||||
initiator -> responder : (the same membership handshake as above)
|
||||
@@ -172,32 +182,46 @@ holding up subsequent packets. Packet IDs are scoped to a QUIC connection;
|
||||
reassembly is created only after the membership handshake. All limits live in
|
||||
`config.rs`. There is no transport-layer retransmission added by this framing.
|
||||
|
||||
Version 4 adds a routing envelope inside the transport fragmentation framing:
|
||||
Version 4 added a routing envelope inside the transport fragmentation framing,
|
||||
and version 5 added the end-to-end protocol to it:
|
||||
|
||||
| field | bytes | encoding |
|
||||
|---|---:|---|
|
||||
| envelope version | 1 | `1` |
|
||||
| envelope version | 1 | `2` |
|
||||
| remaining hops | 1 | initially `16`; transit decrements, drops at `1` |
|
||||
| source peer | 32 | original endpoint public key |
|
||||
| destination peer | 32 | final endpoint public key |
|
||||
| flow id | 8 | big endian opaque stable hash supplied by the plugin |
|
||||
| protocol | 8 | big endian FNV-1a of the end-to-end protocol id (`wg`, `tcp-tls`, ...) |
|
||||
| payload | remaining | end-to-end encrypted plugin bytes |
|
||||
|
||||
The fixed header is 74 bytes. Zero/oversized hop limits, unknown versions,
|
||||
The fixed header is 82 bytes. Zero/oversized hop limits, unknown versions,
|
||||
short frames, oversized datagrams and unknown sources are rejected. A frame
|
||||
addressed here is delivered to its source's protocol inbox; transit goes straight
|
||||
to its next transport. The network and protocol are bound to the authenticated
|
||||
transport connection, not supplied by the frame. Intermediate nodes cannot
|
||||
decrypt or authenticate the inner WireGuard payload; the destination does that.
|
||||
Flow ids are routing hints, not authorization proofs.
|
||||
addressed here is delivered to the inbox of its source and protocol; transit
|
||||
goes straight to its next transport, whatever kind of transport that is. The
|
||||
network is bound to the authenticated transport connection, not supplied by
|
||||
the frame. The protocol names which end-to-end tunnel the payload belongs to
|
||||
and is independent of the links it crosses: a datagram of one protocol can
|
||||
travel over a link opened for another, and a middle member needs no protocol
|
||||
in common with either end. Intermediate nodes cannot decrypt or authenticate
|
||||
the inner payload; the destination's tunnel does that, so a frame naming the
|
||||
wrong protocol or a forged source is simply discarded there. Flow ids are
|
||||
routing hints, not authorization proofs.
|
||||
|
||||
Control ALPN 3 carries `Reachable { links: [{ peer, protocol }] }`. Each row
|
||||
belongs to the authenticated sender and is replaced atomically, expires after
|
||||
90 seconds, and is withdrawn on session closure. Only compatible authenticated
|
||||
members enter a protocol's graph; local edges always come from actual links.
|
||||
`Reachable { links: [{ peer, protocol }] }` lists the sender's direct links;
|
||||
`protocol` is the kind of transport the link is, for diagnostics, and routing
|
||||
ignores it. Each row belongs to the authenticated sender and is replaced
|
||||
atomically, expires after 90 seconds, and is withdrawn on session closure. Only
|
||||
authenticated members enter the graph; local edges always come from actual
|
||||
links.
|
||||
Announcements refresh on the existing control maintenance interval and are
|
||||
sent immediately on link changes. Topology stays volatile, outside signed state.
|
||||
|
||||
Version 5 also adds `hostname_since` to a signed hostname claim (signing domain
|
||||
`tsunagi-signed-record-v3`): a name belongs to whoever claimed it first. Stored
|
||||
records signed before are discarded on upgrade and re-published by their
|
||||
authors.
|
||||
|
||||
Upgrade both endpoints and intermediate peers together. Older control and data
|
||||
ALPNs cannot interoperate. The network secret, device identity, identity
|
||||
derivation, transcript encoding and stored network configuration are unchanged.
|
||||
@@ -216,11 +240,13 @@ not affect other networks.
|
||||
| message | meaning |
|
||||
|---|---|
|
||||
| `Announce { hostname, capabilities, broadcast }` | this agent's hostname, IP-plugin capabilities and local broadcast participation |
|
||||
| `Transport { protocol, payload }` | private, bounded per-peer transport negotiation; never state or a public message event |
|
||||
| `Ping { seq, payload }` | small request used to verify the exchange |
|
||||
| `Pong { seq, payload }` | the echoed reply |
|
||||
| `State { records }` | a snapshot of signed records, merged into what the receiver holds |
|
||||
| `Retirements { markers }` | signed cleanup markers with the target's original signed claim as proof |
|
||||
| `Peers { peers }` | unverified member address hints, requiring their own handshake |
|
||||
| `Reachable { links }` | sender's current direct data links, scoped by protocol |
|
||||
| `Reachable { links }` | sender's current direct data links, by kind of transport |
|
||||
| `Bye { reason }` | graceful goodbye; not a revocation of anything |
|
||||
|
||||
`broadcast` is a per-network local opt-in, enabled by default and treated as
|
||||
|
||||
+47
-9
@@ -15,12 +15,27 @@ the default 1280-byte interface MTU also works on small QUIC paths.
|
||||
## Graph and lifecycle
|
||||
|
||||
Each authenticated control session advertises that peer's own direct data
|
||||
links, including the plugin protocol. Tables are separate for each network
|
||||
and protocol; only members offering a matching enabled protocol version enter
|
||||
the graph. A snapshot refreshes on the maintenance interval, expires after
|
||||
90 seconds, and is withdrawn when the session ends. Link arrival, closure and
|
||||
changed announcements rebuild routes immediately. Unchanged announcements
|
||||
refresh their age without republishing the forwarding snapshot.
|
||||
links. There is one table for each network, whatever protocols run over it: a
|
||||
link is a way to reach a neighbour, and a protocol is an end-to-end tunnel to a
|
||||
peer. The envelope names the protocol, so the two are independent. A neighbour
|
||||
reached over several kinds of transport (UDP, TCP, QUIC datagrams) is sent to
|
||||
over the best open one, in the order the protocols were configured, and over
|
||||
the next if that one fails or cannot carry the frame. Every authenticated
|
||||
member enters the graph; a member in the middle needs no protocol in common
|
||||
with either end, only a link to each. A snapshot refreshes on the maintenance
|
||||
interval, expires after 90 seconds, and is withdrawn when the session ends.
|
||||
Link arrival, closure and changed announcements rebuild routes immediately.
|
||||
Unchanged announcements refresh their age without republishing the forwarding
|
||||
snapshot.
|
||||
|
||||
Which protocol a pair of members runs their tunnel in is settled without
|
||||
asking: the member with the lower endpoint id decides, by the order it
|
||||
announces its protocols in, among those both have enabled at the same wire
|
||||
version. Both ends read the same answer from what they already hold. The
|
||||
tunnel is created as soon as the peer is reachable by any path, and it does not
|
||||
change when a direct link comes or goes underneath it: as soon as a direct
|
||||
link comes up it is shorter than the path through others, and traffic moves to
|
||||
it with nothing to redo. The user sees only a lower round-trip time.
|
||||
|
||||
Breadth-first search computes the shortest directed paths and every equal-cost
|
||||
first hop in sorted order. An actual direct link always wins. The first row is
|
||||
@@ -30,14 +45,29 @@ loops if different nodes have not yet received the same topology update.
|
||||
|
||||
The existing control plane still forms authenticated pairwise sessions.
|
||||
Routing provides multihop **data** paths among these members; it does not add
|
||||
control-plane flooding or carry control sessions through the data plane.
|
||||
control-plane flooding or carry control sessions through the data plane, so a
|
||||
member that has no control session with another has no data path to it either.
|
||||
Announcements are volatile, not durable membership or availability guarantees.
|
||||
|
||||
## What "online" means
|
||||
|
||||
One definition, computed by the agent and reported identically by `tsng status`
|
||||
and the tray: a member is online when this agent can reach it by any path — a
|
||||
live authenticated control session (QUIC keep-alive every 5 seconds, so a
|
||||
vanished peer is noticed within about 15), or a data path to it through other
|
||||
members. It is never inferred from ping, which a firewall may block, or from a
|
||||
tunnel's last handshake, which says only when traffic last flowed.
|
||||
|
||||
How it is reached is reported separately: `direct` over a named transport,
|
||||
`relayed` through a named member over N hops, or no data path yet while one is
|
||||
being set up. Being relayed is not being degraded in any way but distance, and
|
||||
is shown so it is not a surprise.
|
||||
|
||||
## Packet path
|
||||
|
||||
The control loop binds next hops to transport handles in an immutable table
|
||||
and publishes it through `ArcSwap`. Readers do not acquire the topology mutex.
|
||||
The transit routine validates the fixed 74-byte envelope, looks up the source
|
||||
The transit routine validates the fixed 82-byte envelope, looks up the source
|
||||
and destination, selects a cached next hop, decrements one byte, and calls the
|
||||
transport's synchronous datagram send. It never searches the graph, formats an
|
||||
endpoint string, validates an Ed25519 key, decrypts, or touches TUN.
|
||||
@@ -66,6 +96,14 @@ topology change can move a flow; datagrams remain unreliable and unordered,
|
||||
and there is no promise to preserve order across a physical path failure.
|
||||
Logical peer links and end-to-end encryption state survive these changes.
|
||||
|
||||
## Carrier health and switching
|
||||
|
||||
Carrier setup failures and persistent encrypted delivery loss advance the
|
||||
initiator's ordered fallback cycle. Logical end-to-end links survive while
|
||||
physical links are retired explicitly. The AWG carrier negotiates a fresh
|
||||
private pair profile on every reentry, without changing topology's durable
|
||||
state. See [amnezia.md](amnezia.md) for thresholds and bounds.
|
||||
|
||||
## Verification and measurement
|
||||
|
||||
The default offline suite exercises four real agents, iroh transports and
|
||||
@@ -86,5 +124,5 @@ counts sends without storing frames. Results are CPU forwarding cost, **not**
|
||||
end-to-end network latency or VPN throughput; encryption, fragmentation,
|
||||
sockets, congestion and scheduling contribute separately.
|
||||
|
||||
Wire compatibility: control ALPN `tsunagi/ctrl/4`, data ALPN `tsunagi/data/4`.
|
||||
Wire compatibility: control ALPN `tsunagi/ctrl/8`, data ALPN `tsunagi/data/5`.
|
||||
Upgrade every participant together; saved identities and network state persist.
|
||||
+65
-28
@@ -1,18 +1,9 @@
|
||||
# Planned state synchronisation
|
||||
# State synchronisation
|
||||
|
||||
**The first slice of this model is now implemented**, in `crates/tsunagi-core/src/state/`, and is
|
||||
used for one thing: IPv4 overlay addresses. What follows describes the whole
|
||||
model; the section at the end says exactly which parts exist.
|
||||
|
||||
The proof of concept still exchanges hostname and capability announcements
|
||||
over live sessions and keeps no replicated history for those, which is why
|
||||
WireGuard peer *membership* remains session-scoped even though a peer's
|
||||
*address* no longer is. This file records the intended direction so the module
|
||||
boundaries in [architecture.md](architecture.md) stay compatible with it, and so
|
||||
nobody mistakes the current announcements for synchronisation.
|
||||
|
||||
There is no fake "ready CRDT" here and no snapshots that are not actually
|
||||
verified.
|
||||
Signed claims in `crates/tsunagi-core/src/state/` preserve IPv4 reservations and
|
||||
hostnames across an absence. Signed releases and offline-node cleanup markers
|
||||
free those reservations durably. Capabilities, links and authenticated sessions
|
||||
remain volatile; a stored claim never authenticates a live connection.
|
||||
|
||||
## The problem
|
||||
|
||||
@@ -43,7 +34,9 @@ document trusted merely because the neighbour who forwarded it signed it.
|
||||
- Duplicates do not change the result.
|
||||
- Two conflicting signed records at the same version from the same author need
|
||||
explicit handling; they are not resolved by luck.
|
||||
- Neither arrival order nor system clocks decide a winner.
|
||||
- Neither arrival order nor the receiving replica's clock decides a winner. The
|
||||
one thing an author's own clock is used for is the time it first claimed a
|
||||
hostname, signed into its record (see below).
|
||||
- Compaction must not drop what is needed to stop revoked records being
|
||||
resurrected.
|
||||
|
||||
@@ -53,10 +46,59 @@ A hostname is a mutable binding to a persistent author, not an identity. A
|
||||
rename must be a signed record that revokes the specific old binding and
|
||||
announces the new one, ideally atomically in one record.
|
||||
|
||||
A hostname belongs to the member that claimed it **first**. Each claim carries
|
||||
`hostname_since`, the author's own time in milliseconds when it first claimed
|
||||
that exact name, signed with the rest of the record and kept across every later
|
||||
version that keeps the name; renaming starts it afresh. Replicas order claims
|
||||
by `(hostname_since, author id)`, a pure function of the records, so every
|
||||
replica names the same holder whatever order the records arrived in and there
|
||||
is no race to lose. A member that arrives later with a name already held keeps
|
||||
working by address and keeps its claim, but is not found by that name on any
|
||||
member; it is told so in its status and in `tsng status`, and finds the name
|
||||
again if the holder renames or releases it.
|
||||
|
||||
It is the author's word, so a member with a clock set far back can claim an
|
||||
older time. That can reorder who holds a *name* and nothing else — addresses,
|
||||
keys and everything the data plane trusts are decided without it — and anybody
|
||||
who knows the secret could take a name by other means anyway.
|
||||
|
||||
Turning a computer off is not a revocation of its hostname and does not remove
|
||||
the participant. Revocations are not dropped merely because they are old, and
|
||||
no acknowledgement from offline peers is required to keep working.
|
||||
|
||||
## Offline-node cleanup
|
||||
|
||||
Any authenticated member can sign a `SignedRetirement` for another member's
|
||||
observed `Claim`. It includes the original signed claim as proof and uses the
|
||||
separate `tsunagi-node-retirement/v1` signing domain. Both signatures and both
|
||||
network scopes are checked. No owner, vote or quorum is involved, and no member
|
||||
can fabricate the target's release or invent a future version cutoff.
|
||||
|
||||
The strongest observed target version wins; ties use a deterministic canonical
|
||||
byte order. Claims at or below that version do not occupy an address, hostname
|
||||
or visible member row. Original signed records remain stored, including their
|
||||
range, so cleanup does not renumber the network. Markers remain even after a
|
||||
newer claim returns: absence from a snapshot is never deletion and replaying an
|
||||
old snapshot cannot restore the retired claim.
|
||||
|
||||
The local `Agent::forget_node` API checks the exact observed version again and
|
||||
refuses self, online nodes, the selected exit node and missing/changed claims.
|
||||
CLI and tray share this API. A returning live node can sign a newer claim and
|
||||
allocate against the current holders; it resets the old hostname priority, so
|
||||
it cannot reclaim a reused name merely by keeping its old timestamp. Cleanup
|
||||
is not a permanent identity ban or a change of secret.
|
||||
|
||||
The marker and requester's own author counter commit in one SQLite transaction
|
||||
before publication. Receivers persist a verified marker before applying it.
|
||||
Invalid proofs are rejected individually. Periodic bounded marker batches
|
||||
rotate through each peer's send queue, including when a queue fills, so a
|
||||
replica that missed an update can catch up without the requester being online.
|
||||
|
||||
One strongest marker per target is retained, with a default limit of 1024 per
|
||||
network and 1024 bytes per encoded marker. New targets beyond that limit are
|
||||
refused rather than evicting resurrection protection. Corrupt mandatory marker
|
||||
storage or a configured limit smaller than the stored set is an explicit error.
|
||||
|
||||
## Storage requirement this creates
|
||||
|
||||
When signed records land, writing the event and bumping the author's own
|
||||
@@ -86,9 +128,9 @@ Implemented, in `crates/tsunagi-core/src/state/`:
|
||||
* the merge rules above: higher version wins, an older version never rolls
|
||||
back a newer one, duplicates are idempotent, a same-version conflict is
|
||||
resolved identically on every replica and reported;
|
||||
* a release tombstone, which merges correctly and is not undone by a replica
|
||||
that has not heard of it — though nothing emits one yet, so freeing an
|
||||
address still means forgetting the network;
|
||||
* a release tombstone, emitted when leaving or replacing the device key and
|
||||
not undone by a replica that has not heard of it;
|
||||
* durable hostnames and equal-rights offline-node retirement described above;
|
||||
* persistence in `state.sqlite`, with the record and the author's own version
|
||||
counter committed in **one transaction before the record is announced**;
|
||||
* distribution as a `State` control message, merged into what the receiver
|
||||
@@ -96,21 +138,16 @@ Implemented, in `crates/tsunagi-core/src/state/`:
|
||||
* allocation of a free IPv4 address against what everybody else holds, which
|
||||
is what makes an address stable across an absence.
|
||||
|
||||
Deliberately not implemented: compaction, revoking a whole author, record
|
||||
types beyond addressing, and any bound on how large a snapshot may grow
|
||||
beyond the per-message limit.
|
||||
Deliberately not implemented: compaction and permanent author revocation.
|
||||
Claims have a per-message bound; retirement history also has a per-network bound.
|
||||
|
||||
## Future tests
|
||||
|
||||
These are **not implemented and must not be reported as passing**:
|
||||
|
||||
- snapshot merge against the rules above, including conflicting same-version
|
||||
records;
|
||||
- revocation propagation and resistance to resurrection after compaction;
|
||||
- resistance to resurrection after compaction (compaction is not implemented);
|
||||
- long network partitions and rejoin after an extended absence;
|
||||
- hostname rename with atomic revoke-and-announce;
|
||||
- recovery of a returning participant from a replica that is not the author;
|
||||
- NAT traversal and hole punching between real hosts;
|
||||
- relay fallback behaviour against a self-hosted relay;
|
||||
- multi-process and multi-host deployment, as opposed to several library
|
||||
instances inside one test process.
|
||||
- multi-host deployment. CLI tests do run separate local agent processes;
|
||||
library tests run several independent instances inside one process.
|
||||
+86
-4
@@ -98,6 +98,13 @@ a batch does not stop the rest, and allocation is deterministic, spread out,
|
||||
walks past everything taken and reports a full range instead of handing out a
|
||||
duplicate.
|
||||
|
||||
`tests/range_conflicts.rs` checks isolated first members choosing the same
|
||||
range, persistence across restart, preservation of legacy ranges, exclusion
|
||||
of equal and nested ranges, late authenticated conflicts, switching all
|
||||
blockers, automatic resume, cancelling a suspended network's resume and
|
||||
concurrent switch requests. These use real agents, loopback handshakes and
|
||||
SQLite; they do not change host network settings.
|
||||
|
||||
`tests/local_control.rs` covers the local control socket end to end: a client
|
||||
asking a running agent for status over a real Unix socket, joining and
|
||||
leaving a network through it, a leftover socket file being replaced while a
|
||||
@@ -157,10 +164,9 @@ paused clock. The public Mainline/relay smoke test is ignored by default.
|
||||
|
||||
## Not covered, and not claimed to be
|
||||
|
||||
Listed in [sync-model.md](sync-model.md#future-tests): snapshots, revocations,
|
||||
long partitions, hostname renames, recovery of a returning participant, NAT
|
||||
traversal, relay fallback, and multi-process or multi-host deployment. None of
|
||||
these are implemented, and none are marked as passing.
|
||||
Listed in [sync-model.md](sync-model.md#future-tests): compaction, long partitions,
|
||||
NAT traversal, relay fallback and multi-host deployment. These scenarios are
|
||||
not established by the default loopback suite.
|
||||
|
||||
## Debugging a test
|
||||
|
||||
@@ -170,3 +176,79 @@ TSUNAGI_TEST_LOG=tsunagi=debug cargo test --test multi_peer -- --nocapture
|
||||
|
||||
The library never installs a global subscriber; the harness opts in only when
|
||||
that variable is set.
|
||||
|
||||
The `tsunagi-awg` suite runs real pair negotiation and masked UDP WireGuard
|
||||
engines with memory TUNs. A loss wrapper drops datagrams at the carrier boundary
|
||||
without substituting iroh, membership, encryption or SQLite. It checks runtime
|
||||
loss detection, fallback and fresh profiles, in addition to restart and input
|
||||
validation. Tests are offline; public STUN/DPI/NAT effectiveness needs separate
|
||||
real-network validation.
|
||||
|
||||
Runtime protocol selection tests switch real AWG and iroh/QUIC carriers and
|
||||
end-to-end WireGuard tunnels from either endpoint-id role, carry full-size
|
||||
packets, keep another active network's tunnels unchanged, disable and re-enable
|
||||
the data plane without losing control sessions, and preserve selection after
|
||||
restart. CLI tests use a real agent process and control socket to exercise
|
||||
selection by network name, stopped networks, invalid lists, and persisted
|
||||
control-only configuration, offline updates, and enabling a protocol absent
|
||||
from the startup list. A real AWG test starts without a TUN, creates the single
|
||||
memory interface on enablement, and races rapid changes against pending opens.
|
||||
SQLite tests migrate schema 7 to 8 without changing
|
||||
identity, secrets or signed records, preserve order and explicit empty choices,
|
||||
and reject malformed stored selections. These checks use memory TUNs and do
|
||||
not establish GUI rendering or behavior on an actual OS interface.
|
||||
|
||||
`tests/forget_node.rs` uses real agents, handshakes and SQLite to check equal
|
||||
cleanup rights, network isolation, online/self/stale-version refusal, restart,
|
||||
recovery of a stale replica, reuse of a hostname and a returning identity's
|
||||
newer claim. Bounded batch delivery is checked with a two-message send queue
|
||||
and twelve retained markers. Retirement unit tests check signatures, proof tampering, invented
|
||||
future cutoffs, bounds, deterministic merge and resistance to stale snapshots.
|
||||
SQLite tests migrate schema 8 to 9 and inject counter-write failure to prove
|
||||
atomic rollback. The CLI test starts two real agent processes and exercises
|
||||
offline cleanup by network and hostname through the control socket.
|
||||
Native tray rendering and physical screen sizing are not exercised automatically.
|
||||
|
||||
Linux exit-client unit tests cover preservation of foreign mark bits, matching
|
||||
only the agent's UID and locally owned TCP/UDP sockets, exact tagged cleanup,
|
||||
partial IPv6 failure and mark-aware strict kernel RPF validation. The ignored
|
||||
`overlay::hostrules::linux_exit_kernel::strict_firewall_preserves_underlay_and_application_exit_routing`
|
||||
test creates two disposable network namespaces. It reproduces the UID-only
|
||||
failure with real sockets, then runs TCP/UDP IPv4/IPv6 and first inbound TCP/UDP
|
||||
packets, including a UDP port bound after activation, through real mark/conntrack
|
||||
rules with both strict kernel and
|
||||
Netfilter reverse-path checks. It verifies application exit routing, IPv6
|
||||
blocking, repeated apply and cleanup with foreign rules at Tailscale's priority
|
||||
retained. It requires root, iproute2,
|
||||
iptables/ip6tables and Python; it does not run in the default suite or claim
|
||||
to test the entire mesh inside these namespaces.
|
||||
|
||||
Devices-window tests render the egui grid at multiple widths with its real
|
||||
scroll area. They check stretching and shrinking without an expanding first
|
||||
column, separate handshake/control RTT values, unknown measurements and short
|
||||
end-to-end hints that do not substitute the carrier protocol. This does not
|
||||
establish native desktop rendering on every platform.
|
||||
|
||||
|
||||
VIP tests cover signed multi-writer backend merging, stale draft rejection,
|
||||
permanent deletion and replay, separate active/deleted capacity, and mandatory
|
||||
SQLite migration/counter rollback. Real agent instances verify equal editing
|
||||
rights, network isolation, restart and recovery without the creator. Packet
|
||||
checks exercise RTT selection, port translation, UDP failover, a local backend,
|
||||
fragment reassembly/checksums and ICMP quoted-header/PMTU translation. TCP check
|
||||
recovery uses real loopback listening sockets. The WireGuard suite moves VIP
|
||||
TCP/UDP packets through real encrypted tunnels, rejects forged VIP sources and
|
||||
checks UDP failover after the selected agent stops. CLI/DNS tests run the real
|
||||
binary, configure independent listeners, disable checks, reject an invalid
|
||||
change atomically and observe DNS creation/deletion. GUI tests cover draft
|
||||
isolation and form rendering. These do not establish live host TUN behavior or
|
||||
native desktop behavior on Linux, macOS or Windows. See [VIP semantics](vip.md).
|
||||
|
||||
Tray layout tests render bounded table rows with long names, online/offline
|
||||
members, Forget buttons, large counters and enlarged fonts. They assert text
|
||||
stays within its own row and that narrow windows scroll without expanding the
|
||||
viewport. VIP overview tests render live tables and egui scenes at multiple
|
||||
widths, click a diagram backend, and distinguish the new-flow selection from
|
||||
traffic on an existing binding. Sampling checks cover idle open flows, stale
|
||||
observations, counter resets, recreated listeners and removal of cached
|
||||
histories. These are headless egui checks, not native desktop screenshots.
|
||||
+25
-4
@@ -26,8 +26,8 @@ Read this before relying on anything here. The protocol is in
|
||||
neither receive nor forge another member's traffic. See
|
||||
[wireguard.md](wireguard.md#address-ownership-is-enforced-not-announced).
|
||||
- **Tunnelled traffic is end-to-end encrypted by WireGuard**, independently of
|
||||
this crate. The transport underneath is also encrypted by iroh, but the
|
||||
tunnel's confidentiality does not depend on that.
|
||||
this crate. Carriers may be iroh, TLS, direct WG or
|
||||
pairwise AWG; confidentiality does not depend on a particular carrier.
|
||||
- **Resource bounds.** Frame lengths are validated before allocation; strings,
|
||||
lists, queues, concurrent dials and in-flight handshakes are all bounded;
|
||||
handshakes, dials and writes have timeouts.
|
||||
@@ -38,6 +38,14 @@ Read this before relying on anything here. The protocol is in
|
||||
arbitrarily many identities, flood the network with records and collide with
|
||||
other participants' names. This is why a majority is not a root of trust.
|
||||
Signatures protect authorship; they do not make a participant honest.
|
||||
- **Cleanup rights are equal.** Any member can retire an observed signed claim
|
||||
of another device, freeing its hostname and IPv4 reservation. A valid original
|
||||
claim bounds the cutoff; a member cannot invent a future cutoff or forge the
|
||||
target's release. The local API refuses online nodes, but a remote signature
|
||||
cannot prove that a target was offline. A live target may publish a newer
|
||||
claim; a malicious member can repeatedly disrupt reservations. Cleanup is
|
||||
not expulsion or secret revocation. See
|
||||
[sync-model.md](sync-model.md#offline-node-cleanup).
|
||||
- **Weak secrets.** This targets high-entropy secrets. There is no PAKE, so a
|
||||
short human passphrase can be guessed offline by anyone who can reach the
|
||||
handshake. Use `NetworkSecret::generate()`.
|
||||
@@ -55,8 +63,10 @@ Read this before relying on anything here. The protocol is in
|
||||
having iroh under it does not make it so.
|
||||
- **A cloned state directory is a cloned identity.** `state.sqlite` holds the
|
||||
device secret key and the network secrets. Copying it copies the participant.
|
||||
Restoring an old backup rolls the agent's state back, which — once signed
|
||||
records exist — can resurrect revoked information or replay stale versions.
|
||||
Restoring an old backup rolls the agent's state back and can replay stale
|
||||
versions until it meets a replica holding the newer signed state. Retained
|
||||
cleanup markers prevent old claims from resurrecting after that merge; they
|
||||
cannot help if every copy of a marker has been lost.
|
||||
- **A compromised host.** The secret is on disk to survive restarts. File
|
||||
permissions are owner-only where the platform supports it, and the state
|
||||
directory takes an ownership lock, but neither defends against a user who can
|
||||
@@ -113,3 +123,14 @@ for key separation, HMAC-SHA256 for the membership proof, constant-time
|
||||
verification, iroh's Ed25519 endpoint keys and QUIC/TLS for the transport, and
|
||||
RFC 5705 TLS exporter output for channel binding. There is no custom encryption
|
||||
layer and no custom PAKE.
|
||||
|
||||
## Pairwise AWG masking
|
||||
|
||||
AWG profiles, PSKs and outer keys are private to one authenticated pair attempt,
|
||||
not shared network state. Address candidates are unverified until authenticated
|
||||
outer WireGuard data arrives. Header protection and structured signature packets
|
||||
reduce fixed wire markers; they do not guarantee admission by DPI or hide traffic
|
||||
analysis. iroh control availability remains a prerequisite. Encrypted end-to-end
|
||||
delivery probes detect sustained carrier loss, never confer membership and pass
|
||||
the same signed source/destination admission as ordinary IP traffic. See
|
||||
[amnezia.md](amnezia.md) for limits and tested scope.
|
||||
+187
@@ -0,0 +1,187 @@
|
||||
# Named virtual services
|
||||
|
||||
A VIP is a shared name such as `api.home`, with independent TCP/UDP listeners.
|
||||
Each listener maps its port to a set of node identities and application ports.
|
||||
The application uses the name; the automatically allocated IPv4 address is an
|
||||
internal reservation. The current overlay carries IPv4 only.
|
||||
|
||||
## CLI and tray
|
||||
|
||||
```sh
|
||||
tsng vip list home
|
||||
tsng vip create home api
|
||||
tsng vip port add home api tcp/443 --backend server1:6443 --backend server2:7443
|
||||
tsng vip port add home api udp/53 --backend server1:5353
|
||||
tsng vip backend add home api tcp/443 server3:6443
|
||||
tsng vip backend remove home api tcp/443 server2:7443
|
||||
tsng vip port set home api tcp/443 --port-check off
|
||||
tsng vip port set home api tcp/443 --check-interval-ms 3000 --check-timeout-ms 2000 --failures 2 --successes 1 --idle-timeout 7200
|
||||
tsng vip show home api
|
||||
tsng vip status home api --watch
|
||||
tsng vip edit home api
|
||||
tsng vip port remove home api udp/53
|
||||
tsng vip remove home api
|
||||
```
|
||||
|
||||
`list` without a network lists all configured networks. Network and VIP
|
||||
selectors accept exact names, full ids and unique id prefixes; backend node
|
||||
selectors also accept stored node names. Ambiguous selections change nothing.
|
||||
Names resolve to identities when saved, so a renamed node remains a backend.
|
||||
`--json` provides configuration, revision, counters and observations;
|
||||
`list`/`watch` emit one JSON object per service/sample. `edit`
|
||||
opens JSON in `VISUAL`/`EDITOR` and submits one complete version-checked draft.
|
||||
An invalid or rejected edited file is retained with its path reported.
|
||||
|
||||
Each mutating command is atomic. A listener can be added with its full initial
|
||||
backend set, and `edit` can replace the entire configuration at once. Empty
|
||||
VIPs/listeners are valid and resolve, but cannot accept traffic. Configuration
|
||||
updates currently require the network to be active locally. Saved configuration remains visible while a
|
||||
network is stopped.
|
||||
|
||||
The tray's network tile has a **VIP** button. Its separate window edits names,
|
||||
listeners, backend nodes/ports, check settings and flow timeouts. All edits,
|
||||
including deletion, remain drafts until **Save**. Status updates do not
|
||||
replace a dirty draft. A save conflict retains the draft; **Discard / reload**
|
||||
loads the latest configuration. Closing a dirty window offers explicit discard
|
||||
or continued editing.
|
||||
|
||||
The **Overview** tab shows a total VIP traffic plot and one backend table per
|
||||
listener, with health, RTT, open flows, totals, TX/RX rates and individual
|
||||
traffic plots. Plots switch between bytes and packets. Observations describe
|
||||
traffic seen by this device, not a network-wide aggregate.
|
||||
|
||||
An interactive diagram connects the service name through each listener to its
|
||||
node and target port. Drag to pan, Ctrl+scroll/pinch to zoom, and use **Fit**
|
||||
to restore the view. Click a backend in the diagram or table to highlight it
|
||||
and see its identity, address and measurement age. Green branches with moving
|
||||
dots indicate counter growth during the latest status sample; a blue ring
|
||||
marks the backend selected for new connections. Existing flows can therefore
|
||||
keep a different branch active. Idle open flows alone do not light up a branch;
|
||||
activity expires after five seconds without a fresh sample. Graph interaction
|
||||
never edits or saves configuration. The **Configuration** tab retains the
|
||||
explicit Save workflow. Addresses and ids are available in a collapsed detail.
|
||||
|
||||
## Delivery and recovery
|
||||
|
||||
The client agent selects the eligible backend with the lowest smoothed full
|
||||
round-trip latency through the actual encrypted data plane, including relays.
|
||||
Unknown or stale RTT is not treated as zero. Local backends have zero path RTT.
|
||||
A new flow is pinned to its selected backend; a faster RTT does not move it.
|
||||
This can concentrate new traffic on the closest node.
|
||||
|
||||
At local ingress, NAT replaces destination address/port and reserves an OS
|
||||
source port for an unambiguous reverse mapping. Return traffic passes normal
|
||||
network/peer source and destination admission **before** reverse NAT restores
|
||||
the virtual source address/port. Plugins and encrypted transit stay unchanged.
|
||||
A peer never gains permission to claim a VIP source. Fragment assembly is
|
||||
bounded, checksums are updated, packets are fragmented back to the overlay MTU,
|
||||
and ICMP errors have their quoted NAT headers restored. Fragment caches are
|
||||
scoped to network and authenticated sender, with idle expiry independent of
|
||||
further traffic.
|
||||
|
||||
For a backend on the client itself, the request is injected into the same TUN
|
||||
with the nonlocal VIP as source, and the host reply is reverse-translated at
|
||||
local ingress. The local backend therefore sees the VIP as source; remote
|
||||
backends see the client's ordinary overlay address. Firewalls and application
|
||||
bind addresses remain the administrator's responsibility.
|
||||
|
||||
Removing a backend/listener/VIP prevents new bindings while existing bindings
|
||||
continue until idle expiry. Established TCP connections are not migrated. A retransmitted empty SYN may
|
||||
choose another backend before a SYN-ACK has arrived; SYNs carrying application
|
||||
data are not retargeted. A new SYN sequence creates a new binding even if the
|
||||
client reuses its address/port tuple. TCP RST shortens binding retention. An unavailable UDP backend is replaced for subsequent
|
||||
datagrams; already sent datagrams are never replayed. Stateful UDP applications
|
||||
may need to re-establish their application session. Protocols that embed IP
|
||||
addresses or negotiate additional ports require their own configuration; no
|
||||
application payloads are parsed or rewritten.
|
||||
|
||||
Idle expiry defaults to 7200 seconds for TCP and 120 seconds for UDP. Shadow
|
||||
ports are quarantined for 120 seconds after retirement to reject late replies
|
||||
and avoid immediate reuse. Active and quarantined bindings share a bounded
|
||||
capacity. State and sockets are released on network stop or agent shutdown.
|
||||
A restart drops bindings; shared service configuration remains intact.
|
||||
|
||||
## Health
|
||||
|
||||
One probe exchange per second measures each used peer through the encrypted
|
||||
IP tunnel. The EWMA uses 7/8 of the previous sample and 1/8 of the new one.
|
||||
A probe expires after three seconds; two consecutive missing responses or a
|
||||
missing data path exclude the peer. Measurements expire after ten seconds.
|
||||
Probes continue during failure so recovery is automatic.
|
||||
|
||||
TCP connect checks default to on. Defaults are a three-second interval,
|
||||
two-second timeout, two failures to exclude and one success to recover.
|
||||
Checks are shared for the same network/node/address/TCP-port, run off the
|
||||
packet path with at most 32 concurrent connects, and are scheduled fairly.
|
||||
A 250ms worker tick services timers. Connect checks send no application data;
|
||||
they confirm port acceptance and may appear in server logs. They cannot prove
|
||||
application readiness after a successful handshake. Disabling port checks
|
||||
retains data-plane probes.
|
||||
|
||||
UDP uses encrypted path checks and explicit ICMP port errors. No arbitrary
|
||||
probe datagrams are injected into an unknown application protocol. Silence
|
||||
cannot prove failure of an arbitrary UDP application. An explicit unreachable
|
||||
port is excluded for ten seconds and then becomes eligible to retry.
|
||||
|
||||
## Shared state and addresses
|
||||
|
||||
Any authenticated member can create, edit or remove any VIP. Each field is a
|
||||
self-contained signed register: network, service id, field key, optional encoded
|
||||
value, Lamport clock, author, durable author counter and signature. Signing uses
|
||||
the separate `tsunagi-vip-field/v1\0` domain followed by postcard encoding of
|
||||
those fields with the signature vector empty. The encoding/order is a wire
|
||||
contract; the identity scheme and handshake transcript are unchanged.
|
||||
|
||||
Concurrent edits to different backend fields merge. A conflict on one field
|
||||
uses `(Lamport clock, author id, canonical bytes)` ordering, independent of wall
|
||||
time and arrival order. Listener generations prevent backend fields from an
|
||||
old removed listener reappearing when the port is recreated. Service deletion
|
||||
is a permanent signed tombstone for its id; recreation gets a fresh id.
|
||||
Missing snapshot fields are never deletion.
|
||||
|
||||
Saving validates the entire draft and checks the revision shown by the editor.
|
||||
The merged snapshot and author's counter commit together in SQLite before local
|
||||
routing changes or publication. Receivers verify and persist complete service
|
||||
snapshots before exposing them. Periodic rotating snapshots repair queue loss
|
||||
and returning replicas without requiring the creator online. Concurrent edits
|
||||
made before participants have seen each other can still merge after a save.
|
||||
A merged configuration exceeding listener/backend limits is reported as an
|
||||
error and excluded from routing rather than selected according to arrival order.
|
||||
|
||||
VIP allocation walks the agreed overlay range from a service-id-derived seed,
|
||||
excluding known node claims and other allocated VIPs. Nodes avoid known VIP
|
||||
reservations when choosing a new address. Names colliding with node names or
|
||||
another service are reported; node names take precedence and service ties use
|
||||
id order. A partition can temporarily have incomplete state; after merge an
|
||||
address or conflicting name's effective holder can change. Use DNS names.
|
||||
DNS TTL is 30 seconds; DNS does not migrate established connections or force
|
||||
applications to discard cached addresses. Existing NAT bindings retain their
|
||||
original VIP address while draining.
|
||||
|
||||
`Limits` bounds active services (64 by default), permanent deletion markers
|
||||
(1024), signed fields per active service (192), listeners (16), backends per
|
||||
listener (32), active TCP checks (32), flow/port reservations (4096) and incomplete
|
||||
IPv4 datagrams (64). A deleted service retains only its signed permanent
|
||||
marker; obsolete fields can be dropped because replay cannot override deletion.
|
||||
A full deletion history is an explicit error; protection is never silently
|
||||
evicted. Deleted services do not consume the active-service limit. Whole service
|
||||
snapshots must also fit the configured control frame limit. Corrupt mandatory VIP storage
|
||||
is a hard error on activation, never reset as a cache.
|
||||
|
||||
## Observations and verification
|
||||
|
||||
Status is this client's observation. Per VIP/listener/backend it reports the
|
||||
next selected backend, health reason, RTT and sample age, active bindings,
|
||||
user IP packet/byte counters, drops and UDP failovers. Probe bytes do not enter
|
||||
user traffic counters. The GUI and CLI watch derive rates from cumulative
|
||||
counters; counters reset on restart. These are IP forwarding counters, not
|
||||
application acknowledgements or guaranteed delivered-byte totals.
|
||||
|
||||
Automated tests cover field merging, stale drafts, signed deletion, tampering,
|
||||
network isolation, SQLite migration and transactional rollback, real port
|
||||
connect failure/recovery, RTT selection, UDP failover, local hairpin NAT,
|
||||
fragmentation/checksums, real encrypted TCP/UDP packet delivery and source
|
||||
admission, real CLI/DNS wiring and draft form rendering. In-memory TUN tests
|
||||
exercise packets, not an operating system TCP stack. Native desktop behavior,
|
||||
physical interfaces and cross-platform kernel/firewall behavior need live-host
|
||||
validation; automated tests do not establish those.
|
||||
+33
-10
@@ -26,8 +26,8 @@ system can hand us IP packets, and even that is behind a trait
|
||||
| `ManagedTunFactory` | `CAP_NET_ADMIN` | traffic actually reaches the OS |
|
||||
|
||||
`ManagedTunFactory` creates the interface and configures it; see
|
||||
*Provisioning the interface* below and *Privileges* in
|
||||
[../README.md](../README.md#privileges). With no capability the agent runs
|
||||
*Provisioning the interface* below and Privileges in
|
||||
[../dist/linux/README.md](../dist/linux/README.md). With no capability the agent runs
|
||||
with `--no-tun`: everything but the last hop into the kernel still works.
|
||||
|
||||
[boringtun]: https://docs.rs/boringtun
|
||||
@@ -92,8 +92,8 @@ and then recorded as a signed fact**, using the model in
|
||||
[sync-model.md](sync-model.md).
|
||||
|
||||
```text
|
||||
default range 10.13.37.0/24 (override with --ipv4-range)
|
||||
who decides the first member to claim; later ones adopt what they find
|
||||
default range stable pseudorandom /24 in 10/8, derived from NetworkId
|
||||
who decides all members derive the default; signed explicit ranges win
|
||||
who signs the claiming member, with its persistent device key
|
||||
where it is kept state.sqlite, and every replica that has seen it
|
||||
what a return costs nothing: the old address is reclaimed
|
||||
@@ -120,11 +120,34 @@ the same address at the same moment are resolved by a rule both compute
|
||||
identically — the lower endpoint id keeps it — and the loser simply allocates
|
||||
again with a higher version.
|
||||
|
||||
**The range is agreed, not configured per member.** `--ipv4-range` says what
|
||||
this agent would use; a network that has already settled on something else
|
||||
wins, and the agent adopts it. So the flag matters for whoever starts the
|
||||
network and is harmless afterwards. Pass `--ipv4-range none` for an IPv6-only
|
||||
overlay.
|
||||
**The range is agreed, not configured per member.** New networks use the
|
||||
same deterministic `/24` on every device, independently of which network a
|
||||
member activates first or whether any peers are reachable. The derivation
|
||||
is unchanged from the former fallback; network identity and signed record
|
||||
encoding are unchanged. Distinct network ids can still derive the same range.
|
||||
|
||||
`--ipv4-range` proposes an explicit range; existing signed ranges take
|
||||
precedence over the new derived default, including the former `10.13.37.0/24`.
|
||||
Among disagreeing non-default ranges the lowest author id still wins. Updated
|
||||
members propagate their adopted range in their signed claims; older releases
|
||||
still apply their original selection rule until those claims converge.
|
||||
`--ipv4-range none` disables local proposals, as before.
|
||||
|
||||
An agent reserves each active network's range before starting its runtime.
|
||||
Overlapping ranges, including nested blocks such as a `/24` inside a `/16`,
|
||||
cannot run simultaneously. A late authenticated state update that introduces
|
||||
an overlap suspends that network before publishing new allocations to plugins
|
||||
or the interface. Other networks continue running.
|
||||
|
||||
Status includes every blocking network's name, id and range. `tsng network
|
||||
start <name-or-id>` and the tray's switch button stop all blockers before
|
||||
starting the selected network. Suspended networks keep their startup request
|
||||
and resume when the blocker stops or is forgotten. A manual stop cancels that
|
||||
request even for a network already suspended. The user's preference order is
|
||||
stored locally, so the same selection wins at the next startup; it is never
|
||||
advertised or treated as a remote network setting. Local IPC uses wire version
|
||||
19 for the new reports: restart the agent and tray together after upgrading.
|
||||
The peer control protocol and signed record encoding do not change.
|
||||
|
||||
The agent puts the address on the interface itself, as soon as the network has
|
||||
agreed on it — no restart, and the interface is not recreated, which would
|
||||
@@ -184,7 +207,7 @@ QUIC DATAGRAM frames themselves cannot fragment.
|
||||
Reassembly is bounded and expires incomplete packets; one lost fragment loses
|
||||
one packet, without blocking unrelated traffic. The logical data payload limit
|
||||
is 64 KiB, with the relay envelope subtracted before it reaches the plugin.
|
||||
The new framing uses data ALPN `tsunagi/data/4`; both ends and intermediate
|
||||
The new framing uses data ALPN `tsunagi/data/5`; both ends and intermediate
|
||||
peers need the updated binary. Network identities and saved state do not change.
|
||||
See [protocol.md](protocol.md#the-data-plane-protocol) for the wire format.
|
||||
|
||||
|
||||
Generated
+27
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"nodes": {
|
||||
"nixpkgs": {
|
||||
"locked": {
|
||||
"lastModified": 1790769526,
|
||||
"narHash": "sha256-5AIVFLRcx8YAwbFTVPTs7YhwJH7JyHfAQsvXmpovcXI=",
|
||||
"owner": "NixOS",
|
||||
"repo": "nixpkgs",
|
||||
"rev": "c9fe7d12cd78d1adcd12dd15e24432dde5b155a0",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
"owner": "NixOS",
|
||||
"ref": "nixpkgs-unstable",
|
||||
"repo": "nixpkgs",
|
||||
"type": "github"
|
||||
}
|
||||
},
|
||||
"root": {
|
||||
"inputs": {
|
||||
"nixpkgs": "nixpkgs"
|
||||
}
|
||||
}
|
||||
},
|
||||
"root": "root",
|
||||
"version": 7
|
||||
}
|
||||
@@ -0,0 +1,51 @@
|
||||
{
|
||||
description = "tsunagi: serverless mesh VPN. A network name and a secret are all it takes.";
|
||||
|
||||
inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
|
||||
|
||||
outputs =
|
||||
{ self, nixpkgs }:
|
||||
let
|
||||
systems = [
|
||||
"x86_64-linux"
|
||||
"aarch64-linux"
|
||||
];
|
||||
forAllSystems = f: nixpkgs.lib.genAttrs systems (system: f nixpkgs.legacyPackages.${system});
|
||||
in
|
||||
{
|
||||
packages = forAllSystems (pkgs: rec {
|
||||
tsunagi = pkgs.callPackage ./nix/package.nix { };
|
||||
tsunagi-cli = pkgs.callPackage ./nix/package.nix { withTray = false; };
|
||||
default = tsunagi;
|
||||
});
|
||||
|
||||
overlays.default = final: _prev: {
|
||||
tsunagi = final.callPackage ./nix/package.nix { };
|
||||
tsunagi-cli = final.callPackage ./nix/package.nix { withTray = false; };
|
||||
};
|
||||
|
||||
nixosModules.default = import ./nix/module.nix self;
|
||||
|
||||
devShells = forAllSystems (
|
||||
pkgs:
|
||||
let
|
||||
trayPackage = self.packages.${pkgs.stdenv.hostPlatform.system}.tsunagi;
|
||||
in
|
||||
{
|
||||
default = pkgs.mkShell {
|
||||
inputsFrom = [ trayPackage ];
|
||||
packages = [
|
||||
pkgs.rustfmt
|
||||
pkgs.clippy
|
||||
]
|
||||
++ trayPackage.trayRuntimeLibs;
|
||||
# Cargo binaries have no installed wrapper. Expose runtime-loaded
|
||||
# libraries here too, while preserving the caller's library path.
|
||||
shellHook = ''
|
||||
export LD_LIBRARY_PATH="${pkgs.lib.makeLibraryPath trayPackage.trayRuntimeLibs}:/run/opengl-driver/lib''${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
|
||||
'';
|
||||
};
|
||||
}
|
||||
);
|
||||
};
|
||||
}
|
||||
+133
@@ -0,0 +1,133 @@
|
||||
flake:
|
||||
{
|
||||
config,
|
||||
lib,
|
||||
pkgs,
|
||||
...
|
||||
}:
|
||||
|
||||
let
|
||||
cfg = config.services.tsunagi;
|
||||
socket = "/run/tsunagi/agent.sock";
|
||||
in
|
||||
{
|
||||
options.services.tsunagi = {
|
||||
enable = lib.mkEnableOption "the tsunagi mesh agent";
|
||||
|
||||
package = lib.mkOption {
|
||||
type = lib.types.package;
|
||||
default =
|
||||
if cfg.tray.enable then
|
||||
flake.packages.${pkgs.stdenv.hostPlatform.system}.tsunagi
|
||||
else
|
||||
flake.packages.${pkgs.stdenv.hostPlatform.system}.tsunagi-cli;
|
||||
defaultText = lib.literalExpression "tsunagi.packages.\${system}.tsunagi (tsunagi-cli without the tray)";
|
||||
description = "The tsunagi package to run.";
|
||||
};
|
||||
|
||||
tray.enable = lib.mkEnableOption "the tray client, started with every graphical session";
|
||||
|
||||
users = lib.mkOption {
|
||||
type = lib.types.listOf lib.types.str;
|
||||
default = [ ];
|
||||
example = [ "alice" ];
|
||||
description = ''
|
||||
Users allowed to talk to the agent without sudo (`tsng status`, `tsng join`,
|
||||
the tray). They are added to the `tsunagi` group.
|
||||
'';
|
||||
};
|
||||
|
||||
protocols = lib.mkOption {
|
||||
type = lib.types.nullOr (lib.types.listOf lib.types.str);
|
||||
default = null;
|
||||
example = [
|
||||
"wg-quic"
|
||||
"wg"
|
||||
];
|
||||
description = "Packet protocols for `tsng up --protocol`, best first. Null keeps the agent's default.";
|
||||
};
|
||||
};
|
||||
|
||||
config = lib.mkIf cfg.enable {
|
||||
users.groups.tsunagi = { };
|
||||
users.users = {
|
||||
tsunagi = {
|
||||
isSystemUser = true;
|
||||
group = "tsunagi";
|
||||
home = "/var/lib/tsunagi";
|
||||
description = "tsunagi agent";
|
||||
};
|
||||
}
|
||||
// lib.genAttrs cfg.users (_: {
|
||||
extraGroups = [ "tsunagi" ];
|
||||
});
|
||||
|
||||
systemd.services.tsunagi = {
|
||||
description = "tsunagi mesh agent";
|
||||
documentation = [ "https://github.com/house-of-vanity/tsunagi" ];
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
wants = [ "network-online.target" ];
|
||||
after = [ "network-online.target" ];
|
||||
environment = {
|
||||
TSUNAGI_STATE_DIR = "/var/lib/tsunagi";
|
||||
TSUNAGI_CACHE_DIR = "/var/cache/tsunagi";
|
||||
TSUNAGI_CONTROL_SOCKET = socket;
|
||||
TSUNAGI_CONTROL_GROUP = "tsunagi";
|
||||
};
|
||||
serviceConfig = {
|
||||
ExecStart = lib.concatStringsSep " " (
|
||||
[ "${cfg.package}/bin/tsng up" ]
|
||||
++ lib.optional (cfg.protocols != null) "--protocol ${lib.concatStringsSep "," cfg.protocols}"
|
||||
);
|
||||
User = "tsunagi";
|
||||
Group = "tsunagi";
|
||||
AmbientCapabilities = [
|
||||
"CAP_NET_ADMIN"
|
||||
"CAP_NET_BIND_SERVICE"
|
||||
];
|
||||
CapabilityBoundingSet = [
|
||||
"CAP_NET_ADMIN"
|
||||
"CAP_NET_BIND_SERVICE"
|
||||
];
|
||||
StateDirectory = "tsunagi";
|
||||
StateDirectoryMode = "0700";
|
||||
CacheDirectory = "tsunagi";
|
||||
RuntimeDirectory = "tsunagi";
|
||||
RuntimeDirectoryMode = "0755";
|
||||
Restart = "on-failure";
|
||||
RestartSec = 5;
|
||||
};
|
||||
};
|
||||
|
||||
# Exit nodes: the agent looks for iptables only in fixed FHS directories.
|
||||
systemd.tmpfiles.rules = [
|
||||
"L+ /usr/bin/iptables - - - - ${pkgs.iptables}/bin/iptables"
|
||||
"L+ /usr/bin/ip6tables - - - - ${pkgs.iptables}/bin/ip6tables"
|
||||
];
|
||||
|
||||
# Names of mesh members are served locally and handed to the system resolver.
|
||||
services.resolved.enable = lib.mkDefault true;
|
||||
|
||||
# polkit decides by group, not capability, who may configure systemd-resolved.
|
||||
security.polkit.extraConfig = ''
|
||||
polkit.addRule(function(action, subject) {
|
||||
var allowed = [
|
||||
"org.freedesktop.resolve1.set-dns-servers",
|
||||
"org.freedesktop.resolve1.set-domains",
|
||||
"org.freedesktop.resolve1.set-default-route",
|
||||
"org.freedesktop.resolve1.revert"
|
||||
];
|
||||
if (allowed.indexOf(action.id) >= 0 && subject.isInGroup("tsunagi")) {
|
||||
return polkit.Result.YES;
|
||||
}
|
||||
});
|
||||
'';
|
||||
|
||||
environment.systemPackages = [ cfg.package ];
|
||||
environment.sessionVariables.TSUNAGI_CONTROL_SOCKET = socket;
|
||||
|
||||
environment.etc."xdg/autostart/tsunagi-tray.desktop" = lib.mkIf cfg.tray.enable {
|
||||
source = "${cfg.package}/share/applications/tsunagi-tray.desktop";
|
||||
};
|
||||
};
|
||||
}
|
||||
+101
@@ -0,0 +1,101 @@
|
||||
{
|
||||
withTray ? true,
|
||||
lib,
|
||||
rustPlatform,
|
||||
pkg-config,
|
||||
copyDesktopItems,
|
||||
wrapGAppsHook3,
|
||||
gtk3,
|
||||
libayatana-appindicator,
|
||||
libglvnd,
|
||||
libxkbcommon,
|
||||
vulkan-loader,
|
||||
wayland,
|
||||
xdotool,
|
||||
libx11,
|
||||
libxcursor,
|
||||
libxi,
|
||||
libxrandr,
|
||||
}:
|
||||
|
||||
let
|
||||
# The tray is an egui app that loads graphics, windowing and appindicator
|
||||
# libraries at run time, so they have to be put on its library path by hand.
|
||||
trayRuntimeLibs = [
|
||||
libayatana-appindicator
|
||||
libglvnd
|
||||
libxkbcommon
|
||||
vulkan-loader
|
||||
wayland
|
||||
libx11
|
||||
libxcursor
|
||||
libxi
|
||||
libxrandr
|
||||
];
|
||||
in
|
||||
rustPlatform.buildRustPackage {
|
||||
pname = if withTray then "tsunagi" else "tsunagi-cli";
|
||||
version = "0.1.0";
|
||||
|
||||
src = lib.cleanSource ../.;
|
||||
cargoLock.lockFile = ../Cargo.lock;
|
||||
|
||||
cargoBuildFlags = [
|
||||
"--bin"
|
||||
"tsng"
|
||||
]
|
||||
++ lib.optionals withTray [
|
||||
"--bin"
|
||||
"tsunagi-tray"
|
||||
];
|
||||
|
||||
# The suite opens real sockets and interfaces; it runs in CI instead.
|
||||
doCheck = false;
|
||||
|
||||
nativeBuildInputs = [
|
||||
pkg-config
|
||||
]
|
||||
++ lib.optionals withTray [
|
||||
copyDesktopItems
|
||||
wrapGAppsHook3
|
||||
];
|
||||
buildInputs = lib.optionals withTray [
|
||||
gtk3
|
||||
libayatana-appindicator
|
||||
xdotool
|
||||
];
|
||||
|
||||
dontWrapGApps = true;
|
||||
|
||||
# The development shell runs unwrapped Cargo binaries and needs the same
|
||||
# dynamically loaded libraries as the installed tray wrapper.
|
||||
passthru = { inherit trayRuntimeLibs; };
|
||||
|
||||
postInstall = ''
|
||||
install -Dm644 dist/linux/50-tsunagi-resolved.rules \
|
||||
"$out/share/doc/tsunagi/50-tsunagi-resolved.rules"
|
||||
''
|
||||
+ lib.optionalString withTray ''
|
||||
install -Dm644 dist/linux/tsunagi.svg \
|
||||
"$out/share/icons/hicolor/scalable/apps/tsunagi.svg"
|
||||
install -Dm644 dist/linux/tsunagi-tray.desktop \
|
||||
"$out/share/applications/tsunagi-tray.desktop"
|
||||
'';
|
||||
|
||||
postFixup = lib.optionalString withTray ''
|
||||
wrapProgram "$out/bin/tsunagi-tray" \
|
||||
"''${gappsWrapperArgs[@]}" \
|
||||
--prefix LD_LIBRARY_PATH : ${lib.makeLibraryPath trayRuntimeLibs}:/run/opengl-driver/lib
|
||||
'';
|
||||
|
||||
meta = {
|
||||
description = "Serverless mesh VPN: join a network with a name and a secret";
|
||||
homepage = "https://github.com/house-of-vanity/tsunagi";
|
||||
license = lib.licenses.wtfpl;
|
||||
mainProgram = "tsng";
|
||||
platforms = [
|
||||
"x86_64-linux"
|
||||
"aarch64-linux"
|
||||
];
|
||||
};
|
||||
}
|
||||
Loaded 100 of 183 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user