Compare commits

..
Author SHA1 Message Date
Aleksandr Bogomiakov 23af1a2d9d Add shared TCP and UDP virtual services
Provide named VIPs with independent protocol/port listeners and node/port
backends so applications can use one DNS endpoint without a separate load
balancer. Every authenticated member may create, edit or delete any service.
Client agents select the eligible backend with the lowest measured encrypted
data-plane RTT, including relayed paths, and pin existing flows. UDP recovers
on another backend after failure; established TCP is never replayed or moved.
An unanswered empty TCP SYN may retry another backend before establishment.

Keep signed multi-writer field registers in state/vip.rs. Independent backend
edits merge, same-field conflicts use Lamport/author/canonical ordering, and
listener generations prevent old backend entries returning on port recreation.
Permanent service tombstones protect replay while allowing obsolete fields to
be compacted. Active-service and deletion-history limits are separate. Commit
complete snapshots and the author's durable counter atomically before local
routing changes or publication. Restore counters from newer genuine author
updates learned from replicas after an old backup is restored.

Implement bounded client-side address/port NAT at overlay ingress and reverse
translation only after authenticated source/destination admission. Reserve OS
source ports to avoid direct-flow collisions and quarantine retired ports for
late replies. Handle a backend on the client itself, scoped fragment assembly,
checksums, re-fragmentation and quoted ICMP/PMTU headers. Plugins and encrypted
transit retain their existing contracts. Data-path probes measure RTT; bounded
fair TCP connect checks are enabled by default and configurable per listener.
Arbitrary UDP application readiness remains unknown; path checks and explicit
port errors drive eligibility without protocol-specific probe payloads.

Expose the shared draft API through local IPC, simple vip/listener/backend CLI
commands, JSON, editor-based atomic updates and watch output. Add a VIP button
on each network tile and a separate tray editor. Edits and removals stay local
until Save; stale revisions retain the draft, and closing a dirty window offers
explicit discard. Report local routing choices, health reasons, RTT age, active
and draining bindings, IP counters, rates, drops and UDP failovers. Extend DNS
with service names; keep numeric addresses secondary and preserve existing
bindings while VIP addresses or listeners change.

This changes control wire ALPN/version to tsunagi/ctrl/8 and local IPC to 22;
all members and local clients must upgrade together. Data ALPN remains 5.
SQLite schema 10 migrates existing mandatory state without changing identity,
secrets, network ids, derivation labels, handshake encoding or node records.
Crate versions and release tags are unchanged. No installed agent or tray was
replaced. Commands, limits and operational semantics are in docs/vip.md.

Checks on Linux in nix develop:
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets: 559 passed, 3 ignored
- git diff --cached --check

The full suite includes real encrypted TCP/UDP VIP traffic, backend-stop UDP
failover, real loopback TCP check recovery, source admission, signed merges,
stale/deleted state, SQLite migration and rollback, restart/author recovery,
network isolation, actual CLI/DNS processes, and headless text-entry/Save-button
interaction. Discovery and memory TUNs are the existing test substitutions;
authentication, transport, encryption and SQLite remain real.

Initial checks outside Nix failed for missing GTK pkg-config metadata, and an
initial CLI test inherited TSUNAGI_CONTROL_SOCKET. Final checks use the Nix
environment with that variable unset so tests address their temporary sockets.
Root/internet-dependent tests remain ignored. No live host TUN, native desktop
session, macOS or Windows validation was performed.
2026-10-21 15:40:45 +01:00
Aleksandr Bogomiakov 94318e7c16 Keep tray device metrics on one line
The device table sized handshake/RTT and packet columns from fixed samples.
Its own header exceeded the handshake sample, and six-digit counters wrapped
at the slash, making device and total rows taller and misaligning the plots.
Measure actual headers, displayed values and totals before drawing, retain
the widest values across frames, and disable wrapping inside table cells.
When a column grows, request a second egui layout pass so the grid's cached
positions settle before larger text is painted over the next column.
Keep the plot as the stretch column and horizontal scrolling for narrow
windows. This changes only tray presentation.

A headless rendering regression checks the actual text shapes for the
header, handshake age/control RTT, per-device packet counts and totals as
counts grow past five digits, reach billions, shrink again, and the table
gets a narrower container. It checks one-line text and non-overlap with the
next column; it failed on the original wrapped header. Existing resize and
long-relay tests remain in place.

Checks on Linux in nix develop:
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets, with inherited
  TSUNAGI_CONTROL_SOCKET unset so tests use their temporary sockets
- git diff --cached --check

The suite's root/internet-dependent tests remain ignored. No native GUI
session or checks on macOS/Windows were run; rendering was checked through
egui's headless output. No installed tray binary was replaced.
2026-10-14 23:37:38 +01:00
Aleksandr Bogomiakov 2b0c938487 Bound iroh path retries to prevent multi-connection OOM
A long-running rc.33 daemon reached OOM with roughly 22 GiB resident and
24 GiB swapped. Live inspection found 40 GiB and 10/20 GiB anonymous
container mappings; the active worker stack and tracing metadata identified
iroh's scheduled path-open retry loop. In iroh 1.2.0 every failed connection
requeues the same path, while each retry visits every connection again.
Separate control and data connections multiply the queue geometrically.

Vendor the published iroh 1.2.0 source and select it with a workspace Cargo
patch. Deduplicate retry admission and cap distinct pending paths at 64,
evicting the oldest before inserting a new candidate. Keep the current and
next retry batches bounded without combining control/data connections or
changing wire protocols, identity derivation, authentication or state.
Document source provenance, the live evidence and removal conditions.
Upstream PR 4551 merged on October 6, but inspection of the official 1.3.0
crate still shows the unconditional enqueue, so that upgrade alone is not
a fix. No version bump, release tag or installed-service change.

The default workspace suite includes the exact vendored admission module.
Three regressions failed with the original enqueue and pass with the fix:
repeated failures across connections, duplicate admission at capacity, and
10,000 distinct candidates with bounded length/capacity and FIFO eviction.
Transport integration tests continue using real endpoints and encryption.

Checks on Linux in nix develop:
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets, with only the inherited
  TSUNAGI_CONTROL_SOCKET removed from the test environment
- git diff --cached --check

Initial full-suite runs failed six DNS readiness checks because the inherited
system socket path overrode their temporary state directories. Clearing that
environment variable resolves those failures without changing application code.
Root/network-dependent ignored tests and other operating systems were not run.
2026-10-14 23:28:53 +01:00
Ultradesu 4e235cbac6 Keep device data paths on one line
The tray used a fixed sample for the data-path column, so longer relay names wrapped and made device rows uneven. Measure each rendered path and retain the widest value, then render path labels without wrapping. Replace direct/via words with embedded Phosphor route icons while preserving transport, relay name, hop count and explanatory tooltips. The app already ships the font; keep its setup in one function and verify registration alongside a long-relay-width regression test. No wire or stored-state changes. Checks: cargo fmt --all -- --check; cargo clippy --locked --workspace --all-targets -- -D warnings; cargo test --locked --workspace --all-targets.
2026-10-14 13:50:05 +01:00
Aleksandr Bogomiakov 2631eb6975 Protect Linux exit-node connections and tidy the device table
Using an exit node could destroy direct peer connections: the agent's UID
bypass protected outbound routing, but reverse-path checks look up replies
without that UID and selected the overlay default. A real-host A/B check
confirmed that relaxing only strict reverse-path filtering restored direct
connections and exit HTTP traffic; the diagnostic host changes were removed.

Keep strict filtering and protect the Linux underlay with a dedicated masked
mark bit, 0x40000000. Mark the dedicated service UID in OUTPUT, save only that
bit in conntrack and restore it before PREROUTING reverse-path checks. Install
IPv4/IPv6 mark and UID main-table bypass rules before Tailscale's priorities;
leave application-specific and exit-default routing after them. Preserve
foreign mark bits, main-table routes and foreign firewall/policy rules.

Protect the first inbound TCP/UDP packet using bounded, locally owned socket
addresses/ports from /proc and an actual socket match. Refresh socket protection
before sending private transport negotiations and on discovery ticks, without
firewall writes for an unchanged socket set. Remote announcements never become
OS arguments. A failed refresh leaves active exit routing in place. Install
protection before routes, remove routes before protection, roll back partial
activation, and remove only exact interface tags, including legacy UID rules.

Reject strict kernel rp_filter without src_valid_mark with an actionable error;
never weaken filtering or change host sysctls. Document mark-aware Netfilter
requirements and provide the trusted ip6tables path in the NixOS module.
Other platforms retain their existing exit implementation. Peer wire formats,
identity derivation, local IPC and persistent storage are unchanged.

Make the tray devices grid fill and shrink with the window, constrain the first
cell and give remaining width to traffic plots. Keep displayed end-to-end
protocol and its short encryption tooltip consistent, independent of carrier
protocol. Move handshake/session age into its own column beside the existing
iroh control RTT. Use stable per-device hover identities and concise tooltips;
missing protocols/measurements remain explicit rather than invented.

Validation on Linux through nix develop:
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets: 529 passed, 3 ignored;
  inherited TSUNAGI_CONTROL_SOCKET and TSUNAGI_CONTROL_GROUP were unset
- explicit privileged kernel regression: 1 passed, using disposable namespaces,
  real TCP/UDP IPv4/IPv6, strict kernel and Netfilter RPF, first inbound packets,
  dynamic UDP binding, application exit routing/IPv6 blocking, repeated apply
  and exact cleanup with foreign Tailscale-priority rules retained
- headless egui grid/scroll-area rendering at multiple sizes and control RTT,
  mark masks/order, passive socket parsing, partial failure and cleanup tests
- git diff --check

Earlier iterations caught rtnetlink family API and nonexhaustive initializer
errors, Clippy warnings, egui's implicit minimum column width, and kernel-test
fixture issues (wildcard echo source, blocked IPv6 neighbour discovery and a
closed child stdout pipe). Corrected before final checks; the stdout-only test
fix was checked with fmt, Clippy and the explicit kernel regression afterward.
Native tray rendering and macOS/Windows checks were not run. The running host
agent was not replaced or restarted. Requested release tag: v0.1.0-rc.32.
2026-10-14 10:40:30 +01:00
Aleksandr Bogomiakov 90ef5cefec Retire offline node claims with equal network-wide cleanup rights
Offline devices reserved names and IPv4 addresses indefinitely, leaving stale
members in the device history. Give every authenticated network participant
the same right to retire another device's observed signed claim without an
owner, quorum, permanent identity ban or secret change.

Add a separately signed retirement containing the target's original signed
claim as proof. Verify both signatures and network scopes, retain the strongest
proved cutoff per target with deterministic ties, and suppress claim versions
at or below it in membership, hostname ownership and IPv4 allocation. Keep the
old signed records and range agreement, and retain markers after a newer claim
returns so old replicas cannot resurrect obsolete reservations. A returning
device signs a newer claim, allocates against current holders and resets its
previous hostname priority rather than taking a reused name with an old time.

Commit local retirement and the requester's author counter atomically before
announcing it; receivers persist proofs before applying them. Bound history to
1024 targets by default, reject additional targets without evicting protection,
and treat corrupt or over-limit mandatory storage as an explicit error. Repeat
bounded marker batches with a per-session rotating cursor so queue pressure
cannot indefinitely starve later markers. Exclude retired claims and cached
hints from history-derived discovery without preventing a fresh authenticated
device from returning.

Expose Agent::forget_node through the shared local control API. CLI command
tsng network forget-node accepts exact network names, full ids or unique id
prefixes, and exact node hostnames, ids, prefixes or overlay IPv4 addresses.
Ambiguous selections change nothing. Recheck the observed version and refuse
self, online nodes, the selected exit node and absent or changed claims.
Add Forget with an identity confirmation to offline tray device rows. Widen
Show devices to 1440x640 with an 800x400 minimum to fit its contents.

Control ALPN/version advances to tsunagi/ctrl/7 and local IPC to 21; upgrade
all peers and restart the agent with matching CLI/tray clients. Data ALPN 5,
network identity derivation, handshake encoding and existing record signing
bytes remain unchanged. SQLite schema 9 adds retirement storage while retaining
identity, secrets, settings and signed claims. Document behavior and limits.

Validation on Linux through nix develop:
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets: 519 passed, 2 existing ignored
  tests; inherited TSUNAGI_CONTROL_SOCKET and TSUNAGI_CONTROL_GROUP were unset
- git diff --check and the built CLI forget-node help

Tests use real iroh authentication and SQLite. Cover restart, stale-replica
recovery, equal rights and network isolation, hostname reuse and return, twelve
marker batches through a two-message queue, malformed signatures and proofs,
invented future cutoffs, deterministic merge, capacity, migration and injected
counter-write rollback, selectors and cleanup through two real CLI processes.
Earlier iterations caught a private test API, a collapsible tray conditional
and a crash-based CLI test waiting for QUIC's idle timeout; corrected before
the final successful suite. Manual tray rendering, macOS/Windows, privileged
TUNs and public-network traversal were not run. The requested RC tag is
v0.1.0-rc.31; no push was requested.
2026-10-12 18:04:29 +01:00
Aleksandr Bogomiakov 13bdcaf98d Render protocol priority buttons with Phosphor icons
The tray protocol selection used ordinary Unicode up/down arrows for its
priority buttons. The default UI fonts lack these glyphs, so both buttons
rendered missing-glyph boxes. Use the regular Phosphor ARROW_UP and ARROW_DOWN
icons already installed in the application's proportional font fallback.

Validation on Linux through nix develop:
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets, with inherited control socket
  and group overrides removed: 509 passed, 2 existing ignored tests

No new tests were added for the two glyph substitutions. Manual tray rendering
and macOS/Windows checks were not performed. No release tag was requested for
this fix.
2026-10-12 17:19:53 +01:00
Aleksandr Bogomiakov d941cdac7a Change network protocol selections at runtime
Protocol selection previously belonged to tsng up and therefore to the
service command line. A blocked or degraded carrier could only be removed
by editing the service and restarting the whole agent. Make the ordered
selection a local per-network setting instead.

Add Agent::set_protocols and a local control request shared by CLI and tray.
Keep tsng protocols without arguments as the protocol catalog; naming a
network shows its selection and supplying a comma-separated list replaces
it. Accept exact names, full ids and unique prefixes using shared selector
rules; ambiguous selectors and invalid lists change nothing. An explicit
none selects control-only operation. Offline edits take the directory lock
and apply on the next start; live edits never write behind the daemon.

Register every built-in plugin and transport in the CLI. Treat up --protocol
as ordered initial defaults for networks without an override, allowing a
protocol absent from the unit to be enabled later. Retain the targeting of
unqualified startup options and allow explicitly scoped options to configure
initially disabled protocols. Persist NULL for inherited preferences and an
empty selection distinctly in SQLite schema 8; migration preserves existing
identities, secrets, signed records and counters.

Serialize changes with network lifecycle operations and commit preferences
before applying or announcing them. Retire only the affected network's data
links and E2E channels, cancel pending opens, and reject obsolete completion
results by attempt id. Clear peer data channels when authenticated protocol
capabilities change while retaining control sessions and topology. Replay
allocations and peer identities after asynchronously prepared plugins become
ready. A control-only start defers creation of the agent's single interface
until data is enabled; plugins receive a sink that can use it later. Automatic
carrier fallback continues to preserve E2E state; explicit policy changes
renegotiate it. Identity derivation and peer wire formats are unchanged.

Add a collapsible protocols row to each tray network tile, with a catalog
provided by the daemon, descriptions, immediately applied checkboxes and
preference-order buttons. Keep editing disabled until the worker refreshes
status so a second click cannot overwrite an edit using a stale snapshot.
Local IPC wire version is 20 for the new request and report fields; upgrade
and restart agent, CLI and tray together. Crate release versions remain unchanged; the requested RC tag is v0.1.0-rc.30.

Validation on Linux through nix develop:
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets: 509 passed, 2 existing ignored
  tests, with inherited TSUNAGI_CONTROL_SOCKET and TSUNAGI_CONTROL_GROUP removed

Tests use real iroh membership, SQLite, UDP, AWG and WireGuard encryption,
QUIC carriers and memory TUNs. Cover either endpoint-id role, full-size packet
delivery after switching, an unaffected second active network, saved selection
after restart, empty/invalid selections, first enablement without restart,
rapid changes racing old opens, live and offline CLI changes, startup-absent
protocols and preservation of signed state through migration. Earlier compile
and focused-test iterations caught initializer mistakes, a short test secret,
asynchronous plugin allocation readiness, and an inherited system socket
path; corrected before the successful final checks. Plain cargo was unavailable
in the host shell, so validation used the Nix toolchain. Manual tray interaction,
macOS/Windows, privileged host TUNs and public-network traversal were not run.
2026-10-12 16:19:18 +01:00
Aleksandr Bogomiakov e701986a39 Add ephemeral pairwise Amnezia transport and runtime fallback
Plain WG and QUIC can be blocked or severely degraded while the iroh control
connection still works. Add an awg carrier with one independent UDP socket,
WireGuard engine and randomized masking profile per network/peer attempt.
The smaller endpoint id negotiates fresh outer keys, PSK, header protection,
header ranges, padding and signature templates through private authenticated
control messages. Restart or fallback reentry generates a new profile;
ordinary rekeying preserves the current profile. No pair parameters enter
signed state, capabilities, persistent storage or public message events.
Private payloads and serialized control buffers redact and zeroize themselves.

Adapt the bounded packet masking engine from RunaRoo/AmneziaWG-RS at commit
51579a7d6e8882d1d8d6d00822f09e18a9423b73, retaining its Apache license and
provenance. Declare WTFPL AND Apache-2.0 in package metadata to cover both
new glue and the adapted source. Reuse boringtun instead of upstream's Node/global executors and
interface machinery. This is Tsunagi AWG wire version 1, with standard WG
headers restored before authentication; external Amnezia server compatibility
is not implemented. DNS/SIP signature generators vary fields instead of
replaying captures. Each attempt closes independently, bounds allocation,
queues, candidates and fragment assemblies, and supports the full 64 KiB
carrier contract using UDP packets at most 1200 bytes. Yield between fragment
batches so large messages do not starve local socket readers.

Keep carrier encryption separate from end-to-end tunnels so multihop relays
still forward opaque ciphertext. The system owns the sole interface, signed
addresses and admission. Plugins receive neither network secrets nor paths.
Expose a bounded opaque control sender at the transport boundary; core control
messages remain outside protocol crates. Optional STUN on the pair's actual
socket adds IPv4 public candidates in CLI direct/relay mode, with a finite
query deadline and an off/alternate-server option. Local mode remains offline.
Symmetric NAT, IPv6-only paths and arbitrary UDP blocking can still require
fallback; no AWG-over-QUIC or hidden AWG relay is introduced.

Default carrier order is wg, awg, wg-quic, tcp-tls. Advance the fallback cycle
after setup failures and sustained runtime loss, even after earlier cooldowns
expire. Authenticated delivery probes traverse the actual encrypted tunnel,
fit the interface MTU and never enter the TUN. Retire after a 20-second warmup
and at least 60% loss over at least 15 completed recent observations. Preserve
logical tunnels and control sessions while explicitly closing physical links,
including TCP reader/writer tasks. Retain the newest private probe settings
across independently queued control negotiation and inbound installation.
CLI/tray reports keep end-to-end protocol and actual carrier separate.

Control ALPN/version becomes tsunagi/ctrl/6 for the new private message;
upgrade all members together. Data ALPN/envelopes remain version 5. Network
identity derivation, transcript encoding, device identities, signed records,
saved network configuration and release versions are unchanged.

Validation on Linux through nix develop:
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets, with the inherited
  TSUNAGI_CONTROL_SOCKET override removed: 504 passed, 2 pre-existing ignored

Tests use real iroh membership, SQLite, UDP sockets and WireGuard encryption
with memory TUNs. Cover three independent pairs, full-size IP and carrier
frames, restart freshness/socket release, 80% carrier datagram loss, fallback
to real iroh, fresh retries with working delivery checks, unequal small MTUs,
malformed negotiations/profiles, role and resource bounds, probe forgery and
STUN parsing. Earlier compile/lint iterations and focused tests caught RNG
API/initializer mistakes, a retry assertion racing path installation, and a
large-fragment burst overflowing loopback buffers; corrected before these
successful checks. Normalized upstream license line endings for diff checks.
Public NAT/STUN connectivity, actual DPI effectiveness, WAN performance,
per-pair memory measurement, macOS/Windows, privileged TUN operations and
manual tray interaction were not exercised.
2026-10-11 13:52:10 +01:00
Aleksandr Bogomiakov ed28e86748 Derive default network ranges and switch overlapping networks
Every first network previously proposed 10.13.37.0/24. Independently created
networks therefore collided when a device joined more than one of them.
Use the existing stable NetworkId-derived /24 in 10/8 for every new network,
independent of activation order or peer availability. Legacy and explicit
signed ranges outrank the derived default; remaining range disagreements
still use author-id ordering. The derivation, identity labels, handshake and
signed record encoding stay unchanged. There is no creator, timestamp or vote.

Reserve ranges before starting network actors and serialise start, stop and
switching through the agent lifecycle lock. Refuse equal or nested overlaps
without changing a network's subnet locally. A late authenticated range that
collides suspends only its network before updating plugins or the interface.
Coalesced watch notifications let the agent clean up outside the network actor
and reconsider suspended networks when ranges change.

An explicit switch stops every blocker before activating the chosen network.
Suspended networks retain their startup request and resume whenever their
ranges become free, including switching from a wide range to a narrow one.
A manual stop cancels that request even for an already suspended network.
Persist preference order and enabling together in an SQLite transaction so
the user's selection survives restart; no database schema migration is needed.

CLI status and network listing show blocker names, ids, ranges and the start
command. Network start switches explicitly and reports suspended networks;
joining an overlapping network reports it as configured but inactive. The
tray adds conflict details, an amber icon, Switch to this network and Keep
stopped actions. Local IPC wire version is 19 for the extended reports, so
restart the agent and tray together. Release versions and peer wire formats
are unchanged. Update the user documentation and compatibility tests.

Validation on Linux through nix develop:
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets with TSUNAGI_CONTROL_SOCKET
  removed from the test environment: 486 passed, 2 pre-existing ignored tests
- final focused range_conflicts integration suite after making discovery
  fixtures deterministic: 6 passed

Earlier validation attempts caught outdated diagnostic assertions, a new
test helper scope error, and the inherited system control socket override.
Those were corrected before the successful runs. Plain cargo was unavailable
in the host shell, so validation used the Nix development toolchain. macOS,
Windows, privileged host TUN operations and manual tray interaction were not
exercised.
2026-10-11 11:58:27 +01:00
Aleksandr Bogomiakov 781e359775 Make tunnel and transport layers explicit in status
The tray called the end-to-end tunnel proto and the current route link, then put the tunnel handshake age beside the route. A valid wg tunnel over a tcp-tls transport consequently looked like contradictory state. The terminal status used similarly terse wording.\n\nThe tray now separates end-to-end from data path, renders direct paths as direct over TRANSPORT, keeps handshake or uptime beside the tunnel, and explains both layers in hover text. Terminal status uses the same end-to-end tunnel, data path, transport detail, and control path vocabulary. Routing, protocol selection, IPC, and wire behavior are unchanged.\n\nPresentation tests cover wg over tcp-tls, tcp-tls uptime, relayed and missing paths, a missing handshake, and offline devices. The WireGuard broadcast checksum test uses the current slice-array iterator spelling required by Clippy; its behavior is unchanged.\n\nChecks: cargo fmt --all -- --check; cargo clippy --locked --workspace --all-targets -- -D warnings; cargo test --locked --workspace --all-targets with TSUNAGI_CONTROL_SOCKET removed from the test environment so integration agents use their temporary control sockets. All passed on Linux.
2026-10-08 16:03:20 +01:00
Aleksandr Bogomiakov d5e02ec256 fixed gitignore 2026-10-08 15:35:38 +01:00
168 changed files with 51144 additions and 677 deletions

No files matched your search

+1
View File
@@ -1 +1,2 @@
/target
/.claude
Generated
+22 -2
View File
@@ -2806,8 +2806,6 @@ checksum = "791930b43c0d5973160d90a8f3894509f2b273430f5c5c73b668636d0287c5c0"
[[package]]
name = "iroh"
version = "1.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b2f8d1cfffc83efe39a1031aab423ce09cb8048071baba550508931e9a81ce46"
dependencies = [
"backon",
"blake3",
@@ -6241,11 +6239,13 @@ dependencies = [
"iroh",
"netwatch",
"nix 0.31.3",
"serde_json",
"simple-dns",
"tempfile",
"tokio",
"tracing",
"tracing-subscriber",
"tsunagi-awg",
"tsunagi-core",
"tsunagi-tcp-tls",
"tsunagi-wg-quic",
@@ -6253,6 +6253,26 @@ dependencies = [
"windows-service",
]
[[package]]
name = "tsunagi-awg"
version = "0.1.0"
dependencies = [
"boringtun",
"bytes",
"chacha20 0.9.1",
"iroh",
"postcard",
"rand 0.10.3",
"serde",
"tempfile",
"tokio",
"tracing",
"tsunagi-core",
"tsunagi-wg-quic",
"tsunagi-wireguard",
"zeroize",
]
[[package]]
name = "tsunagi-core"
version = "0.1.0"
+3 -1
View File
@@ -4,12 +4,14 @@
[workspace]
resolver = "3"
members = ["crates/*"]
exclude = ["vendor/mainline"]
exclude = ["vendor/mainline", "vendor/iroh"]
# Mainline 8.0.0 treats Windows UDP read timeouts as socket failures.
# Keep the small receive-loop correction reproducible until an upstream fix.
[patch.crates-io]
mainline = { path = "vendor/mainline" }
# Backport bounded path retries until an iroh release includes the OOM fix.
iroh = { path = "vendor/iroh" }
[workspace.package]
edition = "2024"
+66 -1
View File
@@ -81,9 +81,32 @@ tsng join -n home -s tsn1… # every other machine
tsng status # who is online, over which path
```
Every member gets an address in `10.13.37.0/24` and a name like `<host>.home`.
Every new network gets a stable pseudorandom `/24` in `10.0.0.0/8`, derived
from its network id. Its members agree on that range even when starting offline,
and each gets a name like `<host>.home`. Existing signed ranges are preserved.
If two configured networks have overlapping ranges, the agent runs only one.
`tsng status`, `tsng network` and the tray show which network blocks the
other. `tsng network start <name-or-id>` or the tray’s “Switch to this network”
button suspends the conflicting networks. They resume automatically when the
selected network stops; the selection survives restarting the agent. Explicitly
stopping a suspended network cancels its automatic resume.
Machines are called `tsunagi-<id>` until you rename them: `tsng id hostname laptop`.
### Virtual services
```sh
tsng vip create home api
tsng vip port add home api tcp/443 --backend server1:6443 --backend server2:7443
tsng vip status home api --watch
```
Use `api.home:443`. Each client selects the available backend with the lowest
measured data-plane RTT. A VIP can have multiple independent TCP and UDP ports.
Every member may edit it; the tray's **VIP** button opens a draft editor with
explicit Save. [Commands, health, metrics and limits](docs/vip.md).
### Exit nodes
```sh
@@ -97,6 +120,46 @@ The tray does the same with a checkbox. An exit node sees everything you send
through it, so use only members you trust.
[Details and per-OS notes](docs/exit-node.md).
The default data transport order is `wg,awg,wg-quic,tcp-tls`. AWG negotiates a
fresh masking profile and outer keys for each pair connection attempt, including
reentry after fallback. Sustained data loss triggers switching independently of
the iroh control connection. [Configuration and limits](docs/amnezia.md).
Change a network's protocol selection without editing its service or restarting:
```sh
tsng protocols # available protocols and settings
tsng protocols home # this device's choice for home
tsng protocols home awg,wg-quic # prefer AWG, retain QUIC as fallback
tsng protocols home none # control plane only
```
Exact network names, full ids and unique id prefixes work here. The choice is
local to this device, saved per network and applied immediately; an offline
change applies on the next start. In the tray, expand **protocols** on a network
tile to toggle protocols or change their preference order. Both endpoints need
a common protocol at the same wire version. `tsng up --protocol` supplies the
initial defaults for networks without a saved choice; it does not restrict
which built-in protocols can be enabled later.
Remove an obsolete offline device from a network's shared history:
```sh
tsng network forget-node home old-laptop
tsng network forget-node home 10.13.37.143
```
Networks accept exact names, full ids or unique id prefixes; nodes accept those
id selectors, exact hostnames or overlay IPv4 addresses. Ambiguous selectors
change nothing. Start the network first if it is stopped. In **Show devices**,
click **Forget** beside an offline node and confirm its identity.
Every member has the same cleanup right. The signed cleanup frees the observed
claim's name and IPv4 reservation, survives restarts and reaches offline replicas
when they reconnect. It does not ban the device: a newer signed claim can return.
Online nodes and the currently selected exit node cannot be forgotten.
[State and limits](docs/sync-model.md#offline-node-cleanup).
## Platforms
| Platform | Build | CLI `tsng` | Tray | Exit node: offer / use | Packages |
@@ -174,8 +237,10 @@ The [threat model](docs/threat-model.md) says what this does and doesn't protect
[Architecture](docs/architecture.md) ·
[Protocol](docs/protocol.md) ·
[WireGuard data plane](docs/wireguard.md) ·
[Pairwise Amnezia transport](docs/amnezia.md) ·
[Routing](docs/routing.md) ·
[Exit nodes](docs/exit-node.md) ·
[Virtual services](docs/vip.md) ·
[Threat model](docs/threat-model.md) ·
[Contributing](AGENTS.md)
+30
View File
@@ -0,0 +1,30 @@
[package]
name = "tsunagi-awg"
version = "0.1.0"
description = "Ephemeral pairwise Amnezia transport for Tsunagi."
edition.workspace = true
rust-version.workspace = true
license = "WTFPL AND Apache-2.0"
repository.workspace = true
[dependencies]
tsunagi-core = { path = "../tsunagi-core" }
tsunagi-wg-quic = { path = "../tsunagi-wg-quic" }
tsunagi-wireguard = { path = "../tsunagi-wireguard" }
iroh.workspace = true
tokio = { workspace = true, features = ["net", "time", "sync", "macros"] }
serde.workspace = true
postcard.workspace = true
bytes.workspace = true
tracing.workspace = true
rand = "0.10"
zeroize = { version = "1.9", features = ["derive"] }
boringtun = { version = "0.7.1", default-features = false }
chacha20 = "0.9"
[dev-dependencies]
tsunagi-core = { path = "../tsunagi-core", features = ["testing"] }
tempfile.workspace = true
[lints]
workspace = true
+201
View File
@@ -0,0 +1,201 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright 2026 Friderika Coulson (NanoBee)
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
+21
View File
@@ -0,0 +1,21 @@
# Amnezia packet engine
`src/masking.rs` is adapted from `src/core/amnezia.rs` in
https://github.com/RunaRoo/AmneziaWG-RS at
`51579a7d6e8882d1d8d6d00822f09e18a9423b73`, under Apache-2.0.
The upstream license is retained in `LICENSE-AmneziaWG-RS`.
The adaptation replaces configuration-file loading with bounded typed pair
profiles, updates RNG calls to rand 0.10, removes an unnecessary unwrap, and
does not embed upstream's Node, global executors, OS interface operations or
detached tasks. Noise/WireGuard is supplied by boringtun, already used by
Tsunagi. Each pair owns one socket and one cancellable packet-engine task.
This is Tsunagi AWG wire version 1, not a claim of compatibility with external
Amnezia servers. The masking engine substitutes message headers outside the
standard WireGuard state machine and restores them before Noise validation.
There are no remote file paths, shell hooks, or persisted pair profiles.
The adapted masking source remains Apache-2.0. Newly written Tsunagi code
follows the repository's WTFPL. This package declares both licenses in Cargo
metadata; the workspace license alone does not describe its copied source.
+16
View File
@@ -0,0 +1,16 @@
//! Pairwise, ephemeral Amnezia packet transport with independent end-to-end tunnels.
//! All negotiation travels over authenticated Tsunagi control sessions.
mod masking;
mod plugin;
pub mod profile;
mod session;
mod stun;
mod transport;
pub use plugin::AwgPlugin;
pub use transport::{AwgTransport, PairOverview};
/// Protocol identifier and default second-choice direct transport.
pub const AWG_PROTOCOL: &str = "awg";
/// Tsunagi AWG wire version, independent of software releases.
pub const AWG_WIRE_VERSION: u16 = 1;
+336
View File
@@ -0,0 +1,336 @@
// Adapted from RunaRoo/AmneziaWG-RS, commit 51579a7d6e8882d1d8d6d00822f09e18a9423b73.
// Apache-2.0; see LICENSE-AmneziaWG-RS and UPSTREAM.md.
#![allow(dead_code, missing_docs)]
// AmneziaWG configuration, header protection, and traffic shaping.
use bytes::{Bytes, BytesMut};
use chacha20::{
ChaCha20,
cipher::{KeyIvInit, StreamCipher},
};
use rand::RngExt;
pub const HEADER_PROTECTION_NONCE_SIZE: usize = 12;
pub const DEFAULT_UDP_WINDOW: usize = 500;
#[derive(Clone, Copy, Debug, Default, Eq, PartialEq)]
pub struct Range {
pub min: u32,
pub max: u32,
}
impl Range {
pub fn contains(&self, val: u32) -> bool {
val >= self.min && val <= self.max
}
pub fn random(&self) -> u32 {
if self.min >= self.max {
self.min
} else {
rand::rng().random_range(self.min..=self.max)
}
}
pub fn is_zero(&self) -> bool {
self.min == 0 && self.max == 0
}
}
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub enum AwgKind {
Initiation,
Response,
Cookie,
Transport,
}
impl AwgKind {
pub fn fixed_size(self) -> usize {
match self {
Self::Initiation => 148,
Self::Response => 92,
Self::Cookie => 64,
Self::Transport => 32,
}
}
fn standard_type(self) -> u32 {
match self {
Self::Initiation => 1,
Self::Response => 2,
Self::Cookie => 3,
Self::Transport => 4,
}
}
}
#[derive(Debug)]
pub struct DecodedPacket {
pub kind: AwgKind,
pub message: Bytes,
pub wire_len: usize,
}
pub struct AmneziaConfig {
pub jc: u32,
pub jmin: usize,
pub jmax: usize,
pub s1: usize,
pub s2: usize,
pub s3: usize,
pub s4: usize,
pub h1: Range,
pub h2: Range,
pub h3: Range,
pub h4: Range,
pub i1: String,
pub i2: String,
pub i3: String,
pub i4: String,
pub i5: String,
pub header_protection_key: Option<[u8; 32]>,
pub content_padding_addition: Range,
pub random_trailers: bool,
pub mtu: usize,
}
impl AmneziaConfig {
pub fn random_header(&self, range: &Range) -> u32 {
range.random()
}
pub fn random_padding(&self, len: usize) -> Vec<u8> {
let mut padding = vec![0u8; len];
if len > 0 {
rand::fill(&mut padding);
}
padding
}
fn shape(&self, kind: AwgKind) -> (usize, Range) {
match kind {
AwgKind::Initiation => (self.s1, self.h1),
AwgKind::Response => (self.s2, self.h2),
AwgKind::Cookie => (self.s3, self.h3),
AwgKind::Transport => (self.s4, self.h4),
}
}
pub fn encode(&self, kind: AwgKind, payload: Bytes, udp_window: usize) -> Bytes {
let (padding_len, _) = self.shape(kind);
let trailer_len = if kind != AwgKind::Transport && self.random_trailers {
let base = padding_len.saturating_add(payload.len());
let room = udp_window.max(DEFAULT_UDP_WINDOW).saturating_sub(base);
if room == 0 {
0
} else {
rand::rng().random_range(0..room)
}
} else {
0
};
let mut out = BytesMut::zeroed(padding_len + payload.len() + trailer_len);
if padding_len > 0 {
rand::fill(&mut out[..padding_len]);
}
out[padding_len..padding_len + payload.len()].copy_from_slice(&payload);
if trailer_len > 0 {
rand::fill(&mut out[padding_len + payload.len()..]);
}
if self.header_protection_key.is_some() {
let protected_len = if kind == AwgKind::Transport {
16
} else {
kind.fixed_size()
};
self.apply_header_cipher(&mut out, padding_len, protected_len);
}
out.freeze()
}
pub fn decode(&self, datagram: &Bytes) -> Option<DecodedPacket> {
if datagram.len() < 4 {
return None;
}
for kind in [
AwgKind::Initiation,
AwgKind::Response,
AwgKind::Cookie,
AwgKind::Transport,
] {
let (padding_len, headers) = self.shape(kind);
let minimum = padding_len + kind.fixed_size();
let size_ok = match kind {
AwgKind::Transport | AwgKind::Initiation => datagram.len() >= minimum,
AwgKind::Response | AwgKind::Cookie if self.random_trailers => {
datagram.len() >= minimum
}
AwgKind::Response | AwgKind::Cookie => datagram.len() == minimum,
};
if !size_ok || datagram.len() < padding_len + 4 {
continue;
}
let protected_header =
u32::from_le_bytes(datagram[padding_len..padding_len + 4].try_into().ok()?);
let candidate = if self.header_protection_key.is_some() {
protected_header ^ self.header_type_hash(datagram)?
} else {
protected_header
};
if !headers.contains(candidate) {
continue;
}
let message_len = match kind {
AwgKind::Transport | AwgKind::Initiation => datagram.len() - padding_len,
AwgKind::Response | AwgKind::Cookie => kind.fixed_size(),
};
let mut payload = BytesMut::from(&datagram[padding_len..padding_len + message_len]);
if self.header_protection_key.is_some() {
let protected_len = if kind == AwgKind::Transport {
16
} else {
kind.fixed_size()
};
self.apply_header_cipher_with_nonce(
&mut payload,
0,
protected_len,
&datagram[..HEADER_PROTECTION_NONCE_SIZE],
);
}
return Some(DecodedPacket {
kind,
message: payload.freeze(),
wire_len: datagram.len(),
});
}
None
}
pub fn prepare_transport_plaintext(&self, packet: Bytes, udp_window: usize) -> Bytes {
if packet.is_empty() {
return packet;
}
let requested = if !self.content_padding_addition.is_zero() {
self.content_padding_addition.random() as usize
} else if self.random_trailers {
let wire_without_padding = self.s4 + AwgKind::Transport.fixed_size() + packet.len();
let room = udp_window
.max(DEFAULT_UDP_WINDOW)
.saturating_sub(wire_without_padding);
if room == 0 {
0
} else {
rand::rng().random_range(0..room)
}
} else {
let aligned = (packet.len() + 15) & !15;
aligned.saturating_sub(packet.len())
};
let padding = self.clamp_padding_to_mtu(packet.len(), requested);
if padding == 0 {
return packet;
}
let mut out = BytesMut::with_capacity(packet.len() + padding);
out.extend_from_slice(&packet);
out.resize(packet.len() + padding, 0);
out.freeze()
}
pub fn transport_content_length(&self, plaintext: &[u8]) -> Option<usize> {
if plaintext.is_empty() || plaintext[0] == 0 {
return Some(0);
}
match plaintext[0] >> 4 {
4 if plaintext.len() >= 20 => {
let total = u16::from_be_bytes([plaintext[2], plaintext[3]]) as usize;
(total >= 20 && total <= plaintext.len()).then_some(total)
}
6 if plaintext.len() >= 40 => {
let payload = u16::from_be_bytes([plaintext[4], plaintext[5]]) as usize;
let total = 40 + payload;
(total <= plaintext.len()).then_some(total)
}
_ => None,
}
}
fn clamp_padding_to_mtu(&self, packet_len: usize, requested: usize) -> usize {
if self.mtu == 0 {
return requested;
}
let normalized = if packet_len > self.mtu {
packet_len % self.mtu
} else {
packet_len
};
requested.min(self.mtu.saturating_sub(normalized))
}
fn header_type_hash(&self, datagram: &Bytes) -> Option<u32> {
if datagram.len() < HEADER_PROTECTION_NONCE_SIZE {
return None;
}
let mut hash = [0u8; 4];
self.apply_keystream(&mut hash, &datagram[..HEADER_PROTECTION_NONCE_SIZE]);
Some(u32::from_le_bytes(hash))
}
fn apply_header_cipher(&self, packet: &mut [u8], offset: usize, len: usize) {
if packet.len() < HEADER_PROTECTION_NONCE_SIZE || packet.len() < offset + len {
return;
}
let nonce: [u8; HEADER_PROTECTION_NONCE_SIZE] =
match packet[..HEADER_PROTECTION_NONCE_SIZE].try_into() {
Ok(nonce) => nonce,
Err(_) => return,
};
self.apply_header_cipher_with_nonce(packet, offset, len, &nonce);
}
fn apply_header_cipher_with_nonce(
&self,
packet: &mut [u8],
offset: usize,
len: usize,
nonce: &[u8],
) {
if packet.len() < offset + len {
return;
}
self.apply_keystream(&mut packet[offset..offset + len], nonce);
}
fn apply_keystream(&self, data: &mut [u8], nonce: &[u8]) {
let Some(key) = self.header_protection_key.as_ref() else {
return;
};
if nonce.len() != HEADER_PROTECTION_NONCE_SIZE {
return;
}
let Ok(nonce) = <&[u8; HEADER_PROTECTION_NONCE_SIZE]>::try_from(nonce) else {
return;
};
let mut cipher = ChaCha20::new(key.into(), nonce.into());
cipher.apply_keystream(data);
}
}
impl Drop for AmneziaConfig {
fn drop(&mut self) {
use zeroize::Zeroize;
self.header_protection_key.zeroize();
}
}
+132
View File
@@ -0,0 +1,132 @@
//! End-to-end encryption remains independent of physical Amnezia pair links.
use crate::{AWG_PROTOCOL, AWG_WIRE_VERSION, AwgTransport};
use iroh::EndpointId;
use std::sync::Arc;
use tsunagi::dataplane::transport::SharedLink;
use tsunagi::dataplane::{IpPlugin, PluginCapability, PluginContext, PluginError, ProtocolOption};
use tsunagi::state::Ipv4Range;
use tsunagi::{BoxFuture, NetworkId};
use tsunagi_wg_quic::{NetworkOverview, WireguardConfig, WireguardPlugin};
/// AWG capability and an independent end-to-end WireGuard engine.
/// Pair transport keys/profiles never enter its durable end-to-end key store.
#[derive(Debug)]
pub struct AwgPlugin {
inner: Arc<WireguardPlugin>,
transport: Arc<AwgTransport>,
}
impl AwgPlugin {
/// Uses standard tunnel MTU/keepalive options; pair ports and profiles are ephemeral.
pub const OPTIONS: &'static [ProtocolOption] = &[
ProtocolOption {
key: "keepalive",
value: "SECONDS",
help: "end-to-end tunnel keepalive; 0 disables it",
default: Some("25"),
},
ProtocolOption {
key: "mtu",
value: "BYTES",
help: "maximum overlay packet size",
default: Some("1280"),
},
ProtocolOption {
key: "stun",
value: "HOST:PORT|off",
help: "UDP public mapping discovery; disabled in local reach mode",
default: Some("stun.l.google.com:19302"),
},
];
/// Opens the end-to-end engine, retaining separate pair transport state.
pub async fn open(
mut config: WireguardConfig,
transport: Arc<AwgTransport>,
) -> Result<Arc<Self>, PluginError> {
config.protocol = AWG_PROTOCOL;
Ok(Arc::new(Self {
inner: WireguardPlugin::open(config).await?,
transport,
}))
}
/// Reports end-to-end tunnel health and traffic counters.
pub fn overview(&self, network: NetworkId) -> Option<NetworkOverview> {
self.inner.overview(network)
}
}
impl IpPlugin for AwgPlugin {
fn protocol_id(&self) -> &str {
AWG_PROTOCOL
}
fn protocol_version(&self) -> u16 {
AWG_WIRE_VERSION
}
fn options(&self) -> &'static [ProtocolOption] {
Self::OPTIONS
}
fn attach(&self, context: PluginContext) {
self.inner.attach(context);
}
fn local_capability(
&self,
network: NetworkId,
) -> Result<Option<PluginCapability>, PluginError> {
self.inner.local_capability(network).map(|cap| {
cap.map(|mut cap| {
cap.protocol = AWG_PROTOCOL.into();
cap.version = AWG_WIRE_VERSION;
cap
})
})
}
fn on_network_activated(&self, network: NetworkId) {
self.inner.on_network_activated(network);
}
fn on_peer_capability(
&self,
network: NetworkId,
peer: EndpointId,
capability: &PluginCapability,
) -> Result<(), PluginError> {
if capability.protocol != AWG_PROTOCOL
|| capability.version != AWG_WIRE_VERSION
|| !capability.enabled
{
return Err(PluginError::Rejected("unsupported AWG capability".into()));
}
self.inner.on_peer_capability(network, peer, capability)?;
self.transport.allow_peer(network, peer);
Ok(())
}
fn on_address_allocation(
&self,
network: NetworkId,
range: Ipv4Range,
allocations: &[(EndpointId, std::net::Ipv4Addr)],
) {
self.inner
.on_address_allocation(network, range, allocations);
}
fn carry(&self, network: NetworkId, peer: EndpointId, packet: bytes::Bytes) -> bool {
self.inner.carry(network, peer, packet)
}
fn on_peer_link(&self, network: NetworkId, peer: EndpointId, link: SharedLink) {
self.inner.on_peer_link(network, peer, link);
}
fn on_peer_gone(&self, network: NetworkId, peer: EndpointId) {
self.transport.forget_peer(network, peer);
self.inner.on_peer_gone(network, peer);
}
fn on_network_deactivated(&self, network: NetworkId) {
self.transport.stop_network(network);
self.inner.on_network_deactivated(network);
}
fn on_network_forgotten(&self, network: NetworkId) {
self.inner.on_network_forgotten(network);
}
fn shutdown(&self) -> BoxFuture<'_, ()> {
Box::pin(async move {
self.transport.shutdown();
self.inner.shutdown().await;
})
}
}
+226
View File
@@ -0,0 +1,226 @@
//! Bounded profiles, regenerated for every pair connection attempt.
use crate::masking::{AmneziaConfig, AwgKind, Range};
use bytes::Bytes;
use serde::{Deserialize, Serialize};
/// An ephemeral masking profile. Secret fields are redacted and erased on drop.
#[derive(Clone, Serialize, Deserialize, zeroize::ZeroizeOnDrop)]
pub struct PairProfile {
id: [u8; 16],
prefix: u16,
headers: [[u32; 2]; 4],
header_key: [u8; 32],
pub(crate) psk: [u8; 32],
junk_count: u8,
junk_min: u16,
junk_max: u16,
padding: u16,
signature: u8,
}
impl std::fmt::Debug for PairProfile {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str("PairProfile([REDACTED])")
}
}
impl PairProfile {
/// Generates a new safe profile; no deterministic seed or durable state.
pub fn generate() -> Self {
let headers = std::array::from_fn(|i| {
let base = (i as u32) * 0x4000_0000 + rand::random_range(1024..0x3fff_0000);
[base, base + rand::random_range(64..1024)]
});
Self {
id: rand::random(),
prefix: rand::random_range(12..49),
headers,
header_key: rand::random(),
psk: rand::random(),
junk_count: rand::random_range(4..9),
junk_min: rand::random_range(24..65),
junk_max: rand::random_range(80..161),
padding: rand::random_range(16..65),
signature: rand::random_range(0..2),
}
}
/// Public attempt identifier, useful for distinguishing retries without revealing secrets.
pub fn id(&self) -> [u8; 16] {
self.id
}
/// Header key, for packet engines and protocol tests; never log this value.
pub fn header_key(&self) -> &[u8; 32] {
&self.header_key
}
/// Rejects invalid remote profiles before allocating packets or entering the codec.
pub fn validate(&self) -> Result<(), &'static str> {
if !(12..=48).contains(&self.prefix)
|| !(4..=8).contains(&self.junk_count)
|| self.junk_min < 24
|| self.junk_min > self.junk_max
|| self.junk_max > 160
|| self.padding > 64
|| self.signature > 1
|| self.header_key == [0; 32]
|| self.psk == [0; 32]
{
return Err("invalid AWG profile bounds");
}
for (i, [lo, hi]) in self.headers.iter().copied().enumerate() {
if lo < 5 || lo > hi || hi - lo > 1024 {
return Err("invalid AWG header range");
}
for [other_lo, other_hi] in &self.headers[..i] {
if lo <= *other_hi && *other_lo <= hi {
return Err("overlapping AWG header ranges");
}
}
}
Ok(())
}
fn config(&self) -> AmneziaConfig {
let range = |i: usize| Range {
min: self.headers[i][0],
max: self.headers[i][1],
};
AmneziaConfig {
jc: self.junk_count.into(),
jmin: self.junk_min.into(),
jmax: self.junk_max.into(),
s1: self.prefix.into(),
s2: self.prefix.into(),
s3: self.prefix.into(),
s4: self.prefix.into(),
h1: range(0),
h2: range(1),
h3: range(2),
h4: range(3),
i1: String::new(),
i2: String::new(),
i3: String::new(),
i4: String::new(),
i5: String::new(),
header_protection_key: Some(self.header_key),
content_padding_addition: Range {
min: 0,
max: self.padding.into(),
},
random_trailers: true,
mtu: 1050,
}
}
/// Masks one standard WireGuard message; invalid types and oversized packets are refused.
pub fn encode(&self, raw: &[u8]) -> Option<Bytes> {
let kind = match raw.get(..4)? {
[1, 0, 0, 0] => AwgKind::Initiation,
[2, 0, 0, 0] => AwgKind::Response,
[3, 0, 0, 0] => AwgKind::Cookie,
[4, 0, 0, 0] => AwgKind::Transport,
_ => return None,
};
if raw.len() < kind.fixed_size() || raw.len() > 1050 {
return None;
}
let config = self.config();
let range = match kind {
AwgKind::Initiation => config.h1,
AwgKind::Response => config.h2,
AwgKind::Cookie => config.h3,
AwgKind::Transport => config.h4,
};
let mut payload = raw.to_vec();
payload[..4].copy_from_slice(&range.random().to_le_bytes());
Some(config.encode(kind, Bytes::from(payload), 1200))
}
/// Unmasks a packet and restores standard headers before WireGuard validation.
pub fn decode(&self, wire: &[u8]) -> Option<Vec<u8>> {
if wire.len() > 1200 {
return None;
}
let decoded = self.config().decode(&Bytes::copy_from_slice(wire))?;
let (kind, len) = match decoded.kind {
AwgKind::Initiation => (1, 148),
AwgKind::Response => (2, 92),
AwgKind::Cookie => (3, 64),
AwgKind::Transport => (4, decoded.message.len()),
};
let mut message = decoded.message.get(..len)?.to_vec();
message[..4].copy_from_slice(&(kind as u32).to_le_bytes());
Some(message)
}
pub(crate) fn pad(&self, packet: Bytes) -> Bytes {
self.config().prepare_transport_plaintext(packet, 1200)
}
/// Generates structured signature packets with fresh fields, never static captures.
/// These target the other Tsunagi peer, not DNS or SIP services.
pub fn signature_packets(&self) -> Vec<Vec<u8>> {
if self.signature == 0 {
// A complete DNS query. Counts, label lengths and qtype remain valid.
let label = format!("n{:016x}", rand::random::<u64>());
let mut packet = Vec::from(rand::random::<u16>().to_be_bytes());
packet.extend_from_slice(&[1, 0, 0, 1, 0, 0, 0, 0, 0, 0]);
packet.push(label.len() as u8);
packet.extend_from_slice(label.as_bytes());
packet.extend_from_slice(b"\x07example\x03net\0\0");
packet.push(if rand::random() { 1 } else { 28 });
packet.extend_from_slice(&[0, 1]);
vec![packet]
} else {
let id = rand::random::<u64>();
vec![format!("OPTIONS sip:n{id:x}@example.net SIP/2.0\r\nVia: SIP/2.0/UDP example.net;branch=z9hG4bK{id:x}\r\nMax-Forwards: 70\r\nFrom: <sip:a@example.net>;tag={id:x}\r\nTo: <sip:n{id:x}@example.net>\r\nCall-ID: {id:x}@example.net\r\nCSeq: 1 OPTIONS\r\nContent-Length: 0\r\n\r\n").into_bytes()]
}
}
pub(crate) fn junk_packets(&self) -> Vec<Vec<u8>> {
(0..self.junk_count)
.map(|_| {
let mut packet = vec![0; rand::random_range(self.junk_min..=self.junk_max).into()];
rand::fill(&mut packet[..]);
packet
})
.collect()
}
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
#[test]
fn unsafe_peer_profiles_are_rejected_before_the_codec() {
let valid = PairProfile::generate();
let mut bad = valid.clone();
bad.prefix = 11;
assert!(bad.validate().is_err());
let mut bad = valid.clone();
bad.headers[1] = bad.headers[0];
assert!(bad.validate().is_err());
let mut bad = valid.clone();
bad.junk_max = u16::MAX;
assert!(bad.validate().is_err());
let mut bad = valid.clone();
bad.padding = u16::MAX;
assert!(bad.validate().is_err());
let mut bad = valid.clone();
bad.psk = [0; 32];
assert!(bad.validate().is_err());
let mut bad = valid.clone();
bad.signature = 2;
assert!(bad.validate().is_err());
}
#[test]
fn signature_templates_have_consistent_framing_and_fresh_fields() {
let mut profile = PairProfile::generate();
profile.signature = 0;
let dns = profile.signature_packets().remove(0);
assert_eq!(&dns[2..12], &[1, 0, 0, 1, 0, 0, 0, 0, 0, 0]);
let mut at = 12;
while dns[at] != 0 {
at += 1 + usize::from(dns[at]);
}
assert_eq!(at + 5, dns.len());
assert_eq!(&dns[dns.len() - 2..], &[0, 1]);
profile.signature = 1;
let sip = profile.signature_packets().remove(0);
assert!(sip.starts_with(b"OPTIONS sip:"));
assert!(sip.ends_with(b"Content-Length: 0\r\n\r\n"));
assert_ne!(sip, profile.signature_packets().remove(0));
}
}
+402
View File
@@ -0,0 +1,402 @@
//! One cancellable actor and UDP socket per pair. All payload and health frames
//! are inside the Amnezia-masked authenticated WireGuard tunnel.
use crate::profile::PairProfile;
use boringtun::noise::{Tunn, TunnResult};
use bytes::Bytes;
use iroh::EndpointId;
use std::collections::{HashMap, VecDeque};
use std::net::{Ipv4Addr, SocketAddr};
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};
use tokio::net::UdpSocket;
use tokio::sync::{mpsc, watch};
use tsunagi::dataplane::transport::{PacketLink, TransportError};
use tsunagi::{BoxFuture, NetworkId};
pub(crate) const MAX_PAYLOAD: usize = tsunagi::config::MAX_DATA_DATAGRAM;
const CHUNK: usize = 700;
const QUEUE: usize = 64;
const WINDOW: Duration = Duration::from_secs(20);
pub(crate) struct PairLink {
network: NetworkId,
peer: EndpointId,
pub(crate) attempt: [u8; 16],
tx: mpsc::Sender<Bytes>,
rx: tokio::sync::Mutex<mpsc::Receiver<Bytes>>,
stop: watch::Sender<bool>,
closed: AtomicBool,
remote: Mutex<Option<SocketAddr>>,
reason: Mutex<Option<String>>,
}
impl std::fmt::Debug for PairLink {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("AwgPair")
.field("network", &self.network)
.field("peer", &self.peer)
.finish_non_exhaustive()
}
}
impl PairLink {
pub(crate) fn has_authenticated_path(&self) -> bool {
match self.remote.lock() {
Ok(remote) => remote.is_some(),
Err(_) => false,
}
}
}
impl PacketLink for PairLink {
fn network(&self) -> NetworkId {
self.network
}
fn peer(&self) -> EndpointId {
self.peer
}
fn max_datagram_size(&self) -> usize {
MAX_PAYLOAD
}
fn send(&self, payload: Bytes) -> Result<(), TransportError> {
if self.is_closed() {
return Err(TransportError::Closed);
}
if payload.len() > MAX_PAYLOAD {
return Err(TransportError::TooLarge {
size: payload.len(),
limit: MAX_PAYLOAD,
});
}
self.tx
.try_send(payload)
.map_err(|_| TransportError::Other("AWG send queue full or closed".into()))
}
fn recv(&self) -> BoxFuture<'_, Option<Bytes>> {
Box::pin(async move {
let mut stop = self.stop.subscribe();
if *stop.borrow() {
return None;
}
let mut rx = self.rx.lock().await;
tokio::select! {biased; _=stop.changed()=>None,packet=rx.recv()=>packet}
})
}
fn closed(&self) -> BoxFuture<'_, ()> {
Box::pin(async move {
let mut stop = self.stop.subscribe();
if !*stop.borrow() {
let _ = stop.changed().await;
}
})
}
fn is_closed(&self) -> bool {
self.closed.load(Ordering::Acquire) || *self.stop.borrow()
}
fn close(&self) {
self.stop.send_replace(true);
}
fn failure_reason(&self) -> Option<String> {
match self.reason.lock() {
Ok(reason) => reason.clone(),
Err(_) => None,
}
}
fn path_description(&self) -> String {
match self.remote.lock() {
Ok(addr) => match *addr {
Some(addr) => format!("awg UDP via {addr}"),
None => "awg UDP, establishing".into(),
},
Err(_) => "awg UDP".into(),
}
}
}
impl Drop for PairLink {
fn drop(&mut self) {
self.stop.send_replace(true);
}
}
pub(crate) fn start(
network: NetworkId,
peer: EndpointId,
socket: UdpSocket,
tunn: Tunn,
profile: PairProfile,
candidates: Vec<SocketAddr>,
parent: watch::Receiver<bool>,
) -> Arc<PairLink> {
let (tx, rx) = mpsc::channel(QUEUE);
let (in_tx, in_rx) = mpsc::channel(QUEUE);
let (stop, _) = watch::channel(false);
let link = Arc::new(PairLink {
network,
peer,
attempt: profile.id(),
tx,
rx: tokio::sync::Mutex::new(in_rx),
stop: stop.clone(),
closed: AtomicBool::new(false),
remote: Mutex::new(None),
reason: Mutex::new(None),
});
let weak = Arc::downgrade(&link);
tokio::spawn(async move {
let mut engine = Engine {
stop: stop.subscribe(),
socket,
tunn,
profile,
candidates,
remote: None,
inbox: in_tx,
rx,
counter: rand::random(),
sequence: 0,
pending: VecDeque::new(),
samples: VecDeque::new(),
assemblies: HashMap::new(),
started: Instant::now(),
last_authenticated: Instant::now(),
};
engine.run(parent, &weak).await;
if let Some(link) = weak.upgrade() {
link.closed.store(true, Ordering::Release);
link.stop.send_replace(true);
}
});
link
}
struct Assembly {
created: Instant,
count: u16,
chunks: Vec<Option<Bytes>>,
}
struct Engine {
stop: watch::Receiver<bool>,
socket: UdpSocket,
tunn: Tunn,
profile: PairProfile,
candidates: Vec<SocketAddr>,
remote: Option<SocketAddr>,
inbox: mpsc::Sender<Bytes>,
rx: mpsc::Receiver<Bytes>,
counter: u64,
sequence: u64,
pending: VecDeque<(u64, Instant)>,
samples: VecDeque<bool>,
assemblies: HashMap<u64, Assembly>,
started: Instant,
last_authenticated: Instant,
}
enum ResultOwned {
Wire(Vec<u8>),
Plain(Vec<u8>),
Done,
}
fn own(result: TunnResult<'_>) -> ResultOwned {
match result {
TunnResult::WriteToNetwork(p) => ResultOwned::Wire(p.to_vec()),
TunnResult::WriteToTunnelV4(p, _) => ResultOwned::Plain(p.to_vec()),
_ => ResultOwned::Done,
}
}
fn ip(payload: &[u8]) -> Bytes {
let mut packet = vec![0; 20];
packet[0] = 0x45;
packet[8] = 64;
packet[9] = 253;
packet[12..16].copy_from_slice(&Ipv4Addr::new(192, 0, 2, 1).octets());
packet[16..20].copy_from_slice(&Ipv4Addr::new(192, 0, 2, 2).octets());
packet[2..4].copy_from_slice(&((20 + payload.len()) as u16).to_be_bytes());
let sum = packet
.as_chunks::<2>()
.0
.iter()
.map(|b| u32::from(u16::from_be_bytes([b[0], b[1]])))
.sum::<u32>();
let sum = (sum & 0xffff) + (sum >> 16);
packet[10..12].copy_from_slice(&(!(sum as u16)).to_be_bytes());
packet.extend_from_slice(payload);
Bytes::from(packet)
}
impl Engine {
async fn run(&mut self, mut stop: watch::Receiver<bool>, link: &std::sync::Weak<PairLink>) {
if *self.stop.borrow() || *stop.borrow() {
return;
}
let mut buffer = [0; 1201];
let mut scratch = [0; 4096];
let mut timer = tokio::time::interval(Duration::from_millis(250));
timer.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
let mut last_probe = Instant::now() - Duration::from_secs(2);
loop {
tokio::select! {
biased;
_=stop.changed()=>break,
_=self.stop.changed()=>break,
_=timer.tick()=>{
let result=own(self.tunn.update_timers(&mut scratch));self.process(result,None).await;
if last_probe.elapsed()>=Duration::from_secs(1) {
last_probe=Instant::now();self.sequence=self.sequence.wrapping_add(1);
self.pending.push_back((self.sequence,Instant::now()));
let mut probe=vec![1];probe.extend_from_slice(&self.sequence.to_be_bytes());
probe.resize(713,0);rand::fill(&mut probe[9..]);self.encrypt(&probe).await;
}
while self.pending.front().is_some_and(|(_,at)|at.elapsed()>=Duration::from_secs(3)) {
self.pending.pop_front();self.samples.push_back(false);
}
while self.samples.len()>20 {self.samples.pop_front();}
self.assemblies.retain(|_,a|a.created.elapsed()<Duration::from_secs(5));
if self.started.elapsed()>WINDOW && (self.last_authenticated.elapsed()>Duration::from_secs(10)
|| (self.samples.len()>=15&&self.samples.iter().filter(|ok|!**ok).count()*100>=self.samples.len()*60)) {
if let Some(link)=link.upgrade() && let Ok(mut reason)=link.reason.lock() {*reason=Some("AWG data delivery checks: persistent loss or timeout".into());}
break;
}
}
packet=self.socket.recv_from(&mut buffer)=>{
if let Ok((len,source))=packet && let Some(message)=self.profile.decode(&buffer[..len]) {
let result=own(self.tunn.decapsulate(None,&message,&mut scratch));
if matches!(result,ResultOwned::Plain(_)) {
self.remote=Some(source);self.last_authenticated=Instant::now();
if let Some(link)=link.upgrade() && let Ok(mut remote)=link.remote.lock(){*remote=Some(source);}
}
self.process(result,Some(source)).await;
for _ in 0..16 {
let result=own(self.tunn.decapsulate(None,&[],&mut scratch));
if matches!(result,ResultOwned::Done){break;}
self.process(result,Some(source)).await;
}
}
}
packet=self.rx.recv()=>{
let Some(packet)=packet else{break;};
let count=packet.len().max(1).div_ceil(CHUNK) as u16;
self.counter=self.counter.wrapping_add(1);
for (index,chunk) in packet.chunks(CHUNK).enumerate() {
let mut payload=vec![3];payload.extend_from_slice(&self.counter.to_be_bytes());
payload.extend_from_slice(&(index as u16).to_be_bytes());payload.extend_from_slice(&count.to_be_bytes());
payload.extend_from_slice(chunk);self.encrypt(&payload).await;
// Large logical datagrams must let local readers and other
// pairs drain their bounded socket/command queues.
if (index + 1).is_multiple_of(16) {tokio::task::yield_now().await;}
}
}
}
}
}
async fn encrypt(&mut self, payload: &[u8]) {
let packet = self.profile.pad(ip(payload));
let mut scratch = [0; 4096];
let result = own(self.tunn.encapsulate(&packet, &mut scratch));
self.process(result, None).await;
}
async fn wire(&self, raw: &[u8], target: Option<SocketAddr>) {
let Some(wire) = self.profile.encode(raw) else {
return;
};
let targets: Vec<_> = target
.or(self.remote)
.map(|a| vec![a])
.unwrap_or_else(|| self.candidates.clone());
for addr in targets {
if raw.first() == Some(&1) {
for signature in self.profile.signature_packets() {
let _ = self.send_wire(&signature, addr).await;
}
for junk in self.profile.junk_packets() {
let _ = self.send_wire(&junk, addr).await;
}
}
let _ = self.send_wire(&wire, addr).await;
}
}
async fn send_wire(&self, bytes: &[u8], address: SocketAddr) {
let mut stop = self.stop.clone();
if *stop.borrow() {
return;
}
tokio::select! {biased; _=stop.changed()=>{}, _=tokio::time::timeout(Duration::from_millis(100),self.socket.send_to(bytes,address))=>{}}
}
async fn process(&mut self, result: ResultOwned, target: Option<SocketAddr>) {
match result {
ResultOwned::Wire(raw) => self.wire(&raw, target).await,
ResultOwned::Plain(packet) => {
let Some(payload) = packet.get(20..) else {
return;
};
let Some(seq) = payload
.get(1..9)
.and_then(|b| b.try_into().ok())
.map(u64::from_be_bytes)
else {
return;
};
match payload.first() {
Some(1) => {
let mut pong = vec![2];
pong.extend_from_slice(&seq.to_be_bytes());
pong.resize(713, 0);
let packet = self.profile.pad(ip(&pong));
let mut scratch = [0; 4096];
if let ResultOwned::Wire(raw) =
own(self.tunn.encapsulate(&packet, &mut scratch))
{
self.wire(&raw, target).await;
}
}
Some(2) => {
if let Some(index) =
self.pending.iter().position(|(pending, _)| *pending == seq)
{
self.pending.remove(index);
self.samples.push_back(true);
}
}
Some(3) => self.fragment(seq, payload),
_ => {}
}
}
ResultOwned::Done => {}
}
}
fn fragment(&mut self, id: u64, payload: &[u8]) {
let Some(header) = payload.get(9..13) else {
return;
};
let index = u16::from_be_bytes([header[0], header[1]]);
let count = u16::from_be_bytes([header[2], header[3]]);
let chunk = &payload[13..];
if count == 0
|| usize::from(count) > MAX_PAYLOAD.div_ceil(CHUNK)
|| index >= count
|| chunk.len() > CHUNK
{
return;
}
if !self.assemblies.contains_key(&id) && self.assemblies.len() >= 8 {
return;
}
let assembly = self.assemblies.entry(id).or_insert_with(|| Assembly {
created: Instant::now(),
count,
chunks: vec![None; count.into()],
});
if assembly.count != count {
return;
}
assembly.chunks[usize::from(index)] = Some(Bytes::copy_from_slice(chunk));
if assembly.chunks.iter().all(Option::is_some)
&& let Some(assembly) = self.assemblies.remove(&id)
{
let mut packet = Vec::new();
for chunk in assembly.chunks.into_iter().flatten() {
packet.extend_from_slice(&chunk);
}
if packet.len() <= MAX_PAYLOAD {
let _ = self.inbox.try_send(Bytes::from(packet));
}
}
}
}
+101
View File
@@ -0,0 +1,101 @@
//! Optional, bounded server-reflexive discovery on the pair's own UDP socket.
//! A STUN answer is a candidate only; WireGuard authentication verifies a path.
use std::net::{IpAddr, Ipv4Addr, SocketAddr};
use std::time::Duration;
use tokio::net::UdpSocket;
const COOKIE: [u8; 4] = [0x21, 0x12, 0xa4, 0x42];
pub(crate) async fn mapped(socket: &UdpSocket, server: &str) -> Option<SocketAddr> {
tokio::time::timeout(Duration::from_millis(1200), async {
let address = tokio::net::lookup_host(server)
.await
.ok()?
.find(SocketAddr::is_ipv4)?;
let transaction: [u8; 12] = rand::random();
let mut request = [0; 20];
request[..2].copy_from_slice(&[0, 1]);
request[4..8].copy_from_slice(&COOKIE);
request[8..].copy_from_slice(&transaction);
socket.send_to(&request, address).await.ok()?;
let mut bytes = [0; 1024];
loop {
let (len, source) = socket.recv_from(&mut bytes).await.ok()?;
if source == address
&& let Some(mapped) = parse(&bytes[..len], &transaction)
{
return Some(mapped);
}
}
})
.await
.ok()
.flatten()
}
fn parse(bytes: &[u8], transaction: &[u8; 12]) -> Option<SocketAddr> {
if bytes.len() < 20
|| bytes[..2] != [1, 1]
|| bytes[4..8] != COOKIE
|| &bytes[8..20] != transaction
{
return None;
}
let declared = u16::from_be_bytes(bytes[2..4].try_into().ok()?) as usize;
if !declared.is_multiple_of(4) || declared != bytes.len() - 20 {
return None;
}
let mut at = 20;
while at + 4 <= bytes.len() {
let kind = u16::from_be_bytes(bytes[at..at + 2].try_into().ok()?);
let len = u16::from_be_bytes(bytes[at + 2..at + 4].try_into().ok()?) as usize;
at += 4;
let attribute = bytes.get(at..at.checked_add(len)?)?;
if kind == 0x0020 && attribute.len() == 8 && attribute[0..2] == [0, 1] {
let port = u16::from_be_bytes([attribute[2] ^ COOKIE[0], attribute[3] ^ COOKIE[1]]);
let ip = Ipv4Addr::new(
attribute[4] ^ COOKIE[0],
attribute[5] ^ COOKIE[1],
attribute[6] ^ COOKIE[2],
attribute[7] ^ COOKIE[3],
);
if port == 0 || ip.is_unspecified() || ip.is_multicast() || ip.is_broadcast() {
return None;
}
return Some(SocketAddr::new(IpAddr::V4(ip), port));
}
at = at.checked_add((len + 3) & !3)?;
}
None
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
#[tokio::test]
async fn mapping_uses_the_actual_socket_and_rejects_unrelated_or_truncated_answers() {
let server = UdpSocket::bind((Ipv4Addr::LOCALHOST, 0)).await.unwrap();
let address = server.local_addr().unwrap();
let socket = UdpSocket::bind((Ipv4Addr::LOCALHOST, 0)).await.unwrap();
let expected = socket.local_addr().unwrap();
let task = tokio::spawn(async move {
let mut request = [0; 128];
let (len, source) = server.recv_from(&mut request).await.unwrap();
assert_eq!(len, 20);
assert_eq!(source, expected);
let tx: [u8; 12] = request[8..20].try_into().unwrap();
let mut reply = request[..20].to_vec();
reply[..4].copy_from_slice(&[1, 1, 0, 12]);
reply.extend_from_slice(&[0, 0x20, 0, 8, 0, 1]);
let port = source.port().to_be_bytes();
reply.extend_from_slice(&[port[0] ^ COOKIE[0], port[1] ^ COOKIE[1]]);
for (i, b) in Ipv4Addr::LOCALHOST.octets().into_iter().enumerate() {
reply.push(b ^ COOKIE[i]);
}
assert!(parse(&reply, &[0; 12]).is_none());
for len in 0..reply.len() {
assert!(parse(&reply[..len], &tx).is_none());
}
server.send_to(&reply, source).await.unwrap();
});
assert_eq!(mapped(&socket, &address.to_string()).await, Some(expected));
task.await.unwrap();
}
}
+720
View File
@@ -0,0 +1,720 @@
//! Private pair negotiation. No secret profiles in capabilities or durable state.
use crate::{
AWG_PROTOCOL, AWG_WIRE_VERSION,
profile::PairProfile,
session::{self, PairLink},
};
use boringtun::{
noise::Tunn,
x25519::{PublicKey, StaticSecret},
};
use iroh::EndpointId;
use serde::{Deserialize, Serialize};
use std::collections::{HashMap, HashSet};
use std::net::{Ipv4Addr, SocketAddr};
use std::sync::{Arc, Mutex, OnceLock, Weak};
use std::time::Duration;
use tokio::net::UdpSocket;
use tokio::sync::{mpsc, oneshot, watch};
use tsunagi::dataplane::transport::TransportPayload;
use tsunagi::dataplane::transport::{
ControlSender, InboundLink, PacketLink, PacketTransport, SharedLink, TransportError,
};
use tsunagi::{BoxFuture, NetworkId};
type Pair = (NetworkId, EndpointId);
#[derive(Serialize, Deserialize)]
enum Negotiation {
Offer {
version: u16,
profile: PairProfile,
key: [u8; 32],
#[serde(deserialize_with = "bounded_addresses")]
addresses: Vec<SocketAddr>,
},
Answer {
attempt: [u8; 16],
key: [u8; 32],
#[serde(deserialize_with = "bounded_addresses")]
addresses: Vec<SocketAddr>,
},
}
fn bounded_addresses<'de, D: serde::Deserializer<'de>>(
deserializer: D,
) -> Result<Vec<SocketAddr>, D::Error> {
struct Visitor;
impl<'de> serde::de::Visitor<'de> for Visitor {
type Value = Vec<SocketAddr>;
fn expecting(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str("at most 32 UDP candidates")
}
fn visit_seq<A: serde::de::SeqAccess<'de>>(
self,
mut sequence: A,
) -> Result<Self::Value, A::Error> {
if sequence.size_hint().is_some_and(|n| n > 32) {
return Err(serde::de::Error::custom("too many UDP candidates"));
}
let mut addresses = Vec::with_capacity(sequence.size_hint().unwrap_or(0).min(32));
while let Some(address) = sequence.next_element()? {
if addresses.len() == 32 {
return Err(serde::de::Error::custom("too many UDP candidates"));
}
addresses.push(address);
}
Ok(addresses)
}
}
deserializer.deserialize_seq(Visitor)
}
struct Pending {
profile: PairProfile,
key: StaticSecret,
socket: UdpSocket,
reply: oneshot::Sender<Result<SharedLink, TransportError>>,
}
struct Inner {
stopped: watch::Sender<bool>,
local: EndpointId,
bind: Ipv4Addr,
stun: Option<String>,
busy: Mutex<HashSet<Pair>>,
control: OnceLock<Arc<dyn ControlSender>>,
allowed: Mutex<HashSet<Pair>>,
pending: Mutex<HashMap<Pair, Pending>>,
links: Mutex<HashMap<Pair, Arc<PairLink>>>,
inbound: mpsc::Sender<InboundLink>,
port_by_pair: Mutex<HashMap<Pair, u16>>,
}
/// Public diagnostics for one pair. No key or masking parameters are exposed.
#[derive(Debug, Clone)]
pub struct PairOverview {
/// Other endpoint's authenticated identity.
pub peer: EndpointId,
/// Fresh connection-attempt identifier.
pub attempt: [u8; 16],
/// Last known UDP port; None if diagnostics race with pair retirement.
pub local_port: Option<u16>,
/// Actual transport path, if established.
pub path: String,
/// Whether this attempt has been retired.
pub closed: bool,
}
/// A bounded set of independently configured pair engines.
pub struct AwgTransport {
inner: Arc<Inner>,
commands: mpsc::Sender<(NetworkId, EndpointId, Negotiation)>,
incoming: tokio::sync::Mutex<mpsc::Receiver<InboundLink>>,
}
impl std::fmt::Debug for AwgTransport {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("AwgTransport")
.field("local", &self.inner.local)
.finish_non_exhaustive()
}
}
fn lock<T>(mutex: &Mutex<T>) -> std::sync::MutexGuard<'_, T> {
match mutex.lock() {
Ok(g) => g,
Err(p) => p.into_inner(),
}
}
fn error(text: &str) -> TransportError {
TransportError::Other(text.into())
}
async fn addresses(inner: &Inner, socket: &UdpSocket) -> Result<Vec<SocketAddr>, TransportError> {
let local = socket
.local_addr()
.map_err(|_| error("AWG socket address unavailable"))?;
let mut candidates = if inner.bind.is_loopback() {
vec![local]
} else {
tsunagi_wireguard::local_ip_candidates()
.into_iter()
.filter(|ip| ip.is_ipv4())
.take(31)
.map(|ip| SocketAddr::new(ip, local.port()))
.collect()
};
if let Some(server) = &inner.stun
&& let Some(mapped) = crate::stun::mapped(socket, server).await
&& !candidates.contains(&mapped)
{
candidates.push(mapped);
}
if candidates.is_empty() {
return Err(error("AWG has no UDP address candidates"));
}
Ok(candidates)
}
fn validate_endpoint(key: &[u8; 32], addresses: &[SocketAddr]) -> Result<(), TransportError> {
if *key == [0; 32]
|| addresses.is_empty()
|| addresses.len() > 32
|| addresses.iter().any(|address| {
address.port() == 0
|| !address.is_ipv4()
|| address.ip().is_unspecified()
|| address.ip().is_multicast()
|| matches!(address.ip(), std::net::IpAddr::V4(ip) if ip.is_broadcast())
})
{
return Err(error("invalid AWG endpoint"));
}
Ok(())
}
impl AwgTransport {
/// Starts the coordinator. Production binds wildcard IPv4; tests use loopback.
pub fn new(local: EndpointId, bind: Ipv4Addr) -> Arc<Self> {
Self::new_with_stun(local, bind, None)
}
/// Enables an optional STUN server (host:port) for this pair's own socket.
/// Local-only callers pass None; failure still permits local UDP candidates.
pub fn new_with_stun(local: EndpointId, bind: Ipv4Addr, stun: Option<String>) -> Arc<Self> {
let (commands, mut rx) = mpsc::channel(64);
let (inbound, incoming) = mpsc::channel(64);
let (stopped, _) = watch::channel(false);
let inner = Arc::new(Inner {
stopped,
local,
bind,
stun,
busy: Mutex::default(),
control: OnceLock::new(),
allowed: Mutex::default(),
pending: Mutex::default(),
links: Mutex::default(),
inbound,
port_by_pair: Mutex::default(),
});
let weak = Arc::downgrade(&inner);
let mut stopped = inner.stopped.subscribe();
tokio::spawn(async move {
while let Some((network, peer, message)) =
tokio::select! { biased; _=stopped.changed()=>None, message=rx.recv()=>message }
{
let Some(inner) = weak.upgrade() else {
break;
};
if !lock(&inner.allowed).contains(&(network, peer))
|| !lock(&inner.busy).insert((network, peer))
{
continue;
}
tokio::spawn(async move {
let mut stop = inner.stopped.subscribe();
tokio::select! {biased; _=stop.changed()=>{}, _=async {
if let Err(err)=receive(&inner,(network,peer),message).await {tracing::debug!(%err,"AWG pair negotiation rejected");}
}=>{}}
lock(&inner.busy).remove(&(network, peer));
});
}
});
Arc::new(Self {
inner,
commands,
incoming: tokio::sync::Mutex::new(incoming),
})
}
/// Registers a peer after its authenticated capability has been validated.
pub fn allow_peer(&self, network: NetworkId, peer: EndpointId) {
let mut allowed = lock(&self.inner.allowed);
if !*self.inner.stopped.borrow() && allowed.len() < 128 {
allowed.insert((network, peer));
}
}
/// Retires a pair when its network membership session goes away.
pub fn forget_peer(&self, network: NetworkId, peer: EndpointId) {
let pair = (network, peer);
lock(&self.inner.allowed).remove(&pair);
lock(&self.inner.pending).remove(&pair);
lock(&self.inner.port_by_pair).remove(&pair);
if let Some(link) = lock(&self.inner.links).remove(&pair) {
link.close();
}
}
/// Stops all pair engines in this network.
pub fn stop_network(&self, network: NetworkId) {
let peers: Vec<_> = lock(&self.inner.allowed)
.iter()
.filter(|(n, _)| *n == network)
.map(|(_, p)| *p)
.collect();
for peer in peers {
self.forget_peer(network, peer);
}
}
/// Stops all sockets and coordinator work without waiting for remote peers.
pub fn shutdown(&self) {
self.inner.stopped.send_replace(true);
lock(&self.inner.allowed).clear();
lock(&self.inner.pending).clear();
for (_, link) in lock(&self.inner.links).drain() {
link.close();
}
lock(&self.inner.port_by_pair).clear();
}
/// Receives a successfully negotiated and authenticated inbound pair link.
pub async fn next_inbound(&self) -> Option<InboundLink> {
let mut stopped = self.inner.stopped.subscribe();
if *stopped.borrow() {
return None;
}
let mut incoming = self.incoming.lock().await;
tokio::select! {biased; _=stopped.changed()=>None, link=incoming.recv()=>link}
}
/// Takes a public snapshot of this network's pair attempts.
pub fn overview(&self, network: NetworkId) -> Vec<PairOverview> {
let links = lock(&self.inner.links);
let ports = lock(&self.inner.port_by_pair);
links
.iter()
.filter(|((n, _), _)| *n == network)
.map(|((_, peer), link)| PairOverview {
peer: *peer,
attempt: link.attempt,
local_port: ports.get(&(network, *peer)).copied(),
path: link.path_description(),
closed: link.is_closed(),
})
.collect()
}
}
impl PacketTransport for AwgTransport {
fn name(&self) -> &str {
AWG_PROTOCOL
}
fn as_any(&self) -> &dyn std::any::Any {
self
}
fn attach_control(&self, control: Arc<dyn ControlSender>) {
let _ = self.inner.control.set(control);
}
fn on_control(
&self,
network: NetworkId,
peer: EndpointId,
protocol: &str,
payload: &[u8],
) -> Result<(), TransportError> {
if protocol != AWG_PROTOCOL || !lock(&self.inner.allowed).contains(&(network, peer)) {
return Err(TransportError::Declined(protocol.into()));
}
if payload.len() > 4096 {
return Err(error("AWG negotiation too large"));
}
let message: Negotiation =
postcard::from_bytes(payload).map_err(|_| error("malformed AWG negotiation"))?;
match &message {
Negotiation::Offer {
version,
profile,
key,
addresses,
} => {
if *version != AWG_WIRE_VERSION || peer.as_bytes() >= self.inner.local.as_bytes() {
return Err(error("invalid AWG offer role/version"));
}
profile.validate().map_err(error)?;
validate_endpoint(key, addresses)?;
}
Negotiation::Answer { key, addresses, .. } => validate_endpoint(key, addresses)?,
}
self.commands
.try_send((network, peer, message))
.map_err(|_| error("AWG negotiation queue full"))
}
fn open<'a>(
&'a self,
network: NetworkId,
peer: EndpointId,
protocol: &'a str,
) -> BoxFuture<'a, Result<SharedLink, TransportError>> {
Box::pin(async move {
let pair = (network, peer);
if protocol != AWG_PROTOCOL
|| self.inner.local.as_bytes() >= peer.as_bytes()
|| !lock(&self.inner.allowed).contains(&pair)
{
return Err(TransportError::Declined(protocol.into()));
}
let control = self
.inner
.control
.get()
.ok_or_else(|| error("AWG control plane unavailable"))?;
let socket = UdpSocket::bind((self.inner.bind, 0))
.await
.map_err(|_| error("AWG UDP bind failed"))?;
let profile = PairProfile::generate();
let attempt = profile.id();
let key = StaticSecret::from(rand::random::<[u8; 32]>());
let public = PublicKey::from(&key).to_bytes();
let message = Negotiation::Offer {
version: AWG_WIRE_VERSION,
profile: profile.clone(),
key: public,
addresses: addresses(&self.inner, &socket).await?,
};
let (reply, rx) = oneshot::channel();
if lock(&self.inner.pending).len() >= 128 {
return Err(error("too many AWG pair negotiations"));
}
lock(&self.inner.pending).insert(
pair,
Pending {
profile,
key,
socket,
reply,
},
);
// A cancellation guard also retires partially started attempts.
let mut guard = AttemptGuard {
inner: Arc::downgrade(&self.inner),
pair,
attempt,
completed: false,
};
let bytes =
postcard::to_stdvec(&message).map_err(|_| error("AWG offer encoding failed"))?;
control
.send(network, peer, AWG_PROTOCOL, TransportPayload::new(bytes))
.await?;
let result = tokio::time::timeout(Duration::from_secs(15), rx)
.await
.map_err(|_| error("AWG negotiation/handshake timed out"))?
.map_err(|_| error("AWG attempt cancelled"))?;
guard.completed = result.is_ok();
result
})
}
}
struct AttemptGuard {
inner: Weak<Inner>,
pair: Pair,
attempt: [u8; 16],
completed: bool,
}
impl Drop for AttemptGuard {
fn drop(&mut self) {
if let Some(inner) = self.inner.upgrade() {
let mut pending = lock(&inner.pending);
if pending
.get(&self.pair)
.is_some_and(|p| p.profile.id() == self.attempt)
{
pending.remove(&self.pair);
}
drop(pending);
if !self.completed
&& let Some(link) = lock(&inner.links).get(&self.pair)
&& link.attempt == self.attempt
{
link.close();
}
}
}
}
async fn receive(
inner: &Arc<Inner>,
pair: Pair,
message: Negotiation,
) -> Result<(), TransportError> {
if !lock(&inner.allowed).contains(&pair) {
return Err(error("AWG peer no longer authenticated"));
}
match message {
Negotiation::Offer {
profile,
key,
addresses: candidates,
..
} => {
if let Some(link) = lock(&inner.links).get(&pair) {
if link.attempt == profile.id() {
return Err(error("duplicate AWG offer"));
}
link.close();
}
let socket = UdpSocket::bind((inner.bind, 0))
.await
.map_err(|_| error("AWG UDP bind failed"))?;
let local_port = socket
.local_addr()
.map_err(|_| error("AWG socket address unavailable"))?
.port();
let local_key = StaticSecret::from(rand::random::<[u8; 32]>());
let public = PublicKey::from(&local_key).to_bytes();
let answer = Negotiation::Answer {
attempt: profile.id(),
key: public,
addresses: addresses(inner, &socket).await?,
};
let tunn = Tunn::new(
local_key,
PublicKey::from(key),
Some(profile.psk),
Some(1),
rand::random(),
None,
);
let link = session::start(
pair.0,
pair.1,
socket,
tunn,
profile,
candidates,
inner.stopped.subscribe(),
);
if !lock(&inner.allowed).contains(&pair) {
link.close();
return Err(error("AWG membership ended"));
}
lock(&inner.links).insert(pair, link.clone());
lock(&inner.port_by_pair).insert(pair, local_port);
let control = inner
.control
.get()
.ok_or_else(|| error("AWG control unavailable"))?;
if let Err(err) = control
.send(
pair.0,
pair.1,
AWG_PROTOCOL,
TransportPayload::new(
postcard::to_stdvec(&answer)
.map_err(|_| error("AWG answer encoding failed"))?,
),
)
.await
{
link.close();
return Err(err);
}
publish_when_up(Arc::downgrade(inner), pair, link, None);
}
Negotiation::Answer {
attempt,
key,
addresses: candidates,
} => {
if !lock(&inner.pending)
.get(&pair)
.is_some_and(|p| p.profile.id() == attempt)
{
return Err(error("stale AWG answer"));
}
let pending = lock(&inner.pending)
.remove(&pair)
.ok_or_else(|| error("AWG attempt cancelled"))?;
let local_port = pending
.socket
.local_addr()
.map_err(|_| error("AWG socket address unavailable"))?
.port();
let tunn = Tunn::new(
pending.key,
PublicKey::from(key),
Some(pending.profile.psk),
Some(1),
rand::random(),
None,
);
let link = session::start(
pair.0,
pair.1,
pending.socket,
tunn,
pending.profile,
candidates,
inner.stopped.subscribe(),
);
if !lock(&inner.allowed).contains(&pair) {
link.close();
return Err(error("AWG membership ended"));
}
lock(&inner.links).insert(pair, link.clone());
lock(&inner.port_by_pair).insert(pair, local_port);
publish_when_up(Arc::downgrade(inner), pair, link, Some(pending.reply));
}
}
Ok(())
}
fn publish_when_up(
inner: Weak<Inner>,
pair: Pair,
link: Arc<PairLink>,
reply: Option<oneshot::Sender<Result<SharedLink, TransportError>>>,
) {
tokio::spawn(async move {
let mut reply = reply;
let result = tokio::time::timeout(Duration::from_secs(14), async {
loop {
if link.is_closed() || reply.as_ref().is_some_and(oneshot::Sender::is_closed) {
return Err(error("AWG pair closed during handshake"));
}
if link.has_authenticated_path() {
return Ok(());
}
tokio::time::sleep(Duration::from_millis(25)).await;
}
})
.await;
if !matches!(result, Ok(Ok(()))) {
link.close();
if let Some(reply) = reply.take() {
let _ = reply.send(Err(error("AWG authenticated data path timed out")));
}
return;
}
let Some(inner) = inner.upgrade() else {
link.close();
return;
};
if !lock(&inner.allowed).contains(&pair) {
link.close();
return;
}
if link.is_closed()
|| !lock(&inner.links)
.get(&pair)
.is_some_and(|current| current.attempt == link.attempt)
{
link.close();
return;
}
let shared: SharedLink = link.clone();
if let Some(reply) = reply {
if reply.send(Ok(shared)).is_err() {
link.close();
}
} else if inner
.inbound
.try_send(InboundLink {
network: pair.0,
peer: pair.1,
protocol: AWG_PROTOCOL.into(),
link: shared,
})
.is_err()
{
link.close();
}
});
}
impl Drop for AwgTransport {
fn drop(&mut self) {
self.shutdown();
}
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
#[tokio::test]
async fn concurrent_admission_is_bounded_across_networks_and_retirement_releases_slots() {
let identity = tsunagi::identity::DeviceIdentity::generate();
let transport = AwgTransport::new(identity.endpoint_id(), Ipv4Addr::LOCALHOST);
let peer = tsunagi::identity::DeviceIdentity::generate().endpoint_id();
std::thread::scope(|scope| {
for thread in 0..8u8 {
let transport = &transport;
scope.spawn(move || {
for item in 0..32u8 {
let mut id = [0; 32];
id[0] = thread;
id[1] = item;
transport.allow_peer(NetworkId::from_bytes(id), peer);
}
});
}
});
assert_eq!(lock(&transport.inner.allowed).len(), 128);
let network = lock(&transport.inner.allowed).iter().next().unwrap().0;
transport.stop_network(network);
let new = NetworkId::from_bytes([255; 32]);
transport.allow_peer(new, peer);
assert!(lock(&transport.inner.allowed).contains(&(new, peer)));
assert_eq!(lock(&transport.inner.allowed).len(), 128);
transport.shutdown();
assert!(lock(&transport.inner.allowed).is_empty());
}
#[tokio::test]
async fn private_negotiation_bounds_roles_and_membership_before_starting_engines() {
let mut ids = [
tsunagi::identity::DeviceIdentity::generate().endpoint_id(),
tsunagi::identity::DeviceIdentity::generate().endpoint_id(),
];
ids.sort_by_key(|id| *id.as_bytes());
let peer = ids[0];
let local = ids[1];
let network = NetworkId::from_bytes([7; 32]);
let transport = AwgTransport::new(local, Ipv4Addr::LOCALHOST);
let offer = |version, key, addresses| Negotiation::Offer {
version,
profile: PairProfile::generate(),
key,
addresses,
};
let address = SocketAddr::from((Ipv4Addr::LOCALHOST, 12345));
let encode = |message: &Negotiation| postcard::to_stdvec(message).unwrap();
let valid = encode(&offer(AWG_WIRE_VERSION, [1; 32], vec![address]));
assert!(
transport
.on_control(network, peer, AWG_PROTOCOL, &valid)
.is_err()
);
transport.allow_peer(network, peer);
assert!(
transport
.on_control(network, peer, "wrong-protocol", &valid)
.is_err()
);
assert!(
transport
.on_control(network, peer, AWG_PROTOCOL, &[0; 4097])
.is_err()
);
for bytes in [
encode(&offer(99, [1; 32], vec![address])),
encode(&offer(AWG_WIRE_VERSION, [0; 32], vec![address])),
encode(&offer(AWG_WIRE_VERSION, [1; 32], vec![])),
encode(&offer(AWG_WIRE_VERSION, [1; 32], vec![address; 33])),
] {
assert!(
transport
.on_control(network, peer, AWG_PROTOCOL, &bytes)
.is_err()
);
}
for len in 0..valid.len() {
assert!(
transport
.on_control(network, peer, AWG_PROTOCOL, &valid[..len])
.is_err()
);
}
let wrong_role = AwgTransport::new(peer, Ipv4Addr::LOCALHOST);
wrong_role.allow_peer(network, local);
assert!(
wrong_role
.on_control(network, local, AWG_PROTOCOL, &valid)
.is_err()
);
transport.forget_peer(network, peer);
assert!(
transport
.on_control(network, peer, AWG_PROTOCOL, &valid)
.is_err()
);
assert!(transport.overview(network).is_empty());
transport.shutdown();
wrong_role.shutdown();
}
}
+773
View File
@@ -0,0 +1,773 @@
//! Real membership, UDP sockets, pair negotiation and nested WireGuard encryption.
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use bytes::Bytes;
use std::{
net::Ipv4Addr,
path::Path,
sync::{
Arc,
atomic::{AtomicBool, AtomicU64, Ordering},
},
time::Duration,
};
use tempfile::TempDir;
use tsunagi::dataplane::IpPlugin;
use tsunagi::dataplane::transport::{
ControlSender, PacketLink, PacketTransport, SharedLink, TransportError,
};
use tsunagi::discovery::SharedMemoryDiscovery;
use tsunagi::overlay::{MemoryTun, MemoryTunFactory};
use tsunagi::testing::{config_with, network, wait_for_peers, wait_until};
use tsunagi::{Agent, BoxFuture, NetworkId};
use tsunagi_awg::{AWG_PROTOCOL, AwgPlugin, AwgTransport};
use tsunagi_wg_quic::WireguardConfig;
struct Node {
agent: Agent,
plugin: Arc<AwgPlugin>,
quic: Option<Arc<tsunagi_wg_quic::WireguardPlugin>>,
transport: Arc<AwgTransport>,
tuns: MemoryTunFactory,
loss: Arc<AtomicBool>,
}
impl Node {
async fn open(root: &Path, discovery: &SharedMemoryDiscovery, interface: &str) -> Self {
Self::open_with_fallback(root, discovery, interface, false).await
}
async fn open_with_fallback(
root: &Path,
discovery: &SharedMemoryDiscovery,
interface: &str,
fallback: bool,
) -> Self {
Self::open_config(root, discovery, interface, fallback, 1280).await
}
async fn open_config(
root: &Path,
discovery: &SharedMemoryDiscovery,
interface: &str,
fallback: bool,
mtu: u32,
) -> Self {
Self::open_selected(root, discovery, interface, fallback, mtu, None).await
}
async fn open_selected(
root: &Path,
discovery: &SharedMemoryDiscovery,
interface: &str,
fallback: bool,
mtu: u32,
default_protocols: Option<Vec<String>>,
) -> Self {
let identity =
tsunagi::storage::StateStore::open(tsunagi::StoragePaths::under(root).state_db())
.unwrap()
.load_or_create_device_identity()
.unwrap();
let transport = AwgTransport::new(identity.endpoint_id(), Ipv4Addr::LOCALHOST);
let plugin = AwgPlugin::open(
WireguardConfig::new(root.join("awg"))
.with_mtu(mtu)
.with_reconcile(Duration::from_millis(20), Duration::from_millis(250)),
transport.clone(),
)
.await
.unwrap();
let loss = Arc::new(AtomicBool::new(false));
let faults = Arc::new(FaultTransport {
inner: transport.clone(),
loss: loss.clone(),
});
let tuns = MemoryTunFactory::new();
let mut config = config_with(root, discovery)
.with_interface(Arc::new(tuns.clone()), interface, mtu)
.with_plugin(plugin.clone() as Arc<dyn IpPlugin>)
.with_custom_transport(AWG_PROTOCOL, faults);
let mut quic_plugin = None;
if fallback {
let quic = tsunagi_wg_quic::WireguardPlugin::open(
WireguardConfig::new(root.join("wg-quic"))
.with_mtu(mtu)
.with_reconcile(Duration::from_millis(20), Duration::from_millis(250)),
)
.await
.unwrap();
config = config.with_plugin(quic.clone());
quic_plugin = Some(quic);
}
config.default_protocols = default_protocols;
let agent = Agent::spawn(config).await.unwrap();
let inbound_agent = agent.clone();
let inbound_transport = transport.clone();
let inbound_loss = loss.clone();
tokio::spawn(async move {
while let Some(mut link) = inbound_transport.next_inbound().await {
link.link = impaired(link.link, inbound_loss.clone());
let _ = inbound_agent.install_inbound_link(link).await;
}
});
Self {
agent,
plugin,
quic: quic_plugin,
transport,
tuns,
loss,
}
}
async fn ready(&self, id: NetworkId, n: usize) {
wait_until("AWG outer paths and end-to-end handshakes", || async {
(self
.transport
.overview(id)
.iter()
.filter(|p| !p.closed && !p.path.contains("establishing"))
.count()
== n
&& self
.plugin
.overview(id)?
.peers
.iter()
.filter(|p| {
p.tunnel
.as_ref()
.is_some_and(|t| t.health.is_up() && t.path.contains("awg UDP via"))
})
.count()
== n)
.then_some(())
})
.await;
}
fn tun(&self) -> Arc<MemoryTun> {
self.tuns
.device(&self.agent.overlay().unwrap().interface)
.unwrap()
}
fn addr(&self, id: NetworkId) -> Ipv4Addr {
self.plugin
.overview(id)
.unwrap()
.overlay_address_v4
.unwrap()
}
}
fn packet(source: Ipv4Addr, dest: Ipv4Addr, len: usize) -> Bytes {
let mut bytes = vec![0; len];
bytes[0] = 0x45;
bytes[2..4].copy_from_slice(&(len as u16).to_be_bytes());
bytes[8] = 64;
bytes[9] = 253;
bytes[12..16].copy_from_slice(&source.octets());
bytes[16..20].copy_from_slice(&dest.octets());
rand::fill(&mut bytes[20..]);
Bytes::from(bytes)
}
async fn exchange(a: &Node, b: &Node, id: NetworkId) {
for len in [64, 1097, 1280] {
let bytes = packet(a.addr(id), b.addr(id), len);
a.tun().push_from_os(bytes.clone());
assert_eq!(
tokio::time::timeout(Duration::from_secs(5), b.tun().pop_to_os())
.await
.unwrap()
.unwrap(),
bytes
);
}
}
#[tokio::test]
async fn a_control_only_agent_enables_real_data_without_restart_and_cancels_stale_attempts() {
let discovery = SharedMemoryDiscovery::new();
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
let a = Node::open_selected(
roots[0].path(),
&discovery,
"initial-none-a",
true,
1280,
Some(vec![]),
)
.await;
let b = Node::open_selected(
roots[1].path(),
&discovery,
"initial-none-b",
true,
1280,
Some(vec![]),
)
.await;
let (name, secret) = network("enable-after-control-only");
let id = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
wait_for_peers(&a.agent, id, 1).await;
wait_for_peers(&b.agent, id, 1).await;
assert!(a.agent.overlay().is_none());
assert!(b.agent.overlay().is_none());
for node in [&a, &b] {
assert!(
node.agent
.network_status(id)
.await
.unwrap()
.protocols
.is_empty()
);
node.agent
.set_protocols(id, vec!["awg".into()])
.await
.unwrap();
assert!(node.agent.overlay().is_some());
}
a.ready(id, 1).await;
b.ready(id, 1).await;
exchange(&a, &b, id).await;
let before = a.agent.overlay().unwrap().interface;
// Changes race unfinished opens and prepared plugin callbacks; the last
// selection must win even if an earlier open has completed in its queue.
for selection in [
vec!["wg-quic".into()],
vec![],
vec!["awg".into()],
vec![],
vec!["awg".into()],
] {
a.agent.set_protocols(id, selection).await.unwrap();
}
a.ready(id, 1).await;
b.ready(id, 1).await;
exchange(&a, &b, id).await;
assert_eq!(a.agent.overlay().unwrap().interface, before);
for node in [&a, &b] {
let status = node.agent.network_status(id).await.unwrap();
assert_eq!(status.protocols, ["awg"]);
assert_eq!(status.metrics.sessions_established, 1);
assert_eq!(status.metrics.disconnects, 0);
}
a.agent.shutdown().await;
b.agent.shutdown().await;
}
#[tokio::test]
async fn runtime_protocol_selection_replaces_real_tunnels_and_survives_restart() {
let discovery = SharedMemoryDiscovery::new();
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
let a = Node::open_with_fallback(roots[0].path(), &discovery, "runtime-a", true).await;
let b = Node::open_with_fallback(roots[1].path(), &discovery, "runtime-b", true).await;
let (name, secret) = network("runtime-protocols");
let id = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
a.ready(id, 1).await;
b.ready(id, 1).await;
exchange(&a, &b, id).await;
let (other_name, other_secret, other_id) = loop {
let (name, secret) = network("unchanged-runtime-network");
let other_id = tsunagi::identity::NetworkKeys::derive(&name, &secret).network_id();
if tsunagi::state::derived_ipv4_range(id) != tsunagi::state::derived_ipv4_range(other_id) {
break (name, secret, other_id);
}
};
a.agent
.join_network(&other_name, &other_secret)
.await
.unwrap();
b.agent
.join_network(&other_name, &other_secret)
.await
.unwrap();
a.ready(other_id, 1).await;
b.ready(other_id, 1).await;
let other_before = a
.agent
.network_status(other_id)
.await
.unwrap()
.metrics
.data_links_established;
for node in [&a, &b] {
let before = node.agent.network_status(id).await.unwrap().protocols;
for invalid in [vec!["unknown".into()], vec!["awg".into(), "awg".into()]] {
assert!(node.agent.set_protocols(id, invalid).await.is_err());
assert_eq!(
node.agent.network_status(id).await.unwrap().protocols,
before
);
}
}
// Exercise updates initiated by either endpoint-id role, including an
// empty set. Membership and the control session stay alive throughout.
for node in [&a, &b] {
node.agent.set_protocols(id, Vec::new()).await.unwrap();
wait_until("peer accepts control-only selection", || async {
let other = if node.agent.endpoint_id() == a.agent.endpoint_id() {
&b
} else {
&a
};
let status = other.agent.network_status(id).await.ok()?;
(status.peers.len() == 1
&& status.peers[0].protocols.is_empty()
&& matches!(status.peers[0].data_path, tsunagi::agent::DataPath::None))
.then_some(())
})
.await;
assert_eq!(node.agent.network_status(id).await.unwrap().peers.len(), 1);
node.agent
.set_protocols(id, vec!["wg-quic".into(), "awg".into()])
.await
.unwrap();
}
a.agent
.set_protocols(id, vec!["wg-quic".into()])
.await
.unwrap();
wait_until("real QUIC replaces AWG", || async {
let a_status = a.agent.network_status(id).await.ok()?;
let b_status = b.agent.network_status(id).await.ok()?;
(a_status.peers[0].protocols == ["wg-quic"]
&& b_status.peers[0].protocols == ["wg-quic"]
&& !matches!(a_status.peers[0].data_path, tsunagi::agent::DataPath::None)
&& !matches!(b_status.peers[0].data_path, tsunagi::agent::DataPath::None)
&& a.transport.overview(id).iter().all(|p| p.closed)
&& b.transport.overview(id).iter().all(|p| p.closed))
.then_some(())
})
.await;
// Get addresses from the system allocation: the disabled AWG plugin no
// longer has a live view of this network.
for (source, dest) in [(&a, &b), (&b, &a)] {
wait_until("QUIC end-to-end handshake", || async {
source
.quic
.as_ref()?
.overview(id)?
.peers
.iter()
.any(|p| p.tunnel.as_ref().is_some_and(|t| t.health.is_up()))
.then_some(())
})
.await;
let addr = |n: &Node| {
n.quic
.as_ref()
.unwrap()
.overview(id)
.unwrap()
.overlay_address_v4
.unwrap()
};
let bytes = packet(addr(source), addr(dest), 1280);
source.tun().push_from_os(bytes.clone());
assert_eq!(
tokio::time::timeout(Duration::from_secs(5), dest.tun().pop_to_os())
.await
.unwrap()
.unwrap(),
bytes
);
}
assert_eq!(
a.agent
.network_status(other_id)
.await
.unwrap()
.metrics
.data_links_established,
other_before
);
exchange(&a, &b, other_id).await;
a.agent.shutdown().await;
let restarted = Node::open_with_fallback(roots[0].path(), &discovery, "runtime-a", true).await;
wait_for_peers(&restarted.agent, id, 1).await;
assert_eq!(
restarted.agent.network_status(id).await.unwrap().protocols,
["wg-quic"]
);
assert_eq!(
b.agent.network_status(other_id).await.unwrap().protocols,
["awg", "wg-quic"]
);
restarted.ready(other_id, 1).await;
b.ready(other_id, 1).await;
exchange(&restarted, &b, other_id).await;
restarted
.agent
.set_protocols(id, vec!["awg".into()])
.await
.unwrap();
restarted.ready(id, 1).await;
b.ready(id, 1).await;
exchange(&restarted, &b, id).await;
restarted.agent.shutdown().await;
b.agent.shutdown().await;
}
#[tokio::test]
async fn real_pairs_carry_full_size_packets_and_restart_with_fresh_profiles() {
tsunagi::testing::init_tracing();
let discovery = SharedMemoryDiscovery::new();
let roots = [
TempDir::new().unwrap(),
TempDir::new().unwrap(),
TempDir::new().unwrap(),
];
let a = Node::open(roots[0].path(), &discovery, "awga").await;
let b = Node::open(roots[1].path(), &discovery, "awgb").await;
let c = Node::open(roots[2].path(), &discovery, "awgc").await;
let (name, secret) = network("awg-pairs");
let id = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
c.agent.join_network(&name, &secret).await.unwrap();
for node in [&a, &b, &c] {
wait_for_peers(&node.agent, id, 2).await;
node.ready(id, 2).await;
}
let ids: std::collections::HashSet<_> = [&a, &b, &c]
.into_iter()
.flat_map(|n| n.transport.overview(id).into_iter().map(|p| p.attempt))
.collect();
assert_eq!(
ids.len(),
3,
"one independent profile for each of three pairs"
);
exchange(&a, &b, id).await;
exchange(&b, &a, id).await;
exchange(&a, &c, id).await;
let old_ports: Vec<_> = a
.transport
.overview(id)
.into_iter()
.filter_map(|p| p.local_port)
.collect();
a.agent.shutdown().await;
for port in old_ports {
wait_until("old pair socket released", || async {
tokio::net::UdpSocket::bind((Ipv4Addr::LOCALHOST, port))
.await
.ok()
})
.await;
}
let restarted = Node::open(roots[0].path(), &discovery, "awga").await;
restarted.ready(id, 2).await;
assert!(
restarted
.transport
.overview(id)
.iter()
.all(|p| !ids.contains(&p.attempt))
);
exchange(&restarted, &b, id).await;
exchange(&c, &restarted, id).await;
restarted.agent.shutdown().await;
b.agent.shutdown().await;
c.agent.shutdown().await;
assert!(
tokio::time::timeout(Duration::from_secs(1), restarted.transport.next_inbound())
.await
.unwrap()
.is_none()
);
}
// Drops datagrams at the link boundary. Membership, control, sockets, crypto,
// addressing and storage remain real. This substitutes a lossy physical path.
#[derive(Debug)]
struct FaultTransport {
inner: Arc<AwgTransport>,
loss: Arc<AtomicBool>,
}
impl PacketTransport for FaultTransport {
fn name(&self) -> &str {
AWG_PROTOCOL
}
fn as_any(&self) -> &dyn std::any::Any {
self
}
fn attach_control(&self, sender: Arc<dyn ControlSender>) {
self.inner.attach_control(sender);
}
fn on_control(
&self,
network: NetworkId,
peer: iroh::EndpointId,
protocol: &str,
bytes: &[u8],
) -> Result<(), TransportError> {
self.inner.on_control(network, peer, protocol, bytes)
}
fn open<'a>(
&'a self,
network: NetworkId,
peer: iroh::EndpointId,
protocol: &'a str,
) -> BoxFuture<'a, Result<SharedLink, TransportError>> {
Box::pin(async move {
Ok(impaired(
self.inner.open(network, peer, protocol).await?,
self.loss.clone(),
))
})
}
}
fn impaired(raw: SharedLink, loss: Arc<AtomicBool>) -> SharedLink {
Arc::new(FaultLink {
raw,
loss,
sequence: AtomicU64::new(0),
})
}
#[derive(Debug)]
struct FaultLink {
raw: SharedLink,
loss: Arc<AtomicBool>,
sequence: AtomicU64,
}
impl PacketLink for FaultLink {
fn network(&self) -> NetworkId {
self.raw.network()
}
fn peer(&self) -> iroh::EndpointId {
self.raw.peer()
}
fn max_datagram_size(&self) -> usize {
self.raw.max_datagram_size()
}
fn send(&self, bytes: Bytes) -> Result<(), TransportError> {
if self.loss.load(Ordering::Relaxed)
&& !self
.sequence
.fetch_add(1, Ordering::Relaxed)
.is_multiple_of(5)
{
Ok(())
} else {
self.raw.send(bytes)
}
}
fn recv(&self) -> BoxFuture<'_, Option<Bytes>> {
self.raw.recv()
}
fn closed(&self) -> BoxFuture<'_, ()> {
self.raw.closed()
}
fn is_closed(&self) -> bool {
self.raw.is_closed()
}
fn close(&self) {
self.raw.close();
}
fn path_description(&self) -> String {
self.raw.path_description()
}
}
#[tokio::test]
async fn persistent_runtime_loss_retires_the_pair_and_negotiates_fresh_parameters() {
let discovery = SharedMemoryDiscovery::new();
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
let a = Node::open(roots[0].path(), &discovery, "lossa").await;
let b = Node::open(roots[1].path(), &discovery, "lossb").await;
let (name, secret) = network("awg-loss");
let id = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
a.ready(id, 1).await;
b.ready(id, 1).await;
exchange(&a, &b, id).await;
let old = a.transport.overview(id)[0].attempt;
let mut events = a.agent.subscribe();
a.loss.store(true, Ordering::Relaxed);
tsunagi::testing::wait_event(&mut events, |event| match event {
tsunagi::Event::DataLinkDown {
network, reason, ..
} if *network == id && reason.contains("delivery checks") => Some(()),
_ => None,
})
.await;
a.loss.store(false, Ordering::Relaxed);
wait_until("fresh AWG profile after runtime loss", || async {
a.transport
.overview(id)
.iter()
.any(|p| p.attempt != old && !p.closed && !p.path.contains("establishing"))
.then_some(())
})
.await;
a.ready(id, 1).await;
b.ready(id, 1).await;
let initiator = if a.agent.endpoint_id().as_bytes() < b.agent.endpoint_id().as_bytes() {
&a
} else {
&b
};
let received = initiator.plugin.overview(id).unwrap().peers[0]
.tunnel
.as_ref()
.unwrap()
.stats
.rx_packets;
let attempt = initiator.transport.overview(id)[0].attempt;
wait_until(
"fresh pair delivery checks answered after installation",
|| async {
let view = initiator.plugin.overview(id)?;
(view.peers.first()?.tunnel.as_ref()?.stats.rx_packets >= received + 3).then_some(())
},
)
.await;
assert_eq!(initiator.transport.overview(id)[0].attempt, attempt);
wait_for_peers(&a.agent, id, 1).await;
exchange(&a, &b, id).await;
exchange(&b, &a, id).await;
a.agent.shutdown().await;
b.agent.shutdown().await;
}
#[tokio::test]
async fn runtime_loss_falls_back_to_real_iroh_without_interrupting_control_or_ip() {
let discovery = SharedMemoryDiscovery::new();
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
let a = Node::open_with_fallback(roots[0].path(), &discovery, "fallba", true).await;
let b = Node::open_with_fallback(roots[1].path(), &discovery, "fallbb", true).await;
let (name, secret) = network("awg-fallback");
let id = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
a.ready(id, 1).await;
b.ready(id, 1).await;
exchange(&a, &b, id).await;
let mut events = a.agent.subscribe();
a.loss.store(true, Ordering::Relaxed);
tsunagi::testing::wait_event(&mut events, |event| match event {
tsunagi::Event::DataLinkDown {
network, reason, ..
} if *network == id && reason.contains("delivery checks") => Some(()),
_ => None,
})
.await;
for node in [&a, &b] {
wait_until("fallback carrier is wg-quic",||async {
let status=node.agent.status().await.ok()?;
let network=status.networks.iter().find(|n|n.network_id==id)?;
(network.members.iter().any(|m|matches!(&m.data_path,tsunagi::agent::DataPath::Direct{transport} if transport=="wg-quic"))).then_some(())
}).await;
wait_for_peers(&node.agent, id, 1).await;
}
exchange(&a, &b, id).await;
exchange(&b, &a, id).await;
a.agent.shutdown().await;
b.agent.shutdown().await;
}
#[tokio::test]
async fn opaque_pair_carrier_fragments_its_full_advertised_capacity() {
let discovery = SharedMemoryDiscovery::new();
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
let mut nodes = Vec::new();
for root in &roots {
let identity = tsunagi::storage::StateStore::open(
tsunagi::StoragePaths::under(root.path()).state_db(),
)
.unwrap()
.load_or_create_device_identity()
.unwrap();
let transport = AwgTransport::new(identity.endpoint_id(), Ipv4Addr::LOCALHOST);
let agent = Agent::spawn(
config_with(root.path(), &discovery)
.with_custom_transport(AWG_PROTOCOL, transport.clone()),
)
.await
.unwrap();
nodes.push((agent, transport));
}
let (name, secret) = network("awg-capacity");
let id = nodes[0].0.join_network(&name, &secret).await.unwrap();
nodes[1].0.join_network(&name, &secret).await.unwrap();
for (i, (agent, transport)) in nodes.iter().enumerate() {
wait_for_peers(agent, id, 1).await;
// This fixture tests only the transport contract: register the member
// after the real core has authenticated it, without an IP plugin/TUN.
transport.allow_peer(id, nodes[1 - i].0.endpoint_id());
}
nodes.sort_by_key(|(agent, _)| *agent.endpoint_id().as_bytes());
let outgoing = nodes[0]
.1
.open(id, nodes[1].0.endpoint_id(), AWG_PROTOCOL)
.await
.unwrap();
let incoming = tokio::time::timeout(Duration::from_secs(5), nodes[1].1.next_inbound())
.await
.unwrap()
.unwrap()
.link;
for (from, to) in [(&outgoing, &incoming), (&incoming, &outgoing)] {
let mut bytes = vec![0; from.max_datagram_size()];
rand::fill(&mut bytes[..]);
let bytes = Bytes::from(bytes);
from.send(bytes.clone()).unwrap();
assert_eq!(
tokio::time::timeout(Duration::from_secs(5), to.recv())
.await
.unwrap()
.unwrap(),
bytes
);
assert!(matches!(
from.send(Bytes::from(vec![0; from.max_datagram_size() + 1])),
Err(TransportError::TooLarge { .. })
));
}
outgoing.close();
incoming.close();
for (agent, transport) in nodes {
transport.shutdown();
agent.shutdown().await;
}
}
#[tokio::test]
async fn encrypted_health_checks_fit_a_smaller_peers_configured_interface_mtu() {
let discovery = SharedMemoryDiscovery::new();
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
let a = Node::open_config(roots[0].path(), &discovery, "smalla", false, 1280).await;
let b = Node::open_config(roots[1].path(), &discovery, "smallb", false, 576).await;
let (name, secret) = network("awg-small-mtu");
let id = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
a.ready(id, 1).await;
b.ready(id, 1).await;
let bytes = packet(a.addr(id), b.addr(id), 576);
a.tun().push_from_os(bytes.clone());
assert_eq!(
tokio::time::timeout(Duration::from_secs(5), b.tun().pop_to_os())
.await
.unwrap()
.unwrap(),
bytes
);
wait_until(
"encrypted delivery checks answered with a small MTU",
|| async {
let view = a.plugin.overview(id)?;
let peer = view.peers.first()?.tunnel.as_ref()?;
(peer.stats.rx_packets >= 3).then_some(())
},
)
.await;
for node in [&a, &b] {
assert_eq!(
node.plugin.overview(id).unwrap().peers[0]
.tunnel
.as_ref()
.unwrap()
.stats
.dropped_oversize,
0
);
}
a.agent.shutdown().await;
b.agent.shutdown().await;
}
+36
View File
@@ -0,0 +1,36 @@
//! Fresh and bounded masking profiles, and rejection of foreign/truncated wire packets.
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use tsunagi_awg::profile::PairProfile;
#[test]
fn profiles_are_fresh_bounded_and_redacted() {
let profiles: Vec<_> = (0..128).map(|_| PairProfile::generate()).collect();
for (i, profile) in profiles.iter().enumerate() {
profile.validate().unwrap();
assert!(!format!("{profile:?}").contains(&format!("{:?}", profile.header_key())));
for other in &profiles[..i] {
assert_ne!(profile.header_key(), other.header_key());
assert_ne!(profile.id(), other.id());
}
for packet in profile.signature_packets() {
assert!(packet.len() < 1200);
}
}
}
#[test]
fn masked_packets_round_trip_and_reject_other_profiles_and_truncation() {
let profile = PairProfile::generate();
let other = PairProfile::generate();
for (kind, size) in [(1, 148), (2, 92), (3, 64), (4, 800)] {
let mut message = vec![0x7b; size];
message[..4].copy_from_slice(&(kind as u32).to_le_bytes());
let wire = profile.encode(&message).unwrap();
assert_ne!(&wire[..4], &message[..4]);
assert_eq!(profile.decode(&wire).unwrap(), message);
assert!(other.decode(&wire).is_none());
for length in 0..44 {
assert!(profile.decode(&wire[..length]).is_none());
}
}
}
+533 -161
View File
@@ -27,14 +27,14 @@ mod status;
pub use events::Event;
pub use status::{
AgentStatus, CandidateStatus, DataPath, HostnameConflict, MemberStatus, NetworkMetrics,
NetworkState, NetworkStatus, OverlayStatus, PeerStatus,
NetworkState, NetworkStatus, OverlayStatus, PeerStatus, RangeConflict,
};
use std::collections::HashMap;
use std::sync::{Arc, Weak};
use iroh::{EndpointAddr, EndpointId};
use tokio::sync::{RwLock, broadcast, mpsc, oneshot};
use tokio::sync::{Mutex, RwLock, broadcast, mpsc, oneshot, watch};
use tokio::task::JoinHandle;
use crate::BoxFuture;
@@ -58,6 +58,8 @@ use shutdown::Shutdown;
/// Summary of a configured network, whether or not it is running.
#[derive(Debug, Clone)]
pub struct ConfiguredNetwork {
/// Enabled local data-plane protocols, in preference order.
pub protocols: Vec<String>,
/// Saved local broadcast participation.
pub broadcast: bool,
/// Public network identifier.
@@ -70,6 +72,17 @@ pub struct ConfiguredNetwork {
pub active: bool,
}
/// A durable version-scoped cleanup, not an identity ban.
#[derive(Debug, Clone)]
pub struct ForgetNodeOutcome {
/// Identity whose observed claim was retired.
pub target: EndpointId,
/// The retired claim version; future signed claims remain admissible.
pub record_version: u64,
/// Authenticated peers to which cleanup was queued immediately.
pub peers_told: usize,
}
/// A running agent.
///
/// Cloning gives another handle to the same agent.
@@ -90,6 +103,10 @@ struct Inner {
events: broadcast::Sender<Event>,
networks: RwLock<HashMap<NetworkId, NetworkHandle>>,
shutdown: Shutdown,
/// Serialises start, stop, switching and conflict cleanup, including teardown.
lifecycle: Mutex<()>,
range_changed: watch::Sender<()>,
range_task: std::sync::Mutex<Option<JoinHandle<()>>>,
accept_task: std::sync::Mutex<Option<JoinHandle<()>>>,
plugin_task: std::sync::Mutex<Option<JoinHandle<()>>>,
transport: std::sync::OnceLock<Arc<dyn PacketTransport>>,
@@ -100,6 +117,37 @@ struct Inner {
}
impl Inner {
/// Called under the lifecycle lock after startup: an initially
/// control-only agent creates its one interface on first enablement.
async fn ensure_interface(self: &Arc<Self>) {
if self.interface.get().is_some() {
return;
}
if let Some(factory) = self.config.tun_factory.clone() {
let carrier = Arc::new(Carrier(Arc::downgrade(self))) as Arc<dyn PacketCarrier>;
match crate::overlay::Interface::start(
factory,
self.config.interface_name.clone(),
self.config.interface_mtu,
Arc::clone(&self.routes),
carrier,
)
.await
{
Ok(interface) => {
let _ = self.interface.set(Arc::new(interface));
}
Err(err) => {
let _ = self.events.send(Event::PluginError {
network: NetworkId::from_bytes([0u8; 32]),
protocol: "overlay".into(),
reason: err.to_string(),
});
}
}
}
}
/// The name this agent currently answers to.
fn read_hostname(&self) -> String {
match self.hostname.read() {
@@ -165,6 +213,21 @@ impl PacketSink for Sink {
let Some(inner) = self.0.upgrade() else {
return;
};
if let Some(payload) = crate::dataplane::transport::health::ip_payload(&packet) {
if let Some(header) = crate::overlay::IpHeader::parse(&packet)
&& let (std::net::IpAddr::V4(source), std::net::IpAddr::V4(destination)) =
(header.source(), header.destination())
&& inner.routes.may_send_from(network, peer, source)
&& inner.routes.is_local_destination(network, destination)
&& let Some(handle) = inner.networks.read().await.get(&network)
{
let _ = handle.commands.try_send(NetCommand::HealthProbe {
peer,
packet: bytes::Bytes::copy_from_slice(payload),
});
}
return;
}
let Some(interface) = inner.interface.get() else {
return;
};
@@ -183,6 +246,35 @@ impl PacketSink for Sink {
#[derive(Debug)]
struct TransportCtx(Weak<Inner>);
impl crate::dataplane::transport::ControlSender for TransportCtx {
fn send<'a>(
&'a self,
network: NetworkId,
peer: EndpointId,
protocol: &'a str,
payload: crate::proto::message::TransportPayload,
) -> crate::BoxFuture<'a, std::result::Result<(), crate::dataplane::transport::TransportError>>
{
Box::pin(async move {
let inner = self
.0
.upgrade()
.ok_or(crate::dataplane::transport::TransportError::Closed)?;
Agent { inner }
.send(
network,
peer,
crate::proto::message::ControlMessage::Transport {
protocol: protocol.into(),
payload,
},
)
.await
.map_err(|err| crate::dataplane::transport::TransportError::Other(err.to_string()))
})
}
}
impl TransportContext for TransportCtx {
fn snapshot(&self) -> crate::BoxFuture<'_, HashMap<NetworkId, NetworkKeys>> {
Box::pin(async move {
@@ -217,12 +309,133 @@ impl TransportContext for TransportCtx {
}
impl Agent {
/// Atomically saves or removes a version-checked shared VIP draft.
/// Every authenticated network member may edit any service.
pub async fn save_vip(
&self,
network: NetworkId,
config: crate::vip::VipConfig,
expected: Option<String>,
delete: bool,
) -> Result<Vec<crate::vip::VipStatus>> {
let (reply, receive) = oneshot::channel();
self.command(
network,
NetCommand::SaveVip {
config,
expected,
delete,
reply,
},
)
.await?;
receive.await.map_err(|_| Error::Stopped)?
}
/// Retires one offline member's observed claim without revoking membership.
/// Every member has this right. The version must still match the selection.
pub async fn forget_node(
&self,
network: NetworkId,
peer: EndpointId,
expected_version: u64,
) -> Result<ForgetNodeOutcome> {
let _lifecycle = self.inner.lifecycle.lock().await;
let (reply, receive) = oneshot::channel();
self.command(
network,
NetCommand::ForgetNode {
peer,
expected_version,
reply,
},
)
.await?;
receive.await.map_err(|_| Error::Stopped)?
}
fn effective_protocols(&self, saved: Option<&Vec<String>>) -> Vec<String> {
saved
.or(self.inner.config.default_protocols.as_ref())
.cloned()
.unwrap_or_else(|| {
self.inner
.config
.plugins
.iter()
.map(|p| p.protocol_id().to_owned())
.collect()
})
}
fn selected_plugins(&self, protocols: &[String]) -> Vec<crate::dataplane::SharedPlugin> {
protocols
.iter()
.filter_map(|name| {
self.inner
.config
.plugins
.iter()
.find(|p| p.protocol_id() == name)
.cloned()
})
.collect()
}
/// Saves and immediately applies a network's ordered local protocol set.
/// An empty set disables data traffic while retaining control sessions.
pub async fn set_protocols(&self, network_id: NetworkId, protocols: Vec<String>) -> Result<()> {
crate::dataplane::validate_protocols(&protocols, self.inner.limits.max_capabilities)?;
for name in &protocols {
if !self
.inner
.config
.plugins
.iter()
.any(|p| p.protocol_id() == name)
{
return Err(Error::InvalidProtocols(format!(
"this agent has no `{name}` plugin"
)));
}
}
let _lifecycle = self.inner.lifecycle.lock().await;
self.inner
.storage
.set_protocols(network_id, protocols.clone())
.await?;
if self.is_active(network_id).await {
let (reply, receive) = oneshot::channel();
if !protocols.is_empty() {
self.inner.ensure_interface().await;
}
self.command(
network_id,
NetCommand::SetProtocols {
protocols,
interface: self.inner.interface.get().cloned(),
reply,
},
)
.await?;
receive.await.map_err(|_| Error::Stopped)?;
}
Ok(())
}
/// Starts an agent.
///
/// Opens the state store (taking its ownership lock), restores the
/// persistent device identity, binds the iroh endpoint and activates every
/// configured network whose auto-start flag is set.
pub async fn spawn(config: AgentConfig) -> Result<Self> {
if let Some(protocols) = &config.default_protocols {
crate::dataplane::validate_protocols(protocols, config.limits.max_capabilities)?;
for name in protocols {
if !config.plugins.iter().any(|p| p.protocol_id() == name) {
return Err(Error::InvalidProtocols(format!(
"this agent has no `{name}` plugin"
)));
}
}
}
let storage = Storage::open(&config.paths)?;
let identity = storage.device_identity().await?;
let adapter = EndpointAdapter::bind(&config, &identity).await?;
@@ -234,6 +447,7 @@ impl Agent {
let (events, _) = broadcast::channel(config.limits.event_buffer);
let limits = Arc::new(config.limits.clone());
let (range_changed, range_updates) = watch::channel(());
let inner = Arc::new(Inner {
limits,
storage,
@@ -243,6 +457,9 @@ impl Agent {
events,
networks: RwLock::new(HashMap::new()),
shutdown: Shutdown::new(),
lifecycle: Mutex::new(()),
range_changed,
range_task: std::sync::Mutex::new(None),
accept_task: std::sync::Mutex::new(None),
plugin_task: std::sync::Mutex::new(None),
transport: std::sync::OnceLock::new(),
@@ -251,31 +468,13 @@ impl Agent {
config,
});
// One interface for the agent, if it was given a way to make one.
// Failing to is reported and not fatal: the control plane works, and
// so do the protocols, they just have nowhere to put packets.
if let Some(factory) = inner.config.tun_factory.clone() {
let carrier = Arc::new(Carrier(Arc::downgrade(&inner))) as Arc<dyn PacketCarrier>;
match crate::overlay::Interface::start(
factory,
inner.config.interface_name.clone(),
inner.config.interface_mtu,
Arc::clone(&inner.routes),
carrier,
)
.await
{
Ok(interface) => {
let _ = inner.interface.set(Arc::new(interface));
}
Err(err) => {
let _ = inner.events.send(Event::PluginError {
network: NetworkId::from_bytes([0u8; 32]),
protocol: "overlay".into(),
reason: err.to_string(),
});
}
}
if inner
.config
.default_protocols
.as_ref()
.is_none_or(|names| !names.is_empty())
{
inner.ensure_interface().await;
}
// The data plane rides on iroh too, which is where it gets hole
@@ -294,6 +493,7 @@ impl Agent {
inner.config.custom_transports.clone(),
))
};
transport.attach_control(Arc::new(TransportCtx(Arc::downgrade(&inner))));
let _ = inner.transport.set(transport);
if let CacheOutcome::Reset(reason) = inner.storage.cache_outcome().clone() {
@@ -311,8 +511,9 @@ impl Agent {
if !inner.config.plugins.is_empty() {
let (plugin_tx, plugin_rx) = mpsc::channel(64);
let sink = inner
.interface
.get()
.config
.tun_factory
.as_ref()
.map(|_| Arc::new(Sink(Arc::downgrade(&inner))) as Arc<dyn PacketSink>);
let context = PluginContext::new(plugin_tx, inner.identity.endpoint_id(), sink);
for plugin in &inner.config.plugins {
@@ -326,11 +527,16 @@ impl Agent {
let agent = Self { inner };
for stored in agent.inner.storage.list_networks().await? {
if stored.auto_start {
let keys = NetworkKeys::derive(&stored.name, &stored.secret);
agent.activate_with_keys(keys).await?;
}
{
let _guard = agent.inner.lifecycle.lock().await;
agent.resume_networks_locked().await?;
}
let task = tokio::spawn(range_conflict_loop(
Arc::downgrade(&agent.inner),
range_updates,
));
if let Ok(mut guard) = agent.inner.range_task.lock() {
*guard = Some(task);
}
Ok(agent)
@@ -374,100 +580,65 @@ impl Agent {
Ok(())
}
/// Reserves the configured overlay range for a network, if it can.
///
/// Done here, where activations are serialised, rather than inside the
/// runtime: whether a range is free depends on what the other networks
/// took, and deciding that in a task would make the answer depend on
/// which task ran first.
///
/// `None` means this agent will not propose a range for that network and
/// waits to adopt whatever it settles on. One agent has one interface, so
/// proposing a range it could not route would be worse than having none:
/// the lowest author's range wins, and the collision would spread.
async fn reserve_range(&self, network: NetworkId) -> RangePlan {
let reserve = |wanted: Ipv4Range| {
let reservation = crate::overlay::NetworkRoutes {
exit: Default::default(),
broadcast: Default::default(),
range: Some(wanted),
local: None,
peers: Vec::new(),
};
self.inner.routes.set_network(network, reservation).is_ok()
};
// If this network already has signed records in storage, check if its
// members have already agreed on a range. If so, preserve and reserve
// THAT range rather than preemptively seizing the default configured range.
let stored_records = self
.inner
.storage
.signed_records(network)
.await
.unwrap_or_default();
let mut state_set = crate::state::StateSet::new();
state_set.merge_all(network, stored_records);
if let Some(agreed) = state_set.agreed_range() {
if reserve(agreed) {
return RangePlan {
propose: if Some(agreed) == self.inner.config.overlay_ipv4_range {
Some(agreed)
} else {
None
},
..RangePlan::default()
};
/// Signed state always wins over a default or an explicit proposal.
async fn desired_range(&self, network: NetworkId) -> Result<Option<Ipv4Range>> {
let records = self.inner.storage.signed_records(network).await?;
let mut state = crate::state::StateSet::new();
state.merge_all(network, records);
Ok(state.agreed_range().or_else(|| {
if self.inner.config.overlay_ipv4_auto {
Some(crate::state::derived_ipv4_range(network))
} else {
tracing::warn!(
%network,
range = %agreed,
"agreed range for this network overlaps another network already active on this interface"
);
return RangePlan {
conflict: Some(agreed),
..RangePlan::default()
};
self.inner.config.overlay_ipv4_range
}
}))
}
async fn range_conflicts(
&self,
network: NetworkId,
range: Option<Ipv4Range>,
) -> Result<Vec<RangeConflict>> {
let Some(range) = range else {
return Ok(Vec::new());
};
let overlapping = self.inner.routes.overlapping_networks(network, range);
let stored = self.inner.storage.list_networks().await?;
Ok(overlapping
.into_iter()
.filter_map(|(id, range)| {
let name = stored.iter().find(|n| n.network_id == id)?.name.clone();
Some(RangeConflict {
network_id: id,
name,
range,
})
})
.collect())
}
/// Starts requested networks in saved preference order, skipping overlaps.
/// Called with the lifecycle lock held; skipped networks remain requested.
async fn resume_networks_locked(&self) -> Result<()> {
let order = self.inner.storage.network_start_order().await?;
let mut stored = self.inner.storage.list_networks().await?;
stored.sort_by_key(|n| {
order
.iter()
.position(|id| *id == n.network_id)
.unwrap_or(usize::MAX)
});
for network in stored {
if !network.auto_start || self.is_active(network.network_id).await {
continue;
}
let keys = NetworkKeys::derive(&network.name, &network.secret);
match self.activate_with_keys_locked(keys).await {
Ok(()) | Err(Error::NetworkRangeConflict { .. }) => {}
Err(err) => return Err(err),
}
}
let Some(configured) = self.inner.config.overlay_ipv4_range else {
return RangePlan::default();
};
// The configured range first, so a network a device has always had
// keeps the addresses it has always had.
if reserve(configured) {
return RangePlan {
propose: Some(configured),
..RangePlan::default()
};
}
// A second network on the same agent cannot have it — one agent,
// one interface — so it falls back to the range derived from its
// own id, which every one of its members derives identically
// without being told.
let derived = crate::state::derived_ipv4_range(network);
if reserve(derived) {
tracing::info!(
%network,
range = %derived,
"another network here holds the configured range; this one uses the \
range derived from its id unless its members settled another"
);
return RangePlan {
fallback: Some(derived),
conflict: Some(configured),
..RangePlan::default()
};
}
tracing::info!(%network, "not proposing a range for this network");
RangePlan {
conflict: Some(configured),
..RangePlan::default()
}
Ok(())
}
/// The overlay interface this agent owns, when it has one.
@@ -546,6 +717,9 @@ impl Agent {
/// The same `(name, secret)` always produces the same [`NetworkId`], on
/// every device.
///
/// An overlapping network stays configured and requested, but inactive;
/// use [`Agent::switch_network`] to suspend the blockers explicitly.
///
/// This is declarative and therefore **idempotent**: joining a network
/// that is already active succeeds and changes nothing. That matters
/// because a configured network is activated automatically at startup, so
@@ -599,7 +773,9 @@ impl Agent {
}
match self.activate_with_keys(keys).await {
// Already a member of exactly this network space: nothing to do.
Ok(()) | Err(Error::NetworkAlreadyActive(_)) => Ok(network_id),
Ok(())
| Err(Error::NetworkAlreadyActive(_))
| Err(Error::NetworkRangeConflict { .. }) => Ok(network_id),
Err(err) => Err(err),
}
}
@@ -703,7 +879,44 @@ impl Agent {
self.activate_with_keys(keys).await
}
/// Switches to a configured network, suspending every overlapping network.
/// Suspended networks retain their auto-start request and resume when free.
/// Returns the networks suspended by this request.
pub async fn switch_network(&self, network_id: NetworkId) -> Result<Vec<NetworkId>> {
let _guard = self.inner.lifecycle.lock().await;
if self.inner.shutdown.is_triggered() {
return Err(Error::Stopped);
}
let stored = self
.inner
.storage
.list_networks()
.await?
.into_iter()
.find(|n| n.network_id == network_id)
.ok_or(Error::NetworkUnknown(network_id))?;
let range = self.desired_range(network_id).await?;
let conflicts = self.range_conflicts(network_id, range).await?;
self.inner.storage.prefer_network(network_id).await?;
let mut stopped = Vec::new();
for conflict in conflicts {
self.stop_network_locked(conflict.network_id).await?;
stopped.push(conflict.network_id);
}
if !self.is_active(network_id).await {
self.activate_with_keys_locked(NetworkKeys::derive(&stored.name, &stored.secret))
.await?;
}
self.resume_networks_locked().await?;
Ok(stopped)
}
async fn activate_with_keys(&self, keys: NetworkKeys) -> Result<()> {
let _guard = self.inner.lifecycle.lock().await;
self.activate_with_keys_locked(keys).await
}
async fn activate_with_keys_locked(&self, keys: NetworkKeys) -> Result<()> {
if self.inner.shutdown.is_triggered() {
return Err(Error::Stopped);
}
@@ -712,8 +925,70 @@ impl Agent {
if networks.contains_key(&network_id) {
return Err(Error::NetworkAlreadyActive(network_id));
}
let range = self.reserve_range(network_id).await;
let stored = self
.inner
.storage
.list_networks()
.await?
.into_iter()
.find(|stored| stored.network_id == network_id);
// Mandatory cleanup proofs must never be silently dropped on startup.
let retirements = self
.inner
.storage
.retirements(network_id, self.inner.limits.max_retirements)
.await?;
let vip_entries = self
.inner
.storage
.vip_fields(
network_id,
self.inner
.limits
.max_vips
.saturating_mul(self.inner.limits.max_vip_fields)
.saturating_add(self.inner.limits.max_vip_tombstones),
)
.await?;
let mut vips = crate::vip::VipSet::default();
for id in vip_entries
.iter()
.map(|e| e.vip.clone())
.collect::<std::collections::BTreeSet<_>>()
{
vips.merge(
network_id,
vip_entries
.iter()
.filter(|e| e.vip == id)
.cloned()
.collect(),
&self.inner.limits,
)?;
}
let range = self.desired_range(network_id).await?;
if let Some(range) = range {
let conflicts = self.inner.routes.overlapping_networks(network_id, range);
if !conflicts.is_empty() {
return Err(Error::NetworkRangeConflict {
network: network_id,
range,
conflicts: conflicts.into_iter().map(|(id, _)| id).collect(),
});
}
self.inner
.routes
.set_network(
network_id,
crate::overlay::NetworkRoutes {
range: Some(range),
..Default::default()
},
)
.map_err(|err| Error::Rejected {
reason: err.to_string(),
})?;
}
let mut backends = Vec::new();
if let Some(discovery) = &self.inner.config.discovery {
backends.push(discovery.clone());
@@ -729,20 +1004,22 @@ impl Agent {
as Arc<dyn crate::discovery::NetworkDiscovery>,
)
};
let stored = self
.inner
.storage
.list_networks()
.await?
.into_iter()
.find(|stored| stored.network_id == network_id);
let broadcast = stored.as_ref().is_none_or(|stored| stored.broadcast);
let protocols =
self.effective_protocols(stored.as_ref().and_then(|s| s.protocols.as_ref()));
let plugins = self.selected_plugins(&protocols);
if !plugins.is_empty() {
self.inner.ensure_interface().await;
}
let exit_offer = stored.as_ref().is_some_and(|stored| stored.exit_node);
let exit_via = stored
.as_ref()
.and_then(|stored| stored.exit_via)
.and_then(|bytes| EndpointId::from_bytes(&bytes).ok());
let handle = network::spawn(RuntimeParams {
vips,
retirements,
broadcast,
exit_offer,
exit_via,
@@ -755,7 +1032,8 @@ impl Agent {
discovery,
discovery_interval: self.inner.config.discovery_interval,
discovery_policy: self.inner.config.discovery_policy.clone(),
plugins: self.inner.config.plugins.clone(),
available_plugins: self.inner.config.plugins.clone(),
plugins: plugins.clone(),
routes: Arc::clone(&self.inner.routes),
interface: self.inner.interface.get().cloned(),
hostname: self.inner.read_hostname(),
@@ -763,14 +1041,13 @@ impl Agent {
device_secret: self.inner.identity.signing_key(),
#[cfg(feature = "testing")]
unreachable_data_peers: Arc::clone(&self.inner.config.unreachable_data_peers),
ipv4_range: range.propose,
ipv4_fallback: range.fallback,
range_conflict: range.conflict,
ipv4_range: range,
range_changed: self.inner.range_changed.clone(),
});
networks.insert(network_id, handle);
drop(networks);
for plugin in &self.inner.config.plugins {
for plugin in &plugins {
plugin.on_network_activated(network_id);
}
@@ -786,6 +1063,30 @@ impl Agent {
/// it does not remove this agent from anyone else's view of the network.
/// Other networks keep running.
pub async fn deactivate_network(&self, network_id: NetworkId) -> Result<()> {
let _guard = self.inner.lifecycle.lock().await;
if !self.is_active(network_id).await {
return Err(Error::NetworkNotActive(network_id));
}
self.inner.storage.set_auto_start(network_id, false).await?;
self.stop_network_locked(network_id).await?;
self.resume_networks_locked().await
}
/// Stops a configured network, including one already suspended by overlap.
/// Clears its auto-start request so it will not resume behind the user.
pub async fn pause_network(&self, network_id: NetworkId) -> Result<()> {
let _guard = self.inner.lifecycle.lock().await;
if self.inner.shutdown.is_triggered() {
return Err(Error::Stopped);
}
self.inner.storage.set_auto_start(network_id, false).await?;
if self.is_active(network_id).await {
self.stop_network_locked(network_id).await?;
}
self.resume_networks_locked().await
}
async fn stop_network_locked(&self, network_id: NetworkId) -> Result<()> {
let handle = {
let mut networks = self.inner.networks.write().await;
networks
@@ -814,7 +1115,6 @@ impl Agent {
interface.sync_broadcast_rules().await;
interface.sync_exit_rules().await;
}
self.inner.storage.set_auto_start(network_id, false).await?;
Ok(())
}
@@ -824,17 +1124,15 @@ impl Agent {
/// Local and silent: nobody is told. [`Agent::leave_network`] is the one
/// that says goodbye first, and is what a person means by leaving.
pub async fn forget_network(&self, network_id: NetworkId) -> Result<()> {
let _guard = self.inner.lifecycle.lock().await;
if self.is_active(network_id).await {
self.deactivate_network(network_id).await?;
self.stop_network_locked(network_id).await?;
}
self.inner.storage.remove_network(network_id).await?;
// Whatever a protocol kept for this network goes with it. The
// agent cannot know what that is, only that there is no longer any
// reason to hold it.
for plugin in &self.inner.config.plugins {
plugin.on_network_forgotten(network_id);
}
Ok(())
self.resume_networks_locked().await
}
/// Leaves a network: gives up what was claimed, then forgets it.
@@ -890,6 +1188,7 @@ impl Agent {
.await?
.into_iter()
.map(|stored| ConfiguredNetwork {
protocols: self.effective_protocols(stored.protocols.as_ref()),
broadcast: stored.broadcast,
network_id: stored.network_id,
name: stored.name,
@@ -958,13 +1257,54 @@ impl Agent {
let mut networks = Vec::new();
for stored in self.inner.storage.list_networks().await? {
if active.contains(&stored.network_id)
&& let Ok(status) = self.network_status(stored.network_id).await
&& let Ok(mut status) = self.network_status(stored.network_id).await
{
status.auto_start = stored.auto_start;
networks.push(status);
continue;
}
let keys = NetworkKeys::derive(&stored.name, &stored.secret);
let range = self.desired_range(stored.network_id).await?;
let range_conflicts = self.range_conflicts(stored.network_id, range).await?;
let fields = self
.inner
.storage
.vip_fields(
stored.network_id,
self.inner
.limits
.max_vips
.saturating_mul(self.inner.limits.max_vip_fields)
.saturating_add(self.inner.limits.max_vip_tombstones),
)
.await?;
let mut service_state = crate::vip::VipSet::default();
for id in fields
.iter()
.map(|e| e.vip.clone())
.collect::<std::collections::BTreeSet<_>>()
{
service_state.merge(
stored.network_id,
fields.iter().filter(|e| e.vip == id).cloned().collect(),
&self.inner.limits,
)?;
}
let vips = service_state
.configs()
.into_iter()
.map(|config| crate::vip::VipStatus {
revision: service_state.revision(&config.id),
config,
address: None,
error: Some("network is inactive".into()),
listeners: vec![],
})
.collect();
networks.push(NetworkStatus {
vips,
protocols: self.effective_protocols(stored.protocols.as_ref()),
auto_start: stored.auto_start,
broadcast: stored.broadcast,
exit_offer: stored.exit_node,
exit_via: stored
@@ -978,12 +1318,13 @@ impl Agent {
state: NetworkState::Inactive,
peers: Vec::new(),
candidates: Vec::new(),
// An inactive network has no runtime to ask; the roster and
// the range come from one. Empty, not invented.
// The range is available from signed state or the deterministic
// default even while the network has no runtime.
members: Vec::new(),
hostname_conflict: None,
range: None,
range_conflict: None,
range,
range_conflict: range_conflicts.first().map(|conflict| conflict.range),
range_conflicts,
metrics: NetworkMetrics::default(),
relay: Default::default(),
});
@@ -1039,6 +1380,7 @@ impl Agent {
/// After this returns, the state directory can be opened by another agent.
pub async fn shutdown(&self) {
self.inner.shutdown.trigger();
let lifecycle_guard = self.inner.lifecycle.lock().await;
let handles: Vec<NetworkHandle> = {
let mut networks = self.inner.networks.write().await;
@@ -1048,6 +1390,7 @@ impl Agent {
handle.stop().await;
}
drop(lifecycle_guard);
if let Some(dht) = &self.inner.config.dht {
dht.shutdown().await;
}
@@ -1064,6 +1407,7 @@ impl Agent {
for (what, handle) in [
("inbound connections", &self.inner.accept_task),
("plugin requests", &self.inner.plugin_task),
("range conflicts", &self.inner.range_task),
] {
let task = handle.lock().ok().and_then(|mut guard| guard.take());
if let Some(task) = task {
@@ -1107,16 +1451,44 @@ impl Agent {
}
}
/// What a network may propose as its overlay range, and what it cannot have.
#[derive(Debug, Clone, Copy, Default)]
struct RangePlan {
/// Proposed straight away: the configured range, when it is free here.
propose: Option<Ipv4Range>,
/// Proposed after a moment, when the configured range is another
/// network's: the range derived from this network's own id.
fallback: Option<Ipv4Range>,
/// The configured range, when this network cannot have it.
conflict: Option<Ipv4Range>,
/// Processes coalesced conflict notifications without blocking a network actor.
async fn range_conflict_loop(inner: Weak<Inner>, mut updates: watch::Receiver<()>) {
loop {
let Some(owner) = inner.upgrade() else {
return;
};
let shutdown = owner.shutdown.clone();
drop(owner);
tokio::select! {
_ = shutdown.wait() => return,
result = updates.changed() => if result.is_err() { return; },
}
let Some(inner) = inner.upgrade() else {
return;
};
let agent = Agent { inner };
let _guard = agent.inner.lifecycle.lock().await;
if agent.inner.shutdown.is_triggered() {
return;
}
let blocked: Vec<_> = agent
.inner
.networks
.read()
.await
.iter()
.filter(|(_, h)| h.range_blocked.load(std::sync::atomic::Ordering::Acquire))
.map(|(id, _)| *id)
.collect();
for id in blocked {
if let Err(err) = agent.stop_network_locked(id).await {
tracing::warn!(%err, "cannot suspend overlapping network");
}
}
if let Err(err) = agent.resume_networks_locked().await {
tracing::warn!(%err, "cannot resume suspended networks");
}
}
}
/// How long a release is given to reach the sessions it was queued on.
File diff suppressed because it is too large. Load diff
+3 -2
View File
@@ -70,7 +70,7 @@ pub(crate) struct Session {
pub(crate) conn: Connection,
/// Already encoded frame payloads, so the runtime can account for the exact
/// number of control bytes it queues.
pub(crate) outbound: mpsc::Sender<Vec<u8>>,
pub(crate) outbound: mpsc::Sender<zeroize::Zeroizing<Vec<u8>>>,
pub(crate) hostname: Option<String>,
pub(crate) capabilities: Vec<PluginCapability>,
pub(crate) broadcast: bool,
@@ -244,7 +244,7 @@ pub(crate) fn spawn(
async fn writer_task(
mut send: SendStream,
mut outbound: mpsc::Receiver<Vec<u8>>,
mut outbound: mpsc::Receiver<zeroize::Zeroizing<Vec<u8>>>,
limits: Arc<Limits>,
shutdown: Shutdown,
parent: Shutdown,
@@ -331,6 +331,7 @@ async fn reader_task(
}
};
let payload = zeroize::Zeroizing::new(payload);
let bytes = payload.len();
let envelope: Envelope = match decode(&payload) {
Ok(envelope) => envelope,
+22 -6
View File
@@ -108,6 +108,8 @@ pub struct HostnameConflict {
/// connected.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct MemberStatus {
/// Current signed claim version, used to bind cleanup confirmation.
pub record_version: u64,
/// The member's device identity.
pub endpoint_id: EndpointId,
/// The IPv4 overlay address it claimed and signed for.
@@ -209,9 +211,26 @@ pub struct NetworkMetrics {
pub data_link_failures: u64,
}
/// An active network preventing another network from starting locally.
#[derive(Debug, Clone)]
pub struct RangeConflict {
/// The blocking network's public identifier.
pub network_id: NetworkId,
/// Its configured name.
pub name: NetworkName,
/// Its overlapping overlay range.
pub range: crate::state::Ipv4Range,
}
/// Status of one network.
#[derive(Debug, Clone)]
pub struct NetworkStatus {
/// Shared service configuration with this client's routing observations.
pub vips: Vec<crate::vip::VipStatus>,
/// Enabled local data-plane protocols, in preference order.
pub protocols: Vec<String>,
/// Whether this network is requested to start or resume automatically.
pub auto_start: bool,
/// Local broadcast participation in this network.
pub broadcast: bool,
/// This agent offers itself as an exit node in this network.
@@ -242,13 +261,10 @@ pub struct NetworkStatus {
pub hostname_conflict: Option<HostnameConflict>,
/// The overlay range this network uses, once it has one.
pub range: Option<crate::state::Ipv4Range>,
/// The range it could not have, because another network on this agent
/// already holds it.
///
/// One agent has one interface, so an address belongs to one network.
/// This network waits to adopt whatever its members settle on instead of
/// proposing something it could not route.
/// The first overlapping active range; `range_conflicts` identifies all blockers.
pub range_conflict: Option<crate::state::Ipv4Range>,
/// Active networks that would be suspended by switching to this network.
pub range_conflicts: Vec<RangeConflict>,
/// Per-network counters.
pub metrics: NetworkMetrics,
/// What has gone through a peer in the middle, in both directions.
+42 -2
View File
@@ -158,6 +158,22 @@ pub enum TransportPolicy {
/// object where that is possible (notably [`Limits::max_frame_len`]).
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Limits {
/// Maximum active VIP identities per network.
pub max_vips: usize,
/// Maximum permanent VIP deletion tombstones per network.
pub max_vip_tombstones: usize,
/// Maximum concurrent active TCP port checks across the agent.
pub max_vip_checks: usize,
/// Maximum signed fields per VIP, including removed fields.
pub max_vip_fields: usize,
/// Maximum listeners per VIP.
pub max_vip_listeners: usize,
/// Maximum backend endpoints per listener.
pub max_vip_backends: usize,
/// Maximum simultaneous local VIP flow bindings.
pub max_vip_flows: usize,
/// Maximum incomplete IPv4 datagrams per network in VIP reassembly.
pub max_vip_fragments: usize,
/// Largest accepted control frame payload, in bytes.
pub max_frame_len: usize,
/// Largest accepted hostname, in bytes.
@@ -166,12 +182,16 @@ pub struct Limits {
pub max_capabilities: usize,
/// Largest opaque plugin payload, in bytes.
pub max_capability_data_len: usize,
/// Largest private transport negotiation payload (also hard-capped at 4096 bytes).
pub max_transport_payload_len: usize,
/// Largest accepted echo payload in a ping/pong exchange, in bytes.
pub max_echo_payload_len: usize,
/// Largest accepted free-text reason string, in bytes.
pub max_reason_len: usize,
/// Largest number of signed records accepted in one snapshot.
pub max_state_records: usize,
/// Maximum durable retirement markers retained per network.
pub max_retirements: usize,
/// Deadline for the whole handshake.
pub handshake_timeout: Duration,
/// Deadline for one outbound dial attempt.
@@ -201,13 +221,23 @@ pub struct Limits {
impl Default for Limits {
fn default() -> Self {
Self {
max_vips: 64,
max_vip_tombstones: 1024,
max_vip_checks: 32,
max_vip_fields: 192,
max_vip_listeners: 16,
max_vip_backends: 32,
max_vip_flows: 4096,
max_vip_fragments: 64,
max_frame_len: 64 * 1024,
max_hostname_len: 255,
max_capabilities: 16,
max_transport_payload_len: 4096,
max_capability_data_len: 4 * 1024,
max_echo_payload_len: 4 * 1024,
max_reason_len: 256,
max_state_records: crate::state::MAX_RECORDS_PER_MESSAGE,
max_retirements: 1024,
handshake_timeout: Duration::from_secs(10),
dial_timeout: Duration::from_secs(10),
write_timeout: Duration::from_secs(30),
@@ -325,6 +355,9 @@ pub struct AgentConfig {
pub reconnect: ReconnectPolicy,
/// IP plugins whose capabilities are announced and dispatched.
pub plugins: Vec<SharedPlugin>,
/// Initial ordered selection for networks without a saved override.
/// `None` enables all registered plugins; an empty list is control-only.
pub default_protocols: Option<Vec<String>>,
/// Custom packet transports registered per protocol.
pub custom_transports:
std::collections::HashMap<String, Arc<dyn crate::dataplane::PacketTransport>>,
@@ -339,7 +372,11 @@ pub struct AgentConfig {
pub interface_name: String,
/// The interface MTU.
pub interface_mtu: u32,
/// The IPv4 overlay range this agent proposes.
/// Whether new networks derive their IPv4 range from their network id.
///
/// Enabled by default. Explicit ranges and IPv4 opt-out disable this.
pub overlay_ipv4_auto: bool,
/// The IPv4 overlay range this agent proposes when automatic selection is off.
///
/// Addresses are allocated from it and recorded in signed state, so a
/// participant keeps the same one across restarts. A network that has
@@ -369,11 +406,13 @@ impl AgentConfig {
limits: Limits::default(),
reconnect: ReconnectPolicy::default(),
plugins: Vec::new(),
default_protocols: None,
custom_transports: std::collections::HashMap::new(),
tun_factory: None,
interface_name: "tsun0".to_string(),
interface_mtu: 1280,
overlay_ipv4_range: Some(crate::state::DEFAULT_IPV4_RANGE),
overlay_ipv4_auto: true,
overlay_ipv4_range: None,
}
}
@@ -445,6 +484,7 @@ impl AgentConfig {
/// Sets the IPv4 overlay range this agent proposes, or disables IPv4.
pub fn with_overlay_ipv4_range(mut self, range: Option<crate::state::Ipv4Range>) -> Self {
self.overlay_ipv4_auto = false;
self.overlay_ipv4_range = range;
self
}
+27
View File
@@ -40,6 +40,33 @@ pub use tunnel::{GenericTunnelPlugin, TunnelCodec};
/// Maximum length of a plugin protocol identifier.
pub const MAX_PROTOCOL_ID_LEN: usize = 32;
/// Validates bounded local protocol preferences before storing or applying them.
pub(crate) fn validate_protocols(protocols: &[String], max: usize) -> crate::error::Result<()> {
if protocols.len() > max {
return Err(crate::Error::InvalidProtocols(format!(
"at most {max} protocols are allowed"
)));
}
for (index, name) in protocols.iter().enumerate() {
if name.is_empty()
|| name.len() > MAX_PROTOCOL_ID_LEN
|| !name
.bytes()
.all(|c| c.is_ascii_alphanumeric() || c == b'-' || c == b'_')
{
return Err(crate::Error::InvalidProtocols(
"names must be 1–32 ASCII letters, digits, hyphens or underscores".into(),
));
}
if protocols[..index].contains(name) {
return Err(crate::Error::InvalidProtocols(format!(
"`{name}` is listed more than once"
)));
}
}
Ok(())
}
/// An announcement of one IP plugin's capability.
///
/// `data` is opaque to the core. Nothing in it may be interpreted as a shell
+10 -2
View File
@@ -492,11 +492,12 @@ impl RelayHub {
}
}
/// Revokes all links and topology involving a departed authenticated peer.
pub fn remove_peer(&self, peer: EndpointId) {
/// Retires data channels without removing an authenticated peer's topology.
pub fn clear_peer_links(&self, peer: EndpointId) {
let mut state = self.state();
if let Some(links) = state.raw.remove(peer.as_bytes()) {
for raw in links {
raw.link.close();
raw.reader.abort();
}
}
@@ -511,6 +512,13 @@ impl RelayHub {
link.close();
}
}
state.publish();
}
/// Revokes all links and topology involving a departed authenticated peer.
pub fn remove_peer(&self, peer: EndpointId) {
self.clear_peer_links(peer);
let mut state = self.state();
state.members.remove(peer.as_bytes());
state.graph.remove(peer.as_bytes());
for neighbors in state.graph.values_mut() {
@@ -0,0 +1,350 @@
//! Authenticated probes of the actual data link, independent of control RTT.
//! Keys are negotiated privately for each physical link. No fixed wire magic.
use super::{PacketLink, SharedLink, TransportError};
use crate::dataplane::SharedPlugin;
use crate::overlay::RoutingTable;
use crate::{BoxFuture, NetworkId};
use bytes::Bytes;
use hmac::{Hmac, KeyInit, Mac};
use iroh::EndpointId;
use serde::{Deserialize, Serialize};
use sha2::Sha256;
use std::collections::VecDeque;
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};
use tokio::sync::{mpsc, watch};
#[derive(Clone, Serialize, Deserialize, zeroize::ZeroizeOnDrop)]
pub(crate) struct Settings {
marker: [u8; 16],
key: [u8; 32],
}
impl Settings {
pub(crate) fn fresh() -> Self {
Self {
marker: rand::random(),
key: rand::random(),
}
}
}
impl std::fmt::Debug for Settings {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str("HealthSettings([REDACTED])")
}
}
struct Samples {
pending: VecDeque<(u64, Instant)>,
recent: VecDeque<bool>,
}
impl Samples {
fn new() -> Self {
Self {
pending: VecDeque::new(),
recent: VecDeque::new(),
}
}
fn sent(&mut self, seq: u64, now: Instant) {
self.pending.push_back((seq, now));
}
fn acknowledge(&mut self, seq: u64) {
if let Some(i) = self.pending.iter().position(|(n, _)| *n == seq) {
self.pending.remove(i);
self.recent.push_back(true);
}
}
fn expire(&mut self, now: Instant) {
while self
.pending
.front()
.is_some_and(|(_, at)| now.duration_since(*at) >= Duration::from_secs(3))
{
self.pending.pop_front();
self.recent.push_back(false);
}
while self.recent.len() > 20 {
self.recent.pop_front();
}
}
fn degraded(&self) -> bool {
self.recent.len() >= 15
&& self.recent.iter().filter(|ok| !**ok).count() * 100 >= self.recent.len() * 60
}
}
// This prefix is inside end-to-end encryption. It never appears on the UDP wire.
const IP_MAGIC: &[u8; 16] = b"tsunagi-health01";
pub(crate) fn ip_payload(packet: &[u8]) -> Option<&[u8]> {
(packet.len() >= 36 && packet[0] == 0x45 && packet[9] == 253 && &packet[20..36] == IP_MAGIC)
.then(|| &packet[36..])
}
/// Inject probes through the same encrypted tunnel as application IP packets.
/// This matters for pure WG: raw UDP probes could pass DPI while WG is blocked.
pub(crate) struct ProbeRoute {
pub(crate) plugin: SharedPlugin,
pub(crate) routes: Arc<RoutingTable>,
pub(crate) network: NetworkId,
pub(crate) peer: EndpointId,
pub(crate) max_probe_size: usize,
}
impl ProbeRoute {
fn send(&self, payload: Bytes) -> bool {
let Some((source, destination)) = self.routes.probe_addresses(self.network, self.peer)
else {
return false;
};
let mut packet = vec![0; 36];
packet[0] = 0x45;
packet[8] = 64;
packet[9] = 253;
packet[12..16].copy_from_slice(&source.octets());
packet[16..20].copy_from_slice(&destination.octets());
packet[20..36].copy_from_slice(IP_MAGIC);
let length = (packet.len() + payload.len()) as u16;
packet[2..4].copy_from_slice(&length.to_be_bytes());
let sum = packet[..20]
.as_chunks::<2>()
.0
.iter()
.map(|b| u32::from(u16::from_be_bytes(*b)))
.sum::<u32>();
let sum = (sum & 0xffff) + (sum >> 16);
packet[10..12].copy_from_slice(&(!(sum as u16)).to_be_bytes());
packet.extend_from_slice(&payload);
self.plugin
.carry(self.network, self.peer, Bytes::from(packet))
}
}
/// A monitored physical link. It forwards application bytes unchanged.
pub(crate) struct HealthLink {
probes: mpsc::Sender<Bytes>,
raw: SharedLink,
settings: Mutex<Option<Settings>>,
rx: tokio::sync::Mutex<mpsc::Receiver<Bytes>>,
stop: watch::Sender<bool>,
reason: Mutex<Option<String>>,
}
impl std::fmt::Debug for HealthLink {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("HealthLink")
.field("raw", &self.raw)
.finish_non_exhaustive()
}
}
fn lock<T>(m: &Mutex<T>) -> std::sync::MutexGuard<'_, T> {
match m.lock() {
Ok(g) => g,
Err(p) => p.into_inner(),
}
}
impl HealthLink {
pub(crate) fn new(
raw: SharedLink,
local: EndpointId,
settings: Option<Settings>,
route: Option<ProbeRoute>,
) -> Arc<Self> {
let (tx, rx) = mpsc::channel(64);
let (probe_tx, mut probe_rx) = mpsc::channel(64);
let (stop, _) = watch::channel(false);
let link = Arc::new(Self {
probes: probe_tx,
raw: raw.clone(),
settings: Mutex::new(settings),
rx: tokio::sync::Mutex::new(rx),
stop,
reason: Mutex::new(None),
});
let weak = Arc::downgrade(&link);
let mut stopped = link.stop.subscribe();
tokio::spawn(async move {
let mut interval = tokio::time::interval(Duration::from_secs(1));
interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
let mut samples = Samples::new();
let mut sequence = rand::random::<u64>();
let mut since: Option<Instant> = None;
let mut marker = None;
loop {
tokio::select! {
biased;
_=stopped.changed()=>break,
_=interval.tick()=>{
let Some(link)=weak.upgrade()else{break;};let Some(settings)=lock(&link.settings).clone()else{continue;};
if marker != Some(settings.marker) { marker = Some(settings.marker); samples = Samples::new(); since = None; }
let start=*since.get_or_insert_with(Instant::now);let now=Instant::now();samples.expire(now);
if now.duration_since(start)>=Duration::from_secs(20) && samples.degraded() {
*lock(&link.reason)=Some("data delivery checks: at least 60% loss over the last 20 probes".into());break;
}
sequence=sequence.wrapping_add(1);
let size=route.as_ref().map_or(raw.max_datagram_size(),|route|route.max_probe_size);
let packet=probe(&settings,local,1,sequence,size.min(1000));
if let Some(route)=&route { if route.send(packet) {samples.sent(sequence,now);} }
else { let _=raw.send(packet); samples.sent(sequence,now); }
}
packet=raw.recv()=>{
let Some(packet)=packet else{break;};let Some(link)=weak.upgrade()else{break;};
let settings=lock(&link.settings).clone();
if route.is_none() && let Some(settings)=settings && packet.starts_with(&settings.marker) {
if let Some((kind,seq))=verify(&settings,raw.peer(),&packet) {
if kind==1 {let _=raw.send(probe(&settings,local,2,seq,packet.len()));}
else if kind==2 {samples.acknowledge(seq);}
}
} else {let _=tx.try_send(packet);}
}
packet=probe_rx.recv(),if route.is_some()=>{
let Some(packet)=packet else {break;};let Some(link)=weak.upgrade()else{break;};
let Some(settings)=lock(&link.settings).clone()else{continue;};
if let Some((kind,seq))=verify(&settings,raw.peer(),&packet) {
if kind==1 { if let Some(route)=&route { route.send(probe(&settings,local,2,seq,packet.len().min(route.max_probe_size))); } }
else if kind==2 {samples.acknowledge(seq);}
}
}
}
}
raw.close();
if let Some(link) = weak.upgrade() {
link.stop.send_replace(true);
}
});
link
}
pub(crate) fn receive_probe(&self, packet: Bytes) {
let _ = self.probes.try_send(packet);
}
pub(crate) fn configure(&self, settings: Settings) {
*lock(&self.settings) = Some(settings);
}
}
fn probe(settings: &Settings, sender: EndpointId, kind: u8, seq: u64, size: usize) -> Bytes {
let mut bytes = vec![0; size.max(57)];
bytes[..16].copy_from_slice(&settings.marker);
bytes[16] = kind;
bytes[17..25].copy_from_slice(&seq.to_be_bytes());
rand::fill(&mut bytes[25..size.max(57) - 32]);
let tail = bytes.len() - 32;
if let Ok(mut mac) = <Hmac<Sha256> as KeyInit>::new_from_slice(&settings.key) {
mac.update(sender.as_bytes());
mac.update(&bytes[..tail]);
bytes[tail..].copy_from_slice(&mac.finalize().into_bytes());
}
Bytes::from(bytes)
}
fn verify(settings: &Settings, sender: EndpointId, bytes: &[u8]) -> Option<(u8, u64)> {
if bytes.len() < 57 || !bytes.starts_with(&settings.marker) {
return None;
}
let mut mac = <Hmac<Sha256> as KeyInit>::new_from_slice(&settings.key).ok()?;
let tail = bytes.len() - 32;
mac.update(sender.as_bytes());
mac.update(&bytes[..tail]);
mac.verify_slice(&bytes[tail..]).ok()?;
Some((
bytes[16],
u64::from_be_bytes(bytes.get(17..25)?.try_into().ok()?),
))
}
impl PacketLink for HealthLink {
fn network(&self) -> NetworkId {
self.raw.network()
}
fn peer(&self) -> EndpointId {
self.raw.peer()
}
fn max_datagram_size(&self) -> usize {
self.raw.max_datagram_size()
}
fn send(&self, payload: Bytes) -> Result<(), TransportError> {
if self.is_closed() {
Err(TransportError::Closed)
} else {
self.raw.send(payload)
}
}
fn send_flow(
&self,
payload: Bytes,
flow: super::super::routing::FlowId,
) -> Result<(), TransportError> {
if self.is_closed() {
Err(TransportError::Closed)
} else {
self.raw.send_flow(payload, flow)
}
}
fn recv(&self) -> BoxFuture<'_, Option<Bytes>> {
Box::pin(async move {
let mut stop = self.stop.subscribe();
if *stop.borrow() {
return None;
}
let mut rx = self.rx.lock().await;
tokio::select! {biased;_=stop.changed()=>None,packet=rx.recv()=>packet}
})
}
fn closed(&self) -> BoxFuture<'_, ()> {
Box::pin(async move {
let mut stop = self.stop.subscribe();
if !*stop.borrow() {
let _ = stop.changed().await;
}
})
}
fn is_closed(&self) -> bool {
*self.stop.borrow() || self.raw.is_closed()
}
fn close(&self) {
self.stop.send_replace(true);
self.raw.close();
}
fn failure_reason(&self) -> Option<String> {
lock(&self.reason)
.clone()
.or_else(|| self.raw.failure_reason())
}
fn path_description(&self) -> String {
self.raw.path_description()
}
}
impl Drop for HealthLink {
fn drop(&mut self) {
self.stop.send_replace(true);
self.raw.close();
}
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
#[test]
fn no_false_failure_before_window_but_persistent_loss_is_detected() {
let mut samples = Samples::new();
let now = Instant::now();
for seq in 0..20 {
samples.sent(seq, now);
if seq % 5 == 0 {
samples.acknowledge(seq);
}
}
assert!(!samples.degraded());
samples.expire(now + Duration::from_secs(4));
assert!(samples.degraded());
for seq in 20..40 {
samples.sent(seq, now);
samples.acknowledge(seq);
}
samples.expire(now + Duration::from_secs(4));
assert!(!samples.degraded());
}
#[test]
fn a_forged_or_reflected_probe_cannot_claim_delivery() {
let a = crate::identity::DeviceIdentity::generate().endpoint_id();
let b = crate::identity::DeviceIdentity::generate().endpoint_id();
let settings = Settings::fresh();
let packet = probe(&settings, a, 1, 42, 1100);
assert_eq!(verify(&settings, a, &packet), Some((1, 42)));
assert!(verify(&settings, b, &packet).is_none());
let mut forged = packet.to_vec();
forged[16] = 2;
assert!(verify(&settings, a, &forged).is_none());
}
}
@@ -97,6 +97,9 @@ impl IrohLink {
}
impl PacketLink for IrohLink {
fn close(&self) {
self.conn.close(0u32.into(), b"data link retired");
}
fn network(&self) -> NetworkId {
self.network
}
@@ -30,12 +30,14 @@
//! one plugin protocol.
mod fragments;
pub(crate) mod health;
pub mod iroh_link;
use std::sync::Arc;
use bytes::Bytes;
use iroh::EndpointId;
use serde::{Deserialize, Serialize};
use crate::BoxFuture;
use crate::identity::NetworkId;
@@ -112,6 +114,14 @@ pub trait PacketLink: Send + Sync + std::fmt::Debug + 'static {
/// keeping a task parked on [`PacketLink::closed`].
fn is_closed(&self) -> bool;
/// Explicitly retires a physical link. Implementations should wake readers.
fn close(&self) {}
/// Observed reason for retiring this link, if available.
fn failure_reason(&self) -> Option<String> {
None
}
/// A short description of the path in use, for diagnostics.
///
/// Reports what the transport actually knows. It must not invent a value.
@@ -121,6 +131,76 @@ pub trait PacketLink: Send + Sync + std::fmt::Debug + 'static {
/// A shared handle to a link.
pub type SharedLink = std::sync::Arc<dyn PacketLink>;
/// Maximum opaque pair-negotiation payload.
pub const MAX_TRANSPORT_PAYLOAD: usize = 4096;
/// Private transport bytes, never included in Debug output.
#[derive(Clone, PartialEq, Eq, Serialize, zeroize::ZeroizeOnDrop)]
pub struct TransportPayload(Vec<u8>);
impl<'de> Deserialize<'de> for TransportPayload {
fn deserialize<D: serde::Deserializer<'de>>(
deserializer: D,
) -> std::result::Result<Self, D::Error> {
struct Visitor;
impl<'de> serde::de::Visitor<'de> for Visitor {
type Value = TransportPayload;
fn expecting(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "at most {MAX_TRANSPORT_PAYLOAD} private transport bytes")
}
fn visit_seq<A: serde::de::SeqAccess<'de>>(
self,
mut sequence: A,
) -> std::result::Result<Self::Value, A::Error> {
if sequence
.size_hint()
.is_some_and(|size| size > MAX_TRANSPORT_PAYLOAD)
{
return Err(serde::de::Error::custom("transport payload too large"));
}
let mut payload = TransportPayload(Vec::with_capacity(
sequence.size_hint().unwrap_or(0).min(MAX_TRANSPORT_PAYLOAD),
));
while let Some(byte) = sequence.next_element::<u8>()? {
if payload.0.len() == MAX_TRANSPORT_PAYLOAD {
return Err(serde::de::Error::custom("transport payload too large"));
}
payload.0.push(byte);
}
Ok(payload)
}
}
deserializer.deserialize_seq(Visitor)
}
}
impl TransportPayload {
/// Wraps bytes for one authenticated control session.
pub fn new(bytes: Vec<u8>) -> Self {
Self(bytes)
}
/// Borrows private bytes; callers must not log them.
pub fn as_bytes(&self) -> &[u8] {
&self.0
}
}
impl std::fmt::Debug for TransportPayload {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str("TransportPayload([REDACTED])")
}
}
/// Private control-plane sender for transports. Contains no network secret.
pub trait ControlSender: Send + Sync + std::fmt::Debug + 'static {
/// Queues a bounded message to one authenticated member of one network.
fn send<'a>(
&'a self,
network: NetworkId,
peer: EndpointId,
protocol: &'a str,
payload: TransportPayload,
) -> BoxFuture<'a, Result<(), TransportError>>;
}
/// An inbound link a peer opened towards us.
#[derive(Debug)]
pub struct InboundLink {
@@ -139,6 +219,18 @@ pub struct InboundLink {
/// The agent owns one of these and hands links to plugins; plugins never call
/// it directly.
pub trait PacketTransport: Send + Sync + std::fmt::Debug + 'static {
/// Supplies a private sender before network activation.
fn attach_control(&self, _control: Arc<dyn ControlSender>) {}
/// Validates and queues private negotiation received from an authenticated member.
fn on_control(
&self,
_network: NetworkId,
_peer: EndpointId,
_protocol: &str,
_payload: &[u8],
) -> Result<(), TransportError> {
Err(TransportError::Declined(self.name().into()))
}
/// A short name used in diagnostics.
fn name(&self) -> &str;
@@ -184,6 +276,24 @@ impl MultiPacketTransport {
}
impl PacketTransport for MultiPacketTransport {
fn attach_control(&self, control: Arc<dyn ControlSender>) {
self.default.attach_control(control.clone());
for transport in self.by_protocol.values() {
transport.attach_control(control.clone());
}
}
fn on_control(
&self,
network: NetworkId,
peer: EndpointId,
protocol: &str,
payload: &[u8],
) -> Result<(), TransportError> {
match self.by_protocol.get(protocol) {
Some(transport) => transport.on_control(network, peer, protocol, payload),
None => Err(TransportError::Declined(protocol.into())),
}
}
fn name(&self) -> &str {
"multi"
}
+19
View File
@@ -15,6 +15,12 @@ pub type Result<T, E = Error> = std::result::Result<T, E>;
#[derive(Debug, thiserror::Error)]
#[non_exhaustive]
pub enum Error {
/// A node-history cleanup is unsafe, stale or cannot be performed.
#[error("cannot forget node: {0}")]
ForgetNode(String),
/// An ordered local data-plane protocol selection is invalid.
#[error("invalid protocol selection: {0}")]
InvalidProtocols(String),
/// The supplied network name does not satisfy the documented rules.
#[error("invalid network name: {0}")]
InvalidNetworkName(&'static str),
@@ -97,6 +103,19 @@ pub enum Error {
#[error("network {0} is not configured")]
NetworkUnknown(NetworkId),
/// An overlay range overlaps networks already active on this agent.
#[error(
"network {network} cannot start: {range} overlaps active networks {conflicts:?}; switch to it to suspend those networks"
)]
NetworkRangeConflict {
/// Network that could not start.
network: NetworkId,
/// Its agreed or proposed range.
range: crate::state::Ipv4Range,
/// Active networks that must be suspended first.
conflicts: Vec<NetworkId>,
},
/// No session with that peer exists in the given network.
#[error("no authenticated session with peer {peer} in network {network}")]
NoSuchPeer {
+352 -2
View File
@@ -171,6 +171,33 @@ pub fn default_control_socket_path() -> PathBuf {
#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)]
#[non_exhaustive]
pub enum Request {
/// Save a complete version-checked service draft or its deletion.
SaveVip {
/// Exact network id.
network_id: String,
/// Entire desired configuration.
config: crate::vip::VipConfig,
/// Revision originally displayed; absent for creation.
expected: Option<String>,
/// Remove the service permanently.
delete: bool,
},
/// Retire the selected offline member's observed claim.
ForgetNode {
/// Exact public network id.
network_id: String,
/// Exact device identity selected by the client.
peer: String,
/// Claim version displayed when the user selected the node.
expected_version: u64,
},
/// Replace one network's ordered local data-plane selection.
SetProtocols {
/// Public network identifier.
network_id: String,
/// Protocol names, best first; empty selects control only.
protocols: Vec<String>,
},
/// Report what the agent is doing.
Status,
/// Set local broadcast participation in one configured network.
@@ -255,6 +282,24 @@ pub enum Request {
impl std::fmt::Debug for Request {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
Request::SaveVip {
network_id, config, ..
} => write!(f, "SaveVip {{ {network_id}, {} }}", config.id),
Request::ForgetNode {
network_id,
peer,
expected_version,
} => write!(
f,
"ForgetNode {{ {network_id}, peer: {peer}, version: {expected_version} }}"
),
Request::SetProtocols {
network_id,
protocols,
} => write!(
f,
"SetProtocols {{ {network_id}, protocols: {protocols:?} }}"
),
Request::Status => f.write_str("Status"),
Request::SetBroadcast {
network_id,
@@ -286,6 +331,12 @@ impl std::fmt::Debug for Request {
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[non_exhaustive]
pub enum Response {
/// Accepted configuration with fresh local routing observations.
Vips(Vec<crate::vip::VipStatus>),
/// An observed offline claim was durably retired.
NodeForgotten(ForgottenNodeReport),
/// Accepted ordered protocol selection.
Protocols(Vec<String>),
/// A status report.
Status(Box<StatusReport>),
/// Accepted local broadcast participation.
@@ -312,6 +363,192 @@ pub enum Response {
Error(String),
}
/// The durable result of an offline-node cleanup.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct ForgottenNodeReport {
/// Selected device identity.
pub peer: String,
/// The retired claim version, not a future-version ban.
pub record_version: u64,
/// Peers to which cleanup was immediately queued.
pub peers_told: u32,
}
/// Saves a whole VIP draft through the running agent.
pub async fn save_vip(
socket: &Path,
network_id: &str,
config: crate::vip::VipConfig,
expected: Option<String>,
delete: bool,
) -> Result<Vec<crate::vip::VipStatus>> {
match exchange(
socket,
&Request::SaveVip {
network_id: network_id.into(),
config,
expected,
delete,
},
EXCHANGE_TIMEOUT,
)
.await?
{
Response::Vips(vips) => Ok(vips),
Response::Error(reason) => Err(Error::Rejected { reason }),
_ => Err(Error::Rejected {
reason: "unexpected VIP response".into(),
}),
}
}
/// Resolves a service by exact name, full id or unique id prefix.
pub fn resolve_vip<'a>(
network: &'a NetworkReport,
selector: &str,
) -> std::result::Result<&'a crate::vip::VipStatus, String> {
let matches: Vec<_> = network
.vips
.iter()
.filter(|v| v.config.name == selector || v.config.id.starts_with(selector))
.collect();
match matches.as_slice() {
[vip] => Ok(vip),
[] => Err(format!("no VIP matches `{selector}`")),
_ => Err(format!(
"ambiguous VIP `{selector}`: {}",
matches
.iter()
.map(|v| format!("{} ({})", v.config.name, v.config.id))
.collect::<Vec<_>>()
.join(", ")
)),
}
}
/// Resolves a node by exact hostname, full id, unique id prefix or overlay IPv4.
/// Ambiguity is reported with names and full ids and never guessed.
pub fn resolve_node<'a>(
network: &'a NetworkReport,
selector: &str,
) -> std::result::Result<&'a MemberReport, String> {
let selector = selector.trim();
if selector.is_empty() {
return Err("select a node by name, id or IPv4 address".into());
}
if let Some(member) = network.members.iter().find(|m| m.endpoint_id == selector) {
return Ok(member);
}
let prefix = selector.trim_end_matches('…');
let matches: Vec<_> = network
.members
.iter()
.filter(|m| {
m.hostname.as_deref() == Some(selector)
|| m.overlay_address_v4.as_deref() == Some(selector)
|| (!prefix.is_empty() && m.endpoint_id.starts_with(prefix))
})
.collect();
match matches.as_slice() {
[one] => Ok(one),
[] => Err(format!(
"no signed node matches `{selector}` in `{}`",
network.name
)),
many => Err(format!(
"`{selector}` is ambiguous; use a full id or unique id prefix:\n{}",
many.iter()
.map(|m| format!(
" {} {} {}",
m.hostname.as_deref().unwrap_or("unnamed"),
m.endpoint_id,
m.overlay_address_v4.as_deref().unwrap_or("no IPv4")
))
.collect::<Vec<_>>()
.join("\n")
)),
}
}
/// Retires an exact selected claim; an online or changed target is rejected.
pub async fn forget_node(
socket: &Path,
network: &str,
peer: &str,
expected_version: u64,
) -> Result<ForgottenNodeReport> {
let request = Request::ForgetNode {
network_id: network.into(),
peer: peer.into(),
expected_version,
};
match exchange(socket, &request, EXCHANGE_TIMEOUT).await? {
Response::NodeForgotten(report) => Ok(report),
Response::Error(reason) => Err(Error::ForgetNode(reason)),
other => Err(Error::Storage(format!("unexpected response: {other:?}"))),
}
}
#[cfg(test)]
mod node_selector_tests {
use super::*;
#[test]
fn exact_names_addresses_and_unique_prefixes_select_the_same_signed_node() {
let node = MemberReport {
endpoint_id: "abcdef01".into(),
hostname: Some("laptop".into()),
overlay_address_v4: Some("10.2.0.3".into()),
record_version: 42,
..Default::default()
};
let network = NetworkReport {
name: "home".into(),
members: vec![node.clone()],
..Default::default()
};
for selector in ["laptop", "10.2.0.3", "abcdef01", "abc…"] {
assert_eq!(resolve_node(&network, selector), Ok(&node));
}
for selector in ["", "LAPTOP", "missing"] {
assert!(resolve_node(&network, selector).is_err());
}
}
#[test]
fn collisions_between_names_addresses_and_prefixes_require_an_explicit_identity() {
let first = MemberReport {
endpoint_id: "abcdef01".into(),
hostname: Some("duplicate".into()),
overlay_address_v4: Some("10.2.0.3".into()),
..Default::default()
};
let second = MemberReport {
endpoint_id: "abcdef02".into(),
hostname: Some("duplicate".into()),
overlay_address_v4: Some("10.2.0.3".into()),
..Default::default()
};
let third = MemberReport {
endpoint_id: "12345678".into(),
hostname: Some("abc".into()),
..Default::default()
};
let network = NetworkReport {
name: "home".into(),
members: vec![first.clone(), second, third],
..Default::default()
};
for selector in ["duplicate", "10.2.0.3", "abc"] {
let result = resolve_node(&network, selector);
assert!(result.is_err_and(|error| error.contains("ambiguous")
&& error.contains("abcdef01")
&& error.contains("abcdef02")
&& error.contains("duplicate")));
}
assert_eq!(resolve_node(&network, "abcdef01"), Ok(&first));
}
}
/// What happened when a network was stopped or started.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct ActiveReport {
@@ -324,6 +561,9 @@ pub struct ActiveReport {
/// Stopping something already stopped is not an error — it is the state
/// asked for — but it is worth saying which happened.
pub changed: bool,
/// Networks suspended by switching to this network.
#[serde(default)]
pub suspended: Vec<RangeConflictReport>,
}
/// What happened when a network was joined.
@@ -345,6 +585,13 @@ pub struct JoinedReport {
/// Almost always a mistyped secret, and the one thing that makes two
/// sections of a report look like one network.
pub name_shared_with: Option<String>,
/// Whether it started; false means it is configured but blocked by overlap.
#[serde(default = "joined_active_default")]
pub active: bool,
}
fn joined_active_default() -> bool {
true
}
/// What happened when a network was left.
@@ -367,6 +614,8 @@ pub struct LeftReport {
/// Everything the agent is doing, in one snapshot.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct StatusReport {
/// Data-plane protocols this agent can enable at runtime.
pub available_protocols: Vec<ProtocolReport>,
/// This device's persistent endpoint id.
pub endpoint_id: String,
/// Hostname announced to peers.
@@ -437,9 +686,37 @@ pub struct DnsReport {
pub names: u32,
}
/// An active network with an overlapping overlay range.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct RangeConflictReport {
/// Public network id, for unambiguous selection.
pub network_id: String,
/// Configured network name.
pub name: String,
/// The overlapping range.
pub range: String,
}
/// A data-plane protocol available in the running agent.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct ProtocolReport {
/// Stable protocol name used by the CLI and network negotiation.
pub name: String,
/// A short description of encryption and transport.
pub summary: String,
}
/// One network.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct NetworkReport {
/// Shared VIPs and this client's routing observations.
#[serde(default)]
pub vips: Vec<crate::vip::VipStatus>,
/// Enabled local data-plane protocols, in preference order.
pub protocols: Vec<String>,
/// Whether the network starts or resumes automatically when its range is free.
#[serde(default)]
pub auto_start: bool,
/// Whether this agent participates in LAN broadcasts in this network.
#[serde(default)]
pub broadcast: bool,
@@ -482,6 +759,9 @@ pub struct NetworkReport {
/// The range it could not have, because another network on this agent
/// already holds it.
pub range_conflict: Option<String>,
/// Networks that switching to this network would suspend.
#[serde(default)]
pub range_conflicts: Vec<RangeConflictReport>,
/// The overlay, when a protocol is running one.
pub overlay: Option<OverlayReport>,
/// This agent's own name, when another member claimed it first.
@@ -524,6 +804,8 @@ pub struct HostnameConflictReport {
/// One member of the network, from signed state.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct MemberReport {
/// Signed claim version used to reject stale cleanup requests.
pub record_version: u64,
/// The member's device identity.
pub endpoint_id: String,
/// The IPv4 overlay address it claimed and signed for.
@@ -693,6 +975,33 @@ impl OverlayPeerReport {
/// running and what they can report or change. That is what keeps this module
/// free of any knowledge of them.
pub trait ReportSource: Send + Sync + 'static {
/// Atomically saves a complete VIP draft using its original revision.
fn save_vip(
&self,
_network: String,
_config: crate::vip::VipConfig,
_expected: Option<String>,
_delete: bool,
) -> BoxFuture<'_, std::result::Result<Vec<crate::vip::VipStatus>, String>> {
Box::pin(async { Err("this agent cannot manage VIPs".into()) })
}
/// Persists and announces a version-scoped node cleanup.
fn forget_node(
&self,
_network: String,
_peer: String,
_expected_version: u64,
) -> BoxFuture<'_, std::result::Result<ForgottenNodeReport, String>> {
Box::pin(async { Err("this agent cannot forget node history".into()) })
}
/// Saves and applies a local ordered protocol set for one network.
fn set_protocols(
&self,
_network_id: String,
_protocols: Vec<String>,
) -> BoxFuture<'_, std::result::Result<Vec<String>, String>> {
Box::pin(async { Err("this agent cannot change protocols".into()) })
}
/// Produces a fresh report.
fn report(&self) -> BoxFuture<'_, StatusReport>;
@@ -822,7 +1131,7 @@ pub const EXCHANGE_TIMEOUT: Duration = Duration::from_secs(5);
///
/// Bump it whenever [`Request`], [`Response`] or anything they contain
/// changes shape.
pub const CONTROL_PROTOCOL: u32 = u32::from_be_bytes([b'T', b'S', b'N', 18]);
pub const CONTROL_PROTOCOL: u32 = u32::from_be_bytes([b'T', b'S', b'N', 22]);
/// Reads one request off an accepted stream, answers it, writes the response.
///
@@ -846,6 +1155,30 @@ where
}
};
let response = match request {
Request::SaveVip {
network_id,
config,
expected,
delete,
} => match source.save_vip(network_id, config, expected, delete).await {
Ok(vips) => Response::Vips(vips),
Err(reason) => Response::Error(reason),
},
Request::ForgetNode {
network_id,
peer,
expected_version,
} => match source.forget_node(network_id, peer, expected_version).await {
Ok(report) => Response::NodeForgotten(report),
Err(reason) => Response::Error(reason),
},
Request::SetProtocols {
network_id,
protocols,
} => match source.set_protocols(network_id, protocols).await {
Ok(protocols) => Response::Protocols(protocols),
Err(reason) => Response::Error(reason),
},
Request::Status => Response::Status(Box::new(source.report().await)),
Request::SetBroadcast {
network_id,
@@ -1001,7 +1334,24 @@ pub async fn join_network_with_broadcast(
}
}
/// Changes one network's broadcast participation while the agent runs.
/// Replaces the enabled data-plane protocols of one configured network.
pub async fn set_protocols(
socket: &Path,
network_id: &str,
protocols: Vec<String>,
) -> Result<Vec<String>> {
let request = Request::SetProtocols {
network_id: network_id.to_owned(),
protocols,
};
match exchange(socket, &request, EXCHANGE_TIMEOUT).await? {
Response::Protocols(protocols) => Ok(protocols),
Response::Error(reason) => Err(Error::Storage(reason)),
other => Err(Error::Storage(format!("unexpected response: {other:?}"))),
}
}
/// Changes local broadcast participation in a network.
pub async fn set_broadcast(
path: impl AsRef<Path>,
network_id: &str,
+1
View File
@@ -51,6 +51,7 @@ pub mod storage;
pub mod task;
#[cfg(feature = "testing")]
pub mod testing;
pub mod vip;
pub use agent::{Agent, AgentStatus, Event, NetworkStatus, PeerStatus};
pub use config::{AgentConfig, Limits, ReconnectPolicy, StoragePaths, TransportPolicy};
@@ -84,6 +84,11 @@ pub trait ExitHostRules: Send + Sync + std::fmt::Debug + 'static {
/// startup to take away what a crashed run left.
fn clear<'a>(&'a self, interface: &'a str) -> BoxFuture<'a, ()>;
/// Refreshes volatile local socket protection without replacing exit routes.
fn refresh<'a>(&'a self, _plan: &'a ExitHostPlan) -> BoxFuture<'a, Option<RuleOutcome>> {
Box::pin(async { None })
}
/// Whether using an exit node on this host needs the underlay addresses
/// the agent talks to, because the host cannot exempt the agent's own
/// traffic from the default route by user id (Windows). The default is no.
@@ -33,12 +33,13 @@ const CHAIN: &str = "INPUT";
/// Fixed places `iptables` is installed. Never `PATH`: the agent is
/// privileged and a directory a user controls must not decide what it runs.
const IPTABLES_DIRECTORIES: [&str; 5] = [
const IPTABLES_DIRECTORIES: [&str; 6] = [
"/usr/sbin/iptables",
"/sbin/iptables",
"/usr/bin/iptables",
"/bin/iptables",
"/usr/local/sbin/iptables",
"/run/current-system/sw/bin/iptables",
];
/// The first candidate that exists.
@@ -328,7 +329,23 @@ fn clear_firewall(run: Run<'_>, interface: &str) {
/// Runs the real `iptables`.
pub(super) fn run_iptables(args: &[String]) -> Ran {
let Some(program) = pick_iptables(|path| std::path::Path::new(path).is_file()) else {
run_firewall(false, args)
}
pub(super) fn run_firewall(ipv6: bool, args: &[String]) -> Ran {
let v6_paths: Vec<_> = IPTABLES_DIRECTORIES
.iter()
.map(|p| p.replace("iptables", "ip6tables"))
.collect();
let v6_program = v6_paths
.iter()
.find(|path| std::path::Path::new(path).is_file());
let program = if ipv6 {
v6_program.map(String::as_str)
} else {
pick_iptables(|path| std::path::Path::new(path).is_file())
};
let Some(program) = program else {
return Ran::Missing;
};
match std::process::Command::new(program).args(args).output() {
@@ -350,11 +367,17 @@ pub(super) fn run_iptables(args: &[String]) -> Ran {
#[derive(Debug)]
pub struct LinuxHostRules {
pub(super) route: RouteHandle,
/// Interface, sorted socket snapshot and whether exit routes committed.
pub(super) underlay_sockets:
std::sync::Mutex<Option<(String, Vec<super::linux_exit_mark::Listener>, bool)>>,
}
impl LinuxHostRules {
pub(crate) fn new(route: RouteHandle) -> Self {
Self { route }
Self {
route,
underlay_sockets: std::sync::Mutex::new(None),
}
}
async fn set_route(&self, interface: String, source: Ipv4Addr) -> RuleOutcome {
@@ -25,18 +25,19 @@
//!
//! ```text
//! table 0x7473: default dev <if>
//! 5280: from all uidrange <euid>-<euid> lookup main
//! 5205: from all fwmark 0x40000000/0x40000000 lookup main
//! 5206: from all uidrange <euid>-<euid> lookup main
//! 5290: from all lookup main suppress_prefixlength 0
//! 5300: from all lookup 0x7473
//! ```
//!
//! The first keeps the agent's *own* sockets off the tunnel: its QUIC, relay,
//! The first two keep the agent's *own* sockets and marked replies off the tunnel: its QUIC, relay,
//! DHT and DNS traffic would otherwise be routed into the interface it is
//! carrying, a loop. The second lets every specific route — the LAN, another
//! carrying, a loop. The specific-route rule lets the LAN, another
//! VPN, a tunnel to somewhere else — keep working, because only the default
//! route is suppressed. The priorities sit after Tailscale's 5270.
//! route is suppressed. Mark/UID bypass precedes Tailscale; application rules follow it.
//!
//! The first rule exempts a *user id*, so it is only right when the agent has
//! The UID rule exempts a *user id*, so it is only right when the agent has
//! one of its own, as the packaged service does. An agent started from a login
//! session would take that user's own applications out of the tunnel with it,
//! and the exit node would silently do nothing for them. That is refused, not
@@ -48,9 +49,10 @@ use crate::state::Ipv4Range;
use super::RuleOutcome;
use super::exit::{ExitHostPlan, ExitHostReport, ExitHostRules, exit_tag, exit_tag_prefix};
use super::linux::{
LinuxHostRules, Ran, Run, comment_supported, failure, locked, run_iptables, same_rule,
split_rule_line,
LinuxHostRules, Ran, Run, comment_supported, failure, locked, run_firewall, run_iptables,
same_rule, split_rule_line,
};
use super::linux_exit_mark;
/// The chains the offering rules live in: `(table, chain)`.
const CHAINS: [(&str, &str); 2] = [("nat", "POSTROUTING"), ("filter", "FORWARD")];
@@ -377,6 +379,32 @@ fn read_sysctl(path: &str) -> Option<String> {
std::fs::read_to_string(path).ok()
}
fn check_kernel_rpf(overlay: &str) -> Result<(), String> {
let directory = "/proc/sys/net/ipv4/conf";
let value = |name: &str, setting: &str| -> Result<u32, String> {
read_sysctl(&format!("{directory}/{name}/{setting}"))
.and_then(|s| s.trim().parse().ok())
.ok_or_else(|| format!("cannot check {name}/{setting} before installing exit routes"))
};
let mut interfaces = Vec::new();
for entry in std::fs::read_dir(directory).map_err(|err| err.to_string())? {
let entry = entry.map_err(|err| err.to_string())?;
let name = entry.file_name().to_string_lossy().into_owned();
if name != "all" && name != "default" && name != "lo" && name != overlay {
interfaces.push((
name.clone(),
value(&name, "rp_filter")?,
value(&name, "src_valid_mark")?,
));
}
}
linux_exit_mark::check_kernel_rpf(
value("all", "rp_filter")?,
value("all", "src_valid_mark")?,
&interfaces,
)
}
/// The effective user id from the text of `/proc/self/status`.
fn parse_euid(status: &str) -> Option<u32> {
status
@@ -417,7 +445,7 @@ impl LinuxHostRules {
let xdg = std::env::var("XDG_RUNTIME_DIR").ok();
if let Some(reason) = client_refusal(xdg.as_deref()) {
// Whatever an earlier run left must not stay in force.
self.clear_client().await;
let _ = self.clear_client(interface).await;
return RuleOutcome::Failed(reason);
}
let Some(euid) = std::fs::read_to_string("/proc/self/status")
@@ -426,24 +454,147 @@ impl LinuxHostRules {
else {
return RuleOutcome::Failed("cannot tell which user this agent runs as".to_string());
};
let installed = self
.underlay_sockets
.lock()
.unwrap_or_else(|p| p.into_inner())
.as_ref()
.is_some_and(|(name, _, routed)| name == interface && *routed);
if installed {
// A failed socket refresh must never remove an active exit route.
return self
.refresh_sockets(interface)
.await
.unwrap_or(RuleOutcome::Applied);
}
if let Err(reason) = self.clear_client(interface).await {
return RuleOutcome::Failed(reason);
}
let name = interface.to_string();
let prepared = Self::blocking(move || {
check_kernel_rpf(&name)?;
linux_exit_mark::install(
&run_firewall,
&name,
euid,
std::path::Path::new("/proc/sys/net/ipv6").is_dir(),
)
})
.await;
match prepared {
Ok(Ok(sockets)) => {
*self
.underlay_sockets
.lock()
.unwrap_or_else(|p| p.into_inner()) = Some((interface.into(), sockets, false));
}
Ok(Err(reason)) | Err(reason) => return RuleOutcome::Failed(reason),
}
let route = self.route.clone();
let interface = interface.to_string();
match Self::blocking(move || route.set_exit_client(&interface, euid)).await {
let name = interface.to_string();
let outcome = match Self::blocking(move || route.set_exit_client(&name, euid)).await {
Ok(Ok(())) => RuleOutcome::Applied,
Ok(Err(err)) => RuleOutcome::Failed(err.to_string()),
Err(reason) => RuleOutcome::Failed(reason),
};
if let RuleOutcome::Failed(reason) = outcome {
let reason = match self.clear_client(interface).await {
Ok(()) => reason,
Err(cleanup) => format!("{reason}; cannot roll back exit rules: {cleanup}"),
};
RuleOutcome::Failed(reason)
} else {
if let Some((_, _, routed)) = self
.underlay_sockets
.lock()
.unwrap_or_else(|p| p.into_inner())
.as_mut()
{
*routed = true;
}
outcome
}
}
async fn clear_client(&self) {
async fn clear_client(&self, interface: &str) -> Result<(), String> {
let route = self.route.clone();
if let Ok(Err(err)) = Self::blocking(move || route.clear_exit_client()).await {
tracing::warn!(%err, "cannot remove the exit node routing rules");
Self::blocking(move || route.clear_exit_client())
.await?
.map_err(|err| err.to_string())?;
// Route removal must precede removing reply protection.
let name = interface.to_string();
Self::blocking(move || {
linux_exit_mark::clear(
&run_firewall,
&name,
std::path::Path::new("/proc/sys/net/ipv6").is_dir(),
)
})
.await??;
*self
.underlay_sockets
.lock()
.unwrap_or_else(|p| p.into_inner()) = None;
Ok(())
}
async fn refresh_sockets(&self, interface: &str) -> Option<RuleOutcome> {
let previous = self
.underlay_sockets
.lock()
.unwrap_or_else(|p| p.into_inner())
.clone();
if previous
.as_ref()
.is_none_or(|(name, _, routed)| name != interface || !routed)
{
return None;
}
let name = interface.to_string();
let work_name = name.clone();
let result = Self::blocking(move || {
let uid = std::fs::read_to_string("/proc/self/status")
.ok()
.and_then(|s| parse_euid(&s))
.ok_or("cannot read agent UID")?;
check_kernel_rpf(&work_name)?;
let v6 = std::path::Path::new("/proc/sys/net/ipv6").is_dir();
let sockets = linux_exit_mark::local_sockets(uid, v6)?;
if previous
.as_ref()
.is_some_and(|(name, old, _)| name == &work_name && old == &sockets)
{
return Ok(None);
}
linux_exit_mark::refresh(&run_firewall, &work_name, v6, &sockets)?;
Ok(Some(sockets))
})
.await;
match result {
Ok(Ok(None)) => None,
Ok(Ok(Some(sockets))) => {
*self
.underlay_sockets
.lock()
.unwrap_or_else(|p| p.into_inner()) = Some((name, sockets, true));
Some(RuleOutcome::Applied)
}
Ok(Err(reason)) | Err(reason) => Some(RuleOutcome::Failed(reason)),
}
}
}
impl ExitHostRules for LinuxHostRules {
fn refresh<'a>(&'a self, plan: &'a ExitHostPlan) -> BoxFuture<'a, Option<RuleOutcome>> {
Box::pin(async move {
if plan.client {
self.refresh_sockets(&plan.interface).await
} else {
None
}
})
}
fn apply<'a>(&'a self, plan: &'a ExitHostPlan) -> BoxFuture<'a, ExitHostReport> {
Box::pin(async move {
let interface = plan.interface.clone();
@@ -467,7 +618,9 @@ impl ExitHostRules for LinuxHostRules {
let client = if plan.client {
Some(self.apply_client(&interface).await)
} else {
self.clear_client().await;
if let Err(reason) = self.clear_client(&interface).await {
tracing::warn!(%reason, "cannot remove exit-client rules");
}
None
};
ExitHostReport {
@@ -482,7 +635,9 @@ impl ExitHostRules for LinuxHostRules {
Box::pin(async move {
let name = interface.to_string();
let _ = Self::blocking(move || clear_offer(&run_iptables, &name)).await;
self.clear_client().await;
if let Err(reason) = self.clear_client(interface).await {
tracing::warn!(%reason, "cannot remove exit-client rules");
}
})
}
}
@@ -0,0 +1,439 @@
//! Privileged routing regression, isolated from the host in named namespaces.
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use std::io::{BufRead, BufReader, Read, Write};
use std::net::{SocketAddr, TcpListener, TcpStream, UdpSocket};
use std::os::unix::process::CommandExt;
use std::process::{Child, Command};
use std::time::Duration;
use super::linux::run_firewall;
use super::linux_exit_mark::{clear, install, local_sockets, refresh};
use crate::overlay::NetlinkProvisioner;
const TEST: &str = "overlay::hostrules::linux_exit_kernel::strict_firewall_preserves_underlay_and_application_exit_routing";
const ROLE: &str = "TSUNAGI_EXIT_KERNEL_TEST_ROLE";
fn ip(arguments: &[&str]) -> String {
let result = Command::new("ip").args(arguments).output().unwrap();
assert!(
result.status.success(),
"ip {arguments:?}: {}",
String::from_utf8_lossy(&result.stderr)
);
String::from_utf8_lossy(&result.stdout).into_owned()
}
struct Namespace(String);
impl Drop for Namespace {
fn drop(&mut self) {
let _ = Command::new("ip").args(["netns", "del", &self.0]).status();
}
}
struct Echo(Child);
impl Drop for Echo {
fn drop(&mut self) {
let _ = self.0.kill();
let _ = self.0.wait();
}
}
fn probes() {
let should_pass = std::env::var("TSUNAGI_EXIT_KERNEL_EXPECT").unwrap() == "pass";
for address in ["198.51.100.2:42000", "[2001:db8:2::2]:42000"] {
let address: SocketAddr = address.parse().unwrap();
let socket = UdpSocket::bind(if address.is_ipv4() {
"0.0.0.0:0"
} else {
"[::]:0"
})
.unwrap();
socket
.set_read_timeout(Some(Duration::from_secs(1)))
.unwrap();
socket.send_to(b"rpf", address).unwrap();
let mut packet = [0u8; 3];
assert_eq!(
socket.recv(&mut packet).is_ok(),
should_pass,
"UDP {address}"
);
if should_pass {
assert_eq!(&packet, b"rpf");
}
let connected = TcpStream::connect_timeout(&address, Duration::from_secs(1));
assert_eq!(connected.is_ok(), should_pass, "TCP {address}");
if let Ok(mut stream) = connected {
stream
.set_read_timeout(Some(Duration::from_secs(1)))
.unwrap();
stream.write_all(b"rpf").unwrap();
stream.read_exact(&mut packet).unwrap();
assert_eq!(&packet, b"rpf");
}
}
}
fn run_probes(expect: &str) {
let result = Command::new(std::env::current_exe().unwrap())
.args(["--ignored", "--exact", TEST, "--nocapture"])
.env(ROLE, "probe")
.env("TSUNAGI_EXIT_KERNEL_EXPECT", expect)
.uid(990)
.gid(990)
.output()
.unwrap();
assert!(
result.status.success(),
"{}\n{}",
String::from_utf8_lossy(&result.stdout),
String::from_utf8_lossy(&result.stderr)
);
}
fn passive_listener() {
let listeners: Vec<_> = ["192.0.2.1:42010", "[2001:db8:1::1]:42010"]
.into_iter()
.map(|address| TcpListener::bind(address).unwrap())
.collect();
println!("PASSIVE_READY");
let tasks: Vec<_> = listeners
.into_iter()
.map(|listener| {
std::thread::spawn(move || {
let (mut socket, _) = listener.accept().unwrap();
socket
.set_read_timeout(Some(Duration::from_secs(2)))
.unwrap();
let mut bytes = [0; 3];
socket.read_exact(&mut bytes).unwrap();
socket.write_all(&bytes).unwrap();
})
})
.collect();
for task in tasks {
task.join().unwrap();
}
}
fn udp_listener() {
let sockets: Vec<_> = ["192.0.2.1:42011", "[2001:db8:1::1]:42011"]
.into_iter()
.map(|a| UdpSocket::bind(a).unwrap())
.collect();
println!("PASSIVE_READY");
let tasks: Vec<_> = sockets
.into_iter()
.map(|socket| {
std::thread::spawn(move || {
socket
.set_read_timeout(Some(Duration::from_secs(5)))
.unwrap();
let mut bytes = [0; 3];
let (len, peer) = socket.recv_from(&mut bytes).unwrap();
socket.send_to(&bytes[..len], peer).unwrap();
})
})
.collect();
for task in tasks {
task.join().unwrap();
}
}
fn spawn_listener(role: &str) -> Echo {
let process = Command::new(std::env::current_exe().unwrap())
.args(["--ignored", "--exact", TEST, "--nocapture"])
.env(ROLE, role)
.uid(990)
.gid(990)
.stdout(std::process::Stdio::piped())
.spawn()
.unwrap();
let mut child = Echo(process);
let mut output = BufReader::new(child.0.stdout.take().unwrap());
loop {
let mut line = String::new();
assert_ne!(
output.read_line(&mut line).unwrap(),
0,
"listener exited before readiness"
);
if line.contains("PASSIVE_READY") {
// Keep the reader open for libtest's final status after the socket
// worker finishes; closing it here makes success a broken pipe.
child.0.stdout = Some(output.into_inner());
return child;
}
}
}
fn client() {
for setting in ["rp_filter", "src_valid_mark"] {
std::fs::write(format!("/proc/sys/net/ipv4/conf/all/{setting}"), b"1").unwrap();
}
std::fs::write("/proc/sys/net/ipv4/conf/rpf-c/rp_filter", b"1").unwrap();
for family in [false, true] {
let args: Vec<_> = [
"-t",
"mangle",
"-A",
"PREROUTING",
"-m",
"rpfilter",
"--invert",
"--validmark",
"-j",
"DROP",
]
.into_iter()
.map(str::to_string)
.collect();
let result = run_firewall(family, &args);
assert!(
matches!(result, super::linux::Ran::Exited { success: true, .. }),
"{result:?}"
);
}
let provisioner = NetlinkProvisioner::new().unwrap();
let route = provisioner.host_rules().route;
// Model Tailscale's earlier lookup, without running another VPN daemon.
ip(&["-4", "rule", "add", "priority", "5270", "table", "52"]);
ip(&[
"-4", "route", "add", "default", "dev", "tsun0", "table", "52",
]);
ip(&["-6", "rule", "add", "priority", "5270", "table", "52"]);
ip(&[
"-6",
"route",
"add",
"unreachable",
"default",
"table",
"52",
]);
// A foreign VPN must leave neighbour discovery and connected LAN paths
// usable; otherwise the fixture itself cannot resolve its uplink neighbour.
for prefix in ["fe80::/64", "2001:db8:1::/64"] {
ip(&["-6", "route", "add", "throw", prefix, "table", "52"]);
}
let mut passive = spawn_listener("listener");
route.set_exit_client("tsun0", 990).unwrap();
// uid bypass alone cannot protect reverse lookups, reproducing the bug.
run_probes("blocked");
install(&run_firewall, "tsun0", 990, true).unwrap();
run_probes("pass");
// Pair transports may bind fresh UDP ports after exit activation.
let mut dynamic_udp = spawn_listener("udp-listener");
refresh(
&run_firewall,
"tsun0",
true,
&local_sockets(990, true).unwrap(),
)
.unwrap();
let server = std::env::var("TSUNAGI_EXIT_KERNEL_SERVER").unwrap();
let incoming = Command::new("ip").args(["netns", "exec", &server, "python3", "-c", r#"import socket
for family,source,target in [(socket.AF_INET,'198.51.100.2','192.0.2.1'),(socket.AF_INET6,'2001:db8:2::2','2001:db8:1::1')]:
for kind,port in [(socket.SOCK_STREAM,42010),(socket.SOCK_DGRAM,42011)]:
s=socket.socket(family,kind); s.settimeout(2); s.bind((source,0)); s.connect((target,port)); s.sendall(b'rpf'); assert s.recv(3)==b'rpf'; s.close()
"#]).output().unwrap();
assert!(
incoming.status.success(),
"first inbound TCP/UDP must be protected: {}",
String::from_utf8_lossy(&incoming.stderr)
);
assert!(passive.0.wait().unwrap().success());
assert!(dynamic_udp.0.wait().unwrap().success());
// Repeated apply remains usable, and does not redirect application sockets.
install(&run_firewall, "tsun0", 990, true).unwrap();
run_probes("pass");
assert!(ip(&["-4", "route", "get", "198.51.100.2", "uid", "991"]).contains("dev tsun0"));
let blocked = Command::new("ip")
.args(["-6", "route", "get", "2001:db8:2::2", "uid", "991"])
.output()
.unwrap();
assert!(
!blocked.status.success(),
"application IPv6 must stay blocked"
);
route.clear_exit_client().unwrap();
clear(&run_firewall, "tsun0", true).unwrap();
for family in ["-4", "-6"] {
assert!(
ip(&[family, "rule", "show"]).contains("5270:"),
"foreign routing rules survive cleanup"
);
ip(&[family, "rule", "del", "priority", "5270"]);
}
assert!(ip(&["-4", "route", "get", "198.51.100.2", "uid", "991"]).contains("dev rpf-c"));
assert!(ip(&["-6", "route", "get", "2001:db8:2::2", "uid", "991"]).contains("dev rpf-c"));
for family in [false, true] {
let result = run_firewall(family, &["-t".into(), "mangle".into(), "-S".into()]);
if let super::linux::Ran::Exited { stdout, .. } = result {
assert!(!stdout.contains("tsunagi-exit-underlay:"));
assert!(!stdout.contains("tsunagi-exit-sockets:"));
assert!(
stdout.contains("rpfilter"),
"strict host filtering is retained"
);
}
}
}
#[test]
#[ignore = "needs root, iproute2, iptables/ip6tables and Python; changes only temporary namespaces"]
fn strict_firewall_preserves_underlay_and_application_exit_routing() {
match std::env::var(ROLE).as_deref() {
Ok("probe") => {
probes();
return;
}
Ok("client") => {
client();
return;
}
Ok("listener") => {
passive_listener();
return;
}
Ok("udp-listener") => {
udp_listener();
return;
}
_ => {}
}
let suffix = std::process::id();
let client_name = format!("tsng-rpf-c-{suffix}");
let server_name = format!("tsng-rpf-s-{suffix}");
ip(&["netns", "add", &client_name]);
let _client = Namespace(client_name.clone());
ip(&["netns", "add", &server_name]);
let _server = Namespace(server_name.clone());
ip(&[
"link",
"add",
"rpf-c",
"netns",
&client_name,
"type",
"veth",
"peer",
"name",
"rpf-s",
"netns",
&server_name,
]);
for (namespace, interface, v4, v6) in [
(&client_name, "rpf-c", "192.0.2.1/24", "2001:db8:1::1/64"),
(&server_name, "rpf-s", "192.0.2.2/24", "2001:db8:1::2/64"),
] {
ip(&["-n", namespace, "link", "set", "lo", "up"]);
ip(&["-n", namespace, "addr", "add", v4, "dev", interface]);
ip(&[
"-n", namespace, "-6", "addr", "add", v6, "dev", interface, "nodad",
]);
ip(&["-n", namespace, "link", "set", interface, "up"]);
}
ip(&[
"-n",
&server_name,
"addr",
"add",
"198.51.100.2/32",
"dev",
"lo",
]);
ip(&[
"-n",
&server_name,
"-6",
"addr",
"add",
"2001:db8:2::2/128",
"dev",
"lo",
"nodad",
]);
ip(&[
"-n",
&client_name,
"route",
"add",
"default",
"via",
"192.0.2.2",
]);
ip(&[
"-n",
&client_name,
"-6",
"route",
"add",
"default",
"via",
"2001:db8:1::2",
]);
ip(&["-n", &client_name, "link", "add", "tsun0", "type", "dummy"]);
ip(&[
"-n",
&client_name,
"addr",
"add",
"10.13.37.1/24",
"dev",
"tsun0",
]);
ip(&["-n", &client_name, "link", "set", "tsun0", "up"]);
let echo_code = r#"import socket,selectors,sys
selector=selectors.DefaultSelector()
for family in [socket.AF_INET,socket.AF_INET6]:
for kind in [socket.SOCK_DGRAM,socket.SOCK_STREAM]:
sock=socket.socket(family,kind)
if family==socket.AF_INET6: sock.setsockopt(socket.IPPROTO_IPV6,socket.IPV6_V6ONLY,1)
sock.bind(('198.51.100.2' if family==socket.AF_INET else '2001:db8:2::2',42000))
if kind==socket.SOCK_STREAM: sock.listen()
selector.register(sock,selectors.EVENT_READ,kind)
print('ready',flush=True)
while True:
for key,_ in selector.select():
if key.data==socket.SOCK_DGRAM:
data,address=key.fileobj.recvfrom(128); key.fileobj.sendto(data,address)
else:
conn,_=key.fileobj.accept(); conn.settimeout(2)
try: conn.sendall(conn.recv(128))
finally: conn.close()
"#;
let child = Command::new("ip")
.args([
"netns",
"exec",
&server_name,
"python3",
"-u",
"-c",
echo_code,
])
.stdout(std::process::Stdio::piped())
.spawn()
.unwrap();
let mut echo = Echo(child);
let mut ready = String::new();
BufReader::new(echo.0.stdout.take().unwrap())
.read_line(&mut ready)
.unwrap();
assert_eq!(ready.trim(), "ready");
let result = Command::new("ip")
.args(["netns", "exec", &client_name])
.arg(std::env::current_exe().unwrap())
.args(["--ignored", "--exact", TEST, "--nocapture"])
.env(ROLE, "client")
.env("TSUNAGI_EXIT_KERNEL_SERVER", &server_name)
.output()
.unwrap();
assert!(
result.status.success(),
"{}\n{}",
String::from_utf8_lossy(&result.stdout),
String::from_utf8_lossy(&result.stderr)
);
}
@@ -0,0 +1,535 @@
//! Own-bit connection marks for exit-client underlay traffic.
//! Reply marks are restored before host reverse-path filtering, never accepted
//! around it. No peer address becomes a host route or firewall argument.
use super::linux::{Ran, failure, locked, split_rule_line};
use std::net::{IpAddr, Ipv4Addr, Ipv6Addr};
/// Outside Tailscale's 0xff0000 mask and ordinary low-bit WireGuard marks.
pub(crate) const UNDERLAY_MARK: u32 = 0x4000_0000;
pub(crate) const UNDERLAY_MASK: u32 = UNDERLAY_MARK;
const MARK: &str = "0x40000000/0x40000000";
const MASK: &str = "0x40000000";
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord)]
pub(super) struct Listener {
address: IpAddr,
port: u16,
tcp: bool,
}
fn parse_listeners(listing: &str, uid: u32, v6: bool, tcp: bool) -> Result<Vec<Listener>, String> {
let mut result = Vec::new();
for line in listing.lines().skip(1) {
let fields: Vec<_> = line.split_whitespace().collect();
if (tcp && fields.get(3) != Some(&"0A"))
|| fields.get(7).and_then(|s| s.parse::<u32>().ok()) != Some(uid)
{
continue;
}
let (address, port) = fields
.get(1)
.and_then(|s| s.split_once(':'))
.ok_or("malformed local socket")?;
let port = u16::from_str_radix(port, 16).map_err(|e| e.to_string())?;
let address = if v6 {
let mut bytes = [0; 16];
for (word, chunk) in bytes.as_chunks_mut::<4>().0.iter_mut().enumerate() {
let hex = address
.get(word * 8..word * 8 + 8)
.ok_or("malformed local IPv6 listener")?;
chunk.copy_from_slice(
&u32::from_str_radix(hex, 16)
.map_err(|e| e.to_string())?
.to_ne_bytes(),
);
}
IpAddr::V6(Ipv6Addr::from(bytes))
} else {
IpAddr::V4(Ipv4Addr::from(
u32::from_str_radix(address, 16)
.map_err(|e| e.to_string())?
.to_ne_bytes(),
))
};
if result.len() == 128 {
return Err("too many local sockets for exit protection".into());
}
if port != 0 {
result.push(Listener { address, port, tcp });
}
}
Ok(result)
}
pub(super) fn local_sockets(uid: u32, v6: bool) -> Result<Vec<Listener>, String> {
let mut listeners = Vec::new();
for (protocol, tcp) in [("tcp", true), ("udp", false)] {
for family in [false, true].into_iter().filter(|f| !f || v6) {
let path = format!("/proc/net/{protocol}{}", if family { "6" } else { "" });
listeners.extend(parse_listeners(
&std::fs::read_to_string(path).map_err(|e| e.to_string())?,
uid,
family,
tcp,
)?);
}
}
listeners.sort();
listeners.dedup();
Ok(listeners)
}
fn listener_rules(interface: &str, listener: &Listener) -> Vec<Vec<String>> {
let mut base = words(&[
"!",
"-i",
interface,
"-p",
if listener.tcp { "tcp" } else { "udp" },
"--dport",
&listener.port.to_string(),
]);
if !listener.address.is_unspecified() {
base.extend(words(&["-d", &listener.address.to_string()]));
}
// Match an actual local socket, not just a destination port. Listening
// sockets need protection on the first packet, before any outbound connmark.
base.extend(words(&[
"-m",
"socket",
"--nowildcard",
"-m",
"comment",
"--comment",
&listener_tag(interface),
]));
let save = base
.iter()
.cloned()
.chain(words(&[
"-m",
"mark",
"--mark",
MARK,
"-j",
"CONNMARK",
"--save-mark",
"--nfmask",
MASK,
"--ctmask",
MASK,
]))
.collect();
let mark = base
.into_iter()
.chain(words(&["-j", "MARK", "--set-xmark", MARK]))
.collect();
vec![save, mark]
}
pub(super) type RunFamily<'a> = &'a dyn Fn(bool, &[String]) -> Ran;
fn tag(interface: &str) -> String {
format!("tsunagi-exit-underlay:{interface}")
}
fn listener_tag(interface: &str) -> String {
format!("tsunagi-exit-sockets:{interface}")
}
fn words(values: &[&str]) -> Vec<String> {
values.iter().map(|s| (*s).into()).collect()
}
fn rules(interface: &str, uid: u32) -> Vec<(&'static str, Vec<String>)> {
let tag = tag(interface);
let uid = uid.to_string();
// Insert at position one in this order: final OUTPUT order is MARK, SAVE.
vec![
(
"PREROUTING",
words(&[
"!",
"-i",
interface,
"-m",
"connmark",
"--mark",
MARK,
"-m",
"comment",
"--comment",
&tag,
"-j",
"CONNMARK",
"--restore-mark",
"--nfmask",
MASK,
"--ctmask",
MASK,
]),
),
(
"OUTPUT",
words(&[
"-m",
"owner",
"--uid-owner",
&uid,
"-m",
"mark",
"--mark",
MARK,
"-m",
"comment",
"--comment",
&tag,
"-j",
"CONNMARK",
"--save-mark",
"--nfmask",
MASK,
"--ctmask",
MASK,
]),
),
(
"OUTPUT",
words(&[
"-m",
"owner",
"--uid-owner",
&uid,
"-m",
"comment",
"--comment",
&tag,
"-j",
"MARK",
"--set-xmark",
MARK,
]),
),
]
}
fn execute(run: RunFamily<'_>, v6: bool, args: Vec<String>) -> Result<String, String> {
match run(v6, &locked(args)) {
Ran::Exited {
success: true,
stdout,
..
} => Ok(stdout),
Ran::Exited { stderr, .. } => Err(failure(&stderr)),
Ran::Missing => Err(format!(
"{} is required for exit-node connection marks",
if v6 { "ip6tables" } else { "iptables" }
)),
}
}
/// Removes only exact interface tags, retaining other agents and host rules.
pub(super) fn clear(run: RunFamily<'_>, interface: &str, v6: bool) -> Result<(), String> {
clear_tags(run, interface, v6, false)
}
fn clear_tags(
run: RunFamily<'_>,
interface: &str,
v6: bool,
sockets_only: bool,
) -> Result<(), String> {
let wanted = tag(interface);
let sockets = listener_tag(interface);
for family in [false, true].into_iter().filter(|family| !family || v6) {
let chains: &[&str] = if sockets_only {
&["PREROUTING"]
} else {
&["OUTPUT", "PREROUTING"]
};
for chain in chains {
let listing = execute(run, family, words(&["-t", "mangle", "-S", chain]))?;
for line in listing.lines() {
let tokens = split_rule_line(line);
let [append, listed_chain, rest @ ..] = tokens.as_slice() else {
continue;
};
if append == "-A"
&& listed_chain == *chain
&& rest.windows(2).any(|pair| {
pair[0] == "--comment"
&& (pair[1] == sockets || (!sockets_only && pair[1] == wanted))
})
{
let args = words(&["-t", "mangle", "-D", chain])
.into_iter()
.chain(rest.iter().cloned())
.collect();
execute(run, family, args)?;
}
}
}
}
Ok(())
}
/// Establishes protection before routing may redirect application traffic.
pub(super) fn install(
run: RunFamily<'_>,
interface: &str,
uid: u32,
v6: bool,
) -> Result<Vec<Listener>, String> {
let listeners = local_sockets(uid, v6)?;
install_with_listeners(run, interface, uid, v6, &listeners)?;
Ok(listeners)
}
fn install_with_listeners(
run: RunFamily<'_>,
interface: &str,
uid: u32,
v6: bool,
listeners: &[Listener],
) -> Result<(), String> {
clear(run, interface, v6)?;
let result = (|| {
for family in [false, true].into_iter().filter(|family| !family || v6) {
for (chain, spec) in rules(interface, uid) {
let args = words(&["-t", "mangle", "-I", chain, "1"])
.into_iter()
.chain(spec)
.collect();
execute(run, family, args)?;
}
for listener in listeners.iter().filter(|s| s.address.is_ipv6() == family) {
for spec in listener_rules(interface, listener) {
execute(
run,
family,
words(&["-t", "mangle", "-I", "PREROUTING", "1"])
.into_iter()
.chain(spec)
.collect(),
)?;
}
}
}
Ok(())
})();
if result.is_err()
&& let Err(cleanup) = clear(run, interface, v6)
{
return Err(format!(
"{}; cannot roll back connection marks: {cleanup}",
result.err().unwrap_or_default()
));
}
result
}
/// Updates passive protection without interrupting established connmark rules.
pub(super) fn refresh(
run: RunFamily<'_>,
interface: &str,
v6: bool,
listeners: &[Listener],
) -> Result<(), String> {
clear_tags(run, interface, v6, true)?;
for family in [false, true].into_iter().filter(|f| !f || v6) {
for listener in listeners.iter().filter(|s| s.address.is_ipv6() == family) {
for spec in listener_rules(interface, listener) {
execute(
run,
family,
words(&["-t", "mangle", "-I", "PREROUTING", "1"])
.into_iter()
.chain(spec)
.collect(),
)?;
}
}
}
Ok(())
}
/// Kernel strict RPF must include the restored mark, independently of iptables.
pub(super) fn check_kernel_rpf(
all_rpf: u32,
all_valid: u32,
interfaces: &[(String, u32, u32)],
) -> Result<(), String> {
for (interface, rpf, valid) in interfaces {
if all_rpf.max(*rpf) == 1 && all_valid.max(*valid) == 0 {
return Err(format!(
"strict rp_filter on `{interface}` ignores connection marks; enable net.ipv4.conf.all.src_valid_mark=1 before using an exit node (rp_filter stays enabled)"
));
}
}
Ok(())
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
use std::cell::RefCell;
#[test]
fn both_families_restore_before_firewall_and_preserve_foreign_mark_bits() {
let calls = RefCell::new(Vec::new());
let run = |v6, args: &[String]| {
calls.borrow_mut().push((v6, args.to_vec()));
Ran::Exited {
success: true,
stdout: String::new(),
stderr: String::new(),
}
};
install_with_listeners(&run, "tsun0", 990, true, &[]).unwrap();
let calls = calls.borrow();
for family in [false, true] {
let inserts: Vec<_> = calls
.iter()
.filter(|(v6, a)| *v6 == family && a.iter().any(|s| s == "-I"))
.map(|(_, a)| a.join(" "))
.collect();
assert_eq!(inserts.len(), 3);
assert!(inserts[0].contains("-I PREROUTING 1 ! -i tsun0 -m connmark"));
assert!(inserts[0].contains("--restore-mark --nfmask 0x40000000 --ctmask 0x40000000"));
assert!(inserts[1].contains("--uid-owner 990"));
assert!(inserts[1].contains("--save-mark"));
assert!(inserts[2].contains("--set-xmark 0x40000000/0x40000000"));
assert!(
!inserts
.iter()
.any(|s| s.contains("-j ACCEPT") || s.contains("-j RETURN"))
);
}
assert_eq!(UNDERLAY_MASK & 0x00ff_0000, 0);
}
#[test]
fn cleanup_retains_other_interfaces_and_rolls_back_partial_ipv6_installation() {
let calls = RefCell::new(Vec::new());
let run = |v6, args: &[String]| {
calls.borrow_mut().push((v6, args.to_vec()));
let listing = "-A OUTPUT -m comment --comment tsunagi-exit-underlay:tsun0 -j MARK --set-xmark 0x40000000/0x40000000\n-A OUTPUT -m comment --comment tsunagi-exit-underlay:tsun1 -j MARK --set-xmark 0x40000000/0x40000000\n-A OUTPUT -m comment --comment unrelated -j RETURN\n";
Ran::Exited {
success: !(v6 && args.iter().any(|s| s == "-I")),
stdout: if args.iter().any(|s| s == "-S") {
listing.into()
} else {
String::new()
},
stderr: "injected IPv6 failure".into(),
}
};
assert!(install_with_listeners(&run, "tsun0", 990, true, &[]).is_err());
let calls = calls.borrow();
let deletions: Vec<_> = calls
.iter()
.filter(|(_, a)| a.iter().any(|s| s == "-D"))
.collect();
assert!(
deletions.len() >= 4,
"both initial and rollback cleanup run"
);
assert!(
deletions
.iter()
.all(|(_, a)| a.iter().any(|s| s == "tsunagi-exit-underlay:tsun0"))
);
}
#[test]
fn kernel_strict_validation_requires_mark_awareness_without_weakening_rpf() {
let interfaces = vec![("eth0".into(), 1, 0)];
assert!(check_kernel_rpf(0, 0, &interfaces).is_err());
assert!(check_kernel_rpf(0, 1, &interfaces).is_ok());
assert!(check_kernel_rpf(2, 0, &interfaces).is_ok());
assert!(check_kernel_rpf(0, 0, &[("eth0".into(), 1, 1)]).is_ok());
}
#[test]
fn passive_tcp_protection_uses_only_locally_owned_listeners() {
let listing = "header\n0: 00000000:01BB 00000000:0000 0A 0:0 0:0 0 990 0 1\n1: 0100007F:1111 00000000:0000 0A 0:0 0:0 0 1000 0 2\n2: 0100007F:2222 00000000:0000 01 0:0 0:0 0 990 0 3\n";
let listeners = parse_listeners(listing, 990, false, true).unwrap();
assert_eq!(
listeners,
vec![Listener {
address: IpAddr::V4(Ipv4Addr::UNSPECIFIED),
port: 443,
tcp: true
}]
);
let rules = listener_rules("tsun0", &listeners[0]);
assert!(
rules[0]
.join(" ")
.contains("-p tcp --dport 443 -m socket --nowildcard")
);
assert!(rules[0].join(" ").contains("--save-mark"));
assert!(rules[1].join(" ").contains("--set-xmark"));
}
#[test]
fn listener_parser_checks_family_encoding_and_bounds() {
let loopback = Ipv6Addr::LOCALHOST.octets();
let hex: String = loopback
.as_chunks::<4>()
.0
.iter()
.map(|bytes| format!("{:08X}", u32::from_ne_bytes(*bytes)))
.collect();
let listing = format!("header\n0: {hex}:01BB 0:0 0A 0:0 0:0 0 990 0 1\n");
assert_eq!(
parse_listeners(&listing, 990, true, true).unwrap(),
vec![Listener {
address: IpAddr::V6(Ipv6Addr::LOCALHOST),
port: 443,
tcp: true
}]
);
assert!(
parse_listeners(
"header\n0: BAD:01BB 0:0 0A 0:0 0:0 0 990 0 1",
990,
true,
true
)
.is_err()
);
let huge = format!(
"header\n{}",
"0: 00000000:01BB 0:0 0A 0:0 0:0 0 990 0 1\n".repeat(129)
);
assert!(parse_listeners(&huge, 990, false, true).is_err());
}
#[test]
fn udp_refresh_preserves_general_connmark_protection_and_other_interfaces() {
let listing = "header\n0: 00000000:ABA0 0:0 07 0:0 0:0 0 990 0 1\n1: 00000000:0000 0:0 07 0:0 0:0 0 990 0 2\n";
let sockets = parse_listeners(listing, 990, false, false).unwrap();
assert_eq!(sockets.len(), 1);
assert!(!sockets[0].tcp);
let calls = RefCell::new(Vec::new());
let run = |v6, args: &[String]| {
calls.borrow_mut().push((v6, args.join(" ")));
Ran::Exited { success: true, stdout: "-A PREROUTING -m comment --comment tsunagi-exit-underlay:tsun0 -j CONNMARK --restore-mark\n-A PREROUTING -m comment --comment tsunagi-exit-sockets:tsun0 -j MARK\n-A PREROUTING -m comment --comment tsunagi-exit-sockets:tsun1 -j MARK\n".into(), stderr: String::new() }
};
refresh(&run, "tsun0", false, &sockets).unwrap();
let calls = calls.borrow();
assert!(
calls
.iter()
.any(|(_, a)| a.contains("-I PREROUTING 1") && a.contains("-p udp"))
);
assert!(
calls
.iter()
.filter(|(_, a)| a.contains("-D"))
.all(|(_, a)| a.contains("tsunagi-exit-sockets:tsun0"))
);
}
}
@@ -55,6 +55,10 @@ pub use exit::{
mod linux;
#[cfg(all(feature = "tun-device", target_os = "linux"))]
mod linux_exit;
#[cfg(all(test, feature = "tun-device", target_os = "linux"))]
mod linux_exit_kernel;
#[cfg(all(feature = "tun-device", target_os = "linux"))]
pub(in crate::overlay) mod linux_exit_mark;
#[cfg(all(feature = "tun-device", target_os = "linux"))]
pub use linux::LinuxHostRules;
+97 -1
View File
@@ -145,18 +145,19 @@ impl Tally {
}
/// The overlay interface.
#[derive(Debug)]
pub struct Interface {
device: Arc<dyn TunDevice>,
factory: Arc<dyn TunFactory>,
name: String,
mtu: u32,
routes: Arc<RoutingTable>,
carrier: Arc<dyn PacketCarrier>,
tally: Arc<Tally>,
/// What was last applied to the host, so an unchanged table is not
/// re-applied on every pass.
applied: std::sync::Mutex<Vec<Cidr>>,
task: std::sync::Mutex<Option<JoinHandle<()>>>,
vip_task: std::sync::Mutex<Option<JoinHandle<()>>>,
sync_lock: Arc<tokio::sync::Mutex<()>>,
/// The host rules for this interface, when the factory manages the host.
host_rules: Option<Arc<dyn BroadcastHostRules>>,
@@ -200,6 +201,14 @@ impl ExitState {
}
}
impl std::fmt::Debug for Interface {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("Interface")
.field("name", &self.name)
.finish_non_exhaustive()
}
}
impl Interface {
/// Creates the interface and starts moving packets.
pub async fn start(
@@ -221,7 +230,29 @@ impl Interface {
let name = device.name().to_string();
let tally = Arc::new(Tally::default());
let vip_task = {
let routes = Arc::clone(&routes);
let carrier = Arc::clone(&carrier);
tokio::spawn(async move {
let mut interval = tokio::time::interval(std::time::Duration::from_millis(250));
interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
let mut checks = tokio::task::JoinSet::new();
loop {
tokio::select! {
_=interval.tick()=>{
let (probes,jobs)=routes.vips.tick(routes.vips.check_capacity().saturating_sub(checks.len()));
for (route,packet) in probes {carrier.carry(route,packet);}
for job in jobs {checks.spawn(job.run());}
},
Some(result)=checks.join_next(),if !checks.is_empty()=>{
if let Ok((job,ok))=result {routes.vips.checked(job,ok);}
}
}
}
})
};
let task = {
let carrier = Arc::clone(&carrier);
let device = Arc::clone(&device);
let routes = Arc::clone(&routes);
let tally = Arc::clone(&tally);
@@ -231,6 +262,26 @@ impl Interface {
tally.malformed.fetch_add(1, Ordering::Relaxed);
continue;
};
match routes.vips.outgoing(packet.clone()) {
crate::vip::router::Decision::Send(route, packet) => {
for part in crate::vip::packet::fragments(packet, mtu) {
if carrier.carry(route, part) {
tally.sent.fetch_add(1, Ordering::Relaxed);
} else {
tally.undeliverable.fetch_add(1, Ordering::Relaxed);
}
}
continue;
}
crate::vip::router::Decision::Local(packet) => {
for part in crate::vip::packet::fragments(packet, mtu) {
let _ = device.send(part).await;
}
continue;
}
crate::vip::router::Decision::Drop => continue,
crate::vip::router::Decision::Pass => {}
}
let destination = header.destination();
if let (IpAddr::V4(source), IpAddr::V4(destination)) =
(header.source(), destination)
@@ -279,9 +330,11 @@ impl Interface {
name,
mtu,
routes,
carrier,
tally,
applied: std::sync::Mutex::new(Vec::new()),
task: std::sync::Mutex::new(Some(task)),
vip_task: std::sync::Mutex::new(Some(vip_task)),
sync_lock: Arc::new(tokio::sync::Mutex::new(())),
host_rules,
rules_report: std::sync::Mutex::new(None),
@@ -397,6 +450,14 @@ impl Interface {
return;
}
if planned.as_ref() == Some(&wanted) && ok {
if let Some(outcome) = exit.refresh(&wanted).await {
if let RuleOutcome::Failed(reason) = &outcome {
tracing::warn!(interface = %self.name, %reason, "cannot refresh exit-client socket protection");
}
if let Some(report) = self.exit_state().report.as_mut() {
report.client = Some(outcome);
}
}
return;
}
let report = exit.apply(&wanted).await;
@@ -708,6 +769,15 @@ impl Interface {
task.abort();
let _ = task.await;
}
let vip_task = self
.vip_task
.lock()
.unwrap_or_else(|e| e.into_inner())
.take();
if let Some(task) = vip_task {
task.abort();
let _ = task.await;
}
self.factory.destroy(self.device.name()).await;
}
@@ -762,6 +832,24 @@ impl Interface {
return Err(Rejected::WrongDestination);
}
};
// Source ownership was checked before any VIP reverse translation.
let packet = match self.routes.vips.incoming(network, peer, packet.clone()) {
crate::vip::router::Decision::Send(route, packet) => {
// Probe replies use the ordinary peer carrier, never the TUN.
self.carrier.carry(route, packet);
return Ok(());
}
crate::vip::router::Decision::Local(packet) => {
for part in crate::vip::packet::fragments(packet, self.mtu) {
if self.device.send(part).await.is_ok() {
self.tally.received.fetch_add(1, Ordering::Relaxed);
}
}
return Ok(());
}
crate::vip::router::Decision::Drop => return Ok(()),
crate::vip::router::Decision::Pass => packet,
};
// Remote broadcasts terminate here. Only local TUN ingress can fan out.
if self.device.send(packet).await.is_ok() {
if broadcast {
@@ -786,6 +874,14 @@ impl Interface {
impl Drop for Interface {
fn drop(&mut self) {
if let Some(task) = self
.vip_task
.lock()
.unwrap_or_else(|e| e.into_inner())
.take()
{
task.abort();
}
if let Ok(mut guard) = self.task.lock()
&& let Some(task) = guard.take()
{
@@ -55,6 +55,7 @@ use crate::BoxFuture;
use crate::overlay::OverlayError;
use super::super::config::Cidr;
use super::super::hostrules::linux_exit_mark::{UNDERLAY_MARK, UNDERLAY_MASK};
use super::super::tun::{TunDevice, TunRequest};
use super::privilege::{NetAdmin, Privilege, probe_net_admin};
use super::{
@@ -641,8 +642,10 @@ async fn del_broadcast_route(name: &str) -> Result<(), OverlayError> {
const EXIT_TABLE: u32 = 0x7473;
/// The main table.
const MAIN_TABLE: u32 = 254;
/// Where the exit rules sit, after Tailscale's 5270.
const EXIT_BYPASS_PRIORITY: u32 = 5280;
/// Protect marked underlay and initial socket routing before Tailscale's rules.
const EXIT_MARK_PRIORITY: u32 = 5205;
const EXIT_BYPASS_PRIORITY: u32 = 5206;
const LEGACY_EXIT_BYPASS_PRIORITY: u32 = 5280;
const EXIT_SPECIFIC_PRIORITY: u32 = 5290;
const EXIT_DEFAULT_PRIORITY: u32 = 5300;
@@ -666,14 +669,23 @@ fn rule_priority(rule: &RuleMessage) -> Option<u32> {
})
}
/// Whether a rule is one of the three this agent adds for an exit node.
/// Whether a rule is one this agent adds for an exit node, including old UID priority.
///
/// Recognised by shape as well as priority: those numbers are not reserved,
/// and a rule somebody else put at one of them must not be removed.
fn is_exit_rule(rule: &RuleMessage) -> bool {
let table = rule_table(rule);
match rule_priority(rule) {
Some(EXIT_BYPASS_PRIORITY) => {
Some(EXIT_MARK_PRIORITY) => {
table == MAIN_TABLE
&& rule
.attributes
.contains(&RuleAttribute::FwMark(UNDERLAY_MARK))
&& rule
.attributes
.contains(&RuleAttribute::FwMask(UNDERLAY_MASK))
}
Some(EXIT_BYPASS_PRIORITY | LEGACY_EXIT_BYPASS_PRIORITY) => {
table == MAIN_TABLE
&& rule
.attributes
@@ -789,7 +801,8 @@ async fn remove_exit_client(handle: &rtnetlink::Handle) -> Result<(), OverlayErr
///
/// ```text
/// ip route replace default dev <if> table 0x7473
/// ip rule add priority 5280 uidrange <uid>-<uid> lookup main
/// ip rule add priority 5205 fwmark 0x40000000/0x40000000 lookup main
/// ip rule add priority 5206 uidrange <uid>-<uid> lookup main
/// ip rule add priority 5290 lookup main suppress_prefixlength 0
/// ip rule add priority 5300 lookup 0x7473
/// ```
@@ -819,6 +832,8 @@ async fn set_exit_client(name: &str, uid: u32) -> Result<(), OverlayError> {
.await
.map_err(|err| fail("default route", err))?;
add_mark_bypass(&handle, false).await?;
let mut bypass = handle
.rule()
.add()
@@ -891,6 +906,7 @@ async fn set_exit_block_v6(handle: &rtnetlink::Handle, uid: u32) -> Result<(), O
};
}
add_mark_bypass(handle, true).await?;
let mut bypass = handle
.rule()
.add()
@@ -938,6 +954,27 @@ async fn set_exit_block_v6(handle: &rtnetlink::Handle, uid: u32) -> Result<(), O
.map_err(|err| fail("rule for the default route", err))
}
async fn add_mark_bypass(handle: &rtnetlink::Handle, v6: bool) -> Result<(), OverlayError> {
let mut rule = handle
.rule()
.add()
.table_id(MAIN_TABLE)
.action(RuleAction::ToTable)
.priority(EXIT_MARK_PRIORITY)
.fw_mark(UNDERLAY_MARK);
rule.message_mut().header.family = if v6 {
rtnetlink::packet_route::AddressFamily::Inet6
} else {
rtnetlink::packet_route::AddressFamily::Inet
};
rule.message_mut()
.attributes
.push(RuleAttribute::FwMask(UNDERLAY_MASK));
rule.execute().await.map_err(|err| {
OverlayError::Unavailable(format!("cannot protect marked underlay traffic: {err}"))
})
}
async fn configure_link(configure: &Configure) -> Result<(), OverlayError> {
let name = configure.name.as_str();
with_netlink(|handle| async move {
@@ -1110,6 +1147,28 @@ mod tests {
assert!(!is_exit_rule(&rule(5280, 100, vec![uid])));
}
#[test]
fn underlay_bypass_recognises_only_our_exact_bit_and_mask() {
let rule = |table, mark, mask| {
let mut message = RuleMessage::default();
message.attributes.extend([
RuleAttribute::Table(table),
RuleAttribute::Priority(EXIT_MARK_PRIORITY),
RuleAttribute::FwMark(mark),
RuleAttribute::FwMask(mask),
]);
message
};
assert!(is_exit_rule(&rule(
MAIN_TABLE,
UNDERLAY_MARK,
UNDERLAY_MASK
)));
assert!(!is_exit_rule(&rule(MAIN_TABLE, 0x80000, 0xff0000)));
assert!(!is_exit_rule(&rule(MAIN_TABLE, UNDERLAY_MARK, u32::MAX)));
assert!(!is_exit_rule(&rule(52, UNDERLAY_MARK, UNDERLAY_MASK)));
}
#[test]
fn the_ipv6_default_of_the_exit_table_refuses_instead_of_routing() {
let route = exit_blocking_route_v6();
+41 -2
View File
@@ -72,7 +72,7 @@ pub enum RouteError {
#[error(
"network {network} uses {range}, which overlaps {other_range} already in use by \
{other}. One agent has one interface, so an address can belong to only one of \
them: give one network a different range."
them: stop the other network before starting this one."
)]
Overlap {
/// The network being added.
@@ -89,6 +89,7 @@ pub enum RouteError {
/// Address ownership across every network this agent is in.
#[derive(Debug, Default)]
pub struct RoutingTable {
pub(crate) vips: Arc<crate::vip::VipRouter>,
broadcast: ArcSwap<BroadcastTable>,
networks: RwLock<HashMap<NetworkId, NetworkRoutes>>,
}
@@ -146,6 +147,7 @@ impl RoutingTable {
/// Forgets a network.
pub fn remove_network(&self, network: NetworkId) {
self.vips.remove_network(network);
let mut networks = self.write();
networks.remove(&network);
self.broadcast
@@ -365,6 +367,43 @@ impl RoutingTable {
})
}
/// All active networks whose reserved ranges intersect this range.
pub fn overlapping_networks(
&self,
network: NetworkId,
range: Ipv4Range,
) -> Vec<(NetworkId, Ipv4Range)> {
let mut conflicts: Vec<_> = self
.read()
.iter()
.filter_map(|(other, routes)| {
let other_range = routes.range?;
(*other != network && ranges_overlap(range, other_range))
.then_some((*other, other_range))
})
.collect();
conflicts.sort_by_key(|(id, _)| *id);
conflicts
}
/// Current signed allocations used for encrypted data delivery checks.
pub(crate) fn probe_addresses(
&self,
network: NetworkId,
peer: EndpointId,
) -> Option<(Ipv4Addr, Ipv4Addr)> {
let table = self.read();
let routes = table.get(&network)?;
Some((
routes.local?,
routes
.peers
.iter()
.find(|(_, holder)| *holder == peer)
.map(|(address, _)| *address)?,
))
}
/// How many networks the table covers.
pub fn len(&self) -> usize {
self.read().len()
@@ -510,7 +549,7 @@ mod tests {
let err = table.set_network(second, routes()).unwrap_err();
let RouteError::Overlap { other, .. } = err;
assert_eq!(other, first);
assert!(err.to_string().contains("different range"), "{err}");
assert!(err.to_string().contains("stop the other network"), "{err}");
assert_eq!(table.len(), 1, "the overlapping network is not added");
// A range of its own is fine.
+66 -2
View File
@@ -20,7 +20,7 @@ use crate::error::ProtocolError;
/// The version in the ALPN is the wire-compatibility version of the control
/// protocol. It is independent of the network identity scheme version, so
/// bumping it must not change any existing [`crate::NetworkId`].
pub const ALPN: &[u8] = b"tsunagi/ctrl/5";
pub const ALPN: &[u8] = b"tsunagi/ctrl/8";
/// ALPN of the tsunagi data plane.
///
@@ -43,7 +43,9 @@ pub const MAX_DATA_PROTOCOL_LEN: usize = 32;
pub const MAX_SIGNATURE_LEN: usize = 64;
/// Control protocol version carried inside the handshake.
pub const PROTOCOL_VERSION: u16 = 5;
pub const PROTOCOL_VERSION: u16 = 8;
pub use crate::dataplane::transport::{MAX_TRANSPORT_PAYLOAD, TransportPayload};
/// First message of the handshake, sent by the initiator.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
@@ -114,6 +116,13 @@ pub struct DataOpenAck {
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[non_exhaustive]
pub enum ControlMessage {
/// Private transport negotiation; never persisted or rebroadcast.
Transport {
/// Transport protocol id; the payload stays opaque to the core.
protocol: String,
/// Secret negotiation data, redacted in diagnostics.
payload: TransportPayload,
},
/// Hostname and capability announcement.
Announce(Announcement),
/// A small request used to verify that the exchange works.
@@ -172,6 +181,16 @@ pub enum ControlMessage {
/// Short free-text reason.
reason: String,
},
/// One complete signed VIP field snapshot, merged atomically.
Vips {
/// Signed registers bounded by the per-VIP field limit.
entries: Vec<crate::vip::VipEntry>,
},
/// Signed cleanup markers, each proving the retired claim version.
Retirements {
/// Bounded by the per-message signed-state limit.
markers: Vec<crate::state::SignedRetirement>,
},
}
/// Somewhere a member has been seen.
@@ -264,6 +283,48 @@ pub fn validate_capability(
/// network, the other networks or the agent.
pub fn validate(message: &ControlMessage, limits: &Limits) -> Result<(), ProtocolError> {
match message {
ControlMessage::Vips { entries } => {
check_len("vip.fields", entries.len(), limits.max_vip_fields)?;
for e in entries {
check_len("vip.id", e.vip.len(), 32)?;
check_len("vip.key", e.key.len(), 160)?;
check_len("vip.value", e.value.as_ref().map_or(0, Vec::len), 256)?;
check_len("vip.signature", e.signature.len(), 64)?;
}
}
ControlMessage::Retirements { markers } => {
check_len("retirements", markers.len(), limits.max_state_records)?;
for marker in markers {
check_len(
"retirement.signature",
marker.signature.len(),
MAX_SIGNATURE_LEN,
)?;
check_len(
"retirement.target.signature",
marker.target.signature.len(),
MAX_SIGNATURE_LEN,
)?;
if let Some(name) = marker.target.body.hostname() {
check_len(
"retirement.target.hostname",
name.len(),
limits.max_hostname_len,
)?;
}
}
}
ControlMessage::Transport { protocol, payload } => {
if protocol.is_empty() {
return Err(ProtocolError::Malformed("empty transport protocol"));
}
check_len("transport.protocol", protocol.len(), MAX_PROTOCOL_ID_LEN)?;
check_len(
"transport.payload",
payload.as_bytes().len(),
limits.max_transport_payload_len.min(MAX_TRANSPORT_PAYLOAD),
)?;
}
ControlMessage::Peers { peers } => {
check_len("peers", peers.len(), limits.max_state_records)?;
for hint in peers {
@@ -320,10 +381,13 @@ pub fn validate(message: &ControlMessage, limits: &Limits) -> Result<(), Protoco
/// A short, stable label for a message kind, for metrics and diagnostics.
pub fn kind(message: &ControlMessage) -> &'static str {
match message {
ControlMessage::Transport { .. } => "transport",
ControlMessage::Announce(_) => "announce",
ControlMessage::Ping { .. } => "ping",
ControlMessage::Pong { .. } => "pong",
ControlMessage::State { .. } => "state",
ControlMessage::Retirements { .. } => "retirements",
ControlMessage::Vips { .. } => "vips",
ControlMessage::Reachable { .. } => "reachable",
ControlMessage::Peers { .. } => "peers",
ControlMessage::Bye { .. } => "bye",
+136 -26
View File
@@ -28,16 +28,25 @@
//! replica that has the record needs nothing else to interpret it, and
//! recovery never depends on replaying a chain from the beginning.
//!
//! Any member may also sign a separate retirement of another author's proved
//! claim version. It suppresses stale reservations without changing that
//! author's record, and a newer claim remains valid.
//!
//! # What this slice does not do yet
//!
//! Compaction, revocation of a whole author, and snapshots covering more than
//! one record type. See `docs/sync-model.md` for the shape those take.
//! Compaction and permanent revocation of a whole author. See
//! `docs/sync-model.md` for the implemented cleanup semantics.
pub mod allocator;
/// Signed shared virtual service state and its merge rules.
pub mod vip;
use std::collections::HashMap;
use std::net::Ipv4Addr;
mod retirement;
pub use retirement::{MAX_RETIREMENT_LEN, SignedRetirement};
use iroh::{EndpointId, SecretKey, Signature};
use serde::{Deserialize, Serialize};
@@ -109,12 +118,8 @@ impl std::str::FromStr for Ipv4Range {
}
}
/// The IPv4 overlay range used unless something else is configured or agreed.
///
/// A small, specific `/24`: memorable, and far less likely to overlap a
/// network the machine is already on than taking a whole `/8` or `/10` would
/// be. Because addresses are allocated rather than derived, 254 of them is
/// plenty for the size of network this is for.
/// The legacy default range, retained for existing networks and explicit use.
/// New networks use [`derived_ipv4_range`] by default.
pub const DEFAULT_IPV4_RANGE: Ipv4Range = Ipv4Range {
base: Ipv4Addr::new(10, 13, 37, 0),
prefix_len: 24,
@@ -122,15 +127,10 @@ pub const DEFAULT_IPV4_RANGE: Ipv4Range = Ipv4Range {
/// A range derived from a network's own identity.
///
/// One agent has one interface, so two of its networks cannot both use the
/// configured range. The second needs another one — and it cannot be picked
/// locally, because every member has to arrive at the same answer without
/// being told. So it comes from the network id, which every member already
/// has and nobody can choose: the same network derives the same range on
/// every device.
///
/// Inside 10/8 like the default, and never equal to it, so the first
/// network keeps what it has always had.
/// Every member independently derives the same pseudorandom `/24` in 10/8,
/// including isolated first members. Stored signed ranges take precedence.
/// The derivation is kept stable for networks already using it; the legacy
/// default is excluded. Distinct network ids can still produce collisions.
pub fn derived_ipv4_range(network: NetworkId) -> Ipv4Range {
let bytes = network.as_bytes();
let second = bytes[0];
@@ -497,6 +497,7 @@ pub enum Merged {
#[derive(Debug, Clone, Default)]
pub struct StateSet {
records: HashMap<EndpointId, SignedRecord>,
retirements: HashMap<EndpointId, SignedRetirement>,
}
impl StateSet {
@@ -509,6 +510,7 @@ impl StateSet {
pub fn from_verified(records: impl IntoIterator<Item = (EndpointId, SignedRecord)>) -> Self {
Self {
records: records.into_iter().collect(),
retirements: HashMap::new(),
}
}
@@ -537,6 +539,52 @@ impl StateSet {
.collect()
}
/// Visible records, excluding versions retired by authenticated members.
pub fn active_records(&self) -> Vec<SignedRecord> {
self.records()
.into_iter()
.filter(|r| r.author_id().is_ok_and(|id| !self.is_retired(&id)))
.collect()
}
/// Whether the current known version is at or below a proved retirement.
pub fn is_retired(&self, author: &EndpointId) -> bool {
self.retirements
.get(author)
.is_some_and(|m| self.version_of(author) <= m.target.version)
}
/// Strongest retained cleanup proof for an identity.
pub fn retirement_of(&self, target: &EndpointId) -> Option<&SignedRetirement> {
self.retirements.get(target)
}
/// Durable markers in target-id order, including markers surpassed by a fresh claim.
pub fn retirements(&self) -> Vec<SignedRetirement> {
let mut markers: Vec<_> = self.retirements.values().cloned().collect();
markers.sort_by_key(|m| m.target.author);
markers
}
/// Accepts any member's signed cleanup after checking both signatures.
pub fn merge_retirement(
&mut self,
network: NetworkId,
marker: SignedRetirement,
max: usize,
) -> Result<bool, StateError> {
let target = marker.verify(network)?;
if let Some(old) = self.retirements.get(&target) {
if !marker.supersedes(old) {
return Ok(false);
}
} else if self.retirements.len() >= max {
return Err(StateError::Malformed("retirement capacity reached"));
}
self.retirements.insert(target, marker);
Ok(true)
}
/// Merges one record, verifying it first.
///
/// Merging is into the existing set, never a wholesale replacement, and an
@@ -600,6 +648,9 @@ impl StateSet {
pub fn address_holders(&self) -> HashMap<Ipv4Addr, EndpointId> {
let mut holders: HashMap<Ipv4Addr, EndpointId> = HashMap::new();
for (author, record) in &self.records {
if self.is_retired(author) {
continue;
}
let Some(address) = record.body.claimed_address() else {
continue;
};
@@ -627,6 +678,9 @@ impl StateSet {
pub fn hostname_holders(&self) -> HashMap<String, EndpointId> {
let mut holders: HashMap<String, (u64, EndpointId)> = HashMap::new();
for (author, record) in &self.records {
if self.is_retired(author) {
continue;
}
let Some(hostname) = record.body.hostname() else {
continue;
};
@@ -655,6 +709,9 @@ impl StateSet {
/// The name an author claimed and the member that holds it instead, when
/// somebody who claimed it earlier does.
pub fn hostname_taken(&self, author: &EndpointId) -> Option<(&str, EndpointId)> {
if self.is_retired(author) {
return None;
}
let hostname = self.records.get(author)?.body.hostname()?;
let holder = *self.hostname_holders().get(hostname)?;
(holder != *author).then_some((hostname, holder))
@@ -673,21 +730,44 @@ impl StateSet {
/// The range the network settled on, if anybody has said.
///
/// When claims disagree, the one from the lowest author id wins, so every
/// replica reaches the same answer. A participant joining later therefore
/// adopts the range already in use instead of imposing its own.
/// A legacy or explicit range takes precedence over the derived default,
/// so a fresh member's default cannot renumber an established network.
/// Other disagreements use the lowest author id, independently of arrival
/// order. The signed record format and network identity remain unchanged.
pub fn agreed_range(&self) -> Option<Ipv4Range> {
let mut authors: Vec<&EndpointId> = self.records.keys().collect();
authors.sort_by_key(|author| *author.as_bytes());
authors
.into_iter()
.find_map(|author| self.records.get(author)?.body.range())
self.records
.iter()
.chain(
self.retirements
.iter()
.filter(|(author, _)| !self.records.contains_key(*author))
.map(|(author, marker)| (author, &marker.target)),
)
.filter_map(|(author, record)| {
let range = record.body.range()?;
let derived = derived_ipv4_range(NetworkId::from_bytes(record.network));
Some(((range == derived, *author.as_bytes()), range))
})
.min_by_key(|(priority, _)| *priority)
.map(|(_, range)| range)
}
/// The highest version this author has published, as far as is known.
pub fn version_of(&self, author: &EndpointId) -> u64 {
self.records.get(author).map_or(0, |record| record.version)
}
/// Highest witnessed author counter across claims and authored cleanup.
/// This differs from the claim version used to decide retirement.
pub fn authored_version(&self, author: &EndpointId) -> u64 {
self.retirements
.values()
.filter(|m| m.author == *author.as_bytes())
.map(|m| m.version)
.chain(std::iter::once(self.version_of(author)))
.max()
.unwrap_or(0)
}
}
#[cfg(test)]
@@ -1263,7 +1343,7 @@ mod tests {
assert_eq!(once.base.octets()[0], 10, "inside 10/8 like the default");
assert_ne!(
once, DEFAULT_IPV4_RANGE,
"the default belongs to whichever network asked first"
"the legacy default remains excluded"
);
// A different network derives a different range, which is the
@@ -1275,4 +1355,34 @@ mod tests {
.network_id();
assert_ne!(once, derived_ipv4_range(other));
}
#[test]
fn a_derived_default_cannot_displace_a_legacy_or_explicit_range() {
let id = network("range-migration");
let a = SecretKey::generate();
let b = SecretKey::generate();
let (low, high) = if a.public().as_bytes() < b.public().as_bytes() {
(a, b)
} else {
(b, a)
};
for established in [DEFAULT_IPV4_RANGE, "10.90.0.0/16".parse().unwrap()] {
let body = |range: Ipv4Range| RecordBody::Claim {
address: Some(Ipv4Addr::from(u32::from(range.base) + 1)),
range: Some(range),
hostname: None,
hostname_since: None,
};
let fresh = SignedRecord::sign(&low, id, 1, body(derived_ipv4_range(id)));
let existing = SignedRecord::sign(&high, id, 1, body(established));
for records in [
vec![fresh.clone(), existing.clone()],
vec![existing.clone(), fresh.clone()],
] {
let mut state = StateSet::new();
state.merge_all(id, records);
assert_eq!(state.agreed_range(), Some(established));
}
}
}
}
+219
View File
@@ -0,0 +1,219 @@
//! A member may retire an observed claim, never another identity's future.
use iroh::{EndpointId, SecretKey, Signature};
use serde::{Deserialize, Serialize};
use super::{RecordBody, SignedRecord, StateError, push_lp};
use crate::NetworkId;
/// Maximum encoded retirement size, including its bounded claim proof.
pub const MAX_RETIREMENT_LEN: usize = 1024;
/// A network member's signed retirement of a proved claim version.
///
/// The original author's signed claim proves the cutoff really existed.
/// Newer claims remain valid. This is neither a forged Release nor a ban.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct SignedRetirement {
/// Device identity of the member requesting cleanup.
pub author: [u8; 32],
/// Network in which cleanup applies.
pub network: [u8; 32],
/// Requester's durable author counter, committed with this marker.
pub version: u64,
/// The target's original signed claim; bounds the retired version.
pub target: SignedRecord,
/// Requester's Ed25519 signature over this marker and its proof.
pub signature: Vec<u8>,
}
impl SignedRetirement {
fn canonical_bytes(&self) -> Vec<u8> {
let mut out = Vec::with_capacity(768);
push_lp(&mut out, b"tsunagi-node-retirement/v1");
push_lp(&mut out, &self.network);
push_lp(&mut out, &self.author);
push_lp(&mut out, &self.version.to_be_bytes());
// Bind every field of the original signed claim, including its signature.
push_lp(&mut out, &self.target.network);
push_lp(&mut out, &self.target.author);
push_lp(&mut out, &self.target.version.to_be_bytes());
push_lp(&mut out, &self.target.body.canonical());
push_lp(&mut out, &self.target.signature);
out
}
/// Signs a marker with the requesting member's persistent device key.
pub fn sign(
secret: &SecretKey,
network: NetworkId,
version: u64,
target: SignedRecord,
) -> Self {
let mut marker = Self {
author: *secret.public().as_bytes(),
network: *network.as_bytes(),
version,
target,
signature: Vec::new(),
};
marker.signature = secret.sign(&marker.canonical_bytes()).to_bytes().to_vec();
marker
}
/// Verifies both authors, scope and a finite, genuinely signed cutoff.
pub fn verify(&self, network: NetworkId) -> Result<EndpointId, StateError> {
if self.network != *network.as_bytes() {
return Err(StateError::WrongNetwork);
}
if self.version == 0 || self.version > i64::MAX as u64 {
return Err(StateError::Malformed("retirement version is out of range"));
}
if !matches!(self.target.body, RecordBody::Claim { .. }) {
return Err(StateError::Malformed("only a claim can be retired"));
}
let target = self.target.verify(network)?;
let author = EndpointId::from_bytes(&self.author).map_err(|_| StateError::BadAuthor)?;
if author == target {
return Err(StateError::Malformed(
"use Release to retire your own claim",
));
}
let raw: [u8; Signature::LENGTH] =
self.signature
.as_slice()
.try_into()
.map_err(|_| StateError::BadSignatureLength {
expected: Signature::LENGTH,
})?;
author
.verify(&self.canonical_bytes(), &Signature::from_bytes(&raw))
.map_err(|_| StateError::BadSignature)?;
Ok(target)
}
/// A deterministic strongest marker per target, independent of arrival order.
pub(crate) fn supersedes(&self, other: &Self) -> bool {
self.target.version > other.target.version
|| (self.target.version == other.target.version
&& (self.canonical_bytes(), &self.signature)
< (other.canonical_bytes(), &other.signature))
}
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
use crate::state::{Ipv4Range, StateSet};
fn claim(key: &SecretKey, id: NetworkId, version: u64) -> SignedRecord {
SignedRecord::sign(
key,
id,
version,
RecordBody::Claim {
address: Some("10.23.0.2".parse().unwrap()),
range: Some("10.23.0.0/24".parse::<Ipv4Range>().unwrap()),
hostname: Some("retired".into()),
hostname_since: Some(123),
},
)
}
#[test]
fn markers_release_claims_in_any_merge_order_and_older_snapshots_cannot_restore_them() {
let id = NetworkId::from_bytes([7; 32]);
let target = SecretKey::generate();
let cleaner = SecretKey::generate();
let record = claim(&target, id, 5);
let marker = SignedRetirement::sign(&cleaner, id, 8, record.clone());
for first in [true, false] {
let mut state = StateSet::new();
if first {
state.merge_retirement(id, marker.clone(), 16).unwrap();
}
state.merge(id, record.clone()).unwrap();
if !first {
state.merge_retirement(id, marker.clone(), 16).unwrap();
}
assert!(state.address_holders().is_empty());
assert!(state.hostname_holders().is_empty());
state.merge(id, claim(&target, id, 4)).unwrap();
assert!(state.is_retired(&target.public()));
assert!(
state.agreed_range().is_some(),
"cleanup does not renumber the network"
);
state.merge(id, claim(&target, id, 6)).unwrap();
assert!(!state.is_retired(&target.public()));
assert_eq!(state.hostname_of(&target.public()), Some("retired"));
}
}
#[test]
fn proof_signature_scope_tampering_future_cutoffs_and_capacity_are_checked() {
let id = NetworkId::from_bytes([7; 32]);
let target = SecretKey::generate();
let cleaner = SecretKey::generate();
let marker = SignedRetirement::sign(&cleaner, id, 8, claim(&target, id, 5));
assert!(marker.verify(NetworkId::from_bytes([8; 32])).is_err());
let mut forged = marker.clone();
forged.target.version = 99;
assert!(forged.verify(id).is_err());
let mut forged = marker.clone();
forged.signature[0] ^= 1;
assert!(forged.verify(id).is_err());
let mut proof = marker.target.clone();
proof.version = 99;
assert!(
SignedRetirement::sign(&cleaner, id, 8, proof)
.verify(id)
.is_err()
);
let mut state = StateSet::new();
assert!(state.merge_retirement(id, marker.clone(), 0).is_err());
assert!(state.merge_retirement(id, marker.clone(), 1).unwrap());
assert!(!state.merge_retirement(id, marker.clone(), 1).unwrap());
assert!(postcard::to_stdvec(&marker).unwrap().len() <= MAX_RETIREMENT_LEN);
let another = SecretKey::generate();
assert!(
state
.merge_retirement(
id,
SignedRetirement::sign(&cleaner, id, 9, claim(&another, id, 1)),
1
)
.is_err()
);
}
#[test]
fn competing_cleanup_markers_converge_and_keep_the_highest_observed_cutoff() {
let id = NetworkId::from_bytes([7; 32]);
let target = SecretKey::generate();
let a = SecretKey::generate();
let b = SecretKey::generate();
let markers = [
SignedRetirement::sign(&a, id, 1, claim(&target, id, 4)),
SignedRetirement::sign(&a, id, 2, claim(&target, id, 5)),
SignedRetirement::sign(&b, id, 3, claim(&target, id, 5)),
];
let mut forward = StateSet::new();
let mut reverse = StateSet::new();
for marker in &markers {
forward.merge_retirement(id, marker.clone(), 16).unwrap();
}
for marker in markers.iter().rev() {
reverse.merge_retirement(id, marker.clone(), 16).unwrap();
}
assert_eq!(forward.retirements(), reverse.retirements());
assert_eq!(
forward
.retirement_of(&target.public())
.unwrap()
.target
.version,
5
);
}
}
+439
View File
@@ -0,0 +1,439 @@
//! Signed multi-writer virtual service registers and deterministic merge rules.
use crate::error::Result;
use crate::vip::{Backend, Listener, Protocol, VipConfig, VipStatus, invalid};
use crate::{Limits, NetworkId};
use iroh::{EndpointId, SecretKey, Signature};
use serde::{Deserialize, Serialize};
use sha2::{Digest, Sha256};
use std::collections::{BTreeMap, BTreeSet};
use std::net::Ipv4Addr;
/// A signed, independently mergeable field. A removed field keeps its tombstone.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct VipEntry {
/// Network scope.
pub network: [u8; 32],
/// Stable service id.
pub vip: String,
/// Canonical field key.
pub key: String,
/// Encoded field value; absent means deletion.
pub value: Option<Vec<u8>>,
/// Lamport clock, independent of wall time.
pub clock: u64,
/// Persistent signing identity.
pub author: [u8; 32],
/// Author's durable network counter.
pub version: u64,
/// Ed25519 signature.
pub signature: Vec<u8>,
}
impl VipEntry {
fn bytes(&self) -> Vec<u8> {
let mut e = self.clone();
e.signature.clear();
let mut out = b"tsunagi-vip-field/v1\0".to_vec();
if let Ok(bytes) = postcard::to_stdvec(&e) {
out.extend(bytes);
}
out
}
/// Verifies bounds, network and authorship before the entry is merged.
pub fn verify(&self, network: NetworkId) -> Result<()> {
if self.network != *network.as_bytes()
|| self.vip.len() != 32
|| hex::decode(&self.vip).is_err()
|| self.key.len() > 160
|| self.value.as_ref().is_some_and(|v| v.len() > 256)
|| self.version == 0
|| self.version > i64::MAX as u64
|| self.clock == 0
|| self.clock > i64::MAX as u64
{
return Err(invalid("invalid VIP field bounds or network"));
}
let author =
EndpointId::from_bytes(&self.author).map_err(|_| invalid("invalid VIP author"))?;
let raw: [u8; 64] = self
.signature
.as_slice()
.try_into()
.map_err(|_| invalid("invalid VIP signature length"))?;
author
.verify(&self.bytes(), &Signature::from_bytes(&raw))
.map_err(|_| invalid("invalid VIP signature"))?;
if self.key != "deleted"
&& self.key != "name"
&& !self.key.starts_with("l/")
&& !self.key.starts_with("b/")
{
return Err(invalid("unknown VIP field"));
}
match self.key.as_str() {
"deleted" => {
if self.value.as_deref() != Some(&[1][..]) {
return Err(invalid("invalid VIP deletion tombstone"));
}
}
"name" => {
let value = self
.value
.as_ref()
.ok_or_else(|| invalid("VIP name cannot be absent"))?;
let name: String =
postcard::from_bytes(value).map_err(|_| invalid("invalid VIP name field"))?;
VipConfig {
id: self.vip.clone(),
name,
listeners: vec![],
}
.validate(&Limits::default())?;
}
key if key.starts_with("l/") => {
let parts: Vec<_> = key.split('/').collect();
if parts.len() != 3
|| parts[1].parse::<Protocol>().is_err()
|| parts[2].parse::<u16>().ok().is_none_or(|p| p == 0)
{
return Err(invalid("invalid listener field key"));
}
if let Some(value) = &self.value {
let l: Listener = postcard::from_bytes(value)
.map_err(|_| invalid("invalid listener field"))?;
if !l.backends.is_empty() || key != format!("l/{}/{}", l.protocol, l.port) {
return Err(invalid("listener key does not match its value"));
}
VipConfig {
id: self.vip.clone(),
name: "validation".into(),
listeners: vec![l],
}
.validate(&Limits::default())?;
}
}
key if key.starts_with("b/") => {
let parts: Vec<_> = key.split('/').collect();
if parts.len() != 6
|| parts[1].parse::<Protocol>().is_err()
|| parts[2].parse::<u16>().ok().is_none_or(|p| p == 0)
|| parts[3].len() != 32
|| hex::decode(parts[3]).is_err()
|| parts[4].parse::<EndpointId>().is_err()
|| parts[5].parse::<u16>().ok().is_none_or(|p| p == 0)
{
return Err(invalid("invalid backend field key"));
}
if let Some(value) = &self.value {
let b: Backend = postcard::from_bytes(value)
.map_err(|_| invalid("invalid backend field"))?;
if b.peer != parts[4] || b.port.to_string() != parts[5] {
return Err(invalid("backend key does not match its value"));
}
}
}
_ => return Err(invalid("unknown VIP field")),
}
Ok(())
}
fn rank(&self) -> (u64, [u8; 32], Vec<u8>) {
(self.clock, self.author, self.bytes())
}
}
/// Signed registers retained across absences, including deletion tombstones.
#[derive(Debug, Clone, Default)]
pub struct VipSet {
entries: BTreeMap<(String, String), VipEntry>,
}
impl VipSet {
/// Highest retained version genuinely signed by this author.
pub fn authored_version(&self, author: EndpointId) -> u64 {
self.entries
.values()
.filter(|e| e.author == *author.as_bytes())
.map(|e| e.version)
.max()
.unwrap_or(0)
}
/// Complete independently verifiable state of one service, including tombstones.
pub fn entries(&self, id: &str) -> Vec<VipEntry> {
self.entries
.values()
.filter(|e| e.vip == id)
.cloned()
.collect()
}
/// All retained service ids, including deleted services.
pub fn ids(&self) -> Vec<String> {
self.entries
.keys()
.map(|(id, _)| id.clone())
.collect::<BTreeSet<_>>()
.into_iter()
.collect()
}
/// Opaque optimistic concurrency token for a whole service.
pub fn revision(&self, id: &str) -> String {
let mut hash = Sha256::new();
for e in self.entries(id) {
hash.update(e.bytes());
hash.update(&e.signature);
}
hex::encode(hash.finalize())
}
/// Atomically verifies and merges one complete service snapshot.
pub fn merge(
&mut self,
network: NetworkId,
entries: Vec<VipEntry>,
limits: &Limits,
) -> Result<bool> {
if entries.len() > limits.max_vip_fields {
return Err(invalid("too many VIP fields"));
}
let mut next = self.clone();
for e in entries {
e.verify(network)?;
let key = (e.vip.clone(), e.key.clone());
if next
.entries
.get(&key)
.is_none_or(|old| e.rank() > old.rank())
{
next.entries.insert(key, e);
}
}
let deleted: BTreeSet<_> = next
.entries
.values()
.filter(|e| e.key == "deleted" && e.value.is_some())
.map(|e| e.vip.clone())
.collect();
// A permanent service tombstone alone prevents every former field's replay.
next.entries
.retain(|(id, key), _| !deleted.contains(id) || key == "deleted");
let ids = next.ids();
if ids.len().saturating_sub(deleted.len()) > limits.max_vips
|| deleted.len() > limits.max_vip_tombstones
|| ids
.iter()
.any(|id| next.entries(id).len() > limits.max_vip_fields)
{
return Err(invalid(
"VIP state capacity exceeded; deletion protection is retained",
));
}
let changed = self.entries != next.entries;
*self = next;
Ok(changed)
}
/// Materialized configuration. A service tombstone permanently retires its id.
pub fn configs(&self) -> Vec<VipConfig> {
self.ids()
.into_iter()
.filter_map(|id| self.config(&id))
.collect()
}
fn config(&self, id: &str) -> Option<VipConfig> {
let fields = self.entries(id);
if fields
.iter()
.any(|e| e.key == "deleted" && e.value.is_some())
{
return None;
}
let name = fields
.iter()
.find(|e| e.key == "name")?
.value
.as_ref()
.and_then(|v| postcard::from_bytes::<String>(v).ok())?;
let mut listeners: Vec<Listener> = fields
.iter()
.filter(|e| e.key.starts_with("l/"))
.filter_map(|e| e.value.as_ref().and_then(|v| postcard::from_bytes(v).ok()))
.collect();
for l in &mut listeners {
l.backends = fields
.iter()
.filter(|e| {
e.key
.starts_with(&format!("b/{}/{}/{}/", l.protocol, l.port, l.id))
})
.filter_map(|e| e.value.as_ref().and_then(|v| postcard::from_bytes(v).ok()))
.collect();
l.backends.sort();
}
listeners.sort_by_key(|l| (l.protocol, l.port));
Some(VipConfig {
id: id.into(),
name,
listeners,
})
}
/// Builds a signed patch from the user's complete, version-checked draft.
pub fn edit(
&self,
network: NetworkId,
secret: &SecretKey,
version: u64,
config: &VipConfig,
expected: Option<&str>,
limits: &Limits,
) -> Result<Vec<VipEntry>> {
config.validate(limits)?;
if let Some(revision) = expected {
if revision != self.revision(&config.id) {
return Err(invalid("VIP changed while editing; reload before saving"));
}
} else if !self.entries(&config.id).is_empty() {
return Err(invalid("VIP id already exists"));
}
if self
.entries
.get(&(config.id.clone(), "deleted".into()))
.is_some_and(|e| e.value.is_some())
{
return Err(invalid("VIP was deleted; create a new VIP"));
}
let before = self
.config(&config.id)
.map(|c| flatten(&c))
.transpose()?
.unwrap_or_default();
let after = flatten(config)?;
let keys: BTreeSet<_> = before.keys().chain(after.keys()).cloned().collect();
let changes = keys
.into_iter()
.filter_map(|key| {
let value = after.get(&key).cloned();
(before.get(&key) != value.as_ref()).then_some((key, value))
})
.collect();
self.sign(network, secret, version, &config.id, changes)
}
/// Builds a permanent service tombstone after checking the selected revision.
pub fn delete(
&self,
network: NetworkId,
secret: &SecretKey,
version: u64,
id: &str,
expected: &str,
) -> Result<Vec<VipEntry>> {
if self.config(id).is_none() || self.revision(id) != expected {
return Err(invalid("VIP missing or changed while editing"));
}
self.sign(
network,
secret,
version,
id,
vec![("deleted".into(), Some(vec![1]))],
)
}
fn sign(
&self,
network: NetworkId,
secret: &SecretKey,
version: u64,
id: &str,
changes: Vec<(String, Option<Vec<u8>>)>,
) -> Result<Vec<VipEntry>> {
let clock = self
.entries
.values()
.map(|e| e.clock)
.max()
.unwrap_or(0)
.checked_add(1)
.filter(|n| *n <= i64::MAX as u64)
.ok_or_else(|| invalid("VIP clock exhausted"))?;
Ok(changes
.into_iter()
.map(|(key, value)| {
let mut e = VipEntry {
network: *network.as_bytes(),
vip: id.into(),
key,
value,
clock,
author: *secret.public().as_bytes(),
version,
signature: vec![],
};
e.signature = secret.sign(&e.bytes()).to_bytes().to_vec();
e
})
.collect())
}
}
pub(crate) fn flatten(cfg: &VipConfig) -> Result<BTreeMap<String, Vec<u8>>> {
let mut fields = BTreeMap::new();
fields.insert(
"name".into(),
postcard::to_stdvec(&cfg.name).map_err(|e| invalid(e.to_string()))?,
);
for l in &cfg.listeners {
let mut settings = l.clone();
settings.backends.clear();
fields.insert(
format!("l/{}/{}", l.protocol, l.port),
postcard::to_stdvec(&settings).map_err(|e| invalid(e.to_string()))?,
);
for b in &l.backends {
fields.insert(
format!("b/{}/{}/{}/{}/{}", l.protocol, l.port, l.id, b.peer, b.port),
postcard::to_stdvec(b).map_err(|e| invalid(e.to_string()))?,
);
}
}
Ok(fields)
}
impl VipSet {
/// Allocates internal addresses deterministically against durable node claims.
pub(crate) fn views(
&self,
range: Option<crate::state::Ipv4Range>,
mut taken: std::collections::HashSet<Ipv4Addr>,
hostnames: &BTreeSet<String>,
limits: &Limits,
) -> Vec<VipStatus> {
let mut names = hostnames.clone();
let mut result = Vec::new();
for config in self.configs() {
let mut error = config.validate(limits).err().map(|e| e.to_string());
if !names.insert(config.name.clone()) {
error = Some("name conflicts with another VIP or node".into());
}
let address = range.and_then(|r| {
let usable = crate::state::allocator::usable_addresses(r);
if usable == 0 {
return None;
}
let digest = Sha256::digest(config.id.as_bytes());
let seed = u64::from_be_bytes([
digest[0], digest[1], digest[2], digest[3], digest[4], digest[5], digest[6],
digest[7],
]);
let bits = 32 - u32::from(r.prefix_len);
let mask = if bits == 32 { 0 } else { u32::MAX << bits };
(0..usable).find_map(|step| {
let offset = (seed.wrapping_add(step) % usable + 1) as u32;
let ip = Ipv4Addr::from((u32::from(r.base) & mask) | offset);
taken.insert(ip).then_some(ip)
})
});
if address.is_none() {
error = Some("no free overlay address".into());
}
result.push(VipStatus {
revision: self.revision(&config.id),
config,
address,
error,
listeners: vec![],
});
}
result
}
}
+78
View File
@@ -254,6 +254,12 @@ impl Storage {
.await
}
/// Persists the ordered protocol selection off the async executor.
pub async fn set_protocols(&self, network_id: NetworkId, protocols: Vec<String>) -> Result<()> {
self.with_state(move |state| state.set_protocols(network_id, &protocols))
.await
}
/// Persists whether this agent offers itself as an exit node in a network.
pub async fn set_exit_node(&self, network_id: NetworkId, enabled: bool) -> Result<()> {
self.with_state(move |state| state.set_exit_node(network_id, enabled))
@@ -272,11 +278,39 @@ impl Storage {
.await
}
/// Configured networks in the user's most recently selected order.
pub async fn network_start_order(&self) -> Result<Vec<NetworkId>> {
self.with_state(|state| {
let configured = state.list_networks()?;
let text = state
.get_setting("network_start_order")?
.unwrap_or_default();
let mut order = Vec::new();
for word in text.split(',') {
if let Ok(id) = word.parse::<NetworkId>()
&& configured.iter().any(|n| n.network_id == id)
&& !order.contains(&id)
{
order.push(id);
}
}
Ok(order)
})
.await
}
/// Prefers this network at startup without disabling suspended networks.
pub async fn prefer_network(&self, network: NetworkId) -> Result<()> {
self.with_state(move |state| state.prefer_network(network))
.await
}
/// Removes a network configuration and its cached hints.
pub async fn remove_network(&self, network_id: NetworkId) -> Result<()> {
self.with_state(move |state| {
state.remove_network(network_id)?;
state.forget_signed_records(network_id)?;
state.forget_retirements(network_id)?;
state.forget_peer_hostnames(network_id)
})
.await?;
@@ -336,6 +370,50 @@ impl Storage {
.await
}
/// Loads mandatory signed VIP fields with a strict capacity bound.
pub async fn vip_fields(
&self,
network: NetworkId,
max: usize,
) -> Result<Vec<crate::vip::VipEntry>> {
self.with_state(move |state| state.vip_fields(network, max))
.await
}
/// Stores a complete merged service snapshot and optional own counter.
pub async fn put_vip_fields(
&self,
network: NetworkId,
entries: Vec<crate::vip::VipEntry>,
own: Option<([u8; 32], u64)>,
) -> Result<()> {
self.with_state(move |state| state.put_vip_fields(network, &entries, own))
.await
}
/// Loads verified durable cleanup proofs off the executor.
pub async fn retirements(
&self,
network: NetworkId,
max: usize,
) -> Result<Vec<crate::state::SignedRetirement>> {
self.with_state(move |state| state.retirements(network, max))
.await
}
/// Persists a received cleanup proof before applying it.
pub async fn put_retirement(&self, marker: crate::state::SignedRetirement) -> Result<()> {
self.with_state(move |state| state.put_retirement(&marker))
.await
}
/// Commits our cleanup and author counter together before announcement.
pub async fn publish_own_retirement(
&self,
marker: crate::state::SignedRetirement,
) -> Result<()> {
self.with_state(move |state| state.publish_own_retirement(&marker))
.await
}
/// Stores a record received from another replica.
pub async fn put_signed_record(&self, record: SignedRecord) -> Result<()> {
self.with_state(move |state| state.put_signed_record(&record))
+501 -5
View File
@@ -18,10 +18,10 @@ use rusqlite::{Connection, OptionalExtension, params};
use crate::error::{Error, Result};
use crate::identity::{DeviceIdentity, NetworkId, NetworkName, NetworkSecret};
use crate::state::{RecordBody, SignedRecord};
use crate::state::{MAX_RETIREMENT_LEN, RecordBody, SignedRecord, SignedRetirement};
/// Schema version written by this build.
pub const SCHEMA_VERSION: i64 = 7;
pub const SCHEMA_VERSION: i64 = 10;
/// Key of the stored hostname setting.
const SETTING_HOSTNAME: &str = "hostname";
@@ -32,6 +32,8 @@ const SETTING_HOSTNAME: &str = "hostname";
/// and zeroizes on drop.
#[derive(Debug, Clone)]
pub struct StoredNetwork {
/// Ordered local protocol override; absent inherits the agent's defaults.
pub protocols: Option<Vec<String>>,
/// Derived public network identifier.
pub network_id: NetworkId,
/// Network name.
@@ -238,6 +240,35 @@ impl StateStore {
)
.map_err(|err| self.corrupt(format!("cannot migrate schema to 7: {err}")))?;
}
if found < 8 {
self.conn
.execute_batch(
"BEGIN;
ALTER TABLE networks ADD COLUMN protocols TEXT;
PRAGMA user_version = 8;
COMMIT;",
)
.map_err(|err| self.corrupt(format!("cannot migrate schema to 8: {err}")))?;
}
if found < 9 {
self.conn
.execute_batch(
"BEGIN;
CREATE TABLE IF NOT EXISTS retirements (
network_id BLOB NOT NULL, target BLOB NOT NULL, data BLOB NOT NULL,
PRIMARY KEY (network_id, target)
);
PRAGMA user_version = 9;
COMMIT;",
)
.map_err(|err| self.corrupt(format!("cannot migrate schema to 9: {err}")))?;
}
if found < 10 {
self.conn.execute_batch("BEGIN; CREATE TABLE IF NOT EXISTS vip_fields (
network_id BLOB NOT NULL, vip TEXT NOT NULL, field TEXT NOT NULL, data BLOB NOT NULL,
PRIMARY KEY(network_id, vip, field)); PRAGMA user_version = 10; COMMIT;")
.map_err(|err| self.corrupt(format!("cannot migrate VIP state: {err}")))?;
}
Ok(())
}
@@ -250,6 +281,8 @@ impl StateStore {
"settings",
"signed_records",
"peer_hostnames",
"retirements",
"vip_fields",
] {
let present: Option<String> = self
.conn
@@ -448,12 +481,16 @@ impl StateStore {
/// Sets the auto-start flag of a configured network.
pub fn set_auto_start(&self, network_id: NetworkId, auto_start: bool) -> Result<()> {
self.conn
let changed = self
.conn
.execute(
"UPDATE networks SET auto_start = ?2 WHERE network_id = ?1",
params![network_id.as_bytes().as_slice(), auto_start as i64],
)
.map_err(|err| Error::Storage(format!("cannot update network: {err}")))?;
if changed == 0 {
return Err(Error::NetworkUnknown(network_id));
}
Ok(())
}
@@ -472,6 +509,25 @@ impl StateStore {
Ok(())
}
/// Saves a local ordered protocol selection without changing membership.
pub fn set_protocols(&self, network_id: NetworkId, protocols: &[String]) -> Result<()> {
crate::dataplane::validate_protocols(
protocols,
crate::config::Limits::default().max_capabilities,
)?;
let changed = self
.conn
.execute(
"UPDATE networks SET protocols = ?2 WHERE network_id = ?1",
params![network_id.as_bytes().as_slice(), protocols.join(",")],
)
.map_err(|err| Error::Storage(format!("cannot update protocols: {err}")))?;
if changed == 0 {
return Err(Error::NetworkUnknown(network_id));
}
Ok(())
}
/// Removes a network configuration entirely.
pub fn remove_network(&self, network_id: NetworkId) -> Result<()> {
self.conn
@@ -532,7 +588,7 @@ impl StateStore {
pub fn list_networks(&self) -> Result<Vec<StoredNetwork>> {
let mut stmt = self
.conn
.prepare("SELECT network_id, name, secret, auto_start, broadcast, exit_node, exit_via FROM networks ORDER BY name")
.prepare("SELECT network_id, name, secret, auto_start, broadcast, exit_node, exit_via, protocols FROM networks ORDER BY name")
.map_err(|err| Error::Storage(format!("cannot list networks: {err}")))?;
let rows = stmt
.query_map([], |row| {
@@ -548,19 +604,37 @@ impl StateStore {
row.get::<_, bool>(4)?,
row.get::<_, bool>(5)?,
row.get::<_, Option<Vec<u8>>>(6)?,
row.get::<_, Option<String>>(7)?,
))
})
.map_err(|err| Error::Storage(format!("cannot list networks: {err}")))?;
let mut out = Vec::new();
for row in rows {
let (id, name, secret, auto_start, broadcast, exit_node, exit_via) =
let (id, name, secret, auto_start, broadcast, exit_node, exit_via, protocols) =
row.map_err(|err| Error::Storage(format!("cannot read network row: {err}")))?;
let protocols = protocols
.map(|text| {
let max = crate::config::Limits::default().max_capabilities;
if text.len() > max * (crate::dataplane::MAX_PROTOCOL_ID_LEN + 1) {
return Err(self.corrupt("stored protocol selection is too long"));
}
let names = if text.is_empty() {
Vec::new()
} else {
text.split(',').map(str::to_owned).collect()
};
crate::dataplane::validate_protocols(&names, max)
.map_err(|err| self.corrupt(err))?;
Ok(names)
})
.transpose()?;
let id: [u8; 32] = id
.as_slice()
.try_into()
.map_err(|_| self.corrupt("stored network id is not 32 bytes"))?;
out.push(StoredNetwork {
protocols,
network_id: NetworkId::from_bytes(id),
name: NetworkName::new(name)?,
secret: NetworkSecret::from_bytes(secret)?,
@@ -732,6 +806,181 @@ impl StateStore {
Ok(version.unwrap_or(0).max(0) as u64)
}
/// Loads durable cleanup markers without allocating oversized stored blobs.
pub fn retirements(&self, network: NetworkId, max: usize) -> Result<Vec<SignedRetirement>> {
let mut stmt = self
.conn
.prepare(
"SELECT CASE WHEN length(data) <= ?2 THEN data ELSE NULL END
FROM retirements WHERE network_id = ?1 ORDER BY target",
)
.map_err(|err| self.corrupt(err))?;
let rows = stmt
.query_map(
params![network.as_bytes().as_slice(), MAX_RETIREMENT_LEN as i64],
|row| row.get::<_, Option<Vec<u8>>>(0),
)
.map_err(|err| self.corrupt(err))?;
let mut out = Vec::new();
for row in rows {
if out.len() >= max {
return Err(Error::Storage(
"stored retirements exceed the configured limit".into(),
));
}
let bytes = row
.map_err(|err| self.corrupt(err))?
.ok_or_else(|| self.corrupt("oversized retirement"))?;
let marker: SignedRetirement =
postcard::from_bytes(&bytes).map_err(|err| self.corrupt(err))?;
marker.verify(network).map_err(|err| self.corrupt(err))?;
out.push(marker);
}
Ok(out)
}
/// Persists a strongest verified marker received from an authenticated session.
pub fn put_retirement(&self, marker: &SignedRetirement) -> Result<()> {
let network = NetworkId::from_bytes(marker.network);
marker
.verify(network)
.map_err(|err| Error::Storage(err.to_string()))?;
let old: Option<Option<Vec<u8>>> = self.conn.query_row(
"SELECT CASE WHEN length(data) <= ?3 THEN data ELSE NULL END FROM retirements WHERE network_id = ?1 AND target = ?2",
params![marker.network.as_slice(), marker.target.author.as_slice(), MAX_RETIREMENT_LEN as i64], |row| row.get(0))
.optional().map_err(|err| self.corrupt(err))?;
if let Some(bytes) = old {
let bytes = bytes.ok_or_else(|| self.corrupt("oversized retirement"))?;
let old: SignedRetirement =
postcard::from_bytes(&bytes).map_err(|err| self.corrupt(err))?;
old.verify(network).map_err(|err| self.corrupt(err))?;
if !marker.supersedes(&old) {
return Ok(());
}
}
let bytes = postcard::to_stdvec(marker).map_err(|err| Error::Storage(err.to_string()))?;
if bytes.len() > MAX_RETIREMENT_LEN {
return Err(Error::Storage(
"retirement exceeds its encoded bound".into(),
));
}
self.conn
.execute(
"INSERT INTO retirements (network_id, target, data) VALUES (?1, ?2, ?3)
ON CONFLICT(network_id, target) DO UPDATE SET data = excluded.data",
params![
marker.network.as_slice(),
marker.target.author.as_slice(),
bytes
],
)
.map_err(|err| Error::Storage(format!("cannot store retirement: {err}")))?;
Ok(())
}
/// Commits our signed cleanup marker and author counter in one transaction.
pub fn publish_own_retirement(&self, marker: &SignedRetirement) -> Result<()> {
let transaction = self
.conn
.unchecked_transaction()
.map_err(|err| Error::Storage(err.to_string()))?;
self.put_retirement(marker)?;
transaction
.execute(
"INSERT INTO own_record_version (network_id, author, version) VALUES (?1, ?2, ?3)
ON CONFLICT(network_id, author) DO UPDATE SET version = max(version, excluded.version)",
params![
marker.network.as_slice(),
marker.author.as_slice(),
marker.version as i64
],
)
.map_err(|err| Error::Storage(format!("cannot store retirement counter: {err}")))?;
transaction
.commit()
.map_err(|err| Error::Storage(format!("cannot commit retirement: {err}")))
}
/// Reads bounded mandatory VIP state; corrupt records fail activation.
pub fn vip_fields(&self, network: NetworkId, max: usize) -> Result<Vec<crate::vip::VipEntry>> {
let mut stmt = self.conn.prepare("SELECT CASE WHEN length(data) <= 1024 THEN data ELSE NULL END FROM vip_fields WHERE network_id=?1 ORDER BY vip, field")
.map_err(|e| self.corrupt(e.to_string()))?;
let mut rows = stmt
.query(params![network.as_bytes().as_slice()])
.map_err(|e| self.corrupt(e.to_string()))?;
let mut entries = Vec::new();
while let Some(row) = rows.next().map_err(|e| self.corrupt(e.to_string()))? {
if entries.len() >= max {
return Err(self.corrupt("stored VIP state exceeds configured limits"));
}
let bytes: Option<Vec<u8>> = row.get(0).map_err(|e| self.corrupt(e.to_string()))?;
let bytes = bytes.ok_or_else(|| self.corrupt("oversized VIP field"))?;
let entry: crate::vip::VipEntry =
postcard::from_bytes(&bytes).map_err(|e| self.corrupt(e.to_string()))?;
entry
.verify(network)
.map_err(|e| self.corrupt(e.to_string()))?;
entries.push(entry);
}
Ok(entries)
}
/// Commits a validated VIP snapshot and optional author counter atomically.
pub fn put_vip_fields(
&self,
network: NetworkId,
entries: &[crate::vip::VipEntry],
own: Option<([u8; 32], u64)>,
) -> Result<()> {
let tx = self
.conn
.unchecked_transaction()
.map_err(|e| Error::Storage(e.to_string()))?;
if let Some(first) = entries.first() {
if entries.iter().any(|e| e.vip != first.vip) {
return Err(Error::Storage(
"VIP snapshot mixes service identities".into(),
));
}
tx.execute(
"DELETE FROM vip_fields WHERE network_id=?1 AND vip=?2",
params![network.as_bytes().as_slice(), first.vip],
)
.map_err(|e| Error::Storage(e.to_string()))?;
}
for e in entries {
e.verify(network)?;
let bytes = postcard::to_stdvec(e).map_err(|e| Error::Storage(e.to_string()))?;
if bytes.len() > 1024 {
return Err(Error::Storage("oversized VIP field".into()));
}
tx.execute("INSERT INTO vip_fields(network_id,vip,field,data) VALUES(?1,?2,?3,?4) ON CONFLICT(network_id,vip,field) DO UPDATE SET data=excluded.data",
params![network.as_bytes().as_slice(),e.vip,e.key,bytes]).map_err(|e|Error::Storage(e.to_string()))?;
}
if let Some((author, version)) = own {
tx.execute("INSERT INTO own_record_version(network_id,author,version) VALUES(?1,?2,?3) ON CONFLICT(network_id,author) DO UPDATE SET version=max(version,excluded.version)",
params![network.as_bytes().as_slice(),author.as_slice(),version as i64]).map_err(|e|Error::Storage(e.to_string()))?;
}
tx.commit().map_err(|e| Error::Storage(e.to_string()))
}
/// Drops a forgotten network's local cleanup proofs.
pub fn forget_retirements(&self, network: NetworkId) -> Result<()> {
self.conn
.execute(
"DELETE FROM vip_fields WHERE network_id = ?1",
params![network.as_bytes().as_slice()],
)
.map_err(|e| Error::Storage(e.to_string()))?;
self.conn
.execute(
"DELETE FROM retirements WHERE network_id = ?1",
params![network.as_bytes().as_slice()],
)
.map_err(|err| Error::Storage(err.to_string()))?;
Ok(())
}
/// Forgets every record of a network.
pub fn forget_signed_records(&self, network_id: NetworkId) -> Result<()> {
self.conn
@@ -798,6 +1047,44 @@ impl StateStore {
Ok(())
}
/// Atomically enables a network and puts it first in the startup order.
pub fn prefer_network(&self, network: NetworkId) -> Result<()> {
let configured = self.list_networks()?;
if !configured.iter().any(|n| n.network_id == network) {
return Err(Error::NetworkUnknown(network));
}
let existing = self.get_setting("network_start_order")?.unwrap_or_default();
let mut order = vec![network];
for word in existing.split(',') {
if let Ok(id) = word.parse::<NetworkId>()
&& configured.iter().any(|n| n.network_id == id)
&& !order.contains(&id)
{
order.push(id);
}
}
let value = order
.iter()
.map(ToString::to_string)
.collect::<Vec<_>>()
.join(",");
let transaction = self
.conn
.unchecked_transaction()
.map_err(|err| Error::Storage(format!("cannot begin network preference: {err}")))?;
transaction
.execute(
"UPDATE networks SET auto_start = 1 WHERE network_id = ?1",
params![network.as_bytes().as_slice()],
)
.map_err(|err| Error::Storage(format!("cannot enable network: {err}")))?;
transaction.execute("INSERT INTO settings (key, value) VALUES ('network_start_order', ?1) ON CONFLICT(key) DO UPDATE SET value = excluded.value", params![value])
.map_err(|err| Error::Storage(format!("cannot store network preference: {err}")))?;
transaction
.commit()
.map_err(|err| Error::Storage(format!("cannot commit network preference: {err}")))
}
/// Reads an arbitrary setting.
pub fn get_setting(&self, key: &str) -> Result<Option<String>> {
self.conn
@@ -915,6 +1202,158 @@ mod broadcast_storage_tests {
assert!(store.peer_hostnames(id).unwrap().is_empty());
}
#[test]
fn retirement_migration_and_atomic_counter_commit_preserve_existing_state() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("state.sqlite");
let store = StateStore::open(&path).unwrap();
let identity = store.load_or_create_device_identity().unwrap();
let name = NetworkName::new("retirement-migration").unwrap();
let secret = NetworkSecret::from_bytes([6; 32]).unwrap();
let id = NetworkKeys::derive(&name, &secret).network_id();
store.upsert_network(id, &name, &secret, true).unwrap();
store.set_protocols(id, &["awg".into()]).unwrap();
let own = SignedRecord::sign(&identity.signing_key(), id, 8, RecordBody::Release);
store.publish_own_record(&own).unwrap();
store
.conn
.execute_batch("DROP TABLE retirements; PRAGMA user_version = 8;")
.unwrap();
drop(store);
let store = StateStore::open(&path).unwrap();
assert_eq!(
store.device_identity().unwrap().unwrap().endpoint_id(),
identity.endpoint_id()
);
assert_eq!(
store.list_networks().unwrap()[0].protocols,
Some(vec!["awg".into()])
);
assert_eq!(store.signed_records(id).unwrap(), vec![own]);
let target_key = iroh::SecretKey::generate();
let target = SignedRecord::sign(
&target_key,
id,
3,
RecordBody::Claim {
address: Some("10.13.37.2".parse().unwrap()),
range: Some(crate::state::DEFAULT_IPV4_RANGE),
hostname: Some("old-device".into()),
hostname_since: Some(123),
},
);
let marker = SignedRetirement::sign(&identity.signing_key(), id, 9, target);
store
.conn
.execute_batch(
"CREATE TRIGGER fail_retirement_counter BEFORE INSERT ON own_record_version
WHEN NEW.version = 9 BEGIN SELECT RAISE(ABORT, 'counter write refused'); END;",
)
.unwrap();
assert!(store.publish_own_retirement(&marker).is_err());
assert!(
store.retirements(id, 16).unwrap().is_empty(),
"marker must roll back with counter failure"
);
assert_eq!(
store
.own_record_version(id, identity.endpoint_id())
.unwrap(),
8
);
store
.conn
.execute_batch("DROP TRIGGER fail_retirement_counter;")
.unwrap();
store.publish_own_retirement(&marker).unwrap();
drop(store);
let store = StateStore::open(&path).unwrap();
assert_eq!(store.retirements(id, 16).unwrap(), vec![marker.clone()]);
assert_eq!(
store
.own_record_version(id, identity.endpoint_id())
.unwrap(),
9
);
assert!(
store.retirements(id, 0).is_err(),
"capacity changes must not silently drop markers"
);
store
.conn
.execute(
"UPDATE retirements SET data = ?1",
params![vec![0u8; MAX_RETIREMENT_LEN + 1]],
)
.unwrap();
assert!(matches!(
store.retirements(id, 16),
Err(Error::StateCorrupted { .. })
));
}
#[test]
fn v7_migration_keeps_signed_state_and_protocol_overrides_are_local_and_durable() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("state.sqlite");
let store = StateStore::open(&path).unwrap();
let identity = store.load_or_create_device_identity().unwrap();
let name = NetworkName::new("protocol-migration").unwrap();
let secret = NetworkSecret::from_bytes([4; 32]).unwrap();
let id = NetworkKeys::derive(&name, &secret).network_id();
store.upsert_network(id, &name, &secret, true).unwrap();
store.set_broadcast(id, false).unwrap();
let record = SignedRecord::sign(&identity.signing_key(), id, 17, RecordBody::Release);
store.publish_own_record(&record).unwrap();
store
.conn
.execute_batch("ALTER TABLE networks DROP COLUMN protocols; PRAGMA user_version=7;")
.unwrap();
drop(store);
let store = StateStore::open(&path).unwrap();
assert_eq!(
store
.load_or_create_device_identity()
.unwrap()
.endpoint_id(),
identity.endpoint_id()
);
assert_eq!(store.signed_records(id).unwrap(), vec![record]);
let saved = store.list_networks().unwrap();
assert!(saved[0].protocols.is_none());
assert!(!saved[0].broadcast);
assert!(saved[0].auto_start);
assert_eq!(saved[0].secret, secret);
let chosen = vec!["awg".into(), "wg-quic".into()];
store.set_protocols(id, &chosen).unwrap();
assert!(
store
.set_protocols(id, &["awg".into(), "awg".into()])
.is_err()
);
assert!(
store
.set_protocols(NetworkId::from_bytes([1; 32]), &[])
.is_err()
);
drop(store);
let store = StateStore::open(&path).unwrap();
assert_eq!(store.list_networks().unwrap()[0].protocols, Some(chosen));
store.upsert_network(id, &name, &secret, true).unwrap();
store.set_protocols(id, &[]).unwrap();
drop(store);
let store = StateStore::open(&path).unwrap();
assert_eq!(store.list_networks().unwrap()[0].protocols, Some(vec![]));
store
.conn
.execute("UPDATE networks SET protocols = 'awg,awg'", [])
.unwrap();
assert!(matches!(
store.list_networks(),
Err(Error::StateCorrupted { .. })
));
}
#[test]
fn v3_migration_preserves_identity_and_networks_and_defaults_broadcast_on() {
let dir = tempfile::tempdir().unwrap();
@@ -935,6 +1374,7 @@ mod broadcast_storage_tests {
"ALTER TABLE networks DROP COLUMN broadcast; \
ALTER TABLE networks DROP COLUMN exit_node; \
ALTER TABLE networks DROP COLUMN exit_via; \
ALTER TABLE networks DROP COLUMN protocols; \
PRAGMA user_version=3;",
)
.unwrap();
@@ -969,3 +1409,59 @@ mod broadcast_storage_tests {
);
}
}
#[cfg(test)]
mod vip_tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
#[test]
fn vip_migration_and_counter_failure_preserve_identity_and_rollback_fields() {
let dir = tempfile::TempDir::new().unwrap();
let path = dir.path().join("state.sqlite");
let store = StateStore::open(&path).unwrap();
let identity = store.load_or_create_device_identity().unwrap();
store
.conn
.execute_batch("DROP TABLE vip_fields; PRAGMA user_version=9;")
.unwrap();
drop(store);
let store = StateStore::open(&path).unwrap();
assert_eq!(
store.device_identity().unwrap().unwrap().endpoint_id(),
identity.endpoint_id()
);
let net = NetworkId::from_bytes([7; 32]);
let key = identity.signing_key();
let config = crate::vip::VipConfig::new("api");
let fields = crate::vip::VipSet::default()
.edit(net, &key, 7, &config, None, &crate::Limits::default())
.unwrap();
store.conn.execute_batch("CREATE TRIGGER fail_vip_counter BEFORE INSERT ON own_record_version BEGIN SELECT RAISE(ABORT,'counter unavailable'); END;").unwrap();
assert!(
store
.put_vip_fields(net, &fields, Some((*key.public().as_bytes(), 7)))
.is_err()
);
assert!(store.vip_fields(net, 192).unwrap().is_empty());
store
.conn
.execute_batch("DROP TRIGGER fail_vip_counter;")
.unwrap();
store
.put_vip_fields(net, &fields, Some((*key.public().as_bytes(), 7)))
.unwrap();
assert_eq!(store.vip_fields(net, 192).unwrap(), fields);
assert_eq!(store.own_record_version(net, key.public()).unwrap(), 7);
drop(store);
let reopened = StateStore::open(path).unwrap();
assert_eq!(reopened.vip_fields(net, 192).unwrap(), fields);
reopened
.conn
.execute("UPDATE vip_fields SET data=?1", params![vec![0xffu8; 1025]])
.unwrap();
assert!(matches!(
reopened.vip_fields(net, 192),
Err(Error::StateCorrupted { .. })
));
}
}
+239
View File
@@ -0,0 +1,239 @@
//! Named, shared virtual services. Signed field registers converge without an
//! owner; routing, health and connection state remain local to each client.
use crate::Limits;
use crate::error::{Error, Result};
use iroh::EndpointId;
use serde::{Deserialize, Serialize};
use std::collections::BTreeSet;
use std::net::Ipv4Addr;
pub(crate) mod packet;
pub(crate) mod router;
pub use crate::state::vip::{VipEntry, VipSet};
pub(crate) use router::VipRouter;
#[cfg(test)]
mod tests;
/// IP transport carried by a listener.
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum Protocol {
/// TCP.
Tcp,
/// UDP.
Udp,
}
impl std::fmt::Display for Protocol {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str(match self {
Self::Tcp => "tcp",
Self::Udp => "udp",
})
}
}
impl std::str::FromStr for Protocol {
type Err = String;
fn from_str(s: &str) -> std::result::Result<Self, String> {
match s {
"tcp" => Ok(Self::Tcp),
"udp" => Ok(Self::Udp),
_ => Err("protocol must be tcp or udp".into()),
}
}
}
/// A node identity and its application port.
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct Backend {
/// Full device identity, resolved when editing.
pub peer: String,
/// Destination port.
pub port: u16,
}
/// One independently routed port of a VIP.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct Listener {
/// Stable listener generation; prevents removed backend fields resurfacing on re-creation.
pub id: String,
/// TCP or UDP.
pub protocol: Protocol,
/// Client-facing port.
pub port: u16,
/// Enable active TCP connect checks; UDP uses path checks and ICMP errors.
pub port_check: bool,
/// Period between checks in milliseconds.
pub check_interval_ms: u32,
/// Deadline of a port check in milliseconds.
pub check_timeout_ms: u32,
/// Consecutive failed checks before exclusion.
pub failure_threshold: u8,
/// Consecutive successes before recovery.
pub recovery_threshold: u8,
/// Idle expiry for this listener's flows, in seconds.
pub idle_timeout_secs: u32,
/// Allowed destination nodes and ports.
pub backends: Vec<Backend>,
}
impl Listener {
/// Creates a listener with automatic recovery and bounded checks.
pub fn new(protocol: Protocol, port: u16) -> Self {
Self {
id: hex::encode(rand::random::<[u8; 16]>()),
protocol,
port,
port_check: true,
check_interval_ms: 3000,
check_timeout_ms: 2000,
failure_threshold: 2,
recovery_threshold: 1,
idle_timeout_secs: if protocol == Protocol::Tcp { 7200 } else { 120 },
backends: vec![],
}
}
}
/// Entire editable service; its address is an internal derived reservation.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct VipConfig {
/// Stable random 128-bit id in hexadecimal.
pub id: String,
/// One DNS label, unique within the network.
pub name: String,
/// Independent listeners.
pub listeners: Vec<Listener>,
}
impl VipConfig {
/// Creates an empty named service with a fresh stable identity.
pub fn new(name: impl Into<String>) -> Self {
Self {
id: hex::encode(rand::random::<[u8; 16]>()),
name: name.into(),
listeners: vec![],
}
}
/// Validates a complete draft before any state is changed.
pub fn validate(&self, limits: &Limits) -> Result<()> {
if self.id.len() != 32 || hex::decode(&self.id).is_err() {
return Err(invalid("invalid VIP id"));
}
if self.name.is_empty()
|| self.name.len() > 63
|| self.name.starts_with('-')
|| self.name.ends_with('-')
|| !self
.name
.bytes()
.all(|b| b.is_ascii_lowercase() || b.is_ascii_digit() || b == b'-')
{
return Err(invalid(
"VIP name must be a lowercase DNS label (letters, digits and hyphens)",
));
}
if self.listeners.len() > limits.max_vip_listeners {
return Err(invalid("too many VIP listeners"));
}
let mut seen = BTreeSet::new();
for l in &self.listeners {
if l.id.len() != 32 || hex::decode(&l.id).is_err() {
return Err(invalid("invalid listener identity"));
}
if l.port == 0 || !seen.insert((l.protocol, l.port)) {
return Err(invalid(
"listener ports must be nonzero and unique per protocol",
));
}
if !(250..=3_600_000).contains(&l.check_interval_ms)
|| !(100..=60_000).contains(&l.check_timeout_ms)
|| l.failure_threshold == 0
|| l.recovery_threshold == 0
|| !(1..=604800).contains(&l.idle_timeout_secs)
{
return Err(invalid(
"invalid check interval, timeout, thresholds or flow idle timeout",
));
}
if l.backends.len() > limits.max_vip_backends {
return Err(invalid("too many listener backends"));
}
let mut backends = BTreeSet::new();
for b in &l.backends {
if b.port == 0 || b.peer.parse::<EndpointId>().is_err() || !backends.insert(b) {
return Err(invalid(
"backend must have a unique node identity and a nonzero port",
));
}
}
}
if crate::state::vip::flatten(self)?.len() > limits.max_vip_fields {
return Err(invalid("VIP configuration exceeds signed field capacity"));
}
Ok(())
}
}
pub(crate) fn invalid(reason: impl Into<String>) -> Error {
Error::Rejected {
reason: reason.into(),
}
}
/// Cumulative IP traffic, excluding probes. Reset on agent restart.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct Counters {
/// Outbound packets.
pub tx_packets: u64,
/// Outbound bytes.
pub tx_bytes: u64,
/// Inbound packets.
pub rx_packets: u64,
/// Inbound bytes.
pub rx_bytes: u64,
/// Created flow bindings.
pub flows: u64,
/// UDP failovers.
pub failovers: u64,
/// Packets rejected or lacking an eligible backend.
pub dropped: u64,
}
/// One backend, as seen by this client.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct BackendStatus {
/// Configured endpoint.
pub backend: Backend,
/// Eligibility and reason.
pub state: String,
/// Smoothed full data-plane RTT, microseconds; unknown stays absent.
pub rtt_us: Option<u64>,
/// Last successful path measurement age.
pub measurement_age_ms: Option<u64>,
/// Current bindings.
pub active_flows: u32,
/// Local traffic counters.
pub counters: Counters,
}
/// Runtime view of one listener.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct ListenerStatus {
/// Protocol.
pub protocol: Protocol,
/// Frontend port.
pub port: u16,
/// Best backend for a new flow, if measured and eligible.
pub selected: Option<Backend>,
/// Configured and draining endpoints.
pub backends: Vec<BackendStatus>,
/// Local counters.
pub counters: Counters,
}
/// Configuration and local diagnostics returned through the same API to CLI and GUI.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct VipStatus {
/// Persisted configuration.
pub config: VipConfig,
/// Opaque revision used when saving a draft.
pub revision: String,
/// Internal service address.
pub address: Option<Ipv4Addr>,
/// Conflict or allocation issue, if any.
pub error: Option<String>,
/// Local listener status.
pub listeners: Vec<ListenerStatus>,
}
+566
View File
@@ -0,0 +1,566 @@
//! Bounded IPv4 parsing, NAT checksums and fragment reassembly at local ingress.
use super::Protocol;
use bytes::Bytes;
use std::collections::HashMap;
use std::net::Ipv4Addr;
use std::time::{Duration, Instant};
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub(crate) struct Tuple {
pub src: Ipv4Addr,
pub dst: Ipv4Addr,
pub sport: u16,
pub dport: u16,
pub protocol: Protocol,
}
impl Tuple {
pub fn reverse(self) -> Self {
Self {
src: self.dst,
dst: self.src,
sport: self.dport,
dport: self.sport,
protocol: self.protocol,
}
}
}
pub(crate) fn header(p: &[u8]) -> Option<usize> {
if p.len() < 20 || p[0] >> 4 != 4 {
return None;
}
let n = usize::from(p[0] & 15) * 4;
let total = usize::from(u16::from_be_bytes([p[2], p[3]]));
(n >= 20 && n <= p.len() && total == p.len() && checksum(&p[..n]) == 0).then_some(n)
}
pub(crate) fn addresses(p: &[u8]) -> Option<(Ipv4Addr, Ipv4Addr)> {
header(p)?;
Some((
Ipv4Addr::new(p[12], p[13], p[14], p[15]),
Ipv4Addr::new(p[16], p[17], p[18], p[19]),
))
}
pub(crate) fn tuple(p: &[u8]) -> Option<Tuple> {
let n = header(p)?;
if u16::from_be_bytes([p[6], p[7]]) & 0x3fff != 0 {
return None;
}
let protocol = match p[9] {
6 => Protocol::Tcp,
17 => Protocol::Udp,
_ => return None,
};
let min = if protocol == Protocol::Tcp { 20 } else { 8 };
if p.len() < n + min {
return None;
}
if protocol == Protocol::Tcp {
let h = usize::from(p[n + 12] >> 4) * 4;
if h < 20 || n + h > p.len() {
return None;
}
} else if usize::from(u16::from_be_bytes([p[n + 4], p[n + 5]])) != p.len() - n {
return None;
}
let (src, dst) = addresses(p)?;
Some(Tuple {
src,
dst,
sport: u16::from_be_bytes([p[n], p[n + 1]]),
dport: u16::from_be_bytes([p[n + 2], p[n + 3]]),
protocol,
})
}
pub(crate) fn rewrite(p: &[u8], t: Tuple) -> Option<Bytes> {
let n = header(p)?;
if p.len() < n + if t.protocol == Protocol::Tcp { 20 } else { 8 } {
return None;
}
let mut out = p.to_vec();
out[12..16].copy_from_slice(&t.src.octets());
out[16..20].copy_from_slice(&t.dst.octets());
out[n..n + 2].copy_from_slice(&t.sport.to_be_bytes());
out[n + 2..n + 4].copy_from_slice(&t.dport.to_be_bytes());
finish(&mut out, n, t.protocol);
Some(Bytes::from(out))
}
fn finish(p: &mut [u8], n: usize, protocol: Protocol) {
p[10..12].fill(0);
let c = checksum(&p[..n]);
p[10..12].copy_from_slice(&c.to_be_bytes());
let offset = n + if protocol == Protocol::Tcp { 16 } else { 6 };
p[offset..offset + 2].fill(0);
let mut sum = 0u32;
for pair in p[12..20].as_chunks::<2>().0 {
sum += u32::from(u16::from_be_bytes([pair[0], pair[1]]));
}
sum += u32::from(p[9]) + (p.len() - n) as u32;
let c = checksum_seed(&p[n..], sum);
let c = if c == 0 && protocol == Protocol::Udp {
0xffff
} else {
c
};
p[offset..offset + 2].copy_from_slice(&c.to_be_bytes());
}
pub(crate) fn checksum(p: &[u8]) -> u16 {
checksum_seed(p, 0)
}
fn checksum_seed(p: &[u8], mut sum: u32) -> u16 {
for pair in p.chunks(2) {
sum += u32::from(u16::from_be_bytes([pair[0], *pair.get(1).unwrap_or(&0)]));
}
while sum >> 16 != 0 {
sum = (sum & 0xffff) + (sum >> 16);
}
!(sum as u16)
}
pub(crate) fn tcp_syn(p: &[u8]) -> bool {
header(p).is_some_and(|n| p[9] == 6 && p.get(n + 13).is_some_and(|f| f & 0x12 == 2))
}
pub(crate) fn syn_sequence(p: &[u8]) -> Option<u32> {
if !tcp_syn(p) {
return None;
}
let n = header(p)?;
let bytes = p.get(n + 4..n + 8)?;
Some(u32::from_be_bytes(bytes.try_into().ok()?))
}
pub(crate) fn empty_syn(p: &[u8]) -> bool {
header(p).is_some_and(|n| {
tcp_syn(p)
&& p.get(n + 12)
.is_some_and(|h| n + usize::from(h >> 4) * 4 == p.len())
})
}
pub(crate) fn syn_ack(p: &[u8]) -> bool {
header(p).is_some_and(|n| p[9] == 6 && p.get(n + 13).is_some_and(|f| f & 0x12 == 0x12))
}
pub(crate) fn tcp_reset(p: &[u8]) -> bool {
header(p).is_some_and(|n| p[9] == 6 && p.get(n + 13).is_some_and(|f| f & 4 != 0))
}
/// Rewrites an ICMP error and its quoted original header after NAT lookup.
pub(crate) fn icmp_quote(p: &[u8]) -> Option<Tuple> {
let n = header(p)?;
if p[9] != 1 || p.len() < n + 8 + 28 || ![3, 11, 12].contains(&p[n]) {
return None;
}
let q = &p[n + 8..];
let h = usize::from(q[0] & 15) * 4;
if q[0] >> 4 != 4 || h < 20 || q.len() < h + 8 {
return None;
}
let protocol = match q[9] {
6 => Protocol::Tcp,
17 => Protocol::Udp,
_ => return None,
};
Some(Tuple {
src: Ipv4Addr::new(q[12], q[13], q[14], q[15]),
dst: Ipv4Addr::new(q[16], q[17], q[18], q[19]),
sport: u16::from_be_bytes([q[h], q[h + 1]]),
dport: u16::from_be_bytes([q[h + 2], q[h + 3]]),
protocol,
})
}
pub(crate) fn rewrite_icmp(p: &[u8], original: Tuple) -> Option<Bytes> {
let n = header(p)?;
icmp_quote(p)?;
let mut out = p.to_vec();
out[12..16].copy_from_slice(&original.dst.octets());
out[16..20].copy_from_slice(&original.src.octets());
let q = &mut out[n + 8..];
let h = usize::from(q[0] & 15) * 4;
// Incrementally update the quoted transport checksum when it is present.
let off = h + if original.protocol == Protocol::Tcp {
16
} else {
6
};
if q.len() >= off + 2 {
let old = u16::from_be_bytes([q[off], q[off + 1]]);
if old != 0 || original.protocol == Protocol::Tcp {
let mut sum = u32::from(!old);
let mut replacement = Vec::from(original.src.octets());
replacement.extend(original.dst.octets());
replacement.extend(original.sport.to_be_bytes());
replacement.extend(original.dport.to_be_bytes());
let previous = [&q[12..20], &q[h..h + 4]].concat();
for (a, b) in previous
.as_chunks::<2>()
.0
.iter()
.zip(replacement.as_chunks::<2>().0)
{
sum += u32::from(!u16::from_be_bytes([a[0], a[1]]))
+ u32::from(u16::from_be_bytes([b[0], b[1]]));
}
while sum >> 16 != 0 {
sum = (sum & 0xffff) + (sum >> 16);
}
let c = !(sum as u16);
q[off..off + 2].copy_from_slice(
&(if c == 0 && original.protocol == Protocol::Udp {
0xffff
} else {
c
})
.to_be_bytes(),
);
}
}
q[12..16].copy_from_slice(&original.src.octets());
q[16..20].copy_from_slice(&original.dst.octets());
q[h..h + 2].copy_from_slice(&original.sport.to_be_bytes());
q[h + 2..h + 4].copy_from_slice(&original.dport.to_be_bytes());
q[10..12].fill(0);
let c = checksum(&q[..h]);
q[10..12].copy_from_slice(&c.to_be_bytes());
out[n + 2..n + 4].fill(0);
let c = checksum(&out[n..]);
out[n + 2..n + 4].copy_from_slice(&c.to_be_bytes());
out[10..12].fill(0);
let c = checksum(&out[..n]);
out[10..12].copy_from_slice(&c.to_be_bytes());
Some(Bytes::from(out))
}
pub(crate) fn reject(p: &[u8]) -> Option<Bytes> {
let t = tuple(p)?;
let n = header(p)?;
if t.protocol == Protocol::Tcp {
if !tcp_syn(p) {
return None;
}
let mut out = vec![0; 40];
out[0] = 0x45;
out[2..4].copy_from_slice(&40u16.to_be_bytes());
out[8] = 64;
out[9] = 6;
out[12..16].copy_from_slice(&t.dst.octets());
out[16..20].copy_from_slice(&t.src.octets());
out[20..22].copy_from_slice(&t.dport.to_be_bytes());
out[22..24].copy_from_slice(&t.sport.to_be_bytes());
let seq = u32::from_be_bytes(p[n + 4..n + 8].try_into().ok()?);
out[28..32].copy_from_slice(&seq.wrapping_add(1).to_be_bytes());
out[32] = 0x50;
out[33] = 0x14;
finish(&mut out, 20, Protocol::Tcp);
Some(Bytes::from(out))
} else {
let q = p.len().min(n + 8);
let mut out = vec![0; 28 + q];
out[0] = 0x45;
let len = out.len() as u16;
out[2..4].copy_from_slice(&len.to_be_bytes());
out[8] = 64;
out[9] = 1;
out[12..16].copy_from_slice(&t.dst.octets());
out[16..20].copy_from_slice(&t.src.octets());
out[20] = 3;
out[21] = 3;
out[28..].copy_from_slice(&p[..q]);
let c = checksum(&out[20..]);
out[22..24].copy_from_slice(&c.to_be_bytes());
let c = checksum(&out[..20]);
out[10..12].copy_from_slice(&c.to_be_bytes());
Some(Bytes::from(out))
}
}
#[derive(Debug)]
struct Datagram {
at: Instant,
header: Option<Vec<u8>>,
parts: std::collections::BTreeMap<usize, Vec<u8>>,
end: Option<usize>,
bytes: usize,
}
type FragmentKey = (Option<[u8; 32]>, Ipv4Addr, Ipv4Addr, u16, u8);
#[derive(Debug, Default)]
pub(crate) struct Reassembly {
pending: HashMap<FragmentKey, Datagram>,
}
impl Reassembly {
pub fn expire(&mut self, now: Instant) {
self.pending
.retain(|_, d| now.duration_since(d.at) < Duration::from_secs(5));
}
pub fn accept(&mut self, p: Bytes, max: usize, scope: Option<[u8; 32]>) -> Option<Bytes> {
let n = header(&p)?;
let flags = u16::from_be_bytes([p[6], p[7]]);
if flags & 0x3fff == 0 {
return Some(p);
}
let now = Instant::now();
self.pending
.retain(|_, d| now.duration_since(d.at) < Duration::from_secs(5));
let (src, dst) = addresses(&p)?;
let key = (scope, src, dst, u16::from_be_bytes([p[4], p[5]]), p[9]);
let start = usize::from(flags & 0x1fff) * 8;
let payload = &p[n..];
let end = start.checked_add(payload.len())?;
if payload.is_empty()
|| end > 65515
|| flags & 0x2000 != 0 && !payload.len().is_multiple_of(8)
{
self.pending.remove(&key);
return None;
}
if !self.pending.contains_key(&key) && self.pending.len() >= max {
return None;
}
let d = self.pending.entry(key).or_insert_with(|| Datagram {
at: now,
header: None,
parts: Default::default(),
end: None,
bytes: 0,
});
if d.parts
.get(&start)
.is_some_and(|bytes| bytes.as_slice() == payload)
{
return None;
}
if d.parts.iter().any(|(s, b)| *s < end && start < s + b.len()) {
self.pending.remove(&key);
return None;
}
if flags & 0x2000 == 0 {
if d.end.is_some_and(|e| e != end) {
self.pending.remove(&key);
return None;
}
d.end = Some(end);
}
if d.end.is_some_and(|e| end > e) {
self.pending.remove(&key);
return None;
}
if start == 0 {
d.header = Some(p[..n].to_vec());
}
d.bytes += payload.len();
d.parts.insert(start, payload.to_vec());
let complete = d.end.is_some_and(|e| e == d.bytes) && d.header.is_some();
if !complete {
return None;
}
let d = self.pending.remove(&key)?;
let mut out = d.header?;
for part in d.parts.values() {
out.extend(part);
}
if out.len() > 65535 {
return None;
}
let len = out.len() as u16;
out[2..4].copy_from_slice(&len.to_be_bytes());
out[6..8].fill(0);
out[10..12].fill(0);
let h = usize::from(out[0] & 15) * 4;
let c = checksum(&out[..h]);
out[10..12].copy_from_slice(&c.to_be_bytes());
Some(Bytes::from(out))
}
}
/// Re-fragments an assembled NAT datagram for the unchanged overlay MTU.
pub(crate) fn fragments(packet: Bytes, mtu: u32) -> Vec<Bytes> {
if packet.len() <= mtu as usize {
return vec![packet];
}
let Some(n) = header(&packet) else {
return vec![];
};
if u16::from_be_bytes([packet[6], packet[7]]) & 0x4000 != 0 {
return vec![];
}
let size = (mtu as usize).saturating_sub(n) / 8 * 8;
if size == 0 {
return vec![];
}
let mut copied = Vec::new();
let mut offset = 20;
while offset < n {
let kind = packet[offset];
if kind == 0 {
break;
}
if kind == 1 {
offset += 1;
continue;
}
let Some(&len) = packet.get(offset + 1) else {
return vec![];
};
let len = usize::from(len);
if len < 2 || offset + len > n {
return vec![];
}
if kind & 0x80 != 0 {
copied.extend_from_slice(&packet[offset..offset + len]);
}
offset += len;
}
while !copied.len().is_multiple_of(4) {
copied.push(0);
}
let mut result = Vec::new();
let payload = &packet[n..];
for (start, part) in payload.chunks(size).enumerate() {
let offset = start * size;
let mut out = if offset == 0 {
packet[..n].to_vec()
} else {
let mut h = packet[..20].to_vec();
h.extend_from_slice(&copied);
h[0] = 0x40 | (h.len() / 4) as u8;
h
};
let header_len = out.len();
out.extend(part);
let len = out.len() as u16;
out[2..4].copy_from_slice(&len.to_be_bytes());
let flags = (offset / 8) as u16
| if offset + part.len() < payload.len() {
0x2000
} else {
0
};
out[6..8].copy_from_slice(&flags.to_be_bytes());
out[10..12].fill(0);
let c = checksum(&out[..header_len]);
out[10..12].copy_from_slice(&c.to_be_bytes());
result.push(Bytes::from(out));
}
result
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
fn datagram() -> Bytes {
let mut p = vec![0; 3028];
p[0] = 0x45;
p[2..4].copy_from_slice(&3028u16.to_be_bytes());
p[8] = 64;
p[9] = 17;
p[12..16].copy_from_slice(&[10, 0, 0, 1]);
p[16..20].copy_from_slice(&[10, 0, 0, 2]);
p[20..22].copy_from_slice(&1000u16.to_be_bytes());
p[22..24].copy_from_slice(&2000u16.to_be_bytes());
p[24..26].copy_from_slice(&3008u16.to_be_bytes());
finish(&mut p, 20, Protocol::Udp);
Bytes::from(p)
}
#[test]
fn fragmented_udp_reassembles_out_of_order_and_nat_checksums_are_valid() {
let p = datagram();
let parts = fragments(p.clone(), 1280);
assert_eq!(parts.len(), 3);
let mut reassembly = Reassembly::default();
let mut result = None;
for part in parts.into_iter().rev() {
if let Some(p) = reassembly.accept(part, 8, None) {
result = Some(p);
}
}
assert_eq!(result, Some(p.clone()));
let mut t = tuple(&p).unwrap();
t.dst = Ipv4Addr::new(10, 0, 0, 240);
t.dport = 53;
let out = rewrite(&p, t).unwrap();
assert_eq!(header(&out), Some(20));
let mut sum = 0u32;
for pair in out[12..20].as_chunks::<2>().0 {
sum += u32::from(u16::from_be_bytes([pair[0], pair[1]]));
}
sum += 17 + (out.len() - 20) as u32;
assert_eq!(checksum_seed(&out[20..], sum), 0);
}
#[test]
fn overlaps_and_capacity_are_rejected_without_allocating_unbounded_state() {
let parts = fragments(datagram(), 1280);
let mut r = Reassembly::default();
assert!(r.accept(parts[0].clone(), 0, None).is_none());
assert!(r.pending.is_empty());
assert!(r.accept(parts[0].clone(), 1, None).is_none());
assert!(r.accept(parts[0].clone(), 1, None).is_none());
assert_eq!(r.pending.len(), 1);
let mut conflict = parts[0].to_vec();
conflict[20] ^= 1;
assert!(r.accept(Bytes::from(conflict), 1, None).is_none());
assert!(r.pending.is_empty());
for size in 0..80 {
let p = vec![0xff; size];
assert!(tuple(&p).is_none());
assert!(icmp_quote(&p).is_none());
}
}
#[test]
fn fragment_assembly_is_scoped_to_authenticated_sender_and_expires_when_idle() {
let packet = datagram();
let parts = fragments(packet.clone(), 1280);
let mut r = Reassembly::default();
assert!(r.accept(parts[0].clone(), 8, Some([1; 32])).is_none());
assert!(r.accept(parts[1].clone(), 8, Some([2; 32])).is_none());
assert!(r.accept(parts[2].clone(), 8, Some([1; 32])).is_none());
assert_eq!(r.accept(parts[1].clone(), 8, Some([1; 32])), Some(packet));
assert_eq!(r.pending.len(), 1);
r.expire(Instant::now() + Duration::from_secs(6));
assert!(r.pending.is_empty());
}
}
#[cfg(test)]
mod icmp_tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
#[test]
fn pmtu_errors_restore_quoted_addresses_ports_and_checksums() {
let original = Tuple {
src: Ipv4Addr::new(10, 0, 0, 1),
dst: Ipv4Addr::new(10, 0, 0, 240),
sport: 40000,
dport: 53,
protocol: Protocol::Udp,
};
let mut p = vec![0; 36];
p[0] = 0x45;
p[2..4].copy_from_slice(&36u16.to_be_bytes());
p[8] = 64;
p[9] = 17;
p[24..26].copy_from_slice(&16u16.to_be_bytes());
let c = checksum(&p[..20]);
p[10..12].copy_from_slice(&c.to_be_bytes());
let p = rewrite(&p, original).unwrap();
let wire = Tuple {
dst: Ipv4Addr::new(10, 0, 0, 2),
sport: 50000,
dport: 5353,
..original
};
let translated = rewrite(&p, wire).unwrap();
let mut error = vec![0; 56];
error[0] = 0x45;
error[2..4].copy_from_slice(&56u16.to_be_bytes());
error[8] = 64;
error[9] = 1;
error[12..16].copy_from_slice(&wire.dst.octets());
error[16..20].copy_from_slice(&wire.src.octets());
error[20] = 3;
error[21] = 4;
error[26..28].copy_from_slice(&1200u16.to_be_bytes());
error[28..].copy_from_slice(&translated[..28]);
let c = checksum(&error[20..]);
error[22..24].copy_from_slice(&c.to_be_bytes());
let c = checksum(&error[..20]);
error[10..12].copy_from_slice(&c.to_be_bytes());
let restored = rewrite_icmp(&error, original).unwrap();
assert_eq!(icmp_quote(&restored), Some(original));
assert_eq!(checksum(&restored[20..]), 0);
assert_eq!(checksum(&restored[28..48]), 0);
assert_eq!(&restored[54..56], &p[26..28]);
assert_eq!(&restored[26..28], &1200u16.to_be_bytes());
assert_eq!(&restored[12..16], &original.dst.octets());
}
}
File diff suppressed because it is too large. Load diff
+144
View File
@@ -0,0 +1,144 @@
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
use crate::NetworkId;
#[test]
fn concurrent_backend_edits_merge_and_deletion_cannot_resurrect() {
let net = NetworkId::from_bytes([7; 32]);
let a = iroh::SecretKey::generate();
let b = iroh::SecretKey::generate();
let limits = Limits::default();
let mut state = VipSet::default();
let id = hex::encode([9; 16]);
let initial = VipConfig {
id: id.clone(),
name: "api".into(),
listeners: vec![Listener::new(Protocol::Tcp, 443)],
};
let changes = state.edit(net, &a, 1, &initial, None, &limits).unwrap();
state.merge(net, changes, &limits).unwrap();
let revision = state.revision(&id);
let mut first = initial.clone();
first.listeners[0].backends.push(Backend {
peer: a.public().to_string(),
port: 6443,
});
let mut second = initial.clone();
second.listeners[0].backends.push(Backend {
peer: b.public().to_string(),
port: 7443,
});
let x = state
.edit(net, &a, 2, &first, Some(&revision), &limits)
.unwrap();
let y = state
.edit(net, &b, 1, &second, Some(&revision), &limits)
.unwrap();
let mut reverse = state.clone();
state.merge(net, x.clone(), &limits).unwrap();
state.merge(net, y.clone(), &limits).unwrap();
reverse.merge(net, y, &limits).unwrap();
reverse.merge(net, x, &limits).unwrap();
assert_eq!(state.configs(), reverse.configs());
assert_eq!(state.configs()[0].listeners[0].backends.len(), 2);
assert!(
state
.edit(net, &a, 3, &initial, Some(&revision), &limits)
.is_err()
);
let removed = state.delete(net, &a, 3, &id, &state.revision(&id)).unwrap();
let old = state.entries(&id);
state.merge(net, removed, &limits).unwrap();
state.merge(net, old, &limits).unwrap();
assert!(state.configs().is_empty());
}
#[test]
fn signed_fields_reject_tampering_wrong_network_and_limits() {
let net = NetworkId::from_bytes([2; 32]);
let key = iroh::SecretKey::generate();
let limits = Limits::default();
let state = VipSet::default();
let cfg = VipConfig {
id: hex::encode([3; 16]),
name: "dns".into(),
..Default::default()
};
let entries = state.edit(net, &key, 1, &cfg, None, &limits).unwrap();
let mut bad = entries.clone();
bad[0].clock += 1;
assert!(VipSet::default().merge(net, bad, &limits).is_err());
assert!(
VipSet::default()
.merge(NetworkId::from_bytes([4; 32]), entries.clone(), &limits)
.is_err()
);
let mut small = limits.clone();
small.max_vips = 0;
assert!(VipSet::default().merge(net, entries, &small).is_err());
}
#[test]
fn deletion_history_does_not_exhaust_active_capacity_or_accept_stale_fields() {
let network = NetworkId::from_bytes([9; 32]);
let key = iroh::SecretKey::generate();
let limits = Limits {
max_vips: 1,
max_vip_tombstones: 4,
..Default::default()
};
let mut state = VipSet::default();
for index in 0..4 {
let config = VipConfig::new("reused-name");
let fields = state
.edit(network, &key, index * 2 + 1, &config, None, &limits)
.unwrap();
state.merge(network, fields.clone(), &limits).unwrap();
let deletion = state
.delete(
network,
&key,
index * 2 + 2,
&config.id,
&state.revision(&config.id),
)
.unwrap();
state.merge(network, deletion, &limits).unwrap();
assert_eq!(state.entries(&config.id).len(), 1);
assert!(!state.merge(network, fields, &limits).unwrap());
}
let next = VipConfig::new("next");
let fields = state.edit(network, &key, 9, &next, None, &limits).unwrap();
state.merge(network, fields, &limits).unwrap();
assert_eq!(state.configs().len(), 1);
}
#[test]
fn reservations_avoid_nodes_and_name_conflicts_have_a_deterministic_holder() {
let network = NetworkId::from_bytes([4; 32]);
let key = iroh::SecretKey::generate();
let limits = Limits::default();
let mut state = VipSet::default();
for version in 1..=2 {
let config = VipConfig::new("shared-name");
let fields = state
.edit(network, &key, version, &config, None, &limits)
.unwrap();
state.merge(network, fields, &limits).unwrap();
}
let range = "10.0.0.0/29".parse().unwrap();
let taken = ["10.0.0.1".parse().unwrap()].into_iter().collect();
let views = state.views(Some(range), taken, &Default::default(), &limits);
assert!(views[0].error.is_none());
assert!(
views[1]
.error
.as_deref()
.unwrap()
.contains("name conflicts")
);
assert_ne!(views[0].address, views[1].address);
for v in views {
assert_ne!(v.address, Some("10.0.0.1".parse().unwrap()));
}
}
+358
View File
@@ -0,0 +1,358 @@
//! Equal-rights retirement uses real membership, signatures and persistent state.
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use tsunagi::discovery::SharedMemoryDiscovery;
use tsunagi::testing::{TestAgent, config_with, network, wait_for_peers, wait_until};
use tsunagi::{Agent, NetworkId};
#[tokio::test]
async fn bounded_marker_batches_eventually_reach_the_tail_of_a_small_send_queue() {
use tsunagi::config::StoragePaths;
use tsunagi::state::{RecordBody, SignedRecord, SignedRetirement};
use tsunagi::storage::StateStore;
let discovery = SharedMemoryDiscovery::new();
let cleaner = TestAgent::spawn(&discovery).await.unwrap();
let replica = TestAgent::spawn(&discovery).await.unwrap();
let (name, secret) = network("forget-batched-history");
let id = cleaner.agent.join_network(&name, &secret).await.unwrap();
replica.agent.join_network(&name, &secret).await.unwrap();
let cleaner_dir = cleaner.stop().await;
let replica_dir = replica.stop().await;
let cleaner_path = StoragePaths::under(cleaner_dir.path()).state_db();
let replica_path = StoragePaths::under(replica_dir.path()).state_db();
// Seed genuine signed offline history in closed stores. Delivery still uses
// real agents, authenticated loopback sessions and the persistent stores.
let targets = tokio::task::spawn_blocking(move || {
let cleaner = StateStore::open(&cleaner_path).unwrap();
let replica = StateStore::open(&replica_path).unwrap();
let author = cleaner.device_identity().unwrap().unwrap();
let signing_key = iroh::SecretKey::from_bytes(&author.secret_bytes());
let mut targets = Vec::new();
for index in 0..12 {
let target = iroh::SecretKey::generate();
let record = SignedRecord::sign(
&target,
id,
1,
RecordBody::Claim {
address: None,
range: None,
hostname: Some(format!("offline-{index}")),
hostname_since: Some(1),
},
);
cleaner.put_signed_record(&record).unwrap();
replica.put_signed_record(&record).unwrap();
cleaner
.publish_own_retirement(&SignedRetirement::sign(
&signing_key,
id,
100 + index,
record,
))
.unwrap();
targets.push(target.public());
}
targets
})
.await
.unwrap();
let replica = Agent::spawn(config_with(replica_dir.path(), &discovery))
.await
.unwrap();
assert_eq!(
replica
.network_status(id)
.await
.unwrap()
.members
.iter()
.filter(|m| targets.contains(&m.endpoint_id))
.count(),
12
);
let mut config = config_with(cleaner_dir.path(), &discovery);
config.limits.max_state_records = 1;
config.limits.session_send_queue = 2;
let cleaner = Agent::spawn(config).await.unwrap();
wait_for_peers(&cleaner, id, 1).await;
wait_until(
"all cleanup batches arrive despite send queue pressure",
|| async {
replica
.network_status(id)
.await
.ok()?
.members
.iter()
.all(|m| !targets.contains(&m.endpoint_id))
.then_some(())
},
)
.await;
cleaner.shutdown().await;
replica.shutdown().await;
let replica_path = StoragePaths::under(replica_dir.path()).state_db();
tokio::task::spawn_blocking(move || {
let replica = StateStore::open(&replica_path).unwrap();
assert_eq!(replica.retirements(id, 16).unwrap().len(), 12);
})
.await
.unwrap();
}
async fn member(
agent: &Agent,
network: NetworkId,
peer: iroh::EndpointId,
) -> tsunagi::agent::MemberStatus {
wait_until("signed member", || async {
agent
.network_status(network)
.await
.ok()?
.members
.into_iter()
.find(|m| m.endpoint_id == peer)
})
.await
}
#[tokio::test]
async fn retirement_survives_restart_and_stale_replicas_but_allows_a_new_claim() {
let discovery = SharedMemoryDiscovery::new();
let cleaner = TestAgent::spawn_with(|c| c.with_hostname("cleaner"), &discovery)
.await
.unwrap();
let target = TestAgent::spawn_with(|c| c.with_hostname("old-device"), &discovery)
.await
.unwrap();
let replica = TestAgent::spawn_with(|c| c.with_hostname("replica"), &discovery)
.await
.unwrap();
let (name, secret) = network("forget-propagation");
let id = cleaner.agent.join_network(&name, &secret).await.unwrap();
target.agent.join_network(&name, &secret).await.unwrap();
replica.agent.join_network(&name, &secret).await.unwrap();
wait_for_peers(&cleaner.agent, id, 2).await;
let peer = target.agent.endpoint_id();
let original = wait_until("all three agents agree on allocated claims", || async {
let a = cleaner.agent.network_status(id).await.ok()?;
let b = target.agent.network_status(id).await.ok()?;
let c = replica.agent.network_status(id).await.ok()?;
for status in [&a, &b, &c] {
if status.members.len() != 3 {
return None;
}
let addresses: std::collections::HashSet<_> = status
.members
.iter()
.filter_map(|m| m.overlay_address_v4)
.collect();
if addresses.len() != 3 {
return None;
}
}
let record = a.members.iter().find(|m| m.endpoint_id == peer)?;
(b.members
.iter()
.find(|m| m.endpoint_id == peer)?
.record_version
== record.record_version
&& c.members
.iter()
.find(|m| m.endpoint_id == peer)?
.record_version
== record.record_version)
.then(|| record.clone())
})
.await;
assert!(
cleaner
.agent
.forget_node(id, peer, original.record_version)
.await
.is_err(),
"online nodes are not history"
);
assert!(
cleaner
.agent
.forget_node(id, cleaner.agent.endpoint_id(), 1)
.await
.is_err()
);
let replica_dir = replica.stop().await;
let target_dir = target.stop().await;
wait_until("target offline", || async {
cleaner
.agent
.network_status(id)
.await
.ok()?
.members
.iter()
.find(|m| m.endpoint_id == peer)
.filter(|m| !m.online)
.map(|_| ())
})
.await;
assert!(
cleaner
.agent
.forget_node(id, peer, original.record_version + 1)
.await
.is_err(),
"stale confirmation changes nothing"
);
let result = cleaner
.agent
.forget_node(id, peer, original.record_version)
.await
.unwrap();
assert_eq!(result.target, peer);
assert_eq!(result.record_version, original.record_version);
assert_eq!(result.peers_told, 0);
assert!(
!cleaner
.agent
.network_status(id)
.await
.unwrap()
.members
.iter()
.any(|m| m.endpoint_id == peer)
);
let cleaner_dir = cleaner.stop().await;
let cleaner =
Agent::spawn(config_with(cleaner_dir.path(), &discovery).with_hostname("cleaner"))
.await
.unwrap();
let replica =
Agent::spawn(config_with(replica_dir.path(), &discovery).with_hostname("replica"))
.await
.unwrap();
wait_for_peers(&cleaner, id, 1).await;
wait_until("stale replica learns durable retirement", || async {
replica
.network_status(id)
.await
.ok()?
.members
.iter()
.all(|m| m.endpoint_id != peer)
.then_some(())
})
.await;
assert!(
!cleaner
.network_status(id)
.await
.unwrap()
.members
.iter()
.any(|m| m.endpoint_id == peer)
);
replica.set_hostname("old-device").await.unwrap();
let replica_id = replica.endpoint_id();
wait_until("released hostname is reclaimed", || async {
cleaner
.network_status(id)
.await
.ok()?
.members
.iter()
.find(|m| m.endpoint_id == replica_id)
.filter(|m| m.hostname.as_deref() == Some("old-device"))?
.hostname_conflict
.is_none()
.then_some(())
})
.await;
let returned =
Agent::spawn(config_with(target_dir.path(), &discovery).with_hostname("old-device"))
.await
.unwrap();
wait_for_peers(&returned, id, 2).await;
wait_until("returning identity publishes a fresh version", || async {
let m = cleaner
.network_status(id)
.await
.ok()?
.members
.into_iter()
.find(|m| m.endpoint_id == peer)?;
(m.online && m.record_version > original.record_version && m.hostname_conflict.is_some())
.then_some(())
})
.await;
cleaner.shutdown().await;
replica.shutdown().await;
returned.shutdown().await;
}
#[tokio::test]
async fn forgetting_is_network_scoped_and_every_member_has_the_same_right() {
let discovery = SharedMemoryDiscovery::new();
let a = TestAgent::spawn(&discovery).await.unwrap();
let b = TestAgent::spawn(&discovery).await.unwrap();
let target = TestAgent::spawn(&discovery).await.unwrap();
let (name, secret) = network("forget-one-network");
let (other_name, other_secret) = network("keep-other-network");
let id = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
target.agent.join_network(&name, &secret).await.unwrap();
let other = a
.agent
.join_network(&other_name, &other_secret)
.await
.unwrap();
target
.agent
.join_network(&other_name, &other_secret)
.await
.unwrap();
let peer = target.agent.endpoint_id();
let record = member(&b.agent, id, peer).await;
member(&a.agent, other, peer).await;
target.agent.shutdown().await;
wait_until("offline on B", || async {
b.agent
.network_status(id)
.await
.ok()?
.members
.iter()
.find(|m| m.endpoint_id == peer)
.filter(|m| !m.online)
.map(|_| ())
})
.await;
b.agent
.forget_node(id, peer, record.record_version)
.await
.unwrap();
wait_until("A learns B's retirement", || async {
a.agent
.network_status(id)
.await
.ok()?
.members
.iter()
.all(|m| m.endpoint_id != peer)
.then_some(())
})
.await;
assert!(
a.agent
.network_status(other)
.await
.unwrap()
.members
.iter()
.any(|m| m.endpoint_id == peer)
);
a.agent.shutdown().await;
b.agent.shutdown().await;
}
@@ -0,0 +1,4 @@
//! Exercise the vendored retry admission in the default workspace suite.
//! Transport integration tests still use real iroh endpoints.
#[path = "../../../vendor/iroh/src/socket/remote_map/remote_state/pending_paths.rs"]
mod pending_paths;
+1 -1
View File
@@ -162,7 +162,7 @@ async fn rejoining_after_leaving_is_not_taken_for_a_stale_record() {
})
.await;
assert!(
tsunagi::state::DEFAULT_IPV4_RANGE.contains(again),
tsunagi::state::derived_ipv4_range(network_id).contains(again),
"an address in the network's range: {again}"
);
@@ -265,10 +265,7 @@ async fn deactivating_one_network_leaves_the_agent_and_others_running() {
#[tokio::test]
async fn two_networks_on_one_agent_each_get_a_range_of_their_own() {
// One agent, one interface: the configured range can only belong to one
// of them. The second does not go without — it takes the range derived
// from its own id, which every one of its members derives identically,
// so it is an agreement and not a local invention.
// Each network derives its range independently of activation order.
let discovery = SharedMemoryDiscovery::new();
let (first, first_secret) = network("two-ranges-one");
let (second, second_secret) = network("two-ranges-two");
@@ -309,8 +306,8 @@ async fn two_networks_on_one_agent_each_get_a_range_of_their_own() {
"different networks, different addresses"
);
assert!(
tsunagi::state::DEFAULT_IPV4_RANGE.contains(addresses.0),
"the first keeps the configured range: {}",
tsunagi::state::derived_ipv4_range(one).contains(addresses.0),
"the first uses its derived range: {}",
addresses.0
);
assert!(
@@ -319,7 +316,7 @@ async fn two_networks_on_one_agent_each_get_a_range_of_their_own() {
addresses.1
);
assert!(
!tsunagi::state::DEFAULT_IPV4_RANGE.contains(addresses.1),
!tsunagi::state::derived_ipv4_range(one).contains(addresses.1),
"and not in the one the first network holds"
);
@@ -0,0 +1,336 @@
//! Derived defaults and local exclusion of overlapping overlay networks.
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use tsunagi::Agent;
use tsunagi::discovery::SharedMemoryDiscovery;
use tsunagi::state::{DEFAULT_IPV4_RANGE, Ipv4Range, derived_ipv4_range};
use tsunagi::testing::{TestAgent, config_with, network, wait_until};
// These tests need discovery to happen before a collision, so keep the initial
// default outside the established ranges. Derived collisions have their own test.
fn network_outside(
name: &str,
ranges: &[Ipv4Range],
) -> (
tsunagi::identity::NetworkName,
tsunagi::identity::NetworkSecret,
) {
use tsunagi::identity::{NetworkKeys, NetworkName, NetworkSecret};
let name = NetworkName::new(name).unwrap();
let secret = (0..=255u8)
.map(|seed| NetworkSecret::from_bytes([seed; 32]).unwrap())
.find(|secret| {
let derived = derived_ipv4_range(NetworkKeys::derive(&name, secret).network_id());
ranges.iter().all(|range| !range.contains(derived.base))
})
.expect("a deterministic test secret with a non-overlapping default");
(name, secret)
}
#[tokio::test]
async fn colliding_derived_defaults_are_blocked_instead_of_locally_renumbered() {
use std::collections::HashMap;
use tsunagi::identity::{NetworkKeys, NetworkName, NetworkSecret};
let secret = NetworkSecret::from_bytes([19; 32]).unwrap();
let mut seen = HashMap::new();
let (first, second, range) = (0..=65_536)
.find_map(|index| {
let name = NetworkName::new(format!("derived-collision-{index}")).unwrap();
let range = derived_ipv4_range(NetworkKeys::derive(&name, &secret).network_id());
seen.insert(range.base, name.clone())
.map(|first| (first, name, range))
})
.expect("more network names than possible /24 ranges");
let agent = TestAgent::spawn(&SharedMemoryDiscovery::new())
.await
.unwrap();
let first = agent.agent.join_network(&first, &secret).await.unwrap();
let second = agent.agent.join_network(&second, &secret).await.unwrap();
assert!(agent.agent.is_active(first).await);
assert!(!agent.agent.is_active(second).await);
let status = agent.agent.status().await.unwrap();
let blocked = status.network(&second).unwrap();
assert_eq!(blocked.range, Some(range));
assert_eq!(blocked.range_conflicts[0].network_id, first);
assert_eq!(
agent.agent.switch_network(second).await.unwrap(),
vec![first]
);
assert_eq!(
agent.agent.network_status(second).await.unwrap().range,
Some(range)
);
agent.agent.shutdown().await;
}
#[tokio::test]
async fn isolated_first_members_choose_the_same_range_and_keep_it_after_restart() {
let discovery = SharedMemoryDiscovery::new();
let (name, secret) = network("derived-first");
let a = TestAgent::spawn(&discovery).await.unwrap();
let b = TestAgent::spawn(&SharedMemoryDiscovery::new())
.await
.unwrap();
let id = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
let expected = derived_ipv4_range(id);
for agent in [&a.agent, &b.agent] {
wait_until("isolated member allocates in its derived range", || async {
let status = agent.network_status(id).await.ok()?;
(status.range == Some(expected) && !status.members.is_empty()).then_some(())
})
.await;
}
let dir = a.stop().await;
let a = Agent::spawn(config_with(dir.path(), &discovery))
.await
.unwrap();
assert_eq!(a.network_status(id).await.unwrap().range, Some(expected));
a.shutdown().await;
b.agent.shutdown().await;
}
#[tokio::test]
async fn overlap_blocks_activation_switches_and_resumes_across_restart() {
let discovery = SharedMemoryDiscovery::new();
let a = TestAgent::spawn_with(
|c| c.with_overlay_ipv4_range(Some(DEFAULT_IPV4_RANGE)),
&discovery,
)
.await
.unwrap();
let (one_name, one_secret) = network("conflict-one");
let (two_name, two_secret) = network("conflict-two");
let one = a.agent.join_network(&one_name, &one_secret).await.unwrap();
a.agent.network_status(one).await.unwrap();
let two = a.agent.join_network(&two_name, &two_secret).await.unwrap();
assert!(a.agent.is_active(one).await);
assert!(!a.agent.is_active(two).await);
assert!(matches!(
a.agent.activate_network(two).await,
Err(tsunagi::Error::NetworkRangeConflict { .. })
));
let status = a.agent.status().await.unwrap();
let blocked = status.network(&two).unwrap();
assert_eq!(blocked.range, Some(DEFAULT_IPV4_RANGE));
assert_eq!(blocked.range_conflicts[0].network_id, one);
assert_eq!(blocked.range_conflicts[0].name, one_name);
assert_eq!(a.agent.switch_network(two).await.unwrap(), vec![one]);
assert!(a.agent.is_active(two).await);
assert!(!a.agent.is_active(one).await);
a.agent.network_status(two).await.unwrap();
let dir = a.stop().await;
let a = Agent::spawn(config_with(dir.path(), &discovery))
.await
.unwrap();
assert!(a.is_active(two).await, "the user's choice survives restart");
assert!(!a.is_active(one).await);
a.deactivate_network(two).await.unwrap();
assert!(
a.is_active(one).await,
"suspended networks resume when the blocker stops"
);
assert_eq!(
a.network_status(one).await.unwrap().range,
Some(DEFAULT_IPV4_RANGE),
"legacy ranges survive the new default"
);
a.shutdown().await;
}
#[tokio::test]
async fn late_authenticated_state_with_a_nested_range_suspends_only_that_network() {
let discovery = SharedMemoryDiscovery::new();
let broad: Ipv4Range = "10.90.0.0/16".parse().unwrap();
let narrow: Ipv4Range = "10.90.12.0/24".parse().unwrap();
let (name, secret) = network_outside("late-conflict", &[broad]);
let remote = TestAgent::spawn_with(|c| c.with_overlay_ipv4_range(Some(narrow)), &discovery)
.await
.unwrap();
let target = remote.agent.join_network(&name, &secret).await.unwrap();
wait_until("remote range is persisted", || async {
remote
.agent
.network_status(target)
.await
.ok()?
.members
.first()
.map(|_| ())
})
.await;
let (other_name, other_secret) = network("existing-broad");
// Seed an existing network using a real agent and its signed claim.
let seeded = TestAgent::spawn_with(
|c| c.with_overlay_ipv4_range(Some(broad)),
&SharedMemoryDiscovery::new(),
)
.await
.unwrap();
let other = seeded
.agent
.join_network(&other_name, &other_secret)
.await
.unwrap();
wait_until("broad range is persisted", || async {
seeded
.agent
.network_status(other)
.await
.ok()?
.members
.first()
.map(|_| ())
})
.await;
let dir = seeded.stop().await;
let seeded = Agent::spawn(config_with(dir.path(), &discovery))
.await
.unwrap();
seeded.join_network(&name, &secret).await.unwrap();
wait_until("late overlap suspends the joining network", || async {
let status = seeded.status().await.ok()?;
let blocked = status.network(&target)?;
(!seeded.is_active(target).await
&& blocked
.range_conflicts
.iter()
.any(|c| c.network_id == other))
.then_some(())
})
.await;
assert!(seeded.is_active(other).await);
assert_eq!(seeded.switch_network(target).await.unwrap(), vec![other]);
assert_eq!(
seeded.network_status(target).await.unwrap().range,
Some(narrow)
);
seeded.shutdown().await;
remote.agent.shutdown().await;
}
#[tokio::test]
async fn a_wide_range_suspends_all_overlaps_and_manual_stop_cancels_resume() {
let discovery = SharedMemoryDiscovery::new();
let hub = TestAgent::spawn(&discovery).await.unwrap();
let mut members = Vec::new();
let mut ids = Vec::new();
for (name, range) in [
("narrow-one", "10.81.0.0/24"),
("narrow-two", "10.81.1.0/24"),
] {
let peer = TestAgent::spawn_with(
|c| c.with_overlay_ipv4_range(Some(range.parse().unwrap())),
&discovery,
)
.await
.unwrap();
let (name, secret) = network_outside(
name,
&[
"10.81.0.0/24".parse().unwrap(),
"10.81.1.0/24".parse().unwrap(),
],
);
let id = peer.agent.join_network(&name, &secret).await.unwrap();
peer.agent.network_status(id).await.unwrap();
hub.agent.join_network(&name, &secret).await.unwrap();
wait_until("hub adopts the established narrow range", || async {
(hub.agent.network_status(id).await.ok()?.range == Some(range.parse().unwrap()))
.then_some(())
})
.await;
members.push(peer);
ids.push(id);
}
let (name, secret) = network_outside(
"wide-overlap",
&[
"10.81.0.0/24".parse().unwrap(),
"10.81.1.0/24".parse().unwrap(),
],
);
let wide = TestAgent::spawn_with(
|c| c.with_overlay_ipv4_range(Some("10.0.0.0/8".parse().unwrap())),
&discovery,
)
.await
.unwrap();
let id = wide.agent.join_network(&name, &secret).await.unwrap();
wide.agent.network_status(id).await.unwrap();
hub.agent.join_network(&name, &secret).await.unwrap();
wait_until("wide range is blocked by both narrow networks", || async {
let status = hub.agent.status().await.ok()?;
let blocked = status.network(&id)?;
(!hub.agent.is_active(id).await && blocked.range_conflicts.len() == 2).then_some(())
})
.await;
let stopped = hub.agent.switch_network(id).await.unwrap();
assert_eq!(stopped.len(), 2);
assert!(ids.iter().all(|id| stopped.contains(id)));
assert_eq!(hub.agent.switch_network(ids[0]).await.unwrap(), vec![id]);
assert!(
hub.agent.is_active(ids[1]).await,
"switching away from the wide range frees the other narrow range too"
);
assert_eq!(hub.agent.switch_network(id).await.unwrap().len(), 2);
hub.agent.pause_network(ids[0]).await.unwrap();
hub.agent.deactivate_network(id).await.unwrap();
assert!(
!hub.agent.is_active(ids[0]).await,
"explicitly stopped networks do not resume"
);
assert!(hub.agent.is_active(ids[1]).await);
assert!(
hub.agent
.switch_network(tsunagi::NetworkId::from_bytes([0; 32]))
.await
.is_err()
);
assert!(
hub.agent.is_active(ids[1]).await,
"unknown selectors do not stop any network"
);
hub.agent.shutdown().await;
wide.agent.shutdown().await;
for peer in members {
peer.agent.shutdown().await;
}
}
#[tokio::test]
async fn simultaneous_switches_never_leave_overlapping_networks_active() {
let discovery = SharedMemoryDiscovery::new();
let agent = TestAgent::spawn_with(
|c| c.with_overlay_ipv4_range(Some(DEFAULT_IPV4_RANGE)),
&discovery,
)
.await
.unwrap();
let (a, a_secret) = network("switch-a");
let (b, b_secret) = network("switch-b");
let a = agent.agent.join_network(&a, &a_secret).await.unwrap();
agent.agent.network_status(a).await.unwrap();
let b = agent.agent.join_network(&b, &b_secret).await.unwrap();
let (left, right) = tokio::join!(agent.agent.switch_network(b), agent.agent.switch_network(a));
left.unwrap();
right.unwrap();
assert_ne!(
agent.agent.is_active(a).await,
agent.agent.is_active(b).await
);
assert_eq!(
agent
.agent
.list_networks()
.await
.unwrap()
.iter()
.filter(|n| n.active)
.count(),
1
);
agent.agent.shutdown().await;
}
@@ -0,0 +1,130 @@
//! Opaque transport negotiation stays inside authenticated network sessions.
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use std::sync::{Arc, Mutex};
use tsunagi::dataplane::transport::{ControlSender, PacketTransport, SharedLink, TransportError};
use tsunagi::discovery::SharedMemoryDiscovery;
use tsunagi::proto::message::{ControlMessage, TransportPayload};
use tsunagi::testing::{TestAgent, network, wait_for_peers, wait_until};
use tsunagi::{BoxFuture, NetworkId};
#[derive(Debug, Default)]
struct Negotiator {
control: Mutex<Option<Arc<dyn ControlSender>>>,
received: Mutex<Vec<(NetworkId, iroh::EndpointId, Vec<u8>)>>,
}
impl PacketTransport for Negotiator {
fn name(&self) -> &str {
"negotiator"
}
fn as_any(&self) -> &dyn std::any::Any {
self
}
fn attach_control(&self, control: Arc<dyn ControlSender>) {
*self.control.lock().unwrap() = Some(control);
}
fn on_control(
&self,
network: NetworkId,
peer: iroh::EndpointId,
_: &str,
payload: &[u8],
) -> Result<(), TransportError> {
self.received
.lock()
.unwrap()
.push((network, peer, payload.to_vec()));
Ok(())
}
fn open<'a>(
&'a self,
_: NetworkId,
_: iroh::EndpointId,
_: &'a str,
) -> BoxFuture<'a, Result<SharedLink, TransportError>> {
Box::pin(async { Err(TransportError::Closed) })
}
}
#[tokio::test]
async fn negotiated_secrets_are_redacted_scoped_and_delivered_over_real_control() {
let discovery = SharedMemoryDiscovery::new();
let a_transport = Arc::new(Negotiator::default());
let b_transport = Arc::new(Negotiator::default());
let a = TestAgent::spawn_with(
|c| c.with_custom_transport("negotiator", a_transport.clone()),
&discovery,
)
.await
.unwrap();
let b = TestAgent::spawn_with(
|c| c.with_custom_transport("negotiator", b_transport.clone()),
&discovery,
)
.await
.unwrap();
let (name, secret) = network("private-negotiation");
let id = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
wait_for_peers(&a.agent, id, 1).await;
let message = ControlMessage::Transport {
protocol: "negotiator".into(),
payload: TransportPayload::new(b"never-export-this-secret".to_vec()),
};
assert!(!format!("{message:?}").contains("never-export"));
let sender = a_transport.control.lock().unwrap().clone().unwrap();
sender
.send(
id,
b.agent.endpoint_id(),
"negotiator",
TransportPayload::new(b"never-export-this-secret".to_vec()),
)
.await
.unwrap();
wait_until("private transport message", || async {
(!b_transport.received.lock().unwrap().is_empty()).then_some(())
})
.await;
assert_eq!(
b_transport.received.lock().unwrap()[0],
(
id,
a.agent.endpoint_id(),
b"never-export-this-secret".to_vec()
)
);
let unknown = NetworkId::from_bytes([0xa5; 32]);
assert!(
sender
.send(
unknown,
b.agent.endpoint_id(),
"negotiator",
TransportPayload::new(vec![1])
)
.await
.is_err()
);
a.agent.shutdown().await;
b.agent.shutdown().await;
}
#[test]
fn private_bytes_are_bounded_on_decode_and_by_local_policy() {
let limits = tsunagi::Limits {
max_transport_payload_len: 4,
..Default::default()
};
let message = |len| ControlMessage::Transport {
protocol: "awg".into(),
payload: TransportPayload::new(vec![42; len]),
};
assert!(tsunagi::proto::message::validate(&message(4), &limits).is_ok());
assert!(tsunagi::proto::message::validate(&message(5), &limits).is_err());
let bytes = postcard::to_stdvec(&TransportPayload::new(vec![1; 4097])).unwrap();
assert!(postcard::from_bytes::<TransportPayload>(&bytes).is_err());
let bytes = postcard::to_stdvec(&message(4)).unwrap();
let decoded: ControlMessage = postcard::from_bytes(&bytes).unwrap();
assert_eq!(decoded, message(4));
}
+207
View File
@@ -0,0 +1,207 @@
//! Shared services use real membership, signed state and mandatory SQLite.
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use tsunagi::discovery::SharedMemoryDiscovery;
use tsunagi::testing::{TestAgent, config_with, network, wait_for_peers, wait_until};
use tsunagi::vip::{Backend, Listener, Protocol, VipConfig, VipStatus};
use tsunagi::{Agent, NetworkId};
async fn view(agent: &Agent, network: NetworkId, id: &str) -> VipStatus {
wait_until("VIP configuration reaches the replica", || async {
agent
.network_status(network)
.await
.ok()?
.vips
.into_iter()
.find(|v| v.config.id == id)
})
.await
}
#[tokio::test]
async fn any_member_can_edit_services_and_an_offline_creator_is_not_required() {
let discovery = SharedMemoryDiscovery::new();
let a = TestAgent::spawn(&discovery).await.unwrap();
let b = TestAgent::spawn(&discovery).await.unwrap();
let (name, secret) = network("shared-vips");
let net = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
wait_for_peers(&a.agent, net, 1).await;
let cfg = VipConfig::new("api");
a.agent
.save_vip(net, cfg.clone(), None, false)
.await
.unwrap();
let before = view(&b.agent, net, &cfg.id).await;
let mut edited = before.config.clone();
let mut listener = Listener::new(Protocol::Tcp, 443);
listener.backends.push(Backend {
peer: b.agent.endpoint_id().to_string(),
port: 6443,
});
edited.listeners.push(listener);
b.agent
.save_vip(net, edited.clone(), Some(before.revision.clone()), false)
.await
.unwrap();
assert!(
b.agent
.save_vip(net, cfg.clone(), Some(before.revision), false)
.await
.is_err()
);
wait_until("creator observes another member's edit", || async {
(view(&a.agent, net, &cfg.id).await.config == edited).then_some(())
})
.await;
let a_dir = a.stop().await;
let c = TestAgent::spawn(&discovery).await.unwrap();
c.agent.join_network(&name, &secret).await.unwrap();
assert_eq!(view(&c.agent, net, &cfg.id).await.config, edited);
let a = Agent::spawn(config_with(a_dir.path(), &discovery))
.await
.unwrap();
assert_eq!(view(&a, net, &cfg.id).await.config, edited);
let current = view(&c.agent, net, &cfg.id).await;
c.agent
.save_vip(net, current.config, Some(current.revision), true)
.await
.unwrap();
wait_until("deletion reaches the original creator", || async {
a.network_status(net)
.await
.ok()?
.vips
.is_empty()
.then_some(())
})
.await;
a.shutdown().await;
b.stop().await;
c.stop().await;
}
#[tokio::test]
async fn invalid_drafts_are_atomic_and_services_do_not_cross_networks() {
let discovery = SharedMemoryDiscovery::new();
let a = TestAgent::spawn(&discovery).await.unwrap();
let (name, secret) = network("vip-one");
let net = a.agent.join_network(&name, &secret).await.unwrap();
let (other, secret) = network("vip-two");
let second = a.agent.join_network(&other, &secret).await.unwrap();
let mut cfg = VipConfig::new("api");
cfg.listeners.push(Listener::new(Protocol::Udp, 53));
a.agent
.save_vip(net, cfg.clone(), None, false)
.await
.unwrap();
let before = view(&a.agent, net, &cfg.id).await;
let mut invalid = before.config.clone();
invalid.name = "renamed".into();
invalid.listeners.push(invalid.listeners[0].clone());
assert!(
a.agent
.save_vip(net, invalid, Some(before.revision.clone()), false)
.await
.is_err()
);
let after = view(&a.agent, net, &cfg.id).await;
assert_eq!(after.config, before.config);
assert_eq!(after.revision, before.revision);
assert!(
a.agent
.network_status(second)
.await
.unwrap()
.vips
.is_empty()
);
a.stop().await;
}
#[tokio::test]
async fn a_restored_author_advances_its_counter_past_its_own_updates_held_by_a_replica() {
use tsunagi::config::StoragePaths;
use tsunagi::storage::StateStore;
use tsunagi::vip::VipSet;
let discovery = SharedMemoryDiscovery::new();
let a = TestAgent::spawn(&discovery).await.unwrap();
let b = TestAgent::spawn(&discovery).await.unwrap();
let (name, secret) = network("vip-author-recovery");
let net = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
wait_for_peers(&a.agent, net, 1).await;
let config = VipConfig::new("api");
a.agent
.save_vip(net, config.clone(), None, false)
.await
.unwrap();
view(&b.agent, net, &config.id).await;
let author = a.agent.endpoint_id();
let adir = a.stop().await;
let bdir = b.stop().await;
let apath = StoragePaths::under(adir.path()).state_db();
let bpath = StoragePaths::under(bdir.path()).state_db();
// Model an old author backup with a newer, genuinely signed update retained
// by another offline replica. Connections and recovery below remain real.
tokio::task::spawn_blocking(move || {
let a = StateStore::open(apath).unwrap();
let b = StateStore::open(bpath).unwrap();
let mut set = VipSet::default();
set.merge(
net,
a.vip_fields(net, 192).unwrap(),
&tsunagi::Limits::default(),
)
.unwrap();
let mut cfg = set.configs().remove(0);
cfg.name = "newer".into();
let key =
iroh::SecretKey::from_bytes(&a.device_identity().unwrap().unwrap().secret_bytes());
let fields = set
.edit(
net,
&key,
100,
&cfg,
Some(&set.revision(&cfg.id)),
&tsunagi::Limits::default(),
)
.unwrap();
set.merge(net, fields, &tsunagi::Limits::default()).unwrap();
b.put_vip_fields(net, &set.entries(&cfg.id), None).unwrap();
})
.await
.unwrap();
let b = Agent::spawn(config_with(bdir.path(), &discovery))
.await
.unwrap();
let a = Agent::spawn(config_with(adir.path(), &discovery))
.await
.unwrap();
let learned = wait_until(
"the restored author learns its newer signed service",
|| async {
let v = view(&a, net, &config.id).await;
(v.config.name == "newer").then_some(v)
},
)
.await;
let mut next = learned.config;
next.name = "latest".into();
a.save_vip(net, next, Some(learned.revision), false)
.await
.unwrap();
a.shutdown().await;
let path = StoragePaths::under(adir.path()).state_db();
let version = tokio::task::spawn_blocking(move || {
StateStore::open(path)
.unwrap()
.own_record_version(net, author)
.unwrap()
})
.await
.unwrap();
assert!(
version > 100,
"the observed author counter must not be reused"
);
b.shutdown().await;
}
+16 -2
View File
@@ -115,6 +115,7 @@ pub struct TcpTlsLink {
peer: EndpointId,
tx: mpsc::Sender<Bytes>,
rx: tokio::sync::Mutex<mpsc::Receiver<Bytes>>,
tasks: [tokio::task::AbortHandle; 2],
closed_notify: Arc<Notify>,
is_closed: Arc<AtomicBool>,
remote_addr: String,
@@ -136,7 +137,7 @@ impl TcpTlsLink {
let closed_flag_w = Arc::clone(&is_closed);
let closed_notify_w = Arc::clone(&closed_notify);
tokio::spawn(async move {
let writer_task = tokio::spawn(async move {
while let Some(msg) = rx_out.recv().await {
if msg.len() > MAX_FRAME_SIZE {
continue;
@@ -158,7 +159,7 @@ impl TcpTlsLink {
let closed_flag_r = Arc::clone(&is_closed);
let closed_notify_r = Arc::clone(&closed_notify);
tokio::spawn(async move {
let reader_task = tokio::spawn(async move {
let mut len_buf = [0u8; 2];
loop {
if reader.read_exact(&mut len_buf).await.is_err() {
@@ -185,6 +186,7 @@ impl TcpTlsLink {
peer,
tx: tx_out,
rx: tokio::sync::Mutex::new(rx_in),
tasks: [writer_task.abort_handle(), reader_task.abort_handle()],
closed_notify,
is_closed,
remote_addr,
@@ -192,7 +194,19 @@ impl TcpTlsLink {
}
}
impl Drop for TcpTlsLink {
fn drop(&mut self) {
self.close();
}
}
impl PacketLink for TcpTlsLink {
fn close(&self) {
self.is_closed.store(true, Ordering::Release);
self.closed_notify.notify_waiters();
for task in &self.tasks {
task.abort();
}
}
fn network(&self) -> NetworkId {
self.network
}
+116 -4
View File
@@ -21,6 +21,24 @@ const POLL_INTERVAL: Duration = Duration::from_secs(2);
/// A request from the UI to the agent.
pub(crate) enum Command {
SaveVip {
token: String,
network_id: String,
config: tsunagi::vip::VipConfig,
expected: Option<String>,
delete: bool,
},
/// Retire the exact offline-node claim selected in the devices window.
ForgetNode {
network_id: String,
peer: String,
expected_version: u64,
},
/// Replace the network's enabled protocols in preference order.
SetProtocols {
network_id: String,
protocols: Vec<String>,
},
/// Stop serving a network, or start serving it again.
SetActive { network_id: String, active: bool },
/// Turn local broadcast participation on or off for a network.
@@ -44,8 +62,14 @@ pub(crate) enum Command {
}
/// The latest the worker knows, read by the UI each frame.
#[derive(Clone)]
pub(crate) struct VipSaveResult {
pub token: String,
pub result: Result<Vec<tsunagi::vip::VipStatus>, String>,
}
#[derive(Clone, Default)]
pub(crate) struct Snapshot {
pub vip_result: Option<VipSaveResult>,
/// The last status, or the reason it could not be read (agent down, no
/// permission on the socket, …).
pub status: Option<Result<StatusReport, String>>,
@@ -73,6 +97,18 @@ pub(crate) struct AgentClient {
}
impl AgentClient {
#[cfg(test)]
pub(crate) fn test_client() -> (Self, mpsc::UnboundedReceiver<Command>) {
let (tx, rx) = mpsc::unbounded_channel();
(
Self {
tx,
shared: Arc::new(Mutex::new(Snapshot::default())),
pending_copy: Arc::new(Mutex::new(None)),
},
rx,
)
}
/// Starts the worker on `runtime`, reporting to `ctx` (for repaints).
pub(crate) fn spawn(runtime: &tokio::runtime::Handle, ctx: egui::Context) -> Self {
let (tx, rx) = mpsc::unbounded_channel();
@@ -106,7 +142,13 @@ impl AgentClient {
/// Sends a command; a dropped worker simply means nothing happens.
pub(crate) fn send(&self, command: Command) {
let _ = self.tx.send(command);
// Disable protocol editing until the refreshed snapshot arrives;
// otherwise a second checkbox click could replace the first edit
// using the same stale network selection.
lock(&self.shared).busy = true;
if self.tx.send(command).is_err() {
lock(&self.shared).busy = false;
}
}
/// A copy of the latest snapshot.
@@ -142,6 +184,8 @@ async fn worker(
let Some(command) = command else { return };
run(&resolve_socket(), command, &shared, &pending_copy, &ctx).await;
refresh(&shared, &ctx).await;
lock(&shared).busy = false;
ctx.request_repaint();
}
}
}
@@ -179,11 +223,70 @@ async fn run(
ctx.request_repaint();
let outcome: Result<String, String> = match command {
Command::SaveVip {
token,
network_id,
config,
expected,
delete,
} => {
let result = ipc::save_vip(socket, &network_id, config, expected, delete)
.await
.map_err(|e| e.to_string());
lock(shared).vip_result = Some(VipSaveResult {
token,
result: result.clone(),
});
result.map(|_| "VIP configuration saved".into())
}
Command::ForgetNode {
network_id,
peer,
expected_version,
} => ipc::forget_node(socket, &network_id, &peer, expected_version)
.await
.map(|report| {
format!(
"forgot node claim v{}; cleanup queued to {} peers",
report.record_version, report.peers_told
)
})
.map_err(|err| err.to_string()),
Command::SetProtocols {
network_id,
protocols,
} => ipc::set_protocols(socket, &network_id, protocols)
.await
.map(|names| {
format!(
"protocols: {}",
if names.is_empty() {
"none".into()
} else {
names.join(", ")
}
)
})
.map_err(|err| err.to_string()),
Command::SetActive { network_id, active } => ipc::set_active(socket, &network_id, active)
.await
.map(|report| {
let state = if report.active { "started" } else { "stopped" };
format!("{} {}", state, report.name)
if report.suspended.is_empty() {
format!("{} {}", state, report.name)
} else {
format!(
"{} {}; suspended {} until it stops",
state,
report.name,
report
.suspended
.iter()
.map(|n| format!("{} ({})", n.name, n.network_id))
.collect::<Vec<_>>()
.join(", ")
)
}
})
.map_err(|err| err.to_string()),
Command::SetBroadcast {
@@ -211,7 +314,17 @@ async fn run(
}
Command::Join { name, secret } => ipc::join_network(socket, &name, &secret)
.await
.map(|report| format!("joined {}", report.name))
.map(|report| {
format!(
"{} {}",
if report.active {
"joined"
} else {
"configured; stopped by range conflict:"
},
report.name
)
})
.map_err(|err| err.to_string()),
Command::Leave { network_id } => ipc::leave_network(socket, &network_id)
.await
@@ -236,7 +349,6 @@ async fn run(
{
let mut guard = lock(shared);
guard.busy = false;
guard.last_action = if matches!(&outcome, Ok(message) if message.is_empty()) {
None
} else {
+74 -6
View File
@@ -15,6 +15,12 @@ use crate::devices;
use crate::stats::Traffic;
use crate::ui::{self, UiState};
fn app_fonts() -> egui::FontDefinitions {
let mut fonts = egui::FontDefinitions::default();
egui_phosphor::add_to_fonts(&mut fonts, egui_phosphor::Variant::Regular);
fonts
}
/// The window's state.
pub(crate) struct App {
agent: AgentClient,
@@ -34,9 +40,7 @@ impl App {
let ctx = cc.egui_ctx.clone();
// Add the Phosphor icon glyphs to the font set so the icon buttons
// render instead of showing as tofu.
let mut fonts = egui::FontDefinitions::default();
egui_phosphor::add_to_fonts(&mut fonts, egui_phosphor::Variant::Regular);
ctx.set_fonts(fonts);
ctx.set_fonts(app_fonts());
let agent = AgentClient::spawn(&runtime, ctx);
Ok(Self {
@@ -60,6 +64,7 @@ impl eframe::App for App {
ui::draw(ctx, &self.agent, &mut self.state, &snapshot, &self.traffic);
self.show_devices_windows(ctx, &snapshot);
self.show_vip_windows(ctx, &snapshot);
self.show_about_window(ctx, &snapshot);
// A secret the worker fetched for the clipboard: copy it here, on the
@@ -74,6 +79,51 @@ impl eframe::App for App {
}
impl App {
fn show_vip_windows(&mut self, ctx: &egui::Context, snapshot: &crate::agent::Snapshot) {
let Some(Ok(report)) = &snapshot.status else {
return;
};
let ids: Vec<_> = self.state.vip_windows.keys().cloned().collect();
for id in ids {
let Some(network) = report.networks.iter().find(|n| n.network_id == id) else {
self.state.vip_windows.remove(&id);
continue;
};
let Some(state) = self.state.vip_windows.get_mut(&id) else {
continue;
};
let builder = egui::ViewportBuilder::default()
.with_title(format!("tsunagi · VIP · {}", network.name))
.with_app_id(crate::icon::APP_ID)
.with_icon(crate::icon::window())
.with_inner_size([1000., 720.])
.with_min_inner_size([650., 400.]);
let keep = ctx.show_viewport_immediate(
egui::ViewportId::from_hash_of(("vip", &id)),
builder,
|vctx, _| {
let close = crate::vip::show(vctx, &self.agent, state, network, snapshot);
if close {
return false;
}
if vctx.input(|i| i.viewport().close_requested()) {
if state.dirty() {
state.request_close();
vctx.send_viewport_cmd(egui::ViewportCommand::CancelClose);
true
} else {
false
}
} else {
true
}
},
);
if !keep {
self.state.vip_windows.remove(&id);
}
}
}
/// Renders the About window while it is open.
fn show_about_window(&mut self, ctx: &egui::Context, snapshot: &crate::agent::Snapshot) {
if !self.state.about_open {
@@ -116,20 +166,21 @@ impl App {
.with_title(devices::title(network))
.with_app_id(crate::icon::APP_ID)
.with_icon(crate::icon::window())
.with_inner_size([940.0, 540.0])
.with_min_inner_size([480.0, 320.0]);
.with_inner_size([1440.0, 640.0])
.with_min_inner_size([800.0, 400.0]);
let agent = &self.agent;
let traffic = &self.traffic;
// The window shares the single unit setting, so its toggle and the
// main window's stay in step.
let unit = &mut self.state.unit;
let confirm = &mut self.state.exit_confirm;
let forget_confirm = &mut self.state.forget_confirm;
let keep = ctx.show_viewport_immediate(viewport_id, builder, |vctx, _class| {
devices::show(
vctx,
agent,
unit,
confirm,
(confirm, forget_confirm),
network,
&report.endpoint_id,
traffic,
@@ -142,3 +193,20 @@ impl App {
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn route_icons_have_an_embedded_font_in_the_app() {
let fonts = app_fonts();
assert!(fonts.font_data.contains_key("phosphor"));
assert!(
fonts
.families
.get(&egui::FontFamily::Proportional)
.is_some_and(|family| family.iter().any(|name| name == "phosphor"))
);
}
}
+667 -93
View File
@@ -8,6 +8,7 @@
use std::collections::HashMap;
use eframe::egui;
use egui_phosphor::regular as icon;
use tsunagi::ipc::{DataPathReport, NetworkReport, OverlayPeerReport};
@@ -27,6 +28,8 @@ pub(crate) fn title(network: &NetworkReport) -> String {
/// One row of the device table, built from a tunnel and/or a signed member.
struct Device<'a> {
/// Observed claim version, if this row came from signed membership.
record_version: Option<u64>,
/// The agent's own answer to whether it can be reached by any path.
online: bool,
/// How packets reach it.
@@ -42,9 +45,11 @@ struct Device<'a> {
copy: Option<String>,
/// The full endpoint id, which clicking the short form copies.
id: &'a str,
proto: &'a str,
/// End-to-end tunnel protocol, independent of the path carrying it.
tunnel_protocol: &'a str,
address: Option<String>,
handshake_secs: Option<u64>,
control_rtt_ms: Option<u64>,
overlay: Option<&'a OverlayPeerReport>,
/// It is connected and offers to be an exit node.
exit_node: bool,
@@ -64,6 +69,15 @@ pub(crate) struct ExitConfirm {
main: bool,
}
/// An exact offline claim selected for durable cleanup.
pub(crate) struct ForgetConfirm {
network_id: String,
peer: String,
version: u64,
name: String,
address: Option<String>,
}
impl ExitConfirm {
/// A question about stopping the exit node a network's tile shows.
pub(crate) fn stop_from_tile(network: &NetworkReport, peer_id: &str) -> Self {
@@ -108,8 +122,7 @@ fn name_cell(ui: &mut egui::Ui, device: &Device<'_>) {
if let Some((name, holder)) = &device.name_taken {
ui.label(egui::RichText::new(egui_phosphor::regular::WARNING).color(AMBER))
.on_hover_text(format!(
"`{name}` was claimed first by {holder}. This device works by address, \
but nobody finds it by that name."
"Name `{name}` belongs to {holder}. Use this device's address."
));
}
});
@@ -120,11 +133,12 @@ pub(crate) fn show(
ctx: &egui::Context,
agent: &AgentClient,
unit: &mut Unit,
confirm: &mut Option<ExitConfirm>,
confirmations: (&mut Option<ExitConfirm>, &mut Option<ForgetConfirm>),
network: &NetworkReport,
own_id: &str,
traffic: &Traffic,
) {
let (confirm, forget_confirm) = confirmations;
egui::CentralPanel::default().show(ctx, |ui| {
ui.horizontal(|ui| {
ui.heading(&network.name);
@@ -157,10 +171,68 @@ pub(crate) fn show(
egui::ScrollArea::both()
.auto_shrink([false, false])
.show(ui, |ui| {
device_table(ui, unit, network, &devices, traffic, confirm);
device_table(
ui,
unit,
network,
&devices,
traffic,
confirm,
forget_confirm,
);
});
});
confirm_exit(ctx, agent, confirm, Some(&network.network_id));
confirm_forget(ctx, agent, forget_confirm, &network.network_id);
}
fn confirm_forget(
ctx: &egui::Context,
agent: &AgentClient,
confirm: &mut Option<ForgetConfirm>,
network: &str,
) {
let Some(pending) = confirm.as_ref().filter(|c| c.network_id == network) else {
return;
};
egui::Area::new(egui::Id::new("forget node backdrop"))
.order(egui::Order::Middle)
.fixed_pos(egui::Pos2::ZERO)
.show(ctx, |ui| {
let screen = ctx.screen_rect();
ui.allocate_response(screen.size(), egui::Sense::click_and_drag());
ui.painter()
.rect_filled(screen, 0.0, egui::Color32::from_black_alpha(140));
});
let (mut accepted, mut cancelled) = (false, false);
egui::Window::new("Forget offline node")
.order(egui::Order::Foreground)
.collapsible(false)
.resizable(false)
.anchor(egui::Align2::CENTER_CENTER, [0.0, 0.0])
.show(ctx, |ui| {
ui.set_max_width(440.0);
ui.label(format!("Forget {}?", pending.name));
ui.monospace(&pending.peer);
if let Some(address) = &pending.address {
ui.label(format!("IP: {address}"));
}
ui.label("The network will forget this claim and free its name and IP. A returning device can publish a new claim; this does not ban it.");
ui.horizontal(|ui| {
accepted = ui.add_enabled(!agent.snapshot().busy, egui::Button::new("Forget")).clicked();
cancelled = ui.button("Cancel").clicked();
});
});
if accepted {
agent.send(Command::ForgetNode {
network_id: pending.network_id.clone(),
peer: pending.peer.clone(),
expected_version: pending.version,
});
}
if accepted || cancelled || ctx.input(|i| i.key_pressed(egui::Key::Escape)) {
*confirm = None;
}
}
/// The confirmation for an exit-node choice, drawn in the window it was
@@ -353,6 +425,7 @@ fn devices<'a>(network: &'a NetworkReport, own_id: &str) -> Vec<Device<'a>> {
});
let hostname = hostnames.get(id).copied();
Device {
record_version: member.map(|m| m.record_version),
online,
path,
via_name,
@@ -360,11 +433,16 @@ fn devices<'a>(network: &'a NetworkReport, own_id: &str) -> Vec<Device<'a>> {
hostname,
copy: hostname.map(|h| format!("{h}.{}", network.name)),
id,
proto: overlay.map_or("—", |p| p.protocol.as_str()),
tunnel_protocol: overlay.map_or("—", |p| p.protocol.as_str()),
address: overlay
.and_then(|p| p.address.clone())
.or_else(|| member.and_then(|m| m.overlay_address_v4.clone())),
handshake_secs: overlay.and_then(|p| p.handshake_secs_ago),
control_rtt_ms: network
.peers
.iter()
.find(|p| p.endpoint_id == id)
.and_then(|p| p.rtt_ms),
overlay,
exit_node: online
&& network
@@ -384,18 +462,31 @@ fn devices<'a>(network: &'a NetworkReport, own_id: &str) -> Vec<Device<'a>> {
devices
}
/// The widest value each column is expected to hold, so the table starts out
/// wide enough and does not reflow as numbers grow.
const COLUMN_SAMPLES: [&str; 9] = [
"",
const COLUMN_HEADERS: [&str; 10] = [
"device",
"exit",
"tcp-tls",
"end-to-end",
"handshake / RTT",
"address",
"data path",
"pkts",
"bytes",
"rate",
"plot",
];
/// Initial column floors; actual headers, values and totals can grow them.
const COLUMN_SAMPLES: [&str; 10] = [
"device",
"exit",
"end-to-end",
"999d / 9999 ms",
"255.255.255.255",
"23h 59m",
"data path",
"99999/99999",
"1023.9 MB / 1023.9 MB",
"1023.9 kB/s",
"",
"plot",
];
/// Column widths that only ever grow, kept across frames.
@@ -404,11 +495,11 @@ const COLUMN_SAMPLES: [&str; 9] = [
/// gets shorter — `14.4 MB/s` giving way to `382 pkt/s` — would pull the
/// column in and push every cell after it. Remembering the widest seen keeps
/// the table where it is.
fn column_widths(ui: &egui::Ui, id: egui::Id) -> [f32; 9] {
let stored: Option<[f32; 9]> = ui.memory(|m| m.data.get_temp(id));
fn column_widths(ui: &egui::Ui, id: egui::Id) -> [f32; 10] {
let stored: Option<[f32; 10]> = ui.memory(|m| m.data.get_temp(id));
stored.unwrap_or_else(|| {
let font = egui::TextStyle::Body.resolve(ui.style());
let mut widths = [0.0; 9];
let mut widths = [0.0; 10];
for (width, sample) in widths.iter_mut().zip(COLUMN_SAMPLES) {
if !sample.is_empty() {
*width = ui.fonts(|fonts| {
@@ -424,13 +515,25 @@ fn column_widths(ui: &egui::Ui, id: egui::Id) -> [f32; 9] {
}
/// One cell, at least as wide as its column has ever been.
fn cell(ui: &mut egui::Ui, column: usize, widths: &mut [f32; 9], add: impl FnOnce(&mut egui::Ui)) {
fn cell(ui: &mut egui::Ui, column: usize, widths: &[f32; 10], add: impl FnOnce(&mut egui::Ui)) {
let floor = widths[column];
let response = ui.scope(|ui| {
ui.set_min_width(floor);
add(ui);
});
widths[column] = widths[column].max(response.response.rect.width());
ui.allocate_ui_with_layout(
egui::vec2(floor, 0.0),
egui::Layout::top_down(egui::Align::Min),
|ui| {
ui.set_min_width(floor);
ui.set_max_width(floor);
ui.style_mut().wrap_mode = Some(egui::TextWrapMode::Extend);
add(ui);
},
);
}
fn stretch_columns(mut widths: [f32; 10], available: f32) -> [f32; 10] {
let gaps = 10.0 * (widths.len() - 1) as f32;
widths[9] = widths[9].max(100.0);
widths[9] += (available - gaps - widths.iter().sum::<f32>()).max(0.0);
widths
}
/// The exit-node mark of a device: shown when it offers to be one, bright
@@ -456,10 +559,7 @@ fn exit_cell(
};
(color, "in use as your exit node — click to stop")
} else {
(
ui.visuals().text_color(),
"offers to be an exit node — click to send all your internet traffic through it",
)
(ui.visuals().text_color(), "Use this exit node")
};
let response = ui
.add(
@@ -492,6 +592,7 @@ fn device_table(
devices: &[Device<'_>],
traffic: &Traffic,
confirm: &mut Option<ExitConfirm>,
forget_confirm: &mut Option<ForgetConfirm>,
) {
let network_id = network.network_id.as_str();
if devices.is_empty() {
@@ -500,69 +601,198 @@ fn device_table(
}
let widths_id = egui::Id::new(("devices column widths", network_id));
let mut widths = column_widths(ui, widths_id);
let previous_widths = widths;
let font = egui::TextStyle::Body.resolve(ui.style());
let measure = |text: String| {
ui.fonts(|fonts| {
fonts
.layout_no_wrap(text, font.clone(), egui::Color32::WHITE)
.size()
.x
})
};
for (column, header) in COLUMN_HEADERS.into_iter().enumerate() {
widths[column] = widths[column].max(measure(header.to_owned()));
}
let (mut tx_packets, mut rx_packets, mut tx_bytes, mut rx_bytes) = (0u64, 0, 0, 0);
for device in devices {
let name = format!(
"{} ({}){}",
device.name(),
short_id(device.id),
if !device.online { " Forget" } else { "" }
);
widths[0] = widths[0].max(measure(name) + 24.0);
let (path, _) = data_path_text(device.path, device.via_name.as_deref(), device.online);
for (column, text) in [
(2, displayed_protocol(device.tunnel_protocol).to_owned()),
(
3,
handshake_rtt_text(
device.handshake_secs,
device.control_rtt_ms,
displayed_protocol(device.tunnel_protocol) != "—",
),
),
(4, device.address.clone().unwrap_or_default()),
(5, path),
] {
widths[column] = widths[column].max(measure(text));
}
if let Some(peer) = device.overlay {
tx_packets += peer.tx_packets;
rx_packets += peer.rx_packets;
tx_bytes += peer.tx_bytes;
rx_bytes += peer.rx_bytes;
for (column, text) in [
(6, format!("{}/{}", peer.tx_packets, peer.rx_packets)),
(
7,
format!(
"{} / {}",
format::bytes(peer.tx_bytes as f64),
format::bytes(peer.rx_bytes as f64)
),
),
(
8,
format::series_rate(traffic.peer(network_id, &peer.public_key), unit),
),
] {
widths[column] = widths[column].max(measure(text));
}
}
}
for (column, text) in [
(6, format!("{tx_packets}/{rx_packets}")),
(
7,
format!(
"{} / {}",
format::bytes(tx_bytes as f64),
format::bytes(rx_bytes as f64)
),
),
(8, format::series_rate(traffic.network(network_id), unit)),
] {
widths[column] = widths[column].max(measure(text));
}
if widths != previous_widths {
// Grid positions use the previous pass's widths. Let it settle
// before painting newly widened values over the next column.
ui.ctx().request_discard("device table columns widened");
}
ui.memory_mut(|m| m.data.insert_temp(widths_id, widths));
let widths = stretch_columns(widths, ui.available_width());
egui::Grid::new("devices")
.num_columns(9)
.num_columns(10)
.min_col_width(0.0)
.striped(true)
.spacing([10.0, 4.0])
.show(ui, |ui| {
for (column, header) in [
"device", "exit", "proto", "address", "link", "pkts", "bytes", "rate", "plot",
]
.into_iter()
.enumerate()
{
cell(ui, column, &mut widths, |ui| {
ui.weak(header);
for (column, header) in COLUMN_HEADERS.into_iter().enumerate() {
cell(ui, column, &widths, |ui| {
let response = ui.weak(header);
match column {
2 => {
response.on_hover_text("End-to-end encryption.");
}
3 => {
response.on_hover_text("Handshake age / control-connection RTT.");
}
5 => {
response.on_hover_text("Transport carrying the encrypted tunnel.");
}
_ => {}
}
});
}
ui.end_row();
for device in devices {
cell(ui, 0, &mut widths, |ui| name_cell(ui, device));
cell(ui, 1, &mut widths, |ui| {
cell(ui, 0, &widths, |ui| {
ui.horizontal(|ui| {
name_cell(ui, device);
if !device.online
&& !device.in_use
&& let Some(version) = device.record_version
&& ui
.add_enabled(network.active, egui::Button::new("Forget"))
.on_hover_text("Free this offline device's name and IPv4")
.clicked()
{
*forget_confirm = Some(ForgetConfirm {
network_id: network.network_id.clone(),
peer: device.id.into(),
version,
name: device.name(),
address: device.address.clone(),
});
}
});
});
cell(ui, 1, &widths, |ui| {
exit_cell(ui, device, network_id, confirm);
});
cell(ui, 2, &mut widths, |ui| {
ui.label(device.proto);
cell(ui, 2, &widths, |ui| {
ui.push_id((device.id, "end-to-end"), |ui| {
tunnel_cell(ui, device);
});
});
cell(ui, 3, &mut widths, |ui| match &device.address {
cell(ui, 3, &widths, |ui| {
ui.label(handshake_rtt_text(
device.handshake_secs,
device.control_rtt_ms,
displayed_protocol(device.tunnel_protocol) != "—",
))
.on_hover_text(if device.tunnel_protocol == "tcp-tls" {
"Session age / control RTT."
} else {
"Last handshake age / control RTT."
});
});
cell(ui, 4, &widths, |ui| match &device.address {
Some(address) => format::copy_field(ui, address, address),
None => {
ui.weak("—");
}
});
cell(ui, 4, &mut widths, |ui| link_cell(ui, device));
cell(ui, 5, &widths, |ui| {
ui.push_id((device.id, "data-path"), |ui| data_path_cell(ui, device));
});
match device.overlay.filter(|_| device.online) {
Some(peer) => {
let series = traffic.peer(network_id, &peer.public_key);
cell(ui, 5, &mut widths, |ui| {
cell(ui, 6, &widths, |ui| {
ui.label(format!("{}/{}", peer.tx_packets, peer.rx_packets));
});
cell(ui, 6, &mut widths, |ui| {
cell(ui, 7, &widths, |ui| {
ui.label(format!(
"{} / {}",
format::bytes(peer.tx_bytes as f64),
format::bytes(peer.rx_bytes as f64)
));
});
cell(ui, 7, &mut widths, |ui| {
cell(ui, 8, &widths, |ui| {
ui.add(
egui::Label::new(format::series_rate(series, unit))
.wrap_mode(egui::TextWrapMode::Extend),
);
});
ui.allocate_ui(egui::vec2(100.0, 20.0), |ui| {
ui.allocate_ui(egui::vec2(widths[9], 20.0), |ui| {
format::sparkline(ui, series, unit, 18.0);
});
}
None => {
for column in 5..8 {
cell(ui, column, &mut widths, |ui| {
for column in 6..9 {
cell(ui, column, &widths, |ui| {
ui.weak("—");
});
}
ui.weak("—");
cell(ui, 9, &widths, |ui| {
ui.weak("—");
});
}
}
ui.end_row();
@@ -570,32 +800,25 @@ fn device_table(
// Everything this agent has moved through the overlay, including
// devices that have since gone away.
let (mut tx_packets, mut rx_packets, mut tx_bytes, mut rx_bytes) = (0u64, 0, 0, 0);
for peer in devices.iter().filter_map(|d| d.overlay) {
tx_packets += peer.tx_packets;
rx_packets += peer.rx_packets;
tx_bytes += peer.tx_bytes;
rx_bytes += peer.rx_bytes;
}
cell(ui, 0, &mut widths, |ui| {
cell(ui, 0, &widths, |ui| {
ui.strong("total");
});
for column in 1..5 {
cell(ui, column, &mut widths, |ui| {
for column in 1..6 {
cell(ui, column, &widths, |ui| {
ui.label("");
});
}
cell(ui, 5, &mut widths, |ui| {
cell(ui, 6, &widths, |ui| {
ui.strong(format!("{tx_packets}/{rx_packets}"));
});
cell(ui, 6, &mut widths, |ui| {
cell(ui, 7, &widths, |ui| {
ui.strong(format!(
"{} / {}",
format::bytes(tx_bytes as f64),
format::bytes(rx_bytes as f64)
));
});
cell(ui, 7, &mut widths, |ui| {
cell(ui, 8, &widths, |ui| {
ui.add(
egui::Label::new(
egui::RichText::new(format::series_rate(traffic.network(network_id), unit))
@@ -606,50 +829,93 @@ fn device_table(
});
ui.end_row();
});
ui.memory_mut(|m| m.data.insert_temp(widths_id, widths));
}
/// How a device is reached, and how long its tunnel has been up.
///
/// A device reached through others is online, over a longer path, and says
/// so: the colour is the only difference in how it is treated.
fn link_cell(ui: &mut egui::Ui, device: &Device<'_>) {
if !device.online {
ui.label("offline");
/// A protocol name as it should appear in the table.
fn displayed_protocol(protocol: &str) -> &str {
if protocol.is_empty() { "—" } else { protocol }
}
/// The end-to-end tunnel and its own health, without transport details.
fn handshake_rtt_text(handshake: Option<u64>, rtt: Option<u64>, has_tunnel: bool) -> String {
let age = handshake.map_or_else(
|| {
if has_tunnel {
"waiting".into()
} else {
"—".into()
}
},
format::age,
);
let rtt = rtt.map_or_else(|| "—".into(), |value| format!("{value} ms"));
format!("{age} / {rtt}")
}
fn tunnel_hint(protocol: &str) -> String {
let name = displayed_protocol(protocol);
if name == "tcp-tls" {
format!("{name}: Noise end-to-end encryption")
} else {
format!("{name}: WireGuard end-to-end encryption")
}
}
fn tunnel_cell(ui: &mut egui::Ui, device: &Device<'_>) {
let protocol = displayed_protocol(device.tunnel_protocol);
if protocol == "—" {
ui.weak(protocol);
return;
}
let (path, good) = match device.path {
Some(DataPathReport::Direct { transport }) => (format!("direct · {transport}"), true),
ui.label(protocol).on_hover_text(tunnel_hint(protocol));
}
/// Compact route label and whether that route is the preferred direct form.
/// The icon font is embedded by egui-phosphor and installed in `App::new`.
fn data_path_text(
path: Option<&DataPathReport>,
via_name: Option<&str>,
online: bool,
) -> (String, bool) {
if !online {
return ("offline".into(), false);
}
match path {
Some(DataPathReport::Direct { transport }) => (format!("{} {transport}", icon::LINK), true),
Some(DataPathReport::Relayed { hops, .. }) => (
format!(
"via {} · {hops} hops",
device.via_name.as_deref().unwrap_or("another device")
"{} {} · {hops} hops",
icon::SHARE_NETWORK,
via_name.unwrap_or("another device")
),
false,
),
Some(DataPathReport::None) | None => ("no data path".to_string(), false),
};
let age = device.handshake_secs.map(format::age);
let text = match age {
Some(age) => format!("{path} · {age}"),
None => path,
};
let response = if good {
ui.label(text)
} else {
ui.label(egui::RichText::new(text).color(AMBER))
};
if !good {
response.on_hover_text(match device.path {
Some(DataPathReport::Relayed { .. }) => {
"no direct link yet: traffic goes through another device, and moves to a \
direct link by itself as soon as one comes up"
}
_ => "connected, but no path carries packets yet",
});
Some(DataPathReport::None) | None => ("no data path yet".into(), false),
}
}
/// How a device is reached right now.
///
/// A device reached through others is online, over a longer path, and says
/// so: the colour is the only difference in how it is treated.
fn data_path_cell(ui: &mut egui::Ui, device: &Device<'_>) {
let (text, good) = data_path_text(device.path, device.via_name.as_deref(), device.online);
let response = if good {
ui.add(egui::Label::new(text).wrap_mode(egui::TextWrapMode::Extend))
} else {
ui.add(
egui::Label::new(egui::RichText::new(text).color(AMBER))
.wrap_mode(egui::TextWrapMode::Extend),
)
};
response.on_hover_text(match device.path {
Some(DataPathReport::Direct { .. }) => "Direct transport; no mesh relay.",
Some(DataPathReport::Relayed { .. }) => "Traffic forwarded through another device.",
_ if device.online => "Control connected; data transport unavailable.",
_ => "Unreachable.",
});
}
fn yes_no(value: bool) -> &'static str {
if value { "yes" } else { "no" }
}
@@ -662,6 +928,7 @@ mod tests {
fn member(id: &str, online: bool, path: DataPathReport) -> MemberReport {
MemberReport {
record_version: 1,
endpoint_id: id.into(),
online,
path,
@@ -726,4 +993,311 @@ mod tests {
.is_none()
);
}
#[test]
fn tunnel_and_direct_transport_are_named_as_different_layers() {
assert_eq!(displayed_protocol("wg"), "wg");
assert_eq!(tunnel_hint("wg"), "wg: WireGuard end-to-end encryption");
assert_eq!(
data_path_text(
Some(&DataPathReport::Direct {
transport: "tcp-tls".into(),
}),
None,
true,
),
(format!("{} tcp-tls", icon::LINK), true)
);
assert_eq!(
tunnel_hint("tcp-tls"),
"tcp-tls: Noise end-to-end encryption"
);
}
#[test]
fn incomplete_and_longer_paths_stay_explicit() {
assert_eq!(handshake_rtt_text(None, Some(25), true), "waiting / 25 ms");
assert_eq!(
data_path_text(
Some(&DataPathReport::Relayed {
hops: 2,
via: "near".into(),
}),
Some("iris"),
true,
),
(format!("{} iris · 2 hops", icon::SHARE_NETWORK), false)
);
assert_eq!(
data_path_text(Some(&DataPathReport::None), None, true),
("no data path yet".into(), false)
);
assert_eq!(data_path_text(None, None, false), ("offline".into(), false));
}
#[test]
fn handshake_rtt_has_no_invented_values_and_uses_control_measurements() {
use tsunagi::ipc::{OverlayReport, PeerReport};
let network = NetworkReport {
peers: vec![PeerReport {
endpoint_id: "peer".into(),
rtt_ms: Some(27),
..Default::default()
}],
overlay: Some(OverlayReport {
peers: vec![OverlayPeerReport {
endpoint_id: "peer".into(),
protocol: "wg".into(),
handshake_secs_ago: Some(5),
..Default::default()
}],
..Default::default()
}),
..Default::default()
};
let rows = devices(&network, "own");
assert_eq!(
handshake_rtt_text(rows[0].handshake_secs, rows[0].control_rtt_ms, true),
"5s / 27 ms"
);
assert_eq!(handshake_rtt_text(None, None, false), "— / —");
assert_eq!(rows[0].tunnel_protocol, "wg");
assert!(!tunnel_hint(rows[0].tunnel_protocol).contains("wg-quic"));
}
#[test]
fn resized_table_fills_the_window_without_growing_fixed_columns() {
let minimum = [100.0; 10];
let wide = stretch_columns(minimum, 1440.0);
assert!((wide.iter().sum::<f32>() + 90.0 - 1440.0).abs() < 0.1);
assert_eq!(&wide[..9], &minimum[..9]);
let narrow = stretch_columns(minimum, 800.0);
assert_eq!(
narrow, minimum,
"narrow windows scroll rather than clipping columns"
);
assert_eq!(
stretch_columns(minimum, 1440.0),
wide,
"a resize does not accumulate padding"
);
}
#[test]
fn counters_headers_and_handshake_stay_on_one_line_without_overlapping() {
use tsunagi::ipc::{OverlayReport, PeerReport};
fn collect_text<'a>(
shapes: &'a [egui::Shape],
texts: &mut Vec<&'a egui::epaint::TextShape>,
) {
for shape in shapes {
match shape {
egui::Shape::Text(shape) => texts.push(shape),
egui::Shape::Vec(shapes) => collect_text(shapes, texts),
_ => {}
}
}
}
let context = egui::Context::default();
for (width, packets, age) in [
(1600.0, 1569, 56),
(1600.0, 55_555, 172_800),
(1600.0, 137_245, 172_800),
(900.0, 9_876_543_210, 172_800),
(1600.0, 42, 56),
] {
let network = NetworkReport {
network_id: "network".into(),
peers: ["peer", "other"]
.into_iter()
.map(|id| PeerReport {
endpoint_id: id.into(),
rtt_ms: Some(12_345),
..Default::default()
})
.collect(),
overlay: Some(OverlayReport {
peers: ["peer", "other"]
.into_iter()
.map(|id| OverlayPeerReport {
endpoint_id: id.into(),
protocol: "wg-quic".into(),
address: Some("10.13.37.100".into()),
handshake_secs_ago: Some(age),
tx_packets: packets,
rx_packets: packets + 1,
tx_bytes: 123_456_789,
rx_bytes: 987_654_321,
..Default::default()
})
.collect(),
..Default::default()
}),
..Default::default()
};
let rows = devices(&network, "own");
for _ in 0..3 {
let output = context.run(
egui::RawInput {
screen_rect: Some(egui::Rect::from_min_size(
egui::Pos2::ZERO,
egui::vec2(2400.0, 640.0),
)),
..Default::default()
},
|ctx| {
egui::CentralPanel::default().show(ctx, |ui| {
ui.allocate_ui(egui::vec2(width, 480.0), |ui| {
device_table(
ui,
Unit::Packets,
&network,
&rows,
&Traffic::default(),
&mut None,
&mut None,
);
});
});
},
);
let shapes: Vec<_> = output.shapes.into_iter().map(|s| s.shape).collect();
let mut texts = Vec::new();
collect_text(&shapes, &mut texts);
for (label, next_column) in [
("handshake / RTT".to_string(), "address"),
(handshake_rtt_text(Some(age), Some(12_345), true), "address"),
(format!("{packets}/{}", packets + 1), "bytes"),
(format!("{}/{}", packets * 2, (packets + 1) * 2), "bytes"),
] {
let right = texts
.iter()
.find(|shape| shape.galley.text() == next_column)
.unwrap()
.pos
.x;
let matching: Vec<_> = texts
.iter()
.filter(|shape| shape.galley.text() == label)
.collect();
assert!(!matching.is_empty(), "missing {label}");
for shape in &matching {
assert_eq!(shape.galley.rows.len(), 1, "wrapped {label}");
assert!(
shape.visual_bounding_rect().right() <= right,
"overlapping {label}: {} > {right} at width {width}",
shape.visual_bounding_rect().right()
);
}
}
}
}
}
#[test]
fn a_long_relay_name_gets_its_full_column_width() {
let mut relay = member("near", true, DataPathReport::None);
relay.hostname = Some("a-relay-with-a-long-device-name".into());
let network = NetworkReport {
network_id: "network".into(),
members: vec![
relay,
member(
"far",
true,
DataPathReport::Relayed {
hops: 2,
via: "near".into(),
},
),
],
..Default::default()
};
let rows = devices(&network, "own");
let far = rows.iter().find(|device| device.id == "far").unwrap();
let (label, _) = data_path_text(far.path, far.via_name.as_deref(), far.online);
let context = egui::Context::default();
let mut fonts = egui::FontDefinitions::default();
egui_phosphor::add_to_fonts(&mut fonts, egui_phosphor::Variant::Regular);
context.set_fonts(fonts);
let _ = context.run(egui::RawInput::default(), |ctx| {
egui::CentralPanel::default().show(ctx, |ui| {
device_table(
ui,
Unit::Bytes,
&network,
&rows,
&Traffic::default(),
&mut None,
&mut None,
);
let stored: [f32; 10] = ui
.memory(|memory| {
memory
.data
.get_temp(egui::Id::new(("devices column widths", "network")))
})
.unwrap();
let font = egui::TextStyle::Body.resolve(ui.style());
let needed = ui.fonts(|fonts| {
fonts
.layout_no_wrap(label.clone(), font, egui::Color32::WHITE)
.size()
.x
});
assert!(stored[5] >= needed);
});
});
}
#[test]
fn rendered_table_uses_window_width_without_an_expanding_device_cell() {
let context = egui::Context::default();
let network = NetworkReport {
members: vec![MemberReport {
endpoint_id: "0123456789abcdef".into(),
hostname: Some("laptop".into()),
online: true,
..Default::default()
}],
..Default::default()
};
let rows = devices(&network, "own");
for width in [1440.0, 1200.0, 1440.0] {
for _ in 0..3 {
let input = egui::RawInput {
screen_rect: Some(egui::Rect::from_min_size(
egui::Pos2::ZERO,
egui::vec2(width, 640.0),
)),
..Default::default()
};
let _output = context.run(input, |ctx| {
egui::CentralPanel::default().show(ctx, |ui| {
let right = ui.max_rect().right();
egui::ScrollArea::both()
.auto_shrink([false, false])
.show(ui, |ui| {
device_table(
ui,
Unit::Bytes,
&network,
&rows,
&Traffic::default(),
&mut None,
&mut None,
);
});
assert!(
(ui.min_rect().right() - right).abs() < 2.0,
"table width after resize: {} vs {right}",
ui.min_rect().right()
);
});
});
}
}
}
}
+1
View File
@@ -78,6 +78,7 @@ fn colour_of(health: Health) -> [u8; 3] {
match health {
Health::Disconnected => DARK_RED,
Health::Connected => BLUE,
Health::RangeConflict => [0xd9, 0x91, 0x20],
Health::ExitNode => GREEN,
}
}
+1
View File
@@ -19,6 +19,7 @@ mod icon;
mod stats;
mod tray;
mod ui;
mod vip;
mod watch;
use eframe::egui;
+1 -1
View File
@@ -34,7 +34,7 @@ pub(crate) struct Series {
}
impl Series {
fn sample(&mut self, now: Instant, packets: u64, bytes: u64) {
pub(crate) fn sample(&mut self, now: Instant, packets: u64, bytes: u64) {
if let Some((then, prev_packets, prev_bytes)) = self.last {
let dt = now.saturating_duration_since(then).as_secs_f32();
if dt > 0.0 {
+115 -10
View File
@@ -41,10 +41,13 @@ pub(crate) struct UiState {
pub(crate) host_edit: Option<String>,
/// Network ids whose devices window is open.
pub(crate) open_devices: BTreeSet<String>,
pub(crate) vip_windows: std::collections::BTreeMap<String, crate::vip::Window>,
/// Whether the About window is open.
pub(crate) about_open: bool,
/// An exit-node choice waiting for the user to confirm it.
pub(crate) exit_confirm: Option<devices::ExitConfirm>,
/// A selected offline claim waiting for cleanup confirmation.
pub(crate) forget_confirm: Option<devices::ForgetConfirm>,
}
impl Default for UiState {
@@ -54,8 +57,10 @@ impl Default for UiState {
unit: Unit::Packets,
host_edit: None,
open_devices: BTreeSet::new(),
vip_windows: Default::default(),
about_open: false,
exit_confirm: None,
forget_confirm: None,
}
}
}
@@ -123,7 +128,7 @@ pub(crate) fn draw(
ui.weak("No networks yet — join one below.");
}
for network in &report.networks {
draw_network(ui, agent, state, traffic, network, &report.endpoint_id);
draw_network(ui, agent, state, traffic, network, report);
}
});
}
@@ -202,12 +207,13 @@ fn draw_network(
state: &mut UiState,
traffic: &Traffic,
network: &NetworkReport,
own_id: &str,
report: &StatusReport,
) {
ui.add_space(6.0);
egui::Frame::group(ui.style()).show(ui, |ui| {
ui.horizontal(|ui| {
format::copy_field(ui, network.name.as_str(), &network.name);
if ui.button("VIP").clicked(){state.vip_windows.entry(network.network_id.clone()).or_default();}
ui.with_layout(egui::Layout::right_to_left(egui::Align::Center), |ui| {
let leave =
egui::Button::new(egui::RichText::new("Leave").color(egui::Color32::WHITE))
@@ -244,21 +250,40 @@ fn draw_network(
ui.weak("peers");
ui.label(format!(
"{} online · {} known",
devices::online_count(network, own_id),
devices::known_count(network, own_id)
devices::online_count(network, &report.endpoint_id),
devices::known_count(network, &report.endpoint_id)
));
ui.end_row();
});
draw_protocols(ui, agent, network, &report.available_protocols);
format::sparkline(ui, series, state.unit, 34.0);
if !network.range_conflicts.is_empty() {
ui.colored_label(BAD, "Stopped: overlapping network ranges");
for conflict in &network.range_conflicts {
ui.label(format!("{} ({}) uses {}", conflict.name, conflict.network_id, conflict.range));
}
ui.label("Switching suspends these networks until this network stops, then resumes them automatically.");
if network.auto_start { ui.weak("This network will resume automatically when its range is free."); }
}
ui.horizontal(|ui| {
let mut active = network.active;
if ui.checkbox(&mut active, "active").changed() {
agent.send(Command::SetActive {
network_id: network.network_id.clone(),
active,
});
if !network.active && !network.range_conflicts.is_empty() {
if ui.button("Switch to this network").clicked() {
agent.send(Command::SetActive { network_id: network.network_id.clone(), active: true });
}
if network.auto_start && ui.button("Keep stopped").clicked() {
agent.send(Command::SetActive { network_id: network.network_id.clone(), active: false });
}
} else {
let mut active = network.active;
if ui.checkbox(&mut active, "active").changed() {
agent.send(Command::SetActive {
network_id: network.network_id.clone(),
active,
});
}
}
let mut broadcast = network.broadcast;
if ui.checkbox(&mut broadcast, "broadcast").changed() {
@@ -328,6 +353,86 @@ fn draw_network(
});
}
fn draw_protocols(
ui: &mut egui::Ui,
agent: &AgentClient,
network: &NetworkReport,
available: &[tsunagi::ipc::ProtocolReport],
) {
let summary = if network.protocols.is_empty() {
"none".into()
} else {
network.protocols.join(", ")
};
egui::CollapsingHeader::new(format!("protocols {summary}"))
.id_salt(("protocols", &network.network_id))
.show(ui, |ui| {
ui.weak("Preference order; changes apply immediately and are saved.");
let mut selected = network.protocols.clone();
let mut changed = false;
ui.add_enabled_ui(!agent.snapshot().busy, |ui| {
// Enabled items follow the user's preference; unchecked items
// follow the agent's catalog. The worker refreshes after edits.
let ordered: Vec<_> = network
.protocols
.iter()
.filter_map(|name| available.iter().find(|p| &p.name == name))
.chain(
available
.iter()
.filter(|p| !network.protocols.contains(&p.name)),
)
.collect();
for protocol in ordered {
ui.horizontal(|ui| {
let mut on = selected.contains(&protocol.name);
if ui.checkbox(&mut on, &protocol.name).changed() {
if on {
selected.push(protocol.name.clone());
} else {
selected.retain(|p| p != &protocol.name);
}
changed = true;
}
if let Some(index) = selected.iter().position(|p| p == &protocol.name) {
if ui
.add_enabled(index > 0, egui::Button::new(icon::ARROW_UP))
.on_hover_text("prefer earlier")
.clicked()
{
selected.swap(index, index - 1);
changed = true;
}
if ui
.add_enabled(
index + 1 < selected.len(),
egui::Button::new(icon::ARROW_DOWN),
)
.on_hover_text("prefer later")
.clicked()
{
selected.swap(index, index + 1);
changed = true;
}
}
});
ui.indent(("description", &network.network_id, &protocol.name), |ui| {
ui.weak(&protocol.summary);
});
}
});
if selected.is_empty() {
ui.weak("Control only; no IP traffic in this network.");
}
if changed {
agent.send(Command::SetProtocols {
network_id: network.network_id.clone(),
protocols: selected,
});
}
});
}
/// A prominent line about an exit node. One that carries a command copies it
/// when clicked.
fn draw_exit_warning(ui: &mut egui::Ui, warning: &exit::Warning) {
+461
View File
@@ -0,0 +1,461 @@
//! Per-network VIP editor. Drafts never issue commands until Save is pressed.
use crate::agent::{AgentClient, Command, Snapshot};
use crate::stats::{Series, Unit};
use eframe::egui;
use std::collections::BTreeMap;
use tsunagi::ipc::NetworkReport;
use tsunagi::vip::{Backend, Listener, Protocol, VipConfig, VipStatus};
#[derive(Clone)]
struct Draft {
config: VipConfig,
original: VipConfig,
revision: Option<String>,
delete: bool,
}
impl Draft {
fn new(config: VipConfig, revision: Option<String>) -> Self {
Self {
original: config.clone(),
config,
revision,
delete: false,
}
}
fn dirty(&self) -> bool {
self.revision.is_none() || self.delete || self.config != self.original
}
}
#[derive(Default)]
pub(crate) struct Window {
draft: Option<Draft>,
pending: Option<String>,
error: Option<String>,
close_question: bool,
close_after_save: bool,
rates: BTreeMap<String, (Series, Series)>,
generation: u64,
acknowledged_generation: Option<u64>,
}
impl Window {
pub fn request_close(&mut self) {
self.close_question = true;
self.close_after_save = true;
}
pub fn dirty(&self) -> bool {
self.draft.as_ref().is_some_and(Draft::dirty) || self.pending.is_some()
}
}
fn peer_name<'a>(n: &'a NetworkReport, id: &'a str) -> &'a str {
n.members
.iter()
.find(|m| m.endpoint_id == id)
.and_then(|m| m.hostname.as_deref())
.unwrap_or(id)
}
pub(crate) fn show(
ctx: &egui::Context,
agent: &AgentClient,
state: &mut Window,
n: &NetworkReport,
snapshot: &Snapshot,
) -> bool {
if state.generation != snapshot.generation {
state.generation = snapshot.generation;
if let Some(at) = snapshot.at {
for v in &n.vips {
let counters = v.listeners.iter().fold((0, 0, 0, 0), |a, l| {
(
a.0 + l.counters.tx_packets,
a.1 + l.counters.tx_bytes,
a.2 + l.counters.rx_packets,
a.3 + l.counters.rx_bytes,
)
});
let (tx, rx) = state.rates.entry(v.config.id.clone()).or_default();
tx.sample(at, counters.0, counters.1);
rx.sample(at, counters.2, counters.3);
}
}
}
if let Some(result) = &snapshot.vip_result
&& state.pending.as_deref() == Some(result.token.as_str())
{
state.pending = None;
state.acknowledged_generation = Some(snapshot.generation);
match &result.result {
Ok(vips) => {
if let Some(draft) = &state.draft {
state.draft = vips
.iter()
.find(|v| v.config.id == draft.config.id)
.map(|v| Draft::new(v.config.clone(), Some(v.revision.clone())));
}
state.error = None;
}
Err(error) => {
state.error = Some(error.clone());
state.close_after_save = false;
}
}
}
let mut close = state.close_after_save && !state.dirty();
let mut discard_new = false;
egui::CentralPanel::default().show(ctx, |ui| {
ui.heading(format!("VIP · {}", n.name));
ui.weak("Configuration is shared with every member. Changes take effect only after Save.");
if state.close_question {
ui.horizontal(|ui| {
ui.label("Unsaved changes.");
if ui
.add_enabled(
state.pending.is_none(),
egui::Button::new("Discard and close"),
)
.clicked()
{
state.draft = None;
state.close_question = false;
close = true;
}
if ui.button("Keep editing").clicked() {
state.close_question = false;
state.close_after_save = false;
}
});
}
if let Some(error) = &state.error {
ui.colored_label(egui::Color32::LIGHT_RED, error);
}
ui.separator();
ui.horizontal_wrapped(|ui| {
for vip in &n.vips {
let selected = state
.draft
.as_ref()
.is_some_and(|d| d.config.id == vip.config.id);
if ui.selectable_label(selected, &vip.config.name).clicked() {
if state.dirty() {
state.error = Some(
"Save or discard the current draft before selecting another VIP."
.into(),
);
} else {
state.draft =
Some(Draft::new(vip.config.clone(), Some(vip.revision.clone())));
state.error = None;
}
}
}
if ui.button("+ New VIP").clicked() {
if state.dirty() {
state.error = Some("Save or discard the current draft first.".into());
} else {
state.draft = Some(Draft::new(VipConfig::new(""), None));
state.error = None;
}
}
});
let Some(draft) = &mut state.draft else {
ui.label("Select a VIP or create one.");
return;
};
let current = n.vips.iter().find(|v| v.config.id == draft.config.id);
if state.pending.is_none()
&& state.acknowledged_generation != Some(snapshot.generation)
&& !draft.dirty()
&& let Some(current) = current
&& draft.revision.as_ref() != Some(&current.revision)
{
*draft = Draft::new(current.config.clone(), Some(current.revision.clone()));
}
let changed =
current.is_some_and(|v| draft.revision.as_ref().is_some_and(|r| r != &v.revision));
if changed {
ui.colored_label(
egui::Color32::YELLOW,
"This VIP changed on the network. Reload before saving; your draft is retained.",
);
}
if draft.revision.is_some() && current.is_none() {
ui.colored_label(
egui::Color32::YELLOW,
"This VIP was removed on the network. The draft cannot overwrite its deletion.",
);
}
let busy = state.pending.is_some();
ui.horizontal(|ui| {
let valid = draft.config.validate(&tsunagi::Limits::default());
if !n.active {
ui.weak("Start the network to save changes.");
}
if ui
.add_enabled(
n.active
&& !busy
&& draft.dirty()
&& !changed
&& (valid.is_ok() || draft.delete),
egui::Button::new("Save"),
)
.clicked()
{
let token = VipConfig::new("").id;
state.pending = Some(token.clone());
agent.send(Command::SaveVip {
token,
network_id: n.network_id.clone(),
config: draft.config.clone(),
expected: draft.revision.clone(),
delete: draft.delete,
});
}
if ui
.add_enabled(!busy, egui::Button::new("Discard / reload"))
.clicked()
{
if let Some(v) = current {
*draft = Draft::new(v.config.clone(), Some(v.revision.clone()));
} else {
discard_new = true;
}
state.error = None;
}
if busy {
ui.spinner();
ui.label("Saving…");
} else if draft.dirty() {
ui.weak("Unsaved changes");
}
if let Err(error) = valid {
ui.colored_label(egui::Color32::LIGHT_RED, error.to_string());
}
});
ui.separator();
egui::ScrollArea::vertical().show(ui, |ui| {
ui.add_enabled_ui(!busy, |ui| {
ui.horizontal(|ui| {
ui.label("Name");
ui.add(
egui::TextEdit::singleline(&mut draft.config.name)
.id(egui::Id::new(("vip-name", &draft.config.id))),
);
ui.weak(format!(".{}", n.name));
});
if draft.revision.is_some() {
ui.checkbox(&mut draft.delete, "Delete this VIP when saved");
}
if !draft.delete {
edit_listeners(ui, &mut draft.config, n);
}
});
if let Some(current) = current {
ui.separator();
status(ui, current, n, state.rates.get(&current.config.id));
}
});
});
if discard_new {
state.draft = None;
state.error = None;
}
close
}
fn edit_listeners(ui: &mut egui::Ui, config: &mut VipConfig, n: &NetworkReport) {
let mut remove = None;
for (index, l) in config.listeners.iter_mut().enumerate() {
ui.push_id(index,|ui|{egui::Frame::group(ui.style()).show(ui,|ui|{
ui.horizontal(|ui|{egui::ComboBox::from_id_salt("protocol").selected_text(l.protocol.to_string()).show_ui(ui,|ui|{ui.selectable_value(&mut l.protocol,Protocol::Tcp,"TCP");ui.selectable_value(&mut l.protocol,Protocol::Udp,"UDP");});ui.label("Port");ui.add(egui::DragValue::new(&mut l.port).range(1..=65535));if ui.button("Remove listener").clicked(){remove=Some(index);}});
if l.protocol==Protocol::Tcp{ui.checkbox(&mut l.port_check,"Active TCP port checks");}else{ui.weak("UDP health: encrypted path probes and explicit port errors; application readiness is unknown.");}
ui.collapsing("Checks and flow expiry",|ui|{egui::Grid::new("settings").show(ui,|ui|{
ui.label("Check interval, ms");ui.add(egui::DragValue::new(&mut l.check_interval_ms).range(250..=3_600_000));ui.end_row();
ui.label("Check timeout, ms");ui.add(egui::DragValue::new(&mut l.check_timeout_ms).range(100..=60_000));ui.end_row();
ui.label("Failures before exclusion");ui.add(egui::DragValue::new(&mut l.failure_threshold).range(1..=255));ui.end_row();
ui.label("Successes before recovery");ui.add(egui::DragValue::new(&mut l.recovery_threshold).range(1..=255));ui.end_row();
ui.label("Flow idle timeout, seconds");ui.add(egui::DragValue::new(&mut l.idle_timeout_secs).range(1..=604800));ui.end_row();
});});
let mut remove_backend=None;for(i,b)in l.backends.iter_mut().enumerate(){ui.push_id(i,|ui|{ui.horizontal(|ui|{
egui::ComboBox::from_id_salt("node").width(240.).selected_text(peer_name(n,&b.peer)).show_ui(ui,|ui|{for m in &n.members{let label=format!("{} ({})",m.hostname.as_deref().unwrap_or("unnamed"),m.endpoint_id.chars().take(8).collect::<String>());ui.selectable_value(&mut b.peer,m.endpoint_id.clone(),label);}});
ui.label("Target port");ui.add(egui::DragValue::new(&mut b.port).range(1..=65535));if ui.button("Remove").clicked(){remove_backend=Some(i);}
});});}if let Some(i)=remove_backend{l.backends.remove(i);}
if ui.button("+ Backend").clicked(){l.backends.push(Backend{peer:n.members.first().map(|m|m.endpoint_id.clone()).unwrap_or_default(),port:l.port});}
});});
}
if let Some(index) = remove {
config.listeners.remove(index);
}
if ui.button("+ Listener").clicked() {
config.listeners.push(Listener::new(Protocol::Tcp, 443));
}
}
fn status(ui: &mut egui::Ui, v: &VipStatus, n: &NetworkReport, rates: Option<&(Series, Series)>) {
ui.heading("Status · this device");
if let Some(error) = &v.error {
ui.colored_label(egui::Color32::LIGHT_RED, error);
}
if let Some((tx, rx)) = rates {
ui.label(format!(
"TX {:.1} KiB/s · RX {:.1} KiB/s",
tx.rate(Unit::Bytes) / 1024.,
rx.rate(Unit::Bytes) / 1024.
));
}
for l in &v.listeners {
ui.strong(format!("{}/{}", l.protocol, l.port));
let selected = l
.selected
.as_ref()
.map(|b| format!("{}:{}", peer_name(n, &b.peer), b.port))
.unwrap_or_else(|| "none".into());
ui.label(format!("Next new flow: {selected}"));
ui.label(format!(
"TX {} bytes / {} packets · RX {} bytes / {} packets · drops {} · UDP failovers {}",
l.counters.tx_bytes,
l.counters.tx_packets,
l.counters.rx_bytes,
l.counters.rx_packets,
l.counters.dropped,
l.counters.failovers
));
for b in &l.backends {
let rtt = b
.rtt_us
.map(|v| format!("{:.2} ms", v as f64 / 1000.))
.unwrap_or_else(|| "unknown".into());
let age = b
.measurement_age_ms
.map(|v| format!("{:.1}s", v as f64 / 1000.))
.unwrap_or_else(|| "unknown".into());
ui.label(format!(
"{}:{} · {} · RTT {} (age {}) · {} active · TX/RX {}/{} bytes",
peer_name(n, &b.backend.peer),
b.backend.port,
b.state,
rtt,
age,
b.active_flows,
b.counters.tx_bytes,
b.counters.rx_bytes
));
}
}
ui.collapsing("Internal address and identity", |ui| {
ui.monospace(&v.config.id);
if let Some(address) = v.address {
ui.monospace(address.to_string());
}
});
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
#[test]
fn editing_the_actual_form_sends_nothing_until_save_is_clicked() {
let (agent, mut commands) = AgentClient::test_client();
let ctx = egui::Context::default();
let mut config = VipConfig::new("api");
config.listeners.push(Listener::new(Protocol::Tcp, 443));
let mut state = Window {
draft: Some(Draft::new(config.clone(), Some("revision".into()))),
..Default::default()
};
let network = NetworkReport {
active: true,
name: "lab".into(),
vips: vec![VipStatus {
config: config.clone(),
revision: "revision".into(),
address: None,
error: None,
listeners: vec![],
}],
..Default::default()
};
let snapshot = Snapshot::default();
let frame = |events: Vec<egui::Event>, state: &mut Window| {
ctx.run(
egui::RawInput {
screen_rect: Some(egui::Rect::from_min_size(
egui::Pos2::ZERO,
egui::vec2(1000., 800.),
)),
events,
..Default::default()
},
|ctx| {
show(ctx, &agent, state, &network, &snapshot);
},
)
};
frame(vec![], &mut state);
ctx.memory_mut(|m| m.request_focus(egui::Id::new(("vip-name", &config.id))));
frame(vec![egui::Event::Text("new".into())], &mut state);
assert!(state.dirty());
assert_ne!(state.draft.as_ref().unwrap().config.name, config.name);
assert!(commands.try_recv().is_err());
assert_eq!(network.vips[0].config, config);
fn save_position(shape: &egui::Shape) -> Option<egui::Pos2> {
match shape {
egui::Shape::Text(text) if text.galley.text() == "Save" => {
Some(text.pos + text.galley.rect.center().to_vec2())
}
egui::Shape::Vec(shapes) => shapes.iter().find_map(save_position),
_ => None,
}
}
// Save was disabled before this edit; allow the next layout to register
// its now-enabled hit target before dispatching the pointer press.
let ready = frame(vec![], &mut state);
let position = ready
.shapes
.iter()
.find_map(|s| save_position(&s.shape))
.unwrap();
frame(
vec![
egui::Event::PointerMoved(position),
egui::Event::PointerButton {
pos: position,
button: egui::PointerButton::Primary,
pressed: true,
modifiers: egui::Modifiers::NONE,
},
],
&mut state,
);
frame(
vec![egui::Event::PointerButton {
pos: position,
button: egui::PointerButton::Primary,
pressed: false,
modifiers: egui::Modifiers::NONE,
}],
&mut state,
);
let Command::SaveVip {
config: saved,
expected,
delete,
..
} = commands.try_recv().unwrap()
else {
panic!("Save must submit one complete VIP draft");
};
assert_ne!(saved.name, config.name);
assert_eq!(expected.as_deref(), Some("revision"));
assert!(!delete);
assert!(commands.try_recv().is_err());
}
#[test]
fn all_listener_settings_render_without_a_live_agent() {
let ctx = egui::Context::default();
let mut cfg = VipConfig::new("api");
cfg.listeners.push(Listener::new(Protocol::Tcp, 443));
cfg.listeners.push(Listener::new(Protocol::Udp, 53));
let n = NetworkReport::default();
let _ = ctx.run(egui::RawInput::default(), |ctx| {
egui::CentralPanel::default().show(ctx, |ui| edit_listeners(ui, &mut cfg, &n));
});
assert_eq!(cfg.listeners.len(), 2);
}
}
+28
View File
@@ -22,6 +22,8 @@ pub(crate) enum Health {
Disconnected,
/// Connected, nothing out of the ordinary.
Connected,
/// At least one configured network is blocked by an overlapping active range.
RangeConflict,
/// This device's traffic leaves through an exit node, and that works.
ExitNode,
}
@@ -33,6 +35,13 @@ impl Health {
let Some(report) = report else {
return Self::Disconnected;
};
if report
.networks
.iter()
.any(|network| network.auto_start && !network.range_conflicts.is_empty())
{
return Self::RangeConflict;
}
if !report
.networks
.iter()
@@ -56,6 +65,7 @@ impl Health {
match self {
Self::Disconnected => "tsunagi: not connected",
Self::Connected => "tsunagi: connected",
Self::RangeConflict => "tsunagi: network range conflict; open to switch networks",
Self::ExitNode => "tsunagi: using an exit node",
}
}
@@ -108,6 +118,24 @@ mod tests {
}
}
#[test]
fn overlapping_networks_are_visible_even_while_other_peers_are_connected() {
let mut blocked = NetworkReport {
auto_start: true,
..Default::default()
};
blocked
.range_conflicts
.push(tsunagi::ipc::RangeConflictReport {
name: "other".into(),
network_id: "other-id".into(),
range: "10.90.0.0/16".into(),
});
let status = report(vec![network(2, ExitReport::default()), blocked]);
assert_eq!(Health::of(Some(&status)), Health::RangeConflict);
assert!(Health::RangeConflict.label().contains("switch networks"));
}
#[test]
fn no_agent_or_nobody_connected_is_the_red_state() {
assert_eq!(Health::of(None), Health::Disconnected);
+6 -3
View File
@@ -79,6 +79,8 @@ pub const WIREGUARD_OVERHEAD: u32 = 32;
/// Configuration of the WireGuard plugin.
#[derive(Debug, Clone)]
pub struct WireguardConfig {
/// Diagnostic protocol id when this end-to-end engine is reused by another carrier.
pub protocol: &'static str,
/// Directory for the plugin's own key store. Separate from agent state.
pub state_dir: PathBuf,
/// WireGuard keepalive, which keeps tunnels and their links warm.
@@ -100,6 +102,7 @@ impl WireguardConfig {
/// Creates a configuration rooted at `state_dir` with sensible defaults.
pub fn new(state_dir: impl Into<PathBuf>) -> Self {
Self {
protocol: WIREGUARD_PROTOCOL,
state_dir: state_dir.into(),
keepalive: Some(25),
mtu: DEFAULT_MTU,
@@ -394,7 +397,7 @@ impl Worker {
fn report(&self, network: NetworkId, reason: impl std::fmt::Display) {
tracing::warn!(network = %network.fmt_short(), %reason, "wireguard plugin error");
if let Some(context) = self.context.get() {
context.report_error(network, WIREGUARD_PROTOCOL, reason.to_string());
context.report_error(network, self.config.protocol, reason.to_string());
}
}
@@ -647,7 +650,7 @@ async fn run(worker: Arc<Worker>, mut commands: mpsc::Receiver<Command>) {
impl IpPlugin for WireguardPlugin {
fn protocol_id(&self) -> &str {
WIREGUARD_PROTOCOL
self.worker.config.protocol
}
fn protocol_version(&self) -> u16 {
@@ -686,7 +689,7 @@ impl IpPlugin for WireguardPlugin {
// Identity only. Where to send packets is the transport's business.
let announcement = WgAnnouncement::new(network, &state.key.public());
Ok(Some(PluginCapability {
protocol: WIREGUARD_PROTOCOL.to_string(),
protocol: self.worker.config.protocol.to_string(),
version: crate::announcement::ANNOUNCEMENT_VERSION,
enabled: true,
data: announcement.encode()?,
@@ -277,6 +277,7 @@ impl tsunagi::ipc::ReportSource for Control {
network_id: network_id.to_string(),
already_configured: already,
name_shared_with: None,
active: self.0.is_active(network_id).await,
})
})
}
+214 -3
View File
@@ -109,6 +109,30 @@ impl WgAgent {
.await
}
async fn spawn_auto(discovery: &SharedMemoryDiscovery, tag: &str) -> Self {
let dir = TempDir::new().unwrap();
let tuns = MemoryTunFactory::new();
let plugin = WireguardPlugin::open(
WireguardConfig::new(dir.path().join("wireguard"))
.with_reconcile(Duration::from_millis(20), Duration::from_millis(250)),
)
.await
.unwrap();
let agent = Agent::spawn(
config_with(dir.path(), discovery)
.with_interface(Arc::new(tuns.clone()), tag, 1280)
.with_plugin(plugin.clone() as Arc<dyn IpPlugin>),
)
.await
.unwrap();
Self {
dir,
agent,
plugin,
tuns,
}
}
/// Starts an agent with an explicit overlay range, or none at all.
async fn spawn_range(
discovery: &SharedMemoryDiscovery,
@@ -989,7 +1013,7 @@ async fn two_networks_share_one_interface_with_keys_and_ranges_of_their_own() {
let (name_b, secret_b) = network("wg-right");
let beta_range = Some("10.99.0.0/16".parse::<Ipv4Range>().unwrap());
let hub = WgAgent::spawn(&discovery, "th").await;
let hub = WgAgent::spawn_auto(&discovery, "th").await;
let left = WgAgent::spawn(&discovery, "tl").await;
let right = WgAgent::spawn_range(&discovery, "tr", beta_range).await;
@@ -1661,8 +1685,10 @@ async fn lan_broadcast_fanout_is_scoped_opt_in_and_does_not_reflood() {
bytes[28..].copy_from_slice(b"LAN GAME");
// UDP checksum zero is valid for IPv4.
let sum: u32 = bytes[..20]
.chunks_exact(2)
.map(|w| u16::from_be_bytes([w[0], w[1]]) as u32)
.as_chunks::<2>()
.0
.iter()
.map(|w| u16::from_be_bytes(*w) as u32)
.sum();
let checksum = !((sum & 0xffff) + (sum >> 16)) as u16;
bytes[10..12].copy_from_slice(&checksum.to_be_bytes());
@@ -1744,3 +1770,188 @@ async fn lan_broadcast_fanout_is_scoped_opt_in_and_does_not_reflood() {
agent.shutdown().await;
}
}
fn vip_segment(
source: Ipv4Addr,
destination: Ipv4Addr,
sport: u16,
dport: u16,
udp: bool,
flags: u8,
) -> Bytes {
fn sum(bytes: &[u8]) -> u16 {
let mut s: u32 = bytes
.chunks(2)
.map(|b| u32::from(u16::from_be_bytes([b[0], *b.get(1).unwrap_or(&0)])))
.sum();
while s >> 16 != 0 {
s = (s & 0xffff) + (s >> 16);
}
!(s as u16)
}
let mut p = vec![0; if udp { 36 } else { 40 }];
p[0] = 0x45;
let len = p.len() as u16;
p[2..4].copy_from_slice(&len.to_be_bytes());
p[8] = 64;
p[9] = if udp { 17 } else { 6 };
p[12..16].copy_from_slice(&source.octets());
p[16..20].copy_from_slice(&destination.octets());
p[20..22].copy_from_slice(&sport.to_be_bytes());
p[22..24].copy_from_slice(&dport.to_be_bytes());
if udp {
p[24..26].copy_from_slice(&16u16.to_be_bytes());
} else {
p[32] = 0x50;
p[33] = flags;
p[24..28].copy_from_slice(&123u32.to_be_bytes());
}
let mut pseudo = p[12..20].to_vec();
pseudo.extend([0, p[9]]);
pseudo.extend(((p.len() - 20) as u16).to_be_bytes());
pseudo.extend(&p[20..]);
let c = sum(&pseudo);
let offset = if udp { 26 } else { 36 };
p[offset..offset + 2].copy_from_slice(&c.to_be_bytes());
let c = sum(&p[..20]);
p[10..12].copy_from_slice(&c.to_be_bytes());
Bytes::from(p)
}
#[tokio::test]
async fn vip_tcp_and_udp_remap_ports_over_real_encryption_and_preserve_source_admission() {
use tsunagi::vip::{Backend, Listener, Protocol, VipConfig};
let discovery = SharedMemoryDiscovery::new();
let (name, secret) = network("vip-encrypted");
let a = WgAgent::spawn(&discovery, "vipa").await;
let b = WgAgent::spawn(&discovery, "vipb").await;
let net = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
a.wait_for_tunnels(net, 1).await;
b.wait_for_tunnels(net, 1).await;
let aa = a.overlay(net).await;
let ab = b.overlay(net).await;
let ta = a.tun(net).await;
let tb = b.tun(net).await;
let mut cfg = VipConfig::new("service");
for (protocol, port, backend) in [(Protocol::Tcp, 443, 6443), (Protocol::Udp, 53, 5353)] {
let mut l = Listener::new(protocol, port);
l.port_check = false;
l.backends.push(Backend {
peer: b.endpoint_id().to_string(),
port: backend,
});
cfg.listeners.push(l);
}
a.agent
.save_vip(net, cfg.clone(), None, false)
.await
.unwrap();
let vip = wait_until("data-plane probes measure the backend RTT", || async {
let status = a.agent.network_status(net).await.ok()?;
let v = status.vips.iter().find(|v| v.config.id == cfg.id)?;
(v.listeners.iter().all(|l| l.selected.is_some()))
.then_some(v.address?)
.filter(|_| v.listeners.len() == 2)
})
.await;
for (udp, port, target) in [(false, 443, 6443), (true, 53, 5353)] {
ta.push_from_os(vip_segment(aa, vip, 40000, port, udp, 2));
let received = tokio::time::timeout(Duration::from_secs(5), tb.pop_to_os())
.await
.unwrap()
.unwrap();
assert_eq!(&received[12..16], &aa.octets());
assert_eq!(&received[16..20], &ab.octets());
assert_eq!(u16::from_be_bytes([received[22], received[23]]), target);
let translated = u16::from_be_bytes([received[20], received[21]]);
tb.push_from_os(vip_segment(ab, aa, target, translated, udp, 0x12));
let reply = tokio::time::timeout(Duration::from_secs(5), ta.pop_to_os())
.await
.unwrap()
.unwrap();
assert_eq!(&reply[12..16], &vip.octets());
assert_eq!(u16::from_be_bytes([reply[20], reply[21]]), port);
assert_eq!(u16::from_be_bytes([reply[22], reply[23]]), 40000);
// A backend cannot send a VIP source over its authenticated tunnel.
let forged = vip_segment(vip, aa, port, 40000, udp, 0x12);
tb.push_from_os(forged);
}
wait_until("forged VIP sources are rejected before NAT", || async {
(a.agent.overlay()?.counters.wrong_source >= 2).then_some(())
})
.await;
a.shutdown().await;
b.shutdown().await;
}
#[tokio::test]
async fn vip_udp_reaches_another_backend_after_the_selected_agent_stops() {
use tsunagi::vip::{Backend, Listener, Protocol, VipConfig};
let discovery = SharedMemoryDiscovery::new();
let (name, secret) = network("vip-failover");
let a = WgAgent::spawn(&discovery, "vipfa").await;
let b = WgAgent::spawn(&discovery, "vipfb").await;
let c = WgAgent::spawn(&discovery, "vipfc").await;
let net = a.agent.join_network(&name, &secret).await.unwrap();
for node in [&b, &c] {
node.agent.join_network(&name, &secret).await.unwrap();
}
for node in [&a, &b, &c] {
node.wait_for_tunnels(net, 2).await;
}
let mut config = VipConfig::new("datagrams");
let mut listener = Listener::new(Protocol::Udp, 8888);
listener.backends = vec![
Backend {
peer: b.endpoint_id().to_string(),
port: 9000,
},
Backend {
peer: c.endpoint_id().to_string(),
port: 9001,
},
];
config.listeners.push(listener);
a.agent
.save_vip(net, config.clone(), None, false)
.await
.unwrap();
let first = wait_until("both UDP backends have measured paths", || async {
let s = a.agent.network_status(net).await.ok()?;
let v = s.vips.iter().find(|v| v.config.id == config.id)?;
let l = v.listeners.first()?;
(l.backends.iter().all(|b| b.rtt_us.is_some())).then_some((v.address?, l.selected.clone()?))
})
.await;
let (dead, alive) = if first.1.peer == b.endpoint_id().to_string() {
(b, c)
} else {
(c, b)
};
let address = a.overlay(net).await;
let tun = a.tun(net).await;
let dest = dead.tun(net).await;
tun.push_from_os(vip_segment(address, first.0, 41000, 8888, true, 0));
tokio::time::timeout(Duration::from_secs(5), dest.pop_to_os())
.await
.unwrap()
.unwrap();
dead.shutdown().await;
wait_until("the remaining backend becomes the next choice", || async {
let s = a.agent.network_status(net).await.ok()?;
let v = s.vips.iter().find(|v| v.config.id == config.id)?;
(v.listeners.first()?.selected.as_ref()?.peer == alive.endpoint_id().to_string())
.then_some(())
})
.await;
tun.push_from_os(vip_segment(address, first.0, 41000, 8888, true, 0));
let packet = tokio::time::timeout(Duration::from_secs(5), alive.tun(net).await.pop_to_os())
.await
.unwrap()
.unwrap();
assert_eq!(&packet[16..20], &alive.overlay(net).await.octets());
let view = a.agent.network_status(net).await.unwrap();
assert_eq!(view.vips[0].listeners[0].counters.failovers, 1);
a.shutdown().await;
alive.shutdown().await;
}
@@ -231,6 +231,9 @@ impl WireguardLink {
}
impl PacketLink for WireguardLink {
fn close(&self) {
WireguardLink::close(self);
}
fn network(&self) -> NetworkId {
self.network
}
+3
View File
@@ -14,6 +14,7 @@ path = "src/main.rs"
[dependencies]
tsunagi-core = { path = "../tsunagi-core", version = "0.1.0", features = ["tun-device", "dns-publish"] }
tsunagi-wireguard = { path = "../tsunagi-wireguard", version = "0.1.0" }
tsunagi-awg = { path = "../tsunagi-awg", version = "0.1.0" }
tsunagi-wg-quic = { path = "../tsunagi-wg-quic", version = "0.1.0" }
tsunagi-tcp-tls = { path = "../tsunagi-tcp-tls", version = "0.1.0" }
iroh.workspace = true
@@ -26,6 +27,8 @@ clap = { version = "4.5", features = ["derive", "env"] }
# colour is never written where it would show up as rubbish.
anstream = "1.0"
anstyle = "1.0"
serde_json = "1.0"
tempfile.workspace = true
# Local interface addresses, for the diagnostics in `status`.
netwatch = "0.19.3"
+653 -140
View File
File diff suppressed because it is too large. Load diff
+506
View File
@@ -0,0 +1,506 @@
//! Named virtual service administration; every mutation submits one complete draft.
use super::{PathArgs, control_socket, resolve_network_index};
use clap::{Args, Subcommand};
use std::path::PathBuf;
use tsunagi::ipc::{self, NetworkReport};
use tsunagi::vip::{Backend, Listener, Protocol, VipConfig, VipStatus};
type Result<T> = std::result::Result<T, Box<dyn std::error::Error>>;
#[derive(Debug, Args)]
#[command(
after_help = "Examples:\n tsng vip create lab k8s\n tsng vip port add lab k8s tcp/443 --backend cp1:6443 --backend cp2:7443\n tsng vip backend add lab k8s tcp/443 cp3:6443\n tsng vip status lab k8s --watch\n tsng vip edit lab k8s\n\nNames, full ids and unique id prefixes are accepted. Changes are shared with the network.\nTCP port checks default to on; UDP uses data-path probes and explicit ICMP errors."
)]
pub(crate) struct VipArgs {
#[command(flatten)]
paths: PathArgs,
/// Agent control socket; defaults to the running service.
#[arg(long, global = true, env = "TSUNAGI_CONTROL_SOCKET")]
control_socket: Option<PathBuf>,
/// Machine-readable configuration and local observations.
#[arg(long, global = true)]
json: bool,
#[command(subcommand)]
action: Action,
}
#[derive(Debug, Args)]
struct Target {
/// Network exact name, id or unique id prefix.
network: String,
/// VIP exact name, id or unique id prefix.
vip: String,
}
#[derive(Debug, Args)]
struct PortTarget {
#[command(flatten)]
target: Target,
/// Listener as tcp/443 or udp/53.
listener: String,
}
#[derive(Debug, Subcommand)]
enum Action {
/// List services in one network, or every configured network.
List { network: Option<String> },
/// Create a named VIP with an automatically reserved internal address.
Create { network: String, name: String },
/// Show the complete saved configuration and local status.
Show(Target),
/// Show local health, routing choices and cumulative traffic counters.
Status {
#[command(flatten)]
target: Target,
/// Refresh once a second until interrupted.
#[arg(long)]
watch: bool,
},
/// Edit the entire configuration in VISUAL/EDITOR and save it atomically.
Edit(Target),
/// Remove a VIP; existing flows drain until completion or idle expiry.
Remove(Target),
/// Add, configure or remove listeners independently of the VIP.
Port {
#[command(subcommand)]
action: PortAction,
},
/// Change a listener's destination nodes and application ports.
Backend {
#[command(subcommand)]
action: BackendAction,
},
}
#[derive(Debug, Subcommand)]
enum PortAction {
/// Add a listener, optionally with its complete initial backend set.
Add {
#[command(flatten)]
target: PortTarget,
/// Node name/id followed by :port. Repeat for multiple destinations.
#[arg(long)]
backend: Vec<String>,
},
/// Change check settings and flow idle expiry.
Set {
#[command(flatten)]
target: PortTarget,
/// Active TCP connect checks: on or off.
#[arg(long,value_parser=["on","off"])]
port_check: Option<String>,
/// Milliseconds between active checks (250..3600000).
#[arg(long)]
check_interval_ms: Option<u32>,
/// TCP check deadline in milliseconds (100..60000).
#[arg(long)]
check_timeout_ms: Option<u32>,
/// Consecutive failures before exclusion.
#[arg(long)]
failures: Option<u8>,
/// Consecutive successes before recovery.
#[arg(long)]
successes: Option<u8>,
/// Flow idle expiry in seconds (1..604800).
#[arg(long)]
idle_timeout: Option<u32>,
},
/// Remove a listener from new-flow selection; existing flows drain.
Remove(PortTarget),
}
#[derive(Debug, Subcommand)]
enum BackendAction {
/// Add a destination node and port to a listener.
Add {
#[command(flatten)]
target: PortTarget,
backend: String,
},
/// Remove a destination from new-flow selection.
Remove {
#[command(flatten)]
target: PortTarget,
backend: String,
},
}
impl Action {
fn network(&self) -> Option<&str> {
match self {
Self::List { network } => network.as_deref(),
Self::Create { network, .. } => Some(network),
Self::Show(t) | Self::Edit(t) | Self::Remove(t) | Self::Status { target: t, .. } => {
Some(&t.network)
}
Self::Port { action } => Some(
&match action {
PortAction::Add { target, .. }
| PortAction::Set { target, .. }
| PortAction::Remove(target) => target,
}
.target
.network,
),
Self::Backend { action } => Some(
&match action {
BackendAction::Add { target, .. } | BackendAction::Remove { target, .. } => {
target
}
}
.target
.network,
),
}
}
}
fn parse_listener(s: &str) -> Result<(Protocol, u16)> {
let (p, n) = s
.split_once('/')
.ok_or("listener must be tcp/PORT or udp/PORT")?;
let port = n.parse::<u16>()?;
if port == 0 {
return Err("port must not be zero".into());
}
Ok((p.parse()?, port))
}
fn backend(n: &NetworkReport, s: &str) -> Result<Backend> {
let (node, port) = s.rsplit_once(':').ok_or("backend must be NODE:PORT")?;
let peer = match ipc::resolve_node(n, node) {
Ok(member) => member.endpoint_id.clone(),
Err(error) => {
let matches = n
.members
.iter()
.any(|m| m.hostname.as_deref() == Some(node) || m.endpoint_id.starts_with(node));
if matches {
return Err(error.into());
}
node.parse::<iroh::EndpointId>()
.map_err(|_| error)?
.to_string()
}
};
let port = port.parse()?;
if port == 0 {
return Err("backend port must not be zero".into());
}
Ok(Backend { peer, port })
}
fn selected(n: &NetworkReport, t: &Target) -> Result<VipStatus> {
Ok(ipc::resolve_vip(n, &t.vip)?.clone())
}
fn listener<'a>(c: &'a mut VipConfig, target: &PortTarget) -> Result<&'a mut Listener> {
let (p, n) = parse_listener(&target.listener)?;
c.listeners
.iter_mut()
.find(|l| l.protocol == p && l.port == n)
.ok_or_else(|| format!("no listener {} on {}", target.listener, c.name).into())
}
fn name_of<'a>(n: &'a NetworkReport, peer: &'a str) -> &'a str {
n.members
.iter()
.find(|m| m.endpoint_id == peer)
.and_then(|m| m.hostname.as_deref())
.unwrap_or(peer)
}
fn show(n: &NetworkReport, v: &VipStatus, json: bool) -> Result<()> {
if json {
println!("{}", serde_json::to_string(v)?);
return Ok(());
}
println!(
"{}.{} {}",
v.config.name,
n.name,
v.error.as_deref().unwrap_or("configured")
);
for l in &v.listeners {
let choice = l
.selected
.as_ref()
.map(|b| format!("{}:{}", name_of(n, &b.peer), b.port))
.unwrap_or_else(|| "none".into());
println!(
" {}/{} next: {} flows: {} tx/rx: {}/{} bytes drops: {} UDP failovers: {}",
l.protocol,
l.port,
choice,
l.backends.iter().map(|b| b.active_flows).sum::<u32>(),
l.counters.tx_bytes,
l.counters.rx_bytes,
l.counters.dropped,
l.counters.failovers
);
for b in &l.backends {
let rtt = b
.rtt_us
.map(|n| format!("{:.2} ms", n as f64 / 1000.))
.unwrap_or_else(|| "unknown".into());
println!(
" {}:{} {} RTT {} active {}",
name_of(n, &b.backend.peer),
b.backend.port,
b.state,
rtt,
b.active_flows
);
}
}
Ok(())
}
pub(crate) async fn run(args: VipArgs) -> Result<()> {
let custom = args.paths.state_dir.is_some();
let paths = args.paths.resolve()?;
let socket = control_socket(&paths, args.control_socket.as_ref(), custom);
let report = ipc::request_status(&socket).await?;
if let Action::List { network: None } = &args.action {
for n in &report.networks {
for v in &n.vips {
show(n, v, args.json)?;
}
}
return Ok(());
}
let selector = args.action.network().ok_or("network is required")?;
let names: Vec<_> = report
.networks
.iter()
.map(|n| (n.network_id.clone(), n.name.clone()))
.collect();
let network = &report.networks[resolve_network_index(&names, selector)?];
if let Action::Status {
target,
watch: true,
} = &args.action
{
let mut previous: Option<(std::time::Instant, u64, u64)> = None;
loop {
let report = ipc::request_status(&socket).await?;
let n = report
.networks
.iter()
.find(|n| n.network_id == network.network_id)
.ok_or("network was removed")?;
let vip = selected(n, target)?;
let (tx, rx) = vip.listeners.iter().fold((0u64, 0u64), |a, l| {
(
a.0.saturating_add(l.counters.tx_bytes),
a.1.saturating_add(l.counters.rx_bytes),
)
});
let now = std::time::Instant::now();
if let Some((at, old_tx, old_rx)) = previous
&& !args.json
{
let secs = now.duration_since(at).as_secs_f64();
if secs > 0. {
println!(
"TX {:.1} KiB/s · RX {:.1} KiB/s",
tx.saturating_sub(old_tx) as f64 / secs / 1024.,
rx.saturating_sub(old_rx) as f64 / secs / 1024.
);
}
}
previous = Some((now, tx, rx));
show(n, &vip, args.json)?;
tokio::select! {_=tokio::signal::ctrl_c()=>break,_=tokio::time::sleep(std::time::Duration::from_secs(1))=>{}}
}
return Ok(());
}
let mut edited_file = None;
let (mut config, expected, delete) = match &args.action {
Action::List { .. } => {
for v in &network.vips {
show(network, v, args.json)?;
}
return Ok(());
}
Action::Show(t) | Action::Status { target: t, .. } => {
let v = selected(network, t)?;
show(network, &v, args.json)?;
if matches!(args.action, Action::Show(_)) && !args.json {
println!("{}", serde_json::to_string_pretty(&v.config)?);
}
return Ok(());
}
Action::Create { name, .. } => (VipConfig::new(name), None, false),
Action::Remove(t) => {
let v = selected(network, t)?;
(v.config, Some(v.revision), true)
}
Action::Edit(t) => {
let v = selected(network, t)?;
let mut file = tempfile::Builder::new()
.prefix("tsunagi-vip-")
.suffix(".json")
.tempfile()?;
use std::io::Write;
file.write_all(serde_json::to_string_pretty(&v.config)?.as_bytes())?;
file.flush()?;
let editor = std::env::var("VISUAL")
.or_else(|_| std::env::var("EDITOR"))
.unwrap_or_else(|_| "vi".into());
let mut words = editor.split_whitespace();
let program = words.next().ok_or("empty EDITOR")?;
let status = std::process::Command::new(program)
.args(words)
.arg(file.path())
.status()?;
if !status.success() {
return Err("editor failed; VIP unchanged".into());
}
let bytes = std::fs::read(file.path())?;
let cfg: VipConfig = match serde_json::from_slice(&bytes) {
Ok(c) => c,
Err(e) => {
let (_, path) = file.keep()?;
return Err(
format!("invalid draft: {e}; retained at {}", path.display()).into(),
);
}
};
if cfg.id != v.config.id {
return Err("VIP id cannot be edited".into());
}
edited_file = Some(file);
(cfg, Some(v.revision), false)
}
Action::Port { action } => {
let target = match action {
PortAction::Add { target, .. }
| PortAction::Set { target, .. }
| PortAction::Remove(target) => target,
};
let v = selected(network, &target.target)?;
let mut cfg = v.config;
match action {
PortAction::Add {
target,
backend: destinations,
} => {
let (p, port) = parse_listener(&target.listener)?;
let mut l = Listener::new(p, port);
l.backends = destinations
.iter()
.map(|b| backend(network, b))
.collect::<Result<_>>()?;
cfg.listeners.push(l);
}
PortAction::Remove(target) => {
let (p, port) = parse_listener(&target.listener)?;
listener(&mut cfg, target)?;
cfg.listeners.retain(|l| l.protocol != p || l.port != port);
}
PortAction::Set {
target,
port_check,
check_interval_ms,
check_timeout_ms,
failures,
successes,
idle_timeout,
} => {
let l = listener(&mut cfg, target)?;
if let Some(v) = port_check {
l.port_check = v == "on";
}
if let Some(v) = check_interval_ms {
l.check_interval_ms = *v;
}
if let Some(v) = check_timeout_ms {
l.check_timeout_ms = *v;
}
if let Some(v) = failures {
l.failure_threshold = *v;
}
if let Some(v) = successes {
l.recovery_threshold = *v;
}
if let Some(v) = idle_timeout {
l.idle_timeout_secs = *v;
}
}
}
(cfg, Some(v.revision), false)
}
Action::Backend { action } => {
let (target, destination, remove) = match action {
BackendAction::Add { target, backend } => (target, backend, false),
BackendAction::Remove { target, backend } => (target, backend, true),
};
let v = selected(network, &target.target)?;
let mut cfg = v.config;
let b = backend(network, destination)?;
let l = listener(&mut cfg, target)?;
if remove {
if !l.backends.contains(&b) {
return Err("backend is not configured".into());
}
l.backends.retain(|x| x != &b);
} else {
l.backends.push(b);
}
(cfg, Some(v.revision), false)
}
};
// Canonical order makes displayed drafts and shared state deterministic.
config.listeners.sort_by_key(|l| (l.protocol, l.port));
for l in &mut config.listeners {
l.backends.sort();
}
let id = config.id.clone();
let vips = match ipc::save_vip(&socket, &network.network_id, config, expected, delete).await {
Ok(vips) => vips,
Err(error) => {
if let Some(file) = edited_file {
let (_, path) = file.keep()?;
return Err(format!("{error}; draft retained at {}", path.display()).into());
}
return Err(error.into());
}
};
if delete {
if args.json {
println!("{{\"removed\":\"{id}\"}}");
} else {
println!("VIP removed; existing flows drain.");
}
} else if let Some(v) = vips.iter().find(|v| v.config.id == id) {
show(network, v, args.json)?;
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
use clap::Parser;
#[test]
fn vip_commands_parse_complete_listener_and_settings() {
let cli = super::super::Cli::try_parse_from([
"tsng",
"vip",
"port",
"add",
"lab",
"k8s",
"tcp/443",
"--backend",
"cp1:6443",
"--backend",
"cp2:7443",
]);
assert!(cli.is_ok(), "{cli:?}");
assert!(
super::super::Cli::try_parse_from([
"tsng",
"vip",
"port",
"set",
"lab",
"k8s",
"tcp/443",
"--port-check",
"off",
"--failures",
"3"
])
.is_ok()
);
assert!(parse_listener("tcp/0").is_err());
assert!(parse_listener("sctp/53").is_err());
}
}
+92
View File
@@ -345,3 +345,95 @@ fn opposing_dns_flags_are_rejected() {
assert!(!result.status.success());
assert!(String::from_utf8_lossy(&result.stderr).contains("cannot be used with"));
}
#[test]
fn vip_cli_creates_multiple_ports_updates_dns_and_rejects_partial_invalid_changes() {
let agent = start("vip-test", 15367);
let created = agent.run(&["vip", "create", "vip-test", "api", "--json"]);
assert!(
created.status.success(),
"{}",
String::from_utf8_lossy(&created.stderr)
);
let vip: tsunagi::vip::VipStatus = serde_json::from_slice(&created.stdout).unwrap();
assert!(vip.config.listeners.is_empty());
let reply = wait_for_answer("127.0.0.1:15367".parse().unwrap(), "api.vip-test");
assert!(has_answer(&reply));
let socket = tsunagi::ipc::control_socket_path(&agent.dir.path().join("state"));
let runtime = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.unwrap();
let status = runtime
.block_on(tsunagi::ipc::request_status(socket))
.unwrap();
let backend = format!("{}:6443", status.endpoint_id);
let added = agent.run(&[
"vip",
"port",
"add",
"vip-test",
"api",
"tcp/443",
"--backend",
&backend,
]);
assert!(
added.status.success(),
"{}",
String::from_utf8_lossy(&added.stderr)
);
assert!(
agent
.run(&["vip", "port", "add", "vip-test", "api", "udp/53"])
.status
.success()
);
assert!(
!agent
.run(&["vip", "port", "add", "vip-test", "api", "tcp/443"])
.status
.success()
);
assert!(
agent
.run(&[
"vip",
"port",
"set",
"vip-test",
"api",
"tcp/443",
"--port-check",
"off"
])
.status
.success()
);
let shown = agent.run(&["vip", "show", "vip-test", "api", "--json"]);
assert!(shown.status.success());
let shown: tsunagi::vip::VipStatus = serde_json::from_slice(&shown.stdout).unwrap();
assert_eq!(shown.config.listeners.len(), 2);
assert!(
!shown
.config
.listeners
.iter()
.find(|l| l.protocol == tsunagi::vip::Protocol::Tcp)
.unwrap()
.port_check
);
assert_eq!(shown.config.listeners[0].backends[0].port, 6443);
let removed = agent.run(&["vip", "remove", "vip-test", "api"]);
assert!(removed.status.success());
let deadline = Instant::now() + Duration::from_secs(10);
loop {
if let Some(reply) = query("127.0.0.1:15367".parse().unwrap(), "api.vip-test", TYPE::A)
&& !has_answer(&reply)
{
break;
}
assert!(Instant::now() < deadline);
std::thread::sleep(Duration::from_millis(25));
}
}
+435 -7
View File
@@ -6,7 +6,7 @@
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use std::time::Duration;
use std::time::{Duration, Instant};
use tempfile::TempDir;
@@ -50,8 +50,26 @@ fn read_networks(dir: &TempDir) -> Vec<tsunagi::storage::StoredNetwork> {
/// An agent with no network at all, the way a daemon is started before
/// anything has been decided.
fn start_bare(port: u16) -> Running {
start_bare_range(port, None)
}
fn start_bare_range(port: u16, range: Option<&str>) -> Running {
start_bare_config(port, range, None)
}
fn start_bare_config(port: u16, range: Option<&str>, protocols: Option<&str>) -> Running {
start_bare_peers(port, range, protocols, &[])
}
fn start_bare_peers(
port: u16,
range: Option<&str>,
protocols: Option<&str>,
peers: &[String],
) -> Running {
let dir = TempDir::new().unwrap();
let child = std::process::Command::new(env!("CARGO_BIN_EXE_tsng"))
let mut command = std::process::Command::new(env!("CARGO_BIN_EXE_tsng"));
command
.arg("up")
.arg("--state-dir")
.arg(dir.path().join("state"))
@@ -62,11 +80,37 @@ fn start_bare(port: u16) -> Running {
.arg(format!("127.0.0.1:{port}"))
.args(["--log", "error", "--status-interval", "0"])
.stdout(std::process::Stdio::null())
.stderr(std::process::Stdio::null())
.spawn()
.expect("the agent binary starts");
std::thread::sleep(Duration::from_secs(2));
Running { child, dir }
.stderr(std::process::Stdio::null());
if let Some(range) = range {
command.args(["--ipv4-range", range]);
}
if let Some(protocols) = protocols {
command.args(["--protocol", protocols]);
}
for peer in peers {
command.args(["--peer", peer]);
}
let child = command.spawn().expect("the agent binary starts");
let mut agent = Running { child, dir };
let runtime = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.unwrap();
let socket = tsunagi::ipc::control_socket_path(&agent.dir.path().join("state"));
runtime.block_on(async {
tokio::time::timeout(Duration::from_secs(30), async {
while !tsunagi::ipc::is_serving(&socket).await {
assert!(
agent.child.try_wait().unwrap().is_none(),
"agent exited before becoming ready"
);
tokio::time::sleep(Duration::from_millis(25)).await;
}
})
.await
.expect("agent becomes ready");
});
agent
}
/// An agent already in one network, the way most of these start.
@@ -87,6 +131,254 @@ fn start(network: &str, port: u16) -> Running {
agent
}
#[test]
fn cli_forgets_only_the_selected_offline_claim_over_real_control() {
let cleaner = start("forget-cli", 0);
let runtime = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.unwrap();
let socket = tsunagi::ipc::control_socket_path(&cleaner.dir.path().join("state"));
let read = || {
runtime
.block_on(tsunagi::ipc::request_status(&socket))
.unwrap()
};
let before = read();
let bootstrap = format!("{}@{}", before.endpoint_id, before.bound_sockets[0]);
let target = start_bare_peers(0, None, Some("none"), &[bootstrap]);
assert!(
target
.run(&["id", "hostname", "old-cli-node"])
.status
.success()
);
assert!(
target
.run(&[
"join",
"--network",
"forget-cli",
"--secret",
"a-secret-for-the-cli-test"
])
.status
.success()
);
runtime.block_on(async {
tokio::time::timeout(Duration::from_secs(30), async {
loop {
let status = tsunagi::ipc::request_status(&socket).await.unwrap();
if status.networks[0]
.members
.iter()
.any(|m| m.hostname.as_deref() == Some("old-cli-node") && m.online)
{
break;
}
tokio::time::sleep(Duration::from_millis(25)).await;
}
})
.await
.unwrap();
});
assert!(
!cleaner
.run(&["network", "forget-node", "forget-cli", "old-cli-node"])
.status
.success()
);
assert!(
!cleaner
.run(&["network", "forget-node", "forget-cli", &before.endpoint_id])
.status
.success()
);
assert!(
target
.run(&["network", "stop", "forget-cli"])
.status
.success()
);
runtime.block_on(async {
tokio::time::timeout(Duration::from_secs(30), async {
loop {
let status = tsunagi::ipc::request_status(&socket).await.unwrap();
if status.networks[0]
.members
.iter()
.any(|m| m.hostname.as_deref() == Some("old-cli-node") && !m.online)
{
break;
}
tokio::time::sleep(Duration::from_millis(25)).await;
}
})
.await
.unwrap();
});
let forgotten = cleaner.run(&["network", "forget-node", "forget-cli", "old-cli-node"]);
assert!(
forgotten.status.success(),
"{}",
String::from_utf8_lossy(&forgotten.stderr)
);
assert!(String::from_utf8_lossy(&forgotten.stdout).contains("newer signed claim can return"));
assert!(
!read().networks[0]
.members
.iter()
.any(|m| m.hostname.as_deref() == Some("old-cli-node"))
);
assert!(
!cleaner
.run(&["network", "forget-node", "forget-cli", "old-cli-node"])
.status
.success()
);
}
#[test]
fn protocols_absent_from_startup_can_be_enabled_and_changed_offline() {
let mut agent = start_bare_config(0, None, Some("none"));
assert!(
agent
.run(&[
"join",
"--network",
"from-control-only",
"--secret",
"a-control-only-test-secret"
])
.status
.success()
);
let initial = agent.run(&["protocols", "from-control-only"]);
assert!(initial.status.success());
assert!(String::from_utf8_lossy(&initial.stdout).contains("protocols none"));
let enabled = agent.run(&["protocols", "from-control-only", "awg"]);
assert!(
enabled.status.success(),
"{}",
String::from_utf8_lossy(&enabled.stderr)
);
assert_eq!(
read_networks(&agent.dir)[0].protocols,
Some(vec!["awg".into()])
);
agent.child.kill().unwrap();
agent.child.wait().unwrap();
let saved = agent.run(&["protocols", "from-control-only", "wg-quic,awg"]);
assert!(
saved.status.success(),
"{}",
String::from_utf8_lossy(&saved.stderr)
);
assert!(String::from_utf8_lossy(&saved.stdout).contains("agent stopped"));
assert_eq!(
read_networks(&agent.dir)[0].protocols,
Some(vec!["wg-quic".into(), "awg".into()])
);
}
#[test]
fn protocols_are_changed_live_per_network_and_saved_without_changing_the_service() {
let agent = start("protocol-choice", 0);
assert!(
agent
.run(&[
"join",
"--network",
"other-choice",
"--secret",
"other-secret-for-cli-test"
])
.status
.success()
);
let before = read_networks(&agent.dir);
let other_id = before
.iter()
.find(|n| n.name.as_str() == "other-choice")
.unwrap()
.network_id;
let changed = agent.run(&["protocols", "protocol-choice", "awg,wg-quic"]);
assert!(
changed.status.success(),
"{}",
String::from_utf8_lossy(&changed.stderr)
);
let shown = agent.run(&["protocols", "protocol-choice"]);
assert!(shown.status.success());
assert!(String::from_utf8_lossy(&shown.stdout).contains("awg,wg-quic"));
let saved = read_networks(&agent.dir);
assert_eq!(
saved
.iter()
.find(|n| n.name.as_str() == "protocol-choice")
.unwrap()
.protocols,
Some(vec!["awg".into(), "wg-quic".into()])
);
assert_eq!(
saved
.iter()
.find(|n| n.network_id == other_id)
.unwrap()
.protocols,
before
.iter()
.find(|n| n.network_id == other_id)
.unwrap()
.protocols
);
for invalid in ["unknown", "awg,awg", "none,awg"] {
assert!(
!agent
.run(&["protocols", "protocol-choice", invalid])
.status
.success()
);
}
assert!(!agent.run(&["protocols", "missing", "awg"]).status.success());
let disabled = agent.run(&["protocols", "protocol-choice", "none"]);
assert!(
disabled.status.success(),
"{}",
String::from_utf8_lossy(&disabled.stderr)
);
assert_eq!(
read_networks(&agent.dir)
.iter()
.find(|n| n.name.as_str() == "protocol-choice")
.unwrap()
.protocols,
Some(Vec::new())
);
assert!(
agent
.run(&["network", "stop", "protocol-choice"])
.status
.success()
);
assert!(
agent
.run(&["protocols", "protocol-choice", "wg-quic"])
.status
.success()
);
assert!(
agent
.run(&["network", "start", "protocol-choice"])
.status
.success()
);
assert!(
String::from_utf8_lossy(&agent.run(&["protocols", "protocol-choice"]).stdout)
.contains("wg-quic")
);
}
#[test]
fn joining_with_no_secret_makes_one_and_prints_what_to_send() {
// The hurry case: a network with somebody, for as long as it is needed,
@@ -461,3 +753,139 @@ fn an_exit_node_is_offered_per_network_only_when_asked_and_nobody_offers_by_defa
String::from_utf8_lossy(&unknown.stderr)
);
}
#[test]
fn overlapping_networks_are_reported_and_cli_start_switches_then_resumes() {
let agent = start_bare_range(0, Some("10.90.0.0/16"));
for name in ["resident", "games"] {
let joined = agent.run(&["join", "-n", name, "-s", "a-secret-for-the-conflict-test"]);
assert!(
joined.status.success(),
"{}",
String::from_utf8_lossy(&joined.stderr)
);
if name == "games" {
let text = String::from_utf8_lossy(&joined.stdout);
assert!(
text.contains("stopped because its range overlaps"),
"{text}"
);
assert!(!text.contains("it is running"), "{text}");
}
}
let runtime = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.unwrap();
let socket = tsunagi::ipc::control_socket_path(&agent.dir.path().join("state"));
let snapshot = || {
runtime
.block_on(tsunagi::ipc::request_status(&socket))
.unwrap()
};
let before = snapshot();
let resident = before
.networks
.iter()
.find(|n| n.name == "resident")
.unwrap();
let games = before.networks.iter().find(|n| n.name == "games").unwrap();
assert!(resident.active);
assert!(!games.active);
assert_eq!(games.range_conflicts[0].network_id, resident.network_id);
assert_eq!(games.range_conflicts[0].name, "resident");
for args in [vec!["status"], vec!["network"]] {
let result = agent.run(&args);
let text = String::from_utf8_lossy(&result.stdout);
assert!(text.contains("range conflict"), "{text}");
assert!(text.contains(&resident.network_id), "{text}");
assert!(text.contains("tsng network start"), "{text}");
}
let switched = agent.run(&["network", "start", "games"]);
assert!(
switched.status.success(),
"{}",
String::from_utf8_lossy(&switched.stderr)
);
let text = String::from_utf8_lossy(&switched.stdout);
assert!(
text.contains("started `games`") && text.contains("suspended `resident`"),
"{text}"
);
let after = snapshot();
assert!(
after
.networks
.iter()
.find(|n| n.name == "games")
.unwrap()
.active
);
assert!(
!after
.networks
.iter()
.find(|n| n.name == "resident")
.unwrap()
.active
);
assert!(agent.run(&["network", "stop", "games"]).status.success());
assert!(
snapshot()
.networks
.iter()
.find(|n| n.name == "resident")
.unwrap()
.active
);
assert!(agent.run(&["network", "start", "games"]).status.success());
let stop_suspended = agent.run(&["network", "stop", "resident"]);
assert!(stop_suspended.status.success());
assert!(String::from_utf8_lossy(&stop_suspended.stdout).contains("stopped `resident`"));
assert!(agent.run(&["network", "stop", "games"]).status.success());
assert!(
snapshot()
.networks
.iter()
.all(|n| !n.active && !n.auto_start)
);
}
#[test]
fn cli_defaults_derive_each_networks_range() {
let agent = start_bare(0);
for name in ["derived-one", "derived-two"] {
assert!(
agent
.run(&["join", "-n", name, "-s", "a-secret-for-the-derived-test"])
.status
.success()
);
}
let runtime = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.unwrap();
let socket = tsunagi::ipc::control_socket_path(&agent.dir.path().join("state"));
let deadline = Instant::now() + Duration::from_secs(30);
loop {
let report = runtime
.block_on(tsunagi::ipc::request_status(&socket))
.unwrap();
if report.networks.iter().all(|n| n.range.is_some()) {
assert_eq!(report.networks.len(), 2);
for network in report.networks {
let id = network.network_id.parse().unwrap();
assert!(network.active);
assert_eq!(
network.range,
Some(tsunagi::state::derived_ipv4_range(id).to_string())
);
}
break;
}
assert!(Instant::now() < deadline, "networks allocate ranges");
std::thread::sleep(Duration::from_millis(25));
}
}
+131
View File
@@ -0,0 +1,131 @@
# Pairwise Amnezia transport
The CLI's default carrier preference is `wg,awg,wg-quic,tcp-tls`. To select
AWG explicitly:
```sh
tsng up --protocol awg
tsng up --protocol wg,awg,wg-quic,tcp-tls
```
The startup list is the initial per-network preference. A saved network choice
overrides it and can enable any built-in protocol at runtime:
```sh
tsng protocols home awg,wg-quic
tsng protocols home awg
```
Changes are local to this device, persist across restarts, and retire existing
data links and end-to-end channels for that network before renegotiation.
Authenticated control sessions, membership, addresses and other networks stay
live. Removing every protocol (`none` in the CLI, all checkboxes off in the tray)
leaves the network control-only. This is different from automatic carrier
fallback, which preserves the selected end-to-end channel.
`awg` is Tsunagi AWG wire version 1. Its packet masking engine is adapted from
[RunaRoo/AmneziaWG-RS](https://github.com/RunaRoo/AmneziaWG-RS), with the pinned
source and license recorded in
[UPSTREAM.md](../crates/tsunagi-awg/UPSTREAM.md). It uses the existing boringtun
Noise/WireGuard implementation instead of upstream's Node, global executors
and OS interface machinery. Compatibility with ordinary Amnezia servers is
not implemented: this carrier connects authenticated Tsunagi members.
## Negotiation and lifetime
There is one independent UDP socket and cancellable packet-engine task per
`(network, peer)` pair. The smaller endpoint id initiates, following the same
rule as other data carriers. Both sides must announce enabled AWG wire version
1. An authenticated iroh control session is required throughout; AWG supplies
no alternate discovery or membership mechanism.
Every `open` generates new masking parameters, a header-protection key, an
outer WireGuard key pair and a PSK. The initiator sends a private `Offer` with
a random attempt id, its public key, candidate socket addresses and the shared
profile. The responder validates it, creates its own fresh key pair and socket,
and returns an `Answer` naming that attempt. Both send masked handshakes and
encrypted delivery probes to the candidates. Only authenticated WireGuard
**data**, not a STUN response or an incoming UDP address alone, verifies a path.
Negotiation payloads are bounded at 4096 bytes (a caller can lower the limit),
redacted in Debug, excluded from public message events and erased on drop.
They are never capabilities, signed records, cache entries or exported status.
At most 128 pairs are admitted per transport, with bounded command/data queues,
at most 32 candidates per attempt and finite establishment deadlines.
Stale answers and duplicate offers are rejected; a new attempt replaces and
closes an older pair engine. Cancellation, peer departure, network deactivation
and shutdown retire sockets without waiting for a remote reader.
A process restart, a failed setup, or a complete fallback cycle returning to AWG
starts fresh negotiation. Normal WireGuard rekeying within a live attempt keeps
that attempt's masking profile. A failed pair does not reconfigure other pairs
or other networks. A full mesh of 15 members uses 14 engines per member and 105
independent pairs in total; it does not create additional system interfaces.
## Masking
Each profile randomizes disjoint H1–H4 header ranges, a 12–48-byte S1–S4 prefix,
a ChaCha20 header-protection key, handshake trailers, junk packet count/lengths,
and extra content padding. Prefix lengths are equal within one profile so
header detection has one offset. A fresh random nonce is in every packet's
prefix; handshake headers and transport headers/counters are protected.
The standard WireGuard header is restored before boringtun authenticates a
message. This preserves boringtun's standard MAC calculations, which is another
reason this is a Tsunagi-specific AWG wire version.
Before a handshake, an I1 signature packet is generated from either a complete
DNS query or SIP OPTIONS template, with fresh identifiers and consistent length
fields, followed by random junk packets. I2–I5 are unused in this version.
The templates are structural generators, not replayed captures. QUIC imitation
is not selected. Masking can change DPI classification; it is not a promise of
indistinguishability or that any particular firewall will admit it.
## Packets, NAT and fallback
The pair engine carries opaque relay envelopes inside authenticated outer WG
packets. It fragments those envelopes into bounded chunks and expires incomplete
assemblies; a logical datagram is bounded at 64 KiB and an outer UDP packet is
at most 1200 bytes. The encapsulated IPv4 headers are internal framing only: no interface or OS address is created for a
pair. Overlay addresses, routing, ownership checks and end-to-end encryption
remain independent. A relay decrypts its own adjacent AWG carrier to forward
opaque end-to-end ciphertext, never the destination's IP packet.
Local IPv4 candidates and simultaneous encrypted probes provide direct UDP
connectivity. In `direct`/`relay` reach mode the CLI also uses
`stun.l.google.com:19302` to discover a server-reflexive address on **this pair's
socket**, with a 1.2-second bound. A STUN result is only a candidate. Local reach
mode makes no STUN request. Configure or disable it with
`-o awg:stun=HOST:PORT` or `-o awg:stun=off`. Failed discovery still permits local
candidates. IPv6-only paths, symmetric NAT and networks blocking arbitrary UDP
may fail AWG and fall back; there is no AWG-over-QUIC substitution or AWG TURN
relay hidden behind its name.
Data delivery checks go through the actual end-to-end encrypted tunnel, once
per second, independently of control-plane RTT. Requests and replies bind a
fresh private marker/key to the sender's endpoint id. Unanswered requests expire
after three seconds. After a 20-second warmup, at least 15 completed observations
and at least 60% loss in the last 20 observations retire a carrier. Probes are
consumed locally after source/destination admission and never enter the TUN.
AWG additionally checks encrypted outer delivery throughout the pair lifetime.
Setup failure or persistent runtime loss advances to the next mutually supported
carrier in the initiator's order. Earlier expired cooldowns do not jump ahead of
untried carriers. Retry cooldowns remain bounded; reaching AWG again negotiates
new parameters. Control sessions and the logical end-to-end tunnel survive a
carrier change. Status names the end-to-end protocol and the actual carrier
separately, so `wg` over `awg` is a valid result.
Control wire version 6 introduces the private transport message. All members
must upgrade together; device identities, network ids, handshake transcript
encoding, signed state and stored network configuration are unchanged. Data
ALPN remains `tsunagi/data/5`; the relay envelope is unchanged.
## Verification
The default rootless, offline tests use real iroh endpoints, membership proofs,
SQLite, UDP sockets, masked WireGuard handshakes, end-to-end encryption and
memory TUNs. They cover independent three-member pairs, full-size bidirectional
packets, restart freshness/socket release, 80% runtime datagram loss, fallback
to real iroh and fresh negotiation on retry, plus malformed profiles/messages,
role checks, probe forgery and STUN response parsing. These tests do not establish
DPI effectiveness, WAN throughput, public NAT traversal or per-pair memory cost.
+8 -2
View File
@@ -20,8 +20,9 @@ network secret. The core moves an opaque, bounded payload and never parses it.
a `PacketLink` — an authenticated, unreliable datagram channel to one peer for
one protocol — and never learn how it is carried.
The separation between the two planes is **logical, not physical**. Both ride
on iroh, on different ALPNs and different connections. That is deliberate:
The separation between the two planes is **logical, not physical**. The control plane uses
iroh; the data plane can use iroh on its own ALPN and connection, or independent
UDP/TLS carriers. That is deliberate:
iroh's whole value is hole punching a direct path between peers behind NAT,
with a relay as fallback, and a data plane that refused to use it would have to
reimplement all of it. What the separation buys is that `proto` knows nothing
@@ -32,6 +33,10 @@ A plugin talks to the core through three narrow hooks — `on_network_activated`
a `PluginContext` for re-announcements and error reports, and a bounded
`shutdown` — so the core never learns anything protocol-specific.
Private pair negotiation uses a bounded transport sender supplied by the core.
Protocol crates do not encode or parse control messages and never receive the
network secret. No pair profiles enter signed state.
A data plane failure never stops the daemon: the control plane keeps running
and the agent stays manageable.
@@ -68,6 +73,7 @@ And in crates of their own:
| crate | responsibility |
|---|---|
| `tsunagi-awg` | ephemeral pairwise Amnezia masking, private negotiation and independently encrypted UDP carriers; reuses the end-to-end WireGuard engine |
| `tsunagi-wg-quic` | the `wg-quic` protocol: its keys, its announcement, its tunnels |
| `tsunagi` | the command line agent, built as the `tsng` executable: the only place that owns a runtime, a logger and signals |
+30 -6
View File
@@ -43,12 +43,36 @@ be an exit node in one network and not in another.
`tsunagi-exit:<interface>:<range>` and are removed when the offer ends, the
network stops, and when the agent exits. Only networks that offer have
rules, so the others' traffic is not forwarded.
- **Client:** policy routing, with no change to the main table. Table `28787`
holds `default dev <interface>`, and three rules sit after Tailscale's:
`5280 uidrange <agent uid> lookup main` (the agent's own connections never
go through the tunnel, which would be a loop), `5290 lookup main
suppress_prefixlength 0` (LAN and other specific routes keep working) and
`5300 lookup 28787`.
- **Client:** policy routing, with no change to the main table. Table `29811`
holds `default dev <interface>`. Rules `5205 fwmark 0x40000000/0x40000000
lookup main` and `5206 uidrange <agent uid> lookup main` keep the agent's
sockets and their replies outside the tunnel, before Tailscale's rules.
`5290 lookup main suppress_prefixlength 0` preserves LAN and other specific
routes, and `5300 lookup 29811` directs other traffic through the exit.
The exit table is `29811` (`0x7473`).
- **Linux reverse-path filtering:** only the agent's dedicated UID is marked
in mangle `OUTPUT`. Its own bit, `0x40000000`, is saved to conntrack and
restored in `PREROUTING` before the host firewall checks reply routing.
Other mark bits, including Tailscale's `0xff0000` mask, are preserved. The
rules are tagged `tsunagi-exit-underlay:<interface>` and never accept packets
around the host firewall. IPv4 and IPv6 protection is installed before exit
routes; routes are removed before protection, including rollback on failure.
Locally owned TCP listeners and bound UDP sockets also protect the first
incoming packet, before any outgoing packet has created a connection mark.
Socket addresses and ports
come from local kernel socket tables and match an actual socket; remote
announcements never supply these firewall arguments.
The socket set is refreshed before private pair negotiation is sent and on
discovery ticks, so fresh AWG ports are included. An unchanged set causes no
firewall writes; a socket-refresh failure leaves active exit routes in place.
Linux needs both `iptables` and `ip6tables` when IPv6 is present.
- **Kernel strict `rp_filter`:** if an interface uses strict mode (`1`), the
kernel must include marks in reverse lookups: set
`net.ipv4.conf.all.src_valid_mark=1`, or enable it on those interfaces.
The agent checks this before installing exit routes and reports an actionable
error if marks would be ignored. It never disables `rp_filter` or changes
host sysctls. Netfilter strict reverse checks also need their mark-aware
form, such as `rpfilter --validmark`, which NixOS already uses.
- Packets for addresses outside every overlay range are sent to the exit node
by the router; the exit node's replies are accepted from any source. A member
that is not your exit node cannot do that.
+16 -4
View File
@@ -9,16 +9,26 @@ Two versions exist and are independent:
- **Identity scheme**, `tsunagi-network-id-v1`. Frozen. Changing it creates a
different network space for the same name and secret.
- **Control protocol**, ALPN `tsunagi/ctrl/5`, `PROTOCOL_VERSION = 5`.
- **Control protocol**, ALPN `tsunagi/ctrl/8`, `PROTOCOL_VERSION = 8`.
Upgrading the crate or bumping the control protocol must never change an
existing `NetworkId`.
Control version 8 adds signed, multi-writer VIP field snapshots to version 7's
node retirements. Data ALPN remains `tsunagi/data/5`. All members must upgrade
together. Network ids, derivation labels, handshake encoding and existing record
signatures are unchanged. Local CLI/tray IPC is version 22; restart the agent
with matching clients. SQLite schema 10 adds `vip_fields`, preserving identity,
network configuration, signed node records and retirement proofs.
See [virtual services](vip.md), [state sync](sync-model.md) and [AWG](amnezia.md).
## Canonical encoding
Everything that is hashed or MAC'd uses length-prefixed fields, written here as
`LP(x) = u32_be(len(x)) || x`. String concatenation is never used, so no two
different field splits can produce the same bytes.
Network identity derivation and the membership transcript use length-prefixed
fields, written here as `LP(x) = u32_be(len(x)) || x`. String concatenation is
never used, so different field splits cannot produce the same bytes. VIP field
signatures use their separate domain and canonical postcard encoding described
in [virtual services](vip.md#shared-state-and-addresses).
## Network space identity
@@ -230,9 +240,11 @@ not affect other networks.
| message | meaning |
|---|---|
| `Announce { hostname, capabilities, broadcast }` | this agent's hostname, IP-plugin capabilities and local broadcast participation |
| `Transport { protocol, payload }` | private, bounded per-peer transport negotiation; never state or a public message event |
| `Ping { seq, payload }` | small request used to verify the exchange |
| `Pong { seq, payload }` | the echoed reply |
| `State { records }` | a snapshot of signed records, merged into what the receiver holds |
| `Retirements { markers }` | signed cleanup markers with the target's original signed claim as proof |
| `Peers { peers }` | unverified member address hints, requiring their own handshake |
| `Reachable { links }` | sender's current direct data links, by kind of transport |
| `Bye { reason }` | graceful goodbye; not a revocation of anything |
+9 -1
View File
@@ -96,6 +96,14 @@ topology change can move a flow; datagrams remain unreliable and unordered,
and there is no promise to preserve order across a physical path failure.
Logical peer links and end-to-end encryption state survive these changes.
## Carrier health and switching
Carrier setup failures and persistent encrypted delivery loss advance the
initiator's ordered fallback cycle. Logical end-to-end links survive while
physical links are retired explicitly. The AWG carrier negotiates a fresh
private pair profile on every reentry, without changing topology's durable
state. See [amnezia.md](amnezia.md) for thresholds and bounds.
## Verification and measurement
The default offline suite exercises four real agents, iroh transports and
@@ -116,5 +124,5 @@ counts sends without storing frames. Results are CPU forwarding cost, **not**
end-to-end network latency or VPN throughput; encryption, fragmentation,
sockets, congestion and scheduling contribute separately.
Wire compatibility: control ALPN `tsunagi/ctrl/5`, data ALPN `tsunagi/data/5`.
Wire compatibility: control ALPN `tsunagi/ctrl/8`, data ALPN `tsunagi/data/5`.
Upgrade every participant together; saved identities and network state persist.
+46 -27
View File
@@ -1,18 +1,9 @@
# Planned state synchronisation
# State synchronisation
**The first slice of this model is now implemented**, in `crates/tsunagi-core/src/state/`, and is
used for one thing: IPv4 overlay addresses. What follows describes the whole
model; the section at the end says exactly which parts exist.
The proof of concept still exchanges hostname and capability announcements
over live sessions and keeps no replicated history for those, which is why
WireGuard peer *membership* remains session-scoped even though a peer's
*address* no longer is. This file records the intended direction so the module
boundaries in [architecture.md](architecture.md) stay compatible with it, and so
nobody mistakes the current announcements for synchronisation.
There is no fake "ready CRDT" here and no snapshots that are not actually
verified.
Signed claims in `crates/tsunagi-core/src/state/` preserve IPv4 reservations and
hostnames across an absence. Signed releases and offline-node cleanup markers
free those reservations durably. Capabilities, links and authenticated sessions
remain volatile; a stored claim never authenticates a live connection.
## The problem
@@ -75,6 +66,39 @@ Turning a computer off is not a revocation of its hostname and does not remove
the participant. Revocations are not dropped merely because they are old, and
no acknowledgement from offline peers is required to keep working.
## Offline-node cleanup
Any authenticated member can sign a `SignedRetirement` for another member's
observed `Claim`. It includes the original signed claim as proof and uses the
separate `tsunagi-node-retirement/v1` signing domain. Both signatures and both
network scopes are checked. No owner, vote or quorum is involved, and no member
can fabricate the target's release or invent a future version cutoff.
The strongest observed target version wins; ties use a deterministic canonical
byte order. Claims at or below that version do not occupy an address, hostname
or visible member row. Original signed records remain stored, including their
range, so cleanup does not renumber the network. Markers remain even after a
newer claim returns: absence from a snapshot is never deletion and replaying an
old snapshot cannot restore the retired claim.
The local `Agent::forget_node` API checks the exact observed version again and
refuses self, online nodes, the selected exit node and missing/changed claims.
CLI and tray share this API. A returning live node can sign a newer claim and
allocate against the current holders; it resets the old hostname priority, so
it cannot reclaim a reused name merely by keeping its old timestamp. Cleanup
is not a permanent identity ban or a change of secret.
The marker and requester's own author counter commit in one SQLite transaction
before publication. Receivers persist a verified marker before applying it.
Invalid proofs are rejected individually. Periodic bounded marker batches
rotate through each peer's send queue, including when a queue fills, so a
replica that missed an update can catch up without the requester being online.
One strongest marker per target is retained, with a default limit of 1024 per
network and 1024 bytes per encoded marker. New targets beyond that limit are
refused rather than evicting resurrection protection. Corrupt mandatory marker
storage or a configured limit smaller than the stored set is an explicit error.
## Storage requirement this creates
When signed records land, writing the event and bumping the author's own
@@ -104,9 +128,9 @@ Implemented, in `crates/tsunagi-core/src/state/`:
* the merge rules above: higher version wins, an older version never rolls
back a newer one, duplicates are idempotent, a same-version conflict is
resolved identically on every replica and reported;
* a release tombstone, which merges correctly and is not undone by a replica
that has not heard of it — though nothing emits one yet, so freeing an
address still means forgetting the network;
* a release tombstone, emitted when leaving or replacing the device key and
not undone by a replica that has not heard of it;
* durable hostnames and equal-rights offline-node retirement described above;
* persistence in `state.sqlite`, with the record and the author's own version
counter committed in **one transaction before the record is announced**;
* distribution as a `State` control message, merged into what the receiver
@@ -114,21 +138,16 @@ Implemented, in `crates/tsunagi-core/src/state/`:
* allocation of a free IPv4 address against what everybody else holds, which
is what makes an address stable across an absence.
Deliberately not implemented: compaction, revoking a whole author, record
types beyond addressing, and any bound on how large a snapshot may grow
beyond the per-message limit.
Deliberately not implemented: compaction and permanent author revocation.
Claims have a per-message bound; retirement history also has a per-network bound.
## Future tests
These are **not implemented and must not be reported as passing**:
- snapshot merge against the rules above, including conflicting same-version
records;
- revocation propagation and resistance to resurrection after compaction;
- resistance to resurrection after compaction (compaction is not implemented);
- long network partitions and rejoin after an extended absence;
- hostname rename with atomic revoke-and-announce;
- recovery of a returning participant from a replica that is not the author;
- NAT traversal and hole punching between real hosts;
- relay fallback behaviour against a self-hosted relay;
- multi-process and multi-host deployment, as opposed to several library
instances inside one test process.
- multi-host deployment. CLI tests do run separate local agent processes;
library tests run several independent instances inside one process.
+77 -4
View File
@@ -98,6 +98,13 @@ a batch does not stop the rest, and allocation is deterministic, spread out,
walks past everything taken and reports a full range instead of handing out a
duplicate.
`tests/range_conflicts.rs` checks isolated first members choosing the same
range, persistence across restart, preservation of legacy ranges, exclusion
of equal and nested ranges, late authenticated conflicts, switching all
blockers, automatic resume, cancelling a suspended network's resume and
concurrent switch requests. These use real agents, loopback handshakes and
SQLite; they do not change host network settings.
`tests/local_control.rs` covers the local control socket end to end: a client
asking a running agent for status over a real Unix socket, joining and
leaving a network through it, a leftover socket file being replaced while a
@@ -157,10 +164,9 @@ paused clock. The public Mainline/relay smoke test is ignored by default.
## Not covered, and not claimed to be
Listed in [sync-model.md](sync-model.md#future-tests): snapshots, revocations,
long partitions, hostname renames, recovery of a returning participant, NAT
traversal, relay fallback, and multi-process or multi-host deployment. None of
these are implemented, and none are marked as passing.
Listed in [sync-model.md](sync-model.md#future-tests): compaction, long partitions,
NAT traversal, relay fallback and multi-host deployment. These scenarios are
not established by the default loopback suite.
## Debugging a test
@@ -170,3 +176,70 @@ TSUNAGI_TEST_LOG=tsunagi=debug cargo test --test multi_peer -- --nocapture
The library never installs a global subscriber; the harness opts in only when
that variable is set.
The `tsunagi-awg` suite runs real pair negotiation and masked UDP WireGuard
engines with memory TUNs. A loss wrapper drops datagrams at the carrier boundary
without substituting iroh, membership, encryption or SQLite. It checks runtime
loss detection, fallback and fresh profiles, in addition to restart and input
validation. Tests are offline; public STUN/DPI/NAT effectiveness needs separate
real-network validation.
Runtime protocol selection tests switch real AWG and iroh/QUIC carriers and
end-to-end WireGuard tunnels from either endpoint-id role, carry full-size
packets, keep another active network's tunnels unchanged, disable and re-enable
the data plane without losing control sessions, and preserve selection after
restart. CLI tests use a real agent process and control socket to exercise
selection by network name, stopped networks, invalid lists, and persisted
control-only configuration, offline updates, and enabling a protocol absent
from the startup list. A real AWG test starts without a TUN, creates the single
memory interface on enablement, and races rapid changes against pending opens.
SQLite tests migrate schema 7 to 8 without changing
identity, secrets or signed records, preserve order and explicit empty choices,
and reject malformed stored selections. These checks use memory TUNs and do
not establish GUI rendering or behavior on an actual OS interface.
`tests/forget_node.rs` uses real agents, handshakes and SQLite to check equal
cleanup rights, network isolation, online/self/stale-version refusal, restart,
recovery of a stale replica, reuse of a hostname and a returning identity's
newer claim. Bounded batch delivery is checked with a two-message send queue
and twelve retained markers. Retirement unit tests check signatures, proof tampering, invented
future cutoffs, bounds, deterministic merge and resistance to stale snapshots.
SQLite tests migrate schema 8 to 9 and inject counter-write failure to prove
atomic rollback. The CLI test starts two real agent processes and exercises
offline cleanup by network and hostname through the control socket.
Native tray rendering and physical screen sizing are not exercised automatically.
Linux exit-client unit tests cover preservation of foreign mark bits, matching
only the agent's UID and locally owned TCP/UDP sockets, exact tagged cleanup,
partial IPv6 failure and mark-aware strict kernel RPF validation. The ignored
`overlay::hostrules::linux_exit_kernel::strict_firewall_preserves_underlay_and_application_exit_routing`
test creates two disposable network namespaces. It reproduces the UID-only
failure with real sockets, then runs TCP/UDP IPv4/IPv6 and first inbound TCP/UDP
packets, including a UDP port bound after activation, through real mark/conntrack
rules with both strict kernel and
Netfilter reverse-path checks. It verifies application exit routing, IPv6
blocking, repeated apply and cleanup with foreign rules at Tailscale's priority
retained. It requires root, iproute2,
iptables/ip6tables and Python; it does not run in the default suite or claim
to test the entire mesh inside these namespaces.
Devices-window tests render the egui grid at multiple widths with its real
scroll area. They check stretching and shrinking without an expanding first
column, separate handshake/control RTT values, unknown measurements and short
end-to-end hints that do not substitute the carrier protocol. This does not
establish native desktop rendering on every platform.
VIP tests cover signed multi-writer backend merging, stale draft rejection,
permanent deletion and replay, separate active/deleted capacity, and mandatory
SQLite migration/counter rollback. Real agent instances verify equal editing
rights, network isolation, restart and recovery without the creator. Packet
checks exercise RTT selection, port translation, UDP failover, a local backend,
fragment reassembly/checksums and ICMP quoted-header/PMTU translation. TCP check
recovery uses real loopback listening sockets. The WireGuard suite moves VIP
TCP/UDP packets through real encrypted tunnels, rejects forged VIP sources and
checks UDP failover after the selected agent stops. CLI/DNS tests run the real
binary, configure independent listeners, disable checks, reject an invalid
change atomically and observe DNS creation/deletion. GUI tests cover draft
isolation and form rendering. These do not establish live host TUN behavior or
native desktop behavior on Linux, macOS or Windows. See [VIP semantics](vip.md).
+25 -4
View File
@@ -26,8 +26,8 @@ Read this before relying on anything here. The protocol is in
neither receive nor forge another member's traffic. See
[wireguard.md](wireguard.md#address-ownership-is-enforced-not-announced).
- **Tunnelled traffic is end-to-end encrypted by WireGuard**, independently of
this crate. The transport underneath is also encrypted by iroh, but the
tunnel's confidentiality does not depend on that.
this crate. Carriers may be iroh, TLS, direct WG or
pairwise AWG; confidentiality does not depend on a particular carrier.
- **Resource bounds.** Frame lengths are validated before allocation; strings,
lists, queues, concurrent dials and in-flight handshakes are all bounded;
handshakes, dials and writes have timeouts.
@@ -38,6 +38,14 @@ Read this before relying on anything here. The protocol is in
arbitrarily many identities, flood the network with records and collide with
other participants' names. This is why a majority is not a root of trust.
Signatures protect authorship; they do not make a participant honest.
- **Cleanup rights are equal.** Any member can retire an observed signed claim
of another device, freeing its hostname and IPv4 reservation. A valid original
claim bounds the cutoff; a member cannot invent a future cutoff or forge the
target's release. The local API refuses online nodes, but a remote signature
cannot prove that a target was offline. A live target may publish a newer
claim; a malicious member can repeatedly disrupt reservations. Cleanup is
not expulsion or secret revocation. See
[sync-model.md](sync-model.md#offline-node-cleanup).
- **Weak secrets.** This targets high-entropy secrets. There is no PAKE, so a
short human passphrase can be guessed offline by anyone who can reach the
handshake. Use `NetworkSecret::generate()`.
@@ -55,8 +63,10 @@ Read this before relying on anything here. The protocol is in
having iroh under it does not make it so.
- **A cloned state directory is a cloned identity.** `state.sqlite` holds the
device secret key and the network secrets. Copying it copies the participant.
Restoring an old backup rolls the agent's state back, which — once signed
records exist — can resurrect revoked information or replay stale versions.
Restoring an old backup rolls the agent's state back and can replay stale
versions until it meets a replica holding the newer signed state. Retained
cleanup markers prevent old claims from resurrecting after that merge; they
cannot help if every copy of a marker has been lost.
- **A compromised host.** The secret is on disk to survive restarts. File
permissions are owner-only where the platform supports it, and the state
directory takes an ownership lock, but neither defends against a user who can
@@ -113,3 +123,14 @@ for key separation, HMAC-SHA256 for the membership proof, constant-time
verification, iroh's Ed25519 endpoint keys and QUIC/TLS for the transport, and
RFC 5705 TLS exporter output for channel binding. There is no custom encryption
layer and no custom PAKE.
## Pairwise AWG masking
AWG profiles, PSKs and outer keys are private to one authenticated pair attempt,
not shared network state. Address candidates are unverified until authenticated
outer WireGuard data arrives. Header protection and structured signature packets
reduce fixed wire markers; they do not guarantee admission by DPI or hide traffic
analysis. iroh control availability remains a prerequisite. Encrypted end-to-end
delivery probes detect sustained carrier loss, never confer membership and pass
the same signed source/destination admission as ordinary IP traffic. See
[amnezia.md](amnezia.md) for limits and tested scope.
+171
View File
@@ -0,0 +1,171 @@
# Named virtual services
A VIP is a shared name such as `api.home`, with independent TCP/UDP listeners.
Each listener maps its port to a set of node identities and application ports.
The application uses the name; the automatically allocated IPv4 address is an
internal reservation. The current overlay carries IPv4 only.
## CLI and tray
```sh
tsng vip list home
tsng vip create home api
tsng vip port add home api tcp/443 --backend server1:6443 --backend server2:7443
tsng vip port add home api udp/53 --backend server1:5353
tsng vip backend add home api tcp/443 server3:6443
tsng vip backend remove home api tcp/443 server2:7443
tsng vip port set home api tcp/443 --port-check off
tsng vip port set home api tcp/443 --check-interval-ms 3000 --check-timeout-ms 2000 --failures 2 --successes 1 --idle-timeout 7200
tsng vip show home api
tsng vip status home api --watch
tsng vip edit home api
tsng vip port remove home api udp/53
tsng vip remove home api
```
`list` without a network lists all configured networks. Network and VIP
selectors accept exact names, full ids and unique id prefixes; backend node
selectors also accept stored node names. Ambiguous selections change nothing.
Names resolve to identities when saved, so a renamed node remains a backend.
`--json` provides configuration, revision, counters and observations;
`list`/`watch` emit one JSON object per service/sample. `edit`
opens JSON in `VISUAL`/`EDITOR` and submits one complete version-checked draft.
An invalid or rejected edited file is retained with its path reported.
Each mutating command is atomic. A listener can be added with its full initial
backend set, and `edit` can replace the entire configuration at once. Empty
VIPs/listeners are valid and resolve, but cannot accept traffic. Configuration
updates currently require the network to be active locally. Saved configuration remains visible while a
network is stopped.
The tray's network tile has a **VIP** button. Its separate window edits names,
listeners, backend nodes/ports, check settings and flow timeouts. All edits,
including deletion, remain drafts until **Save**. Status updates do not
replace a dirty draft. A save conflict retains the draft; **Discard / reload**
loads the latest configuration. Closing a dirty window offers explicit discard
or continued editing. Addresses and ids are available in a collapsed detail.
## Delivery and recovery
The client agent selects the eligible backend with the lowest smoothed full
round-trip latency through the actual encrypted data plane, including relays.
Unknown or stale RTT is not treated as zero. Local backends have zero path RTT.
A new flow is pinned to its selected backend; a faster RTT does not move it.
This can concentrate new traffic on the closest node.
At local ingress, NAT replaces destination address/port and reserves an OS
source port for an unambiguous reverse mapping. Return traffic passes normal
network/peer source and destination admission **before** reverse NAT restores
the virtual source address/port. Plugins and encrypted transit stay unchanged.
A peer never gains permission to claim a VIP source. Fragment assembly is
bounded, checksums are updated, packets are fragmented back to the overlay MTU,
and ICMP errors have their quoted NAT headers restored. Fragment caches are
scoped to network and authenticated sender, with idle expiry independent of
further traffic.
For a backend on the client itself, the request is injected into the same TUN
with the nonlocal VIP as source, and the host reply is reverse-translated at
local ingress. The local backend therefore sees the VIP as source; remote
backends see the client's ordinary overlay address. Firewalls and application
bind addresses remain the administrator's responsibility.
Removing a backend/listener/VIP prevents new bindings while existing bindings
continue until idle expiry. Established TCP connections are not migrated. A retransmitted empty SYN may
choose another backend before a SYN-ACK has arrived; SYNs carrying application
data are not retargeted. A new SYN sequence creates a new binding even if the
client reuses its address/port tuple. TCP RST shortens binding retention. An unavailable UDP backend is replaced for subsequent
datagrams; already sent datagrams are never replayed. Stateful UDP applications
may need to re-establish their application session. Protocols that embed IP
addresses or negotiate additional ports require their own configuration; no
application payloads are parsed or rewritten.
Idle expiry defaults to 7200 seconds for TCP and 120 seconds for UDP. Shadow
ports are quarantined for 120 seconds after retirement to reject late replies
and avoid immediate reuse. Active and quarantined bindings share a bounded
capacity. State and sockets are released on network stop or agent shutdown.
A restart drops bindings; shared service configuration remains intact.
## Health
One probe exchange per second measures each used peer through the encrypted
IP tunnel. The EWMA uses 7/8 of the previous sample and 1/8 of the new one.
A probe expires after three seconds; two consecutive missing responses or a
missing data path exclude the peer. Measurements expire after ten seconds.
Probes continue during failure so recovery is automatic.
TCP connect checks default to on. Defaults are a three-second interval,
two-second timeout, two failures to exclude and one success to recover.
Checks are shared for the same network/node/address/TCP-port, run off the
packet path with at most 32 concurrent connects, and are scheduled fairly.
A 250ms worker tick services timers. Connect checks send no application data;
they confirm port acceptance and may appear in server logs. They cannot prove
application readiness after a successful handshake. Disabling port checks
retains data-plane probes.
UDP uses encrypted path checks and explicit ICMP port errors. No arbitrary
probe datagrams are injected into an unknown application protocol. Silence
cannot prove failure of an arbitrary UDP application. An explicit unreachable
port is excluded for ten seconds and then becomes eligible to retry.
## Shared state and addresses
Any authenticated member can create, edit or remove any VIP. Each field is a
self-contained signed register: network, service id, field key, optional encoded
value, Lamport clock, author, durable author counter and signature. Signing uses
the separate `tsunagi-vip-field/v1\0` domain followed by postcard encoding of
those fields with the signature vector empty. The encoding/order is a wire
contract; the identity scheme and handshake transcript are unchanged.
Concurrent edits to different backend fields merge. A conflict on one field
uses `(Lamport clock, author id, canonical bytes)` ordering, independent of wall
time and arrival order. Listener generations prevent backend fields from an
old removed listener reappearing when the port is recreated. Service deletion
is a permanent signed tombstone for its id; recreation gets a fresh id.
Missing snapshot fields are never deletion.
Saving validates the entire draft and checks the revision shown by the editor.
The merged snapshot and author's counter commit together in SQLite before local
routing changes or publication. Receivers verify and persist complete service
snapshots before exposing them. Periodic rotating snapshots repair queue loss
and returning replicas without requiring the creator online. Concurrent edits
made before participants have seen each other can still merge after a save.
A merged configuration exceeding listener/backend limits is reported as an
error and excluded from routing rather than selected according to arrival order.
VIP allocation walks the agreed overlay range from a service-id-derived seed,
excluding known node claims and other allocated VIPs. Nodes avoid known VIP
reservations when choosing a new address. Names colliding with node names or
another service are reported; node names take precedence and service ties use
id order. A partition can temporarily have incomplete state; after merge an
address or conflicting name's effective holder can change. Use DNS names.
DNS TTL is 30 seconds; DNS does not migrate established connections or force
applications to discard cached addresses. Existing NAT bindings retain their
original VIP address while draining.
`Limits` bounds active services (64 by default), permanent deletion markers
(1024), signed fields per active service (192), listeners (16), backends per
listener (32), active TCP checks (32), flow/port reservations (4096) and incomplete
IPv4 datagrams (64). A deleted service retains only its signed permanent
marker; obsolete fields can be dropped because replay cannot override deletion.
A full deletion history is an explicit error; protection is never silently
evicted. Deleted services do not consume the active-service limit. Whole service
snapshots must also fit the configured control frame limit. Corrupt mandatory VIP storage
is a hard error on activation, never reset as a cache.
## Observations and verification
Status is this client's observation. Per VIP/listener/backend it reports the
next selected backend, health reason, RTT and sample age, active bindings,
user IP packet/byte counters, drops and UDP failovers. Probe bytes do not enter
user traffic counters. The GUI and CLI watch derive rates from cumulative
counters; counters reset on restart. These are IP forwarding counters, not
application acknowledgements or guaranteed delivered-byte totals.
Automated tests cover field merging, stale drafts, signed deletion, tampering,
network isolation, SQLite migration and transactional rollback, real port
connect failure/recovery, RTT selection, UDP failover, local hairpin NAT,
fragmentation/checksums, real encrypted TCP/UDP packet delivery and source
admission, real CLI/DNS wiring and draft form rendering. In-memory TUN tests
exercise packets, not an operating system TCP stack. Native desktop behavior,
physical interfaces and cross-platform kernel/firewall behavior need live-host
validation; automated tests do not establish those.
+30 -7
View File
@@ -92,8 +92,8 @@ and then recorded as a signed fact**, using the model in
[sync-model.md](sync-model.md).
```text
default range 10.13.37.0/24 (override with --ipv4-range)
who decides the first member to claim; later ones adopt what they find
default range stable pseudorandom /24 in 10/8, derived from NetworkId
who decides all members derive the default; signed explicit ranges win
who signs the claiming member, with its persistent device key
where it is kept state.sqlite, and every replica that has seen it
what a return costs nothing: the old address is reclaimed
@@ -120,11 +120,34 @@ the same address at the same moment are resolved by a rule both compute
identically — the lower endpoint id keeps it — and the loser simply allocates
again with a higher version.
**The range is agreed, not configured per member.** `--ipv4-range` says what
this agent would use; a network that has already settled on something else
wins, and the agent adopts it. So the flag matters for whoever starts the
network and is harmless afterwards. Pass `--ipv4-range none` for an IPv6-only
overlay.
**The range is agreed, not configured per member.** New networks use the
same deterministic `/24` on every device, independently of which network a
member activates first or whether any peers are reachable. The derivation
is unchanged from the former fallback; network identity and signed record
encoding are unchanged. Distinct network ids can still derive the same range.
`--ipv4-range` proposes an explicit range; existing signed ranges take
precedence over the new derived default, including the former `10.13.37.0/24`.
Among disagreeing non-default ranges the lowest author id still wins. Updated
members propagate their adopted range in their signed claims; older releases
still apply their original selection rule until those claims converge.
`--ipv4-range none` disables local proposals, as before.
An agent reserves each active network's range before starting its runtime.
Overlapping ranges, including nested blocks such as a `/24` inside a `/16`,
cannot run simultaneously. A late authenticated state update that introduces
an overlap suspends that network before publishing new allocations to plugins
or the interface. Other networks continue running.
Status includes every blocking network's name, id and range. `tsng network
start <name-or-id>` and the tray's switch button stop all blockers before
starting the selected network. Suspended networks keep their startup request
and resume when the blocker stops or is forgotten. A manual stop cancels that
request even for a network already suspended. The user's preference order is
stored locally, so the same selection wins at the next startup; it is never
advertised or treated as a remote network setting. Local IPC uses wire version
19 for the new reports: restart the agent and tray together after upgrading.
The peer control protocol and signed record encoding do not change.
The agent puts the address on the interface itself, as soon as the network has
agreed on it — no restart, and the interface is not recreated, which would
+1
View File
@@ -102,6 +102,7 @@ in
# Exit nodes: the agent looks for iptables only in fixed FHS directories.
systemd.tmpfiles.rules = [
"L+ /usr/bin/iptables - - - - ${pkgs.iptables}/bin/iptables"
"L+ /usr/bin/ip6tables - - - - ${pkgs.iptables}/bin/ip6tables"
];
# Names of mesh members are served locally and handed to the system resolver.
+539
View File
@@ -0,0 +1,539 @@
# THIS FILE IS AUTOMATICALLY GENERATED BY CARGO
#
# When uploading crates to the registry Cargo will automatically
# "normalize" Cargo.toml files for maximal compatibility
# with all versions of Cargo and also rewrite `path` dependencies
# to registry (e.g., crates.io) dependencies.
#
# If you are reading this file be aware that the original Cargo.toml
# will likely look very different (and much more reasonable).
# See Cargo.toml.orig for the original contents.
[package]
edition = "2024"
rust-version = "1.91"
name = "iroh"
version = "1.2.0"
authors = [
"dignifiedquire <me@dignifiedquire.com>",
"n0 team",
]
build = "build.rs"
autolib = false
autobins = false
autoexamples = false
autotests = false
autobenches = false
description = "p2p quic connections dialed by public key"
readme = "README.md"
keywords = [
"quic",
"networking",
"holepunching",
"p2p",
]
license = "MIT OR Apache-2.0"
repository = "https://github.com/n0-computer/iroh"
resolver = "2"
[package.metadata.docs.rs]
all-features = true
rustdoc-args = [
"--cfg",
"iroh_docsrs",
]
[package.metadata.cargo_check_external_types]
allowed_external_types = [
"iroh_base::*",
"iroh_dns::*",
"iroh_relay::*",
"iroh_metrics::*",
"n0_error::*",
"n0_watcher::*",
"noq::*",
"noq_proto::*",
"noq_udp::*",
"bytes::*",
"http::*",
"serde_core::*",
"tokio::*",
"url::*",
"rustls::*",
"futures_core::stream::Stream",
"futures_lite::stream::Boxed",
]
[features]
default = [
"metrics",
"fast-apple-datapath",
"portmapper",
"tls-ring",
]
fast-apple-datapath = ["noq/fast-apple-datapath"]
metrics = [
"iroh-metrics/metrics",
"iroh-relay/metrics",
]
platform-verifier = ["iroh-relay/platform-verifier"]
portmapper = ["dep:portmapper"]
qlog = ["noq/qlog"]
test-utils = [
"iroh-relay/test-utils",
"iroh-relay/server",
"dep:axum",
"dep:simple-dns",
]
tls-aws-lc-rs = [
"noq/aws-lc-rs",
"iroh-relay/tls-aws-lc-rs",
"iroh-dns/tls-aws-lc-rs",
]
tls-ring = [
"noq/ring",
"iroh-relay/tls-ring",
"iroh-dns/tls-ring",
]
unstable-custom-transports = []
unstable-net-report = []
[lib]
name = "iroh"
crate-type = [
"lib",
"cdylib",
]
path = "src/lib.rs"
[[example]]
name = "0rtt"
path = "examples/0rtt.rs"
required-features = []
[[example]]
name = "auth-hook"
path = "examples/auth-hook.rs"
[[example]]
name = "connect"
path = "examples/connect.rs"
required-features = []
[[example]]
name = "connect-unreliable"
path = "examples/connect-unreliable.rs"
required-features = []
[[example]]
name = "custom-transport"
path = "examples/custom-transport.rs"
required-features = [
"test-utils",
"unstable-custom-transports",
]
[[example]]
name = "echo"
path = "examples/echo.rs"
required-features = []
[[example]]
name = "echo-no-router"
path = "examples/echo-no-router.rs"
required-features = []
[[example]]
name = "home-relay-status"
path = "examples/home-relay-status.rs"
required-features = []
[[example]]
name = "incoming-filter"
path = "examples/incoming-filter.rs"
required-features = []
[[example]]
name = "listen"
path = "examples/listen.rs"
required-features = []
[[example]]
name = "listen-unreliable"
path = "examples/listen-unreliable.rs"
required-features = []
[[example]]
name = "monitor-connections"
path = "examples/monitor-connections.rs"
[[example]]
name = "pq-only-key-exchange"
path = "examples/pq-only-key-exchange.rs"
required-features = ["tls-aws-lc-rs"]
[[example]]
name = "prefer-pq-key-exchange"
path = "examples/prefer-pq-key-exchange.rs"
required-features = ["tls-aws-lc-rs"]
[[example]]
name = "remote-info"
path = "examples/remote-info.rs"
[[example]]
name = "screening-connection"
path = "examples/screening-connection.rs"
[[example]]
name = "search"
path = "examples/search.rs"
required-features = []
[[example]]
name = "transfer"
path = "examples/transfer.rs"
required-features = []
[[test]]
name = "integration"
path = "tests/integration.rs"
[[test]]
name = "patchbay"
path = "tests/patchbay.rs"
[dependencies.axum]
version = "0.8"
optional = true
[dependencies.backon]
version = "1.4"
[dependencies.blake3]
version = "1.8.3"
default-features = false
[dependencies.bytes]
version = "1.11"
[dependencies.ctutils]
version = "0.4.0"
default-features = false
[dependencies.data-encoding]
version = "2.2"
[dependencies.derive_more]
version = "2.0.1"
features = [
"debug",
"display",
"from",
"try_into",
"deref",
"from_str",
"into_iterator",
]
[dependencies.ed25519-dalek]
version = ">=3.0.0-rc.0,<4.0.0"
features = [
"serde",
"rand_core",
"zeroize",
"pkcs8",
"pem",
]
[dependencies.futures-util]
version = "0.3"
[dependencies.http]
version = "1"
[dependencies.ipnet]
version = "2"
[dependencies.iroh-base]
version = "1.2.0"
features = [
"key",
"relay",
]
default-features = false
[dependencies.iroh-dns]
version = "1.3.0"
[dependencies.iroh-metrics]
version = "1.0.1"
default-features = false
[dependencies.iroh-relay]
version = "1.2.0"
default-features = false
[dependencies.n0-error]
version = "1.0.0"
[dependencies.n0-future]
version = "0.3"
[dependencies.n0-watcher]
version = "1.0.0"
[dependencies.netwatch]
version = "0.19.3"
[dependencies.noq]
version = "1.3.0"
features = ["rustls"]
default-features = false
[dependencies.noq-proto]
version = "1.3.0"
default-features = false
[dependencies.noq-udp]
version = "1.3.0"
default-features = false
[dependencies.papaya]
version = "0.2.3"
default-features = false
[dependencies.pin-project]
version = "1"
[dependencies.portable-atomic]
version = "1"
[dependencies.rand]
version = "0.10"
[dependencies.reqwest]
version = "0.13"
features = [
"rustls-no-provider",
"stream",
]
default-features = false
[dependencies.rustc-hash]
version = "2"
[dependencies.rustls]
version = "0.23.33"
default-features = false
[dependencies.serde]
version = "1.0.219"
features = [
"derive",
"rc",
]
[dependencies.simple-dns]
version = "0.12"
optional = true
[dependencies.smallvec]
version = "1.11.1"
[dependencies.strum]
version = "0.28"
features = ["derive"]
[dependencies.tokio]
version = "1.44.1"
features = [
"io-util",
"macros",
"sync",
"rt",
]
[dependencies.tokio-stream]
version = "0.1.15"
features = ["sync"]
[dependencies.tokio-util]
version = "0.7"
features = [
"io-util",
"io",
"rt",
]
[dependencies.tracing]
version = "0.1"
[dependencies.url]
version = "2.5"
features = ["serde"]
[dependencies.webpki_types]
version = "1.12"
package = "rustls-pki-types"
[dev-dependencies.chrono]
version = "0.4.43"
[dev-dependencies.console_error_panic_hook]
version = "0.1"
[dev-dependencies.n0-error]
version = "1.0.0"
features = ["anyhow"]
[dev-dependencies.postcard]
version = "1.1.1"
features = ["use-std"]
[dev-dependencies.rand_chacha]
version = "0.10"
[dev-dependencies.simple-dns]
version = "0.12"
[dev-dependencies.tracing-subscriber]
version = "0.3"
features = ["env-filter"]
[build-dependencies.cfg_aliases]
version = "0.2.2"
[target.'cfg(all(target_family = "wasm", target_os = "unknown"))'.dependencies.getrandom]
version = "0.4"
features = ["wasm_js"]
[target.'cfg(all(target_family = "wasm", target_os = "unknown"))'.dependencies.time]
version = "0.3"
features = ["wasm-bindgen"]
[target.'cfg(all(target_family = "wasm", target_os = "unknown"))'.dependencies.wasm-bindgen-futures]
version = "0.4"
[target.'cfg(all(target_family = "wasm", target_os = "unknown"))'.dev-dependencies.wasm-bindgen-test]
version = "0.3.62"
[target.'cfg(all(target_family = "wasm", target_os = "unknown"))'.dev-dependencies.wasm-tracing]
version = "2.1.0"
[target.'cfg(not(all(target_family = "wasm", target_os = "unknown")))'.dependencies.noq]
version = "1.3.0"
features = [
"runtime-tokio",
"rustls",
]
default-features = false
[target.'cfg(not(all(target_family = "wasm", target_os = "unknown")))'.dependencies.portmapper]
version = "0.19.3"
optional = true
default-features = false
[target.'cfg(not(all(target_family = "wasm", target_os = "unknown")))'.dependencies.tokio]
version = "1"
features = [
"io-util",
"macros",
"sync",
"rt",
"net",
"fs",
"io-std",
]
[target.'cfg(not(all(target_family = "wasm", target_os = "unknown")))'.dev-dependencies.assert_matches]
version = "1.5.0"
[target.'cfg(not(all(target_family = "wasm", target_os = "unknown")))'.dev-dependencies.axum]
version = "0.8"
[target.'cfg(not(all(target_family = "wasm", target_os = "unknown")))'.dev-dependencies.clap]
version = "4"
features = ["derive"]
[target.'cfg(not(all(target_family = "wasm", target_os = "unknown")))'.dev-dependencies.console]
version = "0.16"
[target.'cfg(not(all(target_family = "wasm", target_os = "unknown")))'.dev-dependencies.indicatif]
version = "0.18"
features = ["tokio"]
[target.'cfg(not(all(target_family = "wasm", target_os = "unknown")))'.dev-dependencies.iroh-base]
version = "1.2.0"
features = [
"key",
"relay",
]
default-features = false
[target.'cfg(not(all(target_family = "wasm", target_os = "unknown")))'.dev-dependencies.iroh-relay]
version = "1.2.0"
features = [
"test-utils",
"server",
]
default-features = false
[target.'cfg(not(all(target_family = "wasm", target_os = "unknown")))'.dev-dependencies.n0-tracing-test]
version = "0.3"
[target.'cfg(not(all(target_family = "wasm", target_os = "unknown")))'.dev-dependencies.parse-size]
version = "1.1.0"
features = ["std"]
[target.'cfg(not(all(target_family = "wasm", target_os = "unknown")))'.dev-dependencies.pretty_assertions]
version = "1.4"
[target.'cfg(not(all(target_family = "wasm", target_os = "unknown")))'.dev-dependencies.rand_chacha]
version = "0.10"
[target.'cfg(not(all(target_family = "wasm", target_os = "unknown")))'.dev-dependencies.serde_json]
version = "1"
[target.'cfg(not(all(target_family = "wasm", target_os = "unknown")))'.dev-dependencies.tokio]
version = "1"
features = [
"io-util",
"sync",
"rt",
"rt-multi-thread",
"net",
"fs",
"macros",
"time",
"test-util",
]
[target.'cfg(not(all(target_family = "wasm", target_os = "unknown")))'.dev-dependencies.tracing-subscriber]
version = "0.3"
features = ["env-filter"]
[target.'cfg(target_os = "linux")'.dev-dependencies.ctor]
version = "1.0"
[target.'cfg(target_os = "linux")'.dev-dependencies.patchbay]
version = "0.7"
features = ["iroh-metrics"]
[target.'cfg(target_os = "linux")'.dev-dependencies.testdir]
version = "0.10"
[lints.clippy]
unused-async = "warn"
[lints.rust]
missing_debug_implementations = "warn"
[lints.rust.unexpected_cfgs]
level = "warn"
priority = 0
check-cfg = [
"cfg(iroh_docsrs)",
"cfg(iroh_loom)",
"cfg(skip_patchbay)",
]
+256
View File
@@ -0,0 +1,256 @@
[package]
name = "iroh"
version = "1.2.0"
edition = "2024"
readme = "README.md"
description = "p2p quic connections dialed by public key"
license = "MIT OR Apache-2.0"
authors = ["dignifiedquire <me@dignifiedquire.com>", "n0 team"]
repository = "https://github.com/n0-computer/iroh"
keywords = ["quic", "networking", "holepunching", "p2p"]
# Sadly this also needs to be updated in .github/workflows/ci.yml
rust-version = "1.91"
[lib]
# We need "cdylib" to actually generate .wasm files when we run with --target=wasm32-unknown-unknown.
# It would be nice if we could make this target-dependent, but we can't (yet): https://github.com/rust-lang/cargo/issues/12260
crate-type = ["lib", "cdylib"]
[lints]
workspace = true
[dependencies]
backon = { version = "1.4" }
blake3 = { version = "1.8.3", default-features = false }
bytes = "1.11"
ctutils = { version = "0.4.0", default-features = false }
data-encoding = "2.2"
derive_more = { version = "2.0.1", features = ["debug", "display", "from", "try_into", "deref", "from_str", "into_iterator"] }
ed25519-dalek = { version = ">=3.0.0-rc.0,<4.0.0", features = ["serde", "rand_core", "zeroize", "pkcs8", "pem"] }
http = "1"
ipnet = "2"
iroh-base = { version = "1.2.0", default-features = false, features = ["key", "relay"], path = "../iroh-base" }
iroh-dns = { version = "1.3.0", path = "../iroh-dns" }
iroh-relay = { version = "1.2.0", path = "../iroh-relay", default-features = false }
n0-future = "0.3"
n0-error = "1.0.0"
n0-watcher = "1.0.0"
netwatch = "0.19.3"
papaya = { version = "0.2.3", default-features = false }
pin-project = "1"
portable-atomic = "1"
noq = { version = "1.3.0", default-features = false, features = ["rustls"] }
noq-proto = { version = "1.3.0", default-features = false }
noq-udp = { version = "1.3.0", default-features = false }
rand = "0.10"
reqwest = { version = "0.13", default-features = false, features = ["rustls-no-provider", "stream"] }
rustc-hash = "2"
rustls = { version = "0.23.33", default-features = false }
serde = { version = "1.0.219", features = ["derive", "rc"] }
simple-dns = { version = "0.12", optional = true }
smallvec = "1.11.1"
strum = { version = "0.28", features = ["derive"] }
tokio = { version = "1.44.1", features = [
"io-util",
"macros",
"sync",
"rt",
] }
tokio-stream = { version = "0.1.15", features = ["sync"] }
tokio-util = { version = "0.7", features = ["io-util", "io", "rt"] }
tracing = "0.1"
url = { version = "2.5", features = ["serde"] }
webpki_types = { package = "rustls-pki-types", version = "1.12" }
# metrics
iroh-metrics = { version = "1.0.1", default-features = false }
futures-util = "0.3"
# test_utils
axum = { version = "0.8", optional = true }
# non-wasm-in-browser dependencies
[target.'cfg(not(all(target_family = "wasm", target_os = "unknown")))'.dependencies]
portmapper = { version = "0.19.3", optional = true, default-features = false }
noq = { version = "1.3.0", default-features = false, features = ["runtime-tokio", "rustls"] }
tokio = { version = "1", features = [
"io-util",
"macros",
"sync",
"rt",
"net",
"fs",
"io-std",
] }
# wasm-in-browser dependencies
[target.'cfg(all(target_family = "wasm", target_os = "unknown"))'.dependencies]
wasm-bindgen-futures = "0.4"
# we don't use time directly, but need to enable it because x509_parser uses these in browsers and we need to enable some feature flags for that to work
time = { version = "0.3", features = ["wasm-bindgen"] }
getrandom = { version = "0.4", features = ["wasm_js"] }
# target-common test/dev dependencies
[dev-dependencies]
console_error_panic_hook = "0.1"
n0-error = { version = "1.0.0", features = ["anyhow"] }
postcard = { version = "1.1.1", features = ["use-std"] }
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
rand_chacha = "0.10"
chrono = "0.4.43"
simple-dns = "0.12"
# *non*-wasm-in-browser test/dev dependencies
[target.'cfg(not(all(target_family = "wasm", target_os = "unknown")))'.dev-dependencies]
assert_matches = "1.5.0"
axum = { version = "0.8" }
pretty_assertions = "1.4"
rand_chacha = "0.10"
tokio = { version = "1", features = [
"io-util",
"sync",
"rt",
"rt-multi-thread",
"net",
"fs",
"macros",
"time",
"test-util",
] }
serde_json = "1"
iroh-relay = { version = "1.2.0", path = "../iroh-relay", default-features = false, features = ["test-utils", "server"] }
n0-tracing-test = "0.3"
clap = { version = "4", features = ["derive"] }
tracing-subscriber = { version = "0.3", features = [
"env-filter",
] }
indicatif = { version = "0.18", features = ["tokio"] }
parse-size = { version = "1.1.0", features = ['std'] }
iroh-base = { version = "1.2.0", default-features = false, features = ["key", "relay"], path = "../iroh-base" }
console = { version = "0.16" }
# wasm-in-browser test/dev dependencies
[target.'cfg(all(target_family = "wasm", target_os = "unknown"))'.dev-dependencies]
wasm-tracing = "2.1.0"
wasm-bindgen-test = "0.3.62"
# patchbay netsim test dependencies (linux only)
[target.'cfg(target_os = "linux")'.dev-dependencies]
ctor = "1.0"
patchbay = { version = "0.7", features = ["iroh-metrics"] }
testdir = "0.10"
[build-dependencies]
cfg_aliases = { version = "0.2.2" }
[features]
default = ["metrics", "fast-apple-datapath", "portmapper", "tls-ring"]
portmapper = ["dep:portmapper"]
metrics = ["iroh-metrics/metrics", "iroh-relay/metrics"]
test-utils = ["iroh-relay/test-utils", "iroh-relay/server", "dep:axum", "dep:simple-dns"]
# Enables fetching TLS trust anchors from the operating system
platform-verifier = ["iroh-relay/platform-verifier"]
qlog = ["noq/qlog"]
# Use private Apple APIs to send multiple packets in a single syscall.
fast-apple-datapath = ["noq/fast-apple-datapath"]
# Use ring as the crypto backend.
tls-ring = ["noq/ring", "iroh-relay/tls-ring", "iroh-dns/tls-ring"]
# Use aws-lc-rs as the crypto backend, unless `ring` is also enabled.
tls-aws-lc-rs = ["noq/aws-lc-rs", "iroh-relay/tls-aws-lc-rs", "iroh-dns/tls-aws-lc-rs"]
# Unstable: Custom transport API (may change without notice)
unstable-custom-transports = []
# Unstable: API to access an endpoint's NetReport (may change without notice)
unstable-net-report = []
[package.metadata.docs.rs]
all-features = true
rustdoc-args = ["--cfg", "iroh_docsrs"]
[package.metadata.cargo_check_external_types]
allowed_external_types = [
# workspace crates
"iroh_base::*",
"iroh_dns::*",
"iroh_relay::*",
# crates owned by us that will move to 1.0 as well
"iroh_metrics::*",
"n0_error::*",
"n0_watcher::*",
"noq::*",
"noq_proto::*",
"noq_udp::*",
# 1.0 crates that we deem fine to be part of the public API
"bytes::*",
"http::*",
"serde_core::*",
"tokio::*",
"url::*",
# non-1.0 crates that we decided to accept in the public API
"rustls::*",
"futures_core::stream::Stream",
# only type alias
"futures_lite::stream::Boxed",
]
[[test]]
name = "integration"
path = "tests/integration.rs"
[[example]]
name = "listen"
required-features = []
[[example]]
name = "connect"
required-features = []
[[example]]
name = "custom-transport"
required-features = ["test-utils", "unstable-custom-transports"]
[[example]]
name = "listen-unreliable"
required-features = []
[[example]]
name = "connect-unreliable"
required-features = []
[[example]]
name = "search"
required-features = []
[[example]]
name = "echo"
required-features = []
[[example]]
name = "echo-no-router"
required-features = []
[[example]]
name = "transfer"
required-features = []
[[example]]
name = "0rtt"
required-features = []
[[example]]
name = "incoming-filter"
required-features = []
[[example]]
name = "pq-only-key-exchange"
required-features = ["tls-aws-lc-rs"]
[[example]]
name = "prefer-pq-key-exchange"
required-features = ["tls-aws-lc-rs"]
[[example]]
name = "home-relay-status"
required-features = []
+49
View File
@@ -0,0 +1,49 @@
# Developing iroh
## Structured events
The library uses [tracing] both for logging and for *structured events*.
Events differ from normal logging by convention:
- The [target] is prefixed with `iroh::_events::`, with `::`-separated names.
- There is **no message**; the unique target indicates the meaning.
- The [fields] carry exclusively structured data.
- The [Level] is always `DEBUG`.
This lets automated tooling process events through custom subscribers while
still producing distinct output under the default tracing formatters, and makes
them unlikely to conflict with real modules.
An application can subscribe to the `iroh::_events` target separately. With the
default file logging it is also easy to grep for all events:
```sh
rg 'events::[a-z_\-:]+' path/to/iroh/logs/iroh.YYYY-MM-DD-NN.log
```
When adding events, aim for a high signal-to-noise ratio and design them to be
extracted automatically. To keep them distinct from normal logging, write them
with the `event!()` macro:
```rust,ignore
event!(
target: "iroh::_events::subject",
Level::DEBUG,
field = value,
);
```
## Building documentation
Building the documentation is only supported with `--all-features`. To also
document the cargo features required for certain APIs, pass the `iroh_docsrs`
cfg to rustdoc, which requires nightly Rust:
```sh
RUSTDOCFLAGS="--cfg iroh_docsrs" cargo +nightly doc --workspace --no-deps --all-features
```
[target]: https://docs.rs/tracing/latest/tracing/struct.Metadata.html#method.target
[fields]: https://docs.rs/tracing/latest/tracing/#recording-fields
[Level]: https://docs.rs/tracing/latest/tracing/struct.Level.html
[tracing]: https://docs.rs/tracing
+33
View File
@@ -0,0 +1,33 @@
Parts of the code has been derived from tailscale, which is under the following license.
Specifically the following files are most relevant
- ./src/socket**
BSD 3-Clause License
Copyright (c) 2020 Tailscale Inc & AUTHORS.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
3. Neither the name of the copyright holder nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+183
View File
@@ -0,0 +1,183 @@
<h1 align="center">
<a href="https://iroh.computer">
<img alt="iroh" src="https://raw.githubusercontent.com/n0-computer/iroh/main/.img/iroh_wordmark.svg" width="100" />
</a>
</h1>
<h3 align="center">
less net work for networks
</h3>
[![Documentation](https://img.shields.io/badge/docs-latest-blue.svg?style=flat-square)](https://docs.rs/iroh/)
[![Crates.io](https://img.shields.io/crates/v/iroh.svg?style=flat-square)](https://crates.io/crates/iroh)
[![Chat](https://img.shields.io/discord/1161119546170687619?logo=discord&style=flat-square)](https://discord.com/invite/DpmJgtU7cW)
[![Youtube](https://img.shields.io/badge/YouTube-red?logo=youtube&logoColor=white&style=flat-square)](https://www.youtube.com/@n0computer)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg?style=flat-square)](LICENSE-MIT)
[![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg?style=flat-square)](LICENSE-APACHE)
<div align="center">
<h3>
<a href="https://iroh.computer/docs">
Docs Site
</a>
<span> | </span>
<a href="https://docs.rs/iroh">
Rust Docs
</a>
</h3>
</div>
<br/>
Iroh is a Rust library to establish direct connections between endpoints.
It gives you an API for dialing by public key. You say "connect to that
endpoint", and iroh finds and maintains the best connection for you.
Under the hood iroh establishes peer-to-peer [QUIC] connections between
endpoints. The fastest route is a direct connection, so iroh tries to
[hole-punch] one whenever it can. If that fails it falls back to using
relay servers.
Because iroh is built on [QUIC], all connections are end-to-end encrypted and may
carry any number of concurrent streams. Dialing by public key also makes them mutually
authenticated, because each endpoint's public key is its TLS identity.
## Overview
An iroh endpoint is created and controlled by the [`Endpoint`]. Each endpoint
has a unique [`SecretKey`], whose public key is the endpoint's identity, the
[`EndpointId`]. Connections are authenticated against this key, which means an
[`EndpointId`] can't be impersonated.
A connection is usually established with the help of a *relay server*. When an
endpoint is created it connects to the closest relay and designates it as its
*home relay*. Other endpoints reach it first through this relay, then both
sides use QUIC NAT traversal to establish a direct connection. In the rare
cases where a direct connection is not possible, traffic keeps flowing over the
relay.
Relay servers only forward encrypted packets addressed to Endpoint IDs, they
cannot read any traffic between endpoints.
Endpoints can also connect directly without a relay, as long as the accepting
endpoint is directly reachable at one of its addresses.
To discover addressing information for an endpoint, iroh uses
[address lookup services]. With address lookup, you can connect to other
endpoints with only their [`EndpointId`]. Addressing information will then
be resolved on-demand.
The [`N0` preset] installs the DNS/Pkarr address lookup service, which uses
servers hosted by [n0] to provide global lookup for endpoints.
## Example
This is an echo protocol: the accepting side copies back whatever it receives.
The full, commented version is in [`echo.rs`](examples/echo.rs).
```rust
use iroh::{
Endpoint,
endpoint::{Connection, presets},
protocol::{AcceptError, ProtocolHandler, Router},
};
/// Each protocol is identified by its ALPN, exchanged during the handshake.
const ALPN: &[u8] = b"iroh-example/echo/0";
#[tokio::main]
async fn main() -> anyhow::Result<()> {
// The accepting side: bind an endpoint and route the ALPN to a handler.
let endpoint = Endpoint::bind(presets::N0).await?;
let router = Router::builder(endpoint.clone()).accept(ALPN, Echo).spawn();
endpoint.online().await;
// Get the endpoint's address so that we can connect to it.
let addr = endpoint.addr();
// The connecting side: dial the accepting endpoint by its address.
{
let other_endpoint = Endpoint::bind(presets::N0).await?;
let conn = other_endpoint.connect(addr, ALPN).await?;
let (mut send, mut recv) = conn.open_bi().await?;
send.write_all(b"Hello, world!").await?;
send.finish()?;
let response = recv.read_to_end(1000).await?;
assert_eq!(&response, b"Hello, world!");
conn.close(0u32.into(), b"bye!");
other_endpoint.close().await;
}
router.shutdown().await?;
Ok(())
}
#[derive(Debug, Clone)]
struct Echo;
impl ProtocolHandler for Echo {
async fn accept(&self, connection: Connection) -> Result<(), AcceptError> {
let (mut send, mut recv) = connection.accept_bi().await?;
// Echo bytes back until the sender signals the end of data.
tokio::io::copy(&mut recv, &mut send).await?;
send.finish()?;
// Wait until the other endpoint closes the connection.
connection.closed().await;
Ok(())
}
}
```
More examples live in [`iroh/examples`](examples). Run them with
`cargo run --example NAME`. Details for each are in the file itself.
## Compose protocols
Instead of writing your own, you can build on protocols that already exist on
top of iroh:
- [iroh-blobs] for [BLAKE3]-based content-addressed blob transfer, scaling from
kilobytes to terabytes.
- [iroh-gossip] for publish-subscribe overlay networks that scale down to what
an average phone can handle.
- and many more.
To use iroh from other languages, see [iroh-ffi].
## Development
For notes on iroh's structured events and how to build the documentation, see
[DEVELOPMENT.md](DEVELOPMENT.md).
# License
This project is licensed under either of
* Apache License, Version 2.0, ([LICENSE-APACHE](LICENSE-APACHE) or
https://www.apache.org/licenses/LICENSE-2.0)
* MIT license ([LICENSE-MIT](LICENSE-MIT) or
https://opensource.org/licenses/MIT)
at your option.
### Contribution
See [CONTRIBUTING.md](https://github.com/n0-computer/iroh/blob/main/CONTRIBUTING.md)
for how to get involved.
Unless you explicitly state otherwise, any contribution intentionally submitted
for inclusion in this project by you, as defined in the Apache-2.0 license,
shall be dual licensed as above, without any additional terms or conditions.
[QUIC]: https://en.wikipedia.org/wiki/QUIC
[hole-punch]: https://en.wikipedia.org/wiki/Hole_punching_(networking)
[BLAKE3]: https://github.com/BLAKE3-team/BLAKE3
[`Endpoint`]: https://docs.rs/iroh/latest/iroh/struct.Endpoint.html
[`SecretKey`]: https://docs.rs/iroh/latest/iroh/struct.SecretKey.html
[`EndpointId`]: https://docs.rs/iroh/latest/iroh/struct.EndpointId.html
[address lookup services]: https://docs.rs/iroh/latest/iroh/address_lookup/index.html
[`N0` preset]: https://docs.rs/iroh/latest/iroh/endpoint/presets/struct.N0.html
[iroh-blobs]: https://github.com/n0-computer/iroh-blobs
[iroh-gossip]: https://github.com/n0-computer/iroh-gossip
[iroh-ffi]: https://github.com/n0-computer/iroh-ffi
[n0]: https://n0.computer
+43
View File
@@ -0,0 +1,43 @@
# Bounded path retries
This is the crates.io iroh 1.2.0 source, with registry bookkeeping and its
independent Cargo.lock omitted. The workspace selects it with `[patch.crates-io]`.
- Published source commit: `17c0612f80f78f5288e97b818b1360ae6ea0a51a`.
- Crates.io archive SHA-256:
`b2f8d1cfffc83efe39a1031aab423ce09cb8048071baba550508931e9a81ce46`.
- Upstream source, manifests, examples, tests and license are otherwise retained.
`State::open_path_on_conn` appended a failed path once per connection. The actor
takes the queue every 333 ms and retries every entry on every connection to that
remote. Two failing connections therefore double duplicate work each round.
Tsunagi deliberately keeps control and data on separate QUIC connections, so
merging those connections would violate its architecture and conceal the bug.
On 2026-10-14 a Linux rc.33 daemon reached OOM after roughly five hours. Before
the kill, `/proc` showed anonymous mappings of 40 GiB and 10 GiB, later 20 GiB.
Pointers and container capacities were consistent with the retry queue and its
drained batch. The active worker's stack was in iroh's retry loop,
`open_path_on_all_conns` and `open_path_on_conn`, identified through executable
addresses and tracing metadata in the stripped static binary. This matches
[upstream issue 4509](https://github.com/n0-computer/iroh/issues/4509).
The local change extracts the enqueue operation into `pending_paths.rs`, rejects
duplicates, and limits pending candidates to 64, evicting the oldest before
admitting a new distinct path. The current retry batch and next batch are each
bounded; no wire formats, identity derivation, timers or authentication change.
The upstream deduplication fix was merged in
[PR 4551](https://github.com/n0-computer/iroh/pull/4551) on October 6. Both the
published 1.2.0 source used here and the 1.3.0 crates.io archive (released
September 28) still have the unconditional enqueue. Upgrading to 1.3.0 alone
does not fix this OOM. Remove this override when adopting a release that
includes the fix, retaining a retry regression check.
`crates/tsunagi-core/tests/iroh_path_retries.rs` includes the exact admission
module so its tests run in the default workspace suite, even though vendored
dependencies are excluded from workspace membership. The tests reproduce
multi-connection retry multiplication, preserve FIFO order under duplicate
failures, exercise success draining, and bound length and capacity under 10,000
distinct candidates. They fail with the original unconditional enqueue.
These are queue algorithm tests; the normal integration suite continues to use
real iroh endpoints, membership handshakes, SQLite and encryption.
+10
View File
@@ -0,0 +1,10 @@
use cfg_aliases::cfg_aliases;
fn main() {
// Setup cfg aliases
cfg_aliases! {
// Convenience aliases
wasm_browser: { all(target_family = "wasm", target_os = "unknown") },
with_crypto_provider: { any(feature = "tls-ring", feature = "tls-aws-lc-rs") }
}
}
+36
View File
@@ -0,0 +1,36 @@
# Using a local iroh-relay
It's easy to set up a iroh-relay that runs locally on your machine.
Using cargo:
```shell
$ cargo run --bin iroh-relay --features="iroh-relay" -- --dev
```
This will bind the iroh-relay to `[::]3340` and run it over HTTP.
To connect to this iroh-relay when doing your normal iroh commands, adjust the iroh configuration file to read:
```toml
# iroh.config.toml:
[[relays]]
url = "http://localhost:3340"
```
If you want to give a specific port for the iroh-relay to bind to, you can create a iroh-relay config file and pass that file in using the `--config_path` flag. You need to retain a `secret_key`, so it is recommended to run `iroh-relay --config-path [PATH]` once to generate a secret key and save it to the config file before doing further edits to the file.
To change the port you want to listen on, change the port in the `addr` field:
```
# iroh-relay.toml
secret_key = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
addr = "[::]:12345"
hostname = "my.relay.network"
enable_relay = true
```
Check [the iroh-relay file's](../src/bin/iroh-relay.rs) `Config` struct for documentation on each configuration field.
If you change the local iroh-relay server's configuration, however, be sure to adjust the associated fields in your iroh config as well.
+9
View File
@@ -0,0 +1,9 @@
# Relays
When an Iroh endpoint starts up, it does a latency test to see which known relay endpoint it is “closest to”. That relay server is considered the Iroh endpoint's home relay server.
An endpoint may be connected to multiple relay servers, but it will advertise its home relay endpoint as the one best used to hole-punch or relay packets through.
You do not need to know an endpoint's relay server in order to connect to them directly, if there are no firewalls or NATs between the two endpoints trying to connect. However, to have any hole punching, you must know at least one relay server to which that endpoint is connected.
We currently run 3 relays.
+181
View File
@@ -0,0 +1,181 @@
use std::{env, str::FromStr, time::Instant};
use clap::Parser;
use data_encoding::HEXLOWER;
use iroh::{
EndpointId, SecretKey,
endpoint::{RecvStream, SendStream, ZeroRttStatus, presets},
};
use n0_error::{Result, StackResultExt, StdResultExt};
use n0_future::StreamExt;
use tracing::{info, trace};
const PINGPONG_ALPN: &[u8] = b"0rtt-pingpong";
#[derive(Parser)]
struct Args {
/// The endpoint id to connect to. If not set, the program will start a server.
endpoint_id: Option<EndpointId>,
/// Number of rounds to run.
#[clap(long, default_value = "100")]
rounds: u64,
/// Run without 0-RTT for comparison.
#[clap(long)]
disable_0rtt: bool,
}
/// Gets a secret key from the IROH_SECRET environment variable or generates a new random one.
/// If the environment variable is set, it must be a valid string representation of a secret key.
pub fn get_or_generate_secret_key() -> Result<SecretKey> {
if let Ok(secret) = env::var("IROH_SECRET") {
// Parse the secret key from string
SecretKey::from_str(&secret).context("Invalid secret key format")
} else {
// Generate a new random key
let secret_key = SecretKey::generate();
println!(
"Generated new secret key: {}",
HEXLOWER.encode(&secret_key.to_bytes())
);
println!("To reuse this key, set the IROH_SECRET environment variable to this value");
Ok(secret_key)
}
}
/// Do a simple ping-pong with the given connection.
async fn pingpong(send: SendStream, recv: RecvStream, x: u64) -> Result<()> {
ping(send, x).await?;
pong(recv, x).await
}
async fn ping(mut send: SendStream, x: u64) -> Result<()> {
let data = x.to_be_bytes();
send.write_all(&data).await.anyerr()?;
send.finish().anyerr()
}
async fn pong(mut recv: RecvStream, x: u64) -> Result<()> {
let data = x.to_be_bytes();
let echo = recv.read_to_end(8).await.anyerr()?;
assert!(echo == data);
Ok(())
}
async fn connect(args: Args) -> Result<()> {
let remote_id = args.endpoint_id.unwrap();
let endpoint = iroh::Endpoint::builder(presets::N0)
.relay_mode(iroh::RelayMode::Disabled)
.keylog(true)
.bind()
.await?;
// ensure we have resolved the remote_id before connecting
// so we get a more accurate connection timing
let mut address_lookup_stream = endpoint.address_lookup()?.resolve(remote_id);
let _item = address_lookup_stream
.next()
.await
.context("failed to lookup remote")?;
let t0 = Instant::now();
for i in 0..args.rounds {
let t0 = Instant::now();
let connecting = endpoint
.connect_with_opts(remote_id, PINGPONG_ALPN, Default::default())
.await?;
let connection = if args.disable_0rtt {
let connection = connecting.await.anyerr()?;
trace!("connecting without 0-RTT");
let (send, recv) = connection.open_bi().await.anyerr()?;
pingpong(send, recv, i).await?;
connection
} else {
match connecting.into_0rtt() {
Ok(zrtt_connection) => {
trace!("0-RTT possible from our side");
let (send, recv) = zrtt_connection.open_bi().await.anyerr()?;
// before we get the full handshake, attempt to send 0-RTT data
let zrtt_task = tokio::spawn(ping(send, i));
match zrtt_connection.handshake_completed().await? {
ZeroRttStatus::Accepted(conn) => {
let _ = zrtt_task.await.anyerr()?;
pong(recv, i).await?;
conn
}
ZeroRttStatus::Rejected(conn) => {
zrtt_task.abort();
let (send, recv) = conn.open_bi().await.anyerr()?;
pingpong(send, recv, i).await?;
conn
}
}
}
Err(connecting) => {
trace!("0-RTT not possible from our side");
let conn = connecting.await.anyerr()?;
let (send, recv) = conn.open_bi().await.anyerr()?;
pingpong(send, recv, i).await?;
conn
}
}
};
connection.close(0u8.into(), b"");
let elapsed = t0.elapsed();
println!("round {i}: {} us", elapsed.as_micros());
}
let elapsed = t0.elapsed();
println!("total time: {} us", elapsed.as_micros());
println!(
"time per round: {} us",
elapsed.as_micros() / (args.rounds as u128)
);
Ok(())
}
async fn accept(_args: Args) -> Result<()> {
let secret_key = get_or_generate_secret_key()?;
let endpoint = iroh::Endpoint::builder(presets::N0)
.alpns(vec![PINGPONG_ALPN.to_vec()])
.secret_key(secret_key)
.relay_mode(iroh::RelayMode::Disabled)
.bind()
.await?;
println!("endpoint id: {}", endpoint.id());
let accept = async move {
while let Some(incoming) = endpoint.accept().await {
tokio::spawn(async move {
let accepting = incoming.accept().anyerr()?;
let connection = accepting.into_0rtt();
let (mut send, mut recv) = connection.accept_bi().await.anyerr()?;
trace!("recv.is_0rtt: {}", recv.is_0rtt());
let data = recv.read_to_end(8).await.anyerr()?;
trace!("recv: {}", data.len());
send.write_all(&data).await.anyerr()?;
send.finish().anyerr()?;
connection.closed().await;
n0_error::Ok(())
});
}
};
tokio::select! {
_ = accept => {
info!("accept finished, shutting down");
},
_ = tokio::signal::ctrl_c()=> {
info!("Ctrl-C received, shutting down");
}
}
Ok(())
}
#[tokio::main]
async fn main() -> Result<()> {
tracing_subscriber::fmt::init();
let args = Args::parse();
if args.endpoint_id.is_some() {
connect(args).await?;
} else {
accept(args).await?;
};
Ok(())
}
+355
View File
@@ -0,0 +1,355 @@
//! Implementation of authentication using iroh hooks
//!
//! This implements an auth protocol that works with iroh hooks.
//! It allows to put authentication in front of iroh protocols. The protocols don't need any special support.
//! Authentication is handled prior to establishing the connections, over a separate connection.
use iroh::{Endpoint, EndpointAddr, endpoint::presets, protocol::Router};
use n0_error::{Result, StdResultExt};
use crate::echo::Echo;
#[tokio::main]
async fn main() -> Result<()> {
tracing_subscriber::fmt::init();
let server_router = accept_side(b"secret!!").await?;
server_router.endpoint().online().await;
let server_addr = server_router.endpoint().addr();
println!("-- no --");
let res = connect_side_no_auth(server_addr.clone()).await;
println!("echo without auth: {:#}", res.unwrap_err());
println!("-- wrong --");
let res = connect_side(server_addr.clone(), b"dunno").await;
println!("echo with wrong auth: {:#}", res.unwrap_err());
println!("-- correct --");
let res = connect_side(server_addr.clone(), b"secret!!").await;
println!("echo with correct auth: {res:?}");
server_router.shutdown().await.anyerr()?;
Ok(())
}
async fn connect_side(remote_addr: EndpointAddr, token: &[u8]) -> Result<()> {
let (auth_hook, auth_task) = auth::outgoing(token.to_vec());
let endpoint = Endpoint::builder(presets::N0)
.hooks(auth_hook)
.bind()
.await?;
let _guard = auth_task.spawn(endpoint.clone());
Echo::connect(&endpoint, remote_addr, b"hello there!").await
}
async fn connect_side_no_auth(remote_addr: EndpointAddr) -> Result<()> {
let endpoint = Endpoint::bind(presets::N0).await?;
Echo::connect(&endpoint, remote_addr, b"hello there!").await
}
async fn accept_side(token: &[u8]) -> Result<Router> {
let (auth_hook, auth_protocol) = auth::incoming(token.to_vec());
let endpoint = Endpoint::builder(presets::N0)
.hooks(auth_hook)
.bind()
.await?;
let router = Router::builder(endpoint)
.accept(auth::ALPN, auth_protocol)
.accept(echo::ALPN, Echo)
.spawn();
Ok(router)
}
mod echo {
//! A bare-bones protocol with no knowledge of auth whatsoever.
use iroh::{
Endpoint, EndpointAddr,
endpoint::Connection,
protocol::{AcceptError, ProtocolHandler},
};
use n0_error::{Result, StdResultExt, anyerr};
#[derive(Debug, Clone)]
pub struct Echo;
pub const ALPN: &[u8] = b"iroh-example/echo/0";
impl Echo {
pub async fn connect(
endpoint: &Endpoint,
remote: impl Into<EndpointAddr>,
message: &[u8],
) -> Result<()> {
let conn = endpoint.connect(remote, ALPN).await?;
let (mut send, mut recv) = conn.open_bi().await.anyerr()?;
send.write_all(message).await.anyerr()?;
send.finish().anyerr()?;
let response = recv.read_to_end(1000).await.anyerr()?;
conn.close(0u32.into(), b"bye!");
if response == message {
Ok(())
} else {
Err(anyerr!("Received invalid response"))
}
}
}
impl ProtocolHandler for Echo {
async fn accept(&self, connection: Connection) -> Result<(), AcceptError> {
let (mut send, mut recv) = connection.accept_bi().await?;
tokio::io::copy(&mut recv, &mut send).await?;
send.finish()?;
connection.closed().await;
Ok(())
}
}
}
mod auth {
//! Authentication hook
use std::{
collections::{HashMap, HashSet, hash_map},
sync::{Arc, Mutex},
};
use iroh::{
Endpoint, EndpointAddr, EndpointId,
endpoint::{
AfterHandshakeOutcome, BeforeConnectOutcome, Connection, ConnectionError, EndpointHooks,
},
protocol::{AcceptError, ProtocolHandler},
};
use n0_error::{AnyError, Result, StackResultExt, StdResultExt, anyerr};
use n0_future::task::AbortOnDropHandle;
use tokio::{
sync::{mpsc, oneshot},
task::JoinSet,
};
use tracing::debug;
pub const ALPN: &[u8] = b"iroh-example/auth/0";
const CLOSE_ACCEPTED: u32 = 1;
const CLOSE_DENIED: u32 = 403;
/// Outgoing side: Use this if you want to pre-auth outgoing connections.
pub fn outgoing(token: Vec<u8>) -> (OutgoingAuthHook, OutgoingAuthTask) {
let (tx, rx) = mpsc::channel(16);
let hook = OutgoingAuthHook { tx };
let connector = OutgoingAuthTask {
token,
rx,
allowed_remotes: Default::default(),
pending_remotes: Default::default(),
tasks: JoinSet::new(),
};
(hook, connector)
}
type AuthResult = Result<(), Arc<AnyError>>;
/// Hook to mount on the endpoint builder.
#[derive(Debug)]
pub struct OutgoingAuthHook {
tx: mpsc::Sender<(EndpointId, oneshot::Sender<AuthResult>)>,
}
impl OutgoingAuthHook {
async fn authenticate(&self, remote_id: EndpointId) -> Result<()> {
let (tx, rx) = oneshot::channel();
self.tx
.send((remote_id, tx))
.await
.std_context("authenticator stopped")?;
rx.await
.std_context("authenticator stopped")?
.context("failed to authenticate")
}
}
impl EndpointHooks for OutgoingAuthHook {
async fn before_connect<'a>(
&'a self,
remote_addr: &'a EndpointAddr,
alpn: &'a [u8],
) -> BeforeConnectOutcome {
// Don't intercept auth request themsevles
if alpn == ALPN {
BeforeConnectOutcome::Accept
} else {
match self.authenticate(remote_addr.id).await {
Ok(()) => BeforeConnectOutcome::Accept,
Err(err) => {
debug!("authentication denied: {err:#}");
BeforeConnectOutcome::Reject
}
}
}
}
}
/// Connector task that initiates pre-auth request. Call [`Self::spawn`] once the endpoint is built.
pub struct OutgoingAuthTask {
token: Vec<u8>,
rx: mpsc::Receiver<(EndpointId, oneshot::Sender<AuthResult>)>,
allowed_remotes: HashSet<EndpointId>,
pending_remotes: HashMap<EndpointId, Vec<oneshot::Sender<AuthResult>>>,
tasks: JoinSet<(EndpointId, Result<()>)>,
}
impl OutgoingAuthTask {
pub fn spawn(self, endpoint: Endpoint) -> AbortOnDropHandle<()> {
AbortOnDropHandle::new(tokio::spawn(self.run(endpoint)))
}
async fn run(mut self, endpoint: Endpoint) {
loop {
tokio::select! {
msg = self.rx.recv() => {
let Some((remote_id, tx)) = msg else {
break;
};
self.handle_msg(&endpoint, remote_id, tx);
}
Some(res) = self.tasks.join_next(), if !self.tasks.is_empty() => {
let (remote_id, res) = res.expect("connect task panicked");
let res = res.map_err(Arc::new);
self.handle_task(remote_id, res);
}
}
}
}
fn handle_msg(
&mut self,
endpoint: &Endpoint,
remote_id: EndpointId,
tx: oneshot::Sender<Result<(), Arc<AnyError>>>,
) {
if self.allowed_remotes.contains(&remote_id) {
tx.send(Ok(())).ok();
} else {
match self.pending_remotes.entry(remote_id) {
hash_map::Entry::Occupied(mut entry) => {
entry.get_mut().push(tx);
}
hash_map::Entry::Vacant(entry) => {
let endpoint = endpoint.clone();
let token = self.token.clone();
self.tasks.spawn(async move {
let res = Self::connect(endpoint, remote_id, token).await;
(remote_id, res)
});
entry.insert(vec![tx]);
}
}
}
}
fn handle_task(&mut self, remote_id: EndpointId, res: Result<(), Arc<AnyError>>) {
if res.is_ok() {
self.allowed_remotes.insert(remote_id);
}
let senders = self.pending_remotes.remove(&remote_id);
for tx in senders.into_iter().flatten() {
tx.send(res.clone()).ok();
}
}
async fn connect(endpoint: Endpoint, remote_id: EndpointId, token: Vec<u8>) -> Result<()> {
let conn = endpoint.connect(remote_id, ALPN).await?;
let mut stream = conn.open_uni().await.anyerr()?;
stream.write_all(&token).await.anyerr()?;
stream.finish().anyerr()?;
let reason = conn.closed().await;
if let ConnectionError::ApplicationClosed(code) = &reason
&& code.error_code.into_inner() as u32 == CLOSE_ACCEPTED
{
Ok(())
} else if let ConnectionError::ApplicationClosed(code) = &reason
&& code.error_code.into_inner() as u32 == CLOSE_DENIED
{
Err(anyerr!("authentication denied by remote"))
} else {
Err(AnyError::from_std(reason))
}
}
}
/// Incoming side: Use this if you want to only accept connections from peers with successful pre-auth requests.
pub fn incoming(token: Vec<u8>) -> (IncomingAuthHook, AuthProtocol) {
let allowed_remotes: Arc<Mutex<HashSet<EndpointId>>> = Default::default();
let hook = IncomingAuthHook {
allowed_remotes: allowed_remotes.clone(),
};
let protocol = AuthProtocol {
allowed_remotes,
token,
};
(hook, protocol)
}
/// Accept-side auth hook: Mount this onto the endpoint.
///
/// This will reject incoming connections if the remote did not successfully authenticate before.
#[derive(Debug)]
pub struct IncomingAuthHook {
allowed_remotes: Arc<Mutex<HashSet<EndpointId>>>,
}
impl EndpointHooks for IncomingAuthHook {
async fn after_handshake<'a>(
&'a self,
conn: &'a iroh::endpoint::Connection,
) -> AfterHandshakeOutcome {
if conn.alpn() == ALPN
|| self
.allowed_remotes
.lock()
.expect("poisoned")
.contains(&conn.remote_id())
{
AfterHandshakeOutcome::Accept
} else {
AfterHandshakeOutcome::Reject {
error_code: 403u32.into(),
reason: b"not authenticated".to_vec(),
}
}
}
}
/// Accept-side auth protocol. Mount this on the router to accept authentication requests.
#[derive(Debug, Clone)]
pub struct AuthProtocol {
token: Vec<u8>,
allowed_remotes: Arc<Mutex<HashSet<EndpointId>>>,
}
impl ProtocolHandler for AuthProtocol {
/// The `accept` method is called for each incoming connection for our ALPN.
///
/// The returned future runs on a newly spawned tokio task, so it can run as long as
/// the connection lasts.
async fn accept(&self, connection: Connection) -> Result<(), AcceptError> {
let mut stream = connection.accept_uni().await?;
let token = stream.read_to_end(256).await.anyerr()?;
let remote_id = connection.remote_id();
if token == self.token {
self.allowed_remotes
.lock()
.expect("poisoned")
.insert(remote_id);
connection.close(CLOSE_ACCEPTED.into(), b"accepted");
} else {
connection.close(CLOSE_DENIED.into(), b"rejected");
}
Ok(())
}
}
}
+95
View File
@@ -0,0 +1,95 @@
//! The smallest example showing how to use iroh and [`iroh::Endpoint`] to connect to a remote endpoint and pass bytes using unreliable datagrams.
//!
//! We use the endpoint ID (the PublicKey of the remote endpoint), the direct UDP addresses, and the relay url to achieve a connection.
//!
//! This example uses the default relay servers to attempt to holepunch, and will use that relay server to relay packets if the two devices cannot establish a direct UDP connection.
//!
//! Run the `listen-unreliable` example first (`iroh/examples/listen-unreliable.rs`), which will give you instructions on how to run this example to watch two endpoints connect and exchange bytes.
use std::net::SocketAddr;
use clap::Parser;
use iroh::{Endpoint, EndpointAddr, RelayMode, RelayUrl, SecretKey, endpoint::presets};
use iroh_base::TransportAddr;
use n0_error::{Result, StdResultExt};
use tracing::info;
// An example ALPN that we are using to communicate over the `Endpoint`
const EXAMPLE_ALPN: &[u8] = b"n0/iroh/examples/0";
#[derive(Debug, Parser)]
struct Cli {
/// The id of the remote endpoint.
#[clap(long)]
endpoint_id: iroh::EndpointId,
/// The list of direct UDP addresses for the remote endpoint.
#[clap(long, value_parser, num_args = 1.., value_delimiter = ' ')]
addrs: Vec<SocketAddr>,
/// The url of the relay server the remote endpoint can also be reached at.
#[clap(long)]
relay_url: RelayUrl,
}
#[tokio::main]
async fn main() -> Result<()> {
tracing_subscriber::fmt::init();
println!("\nconnect (unreliable) example!\n");
let args = Cli::parse();
let secret_key = SecretKey::from_bytes(&rand::random());
println!("public key: {}", secret_key.public());
// Build a `Endpoint`, which uses PublicKeys as endpoint identifiers, uses QUIC for directly connecting to other endpoints, and uses the relay protocol and relay servers to holepunch direct connections between endpoints when there are NATs or firewalls preventing direct connections. If no direct connection can be made, packets are relayed over the relay servers.
let endpoint = Endpoint::builder(presets::N0)
// The secret key is used to authenticate with other endpoints. The PublicKey portion of this secret key is how we identify endpoints, often referred to as the `endpoint_id` in our codebase.
.secret_key(secret_key)
// Set the ALPN protocols this endpoint will accept on incoming connections
.alpns(vec![EXAMPLE_ALPN.to_vec()])
// `RelayMode::Default` means that we will use the default relay servers to holepunch and relay.
// Use `RelayMode::Custom` to pass in a `RelayMap` with custom relay urls.
// Use `RelayMode::Disable` to disable holepunching and relaying over HTTPS
// If you want to experiment with relaying using your own relay server, you must pass in the same custom relay url to both the `listen` code AND the `connect` code
.relay_mode(RelayMode::Default)
// You can choose an address to bind to, but passing in `None` will bind the socket to a random available port
.bind()
.await?;
// wait for the endpoint to be online
endpoint.online().await;
let endpoint_addr = endpoint.addr();
let me = endpoint_addr.id;
println!("endpoint id: {me}");
println!("endpoint listening addresses:");
endpoint_addr
.ip_addrs()
.for_each(|addr| println!("\t{addr}"));
let relay_url = endpoint_addr
.relay_urls()
.next()
.expect("Should have a relay URL, assuming a default endpoint setup.");
println!("endpoint relay server url: {relay_url}\n");
// Build a `EndpointAddr` from the endpoint_id, relay url, and UDP addresses.
let addrs = args
.addrs
.into_iter()
.map(TransportAddr::Ip)
.chain(std::iter::once(TransportAddr::Relay(args.relay_url)));
let addr = EndpointAddr::from_parts(args.endpoint_id, addrs);
// Attempt to connect, over the given ALPN.
// Returns a QUIC connection.
let conn = endpoint.connect(addr, EXAMPLE_ALPN).await?;
info!("connected");
// Send a datagram over the connection.
let message = format!("{me} is saying 'hello!'");
conn.send_datagram(message.as_bytes().to_vec().into())
.anyerr()?;
// Read a datagram over the connection.
let message = conn.read_datagram().await.anyerr()?;
let message = String::from_utf8(message.into()).anyerr()?;
println!("received: {message}");
Ok(())
}
+101
View File
@@ -0,0 +1,101 @@
//! The smallest example showing how to use iroh and [`iroh::Endpoint`] to connect to a remote endpoint.
//!
//! We use the endpoint ID (the PublicKey of the remote endpoint), the direct UDP addresses, and the relay url to achieve a connection.
//!
//! This example uses the default relay servers to attempt to holepunch, and will use that relay server to relay packets if the two devices cannot establish a direct UDP connection.
//!
//! Run the `listen` example first (`iroh/examples/listen.rs`), which will give you instructions on how to run this example to watch two endpoints connect and exchange bytes.
use std::net::SocketAddr;
use clap::Parser;
use iroh::{
Endpoint, EndpointAddr, RelayMode, RelayUrl, SecretKey, TransportAddr, endpoint::presets,
};
use n0_error::{Result, StdResultExt};
use tracing::info;
// An example ALPN that we are using to communicate over the `Endpoint`
const EXAMPLE_ALPN: &[u8] = b"n0/iroh/examples/0";
#[derive(Debug, Parser)]
struct Cli {
/// The id of the remote endpoint.
#[clap(long)]
endpoint_id: iroh::EndpointId,
/// The list of direct UDP addresses for the remote endpoint.
#[clap(long, value_parser, num_args = 1.., value_delimiter = ' ')]
addrs: Vec<SocketAddr>,
/// The url of the relay server the remote endpoint can also be reached at.
#[clap(long)]
relay_url: RelayUrl,
}
#[tokio::main]
async fn main() -> Result<()> {
tracing_subscriber::fmt::init();
println!("\nconnect example!\n");
let args = Cli::parse();
let secret_key = SecretKey::generate();
println!("public key: {}", secret_key.public());
// Build a `Endpoint`, which uses PublicKeys as endpoint identifiers, uses QUIC for directly connecting to other endpoints, and uses the relay protocol and relay servers to holepunch direct connections between endpoints when there are NATs or firewalls preventing direct connections. If no direct connection can be made, packets are relayed over the relay servers.
let endpoint = Endpoint::builder(presets::N0)
// The secret key is used to authenticate with other endpoints. The PublicKey portion of this secret key is how we identify endpoints, often referred to as the `endpoint_id` in our codebase.
.secret_key(secret_key)
// Set the ALPN protocols this endpoint will accept on incoming connections
.alpns(vec![EXAMPLE_ALPN.to_vec()])
// `RelayMode::Default` means that we will use the default relay servers to holepunch and relay.
// Use `RelayMode::Custom` to pass in a `RelayMap` with custom relay urls.
// Use `RelayMode::Disable` to disable holepunching and relaying over HTTPS
// If you want to experiment with relaying using your own relay server, you must pass in the same custom relay url to both the `listen` code AND the `connect` code
.relay_mode(RelayMode::Default)
// You can choose an address to bind to, but passing in `None` will bind the socket to a random available port
.bind()
.await?;
// wait for the endpoint to be online
endpoint.online().await;
let endpoint_addr = endpoint.addr();
let me = endpoint.id();
println!("endpoint id: {me}");
println!("endpoint listening addresses:");
for addr in endpoint_addr.ip_addrs() {
println!("\t{addr}")
}
let relay_url = endpoint_addr
.relay_urls()
.next()
.expect("should be connected to a relay server");
println!("endpoint relay server url: {relay_url}\n");
// Build a `EndpointAddr` from the endpoint_id, relay url, and UDP addresses.
let addrs = args
.addrs
.into_iter()
.map(TransportAddr::Ip)
.chain(std::iter::once(TransportAddr::Relay(args.relay_url)));
let addr = EndpointAddr::from_parts(args.endpoint_id, addrs);
// Attempt to connect, over the given ALPN.
// Returns a Noq connection.
let conn = endpoint.connect(addr, EXAMPLE_ALPN).await?;
info!("connected");
// Use the Noq API to send and recv content.
let (mut send, mut recv) = conn.open_bi().await.anyerr()?;
let message = format!("{me} is saying 'hello!'");
send.write_all(message.as_bytes()).await.anyerr()?;
// Call `finish` to close the send side of the connection gracefully.
send.finish().anyerr()?;
let message = recv.read_to_end(100).await.anyerr()?;
let message = String::from_utf8(message).anyerr()?;
println!("received: {message}");
// We received the last message: close all connections and allow for the close
// message to be sent.
endpoint.close().await;
Ok(())
}
+205
View File
@@ -0,0 +1,205 @@
use std::{sync::Arc, time::Duration};
use clap::Parser;
use iroh::{
Endpoint, SecretKey, TransportAddr,
endpoint::{
Builder, Connection, presets,
transports::{Addr, PathSelection, PathSelectionContext, PathSelector},
},
protocol::{AcceptError, ProtocolHandler, Router},
test_utils::test_transport::{TEST_TRANSPORT_ID, TestNetwork, TestTransport},
};
use n0_error::{Result, StdResultExt};
/// Each protocol is identified by its ALPN string.
///
/// The ALPN, or application-layer protocol negotiation, is exchanged in the connection handshake,
/// and the connection is aborted unless both endpoints pass the same bytestring.
const ALPN: &[u8] = b"iroh-example/echo/0";
/// Example demonstrating custom transport usage.
#[derive(Parser, Debug, Clone)]
struct Args {
/// Keep IP transports enabled (in addition to custom transport)
#[arg(long)]
keep_ip: bool,
/// Keep relay transports enabled (in addition to custom transport)
#[arg(long)]
keep_relay: bool,
/// Delay in seconds to wait after connecting before re-checking the selected transport
#[arg(long, default_value = "0")]
delay: u64,
}
/// A [`PathSelector`] that prefers the test custom transport whenever a candidate path on
/// it exists, falling back to the lowest-RTT candidate otherwise.
#[derive(Debug)]
struct PreferTestTransport;
impl PathSelector for PreferTestTransport {
fn select(&self, ctx: &PathSelectionContext<'_>) -> PathSelection {
tracing::debug!("dumping path RTTs");
for p in ctx.paths() {
let network_path = p.network_path();
let rtt = p.stats().map(|s| s.rtt);
tracing::debug!(%network_path, ?rtt);
}
let mut selection = PathSelection::none();
// First preference: any path on our test custom transport.
if let Some(p) = ctx.paths().find(
|p| matches!(p.network_path().remote(), Addr::Custom(c) if c.id() == TEST_TRANSPORT_ID),
) {
selection.set(&p);
return selection;
}
// Otherwise: lowest RTT wins. Paths whose stats can't be read (closed
// concurrently with selection) are skipped entirely.
if let Some(p) = ctx
.paths()
.filter_map(|p| p.stats().map(|s| (p, s.rtt)))
.min_by_key(|(_, rtt)| *rtt)
.map(|(p, _)| p)
{
selection.set(&p);
}
selection
}
}
impl Args {
/// Configure an endpoint builder with the custom transport and optional IP/relay transports.
fn configure(&self, secret_key: SecretKey, transport: Arc<TestTransport>) -> Builder {
let mut builder = Endpoint::builder(presets::N0)
.secret_key(secret_key)
.preset(transport)
// Always prefer the custom transport when it has a working path.
.path_selector(Arc::new(PreferTestTransport));
if !self.keep_ip {
builder = builder.clear_ip_transports();
}
if !self.keep_relay {
builder = builder.clear_relay_transports();
}
builder
}
}
#[derive(Debug, Clone)]
struct Echo;
impl ProtocolHandler for Echo {
/// The `accept` method is called for each incoming connection for our ALPN.
///
/// The returned future runs on a newly spawned tokio task, so it can run as long as
/// the connection lasts.
async fn accept(&self, connection: Connection) -> Result<(), AcceptError> {
// We can get the remote's endpoint id from the connection.
let endpoint_id = connection.remote_id();
println!("accepted connection from {endpoint_id}");
// Our protocol is a simple request-response protocol, so we expect the
// connecting peer to open a single bi-directional stream.
let (mut send, mut recv) = connection.accept_bi().await?;
// Echo any bytes received back directly.
// This will keep copying until the sender signals the end of data on the stream.
let bytes_sent = tokio::io::copy(&mut recv, &mut send).await?;
println!("Copied over {bytes_sent} byte(s)");
// By calling `finish` on the send stream we signal that we will not send anything
// further, which makes the receive stream on the other end terminate.
send.finish()?;
// Wait until the remote closes the connection, which it does once it
// received the response.
connection.closed().await;
Ok(())
}
}
#[tokio::main]
async fn main() -> Result<()> {
tracing_subscriber::fmt::init();
let args = Args::parse();
println!(
"Config: keep_ip={}, keep_relay={}, delay={}s",
args.keep_ip, args.keep_relay, args.delay
);
let network = TestNetwork::new();
let s1 = SecretKey::from([0u8; 32]);
let s2 = SecretKey::from([1u8; 32]);
// Create transports and configure builders with transport + address lookup
let t1 = network.create_transport(s1.public())?;
let ep1 = args.configure(s1.clone(), t1).bind().await?;
let t2 = network.create_transport(s2.public())?;
let ep2 = args.configure(s2.clone(), t2).bind().await?;
println!("ep2 addr: {:?}", ep2.addr());
let server = Router::builder(ep2).accept(ALPN, Echo).spawn();
// Connect using just the endpoint ID - discovery will resolve addresses
// Note: The test network's discovery is very fast (in-memory), so the custom
// transport address is available immediately and wins before IP discovery runs.
println!("Connecting to: {:?}", s2.public());
let conn = ep1.connect(s2.public(), ALPN).await?;
// Helper to print paths and verify test transport is selected
let verify_test_transport = |label: &str| {
let paths = conn.paths();
println!("Paths {}:", label);
for path in paths.iter() {
println!(
" {} selected={} rtt={:?}",
path.remote_addr(),
path.is_selected(),
path.rtt()
);
}
let selected_path = paths.iter().find(|p| p.is_selected());
let is_test_transport = selected_path.as_ref().is_some_and(|p| {
matches!(p.remote_addr(), TransportAddr::Custom(addr) if addr.id() == TEST_TRANSPORT_ID)
});
assert!(
is_test_transport,
"Expected test transport (id={}) to be selected {}, got: {:?}",
TEST_TRANSPORT_ID,
label,
selected_path.as_ref().map(|p| p.remote_addr())
);
println!(
"Verified: test transport (id={}) is selected {}",
TEST_TRANSPORT_ID, label
);
};
// Verify test transport is selected immediately after connecting
verify_test_transport("immediately after connecting");
// If a delay is specified, wait and then re-check to see if the transport is still selected
// after other discovery mechanisms have had time to run.
if args.delay > 0 {
println!(
"Waiting {}s to let other discovery mechanisms run...",
args.delay
);
tokio::time::sleep(Duration::from_secs(args.delay)).await;
verify_test_transport(&format!("after {}s delay", args.delay));
}
let (mut send, mut recv) = conn.open_bi().await.anyerr()?;
send.write_all(b"Hello custom transport!").await.anyerr()?;
send.finish().anyerr()?;
let response = recv.read_to_end(1000).await.anyerr()?;
assert_eq!(&response, b"Hello custom transport!");
conn.close(0u32.into(), b"bye!");
server.shutdown().await.anyerr()?;
drop(server);
Ok(())
}
+121
View File
@@ -0,0 +1,121 @@
//! Very basic example showing how to implement a basic echo protocol,
//! without using the `Router` API. (For the router version, check out the echo.rs example.)
//!
//! The echo protocol echos any data sent to it in the first stream.
//!
//! ## Running the Example
//!
//! cargo run --example echo-no-router
use iroh::{Endpoint, EndpointAddr, endpoint::presets};
use n0_error::{AnyError as Error, Result, StdResultExt};
/// Each protocol is identified by its ALPN string.
///
/// The ALPN, or application-layer protocol negotiation, is exchanged in the connection handshake,
/// and the connection is aborted unless both endpoints pass the same bytestring.
const ALPN: &[u8] = b"iroh-example/echo/0";
#[tokio::main]
async fn main() -> Result<()> {
let endpoint = start_accept_side().await?;
// wait for the endpoint to be online
endpoint.online().await;
connect_side(endpoint.addr()).await?;
// This makes sure the endpoint is closed properly and connections close gracefully
// and will indirectly close the tasks spawned by `start_accept_side`.
endpoint.close().await;
Ok(())
}
async fn connect_side(addr: EndpointAddr) -> Result<()> {
let endpoint = Endpoint::bind(presets::N0).await?;
// Open a connection to the accepting endpoint
let conn = endpoint.connect(addr, ALPN).await?;
// Open a bidirectional QUIC stream
let (mut send, mut recv) = conn.open_bi().await.anyerr()?;
// Send some data to be echoed
send.write_all(b"Hello, world!").await.anyerr()?;
// Signal the end of data for this particular stream
send.finish().anyerr()?;
// Receive the echo, but limit reading up to maximum 1000 bytes
let response = recv.read_to_end(1000).await.anyerr()?;
assert_eq!(&response, b"Hello, world!");
// Explicitly close the whole connection.
conn.close(0u32.into(), b"bye!");
// The above call only queues a close message to be sent (see how it's not async!).
// We need to actually call this to make sure this message is sent out.
endpoint.close().await;
// If we don't call this, but continue using the endpoint, then the queued
// close call will eventually be picked up and sent.
// But always try to wait for endpoint.close().await to go through before dropping
// the endpoint to ensure any queued messages are sent through and connections are
// closed gracefully.
Ok(())
}
async fn start_accept_side() -> Result<Endpoint> {
let endpoint = Endpoint::builder(presets::N0)
// The accept side needs to opt-in to the protocols it accepts,
// as any connection attempts that can't be found with a matching ALPN
// will be rejected.
.alpns(vec![ALPN.to_vec()])
.bind()
.await?;
// spawn a task so that `start_accept_side` returns immediately and we can continue in main().
tokio::spawn({
let endpoint = endpoint.clone();
async move {
// This task won't leak, because we call `endpoint.close()` in `main()`,
// which causes `endpoint.accept().await` to return `None`.
// In a more serious environment, we recommend avoiding `tokio::spawn` and use either a `TaskTracker` or
// `JoinSet` instead to make sure you're not accidentally leaking tasks.
while let Some(incoming) = endpoint.accept().await {
// spawn a task for each incoming connection, so we can serve multiple connections asynchronously
tokio::spawn(async move {
let connection = incoming.await.anyerr()?;
// We can get the remote's endpoint id from the connection.
let endpoint_id = connection.remote_id();
println!("accepted connection from {endpoint_id}");
// Our protocol is a simple request-response protocol, so we expect the
// connecting peer to open a single bi-directional stream.
let (mut send, mut recv) = connection.accept_bi().await.anyerr()?;
// Echo any bytes received back directly.
// This will keep copying until the sender signals the end of data on the stream.
let bytes_sent = tokio::io::copy(&mut recv, &mut send).await.anyerr()?;
println!("Copied over {bytes_sent} byte(s)");
// By calling `finish` on the send stream we signal that we will not send anything
// further, which makes the receive stream on the other end terminate.
send.finish().anyerr()?;
// Wait until the remote closes the connection, which it does once it
// received the response.
connection.closed().await;
Ok::<_, Error>(())
});
}
Ok::<_, Error>(())
}
});
Ok(endpoint)
}
Loaded 100 of 168 files, more files were not shown because too many files have changed in this diff. Show more