ab/tsunagi
1
0 Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Compare commits

base: ab/tsunagi:v0.1.0-rc.31
ab/tsunagi:master ab/tsunagi:worktree-relay-unify ab/tsunagi:worktree-readme-rewrite
ab/tsunagi:v0.1.0-rc.37 ab/tsunagi:v0.1.0-rc.36 ab/tsunagi:v0.1.0-rc.35 ab/tsunagi:v0.1.0-rc.34 ab/tsunagi:v0.1.0-rc.33 ab/tsunagi:v0.1.0-rc.32 ab/tsunagi:v0.1.0-rc.31 ab/tsunagi:v0.1.0-rc.30 ab/tsunagi:v0.1.0-rc.29 ab/tsunagi:v0.1.0-rc.28 ab/tsunagi:v0.1.0-rc.27 ab/tsunagi:v0.1.0-rc.26 ab/tsunagi:v0.1.0-rc.25 ab/tsunagi:v0.1.0-rc.24 ab/tsunagi:v0.1.0-rc.23 ab/tsunagi:v0.1.0-rc.22 ab/tsunagi:v0.1.0-rc.21 ab/tsunagi:v0.1.0-rc.20 ab/tsunagi:v0.1.0-rc.19 ab/tsunagi:v0.1.0-rc.18 ab/tsunagi:v0.1.0-rc.17 ab/tsunagi:v0.1.0-rc.16 ab/tsunagi:v0.1.0-rc.15 ab/tsunagi:v0.1.0-rc.14 ab/tsunagi:v0.1.0-rc.13 ab/tsunagi:v0.1.0-rc.12 ab/tsunagi:v0.1.0-rc.11 ab/tsunagi:v0.1.0-rc.10 ab/tsunagi:v0.1.0-rc.9 ab/tsunagi:v0.1.0-rc.8 ab/tsunagi:v0.1.0-rc.7 ab/tsunagi:v0.1.0-rc.6 ab/tsunagi:v0.1.0-rc.5 ab/tsunagi:v0.1.0-rc.3 ab/tsunagi:v0.1.0-rc.2 ab/tsunagi:v0.1.0-rc.1
...
compare: ab/tsunagi:master
ab/tsunagi:master ab/tsunagi:worktree-relay-unify ab/tsunagi:worktree-readme-rewrite
ab/tsunagi:v0.1.0-rc.37 ab/tsunagi:v0.1.0-rc.36 ab/tsunagi:v0.1.0-rc.35 ab/tsunagi:v0.1.0-rc.34 ab/tsunagi:v0.1.0-rc.33 ab/tsunagi:v0.1.0-rc.32 ab/tsunagi:v0.1.0-rc.31 ab/tsunagi:v0.1.0-rc.30 ab/tsunagi:v0.1.0-rc.29 ab/tsunagi:v0.1.0-rc.28 ab/tsunagi:v0.1.0-rc.27 ab/tsunagi:v0.1.0-rc.26 ab/tsunagi:v0.1.0-rc.25 ab/tsunagi:v0.1.0-rc.24 ab/tsunagi:v0.1.0-rc.23 ab/tsunagi:v0.1.0-rc.22 ab/tsunagi:v0.1.0-rc.21 ab/tsunagi:v0.1.0-rc.20 ab/tsunagi:v0.1.0-rc.19 ab/tsunagi:v0.1.0-rc.18 ab/tsunagi:v0.1.0-rc.17 ab/tsunagi:v0.1.0-rc.16 ab/tsunagi:v0.1.0-rc.15 ab/tsunagi:v0.1.0-rc.14 ab/tsunagi:v0.1.0-rc.13 ab/tsunagi:v0.1.0-rc.12 ab/tsunagi:v0.1.0-rc.11 ab/tsunagi:v0.1.0-rc.10 ab/tsunagi:v0.1.0-rc.9 ab/tsunagi:v0.1.0-rc.8 ab/tsunagi:v0.1.0-rc.7 ab/tsunagi:v0.1.0-rc.6 ab/tsunagi:v0.1.0-rc.5 ab/tsunagi:v0.1.0-rc.3 ab/tsunagi:v0.1.0-rc.2 ab/tsunagi:v0.1.0-rc.1
8 Commits
v0.1.0-rc.31 ... master
Author SHA1 Message Date
Aleksandr Bogomiakov 2166ba4800 Separate internal WireGuard encryption from data carriers 
AWG-only policy also selected a competing end-to-end plugin, so compatible
external links could still have no compatible encrypted data channel. Carrier
changes replaced tunnel state and status selected counters from different
engines. Use one internal WireGuard plugin and a separate DataCarrier contract
for external capabilities/lifecycle. Keep the logical link, keys, handshake,
interface and counter epoch across policy, fallback and direct/relay changes.
Control/discovery transport selection and membership remain independent.

Share the bounded per-neighbour UDP WireGuard engine between ordinary WG and
Amnezia masking. Encrypt the relay envelope inside the outer hop channel with
fresh pair keys/PSK. Remove duplicate WG key/device code and the TCP/TLS inner
cipher. TLS carries opaque internal WireGuard frames. Refuse missing native
transports instead of letting default iroh impersonate their protocol names.
Probe the actual authenticated carrier, independently of the shared IP tunnel;
failed local probe submissions count as loss too.

Apply the vendored iroh interface exclusion to actual direct candidates, not
just public discovery records. Exclude overlay subnets from introductions and
propagate the actual owned interface name, including kernel-assigned utun names,
to every carrier. Production logs showed paths through the overlay itself.
This removes a recursive transport path; it does not establish the historical
cause of the reported 100 GB absolute counter. Exit-node traffic is also real
peer IP traffic even when local applications are idle.

Hide inner cipher choices/handshake clutter in CLI and tray. Report external
routes, readiness and actionable policy/version failures. Count original IP
packets only when their ciphertext is accepted by the logical transport,
including queued packets' exact unpadded lengths. Exclude physical probes and
keepalives. Export counter epochs; sum per-peer rate deltas and discard stale
histories so joins/restarts cannot turn old totals into fresh traffic. Status
also prints exact IP byte totals for comparison with the tray.

Coordinated upgrade: data ALPN 6, internal wireguard-e2e capability v4,
wg/awg v2, QUIC carrier v1, TLS v2, local IPC v23. Control ALPN 8, device/network
identity derivation, signed records and saved network choices stay unchanged.
Keep the historical wg-quic key store; transactionally import missing keys from
wg then awg without replacing canonical keys. Forget removes legacy copies too,
on the blocking pool, to prevent resurrection after restart. Preserve historic
cipher options as shared settings, rejecting contradictory values.

Validation on Linux in the Nix development shell:
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets (562 passed, 3 ignored)
The inherited service socket/secret/group environment overrides were removed
for isolated CLI process tests. A new carrier test initially raced the first
peer announcement; its deadline polling now waits for a peer row explicitly.
Real loopback endpoints, membership, SQLite, WG/AWG/TLS encryption and memory
TUNs cover mixed-carrier relaying and one smoltcp TCP connection surviving
relay/direct/policy changes and a dropped transmission. Counter tests check
exact plaintext lengths, refused sends, epochs, joins and stale histories.
Public DHT, privileged kernel and manual benchmark tests remain ignored;
macOS/Windows native TUNs, desktop GUI and WAN/DPI behavior were not run.
 2026-10-22 01:07:52 +01:00
Aleksandr Bogomiakov 8a6216bccb Expose tray runtime libraries in the Nix development shell 
The development shell inherited build dependencies but Cargo-built tray
binaries lack the installed package's wrapper. AppIndicator was present in
the store yet its dlopen lookup failed because the runtime library path was
missing. The same problem affected dynamically loaded graphics libraries.

Expose the package's existing trayRuntimeLibs through passthru and reuse that
list in mkShell packages and its LD_LIBRARY_PATH hook. Include the existing
NixOS OpenGL driver path and preserve any caller-supplied library paths.
The development shell and installed tray now share one dependency list.
Document direct execution of Cargo-built tray binaries inside nix develop.
No agent, protocol, package version or system configuration changes.

Validation on Linux:
- Reproduced the missing AppIndicator dlopen failure before the change.
- Loaded libayatana-appindicator3.so.1, libxkbcommon-x11.so.0, libEGL.so.1,
  libGL.so.1 and libwayland-client.so.0 inside the updated shell.
- Verified AppIndicator still loads with an inherited LD_LIBRARY_PATH and
  that the existing value is preserved. The first probe used an older host
  Python with an incompatible glibc; probes using Python from the pinned
  nixpkgs input passed.
- Evaluated the x86_64-linux CLI package and shared runtime dependency list,
  and the aarch64-linux development shell; no aarch64 execution claimed.
- nixfmt --check flake.nix nix/package.nix
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets: 566 passed, 3 ignored
Cargo checks ran through nix develop, with TSUNAGI_CONTROL_SOCKET unset for
Clippy and tests. Existing agent and tray processes were not restarted.
 2026-10-21 17:05:25 +01:00
Aleksandr Bogomiakov 2e7e238459 Improve tray VIP monitoring and table layout 
The VIP editor put routing and traffic observations in a dense text block,
while nested layouts in the devices grid could let text escape its row.
Separate the VIP overview from configuration and draw both live tables with
explicit cell bounds, font-aware row heights, clipping and horizontal overflow.
Measure columns before painting and keep widths stable as live values change.

The overview reuses the existing traffic sparklines for service totals and
individual listener backends, with TX/RX rates, health, RTT, open flows and
counters. Retain bounded histories by service/listener/backend identity and
release them when their resources disappear. Recreated listeners start fresh.

Use egui 0.31 Scene and Painter for a pannable, zoomable routing diagram with
Fit, clickable backend details and linked table selection. Highlight recent
counter growth separately from the backend selected for new connections;
idle bindings alone do not imply traffic, and stale activity expires. Draw
legend markers directly rather than depending on optional font glyphs.
The overview only reads saved observations; edits and deletion still require
Save. No dependencies, wire formats, IPC or persistent schemas change.

Validation on Linux through nix develop:
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets: 566 passed, 3 ignored
- cargo build --locked -p tsunagi-tray
The inherited TSUNAGI_CONTROL_SOCKET was unset for Clippy and tests.
Seven new GUI tests cover enlarged fonts, mixed online/offline rows, long
names, large counters, narrow/wide layouts, scene clicks, traffic selection,
stale samples, counter resets, listener recreation and history cleanup.
The first enlarged-font regression exposed an override-font height mismatch;
row sizing now resolves the actual selected font and the regression passes.
A disposable fixture preview rendered successfully with software OpenGL on
an isolated Xvfb display. Initial Xvfb attempts needed explicit xkbcommon and
OpenGL runtime paths in the Nix environment. The preview never contacted an
agent; its temporary source was removed. macOS, Windows, Wayland and live host
TUN traffic were not exercised by this UI change.
 2026-10-21 16:57:27 +01:00
Aleksandr Bogomiakov 23af1a2d9d Add shared TCP and UDP virtual services 
Provide named VIPs with independent protocol/port listeners and node/port
backends so applications can use one DNS endpoint without a separate load
balancer. Every authenticated member may create, edit or delete any service.
Client agents select the eligible backend with the lowest measured encrypted
data-plane RTT, including relayed paths, and pin existing flows. UDP recovers
on another backend after failure; established TCP is never replayed or moved.
An unanswered empty TCP SYN may retry another backend before establishment.

Keep signed multi-writer field registers in state/vip.rs. Independent backend
edits merge, same-field conflicts use Lamport/author/canonical ordering, and
listener generations prevent old backend entries returning on port recreation.
Permanent service tombstones protect replay while allowing obsolete fields to
be compacted. Active-service and deletion-history limits are separate. Commit
complete snapshots and the author's durable counter atomically before local
routing changes or publication. Restore counters from newer genuine author
updates learned from replicas after an old backup is restored.

Implement bounded client-side address/port NAT at overlay ingress and reverse
translation only after authenticated source/destination admission. Reserve OS
source ports to avoid direct-flow collisions and quarantine retired ports for
late replies. Handle a backend on the client itself, scoped fragment assembly,
checksums, re-fragmentation and quoted ICMP/PMTU headers. Plugins and encrypted
transit retain their existing contracts. Data-path probes measure RTT; bounded
fair TCP connect checks are enabled by default and configurable per listener.
Arbitrary UDP application readiness remains unknown; path checks and explicit
port errors drive eligibility without protocol-specific probe payloads.

Expose the shared draft API through local IPC, simple vip/listener/backend CLI
commands, JSON, editor-based atomic updates and watch output. Add a VIP button
on each network tile and a separate tray editor. Edits and removals stay local
until Save; stale revisions retain the draft, and closing a dirty window offers
explicit discard. Report local routing choices, health reasons, RTT age, active
and draining bindings, IP counters, rates, drops and UDP failovers. Extend DNS
with service names; keep numeric addresses secondary and preserve existing
bindings while VIP addresses or listeners change.

This changes control wire ALPN/version to tsunagi/ctrl/8 and local IPC to 22;
all members and local clients must upgrade together. Data ALPN remains 5.
SQLite schema 10 migrates existing mandatory state without changing identity,
secrets, network ids, derivation labels, handshake encoding or node records.
Crate versions and release tags are unchanged. No installed agent or tray was
replaced. Commands, limits and operational semantics are in docs/vip.md.

Checks on Linux in nix develop:
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets: 559 passed, 3 ignored
- git diff --cached --check

The full suite includes real encrypted TCP/UDP VIP traffic, backend-stop UDP
failover, real loopback TCP check recovery, source admission, signed merges,
stale/deleted state, SQLite migration and rollback, restart/author recovery,
network isolation, actual CLI/DNS processes, and headless text-entry/Save-button
interaction. Discovery and memory TUNs are the existing test substitutions;
authentication, transport, encryption and SQLite remain real.

Initial checks outside Nix failed for missing GTK pkg-config metadata, and an
initial CLI test inherited TSUNAGI_CONTROL_SOCKET. Final checks use the Nix
environment with that variable unset so tests address their temporary sockets.
Root/internet-dependent tests remain ignored. No live host TUN, native desktop
session, macOS or Windows validation was performed.
 2026-10-21 15:40:45 +01:00
Aleksandr Bogomiakov 94318e7c16 Keep tray device metrics on one line 
The device table sized handshake/RTT and packet columns from fixed samples.
Its own header exceeded the handshake sample, and six-digit counters wrapped
at the slash, making device and total rows taller and misaligning the plots.
Measure actual headers, displayed values and totals before drawing, retain
the widest values across frames, and disable wrapping inside table cells.
When a column grows, request a second egui layout pass so the grid's cached
positions settle before larger text is painted over the next column.
Keep the plot as the stretch column and horizontal scrolling for narrow
windows. This changes only tray presentation.

A headless rendering regression checks the actual text shapes for the
header, handshake age/control RTT, per-device packet counts and totals as
counts grow past five digits, reach billions, shrink again, and the table
gets a narrower container. It checks one-line text and non-overlap with the
next column; it failed on the original wrapped header. Existing resize and
long-relay tests remain in place.

Checks on Linux in nix develop:
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets, with inherited
  TSUNAGI_CONTROL_SOCKET unset so tests use their temporary sockets
- git diff --cached --check

The suite's root/internet-dependent tests remain ignored. No native GUI
session or checks on macOS/Windows were run; rendering was checked through
egui's headless output. No installed tray binary was replaced.
 2026-10-14 23:37:38 +01:00
Aleksandr Bogomiakov 2b0c938487 Bound iroh path retries to prevent multi-connection OOM 
A long-running rc.33 daemon reached OOM with roughly 22 GiB resident and
24 GiB swapped. Live inspection found 40 GiB and 10/20 GiB anonymous
container mappings; the active worker stack and tracing metadata identified
iroh's scheduled path-open retry loop. In iroh 1.2.0 every failed connection
requeues the same path, while each retry visits every connection again.
Separate control and data connections multiply the queue geometrically.

Vendor the published iroh 1.2.0 source and select it with a workspace Cargo
patch. Deduplicate retry admission and cap distinct pending paths at 64,
evicting the oldest before inserting a new candidate. Keep the current and
next retry batches bounded without combining control/data connections or
changing wire protocols, identity derivation, authentication or state.
Document source provenance, the live evidence and removal conditions.
Upstream PR 4551 merged on October 6, but inspection of the official 1.3.0
crate still shows the unconditional enqueue, so that upgrade alone is not
a fix. No version bump, release tag or installed-service change.

The default workspace suite includes the exact vendored admission module.
Three regressions failed with the original enqueue and pass with the fix:
repeated failures across connections, duplicate admission at capacity, and
10,000 distinct candidates with bounded length/capacity and FIFO eviction.
Transport integration tests continue using real endpoints and encryption.

Checks on Linux in nix develop:
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets, with only the inherited
  TSUNAGI_CONTROL_SOCKET removed from the test environment
- git diff --cached --check

Initial full-suite runs failed six DNS readiness checks because the inherited
system socket path overrode their temporary state directories. Clearing that
environment variable resolves those failures without changing application code.
Root/network-dependent ignored tests and other operating systems were not run.
 2026-10-14 23:28:53 +01:00
Ultradesu 4e235cbac6 Keep device data paths on one line 
The tray used a fixed sample for the data-path column, so longer relay names wrapped and made device rows uneven. Measure each rendered path and retain the widest value, then render path labels without wrapping. Replace direct/via words with embedded Phosphor route icons while preserving transport, relay name, hop count and explanatory tooltips. The app already ships the font; keep its setup in one function and verify registration alongside a long-relay-width regression test. No wire or stored-state changes. Checks: cargo fmt --all -- --check; cargo clippy --locked --workspace --all-targets -- -D warnings; cargo test --locked --workspace --all-targets.
 2026-10-14 13:50:05 +01:00
Aleksandr Bogomiakov 2631eb6975 Protect Linux exit-node connections and tidy the device table 
Using an exit node could destroy direct peer connections: the agent's UID
bypass protected outbound routing, but reverse-path checks look up replies
without that UID and selected the overlay default. A real-host A/B check
confirmed that relaxing only strict reverse-path filtering restored direct
connections and exit HTTP traffic; the diagnostic host changes were removed.

Keep strict filtering and protect the Linux underlay with a dedicated masked
mark bit, 0x40000000. Mark the dedicated service UID in OUTPUT, save only that
bit in conntrack and restore it before PREROUTING reverse-path checks. Install
IPv4/IPv6 mark and UID main-table bypass rules before Tailscale's priorities;
leave application-specific and exit-default routing after them. Preserve
foreign mark bits, main-table routes and foreign firewall/policy rules.

Protect the first inbound TCP/UDP packet using bounded, locally owned socket
addresses/ports from /proc and an actual socket match. Refresh socket protection
before sending private transport negotiations and on discovery ticks, without
firewall writes for an unchanged socket set. Remote announcements never become
OS arguments. A failed refresh leaves active exit routing in place. Install
protection before routes, remove routes before protection, roll back partial
activation, and remove only exact interface tags, including legacy UID rules.

Reject strict kernel rp_filter without src_valid_mark with an actionable error;
never weaken filtering or change host sysctls. Document mark-aware Netfilter
requirements and provide the trusted ip6tables path in the NixOS module.
Other platforms retain their existing exit implementation. Peer wire formats,
identity derivation, local IPC and persistent storage are unchanged.

Make the tray devices grid fill and shrink with the window, constrain the first
cell and give remaining width to traffic plots. Keep displayed end-to-end
protocol and its short encryption tooltip consistent, independent of carrier
protocol. Move handshake/session age into its own column beside the existing
iroh control RTT. Use stable per-device hover identities and concise tooltips;
missing protocols/measurements remain explicit rather than invented.

Validation on Linux through nix develop:
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets: 529 passed, 3 ignored;
  inherited TSUNAGI_CONTROL_SOCKET and TSUNAGI_CONTROL_GROUP were unset
- explicit privileged kernel regression: 1 passed, using disposable namespaces,
  real TCP/UDP IPv4/IPv6, strict kernel and Netfilter RPF, first inbound packets,
  dynamic UDP binding, application exit routing/IPv6 blocking, repeated apply
  and exact cleanup with foreign Tailscale-priority rules retained
- headless egui grid/scroll-area rendering at multiple sizes and control RTT,
  mark masks/order, passive socket parsing, partial failure and cleanup tests
- git diff --check

Earlier iterations caught rtnetlink family API and nonexhaustive initializer
errors, Clippy warnings, egui's implicit minimum column width, and kernel-test
fixture issues (wildcard echo source, blocked IPv6 neighbour discovery and a
closed child stdout pipe). Corrected before final checks; the stdout-only test
fix was checked with fmt, Clippy and the explicit kernel regression afterward.
Native tray rendering and macOS/Windows checks were not run. The running host
agent was not replaced or restarted. Requested release tag: v0.1.0-rc.32.
 2026-10-14 10:40:30 +01:00
190 changed files with 47644 additions and 7027 deletions
Expand all files Collapse all files

No files matched your search

AGENTS.md
+1 -1
View File
@@ -162,7 +162,7 @@ Keep these separate. Crossing them is the main thing to review for.
| `crates/tsunagi-core/src/overlay/` | the one interface an agent owns: provisioning, the TUN, routing, source checks, broadcast host rules | | `crates/tsunagi-core/src/overlay/` | the one interface an agent owns: provisioning, the TUN, routing, source checks, broadcast host rules |
| `crates/tsunagi-core/src/dataplane/` | the protocol contract and the authenticated packet transport | | `crates/tsunagi-core/src/dataplane/` | the protocol contract and the authenticated packet transport |
| `crates/tsunagi-core/src/dns/` | the DNS view of a network: zone, server, resolver publication | | `crates/tsunagi-core/src/dns/` | the DNS view of a network: zone, server, resolver publication |
| `crates/tsunagi-wg-quic/` | the `wg-quic` protocol: its keys, its announcement, its tunnels | | `crates/tsunagi-wg-quic/` | the single internal WireGuard cipher, keys and opaque announcement; historical crate name |
| `crates/tsunagi/` | the command line agent, built as `tsng`; the only place that owns a runtime, a logger and signals | | `crates/tsunagi/` | the command line agent, built as `tsng`; the only place that owns a runtime, a logger and signals |
| `tests/` | integration tests; `tests/common/` is the shared harness | | `tests/` | integration tests; `tests/common/` is the shared harness |
Cargo.lock
Generated
+90 -22
View File
@@ -601,6 +601,12 @@ dependencies = [
"syn 3.0.6", "syn 3.0.6",
] ]
[[package]]
name = "byteorder"
version = "1.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b"
[[package]] [[package]]
name = "byteorder-lite" name = "byteorder-lite"
version = "0.1.0" version = "0.1.0"
@@ -1171,6 +1177,46 @@ dependencies = [
"syn 3.0.6", "syn 3.0.6",
] ]
[[package]]
name = "defmt"
version = "0.3.101"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4deef977660e373c458df605a913d4ab121db0ccd18d1ad6976dff7d37cddb53"
dependencies = [
"defmt 1.2.0",
]
[[package]]
name = "defmt"
version = "1.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e4a0795ac740cb54d7b3775c9c17345f6aaecf884c01733b39d6017738e0e3e1"
dependencies = [
"bitflags 1.3.2",
"defmt-macros",
]
[[package]]
name = "defmt-macros"
version = "1.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "840bb056bd99c495c9188198a258ab787c79f306d864db3e6e8451442d6d8366"
dependencies = [
"defmt-parser",
"proc-macro2",
"quote",
"syn 3.0.6",
]
[[package]]
name = "defmt-parser"
version = "1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "71958f69fb2de856c5dd51ac9377e986ca1aa5919f931bf5395a807277f889a0"
dependencies = [
"thiserror 2.0.20",
]
[[package]] [[package]]
name = "der" name = "der"
version = "0.8.2" version = "0.8.2"
@@ -2366,6 +2412,15 @@ dependencies = [
"zerocopy", "zerocopy",
] ]
[[package]]
name = "hash32"
version = "0.3.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "47d60b12902ba28e2730cd37e95b8c9223af2808df9e902d4df49588d1470606"
dependencies = [
"byteorder",
]
[[package]] [[package]]
name = "hashbrown" name = "hashbrown"
version = "0.15.5" version = "0.15.5"
@@ -2404,6 +2459,16 @@ dependencies = [
"hashbrown 0.17.1", "hashbrown 0.17.1",
] ]
[[package]]
name = "heapless"
version = "0.8.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0bfb9eb618601c89945a70e254898da93b13be0388091d42117462b265bb3fad"
dependencies = [
"hash32",
"stable_deref_trait",
]
[[package]] [[package]]
name = "heck" name = "heck"
version = "0.4.1" version = "0.4.1"
@@ -2806,8 +2871,6 @@ checksum = "791930b43c0d5973160d90a8f3894509f2b273430f5c5c73b668636d0287c5c0"
[[package]] [[package]]
name = "iroh" name = "iroh"
version = "1.2.0" version = "1.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b2f8d1cfffc83efe39a1031aab423ce09cb8048071baba550508931e9a81ce46"
dependencies = [ dependencies = [
"backon", "backon",
"blake3", "blake3",
@@ -3317,6 +3380,12 @@ dependencies = [
"libc", "libc",
] ]
[[package]]
name = "managed"
version = "0.8.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0ca88d725a0a943b096803bd34e73a4437208b6077654cc4ecb2947a5f91618d"
[[package]] [[package]]
name = "matchers" name = "matchers"
version = "0.2.0" version = "0.2.0"
@@ -5518,6 +5587,20 @@ dependencies = [
"serde", "serde",
] ]
[[package]]
name = "smoltcp"
version = "0.12.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "dad095989c1533c1c266d9b1e8d70a1329dd3723c3edac6d03bbd67e7bf6f4bb"
dependencies = [
"bitflags 1.3.2",
"byteorder",
"cfg-if",
"defmt 0.3.101",
"heapless",
"managed",
]
[[package]] [[package]]
name = "socket2" name = "socket2"
version = "0.6.5" version = "0.6.5"
@@ -6241,6 +6324,7 @@ dependencies = [
"iroh", "iroh",
"netwatch", "netwatch",
"nix 0.31.3", "nix 0.31.3",
"serde_json",
"simple-dns", "simple-dns",
"tempfile", "tempfile",
"tokio", "tokio",
@@ -6258,20 +6342,15 @@ dependencies = [
name = "tsunagi-awg" name = "tsunagi-awg"
version = "0.1.0" version = "0.1.0"
dependencies = [ dependencies = [
"boringtun",
"bytes", "bytes",
"chacha20 0.9.1",
"iroh", "iroh",
"postcard",
"rand 0.10.3", "rand 0.10.3",
"serde", "smoltcp",
"tempfile", "tempfile",
"tokio", "tokio",
"tracing",
"tsunagi-core", "tsunagi-core",
"tsunagi-wg-quic", "tsunagi-wg-quic",
"tsunagi-wireguard", "tsunagi-wireguard",
"zeroize",
] ]
[[package]] [[package]]
@@ -6322,27 +6401,18 @@ version = "0.1.0"
dependencies = [ dependencies = [
"bytes", "bytes",
"caps", "caps",
"chacha20poly1305",
"hex",
"hkdf",
"iroh", "iroh",
"netdev 0.46.3", "netdev 0.46.3",
"postcard", "postcard",
"rand 0.10.3",
"rcgen", "rcgen",
"rustls", "rustls",
"serde", "serde",
"sha2",
"subtle",
"tempfile", "tempfile",
"thiserror 2.0.20",
"tokio", "tokio",
"tokio-rustls", "tokio-rustls",
"tracing", "tracing",
"tracing-subscriber", "tracing-subscriber",
"tsunagi-core", "tsunagi-core",
"x25519-dalek",
"zeroize",
] ]
[[package]] [[package]]
@@ -6382,6 +6452,8 @@ dependencies = [
"tracing", "tracing",
"tracing-subscriber", "tracing-subscriber",
"tsunagi-core", "tsunagi-core",
"tsunagi-tcp-tls",
"tsunagi-wireguard",
"zeroize", "zeroize",
] ]
@@ -6391,17 +6463,13 @@ version = "0.1.0"
dependencies = [ dependencies = [
"boringtun", "boringtun",
"bytes", "bytes",
"data-encoding", "chacha20 0.9.1",
"hex",
"iroh", "iroh",
"netdev 0.46.3", "netdev 0.46.3",
"postcard", "postcard",
"rand 0.10.3", "rand 0.10.3",
"rusqlite",
"serde", "serde",
"subtle",
"tempfile", "tempfile",
"thiserror 2.0.20",
"tokio", "tokio",
"tracing", "tracing",
"tracing-subscriber", "tracing-subscriber",
Cargo.toml
+3 -1
View File
@@ -4,12 +4,14 @@
[workspace] [workspace]
resolver = "3" resolver = "3"
members = ["crates/*"] members = ["crates/*"]
exclude = ["vendor/mainline"] exclude = ["vendor/mainline", "vendor/iroh"]
# Mainline 8.0.0 treats Windows UDP read timeouts as socket failures. # Mainline 8.0.0 treats Windows UDP read timeouts as socket failures.
# Keep the small receive-loop correction reproducible until an upstream fix. # Keep the small receive-loop correction reproducible until an upstream fix.
[patch.crates-io] [patch.crates-io]
mainline = { path = "vendor/mainline" } mainline = { path = "vendor/mainline" }
# Backport bounded path retries until an iroh release includes the OOM fix.
iroh = { path = "vendor/iroh" }
[workspace.package] [workspace.package]
edition = "2024" edition = "2024"
README.md
+27 -3
View File
@@ -94,6 +94,19 @@ stopping a suspended network cancels its automatic resume.
Machines are called `tsunagi-<id>` until you rename them: `tsng id hostname laptop`. Machines are called `tsunagi-<id>` until you rename them: `tsng id hostname laptop`.
### Virtual services
```sh
tsng vip create home api
tsng vip port add home api tcp/443 --backend server1:6443 --backend server2:7443
tsng vip status home api --watch
```
Use `api.home:443`. Each client selects the available backend with the lowest
measured data-plane RTT. A VIP can have multiple independent TCP and UDP ports.
Every member may edit it; the tray's **VIP** button opens a draft editor with
explicit Save. [Commands, health, metrics and limits](docs/vip.md).
### Exit nodes ### Exit nodes
```sh ```sh
@@ -123,9 +136,14 @@ tsng protocols home none # control plane only
Exact network names, full ids and unique id prefixes work here. The choice is Exact network names, full ids and unique id prefixes work here. The choice is
local to this device, saved per network and applied immediately; an offline local to this device, saved per network and applied immediately; an offline
change applies on the next start. In the tray, expand **protocols** on a network change applies on the next start. In the tray, expand **data transports** on a network
tile to toggle protocols or change their preference order. Both endpoints need tile to toggle protocols or change their preference order. Adjacent peers need a common external transport at the same wire version.
a common protocol at the same wire version. `tsng up --protocol` supplies the Selection applies to this device's incoming and outgoing data links; control and
discovery are independent. A relay's next hop may use a different transport.
Every pair uses one internal WireGuard cipher, hidden from these controls.
Changing transport or moving direct → relay → direct preserves that channel,
addresses and interface. An existing TCP connection can resume after temporary
loss, subject to its own timeouts. `tsng up --protocol` supplies the
initial defaults for networks without a saved choice; it does not restrict initial defaults for networks without a saved choice; it does not restrict
which built-in protocols can be enabled later. which built-in protocols can be enabled later.
@@ -194,6 +212,11 @@ Just trying it: `nix run github:house-of-vanity/tsunagi#tsunagi-cli -- --help`.
`packages.<system>.tsunagi` (with the tray) and `tsunagi-cli` are also exported, `packages.<system>.tsunagi` (with the tray) and `tsunagi-cli` are also exported,
along with `overlays.default`. along with `overlays.default`.
For development from a checkout, `nix develop` also exposes the tray's native
runtime libraries, including AppIndicator and graphics/windowing libraries.
Cargo-built binaries such as `target/debug/tsunagi-tray` can run directly inside
this shell, without installing the packaged wrapper.
## Build from source ## Build from source
Rust 1.91 or newer. Rust 1.91 or newer.
@@ -227,6 +250,7 @@ The [threat model](docs/threat-model.md) says what this does and doesn't protect
[Pairwise Amnezia transport](docs/amnezia.md) · [Pairwise Amnezia transport](docs/amnezia.md) ·
[Routing](docs/routing.md) · [Routing](docs/routing.md) ·
[Exit nodes](docs/exit-node.md) · [Exit nodes](docs/exit-node.md) ·
[Virtual services](docs/vip.md) ·
[Threat model](docs/threat-model.md) · [Threat model](docs/threat-model.md) ·
[Contributing](AGENTS.md) [Contributing](AGENTS.md)
crates/tsunagi-awg/Cargo.toml
+6 -12
View File
@@ -8,23 +8,17 @@ license = "WTFPL AND Apache-2.0"
repository.workspace = true repository.workspace = true
[dependencies] [dependencies]
tsunagi-core = { path = "../tsunagi-core" }
tsunagi-wg-quic = { path = "../tsunagi-wg-quic" }
tsunagi-wireguard = { path = "../tsunagi-wireguard" } tsunagi-wireguard = { path = "../tsunagi-wireguard" }
iroh.workspace = true
tokio = { workspace = true, features = ["net", "time", "sync", "macros"] }
serde.workspace = true
postcard.workspace = true
bytes.workspace = true
tracing.workspace = true
rand = "0.10"
zeroize = { version = "1.9", features = ["derive"] }
boringtun = { version = "0.7.1", default-features = false }
chacha20 = "0.9"
[dev-dependencies] [dev-dependencies]
tsunagi-wg-quic = { path = "../tsunagi-wg-quic" }
iroh.workspace = true
tokio = { workspace = true, features = ["net", "time", "sync", "macros"] }
bytes.workspace = true
rand = "0.10"
tsunagi-core = { path = "../tsunagi-core", features = ["testing"] } tsunagi-core = { path = "../tsunagi-core", features = ["testing"] }
tempfile.workspace = true tempfile.workspace = true
smoltcp = { version = "0.12", default-features = false, features = ["std", "medium-ip", "proto-ipv4", "socket-tcp"] }
[lints] [lints]
workspace = true workspace = true
crates/tsunagi-awg/UPSTREAM.md
+3 -3
View File
@@ -1,9 +1,9 @@
# Amnezia packet engine # Amnezia packet engine
`src/masking.rs` is adapted from `src/core/amnezia.rs` in `../tsunagi-wireguard/src/masking.rs` is adapted from `src/core/amnezia.rs` in
https://github.com/RunaRoo/AmneziaWG-RS at https://github.com/RunaRoo/AmneziaWG-RS at
`51579a7d6e8882d1d8d6d00822f09e18a9423b73`, under Apache-2.0. `51579a7d6e8882d1d8d6d00822f09e18a9423b73`, under Apache-2.0.
The upstream license is retained in `LICENSE-AmneziaWG-RS`. The upstream license is retained in both packages as `LICENSE-AmneziaWG-RS`.
The adaptation replaces configuration-file loading with bounded typed pair The adaptation replaces configuration-file loading with bounded typed pair
profiles, updates RNG calls to rand 0.10, removes an unnecessary unwrap, and profiles, updates RNG calls to rand 0.10, removes an unnecessary unwrap, and
@@ -11,7 +11,7 @@ does not embed upstream's Node, global executors, OS interface operations or
detached tasks. Noise/WireGuard is supplied by boringtun, already used by detached tasks. Noise/WireGuard is supplied by boringtun, already used by
Tsunagi. Each pair owns one socket and one cancellable packet-engine task. Tsunagi. Each pair owns one socket and one cancellable packet-engine task.
This is Tsunagi AWG wire version 1, not a claim of compatibility with external This is Tsunagi AWG wire version 2, not a claim of compatibility with external
Amnezia servers. The masking engine substitutes message headers outside the Amnezia servers. The masking engine substitutes message headers outside the
standard WireGuard state machine and restores them before Noise validation. standard WireGuard state machine and restores them before Noise validation.
There are no remote file paths, shell hooks, or persisted pair profiles. There are no remote file paths, shell hooks, or persisted pair profiles.
crates/tsunagi-awg/src/lib.rs
+5 -16
View File
@@ -1,16 +1,5 @@
//! Pairwise, ephemeral Amnezia packet transport with independent end-to-end tunnels. //! Amnezia-masked UDP carrier using the independent per-neighbour WireGuard engine.
//! All negotiation travels over authenticated Tsunagi control sessions. pub use tsunagi_wireguard::{
mod masking; AWG_PROTOCOL, AWG_WIRE_VERSION, PairOverview, UdpCarrier as AwgPlugin,
mod plugin; UdpTransport as AwgTransport, profile,
pub mod profile; };
mod session;
mod stun;
mod transport;
pub use plugin::AwgPlugin;
pub use transport::{AwgTransport, PairOverview};
/// Protocol identifier and default second-choice direct transport.
pub const AWG_PROTOCOL: &str = "awg";
/// Tsunagi AWG wire version, independent of software releases.
pub const AWG_WIRE_VERSION: u16 = 1;
crates/tsunagi-awg/src/plugin.rs
-132
View File
@@ -1,132 +0,0 @@
//! End-to-end encryption remains independent of physical Amnezia pair links.
use crate::{AWG_PROTOCOL, AWG_WIRE_VERSION, AwgTransport};
use iroh::EndpointId;
use std::sync::Arc;
use tsunagi::dataplane::transport::SharedLink;
use tsunagi::dataplane::{IpPlugin, PluginCapability, PluginContext, PluginError, ProtocolOption};
use tsunagi::state::Ipv4Range;
use tsunagi::{BoxFuture, NetworkId};
use tsunagi_wg_quic::{NetworkOverview, WireguardConfig, WireguardPlugin};
/// AWG capability and an independent end-to-end WireGuard engine.
/// Pair transport keys/profiles never enter its durable end-to-end key store.
#[derive(Debug)]
pub struct AwgPlugin {
inner: Arc<WireguardPlugin>,
transport: Arc<AwgTransport>,
}
impl AwgPlugin {
/// Uses standard tunnel MTU/keepalive options; pair ports and profiles are ephemeral.
pub const OPTIONS: &'static [ProtocolOption] = &[
ProtocolOption {
key: "keepalive",
value: "SECONDS",
help: "end-to-end tunnel keepalive; 0 disables it",
default: Some("25"),
},
ProtocolOption {
key: "mtu",
value: "BYTES",
help: "maximum overlay packet size",
default: Some("1280"),
},
ProtocolOption {
key: "stun",
value: "HOST:PORT|off",
help: "UDP public mapping discovery; disabled in local reach mode",
default: Some("stun.l.google.com:19302"),
},
];
/// Opens the end-to-end engine, retaining separate pair transport state.
pub async fn open(
mut config: WireguardConfig,
transport: Arc<AwgTransport>,
) -> Result<Arc<Self>, PluginError> {
config.protocol = AWG_PROTOCOL;
Ok(Arc::new(Self {
inner: WireguardPlugin::open(config).await?,
transport,
}))
}
/// Reports end-to-end tunnel health and traffic counters.
pub fn overview(&self, network: NetworkId) -> Option<NetworkOverview> {
self.inner.overview(network)
}
}
impl IpPlugin for AwgPlugin {
fn protocol_id(&self) -> &str {
AWG_PROTOCOL
}
fn protocol_version(&self) -> u16 {
AWG_WIRE_VERSION
}
fn options(&self) -> &'static [ProtocolOption] {
Self::OPTIONS
}
fn attach(&self, context: PluginContext) {
self.inner.attach(context);
}
fn local_capability(
&self,
network: NetworkId,
) -> Result<Option<PluginCapability>, PluginError> {
self.inner.local_capability(network).map(|cap| {
cap.map(|mut cap| {
cap.protocol = AWG_PROTOCOL.into();
cap.version = AWG_WIRE_VERSION;
cap
})
})
}
fn on_network_activated(&self, network: NetworkId) {
self.inner.on_network_activated(network);
}
fn on_peer_capability(
&self,
network: NetworkId,
peer: EndpointId,
capability: &PluginCapability,
) -> Result<(), PluginError> {
if capability.protocol != AWG_PROTOCOL
|| capability.version != AWG_WIRE_VERSION
|| !capability.enabled
{
return Err(PluginError::Rejected("unsupported AWG capability".into()));
}
self.inner.on_peer_capability(network, peer, capability)?;
self.transport.allow_peer(network, peer);
Ok(())
}
fn on_address_allocation(
&self,
network: NetworkId,
range: Ipv4Range,
allocations: &[(EndpointId, std::net::Ipv4Addr)],
) {
self.inner
.on_address_allocation(network, range, allocations);
}
fn carry(&self, network: NetworkId, peer: EndpointId, packet: bytes::Bytes) -> bool {
self.inner.carry(network, peer, packet)
}
fn on_peer_link(&self, network: NetworkId, peer: EndpointId, link: SharedLink) {
self.inner.on_peer_link(network, peer, link);
}
fn on_peer_gone(&self, network: NetworkId, peer: EndpointId) {
self.transport.forget_peer(network, peer);
self.inner.on_peer_gone(network, peer);
}
fn on_network_deactivated(&self, network: NetworkId) {
self.transport.stop_network(network);
self.inner.on_network_deactivated(network);
}
fn on_network_forgotten(&self, network: NetworkId) {
self.inner.on_network_forgotten(network);
}
fn shutdown(&self) -> BoxFuture<'_, ()> {
Box::pin(async move {
self.transport.shutdown();
self.inner.shutdown().await;
})
}
}
crates/tsunagi-awg/src/transport.rs
-720
View File
@@ -1,720 +0,0 @@
//! Private pair negotiation. No secret profiles in capabilities or durable state.
use crate::{
AWG_PROTOCOL, AWG_WIRE_VERSION,
profile::PairProfile,
session::{self, PairLink},
};
use boringtun::{
noise::Tunn,
x25519::{PublicKey, StaticSecret},
};
use iroh::EndpointId;
use serde::{Deserialize, Serialize};
use std::collections::{HashMap, HashSet};
use std::net::{Ipv4Addr, SocketAddr};
use std::sync::{Arc, Mutex, OnceLock, Weak};
use std::time::Duration;
use tokio::net::UdpSocket;
use tokio::sync::{mpsc, oneshot, watch};
use tsunagi::dataplane::transport::TransportPayload;
use tsunagi::dataplane::transport::{
ControlSender, InboundLink, PacketLink, PacketTransport, SharedLink, TransportError,
};
use tsunagi::{BoxFuture, NetworkId};
type Pair = (NetworkId, EndpointId);
#[derive(Serialize, Deserialize)]
enum Negotiation {
Offer {
version: u16,
profile: PairProfile,
key: [u8; 32],
#[serde(deserialize_with = "bounded_addresses")]
addresses: Vec<SocketAddr>,
},
Answer {
attempt: [u8; 16],
key: [u8; 32],
#[serde(deserialize_with = "bounded_addresses")]
addresses: Vec<SocketAddr>,
},
}
fn bounded_addresses<'de, D: serde::Deserializer<'de>>(
deserializer: D,
) -> Result<Vec<SocketAddr>, D::Error> {
struct Visitor;
impl<'de> serde::de::Visitor<'de> for Visitor {
type Value = Vec<SocketAddr>;
fn expecting(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str("at most 32 UDP candidates")
}
fn visit_seq<A: serde::de::SeqAccess<'de>>(
self,
mut sequence: A,
) -> Result<Self::Value, A::Error> {
if sequence.size_hint().is_some_and(|n| n > 32) {
return Err(serde::de::Error::custom("too many UDP candidates"));
}
let mut addresses = Vec::with_capacity(sequence.size_hint().unwrap_or(0).min(32));
while let Some(address) = sequence.next_element()? {
if addresses.len() == 32 {
return Err(serde::de::Error::custom("too many UDP candidates"));
}
addresses.push(address);
}
Ok(addresses)
}
}
deserializer.deserialize_seq(Visitor)
}
struct Pending {
profile: PairProfile,
key: StaticSecret,
socket: UdpSocket,
reply: oneshot::Sender<Result<SharedLink, TransportError>>,
}
struct Inner {
stopped: watch::Sender<bool>,
local: EndpointId,
bind: Ipv4Addr,
stun: Option<String>,
busy: Mutex<HashSet<Pair>>,
control: OnceLock<Arc<dyn ControlSender>>,
allowed: Mutex<HashSet<Pair>>,
pending: Mutex<HashMap<Pair, Pending>>,
links: Mutex<HashMap<Pair, Arc<PairLink>>>,
inbound: mpsc::Sender<InboundLink>,
port_by_pair: Mutex<HashMap<Pair, u16>>,
}
/// Public diagnostics for one pair. No key or masking parameters are exposed.
#[derive(Debug, Clone)]
pub struct PairOverview {
/// Other endpoint's authenticated identity.
pub peer: EndpointId,
/// Fresh connection-attempt identifier.
pub attempt: [u8; 16],
/// Last known UDP port; None if diagnostics race with pair retirement.
pub local_port: Option<u16>,
/// Actual transport path, if established.
pub path: String,
/// Whether this attempt has been retired.
pub closed: bool,
}
/// A bounded set of independently configured pair engines.
pub struct AwgTransport {
inner: Arc<Inner>,
commands: mpsc::Sender<(NetworkId, EndpointId, Negotiation)>,
incoming: tokio::sync::Mutex<mpsc::Receiver<InboundLink>>,
}
impl std::fmt::Debug for AwgTransport {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("AwgTransport")
.field("local", &self.inner.local)
.finish_non_exhaustive()
}
}
fn lock<T>(mutex: &Mutex<T>) -> std::sync::MutexGuard<'_, T> {
match mutex.lock() {
Ok(g) => g,
Err(p) => p.into_inner(),
}
}
fn error(text: &str) -> TransportError {
TransportError::Other(text.into())
}
async fn addresses(inner: &Inner, socket: &UdpSocket) -> Result<Vec<SocketAddr>, TransportError> {
let local = socket
.local_addr()
.map_err(|_| error("AWG socket address unavailable"))?;
let mut candidates = if inner.bind.is_loopback() {
vec![local]
} else {
tsunagi_wireguard::local_ip_candidates()
.into_iter()
.filter(|ip| ip.is_ipv4())
.take(31)
.map(|ip| SocketAddr::new(ip, local.port()))
.collect()
};
if let Some(server) = &inner.stun
&& let Some(mapped) = crate::stun::mapped(socket, server).await
&& !candidates.contains(&mapped)
{
candidates.push(mapped);
}
if candidates.is_empty() {
return Err(error("AWG has no UDP address candidates"));
}
Ok(candidates)
}
fn validate_endpoint(key: &[u8; 32], addresses: &[SocketAddr]) -> Result<(), TransportError> {
if *key == [0; 32]
|| addresses.is_empty()
|| addresses.len() > 32
|| addresses.iter().any(|address| {
address.port() == 0
|| !address.is_ipv4()
|| address.ip().is_unspecified()
|| address.ip().is_multicast()
|| matches!(address.ip(), std::net::IpAddr::V4(ip) if ip.is_broadcast())
})
{
return Err(error("invalid AWG endpoint"));
}
Ok(())
}
impl AwgTransport {
/// Starts the coordinator. Production binds wildcard IPv4; tests use loopback.
pub fn new(local: EndpointId, bind: Ipv4Addr) -> Arc<Self> {
Self::new_with_stun(local, bind, None)
}
/// Enables an optional STUN server (host:port) for this pair's own socket.
/// Local-only callers pass None; failure still permits local UDP candidates.
pub fn new_with_stun(local: EndpointId, bind: Ipv4Addr, stun: Option<String>) -> Arc<Self> {
let (commands, mut rx) = mpsc::channel(64);
let (inbound, incoming) = mpsc::channel(64);
let (stopped, _) = watch::channel(false);
let inner = Arc::new(Inner {
stopped,
local,
bind,
stun,
busy: Mutex::default(),
control: OnceLock::new(),
allowed: Mutex::default(),
pending: Mutex::default(),
links: Mutex::default(),
inbound,
port_by_pair: Mutex::default(),
});
let weak = Arc::downgrade(&inner);
let mut stopped = inner.stopped.subscribe();
tokio::spawn(async move {
while let Some((network, peer, message)) =
tokio::select! { biased; _=stopped.changed()=>None, message=rx.recv()=>message }
{
let Some(inner) = weak.upgrade() else {
break;
};
if !lock(&inner.allowed).contains(&(network, peer))
|| !lock(&inner.busy).insert((network, peer))
{
continue;
}
tokio::spawn(async move {
let mut stop = inner.stopped.subscribe();
tokio::select! {biased; _=stop.changed()=>{}, _=async {
if let Err(err)=receive(&inner,(network,peer),message).await {tracing::debug!(%err,"AWG pair negotiation rejected");}
}=>{}}
lock(&inner.busy).remove(&(network, peer));
});
}
});
Arc::new(Self {
inner,
commands,
incoming: tokio::sync::Mutex::new(incoming),
})
}
/// Registers a peer after its authenticated capability has been validated.
pub fn allow_peer(&self, network: NetworkId, peer: EndpointId) {
let mut allowed = lock(&self.inner.allowed);
if !*self.inner.stopped.borrow() && allowed.len() < 128 {
allowed.insert((network, peer));
}
}
/// Retires a pair when its network membership session goes away.
pub fn forget_peer(&self, network: NetworkId, peer: EndpointId) {
let pair = (network, peer);
lock(&self.inner.allowed).remove(&pair);
lock(&self.inner.pending).remove(&pair);
lock(&self.inner.port_by_pair).remove(&pair);
if let Some(link) = lock(&self.inner.links).remove(&pair) {
link.close();
}
}
/// Stops all pair engines in this network.
pub fn stop_network(&self, network: NetworkId) {
let peers: Vec<_> = lock(&self.inner.allowed)
.iter()
.filter(|(n, _)| *n == network)
.map(|(_, p)| *p)
.collect();
for peer in peers {
self.forget_peer(network, peer);
}
}
/// Stops all sockets and coordinator work without waiting for remote peers.
pub fn shutdown(&self) {
self.inner.stopped.send_replace(true);
lock(&self.inner.allowed).clear();
lock(&self.inner.pending).clear();
for (_, link) in lock(&self.inner.links).drain() {
link.close();
}
lock(&self.inner.port_by_pair).clear();
}
/// Receives a successfully negotiated and authenticated inbound pair link.
pub async fn next_inbound(&self) -> Option<InboundLink> {
let mut stopped = self.inner.stopped.subscribe();
if *stopped.borrow() {
return None;
}
let mut incoming = self.incoming.lock().await;
tokio::select! {biased; _=stopped.changed()=>None, link=incoming.recv()=>link}
}
/// Takes a public snapshot of this network's pair attempts.
pub fn overview(&self, network: NetworkId) -> Vec<PairOverview> {
let links = lock(&self.inner.links);
let ports = lock(&self.inner.port_by_pair);
links
.iter()
.filter(|((n, _), _)| *n == network)
.map(|((_, peer), link)| PairOverview {
peer: *peer,
attempt: link.attempt,
local_port: ports.get(&(network, *peer)).copied(),
path: link.path_description(),
closed: link.is_closed(),
})
.collect()
}
}
impl PacketTransport for AwgTransport {
fn name(&self) -> &str {
AWG_PROTOCOL
}
fn as_any(&self) -> &dyn std::any::Any {
self
}
fn attach_control(&self, control: Arc<dyn ControlSender>) {
let _ = self.inner.control.set(control);
}
fn on_control(
&self,
network: NetworkId,
peer: EndpointId,
protocol: &str,
payload: &[u8],
) -> Result<(), TransportError> {
if protocol != AWG_PROTOCOL || !lock(&self.inner.allowed).contains(&(network, peer)) {
return Err(TransportError::Declined(protocol.into()));
}
if payload.len() > 4096 {
return Err(error("AWG negotiation too large"));
}
let message: Negotiation =
postcard::from_bytes(payload).map_err(|_| error("malformed AWG negotiation"))?;
match &message {
Negotiation::Offer {
version,
profile,
key,
addresses,
} => {
if *version != AWG_WIRE_VERSION || peer.as_bytes() >= self.inner.local.as_bytes() {
return Err(error("invalid AWG offer role/version"));
}
profile.validate().map_err(error)?;
validate_endpoint(key, addresses)?;
}
Negotiation::Answer { key, addresses, .. } => validate_endpoint(key, addresses)?,
}
self.commands
.try_send((network, peer, message))
.map_err(|_| error("AWG negotiation queue full"))
}
fn open<'a>(
&'a self,
network: NetworkId,
peer: EndpointId,
protocol: &'a str,
) -> BoxFuture<'a, Result<SharedLink, TransportError>> {
Box::pin(async move {
let pair = (network, peer);
if protocol != AWG_PROTOCOL
|| self.inner.local.as_bytes() >= peer.as_bytes()
|| !lock(&self.inner.allowed).contains(&pair)
{
return Err(TransportError::Declined(protocol.into()));
}
let control = self
.inner
.control
.get()
.ok_or_else(|| error("AWG control plane unavailable"))?;
let socket = UdpSocket::bind((self.inner.bind, 0))
.await
.map_err(|_| error("AWG UDP bind failed"))?;
let profile = PairProfile::generate();
let attempt = profile.id();
let key = StaticSecret::from(rand::random::<[u8; 32]>());
let public = PublicKey::from(&key).to_bytes();
let message = Negotiation::Offer {
version: AWG_WIRE_VERSION,
profile: profile.clone(),
key: public,
addresses: addresses(&self.inner, &socket).await?,
};
let (reply, rx) = oneshot::channel();
if lock(&self.inner.pending).len() >= 128 {
return Err(error("too many AWG pair negotiations"));
}
lock(&self.inner.pending).insert(
pair,
Pending {
profile,
key,
socket,
reply,
},
);
// A cancellation guard also retires partially started attempts.
let mut guard = AttemptGuard {
inner: Arc::downgrade(&self.inner),
pair,
attempt,
completed: false,
};
let bytes =
postcard::to_stdvec(&message).map_err(|_| error("AWG offer encoding failed"))?;
control
.send(network, peer, AWG_PROTOCOL, TransportPayload::new(bytes))
.await?;
let result = tokio::time::timeout(Duration::from_secs(15), rx)
.await
.map_err(|_| error("AWG negotiation/handshake timed out"))?
.map_err(|_| error("AWG attempt cancelled"))?;
guard.completed = result.is_ok();
result
})
}
}
struct AttemptGuard {
inner: Weak<Inner>,
pair: Pair,
attempt: [u8; 16],
completed: bool,
}
impl Drop for AttemptGuard {
fn drop(&mut self) {
if let Some(inner) = self.inner.upgrade() {
let mut pending = lock(&inner.pending);
if pending
.get(&self.pair)
.is_some_and(|p| p.profile.id() == self.attempt)
{
pending.remove(&self.pair);
}
drop(pending);
if !self.completed
&& let Some(link) = lock(&inner.links).get(&self.pair)
&& link.attempt == self.attempt
{
link.close();
}
}
}
}
async fn receive(
inner: &Arc<Inner>,
pair: Pair,
message: Negotiation,
) -> Result<(), TransportError> {
if !lock(&inner.allowed).contains(&pair) {
return Err(error("AWG peer no longer authenticated"));
}
match message {
Negotiation::Offer {
profile,
key,
addresses: candidates,
..
} => {
if let Some(link) = lock(&inner.links).get(&pair) {
if link.attempt == profile.id() {
return Err(error("duplicate AWG offer"));
}
link.close();
}
let socket = UdpSocket::bind((inner.bind, 0))
.await
.map_err(|_| error("AWG UDP bind failed"))?;
let local_port = socket
.local_addr()
.map_err(|_| error("AWG socket address unavailable"))?
.port();
let local_key = StaticSecret::from(rand::random::<[u8; 32]>());
let public = PublicKey::from(&local_key).to_bytes();
let answer = Negotiation::Answer {
attempt: profile.id(),
key: public,
addresses: addresses(inner, &socket).await?,
};
let tunn = Tunn::new(
local_key,
PublicKey::from(key),
Some(profile.psk),
Some(1),
rand::random(),
None,
);
let link = session::start(
pair.0,
pair.1,
socket,
tunn,
profile,
candidates,
inner.stopped.subscribe(),
);
if !lock(&inner.allowed).contains(&pair) {
link.close();
return Err(error("AWG membership ended"));
}
lock(&inner.links).insert(pair, link.clone());
lock(&inner.port_by_pair).insert(pair, local_port);
let control = inner
.control
.get()
.ok_or_else(|| error("AWG control unavailable"))?;
if let Err(err) = control
.send(
pair.0,
pair.1,
AWG_PROTOCOL,
TransportPayload::new(
postcard::to_stdvec(&answer)
.map_err(|_| error("AWG answer encoding failed"))?,
),
)
.await
{
link.close();
return Err(err);
}
publish_when_up(Arc::downgrade(inner), pair, link, None);
}
Negotiation::Answer {
attempt,
key,
addresses: candidates,
} => {
if !lock(&inner.pending)
.get(&pair)
.is_some_and(|p| p.profile.id() == attempt)
{
return Err(error("stale AWG answer"));
}
let pending = lock(&inner.pending)
.remove(&pair)
.ok_or_else(|| error("AWG attempt cancelled"))?;
let local_port = pending
.socket
.local_addr()
.map_err(|_| error("AWG socket address unavailable"))?
.port();
let tunn = Tunn::new(
pending.key,
PublicKey::from(key),
Some(pending.profile.psk),
Some(1),
rand::random(),
None,
);
let link = session::start(
pair.0,
pair.1,
pending.socket,
tunn,
pending.profile,
candidates,
inner.stopped.subscribe(),
);
if !lock(&inner.allowed).contains(&pair) {
link.close();
return Err(error("AWG membership ended"));
}
lock(&inner.links).insert(pair, link.clone());
lock(&inner.port_by_pair).insert(pair, local_port);
publish_when_up(Arc::downgrade(inner), pair, link, Some(pending.reply));
}
}
Ok(())
}
fn publish_when_up(
inner: Weak<Inner>,
pair: Pair,
link: Arc<PairLink>,
reply: Option<oneshot::Sender<Result<SharedLink, TransportError>>>,
) {
tokio::spawn(async move {
let mut reply = reply;
let result = tokio::time::timeout(Duration::from_secs(14), async {
loop {
if link.is_closed() || reply.as_ref().is_some_and(oneshot::Sender::is_closed) {
return Err(error("AWG pair closed during handshake"));
}
if link.has_authenticated_path() {
return Ok(());
}
tokio::time::sleep(Duration::from_millis(25)).await;
}
})
.await;
if !matches!(result, Ok(Ok(()))) {
link.close();
if let Some(reply) = reply.take() {
let _ = reply.send(Err(error("AWG authenticated data path timed out")));
}
return;
}
let Some(inner) = inner.upgrade() else {
link.close();
return;
};
if !lock(&inner.allowed).contains(&pair) {
link.close();
return;
}
if link.is_closed()
|| !lock(&inner.links)
.get(&pair)
.is_some_and(|current| current.attempt == link.attempt)
{
link.close();
return;
}
let shared: SharedLink = link.clone();
if let Some(reply) = reply {
if reply.send(Ok(shared)).is_err() {
link.close();
}
} else if inner
.inbound
.try_send(InboundLink {
network: pair.0,
peer: pair.1,
protocol: AWG_PROTOCOL.into(),
link: shared,
})
.is_err()
{
link.close();
}
});
}
impl Drop for AwgTransport {
fn drop(&mut self) {
self.shutdown();
}
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
#[tokio::test]
async fn concurrent_admission_is_bounded_across_networks_and_retirement_releases_slots() {
let identity = tsunagi::identity::DeviceIdentity::generate();
let transport = AwgTransport::new(identity.endpoint_id(), Ipv4Addr::LOCALHOST);
let peer = tsunagi::identity::DeviceIdentity::generate().endpoint_id();
std::thread::scope(|scope| {
for thread in 0..8u8 {
let transport = &transport;
scope.spawn(move || {
for item in 0..32u8 {
let mut id = [0; 32];
id[0] = thread;
id[1] = item;
transport.allow_peer(NetworkId::from_bytes(id), peer);
}
});
}
});
assert_eq!(lock(&transport.inner.allowed).len(), 128);
let network = lock(&transport.inner.allowed).iter().next().unwrap().0;
transport.stop_network(network);
let new = NetworkId::from_bytes([255; 32]);
transport.allow_peer(new, peer);
assert!(lock(&transport.inner.allowed).contains(&(new, peer)));
assert_eq!(lock(&transport.inner.allowed).len(), 128);
transport.shutdown();
assert!(lock(&transport.inner.allowed).is_empty());
}
#[tokio::test]
async fn private_negotiation_bounds_roles_and_membership_before_starting_engines() {
let mut ids = [
tsunagi::identity::DeviceIdentity::generate().endpoint_id(),
tsunagi::identity::DeviceIdentity::generate().endpoint_id(),
];
ids.sort_by_key(|id| *id.as_bytes());
let peer = ids[0];
let local = ids[1];
let network = NetworkId::from_bytes([7; 32]);
let transport = AwgTransport::new(local, Ipv4Addr::LOCALHOST);
let offer = |version, key, addresses| Negotiation::Offer {
version,
profile: PairProfile::generate(),
key,
addresses,
};
let address = SocketAddr::from((Ipv4Addr::LOCALHOST, 12345));
let encode = |message: &Negotiation| postcard::to_stdvec(message).unwrap();
let valid = encode(&offer(AWG_WIRE_VERSION, [1; 32], vec![address]));
assert!(
transport
.on_control(network, peer, AWG_PROTOCOL, &valid)
.is_err()
);
transport.allow_peer(network, peer);
assert!(
transport
.on_control(network, peer, "wrong-protocol", &valid)
.is_err()
);
assert!(
transport
.on_control(network, peer, AWG_PROTOCOL, &[0; 4097])
.is_err()
);
for bytes in [
encode(&offer(99, [1; 32], vec![address])),
encode(&offer(AWG_WIRE_VERSION, [0; 32], vec![address])),
encode(&offer(AWG_WIRE_VERSION, [1; 32], vec![])),
encode(&offer(AWG_WIRE_VERSION, [1; 32], vec![address; 33])),
] {
assert!(
transport
.on_control(network, peer, AWG_PROTOCOL, &bytes)
.is_err()
);
}
for len in 0..valid.len() {
assert!(
transport
.on_control(network, peer, AWG_PROTOCOL, &valid[..len])
.is_err()
);
}
let wrong_role = AwgTransport::new(peer, Ipv4Addr::LOCALHOST);
wrong_role.allow_peer(network, local);
assert!(
wrong_role
.on_control(network, local, AWG_PROTOCOL, &valid)
.is_err()
);
transport.forget_peer(network, peer);
assert!(
transport
.on_control(network, peer, AWG_PROTOCOL, &valid)
.is_err()
);
assert!(transport.overview(network).is_empty());
transport.shutdown();
wrong_role.shutdown();
}
}
crates/tsunagi-awg/tests/common/tcp.rs
+221
View File
@@ -0,0 +1,221 @@
//! A real userspace TCP stack attached to each memory TUN, without host routes.
use bytes::Bytes;
use smoltcp::{
iface::{Config, Interface, SocketHandle, SocketSet},
phy::{Device, DeviceCapabilities, Medium, RxToken, TxToken},
socket::tcp,
time::Instant as TcpInstant,
wire::{HardwareAddress, IpAddress, IpCidr},
};
use std::{
collections::VecDeque,
net::Ipv4Addr,
sync::{Arc, Mutex},
time::{Duration, Instant},
};
use tsunagi::overlay::MemoryTun;
struct TunDevice {
input: Arc<Mutex<VecDeque<Vec<u8>>>>,
tun: Arc<MemoryTun>,
}
struct Rx(Vec<u8>);
impl RxToken for Rx {
fn consume<R, F: FnOnce(&[u8]) -> R>(self, f: F) -> R {
f(&self.0)
}
}
struct Tx(Arc<MemoryTun>);
impl TxToken for Tx {
fn consume<R, F: FnOnce(&mut [u8]) -> R>(self, len: usize, f: F) -> R {
let mut packet = vec![0; len];
let result = f(&mut packet);
self.0.push_from_os(Bytes::from(packet));
result
}
}
impl Device for TunDevice {
type RxToken<'a>
= Rx
where
Self: 'a;
type TxToken<'a>
= Tx
where
Self: 'a;
fn receive(&mut self, _: TcpInstant) -> Option<(Rx, Tx)> {
self.input
.lock()
.unwrap()
.pop_front()
.map(|packet| (Rx(packet), Tx(self.tun.clone())))
}
fn transmit(&mut self, _: TcpInstant) -> Option<Tx> {
Some(Tx(self.tun.clone()))
}
fn capabilities(&self) -> DeviceCapabilities {
let mut caps = DeviceCapabilities::default();
caps.medium = Medium::Ip;
caps.max_transmission_unit = 1280;
caps
}
}
struct Stack {
iface: Interface,
sockets: SocketSet<'static>,
socket: SocketHandle,
device: TunDevice,
reader: tokio::task::JoinHandle<()>,
}
impl Stack {
fn new(tun: Arc<MemoryTun>, address: Ipv4Addr) -> Self {
let input = Arc::new(Mutex::new(VecDeque::new()));
let queue = input.clone();
let received = tun.clone();
let reader = tokio::spawn(async move {
while let Some(packet) = received.pop_to_os().await {
let mut queue = queue.lock().unwrap();
assert!(queue.len() < 256);
queue.push_back(packet.to_vec());
}
});
let mut device = TunDevice { input, tun };
let mut config = Config::new(HardwareAddress::Ip);
config.random_seed = rand::random();
let mut iface = Interface::new(config, &mut device, TcpInstant::from_millis(0));
let [a, b, c, d] = address.octets();
iface.update_ip_addrs(|addresses| {
addresses
.push(IpCidr::new(IpAddress::v4(a, b, c, d), 24))
.unwrap()
});
let mut sockets = SocketSet::new(Vec::new());
let socket = sockets.add(tcp::Socket::new(
tcp::SocketBuffer::new(vec![0; 65536]),
tcp::SocketBuffer::new(vec![0; 65536]),
));
Self {
iface,
sockets,
socket,
device,
reader,
}
}
fn tcp(&mut self) -> &mut tcp::Socket<'static> {
self.sockets.get_mut(self.socket)
}
fn poll(&mut self, now: TcpInstant) {
self.iface.poll(now, &mut self.device, &mut self.sockets);
}
}
impl Drop for Stack {
fn drop(&mut self) {
self.reader.abort();
}
}
pub(super) struct TcpPair {
client: Stack,
server: Stack,
started: Instant,
}
impl TcpPair {
pub(super) async fn connect(
a: Arc<MemoryTun>,
a_ip: Ipv4Addr,
b: Arc<MemoryTun>,
b_ip: Ipv4Addr,
) -> Self {
let mut pair = Self {
client: Stack::new(a, a_ip),
server: Stack::new(b, b_ip),
started: Instant::now(),
};
pair.server.tcp().listen(22).unwrap();
let [a, b, c, d] = b_ip.octets();
pair.client
.sockets
.get_mut::<tcp::Socket>(pair.client.socket)
.connect(
pair.client.iface.context(),
(IpAddress::v4(a, b, c, d), 22),
40000,
)
.unwrap();
pair.until(|p| {
p.client.tcp().state() == tcp::State::Established
&& p.server.tcp().state() == tcp::State::Established
})
.await;
pair
}
fn poll(&mut self) {
let now = TcpInstant::from_millis(self.started.elapsed().as_millis() as i64);
self.client.poll(now);
self.server.poll(now);
}
async fn until(&mut self, mut ready: impl FnMut(&mut Self) -> bool) {
tokio::time::timeout(Duration::from_secs(10), async {
let mut ticks = tokio::time::interval(Duration::from_millis(2));
loop {
ticks.tick().await;
self.poll();
if ready(self) {
break;
}
}
})
.await
.expect("same TCP connection must progress across carrier/route changes");
}
pub(super) fn queue_while_unreachable(&mut self, payload: &[u8]) {
assert_eq!(
self.client.tcp().send_slice(payload).unwrap(),
payload.len()
);
self.poll();
}
pub(super) async fn receive_queued(&mut self, payload: &[u8]) {
let mut received = Vec::new();
self.until(|p| {
if p.server.tcp().can_recv() {
p.server
.tcp()
.recv(|bytes| {
received.extend_from_slice(bytes);
(bytes.len(), ())
})
.unwrap();
}
received.len() >= payload.len()
})
.await;
assert_eq!(received, payload);
assert_eq!(self.client.tcp().state(), tcp::State::Established);
assert_eq!(self.server.tcp().state(), tcp::State::Established);
}
pub(super) async fn exchange(&mut self, payload: &[u8]) {
self.queue_while_unreachable(payload);
self.receive_queued(payload).await;
assert_eq!(
self.server.tcp().send_slice(payload).unwrap(),
payload.len()
);
let mut received = Vec::new();
self.until(|p| {
if p.client.tcp().can_recv() {
p.client
.tcp()
.recv(|bytes| {
received.extend_from_slice(bytes);
(bytes.len(), ())
})
.unwrap();
}
received.len() >= payload.len()
})
.await;
assert_eq!(received, payload);
}
}
crates/tsunagi-awg/tests/pairs.rs
+217 -43
View File
@@ -1,5 +1,7 @@
//! Real membership, UDP sockets, pair negotiation and nested WireGuard encryption. //! Real membership, UDP sockets, pair negotiation and nested WireGuard encryption.
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)] #![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
#[path = "common/tcp.rs"]
mod tcp;
use bytes::Bytes; use bytes::Bytes;
use std::{ use std::{
net::Ipv4Addr, net::Ipv4Addr,
@@ -24,8 +26,7 @@ use tsunagi_wg_quic::WireguardConfig;
struct Node { struct Node {
agent: Agent, agent: Agent,
plugin: Arc<AwgPlugin>, plugin: Arc<tsunagi_wg_quic::WireguardPlugin>,
quic: Option<Arc<tsunagi_wg_quic::WireguardPlugin>>,
transport: Arc<AwgTransport>, transport: Arc<AwgTransport>,
tuns: MemoryTunFactory, tuns: MemoryTunFactory,
loss: Arc<AtomicBool>, loss: Arc<AtomicBool>,
@@ -65,11 +66,10 @@ impl Node {
.load_or_create_device_identity() .load_or_create_device_identity()
.unwrap(); .unwrap();
let transport = AwgTransport::new(identity.endpoint_id(), Ipv4Addr::LOCALHOST); let transport = AwgTransport::new(identity.endpoint_id(), Ipv4Addr::LOCALHOST);
let plugin = AwgPlugin::open( let plugin = tsunagi_wg_quic::WireguardPlugin::open(
WireguardConfig::new(root.join("awg")) WireguardConfig::new(root.join("awg"))
.with_mtu(mtu) .with_mtu(mtu)
.with_reconcile(Duration::from_millis(20), Duration::from_millis(250)), .with_reconcile(Duration::from_millis(20), Duration::from_millis(250)),
transport.clone(),
) )
.await .await
.unwrap(); .unwrap();
@@ -81,19 +81,13 @@ impl Node {
let tuns = MemoryTunFactory::new(); let tuns = MemoryTunFactory::new();
let mut config = config_with(root, discovery) let mut config = config_with(root, discovery)
.with_interface(Arc::new(tuns.clone()), interface, mtu) .with_interface(Arc::new(tuns.clone()), interface, mtu)
.with_plugin(plugin.clone() as Arc<dyn IpPlugin>) .with_tunnel(plugin.clone() as Arc<dyn IpPlugin>)
.with_carrier(AwgPlugin::new(transport.clone()))
.with_custom_transport(AWG_PROTOCOL, faults); .with_custom_transport(AWG_PROTOCOL, faults);
let mut quic_plugin = None;
if fallback { if fallback {
let quic = tsunagi_wg_quic::WireguardPlugin::open( config = config.with_carrier(Arc::new(
WireguardConfig::new(root.join("wg-quic")) tsunagi::dataplane::transport::iroh_link::IrohCarrier,
.with_mtu(mtu) ));
.with_reconcile(Duration::from_millis(20), Duration::from_millis(250)),
)
.await
.unwrap();
config = config.with_plugin(quic.clone());
quic_plugin = Some(quic);
} }
config.default_protocols = default_protocols; config.default_protocols = default_protocols;
let agent = Agent::spawn(config).await.unwrap(); let agent = Agent::spawn(config).await.unwrap();
@@ -109,7 +103,6 @@ impl Node {
Self { Self {
agent, agent,
plugin, plugin,
quic: quic_plugin,
transport, transport,
tuns, tuns,
loss, loss,
@@ -252,7 +245,7 @@ async fn a_control_only_agent_enables_real_data_without_restart_and_cancels_stal
} }
#[tokio::test] #[tokio::test]
async fn runtime_protocol_selection_replaces_real_tunnels_and_survives_restart() { async fn runtime_protocol_selection_preserves_real_tunnels_and_survives_restart() {
let discovery = SharedMemoryDiscovery::new(); let discovery = SharedMemoryDiscovery::new();
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()]; let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
let a = Node::open_with_fallback(roots[0].path(), &discovery, "runtime-a", true).await; let a = Node::open_with_fallback(roots[0].path(), &discovery, "runtime-a", true).await;
@@ -263,6 +256,12 @@ async fn runtime_protocol_selection_replaces_real_tunnels_and_survives_restart()
a.ready(id, 1).await; a.ready(id, 1).await;
b.ready(id, 1).await; b.ready(id, 1).await;
exchange(&a, &b, id).await; exchange(&a, &b, id).await;
let original_key = a.plugin.overview(id).unwrap().public_key;
let original_stats = a.plugin.overview(id).unwrap().peers[0]
.tunnel
.as_ref()
.unwrap()
.stats;
let (other_name, other_secret, other_id) = loop { let (other_name, other_secret, other_id) = loop {
let (name, secret) = network("unchanged-runtime-network"); let (name, secret) = network("unchanged-runtime-network");
@@ -338,13 +337,25 @@ async fn runtime_protocol_selection_replaces_real_tunnels_and_survives_restart()
.then_some(()) .then_some(())
}) })
.await; .await;
// Get addresses from the system allocation: the disabled AWG plugin no assert_eq!(a.plugin.overview(id).unwrap().public_key, original_key);
// longer has a live view of this network. let stats = a.plugin.overview(id).unwrap().peers[0]
.tunnel
.as_ref()
.unwrap()
.stats;
assert_eq!(stats.counter_epoch, original_stats.counter_epoch);
assert!(stats.tx_bytes >= original_stats.tx_bytes);
assert!(stats.rx_bytes >= original_stats.rx_bytes);
assert!(a.plugin.overview(id).unwrap().peers.iter().any(|peer| {
peer.tunnel
.as_ref()
.is_some_and(|tunnel| tunnel.health.is_up())
}));
// The same internal plugin supplies addresses and counters on the new carrier.
for (source, dest) in [(&a, &b), (&b, &a)] { for (source, dest) in [(&a, &b), (&b, &a)] {
wait_until("QUIC end-to-end handshake", || async { wait_until("QUIC end-to-end handshake", || async {
source source
.quic .plugin
.as_ref()?
.overview(id)? .overview(id)?
.peers .peers
.iter() .iter()
@@ -352,15 +363,7 @@ async fn runtime_protocol_selection_replaces_real_tunnels_and_survives_restart()
.then_some(()) .then_some(())
}) })
.await; .await;
let addr = |n: &Node| { let addr = |n: &Node| n.plugin.overview(id).unwrap().overlay_address_v4.unwrap();
n.quic
.as_ref()
.unwrap()
.overview(id)
.unwrap()
.overlay_address_v4
.unwrap()
};
let bytes = packet(addr(source), addr(dest), 1280); let bytes = packet(addr(source), addr(dest), 1280);
source.tun().push_from_os(bytes.clone()); source.tun().push_from_os(bytes.clone());
assert_eq!( assert_eq!(
@@ -604,18 +607,13 @@ async fn persistent_runtime_loss_retires_the_pair_and_negotiates_fresh_parameter
} else { } else {
&b &b
}; };
let received = initiator.plugin.overview(id).unwrap().peers[0]
.tunnel
.as_ref()
.unwrap()
.stats
.rx_packets;
let attempt = initiator.transport.overview(id)[0].attempt; let attempt = initiator.transport.overview(id)[0].attempt;
let received = initiator.transport.overview(id)[0].delivery_checks;
wait_until( wait_until(
"fresh pair delivery checks answered after installation", "fresh physical delivery checks answered after installation",
|| async { || async {
let view = initiator.plugin.overview(id)?; (initiator.transport.overview(id).first()?.delivery_checks >= received + 3)
(view.peers.first()?.tunnel.as_ref()?.stats.rx_packets >= received + 3).then_some(()) .then_some(())
}, },
) )
.await; .await;
@@ -749,11 +747,11 @@ async fn encrypted_health_checks_fit_a_smaller_peers_configured_interface_mtu()
bytes bytes
); );
wait_until( wait_until(
"encrypted delivery checks answered with a small MTU", "physical delivery checks do not depend on overlay MTU",
|| async { || async {
let view = a.plugin.overview(id)?; (a.transport.overview(id).first()?.delivery_checks >= 4
let peer = view.peers.first()?.tunnel.as_ref()?; && b.transport.overview(id).first()?.delivery_checks >= 4)
(peer.stats.rx_packets >= 3).then_some(()) .then_some(())
}, },
) )
.await; .await;
@@ -771,3 +769,179 @@ async fn encrypted_health_checks_fit_a_smaller_peers_configured_interface_mtu()
a.agent.shutdown().await; a.agent.shutdown().await;
b.agent.shutdown().await; b.agent.shutdown().await;
} }
#[tokio::test]
async fn disjoint_external_policies_relay_the_same_internal_tunnel_and_keep_it_when_routes_change()
{
let discovery = SharedMemoryDiscovery::new();
let roots = [
TempDir::new().unwrap(),
TempDir::new().unwrap(),
TempDir::new().unwrap(),
];
let a = Node::open_selected(
roots[0].path(),
&discovery,
"only-awg",
true,
1280,
Some(vec!["awg".into()]),
)
.await;
let b = Node::open_with_fallback(roots[1].path(), &discovery, "mixed-relay", true).await;
let c = Node::open_selected(
roots[2].path(),
&discovery,
"only-quic",
true,
1280,
Some(vec!["wg-quic".into()]),
)
.await;
let (name, secret) = network("independent-internal-encryption");
let id = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
c.agent.join_network(&name, &secret).await.unwrap();
for node in [&a, &b, &c] {
wait_for_peers(&node.agent, id, 2).await;
}
wait_until("AWG-only to QUIC-only through an authenticated mixed relay", || async {
let status = a.agent.network_status(id).await.ok()?;
let peer = status.peers.iter().find(|p| p.endpoint_id == c.agent.endpoint_id())?;
(matches!(peer.data_path, tsunagi::agent::DataPath::Relayed { via, .. } if via == b.agent.endpoint_id()) && a.plugin.overview(id)?.established_peers() == 2 && c.plugin.overview(id)?.established_peers() == 2).then_some(())
}).await;
exchange(&a, &c, id).await;
exchange(&c, &a, id).await;
let mut tcp = tcp::TcpPair::connect(a.tun(), a.addr(id), c.tun(), c.addr(id)).await;
tcp.exchange(b"SSH-like TCP stream over the initial relay")
.await;
let key = a.plugin.overview(id).unwrap().public_key;
let before = a
.plugin
.overview(id)
.unwrap()
.peers
.iter()
.find(|p| p.endpoint_id == c.agent.endpoint_id())
.unwrap()
.tunnel
.as_ref()
.unwrap()
.stats
.tx_packets;
// A's additional permission creates a direct QUIC route without replacing its IP tunnel.
a.agent
.set_protocols(id, vec!["wg-quic".into(), "awg".into()])
.await
.unwrap();
wait_until("direct carrier replaces the relay", || async {
let status = a.agent.network_status(id).await.ok()?;
matches!(
status
.peers
.iter()
.find(|p| p.endpoint_id == c.agent.endpoint_id())?
.data_path,
tsunagi::agent::DataPath::Direct { .. }
)
.then_some(())
})
.await;
tcp.exchange(b"same TCP socket after changing the carrier and next hop")
.await;
assert_eq!(a.plugin.overview(id).unwrap().public_key, key);
assert!(
a.plugin
.overview(id)
.unwrap()
.peers
.iter()
.find(|p| p.endpoint_id == c.agent.endpoint_id())
.unwrap()
.tunnel
.as_ref()
.unwrap()
.stats
.tx_packets
> before
);
a.agent.set_protocols(id, Vec::new()).await.unwrap();
// Poll the actual TCP stack while the route is absent: its first transmission is lost.
let pending = b"TCP retransmission survives the temporary absence of every data carrier";
tcp.queue_while_unreachable(pending);
a.agent.set_protocols(id, vec!["awg".into()]).await.unwrap();
wait_until("forbidden QUIC removed and relay restored", || async {
let status = a.agent.network_status(id).await.ok()?;
matches!(
status
.peers
.iter()
.find(|p| p.endpoint_id == c.agent.endpoint_id())?
.data_path,
tsunagi::agent::DataPath::Relayed { .. }
)
.then_some(())
})
.await;
assert_eq!(a.plugin.overview(id).unwrap().public_key, key);
tcp.receive_queued(pending).await;
tcp.exchange(b"same TCP connection after returning to AWG and the relay")
.await;
drop(tcp);
// Transit ciphertext must never be delivered to the middle agent's IP interface.
assert!(
tokio::time::timeout(Duration::from_millis(150), b.tun().pop_to_os())
.await
.is_err()
);
for node in [&a, &b, &c] {
node.agent.shutdown().await;
}
}
#[tokio::test]
async fn no_common_carrier_is_an_actionable_policy_error_without_disabling_control() {
let discovery = SharedMemoryDiscovery::new();
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
let a = Node::open_selected(
roots[0].path(),
&discovery,
"awg-policy",
true,
1280,
Some(vec!["awg".into()]),
)
.await;
let b = Node::open_selected(
roots[1].path(),
&discovery,
"quic-policy",
true,
1280,
Some(vec!["wg-quic".into()]),
)
.await;
let (name, secret) = network("no-common-carrier");
let id = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
wait_until("explain independent external policies", || async {
let status = a.agent.network_status(id).await.ok()?;
let peer = status.peers.first()?;
peer.data_error
.as_ref()?
.contains("no common external transport")
.then_some(())
})
.await;
assert_eq!(
a.agent
.network_status(id)
.await
.unwrap()
.metrics
.data_links_established,
0
);
a.agent.shutdown().await;
b.agent.shutdown().await;
}
crates/tsunagi-core/src/agent/mod.rs
+132 -36
View File
@@ -135,6 +135,10 @@ impl Inner {
.await .await
{ {
Ok(interface) => { Ok(interface) => {
self.adapter.exclude_interface(interface.name());
for carrier in &self.config.carriers {
carrier.on_interface_ready(interface.name());
}
let _ = self.interface.set(Arc::new(interface)); let _ = self.interface.set(Arc::new(interface));
} }
Err(err) => { Err(err) => {
@@ -188,14 +192,13 @@ impl PacketCarrier for Carrier {
let Some(inner) = self.0.upgrade() else { let Some(inner) = self.0.upgrade() else {
return false; return false;
}; };
// Offered to each protocol in turn. With one configured this is a // Every routed IP packet enters the same internal cipher, whatever
// single call; the shape is what allows several to be live at once, // physical carrier currently supplies the logical peer link.
// each carrying the peers it has a link to.
inner inner
.config .config
.plugins .tunnel
.iter() .as_ref()
.any(|plugin| plugin.carry(route.network, route.peer, packet.clone())) .is_some_and(|tunnel| tunnel.carry(route.network, route.peer, packet))
} }
} }
@@ -213,21 +216,6 @@ impl PacketSink for Sink {
let Some(inner) = self.0.upgrade() else { let Some(inner) = self.0.upgrade() else {
return; return;
}; };
if let Some(payload) = crate::dataplane::transport::health::ip_payload(&packet) {
if let Some(header) = crate::overlay::IpHeader::parse(&packet)
&& let (std::net::IpAddr::V4(source), std::net::IpAddr::V4(destination)) =
(header.source(), header.destination())
&& inner.routes.may_send_from(network, peer, source)
&& inner.routes.is_local_destination(network, destination)
&& let Some(handle) = inner.networks.read().await.get(&network)
{
let _ = handle.commands.try_send(NetCommand::HealthProbe {
peer,
packet: bytes::Bytes::copy_from_slice(payload),
});
}
return;
}
let Some(interface) = inner.interface.get() else { let Some(interface) = inner.interface.get() else {
return; return;
}; };
@@ -301,7 +289,7 @@ impl TransportContext for TransportCtx {
} }
inner inner
.config .config
.plugins .carriers
.iter() .iter()
.any(|plugin| plugin.protocol_id() == protocol) .any(|plugin| plugin.protocol_id() == protocol)
}) })
@@ -309,6 +297,28 @@ impl TransportContext for TransportCtx {
} }
impl Agent { impl Agent {
/// Atomically saves or removes a version-checked shared VIP draft.
/// Every authenticated network member may edit any service.
pub async fn save_vip(
&self,
network: NetworkId,
config: crate::vip::VipConfig,
expected: Option<String>,
delete: bool,
) -> Result<Vec<crate::vip::VipStatus>> {
let (reply, receive) = oneshot::channel();
self.command(
network,
NetCommand::SaveVip {
config,
expected,
delete,
reply,
},
)
.await?;
receive.await.map_err(|_| Error::Stopped)?
}
/// Retires one offline member's observed claim without revoking membership. /// Retires one offline member's observed claim without revoking membership.
/// Every member has this right. The version must still match the selection. /// Every member has this right. The version must still match the selection.
pub async fn forget_node( pub async fn forget_node(
@@ -337,20 +347,23 @@ impl Agent {
.unwrap_or_else(|| { .unwrap_or_else(|| {
self.inner self.inner
.config .config
.plugins .carriers
.iter() .iter()
.map(|p| p.protocol_id().to_owned()) .map(|p| p.protocol_id().to_owned())
.collect() .collect()
}) })
} }
fn selected_plugins(&self, protocols: &[String]) -> Vec<crate::dataplane::SharedPlugin> { fn selected_carriers(
&self,
protocols: &[String],
) -> Vec<crate::dataplane::transport::SharedCarrier> {
protocols protocols
.iter() .iter()
.filter_map(|name| { .filter_map(|name| {
self.inner self.inner
.config .config
.plugins .carriers
.iter() .iter()
.find(|p| p.protocol_id() == name) .find(|p| p.protocol_id() == name)
.cloned() .cloned()
@@ -366,7 +379,7 @@ impl Agent {
if !self if !self
.inner .inner
.config .config
.plugins .carriers
.iter() .iter()
.any(|p| p.protocol_id() == name) .any(|p| p.protocol_id() == name)
{ {
@@ -407,7 +420,7 @@ impl Agent {
if let Some(protocols) = &config.default_protocols { if let Some(protocols) = &config.default_protocols {
crate::dataplane::validate_protocols(protocols, config.limits.max_capabilities)?; crate::dataplane::validate_protocols(protocols, config.limits.max_capabilities)?;
for name in protocols { for name in protocols {
if !config.plugins.iter().any(|p| p.protocol_id() == name) { if !config.carriers.iter().any(|p| p.protocol_id() == name) {
return Err(Error::InvalidProtocols(format!( return Err(Error::InvalidProtocols(format!(
"this agent has no `{name}` plugin" "this agent has no `{name}` plugin"
))); )));
@@ -486,7 +499,7 @@ impl Agent {
// Plugins get a handle to ask for re-announcements and report errors. // Plugins get a handle to ask for re-announcements and report errors.
// A bounded queue keeps a noisy plugin from growing memory without // A bounded queue keeps a noisy plugin from growing memory without
// bound; overflow drops the request rather than stalling the plugin. // bound; overflow drops the request rather than stalling the plugin.
if !inner.config.plugins.is_empty() { if !inner.config.carriers.is_empty() || inner.config.tunnel.is_some() {
let (plugin_tx, plugin_rx) = mpsc::channel(64); let (plugin_tx, plugin_rx) = mpsc::channel(64);
let sink = inner let sink = inner
.config .config
@@ -494,8 +507,8 @@ impl Agent {
.as_ref() .as_ref()
.map(|_| Arc::new(Sink(Arc::downgrade(&inner))) as Arc<dyn PacketSink>); .map(|_| Arc::new(Sink(Arc::downgrade(&inner))) as Arc<dyn PacketSink>);
let context = PluginContext::new(plugin_tx, inner.identity.endpoint_id(), sink); let context = PluginContext::new(plugin_tx, inner.identity.endpoint_id(), sink);
for plugin in &inner.config.plugins { if let Some(tunnel) = &inner.config.tunnel {
plugin.attach(context.clone()); tunnel.attach(context);
} }
let task = tokio::spawn(plugin_request_loop(Arc::downgrade(&inner), plugin_rx)); let task = tokio::spawn(plugin_request_loop(Arc::downgrade(&inner), plugin_rx));
if let Ok(mut guard) = inner.plugin_task.lock() { if let Ok(mut guard) = inner.plugin_task.lock() {
@@ -916,6 +929,34 @@ impl Agent {
.storage .storage
.retirements(network_id, self.inner.limits.max_retirements) .retirements(network_id, self.inner.limits.max_retirements)
.await?; .await?;
let vip_entries = self
.inner
.storage
.vip_fields(
network_id,
self.inner
.limits
.max_vips
.saturating_mul(self.inner.limits.max_vip_fields)
.saturating_add(self.inner.limits.max_vip_tombstones),
)
.await?;
let mut vips = crate::vip::VipSet::default();
for id in vip_entries
.iter()
.map(|e| e.vip.clone())
.collect::<std::collections::BTreeSet<_>>()
{
vips.merge(
network_id,
vip_entries
.iter()
.filter(|e| e.vip == id)
.cloned()
.collect(),
&self.inner.limits,
)?;
}
let range = self.desired_range(network_id).await?; let range = self.desired_range(network_id).await?;
if let Some(range) = range { if let Some(range) = range {
let conflicts = self.inner.routes.overlapping_networks(network_id, range); let conflicts = self.inner.routes.overlapping_networks(network_id, range);
@@ -958,7 +999,7 @@ impl Agent {
let broadcast = stored.as_ref().is_none_or(|stored| stored.broadcast); let broadcast = stored.as_ref().is_none_or(|stored| stored.broadcast);
let protocols = let protocols =
self.effective_protocols(stored.as_ref().and_then(|s| s.protocols.as_ref())); self.effective_protocols(stored.as_ref().and_then(|s| s.protocols.as_ref()));
let plugins = self.selected_plugins(&protocols); let plugins = self.selected_carriers(&protocols);
if !plugins.is_empty() { if !plugins.is_empty() {
self.inner.ensure_interface().await; self.inner.ensure_interface().await;
} }
@@ -968,6 +1009,7 @@ impl Agent {
.and_then(|stored| stored.exit_via) .and_then(|stored| stored.exit_via)
.and_then(|bytes| EndpointId::from_bytes(&bytes).ok()); .and_then(|bytes| EndpointId::from_bytes(&bytes).ok());
let handle = network::spawn(RuntimeParams { let handle = network::spawn(RuntimeParams {
vips,
retirements, retirements,
broadcast, broadcast,
exit_offer, exit_offer,
@@ -981,8 +1023,9 @@ impl Agent {
discovery, discovery,
discovery_interval: self.inner.config.discovery_interval, discovery_interval: self.inner.config.discovery_interval,
discovery_policy: self.inner.config.discovery_policy.clone(), discovery_policy: self.inner.config.discovery_policy.clone(),
available_plugins: self.inner.config.plugins.clone(), available_carriers: self.inner.config.carriers.clone(),
plugins: plugins.clone(), tunnel: self.inner.config.tunnel.clone(),
carriers: plugins.clone(),
routes: Arc::clone(&self.inner.routes), routes: Arc::clone(&self.inner.routes),
interface: self.inner.interface.get().cloned(), interface: self.inner.interface.get().cloned(),
hostname: self.inner.read_hostname(), hostname: self.inner.read_hostname(),
@@ -999,6 +1042,9 @@ impl Agent {
for plugin in &plugins { for plugin in &plugins {
plugin.on_network_activated(network_id); plugin.on_network_activated(network_id);
} }
if let Some(tunnel) = &self.inner.config.tunnel {
tunnel.on_network_activated(network_id);
}
let _ = self.inner.events.send(Event::NetworkActivated { let _ = self.inner.events.send(Event::NetworkActivated {
network: network_id, network: network_id,
@@ -1043,9 +1089,12 @@ impl Agent {
.ok_or(Error::NetworkNotActive(network_id))? .ok_or(Error::NetworkNotActive(network_id))?
}; };
handle.stop().await; handle.stop().await;
for plugin in &self.inner.config.plugins { for plugin in &self.inner.config.carriers {
plugin.on_network_deactivated(network_id); plugin.on_network_deactivated(network_id);
} }
if let Some(tunnel) = &self.inner.config.tunnel {
tunnel.on_network_deactivated(network_id);
}
// The network's addresses come off the interface; the interface // The network's addresses come off the interface; the interface
// itself stays, because the agent owns it and other networks may // itself stays, because the agent owns it and other networks may
// still be using it. // still be using it.
@@ -1078,9 +1127,12 @@ impl Agent {
self.stop_network_locked(network_id).await?; self.stop_network_locked(network_id).await?;
} }
self.inner.storage.remove_network(network_id).await?; self.inner.storage.remove_network(network_id).await?;
for plugin in &self.inner.config.plugins { for plugin in &self.inner.config.carriers {
plugin.on_network_forgotten(network_id); plugin.on_network_forgotten(network_id);
} }
if let Some(tunnel) = &self.inner.config.tunnel {
tunnel.on_network_forgotten(network_id).await;
}
self.resume_networks_locked().await self.resume_networks_locked().await
} }
@@ -1215,7 +1267,43 @@ impl Agent {
let keys = NetworkKeys::derive(&stored.name, &stored.secret); let keys = NetworkKeys::derive(&stored.name, &stored.secret);
let range = self.desired_range(stored.network_id).await?; let range = self.desired_range(stored.network_id).await?;
let range_conflicts = self.range_conflicts(stored.network_id, range).await?; let range_conflicts = self.range_conflicts(stored.network_id, range).await?;
let fields = self
.inner
.storage
.vip_fields(
stored.network_id,
self.inner
.limits
.max_vips
.saturating_mul(self.inner.limits.max_vip_fields)
.saturating_add(self.inner.limits.max_vip_tombstones),
)
.await?;
let mut service_state = crate::vip::VipSet::default();
for id in fields
.iter()
.map(|e| e.vip.clone())
.collect::<std::collections::BTreeSet<_>>()
{
service_state.merge(
stored.network_id,
fields.iter().filter(|e| e.vip == id).cloned().collect(),
&self.inner.limits,
)?;
}
let vips = service_state
.configs()
.into_iter()
.map(|config| crate::vip::VipStatus {
revision: service_state.revision(&config.id),
config,
address: None,
error: Some("network is inactive".into()),
listeners: vec![],
})
.collect();
networks.push(NetworkStatus { networks.push(NetworkStatus {
vips,
protocols: self.effective_protocols(stored.protocols.as_ref()), protocols: self.effective_protocols(stored.protocols.as_ref()),
auto_start: stored.auto_start, auto_start: stored.auto_start,
broadcast: stored.broadcast, broadcast: stored.broadcast,
@@ -1336,7 +1424,7 @@ impl Agent {
// Plugins remove whatever system objects they created. A plugin that // Plugins remove whatever system objects they created. A plugin that
// misbehaves here must not hold up the agent, so this is bounded. // misbehaves here must not hold up the agent, so this is bounded.
for plugin in &self.inner.config.plugins { for plugin in &self.inner.config.carriers {
if tokio::time::timeout(PLUGIN_SHUTDOWN_GRACE, plugin.shutdown()) if tokio::time::timeout(PLUGIN_SHUTDOWN_GRACE, plugin.shutdown())
.await .await
.is_err() .is_err()
@@ -1348,6 +1436,14 @@ impl Agent {
} }
} }
if let Some(tunnel) = &self.inner.config.tunnel
&& tokio::time::timeout(PLUGIN_SHUTDOWN_GRACE, tunnel.shutdown())
.await
.is_err()
{
tracing::warn!("data encryption engine did not shut down in time");
}
// Release the directory so another instance can claim it right away. // Release the directory so another instance can claim it right away.
self.inner.storage.release_ownership_lock(); self.inner.storage.release_ownership_lock();
} }
crates/tsunagi-core/src/agent/network.rs
+447 -108
View File
@@ -15,7 +15,7 @@ use tokio::sync::{broadcast, mpsc, oneshot};
use tokio::task::JoinHandle; use tokio::task::JoinHandle;
use crate::config::{DiscoveryPolicy, Limits, ReconnectPolicy}; use crate::config::{DiscoveryPolicy, Limits, ReconnectPolicy};
use crate::dataplane::transport::health::{HealthLink, ProbeRoute, Settings as HealthSettings}; use crate::dataplane::transport::health::{HealthLink, Settings as HealthSettings};
use crate::dataplane::transport::{InboundLink, PacketLink, PacketTransport, SharedLink}; use crate::dataplane::transport::{InboundLink, PacketLink, PacketTransport, SharedLink};
use crate::dataplane::{PluginCapability, SharedPlugin}; use crate::dataplane::{PluginCapability, SharedPlugin};
use crate::discovery::worker::DiscoveryWorker; use crate::discovery::worker::DiscoveryWorker;
@@ -50,6 +50,12 @@ pub(crate) struct InboundSession {
/// Commands accepted by a network runtime. /// Commands accepted by a network runtime.
pub(crate) enum NetCommand { pub(crate) enum NetCommand {
SaveVip {
config: crate::vip::VipConfig,
expected: Option<String>,
delete: bool,
reply: oneshot::Sender<Result<Vec<crate::vip::VipStatus>>>,
},
ForgetNode { ForgetNode {
peer: EndpointId, peer: EndpointId,
expected_version: u64, expected_version: u64,
@@ -67,10 +73,6 @@ pub(crate) enum NetCommand {
}, },
/// A peer opened a data plane link towards us. /// A peer opened a data plane link towards us.
InboundLink(Box<InboundLink>), InboundLink(Box<InboundLink>),
HealthProbe {
peer: EndpointId,
packet: bytes::Bytes,
},
Status { Status {
reply: oneshot::Sender<Box<NetworkStatus>>, reply: oneshot::Sender<Box<NetworkStatus>>,
}, },
@@ -125,6 +127,7 @@ pub(crate) enum NetCommand {
impl std::fmt::Debug for NetCommand { impl std::fmt::Debug for NetCommand {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self { match self {
NetCommand::SaveVip { .. } => f.write_str("SaveVip"),
NetCommand::ForgetNode { peer, .. } => write!(f, "ForgetNode({peer})"), NetCommand::ForgetNode { peer, .. } => write!(f, "ForgetNode({peer})"),
NetCommand::Inbound(_) => f.write_str("Inbound"), NetCommand::Inbound(_) => f.write_str("Inbound"),
NetCommand::InboundLink(link) => write!(f, "InboundLink({})", link.protocol), NetCommand::InboundLink(link) => write!(f, "InboundLink({})", link.protocol),
@@ -142,7 +145,6 @@ impl std::fmt::Debug for NetCommand {
NetCommand::SetHostname(_) => f.write_str("SetHostname"), NetCommand::SetHostname(_) => f.write_str("SetHostname"),
NetCommand::Release { .. } => f.write_str("Release"), NetCommand::Release { .. } => f.write_str("Release"),
NetCommand::PluginError { protocol, .. } => write!(f, "PluginError({protocol})"), NetCommand::PluginError { protocol, .. } => write!(f, "PluginError({protocol})"),
NetCommand::HealthProbe { peer, .. } => write!(f, "HealthProbe({})", peer.fmt_short()),
NetCommand::ProtocolFailure { peer, protocol, .. } => { NetCommand::ProtocolFailure { peer, protocol, .. } => {
write!(f, "ProtocolFailure({}, {protocol})", peer.fmt_short()) write!(f, "ProtocolFailure({}, {protocol})", peer.fmt_short())
} }
@@ -170,6 +172,7 @@ impl NetworkHandle {
/// Everything a network runtime needs to run. /// Everything a network runtime needs to run.
pub(crate) struct RuntimeParams { pub(crate) struct RuntimeParams {
pub(crate) vips: crate::vip::VipSet,
pub(crate) retirements: Vec<SignedRetirement>, pub(crate) retirements: Vec<SignedRetirement>,
pub(crate) broadcast: bool, pub(crate) broadcast: bool,
/// This agent offers itself as an exit node here. /// This agent offers itself as an exit node here.
@@ -185,8 +188,9 @@ pub(crate) struct RuntimeParams {
pub(crate) discovery: Option<Arc<dyn NetworkDiscovery>>, pub(crate) discovery: Option<Arc<dyn NetworkDiscovery>>,
pub(crate) discovery_interval: Duration, pub(crate) discovery_interval: Duration,
pub(crate) discovery_policy: DiscoveryPolicy, pub(crate) discovery_policy: DiscoveryPolicy,
pub(crate) plugins: Vec<SharedPlugin>, pub(crate) carriers: Vec<crate::dataplane::transport::SharedCarrier>,
pub(crate) available_plugins: Vec<SharedPlugin>, pub(crate) available_carriers: Vec<crate::dataplane::transport::SharedCarrier>,
pub(crate) tunnel: Option<SharedPlugin>,
/// Who holds which overlay address, shared with every other network. /// Who holds which overlay address, shared with every other network.
pub(crate) routes: Arc<crate::overlay::RoutingTable>, pub(crate) routes: Arc<crate::overlay::RoutingTable>,
/// The one interface, when the agent has one. /// The one interface, when the agent has one.
@@ -376,6 +380,8 @@ struct Runtime {
/// Snapshots received while dispatching, handled on the next loop pass. /// Snapshots received while dispatching, handled on the next loop pass.
pending_state: Vec<(EndpointId, Vec<SignedRecord>)>, pending_state: Vec<(EndpointId, Vec<SignedRecord>)>,
pending_retirements: Vec<(EndpointId, Vec<SignedRetirement>)>, pending_retirements: Vec<(EndpointId, Vec<SignedRetirement>)>,
pending_vips: Vec<Vec<crate::vip::VipEntry>>,
vip_cursor: HashMap<EndpointId, usize>,
/// Next cleanup batch per live session; prevents queue pressure starving a suffix. /// Next cleanup batch per live session; prevents queue pressure starving a suffix.
retirement_cursor: HashMap<EndpointId, usize>, retirement_cursor: HashMap<EndpointId, usize>,
/// The highest version this agent has ever published for this network. /// The highest version this agent has ever published for this network.
@@ -388,6 +394,8 @@ struct Runtime {
/// The last hostname each peer announced, kept across restarts so a /// The last hostname each peer announced, kept across restarts so a
/// member that is away can still be called by name. /// member that is away can still be called by name.
known_hostnames: HashMap<EndpointId, String>, known_hostnames: HashMap<EndpointId, String>,
data_errors: HashMap<EndpointId, String>,
tunnel_errors: HashMap<EndpointId, String>,
} }
impl Runtime { impl Runtime {
@@ -408,7 +416,7 @@ impl Runtime {
// preferred when a neighbour can be reached over more than one. // preferred when a neighbour can be reached over more than one.
hub.set_priority( hub.set_priority(
params params
.plugins .carriers
.iter() .iter()
.map(|plugin| plugin.protocol_id().to_string()) .map(|plugin| plugin.protocol_id().to_string())
.collect(), .collect(),
@@ -461,11 +469,15 @@ impl Runtime {
state: StateSet::new(), state: StateSet::new(),
pending_state: Vec::new(), pending_state: Vec::new(),
pending_retirements: Vec::new(), pending_retirements: Vec::new(),
pending_vips: Vec::new(),
vip_cursor: HashMap::new(),
retirement_cursor: HashMap::new(), retirement_cursor: HashMap::new(),
own_version: 0, own_version: 0,
exit_via_was_online: false, exit_via_was_online: false,
announced_exit: false, announced_exit: false,
known_hostnames: HashMap::new(), known_hostnames: HashMap::new(),
data_errors: HashMap::new(),
tunnel_errors: HashMap::new(),
} }
} }
@@ -561,6 +573,15 @@ impl Runtime {
async fn handle_command(&mut self, command: NetCommand) { async fn handle_command(&mut self, command: NetCommand) {
match command { match command {
NetCommand::SaveVip {
config,
expected,
delete,
reply,
} => {
let result = self.save_vip(config, expected, delete).await;
let _ = reply.send(result);
}
NetCommand::ForgetNode { NetCommand::ForgetNode {
peer, peer,
expected_version, expected_version,
@@ -578,6 +599,13 @@ impl Runtime {
message, message,
reply, reply,
} => { } => {
if matches!(&message, ControlMessage::Transport { .. })
&& let Some(interface) = self.params.interface.clone()
{
// Locally bound pair sockets must be protected before their
// private offer/accept invites the first inbound datagram.
interface.sync_exit_rules().await;
}
let _ = reply.send(self.send_to(peer, message)); let _ = reply.send(self.send_to(peer, message));
} }
NetCommand::Broadcast { message, reply } => { NetCommand::Broadcast { message, reply } => {
@@ -601,7 +629,7 @@ impl Runtime {
self.params.interface = interface; self.params.interface = interface;
let current: Vec<_> = self let current: Vec<_> = self
.params .params
.plugins .carriers
.iter() .iter()
.map(|p| p.protocol_id().to_owned()) .map(|p| p.protocol_id().to_owned())
.collect(); .collect();
@@ -610,22 +638,22 @@ impl Runtime {
for peer in &peers { for peer in &peers {
self.reset_peer_data(*peer, "local protocol selection changed"); self.reset_peer_data(*peer, "local protocol selection changed");
} }
for plugin in &self.params.plugins { for plugin in &self.params.carriers {
if !protocols.iter().any(|name| name == plugin.protocol_id()) { if !protocols.iter().any(|name| name == plugin.protocol_id()) {
plugin.on_network_deactivated(self.network_id); plugin.on_network_deactivated(self.network_id);
} }
} }
self.params.plugins = protocols self.params.carriers = protocols
.iter() .iter()
.filter_map(|name| { .filter_map(|name| {
self.params self.params
.available_plugins .available_carriers
.iter() .iter()
.find(|p| p.protocol_id() == name) .find(|p| p.protocol_id() == name)
.cloned() .cloned()
}) })
.collect(); .collect();
for plugin in &self.params.plugins { for plugin in &self.params.carriers {
if !current.iter().any(|name| name == plugin.protocol_id()) { if !current.iter().any(|name| name == plugin.protocol_id()) {
plugin.on_network_activated(self.network_id); plugin.on_network_activated(self.network_id);
} }
@@ -725,13 +753,6 @@ impl Runtime {
reason, reason,
}); });
} }
NetCommand::HealthProbe { peer, packet } => {
for ((p, _), link) in &self.links {
if *p == peer {
link.receive_probe(packet.clone());
}
}
}
NetCommand::ProtocolFailure { NetCommand::ProtocolFailure {
peer, peer,
protocol, protocol,
@@ -823,6 +844,7 @@ impl Runtime {
// the check is a comparison when nothing has changed. // the check is a comparison when nothing has changed.
self.ensure_own_claim().await; self.ensure_own_claim().await;
self.report_hostname_taken(); self.report_hostname_taken();
self.broadcast_vips();
self.broadcast_retirements(); self.broadcast_retirements();
// The same timer repairs the relay picture: reachability is soft // The same timer repairs the relay picture: reachability is soft
// state, repeated while it holds and forgotten when it stops. // state, repeated while it holds and forgotten when it stops.
@@ -871,6 +893,9 @@ impl Runtime {
// Last, so it carries what this round learned. // Last, so it carries what this round learned.
self.introduce_peers(); self.introduce_peers();
self.refresh_underlay().await; self.refresh_underlay().await;
if let Some(interface) = self.params.interface.clone() {
interface.sync_exit_rules().await;
}
} }
/// Hands the interface the underlay destinations this network depends /// Hands the interface the underlay destinations this network depends
@@ -900,7 +925,8 @@ impl Runtime {
.await; .await;
} }
fn add_candidate(&mut self, candidate: Candidate) { fn add_candidate(&mut self, mut candidate: Candidate) {
candidate.addr = self.underlay_addr(candidate.addr);
let peer = candidate.endpoint_id(); let peer = candidate.endpoint_id();
if peer == self.local_id { if peer == self.local_id {
return; return;
@@ -1094,7 +1120,8 @@ impl Runtime {
.own_record_version(self.network_id, self.local_id) .own_record_version(self.network_id, self.local_id)
.await .await
.unwrap_or(0) .unwrap_or(0)
.max(self.state.authored_version(&self.local_id)); .max(self.state.authored_version(&self.local_id))
.max(self.params.vips.authored_version(self.local_id));
for (id, hostname) in self for (id, hostname) in self
.params .params
.storage .storage
@@ -1168,11 +1195,26 @@ impl Runtime {
match self.state.address_of(&self.local_id) { match self.state.address_of(&self.local_id) {
Some(mine) if range.contains(mine) => Some(mine), Some(mine) if range.contains(mine) => Some(mine),
_ => { _ => {
let taken: std::collections::HashSet<std::net::Ipv4Addr> = holders let mut taken: std::collections::HashSet<std::net::Ipv4Addr> = holders
.iter() .iter()
.filter(|(_, holder)| **holder != self.local_id) .filter(|(_, holder)| **holder != self.local_id)
.map(|(address, _)| *address) .map(|(address, _)| *address)
.collect(); .collect();
for vip in self.params.vips.views(
Some(range),
taken.clone(),
&std::collections::BTreeSet::new(),
&self.params.limits,
) {
if let Some(address) = vip.address {
taken.insert(address);
}
}
for vip in self.params.routes.vips.statuses(self.network_id) {
if let Some(address) = vip.address {
taken.insert(address);
}
}
match allocate( match allocate(
self.network_id, self.network_id,
self.local_id, self.local_id,
@@ -1450,8 +1492,188 @@ impl Runtime {
self.broadcast_state(); self.broadcast_state();
} }
fn update_vips(&self) {
let taken = self.state.address_holders().into_keys().collect();
let names = self
.state
.active_records()
.into_iter()
.filter_map(|r| {
r.author_id()
.ok()
.and_then(|p| self.state.hostname_of(&p).map(str::to_owned))
})
.collect();
let views =
self.params
.vips
.views(self.effective_range(), taken, &names, &self.params.limits);
let peers = self
.state
.address_holders()
.into_iter()
.map(|(a, p)| (p, a, self.hub.reachable(p)))
.collect();
self.params.routes.vips.configure(
self.network_id,
self.local_id,
self.state.address_of(&self.local_id),
views,
peers,
&self.params.limits,
);
}
async fn save_vip(
&mut self,
config: crate::vip::VipConfig,
expected: Option<String>,
delete: bool,
) -> Result<Vec<crate::vip::VipStatus>> {
if self.released || self.shutdown.is_triggered() {
return Err(Error::Stopped);
}
let version = self
.own_version
.checked_add(1)
.filter(|n| *n <= i64::MAX as u64)
.ok_or_else(|| crate::vip::invalid("author counter exhausted"))?;
let entries = if delete {
self.params.vips.delete(
self.network_id,
&self.params.device_secret,
version,
&config.id,
expected.as_deref().unwrap_or(""),
)?
} else {
self.params.vips.edit(
self.network_id,
&self.params.device_secret,
version,
&config,
expected.as_deref(),
&self.params.limits,
)?
};
let mut candidate = self.params.vips.clone();
candidate.merge(self.network_id, entries, &self.params.limits)?;
if !delete {
let duplicate = candidate
.configs()
.iter()
.any(|v| v.id != config.id && v.name == config.name)
|| self.state.active_records().iter().any(|r| {
r.author_id().ok().and_then(|p| self.state.hostname_of(&p))
== Some(config.name.as_str())
});
if duplicate {
return Err(crate::vip::invalid(
"VIP name is already used by a node or another VIP",
));
}
}
let snapshot = candidate.entries(&config.id);
self.validate_vip_frame(&snapshot)?;
self.params
.storage
.put_vip_fields(
self.network_id,
snapshot.clone(),
Some((*self.local_id.as_bytes(), version)),
)
.await?;
self.own_version = version;
self.params.vips = candidate;
self.update_vips();
let peers: Vec<_> = self.sessions.keys().copied().collect();
for peer in peers {
let _ = self.send_to(
peer,
ControlMessage::Vips {
entries: snapshot.clone(),
},
);
}
Ok(self.params.routes.vips.statuses(self.network_id))
}
async fn receive_vips(&mut self, entries: Vec<crate::vip::VipEntry>) -> Result<()> {
let Some(id) = entries.first().map(|e| e.vip.clone()) else {
return Ok(());
};
if entries.iter().any(|e| e.vip != id) {
return Err(crate::vip::invalid("VIP snapshot mixes service identities"));
}
let own_seen = entries
.iter()
.filter(|e| e.author == *self.local_id.as_bytes())
.map(|e| e.version)
.max()
.unwrap_or(0);
let mut candidate = self.params.vips.clone();
let changed = candidate.merge(self.network_id, entries, &self.params.limits)?;
if !changed && own_seen <= self.own_version {
return Ok(());
}
let counter =
(own_seen > self.own_version).then_some((*self.local_id.as_bytes(), own_seen));
let snapshot = candidate.entries(&id);
self.validate_vip_frame(&snapshot)?;
self.params
.storage
.put_vip_fields(self.network_id, snapshot, counter)
.await?;
self.own_version = self.own_version.max(own_seen);
self.params.vips = candidate;
self.update_vips();
self.broadcast_vips();
Ok(())
}
fn validate_vip_frame(&self, entries: &[crate::vip::VipEntry]) -> Result<()> {
let frame = Envelope {
network_id: *self.network_id.as_bytes(),
message: ControlMessage::Vips {
entries: entries.to_vec(),
},
};
let bytes = encode(&frame).map_err(|e| crate::vip::invalid(e.to_string()))?;
if bytes.len() > self.params.limits.max_frame_len {
return Err(crate::vip::invalid(
"VIP snapshot exceeds the control frame limit",
));
}
Ok(())
}
fn broadcast_vips(&mut self) {
let ids = self.params.vips.ids();
if ids.is_empty() {
return;
}
let peers: Vec<_> = self.sessions.keys().copied().collect();
for peer in peers {
let start = self.vip_cursor.get(&peer).copied().unwrap_or(0) % ids.len();
for step in 0..ids.len() {
let index = (start + step) % ids.len();
let entries = self.params.vips.entries(&ids[index]);
if self
.send_to(peer, ControlMessage::Vips { entries })
.is_err()
{
self.vip_cursor.insert(peer, index);
break;
}
self.vip_cursor.insert(peer, (index + 1) % ids.len());
}
}
}
/// Handles snapshots collected while dispatching messages. /// Handles snapshots collected while dispatching messages.
async fn drain_pending_state(&mut self) { async fn drain_pending_state(&mut self) {
for entries in std::mem::take(&mut self.pending_vips) {
if let Err(err) = self.receive_vips(entries).await {
self.metrics.protocol_violations += 1;
tracing::warn!(%err,"rejecting VIP snapshot");
}
}
for (peer, markers) in std::mem::take(&mut self.pending_retirements) { for (peer, markers) in std::mem::take(&mut self.pending_retirements) {
self.receive_retirements(peer, markers).await; self.receive_retirements(peer, markers).await;
} }
@@ -1463,6 +1685,7 @@ impl Runtime {
/// Sends everything we know to every peer. /// Sends everything we know to every peer.
fn broadcast_state(&mut self) { fn broadcast_state(&mut self) {
self.broadcast_vips();
self.broadcast_retirements(); self.broadcast_retirements();
let mut records = self.state.records(); let mut records = self.state.records();
records.truncate(self.params.limits.max_state_records); records.truncate(self.params.limits.max_state_records);
@@ -1661,8 +1884,9 @@ impl Runtime {
self.params.range_changed.send_replace(()); self.params.range_changed.send_replace(());
return; return;
} }
for plugin in &self.params.plugins { self.update_vips();
plugin.on_address_allocation(self.network_id, range, &allocations); if let Some(tunnel) = &self.params.tunnel {
tunnel.on_address_allocation(self.network_id, range, &allocations);
} }
if self.published_range != Some(range) { if self.published_range != Some(range) {
self.published_range = Some(range); self.published_range = Some(range);
@@ -1754,7 +1978,7 @@ impl Runtime {
/// The protocols this agent speaks, with the version it speaks them at. /// The protocols this agent speaks, with the version it speaks them at.
fn served_protocols(&self) -> Vec<(String, u16)> { fn served_protocols(&self) -> Vec<(String, u16)> {
self.params self.params
.plugins .carriers
.iter() .iter()
.map(|plugin| (plugin.protocol_id().to_string(), plugin.protocol_version())) .map(|plugin| (plugin.protocol_id().to_string(), plugin.protocol_version()))
.collect() .collect()
@@ -1815,6 +2039,8 @@ impl Runtime {
// reachable through somebody is decided below; the protocol's // reachable through somebody is decided below; the protocol's
// link stays either way. // link stays either way.
self.hub.clear_direct(peer, &protocol); self.hub.clear_direct(peer, &protocol);
self.data_errors
.insert(peer, format!("{protocol}: {reason}"));
self.record_protocol_failure(peer, &protocol); self.record_protocol_failure(peer, &protocol);
self.emit(Event::DataLinkDown { self.emit(Event::DataLinkDown {
network: self.network_id, network: self.network_id,
@@ -1939,6 +2165,7 @@ impl Runtime {
} }
} }
let addr = self.underlay_addr(addr);
let fresh = !self.candidate_addrs.contains_key(&peer); let fresh = !self.candidate_addrs.contains_key(&peer);
self.candidate_addrs self.candidate_addrs
.entry(peer) .entry(peer)
@@ -1954,6 +2181,10 @@ impl Runtime {
learned learned
} }
fn underlay_addr(&self, addr: iroh::EndpointAddr) -> iroh::EndpointAddr {
without_overlay_addresses(addr, &self.params.routes.overlay_ranges())
}
/// Tells the peers it is talking to about the peers it knows. /// Tells the peers it is talking to about the peers it knows.
/// ///
/// A device told about one member ends up talking to all of them: a /// A device told about one member ends up talking to all of them: a
@@ -2000,14 +2231,14 @@ impl Runtime {
// other side's own discovery may be able to resolve it. // other side's own discovery may be able to resolve it.
hints.push(crate::proto::message::PeerHint { hints.push(crate::proto::message::PeerHint {
endpoint: *peer.as_bytes(), endpoint: *peer.as_bytes(),
addrs: hint_addrs(addr), addrs: hint_addrs(&self.underlay_addr(addr.clone())),
}); });
} }
// This agent, too: it is the one member the others cannot be told // This agent, too: it is the one member the others cannot be told
// about by anybody else. // about by anybody else.
hints.push(crate::proto::message::PeerHint { hints.push(crate::proto::message::PeerHint {
endpoint: *self.local_id.as_bytes(), endpoint: *self.local_id.as_bytes(),
addrs: hint_addrs(&self.params.adapter.addr()), addrs: hint_addrs(&self.underlay_addr(self.params.adapter.addr())),
}); });
hints.truncate(self.params.limits.max_state_records); hints.truncate(self.params.limits.max_state_records);
if hints.is_empty() { if hints.is_empty() {
@@ -2052,34 +2283,18 @@ impl Runtime {
} }
} }
/// The protocol this agent and a member run their tunnel in, if they have /// Matches the single internal encryption capability, independently of carrier policy.
/// one in common.
///
/// Both ends must name the same one without asking each other, and what
/// each side was configured to prefer can differ. So the member with the
/// lower id decides, by the order it announces its protocols in, and the
/// other side reads that order from the announcement it already has.
fn tunnel_protocol(&self, session: &Session) -> Option<String> { fn tunnel_protocol(&self, session: &Session) -> Option<String> {
let usable = |ours: &[PluginCapability], theirs: &[PluginCapability], name: &str| { let tunnel = self.params.tunnel.as_ref()?;
ours.iter().any(|capability| { session
capability.enabled .capabilities
&& capability.protocol == name
&& theirs.iter().any(|other| {
other.enabled
&& other.protocol == name
&& other.version == capability.version
})
})
};
let (decides, other) = if self.local_id.as_bytes() < session.peer.as_bytes() {
(&self.local_capabilities, &session.capabilities)
} else {
(&session.capabilities, &self.local_capabilities)
};
decides
.iter() .iter()
.find(|capability| usable(decides, other, &capability.protocol)) .any(|cap| {
.map(|capability| capability.protocol.clone()) cap.enabled
&& cap.protocol == tunnel.protocol_id()
&& cap.version == tunnel.protocol_version()
})
.then(|| tunnel.protocol_id().to_owned())
} }
/// Builds the graph off the packet path. Every remote row comes from that /// Builds the graph off the packet path. Every remote row comes from that
@@ -2121,9 +2336,9 @@ impl Runtime {
} }
let Some(plugin) = self let Some(plugin) = self
.params .params
.plugins .tunnel
.iter() .as_ref()
.find(|plugin| plugin.protocol_id() == protocol) .filter(|plugin| plugin.protocol_id() == protocol)
.cloned() .cloned()
else { else {
continue; continue;
@@ -2144,6 +2359,7 @@ impl Runtime {
}); });
} }
} }
self.update_vips();
} }
fn handle_link_result(&mut self, outcome: LinkOutcome) { fn handle_link_result(&mut self, outcome: LinkOutcome) {
@@ -2167,6 +2383,8 @@ impl Runtime {
self.announce_reach(false); self.announce_reach(false);
} }
Err(reason) => { Err(reason) => {
self.data_errors
.insert(outcome.peer, format!("{}: {reason}", outcome.protocol));
// A data plane that cannot be set up is reported, never fatal. // A data plane that cannot be set up is reported, never fatal.
self.metrics.data_link_failures += 1; self.metrics.data_link_failures += 1;
self.record_protocol_failure(outcome.peer, &outcome.protocol); self.record_protocol_failure(outcome.peer, &outcome.protocol);
@@ -2225,7 +2443,7 @@ impl Runtime {
link.close(); link.close();
return; return;
} }
if !self.params.plugins.iter().any(|plugin| { if !self.params.carriers.iter().any(|plugin| {
plugin.protocol_id() == protocol plugin.protocol_id() == protocol
&& self.sessions.get(&peer).is_some_and(|s| { && self.sessions.get(&peer).is_some_and(|s| {
s.capabilities.iter().any(|c| { s.capabilities.iter().any(|c| {
@@ -2239,6 +2457,7 @@ impl Runtime {
return; return;
} }
self.data_errors.remove(&peer);
let key = (peer, protocol.clone()); let key = (peer, protocol.clone());
let settings = if self.local_id.as_bytes() < peer.as_bytes() { let settings = if self.local_id.as_bytes() < peer.as_bytes() {
let settings = HealthSettings::fresh(); let settings = HealthSettings::fresh();
@@ -2255,21 +2474,7 @@ impl Runtime {
} else { } else {
self.pending_health.remove(&key) self.pending_health.remove(&key)
}; };
let route = self let link = HealthLink::new(link, self.local_id, settings);
.sessions
.get(&peer)
.and_then(|s| self.tunnel_protocol(s))
.and_then(|name| self.params.plugins.iter().find(|p| p.protocol_id() == name))
.map(|plugin| ProbeRoute {
plugin: plugin.clone(),
routes: self.params.routes.clone(),
network: self.network_id,
peer,
max_probe_size: self.params.interface.as_ref().map_or(540, |interface| {
(interface.mtu() as usize).saturating_sub(36)
}),
});
let link = HealthLink::new(link, self.local_id, settings, route);
let path = link.path_description(); let path = link.path_description();
let max_datagram = link.max_datagram_size(); let max_datagram = link.max_datagram_size();
if let Some(old) = self.links.insert(key, Arc::clone(&link)) { if let Some(old) = self.links.insert(key, Arc::clone(&link)) {
@@ -2293,14 +2498,21 @@ impl Runtime {
/// Drops every link to a peer. /// Drops every link to a peer.
fn drop_links_for(&mut self, peer: EndpointId) { fn drop_links_for(&mut self, peer: EndpointId) {
self.tunnel_errors.remove(&peer);
self.retirement_cursor.remove(&peer); self.retirement_cursor.remove(&peer);
self.vip_cursor.remove(&peer);
self.reset_peer_data(peer, "peer session ended"); self.reset_peer_data(peer, "peer session ended");
if let Some(tunnel) = &self.params.tunnel {
tunnel.on_peer_gone(self.network_id, peer);
}
self.hub.clear_peer_links(peer);
self.hub.remove_peer(peer); self.hub.remove_peer(peer);
self.reachable.remove(&peer); self.reachable.remove(&peer);
} }
/// Retire only the data plane; authenticated membership and topology remain. /// Retire only the data plane; authenticated membership and topology remain.
fn reset_peer_data(&mut self, peer: EndpointId, reason: &str) { fn reset_peer_data(&mut self, peer: EndpointId, reason: &str) {
self.data_errors.remove(&peer);
self.link_attempts.retain(|(id, _), _| *id != peer); self.link_attempts.retain(|(id, _), _| *id != peer);
self.opening.retain(|(id, _)| *id != peer); self.opening.retain(|(id, _)| *id != peer);
self.link_tasks.retain(|(id, _), task| { self.link_tasks.retain(|(id, _), task| {
@@ -2311,7 +2523,6 @@ impl Runtime {
true true
} }
}); });
self.hub.clear_peer_links(peer);
let keys: Vec<_> = self let keys: Vec<_> = self
.links .links
.keys() .keys()
@@ -2322,6 +2533,7 @@ impl Runtime {
if let Some(link) = self.links.remove(&key) { if let Some(link) = self.links.remove(&key) {
link.close(); link.close();
} }
self.hub.clear_direct(peer, &key.1);
self.emit(Event::DataLinkDown { self.emit(Event::DataLinkDown {
network: self.network_id, network: self.network_id,
peer, peer,
@@ -2332,7 +2544,7 @@ impl Runtime {
self.protocol_dial_states.retain(|(id, _), _| *id != peer); self.protocol_dial_states.retain(|(id, _), _| *id != peer);
self.last_failed_protocol.remove(&peer); self.last_failed_protocol.remove(&peer);
self.pending_health.retain(|(id, _), _| *id != peer); self.pending_health.retain(|(id, _), _| *id != peer);
for plugin in &self.params.plugins { for plugin in &self.params.carriers {
plugin.on_peer_gone(self.network_id, peer); plugin.on_peer_gone(self.network_id, peer);
} }
} }
@@ -2418,11 +2630,27 @@ impl Runtime {
fn local_announcement(&mut self) -> Announcement { fn local_announcement(&mut self) -> Announcement {
let mut capabilities = Vec::new(); let mut capabilities = Vec::new();
let mut errors = Vec::new(); let mut errors = Vec::new();
for plugin in &self.params.plugins { let sources = self
match plugin.local_capability(self.network_id) { .params
Ok(Some(capability)) => capabilities.push(capability), .carriers
.iter()
.map(|carrier| {
(
carrier.protocol_id().to_owned(),
carrier.local_capability(self.network_id),
)
})
.chain(self.params.tunnel.iter().map(|tunnel| {
(
tunnel.protocol_id().to_owned(),
tunnel.local_capability(self.network_id),
)
}));
for (protocol, result) in sources {
match result {
Ok(Some(cap)) => capabilities.push(cap),
Ok(None) => {} Ok(None) => {}
Err(err) => errors.push((plugin.protocol_id().to_string(), err.to_string())), Err(err) => errors.push((protocol, err.to_string())),
} }
} }
for (protocol, reason) in errors { for (protocol, reason) in errors {
@@ -2654,6 +2882,7 @@ impl Runtime {
tracing::debug!(%err, "could not queue pong"); tracing::debug!(%err, "could not queue pong");
} }
} }
ControlMessage::Vips { entries } => self.pending_vips.push(entries.clone()),
ControlMessage::State { records } => { ControlMessage::State { records } => {
self.pending_state.push((peer, records.clone())); self.pending_state.push((peer, records.clone()));
} }
@@ -2689,34 +2918,57 @@ impl Runtime {
fn dispatch_capabilities(&mut self, peer: EndpointId, capabilities: &[PluginCapability]) { fn dispatch_capabilities(&mut self, peer: EndpointId, capabilities: &[PluginCapability]) {
let mut errors = Vec::new(); let mut errors = Vec::new();
for capability in capabilities { for cap in capabilities {
for plugin in &self.params.plugins { let carriers = self
if plugin.protocol_id() != capability.protocol { .params
continue; .carriers
} .iter()
if plugin.protocol_version() != capability.version { .filter(|carrier| carrier.protocol_id() == cap.protocol)
// Said once per peer and protocol. A version that will .map(|carrier| {
// not match this build will not match it on the next (
// announcement either, and repeating it every round carrier.protocol_version(),
// would bury everything else. if carrier.protocol_version() == cap.version {
let key = (peer, capability.protocol.clone()); carrier.on_peer_capability(self.network_id, peer, cap)
if self.reported_mismatch.insert(key) { } else {
errors.push(( Ok(())
capability.protocol.clone(), },
format!( )
"peer speaks {} version {} and this build speaks {}, so there \ });
is no data plane with it; the control plane is unaffected", let tunnel = self
capability.protocol, .params
capability.version, .tunnel
plugin.protocol_version() .iter()
), .filter(|tunnel| tunnel.protocol_id() == cap.protocol)
)); .map(|tunnel| {
(
tunnel.protocol_version(),
if tunnel.protocol_version() == cap.version {
tunnel.on_peer_capability(self.network_id, peer, cap)
} else {
Ok(())
},
)
});
for (version, result) in carriers.chain(tunnel) {
if version != cap.version {
if self.reported_mismatch.insert((peer, cap.protocol.clone())) {
errors.push((cap.protocol.clone(), format!("peer speaks {} version {} and this build speaks {version}; update both agents; control plane is unaffected", cap.protocol, cap.version)));
}
} else {
let is_tunnel = self
.params
.tunnel
.as_ref()
.is_some_and(|t| t.protocol_id() == cap.protocol);
if let Err(err) = result {
if is_tunnel {
self.tunnel_errors
.insert(peer, format!("encrypted data channel: {err}"));
}
errors.push((cap.protocol.clone(), err.to_string()));
} else if is_tunnel {
self.tunnel_errors.remove(&peer);
} }
continue;
}
// The core hands the opaque payload over without interpreting it.
if let Err(err) = plugin.on_peer_capability(self.network_id, peer, capability) {
errors.push((plugin.protocol_id().to_string(), err.to_string()));
} }
} }
} }
@@ -2795,6 +3047,58 @@ impl Runtime {
}) })
} }
fn data_error(&self, session: &Session) -> Option<String> {
if self.params.carriers.is_empty() {
return Some("data transports disabled locally".into());
}
if self.params.tunnel.is_some() && self.tunnel_protocol(session).is_none() {
return Some(
"peer has no compatible encrypted data channel; update both agents".into(),
);
}
if let Some(reason) = self.tunnel_errors.get(&session.peer) {
return Some(reason.clone());
}
if !matches!(self.data_path(session.peer), DataPath::None) {
return None;
}
let served = self.served_protocols();
if !served.iter().any(|(name, version)| {
session
.capabilities
.iter()
.any(|cap| cap.enabled && cap.protocol == *name && cap.version == *version)
}) {
let local = served
.iter()
.map(|(name, _)| name.as_str())
.collect::<Vec<_>>()
.join(",");
let remote = session
.capabilities
.iter()
.filter(|cap| {
cap.enabled
&& self
.params
.tunnel
.as_ref()
.is_none_or(|t| t.protocol_id() != cap.protocol)
})
.map(|cap| cap.protocol.as_str())
.collect::<Vec<_>>()
.join(",");
return Some(format!(
"no common external transport (local: {local}; peer: {})",
if remote.is_empty() { "none" } else { &remote }
));
}
self.data_errors
.get(&session.peer)
.cloned()
.or_else(|| Some("establishing external data transport".into()))
}
fn status(&self) -> NetworkStatus { fn status(&self) -> NetworkStatus {
let served = self.served_protocols(); let served = self.served_protocols();
let mut peers: Vec<PeerStatus> = self let mut peers: Vec<PeerStatus> = self
@@ -2822,6 +3126,7 @@ impl Runtime {
}) })
.map(|capability| capability.protocol.clone()) .map(|capability| capability.protocol.clone())
.collect(), .collect(),
data_error: self.data_error(session),
capabilities: session.capabilities.clone(), capabilities: session.capabilities.clone(),
connected_for: session.established.elapsed(), connected_for: session.established.elapsed(),
paths: snapshot.paths, paths: snapshot.paths,
@@ -2881,9 +3186,10 @@ impl Runtime {
members.dedup_by(|a, b| a.endpoint_id == b.endpoint_id); members.dedup_by(|a, b| a.endpoint_id == b.endpoint_id);
NetworkStatus { NetworkStatus {
vips: self.params.routes.vips.statuses(self.network_id),
protocols: self protocols: self
.params .params
.plugins .carriers
.iter() .iter()
.map(|p| p.protocol_id().to_owned()) .map(|p| p.protocol_id().to_owned())
.collect(), .collect(),
@@ -3031,10 +3337,43 @@ fn merge_addr(existing: &mut iroh::EndpointAddr, incoming: &iroh::EndpointAddr)
} }
} }
/// Introductions and discovery candidates are underlay addresses. An overlay
/// address here would send the tunnel's own transport recursively through itself.
fn without_overlay_addresses(
mut addr: iroh::EndpointAddr,
ranges: &[Ipv4Range],
) -> iroh::EndpointAddr {
addr.addrs.retain(|address| match address {
iroh::TransportAddr::Ip(socket) => match socket.ip() {
std::net::IpAddr::V4(ip) => !ranges.iter().any(|range| range.contains(ip)),
std::net::IpAddr::V6(_) => true,
},
_ => true,
});
addr
}
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*; use super::*;
#[test]
fn underlay_hints_exclude_overlay_subnets_but_preserve_lan_and_relay_candidates() {
let peer = crate::identity::DeviceIdentity::generate().endpoint_id();
let overlay = "10.13.37.99:1234".parse().unwrap();
let lan = "192.168.1.5:1234".parse().unwrap();
let relay: iroh::RelayUrl = "https://relay.example.com".parse().unwrap();
let addr = iroh::EndpointAddr::new(peer)
.with_ip_addr(overlay)
.with_ip_addr(lan)
.with_relay_url(relay.clone());
let filtered = without_overlay_addresses(addr, &[crate::state::DEFAULT_IPV4_RANGE]);
assert_eq!(filtered.ip_addrs().copied().collect::<Vec<_>>(), vec![lan]);
assert!(filtered.addrs.contains(&iroh::TransportAddr::Relay(relay)));
assert_eq!(filtered.id, peer);
}
#[test] #[test]
fn test_data_link_backoff_progression() { fn test_data_link_backoff_progression() {
assert_eq!(data_link_backoff(0), Duration::ZERO); assert_eq!(data_link_backoff(0), Duration::ZERO);
crates/tsunagi-core/src/agent/status.rs
+4
View File
@@ -158,6 +158,8 @@ pub struct PeerStatus {
pub protocols: Vec<String>, pub protocols: Vec<String>,
/// Capabilities the peer announced. Payloads stay opaque. /// Capabilities the peer announced. Payloads stay opaque.
pub capabilities: Vec<PluginCapability>, pub capabilities: Vec<PluginCapability>,
/// Why no data route is available, when known.
pub data_error: Option<String>,
/// How long the session has been up. /// How long the session has been up.
pub connected_for: Duration, pub connected_for: Duration,
/// Verified paths of the underlying connection. /// Verified paths of the underlying connection.
@@ -225,6 +227,8 @@ pub struct RangeConflict {
/// Status of one network. /// Status of one network.
#[derive(Debug, Clone)] #[derive(Debug, Clone)]
pub struct NetworkStatus { pub struct NetworkStatus {
/// Shared service configuration with this client's routing observations.
pub vips: Vec<crate::vip::VipStatus>,
/// Enabled local data-plane protocols, in preference order. /// Enabled local data-plane protocols, in preference order.
pub protocols: Vec<String>, pub protocols: Vec<String>,
/// Whether this network is requested to start or resume automatically. /// Whether this network is requested to start or resume automatically.
crates/tsunagi-core/src/config.rs
+42 -9
View File
@@ -158,6 +158,22 @@ pub enum TransportPolicy {
/// object where that is possible (notably [`Limits::max_frame_len`]). /// object where that is possible (notably [`Limits::max_frame_len`]).
#[derive(Debug, Clone, PartialEq, Eq)] #[derive(Debug, Clone, PartialEq, Eq)]
pub struct Limits { pub struct Limits {
/// Maximum active VIP identities per network.
pub max_vips: usize,
/// Maximum permanent VIP deletion tombstones per network.
pub max_vip_tombstones: usize,
/// Maximum concurrent active TCP port checks across the agent.
pub max_vip_checks: usize,
/// Maximum signed fields per VIP, including removed fields.
pub max_vip_fields: usize,
/// Maximum listeners per VIP.
pub max_vip_listeners: usize,
/// Maximum backend endpoints per listener.
pub max_vip_backends: usize,
/// Maximum simultaneous local VIP flow bindings.
pub max_vip_flows: usize,
/// Maximum incomplete IPv4 datagrams per network in VIP reassembly.
pub max_vip_fragments: usize,
/// Largest accepted control frame payload, in bytes. /// Largest accepted control frame payload, in bytes.
pub max_frame_len: usize, pub max_frame_len: usize,
/// Largest accepted hostname, in bytes. /// Largest accepted hostname, in bytes.
@@ -205,6 +221,14 @@ pub struct Limits {
impl Default for Limits { impl Default for Limits {
fn default() -> Self { fn default() -> Self {
Self { Self {
max_vips: 64,
max_vip_tombstones: 1024,
max_vip_checks: 32,
max_vip_fields: 192,
max_vip_listeners: 16,
max_vip_backends: 32,
max_vip_flows: 4096,
max_vip_fragments: 64,
max_frame_len: 64 * 1024, max_frame_len: 64 * 1024,
max_hostname_len: 255, max_hostname_len: 255,
max_capabilities: 16, max_capabilities: 16,
@@ -329,10 +353,12 @@ pub struct AgentConfig {
pub limits: Limits, pub limits: Limits,
/// Reconnect backoff policy. /// Reconnect backoff policy.
pub reconnect: ReconnectPolicy, pub reconnect: ReconnectPolicy,
/// IP plugins whose capabilities are announced and dispatched. /// External data carriers whose capabilities are announced and dispatched.
pub plugins: Vec<SharedPlugin>, pub carriers: Vec<crate::dataplane::transport::SharedCarrier>,
/// The single internal end-to-end engine, independent of carrier selection.
pub tunnel: Option<SharedPlugin>,
/// Initial ordered selection for networks without a saved override. /// Initial ordered selection for networks without a saved override.
/// `None` enables all registered plugins; an empty list is control-only. /// `None` enables all registered carriers; an empty list is control-only.
pub default_protocols: Option<Vec<String>>, pub default_protocols: Option<Vec<String>>,
/// Custom packet transports registered per protocol. /// Custom packet transports registered per protocol.
pub custom_transports: pub custom_transports:
@@ -381,7 +407,8 @@ impl AgentConfig {
discovery_interval: Duration::from_secs(5), discovery_interval: Duration::from_secs(5),
limits: Limits::default(), limits: Limits::default(),
reconnect: ReconnectPolicy::default(), reconnect: ReconnectPolicy::default(),
plugins: Vec::new(), carriers: Vec::new(),
tunnel: None,
default_protocols: None, default_protocols: None,
custom_transports: std::collections::HashMap::new(), custom_transports: std::collections::HashMap::new(),
tun_factory: None, tun_factory: None,
@@ -465,9 +492,15 @@ impl AgentConfig {
self self
} }
/// Registers an IP plugin. /// Registers an external data carrier.
pub fn with_plugin(mut self, plugin: SharedPlugin) -> Self { pub fn with_carrier(mut self, plugin: crate::dataplane::transport::SharedCarrier) -> Self {
self.plugins.push(plugin); self.carriers.push(plugin);
self
}
/// Installs the internal encryption engine. It is never a selectable carrier.
pub fn with_tunnel(mut self, tunnel: SharedPlugin) -> Self {
self.tunnel = Some(tunnel);
self self
} }
@@ -521,9 +554,9 @@ impl std::fmt::Debug for AgentConfig {
.field("limits", &self.limits) .field("limits", &self.limits)
.field("reconnect", &self.reconnect) .field("reconnect", &self.reconnect)
.field( .field(
"plugins", "carriers",
&self &self
.plugins .carriers
.iter() .iter()
.map(|p| p.protocol_id().to_string()) .map(|p| p.protocol_id().to_string())
.collect::<Vec<_>>(), .collect::<Vec<_>>(),
crates/tsunagi-core/src/dataplane/mod.rs
+13 -21
View File
@@ -1,28 +1,19 @@
//! Boundary between the control plane core and future IP plugins. //! Boundaries between the agent, one IP encryption tunnel, and external carriers.
//! //!
//! The data plane is where actual IP connectivity is created. WireGuard is the //! The core moves bounded opaque capabilities without parsing cipher payloads.
//! first planned plugin; none is implemented here. //! An [`IpPlugin`] sees routed IP packets and a logical [`PacketLink`] per peer;
//! it never learns reachability or the network secret. The agent configures one
//! such plugin. Built-in agents use the same internal WireGuard cipher for every
//! peer, regardless of which external transports carry the encrypted bytes.
//! //!
//! Two rules shape this module: //! [`transport::DataCarrier`] owns external capability negotiation and lifecycle.
//! //! It sees reachability but never plaintext overlay packets. A relay forwards
//! 1. **The core never parses plugin payloads.** A [`PluginCapability`] carries //! the encrypted inner bytes without entering an IP plugin or the TUN.
//! a protocol id, a version, an enabled flag and a bounded opaque blob. The //! Data-plane failure never stops control sessions or discovery.
//! core transports the blob and hands it to the matching plugin. It does not
//! know what a WireGuard configuration looks like.
//! 2. **Plugin keys and lifecycle are separate from iroh identity and from the
//! network secret.** A plugin owns its own keys and its own system objects.
//!
//! An iroh address is *not* automatically a WireGuard address. A future plugin
//! is expected to gather its own reachability information and ship it through
//! the control plane as its announcement payload.
//!
//! A data plane failure never stops the daemon: errors returned here are
//! recorded and surfaced, the control plane keeps running.
pub mod relay; pub mod relay;
pub mod routing; pub mod routing;
pub mod transport; pub mod transport;
pub mod tunnel;
use std::sync::Arc; use std::sync::Arc;
@@ -35,7 +26,6 @@ use crate::identity::NetworkId;
pub use transport::{ pub use transport::{
MultiPacketTransport, PacketLink, PacketTransport, SharedLink, TransportError, MultiPacketTransport, PacketLink, PacketTransport, SharedLink, TransportError,
}; };
pub use tunnel::{GenericTunnelPlugin, TunnelCodec};
/// Maximum length of a plugin protocol identifier. /// Maximum length of a plugin protocol identifier.
pub const MAX_PROTOCOL_ID_LEN: usize = 32; pub const MAX_PROTOCOL_ID_LEN: usize = 32;
@@ -438,7 +428,9 @@ pub trait IpPlugin: Send + Sync + std::fmt::Debug + 'static {
/// Always preceded by [`IpPlugin::on_network_deactivated`], so this is /// Always preceded by [`IpPlugin::on_network_deactivated`], so this is
/// only about what outlives a session. Defaulted to nothing, for a /// only about what outlives a session. Defaulted to nothing, for a
/// plugin that stores nothing. /// plugin that stores nothing.
fn on_network_forgotten(&self, _network: NetworkId) {} fn on_network_forgotten(&self, _network: NetworkId) -> BoxFuture<'_, ()> {
Box::pin(async {})
}
/// Called once when the agent shuts down. /// Called once when the agent shuts down.
/// ///
crates/tsunagi-core/src/dataplane/transport/carrier.rs
+44
View File
@@ -0,0 +1,44 @@
//! External link negotiation, deliberately separate from the IP encryption plugin.
use crate::dataplane::{PluginCapability, PluginError, ProtocolOption};
use crate::{BoxFuture, NetworkId};
use iroh::EndpointId;
use std::sync::Arc;
/// Capability and lifecycle of one external data carrier.
/// Only this layer sees reachability. It never receives plaintext overlay IP packets.
pub trait DataCarrier: Send + Sync + std::fmt::Debug + 'static {
/// External protocol identifier used in local policy and peer negotiation.
fn protocol_id(&self) -> &str;
/// Wire version, independent of software releases and the internal tunnel.
fn protocol_version(&self) -> u16;
/// Supported carrier settings.
fn options(&self) -> &'static [ProtocolOption] {
&[]
}
/// Bounded opaque reachability/negotiation capability for one network.
fn local_capability(&self, network: NetworkId)
-> Result<Option<PluginCapability>, PluginError>;
/// Installs a capability received through an authenticated membership session.
fn on_peer_capability(
&self,
network: NetworkId,
peer: EndpointId,
capability: &PluginCapability,
) -> Result<(), PluginError>;
/// Supplies the actual owned interface name, including kernel-assigned names.
fn on_interface_ready(&self, _name: &str) {}
/// Activates this carrier for one local network.
fn on_network_activated(&self, _network: NetworkId) {}
/// Retires all negotiation and links for this member.
fn on_peer_gone(&self, network: NetworkId, peer: EndpointId);
/// Retires this network's sockets and negotiations.
fn on_network_deactivated(&self, network: NetworkId);
/// Forgets durable carrier settings, if any.
fn on_network_forgotten(&self, _network: NetworkId) {}
/// Stops carrier tasks, with the agent enforcing a shutdown deadline.
fn shutdown(&self) -> BoxFuture<'_, ()> {
Box::pin(async {})
}
}
/// One shared carrier adapter.
pub type SharedCarrier = Arc<dyn DataCarrier>;
crates/tsunagi-core/src/dataplane/transport/health.rs
+5 -69
View File
@@ -1,8 +1,6 @@
//! Authenticated probes of the actual data link, independent of control RTT. //! Authenticated probes of the actual data link, independent of control RTT.
//! Keys are negotiated privately for each physical link. No fixed wire magic. //! Keys are negotiated privately for each physical link. No fixed wire magic.
use super::{PacketLink, SharedLink, TransportError}; use super::{PacketLink, SharedLink, TransportError};
use crate::dataplane::SharedPlugin;
use crate::overlay::RoutingTable;
use crate::{BoxFuture, NetworkId}; use crate::{BoxFuture, NetworkId};
use bytes::Bytes; use bytes::Bytes;
use hmac::{Hmac, KeyInit, Mac}; use hmac::{Hmac, KeyInit, Mac};
@@ -70,52 +68,8 @@ impl Samples {
&& self.recent.iter().filter(|ok| !**ok).count() * 100 >= self.recent.len() * 60 && self.recent.iter().filter(|ok| !**ok).count() * 100 >= self.recent.len() * 60
} }
} }
// This prefix is inside end-to-end encryption. It never appears on the UDP wire.
const IP_MAGIC: &[u8; 16] = b"tsunagi-health01";
pub(crate) fn ip_payload(packet: &[u8]) -> Option<&[u8]> {
(packet.len() >= 36 && packet[0] == 0x45 && packet[9] == 253 && &packet[20..36] == IP_MAGIC)
.then(|| &packet[36..])
}
/// Inject probes through the same encrypted tunnel as application IP packets.
/// This matters for pure WG: raw UDP probes could pass DPI while WG is blocked.
pub(crate) struct ProbeRoute {
pub(crate) plugin: SharedPlugin,
pub(crate) routes: Arc<RoutingTable>,
pub(crate) network: NetworkId,
pub(crate) peer: EndpointId,
pub(crate) max_probe_size: usize,
}
impl ProbeRoute {
fn send(&self, payload: Bytes) -> bool {
let Some((source, destination)) = self.routes.probe_addresses(self.network, self.peer)
else {
return false;
};
let mut packet = vec![0; 36];
packet[0] = 0x45;
packet[8] = 64;
packet[9] = 253;
packet[12..16].copy_from_slice(&source.octets());
packet[16..20].copy_from_slice(&destination.octets());
packet[20..36].copy_from_slice(IP_MAGIC);
let length = (packet.len() + payload.len()) as u16;
packet[2..4].copy_from_slice(&length.to_be_bytes());
let sum = packet[..20]
.as_chunks::<2>()
.0
.iter()
.map(|b| u32::from(u16::from_be_bytes(*b)))
.sum::<u32>();
let sum = (sum & 0xffff) + (sum >> 16);
packet[10..12].copy_from_slice(&(!(sum as u16)).to_be_bytes());
packet.extend_from_slice(&payload);
self.plugin
.carry(self.network, self.peer, Bytes::from(packet))
}
}
/// A monitored physical link. It forwards application bytes unchanged. /// A monitored physical link. It forwards application bytes unchanged.
pub(crate) struct HealthLink { pub(crate) struct HealthLink {
probes: mpsc::Sender<Bytes>,
raw: SharedLink, raw: SharedLink,
settings: Mutex<Option<Settings>>, settings: Mutex<Option<Settings>>,
rx: tokio::sync::Mutex<mpsc::Receiver<Bytes>>, rx: tokio::sync::Mutex<mpsc::Receiver<Bytes>>,
@@ -136,17 +90,10 @@ fn lock<T>(m: &Mutex<T>) -> std::sync::MutexGuard<'_, T> {
} }
} }
impl HealthLink { impl HealthLink {
pub(crate) fn new( pub(crate) fn new(raw: SharedLink, local: EndpointId, settings: Option<Settings>) -> Arc<Self> {
raw: SharedLink,
local: EndpointId,
settings: Option<Settings>,
route: Option<ProbeRoute>,
) -> Arc<Self> {
let (tx, rx) = mpsc::channel(64); let (tx, rx) = mpsc::channel(64);
let (probe_tx, mut probe_rx) = mpsc::channel(64);
let (stop, _) = watch::channel(false); let (stop, _) = watch::channel(false);
let link = Arc::new(Self { let link = Arc::new(Self {
probes: probe_tx,
raw: raw.clone(), raw: raw.clone(),
settings: Mutex::new(settings), settings: Mutex::new(settings),
rx: tokio::sync::Mutex::new(rx), rx: tokio::sync::Mutex::new(rx),
@@ -174,29 +121,21 @@ impl HealthLink {
*lock(&link.reason)=Some("data delivery checks: at least 60% loss over the last 20 probes".into());break; *lock(&link.reason)=Some("data delivery checks: at least 60% loss over the last 20 probes".into());break;
} }
sequence=sequence.wrapping_add(1); sequence=sequence.wrapping_add(1);
let size=route.as_ref().map_or(raw.max_datagram_size(),|route|route.max_probe_size); let size=raw.max_datagram_size();
let packet=probe(&settings,local,1,sequence,size.min(1000)); let packet=probe(&settings,local,1,sequence,size.min(1000));
if let Some(route)=&route { if route.send(packet) {samples.sent(sequence,now);} } samples.sent(sequence,now);
else { let _=raw.send(packet); samples.sent(sequence,now); } let _=raw.send(packet);
} }
packet=raw.recv()=>{ packet=raw.recv()=>{
let Some(packet)=packet else{break;};let Some(link)=weak.upgrade()else{break;}; let Some(packet)=packet else{break;};let Some(link)=weak.upgrade()else{break;};
let settings=lock(&link.settings).clone(); let settings=lock(&link.settings).clone();
if route.is_none() && let Some(settings)=settings && packet.starts_with(&settings.marker) { if let Some(settings)=settings && packet.starts_with(&settings.marker) {
if let Some((kind,seq))=verify(&settings,raw.peer(),&packet) { if let Some((kind,seq))=verify(&settings,raw.peer(),&packet) {
if kind==1 {let _=raw.send(probe(&settings,local,2,seq,packet.len()));} if kind==1 {let _=raw.send(probe(&settings,local,2,seq,packet.len()));}
else if kind==2 {samples.acknowledge(seq);} else if kind==2 {samples.acknowledge(seq);}
} }
} else {let _=tx.try_send(packet);} } else {let _=tx.try_send(packet);}
} }
packet=probe_rx.recv(),if route.is_some()=>{
let Some(packet)=packet else {break;};let Some(link)=weak.upgrade()else{break;};
let Some(settings)=lock(&link.settings).clone()else{continue;};
if let Some((kind,seq))=verify(&settings,raw.peer(),&packet) {
if kind==1 { if let Some(route)=&route { route.send(probe(&settings,local,2,seq,packet.len().min(route.max_probe_size))); } }
else if kind==2 {samples.acknowledge(seq);}
}
}
} }
} }
@@ -207,9 +146,6 @@ impl HealthLink {
}); });
link link
} }
pub(crate) fn receive_probe(&self, packet: Bytes) {
let _ = self.probes.try_send(packet);
}
pub(crate) fn configure(&self, settings: Settings) { pub(crate) fn configure(&self, settings: Settings) {
*lock(&self.settings) = Some(settings); *lock(&self.settings) = Some(settings);
} }
crates/tsunagi-core/src/dataplane/transport/iroh_link.rs
+55 -1
View File
@@ -40,6 +40,49 @@ use crate::proto::{read_frame, write_frame};
use super::fragments::{self, Reassembler}; use super::fragments::{self, Reassembler};
use super::{InboundLink, PacketLink, PacketTransport, SharedLink, TransportError}; use super::{InboundLink, PacketLink, PacketTransport, SharedLink, TransportError};
/// QUIC carrier capability. Encryption of end-to-end IP packets is independent.
#[derive(Debug, Default)]
pub struct IrohCarrier;
impl super::DataCarrier for IrohCarrier {
fn protocol_id(&self) -> &str {
"wg-quic"
}
fn protocol_version(&self) -> u16 {
1
}
fn local_capability(
&self,
_: NetworkId,
) -> Result<Option<crate::dataplane::PluginCapability>, crate::dataplane::PluginError> {
Ok(Some(crate::dataplane::PluginCapability {
protocol: self.protocol_id().into(),
version: self.protocol_version(),
enabled: true,
data: Vec::new(),
}))
}
fn on_peer_capability(
&self,
_: NetworkId,
_: EndpointId,
capability: &crate::dataplane::PluginCapability,
) -> Result<(), crate::dataplane::PluginError> {
if capability.protocol != self.protocol_id()
|| capability.version != self.protocol_version()
|| !capability.enabled
|| !capability.data.is_empty()
{
return Err(crate::dataplane::PluginError::Rejected(
"unsupported QUIC carrier capability".into(),
));
}
Ok(())
}
fn on_peer_gone(&self, _: NetworkId, _: EndpointId) {}
fn on_network_deactivated(&self, _: NetworkId) {}
}
/// What the iroh transport needs from the agent. /// What the iroh transport needs from the agent.
/// ///
/// Implemented by the agent, which is the only thing that knows which networks /// Implemented by the agent, which is the only thing that knows which networks
@@ -190,6 +233,10 @@ impl PacketLink for IrohLink {
self.conn.close_reason().is_some() self.conn.close_reason().is_some()
} }
fn failure_reason(&self) -> Option<String> {
self.conn.close_reason().map(|reason| reason.to_string())
}
fn path_description(&self) -> String { fn path_description(&self) -> String {
// Report what iroh actually knows, never a guess. // Report what iroh actually knows, never a guess.
let snapshot = crate::net::snapshot_connection(&self.conn); let snapshot = crate::net::snapshot_connection(&self.conn);
@@ -264,7 +311,8 @@ impl IrohTransport {
)); ));
} }
let serves = self.lookup.serves(outcome.network_id, &open.protocol).await; let serves = open.protocol == "wg-quic"
&& self.lookup.serves(outcome.network_id, &open.protocol).await;
if open.max_datagram == 0 || conn.max_datagram_size().is_none() { if open.max_datagram == 0 || conn.max_datagram_size().is_none() {
return Err(TransportError::Other( return Err(TransportError::Other(
"data channel has no datagram support".into(), "data channel has no datagram support".into(),
@@ -325,6 +373,12 @@ impl PacketTransport for IrohTransport {
"data channel protocol id is out of bounds".into(), "data channel protocol id is out of bounds".into(),
)); ));
} }
// This default must never impersonate a missing native carrier.
if protocol != "wg-quic" {
return Err(TransportError::Declined(format!(
"no native `{protocol}` data transport registered"
)));
}
let keys = self let keys = self
.lookup .lookup
.snapshot() .snapshot()
crates/tsunagi-core/src/dataplane/transport/mod.rs
+3
View File
@@ -33,6 +33,9 @@ mod fragments;
pub(crate) mod health; pub(crate) mod health;
pub mod iroh_link; pub mod iroh_link;
pub mod carrier;
pub use carrier::{DataCarrier, SharedCarrier};
use std::sync::Arc; use std::sync::Arc;
use bytes::Bytes; use bytes::Bytes;
crates/tsunagi-core/src/dataplane/tunnel.rs
-380
View File
@@ -1,380 +0,0 @@
//! Generic harness for IP tunnel plugins.
//!
//! Implementing [`super::IpPlugin`] directly requires managing per-peer background
//! tasks, synchronization, packet queues, and graceful shutdown.
//!
//! [`TunnelCodec`] provides a simplified interface where a plugin author only
//! defines cryptographic transformation and handshake payloads.
//! [`GenericTunnelPlugin`] adapts any [`TunnelCodec`] into a full [`super::IpPlugin`].
use std::collections::HashMap;
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::{Arc, Mutex, RwLock};
use bytes::Bytes;
use iroh::EndpointId;
use tokio::task::JoinHandle;
use super::{IpPlugin, PluginCapability, PluginContext, PluginError, ProtocolOption, SharedLink};
use crate::BoxFuture;
use crate::identity::NetworkId;
/// The simplified contract for an IP tunnel protocol implementation.
///
/// Implementors only handle key announcements, link lifecycle events,
/// and packet encrypt/decrypt. Concurrency and task management are handled
/// by [`GenericTunnelPlugin`].
pub trait TunnelCodec: Send + Sync + 'static {
/// Protocol identifier, e.g. `tcp-tls`.
fn protocol_id(&self) -> &str;
/// Protocol wire version.
fn protocol_version(&self) -> u16;
/// Options accepted by the protocol.
fn options(&self) -> &'static [ProtocolOption] {
&[]
}
/// Generates local capability announcement data for a network.
fn local_capability_data(&self, network: NetworkId) -> Result<Vec<u8>, PluginError>;
/// Handles a peer's announced capability payload.
fn on_peer_capability_data(
&self,
network: NetworkId,
peer: EndpointId,
data: &[u8],
) -> Result<(), PluginError>;
/// Called when an authenticated link to a peer is established.
fn on_peer_link_up(&self, network: NetworkId, peer: EndpointId, link: &SharedLink) {
let _ = (network, peer, link);
}
/// Called when a peer goes away.
fn on_peer_down(&self, network: NetworkId, peer: EndpointId) {
let _ = (network, peer);
}
/// Encrypts an outbound IP packet into a payload to send over the link.
fn encrypt(
&self,
network: NetworkId,
peer: EndpointId,
packet: &[u8],
) -> Result<Bytes, PluginError>;
/// Decrypts an inbound payload from the link into an IP packet.
fn decrypt(
&self,
network: NetworkId,
peer: EndpointId,
payload: &[u8],
) -> Result<Bytes, PluginError>;
/// Called during shutdown.
fn shutdown<'a>(&'a self) -> BoxFuture<'a, ()> {
Box::pin(async {})
}
}
struct TunnelPeerState {
link: SharedLink,
tx_packets: AtomicU64,
rx_packets: AtomicU64,
tx_bytes: AtomicU64,
rx_bytes: AtomicU64,
dropped: AtomicU64,
established_at: std::time::Instant,
}
/// Statistics and status for one active peer in [`GenericTunnelPlugin`].
#[derive(Debug, Clone)]
pub struct TunnelPeerReport {
/// Peer endpoint identity.
pub peer: EndpointId,
/// Protocol identifier.
pub protocol: String,
/// Human-readable path description.
pub path: String,
/// Packets successfully encrypted and sent.
pub tx_packets: u64,
/// Packets successfully received and decrypted.
pub rx_packets: u64,
/// Plaintext bytes successfully encrypted and sent.
pub tx_bytes: u64,
/// Plaintext bytes successfully received and decrypted.
pub rx_bytes: u64,
/// Packets dropped due to encryption/decryption/send failures.
pub dropped: u64,
/// Uptime in seconds since the data link was established.
pub uptime_secs: u64,
}
/// Generic wrapper that turns a [`TunnelCodec`] into a full [`IpPlugin`].
pub struct GenericTunnelPlugin<C: TunnelCodec> {
codec: Arc<C>,
context: Mutex<Option<PluginContext>>,
links: RwLock<HashMap<(NetworkId, EndpointId), Arc<TunnelPeerState>>>,
tasks: Mutex<HashMap<(NetworkId, EndpointId), JoinHandle<()>>>,
}
impl<C: TunnelCodec> GenericTunnelPlugin<C> {
/// Wraps `codec` into an [`IpPlugin`].
pub fn new(codec: C) -> Self {
Self {
codec: Arc::new(codec),
context: Mutex::new(None),
links: RwLock::new(HashMap::new()),
tasks: Mutex::new(HashMap::new()),
}
}
/// Returns a reference to the underlying codec.
pub fn codec(&self) -> &C {
&self.codec
}
/// Returns the active peers with established data links for a network.
pub fn active_peers(&self, network: NetworkId) -> Vec<TunnelPeerReport> {
let guard = match self.links.read() {
Ok(g) => g,
Err(p) => p.into_inner(),
};
guard
.iter()
.filter(|((net, _), _)| *net == network)
.map(|((_, peer), state)| TunnelPeerReport {
peer: *peer,
protocol: self.codec.protocol_id().to_string(),
path: state.link.path_description(),
tx_packets: state.tx_packets.load(Ordering::Relaxed),
rx_packets: state.rx_packets.load(Ordering::Relaxed),
tx_bytes: state.tx_bytes.load(Ordering::Relaxed),
rx_bytes: state.rx_bytes.load(Ordering::Relaxed),
dropped: state.dropped.load(Ordering::Relaxed),
uptime_secs: state.established_at.elapsed().as_secs(),
})
.collect()
}
}
impl<C: TunnelCodec> std::fmt::Debug for GenericTunnelPlugin<C> {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("GenericTunnelPlugin")
.field("protocol_id", &self.codec.protocol_id())
.field("protocol_version", &self.codec.protocol_version())
.finish()
}
}
impl<C: TunnelCodec> IpPlugin for GenericTunnelPlugin<C> {
fn protocol_id(&self) -> &str {
self.codec.protocol_id()
}
fn protocol_version(&self) -> u16 {
self.codec.protocol_version()
}
fn options(&self) -> &'static [ProtocolOption] {
self.codec.options()
}
fn attach(&self, context: PluginContext) {
if let Ok(mut guard) = self.context.lock() {
*guard = Some(context);
}
}
fn local_capability(
&self,
network: NetworkId,
) -> Result<Option<PluginCapability>, PluginError> {
let data = self.codec.local_capability_data(network)?;
Ok(Some(PluginCapability {
protocol: self.codec.protocol_id().to_string(),
version: self.codec.protocol_version(),
enabled: true,
data,
}))
}
fn on_peer_capability(
&self,
network: NetworkId,
peer: EndpointId,
capability: &PluginCapability,
) -> Result<(), PluginError> {
if capability.protocol != self.codec.protocol_id() {
return Ok(());
}
self.codec
.on_peer_capability_data(network, peer, &capability.data)
}
fn carry(&self, network: NetworkId, peer: EndpointId, packet: bytes::Bytes) -> bool {
let state = {
let guard = match self.links.read() {
Ok(g) => g,
Err(p) => p.into_inner(),
};
guard.get(&(network, peer)).cloned()
};
let Some(state) = state else {
return false;
};
match self.codec.encrypt(network, peer, &packet) {
Ok(ciphertext) => {
if state.link.send(ciphertext).is_ok() {
state.tx_packets.fetch_add(1, Ordering::Relaxed);
state
.tx_bytes
.fetch_add(packet.len() as u64, Ordering::Relaxed);
true
} else {
state.dropped.fetch_add(1, Ordering::Relaxed);
false
}
}
Err(err) => {
state.dropped.fetch_add(1, Ordering::Relaxed);
tracing::debug!(%peer, %err, "tunnel encrypt failed");
false
}
}
}
fn on_peer_link(&self, network: NetworkId, peer: EndpointId, link: SharedLink) {
let state = Arc::new(TunnelPeerState {
link: Arc::clone(&link),
tx_packets: AtomicU64::new(0),
rx_packets: AtomicU64::new(0),
tx_bytes: AtomicU64::new(0),
rx_bytes: AtomicU64::new(0),
dropped: AtomicU64::new(0),
established_at: std::time::Instant::now(),
});
{
let mut guard = match self.links.write() {
Ok(g) => g,
Err(p) => p.into_inner(),
};
guard.insert((network, peer), Arc::clone(&state));
}
self.codec.on_peer_link_up(network, peer, &link);
let codec = Arc::clone(&self.codec);
let sink = match self.context.lock() {
Ok(g) => g.as_ref().map(|ctx| ctx.packet_sink()),
Err(p) => p.into_inner().as_ref().map(|ctx| ctx.packet_sink()),
};
let rx_link = Arc::clone(&link);
let rx_state = Arc::clone(&state);
let handle = tokio::spawn(async move {
while let Some(payload) = rx_link.recv().await {
match codec.decrypt(network, peer, &payload) {
Ok(packet) => {
rx_state.rx_packets.fetch_add(1, Ordering::Relaxed);
rx_state
.rx_bytes
.fetch_add(packet.len() as u64, Ordering::Relaxed);
if let Some(ref sink) = sink {
sink.deliver(network, peer, packet).await;
}
}
Err(err) => {
rx_state.dropped.fetch_add(1, Ordering::Relaxed);
tracing::debug!(%peer, %err, "tunnel decrypt failed");
}
}
}
});
let mut guard = match self.tasks.lock() {
Ok(g) => g,
Err(p) => p.into_inner(),
};
if let Some(old) = guard.insert((network, peer), handle) {
old.abort();
}
}
fn on_peer_gone(&self, network: NetworkId, peer: EndpointId) {
{
let mut guard = match self.tasks.lock() {
Ok(g) => g,
Err(p) => p.into_inner(),
};
if let Some(task) = guard.remove(&(network, peer)) {
task.abort();
}
}
{
let mut guard = match self.links.write() {
Ok(g) => g,
Err(p) => p.into_inner(),
};
guard.remove(&(network, peer));
}
self.codec.on_peer_down(network, peer);
}
fn on_network_deactivated(&self, network: NetworkId) {
let keys_to_remove: Vec<(NetworkId, EndpointId)> = {
let guard = match self.tasks.lock() {
Ok(g) => g,
Err(p) => p.into_inner(),
};
guard
.keys()
.filter(|(net, _)| *net == network)
.copied()
.collect()
};
{
let mut guard = match self.tasks.lock() {
Ok(g) => g,
Err(p) => p.into_inner(),
};
for key in &keys_to_remove {
if let Some(task) = guard.remove(key) {
task.abort();
}
}
}
{
let mut guard = match self.links.write() {
Ok(g) => g,
Err(p) => p.into_inner(),
};
guard.retain(|(net, _), _| *net != network);
}
}
fn shutdown<'a>(&'a self) -> BoxFuture<'a, ()> {
Box::pin(async move {
let mut tasks = Vec::new();
{
let mut guard = match self.tasks.lock() {
Ok(g) => g,
Err(p) => p.into_inner(),
};
for (_, task) in guard.drain() {
tasks.push(task);
}
}
for task in tasks {
task.abort();
}
self.codec.shutdown().await;
})
}
}
crates/tsunagi-core/src/ipc/mod.rs
+95 -2
View File
@@ -171,6 +171,17 @@ pub fn default_control_socket_path() -> PathBuf {
#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] #[derive(Clone, PartialEq, Eq, Serialize, Deserialize)]
#[non_exhaustive] #[non_exhaustive]
pub enum Request { pub enum Request {
/// Save a complete version-checked service draft or its deletion.
SaveVip {
/// Exact network id.
network_id: String,
/// Entire desired configuration.
config: crate::vip::VipConfig,
/// Revision originally displayed; absent for creation.
expected: Option<String>,
/// Remove the service permanently.
delete: bool,
},
/// Retire the selected offline member's observed claim. /// Retire the selected offline member's observed claim.
ForgetNode { ForgetNode {
/// Exact public network id. /// Exact public network id.
@@ -271,6 +282,9 @@ pub enum Request {
impl std::fmt::Debug for Request { impl std::fmt::Debug for Request {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self { match self {
Request::SaveVip {
network_id, config, ..
} => write!(f, "SaveVip {{ {network_id}, {} }}", config.id),
Request::ForgetNode { Request::ForgetNode {
network_id, network_id,
peer, peer,
@@ -317,6 +331,8 @@ impl std::fmt::Debug for Request {
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[non_exhaustive] #[non_exhaustive]
pub enum Response { pub enum Response {
/// Accepted configuration with fresh local routing observations.
Vips(Vec<crate::vip::VipStatus>),
/// An observed offline claim was durably retired. /// An observed offline claim was durably retired.
NodeForgotten(ForgottenNodeReport), NodeForgotten(ForgottenNodeReport),
/// Accepted ordered protocol selection. /// Accepted ordered protocol selection.
@@ -358,6 +374,57 @@ pub struct ForgottenNodeReport {
pub peers_told: u32, pub peers_told: u32,
} }
/// Saves a whole VIP draft through the running agent.
pub async fn save_vip(
socket: &Path,
network_id: &str,
config: crate::vip::VipConfig,
expected: Option<String>,
delete: bool,
) -> Result<Vec<crate::vip::VipStatus>> {
match exchange(
socket,
&Request::SaveVip {
network_id: network_id.into(),
config,
expected,
delete,
},
EXCHANGE_TIMEOUT,
)
.await?
{
Response::Vips(vips) => Ok(vips),
Response::Error(reason) => Err(Error::Rejected { reason }),
_ => Err(Error::Rejected {
reason: "unexpected VIP response".into(),
}),
}
}
/// Resolves a service by exact name, full id or unique id prefix.
pub fn resolve_vip<'a>(
network: &'a NetworkReport,
selector: &str,
) -> std::result::Result<&'a crate::vip::VipStatus, String> {
let matches: Vec<_> = network
.vips
.iter()
.filter(|v| v.config.name == selector || v.config.id.starts_with(selector))
.collect();
match matches.as_slice() {
[vip] => Ok(vip),
[] => Err(format!("no VIP matches `{selector}`")),
_ => Err(format!(
"ambiguous VIP `{selector}`: {}",
matches
.iter()
.map(|v| format!("{} ({})", v.config.name, v.config.id))
.collect::<Vec<_>>()
.join(", ")
)),
}
}
/// Resolves a node by exact hostname, full id, unique id prefix or overlay IPv4. /// Resolves a node by exact hostname, full id, unique id prefix or overlay IPv4.
/// Ambiguity is reported with names and full ids and never guessed. /// Ambiguity is reported with names and full ids and never guessed.
pub fn resolve_node<'a>( pub fn resolve_node<'a>(
@@ -642,6 +709,9 @@ pub struct ProtocolReport {
/// One network. /// One network.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)] #[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct NetworkReport { pub struct NetworkReport {
/// Shared VIPs and this client's routing observations.
#[serde(default)]
pub vips: Vec<crate::vip::VipStatus>,
/// Enabled local data-plane protocols, in preference order. /// Enabled local data-plane protocols, in preference order.
pub protocols: Vec<String>, pub protocols: Vec<String>,
/// Whether the network starts or resumes automatically when its range is free. /// Whether the network starts or resumes automatically when its range is free.
@@ -758,6 +828,8 @@ pub struct MemberReport {
/// One control plane peer. /// One control plane peer.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)] #[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct PeerReport { pub struct PeerReport {
/// Actionable carrier failure or policy mismatch; absent for a usable route.
pub data_error: Option<String>,
/// The peer's endpoint id. /// The peer's endpoint id.
pub endpoint_id: String, pub endpoint_id: String,
/// Hostname it announced, if any. /// Hostname it announced, if any.
@@ -847,10 +919,12 @@ pub struct HostRulesReport {
/// One overlay peer. /// One overlay peer.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)] #[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct OverlayPeerReport { pub struct OverlayPeerReport {
/// Opaque counter lifetime. A new tunnel or agent restart starts a new epoch.
pub counter_epoch: u64,
/// The peer's control plane identity, so a tunnel can be matched to the /// The peer's control plane identity, so a tunnel can be matched to the
/// session and the member it belongs to. /// session and the member it belongs to.
pub endpoint_id: String, pub endpoint_id: String,
/// Protocol carrying overlay traffic for this peer (e.g. "tcp-tls", "wg-quic"). /// Reserved legacy field; the internal encryption protocol is not user selectable.
pub protocol: String, pub protocol: String,
/// The peer's WireGuard public key (if applicable). /// The peer's WireGuard public key (if applicable).
pub public_key: String, pub public_key: String,
@@ -905,6 +979,16 @@ impl OverlayPeerReport {
/// running and what they can report or change. That is what keeps this module /// running and what they can report or change. That is what keeps this module
/// free of any knowledge of them. /// free of any knowledge of them.
pub trait ReportSource: Send + Sync + 'static { pub trait ReportSource: Send + Sync + 'static {
/// Atomically saves a complete VIP draft using its original revision.
fn save_vip(
&self,
_network: String,
_config: crate::vip::VipConfig,
_expected: Option<String>,
_delete: bool,
) -> BoxFuture<'_, std::result::Result<Vec<crate::vip::VipStatus>, String>> {
Box::pin(async { Err("this agent cannot manage VIPs".into()) })
}
/// Persists and announces a version-scoped node cleanup. /// Persists and announces a version-scoped node cleanup.
fn forget_node( fn forget_node(
&self, &self,
@@ -1051,7 +1135,7 @@ pub const EXCHANGE_TIMEOUT: Duration = Duration::from_secs(5);
/// ///
/// Bump it whenever [`Request`], [`Response`] or anything they contain /// Bump it whenever [`Request`], [`Response`] or anything they contain
/// changes shape. /// changes shape.
pub const CONTROL_PROTOCOL: u32 = u32::from_be_bytes([b'T', b'S', b'N', 21]); pub const CONTROL_PROTOCOL: u32 = u32::from_be_bytes([b'T', b'S', b'N', 23]);
/// Reads one request off an accepted stream, answers it, writes the response. /// Reads one request off an accepted stream, answers it, writes the response.
/// ///
@@ -1075,6 +1159,15 @@ where
} }
}; };
let response = match request { let response = match request {
Request::SaveVip {
network_id,
config,
expected,
delete,
} => match source.save_vip(network_id, config, expected, delete).await {
Ok(vips) => Response::Vips(vips),
Err(reason) => Response::Error(reason),
},
Request::ForgetNode { Request::ForgetNode {
network_id, network_id,
peer, peer,
crates/tsunagi-core/src/lib.rs
+1
View File
@@ -51,6 +51,7 @@ pub mod storage;
pub mod task; pub mod task;
#[cfg(feature = "testing")] #[cfg(feature = "testing")]
pub mod testing; pub mod testing;
pub mod vip;
pub use agent::{Agent, AgentStatus, Event, NetworkStatus, PeerStatus}; pub use agent::{Agent, AgentStatus, Event, NetworkStatus, PeerStatus};
pub use config::{AgentConfig, Limits, ReconnectPolicy, StoragePaths, TransportPolicy}; pub use config::{AgentConfig, Limits, ReconnectPolicy, StoragePaths, TransportPolicy};
crates/tsunagi-core/src/net.rs
+29 -8
View File
@@ -22,6 +22,7 @@
//! own reachability data. //! own reachability data.
use std::net::SocketAddr; use std::net::SocketAddr;
use std::sync::Arc;
use std::time::Duration; use std::time::Duration;
use iroh::endpoint::{ use iroh::endpoint::{
@@ -221,9 +222,18 @@ fn local_addr_string(addr: &iroh::endpoint::LocalTransportAddr) -> Option<String
#[derive(Debug, Clone)] #[derive(Debug, Clone)]
pub struct EndpointAdapter { pub struct EndpointAdapter {
endpoint: Endpoint, endpoint: Endpoint,
owned_interface: Arc<std::sync::RwLock<String>>,
} }
impl EndpointAdapter { impl EndpointAdapter {
/// Excludes the real name after interface creation (macOS assigns utunN).
pub(crate) fn exclude_interface(&self, name: &str) {
*self
.owned_interface
.write()
.unwrap_or_else(|p| p.into_inner()) = name.to_lowercase();
}
/// Binds an endpoint according to `config`, reusing the persistent device key. /// Binds an endpoint according to `config`, reusing the persistent device key.
pub async fn bind(config: &AgentConfig, identity: &DeviceIdentity) -> Result<Self> { pub async fn bind(config: &AgentConfig, identity: &DeviceIdentity) -> Result<Self> {
let mut builder = Endpoint::builder(presets::Minimal) let mut builder = Endpoint::builder(presets::Minimal)
@@ -240,11 +250,15 @@ impl EndpointAdapter {
TransportPolicy::N0Defaults => builder.preset(presets::N0), TransportPolicy::N0Defaults => builder.preset(presets::N0),
}; };
if matches!( let owned_interface =
config.transport, Arc::new(std::sync::RwLock::new(config.interface_name.to_lowercase()));
TransportPolicy::N0Defaults | TransportPolicy::DirectOnly {
) { let interface_filter = Arc::clone(&owned_interface);
let filter = iroh::address_lookup::AddrFilter::new(|addrs| { let filter = iroh::address_lookup::AddrFilter::new(move |addrs| {
let owned_interface = interface_filter
.read()
.unwrap_or_else(|p| p.into_inner())
.clone();
let mut overlay_ips = std::collections::HashSet::new(); let mut overlay_ips = std::collections::HashSet::new();
for iface in netdev::get_interfaces() { for iface in netdev::get_interfaces() {
let name = iface.name.to_lowercase(); let name = iface.name.to_lowercase();
@@ -258,7 +272,9 @@ impl EndpointAdapter {
.as_deref() .as_deref()
.map(str::to_lowercase) .map(str::to_lowercase)
.unwrap_or_default(); .unwrap_or_default();
if name.starts_with("tsun") if name == owned_interface
|| friendly == owned_interface
|| name.starts_with("tsun")
|| name.contains("tsunagi") || name.contains("tsunagi")
|| name.contains("wintun") || name.contains("wintun")
|| friendly.starts_with("tsun") || friendly.starts_with("tsun")
@@ -288,7 +304,9 @@ impl EndpointAdapter {
.collect(); .collect();
std::borrow::Cow::Owned(filtered) std::borrow::Cow::Owned(filtered)
}); });
builder = builder.addr_filter(filter); builder = builder
.direct_addr_filter(filter.clone())
.addr_filter(filter);
} }
#[cfg(feature = "testing")] #[cfg(feature = "testing")]
@@ -395,7 +413,10 @@ impl EndpointAdapter {
} }
}); });
Ok(Self { endpoint }) Ok(Self {
endpoint,
owned_interface,
})
} }
/// The underlying iroh endpoint. /// The underlying iroh endpoint.
crates/tsunagi-core/src/overlay/hostrules/exit.rs
+5
View File
@@ -84,6 +84,11 @@ pub trait ExitHostRules: Send + Sync + std::fmt::Debug + 'static {
/// startup to take away what a crashed run left. /// startup to take away what a crashed run left.
fn clear<'a>(&'a self, interface: &'a str) -> BoxFuture<'a, ()>; fn clear<'a>(&'a self, interface: &'a str) -> BoxFuture<'a, ()>;
/// Refreshes volatile local socket protection without replacing exit routes.
fn refresh<'a>(&'a self, _plan: &'a ExitHostPlan) -> BoxFuture<'a, Option<RuleOutcome>> {
Box::pin(async { None })
}
/// Whether using an exit node on this host needs the underlay addresses /// Whether using an exit node on this host needs the underlay addresses
/// the agent talks to, because the host cannot exempt the agent's own /// the agent talks to, because the host cannot exempt the agent's own
/// traffic from the default route by user id (Windows). The default is no. /// traffic from the default route by user id (Windows). The default is no.
crates/tsunagi-core/src/overlay/hostrules/linux.rs
+26 -3
View File
@@ -33,12 +33,13 @@ const CHAIN: &str = "INPUT";
/// Fixed places `iptables` is installed. Never `PATH`: the agent is /// Fixed places `iptables` is installed. Never `PATH`: the agent is
/// privileged and a directory a user controls must not decide what it runs. /// privileged and a directory a user controls must not decide what it runs.
const IPTABLES_DIRECTORIES: [&str; 5] = [ const IPTABLES_DIRECTORIES: [&str; 6] = [
"/usr/sbin/iptables", "/usr/sbin/iptables",
"/sbin/iptables", "/sbin/iptables",
"/usr/bin/iptables", "/usr/bin/iptables",
"/bin/iptables", "/bin/iptables",
"/usr/local/sbin/iptables", "/usr/local/sbin/iptables",
"/run/current-system/sw/bin/iptables",
]; ];
/// The first candidate that exists. /// The first candidate that exists.
@@ -328,7 +329,23 @@ fn clear_firewall(run: Run<'_>, interface: &str) {
/// Runs the real `iptables`. /// Runs the real `iptables`.
pub(super) fn run_iptables(args: &[String]) -> Ran { pub(super) fn run_iptables(args: &[String]) -> Ran {
let Some(program) = pick_iptables(|path| std::path::Path::new(path).is_file()) else { run_firewall(false, args)
}
pub(super) fn run_firewall(ipv6: bool, args: &[String]) -> Ran {
let v6_paths: Vec<_> = IPTABLES_DIRECTORIES
.iter()
.map(|p| p.replace("iptables", "ip6tables"))
.collect();
let v6_program = v6_paths
.iter()
.find(|path| std::path::Path::new(path).is_file());
let program = if ipv6 {
v6_program.map(String::as_str)
} else {
pick_iptables(|path| std::path::Path::new(path).is_file())
};
let Some(program) = program else {
return Ran::Missing; return Ran::Missing;
}; };
match std::process::Command::new(program).args(args).output() { match std::process::Command::new(program).args(args).output() {
@@ -350,11 +367,17 @@ pub(super) fn run_iptables(args: &[String]) -> Ran {
#[derive(Debug)] #[derive(Debug)]
pub struct LinuxHostRules { pub struct LinuxHostRules {
pub(super) route: RouteHandle, pub(super) route: RouteHandle,
/// Interface, sorted socket snapshot and whether exit routes committed.
pub(super) underlay_sockets:
std::sync::Mutex<Option<(String, Vec<super::linux_exit_mark::Listener>, bool)>>,
} }
impl LinuxHostRules { impl LinuxHostRules {
pub(crate) fn new(route: RouteHandle) -> Self { pub(crate) fn new(route: RouteHandle) -> Self {
Self { route } Self {
route,
underlay_sockets: std::sync::Mutex::new(None),
}
} }
async fn set_route(&self, interface: String, source: Ipv4Addr) -> RuleOutcome { async fn set_route(&self, interface: String, source: Ipv4Addr) -> RuleOutcome {
crates/tsunagi-core/src/overlay/hostrules/linux_exit.rs
+170 -15
View File
@@ -25,18 +25,19 @@
//! //!
//! ```text //! ```text
//! table 0x7473: default dev <if> //! table 0x7473: default dev <if>
//! 5280: from all uidrange <euid>-<euid> lookup main //! 5205: from all fwmark 0x40000000/0x40000000 lookup main
//! 5206: from all uidrange <euid>-<euid> lookup main
//! 5290: from all lookup main suppress_prefixlength 0 //! 5290: from all lookup main suppress_prefixlength 0
//! 5300: from all lookup 0x7473 //! 5300: from all lookup 0x7473
//! ``` //! ```
//! //!
//! The first keeps the agent's *own* sockets off the tunnel: its QUIC, relay, //! The first two keep the agent's *own* sockets and marked replies off the tunnel: its QUIC, relay,
//! DHT and DNS traffic would otherwise be routed into the interface it is //! DHT and DNS traffic would otherwise be routed into the interface it is
//! carrying, a loop. The second lets every specific route — the LAN, another //! carrying, a loop. The specific-route rule lets the LAN, another
//! VPN, a tunnel to somewhere else — keep working, because only the default //! VPN, a tunnel to somewhere else — keep working, because only the default
//! route is suppressed. The priorities sit after Tailscale's 5270. //! route is suppressed. Mark/UID bypass precedes Tailscale; application rules follow it.
//! //!
//! The first rule exempts a *user id*, so it is only right when the agent has //! The UID rule exempts a *user id*, so it is only right when the agent has
//! one of its own, as the packaged service does. An agent started from a login //! one of its own, as the packaged service does. An agent started from a login
//! session would take that user's own applications out of the tunnel with it, //! session would take that user's own applications out of the tunnel with it,
//! and the exit node would silently do nothing for them. That is refused, not //! and the exit node would silently do nothing for them. That is refused, not
@@ -48,9 +49,10 @@ use crate::state::Ipv4Range;
use super::RuleOutcome; use super::RuleOutcome;
use super::exit::{ExitHostPlan, ExitHostReport, ExitHostRules, exit_tag, exit_tag_prefix}; use super::exit::{ExitHostPlan, ExitHostReport, ExitHostRules, exit_tag, exit_tag_prefix};
use super::linux::{ use super::linux::{
LinuxHostRules, Ran, Run, comment_supported, failure, locked, run_iptables, same_rule, LinuxHostRules, Ran, Run, comment_supported, failure, locked, run_firewall, run_iptables,
split_rule_line, same_rule, split_rule_line,
}; };
use super::linux_exit_mark;
/// The chains the offering rules live in: `(table, chain)`. /// The chains the offering rules live in: `(table, chain)`.
const CHAINS: [(&str, &str); 2] = [("nat", "POSTROUTING"), ("filter", "FORWARD")]; const CHAINS: [(&str, &str); 2] = [("nat", "POSTROUTING"), ("filter", "FORWARD")];
@@ -377,6 +379,32 @@ fn read_sysctl(path: &str) -> Option<String> {
std::fs::read_to_string(path).ok() std::fs::read_to_string(path).ok()
} }
fn check_kernel_rpf(overlay: &str) -> Result<(), String> {
let directory = "/proc/sys/net/ipv4/conf";
let value = |name: &str, setting: &str| -> Result<u32, String> {
read_sysctl(&format!("{directory}/{name}/{setting}"))
.and_then(|s| s.trim().parse().ok())
.ok_or_else(|| format!("cannot check {name}/{setting} before installing exit routes"))
};
let mut interfaces = Vec::new();
for entry in std::fs::read_dir(directory).map_err(|err| err.to_string())? {
let entry = entry.map_err(|err| err.to_string())?;
let name = entry.file_name().to_string_lossy().into_owned();
if name != "all" && name != "default" && name != "lo" && name != overlay {
interfaces.push((
name.clone(),
value(&name, "rp_filter")?,
value(&name, "src_valid_mark")?,
));
}
}
linux_exit_mark::check_kernel_rpf(
value("all", "rp_filter")?,
value("all", "src_valid_mark")?,
&interfaces,
)
}
/// The effective user id from the text of `/proc/self/status`. /// The effective user id from the text of `/proc/self/status`.
fn parse_euid(status: &str) -> Option<u32> { fn parse_euid(status: &str) -> Option<u32> {
status status
@@ -417,7 +445,7 @@ impl LinuxHostRules {
let xdg = std::env::var("XDG_RUNTIME_DIR").ok(); let xdg = std::env::var("XDG_RUNTIME_DIR").ok();
if let Some(reason) = client_refusal(xdg.as_deref()) { if let Some(reason) = client_refusal(xdg.as_deref()) {
// Whatever an earlier run left must not stay in force. // Whatever an earlier run left must not stay in force.
self.clear_client().await; let _ = self.clear_client(interface).await;
return RuleOutcome::Failed(reason); return RuleOutcome::Failed(reason);
} }
let Some(euid) = std::fs::read_to_string("/proc/self/status") let Some(euid) = std::fs::read_to_string("/proc/self/status")
@@ -426,24 +454,147 @@ impl LinuxHostRules {
else { else {
return RuleOutcome::Failed("cannot tell which user this agent runs as".to_string()); return RuleOutcome::Failed("cannot tell which user this agent runs as".to_string());
}; };
let installed = self
.underlay_sockets
.lock()
.unwrap_or_else(|p| p.into_inner())
.as_ref()
.is_some_and(|(name, _, routed)| name == interface && *routed);
if installed {
// A failed socket refresh must never remove an active exit route.
return self
.refresh_sockets(interface)
.await
.unwrap_or(RuleOutcome::Applied);
}
if let Err(reason) = self.clear_client(interface).await {
return RuleOutcome::Failed(reason);
}
let name = interface.to_string();
let prepared = Self::blocking(move || {
check_kernel_rpf(&name)?;
linux_exit_mark::install(
&run_firewall,
&name,
euid,
std::path::Path::new("/proc/sys/net/ipv6").is_dir(),
)
})
.await;
match prepared {
Ok(Ok(sockets)) => {
*self
.underlay_sockets
.lock()
.unwrap_or_else(|p| p.into_inner()) = Some((interface.into(), sockets, false));
}
Ok(Err(reason)) | Err(reason) => return RuleOutcome::Failed(reason),
}
let route = self.route.clone(); let route = self.route.clone();
let interface = interface.to_string(); let name = interface.to_string();
match Self::blocking(move || route.set_exit_client(&interface, euid)).await { let outcome = match Self::blocking(move || route.set_exit_client(&name, euid)).await {
Ok(Ok(())) => RuleOutcome::Applied, Ok(Ok(())) => RuleOutcome::Applied,
Ok(Err(err)) => RuleOutcome::Failed(err.to_string()), Ok(Err(err)) => RuleOutcome::Failed(err.to_string()),
Err(reason) => RuleOutcome::Failed(reason), Err(reason) => RuleOutcome::Failed(reason),
};
if let RuleOutcome::Failed(reason) = outcome {
let reason = match self.clear_client(interface).await {
Ok(()) => reason,
Err(cleanup) => format!("{reason}; cannot roll back exit rules: {cleanup}"),
};
RuleOutcome::Failed(reason)
} else {
if let Some((_, _, routed)) = self
.underlay_sockets
.lock()
.unwrap_or_else(|p| p.into_inner())
.as_mut()
{
*routed = true;
}
outcome
} }
} }
async fn clear_client(&self) { async fn clear_client(&self, interface: &str) -> Result<(), String> {
let route = self.route.clone(); let route = self.route.clone();
if let Ok(Err(err)) = Self::blocking(move || route.clear_exit_client()).await { Self::blocking(move || route.clear_exit_client())
tracing::warn!(%err, "cannot remove the exit node routing rules"); .await?
.map_err(|err| err.to_string())?;
// Route removal must precede removing reply protection.
let name = interface.to_string();
Self::blocking(move || {
linux_exit_mark::clear(
&run_firewall,
&name,
std::path::Path::new("/proc/sys/net/ipv6").is_dir(),
)
})
.await??;
*self
.underlay_sockets
.lock()
.unwrap_or_else(|p| p.into_inner()) = None;
Ok(())
}
async fn refresh_sockets(&self, interface: &str) -> Option<RuleOutcome> {
let previous = self
.underlay_sockets
.lock()
.unwrap_or_else(|p| p.into_inner())
.clone();
if previous
.as_ref()
.is_none_or(|(name, _, routed)| name != interface || !routed)
{
return None;
}
let name = interface.to_string();
let work_name = name.clone();
let result = Self::blocking(move || {
let uid = std::fs::read_to_string("/proc/self/status")
.ok()
.and_then(|s| parse_euid(&s))
.ok_or("cannot read agent UID")?;
check_kernel_rpf(&work_name)?;
let v6 = std::path::Path::new("/proc/sys/net/ipv6").is_dir();
let sockets = linux_exit_mark::local_sockets(uid, v6)?;
if previous
.as_ref()
.is_some_and(|(name, old, _)| name == &work_name && old == &sockets)
{
return Ok(None);
}
linux_exit_mark::refresh(&run_firewall, &work_name, v6, &sockets)?;
Ok(Some(sockets))
})
.await;
match result {
Ok(Ok(None)) => None,
Ok(Ok(Some(sockets))) => {
*self
.underlay_sockets
.lock()
.unwrap_or_else(|p| p.into_inner()) = Some((name, sockets, true));
Some(RuleOutcome::Applied)
}
Ok(Err(reason)) | Err(reason) => Some(RuleOutcome::Failed(reason)),
} }
} }
} }
impl ExitHostRules for LinuxHostRules { impl ExitHostRules for LinuxHostRules {
fn refresh<'a>(&'a self, plan: &'a ExitHostPlan) -> BoxFuture<'a, Option<RuleOutcome>> {
Box::pin(async move {
if plan.client {
self.refresh_sockets(&plan.interface).await
} else {
None
}
})
}
fn apply<'a>(&'a self, plan: &'a ExitHostPlan) -> BoxFuture<'a, ExitHostReport> { fn apply<'a>(&'a self, plan: &'a ExitHostPlan) -> BoxFuture<'a, ExitHostReport> {
Box::pin(async move { Box::pin(async move {
let interface = plan.interface.clone(); let interface = plan.interface.clone();
@@ -467,7 +618,9 @@ impl ExitHostRules for LinuxHostRules {
let client = if plan.client { let client = if plan.client {
Some(self.apply_client(&interface).await) Some(self.apply_client(&interface).await)
} else { } else {
self.clear_client().await; if let Err(reason) = self.clear_client(&interface).await {
tracing::warn!(%reason, "cannot remove exit-client rules");
}
None None
}; };
ExitHostReport { ExitHostReport {
@@ -482,7 +635,9 @@ impl ExitHostRules for LinuxHostRules {
Box::pin(async move { Box::pin(async move {
let name = interface.to_string(); let name = interface.to_string();
let _ = Self::blocking(move || clear_offer(&run_iptables, &name)).await; let _ = Self::blocking(move || clear_offer(&run_iptables, &name)).await;
self.clear_client().await; if let Err(reason) = self.clear_client(interface).await {
tracing::warn!(%reason, "cannot remove exit-client rules");
}
}) })
} }
} }
crates/tsunagi-core/src/overlay/hostrules/linux_exit_kernel.rs
+439
View File
@@ -0,0 +1,439 @@
//! Privileged routing regression, isolated from the host in named namespaces.
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use std::io::{BufRead, BufReader, Read, Write};
use std::net::{SocketAddr, TcpListener, TcpStream, UdpSocket};
use std::os::unix::process::CommandExt;
use std::process::{Child, Command};
use std::time::Duration;
use super::linux::run_firewall;
use super::linux_exit_mark::{clear, install, local_sockets, refresh};
use crate::overlay::NetlinkProvisioner;
const TEST: &str = "overlay::hostrules::linux_exit_kernel::strict_firewall_preserves_underlay_and_application_exit_routing";
const ROLE: &str = "TSUNAGI_EXIT_KERNEL_TEST_ROLE";
fn ip(arguments: &[&str]) -> String {
let result = Command::new("ip").args(arguments).output().unwrap();
assert!(
result.status.success(),
"ip {arguments:?}: {}",
String::from_utf8_lossy(&result.stderr)
);
String::from_utf8_lossy(&result.stdout).into_owned()
}
struct Namespace(String);
impl Drop for Namespace {
fn drop(&mut self) {
let _ = Command::new("ip").args(["netns", "del", &self.0]).status();
}
}
struct Echo(Child);
impl Drop for Echo {
fn drop(&mut self) {
let _ = self.0.kill();
let _ = self.0.wait();
}
}
fn probes() {
let should_pass = std::env::var("TSUNAGI_EXIT_KERNEL_EXPECT").unwrap() == "pass";
for address in ["198.51.100.2:42000", "[2001:db8:2::2]:42000"] {
let address: SocketAddr = address.parse().unwrap();
let socket = UdpSocket::bind(if address.is_ipv4() {
"0.0.0.0:0"
} else {
"[::]:0"
})
.unwrap();
socket
.set_read_timeout(Some(Duration::from_secs(1)))
.unwrap();
socket.send_to(b"rpf", address).unwrap();
let mut packet = [0u8; 3];
assert_eq!(
socket.recv(&mut packet).is_ok(),
should_pass,
"UDP {address}"
);
if should_pass {
assert_eq!(&packet, b"rpf");
}
let connected = TcpStream::connect_timeout(&address, Duration::from_secs(1));
assert_eq!(connected.is_ok(), should_pass, "TCP {address}");
if let Ok(mut stream) = connected {
stream
.set_read_timeout(Some(Duration::from_secs(1)))
.unwrap();
stream.write_all(b"rpf").unwrap();
stream.read_exact(&mut packet).unwrap();
assert_eq!(&packet, b"rpf");
}
}
}
fn run_probes(expect: &str) {
let result = Command::new(std::env::current_exe().unwrap())
.args(["--ignored", "--exact", TEST, "--nocapture"])
.env(ROLE, "probe")
.env("TSUNAGI_EXIT_KERNEL_EXPECT", expect)
.uid(990)
.gid(990)
.output()
.unwrap();
assert!(
result.status.success(),
"{}\n{}",
String::from_utf8_lossy(&result.stdout),
String::from_utf8_lossy(&result.stderr)
);
}
fn passive_listener() {
let listeners: Vec<_> = ["192.0.2.1:42010", "[2001:db8:1::1]:42010"]
.into_iter()
.map(|address| TcpListener::bind(address).unwrap())
.collect();
println!("PASSIVE_READY");
let tasks: Vec<_> = listeners
.into_iter()
.map(|listener| {
std::thread::spawn(move || {
let (mut socket, _) = listener.accept().unwrap();
socket
.set_read_timeout(Some(Duration::from_secs(2)))
.unwrap();
let mut bytes = [0; 3];
socket.read_exact(&mut bytes).unwrap();
socket.write_all(&bytes).unwrap();
})
})
.collect();
for task in tasks {
task.join().unwrap();
}
}
fn udp_listener() {
let sockets: Vec<_> = ["192.0.2.1:42011", "[2001:db8:1::1]:42011"]
.into_iter()
.map(|a| UdpSocket::bind(a).unwrap())
.collect();
println!("PASSIVE_READY");
let tasks: Vec<_> = sockets
.into_iter()
.map(|socket| {
std::thread::spawn(move || {
socket
.set_read_timeout(Some(Duration::from_secs(5)))
.unwrap();
let mut bytes = [0; 3];
let (len, peer) = socket.recv_from(&mut bytes).unwrap();
socket.send_to(&bytes[..len], peer).unwrap();
})
})
.collect();
for task in tasks {
task.join().unwrap();
}
}
fn spawn_listener(role: &str) -> Echo {
let process = Command::new(std::env::current_exe().unwrap())
.args(["--ignored", "--exact", TEST, "--nocapture"])
.env(ROLE, role)
.uid(990)
.gid(990)
.stdout(std::process::Stdio::piped())
.spawn()
.unwrap();
let mut child = Echo(process);
let mut output = BufReader::new(child.0.stdout.take().unwrap());
loop {
let mut line = String::new();
assert_ne!(
output.read_line(&mut line).unwrap(),
0,
"listener exited before readiness"
);
if line.contains("PASSIVE_READY") {
// Keep the reader open for libtest's final status after the socket
// worker finishes; closing it here makes success a broken pipe.
child.0.stdout = Some(output.into_inner());
return child;
}
}
}
fn client() {
for setting in ["rp_filter", "src_valid_mark"] {
std::fs::write(format!("/proc/sys/net/ipv4/conf/all/{setting}"), b"1").unwrap();
}
std::fs::write("/proc/sys/net/ipv4/conf/rpf-c/rp_filter", b"1").unwrap();
for family in [false, true] {
let args: Vec<_> = [
"-t",
"mangle",
"-A",
"PREROUTING",
"-m",
"rpfilter",
"--invert",
"--validmark",
"-j",
"DROP",
]
.into_iter()
.map(str::to_string)
.collect();
let result = run_firewall(family, &args);
assert!(
matches!(result, super::linux::Ran::Exited { success: true, .. }),
"{result:?}"
);
}
let provisioner = NetlinkProvisioner::new().unwrap();
let route = provisioner.host_rules().route;
// Model Tailscale's earlier lookup, without running another VPN daemon.
ip(&["-4", "rule", "add", "priority", "5270", "table", "52"]);
ip(&[
"-4", "route", "add", "default", "dev", "tsun0", "table", "52",
]);
ip(&["-6", "rule", "add", "priority", "5270", "table", "52"]);
ip(&[
"-6",
"route",
"add",
"unreachable",
"default",
"table",
"52",
]);
// A foreign VPN must leave neighbour discovery and connected LAN paths
// usable; otherwise the fixture itself cannot resolve its uplink neighbour.
for prefix in ["fe80::/64", "2001:db8:1::/64"] {
ip(&["-6", "route", "add", "throw", prefix, "table", "52"]);
}
let mut passive = spawn_listener("listener");
route.set_exit_client("tsun0", 990).unwrap();
// uid bypass alone cannot protect reverse lookups, reproducing the bug.
run_probes("blocked");
install(&run_firewall, "tsun0", 990, true).unwrap();
run_probes("pass");
// Pair transports may bind fresh UDP ports after exit activation.
let mut dynamic_udp = spawn_listener("udp-listener");
refresh(
&run_firewall,
"tsun0",
true,
&local_sockets(990, true).unwrap(),
)
.unwrap();
let server = std::env::var("TSUNAGI_EXIT_KERNEL_SERVER").unwrap();
let incoming = Command::new("ip").args(["netns", "exec", &server, "python3", "-c", r#"import socket
for family,source,target in [(socket.AF_INET,'198.51.100.2','192.0.2.1'),(socket.AF_INET6,'2001:db8:2::2','2001:db8:1::1')]:
for kind,port in [(socket.SOCK_STREAM,42010),(socket.SOCK_DGRAM,42011)]:
s=socket.socket(family,kind); s.settimeout(2); s.bind((source,0)); s.connect((target,port)); s.sendall(b'rpf'); assert s.recv(3)==b'rpf'; s.close()
"#]).output().unwrap();
assert!(
incoming.status.success(),
"first inbound TCP/UDP must be protected: {}",
String::from_utf8_lossy(&incoming.stderr)
);
assert!(passive.0.wait().unwrap().success());
assert!(dynamic_udp.0.wait().unwrap().success());
// Repeated apply remains usable, and does not redirect application sockets.
install(&run_firewall, "tsun0", 990, true).unwrap();
run_probes("pass");
assert!(ip(&["-4", "route", "get", "198.51.100.2", "uid", "991"]).contains("dev tsun0"));
let blocked = Command::new("ip")
.args(["-6", "route", "get", "2001:db8:2::2", "uid", "991"])
.output()
.unwrap();
assert!(
!blocked.status.success(),
"application IPv6 must stay blocked"
);
route.clear_exit_client().unwrap();
clear(&run_firewall, "tsun0", true).unwrap();
for family in ["-4", "-6"] {
assert!(
ip(&[family, "rule", "show"]).contains("5270:"),
"foreign routing rules survive cleanup"
);
ip(&[family, "rule", "del", "priority", "5270"]);
}
assert!(ip(&["-4", "route", "get", "198.51.100.2", "uid", "991"]).contains("dev rpf-c"));
assert!(ip(&["-6", "route", "get", "2001:db8:2::2", "uid", "991"]).contains("dev rpf-c"));
for family in [false, true] {
let result = run_firewall(family, &["-t".into(), "mangle".into(), "-S".into()]);
if let super::linux::Ran::Exited { stdout, .. } = result {
assert!(!stdout.contains("tsunagi-exit-underlay:"));
assert!(!stdout.contains("tsunagi-exit-sockets:"));
assert!(
stdout.contains("rpfilter"),
"strict host filtering is retained"
);
}
}
}
#[test]
#[ignore = "needs root, iproute2, iptables/ip6tables and Python; changes only temporary namespaces"]
fn strict_firewall_preserves_underlay_and_application_exit_routing() {
match std::env::var(ROLE).as_deref() {
Ok("probe") => {
probes();
return;
}
Ok("client") => {
client();
return;
}
Ok("listener") => {
passive_listener();
return;
}
Ok("udp-listener") => {
udp_listener();
return;
}
_ => {}
}
let suffix = std::process::id();
let client_name = format!("tsng-rpf-c-{suffix}");
let server_name = format!("tsng-rpf-s-{suffix}");
ip(&["netns", "add", &client_name]);
let _client = Namespace(client_name.clone());
ip(&["netns", "add", &server_name]);
let _server = Namespace(server_name.clone());
ip(&[
"link",
"add",
"rpf-c",
"netns",
&client_name,
"type",
"veth",
"peer",
"name",
"rpf-s",
"netns",
&server_name,
]);
for (namespace, interface, v4, v6) in [
(&client_name, "rpf-c", "192.0.2.1/24", "2001:db8:1::1/64"),
(&server_name, "rpf-s", "192.0.2.2/24", "2001:db8:1::2/64"),
] {
ip(&["-n", namespace, "link", "set", "lo", "up"]);
ip(&["-n", namespace, "addr", "add", v4, "dev", interface]);
ip(&[
"-n", namespace, "-6", "addr", "add", v6, "dev", interface, "nodad",
]);
ip(&["-n", namespace, "link", "set", interface, "up"]);
}
ip(&[
"-n",
&server_name,
"addr",
"add",
"198.51.100.2/32",
"dev",
"lo",
]);
ip(&[
"-n",
&server_name,
"-6",
"addr",
"add",
"2001:db8:2::2/128",
"dev",
"lo",
"nodad",
]);
ip(&[
"-n",
&client_name,
"route",
"add",
"default",
"via",
"192.0.2.2",
]);
ip(&[
"-n",
&client_name,
"-6",
"route",
"add",
"default",
"via",
"2001:db8:1::2",
]);
ip(&["-n", &client_name, "link", "add", "tsun0", "type", "dummy"]);
ip(&[
"-n",
&client_name,
"addr",
"add",
"10.13.37.1/24",
"dev",
"tsun0",
]);
ip(&["-n", &client_name, "link", "set", "tsun0", "up"]);
let echo_code = r#"import socket,selectors,sys
selector=selectors.DefaultSelector()
for family in [socket.AF_INET,socket.AF_INET6]:
for kind in [socket.SOCK_DGRAM,socket.SOCK_STREAM]:
sock=socket.socket(family,kind)
if family==socket.AF_INET6: sock.setsockopt(socket.IPPROTO_IPV6,socket.IPV6_V6ONLY,1)
sock.bind(('198.51.100.2' if family==socket.AF_INET else '2001:db8:2::2',42000))
if kind==socket.SOCK_STREAM: sock.listen()
selector.register(sock,selectors.EVENT_READ,kind)
print('ready',flush=True)
while True:
for key,_ in selector.select():
if key.data==socket.SOCK_DGRAM:
data,address=key.fileobj.recvfrom(128); key.fileobj.sendto(data,address)
else:
conn,_=key.fileobj.accept(); conn.settimeout(2)
try: conn.sendall(conn.recv(128))
finally: conn.close()
"#;
let child = Command::new("ip")
.args([
"netns",
"exec",
&server_name,
"python3",
"-u",
"-c",
echo_code,
])
.stdout(std::process::Stdio::piped())
.spawn()
.unwrap();
let mut echo = Echo(child);
let mut ready = String::new();
BufReader::new(echo.0.stdout.take().unwrap())
.read_line(&mut ready)
.unwrap();
assert_eq!(ready.trim(), "ready");
let result = Command::new("ip")
.args(["netns", "exec", &client_name])
.arg(std::env::current_exe().unwrap())
.args(["--ignored", "--exact", TEST, "--nocapture"])
.env(ROLE, "client")
.env("TSUNAGI_EXIT_KERNEL_SERVER", &server_name)
.output()
.unwrap();
assert!(
result.status.success(),
"{}\n{}",
String::from_utf8_lossy(&result.stdout),
String::from_utf8_lossy(&result.stderr)
);
}
crates/tsunagi-core/src/overlay/hostrules/linux_exit_mark.rs
+535
View File
@@ -0,0 +1,535 @@
//! Own-bit connection marks for exit-client underlay traffic.
//! Reply marks are restored before host reverse-path filtering, never accepted
//! around it. No peer address becomes a host route or firewall argument.
use super::linux::{Ran, failure, locked, split_rule_line};
use std::net::{IpAddr, Ipv4Addr, Ipv6Addr};
/// Outside Tailscale's 0xff0000 mask and ordinary low-bit WireGuard marks.
pub(crate) const UNDERLAY_MARK: u32 = 0x4000_0000;
pub(crate) const UNDERLAY_MASK: u32 = UNDERLAY_MARK;
const MARK: &str = "0x40000000/0x40000000";
const MASK: &str = "0x40000000";
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord)]
pub(super) struct Listener {
address: IpAddr,
port: u16,
tcp: bool,
}
fn parse_listeners(listing: &str, uid: u32, v6: bool, tcp: bool) -> Result<Vec<Listener>, String> {
let mut result = Vec::new();
for line in listing.lines().skip(1) {
let fields: Vec<_> = line.split_whitespace().collect();
if (tcp && fields.get(3) != Some(&"0A"))
|| fields.get(7).and_then(|s| s.parse::<u32>().ok()) != Some(uid)
{
continue;
}
let (address, port) = fields
.get(1)
.and_then(|s| s.split_once(':'))
.ok_or("malformed local socket")?;
let port = u16::from_str_radix(port, 16).map_err(|e| e.to_string())?;
let address = if v6 {
let mut bytes = [0; 16];
for (word, chunk) in bytes.as_chunks_mut::<4>().0.iter_mut().enumerate() {
let hex = address
.get(word * 8..word * 8 + 8)
.ok_or("malformed local IPv6 listener")?;
chunk.copy_from_slice(
&u32::from_str_radix(hex, 16)
.map_err(|e| e.to_string())?
.to_ne_bytes(),
);
}
IpAddr::V6(Ipv6Addr::from(bytes))
} else {
IpAddr::V4(Ipv4Addr::from(
u32::from_str_radix(address, 16)
.map_err(|e| e.to_string())?
.to_ne_bytes(),
))
};
if result.len() == 128 {
return Err("too many local sockets for exit protection".into());
}
if port != 0 {
result.push(Listener { address, port, tcp });
}
}
Ok(result)
}
pub(super) fn local_sockets(uid: u32, v6: bool) -> Result<Vec<Listener>, String> {
let mut listeners = Vec::new();
for (protocol, tcp) in [("tcp", true), ("udp", false)] {
for family in [false, true].into_iter().filter(|f| !f || v6) {
let path = format!("/proc/net/{protocol}{}", if family { "6" } else { "" });
listeners.extend(parse_listeners(
&std::fs::read_to_string(path).map_err(|e| e.to_string())?,
uid,
family,
tcp,
)?);
}
}
listeners.sort();
listeners.dedup();
Ok(listeners)
}
fn listener_rules(interface: &str, listener: &Listener) -> Vec<Vec<String>> {
let mut base = words(&[
"!",
"-i",
interface,
"-p",
if listener.tcp { "tcp" } else { "udp" },
"--dport",
&listener.port.to_string(),
]);
if !listener.address.is_unspecified() {
base.extend(words(&["-d", &listener.address.to_string()]));
}
// Match an actual local socket, not just a destination port. Listening
// sockets need protection on the first packet, before any outbound connmark.
base.extend(words(&[
"-m",
"socket",
"--nowildcard",
"-m",
"comment",
"--comment",
&listener_tag(interface),
]));
let save = base
.iter()
.cloned()
.chain(words(&[
"-m",
"mark",
"--mark",
MARK,
"-j",
"CONNMARK",
"--save-mark",
"--nfmask",
MASK,
"--ctmask",
MASK,
]))
.collect();
let mark = base
.into_iter()
.chain(words(&["-j", "MARK", "--set-xmark", MARK]))
.collect();
vec![save, mark]
}
pub(super) type RunFamily<'a> = &'a dyn Fn(bool, &[String]) -> Ran;
fn tag(interface: &str) -> String {
format!("tsunagi-exit-underlay:{interface}")
}
fn listener_tag(interface: &str) -> String {
format!("tsunagi-exit-sockets:{interface}")
}
fn words(values: &[&str]) -> Vec<String> {
values.iter().map(|s| (*s).into()).collect()
}
fn rules(interface: &str, uid: u32) -> Vec<(&'static str, Vec<String>)> {
let tag = tag(interface);
let uid = uid.to_string();
// Insert at position one in this order: final OUTPUT order is MARK, SAVE.
vec![
(
"PREROUTING",
words(&[
"!",
"-i",
interface,
"-m",
"connmark",
"--mark",
MARK,
"-m",
"comment",
"--comment",
&tag,
"-j",
"CONNMARK",
"--restore-mark",
"--nfmask",
MASK,
"--ctmask",
MASK,
]),
),
(
"OUTPUT",
words(&[
"-m",
"owner",
"--uid-owner",
&uid,
"-m",
"mark",
"--mark",
MARK,
"-m",
"comment",
"--comment",
&tag,
"-j",
"CONNMARK",
"--save-mark",
"--nfmask",
MASK,
"--ctmask",
MASK,
]),
),
(
"OUTPUT",
words(&[
"-m",
"owner",
"--uid-owner",
&uid,
"-m",
"comment",
"--comment",
&tag,
"-j",
"MARK",
"--set-xmark",
MARK,
]),
),
]
}
fn execute(run: RunFamily<'_>, v6: bool, args: Vec<String>) -> Result<String, String> {
match run(v6, &locked(args)) {
Ran::Exited {
success: true,
stdout,
..
} => Ok(stdout),
Ran::Exited { stderr, .. } => Err(failure(&stderr)),
Ran::Missing => Err(format!(
"{} is required for exit-node connection marks",
if v6 { "ip6tables" } else { "iptables" }
)),
}
}
/// Removes only exact interface tags, retaining other agents and host rules.
pub(super) fn clear(run: RunFamily<'_>, interface: &str, v6: bool) -> Result<(), String> {
clear_tags(run, interface, v6, false)
}
fn clear_tags(
run: RunFamily<'_>,
interface: &str,
v6: bool,
sockets_only: bool,
) -> Result<(), String> {
let wanted = tag(interface);
let sockets = listener_tag(interface);
for family in [false, true].into_iter().filter(|family| !family || v6) {
let chains: &[&str] = if sockets_only {
&["PREROUTING"]
} else {
&["OUTPUT", "PREROUTING"]
};
for chain in chains {
let listing = execute(run, family, words(&["-t", "mangle", "-S", chain]))?;
for line in listing.lines() {
let tokens = split_rule_line(line);
let [append, listed_chain, rest @ ..] = tokens.as_slice() else {
continue;
};
if append == "-A"
&& listed_chain == *chain
&& rest.windows(2).any(|pair| {
pair[0] == "--comment"
&& (pair[1] == sockets || (!sockets_only && pair[1] == wanted))
})
{
let args = words(&["-t", "mangle", "-D", chain])
.into_iter()
.chain(rest.iter().cloned())
.collect();
execute(run, family, args)?;
}
}
}
}
Ok(())
}
/// Establishes protection before routing may redirect application traffic.
pub(super) fn install(
run: RunFamily<'_>,
interface: &str,
uid: u32,
v6: bool,
) -> Result<Vec<Listener>, String> {
let listeners = local_sockets(uid, v6)?;
install_with_listeners(run, interface, uid, v6, &listeners)?;
Ok(listeners)
}
fn install_with_listeners(
run: RunFamily<'_>,
interface: &str,
uid: u32,
v6: bool,
listeners: &[Listener],
) -> Result<(), String> {
clear(run, interface, v6)?;
let result = (|| {
for family in [false, true].into_iter().filter(|family| !family || v6) {
for (chain, spec) in rules(interface, uid) {
let args = words(&["-t", "mangle", "-I", chain, "1"])
.into_iter()
.chain(spec)
.collect();
execute(run, family, args)?;
}
for listener in listeners.iter().filter(|s| s.address.is_ipv6() == family) {
for spec in listener_rules(interface, listener) {
execute(
run,
family,
words(&["-t", "mangle", "-I", "PREROUTING", "1"])
.into_iter()
.chain(spec)
.collect(),
)?;
}
}
}
Ok(())
})();
if result.is_err()
&& let Err(cleanup) = clear(run, interface, v6)
{
return Err(format!(
"{}; cannot roll back connection marks: {cleanup}",
result.err().unwrap_or_default()
));
}
result
}
/// Updates passive protection without interrupting established connmark rules.
pub(super) fn refresh(
run: RunFamily<'_>,
interface: &str,
v6: bool,
listeners: &[Listener],
) -> Result<(), String> {
clear_tags(run, interface, v6, true)?;
for family in [false, true].into_iter().filter(|f| !f || v6) {
for listener in listeners.iter().filter(|s| s.address.is_ipv6() == family) {
for spec in listener_rules(interface, listener) {
execute(
run,
family,
words(&["-t", "mangle", "-I", "PREROUTING", "1"])
.into_iter()
.chain(spec)
.collect(),
)?;
}
}
}
Ok(())
}
/// Kernel strict RPF must include the restored mark, independently of iptables.
pub(super) fn check_kernel_rpf(
all_rpf: u32,
all_valid: u32,
interfaces: &[(String, u32, u32)],
) -> Result<(), String> {
for (interface, rpf, valid) in interfaces {
if all_rpf.max(*rpf) == 1 && all_valid.max(*valid) == 0 {
return Err(format!(
"strict rp_filter on `{interface}` ignores connection marks; enable net.ipv4.conf.all.src_valid_mark=1 before using an exit node (rp_filter stays enabled)"
));
}
}
Ok(())
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
use std::cell::RefCell;
#[test]
fn both_families_restore_before_firewall_and_preserve_foreign_mark_bits() {
let calls = RefCell::new(Vec::new());
let run = |v6, args: &[String]| {
calls.borrow_mut().push((v6, args.to_vec()));
Ran::Exited {
success: true,
stdout: String::new(),
stderr: String::new(),
}
};
install_with_listeners(&run, "tsun0", 990, true, &[]).unwrap();
let calls = calls.borrow();
for family in [false, true] {
let inserts: Vec<_> = calls
.iter()
.filter(|(v6, a)| *v6 == family && a.iter().any(|s| s == "-I"))
.map(|(_, a)| a.join(" "))
.collect();
assert_eq!(inserts.len(), 3);
assert!(inserts[0].contains("-I PREROUTING 1 ! -i tsun0 -m connmark"));
assert!(inserts[0].contains("--restore-mark --nfmask 0x40000000 --ctmask 0x40000000"));
assert!(inserts[1].contains("--uid-owner 990"));
assert!(inserts[1].contains("--save-mark"));
assert!(inserts[2].contains("--set-xmark 0x40000000/0x40000000"));
assert!(
!inserts
.iter()
.any(|s| s.contains("-j ACCEPT") || s.contains("-j RETURN"))
);
}
assert_eq!(UNDERLAY_MASK & 0x00ff_0000, 0);
}
#[test]
fn cleanup_retains_other_interfaces_and_rolls_back_partial_ipv6_installation() {
let calls = RefCell::new(Vec::new());
let run = |v6, args: &[String]| {
calls.borrow_mut().push((v6, args.to_vec()));
let listing = "-A OUTPUT -m comment --comment tsunagi-exit-underlay:tsun0 -j MARK --set-xmark 0x40000000/0x40000000\n-A OUTPUT -m comment --comment tsunagi-exit-underlay:tsun1 -j MARK --set-xmark 0x40000000/0x40000000\n-A OUTPUT -m comment --comment unrelated -j RETURN\n";
Ran::Exited {
success: !(v6 && args.iter().any(|s| s == "-I")),
stdout: if args.iter().any(|s| s == "-S") {
listing.into()
} else {
String::new()
},
stderr: "injected IPv6 failure".into(),
}
};
assert!(install_with_listeners(&run, "tsun0", 990, true, &[]).is_err());
let calls = calls.borrow();
let deletions: Vec<_> = calls
.iter()
.filter(|(_, a)| a.iter().any(|s| s == "-D"))
.collect();
assert!(
deletions.len() >= 4,
"both initial and rollback cleanup run"
);
assert!(
deletions
.iter()
.all(|(_, a)| a.iter().any(|s| s == "tsunagi-exit-underlay:tsun0"))
);
}
#[test]
fn kernel_strict_validation_requires_mark_awareness_without_weakening_rpf() {
let interfaces = vec![("eth0".into(), 1, 0)];
assert!(check_kernel_rpf(0, 0, &interfaces).is_err());
assert!(check_kernel_rpf(0, 1, &interfaces).is_ok());
assert!(check_kernel_rpf(2, 0, &interfaces).is_ok());
assert!(check_kernel_rpf(0, 0, &[("eth0".into(), 1, 1)]).is_ok());
}
#[test]
fn passive_tcp_protection_uses_only_locally_owned_listeners() {
let listing = "header\n0: 00000000:01BB 00000000:0000 0A 0:0 0:0 0 990 0 1\n1: 0100007F:1111 00000000:0000 0A 0:0 0:0 0 1000 0 2\n2: 0100007F:2222 00000000:0000 01 0:0 0:0 0 990 0 3\n";
let listeners = parse_listeners(listing, 990, false, true).unwrap();
assert_eq!(
listeners,
vec![Listener {
address: IpAddr::V4(Ipv4Addr::UNSPECIFIED),
port: 443,
tcp: true
}]
);
let rules = listener_rules("tsun0", &listeners[0]);
assert!(
rules[0]
.join(" ")
.contains("-p tcp --dport 443 -m socket --nowildcard")
);
assert!(rules[0].join(" ").contains("--save-mark"));
assert!(rules[1].join(" ").contains("--set-xmark"));
}
#[test]
fn listener_parser_checks_family_encoding_and_bounds() {
let loopback = Ipv6Addr::LOCALHOST.octets();
let hex: String = loopback
.as_chunks::<4>()
.0
.iter()
.map(|bytes| format!("{:08X}", u32::from_ne_bytes(*bytes)))
.collect();
let listing = format!("header\n0: {hex}:01BB 0:0 0A 0:0 0:0 0 990 0 1\n");
assert_eq!(
parse_listeners(&listing, 990, true, true).unwrap(),
vec![Listener {
address: IpAddr::V6(Ipv6Addr::LOCALHOST),
port: 443,
tcp: true
}]
);
assert!(
parse_listeners(
"header\n0: BAD:01BB 0:0 0A 0:0 0:0 0 990 0 1",
990,
true,
true
)
.is_err()
);
let huge = format!(
"header\n{}",
"0: 00000000:01BB 0:0 0A 0:0 0:0 0 990 0 1\n".repeat(129)
);
assert!(parse_listeners(&huge, 990, false, true).is_err());
}
#[test]
fn udp_refresh_preserves_general_connmark_protection_and_other_interfaces() {
let listing = "header\n0: 00000000:ABA0 0:0 07 0:0 0:0 0 990 0 1\n1: 00000000:0000 0:0 07 0:0 0:0 0 990 0 2\n";
let sockets = parse_listeners(listing, 990, false, false).unwrap();
assert_eq!(sockets.len(), 1);
assert!(!sockets[0].tcp);
let calls = RefCell::new(Vec::new());
let run = |v6, args: &[String]| {
calls.borrow_mut().push((v6, args.join(" ")));
Ran::Exited { success: true, stdout: "-A PREROUTING -m comment --comment tsunagi-exit-underlay:tsun0 -j CONNMARK --restore-mark\n-A PREROUTING -m comment --comment tsunagi-exit-sockets:tsun0 -j MARK\n-A PREROUTING -m comment --comment tsunagi-exit-sockets:tsun1 -j MARK\n".into(), stderr: String::new() }
};
refresh(&run, "tsun0", false, &sockets).unwrap();
let calls = calls.borrow();
assert!(
calls
.iter()
.any(|(_, a)| a.contains("-I PREROUTING 1") && a.contains("-p udp"))
);
assert!(
calls
.iter()
.filter(|(_, a)| a.contains("-D"))
.all(|(_, a)| a.contains("tsunagi-exit-sockets:tsun0"))
);
}
}
crates/tsunagi-core/src/overlay/hostrules/mod.rs
+4
View File
@@ -55,6 +55,10 @@ pub use exit::{
mod linux; mod linux;
#[cfg(all(feature = "tun-device", target_os = "linux"))] #[cfg(all(feature = "tun-device", target_os = "linux"))]
mod linux_exit; mod linux_exit;
#[cfg(all(test, feature = "tun-device", target_os = "linux"))]
mod linux_exit_kernel;
#[cfg(all(feature = "tun-device", target_os = "linux"))]
pub(in crate::overlay) mod linux_exit_mark;
#[cfg(all(feature = "tun-device", target_os = "linux"))] #[cfg(all(feature = "tun-device", target_os = "linux"))]
pub use linux::LinuxHostRules; pub use linux::LinuxHostRules;
crates/tsunagi-core/src/overlay/interface.rs
+97 -1
View File
@@ -145,18 +145,19 @@ impl Tally {
} }
/// The overlay interface. /// The overlay interface.
#[derive(Debug)]
pub struct Interface { pub struct Interface {
device: Arc<dyn TunDevice>, device: Arc<dyn TunDevice>,
factory: Arc<dyn TunFactory>, factory: Arc<dyn TunFactory>,
name: String, name: String,
mtu: u32, mtu: u32,
routes: Arc<RoutingTable>, routes: Arc<RoutingTable>,
carrier: Arc<dyn PacketCarrier>,
tally: Arc<Tally>, tally: Arc<Tally>,
/// What was last applied to the host, so an unchanged table is not /// What was last applied to the host, so an unchanged table is not
/// re-applied on every pass. /// re-applied on every pass.
applied: std::sync::Mutex<Vec<Cidr>>, applied: std::sync::Mutex<Vec<Cidr>>,
task: std::sync::Mutex<Option<JoinHandle<()>>>, task: std::sync::Mutex<Option<JoinHandle<()>>>,
vip_task: std::sync::Mutex<Option<JoinHandle<()>>>,
sync_lock: Arc<tokio::sync::Mutex<()>>, sync_lock: Arc<tokio::sync::Mutex<()>>,
/// The host rules for this interface, when the factory manages the host. /// The host rules for this interface, when the factory manages the host.
host_rules: Option<Arc<dyn BroadcastHostRules>>, host_rules: Option<Arc<dyn BroadcastHostRules>>,
@@ -200,6 +201,14 @@ impl ExitState {
} }
} }
impl std::fmt::Debug for Interface {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("Interface")
.field("name", &self.name)
.finish_non_exhaustive()
}
}
impl Interface { impl Interface {
/// Creates the interface and starts moving packets. /// Creates the interface and starts moving packets.
pub async fn start( pub async fn start(
@@ -221,7 +230,29 @@ impl Interface {
let name = device.name().to_string(); let name = device.name().to_string();
let tally = Arc::new(Tally::default()); let tally = Arc::new(Tally::default());
let vip_task = {
let routes = Arc::clone(&routes);
let carrier = Arc::clone(&carrier);
tokio::spawn(async move {
let mut interval = tokio::time::interval(std::time::Duration::from_millis(250));
interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
let mut checks = tokio::task::JoinSet::new();
loop {
tokio::select! {
_=interval.tick()=>{
let (probes,jobs)=routes.vips.tick(routes.vips.check_capacity().saturating_sub(checks.len()));
for (route,packet) in probes {carrier.carry(route,packet);}
for job in jobs {checks.spawn(job.run());}
},
Some(result)=checks.join_next(),if !checks.is_empty()=>{
if let Ok((job,ok))=result {routes.vips.checked(job,ok);}
}
}
}
})
};
let task = { let task = {
let carrier = Arc::clone(&carrier);
let device = Arc::clone(&device); let device = Arc::clone(&device);
let routes = Arc::clone(&routes); let routes = Arc::clone(&routes);
let tally = Arc::clone(&tally); let tally = Arc::clone(&tally);
@@ -231,6 +262,26 @@ impl Interface {
tally.malformed.fetch_add(1, Ordering::Relaxed); tally.malformed.fetch_add(1, Ordering::Relaxed);
continue; continue;
}; };
match routes.vips.outgoing(packet.clone()) {
crate::vip::router::Decision::Send(route, packet) => {
for part in crate::vip::packet::fragments(packet, mtu) {
if carrier.carry(route, part) {
tally.sent.fetch_add(1, Ordering::Relaxed);
} else {
tally.undeliverable.fetch_add(1, Ordering::Relaxed);
}
}
continue;
}
crate::vip::router::Decision::Local(packet) => {
for part in crate::vip::packet::fragments(packet, mtu) {
let _ = device.send(part).await;
}
continue;
}
crate::vip::router::Decision::Drop => continue,
crate::vip::router::Decision::Pass => {}
}
let destination = header.destination(); let destination = header.destination();
if let (IpAddr::V4(source), IpAddr::V4(destination)) = if let (IpAddr::V4(source), IpAddr::V4(destination)) =
(header.source(), destination) (header.source(), destination)
@@ -279,9 +330,11 @@ impl Interface {
name, name,
mtu, mtu,
routes, routes,
carrier,
tally, tally,
applied: std::sync::Mutex::new(Vec::new()), applied: std::sync::Mutex::new(Vec::new()),
task: std::sync::Mutex::new(Some(task)), task: std::sync::Mutex::new(Some(task)),
vip_task: std::sync::Mutex::new(Some(vip_task)),
sync_lock: Arc::new(tokio::sync::Mutex::new(())), sync_lock: Arc::new(tokio::sync::Mutex::new(())),
host_rules, host_rules,
rules_report: std::sync::Mutex::new(None), rules_report: std::sync::Mutex::new(None),
@@ -397,6 +450,14 @@ impl Interface {
return; return;
} }
if planned.as_ref() == Some(&wanted) && ok { if planned.as_ref() == Some(&wanted) && ok {
if let Some(outcome) = exit.refresh(&wanted).await {
if let RuleOutcome::Failed(reason) = &outcome {
tracing::warn!(interface = %self.name, %reason, "cannot refresh exit-client socket protection");
}
if let Some(report) = self.exit_state().report.as_mut() {
report.client = Some(outcome);
}
}
return; return;
} }
let report = exit.apply(&wanted).await; let report = exit.apply(&wanted).await;
@@ -708,6 +769,15 @@ impl Interface {
task.abort(); task.abort();
let _ = task.await; let _ = task.await;
} }
let vip_task = self
.vip_task
.lock()
.unwrap_or_else(|e| e.into_inner())
.take();
if let Some(task) = vip_task {
task.abort();
let _ = task.await;
}
self.factory.destroy(self.device.name()).await; self.factory.destroy(self.device.name()).await;
} }
@@ -762,6 +832,24 @@ impl Interface {
return Err(Rejected::WrongDestination); return Err(Rejected::WrongDestination);
} }
}; };
// Source ownership was checked before any VIP reverse translation.
let packet = match self.routes.vips.incoming(network, peer, packet.clone()) {
crate::vip::router::Decision::Send(route, packet) => {
// Probe replies use the ordinary peer carrier, never the TUN.
self.carrier.carry(route, packet);
return Ok(());
}
crate::vip::router::Decision::Local(packet) => {
for part in crate::vip::packet::fragments(packet, self.mtu) {
if self.device.send(part).await.is_ok() {
self.tally.received.fetch_add(1, Ordering::Relaxed);
}
}
return Ok(());
}
crate::vip::router::Decision::Drop => return Ok(()),
crate::vip::router::Decision::Pass => packet,
};
// Remote broadcasts terminate here. Only local TUN ingress can fan out. // Remote broadcasts terminate here. Only local TUN ingress can fan out.
if self.device.send(packet).await.is_ok() { if self.device.send(packet).await.is_ok() {
if broadcast { if broadcast {
@@ -786,6 +874,14 @@ impl Interface {
impl Drop for Interface { impl Drop for Interface {
fn drop(&mut self) { fn drop(&mut self) {
if let Some(task) = self
.vip_task
.lock()
.unwrap_or_else(|e| e.into_inner())
.take()
{
task.abort();
}
if let Ok(mut guard) = self.task.lock() if let Ok(mut guard) = self.task.lock()
&& let Some(task) = guard.take() && let Some(task) = guard.take()
{ {
crates/tsunagi-core/src/overlay/provision/linux.rs
+64 -5
View File
@@ -55,6 +55,7 @@ use crate::BoxFuture;
use crate::overlay::OverlayError; use crate::overlay::OverlayError;
use super::super::config::Cidr; use super::super::config::Cidr;
use super::super::hostrules::linux_exit_mark::{UNDERLAY_MARK, UNDERLAY_MASK};
use super::super::tun::{TunDevice, TunRequest}; use super::super::tun::{TunDevice, TunRequest};
use super::privilege::{NetAdmin, Privilege, probe_net_admin}; use super::privilege::{NetAdmin, Privilege, probe_net_admin};
use super::{ use super::{
@@ -641,8 +642,10 @@ async fn del_broadcast_route(name: &str) -> Result<(), OverlayError> {
const EXIT_TABLE: u32 = 0x7473; const EXIT_TABLE: u32 = 0x7473;
/// The main table. /// The main table.
const MAIN_TABLE: u32 = 254; const MAIN_TABLE: u32 = 254;
/// Where the exit rules sit, after Tailscale's 5270. /// Protect marked underlay and initial socket routing before Tailscale's rules.
const EXIT_BYPASS_PRIORITY: u32 = 5280; const EXIT_MARK_PRIORITY: u32 = 5205;
const EXIT_BYPASS_PRIORITY: u32 = 5206;
const LEGACY_EXIT_BYPASS_PRIORITY: u32 = 5280;
const EXIT_SPECIFIC_PRIORITY: u32 = 5290; const EXIT_SPECIFIC_PRIORITY: u32 = 5290;
const EXIT_DEFAULT_PRIORITY: u32 = 5300; const EXIT_DEFAULT_PRIORITY: u32 = 5300;
@@ -666,14 +669,23 @@ fn rule_priority(rule: &RuleMessage) -> Option<u32> {
}) })
} }
/// Whether a rule is one of the three this agent adds for an exit node. /// Whether a rule is one this agent adds for an exit node, including old UID priority.
/// ///
/// Recognised by shape as well as priority: those numbers are not reserved, /// Recognised by shape as well as priority: those numbers are not reserved,
/// and a rule somebody else put at one of them must not be removed. /// and a rule somebody else put at one of them must not be removed.
fn is_exit_rule(rule: &RuleMessage) -> bool { fn is_exit_rule(rule: &RuleMessage) -> bool {
let table = rule_table(rule); let table = rule_table(rule);
match rule_priority(rule) { match rule_priority(rule) {
Some(EXIT_BYPASS_PRIORITY) => { Some(EXIT_MARK_PRIORITY) => {
table == MAIN_TABLE
&& rule
.attributes
.contains(&RuleAttribute::FwMark(UNDERLAY_MARK))
&& rule
.attributes
.contains(&RuleAttribute::FwMask(UNDERLAY_MASK))
}
Some(EXIT_BYPASS_PRIORITY | LEGACY_EXIT_BYPASS_PRIORITY) => {
table == MAIN_TABLE table == MAIN_TABLE
&& rule && rule
.attributes .attributes
@@ -789,7 +801,8 @@ async fn remove_exit_client(handle: &rtnetlink::Handle) -> Result<(), OverlayErr
/// ///
/// ```text /// ```text
/// ip route replace default dev <if> table 0x7473 /// ip route replace default dev <if> table 0x7473
/// ip rule add priority 5280 uidrange <uid>-<uid> lookup main /// ip rule add priority 5205 fwmark 0x40000000/0x40000000 lookup main
/// ip rule add priority 5206 uidrange <uid>-<uid> lookup main
/// ip rule add priority 5290 lookup main suppress_prefixlength 0 /// ip rule add priority 5290 lookup main suppress_prefixlength 0
/// ip rule add priority 5300 lookup 0x7473 /// ip rule add priority 5300 lookup 0x7473
/// ``` /// ```
@@ -819,6 +832,8 @@ async fn set_exit_client(name: &str, uid: u32) -> Result<(), OverlayError> {
.await .await
.map_err(|err| fail("default route", err))?; .map_err(|err| fail("default route", err))?;
add_mark_bypass(&handle, false).await?;
let mut bypass = handle let mut bypass = handle
.rule() .rule()
.add() .add()
@@ -891,6 +906,7 @@ async fn set_exit_block_v6(handle: &rtnetlink::Handle, uid: u32) -> Result<(), O
}; };
} }
add_mark_bypass(handle, true).await?;
let mut bypass = handle let mut bypass = handle
.rule() .rule()
.add() .add()
@@ -938,6 +954,27 @@ async fn set_exit_block_v6(handle: &rtnetlink::Handle, uid: u32) -> Result<(), O
.map_err(|err| fail("rule for the default route", err)) .map_err(|err| fail("rule for the default route", err))
} }
async fn add_mark_bypass(handle: &rtnetlink::Handle, v6: bool) -> Result<(), OverlayError> {
let mut rule = handle
.rule()
.add()
.table_id(MAIN_TABLE)
.action(RuleAction::ToTable)
.priority(EXIT_MARK_PRIORITY)
.fw_mark(UNDERLAY_MARK);
rule.message_mut().header.family = if v6 {
rtnetlink::packet_route::AddressFamily::Inet6
} else {
rtnetlink::packet_route::AddressFamily::Inet
};
rule.message_mut()
.attributes
.push(RuleAttribute::FwMask(UNDERLAY_MASK));
rule.execute().await.map_err(|err| {
OverlayError::Unavailable(format!("cannot protect marked underlay traffic: {err}"))
})
}
async fn configure_link(configure: &Configure) -> Result<(), OverlayError> { async fn configure_link(configure: &Configure) -> Result<(), OverlayError> {
let name = configure.name.as_str(); let name = configure.name.as_str();
with_netlink(|handle| async move { with_netlink(|handle| async move {
@@ -1110,6 +1147,28 @@ mod tests {
assert!(!is_exit_rule(&rule(5280, 100, vec![uid]))); assert!(!is_exit_rule(&rule(5280, 100, vec![uid])));
} }
#[test]
fn underlay_bypass_recognises_only_our_exact_bit_and_mask() {
let rule = |table, mark, mask| {
let mut message = RuleMessage::default();
message.attributes.extend([
RuleAttribute::Table(table),
RuleAttribute::Priority(EXIT_MARK_PRIORITY),
RuleAttribute::FwMark(mark),
RuleAttribute::FwMask(mask),
]);
message
};
assert!(is_exit_rule(&rule(
MAIN_TABLE,
UNDERLAY_MARK,
UNDERLAY_MASK
)));
assert!(!is_exit_rule(&rule(MAIN_TABLE, 0x80000, 0xff0000)));
assert!(!is_exit_rule(&rule(MAIN_TABLE, UNDERLAY_MARK, u32::MAX)));
assert!(!is_exit_rule(&rule(52, UNDERLAY_MARK, UNDERLAY_MASK)));
}
#[test] #[test]
fn the_ipv6_default_of_the_exit_table_refuses_instead_of_routing() { fn the_ipv6_default_of_the_exit_table_refuses_instead_of_routing() {
let route = exit_blocking_route_v6(); let route = exit_blocking_route_v6();
crates/tsunagi-core/src/overlay/router.rs
+2 -18
View File
@@ -89,6 +89,7 @@ pub enum RouteError {
/// Address ownership across every network this agent is in. /// Address ownership across every network this agent is in.
#[derive(Debug, Default)] #[derive(Debug, Default)]
pub struct RoutingTable { pub struct RoutingTable {
pub(crate) vips: Arc<crate::vip::VipRouter>,
broadcast: ArcSwap<BroadcastTable>, broadcast: ArcSwap<BroadcastTable>,
networks: RwLock<HashMap<NetworkId, NetworkRoutes>>, networks: RwLock<HashMap<NetworkId, NetworkRoutes>>,
} }
@@ -146,6 +147,7 @@ impl RoutingTable {
/// Forgets a network. /// Forgets a network.
pub fn remove_network(&self, network: NetworkId) { pub fn remove_network(&self, network: NetworkId) {
self.vips.remove_network(network);
let mut networks = self.write(); let mut networks = self.write();
networks.remove(&network); networks.remove(&network);
self.broadcast self.broadcast
@@ -384,24 +386,6 @@ impl RoutingTable {
conflicts conflicts
} }
/// Current signed allocations used for encrypted data delivery checks.
pub(crate) fn probe_addresses(
&self,
network: NetworkId,
peer: EndpointId,
) -> Option<(Ipv4Addr, Ipv4Addr)> {
let table = self.read();
let routes = table.get(&network)?;
Some((
routes.local?,
routes
.peers
.iter()
.find(|(_, holder)| *holder == peer)
.map(|(address, _)| *address)?,
))
}
/// How many networks the table covers. /// How many networks the table covers.
pub fn len(&self) -> usize { pub fn len(&self) -> usize {
self.read().len() self.read().len()
crates/tsunagi-core/src/proto/message.rs
+21 -4
View File
@@ -20,7 +20,7 @@ use crate::error::ProtocolError;
/// The version in the ALPN is the wire-compatibility version of the control /// The version in the ALPN is the wire-compatibility version of the control
/// protocol. It is independent of the network identity scheme version, so /// protocol. It is independent of the network identity scheme version, so
/// bumping it must not change any existing [`crate::NetworkId`]. /// bumping it must not change any existing [`crate::NetworkId`].
pub const ALPN: &[u8] = b"tsunagi/ctrl/7"; pub const ALPN: &[u8] = b"tsunagi/ctrl/8";
/// ALPN of the tsunagi data plane. /// ALPN of the tsunagi data plane.
/// ///
@@ -33,8 +33,10 @@ pub const ALPN: &[u8] = b"tsunagi/ctrl/7";
/// transport fragmentation. Version 5 adds the end-to-end protocol to that /// transport fragmentation. Version 5 adds the end-to-end protocol to that
/// envelope, so a tunnel no longer depends on the kind of link under it, and /// envelope, so a tunnel no longer depends on the kind of link under it, and
/// a first-claimed time to a signed hostname claim. All members must upgrade /// a first-claimed time to a signed hostname claim. All members must upgrade
/// together. Persistent identities and network configuration do not change. /// together. Version 6 uses one fixed inner WireGuard channel and independent
pub const DATA_ALPN: &[u8] = b"tsunagi/data/5"; /// external carrier capabilities. Control authentication, persistent identities
/// and network configuration do not change.
pub const DATA_ALPN: &[u8] = b"tsunagi/data/6";
/// Largest plugin protocol identifier accepted when opening a data channel. /// Largest plugin protocol identifier accepted when opening a data channel.
pub const MAX_DATA_PROTOCOL_LEN: usize = 32; pub const MAX_DATA_PROTOCOL_LEN: usize = 32;
@@ -43,7 +45,7 @@ pub const MAX_DATA_PROTOCOL_LEN: usize = 32;
pub const MAX_SIGNATURE_LEN: usize = 64; pub const MAX_SIGNATURE_LEN: usize = 64;
/// Control protocol version carried inside the handshake. /// Control protocol version carried inside the handshake.
pub const PROTOCOL_VERSION: u16 = 7; pub const PROTOCOL_VERSION: u16 = 8;
pub use crate::dataplane::transport::{MAX_TRANSPORT_PAYLOAD, TransportPayload}; pub use crate::dataplane::transport::{MAX_TRANSPORT_PAYLOAD, TransportPayload};
@@ -181,6 +183,11 @@ pub enum ControlMessage {
/// Short free-text reason. /// Short free-text reason.
reason: String, reason: String,
}, },
/// One complete signed VIP field snapshot, merged atomically.
Vips {
/// Signed registers bounded by the per-VIP field limit.
entries: Vec<crate::vip::VipEntry>,
},
/// Signed cleanup markers, each proving the retired claim version. /// Signed cleanup markers, each proving the retired claim version.
Retirements { Retirements {
/// Bounded by the per-message signed-state limit. /// Bounded by the per-message signed-state limit.
@@ -278,6 +285,15 @@ pub fn validate_capability(
/// network, the other networks or the agent. /// network, the other networks or the agent.
pub fn validate(message: &ControlMessage, limits: &Limits) -> Result<(), ProtocolError> { pub fn validate(message: &ControlMessage, limits: &Limits) -> Result<(), ProtocolError> {
match message { match message {
ControlMessage::Vips { entries } => {
check_len("vip.fields", entries.len(), limits.max_vip_fields)?;
for e in entries {
check_len("vip.id", e.vip.len(), 32)?;
check_len("vip.key", e.key.len(), 160)?;
check_len("vip.value", e.value.as_ref().map_or(0, Vec::len), 256)?;
check_len("vip.signature", e.signature.len(), 64)?;
}
}
ControlMessage::Retirements { markers } => { ControlMessage::Retirements { markers } => {
check_len("retirements", markers.len(), limits.max_state_records)?; check_len("retirements", markers.len(), limits.max_state_records)?;
for marker in markers { for marker in markers {
@@ -373,6 +389,7 @@ pub fn kind(message: &ControlMessage) -> &'static str {
ControlMessage::Pong { .. } => "pong", ControlMessage::Pong { .. } => "pong",
ControlMessage::State { .. } => "state", ControlMessage::State { .. } => "state",
ControlMessage::Retirements { .. } => "retirements", ControlMessage::Retirements { .. } => "retirements",
ControlMessage::Vips { .. } => "vips",
ControlMessage::Reachable { .. } => "reachable", ControlMessage::Reachable { .. } => "reachable",
ControlMessage::Peers { .. } => "peers", ControlMessage::Peers { .. } => "peers",
ControlMessage::Bye { .. } => "bye", ControlMessage::Bye { .. } => "bye",
crates/tsunagi-core/src/state/mod.rs
+2
View File
@@ -38,6 +38,8 @@
//! `docs/sync-model.md` for the implemented cleanup semantics. //! `docs/sync-model.md` for the implemented cleanup semantics.
pub mod allocator; pub mod allocator;
/// Signed shared virtual service state and its merge rules.
pub mod vip;
use std::collections::HashMap; use std::collections::HashMap;
use std::net::Ipv4Addr; use std::net::Ipv4Addr;
crates/tsunagi-core/src/state/vip.rs
+439
View File
@@ -0,0 +1,439 @@
//! Signed multi-writer virtual service registers and deterministic merge rules.
use crate::error::Result;
use crate::vip::{Backend, Listener, Protocol, VipConfig, VipStatus, invalid};
use crate::{Limits, NetworkId};
use iroh::{EndpointId, SecretKey, Signature};
use serde::{Deserialize, Serialize};
use sha2::{Digest, Sha256};
use std::collections::{BTreeMap, BTreeSet};
use std::net::Ipv4Addr;
/// A signed, independently mergeable field. A removed field keeps its tombstone.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct VipEntry {
/// Network scope.
pub network: [u8; 32],
/// Stable service id.
pub vip: String,
/// Canonical field key.
pub key: String,
/// Encoded field value; absent means deletion.
pub value: Option<Vec<u8>>,
/// Lamport clock, independent of wall time.
pub clock: u64,
/// Persistent signing identity.
pub author: [u8; 32],
/// Author's durable network counter.
pub version: u64,
/// Ed25519 signature.
pub signature: Vec<u8>,
}
impl VipEntry {
fn bytes(&self) -> Vec<u8> {
let mut e = self.clone();
e.signature.clear();
let mut out = b"tsunagi-vip-field/v1\0".to_vec();
if let Ok(bytes) = postcard::to_stdvec(&e) {
out.extend(bytes);
}
out
}
/// Verifies bounds, network and authorship before the entry is merged.
pub fn verify(&self, network: NetworkId) -> Result<()> {
if self.network != *network.as_bytes()
|| self.vip.len() != 32
|| hex::decode(&self.vip).is_err()
|| self.key.len() > 160
|| self.value.as_ref().is_some_and(|v| v.len() > 256)
|| self.version == 0
|| self.version > i64::MAX as u64
|| self.clock == 0
|| self.clock > i64::MAX as u64
{
return Err(invalid("invalid VIP field bounds or network"));
}
let author =
EndpointId::from_bytes(&self.author).map_err(|_| invalid("invalid VIP author"))?;
let raw: [u8; 64] = self
.signature
.as_slice()
.try_into()
.map_err(|_| invalid("invalid VIP signature length"))?;
author
.verify(&self.bytes(), &Signature::from_bytes(&raw))
.map_err(|_| invalid("invalid VIP signature"))?;
if self.key != "deleted"
&& self.key != "name"
&& !self.key.starts_with("l/")
&& !self.key.starts_with("b/")
{
return Err(invalid("unknown VIP field"));
}
match self.key.as_str() {
"deleted" => {
if self.value.as_deref() != Some(&[1][..]) {
return Err(invalid("invalid VIP deletion tombstone"));
}
}
"name" => {
let value = self
.value
.as_ref()
.ok_or_else(|| invalid("VIP name cannot be absent"))?;
let name: String =
postcard::from_bytes(value).map_err(|_| invalid("invalid VIP name field"))?;
VipConfig {
id: self.vip.clone(),
name,
listeners: vec![],
}
.validate(&Limits::default())?;
}
key if key.starts_with("l/") => {
let parts: Vec<_> = key.split('/').collect();
if parts.len() != 3
|| parts[1].parse::<Protocol>().is_err()
|| parts[2].parse::<u16>().ok().is_none_or(|p| p == 0)
{
return Err(invalid("invalid listener field key"));
}
if let Some(value) = &self.value {
let l: Listener = postcard::from_bytes(value)
.map_err(|_| invalid("invalid listener field"))?;
if !l.backends.is_empty() || key != format!("l/{}/{}", l.protocol, l.port) {
return Err(invalid("listener key does not match its value"));
}
VipConfig {
id: self.vip.clone(),
name: "validation".into(),
listeners: vec![l],
}
.validate(&Limits::default())?;
}
}
key if key.starts_with("b/") => {
let parts: Vec<_> = key.split('/').collect();
if parts.len() != 6
|| parts[1].parse::<Protocol>().is_err()
|| parts[2].parse::<u16>().ok().is_none_or(|p| p == 0)
|| parts[3].len() != 32
|| hex::decode(parts[3]).is_err()
|| parts[4].parse::<EndpointId>().is_err()
|| parts[5].parse::<u16>().ok().is_none_or(|p| p == 0)
{
return Err(invalid("invalid backend field key"));
}
if let Some(value) = &self.value {
let b: Backend = postcard::from_bytes(value)
.map_err(|_| invalid("invalid backend field"))?;
if b.peer != parts[4] || b.port.to_string() != parts[5] {
return Err(invalid("backend key does not match its value"));
}
}
}
_ => return Err(invalid("unknown VIP field")),
}
Ok(())
}
fn rank(&self) -> (u64, [u8; 32], Vec<u8>) {
(self.clock, self.author, self.bytes())
}
}
/// Signed registers retained across absences, including deletion tombstones.
#[derive(Debug, Clone, Default)]
pub struct VipSet {
entries: BTreeMap<(String, String), VipEntry>,
}
impl VipSet {
/// Highest retained version genuinely signed by this author.
pub fn authored_version(&self, author: EndpointId) -> u64 {
self.entries
.values()
.filter(|e| e.author == *author.as_bytes())
.map(|e| e.version)
.max()
.unwrap_or(0)
}
/// Complete independently verifiable state of one service, including tombstones.
pub fn entries(&self, id: &str) -> Vec<VipEntry> {
self.entries
.values()
.filter(|e| e.vip == id)
.cloned()
.collect()
}
/// All retained service ids, including deleted services.
pub fn ids(&self) -> Vec<String> {
self.entries
.keys()
.map(|(id, _)| id.clone())
.collect::<BTreeSet<_>>()
.into_iter()
.collect()
}
/// Opaque optimistic concurrency token for a whole service.
pub fn revision(&self, id: &str) -> String {
let mut hash = Sha256::new();
for e in self.entries(id) {
hash.update(e.bytes());
hash.update(&e.signature);
}
hex::encode(hash.finalize())
}
/// Atomically verifies and merges one complete service snapshot.
pub fn merge(
&mut self,
network: NetworkId,
entries: Vec<VipEntry>,
limits: &Limits,
) -> Result<bool> {
if entries.len() > limits.max_vip_fields {
return Err(invalid("too many VIP fields"));
}
let mut next = self.clone();
for e in entries {
e.verify(network)?;
let key = (e.vip.clone(), e.key.clone());
if next
.entries
.get(&key)
.is_none_or(|old| e.rank() > old.rank())
{
next.entries.insert(key, e);
}
}
let deleted: BTreeSet<_> = next
.entries
.values()
.filter(|e| e.key == "deleted" && e.value.is_some())
.map(|e| e.vip.clone())
.collect();
// A permanent service tombstone alone prevents every former field's replay.
next.entries
.retain(|(id, key), _| !deleted.contains(id) || key == "deleted");
let ids = next.ids();
if ids.len().saturating_sub(deleted.len()) > limits.max_vips
|| deleted.len() > limits.max_vip_tombstones
|| ids
.iter()
.any(|id| next.entries(id).len() > limits.max_vip_fields)
{
return Err(invalid(
"VIP state capacity exceeded; deletion protection is retained",
));
}
let changed = self.entries != next.entries;
*self = next;
Ok(changed)
}
/// Materialized configuration. A service tombstone permanently retires its id.
pub fn configs(&self) -> Vec<VipConfig> {
self.ids()
.into_iter()
.filter_map(|id| self.config(&id))
.collect()
}
fn config(&self, id: &str) -> Option<VipConfig> {
let fields = self.entries(id);
if fields
.iter()
.any(|e| e.key == "deleted" && e.value.is_some())
{
return None;
}
let name = fields
.iter()
.find(|e| e.key == "name")?
.value
.as_ref()
.and_then(|v| postcard::from_bytes::<String>(v).ok())?;
let mut listeners: Vec<Listener> = fields
.iter()
.filter(|e| e.key.starts_with("l/"))
.filter_map(|e| e.value.as_ref().and_then(|v| postcard::from_bytes(v).ok()))
.collect();
for l in &mut listeners {
l.backends = fields
.iter()
.filter(|e| {
e.key
.starts_with(&format!("b/{}/{}/{}/", l.protocol, l.port, l.id))
})
.filter_map(|e| e.value.as_ref().and_then(|v| postcard::from_bytes(v).ok()))
.collect();
l.backends.sort();
}
listeners.sort_by_key(|l| (l.protocol, l.port));
Some(VipConfig {
id: id.into(),
name,
listeners,
})
}
/// Builds a signed patch from the user's complete, version-checked draft.
pub fn edit(
&self,
network: NetworkId,
secret: &SecretKey,
version: u64,
config: &VipConfig,
expected: Option<&str>,
limits: &Limits,
) -> Result<Vec<VipEntry>> {
config.validate(limits)?;
if let Some(revision) = expected {
if revision != self.revision(&config.id) {
return Err(invalid("VIP changed while editing; reload before saving"));
}
} else if !self.entries(&config.id).is_empty() {
return Err(invalid("VIP id already exists"));
}
if self
.entries
.get(&(config.id.clone(), "deleted".into()))
.is_some_and(|e| e.value.is_some())
{
return Err(invalid("VIP was deleted; create a new VIP"));
}
let before = self
.config(&config.id)
.map(|c| flatten(&c))
.transpose()?
.unwrap_or_default();
let after = flatten(config)?;
let keys: BTreeSet<_> = before.keys().chain(after.keys()).cloned().collect();
let changes = keys
.into_iter()
.filter_map(|key| {
let value = after.get(&key).cloned();
(before.get(&key) != value.as_ref()).then_some((key, value))
})
.collect();
self.sign(network, secret, version, &config.id, changes)
}
/// Builds a permanent service tombstone after checking the selected revision.
pub fn delete(
&self,
network: NetworkId,
secret: &SecretKey,
version: u64,
id: &str,
expected: &str,
) -> Result<Vec<VipEntry>> {
if self.config(id).is_none() || self.revision(id) != expected {
return Err(invalid("VIP missing or changed while editing"));
}
self.sign(
network,
secret,
version,
id,
vec![("deleted".into(), Some(vec![1]))],
)
}
fn sign(
&self,
network: NetworkId,
secret: &SecretKey,
version: u64,
id: &str,
changes: Vec<(String, Option<Vec<u8>>)>,
) -> Result<Vec<VipEntry>> {
let clock = self
.entries
.values()
.map(|e| e.clock)
.max()
.unwrap_or(0)
.checked_add(1)
.filter(|n| *n <= i64::MAX as u64)
.ok_or_else(|| invalid("VIP clock exhausted"))?;
Ok(changes
.into_iter()
.map(|(key, value)| {
let mut e = VipEntry {
network: *network.as_bytes(),
vip: id.into(),
key,
value,
clock,
author: *secret.public().as_bytes(),
version,
signature: vec![],
};
e.signature = secret.sign(&e.bytes()).to_bytes().to_vec();
e
})
.collect())
}
}
pub(crate) fn flatten(cfg: &VipConfig) -> Result<BTreeMap<String, Vec<u8>>> {
let mut fields = BTreeMap::new();
fields.insert(
"name".into(),
postcard::to_stdvec(&cfg.name).map_err(|e| invalid(e.to_string()))?,
);
for l in &cfg.listeners {
let mut settings = l.clone();
settings.backends.clear();
fields.insert(
format!("l/{}/{}", l.protocol, l.port),
postcard::to_stdvec(&settings).map_err(|e| invalid(e.to_string()))?,
);
for b in &l.backends {
fields.insert(
format!("b/{}/{}/{}/{}/{}", l.protocol, l.port, l.id, b.peer, b.port),
postcard::to_stdvec(b).map_err(|e| invalid(e.to_string()))?,
);
}
}
Ok(fields)
}
impl VipSet {
/// Allocates internal addresses deterministically against durable node claims.
pub(crate) fn views(
&self,
range: Option<crate::state::Ipv4Range>,
mut taken: std::collections::HashSet<Ipv4Addr>,
hostnames: &BTreeSet<String>,
limits: &Limits,
) -> Vec<VipStatus> {
let mut names = hostnames.clone();
let mut result = Vec::new();
for config in self.configs() {
let mut error = config.validate(limits).err().map(|e| e.to_string());
if !names.insert(config.name.clone()) {
error = Some("name conflicts with another VIP or node".into());
}
let address = range.and_then(|r| {
let usable = crate::state::allocator::usable_addresses(r);
if usable == 0 {
return None;
}
let digest = Sha256::digest(config.id.as_bytes());
let seed = u64::from_be_bytes([
digest[0], digest[1], digest[2], digest[3], digest[4], digest[5], digest[6],
digest[7],
]);
let bits = 32 - u32::from(r.prefix_len);
let mask = if bits == 32 { 0 } else { u32::MAX << bits };
(0..usable).find_map(|step| {
let offset = (seed.wrapping_add(step) % usable + 1) as u32;
let ip = Ipv4Addr::from((u32::from(r.base) & mask) | offset);
taken.insert(ip).then_some(ip)
})
});
if address.is_none() {
error = Some("no free overlay address".into());
}
result.push(VipStatus {
revision: self.revision(&config.id),
config,
address,
error,
listeners: vec![],
});
}
result
}
}
crates/tsunagi-core/src/storage/mod.rs
+19
View File
@@ -370,6 +370,25 @@ impl Storage {
.await .await
} }
/// Loads mandatory signed VIP fields with a strict capacity bound.
pub async fn vip_fields(
&self,
network: NetworkId,
max: usize,
) -> Result<Vec<crate::vip::VipEntry>> {
self.with_state(move |state| state.vip_fields(network, max))
.await
}
/// Stores a complete merged service snapshot and optional own counter.
pub async fn put_vip_fields(
&self,
network: NetworkId,
entries: Vec<crate::vip::VipEntry>,
own: Option<([u8; 32], u64)>,
) -> Result<()> {
self.with_state(move |state| state.put_vip_fields(network, &entries, own))
.await
}
/// Loads verified durable cleanup proofs off the executor. /// Loads verified durable cleanup proofs off the executor.
pub async fn retirements( pub async fn retirements(
&self, &self,
crates/tsunagi-core/src/storage/state.rs
+133 -1
View File
@@ -21,7 +21,7 @@ use crate::identity::{DeviceIdentity, NetworkId, NetworkName, NetworkSecret};
use crate::state::{MAX_RETIREMENT_LEN, RecordBody, SignedRecord, SignedRetirement}; use crate::state::{MAX_RETIREMENT_LEN, RecordBody, SignedRecord, SignedRetirement};
/// Schema version written by this build. /// Schema version written by this build.
pub const SCHEMA_VERSION: i64 = 9; pub const SCHEMA_VERSION: i64 = 10;
/// Key of the stored hostname setting. /// Key of the stored hostname setting.
const SETTING_HOSTNAME: &str = "hostname"; const SETTING_HOSTNAME: &str = "hostname";
@@ -263,6 +263,12 @@ impl StateStore {
) )
.map_err(|err| self.corrupt(format!("cannot migrate schema to 9: {err}")))?; .map_err(|err| self.corrupt(format!("cannot migrate schema to 9: {err}")))?;
} }
if found < 10 {
self.conn.execute_batch("BEGIN; CREATE TABLE IF NOT EXISTS vip_fields (
network_id BLOB NOT NULL, vip TEXT NOT NULL, field TEXT NOT NULL, data BLOB NOT NULL,
PRIMARY KEY(network_id, vip, field)); PRAGMA user_version = 10; COMMIT;")
.map_err(|err| self.corrupt(format!("cannot migrate VIP state: {err}")))?;
}
Ok(()) Ok(())
} }
@@ -276,6 +282,7 @@ impl StateStore {
"signed_records", "signed_records",
"peer_hostnames", "peer_hostnames",
"retirements", "retirements",
"vip_fields",
] { ] {
let present: Option<String> = self let present: Option<String> = self
.conn .conn
@@ -894,8 +901,77 @@ impl StateStore {
.map_err(|err| Error::Storage(format!("cannot commit retirement: {err}"))) .map_err(|err| Error::Storage(format!("cannot commit retirement: {err}")))
} }
/// Reads bounded mandatory VIP state; corrupt records fail activation.
pub fn vip_fields(&self, network: NetworkId, max: usize) -> Result<Vec<crate::vip::VipEntry>> {
let mut stmt = self.conn.prepare("SELECT CASE WHEN length(data) <= 1024 THEN data ELSE NULL END FROM vip_fields WHERE network_id=?1 ORDER BY vip, field")
.map_err(|e| self.corrupt(e.to_string()))?;
let mut rows = stmt
.query(params![network.as_bytes().as_slice()])
.map_err(|e| self.corrupt(e.to_string()))?;
let mut entries = Vec::new();
while let Some(row) = rows.next().map_err(|e| self.corrupt(e.to_string()))? {
if entries.len() >= max {
return Err(self.corrupt("stored VIP state exceeds configured limits"));
}
let bytes: Option<Vec<u8>> = row.get(0).map_err(|e| self.corrupt(e.to_string()))?;
let bytes = bytes.ok_or_else(|| self.corrupt("oversized VIP field"))?;
let entry: crate::vip::VipEntry =
postcard::from_bytes(&bytes).map_err(|e| self.corrupt(e.to_string()))?;
entry
.verify(network)
.map_err(|e| self.corrupt(e.to_string()))?;
entries.push(entry);
}
Ok(entries)
}
/// Commits a validated VIP snapshot and optional author counter atomically.
pub fn put_vip_fields(
&self,
network: NetworkId,
entries: &[crate::vip::VipEntry],
own: Option<([u8; 32], u64)>,
) -> Result<()> {
let tx = self
.conn
.unchecked_transaction()
.map_err(|e| Error::Storage(e.to_string()))?;
if let Some(first) = entries.first() {
if entries.iter().any(|e| e.vip != first.vip) {
return Err(Error::Storage(
"VIP snapshot mixes service identities".into(),
));
}
tx.execute(
"DELETE FROM vip_fields WHERE network_id=?1 AND vip=?2",
params![network.as_bytes().as_slice(), first.vip],
)
.map_err(|e| Error::Storage(e.to_string()))?;
}
for e in entries {
e.verify(network)?;
let bytes = postcard::to_stdvec(e).map_err(|e| Error::Storage(e.to_string()))?;
if bytes.len() > 1024 {
return Err(Error::Storage("oversized VIP field".into()));
}
tx.execute("INSERT INTO vip_fields(network_id,vip,field,data) VALUES(?1,?2,?3,?4) ON CONFLICT(network_id,vip,field) DO UPDATE SET data=excluded.data",
params![network.as_bytes().as_slice(),e.vip,e.key,bytes]).map_err(|e|Error::Storage(e.to_string()))?;
}
if let Some((author, version)) = own {
tx.execute("INSERT INTO own_record_version(network_id,author,version) VALUES(?1,?2,?3) ON CONFLICT(network_id,author) DO UPDATE SET version=max(version,excluded.version)",
params![network.as_bytes().as_slice(),author.as_slice(),version as i64]).map_err(|e|Error::Storage(e.to_string()))?;
}
tx.commit().map_err(|e| Error::Storage(e.to_string()))
}
/// Drops a forgotten network's local cleanup proofs. /// Drops a forgotten network's local cleanup proofs.
pub fn forget_retirements(&self, network: NetworkId) -> Result<()> { pub fn forget_retirements(&self, network: NetworkId) -> Result<()> {
self.conn
.execute(
"DELETE FROM vip_fields WHERE network_id = ?1",
params![network.as_bytes().as_slice()],
)
.map_err(|e| Error::Storage(e.to_string()))?;
self.conn self.conn
.execute( .execute(
"DELETE FROM retirements WHERE network_id = ?1", "DELETE FROM retirements WHERE network_id = ?1",
@@ -1333,3 +1409,59 @@ mod broadcast_storage_tests {
); );
} }
} }
#[cfg(test)]
mod vip_tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
#[test]
fn vip_migration_and_counter_failure_preserve_identity_and_rollback_fields() {
let dir = tempfile::TempDir::new().unwrap();
let path = dir.path().join("state.sqlite");
let store = StateStore::open(&path).unwrap();
let identity = store.load_or_create_device_identity().unwrap();
store
.conn
.execute_batch("DROP TABLE vip_fields; PRAGMA user_version=9;")
.unwrap();
drop(store);
let store = StateStore::open(&path).unwrap();
assert_eq!(
store.device_identity().unwrap().unwrap().endpoint_id(),
identity.endpoint_id()
);
let net = NetworkId::from_bytes([7; 32]);
let key = identity.signing_key();
let config = crate::vip::VipConfig::new("api");
let fields = crate::vip::VipSet::default()
.edit(net, &key, 7, &config, None, &crate::Limits::default())
.unwrap();
store.conn.execute_batch("CREATE TRIGGER fail_vip_counter BEFORE INSERT ON own_record_version BEGIN SELECT RAISE(ABORT,'counter unavailable'); END;").unwrap();
assert!(
store
.put_vip_fields(net, &fields, Some((*key.public().as_bytes(), 7)))
.is_err()
);
assert!(store.vip_fields(net, 192).unwrap().is_empty());
store
.conn
.execute_batch("DROP TRIGGER fail_vip_counter;")
.unwrap();
store
.put_vip_fields(net, &fields, Some((*key.public().as_bytes(), 7)))
.unwrap();
assert_eq!(store.vip_fields(net, 192).unwrap(), fields);
assert_eq!(store.own_record_version(net, key.public()).unwrap(), 7);
drop(store);
let reopened = StateStore::open(path).unwrap();
assert_eq!(reopened.vip_fields(net, 192).unwrap(), fields);
reopened
.conn
.execute("UPDATE vip_fields SET data=?1", params![vec![0xffu8; 1025]])
.unwrap();
assert!(matches!(
reopened.vip_fields(net, 192),
Err(Error::StateCorrupted { .. })
));
}
}
crates/tsunagi-core/src/vip/mod.rs
+239
View File
@@ -0,0 +1,239 @@
//! Named, shared virtual services. Signed field registers converge without an
//! owner; routing, health and connection state remain local to each client.
use crate::Limits;
use crate::error::{Error, Result};
use iroh::EndpointId;
use serde::{Deserialize, Serialize};
use std::collections::BTreeSet;
use std::net::Ipv4Addr;
pub(crate) mod packet;
pub(crate) mod router;
pub use crate::state::vip::{VipEntry, VipSet};
pub(crate) use router::VipRouter;
#[cfg(test)]
mod tests;
/// IP transport carried by a listener.
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum Protocol {
/// TCP.
Tcp,
/// UDP.
Udp,
}
impl std::fmt::Display for Protocol {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str(match self {
Self::Tcp => "tcp",
Self::Udp => "udp",
})
}
}
impl std::str::FromStr for Protocol {
type Err = String;
fn from_str(s: &str) -> std::result::Result<Self, String> {
match s {
"tcp" => Ok(Self::Tcp),
"udp" => Ok(Self::Udp),
_ => Err("protocol must be tcp or udp".into()),
}
}
}
/// A node identity and its application port.
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct Backend {
/// Full device identity, resolved when editing.
pub peer: String,
/// Destination port.
pub port: u16,
}
/// One independently routed port of a VIP.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct Listener {
/// Stable listener generation; prevents removed backend fields resurfacing on re-creation.
pub id: String,
/// TCP or UDP.
pub protocol: Protocol,
/// Client-facing port.
pub port: u16,
/// Enable active TCP connect checks; UDP uses path checks and ICMP errors.
pub port_check: bool,
/// Period between checks in milliseconds.
pub check_interval_ms: u32,
/// Deadline of a port check in milliseconds.
pub check_timeout_ms: u32,
/// Consecutive failed checks before exclusion.
pub failure_threshold: u8,
/// Consecutive successes before recovery.
pub recovery_threshold: u8,
/// Idle expiry for this listener's flows, in seconds.
pub idle_timeout_secs: u32,
/// Allowed destination nodes and ports.
pub backends: Vec<Backend>,
}
impl Listener {
/// Creates a listener with automatic recovery and bounded checks.
pub fn new(protocol: Protocol, port: u16) -> Self {
Self {
id: hex::encode(rand::random::<[u8; 16]>()),
protocol,
port,
port_check: true,
check_interval_ms: 3000,
check_timeout_ms: 2000,
failure_threshold: 2,
recovery_threshold: 1,
idle_timeout_secs: if protocol == Protocol::Tcp { 7200 } else { 120 },
backends: vec![],
}
}
}
/// Entire editable service; its address is an internal derived reservation.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct VipConfig {
/// Stable random 128-bit id in hexadecimal.
pub id: String,
/// One DNS label, unique within the network.
pub name: String,
/// Independent listeners.
pub listeners: Vec<Listener>,
}
impl VipConfig {
/// Creates an empty named service with a fresh stable identity.
pub fn new(name: impl Into<String>) -> Self {
Self {
id: hex::encode(rand::random::<[u8; 16]>()),
name: name.into(),
listeners: vec![],
}
}
/// Validates a complete draft before any state is changed.
pub fn validate(&self, limits: &Limits) -> Result<()> {
if self.id.len() != 32 || hex::decode(&self.id).is_err() {
return Err(invalid("invalid VIP id"));
}
if self.name.is_empty()
|| self.name.len() > 63
|| self.name.starts_with('-')
|| self.name.ends_with('-')
|| !self
.name
.bytes()
.all(|b| b.is_ascii_lowercase() || b.is_ascii_digit() || b == b'-')
{
return Err(invalid(
"VIP name must be a lowercase DNS label (letters, digits and hyphens)",
));
}
if self.listeners.len() > limits.max_vip_listeners {
return Err(invalid("too many VIP listeners"));
}
let mut seen = BTreeSet::new();
for l in &self.listeners {
if l.id.len() != 32 || hex::decode(&l.id).is_err() {
return Err(invalid("invalid listener identity"));
}
if l.port == 0 || !seen.insert((l.protocol, l.port)) {
return Err(invalid(
"listener ports must be nonzero and unique per protocol",
));
}
if !(250..=3_600_000).contains(&l.check_interval_ms)
|| !(100..=60_000).contains(&l.check_timeout_ms)
|| l.failure_threshold == 0
|| l.recovery_threshold == 0
|| !(1..=604800).contains(&l.idle_timeout_secs)
{
return Err(invalid(
"invalid check interval, timeout, thresholds or flow idle timeout",
));
}
if l.backends.len() > limits.max_vip_backends {
return Err(invalid("too many listener backends"));
}
let mut backends = BTreeSet::new();
for b in &l.backends {
if b.port == 0 || b.peer.parse::<EndpointId>().is_err() || !backends.insert(b) {
return Err(invalid(
"backend must have a unique node identity and a nonzero port",
));
}
}
}
if crate::state::vip::flatten(self)?.len() > limits.max_vip_fields {
return Err(invalid("VIP configuration exceeds signed field capacity"));
}
Ok(())
}
}
pub(crate) fn invalid(reason: impl Into<String>) -> Error {
Error::Rejected {
reason: reason.into(),
}
}
/// Cumulative IP traffic, excluding probes. Reset on agent restart.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct Counters {
/// Outbound packets.
pub tx_packets: u64,
/// Outbound bytes.
pub tx_bytes: u64,
/// Inbound packets.
pub rx_packets: u64,
/// Inbound bytes.
pub rx_bytes: u64,
/// Created flow bindings.
pub flows: u64,
/// UDP failovers.
pub failovers: u64,
/// Packets rejected or lacking an eligible backend.
pub dropped: u64,
}
/// One backend, as seen by this client.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct BackendStatus {
/// Configured endpoint.
pub backend: Backend,
/// Eligibility and reason.
pub state: String,
/// Smoothed full data-plane RTT, microseconds; unknown stays absent.
pub rtt_us: Option<u64>,
/// Last successful path measurement age.
pub measurement_age_ms: Option<u64>,
/// Current bindings.
pub active_flows: u32,
/// Local traffic counters.
pub counters: Counters,
}
/// Runtime view of one listener.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct ListenerStatus {
/// Protocol.
pub protocol: Protocol,
/// Frontend port.
pub port: u16,
/// Best backend for a new flow, if measured and eligible.
pub selected: Option<Backend>,
/// Configured and draining endpoints.
pub backends: Vec<BackendStatus>,
/// Local counters.
pub counters: Counters,
}
/// Configuration and local diagnostics returned through the same API to CLI and GUI.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct VipStatus {
/// Persisted configuration.
pub config: VipConfig,
/// Opaque revision used when saving a draft.
pub revision: String,
/// Internal service address.
pub address: Option<Ipv4Addr>,
/// Conflict or allocation issue, if any.
pub error: Option<String>,
/// Local listener status.
pub listeners: Vec<ListenerStatus>,
}
crates/tsunagi-core/src/vip/packet.rs
+566
View File
@@ -0,0 +1,566 @@
//! Bounded IPv4 parsing, NAT checksums and fragment reassembly at local ingress.
use super::Protocol;
use bytes::Bytes;
use std::collections::HashMap;
use std::net::Ipv4Addr;
use std::time::{Duration, Instant};
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub(crate) struct Tuple {
pub src: Ipv4Addr,
pub dst: Ipv4Addr,
pub sport: u16,
pub dport: u16,
pub protocol: Protocol,
}
impl Tuple {
pub fn reverse(self) -> Self {
Self {
src: self.dst,
dst: self.src,
sport: self.dport,
dport: self.sport,
protocol: self.protocol,
}
}
}
pub(crate) fn header(p: &[u8]) -> Option<usize> {
if p.len() < 20 || p[0] >> 4 != 4 {
return None;
}
let n = usize::from(p[0] & 15) * 4;
let total = usize::from(u16::from_be_bytes([p[2], p[3]]));
(n >= 20 && n <= p.len() && total == p.len() && checksum(&p[..n]) == 0).then_some(n)
}
pub(crate) fn addresses(p: &[u8]) -> Option<(Ipv4Addr, Ipv4Addr)> {
header(p)?;
Some((
Ipv4Addr::new(p[12], p[13], p[14], p[15]),
Ipv4Addr::new(p[16], p[17], p[18], p[19]),
))
}
pub(crate) fn tuple(p: &[u8]) -> Option<Tuple> {
let n = header(p)?;
if u16::from_be_bytes([p[6], p[7]]) & 0x3fff != 0 {
return None;
}
let protocol = match p[9] {
6 => Protocol::Tcp,
17 => Protocol::Udp,
_ => return None,
};
let min = if protocol == Protocol::Tcp { 20 } else { 8 };
if p.len() < n + min {
return None;
}
if protocol == Protocol::Tcp {
let h = usize::from(p[n + 12] >> 4) * 4;
if h < 20 || n + h > p.len() {
return None;
}
} else if usize::from(u16::from_be_bytes([p[n + 4], p[n + 5]])) != p.len() - n {
return None;
}
let (src, dst) = addresses(p)?;
Some(Tuple {
src,
dst,
sport: u16::from_be_bytes([p[n], p[n + 1]]),
dport: u16::from_be_bytes([p[n + 2], p[n + 3]]),
protocol,
})
}
pub(crate) fn rewrite(p: &[u8], t: Tuple) -> Option<Bytes> {
let n = header(p)?;
if p.len() < n + if t.protocol == Protocol::Tcp { 20 } else { 8 } {
return None;
}
let mut out = p.to_vec();
out[12..16].copy_from_slice(&t.src.octets());
out[16..20].copy_from_slice(&t.dst.octets());
out[n..n + 2].copy_from_slice(&t.sport.to_be_bytes());
out[n + 2..n + 4].copy_from_slice(&t.dport.to_be_bytes());
finish(&mut out, n, t.protocol);
Some(Bytes::from(out))
}
fn finish(p: &mut [u8], n: usize, protocol: Protocol) {
p[10..12].fill(0);
let c = checksum(&p[..n]);
p[10..12].copy_from_slice(&c.to_be_bytes());
let offset = n + if protocol == Protocol::Tcp { 16 } else { 6 };
p[offset..offset + 2].fill(0);
let mut sum = 0u32;
for pair in p[12..20].as_chunks::<2>().0 {
sum += u32::from(u16::from_be_bytes([pair[0], pair[1]]));
}
sum += u32::from(p[9]) + (p.len() - n) as u32;
let c = checksum_seed(&p[n..], sum);
let c = if c == 0 && protocol == Protocol::Udp {
0xffff
} else {
c
};
p[offset..offset + 2].copy_from_slice(&c.to_be_bytes());
}
pub(crate) fn checksum(p: &[u8]) -> u16 {
checksum_seed(p, 0)
}
fn checksum_seed(p: &[u8], mut sum: u32) -> u16 {
for pair in p.chunks(2) {
sum += u32::from(u16::from_be_bytes([pair[0], *pair.get(1).unwrap_or(&0)]));
}
while sum >> 16 != 0 {
sum = (sum & 0xffff) + (sum >> 16);
}
!(sum as u16)
}
pub(crate) fn tcp_syn(p: &[u8]) -> bool {
header(p).is_some_and(|n| p[9] == 6 && p.get(n + 13).is_some_and(|f| f & 0x12 == 2))
}
pub(crate) fn syn_sequence(p: &[u8]) -> Option<u32> {
if !tcp_syn(p) {
return None;
}
let n = header(p)?;
let bytes = p.get(n + 4..n + 8)?;
Some(u32::from_be_bytes(bytes.try_into().ok()?))
}
pub(crate) fn empty_syn(p: &[u8]) -> bool {
header(p).is_some_and(|n| {
tcp_syn(p)
&& p.get(n + 12)
.is_some_and(|h| n + usize::from(h >> 4) * 4 == p.len())
})
}
pub(crate) fn syn_ack(p: &[u8]) -> bool {
header(p).is_some_and(|n| p[9] == 6 && p.get(n + 13).is_some_and(|f| f & 0x12 == 0x12))
}
pub(crate) fn tcp_reset(p: &[u8]) -> bool {
header(p).is_some_and(|n| p[9] == 6 && p.get(n + 13).is_some_and(|f| f & 4 != 0))
}
/// Rewrites an ICMP error and its quoted original header after NAT lookup.
pub(crate) fn icmp_quote(p: &[u8]) -> Option<Tuple> {
let n = header(p)?;
if p[9] != 1 || p.len() < n + 8 + 28 || ![3, 11, 12].contains(&p[n]) {
return None;
}
let q = &p[n + 8..];
let h = usize::from(q[0] & 15) * 4;
if q[0] >> 4 != 4 || h < 20 || q.len() < h + 8 {
return None;
}
let protocol = match q[9] {
6 => Protocol::Tcp,
17 => Protocol::Udp,
_ => return None,
};
Some(Tuple {
src: Ipv4Addr::new(q[12], q[13], q[14], q[15]),
dst: Ipv4Addr::new(q[16], q[17], q[18], q[19]),
sport: u16::from_be_bytes([q[h], q[h + 1]]),
dport: u16::from_be_bytes([q[h + 2], q[h + 3]]),
protocol,
})
}
pub(crate) fn rewrite_icmp(p: &[u8], original: Tuple) -> Option<Bytes> {
let n = header(p)?;
icmp_quote(p)?;
let mut out = p.to_vec();
out[12..16].copy_from_slice(&original.dst.octets());
out[16..20].copy_from_slice(&original.src.octets());
let q = &mut out[n + 8..];
let h = usize::from(q[0] & 15) * 4;
// Incrementally update the quoted transport checksum when it is present.
let off = h + if original.protocol == Protocol::Tcp {
16
} else {
6
};
if q.len() >= off + 2 {
let old = u16::from_be_bytes([q[off], q[off + 1]]);
if old != 0 || original.protocol == Protocol::Tcp {
let mut sum = u32::from(!old);
let mut replacement = Vec::from(original.src.octets());
replacement.extend(original.dst.octets());
replacement.extend(original.sport.to_be_bytes());
replacement.extend(original.dport.to_be_bytes());
let previous = [&q[12..20], &q[h..h + 4]].concat();
for (a, b) in previous
.as_chunks::<2>()
.0
.iter()
.zip(replacement.as_chunks::<2>().0)
{
sum += u32::from(!u16::from_be_bytes([a[0], a[1]]))
+ u32::from(u16::from_be_bytes([b[0], b[1]]));
}
while sum >> 16 != 0 {
sum = (sum & 0xffff) + (sum >> 16);
}
let c = !(sum as u16);
q[off..off + 2].copy_from_slice(
&(if c == 0 && original.protocol == Protocol::Udp {
0xffff
} else {
c
})
.to_be_bytes(),
);
}
}
q[12..16].copy_from_slice(&original.src.octets());
q[16..20].copy_from_slice(&original.dst.octets());
q[h..h + 2].copy_from_slice(&original.sport.to_be_bytes());
q[h + 2..h + 4].copy_from_slice(&original.dport.to_be_bytes());
q[10..12].fill(0);
let c = checksum(&q[..h]);
q[10..12].copy_from_slice(&c.to_be_bytes());
out[n + 2..n + 4].fill(0);
let c = checksum(&out[n..]);
out[n + 2..n + 4].copy_from_slice(&c.to_be_bytes());
out[10..12].fill(0);
let c = checksum(&out[..n]);
out[10..12].copy_from_slice(&c.to_be_bytes());
Some(Bytes::from(out))
}
pub(crate) fn reject(p: &[u8]) -> Option<Bytes> {
let t = tuple(p)?;
let n = header(p)?;
if t.protocol == Protocol::Tcp {
if !tcp_syn(p) {
return None;
}
let mut out = vec![0; 40];
out[0] = 0x45;
out[2..4].copy_from_slice(&40u16.to_be_bytes());
out[8] = 64;
out[9] = 6;
out[12..16].copy_from_slice(&t.dst.octets());
out[16..20].copy_from_slice(&t.src.octets());
out[20..22].copy_from_slice(&t.dport.to_be_bytes());
out[22..24].copy_from_slice(&t.sport.to_be_bytes());
let seq = u32::from_be_bytes(p[n + 4..n + 8].try_into().ok()?);
out[28..32].copy_from_slice(&seq.wrapping_add(1).to_be_bytes());
out[32] = 0x50;
out[33] = 0x14;
finish(&mut out, 20, Protocol::Tcp);
Some(Bytes::from(out))
} else {
let q = p.len().min(n + 8);
let mut out = vec![0; 28 + q];
out[0] = 0x45;
let len = out.len() as u16;
out[2..4].copy_from_slice(&len.to_be_bytes());
out[8] = 64;
out[9] = 1;
out[12..16].copy_from_slice(&t.dst.octets());
out[16..20].copy_from_slice(&t.src.octets());
out[20] = 3;
out[21] = 3;
out[28..].copy_from_slice(&p[..q]);
let c = checksum(&out[20..]);
out[22..24].copy_from_slice(&c.to_be_bytes());
let c = checksum(&out[..20]);
out[10..12].copy_from_slice(&c.to_be_bytes());
Some(Bytes::from(out))
}
}
#[derive(Debug)]
struct Datagram {
at: Instant,
header: Option<Vec<u8>>,
parts: std::collections::BTreeMap<usize, Vec<u8>>,
end: Option<usize>,
bytes: usize,
}
type FragmentKey = (Option<[u8; 32]>, Ipv4Addr, Ipv4Addr, u16, u8);
#[derive(Debug, Default)]
pub(crate) struct Reassembly {
pending: HashMap<FragmentKey, Datagram>,
}
impl Reassembly {
pub fn expire(&mut self, now: Instant) {
self.pending
.retain(|_, d| now.duration_since(d.at) < Duration::from_secs(5));
}
pub fn accept(&mut self, p: Bytes, max: usize, scope: Option<[u8; 32]>) -> Option<Bytes> {
let n = header(&p)?;
let flags = u16::from_be_bytes([p[6], p[7]]);
if flags & 0x3fff == 0 {
return Some(p);
}
let now = Instant::now();
self.pending
.retain(|_, d| now.duration_since(d.at) < Duration::from_secs(5));
let (src, dst) = addresses(&p)?;
let key = (scope, src, dst, u16::from_be_bytes([p[4], p[5]]), p[9]);
let start = usize::from(flags & 0x1fff) * 8;
let payload = &p[n..];
let end = start.checked_add(payload.len())?;
if payload.is_empty()
|| end > 65515
|| flags & 0x2000 != 0 && !payload.len().is_multiple_of(8)
{
self.pending.remove(&key);
return None;
}
if !self.pending.contains_key(&key) && self.pending.len() >= max {
return None;
}
let d = self.pending.entry(key).or_insert_with(|| Datagram {
at: now,
header: None,
parts: Default::default(),
end: None,
bytes: 0,
});
if d.parts
.get(&start)
.is_some_and(|bytes| bytes.as_slice() == payload)
{
return None;
}
if d.parts.iter().any(|(s, b)| *s < end && start < s + b.len()) {
self.pending.remove(&key);
return None;
}
if flags & 0x2000 == 0 {
if d.end.is_some_and(|e| e != end) {
self.pending.remove(&key);
return None;
}
d.end = Some(end);
}
if d.end.is_some_and(|e| end > e) {
self.pending.remove(&key);
return None;
}
if start == 0 {
d.header = Some(p[..n].to_vec());
}
d.bytes += payload.len();
d.parts.insert(start, payload.to_vec());
let complete = d.end.is_some_and(|e| e == d.bytes) && d.header.is_some();
if !complete {
return None;
}
let d = self.pending.remove(&key)?;
let mut out = d.header?;
for part in d.parts.values() {
out.extend(part);
}
if out.len() > 65535 {
return None;
}
let len = out.len() as u16;
out[2..4].copy_from_slice(&len.to_be_bytes());
out[6..8].fill(0);
out[10..12].fill(0);
let h = usize::from(out[0] & 15) * 4;
let c = checksum(&out[..h]);
out[10..12].copy_from_slice(&c.to_be_bytes());
Some(Bytes::from(out))
}
}
/// Re-fragments an assembled NAT datagram for the unchanged overlay MTU.
pub(crate) fn fragments(packet: Bytes, mtu: u32) -> Vec<Bytes> {
if packet.len() <= mtu as usize {
return vec![packet];
}
let Some(n) = header(&packet) else {
return vec![];
};
if u16::from_be_bytes([packet[6], packet[7]]) & 0x4000 != 0 {
return vec![];
}
let size = (mtu as usize).saturating_sub(n) / 8 * 8;
if size == 0 {
return vec![];
}
let mut copied = Vec::new();
let mut offset = 20;
while offset < n {
let kind = packet[offset];
if kind == 0 {
break;
}
if kind == 1 {
offset += 1;
continue;
}
let Some(&len) = packet.get(offset + 1) else {
return vec![];
};
let len = usize::from(len);
if len < 2 || offset + len > n {
return vec![];
}
if kind & 0x80 != 0 {
copied.extend_from_slice(&packet[offset..offset + len]);
}
offset += len;
}
while !copied.len().is_multiple_of(4) {
copied.push(0);
}
let mut result = Vec::new();
let payload = &packet[n..];
for (start, part) in payload.chunks(size).enumerate() {
let offset = start * size;
let mut out = if offset == 0 {
packet[..n].to_vec()
} else {
let mut h = packet[..20].to_vec();
h.extend_from_slice(&copied);
h[0] = 0x40 | (h.len() / 4) as u8;
h
};
let header_len = out.len();
out.extend(part);
let len = out.len() as u16;
out[2..4].copy_from_slice(&len.to_be_bytes());
let flags = (offset / 8) as u16
| if offset + part.len() < payload.len() {
0x2000
} else {
0
};
out[6..8].copy_from_slice(&flags.to_be_bytes());
out[10..12].fill(0);
let c = checksum(&out[..header_len]);
out[10..12].copy_from_slice(&c.to_be_bytes());
result.push(Bytes::from(out));
}
result
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
fn datagram() -> Bytes {
let mut p = vec![0; 3028];
p[0] = 0x45;
p[2..4].copy_from_slice(&3028u16.to_be_bytes());
p[8] = 64;
p[9] = 17;
p[12..16].copy_from_slice(&[10, 0, 0, 1]);
p[16..20].copy_from_slice(&[10, 0, 0, 2]);
p[20..22].copy_from_slice(&1000u16.to_be_bytes());
p[22..24].copy_from_slice(&2000u16.to_be_bytes());
p[24..26].copy_from_slice(&3008u16.to_be_bytes());
finish(&mut p, 20, Protocol::Udp);
Bytes::from(p)
}
#[test]
fn fragmented_udp_reassembles_out_of_order_and_nat_checksums_are_valid() {
let p = datagram();
let parts = fragments(p.clone(), 1280);
assert_eq!(parts.len(), 3);
let mut reassembly = Reassembly::default();
let mut result = None;
for part in parts.into_iter().rev() {
if let Some(p) = reassembly.accept(part, 8, None) {
result = Some(p);
}
}
assert_eq!(result, Some(p.clone()));
let mut t = tuple(&p).unwrap();
t.dst = Ipv4Addr::new(10, 0, 0, 240);
t.dport = 53;
let out = rewrite(&p, t).unwrap();
assert_eq!(header(&out), Some(20));
let mut sum = 0u32;
for pair in out[12..20].as_chunks::<2>().0 {
sum += u32::from(u16::from_be_bytes([pair[0], pair[1]]));
}
sum += 17 + (out.len() - 20) as u32;
assert_eq!(checksum_seed(&out[20..], sum), 0);
}
#[test]
fn overlaps_and_capacity_are_rejected_without_allocating_unbounded_state() {
let parts = fragments(datagram(), 1280);
let mut r = Reassembly::default();
assert!(r.accept(parts[0].clone(), 0, None).is_none());
assert!(r.pending.is_empty());
assert!(r.accept(parts[0].clone(), 1, None).is_none());
assert!(r.accept(parts[0].clone(), 1, None).is_none());
assert_eq!(r.pending.len(), 1);
let mut conflict = parts[0].to_vec();
conflict[20] ^= 1;
assert!(r.accept(Bytes::from(conflict), 1, None).is_none());
assert!(r.pending.is_empty());
for size in 0..80 {
let p = vec![0xff; size];
assert!(tuple(&p).is_none());
assert!(icmp_quote(&p).is_none());
}
}
#[test]
fn fragment_assembly_is_scoped_to_authenticated_sender_and_expires_when_idle() {
let packet = datagram();
let parts = fragments(packet.clone(), 1280);
let mut r = Reassembly::default();
assert!(r.accept(parts[0].clone(), 8, Some([1; 32])).is_none());
assert!(r.accept(parts[1].clone(), 8, Some([2; 32])).is_none());
assert!(r.accept(parts[2].clone(), 8, Some([1; 32])).is_none());
assert_eq!(r.accept(parts[1].clone(), 8, Some([1; 32])), Some(packet));
assert_eq!(r.pending.len(), 1);
r.expire(Instant::now() + Duration::from_secs(6));
assert!(r.pending.is_empty());
}
}
#[cfg(test)]
mod icmp_tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
#[test]
fn pmtu_errors_restore_quoted_addresses_ports_and_checksums() {
let original = Tuple {
src: Ipv4Addr::new(10, 0, 0, 1),
dst: Ipv4Addr::new(10, 0, 0, 240),
sport: 40000,
dport: 53,
protocol: Protocol::Udp,
};
let mut p = vec![0; 36];
p[0] = 0x45;
p[2..4].copy_from_slice(&36u16.to_be_bytes());
p[8] = 64;
p[9] = 17;
p[24..26].copy_from_slice(&16u16.to_be_bytes());
let c = checksum(&p[..20]);
p[10..12].copy_from_slice(&c.to_be_bytes());
let p = rewrite(&p, original).unwrap();
let wire = Tuple {
dst: Ipv4Addr::new(10, 0, 0, 2),
sport: 50000,
dport: 5353,
..original
};
let translated = rewrite(&p, wire).unwrap();
let mut error = vec![0; 56];
error[0] = 0x45;
error[2..4].copy_from_slice(&56u16.to_be_bytes());
error[8] = 64;
error[9] = 1;
error[12..16].copy_from_slice(&wire.dst.octets());
error[16..20].copy_from_slice(&wire.src.octets());
error[20] = 3;
error[21] = 4;
error[26..28].copy_from_slice(&1200u16.to_be_bytes());
error[28..].copy_from_slice(&translated[..28]);
let c = checksum(&error[20..]);
error[22..24].copy_from_slice(&c.to_be_bytes());
let c = checksum(&error[..20]);
error[10..12].copy_from_slice(&c.to_be_bytes());
let restored = rewrite_icmp(&error, original).unwrap();
assert_eq!(icmp_quote(&restored), Some(original));
assert_eq!(checksum(&restored[20..]), 0);
assert_eq!(checksum(&restored[28..48]), 0);
assert_eq!(&restored[54..56], &p[26..28]);
assert_eq!(&restored[26..28], &1200u16.to_be_bytes());
assert_eq!(&restored[12..16], &original.dst.octets());
}
}
crates/tsunagi-core/src/vip/router.rs
+1186
View File
File diff suppressed because it is too large. Load diff
crates/tsunagi-core/src/vip/tests.rs
+144
View File
@@ -0,0 +1,144 @@
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
use crate::NetworkId;
#[test]
fn concurrent_backend_edits_merge_and_deletion_cannot_resurrect() {
let net = NetworkId::from_bytes([7; 32]);
let a = iroh::SecretKey::generate();
let b = iroh::SecretKey::generate();
let limits = Limits::default();
let mut state = VipSet::default();
let id = hex::encode([9; 16]);
let initial = VipConfig {
id: id.clone(),
name: "api".into(),
listeners: vec![Listener::new(Protocol::Tcp, 443)],
};
let changes = state.edit(net, &a, 1, &initial, None, &limits).unwrap();
state.merge(net, changes, &limits).unwrap();
let revision = state.revision(&id);
let mut first = initial.clone();
first.listeners[0].backends.push(Backend {
peer: a.public().to_string(),
port: 6443,
});
let mut second = initial.clone();
second.listeners[0].backends.push(Backend {
peer: b.public().to_string(),
port: 7443,
});
let x = state
.edit(net, &a, 2, &first, Some(&revision), &limits)
.unwrap();
let y = state
.edit(net, &b, 1, &second, Some(&revision), &limits)
.unwrap();
let mut reverse = state.clone();
state.merge(net, x.clone(), &limits).unwrap();
state.merge(net, y.clone(), &limits).unwrap();
reverse.merge(net, y, &limits).unwrap();
reverse.merge(net, x, &limits).unwrap();
assert_eq!(state.configs(), reverse.configs());
assert_eq!(state.configs()[0].listeners[0].backends.len(), 2);
assert!(
state
.edit(net, &a, 3, &initial, Some(&revision), &limits)
.is_err()
);
let removed = state.delete(net, &a, 3, &id, &state.revision(&id)).unwrap();
let old = state.entries(&id);
state.merge(net, removed, &limits).unwrap();
state.merge(net, old, &limits).unwrap();
assert!(state.configs().is_empty());
}
#[test]
fn signed_fields_reject_tampering_wrong_network_and_limits() {
let net = NetworkId::from_bytes([2; 32]);
let key = iroh::SecretKey::generate();
let limits = Limits::default();
let state = VipSet::default();
let cfg = VipConfig {
id: hex::encode([3; 16]),
name: "dns".into(),
..Default::default()
};
let entries = state.edit(net, &key, 1, &cfg, None, &limits).unwrap();
let mut bad = entries.clone();
bad[0].clock += 1;
assert!(VipSet::default().merge(net, bad, &limits).is_err());
assert!(
VipSet::default()
.merge(NetworkId::from_bytes([4; 32]), entries.clone(), &limits)
.is_err()
);
let mut small = limits.clone();
small.max_vips = 0;
assert!(VipSet::default().merge(net, entries, &small).is_err());
}
#[test]
fn deletion_history_does_not_exhaust_active_capacity_or_accept_stale_fields() {
let network = NetworkId::from_bytes([9; 32]);
let key = iroh::SecretKey::generate();
let limits = Limits {
max_vips: 1,
max_vip_tombstones: 4,
..Default::default()
};
let mut state = VipSet::default();
for index in 0..4 {
let config = VipConfig::new("reused-name");
let fields = state
.edit(network, &key, index * 2 + 1, &config, None, &limits)
.unwrap();
state.merge(network, fields.clone(), &limits).unwrap();
let deletion = state
.delete(
network,
&key,
index * 2 + 2,
&config.id,
&state.revision(&config.id),
)
.unwrap();
state.merge(network, deletion, &limits).unwrap();
assert_eq!(state.entries(&config.id).len(), 1);
assert!(!state.merge(network, fields, &limits).unwrap());
}
let next = VipConfig::new("next");
let fields = state.edit(network, &key, 9, &next, None, &limits).unwrap();
state.merge(network, fields, &limits).unwrap();
assert_eq!(state.configs().len(), 1);
}
#[test]
fn reservations_avoid_nodes_and_name_conflicts_have_a_deterministic_holder() {
let network = NetworkId::from_bytes([4; 32]);
let key = iroh::SecretKey::generate();
let limits = Limits::default();
let mut state = VipSet::default();
for version in 1..=2 {
let config = VipConfig::new("shared-name");
let fields = state
.edit(network, &key, version, &config, None, &limits)
.unwrap();
state.merge(network, fields, &limits).unwrap();
}
let range = "10.0.0.0/29".parse().unwrap();
let taken = ["10.0.0.1".parse().unwrap()].into_iter().collect();
let views = state.views(Some(range), taken, &Default::default(), &limits);
assert!(views[0].error.is_none());
assert!(
views[1]
.error
.as_deref()
.unwrap()
.contains("name conflicts")
);
assert_ne!(views[0].address, views[1].address);
for v in views {
assert_ne!(v.address, Some("10.0.0.1".parse().unwrap()));
}
}
crates/tsunagi-core/tests/iroh_candidates.rs
+135
View File
@@ -0,0 +1,135 @@
//! The local candidate filter must affect actual hole-punch addresses, not only publications.
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use iroh::{
Endpoint, RelayMode, TransportAddr,
endpoint::{PortmapperConfig, presets},
};
use std::borrow::Cow;
#[tokio::test]
async fn direct_candidate_filter_changes_the_address_watch_used_by_connections() {
let observed = std::sync::Arc::new(std::sync::atomic::AtomicBool::new(false));
let called = observed.clone();
let endpoint = Endpoint::builder(presets::Minimal)
.relay_mode(RelayMode::Disabled)
.clear_address_lookup()
.portmapper_config(PortmapperConfig::Disabled)
.net_report_config(iroh::NetReportConfig::minimal())
.clear_ip_transports()
.bind_addr("127.0.0.1:0")
.unwrap()
// Stand in for addresses on the agent's own overlay interface.
.direct_addr_filter(iroh::address_lookup::AddrFilter::new(move |addrs| {
if addrs
.iter()
.any(|addr| matches!(addr, TransportAddr::Ip(_)))
{
called.store(true, std::sync::atomic::Ordering::Relaxed);
}
Cow::Owned(
addrs
.iter()
.filter(|addr| !matches!(addr, TransportAddr::Ip(_)))
.cloned()
.collect(),
)
}))
.bind()
.await
.unwrap();
tsunagi::testing::wait_until("filter applied to actual bound candidates", || async {
observed
.load(std::sync::atomic::Ordering::Relaxed)
.then_some(())
})
.await;
assert!(
!endpoint.bound_sockets().is_empty(),
"real UDP sockets are still bound"
);
assert!(
endpoint.addr().ip_addrs().next().is_none(),
"excluded local paths never reach iroh's connection candidates"
);
endpoint.close().await;
}
#[tokio::test]
async fn carriers_exclude_the_kernel_assigned_interface_name_instead_of_the_requested_alias() {
use std::sync::{Arc, Mutex};
use tsunagi::dataplane::transport::DataCarrier;
use tsunagi::dataplane::{PluginCapability, PluginError};
use tsunagi::overlay::{MemoryTunFactory, OverlayError, TunDevice, TunFactory, TunRequest};
use tsunagi::{BoxFuture, NetworkId};
#[derive(Debug)]
struct RenamingFactory(MemoryTunFactory);
impl TunFactory for RenamingFactory {
fn name(&self) -> &str {
"kernel-name-test"
}
fn on_host(&self) -> bool {
false
}
fn create<'a>(
&'a self,
mut request: TunRequest,
) -> BoxFuture<'a, Result<Arc<dyn TunDevice>, OverlayError>> {
request.name = "kernel-assigned".into();
self.0.create(request)
}
fn reconfigure<'a>(
&'a self,
request: TunRequest,
) -> BoxFuture<'a, Result<(), OverlayError>> {
self.0.reconfigure(request)
}
fn destroy<'a>(&'a self, name: &'a str) -> BoxFuture<'a, ()> {
self.0.destroy(name)
}
}
#[derive(Debug, Default)]
struct Carrier(Mutex<Option<String>>);
impl DataCarrier for Carrier {
fn protocol_id(&self) -> &str {
"candidate-test"
}
fn protocol_version(&self) -> u16 {
1
}
fn on_interface_ready(&self, name: &str) {
*self.0.lock().unwrap() = Some(name.into());
}
fn local_capability(&self, _: NetworkId) -> Result<Option<PluginCapability>, PluginError> {
Ok(None)
}
fn on_peer_capability(
&self,
_: NetworkId,
_: iroh::EndpointId,
_: &PluginCapability,
) -> Result<(), PluginError> {
Ok(())
}
fn on_peer_gone(&self, _: NetworkId, _: iroh::EndpointId) {}
fn on_network_deactivated(&self, _: NetworkId) {}
}
let root = tempfile::TempDir::new().unwrap();
let discovery = tsunagi::discovery::SharedMemoryDiscovery::new();
let carrier = Arc::new(Carrier::default());
let config = tsunagi::testing::config_with(root.path(), &discovery)
.with_interface(
Arc::new(RenamingFactory(MemoryTunFactory::new())),
"requested-alias",
1280,
)
.with_carrier(carrier.clone());
let agent = tsunagi::Agent::spawn(config).await.unwrap();
assert_eq!(agent.overlay().unwrap().interface, "kernel-assigned");
assert_eq!(
carrier.0.lock().unwrap().as_deref(),
Some("kernel-assigned")
);
agent.shutdown().await;
}
crates/tsunagi-core/tests/iroh_path_retries.rs
+4
View File
@@ -0,0 +1,4 @@
//! Exercise the vendored retry admission in the default workspace suite.
//! Transport integration tests still use real iroh endpoints.
#[path = "../../../vendor/iroh/src/socket/remote_map/remote_state/pending_paths.rs"]
mod pending_paths;
crates/tsunagi-core/tests/mixed_transports.rs
+5 -216
View File
@@ -1,220 +1,9 @@
//! Connectivity does not depend on what kind of link each hop happens to be, //! Signed hostname ownership remains independent of connectivity and identity order.
//! and a member is online however far away it is.
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)] #![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use tsunagi::{
use std::collections::{HashMap, HashSet}; discovery::SharedMemoryDiscovery,
use std::sync::{Arc, Mutex}; testing::{TestAgent, network, wait_for_peers, wait_until},
};
use bytes::Bytes;
use iroh::EndpointId;
use tsunagi::NetworkId;
use tsunagi::agent::DataPath;
use tsunagi::dataplane::transport::SharedLink;
use tsunagi::dataplane::{IpPlugin, PluginCapability, PluginError};
use tsunagi::discovery::SharedMemoryDiscovery;
use tsunagi::testing::{TestAgent, network, wait_for_peers, wait_until};
/// A protocol that moves bytes between two members unchanged and keeps the
/// link it was handed, so a test can use the link directly.
#[derive(Debug)]
struct Carrier {
protocol: String,
links: Mutex<HashMap<EndpointId, SharedLink>>,
}
impl Carrier {
fn new(protocol: &str) -> Arc<Self> {
Arc::new(Self {
protocol: protocol.into(),
links: Mutex::default(),
})
}
fn link(&self, peer: EndpointId) -> Option<SharedLink> {
self.links.lock().unwrap().get(&peer).cloned()
}
}
impl IpPlugin for Carrier {
fn protocol_id(&self) -> &str {
&self.protocol
}
fn protocol_version(&self) -> u16 {
1
}
fn local_capability(
&self,
_network: NetworkId,
) -> Result<Option<PluginCapability>, PluginError> {
Ok(Some(PluginCapability {
protocol: self.protocol.clone(),
version: 1,
enabled: true,
data: Vec::new(),
}))
}
fn on_peer_capability(
&self,
_network: NetworkId,
_peer: EndpointId,
_capability: &PluginCapability,
) -> Result<(), PluginError> {
Ok(())
}
fn on_peer_link(&self, _network: NetworkId, peer: EndpointId, link: SharedLink) {
self.links.lock().unwrap().insert(peer, link);
}
fn on_peer_gone(&self, _network: NetworkId, peer: EndpointId) {
self.links.lock().unwrap().remove(&peer);
}
fn on_network_deactivated(&self, _network: NetworkId) {}
}
type Blocked = Arc<Mutex<HashSet<EndpointId>>>;
async fn member(
discovery: &SharedMemoryDiscovery,
plugins: &[Arc<Carrier>],
blocked: &Blocked,
) -> TestAgent {
let plugins: Vec<_> = plugins.to_vec();
let blocked = Arc::clone(blocked);
TestAgent::spawn_with(
move |config| {
let config = config.with_unreachable_data_peers(blocked);
plugins
.into_iter()
.fold(config, |config, plugin| config.with_plugin(plugin))
},
discovery,
)
.await
.unwrap()
}
async fn exchange(from: &SharedLink, to: &SharedLink, payload: &'static [u8]) {
from.send(Bytes::from_static(payload)).unwrap();
let got = tokio::time::timeout(tsunagi::testing::DEADLINE, to.recv())
.await
.expect("the datagram should arrive")
.expect("the link stays open");
assert_eq!(got, Bytes::from_static(payload));
}
#[tokio::test]
async fn a_tunnel_crosses_a_member_that_shares_no_protocol_with_either_end_and_moves_to_a_direct_link_by_itself()
{
// A and B share one protocol and nothing else. C shares a different
// one with each, so every link here is of a different kind and the
// middle has never heard of what A and B speak. A and B cannot reach
// each other directly, which is the case a blocked UDP path produces.
let discovery = SharedMemoryDiscovery::new();
let (name, secret) = network("mixed-transports");
let blocked_a: Blocked = Arc::default();
let blocked_b: Blocked = Arc::default();
let a_shared = Carrier::new("shared");
let b_shared = Carrier::new("shared");
let a = member(
&discovery,
&[Carrier::new("a-and-c"), Arc::clone(&a_shared)],
&blocked_a,
)
.await;
let b = member(
&discovery,
&[Carrier::new("c-and-b"), Arc::clone(&b_shared)],
&blocked_b,
)
.await;
let c = member(
&discovery,
&[Carrier::new("a-and-c"), Carrier::new("c-and-b")],
&Blocked::default(),
)
.await;
let (a_id, b_id, c_id) = (
a.agent.endpoint_id(),
b.agent.endpoint_id(),
c.agent.endpoint_id(),
);
blocked_a.lock().unwrap().insert(b_id);
blocked_b.lock().unwrap().insert(a_id);
let network_id = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
c.agent.join_network(&name, &secret).await.unwrap();
for agent in [&a, &b, &c] {
wait_for_peers(&agent.agent, network_id, 2).await;
}
// The tunnel comes up through C, and data goes both ways.
let to_b = wait_until("a has a link to b", || async { a_shared.link(b_id) }).await;
let to_a = wait_until("b has a link to a", || async { b_shared.link(a_id) }).await;
exchange(&to_b, &to_a, b"a to b, sealed").await;
exchange(&to_a, &to_b, b"b to a, sealed").await;
// The status says what is true: online, and through whom.
let status = a.agent.network_status(network_id).await.unwrap();
let b_peer = status.peers.iter().find(|p| p.endpoint_id == b_id).unwrap();
assert_eq!(b_peer.data_path, DataPath::Relayed { hops: 2, via: c_id });
let c_peer = status.peers.iter().find(|p| p.endpoint_id == c_id).unwrap();
assert!(matches!(&c_peer.data_path, DataPath::Direct { transport } if transport == "a-and-c"));
let b_member = wait_until("a lists b as a member", || async {
let status = a.agent.network_status(network_id).await.ok()?;
status.members.into_iter().find(|m| m.endpoint_id == b_id)
})
.await;
assert!(b_member.online);
let carried = c.agent.network_status(network_id).await.unwrap().relay;
assert!(carried.forwarded >= 2, "{carried:?}");
// The direct path becomes possible: the same link carries on, now
// without C, and nobody has to do anything about it.
blocked_a.lock().unwrap().clear();
blocked_b.lock().unwrap().clear();
wait_until("a reaches b directly", || async {
let status = a.agent.network_status(network_id).await.ok()?;
let peer = status.peers.iter().find(|p| p.endpoint_id == b_id)?;
matches!(&peer.data_path, DataPath::Direct { transport } if transport == "shared")
.then_some(())
})
.await;
let forwarded = c
.agent
.network_status(network_id)
.await
.unwrap()
.relay
.forwarded;
exchange(&to_b, &to_a, b"direct now").await;
assert!(
Arc::ptr_eq(&to_b, &a_shared.link(b_id).unwrap()),
"the tunnel's link is the one it always had"
);
assert_eq!(
c.agent
.network_status(network_id)
.await
.unwrap()
.relay
.forwarded,
forwarded,
"nothing goes through C any more"
);
a.stop().await;
b.stop().await;
c.stop().await;
}
#[tokio::test] #[tokio::test]
async fn a_name_belongs_to_the_member_that_claimed_it_first_on_every_member() { async fn a_name_belongs_to_the_member_that_claimed_it_first_on_every_member() {
let discovery = SharedMemoryDiscovery::new(); let discovery = SharedMemoryDiscovery::new();
crates/tsunagi-core/tests/multi_peer.rs
+2 -2
View File
@@ -132,7 +132,7 @@ async fn opaque_plugin_capabilities_cross_the_control_plane() {
let a = TestAgent::spawn_with( let a = TestAgent::spawn_with(
{ {
let plugin = Arc::clone(&plugin_a); let plugin = Arc::clone(&plugin_a);
move |cfg| cfg.with_plugin(plugin) move |cfg| cfg.with_tunnel(plugin)
}, },
&discovery, &discovery,
) )
@@ -141,7 +141,7 @@ async fn opaque_plugin_capabilities_cross_the_control_plane() {
let b = TestAgent::spawn_with( let b = TestAgent::spawn_with(
{ {
let plugin = Arc::clone(&plugin_b); let plugin = Arc::clone(&plugin_b);
move |cfg| cfg.with_plugin(plugin) move |cfg| cfg.with_tunnel(plugin)
}, },
&discovery, &discovery,
) )
crates/tsunagi-core/tests/vip.rs
+207
View File
@@ -0,0 +1,207 @@
//! Shared services use real membership, signed state and mandatory SQLite.
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use tsunagi::discovery::SharedMemoryDiscovery;
use tsunagi::testing::{TestAgent, config_with, network, wait_for_peers, wait_until};
use tsunagi::vip::{Backend, Listener, Protocol, VipConfig, VipStatus};
use tsunagi::{Agent, NetworkId};
async fn view(agent: &Agent, network: NetworkId, id: &str) -> VipStatus {
wait_until("VIP configuration reaches the replica", || async {
agent
.network_status(network)
.await
.ok()?
.vips
.into_iter()
.find(|v| v.config.id == id)
})
.await
}
#[tokio::test]
async fn any_member_can_edit_services_and_an_offline_creator_is_not_required() {
let discovery = SharedMemoryDiscovery::new();
let a = TestAgent::spawn(&discovery).await.unwrap();
let b = TestAgent::spawn(&discovery).await.unwrap();
let (name, secret) = network("shared-vips");
let net = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
wait_for_peers(&a.agent, net, 1).await;
let cfg = VipConfig::new("api");
a.agent
.save_vip(net, cfg.clone(), None, false)
.await
.unwrap();
let before = view(&b.agent, net, &cfg.id).await;
let mut edited = before.config.clone();
let mut listener = Listener::new(Protocol::Tcp, 443);
listener.backends.push(Backend {
peer: b.agent.endpoint_id().to_string(),
port: 6443,
});
edited.listeners.push(listener);
b.agent
.save_vip(net, edited.clone(), Some(before.revision.clone()), false)
.await
.unwrap();
assert!(
b.agent
.save_vip(net, cfg.clone(), Some(before.revision), false)
.await
.is_err()
);
wait_until("creator observes another member's edit", || async {
(view(&a.agent, net, &cfg.id).await.config == edited).then_some(())
})
.await;
let a_dir = a.stop().await;
let c = TestAgent::spawn(&discovery).await.unwrap();
c.agent.join_network(&name, &secret).await.unwrap();
assert_eq!(view(&c.agent, net, &cfg.id).await.config, edited);
let a = Agent::spawn(config_with(a_dir.path(), &discovery))
.await
.unwrap();
assert_eq!(view(&a, net, &cfg.id).await.config, edited);
let current = view(&c.agent, net, &cfg.id).await;
c.agent
.save_vip(net, current.config, Some(current.revision), true)
.await
.unwrap();
wait_until("deletion reaches the original creator", || async {
a.network_status(net)
.await
.ok()?
.vips
.is_empty()
.then_some(())
})
.await;
a.shutdown().await;
b.stop().await;
c.stop().await;
}
#[tokio::test]
async fn invalid_drafts_are_atomic_and_services_do_not_cross_networks() {
let discovery = SharedMemoryDiscovery::new();
let a = TestAgent::spawn(&discovery).await.unwrap();
let (name, secret) = network("vip-one");
let net = a.agent.join_network(&name, &secret).await.unwrap();
let (other, secret) = network("vip-two");
let second = a.agent.join_network(&other, &secret).await.unwrap();
let mut cfg = VipConfig::new("api");
cfg.listeners.push(Listener::new(Protocol::Udp, 53));
a.agent
.save_vip(net, cfg.clone(), None, false)
.await
.unwrap();
let before = view(&a.agent, net, &cfg.id).await;
let mut invalid = before.config.clone();
invalid.name = "renamed".into();
invalid.listeners.push(invalid.listeners[0].clone());
assert!(
a.agent
.save_vip(net, invalid, Some(before.revision.clone()), false)
.await
.is_err()
);
let after = view(&a.agent, net, &cfg.id).await;
assert_eq!(after.config, before.config);
assert_eq!(after.revision, before.revision);
assert!(
a.agent
.network_status(second)
.await
.unwrap()
.vips
.is_empty()
);
a.stop().await;
}
#[tokio::test]
async fn a_restored_author_advances_its_counter_past_its_own_updates_held_by_a_replica() {
use tsunagi::config::StoragePaths;
use tsunagi::storage::StateStore;
use tsunagi::vip::VipSet;
let discovery = SharedMemoryDiscovery::new();
let a = TestAgent::spawn(&discovery).await.unwrap();
let b = TestAgent::spawn(&discovery).await.unwrap();
let (name, secret) = network("vip-author-recovery");
let net = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
wait_for_peers(&a.agent, net, 1).await;
let config = VipConfig::new("api");
a.agent
.save_vip(net, config.clone(), None, false)
.await
.unwrap();
view(&b.agent, net, &config.id).await;
let author = a.agent.endpoint_id();
let adir = a.stop().await;
let bdir = b.stop().await;
let apath = StoragePaths::under(adir.path()).state_db();
let bpath = StoragePaths::under(bdir.path()).state_db();
// Model an old author backup with a newer, genuinely signed update retained
// by another offline replica. Connections and recovery below remain real.
tokio::task::spawn_blocking(move || {
let a = StateStore::open(apath).unwrap();
let b = StateStore::open(bpath).unwrap();
let mut set = VipSet::default();
set.merge(
net,
a.vip_fields(net, 192).unwrap(),
&tsunagi::Limits::default(),
)
.unwrap();
let mut cfg = set.configs().remove(0);
cfg.name = "newer".into();
let key =
iroh::SecretKey::from_bytes(&a.device_identity().unwrap().unwrap().secret_bytes());
let fields = set
.edit(
net,
&key,
100,
&cfg,
Some(&set.revision(&cfg.id)),
&tsunagi::Limits::default(),
)
.unwrap();
set.merge(net, fields, &tsunagi::Limits::default()).unwrap();
b.put_vip_fields(net, &set.entries(&cfg.id), None).unwrap();
})
.await
.unwrap();
let b = Agent::spawn(config_with(bdir.path(), &discovery))
.await
.unwrap();
let a = Agent::spawn(config_with(adir.path(), &discovery))
.await
.unwrap();
let learned = wait_until(
"the restored author learns its newer signed service",
|| async {
let v = view(&a, net, &config.id).await;
(v.config.name == "newer").then_some(v)
},
)
.await;
let mut next = learned.config;
next.name = "latest".into();
a.save_vip(net, next, Some(learned.revision), false)
.await
.unwrap();
a.shutdown().await;
let path = StoragePaths::under(adir.path()).state_db();
let version = tokio::task::spawn_blocking(move || {
StateStore::open(path)
.unwrap()
.own_record_version(net, author)
.unwrap()
})
.await
.unwrap();
assert!(
version > 100,
"the observed author counter must not be reused"
);
b.shutdown().await;
}
crates/tsunagi-tcp-tls/Cargo.toml
+1 -10
View File
@@ -1,7 +1,7 @@
[package] [package]
name = "tsunagi-tcp-tls" name = "tsunagi-tcp-tls"
version = "0.1.0" version = "0.1.0"
description = "TCP TLS 443 transport and E2EE tunnel protocol for tsunagi mesh networks." description = "TCP TLS carrier for independently encrypted tsunagi mesh traffic."
edition.workspace = true edition.workspace = true
rust-version.workspace = true rust-version.workspace = true
license.workspace = true license.workspace = true
@@ -14,19 +14,10 @@ tokio = { workspace = true, features = ["net", "io-util", "time", "sync", "macro
rustls = "0.23" rustls = "0.23"
tokio-rustls = "0.26" tokio-rustls = "0.26"
rcgen = { version = "0.14", default-features = false, features = ["ring"] } rcgen = { version = "0.14", default-features = false, features = ["ring"] }
x25519-dalek = { version = "2.0", features = ["static_secrets"] }
chacha20poly1305 = "0.10"
sha2 = "0.11"
hkdf = "0.13"
rand = "0.10"
subtle = "2.6"
zeroize = { version = "1.9", features = ["derive"] }
bytes.workspace = true bytes.workspace = true
serde.workspace = true serde.workspace = true
postcard.workspace = true postcard.workspace = true
tracing.workspace = true tracing.workspace = true
thiserror.workspace = true
hex.workspace = true
netdev = "0.46" netdev = "0.46"
[target.'cfg(target_os = "linux")'.dependencies] [target.'cfg(target_os = "linux")'.dependencies]
crates/tsunagi-tcp-tls/src/crypto.rs
-104
View File
@@ -1,104 +0,0 @@
//! End-to-end packet encryption for `tcp-tls`.
//!
//! Uses X25519 Diffie-Hellman key exchange and ChaCha20-Poly1305 AEAD.
//! This ensures that even when traffic passes through intermediate relay nodes,
//! the intermediate nodes cannot inspect or tamper with the payload.
use std::sync::atomic::{AtomicU64, Ordering};
use chacha20poly1305::aead::{Aead, KeyInit};
use chacha20poly1305::{ChaCha20Poly1305, Nonce};
use hkdf::Hkdf;
use iroh::EndpointId;
use sha2::Sha256;
use tsunagi::dataplane::PluginError;
use x25519_dalek::{PublicKey, StaticSecret};
/// Length of the X25519 public key.
pub const KEY_LEN: usize = 32;
/// Domain separator for session key derivation.
const HKDF_SALT: &[u8] = b"tsunagi/e2ee/noise/v1";
/// A session cipher for communicating with one peer.
pub struct PeerCipher {
tx_cipher: ChaCha20Poly1305,
rx_cipher: ChaCha20Poly1305,
tx_nonce: AtomicU64,
}
impl PeerCipher {
/// Establishes a new cipher between `local_id` and `remote_id` given
/// our local secret and their public key.
pub fn new(
local_id: EndpointId,
remote_id: EndpointId,
local_secret: &StaticSecret,
remote_public: &PublicKey,
) -> Self {
let shared = local_secret.diffie_hellman(remote_public);
let hk = Hkdf::<Sha256>::new(Some(HKDF_SALT), shared.as_bytes());
let mut key_a_to_b = [0u8; 32];
let mut key_b_to_a = [0u8; 32];
let _ = hk.expand(b"tsunagi-key-a-to-b", &mut key_a_to_b);
let _ = hk.expand(b"tsunagi-key-b-to-a", &mut key_b_to_a);
// Deterministic role: smaller EndpointId is party A.
let is_party_a = local_id.as_bytes() < remote_id.as_bytes();
let (tx_key, rx_key) = if is_party_a {
(key_a_to_b, key_b_to_a)
} else {
(key_b_to_a, key_a_to_b)
};
let tx_cipher = ChaCha20Poly1305::new(&tx_key.into());
let rx_cipher = ChaCha20Poly1305::new(&rx_key.into());
Self {
tx_cipher,
rx_cipher,
tx_nonce: AtomicU64::new(0),
}
}
/// Encrypts an outbound packet. Prefixes the ciphertext with the 8-byte nonce.
pub fn encrypt(&self, packet: &[u8]) -> Result<bytes::Bytes, PluginError> {
let counter = self.tx_nonce.fetch_add(1, Ordering::Relaxed);
let mut nonce_bytes = [0u8; 12];
nonce_bytes[4..12].copy_from_slice(&counter.to_be_bytes());
let nonce = Nonce::from_slice(&nonce_bytes);
let ciphertext = self
.tx_cipher
.encrypt(nonce, packet)
.map_err(|err| PluginError::Other(format!("encryption failed: {err}")))?;
let mut out = Vec::with_capacity(8 + ciphertext.len());
out.extend_from_slice(&counter.to_be_bytes());
out.extend_from_slice(&ciphertext);
Ok(bytes::Bytes::from(out))
}
/// Decrypts an inbound packet prefixed with the 8-byte nonce.
pub fn decrypt(&self, payload: &[u8]) -> Result<bytes::Bytes, PluginError> {
if payload.len() < 8 + 16 {
return Err(PluginError::Other("payload too short".into()));
}
let counter_bytes: [u8; 8] = match payload[..8].try_into() {
Ok(b) => b,
Err(_) => return Err(PluginError::Other("invalid nonce".into())),
};
let mut nonce_bytes = [0u8; 12];
nonce_bytes[4..12].copy_from_slice(&counter_bytes);
let nonce = Nonce::from_slice(&nonce_bytes);
let plaintext = self
.rx_cipher
.decrypt(nonce, &payload[8..])
.map_err(|err| PluginError::Other(format!("decryption failed: {err}")))?;
Ok(bytes::Bytes::from(plaintext))
}
}
crates/tsunagi-tcp-tls/src/lib.rs
+2 -5
View File
@@ -1,11 +1,8 @@
//! The `tcp-tls` protocol: TCP with TLS 1.3 on port 443 with Noise E2EE. //! TCP TLS 1.3 carrier for opaque, independently encrypted mesh datagrams.
pub mod cert; pub mod cert;
pub mod crypto;
pub mod plugin; pub mod plugin;
pub mod transport; pub mod transport;
pub use plugin::{ pub use plugin::{
TCP_TLS_PROTOCOL, TCP_TLS_VERSION, TcpTlsCodec, TcpTlsConfig, TcpTlsPlugin, local_ip_candidates, TCP_TLS_PROTOCOL, TCP_TLS_VERSION, TcpTlsCarrier, TcpTlsConfig, local_ip_candidates,
}; };
pub use transport::TcpTlsTransport; pub use transport::TcpTlsTransport;
crates/tsunagi-tcp-tls/src/plugin.rs
+168 -297
View File
@@ -1,26 +1,14 @@
//! The `tcp-tls` IP plugin implementation. //! TLS carrier options and bounded reachability announcements. No IP cipher lives here.
use std::collections::HashMap;
use std::sync::{Arc, RwLock};
use bytes::Bytes;
use iroh::EndpointId;
use serde::{Deserialize, Serialize};
use x25519_dalek::{PublicKey, StaticSecret};
use tsunagi::dataplane::tunnel::{GenericTunnelPlugin, TunnelCodec};
use tsunagi::dataplane::{PluginError, ProtocolOption};
use tsunagi::identity::NetworkId;
use crate::crypto::PeerCipher;
use crate::transport::TcpTlsTransport; use crate::transport::TcpTlsTransport;
use iroh::EndpointId;
/// The wire protocol identifier for `tcp-tls`. use serde::Deserialize;
use std::sync::Arc;
use tsunagi::NetworkId;
use tsunagi::dataplane::{PluginError, ProtocolOption};
/// External TLS carrier identifier.
pub const TCP_TLS_PROTOCOL: &str = "tcp-tls"; pub const TCP_TLS_PROTOCOL: &str = "tcp-tls";
/// Carrier announcement wire version.
/// Wire version for `tcp-tls`. pub const TCP_TLS_VERSION: u16 = 2;
pub const TCP_TLS_VERSION: u16 = 1;
/// Configuration for the `tcp-tls` protocol plugin. /// Configuration for the `tcp-tls` protocol plugin.
#[derive(Debug, Clone, Default)] #[derive(Debug, Clone, Default)]
pub struct TcpTlsConfig { pub struct TcpTlsConfig {
@@ -49,23 +37,22 @@ impl TcpTlsConfig {
} }
} }
/// Announcement payload exchanged between peers for `tcp-tls`.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct TcpTlsAnnouncement {
/// X25519 public key for E2EE noise encryption.
pub noise_public: [u8; 32],
/// TCP port the peer is listening on.
pub port: u16,
/// Candidate IP addresses where this peer can be reached on `port`.
#[serde(default)]
pub addrs: Vec<std::net::IpAddr>,
}
/// Discovers local network interface IPs to announce to peers. /// Discovers local network interface IPs to announce to peers.
pub fn local_ip_candidates() -> Vec<std::net::IpAddr> { pub fn local_ip_candidates() -> Vec<std::net::IpAddr> {
local_ip_candidates_excluding(None)
}
fn local_ip_candidates_excluding(excluded_interface: Option<&str>) -> Vec<std::net::IpAddr> {
let mut candidates = Vec::new(); let mut candidates = Vec::new();
for iface in netdev::get_interfaces() { for iface in netdev::get_interfaces() {
if !is_candidate_interface(&iface) { if excluded_interface.is_some_and(|name| {
iface.name.eq_ignore_ascii_case(name)
|| iface
.friendly_name
.as_deref()
.is_some_and(|friendly| friendly.eq_ignore_ascii_case(name))
}) || !is_candidate_interface(&iface)
{
continue; continue;
} }
for ip in &iface.ipv4 { for ip in &iface.ipv4 {
@@ -203,26 +190,7 @@ pub(crate) fn ip_priority(iface: &netdev::Interface, ip: std::net::IpAddr) -> u3
} }
} }
} }
impl TcpTlsConfig {
/// The codec implementing cryptographic transformation and announcements for `tcp-tls`.
pub struct TcpTlsCodec {
local_id: EndpointId,
static_secret: StaticSecret,
public_key: PublicKey,
transport: Arc<TcpTlsTransport>,
ciphers: RwLock<HashMap<(NetworkId, EndpointId), Arc<PeerCipher>>>,
}
impl std::fmt::Debug for TcpTlsCodec {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("TcpTlsCodec")
.field("local_id", &self.local_id.fmt_short().to_string())
.field("port", &self.transport.bound_port())
.finish()
}
}
impl TcpTlsCodec {
/// Supported options for `tcp-tls`. /// Supported options for `tcp-tls`.
pub const OPTIONS: &'static [ProtocolOption] = &[ pub const OPTIONS: &'static [ProtocolOption] = &[
ProtocolOption { ProtocolOption {
@@ -239,22 +207,6 @@ impl TcpTlsCodec {
}, },
]; ];
/// Creates a new `TcpTlsCodec`.
pub fn new(local_id: EndpointId, transport: Arc<TcpTlsTransport>) -> Self {
let mut secret_bytes = [0u8; 32];
rand::fill(&mut secret_bytes);
let static_secret = StaticSecret::from(secret_bytes);
let public_key = PublicKey::from(&static_secret);
Self {
local_id,
static_secret,
public_key,
transport,
ciphers: RwLock::new(HashMap::new()),
}
}
/// Configures settings from parsed `-o` options. /// Configures settings from parsed `-o` options.
pub fn configure( pub fn configure(
mut config: TcpTlsConfig, mut config: TcpTlsConfig,
@@ -281,250 +233,169 @@ impl TcpTlsCodec {
Ok(config) Ok(config)
} }
} }
/// TLS carrier announcements and lifecycle, independent of IP encryption.
impl TunnelCodec for TcpTlsCodec { #[derive(Debug)]
pub struct TcpTlsCarrier {
transport: Arc<TcpTlsTransport>,
excluded_interface: std::sync::Mutex<Option<String>>,
peers: std::sync::Mutex<
std::collections::HashMap<(NetworkId, EndpointId), Vec<std::net::SocketAddr>>,
>,
}
impl TcpTlsCarrier {
/// Shares the transport's listener; creates no end-to-end cipher.
pub fn new(transport: Arc<TcpTlsTransport>) -> Arc<Self> {
Arc::new(Self {
transport,
excluded_interface: std::sync::Mutex::new(None),
peers: Default::default(),
})
}
/// Registers a carrier while excluding the agent's own, possibly custom-named interface.
pub fn with_interface(transport: Arc<TcpTlsTransport>, interface: String) -> Arc<Self> {
Arc::new(Self {
transport,
excluded_interface: std::sync::Mutex::new(Some(interface)),
peers: Default::default(),
})
}
fn update_peer(
&self,
network: NetworkId,
peer: EndpointId,
addrs: Option<Vec<std::net::SocketAddr>>,
) {
let mut peers = self.peers.lock().unwrap_or_else(|p| p.into_inner());
match addrs {
Some(addrs) => {
peers.insert((network, peer), addrs);
}
None => {
peers.remove(&(network, peer));
}
}
let mut combined = peers
.iter()
.filter(|((_, id), _)| *id == peer)
.flat_map(|(_, addrs)| addrs.iter().copied())
.collect::<Vec<_>>();
combined.sort_unstable();
combined.dedup();
combined.truncate(32);
self.transport.replace_peer_addrs(peer, combined);
}
}
impl tsunagi::dataplane::transport::DataCarrier for TcpTlsCarrier {
fn protocol_id(&self) -> &str { fn protocol_id(&self) -> &str {
TCP_TLS_PROTOCOL TCP_TLS_PROTOCOL
} }
fn protocol_version(&self) -> u16 { fn protocol_version(&self) -> u16 {
TCP_TLS_VERSION 2
} }
fn on_interface_ready(&self, name: &str) {
fn options(&self) -> &'static [ProtocolOption] { *self
Self::OPTIONS .excluded_interface
.lock()
.unwrap_or_else(|p| p.into_inner()) = Some(name.into());
} }
fn local_capability(
fn local_capability_data(&self, _network: NetworkId) -> Result<Vec<u8>, PluginError> { &self,
let announcement = TcpTlsAnnouncement { _: NetworkId,
noise_public: *self.public_key.as_bytes(), ) -> Result<Option<tsunagi::dataplane::PluginCapability>, PluginError> {
port: self.transport.bound_port(), let data = postcard::to_allocvec(&(
addrs: local_ip_candidates(), self.transport.bound_port(),
}; local_ip_candidates_excluding(
postcard::to_allocvec(&announcement) self.excluded_interface
.map_err(|err| PluginError::Other(format!("encode announcement failed: {err}"))) .lock()
.unwrap_or_else(|p| p.into_inner())
.as_deref(),
)
.into_iter()
.take(32)
.collect::<Vec<_>>(),
))
.map_err(|_| PluginError::Other("cannot encode TLS addresses".into()))?;
Ok(Some(tsunagi::dataplane::PluginCapability {
protocol: TCP_TLS_PROTOCOL.into(),
version: 2,
enabled: true,
data,
}))
} }
fn on_peer_capability(
fn on_peer_capability_data(
&self, &self,
network: NetworkId, network: NetworkId,
peer: EndpointId, peer: EndpointId,
data: &[u8], cap: &tsunagi::dataplane::PluginCapability,
) -> Result<(), PluginError> { ) -> Result<(), PluginError> {
let announcement: TcpTlsAnnouncement = postcard::from_bytes(data) if !cap.enabled || cap.protocol != TCP_TLS_PROTOCOL || cap.version != 2 {
.map_err(|err| PluginError::Rejected(format!("invalid announcement: {err}")))?; return Err(PluginError::Rejected("unsupported TLS carrier".into()));
for ip in &announcement.addrs {
if is_candidate_ip(*ip) {
self.transport
.set_peer_addr(peer, std::net::SocketAddr::new(*ip, announcement.port));
}
} }
// Decode a fixed bounded array through the existing common announcement limit.
let peer_noise_pubkey = PublicKey::from(announcement.noise_public); #[derive(Deserialize)]
let cipher = Arc::new(PeerCipher::new( struct Addresses(
self.local_id, u16,
#[serde(deserialize_with = "bounded_ips")] Vec<std::net::IpAddr>,
);
let Addresses(port, ips) = postcard::from_bytes(&cap.data)
.map_err(|_| PluginError::Rejected("invalid TLS carrier addresses".into()))?;
if port == 0 {
return Err(PluginError::Rejected("invalid TLS carrier port".into()));
}
self.update_peer(
network,
peer, peer,
&self.static_secret, Some(
&peer_noise_pubkey, ips.into_iter()
)); .filter(|ip| is_candidate_ip(*ip))
.map(|ip| std::net::SocketAddr::new(ip, port))
let mut guard = match self.ciphers.write() { .collect(),
Ok(g) => g, ),
Err(p) => p.into_inner(), );
};
guard.insert((network, peer), cipher);
Ok(()) Ok(())
} }
fn on_peer_gone(&self, network: NetworkId, peer: EndpointId) {
fn on_peer_down(&self, network: NetworkId, peer: EndpointId) { self.update_peer(network, peer, None);
let mut guard = match self.ciphers.write() {
Ok(g) => g,
Err(p) => p.into_inner(),
};
guard.remove(&(network, peer));
} }
fn on_network_deactivated(&self, network: NetworkId) {
fn encrypt( let peers: Vec<_> = self
&self, .peers
network: NetworkId, .lock()
peer: EndpointId, .unwrap_or_else(|p| p.into_inner())
packet: &[u8], .keys()
) -> Result<Bytes, PluginError> { .filter(|(id, _)| *id == network)
let cipher = { .map(|(_, peer)| *peer)
let guard = match self.ciphers.read() { .collect();
Ok(g) => g, for peer in peers {
Err(p) => p.into_inner(), self.update_peer(network, peer, None);
}; }
guard.get(&(network, peer)).cloned()
};
let Some(cipher) = cipher else {
return Err(PluginError::Unavailable("no cipher for peer".into()));
};
cipher.encrypt(packet)
}
fn decrypt(
&self,
network: NetworkId,
peer: EndpointId,
payload: &[u8],
) -> Result<Bytes, PluginError> {
let cipher = {
let guard = match self.ciphers.read() {
Ok(g) => g,
Err(p) => p.into_inner(),
};
guard.get(&(network, peer)).cloned()
};
let Some(cipher) = cipher else {
return Err(PluginError::Unavailable("no cipher for peer".into()));
};
cipher.decrypt(payload)
} }
} }
fn bounded_ips<'de, D: serde::Deserializer<'de>>(
/// The complete `tcp-tls` plugin. deserializer: D,
pub type TcpTlsPlugin = GenericTunnelPlugin<TcpTlsCodec>; ) -> Result<Vec<std::net::IpAddr>, D::Error> {
struct Visitor;
#[cfg(test)] impl<'de> serde::de::Visitor<'de> for Visitor {
mod tests { type Value = Vec<std::net::IpAddr>;
use super::*; fn expecting(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
use std::net::{Ipv4Addr, Ipv6Addr}; f.write_str("at most 32 IP candidates")
}
#[test] fn visit_seq<A: serde::de::SeqAccess<'de>>(
fn test_is_candidate_interface_filters_overlay_and_bridges() { self,
let mut iface = netdev::Interface::dummy(); mut seq: A,
iface.flags = 1; // UP ) -> Result<Self::Value, A::Error> {
if seq.size_hint().is_some_and(|n| n > 32) {
// Standard Ethernet - should pass return Err(serde::de::Error::custom("too many IP candidates"));
iface.name = "eth0".to_string(); }
assert!(is_candidate_interface(&iface)); let mut ips = Vec::new();
while let Some(ip) = seq.next_element()? {
// Windows Ethernet with GUID name - should pass if ips.len() == 32 {
iface.name = "{B6149D2C-1234-5678-9ABC-DEF012345678}".to_string(); return Err(serde::de::Error::custom("too many IP candidates"));
iface.friendly_name = Some("Ethernet".to_string()); }
iface.description = Some("Intel(R) Ethernet Connection".to_string()); ips.push(ip);
assert!(is_candidate_interface(&iface)); }
Ok(ips)
// Linux tsun0 overlay - must be filtered }
iface.name = "tsun0".to_string();
iface.friendly_name = None;
iface.description = None;
assert!(!is_candidate_interface(&iface));
// Windows Wintun overlay (GUID name with tsun0 friendly name) - must be filtered!
iface.name = "{72D85D76-F1A5-42AC-BC49-2169527CDDDF}".to_string();
iface.friendly_name = Some("tsun0".to_string());
iface.description = Some("Wintun Userspace Tunnel".to_string());
assert!(!is_candidate_interface(&iface));
// Windows Wintun overlay with description containing Wintun - must be filtered!
iface.friendly_name = Some("Local Area Connection 2".to_string());
iface.description = Some("Wintun Userspace Tunnel".to_string());
assert!(!is_candidate_interface(&iface));
// Docker bridge - must be filtered
iface.name = "docker0".to_string();
iface.friendly_name = None;
iface.description = None;
assert!(!is_candidate_interface(&iface));
// Docker network bridge br-* - must be filtered
iface.name = "br-4f6d3a9b1c2e".to_string();
assert!(!is_candidate_interface(&iface));
// Virtual ethernet veth* - must be filtered
iface.name = "veth12345".to_string();
assert!(!is_candidate_interface(&iface));
// Libvirt NAT bridge virbr* - must be filtered
iface.name = "virbr0".to_string();
assert!(!is_candidate_interface(&iface));
// CNI bridge - must be filtered
iface.name = "cni0".to_string();
assert!(!is_candidate_interface(&iface));
// Hyper-V / WSL vEthernet switch on Windows - must be filtered
iface.name = "{A1B2C3D4-E5F6-7890-1234-567890ABCDEF}".to_string();
iface.friendly_name = Some("vEthernet (WSL)".to_string());
iface.description = Some("Hyper-V Virtual Ethernet Adapter".to_string());
assert!(!is_candidate_interface(&iface));
// Down interface - must be filtered
let mut down_iface = netdev::Interface::dummy();
down_iface.flags = 0; // Not UP
down_iface.name = "eth0".to_string();
assert!(!is_candidate_interface(&down_iface));
}
#[test]
fn test_is_candidate_ip() {
// Valid LAN IP
assert!(is_candidate_ip(std::net::IpAddr::V4(Ipv4Addr::new(
192, 168, 1, 117
))));
// Valid Tailscale IP
assert!(is_candidate_ip(std::net::IpAddr::V4(Ipv4Addr::new(
100, 77, 155, 120
))));
// Valid Public IP
assert!(is_candidate_ip(std::net::IpAddr::V4(Ipv4Addr::new(
1, 1, 1, 1
))));
// Valid IPv6 GUA
assert!(is_candidate_ip(std::net::IpAddr::V6(Ipv6Addr::new(
0x2a01, 0x4b00, 0xb8e3, 0x4e00, 0, 0, 0, 1
))));
// Loopback - must be rejected
assert!(!is_candidate_ip(std::net::IpAddr::V4(Ipv4Addr::new(
127, 0, 0, 1
))));
assert!(!is_candidate_ip(std::net::IpAddr::V6(Ipv6Addr::LOCALHOST)));
// Unspecified - must be rejected
assert!(!is_candidate_ip(std::net::IpAddr::V4(
Ipv4Addr::UNSPECIFIED
)));
assert!(!is_candidate_ip(std::net::IpAddr::V6(
Ipv6Addr::UNSPECIFIED
)));
// Link-local - must be rejected
assert!(!is_candidate_ip(std::net::IpAddr::V4(Ipv4Addr::new(
169, 254, 1, 1
))));
assert!(!is_candidate_ip(std::net::IpAddr::V6(Ipv6Addr::new(
0xfe80, 0, 0, 0, 0, 0x1ff, 0xfe00, 1
))));
// Broadcast - must be rejected
assert!(!is_candidate_ip(std::net::IpAddr::V4(Ipv4Addr::BROADCAST)));
}
#[test]
fn test_ip_priority_ordering() {
let mut iface_lan = netdev::Interface::dummy();
iface_lan.flags = 1;
iface_lan.gateway = Some(netdev::net::device::NetworkDevice::new());
let mut iface_vpn = netdev::Interface::dummy();
iface_vpn.flags = 1;
iface_vpn.name = "tailscale0".to_string();
let lan_ip = std::net::IpAddr::V4(Ipv4Addr::new(192, 168, 1, 117));
let tailscale_ip = std::net::IpAddr::V4(Ipv4Addr::new(100, 77, 155, 120));
let p_lan = ip_priority(&iface_lan, lan_ip);
let p_ts = ip_priority(&iface_vpn, tailscale_ip);
// LAN IP with gateway should score higher than VPN IP without gateway
assert!(p_lan > p_ts);
} }
deserializer.deserialize_seq(Visitor)
} }
crates/tsunagi-tcp-tls/src/transport.rs
+11
View File
@@ -464,6 +464,17 @@ impl TcpTlsTransport {
} }
} }
/// Replaces a bounded candidate set learned through an authenticated session.
pub fn replace_peer_addrs(&self, peer: EndpointId, addresses: Vec<SocketAddr>) {
if let Ok(mut peers) = self.peer_addresses.write() {
if addresses.is_empty() {
peers.remove(&peer);
} else {
peers.insert(peer, addresses.into_iter().take(32).collect());
}
}
}
/// Runs the accept loop for inbound TCP TLS connections. /// Runs the accept loop for inbound TCP TLS connections.
pub async fn accept_loop<F>(&self, on_inbound: F) pub async fn accept_loop<F>(&self, on_inbound: F)
where where
crates/tsunagi-tcp-tls/tests/crypto_tests.rs
-73
View File
@@ -1,73 +0,0 @@
#![allow(missing_docs)]
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use tsunagi_tcp_tls::crypto::PeerCipher;
use x25519_dalek::{PublicKey, StaticSecret};
#[test]
fn test_peer_cipher_bidirectional_roundtrip() {
let peer_a_id = iroh::SecretKey::generate().public();
let peer_b_id = iroh::SecretKey::generate().public();
let secret_a = StaticSecret::from([10u8; 32]);
let public_a = PublicKey::from(&secret_a);
let secret_b = StaticSecret::from([20u8; 32]);
let public_b = PublicKey::from(&secret_b);
let cipher_a = PeerCipher::new(peer_a_id, peer_b_id, &secret_a, &public_b);
let cipher_b = PeerCipher::new(peer_b_id, peer_a_id, &secret_b, &public_a);
// A -> B
let msg_from_a = b"hello from node A to node B over Noise E2EE";
let encrypted_by_a = cipher_a
.encrypt(msg_from_a)
.expect("encryption should succeed");
let decrypted_by_b = cipher_b
.decrypt(&encrypted_by_a)
.expect("decryption should succeed");
assert_eq!(&decrypted_by_b[..], msg_from_a);
// B -> A
let msg_from_b = b"hello back from node B to node A";
let encrypted_by_b = cipher_b
.encrypt(msg_from_b)
.expect("encryption should succeed");
let decrypted_by_a = cipher_a
.decrypt(&encrypted_by_b)
.expect("decryption should succeed");
assert_eq!(&decrypted_by_a[..], msg_from_b);
// Sequence nonces increment
let msg2 = b"packet 2";
let enc2 = cipher_a.encrypt(msg2).expect("encryption should succeed");
assert_ne!(encrypted_by_a[..8], enc2[..8], "nonces should differ");
let dec2 = cipher_b.decrypt(&enc2).expect("decryption should succeed");
assert_eq!(&dec2[..], msg2);
}
#[test]
fn test_tampered_payload_rejected() {
let peer_a_id = iroh::SecretKey::generate().public();
let peer_b_id = iroh::SecretKey::generate().public();
let secret_a = StaticSecret::from([10u8; 32]);
let public_a = PublicKey::from(&secret_a);
let secret_b = StaticSecret::from([20u8; 32]);
let public_b = PublicKey::from(&secret_b);
let cipher_a = PeerCipher::new(peer_a_id, peer_b_id, &secret_a, &public_b);
let cipher_b = PeerCipher::new(peer_b_id, peer_a_id, &secret_b, &public_a);
let encrypted = cipher_a.encrypt(b"secret payload").expect("encrypt");
// Corrupt one byte of ciphertext
let len = encrypted.len();
let mut modified = encrypted.to_vec();
modified[len - 1] ^= 0xff;
assert!(
cipher_b.decrypt(&modified).is_err(),
"corrupted ciphertext must fail AEAD verification"
);
}
crates/tsunagi-tcp-tls/tests/plugin_tests.rs
+27 -143
View File
@@ -1,149 +1,33 @@
#![allow(missing_docs)] //! Bounded TLS carrier announcements contain no end-to-end cipher selection.
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)] #![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use std::{net::IpAddr, sync::Arc};
use std::sync::Arc; use tsunagi::{
use std::time::Duration; dataplane::transport::DataCarrier,
identity::{DeviceIdentity, NetworkId},
use bytes::Bytes; };
use tokio::sync::mpsc; use tsunagi_tcp_tls::{TcpTlsCarrier, TcpTlsTransport};
use tsunagi::dataplane::transport::{InboundLink, PacketTransport};
use tsunagi::dataplane::{IpPlugin, PacketSink, PluginContext};
use tsunagi::identity::{DeviceIdentity, NetworkId};
use tsunagi_tcp_tls::{TcpTlsCodec, TcpTlsPlugin, TcpTlsTransport};
struct TestSink {
delivered: mpsc::Sender<(NetworkId, iroh::EndpointId, Bytes)>,
}
impl PacketSink for TestSink {
fn deliver(
&self,
network: NetworkId,
from: iroh::EndpointId,
packet: Bytes,
) -> tsunagi::BoxFuture<'_, ()> {
let delivered = self.delivered.clone();
Box::pin(async move {
let _ = delivered.send((network, from, packet)).await;
})
}
}
#[tokio::test] #[tokio::test]
async fn test_tcp_tls_plugin_end_to_end() { async fn tls_capabilities_bound_addresses_and_reject_invalid_ports_and_versions() {
let identity_a = DeviceIdentity::generate(); let identity = DeviceIdentity::generate();
let peer_a = identity_a.endpoint_id(); let peer = DeviceIdentity::generate().endpoint_id();
let transport = Arc::new(
let identity_b = DeviceIdentity::generate(); TcpTlsTransport::bind(&identity, Some(0), None)
let peer_b = identity_b.endpoint_id();
let network = NetworkId::from_bytes([77u8; 32]);
// Transports
let transport_b = Arc::new(
TcpTlsTransport::bind(&identity_b, Some(0), None)
.await .await
.expect("bind B"), .unwrap(),
); );
let port_b = transport_b.bound_port(); let carrier = TcpTlsCarrier::new(transport);
let network = NetworkId::from_bytes([7; 32]);
let transport_a = Arc::new( let mut cap = carrier.local_capability(network).unwrap().unwrap();
TcpTlsTransport::bind(&identity_a, Some(0), None) assert_eq!(cap.protocol, "tcp-tls");
.await let addresses: Vec<IpAddr> = vec!["192.0.2.1".parse().unwrap(); 33];
.expect("bind A"), cap.data = postcard::to_allocvec(&(443u16, addresses)).unwrap();
); assert!(carrier.on_peer_capability(network, peer, &cap).is_err());
cap.data = postcard::to_allocvec(&(0u16, Vec::<IpAddr>::new())).unwrap();
// Codecs and plugins assert!(carrier.on_peer_capability(network, peer, &cap).is_err());
let codec_a = TcpTlsCodec::new(peer_a, Arc::clone(&transport_a)); cap.data =
let plugin_a = Arc::new(TcpTlsPlugin::new(codec_a)); postcard::to_allocvec(&(443u16, vec!["192.0.2.1".parse::<IpAddr>().unwrap()])).unwrap();
carrier.on_peer_capability(network, peer, &cap).unwrap();
let codec_b = TcpTlsCodec::new(peer_b, Arc::clone(&transport_b)); cap.version += 1;
let plugin_b = Arc::new(TcpTlsPlugin::new(codec_b)); assert!(carrier.on_peer_capability(network, peer, &cap).is_err());
// Attach sinks
let (sink_a_tx, mut sink_a_rx) = mpsc::channel(10);
plugin_a.attach(PluginContext::with_sink(Arc::new(TestSink {
delivered: sink_a_tx,
})));
let (sink_b_tx, mut sink_b_rx) = mpsc::channel(10);
plugin_b.attach(PluginContext::with_sink(Arc::new(TestSink {
delivered: sink_b_tx,
})));
// Capability exchange
let cap_a = plugin_a
.local_capability(network)
.expect("cap A")
.expect("some cap A");
let cap_b = plugin_b
.local_capability(network)
.expect("cap B")
.expect("some cap B");
plugin_a
.on_peer_capability(network, peer_b, &cap_b)
.expect("A on cap B");
plugin_b
.on_peer_capability(network, peer_a, &cap_a)
.expect("B on cap A");
// Spawn accept loop on B
let (inbound_tx, mut inbound_rx) = mpsc::channel::<InboundLink>(1);
let transport_b_clone = Arc::clone(&transport_b);
tokio::spawn(async move {
transport_b_clone
.accept_loop(move |inbound| {
let _ = inbound_tx.try_send(inbound);
})
.await;
});
// Make sure A knows B's loopback address
transport_a.set_peer_addr(peer_b, format!("127.0.0.1:{port_b}").parse().unwrap());
// Connect link from A to B
let link_a = transport_a
.open(network, peer_b, "tcp-tls")
.await
.expect("open link from A");
let inbound_b = tokio::time::timeout(Duration::from_secs(5), inbound_rx.recv())
.await
.expect("recv timeout")
.expect("recv inbound");
let link_b = inbound_b.link;
// Install links into plugins
plugin_a.on_peer_link(network, peer_b, link_a);
plugin_b.on_peer_link(network, peer_a, link_b);
// Carry packet from A to B
let ip_packet_a = Bytes::from_static(b"4500003c000100004011e0e8c0a80101c0a8010200000000");
let carried_a = plugin_a.carry(network, peer_b, ip_packet_a.clone());
assert!(carried_a, "plugin A should carry packet");
let (net_received, from_peer, delivered_pkt) =
tokio::time::timeout(Duration::from_secs(3), sink_b_rx.recv())
.await
.expect("timeout waiting for delivery on B")
.expect("packet on B");
assert_eq!(net_received, network);
assert_eq!(from_peer, peer_a);
assert_eq!(delivered_pkt, ip_packet_a);
// Carry reply packet from B to A
let ip_packet_b = Bytes::from_static(b"reply-packet-from-b");
let carried_b = plugin_b.carry(network, peer_a, ip_packet_b.clone());
assert!(carried_b, "plugin B should carry packet");
let (net_received, from_peer, delivered_pkt) =
tokio::time::timeout(Duration::from_secs(3), sink_a_rx.recv())
.await
.expect("timeout waiting for delivery on A")
.expect("packet on A");
assert_eq!(net_received, network);
assert_eq!(from_peer, peer_b);
assert_eq!(delivered_pkt, ip_packet_b);
} }
crates/tsunagi-tray/src/agent.rs
+41
View File
@@ -21,6 +21,13 @@ const POLL_INTERVAL: Duration = Duration::from_secs(2);
/// A request from the UI to the agent. /// A request from the UI to the agent.
pub(crate) enum Command { pub(crate) enum Command {
SaveVip {
token: String,
network_id: String,
config: tsunagi::vip::VipConfig,
expected: Option<String>,
delete: bool,
},
/// Retire the exact offline-node claim selected in the devices window. /// Retire the exact offline-node claim selected in the devices window.
ForgetNode { ForgetNode {
network_id: String, network_id: String,
@@ -55,8 +62,14 @@ pub(crate) enum Command {
} }
/// The latest the worker knows, read by the UI each frame. /// The latest the worker knows, read by the UI each frame.
#[derive(Clone)]
pub(crate) struct VipSaveResult {
pub token: String,
pub result: Result<Vec<tsunagi::vip::VipStatus>, String>,
}
#[derive(Clone, Default)] #[derive(Clone, Default)]
pub(crate) struct Snapshot { pub(crate) struct Snapshot {
pub vip_result: Option<VipSaveResult>,
/// The last status, or the reason it could not be read (agent down, no /// The last status, or the reason it could not be read (agent down, no
/// permission on the socket, …). /// permission on the socket, …).
pub status: Option<Result<StatusReport, String>>, pub status: Option<Result<StatusReport, String>>,
@@ -84,6 +97,18 @@ pub(crate) struct AgentClient {
} }
impl AgentClient { impl AgentClient {
#[cfg(test)]
pub(crate) fn test_client() -> (Self, mpsc::UnboundedReceiver<Command>) {
let (tx, rx) = mpsc::unbounded_channel();
(
Self {
tx,
shared: Arc::new(Mutex::new(Snapshot::default())),
pending_copy: Arc::new(Mutex::new(None)),
},
rx,
)
}
/// Starts the worker on `runtime`, reporting to `ctx` (for repaints). /// Starts the worker on `runtime`, reporting to `ctx` (for repaints).
pub(crate) fn spawn(runtime: &tokio::runtime::Handle, ctx: egui::Context) -> Self { pub(crate) fn spawn(runtime: &tokio::runtime::Handle, ctx: egui::Context) -> Self {
let (tx, rx) = mpsc::unbounded_channel(); let (tx, rx) = mpsc::unbounded_channel();
@@ -198,6 +223,22 @@ async fn run(
ctx.request_repaint(); ctx.request_repaint();
let outcome: Result<String, String> = match command { let outcome: Result<String, String> = match command {
Command::SaveVip {
token,
network_id,
config,
expected,
delete,
} => {
let result = ipc::save_vip(socket, &network_id, config, expected, delete)
.await
.map_err(|e| e.to_string());
lock(shared).vip_result = Some(VipSaveResult {
token,
result: result.clone(),
});
result.map(|_| "VIP configuration saved".into())
}
Command::ForgetNode { Command::ForgetNode {
network_id, network_id,
peer, peer,
crates/tsunagi-tray/src/app.rs
+70 -3
View File
@@ -15,6 +15,12 @@ use crate::devices;
use crate::stats::Traffic; use crate::stats::Traffic;
use crate::ui::{self, UiState}; use crate::ui::{self, UiState};
fn app_fonts() -> egui::FontDefinitions {
let mut fonts = egui::FontDefinitions::default();
egui_phosphor::add_to_fonts(&mut fonts, egui_phosphor::Variant::Regular);
fonts
}
/// The window's state. /// The window's state.
pub(crate) struct App { pub(crate) struct App {
agent: AgentClient, agent: AgentClient,
@@ -34,9 +40,7 @@ impl App {
let ctx = cc.egui_ctx.clone(); let ctx = cc.egui_ctx.clone();
// Add the Phosphor icon glyphs to the font set so the icon buttons // Add the Phosphor icon glyphs to the font set so the icon buttons
// render instead of showing as tofu. // render instead of showing as tofu.
let mut fonts = egui::FontDefinitions::default(); ctx.set_fonts(app_fonts());
egui_phosphor::add_to_fonts(&mut fonts, egui_phosphor::Variant::Regular);
ctx.set_fonts(fonts);
let agent = AgentClient::spawn(&runtime, ctx); let agent = AgentClient::spawn(&runtime, ctx);
Ok(Self { Ok(Self {
@@ -60,6 +64,7 @@ impl eframe::App for App {
ui::draw(ctx, &self.agent, &mut self.state, &snapshot, &self.traffic); ui::draw(ctx, &self.agent, &mut self.state, &snapshot, &self.traffic);
self.show_devices_windows(ctx, &snapshot); self.show_devices_windows(ctx, &snapshot);
self.show_vip_windows(ctx, &snapshot);
self.show_about_window(ctx, &snapshot); self.show_about_window(ctx, &snapshot);
// A secret the worker fetched for the clipboard: copy it here, on the // A secret the worker fetched for the clipboard: copy it here, on the
@@ -74,6 +79,51 @@ impl eframe::App for App {
} }
impl App { impl App {
fn show_vip_windows(&mut self, ctx: &egui::Context, snapshot: &crate::agent::Snapshot) {
let Some(Ok(report)) = &snapshot.status else {
return;
};
let ids: Vec<_> = self.state.vip_windows.keys().cloned().collect();
for id in ids {
let Some(network) = report.networks.iter().find(|n| n.network_id == id) else {
self.state.vip_windows.remove(&id);
continue;
};
let Some(state) = self.state.vip_windows.get_mut(&id) else {
continue;
};
let builder = egui::ViewportBuilder::default()
.with_title(format!("tsunagi · VIP · {}", network.name))
.with_app_id(crate::icon::APP_ID)
.with_icon(crate::icon::window())
.with_inner_size([1280., 860.])
.with_min_inner_size([650., 400.]);
let keep = ctx.show_viewport_immediate(
egui::ViewportId::from_hash_of(("vip", &id)),
builder,
|vctx, _| {
let close = crate::vip::show(vctx, &self.agent, state, network, snapshot);
if close {
return false;
}
if vctx.input(|i| i.viewport().close_requested()) {
if state.dirty() {
state.request_close();
vctx.send_viewport_cmd(egui::ViewportCommand::CancelClose);
true
} else {
false
}
} else {
true
}
},
);
if !keep {
self.state.vip_windows.remove(&id);
}
}
}
/// Renders the About window while it is open. /// Renders the About window while it is open.
fn show_about_window(&mut self, ctx: &egui::Context, snapshot: &crate::agent::Snapshot) { fn show_about_window(&mut self, ctx: &egui::Context, snapshot: &crate::agent::Snapshot) {
if !self.state.about_open { if !self.state.about_open {
@@ -143,3 +193,20 @@ impl App {
} }
} }
} }
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn route_icons_have_an_embedded_font_in_the_app() {
let fonts = app_fonts();
assert!(fonts.font_data.contains_key("phosphor"));
assert!(
fonts
.families
.get(&egui::FontFamily::Proportional)
.is_some_and(|family| family.iter().any(|name| name == "phosphor"))
);
}
}
crates/tsunagi-tray/src/devices.rs
+627 -240
View File
File diff suppressed because it is too large. Load diff
crates/tsunagi-tray/src/exit.rs
+1
View File
@@ -109,6 +109,7 @@ mod tests {
fn network(exit: ExitReport) -> NetworkReport { fn network(exit: ExitReport) -> NetworkReport {
NetworkReport { NetworkReport {
peers: vec![PeerReport { peers: vec![PeerReport {
data_error: None,
endpoint_id: "abcdef0123456789".into(), endpoint_id: "abcdef0123456789".into(),
hostname: Some("music".into()), hostname: Some("music".into()),
exit_node: true, exit_node: true,
crates/tsunagi-tray/src/format.rs
-13
View File
@@ -24,19 +24,6 @@ pub(crate) fn short(id: &str) -> String {
} }
} }
/// A duration in seconds as a compact age (`5s`, `12m`, `3h 20m`, `2d 4h`).
pub(crate) fn age(secs: u64) -> String {
if secs < 60 {
format!("{secs}s")
} else if secs < 3_600 {
format!("{}m", secs / 60)
} else if secs < 86_400 {
format!("{}h {}m", secs / 3_600, (secs % 3_600) / 60)
} else {
format!("{}d {}h", secs / 86_400, (secs % 86_400) / 3_600)
}
}
/// A byte count in binary units. /// A byte count in binary units.
pub(crate) fn bytes(n: f64) -> String { pub(crate) fn bytes(n: f64) -> String {
const UNITS: [&str; 5] = ["B", "kB", "MB", "GB", "TB"]; const UNITS: [&str; 5] = ["B", "kB", "MB", "GB", "TB"];
crates/tsunagi-tray/src/main.rs
+3
View File
@@ -17,8 +17,11 @@ mod exit;
mod format; mod format;
mod icon; mod icon;
mod stats; mod stats;
mod table;
mod tray; mod tray;
mod ui; mod ui;
mod vip;
mod vip_status;
mod watch; mod watch;
use eframe::egui; use eframe::egui;
crates/tsunagi-tray/src/stats.rs
+193 -22
View File
@@ -6,10 +6,10 @@
//! key: a network total, and each peer within it. Both packets and bytes are //! key: a network total, and each peer within it. Both packets and bytes are
//! tracked so the UI can switch between them. //! tracked so the UI can switch between them.
use std::collections::{HashMap, VecDeque}; use std::collections::{HashMap, HashSet, VecDeque};
use std::time::Instant; use std::time::Instant;
use tsunagi::ipc::StatusReport; use tsunagi::ipc::{OverlayPeerReport, StatusReport};
/// How many rate samples to keep for a sparkline. /// How many rate samples to keep for a sparkline.
const HISTORY: usize = 60; const HISTORY: usize = 60;
@@ -26,7 +26,7 @@ pub(crate) enum Unit {
/// One series' rolling packet and byte rates. /// One series' rolling packet and byte rates.
#[derive(Default)] #[derive(Default)]
pub(crate) struct Series { pub(crate) struct Series {
last: Option<(Instant, u64, u64)>, last: Option<(Instant, Counters)>,
packets_rate: f32, packets_rate: f32,
bytes_rate: f32, bytes_rate: f32,
packets_history: VecDeque<f32>, packets_history: VecDeque<f32>,
@@ -34,17 +34,52 @@ pub(crate) struct Series {
} }
impl Series { impl Series {
fn sample(&mut self, now: Instant, packets: u64, bytes: u64) { /// Samples a single cumulative counter pair (VIP histories own their lifetime).
if let Some((then, prev_packets, prev_bytes)) = self.last { pub(crate) fn sample(&mut self, now: Instant, packets: u64, bytes: u64) {
self.sample_peer(
now,
Counters {
epoch: 0,
tx_packets: packets,
tx_bytes: bytes,
rx_packets: 0,
rx_bytes: 0,
},
);
}
fn sample_peer(&mut self, now: Instant, counters: Counters) {
if let Some((then, previous)) = self.last {
let dt = now.saturating_duration_since(then).as_secs_f32(); let dt = now.saturating_duration_since(then).as_secs_f32();
if dt > 0.0 { // Ignore duplicate/out-of-order observations without changing the baseline.
self.packets_rate = packets.saturating_sub(prev_packets) as f32 / dt; if dt <= 0.0 {
self.bytes_rate = bytes.saturating_sub(prev_bytes) as f32 / dt; return;
push(&mut self.packets_history, self.packets_rate); }
push(&mut self.bytes_history, self.bytes_rate); if counters.epoch == previous.epoch
&& counters.tx_packets >= previous.tx_packets
&& counters.rx_packets >= previous.rx_packets
&& counters.tx_bytes >= previous.tx_bytes
&& counters.rx_bytes >= previous.rx_bytes
{
self.record(
(counters.tx_packets - previous.tx_packets) as f32 / dt
+ (counters.rx_packets - previous.rx_packets) as f32 / dt,
(counters.tx_bytes - previous.tx_bytes) as f32 / dt
+ (counters.rx_bytes - previous.rx_bytes) as f32 / dt,
);
} else {
// A counter reset is a new baseline, never traffic.
*self = Self::default();
} }
} }
self.last = Some((now, packets, bytes)); self.last = Some((now, counters));
}
fn record(&mut self, packets: f32, bytes: f32) {
self.packets_rate = packets;
self.bytes_rate = bytes;
push(&mut self.packets_history, packets);
push(&mut self.bytes_history, bytes);
} }
/// The current rate in the chosen unit. /// The current rate in the chosen unit.
@@ -71,6 +106,27 @@ fn push(history: &mut VecDeque<f32>, value: f32) {
} }
} }
#[derive(Clone, Copy)]
struct Counters {
epoch: u64,
tx_packets: u64,
rx_packets: u64,
tx_bytes: u64,
rx_bytes: u64,
}
impl From<&OverlayPeerReport> for Counters {
fn from(peer: &OverlayPeerReport) -> Self {
Self {
epoch: peer.counter_epoch,
tx_packets: peer.tx_packets,
rx_packets: peer.rx_packets,
tx_bytes: peer.tx_bytes,
rx_bytes: peer.rx_bytes,
}
}
}
/// All traffic series, keyed by a stable string. /// All traffic series, keyed by a stable string.
#[derive(Default)] #[derive(Default)]
pub(crate) struct Traffic { pub(crate) struct Traffic {
@@ -80,26 +136,32 @@ pub(crate) struct Traffic {
impl Traffic { impl Traffic {
/// Folds a fresh snapshot into the rates. Call once per new generation. /// Folds a fresh snapshot into the rates. Call once per new generation.
pub(crate) fn observe(&mut self, report: &StatusReport, now: Instant) { pub(crate) fn observe(&mut self, report: &StatusReport, now: Instant) {
let mut active = HashSet::new();
for network in &report.networks { for network in &report.networks {
let Some(overlay) = &network.overlay else { let Some(overlay) = &network.overlay else {
continue; continue;
}; };
let (mut net_packets, mut net_bytes) = (0u64, 0u64); let (mut packets_rate, mut bytes_rate) = (0.0, 0.0);
for peer in &overlay.peers { for peer in &overlay.peers {
let packets = peer.tx_packets.saturating_add(peer.rx_packets); let key = peer_key(&network.network_id, &peer.public_key);
let bytes = peer.tx_bytes.saturating_add(peer.rx_bytes); // Duplicate rows are not additional traffic.
net_packets = net_packets.saturating_add(packets); if !active.insert(key.clone()) {
net_bytes = net_bytes.saturating_add(bytes); continue;
self.series }
.entry(peer_key(&network.network_id, &peer.public_key)) let series = self.series.entry(key).or_default();
.or_default() series.sample_peer(now, peer.into());
.sample(now, packets, bytes); packets_rate += series.rate(Unit::Packets);
bytes_rate += series.rate(Unit::Bytes);
} }
// Sum rates, not cumulative counters of a changing set of peers.
let key = net_key(&network.network_id);
active.insert(key.clone());
self.series self.series
.entry(net_key(&network.network_id)) .entry(key)
.or_default() .or_default()
.sample(now, net_packets, net_bytes); .record(packets_rate, bytes_rate);
} }
self.series.retain(|key, _| active.contains(key));
} }
/// The network-total series, if sampled. /// The network-total series, if sampled.
@@ -120,3 +182,112 @@ fn net_key(network_id: &str) -> String {
fn peer_key(network_id: &str, public_key: &str) -> String { fn peer_key(network_id: &str, public_key: &str) -> String {
format!("peer:{network_id}:{public_key}") format!("peer:{network_id}:{public_key}")
} }
#[cfg(test)]
mod tests {
use super::*;
use std::time::Duration;
use tsunagi::ipc::{NetworkReport, OverlayPeerReport, OverlayReport};
fn peer(key: &str, bytes: u64) -> OverlayPeerReport {
OverlayPeerReport {
public_key: key.into(),
tx_bytes: bytes,
..Default::default()
}
}
fn report(peers: Vec<OverlayPeerReport>) -> StatusReport {
StatusReport {
networks: vec![NetworkReport {
network_id: "network".into(),
overlay: Some(OverlayReport {
peers,
..Default::default()
}),
..Default::default()
}],
..Default::default()
}
}
#[test]
fn joining_peer_does_not_add_its_past_traffic_to_the_network_rate() {
let mut traffic = Traffic::default();
let now = Instant::now();
traffic.observe(&report(vec![peer("a", 100)]), now);
traffic.observe(
&report(vec![peer("a", 120), peer("b", 100_000_000_000)]),
now + Duration::from_secs(1),
);
assert_eq!(
traffic.network("network").map(|s| s.rate(Unit::Bytes)),
Some(20.0)
);
traffic.observe(
&report(vec![peer("a", 150), peer("b", 100_000_000_040)]),
now + Duration::from_secs(2),
);
assert_eq!(
traffic.network("network").map(|s| s.rate(Unit::Bytes)),
Some(70.0)
);
}
#[test]
fn departed_peers_and_removed_networks_do_not_keep_stale_rates() {
let mut traffic = Traffic::default();
let now = Instant::now();
traffic.observe(&report(vec![peer("a", 100)]), now);
traffic.observe(&report(vec![peer("a", 140)]), now + Duration::from_secs(1));
traffic.observe(&report(vec![]), now + Duration::from_secs(2));
assert!(traffic.peer("network", "a").is_none());
assert_eq!(
traffic.network("network").map(|s| s.rate(Unit::Bytes)),
Some(0.0)
);
traffic.observe(&StatusReport::default(), now + Duration::from_secs(3));
assert!(traffic.network("network").is_none());
}
#[test]
fn restarted_counters_are_a_new_baseline_even_if_they_already_exceed_the_old_value() {
let mut traffic = Traffic::default();
let now = Instant::now();
let mut old = peer("a", 100);
old.counter_epoch = 1;
traffic.observe(&report(vec![old]), now);
let mut restarted = peer("a", 100_000_000_000);
restarted.counter_epoch = 2;
traffic.observe(
&report(vec![restarted.clone()]),
now + Duration::from_secs(1),
);
assert_eq!(
traffic.peer("network", "a").map(|s| s.rate(Unit::Bytes)),
Some(0.0)
);
restarted.tx_bytes += 80;
traffic.observe(&report(vec![restarted]), now + Duration::from_secs(2));
assert_eq!(
traffic.network("network").map(|s| s.rate(Unit::Bytes)),
Some(80.0)
);
}
#[test]
fn reset_in_one_direction_cannot_be_hidden_by_growth_in_the_other() {
let mut traffic = Traffic::default();
let now = Instant::now();
let mut old = peer("a", 100);
old.rx_bytes = 1000;
traffic.observe(&report(vec![old]), now);
let mut reset = peer("a", 50);
reset.rx_bytes = 10_000;
traffic.observe(&report(vec![reset]), now + Duration::from_secs(1));
assert_eq!(
traffic.network("network").map(|s| s.rate(Unit::Bytes)),
Some(0.0)
);
}
}
crates/tsunagi-tray/src/table.rs
+88
View File
@@ -0,0 +1,88 @@
//! Fixed row geometry for dense live tables. Children cannot resize a row or
//! paint into its neighbours; the enclosing scroll area owns overflow.
use eframe::egui;
pub(crate) const GAP: f32 = 10.0;
pub(crate) fn row(
ui: &mut egui::Ui,
id: impl std::hash::Hash,
widths: &[f32],
stripe: bool,
mut cell: impl FnMut(usize, &mut egui::Ui),
) {
let font = egui::FontSelection::Default.resolve(ui.style());
let text_height = ui.fonts(|fonts| fonts.row_height(&font));
let height = ui.spacing().interact_size.y.max(text_height) + 10.0;
let width = widths.iter().sum::<f32>() + GAP * widths.len().saturating_sub(1) as f32;
let (rect, _) = ui.allocate_exact_size(egui::vec2(width, height), egui::Sense::hover());
if stripe {
ui.painter()
.rect_filled(rect, 2.0, ui.visuals().faint_bg_color);
}
let mut x = rect.left();
for (column, width) in widths.iter().enumerate() {
let bounds = egui::Rect::from_min_size(
egui::pos2(x, rect.top() + 3.0),
egui::vec2(*width, height - 6.0),
);
let mut child = ui.new_child(
egui::UiBuilder::new()
.id_salt((&id, column))
.max_rect(bounds)
.layout(egui::Layout::left_to_right(egui::Align::Center)),
);
child.set_clip_rect(ui.clip_rect().intersect(bounds));
child.style_mut().wrap_mode = Some(egui::TextWrapMode::Truncate);
cell(column, &mut child);
x += width + GAP;
}
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
#[test]
fn oversized_content_is_clipped_to_its_cell_at_every_scale() {
for size in [13.0, 20.0, 28.0] {
let ctx = egui::Context::default();
ctx.style_mut(|style| {
style.override_font_id = Some(egui::FontId::proportional(size));
});
let output = ctx.run(egui::RawInput::default(), |ctx| {
egui::CentralPanel::default().show(ctx, |ui| {
for index in 0..4 {
row(ui, index, &[90.0, 120.0], index % 2 == 0, |column, ui| {
if column == 0 {
ui.horizontal(|ui| {
ui.label("very long device name that cannot fit");
ui.button("Forget").on_hover_text("Forget");
});
} else {
ui.label(format!("row {index}"));
}
});
}
});
});
let mut previous_bottom = 0.0;
for index in 0..4 {
let shape = output.shapes.iter().find(|shape| matches!(&shape.shape, egui::Shape::Text(t) if t.galley.text() == format!("row {index}"))).unwrap();
let egui::Shape::Text(text) = &shape.shape else {
unreachable!()
};
assert!(text.visual_bounding_rect().top() >= previous_bottom);
assert!(
shape.clip_rect.contains_rect(text.visual_bounding_rect()),
"font {size}: clip {:?}, text {:?}",
shape.clip_rect,
text.visual_bounding_rect()
);
assert!(shape.clip_rect.left() >= 108.0);
previous_bottom = shape.clip_rect.bottom();
}
}
}
}
crates/tsunagi-tray/src/ui.rs
+4 -1
View File
@@ -41,6 +41,7 @@ pub(crate) struct UiState {
pub(crate) host_edit: Option<String>, pub(crate) host_edit: Option<String>,
/// Network ids whose devices window is open. /// Network ids whose devices window is open.
pub(crate) open_devices: BTreeSet<String>, pub(crate) open_devices: BTreeSet<String>,
pub(crate) vip_windows: std::collections::BTreeMap<String, crate::vip::Window>,
/// Whether the About window is open. /// Whether the About window is open.
pub(crate) about_open: bool, pub(crate) about_open: bool,
/// An exit-node choice waiting for the user to confirm it. /// An exit-node choice waiting for the user to confirm it.
@@ -56,6 +57,7 @@ impl Default for UiState {
unit: Unit::Packets, unit: Unit::Packets,
host_edit: None, host_edit: None,
open_devices: BTreeSet::new(), open_devices: BTreeSet::new(),
vip_windows: Default::default(),
about_open: false, about_open: false,
exit_confirm: None, exit_confirm: None,
forget_confirm: None, forget_confirm: None,
@@ -211,6 +213,7 @@ fn draw_network(
egui::Frame::group(ui.style()).show(ui, |ui| { egui::Frame::group(ui.style()).show(ui, |ui| {
ui.horizontal(|ui| { ui.horizontal(|ui| {
format::copy_field(ui, network.name.as_str(), &network.name); format::copy_field(ui, network.name.as_str(), &network.name);
if ui.button("VIP").clicked(){state.vip_windows.entry(network.network_id.clone()).or_default();}
ui.with_layout(egui::Layout::right_to_left(egui::Align::Center), |ui| { ui.with_layout(egui::Layout::right_to_left(egui::Align::Center), |ui| {
let leave = let leave =
egui::Button::new(egui::RichText::new("Leave").color(egui::Color32::WHITE)) egui::Button::new(egui::RichText::new("Leave").color(egui::Color32::WHITE))
@@ -361,7 +364,7 @@ fn draw_protocols(
} else { } else {
network.protocols.join(", ") network.protocols.join(", ")
}; };
egui::CollapsingHeader::new(format!("protocols {summary}")) egui::CollapsingHeader::new(format!("data transports {summary}"))
.id_salt(("protocols", &network.network_id)) .id_salt(("protocols", &network.network_id))
.show(ui, |ui| { .show(ui, |ui| {
ui.weak("Preference order; changes apply immediately and are saved."); ui.weak("Preference order; changes apply immediately and are saved.");
crates/tsunagi-tray/src/vip.rs
+402
View File
@@ -0,0 +1,402 @@
//! Per-network VIP editor. Drafts never issue commands until Save is pressed.
use crate::agent::{AgentClient, Command, Snapshot};
use crate::vip_status;
use eframe::egui;
use tsunagi::ipc::NetworkReport;
use tsunagi::vip::{Backend, Listener, Protocol, VipConfig};
#[derive(Clone)]
struct Draft {
config: VipConfig,
original: VipConfig,
revision: Option<String>,
delete: bool,
}
impl Draft {
fn new(config: VipConfig, revision: Option<String>) -> Self {
Self {
original: config.clone(),
config,
revision,
delete: false,
}
}
fn dirty(&self) -> bool {
self.revision.is_none() || self.delete || self.config != self.original
}
}
#[derive(Default)]
pub(crate) struct Window {
draft: Option<Draft>,
pending: Option<String>,
error: Option<String>,
close_question: bool,
close_after_save: bool,
live: vip_status::View,
configure: bool,
generation: u64,
acknowledged_generation: Option<u64>,
}
impl Window {
pub fn request_close(&mut self) {
self.close_question = true;
self.close_after_save = true;
}
pub fn dirty(&self) -> bool {
self.draft.as_ref().is_some_and(Draft::dirty) || self.pending.is_some()
}
}
fn peer_name<'a>(n: &'a NetworkReport, id: &'a str) -> &'a str {
n.members
.iter()
.find(|m| m.endpoint_id == id)
.and_then(|m| m.hostname.as_deref())
.unwrap_or(id)
}
pub(crate) fn show(
ctx: &egui::Context,
agent: &AgentClient,
state: &mut Window,
n: &NetworkReport,
snapshot: &Snapshot,
) -> bool {
if state.generation != snapshot.generation {
state.generation = snapshot.generation;
if let Some(at) = snapshot.at {
state.live.observe(n, at);
}
}
if let Some(result) = &snapshot.vip_result
&& state.pending.as_deref() == Some(result.token.as_str())
{
state.pending = None;
state.acknowledged_generation = Some(snapshot.generation);
match &result.result {
Ok(vips) => {
if let Some(draft) = &state.draft {
state.draft = vips
.iter()
.find(|v| v.config.id == draft.config.id)
.map(|v| Draft::new(v.config.clone(), Some(v.revision.clone())));
}
state.error = None;
}
Err(error) => {
state.error = Some(error.clone());
state.close_after_save = false;
}
}
}
let mut close = state.close_after_save && !state.dirty();
let mut discard_new = false;
egui::CentralPanel::default().show(ctx, |ui| {
ui.heading(format!("VIP · {}", n.name));
ui.weak("Configuration is shared with every member. Changes take effect only after Save.");
if state.close_question {
ui.horizontal(|ui| {
ui.label("Unsaved changes.");
if ui
.add_enabled(
state.pending.is_none(),
egui::Button::new("Discard and close"),
)
.clicked()
{
state.draft = None;
state.close_question = false;
close = true;
}
if ui.button("Keep editing").clicked() {
state.close_question = false;
state.close_after_save = false;
}
});
}
if let Some(error) = &state.error {
ui.colored_label(egui::Color32::LIGHT_RED, error);
}
ui.separator();
ui.horizontal_wrapped(|ui| {
for vip in &n.vips {
let selected = state
.draft
.as_ref()
.is_some_and(|d| d.config.id == vip.config.id);
if ui.selectable_label(selected, &vip.config.name).clicked() {
if state.dirty() {
state.error = Some(
"Save or discard the current draft before selecting another VIP."
.into(),
);
} else {
state.draft =
Some(Draft::new(vip.config.clone(), Some(vip.revision.clone())));
state.error = None;
}
}
}
if ui.button("+ New VIP").clicked() {
if state.dirty() {
state.error = Some("Save or discard the current draft first.".into());
} else {
state.draft = Some(Draft::new(VipConfig::new(""), None));
state.configure = true;
state.error = None;
}
}
});
let Some(draft) = &mut state.draft else {
ui.label("Select a VIP or create one.");
return;
};
let current = n.vips.iter().find(|v| v.config.id == draft.config.id);
if state.pending.is_none()
&& state.acknowledged_generation != Some(snapshot.generation)
&& !draft.dirty()
&& let Some(current) = current
&& draft.revision.as_ref() != Some(&current.revision)
{
*draft = Draft::new(current.config.clone(), Some(current.revision.clone()));
}
let changed =
current.is_some_and(|v| draft.revision.as_ref().is_some_and(|r| r != &v.revision));
if changed {
ui.colored_label(
egui::Color32::YELLOW,
"This VIP changed on the network. Reload before saving; your draft is retained.",
);
}
if draft.revision.is_some() && current.is_none() {
ui.colored_label(
egui::Color32::YELLOW,
"This VIP was removed on the network. The draft cannot overwrite its deletion.",
);
}
let busy = state.pending.is_some();
ui.horizontal(|ui| {
let valid = draft.config.validate(&tsunagi::Limits::default());
if !n.active {
ui.weak("Start the network to save changes.");
}
if ui
.add_enabled(
n.active
&& !busy
&& draft.dirty()
&& !changed
&& (valid.is_ok() || draft.delete),
egui::Button::new("Save"),
)
.clicked()
{
let token = VipConfig::new("").id;
state.pending = Some(token.clone());
agent.send(Command::SaveVip {
token,
network_id: n.network_id.clone(),
config: draft.config.clone(),
expected: draft.revision.clone(),
delete: draft.delete,
});
}
if ui
.add_enabled(!busy, egui::Button::new("Discard / reload"))
.clicked()
{
if let Some(v) = current {
*draft = Draft::new(v.config.clone(), Some(v.revision.clone()));
} else {
discard_new = true;
}
state.error = None;
}
if busy {
ui.spinner();
ui.label("Saving…");
} else if draft.dirty() {
ui.weak("Unsaved changes");
}
if let Err(error) = valid {
ui.colored_label(egui::Color32::LIGHT_RED, error.to_string());
}
});
ui.separator();
ui.horizontal(|ui| {
ui.selectable_value(&mut state.configure, false, "Overview");
ui.selectable_value(&mut state.configure, true, "Configuration");
});
egui::ScrollArea::vertical()
.auto_shrink([false, false])
.show(ui, |ui| {
if state.configure {
ui.add_enabled_ui(!busy, |ui| {
ui.horizontal(|ui| {
ui.label("Name");
ui.add(
egui::TextEdit::singleline(&mut draft.config.name)
.id(egui::Id::new(("vip-name", &draft.config.id))),
);
ui.weak(format!(".{}", n.name));
});
if draft.revision.is_some() {
ui.checkbox(&mut draft.delete, "Delete this VIP when saved");
}
if !draft.delete {
edit_listeners(ui, &mut draft.config, n);
}
});
} else if let Some(current) = current {
vip_status::show(ui, current, n, &mut state.live);
} else {
ui.weak("Save this VIP to see its live status.");
}
});
});
if discard_new {
state.draft = None;
state.error = None;
}
close
}
fn edit_listeners(ui: &mut egui::Ui, config: &mut VipConfig, n: &NetworkReport) {
let mut remove = None;
for (index, l) in config.listeners.iter_mut().enumerate() {
ui.push_id(index,|ui|{egui::Frame::group(ui.style()).show(ui,|ui|{
ui.horizontal(|ui|{egui::ComboBox::from_id_salt("protocol").selected_text(l.protocol.to_string()).show_ui(ui,|ui|{ui.selectable_value(&mut l.protocol,Protocol::Tcp,"TCP");ui.selectable_value(&mut l.protocol,Protocol::Udp,"UDP");});ui.label("Port");ui.add(egui::DragValue::new(&mut l.port).range(1..=65535));if ui.button("Remove listener").clicked(){remove=Some(index);}});
if l.protocol==Protocol::Tcp{ui.checkbox(&mut l.port_check,"Active TCP port checks");}else{ui.weak("UDP health: encrypted path probes and explicit port errors; application readiness is unknown.");}
ui.collapsing("Checks and flow expiry",|ui|{egui::Grid::new("settings").show(ui,|ui|{
ui.label("Check interval, ms");ui.add(egui::DragValue::new(&mut l.check_interval_ms).range(250..=3_600_000));ui.end_row();
ui.label("Check timeout, ms");ui.add(egui::DragValue::new(&mut l.check_timeout_ms).range(100..=60_000));ui.end_row();
ui.label("Failures before exclusion");ui.add(egui::DragValue::new(&mut l.failure_threshold).range(1..=255));ui.end_row();
ui.label("Successes before recovery");ui.add(egui::DragValue::new(&mut l.recovery_threshold).range(1..=255));ui.end_row();
ui.label("Flow idle timeout, seconds");ui.add(egui::DragValue::new(&mut l.idle_timeout_secs).range(1..=604800));ui.end_row();
});});
let mut remove_backend=None;for(i,b)in l.backends.iter_mut().enumerate(){ui.push_id(i,|ui|{ui.horizontal(|ui|{
egui::ComboBox::from_id_salt("node").width(240.).selected_text(peer_name(n,&b.peer)).show_ui(ui,|ui|{for m in &n.members{let label=format!("{} ({})",m.hostname.as_deref().unwrap_or("unnamed"),m.endpoint_id.chars().take(8).collect::<String>());ui.selectable_value(&mut b.peer,m.endpoint_id.clone(),label);}});
ui.label("Target port");ui.add(egui::DragValue::new(&mut b.port).range(1..=65535));if ui.button("Remove").clicked(){remove_backend=Some(i);}
});});}if let Some(i)=remove_backend{l.backends.remove(i);}
if ui.button("+ Backend").clicked(){l.backends.push(Backend{peer:n.members.first().map(|m|m.endpoint_id.clone()).unwrap_or_default(),port:l.port});}
});});
}
if let Some(index) = remove {
config.listeners.remove(index);
}
if ui.button("+ Listener").clicked() {
config.listeners.push(Listener::new(Protocol::Tcp, 443));
}
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
use tsunagi::vip::VipStatus;
#[test]
fn editing_the_actual_form_sends_nothing_until_save_is_clicked() {
let (agent, mut commands) = AgentClient::test_client();
let ctx = egui::Context::default();
let mut config = VipConfig::new("api");
config.listeners.push(Listener::new(Protocol::Tcp, 443));
let mut state = Window {
draft: Some(Draft::new(config.clone(), Some("revision".into()))),
configure: true,
..Default::default()
};
let network = NetworkReport {
active: true,
name: "lab".into(),
vips: vec![VipStatus {
config: config.clone(),
revision: "revision".into(),
address: None,
error: None,
listeners: vec![],
}],
..Default::default()
};
let snapshot = Snapshot::default();
let frame = |events: Vec<egui::Event>, state: &mut Window| {
ctx.run(
egui::RawInput {
screen_rect: Some(egui::Rect::from_min_size(
egui::Pos2::ZERO,
egui::vec2(1000., 800.),
)),
events,
..Default::default()
},
|ctx| {
show(ctx, &agent, state, &network, &snapshot);
},
)
};
frame(vec![], &mut state);
ctx.memory_mut(|m| m.request_focus(egui::Id::new(("vip-name", &config.id))));
frame(vec![egui::Event::Text("new".into())], &mut state);
assert!(state.dirty());
assert_ne!(state.draft.as_ref().unwrap().config.name, config.name);
assert!(commands.try_recv().is_err());
assert_eq!(network.vips[0].config, config);
fn save_position(shape: &egui::Shape) -> Option<egui::Pos2> {
match shape {
egui::Shape::Text(text) if text.galley.text() == "Save" => {
Some(text.pos + text.galley.rect.center().to_vec2())
}
egui::Shape::Vec(shapes) => shapes.iter().find_map(save_position),
_ => None,
}
}
// Save was disabled before this edit; allow the next layout to register
// its now-enabled hit target before dispatching the pointer press.
let ready = frame(vec![], &mut state);
let position = ready
.shapes
.iter()
.find_map(|s| save_position(&s.shape))
.unwrap();
frame(
vec![
egui::Event::PointerMoved(position),
egui::Event::PointerButton {
pos: position,
button: egui::PointerButton::Primary,
pressed: true,
modifiers: egui::Modifiers::NONE,
},
],
&mut state,
);
frame(
vec![egui::Event::PointerButton {
pos: position,
button: egui::PointerButton::Primary,
pressed: false,
modifiers: egui::Modifiers::NONE,
}],
&mut state,
);
let Command::SaveVip {
config: saved,
expected,
delete,
..
} = commands.try_recv().unwrap()
else {
panic!("Save must submit one complete VIP draft");
};
assert_ne!(saved.name, config.name);
assert_eq!(expected.as_deref(), Some("revision"));
assert!(!delete);
assert!(commands.try_recv().is_err());
}
#[test]
fn all_listener_settings_render_without_a_live_agent() {
let ctx = egui::Context::default();
let mut cfg = VipConfig::new("api");
cfg.listeners.push(Listener::new(Protocol::Tcp, 443));
cfg.listeners.push(Listener::new(Protocol::Udp, 53));
let n = NetworkReport::default();
let _ = ctx.run(egui::RawInput::default(), |ctx| {
egui::CentralPanel::default().show(ctx, |ui| edit_listeners(ui, &mut cfg, &n));
});
assert_eq!(cfg.listeners.len(), 2);
}
}
crates/tsunagi-tray/src/vip_status.rs
+885
View File
@@ -0,0 +1,885 @@
//! Local VIP observations, kept apart from editable configuration drafts.
use std::collections::BTreeMap;
use std::time::{Duration, Instant};
use eframe::egui;
use tsunagi::ipc::NetworkReport;
use tsunagi::vip::{Backend, BackendStatus, Counters, ListenerStatus, Protocol, VipStatus};
use crate::{
format,
stats::{Series, Unit},
table,
};
const ACTIVE: egui::Color32 = egui::Color32::from_rgb(73, 190, 133);
const NEXT: egui::Color32 = egui::Color32::from_rgb(78, 157, 237);
const WARN: egui::Color32 = egui::Color32::from_rgb(225, 170, 73);
const BAD: egui::Color32 = egui::Color32::from_rgb(228, 112, 112);
#[derive(Default)]
struct Rates {
tx: Series,
rx: Series,
total: Series,
at: Option<Instant>,
}
impl Rates {
fn sample(&mut self, at: Instant, c: &Counters) {
self.tx.sample(at, c.tx_packets, c.tx_bytes);
self.rx.sample(at, c.rx_packets, c.rx_bytes);
self.total.sample(
at,
c.tx_packets.saturating_add(c.rx_packets),
c.tx_bytes.saturating_add(c.rx_bytes),
);
self.at = Some(at);
}
fn active(&self, now: Instant) -> bool {
self.at
.is_some_and(|at| now.saturating_duration_since(at) <= Duration::from_secs(5))
&& self.total.rate(Unit::Packets) > 0.0
}
}
#[derive(Clone, PartialEq, Eq, PartialOrd, Ord, Hash)]
struct Key(String, String, u16, String, u16);
fn key(v: &VipStatus, l: &ListenerStatus, b: &Backend) -> Key {
// Listener identity separates counter histories when a port is recreated.
let listener = v
.config
.listeners
.iter()
.find(|entry| entry.protocol == l.protocol && entry.port == l.port)
.map(|entry| entry.id.clone())
.unwrap_or_else(|| l.protocol.to_string());
Key(
v.config.id.clone(),
listener,
l.port,
b.peer.clone(),
b.port,
)
}
struct Camera {
rect: egui::Rect,
members: Vec<Backend>,
}
impl Default for Camera {
fn default() -> Self {
Self {
rect: egui::Rect::NOTHING,
members: Vec::new(),
}
}
}
pub(crate) struct View {
totals: BTreeMap<String, Rates>,
backends: BTreeMap<Key, Rates>,
cameras: BTreeMap<(String, String, u16), Camera>,
selected: Option<Key>,
unit: Unit,
}
impl Default for View {
fn default() -> Self {
Self {
totals: BTreeMap::new(),
backends: BTreeMap::new(),
cameras: BTreeMap::new(),
selected: None,
unit: Unit::Bytes,
}
}
}
impl View {
pub(crate) fn observe(&mut self, network: &NetworkReport, at: Instant) {
self.totals
.retain(|id, _| network.vips.iter().any(|v| &v.config.id == id));
let mut present = std::collections::BTreeSet::new();
for v in &network.vips {
let mut counters = Counters::default();
for l in &v.listeners {
counters.tx_packets = counters.tx_packets.saturating_add(l.counters.tx_packets);
counters.rx_packets = counters.rx_packets.saturating_add(l.counters.rx_packets);
counters.tx_bytes = counters.tx_bytes.saturating_add(l.counters.tx_bytes);
counters.rx_bytes = counters.rx_bytes.saturating_add(l.counters.rx_bytes);
for b in &l.backends {
let key = key(v, l, &b.backend);
self.backends
.entry(key.clone())
.or_default()
.sample(at, &b.counters);
present.insert(key);
}
}
self.totals
.entry(v.config.id.clone())
.or_default()
.sample(at, &counters);
}
self.backends.retain(|key, _| present.contains(key));
if self
.selected
.as_ref()
.is_some_and(|key| !present.contains(key))
{
self.selected = None;
}
self.cameras.retain(|(id, protocol, port), _| {
network.vips.iter().any(|v| {
&v.config.id == id
&& v.listeners
.iter()
.any(|l| l.protocol.to_string() == *protocol && l.port == *port)
})
});
}
}
fn name<'a>(n: &'a NetworkReport, id: &'a str) -> &'a str {
n.members
.iter()
.find(|m| m.endpoint_id == id)
.and_then(|m| m.hostname.as_deref())
.unwrap_or(id)
}
fn backend_name(n: &NetworkReport, b: &Backend) -> String {
format!("{}:{}", name(n, &b.peer), b.port)
}
fn rtt(b: &BackendStatus) -> String {
b.rtt_us
.map_or_else(|| "—".into(), |us| format!("{:.2} ms", us as f64 / 1000.0))
}
fn health_color(state: &str) -> egui::Color32 {
match state {
"ready" => ACTIVE,
"checking port" | "measuring RTT" | "draining" => WARN,
_ => BAD,
}
}
pub(crate) fn show(ui: &mut egui::Ui, v: &VipStatus, n: &NetworkReport, view: &mut View) {
ui.horizontal_wrapped(|ui| {
ui.heading(format!("{}.{}", v.config.name, n.name));
ui.weak("Traffic seen by this device");
ui.selectable_value(&mut view.unit, Unit::Bytes, "bytes");
ui.selectable_value(&mut view.unit, Unit::Packets, "pkts");
});
if let Some(error) = &v.error {
ui.colored_label(BAD, error);
}
let total = view.totals.get(&v.config.id);
ui.horizontal_wrapped(|ui| {
metric(
ui,
"TX",
&format::series_rate(total.map(|r| &r.tx), view.unit),
);
metric(
ui,
"RX",
&format::series_rate(total.map(|r| &r.rx), view.unit),
);
let flows: u64 = v
.listeners
.iter()
.flat_map(|l| &l.backends)
.map(|b| u64::from(b.active_flows))
.sum();
metric(ui, "Open flows", &flows.to_string());
});
format::sparkline(ui, total.map(|r| &r.total), view.unit, 64.0);
ui.add_space(8.0);
if v.listeners.is_empty() {
ui.weak("No listeners configured. Add one in Configuration.");
}
for l in &v.listeners {
ui.push_id(
(
"vip live listener",
&v.config.id,
l.protocol.to_string(),
l.port,
),
|ui| {
egui::Frame::group(ui.style()).show(ui, |ui| {
ui.set_min_width((ui.available_width() - 2.0).max(0.0));
ui.horizontal_wrapped(|ui| {
ui.heading(format!("{}/{}", l.protocol, l.port));
ui.weak("New connections:");
if let Some(b) = &l.selected {
ui.colored_label(NEXT, backend_name(n, b));
} else {
ui.colored_label(WARN, "No eligible backend");
}
});
ui.horizontal_wrapped(|ui| {
metric(ui, "TX", &format::bytes(l.counters.tx_bytes as f64));
metric(ui, "RX", &format::bytes(l.counters.rx_bytes as f64));
metric(
ui,
"Packets TX / RX",
&format!("{} / {}", l.counters.tx_packets, l.counters.rx_packets),
);
metric(ui, "Drops", &l.counters.dropped.to_string());
if l.protocol == Protocol::Udp {
metric(ui, "Failovers", &l.counters.failovers.to_string());
}
});
ui.separator();
diagram(ui, v, l, n, view);
ui.add_space(6.0);
egui::ScrollArea::horizontal()
.id_salt("backend table scroll")
.auto_shrink([false, true])
.show(ui, |ui| {
backend_table(ui, v, l, n, view);
});
if let Some(b) = l
.backends
.iter()
.find(|b| view.selected.as_ref() == Some(&key(v, l, &b.backend)))
{
egui::Frame::group(ui.style()).show(ui, |ui| {
ui.horizontal_wrapped(|ui| {
ui.strong(backend_name(n, &b.backend));
ui.colored_label(health_color(&b.state), &b.state);
ui.weak(format!("{} open flows", b.active_flows));
if let Some(age) = b.measurement_age_ms {
ui.weak(format!(
"RTT measured {:.1}s ago",
age as f64 / 1000.0
));
} else {
ui.weak("No RTT measurement yet");
}
});
ui.horizontal_wrapped(|ui| {
ui.weak("Node");
format::copy_field(
ui,
&format::short(&b.backend.peer),
&b.backend.peer,
);
if let Some(address) = n
.members
.iter()
.find(|m| m.endpoint_id == b.backend.peer)
.and_then(|m| m.overlay_address_v4.as_deref())
{
ui.weak("Address");
format::copy_field(ui, address, address);
}
});
});
}
});
},
);
ui.add_space(8.0);
}
ui.collapsing("Internal address and identity", |ui| {
egui::Grid::new(("vip identity", &v.config.id)).show(ui, |ui| {
ui.weak("Address");
if let Some(address) = v.address {
format::copy_field(ui, &address.to_string(), &address.to_string());
} else {
ui.weak("Not allocated");
}
ui.end_row();
ui.weak("ID");
format::copy_field(ui, &v.config.id, &v.config.id);
ui.end_row();
});
});
}
fn metric(ui: &mut egui::Ui, title: &str, value: &str) {
ui.horizontal(|ui| {
ui.weak(title);
ui.strong(value);
});
ui.add_space(8.0);
}
fn backend_table(
ui: &mut egui::Ui,
v: &VipStatus,
l: &ListenerStatus,
n: &NetworkReport,
view: &mut View,
) {
let headers = [
"Backend",
"Health",
"Route",
"RTT",
"Flows",
"TX / RX",
"TX / RX rate",
"Traffic",
];
let mut widths = [170.0_f32, 110.0, 100.0, 90.0, 50.0, 160.0, 160.0, 160.0];
// Measure every value before layout, including long identities and counters.
let values: Vec<_> = l
.backends
.iter()
.map(|b| {
let rates = view.backends.get(&key(v, l, &b.backend));
let route = if rates.is_some_and(|r| r.active(Instant::now())) {
"Traffic"
} else if l.selected.as_ref() == Some(&b.backend) {
"Next flow"
} else {
"—"
};
[
backend_name(n, &b.backend),
b.state.clone(),
route.into(),
rtt(b),
b.active_flows.to_string(),
format!(
"{} / {}",
format::bytes(b.counters.tx_bytes as f64),
format::bytes(b.counters.rx_bytes as f64)
),
format!(
"{} / {}",
format::series_rate(rates.map(|r| &r.tx), view.unit),
format::series_rate(rates.map(|r| &r.rx), view.unit)
),
]
})
.collect();
for (column, width) in widths.iter_mut().enumerate().take(7) {
let font = egui::FontSelection::Default.resolve(ui.style());
for text in
std::iter::once(headers[column]).chain(values.iter().map(|row| row[column].as_str()))
{
*width = width.max(
ui.fonts(|f| {
f.layout_no_wrap(text.to_owned(), font.clone(), egui::Color32::WHITE)
.size()
.x
}) + 12.0,
);
}
}
let widths_id = ui.id().with("backend column widths");
if let Some(previous) = ui.memory(|memory| memory.data.get_temp::<[f32; 8]>(widths_id)) {
for (width, previous) in widths.iter_mut().zip(previous) {
*width = width.max(previous);
}
}
ui.memory_mut(|memory| memory.data.insert_temp(widths_id, widths));
let fixed = widths.iter().sum::<f32>() + table::GAP * 7.0;
widths[7] += (ui.available_width() - fixed).max(0.0);
table::row(ui, "backend headers", &widths, false, |column, ui| {
ui.weak(headers[column]);
});
for (index, b) in l.backends.iter().enumerate() {
let key = key(v, l, &b.backend);
let rates = view.backends.get(&key);
table::row(
ui,
&key,
&widths,
index % 2 == 0,
|column, ui| match column {
0 => {
if ui
.selectable_label(view.selected.as_ref() == Some(&key), &values[index][0])
.clicked()
{
view.selected = Some(key.clone());
}
}
1 => {
ui.colored_label(health_color(&b.state), &values[index][1])
.on_hover_text(&b.state);
}
2 => {
ui.colored_label(
if values[index][2] == "Traffic" {
ACTIVE
} else {
NEXT
},
&values[index][2],
)
.on_hover_text(
if l.selected.as_ref() == Some(&b.backend) {
"Selected for new connections. Existing flows can use other backends."
} else {
"Traffic during the latest sample interval."
},
);
}
3..=6 => {
ui.label(&values[index][column]);
}
_ => format::sparkline(ui, rates.map(|r| &r.total), view.unit, 24.0),
},
);
}
if l.backends.is_empty() {
ui.weak("No backends. Add a node and target port in Configuration.");
}
}
#[derive(Clone, Copy)]
struct Branch {
active: bool,
next: bool,
}
fn branch(l: &ListenerStatus, b: &BackendStatus, rates: Option<&Rates>, now: Instant) -> Branch {
Branch {
active: rates.is_some_and(|r| r.active(now)),
next: l.selected.as_ref() == Some(&b.backend),
}
}
fn diagram(
ui: &mut egui::Ui,
v: &VipStatus,
l: &ListenerStatus,
n: &NetworkReport,
view: &mut View,
) {
let camera = view
.cameras
.entry((v.config.id.clone(), l.protocol.to_string(), l.port))
.or_default();
let members: Vec<_> = l.backends.iter().map(|b| b.backend.clone()).collect();
let height = (members.len().max(1) as f32 * 92.0 + 32.0).max(180.0);
let bounds = egui::Rect::from_min_size(egui::Pos2::ZERO, egui::vec2(860.0, height));
if camera.members != members || !camera.rect.is_positive() {
camera.rect = bounds;
camera.members = members;
}
ui.horizontal_wrapped(|ui| {
legend(ui, "Traffic", ACTIVE, true);
legend(ui, "Next new flow", NEXT, false);
ui.weak("Click a backend · drag to pan · Ctrl+scroll to zoom");
if ui.small_button("Fit").clicked() {
camera.rect = bounds;
}
});
let size = egui::vec2(ui.available_width(), height.clamp(180.0, 360.0));
let (outer, _) = ui.allocate_exact_size(size, egui::Sense::hover());
ui.painter()
.rect_filled(outer, 6.0, ui.visuals().extreme_bg_color);
let mut child = ui.new_child(egui::UiBuilder::new().id_salt("vip graph").max_rect(outer));
egui::Scene::new()
.zoom_range(0.05..=1.5)
.max_inner_size(bounds.size())
.show(&mut child, &mut camera.rect, |ui| {
ui.set_min_size(bounds.size());
let center = height / 2.0;
let source =
egui::Rect::from_min_size(egui::pos2(18.0, center - 32.0), egui::vec2(204.0, 64.0));
let listener = egui::Rect::from_min_size(
egui::pos2(290.0, center - 32.0),
egui::vec2(146.0, 64.0),
);
let active = l.backends.iter().any(|b| {
view.backends
.get(&key(v, l, &b.backend))
.is_some_and(|r| r.active(Instant::now()))
});
edge(
ui,
source.right_center(),
listener.left_center(),
if active { ACTIVE } else { NEXT },
active,
);
node(
ui,
source,
&format!("{}.{}", v.config.name, n.name),
"This device",
NEXT,
false,
);
node(
ui,
listener,
&format!("{}/{}", l.protocol, l.port),
"Listener",
NEXT,
false,
);
for (index, b) in l.backends.iter().enumerate() {
let key = key(v, l, &b.backend);
let rates = view.backends.get(&key);
let state = branch(l, b, rates, Instant::now());
let rect = egui::Rect::from_min_size(
egui::pos2(558.0, 24.0 + index as f32 * 92.0),
egui::vec2(278.0, 68.0),
);
let color = if state.active {
ACTIVE
} else if state.next {
NEXT
} else if b.state == "ready" {
ui.visuals().weak_text_color()
} else {
health_color(&b.state)
};
edge(
ui,
listener.right_center(),
rect.left_center(),
color,
state.active,
);
if state.next {
ui.painter().circle_stroke(
rect.left_center(),
5.0,
egui::Stroke::new(2.0_f32, NEXT),
);
}
let response = ui
.interact(rect, ui.id().with((&key.3, key.4)), egui::Sense::click())
.on_hover_cursor(egui::CursorIcon::PointingHand)
.on_hover_text(format!(
"{}\n{} · RTT {}\n{} open flows\n{}{}",
backend_name(n, &b.backend),
b.state,
rtt(b),
b.active_flows,
if state.active {
"Traffic in latest sample. "
} else {
""
},
if state.next {
"Selected for new connections."
} else {
""
}
));
if response.clicked() {
view.selected = Some(key.clone());
}
let detail = format!("{} · {} · {} flows", b.state, rtt(b), b.active_flows);
node(
ui,
rect,
&backend_name(n, &b.backend),
&detail,
color,
view.selected.as_ref() == Some(&key) || response.hovered(),
);
}
if l.backends.is_empty() {
ui.painter().text(
egui::pos2(570.0, center),
egui::Align2::LEFT_CENTER,
"No backends",
egui::FontId::proportional(15.0),
ui.visuals().weak_text_color(),
);
}
});
}
fn legend(ui: &mut egui::Ui, text: &str, color: egui::Color32, filled: bool) {
ui.horizontal(|ui| {
let (rect, _) = ui.allocate_exact_size(egui::vec2(12.0, 12.0), egui::Sense::hover());
if filled {
ui.painter().circle_filled(rect.center(), 4.0, color);
} else {
ui.painter()
.circle_stroke(rect.center(), 4.0, egui::Stroke::new(1.5_f32, color));
}
ui.colored_label(color, text);
});
}
fn node(
ui: &egui::Ui,
rect: egui::Rect,
title: &str,
detail: &str,
color: egui::Color32,
selected: bool,
) {
let painter = ui.painter_at(rect);
painter.rect_filled(rect, 8.0, ui.visuals().widgets.inactive.bg_fill);
painter.rect_stroke(
rect.shrink(1.0),
8.0,
egui::Stroke::new(if selected { 3.0_f32 } else { 1.5_f32 }, color),
egui::StrokeKind::Inside,
);
text_line(
&painter,
rect.min + egui::vec2(12.0, 9.0),
title,
16.0,
ui.visuals().strong_text_color(),
rect.width() - 24.0,
);
text_line(
&painter,
rect.min + egui::vec2(12.0, 34.0),
detail,
12.0,
ui.visuals().text_color(),
rect.width() - 24.0,
);
}
fn text_line(
painter: &egui::Painter,
position: egui::Pos2,
text: &str,
size: f32,
color: egui::Color32,
width: f32,
) {
let mut job = egui::text::LayoutJob::simple_singleline(
text.into(),
egui::FontId::proportional(size),
color,
);
job.wrap.max_width = width;
job.wrap.max_rows = 1;
job.wrap.break_anywhere = true;
painter.galley(position, painter.layout_job(job), color);
}
fn edge(ui: &egui::Ui, start: egui::Pos2, end: egui::Pos2, color: egui::Color32, active: bool) {
let bend = (end.x - start.x) * 0.5;
let curve = egui::epaint::CubicBezierShape::from_points_stroke(
[
start,
start + egui::vec2(bend, 0.0),
end - egui::vec2(bend, 0.0),
end,
],
false,
egui::Color32::TRANSPARENT,
egui::Stroke::new(if active { 3.0_f32 } else { 1.5_f32 }, color),
);
ui.painter().add(curve.clone());
if active {
let t = ui.input(|i| i.time) as f32 * 0.5 % 1.0;
let position = curve.sample(t);
ui.painter().circle_filled(position, 4.0, color);
ui.ctx().request_repaint_after(Duration::from_millis(40));
}
ui.painter().arrow(
end - egui::vec2(12.0, 0.0),
egui::vec2(10.0, 0.0),
egui::Stroke::new(1.5_f32, color),
);
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
use tsunagi::vip::{Listener, VipConfig};
fn fixture() -> (NetworkReport, VipStatus) {
let mut config = VipConfig::new("k8s");
config.listeners.push(Listener::new(Protocol::Tcp, 6443));
let backends: Vec<_> = ["ai", "master", "music"]
.into_iter()
.map(|peer| BackendStatus {
backend: Backend {
peer: peer.into(),
port: 7443,
},
state: "ready".into(),
rtt_us: Some(4500),
measurement_age_ms: Some(500),
active_flows: 5,
counters: Counters::default(),
})
.collect();
let v = VipStatus {
config,
revision: "r1".into(),
address: None,
error: None,
listeners: vec![ListenerStatus {
protocol: Protocol::Tcp,
port: 6443,
selected: Some(backends[0].backend.clone()),
backends,
counters: Counters::default(),
}],
};
let n = NetworkReport {
name: "test".into(),
vips: vec![v.clone()],
..Default::default()
};
(n, v)
}
#[test]
fn traffic_branches_are_sampled_independently_of_new_flow_selection_and_open_flows() {
let (mut n, _) = fixture();
let now = Instant::now();
let mut view = View::default();
view.observe(&n, now);
n.vips[0].listeners[0].backends[2].counters.tx_packets = 20;
n.vips[0].listeners[0].backends[2].counters.tx_bytes = 4000;
n.vips[0].listeners[0].counters.tx_packets = 20;
n.vips[0].listeners[0].counters.tx_bytes = 4000;
view.observe(&n, now + Duration::from_secs(2));
let v = &n.vips[0];
let l = &v.listeners[0];
let selected = branch(
l,
&l.backends[0],
view.backends.get(&key(v, l, &l.backends[0].backend)),
now + Duration::from_secs(2),
);
let carrying = branch(
l,
&l.backends[2],
view.backends.get(&key(v, l, &l.backends[2].backend)),
now + Duration::from_secs(2),
);
assert!(selected.next && !selected.active);
assert!(carrying.active && !carrying.next);
assert_eq!(view.totals[&v.config.id].total.rate(Unit::Bytes), 2000.0);
assert!(
!view.backends[&key(v, l, &l.backends[2].backend)].active(now + Duration::from_secs(8))
);
view.observe(&n, now + Duration::from_secs(4));
assert!(
!view.backends[&key(v, l, &l.backends[2].backend)].active(now + Duration::from_secs(4))
);
}
#[test]
fn removed_services_and_backends_release_history_selection_and_cameras() {
let (mut n, v) = fixture();
let mut view = View::default();
view.observe(&n, Instant::now());
view.selected = Some(key(
&v,
&v.listeners[0],
&v.listeners[0].backends[0].backend,
));
n.vips.clear();
view.observe(&n, Instant::now());
assert!(
view.backends.is_empty()
&& view.totals.is_empty()
&& view.selected.is_none()
&& view.cameras.is_empty()
);
}
#[test]
fn clicking_a_scene_node_selects_its_backend_without_changing_configuration() {
let (n, v) = fixture();
let ctx = egui::Context::default();
let mut view = View::default();
let frame = |view: &mut View, events| {
ctx.run(
egui::RawInput {
screen_rect: Some(egui::Rect::from_min_size(
egui::Pos2::ZERO,
egui::vec2(1000.0, 600.0),
)),
events,
..Default::default()
},
|ctx| {
egui::CentralPanel::default()
.show(ctx, |ui| diagram(ui, &v, &v.listeners[0], &n, view));
},
)
};
frame(&mut view, vec![]);
let output = frame(&mut view, vec![]);
let pos = output
.shapes
.iter()
.find_map(|s| match &s.shape {
egui::Shape::Text(t) if t.galley.text() == "music:7443" => {
Some(t.pos + egui::vec2(20.0, 10.0))
}
_ => None,
})
.expect("scene draws the backend node");
for pressed in [true, false] {
frame(
&mut view,
vec![
egui::Event::PointerMoved(pos),
egui::Event::PointerButton {
pos,
button: egui::PointerButton::Primary,
pressed,
modifiers: egui::Modifiers::NONE,
},
],
);
}
assert!(
view.selected
== Some(key(
&v,
&v.listeners[0],
&v.listeners[0].backends[2].backend
))
);
assert_eq!(n.vips[0].config, v.config);
}
#[test]
fn reset_counters_and_recreated_listeners_do_not_inherit_active_traffic() {
let (mut n, _) = fixture();
let mut view = View::default();
let at = Instant::now();
n.vips[0].listeners[0].backends[0].counters.tx_packets = 100;
view.observe(&n, at);
n.vips[0].listeners[0].backends[0].counters.tx_packets = 0;
view.observe(&n, at + Duration::from_secs(2));
let v = &n.vips[0];
let l = &v.listeners[0];
assert!(
!view.backends[&key(v, l, &l.backends[0].backend)].active(at + Duration::from_secs(2))
);
n.vips[0].config.listeners[0].id = "new listener".into();
n.vips[0].listeners[0].backends[0].counters.tx_packets = 200;
view.observe(&n, at + Duration::from_secs(4));
let v = &n.vips[0];
let l = &v.listeners[0];
assert!(
!view.backends[&key(v, l, &l.backends[0].backend)].active(at + Duration::from_secs(4))
);
assert_eq!(view.backends.len(), 3);
}
#[test]
fn live_tables_and_scenes_render_at_narrow_and_wide_sizes() {
let (n, v) = fixture();
let ctx = egui::Context::default();
let mut view = View::default();
for width in [650.0, 1000.0, 1800.0, 650.0] {
for _ in 0..2 {
let _ = ctx.run(
egui::RawInput {
screen_rect: Some(egui::Rect::from_min_size(
egui::Pos2::ZERO,
egui::vec2(width, 1000.0),
)),
..Default::default()
},
|ctx| {
egui::CentralPanel::default().show(ctx, |ui| {
egui::ScrollArea::vertical().show(ui, |ui| show(ui, &v, &n, &mut view));
assert!(
ui.min_rect().right() <= width + 1.0,
"VIP content expands the window at {width}"
);
});
},
);
}
}
}
}
crates/tsunagi-wg-quic/Cargo.toml
+3 -1
View File
@@ -4,7 +4,7 @@ name = "tsunagi-wg-quic"
# negotiates: a release here does not stop it talking to a peer on an older # negotiates: a release here does not stop it talking to a peer on an older
# build, because what peers compare is `ANNOUNCEMENT_VERSION`. # build, because what peers compare is `ANNOUNCEMENT_VERSION`.
version = "0.1.0" version = "0.1.0"
description = "The wg-quic protocol for tsunagi: WireGuard's cryptography carried in iroh's QUIC datagrams." description = "The single internal WireGuard tunnel for tsunagi, independent of external carriers."
edition.workspace = true edition.workspace = true
rust-version.workspace = true rust-version.workspace = true
license.workspace = true license.workspace = true
@@ -30,6 +30,8 @@ rand = "0.10"
boringtun = { version = "0.7.1", default-features = false } boringtun = { version = "0.7.1", default-features = false }
[dev-dependencies] [dev-dependencies]
tsunagi-wireguard = { path = "../tsunagi-wireguard" }
tsunagi-tcp-tls = { path = "../tsunagi-tcp-tls" }
tsunagi-core = { path = "../tsunagi-core", features = ["testing"] } tsunagi-core = { path = "../tsunagi-core", features = ["testing"] }
tokio = { version = "1.53", features = ["rt", "rt-multi-thread", "sync", "time", "macros"] } tokio = { version = "1.53", features = ["rt", "rt-multi-thread", "sync", "time", "macros"] }
tempfile.workspace = true tempfile.workspace = true
crates/tsunagi-wg-quic/src/device.rs
+54 -23
View File
@@ -59,7 +59,7 @@ const SCRATCH: usize = 4096;
/// All three operations share the same mutex as the cryptographic state. /// All three operations share the same mutex as the cryptographic state.
struct FlowTunnel { struct FlowTunnel {
tunn: Tunn, tunn: Tunn,
queued: VecDeque<FlowId>, queued: VecDeque<(FlowId, usize)>,
} }
fn is_data(result: &TunnResult<'_>) -> bool { fn is_data(result: &TunnResult<'_>) -> bool {
@@ -79,7 +79,7 @@ impl FlowTunnel {
} }
// MAX_QUEUE_DEPTH in boringtun 0.7: the newest packet is dropped at 256. // MAX_QUEUE_DEPTH in boringtun 0.7: the newest packet is dropped at 256.
if self.queued.len() < 256 { if self.queued.len() < 256 {
self.queued.push_back(flow); self.queued.push_back((flow, packet.len()));
} }
(result, 0) (result, 0)
} }
@@ -88,19 +88,22 @@ impl FlowTunnel {
&mut self, &mut self,
packet: &[u8], packet: &[u8],
scratch: &'a mut [u8], scratch: &'a mut [u8],
) -> (TunnResult<'a>, FlowId) { ) -> (TunnResult<'a>, FlowId, Option<usize>) {
let result = self.tunn.decapsulate(None, packet, scratch); let result = self.tunn.decapsulate(None, packet, scratch);
let mut flow = 0; let (mut flow, mut plaintext_len) = (0, None);
if packet.is_empty() { if packet.is_empty() {
if is_data(&result) { if is_data(&result) {
flow = self.queued.pop_front().unwrap_or(0); if let Some((tag, len)) = self.queued.pop_front() {
flow = tag;
plaintext_len = Some(len);
}
} else if let Some(pending) = self.queued.pop_front() { } else if let Some(pending) = self.queued.pop_front() {
// Without a session, draining pops the oldest IP packet and // Without a session, draining pops the oldest IP packet and
// encapsulate queues it again at the back. // encapsulate queues it again at the back.
self.queued.push_back(pending); self.queued.push_back(pending);
} }
} }
(result, flow) (result, flow, plaintext_len)
} }
fn update_timers<'a>(&mut self, scratch: &'a mut [u8]) -> TunnResult<'a> { fn update_timers<'a>(&mut self, scratch: &'a mut [u8]) -> TunnResult<'a> {
@@ -118,6 +121,7 @@ impl FlowTunnel {
/// Counters for one peer's tunnel. /// Counters for one peer's tunnel.
#[derive(Debug, Default)] #[derive(Debug, Default)]
struct PeerCounters { struct PeerCounters {
epoch: u64,
tx_packets: AtomicU64, tx_packets: AtomicU64,
tx_bytes: AtomicU64, tx_bytes: AtomicU64,
rx_packets: AtomicU64, rx_packets: AtomicU64,
@@ -132,6 +136,8 @@ struct PeerCounters {
/// A snapshot of one peer's tunnel counters. /// A snapshot of one peer's tunnel counters.
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] #[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub struct PeerStats { pub struct PeerStats {
/// Identifies this counter lifetime; unchanged by rekeying or route switches.
pub counter_epoch: u64,
/// Plaintext packets encrypted and sent to this peer. /// Plaintext packets encrypted and sent to this peer.
pub tx_packets: u64, pub tx_packets: u64,
/// Plaintext bytes encrypted and sent to this peer. /// Plaintext bytes encrypted and sent to this peer.
@@ -209,6 +215,7 @@ impl Drop for Peer {
impl Peer { impl Peer {
fn stats(&self) -> PeerStats { fn stats(&self) -> PeerStats {
PeerStats { PeerStats {
counter_epoch: self.counters.epoch,
tx_packets: self.counters.tx_packets.load(Ordering::Relaxed), tx_packets: self.counters.tx_packets.load(Ordering::Relaxed),
tx_bytes: self.counters.tx_bytes.load(Ordering::Relaxed), tx_bytes: self.counters.tx_bytes.load(Ordering::Relaxed),
rx_packets: self.counters.rx_packets.load(Ordering::Relaxed), rx_packets: self.counters.rx_packets.load(Ordering::Relaxed),
@@ -328,26 +335,27 @@ impl WireguardDevice {
}; };
let (result, selected_flow) = tunn.encapsulate(packet, &mut scratch, flow); let (result, selected_flow) = tunn.encapsulate(packet, &mut scratch, flow);
match result { match result {
TunnResult::WriteToNetwork(out) => Some((out.len(), selected_flow)), TunnResult::WriteToNetwork(out) => {
Some((out.len(), selected_flow, out.starts_with(&[4, 0, 0, 0])))
}
// No session yet, so nothing to send. Counted as dropped // No session yet, so nothing to send. Counted as dropped
// rather than reported: the handshake is in flight and the // rather than reported: the handshake is in flight and the
// next packet will go. // next packet will go.
_ => None, _ => None,
} }
}; };
let Some((len, selected_flow)) = result else { let Some((len, selected_flow, data)) = result else {
peer.counters peer.counters
.dropped_no_session .dropped_no_session
.fetch_add(1, Ordering::Relaxed); .fetch_add(1, Ordering::Relaxed);
return false; return false;
}; };
send_to_peer_flow(&peer, &scratch[..len], selected_flow); let sent = send_to_peer_flow(&peer, &scratch[..len], selected_flow);
peer.counters.tx_packets.fetch_add(1, Ordering::Relaxed); if sent && data {
peer.counters count_sent(&peer, packet.len());
.tx_bytes }
.fetch_add(packet.len() as u64, Ordering::Relaxed); sent
true
} }
/// Adds or replaces a peer and starts its tunnel. /// Adds or replaces a peer and starts its tunnel.
@@ -388,7 +396,10 @@ impl WireguardDevice {
queued: VecDeque::new(), queued: VecDeque::new(),
}), }),
link, link,
counters: Arc::new(PeerCounters::default()), counters: Arc::new(PeerCounters {
epoch: rand::random(),
..Default::default()
}),
task: Mutex::new(None), task: Mutex::new(None),
}); });
@@ -501,9 +512,9 @@ fn send_to_peer(peer: &Peer, payload: &[u8]) {
send_to_peer_flow(peer, payload, 0); send_to_peer_flow(peer, payload, 0);
} }
fn send_to_peer_flow(peer: &Peer, payload: &[u8], flow: FlowId) { fn send_to_peer_flow(peer: &Peer, payload: &[u8], flow: FlowId) -> bool {
match peer.link.send_flow(Bytes::copy_from_slice(payload), flow) { match peer.link.send_flow(Bytes::copy_from_slice(payload), flow) {
Ok(()) => {} Ok(()) => return true,
Err(TransportError::TooLarge { .. }) => { Err(TransportError::TooLarge { .. }) => {
peer.counters peer.counters
.dropped_oversize .dropped_oversize
@@ -514,6 +525,14 @@ fn send_to_peer_flow(peer: &Peer, payload: &[u8], flow: FlowId) {
tracing::trace!(%err, "dropping a WireGuard packet the link refused"); tracing::trace!(%err, "dropping a WireGuard packet the link refused");
} }
} }
false
}
fn count_sent(peer: &Peer, plaintext_len: usize) {
peer.counters.tx_packets.fetch_add(1, Ordering::Relaxed);
peer.counters
.tx_bytes
.fetch_add(plaintext_len as u64, Ordering::Relaxed);
} }
async fn read_from_link(inner: Arc<Inner>, peer: Arc<Peer>) { async fn read_from_link(inner: Arc<Inner>, peer: Arc<Peer>) {
@@ -532,9 +551,12 @@ async fn read_from_link(inner: Arc<Inner>, peer: Arc<Peer>) {
Ok(guard) => guard, Ok(guard) => guard,
Err(poisoned) => poisoned.into_inner(), Err(poisoned) => poisoned.into_inner(),
}; };
let (result, flow) = tunn.decapsulate(input.unwrap_or(&[]), &mut scratch); let (result, flow, plaintext_len) =
tunn.decapsulate(input.unwrap_or(&[]), &mut scratch);
match result { match result {
TunnResult::WriteToNetwork(out) => Outcome::ToNetwork(out.len(), flow), TunnResult::WriteToNetwork(out) => {
Outcome::ToNetwork(out.len(), flow, plaintext_len)
}
// The source boringtun reports is not consulted here: // The source boringtun reports is not consulted here:
// whether the peer may use it is checked where the // whether the peer may use it is checked where the
// claims live. // claims live.
@@ -549,8 +571,12 @@ async fn read_from_link(inner: Arc<Inner>, peer: Arc<Peer>) {
}; };
match outcome { match outcome {
Outcome::ToNetwork(len, flow) => { Outcome::ToNetwork(len, flow, plaintext_len) => {
send_to_peer_flow(&peer, &scratch[..len], flow); if send_to_peer_flow(&peer, &scratch[..len], flow)
&& let Some(len) = plaintext_len
{
count_sent(&peer, len);
}
// Keep draining with an empty datagram, as boringtun asks. // Keep draining with an empty datagram, as boringtun asks.
input = None; input = None;
continue; continue;
@@ -584,7 +610,7 @@ async fn read_from_link(inner: Arc<Inner>, peer: Arc<Peer>) {
} }
enum Outcome { enum Outcome {
ToNetwork(usize, FlowId), ToNetwork(usize, FlowId, Option<usize>),
ToTunnel(usize), ToTunnel(usize),
Done, Done,
Failed, Failed,
@@ -671,7 +697,12 @@ mod flow_tests {
let keepalive = wire(a.decapsulate(&response, &mut scratch).0); let keepalive = wire(a.decapsulate(&response, &mut scratch).0);
let _ = b.decapsulate(&keepalive, &mut scratch); let _ = b.decapsulate(&keepalive, &mut scratch);
for expected in 100..356 { for expected in 100..356 {
let (result, flow) = a.decapsulate(&[], &mut scratch); let (result, flow, len) = a.decapsulate(&[], &mut scratch);
assert_eq!(
len,
Some(packet.len()),
"queued counters retain the exact unpadded IP length"
);
assert_eq!(flow, expected); assert_eq!(flow, expected);
let ciphertext = wire(result); let ciphertext = wire(result);
match b.decapsulate(&ciphertext, &mut scratch).0 { match b.decapsulate(&ciphertext, &mut scratch).0 {
crates/tsunagi-wg-quic/src/plugin.rs
+39 -24
View File
@@ -47,13 +47,9 @@ use crate::store::WgKeyStore;
use tsunagi::state::Ipv4Range; use tsunagi::state::Ipv4Range;
use tsunagi::task::{TASK_GRACE, wind_down}; use tsunagi::task::{TASK_GRACE, wind_down};
/// The protocol identifier this plugin announces. /// Fixed internal encryption capability, independent of external carrier names.
/// The protocol id of this plugin. /// The historical crate/storage name does not select a physical transport.
/// pub const WIREGUARD_PROTOCOL: &str = "wireguard-e2e";
/// `wg-quic`, because that is what it is: WireGuard's cryptography carried
/// in QUIC datagrams. The name is on the wire, so it is a protocol name and
/// not a description of the implementation.
pub const WIREGUARD_PROTOCOL: &str = "wg-quic";
/// Smallest interface MTU the overlay accepts. /// Smallest interface MTU the overlay accepts.
/// ///
@@ -79,8 +75,8 @@ pub const WIREGUARD_OVERHEAD: u32 = 32;
/// Configuration of the WireGuard plugin. /// Configuration of the WireGuard plugin.
#[derive(Debug, Clone)] #[derive(Debug, Clone)]
pub struct WireguardConfig { pub struct WireguardConfig {
/// Diagnostic protocol id when this end-to-end engine is reused by another carrier. /// Previous key stores to import before creating any network key.
pub protocol: &'static str, pub legacy_key_stores: Vec<PathBuf>,
/// Directory for the plugin's own key store. Separate from agent state. /// Directory for the plugin's own key store. Separate from agent state.
pub state_dir: PathBuf, pub state_dir: PathBuf,
/// WireGuard keepalive, which keeps tunnels and their links warm. /// WireGuard keepalive, which keeps tunnels and their links warm.
@@ -102,7 +98,7 @@ impl WireguardConfig {
/// Creates a configuration rooted at `state_dir` with sensible defaults. /// Creates a configuration rooted at `state_dir` with sensible defaults.
pub fn new(state_dir: impl Into<PathBuf>) -> Self { pub fn new(state_dir: impl Into<PathBuf>) -> Self {
Self { Self {
protocol: WIREGUARD_PROTOCOL, legacy_key_stores: Vec::new(),
state_dir: state_dir.into(), state_dir: state_dir.into(),
keepalive: Some(25), keepalive: Some(25),
mtu: DEFAULT_MTU, mtu: DEFAULT_MTU,
@@ -302,9 +298,16 @@ impl WireguardPlugin {
} }
let path = config.key_store_path(); let path = config.key_store_path();
let store = tokio::task::spawn_blocking(move || WgKeyStore::open(path)) let legacy = config.legacy_key_stores.clone();
.await let store = tokio::task::spawn_blocking(move || {
.map_err(|err| PluginError::Other(format!("key store task failed: {err}")))??; let store = WgKeyStore::open(path)?;
for path in legacy {
store.import_legacy(&path)?;
}
Ok::<_, PluginError>(store)
})
.await
.map_err(|err| PluginError::Other(format!("key store task failed: {err}")))??;
let worker = Arc::new(Worker { let worker = Arc::new(Worker {
config, config,
@@ -397,7 +400,7 @@ impl Worker {
fn report(&self, network: NetworkId, reason: impl std::fmt::Display) { fn report(&self, network: NetworkId, reason: impl std::fmt::Display) {
tracing::warn!(network = %network.fmt_short(), %reason, "wireguard plugin error"); tracing::warn!(network = %network.fmt_short(), %reason, "wireguard plugin error");
if let Some(context) = self.context.get() { if let Some(context) = self.context.get() {
context.report_error(network, self.config.protocol, reason.to_string()); context.report_error(network, WIREGUARD_PROTOCOL, reason.to_string());
} }
} }
@@ -650,7 +653,7 @@ async fn run(worker: Arc<Worker>, mut commands: mpsc::Receiver<Command>) {
impl IpPlugin for WireguardPlugin { impl IpPlugin for WireguardPlugin {
fn protocol_id(&self) -> &str { fn protocol_id(&self) -> &str {
self.worker.config.protocol WIREGUARD_PROTOCOL
} }
fn protocol_version(&self) -> u16 { fn protocol_version(&self) -> u16 {
@@ -689,7 +692,7 @@ impl IpPlugin for WireguardPlugin {
// Identity only. Where to send packets is the transport's business. // Identity only. Where to send packets is the transport's business.
let announcement = WgAnnouncement::new(network, &state.key.public()); let announcement = WgAnnouncement::new(network, &state.key.public());
Ok(Some(PluginCapability { Ok(Some(PluginCapability {
protocol: self.worker.config.protocol.to_string(), protocol: WIREGUARD_PROTOCOL.to_string(),
version: crate::announcement::ANNOUNCEMENT_VERSION, version: crate::announcement::ANNOUNCEMENT_VERSION,
enabled: true, enabled: true,
data: announcement.encode()?, data: announcement.encode()?,
@@ -795,14 +798,26 @@ impl IpPlugin for WireguardPlugin {
self.nudge(Command::Teardown(network)); self.nudge(Command::Teardown(network));
} }
fn on_network_forgotten(&self, network: NetworkId) { fn on_network_forgotten(&self, network: NetworkId) -> BoxFuture<'_, ()> {
// The key is this protocol's identity in that network and nothing let worker = Arc::clone(&self.worker);
// else's. Keeping it after leaving would keep a secret for a Box::pin(async move {
// network this agent is no longer in — and hand back the same // Remove old copies as well, otherwise the next startup's import
// overlay address on a rejoin that everyone else has moved past. // would resurrect a key that the user explicitly forgot.
if let Err(err) = self.worker.store.forget(network) { let result = tokio::task::spawn_blocking(move || {
tracing::warn!(%err, "cannot remove the WireGuard key of a network we left"); for path in &worker.config.legacy_key_stores {
} if path.exists() {
WgKeyStore::open(path)?.forget(network)?;
}
}
worker.store.forget(network)
})
.await;
match result {
Ok(Ok(())) => {}
Ok(Err(err)) => tracing::warn!(%err, "cannot remove forgotten network data keys"),
Err(err) => tracing::warn!(%err, "data key cleanup task failed"),
}
})
} }
fn shutdown<'a>(&'a self) -> BoxFuture<'a, ()> { fn shutdown<'a>(&'a self) -> BoxFuture<'a, ()> {
crates/tsunagi-wg-quic/src/store.rs
+109
View File
@@ -106,6 +106,49 @@ impl WgKeyStore {
} }
} }
/// Imports missing network keys from an older carrier-specific store, atomically.
/// The source is read-only and existing canonical keys always win.
pub fn import_legacy(&self, path: &Path) -> Result<(), PluginError> {
if !path.exists() || path == self.path {
return Ok(());
}
let fail = |err: rusqlite::Error| {
PluginError::Other(format!("cannot import legacy data keys: {err}"))
};
let source = Connection::open_with_flags(path, rusqlite::OpenFlags::SQLITE_OPEN_READ_ONLY)
.map_err(fail)?;
let integrity: String = source
.query_row("PRAGMA integrity_check", [], |row| row.get(0))
.map_err(fail)?;
let schema: i64 = source
.query_row("PRAGMA user_version", [], |row| row.get(0))
.map_err(fail)?;
if integrity != "ok" || schema != SCHEMA_VERSION {
return Err(PluginError::Other(
"legacy data key store is damaged or has an unsupported schema".into(),
));
}
let mut statement = source.prepare("SELECT network_id, secret, created_at, length(network_id), length(secret) FROM network_keys").map_err(fail)?;
let mut rows = statement.query([]).map_err(fail)?;
let mut target = self.lock();
let transaction = target.transaction().map_err(fail)?;
while let Some(row) = rows.next().map_err(fail)? {
let id_len: i64 = row.get(3).map_err(fail)?;
let key_len: i64 = row.get(4).map_err(fail)?;
if id_len != 32 || key_len != KEY_LEN as i64 {
return Err(PluginError::Other(
"legacy data key has an invalid length".into(),
));
}
let id: Vec<u8> = row.get(0).map_err(fail)?;
let key = zeroize::Zeroizing::new(row.get::<_, Vec<u8>>(1).map_err(fail)?);
let created: i64 = row.get(2).map_err(fail)?;
transaction.execute("INSERT OR IGNORE INTO network_keys(network_id, secret, created_at) VALUES (?1, ?2, ?3)", params![id, key.as_slice(), created]).map_err(fail)?;
}
transaction.commit().map_err(fail)?;
Ok(())
}
/// Returns this agent's key for a network, creating it on first use. /// Returns this agent's key for a network, creating it on first use.
pub fn load_or_create(&self, network: NetworkId) -> Result<WgSecretKey, PluginError> { pub fn load_or_create(&self, network: NetworkId) -> Result<WgSecretKey, PluginError> {
let conn = self.lock(); let conn = self.lock();
@@ -219,4 +262,70 @@ mod tests {
"a damaged key store must not silently mint a new identity (was {original})" "a damaged key store must not silently mint a new identity (was {original})"
); );
} }
#[tokio::test]
async fn forgetting_an_imported_key_cannot_resurrect_it_on_restart() {
use crate::{WireguardConfig, WireguardPlugin};
use tsunagi::dataplane::IpPlugin;
let dir = tempfile::TempDir::new().unwrap();
let id = network("forgotten-import");
let legacy_path = dir.path().join("old.sqlite");
let original = WgKeyStore::open(&legacy_path)
.unwrap()
.load_or_create(id)
.unwrap()
.public();
let mut config = WireguardConfig::new(dir.path().join("canonical"));
config.legacy_key_stores.push(legacy_path);
let plugin = WireguardPlugin::open(config.clone()).await.unwrap();
plugin.on_network_forgotten(id).await;
plugin.shutdown().await;
let restarted = WireguardPlugin::open(config.clone()).await.unwrap();
let key = WgKeyStore::open(config.key_store_path())
.unwrap()
.load_or_create(id)
.unwrap()
.public();
assert_ne!(key, original);
restarted.shutdown().await;
}
#[test]
fn legacy_import_preserves_keys_and_never_overwrites_the_canonical_identity() {
let dir = tempfile::TempDir::new().unwrap();
let legacy = WgKeyStore::open(dir.path().join("legacy.sqlite")).unwrap();
let canonical = WgKeyStore::open(dir.path().join("canonical.sqlite")).unwrap();
let imported = network("imported");
let existing = network("existing");
let imported_key = legacy.load_or_create(imported).unwrap().public();
legacy.load_or_create(existing).unwrap();
let existing_key = canonical.load_or_create(existing).unwrap().public();
canonical.import_legacy(legacy.path()).unwrap();
canonical.import_legacy(legacy.path()).unwrap();
assert_eq!(
canonical.load_or_create(imported).unwrap().public(),
imported_key
);
assert_eq!(
canonical.load_or_create(existing).unwrap().public(),
existing_key
);
assert_eq!(
legacy.load_or_create(imported).unwrap().public(),
imported_key
);
}
#[test]
fn a_damaged_legacy_store_aborts_import_without_regenerating_keys() {
let dir = tempfile::TempDir::new().unwrap();
let path = dir.path().join("legacy.sqlite");
std::fs::write(&path, b"damaged SQLite").unwrap();
let canonical = WgKeyStore::open(dir.path().join("canonical.sqlite")).unwrap();
assert!(canonical.import_legacy(&path).is_err());
canonical
.import_legacy(&dir.path().join("absent.sqlite"))
.unwrap();
assert!(!dir.path().join("absent.sqlite").exists());
}
} }
crates/tsunagi-wg-quic/tests/carriers.rs
+285
View File
@@ -0,0 +1,285 @@
//! Real membership and one internal cipher over each independent external carrier.
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use bytes::Bytes;
use iroh::EndpointId;
use std::{
net::{Ipv4Addr, SocketAddr},
sync::Arc,
time::Duration,
};
use tempfile::TempDir;
use tsunagi::dataplane::transport::{DataCarrier, PacketTransport, SharedLink, TransportError};
use tsunagi::discovery::SharedMemoryDiscovery;
use tsunagi::overlay::{MemoryTun, MemoryTunFactory};
use tsunagi::testing::{config_with, network, wait_for_peers, wait_until};
use tsunagi::{Agent, BoxFuture, NetworkId};
use tsunagi_tcp_tls::{TcpTlsCarrier, TcpTlsTransport};
use tsunagi_wg_quic::{WireguardConfig, WireguardPlugin};
use tsunagi_wireguard::{UdpCarrier, UdpTransport};
// Only address selection is constrained to loopback. TLS sockets, identity
// verification, framing, membership and both encryption layers remain real.
#[derive(Debug)]
struct LoopbackTls {
inner: Arc<TcpTlsTransport>,
ports: Arc<std::sync::Mutex<Vec<(EndpointId, u16)>>>,
}
impl PacketTransport for LoopbackTls {
fn name(&self) -> &str {
"tcp-tls"
}
fn as_any(&self) -> &dyn std::any::Any {
self
}
fn open<'a>(
&'a self,
network: NetworkId,
peer: EndpointId,
protocol: &'a str,
) -> BoxFuture<'a, Result<SharedLink, TransportError>> {
let port = self
.ports
.lock()
.unwrap()
.iter()
.find(|(id, _)| *id == peer)
.unwrap()
.1;
self.inner
.replace_peer_addrs(peer, vec![SocketAddr::from((Ipv4Addr::LOCALHOST, port))]);
self.inner.open(network, peer, protocol)
}
}
struct Node {
_root: TempDir,
agent: Agent,
tunnel: Arc<WireguardPlugin>,
tuns: MemoryTunFactory,
accept: tokio::task::JoinHandle<()>,
}
impl Node {
async fn new(
discovery: &SharedMemoryDiscovery,
protocol: &str,
ports: Arc<std::sync::Mutex<Vec<(EndpointId, u16)>>>,
) -> Self {
Self::configured(discovery, protocol, ports, true).await
}
async fn configured(
discovery: &SharedMemoryDiscovery,
protocol: &str,
ports: Arc<std::sync::Mutex<Vec<(EndpointId, u16)>>>,
register_transport: bool,
) -> Self {
let root = TempDir::new().unwrap();
let identity = tsunagi::storage::StateStore::open(
tsunagi::StoragePaths::under(root.path()).state_db(),
)
.unwrap()
.load_or_create_device_identity()
.unwrap();
let tunnel = WireguardPlugin::open(
WireguardConfig::new(root.path().join("cipher"))
.with_reconcile(Duration::from_millis(20), Duration::from_millis(250)),
)
.await
.unwrap();
let tuns = MemoryTunFactory::new();
let config = config_with(root.path(), discovery)
.with_interface(Arc::new(tuns.clone()), "carrier-test", 1280)
.with_tunnel(tunnel.clone());
let (agent, accept) = if protocol == "wg" {
let transport =
UdpTransport::wireguard(identity.endpoint_id(), Ipv4Addr::LOCALHOST, None);
let carrier: Arc<dyn DataCarrier> = UdpCarrier::new(transport.clone());
let mut config = config.with_carrier(carrier);
if register_transport {
config = config.with_custom_transport(protocol, transport.clone());
}
let agent = Agent::spawn(config).await.unwrap();
let incoming = agent.clone();
let accept = tokio::spawn(async move {
while let Some(link) = transport.next_inbound().await {
let _ = incoming.install_inbound_link(link).await;
}
});
(agent, accept)
} else {
let transport = Arc::new(
TcpTlsTransport::bind(&identity, Some(0), None)
.await
.unwrap(),
);
ports
.lock()
.unwrap()
.push((identity.endpoint_id(), transport.bound_port()));
let carrier: Arc<dyn DataCarrier> = TcpTlsCarrier::new(transport.clone());
let custom = Arc::new(LoopbackTls {
inner: transport.clone(),
ports,
});
let agent = Agent::spawn(
config
.with_carrier(carrier)
.with_custom_transport(protocol, custom),
)
.await
.unwrap();
let incoming = agent.clone();
let accept = tokio::spawn(async move {
transport
.accept_loop(move |link| {
let incoming = incoming.clone();
tokio::spawn(async move {
let _ = incoming.install_inbound_link(link).await;
});
})
.await;
});
(agent, accept)
};
Self {
_root: root,
agent,
tunnel,
tuns,
accept,
}
}
fn tun(&self) -> Arc<MemoryTun> {
self.tuns
.device(&self.agent.overlay().unwrap().interface)
.unwrap()
}
fn addr(&self, network: NetworkId) -> Ipv4Addr {
self.tunnel
.overview(network)
.unwrap()
.overlay_address_v4
.unwrap()
}
}
#[tokio::test]
async fn missing_native_transport_cannot_silently_turn_wireguard_into_quic() {
let discovery = SharedMemoryDiscovery::new();
let ports = Arc::default();
let a = Node::configured(&discovery, "wg", Arc::clone(&ports), false).await;
let b = Node::configured(&discovery, "wg", ports, false).await;
let (name, secret) = network("no-disguised-quic-fallback");
let id = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
wait_for_peers(&a.agent, id, 1).await;
let initiator = if a.agent.endpoint_id().as_bytes() < b.agent.endpoint_id().as_bytes() {
&a
} else {
&b
};
wait_until(
"missing carrier implementation is explicitly refused",
|| async {
let status = initiator.agent.network_status(id).await.ok()?;
let peer = status.peers.first()?;
(peer
.data_error
.as_deref()
.is_some_and(|reason| reason.contains("registered"))
&& matches!(peer.data_path, tsunagi::agent::DataPath::None))
.then_some(())
},
)
.await;
a.agent.shutdown().await;
b.agent.shutdown().await;
a.accept.abort();
b.accept.abort();
}
#[tokio::test]
async fn wireguard_and_tls_carriers_both_carry_the_same_internal_cipher_and_exact_plaintext_counters()
{
for protocol in ["wg", "tcp-tls"] {
let discovery = SharedMemoryDiscovery::new();
let ports = Arc::default();
let a = Node::new(&discovery, protocol, Arc::clone(&ports)).await;
let b = Node::new(&discovery, protocol, ports).await;
let (name, secret) = network(protocol);
let id = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
wait_for_peers(&a.agent, id, 1).await;
for node in [&a, &b] {
wait_until("real internal handshake over selected carrier", || async {
node.tunnel
.overview(id)?
.peers
.first()?
.tunnel
.as_ref()?
.health
.is_up()
.then_some(())
})
.await;
let status = node.agent.network_status(id).await.unwrap();
assert_eq!(status.protocols, [protocol]);
assert!(
matches!(&status.peers[0].data_path, tsunagi::agent::DataPath::Direct { transport } if transport == protocol)
);
}
for (source, destination) in [(&a, &b), (&b, &a)] {
let mut total = 0;
for len in [60, 1097, 1280] {
let mut packet = vec![0; len];
packet[0] = 0x45;
packet[2..4].copy_from_slice(&(len as u16).to_be_bytes());
packet[8] = 64;
packet[9] = 253;
packet[12..16].copy_from_slice(&source.addr(id).octets());
packet[16..20].copy_from_slice(&destination.addr(id).octets());
let packet = Bytes::from(packet);
source.tun().push_from_os(packet.clone());
assert_eq!(
tokio::time::timeout(tsunagi::testing::DEADLINE, destination.tun().pop_to_os())
.await
.unwrap()
.unwrap(),
packet
);
total += len as u64;
}
wait_until(
"plaintext bytes count once without health probes or nested encryption overhead",
|| async {
let tx = source
.tunnel
.overview(id)?
.peers
.first()?
.tunnel
.as_ref()?
.stats;
let rx = destination
.tunnel
.overview(id)?
.peers
.first()?
.tunnel
.as_ref()?
.stats;
(tx.tx_bytes == total
&& rx.rx_bytes == total
&& tx.tx_packets == 3
&& rx.rx_packets == 3)
.then_some(())
},
)
.await;
}
a.agent.shutdown().await;
b.agent.shutdown().await;
a.accept.abort();
b.accept.abort();
}
}
crates/tsunagi-wg-quic/tests/exit_node.rs
+4 -1
View File
@@ -101,7 +101,10 @@ impl Node {
tag, tag,
1280, 1280,
) )
.with_plugin(plugin.clone() as Arc<dyn IpPlugin>), .with_tunnel(plugin.clone() as Arc<dyn IpPlugin>)
.with_carrier(Arc::new(
tsunagi::dataplane::transport::iroh_link::IrohCarrier,
)),
) )
.await .await
.unwrap(); .unwrap();
crates/tsunagi-wg-quic/tests/interface_provisioning.rs
+4 -1
View File
@@ -47,7 +47,10 @@ impl HostedAgent {
config_with(dir.path(), discovery) config_with(dir.path(), discovery)
.with_overlay_ipv4_range(Some(DEFAULT_IPV4_RANGE)) .with_overlay_ipv4_range(Some(DEFAULT_IPV4_RANGE))
.with_interface(factory, tag, 1280) .with_interface(factory, tag, 1280)
.with_plugin(plugin.clone() as Arc<dyn IpPlugin>), .with_tunnel(plugin.clone() as Arc<dyn IpPlugin>)
.with_carrier(Arc::new(
tsunagi::dataplane::transport::iroh_link::IrohCarrier,
)),
) )
.await .await
.unwrap(); .unwrap();
crates/tsunagi-wg-quic/tests/local_control.rs
+9 -2
View File
@@ -37,6 +37,7 @@ fn source(agent: Agent, plugin: Arc<WireguardPlugin>) -> Arc<dyn tsunagi::ipc::R
.peers .peers
.iter() .iter()
.map(|peer| tsunagi::ipc::PeerReport { .map(|peer| tsunagi::ipc::PeerReport {
data_error: None,
endpoint_id: peer.endpoint_id.to_string(), endpoint_id: peer.endpoint_id.to_string(),
hostname: peer.hostname.clone(), hostname: peer.hostname.clone(),
transport: format!("{:?}", peer.transport), transport: format!("{:?}", peer.transport),
@@ -107,7 +108,10 @@ async fn a_client_sees_the_agent_and_its_overlay() {
let agent = Agent::spawn( let agent = Agent::spawn(
config_with(dir_a.path(), &discovery) config_with(dir_a.path(), &discovery)
.with_interface(Arc::new(tuns), "tca0", 1280) .with_interface(Arc::new(tuns), "tca0", 1280)
.with_plugin(plugin.clone() as Arc<dyn IpPlugin>), .with_tunnel(plugin.clone() as Arc<dyn IpPlugin>)
.with_carrier(Arc::new(
tsunagi::dataplane::transport::iroh_link::IrohCarrier,
)),
) )
.await .await
.unwrap(); .unwrap();
@@ -122,7 +126,10 @@ async fn a_client_sees_the_agent_and_its_overlay() {
let agent_b = Agent::spawn( let agent_b = Agent::spawn(
config_with(dir_b.path(), &discovery) config_with(dir_b.path(), &discovery)
.with_interface(Arc::new(MemoryTunFactory::new()), "tcb0", 1280) .with_interface(Arc::new(MemoryTunFactory::new()), "tcb0", 1280)
.with_plugin(plugin_b.clone() as Arc<dyn IpPlugin>), .with_tunnel(plugin_b.clone() as Arc<dyn IpPlugin>)
.with_carrier(Arc::new(
tsunagi::dataplane::transport::iroh_link::IrohCarrier,
)),
) )
.await .await
.unwrap(); .unwrap();
crates/tsunagi-wg-quic/tests/wireguard.rs
+280 -18
View File
@@ -121,7 +121,10 @@ impl WgAgent {
let agent = Agent::spawn( let agent = Agent::spawn(
config_with(dir.path(), discovery) config_with(dir.path(), discovery)
.with_interface(Arc::new(tuns.clone()), tag, 1280) .with_interface(Arc::new(tuns.clone()), tag, 1280)
.with_plugin(plugin.clone() as Arc<dyn IpPlugin>), .with_tunnel(plugin.clone() as Arc<dyn IpPlugin>)
.with_carrier(Arc::new(
tsunagi::dataplane::transport::iroh_link::IrohCarrier,
)),
) )
.await .await
.unwrap(); .unwrap();
@@ -193,7 +196,10 @@ impl WgAgent {
config_with(root, discovery) config_with(root, discovery)
.with_overlay_ipv4_range(range) .with_overlay_ipv4_range(range)
.with_interface(Arc::new(tuns.clone()), tag, 1280) .with_interface(Arc::new(tuns.clone()), tag, 1280)
.with_plugin(plugin.clone() as Arc<dyn IpPlugin>), .with_tunnel(plugin.clone() as Arc<dyn IpPlugin>)
.with_carrier(Arc::new(
tsunagi::dataplane::transport::iroh_link::IrohCarrier,
)),
) )
.await .await
.unwrap(); .unwrap();
@@ -225,7 +231,10 @@ impl WgAgent {
.with_overlay_ipv4_range(Some(tsunagi::state::DEFAULT_IPV4_RANGE)) .with_overlay_ipv4_range(Some(tsunagi::state::DEFAULT_IPV4_RANGE))
.with_interface(Arc::new(tuns.clone()), tag, 1280) .with_interface(Arc::new(tuns.clone()), tag, 1280)
.with_unreachable_data_peers(blocked) .with_unreachable_data_peers(blocked)
.with_plugin(plugin.clone() as Arc<dyn IpPlugin>), .with_tunnel(plugin.clone() as Arc<dyn IpPlugin>)
.with_carrier(Arc::new(
tsunagi::dataplane::transport::iroh_link::IrohCarrier,
)),
) )
.await .await
.unwrap(); .unwrap();
@@ -254,7 +263,10 @@ impl WgAgent {
config_with(dir.path(), discovery) config_with(dir.path(), discovery)
.with_overlay_ipv4_range(Some(tsunagi::state::DEFAULT_IPV4_RANGE)) .with_overlay_ipv4_range(Some(tsunagi::state::DEFAULT_IPV4_RANGE))
.with_interface(Arc::new(PretendHostTuns(tuns.clone())), tag, 1280) .with_interface(Arc::new(PretendHostTuns(tuns.clone())), tag, 1280)
.with_plugin(plugin.clone() as Arc<dyn IpPlugin>), .with_tunnel(plugin.clone() as Arc<dyn IpPlugin>)
.with_carrier(Arc::new(
tsunagi::dataplane::transport::iroh_link::IrohCarrier,
)),
) )
.await .await
.unwrap(); .unwrap();
@@ -458,10 +470,52 @@ async fn two_agents_carry_real_ip_packets_through_a_wireguard_tunnel() {
let view = a.plugin.overview(network_id).unwrap(); let view = a.plugin.overview(network_id).unwrap();
let tunnel = view.peers[0].tunnel.as_ref().unwrap(); let tunnel = view.peers[0].tunnel.as_ref().unwrap();
assert!(tunnel.health.is_up()); assert!(tunnel.health.is_up());
assert!(tunnel.stats.tx_packets >= 1); wait_until(
assert!(tunnel.stats.rx_packets >= 1); "plaintext counters exclude carrier probes and encryption overhead",
|| async {
let view = a.plugin.overview(network_id)?;
let stats = view.peers[0].tunnel.as_ref()?.stats;
(stats.tx_packets == 1
&& stats.rx_packets == 1
&& stats.tx_bytes == (20 + payload.len()) as u64
&& stats.rx_bytes == (20 + b"and back".len()) as u64)
.then_some(())
},
)
.await;
assert_eq!(tunnel.stats.dropped_wrong_source, 0); assert_eq!(tunnel.stats.dropped_wrong_source, 0);
let before = a.plugin.overview(network_id).unwrap().peers[0]
.tunnel
.as_ref()
.unwrap()
.stats;
a.agent.set_protocols(network_id, Vec::new()).await.unwrap();
wait_until("no remaining physical path", || async {
matches!(
a.agent.network_status(network_id).await.ok()?.peers[0].data_path,
tsunagi::agent::DataPath::None
)
.then_some(())
})
.await;
assert!(!a.plugin.carry(
network_id,
b.agent.endpoint_id(),
ipv4_packet(addr_a, addr_b, b"refused by transport")
));
let after = a.plugin.overview(network_id).unwrap().peers[0]
.tunnel
.as_ref()
.unwrap()
.stats;
assert_eq!(after.counter_epoch, before.counter_epoch);
assert_eq!(
after.tx_bytes, before.tx_bytes,
"a refused send is not transmitted traffic"
);
assert_eq!(after.tx_packets, before.tx_packets);
a.shutdown().await; a.shutdown().await;
b.shutdown().await; b.shutdown().await;
} }
@@ -1151,10 +1205,12 @@ async fn the_core_carries_the_payload_without_interpreting_it() {
let capability = wait_until("the peer's capability arrived", || async { let capability = wait_until("the peer's capability arrived", || async {
let status: NetworkStatus = a.agent.network_status(network_id).await.ok()?; let status: NetworkStatus = a.agent.network_status(network_id).await.ok()?;
status status.peers.first().and_then(|peer| {
.peers peer.capabilities
.first() .iter()
.and_then(|peer| peer.capabilities.first().cloned()) .find(|cap| cap.protocol == WIREGUARD_PROTOCOL)
.cloned()
})
}) })
.await; .await;
assert_eq!(capability.protocol, WIREGUARD_PROTOCOL); assert_eq!(capability.protocol, WIREGUARD_PROTOCOL);
@@ -1197,7 +1253,10 @@ async fn an_announcement_for_another_network_never_reaches_a_tunnel() {
let attacker_dir = TempDir::new().unwrap(); let attacker_dir = TempDir::new().unwrap();
let attacker = Agent::spawn( let attacker = Agent::spawn(
config_with(attacker_dir.path(), &discovery) config_with(attacker_dir.path(), &discovery)
.with_plugin(forger.clone() as Arc<dyn IpPlugin>), .with_tunnel(forger.clone() as Arc<dyn IpPlugin>)
.with_carrier(Arc::new(
tsunagi::dataplane::transport::iroh_link::IrohCarrier,
)),
) )
.await .await
.unwrap(); .unwrap();
@@ -1246,7 +1305,11 @@ async fn a_peer_at_another_protocol_version_gets_no_data_plane_and_keeps_the_con
let ahead = Arc::new(FromTheFuture); let ahead = Arc::new(FromTheFuture);
let dir = TempDir::new().unwrap(); let dir = TempDir::new().unwrap();
let other = Agent::spawn( let other = Agent::spawn(
config_with(dir.path(), &discovery).with_plugin(ahead.clone() as Arc<dyn IpPlugin>), config_with(dir.path(), &discovery)
.with_tunnel(ahead.clone() as Arc<dyn IpPlugin>)
.with_carrier(Arc::new(
tsunagi::dataplane::transport::iroh_link::IrohCarrier,
)),
) )
.await .await
.unwrap(); .unwrap();
@@ -1273,10 +1336,16 @@ async fn a_peer_at_another_protocol_version_gets_no_data_plane_and_keeps_the_con
); );
let status = here.agent.network_status(network_id).await.unwrap(); let status = here.agent.network_status(network_id).await.unwrap();
assert_eq!(status.peers.len(), 1, "the session is up all the same"); assert_eq!(status.peers.len(), 1, "the session is up all the same");
assert_eq!(
status.peers[0].protocols,
["wg-quic"],
"external capability remains independent"
);
assert!( assert!(
status.peers[0].protocols.is_empty(), status.peers[0]
"nothing was agreed with it: {:?}", .data_error
status.peers[0].protocols .as_deref()
.is_some_and(|reason| reason.contains("update both agents"))
); );
// And the control plane carries a message to it regardless. // And the control plane carries a message to it regardless.
@@ -1317,7 +1386,11 @@ async fn agreement_turns_on_the_protocol_version_and_nothing_else() {
}); });
let dir = TempDir::new().unwrap(); let dir = TempDir::new().unwrap();
let other = Agent::spawn( let other = Agent::spawn(
config_with(dir.path(), &discovery).with_plugin(stranger.clone() as Arc<dyn IpPlugin>), config_with(dir.path(), &discovery)
.with_tunnel(stranger.clone() as Arc<dyn IpPlugin>)
.with_carrier(Arc::new(
tsunagi::dataplane::transport::iroh_link::IrohCarrier,
)),
) )
.await .await
.unwrap(); .unwrap();
@@ -1332,8 +1405,12 @@ async fn agreement_turns_on_the_protocol_version_and_nothing_else() {
status status
.peers .peers
.first()? .first()?
.protocols .capabilities
.contains(&WIREGUARD_PROTOCOL.to_string()) .iter()
.any(|cap| {
cap.protocol == WIREGUARD_PROTOCOL
&& cap.version == tsunagi_wg_quic::ANNOUNCEMENT_VERSION
})
.then_some(()) .then_some(())
}) })
.await; .await;
@@ -1770,3 +1847,188 @@ async fn lan_broadcast_fanout_is_scoped_opt_in_and_does_not_reflood() {
agent.shutdown().await; agent.shutdown().await;
} }
} }
fn vip_segment(
source: Ipv4Addr,
destination: Ipv4Addr,
sport: u16,
dport: u16,
udp: bool,
flags: u8,
) -> Bytes {
fn sum(bytes: &[u8]) -> u16 {
let mut s: u32 = bytes
.chunks(2)
.map(|b| u32::from(u16::from_be_bytes([b[0], *b.get(1).unwrap_or(&0)])))
.sum();
while s >> 16 != 0 {
s = (s & 0xffff) + (s >> 16);
}
!(s as u16)
}
let mut p = vec![0; if udp { 36 } else { 40 }];
p[0] = 0x45;
let len = p.len() as u16;
p[2..4].copy_from_slice(&len.to_be_bytes());
p[8] = 64;
p[9] = if udp { 17 } else { 6 };
p[12..16].copy_from_slice(&source.octets());
p[16..20].copy_from_slice(&destination.octets());
p[20..22].copy_from_slice(&sport.to_be_bytes());
p[22..24].copy_from_slice(&dport.to_be_bytes());
if udp {
p[24..26].copy_from_slice(&16u16.to_be_bytes());
} else {
p[32] = 0x50;
p[33] = flags;
p[24..28].copy_from_slice(&123u32.to_be_bytes());
}
let mut pseudo = p[12..20].to_vec();
pseudo.extend([0, p[9]]);
pseudo.extend(((p.len() - 20) as u16).to_be_bytes());
pseudo.extend(&p[20..]);
let c = sum(&pseudo);
let offset = if udp { 26 } else { 36 };
p[offset..offset + 2].copy_from_slice(&c.to_be_bytes());
let c = sum(&p[..20]);
p[10..12].copy_from_slice(&c.to_be_bytes());
Bytes::from(p)
}
#[tokio::test]
async fn vip_tcp_and_udp_remap_ports_over_real_encryption_and_preserve_source_admission() {
use tsunagi::vip::{Backend, Listener, Protocol, VipConfig};
let discovery = SharedMemoryDiscovery::new();
let (name, secret) = network("vip-encrypted");
let a = WgAgent::spawn(&discovery, "vipa").await;
let b = WgAgent::spawn(&discovery, "vipb").await;
let net = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
a.wait_for_tunnels(net, 1).await;
b.wait_for_tunnels(net, 1).await;
let aa = a.overlay(net).await;
let ab = b.overlay(net).await;
let ta = a.tun(net).await;
let tb = b.tun(net).await;
let mut cfg = VipConfig::new("service");
for (protocol, port, backend) in [(Protocol::Tcp, 443, 6443), (Protocol::Udp, 53, 5353)] {
let mut l = Listener::new(protocol, port);
l.port_check = false;
l.backends.push(Backend {
peer: b.endpoint_id().to_string(),
port: backend,
});
cfg.listeners.push(l);
}
a.agent
.save_vip(net, cfg.clone(), None, false)
.await
.unwrap();
let vip = wait_until("data-plane probes measure the backend RTT", || async {
let status = a.agent.network_status(net).await.ok()?;
let v = status.vips.iter().find(|v| v.config.id == cfg.id)?;
(v.listeners.iter().all(|l| l.selected.is_some()))
.then_some(v.address?)
.filter(|_| v.listeners.len() == 2)
})
.await;
for (udp, port, target) in [(false, 443, 6443), (true, 53, 5353)] {
ta.push_from_os(vip_segment(aa, vip, 40000, port, udp, 2));
let received = tokio::time::timeout(Duration::from_secs(5), tb.pop_to_os())
.await
.unwrap()
.unwrap();
assert_eq!(&received[12..16], &aa.octets());
assert_eq!(&received[16..20], &ab.octets());
assert_eq!(u16::from_be_bytes([received[22], received[23]]), target);
let translated = u16::from_be_bytes([received[20], received[21]]);
tb.push_from_os(vip_segment(ab, aa, target, translated, udp, 0x12));
let reply = tokio::time::timeout(Duration::from_secs(5), ta.pop_to_os())
.await
.unwrap()
.unwrap();
assert_eq!(&reply[12..16], &vip.octets());
assert_eq!(u16::from_be_bytes([reply[20], reply[21]]), port);
assert_eq!(u16::from_be_bytes([reply[22], reply[23]]), 40000);
// A backend cannot send a VIP source over its authenticated tunnel.
let forged = vip_segment(vip, aa, port, 40000, udp, 0x12);
tb.push_from_os(forged);
}
wait_until("forged VIP sources are rejected before NAT", || async {
(a.agent.overlay()?.counters.wrong_source >= 2).then_some(())
})
.await;
a.shutdown().await;
b.shutdown().await;
}
#[tokio::test]
async fn vip_udp_reaches_another_backend_after_the_selected_agent_stops() {
use tsunagi::vip::{Backend, Listener, Protocol, VipConfig};
let discovery = SharedMemoryDiscovery::new();
let (name, secret) = network("vip-failover");
let a = WgAgent::spawn(&discovery, "vipfa").await;
let b = WgAgent::spawn(&discovery, "vipfb").await;
let c = WgAgent::spawn(&discovery, "vipfc").await;
let net = a.agent.join_network(&name, &secret).await.unwrap();
for node in [&b, &c] {
node.agent.join_network(&name, &secret).await.unwrap();
}
for node in [&a, &b, &c] {
node.wait_for_tunnels(net, 2).await;
}
let mut config = VipConfig::new("datagrams");
let mut listener = Listener::new(Protocol::Udp, 8888);
listener.backends = vec![
Backend {
peer: b.endpoint_id().to_string(),
port: 9000,
},
Backend {
peer: c.endpoint_id().to_string(),
port: 9001,
},
];
config.listeners.push(listener);
a.agent
.save_vip(net, config.clone(), None, false)
.await
.unwrap();
let first = wait_until("both UDP backends have measured paths", || async {
let s = a.agent.network_status(net).await.ok()?;
let v = s.vips.iter().find(|v| v.config.id == config.id)?;
let l = v.listeners.first()?;
(l.backends.iter().all(|b| b.rtt_us.is_some())).then_some((v.address?, l.selected.clone()?))
})
.await;
let (dead, alive) = if first.1.peer == b.endpoint_id().to_string() {
(b, c)
} else {
(c, b)
};
let address = a.overlay(net).await;
let tun = a.tun(net).await;
let dest = dead.tun(net).await;
tun.push_from_os(vip_segment(address, first.0, 41000, 8888, true, 0));
tokio::time::timeout(Duration::from_secs(5), dest.pop_to_os())
.await
.unwrap()
.unwrap();
dead.shutdown().await;
wait_until("the remaining backend becomes the next choice", || async {
let s = a.agent.network_status(net).await.ok()?;
let v = s.vips.iter().find(|v| v.config.id == config.id)?;
(v.listeners.first()?.selected.as_ref()?.peer == alive.endpoint_id().to_string())
.then_some(())
})
.await;
tun.push_from_os(vip_segment(address, first.0, 41000, 8888, true, 0));
let packet = tokio::time::timeout(Duration::from_secs(5), alive.tun(net).await.pop_to_os())
.await
.unwrap()
.unwrap();
assert_eq!(&packet[16..20], &alive.overlay(net).await.octets());
let view = a.agent.network_status(net).await.unwrap();
assert_eq!(view.vips[0].listeners[0].counters.failovers, 1);
a.shutdown().await;
alive.shutdown().await;
}
crates/tsunagi-wireguard/Cargo.toml
+3 -7
View File
@@ -1,10 +1,10 @@
[package] [package]
name = "tsunagi-wireguard" name = "tsunagi-wireguard"
version = "0.1.0" version = "0.1.0"
description = "Pure WireGuard direct UDP transport and protocol plugin for tsunagi mesh networks." description = "Independent authenticated WireGuard and Amnezia UDP carriers for tsunagi."
edition.workspace = true edition.workspace = true
rust-version.workspace = true rust-version.workspace = true
license.workspace = true license = "WTFPL AND Apache-2.0"
repository.workspace = true repository.workspace = true
[dependencies] [dependencies]
@@ -14,14 +14,10 @@ tokio = { workspace = true, features = ["net", "io-util", "time", "sync", "macro
serde.workspace = true serde.workspace = true
postcard.workspace = true postcard.workspace = true
bytes.workspace = true bytes.workspace = true
thiserror.workspace = true
tracing.workspace = true tracing.workspace = true
data-encoding.workspace = true
hex.workspace = true
rusqlite = { version = "0.40", features = ["bundled"] }
subtle = "2.6"
zeroize = { version = "1.9", features = ["derive"] } zeroize = { version = "1.9", features = ["derive"] }
rand = "0.10" rand = "0.10"
chacha20 = "0.9"
boringtun = { version = "0.7.1", default-features = false } boringtun = { version = "0.7.1", default-features = false }
netdev = "0.46" netdev = "0.46"
crates/tsunagi-wireguard/LICENSE-AmneziaWG-RS
+201
View File
@@ -0,0 +1,201 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright 2026 Friderika Coulson (NanoBee)
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
crates/tsunagi-wireguard/src/addresses.rs
+168
View File
@@ -0,0 +1,168 @@
//! Local underlay candidates, shared by the UDP carriers.
use std::net::{IpAddr, Ipv4Addr, SocketAddr};
/// Discovers local network interface IPs to announce to peers, excluding loopback, overlay interfaces, and virtual bridges.
pub fn local_ip_candidates() -> Vec<IpAddr> {
local_ip_candidates_excluding(None)
}
pub(crate) fn local_ip_candidates_excluding(
excluded_interface: Option<&str>,
) -> Vec<std::net::IpAddr> {
let mut candidates = Vec::new();
for iface in netdev::get_interfaces() {
if excluded_interface.is_some_and(|name| {
iface.name.eq_ignore_ascii_case(name)
|| iface
.friendly_name
.as_deref()
.is_some_and(|friendly| friendly.eq_ignore_ascii_case(name))
}) || !is_candidate_interface(&iface)
{
continue;
}
for ip in &iface.ipv4 {
let addr = ip.addr();
if is_candidate_ipv4(addr) {
let priority = ip_priority(&iface, IpAddr::V4(addr));
candidates.push((priority, IpAddr::V4(addr)));
}
}
}
// Sort descending by priority so the most viable physical / WAN / LAN IP is candidate #0
candidates.sort_by_key(|b| std::cmp::Reverse(b.0));
let mut seen = std::collections::HashSet::new();
let mut ips = Vec::new();
for (_, ip) in candidates {
if seen.insert(ip) {
ips.push(ip);
}
}
ips
}
pub(crate) fn is_candidate_interface(iface: &netdev::Interface) -> bool {
if !iface.is_up() || iface.is_loopback() {
return false;
}
let name = iface.name.to_lowercase();
let friendly = iface
.friendly_name
.as_deref()
.map(str::to_lowercase)
.unwrap_or_default();
let desc = iface
.description
.as_deref()
.map(str::to_lowercase)
.unwrap_or_default();
// Skip Tsunagi / Wintun overlay interfaces
if name.starts_with("tsun")
|| name.contains("tsunagi")
|| name.contains("wintun")
|| friendly.starts_with("tsun")
|| friendly.contains("tsunagi")
|| friendly.contains("wintun")
|| desc.contains("wintun")
|| desc.contains("tsunagi")
{
return false;
}
// Skip container interfaces and virtual bridges purely internal to container engines
if name.starts_with("docker")
|| name.starts_with("br-")
|| name.starts_with("veth")
|| name.starts_with("cni")
|| name.starts_with("flannel")
|| name.starts_with("cali")
|| name.starts_with("kube")
|| name.starts_with("dummy")
|| name.starts_with("lxc")
|| name.starts_with("podman")
|| friendly.contains("wsl")
{
return false;
}
true
}
/// Reorders candidate addresses so that candidates on the same local subnet as one of our
/// active network interfaces appear first.
pub fn prioritize_local_subnet(candidates: &mut [SocketAddr]) {
let local_subnets: Vec<(Ipv4Addr, u8)> = netdev::get_interfaces()
.into_iter()
.filter(|iface| iface.is_up() && !iface.is_loopback())
.flat_map(|iface| {
iface
.ipv4
.into_iter()
.map(|net| (net.addr(), net.prefix_len()))
})
.collect();
candidates.sort_by_key(|addr| {
let is_local = match addr.ip() {
IpAddr::V4(v4) => local_subnets
.iter()
.any(|(local_ip, prefix_len)| matches_subnet(*local_ip, *prefix_len, v4)),
IpAddr::V6(_) => false,
};
if is_local { 0 } else { 1 }
});
}
fn matches_subnet(local_ip: Ipv4Addr, prefix_len: u8, target_ip: Ipv4Addr) -> bool {
if prefix_len == 0 {
return false;
}
let mask = if prefix_len >= 32 {
!0u32
} else {
!0u32 << (32 - prefix_len)
};
(u32::from(local_ip) & mask) == (u32::from(target_ip) & mask)
}
pub(crate) fn is_candidate_ipv4(addr: std::net::Ipv4Addr) -> bool {
if addr.is_loopback() || addr.is_unspecified() || addr.is_link_local() || addr.is_broadcast() {
return false;
}
if addr.is_documentation() {
return false;
}
true
}
pub(crate) fn ip_priority(iface: &netdev::Interface, ip: IpAddr) -> u32 {
let has_gateway = iface.gateway.is_some();
let base = if has_gateway { 100 } else { 0 };
match ip {
IpAddr::V4(v4) => {
let octets = v4.octets();
// Tailscale: 100.64.0.0/10
let is_tailscale = octets[0] == 100 && (octets[1] & 0xc0) == 64;
let is_private = v4.is_private();
if !is_private && !is_tailscale {
// Public / globally routable IP
base + 50
} else if has_gateway && is_private {
// Primary LAN IP with gateway (e.g. 192.168.1.117)
base + 40
} else if is_tailscale {
// Tailscale / CGNAT
base + 30
} else {
// Other private / virtual
base + 10
}
}
IpAddr::V6(v6) => {
let segments = v6.segments();
let is_global = (segments[0] & 0xe000) == 0x2000;
if is_global { base + 45 } else { base + 20 }
}
}
}
crates/tsunagi-wireguard/src/announcement.rs
-142
View File
@@ -1,142 +0,0 @@
//! What a pure WireGuard peer tells the network about itself.
//!
//! Exchanged via [`tsunagi::dataplane::PluginCapability`]. The announcement
//! carries the WireGuard public key, the listening UDP port, and candidate IP addresses
//! for direct UDP packet transport.
use serde::{Deserialize, Serialize};
use tsunagi::dataplane::PluginError;
use tsunagi::identity::NetworkId;
use crate::keys::WgPublicKey;
/// The wire protocol identifier for pure WireGuard.
pub const WG_PROTOCOL: &str = "wg";
/// Wire version of the `wg` announcement format.
pub const ANNOUNCEMENT_VERSION: u16 = 1;
/// Wire version peers compare during capability exchange.
///
/// This is the same as [`ANNOUNCEMENT_VERSION`]; keeping a separate name
/// makes the intent explicit in the CLI protocol registry.
pub const WG_WIRE_VERSION: u16 = ANNOUNCEMENT_VERSION;
/// Maximum number of IP candidates allowed in an announcement.
pub const MAX_CANDIDATE_ADDRS: usize = 32;
/// What one participant advertises for the pure WireGuard UDP data plane.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct WgAnnouncement {
/// Announcement format version.
pub version: u16,
/// The peer's WireGuard public key.
pub public_key: [u8; 32],
/// The network this key is for.
pub network: [u8; 32],
/// The UDP port the peer is listening on for WireGuard packets.
pub port: u16,
/// Candidate IP addresses where this peer can be reached on `port`.
#[serde(default)]
pub addrs: Vec<std::net::IpAddr>,
}
/// A peer announcement that has been validated against a specific network.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ValidatedAnnouncement {
/// The peer's WireGuard public key.
pub public_key: WgPublicKey,
/// The UDP port the peer listens on.
pub port: u16,
/// Validated candidate IP addresses.
pub addrs: Vec<std::net::IpAddr>,
}
impl WgAnnouncement {
/// Builds this agent's announcement.
pub fn new(
network: NetworkId,
public_key: &WgPublicKey,
port: u16,
addrs: Vec<std::net::IpAddr>,
) -> Self {
Self {
version: ANNOUNCEMENT_VERSION,
public_key: *public_key.as_bytes(),
network: *network.as_bytes(),
port,
addrs,
}
}
/// Encodes the announcement into the opaque capability payload.
pub fn encode(&self) -> Result<Vec<u8>, PluginError> {
postcard::to_allocvec(self)
.map_err(|err| PluginError::Other(format!("cannot encode announcement: {err}")))
}
/// Decodes and validates a payload received from a peer.
pub fn decode_and_validate(
payload: &[u8],
network: NetworkId,
local_key: &WgPublicKey,
) -> Result<ValidatedAnnouncement, PluginError> {
let announcement: Self = postcard::from_bytes(payload)
.map_err(|_| PluginError::Rejected("malformed WireGuard announcement".into()))?;
announcement.validate(network, local_key)
}
fn validate(
self,
network: NetworkId,
local_key: &WgPublicKey,
) -> Result<ValidatedAnnouncement, PluginError> {
if self.version != ANNOUNCEMENT_VERSION {
return Err(PluginError::Rejected(format!(
"peer speaks WireGuard announcement version {} but this build speaks \
{ANNOUNCEMENT_VERSION}; one of the two needs updating",
self.version
)));
}
let public_key = WgPublicKey::from_bytes(self.public_key);
if public_key.is_zero() {
return Err(PluginError::Rejected(
"WireGuard public key is all zeroes".into(),
));
}
if &public_key == local_key {
return Err(PluginError::Rejected(
"peer announced this agent's own WireGuard key".into(),
));
}
if self.network != *network.as_bytes() {
return Err(PluginError::Rejected(
"announcement is for a different network".into(),
));
}
if self.port == 0 {
return Err(PluginError::Rejected("WireGuard port cannot be 0".into()));
}
if self.addrs.len() > MAX_CANDIDATE_ADDRS {
return Err(PluginError::Rejected(format!(
"too many candidate addresses: {} > {MAX_CANDIDATE_ADDRS}",
self.addrs.len()
)));
}
let mut valid_addrs = Vec::new();
for addr in self.addrs {
if !addr.is_unspecified() && !valid_addrs.contains(&addr) {
valid_addrs.push(addr);
}
}
Ok(ValidatedAnnouncement {
public_key,
port: self.port,
addrs: valid_addrs,
})
}
}
crates/tsunagi-wireguard/src/device.rs
-686
View File
@@ -1,686 +0,0 @@
//! Userspace WireGuard device implementation over direct UDP links.
//!
//! Uses [`boringtun::noise::Tunn`] state machine to encrypt and decrypt
//! standard WireGuard protocol packets directly over a [`SharedLink`].
use std::collections::{HashMap, VecDeque};
use std::net::Ipv4Addr;
use std::sync::atomic::{AtomicU32, AtomicU64, Ordering};
use std::sync::{Arc, Mutex, RwLock};
use std::time::Duration;
use boringtun::noise::{Tunn, TunnResult};
use bytes::Bytes;
use iroh::EndpointId;
use tokio::task::JoinHandle;
use tsunagi::dataplane::routing::{FlowId, flow::ip_flow};
use tsunagi::dataplane::transport::{SharedLink, TransportError};
use tsunagi::dataplane::{PacketSink, PluginContext, PluginError};
use tsunagi::identity::NetworkId;
use crate::keys::{WgPublicKey, WgSecretKey};
use crate::transport::WireguardTransport;
const TIMER_INTERVAL: Duration = Duration::from_millis(250);
const SCRATCH: usize = 4096;
/// Maximum consecutive rekey timeout retransmissions before declaring a tunnel dead.
pub const MAX_REKEY_TIMEOUTS: u32 = 3;
/// Overall maximum duration trying to establish or rekey a handshake before declaring it dead.
pub const HANDSHAKE_TIMEOUT: Duration = Duration::from_secs(15);
struct FlowTunnel {
tunn: Tunn,
queued: VecDeque<FlowId>,
handshake_started_at: Option<std::time::Instant>,
rekey_timeouts: u32,
failed: bool,
}
fn is_data(result: &TunnResult<'_>) -> bool {
matches!(result, TunnResult::WriteToNetwork(bytes) if bytes.starts_with(&[4, 0, 0, 0]))
}
impl FlowTunnel {
fn encapsulate<'a>(
&mut self,
packet: &[u8],
scratch: &'a mut [u8],
flow: FlowId,
) -> (TunnResult<'a>, FlowId) {
let result = self.tunn.encapsulate(packet, scratch);
if is_data(&result) {
return (result, flow);
}
if matches!(result, TunnResult::WriteToNetwork(ref b) if !b.is_empty() && b[0] == 1)
&& self.handshake_started_at.is_none()
{
self.handshake_started_at = Some(std::time::Instant::now());
self.rekey_timeouts = 0;
self.failed = false;
}
if self.queued.len() < 256 {
self.queued.push_back(flow);
}
(result, 0)
}
fn decapsulate<'a>(
&mut self,
packet: &[u8],
scratch: &'a mut [u8],
) -> (TunnResult<'a>, FlowId) {
let result = self.tunn.decapsulate(None, packet, scratch);
if self.tunn.time_since_last_handshake().is_some() {
self.handshake_started_at = None;
self.rekey_timeouts = 0;
self.failed = false;
}
let mut flow = 0;
if packet.is_empty() {
if is_data(&result) {
flow = self.queued.pop_front().unwrap_or(0);
} else if let Some(pending) = self.queued.pop_front() {
self.queued.push_back(pending);
}
}
(result, flow)
}
fn update_timers<'a>(&mut self, scratch: &'a mut [u8]) -> (TunnResult<'a>, bool) {
if self.failed {
return (TunnResult::Done, false);
}
let result = self.tunn.update_timers(scratch);
if matches!(
result,
TunnResult::Err(boringtun::noise::errors::WireGuardError::ConnectionExpired)
) || self.tunn.is_expired()
{
self.failed = true;
self.queued.clear();
self.handshake_started_at = None;
self.rekey_timeouts = 0;
return (result, true);
}
if let TunnResult::WriteToNetwork(ref bytes) = result
&& !bytes.is_empty()
&& bytes[0] == 1
&& self.handshake_started_at.is_none()
{
self.handshake_started_at = Some(std::time::Instant::now());
}
if let Some(started) = self.handshake_started_at
&& started.elapsed() >= HANDSHAKE_TIMEOUT
{
self.failed = true;
self.queued.clear();
self.handshake_started_at = None;
self.rekey_timeouts = 0;
return (result, true);
}
(result, false)
}
}
#[derive(Debug, Default)]
struct PeerCounters {
tx_packets: AtomicU64,
tx_bytes: AtomicU64,
rx_packets: AtomicU64,
rx_bytes: AtomicU64,
dropped_wrong_source: AtomicU64,
dropped_oversize: AtomicU64,
dropped_no_session: AtomicU64,
protocol_errors: AtomicU64,
}
/// A snapshot of one peer's tunnel counters.
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub struct PeerStats {
/// Plaintext packets encrypted and sent to this peer.
pub tx_packets: u64,
/// Plaintext bytes encrypted and sent to this peer.
pub tx_bytes: u64,
/// Plaintext packets decrypted from this peer and given to the OS.
pub rx_packets: u64,
/// Plaintext bytes decrypted from this peer and given to the OS.
pub rx_bytes: u64,
/// Packets dropped because their source was not this peer's address.
pub dropped_wrong_source: u64,
/// Packets dropped because they did not fit in one link datagram.
pub dropped_oversize: u64,
/// Packets dropped because there was no session to encrypt with yet.
pub dropped_no_session: u64,
/// WireGuard protocol errors, including packets that failed to decrypt.
pub protocol_errors: u64,
}
/// Whether a peer's tunnel has completed a handshake.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct PeerHealth {
/// Time since the last successful WireGuard handshake.
pub since_handshake: Option<Duration>,
}
impl PeerHealth {
/// Whether the tunnel has ever completed a handshake.
pub fn is_up(&self) -> bool {
self.since_handshake.is_some()
}
}
struct Peer {
endpoint_id: EndpointId,
public_key: WgPublicKey,
overlay_v4: Mutex<Option<Ipv4Addr>>,
tunn: Mutex<FlowTunnel>,
link: SharedLink,
counters: Arc<PeerCounters>,
task: Mutex<Option<JoinHandle<()>>>,
}
impl std::fmt::Debug for Peer {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("Peer")
.field("peer", &self.endpoint_id.fmt_short().to_string())
.field("public_key", &self.public_key)
.finish()
}
}
impl Drop for Peer {
fn drop(&mut self) {
if let Ok(mut guard) = self.task.lock()
&& let Some(task) = guard.take()
{
task.abort();
}
}
}
impl Peer {
fn stats(&self) -> PeerStats {
PeerStats {
tx_packets: self.counters.tx_packets.load(Ordering::Relaxed),
tx_bytes: self.counters.tx_bytes.load(Ordering::Relaxed),
rx_packets: self.counters.rx_packets.load(Ordering::Relaxed),
rx_bytes: self.counters.rx_bytes.load(Ordering::Relaxed),
dropped_wrong_source: self.counters.dropped_wrong_source.load(Ordering::Relaxed),
dropped_oversize: self.counters.dropped_oversize.load(Ordering::Relaxed),
dropped_no_session: self.counters.dropped_no_session.load(Ordering::Relaxed),
protocol_errors: self.counters.protocol_errors.load(Ordering::Relaxed),
}
}
fn health(&self) -> PeerHealth {
let guard = match self.tunn.lock() {
Ok(guard) => guard,
Err(poisoned) => poisoned.into_inner(),
};
PeerHealth {
since_handshake: guard.tunn.time_since_last_handshake(),
}
}
}
/// What a peer's tunnel looks like from outside.
#[derive(Debug, Clone)]
pub struct PeerSummary {
/// The peer's control plane identity.
pub endpoint_id: EndpointId,
/// The peer's WireGuard public key.
pub public_key: WgPublicKey,
/// The overlay address it holds, once agreed.
pub overlay_address_v4: Option<Ipv4Addr>,
/// Whether the tunnel has handshaken.
pub health: PeerHealth,
/// Traffic counters.
pub stats: PeerStats,
/// What the transport reports about the path carrying this tunnel.
pub path: String,
/// Largest datagram the link accepts.
pub max_datagram: usize,
}
struct Inner {
network: NetworkId,
private_key: WgSecretKey,
peers: RwLock<HashMap<WgPublicKey, Arc<Peer>>>,
sink: Arc<dyn PacketSink>,
transport: Arc<WireguardTransport>,
context: PluginContext,
next_index: AtomicU32,
}
impl Inner {
fn on_tunnel_failed(&self, peer: EndpointId, public_key: WgPublicKey) {
self.transport.close_link(self.network, peer);
write_lock(&self.peers).remove(&public_key);
self.context.report_peer_protocol_failure(
self.network,
peer,
crate::announcement::WG_PROTOCOL,
"WireGuard rekey timeout",
);
}
}
impl std::fmt::Debug for Inner {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("Inner")
.field("network", &self.network.fmt_short())
.finish()
}
}
/// The userspace WireGuard tunnels of one network.
#[derive(Debug)]
pub struct WireguardDevice {
inner: Arc<Inner>,
tasks: Vec<JoinHandle<()>>,
}
impl WireguardDevice {
/// Starts the WireGuard tunnels for one network.
pub fn start(
network: NetworkId,
private_key: WgSecretKey,
sink: Arc<dyn PacketSink>,
transport: Arc<WireguardTransport>,
context: PluginContext,
) -> Self {
let inner = Arc::new(Inner {
network,
private_key,
peers: RwLock::new(HashMap::new()),
sink,
transport,
context,
next_index: AtomicU32::new(1),
});
let timers = tokio::spawn(drive_timers(Arc::clone(&inner)));
Self {
inner,
tasks: vec![timers],
}
}
/// Encrypts an IP packet and sends it to a peer over its direct link.
pub fn carry(&self, peer: EndpointId, packet: &[u8]) -> bool {
let found = read_lock(&self.inner.peers)
.values()
.find(|candidate| candidate.endpoint_id == peer)
.cloned();
let Some(peer) = found else {
return false;
};
let mut scratch = [0u8; SCRATCH];
let flow = ip_flow(packet);
let result = {
let mut tunn = match peer.tunn.lock() {
Ok(guard) => guard,
Err(poisoned) => poisoned.into_inner(),
};
let (result, selected_flow) = tunn.encapsulate(packet, &mut scratch, flow);
match result {
TunnResult::WriteToNetwork(out) => Some((out.len(), selected_flow)),
_ => None,
}
};
let Some((len, selected_flow)) = result else {
peer.counters
.dropped_no_session
.fetch_add(1, Ordering::Relaxed);
return false;
};
send_to_peer_flow(&peer, &scratch[..len], selected_flow);
peer.counters.tx_packets.fetch_add(1, Ordering::Relaxed);
peer.counters
.tx_bytes
.fetch_add(packet.len() as u64, Ordering::Relaxed);
true
}
/// Adds or replaces a peer and starts its tunnel.
pub fn add_peer(
&self,
endpoint_id: EndpointId,
public_key: WgPublicKey,
overlay_v4: Option<Ipv4Addr>,
link: SharedLink,
keepalive: Option<u16>,
) -> Result<(), PluginError> {
if public_key == self.inner.private_key.public() {
return Err(PluginError::Rejected(
"refusing to add ourselves as a WireGuard peer".into(),
));
}
let index = self.inner.next_index.fetch_add(1, Ordering::Relaxed);
let tunn = Tunn::new(
self.inner.private_key.to_static_secret(),
public_key.into_x25519(),
None,
keepalive,
index,
None,
);
let peer = Arc::new(Peer {
endpoint_id,
public_key,
overlay_v4: Mutex::new(overlay_v4),
tunn: Mutex::new(FlowTunnel {
tunn,
queued: VecDeque::new(),
handshake_started_at: None,
rekey_timeouts: 0,
failed: false,
}),
link,
counters: Arc::new(PeerCounters::default()),
task: Mutex::new(None),
});
let task = tokio::spawn(read_from_link(Arc::clone(&self.inner), Arc::clone(&peer)));
if let Ok(mut guard) = peer.task.lock() {
*guard = Some(task);
}
write_lock(&self.inner.peers).insert(public_key, Arc::clone(&peer));
kick_handshake(&peer);
Ok(())
}
/// Removes a peer and stops its tunnel.
pub fn remove_peer(&self, public_key: &WgPublicKey) {
write_lock(&self.inner.peers).remove(public_key);
}
/// Removes every peer whose key is not in `keep`.
pub fn retain_peers(&self, keep: &[WgPublicKey]) {
let stale: Vec<WgPublicKey> = read_lock(&self.inner.peers)
.keys()
.filter(|key| !keep.contains(key))
.copied()
.collect();
for key in stale {
self.remove_peer(&key);
}
}
/// Whether a peer's tunnel exists.
pub fn has_peer(&self, public_key: &WgPublicKey) -> bool {
read_lock(&self.inner.peers).contains_key(public_key)
}
/// A snapshot of every peer.
pub fn peers(&self) -> Vec<PeerSummary> {
let mut peers: Vec<PeerSummary> = read_lock(&self.inner.peers)
.values()
.map(|peer| {
let overlay_v4 = match peer.overlay_v4.lock() {
Ok(guard) => *guard,
Err(poisoned) => *poisoned.into_inner(),
};
PeerSummary {
endpoint_id: peer.endpoint_id,
public_key: peer.public_key,
overlay_address_v4: overlay_v4,
health: peer.health(),
stats: peer.stats(),
path: peer.link.path_description(),
max_datagram: peer.link.max_datagram_size(),
}
})
.collect();
peers.sort_by_key(|peer| peer.public_key);
peers
}
}
impl Drop for WireguardDevice {
fn drop(&mut self) {
for task in &self.tasks {
task.abort();
}
}
}
fn read_lock<T>(lock: &RwLock<T>) -> std::sync::RwLockReadGuard<'_, T> {
match lock.read() {
Ok(guard) => guard,
Err(poisoned) => poisoned.into_inner(),
}
}
fn write_lock<T>(lock: &RwLock<T>) -> std::sync::RwLockWriteGuard<'_, T> {
match lock.write() {
Ok(guard) => guard,
Err(poisoned) => poisoned.into_inner(),
}
}
fn kick_handshake(peer: &Peer) {
let mut scratch = vec![0u8; SCRATCH];
let len = {
let mut tunn = match peer.tunn.lock() {
Ok(guard) => guard,
Err(poisoned) => poisoned.into_inner(),
};
match tunn.tunn.format_handshake_initiation(&mut scratch, false) {
TunnResult::WriteToNetwork(out) => {
tunn.handshake_started_at = Some(std::time::Instant::now());
tunn.rekey_timeouts = 0;
tunn.failed = false;
Some(out.len())
}
_ => None,
}
};
if let Some(len) = len {
send_to_peer(peer, &scratch[..len]);
}
}
fn send_to_peer(peer: &Peer, payload: &[u8]) {
send_to_peer_flow(peer, payload, 0);
}
fn send_to_peer_flow(peer: &Peer, payload: &[u8], flow: FlowId) {
match peer.link.send_flow(Bytes::copy_from_slice(payload), flow) {
Ok(()) => {}
Err(TransportError::TooLarge { .. }) => {
peer.counters
.dropped_oversize
.fetch_add(1, Ordering::Relaxed);
}
Err(TransportError::Closed) => {}
Err(err) => {
tracing::trace!(%err, "dropping a WireGuard packet the link refused");
}
}
}
async fn read_from_link(inner: Arc<Inner>, peer: Arc<Peer>) {
let mut scratch = vec![0u8; SCRATCH];
loop {
let Some(datagram) = peer.link.recv().await else {
return;
};
let mut input: Option<&[u8]> = Some(&datagram);
loop {
let outcome = {
let mut tunn = match peer.tunn.lock() {
Ok(guard) => guard,
Err(poisoned) => poisoned.into_inner(),
};
let (result, flow) = tunn.decapsulate(input.unwrap_or(&[]), &mut scratch);
match result {
TunnResult::WriteToNetwork(out) => Outcome::ToNetwork(out.len(), flow),
TunnResult::WriteToTunnelV6(out, _) => Outcome::ToTunnel(out.len()),
TunnResult::WriteToTunnelV4(out, _) => Outcome::ToTunnel(out.len()),
TunnResult::Done => Outcome::Done,
TunnResult::Err(err) => {
tracing::trace!(?err, "WireGuard decapsulation failed");
Outcome::Failed
}
}
};
match outcome {
Outcome::ToNetwork(len, flow) => {
send_to_peer_flow(&peer, &scratch[..len], flow);
input = None;
continue;
}
Outcome::ToTunnel(len) => {
let payload = Bytes::copy_from_slice(&scratch[..len]);
inner
.sink
.deliver(inner.network, peer.endpoint_id, payload)
.await;
peer.counters.rx_packets.fetch_add(1, Ordering::Relaxed);
peer.counters
.rx_bytes
.fetch_add(len as u64, Ordering::Relaxed);
break;
}
Outcome::Failed => {
peer.counters
.protocol_errors
.fetch_add(1, Ordering::Relaxed);
break;
}
Outcome::Done => break,
}
}
}
}
enum Outcome {
ToNetwork(usize, FlowId),
ToTunnel(usize),
Done,
Failed,
}
async fn drive_timers(inner: Arc<Inner>) {
let mut ticker = tokio::time::interval(TIMER_INTERVAL);
ticker.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay);
loop {
ticker.tick().await;
let peers: Vec<Arc<Peer>> = read_lock(&inner.peers).values().cloned().collect();
for peer in peers {
let mut scratch = vec![0u8; SCRATCH];
let (len, failed) = {
let mut tunn = match peer.tunn.lock() {
Ok(guard) => guard,
Err(poisoned) => poisoned.into_inner(),
};
let (result, failed) = tunn.update_timers(&mut scratch);
let len = match result {
TunnResult::WriteToNetwork(out) => Some(out.len()),
TunnResult::Err(err) => {
tracing::trace!(?err, "WireGuard timer error");
None
}
_ => None,
};
(len, failed)
};
if let Some(len) = len {
send_to_peer(&peer, &scratch[..len]);
}
if failed {
tracing::warn!(
peer = %peer.endpoint_id.fmt_short(),
"WireGuard tunnel failed due to rekey timeout, triggering fallback"
);
inner.on_tunnel_failed(peer.endpoint_id, peer.public_key);
}
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_flow_tunnel_rekey_timeout_triggers_failure() {
let my_secret = WgSecretKey::generate();
let their_secret = WgSecretKey::generate();
let their_public = their_secret.public();
let tunn = Tunn::new(
my_secret.to_static_secret(),
their_public.into_x25519(),
None,
None,
1,
None,
);
let mut tunnel = FlowTunnel {
tunn,
queued: VecDeque::new(),
handshake_started_at: Some(std::time::Instant::now() - Duration::from_secs(16)),
rekey_timeouts: 0,
failed: false,
};
let mut scratch = vec![0u8; SCRATCH];
// When handshake timeout exceeds HANDSHAKE_TIMEOUT (15s), update_timers returns true (failed)
let (_, failed) = tunnel.update_timers(&mut scratch);
assert!(failed, "handshake timeout should report failure");
assert!(tunnel.handshake_started_at.is_none());
assert!(tunnel.failed, "tunnel should be marked as failed");
// Subsequent timer tick must not report failure again
let (_, failed_again) = tunnel.update_timers(&mut scratch);
assert!(
!failed_again,
"subsequent timer tick should not re-trigger failure"
);
}
#[test]
fn test_flow_tunnel_reset_on_handshake() {
let my_secret = WgSecretKey::generate();
let their_secret = WgSecretKey::generate();
let their_public = their_secret.public();
let tunn = Tunn::new(
my_secret.to_static_secret(),
their_public.into_x25519(),
None,
None,
1,
None,
);
let mut tunnel = FlowTunnel {
tunn,
queued: VecDeque::new(),
handshake_started_at: Some(std::time::Instant::now()),
rekey_timeouts: 2,
failed: true,
};
let mut scratch = vec![0u8; SCRATCH];
let _ = tunnel.decapsulate(&[], &mut scratch);
// decapsulate without handshake response doesn't establish session
assert_eq!(tunnel.rekey_timeouts, 2);
}
}
crates/tsunagi-wireguard/src/keys.rs
-160
View File
@@ -1,160 +0,0 @@
//! WireGuard key material for pure WireGuard transport.
//!
//! Keys belong to this protocol plugin alone. They are not derived from
//! the iroh device key and not derived from the network secret, so compromising
//! or rotating one does not affect the others.
//!
//! Keys are X25519, encoded the way WireGuard encodes them: standard base64
//! with padding, 44 characters.
use boringtun::x25519;
use data_encoding::BASE64;
use zeroize::{Zeroize, Zeroizing};
use tsunagi::dataplane::PluginError;
/// Length of a raw WireGuard key in bytes.
pub const KEY_LEN: usize = 32;
/// Length of the base64 text form of a key.
pub const KEY_TEXT_LEN: usize = 44;
/// Applies the X25519 clamping WireGuard applies to private keys.
fn clamp(bytes: &mut [u8; KEY_LEN]) {
bytes[0] &= 248;
bytes[31] &= 127;
bytes[31] |= 64;
}
/// A WireGuard public key.
///
/// Public, safe to log, and the identity a peer is known by inside the overlay.
#[derive(Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub struct WgPublicKey([u8; KEY_LEN]);
impl WgPublicKey {
/// Wraps raw key bytes.
pub fn from_bytes(bytes: [u8; KEY_LEN]) -> Self {
Self(bytes)
}
/// The raw key bytes.
pub fn as_bytes(&self) -> &[u8; KEY_LEN] {
&self.0
}
/// The key in the form the WireGuard implementation expects.
pub(crate) fn into_x25519(self) -> x25519::PublicKey {
x25519::PublicKey::from(self.0)
}
/// Whether this is the all-zero key, which is never a valid peer.
pub fn is_zero(&self) -> bool {
self.0 == [0u8; KEY_LEN]
}
/// The base64 form WireGuard uses.
pub fn encode(&self) -> String {
BASE64.encode(&self.0)
}
/// Parses the base64 form WireGuard uses.
pub fn decode(text: &str) -> Result<Self, PluginError> {
if text.len() != KEY_TEXT_LEN {
return Err(PluginError::Rejected(format!(
"a WireGuard key is {KEY_TEXT_LEN} base64 characters, got {}",
text.len()
)));
}
let raw = BASE64
.decode(text.as_bytes())
.map_err(|_| PluginError::Rejected("key is not valid base64".into()))?;
let bytes = <[u8; KEY_LEN]>::try_from(raw.as_slice())
.map_err(|_| PluginError::Rejected("key is not 32 bytes".into()))?;
Ok(Self(bytes))
}
/// A short prefix for logs and diagnostics.
pub fn fmt_short(&self) -> String {
self.encode().chars().take(8).collect()
}
}
impl std::fmt::Display for WgPublicKey {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str(&self.encode())
}
}
impl std::fmt::Debug for WgPublicKey {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "WgPublicKey({})", self.fmt_short())
}
}
/// A WireGuard private key.
///
/// Clamped on creation so it matches what `wg genkey` produces. Zeroized on
/// drop, and prints as `[redacted]` in Debug so it cannot leak into logs.
#[derive(Clone)]
pub struct WgSecretKey {
bytes: Zeroizing<[u8; KEY_LEN]>,
public: WgPublicKey,
}
impl WgSecretKey {
/// Generates a fresh clamped random private key.
pub fn generate() -> Self {
let mut bytes = [0u8; KEY_LEN];
rand::fill(&mut bytes);
clamp(&mut bytes);
let secret = x25519::StaticSecret::from(bytes);
let public = WgPublicKey(*x25519::PublicKey::from(&secret).as_bytes());
Self {
bytes: Zeroizing::new(bytes),
public,
}
}
/// Wraps existing key bytes, clamping them to a valid X25519 private key.
pub fn from_bytes(raw: &[u8; KEY_LEN]) -> Self {
let mut bytes = *raw;
clamp(&mut bytes);
let secret = x25519::StaticSecret::from(bytes);
let public = WgPublicKey(*x25519::PublicKey::from(&secret).as_bytes());
Self {
bytes: Zeroizing::new(bytes),
public,
}
}
/// The matching public key.
pub fn public(&self) -> WgPublicKey {
self.public
}
/// The key in the form the WireGuard implementation expects.
pub(crate) fn to_static_secret(&self) -> x25519::StaticSecret {
x25519::StaticSecret::from(*self.bytes)
}
/// The raw key bytes.
pub fn as_bytes(&self) -> &[u8; KEY_LEN] {
&self.bytes
}
}
impl Drop for WgSecretKey {
fn drop(&mut self) {
self.bytes.zeroize();
}
}
impl std::fmt::Debug for WgSecretKey {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("WgSecretKey")
.field("bytes", &"[redacted]")
.field("public", &self.public)
.finish()
}
}
crates/tsunagi-wireguard/src/lib.rs
+19 -26
View File
@@ -1,28 +1,21 @@
//! The pure WireGuard protocol plugin for Tsunagi: WireGuard cryptography over direct UDP. //! Authenticated, per-neighbour WireGuard UDP carriers. IP encryption is independent.
//!
//! Deliberately a separate crate from the core, ensuring clear boundaries:
//! - Direct UDP datagram transport without QUIC encapsulation or double-encryption.
//! - Standard WireGuard cryptography via `boringtun`.
//! - Peer discovery and key exchange managed through the Tsunagi control plane.
#![deny(unsafe_code)] #![deny(unsafe_code)]
#![warn(missing_docs)] #![warn(missing_docs)]
mod addresses;
pub mod announcement; mod masking;
pub mod device; mod plugin;
pub mod keys; pub mod profile;
pub mod plugin; mod session;
pub mod store; mod stun;
pub mod transport; mod transport;
pub use addresses::{local_ip_candidates, prioritize_local_subnet};
pub use announcement::{ pub use plugin::AwgPlugin as UdpCarrier;
ANNOUNCEMENT_VERSION, ValidatedAnnouncement, WG_PROTOCOL, WG_WIRE_VERSION, WgAnnouncement, pub use transport::{AwgTransport as UdpTransport, PairOverview};
}; /// External ordinary WireGuard carrier identifier.
pub use device::{PeerHealth, PeerStats, PeerSummary, WireguardDevice}; pub const WG_PROTOCOL: &str = "wg";
pub use keys::{WgPublicKey, WgSecretKey}; /// Wire version of the private UDP negotiation and carrier frames.
pub use plugin::{ pub const WG_WIRE_VERSION: u16 = 2;
DEFAULT_MTU, MIN_MTU, NetworkOverview, PeerOverview, WIREGUARD_OVERHEAD, WireguardConfig, /// Masked carrier identifier used by the shared pair engine.
WireguardPlugin, local_ip_candidates, pub const AWG_PROTOCOL: &str = "awg";
}; /// Wire version of masked carrier frames and private negotiation.
pub use store::WgKeyStore; pub const AWG_WIRE_VERSION: u16 = 2;
pub use transport::{DEFAULT_PORT, MAX_UDP_DATAGRAM_SIZE, WireguardLink, WireguardTransport};
crates/tsunagi-awg/src/masking.rs → crates/tsunagi-wireguard/src/masking.rs
View File
File renamed without changes.
crates/tsunagi-wireguard/src/plugin.rs
+41 -1020
View File
File diff suppressed because it is too large. Load diff
crates/tsunagi-awg/src/profile.rs → crates/tsunagi-wireguard/src/profile.rs
+29 -1
View File
@@ -7,6 +7,7 @@ use serde::{Deserialize, Serialize};
#[derive(Clone, Serialize, Deserialize, zeroize::ZeroizeOnDrop)] #[derive(Clone, Serialize, Deserialize, zeroize::ZeroizeOnDrop)]
pub struct PairProfile { pub struct PairProfile {
id: [u8; 16], id: [u8; 16],
masked: bool,
prefix: u16, prefix: u16,
headers: [[u32; 2]; 4], headers: [[u32; 2]; 4],
header_key: [u8; 32], header_key: [u8; 32],
@@ -31,6 +32,7 @@ impl PairProfile {
}); });
Self { Self {
id: rand::random(), id: rand::random(),
masked: true,
prefix: rand::random_range(12..49), prefix: rand::random_range(12..49),
headers, headers,
header_key: rand::random(), header_key: rand::random(),
@@ -42,6 +44,16 @@ impl PairProfile {
signature: rand::random_range(0..2), signature: rand::random_range(0..2),
} }
} }
/// A fresh ordinary WireGuard profile, without masking or signature packets.
pub fn wireguard() -> Self {
let mut profile = Self::generate();
profile.masked = false;
profile
}
/// Whether this profile uses Amnezia masking.
pub fn is_masked(&self) -> bool {
self.masked
}
/// Public attempt identifier, useful for distinguishing retries without revealing secrets. /// Public attempt identifier, useful for distinguishing retries without revealing secrets.
pub fn id(&self) -> [u8; 16] { pub fn id(&self) -> [u8; 16] {
self.id self.id
@@ -109,6 +121,9 @@ impl PairProfile {
} }
/// Masks one standard WireGuard message; invalid types and oversized packets are refused. /// Masks one standard WireGuard message; invalid types and oversized packets are refused.
pub fn encode(&self, raw: &[u8]) -> Option<Bytes> { pub fn encode(&self, raw: &[u8]) -> Option<Bytes> {
if !self.masked {
return (raw.len() <= 1200).then(|| Bytes::copy_from_slice(raw));
}
let kind = match raw.get(..4)? { let kind = match raw.get(..4)? {
[1, 0, 0, 0] => AwgKind::Initiation, [1, 0, 0, 0] => AwgKind::Initiation,
[2, 0, 0, 0] => AwgKind::Response, [2, 0, 0, 0] => AwgKind::Response,
@@ -132,6 +147,9 @@ impl PairProfile {
} }
/// Unmasks a packet and restores standard headers before WireGuard validation. /// Unmasks a packet and restores standard headers before WireGuard validation.
pub fn decode(&self, wire: &[u8]) -> Option<Vec<u8>> { pub fn decode(&self, wire: &[u8]) -> Option<Vec<u8>> {
if !self.masked {
return (wire.len() <= 1200).then(|| wire.to_vec());
}
if wire.len() > 1200 { if wire.len() > 1200 {
return None; return None;
} }
@@ -147,11 +165,18 @@ impl PairProfile {
Some(message) Some(message)
} }
pub(crate) fn pad(&self, packet: Bytes) -> Bytes { pub(crate) fn pad(&self, packet: Bytes) -> Bytes {
self.config().prepare_transport_plaintext(packet, 1200) if self.masked {
self.config().prepare_transport_plaintext(packet, 1200)
} else {
packet
}
} }
/// Generates structured signature packets with fresh fields, never static captures. /// Generates structured signature packets with fresh fields, never static captures.
/// These target the other Tsunagi peer, not DNS or SIP services. /// These target the other Tsunagi peer, not DNS or SIP services.
pub fn signature_packets(&self) -> Vec<Vec<u8>> { pub fn signature_packets(&self) -> Vec<Vec<u8>> {
if !self.masked {
return Vec::new();
}
if self.signature == 0 { if self.signature == 0 {
// A complete DNS query. Counts, label lengths and qtype remain valid. // A complete DNS query. Counts, label lengths and qtype remain valid.
let label = format!("n{:016x}", rand::random::<u64>()); let label = format!("n{:016x}", rand::random::<u64>());
@@ -169,6 +194,9 @@ impl PairProfile {
} }
} }
pub(crate) fn junk_packets(&self) -> Vec<Vec<u8>> { pub(crate) fn junk_packets(&self) -> Vec<Vec<u8>> {
if !self.masked {
return Vec::new();
}
(0..self.junk_count) (0..self.junk_count)
.map(|_| { .map(|_| {
let mut packet = vec![0; rand::random_range(self.junk_min..=self.junk_max).into()]; let mut packet = vec![0; rand::random_range(self.junk_min..=self.junk_max).into()];
crates/tsunagi-awg/src/session.rs → crates/tsunagi-wireguard/src/session.rs
+17 -5
View File
@@ -6,7 +6,7 @@ use bytes::Bytes;
use iroh::EndpointId; use iroh::EndpointId;
use std::collections::{HashMap, VecDeque}; use std::collections::{HashMap, VecDeque};
use std::net::{Ipv4Addr, SocketAddr}; use std::net::{Ipv4Addr, SocketAddr};
use std::sync::atomic::{AtomicBool, Ordering}; use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
use std::sync::{Arc, Mutex}; use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant}; use std::time::{Duration, Instant};
use tokio::net::UdpSocket; use tokio::net::UdpSocket;
@@ -21,6 +21,7 @@ const WINDOW: Duration = Duration::from_secs(20);
pub(crate) struct PairLink { pub(crate) struct PairLink {
network: NetworkId, network: NetworkId,
protocol: &'static str,
peer: EndpointId, peer: EndpointId,
pub(crate) attempt: [u8; 16], pub(crate) attempt: [u8; 16],
tx: mpsc::Sender<Bytes>, tx: mpsc::Sender<Bytes>,
@@ -29,6 +30,7 @@ pub(crate) struct PairLink {
closed: AtomicBool, closed: AtomicBool,
remote: Mutex<Option<SocketAddr>>, remote: Mutex<Option<SocketAddr>>,
reason: Mutex<Option<String>>, reason: Mutex<Option<String>>,
checks: Arc<AtomicU64>,
} }
impl std::fmt::Debug for PairLink { impl std::fmt::Debug for PairLink {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
@@ -39,6 +41,9 @@ impl std::fmt::Debug for PairLink {
} }
} }
impl PairLink { impl PairLink {
pub(crate) fn delivery_checks(&self) -> u64 {
self.checks.load(Ordering::Relaxed)
}
pub(crate) fn has_authenticated_path(&self) -> bool { pub(crate) fn has_authenticated_path(&self) -> bool {
match self.remote.lock() { match self.remote.lock() {
Ok(remote) => remote.is_some(), Ok(remote) => remote.is_some(),
@@ -103,8 +108,8 @@ impl PacketLink for PairLink {
fn path_description(&self) -> String { fn path_description(&self) -> String {
match self.remote.lock() { match self.remote.lock() {
Ok(addr) => match *addr { Ok(addr) => match *addr {
Some(addr) => format!("awg UDP via {addr}"), Some(addr) => format!("{} UDP via {addr}", self.protocol),
None => "awg UDP, establishing".into(), None => format!("{} UDP, establishing", self.protocol),
}, },
Err(_) => "awg UDP".into(), Err(_) => "awg UDP".into(),
} }
@@ -117,19 +122,21 @@ impl Drop for PairLink {
} }
pub(crate) fn start( pub(crate) fn start(
network: NetworkId, pair: (NetworkId, EndpointId),
peer: EndpointId,
socket: UdpSocket, socket: UdpSocket,
tunn: Tunn, tunn: Tunn,
profile: PairProfile, profile: PairProfile,
candidates: Vec<SocketAddr>, candidates: Vec<SocketAddr>,
protocol: &'static str,
parent: watch::Receiver<bool>, parent: watch::Receiver<bool>,
) -> Arc<PairLink> { ) -> Arc<PairLink> {
let (network, peer) = pair;
let (tx, rx) = mpsc::channel(QUEUE); let (tx, rx) = mpsc::channel(QUEUE);
let (in_tx, in_rx) = mpsc::channel(QUEUE); let (in_tx, in_rx) = mpsc::channel(QUEUE);
let (stop, _) = watch::channel(false); let (stop, _) = watch::channel(false);
let link = Arc::new(PairLink { let link = Arc::new(PairLink {
network, network,
protocol,
peer, peer,
attempt: profile.id(), attempt: profile.id(),
tx, tx,
@@ -138,8 +145,10 @@ pub(crate) fn start(
closed: AtomicBool::new(false), closed: AtomicBool::new(false),
remote: Mutex::new(None), remote: Mutex::new(None),
reason: Mutex::new(None), reason: Mutex::new(None),
checks: Arc::new(AtomicU64::new(0)),
}); });
let weak = Arc::downgrade(&link); let weak = Arc::downgrade(&link);
let checks = link.checks.clone();
tokio::spawn(async move { tokio::spawn(async move {
let mut engine = Engine { let mut engine = Engine {
stop: stop.subscribe(), stop: stop.subscribe(),
@@ -157,6 +166,7 @@ pub(crate) fn start(
assemblies: HashMap::new(), assemblies: HashMap::new(),
started: Instant::now(), started: Instant::now(),
last_authenticated: Instant::now(), last_authenticated: Instant::now(),
checks,
}; };
engine.run(parent, &weak).await; engine.run(parent, &weak).await;
if let Some(link) = weak.upgrade() { if let Some(link) = weak.upgrade() {
@@ -188,6 +198,7 @@ struct Engine {
assemblies: HashMap<u64, Assembly>, assemblies: HashMap<u64, Assembly>,
started: Instant, started: Instant,
last_authenticated: Instant, last_authenticated: Instant,
checks: Arc<AtomicU64>,
} }
enum ResultOwned { enum ResultOwned {
Wire(Vec<u8>), Wire(Vec<u8>),
@@ -352,6 +363,7 @@ impl Engine {
{ {
self.pending.remove(index); self.pending.remove(index);
self.samples.push_back(true); self.samples.push_back(true);
self.checks.fetch_add(1, Ordering::Relaxed);
} }
} }
Some(3) => self.fragment(seq, payload), Some(3) => self.fragment(seq, payload),
crates/tsunagi-wireguard/src/store.rs
-160
View File
@@ -1,160 +0,0 @@
//! The plugin's own key store.
//!
//! Deliberately a separate SQLite file from the agent's `state.sqlite`: plugin
//! keys are not the iroh identity and not the network secret, and their
//! lifecycle is the plugin's business alone.
//!
//! One key per network, so a participant presents a different WireGuard
//! identity in each network it belongs to.
use std::path::{Path, PathBuf};
use std::sync::Mutex;
use rusqlite::{Connection, OptionalExtension, params};
use tsunagi::dataplane::PluginError;
use tsunagi::identity::NetworkId;
use crate::keys::{KEY_LEN, WgSecretKey};
/// Schema version written by this build.
pub const SCHEMA_VERSION: i64 = 1;
/// Per-network WireGuard private keys.
#[derive(Debug)]
pub struct WgKeyStore {
conn: Mutex<Connection>,
path: PathBuf,
}
impl WgKeyStore {
/// Opens, creating the file and its directory if needed.
pub fn open(path: impl AsRef<Path>) -> Result<Self, PluginError> {
let path = path.as_ref().to_path_buf();
if let Some(parent) = path.parent() {
tsunagi::storage::create_dir(parent)
.map_err(|err| PluginError::Other(format!("cannot create {parent:?}: {err}")))?;
}
let existed = path.exists();
let conn = Connection::open(&path).map_err(|err| {
PluginError::Other(format!("cannot open the WireGuard key store: {err}"))
})?;
tsunagi::storage::restrict_path_permissions(&path)
.map_err(|err| PluginError::Other(format!("cannot secure the key store: {err}")))?;
conn.busy_timeout(std::time::Duration::from_secs(5))
.and_then(|()| conn.pragma_update(None, "journal_mode", "WAL"))
.and_then(|()| conn.pragma_update(None, "synchronous", "NORMAL"))
.map_err(|err| PluginError::Other(format!("cannot configure the key store: {err}")))?;
if existed {
let integrity: String = conn
.query_row("PRAGMA integrity_check", [], |row| row.get(0))
.map_err(|err| {
PluginError::Other(format!("WireGuard key store is unusable: {err}"))
})?;
if integrity != "ok" {
return Err(PluginError::Other(format!(
"WireGuard key store at {} is corrupt and will not be recreated: {integrity}",
path.display()
)));
}
}
let found: i64 = conn
.query_row("PRAGMA user_version", [], |row| row.get(0))
.map_err(|err| PluginError::Other(format!("cannot read the schema version: {err}")))?;
if found > SCHEMA_VERSION {
return Err(PluginError::Other(format!(
"WireGuard key store schema {found} is newer than {SCHEMA_VERSION}"
)));
}
if found < SCHEMA_VERSION {
conn.execute_batch(
"BEGIN;
CREATE TABLE IF NOT EXISTS network_keys (
network_id BLOB PRIMARY KEY,
secret BLOB NOT NULL,
created_at INTEGER NOT NULL
);
PRAGMA user_version = 1;
COMMIT;",
)
.map_err(|err| PluginError::Other(format!("cannot create the schema: {err}")))?;
}
Ok(Self {
conn: Mutex::new(conn),
path,
})
}
/// Path of the underlying file.
pub fn path(&self) -> &Path {
&self.path
}
/// Loads the stored key for `network`, or generates, stores and returns a fresh one.
pub fn load_or_create(&self, network: NetworkId) -> Result<WgSecretKey, PluginError> {
let guard = self
.conn
.lock()
.map_err(|_| PluginError::Other("key store lock is poisoned".into()))?;
let existing: Option<Vec<u8>> = guard
.query_row(
"SELECT secret FROM network_keys WHERE network_id = ?1",
params![network.as_bytes().as_slice()],
|row| row.get(0),
)
.optional()
.map_err(|err| PluginError::Other(format!("cannot read the key store: {err}")))?;
if let Some(secret) = existing {
let bytes = <[u8; KEY_LEN]>::try_from(secret.as_slice()).map_err(|_| {
PluginError::Other(format!(
"stored WireGuard key for {} is {} bytes, expected {KEY_LEN}",
network.fmt_short(),
secret.len()
))
})?;
return Ok(WgSecretKey::from_bytes(&bytes));
}
let fresh = WgSecretKey::generate();
let now = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map_err(|err| PluginError::Other(format!("system clock is broken: {err}")))?
.as_secs() as i64;
guard
.execute(
"INSERT INTO network_keys (network_id, secret, created_at) VALUES (?1, ?2, ?3)",
params![
network.as_bytes().as_slice(),
fresh.as_bytes().as_slice(),
now
],
)
.map_err(|err| PluginError::Other(format!("cannot write the key store: {err}")))?;
Ok(fresh)
}
/// Removes the stored key for `network`.
pub fn forget(&self, network: NetworkId) -> Result<(), PluginError> {
let guard = self
.conn
.lock()
.map_err(|_| PluginError::Other("key store lock is poisoned".into()))?;
guard
.execute(
"DELETE FROM network_keys WHERE network_id = ?1",
params![network.as_bytes().as_slice()],
)
.map_err(|err| {
PluginError::Other(format!("cannot delete from the key store: {err}"))
})?;
Ok(())
}
}
crates/tsunagi-awg/src/stun.rs → crates/tsunagi-wireguard/src/stun.rs
View File
File renamed without changes.
crates/tsunagi-wireguard/src/transport.rs
+746 -755
View File
File diff suppressed because it is too large. Load diff
crates/tsunagi-wireguard/tests/announcement_tests.rs
-86
View File
@@ -1,86 +0,0 @@
#![allow(missing_docs)]
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use std::net::IpAddr;
use tsunagi::identity::NetworkId;
use tsunagi_wireguard::{ANNOUNCEMENT_VERSION, WgAnnouncement, WgPublicKey, WgSecretKey};
#[test]
fn announcement_round_trip() {
let network = NetworkId::from_bytes([42u8; 32]);
let key = WgSecretKey::generate().public();
let local = WgSecretKey::generate().public();
let addrs: Vec<IpAddr> = vec![
"192.168.1.100".parse().unwrap(),
"10.0.0.5".parse().unwrap(),
];
let announcement = WgAnnouncement::new(network, &key, 51820, addrs.clone());
let encoded = announcement.encode().expect("encode");
let validated = WgAnnouncement::decode_and_validate(&encoded, network, &local).expect("decode");
assert_eq!(validated.public_key, key);
assert_eq!(validated.port, 51820);
assert_eq!(validated.addrs, addrs);
}
#[test]
fn announcement_rejects_zero_key() {
let network = NetworkId::from_bytes([42u8; 32]);
let local = WgSecretKey::generate().public();
let zero = WgPublicKey::from_bytes([0u8; 32]);
let announcement = WgAnnouncement::new(network, &zero, 51820, vec![]);
let encoded = announcement.encode().unwrap();
let err = WgAnnouncement::decode_and_validate(&encoded, network, &local).unwrap_err();
assert!(err.to_string().contains("all zeroes"));
}
#[test]
fn announcement_rejects_own_key() {
let network = NetworkId::from_bytes([42u8; 32]);
let local = WgSecretKey::generate().public();
let announcement = WgAnnouncement::new(network, &local, 51820, vec![]);
let encoded = announcement.encode().unwrap();
let err = WgAnnouncement::decode_and_validate(&encoded, network, &local).unwrap_err();
assert!(err.to_string().contains("own WireGuard key"));
}
#[test]
fn announcement_rejects_wrong_network() {
let network_a = NetworkId::from_bytes([1u8; 32]);
let network_b = NetworkId::from_bytes([2u8; 32]);
let key = WgSecretKey::generate().public();
let local = WgSecretKey::generate().public();
let announcement = WgAnnouncement::new(network_a, &key, 51820, vec![]);
let encoded = announcement.encode().unwrap();
let err = WgAnnouncement::decode_and_validate(&encoded, network_b, &local).unwrap_err();
assert!(err.to_string().contains("different network"));
}
#[test]
fn announcement_rejects_zero_port() {
let network = NetworkId::from_bytes([42u8; 32]);
let key = WgSecretKey::generate().public();
let local = WgSecretKey::generate().public();
let announcement = WgAnnouncement {
version: ANNOUNCEMENT_VERSION,
public_key: *key.as_bytes(),
network: *network.as_bytes(),
port: 0,
addrs: vec![],
};
let encoded = announcement.encode().unwrap();
let err = WgAnnouncement::decode_and_validate(&encoded, network, &local).unwrap_err();
assert!(err.to_string().contains("cannot be 0"));
}
crates/tsunagi-wireguard/tests/plugin_tests.rs
-378
View File
@@ -1,378 +0,0 @@
#![allow(missing_docs)]
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use std::net::SocketAddr;
use std::sync::Arc;
use std::time::Duration;
use bytes::Bytes;
use tokio::sync::mpsc;
use tsunagi::dataplane::transport::{InboundLink, PacketTransport};
use tsunagi::dataplane::{IpPlugin, PacketSink, PluginContext};
use tsunagi::identity::{DeviceIdentity, NetworkId};
use tsunagi_wireguard::{WireguardConfig, WireguardPlugin, WireguardTransport};
struct TestSink {
delivered: mpsc::Sender<(NetworkId, iroh::EndpointId, Bytes)>,
}
impl PacketSink for TestSink {
fn deliver(
&self,
network: NetworkId,
from: iroh::EndpointId,
packet: Bytes,
) -> tsunagi::BoxFuture<'_, ()> {
let delivered = self.delivered.clone();
Box::pin(async move {
let _ = delivered.send((network, from, packet)).await;
})
}
}
fn sample_ipv4_packet() -> Bytes {
let mut packet = vec![0u8; 40];
packet[0] = 0x45; // IPv4, IHL 5
packet[2..4].copy_from_slice(&40u16.to_be_bytes()); // total len 40
packet[8] = 64; // TTL
packet[9] = 17; // UDP
packet[12..16].copy_from_slice(&[10, 0, 0, 1]); // src 10.0.0.1
packet[16..20].copy_from_slice(&[10, 0, 0, 2]); // dst 10.0.0.2
Bytes::from(packet)
}
#[tokio::test]
async fn test_wireguard_plugin_end_to_end() {
let _ = tracing_subscriber::fmt()
.with_env_filter("trace")
.with_test_writer()
.try_init();
let identity_a = DeviceIdentity::generate();
let peer_a = identity_a.endpoint_id();
let identity_b = DeviceIdentity::generate();
let peer_b = identity_b.endpoint_id();
let network = NetworkId::from_bytes([77u8; 32]);
let tmp_a = tempfile::tempdir().expect("tmp A");
let tmp_b = tempfile::tempdir().expect("tmp B");
// Transports
let transport_b = Arc::new(
WireguardTransport::bind(&identity_b, Some(0))
.await
.expect("bind B"),
);
let port_b = transport_b.bound_port();
let transport_a = Arc::new(
WireguardTransport::bind(&identity_a, Some(0))
.await
.expect("bind A"),
);
let port_a = transport_a.bound_port();
// Plugins
let cfg_a = WireguardConfig::new(tmp_a.path());
let plugin_a = WireguardPlugin::open(cfg_a, Arc::clone(&transport_a))
.await
.expect("open plugin A");
let cfg_b = WireguardConfig::new(tmp_b.path());
let plugin_b = WireguardPlugin::open(cfg_b, Arc::clone(&transport_b))
.await
.expect("open plugin B");
// Attach sinks
let (sink_a_tx, mut sink_a_rx) = mpsc::channel(10);
plugin_a.attach(PluginContext::with_sink(Arc::new(TestSink {
delivered: sink_a_tx,
})));
let (sink_b_tx, mut sink_b_rx) = mpsc::channel(10);
plugin_b.attach(PluginContext::with_sink(Arc::new(TestSink {
delivered: sink_b_tx,
})));
// Capability exchange
plugin_a.on_network_activated(network);
plugin_b.on_network_activated(network);
let cap_a = loop {
if let Some(cap) = plugin_a.local_capability(network).expect("cap A") {
break cap;
}
tokio::time::sleep(Duration::from_millis(50)).await;
};
let cap_b = loop {
if let Some(cap) = plugin_b.local_capability(network).expect("cap B") {
break cap;
}
tokio::time::sleep(Duration::from_millis(50)).await;
};
plugin_a
.on_peer_capability(network, peer_b, &cap_b)
.expect("A on cap B");
plugin_b
.on_peer_capability(network, peer_a, &cap_a)
.expect("B on cap A");
// Register loopback address candidates
let addr_b: SocketAddr = format!("127.0.0.1:{port_b}").parse().unwrap();
transport_a.set_peer_addr(peer_b, addr_b);
let addr_a: SocketAddr = format!("127.0.0.1:{port_a}").parse().unwrap();
transport_b.set_peer_addr(peer_a, addr_a);
// Spawn accept loop on A
let transport_a_clone = Arc::clone(&transport_a);
tokio::spawn(async move {
transport_a_clone.accept_loop(|_| {}).await;
});
// Spawn accept loop on B
let (inbound_tx, mut inbound_rx) = mpsc::channel::<InboundLink>(1);
let transport_b_clone = Arc::clone(&transport_b);
tokio::spawn(async move {
transport_b_clone
.accept_loop(move |inbound| {
let _ = inbound_tx.try_send(inbound);
})
.await;
});
// Open link from A to B
let link_a = tokio::time::timeout(
Duration::from_secs(5),
transport_a.open(network, peer_b, "wg"),
)
.await
.expect("open should not timeout")
.expect("open should succeed");
plugin_a.on_peer_link(network, peer_b, Arc::clone(&link_a));
// Wait for inbound link on B and hand to plugin B
let inbound_b = tokio::time::timeout(Duration::from_secs(5), inbound_rx.recv())
.await
.expect("accept should not timeout")
.expect("inbound link should arrive");
plugin_b.on_peer_link(network, peer_a, inbound_b.link);
// Wait briefly for WireGuard handshake to complete
let mut up = false;
for _ in 0..50 {
if let Some(overview_a) = plugin_a.overview(network)
&& overview_a.peers.iter().any(|p| p.is_up())
{
up = true;
break;
}
tokio::time::sleep(Duration::from_millis(50)).await;
}
assert!(up, "WireGuard handshake should complete between peers");
// Send IP packet from A to B
let packet_a_to_b = sample_ipv4_packet();
let sent = plugin_a.carry(network, peer_b, packet_a_to_b.clone());
assert!(sent, "plugin A carry should succeed");
let received_b = tokio::time::timeout(Duration::from_secs(5), sink_b_rx.recv())
.await
.expect("recv B timeout")
.expect("recv B option");
assert_eq!(received_b.0, network);
assert_eq!(received_b.1, peer_a);
assert_eq!(received_b.2, packet_a_to_b);
// Send IP packet from B to A
let packet_b_to_a = sample_ipv4_packet();
let sent_reply = plugin_b.carry(network, peer_a, packet_b_to_a.clone());
assert!(sent_reply, "plugin B carry should succeed");
let received_a = tokio::time::timeout(Duration::from_secs(5), sink_a_rx.recv())
.await
.expect("recv A timeout")
.expect("recv A option");
assert_eq!(received_a.0, network);
assert_eq!(received_a.1, peer_b);
assert_eq!(received_a.2, packet_b_to_a);
}
#[tokio::test]
async fn test_wireguard_two_networks_same_peers() {
let identity_a = DeviceIdentity::generate();
let peer_a = identity_a.endpoint_id();
let identity_b = DeviceIdentity::generate();
let peer_b = identity_b.endpoint_id();
let net1 = NetworkId::from_bytes([11u8; 32]);
let net2 = NetworkId::from_bytes([22u8; 32]);
let tmp_a = tempfile::tempdir().expect("tmp A");
let tmp_b = tempfile::tempdir().expect("tmp B");
let transport_b = Arc::new(
WireguardTransport::bind(&identity_b, Some(0))
.await
.expect("bind B"),
);
let port_b = transport_b.bound_port();
let transport_a = Arc::new(
WireguardTransport::bind(&identity_a, Some(0))
.await
.expect("bind A"),
);
let port_a = transport_a.bound_port();
let plugin_a =
WireguardPlugin::open(WireguardConfig::new(tmp_a.path()), Arc::clone(&transport_a))
.await
.expect("open plugin A");
let plugin_b =
WireguardPlugin::open(WireguardConfig::new(tmp_b.path()), Arc::clone(&transport_b))
.await
.expect("open plugin B");
let (sink_a_tx, mut sink_a_rx) = mpsc::channel(20);
plugin_a.attach(PluginContext::with_sink(Arc::new(TestSink {
delivered: sink_a_tx,
})));
let (sink_b_tx, mut sink_b_rx) = mpsc::channel(20);
plugin_b.attach(PluginContext::with_sink(Arc::new(TestSink {
delivered: sink_b_tx,
})));
// Activate both networks
for net in [net1, net2] {
plugin_a.on_network_activated(net);
plugin_b.on_network_activated(net);
}
// Exchange capabilities for both networks
for net in [net1, net2] {
let cap_a = loop {
if let Some(cap) = plugin_a.local_capability(net).expect("cap A") {
break cap;
}
tokio::time::sleep(Duration::from_millis(20)).await;
};
let cap_b = loop {
if let Some(cap) = plugin_b.local_capability(net).expect("cap B") {
break cap;
}
tokio::time::sleep(Duration::from_millis(20)).await;
};
plugin_a
.on_peer_capability(net, peer_b, &cap_b)
.expect("A on cap B");
plugin_b
.on_peer_capability(net, peer_a, &cap_a)
.expect("B on cap A");
}
let addr_b: SocketAddr = format!("127.0.0.1:{port_b}").parse().unwrap();
transport_a.set_peer_addr(peer_b, addr_b);
let addr_a: SocketAddr = format!("127.0.0.1:{port_a}").parse().unwrap();
transport_b.set_peer_addr(peer_a, addr_a);
// Accept loops
let transport_a_clone = Arc::clone(&transport_a);
tokio::spawn(async move {
transport_a_clone.accept_loop(|_| {}).await;
});
let (inbound_tx, mut inbound_rx) = mpsc::channel::<InboundLink>(10);
let transport_b_clone = Arc::clone(&transport_b);
tokio::spawn(async move {
transport_b_clone
.accept_loop(move |inbound| {
let _ = inbound_tx.try_send(inbound);
})
.await;
});
// Open links for both networks from A to B
for net in [net1, net2] {
let link_a = transport_a
.open(net, peer_b, "wg")
.await
.expect("open A link");
plugin_a.on_peer_link(net, peer_b, link_a);
}
// Deliver inbound links to B
for _ in 0..2 {
let inbound = tokio::time::timeout(Duration::from_secs(5), inbound_rx.recv())
.await
.expect("inbound timeout")
.expect("inbound option");
plugin_b.on_peer_link(inbound.network, peer_a, inbound.link);
}
// Wait for both handshakes to complete on both networks
for net in [net1, net2] {
let mut up = false;
for _ in 0..50 {
if let Some(overview) = plugin_a.overview(net)
&& overview.peers.iter().any(|p| p.is_up())
{
up = true;
break;
}
tokio::time::sleep(Duration::from_millis(50)).await;
}
assert!(up, "handshake on {net:?} should complete");
}
// Send on net1
let pkt1 = sample_ipv4_packet();
assert!(plugin_a.carry(net1, peer_b, pkt1.clone()));
let recv1 = tokio::time::timeout(Duration::from_secs(5), sink_b_rx.recv())
.await
.expect("recv1 timeout")
.expect("recv1 option");
assert_eq!(recv1.0, net1);
assert_eq!(recv1.2, pkt1);
// Send on net2 from A to B
let mut pkt2_bytes = sample_ipv4_packet().to_vec();
pkt2_bytes[15] = 99; // different byte
let pkt2 = Bytes::from(pkt2_bytes);
assert!(plugin_a.carry(net2, peer_b, pkt2.clone()));
let recv2 = tokio::time::timeout(Duration::from_secs(5), sink_b_rx.recv())
.await
.expect("recv2 timeout")
.expect("recv2 option");
assert_eq!(recv2.0, net2);
assert_eq!(recv2.2, pkt2);
// Send reply on net1 from B to A
assert!(plugin_b.carry(net1, peer_a, pkt1.clone()));
let reply1 = tokio::time::timeout(Duration::from_secs(5), sink_a_rx.recv())
.await
.expect("reply1 timeout")
.expect("reply1 option");
assert_eq!(reply1.0, net1);
assert_eq!(reply1.2, pkt1);
// Send reply on net2 from B to A
assert!(plugin_b.carry(net2, peer_a, pkt2.clone()));
let reply2 = tokio::time::timeout(Duration::from_secs(5), sink_a_rx.recv())
.await
.expect("reply2 timeout")
.expect("reply2 option");
assert_eq!(reply2.0, net2);
assert_eq!(reply2.2, pkt2);
}
crates/tsunagi-wireguard/tests/transport_tests.rs
+82 -208
View File
@@ -1,220 +1,94 @@
#![allow(missing_docs)] //! Real control authentication and independently encrypted ordinary WG carriers.
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)] #![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use std::net::SocketAddr;
use std::sync::Arc;
use std::time::Duration;
use bytes::Bytes; use bytes::Bytes;
use tokio::sync::mpsc; use std::{net::Ipv4Addr, time::Duration};
use tsunagi::dataplane::transport::{InboundLink, PacketTransport}; use tempfile::TempDir;
use tsunagi::identity::{DeviceIdentity, NetworkId}; use tsunagi::{
use tsunagi_wireguard::WireguardTransport; Agent,
dataplane::transport::{PacketTransport, TransportError},
discovery::SharedMemoryDiscovery,
identity::DeviceIdentity,
testing::{config_with, network, wait_for_peers},
};
use tsunagi_wireguard::{UdpTransport, profile::PairProfile};
#[tokio::test] #[tokio::test]
async fn test_wireguard_transport_loopback() { async fn ordinary_wireguard_carries_opaque_frames_after_real_membership_authentication() {
let identity_a = DeviceIdentity::generate(); let discovery = SharedMemoryDiscovery::new();
let peer_a = identity_a.endpoint_id(); let mut nodes = Vec::new();
for _ in 0..2 {
let identity_b = DeviceIdentity::generate(); let root = TempDir::new().unwrap();
let peer_b = identity_b.endpoint_id(); let config = config_with(root.path(), &discovery);
let identity = tsunagi::storage::StateStore::open(config.paths.state_db())
let network = NetworkId::from_bytes([42u8; 32]); .unwrap()
.load_or_create_device_identity()
// Bind transport B on port 0 (ephemeral) .unwrap();
let transport_b = Arc::new( let transport = UdpTransport::wireguard(identity.endpoint_id(), Ipv4Addr::LOCALHOST, None);
WireguardTransport::bind(&identity_b, Some(0)) let agent = Agent::spawn(config.with_custom_transport("wg", transport.clone()))
.await .await
.expect("bind server should succeed"), .unwrap();
); nodes.push((root, agent, transport));
let port_b = transport_b.bound_port(); }
assert!(port_b > 0); nodes.sort_by_key(|(_, agent, _)| *agent.endpoint_id().as_bytes());
let (name, secret) = network("outer-wireguard");
// Bind transport A on port 0 (ephemeral) let id = nodes[0].1.join_network(&name, &secret).await.unwrap();
let transport_a = Arc::new( nodes[1].1.join_network(&name, &secret).await.unwrap();
WireguardTransport::bind(&identity_a, Some(0)) for (index, (_, agent, transport)) in nodes.iter().enumerate() {
.await wait_for_peers(agent, id, 1).await;
.expect("bind client should succeed"), transport.allow_peer(id, nodes[1 - index].1.endpoint_id());
); }
let port_a = transport_a.bound_port(); let outgoing = nodes[0]
assert!(port_a > 0); .2
.open(id, nodes[1].1.endpoint_id(), "wg")
// Register B's address and network on transport A
let addr_b: SocketAddr = format!("127.0.0.1:{port_b}").parse().unwrap();
transport_a.set_peer_addr(peer_b, addr_b);
transport_a.set_peer_network(peer_b, network);
// Register A's address and network on transport B
let addr_a: SocketAddr = format!("127.0.0.1:{port_a}").parse().unwrap();
transport_b.set_peer_addr(peer_a, addr_a);
transport_b.set_peer_network(peer_a, network);
// Spawn accept loop on A
let transport_a_clone = Arc::clone(&transport_a);
tokio::spawn(async move {
transport_a_clone.accept_loop(|_| {}).await;
});
// Spawn accept loop on B
let (inbound_tx, mut inbound_rx) = mpsc::channel::<InboundLink>(1);
let transport_b_clone = Arc::clone(&transport_b);
tokio::spawn(async move {
transport_b_clone
.accept_loop(move |inbound| {
let _ = inbound_tx.try_send(inbound);
})
.await;
});
// Open link from A to B
let link_a = tokio::time::timeout(
Duration::from_secs(5),
transport_a.open(network, peer_b, "wg"),
)
.await
.expect("open should not timeout")
.expect("open should succeed");
// Send from A to B to trigger inbound link discovery on B
let data_a_to_b = Bytes::from_static(b"wireguard packet from A to B");
link_a.send(data_a_to_b.clone()).expect("send A->B");
// Receive inbound link on B
let inbound_b = tokio::time::timeout(Duration::from_secs(5), inbound_rx.recv())
.await .await
.expect("accept should not timeout") .unwrap();
.expect("inbound link should arrive"); let incoming = tokio::time::timeout(Duration::from_secs(5), nodes[1].2.next_inbound())
assert_eq!(inbound_b.network, network);
assert_eq!(inbound_b.peer, peer_a);
assert_eq!(inbound_b.protocol, "wg");
let link_b = inbound_b.link;
// Check packet received on B
let received_on_b = tokio::time::timeout(Duration::from_secs(2), link_b.recv())
.await .await
.expect("recv B timeout") .unwrap()
.expect("recv B option"); .unwrap()
assert_eq!(received_on_b, data_a_to_b); .link;
for (from, to) in [(&outgoing, &incoming), (&incoming, &outgoing)] {
// Send from B to A let payload = Bytes::from(vec![0x37; 65_536]);
let data_b_to_a = Bytes::from_static(b"wireguard packet from B to A"); from.send(payload.clone()).unwrap();
link_b.send(data_b_to_a.clone()).expect("send B->A"); assert_eq!(
let received_on_a = tokio::time::timeout(Duration::from_secs(2), link_a.recv()) tokio::time::timeout(Duration::from_secs(5), to.recv())
.await .await
.expect("recv A timeout") .unwrap()
.expect("recv A option"); .unwrap(),
assert_eq!(received_on_a, data_b_to_a); payload
);
// Path descriptions assert!(from.path_description().starts_with("wg UDP via"));
assert!(link_a.path_description().contains("wireguard via")); assert!(matches!(
assert!(link_b.path_description().contains("wireguard via")); from.send(Bytes::from(vec![0; 65_537])),
} Err(TransportError::TooLarge { .. })
));
#[tokio::test] }
async fn test_wireguard_transport_unreachable_probe_times_out() { outgoing.close();
let identity_a = DeviceIdentity::generate(); incoming.close();
let identity_b = DeviceIdentity::generate(); for (_, agent, transport) in nodes {
let peer_b = identity_b.endpoint_id(); transport.shutdown();
let network = NetworkId::from_bytes([99u8; 32]); agent.shutdown().await;
let transport_a = Arc::new(
WireguardTransport::bind(&identity_a, Some(0))
.await
.expect("bind client should succeed"),
);
// Register a non-responsive address for peer B (TEST-NET-1 documentation IP which drops packets)
let unreachable_addr: SocketAddr = "192.0.2.1:51820".parse().unwrap();
transport_a.set_peer_addr(peer_b, unreachable_addr);
transport_a.set_peer_network(peer_b, network);
// Spawn accept loop on A
let transport_a_clone = Arc::clone(&transport_a);
tokio::spawn(async move {
transport_a_clone.accept_loop(|_| {}).await;
});
// Opening to unreachable candidate must fail with Unreachable rather than returning a dead direct link
let result = transport_a.open(network, peer_b, "wg").await;
match result {
Err(tsunagi::dataplane::transport::TransportError::Unreachable(msg)) => {
assert!(msg.contains("timed out"));
}
other => panic!("expected Unreachable error, got {other:?}"),
} }
} }
#[tokio::test] #[tokio::test]
async fn test_wireguard_link_close_and_transport_close_link() { async fn an_unannounced_peer_cannot_open_or_negotiate_a_wireguard_carrier() {
let identity_a = DeviceIdentity::generate(); let local = DeviceIdentity::generate().endpoint_id();
let peer_a = identity_a.endpoint_id(); let peer = DeviceIdentity::generate().endpoint_id();
let identity_b = DeviceIdentity::generate(); let transport = UdpTransport::wireguard(local, Ipv4Addr::LOCALHOST, None);
let peer_b = identity_b.endpoint_id(); let id = tsunagi::NetworkId::from_bytes([7; 32]);
let network = NetworkId::from_bytes([55u8; 32]); assert!(transport.open(id, peer, "wg").await.is_err());
assert!(transport.on_control(id, peer, "wg", &[0; 40]).is_err());
let transport_b = Arc::new( transport.shutdown();
WireguardTransport::bind(&identity_b, Some(0)) }
.await
.unwrap(), #[test]
); fn ordinary_profiles_preserve_the_wireguard_header_and_emit_no_masking_packets() {
let port_b = transport_b.bound_port(); let profile = PairProfile::wireguard();
let mut message = vec![0x42; 148];
let transport_a = Arc::new( message[..4].copy_from_slice(&[1, 0, 0, 0]);
WireguardTransport::bind(&identity_a, Some(0)) assert_eq!(profile.encode(&message).unwrap().as_ref(), message);
.await assert_eq!(profile.decode(&message).unwrap(), message);
.unwrap(), assert!(profile.signature_packets().is_empty());
); assert!(!profile.is_masked());
let port_a = transport_a.bound_port();
let addr_b: SocketAddr = format!("127.0.0.1:{port_b}").parse().unwrap();
transport_a.set_peer_addr(peer_b, addr_b);
transport_a.set_peer_network(peer_b, network);
let addr_a: SocketAddr = format!("127.0.0.1:{port_a}").parse().unwrap();
transport_b.set_peer_addr(peer_a, addr_a);
transport_b.set_peer_network(peer_a, network);
let transport_a_clone = Arc::clone(&transport_a);
tokio::spawn(async move {
transport_a_clone.accept_loop(|_| {}).await;
});
let transport_b_clone = Arc::clone(&transport_b);
tokio::spawn(async move {
transport_b_clone.accept_loop(|_| {}).await;
});
let link_a = transport_a
.open(network, peer_b, "wg")
.await
.expect("open link A");
assert!(!link_a.is_closed());
// Spawn a reader task waiting on recv()
let link_a_clone = Arc::clone(&link_a);
let recv_task = tokio::spawn(async move { link_a_clone.recv().await });
// Explicitly closing via transport.close_link()
transport_a.close_link(network, peer_b);
assert!(link_a.is_closed());
// Wait on closed() notification
tokio::time::timeout(Duration::from_secs(1), link_a.closed())
.await
.expect("link.closed() should resolve promptly");
// recv() must have returned None
let recv_res = tokio::time::timeout(Duration::from_secs(1), recv_task)
.await
.expect("recv task should complete")
.expect("task join");
assert_eq!(recv_res, None);
// Send after close must return Closed error
let send_res = link_a.send(Bytes::from_static(b"after close"));
assert!(matches!(
send_res,
Err(tsunagi::dataplane::transport::TransportError::Closed)
));
} }
crates/tsunagi/Cargo.toml
+2
View File
@@ -27,6 +27,8 @@ clap = { version = "4.5", features = ["derive", "env"] }
# colour is never written where it would show up as rubbish. # colour is never written where it would show up as rubbish.
anstream = "1.0" anstream = "1.0"
anstyle = "1.0" anstyle = "1.0"
serde_json = "1.0"
tempfile.workspace = true
# Local interface addresses, for the diagnostics in `status`. # Local interface addresses, for the diagnostics in `status`.
netwatch = "0.19.3" netwatch = "0.19.3"
crates/tsunagi/src/main.rs
+247 -398
View File
@@ -14,10 +14,10 @@ use std::time::Duration;
#[cfg(windows)] #[cfg(windows)]
mod winservice; mod winservice;
mod vip_cli;
use clap::{Args, Parser, Subcommand, ValueEnum}; use clap::{Args, Parser, Subcommand, ValueEnum};
use tsunagi::agent::Event; use tsunagi::agent::Event;
use tsunagi::config::{AgentConfig, StoragePaths, TransportPolicy}; use tsunagi::config::{AgentConfig, StoragePaths, TransportPolicy};
use tsunagi::dataplane::IpPlugin;
use tsunagi::discovery::{ use tsunagi::discovery::{
CompositeDiscovery, MainlineDiscovery, NetworkDiscovery, StaticBootstrap, CompositeDiscovery, MainlineDiscovery, NetworkDiscovery, StaticBootstrap,
}; };
@@ -27,14 +27,9 @@ use tsunagi::overlay::{MemoryTunFactory, TunFactory};
use tsunagi::state::Ipv4Range; use tsunagi::state::Ipv4Range;
use tsunagi::{Agent, NetworkId}; use tsunagi::{Agent, NetworkId};
use tsunagi_awg::{AWG_PROTOCOL, AWG_WIRE_VERSION, AwgPlugin, AwgTransport}; use tsunagi_awg::{AWG_PROTOCOL, AWG_WIRE_VERSION, AwgPlugin, AwgTransport};
use tsunagi_tcp_tls::{ use tsunagi_tcp_tls::{TCP_TLS_PROTOCOL, TcpTlsCarrier, TcpTlsConfig, TcpTlsTransport};
TCP_TLS_PROTOCOL, TCP_TLS_VERSION, TcpTlsCodec, TcpTlsConfig, TcpTlsPlugin, TcpTlsTransport,
};
use tsunagi_wg_quic::{WireguardConfig, WireguardPlugin}; use tsunagi_wg_quic::{WireguardConfig, WireguardPlugin};
use tsunagi_wireguard::{ use tsunagi_wireguard::WG_PROTOCOL;
WG_PROTOCOL, WG_WIRE_VERSION, WireguardConfig as PureWgConfig, WireguardPlugin as PureWgPlugin,
WireguardTransport as PureWgTransport,
};
/// A small agent for private mesh networks. /// A small agent for private mesh networks.
#[derive(Debug, Parser)] #[derive(Debug, Parser)]
@@ -52,6 +47,8 @@ struct Cli {
#[derive(Debug, Subcommand)] #[derive(Debug, Subcommand)]
enum Command { enum Command {
/// Manage named shared TCP/UDP virtual services.
Vip(vip_cli::VipArgs),
/// Shows this device's identity, and changes it. /// Shows this device's identity, and changes it.
/// ///
/// Every item follows the same shape: name it to see it, name it with a /// Every item follows the same shape: name it to see it, name it with a
@@ -109,7 +106,8 @@ struct DnsArgs {
#[derive(Debug, Args)] #[derive(Debug, Args)]
#[command( #[command(
after_help = "Examples:\n tsng protocols\n tsng protocols home\n tsng protocols home awg,wg-quic\n tsng protocols home none\n\nThe selection is local to this device, saved per network and applied immediately.\nBoth endpoints need a common protocol at the same wire version." after_help = "Examples:\n tsng protocols\n tsng protocols home\n tsng protocols home awg,wg-quic\n tsng protocols home none\n\nThe selection is local to this device, saved per network and applied immediately.\nThe choice governs local incoming and outgoing data links only.
Adjacent peers need a common external transport; control and discovery are independent."
)] )]
struct ProtocolsArgs { struct ProtocolsArgs {
#[command(flatten)] #[command(flatten)]
@@ -927,6 +925,7 @@ async fn run(command: Command) -> Result<(), Box<dyn std::error::Error>> {
Command::Join(args) => join_command(args).await, Command::Join(args) => join_command(args).await,
Command::Network(args) => network_command(args).await, Command::Network(args) => network_command(args).await,
Command::Dns(args) => dns_command(args).await, Command::Dns(args) => dns_command(args).await,
Command::Vip(args) => vip_cli::run(args).await,
Command::ExitNode(args) => exit_node_command(args).await, Command::ExitNode(args) => exit_node_command(args).await,
Command::Wipe(args) => wipe(args).await, Command::Wipe(args) => wipe(args).await,
// Taken in `main`, before there is a runtime: the service manager // Taken in `main`, before there is a runtime: the service manager
@@ -1317,7 +1316,12 @@ fn spawn_dns(agent: Agent, port: u16) -> DnsService {
let members = network.members.iter().filter_map(|member| { let members = network.members.iter().filter_map(|member| {
Some((member.hostname.clone()?, member.overlay_address_v4?)) Some((member.hostname.clone()?, member.overlay_address_v4?))
}); });
zones.push(Zone::new(zone, members)); let services = network
.vips
.iter()
.filter(|v| v.error.is_none())
.filter_map(|v| Some((v.config.name.clone(), v.address?)));
zones.push(Zone::new(zone, members.chain(services)));
} }
let zones = Zones::new(zones); let zones = Zones::new(zones);
let names = zones.names() as u32; let names = zones.names() as u32;
@@ -1451,13 +1455,20 @@ struct ProtocolSpec {
options: &'static [tsunagi::dataplane::ProtocolOption], options: &'static [tsunagi::dataplane::ProtocolOption],
} }
const WG_OPTIONS: &[tsunagi::dataplane::ProtocolOption] = &[tsunagi::dataplane::ProtocolOption {
key: "port",
value: "PORT",
help: "preferred local UDP port; additional pairs use dynamic ports",
default: Some("51820"),
}];
/// Every protocol this build has. /// Every protocol this build has.
const PROTOCOLS: &[ProtocolSpec] = &[ const PROTOCOLS: &[ProtocolSpec] = &[
ProtocolSpec { ProtocolSpec {
name: WG_PROTOCOL, name: WG_PROTOCOL,
version: WG_WIRE_VERSION, version: tsunagi_wireguard::WG_WIRE_VERSION,
summary: "Pure WireGuard direct UDP transport", summary: "Pure WireGuard direct UDP transport",
options: PureWgPlugin::OPTIONS, options: WG_OPTIONS,
}, },
ProtocolSpec { ProtocolSpec {
name: AWG_PROTOCOL, name: AWG_PROTOCOL,
@@ -1466,16 +1477,16 @@ const PROTOCOLS: &[ProtocolSpec] = &[
options: AwgPlugin::OPTIONS, options: AwgPlugin::OPTIONS,
}, },
ProtocolSpec { ProtocolSpec {
name: tsunagi_wg_quic::WIREGUARD_PROTOCOL, name: "wg-quic",
version: tsunagi_wg_quic::ANNOUNCEMENT_VERSION, version: 1,
summary: "WireGuard encryption over iroh QUIC datagrams with NAT traversal and relays", summary: "iroh QUIC datagram carrier with NAT traversal and relays",
options: WireguardPlugin::OPTIONS, options: &[],
}, },
ProtocolSpec { ProtocolSpec {
name: TCP_TLS_PROTOCOL, name: TCP_TLS_PROTOCOL,
version: TCP_TLS_VERSION, version: 2,
summary: "Noise E2EE carried over TCP TLS 1.3", summary: "TCP TLS 1.3 carrier",
options: TcpTlsCodec::OPTIONS, options: TcpTlsConfig::OPTIONS,
}, },
]; ];
@@ -1511,12 +1522,44 @@ fn parse_settings(raw: &[String]) -> Result<Vec<Setting>, Box<dyn std::error::Er
.collect() .collect()
} }
/// Accept historic encryption settings without advertising internal protocol choices.
/// All spellings now configure the one shared tunnel.
fn take_tunnel_settings(settings: &mut Vec<Setting>) -> Result<Vec<(String, String)>, String> {
let mut tunnel: Vec<(String, String)> = Vec::new();
for setting in settings
.iter()
.filter(|s| matches!(s.key.as_str(), "keepalive" | "mtu"))
{
if setting
.protocol
.as_ref()
.is_some_and(|p| !PROTOCOLS.iter().any(|s| s.name == p))
{
return Err(format!(
"unknown protocol in legacy `{}` setting",
setting.key
));
}
if let Some((_, previous)) = tunnel.iter().find(|(key, _)| key == &setting.key) {
if previous != &setting.value {
return Err(format!("conflicting shared `{}` settings", setting.key));
}
} else {
tunnel.push((setting.key.clone(), setting.value.clone()));
}
}
settings.retain(|s| !matches!(s.key.as_str(), "keepalive" | "mtu"));
Ok(tunnel)
}
/// The settings meant for one protocol, refusing any that fit nowhere. /// The settings meant for one protocol, refusing any that fit nowhere.
fn settings_for(spec: &ProtocolSpec, settings: &[Setting]) -> Vec<(String, String)> { fn settings_for(spec: &ProtocolSpec, settings: &[Setting]) -> Vec<(String, String)> {
let mut taken = Vec::new(); let mut taken = Vec::new();
for setting in settings { for setting in settings {
let aimed_here = match &setting.protocol { let aimed_here = match &setting.protocol {
Some(name) => name == spec.name, Some(name) => {
name == spec.name && spec.options.iter().any(|option| option.key == setting.key)
}
// Unqualified settings go to whichever protocol declares the // Unqualified settings go to whichever protocol declares the
// key. With one selected that is the obvious reading; with // key. With one selected that is the obvious reading; with
// several, write `protocol:key=value`. // several, write `protocol:key=value`.
@@ -1566,10 +1609,7 @@ fn show_protocols() -> Result<(), Box<dyn std::error::Error>> {
/// published. /// published.
struct AgentControl { struct AgentControl {
agent: Agent, agent: Agent,
pure_wg: Option<Arc<PureWgPlugin>>,
awg: Option<Arc<AwgPlugin>>,
plugin: Option<Arc<WireguardPlugin>>, plugin: Option<Arc<WireguardPlugin>>,
tcp_tls: Option<Arc<TcpTlsPlugin>>,
/// The resolver, which this owns so it can be switched while running. /// The resolver, which this owns so it can be switched while running.
dns: Arc<tokio::sync::Mutex<Option<DnsService>>>, dns: Arc<tokio::sync::Mutex<Option<DnsService>>>,
/// Where the setting is remembered, so it survives a restart. /// Where the setting is remembered, so it survives a restart.
@@ -1577,6 +1617,23 @@ struct AgentControl {
} }
impl tsunagi::ipc::ReportSource for AgentControl { impl tsunagi::ipc::ReportSource for AgentControl {
fn save_vip(
&self,
network: String,
config: tsunagi::vip::VipConfig,
expected: Option<String>,
delete: bool,
) -> tsunagi::BoxFuture<'_, Result<Vec<tsunagi::vip::VipStatus>, String>> {
Box::pin(async move {
let network = network
.parse()
.map_err(|e| format!("invalid network id: {e}"))?;
self.agent
.save_vip(network, config, expected, delete)
.await
.map_err(|e| e.to_string())
})
}
fn forget_node( fn forget_node(
&self, &self,
network: String, network: String,
@@ -1605,16 +1662,7 @@ impl tsunagi::ipc::ReportSource for AgentControl {
fn report(&self) -> tsunagi::BoxFuture<'_, tsunagi::ipc::StatusReport> { fn report(&self) -> tsunagi::BoxFuture<'_, tsunagi::ipc::StatusReport> {
Box::pin(async move { Box::pin(async move {
let dns = self.dns_state().await; let dns = self.dns_state().await;
build_report( build_report(&self.agent, self.plugin.as_deref(), dns, &self.paths).await
&self.agent,
self.pure_wg.as_deref(),
self.awg.as_deref(),
self.plugin.as_deref(),
self.tcp_tls.as_deref(),
dns,
&self.paths,
)
.await
}) })
} }
@@ -2843,7 +2891,10 @@ async fn leave_network(
storage.remove_network(network.network_id).await?; storage.remove_network(network.network_id).await?;
// What a protocol kept for it goes too; there is no plugin loaded here // What a protocol kept for it goes too; there is no plugin loaded here
// to be asked, so the one this build has is asked directly. // to be asked, so the one this build has is asked directly.
forget_protocol_state(paths, network.network_id); let cleanup_paths = paths.clone();
let cleanup_network = network.network_id;
tokio::task::spawn_blocking(move || forget_protocol_state(&cleanup_paths, cleanup_network))
.await?;
storage.release_ownership_lock(); storage.release_ownership_lock();
println!("left `{name}` ({}) locally", short(&id, 10)); println!("left `{name}` ({}) locally", short(&id, 10));
eprintln!( eprintln!(
@@ -2872,17 +2923,18 @@ fn forget_protocol_state(paths: &StoragePaths, network: tsunagi::NetworkId) {
Err(err) => eprintln!("warning: the wg-quic key store could not be opened: {err}"), Err(err) => eprintln!("warning: the wg-quic key store could not be opened: {err}"),
} }
let pure_store = PureWgConfig::new(paths.state_dir.join("wg")); for name in ["wg", "awg"] {
match tsunagi_wireguard::WgKeyStore::open(pure_store.key_store_path()) { let path = paths.state_dir.join(name).join("wireguard.sqlite");
Ok(store) => { if path.exists() {
if let Err(err) = store.forget(network) { match tsunagi_wg_quic::WgKeyStore::open(path) {
eprintln!("warning: the wg key for it could not be removed: {err}"); Ok(store) => {
if let Err(err) = store.forget(network) {
eprintln!("warning: old data key could not be removed: {err}");
}
}
Err(err) => eprintln!("warning: old data key store could not be opened: {err}"),
} }
} }
Err(err) if !pure_store.key_store_path().exists() => {
let _ = err;
}
Err(err) => eprintln!("warning: the wg key store could not be opened: {err}"),
} }
} }
@@ -3417,6 +3469,7 @@ struct MemberRow<'a> {
path: Option<&'a tsunagi::ipc::DataPathReport>, path: Option<&'a tsunagi::ipc::DataPathReport>,
/// What to call the member the first of those links goes to. /// What to call the member the first of those links goes to.
via_label: Option<String>, via_label: Option<String>,
data_error: Option<&'a str>,
/// The name it claimed that another member holds, and what to call the /// The name it claimed that another member holds, and what to call the
/// holder. /// holder.
name_taken: Option<(&'a str, String)>, name_taken: Option<(&'a str, String)>,
@@ -3462,6 +3515,7 @@ fn member_rows<'a>(network: &'a tsunagi::ipc::NetworkReport, own_id: &str) -> Ve
online: false, online: false,
path: None, path: None,
via_label: None, via_label: None,
data_error: None,
name_taken: None, name_taken: None,
transport: None, transport: None,
rtt_ms: None, rtt_ms: None,
@@ -3494,6 +3548,7 @@ fn member_rows<'a>(network: &'a tsunagi::ipc::NetworkReport, own_id: &str) -> Ve
row.path = Some(&peer.path); row.path = Some(&peer.path);
row.transport = Some(&peer.transport); row.transport = Some(&peer.transport);
row.rtt_ms = peer.rtt_ms; row.rtt_ms = peer.rtt_ms;
row.data_error = peer.data_error.as_deref();
} }
if let Some(overlay) = &network.overlay { if let Some(overlay) = &network.overlay {
for peer in &overlay.peers { for peer in &overlay.peers {
@@ -3874,9 +3929,6 @@ fn member_row(row: &MemberRow<'_>) -> report::Row {
} }
use tsunagi::ipc::DataPathReport; use tsunagi::ipc::DataPathReport;
let direct_control = row
.transport
.is_none_or(|transport| transport.eq_ignore_ascii_case("direct"));
let tunnel_up = row.tunnel.is_some_and(|tunnel| tunnel.is_up()); let tunnel_up = row.tunnel.is_some_and(|tunnel| tunnel.is_up());
let has_overlay = row.tunnel.is_some(); let has_overlay = row.tunnel.is_some();
// Online is online: a member reached through others is as much here as // Online is online: a member reached through others is as much here as
@@ -3890,7 +3942,7 @@ fn member_row(row: &MemberRow<'_>) -> report::Row {
"data path via {} · {hops} hops", "data path via {} · {hops} hops",
row.via_label.as_deref().unwrap_or("another member") row.via_label.as_deref().unwrap_or("another member")
), ),
false, true,
), ),
Some(DataPathReport::None) | None => ("connected · no data path yet".to_string(), false), Some(DataPathReport::None) | None => ("connected · no data path yet".to_string(), false),
}; };
@@ -3913,51 +3965,32 @@ fn member_row(row: &MemberRow<'_>) -> report::Row {
} }
detail.extend(taken); detail.extend(taken);
let health = if !data_path_good || !direct_control || (has_overlay && !tunnel_up) { let health = if !data_path_good || row.data_error.is_some() || (has_overlay && !tunnel_up) {
Health::Degraded Health::Degraded
} else { } else {
Health::Good Health::Good
}; };
let mut out = Row::new(health, row.label(), detail); let mut out = Row::new(health, row.label(), detail);
if let Some(reason) = row.data_error {
out = out.with_note(reason.to_owned());
}
if let Some(tunnel) = row.tunnel { if let Some(tunnel) = row.tunnel {
let proto = if tunnel.protocol.is_empty() {
"wg-quic"
} else {
&tunnel.protocol
};
out = out.with_note(match tunnel.handshake_secs_ago { out = out.with_note(match tunnel.handshake_secs_ago {
Some(secs) => { Some(_) => format!(
let health = if proto == "tcp-tls" { "tx {} rx {} · IP bytes ↑{} ↓{}{} · {}",
format!("uptime {secs}s") tunnel.tx_packets,
tunnel.rx_packets,
tunnel.tx_bytes,
tunnel.rx_bytes,
if tunnel.dropped > 0 {
format!(", {} dropped", tunnel.dropped)
} else { } else {
format!("handshake {secs}s ago") String::new()
}; },
format!( tunnel.path
"end-to-end tunnel {proto} · {health} · tx {} rx {}{} · transport detail: {}", ),
tunnel.tx_packets, None => "encrypted data channel is not ready yet".into(),
tunnel.rx_packets,
if tunnel.dropped > 0 {
format!(", {} dropped", tunnel.dropped)
} else {
String::new()
},
tunnel.path
)
}
None => {
if proto == "wg-quic" || proto == "wg" {
format!(
"end-to-end tunnel {proto} · no WireGuard handshake yet; the tunnel \
cannot carry traffic"
)
} else {
format!(
"end-to-end tunnel {proto} · no {proto} handshake yet; the tunnel \
cannot carry traffic"
)
}
}
}); });
} }
out out
@@ -3992,7 +4025,7 @@ fn host_section(observed: &Observed) -> report::Section {
host.push(Row::new( host.push(Row::new(
Health::Info, Health::Info,
"implementation", "implementation",
"userspace overlay (boringtun WireGuard, TCP TLS 1.3); no kernel module needed", "userspace overlay with independent encrypted data transports; no kernel module needed",
)); ));
{ {
if cfg!(target_os = "linux") { if cfg!(target_os = "linux") {
@@ -4657,7 +4690,8 @@ async fn up(args: UpArgs) -> Result<(), Box<dyn std::error::Error>> {
.collect(); .collect();
config.default_protocols = Some(initial); config.default_protocols = Some(initial);
let settings = parse_settings(&args.protocol_options)?; let mut settings = parse_settings(&args.protocol_options)?;
let tunnel_settings = take_tunnel_settings(&mut settings)?;
// A setting nobody takes is a mistake, not a preference: one that was // A setting nobody takes is a mistake, not a preference: one that was
// silently dropped looks exactly like one that did not work. // silently dropped looks exactly like one that did not work.
for setting in &settings { for setting in &settings {
@@ -4676,13 +4710,9 @@ async fn up(args: UpArgs) -> Result<(), Box<dyn std::error::Error>> {
// The interface belongs to the agent, so it is configured once whatever // The interface belongs to the agent, so it is configured once whatever
// was selected to carry traffic over it. // was selected to carry traffic over it.
let mut wireguard = None;
let mut awg_transport = None; let mut awg_transport = None;
let mut awg_plugin = None;
let mut tcp_tls_transport = None; let mut tcp_tls_transport = None;
let mut tcp_tls_plugin = None; let mut pure_wg_transport: Option<Arc<AwgTransport>> = None;
let mut pure_wg_plugin: Option<Arc<PureWgPlugin>> = None;
let mut pure_wg_transport: Option<Arc<PureWgTransport>> = None;
if !PROTOCOLS.is_empty() { if !PROTOCOLS.is_empty() {
let tun_factory: Arc<dyn TunFactory> = if args.no_tun { let tun_factory: Arc<dyn TunFactory> = if args.no_tun {
Arc::new(MemoryTunFactory::new()) Arc::new(MemoryTunFactory::new())
@@ -4693,6 +4723,25 @@ async fn up(args: UpArgs) -> Result<(), Box<dyn std::error::Error>> {
config = config.with_interface(tun_factory, args.interface.clone(), mtu); config = config.with_interface(tun_factory, args.interface.clone(), mtu);
} }
let mut tunnel_config = WireguardConfig::new(paths.state_dir.join("wg-quic"));
tunnel_config.legacy_key_stores = ["wg", "awg"]
.into_iter()
.map(|name| paths.state_dir.join(name).join("wireguard.sqlite"))
.collect();
if let Some(mtu) = args.mtu {
tunnel_config = tunnel_config.with_mtu(mtu);
}
let tunnel =
WireguardPlugin::open(WireguardPlugin::configure(tunnel_config, &tunnel_settings)?).await?;
config = config.with_tunnel(tunnel.clone());
let wireguard = Some(tunnel);
let identity_db = paths.state_db();
let identity = tokio::task::spawn_blocking(move || {
tsunagi::storage::StateStore::open(identity_db)?.load_or_create_device_identity()
})
.await??;
for spec in PROTOCOLS { for spec in PROTOCOLS {
// Unqualified options keep targeting the initial selection; explicitly // Unqualified options keep targeting the initial selection; explicitly
// scoped options also configure protocols that will be enabled later. // scoped options also configure protocols that will be enabled later.
@@ -4706,16 +4755,6 @@ async fn up(args: UpArgs) -> Result<(), Box<dyn std::error::Error>> {
let options = settings_for(spec, &scoped); let options = settings_for(spec, &scoped);
match spec.name { match spec.name {
AWG_PROTOCOL => { AWG_PROTOCOL => {
let mut wg = WireguardConfig::new(paths.state_dir.join("awg"));
if let Some(mtu) = args.mtu {
wg = wg.with_mtu(mtu);
}
let tunnel_options: Vec<_> = options
.iter()
.filter(|(key, _)| key != "stun")
.cloned()
.collect();
let wg = WireguardPlugin::configure(wg, &tunnel_options)?;
let server = options let server = options
.iter() .iter()
.find(|(key, _)| key == "stun") .find(|(key, _)| key == "stun")
@@ -4726,42 +4765,32 @@ async fn up(args: UpArgs) -> Result<(), Box<dyn std::error::Error>> {
} }
let stun = (!matches!(args.reach, Reach::Local) && server != "off") let stun = (!matches!(args.reach, Reach::Local) && server != "off")
.then(|| server.to_owned()); .then(|| server.to_owned());
let state_db = paths.state_db();
let identity = tokio::task::spawn_blocking(move || {
tsunagi::storage::StateStore::open(state_db)?.load_or_create_device_identity()
})
.await??;
let transport = AwgTransport::new_with_stun( let transport = AwgTransport::new_with_stun(
identity.endpoint_id(), identity.endpoint_id(),
std::net::Ipv4Addr::UNSPECIFIED, std::net::Ipv4Addr::UNSPECIFIED,
stun, stun,
); );
let plugin = AwgPlugin::open(wg, transport.clone()).await?; transport.exclude_interface(args.interface.clone());
let plugin = AwgPlugin::new(transport.clone());
config = config config = config
.with_plugin(plugin.clone() as Arc<dyn IpPlugin>) .with_carrier(
plugin.clone() as Arc<dyn tsunagi::dataplane::transport::DataCarrier>
)
.with_custom_transport( .with_custom_transport(
AWG_PROTOCOL, AWG_PROTOCOL,
transport.clone() transport.clone()
as Arc<dyn tsunagi::dataplane::transport::PacketTransport>, as Arc<dyn tsunagi::dataplane::transport::PacketTransport>,
); );
awg_transport = Some(transport); awg_transport = Some(transport);
awg_plugin = Some(plugin);
} }
tsunagi_wg_quic::WIREGUARD_PROTOCOL => { "wg-quic" => {
let mut wg = WireguardConfig::new(paths.state_dir.join("wg-quic")); config = config.with_carrier(Arc::new(
if let Some(mtu) = args.mtu { tsunagi::dataplane::transport::iroh_link::IrohCarrier,
wg = wg.with_mtu(mtu); ));
}
let wg = WireguardPlugin::configure(wg, &options)?;
let plugin = WireguardPlugin::open(wg).await?;
config = config.with_plugin(plugin.clone() as Arc<dyn IpPlugin>);
wireguard = Some(plugin);
} }
TCP_TLS_PROTOCOL => { TCP_TLS_PROTOCOL => {
let tls_cfg = TcpTlsConfig::new(); let tls_cfg = TcpTlsConfig::new();
let tls_cfg = TcpTlsCodec::configure(tls_cfg, &options)?; let tls_cfg = TcpTlsConfig::configure(tls_cfg, &options)?;
let identity = tsunagi::storage::StateStore::open(paths.state_db())?
.load_or_create_device_identity()?;
let transport = Arc::new( let transport = Arc::new(
TcpTlsTransport::bind(&identity, tls_cfg.port, tls_cfg.sni) TcpTlsTransport::bind(&identity, tls_cfg.port, tls_cfg.sni)
.await .await
@@ -4772,46 +4801,44 @@ async fn up(args: UpArgs) -> Result<(), Box<dyn std::error::Error>> {
transport.set_peer_addr(ep_addr.id, *ip); transport.set_peer_addr(ep_addr.id, *ip);
} }
} }
let codec = TcpTlsCodec::new(identity.endpoint_id(), Arc::clone(&transport)); let plugin =
let plugin = Arc::new(TcpTlsPlugin::new(codec)); TcpTlsCarrier::with_interface(transport.clone(), args.interface.clone());
config = config config = config
.with_plugin(plugin.clone() as Arc<dyn IpPlugin>) .with_carrier(
plugin.clone() as Arc<dyn tsunagi::dataplane::transport::DataCarrier>
)
.with_custom_transport( .with_custom_transport(
TCP_TLS_PROTOCOL, TCP_TLS_PROTOCOL,
Arc::clone(&transport) Arc::clone(&transport)
as Arc<dyn tsunagi::dataplane::transport::PacketTransport>, as Arc<dyn tsunagi::dataplane::transport::PacketTransport>,
); );
tcp_tls_transport = Some(transport); tcp_tls_transport = Some(transport);
tcp_tls_plugin = Some(plugin);
} }
WG_PROTOCOL => { WG_PROTOCOL => {
let mut wg_cfg = PureWgConfig::new(paths.state_dir.join("wg")); let port = options
if let Some(mtu) = args.mtu { .iter()
wg_cfg = wg_cfg.with_mtu(mtu); .find(|(key, _)| key == "port")
} .map(|(_, value)| value.parse::<u16>())
let wg_cfg = PureWgPlugin::configure(wg_cfg, &options)?; .transpose()?
let identity = tsunagi::storage::StateStore::open(paths.state_db())? .unwrap_or(51820);
.load_or_create_device_identity()?; let transport = AwgTransport::wireguard_with_port(
let transport = Arc::new( identity.endpoint_id(),
PureWgTransport::bind(&identity, wg_cfg.port) std::net::Ipv4Addr::UNSPECIFIED,
.await (!matches!(args.reach, Reach::Local)).then(|| "stun.l.google.com:19302".into()),
.map_err(|err| err.to_string())?, port,
); );
for ep_addr in &bootstrap { transport.exclude_interface(args.interface.clone());
for addr in ep_addr.ip_addrs() { let plugin = AwgPlugin::new(transport.clone());
transport.set_peer_addr(ep_addr.id, *addr);
}
}
let plugin = PureWgPlugin::open(wg_cfg, Arc::clone(&transport)).await?;
config = config config = config
.with_plugin(plugin.clone() as Arc<dyn IpPlugin>) .with_carrier(
plugin.clone() as Arc<dyn tsunagi::dataplane::transport::DataCarrier>
)
.with_custom_transport( .with_custom_transport(
WG_PROTOCOL, WG_PROTOCOL,
Arc::clone(&transport) Arc::clone(&transport)
as Arc<dyn tsunagi::dataplane::transport::PacketTransport>, as Arc<dyn tsunagi::dataplane::transport::PacketTransport>,
); );
pure_wg_transport = Some(transport); pure_wg_transport = Some(transport);
pure_wg_plugin = Some(plugin);
} }
other => return Err(format!("`{other}` is listed but not built in").into()), other => return Err(format!("`{other}` is listed but not built in").into()),
} }
@@ -4876,18 +4903,13 @@ async fn up(args: UpArgs) -> Result<(), Box<dyn std::error::Error>> {
}); });
} }
if let Some(transport) = pure_wg_transport { if let Some(transport) = pure_wg_transport {
let agent_inbound = agent.clone(); let inbound_agent = agent.clone();
tokio::spawn(async move { tokio::spawn(async move {
transport while let Some(inbound) = transport.next_inbound().await {
.accept_loop(move |inbound| { if let Err(err) = inbound_agent.install_inbound_link(inbound).await {
let agent = agent_inbound.clone(); tracing::debug!(%err, "failed to install inbound wg link");
tokio::spawn(async move { }
if let Err(err) = agent.install_inbound_link(inbound).await { }
tracing::debug!(%err, "failed to install inbound wg link");
}
});
})
.await;
}); });
} }
@@ -4926,17 +4948,11 @@ async fn up(args: UpArgs) -> Result<(), Box<dyn std::error::Error>> {
// is not fatal: the agent itself works fine without it. // is not fatal: the agent itself works fine without it.
let control = { let control = {
let agent = agent.clone(); let agent = agent.clone();
let pure_wg = pure_wg_plugin.clone();
let awg = awg_plugin.clone();
let plugin = wireguard.clone(); let plugin = wireguard.clone();
let tcp_tls = tcp_tls_plugin.clone();
let dns = Arc::clone(&dns); let dns = Arc::clone(&dns);
let source: Arc<dyn tsunagi::ipc::ReportSource> = Arc::new(AgentControl { let source: Arc<dyn tsunagi::ipc::ReportSource> = Arc::new(AgentControl {
agent, agent,
pure_wg,
awg,
plugin, plugin,
tcp_tls,
dns, dns,
paths: paths.clone(), paths: paths.clone(),
}); });
@@ -5019,10 +5035,7 @@ async fn up(args: UpArgs) -> Result<(), Box<dyn std::error::Error>> {
/// both. /// both.
async fn build_report( async fn build_report(
agent: &Agent, agent: &Agent,
pure_wg: Option<&PureWgPlugin>,
awg: Option<&AwgPlugin>,
wireguard: Option<&WireguardPlugin>, wireguard: Option<&WireguardPlugin>,
tcp_tls: Option<&TcpTlsPlugin>,
dns: Option<DnsState>, dns: Option<DnsState>,
paths: &StoragePaths, paths: &StoragePaths,
) -> tsunagi::ipc::StatusReport { ) -> tsunagi::ipc::StatusReport {
@@ -5041,226 +5054,37 @@ async fn build_report(
.networks .networks
.iter() .iter()
.map(|network| { .map(|network| {
let awg_view = awg.and_then(|plugin| plugin.overview(network.network_id));
let wg_view = wireguard.and_then(|plugin| plugin.overview(network.network_id)); let wg_view = wireguard.and_then(|plugin| plugin.overview(network.network_id));
let pure_wg_view = pure_wg.and_then(|plugin| plugin.overview(network.network_id)); let wg_overlay_addr = wg_view.as_ref().and_then(|v| v.overlay_address_v4);
let tcp_tls_reports = tcp_tls
.map(|p| p.active_peers(network.network_id))
.unwrap_or_default();
// Use whichever WireGuard view is available for the overlay
// address and range — wg-quic and pure wg both derive them
// identically from the control plane's allocations.
let wg_overlay_addr = wg_view
.as_ref()
.and_then(|v| v.overlay_address_v4)
.or_else(|| pure_wg_view.as_ref().and_then(|v| v.overlay_address_v4))
.or_else(|| awg_view.as_ref().and_then(|v| v.overlay_address_v4));
let wg_prefix_len = wg_view let wg_prefix_len = wg_view
.as_ref() .as_ref()
.and_then(|v| v.ipv4_range) .and_then(|v| v.ipv4_range)
.or_else(|| pure_wg_view.as_ref().and_then(|v| v.ipv4_range))
.or_else(|| awg_view.as_ref().and_then(|v| v.ipv4_range))
.map_or(0, |r| r.prefix_len); .map_or(0, |r| r.prefix_len);
let has_any_plugin = wg_view.is_some() let has_any_plugin = wg_view.is_some();
|| pure_wg_view.is_some()
|| awg_view.is_some()
|| !tcp_tls_reports.is_empty();
let overlay = if has_any_plugin && overlay.is_some() { let overlay = if has_any_plugin && overlay.is_some() {
let mut peers: Vec<OverlayPeerReport> = Vec::new(); let mut peers: Vec<OverlayPeerReport> = Vec::new();
// Collect all unique peer IDs known in this network. if let Some(view) = &wg_view {
let mut all_peer_ids: Vec<EndpointId> = Vec::new();
for member in &network.members {
if member.endpoint_id != status.endpoint_id
&& !all_peer_ids.contains(&member.endpoint_id)
{
all_peer_ids.push(member.endpoint_id);
}
}
if let Some(ref view) = pure_wg_view {
for peer in &view.peers { for peer in &view.peers {
if !all_peer_ids.contains(&peer.endpoint_id) { let tunnel = peer.tunnel.as_ref();
all_peer_ids.push(peer.endpoint_id);
}
}
}
if let Some(ref view) = wg_view {
for peer in &view.peers {
if !all_peer_ids.contains(&peer.endpoint_id) {
all_peer_ids.push(peer.endpoint_id);
}
}
}
if let Some(ref view) = awg_view {
for peer in &view.peers {
if !all_peer_ids.contains(&peer.endpoint_id) {
all_peer_ids.push(peer.endpoint_id);
}
}
}
for report in &tcp_tls_reports {
if !all_peer_ids.contains(&report.peer) {
all_peer_ids.push(report.peer);
}
}
for peer_id in all_peer_ids {
let pure_wg_match = pure_wg_view
.as_ref()
.and_then(|v| v.peers.iter().find(|p| p.endpoint_id == peer_id));
let awg_match = awg_view
.as_ref()
.and_then(|v| v.peers.iter().find(|p| p.endpoint_id == peer_id));
let tcp_tls_match = tcp_tls_reports.iter().find(|r| r.peer == peer_id);
let wg_quic_match = wg_view
.as_ref()
.and_then(|v| v.peers.iter().find(|p| p.endpoint_id == peer_id));
let member_addr = network
.members
.iter()
.find(|m| m.endpoint_id == peer_id)
.and_then(|m| m.overlay_address_v4)
.map(|a| a.to_string());
if let Some(pure_peer) = pure_wg_match
&& let Some(tunnel) = &pure_peer.tunnel
{
peers.push(OverlayPeerReport { peers.push(OverlayPeerReport {
endpoint_id: peer_id.to_string(), counter_epoch: tunnel.map_or(0, |t| t.stats.counter_epoch),
protocol: "wg".into(), endpoint_id: peer.endpoint_id.to_string(),
public_key: pure_peer.public_key.to_string(), protocol: String::new(),
address: pure_peer public_key: peer.public_key.to_string(),
.overlay_address_v4 address: peer.overlay_address_v4.map(|a| a.to_string()),
.map(|a| a.to_string()) handshake_secs_ago: tunnel
.or(member_addr), .and_then(|t| t.health.since_handshake.map(|s| s.as_secs())),
handshake_secs_ago: tunnel.health.since_handshake.map(|s| s.as_secs()), tx_packets: tunnel.map_or(0, |t| t.stats.tx_packets),
tx_packets: tunnel.stats.tx_packets, rx_packets: tunnel.map_or(0, |t| t.stats.rx_packets),
rx_packets: tunnel.stats.rx_packets, tx_bytes: tunnel.map_or(0, |t| t.stats.tx_bytes),
tx_bytes: tunnel.stats.tx_bytes, rx_bytes: tunnel.map_or(0, |t| t.stats.rx_bytes),
rx_bytes: tunnel.stats.rx_bytes, dropped: tunnel.map_or(0, |t| {
dropped: tunnel.stats.dropped_wrong_source t.stats.dropped_wrong_source + t.stats.dropped_oversize
+ tunnel.stats.dropped_oversize, }),
protocol_errors: tunnel.stats.protocol_errors, protocol_errors: tunnel.map_or(0, |t| t.stats.protocol_errors),
path: tunnel.path.clone(), path: tunnel.map_or_else(|| "no data path".into(), |t| t.path.clone()),
});
} else if let Some(tcp_tls) = tcp_tls_match {
let pubkey = pure_wg_match
.map(|p| p.public_key.to_string())
.or_else(|| wg_quic_match.map(|p| p.public_key.to_string()))
.or_else(|| awg_match.map(|p| p.public_key.to_string()))
.unwrap_or_default();
peers.push(OverlayPeerReport {
endpoint_id: peer_id.to_string(),
protocol: tcp_tls.protocol.clone(),
public_key: pubkey,
address: member_addr,
handshake_secs_ago: Some(tcp_tls.uptime_secs),
tx_packets: tcp_tls.tx_packets,
rx_packets: tcp_tls.rx_packets,
tx_bytes: tcp_tls.tx_bytes,
rx_bytes: tcp_tls.rx_bytes,
dropped: tcp_tls.dropped,
protocol_errors: 0,
path: tcp_tls.path.clone(),
});
} else if let Some(awg_peer) = awg_match
&& let Some(tunnel) = &awg_peer.tunnel
{
peers.push(OverlayPeerReport {
endpoint_id: peer_id.to_string(),
protocol: AWG_PROTOCOL.into(),
public_key: awg_peer.public_key.to_string(),
address: awg_peer
.overlay_address_v4
.map(|a| a.to_string())
.or(member_addr),
handshake_secs_ago: tunnel.health.since_handshake.map(|s| s.as_secs()),
tx_packets: tunnel.stats.tx_packets,
rx_packets: tunnel.stats.rx_packets,
tx_bytes: tunnel.stats.tx_bytes,
rx_bytes: tunnel.stats.rx_bytes,
dropped: tunnel.stats.dropped_wrong_source
+ tunnel.stats.dropped_oversize,
protocol_errors: tunnel.stats.protocol_errors,
path: tunnel.path.clone(),
});
} else if let Some(wg_peer) = wg_quic_match
&& let Some(tunnel) = &wg_peer.tunnel
{
peers.push(OverlayPeerReport {
endpoint_id: peer_id.to_string(),
protocol: "wg-quic".into(),
public_key: wg_peer.public_key.to_string(),
address: wg_peer
.overlay_address_v4
.map(|a| a.to_string())
.or(member_addr),
handshake_secs_ago: tunnel.health.since_handshake.map(|s| s.as_secs()),
tx_packets: tunnel.stats.tx_packets,
rx_packets: tunnel.stats.rx_packets,
tx_bytes: tunnel.stats.tx_bytes,
rx_bytes: tunnel.stats.rx_bytes,
dropped: tunnel.stats.dropped_wrong_source
+ tunnel.stats.dropped_oversize,
protocol_errors: tunnel.stats.protocol_errors,
path: tunnel.path.clone(),
});
} else if let Some(pure_peer) = pure_wg_match {
peers.push(OverlayPeerReport {
endpoint_id: peer_id.to_string(),
protocol: "wg".into(),
public_key: pure_peer.public_key.to_string(),
address: pure_peer
.overlay_address_v4
.map(|a| a.to_string())
.or(member_addr),
handshake_secs_ago: None,
tx_packets: 0,
rx_packets: 0,
tx_bytes: 0,
rx_bytes: 0,
dropped: 0,
protocol_errors: 0,
path: "no data link".into(),
});
} else if let Some(awg_peer) = awg_match {
peers.push(OverlayPeerReport {
endpoint_id: peer_id.to_string(),
protocol: AWG_PROTOCOL.into(),
public_key: awg_peer.public_key.to_string(),
address: awg_peer
.overlay_address_v4
.map(|a| a.to_string())
.or(member_addr),
handshake_secs_ago: None,
tx_packets: 0,
rx_packets: 0,
tx_bytes: 0,
rx_bytes: 0,
dropped: 0,
protocol_errors: 0,
path: "no data link".into(),
});
} else if let Some(wg_peer) = wg_quic_match {
peers.push(OverlayPeerReport {
endpoint_id: peer_id.to_string(),
protocol: "wg-quic".into(),
public_key: wg_peer.public_key.to_string(),
address: wg_peer
.overlay_address_v4
.map(|a| a.to_string())
.or(member_addr),
handshake_secs_ago: None,
tx_packets: 0,
rx_packets: 0,
tx_bytes: 0,
rx_bytes: 0,
dropped: 0,
protocol_errors: 0,
path: "no data link".into(),
}); });
} }
} }
@@ -5307,6 +5131,7 @@ async fn build_report(
}; };
NetworkReport { NetworkReport {
vips: network.vips.clone(),
protocols: network.protocols.clone(), protocols: network.protocols.clone(),
auto_start: network.auto_start, auto_start: network.auto_start,
exit: exit_report(network), exit: exit_report(network),
@@ -5322,6 +5147,7 @@ async fn build_report(
.peers .peers
.iter() .iter()
.map(|peer| PeerReport { .map(|peer| PeerReport {
data_error: peer.data_error.clone(),
endpoint_id: peer.endpoint_id.to_string(), endpoint_id: peer.endpoint_id.to_string(),
hostname: peer.hostname.clone(), hostname: peer.hostname.clone(),
transport: peer.transport.to_string(), transport: peer.transport.to_string(),
@@ -5601,11 +5427,11 @@ fn log_event(event: &Event) {
path, path,
max_datagram, max_datagram,
} => { } => {
tracing::info!( if protocol == tsunagi_wg_quic::WIREGUARD_PROTOCOL {
network = %network.fmt_short(), tracing::info!(network = %network.fmt_short(), peer = %peer.fmt_short(), "Data route available ({path})");
peer = %peer.fmt_short(), } else {
"Data link established for {protocol} ({path}, MTU datagram: {max_datagram}B)" tracing::info!(network = %network.fmt_short(), peer = %peer.fmt_short(), "Data link established for {protocol} ({path}, MTU datagram: {max_datagram}B)");
); }
} }
Event::DataLinkDown { Event::DataLinkDown {
network, network,
@@ -5740,6 +5566,7 @@ mod status_tests {
/// The situation that prompted this: one peer left and came back. /// The situation that prompted this: one peer left and came back.
fn network_after_a_peer_returned() -> NetworkReport { fn network_after_a_peer_returned() -> NetworkReport {
NetworkReport { NetworkReport {
vips: Vec::new(),
protocols: vec!["wg-quic".into()], protocols: vec!["wg-quic".into()],
auto_start: true, auto_start: true,
exit: Default::default(), exit: Default::default(),
@@ -5753,6 +5580,7 @@ mod status_tests {
relay_sent_via: 0, relay_sent_via: 0,
relay_received_via: 0, relay_received_via: 0,
peers: vec![PeerReport { peers: vec![PeerReport {
data_error: None,
endpoint_id: ONLINE.into(), endpoint_id: ONLINE.into(),
hostname: Some("music".into()), hostname: Some("music".into()),
transport: "direct".into(), transport: "direct".into(),
@@ -6243,7 +6071,7 @@ mod status_tests {
"names the member it goes through: {text}" "names the member it goes through: {text}"
); );
assert!(!text.contains("offline"), "{text}"); assert!(!text.contains("offline"), "{text}");
assert_eq!(out.worst(), Health::Degraded, "{text}"); assert_eq!(out.worst(), Health::Good, "{text}");
} }
#[test] #[test]
@@ -6277,14 +6105,14 @@ mod status_tests {
} }
#[test] #[test]
fn a_relayed_peer_is_graded_as_degraded_quality() { fn relayed_control_does_not_degrade_a_working_data_path() {
let mut network = network_after_a_peer_returned(); let mut network = network_after_a_peer_returned();
network.peers[0].transport = "relay".into(); network.peers[0].transport = "relay".into();
let mut out = report::Report::new(); let mut out = report::Report::new();
out.push(network_section(&network, OWN, false)); out.push(network_section(&network, OWN, false));
assert_eq!(out.worst(), Health::Degraded, "{}", out.render(false)); assert_eq!(out.worst(), Health::Good, "{}", out.render(false));
assert!(out.render(false).contains("relay")); assert!(out.render(false).contains("control path relay"));
} }
#[test] #[test]
@@ -6296,7 +6124,10 @@ mod status_tests {
out.push(network_section(&network, OWN, false)); out.push(network_section(&network, OWN, false));
let text = out.render(false); let text = out.render(false);
assert_eq!(out.worst(), Health::Degraded, "{text}"); assert_eq!(out.worst(), Health::Degraded, "{text}");
assert!(text.contains("no WireGuard handshake yet"), "{text}"); assert!(
text.contains("encrypted data channel is not ready yet"),
"{text}"
);
} }
#[test] #[test]
@@ -6319,7 +6150,7 @@ mod status_tests {
} }
#[test] #[test]
fn a_tcp_tls_tunnel_shows_healthy_state_and_metrics() { fn a_tcp_tls_carrier_shows_healthy_state_and_metrics() {
let mut network = network_after_a_peer_returned(); let mut network = network_after_a_peer_returned();
let mut t = tunnel(ONLINE, Some(42)); let mut t = tunnel(ONLINE, Some(42));
t.protocol = "tcp-tls".into(); t.protocol = "tcp-tls".into();
@@ -6332,17 +6163,14 @@ mod status_tests {
out.push(network_section(&network, OWN, false)); out.push(network_section(&network, OWN, false));
let text = out.render(false); let text = out.render(false);
assert_eq!(out.worst(), Health::Good, "{text}"); assert_eq!(out.worst(), Health::Good, "{text}");
assert!(text.contains("end-to-end tunnel tcp-tls"), "{text}"); assert!(!text.contains("end-to-end"), "{text}");
assert!(text.contains("uptime 42s"), "{text}"); assert!(!text.contains("handshake"), "{text}");
assert!(text.contains("tx 100 rx 200"), "{text}"); assert!(text.contains("tx 100 rx 200"), "{text}");
assert!( assert!(text.contains("tcp-tls via 192.0.2.1:443"), "{text}");
text.contains("transport detail: tcp-tls via 192.0.2.1:443"),
"{text}"
);
} }
#[test] #[test]
fn end_to_end_tunnel_and_data_transport_are_named_separately() { fn status_names_only_the_external_data_transport() {
let mut network = network_after_a_peer_returned(); let mut network = network_after_a_peer_returned();
network.peers[0].path = tsunagi::ipc::DataPathReport::Direct { network.peers[0].path = tsunagi::ipc::DataPathReport::Direct {
transport: "tcp-tls".into(), transport: "tcp-tls".into(),
@@ -6358,20 +6186,41 @@ mod status_tests {
let text = out.render(false); let text = out.render(false);
assert!(text.contains("data path direct over tcp-tls"), "{text}"); assert!(text.contains("data path direct over tcp-tls"), "{text}");
assert!(text.contains("control path direct"), "{text}"); assert!(text.contains("control path direct"), "{text}");
assert!( assert!(!text.contains("end-to-end"), "{text}");
text.contains("end-to-end tunnel wg · handshake 1s ago"), assert!(!text.contains("handshake"), "{text}");
"{text}" assert!(text.contains("tcp-tls via 192.0.2.1:443"), "{text}");
);
assert!(
text.contains("transport detail: tcp-tls via 192.0.2.1:443"),
"{text}"
);
} }
} }
#[cfg(test)] #[cfg(test)]
mod network_tests { mod network_tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)] #![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
#[test]
fn historic_cipher_options_configure_one_tunnel_and_never_reach_carriers() {
let mut settings = super::parse_settings(&[
"keepalive=10".into(),
"wg:keepalive=10".into(),
"wg-quic:mtu=1400".into(),
"wg:port=0".into(),
])
.unwrap();
let shared = super::take_tunnel_settings(&mut settings).unwrap();
assert_eq!(
shared,
[
("keepalive".into(), "10".into()),
("mtu".into(), "1400".into())
]
);
assert_eq!(settings.len(), 1);
assert_eq!(settings[0].key, "port");
let mut conflict =
super::parse_settings(&["wg:keepalive=10".into(), "awg:keepalive=20".into()]).unwrap();
assert!(super::take_tunnel_settings(&mut conflict).is_err());
let mut unknown = super::parse_settings(&["typo:mtu=1400".into()]).unwrap();
assert!(super::take_tunnel_settings(&mut unknown).is_err());
}
#[test] #[test]
fn default_carriers_include_awg_and_explicit_order_is_preserved() { fn default_carriers_include_awg_and_explicit_order_is_preserved() {
use clap::Parser; use clap::Parser;
crates/tsunagi/src/vip_cli.rs
+506
View File
@@ -0,0 +1,506 @@
//! Named virtual service administration; every mutation submits one complete draft.
use super::{PathArgs, control_socket, resolve_network_index};
use clap::{Args, Subcommand};
use std::path::PathBuf;
use tsunagi::ipc::{self, NetworkReport};
use tsunagi::vip::{Backend, Listener, Protocol, VipConfig, VipStatus};
type Result<T> = std::result::Result<T, Box<dyn std::error::Error>>;
#[derive(Debug, Args)]
#[command(
after_help = "Examples:\n tsng vip create lab k8s\n tsng vip port add lab k8s tcp/443 --backend cp1:6443 --backend cp2:7443\n tsng vip backend add lab k8s tcp/443 cp3:6443\n tsng vip status lab k8s --watch\n tsng vip edit lab k8s\n\nNames, full ids and unique id prefixes are accepted. Changes are shared with the network.\nTCP port checks default to on; UDP uses data-path probes and explicit ICMP errors."
)]
pub(crate) struct VipArgs {
#[command(flatten)]
paths: PathArgs,
/// Agent control socket; defaults to the running service.
#[arg(long, global = true, env = "TSUNAGI_CONTROL_SOCKET")]
control_socket: Option<PathBuf>,
/// Machine-readable configuration and local observations.
#[arg(long, global = true)]
json: bool,
#[command(subcommand)]
action: Action,
}
#[derive(Debug, Args)]
struct Target {
/// Network exact name, id or unique id prefix.
network: String,
/// VIP exact name, id or unique id prefix.
vip: String,
}
#[derive(Debug, Args)]
struct PortTarget {
#[command(flatten)]
target: Target,
/// Listener as tcp/443 or udp/53.
listener: String,
}
#[derive(Debug, Subcommand)]
enum Action {
/// List services in one network, or every configured network.
List { network: Option<String> },
/// Create a named VIP with an automatically reserved internal address.
Create { network: String, name: String },
/// Show the complete saved configuration and local status.
Show(Target),
/// Show local health, routing choices and cumulative traffic counters.
Status {
#[command(flatten)]
target: Target,
/// Refresh once a second until interrupted.
#[arg(long)]
watch: bool,
},
/// Edit the entire configuration in VISUAL/EDITOR and save it atomically.
Edit(Target),
/// Remove a VIP; existing flows drain until completion or idle expiry.
Remove(Target),
/// Add, configure or remove listeners independently of the VIP.
Port {
#[command(subcommand)]
action: PortAction,
},
/// Change a listener's destination nodes and application ports.
Backend {
#[command(subcommand)]
action: BackendAction,
},
}
#[derive(Debug, Subcommand)]
enum PortAction {
/// Add a listener, optionally with its complete initial backend set.
Add {
#[command(flatten)]
target: PortTarget,
/// Node name/id followed by :port. Repeat for multiple destinations.
#[arg(long)]
backend: Vec<String>,
},
/// Change check settings and flow idle expiry.
Set {
#[command(flatten)]
target: PortTarget,
/// Active TCP connect checks: on or off.
#[arg(long,value_parser=["on","off"])]
port_check: Option<String>,
/// Milliseconds between active checks (250..3600000).
#[arg(long)]
check_interval_ms: Option<u32>,
/// TCP check deadline in milliseconds (100..60000).
#[arg(long)]
check_timeout_ms: Option<u32>,
/// Consecutive failures before exclusion.
#[arg(long)]
failures: Option<u8>,
/// Consecutive successes before recovery.
#[arg(long)]
successes: Option<u8>,
/// Flow idle expiry in seconds (1..604800).
#[arg(long)]
idle_timeout: Option<u32>,
},
/// Remove a listener from new-flow selection; existing flows drain.
Remove(PortTarget),
}
#[derive(Debug, Subcommand)]
enum BackendAction {
/// Add a destination node and port to a listener.
Add {
#[command(flatten)]
target: PortTarget,
backend: String,
},
/// Remove a destination from new-flow selection.
Remove {
#[command(flatten)]
target: PortTarget,
backend: String,
},
}
impl Action {
fn network(&self) -> Option<&str> {
match self {
Self::List { network } => network.as_deref(),
Self::Create { network, .. } => Some(network),
Self::Show(t) | Self::Edit(t) | Self::Remove(t) | Self::Status { target: t, .. } => {
Some(&t.network)
}
Self::Port { action } => Some(
&match action {
PortAction::Add { target, .. }
| PortAction::Set { target, .. }
| PortAction::Remove(target) => target,
}
.target
.network,
),
Self::Backend { action } => Some(
&match action {
BackendAction::Add { target, .. } | BackendAction::Remove { target, .. } => {
target
}
}
.target
.network,
),
}
}
}
fn parse_listener(s: &str) -> Result<(Protocol, u16)> {
let (p, n) = s
.split_once('/')
.ok_or("listener must be tcp/PORT or udp/PORT")?;
let port = n.parse::<u16>()?;
if port == 0 {
return Err("port must not be zero".into());
}
Ok((p.parse()?, port))
}
fn backend(n: &NetworkReport, s: &str) -> Result<Backend> {
let (node, port) = s.rsplit_once(':').ok_or("backend must be NODE:PORT")?;
let peer = match ipc::resolve_node(n, node) {
Ok(member) => member.endpoint_id.clone(),
Err(error) => {
let matches = n
.members
.iter()
.any(|m| m.hostname.as_deref() == Some(node) || m.endpoint_id.starts_with(node));
if matches {
return Err(error.into());
}
node.parse::<iroh::EndpointId>()
.map_err(|_| error)?
.to_string()
}
};
let port = port.parse()?;
if port == 0 {
return Err("backend port must not be zero".into());
}
Ok(Backend { peer, port })
}
fn selected(n: &NetworkReport, t: &Target) -> Result<VipStatus> {
Ok(ipc::resolve_vip(n, &t.vip)?.clone())
}
fn listener<'a>(c: &'a mut VipConfig, target: &PortTarget) -> Result<&'a mut Listener> {
let (p, n) = parse_listener(&target.listener)?;
c.listeners
.iter_mut()
.find(|l| l.protocol == p && l.port == n)
.ok_or_else(|| format!("no listener {} on {}", target.listener, c.name).into())
}
fn name_of<'a>(n: &'a NetworkReport, peer: &'a str) -> &'a str {
n.members
.iter()
.find(|m| m.endpoint_id == peer)
.and_then(|m| m.hostname.as_deref())
.unwrap_or(peer)
}
fn show(n: &NetworkReport, v: &VipStatus, json: bool) -> Result<()> {
if json {
println!("{}", serde_json::to_string(v)?);
return Ok(());
}
println!(
"{}.{} {}",
v.config.name,
n.name,
v.error.as_deref().unwrap_or("configured")
);
for l in &v.listeners {
let choice = l
.selected
.as_ref()
.map(|b| format!("{}:{}", name_of(n, &b.peer), b.port))
.unwrap_or_else(|| "none".into());
println!(
" {}/{} next: {} flows: {} tx/rx: {}/{} bytes drops: {} UDP failovers: {}",
l.protocol,
l.port,
choice,
l.backends.iter().map(|b| b.active_flows).sum::<u32>(),
l.counters.tx_bytes,
l.counters.rx_bytes,
l.counters.dropped,
l.counters.failovers
);
for b in &l.backends {
let rtt = b
.rtt_us
.map(|n| format!("{:.2} ms", n as f64 / 1000.))
.unwrap_or_else(|| "unknown".into());
println!(
" {}:{} {} RTT {} active {}",
name_of(n, &b.backend.peer),
b.backend.port,
b.state,
rtt,
b.active_flows
);
}
}
Ok(())
}
pub(crate) async fn run(args: VipArgs) -> Result<()> {
let custom = args.paths.state_dir.is_some();
let paths = args.paths.resolve()?;
let socket = control_socket(&paths, args.control_socket.as_ref(), custom);
let report = ipc::request_status(&socket).await?;
if let Action::List { network: None } = &args.action {
for n in &report.networks {
for v in &n.vips {
show(n, v, args.json)?;
}
}
return Ok(());
}
let selector = args.action.network().ok_or("network is required")?;
let names: Vec<_> = report
.networks
.iter()
.map(|n| (n.network_id.clone(), n.name.clone()))
.collect();
let network = &report.networks[resolve_network_index(&names, selector)?];
if let Action::Status {
target,
watch: true,
} = &args.action
{
let mut previous: Option<(std::time::Instant, u64, u64)> = None;
loop {
let report = ipc::request_status(&socket).await?;
let n = report
.networks
.iter()
.find(|n| n.network_id == network.network_id)
.ok_or("network was removed")?;
let vip = selected(n, target)?;
let (tx, rx) = vip.listeners.iter().fold((0u64, 0u64), |a, l| {
(
a.0.saturating_add(l.counters.tx_bytes),
a.1.saturating_add(l.counters.rx_bytes),
)
});
let now = std::time::Instant::now();
if let Some((at, old_tx, old_rx)) = previous
&& !args.json
{
let secs = now.duration_since(at).as_secs_f64();
if secs > 0. {
println!(
"TX {:.1} KiB/s · RX {:.1} KiB/s",
tx.saturating_sub(old_tx) as f64 / secs / 1024.,
rx.saturating_sub(old_rx) as f64 / secs / 1024.
);
}
}
previous = Some((now, tx, rx));
show(n, &vip, args.json)?;
tokio::select! {_=tokio::signal::ctrl_c()=>break,_=tokio::time::sleep(std::time::Duration::from_secs(1))=>{}}
}
return Ok(());
}
let mut edited_file = None;
let (mut config, expected, delete) = match &args.action {
Action::List { .. } => {
for v in &network.vips {
show(network, v, args.json)?;
}
return Ok(());
}
Action::Show(t) | Action::Status { target: t, .. } => {
let v = selected(network, t)?;
show(network, &v, args.json)?;
if matches!(args.action, Action::Show(_)) && !args.json {
println!("{}", serde_json::to_string_pretty(&v.config)?);
}
return Ok(());
}
Action::Create { name, .. } => (VipConfig::new(name), None, false),
Action::Remove(t) => {
let v = selected(network, t)?;
(v.config, Some(v.revision), true)
}
Action::Edit(t) => {
let v = selected(network, t)?;
let mut file = tempfile::Builder::new()
.prefix("tsunagi-vip-")
.suffix(".json")
.tempfile()?;
use std::io::Write;
file.write_all(serde_json::to_string_pretty(&v.config)?.as_bytes())?;
file.flush()?;
let editor = std::env::var("VISUAL")
.or_else(|_| std::env::var("EDITOR"))
.unwrap_or_else(|_| "vi".into());
let mut words = editor.split_whitespace();
let program = words.next().ok_or("empty EDITOR")?;
let status = std::process::Command::new(program)
.args(words)
.arg(file.path())
.status()?;
if !status.success() {
return Err("editor failed; VIP unchanged".into());
}
let bytes = std::fs::read(file.path())?;
let cfg: VipConfig = match serde_json::from_slice(&bytes) {
Ok(c) => c,
Err(e) => {
let (_, path) = file.keep()?;
return Err(
format!("invalid draft: {e}; retained at {}", path.display()).into(),
);
}
};
if cfg.id != v.config.id {
return Err("VIP id cannot be edited".into());
}
edited_file = Some(file);
(cfg, Some(v.revision), false)
}
Action::Port { action } => {
let target = match action {
PortAction::Add { target, .. }
| PortAction::Set { target, .. }
| PortAction::Remove(target) => target,
};
let v = selected(network, &target.target)?;
let mut cfg = v.config;
match action {
PortAction::Add {
target,
backend: destinations,
} => {
let (p, port) = parse_listener(&target.listener)?;
let mut l = Listener::new(p, port);
l.backends = destinations
.iter()
.map(|b| backend(network, b))
.collect::<Result<_>>()?;
cfg.listeners.push(l);
}
PortAction::Remove(target) => {
let (p, port) = parse_listener(&target.listener)?;
listener(&mut cfg, target)?;
cfg.listeners.retain(|l| l.protocol != p || l.port != port);
}
PortAction::Set {
target,
port_check,
check_interval_ms,
check_timeout_ms,
failures,
successes,
idle_timeout,
} => {
let l = listener(&mut cfg, target)?;
if let Some(v) = port_check {
l.port_check = v == "on";
}
if let Some(v) = check_interval_ms {
l.check_interval_ms = *v;
}
if let Some(v) = check_timeout_ms {
l.check_timeout_ms = *v;
}
if let Some(v) = failures {
l.failure_threshold = *v;
}
if let Some(v) = successes {
l.recovery_threshold = *v;
}
if let Some(v) = idle_timeout {
l.idle_timeout_secs = *v;
}
}
}
(cfg, Some(v.revision), false)
}
Action::Backend { action } => {
let (target, destination, remove) = match action {
BackendAction::Add { target, backend } => (target, backend, false),
BackendAction::Remove { target, backend } => (target, backend, true),
};
let v = selected(network, &target.target)?;
let mut cfg = v.config;
let b = backend(network, destination)?;
let l = listener(&mut cfg, target)?;
if remove {
if !l.backends.contains(&b) {
return Err("backend is not configured".into());
}
l.backends.retain(|x| x != &b);
} else {
l.backends.push(b);
}
(cfg, Some(v.revision), false)
}
};
// Canonical order makes displayed drafts and shared state deterministic.
config.listeners.sort_by_key(|l| (l.protocol, l.port));
for l in &mut config.listeners {
l.backends.sort();
}
let id = config.id.clone();
let vips = match ipc::save_vip(&socket, &network.network_id, config, expected, delete).await {
Ok(vips) => vips,
Err(error) => {
if let Some(file) = edited_file {
let (_, path) = file.keep()?;
return Err(format!("{error}; draft retained at {}", path.display()).into());
}
return Err(error.into());
}
};
if delete {
if args.json {
println!("{{\"removed\":\"{id}\"}}");
} else {
println!("VIP removed; existing flows drain.");
}
} else if let Some(v) = vips.iter().find(|v| v.config.id == id) {
show(network, v, args.json)?;
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
use clap::Parser;
#[test]
fn vip_commands_parse_complete_listener_and_settings() {
let cli = super::super::Cli::try_parse_from([
"tsng",
"vip",
"port",
"add",
"lab",
"k8s",
"tcp/443",
"--backend",
"cp1:6443",
"--backend",
"cp2:7443",
]);
assert!(cli.is_ok(), "{cli:?}");
assert!(
super::super::Cli::try_parse_from([
"tsng",
"vip",
"port",
"set",
"lab",
"k8s",
"tcp/443",
"--port-check",
"off",
"--failures",
"3"
])
.is_ok()
);
assert!(parse_listener("tcp/0").is_err());
assert!(parse_listener("sctp/53").is_err());
}
}
crates/tsunagi/tests/dns_service.rs
+92
View File
@@ -345,3 +345,95 @@ fn opposing_dns_flags_are_rejected() {
assert!(!result.status.success()); assert!(!result.status.success());
assert!(String::from_utf8_lossy(&result.stderr).contains("cannot be used with")); assert!(String::from_utf8_lossy(&result.stderr).contains("cannot be used with"));
} }
#[test]
fn vip_cli_creates_multiple_ports_updates_dns_and_rejects_partial_invalid_changes() {
let agent = start("vip-test", 15367);
let created = agent.run(&["vip", "create", "vip-test", "api", "--json"]);
assert!(
created.status.success(),
"{}",
String::from_utf8_lossy(&created.stderr)
);
let vip: tsunagi::vip::VipStatus = serde_json::from_slice(&created.stdout).unwrap();
assert!(vip.config.listeners.is_empty());
let reply = wait_for_answer("127.0.0.1:15367".parse().unwrap(), "api.vip-test");
assert!(has_answer(&reply));
let socket = tsunagi::ipc::control_socket_path(&agent.dir.path().join("state"));
let runtime = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.unwrap();
let status = runtime
.block_on(tsunagi::ipc::request_status(socket))
.unwrap();
let backend = format!("{}:6443", status.endpoint_id);
let added = agent.run(&[
"vip",
"port",
"add",
"vip-test",
"api",
"tcp/443",
"--backend",
&backend,
]);
assert!(
added.status.success(),
"{}",
String::from_utf8_lossy(&added.stderr)
);
assert!(
agent
.run(&["vip", "port", "add", "vip-test", "api", "udp/53"])
.status
.success()
);
assert!(
!agent
.run(&["vip", "port", "add", "vip-test", "api", "tcp/443"])
.status
.success()
);
assert!(
agent
.run(&[
"vip",
"port",
"set",
"vip-test",
"api",
"tcp/443",
"--port-check",
"off"
])
.status
.success()
);
let shown = agent.run(&["vip", "show", "vip-test", "api", "--json"]);
assert!(shown.status.success());
let shown: tsunagi::vip::VipStatus = serde_json::from_slice(&shown.stdout).unwrap();
assert_eq!(shown.config.listeners.len(), 2);
assert!(
!shown
.config
.listeners
.iter()
.find(|l| l.protocol == tsunagi::vip::Protocol::Tcp)
.unwrap()
.port_check
);
assert_eq!(shown.config.listeners[0].backends[0].port, 6443);
let removed = agent.run(&["vip", "remove", "vip-test", "api"]);
assert!(removed.status.success());
let deadline = Instant::now() + Duration::from_secs(10);
loop {
if let Some(reply) = query("127.0.0.1:15367".parse().unwrap(), "api.vip-test", TYPE::A)
&& !has_answer(&reply)
{
break;
}
assert!(Instant::now() < deadline);
std::thread::sleep(Duration::from_millis(25));
}
}
docs/amnezia.md
+29 -14
View File
@@ -17,13 +17,16 @@ tsng protocols home awg
``` ```
Changes are local to this device, persist across restarts, and retire existing Changes are local to this device, persist across restarts, and retire existing
data links and end-to-end channels for that network before renegotiation. physical data links for that network before renegotiation. The single internal
WireGuard channel, its keys, session and counters survive.
Authenticated control sessions, membership, addresses and other networks stay Authenticated control sessions, membership, addresses and other networks stay
live. Removing every protocol (`none` in the CLI, all checkboxes off in the tray) live. Removing every protocol (`none` in the CLI, all checkboxes off in the tray)
leaves the network control-only. This is different from automatic carrier leaves the network control-only. Re-enabling a carrier resumes the existing
fallback, which preserves the selected end-to-end channel. internal channel, subject to ordinary WireGuard and application timeouts. The
choice applies to both incoming and outgoing local data links; a relay may use
a different transport on its next hop. Control and discovery are independent.
`awg` is Tsunagi AWG wire version 1. Its packet masking engine is adapted from `awg` is Tsunagi AWG wire version 2. Its packet masking engine is adapted from
[RunaRoo/AmneziaWG-RS](https://github.com/RunaRoo/AmneziaWG-RS), with the pinned [RunaRoo/AmneziaWG-RS](https://github.com/RunaRoo/AmneziaWG-RS), with the pinned
source and license recorded in source and license recorded in
[UPSTREAM.md](../crates/tsunagi-awg/UPSTREAM.md). It uses the existing boringtun [UPSTREAM.md](../crates/tsunagi-awg/UPSTREAM.md). It uses the existing boringtun
@@ -36,7 +39,7 @@ not implemented: this carrier connects authenticated Tsunagi members.
There is one independent UDP socket and cancellable packet-engine task per There is one independent UDP socket and cancellable packet-engine task per
`(network, peer)` pair. The smaller endpoint id initiates, following the same `(network, peer)` pair. The smaller endpoint id initiates, following the same
rule as other data carriers. Both sides must announce enabled AWG wire version rule as other data carriers. Both sides must announce enabled AWG wire version
1. An authenticated iroh control session is required throughout; AWG supplies 2. An authenticated iroh control session is required throughout; AWG supplies
no alternate discovery or membership mechanism. no alternate discovery or membership mechanism.
Every `open` generates new masking parameters, a header-protection key, an Every `open` generates new masking parameters, a header-protection key, an
@@ -100,25 +103,37 @@ candidates. IPv6-only paths, symmetric NAT and networks blocking arbitrary UDP
may fail AWG and fall back; there is no AWG-over-QUIC substitution or AWG TURN may fail AWG and fall back; there is no AWG-over-QUIC substitution or AWG TURN
relay hidden behind its name. relay hidden behind its name.
Data delivery checks go through the actual end-to-end encrypted tunnel, once Data delivery checks go through the specific authenticated physical carrier,
per second, independently of control-plane RTT. Requests and replies bind a once per second, independently of control-plane RTT or the shared internal
tunnel. A successful relay path cannot mask a broken direct carrier. Requests and replies bind a
fresh private marker/key to the sender's endpoint id. Unanswered requests expire fresh private marker/key to the sender's endpoint id. Unanswered requests expire
after three seconds. After a 20-second warmup, at least 15 completed observations after three seconds. After a 20-second warmup, at least 15 completed observations
and at least 60% loss in the last 20 observations retire a carrier. Probes are and at least 60% loss in the last 20 observations retire a carrier. Probes are
consumed locally after source/destination admission and never enter the TUN. consumed at the carrier boundary and never enter the TUN or user IP counters.
AWG additionally checks encrypted outer delivery throughout the pair lifetime. AWG additionally checks encrypted outer delivery throughout the pair lifetime.
Setup failure or persistent runtime loss advances to the next mutually supported Setup failure or persistent runtime loss advances to the next mutually supported
carrier in the initiator's order. Earlier expired cooldowns do not jump ahead of carrier in the initiator's order. Earlier expired cooldowns do not jump ahead of
untried carriers. Retry cooldowns remain bounded; reaching AWG again negotiates untried carriers. Retry cooldowns remain bounded; reaching AWG again negotiates
new parameters. Control sessions and the logical end-to-end tunnel survive a new parameters. Control sessions and the logical end-to-end tunnel survive a
carrier change. Status names the end-to-end protocol and the actual carrier carrier change. Status and the tray show the actual external transport or relay
separately, so `wg` over `awg` is a valid result. route and data readiness. There is no user-selectable internal cipher.
Control wire version 6 introduces the private transport message. All members The internal capability is `wireguard-e2e` v4; external `wg` and `awg` use v2,
must upgrade together; device identities, network ids, handshake transcript `wg-quic` v1 and `tcp-tls` v2. All members must upgrade together. Control ALPN
encoding, signed state and stored network configuration are unchanged. Data `tsunagi/ctrl/8`, discovery, membership and signed state are unchanged. The data
ALPN remains `tsunagi/data/5`; the relay envelope is unchanged. ALPN is `tsunagi/data/6`; the relay envelope format is unchanged. Local IPC is v23,
so agent, CLI and tray must match. Historical `wg-quic/wireguard.sqlite` is the
canonical internal key store. Missing keys are imported transactionally from
`wg` then `awg`, without replacing an existing canonical key or modifying a
source database. Corrupt key stores fail rather than silently regenerating keys.
Device identities, network ids, addresses and saved transport choices survive.
Ordinary `wg` uses the same independent pair engine with standard WireGuard
headers, no masking/junk, fresh hop keys and PSK, and a preferred local UDP port
(default 51820; additional pairs use dynamic ports). The whole relay envelope is
inside the hop encryption. It is separate from the internal WireGuard channel.
Neither UDP mode is a client for an externally configured WireGuard server.
## Verification ## Verification
docs/architecture.md
+13 -8
View File
@@ -11,14 +11,15 @@ messages. On top of it, this crate's protocol handles membership
authentication, participant announcements, capability exchange and — later — authentication, participant announcements, capability exchange and — later —
state synchronisation and delivery of IP-plugin data. state synchronisation and delivery of IP-plugin data.
**Data plane.** Separate plugins create IP connectivity. WireGuard is the first **Data plane.** One internal WireGuard plugin creates IP connectivity in userspace — see [wireguard.md](wireguard.md). Plugin
one and is implemented in userspace — see [wireguard.md](wireguard.md). Plugin
keys, configuration and lifecycle are separate from iroh identity and from the keys, configuration and lifecycle are separate from iroh identity and from the
network secret. The core moves an opaque, bounded payload and never parses it. network secret. The core moves an opaque, bounded payload and never parses it.
**The transport in between.** Plugins do not open connections. They are handed **The transport in between.** Plugins do not open connections. They are handed
a `PacketLink` — an authenticated, unreliable datagram channel to one peer for a `PacketLink` — an authenticated, unreliable datagram channel to one peer for
one protocol — and never learn how it is carried. the internal cipher — and never learns how it is carried. External `DataCarrier`
adapters own their reachability capabilities and physical link lifecycle; they
never receive plaintext overlay IP packets.
The separation between the two planes is **logical, not physical**. The control plane uses The separation between the two planes is **logical, not physical**. The control plane uses
iroh; the data plane can use iroh on its own ALPN and connection, or independent iroh; the data plane can use iroh on its own ALPN and connection, or independent
@@ -73,8 +74,10 @@ And in crates of their own:
| crate | responsibility | | crate | responsibility |
|---|---| |---|---|
| `tsunagi-awg` | ephemeral pairwise Amnezia masking, private negotiation and independently encrypted UDP carriers; reuses the end-to-end WireGuard engine | | `tsunagi-wireguard` | independent ordinary WG and masked UDP hop engines, private pair negotiation and local candidates |
| `tsunagi-wg-quic` | the `wg-quic` protocol: its keys, its announcement, its tunnels | | `tsunagi-tcp-tls` | TLS hop encryption and bounded reachability announcements; no inner IP cipher |
| `tsunagi-awg` | ephemeral pairwise Amnezia masking, private negotiation and independently encrypted UDP carriers; shares the independent UDP hop engine with `wg` |
| `tsunagi-wg-quic` | the single internal WireGuard cipher, keys and opaque announcement (historical crate name) |
| `tsunagi` | the command line agent, built as the `tsng` executable: the only place that owns a runtime, a logger and signals | | `tsunagi` | the command line agent, built as the `tsng` executable: the only place that owns a runtime, a logger and signals |
The core library is the package `tsunagi-core`; its library target is still called `tsunagi`, so code says `use tsunagi::…`. The core library is the package `tsunagi-core`; its library target is still called `tsunagi`, so code says `use tsunagi::…`.
@@ -85,7 +88,7 @@ they compare the *wire* version, which moves only when the bytes do, so two
peers on different releases keep working. peers on different releases keep working.
Abstractions exist only where something is really substituted or really needs Abstractions exist only where something is really substituted or really needs
isolating for tests: `NetworkDiscovery`, `IpPlugin`, `PacketTransport` / isolating for tests: `NetworkDiscovery`, one `IpPlugin`, external `DataCarrier`, `PacketTransport` /
`PacketLink` (so a protocol's carrier can change), `TunFactory` (which is what `PacketLink` (so a protocol's carrier can change), `TunFactory` (which is what
lets the whole data plane be tested without privileges), `InterfaceProvisioner` lets the whole data plane be tested without privileges), `InterfaceProvisioner`
and `DnsPublisher` (which are where the platforms differ). Everything else is a and `DnsPublisher` (which are where the platforms differ). Everything else is a
@@ -161,8 +164,10 @@ Values that cannot honestly be attributed to one network stay at the endpoint
level. A value iroh does not report is `None`. level. A value iroh does not report is `None`.
An iroh address is an address for iroh. It is not assumed usable by any other An iroh address is an address for iroh. It is not assumed usable by any other
protocol; a future WireGuard plugin gathers its own reachability data and ships protocol; each external carrier gathers its own reachability data and ships it through
it through the control plane as an opaque announcement. the control plane as an opaque announcement or private pair negotiation.
Local iroh candidates and introductions exclude the agent's own overlay, so
transport traffic cannot recursively tunnel through itself.
### Relays ### Relays
docs/exit-node.md
+30 -6
View File
@@ -43,12 +43,36 @@ be an exit node in one network and not in another.
`tsunagi-exit:<interface>:<range>` and are removed when the offer ends, the `tsunagi-exit:<interface>:<range>` and are removed when the offer ends, the
network stops, and when the agent exits. Only networks that offer have network stops, and when the agent exits. Only networks that offer have
rules, so the others' traffic is not forwarded. rules, so the others' traffic is not forwarded.
- **Client:** policy routing, with no change to the main table. Table `28787` - **Client:** policy routing, with no change to the main table. Table `29811`
holds `default dev <interface>`, and three rules sit after Tailscale's: holds `default dev <interface>`. Rules `5205 fwmark 0x40000000/0x40000000
`5280 uidrange <agent uid> lookup main` (the agent's own connections never lookup main` and `5206 uidrange <agent uid> lookup main` keep the agent's
go through the tunnel, which would be a loop), `5290 lookup main sockets and their replies outside the tunnel, before Tailscale's rules.
suppress_prefixlength 0` (LAN and other specific routes keep working) and `5290 lookup main suppress_prefixlength 0` preserves LAN and other specific
`5300 lookup 28787`. routes, and `5300 lookup 29811` directs other traffic through the exit.
The exit table is `29811` (`0x7473`).
- **Linux reverse-path filtering:** only the agent's dedicated UID is marked
in mangle `OUTPUT`. Its own bit, `0x40000000`, is saved to conntrack and
restored in `PREROUTING` before the host firewall checks reply routing.
Other mark bits, including Tailscale's `0xff0000` mask, are preserved. The
rules are tagged `tsunagi-exit-underlay:<interface>` and never accept packets
around the host firewall. IPv4 and IPv6 protection is installed before exit
routes; routes are removed before protection, including rollback on failure.
Locally owned TCP listeners and bound UDP sockets also protect the first
incoming packet, before any outgoing packet has created a connection mark.
Socket addresses and ports
come from local kernel socket tables and match an actual socket; remote
announcements never supply these firewall arguments.
The socket set is refreshed before private pair negotiation is sent and on
discovery ticks, so fresh AWG ports are included. An unchanged set causes no
firewall writes; a socket-refresh failure leaves active exit routes in place.
Linux needs both `iptables` and `ip6tables` when IPv6 is present.
- **Kernel strict `rp_filter`:** if an interface uses strict mode (`1`), the
kernel must include marks in reverse lookups: set
`net.ipv4.conf.all.src_valid_mark=1`, or enable it on those interfaces.
The agent checks this before installing exit routes and reports an actionable
error if marks would be ignored. It never disables `rp_filter` or changes
host sysctls. Netfilter strict reverse checks also need their mark-aware
form, such as `rpfilter --validmark`, which NixOS already uses.
- Packets for addresses outside every overlay range are sent to the exit node - Packets for addresses outside every overlay range are sent to the exit node
by the router; the exit node's replies are accepted from any source. A member by the router; the exit node's replies are accepted from any source. A member
that is not your exit node cannot do that. that is not your exit node cannot do that.
docs/protocol.md
+15 -13
View File
@@ -9,24 +9,26 @@ Two versions exist and are independent:
- **Identity scheme**, `tsunagi-network-id-v1`. Frozen. Changing it creates a - **Identity scheme**, `tsunagi-network-id-v1`. Frozen. Changing it creates a
different network space for the same name and secret. different network space for the same name and secret.
- **Control protocol**, ALPN `tsunagi/ctrl/7`, `PROTOCOL_VERSION = 7`. - **Control protocol**, ALPN `tsunagi/ctrl/8`, `PROTOCOL_VERSION = 8`.
Upgrading the crate or bumping the control protocol must never change an Upgrading the crate or bumping the control protocol must never change an
existing `NetworkId`. existing `NetworkId`.
Control version 7 adds signed offline-node retirement messages to version 6's Control version 8 adds signed, multi-writer VIP field snapshots to version 7's
private AWG transport negotiation and delivery checks. Data ALPN remains node retirements. Data ALPN is `tsunagi/data/6` for the single internal WireGuard channel. All members must upgrade
`tsunagi/data/5`. All members must upgrade together; network ids, handshake together. Network ids, derivation labels, handshake encoding and existing record
encoding and existing record signatures are unchanged. Local CLI/tray IPC is signatures are unchanged. Local CLI/tray IPC is version 23; restart the agent
version 21, so restart the agent with matching clients. SQLite schema 9 adds with matching clients. SQLite schema 10 adds `vip_fields`, preserving identity,
retirement storage while preserving existing identities, settings and records. network configuration, signed node records and retirement proofs.
See [sync-model.md](sync-model.md#offline-node-cleanup) and [amnezia.md](amnezia.md). See [virtual services](vip.md), [state sync](sync-model.md) and [AWG](amnezia.md).
## Canonical encoding ## Canonical encoding
Everything that is hashed or MAC'd uses length-prefixed fields, written here as Network identity derivation and the membership transcript use length-prefixed
`LP(x) = u32_be(len(x)) || x`. String concatenation is never used, so no two fields, written here as `LP(x) = u32_be(len(x)) || x`. String concatenation is
different field splits can produce the same bytes. never used, so different field splits cannot produce the same bytes. VIP field
signatures use their separate domain and canonical postcard encoding described
in [virtual services](vip.md#shared-state-and-addresses).
## Network space identity ## Network space identity
@@ -143,7 +145,7 @@ event does not report a network id.
## The data plane protocol ## The data plane protocol
IP plugin packets never travel on a control connection. They use their own IP plugin packets never travel on a control connection. They use their own
ALPN, `tsunagi/data/5`, on their own iroh connection: ALPN, `tsunagi/data/6`, on their own iroh connection:
```text ```text
initiator -> responder : (the same membership handshake as above) initiator -> responder : (the same membership handshake as above)
@@ -190,7 +192,7 @@ and version 5 added the end-to-end protocol to it:
| source peer | 32 | original endpoint public key | | source peer | 32 | original endpoint public key |
| destination peer | 32 | final endpoint public key | | destination peer | 32 | final endpoint public key |
| flow id | 8 | big endian opaque stable hash supplied by the plugin | | flow id | 8 | big endian opaque stable hash supplied by the plugin |
| protocol | 8 | big endian FNV-1a of the end-to-end protocol id (`wg`, `tcp-tls`, ...) | | protocol | 8 | big endian FNV-1a of the internal protocol id (`wireguard-e2e`) |
| payload | remaining | end-to-end encrypted plugin bytes | | payload | remaining | end-to-end encrypted plugin bytes |
The fixed header is 82 bytes. Zero/oversized hop limits, unknown versions, The fixed header is 82 bytes. Zero/oversized hop limits, unknown versions,
docs/routing.md
+13 -11
View File
@@ -16,8 +16,8 @@ the default 1280-byte interface MTU also works on small QUIC paths.
Each authenticated control session advertises that peer's own direct data Each authenticated control session advertises that peer's own direct data
links. There is one table for each network, whatever protocols run over it: a links. There is one table for each network, whatever protocols run over it: a
link is a way to reach a neighbour, and a protocol is an end-to-end tunnel to a link is a way to reach a neighbour, while the single internal protocol is an
peer. The envelope names the protocol, so the two are independent. A neighbour end-to-end tunnel to a peer. The envelope names the protocol, so the two are independent. A neighbour
reached over several kinds of transport (UDP, TCP, QUIC datagrams) is sent to reached over several kinds of transport (UDP, TCP, QUIC datagrams) is sent to
over the best open one, in the order the protocols were configured, and over over the best open one, in the order the protocols were configured, and over
the next if that one fails or cannot carry the frame. Every authenticated the next if that one fails or cannot carry the frame. Every authenticated
@@ -28,14 +28,16 @@ Link arrival, closure and changed announcements rebuild routes immediately.
Unchanged announcements refresh their age without republishing the forwarding Unchanged announcements refresh their age without republishing the forwarding
snapshot. snapshot.
Which protocol a pair of members runs their tunnel in is settled without Every pair uses one internal WireGuard channel, matched by its fixed opaque
asking: the member with the lower endpoint id decides, by the order it capability and independent of the selected external protocols. The tunnel is
announces its protocols in, among those both have enabled at the same wire created when a member and its key are known; its logical link survives temporary
version. Both ends read the same answer from what they already hold. The loss of all physical paths. Carrier selection changes, automatic fallback and
tunnel is created as soon as the peer is reachable by any path, and it does not shorter routes keep the same keys, interface and counters. The lower endpoint
change when a direct link comes or goes underneath it: as soon as a direct id still chooses the mutually enabled external carrier in its preference order.
link comes up it is shorter than the path through others, and traffic moves to A relayed AWG first hop can lead to a QUIC next hop: the user's choice governs
it with nothing to redo. The user sees only a lower round-trip time. this device's adjacent links, while the inner ciphertext stays unchanged.
An existing TCP connection continues after transient loss if its own timeout
has not expired; a sufficiently long outage can still terminate it.
Breadth-first search computes the shortest directed paths and every equal-cost Breadth-first search computes the shortest directed paths and every equal-cost
first hop in sorted order. An actual direct link always wins. The first row is first hop in sorted order. An actual direct link always wins. The first row is
@@ -124,5 +126,5 @@ counts sends without storing frames. Results are CPU forwarding cost, **not**
end-to-end network latency or VPN throughput; encryption, fragmentation, end-to-end network latency or VPN throughput; encryption, fragmentation,
sockets, congestion and scheduling contribute separately. sockets, congestion and scheduling contribute separately.
Wire compatibility: control ALPN `tsunagi/ctrl/7`, data ALPN `tsunagi/data/5`. Wire compatibility: control ALPN `tsunagi/ctrl/8`, data ALPN `tsunagi/data/6`.
Upgrade every participant together; saved identities and network state persist. Upgrade every participant together; saved identities and network state persist.
Loaded 100 of 190 files, more files were not shown because too many files have changed in this diff. Show more