ab/tsunagi
1
0 Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Compare commits

base: ab/tsunagi:v0.1.0-rc.10
ab/tsunagi:master ab/tsunagi:worktree-relay-unify ab/tsunagi:worktree-readme-rewrite
ab/tsunagi:v0.1.0-rc.37 ab/tsunagi:v0.1.0-rc.36 ab/tsunagi:v0.1.0-rc.35 ab/tsunagi:v0.1.0-rc.34 ab/tsunagi:v0.1.0-rc.33 ab/tsunagi:v0.1.0-rc.32 ab/tsunagi:v0.1.0-rc.31 ab/tsunagi:v0.1.0-rc.30 ab/tsunagi:v0.1.0-rc.29 ab/tsunagi:v0.1.0-rc.28 ab/tsunagi:v0.1.0-rc.27 ab/tsunagi:v0.1.0-rc.26 ab/tsunagi:v0.1.0-rc.25 ab/tsunagi:v0.1.0-rc.24 ab/tsunagi:v0.1.0-rc.23 ab/tsunagi:v0.1.0-rc.22 ab/tsunagi:v0.1.0-rc.21 ab/tsunagi:v0.1.0-rc.20 ab/tsunagi:v0.1.0-rc.19 ab/tsunagi:v0.1.0-rc.18 ab/tsunagi:v0.1.0-rc.17 ab/tsunagi:v0.1.0-rc.16 ab/tsunagi:v0.1.0-rc.15 ab/tsunagi:v0.1.0-rc.14 ab/tsunagi:v0.1.0-rc.13 ab/tsunagi:v0.1.0-rc.12 ab/tsunagi:v0.1.0-rc.11 ab/tsunagi:v0.1.0-rc.10 ab/tsunagi:v0.1.0-rc.9 ab/tsunagi:v0.1.0-rc.8 ab/tsunagi:v0.1.0-rc.7 ab/tsunagi:v0.1.0-rc.6 ab/tsunagi:v0.1.0-rc.5 ab/tsunagi:v0.1.0-rc.3 ab/tsunagi:v0.1.0-rc.2 ab/tsunagi:v0.1.0-rc.1
...
compare: ab/tsunagi:23af1a2d9d1ef7899359ede127552ee6a8b98e1a
ab/tsunagi:master ab/tsunagi:worktree-relay-unify ab/tsunagi:worktree-readme-rewrite
ab/tsunagi:v0.1.0-rc.37 ab/tsunagi:v0.1.0-rc.36 ab/tsunagi:v0.1.0-rc.35 ab/tsunagi:v0.1.0-rc.34 ab/tsunagi:v0.1.0-rc.33 ab/tsunagi:v0.1.0-rc.32 ab/tsunagi:v0.1.0-rc.31 ab/tsunagi:v0.1.0-rc.30 ab/tsunagi:v0.1.0-rc.29 ab/tsunagi:v0.1.0-rc.28 ab/tsunagi:v0.1.0-rc.27 ab/tsunagi:v0.1.0-rc.26 ab/tsunagi:v0.1.0-rc.25 ab/tsunagi:v0.1.0-rc.24 ab/tsunagi:v0.1.0-rc.23 ab/tsunagi:v0.1.0-rc.22 ab/tsunagi:v0.1.0-rc.21 ab/tsunagi:v0.1.0-rc.20 ab/tsunagi:v0.1.0-rc.19 ab/tsunagi:v0.1.0-rc.18 ab/tsunagi:v0.1.0-rc.17 ab/tsunagi:v0.1.0-rc.16 ab/tsunagi:v0.1.0-rc.15 ab/tsunagi:v0.1.0-rc.14 ab/tsunagi:v0.1.0-rc.13 ab/tsunagi:v0.1.0-rc.12 ab/tsunagi:v0.1.0-rc.11 ab/tsunagi:v0.1.0-rc.10 ab/tsunagi:v0.1.0-rc.9 ab/tsunagi:v0.1.0-rc.8 ab/tsunagi:v0.1.0-rc.7 ab/tsunagi:v0.1.0-rc.6 ab/tsunagi:v0.1.0-rc.5 ab/tsunagi:v0.1.0-rc.3 ab/tsunagi:v0.1.0-rc.2 ab/tsunagi:v0.1.0-rc.1
46 Commits
v0.1.0-rc.10 ... 23af1a2d9d
Author SHA1 Message Date
Aleksandr Bogomiakov 23af1a2d9d Add shared TCP and UDP virtual services 
Provide named VIPs with independent protocol/port listeners and node/port
backends so applications can use one DNS endpoint without a separate load
balancer. Every authenticated member may create, edit or delete any service.
Client agents select the eligible backend with the lowest measured encrypted
data-plane RTT, including relayed paths, and pin existing flows. UDP recovers
on another backend after failure; established TCP is never replayed or moved.
An unanswered empty TCP SYN may retry another backend before establishment.

Keep signed multi-writer field registers in state/vip.rs. Independent backend
edits merge, same-field conflicts use Lamport/author/canonical ordering, and
listener generations prevent old backend entries returning on port recreation.
Permanent service tombstones protect replay while allowing obsolete fields to
be compacted. Active-service and deletion-history limits are separate. Commit
complete snapshots and the author's durable counter atomically before local
routing changes or publication. Restore counters from newer genuine author
updates learned from replicas after an old backup is restored.

Implement bounded client-side address/port NAT at overlay ingress and reverse
translation only after authenticated source/destination admission. Reserve OS
source ports to avoid direct-flow collisions and quarantine retired ports for
late replies. Handle a backend on the client itself, scoped fragment assembly,
checksums, re-fragmentation and quoted ICMP/PMTU headers. Plugins and encrypted
transit retain their existing contracts. Data-path probes measure RTT; bounded
fair TCP connect checks are enabled by default and configurable per listener.
Arbitrary UDP application readiness remains unknown; path checks and explicit
port errors drive eligibility without protocol-specific probe payloads.

Expose the shared draft API through local IPC, simple vip/listener/backend CLI
commands, JSON, editor-based atomic updates and watch output. Add a VIP button
on each network tile and a separate tray editor. Edits and removals stay local
until Save; stale revisions retain the draft, and closing a dirty window offers
explicit discard. Report local routing choices, health reasons, RTT age, active
and draining bindings, IP counters, rates, drops and UDP failovers. Extend DNS
with service names; keep numeric addresses secondary and preserve existing
bindings while VIP addresses or listeners change.

This changes control wire ALPN/version to tsunagi/ctrl/8 and local IPC to 22;
all members and local clients must upgrade together. Data ALPN remains 5.
SQLite schema 10 migrates existing mandatory state without changing identity,
secrets, network ids, derivation labels, handshake encoding or node records.
Crate versions and release tags are unchanged. No installed agent or tray was
replaced. Commands, limits and operational semantics are in docs/vip.md.

Checks on Linux in nix develop:
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets: 559 passed, 3 ignored
- git diff --cached --check

The full suite includes real encrypted TCP/UDP VIP traffic, backend-stop UDP
failover, real loopback TCP check recovery, source admission, signed merges,
stale/deleted state, SQLite migration and rollback, restart/author recovery,
network isolation, actual CLI/DNS processes, and headless text-entry/Save-button
interaction. Discovery and memory TUNs are the existing test substitutions;
authentication, transport, encryption and SQLite remain real.

Initial checks outside Nix failed for missing GTK pkg-config metadata, and an
initial CLI test inherited TSUNAGI_CONTROL_SOCKET. Final checks use the Nix
environment with that variable unset so tests address their temporary sockets.
Root/internet-dependent tests remain ignored. No live host TUN, native desktop
session, macOS or Windows validation was performed.
 2026-10-21 15:40:45 +01:00
Aleksandr Bogomiakov 94318e7c16 Keep tray device metrics on one line 
The device table sized handshake/RTT and packet columns from fixed samples.
Its own header exceeded the handshake sample, and six-digit counters wrapped
at the slash, making device and total rows taller and misaligning the plots.
Measure actual headers, displayed values and totals before drawing, retain
the widest values across frames, and disable wrapping inside table cells.
When a column grows, request a second egui layout pass so the grid's cached
positions settle before larger text is painted over the next column.
Keep the plot as the stretch column and horizontal scrolling for narrow
windows. This changes only tray presentation.

A headless rendering regression checks the actual text shapes for the
header, handshake age/control RTT, per-device packet counts and totals as
counts grow past five digits, reach billions, shrink again, and the table
gets a narrower container. It checks one-line text and non-overlap with the
next column; it failed on the original wrapped header. Existing resize and
long-relay tests remain in place.

Checks on Linux in nix develop:
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets, with inherited
  TSUNAGI_CONTROL_SOCKET unset so tests use their temporary sockets
- git diff --cached --check

The suite's root/internet-dependent tests remain ignored. No native GUI
session or checks on macOS/Windows were run; rendering was checked through
egui's headless output. No installed tray binary was replaced.
 2026-10-14 23:37:38 +01:00
Aleksandr Bogomiakov 2b0c938487 Bound iroh path retries to prevent multi-connection OOM 
A long-running rc.33 daemon reached OOM with roughly 22 GiB resident and
24 GiB swapped. Live inspection found 40 GiB and 10/20 GiB anonymous
container mappings; the active worker stack and tracing metadata identified
iroh's scheduled path-open retry loop. In iroh 1.2.0 every failed connection
requeues the same path, while each retry visits every connection again.
Separate control and data connections multiply the queue geometrically.

Vendor the published iroh 1.2.0 source and select it with a workspace Cargo
patch. Deduplicate retry admission and cap distinct pending paths at 64,
evicting the oldest before inserting a new candidate. Keep the current and
next retry batches bounded without combining control/data connections or
changing wire protocols, identity derivation, authentication or state.
Document source provenance, the live evidence and removal conditions.
Upstream PR 4551 merged on October 6, but inspection of the official 1.3.0
crate still shows the unconditional enqueue, so that upgrade alone is not
a fix. No version bump, release tag or installed-service change.

The default workspace suite includes the exact vendored admission module.
Three regressions failed with the original enqueue and pass with the fix:
repeated failures across connections, duplicate admission at capacity, and
10,000 distinct candidates with bounded length/capacity and FIFO eviction.
Transport integration tests continue using real endpoints and encryption.

Checks on Linux in nix develop:
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets, with only the inherited
  TSUNAGI_CONTROL_SOCKET removed from the test environment
- git diff --cached --check

Initial full-suite runs failed six DNS readiness checks because the inherited
system socket path overrode their temporary state directories. Clearing that
environment variable resolves those failures without changing application code.
Root/network-dependent ignored tests and other operating systems were not run.
 2026-10-14 23:28:53 +01:00
Ultradesu 4e235cbac6 Keep device data paths on one line 
The tray used a fixed sample for the data-path column, so longer relay names wrapped and made device rows uneven. Measure each rendered path and retain the widest value, then render path labels without wrapping. Replace direct/via words with embedded Phosphor route icons while preserving transport, relay name, hop count and explanatory tooltips. The app already ships the font; keep its setup in one function and verify registration alongside a long-relay-width regression test. No wire or stored-state changes. Checks: cargo fmt --all -- --check; cargo clippy --locked --workspace --all-targets -- -D warnings; cargo test --locked --workspace --all-targets.
 2026-10-14 13:50:05 +01:00
Aleksandr Bogomiakov 2631eb6975 Protect Linux exit-node connections and tidy the device table 
Using an exit node could destroy direct peer connections: the agent's UID
bypass protected outbound routing, but reverse-path checks look up replies
without that UID and selected the overlay default. A real-host A/B check
confirmed that relaxing only strict reverse-path filtering restored direct
connections and exit HTTP traffic; the diagnostic host changes were removed.

Keep strict filtering and protect the Linux underlay with a dedicated masked
mark bit, 0x40000000. Mark the dedicated service UID in OUTPUT, save only that
bit in conntrack and restore it before PREROUTING reverse-path checks. Install
IPv4/IPv6 mark and UID main-table bypass rules before Tailscale's priorities;
leave application-specific and exit-default routing after them. Preserve
foreign mark bits, main-table routes and foreign firewall/policy rules.

Protect the first inbound TCP/UDP packet using bounded, locally owned socket
addresses/ports from /proc and an actual socket match. Refresh socket protection
before sending private transport negotiations and on discovery ticks, without
firewall writes for an unchanged socket set. Remote announcements never become
OS arguments. A failed refresh leaves active exit routing in place. Install
protection before routes, remove routes before protection, roll back partial
activation, and remove only exact interface tags, including legacy UID rules.

Reject strict kernel rp_filter without src_valid_mark with an actionable error;
never weaken filtering or change host sysctls. Document mark-aware Netfilter
requirements and provide the trusted ip6tables path in the NixOS module.
Other platforms retain their existing exit implementation. Peer wire formats,
identity derivation, local IPC and persistent storage are unchanged.

Make the tray devices grid fill and shrink with the window, constrain the first
cell and give remaining width to traffic plots. Keep displayed end-to-end
protocol and its short encryption tooltip consistent, independent of carrier
protocol. Move handshake/session age into its own column beside the existing
iroh control RTT. Use stable per-device hover identities and concise tooltips;
missing protocols/measurements remain explicit rather than invented.

Validation on Linux through nix develop:
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets: 529 passed, 3 ignored;
  inherited TSUNAGI_CONTROL_SOCKET and TSUNAGI_CONTROL_GROUP were unset
- explicit privileged kernel regression: 1 passed, using disposable namespaces,
  real TCP/UDP IPv4/IPv6, strict kernel and Netfilter RPF, first inbound packets,
  dynamic UDP binding, application exit routing/IPv6 blocking, repeated apply
  and exact cleanup with foreign Tailscale-priority rules retained
- headless egui grid/scroll-area rendering at multiple sizes and control RTT,
  mark masks/order, passive socket parsing, partial failure and cleanup tests
- git diff --check

Earlier iterations caught rtnetlink family API and nonexhaustive initializer
errors, Clippy warnings, egui's implicit minimum column width, and kernel-test
fixture issues (wildcard echo source, blocked IPv6 neighbour discovery and a
closed child stdout pipe). Corrected before final checks; the stdout-only test
fix was checked with fmt, Clippy and the explicit kernel regression afterward.
Native tray rendering and macOS/Windows checks were not run. The running host
agent was not replaced or restarted. Requested release tag: v0.1.0-rc.32.
 2026-10-14 10:40:30 +01:00
Aleksandr Bogomiakov 90ef5cefec Retire offline node claims with equal network-wide cleanup rights 
Offline devices reserved names and IPv4 addresses indefinitely, leaving stale
members in the device history. Give every authenticated network participant
the same right to retire another device's observed signed claim without an
owner, quorum, permanent identity ban or secret change.

Add a separately signed retirement containing the target's original signed
claim as proof. Verify both signatures and network scopes, retain the strongest
proved cutoff per target with deterministic ties, and suppress claim versions
at or below it in membership, hostname ownership and IPv4 allocation. Keep the
old signed records and range agreement, and retain markers after a newer claim
returns so old replicas cannot resurrect obsolete reservations. A returning
device signs a newer claim, allocates against current holders and resets its
previous hostname priority rather than taking a reused name with an old time.

Commit local retirement and the requester's author counter atomically before
announcing it; receivers persist proofs before applying them. Bound history to
1024 targets by default, reject additional targets without evicting protection,
and treat corrupt or over-limit mandatory storage as an explicit error. Repeat
bounded marker batches with a per-session rotating cursor so queue pressure
cannot indefinitely starve later markers. Exclude retired claims and cached
hints from history-derived discovery without preventing a fresh authenticated
device from returning.

Expose Agent::forget_node through the shared local control API. CLI command
tsng network forget-node accepts exact network names, full ids or unique id
prefixes, and exact node hostnames, ids, prefixes or overlay IPv4 addresses.
Ambiguous selections change nothing. Recheck the observed version and refuse
self, online nodes, the selected exit node and absent or changed claims.
Add Forget with an identity confirmation to offline tray device rows. Widen
Show devices to 1440x640 with an 800x400 minimum to fit its contents.

Control ALPN/version advances to tsunagi/ctrl/7 and local IPC to 21; upgrade
all peers and restart the agent with matching CLI/tray clients. Data ALPN 5,
network identity derivation, handshake encoding and existing record signing
bytes remain unchanged. SQLite schema 9 adds retirement storage while retaining
identity, secrets, settings and signed claims. Document behavior and limits.

Validation on Linux through nix develop:
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets: 519 passed, 2 existing ignored
  tests; inherited TSUNAGI_CONTROL_SOCKET and TSUNAGI_CONTROL_GROUP were unset
- git diff --check and the built CLI forget-node help

Tests use real iroh authentication and SQLite. Cover restart, stale-replica
recovery, equal rights and network isolation, hostname reuse and return, twelve
marker batches through a two-message queue, malformed signatures and proofs,
invented future cutoffs, deterministic merge, capacity, migration and injected
counter-write rollback, selectors and cleanup through two real CLI processes.
Earlier iterations caught a private test API, a collapsible tray conditional
and a crash-based CLI test waiting for QUIC's idle timeout; corrected before
the final successful suite. Manual tray rendering, macOS/Windows, privileged
TUNs and public-network traversal were not run. The requested RC tag is
v0.1.0-rc.31; no push was requested.
 2026-10-12 18:04:29 +01:00
Aleksandr Bogomiakov 13bdcaf98d Render protocol priority buttons with Phosphor icons 
The tray protocol selection used ordinary Unicode up/down arrows for its
priority buttons. The default UI fonts lack these glyphs, so both buttons
rendered missing-glyph boxes. Use the regular Phosphor ARROW_UP and ARROW_DOWN
icons already installed in the application's proportional font fallback.

Validation on Linux through nix develop:
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets, with inherited control socket
  and group overrides removed: 509 passed, 2 existing ignored tests

No new tests were added for the two glyph substitutions. Manual tray rendering
and macOS/Windows checks were not performed. No release tag was requested for
this fix.
 2026-10-12 17:19:53 +01:00
Aleksandr Bogomiakov d941cdac7a Change network protocol selections at runtime 
Protocol selection previously belonged to tsng up and therefore to the
service command line. A blocked or degraded carrier could only be removed
by editing the service and restarting the whole agent. Make the ordered
selection a local per-network setting instead.

Add Agent::set_protocols and a local control request shared by CLI and tray.
Keep tsng protocols without arguments as the protocol catalog; naming a
network shows its selection and supplying a comma-separated list replaces
it. Accept exact names, full ids and unique prefixes using shared selector
rules; ambiguous selectors and invalid lists change nothing. An explicit
none selects control-only operation. Offline edits take the directory lock
and apply on the next start; live edits never write behind the daemon.

Register every built-in plugin and transport in the CLI. Treat up --protocol
as ordered initial defaults for networks without an override, allowing a
protocol absent from the unit to be enabled later. Retain the targeting of
unqualified startup options and allow explicitly scoped options to configure
initially disabled protocols. Persist NULL for inherited preferences and an
empty selection distinctly in SQLite schema 8; migration preserves existing
identities, secrets, signed records and counters.

Serialize changes with network lifecycle operations and commit preferences
before applying or announcing them. Retire only the affected network's data
links and E2E channels, cancel pending opens, and reject obsolete completion
results by attempt id. Clear peer data channels when authenticated protocol
capabilities change while retaining control sessions and topology. Replay
allocations and peer identities after asynchronously prepared plugins become
ready. A control-only start defers creation of the agent's single interface
until data is enabled; plugins receive a sink that can use it later. Automatic
carrier fallback continues to preserve E2E state; explicit policy changes
renegotiate it. Identity derivation and peer wire formats are unchanged.

Add a collapsible protocols row to each tray network tile, with a catalog
provided by the daemon, descriptions, immediately applied checkboxes and
preference-order buttons. Keep editing disabled until the worker refreshes
status so a second click cannot overwrite an edit using a stale snapshot.
Local IPC wire version is 20 for the new request and report fields; upgrade
and restart agent, CLI and tray together. Crate release versions remain unchanged; the requested RC tag is v0.1.0-rc.30.

Validation on Linux through nix develop:
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets: 509 passed, 2 existing ignored
  tests, with inherited TSUNAGI_CONTROL_SOCKET and TSUNAGI_CONTROL_GROUP removed

Tests use real iroh membership, SQLite, UDP, AWG and WireGuard encryption,
QUIC carriers and memory TUNs. Cover either endpoint-id role, full-size packet
delivery after switching, an unaffected second active network, saved selection
after restart, empty/invalid selections, first enablement without restart,
rapid changes racing old opens, live and offline CLI changes, startup-absent
protocols and preservation of signed state through migration. Earlier compile
and focused-test iterations caught initializer mistakes, a short test secret,
asynchronous plugin allocation readiness, and an inherited system socket
path; corrected before the successful final checks. Plain cargo was unavailable
in the host shell, so validation used the Nix toolchain. Manual tray interaction,
macOS/Windows, privileged host TUNs and public-network traversal were not run.
 2026-10-12 16:19:18 +01:00
Aleksandr Bogomiakov e701986a39 Add ephemeral pairwise Amnezia transport and runtime fallback 
Plain WG and QUIC can be blocked or severely degraded while the iroh control
connection still works. Add an awg carrier with one independent UDP socket,
WireGuard engine and randomized masking profile per network/peer attempt.
The smaller endpoint id negotiates fresh outer keys, PSK, header protection,
header ranges, padding and signature templates through private authenticated
control messages. Restart or fallback reentry generates a new profile;
ordinary rekeying preserves the current profile. No pair parameters enter
signed state, capabilities, persistent storage or public message events.
Private payloads and serialized control buffers redact and zeroize themselves.

Adapt the bounded packet masking engine from RunaRoo/AmneziaWG-RS at commit
51579a7d6e8882d1d8d6d00822f09e18a9423b73, retaining its Apache license and
provenance. Declare WTFPL AND Apache-2.0 in package metadata to cover both
new glue and the adapted source. Reuse boringtun instead of upstream's Node/global executors and
interface machinery. This is Tsunagi AWG wire version 1, with standard WG
headers restored before authentication; external Amnezia server compatibility
is not implemented. DNS/SIP signature generators vary fields instead of
replaying captures. Each attempt closes independently, bounds allocation,
queues, candidates and fragment assemblies, and supports the full 64 KiB
carrier contract using UDP packets at most 1200 bytes. Yield between fragment
batches so large messages do not starve local socket readers.

Keep carrier encryption separate from end-to-end tunnels so multihop relays
still forward opaque ciphertext. The system owns the sole interface, signed
addresses and admission. Plugins receive neither network secrets nor paths.
Expose a bounded opaque control sender at the transport boundary; core control
messages remain outside protocol crates. Optional STUN on the pair's actual
socket adds IPv4 public candidates in CLI direct/relay mode, with a finite
query deadline and an off/alternate-server option. Local mode remains offline.
Symmetric NAT, IPv6-only paths and arbitrary UDP blocking can still require
fallback; no AWG-over-QUIC or hidden AWG relay is introduced.

Default carrier order is wg, awg, wg-quic, tcp-tls. Advance the fallback cycle
after setup failures and sustained runtime loss, even after earlier cooldowns
expire. Authenticated delivery probes traverse the actual encrypted tunnel,
fit the interface MTU and never enter the TUN. Retire after a 20-second warmup
and at least 60% loss over at least 15 completed recent observations. Preserve
logical tunnels and control sessions while explicitly closing physical links,
including TCP reader/writer tasks. Retain the newest private probe settings
across independently queued control negotiation and inbound installation.
CLI/tray reports keep end-to-end protocol and actual carrier separate.

Control ALPN/version becomes tsunagi/ctrl/6 for the new private message;
upgrade all members together. Data ALPN/envelopes remain version 5. Network
identity derivation, transcript encoding, device identities, signed records,
saved network configuration and release versions are unchanged.

Validation on Linux through nix develop:
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets, with the inherited
  TSUNAGI_CONTROL_SOCKET override removed: 504 passed, 2 pre-existing ignored

Tests use real iroh membership, SQLite, UDP sockets and WireGuard encryption
with memory TUNs. Cover three independent pairs, full-size IP and carrier
frames, restart freshness/socket release, 80% carrier datagram loss, fallback
to real iroh, fresh retries with working delivery checks, unequal small MTUs,
malformed negotiations/profiles, role and resource bounds, probe forgery and
STUN parsing. Earlier compile/lint iterations and focused tests caught RNG
API/initializer mistakes, a retry assertion racing path installation, and a
large-fragment burst overflowing loopback buffers; corrected before these
successful checks. Normalized upstream license line endings for diff checks.
Public NAT/STUN connectivity, actual DPI effectiveness, WAN performance,
per-pair memory measurement, macOS/Windows, privileged TUN operations and
manual tray interaction were not exercised.
 2026-10-11 13:52:10 +01:00
Aleksandr Bogomiakov ed28e86748 Derive default network ranges and switch overlapping networks 
Every first network previously proposed 10.13.37.0/24. Independently created
networks therefore collided when a device joined more than one of them.
Use the existing stable NetworkId-derived /24 in 10/8 for every new network,
independent of activation order or peer availability. Legacy and explicit
signed ranges outrank the derived default; remaining range disagreements
still use author-id ordering. The derivation, identity labels, handshake and
signed record encoding stay unchanged. There is no creator, timestamp or vote.

Reserve ranges before starting network actors and serialise start, stop and
switching through the agent lifecycle lock. Refuse equal or nested overlaps
without changing a network's subnet locally. A late authenticated range that
collides suspends only its network before updating plugins or the interface.
Coalesced watch notifications let the agent clean up outside the network actor
and reconsider suspended networks when ranges change.

An explicit switch stops every blocker before activating the chosen network.
Suspended networks retain their startup request and resume whenever their
ranges become free, including switching from a wide range to a narrow one.
A manual stop cancels that request even for an already suspended network.
Persist preference order and enabling together in an SQLite transaction so
the user's selection survives restart; no database schema migration is needed.

CLI status and network listing show blocker names, ids, ranges and the start
command. Network start switches explicitly and reports suspended networks;
joining an overlapping network reports it as configured but inactive. The
tray adds conflict details, an amber icon, Switch to this network and Keep
stopped actions. Local IPC wire version is 19 for the extended reports, so
restart the agent and tray together. Release versions and peer wire formats
are unchanged. Update the user documentation and compatibility tests.

Validation on Linux through nix develop:
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets with TSUNAGI_CONTROL_SOCKET
  removed from the test environment: 486 passed, 2 pre-existing ignored tests
- final focused range_conflicts integration suite after making discovery
  fixtures deterministic: 6 passed

Earlier validation attempts caught outdated diagnostic assertions, a new
test helper scope error, and the inherited system control socket override.
Those were corrected before the successful runs. Plain cargo was unavailable
in the host shell, so validation used the Nix development toolchain. macOS,
Windows, privileged host TUN operations and manual tray interaction were not
exercised.
 2026-10-11 11:58:27 +01:00
Aleksandr Bogomiakov 781e359775 Make tunnel and transport layers explicit in status 
The tray called the end-to-end tunnel proto and the current route link, then put the tunnel handshake age beside the route. A valid wg tunnel over a tcp-tls transport consequently looked like contradictory state. The terminal status used similarly terse wording.\n\nThe tray now separates end-to-end from data path, renders direct paths as direct over TRANSPORT, keeps handshake or uptime beside the tunnel, and explains both layers in hover text. Terminal status uses the same end-to-end tunnel, data path, transport detail, and control path vocabulary. Routing, protocol selection, IPC, and wire behavior are unchanged.\n\nPresentation tests cover wg over tcp-tls, tcp-tls uptime, relayed and missing paths, a missing handshake, and offline devices. The WireGuard broadcast checksum test uses the current slice-array iterator spelling required by Clippy; its behavior is unchanged.\n\nChecks: cargo fmt --all -- --check; cargo clippy --locked --workspace --all-targets -- -D warnings; cargo test --locked --workspace --all-targets with TSUNAGI_CONTROL_SOCKET removed from the test environment so integration agents use their temporary control sockets. All passed on Linux.
 2026-10-08 16:03:20 +01:00
Aleksandr Bogomiakov d5e02ec256 fixed gitignore 2026-10-08 15:35:38 +01:00
Aleksandr BogomiakovandClaude 97377e3703 Relay across any mix of links, one online answer for every client, first claim owns a hostname 
A tunnel no longer depends on the kind of link under it. The relay hub keeps
one forwarding table per network instead of one per protocol, the envelope
names the end-to-end protocol, and a neighbour reached over several kinds of
transport is sent to over the best open one. A member in the middle needs no
protocol in common with either end. Which protocol a pair runs is settled by
the lower endpoint id's announced order, so both ends agree without asking.
A direct link that comes up later is shorter than the path through others, so
traffic moves to it with the same logical link.

The agent now answers "is it online" once: a live control session or any data
path. Members and peers report how they are reached (direct over a named
transport, relayed through a named member, or no data path yet), and `tsng
status` and the tray both read that instead of a session list and a WireGuard
handshake respectively.

A hostname belongs to whoever claimed it first. A claim carries the time its
author first claimed the name, signed with the record, and replicas order by
(time, id) so every member names the same holder. A later claimant keeps
working by address, is not found by the name, and is told so in its status,
`tsng status`, the tray and a HostnameTaken event.

Wire change: control and data ALPN 5, control PROTOCOL_VERSION 5, envelope
version 2, signing domain v3, state schema 7, local control protocol 18.
Upgrade every member together.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-10-05 11:39:33 +01:00
Aleksandr BogomiakovandClaude c184ef0914 Rewrite README as a short project front page; add a Nix flake 
The README is now a landing page: the idea (a mesh VPN with no server, joined
with a network name and a secret), install and platform tables, exit nodes,
and a short build section. Implementation detail stays in docs/.

The flake builds tsng and the tray from source and ships a NixOS module
(services.tsunagi). Dist install notes now show the real `join -n -s` syntax,
and two links that pointed into the old README are redirected.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-10-03 17:47:44 +01:00
Aleksandr BogomiakovandClaude e6ec62bcba Windows installer: find the Wintun licence at the archive root 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-10-03 14:38:16 +01:00
UltradesuandClaude Sonnet 5.5 21328bf818 Windows installer: agent service, Wintun, tray and PATH in one setup program 
Running on Windows needed wintun.dll by hand, an elevated console and a
manually started agent; without the DLL the agent silently ran with no
interface (and so no exit node or routing at all).

`tsng service` (hidden) runs the same `up` under the Windows service manager:
src/winservice.rs registers with the SCM through the windows-service crate,
reports Running/StopPending/Stopped, and maps a stop request onto the existing
shutdown path (stop_signal). The only unsafe is the extern "system" entry point
that define_windows_service! expands to, allowed in that module alone. A
service has no console, so it logs to %ProgramData%\Tsunagi\logs\tsng.log
(one older file kept past 8 MiB). Its state defaults to
%ProgramData%\Tsunagi\{data,cache} and its control socket to the system-wide
default every client already uses, so an unelevated `tsng status` and the tray
find it.

dist/windows/tsunagi.iss (Inno Setup) installs tsng.exe, the tray and Wintun
0.14.1 (pinned by SHA-256 in CI, licence text shipped), registers the
"Tsunagi" service as LocalSystem with auto start and restart-on-failure, locks
the data/cache/log directories to SYSTEM and Administrators by SID, and by
tasks adds tsng to the PATH, an inbound firewall rule for the agent and a tray
autostart. It stops the service, the tray and any hand-started agent before
replacing files; uninstall removes the service, rule and PATH entry and asks
before deleting the data (never on a silent uninstall). The release workflows
(GitHub and Gitea) build it on the Windows job and publish
tsunagi-setup-<version>-x86_64.exe.

.cargo/config.toml links the MSVC C runtime statically so nothing needs a
Visual C++ redistributable on a clean machine.

Compatibility: an agent run by hand keeps its data in the user profile; the
service has its own, so networks must be joined again once.

Checks: cargo fmt --check, clippy -D warnings, cargo test --workspace passed on
macOS (one unrelated wireguard restart test failed once under load and passed
on three reruns). winservice.rs was cargo-checked for x86_64-pc-windows-msvc in
a scratch crate; the whole tsunagi crate cannot be built for Windows from
macOS (aws-lc needs the Windows SDK). The .iss script and the CI step were NOT
compiled or run: no Inno Setup or Windows here.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
 2026-10-03 14:16:05 +01:00
UltradesuandClaude Sonnet 5.5 7c176e0b74 Exit nodes on Windows: use a member as exit node via destination bypass routes 
Choosing an exit node on Windows did nothing: the using side was refused,
because Windows has no per-user routing and the agent's own traffic would loop
into the tunnel it carries.

Now the overlay interface gets 0.0.0.0/1 and 128.0.0.0/1 routes (and ::/1,
8000::/1 as an IPv6 blackhole), and every underlay address the agent talks to
directly gets a /32 route through the physical default gateway, marked with
route metric 7373 so stale ones are found and removed. Bypass routes are
written before the halves and removed after them; a failed step removes what
was written. All routes are in the active store.

The addresses come from the runtime: each network runtime hands the interface
the verified direct IPv4 paths of its control connections and the relay hosts
in use (Interface::set_underlay, refreshed every dial round); relay names are
resolved when the rules are applied. ExitHostRules::needs_bypass (true only on
Windows) gates it, so Linux and macOS plans never carry or re-apply on path
changes. ExitHostPlan gains `bypass` and `bypass_hosts`; networks leaving
forget their entry.

Limits: a path that appears after the last apply goes through the tunnel until
the next apply; data-plane-only addresses are not collected separately (the
data links ride the same iroh paths). Nothing was run on Windows.

Checks: cargo fmt --check, cargo clippy --locked --workspace --all-targets
-D warnings and cargo test --locked --workspace --all-targets passed on macOS;
scripts and ordering are unit-tested with an injected runner and resolver, and
Interface::set_underlay with the mock host.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
 2026-10-03 13:43:01 +01:00
UltradesuandClaude Sonnet 5.5 637915600e Exit nodes on Windows: offer through WinNAT; using is refused 
Windows reported exit nodes as unsupported. The offering side now creates one
WinNAT object per offered range (New-NetNat, named tsunagi-exit:<if>:<range>),
prunes this interface's stale ones, reads (never writes) IPv4 forwarding of
the overlay interface, and removes everything on clear. All values are
single-quoted PowerShell literals.

The using side is deliberately not implemented: Windows has no per-user or
per-process routing, so the agent's own QUIC/relay traffic cannot be kept out
of a default route through the overlay and would loop. It reports that reason
and installs no route. The CLI/tray forwarding hint and the kernel-forwarding
warning name the Windows command; docs/exit-node.md has a Windows section.

Checks: cargo fmt --check, cargo clippy --locked --workspace --all-targets
-D warnings and cargo test --locked --workspace --all-targets passed on macOS.
The script builders and plan logic are unit-tested with an injected runner on
every platform; nothing was run against a real Windows host or cross-compiled
(no Windows target installed).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
 2026-10-03 12:33:22 +01:00
UltradesuandClaude Sonnet 5.5 47c5d38d0c Exit nodes on macOS: pf anchor for offering, routes plus route-to for using 
The exit-node host rules existed only for Linux (iptables + policy routing);
macOS reported "only supported on Linux yet".

Offering: a pf anchor com.apple/tsunagi-exit-<if> (evaluated by the stock
pf.conf) holds a nat rule per offered range and a pass-in rule. pf is enabled
if needed and the pfctl -E reference is released when the rules are removed.
net.inet.ip.forwarding is only read, never written, as on Linux.

Using: macOS has no uidrange policy routing and the agent is root, so the
overlay gets 0.0.0.0/1 and 128.0.0.0/1 routes and a pf route-to rule (matched
by the agent's uid) sends the agent's own QUIC/relay traffic back out the
physical interface, translated to its address. The rules are loaded before the
routes, so a failed load never sends traffic into a tunnel it cannot exempt
the agent from. IPv6 is blocked with rejecting ::/1 and 8000::/1 routes.
ExitHostPlan gains `overlay` (every network range) for the route-to exemption;
Linux ignores it.

Known limits: the egress is the default route's interface at apply time; other
root processes also bypass the tunnel. Forwarding hints in the CLI and tray
name the macOS sysctl. The rule text and command order are unit-tested with a
scripted host; nothing was run against a live pf/route (needs root), so the
route-to + nat second-pass behaviour is unverified.

Checks: cargo fmt --check, cargo clippy --locked --workspace --all-targets
-D warnings and cargo test --locked --workspace --all-targets, run on macOS
only.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
 2026-10-03 12:29:40 +01:00
Aleksandr BogomiakovandClaude 0cfc1cbafc Tray state icon and window picture; firewall rules without the comment match; macOS app bundle 
The tray icon is dark red with no connection (or a lost exit node), blue when
connected and green while an exit node is in use; the windows get the logo and
an app id that matches the desktop entry, and the exit icon moves to the
bottom left of the tile. Exit-node and broadcast firewall rules fall back to
being recognised by their shape on kernels that have no `comment` match, as
small router builds do. The macOS archive carries Tsunagi.app for the Homebrew
cask, and the daemon plist opens the control socket to the admin group.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-10-03 03:30:47 +01:00
Aleksandr BogomiakovandClaude c9c653023d Release: build a static ARMv7 (armhf) Linux agent 
Adds armv7-unknown-linux-musleabihf to both release pipelines: the archive,
a .deb for armhf and an Arch package for armv7h. Agent and CLI only, like the
other non-x86_64 Linux builds.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-10-03 02:25:43 +01:00
Aleksandr BogomiakovandClaude 9966b50407 Rename the CLI package to tsunagi and its executable to tsng 
The command line package is now `tsunagi` and builds the `tsng` executable,
which the packages, the unit, the installers and the workflows install and
run. The core library package becomes `tsunagi-core`, keeping the library name
`tsunagi`, so no code changes `use tsunagi::...`. Directories follow
(crates/tsunagi-core, crates/tsunagi). Every command shown to a person is now
`tsng ...`.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-10-03 02:14:43 +01:00
Aleksandr BogomiakovandClaude bb38562b0a Exit nodes: block IPv6 while one is in use; exit icon on the network tile 
IPv6 has no way through an exit node, so the same policy rules get an
unreachable default for IPv6 instead of leaking it. The tray shows a green
icon in the tile corner while an exit node is in use, with a one-line
confirmation to stop it; confirmations are shortened.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-10-03 02:09:14 +01:00
Aleksandr BogomiakovandClaude b2431d7eb1 Exit nodes: offer one per network, send all traffic through a member 
A member can offer itself as an exit node per network (off by default); it is
announced over the control channel (ctrl/4) once its iptables masquerade and
forward rules are in place, with a warning when kernel forwarding is off. A
device picks one member as its exit node via policy routing, with the agent's
own uid exempt from the tunnel; if the exit node disappears the rules stay
and traffic is dropped, loudly reported. CLI: `network exit-node`,
`exit-node`; tray: tile checkbox, warnings and a per-device icon with
confirmation. Linux only.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-10-03 01:14:51 +01:00
Aleksandr BogomiakovandClaude c62d29fa14 Arch package: start the service and let the GUI reach the agent 
The control socket is now opened to the tsunagi group (unit), the Arch
package enables and starts the service on install, adds the installing user
to the group, restarts a running agent on upgrade, and depends on the
libraries the tray loads at run time. The tray looks for the agent's socket
again on every poll instead of once at start, and a missing indicator
library is reported instead of panicking. Repository URLs unified.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-10-02 23:29:51 +01:00
Aleksandr BogomiakovandClaude a6d9edabee Point the project repository at github.com/house-of-vanity/tsunagi 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-10-02 23:06:48 +01:00
Aleksandr BogomiakovandClaude 2f7f9980c7 Tray: open the devices window wider and let its table scroll sideways 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-10-02 23:05:08 +01:00
Aleksandr BogomiakovandClaude 2e178a6f9d Tray: About window, fixed-width device table with totals; WTFPL
Release / tsunagi-linux-aarch64 (push) Successful in 2m19s
Release / tsunagi-linux-x86_64 (push) Successful in 3m56s
Release / tsunagi-macos-aarch64 (push) Canceled after 0s
Release / tsunagi-windows-x86_64 (push) Canceled after 0s
Release / Publish release (push) Canceled after 0s 
Add an About window (client and agent details, protocol versions), pin the
device table's column widths and add a total row, show the version in the
header and drop the refresh button. Relicense under WTFPL.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-10-02 20:16:01 +01:00
Aleksandr BogomiakovandClaude f4dcdec662 Tray: separate tray and window processes; device list without self
Release / tsunagi-linux-aarch64 (push) Successful in 2m44s
Release / tsunagi-linux-x86_64 (push) Successful in 4m37s
Release / tsunagi-macos-aarch64 (push) Canceled after 0s
Release / tsunagi-windows-x86_64 (push) Canceled after 0s
Release / Publish release (push) Canceled after 0s 
The tray process holds only the icon and starts the same binary with
--window on Open; the window ends when closed, which works on Wayland where
a window cannot be hidden. The devices table now names devices as
hostname (id5), leaves out this device and counts the others.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-10-02 19:23:06 +01:00
Aleksandr BogomiakovandClaude 69310108f4 Remember the last hostname each peer announced 
Persist announced hostnames per network (state schema 5) and report them for
members, so a device that is away is still named in `status` and the tray.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-10-02 19:23:06 +01:00
Aleksandr BogomiakovandClaude e7dff34ec2 Status: ask the agent for its state directory and its own CAP_NET_ADMIN 
The CLI reached a packaged service's socket but read state from its own
default directory, so `network` saw nothing next to a running network, and
`status` reported the CLI binary's capabilities instead of the agent's.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-10-02 18:31:01 +01:00
Aleksandr BogomiakovandClaude a55ea883a1 Tray: initialise GTK and run its main loop on Linux
Release / tsunagi-linux-aarch64 (push) Successful in 3m14s
Release / tsunagi-linux-x86_64 (push) Successful in 5m21s
Release / tsunagi-macos-aarch64 (push) Canceled after 0s
Release / tsunagi-windows-x86_64 (push) Canceled after 0s
Release / Publish release (push) Canceled after 0s 
tray-icon needs gtk::init() and a running GTK loop on Linux, which
eframe/winit never provide, so the tray panicked at startup with "GTK has
not been initialized". Build the tray on a dedicated GTK thread instead.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-10-02 17:54:38 +01:00
UltradesuandClaude Opus 4.8 3e066f991f CI: install libxdo-dev for the Linux tray link
Release / tsunagi-linux-aarch64 (push) Successful in 3m23s
Release / tsunagi-linux-x86_64 (push) Successful in 6m58s
Release / tsunagi-macos-aarch64 (push) Canceled after 0s
Release / tsunagi-windows-x86_64 (push) Canceled after 0s
Release / Publish release (push) Canceled after 0s 
The x86_64 Linux tray failed at link with "unable to find library -lxdo":
tray-icon pulls libxdo, whose .so was not present. Add libxdo-dev to the GUI
build's apt install, alongside libgtk-3-dev and libayatana-appindicator3-dev.
Everything else (aarch64 agent, macOS, Windows) already built.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-10-02 16:51:53 +01:00
UltradesuandClaude Opus 4.8 3752419075 CI: build the Linux tray natively with GTK; drop the aarch64 GUI
Release / tsunagi-linux-aarch64 (push) Successful in 4m14s
Release / tsunagi-linux-x86_64 (push) Failing after 4m29s
Release / tsunagi-macos-aarch64 (push) Canceled after 0s
Release / tsunagi-windows-x86_64 (push) Canceled after 0s
Release / Publish release (push) Canceled after 0s 
The aarch64 GUI cross-build failed: tray-icon links GTK and libappindicator
through pkg-config, which zig cross-compilation cannot satisfy without a target
sysroot. Rather than set one up, the Linux tray is now built natively on the
x86_64 runner with libgtk-3-dev and libayatana-appindicator3-dev installed, so
pkg-config just works. The agent is still static musl for both arches.

The aarch64 Linux GUI is dropped for now: that archive stays agent-only and no
aarch64 tsunagi-gui package is produced. x86_64 gets the tray in its archive
and the tsunagi-gui .deb/.pkg.tar.zst. macOS and Windows build the tray
natively as before. Guarded on matrix.gnu_target (set only on x86_64 Linux).

Note: the native build links the runner's glibc, so the Linux GUI binary needs a
reasonably recent glibc; fine for current desktops. A lower baseline would need
a GTK sysroot or an older runner image.

YAML validated; the Linux GUI build is exercised only by a real CI run.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-10-02 16:38:58 +01:00
UltradesuandClaude Opus 4.8 f63343a6a0 CI: ship the tray GUI — in the archives and as tsunagi-gui deb/zst
Release / tsunagi-linux-aarch64 (push) Failing after 2m40s
Release / tsunagi-linux-x86_64 (push) Failing after 2m40s
Release / tsunagi-macos-aarch64 (push) Canceled after 0s
Release / tsunagi-windows-x86_64 (push) Canceled after 0s
Release / Publish release (push) Canceled after 0s 
Adds the tray application to the release:

- Every target's archive now also contains the tsunagi-tray binary. On Linux it
  is built against an old glibc baseline (via cargo-zigbuild) rather than static
  musl, because a GUI must dlopen libGL/X11/Wayland at runtime and a static musl
  binary cannot; the agent stays static musl. macOS and Windows build the tray
  natively.
- New GUI package variants for Linux: `tsunagi-gui` .deb and .pkg.tar.zst, built
  by the existing scripts with an extra tray argument. They install the agent
  (service, polkit, user — same as the headless packages) plus the tray binary,
  a desktop launcher (dist/linux/tsunagi-tray.desktop) and an icon
  (dist/linux/tsunagi.svg), so the GUI appears in the applications menu. The GUI
  package conflicts/replaces/provides the headless tsunagi so the two do not
  coexist.
- The tray finds the system service with no configuration: the library's default
  control-socket path now prefers /run/tsunagi/agent.sock when present (both the
  tray and CLI use this), falling back to the per-user path.

No control-protocol change. Builds, fmt and clippy clean on macOS; the Linux GUI
cross-build and the packages are exercised only by a real CI run.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-10-02 16:26:45 +01:00
UltradesuandClaude Opus 4.8 d13bb00623 Tray: one device table (online + offline), FQDN copy, green-fade copy feedback 
- The devices window now shows a single device table instead of a separate
  "connected" box and an easily-missed "offline" box below it. Every endpoint
  known from a live tunnel or the signed membership appears, online first;
  offline rows show name and address with dashes for the live columns. The
  table fills the window and scrolls, so nothing hides off the bottom.
- Clicking an online peer's name copies hostname.network (the FQDN) so it
  resolves; a name without a hostname copies its short id.
- Copy feedback no longer appends "Copied" text (which reflowed the table for a
  second). The field just flashes green and fades back over a second, with no
  change to its text or width.

Builds and is clippy-clean (-D warnings) on macOS; not run in this headless
session.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-10-02 16:09:33 +01:00
UltradesuandClaude Opus 4.8 2a0a8abd60 Tray: split rate/plot columns, add unit toggle to the devices window 
The connected-peers rate and sparkline shared one grid cell, so the plot still
moved right as the rate text grew (e.g. "1.0 kB/s"). They are now two separate
grid columns — the plot column starts at the same x on every row, since grid
columns align.

The devices window also gets its own pkts/bytes toggle, bound to the same unit
setting as the main window so the two stay in step.

Builds and is clippy-clean (-D warnings) on macOS; not run in this headless
session.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-10-02 16:04:39 +01:00
UltradesuandClaude Opus 4.8 db01c58a6f Tray: Phosphor icons, click-to-copy fields with feedback, steady plot column 
- Bundle egui-phosphor (pinned to egui 0.31) and register its glyphs, so the
  icon buttons render real icons instead of tofu: refresh, shuffle (generate
  secret), pencil/check/x (rename host), copy (Copy secret).
- Copying is now the field itself, not a separate button: clicking the
  shortened device id, the network name, the network id, a range or an address
  copies the full value and the field flashes green "Copied" for a second
  (tracked in egui temp memory, so it clears itself). The standalone copy
  buttons are gone.
- In the devices window the connected-peers "rate / plot" column put the rate
  and the inline sparkline in a shared row, so the plot shifted as the rate
  text changed width ("0 B/s" vs "1.1 kB/s"). The rate now sits in a
  fixed-width, truncating cell, so the plot stays put.

Builds and is clippy-clean (-D warnings) on macOS; the UI has not been run in
this headless session.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-10-02 16:00:23 +01:00
UltradesuandClaude Opus 4.8 de56258eb1 Tray: copy a network secret via ipc; use text buttons, not missing glyphs 
Two fixes.

Secret copy: a new local control request returns a configured network's secret
in its tsn1 text form, so the tray's "Copy secret" button can put it on the
clipboard. The agent reads it from local state (same source as the CLI's
`network secret show`); it crosses only the owner/group socket and is wrapped in
a RedactedText type whose Debug prints "<redacted>", so it can never land in a
log. In the GUI the worker fetches it and hands it to the UI thread (which owns
the clipboard) through a one-shot slot kept out of the status snapshot, so it is
copied once and never stored in the cloned snapshot. Control protocol bumped
16 -> 17.

Icons: the button glyphs (⧉ ⟳ ✎ ✔ ✕) are not in egui's bundled fonts and showed
as tofu squares, so they are now plain text: copy / refresh / random / edit /
save / cancel. No icon font to embed.

Checks: fmt, clippy --workspace --all-targets -D warnings, and the tsunagi lib,
tsunagi-cli and wg-quic local_control tests pass on macOS.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-10-02 15:48:32 +01:00
UltradesuandClaude Opus 4.8 8f5ce21ce9 Tray: devices window, bytes/packets toggle, copy & edit affordances 
Replaces the in-tile collapsing peer list with a "Show devices" button that
opens a separate, normal-sized window per network, and fills in the rest of the
feedback.

Main window:
- A global show: pkts/bytes toggle; all rates and sparklines follow it.
- Sparklines now reserve their space always (fixed height, "no data yet"
  placeholder) so the layout no longer jumps when data arrives, and each one
  overlays its peak (top-left) and current value (bottom-right) with units.
- Device id has a copy button; the hostname is editable in place (✎ → edit →
  ✔/Enter sends SetHostname); network name, range and address copy on click.
- The Leave button is a neutral red. Per-tile "Show devices" opens the detail.

Devices window (a separate egui viewport per network):
- Network summary: id (copy), active/broadcast, range, this device's address,
  candidates, relay fwd/via/recv, dial/handshake failures.
- A larger network traffic sparkline.
- Connected peers in a table: hostname when known (else short key), protocol,
  address, handshake age formatted (s/m/h/d), tx/rx packets, tx/rx bytes, and
  current rate with an inline sparkline. Names and addresses copy on click.
- Offline members listed below with address and failed-dial count.

Agent: a SetHostname command (over the existing ipc) backs the in-place rename.
Traffic sampler now tracks both packet and byte rates; the toggle selects which.

Deferred: a "copy secret" button on the tile — the secret is deliberately not
exported to the GUI (secrets never leave the agent), so it needs a small local
secret-export ipc, which is the next step.

Builds and is clippy-clean (-D warnings) on macOS; the UI has not been run in
this headless session.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-10-02 15:37:08 +01:00
UltradesuandClaude Opus 4.8 0a344c1b4f Report per-peer byte counters, not just packets 
The overlay peer report carried packet counts only, so the GUI could show
packets/s but no byte throughput. Adds tx_bytes/rx_bytes to OverlayPeerReport
and fills them from each data plane:

- the wg and wg-quic plugins already counted plaintext bytes per peer; the CLI
  now maps them into the report;
- the generic tunnel (tcp-tls) gains tx_bytes/rx_bytes on its peer state and
  report, incremented by the plaintext length at the same points it counts
  packets.

The local control protocol version is bumped (TSN 15 -> 16) since the report
shape changed; a client and agent on different builds already detect and report
the mismatch.

Checks: fmt, clippy --workspace --all-targets -D warnings, and the tsunagi lib,
tsunagi-cli and wg-quic local_control tests pass on macOS.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-10-02 15:30:37 +01:00
UltradesuandClaude Opus 4.8 2cda56e1f8 Log inbound handshake rejections at debug, not warn 
Creating and leaving networks produced a stream of WARN lines like "Handshake
rejected: peer requested an unknown or inactive network network=None". This is
not a leave bug: leaving deactivates the network and publishes a release, but
remote peers that still consider this device a member keep dialing, and each
dial for a network no longer served is correctly rejected by the responder
(lookup returns None, so the network stays unidentified — network=None by
design, since a pre-handshake claim is unverified).

Such a rejection is routine and remotely triggerable — a left network a peer
has not caught up on, a stranger, a port scan, an old client — so logging it at
WARN let any peer spam the log. It is now debug; enable debug to see rejections
when diagnosing. The rejection behaviour itself is unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-10-02 15:27:49 +01:00
UltradesuandClaude Opus 4.8 d1de61ba29 Tray UI: start hidden, fixed window, grid layout, traffic + peers 
Reworks the tray window per feedback and fills it in.

- Starts hidden in the tray (no window on launch); the window opens from the
  menu. On macOS the app now uses the Accessory activation policy, so it lives
  in the menu bar with no Dock icon (via a winit event-loop hook, pinned to
  eframe's winit). Fixed, non-resizable 360x600 window; the layout targets it.
- Tidied layout: top bar (title + refresh), a bottom panel with the join form
  and the action banner, and a scrolling list of network tiles between them.
  Labels and values sit on two-column grids so they align; the join name and
  secret inputs are the same fixed width.
- A ⟳ button beside the secret generates a fresh random secret
  (NetworkSecret::generate), for creating a network.
- Each network tile shows id, range, this device's address, peers
  (connected/known), the active and broadcast switches, a Leave button, and a
  collapsing "peers (N)" section with a formatted per-peer table (short key,
  address, handshake age, tx/rx packets, rate).
- Traffic: a new sampler turns the agent's cumulative packet counters into a
  packets-per-second rate per network and per peer, from the deltas between
  snapshots (the snapshot now carries a generation and timestamp so each
  refresh is sampled once). The network tile shows the current rate and a small
  hand-drawn sparkline of recent rates (no extra dependency).

Note: rates are packets/s — the report has no byte counters yet; bytes would
need an agent change. Builds and is clippy-clean (-D warnings) on macOS; the UI
has not been run in this headless session.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-10-02 15:08:29 +01:00
UltradesuandClaude Opus 4.8 19509dc833 Add the tsunagi-tray GUI (egui) — first increment 
A native, lightweight tray application to manage a running agent — no web
engine. It is a thin client over the agent's local control socket, reusing
tsunagi::ipc and the report types directly; it owns its own runtime, logger and
event loop, like the CLI.

This first increment:
- tray icon with an Open/Quit menu (tray-icon); menu events are read on a
  background thread so Open works even while the window is hidden;
- a small window that hides to the tray on close and reopens from the menu;
- a background tokio worker that polls status every 2s and runs commands off the
  UI thread, writing a snapshot the UI renders from and repainting egui;
- UI: device header, a per-network card (name, id, range, this device's
  address, peers connected/known, a Leave button, and the per-network active
  and broadcast switches), a join form (name + secret), and a transient
  action/error banner; an "agent unreachable" state that names the socket.

Socket resolution: TSUNAGI_CONTROL_SOCKET if set, else the platform default
(/var/run/tsunagi/agent.sock on macOS) — the same socket the group-access agent
serves.

Scope/notes: macOS first (Linux and Windows follow). Not yet done — macOS
accessory activation policy (so there is no dock icon), light/dark tray icons
from assets (the icon is drawn in code for now), and packaging/autostart. Builds
and is clippy-clean (-D warnings) on macOS; the UI itself has not been run in
this headless session.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-10-02 14:48:52 +01:00
UltradesuandClaude Opus 4.8 de8e48bee7 Let the agent grant the control socket to a group (0660) 
For a GUI (or a CLI) running as a different user to reach the agent, the control
socket needs to be group-accessible. This is set by the agent when it binds the
socket — correct from creation, not patched afterwards.

- ipc::ControlSocketAccess { Private, Group(gid) }. Private is the existing
  owner-only 0600. Group sets the socket's group to the gid and mode 0660, and
  creates the socket's parent directory 0755 so the group can traverse to it; if
  the group cannot be set it falls back to owner-only rather than leaving it
  wrong. The chown uses std::os::unix::fs::chown — no unsafe, no new lib
  dependency.
- ControlSocket::bind takes the access policy (Windows accepts it and keeps its
  pipe default for now; a pipe security descriptor is a later follow-up).
- CLI: `tsunagi up --control-group <name>` (env TSUNAGI_CONTROL_GROUP) resolves
  the group to its gid via nix (safe wrapper, unix-only dep) and passes
  Group(gid); without it the socket stays Private. A name that does not resolve
  is reported, and the socket stays owner-only.
- control_socket() now serves the system path (default_control_socket_path) on
  macOS too, not just Windows: /var/run/tsunagi/agent.sock, a runtime directory
  the GUI can reach, rather than the owner-only state directory. A custom
  --state-dir still keeps its own per-user socket.

So the packaged/daemon agent runs e.g. `tsunagi up --control-group tsunagi` and a
user in that group talks to it; a plain per-user agent is unchanged (0600).

Checks: fmt, clippy --workspace --all-targets -D warnings, tsunagi lib tests and
the wg-quic local_control tests pass on macOS.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-10-02 14:41:58 +01:00
UltradesuandClaude Opus 4.8 8c812e6874 Bump the GitHub workflow actions off deprecated Node 20
Release / tsunagi-linux-aarch64 (push) Successful in 5m25s
Release / tsunagi-linux-x86_64 (push) Failing after 3h12m37s
Release / tsunagi-macos-aarch64 (push) Canceled after 0s
Release / tsunagi-windows-x86_64 (push) Canceled after 0s
Release / Publish release (push) Canceled after 0s 
GitHub now forces actions/checkout@v4, actions/upload-artifact@v4 and
actions/download-artifact@v4 (all Node 20) onto Node 24 with a deprecation
warning. Move them to the current Node 24 majors — checkout@v5,
upload-artifact@v7, download-artifact@v8 — which keep the same inputs and the
per-artifact subdirectory layout (merge-multiple defaults to false), so the
publish step is unchanged.

Only the .github workflow is touched. The .gitea workflow stays on
checkout@v4 and the @v3 artifact actions, because Gitea does not support the v4+
artifact backend and may not have Node 24 on its runner image; it also does not
emit these deprecation warnings. mlugg/setup-zig has no Node 24 release yet
(latest v2.2.1 is still Node 20), so that one warning remains until upstream
updates — nothing to change on our side.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-10-02 14:01:33 +01:00
270 changed files with 67880 additions and 3450 deletions
Expand all files Collapse all files

No files matched your search

.cargo/config.toml
+5
View File
@@ -0,0 +1,5 @@
# Windows: link the C runtime statically, so the agent and the tray start on a
# machine that has no Visual C++ redistributable installed. The installer ships
# nothing but our own files and Wintun.
[target.x86_64-pc-windows-msvc]
rustflags = ["-C", "target-feature=+crt-static"]
.gitea/workflows/release.yml
+100 -7
View File
@@ -36,6 +36,7 @@ jobs:
deb: true
deb_arch: amd64
pkg_arch: x86_64
gnu_target: x86_64-unknown-linux-gnu
- os: ubuntu-latest
target: aarch64-unknown-linux-musl
asset: tsunagi-linux-aarch64
@@ -44,6 +45,16 @@ jobs:
deb: true
deb_arch: arm64
pkg_arch: aarch64
# 32-bit ARM with hardware float (ARMv7, `v7l`): single-board and edge
# devices. Statically linked like the others.
- os: ubuntu-latest
target: armv7-unknown-linux-musleabihf
asset: tsunagi-linux-armv7
archive: tar.gz
zig: true
deb: true
deb_arch: armhf
pkg_arch: armv7h
- os: macos-latest
target: aarch64-apple-darwin
asset: tsunagi-macos-aarch64
@@ -97,9 +108,33 @@ jobs:
# one lint to keep the musl build log clean. A real link error still
# fails the build.
export RUSTFLAGS="${RUSTFLAGS:-} -A linker_messages"
cargo zigbuild --release --locked --target "${{ matrix.target }}" --bin tsunagi
cargo zigbuild --release --locked --target "${{ matrix.target }}" --bin tsng
else
cargo build --release --locked --target "${{ matrix.target }}" --bin tsunagi
cargo build --release --locked --target "${{ matrix.target }}" --bin tsng
fi
# The tray GUI. On Linux it is built against glibc (an old baseline for
# portability), not musl: a GUI must dlopen libGL/X11/Wayland at runtime,
# which a static musl binary cannot. macOS and Windows build it natively.
- name: Build the tray GUI
shell: bash
run: |
set -euo pipefail
if [[ "${{ runner.os }}" == "Linux" ]]; then
if [[ -n "${{ matrix.gnu_target }}" ]]; then
# tray-icon links GTK and libappindicator through pkg-config, which
# has no cross story under zig, so the Linux GUI is built natively
# (x86_64 only) against the runner's glibc.
sudo apt-get update
sudo apt-get install -y --no-install-recommends \
libgtk-3-dev libayatana-appindicator3-dev libxdo-dev
cargo build --release --locked \
--target "${{ matrix.gnu_target }}" --bin tsunagi-tray
else
echo "no GUI build for ${{ matrix.target }}"
fi
else
cargo build --release --locked --target "${{ matrix.target }}" --bin tsunagi-tray
fi
- name: Package
@@ -111,10 +146,21 @@ jobs:
stage="dist/pkg/${{ matrix.asset }}"
mkdir -p "$stage"
bin="tsunagi"
[[ "${{ runner.os }}" == "Windows" ]] && bin="tsunagi.exe"
bin="tsng"
[[ "${{ runner.os }}" == "Windows" ]] && bin="tsng.exe"
cp "target/${{ matrix.target }}/release/${bin}" "$stage/"
# The tray GUI alongside it, where it was built (Linux: x86_64 only).
tray="tsunagi-tray"
[[ "${{ runner.os }}" == "Windows" ]] && tray="tsunagi-tray.exe"
if [[ "${{ runner.os }}" == "Linux" ]]; then
if [[ -n "${{ matrix.gnu_target }}" ]]; then
cp "target/${{ matrix.gnu_target }}/release/${tray}" "$stage/"
fi
else
cp "target/${{ matrix.target }}/release/${tray}" "$stage/"
fi
# The top-level README, plus the per-OS install notes and integration
# files, so the archive carries what the platform needs to set up.
cp README.md "$stage/"
@@ -127,6 +173,10 @@ jobs:
macOS)
cp dist/macos/cy.hexor.tsunagi.plist "$stage/"
cp dist/macos/README.md "$stage/INSTALL.md"
# The tray as an application bundle, for the Homebrew cask.
bash dist/macos/build-app.sh \
"target/${{ matrix.target }}/release/tsunagi-tray" \
"$version" "$stage/Tsunagi.app"
;;
Windows)
cp dist/windows/README.md "$stage/INSTALL.md"
@@ -140,6 +190,29 @@ jobs:
tar -C dist/pkg -czf "${archive}" "${{ matrix.asset }}"
fi
# The Windows installer: the agent as a service, Wintun, the tray and the
# PATH entry in one setup program. Wintun is the signed library from
# wintun.net, pinned by hash so a changed download fails the build.
- name: Build the Windows installer
if: runner.os == 'Windows'
shell: pwsh
run: |
$ErrorActionPreference = 'Stop'
$version = $env:GITHUB_REF_NAME.TrimStart('v')
Invoke-WebRequest -Uri https://www.wintun.net/builds/wintun-0.14.1.zip -OutFile wintun.zip
$hash = (Get-FileHash wintun.zip -Algorithm SHA256).Hash
if ($hash -ne '07C256185D6EE3652E09FA55C0B673E2624B565E02C4B9091C79CA7D2F24EF51') {
throw "wintun.zip has an unexpected hash: $hash"
}
Expand-Archive wintun.zip -DestinationPath wintun-dl
choco install innosetup -y --no-progress
$iscc = Join-Path ${env:ProgramFiles(x86)} 'Inno Setup 6\ISCC.exe'
$stage = Join-Path $PWD 'dist/pkg/${{ matrix.asset }}'
& $iscc "/DAppVersion=$version" "/DSourceDir=$stage" `
"/DWintunDir=$(Join-Path $PWD 'wintun-dl/wintun/bin/amd64')" `
"/O$PWD" dist/windows/tsunagi.iss
if ($LASTEXITCODE -ne 0) { throw "ISCC failed with $LASTEXITCODE" }
# A dependency-free .deb for the x86_64 and aarch64 Linux builds: it drops
# the static binary, the systemd service and the polkit rule. Built with
# dpkg-deb (present on the runner); see dist/deb/build-deb.sh.
@@ -150,7 +223,7 @@ jobs:
set -euo pipefail
version="${GITHUB_REF_NAME#v}"
bash dist/deb/build-deb.sh \
"target/${{ matrix.target }}/release/tsunagi" \
"target/${{ matrix.target }}/release/tsng" \
"${{ matrix.deb_arch }}" \
"$version" \
"tsunagi_${version}_${{ matrix.deb_arch }}.deb"
@@ -166,11 +239,30 @@ jobs:
version="${GITHUB_REF_NAME#v}"
pkgver="${version//-/_}"
bash dist/arch/build-pkg.sh \
"target/${{ matrix.target }}/release/tsunagi" \
"target/${{ matrix.target }}/release/tsng" \
"${{ matrix.pkg_arch }}" \
"$version" \
"tsunagi-${pkgver}-1-${{ matrix.pkg_arch }}.pkg.tar.zst"
# The GUI variants: the same agent plus the tray binary and a desktop
# launcher, named tsunagi-gui (conflicting with the headless packages).
- name: Build the GUI packages
if: matrix.gnu_target
shell: bash
run: |
set -euo pipefail
version="${GITHUB_REF_NAME#v}"
pkgver="${version//-/_}"
tray="target/${{ matrix.gnu_target }}/release/tsunagi-tray"
bash dist/deb/build-deb.sh \
"target/${{ matrix.target }}/release/tsng" \
"${{ matrix.deb_arch }}" "$version" \
"tsunagi-gui_${version}_${{ matrix.deb_arch }}.deb" "$tray"
bash dist/arch/build-pkg.sh \
"target/${{ matrix.target }}/release/tsng" \
"${{ matrix.pkg_arch }}" "$version" \
"tsunagi-gui-${pkgver}-1-${{ matrix.pkg_arch }}.pkg.tar.zst" "$tray"
# @v3 on purpose: Gitea does not support the v4 artifact backend. This is
# what makes the archive (and the .deb, where built) show up under the run
# and hands it to `publish`.
@@ -182,6 +274,7 @@ jobs:
${{ matrix.asset }}-*.*
*.deb
*.pkg.tar.zst
tsunagi-setup-*.exe
if-no-files-found: error
publish:
@@ -253,5 +346,5 @@ jobs:
cat up.json >&2 || true
rc=1
fi
done < <(find artifacts -type f \( -name '*.tar.gz' -o -name '*.zip' -o -name '*.deb' -o -name '*.pkg.tar.zst' \) -print0)
done < <(find artifacts -type f \( -name '*.tar.gz' -o -name '*.zip' -o -name '*.deb' -o -name '*.pkg.tar.zst' -o -name 'tsunagi-setup-*.exe' \) -print0)
exit "$rc"
.github/workflows/release.yml
+110 -16
View File
@@ -2,13 +2,14 @@ name: Release
# GitHub version of the release pipeline. The build half mirrors the Gitea one
# in .gitea/workflows/release.yml (static musl Linux via cargo-zigbuild, native
# macOS and Windows); only the publish half differs: GitHub needs the v4
# artifact backend and fills the release with `gh`.
# macOS and Windows); only the publish half differs: it uses the current
# artifact actions (Node 24) and fills the release with `gh`.
#
# Every job is gated on `github.server_url == 'https://github.com'`, so this file
# runs only on GitHub. Gitea runs its own copy in .gitea/workflows; should a
# Gitea instance also read .github/workflows, the guard skips these jobs there
# so they never fail on the @v4 artifact backend or `gh`.
# runs only on GitHub. Gitea runs its own copy in .gitea/workflows (which pins
# the older @v3 artifact actions it supports); should a Gitea instance also read
# .github/workflows, the guard skips these jobs there so they never fail on the
# artifact backend or `gh`.
on:
push:
tags: ["v*"]
@@ -37,6 +38,7 @@ jobs:
deb: true
deb_arch: amd64
pkg_arch: x86_64
gnu_target: x86_64-unknown-linux-gnu
- os: ubuntu-latest
target: aarch64-unknown-linux-musl
asset: tsunagi-linux-aarch64
@@ -45,6 +47,16 @@ jobs:
deb: true
deb_arch: arm64
pkg_arch: aarch64
# 32-bit ARM with hardware float (ARMv7, `v7l`): single-board and edge
# devices. Statically linked like the others.
- os: ubuntu-latest
target: armv7-unknown-linux-musleabihf
asset: tsunagi-linux-armv7
archive: tar.gz
zig: true
deb: true
deb_arch: armhf
pkg_arch: armv7h
- os: macos-latest
target: aarch64-apple-darwin
asset: tsunagi-macos-aarch64
@@ -57,7 +69,7 @@ jobs:
zig: false
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v5
- uses: dtolnay/rust-toolchain@stable
with:
@@ -98,9 +110,33 @@ jobs:
# one lint to keep the musl build log clean. A real link error still
# fails the build.
export RUSTFLAGS="${RUSTFLAGS:-} -A linker_messages"
cargo zigbuild --release --locked --target "${{ matrix.target }}" --bin tsunagi
cargo zigbuild --release --locked --target "${{ matrix.target }}" --bin tsng
else
cargo build --release --locked --target "${{ matrix.target }}" --bin tsunagi
cargo build --release --locked --target "${{ matrix.target }}" --bin tsng
fi
# The tray GUI. On Linux it is built against glibc (an old baseline for
# portability), not musl: a GUI must dlopen libGL/X11/Wayland at runtime,
# which a static musl binary cannot. macOS and Windows build it natively.
- name: Build the tray GUI
shell: bash
run: |
set -euo pipefail
if [[ "${{ runner.os }}" == "Linux" ]]; then
if [[ -n "${{ matrix.gnu_target }}" ]]; then
# tray-icon links GTK and libappindicator through pkg-config, which
# has no cross story under zig, so the Linux GUI is built natively
# (x86_64 only) against the runner's glibc.
sudo apt-get update
sudo apt-get install -y --no-install-recommends \
libgtk-3-dev libayatana-appindicator3-dev libxdo-dev
cargo build --release --locked \
--target "${{ matrix.gnu_target }}" --bin tsunagi-tray
else
echo "no GUI build for ${{ matrix.target }}"
fi
else
cargo build --release --locked --target "${{ matrix.target }}" --bin tsunagi-tray
fi
- name: Package
@@ -112,10 +148,21 @@ jobs:
stage="dist/pkg/${{ matrix.asset }}"
mkdir -p "$stage"
bin="tsunagi"
[[ "${{ runner.os }}" == "Windows" ]] && bin="tsunagi.exe"
bin="tsng"
[[ "${{ runner.os }}" == "Windows" ]] && bin="tsng.exe"
cp "target/${{ matrix.target }}/release/${bin}" "$stage/"
# The tray GUI alongside it, where it was built (Linux: x86_64 only).
tray="tsunagi-tray"
[[ "${{ runner.os }}" == "Windows" ]] && tray="tsunagi-tray.exe"
if [[ "${{ runner.os }}" == "Linux" ]]; then
if [[ -n "${{ matrix.gnu_target }}" ]]; then
cp "target/${{ matrix.gnu_target }}/release/${tray}" "$stage/"
fi
else
cp "target/${{ matrix.target }}/release/${tray}" "$stage/"
fi
# The top-level README, plus the per-OS install notes and integration
# files, so the archive carries what the platform needs to set up.
cp README.md "$stage/"
@@ -128,6 +175,10 @@ jobs:
macOS)
cp dist/macos/cy.hexor.tsunagi.plist "$stage/"
cp dist/macos/README.md "$stage/INSTALL.md"
# The tray as an application bundle, for the Homebrew cask.
bash dist/macos/build-app.sh \
"target/${{ matrix.target }}/release/tsunagi-tray" \
"$version" "$stage/Tsunagi.app"
;;
Windows)
cp dist/windows/README.md "$stage/INSTALL.md"
@@ -141,6 +192,29 @@ jobs:
tar -C dist/pkg -czf "${archive}" "${{ matrix.asset }}"
fi
# The Windows installer: the agent as a service, Wintun, the tray and the
# PATH entry in one setup program. Wintun is the signed library from
# wintun.net, pinned by hash so a changed download fails the build.
- name: Build the Windows installer
if: runner.os == 'Windows'
shell: pwsh
run: |
$ErrorActionPreference = 'Stop'
$version = $env:GITHUB_REF_NAME.TrimStart('v')
Invoke-WebRequest -Uri https://www.wintun.net/builds/wintun-0.14.1.zip -OutFile wintun.zip
$hash = (Get-FileHash wintun.zip -Algorithm SHA256).Hash
if ($hash -ne '07C256185D6EE3652E09FA55C0B673E2624B565E02C4B9091C79CA7D2F24EF51') {
throw "wintun.zip has an unexpected hash: $hash"
}
Expand-Archive wintun.zip -DestinationPath wintun-dl
choco install innosetup -y --no-progress
$iscc = Join-Path ${env:ProgramFiles(x86)} 'Inno Setup 6\ISCC.exe'
$stage = Join-Path $PWD 'dist/pkg/${{ matrix.asset }}'
& $iscc "/DAppVersion=$version" "/DSourceDir=$stage" `
"/DWintunDir=$(Join-Path $PWD 'wintun-dl/wintun/bin/amd64')" `
"/O$PWD" dist/windows/tsunagi.iss
if ($LASTEXITCODE -ne 0) { throw "ISCC failed with $LASTEXITCODE" }
# A dependency-free .deb for the x86_64 and aarch64 Linux builds: it drops
# the static binary, the systemd service and the polkit rule. Built with
# dpkg-deb (present on the runner); see dist/deb/build-deb.sh.
@@ -151,7 +225,7 @@ jobs:
set -euo pipefail
version="${GITHUB_REF_NAME#v}"
bash dist/deb/build-deb.sh \
"target/${{ matrix.target }}/release/tsunagi" \
"target/${{ matrix.target }}/release/tsng" \
"${{ matrix.deb_arch }}" \
"$version" \
"tsunagi_${version}_${{ matrix.deb_arch }}.deb"
@@ -167,19 +241,39 @@ jobs:
version="${GITHUB_REF_NAME#v}"
pkgver="${version//-/_}"
bash dist/arch/build-pkg.sh \
"target/${{ matrix.target }}/release/tsunagi" \
"target/${{ matrix.target }}/release/tsng" \
"${{ matrix.pkg_arch }}" \
"$version" \
"tsunagi-${pkgver}-1-${{ matrix.pkg_arch }}.pkg.tar.zst"
# The GUI variants: the same agent plus the tray binary and a desktop
# launcher, named tsunagi-gui (conflicting with the headless packages).
- name: Build the GUI packages
if: matrix.gnu_target
shell: bash
run: |
set -euo pipefail
version="${GITHUB_REF_NAME#v}"
pkgver="${version//-/_}"
tray="target/${{ matrix.gnu_target }}/release/tsunagi-tray"
bash dist/deb/build-deb.sh \
"target/${{ matrix.target }}/release/tsng" \
"${{ matrix.deb_arch }}" "$version" \
"tsunagi-gui_${version}_${{ matrix.deb_arch }}.deb" "$tray"
bash dist/arch/build-pkg.sh \
"target/${{ matrix.target }}/release/tsng" \
"${{ matrix.pkg_arch }}" "$version" \
"tsunagi-gui-${pkgver}-1-${{ matrix.pkg_arch }}.pkg.tar.zst" "$tray"
- name: Upload the build outputs as run artifacts
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@v7
with:
name: ${{ matrix.asset }}
path: |
${{ matrix.asset }}-*.*
*.deb
*.pkg.tar.zst
tsunagi-setup-*.exe
if-no-files-found: error
publish:
@@ -191,7 +285,7 @@ jobs:
contents: write
steps:
- name: Download every build artifact
uses: actions/download-artifact@v4
uses: actions/download-artifact@v8
with:
path: artifacts
@@ -206,9 +300,9 @@ jobs:
run: |
set -euo pipefail
# download-artifact@v4 nests each artifact under its own directory, so
# download-artifact nests each artifact under its own directory, so
# the archives are artifacts/<asset>/<file>.
mapfile -t files < <(find artifacts -type f \( -name '*.tar.gz' -o -name '*.zip' -o -name '*.deb' -o -name '*.pkg.tar.zst' \))
mapfile -t files < <(find artifacts -type f \( -name '*.tar.gz' -o -name '*.zip' -o -name '*.deb' -o -name '*.pkg.tar.zst' -o -name 'tsunagi-setup-*.exe' \))
printf 'asset: %s\n' "${files[@]}"
prerelease=()
.gitignore
+1
View File
@@ -1 +1,2 @@
/target
/.claude
AGENTS.md
+22 -20
View File
@@ -8,7 +8,7 @@ before touching the code. Do not restate the full design here; follow the links.
A proof-of-concept agent library for small private mesh networks. A user
supplies a network name and one shared secret; agents derive the same network
space independently, find each other, prove membership and exchange control
messages. Scope and non-scope are in [README.md](README.md).
messages. What it does and does not do is in [docs/architecture.md](docs/architecture.md) and [docs/threat-model.md](docs/threat-model.md).
There is no central server, no owner, no registration and no majority vote.
Design accordingly: a majority is not a root of trust.
@@ -30,7 +30,7 @@ scripts when improving the interactive workflow.
Keep these separate. Crossing them is the main thing to review for.
- **Control plane vs data plane.** The separation is **logical, not physical**.
The control protocol in `crates/tsunagi/src/proto/` knows nothing about
The control protocol in `crates/tsunagi-core/src/proto/` knows nothing about
packets, and a protocol crate knows nothing about the control protocol;
either can be replaced on its own. Both may ride on iroh — refusing to
would throw away iroh's NAT traversal and force the data plane to
@@ -59,8 +59,10 @@ Keep these separate. Crossing them is the main thing to review for.
other peers goes link in, link out: it is never written to the middle's
interface or decrypted there. Multihop routing has a bounded hop limit;
equal-cost next hops are chosen per flow. Reachability is first-hand and
volatile: each authenticated member advertises its own protocol-specific
links. Build routing tables on topology changes, never per packet. Transit
volatile: each authenticated member advertises its own direct links, of
whatever kind. A link is a way to reach a neighbour and a protocol is an
end-to-end tunnel to a peer; the two meet only in the envelope, so a
tunnel never depends on which transports carry it. Build routing tables on topology changes, never per packet. Transit
must not acquire a routing mutex or wait for a protocol/TUN reader.
- **Broadcast fanout belongs at local IP ingress.** Participation is local to
each network, enabled by default, persisted and advertised to authenticated
@@ -70,9 +72,9 @@ Keep these separate. Crossing them is the main thing to review for.
ownership checks or add broadcast flooding to encrypted transit.
- **Plugins never learn reachability.** An `IpPlugin` is handed a `PacketLink`
per peer and moves datagrams over it. Addresses, hole punching and relays
belong to `crates/tsunagi/src/dataplane/transport/`. A plugin announcement says *who*, never
belong to `crates/tsunagi-core/src/dataplane/transport/`. A plugin announcement says *who*, never
*where*.
- **The core never parses a plugin payload.** See `crates/tsunagi/src/dataplane/mod.rs`. Only
- **The core never parses a plugin payload.** See `crates/tsunagi-core/src/dataplane/mod.rs`. Only
`crates/tsunagi-wg-quic/src/announcement.rs` interprets `wg-quic` payloads, and
only after bounding every field. A data plane failure must never stop the
control plane.
@@ -81,7 +83,7 @@ Keep these separate. Crossing them is the main thing to review for.
inbound packets are dropped unless their source is the address derived for
the peer that sent them. Never trust an address a peer announces.
- **Signed state is the only durable agreement.** A fact that must survive a
participant being away goes in `crates/tsunagi/src/state/` as a record signed by its
participant being away goes in `crates/tsunagi-core/src/state/` as a record signed by its
author, never in a session. Merging is deterministic, an older version never
rolls back a newer one, and absence from a snapshot is not deletion. Never
add a vote or a quorum: a majority is not a trust root here, and it would
@@ -115,7 +117,7 @@ Keep these separate. Crossing them is the main thing to review for.
normal path *and* the important failures. See [docs/testing.md](docs/testing.md).
Do not add tests for getters or to move a coverage number.
2. **Never change `IDENTITY_SCHEME`, the derivation labels, or the transcript
encoding** in `crates/tsunagi/src/identity/network.rs` and `crates/tsunagi/src/proto/handshake.rs` without
encoding** in `crates/tsunagi-core/src/identity/network.rs` and `crates/tsunagi-core/src/proto/handshake.rs` without
treating it as an incompatible protocol change. Bumping the crate version or
the control protocol version must not change an existing `NetworkId`.
3. **Secrets never leak.** Not into logs, not into `Debug`, not into status
@@ -126,7 +128,7 @@ Keep these separate. Crossing them is the main thing to review for.
the library; tests opt out explicitly at the top of each file.
5. **Bounds before allocation.** Frame lengths are checked against
`Limits::max_frame_len` before a buffer is allocated. Every string, list and
queue has a limit in `crates/tsunagi/src/config.rs`.
queue has a limit in `crates/tsunagi-core/src/config.rs`.
6. **Failure is contained.** A bad signature, wrong secret, malformed packet or
unknown version rejects one message or one session. It never stops another
network and never stops the agent. There is no irreversible global error
@@ -150,18 +152,18 @@ Keep these separate. Crossing them is the main thing to review for.
| path | responsibility |
|---------------------|----------------|
| `crates/tsunagi/src/identity/` | device identity; deterministic network space identity and derived keys |
| `crates/tsunagi/src/storage/` | `state.sqlite`, `cache.sqlite`, directory ownership lock |
| `crates/tsunagi/src/discovery.rs` | candidate sources; test and static backends |
| `crates/tsunagi/src/proto/` | framing, message formats, membership handshake |
| `crates/tsunagi/src/net.rs` | iroh endpoint adapter and observability snapshots |
| `crates/tsunagi/src/agent/` | agent lifecycle, per-network runtimes, sessions, events, status |
| `crates/tsunagi/src/state/` | signed records that outlive a session, their merge rules and address allocation |
| `crates/tsunagi/src/overlay/` | the one interface an agent owns: provisioning, the TUN, routing, source checks, broadcast host rules |
| `crates/tsunagi/src/dataplane/` | the protocol contract and the authenticated packet transport |
| `crates/tsunagi/src/dns/` | the DNS view of a network: zone, server, resolver publication |
| `crates/tsunagi-core/src/identity/` | device identity; deterministic network space identity and derived keys |
| `crates/tsunagi-core/src/storage/` | `state.sqlite`, `cache.sqlite`, directory ownership lock |
| `crates/tsunagi-core/src/discovery.rs` | candidate sources; test and static backends |
| `crates/tsunagi-core/src/proto/` | framing, message formats, membership handshake |
| `crates/tsunagi-core/src/net.rs` | iroh endpoint adapter and observability snapshots |
| `crates/tsunagi-core/src/agent/` | agent lifecycle, per-network runtimes, sessions, events, status |
| `crates/tsunagi-core/src/state/` | signed records that outlive a session, their merge rules and address allocation |
| `crates/tsunagi-core/src/overlay/` | the one interface an agent owns: provisioning, the TUN, routing, source checks, broadcast host rules |
| `crates/tsunagi-core/src/dataplane/` | the protocol contract and the authenticated packet transport |
| `crates/tsunagi-core/src/dns/` | the DNS view of a network: zone, server, resolver publication |
| `crates/tsunagi-wg-quic/` | the `wg-quic` protocol: its keys, its announcement, its tunnels |
| `crates/tsunagi-cli/` | the command line agent; the only place that owns a runtime, a logger and signals |
| `crates/tsunagi/` | the command line agent, built as `tsng`; the only place that owns a runtime, a logger and signals |
| `tests/` | integration tests; `tests/common/` is the shared harness |
Add abstractions only at real substitution or testing boundaries. Do not add a
Cargo.lock
Generated
+2798 -115
View File
File diff suppressed because it is too large. Load diff
Cargo.toml
+5 -3
View File
@@ -4,18 +4,20 @@
[workspace]
resolver = "3"
members = ["crates/*"]
exclude = ["vendor/mainline"]
exclude = ["vendor/mainline", "vendor/iroh"]
# Mainline 8.0.0 treats Windows UDP read timeouts as socket failures.
# Keep the small receive-loop correction reproducible until an upstream fix.
[patch.crates-io]
mainline = { path = "vendor/mainline" }
# Backport bounded path retries until an iroh release includes the OOM fix.
iroh = { path = "vendor/iroh" }
[workspace.package]
edition = "2024"
rust-version = "1.91"
license = "MIT OR Apache-2.0"
repository = "https://github.com/tsunagi-net/tsunagi"
license = "WTFPL"
repository = "https://github.com/house-of-vanity/tsunagi"
[workspace.lints.rust]
missing_docs = "warn"
LICENSE
+13
View File
@@ -0,0 +1,13 @@
DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
Version 2, December 2004
Copyright (C) 2004 Sam Hocevar <sam@hocevar.net>
Everyone is permitted to copy and distribute verbatim or modified
copies of this license document, and changing it is allowed as long
as the name is changed.
DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
0. You just DO WHAT THE FUCK YOU WANT TO.
README.md
+201 -718
View File
@@ -1,766 +1,249 @@
# tsunagi
A proof-of-concept Rust library for **small private mesh networks** — a handful
of friends, home machines, a few servers. Units to dozens of participants, not
thousands.
The end user configures exactly two things:
```text
network_name
secret # one shared secret; "password" and "secret" mean the same value
```
From those, every agent independently derives the same network space. There is
no central server, no network owner with special powers, no registration and no
majority vote. Anyone who knows the parameters can join; nobody has to trust
anybody else.
## What this proof of concept actually does
A working library with **real iroh connections** and integration tests:
- persistent device identity stored in SQLite, stable across restarts;
- several independent networks at once in one agent;
- deterministic network identity derived from name + secret;
- candidates supplied by a replaceable discovery component;
- automatic first contact through the public Mainline DHT (BEP44);
- real iroh connections plus an explicit mutual proof of network membership;
- a small versioned control protocol: handshake, hostname/capability
announcement, ping/pong;
- automatic reconnect with bounded exponential backoff and jitter;
- status snapshots, an event stream and honest diagnostics;
- configuration restored after a restart;
- correct behaviour when the disposable cache is missing or corrupt;
- a **WireGuard data plane**, in userspace: its own key per network, an IPv6
overlay with deterministically derived addresses and optional IPv4, real
tunnels carried over iroh, and address ownership enforced rather than
believed;
- a **command line agent**, `tsunagi`, with a local control socket.
### What it deliberately does **not** do
Not implemented, and not pretended to be: DNS, routing through
intermediate participants, a full CRDT, dynamically loaded plugins, a system
service, a complete CLI, or a local control socket. Snapshot synchronisation
and signed revocations are designed for but not implemented — see
[docs/sync-model.md](docs/sync-model.md). The WireGuard plugin's own limits,
including that its system backend is Linux-only, are in
[docs/wireguard.md](docs/wireguard.md#limits-and-future-work).
**Control and data are separated logically, not physically.** Both ride on
iroh, on different ALPNs and different connections, so the data plane inherits
iroh's hole punching and relay fallback instead of reimplementing them — while
the control protocol still knows nothing about packets and can keep a different
transport underneath it later. Filtering user traffic remains the operating
system's and the user's responsibility, not this library's.
## Requirements
- Rust 1.91 or newer (iroh 1.2 requires it), edition 2024. Pinned dependencies
in `Cargo.lock`.
- No internet, no public DHT, no public relay, no administrator rights and no changes
to OS network settings are needed to build or test.
- WireGuard runs in userspace (boringtun): **no kernel module and no `wg`
tool**. Only creating a real network interface needs `CAP_NET_ADMIN`, and
`--no-tun` skips even that.
## Trying it on two machines
On the first machine:
```bash
cargo build --release
./target/release/tsunagi status # this device, the agent, and this host
./target/release/tsunagi network # the networks this device belongs to
```
Two commands do the work, and they are separate on purpose: `up` runs the
agent — one per device, for as long as you want it — and `join` decides
what it belongs to, at any time, while it runs.
```bash
# Terminal one: the agent. It prints its endpoint id and then serves.
./target/release/tsunagi up
# Terminal two: make a network. With no --secret it invents one and prints
# it, along with the line to send the other machine.
./target/release/tsunagi join --network lab
```
On the second machine, start its agent and join the same network with the
secret that was printed:
```bash
./target/release/tsunagi up
./target/release/tsunagi join --network lab --secret tsn1...
```
**Name and secret are enough for the first meeting.** Mainline DHT discovery
is enabled by default. Each active network publishes this device's endpoint
and searches until the first authenticated connection. While connected it
only republishes; after 60 seconds without any authenticated connection it
searches again. Publication is approximately every five minutes, and records
older than 15 minutes are ignored. DHT storage nodes may retain them longer.
Use `--no-dht` to disable DHT, or `--peer <endpoint-id>` for optional manual
bootstrap. `--reach local` disables public DHT regardless of the DHT flag.
Endpoint records are public, signed, and contain no network secret. Discovery
is not authentication: every connection must still prove network membership.
Details and failure behavior are in [docs/mainline-dht.md](docs/mainline-dht.md).
**One introduction is enough for the whole network.** Members tell each
other about the members they know, and every author of a signed record is
somebody to try, so a device pointed at one member ends up talking to all
of them rather than to the one that happened to be on its command line.
What travels is a candidate — an address somebody has seen — and it is
authenticated by the handshake like any other; being introduced grants
nothing. It is deliberately the same shape a lookup in a distributed hash
table would return, so that is a source to add beside this one rather than
a redesign.
An agent with nobody to contact says so in `status` rather than sitting
there looking patient.
Within a few seconds both print something like:
```text
+ peer b47c958462 connected over direct rtt=Some(4.5ms)
+ data link to b47c958462 for wg-quic: direct via 192.0.2.7:41234, datagram 1382
--- status ---
control: 1 peer(s), 0 dial failure(s), 0 handshake failure(s)
wg-quic: tsun0 on 10.13.37.69/24 mtu 1280, 1/1 tunnel(s) established
4jO4kx9Z 10.13.37.237 handshake 3s ago tx=0 rx=0 dropped=0 path=direct via 192.0.2.7:41234
```
`1/1 tunnel(s) established` means a real WireGuard handshake completed.
## Checking that it works
From another shell on either machine:
```bash
tsunagi status
```
```text
endpoint 7d76ccbbc21bf30767e14422c0494740a2cecc02aa9f82d5b8d57bdae350e7fc
hostname tsunagi-7d76ccbbc2
bound 0.0.0.0:41641
network lab (z2o4qwrvnj3zb6st2aoqg4abf342j662q2ujttqrsmz22argk2ba) active
peer b345d5271b tsunagi-b345d5271b Direct rtt 24ms
overlay tsunz2o4qwrvnj3 fd09:…:c1c6/64 and 100.110.49.177 mtu 1280 1/1 tunnel(s) up
SDsEb/WF fd09:…:c4b / 100.65.243.53 handshake 4s ago tx 0 rx 0 Direct via Ip(…)
```
`1/1 tunnel(s) up` and a recent handshake mean the tunnel is live. Then send
real traffic to the peer's overlay address:
```bash
ping6 fd09:…:c4b # or
ping 100.65.243.53
```
`tx` and `rx` in the status should start moving.
**Addresses are allocated and then remembered.** The default range is
`10.13.37.0/24`; the first member to join settles it and later members adopt
what they find, so `--ipv4-range` only matters for whoever starts the network:
```bash
tsunagi up --ipv4-range 10.44.0.0/16
tsunagi up --ipv4-range none # no data plane at all
```
One agent has one interface, so two of its networks cannot both use that
range. The second takes the range **derived from its own network id**: not
picked locally — every member derives the same one from something they all
already have — so it is an agreement rather than a guess, and a device in
several networks gets an address in each.
An address is claimed with a record signed by that member's persistent device
key, stored, and merged between every replica. A member that disappears for a
month comes back to the same address, because the claim outlived the session.
No vote is involved — see
[docs/wireguard.md](docs/wireguard.md#ipv4-allocated-signed-and-kept) and
[docs/sync-model.md](docs/sync-model.md).
Because the address is allocated at run time rather than derived, it is not
known until the agent has started and agreed with its peers. The agent then
assigns it to the interface itself.
## Levels
The command line is split the way the design is. A bare `tsunagi up` needs
only a network name and a secret; everything else sits under the level it
belongs to, which `--help` shows as two sections:
* **System** — what the agent itself does: how it reaches peers (`--reach`),
the one overlay interface it owns (`--interface`, `--mtu`, `--no-tun`), the
address range (`--ipv4-range`), and the local resolver (`--no-dns` to disable).
* **Transport** — which protocols carry packets (`--protocol`, a list) and
their own settings (`-o key=value`, or `-o protocol:key=value`).
```bash
tsunagi protocols # what this build can carry packets with
```
```text
wg-quic (wire version 4)
what WireGuard's cryptography carried in iroh's QUIC datagrams
-o keepalive=SECONDS keeps a tunnel and its link warm through a NAT
-o mtu=BYTES largest packet a tunnel will carry, at least 576
```
Each protocol declares its own settings, so the agent can list them without
knowing anything about the protocol, and a setting no selected protocol takes
is refused rather than ignored. `--protocol none` runs the control plane by
itself.
A pair of peers uses a protocol they both have **at the same wire version**.
That is not the software version: two peers on different builds carry traffic
for each other for as long as the bytes between them have not changed. A peer
with nothing in common keeps its control plane — messages and signed state
still flow — and simply has no data plane, which `tsunagi status` shows as a
session with no agreed protocol.
## Names
DNS is enabled by default and serves a local zone for **every network this device is in**,
each named after the network, so members can be reached by name instead of
by address:
```bash
tsunagi up # DNS is already enabled
dig @127.0.0.1 -p 5354 music.lab
```
An explicit choice is remembered across restarts. `--no-dns` or `tsunagi dns off`
disables it; `--dns` or `tsunagi dns on` enables it again. Existing saved opt-outs
remain respected. The `dns on` and `dns off` commands also take effect immediately
on a running agent. `--dns` and `--no-dns` cannot be combined.
The zone of a network is its name — `--dns-zone` is gone, because with
several networks there is no single zone to name. A network name may contain
dots, so `--network lab.internal` is how you get `music.lab.internal`.
Names come from signed state, which is the point: **a member that is
switched off still resolves**, because its claim outlived the session.
The answers are IPv4 addresses, because that is what the overlay is. The
*questions* are taken on `127.0.0.1` and `[::1]`, over UDP and TCP, so a
resolver reaches it over whichever family it uses; both are published to the
system resolver together. Loopback and nothing else: the zones are a view
for the host running the agent, and binding an overlay address would put
them in front of the whole mesh — an agent in two networks would then answer
one network's questions about the other's names.
The zone is the network name, and it is yours to choose. A name that
shadows a real public domain is reported and then used: a network called
`ru` warns that every public `.ru` name becomes unreachable from this host,
and then does it. Anything that collides with nothing — which is most names
— is said nothing about, because a warning that fires on every private name
anybody picks is how people learn to ignore warnings.
On Linux the agent tells systemd-resolved to send questions for that suffix
here, over D-Bus, scoped to the overlay interface and as a *routing* domain
so it never becomes the resolver for anything else. resolved drops the whole
setting when the interface goes, and the interface goes with the agent.
Only an interface the agent created, though. Under `--no-tun` there is no
host interface at all, and the agent says so and serves the zone on
loopback rather than configuring whatever else on the host happens to share
the name.
That last step needs permission that `CAP_NET_ADMIN` does not give:
systemd-resolved asks polkit, and polkit decides by **user**, not by
capability, so there is no way for the agent to arrange it from inside. On
a desktop the refusal reads `Interactive authentication required`.
Running as a system service is enough. Otherwise the agent prints the rule
that grants it — the four actions it calls and nothing else — ready to
paste:
```bash
sudo tee /etc/polkit-1/rules.d/50-tsunagi-resolved.rules > /dev/null <<'RULE'
polkit.addRule(function(action, subject) {
var allowed = [
"org.freedesktop.resolve1.set-dns-servers",
"org.freedesktop.resolve1.set-domains",
"org.freedesktop.resolve1.set-default-route",
"org.freedesktop.resolve1.revert"
];
if (allowed.indexOf(action.id) >= 0 && subject.user == "YOUR-USER") {
return polkit.Result.YES;
}
});
RULE
```
**Without it the server still runs** — `tsunagi status` prints where it is
listening and the exact `dig` line — so the automatic part is missing, not
the feature. The refusal is said once rather than on every pass, and retried
slowly, because nothing but a person will change it.
The server is authoritative for its zone and nothing else. No recursion, no
forwarding, no cache: pointing a resolver at it can never make it a route to
the outside.
## Privileges
On Linux the agent **manages its own overlay interface**. It creates the TUN
interface, sets the MTU, brings it up and assigns both overlay addresses, all
over netlink in process — no `ip` invocation, no shell, nothing that a
remote peer could influence.
That needs `CAP_NET_ADMIN`, granted once:
```bash
sudo setcap cap_net_admin+p /usr/local/bin/tsunagi
```
`+p` rather than `+ep`: the capability is then *permitted* but not
*effective*, and the agent raises it only around the handful of netlink calls
that need it — a few milliseconds at startup, and again if its address
allocation changes. Everything else, including every byte from the network,
is handled with it lowered. `+ep` works too; the agent lowers it on the way
in.
`tsunagi status` says which of these applies on the host it runs on, along
with what the agent is doing. It grades each finding: **ok** for what works,
**warn** for what the agent runs without and you can fix from the line it
prints, **FAIL** for what it cannot work around. The words carry the grade as
well as the colour, so the report reads the same piped to a file or on a
terminal without colour, and it honours `NO_COLOR`.
Members are listed online first, then the ones that are away. A member that
is away is reported plainly rather than flagged: in a mesh of laptops it is
the ordinary condition, not a fault. The signed state is what makes that
sayable — it remembers who belongs while they are gone, so the report can say
"offline, 10.13.37.99 still reserved for it" instead of leaving a
dial-failure counter to imply it. Counters are history and are never graded:
a peer that left and came back should not leave the report looking broken.
The other two commands split along the line the system itself draws. `id` is
this **device**: the key it signs with and the name it answers to. `network`
is what it **belongs to**: which networks, their secrets, joining and
leaving. A device outlives every network it is in, and a network outlives
any device in it, so a command that mixed them had to be read twice.
Every item takes the same shape, so there is nothing to remember: name it to
see it, name it with a value to change it.
```
tsunagi id everything about this device
tsunagi id hostname the name it answers to
tsunagi id hostname mango change it
tsunagi id key the key it signs with
tsunagi id key rotate replace that key
tsunagi network the networks this device belongs to
tsunagi network join -n lab -s tsn1… join one; adds it to a running agent
tsunagi network join -n lab resume one this device has, or make it on the spot
tsunagi network stop lab stop serving it, keeping everything
tsunagi network start lab serve it again, from where it left off
tsunagi network leave lab give up the address and name, then forget it
tsunagi network secret the secret of each joined network
tsunagi network secret lab just that one, for copying
tsunagi network secret generate a fresh secret for a network that does not exist yet
tsunagi dns whether the local resolver is serving, and what
tsunagi dns on start it, now and after every restart
tsunagi dns off stop it, now and after every restart
```
**A network without a secret makes one.** `tsunagi join --network lab`
resolves a bare name in the obvious way: if this device is already in
exactly one network called `lab`, that one — so the name alone resumes what
you have; if it is in none, a fresh random secret, printed in full with the
line to send the other machine:
```
joined `lab` (k2on43wadbi5x267vp6z3ogkm7nbjedfdoxyauxhtxhwqrprylba)
<p align="center">
<img src="dist/linux/tsunagi.svg" width="96" alt="">
</p>
<h1 align="center">tsunagi</h1>
<p align="center">
<b>A mesh VPN with no server.</b><br>
Pick a network name, share a secret, and your machines are on one private network.
</p>
<p align="center">
<a href="https://github.com/house-of-vanity/tsunagi/releases/latest"><img alt="release" src="https://img.shields.io/github/v/release/house-of-vanity/tsunagi?include_prereleases"></a>
<a href="https://github.com/house-of-vanity/tsunagi/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/house-of-vanity/tsunagi/actions/workflows/ci.yml/badge.svg"></a>
<a href="LICENSE"><img alt="license" src="https://img.shields.io/badge/license-WTFPL-blue"></a>
</p>
```console
laptop$ tsng join -n home
joined `home`
secret tsn1u7c…
Run this on the other machine:
server$ tsng join -n home -s tsn1u7c…
server$ tsng id hostname server
tsunagi join --network lab --secret tsn1u7c…
Its agent has to be running. If it is not:
tsunagi up --peer 91e83a6e2b7a…
laptop$ ssh server.home
```
Two networks of one name is a thing that happens — a mistyped secret makes
one — so joining says whether the network was already here, and warns when
another configured network answers to the same name. A name is a label and
the id is the identity.
No account, no coordination server, no admin console, no keys to approve.
Anyone who knows the **name** and the **secret** is in. Nobody else can find
the network, let alone enter it.
That is the ad-hoc case: one person makes a network and sends the command
round. The secret is printed *only* when the agent invented it — there is
nowhere else to read it from — and never when it was supplied, because then
it is already yours. Two networks of one name and no secret is the one case
with no answer, and it says so instead of choosing.
## Why
A secret is printed by `network secret` and nowhere else — not by `id`, not
by `status`, not in a log, a `Debug` rendering or anything sent to a peer.
Asking for it is deliberate, because these reports get pasted into chats.
Tailscale, Headscale and ZeroTier are great, but each needs a control server:
theirs, or one you have to run and keep alive. tsunagi doesn't. Machines find
each other through the public BitTorrent DHT, connect directly (hole punching,
with relay fallback), and keep the network state among themselves.
**`up` takes no network at all** if you would rather decide later: it
brings up the agent and whatever it is already configured for, and waits.
That is the shape of a daemon in one terminal and `tsunagi network join`
in another.
| | Tailscale | Headscale | ZeroTier | **tsunagi** |
| ---------------------- | ------------------ | ---------------- | ------------------- | ------------------- |
| Control server | hosted by vendor | you run it | hosted or self-run | **none** |
| Account / sign-in | yes | yes | yes | **no** |
| To join a network | sign in, get approved | pre-auth key | network ID, approval | **name + secret** |
| Direct connections | yes | yes | yes | yes |
| Exit nodes | yes | yes | via routes | yes |
`network join` is also the answer to a question `up` cannot: a state
directory belongs to one live agent, so a second `tsunagi up` cannot add a
network to the one already running. This adds it over the control socket and
it starts at once. With no agent running it is written to the configuration
and starts with the next `up`.
## Features
The name is part of the signed state, so changing it revokes the previous
one: there is one record per author, a new version replaces the whole claim,
and no replica can keep the old name standing. Changing it while the agent
runs goes through the agent, which republishes and tells its peers straight
away.
- **Two things to know:** a network name and a secret. A secret is generated for you when you make a network.
- **Direct, encrypted links** between members (WireGuard cryptography), with relay fallback behind strict NATs.
- **Names for machines:** `server.home` just works, even while `server` is switched off. A name belongs to whoever claimed it first; a later device with the same name keeps working by address and is told so.
- **Exit nodes:** send all your internet traffic through any member that offers one.
- **Multi-hop:** members that can't reach each other directly are routed through a member that can, over whatever kind of link each hop has. A member reached that way is online, and shown as relayed; traffic moves to a direct link by itself the moment one comes up.
- **LAN games:** UDP broadcasts are relayed, so "LAN" lobbies show up across the mesh.
- **Several networks at once** on one device.
- **Tray app** for Windows, macOS and Linux; a command line for everything.
Replacing the signing key makes this device a different member, and it loses
the address and name the old key held — nothing can sign on a retired key's
behalf, and by design there is no authority that could overrule an author. So
the outgoing key signs a release for every network on its way out, which
frees them for whoever wants them next, and the whole thing commits at once.
## Install
`status` and `id` both prefer a running agent, which is live and
authoritative, and fall back to reading the state store when there is none.
Reading takes no directory lock, so neither has to wait for the agent it is
asking about — nor does either need one to be running.
Grab a build from the [latest release](https://github.com/house-of-vanity/tsunagi/releases/latest).
### It cleans up after itself
| Platform | How |
| --------------------- | -------------------------------------------------------------------------------- |
| **Windows** | Run `tsunagi-setup-<version>-x86_64.exe`. Installs the service, the tray and `tsng`. |
| **macOS** | `brew tap house-of-vanity/tap && brew install --cask tsunagi-gui`<br>CLI only: `brew install tsunagi` |
| **Debian / Ubuntu** | `sudo apt install ./tsunagi-gui_<version>_amd64.deb` (`tsunagi_…` for no tray) |
| **Arch** | `sudo pacman -U tsunagi-gui-<version>-1-x86_64.pkg.tar.zst` |
| **NixOS** | [Flake and module](#nixos) below |
| **Any other Linux** | Unpack the tarball, then `sudo ./install.sh && sudo systemctl enable --now tsunagi` |
The interface is tied to an open file descriptor and is deliberately **not**
made persistent, so the kernel removes it when the agent exits — on a clean
shutdown, on a panic, on `SIGKILL`, on power loss alike. Keeping it is what
would take an action; removing it is the default.
Linux packages run the agent as an unprivileged service user. Add yourself to
the `tsunagi` group (`sudo usermod -aG tsunagi $USER`, then log in again) to
use `tsng` and the tray without `sudo`.
If something is left behind anyway — an interface made by an older version's
manual recipe, or one from a run killed in the instant between creating it and
recording it — the next start **replaces it**, along with any stale addresses
it carried. Two things are never touched:
## Quick start
* an interface that is not a TUN, because the name colliding with somebody's
bridge is not a reason to destroy the bridge;
* a TUN that another process is holding open, because that is a working
overlay belonging to somebody else — most likely a second agent on this
host, which should be given a different `--wg-prefix`.
Both of those refuse with an explanation rather than guessing.
Two settings the manual recipe used to need are gone with it.
`keep_addr_on_down` existed only because an interface nobody held open lost
carrier and had its IPv6 addresses flushed, and `nodad` only because duplicate
address detection can never finish without carrier. An interface held open for
its whole life has carrier for its whole life.
### The MTU is 1280
The default interface MTU stays at 1280 on direct paths, relays and paths
inside another VPN. Large encrypted packets are split into smaller QUIC
datagrams and reassembled before reaching WireGuard. SSH and full-size TCP
segments therefore do not require a manual MTU override or MSS adjustment.
The original IP packet, including its DF bit, is preserved. See
[docs/wireguard.md](docs/wireguard.md#mtu) for limits and compatibility.
### Summary
| approach | agent runs as | notes |
|---|---|---|
| `setcap cap_net_admin+p` | ordinary user, one capability | recommended: nothing to prepare, nothing left behind. Lost on every rebuild or copy of the binary. |
| systemd service | `User=`, `AmbientCapabilities=CAP_NET_ADMIN` | the same, for an installed service |
| `sudo tsunagi up` | root | everything works, nothing is isolated |
| macOS root LaunchDaemon | root | the native macOS deployment; see below |
| `--no-tun` | ordinary user, no capabilities | tunnels run and handshake, traffic never reaches the OS |
On Windows, put `wintun.dll` beside the executable and start `tsunagi up`
from PowerShell or Command Prompt opened with **Run as administrator**.
Run commands controlling that agent (`join`, `status`, `dns`, `network`)
as the same Windows user with the same elevation. If Windows denies access
to the agent's control pipe, the command reports a permission error instead
of claiming the agent is absent or trying to edit its locked state.
Creating a TUN without sufficient privileges also explains how to restart
the agent with the required permissions.
On macOS the agent creates a `utun` interface, configures it with `ifconfig`
and `route`, and points the system resolver at the overlay by writing
`/etc/resolver/<network>` files (which carry a port, so the DNS server needs no
privileged socket). All of this needs **root**: macOS has no per-capability
grant like Linux, and a `utun` plus `/etc/resolver` are root-only. The kernel
assigns the interface its `utunN` name, so the name derived from the network id
is only a request and the real one is adopted once the interface is up.
Run it as root — `sudo tsunagi up` for a quick start, or install it as a root
**LaunchDaemon** for the native always-on deployment. As a daemon the agent runs
unprivileged commands (`join`, `status`, `dns`, `network`) over its control
socket like any other platform; the socket lives at `/var/run/tsunagi/agent.sock`
and the stores at `/var/db/tsunagi`, both fixed system paths so they do not
depend on whose shell started `sudo`. A ready-to-edit plist and install notes
are in [dist/macos/](dist/macos/). The broadcast route is installed; the
inbound-UDP firewall allowance (a `pf` anchor) is not yet, so `status` reports
that half as incomplete and names what to allow by hand. The decision logic that
says *what* to change is shared and tested on every platform.
## Checks
```bash
cargo fmt --all -- --check
cargo clippy --locked --workspace --all-targets -- -D warnings
cargo test --locked --workspace --all-targets
```sh
tsng join -n home # first machine: makes the network and prints a secret
tsng join -n home -s tsn1… # every other machine
tsng status # who is online, over which path
```
The whole suite runs offline on loopback. Set `TSUNAGI_TEST_LOG=tsunagi=debug`
to see agent logs while a test runs.
Every new network gets a stable pseudorandom `/24` in `10.0.0.0/8`, derived
from its network id. Its members agree on that range even when starting offline,
and each gets a name like `<host>.home`. Existing signed ranges are preserved.
There are also two runnable demos, which are demos and not substitutes for the
tests:
If two configured networks have overlapping ranges, the agent runs only one.
`tsng status`, `tsng network` and the tray show which network blocks the
other. `tsng network start <name-or-id>` or the tray’s “Switch to this network”
button suspends the conflicting networks. They resume automatically when the
selected network stops; the selection survives restarting the agent. Explicitly
stopping a suspended network cancels its automatic resume.
```bash
cargo run --example two_agents # control plane only
Machines are called `tsunagi-<id>` until you rename them: `tsng id hostname laptop`.
### Virtual services
```sh
tsng vip create home api
tsng vip port add home api tcp/443 --backend server1:6443 --backend server2:7443
tsng vip status home api --watch
```
Both run with no privileges and change nothing on the host.
Use `api.home:443`. Each client selects the available backend with the lowest
measured data-plane RTT. A VIP can have multiple independent TCP and UDP ports.
Every member may edit it; the tray's **VIP** button opens a draft editor with
explicit Save. [Commands, health, metrics and limits](docs/vip.md).
## Usage
### Exit nodes
```rust,no_run
use std::sync::Arc;
use tsunagi::config::{AgentConfig, StoragePaths, TransportPolicy};
use tsunagi::discovery::SharedMemoryDiscovery;
use tsunagi::identity::{NetworkName, NetworkSecret};
use tsunagi::proto::ControlMessage;
use tsunagi::{Agent, Result};
```sh
tsng network exit-node home on # on the machine that should be the exit
tsng exit-node # on another one: who offers an exit
tsng exit-node laptop # send everything through it
tsng exit-node off
```
// The library never starts a runtime, installs a logger, handles signals,
// forks, or calls process::exit. The binary owns all of that.
#[tokio::main]
async fn main() -> Result<()> {
let config = AgentConfig::new(StoragePaths::user_default()?)
.with_transport(TransportPolicy::N0Defaults)
.with_discovery(Arc::new(SharedMemoryDiscovery::new()));
The tray does the same with a checkbox. An exit node sees everything you send
through it, so use only members you trust.
[Details and per-OS notes](docs/exit-node.md).
let agent = Agent::spawn(config).await?;
The default data transport order is `wg,awg,wg-quic,tcp-tls`. AWG negotiates a
fresh masking profile and outer keys for each pair connection attempt, including
reentry after fallback. Sustained data loss triggers switching independently of
the iroh control connection. [Configuration and limits](docs/amnezia.md).
let name = NetworkName::new("kitchen-table")?;
let secret = NetworkSecret::generate(); // 32 random bytes
println!("share this: {}", secret.encode().as_str());
Change a network's protocol selection without editing its service or restarting:
let network = agent.join_network(&name, &secret).await?;
```sh
tsng protocols # available protocols and settings
tsng protocols home # this device's choice for home
tsng protocols home awg,wg-quic # prefer AWG, retain QUIC as fallback
tsng protocols home none # control plane only
```
let mut events = agent.subscribe();
tokio::spawn(async move {
while let Ok(event) = events.recv().await {
println!("{event:?}");
Exact network names, full ids and unique id prefixes work here. The choice is
local to this device, saved per network and applied immediately; an offline
change applies on the next start. In the tray, expand **protocols** on a network
tile to toggle protocols or change their preference order. Both endpoints need
a common protocol at the same wire version. `tsng up --protocol` supplies the
initial defaults for networks without a saved choice; it does not restrict
which built-in protocols can be enabled later.
Remove an obsolete offline device from a network's shared history:
```sh
tsng network forget-node home old-laptop
tsng network forget-node home 10.13.37.143
```
Networks accept exact names, full ids or unique id prefixes; nodes accept those
id selectors, exact hostnames or overlay IPv4 addresses. Ambiguous selectors
change nothing. Start the network first if it is stopped. In **Show devices**,
click **Forget** beside an offline node and confirm its identity.
Every member has the same cleanup right. The signed cleanup frees the observed
claim's name and IPv4 reservation, survives restarts and reaches offline replicas
when they reconnect. It does not ban the device: a newer signed claim can return.
Online nodes and the currently selected exit node cannot be forgotten.
[State and limits](docs/sync-model.md#offline-node-cleanup).
## Platforms
| Platform | Build | CLI `tsng` | Tray | Exit node: offer / use | Packages |
| ------------------------- | ------------ | :--------: | :--: | :--------------------: | ----------------------------------- |
| Linux x86_64 | release | ✔ | ✔ | ✔ / ✔ | `.deb`, `.pkg.tar.zst`, tarball, Nix |
| Linux aarch64 | release | ✔ | –¹ | ✔ / ✔ | `.deb`, `.pkg.tar.zst`, tarball, Nix |
| Linux armv7 | release | ✔ | –¹ | ✔ / ✔ | `.deb`, `.pkg.tar.zst`, tarball |
| macOS, Apple silicon | release | ✔ | ✔ | ✔ / ✔² | Homebrew, tarball |
| macOS, Intel | from source | ✔³ | ✔³ | ✔ / ✔²³ | – |
| Windows x86_64 | release | ✔ | ✔ | ✔ / ✔² | installer, zip |
<sup>¹ Not in the release; builds from source (and through Nix).
² Newer and less battle-tested than on Linux.
³ Builds from source, not covered by CI.</sup>
The Linux CLI is a single static binary: it runs on any distribution.
## NixOS
The repository is a flake with a package and a module.
```nix
# flake.nix
{
inputs.tsunagi.url = "github:house-of-vanity/tsunagi";
outputs = { nixpkgs, tsunagi, ... }: {
nixosConfigurations.myhost = nixpkgs.lib.nixosSystem {
modules = [
tsunagi.nixosModules.default
{
services.tsunagi = {
enable = true;
tray.enable = true; # optional, for desktops
users = [ "alice" ]; # may use tsng and the tray without sudo
};
}
});
for peer in agent.network_status(network).await?.connected_peers() {
agent
.send(network, peer, ControlMessage::Ping { seq: 1, payload: vec![] })
.await?;
}
agent.shutdown().await;
Ok(())
];
};
};
}
```
`TransportPolicy::LocalOnly` is the default, so a plain `AgentConfig::new` never
reaches the internet by accident. Opt into `DirectOnly` or `N0Defaults`
explicitly.
Just trying it: `nix run github:house-of-vanity/tsunagi#tsunagi-cli -- --help`.
`packages.<system>.tsunagi` (with the tray) and `tsunagi-cli` are also exported,
along with `overlays.default`.
### Through somebody in the middle
## Build from source
Tsunagi routes the overlay in userspace. A direct data link always wins;
otherwise the shortest available path can pass through several members.
Each authenticated member advertises only its own live, protocol-specific
transport links. The control plane collects these into a graph and replaces
the routing table when connectivity changes; announcements expire after
90 seconds without refresh.
Transit goes transport → router → transport, without entering WireGuard or
the intermediate host's TUN. The payload stays encrypted between the original
endpoints. TUN connects only the local OS to Tsunagi; the kernel needs only
the overlay route through that interface, with no IP forwarding configuration.
A 16-hop limit bounds loops while topology updates converge. Equal shortest
paths use a stable flow hash derived before encryption, so packets of a TCP
or UDP flow keep their path while topology is unchanged. Physical link changes
do not replace end-to-end WireGuard tunnels or change the 1280-byte default MTU.
`status` reports the hop count and first next hop for relayed paths.
The hot path reads an immutable routing snapshot and sends directly to a cached
transport handle. It takes no routing mutex, walks no graph and does not parse
the encrypted payload. See [routing.md](docs/routing.md) for the architecture,
limits and reproducible forwarding microbenchmark.
This wire format requires all members to upgrade together (control ALPN 3,
data ALPN 4); saved identities, network names, secrets and addresses survive.
### LAN game discovery
IPv4 UDP broadcast relay is enabled by default for each network. The IP router
sends one encrypted copy to each participating peer, including through multihop
paths. It supports `255.255.255.255` and the overlay subnet's broadcast address;
received broadcasts never trigger another fanout.
Rust 1.91 or newer.
```sh
tsunagi join -n games --no-broadcast
tsunagi join -n games --broadcast
tsunagi network broadcast games off
tsunagi network broadcast games on
cargo build --release --bin tsng # the agent and CLI
cargo build --release --bin tsunagi-tray # the tray (Linux needs GTK 3 and libayatana-appindicator)
```
The choice persists across restarts and a plain `join`. It can be changed while
the agent runs; `status` shows it for each network. While broadcast is on, the
agent routes `255.255.255.255` through the Tsunagi interface and allows inbound
UDP from the overlay range on it (Linux and Windows, needs the same privilege as
the interface), and removes both when it is turned off or the agent stops. Games
bound to a physical address still bypass it. Physical LAN capture/subnet sharing
is not implemented.
See [broadcast.md](docs/broadcast.md) for domain isolation and future LAN gateways.
Windows needs [Wintun](https://www.wintun.net) next to `tsng.exe`.
Per-platform notes: [Linux](dist/linux/README.md), [Arch](dist/arch/README.md),
[macOS](dist/macos/README.md), [Windows](dist/windows/README.md).
## How peers find each other
## How it works
Two different lookups are involved, and only one of them is this project's:
- Your name and secret derive a network identity. Members publish signed
contact records to the public Mainline DHT under it, and find each other there.
- Every connection proves knowledge of the secret before it's accepted.
Traffic between members is end-to-end encrypted, relays included.
- Addresses and names are claimed with each device's own key and replicated
between members. There is no one in charge, so nothing to be taken down.
**1. Resolving one endpoint's address — iroh's, and it works today.**
With `--reach relay` or `--reach direct`, iroh publishes a signed
record of this endpoint's addresses, keyed by its endpoint id, to the public
service run by Number 0 — "n0", the company behind iroh — at `dns.iroh.link`,
over pkarr and DNS, and resolves other endpoints the same way. That is why `--peer <endpoint-id>` works with no address attached:
iroh looks it up. None of that code is ours.
**2. Finding who is in a network — Mainline DHT by default.**
`NetworkDiscovery` supplies unverified candidates. The Mainline backend uses
a signing key derived from the network name and secret to publish signed
BEP44 records containing each writer's endpoint ID and addresses. Its 16
slots provide starting candidates, not a complete membership list or a limit
on network size. After the first authenticated connection, introductions and
signed state reveal the other members. `StaticBootstrap` (`--peer`) remains
available, and tests use either a local Mainline Testnet or in-memory discovery.
What this means in practice:
- With `relay` or `direct`, **your endpoint id and IP addresses are published
to a public third-party service** (Number 0's, unless you change it). They are not secret, and the network secret is
never published, but an observer of that service learns that your endpoint
exists and where it is. `--reach local` publishes nothing.
- With DHT enabled, endpoint records are also published to the public Mainline
DHT. They are signed, not encrypted; membership still requires the secret.
- A relay, when one is needed, sees the volume and timing of your traffic — not
its contents. The default relays are Number 0's, in the US, EU and
Asia-Pacific.
## Storage
Two physically separate SQLite files, placed wherever the library's
configuration says (`StoragePaths`). A future system service supplies its own
paths; tests always use temporary directories.
| file | holds | when damaged |
|----------------|--------------------------------------------------|-------------------------|
| `state.sqlite` | device identity, network configuration, hostname | clear error, never reset |
| `cache.sqlite` | address hints and other recoverable data | discarded and recreated |
The `wg-quic` protocol keeps its own keys in its own store under `wg-quic/`,
because a protocol's keys are neither the iroh identity nor the network
secret.
The command line agent puts everything under the platform's per-user
directories by default; `--state-dir` and `--cache-dir` override them.
One state directory belongs to one live agent, enforced with a real OS file
lock rather than an existence check. That directory *is* the identity: one
agent, one device key, one interface, and as many networks as you like on it
— `tsunagi network join` adds them to the agent that is already running,
which is why a second `tsunagi up` on the same directory is refused rather
than made to work. A second agent on the same host is a second identity, and
needs everything of its own: its own state and cache directories, its own
interface name and its own overlay range. The library owns no globals, so
several of them run side by side in one process as readily as in one host.
### Leaving, and starting over
Membership outlives a session, so it also has to be possible to end it.
```bash
tsunagi network # what this device belongs to
tsunagi network leave lab # give up the address and the name, then forget it
tsunagi wipe --yes # remove everything and be a stranger again
```
**Stopping is not leaving.** `stop` closes this network's sessions, takes
its address off the interface and keeps it from starting again, and that is
all: the configuration, the secret, the signed state and the protocol key
stay exactly as they are, nothing is announced, and to the others this
device is simply away — an ordinary condition they already handle, with its
address and name still reserved for it. `start` picks it up where it left
off. A network named on the `up` command line is started by that command
whatever its stored state, and the start-up banner says so rather than
letting a `stop` quietly come back.
`leave` is the other one, and it removes, locally: the configuration and
the secret, this network's signed records, its cached address hints and the
protocol key it used there. What it keeps is this author's version counter
for that network — a rejoin has to continue above the release, or every
replica would treat the new claim as stale. Everything about *other*
networks, and the device identity itself, is untouched.
`leave` publishes a signed `Release` **first**, while the agent is running and
its sessions are up, so the address and the name are freed for the others
instead of staying reserved to a member that has gone. They pass the tombstone
on, so a member that was away hears it from them. With no agent running,
nothing can sign or send it: the command says so and refuses, and `--offline`
drops the network locally while leaving the others holding the old claim. The
protocol key for that network goes too — rejoining is joining, not resuming.
`leave`, `stop`, `start`, `secret` and `broadcast` accept an exact stored network
name, its full id, or a unique id prefix (including a copied report ellipsis).
Names are case-sensitive. If several networks share a name, or a name matches
another network's id prefix, the command lists matching names and full ids and
changes nothing. Use a full id to choose explicitly; a full id always takes
precedence over a name. No secret is needed to select a configured network.
`wipe` removes both directories' contents: the device identity, every network,
every signed record and everything a protocol kept beside them. It refuses
while an agent is running, and refuses a directory with no `state.sqlite` in
it, so a mistyped `--state-dir` cannot take somebody's documents with it.
Without `--yes` it only says what it would remove. It is not a goodbye: nobody
is told, because after it there is no key left to sign anything with. Leave the
networks first if the addresses should be freed.
The secret is the only key, so make it a good one (the generated ones are).
The [threat model](docs/threat-model.md) says what this does and doesn't protect against.
## Documentation
- [docs/architecture.md](docs/architecture.md) — module boundaries and runtime.
- [docs/wireguard.md](docs/wireguard.md) — the WireGuard plugin: overlay
addressing, announcements, backends, reconciliation.
- [docs/protocol.md](docs/protocol.md) — identity derivation, framing, handshake.
- [docs/sync-model.md](docs/sync-model.md) — the planned signed-state model and
what is deliberately not built yet.
- [docs/threat-model.md](docs/threat-model.md) — threat model and known limits.
- [docs/testing.md](docs/testing.md) — what the suite covers and what it does not.
- [AGENTS.md](AGENTS.md) — rules for anyone (human or agent) changing this repo.
## Security in one paragraph
Membership is proved by an HMAC over a transcript keyed by a value derived from
the shared secret, bound to the specific iroh connection through the TLS
exporter, to the network id, to both endpoint identities and to distinct role
labels. This targets high-entropy secrets: there is no PAKE here, so a short
human passphrase is guessable offline by anyone who can reach the handshake.
Anyone who knows the secret is a full participant and can create many
identities. Read [docs/threat-model.md](docs/threat-model.md) before relying on
any of this.
[Architecture](docs/architecture.md) ·
[Protocol](docs/protocol.md) ·
[WireGuard data plane](docs/wireguard.md) ·
[Pairwise Amnezia transport](docs/amnezia.md) ·
[Routing](docs/routing.md) ·
[Exit nodes](docs/exit-node.md) ·
[Virtual services](docs/vip.md) ·
[Threat model](docs/threat-model.md) ·
[Contributing](AGENTS.md)
## Licence
MIT OR Apache-2.0.
[WTFPL](LICENSE): do what the fuck you want to.
crates/tsunagi-awg/Cargo.toml
+30
View File
@@ -0,0 +1,30 @@
[package]
name = "tsunagi-awg"
version = "0.1.0"
description = "Ephemeral pairwise Amnezia transport for Tsunagi."
edition.workspace = true
rust-version.workspace = true
license = "WTFPL AND Apache-2.0"
repository.workspace = true
[dependencies]
tsunagi-core = { path = "../tsunagi-core" }
tsunagi-wg-quic = { path = "../tsunagi-wg-quic" }
tsunagi-wireguard = { path = "../tsunagi-wireguard" }
iroh.workspace = true
tokio = { workspace = true, features = ["net", "time", "sync", "macros"] }
serde.workspace = true
postcard.workspace = true
bytes.workspace = true
tracing.workspace = true
rand = "0.10"
zeroize = { version = "1.9", features = ["derive"] }
boringtun = { version = "0.7.1", default-features = false }
chacha20 = "0.9"
[dev-dependencies]
tsunagi-core = { path = "../tsunagi-core", features = ["testing"] }
tempfile.workspace = true
[lints]
workspace = true
crates/tsunagi-awg/LICENSE-AmneziaWG-RS
+201
View File
@@ -0,0 +1,201 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright 2026 Friderika Coulson (NanoBee)
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
crates/tsunagi-awg/UPSTREAM.md
+21
View File
@@ -0,0 +1,21 @@
# Amnezia packet engine
`src/masking.rs` is adapted from `src/core/amnezia.rs` in
https://github.com/RunaRoo/AmneziaWG-RS at
`51579a7d6e8882d1d8d6d00822f09e18a9423b73`, under Apache-2.0.
The upstream license is retained in `LICENSE-AmneziaWG-RS`.
The adaptation replaces configuration-file loading with bounded typed pair
profiles, updates RNG calls to rand 0.10, removes an unnecessary unwrap, and
does not embed upstream's Node, global executors, OS interface operations or
detached tasks. Noise/WireGuard is supplied by boringtun, already used by
Tsunagi. Each pair owns one socket and one cancellable packet-engine task.
This is Tsunagi AWG wire version 1, not a claim of compatibility with external
Amnezia servers. The masking engine substitutes message headers outside the
standard WireGuard state machine and restores them before Noise validation.
There are no remote file paths, shell hooks, or persisted pair profiles.
The adapted masking source remains Apache-2.0. Newly written Tsunagi code
follows the repository's WTFPL. This package declares both licenses in Cargo
metadata; the workspace license alone does not describe its copied source.
crates/tsunagi-awg/src/lib.rs
+16
View File
@@ -0,0 +1,16 @@
//! Pairwise, ephemeral Amnezia packet transport with independent end-to-end tunnels.
//! All negotiation travels over authenticated Tsunagi control sessions.
mod masking;
mod plugin;
pub mod profile;
mod session;
mod stun;
mod transport;
pub use plugin::AwgPlugin;
pub use transport::{AwgTransport, PairOverview};
/// Protocol identifier and default second-choice direct transport.
pub const AWG_PROTOCOL: &str = "awg";
/// Tsunagi AWG wire version, independent of software releases.
pub const AWG_WIRE_VERSION: u16 = 1;
crates/tsunagi-awg/src/masking.rs
+336
View File
@@ -0,0 +1,336 @@
// Adapted from RunaRoo/AmneziaWG-RS, commit 51579a7d6e8882d1d8d6d00822f09e18a9423b73.
// Apache-2.0; see LICENSE-AmneziaWG-RS and UPSTREAM.md.
#![allow(dead_code, missing_docs)]
// AmneziaWG configuration, header protection, and traffic shaping.
use bytes::{Bytes, BytesMut};
use chacha20::{
ChaCha20,
cipher::{KeyIvInit, StreamCipher},
};
use rand::RngExt;
pub const HEADER_PROTECTION_NONCE_SIZE: usize = 12;
pub const DEFAULT_UDP_WINDOW: usize = 500;
#[derive(Clone, Copy, Debug, Default, Eq, PartialEq)]
pub struct Range {
pub min: u32,
pub max: u32,
}
impl Range {
pub fn contains(&self, val: u32) -> bool {
val >= self.min && val <= self.max
}
pub fn random(&self) -> u32 {
if self.min >= self.max {
self.min
} else {
rand::rng().random_range(self.min..=self.max)
}
}
pub fn is_zero(&self) -> bool {
self.min == 0 && self.max == 0
}
}
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub enum AwgKind {
Initiation,
Response,
Cookie,
Transport,
}
impl AwgKind {
pub fn fixed_size(self) -> usize {
match self {
Self::Initiation => 148,
Self::Response => 92,
Self::Cookie => 64,
Self::Transport => 32,
}
}
fn standard_type(self) -> u32 {
match self {
Self::Initiation => 1,
Self::Response => 2,
Self::Cookie => 3,
Self::Transport => 4,
}
}
}
#[derive(Debug)]
pub struct DecodedPacket {
pub kind: AwgKind,
pub message: Bytes,
pub wire_len: usize,
}
pub struct AmneziaConfig {
pub jc: u32,
pub jmin: usize,
pub jmax: usize,
pub s1: usize,
pub s2: usize,
pub s3: usize,
pub s4: usize,
pub h1: Range,
pub h2: Range,
pub h3: Range,
pub h4: Range,
pub i1: String,
pub i2: String,
pub i3: String,
pub i4: String,
pub i5: String,
pub header_protection_key: Option<[u8; 32]>,
pub content_padding_addition: Range,
pub random_trailers: bool,
pub mtu: usize,
}
impl AmneziaConfig {
pub fn random_header(&self, range: &Range) -> u32 {
range.random()
}
pub fn random_padding(&self, len: usize) -> Vec<u8> {
let mut padding = vec![0u8; len];
if len > 0 {
rand::fill(&mut padding);
}
padding
}
fn shape(&self, kind: AwgKind) -> (usize, Range) {
match kind {
AwgKind::Initiation => (self.s1, self.h1),
AwgKind::Response => (self.s2, self.h2),
AwgKind::Cookie => (self.s3, self.h3),
AwgKind::Transport => (self.s4, self.h4),
}
}
pub fn encode(&self, kind: AwgKind, payload: Bytes, udp_window: usize) -> Bytes {
let (padding_len, _) = self.shape(kind);
let trailer_len = if kind != AwgKind::Transport && self.random_trailers {
let base = padding_len.saturating_add(payload.len());
let room = udp_window.max(DEFAULT_UDP_WINDOW).saturating_sub(base);
if room == 0 {
0
} else {
rand::rng().random_range(0..room)
}
} else {
0
};
let mut out = BytesMut::zeroed(padding_len + payload.len() + trailer_len);
if padding_len > 0 {
rand::fill(&mut out[..padding_len]);
}
out[padding_len..padding_len + payload.len()].copy_from_slice(&payload);
if trailer_len > 0 {
rand::fill(&mut out[padding_len + payload.len()..]);
}
if self.header_protection_key.is_some() {
let protected_len = if kind == AwgKind::Transport {
16
} else {
kind.fixed_size()
};
self.apply_header_cipher(&mut out, padding_len, protected_len);
}
out.freeze()
}
pub fn decode(&self, datagram: &Bytes) -> Option<DecodedPacket> {
if datagram.len() < 4 {
return None;
}
for kind in [
AwgKind::Initiation,
AwgKind::Response,
AwgKind::Cookie,
AwgKind::Transport,
] {
let (padding_len, headers) = self.shape(kind);
let minimum = padding_len + kind.fixed_size();
let size_ok = match kind {
AwgKind::Transport | AwgKind::Initiation => datagram.len() >= minimum,
AwgKind::Response | AwgKind::Cookie if self.random_trailers => {
datagram.len() >= minimum
}
AwgKind::Response | AwgKind::Cookie => datagram.len() == minimum,
};
if !size_ok || datagram.len() < padding_len + 4 {
continue;
}
let protected_header =
u32::from_le_bytes(datagram[padding_len..padding_len + 4].try_into().ok()?);
let candidate = if self.header_protection_key.is_some() {
protected_header ^ self.header_type_hash(datagram)?
} else {
protected_header
};
if !headers.contains(candidate) {
continue;
}
let message_len = match kind {
AwgKind::Transport | AwgKind::Initiation => datagram.len() - padding_len,
AwgKind::Response | AwgKind::Cookie => kind.fixed_size(),
};
let mut payload = BytesMut::from(&datagram[padding_len..padding_len + message_len]);
if self.header_protection_key.is_some() {
let protected_len = if kind == AwgKind::Transport {
16
} else {
kind.fixed_size()
};
self.apply_header_cipher_with_nonce(
&mut payload,
0,
protected_len,
&datagram[..HEADER_PROTECTION_NONCE_SIZE],
);
}
return Some(DecodedPacket {
kind,
message: payload.freeze(),
wire_len: datagram.len(),
});
}
None
}
pub fn prepare_transport_plaintext(&self, packet: Bytes, udp_window: usize) -> Bytes {
if packet.is_empty() {
return packet;
}
let requested = if !self.content_padding_addition.is_zero() {
self.content_padding_addition.random() as usize
} else if self.random_trailers {
let wire_without_padding = self.s4 + AwgKind::Transport.fixed_size() + packet.len();
let room = udp_window
.max(DEFAULT_UDP_WINDOW)
.saturating_sub(wire_without_padding);
if room == 0 {
0
} else {
rand::rng().random_range(0..room)
}
} else {
let aligned = (packet.len() + 15) & !15;
aligned.saturating_sub(packet.len())
};
let padding = self.clamp_padding_to_mtu(packet.len(), requested);
if padding == 0 {
return packet;
}
let mut out = BytesMut::with_capacity(packet.len() + padding);
out.extend_from_slice(&packet);
out.resize(packet.len() + padding, 0);
out.freeze()
}
pub fn transport_content_length(&self, plaintext: &[u8]) -> Option<usize> {
if plaintext.is_empty() || plaintext[0] == 0 {
return Some(0);
}
match plaintext[0] >> 4 {
4 if plaintext.len() >= 20 => {
let total = u16::from_be_bytes([plaintext[2], plaintext[3]]) as usize;
(total >= 20 && total <= plaintext.len()).then_some(total)
}
6 if plaintext.len() >= 40 => {
let payload = u16::from_be_bytes([plaintext[4], plaintext[5]]) as usize;
let total = 40 + payload;
(total <= plaintext.len()).then_some(total)
}
_ => None,
}
}
fn clamp_padding_to_mtu(&self, packet_len: usize, requested: usize) -> usize {
if self.mtu == 0 {
return requested;
}
let normalized = if packet_len > self.mtu {
packet_len % self.mtu
} else {
packet_len
};
requested.min(self.mtu.saturating_sub(normalized))
}
fn header_type_hash(&self, datagram: &Bytes) -> Option<u32> {
if datagram.len() < HEADER_PROTECTION_NONCE_SIZE {
return None;
}
let mut hash = [0u8; 4];
self.apply_keystream(&mut hash, &datagram[..HEADER_PROTECTION_NONCE_SIZE]);
Some(u32::from_le_bytes(hash))
}
fn apply_header_cipher(&self, packet: &mut [u8], offset: usize, len: usize) {
if packet.len() < HEADER_PROTECTION_NONCE_SIZE || packet.len() < offset + len {
return;
}
let nonce: [u8; HEADER_PROTECTION_NONCE_SIZE] =
match packet[..HEADER_PROTECTION_NONCE_SIZE].try_into() {
Ok(nonce) => nonce,
Err(_) => return,
};
self.apply_header_cipher_with_nonce(packet, offset, len, &nonce);
}
fn apply_header_cipher_with_nonce(
&self,
packet: &mut [u8],
offset: usize,
len: usize,
nonce: &[u8],
) {
if packet.len() < offset + len {
return;
}
self.apply_keystream(&mut packet[offset..offset + len], nonce);
}
fn apply_keystream(&self, data: &mut [u8], nonce: &[u8]) {
let Some(key) = self.header_protection_key.as_ref() else {
return;
};
if nonce.len() != HEADER_PROTECTION_NONCE_SIZE {
return;
}
let Ok(nonce) = <&[u8; HEADER_PROTECTION_NONCE_SIZE]>::try_from(nonce) else {
return;
};
let mut cipher = ChaCha20::new(key.into(), nonce.into());
cipher.apply_keystream(data);
}
}
impl Drop for AmneziaConfig {
fn drop(&mut self) {
use zeroize::Zeroize;
self.header_protection_key.zeroize();
}
}
crates/tsunagi-awg/src/plugin.rs
+132
View File
@@ -0,0 +1,132 @@
//! End-to-end encryption remains independent of physical Amnezia pair links.
use crate::{AWG_PROTOCOL, AWG_WIRE_VERSION, AwgTransport};
use iroh::EndpointId;
use std::sync::Arc;
use tsunagi::dataplane::transport::SharedLink;
use tsunagi::dataplane::{IpPlugin, PluginCapability, PluginContext, PluginError, ProtocolOption};
use tsunagi::state::Ipv4Range;
use tsunagi::{BoxFuture, NetworkId};
use tsunagi_wg_quic::{NetworkOverview, WireguardConfig, WireguardPlugin};
/// AWG capability and an independent end-to-end WireGuard engine.
/// Pair transport keys/profiles never enter its durable end-to-end key store.
#[derive(Debug)]
pub struct AwgPlugin {
inner: Arc<WireguardPlugin>,
transport: Arc<AwgTransport>,
}
impl AwgPlugin {
/// Uses standard tunnel MTU/keepalive options; pair ports and profiles are ephemeral.
pub const OPTIONS: &'static [ProtocolOption] = &[
ProtocolOption {
key: "keepalive",
value: "SECONDS",
help: "end-to-end tunnel keepalive; 0 disables it",
default: Some("25"),
},
ProtocolOption {
key: "mtu",
value: "BYTES",
help: "maximum overlay packet size",
default: Some("1280"),
},
ProtocolOption {
key: "stun",
value: "HOST:PORT|off",
help: "UDP public mapping discovery; disabled in local reach mode",
default: Some("stun.l.google.com:19302"),
},
];
/// Opens the end-to-end engine, retaining separate pair transport state.
pub async fn open(
mut config: WireguardConfig,
transport: Arc<AwgTransport>,
) -> Result<Arc<Self>, PluginError> {
config.protocol = AWG_PROTOCOL;
Ok(Arc::new(Self {
inner: WireguardPlugin::open(config).await?,
transport,
}))
}
/// Reports end-to-end tunnel health and traffic counters.
pub fn overview(&self, network: NetworkId) -> Option<NetworkOverview> {
self.inner.overview(network)
}
}
impl IpPlugin for AwgPlugin {
fn protocol_id(&self) -> &str {
AWG_PROTOCOL
}
fn protocol_version(&self) -> u16 {
AWG_WIRE_VERSION
}
fn options(&self) -> &'static [ProtocolOption] {
Self::OPTIONS
}
fn attach(&self, context: PluginContext) {
self.inner.attach(context);
}
fn local_capability(
&self,
network: NetworkId,
) -> Result<Option<PluginCapability>, PluginError> {
self.inner.local_capability(network).map(|cap| {
cap.map(|mut cap| {
cap.protocol = AWG_PROTOCOL.into();
cap.version = AWG_WIRE_VERSION;
cap
})
})
}
fn on_network_activated(&self, network: NetworkId) {
self.inner.on_network_activated(network);
}
fn on_peer_capability(
&self,
network: NetworkId,
peer: EndpointId,
capability: &PluginCapability,
) -> Result<(), PluginError> {
if capability.protocol != AWG_PROTOCOL
|| capability.version != AWG_WIRE_VERSION
|| !capability.enabled
{
return Err(PluginError::Rejected("unsupported AWG capability".into()));
}
self.inner.on_peer_capability(network, peer, capability)?;
self.transport.allow_peer(network, peer);
Ok(())
}
fn on_address_allocation(
&self,
network: NetworkId,
range: Ipv4Range,
allocations: &[(EndpointId, std::net::Ipv4Addr)],
) {
self.inner
.on_address_allocation(network, range, allocations);
}
fn carry(&self, network: NetworkId, peer: EndpointId, packet: bytes::Bytes) -> bool {
self.inner.carry(network, peer, packet)
}
fn on_peer_link(&self, network: NetworkId, peer: EndpointId, link: SharedLink) {
self.inner.on_peer_link(network, peer, link);
}
fn on_peer_gone(&self, network: NetworkId, peer: EndpointId) {
self.transport.forget_peer(network, peer);
self.inner.on_peer_gone(network, peer);
}
fn on_network_deactivated(&self, network: NetworkId) {
self.transport.stop_network(network);
self.inner.on_network_deactivated(network);
}
fn on_network_forgotten(&self, network: NetworkId) {
self.inner.on_network_forgotten(network);
}
fn shutdown(&self) -> BoxFuture<'_, ()> {
Box::pin(async move {
self.transport.shutdown();
self.inner.shutdown().await;
})
}
}
crates/tsunagi-awg/src/profile.rs
+226
View File
@@ -0,0 +1,226 @@
//! Bounded profiles, regenerated for every pair connection attempt.
use crate::masking::{AmneziaConfig, AwgKind, Range};
use bytes::Bytes;
use serde::{Deserialize, Serialize};
/// An ephemeral masking profile. Secret fields are redacted and erased on drop.
#[derive(Clone, Serialize, Deserialize, zeroize::ZeroizeOnDrop)]
pub struct PairProfile {
id: [u8; 16],
prefix: u16,
headers: [[u32; 2]; 4],
header_key: [u8; 32],
pub(crate) psk: [u8; 32],
junk_count: u8,
junk_min: u16,
junk_max: u16,
padding: u16,
signature: u8,
}
impl std::fmt::Debug for PairProfile {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str("PairProfile([REDACTED])")
}
}
impl PairProfile {
/// Generates a new safe profile; no deterministic seed or durable state.
pub fn generate() -> Self {
let headers = std::array::from_fn(|i| {
let base = (i as u32) * 0x4000_0000 + rand::random_range(1024..0x3fff_0000);
[base, base + rand::random_range(64..1024)]
});
Self {
id: rand::random(),
prefix: rand::random_range(12..49),
headers,
header_key: rand::random(),
psk: rand::random(),
junk_count: rand::random_range(4..9),
junk_min: rand::random_range(24..65),
junk_max: rand::random_range(80..161),
padding: rand::random_range(16..65),
signature: rand::random_range(0..2),
}
}
/// Public attempt identifier, useful for distinguishing retries without revealing secrets.
pub fn id(&self) -> [u8; 16] {
self.id
}
/// Header key, for packet engines and protocol tests; never log this value.
pub fn header_key(&self) -> &[u8; 32] {
&self.header_key
}
/// Rejects invalid remote profiles before allocating packets or entering the codec.
pub fn validate(&self) -> Result<(), &'static str> {
if !(12..=48).contains(&self.prefix)
|| !(4..=8).contains(&self.junk_count)
|| self.junk_min < 24
|| self.junk_min > self.junk_max
|| self.junk_max > 160
|| self.padding > 64
|| self.signature > 1
|| self.header_key == [0; 32]
|| self.psk == [0; 32]
{
return Err("invalid AWG profile bounds");
}
for (i, [lo, hi]) in self.headers.iter().copied().enumerate() {
if lo < 5 || lo > hi || hi - lo > 1024 {
return Err("invalid AWG header range");
}
for [other_lo, other_hi] in &self.headers[..i] {
if lo <= *other_hi && *other_lo <= hi {
return Err("overlapping AWG header ranges");
}
}
}
Ok(())
}
fn config(&self) -> AmneziaConfig {
let range = |i: usize| Range {
min: self.headers[i][0],
max: self.headers[i][1],
};
AmneziaConfig {
jc: self.junk_count.into(),
jmin: self.junk_min.into(),
jmax: self.junk_max.into(),
s1: self.prefix.into(),
s2: self.prefix.into(),
s3: self.prefix.into(),
s4: self.prefix.into(),
h1: range(0),
h2: range(1),
h3: range(2),
h4: range(3),
i1: String::new(),
i2: String::new(),
i3: String::new(),
i4: String::new(),
i5: String::new(),
header_protection_key: Some(self.header_key),
content_padding_addition: Range {
min: 0,
max: self.padding.into(),
},
random_trailers: true,
mtu: 1050,
}
}
/// Masks one standard WireGuard message; invalid types and oversized packets are refused.
pub fn encode(&self, raw: &[u8]) -> Option<Bytes> {
let kind = match raw.get(..4)? {
[1, 0, 0, 0] => AwgKind::Initiation,
[2, 0, 0, 0] => AwgKind::Response,
[3, 0, 0, 0] => AwgKind::Cookie,
[4, 0, 0, 0] => AwgKind::Transport,
_ => return None,
};
if raw.len() < kind.fixed_size() || raw.len() > 1050 {
return None;
}
let config = self.config();
let range = match kind {
AwgKind::Initiation => config.h1,
AwgKind::Response => config.h2,
AwgKind::Cookie => config.h3,
AwgKind::Transport => config.h4,
};
let mut payload = raw.to_vec();
payload[..4].copy_from_slice(&range.random().to_le_bytes());
Some(config.encode(kind, Bytes::from(payload), 1200))
}
/// Unmasks a packet and restores standard headers before WireGuard validation.
pub fn decode(&self, wire: &[u8]) -> Option<Vec<u8>> {
if wire.len() > 1200 {
return None;
}
let decoded = self.config().decode(&Bytes::copy_from_slice(wire))?;
let (kind, len) = match decoded.kind {
AwgKind::Initiation => (1, 148),
AwgKind::Response => (2, 92),
AwgKind::Cookie => (3, 64),
AwgKind::Transport => (4, decoded.message.len()),
};
let mut message = decoded.message.get(..len)?.to_vec();
message[..4].copy_from_slice(&(kind as u32).to_le_bytes());
Some(message)
}
pub(crate) fn pad(&self, packet: Bytes) -> Bytes {
self.config().prepare_transport_plaintext(packet, 1200)
}
/// Generates structured signature packets with fresh fields, never static captures.
/// These target the other Tsunagi peer, not DNS or SIP services.
pub fn signature_packets(&self) -> Vec<Vec<u8>> {
if self.signature == 0 {
// A complete DNS query. Counts, label lengths and qtype remain valid.
let label = format!("n{:016x}", rand::random::<u64>());
let mut packet = Vec::from(rand::random::<u16>().to_be_bytes());
packet.extend_from_slice(&[1, 0, 0, 1, 0, 0, 0, 0, 0, 0]);
packet.push(label.len() as u8);
packet.extend_from_slice(label.as_bytes());
packet.extend_from_slice(b"\x07example\x03net\0\0");
packet.push(if rand::random() { 1 } else { 28 });
packet.extend_from_slice(&[0, 1]);
vec![packet]
} else {
let id = rand::random::<u64>();
vec![format!("OPTIONS sip:n{id:x}@example.net SIP/2.0\r\nVia: SIP/2.0/UDP example.net;branch=z9hG4bK{id:x}\r\nMax-Forwards: 70\r\nFrom: <sip:a@example.net>;tag={id:x}\r\nTo: <sip:n{id:x}@example.net>\r\nCall-ID: {id:x}@example.net\r\nCSeq: 1 OPTIONS\r\nContent-Length: 0\r\n\r\n").into_bytes()]
}
}
pub(crate) fn junk_packets(&self) -> Vec<Vec<u8>> {
(0..self.junk_count)
.map(|_| {
let mut packet = vec![0; rand::random_range(self.junk_min..=self.junk_max).into()];
rand::fill(&mut packet[..]);
packet
})
.collect()
}
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
#[test]
fn unsafe_peer_profiles_are_rejected_before_the_codec() {
let valid = PairProfile::generate();
let mut bad = valid.clone();
bad.prefix = 11;
assert!(bad.validate().is_err());
let mut bad = valid.clone();
bad.headers[1] = bad.headers[0];
assert!(bad.validate().is_err());
let mut bad = valid.clone();
bad.junk_max = u16::MAX;
assert!(bad.validate().is_err());
let mut bad = valid.clone();
bad.padding = u16::MAX;
assert!(bad.validate().is_err());
let mut bad = valid.clone();
bad.psk = [0; 32];
assert!(bad.validate().is_err());
let mut bad = valid.clone();
bad.signature = 2;
assert!(bad.validate().is_err());
}
#[test]
fn signature_templates_have_consistent_framing_and_fresh_fields() {
let mut profile = PairProfile::generate();
profile.signature = 0;
let dns = profile.signature_packets().remove(0);
assert_eq!(&dns[2..12], &[1, 0, 0, 1, 0, 0, 0, 0, 0, 0]);
let mut at = 12;
while dns[at] != 0 {
at += 1 + usize::from(dns[at]);
}
assert_eq!(at + 5, dns.len());
assert_eq!(&dns[dns.len() - 2..], &[0, 1]);
profile.signature = 1;
let sip = profile.signature_packets().remove(0);
assert!(sip.starts_with(b"OPTIONS sip:"));
assert!(sip.ends_with(b"Content-Length: 0\r\n\r\n"));
assert_ne!(sip, profile.signature_packets().remove(0));
}
}
crates/tsunagi-awg/src/session.rs
+402
View File
@@ -0,0 +1,402 @@
//! One cancellable actor and UDP socket per pair. All payload and health frames
//! are inside the Amnezia-masked authenticated WireGuard tunnel.
use crate::profile::PairProfile;
use boringtun::noise::{Tunn, TunnResult};
use bytes::Bytes;
use iroh::EndpointId;
use std::collections::{HashMap, VecDeque};
use std::net::{Ipv4Addr, SocketAddr};
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};
use tokio::net::UdpSocket;
use tokio::sync::{mpsc, watch};
use tsunagi::dataplane::transport::{PacketLink, TransportError};
use tsunagi::{BoxFuture, NetworkId};
pub(crate) const MAX_PAYLOAD: usize = tsunagi::config::MAX_DATA_DATAGRAM;
const CHUNK: usize = 700;
const QUEUE: usize = 64;
const WINDOW: Duration = Duration::from_secs(20);
pub(crate) struct PairLink {
network: NetworkId,
peer: EndpointId,
pub(crate) attempt: [u8; 16],
tx: mpsc::Sender<Bytes>,
rx: tokio::sync::Mutex<mpsc::Receiver<Bytes>>,
stop: watch::Sender<bool>,
closed: AtomicBool,
remote: Mutex<Option<SocketAddr>>,
reason: Mutex<Option<String>>,
}
impl std::fmt::Debug for PairLink {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("AwgPair")
.field("network", &self.network)
.field("peer", &self.peer)
.finish_non_exhaustive()
}
}
impl PairLink {
pub(crate) fn has_authenticated_path(&self) -> bool {
match self.remote.lock() {
Ok(remote) => remote.is_some(),
Err(_) => false,
}
}
}
impl PacketLink for PairLink {
fn network(&self) -> NetworkId {
self.network
}
fn peer(&self) -> EndpointId {
self.peer
}
fn max_datagram_size(&self) -> usize {
MAX_PAYLOAD
}
fn send(&self, payload: Bytes) -> Result<(), TransportError> {
if self.is_closed() {
return Err(TransportError::Closed);
}
if payload.len() > MAX_PAYLOAD {
return Err(TransportError::TooLarge {
size: payload.len(),
limit: MAX_PAYLOAD,
});
}
self.tx
.try_send(payload)
.map_err(|_| TransportError::Other("AWG send queue full or closed".into()))
}
fn recv(&self) -> BoxFuture<'_, Option<Bytes>> {
Box::pin(async move {
let mut stop = self.stop.subscribe();
if *stop.borrow() {
return None;
}
let mut rx = self.rx.lock().await;
tokio::select! {biased; _=stop.changed()=>None,packet=rx.recv()=>packet}
})
}
fn closed(&self) -> BoxFuture<'_, ()> {
Box::pin(async move {
let mut stop = self.stop.subscribe();
if !*stop.borrow() {
let _ = stop.changed().await;
}
})
}
fn is_closed(&self) -> bool {
self.closed.load(Ordering::Acquire) || *self.stop.borrow()
}
fn close(&self) {
self.stop.send_replace(true);
}
fn failure_reason(&self) -> Option<String> {
match self.reason.lock() {
Ok(reason) => reason.clone(),
Err(_) => None,
}
}
fn path_description(&self) -> String {
match self.remote.lock() {
Ok(addr) => match *addr {
Some(addr) => format!("awg UDP via {addr}"),
None => "awg UDP, establishing".into(),
},
Err(_) => "awg UDP".into(),
}
}
}
impl Drop for PairLink {
fn drop(&mut self) {
self.stop.send_replace(true);
}
}
pub(crate) fn start(
network: NetworkId,
peer: EndpointId,
socket: UdpSocket,
tunn: Tunn,
profile: PairProfile,
candidates: Vec<SocketAddr>,
parent: watch::Receiver<bool>,
) -> Arc<PairLink> {
let (tx, rx) = mpsc::channel(QUEUE);
let (in_tx, in_rx) = mpsc::channel(QUEUE);
let (stop, _) = watch::channel(false);
let link = Arc::new(PairLink {
network,
peer,
attempt: profile.id(),
tx,
rx: tokio::sync::Mutex::new(in_rx),
stop: stop.clone(),
closed: AtomicBool::new(false),
remote: Mutex::new(None),
reason: Mutex::new(None),
});
let weak = Arc::downgrade(&link);
tokio::spawn(async move {
let mut engine = Engine {
stop: stop.subscribe(),
socket,
tunn,
profile,
candidates,
remote: None,
inbox: in_tx,
rx,
counter: rand::random(),
sequence: 0,
pending: VecDeque::new(),
samples: VecDeque::new(),
assemblies: HashMap::new(),
started: Instant::now(),
last_authenticated: Instant::now(),
};
engine.run(parent, &weak).await;
if let Some(link) = weak.upgrade() {
link.closed.store(true, Ordering::Release);
link.stop.send_replace(true);
}
});
link
}
struct Assembly {
created: Instant,
count: u16,
chunks: Vec<Option<Bytes>>,
}
struct Engine {
stop: watch::Receiver<bool>,
socket: UdpSocket,
tunn: Tunn,
profile: PairProfile,
candidates: Vec<SocketAddr>,
remote: Option<SocketAddr>,
inbox: mpsc::Sender<Bytes>,
rx: mpsc::Receiver<Bytes>,
counter: u64,
sequence: u64,
pending: VecDeque<(u64, Instant)>,
samples: VecDeque<bool>,
assemblies: HashMap<u64, Assembly>,
started: Instant,
last_authenticated: Instant,
}
enum ResultOwned {
Wire(Vec<u8>),
Plain(Vec<u8>),
Done,
}
fn own(result: TunnResult<'_>) -> ResultOwned {
match result {
TunnResult::WriteToNetwork(p) => ResultOwned::Wire(p.to_vec()),
TunnResult::WriteToTunnelV4(p, _) => ResultOwned::Plain(p.to_vec()),
_ => ResultOwned::Done,
}
}
fn ip(payload: &[u8]) -> Bytes {
let mut packet = vec![0; 20];
packet[0] = 0x45;
packet[8] = 64;
packet[9] = 253;
packet[12..16].copy_from_slice(&Ipv4Addr::new(192, 0, 2, 1).octets());
packet[16..20].copy_from_slice(&Ipv4Addr::new(192, 0, 2, 2).octets());
packet[2..4].copy_from_slice(&((20 + payload.len()) as u16).to_be_bytes());
let sum = packet
.as_chunks::<2>()
.0
.iter()
.map(|b| u32::from(u16::from_be_bytes([b[0], b[1]])))
.sum::<u32>();
let sum = (sum & 0xffff) + (sum >> 16);
packet[10..12].copy_from_slice(&(!(sum as u16)).to_be_bytes());
packet.extend_from_slice(payload);
Bytes::from(packet)
}
impl Engine {
async fn run(&mut self, mut stop: watch::Receiver<bool>, link: &std::sync::Weak<PairLink>) {
if *self.stop.borrow() || *stop.borrow() {
return;
}
let mut buffer = [0; 1201];
let mut scratch = [0; 4096];
let mut timer = tokio::time::interval(Duration::from_millis(250));
timer.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
let mut last_probe = Instant::now() - Duration::from_secs(2);
loop {
tokio::select! {
biased;
_=stop.changed()=>break,
_=self.stop.changed()=>break,
_=timer.tick()=>{
let result=own(self.tunn.update_timers(&mut scratch));self.process(result,None).await;
if last_probe.elapsed()>=Duration::from_secs(1) {
last_probe=Instant::now();self.sequence=self.sequence.wrapping_add(1);
self.pending.push_back((self.sequence,Instant::now()));
let mut probe=vec![1];probe.extend_from_slice(&self.sequence.to_be_bytes());
probe.resize(713,0);rand::fill(&mut probe[9..]);self.encrypt(&probe).await;
}
while self.pending.front().is_some_and(|(_,at)|at.elapsed()>=Duration::from_secs(3)) {
self.pending.pop_front();self.samples.push_back(false);
}
while self.samples.len()>20 {self.samples.pop_front();}
self.assemblies.retain(|_,a|a.created.elapsed()<Duration::from_secs(5));
if self.started.elapsed()>WINDOW && (self.last_authenticated.elapsed()>Duration::from_secs(10)
|| (self.samples.len()>=15&&self.samples.iter().filter(|ok|!**ok).count()*100>=self.samples.len()*60)) {
if let Some(link)=link.upgrade() && let Ok(mut reason)=link.reason.lock() {*reason=Some("AWG data delivery checks: persistent loss or timeout".into());}
break;
}
}
packet=self.socket.recv_from(&mut buffer)=>{
if let Ok((len,source))=packet && let Some(message)=self.profile.decode(&buffer[..len]) {
let result=own(self.tunn.decapsulate(None,&message,&mut scratch));
if matches!(result,ResultOwned::Plain(_)) {
self.remote=Some(source);self.last_authenticated=Instant::now();
if let Some(link)=link.upgrade() && let Ok(mut remote)=link.remote.lock(){*remote=Some(source);}
}
self.process(result,Some(source)).await;
for _ in 0..16 {
let result=own(self.tunn.decapsulate(None,&[],&mut scratch));
if matches!(result,ResultOwned::Done){break;}
self.process(result,Some(source)).await;
}
}
}
packet=self.rx.recv()=>{
let Some(packet)=packet else{break;};
let count=packet.len().max(1).div_ceil(CHUNK) as u16;
self.counter=self.counter.wrapping_add(1);
for (index,chunk) in packet.chunks(CHUNK).enumerate() {
let mut payload=vec![3];payload.extend_from_slice(&self.counter.to_be_bytes());
payload.extend_from_slice(&(index as u16).to_be_bytes());payload.extend_from_slice(&count.to_be_bytes());
payload.extend_from_slice(chunk);self.encrypt(&payload).await;
// Large logical datagrams must let local readers and other
// pairs drain their bounded socket/command queues.
if (index + 1).is_multiple_of(16) {tokio::task::yield_now().await;}
}
}
}
}
}
async fn encrypt(&mut self, payload: &[u8]) {
let packet = self.profile.pad(ip(payload));
let mut scratch = [0; 4096];
let result = own(self.tunn.encapsulate(&packet, &mut scratch));
self.process(result, None).await;
}
async fn wire(&self, raw: &[u8], target: Option<SocketAddr>) {
let Some(wire) = self.profile.encode(raw) else {
return;
};
let targets: Vec<_> = target
.or(self.remote)
.map(|a| vec![a])
.unwrap_or_else(|| self.candidates.clone());
for addr in targets {
if raw.first() == Some(&1) {
for signature in self.profile.signature_packets() {
let _ = self.send_wire(&signature, addr).await;
}
for junk in self.profile.junk_packets() {
let _ = self.send_wire(&junk, addr).await;
}
}
let _ = self.send_wire(&wire, addr).await;
}
}
async fn send_wire(&self, bytes: &[u8], address: SocketAddr) {
let mut stop = self.stop.clone();
if *stop.borrow() {
return;
}
tokio::select! {biased; _=stop.changed()=>{}, _=tokio::time::timeout(Duration::from_millis(100),self.socket.send_to(bytes,address))=>{}}
}
async fn process(&mut self, result: ResultOwned, target: Option<SocketAddr>) {
match result {
ResultOwned::Wire(raw) => self.wire(&raw, target).await,
ResultOwned::Plain(packet) => {
let Some(payload) = packet.get(20..) else {
return;
};
let Some(seq) = payload
.get(1..9)
.and_then(|b| b.try_into().ok())
.map(u64::from_be_bytes)
else {
return;
};
match payload.first() {
Some(1) => {
let mut pong = vec![2];
pong.extend_from_slice(&seq.to_be_bytes());
pong.resize(713, 0);
let packet = self.profile.pad(ip(&pong));
let mut scratch = [0; 4096];
if let ResultOwned::Wire(raw) =
own(self.tunn.encapsulate(&packet, &mut scratch))
{
self.wire(&raw, target).await;
}
}
Some(2) => {
if let Some(index) =
self.pending.iter().position(|(pending, _)| *pending == seq)
{
self.pending.remove(index);
self.samples.push_back(true);
}
}
Some(3) => self.fragment(seq, payload),
_ => {}
}
}
ResultOwned::Done => {}
}
}
fn fragment(&mut self, id: u64, payload: &[u8]) {
let Some(header) = payload.get(9..13) else {
return;
};
let index = u16::from_be_bytes([header[0], header[1]]);
let count = u16::from_be_bytes([header[2], header[3]]);
let chunk = &payload[13..];
if count == 0
|| usize::from(count) > MAX_PAYLOAD.div_ceil(CHUNK)
|| index >= count
|| chunk.len() > CHUNK
{
return;
}
if !self.assemblies.contains_key(&id) && self.assemblies.len() >= 8 {
return;
}
let assembly = self.assemblies.entry(id).or_insert_with(|| Assembly {
created: Instant::now(),
count,
chunks: vec![None; count.into()],
});
if assembly.count != count {
return;
}
assembly.chunks[usize::from(index)] = Some(Bytes::copy_from_slice(chunk));
if assembly.chunks.iter().all(Option::is_some)
&& let Some(assembly) = self.assemblies.remove(&id)
{
let mut packet = Vec::new();
for chunk in assembly.chunks.into_iter().flatten() {
packet.extend_from_slice(&chunk);
}
if packet.len() <= MAX_PAYLOAD {
let _ = self.inbox.try_send(Bytes::from(packet));
}
}
}
}
crates/tsunagi-awg/src/stun.rs
+101
View File
@@ -0,0 +1,101 @@
//! Optional, bounded server-reflexive discovery on the pair's own UDP socket.
//! A STUN answer is a candidate only; WireGuard authentication verifies a path.
use std::net::{IpAddr, Ipv4Addr, SocketAddr};
use std::time::Duration;
use tokio::net::UdpSocket;
const COOKIE: [u8; 4] = [0x21, 0x12, 0xa4, 0x42];
pub(crate) async fn mapped(socket: &UdpSocket, server: &str) -> Option<SocketAddr> {
tokio::time::timeout(Duration::from_millis(1200), async {
let address = tokio::net::lookup_host(server)
.await
.ok()?
.find(SocketAddr::is_ipv4)?;
let transaction: [u8; 12] = rand::random();
let mut request = [0; 20];
request[..2].copy_from_slice(&[0, 1]);
request[4..8].copy_from_slice(&COOKIE);
request[8..].copy_from_slice(&transaction);
socket.send_to(&request, address).await.ok()?;
let mut bytes = [0; 1024];
loop {
let (len, source) = socket.recv_from(&mut bytes).await.ok()?;
if source == address
&& let Some(mapped) = parse(&bytes[..len], &transaction)
{
return Some(mapped);
}
}
})
.await
.ok()
.flatten()
}
fn parse(bytes: &[u8], transaction: &[u8; 12]) -> Option<SocketAddr> {
if bytes.len() < 20
|| bytes[..2] != [1, 1]
|| bytes[4..8] != COOKIE
|| &bytes[8..20] != transaction
{
return None;
}
let declared = u16::from_be_bytes(bytes[2..4].try_into().ok()?) as usize;
if !declared.is_multiple_of(4) || declared != bytes.len() - 20 {
return None;
}
let mut at = 20;
while at + 4 <= bytes.len() {
let kind = u16::from_be_bytes(bytes[at..at + 2].try_into().ok()?);
let len = u16::from_be_bytes(bytes[at + 2..at + 4].try_into().ok()?) as usize;
at += 4;
let attribute = bytes.get(at..at.checked_add(len)?)?;
if kind == 0x0020 && attribute.len() == 8 && attribute[0..2] == [0, 1] {
let port = u16::from_be_bytes([attribute[2] ^ COOKIE[0], attribute[3] ^ COOKIE[1]]);
let ip = Ipv4Addr::new(
attribute[4] ^ COOKIE[0],
attribute[5] ^ COOKIE[1],
attribute[6] ^ COOKIE[2],
attribute[7] ^ COOKIE[3],
);
if port == 0 || ip.is_unspecified() || ip.is_multicast() || ip.is_broadcast() {
return None;
}
return Some(SocketAddr::new(IpAddr::V4(ip), port));
}
at = at.checked_add((len + 3) & !3)?;
}
None
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
#[tokio::test]
async fn mapping_uses_the_actual_socket_and_rejects_unrelated_or_truncated_answers() {
let server = UdpSocket::bind((Ipv4Addr::LOCALHOST, 0)).await.unwrap();
let address = server.local_addr().unwrap();
let socket = UdpSocket::bind((Ipv4Addr::LOCALHOST, 0)).await.unwrap();
let expected = socket.local_addr().unwrap();
let task = tokio::spawn(async move {
let mut request = [0; 128];
let (len, source) = server.recv_from(&mut request).await.unwrap();
assert_eq!(len, 20);
assert_eq!(source, expected);
let tx: [u8; 12] = request[8..20].try_into().unwrap();
let mut reply = request[..20].to_vec();
reply[..4].copy_from_slice(&[1, 1, 0, 12]);
reply.extend_from_slice(&[0, 0x20, 0, 8, 0, 1]);
let port = source.port().to_be_bytes();
reply.extend_from_slice(&[port[0] ^ COOKIE[0], port[1] ^ COOKIE[1]]);
for (i, b) in Ipv4Addr::LOCALHOST.octets().into_iter().enumerate() {
reply.push(b ^ COOKIE[i]);
}
assert!(parse(&reply, &[0; 12]).is_none());
for len in 0..reply.len() {
assert!(parse(&reply[..len], &tx).is_none());
}
server.send_to(&reply, source).await.unwrap();
});
assert_eq!(mapped(&socket, &address.to_string()).await, Some(expected));
task.await.unwrap();
}
}
crates/tsunagi-awg/src/transport.rs
+720
View File
@@ -0,0 +1,720 @@
//! Private pair negotiation. No secret profiles in capabilities or durable state.
use crate::{
AWG_PROTOCOL, AWG_WIRE_VERSION,
profile::PairProfile,
session::{self, PairLink},
};
use boringtun::{
noise::Tunn,
x25519::{PublicKey, StaticSecret},
};
use iroh::EndpointId;
use serde::{Deserialize, Serialize};
use std::collections::{HashMap, HashSet};
use std::net::{Ipv4Addr, SocketAddr};
use std::sync::{Arc, Mutex, OnceLock, Weak};
use std::time::Duration;
use tokio::net::UdpSocket;
use tokio::sync::{mpsc, oneshot, watch};
use tsunagi::dataplane::transport::TransportPayload;
use tsunagi::dataplane::transport::{
ControlSender, InboundLink, PacketLink, PacketTransport, SharedLink, TransportError,
};
use tsunagi::{BoxFuture, NetworkId};
type Pair = (NetworkId, EndpointId);
#[derive(Serialize, Deserialize)]
enum Negotiation {
Offer {
version: u16,
profile: PairProfile,
key: [u8; 32],
#[serde(deserialize_with = "bounded_addresses")]
addresses: Vec<SocketAddr>,
},
Answer {
attempt: [u8; 16],
key: [u8; 32],
#[serde(deserialize_with = "bounded_addresses")]
addresses: Vec<SocketAddr>,
},
}
fn bounded_addresses<'de, D: serde::Deserializer<'de>>(
deserializer: D,
) -> Result<Vec<SocketAddr>, D::Error> {
struct Visitor;
impl<'de> serde::de::Visitor<'de> for Visitor {
type Value = Vec<SocketAddr>;
fn expecting(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str("at most 32 UDP candidates")
}
fn visit_seq<A: serde::de::SeqAccess<'de>>(
self,
mut sequence: A,
) -> Result<Self::Value, A::Error> {
if sequence.size_hint().is_some_and(|n| n > 32) {
return Err(serde::de::Error::custom("too many UDP candidates"));
}
let mut addresses = Vec::with_capacity(sequence.size_hint().unwrap_or(0).min(32));
while let Some(address) = sequence.next_element()? {
if addresses.len() == 32 {
return Err(serde::de::Error::custom("too many UDP candidates"));
}
addresses.push(address);
}
Ok(addresses)
}
}
deserializer.deserialize_seq(Visitor)
}
struct Pending {
profile: PairProfile,
key: StaticSecret,
socket: UdpSocket,
reply: oneshot::Sender<Result<SharedLink, TransportError>>,
}
struct Inner {
stopped: watch::Sender<bool>,
local: EndpointId,
bind: Ipv4Addr,
stun: Option<String>,
busy: Mutex<HashSet<Pair>>,
control: OnceLock<Arc<dyn ControlSender>>,
allowed: Mutex<HashSet<Pair>>,
pending: Mutex<HashMap<Pair, Pending>>,
links: Mutex<HashMap<Pair, Arc<PairLink>>>,
inbound: mpsc::Sender<InboundLink>,
port_by_pair: Mutex<HashMap<Pair, u16>>,
}
/// Public diagnostics for one pair. No key or masking parameters are exposed.
#[derive(Debug, Clone)]
pub struct PairOverview {
/// Other endpoint's authenticated identity.
pub peer: EndpointId,
/// Fresh connection-attempt identifier.
pub attempt: [u8; 16],
/// Last known UDP port; None if diagnostics race with pair retirement.
pub local_port: Option<u16>,
/// Actual transport path, if established.
pub path: String,
/// Whether this attempt has been retired.
pub closed: bool,
}
/// A bounded set of independently configured pair engines.
pub struct AwgTransport {
inner: Arc<Inner>,
commands: mpsc::Sender<(NetworkId, EndpointId, Negotiation)>,
incoming: tokio::sync::Mutex<mpsc::Receiver<InboundLink>>,
}
impl std::fmt::Debug for AwgTransport {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("AwgTransport")
.field("local", &self.inner.local)
.finish_non_exhaustive()
}
}
fn lock<T>(mutex: &Mutex<T>) -> std::sync::MutexGuard<'_, T> {
match mutex.lock() {
Ok(g) => g,
Err(p) => p.into_inner(),
}
}
fn error(text: &str) -> TransportError {
TransportError::Other(text.into())
}
async fn addresses(inner: &Inner, socket: &UdpSocket) -> Result<Vec<SocketAddr>, TransportError> {
let local = socket
.local_addr()
.map_err(|_| error("AWG socket address unavailable"))?;
let mut candidates = if inner.bind.is_loopback() {
vec![local]
} else {
tsunagi_wireguard::local_ip_candidates()
.into_iter()
.filter(|ip| ip.is_ipv4())
.take(31)
.map(|ip| SocketAddr::new(ip, local.port()))
.collect()
};
if let Some(server) = &inner.stun
&& let Some(mapped) = crate::stun::mapped(socket, server).await
&& !candidates.contains(&mapped)
{
candidates.push(mapped);
}
if candidates.is_empty() {
return Err(error("AWG has no UDP address candidates"));
}
Ok(candidates)
}
fn validate_endpoint(key: &[u8; 32], addresses: &[SocketAddr]) -> Result<(), TransportError> {
if *key == [0; 32]
|| addresses.is_empty()
|| addresses.len() > 32
|| addresses.iter().any(|address| {
address.port() == 0
|| !address.is_ipv4()
|| address.ip().is_unspecified()
|| address.ip().is_multicast()
|| matches!(address.ip(), std::net::IpAddr::V4(ip) if ip.is_broadcast())
})
{
return Err(error("invalid AWG endpoint"));
}
Ok(())
}
impl AwgTransport {
/// Starts the coordinator. Production binds wildcard IPv4; tests use loopback.
pub fn new(local: EndpointId, bind: Ipv4Addr) -> Arc<Self> {
Self::new_with_stun(local, bind, None)
}
/// Enables an optional STUN server (host:port) for this pair's own socket.
/// Local-only callers pass None; failure still permits local UDP candidates.
pub fn new_with_stun(local: EndpointId, bind: Ipv4Addr, stun: Option<String>) -> Arc<Self> {
let (commands, mut rx) = mpsc::channel(64);
let (inbound, incoming) = mpsc::channel(64);
let (stopped, _) = watch::channel(false);
let inner = Arc::new(Inner {
stopped,
local,
bind,
stun,
busy: Mutex::default(),
control: OnceLock::new(),
allowed: Mutex::default(),
pending: Mutex::default(),
links: Mutex::default(),
inbound,
port_by_pair: Mutex::default(),
});
let weak = Arc::downgrade(&inner);
let mut stopped = inner.stopped.subscribe();
tokio::spawn(async move {
while let Some((network, peer, message)) =
tokio::select! { biased; _=stopped.changed()=>None, message=rx.recv()=>message }
{
let Some(inner) = weak.upgrade() else {
break;
};
if !lock(&inner.allowed).contains(&(network, peer))
|| !lock(&inner.busy).insert((network, peer))
{
continue;
}
tokio::spawn(async move {
let mut stop = inner.stopped.subscribe();
tokio::select! {biased; _=stop.changed()=>{}, _=async {
if let Err(err)=receive(&inner,(network,peer),message).await {tracing::debug!(%err,"AWG pair negotiation rejected");}
}=>{}}
lock(&inner.busy).remove(&(network, peer));
});
}
});
Arc::new(Self {
inner,
commands,
incoming: tokio::sync::Mutex::new(incoming),
})
}
/// Registers a peer after its authenticated capability has been validated.
pub fn allow_peer(&self, network: NetworkId, peer: EndpointId) {
let mut allowed = lock(&self.inner.allowed);
if !*self.inner.stopped.borrow() && allowed.len() < 128 {
allowed.insert((network, peer));
}
}
/// Retires a pair when its network membership session goes away.
pub fn forget_peer(&self, network: NetworkId, peer: EndpointId) {
let pair = (network, peer);
lock(&self.inner.allowed).remove(&pair);
lock(&self.inner.pending).remove(&pair);
lock(&self.inner.port_by_pair).remove(&pair);
if let Some(link) = lock(&self.inner.links).remove(&pair) {
link.close();
}
}
/// Stops all pair engines in this network.
pub fn stop_network(&self, network: NetworkId) {
let peers: Vec<_> = lock(&self.inner.allowed)
.iter()
.filter(|(n, _)| *n == network)
.map(|(_, p)| *p)
.collect();
for peer in peers {
self.forget_peer(network, peer);
}
}
/// Stops all sockets and coordinator work without waiting for remote peers.
pub fn shutdown(&self) {
self.inner.stopped.send_replace(true);
lock(&self.inner.allowed).clear();
lock(&self.inner.pending).clear();
for (_, link) in lock(&self.inner.links).drain() {
link.close();
}
lock(&self.inner.port_by_pair).clear();
}
/// Receives a successfully negotiated and authenticated inbound pair link.
pub async fn next_inbound(&self) -> Option<InboundLink> {
let mut stopped = self.inner.stopped.subscribe();
if *stopped.borrow() {
return None;
}
let mut incoming = self.incoming.lock().await;
tokio::select! {biased; _=stopped.changed()=>None, link=incoming.recv()=>link}
}
/// Takes a public snapshot of this network's pair attempts.
pub fn overview(&self, network: NetworkId) -> Vec<PairOverview> {
let links = lock(&self.inner.links);
let ports = lock(&self.inner.port_by_pair);
links
.iter()
.filter(|((n, _), _)| *n == network)
.map(|((_, peer), link)| PairOverview {
peer: *peer,
attempt: link.attempt,
local_port: ports.get(&(network, *peer)).copied(),
path: link.path_description(),
closed: link.is_closed(),
})
.collect()
}
}
impl PacketTransport for AwgTransport {
fn name(&self) -> &str {
AWG_PROTOCOL
}
fn as_any(&self) -> &dyn std::any::Any {
self
}
fn attach_control(&self, control: Arc<dyn ControlSender>) {
let _ = self.inner.control.set(control);
}
fn on_control(
&self,
network: NetworkId,
peer: EndpointId,
protocol: &str,
payload: &[u8],
) -> Result<(), TransportError> {
if protocol != AWG_PROTOCOL || !lock(&self.inner.allowed).contains(&(network, peer)) {
return Err(TransportError::Declined(protocol.into()));
}
if payload.len() > 4096 {
return Err(error("AWG negotiation too large"));
}
let message: Negotiation =
postcard::from_bytes(payload).map_err(|_| error("malformed AWG negotiation"))?;
match &message {
Negotiation::Offer {
version,
profile,
key,
addresses,
} => {
if *version != AWG_WIRE_VERSION || peer.as_bytes() >= self.inner.local.as_bytes() {
return Err(error("invalid AWG offer role/version"));
}
profile.validate().map_err(error)?;
validate_endpoint(key, addresses)?;
}
Negotiation::Answer { key, addresses, .. } => validate_endpoint(key, addresses)?,
}
self.commands
.try_send((network, peer, message))
.map_err(|_| error("AWG negotiation queue full"))
}
fn open<'a>(
&'a self,
network: NetworkId,
peer: EndpointId,
protocol: &'a str,
) -> BoxFuture<'a, Result<SharedLink, TransportError>> {
Box::pin(async move {
let pair = (network, peer);
if protocol != AWG_PROTOCOL
|| self.inner.local.as_bytes() >= peer.as_bytes()
|| !lock(&self.inner.allowed).contains(&pair)
{
return Err(TransportError::Declined(protocol.into()));
}
let control = self
.inner
.control
.get()
.ok_or_else(|| error("AWG control plane unavailable"))?;
let socket = UdpSocket::bind((self.inner.bind, 0))
.await
.map_err(|_| error("AWG UDP bind failed"))?;
let profile = PairProfile::generate();
let attempt = profile.id();
let key = StaticSecret::from(rand::random::<[u8; 32]>());
let public = PublicKey::from(&key).to_bytes();
let message = Negotiation::Offer {
version: AWG_WIRE_VERSION,
profile: profile.clone(),
key: public,
addresses: addresses(&self.inner, &socket).await?,
};
let (reply, rx) = oneshot::channel();
if lock(&self.inner.pending).len() >= 128 {
return Err(error("too many AWG pair negotiations"));
}
lock(&self.inner.pending).insert(
pair,
Pending {
profile,
key,
socket,
reply,
},
);
// A cancellation guard also retires partially started attempts.
let mut guard = AttemptGuard {
inner: Arc::downgrade(&self.inner),
pair,
attempt,
completed: false,
};
let bytes =
postcard::to_stdvec(&message).map_err(|_| error("AWG offer encoding failed"))?;
control
.send(network, peer, AWG_PROTOCOL, TransportPayload::new(bytes))
.await?;
let result = tokio::time::timeout(Duration::from_secs(15), rx)
.await
.map_err(|_| error("AWG negotiation/handshake timed out"))?
.map_err(|_| error("AWG attempt cancelled"))?;
guard.completed = result.is_ok();
result
})
}
}
struct AttemptGuard {
inner: Weak<Inner>,
pair: Pair,
attempt: [u8; 16],
completed: bool,
}
impl Drop for AttemptGuard {
fn drop(&mut self) {
if let Some(inner) = self.inner.upgrade() {
let mut pending = lock(&inner.pending);
if pending
.get(&self.pair)
.is_some_and(|p| p.profile.id() == self.attempt)
{
pending.remove(&self.pair);
}
drop(pending);
if !self.completed
&& let Some(link) = lock(&inner.links).get(&self.pair)
&& link.attempt == self.attempt
{
link.close();
}
}
}
}
async fn receive(
inner: &Arc<Inner>,
pair: Pair,
message: Negotiation,
) -> Result<(), TransportError> {
if !lock(&inner.allowed).contains(&pair) {
return Err(error("AWG peer no longer authenticated"));
}
match message {
Negotiation::Offer {
profile,
key,
addresses: candidates,
..
} => {
if let Some(link) = lock(&inner.links).get(&pair) {
if link.attempt == profile.id() {
return Err(error("duplicate AWG offer"));
}
link.close();
}
let socket = UdpSocket::bind((inner.bind, 0))
.await
.map_err(|_| error("AWG UDP bind failed"))?;
let local_port = socket
.local_addr()
.map_err(|_| error("AWG socket address unavailable"))?
.port();
let local_key = StaticSecret::from(rand::random::<[u8; 32]>());
let public = PublicKey::from(&local_key).to_bytes();
let answer = Negotiation::Answer {
attempt: profile.id(),
key: public,
addresses: addresses(inner, &socket).await?,
};
let tunn = Tunn::new(
local_key,
PublicKey::from(key),
Some(profile.psk),
Some(1),
rand::random(),
None,
);
let link = session::start(
pair.0,
pair.1,
socket,
tunn,
profile,
candidates,
inner.stopped.subscribe(),
);
if !lock(&inner.allowed).contains(&pair) {
link.close();
return Err(error("AWG membership ended"));
}
lock(&inner.links).insert(pair, link.clone());
lock(&inner.port_by_pair).insert(pair, local_port);
let control = inner
.control
.get()
.ok_or_else(|| error("AWG control unavailable"))?;
if let Err(err) = control
.send(
pair.0,
pair.1,
AWG_PROTOCOL,
TransportPayload::new(
postcard::to_stdvec(&answer)
.map_err(|_| error("AWG answer encoding failed"))?,
),
)
.await
{
link.close();
return Err(err);
}
publish_when_up(Arc::downgrade(inner), pair, link, None);
}
Negotiation::Answer {
attempt,
key,
addresses: candidates,
} => {
if !lock(&inner.pending)
.get(&pair)
.is_some_and(|p| p.profile.id() == attempt)
{
return Err(error("stale AWG answer"));
}
let pending = lock(&inner.pending)
.remove(&pair)
.ok_or_else(|| error("AWG attempt cancelled"))?;
let local_port = pending
.socket
.local_addr()
.map_err(|_| error("AWG socket address unavailable"))?
.port();
let tunn = Tunn::new(
pending.key,
PublicKey::from(key),
Some(pending.profile.psk),
Some(1),
rand::random(),
None,
);
let link = session::start(
pair.0,
pair.1,
pending.socket,
tunn,
pending.profile,
candidates,
inner.stopped.subscribe(),
);
if !lock(&inner.allowed).contains(&pair) {
link.close();
return Err(error("AWG membership ended"));
}
lock(&inner.links).insert(pair, link.clone());
lock(&inner.port_by_pair).insert(pair, local_port);
publish_when_up(Arc::downgrade(inner), pair, link, Some(pending.reply));
}
}
Ok(())
}
fn publish_when_up(
inner: Weak<Inner>,
pair: Pair,
link: Arc<PairLink>,
reply: Option<oneshot::Sender<Result<SharedLink, TransportError>>>,
) {
tokio::spawn(async move {
let mut reply = reply;
let result = tokio::time::timeout(Duration::from_secs(14), async {
loop {
if link.is_closed() || reply.as_ref().is_some_and(oneshot::Sender::is_closed) {
return Err(error("AWG pair closed during handshake"));
}
if link.has_authenticated_path() {
return Ok(());
}
tokio::time::sleep(Duration::from_millis(25)).await;
}
})
.await;
if !matches!(result, Ok(Ok(()))) {
link.close();
if let Some(reply) = reply.take() {
let _ = reply.send(Err(error("AWG authenticated data path timed out")));
}
return;
}
let Some(inner) = inner.upgrade() else {
link.close();
return;
};
if !lock(&inner.allowed).contains(&pair) {
link.close();
return;
}
if link.is_closed()
|| !lock(&inner.links)
.get(&pair)
.is_some_and(|current| current.attempt == link.attempt)
{
link.close();
return;
}
let shared: SharedLink = link.clone();
if let Some(reply) = reply {
if reply.send(Ok(shared)).is_err() {
link.close();
}
} else if inner
.inbound
.try_send(InboundLink {
network: pair.0,
peer: pair.1,
protocol: AWG_PROTOCOL.into(),
link: shared,
})
.is_err()
{
link.close();
}
});
}
impl Drop for AwgTransport {
fn drop(&mut self) {
self.shutdown();
}
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
#[tokio::test]
async fn concurrent_admission_is_bounded_across_networks_and_retirement_releases_slots() {
let identity = tsunagi::identity::DeviceIdentity::generate();
let transport = AwgTransport::new(identity.endpoint_id(), Ipv4Addr::LOCALHOST);
let peer = tsunagi::identity::DeviceIdentity::generate().endpoint_id();
std::thread::scope(|scope| {
for thread in 0..8u8 {
let transport = &transport;
scope.spawn(move || {
for item in 0..32u8 {
let mut id = [0; 32];
id[0] = thread;
id[1] = item;
transport.allow_peer(NetworkId::from_bytes(id), peer);
}
});
}
});
assert_eq!(lock(&transport.inner.allowed).len(), 128);
let network = lock(&transport.inner.allowed).iter().next().unwrap().0;
transport.stop_network(network);
let new = NetworkId::from_bytes([255; 32]);
transport.allow_peer(new, peer);
assert!(lock(&transport.inner.allowed).contains(&(new, peer)));
assert_eq!(lock(&transport.inner.allowed).len(), 128);
transport.shutdown();
assert!(lock(&transport.inner.allowed).is_empty());
}
#[tokio::test]
async fn private_negotiation_bounds_roles_and_membership_before_starting_engines() {
let mut ids = [
tsunagi::identity::DeviceIdentity::generate().endpoint_id(),
tsunagi::identity::DeviceIdentity::generate().endpoint_id(),
];
ids.sort_by_key(|id| *id.as_bytes());
let peer = ids[0];
let local = ids[1];
let network = NetworkId::from_bytes([7; 32]);
let transport = AwgTransport::new(local, Ipv4Addr::LOCALHOST);
let offer = |version, key, addresses| Negotiation::Offer {
version,
profile: PairProfile::generate(),
key,
addresses,
};
let address = SocketAddr::from((Ipv4Addr::LOCALHOST, 12345));
let encode = |message: &Negotiation| postcard::to_stdvec(message).unwrap();
let valid = encode(&offer(AWG_WIRE_VERSION, [1; 32], vec![address]));
assert!(
transport
.on_control(network, peer, AWG_PROTOCOL, &valid)
.is_err()
);
transport.allow_peer(network, peer);
assert!(
transport
.on_control(network, peer, "wrong-protocol", &valid)
.is_err()
);
assert!(
transport
.on_control(network, peer, AWG_PROTOCOL, &[0; 4097])
.is_err()
);
for bytes in [
encode(&offer(99, [1; 32], vec![address])),
encode(&offer(AWG_WIRE_VERSION, [0; 32], vec![address])),
encode(&offer(AWG_WIRE_VERSION, [1; 32], vec![])),
encode(&offer(AWG_WIRE_VERSION, [1; 32], vec![address; 33])),
] {
assert!(
transport
.on_control(network, peer, AWG_PROTOCOL, &bytes)
.is_err()
);
}
for len in 0..valid.len() {
assert!(
transport
.on_control(network, peer, AWG_PROTOCOL, &valid[..len])
.is_err()
);
}
let wrong_role = AwgTransport::new(peer, Ipv4Addr::LOCALHOST);
wrong_role.allow_peer(network, local);
assert!(
wrong_role
.on_control(network, local, AWG_PROTOCOL, &valid)
.is_err()
);
transport.forget_peer(network, peer);
assert!(
transport
.on_control(network, peer, AWG_PROTOCOL, &valid)
.is_err()
);
assert!(transport.overview(network).is_empty());
transport.shutdown();
wrong_role.shutdown();
}
}
crates/tsunagi-awg/tests/pairs.rs
+773
View File
@@ -0,0 +1,773 @@
//! Real membership, UDP sockets, pair negotiation and nested WireGuard encryption.
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use bytes::Bytes;
use std::{
net::Ipv4Addr,
path::Path,
sync::{
Arc,
atomic::{AtomicBool, AtomicU64, Ordering},
},
time::Duration,
};
use tempfile::TempDir;
use tsunagi::dataplane::IpPlugin;
use tsunagi::dataplane::transport::{
ControlSender, PacketLink, PacketTransport, SharedLink, TransportError,
};
use tsunagi::discovery::SharedMemoryDiscovery;
use tsunagi::overlay::{MemoryTun, MemoryTunFactory};
use tsunagi::testing::{config_with, network, wait_for_peers, wait_until};
use tsunagi::{Agent, BoxFuture, NetworkId};
use tsunagi_awg::{AWG_PROTOCOL, AwgPlugin, AwgTransport};
use tsunagi_wg_quic::WireguardConfig;
struct Node {
agent: Agent,
plugin: Arc<AwgPlugin>,
quic: Option<Arc<tsunagi_wg_quic::WireguardPlugin>>,
transport: Arc<AwgTransport>,
tuns: MemoryTunFactory,
loss: Arc<AtomicBool>,
}
impl Node {
async fn open(root: &Path, discovery: &SharedMemoryDiscovery, interface: &str) -> Self {
Self::open_with_fallback(root, discovery, interface, false).await
}
async fn open_with_fallback(
root: &Path,
discovery: &SharedMemoryDiscovery,
interface: &str,
fallback: bool,
) -> Self {
Self::open_config(root, discovery, interface, fallback, 1280).await
}
async fn open_config(
root: &Path,
discovery: &SharedMemoryDiscovery,
interface: &str,
fallback: bool,
mtu: u32,
) -> Self {
Self::open_selected(root, discovery, interface, fallback, mtu, None).await
}
async fn open_selected(
root: &Path,
discovery: &SharedMemoryDiscovery,
interface: &str,
fallback: bool,
mtu: u32,
default_protocols: Option<Vec<String>>,
) -> Self {
let identity =
tsunagi::storage::StateStore::open(tsunagi::StoragePaths::under(root).state_db())
.unwrap()
.load_or_create_device_identity()
.unwrap();
let transport = AwgTransport::new(identity.endpoint_id(), Ipv4Addr::LOCALHOST);
let plugin = AwgPlugin::open(
WireguardConfig::new(root.join("awg"))
.with_mtu(mtu)
.with_reconcile(Duration::from_millis(20), Duration::from_millis(250)),
transport.clone(),
)
.await
.unwrap();
let loss = Arc::new(AtomicBool::new(false));
let faults = Arc::new(FaultTransport {
inner: transport.clone(),
loss: loss.clone(),
});
let tuns = MemoryTunFactory::new();
let mut config = config_with(root, discovery)
.with_interface(Arc::new(tuns.clone()), interface, mtu)
.with_plugin(plugin.clone() as Arc<dyn IpPlugin>)
.with_custom_transport(AWG_PROTOCOL, faults);
let mut quic_plugin = None;
if fallback {
let quic = tsunagi_wg_quic::WireguardPlugin::open(
WireguardConfig::new(root.join("wg-quic"))
.with_mtu(mtu)
.with_reconcile(Duration::from_millis(20), Duration::from_millis(250)),
)
.await
.unwrap();
config = config.with_plugin(quic.clone());
quic_plugin = Some(quic);
}
config.default_protocols = default_protocols;
let agent = Agent::spawn(config).await.unwrap();
let inbound_agent = agent.clone();
let inbound_transport = transport.clone();
let inbound_loss = loss.clone();
tokio::spawn(async move {
while let Some(mut link) = inbound_transport.next_inbound().await {
link.link = impaired(link.link, inbound_loss.clone());
let _ = inbound_agent.install_inbound_link(link).await;
}
});
Self {
agent,
plugin,
quic: quic_plugin,
transport,
tuns,
loss,
}
}
async fn ready(&self, id: NetworkId, n: usize) {
wait_until("AWG outer paths and end-to-end handshakes", || async {
(self
.transport
.overview(id)
.iter()
.filter(|p| !p.closed && !p.path.contains("establishing"))
.count()
== n
&& self
.plugin
.overview(id)?
.peers
.iter()
.filter(|p| {
p.tunnel
.as_ref()
.is_some_and(|t| t.health.is_up() && t.path.contains("awg UDP via"))
})
.count()
== n)
.then_some(())
})
.await;
}
fn tun(&self) -> Arc<MemoryTun> {
self.tuns
.device(&self.agent.overlay().unwrap().interface)
.unwrap()
}
fn addr(&self, id: NetworkId) -> Ipv4Addr {
self.plugin
.overview(id)
.unwrap()
.overlay_address_v4
.unwrap()
}
}
fn packet(source: Ipv4Addr, dest: Ipv4Addr, len: usize) -> Bytes {
let mut bytes = vec![0; len];
bytes[0] = 0x45;
bytes[2..4].copy_from_slice(&(len as u16).to_be_bytes());
bytes[8] = 64;
bytes[9] = 253;
bytes[12..16].copy_from_slice(&source.octets());
bytes[16..20].copy_from_slice(&dest.octets());
rand::fill(&mut bytes[20..]);
Bytes::from(bytes)
}
async fn exchange(a: &Node, b: &Node, id: NetworkId) {
for len in [64, 1097, 1280] {
let bytes = packet(a.addr(id), b.addr(id), len);
a.tun().push_from_os(bytes.clone());
assert_eq!(
tokio::time::timeout(Duration::from_secs(5), b.tun().pop_to_os())
.await
.unwrap()
.unwrap(),
bytes
);
}
}
#[tokio::test]
async fn a_control_only_agent_enables_real_data_without_restart_and_cancels_stale_attempts() {
let discovery = SharedMemoryDiscovery::new();
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
let a = Node::open_selected(
roots[0].path(),
&discovery,
"initial-none-a",
true,
1280,
Some(vec![]),
)
.await;
let b = Node::open_selected(
roots[1].path(),
&discovery,
"initial-none-b",
true,
1280,
Some(vec![]),
)
.await;
let (name, secret) = network("enable-after-control-only");
let id = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
wait_for_peers(&a.agent, id, 1).await;
wait_for_peers(&b.agent, id, 1).await;
assert!(a.agent.overlay().is_none());
assert!(b.agent.overlay().is_none());
for node in [&a, &b] {
assert!(
node.agent
.network_status(id)
.await
.unwrap()
.protocols
.is_empty()
);
node.agent
.set_protocols(id, vec!["awg".into()])
.await
.unwrap();
assert!(node.agent.overlay().is_some());
}
a.ready(id, 1).await;
b.ready(id, 1).await;
exchange(&a, &b, id).await;
let before = a.agent.overlay().unwrap().interface;
// Changes race unfinished opens and prepared plugin callbacks; the last
// selection must win even if an earlier open has completed in its queue.
for selection in [
vec!["wg-quic".into()],
vec![],
vec!["awg".into()],
vec![],
vec!["awg".into()],
] {
a.agent.set_protocols(id, selection).await.unwrap();
}
a.ready(id, 1).await;
b.ready(id, 1).await;
exchange(&a, &b, id).await;
assert_eq!(a.agent.overlay().unwrap().interface, before);
for node in [&a, &b] {
let status = node.agent.network_status(id).await.unwrap();
assert_eq!(status.protocols, ["awg"]);
assert_eq!(status.metrics.sessions_established, 1);
assert_eq!(status.metrics.disconnects, 0);
}
a.agent.shutdown().await;
b.agent.shutdown().await;
}
#[tokio::test]
async fn runtime_protocol_selection_replaces_real_tunnels_and_survives_restart() {
let discovery = SharedMemoryDiscovery::new();
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
let a = Node::open_with_fallback(roots[0].path(), &discovery, "runtime-a", true).await;
let b = Node::open_with_fallback(roots[1].path(), &discovery, "runtime-b", true).await;
let (name, secret) = network("runtime-protocols");
let id = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
a.ready(id, 1).await;
b.ready(id, 1).await;
exchange(&a, &b, id).await;
let (other_name, other_secret, other_id) = loop {
let (name, secret) = network("unchanged-runtime-network");
let other_id = tsunagi::identity::NetworkKeys::derive(&name, &secret).network_id();
if tsunagi::state::derived_ipv4_range(id) != tsunagi::state::derived_ipv4_range(other_id) {
break (name, secret, other_id);
}
};
a.agent
.join_network(&other_name, &other_secret)
.await
.unwrap();
b.agent
.join_network(&other_name, &other_secret)
.await
.unwrap();
a.ready(other_id, 1).await;
b.ready(other_id, 1).await;
let other_before = a
.agent
.network_status(other_id)
.await
.unwrap()
.metrics
.data_links_established;
for node in [&a, &b] {
let before = node.agent.network_status(id).await.unwrap().protocols;
for invalid in [vec!["unknown".into()], vec!["awg".into(), "awg".into()]] {
assert!(node.agent.set_protocols(id, invalid).await.is_err());
assert_eq!(
node.agent.network_status(id).await.unwrap().protocols,
before
);
}
}
// Exercise updates initiated by either endpoint-id role, including an
// empty set. Membership and the control session stay alive throughout.
for node in [&a, &b] {
node.agent.set_protocols(id, Vec::new()).await.unwrap();
wait_until("peer accepts control-only selection", || async {
let other = if node.agent.endpoint_id() == a.agent.endpoint_id() {
&b
} else {
&a
};
let status = other.agent.network_status(id).await.ok()?;
(status.peers.len() == 1
&& status.peers[0].protocols.is_empty()
&& matches!(status.peers[0].data_path, tsunagi::agent::DataPath::None))
.then_some(())
})
.await;
assert_eq!(node.agent.network_status(id).await.unwrap().peers.len(), 1);
node.agent
.set_protocols(id, vec!["wg-quic".into(), "awg".into()])
.await
.unwrap();
}
a.agent
.set_protocols(id, vec!["wg-quic".into()])
.await
.unwrap();
wait_until("real QUIC replaces AWG", || async {
let a_status = a.agent.network_status(id).await.ok()?;
let b_status = b.agent.network_status(id).await.ok()?;
(a_status.peers[0].protocols == ["wg-quic"]
&& b_status.peers[0].protocols == ["wg-quic"]
&& !matches!(a_status.peers[0].data_path, tsunagi::agent::DataPath::None)
&& !matches!(b_status.peers[0].data_path, tsunagi::agent::DataPath::None)
&& a.transport.overview(id).iter().all(|p| p.closed)
&& b.transport.overview(id).iter().all(|p| p.closed))
.then_some(())
})
.await;
// Get addresses from the system allocation: the disabled AWG plugin no
// longer has a live view of this network.
for (source, dest) in [(&a, &b), (&b, &a)] {
wait_until("QUIC end-to-end handshake", || async {
source
.quic
.as_ref()?
.overview(id)?
.peers
.iter()
.any(|p| p.tunnel.as_ref().is_some_and(|t| t.health.is_up()))
.then_some(())
})
.await;
let addr = |n: &Node| {
n.quic
.as_ref()
.unwrap()
.overview(id)
.unwrap()
.overlay_address_v4
.unwrap()
};
let bytes = packet(addr(source), addr(dest), 1280);
source.tun().push_from_os(bytes.clone());
assert_eq!(
tokio::time::timeout(Duration::from_secs(5), dest.tun().pop_to_os())
.await
.unwrap()
.unwrap(),
bytes
);
}
assert_eq!(
a.agent
.network_status(other_id)
.await
.unwrap()
.metrics
.data_links_established,
other_before
);
exchange(&a, &b, other_id).await;
a.agent.shutdown().await;
let restarted = Node::open_with_fallback(roots[0].path(), &discovery, "runtime-a", true).await;
wait_for_peers(&restarted.agent, id, 1).await;
assert_eq!(
restarted.agent.network_status(id).await.unwrap().protocols,
["wg-quic"]
);
assert_eq!(
b.agent.network_status(other_id).await.unwrap().protocols,
["awg", "wg-quic"]
);
restarted.ready(other_id, 1).await;
b.ready(other_id, 1).await;
exchange(&restarted, &b, other_id).await;
restarted
.agent
.set_protocols(id, vec!["awg".into()])
.await
.unwrap();
restarted.ready(id, 1).await;
b.ready(id, 1).await;
exchange(&restarted, &b, id).await;
restarted.agent.shutdown().await;
b.agent.shutdown().await;
}
#[tokio::test]
async fn real_pairs_carry_full_size_packets_and_restart_with_fresh_profiles() {
tsunagi::testing::init_tracing();
let discovery = SharedMemoryDiscovery::new();
let roots = [
TempDir::new().unwrap(),
TempDir::new().unwrap(),
TempDir::new().unwrap(),
];
let a = Node::open(roots[0].path(), &discovery, "awga").await;
let b = Node::open(roots[1].path(), &discovery, "awgb").await;
let c = Node::open(roots[2].path(), &discovery, "awgc").await;
let (name, secret) = network("awg-pairs");
let id = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
c.agent.join_network(&name, &secret).await.unwrap();
for node in [&a, &b, &c] {
wait_for_peers(&node.agent, id, 2).await;
node.ready(id, 2).await;
}
let ids: std::collections::HashSet<_> = [&a, &b, &c]
.into_iter()
.flat_map(|n| n.transport.overview(id).into_iter().map(|p| p.attempt))
.collect();
assert_eq!(
ids.len(),
3,
"one independent profile for each of three pairs"
);
exchange(&a, &b, id).await;
exchange(&b, &a, id).await;
exchange(&a, &c, id).await;
let old_ports: Vec<_> = a
.transport
.overview(id)
.into_iter()
.filter_map(|p| p.local_port)
.collect();
a.agent.shutdown().await;
for port in old_ports {
wait_until("old pair socket released", || async {
tokio::net::UdpSocket::bind((Ipv4Addr::LOCALHOST, port))
.await
.ok()
})
.await;
}
let restarted = Node::open(roots[0].path(), &discovery, "awga").await;
restarted.ready(id, 2).await;
assert!(
restarted
.transport
.overview(id)
.iter()
.all(|p| !ids.contains(&p.attempt))
);
exchange(&restarted, &b, id).await;
exchange(&c, &restarted, id).await;
restarted.agent.shutdown().await;
b.agent.shutdown().await;
c.agent.shutdown().await;
assert!(
tokio::time::timeout(Duration::from_secs(1), restarted.transport.next_inbound())
.await
.unwrap()
.is_none()
);
}
// Drops datagrams at the link boundary. Membership, control, sockets, crypto,
// addressing and storage remain real. This substitutes a lossy physical path.
#[derive(Debug)]
struct FaultTransport {
inner: Arc<AwgTransport>,
loss: Arc<AtomicBool>,
}
impl PacketTransport for FaultTransport {
fn name(&self) -> &str {
AWG_PROTOCOL
}
fn as_any(&self) -> &dyn std::any::Any {
self
}
fn attach_control(&self, sender: Arc<dyn ControlSender>) {
self.inner.attach_control(sender);
}
fn on_control(
&self,
network: NetworkId,
peer: iroh::EndpointId,
protocol: &str,
bytes: &[u8],
) -> Result<(), TransportError> {
self.inner.on_control(network, peer, protocol, bytes)
}
fn open<'a>(
&'a self,
network: NetworkId,
peer: iroh::EndpointId,
protocol: &'a str,
) -> BoxFuture<'a, Result<SharedLink, TransportError>> {
Box::pin(async move {
Ok(impaired(
self.inner.open(network, peer, protocol).await?,
self.loss.clone(),
))
})
}
}
fn impaired(raw: SharedLink, loss: Arc<AtomicBool>) -> SharedLink {
Arc::new(FaultLink {
raw,
loss,
sequence: AtomicU64::new(0),
})
}
#[derive(Debug)]
struct FaultLink {
raw: SharedLink,
loss: Arc<AtomicBool>,
sequence: AtomicU64,
}
impl PacketLink for FaultLink {
fn network(&self) -> NetworkId {
self.raw.network()
}
fn peer(&self) -> iroh::EndpointId {
self.raw.peer()
}
fn max_datagram_size(&self) -> usize {
self.raw.max_datagram_size()
}
fn send(&self, bytes: Bytes) -> Result<(), TransportError> {
if self.loss.load(Ordering::Relaxed)
&& !self
.sequence
.fetch_add(1, Ordering::Relaxed)
.is_multiple_of(5)
{
Ok(())
} else {
self.raw.send(bytes)
}
}
fn recv(&self) -> BoxFuture<'_, Option<Bytes>> {
self.raw.recv()
}
fn closed(&self) -> BoxFuture<'_, ()> {
self.raw.closed()
}
fn is_closed(&self) -> bool {
self.raw.is_closed()
}
fn close(&self) {
self.raw.close();
}
fn path_description(&self) -> String {
self.raw.path_description()
}
}
#[tokio::test]
async fn persistent_runtime_loss_retires_the_pair_and_negotiates_fresh_parameters() {
let discovery = SharedMemoryDiscovery::new();
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
let a = Node::open(roots[0].path(), &discovery, "lossa").await;
let b = Node::open(roots[1].path(), &discovery, "lossb").await;
let (name, secret) = network("awg-loss");
let id = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
a.ready(id, 1).await;
b.ready(id, 1).await;
exchange(&a, &b, id).await;
let old = a.transport.overview(id)[0].attempt;
let mut events = a.agent.subscribe();
a.loss.store(true, Ordering::Relaxed);
tsunagi::testing::wait_event(&mut events, |event| match event {
tsunagi::Event::DataLinkDown {
network, reason, ..
} if *network == id && reason.contains("delivery checks") => Some(()),
_ => None,
})
.await;
a.loss.store(false, Ordering::Relaxed);
wait_until("fresh AWG profile after runtime loss", || async {
a.transport
.overview(id)
.iter()
.any(|p| p.attempt != old && !p.closed && !p.path.contains("establishing"))
.then_some(())
})
.await;
a.ready(id, 1).await;
b.ready(id, 1).await;
let initiator = if a.agent.endpoint_id().as_bytes() < b.agent.endpoint_id().as_bytes() {
&a
} else {
&b
};
let received = initiator.plugin.overview(id).unwrap().peers[0]
.tunnel
.as_ref()
.unwrap()
.stats
.rx_packets;
let attempt = initiator.transport.overview(id)[0].attempt;
wait_until(
"fresh pair delivery checks answered after installation",
|| async {
let view = initiator.plugin.overview(id)?;
(view.peers.first()?.tunnel.as_ref()?.stats.rx_packets >= received + 3).then_some(())
},
)
.await;
assert_eq!(initiator.transport.overview(id)[0].attempt, attempt);
wait_for_peers(&a.agent, id, 1).await;
exchange(&a, &b, id).await;
exchange(&b, &a, id).await;
a.agent.shutdown().await;
b.agent.shutdown().await;
}
#[tokio::test]
async fn runtime_loss_falls_back_to_real_iroh_without_interrupting_control_or_ip() {
let discovery = SharedMemoryDiscovery::new();
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
let a = Node::open_with_fallback(roots[0].path(), &discovery, "fallba", true).await;
let b = Node::open_with_fallback(roots[1].path(), &discovery, "fallbb", true).await;
let (name, secret) = network("awg-fallback");
let id = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
a.ready(id, 1).await;
b.ready(id, 1).await;
exchange(&a, &b, id).await;
let mut events = a.agent.subscribe();
a.loss.store(true, Ordering::Relaxed);
tsunagi::testing::wait_event(&mut events, |event| match event {
tsunagi::Event::DataLinkDown {
network, reason, ..
} if *network == id && reason.contains("delivery checks") => Some(()),
_ => None,
})
.await;
for node in [&a, &b] {
wait_until("fallback carrier is wg-quic",||async {
let status=node.agent.status().await.ok()?;
let network=status.networks.iter().find(|n|n.network_id==id)?;
(network.members.iter().any(|m|matches!(&m.data_path,tsunagi::agent::DataPath::Direct{transport} if transport=="wg-quic"))).then_some(())
}).await;
wait_for_peers(&node.agent, id, 1).await;
}
exchange(&a, &b, id).await;
exchange(&b, &a, id).await;
a.agent.shutdown().await;
b.agent.shutdown().await;
}
#[tokio::test]
async fn opaque_pair_carrier_fragments_its_full_advertised_capacity() {
let discovery = SharedMemoryDiscovery::new();
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
let mut nodes = Vec::new();
for root in &roots {
let identity = tsunagi::storage::StateStore::open(
tsunagi::StoragePaths::under(root.path()).state_db(),
)
.unwrap()
.load_or_create_device_identity()
.unwrap();
let transport = AwgTransport::new(identity.endpoint_id(), Ipv4Addr::LOCALHOST);
let agent = Agent::spawn(
config_with(root.path(), &discovery)
.with_custom_transport(AWG_PROTOCOL, transport.clone()),
)
.await
.unwrap();
nodes.push((agent, transport));
}
let (name, secret) = network("awg-capacity");
let id = nodes[0].0.join_network(&name, &secret).await.unwrap();
nodes[1].0.join_network(&name, &secret).await.unwrap();
for (i, (agent, transport)) in nodes.iter().enumerate() {
wait_for_peers(agent, id, 1).await;
// This fixture tests only the transport contract: register the member
// after the real core has authenticated it, without an IP plugin/TUN.
transport.allow_peer(id, nodes[1 - i].0.endpoint_id());
}
nodes.sort_by_key(|(agent, _)| *agent.endpoint_id().as_bytes());
let outgoing = nodes[0]
.1
.open(id, nodes[1].0.endpoint_id(), AWG_PROTOCOL)
.await
.unwrap();
let incoming = tokio::time::timeout(Duration::from_secs(5), nodes[1].1.next_inbound())
.await
.unwrap()
.unwrap()
.link;
for (from, to) in [(&outgoing, &incoming), (&incoming, &outgoing)] {
let mut bytes = vec![0; from.max_datagram_size()];
rand::fill(&mut bytes[..]);
let bytes = Bytes::from(bytes);
from.send(bytes.clone()).unwrap();
assert_eq!(
tokio::time::timeout(Duration::from_secs(5), to.recv())
.await
.unwrap()
.unwrap(),
bytes
);
assert!(matches!(
from.send(Bytes::from(vec![0; from.max_datagram_size() + 1])),
Err(TransportError::TooLarge { .. })
));
}
outgoing.close();
incoming.close();
for (agent, transport) in nodes {
transport.shutdown();
agent.shutdown().await;
}
}
#[tokio::test]
async fn encrypted_health_checks_fit_a_smaller_peers_configured_interface_mtu() {
let discovery = SharedMemoryDiscovery::new();
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
let a = Node::open_config(roots[0].path(), &discovery, "smalla", false, 1280).await;
let b = Node::open_config(roots[1].path(), &discovery, "smallb", false, 576).await;
let (name, secret) = network("awg-small-mtu");
let id = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
a.ready(id, 1).await;
b.ready(id, 1).await;
let bytes = packet(a.addr(id), b.addr(id), 576);
a.tun().push_from_os(bytes.clone());
assert_eq!(
tokio::time::timeout(Duration::from_secs(5), b.tun().pop_to_os())
.await
.unwrap()
.unwrap(),
bytes
);
wait_until(
"encrypted delivery checks answered with a small MTU",
|| async {
let view = a.plugin.overview(id)?;
let peer = view.peers.first()?.tunnel.as_ref()?;
(peer.stats.rx_packets >= 3).then_some(())
},
)
.await;
for node in [&a, &b] {
assert_eq!(
node.plugin.overview(id).unwrap().peers[0]
.tunnel
.as_ref()
.unwrap()
.stats
.dropped_oversize,
0
);
}
a.agent.shutdown().await;
b.agent.shutdown().await;
}
crates/tsunagi-awg/tests/profiles.rs
+36
View File
@@ -0,0 +1,36 @@
//! Fresh and bounded masking profiles, and rejection of foreign/truncated wire packets.
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use tsunagi_awg::profile::PairProfile;
#[test]
fn profiles_are_fresh_bounded_and_redacted() {
let profiles: Vec<_> = (0..128).map(|_| PairProfile::generate()).collect();
for (i, profile) in profiles.iter().enumerate() {
profile.validate().unwrap();
assert!(!format!("{profile:?}").contains(&format!("{:?}", profile.header_key())));
for other in &profiles[..i] {
assert_ne!(profile.header_key(), other.header_key());
assert_ne!(profile.id(), other.id());
}
for packet in profile.signature_packets() {
assert!(packet.len() < 1200);
}
}
}
#[test]
fn masked_packets_round_trip_and_reject_other_profiles_and_truncation() {
let profile = PairProfile::generate();
let other = PairProfile::generate();
for (kind, size) in [(1, 148), (2, 92), (3, 64), (4, 800)] {
let mut message = vec![0x7b; size];
message[..4].copy_from_slice(&(kind as u32).to_le_bytes());
let wire = profile.encode(&message).unwrap();
assert_ne!(&wire[..4], &message[..4]);
assert_eq!(profile.decode(&wire).unwrap(), message);
assert!(other.decode(&wire).is_none());
for length in 0..44 {
assert!(profile.decode(&wire[..length]).is_none());
}
}
}
crates/tsunagi-cli/Cargo.toml
-42
View File
@@ -1,42 +0,0 @@
[package]
name = "tsunagi-cli"
version = "0.1.0"
description = "The tsunagi command line agent."
edition.workspace = true
rust-version.workspace = true
license.workspace = true
repository.workspace = true
[[bin]]
name = "tsunagi"
path = "src/main.rs"
# The binary and the library share a name, which `cargo doc` cannot map to
# two directories. The command line is documented by `--help`, not rustdoc,
# so the library keeps the name and this opts out.
doc = false
[dependencies]
tsunagi = { path = "../tsunagi", version = "0.1.0", features = ["tun-device", "dns-publish"] }
tsunagi-wireguard = { path = "../tsunagi-wireguard", version = "0.1.0" }
tsunagi-wg-quic = { path = "../tsunagi-wg-quic", version = "0.1.0" }
tsunagi-tcp-tls = { path = "../tsunagi-tcp-tls", version = "0.1.0" }
iroh.workspace = true
tokio = { version = "1.53", features = ["rt", "rt-multi-thread", "sync", "time", "macros", "signal"] }
tracing.workspace = true
tracing-subscriber.workspace = true
clap = { version = "4.5", features = ["derive", "env"] }
# Already in the tree through clap. `anstream` strips the escapes when stdout
# is not a terminal and turns on virtual terminal processing on Windows, so
# colour is never written where it would show up as rubbish.
anstream = "1.0"
anstyle = "1.0"
# Local interface addresses, for the diagnostics in `status`.
netwatch = "0.19.3"
[dev-dependencies]
tokio = { version = "1.53", features = ["rt", "rt-multi-thread", "sync", "time", "macros", "process"] }
tempfile.workspace = true
simple-dns = "0.12"
[lints]
workspace = true
crates/tsunagi-cli/tests/identity.rs
-56
View File
@@ -1,56 +0,0 @@
//! What the command line says about this device, against a running agent.
//!
//! These drive the real binary, which is why they live beside it rather
//! than with the library's own tests: the library cannot depend on a binary
//! built from a crate that depends on it.
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use tempfile::TempDir;
use tsunagi::Agent;
use tsunagi::config::{AgentConfig, StoragePaths, TransportPolicy};
/// Asking who this device is must not need the directory lock.
///
/// The lock belongs to the one agent allowed to *write* the state. `id` only
/// reads, so making it take the lock would mean the question could never be
/// answered while an agent was running — which is exactly when you want to
/// ask it.
#[tokio::test]
async fn identity_can_be_read_while_an_agent_holds_the_directory() {
let dir = TempDir::new().unwrap();
// Hold the directory the way a running agent does.
let agent = Agent::spawn(
AgentConfig::new(StoragePaths::under(dir.path()))
.with_transport(TransportPolicy::LocalOnly)
.with_loopback_bind(),
)
.await
.unwrap();
let expected = agent.endpoint_id().to_string();
let output = tokio::process::Command::new(env!("CARGO_BIN_EXE_tsunagi"))
.arg("id")
// The same layout `StoragePaths::under` gives the agent above.
.arg("--state-dir")
.arg(dir.path().join("state"))
.arg("--cache-dir")
.arg(dir.path().join("cache"))
.output()
.await
.unwrap();
let stdout = String::from_utf8_lossy(&output.stdout);
let stderr = String::from_utf8_lossy(&output.stderr);
assert!(
output.status.success(),
"`id` failed while an agent was running:\n{stderr}"
);
assert!(
stdout.contains(&expected),
"expected {expected} in:\n{stdout}"
);
agent.shutdown().await;
}
crates/tsunagi-cli/tests/network_cli.rs
-420
View File
@@ -1,420 +0,0 @@
//! Making and joining a network from the command line, against a real agent.
//!
//! The rules here are small and easy to get wrong in a way no unit test
//! notices: what a bare network name means, when a secret is invented, and
//! whether what is printed is enough for the other person to paste.
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use std::time::Duration;
use tempfile::TempDir;
/// An agent running as a real process, with its own directories.
struct Running {
child: std::process::Child,
dir: TempDir,
}
impl Drop for Running {
fn drop(&mut self) {
let _ = self.child.kill();
let _ = self.child.wait();
}
}
impl Running {
fn run(&self, args: &[&str]) -> std::process::Output {
run_in(&self.dir, args)
}
}
fn run_in(dir: &TempDir, args: &[&str]) -> std::process::Output {
std::process::Command::new(env!("CARGO_BIN_EXE_tsunagi"))
.args(args)
.arg("--state-dir")
.arg(dir.path().join("state"))
.arg("--cache-dir")
.arg(dir.path().join("cache"))
.output()
.expect("the agent binary runs")
}
fn read_networks(dir: &TempDir) -> Vec<tsunagi::storage::StoredNetwork> {
tsunagi::storage::StateStore::open(dir.path().join("state/state.sqlite"))
.unwrap()
.list_networks()
.unwrap()
}
/// An agent with no network at all, the way a daemon is started before
/// anything has been decided.
fn start_bare(port: u16) -> Running {
let dir = TempDir::new().unwrap();
let child = std::process::Command::new(env!("CARGO_BIN_EXE_tsunagi"))
.arg("up")
.arg("--state-dir")
.arg(dir.path().join("state"))
.arg("--cache-dir")
.arg(dir.path().join("cache"))
.args(["--reach", "local", "--no-tun", "--no-dns"])
.arg("--bind")
.arg(format!("127.0.0.1:{port}"))
.args(["--log", "error", "--status-interval", "0"])
.stdout(std::process::Stdio::null())
.stderr(std::process::Stdio::null())
.spawn()
.expect("the agent binary starts");
std::thread::sleep(Duration::from_secs(2));
Running { child, dir }
}
/// An agent already in one network, the way most of these start.
fn start(network: &str, port: u16) -> Running {
let agent = start_bare(port);
let joined = agent.run(&[
"join",
"--network",
network,
"--secret",
"a-secret-for-the-cli-test",
]);
assert!(
joined.status.success(),
"{}",
String::from_utf8_lossy(&joined.stderr)
);
agent
}
#[test]
fn joining_with_no_secret_makes_one_and_prints_what_to_send() {
// The hurry case: a network with somebody, for as long as it is needed,
// and then gone. Asking for a secret first is a step with no purpose,
// and a secret that is not printed is a network nobody else can join.
let agent = start("resident", 45071);
let joined = agent.run(&["network", "join", "--network", "spontaneous"]);
assert!(
joined.status.success(),
"joining failed: {}",
String::from_utf8_lossy(&joined.stderr)
);
let out = String::from_utf8_lossy(&joined.stdout);
assert!(out.contains("joined `spontaneous`"), "{out}");
// The secret in full, and one line the other side can paste as it is.
let secret = out
.lines()
.find_map(|line| line.trim().strip_prefix("secret "))
.expect("the invented secret is printed");
assert!(secret.starts_with("tsn1"), "{out}");
let command = out
.lines()
.find(|line| line.contains("tsunagi join --network spontaneous"))
.expect("a command to send");
assert!(command.contains(secret), "with the secret in it: {out}");
// And where to find this device, for an agent that is not up yet.
assert!(out.contains("--peer "), "somewhere to find us: {out}");
// Joining the same name again resumes it rather than making another
// network that merely looks the same.
let again = agent.run(&["network", "join", "--network", "spontaneous"]);
let out = String::from_utf8_lossy(&again.stdout);
assert!(again.status.success());
assert!(out.contains("already here"), "{out}");
assert!(!out.contains("secret tsn1"), "no second secret: {out}");
}
#[test]
fn a_bare_name_resumes_the_network_of_that_name_rather_than_inventing_one() {
// `join --network resident` with no secret is the same rule: this device
// has exactly one network of that name, so that is the one meant.
let agent = start("resident", 45072);
let listed = agent.run(&["network"]);
let before = String::from_utf8_lossy(&listed.stdout).to_string();
let id = before
.lines()
.find(|line| line.contains("resident"))
.and_then(|line| line.split_whitespace().nth(1))
.expect("the network is listed")
.to_string();
let joined = agent.run(&["network", "join", "--network", "resident"]);
assert!(joined.status.success());
let out = String::from_utf8_lossy(&joined.stdout);
assert!(out.contains(&id), "the same network, not a new one: {out}");
assert!(out.contains("already here"), "{out}");
}
#[test]
fn a_network_can_be_stopped_and_resumed_without_losing_anything() {
// Leaving gives everything up; stopping is being away. The difference
// is what somebody wants when they will be back.
let agent = start("resident", 45073);
let listed = agent.run(&["network"]);
let out = String::from_utf8_lossy(&listed.stdout).to_string();
let id = out
.lines()
.find(|line| line.contains("resident"))
.and_then(|line| line.split_whitespace().nth(1))
.expect("the network is listed")
.to_string();
let secret_before = agent.run(&["network", "secret", &id[..10]]);
let secret_before = String::from_utf8_lossy(&secret_before.stdout)
.trim()
.to_string();
assert!(secret_before.starts_with("tsn1"));
let stopped = agent.run(&["network", "stop", "resident"]);
assert!(
stopped.status.success(),
"{}",
String::from_utf8_lossy(&stopped.stderr)
);
assert!(
String::from_utf8_lossy(&stopped.stdout).contains("stopped `resident`"),
"{}",
String::from_utf8_lossy(&stopped.stdout)
);
// Still configured, and said to be stopped rather than missing.
let listed = String::from_utf8_lossy(&agent.run(&["network"]).stdout).to_string();
assert!(listed.contains(&id), "still configured: {listed}");
assert!(listed.contains("stopped"), "{listed}");
// Stopping what is stopped is the state asked for, not an error.
let again = agent.run(&["network", "stop", &id[..10]]);
assert!(again.status.success());
assert!(
String::from_utf8_lossy(&again.stdout).contains("already stopped"),
"{}",
String::from_utf8_lossy(&again.stdout)
);
let started = agent.run(&["network", "start", "resident"]);
assert!(started.status.success());
assert!(
String::from_utf8_lossy(&started.stdout).contains("started `resident`"),
"{}",
String::from_utf8_lossy(&started.stdout)
);
let listed = String::from_utf8_lossy(&agent.run(&["network"]).stdout).to_string();
assert!(listed.contains("running"), "{listed}");
// And nothing was given up on the way: the same network, same secret.
let secret_after = agent.run(&["network", "secret", "resident"]);
assert_eq!(
String::from_utf8_lossy(&secret_after.stdout).trim(),
secret_before,
"the secret is kept, so this is the same network"
);
}
#[test]
fn an_agent_starts_with_no_network_and_takes_one_later() {
// A daemon in one terminal and the deciding done in another: the
// agent is the identity, and which networks it is in is a separate
// question it can answer at any time.
let agent = start_bare(45074);
let listed = agent.run(&["network"]);
assert!(listed.status.success());
assert!(
String::from_utf8_lossy(&listed.stderr).contains("no network has been joined"),
"{}",
String::from_utf8_lossy(&listed.stderr)
);
let joined = agent.run(&["network", "join", "--network", "afterwards"]);
assert!(
joined.status.success(),
"{}",
String::from_utf8_lossy(&joined.stderr)
);
let listed = String::from_utf8_lossy(&agent.run(&["network"]).stdout).to_string();
assert!(listed.contains("afterwards"), "{listed}");
assert!(listed.contains("running"), "and it is live: {listed}");
}
#[test]
fn up_is_the_agent_and_takes_no_network_at_all() {
// The two commands are separate on purpose: `up` runs the device's
// agent, `join` decides what it is in. A network on `up` would be a
// second way to do the same thing, and the one that cannot be undone
// without a restart.
for argument in ["--network", "--secret"] {
let out = std::process::Command::new(env!("CARGO_BIN_EXE_tsunagi"))
.args(["up", argument, "whatever"])
.arg("--state-dir")
.arg(TempDir::new().unwrap().path().join("state"))
.output()
.expect("the agent binary runs");
assert!(
!out.status.success(),
"`up {argument}` should not be a thing"
);
assert!(
String::from_utf8_lossy(&out.stderr).contains("unexpected argument"),
"{}",
String::from_utf8_lossy(&out.stderr)
);
}
}
#[test]
fn broadcast_choice_is_per_network_live_persistent_and_preserved_by_rejoin() {
let agent = start_bare(0);
for args in [
vec!["join", "-n", "broadcast-default"],
vec!["join", "-n", "broadcast-off", "--no-broadcast"],
] {
let result = agent.run(&args);
assert!(
result.status.success(),
"{}",
String::from_utf8_lossy(&result.stderr)
);
}
let read = || {
tsunagi::storage::StateStore::open(agent.dir.path().join("state/state.sqlite"))
.unwrap()
.list_networks()
.unwrap()
};
let networks = read();
let enabled = networks
.iter()
.find(|n| n.name.as_str() == "broadcast-default")
.unwrap();
let disabled = networks
.iter()
.find(|n| n.name.as_str() == "broadcast-off")
.unwrap();
assert!(enabled.broadcast);
assert!(!disabled.broadcast);
assert!(agent.run(&["join", "-n", "broadcast-off"]).status.success());
assert!(
!read()
.iter()
.find(|n| n.name.as_str() == "broadcast-off")
.unwrap()
.broadcast
);
let id = disabled.network_id.to_string();
let change = agent.run(&["network", "broadcast", "broadcast-off", "on"]);
assert!(
change.status.success(),
"{}",
String::from_utf8_lossy(&change.stderr)
);
assert!(read().iter().all(|n| n.broadcast));
let change = agent.run(&["join", "-n", "broadcast-off", "--no-broadcast"]);
assert!(
change.status.success(),
"{}",
String::from_utf8_lossy(&change.stderr)
);
let status = agent.run(&["network", "broadcast", &id]);
assert!(String::from_utf8_lossy(&status.stdout).contains("broadcast off"));
assert!(
!agent
.run(&["join", "-n", "conflict", "--broadcast", "--no-broadcast"])
.status
.success()
);
}
#[test]
fn leaving_by_name_removes_only_that_network_from_a_running_agent() {
let agent = start("resident", 0);
assert!(agent.run(&["join", "-n", "games"]).status.success());
let resident = read_networks(&agent.dir)
.into_iter()
.find(|network| network.name.as_str() == "resident")
.unwrap();
let left = agent.run(&["network", "leave", "games"]);
assert!(
left.status.success(),
"{}",
String::from_utf8_lossy(&left.stderr)
);
assert!(String::from_utf8_lossy(&left.stdout).contains("left `games`"));
let remaining = read_networks(&agent.dir);
assert_eq!(remaining.len(), 1);
assert_eq!(remaining[0].network_id, resident.network_id);
assert_eq!(
remaining[0].secret.encode().as_str(),
resident.secret.encode().as_str()
);
let status = agent.run(&["network"]);
assert!(status.status.success());
let out = String::from_utf8_lossy(&status.stdout);
assert!(out.contains("resident") && out.contains("running"), "{out}");
assert!(!out.contains("games"), "{out}");
}
#[test]
fn leaving_by_name_offline_keeps_the_explicit_offline_requirement() {
let dir = TempDir::new().unwrap();
for name in ["resident", "games"] {
assert!(run_in(&dir, &["join", "-n", name]).status.success());
}
let refused = run_in(&dir, &["network", "leave", "games"]);
assert!(!refused.status.success());
assert!(String::from_utf8_lossy(&refused.stderr).contains("--offline"));
assert_eq!(read_networks(&dir).len(), 2);
let left = run_in(&dir, &["network", "leave", "games", "--offline"]);
assert!(
left.status.success(),
"{}",
String::from_utf8_lossy(&left.stderr)
);
let remaining = read_networks(&dir);
assert_eq!(remaining.len(), 1);
assert_eq!(remaining[0].name.as_str(), "resident");
}
#[test]
fn ambiguous_and_unknown_names_leave_every_network_untouched() {
let agent = start_bare(0);
for secret in [
"first-secret-for-the-cli-test",
"second-secret-for-the-cli-test",
] {
assert!(
agent
.run(&["join", "-n", "games", "-s", secret])
.status
.success()
);
}
let before = read_networks(&agent.dir);
assert_eq!(before.len(), 2);
for name in ["games", "missing-network"] {
let result = agent.run(&["network", "leave", name]);
assert!(!result.status.success());
let err = String::from_utf8_lossy(&result.stderr);
if name == "games" {
assert!(err.contains("matches 2 networks"), "{err}");
for network in &before {
assert!(err.contains(&network.network_id.to_string()), "{err}");
assert!(!err.contains(network.secret.encode().as_str()), "{err}");
}
} else {
assert!(err.contains("no configured network"), "{err}");
}
assert_eq!(read_networks(&agent.dir).len(), 2);
}
// The suggested full id still selects exactly one of the same-name networks.
let left = agent.run(&["network", "leave", &before[0].network_id.to_string()]);
assert!(
left.status.success(),
"{}",
String::from_utf8_lossy(&left.stderr)
);
let remaining = read_networks(&agent.dir);
assert_eq!(remaining.len(), 1);
assert_eq!(remaining[0].network_id, before[1].network_id);
}
crates/tsunagi-core/Cargo.toml
+113
View File
@@ -0,0 +1,113 @@
[package]
name = "tsunagi-core"
version = "0.1.0"
description = "Small private mesh networks: persistent agent identity, deterministic network spaces, an iroh control plane, signed state and a local DNS view."
readme = "../../README.md"
keywords = ["mesh", "p2p", "iroh", "networking"]
categories = ["network-programming"]
edition.workspace = true
rust-version.workspace = true
license.workspace = true
repository.workspace = true
[lib]
name = "tsunagi"
[features]
default = []
# A real TUN device, so a plugin can carry actual IP traffic. On Linux it
# needs CAP_NET_ADMIN; on Windows the process must run elevated and
# `wintun.dll` must be reachable. Without either the in-memory device serves
# tests. The platform-specific dependencies are declared per target, so
# enabling this only pulls what the target it is built for actually uses.
tun-device = [
"dep:tun",
"dep:rtnetlink",
"dep:caps",
"dep:futures-util",
]
# Telling the operating system where to send its DNS questions. systemd-resolved
# on Linux, the Name Resolution Policy Table on Windows; the zone and the
# server work without it either way.
dns-publish = ["dep:zbus"]
# The test harness in `testing`, for this crate's own tests and for a
# protocol crate's. Not on by default: it is only of use to a test.
testing = ["dep:tempfile", "dep:tracing-subscriber"]
[dependencies]
arc-swap = "1.9"
iroh.workspace = true
# `net` for the local control interface: a Unix socket on Unix, a named pipe
# on Windows. Both live behind the same `net` feature.
tokio = { workspace = true, features = ["net"] }
tokio-stream = "0.1"
serde.workspace = true
postcard.workspace = true
bytes.workspace = true
thiserror.workspace = true
tracing.workspace = true
hex.workspace = true
data-encoding.workspace = true
rusqlite = { version = "0.40", features = ["bundled"] }
hkdf = "0.13"
hmac = "0.13"
sha2 = "0.11"
subtle = "2.6"
zeroize = { version = "1.9", features = ["derive"] }
rand = "0.10"
# Already in the tree through iroh. A packet codec, not a DNS server: the
# zone logic is ours and a full server framework would be a large dependency
# for answering A records from memory.
simple-dns = "0.12"
# The IANA top-level domain list, compiled in: one function, no
# dependencies, no network. Used only to warn that a zone name shadows a
# real public domain, never to refuse one.
tld = "2.40"
fs4 = { version = "1.1", features = ["sync"] }
directories = "6.0"
# The real system hostname, without a libc call of our own: this crate is
# forbidden `unsafe` and will not make one.
gethostname = "1.1"
netwatch = "0.19.3"
netdev = "0.46"
mainline = "8.0"
futures-lite = "2.6"
tun = { version = "0.8", features = ["async"], optional = true }
# Only for the `testing` harness, which a protocol crate's tests use too.
tempfile = { version = "3.24", optional = true }
tracing-subscriber = { version = "0.3", features = ["env-filter"], optional = true }
# Linux-only interface provisioning. `rtnetlink` configures the interface in
# process, so no `ip` invocation is ever needed; `caps` keeps CAP_NET_ADMIN
# out of the effective set except during the moments it is used.
[target.'cfg(target_os = "linux")'.dependencies]
rtnetlink = { version = "0.23", optional = true }
# Pure Rust D-Bus, no libdbus to link against. `tokio` rather than the
# default reactor, because the agent brings its own.
zbus = { version = "5.19", default-features = false, features = ["tokio"], optional = true }
caps = { version = "0.5", optional = true }
futures-util = { version = "0.3", default-features = false, optional = true }
# Windows-only interface provisioning. The Wintun adapter itself is created
# and driven through the `tun` dependency, whose safe wrapper is the only way
# in — loading `wintun.dll` is an `unsafe` call this crate forbids. `netdev`
# (already in the tree through `netwatch`) reads the interface table through a
# safe wrapper, which is how the provisioner observes an adapter by name
# before deciding what to change. Creating an adapter needs the process to be
# elevated and `wintun.dll` to be reachable at run time.
[target.'cfg(target_os = "windows")'.dependencies]
windows-sys = { version = "0.61", features = [
"Win32_Security",
"Win32_Security_Authorization",
"Win32_Foundation",
] }
[dev-dependencies]
# Its own tests use the harness it publishes.
tsunagi-core = { path = ".", features = ["testing"] }
tokio = { version = "1.53", features = ["rt", "rt-multi-thread", "sync", "time", "macros", "process", "test-util"] }
tempfile.workspace = true
tracing-subscriber.workspace = true
[lints]
workspace = true
crates/tsunagi/src/agent/events.rs → crates/tsunagi-core/src/agent/events.rs
+13
View File
@@ -129,6 +129,19 @@ pub enum Event {
/// Why it is not up.
reason: String,
},
/// The name this agent claimed was claimed first by another member.
///
/// Nothing stops working: this agent keeps its address and every
/// connection. It just is not found by that name until it is renamed or
/// the member that holds the name lets it go.
HostnameTaken {
/// The network.
network: NetworkId,
/// The name that is taken.
name: String,
/// The member that holds it.
holder: EndpointId,
},
/// An IP plugin reported an error. Never fatal.
PluginError {
/// The network the call was scoped to.
crates/tsunagi/src/agent/mod.rs → crates/tsunagi-core/src/agent/mod.rs
+622 -163
View File
File diff suppressed because it is too large. Load diff
crates/tsunagi/src/agent/network.rs → crates/tsunagi-core/src/agent/network.rs
+1317 -178
View File
File diff suppressed because it is too large. Load diff
crates/tsunagi/src/agent/session.rs → crates/tsunagi-core/src/agent/session.rs
+6 -2
View File
@@ -70,10 +70,12 @@ pub(crate) struct Session {
pub(crate) conn: Connection,
/// Already encoded frame payloads, so the runtime can account for the exact
/// number of control bytes it queues.
pub(crate) outbound: mpsc::Sender<Vec<u8>>,
pub(crate) outbound: mpsc::Sender<zeroize::Zeroizing<Vec<u8>>>,
pub(crate) hostname: Option<String>,
pub(crate) capabilities: Vec<PluginCapability>,
pub(crate) broadcast: bool,
/// The peer offers itself as an exit node in this network.
pub(crate) exit_node: bool,
pub(crate) messages_sent: u64,
pub(crate) messages_received: u64,
pub(crate) bytes_sent: u64,
@@ -228,6 +230,7 @@ pub(crate) fn spawn(
hostname: None,
capabilities: Vec::new(),
broadcast: false,
exit_node: false,
messages_sent: 0,
messages_received: 0,
bytes_sent: 0,
@@ -241,7 +244,7 @@ pub(crate) fn spawn(
async fn writer_task(
mut send: SendStream,
mut outbound: mpsc::Receiver<Vec<u8>>,
mut outbound: mpsc::Receiver<zeroize::Zeroizing<Vec<u8>>>,
limits: Arc<Limits>,
shutdown: Shutdown,
parent: Shutdown,
@@ -328,6 +331,7 @@ async fn reader_task(
}
};
let payload = zeroize::Zeroizing::new(payload);
let bytes = payload.len();
let envelope: Envelope = match decode(&payload) {
Ok(envelope) => envelope,
crates/tsunagi/src/agent/shutdown.rs → crates/tsunagi-core/src/agent/shutdown.rs
View File
File renamed without changes.
crates/tsunagi/src/agent/status.rs → crates/tsunagi-core/src/agent/status.rs
+89 -6
View File
@@ -59,6 +59,43 @@ pub struct OverlayStatus {
pub broadcast_rules: Option<crate::overlay::BroadcastRulesReport>,
}
/// How packets reach a member right now.
///
/// Being reachable is one thing and being reachable well is another, and
/// the second is never a reason to call a member away: a member that can only
/// be reached through somebody else is online, over a longer path.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum DataPath {
/// Nothing carries packets to it yet. The member is connected for
/// control only, which is the state of a data path being set up.
None,
/// A direct link, of the named kind of transport.
Direct {
/// The kind of transport carrying it, e.g. `wg`.
transport: String,
},
/// Through other members.
Relayed {
/// Number of links on the way, at least two.
hops: u8,
/// The member the first of them goes to.
via: EndpointId,
},
}
/// A name that a member claimed and somebody else holds.
///
/// A name belongs to whoever claimed it first. The member that claimed it
/// later is still a member and still reachable by its address; it is just
/// not found by that name, on any member of the network.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct HostnameConflict {
/// The name that is taken.
pub name: String,
/// The member that holds it.
pub holder: EndpointId,
}
/// A member the signed state knows about, connected or not.
///
/// This is the durable roster: it comes from signed records, so a member that
@@ -71,6 +108,8 @@ pub struct OverlayStatus {
/// connected.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct MemberStatus {
/// Current signed claim version, used to bind cleanup confirmation.
pub record_version: u64,
/// The member's device identity.
pub endpoint_id: EndpointId,
/// The IPv4 overlay address it claimed and signed for.
@@ -80,6 +119,20 @@ pub struct MemberStatus {
/// Signed, so it is still known while the member is away — which is what
/// lets an absent member be named rather than shown as a bare id.
pub hostname: Option<String>,
/// The last name this agent saw it announce, whether or not it is here.
///
/// Not signed, so it is a convenience for naming a member that is away
/// and never a claim: the zone is served from `hostname`.
pub last_hostname: Option<String>,
/// The name it claimed, when somebody else claimed it first.
pub hostname_conflict: Option<HostnameConflict>,
/// Whether this agent can reach it now, by any path: a live control
/// session, or a data path through other members. The one answer to "is
/// it online" for every client, so that none of them has to work it out
/// from a different part of the picture. This agent counts as online.
pub online: bool,
/// How packets reach it. [`DataPath::None`] while it is offline.
pub data_path: DataPath,
}
/// Status of one authenticated session.
@@ -87,6 +140,8 @@ pub struct MemberStatus {
pub struct PeerStatus {
/// This authenticated peer accepts network broadcasts.
pub broadcast: bool,
/// This authenticated peer offers itself as an exit node.
pub exit_node: bool,
/// Authenticated endpoint id.
pub endpoint_id: EndpointId,
/// Which side this agent played in the handshake.
@@ -121,6 +176,8 @@ pub struct PeerStatus {
pub control_bytes_sent: u64,
/// Control payload bytes read from this session.
pub control_bytes_received: u64,
/// How packets reach this peer right now.
pub data_path: DataPath,
}
/// Counters scoped to one logical network.
@@ -154,11 +211,36 @@ pub struct NetworkMetrics {
pub data_link_failures: u64,
}
/// An active network preventing another network from starting locally.
#[derive(Debug, Clone)]
pub struct RangeConflict {
/// The blocking network's public identifier.
pub network_id: NetworkId,
/// Its configured name.
pub name: NetworkName,
/// Its overlapping overlay range.
pub range: crate::state::Ipv4Range,
}
/// Status of one network.
#[derive(Debug, Clone)]
pub struct NetworkStatus {
/// Shared service configuration with this client's routing observations.
pub vips: Vec<crate::vip::VipStatus>,
/// Enabled local data-plane protocols, in preference order.
pub protocols: Vec<String>,
/// Whether this network is requested to start or resume automatically.
pub auto_start: bool,
/// Local broadcast participation in this network.
pub broadcast: bool,
/// This agent offers itself as an exit node in this network.
pub exit_offer: bool,
/// The member this device sends all its internet traffic through.
pub exit_via: Option<EndpointId>,
/// Whether that member is connected and still offering to be one.
pub exit_via_online: bool,
/// What the host rules for the exit settings are doing.
pub exit_rules: crate::overlay::ExitRulesReport,
/// Immutable deterministic description of the network space.
pub descriptor: NetworkDescriptor,
/// Network name, for convenience.
@@ -173,15 +255,16 @@ pub struct NetworkStatus {
pub candidates: Vec<CandidateStatus>,
/// Members the signed state knows about, whether connected or not.
pub members: Vec<MemberStatus>,
/// This agent's own name, when another member claimed it first and so
/// holds it. This agent keeps working by address; nobody finds it by
/// this name until it is renamed or the other member lets it go.
pub hostname_conflict: Option<HostnameConflict>,
/// The overlay range this network uses, once it has one.
pub range: Option<crate::state::Ipv4Range>,
/// The range it could not have, because another network on this agent
/// already holds it.
///
/// One agent has one interface, so an address belongs to one network.
/// This network waits to adopt whatever its members settle on instead of
/// proposing something it could not route.
/// The first overlapping active range; `range_conflicts` identifies all blockers.
pub range_conflict: Option<crate::state::Ipv4Range>,
/// Active networks that would be suspended by switching to this network.
pub range_conflicts: Vec<RangeConflict>,
/// Per-network counters.
pub metrics: NetworkMetrics,
/// What has gone through a peer in the middle, in both directions.
crates/tsunagi/src/config.rs → crates/tsunagi-core/src/config.rs
+42 -2
View File
@@ -158,6 +158,22 @@ pub enum TransportPolicy {
/// object where that is possible (notably [`Limits::max_frame_len`]).
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Limits {
/// Maximum active VIP identities per network.
pub max_vips: usize,
/// Maximum permanent VIP deletion tombstones per network.
pub max_vip_tombstones: usize,
/// Maximum concurrent active TCP port checks across the agent.
pub max_vip_checks: usize,
/// Maximum signed fields per VIP, including removed fields.
pub max_vip_fields: usize,
/// Maximum listeners per VIP.
pub max_vip_listeners: usize,
/// Maximum backend endpoints per listener.
pub max_vip_backends: usize,
/// Maximum simultaneous local VIP flow bindings.
pub max_vip_flows: usize,
/// Maximum incomplete IPv4 datagrams per network in VIP reassembly.
pub max_vip_fragments: usize,
/// Largest accepted control frame payload, in bytes.
pub max_frame_len: usize,
/// Largest accepted hostname, in bytes.
@@ -166,12 +182,16 @@ pub struct Limits {
pub max_capabilities: usize,
/// Largest opaque plugin payload, in bytes.
pub max_capability_data_len: usize,
/// Largest private transport negotiation payload (also hard-capped at 4096 bytes).
pub max_transport_payload_len: usize,
/// Largest accepted echo payload in a ping/pong exchange, in bytes.
pub max_echo_payload_len: usize,
/// Largest accepted free-text reason string, in bytes.
pub max_reason_len: usize,
/// Largest number of signed records accepted in one snapshot.
pub max_state_records: usize,
/// Maximum durable retirement markers retained per network.
pub max_retirements: usize,
/// Deadline for the whole handshake.
pub handshake_timeout: Duration,
/// Deadline for one outbound dial attempt.
@@ -201,13 +221,23 @@ pub struct Limits {
impl Default for Limits {
fn default() -> Self {
Self {
max_vips: 64,
max_vip_tombstones: 1024,
max_vip_checks: 32,
max_vip_fields: 192,
max_vip_listeners: 16,
max_vip_backends: 32,
max_vip_flows: 4096,
max_vip_fragments: 64,
max_frame_len: 64 * 1024,
max_hostname_len: 255,
max_capabilities: 16,
max_transport_payload_len: 4096,
max_capability_data_len: 4 * 1024,
max_echo_payload_len: 4 * 1024,
max_reason_len: 256,
max_state_records: crate::state::MAX_RECORDS_PER_MESSAGE,
max_retirements: 1024,
handshake_timeout: Duration::from_secs(10),
dial_timeout: Duration::from_secs(10),
write_timeout: Duration::from_secs(30),
@@ -325,6 +355,9 @@ pub struct AgentConfig {
pub reconnect: ReconnectPolicy,
/// IP plugins whose capabilities are announced and dispatched.
pub plugins: Vec<SharedPlugin>,
/// Initial ordered selection for networks without a saved override.
/// `None` enables all registered plugins; an empty list is control-only.
pub default_protocols: Option<Vec<String>>,
/// Custom packet transports registered per protocol.
pub custom_transports:
std::collections::HashMap<String, Arc<dyn crate::dataplane::PacketTransport>>,
@@ -339,7 +372,11 @@ pub struct AgentConfig {
pub interface_name: String,
/// The interface MTU.
pub interface_mtu: u32,
/// The IPv4 overlay range this agent proposes.
/// Whether new networks derive their IPv4 range from their network id.
///
/// Enabled by default. Explicit ranges and IPv4 opt-out disable this.
pub overlay_ipv4_auto: bool,
/// The IPv4 overlay range this agent proposes when automatic selection is off.
///
/// Addresses are allocated from it and recorded in signed state, so a
/// participant keeps the same one across restarts. A network that has
@@ -369,11 +406,13 @@ impl AgentConfig {
limits: Limits::default(),
reconnect: ReconnectPolicy::default(),
plugins: Vec::new(),
default_protocols: None,
custom_transports: std::collections::HashMap::new(),
tun_factory: None,
interface_name: "tsun0".to_string(),
interface_mtu: 1280,
overlay_ipv4_range: Some(crate::state::DEFAULT_IPV4_RANGE),
overlay_ipv4_auto: true,
overlay_ipv4_range: None,
}
}
@@ -445,6 +484,7 @@ impl AgentConfig {
/// Sets the IPv4 overlay range this agent proposes, or disables IPv4.
pub fn with_overlay_ipv4_range(mut self, range: Option<crate::state::Ipv4Range>) -> Self {
self.overlay_ipv4_auto = false;
self.overlay_ipv4_range = range;
self
}
crates/tsunagi/src/dataplane/mod.rs → crates/tsunagi-core/src/dataplane/mod.rs
+27
View File
@@ -40,6 +40,33 @@ pub use tunnel::{GenericTunnelPlugin, TunnelCodec};
/// Maximum length of a plugin protocol identifier.
pub const MAX_PROTOCOL_ID_LEN: usize = 32;
/// Validates bounded local protocol preferences before storing or applying them.
pub(crate) fn validate_protocols(protocols: &[String], max: usize) -> crate::error::Result<()> {
if protocols.len() > max {
return Err(crate::Error::InvalidProtocols(format!(
"at most {max} protocols are allowed"
)));
}
for (index, name) in protocols.iter().enumerate() {
if name.is_empty()
|| name.len() > MAX_PROTOCOL_ID_LEN
|| !name
.bytes()
.all(|c| c.is_ascii_alphanumeric() || c == b'-' || c == b'_')
{
return Err(crate::Error::InvalidProtocols(
"names must be 1–32 ASCII letters, digits, hyphens or underscores".into(),
));
}
if protocols[..index].contains(name) {
return Err(crate::Error::InvalidProtocols(format!(
"`{name}` is listed more than once"
)));
}
}
Ok(())
}
/// An announcement of one IP plugin's capability.
///
/// `data` is opaque to the core. Nothing in it may be interpreted as a shell
crates/tsunagi/src/dataplane/relay.rs → crates/tsunagi-core/src/dataplane/relay.rs
+270 -130
View File
@@ -67,10 +67,41 @@ impl Tally {
}
}
/// The links to one neighbour, best first. Which transport carries a frame
/// is decided here and nowhere else: the end-to-end protocol is in the
/// envelope and never depends on it.
#[derive(Debug)]
struct NextHop {
peer: PeerId,
link: SharedLink,
links: Box<[SharedLink]>,
}
impl NextHop {
fn usable(&self) -> bool {
self.links.iter().any(|link| !link.is_closed())
}
fn first_open(&self) -> Option<&SharedLink> {
self.links.iter().find(|link| !link.is_closed())
}
/// Hands a frame to the best open link that accepts it. A link that
/// cannot carry the frame (too large for its path, closed a moment ago)
/// leaves it to the next one; the frame is only copied when there is
/// another to try.
fn send(&self, frame: Bytes) -> Result<(), TransportError> {
let mut open = self.links.iter().filter(|link| !link.is_closed());
let Some(mut current) = open.next() else {
return Err(TransportError::Closed);
};
for next in open {
if current.send(frame.clone()).is_ok() {
return Ok(());
}
current = next;
}
current.send(frame)
}
}
#[derive(Debug)]
@@ -82,21 +113,22 @@ struct ForwardRoute {
impl ForwardRoute {
fn select(&self, source: &PeerId, flow: FlowId) -> Option<&NextHop> {
if self.next.len() == 1 {
return self.next.first().filter(|hop| !hop.link.is_closed());
return self.next.first().filter(|hop| hop.usable());
}
let seed = u64::from_le_bytes(source[..8].try_into().ok()?);
let start = mix64(flow ^ seed) as usize % self.next.len();
// Stable fallback if a link closed just before its table was replaced.
(0..self.next.len())
.map(|i| &self.next[(start + i) % self.next.len()])
.find(|hop| !hop.link.is_closed())
.find(|hop| hop.usable())
}
}
#[derive(Debug, Default)]
struct ForwardingTable {
routes: HashMap<PeerId, ForwardRoute>,
inboxes: HashMap<PeerId, mpsc::Sender<Bytes>>,
/// Local delivery, by who sent it and which end-to-end protocol it is.
inboxes: HashMap<(PeerId, u64), mpsc::Sender<Bytes>>,
members: HashSet<PeerId>,
}
@@ -121,7 +153,7 @@ impl Plane {
return;
}
if header.destination == self.local {
let Some(inbox) = table.inboxes.get(&header.source) else {
let Some(inbox) = table.inboxes.get(&(header.source, header.protocol)) else {
self.tally.dropped_unknown.fetch_add(1, Ordering::Relaxed);
return;
};
@@ -144,7 +176,7 @@ impl Plane {
self.tally.dropped_no_link.fetch_add(1, Ordering::Relaxed);
return;
};
if hop.link.send(envelope::decrement(frame)).is_ok() {
if hop.send(envelope::decrement(frame)).is_ok() {
self.tally.forwarded.fetch_add(1, Ordering::Relaxed);
} else {
self.tally.dropped_no_link.fetch_add(1, Ordering::Relaxed);
@@ -152,27 +184,65 @@ impl Plane {
}
}
/// One physical link to a neighbour, of whatever kind of transport.
#[derive(Debug)]
struct Protocol {
plane: Arc<Plane>,
graph: HashMap<PeerId, Vec<PeerId>>,
members: HashSet<PeerId>,
raw: HashMap<PeerId, SharedLink>,
readers: HashMap<PeerId, JoinHandle<()>>,
links: HashMap<PeerId, Arc<PeerLink>>,
struct RawLink {
kind: String,
link: SharedLink,
reader: JoinHandle<()>,
}
impl Protocol {
/// How a destination is reached right now.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct RouteSummary {
/// Number of transport links to it. One is a direct link.
pub hops: u8,
/// The neighbour the first link goes to. For a direct link that is the
/// destination itself.
pub via: EndpointId,
}
#[derive(Debug)]
struct State {
plane: Arc<Plane>,
/// Transport kinds, best first. A neighbour with several links is
/// reached over the first of these it has.
priority: Vec<String>,
graph: HashMap<PeerId, Vec<PeerId>>,
members: HashSet<PeerId>,
raw: HashMap<PeerId, Vec<RawLink>>,
links: HashMap<(PeerId, String), Arc<PeerLink>>,
}
impl State {
fn rank(&self, kind: &str) -> usize {
self.priority
.iter()
.position(|known| known == kind)
.unwrap_or(usize::MAX)
}
/// A neighbour's links, best kind first.
fn ordered(&self, peer: &PeerId) -> Vec<&RawLink> {
let mut links: Vec<&RawLink> = self
.raw
.get(peer)
.map(|links| links.iter().collect())
.unwrap_or_default();
links.sort_by_key(|raw| (self.rank(&raw.kind), raw.kind.clone()));
links
}
fn publish(&self) {
let mut graph = self.graph.clone();
graph.insert(
self.plane.local,
self.raw
.iter()
.filter(|(_, link)| !link.is_closed())
.map(|(peer, _)| *peer)
.collect(),
);
let mut neighbours: Vec<PeerId> = self
.raw
.iter()
.filter(|(_, links)| links.iter().any(|raw| !raw.link.is_closed()))
.map(|(peer, _)| *peer)
.collect();
neighbours.sort_unstable();
graph.insert(self.plane.local, neighbours);
let routing = RoutingTable::build(self.plane.local, &graph, ROUTING_HOP_LIMIT);
let routes = routing
.iter()
@@ -181,10 +251,12 @@ impl Protocol {
.next_hops
.iter()
.filter_map(|id| {
self.raw.get(id).map(|link| NextHop {
peer: *id,
link: link.clone(),
})
let links: Box<[_]> = self
.ordered(id)
.into_iter()
.map(|raw| raw.link.clone())
.collect();
(!links.is_empty()).then_some(NextHop { peer: *id, links })
})
.collect();
(!next.is_empty()).then_some((
@@ -204,17 +276,21 @@ impl Protocol {
inboxes: self
.links
.iter()
.map(|(peer, link)| (*peer, link.inbox_tx.clone()))
.map(|((peer, _), link)| ((*peer, link.tag), link.inbox_tx.clone()))
.collect(),
}));
}
fn stop_readers(&mut self) {
for raw in self.raw.values().flatten() {
raw.reader.abort();
}
}
}
impl Drop for Protocol {
impl Drop for State {
fn drop(&mut self) {
for task in self.readers.values() {
task.abort();
}
self.stop_readers();
for link in self.links.values() {
link.close();
}
@@ -222,12 +298,17 @@ impl Drop for Protocol {
}
}
/// One network's transport links and independently published protocol tables.
/// One network's transport links and its published forwarding table.
///
/// There is one table for the whole network, whatever protocols run over it.
/// A link is a way to reach a neighbour; a protocol is an end-to-end channel
/// to a peer. The two meet only in the envelope, which names the protocol, so
/// a tunnel keeps working over any mix of transports and a middle member
/// needs no protocol in common with either end.
#[derive(Debug)]
pub struct RelayHub {
network: NetworkId,
local: PeerId,
protocols: Mutex<HashMap<String, Protocol>>,
state: Mutex<State>,
tally: Arc<Tally>,
changed: Arc<Notify>,
}
@@ -235,40 +316,38 @@ pub struct RelayHub {
impl RelayHub {
/// Creates the router at this network's local endpoint.
pub fn new(network: NetworkId, local: EndpointId) -> Arc<Self> {
let tally = Arc::new(Tally::default());
Arc::new(Self {
network,
local: *local.as_bytes(),
protocols: Mutex::default(),
tally: Arc::default(),
state: Mutex::new(State {
plane: Arc::new(Plane {
local: *local.as_bytes(),
table: ArcSwap::from_pointee(ForwardingTable::default()),
tally: tally.clone(),
}),
priority: Vec::new(),
graph: HashMap::new(),
members: HashSet::new(),
raw: HashMap::new(),
links: HashMap::new(),
}),
tally,
changed: Arc::default(),
})
}
fn protocols(&self) -> std::sync::MutexGuard<'_, HashMap<String, Protocol>> {
self.protocols
.lock()
.unwrap_or_else(|error| error.into_inner())
fn state(&self) -> std::sync::MutexGuard<'_, State> {
self.state.lock().unwrap_or_else(|error| error.into_inner())
}
fn protocol<'a>(
&self,
protocols: &'a mut HashMap<String, Protocol>,
name: &str,
) -> &'a mut Protocol {
protocols
.entry(name.to_owned())
.or_insert_with(|| Protocol {
plane: Arc::new(Plane {
local: self.local,
table: ArcSwap::from_pointee(ForwardingTable::default()),
tally: self.tally.clone(),
}),
graph: HashMap::new(),
members: HashSet::new(),
raw: HashMap::new(),
readers: HashMap::new(),
links: HashMap::new(),
})
/// Sets which kinds of transport are preferred, best first, when a
/// neighbour can be reached over more than one.
pub fn set_priority(&self, kinds: Vec<String>) {
let mut state = self.state();
if state.priority != kinds {
state.priority = kinds;
state.publish();
}
}
/// Wakes the control loop promptly when a transport reader ends.
@@ -282,14 +361,11 @@ impl RelayHub {
}
/// Publishes authenticated topology. Refreshing identical state is cheap.
pub fn set_topology(
&self,
protocol: &str,
graph: HashMap<PeerId, Vec<PeerId>>,
members: HashSet<PeerId>,
) {
let mut protocols = self.protocols();
let state = self.protocol(&mut protocols, protocol);
///
/// Every row of `graph` is one member's own account of who it has a link
/// to; what kind of link it is does not matter here.
pub fn set_topology(&self, graph: HashMap<PeerId, Vec<PeerId>>, members: HashSet<PeerId>) {
let mut state = self.state();
if state.graph != graph || state.members != members {
state.graph = graph;
state.members = members;
@@ -298,22 +374,44 @@ impl RelayHub {
}
/// Whether the current table has a path to this destination.
pub fn reachable(&self, peer: EndpointId, protocol: &str) -> bool {
self.protocols().get(protocol).is_some_and(|state| {
state
.plane
.table
.load()
.routes
.contains_key(peer.as_bytes())
pub fn reachable(&self, peer: EndpointId) -> bool {
self.state()
.plane
.table
.load()
.routes
.contains_key(peer.as_bytes())
}
/// How a destination is reached right now, if it is.
pub fn route(&self, peer: EndpointId) -> Option<RouteSummary> {
let table = self.state().plane.table.load_full();
let route = table.routes.get(peer.as_bytes())?;
let via = EndpointId::from_bytes(&route.next.first()?.peer).ok()?;
Some(RouteSummary {
hops: route.hops,
via,
})
}
/// Stable logical link, preserved across topology changes.
/// The kinds of transport with an open direct link to a peer, best
/// first.
pub fn direct_kinds(&self, peer: EndpointId) -> Vec<String> {
let state = self.state();
state
.ordered(peer.as_bytes())
.into_iter()
.filter(|raw| !raw.link.is_closed())
.map(|raw| raw.kind.clone())
.collect()
}
/// Stable end-to-end channel for a protocol, preserved across topology
/// and transport changes.
pub fn link(&self, peer: EndpointId, protocol: &str) -> Arc<PeerLink> {
let mut protocols = self.protocols();
let state = self.protocol(&mut protocols, protocol);
if let Some(link) = state.links.get(peer.as_bytes()) {
let mut state = self.state();
let key = (*peer.as_bytes(), protocol.to_owned());
if let Some(link) = state.links.get(&key) {
return link.clone();
}
let (inbox_tx, inbox_rx) = mpsc::channel(INBOX);
@@ -321,88 +419,126 @@ impl RelayHub {
let link = Arc::new(PeerLink {
network: self.network,
peer,
tag: envelope::protocol_tag(protocol),
plane: state.plane.clone(),
inbox_tx,
inbox_rx: tokio::sync::Mutex::new(inbox_rx),
closed,
});
state.links.insert(*peer.as_bytes(), link.clone());
state.links.insert(key, link.clone());
state.publish();
link
}
/// Whether the protocol already owns its logical link.
/// Whether the protocol already owns its end-to-end channel to a peer.
pub fn has_link(&self, peer: EndpointId, protocol: &str) -> bool {
self.protocols()
.get(protocol)
.is_some_and(|state| state.links.contains_key(peer.as_bytes()))
self.state()
.links
.contains_key(&(*peer.as_bytes(), protocol.to_owned()))
}
/// Installs a raw transport link and starts its independent ingress reader.
pub fn set_direct(&self, peer: EndpointId, protocol: &str, raw: SharedLink) {
///
/// A neighbour may have one link per kind of transport; installing a
/// second of the same kind replaces the first.
pub fn set_direct(&self, peer: EndpointId, kind: &str, raw: SharedLink) {
if raw.network() != self.network || raw.peer() != peer {
return;
}
let mut protocols = self.protocols();
let state = self.protocol(&mut protocols, protocol);
let mut state = self.state();
let id = *peer.as_bytes();
if let Some(old) = state.readers.remove(&id) {
old.abort();
}
state.raw.insert(id, raw.clone());
state.publish();
let plane = state.plane.clone();
let changed = self.changed.clone();
state.readers.insert(
id,
tokio::spawn(async move {
let mut batch = 0;
while let Some(frame) = raw.recv().await {
plane.receive(id, frame);
batch += 1;
if batch == 64 {
tokio::task::yield_now().await;
batch = 0;
}
let reader_link = raw.clone();
let reader = tokio::spawn(async move {
let mut batch = 0;
while let Some(frame) = reader_link.recv().await {
plane.receive(id, frame);
batch += 1;
if batch == 64 {
tokio::task::yield_now().await;
batch = 0;
}
changed.notify_one();
}),
);
}
changed.notify_one();
});
let links = state.raw.entry(id).or_default();
if let Some(position) = links.iter().position(|known| known.kind == kind) {
links.remove(position).reader.abort();
}
links.push(RawLink {
kind: kind.to_owned(),
link: raw,
reader,
});
state.publish();
}
/// Removes a physical link without tearing down end-to-end protocol state.
pub fn clear_direct(&self, peer: EndpointId, protocol: &str) {
if let Some(state) = self.protocols().get_mut(protocol) {
state.raw.remove(peer.as_bytes());
if let Some(task) = state.readers.remove(peer.as_bytes()) {
task.abort();
/// Removes one physical link without tearing down end-to-end protocol state.
pub fn clear_direct(&self, peer: EndpointId, kind: &str) {
let mut state = self.state();
let mut changed = false;
if let Some(links) = state.raw.get_mut(peer.as_bytes()) {
if let Some(position) = links.iter().position(|known| known.kind == kind) {
links.remove(position).reader.abort();
changed = true;
}
if links.is_empty() {
state.raw.remove(peer.as_bytes());
}
}
if changed {
state.publish();
}
}
/// Retires data channels without removing an authenticated peer's topology.
pub fn clear_peer_links(&self, peer: EndpointId) {
let mut state = self.state();
if let Some(links) = state.raw.remove(peer.as_bytes()) {
for raw in links {
raw.link.close();
raw.reader.abort();
}
}
let gone: Vec<_> = state
.links
.keys()
.filter(|(id, _)| id == peer.as_bytes())
.cloned()
.collect();
for key in gone {
if let Some(link) = state.links.remove(&key) {
link.close();
}
}
state.publish();
}
/// Revokes all links and topology involving a departed authenticated peer.
pub fn remove_peer(&self, peer: EndpointId) {
for state in self.protocols().values_mut() {
state.raw.remove(peer.as_bytes());
if let Some(task) = state.readers.remove(peer.as_bytes()) {
task.abort();
}
if let Some(link) = state.links.remove(peer.as_bytes()) {
link.close();
}
state.members.remove(peer.as_bytes());
state.graph.remove(peer.as_bytes());
for neighbors in state.graph.values_mut() {
neighbors.retain(|id| id != peer.as_bytes());
}
state.publish();
self.clear_peer_links(peer);
let mut state = self.state();
state.members.remove(peer.as_bytes());
state.graph.remove(peer.as_bytes());
for neighbors in state.graph.values_mut() {
neighbors.retain(|id| id != peer.as_bytes());
}
state.publish();
}
/// Stops every reader and releases all transport handles.
pub fn close(&self) {
self.protocols().clear();
let mut state = self.state();
state.stop_readers();
state.raw.clear();
for link in state.links.values() {
link.close();
}
state.links.clear();
state.graph.clear();
state.members.clear();
state.plane.table.store(Arc::default());
}
}
@@ -411,6 +547,7 @@ impl RelayHub {
pub struct PeerLink {
network: NetworkId,
peer: EndpointId,
tag: u64,
plane: Arc<Plane>,
inbox_tx: mpsc::Sender<Bytes>,
inbox_rx: tokio::sync::Mutex<mpsc::Receiver<Bytes>>,
@@ -454,10 +591,11 @@ impl PacketLink for PeerLink {
let hop = route
.select(&self.plane.local, flow)
.ok_or(TransportError::Closed)?;
hop.link.send(envelope::encode(
hop.send(envelope::encode(
self.plane.local,
*self.peer.as_bytes(),
flow,
self.tag,
&payload,
))?;
if route.hops > 1 {
@@ -489,7 +627,9 @@ impl PacketLink for PeerLink {
fn path_description(&self) -> String {
let table = self.plane.table.load();
match table.routes.get(self.peer.as_bytes()) {
Some(route) if route.hops == 1 => route.next[0].link.path_description(),
Some(route) if route.hops == 1 => route.next[0]
.first_open()
.map_or_else(|| "unreachable".into(), |link| link.path_description()),
Some(route) => format!(
"relay {} hops via {} ({} equal paths)",
route.hops,
crates/tsunagi/src/dataplane/relay_tests.rs → crates/tsunagi-core/src/dataplane/relay_tests.rs
+147 -16
View File
@@ -16,6 +16,9 @@ fn peer(seed: u8) -> EndpointId {
fn id(seed: u8) -> PeerId {
*peer(seed).as_bytes()
}
fn tag() -> u64 {
envelope::protocol_tag("ip")
}
#[derive(Debug)]
struct Wire {
@@ -78,9 +81,8 @@ impl PacketLink for Wire {
}
}
fn topology(hub: &RelayHub, protocol: &str, edges: &[(u8, &[u8])], members: &[u8]) {
fn topology(hub: &RelayHub, edges: &[(u8, &[u8])], members: &[u8]) {
hub.set_topology(
protocol,
edges
.iter()
.map(|(a, bs)| (id(*a), bs.iter().map(|b| id(*b)).collect()))
@@ -96,7 +98,7 @@ async fn ecmp_is_per_flow_direct_wins_and_existing_tunnel_survives_changes() {
let c = Wire::new(3, true);
hub.set_direct(peer(2), "ip", b.clone());
hub.set_direct(peer(3), "ip", c.clone());
topology(&hub, "ip", &[(2, &[4]), (3, &[4])], &[1, 2, 3, 4]);
topology(&hub, &[(2, &[4]), (3, &[4])], &[1, 2, 3, 4]);
let link = hub.link(peer(4), "ip");
let mut chosen = HashMap::new();
for _ in 0..8 {
@@ -132,7 +134,7 @@ async fn ecmp_is_per_flow_direct_wins_and_existing_tunnel_survives_changes() {
}
assert_eq!(c.take().len(), 100);
hub.clear_direct(peer(2), "ip");
topology(&hub, "ip", &[(3, &[])], &[1, 2, 3, 4]);
topology(&hub, &[(3, &[])], &[1, 2, 3, 4]);
assert!(link.send(Bytes::new()).is_err());
assert!(!link.is_closed());
assert!(Arc::ptr_eq(&link, &hub.link(peer(4), "ip")));
@@ -152,8 +154,8 @@ async fn transport_reader_forwards_without_any_plugin_reader_and_isolates_protoc
let c = Wire::new(3, true);
hub.set_direct(peer(1), "ip", a.clone());
hub.set_direct(peer(3), "ip", c.clone());
topology(&hub, "ip", &[(3, &[4])], &[1, 2, 3, 4]);
let frame = envelope::encode(id(1), id(4), 91, b"end-to-end encrypted");
topology(&hub, &[(3, &[4])], &[1, 2, 3, 4]);
let frame = envelope::encode(id(1), id(4), 91, tag(), b"end-to-end encrypted");
let pointer = frame.as_ptr();
a.feed.send(frame).await.unwrap();
tokio::time::timeout(std::time::Duration::from_secs(1), async {
@@ -170,12 +172,136 @@ async fn transport_reader_forwards_without_any_plugin_reader_and_isolates_protoc
ROUTING_HOP_LIMIT - 1
);
assert_eq!(&sent[0][RELAY_OVERHEAD..], b"end-to-end encrypted");
let isolated = hub.link(peer(4), "other-ip");
assert!(isolated.send(Bytes::new()).is_err());
assert_eq!(hub.counters().forwarded, 1);
hub.close();
}
#[tokio::test]
async fn a_frame_is_delivered_only_to_the_protocol_it_names() {
let hub = RelayHub::new(network(), peer(2));
hub.set_direct(peer(1), "ip", Wire::new(1, true));
let ip = hub.link(peer(1), "ip");
let other = hub.link(peer(1), "other-ip");
let plane = ip.plane.clone();
plane.receive(id(1), envelope::encode(id(1), id(2), 0, tag(), b"for ip"));
plane.receive(
id(1),
envelope::encode(
id(1),
id(2),
0,
envelope::protocol_tag("other-ip"),
b"for other",
),
);
plane.receive(
id(1),
envelope::encode(
id(1),
id(2),
0,
envelope::protocol_tag("nobody-runs-this"),
b"for nobody",
),
);
assert_eq!(ip.recv().await.unwrap(), Bytes::from_static(b"for ip"));
assert_eq!(
other.recv().await.unwrap(),
Bytes::from_static(b"for other")
);
assert_eq!(hub.counters().dropped_unknown, 1);
hub.close();
}
/// A middle member joins two neighbours over different transports, and the
/// two ends share a protocol the middle has never heard of.
#[tokio::test]
async fn a_tunnel_crosses_a_middle_member_whose_links_are_of_different_kinds() {
let a = RelayHub::new(network(), peer(1));
let c = RelayHub::new(network(), peer(3));
let b = RelayHub::new(network(), peer(2));
let a_to_c = Wire::new(3, true);
let c_to_a = Wire::new(1, true);
let c_to_b = Wire::new(2, true);
let b_to_c = Wire::new(3, true);
a.set_direct(peer(3), "wg", a_to_c.clone());
c.set_direct(peer(1), "wg", c_to_a);
c.set_direct(peer(2), "wg-quic", c_to_b.clone());
b.set_direct(peer(3), "wg-quic", b_to_c.clone());
topology(&a, &[(3, &[1, 2])], &[1, 2, 3]);
topology(&b, &[(3, &[1, 2])], &[1, 2, 3]);
topology(&c, &[(1, &[3]), (2, &[3])], &[1, 2, 3]);
let outgoing = a.link(peer(2), "tunnel");
let incoming = b.link(peer(1), "tunnel");
outgoing
.send_flow(Bytes::from_static(b"sealed"), 5)
.unwrap();
let frame = a_to_c.take().remove(0);
// The middle forwards it onto its other kind of link without looking
// inside.
c.state().plane.receive(id(1), frame);
let frame = c_to_b.take().remove(0);
assert_eq!(c.counters().forwarded, 1);
b.state().plane.receive(id(3), frame);
assert_eq!(
incoming.recv().await.unwrap(),
Bytes::from_static(b"sealed")
);
assert_eq!(b.counters().received_via, 1);
assert_eq!(
b.route(peer(1)),
Some(RouteSummary {
hops: 2,
via: peer(3)
})
);
a.close();
b.close();
c.close();
}
#[tokio::test]
async fn a_neighbour_is_reached_over_the_preferred_kind_and_falls_back_when_it_is_lost() {
let hub = RelayHub::new(network(), peer(1));
hub.set_priority(vec!["wg".into(), "tcp-tls".into(), "wg-quic".into()]);
let slow = Wire::new(2, true);
let fast = Wire::new(2, true);
hub.set_direct(peer(2), "wg-quic", slow.clone());
let link = hub.link(peer(2), "tunnel");
link.send(Bytes::from_static(b"one")).unwrap();
assert_eq!(slow.take().len(), 1);
assert_eq!(hub.direct_kinds(peer(2)), ["wg-quic"]);
// A better kind appears: traffic moves to it with nothing to redo.
hub.set_direct(peer(2), "wg", fast.clone());
assert_eq!(hub.direct_kinds(peer(2)), ["wg", "wg-quic"]);
link.send(Bytes::from_static(b"two")).unwrap();
assert_eq!(fast.take().len(), 1);
assert!(slow.take().is_empty());
// It dies: the worse one carries on, and the logical link is untouched.
fast.dead.store(true, Ordering::Relaxed);
link.send(Bytes::from_static(b"three")).unwrap();
assert_eq!(slow.take().len(), 1);
assert!(!link.is_closed());
assert_eq!(
hub.route(peer(2)),
Some(RouteSummary {
hops: 1,
via: peer(2)
})
);
hub.clear_direct(peer(2), "wg-quic");
hub.clear_direct(peer(2), "wg");
assert!(!hub.reachable(peer(2)));
assert!(Arc::ptr_eq(&link, &hub.link(peer(2), "tunnel")));
hub.close();
}
#[tokio::test]
async fn local_delivery_is_bounded_and_malformed_or_unknown_sources_are_dropped() {
let hub = RelayHub::new(network(), peer(2));
@@ -184,18 +310,21 @@ async fn local_delivery_is_bounded_and_malformed_or_unknown_sources_are_dropped(
let link = hub.link(peer(1), "ip");
let plane = link.plane.clone();
for _ in 0..INBOX + 1 {
plane.receive(id(1), envelope::encode(id(1), id(2), 0, b"ip"));
plane.receive(id(1), envelope::encode(id(1), id(2), 0, tag(), b"ip"));
}
assert_eq!(hub.counters().dropped_congested, 1);
for _ in 0..INBOX {
assert_eq!(link.recv().await.unwrap(), Bytes::from_static(b"ip"));
}
plane.receive(id(1), Bytes::from_static(b"broken"));
plane.receive(id(1), envelope::encode(id(9), id(2), 0, b"unknown member"));
plane.receive(
id(1),
envelope::encode(id(9), id(2), 0, tag(), b"unknown member"),
);
assert_eq!(hub.counters().dropped_unknown, 2);
plane.receive(
id(1),
envelope::encode(id(1), id(9), 0, b"unknown destination"),
envelope::encode(id(1), id(9), 0, tag(), b"unknown destination"),
);
assert_eq!(hub.counters().dropped_no_link, 1);
hub.close();
@@ -210,11 +339,11 @@ async fn inconsistent_tables_cannot_loop_beyond_the_hop_limit() {
a.set_direct(peer(2), "ip", ab.clone());
b.set_direct(peer(1), "ip", ba.clone());
// Deliberately inconsistent snapshots while updates are in flight.
topology(&a, "ip", &[(2, &[4])], &[1, 2, 3, 4]);
topology(&b, "ip", &[(1, &[4])], &[1, 2, 3, 4]);
topology(&a, &[(2, &[4])], &[1, 2, 3, 4]);
topology(&b, &[(1, &[4])], &[1, 2, 3, 4]);
let ap = a.link(peer(4), "ip").plane.clone();
let bp = b.link(peer(4), "ip").plane.clone();
let mut frame = envelope::encode(id(3), id(4), 7, b"transit");
let mut frame = envelope::encode(id(3), id(4), 7, tag(), b"transit");
for turn in 0..ROUTING_HOP_LIMIT {
let (plane, incoming, wire) = if turn % 2 == 0 {
(&ap, id(2), &ab)
@@ -253,7 +382,7 @@ async fn forwarding_benchmark() {
hub.set_direct(peer(3), "ip", wire.clone());
let alternate = Wire::new(5, false);
hub.set_direct(peer(5), "ip", alternate.clone());
topology(&hub, "ip", &[(3, &[4]), (5, &[4])], &[1, 2, 3, 4, 5]);
topology(&hub, &[(3, &[4]), (5, &[4])], &[1, 2, 3, 4, 5]);
let plane = hub.link(peer(4), "ip").plane.clone();
let source = id(1);
let destination = id(4);
@@ -264,7 +393,9 @@ async fn forwarding_benchmark() {
const BATCHES: usize = 100;
for _ in 0..BATCHES {
let frames: Vec<_> = (0..COUNT)
.map(|flow| envelope::encode(source, destination, flow as u64, &vec![42; size]))
.map(|flow| {
envelope::encode(source, destination, flow as u64, tag(), &vec![42; size])
})
.collect();
let start = std::time::Instant::now();
for frame in frames {
crates/tsunagi/src/dataplane/routing/envelope.rs → crates/tsunagi-core/src/dataplane/routing/envelope.rs
+25 -5
View File
@@ -3,8 +3,8 @@ use super::{FlowId, PeerId};
use crate::config::{MAX_DATA_DATAGRAM, ROUTING_HOP_LIMIT};
use bytes::{BufMut, Bytes, BytesMut};
pub(crate) const HEADER: usize = 2 + 32 + 32 + 8;
const VERSION: u8 = 1;
pub(crate) const HEADER: usize = 2 + 32 + 32 + 8 + 8;
const VERSION: u8 = 2;
#[derive(Debug, PartialEq, Eq)]
pub(crate) struct Header {
@@ -12,6 +12,15 @@ pub(crate) struct Header {
pub destination: PeerId,
pub remaining: u8,
pub flow: FlowId,
/// Which end-to-end protocol the payload belongs to, whatever links carry it.
pub protocol: u64,
}
/// A stable 64-bit tag for a protocol id (FNV-1a), the same on every member.
pub(crate) fn protocol_tag(name: &str) -> u64 {
name.bytes().fold(0xcbf2_9ce4_8422_2325, |hash, byte| {
(hash ^ u64::from(byte)).wrapping_mul(0x0000_0100_0000_01b3)
})
}
pub(crate) fn decode(bytes: &[u8]) -> Option<Header> {
@@ -28,16 +37,24 @@ pub(crate) fn decode(bytes: &[u8]) -> Option<Header> {
destination: bytes[34..66].try_into().ok()?,
remaining: bytes[1],
flow: u64::from_be_bytes(bytes[66..74].try_into().ok()?),
protocol: u64::from_be_bytes(bytes[74..82].try_into().ok()?),
})
}
pub(crate) fn encode(source: PeerId, destination: PeerId, flow: FlowId, payload: &[u8]) -> Bytes {
pub(crate) fn encode(
source: PeerId,
destination: PeerId,
flow: FlowId,
protocol: u64,
payload: &[u8],
) -> Bytes {
let mut bytes = BytesMut::with_capacity(HEADER + payload.len());
bytes.put_u8(VERSION);
bytes.put_u8(ROUTING_HOP_LIMIT);
bytes.extend_from_slice(&source);
bytes.extend_from_slice(&destination);
bytes.put_u64(flow);
bytes.put_u64(protocol);
bytes.extend_from_slice(payload);
bytes.freeze()
}
@@ -58,7 +75,7 @@ mod tests {
use super::*;
#[test]
fn transit_reuses_the_buffer_and_rejects_invalid_headers() {
let frame = encode([1; 32], [2; 32], 17, &[42; 1280]);
let frame = encode([1; 32], [2; 32], 17, protocol_tag("wg"), &[42; 1280]);
let address = frame.as_ptr();
let frame = decrement(frame);
assert_eq!(
@@ -66,7 +83,10 @@ mod tests {
address,
"no payload copy for an owned buffer"
);
assert_eq!(decode(&frame).unwrap().remaining, ROUTING_HOP_LIMIT - 1);
let header = decode(&frame).unwrap();
assert_eq!(header.remaining, ROUTING_HOP_LIMIT - 1);
assert_eq!(header.protocol, protocol_tag("wg"));
assert_ne!(protocol_tag("wg"), protocol_tag("wg-quic"));
assert_eq!(&frame[HEADER..], &[42; 1280]);
for length in 0..HEADER {
assert!(decode(&frame[..length]).is_none());
crates/tsunagi/src/dataplane/routing/flow.rs → crates/tsunagi-core/src/dataplane/routing/flow.rs
View File
File renamed without changes.
crates/tsunagi/src/dataplane/routing/mod.rs → crates/tsunagi-core/src/dataplane/routing/mod.rs
View File
File renamed without changes.
crates/tsunagi/src/dataplane/transport/fragments.rs → crates/tsunagi-core/src/dataplane/transport/fragments.rs
View File
File renamed without changes.
crates/tsunagi-core/src/dataplane/transport/health.rs
+350
View File
@@ -0,0 +1,350 @@
//! Authenticated probes of the actual data link, independent of control RTT.
//! Keys are negotiated privately for each physical link. No fixed wire magic.
use super::{PacketLink, SharedLink, TransportError};
use crate::dataplane::SharedPlugin;
use crate::overlay::RoutingTable;
use crate::{BoxFuture, NetworkId};
use bytes::Bytes;
use hmac::{Hmac, KeyInit, Mac};
use iroh::EndpointId;
use serde::{Deserialize, Serialize};
use sha2::Sha256;
use std::collections::VecDeque;
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};
use tokio::sync::{mpsc, watch};
#[derive(Clone, Serialize, Deserialize, zeroize::ZeroizeOnDrop)]
pub(crate) struct Settings {
marker: [u8; 16],
key: [u8; 32],
}
impl Settings {
pub(crate) fn fresh() -> Self {
Self {
marker: rand::random(),
key: rand::random(),
}
}
}
impl std::fmt::Debug for Settings {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str("HealthSettings([REDACTED])")
}
}
struct Samples {
pending: VecDeque<(u64, Instant)>,
recent: VecDeque<bool>,
}
impl Samples {
fn new() -> Self {
Self {
pending: VecDeque::new(),
recent: VecDeque::new(),
}
}
fn sent(&mut self, seq: u64, now: Instant) {
self.pending.push_back((seq, now));
}
fn acknowledge(&mut self, seq: u64) {
if let Some(i) = self.pending.iter().position(|(n, _)| *n == seq) {
self.pending.remove(i);
self.recent.push_back(true);
}
}
fn expire(&mut self, now: Instant) {
while self
.pending
.front()
.is_some_and(|(_, at)| now.duration_since(*at) >= Duration::from_secs(3))
{
self.pending.pop_front();
self.recent.push_back(false);
}
while self.recent.len() > 20 {
self.recent.pop_front();
}
}
fn degraded(&self) -> bool {
self.recent.len() >= 15
&& self.recent.iter().filter(|ok| !**ok).count() * 100 >= self.recent.len() * 60
}
}
// This prefix is inside end-to-end encryption. It never appears on the UDP wire.
const IP_MAGIC: &[u8; 16] = b"tsunagi-health01";
pub(crate) fn ip_payload(packet: &[u8]) -> Option<&[u8]> {
(packet.len() >= 36 && packet[0] == 0x45 && packet[9] == 253 && &packet[20..36] == IP_MAGIC)
.then(|| &packet[36..])
}
/// Inject probes through the same encrypted tunnel as application IP packets.
/// This matters for pure WG: raw UDP probes could pass DPI while WG is blocked.
pub(crate) struct ProbeRoute {
pub(crate) plugin: SharedPlugin,
pub(crate) routes: Arc<RoutingTable>,
pub(crate) network: NetworkId,
pub(crate) peer: EndpointId,
pub(crate) max_probe_size: usize,
}
impl ProbeRoute {
fn send(&self, payload: Bytes) -> bool {
let Some((source, destination)) = self.routes.probe_addresses(self.network, self.peer)
else {
return false;
};
let mut packet = vec![0; 36];
packet[0] = 0x45;
packet[8] = 64;
packet[9] = 253;
packet[12..16].copy_from_slice(&source.octets());
packet[16..20].copy_from_slice(&destination.octets());
packet[20..36].copy_from_slice(IP_MAGIC);
let length = (packet.len() + payload.len()) as u16;
packet[2..4].copy_from_slice(&length.to_be_bytes());
let sum = packet[..20]
.as_chunks::<2>()
.0
.iter()
.map(|b| u32::from(u16::from_be_bytes(*b)))
.sum::<u32>();
let sum = (sum & 0xffff) + (sum >> 16);
packet[10..12].copy_from_slice(&(!(sum as u16)).to_be_bytes());
packet.extend_from_slice(&payload);
self.plugin
.carry(self.network, self.peer, Bytes::from(packet))
}
}
/// A monitored physical link. It forwards application bytes unchanged.
pub(crate) struct HealthLink {
probes: mpsc::Sender<Bytes>,
raw: SharedLink,
settings: Mutex<Option<Settings>>,
rx: tokio::sync::Mutex<mpsc::Receiver<Bytes>>,
stop: watch::Sender<bool>,
reason: Mutex<Option<String>>,
}
impl std::fmt::Debug for HealthLink {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("HealthLink")
.field("raw", &self.raw)
.finish_non_exhaustive()
}
}
fn lock<T>(m: &Mutex<T>) -> std::sync::MutexGuard<'_, T> {
match m.lock() {
Ok(g) => g,
Err(p) => p.into_inner(),
}
}
impl HealthLink {
pub(crate) fn new(
raw: SharedLink,
local: EndpointId,
settings: Option<Settings>,
route: Option<ProbeRoute>,
) -> Arc<Self> {
let (tx, rx) = mpsc::channel(64);
let (probe_tx, mut probe_rx) = mpsc::channel(64);
let (stop, _) = watch::channel(false);
let link = Arc::new(Self {
probes: probe_tx,
raw: raw.clone(),
settings: Mutex::new(settings),
rx: tokio::sync::Mutex::new(rx),
stop,
reason: Mutex::new(None),
});
let weak = Arc::downgrade(&link);
let mut stopped = link.stop.subscribe();
tokio::spawn(async move {
let mut interval = tokio::time::interval(Duration::from_secs(1));
interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
let mut samples = Samples::new();
let mut sequence = rand::random::<u64>();
let mut since: Option<Instant> = None;
let mut marker = None;
loop {
tokio::select! {
biased;
_=stopped.changed()=>break,
_=interval.tick()=>{
let Some(link)=weak.upgrade()else{break;};let Some(settings)=lock(&link.settings).clone()else{continue;};
if marker != Some(settings.marker) { marker = Some(settings.marker); samples = Samples::new(); since = None; }
let start=*since.get_or_insert_with(Instant::now);let now=Instant::now();samples.expire(now);
if now.duration_since(start)>=Duration::from_secs(20) && samples.degraded() {
*lock(&link.reason)=Some("data delivery checks: at least 60% loss over the last 20 probes".into());break;
}
sequence=sequence.wrapping_add(1);
let size=route.as_ref().map_or(raw.max_datagram_size(),|route|route.max_probe_size);
let packet=probe(&settings,local,1,sequence,size.min(1000));
if let Some(route)=&route { if route.send(packet) {samples.sent(sequence,now);} }
else { let _=raw.send(packet); samples.sent(sequence,now); }
}
packet=raw.recv()=>{
let Some(packet)=packet else{break;};let Some(link)=weak.upgrade()else{break;};
let settings=lock(&link.settings).clone();
if route.is_none() && let Some(settings)=settings && packet.starts_with(&settings.marker) {
if let Some((kind,seq))=verify(&settings,raw.peer(),&packet) {
if kind==1 {let _=raw.send(probe(&settings,local,2,seq,packet.len()));}
else if kind==2 {samples.acknowledge(seq);}
}
} else {let _=tx.try_send(packet);}
}
packet=probe_rx.recv(),if route.is_some()=>{
let Some(packet)=packet else {break;};let Some(link)=weak.upgrade()else{break;};
let Some(settings)=lock(&link.settings).clone()else{continue;};
if let Some((kind,seq))=verify(&settings,raw.peer(),&packet) {
if kind==1 { if let Some(route)=&route { route.send(probe(&settings,local,2,seq,packet.len().min(route.max_probe_size))); } }
else if kind==2 {samples.acknowledge(seq);}
}
}
}
}
raw.close();
if let Some(link) = weak.upgrade() {
link.stop.send_replace(true);
}
});
link
}
pub(crate) fn receive_probe(&self, packet: Bytes) {
let _ = self.probes.try_send(packet);
}
pub(crate) fn configure(&self, settings: Settings) {
*lock(&self.settings) = Some(settings);
}
}
fn probe(settings: &Settings, sender: EndpointId, kind: u8, seq: u64, size: usize) -> Bytes {
let mut bytes = vec![0; size.max(57)];
bytes[..16].copy_from_slice(&settings.marker);
bytes[16] = kind;
bytes[17..25].copy_from_slice(&seq.to_be_bytes());
rand::fill(&mut bytes[25..size.max(57) - 32]);
let tail = bytes.len() - 32;
if let Ok(mut mac) = <Hmac<Sha256> as KeyInit>::new_from_slice(&settings.key) {
mac.update(sender.as_bytes());
mac.update(&bytes[..tail]);
bytes[tail..].copy_from_slice(&mac.finalize().into_bytes());
}
Bytes::from(bytes)
}
fn verify(settings: &Settings, sender: EndpointId, bytes: &[u8]) -> Option<(u8, u64)> {
if bytes.len() < 57 || !bytes.starts_with(&settings.marker) {
return None;
}
let mut mac = <Hmac<Sha256> as KeyInit>::new_from_slice(&settings.key).ok()?;
let tail = bytes.len() - 32;
mac.update(sender.as_bytes());
mac.update(&bytes[..tail]);
mac.verify_slice(&bytes[tail..]).ok()?;
Some((
bytes[16],
u64::from_be_bytes(bytes.get(17..25)?.try_into().ok()?),
))
}
impl PacketLink for HealthLink {
fn network(&self) -> NetworkId {
self.raw.network()
}
fn peer(&self) -> EndpointId {
self.raw.peer()
}
fn max_datagram_size(&self) -> usize {
self.raw.max_datagram_size()
}
fn send(&self, payload: Bytes) -> Result<(), TransportError> {
if self.is_closed() {
Err(TransportError::Closed)
} else {
self.raw.send(payload)
}
}
fn send_flow(
&self,
payload: Bytes,
flow: super::super::routing::FlowId,
) -> Result<(), TransportError> {
if self.is_closed() {
Err(TransportError::Closed)
} else {
self.raw.send_flow(payload, flow)
}
}
fn recv(&self) -> BoxFuture<'_, Option<Bytes>> {
Box::pin(async move {
let mut stop = self.stop.subscribe();
if *stop.borrow() {
return None;
}
let mut rx = self.rx.lock().await;
tokio::select! {biased;_=stop.changed()=>None,packet=rx.recv()=>packet}
})
}
fn closed(&self) -> BoxFuture<'_, ()> {
Box::pin(async move {
let mut stop = self.stop.subscribe();
if !*stop.borrow() {
let _ = stop.changed().await;
}
})
}
fn is_closed(&self) -> bool {
*self.stop.borrow() || self.raw.is_closed()
}
fn close(&self) {
self.stop.send_replace(true);
self.raw.close();
}
fn failure_reason(&self) -> Option<String> {
lock(&self.reason)
.clone()
.or_else(|| self.raw.failure_reason())
}
fn path_description(&self) -> String {
self.raw.path_description()
}
}
impl Drop for HealthLink {
fn drop(&mut self) {
self.stop.send_replace(true);
self.raw.close();
}
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
#[test]
fn no_false_failure_before_window_but_persistent_loss_is_detected() {
let mut samples = Samples::new();
let now = Instant::now();
for seq in 0..20 {
samples.sent(seq, now);
if seq % 5 == 0 {
samples.acknowledge(seq);
}
}
assert!(!samples.degraded());
samples.expire(now + Duration::from_secs(4));
assert!(samples.degraded());
for seq in 20..40 {
samples.sent(seq, now);
samples.acknowledge(seq);
}
samples.expire(now + Duration::from_secs(4));
assert!(!samples.degraded());
}
#[test]
fn a_forged_or_reflected_probe_cannot_claim_delivery() {
let a = crate::identity::DeviceIdentity::generate().endpoint_id();
let b = crate::identity::DeviceIdentity::generate().endpoint_id();
let settings = Settings::fresh();
let packet = probe(&settings, a, 1, 42, 1100);
assert_eq!(verify(&settings, a, &packet), Some((1, 42)));
assert!(verify(&settings, b, &packet).is_none());
let mut forged = packet.to_vec();
forged[16] = 2;
assert!(verify(&settings, a, &forged).is_none());
}
}
crates/tsunagi/src/dataplane/transport/iroh_link.rs → crates/tsunagi-core/src/dataplane/transport/iroh_link.rs
+3
View File
@@ -97,6 +97,9 @@ impl IrohLink {
}
impl PacketLink for IrohLink {
fn close(&self) {
self.conn.close(0u32.into(), b"data link retired");
}
fn network(&self) -> NetworkId {
self.network
}
crates/tsunagi/src/dataplane/transport/mod.rs → crates/tsunagi-core/src/dataplane/transport/mod.rs
+110
View File
@@ -30,12 +30,14 @@
//! one plugin protocol.
mod fragments;
pub(crate) mod health;
pub mod iroh_link;
use std::sync::Arc;
use bytes::Bytes;
use iroh::EndpointId;
use serde::{Deserialize, Serialize};
use crate::BoxFuture;
use crate::identity::NetworkId;
@@ -112,6 +114,14 @@ pub trait PacketLink: Send + Sync + std::fmt::Debug + 'static {
/// keeping a task parked on [`PacketLink::closed`].
fn is_closed(&self) -> bool;
/// Explicitly retires a physical link. Implementations should wake readers.
fn close(&self) {}
/// Observed reason for retiring this link, if available.
fn failure_reason(&self) -> Option<String> {
None
}
/// A short description of the path in use, for diagnostics.
///
/// Reports what the transport actually knows. It must not invent a value.
@@ -121,6 +131,76 @@ pub trait PacketLink: Send + Sync + std::fmt::Debug + 'static {
/// A shared handle to a link.
pub type SharedLink = std::sync::Arc<dyn PacketLink>;
/// Maximum opaque pair-negotiation payload.
pub const MAX_TRANSPORT_PAYLOAD: usize = 4096;
/// Private transport bytes, never included in Debug output.
#[derive(Clone, PartialEq, Eq, Serialize, zeroize::ZeroizeOnDrop)]
pub struct TransportPayload(Vec<u8>);
impl<'de> Deserialize<'de> for TransportPayload {
fn deserialize<D: serde::Deserializer<'de>>(
deserializer: D,
) -> std::result::Result<Self, D::Error> {
struct Visitor;
impl<'de> serde::de::Visitor<'de> for Visitor {
type Value = TransportPayload;
fn expecting(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "at most {MAX_TRANSPORT_PAYLOAD} private transport bytes")
}
fn visit_seq<A: serde::de::SeqAccess<'de>>(
self,
mut sequence: A,
) -> std::result::Result<Self::Value, A::Error> {
if sequence
.size_hint()
.is_some_and(|size| size > MAX_TRANSPORT_PAYLOAD)
{
return Err(serde::de::Error::custom("transport payload too large"));
}
let mut payload = TransportPayload(Vec::with_capacity(
sequence.size_hint().unwrap_or(0).min(MAX_TRANSPORT_PAYLOAD),
));
while let Some(byte) = sequence.next_element::<u8>()? {
if payload.0.len() == MAX_TRANSPORT_PAYLOAD {
return Err(serde::de::Error::custom("transport payload too large"));
}
payload.0.push(byte);
}
Ok(payload)
}
}
deserializer.deserialize_seq(Visitor)
}
}
impl TransportPayload {
/// Wraps bytes for one authenticated control session.
pub fn new(bytes: Vec<u8>) -> Self {
Self(bytes)
}
/// Borrows private bytes; callers must not log them.
pub fn as_bytes(&self) -> &[u8] {
&self.0
}
}
impl std::fmt::Debug for TransportPayload {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str("TransportPayload([REDACTED])")
}
}
/// Private control-plane sender for transports. Contains no network secret.
pub trait ControlSender: Send + Sync + std::fmt::Debug + 'static {
/// Queues a bounded message to one authenticated member of one network.
fn send<'a>(
&'a self,
network: NetworkId,
peer: EndpointId,
protocol: &'a str,
payload: TransportPayload,
) -> BoxFuture<'a, Result<(), TransportError>>;
}
/// An inbound link a peer opened towards us.
#[derive(Debug)]
pub struct InboundLink {
@@ -139,6 +219,18 @@ pub struct InboundLink {
/// The agent owns one of these and hands links to plugins; plugins never call
/// it directly.
pub trait PacketTransport: Send + Sync + std::fmt::Debug + 'static {
/// Supplies a private sender before network activation.
fn attach_control(&self, _control: Arc<dyn ControlSender>) {}
/// Validates and queues private negotiation received from an authenticated member.
fn on_control(
&self,
_network: NetworkId,
_peer: EndpointId,
_protocol: &str,
_payload: &[u8],
) -> Result<(), TransportError> {
Err(TransportError::Declined(self.name().into()))
}
/// A short name used in diagnostics.
fn name(&self) -> &str;
@@ -184,6 +276,24 @@ impl MultiPacketTransport {
}
impl PacketTransport for MultiPacketTransport {
fn attach_control(&self, control: Arc<dyn ControlSender>) {
self.default.attach_control(control.clone());
for transport in self.by_protocol.values() {
transport.attach_control(control.clone());
}
}
fn on_control(
&self,
network: NetworkId,
peer: EndpointId,
protocol: &str,
payload: &[u8],
) -> Result<(), TransportError> {
match self.by_protocol.get(protocol) {
Some(transport) => transport.on_control(network, peer, protocol, payload),
None => Err(TransportError::Declined(protocol.into())),
}
}
fn name(&self) -> &str {
"multi"
}
crates/tsunagi/src/dataplane/tunnel.rs → crates/tsunagi-core/src/dataplane/tunnel.rs
+16
View File
@@ -83,6 +83,8 @@ struct TunnelPeerState {
link: SharedLink,
tx_packets: AtomicU64,
rx_packets: AtomicU64,
tx_bytes: AtomicU64,
rx_bytes: AtomicU64,
dropped: AtomicU64,
established_at: std::time::Instant,
}
@@ -100,6 +102,10 @@ pub struct TunnelPeerReport {
pub tx_packets: u64,
/// Packets successfully received and decrypted.
pub rx_packets: u64,
/// Plaintext bytes successfully encrypted and sent.
pub tx_bytes: u64,
/// Plaintext bytes successfully received and decrypted.
pub rx_bytes: u64,
/// Packets dropped due to encryption/decryption/send failures.
pub dropped: u64,
/// Uptime in seconds since the data link was established.
@@ -145,6 +151,8 @@ impl<C: TunnelCodec> GenericTunnelPlugin<C> {
path: state.link.path_description(),
tx_packets: state.tx_packets.load(Ordering::Relaxed),
rx_packets: state.rx_packets.load(Ordering::Relaxed),
tx_bytes: state.tx_bytes.load(Ordering::Relaxed),
rx_bytes: state.rx_bytes.load(Ordering::Relaxed),
dropped: state.dropped.load(Ordering::Relaxed),
uptime_secs: state.established_at.elapsed().as_secs(),
})
@@ -223,6 +231,9 @@ impl<C: TunnelCodec> IpPlugin for GenericTunnelPlugin<C> {
Ok(ciphertext) => {
if state.link.send(ciphertext).is_ok() {
state.tx_packets.fetch_add(1, Ordering::Relaxed);
state
.tx_bytes
.fetch_add(packet.len() as u64, Ordering::Relaxed);
true
} else {
state.dropped.fetch_add(1, Ordering::Relaxed);
@@ -242,6 +253,8 @@ impl<C: TunnelCodec> IpPlugin for GenericTunnelPlugin<C> {
link: Arc::clone(&link),
tx_packets: AtomicU64::new(0),
rx_packets: AtomicU64::new(0),
tx_bytes: AtomicU64::new(0),
rx_bytes: AtomicU64::new(0),
dropped: AtomicU64::new(0),
established_at: std::time::Instant::now(),
});
@@ -268,6 +281,9 @@ impl<C: TunnelCodec> IpPlugin for GenericTunnelPlugin<C> {
match codec.decrypt(network, peer, &payload) {
Ok(packet) => {
rx_state.rx_packets.fetch_add(1, Ordering::Relaxed);
rx_state
.rx_bytes
.fetch_add(packet.len() as u64, Ordering::Relaxed);
if let Some(ref sink) = sink {
sink.deliver(network, peer, packet).await;
}
crates/tsunagi/src/discovery.rs → crates/tsunagi-core/src/discovery.rs
View File
File renamed without changes.
crates/tsunagi/src/discovery/mainline.rs → crates/tsunagi-core/src/discovery/mainline.rs
View File
File renamed without changes.
crates/tsunagi/src/discovery/worker.rs → crates/tsunagi-core/src/discovery/worker.rs
View File
File renamed without changes.
crates/tsunagi/src/dns/mod.rs → crates/tsunagi-core/src/dns/mod.rs
View File
File renamed without changes.
crates/tsunagi/src/dns/publish/macos.rs → crates/tsunagi-core/src/dns/publish/macos.rs
View File
File renamed without changes.
crates/tsunagi/src/dns/publish/mock.rs → crates/tsunagi-core/src/dns/publish/mock.rs
View File
File renamed without changes.
crates/tsunagi/src/dns/publish/mod.rs → crates/tsunagi-core/src/dns/publish/mod.rs
View File
File renamed without changes.
crates/tsunagi/src/dns/publish/resolved.rs → crates/tsunagi-core/src/dns/publish/resolved.rs
View File
File renamed without changes.
crates/tsunagi/src/dns/publish/unsupported.rs → crates/tsunagi-core/src/dns/publish/unsupported.rs
View File
File renamed without changes.
crates/tsunagi/src/dns/publish/windows.rs → crates/tsunagi-core/src/dns/publish/windows.rs
View File
File renamed without changes.
crates/tsunagi/src/dns/server.rs → crates/tsunagi-core/src/dns/server.rs
View File
File renamed without changes.
crates/tsunagi/src/dns/zone.rs → crates/tsunagi-core/src/dns/zone.rs
View File
File renamed without changes.
crates/tsunagi/src/error.rs → crates/tsunagi-core/src/error.rs
+26
View File
@@ -15,6 +15,12 @@ pub type Result<T, E = Error> = std::result::Result<T, E>;
#[derive(Debug, thiserror::Error)]
#[non_exhaustive]
pub enum Error {
/// A node-history cleanup is unsafe, stale or cannot be performed.
#[error("cannot forget node: {0}")]
ForgetNode(String),
/// An ordered local data-plane protocol selection is invalid.
#[error("invalid protocol selection: {0}")]
InvalidProtocols(String),
/// The supplied network name does not satisfy the documented rules.
#[error("invalid network name: {0}")]
InvalidNetworkName(&'static str),
@@ -97,6 +103,19 @@ pub enum Error {
#[error("network {0} is not configured")]
NetworkUnknown(NetworkId),
/// An overlay range overlaps networks already active on this agent.
#[error(
"network {network} cannot start: {range} overlaps active networks {conflicts:?}; switch to it to suspend those networks"
)]
NetworkRangeConflict {
/// Network that could not start.
network: NetworkId,
/// Its agreed or proposed range.
range: crate::state::Ipv4Range,
/// Active networks that must be suspended first.
conflicts: Vec<NetworkId>,
},
/// No session with that peer exists in the given network.
#[error("no authenticated session with peer {peer} in network {network}")]
NoSuchPeer {
@@ -120,6 +139,13 @@ pub enum Error {
what: String,
},
/// The request is understood and cannot be done as things stand.
#[error("{reason}")]
Rejected {
/// Why, phrased so the message stands on its own.
reason: String,
},
/// Discovery backend failure. Never fatal for the agent.
#[error("discovery error: {0}")]
Discovery(String),
crates/tsunagi/src/identity/mod.rs → crates/tsunagi-core/src/identity/mod.rs
View File
File renamed without changes.
crates/tsunagi/src/identity/network.rs → crates/tsunagi-core/src/identity/network.rs
View File
File renamed without changes.
crates/tsunagi/src/ipc/mod.rs → crates/tsunagi-core/src/ipc/mod.rs
+654 -5
View File
@@ -44,6 +44,21 @@ use windows as transport;
#[cfg(any(unix, windows))]
pub use transport::ControlSocket;
/// Who may open the local control socket.
///
/// The default is owner-only. A system service shares its socket with a GUI or
/// CLI running as another user by granting a group; the agent sets that on the
/// socket when it binds it, so the access is correct from creation rather than
/// patched afterwards.
#[derive(Debug, Clone)]
pub enum ControlSocketAccess {
/// Owner only — mode `0600`. The per-user agent's default.
Private,
/// The owner and members of this group — the socket's group is set to this
/// gid and the mode to `0660`. The caller resolves a group name to its gid.
Group(u32),
}
use std::path::{Path, PathBuf};
use std::sync::Arc;
use std::time::Duration;
@@ -58,6 +73,31 @@ use crate::error::{Error, Result};
/// Largest accepted local control message.
pub const MAX_MESSAGE_LEN: usize = 1024 * 1024;
/// A string the control protocol must carry but that must never appear in a
/// log: it serialises transparently (the text crosses the socket intact) but
/// its `Debug` is redacted. Used for a network secret handed back to a local
/// client that asked to copy it.
#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct RedactedText(String);
impl RedactedText {
/// Wraps a string.
pub fn new(text: impl Into<String>) -> Self {
Self(text.into())
}
/// Unwraps to the plain string.
pub fn into_string(self) -> String {
self.0
}
}
impl std::fmt::Debug for RedactedText {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str("<redacted>")
}
}
/// Where the control socket for a state directory lives.
///
/// A Unix socket path is limited to around 100 bytes, which a state directory
@@ -108,10 +148,17 @@ pub fn default_control_socket_path() -> PathBuf {
}
#[cfg(all(unix, not(target_os = "macos")))]
{
// Prefer a running system service's socket when it is present, so a
// GUI or CLI reaches the packaged daemon with no configuration; fall
// back to the per-user location otherwise.
let system = PathBuf::from("/run/tsunagi/agent.sock");
if system.exists() {
return system;
}
if let Ok(paths) = crate::config::StoragePaths::user_default() {
control_socket_path(&paths.state_dir)
} else {
PathBuf::from("/run/tsunagi/agent.sock")
system
}
}
}
@@ -124,6 +171,33 @@ pub fn default_control_socket_path() -> PathBuf {
#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)]
#[non_exhaustive]
pub enum Request {
/// Save a complete version-checked service draft or its deletion.
SaveVip {
/// Exact network id.
network_id: String,
/// Entire desired configuration.
config: crate::vip::VipConfig,
/// Revision originally displayed; absent for creation.
expected: Option<String>,
/// Remove the service permanently.
delete: bool,
},
/// Retire the selected offline member's observed claim.
ForgetNode {
/// Exact public network id.
network_id: String,
/// Exact device identity selected by the client.
peer: String,
/// Claim version displayed when the user selected the node.
expected_version: u64,
},
/// Replace one network's ordered local data-plane selection.
SetProtocols {
/// Public network identifier.
network_id: String,
/// Protocol names, best first; empty selects control only.
protocols: Vec<String>,
},
/// Report what the agent is doing.
Status,
/// Set local broadcast participation in one configured network.
@@ -167,7 +241,7 @@ pub enum Request {
///
/// Asked of the running agent because that is the only way to add a
/// network to an agent that is already up: the state directory belongs
/// to one live agent, so a second `tsunagi up` cannot.
/// to one live agent, so a second `tsng up` cannot.
Join {
/// The network name.
name: String,
@@ -180,11 +254,52 @@ pub enum Request {
#[serde(default)]
broadcast: Option<bool>,
},
/// Ask for a configured network's shared secret, in its `tsn1…` text form,
/// so a local GUI or script can copy it to re-share. The reply travels only
/// over the owner/group socket and the secret is never logged.
Secret {
/// Which network, by the id text `status` prints.
network_id: String,
},
/// Offer this agent as an exit node in one network, or stop. Off unless
/// asked.
SetExitOffer {
/// Public network identifier.
network_id: String,
/// Whether members may send their internet traffic through this agent.
enabled: bool,
},
/// Send all this device's internet traffic through a member of one
/// network, or back the ordinary way.
SetExitNode {
/// Public network identifier.
network_id: String,
/// The member's endpoint id, or `None` to stop using an exit node.
peer: Option<String>,
},
}
impl std::fmt::Debug for Request {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
Request::SaveVip {
network_id, config, ..
} => write!(f, "SaveVip {{ {network_id}, {} }}", config.id),
Request::ForgetNode {
network_id,
peer,
expected_version,
} => write!(
f,
"ForgetNode {{ {network_id}, peer: {peer}, version: {expected_version} }}"
),
Request::SetProtocols {
network_id,
protocols,
} => write!(
f,
"SetProtocols {{ {network_id}, protocols: {protocols:?} }}"
),
Request::Status => f.write_str("Status"),
Request::SetBroadcast {
network_id,
@@ -200,6 +315,14 @@ impl std::fmt::Debug for Request {
write!(f, "Dns {{ enable: {enable}, port: {port:?} }}")
}
Request::Join { name, .. } => write!(f, "Join {{ name: {name}, secret: <redacted> }}"),
Request::Secret { network_id } => write!(f, "Secret {{ {network_id} }}"),
Request::SetExitOffer {
network_id,
enabled,
} => write!(f, "SetExitOffer {{ {network_id}, enabled: {enabled} }}"),
Request::SetExitNode { network_id, peer } => {
write!(f, "SetExitNode {{ {network_id}, peer: {peer:?} }}")
}
}
}
}
@@ -208,6 +331,12 @@ impl std::fmt::Debug for Request {
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[non_exhaustive]
pub enum Response {
/// Accepted configuration with fresh local routing observations.
Vips(Vec<crate::vip::VipStatus>),
/// An observed offline claim was durably retired.
NodeForgotten(ForgottenNodeReport),
/// Accepted ordered protocol selection.
Protocols(Vec<String>),
/// A status report.
Status(Box<StatusReport>),
/// Accepted local broadcast participation.
@@ -224,10 +353,202 @@ pub enum Response {
///
/// `None` means it is not serving at all.
Dns(Option<DnsReport>),
/// A network's shared secret, redacted in `Debug`.
Secret(RedactedText),
/// Whether this agent now offers itself as an exit node.
ExitOffer(bool),
/// The member this device now sends its internet traffic through.
ExitNode(Option<String>),
/// The request could not be served.
Error(String),
}
/// The durable result of an offline-node cleanup.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct ForgottenNodeReport {
/// Selected device identity.
pub peer: String,
/// The retired claim version, not a future-version ban.
pub record_version: u64,
/// Peers to which cleanup was immediately queued.
pub peers_told: u32,
}
/// Saves a whole VIP draft through the running agent.
pub async fn save_vip(
socket: &Path,
network_id: &str,
config: crate::vip::VipConfig,
expected: Option<String>,
delete: bool,
) -> Result<Vec<crate::vip::VipStatus>> {
match exchange(
socket,
&Request::SaveVip {
network_id: network_id.into(),
config,
expected,
delete,
},
EXCHANGE_TIMEOUT,
)
.await?
{
Response::Vips(vips) => Ok(vips),
Response::Error(reason) => Err(Error::Rejected { reason }),
_ => Err(Error::Rejected {
reason: "unexpected VIP response".into(),
}),
}
}
/// Resolves a service by exact name, full id or unique id prefix.
pub fn resolve_vip<'a>(
network: &'a NetworkReport,
selector: &str,
) -> std::result::Result<&'a crate::vip::VipStatus, String> {
let matches: Vec<_> = network
.vips
.iter()
.filter(|v| v.config.name == selector || v.config.id.starts_with(selector))
.collect();
match matches.as_slice() {
[vip] => Ok(vip),
[] => Err(format!("no VIP matches `{selector}`")),
_ => Err(format!(
"ambiguous VIP `{selector}`: {}",
matches
.iter()
.map(|v| format!("{} ({})", v.config.name, v.config.id))
.collect::<Vec<_>>()
.join(", ")
)),
}
}
/// Resolves a node by exact hostname, full id, unique id prefix or overlay IPv4.
/// Ambiguity is reported with names and full ids and never guessed.
pub fn resolve_node<'a>(
network: &'a NetworkReport,
selector: &str,
) -> std::result::Result<&'a MemberReport, String> {
let selector = selector.trim();
if selector.is_empty() {
return Err("select a node by name, id or IPv4 address".into());
}
if let Some(member) = network.members.iter().find(|m| m.endpoint_id == selector) {
return Ok(member);
}
let prefix = selector.trim_end_matches('…');
let matches: Vec<_> = network
.members
.iter()
.filter(|m| {
m.hostname.as_deref() == Some(selector)
|| m.overlay_address_v4.as_deref() == Some(selector)
|| (!prefix.is_empty() && m.endpoint_id.starts_with(prefix))
})
.collect();
match matches.as_slice() {
[one] => Ok(one),
[] => Err(format!(
"no signed node matches `{selector}` in `{}`",
network.name
)),
many => Err(format!(
"`{selector}` is ambiguous; use a full id or unique id prefix:\n{}",
many.iter()
.map(|m| format!(
" {} {} {}",
m.hostname.as_deref().unwrap_or("unnamed"),
m.endpoint_id,
m.overlay_address_v4.as_deref().unwrap_or("no IPv4")
))
.collect::<Vec<_>>()
.join("\n")
)),
}
}
/// Retires an exact selected claim; an online or changed target is rejected.
pub async fn forget_node(
socket: &Path,
network: &str,
peer: &str,
expected_version: u64,
) -> Result<ForgottenNodeReport> {
let request = Request::ForgetNode {
network_id: network.into(),
peer: peer.into(),
expected_version,
};
match exchange(socket, &request, EXCHANGE_TIMEOUT).await? {
Response::NodeForgotten(report) => Ok(report),
Response::Error(reason) => Err(Error::ForgetNode(reason)),
other => Err(Error::Storage(format!("unexpected response: {other:?}"))),
}
}
#[cfg(test)]
mod node_selector_tests {
use super::*;
#[test]
fn exact_names_addresses_and_unique_prefixes_select_the_same_signed_node() {
let node = MemberReport {
endpoint_id: "abcdef01".into(),
hostname: Some("laptop".into()),
overlay_address_v4: Some("10.2.0.3".into()),
record_version: 42,
..Default::default()
};
let network = NetworkReport {
name: "home".into(),
members: vec![node.clone()],
..Default::default()
};
for selector in ["laptop", "10.2.0.3", "abcdef01", "abc…"] {
assert_eq!(resolve_node(&network, selector), Ok(&node));
}
for selector in ["", "LAPTOP", "missing"] {
assert!(resolve_node(&network, selector).is_err());
}
}
#[test]
fn collisions_between_names_addresses_and_prefixes_require_an_explicit_identity() {
let first = MemberReport {
endpoint_id: "abcdef01".into(),
hostname: Some("duplicate".into()),
overlay_address_v4: Some("10.2.0.3".into()),
..Default::default()
};
let second = MemberReport {
endpoint_id: "abcdef02".into(),
hostname: Some("duplicate".into()),
overlay_address_v4: Some("10.2.0.3".into()),
..Default::default()
};
let third = MemberReport {
endpoint_id: "12345678".into(),
hostname: Some("abc".into()),
..Default::default()
};
let network = NetworkReport {
name: "home".into(),
members: vec![first.clone(), second, third],
..Default::default()
};
for selector in ["duplicate", "10.2.0.3", "abc"] {
let result = resolve_node(&network, selector);
assert!(result.is_err_and(|error| error.contains("ambiguous")
&& error.contains("abcdef01")
&& error.contains("abcdef02")
&& error.contains("duplicate")));
}
assert_eq!(resolve_node(&network, "abcdef01"), Ok(&first));
}
}
/// What happened when a network was stopped or started.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct ActiveReport {
@@ -240,6 +561,9 @@ pub struct ActiveReport {
/// Stopping something already stopped is not an error — it is the state
/// asked for — but it is worth saying which happened.
pub changed: bool,
/// Networks suspended by switching to this network.
#[serde(default)]
pub suspended: Vec<RangeConflictReport>,
}
/// What happened when a network was joined.
@@ -261,6 +585,13 @@ pub struct JoinedReport {
/// Almost always a mistyped secret, and the one thing that makes two
/// sections of a report look like one network.
pub name_shared_with: Option<String>,
/// Whether it started; false means it is configured but blocked by overlap.
#[serde(default = "joined_active_default")]
pub active: bool,
}
fn joined_active_default() -> bool {
true
}
/// What happened when a network was left.
@@ -283,6 +614,8 @@ pub struct LeftReport {
/// Everything the agent is doing, in one snapshot.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct StatusReport {
/// Data-plane protocols this agent can enable at runtime.
pub available_protocols: Vec<ProtocolReport>,
/// This device's persistent endpoint id.
pub endpoint_id: String,
/// Hostname announced to peers.
@@ -295,6 +628,38 @@ pub struct StatusReport {
pub networks: Vec<NetworkReport>,
/// The local DNS service, when one was asked for.
pub dns: Option<DnsReport>,
/// The state directory the agent is actually using.
///
/// A client that was not told one would look in its own default, which is
/// not the agent's when the agent runs as a service under another user.
pub state_dir: String,
/// The cache directory the agent is actually using.
pub cache_dir: String,
/// The executable the agent runs, which is what needs the capability.
pub program: String,
/// Whether the agent itself can manage the overlay interface.
///
/// It is the agent that creates the interface, so the client's own
/// capabilities say nothing about it.
pub privilege: PrivilegeReport,
/// The version of the agent's software.
pub version: String,
/// The data plane protocols it has, each as `name vN` (wire version).
pub protocols: Vec<String>,
}
/// What the agent found when it checked whether it may manage an interface.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub enum PrivilegeReport {
/// Not checked.
#[default]
Unknown,
/// It can.
Available,
/// It cannot, with what was found.
Missing(String),
/// This platform has no provisioner yet.
Unsupported,
}
/// The local DNS service.
@@ -321,9 +686,37 @@ pub struct DnsReport {
pub names: u32,
}
/// An active network with an overlapping overlay range.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct RangeConflictReport {
/// Public network id, for unambiguous selection.
pub network_id: String,
/// Configured network name.
pub name: String,
/// The overlapping range.
pub range: String,
}
/// A data-plane protocol available in the running agent.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct ProtocolReport {
/// Stable protocol name used by the CLI and network negotiation.
pub name: String,
/// A short description of encryption and transport.
pub summary: String,
}
/// One network.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct NetworkReport {
/// Shared VIPs and this client's routing observations.
#[serde(default)]
pub vips: Vec<crate::vip::VipStatus>,
/// Enabled local data-plane protocols, in preference order.
pub protocols: Vec<String>,
/// Whether the network starts or resumes automatically when its range is free.
#[serde(default)]
pub auto_start: bool,
/// Whether this agent participates in LAN broadcasts in this network.
#[serde(default)]
pub broadcast: bool,
@@ -366,19 +759,70 @@ pub struct NetworkReport {
/// The range it could not have, because another network on this agent
/// already holds it.
pub range_conflict: Option<String>,
/// Networks that switching to this network would suspend.
#[serde(default)]
pub range_conflicts: Vec<RangeConflictReport>,
/// The overlay, when a protocol is running one.
pub overlay: Option<OverlayReport>,
/// This agent's own name, when another member claimed it first.
pub hostname_conflict: Option<HostnameConflictReport>,
/// Exit-node settings: whether this agent offers one, and whether it
/// uses one.
pub exit: ExitReport,
}
/// How packets reach a member right now.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub enum DataPathReport {
/// Nothing carries packets to it: it is away, or only connected for
/// control while a data path is being set up.
#[default]
None,
/// A direct link.
Direct {
/// The kind of transport carrying it, e.g. `wg`.
transport: String,
},
/// Through other members.
Relayed {
/// Number of links on the way, at least two.
hops: u8,
/// The member the first of them goes to, as an endpoint id.
via: String,
},
}
/// A name somebody else claimed first.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct HostnameConflictReport {
/// The name that is taken.
pub name: String,
/// The member that holds it, as an endpoint id.
pub holder: String,
}
/// One member of the network, from signed state.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct MemberReport {
/// Signed claim version used to reject stale cleanup requests.
pub record_version: u64,
/// The member's device identity.
pub endpoint_id: String,
/// The IPv4 overlay address it claimed and signed for.
pub overlay_address_v4: Option<String>,
/// Consecutive failed dial attempts, when this agent is trying to reach it.
pub failed_dials: u32,
/// The last hostname this agent saw it announce, kept while it is away.
pub hostname: Option<String>,
/// Whether this agent can reach it now, by any path. The one answer
/// every client gives to "is it online": a member reached through
/// others is online, over a longer path.
pub online: bool,
/// How packets reach it.
pub path: DataPathReport,
/// The name it claimed, when somebody else claimed it first. It still
/// works by address; it is not found by that name.
pub hostname_conflict: Option<HostnameConflictReport>,
}
/// One control plane peer.
@@ -393,6 +837,10 @@ pub struct PeerReport {
pub transport: String,
/// Round-trip time in milliseconds, when a path is selected.
pub rtt_ms: Option<u64>,
/// The peer offers itself as an exit node in this network.
pub exit_node: bool,
/// How packets reach it.
pub path: DataPathReport,
}
/// The WireGuard overlay of one network.
@@ -425,6 +873,35 @@ pub struct OverlayReport {
pub broadcast_rules: Option<HostRulesReport>,
}
/// How one set of exit-node host rules went.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct RuleSetReport {
/// Whether every step took.
pub ok: bool,
/// One line: what is in place, or what is missing and why.
pub detail: String,
}
/// One network's exit-node settings and what they are doing.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct ExitReport {
/// This agent offers itself as an exit node in this network.
pub offering: bool,
/// Masquerading and forwarding rules for members' traffic, when offering.
pub offer_rules: Option<RuleSetReport>,
/// Whether the kernel forwards packets for the overlay interface.
/// `Some(false)` means the rules are in place and nothing passes until it
/// is turned on. `None` when unknown.
pub forwarding: Option<bool>,
/// The member this device sends its internet traffic through, as an
/// endpoint id.
pub via: Option<String>,
/// Whether that member is connected and still offering to be one.
pub via_online: bool,
/// The routes that send this device's traffic through it.
pub client_rules: Option<RuleSetReport>,
}
/// What the agent installed on the host so LAN broadcast reaches the overlay.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct HostRulesReport {
@@ -457,6 +934,10 @@ pub struct OverlayPeerReport {
pub tx_packets: u64,
/// Packets decrypted from this peer.
pub rx_packets: u64,
/// Plaintext bytes encrypted and sent to this peer.
pub tx_bytes: u64,
/// Plaintext bytes decrypted from this peer.
pub rx_bytes: u64,
/// Data packets dropped: wrong source address, or too large for the path.
pub dropped: u64,
/// WireGuard protocol errors.
@@ -494,6 +975,33 @@ impl OverlayPeerReport {
/// running and what they can report or change. That is what keeps this module
/// free of any knowledge of them.
pub trait ReportSource: Send + Sync + 'static {
/// Atomically saves a complete VIP draft using its original revision.
fn save_vip(
&self,
_network: String,
_config: crate::vip::VipConfig,
_expected: Option<String>,
_delete: bool,
) -> BoxFuture<'_, std::result::Result<Vec<crate::vip::VipStatus>, String>> {
Box::pin(async { Err("this agent cannot manage VIPs".into()) })
}
/// Persists and announces a version-scoped node cleanup.
fn forget_node(
&self,
_network: String,
_peer: String,
_expected_version: u64,
) -> BoxFuture<'_, std::result::Result<ForgottenNodeReport, String>> {
Box::pin(async { Err("this agent cannot forget node history".into()) })
}
/// Saves and applies a local ordered protocol set for one network.
fn set_protocols(
&self,
_network_id: String,
_protocols: Vec<String>,
) -> BoxFuture<'_, std::result::Result<Vec<String>, String>> {
Box::pin(async { Err("this agent cannot change protocols".into()) })
}
/// Produces a fresh report.
fn report(&self) -> BoxFuture<'_, StatusReport>;
@@ -560,6 +1068,39 @@ pub trait ReportSource: Send + Sync + 'static {
) -> BoxFuture<'_, std::result::Result<Option<DnsReport>, String>> {
Box::pin(async move { Err("this agent cannot serve DNS".to_string()) })
}
/// Returns a configured network's shared secret in its `tsn1…` text form.
///
/// Defaulted to a refusal, like the others. An implementor reads it from
/// local state; it must never be logged.
fn network_secret(
&self,
_network_id: String,
) -> BoxFuture<'_, std::result::Result<String, String>> {
Box::pin(async move { Err("this agent cannot read secrets".to_string()) })
}
/// Offers this agent as an exit node in a network, or stops.
///
/// Defaulted to a refusal, like the others.
fn set_exit_offer(
&self,
_network_id: String,
_enabled: bool,
) -> BoxFuture<'_, std::result::Result<bool, String>> {
Box::pin(async move { Err("this agent cannot be an exit node".to_string()) })
}
/// Sends this device's internet traffic through a member, or stops.
///
/// Defaulted to a refusal, like the others.
fn set_exit_node(
&self,
_network_id: String,
_peer: Option<String>,
) -> BoxFuture<'_, std::result::Result<Option<String>, String>> {
Box::pin(async move { Err("this agent cannot use an exit node".to_string()) })
}
}
impl<F> ReportSource for F
@@ -575,7 +1116,7 @@ where
///
/// A local answer comes from memory, so anything this slow means the agent is
/// wedged rather than busy. Saying so beats waiting: unbounded, one wedged
/// runtime leaves `tsunagi status` hanging with nothing on screen and no way
/// runtime leaves `tsng status` hanging with nothing on screen and no way
/// out but Ctrl-C.
pub const EXCHANGE_TIMEOUT: Duration = Duration::from_secs(5);
@@ -590,7 +1131,7 @@ pub const EXCHANGE_TIMEOUT: Duration = Duration::from_secs(5);
///
/// Bump it whenever [`Request`], [`Response`] or anything they contain
/// changes shape.
pub const CONTROL_PROTOCOL: u32 = u32::from_be_bytes([b'T', b'S', b'N', 15]);
pub const CONTROL_PROTOCOL: u32 = u32::from_be_bytes([b'T', b'S', b'N', 22]);
/// Reads one request off an accepted stream, answers it, writes the response.
///
@@ -614,6 +1155,30 @@ where
}
};
let response = match request {
Request::SaveVip {
network_id,
config,
expected,
delete,
} => match source.save_vip(network_id, config, expected, delete).await {
Ok(vips) => Response::Vips(vips),
Err(reason) => Response::Error(reason),
},
Request::ForgetNode {
network_id,
peer,
expected_version,
} => match source.forget_node(network_id, peer, expected_version).await {
Ok(report) => Response::NodeForgotten(report),
Err(reason) => Response::Error(reason),
},
Request::SetProtocols {
network_id,
protocols,
} => match source.set_protocols(network_id, protocols).await {
Ok(protocols) => Response::Protocols(protocols),
Err(reason) => Response::Error(reason),
},
Request::Status => Response::Status(Box::new(source.report().await)),
Request::SetBroadcast {
network_id,
@@ -648,6 +1213,23 @@ where
Ok(report) => Response::Dns(report),
Err(reason) => Response::Error(reason),
},
Request::Secret { network_id } => match source.network_secret(network_id).await {
Ok(secret) => Response::Secret(RedactedText::new(secret)),
Err(reason) => Response::Error(reason),
},
Request::SetExitOffer {
network_id,
enabled,
} => match source.set_exit_offer(network_id, enabled).await {
Ok(enabled) => Response::ExitOffer(enabled),
Err(reason) => Response::Error(reason),
},
Request::SetExitNode { network_id, peer } => {
match source.set_exit_node(network_id, peer).await {
Ok(peer) => Response::ExitNode(peer),
Err(reason) => Response::Error(reason),
}
}
};
write_message(&mut stream, &response).await
}
@@ -752,7 +1334,24 @@ pub async fn join_network_with_broadcast(
}
}
/// Changes one network's broadcast participation while the agent runs.
/// Replaces the enabled data-plane protocols of one configured network.
pub async fn set_protocols(
socket: &Path,
network_id: &str,
protocols: Vec<String>,
) -> Result<Vec<String>> {
let request = Request::SetProtocols {
network_id: network_id.to_owned(),
protocols,
};
match exchange(socket, &request, EXCHANGE_TIMEOUT).await? {
Response::Protocols(protocols) => Ok(protocols),
Response::Error(reason) => Err(Error::Storage(reason)),
other => Err(Error::Storage(format!("unexpected response: {other:?}"))),
}
}
/// Changes local broadcast participation in a network.
pub async fn set_broadcast(
path: impl AsRef<Path>,
network_id: &str,
@@ -802,6 +1401,56 @@ pub async fn set_dns(
}
}
/// Offers the running agent as an exit node in one network, or stops.
pub async fn set_exit_offer(
path: impl AsRef<Path>,
network_id: &str,
enabled: bool,
) -> Result<bool> {
let request = Request::SetExitOffer {
network_id: network_id.to_owned(),
enabled,
};
match exchange(path.as_ref(), &request, EXCHANGE_TIMEOUT).await? {
Response::ExitOffer(enabled) => Ok(enabled),
Response::Error(message) => Err(Error::Storage(message)),
other => Err(Error::Storage(format!("unexpected answer: {other:?}"))),
}
}
/// Sends the running agent's internet traffic through a member of one
/// network, or back the ordinary way with `None`.
pub async fn set_exit_node(
path: impl AsRef<Path>,
network_id: &str,
peer: Option<&str>,
) -> Result<Option<String>> {
let request = Request::SetExitNode {
network_id: network_id.to_owned(),
peer: peer.map(str::to_owned),
};
match exchange(path.as_ref(), &request, EXCHANGE_TIMEOUT).await? {
Response::ExitNode(peer) => Ok(peer),
Response::Error(message) => Err(Error::Storage(message)),
other => Err(Error::Storage(format!("unexpected answer: {other:?}"))),
}
}
/// Asks a running agent for a configured network's shared secret, in its
/// `tsn1…` text form, so a local client can copy it. The secret is never
/// logged by this path.
pub async fn network_secret(path: impl AsRef<Path>, network_id: &str) -> Result<String> {
let path = path.as_ref();
let request = Request::Secret {
network_id: network_id.to_string(),
};
match exchange(path, &request, EXCHANGE_TIMEOUT).await? {
Response::Secret(secret) => Ok(secret.into_string()),
Response::Error(reason) => Err(Error::Storage(reason)),
other => Err(Error::Storage(format!("unexpected answer: {other:?}"))),
}
}
pub(crate) async fn write_message<S, T>(stream: &mut S, value: &T) -> Result<()>
where
S: AsyncWrite + Unpin,
crates/tsunagi/src/ipc/unix.rs → crates/tsunagi-core/src/ipc/unix.rs
+53 -8
View File
@@ -22,8 +22,8 @@ use crate::error::{Error, Result};
// The transport-agnostic surface, re-exported so this module is a complete
// view of the local control interface on its own.
pub use super::{
CONTROL_PROTOCOL, EXCHANGE_TIMEOUT, ReportSource, is_serving, join_network, leave_network,
request_status, set_active, set_dns, set_hostname,
CONTROL_PROTOCOL, ControlSocketAccess, EXCHANGE_TIMEOUT, ReportSource, is_serving,
join_network, leave_network, request_status, set_active, set_dns, set_hostname,
};
/// Serves the local control interface on a Unix socket.
@@ -39,11 +39,15 @@ impl ControlSocket {
/// A socket file left behind by a crashed agent is replaced, but only
/// after checking that nothing is listening on it, so two live agents
/// never fight over one path.
pub async fn bind(path: impl AsRef<Path>, source: Arc<dyn ReportSource>) -> Result<Self> {
pub async fn bind(
path: impl AsRef<Path>,
source: Arc<dyn ReportSource>,
access: ControlSocketAccess,
) -> Result<Self> {
let path = path.as_ref().to_path_buf();
if let Some(parent) = path.parent() {
crate::storage::create_dir(parent)?;
create_parent(parent, &access)?;
}
if path.exists() {
@@ -61,7 +65,7 @@ impl ControlSocket {
path: path.clone(),
source,
})?;
restrict(&path)?;
apply_access(&path, &access)?;
let task = tokio::spawn(serve(listener, source));
Ok(Self {
@@ -94,9 +98,48 @@ impl Drop for ControlSocket {
}
}
fn restrict(path: &Path) -> Result<()> {
/// Creates the socket's parent directory.
///
/// A group-shared socket must sit in a directory the group can traverse, so it
/// is `0755`; a private socket keeps the owner-only state-directory default.
fn create_parent(parent: &Path, access: &ControlSocketAccess) -> Result<()> {
match access {
ControlSocketAccess::Private => crate::storage::create_dir(parent),
ControlSocketAccess::Group(_) => {
std::fs::create_dir_all(parent).map_err(|source| Error::Io {
path: parent.to_path_buf(),
source,
})?;
set_mode(parent, 0o755)
}
}
}
/// Applies the access policy to the bound socket.
///
/// Private is owner-only `0600`. Group sets the socket's group to the gid and
/// the mode to `0660`; if the group cannot be set — the agent is not a member,
/// say — it falls back to owner-only rather than leaving the group wrong, so
/// the socket is never more open than intended.
fn apply_access(path: &Path, access: &ControlSocketAccess) -> Result<()> {
match access {
ControlSocketAccess::Private => set_mode(path, 0o600),
ControlSocketAccess::Group(gid) => {
if let Err(source) = std::os::unix::fs::chown(path, None, Some(*gid)) {
tracing::warn!(
path = %path.display(), gid, %source,
"cannot set the control socket group; keeping it owner-only"
);
return set_mode(path, 0o600);
}
set_mode(path, 0o660)
}
}
}
fn set_mode(path: &Path, mode: u32) -> Result<()> {
use std::os::unix::fs::PermissionsExt;
std::fs::set_permissions(path, std::fs::Permissions::from_mode(0o600)).map_err(|source| {
std::fs::set_permissions(path, std::fs::Permissions::from_mode(mode)).map_err(|source| {
Error::Io {
path: path.to_path_buf(),
source,
@@ -170,7 +213,9 @@ mod tests {
let source: Arc<dyn ReportSource> = Arc::new(|| -> BoxFuture<'static, StatusReport> {
Box::pin(async { StatusReport::default() })
});
let control = ControlSocket::bind(&path, source).await.unwrap();
let control = ControlSocket::bind(&path, source, ControlSocketAccess::Private)
.await
.unwrap();
let answer = exchange(&path, &Request::Status, EXCHANGE_TIMEOUT)
.await
crates/tsunagi/src/ipc/windows.rs → crates/tsunagi-core/src/ipc/windows.rs
+14 -5
View File
@@ -40,8 +40,8 @@ use tokio::task::JoinHandle;
use crate::error::{Error, Result};
pub use super::{
CONTROL_PROTOCOL, EXCHANGE_TIMEOUT, ReportSource, is_serving, join_network, leave_network,
request_status, set_active, set_dns, set_hostname,
CONTROL_PROTOCOL, ControlSocketAccess, EXCHANGE_TIMEOUT, ReportSource, is_serving,
join_network, leave_network, request_status, set_active, set_dns, set_hostname,
};
/// `ERROR_ACCESS_DENIED`: creating a first instance finds an existing pipe,
@@ -149,7 +149,16 @@ impl ControlSocket {
/// Claiming the first instance of the name is what detects a second agent:
/// if the name already exists, the create is refused and that is reported
/// as the state being locked, so two live agents never share one pipe.
pub async fn bind(path: impl AsRef<Path>, source: Arc<dyn ReportSource>) -> Result<Self> {
///
/// `_access` is accepted for parity with the Unix socket; granting a group
/// here means a pipe security descriptor and is a separate follow-up, so for
/// now the pipe keeps its default (creator plus the built-in access the
/// `PipeSecurityAttributes` set).
pub async fn bind(
path: impl AsRef<Path>,
source: Arc<dyn ReportSource>,
_access: ControlSocketAccess,
) -> Result<Self> {
let path = path.as_ref().to_path_buf();
let name = pipe_name(&path);
@@ -201,7 +210,7 @@ impl Drop for ControlSocket {
/// Accepts one client at a time, keeping a fresh instance ready for the next.
///
/// A named pipe server instance serves a single client, so a new instance is
/// created as soon as one is taken — otherwise a second `tsunagi status` while
/// created as soon as one is taken — otherwise a second `tsng status` while
/// the first is mid-flight would find nothing listening.
async fn serve(
name: String,
@@ -285,7 +294,7 @@ pub(crate) async fn connect(path: &Path) -> Result<NamedPipeClient> {
format!(
"the agent's control pipe exists, but Windows denied access. \
Run this command as the same Windows user and with the same \
elevation as `tsunagi up` (use an administrator terminal if \
elevation as `tsng up` (use an administrator terminal if \
the agent is elevated). Windows error: {err}"
),
),
crates/tsunagi/src/lib.rs → crates/tsunagi-core/src/lib.rs
+1
View File
@@ -51,6 +51,7 @@ pub mod storage;
pub mod task;
#[cfg(feature = "testing")]
pub mod testing;
pub mod vip;
pub use agent::{Agent, AgentStatus, Event, NetworkStatus, PeerStatus};
pub use config::{AgentConfig, Limits, ReconnectPolicy, StoragePaths, TransportPolicy};
crates/tsunagi/src/net.rs → crates/tsunagi-core/src/net.rs
View File
File renamed without changes.
crates/tsunagi/src/overlay/broadcast.rs → crates/tsunagi-core/src/overlay/broadcast.rs
+3
View File
@@ -140,6 +140,7 @@ mod tests {
fn limited_and_directed_broadcasts_never_cross_domains_or_target_disabled_peers() {
let peer = iroh::SecretKey::from_bytes(&[8; 32]).public();
let a = NetworkRoutes {
exit: Default::default(),
range: Some("10.0.0.0/24".parse().unwrap()),
local: Some("10.0.0.1".parse().unwrap()),
peers: vec![("10.0.0.2".parse().unwrap(), peer)],
@@ -149,6 +150,7 @@ mod tests {
},
};
let b = NetworkRoutes {
exit: Default::default(),
range: Some("10.1.0.0/24".parse().unwrap()),
local: Some("10.1.0.1".parse().unwrap()),
peers: vec![("10.1.0.2".parse().unwrap(), peer)],
@@ -185,6 +187,7 @@ mod tests {
assert!(!table.accepts(id("b"), peer, Ipv4Addr::BROADCAST));
assert!(table.accepts(id("a"), peer, Ipv4Addr::BROADCAST));
let disabled = NetworkRoutes {
exit: Default::default(),
broadcast: BroadcastPolicy {
enabled: false,
..a.broadcast.clone()
crates/tsunagi/src/overlay/config.rs → crates/tsunagi-core/src/overlay/config.rs
View File
File renamed without changes.
crates/tsunagi-core/src/overlay/exit.rs
+32
View File
@@ -0,0 +1,32 @@
//! Exit-node host rules: what makes this device an exit node for others, and
//! what sends this device's own traffic through one.
//!
//! The packet side lives in [`super::router`]; this is the side the operating
//! system sees. It is kept apart from the broadcast rules, which have a
//! different lifetime and different tags.
/// How one set of host rules went.
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct RuleSetReport {
/// Whether every step took.
pub ok: bool,
/// One line: what is in place, or what is missing and why.
pub detail: String,
}
/// What the host rules of one network's exit-node settings are doing.
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct ExitRulesReport {
/// Masquerading and forwarding for members' traffic, when this agent
/// offers itself as an exit node here. `None` means none are wanted.
pub offer: Option<RuleSetReport>,
/// Whether the kernel forwards packets for the overlay interface.
///
/// `Some(false)` means the rules are in place but nothing passes until
/// forwarding is turned on, which the agent never does itself. `None`
/// when it could not be read.
pub forwarding: Option<bool>,
/// The routes that send this device's traffic through its exit node.
/// `None` means it uses none through this network.
pub client: Option<RuleSetReport>,
}
crates/tsunagi-core/src/overlay/hostrules/exit.rs
+268
View File
@@ -0,0 +1,268 @@
//! Host rules for exit nodes: what the operating system needs for this agent
//! to carry other members' internet traffic, and for this device's own
//! traffic to go through a member.
//!
//! Two independent halves, planned together because one interface carries
//! both:
//!
//! * **Offering** — one set of rules per network that offers this agent as an
//! exit node, chosen by that network's range: masquerade the range out of
//! every other interface and let it through the forward chain. A network
//! that does not offer has no rules, so its members' packets are dropped by
//! the firewall as well as by the agent.
//! * **Using** — policy routing that sends everything the host does not have
//! a more specific route for into the overlay interface, while the agent's
//! own sockets keep the ordinary route (see `linux_exit`).
//!
//! Nothing a peer announces becomes a host setting. Offering is the owner's
//! switch; using is the user's choice of a member. Every object is tagged,
//! reversible, and removed when it is no longer wanted, when the network
//! leaves or stops, and when the agent stops.
use std::sync::{Arc, Mutex};
use crate::BoxFuture;
use crate::state::Ipv4Range;
use super::RuleOutcome;
/// What the host should hold for one interface's exit settings.
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct ExitHostPlan {
/// The overlay interface.
pub interface: String,
/// The ranges of every network that offers this agent as an exit node.
pub offer: Vec<Ipv4Range>,
/// Whether this device sends its internet traffic through a member.
pub client: bool,
/// The range of every network on the interface. Only the using side
/// reads it, to keep this agent's own traffic *to* the overlay inside it
/// where the host has no per-user routing (macOS).
pub overlay: Vec<Ipv4Range>,
/// Underlay addresses this agent talks to directly and must keep reaching
/// outside the overlay's default route. Only filled while `client` is set
/// and only for hosts that report [`ExitHostRules::needs_bypass`].
pub bypass: Vec<std::net::Ipv4Addr>,
/// Host names (relay servers) that must stay reachable the same way; the
/// host rules resolve them.
pub bypass_hosts: Vec<String>,
}
impl ExitHostPlan {
/// Whether there is nothing to hold.
pub fn is_empty(&self) -> bool {
self.offer.is_empty() && !self.client
}
}
/// What applying a plan achieved.
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct ExitHostReport {
/// One outcome per offered range: its masquerade and forward rules.
pub offer: Vec<(Ipv4Range, RuleOutcome)>,
/// Whether the kernel forwards packets for the interface. `None` when it
/// could not be read. The agent only reads this: turning forwarding on is
/// the owner's decision.
pub forwarding: Option<bool>,
/// The routes and rules that send this device's traffic through the
/// interface, when the plan asked for them.
pub client: Option<RuleOutcome>,
}
/// Installs and removes exit-node rules for an overlay interface.
///
/// Like the broadcast rules, no call fails as a whole: a step that cannot be
/// done is in the report, because the caller's only useful response is to
/// say so.
pub trait ExitHostRules: Send + Sync + std::fmt::Debug + 'static {
/// Makes the host match the plan, replacing what this agent put there
/// for the interface before.
fn apply<'a>(&'a self, plan: &'a ExitHostPlan) -> BoxFuture<'a, ExitHostReport>;
/// Removes everything this agent put there for the interface. Succeeds
/// when nothing is there: it runs on the shutdown path, and once at
/// startup to take away what a crashed run left.
fn clear<'a>(&'a self, interface: &'a str) -> BoxFuture<'a, ()>;
/// Refreshes volatile local socket protection without replacing exit routes.
fn refresh<'a>(&'a self, _plan: &'a ExitHostPlan) -> BoxFuture<'a, Option<RuleOutcome>> {
Box::pin(async { None })
}
/// Whether using an exit node on this host needs the underlay addresses
/// the agent talks to, because the host cannot exempt the agent's own
/// traffic from the default route by user id (Windows). The default is no.
fn needs_bypass(&self) -> bool {
false
}
}
/// The tag every firewall object of one offered range carries.
pub fn exit_tag(interface: &str, range: &Ipv4Range) -> String {
format!("{}{range}", exit_tag_prefix(interface))
}
/// The start of every tag for an interface; what cleanup selects by.
pub fn exit_tag_prefix(interface: &str) -> String {
format!("tsunagi-exit:{interface}:")
}
/// The one-line detail for a set of offer outcomes.
pub fn offer_detail(outcome: &RuleOutcome, range: &Ipv4Range) -> String {
match outcome {
RuleOutcome::Applied => format!("masquerade + forward for {range} in place"),
RuleOutcome::Failed(reason) => reason.clone(),
}
}
/// An in-memory host for exit rules, for tests: nothing is touched.
#[derive(Debug, Clone, Default)]
pub struct MockExitRules {
state: Arc<Mutex<Option<ExitHostPlan>>>,
calls: Arc<Mutex<Vec<String>>>,
forwarding: Arc<Mutex<Option<bool>>>,
failure: Arc<Mutex<Option<String>>>,
needs_bypass: Arc<Mutex<bool>>,
}
impl MockExitRules {
/// An empty host with forwarding turned on.
pub fn new() -> Self {
let rules = Self::default();
*lock(&rules.forwarding) = Some(true);
rules
}
/// What the host reports for kernel forwarding from now on.
pub fn set_forwarding(&self, forwarding: Option<bool>) {
*lock(&self.forwarding) = forwarding;
}
/// Whether this host asks for the underlay addresses, like Windows.
pub fn set_needs_bypass(&self, needs: bool) {
*lock(&self.needs_bypass) = needs;
}
/// Makes every step fail with this reason from now on.
pub fn fail_with(&self, reason: Option<String>) {
*lock(&self.failure) = reason;
}
/// What is installed now.
pub fn installed(&self) -> Option<ExitHostPlan> {
lock(&self.state).clone()
}
/// Every `apply:<interface>` / `clear:<interface>` call, in order.
pub fn calls(&self) -> Vec<String> {
lock(&self.calls).clone()
}
}
fn lock<T>(mutex: &Mutex<T>) -> std::sync::MutexGuard<'_, T> {
match mutex.lock() {
Ok(guard) => guard,
Err(poisoned) => poisoned.into_inner(),
}
}
impl ExitHostRules for MockExitRules {
fn needs_bypass(&self) -> bool {
*lock(&self.needs_bypass)
}
fn apply<'a>(&'a self, plan: &'a ExitHostPlan) -> BoxFuture<'a, ExitHostReport> {
Box::pin(async move {
lock(&self.calls).push(format!("apply:{}", plan.interface));
let outcome = match lock(&self.failure).clone() {
Some(reason) => RuleOutcome::Failed(reason),
None => {
*lock(&self.state) = Some(plan.clone());
RuleOutcome::Applied
}
};
ExitHostReport {
offer: plan
.offer
.iter()
.map(|range| (*range, outcome.clone()))
.collect(),
forwarding: *lock(&self.forwarding),
client: plan.client.then_some(outcome),
}
})
}
fn clear<'a>(&'a self, interface: &'a str) -> BoxFuture<'a, ()> {
Box::pin(async move {
lock(&self.calls).push(format!("clear:{interface}"));
*lock(&self.state) = None;
})
}
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
fn range() -> Ipv4Range {
"10.13.37.0/24".parse().unwrap()
}
#[test]
fn tags_name_the_interface_and_the_range() {
assert_eq!(
exit_tag("tsun0", &range()),
"tsunagi-exit:tsun0:10.13.37.0/24"
);
assert!(exit_tag("tsun0", &range()).starts_with(&exit_tag_prefix("tsun0")));
// Another interface's tag never matches this one's prefix.
assert!(!exit_tag("tsun01", &range()).starts_with(&exit_tag_prefix("tsun0")));
}
#[test]
fn an_empty_plan_is_nothing_to_hold() {
let mut plan = ExitHostPlan::default();
assert!(plan.is_empty());
plan.client = true;
assert!(!plan.is_empty());
}
#[tokio::test]
async fn the_mock_applies_reports_and_clears() {
let rules = MockExitRules::new();
let plan = ExitHostPlan {
interface: "tsun0".into(),
offer: vec![range()],
client: true,
overlay: vec![range()],
..Default::default()
};
let report = rules.apply(&plan).await;
assert_eq!(report.offer[0].1, RuleOutcome::Applied);
assert_eq!(report.forwarding, Some(true));
assert_eq!(report.client, Some(RuleOutcome::Applied));
assert_eq!(rules.installed(), Some(plan));
rules.set_forwarding(Some(false));
rules.fail_with(Some("denied".into()));
let report = rules
.apply(&ExitHostPlan {
interface: "tsun0".into(),
offer: vec![range()],
client: false,
overlay: Vec::new(),
..Default::default()
})
.await;
assert_eq!(report.forwarding, Some(false));
assert!(matches!(report.offer[0].1, RuleOutcome::Failed(_)));
assert_eq!(report.client, None);
rules.clear("tsun0").await;
assert_eq!(rules.installed(), None);
assert_eq!(rules.calls(), ["apply:tsun0", "apply:tsun0", "clear:tsun0"]);
}
}
crates/tsunagi/src/overlay/hostrules/linux.rs → crates/tsunagi-core/src/overlay/hostrules/linux.rs
+183 -39
View File
@@ -33,12 +33,13 @@ const CHAIN: &str = "INPUT";
/// Fixed places `iptables` is installed. Never `PATH`: the agent is
/// privileged and a directory a user controls must not decide what it runs.
const IPTABLES_DIRECTORIES: [&str; 5] = [
const IPTABLES_DIRECTORIES: [&str; 6] = [
"/usr/sbin/iptables",
"/sbin/iptables",
"/usr/bin/iptables",
"/bin/iptables",
"/usr/local/sbin/iptables",
"/run/current-system/sw/bin/iptables",
];
/// The first candidate that exists.
@@ -50,7 +51,7 @@ fn pick_iptables(exists: impl Fn(&str) -> bool) -> Option<&'static str> {
/// What running `iptables` produced.
#[derive(Debug, Clone, PartialEq, Eq)]
enum Ran {
pub(super) enum Ran {
/// There is no `iptables` to run.
Missing,
/// It ran, and exited like this.
@@ -63,32 +64,76 @@ enum Ran {
/// Runs `iptables` with these arguments. A function value so the flow below
/// can be tested without touching the host.
type Run<'a> = &'a dyn Fn(&[String]) -> Ran;
pub(super) type Run<'a> = &'a dyn Fn(&[String]) -> Ran;
/// The match and target of the rule, without the chain or position.
fn rule_spec(plan: &BroadcastRulesPlan) -> Vec<String> {
[
"-i",
plan.interface.as_str(),
"-p",
"udp",
"-s",
&plan.range.to_string(),
"-m",
"comment",
"--comment",
&plan.tag(),
"-j",
"ACCEPT",
]
.into_iter()
.map(str::to_string)
.collect()
///
/// `comment` false leaves the tag off, for a kernel that has no `comment`
/// match; the rule is then recognised by its shape, see [`tagged_deletions`].
fn rule_spec(plan: &BroadcastRulesPlan, comment: bool) -> Vec<String> {
let range = plan.range.to_string();
let tag = plan.tag();
let mut words = vec!["-i", plan.interface.as_str(), "-p", "udp", "-s", &range];
if comment {
words.extend(["-m", "comment", "--comment", &tag]);
}
words.extend(["-j", "ACCEPT"]);
words.into_iter().map(str::to_string).collect()
}
/// Whether the kernel has the `comment` match, which a rule needs to carry
/// its tag. Asked of iptables itself: checking a rule that does not exist
/// fails either way, but only a missing match says so.
pub(super) fn comment_supported(run: Run<'_>) -> bool {
let probe = locked(
[
"-t",
"filter",
"-C",
"FORWARD",
"-m",
"comment",
"--comment",
"tsunagi-probe",
"-j",
"ACCEPT",
]
.map(str::to_string),
);
match run(&probe) {
Ran::Exited {
success: false,
stderr,
..
} => !stderr.contains("No chain/target/match by that name"),
_ => true,
}
}
/// `iptables -S` words as option clauses: an option with its value, a negated
/// one with its `!`. It prints the options in its own order, not the one a
/// rule was written in, so rules are compared by their clauses, sorted.
fn clauses(words: &[String]) -> Vec<String> {
let mut out = Vec::new();
let mut at = 0;
while at < words.len() {
let take = if words[at] == "!" { 3 } else { 2 };
let end = (at + take).min(words.len());
out.push(words[at..end].join(" "));
at = end;
}
out.sort();
out
}
/// Whether two rules say the same thing, whatever order they list it in.
pub(super) fn same_rule(a: &[String], b: &[String]) -> bool {
clauses(a) == clauses(b)
}
/// Waits briefly for the xtables lock instead of failing when another tool
/// holds it.
fn locked(args: impl IntoIterator<Item = String>) -> Vec<String> {
pub(super) fn locked(args: impl IntoIterator<Item = String>) -> Vec<String> {
["-w", "5"]
.into_iter()
.map(str::to_string)
@@ -96,19 +141,19 @@ fn locked(args: impl IntoIterator<Item = String>) -> Vec<String> {
.collect()
}
fn check_args(plan: &BroadcastRulesPlan) -> Vec<String> {
fn check_args(plan: &BroadcastRulesPlan, comment: bool) -> Vec<String> {
locked(
["-C".to_string(), CHAIN.to_string()]
.into_iter()
.chain(rule_spec(plan)),
.chain(rule_spec(plan, comment)),
)
}
fn insert_args(plan: &BroadcastRulesPlan) -> Vec<String> {
fn insert_args(plan: &BroadcastRulesPlan, comment: bool) -> Vec<String> {
locked(
["-I".to_string(), CHAIN.to_string(), "1".to_string()]
.into_iter()
.chain(rule_spec(plan)),
.chain(rule_spec(plan, comment)),
)
}
@@ -118,7 +163,7 @@ fn list_args() -> Vec<String> {
/// Splits one `iptables -S` line into words, honouring double quotes, which
/// `iptables` puts around comments that need them.
fn split_rule_line(line: &str) -> Vec<String> {
pub(super) fn split_rule_line(line: &str) -> Vec<String> {
let mut words = Vec::new();
let mut current = String::new();
let mut quoted = false;
@@ -147,8 +192,28 @@ fn split_rule_line(line: &str) -> Vec<String> {
words
}
/// Whether a rule is the one this module writes for some range of the
/// interface the tag belongs to, with no comment on it.
fn is_uncommented_rule(tag: &str, rest: &[String]) -> bool {
let Some(interface) = tag.strip_prefix("tsunagi:") else {
return false;
};
let Some(range) = rest
.windows(2)
.find(|pair| pair[0] == "-s")
.map(|pair| pair[1].as_str())
else {
return false;
};
let ours: Vec<String> = ["-i", interface, "-p", "udp", "-s", range, "-j", "ACCEPT"]
.map(str::to_string)
.to_vec();
same_rule(&ours, rest)
}
/// The delete commands for every rule in an `iptables -S INPUT` listing that
/// carries exactly this tag. Anything else in the chain is left alone.
/// carries exactly this tag, or, with no `comment` match in the kernel, is
/// exactly the rule written for some range. Anything else is left alone.
fn tagged_deletions(listing: &str, tag: &str) -> Vec<Vec<String>> {
listing
.lines()
@@ -162,7 +227,8 @@ fn tagged_deletions(listing: &str, tag: &str) -> Vec<Vec<String>> {
}
let tagged = rest
.windows(2)
.any(|pair| pair[0] == "--comment" && pair[1] == tag);
.any(|pair| pair[0] == "--comment" && pair[1] == tag)
|| is_uncommented_rule(tag, rest);
tagged.then(|| {
locked(
["-D".to_string(), CHAIN.to_string()]
@@ -175,7 +241,7 @@ fn tagged_deletions(listing: &str, tag: &str) -> Vec<Vec<String>> {
}
/// A failure message with the likely fix for the likely cause.
fn failure(stderr: &str) -> String {
pub(super) fn failure(stderr: &str) -> String {
let text = stderr.trim();
let lower = text.to_ascii_lowercase();
if lower.contains("permission denied") || lower.contains("must be root") {
@@ -197,7 +263,8 @@ fn missing(plan: &BroadcastRulesPlan) -> String {
/// Makes the firewall hold exactly one rule for this interface, the planned one.
fn apply_firewall(run: Run<'_>, plan: &BroadcastRulesPlan) -> RuleOutcome {
let present = match run(&check_args(plan)) {
let comment = comment_supported(run);
let present = match run(&check_args(plan, comment)) {
Ran::Missing => return RuleOutcome::Failed(missing(plan)),
Ran::Exited { success, .. } => success,
};
@@ -227,7 +294,7 @@ fn apply_firewall(run: Run<'_>, plan: &BroadcastRulesPlan) -> RuleOutcome {
return RuleOutcome::Failed(failure(&stderr));
}
}
match run(&insert_args(plan)) {
match run(&insert_args(plan, comment)) {
Ran::Missing => RuleOutcome::Failed(missing(plan)),
Ran::Exited { success: true, .. } => RuleOutcome::Applied,
Ran::Exited { stderr, .. } => RuleOutcome::Failed(failure(&stderr)),
@@ -261,8 +328,24 @@ fn clear_firewall(run: Run<'_>, interface: &str) {
}
/// Runs the real `iptables`.
fn run_iptables(args: &[String]) -> Ran {
let Some(program) = pick_iptables(|path| std::path::Path::new(path).is_file()) else {
pub(super) fn run_iptables(args: &[String]) -> Ran {
run_firewall(false, args)
}
pub(super) fn run_firewall(ipv6: bool, args: &[String]) -> Ran {
let v6_paths: Vec<_> = IPTABLES_DIRECTORIES
.iter()
.map(|p| p.replace("iptables", "ip6tables"))
.collect();
let v6_program = v6_paths
.iter()
.find(|path| std::path::Path::new(path).is_file());
let program = if ipv6 {
v6_program.map(String::as_str)
} else {
pick_iptables(|path| std::path::Path::new(path).is_file())
};
let Some(program) = program else {
return Ran::Missing;
};
match std::process::Command::new(program).args(args).output() {
@@ -283,12 +366,18 @@ fn run_iptables(args: &[String]) -> Ran {
/// The Linux implementation: netlink for the route, `iptables` for the rule.
#[derive(Debug)]
pub struct LinuxHostRules {
route: RouteHandle,
pub(super) route: RouteHandle,
/// Interface, sorted socket snapshot and whether exit routes committed.
pub(super) underlay_sockets:
std::sync::Mutex<Option<(String, Vec<super::linux_exit_mark::Listener>, bool)>>,
}
impl LinuxHostRules {
pub(crate) fn new(route: RouteHandle) -> Self {
Self { route }
Self {
route,
underlay_sockets: std::sync::Mutex::new(None),
}
}
async fn set_route(&self, interface: String, source: Ipv4Addr) -> RuleOutcome {
@@ -306,6 +395,10 @@ impl BroadcastHostRules for LinuxHostRules {
"netlink+iptables"
}
fn exit_rules(&self) -> Option<&dyn super::ExitHostRules> {
Some(self)
}
fn apply<'a>(&'a self, plan: &'a BroadcastRulesPlan) -> BoxFuture<'a, BroadcastRulesReport> {
Box::pin(async move {
let route = self.set_route(plan.interface.clone(), plan.source).await;
@@ -361,7 +454,7 @@ mod tests {
#[test]
fn the_rule_is_udp_only_from_the_overlay_range_on_this_interface() {
let spec = words(&rule_spec(&plan()));
let spec = words(&rule_spec(&plan(), true));
assert_eq!(
spec,
"-i tsun0 -p udp -s 10.13.37.0/24 -m comment --comment tsunagi:tsun0 -j ACCEPT"
@@ -369,8 +462,8 @@ mod tests {
// Nothing here may widen to other protocols or every source.
assert!(spec.contains("-p udp"));
assert!(!spec.contains("0.0.0.0/0"));
assert!(words(&insert_args(&plan())).starts_with("-w 5 -I INPUT 1 -i tsun0"));
assert!(words(&check_args(&plan())).starts_with("-w 5 -C INPUT -i tsun0"));
assert!(words(&insert_args(&plan(), true)).starts_with("-w 5 -I INPUT 1 -i tsun0"));
assert!(words(&check_args(&plan(), true)).starts_with("-w 5 -C INPUT -i tsun0"));
}
#[test]
@@ -404,6 +497,49 @@ mod tests {
assert!(tagged_deletions("", "tsunagi:tsun0").is_empty());
}
#[test]
fn a_kernel_without_the_comment_match_says_so_in_its_error() {
let answer = |stderr: &'static str| {
move |_: &[String]| Ran::Exited {
success: false,
stdout: String::new(),
stderr: stderr.to_string(),
}
};
assert!(!comment_supported(&answer(
"iptables: No chain/target/match by that name."
)));
assert!(comment_supported(&answer(
"iptables: Bad rule (does a matching rule exist in that chain?)."
)));
assert!(comment_supported(&|_: &[String]| Ran::Missing));
}
#[test]
fn a_rule_written_without_a_comment_is_found_by_its_shape() {
// Listed the way iptables prints it: its own option order.
let listing = "\
-P INPUT ACCEPT
-A INPUT -s 10.13.37.0/24 -i tsun0 -p udp -j ACCEPT
-A INPUT -s 10.99.0.0/16 -i tsun0 -p udp -j ACCEPT
-A INPUT -s 10.13.37.0/24 -i tsun1 -p udp -j ACCEPT
-A INPUT -s 10.13.37.0/24 -i tsun0 -p udp -j DROP
-A INPUT -s 10.13.37.0/24 -i tsun0 -p tcp -j ACCEPT
";
let deletions = tagged_deletions(listing, "tsunagi:tsun0");
assert_eq!(deletions.len(), 2, "{deletions:?}");
assert!(words(&deletions[0]).ends_with("-s 10.13.37.0/24 -i tsun0 -p udp -j ACCEPT"));
assert!(words(&deletions[1]).contains("10.99.0.0/16"));
}
#[test]
fn without_a_comment_the_rule_has_none() {
let with = words(&rule_spec(&plan(), true));
let without = words(&rule_spec(&plan(), false));
assert!(with.contains("-m comment --comment tsunagi:tsun0"));
assert_eq!(without, "-i tsun0 -p udp -s 10.13.37.0/24 -j ACCEPT");
}
/// A scripted `iptables`: answers by the leading verb, records every call.
struct Fake {
calls: RefCell<Vec<String>>,
@@ -424,6 +560,14 @@ mod tests {
}
}
fn run(&self, args: &[String]) -> Ran {
// The probe for the `comment` match: not part of what is tested.
if args.iter().any(|word| word == "tsunagi-probe") {
return Ran::Exited {
success: false,
stdout: String::new(),
stderr: "Bad rule".into(),
};
}
self.calls.borrow_mut().push(words(args));
if self.missing {
return Ran::Missing;
crates/tsunagi-core/src/overlay/hostrules/linux_exit.rs
+968
View File
@@ -0,0 +1,968 @@
//! Exit-node host rules on Linux: `iptables` for the offering side, policy
//! routing over netlink for the using side.
//!
//! # Offering
//!
//! For every network that offers this agent as an exit node, three rules,
//! selected by that network's range and carrying the tag
//! `tsunagi-exit:<if>:<range>`:
//!
//! ```text
//! -t nat POSTROUTING -s <range> ! -o <if> -j MASQUERADE
//! -t filter FORWARD -i <if> -s <range> ! -o <if> -j ACCEPT
//! -t filter FORWARD -o <if> -d <range> -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
//! ```
//!
//! `! -o <if>` is the egress without naming it: whatever interface the
//! default route uses, now or after it changes. Kernel forwarding is only
//! *read*. Turning it on changes how the whole host behaves, which is the
//! owner's decision, so when it is off the rules are installed anyway and the
//! report says so.
//!
//! # Using
//!
//! Policy routing, with no `/1` routes and nothing in the main table:
//!
//! ```text
//! table 0x7473: default dev <if>
//! 5205: from all fwmark 0x40000000/0x40000000 lookup main
//! 5206: from all uidrange <euid>-<euid> lookup main
//! 5290: from all lookup main suppress_prefixlength 0
//! 5300: from all lookup 0x7473
//! ```
//!
//! The first two keep the agent's *own* sockets and marked replies off the tunnel: its QUIC, relay,
//! DHT and DNS traffic would otherwise be routed into the interface it is
//! carrying, a loop. The specific-route rule lets the LAN, another
//! VPN, a tunnel to somewhere else — keep working, because only the default
//! route is suppressed. Mark/UID bypass precedes Tailscale; application rules follow it.
//!
//! The UID rule exempts a *user id*, so it is only right when the agent has
//! one of its own, as the packaged service does. An agent started from a login
//! session would take that user's own applications out of the tunnel with it,
//! and the exit node would silently do nothing for them. That is refused, not
//! guessed at: see [`client_refusal`].
use crate::BoxFuture;
use crate::state::Ipv4Range;
use super::RuleOutcome;
use super::exit::{ExitHostPlan, ExitHostReport, ExitHostRules, exit_tag, exit_tag_prefix};
use super::linux::{
LinuxHostRules, Ran, Run, comment_supported, failure, locked, run_firewall, run_iptables,
same_rule, split_rule_line,
};
use super::linux_exit_mark;
/// The chains the offering rules live in: `(table, chain)`.
const CHAINS: [(&str, &str); 2] = [("nat", "POSTROUTING"), ("filter", "FORWARD")];
/// One firewall rule: where it goes, and its match and target.
#[derive(Debug, Clone, PartialEq, Eq)]
struct Rule {
table: &'static str,
chain: &'static str,
spec: Vec<String>,
}
fn owned(words: &[&str]) -> Vec<String> {
words.iter().map(|word| word.to_string()).collect()
}
/// The three rules of one offered range.
///
/// Each carries a comment naming its interface and range, which is how it is
/// found again. A kernel without the `comment` match (small router builds
/// leave it out) cannot take that, so `comment` false leaves it off and the
/// rules are recognised by their shape instead; see [`tagged_rules`].
fn offer_rules(interface: &str, range: &Ipv4Range, comment: bool) -> Vec<Rule> {
let range_text = range.to_string();
let tag = exit_tag(interface, range);
let comment: Vec<String> = if comment {
owned(&["-m", "comment", "--comment", tag.as_str()])
} else {
Vec::new()
};
let spec = |head: &[&str], tail: &[&str]| -> Vec<String> {
owned(head)
.into_iter()
.chain(comment.iter().cloned())
.chain(owned(tail))
.collect()
};
vec![
Rule {
table: "nat",
chain: "POSTROUTING",
spec: spec(
&["-s", &range_text, "!", "-o", interface],
&["-j", "MASQUERADE"],
),
},
Rule {
table: "filter",
chain: "FORWARD",
spec: spec(
&["-i", interface, "-s", &range_text, "!", "-o", interface],
&["-j", "ACCEPT"],
),
},
Rule {
table: "filter",
chain: "FORWARD",
spec: spec(
&[
"-o",
interface,
"-d",
&range_text,
"-m",
"conntrack",
"--ctstate",
"RELATED,ESTABLISHED",
],
&["-j", "ACCEPT"],
),
},
]
}
fn check_args(rule: &Rule) -> Vec<String> {
locked(
owned(&["-t", rule.table, "-C", rule.chain])
.into_iter()
.chain(rule.spec.iter().cloned()),
)
}
fn insert_args(rule: &Rule) -> Vec<String> {
locked(
owned(&["-t", rule.table, "-I", rule.chain, "1"])
.into_iter()
.chain(rule.spec.iter().cloned()),
)
}
fn list_args(table: &str, chain: &str) -> Vec<String> {
locked(owned(&["-t", table, "-S", chain]))
}
/// A rule found in a listing whose comment starts with our prefix.
#[derive(Debug, Clone, PartialEq, Eq)]
struct Found {
/// The whole tag.
tag: String,
/// The words after `-A <chain>`, which is what `-D` needs.
rest: Vec<String>,
}
/// Every rule in an `iptables -S <chain>` listing that is ours: carrying a
/// tag that starts with `prefix`, or, on a kernel with no `comment` match,
/// being exactly one of the three rules this module writes for some range.
/// Anything else in the chain is left alone.
fn tagged_rules(listing: &str, chain: &str, prefix: &str) -> Vec<Found> {
// `tsunagi-exit:<interface>:`
let interface = prefix
.strip_prefix("tsunagi-exit:")
.and_then(|rest| rest.strip_suffix(':'))
.unwrap_or_default();
listing
.lines()
.filter_map(|line| {
let words = split_rule_line(line);
let [append, found_chain, rest @ ..] = words.as_slice() else {
return None;
};
if append != "-A" || found_chain != chain {
return None;
}
let tag = rest
.windows(2)
.find(|pair| pair[0] == "--comment" && pair[1].starts_with(prefix))
.map(|pair| pair[1].clone())
.or_else(|| uncommented_tag(interface, chain, rest))?;
Some(Found {
tag,
rest: rest.to_vec(),
})
})
.collect()
}
/// The tag a rule would have carried, when it is one of ours written without
/// a comment: its words are those of the rule for the range it names.
fn uncommented_tag(interface: &str, chain: &str, rest: &[String]) -> Option<String> {
if interface.is_empty() {
return None;
}
let range: Ipv4Range = rest
.windows(2)
.find(|pair| pair[0] == "-s" || pair[0] == "-d")
.and_then(|pair| pair[1].parse().ok())?;
offer_rules(interface, &range, false)
.iter()
.any(|rule| rule.chain == chain && same_rule(&rule.spec, rest))
.then(|| exit_tag(interface, &range))
}
fn delete_args(table: &str, chain: &str, found: &Found) -> Vec<String> {
locked(
owned(&["-t", table, "-D", chain])
.into_iter()
.chain(found.rest.iter().cloned()),
)
}
fn missing_iptables(range: &Ipv4Range, interface: &str) -> String {
format!(
"iptables not found; install it (the iptables-nft package is enough), or masquerade \
{range} out of every interface but `{interface}` and forward it yourself"
)
}
/// Reads one chain, or says why not.
fn read_chain(run: Run<'_>, table: &str, chain: &str) -> Result<String, Failure> {
match run(&list_args(table, chain)) {
Ran::Missing => Err(Failure::Missing),
Ran::Exited {
success: true,
stdout,
..
} => Ok(stdout),
Ran::Exited { stderr, .. } => Err(Failure::Failed(failure(&stderr))),
}
}
enum Failure {
Missing,
Failed(String),
}
/// Deletes found rules; the first failure ends it.
fn delete_all(
run: Run<'_>,
table: &str,
chain: &str,
found: &[Found],
keep: impl Fn(&Found) -> bool,
) -> Result<(), String> {
for rule in found.iter().filter(|rule| !keep(rule)) {
if let Ran::Exited {
success: false,
stderr,
..
} = run(&delete_args(table, chain, rule))
{
return Err(failure(&stderr));
}
}
Ok(())
}
/// Makes the firewall hold exactly the offering rules of these ranges for the
/// interface, and no others of ours.
fn apply_offer(
run: Run<'_>,
interface: &str,
ranges: &[Ipv4Range],
) -> Vec<(Ipv4Range, RuleOutcome)> {
let fail_all = |reason: String| -> Vec<(Ipv4Range, RuleOutcome)> {
ranges
.iter()
.map(|range| (*range, RuleOutcome::Failed(reason.clone())))
.collect()
};
let prefix = exit_tag_prefix(interface);
let wanted: Vec<String> = ranges.iter().map(|r| exit_tag(interface, r)).collect();
let comment = comment_supported(run);
// What is there now, per chain.
let mut present: Vec<(&str, &str, Vec<Found>)> = Vec::new();
for (table, chain) in CHAINS {
match read_chain(run, table, chain) {
Ok(listing) => present.push((table, chain, tagged_rules(&listing, chain, &prefix))),
Err(Failure::Missing) => {
return match ranges.first() {
Some(range) => fail_all(missing_iptables(range, interface)),
None => Vec::new(),
};
}
Err(Failure::Failed(reason)) => return fail_all(reason),
}
}
// Rules for a range no longer offered, or left by an earlier run.
for (table, chain, found) in &present {
if let Err(reason) = delete_all(run, table, chain, found, |rule| wanted.contains(&rule.tag))
{
return fail_all(reason);
}
}
ranges
.iter()
.map(|range| {
let rules = offer_rules(interface, range, comment);
let tag = exit_tag(interface, range);
let in_place = rules
.iter()
.all(|rule| matches!(run(&check_args(rule)), Ran::Exited { success: true, .. }))
&& present.iter().all(|(_, chain, found)| {
let have = found.iter().filter(|rule| rule.tag == tag).count();
have == rules.iter().filter(|rule| rule.chain == *chain).count()
});
if in_place {
return (*range, RuleOutcome::Applied);
}
// Start this range from a clean slate: a half-installed set, or
// one that differs from the plan, is replaced as a whole.
for (table, chain, found) in &present {
if let Err(reason) = delete_all(run, table, chain, found, |rule| rule.tag != tag) {
return (*range, RuleOutcome::Failed(reason));
}
}
for rule in &rules {
match run(&insert_args(rule)) {
Ran::Missing => {
return (
*range,
RuleOutcome::Failed(missing_iptables(range, interface)),
);
}
Ran::Exited { success: true, .. } => {}
Ran::Exited { stderr, .. } => {
return (*range, RuleOutcome::Failed(failure(&stderr)));
}
}
}
(*range, RuleOutcome::Applied)
})
.collect()
}
/// Removes every offering rule of ours for the interface. Best effort.
fn clear_offer(run: Run<'_>, interface: &str) {
let prefix = exit_tag_prefix(interface);
for (table, chain) in CHAINS {
let Ok(listing) = read_chain(run, table, chain) else {
continue;
};
let found = tagged_rules(&listing, chain, &prefix);
if let Err(reason) = delete_all(run, table, chain, &found, |_| false) {
tracing::warn!(
interface,
"cannot remove an exit node firewall rule: {reason}"
);
}
}
}
/// The kernel's forwarding switch for packets arriving on the interface,
/// read from the text of the sysctl files. Never written.
fn forwarding_state(read: impl Fn(&str) -> Option<String>, interface: &str) -> Option<bool> {
let flag = |text: String| match text.trim() {
"1" => Some(true),
"0" => Some(false),
_ => None,
};
// The name is derived here, but it ends up in a path.
if !interface.is_empty() && !interface.contains(['/', '\0']) && interface != ".." {
let per_interface = format!("/proc/sys/net/ipv4/conf/{interface}/forwarding");
if let Some(state) = read(&per_interface).and_then(flag) {
return Some(state);
}
}
read("/proc/sys/net/ipv4/ip_forward").and_then(flag)
}
fn read_sysctl(path: &str) -> Option<String> {
std::fs::read_to_string(path).ok()
}
fn check_kernel_rpf(overlay: &str) -> Result<(), String> {
let directory = "/proc/sys/net/ipv4/conf";
let value = |name: &str, setting: &str| -> Result<u32, String> {
read_sysctl(&format!("{directory}/{name}/{setting}"))
.and_then(|s| s.trim().parse().ok())
.ok_or_else(|| format!("cannot check {name}/{setting} before installing exit routes"))
};
let mut interfaces = Vec::new();
for entry in std::fs::read_dir(directory).map_err(|err| err.to_string())? {
let entry = entry.map_err(|err| err.to_string())?;
let name = entry.file_name().to_string_lossy().into_owned();
if name != "all" && name != "default" && name != "lo" && name != overlay {
interfaces.push((
name.clone(),
value(&name, "rp_filter")?,
value(&name, "src_valid_mark")?,
));
}
}
linux_exit_mark::check_kernel_rpf(
value("all", "rp_filter")?,
value("all", "src_valid_mark")?,
&interfaces,
)
}
/// The effective user id from the text of `/proc/self/status`.
fn parse_euid(status: &str) -> Option<u32> {
status
.lines()
.find_map(|line| line.strip_prefix("Uid:"))
.and_then(|rest| rest.split_whitespace().nth(1))
.and_then(|effective| effective.parse().ok())
}
/// Why the using side must not be installed for this process, if it must not.
///
/// The bypass rule exempts this process's *user id* from the tunnel. That is
/// right for a service with an id of its own and wrong for a process started
/// from somebody's login session, which shares its id with every application
/// that person runs. A login session is recognised by `XDG_RUNTIME_DIR`,
/// which `systemd --user` and the login managers set and a system service
/// does not. Crude on purpose: refusing a service that was started oddly is
/// a message, and exempting a user's applications is a silent failure.
fn client_refusal(xdg_runtime_dir: Option<&str>) -> Option<String> {
xdg_runtime_dir.filter(|dir| !dir.is_empty()).map(|_| {
"the exit node needs the agent to run as a system service with a user of its own: \
this agent was started from a login session, so routing everything through the \
overlay would take that user's own applications out of the tunnel with it"
.to_string()
})
}
impl LinuxHostRules {
async fn blocking<T: Send + 'static>(
work: impl FnOnce() -> T + Send + 'static,
) -> Result<T, String> {
tokio::task::spawn_blocking(work)
.await
.map_err(|err| format!("the task failed: {err}"))
}
async fn apply_client(&self, interface: &str) -> RuleOutcome {
let xdg = std::env::var("XDG_RUNTIME_DIR").ok();
if let Some(reason) = client_refusal(xdg.as_deref()) {
// Whatever an earlier run left must not stay in force.
let _ = self.clear_client(interface).await;
return RuleOutcome::Failed(reason);
}
let Some(euid) = std::fs::read_to_string("/proc/self/status")
.ok()
.and_then(|status| parse_euid(&status))
else {
return RuleOutcome::Failed("cannot tell which user this agent runs as".to_string());
};
let installed = self
.underlay_sockets
.lock()
.unwrap_or_else(|p| p.into_inner())
.as_ref()
.is_some_and(|(name, _, routed)| name == interface && *routed);
if installed {
// A failed socket refresh must never remove an active exit route.
return self
.refresh_sockets(interface)
.await
.unwrap_or(RuleOutcome::Applied);
}
if let Err(reason) = self.clear_client(interface).await {
return RuleOutcome::Failed(reason);
}
let name = interface.to_string();
let prepared = Self::blocking(move || {
check_kernel_rpf(&name)?;
linux_exit_mark::install(
&run_firewall,
&name,
euid,
std::path::Path::new("/proc/sys/net/ipv6").is_dir(),
)
})
.await;
match prepared {
Ok(Ok(sockets)) => {
*self
.underlay_sockets
.lock()
.unwrap_or_else(|p| p.into_inner()) = Some((interface.into(), sockets, false));
}
Ok(Err(reason)) | Err(reason) => return RuleOutcome::Failed(reason),
}
let route = self.route.clone();
let name = interface.to_string();
let outcome = match Self::blocking(move || route.set_exit_client(&name, euid)).await {
Ok(Ok(())) => RuleOutcome::Applied,
Ok(Err(err)) => RuleOutcome::Failed(err.to_string()),
Err(reason) => RuleOutcome::Failed(reason),
};
if let RuleOutcome::Failed(reason) = outcome {
let reason = match self.clear_client(interface).await {
Ok(()) => reason,
Err(cleanup) => format!("{reason}; cannot roll back exit rules: {cleanup}"),
};
RuleOutcome::Failed(reason)
} else {
if let Some((_, _, routed)) = self
.underlay_sockets
.lock()
.unwrap_or_else(|p| p.into_inner())
.as_mut()
{
*routed = true;
}
outcome
}
}
async fn clear_client(&self, interface: &str) -> Result<(), String> {
let route = self.route.clone();
Self::blocking(move || route.clear_exit_client())
.await?
.map_err(|err| err.to_string())?;
// Route removal must precede removing reply protection.
let name = interface.to_string();
Self::blocking(move || {
linux_exit_mark::clear(
&run_firewall,
&name,
std::path::Path::new("/proc/sys/net/ipv6").is_dir(),
)
})
.await??;
*self
.underlay_sockets
.lock()
.unwrap_or_else(|p| p.into_inner()) = None;
Ok(())
}
async fn refresh_sockets(&self, interface: &str) -> Option<RuleOutcome> {
let previous = self
.underlay_sockets
.lock()
.unwrap_or_else(|p| p.into_inner())
.clone();
if previous
.as_ref()
.is_none_or(|(name, _, routed)| name != interface || !routed)
{
return None;
}
let name = interface.to_string();
let work_name = name.clone();
let result = Self::blocking(move || {
let uid = std::fs::read_to_string("/proc/self/status")
.ok()
.and_then(|s| parse_euid(&s))
.ok_or("cannot read agent UID")?;
check_kernel_rpf(&work_name)?;
let v6 = std::path::Path::new("/proc/sys/net/ipv6").is_dir();
let sockets = linux_exit_mark::local_sockets(uid, v6)?;
if previous
.as_ref()
.is_some_and(|(name, old, _)| name == &work_name && old == &sockets)
{
return Ok(None);
}
linux_exit_mark::refresh(&run_firewall, &work_name, v6, &sockets)?;
Ok(Some(sockets))
})
.await;
match result {
Ok(Ok(None)) => None,
Ok(Ok(Some(sockets))) => {
*self
.underlay_sockets
.lock()
.unwrap_or_else(|p| p.into_inner()) = Some((name, sockets, true));
Some(RuleOutcome::Applied)
}
Ok(Err(reason)) | Err(reason) => Some(RuleOutcome::Failed(reason)),
}
}
}
impl ExitHostRules for LinuxHostRules {
fn refresh<'a>(&'a self, plan: &'a ExitHostPlan) -> BoxFuture<'a, Option<RuleOutcome>> {
Box::pin(async move {
if plan.client {
self.refresh_sockets(&plan.interface).await
} else {
None
}
})
}
fn apply<'a>(&'a self, plan: &'a ExitHostPlan) -> BoxFuture<'a, ExitHostReport> {
Box::pin(async move {
let interface = plan.interface.clone();
let ranges = plan.offer.clone();
let offer = {
let (job_interface, job_ranges) = (interface.clone(), ranges.clone());
Self::blocking(move || apply_offer(&run_iptables, &job_interface, &job_ranges))
.await
.unwrap_or_else(|reason| {
ranges
.iter()
.map(|range| (*range, RuleOutcome::Failed(reason.clone())))
.collect()
})
};
let forwarding = if ranges.is_empty() {
None
} else {
forwarding_state(read_sysctl, &interface)
};
let client = if plan.client {
Some(self.apply_client(&interface).await)
} else {
if let Err(reason) = self.clear_client(&interface).await {
tracing::warn!(%reason, "cannot remove exit-client rules");
}
None
};
ExitHostReport {
offer,
forwarding,
client,
}
})
}
fn clear<'a>(&'a self, interface: &'a str) -> BoxFuture<'a, ()> {
Box::pin(async move {
let name = interface.to_string();
let _ = Self::blocking(move || clear_offer(&run_iptables, &name)).await;
if let Err(reason) = self.clear_client(interface).await {
tracing::warn!(%reason, "cannot remove exit-client rules");
}
})
}
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use std::cell::RefCell;
use super::*;
fn range() -> Ipv4Range {
"10.13.37.0/24".parse().unwrap()
}
fn words(args: &[String]) -> String {
args.join(" ")
}
#[test]
fn the_rules_masquerade_the_range_out_of_every_other_interface_and_forward_it() {
let rules = offer_rules("tsun0", &range(), true);
assert_eq!(rules.len(), 3);
assert_eq!(
words(&rules[0].spec),
"-s 10.13.37.0/24 ! -o tsun0 -m comment --comment tsunagi-exit:tsun0:10.13.37.0/24 -j MASQUERADE"
);
assert_eq!((rules[0].table, rules[0].chain), ("nat", "POSTROUTING"));
assert_eq!(
words(&rules[1].spec),
"-i tsun0 -s 10.13.37.0/24 ! -o tsun0 -m comment --comment tsunagi-exit:tsun0:10.13.37.0/24 -j ACCEPT"
);
assert_eq!((rules[1].table, rules[1].chain), ("filter", "FORWARD"));
assert_eq!(
words(&rules[2].spec),
"-o tsun0 -d 10.13.37.0/24 -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment tsunagi-exit:tsun0:10.13.37.0/24 -j ACCEPT"
);
// Nothing may widen to every source or to the other direction.
for rule in &rules {
assert!(!words(&rule.spec).contains("0.0.0.0/0"));
}
assert!(words(&insert_args(&rules[1])).starts_with("-w 5 -t filter -I FORWARD 1 -i tsun0"));
assert!(words(&check_args(&rules[0])).starts_with("-w 5 -t nat -C POSTROUTING -s"));
}
#[test]
fn only_rules_with_our_tag_for_this_interface_are_selected() {
let listing = "\
-P FORWARD DROP
-A FORWARD -i tsun0 -s 10.13.37.0/24 ! -o tsun0 -m comment --comment tsunagi-exit:tsun0:10.13.37.0/24 -j ACCEPT
-A FORWARD -i tsun1 -m comment --comment tsunagi-exit:tsun1:10.99.0.0/24 -j ACCEPT
-A FORWARD -i tsun01 -m comment --comment tsunagi-exit:tsun01:10.5.0.0/24 -j ACCEPT
-A FORWARD -m comment --comment \"allow tsunagi-exit:tsun0:x\" -j ACCEPT
-A FORWARD -m comment --comment tsunagi:tsun0 -j ACCEPT
-A INPUT -m comment --comment tsunagi-exit:tsun0:10.13.37.0/24 -j ACCEPT
-A FORWARD -j ACCEPT
";
let found = tagged_rules(listing, "FORWARD", &exit_tag_prefix("tsun0"));
assert_eq!(found.len(), 1, "{found:?}");
assert_eq!(found[0].tag, "tsunagi-exit:tsun0:10.13.37.0/24");
assert_eq!(
words(&delete_args("filter", "FORWARD", &found[0])),
"-w 5 -t filter -D FORWARD -i tsun0 -s 10.13.37.0/24 ! -o tsun0 -m comment --comment tsunagi-exit:tsun0:10.13.37.0/24 -j ACCEPT"
);
assert!(tagged_rules("", "FORWARD", "tsunagi-exit:tsun0:").is_empty());
}
/// A scripted `iptables` that remembers which rules it was given.
struct Fake {
calls: RefCell<Vec<String>>,
/// Rules present, as `-S` lines per `(table, chain)`.
listing: RefCell<Vec<(String, String)>>,
missing: bool,
check_ok: bool,
/// A kernel with no `comment` match, as small router builds have.
no_comment: bool,
}
impl Fake {
fn new(check_ok: bool) -> Self {
Self {
calls: RefCell::new(Vec::new()),
listing: RefCell::new(Vec::new()),
missing: false,
check_ok,
no_comment: false,
}
}
fn with_rule(self, table: &str, line: &str) -> Self {
self.listing
.borrow_mut()
.push((table.to_string(), line.to_string()));
self
}
fn run(&self, args: &[String]) -> Ran {
if self.missing {
return Ran::Missing;
}
let line = args.join(" ");
self.calls.borrow_mut().push(line.clone());
let table = args
.iter()
.position(|word| word == "-t")
.map(|at| args[at + 1].clone())
.unwrap_or_default();
let ok = |stdout: String| Ran::Exited {
success: true,
stdout,
stderr: String::new(),
};
if self.no_comment && line.contains("-m comment") {
return Ran::Exited {
success: false,
stdout: String::new(),
stderr: "iptables: No chain/target/match by that name.".to_string(),
};
}
if line.contains(" -S ") {
let listing: Vec<String> = self
.listing
.borrow()
.iter()
.filter(|(t, _)| *t == table)
.map(|(_, l)| l.clone())
.collect();
return ok(listing.join("\n"));
}
if line.contains(" -C ") {
return Ran::Exited {
success: self.check_ok,
stdout: String::new(),
stderr: String::new(),
};
}
ok(String::new())
}
fn verbs(&self, verb: &str) -> Vec<String> {
self.calls
.borrow()
.iter()
.filter(|call| call.contains(&format!(" {verb} ")))
.cloned()
.collect()
}
}
#[test]
fn a_kernel_without_the_comment_match_gets_the_rules_without_comments() {
let mut fake = Fake::new(false);
fake.no_comment = true;
let outcome = apply_offer(&|args| fake.run(args), "tsun0", &[range()]);
assert_eq!(outcome, vec![(range(), RuleOutcome::Applied)]);
let inserted = fake.verbs("-I");
assert_eq!(inserted.len(), 3, "{inserted:?}");
assert!(
inserted.iter().all(|call| !call.contains("comment")),
"{inserted:?}"
);
}
#[test]
fn rules_written_without_a_comment_are_ours_and_are_removed_with_them() {
// The way iptables prints them: its own option order.
let listing = "\
-P FORWARD ACCEPT
-A FORWARD -s 10.13.37.0/24 -i tsun0 ! -o tsun0 -j ACCEPT
-A FORWARD -d 10.13.37.0/24 -o tsun0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i tsun0 -j ACCEPT
-A FORWARD -s 10.13.37.0/24 -i tsun1 ! -o tsun1 -j ACCEPT
";
let found = tagged_rules(listing, "FORWARD", &exit_tag_prefix("tsun0"));
assert_eq!(found.len(), 2, "{found:?}");
assert!(
found
.iter()
.all(|rule| rule.tag == exit_tag("tsun0", &range()))
);
let nat = "-A POSTROUTING -s 10.13.37.0/24 ! -o tsun0 -j MASQUERADE\n";
assert_eq!(
tagged_rules(nat, "POSTROUTING", &exit_tag_prefix("tsun0")).len(),
1
);
}
#[test]
fn a_fresh_host_gets_the_three_rules() {
let fake = Fake::new(false);
let outcome = apply_offer(&|args| fake.run(args), "tsun0", &[range()]);
assert_eq!(outcome, vec![(range(), RuleOutcome::Applied)]);
let inserted = fake.verbs("-I");
assert_eq!(inserted.len(), 3, "{inserted:?}");
assert!(inserted[0].contains("-t nat -I POSTROUTING 1"));
assert!(fake.verbs("-D").is_empty());
}
#[test]
fn rules_already_in_place_are_left_alone() {
let tag = exit_tag("tsun0", &range());
let fake = Fake::new(true)
.with_rule(
"nat",
&format!("-A POSTROUTING -s 10.13.37.0/24 ! -o tsun0 -m comment --comment {tag} -j MASQUERADE"),
)
.with_rule("filter", &format!("-A FORWARD -i tsun0 -m comment --comment {tag} -j ACCEPT"))
.with_rule("filter", &format!("-A FORWARD -o tsun0 -m comment --comment {tag} -j ACCEPT"));
let outcome = apply_offer(&|args| fake.run(args), "tsun0", &[range()]);
assert_eq!(outcome, vec![(range(), RuleOutcome::Applied)]);
assert!(fake.verbs("-I").is_empty());
assert!(fake.verbs("-D").is_empty());
}
#[test]
fn a_stale_range_is_removed_and_a_half_installed_set_is_replaced() {
let old: Ipv4Range = "10.99.0.0/24".parse().unwrap();
let old_tag = exit_tag("tsun0", &old);
let tag = exit_tag("tsun0", &range());
let fake = Fake::new(true)
.with_rule(
"nat",
&format!(
"-A POSTROUTING -s 10.99.0.0/24 -m comment --comment {old_tag} -j MASQUERADE"
),
)
// Only one of the three for the current range.
.with_rule(
"filter",
&format!("-A FORWARD -i tsun0 -m comment --comment {tag} -j ACCEPT"),
)
// Somebody else's rule, which must survive.
.with_rule("filter", "-A FORWARD -i eth0 -j ACCEPT");
let outcome = apply_offer(&|args| fake.run(args), "tsun0", &[range()]);
assert_eq!(outcome, vec![(range(), RuleOutcome::Applied)]);
let deleted = fake.verbs("-D");
assert!(
deleted.iter().any(|call| call.contains(&old_tag)),
"{deleted:?}"
);
assert!(
deleted.iter().any(|call| call.contains(&tag)),
"{deleted:?}"
);
assert!(
deleted.iter().all(|call| !call.contains("eth0")),
"{deleted:?}"
);
assert_eq!(fake.verbs("-I").len(), 3);
}
#[test]
fn clearing_removes_only_our_rules_for_the_interface() {
let tag = exit_tag("tsun0", &range());
let other = exit_tag("tsun1", &range());
let fake = Fake::new(true)
.with_rule(
"nat",
&format!("-A POSTROUTING -m comment --comment {tag} -j MASQUERADE"),
)
.with_rule(
"nat",
&format!("-A POSTROUTING -m comment --comment {other} -j MASQUERADE"),
)
.with_rule("filter", "-A FORWARD -j ACCEPT");
clear_offer(&|args| fake.run(args), "tsun0");
let deleted = fake.verbs("-D");
assert_eq!(deleted.len(), 1, "{deleted:?}");
assert!(deleted[0].contains(&tag));
}
#[test]
fn a_missing_iptables_is_a_failure_that_says_what_to_do() {
let mut fake = Fake::new(false);
fake.missing = true;
let outcome = apply_offer(&|args| fake.run(args), "tsun0", &[range()]);
let RuleOutcome::Failed(reason) = &outcome[0].1 else {
panic!("{outcome:?}");
};
assert!(reason.contains("iptables not found"), "{reason}");
assert!(reason.contains("10.13.37.0/24"), "{reason}");
// Nothing wanted and nothing to run is not a failure of anything.
assert!(apply_offer(&|args| fake.run(args), "tsun0", &[]).is_empty());
}
#[test]
fn forwarding_is_read_per_interface_then_globally_and_never_written() {
let files = |per: Option<&'static str>, global: Option<&'static str>| {
move |path: &str| match path {
"/proc/sys/net/ipv4/conf/tsun0/forwarding" => per.map(str::to_string),
"/proc/sys/net/ipv4/ip_forward" => global.map(str::to_string),
_ => None,
}
};
assert_eq!(
forwarding_state(files(Some("1\n"), Some("0\n")), "tsun0"),
Some(true)
);
assert_eq!(
forwarding_state(files(Some("0\n"), Some("1\n")), "tsun0"),
Some(false)
);
assert_eq!(
forwarding_state(files(None, Some("1\n")), "tsun0"),
Some(true)
);
assert_eq!(forwarding_state(files(None, None), "tsun0"), None);
assert_eq!(forwarding_state(files(Some("junk"), None), "tsun0"), None);
// A name that would leave the directory is not looked up.
assert_eq!(forwarding_state(files(Some("1"), None), "../../x"), None);
}
#[test]
fn the_effective_uid_is_the_second_field() {
let status = "Name:\ttsunagi\nUid:\t1001\t1002\t1001\t1001\nGid:\t5\t5\t5\t5\n";
assert_eq!(parse_euid(status), Some(1002));
assert_eq!(parse_euid("Name:\tx\n"), None);
}
#[test]
fn a_login_session_is_refused_and_a_service_is_not() {
let reason = client_refusal(Some("/run/user/1000")).unwrap();
assert!(reason.contains("system service"), "{reason}");
assert!(client_refusal(None).is_none());
assert!(client_refusal(Some("")).is_none());
}
}
crates/tsunagi-core/src/overlay/hostrules/linux_exit_kernel.rs
+439
View File
@@ -0,0 +1,439 @@
//! Privileged routing regression, isolated from the host in named namespaces.
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use std::io::{BufRead, BufReader, Read, Write};
use std::net::{SocketAddr, TcpListener, TcpStream, UdpSocket};
use std::os::unix::process::CommandExt;
use std::process::{Child, Command};
use std::time::Duration;
use super::linux::run_firewall;
use super::linux_exit_mark::{clear, install, local_sockets, refresh};
use crate::overlay::NetlinkProvisioner;
const TEST: &str = "overlay::hostrules::linux_exit_kernel::strict_firewall_preserves_underlay_and_application_exit_routing";
const ROLE: &str = "TSUNAGI_EXIT_KERNEL_TEST_ROLE";
fn ip(arguments: &[&str]) -> String {
let result = Command::new("ip").args(arguments).output().unwrap();
assert!(
result.status.success(),
"ip {arguments:?}: {}",
String::from_utf8_lossy(&result.stderr)
);
String::from_utf8_lossy(&result.stdout).into_owned()
}
struct Namespace(String);
impl Drop for Namespace {
fn drop(&mut self) {
let _ = Command::new("ip").args(["netns", "del", &self.0]).status();
}
}
struct Echo(Child);
impl Drop for Echo {
fn drop(&mut self) {
let _ = self.0.kill();
let _ = self.0.wait();
}
}
fn probes() {
let should_pass = std::env::var("TSUNAGI_EXIT_KERNEL_EXPECT").unwrap() == "pass";
for address in ["198.51.100.2:42000", "[2001:db8:2::2]:42000"] {
let address: SocketAddr = address.parse().unwrap();
let socket = UdpSocket::bind(if address.is_ipv4() {
"0.0.0.0:0"
} else {
"[::]:0"
})
.unwrap();
socket
.set_read_timeout(Some(Duration::from_secs(1)))
.unwrap();
socket.send_to(b"rpf", address).unwrap();
let mut packet = [0u8; 3];
assert_eq!(
socket.recv(&mut packet).is_ok(),
should_pass,
"UDP {address}"
);
if should_pass {
assert_eq!(&packet, b"rpf");
}
let connected = TcpStream::connect_timeout(&address, Duration::from_secs(1));
assert_eq!(connected.is_ok(), should_pass, "TCP {address}");
if let Ok(mut stream) = connected {
stream
.set_read_timeout(Some(Duration::from_secs(1)))
.unwrap();
stream.write_all(b"rpf").unwrap();
stream.read_exact(&mut packet).unwrap();
assert_eq!(&packet, b"rpf");
}
}
}
fn run_probes(expect: &str) {
let result = Command::new(std::env::current_exe().unwrap())
.args(["--ignored", "--exact", TEST, "--nocapture"])
.env(ROLE, "probe")
.env("TSUNAGI_EXIT_KERNEL_EXPECT", expect)
.uid(990)
.gid(990)
.output()
.unwrap();
assert!(
result.status.success(),
"{}\n{}",
String::from_utf8_lossy(&result.stdout),
String::from_utf8_lossy(&result.stderr)
);
}
fn passive_listener() {
let listeners: Vec<_> = ["192.0.2.1:42010", "[2001:db8:1::1]:42010"]
.into_iter()
.map(|address| TcpListener::bind(address).unwrap())
.collect();
println!("PASSIVE_READY");
let tasks: Vec<_> = listeners
.into_iter()
.map(|listener| {
std::thread::spawn(move || {
let (mut socket, _) = listener.accept().unwrap();
socket
.set_read_timeout(Some(Duration::from_secs(2)))
.unwrap();
let mut bytes = [0; 3];
socket.read_exact(&mut bytes).unwrap();
socket.write_all(&bytes).unwrap();
})
})
.collect();
for task in tasks {
task.join().unwrap();
}
}
fn udp_listener() {
let sockets: Vec<_> = ["192.0.2.1:42011", "[2001:db8:1::1]:42011"]
.into_iter()
.map(|a| UdpSocket::bind(a).unwrap())
.collect();
println!("PASSIVE_READY");
let tasks: Vec<_> = sockets
.into_iter()
.map(|socket| {
std::thread::spawn(move || {
socket
.set_read_timeout(Some(Duration::from_secs(5)))
.unwrap();
let mut bytes = [0; 3];
let (len, peer) = socket.recv_from(&mut bytes).unwrap();
socket.send_to(&bytes[..len], peer).unwrap();
})
})
.collect();
for task in tasks {
task.join().unwrap();
}
}
fn spawn_listener(role: &str) -> Echo {
let process = Command::new(std::env::current_exe().unwrap())
.args(["--ignored", "--exact", TEST, "--nocapture"])
.env(ROLE, role)
.uid(990)
.gid(990)
.stdout(std::process::Stdio::piped())
.spawn()
.unwrap();
let mut child = Echo(process);
let mut output = BufReader::new(child.0.stdout.take().unwrap());
loop {
let mut line = String::new();
assert_ne!(
output.read_line(&mut line).unwrap(),
0,
"listener exited before readiness"
);
if line.contains("PASSIVE_READY") {
// Keep the reader open for libtest's final status after the socket
// worker finishes; closing it here makes success a broken pipe.
child.0.stdout = Some(output.into_inner());
return child;
}
}
}
fn client() {
for setting in ["rp_filter", "src_valid_mark"] {
std::fs::write(format!("/proc/sys/net/ipv4/conf/all/{setting}"), b"1").unwrap();
}
std::fs::write("/proc/sys/net/ipv4/conf/rpf-c/rp_filter", b"1").unwrap();
for family in [false, true] {
let args: Vec<_> = [
"-t",
"mangle",
"-A",
"PREROUTING",
"-m",
"rpfilter",
"--invert",
"--validmark",
"-j",
"DROP",
]
.into_iter()
.map(str::to_string)
.collect();
let result = run_firewall(family, &args);
assert!(
matches!(result, super::linux::Ran::Exited { success: true, .. }),
"{result:?}"
);
}
let provisioner = NetlinkProvisioner::new().unwrap();
let route = provisioner.host_rules().route;
// Model Tailscale's earlier lookup, without running another VPN daemon.
ip(&["-4", "rule", "add", "priority", "5270", "table", "52"]);
ip(&[
"-4", "route", "add", "default", "dev", "tsun0", "table", "52",
]);
ip(&["-6", "rule", "add", "priority", "5270", "table", "52"]);
ip(&[
"-6",
"route",
"add",
"unreachable",
"default",
"table",
"52",
]);
// A foreign VPN must leave neighbour discovery and connected LAN paths
// usable; otherwise the fixture itself cannot resolve its uplink neighbour.
for prefix in ["fe80::/64", "2001:db8:1::/64"] {
ip(&["-6", "route", "add", "throw", prefix, "table", "52"]);
}
let mut passive = spawn_listener("listener");
route.set_exit_client("tsun0", 990).unwrap();
// uid bypass alone cannot protect reverse lookups, reproducing the bug.
run_probes("blocked");
install(&run_firewall, "tsun0", 990, true).unwrap();
run_probes("pass");
// Pair transports may bind fresh UDP ports after exit activation.
let mut dynamic_udp = spawn_listener("udp-listener");
refresh(
&run_firewall,
"tsun0",
true,
&local_sockets(990, true).unwrap(),
)
.unwrap();
let server = std::env::var("TSUNAGI_EXIT_KERNEL_SERVER").unwrap();
let incoming = Command::new("ip").args(["netns", "exec", &server, "python3", "-c", r#"import socket
for family,source,target in [(socket.AF_INET,'198.51.100.2','192.0.2.1'),(socket.AF_INET6,'2001:db8:2::2','2001:db8:1::1')]:
for kind,port in [(socket.SOCK_STREAM,42010),(socket.SOCK_DGRAM,42011)]:
s=socket.socket(family,kind); s.settimeout(2); s.bind((source,0)); s.connect((target,port)); s.sendall(b'rpf'); assert s.recv(3)==b'rpf'; s.close()
"#]).output().unwrap();
assert!(
incoming.status.success(),
"first inbound TCP/UDP must be protected: {}",
String::from_utf8_lossy(&incoming.stderr)
);
assert!(passive.0.wait().unwrap().success());
assert!(dynamic_udp.0.wait().unwrap().success());
// Repeated apply remains usable, and does not redirect application sockets.
install(&run_firewall, "tsun0", 990, true).unwrap();
run_probes("pass");
assert!(ip(&["-4", "route", "get", "198.51.100.2", "uid", "991"]).contains("dev tsun0"));
let blocked = Command::new("ip")
.args(["-6", "route", "get", "2001:db8:2::2", "uid", "991"])
.output()
.unwrap();
assert!(
!blocked.status.success(),
"application IPv6 must stay blocked"
);
route.clear_exit_client().unwrap();
clear(&run_firewall, "tsun0", true).unwrap();
for family in ["-4", "-6"] {
assert!(
ip(&[family, "rule", "show"]).contains("5270:"),
"foreign routing rules survive cleanup"
);
ip(&[family, "rule", "del", "priority", "5270"]);
}
assert!(ip(&["-4", "route", "get", "198.51.100.2", "uid", "991"]).contains("dev rpf-c"));
assert!(ip(&["-6", "route", "get", "2001:db8:2::2", "uid", "991"]).contains("dev rpf-c"));
for family in [false, true] {
let result = run_firewall(family, &["-t".into(), "mangle".into(), "-S".into()]);
if let super::linux::Ran::Exited { stdout, .. } = result {
assert!(!stdout.contains("tsunagi-exit-underlay:"));
assert!(!stdout.contains("tsunagi-exit-sockets:"));
assert!(
stdout.contains("rpfilter"),
"strict host filtering is retained"
);
}
}
}
#[test]
#[ignore = "needs root, iproute2, iptables/ip6tables and Python; changes only temporary namespaces"]
fn strict_firewall_preserves_underlay_and_application_exit_routing() {
match std::env::var(ROLE).as_deref() {
Ok("probe") => {
probes();
return;
}
Ok("client") => {
client();
return;
}
Ok("listener") => {
passive_listener();
return;
}
Ok("udp-listener") => {
udp_listener();
return;
}
_ => {}
}
let suffix = std::process::id();
let client_name = format!("tsng-rpf-c-{suffix}");
let server_name = format!("tsng-rpf-s-{suffix}");
ip(&["netns", "add", &client_name]);
let _client = Namespace(client_name.clone());
ip(&["netns", "add", &server_name]);
let _server = Namespace(server_name.clone());
ip(&[
"link",
"add",
"rpf-c",
"netns",
&client_name,
"type",
"veth",
"peer",
"name",
"rpf-s",
"netns",
&server_name,
]);
for (namespace, interface, v4, v6) in [
(&client_name, "rpf-c", "192.0.2.1/24", "2001:db8:1::1/64"),
(&server_name, "rpf-s", "192.0.2.2/24", "2001:db8:1::2/64"),
] {
ip(&["-n", namespace, "link", "set", "lo", "up"]);
ip(&["-n", namespace, "addr", "add", v4, "dev", interface]);
ip(&[
"-n", namespace, "-6", "addr", "add", v6, "dev", interface, "nodad",
]);
ip(&["-n", namespace, "link", "set", interface, "up"]);
}
ip(&[
"-n",
&server_name,
"addr",
"add",
"198.51.100.2/32",
"dev",
"lo",
]);
ip(&[
"-n",
&server_name,
"-6",
"addr",
"add",
"2001:db8:2::2/128",
"dev",
"lo",
"nodad",
]);
ip(&[
"-n",
&client_name,
"route",
"add",
"default",
"via",
"192.0.2.2",
]);
ip(&[
"-n",
&client_name,
"-6",
"route",
"add",
"default",
"via",
"2001:db8:1::2",
]);
ip(&["-n", &client_name, "link", "add", "tsun0", "type", "dummy"]);
ip(&[
"-n",
&client_name,
"addr",
"add",
"10.13.37.1/24",
"dev",
"tsun0",
]);
ip(&["-n", &client_name, "link", "set", "tsun0", "up"]);
let echo_code = r#"import socket,selectors,sys
selector=selectors.DefaultSelector()
for family in [socket.AF_INET,socket.AF_INET6]:
for kind in [socket.SOCK_DGRAM,socket.SOCK_STREAM]:
sock=socket.socket(family,kind)
if family==socket.AF_INET6: sock.setsockopt(socket.IPPROTO_IPV6,socket.IPV6_V6ONLY,1)
sock.bind(('198.51.100.2' if family==socket.AF_INET else '2001:db8:2::2',42000))
if kind==socket.SOCK_STREAM: sock.listen()
selector.register(sock,selectors.EVENT_READ,kind)
print('ready',flush=True)
while True:
for key,_ in selector.select():
if key.data==socket.SOCK_DGRAM:
data,address=key.fileobj.recvfrom(128); key.fileobj.sendto(data,address)
else:
conn,_=key.fileobj.accept(); conn.settimeout(2)
try: conn.sendall(conn.recv(128))
finally: conn.close()
"#;
let child = Command::new("ip")
.args([
"netns",
"exec",
&server_name,
"python3",
"-u",
"-c",
echo_code,
])
.stdout(std::process::Stdio::piped())
.spawn()
.unwrap();
let mut echo = Echo(child);
let mut ready = String::new();
BufReader::new(echo.0.stdout.take().unwrap())
.read_line(&mut ready)
.unwrap();
assert_eq!(ready.trim(), "ready");
let result = Command::new("ip")
.args(["netns", "exec", &client_name])
.arg(std::env::current_exe().unwrap())
.args(["--ignored", "--exact", TEST, "--nocapture"])
.env(ROLE, "client")
.env("TSUNAGI_EXIT_KERNEL_SERVER", &server_name)
.output()
.unwrap();
assert!(
result.status.success(),
"{}\n{}",
String::from_utf8_lossy(&result.stdout),
String::from_utf8_lossy(&result.stderr)
);
}
crates/tsunagi-core/src/overlay/hostrules/linux_exit_mark.rs
+535
View File
@@ -0,0 +1,535 @@
//! Own-bit connection marks for exit-client underlay traffic.
//! Reply marks are restored before host reverse-path filtering, never accepted
//! around it. No peer address becomes a host route or firewall argument.
use super::linux::{Ran, failure, locked, split_rule_line};
use std::net::{IpAddr, Ipv4Addr, Ipv6Addr};
/// Outside Tailscale's 0xff0000 mask and ordinary low-bit WireGuard marks.
pub(crate) const UNDERLAY_MARK: u32 = 0x4000_0000;
pub(crate) const UNDERLAY_MASK: u32 = UNDERLAY_MARK;
const MARK: &str = "0x40000000/0x40000000";
const MASK: &str = "0x40000000";
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord)]
pub(super) struct Listener {
address: IpAddr,
port: u16,
tcp: bool,
}
fn parse_listeners(listing: &str, uid: u32, v6: bool, tcp: bool) -> Result<Vec<Listener>, String> {
let mut result = Vec::new();
for line in listing.lines().skip(1) {
let fields: Vec<_> = line.split_whitespace().collect();
if (tcp && fields.get(3) != Some(&"0A"))
|| fields.get(7).and_then(|s| s.parse::<u32>().ok()) != Some(uid)
{
continue;
}
let (address, port) = fields
.get(1)
.and_then(|s| s.split_once(':'))
.ok_or("malformed local socket")?;
let port = u16::from_str_radix(port, 16).map_err(|e| e.to_string())?;
let address = if v6 {
let mut bytes = [0; 16];
for (word, chunk) in bytes.as_chunks_mut::<4>().0.iter_mut().enumerate() {
let hex = address
.get(word * 8..word * 8 + 8)
.ok_or("malformed local IPv6 listener")?;
chunk.copy_from_slice(
&u32::from_str_radix(hex, 16)
.map_err(|e| e.to_string())?
.to_ne_bytes(),
);
}
IpAddr::V6(Ipv6Addr::from(bytes))
} else {
IpAddr::V4(Ipv4Addr::from(
u32::from_str_radix(address, 16)
.map_err(|e| e.to_string())?
.to_ne_bytes(),
))
};
if result.len() == 128 {
return Err("too many local sockets for exit protection".into());
}
if port != 0 {
result.push(Listener { address, port, tcp });
}
}
Ok(result)
}
pub(super) fn local_sockets(uid: u32, v6: bool) -> Result<Vec<Listener>, String> {
let mut listeners = Vec::new();
for (protocol, tcp) in [("tcp", true), ("udp", false)] {
for family in [false, true].into_iter().filter(|f| !f || v6) {
let path = format!("/proc/net/{protocol}{}", if family { "6" } else { "" });
listeners.extend(parse_listeners(
&std::fs::read_to_string(path).map_err(|e| e.to_string())?,
uid,
family,
tcp,
)?);
}
}
listeners.sort();
listeners.dedup();
Ok(listeners)
}
fn listener_rules(interface: &str, listener: &Listener) -> Vec<Vec<String>> {
let mut base = words(&[
"!",
"-i",
interface,
"-p",
if listener.tcp { "tcp" } else { "udp" },
"--dport",
&listener.port.to_string(),
]);
if !listener.address.is_unspecified() {
base.extend(words(&["-d", &listener.address.to_string()]));
}
// Match an actual local socket, not just a destination port. Listening
// sockets need protection on the first packet, before any outbound connmark.
base.extend(words(&[
"-m",
"socket",
"--nowildcard",
"-m",
"comment",
"--comment",
&listener_tag(interface),
]));
let save = base
.iter()
.cloned()
.chain(words(&[
"-m",
"mark",
"--mark",
MARK,
"-j",
"CONNMARK",
"--save-mark",
"--nfmask",
MASK,
"--ctmask",
MASK,
]))
.collect();
let mark = base
.into_iter()
.chain(words(&["-j", "MARK", "--set-xmark", MARK]))
.collect();
vec![save, mark]
}
pub(super) type RunFamily<'a> = &'a dyn Fn(bool, &[String]) -> Ran;
fn tag(interface: &str) -> String {
format!("tsunagi-exit-underlay:{interface}")
}
fn listener_tag(interface: &str) -> String {
format!("tsunagi-exit-sockets:{interface}")
}
fn words(values: &[&str]) -> Vec<String> {
values.iter().map(|s| (*s).into()).collect()
}
fn rules(interface: &str, uid: u32) -> Vec<(&'static str, Vec<String>)> {
let tag = tag(interface);
let uid = uid.to_string();
// Insert at position one in this order: final OUTPUT order is MARK, SAVE.
vec![
(
"PREROUTING",
words(&[
"!",
"-i",
interface,
"-m",
"connmark",
"--mark",
MARK,
"-m",
"comment",
"--comment",
&tag,
"-j",
"CONNMARK",
"--restore-mark",
"--nfmask",
MASK,
"--ctmask",
MASK,
]),
),
(
"OUTPUT",
words(&[
"-m",
"owner",
"--uid-owner",
&uid,
"-m",
"mark",
"--mark",
MARK,
"-m",
"comment",
"--comment",
&tag,
"-j",
"CONNMARK",
"--save-mark",
"--nfmask",
MASK,
"--ctmask",
MASK,
]),
),
(
"OUTPUT",
words(&[
"-m",
"owner",
"--uid-owner",
&uid,
"-m",
"comment",
"--comment",
&tag,
"-j",
"MARK",
"--set-xmark",
MARK,
]),
),
]
}
fn execute(run: RunFamily<'_>, v6: bool, args: Vec<String>) -> Result<String, String> {
match run(v6, &locked(args)) {
Ran::Exited {
success: true,
stdout,
..
} => Ok(stdout),
Ran::Exited { stderr, .. } => Err(failure(&stderr)),
Ran::Missing => Err(format!(
"{} is required for exit-node connection marks",
if v6 { "ip6tables" } else { "iptables" }
)),
}
}
/// Removes only exact interface tags, retaining other agents and host rules.
pub(super) fn clear(run: RunFamily<'_>, interface: &str, v6: bool) -> Result<(), String> {
clear_tags(run, interface, v6, false)
}
fn clear_tags(
run: RunFamily<'_>,
interface: &str,
v6: bool,
sockets_only: bool,
) -> Result<(), String> {
let wanted = tag(interface);
let sockets = listener_tag(interface);
for family in [false, true].into_iter().filter(|family| !family || v6) {
let chains: &[&str] = if sockets_only {
&["PREROUTING"]
} else {
&["OUTPUT", "PREROUTING"]
};
for chain in chains {
let listing = execute(run, family, words(&["-t", "mangle", "-S", chain]))?;
for line in listing.lines() {
let tokens = split_rule_line(line);
let [append, listed_chain, rest @ ..] = tokens.as_slice() else {
continue;
};
if append == "-A"
&& listed_chain == *chain
&& rest.windows(2).any(|pair| {
pair[0] == "--comment"
&& (pair[1] == sockets || (!sockets_only && pair[1] == wanted))
})
{
let args = words(&["-t", "mangle", "-D", chain])
.into_iter()
.chain(rest.iter().cloned())
.collect();
execute(run, family, args)?;
}
}
}
}
Ok(())
}
/// Establishes protection before routing may redirect application traffic.
pub(super) fn install(
run: RunFamily<'_>,
interface: &str,
uid: u32,
v6: bool,
) -> Result<Vec<Listener>, String> {
let listeners = local_sockets(uid, v6)?;
install_with_listeners(run, interface, uid, v6, &listeners)?;
Ok(listeners)
}
fn install_with_listeners(
run: RunFamily<'_>,
interface: &str,
uid: u32,
v6: bool,
listeners: &[Listener],
) -> Result<(), String> {
clear(run, interface, v6)?;
let result = (|| {
for family in [false, true].into_iter().filter(|family| !family || v6) {
for (chain, spec) in rules(interface, uid) {
let args = words(&["-t", "mangle", "-I", chain, "1"])
.into_iter()
.chain(spec)
.collect();
execute(run, family, args)?;
}
for listener in listeners.iter().filter(|s| s.address.is_ipv6() == family) {
for spec in listener_rules(interface, listener) {
execute(
run,
family,
words(&["-t", "mangle", "-I", "PREROUTING", "1"])
.into_iter()
.chain(spec)
.collect(),
)?;
}
}
}
Ok(())
})();
if result.is_err()
&& let Err(cleanup) = clear(run, interface, v6)
{
return Err(format!(
"{}; cannot roll back connection marks: {cleanup}",
result.err().unwrap_or_default()
));
}
result
}
/// Updates passive protection without interrupting established connmark rules.
pub(super) fn refresh(
run: RunFamily<'_>,
interface: &str,
v6: bool,
listeners: &[Listener],
) -> Result<(), String> {
clear_tags(run, interface, v6, true)?;
for family in [false, true].into_iter().filter(|f| !f || v6) {
for listener in listeners.iter().filter(|s| s.address.is_ipv6() == family) {
for spec in listener_rules(interface, listener) {
execute(
run,
family,
words(&["-t", "mangle", "-I", "PREROUTING", "1"])
.into_iter()
.chain(spec)
.collect(),
)?;
}
}
}
Ok(())
}
/// Kernel strict RPF must include the restored mark, independently of iptables.
pub(super) fn check_kernel_rpf(
all_rpf: u32,
all_valid: u32,
interfaces: &[(String, u32, u32)],
) -> Result<(), String> {
for (interface, rpf, valid) in interfaces {
if all_rpf.max(*rpf) == 1 && all_valid.max(*valid) == 0 {
return Err(format!(
"strict rp_filter on `{interface}` ignores connection marks; enable net.ipv4.conf.all.src_valid_mark=1 before using an exit node (rp_filter stays enabled)"
));
}
}
Ok(())
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
use std::cell::RefCell;
#[test]
fn both_families_restore_before_firewall_and_preserve_foreign_mark_bits() {
let calls = RefCell::new(Vec::new());
let run = |v6, args: &[String]| {
calls.borrow_mut().push((v6, args.to_vec()));
Ran::Exited {
success: true,
stdout: String::new(),
stderr: String::new(),
}
};
install_with_listeners(&run, "tsun0", 990, true, &[]).unwrap();
let calls = calls.borrow();
for family in [false, true] {
let inserts: Vec<_> = calls
.iter()
.filter(|(v6, a)| *v6 == family && a.iter().any(|s| s == "-I"))
.map(|(_, a)| a.join(" "))
.collect();
assert_eq!(inserts.len(), 3);
assert!(inserts[0].contains("-I PREROUTING 1 ! -i tsun0 -m connmark"));
assert!(inserts[0].contains("--restore-mark --nfmask 0x40000000 --ctmask 0x40000000"));
assert!(inserts[1].contains("--uid-owner 990"));
assert!(inserts[1].contains("--save-mark"));
assert!(inserts[2].contains("--set-xmark 0x40000000/0x40000000"));
assert!(
!inserts
.iter()
.any(|s| s.contains("-j ACCEPT") || s.contains("-j RETURN"))
);
}
assert_eq!(UNDERLAY_MASK & 0x00ff_0000, 0);
}
#[test]
fn cleanup_retains_other_interfaces_and_rolls_back_partial_ipv6_installation() {
let calls = RefCell::new(Vec::new());
let run = |v6, args: &[String]| {
calls.borrow_mut().push((v6, args.to_vec()));
let listing = "-A OUTPUT -m comment --comment tsunagi-exit-underlay:tsun0 -j MARK --set-xmark 0x40000000/0x40000000\n-A OUTPUT -m comment --comment tsunagi-exit-underlay:tsun1 -j MARK --set-xmark 0x40000000/0x40000000\n-A OUTPUT -m comment --comment unrelated -j RETURN\n";
Ran::Exited {
success: !(v6 && args.iter().any(|s| s == "-I")),
stdout: if args.iter().any(|s| s == "-S") {
listing.into()
} else {
String::new()
},
stderr: "injected IPv6 failure".into(),
}
};
assert!(install_with_listeners(&run, "tsun0", 990, true, &[]).is_err());
let calls = calls.borrow();
let deletions: Vec<_> = calls
.iter()
.filter(|(_, a)| a.iter().any(|s| s == "-D"))
.collect();
assert!(
deletions.len() >= 4,
"both initial and rollback cleanup run"
);
assert!(
deletions
.iter()
.all(|(_, a)| a.iter().any(|s| s == "tsunagi-exit-underlay:tsun0"))
);
}
#[test]
fn kernel_strict_validation_requires_mark_awareness_without_weakening_rpf() {
let interfaces = vec![("eth0".into(), 1, 0)];
assert!(check_kernel_rpf(0, 0, &interfaces).is_err());
assert!(check_kernel_rpf(0, 1, &interfaces).is_ok());
assert!(check_kernel_rpf(2, 0, &interfaces).is_ok());
assert!(check_kernel_rpf(0, 0, &[("eth0".into(), 1, 1)]).is_ok());
}
#[test]
fn passive_tcp_protection_uses_only_locally_owned_listeners() {
let listing = "header\n0: 00000000:01BB 00000000:0000 0A 0:0 0:0 0 990 0 1\n1: 0100007F:1111 00000000:0000 0A 0:0 0:0 0 1000 0 2\n2: 0100007F:2222 00000000:0000 01 0:0 0:0 0 990 0 3\n";
let listeners = parse_listeners(listing, 990, false, true).unwrap();
assert_eq!(
listeners,
vec![Listener {
address: IpAddr::V4(Ipv4Addr::UNSPECIFIED),
port: 443,
tcp: true
}]
);
let rules = listener_rules("tsun0", &listeners[0]);
assert!(
rules[0]
.join(" ")
.contains("-p tcp --dport 443 -m socket --nowildcard")
);
assert!(rules[0].join(" ").contains("--save-mark"));
assert!(rules[1].join(" ").contains("--set-xmark"));
}
#[test]
fn listener_parser_checks_family_encoding_and_bounds() {
let loopback = Ipv6Addr::LOCALHOST.octets();
let hex: String = loopback
.as_chunks::<4>()
.0
.iter()
.map(|bytes| format!("{:08X}", u32::from_ne_bytes(*bytes)))
.collect();
let listing = format!("header\n0: {hex}:01BB 0:0 0A 0:0 0:0 0 990 0 1\n");
assert_eq!(
parse_listeners(&listing, 990, true, true).unwrap(),
vec![Listener {
address: IpAddr::V6(Ipv6Addr::LOCALHOST),
port: 443,
tcp: true
}]
);
assert!(
parse_listeners(
"header\n0: BAD:01BB 0:0 0A 0:0 0:0 0 990 0 1",
990,
true,
true
)
.is_err()
);
let huge = format!(
"header\n{}",
"0: 00000000:01BB 0:0 0A 0:0 0:0 0 990 0 1\n".repeat(129)
);
assert!(parse_listeners(&huge, 990, false, true).is_err());
}
#[test]
fn udp_refresh_preserves_general_connmark_protection_and_other_interfaces() {
let listing = "header\n0: 00000000:ABA0 0:0 07 0:0 0:0 0 990 0 1\n1: 00000000:0000 0:0 07 0:0 0:0 0 990 0 2\n";
let sockets = parse_listeners(listing, 990, false, false).unwrap();
assert_eq!(sockets.len(), 1);
assert!(!sockets[0].tcp);
let calls = RefCell::new(Vec::new());
let run = |v6, args: &[String]| {
calls.borrow_mut().push((v6, args.join(" ")));
Ran::Exited { success: true, stdout: "-A PREROUTING -m comment --comment tsunagi-exit-underlay:tsun0 -j CONNMARK --restore-mark\n-A PREROUTING -m comment --comment tsunagi-exit-sockets:tsun0 -j MARK\n-A PREROUTING -m comment --comment tsunagi-exit-sockets:tsun1 -j MARK\n".into(), stderr: String::new() }
};
refresh(&run, "tsun0", false, &sockets).unwrap();
let calls = calls.borrow();
assert!(
calls
.iter()
.any(|(_, a)| a.contains("-I PREROUTING 1") && a.contains("-p udp"))
);
assert!(
calls
.iter()
.filter(|(_, a)| a.contains("-D"))
.all(|(_, a)| a.contains("tsunagi-exit-sockets:tsun0"))
);
}
}
crates/tsunagi/src/overlay/hostrules/macos.rs → crates/tsunagi-core/src/overlay/hostrules/macos.rs
+47 -3
View File
@@ -19,6 +19,10 @@
use crate::BoxFuture;
use std::sync::{Arc, Mutex};
use super::exit::{ExitHostPlan, ExitHostReport, ExitHostRules};
use super::macos_exit::{apply_plan, clear_all, run_tool};
use super::{BroadcastHostRules, BroadcastRulesPlan, BroadcastRulesReport, RuleOutcome};
/// The limited-broadcast destination routed through the overlay interface.
@@ -60,14 +64,19 @@ fn firewall_todo(plan: &BroadcastRulesPlan) -> String {
)
}
/// The macOS implementation: `route` for the route, no firewall yet.
/// The macOS implementation: `route` for the route, no firewall yet, and
/// `pf` plus `route` for exit nodes (see [`super::macos_exit`]).
#[derive(Debug, Default)]
pub struct MacosHostRules;
pub struct MacosHostRules {
/// The reference `pfctl -E` handed out, held while exit-node rules need
/// `pf` on and released when they are removed.
pub(super) pf_token: Arc<Mutex<Option<String>>>,
}
impl MacosHostRules {
/// Creates the host rules.
pub fn new() -> Self {
Self
Self::default()
}
/// Installs the route, replacing one this agent added earlier so the call
@@ -82,11 +91,46 @@ impl MacosHostRules {
}
}
impl ExitHostRules for MacosHostRules {
fn apply<'a>(&'a self, plan: &'a ExitHostPlan) -> BoxFuture<'a, ExitHostReport> {
Box::pin(async move {
let job_plan = plan.clone();
let token = Arc::clone(&self.pf_token);
tokio::task::spawn_blocking(move || apply_plan(&run_tool, &token, &job_plan))
.await
.unwrap_or_else(|err| {
let failed = RuleOutcome::Failed(format!("the task failed: {err}"));
ExitHostReport {
offer: plan
.offer
.iter()
.map(|range| (*range, failed.clone()))
.collect(),
forwarding: None,
client: plan.client.then_some(failed),
}
})
})
}
fn clear<'a>(&'a self, interface: &'a str) -> BoxFuture<'a, ()> {
Box::pin(async move {
let name = interface.to_string();
let token = Arc::clone(&self.pf_token);
let _ = tokio::task::spawn_blocking(move || clear_all(&run_tool, &token, &name)).await;
})
}
}
impl BroadcastHostRules for MacosHostRules {
fn name(&self) -> &str {
"route"
}
fn exit_rules(&self) -> Option<&dyn ExitHostRules> {
Some(self)
}
fn apply<'a>(&'a self, plan: &'a BroadcastRulesPlan) -> BoxFuture<'a, BroadcastRulesReport> {
Box::pin(async move {
let route = self.set_route(plan.interface.clone()).await;
crates/tsunagi-core/src/overlay/hostrules/macos_exit.rs
+915
View File
@@ -0,0 +1,915 @@
//! Exit-node host rules on macOS: a `pf` anchor and `route` entries.
//!
//! Compiled on every platform so the rule text and the parsers are tested
//! everywhere; only `MacosHostRules` runs them, and only on macOS.
//!
//! # Offering
//!
//! Per offered range, in the anchor `com.apple/tsunagi-exit-<if>`:
//!
//! ```text
//! nat on <egress> inet from <range> to any -> (<egress>)
//! pass in quick on <if> inet from <range> to ! <range>
//! ```
//!
//! The anchor lives under `com.apple/` on purpose: the stock `/etc/pf.conf`
//! evaluates `nat-anchor "com.apple/*"` and `anchor "com.apple/*"`, so the
//! rules take effect without editing the user's ruleset. `pf` cannot say
//! "every interface but one" in a translation address, so the egress is the
//! interface of the default route **when the rules are applied**; after the
//! default route moves (Wi-Fi to Ethernet) the rules are re-applied by
//! switching the exit node off and on, or restarting the agent.
//!
//! The kernel's `net.inet.ip.forwarding` is only read, never written: it
//! changes how the whole host behaves, which is the owner's decision.
//!
//! # Using
//!
//! macOS has no `uidrange` policy routing, and the agent is root, so the
//! Linux scheme does not carry over. Instead:
//!
//! ```text
//! route add -net 0.0.0.0/1 -interface <if>
//! route add -net 128.0.0.0/1 -interface <if>
//! pass out quick on <if> route-to (<egress> <gateway>) inet proto { tcp udp }
//! from any to ! <overlay ranges> user <agent uid>
//! nat on <egress> inet from (<if>) to any -> (<egress>)
//! ```
//!
//! The two halves outrank the default route without replacing it, and every
//! more specific route (LAN, another VPN) keeps working. The `route-to` rule
//! sends the *agent's own* traffic — QUIC, relay, DHT, which would otherwise
//! loop into the tunnel it carries — back out the physical interface, and the
//! translation gives it that interface's address, since an unbound socket
//! picked the overlay address when the route pointed at the overlay. Like the
//! Linux exemption this is by user id, so anything else running as root also
//! bypasses the tunnel.
//!
//! IPv6 is blocked while an exit node is in use, as on Linux: `::/1` and
//! `8000::/1` are rejecting routes, so applications fall back to IPv4 at once
//! instead of leaking.
//!
//! Every object is derived from local state and the interface the agent owns;
//! none of it comes from a peer.
use std::net::Ipv4Addr;
use std::sync::Mutex;
use crate::state::Ipv4Range;
use super::RuleOutcome;
use super::exit::{ExitHostPlan, ExitHostReport};
/// The programs this module runs, always by absolute path.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(super) enum Tool {
Route,
Pfctl,
Sysctl,
Id,
}
impl Tool {
fn path(self) -> &'static str {
match self {
Self::Route => "/sbin/route",
Self::Pfctl => "/sbin/pfctl",
Self::Sysctl => "/usr/sbin/sysctl",
Self::Id => "/usr/bin/id",
}
}
}
/// What running a tool came to.
#[derive(Debug, Clone, PartialEq, Eq)]
pub(super) enum Ran {
/// The program is not there.
Missing,
/// It ran.
Exited {
success: bool,
stdout: String,
stderr: String,
},
}
/// Runs a tool with optional standard input; injectable so the logic is
/// tested without touching the host.
pub(super) type Run<'a> = &'a dyn Fn(Tool, &[String], Option<&str>) -> Ran;
/// Runs a real tool. Blocking: callers use `spawn_blocking`.
pub(super) fn run_tool(tool: Tool, args: &[String], stdin: Option<&str>) -> Ran {
use std::io::Write;
use std::process::{Command, Stdio};
let mut command = Command::new(tool.path());
command
.args(args)
.stdin(if stdin.is_some() {
Stdio::piped()
} else {
Stdio::null()
})
.stdout(Stdio::piped())
.stderr(Stdio::piped());
let mut child = match command.spawn() {
Ok(child) => child,
Err(err) if err.kind() == std::io::ErrorKind::NotFound => return Ran::Missing,
Err(err) => {
return Ran::Exited {
success: false,
stdout: String::new(),
stderr: format!("could not run {}: {err}", tool.path()),
};
}
};
if let (Some(text), Some(mut pipe)) = (stdin, child.stdin.take()) {
// A write error shows up as the tool's own failure below.
let _ = pipe.write_all(text.as_bytes());
}
match child.wait_with_output() {
Ok(output) => Ran::Exited {
success: output.status.success(),
stdout: String::from_utf8_lossy(&output.stdout).into_owned(),
stderr: String::from_utf8_lossy(&output.stderr).into_owned(),
},
Err(err) => Ran::Exited {
success: false,
stdout: String::new(),
stderr: format!("could not run {}: {err}", tool.path()),
},
}
}
fn owned(words: &[&str]) -> Vec<String> {
words.iter().map(|word| word.to_string()).collect()
}
/// A name that is safe in a `pf` rule, an anchor path and a table name.
fn safe_name(name: &str) -> bool {
!name.is_empty() && name.len() <= 32 && name.chars().all(|c| c.is_ascii_alphanumeric())
}
/// The anchor holding every rule of one interface.
fn anchor_name(interface: &str) -> Result<String, String> {
if safe_name(interface) {
Ok(format!("com.apple/tsunagi-exit-{interface}"))
} else {
Err(format!(
"`{interface}` is not an interface name this agent can write a pf rule for"
))
}
}
/// The default route as `route -n get default` reports it.
#[derive(Debug, Clone, PartialEq, Eq)]
pub(super) struct DefaultRoute {
pub interface: String,
pub gateway: Option<Ipv4Addr>,
}
/// Reads `interface:` and `gateway:` out of `route -n get default`.
fn parse_default_route(text: &str) -> Option<DefaultRoute> {
let field = |name: &str| {
text.lines().find_map(|line| {
let (key, value) = line.split_once(':')?;
(key.trim() == name).then(|| value.trim().to_string())
})
};
let interface = field("interface").filter(|name| safe_name(name))?;
let gateway = field("gateway").and_then(|value| value.parse().ok());
Some(DefaultRoute { interface, gateway })
}
fn failure(stderr: &str, stdout: &str) -> String {
let text = if stderr.trim().is_empty() {
stdout.trim()
} else {
stderr.trim()
};
if text.is_empty() {
"the command failed without saying why".to_string()
} else {
text.to_string()
}
}
/// Runs a tool and turns a failure into a sentence a person can act on.
fn run_ok(
run: Run<'_>,
tool: Tool,
args: &[String],
stdin: Option<&str>,
) -> Result<String, String> {
match run(tool, args, stdin) {
Ran::Missing => Err(format!("{} not found", tool.path())),
Ran::Exited {
success: true,
stdout,
..
} => Ok(stdout),
Ran::Exited { stderr, stdout, .. } => Err(format!(
"{} {} failed: {}",
tool.path(),
args.join(" "),
failure(&stderr, &stdout)
)),
}
}
/// The route traffic leaves by now, which is the egress for translation and
/// for the agent's own sockets. Read *before* the overlay routes are added.
fn default_route(run: Run<'_>, overlay: &str) -> Result<DefaultRoute, String> {
let out = run_ok(run, Tool::Route, &owned(&["-n", "get", "default"]), None)
.map_err(|why| format!("cannot find the default route: {why}"))?;
let route = parse_default_route(&out)
.ok_or_else(|| "the host has no default route to send traffic out of".to_string())?;
if route.interface == overlay {
return Err(format!(
"the default route already goes through `{overlay}`"
));
}
Ok(route)
}
/// Whether the kernel forwards IPv4. Read only.
fn forwarding_state(run: Run<'_>) -> Option<bool> {
let out = run_ok(
run,
Tool::Sysctl,
&owned(&["-n", "net.inet.ip.forwarding"]),
None,
)
.ok()?;
match out.trim() {
"1" => Some(true),
"0" => Some(false),
_ => None,
}
}
/// The effective user id of this process, from `id -u`.
fn agent_uid(run: Run<'_>) -> Result<u32, String> {
run_ok(run, Tool::Id, &owned(&["-u"]), None)
.ok()
.and_then(|out| out.trim().parse().ok())
.ok_or_else(|| "cannot tell which user this agent runs as".to_string())
}
/// What the using side adds to the rules.
#[derive(Debug, Clone, PartialEq, Eq)]
struct ClientRules<'a> {
uid: u32,
/// Every overlay range: the agent's own traffic *to* those still goes
/// through the interface.
overlay: &'a [Ipv4Range],
}
/// The text of the anchor: translation first, filtering after, which is the
/// order `pf` insists on.
fn pf_rules(
interface: &str,
egress: &DefaultRoute,
offer: &[Ipv4Range],
client: Option<&ClientRules<'_>>,
) -> String {
let out = &egress.interface;
let mut nat = Vec::new();
let mut filter = Vec::new();
for range in offer {
nat.push(format!("nat on {out} inet from {range} to any -> ({out})"));
filter.push(format!(
"pass in quick on {interface} inet from {range} to ! {range}"
));
}
let mut tables = Vec::new();
if let Some(client) = client {
nat.push(format!(
"nat on {out} inet from ({interface}) to any -> ({out})"
));
let target = if client.overlay.is_empty() {
"any".to_string()
} else {
let ranges: Vec<String> = client.overlay.iter().map(|r| r.to_string()).collect();
// A table, because `! { a, b }` expands to one rule per element
// in older `pf`, which matches everything.
tables.push(format!(
"table <tsunagi_overlay_{interface}> const {{ {} }}",
ranges.join(", ")
));
format!("! <tsunagi_overlay_{interface}>")
};
let via = match egress.gateway {
Some(gateway) => format!("({out} {gateway})"),
None => out.clone(),
};
filter.push(format!(
"pass out quick on {interface} route-to {via} inet proto {{ tcp udp }} \
from any to {target} user {}",
client.uid
));
}
tables
.into_iter()
.chain(nat)
.chain(filter)
.collect::<Vec<_>>()
.join("\n")
+ "\n"
}
/// Turns `pf` on if nobody has, holding the reference `pfctl` hands out so it
/// can be released when the agent is done. One reference at most.
fn acquire_pf(run: Run<'_>, slot: &Mutex<Option<String>>) -> Result<(), String> {
if lock(slot).is_some() {
return Ok(());
}
match run(Tool::Pfctl, &owned(&["-E"]), None) {
Ran::Missing => Err(format!("{} not found", Tool::Pfctl.path())),
Ran::Exited {
success,
stdout,
stderr,
} => {
// `pfctl -E` reports `Token : <n>` on stderr, and also says so
// when pf is already enabled.
let token = stdout.lines().chain(stderr.lines()).find_map(parse_token);
if !success && token.is_none() {
return Err(format!("cannot enable pf: {}", failure(&stderr, &stdout)));
}
*lock(slot) = token;
Ok(())
}
}
}
fn parse_token(line: &str) -> Option<String> {
let (key, value) = line.split_once(':')?;
let value = value.trim();
(key.trim().eq_ignore_ascii_case("token")
&& !value.is_empty()
&& value.chars().all(|c| c.is_ascii_digit()))
.then(|| value.to_string())
}
fn release_pf(run: Run<'_>, slot: &Mutex<Option<String>>) {
if let Some(token) = lock(slot).take()
&& let Err(why) = run_ok(run, Tool::Pfctl, &["-X".to_string(), token], None)
{
tracing::debug!(%why, "cannot release the pf reference");
}
}
fn lock<T>(mutex: &Mutex<T>) -> std::sync::MutexGuard<'_, T> {
match mutex.lock() {
Ok(guard) => guard,
Err(poisoned) => poisoned.into_inner(),
}
}
fn flush_args(anchor: &str) -> Vec<String> {
owned(&["-q", "-a", anchor, "-F", "all"])
}
fn load_args(anchor: &str) -> Vec<String> {
owned(&["-q", "-a", anchor, "-f", "-"])
}
/// The two halves that out-rank the default route.
const HALVES: [&str; 2] = ["0.0.0.0/1", "128.0.0.0/1"];
/// The IPv6 halves, made unreachable.
const HALVES_V6: [&str; 2] = ["::/1", "8000::/1"];
fn route_add_args(half: &str, interface: &str) -> Vec<String> {
owned(&["-q", "-n", "add", "-net", half, "-interface", interface])
}
fn route_del_args(half: &str, interface: &str) -> Vec<String> {
owned(&["-q", "-n", "delete", "-net", half, "-interface", interface])
}
fn route_add_v6_args(half: &str) -> Vec<String> {
owned(&["-q", "-n", "add", "-inet6", "-net", half, "::1", "-reject"])
}
fn route_del_v6_args(half: &str) -> Vec<String> {
owned(&["-q", "-n", "delete", "-inet6", "-net", half, "::1"])
}
/// Takes away the overlay default routes. Best effort: what is not there is
/// the state wanted.
fn clear_routes(run: Run<'_>, interface: &str) {
for half in HALVES {
let _ = run(Tool::Route, &route_del_args(half, interface), None);
}
for half in HALVES_V6 {
let _ = run(Tool::Route, &route_del_v6_args(half), None);
}
}
fn set_routes(run: Run<'_>, interface: &str) -> Result<(), String> {
clear_routes(run, interface);
for half in HALVES {
if let Err(why) = run_ok(run, Tool::Route, &route_add_args(half, interface), None) {
clear_routes(run, interface);
return Err(why);
}
}
for half in HALVES_V6 {
// A host without IPv6 has nothing to block; not a failure.
if let Err(why) = run_ok(run, Tool::Route, &route_add_v6_args(half), None) {
tracing::debug!(%why, "cannot block IPv6 for the exit node");
}
}
Ok(())
}
/// Makes the host match the plan.
pub(super) fn apply_plan(
run: Run<'_>,
token: &Mutex<Option<String>>,
plan: &ExitHostPlan,
) -> ExitHostReport {
let forwarding = if plan.offer.is_empty() {
None
} else {
forwarding_state(run)
};
let outcome = apply_rules(run, token, plan);
let verdict = |ok: bool, why: &str| {
if ok {
RuleOutcome::Applied
} else {
RuleOutcome::Failed(why.to_string())
}
};
match outcome {
Ok(()) => ExitHostReport {
offer: plan
.offer
.iter()
.map(|range| (*range, RuleOutcome::Applied))
.collect(),
forwarding,
client: plan.client.then_some(RuleOutcome::Applied),
},
Err(why) => ExitHostReport {
offer: plan
.offer
.iter()
.map(|range| (*range, verdict(false, &why)))
.collect(),
forwarding,
client: plan.client.then(|| verdict(false, &why)),
},
}
}
fn apply_rules(
run: Run<'_>,
token: &Mutex<Option<String>>,
plan: &ExitHostPlan,
) -> Result<(), String> {
let interface = plan.interface.as_str();
let anchor = anchor_name(interface)?;
let egress = default_route(run, interface)?;
let uid = if plan.client {
Some(agent_uid(run)?)
} else {
None
};
let client = uid.map(|uid| ClientRules {
uid,
overlay: &plan.overlay,
});
let rules = pf_rules(interface, &egress, &plan.offer, client.as_ref());
acquire_pf(run, token)?;
run_ok(run, Tool::Pfctl, &load_args(&anchor), Some(&rules))
.map_err(|why| format!("cannot load the pf rules: {why}"))?;
// The routes come after the rules that keep the agent's own traffic off
// them: the other order would loop for as long as it takes.
if plan.client {
set_routes(run, interface)?;
} else {
clear_routes(run, interface);
}
Ok(())
}
/// Removes everything this agent put there for the interface.
pub(super) fn clear_all(run: Run<'_>, token: &Mutex<Option<String>>, interface: &str) {
clear_routes(run, interface);
if let Ok(anchor) = anchor_name(interface) {
// Flushing an anchor nothing was ever loaded into is not an error
// worth reporting.
if let Err(why) = run_ok(run, Tool::Pfctl, &flush_args(&anchor), None) {
tracing::debug!(%why, "cannot flush the exit node pf anchor");
}
}
release_pf(run, token);
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use std::cell::RefCell;
use super::*;
fn range() -> Ipv4Range {
"10.13.37.0/24".parse().unwrap()
}
const DEFAULT_ROUTE: &str = " route to: default\n\
destination: default\n\
mask: default\n\
gateway: 192.168.1.1\n\
interface: en0\n\
flags: <UP,GATEWAY,DONE,STATIC,PRCLONING,GLOBAL>\n";
fn egress() -> DefaultRoute {
DefaultRoute {
interface: "en0".into(),
gateway: Some(Ipv4Addr::new(192, 168, 1, 1)),
}
}
/// A scripted host that records every call.
struct Fake {
calls: RefCell<Vec<String>>,
stdins: RefCell<Vec<String>>,
default_route: Option<String>,
forwarding: &'static str,
pf_missing: bool,
pf_load_fails: bool,
route_add_fails: bool,
pf_enable_output: &'static str,
}
impl Fake {
fn new() -> Self {
Self {
calls: RefCell::new(Vec::new()),
stdins: RefCell::new(Vec::new()),
default_route: Some(DEFAULT_ROUTE.to_string()),
forwarding: "1\n",
pf_missing: false,
pf_load_fails: false,
route_add_fails: false,
pf_enable_output: "pf enabled\nToken : 4242\n",
}
}
fn run(&self, tool: Tool, args: &[String], stdin: Option<&str>) -> Ran {
let line = format!("{:?} {}", tool, args.join(" "));
self.calls.borrow_mut().push(line);
if let Some(text) = stdin {
self.stdins.borrow_mut().push(text.to_string());
}
let ok = |stdout: &str, stderr: &str| Ran::Exited {
success: true,
stdout: stdout.to_string(),
stderr: stderr.to_string(),
};
let fail = |why: &str| Ran::Exited {
success: false,
stdout: String::new(),
stderr: why.to_string(),
};
match tool {
Tool::Id => ok("0\n", ""),
Tool::Sysctl => ok(self.forwarding, ""),
Tool::Route if args.first().map(String::as_str) == Some("-n") => {
match &self.default_route {
Some(text) => ok(text, ""),
None => fail("route: writing to routing socket: not in table"),
}
}
Tool::Route if args.contains(&"add".to_string()) && self.route_add_fails => {
fail("route: writing to routing socket: File exists")
}
Tool::Route => ok("", ""),
Tool::Pfctl if self.pf_missing => Ran::Missing,
Tool::Pfctl if args == ["-E"] => ok("", self.pf_enable_output),
Tool::Pfctl if args.contains(&"-f".to_string()) && self.pf_load_fails => {
fail("pfctl: Syntax error")
}
Tool::Pfctl => ok("", ""),
}
}
fn calls(&self) -> Vec<String> {
self.calls.borrow().clone()
}
fn position(&self, needle: &str) -> Option<usize> {
self.calls.borrow().iter().position(|c| c.contains(needle))
}
fn count(&self, needle: &str) -> usize {
self.calls
.borrow()
.iter()
.filter(|c| c.contains(needle))
.count()
}
}
fn plan(offer: bool, client: bool) -> ExitHostPlan {
ExitHostPlan {
interface: "utun5".into(),
offer: if offer { vec![range()] } else { Vec::new() },
client,
overlay: vec![range()],
..Default::default()
}
}
#[test]
fn the_default_route_is_read_from_route_output() {
assert_eq!(parse_default_route(DEFAULT_ROUTE), Some(egress()));
// A point-to-point default has no gateway line to use.
let ppp = "destination: default\n interface: ppp0\n";
assert_eq!(
parse_default_route(ppp),
Some(DefaultRoute {
interface: "ppp0".into(),
gateway: None
})
);
assert_eq!(parse_default_route("route: not in table"), None);
// An interface name that is not plain letters and digits never ends
// up in a rule.
assert_eq!(parse_default_route("interface: en0; pass all\n"), None);
}
#[test]
fn the_anchor_is_under_com_apple_and_only_for_plain_names() {
assert_eq!(
anchor_name("utun5").unwrap(),
"com.apple/tsunagi-exit-utun5"
);
assert!(anchor_name("").is_err());
assert!(anchor_name("utun5 pass all").is_err());
assert!(anchor_name("../x").is_err());
}
#[test]
fn offering_translates_the_range_and_admits_it_in() {
let text = pf_rules("utun5", &egress(), &[range()], None);
assert_eq!(
text,
"nat on en0 inet from 10.13.37.0/24 to any -> (en0)\n\
pass in quick on utun5 inet from 10.13.37.0/24 to ! 10.13.37.0/24\n"
);
// Translation precedes filtering, as pf requires.
assert!(text.find("nat on").unwrap() < text.find("pass in").unwrap());
}
#[test]
fn using_sends_the_agents_own_traffic_back_out_the_physical_interface() {
let other: Ipv4Range = "10.99.0.0/16".parse().unwrap();
let overlay = [range(), other];
let client = ClientRules {
uid: 0,
overlay: &overlay,
};
let text = pf_rules("utun5", &egress(), &[], Some(&client));
assert!(
text.starts_with(
"table <tsunagi_overlay_utun5> const { 10.13.37.0/24, 10.99.0.0/16 }\n"
),
"{text}"
);
assert!(
text.contains("nat on en0 inet from (utun5) to any -> (en0)\n"),
"{text}"
);
assert!(
text.contains(
"pass out quick on utun5 route-to (en0 192.168.1.1) inet proto { tcp udp } \
from any to ! <tsunagi_overlay_utun5> user 0\n"
),
"{text}"
);
// Only the agent's user, and only what is not the overlay.
assert!(!text.contains("from any to any"));
let direct = DefaultRoute {
interface: "ppp0".into(),
gateway: None,
};
let text = pf_rules(
"utun5",
&direct,
&[],
Some(&ClientRules {
uid: 501,
overlay: &[],
}),
);
assert!(text.contains("route-to ppp0 inet"), "{text}");
assert!(text.contains("to any user 501"), "{text}");
assert!(!text.contains("table"), "{text}");
}
#[test]
fn the_overlay_default_routes_are_halves_and_ipv6_is_rejected() {
assert_eq!(
route_add_args("0.0.0.0/1", "utun5").join(" "),
"-q -n add -net 0.0.0.0/1 -interface utun5"
);
assert_eq!(
route_del_args("128.0.0.0/1", "utun5").join(" "),
"-q -n delete -net 128.0.0.0/1 -interface utun5"
);
assert_eq!(
route_add_v6_args("8000::/1").join(" "),
"-q -n add -inet6 -net 8000::/1 ::1 -reject"
);
assert_eq!(
route_del_v6_args("::/1").join(" "),
"-q -n delete -inet6 -net ::/1 ::1"
);
}
#[test]
fn the_pf_token_is_found_in_either_stream() {
assert_eq!(parse_token("Token : 1234"), Some("1234".into()));
assert_eq!(parse_token("token: 9"), Some("9".into()));
assert_eq!(parse_token("pf enabled"), None);
assert_eq!(parse_token("Token : x; rm"), None);
}
#[test]
fn offering_loads_the_anchor_and_reads_but_never_writes_forwarding() {
let fake = Fake::new();
let token = Mutex::new(None);
let report = apply_plan(&|t, a, s| fake.run(t, a, s), &token, &plan(true, false));
assert_eq!(report.offer, vec![(range(), RuleOutcome::Applied)]);
assert_eq!(report.forwarding, Some(true));
assert_eq!(report.client, None);
let calls = fake.calls();
assert!(
calls
.iter()
.any(|c| c == "Pfctl -q -a com.apple/tsunagi-exit-utun5 -f -"),
"{calls:?}"
);
assert_eq!(
fake.stdins.borrow()[0],
"nat on en0 inet from 10.13.37.0/24 to any -> (en0)\n\
pass in quick on utun5 inet from 10.13.37.0/24 to ! 10.13.37.0/24\n"
);
// sysctl is only ever asked, never told.
assert!(
calls
.iter()
.filter(|c| c.starts_with("Sysctl"))
.all(|c| c == "Sysctl -n net.inet.ip.forwarding"),
"{calls:?}"
);
// No client routes were added for an offer alone.
assert_eq!(fake.count(" add "), 0);
assert_eq!(token.lock().unwrap().as_deref(), Some("4242"));
}
#[test]
fn forwarding_off_is_reported_and_unreadable_is_none() {
let mut fake = Fake::new();
fake.forwarding = "0\n";
let token = Mutex::new(None);
let report = apply_plan(&|t, a, s| fake.run(t, a, s), &token, &plan(true, false));
assert_eq!(report.forwarding, Some(false));
fake.forwarding = "junk";
let report = apply_plan(&|t, a, s| fake.run(t, a, s), &token, &plan(true, false));
assert_eq!(report.forwarding, None);
}
#[test]
fn the_client_rules_are_loaded_before_the_routes_that_need_them() {
let fake = Fake::new();
let token = Mutex::new(None);
let report = apply_plan(&|t, a, s| fake.run(t, a, s), &token, &plan(false, true));
assert_eq!(report.client, Some(RuleOutcome::Applied));
assert!(report.offer.is_empty());
let load = fake.position("-f -").unwrap();
let add = fake.position(" add -net 0.0.0.0/1").unwrap();
assert!(load < add, "{:?}", fake.calls());
assert!(fake.position("add -net 128.0.0.0/1").is_some());
assert!(fake.position("-inet6 -net ::/1 ::1 -reject").is_some());
assert!(fake.stdins.borrow()[0].contains("user 0"));
// The default route was read before any overlay route existed.
assert!(fake.position("-n get default").unwrap() < add);
}
#[test]
fn no_routes_are_added_when_the_rules_did_not_load() {
let mut fake = Fake::new();
fake.pf_load_fails = true;
let token = Mutex::new(None);
let report = apply_plan(&|t, a, s| fake.run(t, a, s), &token, &plan(true, true));
let Some(RuleOutcome::Failed(why)) = report.client else {
panic!("{report:?}");
};
assert!(why.contains("cannot load the pf rules"), "{why}");
assert!(why.contains("Syntax error"), "{why}");
assert!(matches!(report.offer[0].1, RuleOutcome::Failed(_)));
// Nothing sent the host's traffic into a tunnel it could not exempt
// the agent from.
assert_eq!(fake.count(" add "), 0);
}
#[test]
fn a_failing_route_is_a_failure_and_leaves_nothing_behind() {
let mut fake = Fake::new();
fake.route_add_fails = true;
let token = Mutex::new(None);
let report = apply_plan(&|t, a, s| fake.run(t, a, s), &token, &plan(false, true));
let Some(RuleOutcome::Failed(why)) = report.client else {
panic!("{report:?}");
};
assert!(why.contains("File exists"), "{why}");
// The cleanup after the failed add removed whatever did land.
let last_add = fake
.calls()
.iter()
.rposition(|c| c.contains(" add "))
.unwrap();
assert!(
fake.calls()[last_add..]
.iter()
.any(|c| c.contains("delete"))
);
}
#[test]
fn a_missing_pfctl_or_default_route_says_what_is_wrong() {
let mut fake = Fake::new();
fake.pf_missing = true;
let token = Mutex::new(None);
let report = apply_plan(&|t, a, s| fake.run(t, a, s), &token, &plan(true, false));
let RuleOutcome::Failed(why) = &report.offer[0].1 else {
panic!("{report:?}");
};
assert!(why.contains("/sbin/pfctl not found"), "{why}");
let mut fake = Fake::new();
fake.default_route = None;
let report = apply_plan(&|t, a, s| fake.run(t, a, s), &token, &plan(true, false));
let RuleOutcome::Failed(why) = &report.offer[0].1 else {
panic!("{report:?}");
};
assert!(why.contains("default route"), "{why}");
assert_eq!(fake.count("-f -"), 0);
}
#[test]
fn a_default_route_through_the_overlay_is_refused() {
let mut fake = Fake::new();
fake.default_route = Some("gateway: 10.13.37.1\ninterface: utun5\n".into());
let token = Mutex::new(None);
let report = apply_plan(&|t, a, s| fake.run(t, a, s), &token, &plan(true, false));
let RuleOutcome::Failed(why) = &report.offer[0].1 else {
panic!("{report:?}");
};
assert!(why.contains("already goes through"), "{why}");
}
#[test]
fn pf_is_enabled_once_and_released_once_on_clear() {
let fake = Fake::new();
let token = Mutex::new(None);
let run = |t, a: &[String], s: Option<&str>| fake.run(t, a, s);
apply_plan(&run, &token, &plan(true, false));
apply_plan(&run, &token, &plan(true, true));
assert_eq!(fake.count("Pfctl -E"), 1, "{:?}", fake.calls());
clear_all(&run, &token, "utun5");
assert_eq!(fake.count("Pfctl -X 4242"), 1);
assert!(
fake.position("-a com.apple/tsunagi-exit-utun5 -F all")
.is_some()
);
assert!(
fake.position("delete -net 0.0.0.0/1 -interface utun5")
.is_some()
);
assert_eq!(*token.lock().unwrap(), None);
// A second clear has nothing to release and still succeeds.
clear_all(&run, &token, "utun5");
assert_eq!(fake.count("Pfctl -X"), 1);
}
#[test]
fn dropping_the_client_removes_its_routes() {
let fake = Fake::new();
let token = Mutex::new(None);
let run = |t, a: &[String], s: Option<&str>| fake.run(t, a, s);
apply_plan(&run, &token, &plan(true, false));
assert!(fake.position("delete -net 0.0.0.0/1").is_some());
assert_eq!(fake.count(" add "), 0);
}
}
crates/tsunagi/src/overlay/hostrules/mod.rs → crates/tsunagi-core/src/overlay/hostrules/mod.rs
+40 -1
View File
@@ -45,19 +45,36 @@ use std::sync::{Arc, Mutex};
use crate::BoxFuture;
use crate::state::Ipv4Range;
mod exit;
pub use exit::{
ExitHostPlan, ExitHostReport, ExitHostRules, MockExitRules, exit_tag, exit_tag_prefix,
offer_detail,
};
#[cfg(all(feature = "tun-device", target_os = "linux"))]
mod linux;
#[cfg(all(feature = "tun-device", target_os = "linux"))]
mod linux_exit;
#[cfg(all(test, feature = "tun-device", target_os = "linux"))]
mod linux_exit_kernel;
#[cfg(all(feature = "tun-device", target_os = "linux"))]
pub(in crate::overlay) mod linux_exit_mark;
#[cfg(all(feature = "tun-device", target_os = "linux"))]
pub use linux::LinuxHostRules;
// Compiled everywhere so the scripts it builds are tested everywhere; only
// the type that runs them is exported on Windows.
mod windows;
#[cfg_attr(not(target_os = "windows"), allow(dead_code))]
mod windows_exit;
#[cfg(all(feature = "tun-device", target_os = "windows"))]
pub use windows::WindowsHostRules;
// Compiled everywhere so its rule text and parsers are tested everywhere.
#[cfg(all(feature = "tun-device", target_os = "macos"))]
mod macos;
#[cfg_attr(not(target_os = "macos"), allow(dead_code))]
mod macos_exit;
#[cfg(all(feature = "tun-device", target_os = "macos"))]
pub use macos::MacosHostRules;
@@ -159,6 +176,15 @@ pub trait BroadcastHostRules: Send + Sync + std::fmt::Debug + 'static {
/// Removes what this agent put there for the interface. Succeeds when
/// nothing is there: it runs on the shutdown path.
fn clear<'a>(&'a self, interface: &'a str) -> BoxFuture<'a, ()>;
/// The exit-node rules that go with these, when the platform has them.
///
/// One object per host, because both halves need the same privileged
/// helpers; the default is that this platform has none, which is said
/// where it matters rather than pretended.
fn exit_rules(&self) -> Option<&dyn ExitHostRules> {
None
}
}
/// The platform has no implementation, which is said rather than pretended.
@@ -195,6 +221,7 @@ pub type MockRulesState = std::collections::BTreeMap<String, BroadcastRulesPlan>
/// An in-memory host, for tests: nothing is touched.
#[derive(Debug, Clone, Default)]
pub struct MockHostRules {
exit: MockExitRules,
state: Arc<Mutex<MockRulesState>>,
calls: Arc<Mutex<Vec<String>>>,
firewall_failure: Arc<Mutex<Option<String>>>,
@@ -203,7 +230,15 @@ pub struct MockHostRules {
impl MockHostRules {
/// An empty host.
pub fn new() -> Self {
Self::default()
Self {
exit: MockExitRules::new(),
..Self::default()
}
}
/// The exit-node half of this host, for inspecting and steering tests.
pub fn exit(&self) -> &MockExitRules {
&self.exit
}
/// Makes the firewall step fail with this reason from now on.
@@ -234,6 +269,10 @@ impl BroadcastHostRules for MockHostRules {
"mock"
}
fn exit_rules(&self) -> Option<&dyn ExitHostRules> {
Some(&self.exit)
}
fn apply<'a>(&'a self, plan: &'a BroadcastRulesPlan) -> BoxFuture<'a, BroadcastRulesReport> {
Box::pin(async move {
lock(&self.calls).push(format!("apply:{}", plan.interface));
crates/tsunagi/src/overlay/hostrules/windows.rs → crates/tsunagi-core/src/overlay/hostrules/windows.rs
+71 -2
View File
@@ -29,6 +29,8 @@
use crate::BoxFuture;
use super::exit::{ExitHostPlan, ExitHostReport, ExitHostRules};
use super::windows_exit;
use super::{BroadcastHostRules, BroadcastRulesPlan, BroadcastRulesReport, RuleOutcome, tag_for};
/// A PowerShell single-quoted literal, with any embedded quote doubled.
@@ -90,7 +92,7 @@ fn classify(what: &str, text: &str) -> String {
|| lower.contains("run as administrator")
|| lower.contains("permissiondenied")
{
format!("{what}: Windows refused; run tsunagi from an elevated (administrator) process")
format!("{what}: Windows refused; run tsng from an elevated (administrator) process")
} else if lower.contains("is not recognized") || lower.contains("commandnotfound") {
format!("{what}: the NetTCPIP/NetSecurity PowerShell modules are not available: {text}")
} else if text.is_empty() {
@@ -113,6 +115,11 @@ fn powershell_path() -> std::path::PathBuf {
/// Runs a script and returns what it said on failure.
fn run_powershell(script: &str) -> Result<(), String> {
run_powershell_output(script).map(|_| ())
}
/// Runs a script and returns what it printed, or what it said on failure.
fn run_powershell_output(script: &str) -> Result<String, String> {
let output = std::process::Command::new(powershell_path())
.args([
"-NoProfile",
@@ -125,7 +132,7 @@ fn run_powershell(script: &str) -> Result<(), String> {
.output()
.map_err(|err| format!("could not run PowerShell: {err}"))?;
if output.status.success() {
return Ok(());
return Ok(String::from_utf8_lossy(&output.stdout).into_owned());
}
let stderr = String::from_utf8_lossy(&output.stderr);
let text = if stderr.trim().is_empty() {
@@ -155,11 +162,73 @@ impl WindowsHostRules {
}
}
/// The IPv4 addresses a host name resolves to; empty when it does not.
fn resolve_host(host: &str) -> Vec<std::net::Ipv4Addr> {
use std::net::{IpAddr, ToSocketAddrs};
(host, 443)
.to_socket_addrs()
.map(|addrs| {
addrs
.filter_map(|addr| match addr.ip() {
IpAddr::V4(ip) => Some(ip),
IpAddr::V6(_) => None,
})
.collect()
})
.unwrap_or_default()
}
impl ExitHostRules for WindowsHostRules {
fn needs_bypass(&self) -> bool {
true
}
fn apply<'a>(&'a self, plan: &'a ExitHostPlan) -> BoxFuture<'a, ExitHostReport> {
Box::pin(async move {
let job = plan.clone();
tokio::task::spawn_blocking(move || {
windows_exit::apply_plan(
&|script| run_powershell_output(script),
&resolve_host,
&job,
)
})
.await
.unwrap_or_else(|err| {
let failed = RuleOutcome::Failed(format!("the task failed: {err}"));
ExitHostReport {
offer: plan
.offer
.iter()
.map(|range| (*range, failed.clone()))
.collect(),
forwarding: None,
client: plan.client.then_some(failed),
}
})
})
}
fn clear<'a>(&'a self, interface: &'a str) -> BoxFuture<'a, ()> {
Box::pin(async move {
let name = interface.to_string();
let _ = tokio::task::spawn_blocking(move || {
windows_exit::clear_all(&|script| run_powershell_output(script), &name)
})
.await;
})
}
}
impl BroadcastHostRules for WindowsHostRules {
fn name(&self) -> &str {
"windows-netsecurity"
}
fn exit_rules(&self) -> Option<&dyn ExitHostRules> {
Some(self)
}
fn apply<'a>(&'a self, plan: &'a BroadcastRulesPlan) -> BoxFuture<'a, BroadcastRulesReport> {
Box::pin(async move {
let route = step("route", route_script(plan)).await;
crates/tsunagi-core/src/overlay/hostrules/windows_exit.rs
+603
View File
@@ -0,0 +1,603 @@
//! Exit-node host rules on Windows: WinNAT through PowerShell.
//!
//! Compiled on every platform so the scripts and the plan logic are tested
//! everywhere; only `WindowsHostRules` runs them, and only on Windows. It
//! could not be run against a real Windows host while it was written.
//!
//! # Offering
//!
//! One NAT object per offered range, named with the usual tag
//! `tsunagi-exit:<if>:<range>`:
//!
//! ```text
//! New-NetNat -Name <tag> -InternalIPInterfaceAddressPrefix <range>
//! ```
//!
//! WinNAT translates that range out of whichever interface the traffic leaves
//! by, which is the Windows counterpart of the Linux masquerade. Windows
//! forwards per interface, and the egress is not known here, so forwarding is
//! only *read* for the overlay interface; turning it on for the interfaces
//! involved is the owner's decision (`Set-NetIPInterface -Forwarding Enabled`).
//! Some Windows versions allow a single NAT per host: a second offered range
//! then reports its own failure.
//!
//! # Using
//!
//! Windows has no per-user or per-process routing, so the Linux scheme (an
//! exemption for the agent's user) does not carry over. Instead the agent's
//! own underlay traffic is exempted by *destination*:
//!
//! ```text
//! New-NetRoute 0.0.0.0/1 on the overlay interface, metric 1
//! New-NetRoute 128.0.0.0/1 on the overlay interface, metric 1
//! New-NetRoute <ip>/32 via the physical default gateway, metric 7373
//! ```
//!
//! The halves out-rank the default route without replacing it, so every more
//! specific route (LAN, another VPN) keeps working. A `/32` per address the
//! agent talks to directly (the verified direct paths of its connections, and
//! the relays it uses, resolved when the rules are applied) keeps that traffic
//! on the physical network instead of looping into the tunnel it carries. The
//! metric 7373 marks those routes as ours, which is how stale ones are found
//! and removed. Everything is written to the active store, so a reboot clears
//! it. IPv6 is blackholed with `::/1` and `8000::/1` on the overlay interface,
//! which has no IPv6: applications fall back to IPv4 after a delay, where
//! Linux and macOS answer at once. The bypass follows the agent's paths as
//! they change; a path that appears after the last apply is reached through
//! the tunnel until the next one.
//!
//! The bypass routes are written before the halves, and removed after them.
use super::RuleOutcome;
use std::net::Ipv4Addr;
use super::exit::{ExitHostPlan, ExitHostReport, exit_tag, exit_tag_prefix};
use crate::state::Ipv4Range;
/// The route metric that marks a bypass route as this agent's.
const BYPASS_METRIC: u32 = 7373;
/// Runs a PowerShell script and returns what it printed.
pub(super) type RunScript<'a> = &'a dyn Fn(&str) -> Result<String, String>;
/// Resolves a host name to its IPv4 addresses.
pub(super) type Resolve<'a> = &'a dyn Fn(&str) -> Vec<Ipv4Addr>;
/// A PowerShell single-quoted literal, with any embedded quote doubled.
fn ps_literal(value: &str) -> String {
format!("'{}'", value.replace('\'', "''"))
}
/// Creates the NAT of one range, replacing one this agent made before.
fn nat_script(interface: &str, range: &Ipv4Range) -> String {
let name = ps_literal(&exit_tag(interface, range));
let prefix = ps_literal(&range.to_string());
format!(
"$ErrorActionPreference = 'Stop'\n\
$name = {name}\n\
Get-NetNat -Name $name -ErrorAction SilentlyContinue | \
Remove-NetNat -Confirm:$false -ErrorAction SilentlyContinue\n\
New-NetNat -Name $name -InternalIPInterfaceAddressPrefix {prefix} | Out-Null\n"
)
}
/// Removes this interface's NAT objects except those named in `keep`.
fn prune_script(interface: &str, keep: &[Ipv4Range]) -> String {
let prefix = ps_literal(&exit_tag_prefix(interface));
let keep: Vec<String> = keep
.iter()
.map(|range| ps_literal(&exit_tag(interface, range)))
.collect();
format!(
"$ErrorActionPreference = 'SilentlyContinue'\n\
$prefix = {prefix}\n\
$keep = @({})\n\
Get-NetNat | Where-Object {{ $_.Name.StartsWith($prefix) -and \
($keep -notcontains $_.Name) }} | Remove-NetNat -Confirm:$false\n\
exit 0\n",
keep.join(", ")
)
}
/// Prints `Enabled` or `Disabled` for the interface's IPv4 forwarding.
fn forwarding_script(interface: &str) -> String {
let alias = ps_literal(interface);
format!(
"$ErrorActionPreference = 'Stop'\n\
(Get-NetIPInterface -InterfaceAlias {alias} -AddressFamily IPv4).Forwarding\n"
)
}
/// Prints `<interface index> <next hop>` of the best default route that is
/// not through the overlay interface.
fn gateway_script(interface: &str) -> String {
let alias = ps_literal(interface);
format!(
"$ErrorActionPreference = 'Stop'\n\
$alias = {alias}\n\
$best = Get-NetRoute -DestinationPrefix '0.0.0.0/0' -AddressFamily IPv4 \
-PolicyStore ActiveStore | Where-Object {{ $_.InterfaceAlias -ne $alias -and \
$_.NextHop -ne '0.0.0.0' }} | Sort-Object {{ $_.RouteMetric + \
(Get-NetIPInterface -InterfaceIndex $_.InterfaceIndex -AddressFamily IPv4).InterfaceMetric }} | \
Select-Object -First 1\n\
if (-not $best) {{ throw 'the host has no default gateway outside the overlay' }}\n\
\"$($best.InterfaceIndex) $($best.NextHop)\"\n"
)
}
/// The default gateway as the script above printed it.
fn parse_gateway(text: &str) -> Option<(u32, Ipv4Addr)> {
let mut words = text.split_whitespace();
let index = words.next()?.parse().ok()?;
let hop = words.next()?.parse().ok()?;
words.next().is_none().then_some((index, hop))
}
/// Replaces the bypass routes with one per address, through the gateway.
fn bypass_script(index: u32, gateway: Ipv4Addr, addresses: &[Ipv4Addr]) -> String {
let list: Vec<String> = addresses
.iter()
.map(|address| ps_literal(&address.to_string()))
.collect();
format!(
"$ErrorActionPreference = 'Stop'\n\
Get-NetRoute -AddressFamily IPv4 -PolicyStore ActiveStore -ErrorAction SilentlyContinue | \
Where-Object {{ $_.RouteMetric -eq {BYPASS_METRIC} -and $_.DestinationPrefix.EndsWith('/32') }} | \
Remove-NetRoute -Confirm:$false -ErrorAction SilentlyContinue\n\
foreach ($ip in @({})) {{\n\
New-NetRoute -DestinationPrefix \"$ip/32\" -InterfaceIndex {index} -NextHop '{gateway}' \
-RouteMetric {BYPASS_METRIC} -PolicyStore ActiveStore -ErrorAction SilentlyContinue | Out-Null\n\
}}\n",
list.join(", ")
)
}
/// Sends everything else through the overlay interface.
fn halves_script(interface: &str) -> String {
let alias = ps_literal(interface);
format!(
"$ErrorActionPreference = 'Stop'\n\
$alias = {alias}\n\
foreach ($p in '0.0.0.0/1', '128.0.0.0/1') {{\n\
Get-NetRoute -DestinationPrefix $p -InterfaceAlias $alias -PolicyStore ActiveStore \
-ErrorAction SilentlyContinue | Remove-NetRoute -Confirm:$false -ErrorAction SilentlyContinue\n\
New-NetRoute -DestinationPrefix $p -InterfaceAlias $alias -NextHop '0.0.0.0' \
-RouteMetric 1 -PolicyStore ActiveStore | Out-Null\n\
}}\n\
foreach ($p in '::/1', '8000::/1') {{\n\
Get-NetRoute -DestinationPrefix $p -InterfaceAlias $alias -PolicyStore ActiveStore \
-ErrorAction SilentlyContinue | Remove-NetRoute -Confirm:$false -ErrorAction SilentlyContinue\n\
New-NetRoute -DestinationPrefix $p -InterfaceAlias $alias -NextHop '::' \
-RouteMetric 1 -PolicyStore ActiveStore -ErrorAction SilentlyContinue | Out-Null\n\
}}\n"
)
}
/// Removes the halves and every bypass route. What is not there is the state
/// wanted.
fn remove_client_script(interface: &str) -> String {
let alias = ps_literal(interface);
format!(
"$ErrorActionPreference = 'SilentlyContinue'\n\
$alias = {alias}\n\
foreach ($p in '0.0.0.0/1', '128.0.0.0/1', '::/1', '8000::/1') {{\n\
Get-NetRoute -DestinationPrefix $p -InterfaceAlias $alias -PolicyStore ActiveStore | \
Remove-NetRoute -Confirm:$false\n\
}}\n\
Get-NetRoute -AddressFamily IPv4 -PolicyStore ActiveStore | \
Where-Object {{ $_.RouteMetric -eq {BYPASS_METRIC} -and $_.DestinationPrefix.EndsWith('/32') }} | \
Remove-NetRoute -Confirm:$false\n\
exit 0\n"
)
}
/// Every address to keep direct: the ones given and what the names resolve
/// to, sorted and without duplicates.
fn bypass_addresses(
resolve: Resolve<'_>,
addresses: &[Ipv4Addr],
hosts: &[String],
) -> Vec<Ipv4Addr> {
let mut all: std::collections::BTreeSet<Ipv4Addr> = addresses.iter().copied().collect();
for host in hosts {
all.extend(resolve(host));
}
all.into_iter()
.filter(|a| !a.is_loopback() && !a.is_unspecified() && !a.is_multicast())
.collect()
}
fn apply_client(
run: RunScript<'_>,
resolve: Resolve<'_>,
plan: &ExitHostPlan,
) -> Result<(), String> {
let interface = plan.interface.as_str();
let gateway = run(&gateway_script(interface))?;
let (index, hop) = parse_gateway(&gateway)
.ok_or_else(|| format!("cannot read the default gateway from `{}`", gateway.trim()))?;
let addresses = bypass_addresses(resolve, &plan.bypass, &plan.bypass_hosts);
run(&bypass_script(index, hop, &addresses))?;
if let Err(why) = run(&halves_script(interface)) {
// Do not leave a half-installed set: bypass routes alone are harmless
// but the next apply starts clean.
let _ = run(&remove_client_script(interface));
return Err(why);
}
Ok(())
}
fn parse_forwarding(text: &str) -> Option<bool> {
match text.trim().to_ascii_lowercase().as_str() {
"enabled" | "1" => Some(true),
"disabled" | "0" => Some(false),
_ => None,
}
}
/// Makes the host match the plan.
pub(super) fn apply_plan(
run: RunScript<'_>,
resolve: Resolve<'_>,
plan: &ExitHostPlan,
) -> ExitHostReport {
let interface = plan.interface.as_str();
// What an earlier plan left and this one no longer wants.
let pruned = run(&prune_script(interface, &plan.offer));
let offer = plan
.offer
.iter()
.map(|range| {
let outcome = match &pruned {
Err(why) => RuleOutcome::Failed(why.clone()),
Ok(_) => match run(&nat_script(interface, range)) {
Ok(_) => RuleOutcome::Applied,
Err(why) => RuleOutcome::Failed(why),
},
};
(*range, outcome)
})
.collect();
let client = if plan.client {
Some(match apply_client(run, resolve, plan) {
Ok(()) => RuleOutcome::Applied,
Err(why) => RuleOutcome::Failed(why),
})
} else {
// Whatever an earlier plan left must not stay in force.
let _ = run(&remove_client_script(interface));
None
};
let forwarding = if plan.offer.is_empty() {
None
} else {
run(&forwarding_script(interface))
.ok()
.and_then(|text| parse_forwarding(&text))
};
ExitHostReport {
offer,
forwarding,
client,
}
}
/// Removes every NAT object of this agent's for the interface.
pub(super) fn clear_all(run: RunScript<'_>, interface: &str) {
if let Err(why) = run(&prune_script(interface, &[])) {
tracing::warn!(interface, "cannot remove the exit node NAT: {why}");
}
if let Err(why) = run(&remove_client_script(interface)) {
tracing::warn!(interface, "cannot remove the exit node routes: {why}");
}
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use std::cell::RefCell;
use super::*;
fn range() -> Ipv4Range {
"10.13.37.0/24".parse().unwrap()
}
fn plan(client: bool) -> ExitHostPlan {
ExitHostPlan {
interface: "tsun0".into(),
offer: vec![range()],
client,
overlay: vec![range()],
..Default::default()
}
}
#[test]
fn the_nat_is_named_with_the_tag_and_replaces_its_own() {
let script = nat_script("tsun0", &range());
assert!(
script.contains("$name = 'tsunagi-exit:tsun0:10.13.37.0/24'"),
"{script}"
);
assert!(script.contains("-InternalIPInterfaceAddressPrefix '10.13.37.0/24'"));
assert!(
script.find("Remove-NetNat").unwrap() < script.find("New-NetNat").unwrap(),
"{script}"
);
}
#[test]
fn a_quote_in_a_value_cannot_escape_the_literal() {
let script = nat_script("x'; Remove-Item C:\\ #", &range());
assert!(script.contains("'tsunagi-exit:x''; Remove-Item C:\\ #:10.13.37.0/24'"));
assert!(prune_script("x'y", &[]).contains("'tsunagi-exit:x''y:'"));
assert!(forwarding_script("x'y").contains("'x''y'"));
}
#[test]
fn pruning_keeps_wanted_ranges_and_only_selects_this_interfaces_nats() {
let script = prune_script("tsun0", &[range()]);
assert!(script.contains("$prefix = 'tsunagi-exit:tsun0:'"));
assert!(script.contains("$keep = @('tsunagi-exit:tsun0:10.13.37.0/24')"));
assert!(script.contains("StartsWith($prefix)"));
assert!(script.contains("exit 0"));
assert!(prune_script("tsun0", &[]).contains("$keep = @()"));
}
#[test]
fn forwarding_text_is_read_never_written() {
assert_eq!(parse_forwarding("Enabled\r\n"), Some(true));
assert_eq!(parse_forwarding("Disabled"), Some(false));
assert_eq!(parse_forwarding("what"), None);
let script = forwarding_script("tsun0");
assert!(script.contains("Get-NetIPInterface"));
assert!(!script.contains("Set-NetIPInterface"));
}
#[test]
fn offering_prunes_then_creates_and_reads_forwarding() {
let calls = RefCell::new(Vec::new());
let run = |script: &str| -> Result<String, String> {
calls.borrow_mut().push(script.to_string());
Ok(if script.contains("Get-NetIPInterface") {
"Disabled\n".to_string()
} else {
String::new()
})
};
let report = apply_plan(&run, &no_names, &plan(false));
assert_eq!(report.offer, vec![(range(), RuleOutcome::Applied)]);
assert_eq!(report.forwarding, Some(false));
assert_eq!(report.client, None);
let calls = calls.borrow();
assert!(calls[0].contains("StartsWith"), "{calls:?}");
assert!(calls[1].contains("New-NetNat"), "{calls:?}");
// Without a client, the routes of an earlier plan are taken away.
assert!(calls[2].contains("RouteMetric -eq 7373"), "{calls:?}");
assert!(calls[3].contains("Get-NetIPInterface"), "{calls:?}");
}
#[test]
fn a_failing_nat_is_that_ranges_failure() {
let run = |script: &str| -> Result<String, String> {
if script.contains("New-NetNat") {
Err("A NAT with this prefix already exists".into())
} else {
Ok(String::new())
}
};
let report = apply_plan(&run, &no_names, &plan(false));
let RuleOutcome::Failed(why) = &report.offer[0].1 else {
panic!("{report:?}");
};
assert!(why.contains("already exists"), "{why}");
}
fn no_names(_: &str) -> Vec<Ipv4Addr> {
Vec::new()
}
fn client_plan() -> ExitHostPlan {
ExitHostPlan {
interface: "tsun0".into(),
offer: Vec::new(),
client: true,
overlay: vec![range()],
bypass: vec![Ipv4Addr::new(138, 201, 61, 182)],
bypass_hosts: vec!["relay.example".into()],
}
}
#[test]
fn the_gateway_is_read_from_the_script_output() {
assert_eq!(
parse_gateway("12 192.168.1.1\r\n"),
Some((12, Ipv4Addr::new(192, 168, 1, 1)))
);
assert_eq!(parse_gateway(""), None);
assert_eq!(parse_gateway("12"), None);
assert_eq!(parse_gateway("12 gw"), None);
assert_eq!(parse_gateway("12 1.1.1.1 extra"), None);
let script = gateway_script("tsun0");
assert!(script.contains("-ne $alias"));
assert!(script.contains("-ne '0.0.0.0'"));
}
#[test]
fn bypass_routes_are_marked_host_routes_through_the_physical_gateway() {
let script = bypass_script(
12,
Ipv4Addr::new(192, 168, 1, 1),
&[Ipv4Addr::new(1, 2, 3, 4), Ipv4Addr::new(5, 6, 7, 8)],
);
assert!(script.contains("@('1.2.3.4', '5.6.7.8')"), "{script}");
assert!(script.contains("-InterfaceIndex 12 -NextHop '192.168.1.1'"));
assert!(script.contains("-RouteMetric 7373"));
assert!(script.contains("-PolicyStore ActiveStore"));
// Stale marked routes go first, and only /32 ones with our metric.
assert!(script.find("Remove-NetRoute").unwrap() < script.find("New-NetRoute").unwrap());
assert!(script.contains("RouteMetric -eq 7373"));
assert!(script.contains("EndsWith('/32')"));
}
#[test]
fn the_halves_go_through_the_overlay_and_ipv6_is_blackholed() {
let script = halves_script("tsun0");
assert!(script.contains("'0.0.0.0/1', '128.0.0.0/1'"));
assert!(script.contains("-InterfaceAlias $alias -NextHop '0.0.0.0'"));
assert!(script.contains("'::/1', '8000::/1'"));
assert!(script.contains("-PolicyStore ActiveStore"));
assert!(
!script.contains("0.0.0.0/0"),
"the default route is left alone"
);
assert!(halves_script("x'y").contains("'x''y'"));
}
#[test]
fn addresses_are_resolved_merged_and_cleaned() {
let resolve = |host: &str| -> Vec<Ipv4Addr> {
match host {
"relay.example" => vec![Ipv4Addr::new(9, 9, 9, 9), Ipv4Addr::new(1, 2, 3, 4)],
_ => Vec::new(),
}
};
let all = bypass_addresses(
&resolve,
&[
Ipv4Addr::new(1, 2, 3, 4),
Ipv4Addr::LOCALHOST,
Ipv4Addr::UNSPECIFIED,
],
&["relay.example".into(), "gone.example".into()],
);
assert_eq!(
all,
vec![Ipv4Addr::new(1, 2, 3, 4), Ipv4Addr::new(9, 9, 9, 9)]
);
}
#[test]
fn the_bypass_is_written_before_the_halves_that_need_it() {
let calls = RefCell::new(Vec::new());
let run = |script: &str| -> Result<String, String> {
calls.borrow_mut().push(script.to_string());
Ok(if script.contains("Select-Object -First 1") {
"12 192.168.1.1\n".to_string()
} else {
String::new()
})
};
let resolve = |_: &str| vec![Ipv4Addr::new(9, 9, 9, 9)];
let report = apply_plan(&run, &resolve, &client_plan());
assert_eq!(report.client, Some(RuleOutcome::Applied));
let calls = calls.borrow();
let at = |needle: &str| calls.iter().position(|c| c.contains(needle)).unwrap();
assert!(at("Select-Object -First 1") < at("-RouteMetric 7373"));
assert!(at("-RouteMetric 7373") < at("'0.0.0.0/1', '128.0.0.0/1'"));
let bypass = &calls[at("-RouteMetric 7373")];
assert!(
bypass.contains("'9.9.9.9'") && bypass.contains("'138.201.61.182'"),
"{bypass}"
);
}
#[test]
fn nothing_goes_through_the_overlay_when_the_bypass_cannot_be_written() {
let run = |script: &str| -> Result<String, String> {
if script.contains("Select-Object -First 1") {
Ok("12 192.168.1.1".to_string())
} else if script.contains("-RouteMetric 7373") && script.contains("foreach ($ip") {
Err("Access is denied".to_string())
} else {
assert!(!script.contains("-NextHop '0.0.0.0'"), "{script}");
Ok(String::new())
}
};
let report = apply_plan(&run, &no_names, &client_plan());
assert_eq!(
report.client,
Some(RuleOutcome::Failed("Access is denied".into()))
);
}
#[test]
fn a_missing_gateway_is_a_failure_that_installs_no_route() {
let run = |script: &str| -> Result<String, String> {
assert!(!script.contains("New-NetRoute"), "{script}");
if script.contains("Select-Object -First 1") {
Err("the host has no default gateway outside the overlay".into())
} else {
Ok(String::new())
}
};
let report = apply_plan(&run, &no_names, &client_plan());
let Some(RuleOutcome::Failed(why)) = report.client else {
panic!("{report:?}");
};
assert!(why.contains("no default gateway"), "{why}");
}
#[test]
fn a_failed_halves_step_removes_what_was_written() {
let calls = RefCell::new(Vec::new());
let run = |script: &str| -> Result<String, String> {
calls.borrow_mut().push(script.to_string());
if script.contains("Select-Object -First 1") {
Ok("12 192.168.1.1".to_string())
} else if script.contains("'128.0.0.0/1'") && script.contains("New-NetRoute") {
Err("boom".to_string())
} else {
Ok(String::new())
}
};
let report = apply_plan(&run, &no_names, &client_plan());
assert!(matches!(report.client, Some(RuleOutcome::Failed(_))));
assert!(calls.borrow().last().unwrap().contains("Remove-NetRoute"));
}
#[test]
fn dropping_the_client_removes_its_routes_and_clearing_does_too() {
let calls = RefCell::new(Vec::new());
let run = |script: &str| -> Result<String, String> {
calls.borrow_mut().push(script.to_string());
Ok(String::new())
};
let report = apply_plan(&run, &no_names, &plan(false));
assert_eq!(report.client, None);
assert!(
calls
.borrow()
.iter()
.any(|c| c.contains("RouteMetric -eq 7373"))
);
let script = remove_client_script("tsun0");
assert!(script.contains("'0.0.0.0/1', '128.0.0.0/1', '::/1', '8000::/1'"));
assert!(script.contains("exit 0"));
// Only routes on our interface, or marked ours, are touched.
assert!(script.contains("-InterfaceAlias $alias"));
calls.borrow_mut().clear();
clear_all(&run, "tsun0");
assert!(
calls
.borrow()
.iter()
.any(|c| c.contains("RouteMetric -eq 7373"))
);
}
#[test]
fn clearing_prunes_everything_for_the_interface() {
let seen = RefCell::new(Vec::<String>::new());
let run = |script: &str| -> Result<String, String> {
seen.borrow_mut().push(script.to_string());
Ok(String::new())
};
clear_all(&run, "tsun0");
let seen = seen.borrow();
assert!(seen[0].contains("$keep = @()"));
assert!(seen[0].contains("'tsunagi-exit:tsun0:'"));
}
}
crates/tsunagi/src/overlay/interface.rs → crates/tsunagi-core/src/overlay/interface.rs
+741 -3
View File
@@ -29,7 +29,11 @@ use tokio::task::JoinHandle;
use crate::identity::NetworkId;
use super::hostrules::{BroadcastHostRules, BroadcastRulesPlan, BroadcastRulesReport};
use super::exit::{ExitRulesReport, RuleSetReport};
use super::hostrules::{
BroadcastHostRules, BroadcastRulesPlan, BroadcastRulesReport, ExitHostPlan, ExitHostReport,
RuleOutcome, offer_detail,
};
use super::packet::IpHeader;
use super::router::{Route, RoutingTable};
use super::tun::{TunDevice, TunFactory, TunRequest};
@@ -141,23 +145,68 @@ impl Tally {
}
/// The overlay interface.
#[derive(Debug)]
pub struct Interface {
device: Arc<dyn TunDevice>,
factory: Arc<dyn TunFactory>,
name: String,
mtu: u32,
routes: Arc<RoutingTable>,
carrier: Arc<dyn PacketCarrier>,
tally: Arc<Tally>,
/// What was last applied to the host, so an unchanged table is not
/// re-applied on every pass.
applied: std::sync::Mutex<Vec<Cidr>>,
task: std::sync::Mutex<Option<JoinHandle<()>>>,
vip_task: std::sync::Mutex<Option<JoinHandle<()>>>,
sync_lock: Arc<tokio::sync::Mutex<()>>,
/// The host rules for this interface, when the factory manages the host.
host_rules: Option<Arc<dyn BroadcastHostRules>>,
/// What those rules last reported, `None` while none are installed.
rules_report: std::sync::Mutex<Option<BroadcastRulesReport>>,
/// What the exit-node rules were last asked for, and what came of it.
exit_state: std::sync::Mutex<ExitState>,
/// Where each network's runtime reaches its peers and relays outside the
/// overlay; read only by hosts that cannot exempt the agent from the
/// exit node's default route.
underlay: std::sync::Mutex<std::collections::HashMap<NetworkId, Underlay>>,
}
/// Underlay destinations one network's runtime currently depends on.
#[derive(Debug, Default, Clone, PartialEq, Eq)]
struct Underlay {
addresses: std::collections::BTreeSet<std::net::Ipv4Addr>,
hosts: std::collections::BTreeSet<String>,
}
/// The exit-node host rules as this interface last left them.
#[derive(Debug, Default)]
struct ExitState {
/// Whether what an earlier run left behind has been taken away. Done once,
/// at the first sync, whether or not anything is wanted now: a firewall
/// rule that forwards a range must not outlive the switch that offered it.
cleaned: bool,
/// What was last applied, `None` while nothing is.
plan: Option<ExitHostPlan>,
/// What came of it.
report: Option<ExitHostReport>,
}
impl ExitState {
/// Whether every step of the last apply took.
fn is_ok(&self) -> bool {
self.report.as_ref().is_some_and(|report| {
report.offer.iter().all(|(_, outcome)| outcome.is_applied())
&& report.client.as_ref().is_none_or(RuleOutcome::is_applied)
})
}
}
impl std::fmt::Debug for Interface {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("Interface")
.field("name", &self.name)
.finish_non_exhaustive()
}
}
impl Interface {
@@ -181,7 +230,29 @@ impl Interface {
let name = device.name().to_string();
let tally = Arc::new(Tally::default());
let vip_task = {
let routes = Arc::clone(&routes);
let carrier = Arc::clone(&carrier);
tokio::spawn(async move {
let mut interval = tokio::time::interval(std::time::Duration::from_millis(250));
interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
let mut checks = tokio::task::JoinSet::new();
loop {
tokio::select! {
_=interval.tick()=>{
let (probes,jobs)=routes.vips.tick(routes.vips.check_capacity().saturating_sub(checks.len()));
for (route,packet) in probes {carrier.carry(route,packet);}
for job in jobs {checks.spawn(job.run());}
},
Some(result)=checks.join_next(),if !checks.is_empty()=>{
if let Ok((job,ok))=result {routes.vips.checked(job,ok);}
}
}
}
})
};
let task = {
let carrier = Arc::clone(&carrier);
let device = Arc::clone(&device);
let routes = Arc::clone(&routes);
let tally = Arc::clone(&tally);
@@ -191,6 +262,26 @@ impl Interface {
tally.malformed.fetch_add(1, Ordering::Relaxed);
continue;
};
match routes.vips.outgoing(packet.clone()) {
crate::vip::router::Decision::Send(route, packet) => {
for part in crate::vip::packet::fragments(packet, mtu) {
if carrier.carry(route, part) {
tally.sent.fetch_add(1, Ordering::Relaxed);
} else {
tally.undeliverable.fetch_add(1, Ordering::Relaxed);
}
}
continue;
}
crate::vip::router::Decision::Local(packet) => {
for part in crate::vip::packet::fragments(packet, mtu) {
let _ = device.send(part).await;
}
continue;
}
crate::vip::router::Decision::Drop => continue,
crate::vip::router::Decision::Pass => {}
}
let destination = header.destination();
if let (IpAddr::V4(source), IpAddr::V4(destination)) =
(header.source(), destination)
@@ -239,12 +330,16 @@ impl Interface {
name,
mtu,
routes,
carrier,
tally,
applied: std::sync::Mutex::new(Vec::new()),
task: std::sync::Mutex::new(Some(task)),
vip_task: std::sync::Mutex::new(Some(vip_task)),
sync_lock: Arc::new(tokio::sync::Mutex::new(())),
host_rules,
rules_report: std::sync::Mutex::new(None),
exit_state: std::sync::Mutex::new(ExitState::default()),
underlay: std::sync::Mutex::new(std::collections::HashMap::new()),
})
}
@@ -298,6 +393,279 @@ impl Interface {
}
}
/// Brings the host's exit-node rules in line with the routing table.
///
/// Idempotent, and cheap when nothing changed. It never fails: the
/// outcome is kept for [`Self::exit_rules`]. The first call also takes
/// away whatever an earlier run left, even when nothing is wanted now.
pub async fn sync_exit_rules(&self) {
let Some(exit) = self
.host_rules
.as_ref()
.and_then(|rules| rules.exit_rules())
else {
return;
};
let _guard = self.sync_lock.lock().await;
let mut ranges: Vec<_> = self
.routes
.exit_offers()
.into_iter()
.map(|(_, range)| range)
.collect();
ranges.sort_by_key(|range| (range.base, range.prefix_len));
ranges.dedup();
let client = self.routes.exit_via().is_some();
let overlay = self.routes.overlay_ranges();
let (bypass, bypass_hosts) = if client && exit.needs_bypass() {
self.underlay_targets(&overlay)
} else {
(Vec::new(), Vec::new())
};
let wanted = ExitHostPlan {
interface: self.name.clone(),
offer: ranges,
client,
overlay,
bypass,
bypass_hosts,
};
let first = !std::mem::replace(&mut self.exit_state().cleaned, true);
if first {
exit.clear(&self.name).await;
}
let (planned, ok) = {
let state = self.exit_state();
(state.plan.clone(), state.is_ok())
};
if wanted.is_empty() {
if planned.is_some() {
exit.clear(&self.name).await;
let mut state = self.exit_state();
state.plan = None;
state.report = None;
tracing::info!(interface = %self.name, "exit node host rules removed");
}
return;
}
if planned.as_ref() == Some(&wanted) && ok {
if let Some(outcome) = exit.refresh(&wanted).await {
if let RuleOutcome::Failed(reason) = &outcome {
tracing::warn!(interface = %self.name, %reason, "cannot refresh exit-client socket protection");
}
if let Some(report) = self.exit_state().report.as_mut() {
report.client = Some(outcome);
}
}
return;
}
let report = exit.apply(&wanted).await;
for (range, outcome) in &report.offer {
match outcome {
RuleOutcome::Applied => tracing::info!(
interface = %self.name,
%range,
"exit node firewall rules applied"
),
RuleOutcome::Failed(reason) => tracing::warn!(
interface = %self.name,
%range,
"exit node firewall rules are incomplete: {reason}"
),
}
}
if report.offer.iter().any(|(_, outcome)| outcome.is_applied())
&& report.forwarding == Some(false)
{
tracing::warn!(
interface = %self.name,
"kernel forwarding is off for this interface, so nothing is forwarded for the \
exit node yet: {}",
if cfg!(target_os = "windows") {
"`Set-NetIPInterface -Forwarding Enabled` on the interfaces involved"
} else if cfg!(target_os = "macos") {
"`sysctl -w net.inet.ip.forwarding=1`, set again after every reboot"
} else {
"`sysctl -w net.ipv4.ip_forward=1` (or `net.ipv4.conf.<interface>.forwarding=1`), \
and set it in sysctl.d to keep it"
}
);
}
match &report.client {
Some(RuleOutcome::Applied) => tracing::info!(
interface = %self.name,
"this device's traffic is routed through its exit node"
),
Some(RuleOutcome::Failed(reason)) => tracing::warn!(
interface = %self.name,
"cannot route this device's traffic through the exit node: {reason}"
),
None => {}
}
let mut state = self.exit_state();
state.plan = Some(wanted);
state.report = Some(report);
}
/// Tells the interface where a network's runtime reaches peers and
/// relays outside the overlay. When the host needs those kept out of the
/// exit node's default route and the set changed, the rules follow.
pub async fn set_underlay(
&self,
network: NetworkId,
addresses: impl IntoIterator<Item = std::net::Ipv4Addr>,
hosts: impl IntoIterator<Item = String>,
) {
let new = Underlay {
addresses: addresses.into_iter().collect(),
hosts: hosts.into_iter().collect(),
};
let changed = {
let mut map = match self.underlay.lock() {
Ok(guard) => guard,
Err(poisoned) => poisoned.into_inner(),
};
if new == Underlay::default() {
map.remove(&network).is_some()
} else {
map.insert(network, new.clone()) != Some(new)
}
};
let needed = changed
&& self.routes.exit_via().is_some()
&& self
.host_rules
.as_ref()
.and_then(|rules| rules.exit_rules())
.is_some_and(|exit| exit.needs_bypass());
if needed {
self.sync_exit_rules().await;
}
}
/// Every underlay destination worth keeping direct, minus anything that
/// is the overlay itself, loopback or link-local.
fn underlay_targets(
&self,
overlay: &[crate::state::Ipv4Range],
) -> (Vec<std::net::Ipv4Addr>, Vec<String>) {
let map = match self.underlay.lock() {
Ok(guard) => guard,
Err(poisoned) => poisoned.into_inner(),
};
let mut addresses = std::collections::BTreeSet::new();
let mut hosts = std::collections::BTreeSet::new();
for underlay in map.values() {
addresses.extend(underlay.addresses.iter().copied().filter(|address| {
!address.is_loopback()
&& !address.is_link_local()
&& !address.is_multicast()
&& !address.is_unspecified()
&& !address.is_broadcast()
&& !overlay.iter().any(|range| range.contains(*address))
}));
hosts.extend(underlay.hosts.iter().cloned());
}
(addresses.into_iter().collect(), hosts.into_iter().collect())
}
fn exit_state(&self) -> std::sync::MutexGuard<'_, ExitState> {
match self.exit_state.lock() {
Ok(guard) => guard,
Err(poisoned) => poisoned.into_inner(),
}
}
/// What the exit-node rules of a network are doing now.
///
/// `offer` is `Some` while the network offers this agent as an exit node,
/// `client` while this device sends its traffic through a member of it.
pub fn exit_rules(&self, network: NetworkId) -> ExitRulesReport {
let Some((policy, range)) = self.routes.exit_settings(network) else {
return ExitRulesReport::default();
};
let unavailable = match self
.host_rules
.as_ref()
.map(|rules| rules.exit_rules().is_some())
{
None => Some(
"this overlay interface is not on the host (no TUN device), so there is \
nothing to forward to",
),
Some(false) => Some("exit nodes are not supported on this platform yet"),
Some(true) => None,
};
let failed = |detail: &str| RuleSetReport {
ok: false,
detail: detail.to_string(),
};
let state = self.exit_state();
let mut out = ExitRulesReport::default();
if policy.offer {
out.offer = Some(match (unavailable, range) {
(Some(why), _) => failed(why),
(None, None) => failed("waiting for the network to agree an address range"),
(None, Some(range)) => {
match state
.report
.as_ref()
.and_then(|report| report.offer.iter().find(|(r, _)| *r == range))
{
Some((_, outcome)) => RuleSetReport {
ok: outcome.is_applied(),
detail: offer_detail(outcome, &range),
},
None => failed("not applied yet"),
}
}
});
if unavailable.is_none() {
out.forwarding = state.report.as_ref().and_then(|report| report.forwarding);
}
}
if policy.via.is_some()
&& self
.routes
.exit_via()
.is_some_and(|(selected, _)| selected == network)
{
out.client = Some(match unavailable {
Some(why) => failed(why),
None => match state
.report
.as_ref()
.and_then(|report| report.client.as_ref())
{
Some(RuleOutcome::Applied) => RuleSetReport {
ok: true,
detail: "this device's traffic is routed through the exit node".to_string(),
},
Some(RuleOutcome::Failed(reason)) => failed(reason),
None => failed("not applied yet"),
},
});
}
out
}
/// Whether this agent can really act as an exit node in a network: the
/// rules members' traffic needs are in place. Kernel forwarding is not
/// part of it, because the agent never turns that on itself.
pub fn exit_offer_ready(&self, network: NetworkId) -> bool {
let Some((policy, Some(range))) = self.routes.exit_settings(network) else {
return false;
};
policy.offer
&& self.exit_state().report.as_ref().is_some_and(|report| {
report
.offer
.iter()
.any(|(r, outcome)| *r == range && outcome.is_applied())
})
}
/// What the broadcast host rules achieved, `None` when none are installed
/// or this interface has no host to configure.
pub fn broadcast_rules(&self) -> Option<BroadcastRulesReport> {
@@ -379,6 +747,16 @@ impl Interface {
if let Some(rules) = &self.host_rules {
rules.clear(&self.name).await;
self.store_rules_report(None);
let touched = {
let state = self.exit_state();
state.cleaned || state.plan.is_some()
};
if let (true, Some(exit)) = (touched, rules.exit_rules()) {
exit.clear(&self.name).await;
let mut state = self.exit_state();
state.plan = None;
state.report = None;
}
}
// The packet loop holds the device open, and it ends only when the
// device reports end of stream — which a real interface never does
@@ -391,6 +769,15 @@ impl Interface {
task.abort();
let _ = task.await;
}
let vip_task = self
.vip_task
.lock()
.unwrap_or_else(|e| e.into_inner())
.take();
if let Some(task) = vip_task {
task.abort();
let _ = task.await;
}
self.factory.destroy(self.device.name()).await;
}
@@ -435,11 +822,34 @@ impl Interface {
IpAddr::V4(destination) if self.routes.is_local_destination(network, destination) => {
false
}
// This agent is an exit node of this network: a member's packet
// for the internet goes to the host, which masquerades it.
IpAddr::V4(destination) if self.routes.accepts_exit_traffic(network, destination) => {
false
}
_ => {
self.tally.wrong_destination.fetch_add(1, Ordering::Relaxed);
return Err(Rejected::WrongDestination);
}
};
// Source ownership was checked before any VIP reverse translation.
let packet = match self.routes.vips.incoming(network, peer, packet.clone()) {
crate::vip::router::Decision::Send(route, packet) => {
// Probe replies use the ordinary peer carrier, never the TUN.
self.carrier.carry(route, packet);
return Ok(());
}
crate::vip::router::Decision::Local(packet) => {
for part in crate::vip::packet::fragments(packet, self.mtu) {
if self.device.send(part).await.is_ok() {
self.tally.received.fetch_add(1, Ordering::Relaxed);
}
}
return Ok(());
}
crate::vip::router::Decision::Drop => return Ok(()),
crate::vip::router::Decision::Pass => packet,
};
// Remote broadcasts terminate here. Only local TUN ingress can fan out.
if self.device.send(packet).await.is_ok() {
if broadcast {
@@ -464,6 +874,14 @@ impl Interface {
impl Drop for Interface {
fn drop(&mut self) {
if let Some(task) = self
.vip_task
.lock()
.unwrap_or_else(|e| e.into_inner())
.take()
{
task.abort();
}
if let Ok(mut guard) = self.task.lock()
&& let Some(task) = guard.take()
{
@@ -525,7 +943,7 @@ mod tests {
use crate::overlay::broadcast::BroadcastPolicy;
use crate::overlay::hostrules::{MockHostRules, RuleOutcome};
use crate::overlay::provision::{ManagedTunFactory, MockProvisioner};
use crate::overlay::router::NetworkRoutes;
use crate::overlay::router::{ExitPolicy, NetworkRoutes};
use crate::overlay::tun::{MemoryTun, MemoryTunFactory};
fn network(name: &str) -> NetworkId {
@@ -667,6 +1085,7 @@ mod tests {
.set_network(
id,
NetworkRoutes {
exit: Default::default(),
broadcast: Default::default(),
range: Some("10.13.37.0/24".parse().unwrap()),
local: Some(addr(1)),
@@ -833,6 +1252,7 @@ mod tests {
.set_network(
id,
NetworkRoutes {
exit: Default::default(),
broadcast: Default::default(),
range: Some("10.13.37.0/24".parse().unwrap()),
local: Some(addr(9)),
@@ -856,6 +1276,7 @@ mod tests {
.set_network(
id,
NetworkRoutes {
exit: Default::default(),
broadcast: BroadcastPolicy {
enabled: broadcast_enabled,
..Default::default()
@@ -910,6 +1331,7 @@ mod tests {
.set_network(
id,
NetworkRoutes {
exit: Default::default(),
broadcast: BroadcastPolicy::default(),
range: Some("10.13.37.0/24".parse().unwrap()),
local: Some(addr(9)),
@@ -978,4 +1400,320 @@ mod tests {
assert!(interface.broadcast_rules().is_none());
interface.remove().await;
}
const INTERNET: Ipv4Addr = Ipv4Addr::new(93, 184, 216, 34);
fn offering() -> ExitPolicy {
ExitPolicy {
offer: true,
via: None,
}
}
fn using(exit: EndpointId) -> ExitPolicy {
ExitPolicy {
offer: false,
via: Some(exit),
}
}
#[tokio::test]
async fn internet_traffic_goes_to_the_exit_node_and_only_once_one_is_chosen() {
let (interface, device, routes, carrier, id) = interface(false).await;
device.push_from_os(ipv4(addr(1), INTERNET, b"before"));
wait_for(&interface, |counters| counters.unroutable).await;
assert!(carrier.carried().is_empty(), "no exit node, no route");
routes.set_exit(id, using(peer(2)));
device.push_from_os(ipv4(addr(1), INTERNET, b"after"));
wait_for(&interface, |counters| counters.sent).await;
let carried = carrier.carried();
assert_eq!(carried.len(), 1);
assert_eq!(carried[0].0.peer, peer(2));
assert_eq!(&carried[0].1[20..], b"after");
}
#[tokio::test]
async fn the_exit_node_may_answer_with_any_source_and_nobody_else_may() {
let (interface, device, routes, _carrier, id) = interface(false).await;
routes.set_exit(id, using(peer(2)));
interface
.deliver(id, peer(2), ipv4(INTERNET, addr(1), b"reply"))
.await
.unwrap();
let written = tokio::time::timeout(std::time::Duration::from_secs(5), device.pop_to_os())
.await
.expect("the reply reaches the interface")
.unwrap();
assert_eq!(&written[20..], b"reply");
// Another member cannot use the internet's addresses.
assert_eq!(
interface
.deliver(id, peer(3), ipv4(INTERNET, addr(1), b"spoof"))
.await,
Err(Rejected::WrongSource)
);
// Nor can the exit node speak for a member.
assert_eq!(
interface
.deliver(id, peer(2), ipv4(addr(3), addr(1), b"spoof"))
.await,
Err(Rejected::WrongSource)
);
// And what it sends still has to be for this agent.
assert_eq!(
interface
.deliver(id, peer(2), ipv4(INTERNET, addr(200), b"elsewhere"))
.await,
Err(Rejected::WrongDestination)
);
assert_eq!(interface.counters().received, 1);
}
#[tokio::test]
async fn an_exit_node_hands_a_members_internet_packet_to_the_host_only_while_offering() {
let (interface, device, routes, _carrier, id) = interface(false).await;
// Not offering: dropped exactly as before.
assert_eq!(
interface
.deliver(id, peer(2), ipv4(addr(2), INTERNET, b"request"))
.await,
Err(Rejected::WrongDestination)
);
assert_eq!(interface.counters().received, 0);
routes.set_exit(id, offering());
interface
.deliver(id, peer(2), ipv4(addr(2), INTERNET, b"request"))
.await
.unwrap();
let written = tokio::time::timeout(std::time::Duration::from_secs(5), device.pop_to_os())
.await
.expect("the request reaches the interface")
.unwrap();
assert_eq!(&written[20..], b"request");
// The source must still be the sender's own address.
assert_eq!(
interface
.deliver(id, peer(2), ipv4(addr(3), INTERNET, b"spoof"))
.await,
Err(Rejected::WrongSource)
);
assert_eq!(
interface
.deliver(id, peer(2), ipv4(INTERNET, INTERNET, b"spoof"))
.await,
Err(Rejected::WrongSource)
);
// Offering does not make this a door to the overlay or to this host.
for destination in [addr(3), addr(200), Ipv4Addr::LOCALHOST] {
assert_eq!(
interface
.deliver(id, peer(2), ipv4(addr(2), destination, b"no"))
.await,
Err(Rejected::WrongDestination),
"{destination}"
);
}
assert_eq!(interface.counters().received, 1);
}
#[tokio::test]
async fn the_underlay_reaches_the_rules_only_for_hosts_that_need_it_and_only_while_using_one() {
let rules = MockHostRules::new();
let (interface, routes, id) = managed_interface(&rules, false).await;
let relay = || vec!["relay.example".to_string()];
let direct = || {
vec![
Ipv4Addr::new(138, 201, 61, 182),
Ipv4Addr::new(10, 13, 37, 9), // the overlay itself
Ipv4Addr::LOCALHOST,
Ipv4Addr::new(169, 254, 1, 1),
]
};
// A host that does not ask never gets them, so Linux and macOS do not
// re-apply their rules whenever a path moves.
routes.set_exit(id, using(peer(2)));
interface.set_underlay(id, direct(), relay()).await;
interface.sync_exit_rules().await;
let plan = rules.exit().installed().unwrap();
assert!(plan.client);
assert!(plan.bypass.is_empty() && plan.bypass_hosts.is_empty());
// A host that does gets the usable ones, and follows changes.
rules.exit().set_needs_bypass(true);
interface.set_underlay(id, direct(), relay()).await;
interface.sync_exit_rules().await;
let plan = rules.exit().installed().unwrap();
assert_eq!(plan.bypass, vec![Ipv4Addr::new(138, 201, 61, 182)]);
assert_eq!(plan.bypass_hosts, relay());
let applies = || {
rules
.exit()
.calls()
.iter()
.filter(|c| c.starts_with("apply"))
.count()
};
let before = applies();
interface.set_underlay(id, direct(), relay()).await;
assert_eq!(applies(), before, "an unchanged set is not applied again");
interface
.set_underlay(id, vec![Ipv4Addr::new(5, 6, 7, 8)], relay())
.await;
assert_eq!(applies(), before + 1);
assert_eq!(
rules.exit().installed().unwrap().bypass,
vec![Ipv4Addr::new(5, 6, 7, 8)]
);
// Not using an exit node: nothing to keep direct.
routes.set_exit(id, Default::default());
interface.sync_exit_rules().await;
interface
.set_underlay(id, vec![Ipv4Addr::new(1, 1, 1, 1)], relay())
.await;
assert!(rules.exit().installed().is_none());
}
#[tokio::test]
async fn the_exit_rules_follow_the_routing_table_and_leave_with_the_interface() {
let rules = MockHostRules::new();
let (interface, routes, id) = managed_interface(&rules, false).await;
let range: crate::state::Ipv4Range = "10.13.37.0/24".parse().unwrap();
// Nothing wanted: nothing installed, but what a crashed run left is
// taken away once.
interface.sync_exit_rules().await;
interface.sync_exit_rules().await;
assert_eq!(rules.exit().calls(), ["clear:tsunrules"]);
assert!(rules.exit().installed().is_none());
assert!(!interface.exit_offer_ready(id));
assert_eq!(interface.exit_rules(id), ExitRulesReport::default());
// Offering: the range is masqueraded, and the agent can say so.
routes.set_exit(id, offering());
assert!(!interface.exit_offer_ready(id), "not applied yet");
let waiting = interface.exit_rules(id).offer.unwrap();
assert!(!waiting.ok);
assert!(waiting.detail.contains("not applied yet"), "{waiting:?}");
interface.sync_exit_rules().await;
assert_eq!(rules.exit().installed().unwrap().offer, vec![range]);
assert!(interface.exit_offer_ready(id));
let report = interface.exit_rules(id);
assert!(report.offer.as_ref().unwrap().ok);
assert_eq!(report.forwarding, Some(true));
assert!(report.client.is_none());
// Unchanged: not applied again.
interface.sync_exit_rules().await;
assert_eq!(
rules
.exit()
.calls()
.iter()
.filter(|call| call.starts_with("apply"))
.count(),
1
);
// Forwarding off in the kernel: still installed, still ready, and said.
rules.exit().set_forwarding(Some(false));
routes.set_exit(
id,
ExitPolicy {
offer: true,
via: Some(peer(2)),
},
);
interface.sync_exit_rules().await;
assert!(rules.exit().installed().unwrap().client);
assert!(interface.exit_offer_ready(id));
let report = interface.exit_rules(id);
assert_eq!(report.forwarding, Some(false));
assert!(report.offer.unwrap().ok);
assert!(report.client.unwrap().ok);
// Turned off: removed, and no longer announced as ready.
routes.set_exit(id, ExitPolicy::default());
interface.sync_exit_rules().await;
assert!(rules.exit().installed().is_none());
assert!(!interface.exit_offer_ready(id));
assert_eq!(interface.exit_rules(id), ExitRulesReport::default());
routes.set_exit(id, offering());
interface.sync_exit_rules().await;
assert!(rules.exit().installed().is_some());
interface.remove().await;
assert!(
rules.exit().installed().is_none(),
"gone with the interface"
);
}
#[tokio::test]
async fn a_failed_exit_step_is_reported_and_the_agent_does_not_claim_to_be_an_exit_node() {
let rules = MockHostRules::new();
rules.exit().fail_with(Some("iptables needs root".into()));
let (interface, routes, id) = managed_interface(&rules, false).await;
routes.set_exit(id, offering());
interface.sync_exit_rules().await;
assert!(!interface.exit_offer_ready(id));
let offer = interface.exit_rules(id).offer.unwrap();
assert!(!offer.ok);
assert!(offer.detail.contains("iptables needs root"), "{offer:?}");
// It is tried again, not given up on, the next time it is asked.
rules.exit().fail_with(None);
interface.sync_exit_rules().await;
assert!(interface.exit_offer_ready(id));
interface.remove().await;
}
#[tokio::test]
async fn an_interface_without_a_host_cannot_be_an_exit_node_and_says_why() {
let (interface, _device, routes, _carrier, id) = interface(false).await;
routes.set_exit(
id,
ExitPolicy {
offer: true,
via: Some(peer(2)),
},
);
interface.sync_exit_rules().await;
assert!(!interface.exit_offer_ready(id));
let report = interface.exit_rules(id);
let offer = report.offer.unwrap();
assert!(!offer.ok);
assert!(offer.detail.contains("not on the host"), "{offer:?}");
assert!(!report.client.unwrap().ok);
interface.remove().await;
}
#[tokio::test]
async fn a_network_waiting_for_its_range_has_nothing_to_masquerade_yet() {
let rules = MockHostRules::new();
let (interface, routes, _id) = managed_interface(&rules, false).await;
let waiting = network("waiting");
routes
.set_network(
waiting,
NetworkRoutes {
exit: offering(),
..Default::default()
},
)
.unwrap();
interface.sync_exit_rules().await;
assert!(rules.exit().installed().is_none());
let offer = interface.exit_rules(waiting).offer.unwrap();
assert!(offer.detail.contains("address range"), "{offer:?}");
interface.remove().await;
}
}
crates/tsunagi/src/overlay/mod.rs → crates/tsunagi-core/src/overlay/mod.rs
+5 -3
View File
@@ -40,6 +40,7 @@ pub enum OverlayError {
pub mod broadcast;
pub mod config;
pub mod exit;
pub mod hostrules;
pub mod interface;
pub mod packet;
@@ -48,9 +49,10 @@ pub mod router;
pub mod tun;
pub use config::{Cidr, DEFAULT_INTERFACE_PREFIX, MAX_INTERFACE_NAME_LEN, interface_name};
pub use exit::{ExitRulesReport, RuleSetReport};
pub use hostrules::{
BroadcastHostRules, BroadcastRulesPlan, BroadcastRulesReport, MockHostRules, RuleOutcome,
UnsupportedHostRules,
BroadcastHostRules, BroadcastRulesPlan, BroadcastRulesReport, ExitHostPlan, ExitHostReport,
ExitHostRules, MockExitRules, MockHostRules, RuleOutcome, UnsupportedHostRules,
};
pub use interface::{Counters, Interface, PacketCarrier, Rejected};
pub use packet::IpHeader;
@@ -59,7 +61,7 @@ pub use provision::{
MockHost, MockProvisioner, Privilege, Provisioned, UnsupportedProvisioner, plan_changes,
probe_net_admin,
};
pub use router::{NetworkRoutes, Route, RouteError, RoutingTable};
pub use router::{ExitPolicy, NetworkRoutes, Route, RouteError, RoutingTable};
pub use tun::{MemoryTun, MemoryTunFactory, TunDevice, TunFactory, TunRequest, address_is_local};
#[cfg(all(feature = "tun-device", target_os = "linux"))]
crates/tsunagi/src/overlay/packet.rs → crates/tsunagi-core/src/overlay/packet.rs
View File
File renamed without changes.
crates/tsunagi/src/overlay/provision/factory.rs → crates/tsunagi-core/src/overlay/provision/factory.rs
View File
File renamed without changes.
crates/tsunagi/src/overlay/provision/linux.rs → crates/tsunagi-core/src/overlay/provision/linux.rs
+464 -2
View File
@@ -48,12 +48,14 @@ use futures_util::TryStreamExt;
use rtnetlink::packet_route::address::{AddressAttribute, AddressMessage};
use rtnetlink::packet_route::link::{InfoKind, LinkAttribute, LinkFlags, LinkInfo, LinkMessage};
use rtnetlink::packet_route::route::RouteScope;
use rtnetlink::{LinkMessageBuilder, LinkUnspec, RouteMessageBuilder};
use rtnetlink::packet_route::rule::{RuleAction, RuleAttribute, RuleMessage, RuleUidRange};
use rtnetlink::{IpVersion, LinkMessageBuilder, LinkUnspec, RouteMessageBuilder};
use crate::BoxFuture;
use crate::overlay::OverlayError;
use super::super::config::Cidr;
use super::super::hostrules::linux_exit_mark::{UNDERLAY_MARK, UNDERLAY_MASK};
use super::super::tun::{TunDevice, TunRequest};
use super::privilege::{NetAdmin, Privilege, probe_net_admin};
use super::{
@@ -72,6 +74,11 @@ enum Command {
SetBroadcastRoute(String, Ipv4Addr, Reply<()>),
/// Remove that route from this interface.
DelBroadcastRoute(String, Reply<()>),
/// Send the host's default traffic through this interface, except what
/// this user id sends.
SetExitClient(String, u32, Reply<()>),
/// Take those routing rules away again.
ClearExitClient(Reply<()>),
/// Stop the thread.
Stop,
}
@@ -203,6 +210,20 @@ impl RouteHandle {
Command::DelBroadcastRoute(interface, reply)
})
}
/// Sends the host's default traffic through `interface`, except what the
/// user id `uid` sends. Replaces what an earlier call set up.
pub(crate) fn set_exit_client(&self, interface: &str, uid: u32) -> Result<(), OverlayError> {
let interface = interface.to_string();
call_thread(&self.commands, |reply| {
Command::SetExitClient(interface, uid, reply)
})
}
/// Removes those rules. Succeeds when none are there.
pub(crate) fn clear_exit_client(&self) -> Result<(), OverlayError> {
call_thread(&self.commands, Command::ClearExitClient)
}
}
impl Drop for NetlinkProvisioner {
@@ -324,7 +345,11 @@ fn netlink_thread(requests: mpsc::Receiver<Command>) {
Command::Delete(_, reply)
| Command::Configure(_, reply)
| Command::SetBroadcastRoute(_, _, reply)
| Command::DelBroadcastRoute(_, reply) => {
| Command::DelBroadcastRoute(_, reply)
| Command::SetExitClient(_, _, reply) => {
let _ = reply.send(Err(OverlayError::Unavailable(message)));
}
Command::ClearExitClient(reply) => {
let _ = reply.send(Err(OverlayError::Unavailable(message)));
}
Command::Stop => return,
@@ -373,6 +398,22 @@ fn netlink_thread(requests: mpsc::Receiver<Command>) {
});
let _ = reply.send(result);
}
Command::SetExitClient(name, uid, reply) => {
let result = NetAdmin::acquire().and_then(|guard| {
let result = runtime.block_on(set_exit_client(&name, uid));
drop(guard);
result
});
let _ = reply.send(result);
}
Command::ClearExitClient(reply) => {
let result = NetAdmin::acquire().and_then(|guard| {
let result = runtime.block_on(clear_exit_client());
drop(guard);
result
});
let _ = reply.send(result);
}
}
}
}
@@ -596,6 +637,344 @@ async fn del_broadcast_route(name: &str) -> Result<(), OverlayError> {
.await
}
/// The routing table that holds the exit node's default route: `tsun` in
/// ASCII-ish hex, and nothing else uses it.
const EXIT_TABLE: u32 = 0x7473;
/// The main table.
const MAIN_TABLE: u32 = 254;
/// Protect marked underlay and initial socket routing before Tailscale's rules.
const EXIT_MARK_PRIORITY: u32 = 5205;
const EXIT_BYPASS_PRIORITY: u32 = 5206;
const LEGACY_EXIT_BYPASS_PRIORITY: u32 = 5280;
const EXIT_SPECIFIC_PRIORITY: u32 = 5290;
const EXIT_DEFAULT_PRIORITY: u32 = 5300;
/// The table a rule looks up.
fn rule_table(rule: &RuleMessage) -> u32 {
rule.attributes
.iter()
.find_map(|attribute| match attribute {
RuleAttribute::Table(table) => Some(*table),
_ => None,
})
.unwrap_or(u32::from(rule.header.table))
}
fn rule_priority(rule: &RuleMessage) -> Option<u32> {
rule.attributes
.iter()
.find_map(|attribute| match attribute {
RuleAttribute::Priority(priority) => Some(*priority),
_ => None,
})
}
/// Whether a rule is one this agent adds for an exit node, including old UID priority.
///
/// Recognised by shape as well as priority: those numbers are not reserved,
/// and a rule somebody else put at one of them must not be removed.
fn is_exit_rule(rule: &RuleMessage) -> bool {
let table = rule_table(rule);
match rule_priority(rule) {
Some(EXIT_MARK_PRIORITY) => {
table == MAIN_TABLE
&& rule
.attributes
.contains(&RuleAttribute::FwMark(UNDERLAY_MARK))
&& rule
.attributes
.contains(&RuleAttribute::FwMask(UNDERLAY_MASK))
}
Some(EXIT_BYPASS_PRIORITY | LEGACY_EXIT_BYPASS_PRIORITY) => {
table == MAIN_TABLE
&& rule
.attributes
.iter()
.any(|attribute| matches!(attribute, RuleAttribute::UidRange(_)))
}
Some(EXIT_SPECIFIC_PRIORITY) => {
table == MAIN_TABLE
&& rule
.attributes
.iter()
.any(|attribute| matches!(attribute, RuleAttribute::SuppressPrefixLen(0)))
}
Some(EXIT_DEFAULT_PRIORITY) => table == EXIT_TABLE,
_ => false,
}
}
fn exit_default_route(index: Option<u32>) -> rtnetlink::packet_route::route::RouteMessage {
let mut builder = RouteMessageBuilder::<Ipv4Addr>::new()
.destination_prefix(Ipv4Addr::UNSPECIFIED, 0)
.table_id(EXIT_TABLE);
if let Some(index) = index {
builder = builder.output_interface(index).scope(RouteScope::Link);
}
builder.build()
}
/// The exit table's IPv6 default: nothing is reachable through it.
///
/// There is no IPv6 through an exit node, and what the ordinary route would do
/// is send it out in the clear. This turns it into an immediate refusal, which
/// is what makes applications fall back to IPv4.
fn exit_blocking_route_v6() -> rtnetlink::packet_route::route::RouteMessage {
RouteMessageBuilder::<std::net::Ipv6Addr>::new()
.destination_prefix(std::net::Ipv6Addr::UNSPECIFIED, 0)
.table_id(EXIT_TABLE)
.kind(rtnetlink::packet_route::route::RouteType::Unreachable)
.build()
}
/// A host with no IPv6 at all has nothing to block, and says so this way.
fn family_unsupported(err: &impl std::fmt::Display) -> bool {
let text = err.to_string();
text.contains("not supported") || text.contains("Address family")
}
/// Removes the exit node's routing rules and its tables. Nothing there is
/// success: it also runs once at startup, for what a crashed run left.
async fn clear_exit_client() -> Result<(), OverlayError> {
with_netlink(|handle| async move { remove_exit_client(&handle).await }).await
}
/// Deletes this agent's exit rules of one address family.
async fn remove_exit_rules(
handle: &rtnetlink::Handle,
version: IpVersion,
) -> Result<(), OverlayError> {
let mut rules = handle.rule().get(version).execute();
let mut ours = Vec::new();
while let Some(rule) = rules
.try_next()
.await
.map_err(|err| OverlayError::Unavailable(format!("cannot read the routing rules: {err}")))?
{
if is_exit_rule(&rule) {
ours.push(rule);
}
}
for rule in ours {
match handle.rule().del(rule).execute().await {
Ok(()) => {}
// Gone already.
Err(err) if err.to_string().contains("No such") => {}
Err(err) => {
return Err(OverlayError::Unavailable(format!(
"cannot remove an exit node routing rule: {err}"
)));
}
}
}
Ok(())
}
async fn remove_exit_client(handle: &rtnetlink::Handle) -> Result<(), OverlayError> {
remove_exit_rules(handle, IpVersion::V4).await?;
match handle.route().del(exit_default_route(None)).execute().await {
Ok(()) => {}
// "No such process": no such route, which is the goal.
Err(err) if err.to_string().contains("No such") => {}
Err(err) => {
return Err(OverlayError::Unavailable(format!(
"cannot remove the exit node default route: {err}"
)));
}
}
match remove_exit_rules(handle, IpVersion::V6).await {
Ok(()) => {}
Err(OverlayError::Unavailable(reason)) if family_unsupported(&reason) => return Ok(()),
Err(err) => return Err(err),
}
match handle.route().del(exit_blocking_route_v6()).execute().await {
Ok(()) => Ok(()),
Err(err) if err.to_string().contains("No such") || family_unsupported(&err) => Ok(()),
Err(err) => Err(OverlayError::Unavailable(format!(
"cannot remove the exit node IPv6 block: {err}"
))),
}
}
/// Sets up the using side of an exit node.
///
/// ```text
/// ip route replace default dev <if> table 0x7473
/// ip rule add priority 5205 fwmark 0x40000000/0x40000000 lookup main
/// ip rule add priority 5206 uidrange <uid>-<uid> lookup main
/// ip rule add priority 5290 lookup main suppress_prefixlength 0
/// ip rule add priority 5300 lookup 0x7473
/// ```
///
/// and the same for IPv6 with `ip -6 route replace unreachable default table
/// 0x7473`, so that IPv6 is refused instead of leaving the ordinary way.
///
/// What an earlier call left is removed first, so repeating it is safe, and
/// the rules go in last, so a failure part way never leaves a rule that
/// sends traffic to a table with no route in it.
async fn set_exit_client(name: &str, uid: u32) -> Result<(), OverlayError> {
with_netlink(|handle| async move {
let index = link_index(&handle, name).await.ok_or_else(|| {
OverlayError::Unavailable(format!(
"interface `{name}` disappeared before the exit node routes could be added"
))
})?;
remove_exit_client(&handle).await?;
let fail = |what: &str, err: rtnetlink::Error| {
OverlayError::Unavailable(format!("cannot add the exit node {what}: {err}"))
};
handle
.route()
.add(exit_default_route(Some(index)))
.replace()
.execute()
.await
.map_err(|err| fail("default route", err))?;
add_mark_bypass(&handle, false).await?;
let mut bypass = handle
.rule()
.add()
.v4()
.table_id(MAIN_TABLE)
.action(RuleAction::ToTable)
.priority(EXIT_BYPASS_PRIORITY);
bypass
.message_mut()
.attributes
.push(RuleAttribute::UidRange(RuleUidRange {
start: uid,
end: uid,
}));
bypass
.execute()
.await
.map_err(|err| fail("rule for the agent's own traffic", err))?;
let mut specific = handle
.rule()
.add()
.v4()
.table_id(MAIN_TABLE)
.action(RuleAction::ToTable)
.priority(EXIT_SPECIFIC_PRIORITY);
specific
.message_mut()
.attributes
.push(RuleAttribute::SuppressPrefixLen(0));
specific
.execute()
.await
.map_err(|err| fail("rule for the specific routes", err))?;
handle
.rule()
.add()
.v4()
.table_id(EXIT_TABLE)
.action(RuleAction::ToTable)
.priority(EXIT_DEFAULT_PRIORITY)
.execute()
.await
.map_err(|err| fail("rule for the default route", err))?;
set_exit_block_v6(&handle, uid).await
})
.await
}
/// The IPv6 half: an unreachable default for everything but the agent's own
/// connections and the specific routes, which keep working.
async fn set_exit_block_v6(handle: &rtnetlink::Handle, uid: u32) -> Result<(), OverlayError> {
let fail = |what: &str, err: rtnetlink::Error| {
OverlayError::Unavailable(format!("cannot add the exit node IPv6 {what}: {err}"))
};
if let Err(err) = handle
.route()
.add(exit_blocking_route_v6())
.replace()
.execute()
.await
{
// Without IPv6 there is nothing to leak.
return if family_unsupported(&err) {
Ok(())
} else {
Err(fail("block", err))
};
}
add_mark_bypass(handle, true).await?;
let mut bypass = handle
.rule()
.add()
.v6()
.table_id(MAIN_TABLE)
.action(RuleAction::ToTable)
.priority(EXIT_BYPASS_PRIORITY);
bypass
.message_mut()
.attributes
.push(RuleAttribute::UidRange(RuleUidRange {
start: uid,
end: uid,
}));
bypass
.execute()
.await
.map_err(|err| fail("rule for the agent's own traffic", err))?;
let mut specific = handle
.rule()
.add()
.v6()
.table_id(MAIN_TABLE)
.action(RuleAction::ToTable)
.priority(EXIT_SPECIFIC_PRIORITY);
specific
.message_mut()
.attributes
.push(RuleAttribute::SuppressPrefixLen(0));
specific
.execute()
.await
.map_err(|err| fail("rule for the specific routes", err))?;
handle
.rule()
.add()
.v6()
.table_id(EXIT_TABLE)
.action(RuleAction::ToTable)
.priority(EXIT_DEFAULT_PRIORITY)
.execute()
.await
.map_err(|err| fail("rule for the default route", err))
}
async fn add_mark_bypass(handle: &rtnetlink::Handle, v6: bool) -> Result<(), OverlayError> {
let mut rule = handle
.rule()
.add()
.table_id(MAIN_TABLE)
.action(RuleAction::ToTable)
.priority(EXIT_MARK_PRIORITY)
.fw_mark(UNDERLAY_MARK);
rule.message_mut().header.family = if v6 {
rtnetlink::packet_route::AddressFamily::Inet6
} else {
rtnetlink::packet_route::AddressFamily::Inet
};
rule.message_mut()
.attributes
.push(RuleAttribute::FwMask(UNDERLAY_MASK));
rule.execute().await.map_err(|err| {
OverlayError::Unavailable(format!("cannot protect marked underlay traffic: {err}"))
})
}
async fn configure_link(configure: &Configure) -> Result<(), OverlayError> {
let name = configure.name.as_str();
with_netlink(|handle| async move {
@@ -737,6 +1116,89 @@ mod tests {
);
}
fn rule(priority: u32, table: u32, extra: Vec<RuleAttribute>) -> RuleMessage {
let mut message = RuleMessage::default();
message.attributes.push(RuleAttribute::Priority(priority));
if table > 255 {
message.attributes.push(RuleAttribute::Table(table));
} else {
message.header.table = table as u8;
}
message.attributes.extend(extra);
message
}
#[test]
fn only_the_three_exit_rules_are_recognised_as_ours() {
let uid = RuleAttribute::UidRange(RuleUidRange { start: 5, end: 5 });
assert!(is_exit_rule(&rule(5280, 254, vec![uid.clone()])));
assert!(is_exit_rule(&rule(
5290,
254,
vec![RuleAttribute::SuppressPrefixLen(0)]
)));
assert!(is_exit_rule(&rule(5300, EXIT_TABLE, vec![])));
// Same priorities, somebody else's rules.
assert!(!is_exit_rule(&rule(5280, 254, vec![])));
assert!(!is_exit_rule(&rule(5290, 254, vec![])));
assert!(!is_exit_rule(&rule(5300, 52, vec![])));
assert!(!is_exit_rule(&rule(5270, 52, vec![])));
assert!(!is_exit_rule(&rule(5280, 100, vec![uid])));
}
#[test]
fn underlay_bypass_recognises_only_our_exact_bit_and_mask() {
let rule = |table, mark, mask| {
let mut message = RuleMessage::default();
message.attributes.extend([
RuleAttribute::Table(table),
RuleAttribute::Priority(EXIT_MARK_PRIORITY),
RuleAttribute::FwMark(mark),
RuleAttribute::FwMask(mask),
]);
message
};
assert!(is_exit_rule(&rule(
MAIN_TABLE,
UNDERLAY_MARK,
UNDERLAY_MASK
)));
assert!(!is_exit_rule(&rule(MAIN_TABLE, 0x80000, 0xff0000)));
assert!(!is_exit_rule(&rule(MAIN_TABLE, UNDERLAY_MARK, u32::MAX)));
assert!(!is_exit_rule(&rule(52, UNDERLAY_MARK, UNDERLAY_MASK)));
}
#[test]
fn the_ipv6_default_of_the_exit_table_refuses_instead_of_routing() {
let route = exit_blocking_route_v6();
assert_eq!(route.header.destination_prefix_length, 0);
let debug = format!("{route:?}");
assert!(debug.contains("Unreachable"), "{debug}");
assert!(debug.contains(&format!("Table({EXIT_TABLE})")), "{debug}");
assert!(!debug.contains("Oif"), "{debug}");
}
#[test]
fn a_host_without_ipv6_is_not_a_failure() {
assert!(family_unsupported(
&"Address family not supported by protocol"
));
assert!(family_unsupported(&"Operation not supported"));
assert!(!family_unsupported(&"Operation not permitted"));
}
#[test]
fn the_default_route_of_the_exit_table_names_the_table_and_the_interface() {
let route = exit_default_route(Some(7));
assert_eq!(route.header.destination_prefix_length, 0);
let debug = format!("{route:?}");
assert!(debug.contains(&format!("Table({EXIT_TABLE})")), "{debug}");
assert!(debug.contains("Oif(7)"), "{debug}");
// Deleting names no interface, so it matches whatever it was added on.
assert!(!format!("{:?}", exit_default_route(None)).contains("Oif"));
}
#[test]
fn an_interface_that_is_not_a_tun_is_reported_foreign() {
let message = LinkMessage::default();
crates/tsunagi/src/overlay/provision/macos.rs → crates/tsunagi-core/src/overlay/provision/macos.rs
View File
File renamed without changes.
crates/tsunagi/src/overlay/provision/mock.rs → crates/tsunagi-core/src/overlay/provision/mock.rs
View File
File renamed without changes.
crates/tsunagi/src/overlay/provision/mod.rs → crates/tsunagi-core/src/overlay/provision/mod.rs
View File
File renamed without changes.
crates/tsunagi/src/overlay/provision/privilege.rs → crates/tsunagi-core/src/overlay/provision/privilege.rs
+4 -6
View File
@@ -15,7 +15,7 @@
//! Grant it with:
//!
//! ```text
//! sudo setcap cap_net_admin,cap_net_bind_service+p /usr/local/bin/tsunagi
//! sudo setcap cap_net_admin,cap_net_bind_service+p /usr/local/bin/tsng
//! ```
//!
//! `+p` rather than `+ep`: with `+p` the capability is permitted but not
@@ -171,7 +171,7 @@ mod linux_impl {
caps::raise(None, CapSet::Effective, Capability::CAP_NET_ADMIN).map_err(|err| {
OverlayError::Unavailable(format!(
"cannot raise CAP_NET_ADMIN: {err}. {}",
Privilege::how_to_grant("tsunagi")
Privilege::how_to_grant("tsng")
))
})?;
Ok(Self { raised: true })
@@ -204,10 +204,8 @@ mod tests {
#[test]
fn the_grant_instructions_name_the_program() {
let text = Privilege::how_to_grant("/usr/local/bin/tsunagi");
assert!(
text.contains("setcap cap_net_admin,cap_net_bind_service+p /usr/local/bin/tsunagi")
);
let text = Privilege::how_to_grant("/usr/local/bin/tsng");
assert!(text.contains("setcap cap_net_admin,cap_net_bind_service+p /usr/local/bin/tsng"));
assert!(text.contains("--no-tun"), "the fallback is offered too");
}
crates/tsunagi/src/overlay/provision/unsupported.rs → crates/tsunagi-core/src/overlay/provision/unsupported.rs
View File
File renamed without changes.
crates/tsunagi/src/overlay/provision/windows.rs → crates/tsunagi-core/src/overlay/provision/windows.rs
View File
File renamed without changes.
crates/tsunagi/src/overlay/router.rs → crates/tsunagi-core/src/overlay/router.rs
+409 -9
View File
@@ -40,9 +40,21 @@ pub struct Route {
pub peer: EndpointId,
}
/// A network's exit-node settings, as far as packets are concerned.
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub struct ExitPolicy {
/// This agent offers itself as an exit node here: packets from members
/// for addresses outside every overlay range are handed to the host.
pub offer: bool,
/// The member all other traffic is sent to, if this device uses one.
pub via: Option<EndpointId>,
}
/// What one network contributes to the table.
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct NetworkRoutes {
/// Exit-node settings of this network.
pub exit: ExitPolicy,
/// Local broadcast policy and authenticated willing recipients.
pub broadcast: BroadcastPolicy,
/// The range this network allocates from, once it has agreed one.
@@ -60,7 +72,7 @@ pub enum RouteError {
#[error(
"network {network} uses {range}, which overlaps {other_range} already in use by \
{other}. One agent has one interface, so an address can belong to only one of \
them: give one network a different range."
them: stop the other network before starting this one."
)]
Overlap {
/// The network being added.
@@ -77,6 +89,7 @@ pub enum RouteError {
/// Address ownership across every network this agent is in.
#[derive(Debug, Default)]
pub struct RoutingTable {
pub(crate) vips: Arc<crate::vip::VipRouter>,
broadcast: ArcSwap<BroadcastTable>,
networks: RwLock<HashMap<NetworkId, NetworkRoutes>>,
}
@@ -134,6 +147,7 @@ impl RoutingTable {
/// Forgets a network.
pub fn remove_network(&self, network: NetworkId) {
self.vips.remove_network(network);
let mut networks = self.write();
networks.remove(&network);
self.broadcast
@@ -153,6 +167,16 @@ impl RoutingTable {
}
}
/// Replaces only a network's exit-node settings, without changing addresses.
///
/// A network that has no entry yet (its range is not agreed) takes them
/// from [`NetworkRoutes::exit`] when it is added.
pub fn set_exit(&self, network: NetworkId, policy: ExitPolicy) {
if let Some(routes) = self.write().get_mut(&network) {
routes.exit = policy;
}
}
/// Recognizes limited and configured subnet broadcasts before unicast lookup.
pub fn is_broadcast(&self, destination: Ipv4Addr) -> bool {
self.broadcast.load().is_destination(destination)
@@ -181,6 +205,10 @@ impl RoutingTable {
///
/// This agent's own address returns `None`: a packet for ourselves does
/// not go over a tunnel, and answering with a peer would send it to one.
///
/// An address nobody holds, outside every overlay range and an ordinary
/// unicast destination, goes to this device's exit node when it has one:
/// that is what sending *all* traffic through a member means.
pub fn route(&self, destination: Ipv4Addr) -> Option<Route> {
let networks = self.read();
for (network, routes) in networks.iter() {
@@ -198,17 +226,92 @@ impl RoutingTable {
});
}
}
None
// Inside an overlay range, an unowned address is nobody's, not the
// internet's.
if !is_internet_destination(destination) || in_any_range(&networks, destination) {
return None;
}
exit_via(&networks).map(|(network, peer)| Route { network, peer })
}
/// Whether a peer may send from a source address.
///
/// A peer speaks only for the address the network agreed it holds. The
/// one exception is this device's exit node: what comes back from the
/// internet carries the internet's addresses, so from *that* peer any
/// source outside the overlay ranges is accepted.
pub fn may_send_from(&self, network: NetworkId, peer: EndpointId, source: Ipv4Addr) -> bool {
self.read().get(&network).is_some_and(|routes| {
routes
.peers
.iter()
.any(|(address, holder)| *address == source && *holder == peer)
})
let networks = self.read();
let Some(routes) = networks.get(&network) else {
return false;
};
if routes
.peers
.iter()
.any(|(address, holder)| *address == source && *holder == peer)
{
return true;
}
routes.exit.via == Some(peer)
&& is_internet_destination(source)
&& !in_any_range(&networks, source)
}
/// Whether this agent, as an exit node of `network`, takes a member's
/// packet for `destination` to the host.
///
/// Only an ordinary unicast address outside every overlay range, so a
/// packet for another member, for this agent or for the local network
/// segment never leaves by this door. The sender's own source address is
/// checked separately, by [`Self::may_send_from`].
pub fn accepts_exit_traffic(&self, network: NetworkId, destination: Ipv4Addr) -> bool {
let networks = self.read();
networks
.get(&network)
.is_some_and(|routes| routes.exit.offer)
&& is_internet_destination(destination)
&& !in_any_range(&networks, destination)
}
/// The network and member this device sends its internet traffic through.
///
/// One default route, so one answer; if several networks claim one (they
/// should not) the lowest network id decides, so two runs agree.
pub fn exit_via(&self) -> Option<(NetworkId, EndpointId)> {
exit_via(&self.read())
}
/// Networks that offer this agent as an exit node, with the range each
/// agreed. A network that has not agreed one yet has nothing to
/// masquerade and is left out.
pub fn exit_offers(&self) -> Vec<(NetworkId, Ipv4Range)> {
let mut offers: Vec<(NetworkId, Ipv4Range)> = self
.read()
.iter()
.filter(|(_, routes)| routes.exit.offer)
.filter_map(|(network, routes)| Some((*network, routes.range?)))
.collect();
offers.sort_by_key(|(network, _)| *network);
offers
}
/// The range of every network in the table that has agreed one, sorted.
pub fn overlay_ranges(&self) -> Vec<Ipv4Range> {
let mut ranges: Vec<Ipv4Range> = self
.read()
.values()
.filter_map(|routes| routes.range)
.collect();
ranges.sort_by_key(|range| (range.base, range.prefix_len));
ranges.dedup();
ranges
}
/// One network's exit-node settings and its range, if it is in the table.
pub fn exit_settings(&self, network: NetworkId) -> Option<(ExitPolicy, Option<Ipv4Range>)> {
self.read()
.get(&network)
.map(|routes| (routes.exit, routes.range))
}
/// Whether an ordinary packet terminates on this host in this network.
@@ -264,6 +367,43 @@ impl RoutingTable {
})
}
/// All active networks whose reserved ranges intersect this range.
pub fn overlapping_networks(
&self,
network: NetworkId,
range: Ipv4Range,
) -> Vec<(NetworkId, Ipv4Range)> {
let mut conflicts: Vec<_> = self
.read()
.iter()
.filter_map(|(other, routes)| {
let other_range = routes.range?;
(*other != network && ranges_overlap(range, other_range))
.then_some((*other, other_range))
})
.collect();
conflicts.sort_by_key(|(id, _)| *id);
conflicts
}
/// Current signed allocations used for encrypted data delivery checks.
pub(crate) fn probe_addresses(
&self,
network: NetworkId,
peer: EndpointId,
) -> Option<(Ipv4Addr, Ipv4Addr)> {
let table = self.read();
let routes = table.get(&network)?;
Some((
routes.local?,
routes
.peers
.iter()
.find(|(_, holder)| *holder == peer)
.map(|(address, _)| *address)?,
))
}
/// How many networks the table covers.
pub fn len(&self) -> usize {
self.read().len()
@@ -275,6 +415,31 @@ impl RoutingTable {
}
}
/// Whether an address is an ordinary unicast one: not this host, not a
/// multicast or broadcast group, not link-local, not unspecified.
fn is_internet_destination(address: Ipv4Addr) -> bool {
!(address.is_unspecified()
|| address.is_loopback()
|| address.is_multicast()
|| address.is_broadcast()
|| address.is_link_local())
}
/// Whether an address lies in any network's range.
fn in_any_range(networks: &HashMap<NetworkId, NetworkRoutes>, address: Ipv4Addr) -> bool {
networks
.values()
.any(|routes| routes.range.is_some_and(|range| range.contains(address)))
}
/// The exit node in use, if any, from the lowest network that names one.
fn exit_via(networks: &HashMap<NetworkId, NetworkRoutes>) -> Option<(NetworkId, EndpointId)> {
networks
.iter()
.filter_map(|(network, routes)| Some((*network, routes.exit.via?)))
.min_by_key(|(network, _)| *network)
}
/// Whether two ranges share any address.
fn ranges_overlap(one: Ipv4Range, other: Ipv4Range) -> bool {
// A range contains the other's base, or the other way round. With
@@ -308,6 +473,7 @@ mod tests {
fn routes() -> NetworkRoutes {
NetworkRoutes {
exit: Default::default(),
broadcast: Default::default(),
range: Some("10.13.37.0/24".parse().unwrap()),
local: Some(addr(1)),
@@ -383,7 +549,7 @@ mod tests {
let err = table.set_network(second, routes()).unwrap_err();
let RouteError::Overlap { other, .. } = err;
assert_eq!(other, first);
assert!(err.to_string().contains("different range"), "{err}");
assert!(err.to_string().contains("stop the other network"), "{err}");
assert_eq!(table.len(), 1, "the overlapping network is not added");
// A range of its own is fine.
@@ -391,6 +557,7 @@ mod tests {
.set_network(
second,
NetworkRoutes {
exit: Default::default(),
broadcast: Default::default(),
range: Some("10.99.0.0/16".parse().unwrap()),
local: Some(Ipv4Addr::new(10, 99, 0, 1)),
@@ -420,6 +587,7 @@ mod tests {
if a < b { (a, b) } else { (b, a) }
};
let mk = |enabled: bool, base: [u8; 4], local: Option<Ipv4Addr>| NetworkRoutes {
exit: Default::default(),
broadcast: BroadcastPolicy {
enabled,
..Default::default()
@@ -495,6 +663,7 @@ mod tests {
.set_network(
network("wide"),
NetworkRoutes {
exit: Default::default(),
broadcast: Default::default(),
range: Some("10.0.0.0/8".parse().unwrap()),
..Default::default()
@@ -507,6 +676,7 @@ mod tests {
.set_network(
network("narrow"),
NetworkRoutes {
exit: Default::default(),
broadcast: Default::default(),
range: Some("10.13.37.0/24".parse().unwrap()),
..Default::default()
@@ -549,6 +719,7 @@ mod tests {
.set_network(
network("two"),
NetworkRoutes {
exit: Default::default(),
broadcast: Default::default(),
range: Some("10.99.0.0/16".parse().unwrap()),
local: Some(Ipv4Addr::new(10, 99, 0, 1)),
@@ -573,4 +744,233 @@ mod tests {
assert_eq!(table.route(addr(2)), None);
assert_eq!(table.len(), 1, "the network is known, it just has nothing");
}
fn with_exit(offer: bool, via: Option<EndpointId>) -> NetworkRoutes {
NetworkRoutes {
exit: ExitPolicy { offer, via },
..routes()
}
}
const INTERNET: Ipv4Addr = Ipv4Addr::new(93, 184, 216, 34);
#[test]
fn without_an_exit_node_the_internet_has_nowhere_to_go() {
let table = RoutingTable::new();
table.set_network(network("plain"), routes()).unwrap();
assert_eq!(table.route(INTERNET), None);
assert_eq!(table.exit_via(), None);
}
#[test]
fn with_an_exit_node_every_ordinary_destination_goes_to_it() {
let table = RoutingTable::new();
let id = network("client");
table
.set_network(id, with_exit(false, Some(peer(2))))
.unwrap();
let exit = Route {
network: id,
peer: peer(2),
};
assert_eq!(table.route(INTERNET), Some(exit));
assert_eq!(table.route(Ipv4Addr::new(192, 168, 1, 10)), Some(exit));
assert_eq!(table.exit_via(), Some((id, peer(2))));
// A member's own address still goes to that member.
assert_eq!(
table.route(addr(3)),
Some(Route {
network: id,
peer: peer(3)
})
);
}
#[test]
fn the_exit_node_is_never_the_answer_for_what_is_not_the_internet() {
let table = RoutingTable::new();
table
.set_network(network("client"), with_exit(false, Some(peer(2))))
.unwrap();
// Ourselves, a free address inside the overlay range, and everything
// that is not an ordinary unicast destination.
for destination in [
addr(1),
addr(200),
Ipv4Addr::new(127, 0, 0, 1),
Ipv4Addr::new(224, 0, 0, 251),
Ipv4Addr::new(239, 255, 255, 250),
Ipv4Addr::BROADCAST,
Ipv4Addr::UNSPECIFIED,
Ipv4Addr::new(169, 254, 1, 1),
] {
assert_eq!(table.route(destination), None, "{destination}");
}
}
#[test]
fn another_networks_range_is_not_the_internet_either() {
let table = RoutingTable::new();
table
.set_network(network("client"), with_exit(false, Some(peer(2))))
.unwrap();
table
.set_network(
network("other"),
NetworkRoutes {
range: Some("10.99.0.0/16".parse().unwrap()),
local: Some(Ipv4Addr::new(10, 99, 0, 1)),
..Default::default()
},
)
.unwrap();
assert_eq!(table.route(Ipv4Addr::new(10, 99, 7, 7)), None);
}
#[test]
fn two_exit_choices_resolve_to_the_lowest_network_id() {
let table = RoutingTable::new();
let (a, b) = (network("one"), network("two"));
table
.set_network(a, with_exit(false, Some(peer(2))))
.unwrap();
table
.set_network(
b,
NetworkRoutes {
exit: ExitPolicy {
offer: false,
via: Some(peer(9)),
},
range: Some("10.99.0.0/16".parse().unwrap()),
..Default::default()
},
)
.unwrap();
let (chosen, _) = table.exit_via().unwrap();
assert_eq!(chosen, a.min(b));
assert_eq!(table.route(INTERNET).unwrap().network, a.min(b));
}
#[test]
fn the_exit_peer_may_send_from_any_internet_address_but_nobody_else_may() {
let table = RoutingTable::new();
let id = network("client");
table
.set_network(id, with_exit(false, Some(peer(2))))
.unwrap();
// What comes back from the internet carries the internet's addresses.
assert!(table.may_send_from(id, peer(2), INTERNET));
// Another member still speaks only for its own address.
assert!(!table.may_send_from(id, peer(3), INTERNET));
assert!(table.may_send_from(id, peer(3), addr(3)));
// The exit peer cannot pass itself off as somebody else in the
// overlay, nor as this agent, nor as something that is not a source.
assert!(!table.may_send_from(id, peer(2), addr(3)));
assert!(!table.may_send_from(id, peer(2), addr(1)));
assert!(!table.may_send_from(id, peer(2), addr(200)));
for source in [
Ipv4Addr::LOCALHOST,
Ipv4Addr::UNSPECIFIED,
Ipv4Addr::new(224, 0, 0, 1),
Ipv4Addr::BROADCAST,
] {
assert!(!table.may_send_from(id, peer(2), source), "{source}");
}
// Without choosing an exit node nobody gets the exception.
let plain = RoutingTable::new();
plain.set_network(id, routes()).unwrap();
assert!(!plain.may_send_from(id, peer(2), INTERNET));
}
#[test]
fn an_exit_node_takes_a_members_internet_packets_only_while_it_offers() {
let table = RoutingTable::new();
let id = network("server");
table.set_network(id, with_exit(false, None)).unwrap();
assert!(!table.accepts_exit_traffic(id, INTERNET));
table.set_exit(
id,
ExitPolicy {
offer: true,
via: None,
},
);
assert!(table.accepts_exit_traffic(id, INTERNET));
assert!(table.accepts_exit_traffic(id, Ipv4Addr::new(192, 168, 1, 10)));
// Never for another member, for this agent, a free overlay address,
// or anything that is not an ordinary unicast destination.
for destination in [
addr(1),
addr(2),
addr(200),
Ipv4Addr::LOCALHOST,
Ipv4Addr::new(224, 0, 0, 251),
Ipv4Addr::BROADCAST,
Ipv4Addr::UNSPECIFIED,
] {
assert!(
!table.accepts_exit_traffic(id, destination),
"{destination}"
);
}
// An unknown network offers nothing.
assert!(!table.accepts_exit_traffic(network("elsewhere"), INTERNET));
table.set_exit(id, ExitPolicy::default());
assert!(!table.accepts_exit_traffic(id, INTERNET));
}
#[test]
fn offers_are_listed_by_network_with_the_range_each_agreed() {
let table = RoutingTable::new();
let (a, b) = (network("a"), network("b"));
table.set_network(a, with_exit(true, None)).unwrap();
table
.set_network(
b,
NetworkRoutes {
exit: ExitPolicy {
offer: true,
via: None,
},
// No range agreed yet: nothing to masquerade.
..Default::default()
},
)
.unwrap();
assert_eq!(
table.exit_offers(),
vec![(a, "10.13.37.0/24".parse().unwrap())]
);
assert_eq!(
table.exit_settings(a),
Some((
ExitPolicy {
offer: true,
via: None
},
Some("10.13.37.0/24".parse().unwrap())
))
);
assert_eq!(table.exit_settings(network("none")), None);
}
#[test]
fn setting_exit_for_a_network_with_no_entry_is_remembered_by_the_caller() {
// The runtime keeps the policy and passes it with the network when
// its range is agreed; a setter for a network the table does not know
// yet changes nothing.
let table = RoutingTable::new();
table.set_exit(
network("later"),
ExitPolicy {
offer: true,
via: None,
},
);
assert!(table.is_empty());
}
}
crates/tsunagi/src/overlay/tun.rs → crates/tsunagi-core/src/overlay/tun.rs
+5 -5
View File
@@ -483,15 +483,15 @@ mod system {
if denied {
let hint = if cfg!(target_os = "windows") {
"Open PowerShell or Command Prompt with 'Run as administrator' and \
start `tsunagi up` there."
start `tsng up` there."
} else if cfg!(target_os = "linux") {
"Start `tsunagi up` as root or grant CAP_NET_ADMIN with \
"Start `tsng up` as root or grant CAP_NET_ADMIN with \
`sudo setcap cap_net_admin,cap_net_bind_service+p /path/to/tsunagi`."
} else if cfg!(target_os = "macos") {
"Creating a utun interface needs root on macOS; start `tsunagi up` \
"Creating a utun interface needs root on macOS; start `tsng up` \
with `sudo`, or install it as a root LaunchDaemon."
} else {
"Start `tsunagi up` with the privileges required to create a TUN interface."
"Start `tsng up` with the privileges required to create a TUN interface."
};
return OverlayError::Unavailable(format!(
"permission denied while creating the TUN interface `{name}`. \
@@ -539,7 +539,7 @@ mod system {
let message = open_error("tsun0", error).to_string();
assert!(message.contains("permission denied"), "{message}");
assert!(message.contains("tsun0"), "{message}");
assert!(message.contains("tsunagi up"), "{message}");
assert!(message.contains("tsng up"), "{message}");
#[cfg(target_os = "windows")]
assert!(message.contains("Run as administrator"), "{message}");
#[cfg(target_os = "linux")]
crates/tsunagi/src/proto/frame.rs → crates/tsunagi-core/src/proto/frame.rs
View File
File renamed without changes.
crates/tsunagi/src/proto/handshake.rs → crates/tsunagi-core/src/proto/handshake.rs
View File
File renamed without changes.
crates/tsunagi/src/proto/message.rs → crates/tsunagi-core/src/proto/message.rs
+76 -5
View File
@@ -20,7 +20,7 @@ use crate::error::ProtocolError;
/// The version in the ALPN is the wire-compatibility version of the control
/// protocol. It is independent of the network identity scheme version, so
/// bumping it must not change any existing [`crate::NetworkId`].
pub const ALPN: &[u8] = b"tsunagi/ctrl/3";
pub const ALPN: &[u8] = b"tsunagi/ctrl/8";
/// ALPN of the tsunagi data plane.
///
@@ -30,9 +30,11 @@ pub const ALPN: &[u8] = b"tsunagi/ctrl/3";
/// transport underneath can be replaced without touching the control protocol.
///
/// Version 4 adds source, destination, hop limit and stable flow id above
/// transport fragmentation. All members must upgrade together. Persistent
/// identities and network configuration do not change.
pub const DATA_ALPN: &[u8] = b"tsunagi/data/4";
/// transport fragmentation. Version 5 adds the end-to-end protocol to that
/// envelope, so a tunnel no longer depends on the kind of link under it, and
/// a first-claimed time to a signed hostname claim. All members must upgrade
/// together. Persistent identities and network configuration do not change.
pub const DATA_ALPN: &[u8] = b"tsunagi/data/5";
/// Largest plugin protocol identifier accepted when opening a data channel.
pub const MAX_DATA_PROTOCOL_LEN: usize = 32;
@@ -41,7 +43,9 @@ pub const MAX_DATA_PROTOCOL_LEN: usize = 32;
pub const MAX_SIGNATURE_LEN: usize = 64;
/// Control protocol version carried inside the handshake.
pub const PROTOCOL_VERSION: u16 = 3;
pub const PROTOCOL_VERSION: u16 = 8;
pub use crate::dataplane::transport::{MAX_TRANSPORT_PAYLOAD, TransportPayload};
/// First message of the handshake, sent by the initiator.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
@@ -75,6 +79,11 @@ pub struct AuthProof {
pub struct Announcement {
/// Local opt-in to receive LAN broadcasts in this network.
pub broadcast: bool,
/// This agent offers itself as an exit node in this network: members may
/// send all their internet traffic through it. Off unless its owner
/// turned it on, and only said once the rules that make it work are in
/// place.
pub exit_node: bool,
/// Human-readable hostname. A mutable binding, not an identity.
pub hostname: String,
/// Announced IP plugin capabilities. Opaque to the core.
@@ -107,6 +116,13 @@ pub struct DataOpenAck {
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[non_exhaustive]
pub enum ControlMessage {
/// Private transport negotiation; never persisted or rebroadcast.
Transport {
/// Transport protocol id; the payload stays opaque to the core.
protocol: String,
/// Secret negotiation data, redacted in diagnostics.
payload: TransportPayload,
},
/// Hostname and capability announcement.
Announce(Announcement),
/// A small request used to verify that the exchange works.
@@ -165,6 +181,16 @@ pub enum ControlMessage {
/// Short free-text reason.
reason: String,
},
/// One complete signed VIP field snapshot, merged atomically.
Vips {
/// Signed registers bounded by the per-VIP field limit.
entries: Vec<crate::vip::VipEntry>,
},
/// Signed cleanup markers, each proving the retired claim version.
Retirements {
/// Bounded by the per-message signed-state limit.
markers: Vec<crate::state::SignedRetirement>,
},
}
/// Somewhere a member has been seen.
@@ -257,6 +283,48 @@ pub fn validate_capability(
/// network, the other networks or the agent.
pub fn validate(message: &ControlMessage, limits: &Limits) -> Result<(), ProtocolError> {
match message {
ControlMessage::Vips { entries } => {
check_len("vip.fields", entries.len(), limits.max_vip_fields)?;
for e in entries {
check_len("vip.id", e.vip.len(), 32)?;
check_len("vip.key", e.key.len(), 160)?;
check_len("vip.value", e.value.as_ref().map_or(0, Vec::len), 256)?;
check_len("vip.signature", e.signature.len(), 64)?;
}
}
ControlMessage::Retirements { markers } => {
check_len("retirements", markers.len(), limits.max_state_records)?;
for marker in markers {
check_len(
"retirement.signature",
marker.signature.len(),
MAX_SIGNATURE_LEN,
)?;
check_len(
"retirement.target.signature",
marker.target.signature.len(),
MAX_SIGNATURE_LEN,
)?;
if let Some(name) = marker.target.body.hostname() {
check_len(
"retirement.target.hostname",
name.len(),
limits.max_hostname_len,
)?;
}
}
}
ControlMessage::Transport { protocol, payload } => {
if protocol.is_empty() {
return Err(ProtocolError::Malformed("empty transport protocol"));
}
check_len("transport.protocol", protocol.len(), MAX_PROTOCOL_ID_LEN)?;
check_len(
"transport.payload",
payload.as_bytes().len(),
limits.max_transport_payload_len.min(MAX_TRANSPORT_PAYLOAD),
)?;
}
ControlMessage::Peers { peers } => {
check_len("peers", peers.len(), limits.max_state_records)?;
for hint in peers {
@@ -313,10 +381,13 @@ pub fn validate(message: &ControlMessage, limits: &Limits) -> Result<(), Protoco
/// A short, stable label for a message kind, for metrics and diagnostics.
pub fn kind(message: &ControlMessage) -> &'static str {
match message {
ControlMessage::Transport { .. } => "transport",
ControlMessage::Announce(_) => "announce",
ControlMessage::Ping { .. } => "ping",
ControlMessage::Pong { .. } => "pong",
ControlMessage::State { .. } => "state",
ControlMessage::Retirements { .. } => "retirements",
ControlMessage::Vips { .. } => "vips",
ControlMessage::Reachable { .. } => "reachable",
ControlMessage::Peers { .. } => "peers",
ControlMessage::Bye { .. } => "bye",
crates/tsunagi/src/proto/mod.rs → crates/tsunagi-core/src/proto/mod.rs
View File
File renamed without changes.
crates/tsunagi/src/state/allocator.rs → crates/tsunagi-core/src/state/allocator.rs
View File
File renamed without changes.
crates/tsunagi/src/state/mod.rs → crates/tsunagi-core/src/state/mod.rs
+309 -34
View File
@@ -28,16 +28,25 @@
//! replica that has the record needs nothing else to interpret it, and
//! recovery never depends on replaying a chain from the beginning.
//!
//! Any member may also sign a separate retirement of another author's proved
//! claim version. It suppresses stale reservations without changing that
//! author's record, and a newer claim remains valid.
//!
//! # What this slice does not do yet
//!
//! Compaction, revocation of a whole author, and snapshots covering more than
//! one record type. See `docs/sync-model.md` for the shape those take.
//! Compaction and permanent revocation of a whole author. See
//! `docs/sync-model.md` for the implemented cleanup semantics.
pub mod allocator;
/// Signed shared virtual service state and its merge rules.
pub mod vip;
use std::collections::HashMap;
use std::net::Ipv4Addr;
mod retirement;
pub use retirement::{MAX_RETIREMENT_LEN, SignedRetirement};
use iroh::{EndpointId, SecretKey, Signature};
use serde::{Deserialize, Serialize};
@@ -109,12 +118,8 @@ impl std::str::FromStr for Ipv4Range {
}
}
/// The IPv4 overlay range used unless something else is configured or agreed.
///
/// A small, specific `/24`: memorable, and far less likely to overlap a
/// network the machine is already on than taking a whole `/8` or `/10` would
/// be. Because addresses are allocated rather than derived, 254 of them is
/// plenty for the size of network this is for.
/// The legacy default range, retained for existing networks and explicit use.
/// New networks use [`derived_ipv4_range`] by default.
pub const DEFAULT_IPV4_RANGE: Ipv4Range = Ipv4Range {
base: Ipv4Addr::new(10, 13, 37, 0),
prefix_len: 24,
@@ -122,15 +127,10 @@ pub const DEFAULT_IPV4_RANGE: Ipv4Range = Ipv4Range {
/// A range derived from a network's own identity.
///
/// One agent has one interface, so two of its networks cannot both use the
/// configured range. The second needs another one — and it cannot be picked
/// locally, because every member has to arrive at the same answer without
/// being told. So it comes from the network id, which every member already
/// has and nobody can choose: the same network derives the same range on
/// every device.
///
/// Inside 10/8 like the default, and never equal to it, so the first
/// network keeps what it has always had.
/// Every member independently derives the same pseudorandom `/24` in 10/8,
/// including isolated first members. Stored signed ranges take precedence.
/// The derivation is kept stable for networks already using it; the legacy
/// default is excluded. Distinct network ids can still produce collisions.
pub fn derived_ipv4_range(network: NetworkId) -> Ipv4Range {
let bytes = network.as_bytes();
let second = bytes[0];
@@ -150,7 +150,7 @@ pub fn derived_ipv4_range(network: NetworkId) -> Ipv4Range {
}
/// Frozen domain separator for the bytes a record signature covers.
pub const RECORD_DOMAIN: &str = "tsunagi-signed-record-v2";
pub const RECORD_DOMAIN: &str = "tsunagi-signed-record-v3";
/// Longest hostname a record may carry.
///
@@ -207,6 +207,18 @@ pub enum RecordBody {
range: Option<Ipv4Range>,
/// The name this author answers to.
hostname: Option<String>,
/// When this author first claimed that name, in milliseconds since
/// the Unix epoch, kept across every later version that keeps the
/// name. It is what makes a name belong to whoever claimed it first
/// rather than to whoever happens to sort lowest, so a member that
/// arrives later with the same name cannot take it over.
///
/// The author's own word: a member with a wrong clock can claim an
/// earlier time. That costs it nothing it could not do by other
/// means, since anybody who knows the network secret is a member,
/// and it cannot take an address or reach anything it was not
/// already entitled to.
hostname_since: Option<u64>,
},
/// This author gives up everything it claimed.
///
@@ -243,6 +255,14 @@ impl RecordBody {
}
}
/// When the author first claimed the hostname, if it has one.
pub fn hostname_since(&self) -> Option<u64> {
match self {
RecordBody::Claim { hostname_since, .. } => *hostname_since,
RecordBody::Release => None,
}
}
fn canonical(&self) -> Vec<u8> {
let mut out = Vec::with_capacity(96);
match self {
@@ -250,6 +270,7 @@ impl RecordBody {
address,
range,
hostname,
hostname_since,
} => {
push_lp(&mut out, b"claim");
push_opt(
@@ -270,6 +291,13 @@ impl RecordBody {
.as_deref(),
);
push_opt(&mut out, hostname.as_deref().map(str::as_bytes));
push_opt(
&mut out,
hostname_since
.map(|since| since.to_be_bytes())
.as_ref()
.map(|bytes| &bytes[..]),
);
}
RecordBody::Release => {
push_lp(&mut out, b"release");
@@ -391,8 +419,12 @@ impl SignedRecord {
address,
range,
hostname,
hostname_since,
} = &self.body
{
if hostname.is_none() && hostname_since.is_some() {
return Err(StateError::Malformed("a time was claimed for no hostname"));
}
// Bounds before anything is believed, because all of this came
// off the network.
if let Some(range) = range {
@@ -465,6 +497,7 @@ pub enum Merged {
#[derive(Debug, Clone, Default)]
pub struct StateSet {
records: HashMap<EndpointId, SignedRecord>,
retirements: HashMap<EndpointId, SignedRetirement>,
}
impl StateSet {
@@ -477,6 +510,7 @@ impl StateSet {
pub fn from_verified(records: impl IntoIterator<Item = (EndpointId, SignedRecord)>) -> Self {
Self {
records: records.into_iter().collect(),
retirements: HashMap::new(),
}
}
@@ -505,6 +539,52 @@ impl StateSet {
.collect()
}
/// Visible records, excluding versions retired by authenticated members.
pub fn active_records(&self) -> Vec<SignedRecord> {
self.records()
.into_iter()
.filter(|r| r.author_id().is_ok_and(|id| !self.is_retired(&id)))
.collect()
}
/// Whether the current known version is at or below a proved retirement.
pub fn is_retired(&self, author: &EndpointId) -> bool {
self.retirements
.get(author)
.is_some_and(|m| self.version_of(author) <= m.target.version)
}
/// Strongest retained cleanup proof for an identity.
pub fn retirement_of(&self, target: &EndpointId) -> Option<&SignedRetirement> {
self.retirements.get(target)
}
/// Durable markers in target-id order, including markers surpassed by a fresh claim.
pub fn retirements(&self) -> Vec<SignedRetirement> {
let mut markers: Vec<_> = self.retirements.values().cloned().collect();
markers.sort_by_key(|m| m.target.author);
markers
}
/// Accepts any member's signed cleanup after checking both signatures.
pub fn merge_retirement(
&mut self,
network: NetworkId,
marker: SignedRetirement,
max: usize,
) -> Result<bool, StateError> {
let target = marker.verify(network)?;
if let Some(old) = self.retirements.get(&target) {
if !marker.supersedes(old) {
return Ok(false);
}
} else if self.retirements.len() >= max {
return Err(StateError::Malformed("retirement capacity reached"));
}
self.retirements.insert(target, marker);
Ok(true)
}
/// Merges one record, verifying it first.
///
/// Merging is into the existing set, never a wholesale replacement, and an
@@ -568,6 +648,9 @@ impl StateSet {
pub fn address_holders(&self) -> HashMap<Ipv4Addr, EndpointId> {
let mut holders: HashMap<Ipv4Addr, EndpointId> = HashMap::new();
for (author, record) in &self.records {
if self.is_retired(author) {
continue;
}
let Some(address) = record.body.claimed_address() else {
continue;
};
@@ -585,25 +668,36 @@ impl StateSet {
/// Who currently holds each claimed hostname.
///
/// The same rule as addresses, for the same reason: a name is owned, two
/// members may claim one, and every replica has to reach the same answer
/// about who has it without asking anybody.
/// A name belongs to the member that claimed it first, and the others
/// that claim it keep working by address but are not found by that name.
/// "First" is the time each author signed into its own claim, with the
/// lower id deciding a tie and a claim with no time counting as last, so
/// every replica reaches the same answer about who has it without
/// asking anybody. It is a pure function of the records: nothing depends
/// on the order they arrived in, so there is no race to lose.
pub fn hostname_holders(&self) -> HashMap<String, EndpointId> {
let mut holders: HashMap<String, EndpointId> = HashMap::new();
let mut holders: HashMap<String, (u64, EndpointId)> = HashMap::new();
for (author, record) in &self.records {
if self.is_retired(author) {
continue;
}
let Some(hostname) = record.body.hostname() else {
continue;
};
let claim = (record.body.hostname_since().unwrap_or(u64::MAX), *author);
holders
.entry(hostname.to_string())
.and_modify(|held| {
if author.as_bytes() < held.as_bytes() {
*held = *author;
if (claim.0, claim.1.as_bytes()) < (held.0, held.1.as_bytes()) {
*held = claim;
}
})
.or_insert(*author);
.or_insert(claim);
}
holders
.into_iter()
.map(|(hostname, (_, author))| (hostname, author))
.collect()
}
/// The hostname an author holds, if it holds one uncontested.
@@ -612,6 +706,22 @@ impl StateSet {
(self.hostname_holders().get(hostname) == Some(author)).then_some(hostname)
}
/// The name an author claimed and the member that holds it instead, when
/// somebody who claimed it earlier does.
pub fn hostname_taken(&self, author: &EndpointId) -> Option<(&str, EndpointId)> {
if self.is_retired(author) {
return None;
}
let hostname = self.records.get(author)?.body.hostname()?;
let holder = *self.hostname_holders().get(hostname)?;
(holder != *author).then_some((hostname, holder))
}
/// When an author first claimed the name it holds now, if it has one.
pub fn hostname_since_of(&self, author: &EndpointId) -> Option<u64> {
self.records.get(author)?.body.hostname_since()
}
/// The address an author holds, if it holds one uncontested.
pub fn address_of(&self, author: &EndpointId) -> Option<Ipv4Addr> {
let address = self.records.get(author)?.body.claimed_address()?;
@@ -620,21 +730,44 @@ impl StateSet {
/// The range the network settled on, if anybody has said.
///
/// When claims disagree, the one from the lowest author id wins, so every
/// replica reaches the same answer. A participant joining later therefore
/// adopts the range already in use instead of imposing its own.
/// A legacy or explicit range takes precedence over the derived default,
/// so a fresh member's default cannot renumber an established network.
/// Other disagreements use the lowest author id, independently of arrival
/// order. The signed record format and network identity remain unchanged.
pub fn agreed_range(&self) -> Option<Ipv4Range> {
let mut authors: Vec<&EndpointId> = self.records.keys().collect();
authors.sort_by_key(|author| *author.as_bytes());
authors
.into_iter()
.find_map(|author| self.records.get(author)?.body.range())
self.records
.iter()
.chain(
self.retirements
.iter()
.filter(|(author, _)| !self.records.contains_key(*author))
.map(|(author, marker)| (author, &marker.target)),
)
.filter_map(|(author, record)| {
let range = record.body.range()?;
let derived = derived_ipv4_range(NetworkId::from_bytes(record.network));
Some(((range == derived, *author.as_bytes()), range))
})
.min_by_key(|(priority, _)| *priority)
.map(|(_, range)| range)
}
/// The highest version this author has published, as far as is known.
pub fn version_of(&self, author: &EndpointId) -> u64 {
self.records.get(author).map_or(0, |record| record.version)
}
/// Highest witnessed author counter across claims and authored cleanup.
/// This differs from the claim version used to decide retirement.
pub fn authored_version(&self, author: &EndpointId) -> u64 {
self.retirements
.values()
.filter(|m| m.author == *author.as_bytes())
.map(|m| m.version)
.chain(std::iter::once(self.version_of(author)))
.max()
.unwrap_or(0)
}
}
#[cfg(test)]
@@ -661,6 +794,7 @@ mod tests {
address: Some(address.parse().unwrap()),
range: Some(range()),
hostname: None,
hostname_since: None,
}
}
@@ -720,6 +854,7 @@ mod tests {
address: Some("10.99.0.1".parse().unwrap()),
range: Some(range()),
hostname: None,
hostname_since: None,
},
);
assert!(matches!(outside.verify(id), Err(StateError::Malformed(_))));
@@ -735,6 +870,7 @@ mod tests {
prefix_len: 31,
}),
hostname: None,
hostname_since: None,
},
);
assert!(matches!(no_hosts.verify(id), Err(StateError::Malformed(_))));
@@ -776,6 +912,7 @@ mod tests {
address: None,
range: None,
hostname: Some(bad.to_string()),
hostname_since: None,
},
);
assert!(
@@ -797,6 +934,7 @@ mod tests {
address: Some("10.13.37.5".parse().unwrap()),
range: None,
hostname: None,
hostname_since: None,
},
);
assert!(matches!(record.verify(id), Err(StateError::Malformed(_))));
@@ -818,6 +956,7 @@ mod tests {
address: None,
range: None,
hostname: Some("music".into()),
hostname_since: None,
},
)
};
@@ -860,6 +999,7 @@ mod tests {
address: None,
range: None,
hostname: Some(name.to_string()),
hostname_since: None,
},
)
};
@@ -879,6 +1019,109 @@ mod tests {
assert_eq!(set.hostname_of(&secret.public()), Some("new"));
}
fn named_at(
secret: &SecretKey,
id: NetworkId,
version: u64,
name: &str,
since: u64,
) -> SignedRecord {
SignedRecord::sign(
secret,
id,
version,
RecordBody::Claim {
address: None,
range: None,
hostname: Some(name.into()),
hostname_since: Some(since),
},
)
}
#[test]
fn a_name_belongs_to_whoever_claimed_it_first_whatever_their_ids_or_arrival_order() {
let id = network("naming");
let mut keys = [SecretKey::generate(), SecretKey::generate()];
keys.sort_by_key(|key| *key.public().as_bytes());
let [low, high] = keys;
// The member with the higher id claimed first, so it holds the name,
// and the lower id does not take it back by sorting first.
let early = named_at(&high, id, 1, "music", 1_000);
let late = named_at(&low, id, 1, "music", 2_000);
for order in [[&early, &late], [&late, &early]] {
let mut set = StateSet::new();
for record in order {
set.merge(id, (*record).clone()).unwrap();
}
assert_eq!(set.hostname_of(&high.public()), Some("music"));
assert_eq!(set.hostname_of(&low.public()), None);
assert_eq!(
set.hostname_taken(&low.public()),
Some(("music", high.public()))
);
assert_eq!(set.hostname_taken(&high.public()), None);
}
// At the same instant the lower id decides, the same way everywhere.
let mut set = StateSet::new();
set.merge(id, named_at(&high, id, 1, "music", 5)).unwrap();
set.merge(id, named_at(&low, id, 1, "music", 5)).unwrap();
assert_eq!(set.hostname_of(&low.public()), Some("music"));
assert_eq!(set.hostname_of(&high.public()), None);
}
#[test]
fn a_name_that_loses_is_found_again_once_the_holder_lets_it_go() {
let id = network("naming");
let first = SecretKey::generate();
let second = SecretKey::generate();
let mut set = StateSet::new();
set.merge(id, named_at(&first, id, 1, "music", 1)).unwrap();
set.merge(id, named_at(&second, id, 1, "music", 2)).unwrap();
assert_eq!(set.hostname_of(&second.public()), None);
// Renaming starts the clock afresh, so it cannot be used to jump the
// queue for a name somebody else already holds.
set.merge(id, named_at(&first, id, 2, "other", 3)).unwrap();
assert_eq!(set.hostname_of(&second.public()), Some("music"));
set.merge(id, named_at(&first, id, 3, "music", 4)).unwrap();
assert_eq!(set.hostname_of(&second.public()), Some("music"));
assert_eq!(set.hostname_of(&first.public()), None);
// And a release hands it on.
set.merge(id, SignedRecord::sign(&second, id, 2, RecordBody::Release))
.unwrap();
assert_eq!(set.hostname_of(&first.public()), Some("music"));
}
#[test]
fn a_time_with_no_name_is_not_a_valid_claim() {
let id = network("naming");
let secret = SecretKey::generate();
let record = SignedRecord::sign(
&secret,
id,
1,
RecordBody::Claim {
address: None,
range: None,
hostname: None,
hostname_since: Some(1),
},
);
assert!(matches!(record.verify(id), Err(StateError::Malformed(_))));
// The time is signed: moving it breaks the signature, so nobody can
// improve another member's place in the queue.
let mut forged = named_at(&secret, id, 1, "music", 9_000);
if let RecordBody::Claim { hostname_since, .. } = &mut forged.body {
*hostname_since = Some(1);
}
assert_eq!(forged.verify(id).unwrap_err(), StateError::BadSignature);
}
#[test]
fn a_release_gives_up_the_name_as_well_as_the_address() {
let secret = SecretKey::generate();
@@ -894,6 +1137,7 @@ mod tests {
address: Some("10.13.37.5".parse().unwrap()),
range: Some(range()),
hostname: Some("music".into()),
hostname_since: None,
},
),
)
@@ -1057,6 +1301,7 @@ mod tests {
address: Some("10.99.0.7".parse().unwrap()),
range: Some(custom),
hostname: None,
hostname_since: None,
},
),
)
@@ -1098,7 +1343,7 @@ mod tests {
assert_eq!(once.base.octets()[0], 10, "inside 10/8 like the default");
assert_ne!(
once, DEFAULT_IPV4_RANGE,
"the default belongs to whichever network asked first"
"the legacy default remains excluded"
);
// A different network derives a different range, which is the
@@ -1110,4 +1355,34 @@ mod tests {
.network_id();
assert_ne!(once, derived_ipv4_range(other));
}
#[test]
fn a_derived_default_cannot_displace_a_legacy_or_explicit_range() {
let id = network("range-migration");
let a = SecretKey::generate();
let b = SecretKey::generate();
let (low, high) = if a.public().as_bytes() < b.public().as_bytes() {
(a, b)
} else {
(b, a)
};
for established in [DEFAULT_IPV4_RANGE, "10.90.0.0/16".parse().unwrap()] {
let body = |range: Ipv4Range| RecordBody::Claim {
address: Some(Ipv4Addr::from(u32::from(range.base) + 1)),
range: Some(range),
hostname: None,
hostname_since: None,
};
let fresh = SignedRecord::sign(&low, id, 1, body(derived_ipv4_range(id)));
let existing = SignedRecord::sign(&high, id, 1, body(established));
for records in [
vec![fresh.clone(), existing.clone()],
vec![existing.clone(), fresh.clone()],
] {
let mut state = StateSet::new();
state.merge_all(id, records);
assert_eq!(state.agreed_range(), Some(established));
}
}
}
}
crates/tsunagi-core/src/state/retirement.rs
+219
View File
@@ -0,0 +1,219 @@
//! A member may retire an observed claim, never another identity's future.
use iroh::{EndpointId, SecretKey, Signature};
use serde::{Deserialize, Serialize};
use super::{RecordBody, SignedRecord, StateError, push_lp};
use crate::NetworkId;
/// Maximum encoded retirement size, including its bounded claim proof.
pub const MAX_RETIREMENT_LEN: usize = 1024;
/// A network member's signed retirement of a proved claim version.
///
/// The original author's signed claim proves the cutoff really existed.
/// Newer claims remain valid. This is neither a forged Release nor a ban.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct SignedRetirement {
/// Device identity of the member requesting cleanup.
pub author: [u8; 32],
/// Network in which cleanup applies.
pub network: [u8; 32],
/// Requester's durable author counter, committed with this marker.
pub version: u64,
/// The target's original signed claim; bounds the retired version.
pub target: SignedRecord,
/// Requester's Ed25519 signature over this marker and its proof.
pub signature: Vec<u8>,
}
impl SignedRetirement {
fn canonical_bytes(&self) -> Vec<u8> {
let mut out = Vec::with_capacity(768);
push_lp(&mut out, b"tsunagi-node-retirement/v1");
push_lp(&mut out, &self.network);
push_lp(&mut out, &self.author);
push_lp(&mut out, &self.version.to_be_bytes());
// Bind every field of the original signed claim, including its signature.
push_lp(&mut out, &self.target.network);
push_lp(&mut out, &self.target.author);
push_lp(&mut out, &self.target.version.to_be_bytes());
push_lp(&mut out, &self.target.body.canonical());
push_lp(&mut out, &self.target.signature);
out
}
/// Signs a marker with the requesting member's persistent device key.
pub fn sign(
secret: &SecretKey,
network: NetworkId,
version: u64,
target: SignedRecord,
) -> Self {
let mut marker = Self {
author: *secret.public().as_bytes(),
network: *network.as_bytes(),
version,
target,
signature: Vec::new(),
};
marker.signature = secret.sign(&marker.canonical_bytes()).to_bytes().to_vec();
marker
}
/// Verifies both authors, scope and a finite, genuinely signed cutoff.
pub fn verify(&self, network: NetworkId) -> Result<EndpointId, StateError> {
if self.network != *network.as_bytes() {
return Err(StateError::WrongNetwork);
}
if self.version == 0 || self.version > i64::MAX as u64 {
return Err(StateError::Malformed("retirement version is out of range"));
}
if !matches!(self.target.body, RecordBody::Claim { .. }) {
return Err(StateError::Malformed("only a claim can be retired"));
}
let target = self.target.verify(network)?;
let author = EndpointId::from_bytes(&self.author).map_err(|_| StateError::BadAuthor)?;
if author == target {
return Err(StateError::Malformed(
"use Release to retire your own claim",
));
}
let raw: [u8; Signature::LENGTH] =
self.signature
.as_slice()
.try_into()
.map_err(|_| StateError::BadSignatureLength {
expected: Signature::LENGTH,
})?;
author
.verify(&self.canonical_bytes(), &Signature::from_bytes(&raw))
.map_err(|_| StateError::BadSignature)?;
Ok(target)
}
/// A deterministic strongest marker per target, independent of arrival order.
pub(crate) fn supersedes(&self, other: &Self) -> bool {
self.target.version > other.target.version
|| (self.target.version == other.target.version
&& (self.canonical_bytes(), &self.signature)
< (other.canonical_bytes(), &other.signature))
}
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
use crate::state::{Ipv4Range, StateSet};
fn claim(key: &SecretKey, id: NetworkId, version: u64) -> SignedRecord {
SignedRecord::sign(
key,
id,
version,
RecordBody::Claim {
address: Some("10.23.0.2".parse().unwrap()),
range: Some("10.23.0.0/24".parse::<Ipv4Range>().unwrap()),
hostname: Some("retired".into()),
hostname_since: Some(123),
},
)
}
#[test]
fn markers_release_claims_in_any_merge_order_and_older_snapshots_cannot_restore_them() {
let id = NetworkId::from_bytes([7; 32]);
let target = SecretKey::generate();
let cleaner = SecretKey::generate();
let record = claim(&target, id, 5);
let marker = SignedRetirement::sign(&cleaner, id, 8, record.clone());
for first in [true, false] {
let mut state = StateSet::new();
if first {
state.merge_retirement(id, marker.clone(), 16).unwrap();
}
state.merge(id, record.clone()).unwrap();
if !first {
state.merge_retirement(id, marker.clone(), 16).unwrap();
}
assert!(state.address_holders().is_empty());
assert!(state.hostname_holders().is_empty());
state.merge(id, claim(&target, id, 4)).unwrap();
assert!(state.is_retired(&target.public()));
assert!(
state.agreed_range().is_some(),
"cleanup does not renumber the network"
);
state.merge(id, claim(&target, id, 6)).unwrap();
assert!(!state.is_retired(&target.public()));
assert_eq!(state.hostname_of(&target.public()), Some("retired"));
}
}
#[test]
fn proof_signature_scope_tampering_future_cutoffs_and_capacity_are_checked() {
let id = NetworkId::from_bytes([7; 32]);
let target = SecretKey::generate();
let cleaner = SecretKey::generate();
let marker = SignedRetirement::sign(&cleaner, id, 8, claim(&target, id, 5));
assert!(marker.verify(NetworkId::from_bytes([8; 32])).is_err());
let mut forged = marker.clone();
forged.target.version = 99;
assert!(forged.verify(id).is_err());
let mut forged = marker.clone();
forged.signature[0] ^= 1;
assert!(forged.verify(id).is_err());
let mut proof = marker.target.clone();
proof.version = 99;
assert!(
SignedRetirement::sign(&cleaner, id, 8, proof)
.verify(id)
.is_err()
);
let mut state = StateSet::new();
assert!(state.merge_retirement(id, marker.clone(), 0).is_err());
assert!(state.merge_retirement(id, marker.clone(), 1).unwrap());
assert!(!state.merge_retirement(id, marker.clone(), 1).unwrap());
assert!(postcard::to_stdvec(&marker).unwrap().len() <= MAX_RETIREMENT_LEN);
let another = SecretKey::generate();
assert!(
state
.merge_retirement(
id,
SignedRetirement::sign(&cleaner, id, 9, claim(&another, id, 1)),
1
)
.is_err()
);
}
#[test]
fn competing_cleanup_markers_converge_and_keep_the_highest_observed_cutoff() {
let id = NetworkId::from_bytes([7; 32]);
let target = SecretKey::generate();
let a = SecretKey::generate();
let b = SecretKey::generate();
let markers = [
SignedRetirement::sign(&a, id, 1, claim(&target, id, 4)),
SignedRetirement::sign(&a, id, 2, claim(&target, id, 5)),
SignedRetirement::sign(&b, id, 3, claim(&target, id, 5)),
];
let mut forward = StateSet::new();
let mut reverse = StateSet::new();
for marker in &markers {
forward.merge_retirement(id, marker.clone(), 16).unwrap();
}
for marker in markers.iter().rev() {
reverse.merge_retirement(id, marker.clone(), 16).unwrap();
}
assert_eq!(forward.retirements(), reverse.retirements());
assert_eq!(
forward
.retirement_of(&target.public())
.unwrap()
.target
.version,
5
);
}
}
crates/tsunagi-core/src/state/vip.rs
+439
View File
@@ -0,0 +1,439 @@
//! Signed multi-writer virtual service registers and deterministic merge rules.
use crate::error::Result;
use crate::vip::{Backend, Listener, Protocol, VipConfig, VipStatus, invalid};
use crate::{Limits, NetworkId};
use iroh::{EndpointId, SecretKey, Signature};
use serde::{Deserialize, Serialize};
use sha2::{Digest, Sha256};
use std::collections::{BTreeMap, BTreeSet};
use std::net::Ipv4Addr;
/// A signed, independently mergeable field. A removed field keeps its tombstone.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct VipEntry {
/// Network scope.
pub network: [u8; 32],
/// Stable service id.
pub vip: String,
/// Canonical field key.
pub key: String,
/// Encoded field value; absent means deletion.
pub value: Option<Vec<u8>>,
/// Lamport clock, independent of wall time.
pub clock: u64,
/// Persistent signing identity.
pub author: [u8; 32],
/// Author's durable network counter.
pub version: u64,
/// Ed25519 signature.
pub signature: Vec<u8>,
}
impl VipEntry {
fn bytes(&self) -> Vec<u8> {
let mut e = self.clone();
e.signature.clear();
let mut out = b"tsunagi-vip-field/v1\0".to_vec();
if let Ok(bytes) = postcard::to_stdvec(&e) {
out.extend(bytes);
}
out
}
/// Verifies bounds, network and authorship before the entry is merged.
pub fn verify(&self, network: NetworkId) -> Result<()> {
if self.network != *network.as_bytes()
|| self.vip.len() != 32
|| hex::decode(&self.vip).is_err()
|| self.key.len() > 160
|| self.value.as_ref().is_some_and(|v| v.len() > 256)
|| self.version == 0
|| self.version > i64::MAX as u64
|| self.clock == 0
|| self.clock > i64::MAX as u64
{
return Err(invalid("invalid VIP field bounds or network"));
}
let author =
EndpointId::from_bytes(&self.author).map_err(|_| invalid("invalid VIP author"))?;
let raw: [u8; 64] = self
.signature
.as_slice()
.try_into()
.map_err(|_| invalid("invalid VIP signature length"))?;
author
.verify(&self.bytes(), &Signature::from_bytes(&raw))
.map_err(|_| invalid("invalid VIP signature"))?;
if self.key != "deleted"
&& self.key != "name"
&& !self.key.starts_with("l/")
&& !self.key.starts_with("b/")
{
return Err(invalid("unknown VIP field"));
}
match self.key.as_str() {
"deleted" => {
if self.value.as_deref() != Some(&[1][..]) {
return Err(invalid("invalid VIP deletion tombstone"));
}
}
"name" => {
let value = self
.value
.as_ref()
.ok_or_else(|| invalid("VIP name cannot be absent"))?;
let name: String =
postcard::from_bytes(value).map_err(|_| invalid("invalid VIP name field"))?;
VipConfig {
id: self.vip.clone(),
name,
listeners: vec![],
}
.validate(&Limits::default())?;
}
key if key.starts_with("l/") => {
let parts: Vec<_> = key.split('/').collect();
if parts.len() != 3
|| parts[1].parse::<Protocol>().is_err()
|| parts[2].parse::<u16>().ok().is_none_or(|p| p == 0)
{
return Err(invalid("invalid listener field key"));
}
if let Some(value) = &self.value {
let l: Listener = postcard::from_bytes(value)
.map_err(|_| invalid("invalid listener field"))?;
if !l.backends.is_empty() || key != format!("l/{}/{}", l.protocol, l.port) {
return Err(invalid("listener key does not match its value"));
}
VipConfig {
id: self.vip.clone(),
name: "validation".into(),
listeners: vec![l],
}
.validate(&Limits::default())?;
}
}
key if key.starts_with("b/") => {
let parts: Vec<_> = key.split('/').collect();
if parts.len() != 6
|| parts[1].parse::<Protocol>().is_err()
|| parts[2].parse::<u16>().ok().is_none_or(|p| p == 0)
|| parts[3].len() != 32
|| hex::decode(parts[3]).is_err()
|| parts[4].parse::<EndpointId>().is_err()
|| parts[5].parse::<u16>().ok().is_none_or(|p| p == 0)
{
return Err(invalid("invalid backend field key"));
}
if let Some(value) = &self.value {
let b: Backend = postcard::from_bytes(value)
.map_err(|_| invalid("invalid backend field"))?;
if b.peer != parts[4] || b.port.to_string() != parts[5] {
return Err(invalid("backend key does not match its value"));
}
}
}
_ => return Err(invalid("unknown VIP field")),
}
Ok(())
}
fn rank(&self) -> (u64, [u8; 32], Vec<u8>) {
(self.clock, self.author, self.bytes())
}
}
/// Signed registers retained across absences, including deletion tombstones.
#[derive(Debug, Clone, Default)]
pub struct VipSet {
entries: BTreeMap<(String, String), VipEntry>,
}
impl VipSet {
/// Highest retained version genuinely signed by this author.
pub fn authored_version(&self, author: EndpointId) -> u64 {
self.entries
.values()
.filter(|e| e.author == *author.as_bytes())
.map(|e| e.version)
.max()
.unwrap_or(0)
}
/// Complete independently verifiable state of one service, including tombstones.
pub fn entries(&self, id: &str) -> Vec<VipEntry> {
self.entries
.values()
.filter(|e| e.vip == id)
.cloned()
.collect()
}
/// All retained service ids, including deleted services.
pub fn ids(&self) -> Vec<String> {
self.entries
.keys()
.map(|(id, _)| id.clone())
.collect::<BTreeSet<_>>()
.into_iter()
.collect()
}
/// Opaque optimistic concurrency token for a whole service.
pub fn revision(&self, id: &str) -> String {
let mut hash = Sha256::new();
for e in self.entries(id) {
hash.update(e.bytes());
hash.update(&e.signature);
}
hex::encode(hash.finalize())
}
/// Atomically verifies and merges one complete service snapshot.
pub fn merge(
&mut self,
network: NetworkId,
entries: Vec<VipEntry>,
limits: &Limits,
) -> Result<bool> {
if entries.len() > limits.max_vip_fields {
return Err(invalid("too many VIP fields"));
}
let mut next = self.clone();
for e in entries {
e.verify(network)?;
let key = (e.vip.clone(), e.key.clone());
if next
.entries
.get(&key)
.is_none_or(|old| e.rank() > old.rank())
{
next.entries.insert(key, e);
}
}
let deleted: BTreeSet<_> = next
.entries
.values()
.filter(|e| e.key == "deleted" && e.value.is_some())
.map(|e| e.vip.clone())
.collect();
// A permanent service tombstone alone prevents every former field's replay.
next.entries
.retain(|(id, key), _| !deleted.contains(id) || key == "deleted");
let ids = next.ids();
if ids.len().saturating_sub(deleted.len()) > limits.max_vips
|| deleted.len() > limits.max_vip_tombstones
|| ids
.iter()
.any(|id| next.entries(id).len() > limits.max_vip_fields)
{
return Err(invalid(
"VIP state capacity exceeded; deletion protection is retained",
));
}
let changed = self.entries != next.entries;
*self = next;
Ok(changed)
}
/// Materialized configuration. A service tombstone permanently retires its id.
pub fn configs(&self) -> Vec<VipConfig> {
self.ids()
.into_iter()
.filter_map(|id| self.config(&id))
.collect()
}
fn config(&self, id: &str) -> Option<VipConfig> {
let fields = self.entries(id);
if fields
.iter()
.any(|e| e.key == "deleted" && e.value.is_some())
{
return None;
}
let name = fields
.iter()
.find(|e| e.key == "name")?
.value
.as_ref()
.and_then(|v| postcard::from_bytes::<String>(v).ok())?;
let mut listeners: Vec<Listener> = fields
.iter()
.filter(|e| e.key.starts_with("l/"))
.filter_map(|e| e.value.as_ref().and_then(|v| postcard::from_bytes(v).ok()))
.collect();
for l in &mut listeners {
l.backends = fields
.iter()
.filter(|e| {
e.key
.starts_with(&format!("b/{}/{}/{}/", l.protocol, l.port, l.id))
})
.filter_map(|e| e.value.as_ref().and_then(|v| postcard::from_bytes(v).ok()))
.collect();
l.backends.sort();
}
listeners.sort_by_key(|l| (l.protocol, l.port));
Some(VipConfig {
id: id.into(),
name,
listeners,
})
}
/// Builds a signed patch from the user's complete, version-checked draft.
pub fn edit(
&self,
network: NetworkId,
secret: &SecretKey,
version: u64,
config: &VipConfig,
expected: Option<&str>,
limits: &Limits,
) -> Result<Vec<VipEntry>> {
config.validate(limits)?;
if let Some(revision) = expected {
if revision != self.revision(&config.id) {
return Err(invalid("VIP changed while editing; reload before saving"));
}
} else if !self.entries(&config.id).is_empty() {
return Err(invalid("VIP id already exists"));
}
if self
.entries
.get(&(config.id.clone(), "deleted".into()))
.is_some_and(|e| e.value.is_some())
{
return Err(invalid("VIP was deleted; create a new VIP"));
}
let before = self
.config(&config.id)
.map(|c| flatten(&c))
.transpose()?
.unwrap_or_default();
let after = flatten(config)?;
let keys: BTreeSet<_> = before.keys().chain(after.keys()).cloned().collect();
let changes = keys
.into_iter()
.filter_map(|key| {
let value = after.get(&key).cloned();
(before.get(&key) != value.as_ref()).then_some((key, value))
})
.collect();
self.sign(network, secret, version, &config.id, changes)
}
/// Builds a permanent service tombstone after checking the selected revision.
pub fn delete(
&self,
network: NetworkId,
secret: &SecretKey,
version: u64,
id: &str,
expected: &str,
) -> Result<Vec<VipEntry>> {
if self.config(id).is_none() || self.revision(id) != expected {
return Err(invalid("VIP missing or changed while editing"));
}
self.sign(
network,
secret,
version,
id,
vec![("deleted".into(), Some(vec![1]))],
)
}
fn sign(
&self,
network: NetworkId,
secret: &SecretKey,
version: u64,
id: &str,
changes: Vec<(String, Option<Vec<u8>>)>,
) -> Result<Vec<VipEntry>> {
let clock = self
.entries
.values()
.map(|e| e.clock)
.max()
.unwrap_or(0)
.checked_add(1)
.filter(|n| *n <= i64::MAX as u64)
.ok_or_else(|| invalid("VIP clock exhausted"))?;
Ok(changes
.into_iter()
.map(|(key, value)| {
let mut e = VipEntry {
network: *network.as_bytes(),
vip: id.into(),
key,
value,
clock,
author: *secret.public().as_bytes(),
version,
signature: vec![],
};
e.signature = secret.sign(&e.bytes()).to_bytes().to_vec();
e
})
.collect())
}
}
pub(crate) fn flatten(cfg: &VipConfig) -> Result<BTreeMap<String, Vec<u8>>> {
let mut fields = BTreeMap::new();
fields.insert(
"name".into(),
postcard::to_stdvec(&cfg.name).map_err(|e| invalid(e.to_string()))?,
);
for l in &cfg.listeners {
let mut settings = l.clone();
settings.backends.clear();
fields.insert(
format!("l/{}/{}", l.protocol, l.port),
postcard::to_stdvec(&settings).map_err(|e| invalid(e.to_string()))?,
);
for b in &l.backends {
fields.insert(
format!("b/{}/{}/{}/{}/{}", l.protocol, l.port, l.id, b.peer, b.port),
postcard::to_stdvec(b).map_err(|e| invalid(e.to_string()))?,
);
}
}
Ok(fields)
}
impl VipSet {
/// Allocates internal addresses deterministically against durable node claims.
pub(crate) fn views(
&self,
range: Option<crate::state::Ipv4Range>,
mut taken: std::collections::HashSet<Ipv4Addr>,
hostnames: &BTreeSet<String>,
limits: &Limits,
) -> Vec<VipStatus> {
let mut names = hostnames.clone();
let mut result = Vec::new();
for config in self.configs() {
let mut error = config.validate(limits).err().map(|e| e.to_string());
if !names.insert(config.name.clone()) {
error = Some("name conflicts with another VIP or node".into());
}
let address = range.and_then(|r| {
let usable = crate::state::allocator::usable_addresses(r);
if usable == 0 {
return None;
}
let digest = Sha256::digest(config.id.as_bytes());
let seed = u64::from_be_bytes([
digest[0], digest[1], digest[2], digest[3], digest[4], digest[5], digest[6],
digest[7],
]);
let bits = 32 - u32::from(r.prefix_len);
let mask = if bits == 32 { 0 } else { u32::MAX << bits };
(0..usable).find_map(|step| {
let offset = (seed.wrapping_add(step) % usable + 1) as u32;
let ip = Ipv4Addr::from((u32::from(r.base) & mask) | offset);
taken.insert(ip).then_some(ip)
})
});
if address.is_none() {
error = Some("no free overlay address".into());
}
result.push(VipStatus {
revision: self.revision(&config.id),
config,
address,
error,
listeners: vec![],
});
}
result
}
}
crates/tsunagi/src/storage/cache.rs → crates/tsunagi-core/src/storage/cache.rs
View File
File renamed without changes.
crates/tsunagi/src/storage/lock.rs → crates/tsunagi-core/src/storage/lock.rs
View File
File renamed without changes.
crates/tsunagi/src/storage/mod.rs → crates/tsunagi-core/src/storage/mod.rs
+112 -1
View File
@@ -254,17 +254,64 @@ impl Storage {
.await
}
/// Persists the ordered protocol selection off the async executor.
pub async fn set_protocols(&self, network_id: NetworkId, protocols: Vec<String>) -> Result<()> {
self.with_state(move |state| state.set_protocols(network_id, &protocols))
.await
}
/// Persists whether this agent offers itself as an exit node in a network.
pub async fn set_exit_node(&self, network_id: NetworkId, enabled: bool) -> Result<()> {
self.with_state(move |state| state.set_exit_node(network_id, enabled))
.await
}
/// Persists the member this device sends its internet traffic through.
pub async fn set_exit_via(&self, network_id: NetworkId, via: Option<[u8; 32]>) -> Result<()> {
self.with_state(move |state| state.set_exit_via(network_id, via.as_ref()))
.await
}
/// Updates the auto-start flag of a network.
pub async fn set_auto_start(&self, network_id: NetworkId, auto_start: bool) -> Result<()> {
self.with_state(move |state| state.set_auto_start(network_id, auto_start))
.await
}
/// Configured networks in the user's most recently selected order.
pub async fn network_start_order(&self) -> Result<Vec<NetworkId>> {
self.with_state(|state| {
let configured = state.list_networks()?;
let text = state
.get_setting("network_start_order")?
.unwrap_or_default();
let mut order = Vec::new();
for word in text.split(',') {
if let Ok(id) = word.parse::<NetworkId>()
&& configured.iter().any(|n| n.network_id == id)
&& !order.contains(&id)
{
order.push(id);
}
}
Ok(order)
})
.await
}
/// Prefers this network at startup without disabling suspended networks.
pub async fn prefer_network(&self, network: NetworkId) -> Result<()> {
self.with_state(move |state| state.prefer_network(network))
.await
}
/// Removes a network configuration and its cached hints.
pub async fn remove_network(&self, network_id: NetworkId) -> Result<()> {
self.with_state(move |state| {
state.remove_network(network_id)?;
state.forget_signed_records(network_id)
state.forget_signed_records(network_id)?;
state.forget_retirements(network_id)?;
state.forget_peer_hostnames(network_id)
})
.await?;
self.with_cache((), move |cache| cache.forget_network(network_id))
@@ -283,6 +330,26 @@ impl Storage {
.await
}
/// Remembers the hostname a member announced, so it can still be named
/// once it is away.
pub async fn remember_peer_hostname(
&self,
network_id: NetworkId,
endpoint_id: [u8; 32],
hostname: String,
) -> Result<()> {
self.with_state(move |state| {
state.remember_peer_hostname(network_id, &endpoint_id, &hostname)
})
.await
}
/// The hostnames members of a network last announced.
pub async fn peer_hostnames(&self, network_id: NetworkId) -> Result<Vec<([u8; 32], String)>> {
self.with_state(move |state| state.peer_hostnames(network_id))
.await
}
/// Records an address hint. Failures are non-fatal.
pub async fn record_hint(
&self,
@@ -303,6 +370,50 @@ impl Storage {
.await
}
/// Loads mandatory signed VIP fields with a strict capacity bound.
pub async fn vip_fields(
&self,
network: NetworkId,
max: usize,
) -> Result<Vec<crate::vip::VipEntry>> {
self.with_state(move |state| state.vip_fields(network, max))
.await
}
/// Stores a complete merged service snapshot and optional own counter.
pub async fn put_vip_fields(
&self,
network: NetworkId,
entries: Vec<crate::vip::VipEntry>,
own: Option<([u8; 32], u64)>,
) -> Result<()> {
self.with_state(move |state| state.put_vip_fields(network, &entries, own))
.await
}
/// Loads verified durable cleanup proofs off the executor.
pub async fn retirements(
&self,
network: NetworkId,
max: usize,
) -> Result<Vec<crate::state::SignedRetirement>> {
self.with_state(move |state| state.retirements(network, max))
.await
}
/// Persists a received cleanup proof before applying it.
pub async fn put_retirement(&self, marker: crate::state::SignedRetirement) -> Result<()> {
self.with_state(move |state| state.put_retirement(&marker))
.await
}
/// Commits our cleanup and author counter together before announcement.
pub async fn publish_own_retirement(
&self,
marker: crate::state::SignedRetirement,
) -> Result<()> {
self.with_state(move |state| state.publish_own_retirement(&marker))
.await
}
/// Stores a record received from another replica.
pub async fn put_signed_record(&self, record: SignedRecord) -> Result<()> {
self.with_state(move |state| state.put_signed_record(&record))
Loaded 100 of 270 files, more files were not shown because too many files have changed in this diff. Show more