Compare commits

..
Author SHA1 Message Date
Aleksandr Bogomiakov 2631eb6975 Protect Linux exit-node connections and tidy the device table
Using an exit node could destroy direct peer connections: the agent's UID
bypass protected outbound routing, but reverse-path checks look up replies
without that UID and selected the overlay default. A real-host A/B check
confirmed that relaxing only strict reverse-path filtering restored direct
connections and exit HTTP traffic; the diagnostic host changes were removed.

Keep strict filtering and protect the Linux underlay with a dedicated masked
mark bit, 0x40000000. Mark the dedicated service UID in OUTPUT, save only that
bit in conntrack and restore it before PREROUTING reverse-path checks. Install
IPv4/IPv6 mark and UID main-table bypass rules before Tailscale's priorities;
leave application-specific and exit-default routing after them. Preserve
foreign mark bits, main-table routes and foreign firewall/policy rules.

Protect the first inbound TCP/UDP packet using bounded, locally owned socket
addresses/ports from /proc and an actual socket match. Refresh socket protection
before sending private transport negotiations and on discovery ticks, without
firewall writes for an unchanged socket set. Remote announcements never become
OS arguments. A failed refresh leaves active exit routing in place. Install
protection before routes, remove routes before protection, roll back partial
activation, and remove only exact interface tags, including legacy UID rules.

Reject strict kernel rp_filter without src_valid_mark with an actionable error;
never weaken filtering or change host sysctls. Document mark-aware Netfilter
requirements and provide the trusted ip6tables path in the NixOS module.
Other platforms retain their existing exit implementation. Peer wire formats,
identity derivation, local IPC and persistent storage are unchanged.

Make the tray devices grid fill and shrink with the window, constrain the first
cell and give remaining width to traffic plots. Keep displayed end-to-end
protocol and its short encryption tooltip consistent, independent of carrier
protocol. Move handshake/session age into its own column beside the existing
iroh control RTT. Use stable per-device hover identities and concise tooltips;
missing protocols/measurements remain explicit rather than invented.

Validation on Linux through nix develop:
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets: 529 passed, 3 ignored;
  inherited TSUNAGI_CONTROL_SOCKET and TSUNAGI_CONTROL_GROUP were unset
- explicit privileged kernel regression: 1 passed, using disposable namespaces,
  real TCP/UDP IPv4/IPv6, strict kernel and Netfilter RPF, first inbound packets,
  dynamic UDP binding, application exit routing/IPv6 blocking, repeated apply
  and exact cleanup with foreign Tailscale-priority rules retained
- headless egui grid/scroll-area rendering at multiple sizes and control RTT,
  mark masks/order, passive socket parsing, partial failure and cleanup tests
- git diff --check

Earlier iterations caught rtnetlink family API and nonexhaustive initializer
errors, Clippy warnings, egui's implicit minimum column width, and kernel-test
fixture issues (wildcard echo source, blocked IPv6 neighbour discovery and a
closed child stdout pipe). Corrected before final checks; the stdout-only test
fix was checked with fmt, Clippy and the explicit kernel regression afterward.
Native tray rendering and macOS/Windows checks were not run. The running host
agent was not replaced or restarted. Requested release tag: v0.1.0-rc.32.
2026-10-14 10:40:30 +01:00
Aleksandr Bogomiakov 90ef5cefec Retire offline node claims with equal network-wide cleanup rights
Offline devices reserved names and IPv4 addresses indefinitely, leaving stale
members in the device history. Give every authenticated network participant
the same right to retire another device's observed signed claim without an
owner, quorum, permanent identity ban or secret change.

Add a separately signed retirement containing the target's original signed
claim as proof. Verify both signatures and network scopes, retain the strongest
proved cutoff per target with deterministic ties, and suppress claim versions
at or below it in membership, hostname ownership and IPv4 allocation. Keep the
old signed records and range agreement, and retain markers after a newer claim
returns so old replicas cannot resurrect obsolete reservations. A returning
device signs a newer claim, allocates against current holders and resets its
previous hostname priority rather than taking a reused name with an old time.

Commit local retirement and the requester's author counter atomically before
announcing it; receivers persist proofs before applying them. Bound history to
1024 targets by default, reject additional targets without evicting protection,
and treat corrupt or over-limit mandatory storage as an explicit error. Repeat
bounded marker batches with a per-session rotating cursor so queue pressure
cannot indefinitely starve later markers. Exclude retired claims and cached
hints from history-derived discovery without preventing a fresh authenticated
device from returning.

Expose Agent::forget_node through the shared local control API. CLI command
tsng network forget-node accepts exact network names, full ids or unique id
prefixes, and exact node hostnames, ids, prefixes or overlay IPv4 addresses.
Ambiguous selections change nothing. Recheck the observed version and refuse
self, online nodes, the selected exit node and absent or changed claims.
Add Forget with an identity confirmation to offline tray device rows. Widen
Show devices to 1440x640 with an 800x400 minimum to fit its contents.

Control ALPN/version advances to tsunagi/ctrl/7 and local IPC to 21; upgrade
all peers and restart the agent with matching CLI/tray clients. Data ALPN 5,
network identity derivation, handshake encoding and existing record signing
bytes remain unchanged. SQLite schema 9 adds retirement storage while retaining
identity, secrets, settings and signed claims. Document behavior and limits.

Validation on Linux through nix develop:
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets: 519 passed, 2 existing ignored
  tests; inherited TSUNAGI_CONTROL_SOCKET and TSUNAGI_CONTROL_GROUP were unset
- git diff --check and the built CLI forget-node help

Tests use real iroh authentication and SQLite. Cover restart, stale-replica
recovery, equal rights and network isolation, hostname reuse and return, twelve
marker batches through a two-message queue, malformed signatures and proofs,
invented future cutoffs, deterministic merge, capacity, migration and injected
counter-write rollback, selectors and cleanup through two real CLI processes.
Earlier iterations caught a private test API, a collapsible tray conditional
and a crash-based CLI test waiting for QUIC's idle timeout; corrected before
the final successful suite. Manual tray rendering, macOS/Windows, privileged
TUNs and public-network traversal were not run. The requested RC tag is
v0.1.0-rc.31; no push was requested.
2026-10-12 18:04:29 +01:00
Aleksandr Bogomiakov 13bdcaf98d Render protocol priority buttons with Phosphor icons
The tray protocol selection used ordinary Unicode up/down arrows for its
priority buttons. The default UI fonts lack these glyphs, so both buttons
rendered missing-glyph boxes. Use the regular Phosphor ARROW_UP and ARROW_DOWN
icons already installed in the application's proportional font fallback.

Validation on Linux through nix develop:
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets, with inherited control socket
  and group overrides removed: 509 passed, 2 existing ignored tests

No new tests were added for the two glyph substitutions. Manual tray rendering
and macOS/Windows checks were not performed. No release tag was requested for
this fix.
2026-10-12 17:19:53 +01:00
Aleksandr Bogomiakov d941cdac7a Change network protocol selections at runtime
Protocol selection previously belonged to tsng up and therefore to the
service command line. A blocked or degraded carrier could only be removed
by editing the service and restarting the whole agent. Make the ordered
selection a local per-network setting instead.

Add Agent::set_protocols and a local control request shared by CLI and tray.
Keep tsng protocols without arguments as the protocol catalog; naming a
network shows its selection and supplying a comma-separated list replaces
it. Accept exact names, full ids and unique prefixes using shared selector
rules; ambiguous selectors and invalid lists change nothing. An explicit
none selects control-only operation. Offline edits take the directory lock
and apply on the next start; live edits never write behind the daemon.

Register every built-in plugin and transport in the CLI. Treat up --protocol
as ordered initial defaults for networks without an override, allowing a
protocol absent from the unit to be enabled later. Retain the targeting of
unqualified startup options and allow explicitly scoped options to configure
initially disabled protocols. Persist NULL for inherited preferences and an
empty selection distinctly in SQLite schema 8; migration preserves existing
identities, secrets, signed records and counters.

Serialize changes with network lifecycle operations and commit preferences
before applying or announcing them. Retire only the affected network's data
links and E2E channels, cancel pending opens, and reject obsolete completion
results by attempt id. Clear peer data channels when authenticated protocol
capabilities change while retaining control sessions and topology. Replay
allocations and peer identities after asynchronously prepared plugins become
ready. A control-only start defers creation of the agent's single interface
until data is enabled; plugins receive a sink that can use it later. Automatic
carrier fallback continues to preserve E2E state; explicit policy changes
renegotiate it. Identity derivation and peer wire formats are unchanged.

Add a collapsible protocols row to each tray network tile, with a catalog
provided by the daemon, descriptions, immediately applied checkboxes and
preference-order buttons. Keep editing disabled until the worker refreshes
status so a second click cannot overwrite an edit using a stale snapshot.
Local IPC wire version is 20 for the new request and report fields; upgrade
and restart agent, CLI and tray together. Crate release versions remain unchanged; the requested RC tag is v0.1.0-rc.30.

Validation on Linux through nix develop:
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets: 509 passed, 2 existing ignored
  tests, with inherited TSUNAGI_CONTROL_SOCKET and TSUNAGI_CONTROL_GROUP removed

Tests use real iroh membership, SQLite, UDP, AWG and WireGuard encryption,
QUIC carriers and memory TUNs. Cover either endpoint-id role, full-size packet
delivery after switching, an unaffected second active network, saved selection
after restart, empty/invalid selections, first enablement without restart,
rapid changes racing old opens, live and offline CLI changes, startup-absent
protocols and preservation of signed state through migration. Earlier compile
and focused-test iterations caught initializer mistakes, a short test secret,
asynchronous plugin allocation readiness, and an inherited system socket
path; corrected before the successful final checks. Plain cargo was unavailable
in the host shell, so validation used the Nix toolchain. Manual tray interaction,
macOS/Windows, privileged host TUNs and public-network traversal were not run.
2026-10-12 16:19:18 +01:00
Aleksandr Bogomiakov e701986a39 Add ephemeral pairwise Amnezia transport and runtime fallback
Plain WG and QUIC can be blocked or severely degraded while the iroh control
connection still works. Add an awg carrier with one independent UDP socket,
WireGuard engine and randomized masking profile per network/peer attempt.
The smaller endpoint id negotiates fresh outer keys, PSK, header protection,
header ranges, padding and signature templates through private authenticated
control messages. Restart or fallback reentry generates a new profile;
ordinary rekeying preserves the current profile. No pair parameters enter
signed state, capabilities, persistent storage or public message events.
Private payloads and serialized control buffers redact and zeroize themselves.

Adapt the bounded packet masking engine from RunaRoo/AmneziaWG-RS at commit
51579a7d6e8882d1d8d6d00822f09e18a9423b73, retaining its Apache license and
provenance. Declare WTFPL AND Apache-2.0 in package metadata to cover both
new glue and the adapted source. Reuse boringtun instead of upstream's Node/global executors and
interface machinery. This is Tsunagi AWG wire version 1, with standard WG
headers restored before authentication; external Amnezia server compatibility
is not implemented. DNS/SIP signature generators vary fields instead of
replaying captures. Each attempt closes independently, bounds allocation,
queues, candidates and fragment assemblies, and supports the full 64 KiB
carrier contract using UDP packets at most 1200 bytes. Yield between fragment
batches so large messages do not starve local socket readers.

Keep carrier encryption separate from end-to-end tunnels so multihop relays
still forward opaque ciphertext. The system owns the sole interface, signed
addresses and admission. Plugins receive neither network secrets nor paths.
Expose a bounded opaque control sender at the transport boundary; core control
messages remain outside protocol crates. Optional STUN on the pair's actual
socket adds IPv4 public candidates in CLI direct/relay mode, with a finite
query deadline and an off/alternate-server option. Local mode remains offline.
Symmetric NAT, IPv6-only paths and arbitrary UDP blocking can still require
fallback; no AWG-over-QUIC or hidden AWG relay is introduced.

Default carrier order is wg, awg, wg-quic, tcp-tls. Advance the fallback cycle
after setup failures and sustained runtime loss, even after earlier cooldowns
expire. Authenticated delivery probes traverse the actual encrypted tunnel,
fit the interface MTU and never enter the TUN. Retire after a 20-second warmup
and at least 60% loss over at least 15 completed recent observations. Preserve
logical tunnels and control sessions while explicitly closing physical links,
including TCP reader/writer tasks. Retain the newest private probe settings
across independently queued control negotiation and inbound installation.
CLI/tray reports keep end-to-end protocol and actual carrier separate.

Control ALPN/version becomes tsunagi/ctrl/6 for the new private message;
upgrade all members together. Data ALPN/envelopes remain version 5. Network
identity derivation, transcript encoding, device identities, signed records,
saved network configuration and release versions are unchanged.

Validation on Linux through nix develop:
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets, with the inherited
  TSUNAGI_CONTROL_SOCKET override removed: 504 passed, 2 pre-existing ignored

Tests use real iroh membership, SQLite, UDP sockets and WireGuard encryption
with memory TUNs. Cover three independent pairs, full-size IP and carrier
frames, restart freshness/socket release, 80% carrier datagram loss, fallback
to real iroh, fresh retries with working delivery checks, unequal small MTUs,
malformed negotiations/profiles, role and resource bounds, probe forgery and
STUN parsing. Earlier compile/lint iterations and focused tests caught RNG
API/initializer mistakes, a retry assertion racing path installation, and a
large-fragment burst overflowing loopback buffers; corrected before these
successful checks. Normalized upstream license line endings for diff checks.
Public NAT/STUN connectivity, actual DPI effectiveness, WAN performance,
per-pair memory measurement, macOS/Windows, privileged TUN operations and
manual tray interaction were not exercised.
2026-10-11 13:52:10 +01:00
Aleksandr Bogomiakov ed28e86748 Derive default network ranges and switch overlapping networks
Every first network previously proposed 10.13.37.0/24. Independently created
networks therefore collided when a device joined more than one of them.
Use the existing stable NetworkId-derived /24 in 10/8 for every new network,
independent of activation order or peer availability. Legacy and explicit
signed ranges outrank the derived default; remaining range disagreements
still use author-id ordering. The derivation, identity labels, handshake and
signed record encoding stay unchanged. There is no creator, timestamp or vote.

Reserve ranges before starting network actors and serialise start, stop and
switching through the agent lifecycle lock. Refuse equal or nested overlaps
without changing a network's subnet locally. A late authenticated range that
collides suspends only its network before updating plugins or the interface.
Coalesced watch notifications let the agent clean up outside the network actor
and reconsider suspended networks when ranges change.

An explicit switch stops every blocker before activating the chosen network.
Suspended networks retain their startup request and resume whenever their
ranges become free, including switching from a wide range to a narrow one.
A manual stop cancels that request even for an already suspended network.
Persist preference order and enabling together in an SQLite transaction so
the user's selection survives restart; no database schema migration is needed.

CLI status and network listing show blocker names, ids, ranges and the start
command. Network start switches explicitly and reports suspended networks;
joining an overlapping network reports it as configured but inactive. The
tray adds conflict details, an amber icon, Switch to this network and Keep
stopped actions. Local IPC wire version is 19 for the extended reports, so
restart the agent and tray together. Release versions and peer wire formats
are unchanged. Update the user documentation and compatibility tests.

Validation on Linux through nix develop:
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets with TSUNAGI_CONTROL_SOCKET
  removed from the test environment: 486 passed, 2 pre-existing ignored tests
- final focused range_conflicts integration suite after making discovery
  fixtures deterministic: 6 passed

Earlier validation attempts caught outdated diagnostic assertions, a new
test helper scope error, and the inherited system control socket override.
Those were corrected before the successful runs. Plain cargo was unavailable
in the host shell, so validation used the Nix development toolchain. macOS,
Windows, privileged host TUN operations and manual tray interaction were not
exercised.
2026-10-11 11:58:27 +01:00
Aleksandr Bogomiakov 781e359775 Make tunnel and transport layers explicit in status
The tray called the end-to-end tunnel proto and the current route link, then put the tunnel handshake age beside the route. A valid wg tunnel over a tcp-tls transport consequently looked like contradictory state. The terminal status used similarly terse wording.\n\nThe tray now separates end-to-end from data path, renders direct paths as direct over TRANSPORT, keeps handshake or uptime beside the tunnel, and explains both layers in hover text. Terminal status uses the same end-to-end tunnel, data path, transport detail, and control path vocabulary. Routing, protocol selection, IPC, and wire behavior are unchanged.\n\nPresentation tests cover wg over tcp-tls, tcp-tls uptime, relayed and missing paths, a missing handshake, and offline devices. The WireGuard broadcast checksum test uses the current slice-array iterator spelling required by Clippy; its behavior is unchanged.\n\nChecks: cargo fmt --all -- --check; cargo clippy --locked --workspace --all-targets -- -D warnings; cargo test --locked --workspace --all-targets with TSUNAGI_CONTROL_SOCKET removed from the test environment so integration agents use their temporary control sockets. All passed on Linux.
2026-10-08 16:03:20 +01:00
Aleksandr Bogomiakov d5e02ec256 fixed gitignore 2026-10-08 15:35:38 +01:00
70 changed files with 10020 additions and 650 deletions

No files matched your search

+1
View File
@@ -1 +1,2 @@
/target
/.claude
Generated
+21
View File
@@ -6246,6 +6246,7 @@ dependencies = [
"tokio",
"tracing",
"tracing-subscriber",
"tsunagi-awg",
"tsunagi-core",
"tsunagi-tcp-tls",
"tsunagi-wg-quic",
@@ -6253,6 +6254,26 @@ dependencies = [
"windows-service",
]
[[package]]
name = "tsunagi-awg"
version = "0.1.0"
dependencies = [
"boringtun",
"bytes",
"chacha20 0.9.1",
"iroh",
"postcard",
"rand 0.10.3",
"serde",
"tempfile",
"tokio",
"tracing",
"tsunagi-core",
"tsunagi-wg-quic",
"tsunagi-wireguard",
"zeroize",
]
[[package]]
name = "tsunagi-core"
version = "0.1.0"
+52 -1
View File
@@ -81,7 +81,17 @@ tsng join -n home -s tsn1… # every other machine
tsng status # who is online, over which path
```
Every member gets an address in `10.13.37.0/24` and a name like `<host>.home`.
Every new network gets a stable pseudorandom `/24` in `10.0.0.0/8`, derived
from its network id. Its members agree on that range even when starting offline,
and each gets a name like `<host>.home`. Existing signed ranges are preserved.
If two configured networks have overlapping ranges, the agent runs only one.
`tsng status`, `tsng network` and the tray show which network blocks the
other. `tsng network start <name-or-id>` or the tray’s “Switch to this network”
button suspends the conflicting networks. They resume automatically when the
selected network stops; the selection survives restarting the agent. Explicitly
stopping a suspended network cancels its automatic resume.
Machines are called `tsunagi-<id>` until you rename them: `tsng id hostname laptop`.
### Exit nodes
@@ -97,6 +107,46 @@ The tray does the same with a checkbox. An exit node sees everything you send
through it, so use only members you trust.
[Details and per-OS notes](docs/exit-node.md).
The default data transport order is `wg,awg,wg-quic,tcp-tls`. AWG negotiates a
fresh masking profile and outer keys for each pair connection attempt, including
reentry after fallback. Sustained data loss triggers switching independently of
the iroh control connection. [Configuration and limits](docs/amnezia.md).
Change a network's protocol selection without editing its service or restarting:
```sh
tsng protocols # available protocols and settings
tsng protocols home # this device's choice for home
tsng protocols home awg,wg-quic # prefer AWG, retain QUIC as fallback
tsng protocols home none # control plane only
```
Exact network names, full ids and unique id prefixes work here. The choice is
local to this device, saved per network and applied immediately; an offline
change applies on the next start. In the tray, expand **protocols** on a network
tile to toggle protocols or change their preference order. Both endpoints need
a common protocol at the same wire version. `tsng up --protocol` supplies the
initial defaults for networks without a saved choice; it does not restrict
which built-in protocols can be enabled later.
Remove an obsolete offline device from a network's shared history:
```sh
tsng network forget-node home old-laptop
tsng network forget-node home 10.13.37.143
```
Networks accept exact names, full ids or unique id prefixes; nodes accept those
id selectors, exact hostnames or overlay IPv4 addresses. Ambiguous selectors
change nothing. Start the network first if it is stopped. In **Show devices**,
click **Forget** beside an offline node and confirm its identity.
Every member has the same cleanup right. The signed cleanup frees the observed
claim's name and IPv4 reservation, survives restarts and reaches offline replicas
when they reconnect. It does not ban the device: a newer signed claim can return.
Online nodes and the currently selected exit node cannot be forgotten.
[State and limits](docs/sync-model.md#offline-node-cleanup).
## Platforms
| Platform | Build | CLI `tsng` | Tray | Exit node: offer / use | Packages |
@@ -174,6 +224,7 @@ The [threat model](docs/threat-model.md) says what this does and doesn't protect
[Architecture](docs/architecture.md) ·
[Protocol](docs/protocol.md) ·
[WireGuard data plane](docs/wireguard.md) ·
[Pairwise Amnezia transport](docs/amnezia.md) ·
[Routing](docs/routing.md) ·
[Exit nodes](docs/exit-node.md) ·
[Threat model](docs/threat-model.md) ·
+30
View File
@@ -0,0 +1,30 @@
[package]
name = "tsunagi-awg"
version = "0.1.0"
description = "Ephemeral pairwise Amnezia transport for Tsunagi."
edition.workspace = true
rust-version.workspace = true
license = "WTFPL AND Apache-2.0"
repository.workspace = true
[dependencies]
tsunagi-core = { path = "../tsunagi-core" }
tsunagi-wg-quic = { path = "../tsunagi-wg-quic" }
tsunagi-wireguard = { path = "../tsunagi-wireguard" }
iroh.workspace = true
tokio = { workspace = true, features = ["net", "time", "sync", "macros"] }
serde.workspace = true
postcard.workspace = true
bytes.workspace = true
tracing.workspace = true
rand = "0.10"
zeroize = { version = "1.9", features = ["derive"] }
boringtun = { version = "0.7.1", default-features = false }
chacha20 = "0.9"
[dev-dependencies]
tsunagi-core = { path = "../tsunagi-core", features = ["testing"] }
tempfile.workspace = true
[lints]
workspace = true
+201
View File
@@ -0,0 +1,201 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright 2026 Friderika Coulson (NanoBee)
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
+21
View File
@@ -0,0 +1,21 @@
# Amnezia packet engine
`src/masking.rs` is adapted from `src/core/amnezia.rs` in
https://github.com/RunaRoo/AmneziaWG-RS at
`51579a7d6e8882d1d8d6d00822f09e18a9423b73`, under Apache-2.0.
The upstream license is retained in `LICENSE-AmneziaWG-RS`.
The adaptation replaces configuration-file loading with bounded typed pair
profiles, updates RNG calls to rand 0.10, removes an unnecessary unwrap, and
does not embed upstream's Node, global executors, OS interface operations or
detached tasks. Noise/WireGuard is supplied by boringtun, already used by
Tsunagi. Each pair owns one socket and one cancellable packet-engine task.
This is Tsunagi AWG wire version 1, not a claim of compatibility with external
Amnezia servers. The masking engine substitutes message headers outside the
standard WireGuard state machine and restores them before Noise validation.
There are no remote file paths, shell hooks, or persisted pair profiles.
The adapted masking source remains Apache-2.0. Newly written Tsunagi code
follows the repository's WTFPL. This package declares both licenses in Cargo
metadata; the workspace license alone does not describe its copied source.
+16
View File
@@ -0,0 +1,16 @@
//! Pairwise, ephemeral Amnezia packet transport with independent end-to-end tunnels.
//! All negotiation travels over authenticated Tsunagi control sessions.
mod masking;
mod plugin;
pub mod profile;
mod session;
mod stun;
mod transport;
pub use plugin::AwgPlugin;
pub use transport::{AwgTransport, PairOverview};
/// Protocol identifier and default second-choice direct transport.
pub const AWG_PROTOCOL: &str = "awg";
/// Tsunagi AWG wire version, independent of software releases.
pub const AWG_WIRE_VERSION: u16 = 1;
+336
View File
@@ -0,0 +1,336 @@
// Adapted from RunaRoo/AmneziaWG-RS, commit 51579a7d6e8882d1d8d6d00822f09e18a9423b73.
// Apache-2.0; see LICENSE-AmneziaWG-RS and UPSTREAM.md.
#![allow(dead_code, missing_docs)]
// AmneziaWG configuration, header protection, and traffic shaping.
use bytes::{Bytes, BytesMut};
use chacha20::{
ChaCha20,
cipher::{KeyIvInit, StreamCipher},
};
use rand::RngExt;
pub const HEADER_PROTECTION_NONCE_SIZE: usize = 12;
pub const DEFAULT_UDP_WINDOW: usize = 500;
#[derive(Clone, Copy, Debug, Default, Eq, PartialEq)]
pub struct Range {
pub min: u32,
pub max: u32,
}
impl Range {
pub fn contains(&self, val: u32) -> bool {
val >= self.min && val <= self.max
}
pub fn random(&self) -> u32 {
if self.min >= self.max {
self.min
} else {
rand::rng().random_range(self.min..=self.max)
}
}
pub fn is_zero(&self) -> bool {
self.min == 0 && self.max == 0
}
}
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub enum AwgKind {
Initiation,
Response,
Cookie,
Transport,
}
impl AwgKind {
pub fn fixed_size(self) -> usize {
match self {
Self::Initiation => 148,
Self::Response => 92,
Self::Cookie => 64,
Self::Transport => 32,
}
}
fn standard_type(self) -> u32 {
match self {
Self::Initiation => 1,
Self::Response => 2,
Self::Cookie => 3,
Self::Transport => 4,
}
}
}
#[derive(Debug)]
pub struct DecodedPacket {
pub kind: AwgKind,
pub message: Bytes,
pub wire_len: usize,
}
pub struct AmneziaConfig {
pub jc: u32,
pub jmin: usize,
pub jmax: usize,
pub s1: usize,
pub s2: usize,
pub s3: usize,
pub s4: usize,
pub h1: Range,
pub h2: Range,
pub h3: Range,
pub h4: Range,
pub i1: String,
pub i2: String,
pub i3: String,
pub i4: String,
pub i5: String,
pub header_protection_key: Option<[u8; 32]>,
pub content_padding_addition: Range,
pub random_trailers: bool,
pub mtu: usize,
}
impl AmneziaConfig {
pub fn random_header(&self, range: &Range) -> u32 {
range.random()
}
pub fn random_padding(&self, len: usize) -> Vec<u8> {
let mut padding = vec![0u8; len];
if len > 0 {
rand::fill(&mut padding);
}
padding
}
fn shape(&self, kind: AwgKind) -> (usize, Range) {
match kind {
AwgKind::Initiation => (self.s1, self.h1),
AwgKind::Response => (self.s2, self.h2),
AwgKind::Cookie => (self.s3, self.h3),
AwgKind::Transport => (self.s4, self.h4),
}
}
pub fn encode(&self, kind: AwgKind, payload: Bytes, udp_window: usize) -> Bytes {
let (padding_len, _) = self.shape(kind);
let trailer_len = if kind != AwgKind::Transport && self.random_trailers {
let base = padding_len.saturating_add(payload.len());
let room = udp_window.max(DEFAULT_UDP_WINDOW).saturating_sub(base);
if room == 0 {
0
} else {
rand::rng().random_range(0..room)
}
} else {
0
};
let mut out = BytesMut::zeroed(padding_len + payload.len() + trailer_len);
if padding_len > 0 {
rand::fill(&mut out[..padding_len]);
}
out[padding_len..padding_len + payload.len()].copy_from_slice(&payload);
if trailer_len > 0 {
rand::fill(&mut out[padding_len + payload.len()..]);
}
if self.header_protection_key.is_some() {
let protected_len = if kind == AwgKind::Transport {
16
} else {
kind.fixed_size()
};
self.apply_header_cipher(&mut out, padding_len, protected_len);
}
out.freeze()
}
pub fn decode(&self, datagram: &Bytes) -> Option<DecodedPacket> {
if datagram.len() < 4 {
return None;
}
for kind in [
AwgKind::Initiation,
AwgKind::Response,
AwgKind::Cookie,
AwgKind::Transport,
] {
let (padding_len, headers) = self.shape(kind);
let minimum = padding_len + kind.fixed_size();
let size_ok = match kind {
AwgKind::Transport | AwgKind::Initiation => datagram.len() >= minimum,
AwgKind::Response | AwgKind::Cookie if self.random_trailers => {
datagram.len() >= minimum
}
AwgKind::Response | AwgKind::Cookie => datagram.len() == minimum,
};
if !size_ok || datagram.len() < padding_len + 4 {
continue;
}
let protected_header =
u32::from_le_bytes(datagram[padding_len..padding_len + 4].try_into().ok()?);
let candidate = if self.header_protection_key.is_some() {
protected_header ^ self.header_type_hash(datagram)?
} else {
protected_header
};
if !headers.contains(candidate) {
continue;
}
let message_len = match kind {
AwgKind::Transport | AwgKind::Initiation => datagram.len() - padding_len,
AwgKind::Response | AwgKind::Cookie => kind.fixed_size(),
};
let mut payload = BytesMut::from(&datagram[padding_len..padding_len + message_len]);
if self.header_protection_key.is_some() {
let protected_len = if kind == AwgKind::Transport {
16
} else {
kind.fixed_size()
};
self.apply_header_cipher_with_nonce(
&mut payload,
0,
protected_len,
&datagram[..HEADER_PROTECTION_NONCE_SIZE],
);
}
return Some(DecodedPacket {
kind,
message: payload.freeze(),
wire_len: datagram.len(),
});
}
None
}
pub fn prepare_transport_plaintext(&self, packet: Bytes, udp_window: usize) -> Bytes {
if packet.is_empty() {
return packet;
}
let requested = if !self.content_padding_addition.is_zero() {
self.content_padding_addition.random() as usize
} else if self.random_trailers {
let wire_without_padding = self.s4 + AwgKind::Transport.fixed_size() + packet.len();
let room = udp_window
.max(DEFAULT_UDP_WINDOW)
.saturating_sub(wire_without_padding);
if room == 0 {
0
} else {
rand::rng().random_range(0..room)
}
} else {
let aligned = (packet.len() + 15) & !15;
aligned.saturating_sub(packet.len())
};
let padding = self.clamp_padding_to_mtu(packet.len(), requested);
if padding == 0 {
return packet;
}
let mut out = BytesMut::with_capacity(packet.len() + padding);
out.extend_from_slice(&packet);
out.resize(packet.len() + padding, 0);
out.freeze()
}
pub fn transport_content_length(&self, plaintext: &[u8]) -> Option<usize> {
if plaintext.is_empty() || plaintext[0] == 0 {
return Some(0);
}
match plaintext[0] >> 4 {
4 if plaintext.len() >= 20 => {
let total = u16::from_be_bytes([plaintext[2], plaintext[3]]) as usize;
(total >= 20 && total <= plaintext.len()).then_some(total)
}
6 if plaintext.len() >= 40 => {
let payload = u16::from_be_bytes([plaintext[4], plaintext[5]]) as usize;
let total = 40 + payload;
(total <= plaintext.len()).then_some(total)
}
_ => None,
}
}
fn clamp_padding_to_mtu(&self, packet_len: usize, requested: usize) -> usize {
if self.mtu == 0 {
return requested;
}
let normalized = if packet_len > self.mtu {
packet_len % self.mtu
} else {
packet_len
};
requested.min(self.mtu.saturating_sub(normalized))
}
fn header_type_hash(&self, datagram: &Bytes) -> Option<u32> {
if datagram.len() < HEADER_PROTECTION_NONCE_SIZE {
return None;
}
let mut hash = [0u8; 4];
self.apply_keystream(&mut hash, &datagram[..HEADER_PROTECTION_NONCE_SIZE]);
Some(u32::from_le_bytes(hash))
}
fn apply_header_cipher(&self, packet: &mut [u8], offset: usize, len: usize) {
if packet.len() < HEADER_PROTECTION_NONCE_SIZE || packet.len() < offset + len {
return;
}
let nonce: [u8; HEADER_PROTECTION_NONCE_SIZE] =
match packet[..HEADER_PROTECTION_NONCE_SIZE].try_into() {
Ok(nonce) => nonce,
Err(_) => return,
};
self.apply_header_cipher_with_nonce(packet, offset, len, &nonce);
}
fn apply_header_cipher_with_nonce(
&self,
packet: &mut [u8],
offset: usize,
len: usize,
nonce: &[u8],
) {
if packet.len() < offset + len {
return;
}
self.apply_keystream(&mut packet[offset..offset + len], nonce);
}
fn apply_keystream(&self, data: &mut [u8], nonce: &[u8]) {
let Some(key) = self.header_protection_key.as_ref() else {
return;
};
if nonce.len() != HEADER_PROTECTION_NONCE_SIZE {
return;
}
let Ok(nonce) = <&[u8; HEADER_PROTECTION_NONCE_SIZE]>::try_from(nonce) else {
return;
};
let mut cipher = ChaCha20::new(key.into(), nonce.into());
cipher.apply_keystream(data);
}
}
impl Drop for AmneziaConfig {
fn drop(&mut self) {
use zeroize::Zeroize;
self.header_protection_key.zeroize();
}
}
+132
View File
@@ -0,0 +1,132 @@
//! End-to-end encryption remains independent of physical Amnezia pair links.
use crate::{AWG_PROTOCOL, AWG_WIRE_VERSION, AwgTransport};
use iroh::EndpointId;
use std::sync::Arc;
use tsunagi::dataplane::transport::SharedLink;
use tsunagi::dataplane::{IpPlugin, PluginCapability, PluginContext, PluginError, ProtocolOption};
use tsunagi::state::Ipv4Range;
use tsunagi::{BoxFuture, NetworkId};
use tsunagi_wg_quic::{NetworkOverview, WireguardConfig, WireguardPlugin};
/// AWG capability and an independent end-to-end WireGuard engine.
/// Pair transport keys/profiles never enter its durable end-to-end key store.
#[derive(Debug)]
pub struct AwgPlugin {
inner: Arc<WireguardPlugin>,
transport: Arc<AwgTransport>,
}
impl AwgPlugin {
/// Uses standard tunnel MTU/keepalive options; pair ports and profiles are ephemeral.
pub const OPTIONS: &'static [ProtocolOption] = &[
ProtocolOption {
key: "keepalive",
value: "SECONDS",
help: "end-to-end tunnel keepalive; 0 disables it",
default: Some("25"),
},
ProtocolOption {
key: "mtu",
value: "BYTES",
help: "maximum overlay packet size",
default: Some("1280"),
},
ProtocolOption {
key: "stun",
value: "HOST:PORT|off",
help: "UDP public mapping discovery; disabled in local reach mode",
default: Some("stun.l.google.com:19302"),
},
];
/// Opens the end-to-end engine, retaining separate pair transport state.
pub async fn open(
mut config: WireguardConfig,
transport: Arc<AwgTransport>,
) -> Result<Arc<Self>, PluginError> {
config.protocol = AWG_PROTOCOL;
Ok(Arc::new(Self {
inner: WireguardPlugin::open(config).await?,
transport,
}))
}
/// Reports end-to-end tunnel health and traffic counters.
pub fn overview(&self, network: NetworkId) -> Option<NetworkOverview> {
self.inner.overview(network)
}
}
impl IpPlugin for AwgPlugin {
fn protocol_id(&self) -> &str {
AWG_PROTOCOL
}
fn protocol_version(&self) -> u16 {
AWG_WIRE_VERSION
}
fn options(&self) -> &'static [ProtocolOption] {
Self::OPTIONS
}
fn attach(&self, context: PluginContext) {
self.inner.attach(context);
}
fn local_capability(
&self,
network: NetworkId,
) -> Result<Option<PluginCapability>, PluginError> {
self.inner.local_capability(network).map(|cap| {
cap.map(|mut cap| {
cap.protocol = AWG_PROTOCOL.into();
cap.version = AWG_WIRE_VERSION;
cap
})
})
}
fn on_network_activated(&self, network: NetworkId) {
self.inner.on_network_activated(network);
}
fn on_peer_capability(
&self,
network: NetworkId,
peer: EndpointId,
capability: &PluginCapability,
) -> Result<(), PluginError> {
if capability.protocol != AWG_PROTOCOL
|| capability.version != AWG_WIRE_VERSION
|| !capability.enabled
{
return Err(PluginError::Rejected("unsupported AWG capability".into()));
}
self.inner.on_peer_capability(network, peer, capability)?;
self.transport.allow_peer(network, peer);
Ok(())
}
fn on_address_allocation(
&self,
network: NetworkId,
range: Ipv4Range,
allocations: &[(EndpointId, std::net::Ipv4Addr)],
) {
self.inner
.on_address_allocation(network, range, allocations);
}
fn carry(&self, network: NetworkId, peer: EndpointId, packet: bytes::Bytes) -> bool {
self.inner.carry(network, peer, packet)
}
fn on_peer_link(&self, network: NetworkId, peer: EndpointId, link: SharedLink) {
self.inner.on_peer_link(network, peer, link);
}
fn on_peer_gone(&self, network: NetworkId, peer: EndpointId) {
self.transport.forget_peer(network, peer);
self.inner.on_peer_gone(network, peer);
}
fn on_network_deactivated(&self, network: NetworkId) {
self.transport.stop_network(network);
self.inner.on_network_deactivated(network);
}
fn on_network_forgotten(&self, network: NetworkId) {
self.inner.on_network_forgotten(network);
}
fn shutdown(&self) -> BoxFuture<'_, ()> {
Box::pin(async move {
self.transport.shutdown();
self.inner.shutdown().await;
})
}
}
+226
View File
@@ -0,0 +1,226 @@
//! Bounded profiles, regenerated for every pair connection attempt.
use crate::masking::{AmneziaConfig, AwgKind, Range};
use bytes::Bytes;
use serde::{Deserialize, Serialize};
/// An ephemeral masking profile. Secret fields are redacted and erased on drop.
#[derive(Clone, Serialize, Deserialize, zeroize::ZeroizeOnDrop)]
pub struct PairProfile {
id: [u8; 16],
prefix: u16,
headers: [[u32; 2]; 4],
header_key: [u8; 32],
pub(crate) psk: [u8; 32],
junk_count: u8,
junk_min: u16,
junk_max: u16,
padding: u16,
signature: u8,
}
impl std::fmt::Debug for PairProfile {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str("PairProfile([REDACTED])")
}
}
impl PairProfile {
/// Generates a new safe profile; no deterministic seed or durable state.
pub fn generate() -> Self {
let headers = std::array::from_fn(|i| {
let base = (i as u32) * 0x4000_0000 + rand::random_range(1024..0x3fff_0000);
[base, base + rand::random_range(64..1024)]
});
Self {
id: rand::random(),
prefix: rand::random_range(12..49),
headers,
header_key: rand::random(),
psk: rand::random(),
junk_count: rand::random_range(4..9),
junk_min: rand::random_range(24..65),
junk_max: rand::random_range(80..161),
padding: rand::random_range(16..65),
signature: rand::random_range(0..2),
}
}
/// Public attempt identifier, useful for distinguishing retries without revealing secrets.
pub fn id(&self) -> [u8; 16] {
self.id
}
/// Header key, for packet engines and protocol tests; never log this value.
pub fn header_key(&self) -> &[u8; 32] {
&self.header_key
}
/// Rejects invalid remote profiles before allocating packets or entering the codec.
pub fn validate(&self) -> Result<(), &'static str> {
if !(12..=48).contains(&self.prefix)
|| !(4..=8).contains(&self.junk_count)
|| self.junk_min < 24
|| self.junk_min > self.junk_max
|| self.junk_max > 160
|| self.padding > 64
|| self.signature > 1
|| self.header_key == [0; 32]
|| self.psk == [0; 32]
{
return Err("invalid AWG profile bounds");
}
for (i, [lo, hi]) in self.headers.iter().copied().enumerate() {
if lo < 5 || lo > hi || hi - lo > 1024 {
return Err("invalid AWG header range");
}
for [other_lo, other_hi] in &self.headers[..i] {
if lo <= *other_hi && *other_lo <= hi {
return Err("overlapping AWG header ranges");
}
}
}
Ok(())
}
fn config(&self) -> AmneziaConfig {
let range = |i: usize| Range {
min: self.headers[i][0],
max: self.headers[i][1],
};
AmneziaConfig {
jc: self.junk_count.into(),
jmin: self.junk_min.into(),
jmax: self.junk_max.into(),
s1: self.prefix.into(),
s2: self.prefix.into(),
s3: self.prefix.into(),
s4: self.prefix.into(),
h1: range(0),
h2: range(1),
h3: range(2),
h4: range(3),
i1: String::new(),
i2: String::new(),
i3: String::new(),
i4: String::new(),
i5: String::new(),
header_protection_key: Some(self.header_key),
content_padding_addition: Range {
min: 0,
max: self.padding.into(),
},
random_trailers: true,
mtu: 1050,
}
}
/// Masks one standard WireGuard message; invalid types and oversized packets are refused.
pub fn encode(&self, raw: &[u8]) -> Option<Bytes> {
let kind = match raw.get(..4)? {
[1, 0, 0, 0] => AwgKind::Initiation,
[2, 0, 0, 0] => AwgKind::Response,
[3, 0, 0, 0] => AwgKind::Cookie,
[4, 0, 0, 0] => AwgKind::Transport,
_ => return None,
};
if raw.len() < kind.fixed_size() || raw.len() > 1050 {
return None;
}
let config = self.config();
let range = match kind {
AwgKind::Initiation => config.h1,
AwgKind::Response => config.h2,
AwgKind::Cookie => config.h3,
AwgKind::Transport => config.h4,
};
let mut payload = raw.to_vec();
payload[..4].copy_from_slice(&range.random().to_le_bytes());
Some(config.encode(kind, Bytes::from(payload), 1200))
}
/// Unmasks a packet and restores standard headers before WireGuard validation.
pub fn decode(&self, wire: &[u8]) -> Option<Vec<u8>> {
if wire.len() > 1200 {
return None;
}
let decoded = self.config().decode(&Bytes::copy_from_slice(wire))?;
let (kind, len) = match decoded.kind {
AwgKind::Initiation => (1, 148),
AwgKind::Response => (2, 92),
AwgKind::Cookie => (3, 64),
AwgKind::Transport => (4, decoded.message.len()),
};
let mut message = decoded.message.get(..len)?.to_vec();
message[..4].copy_from_slice(&(kind as u32).to_le_bytes());
Some(message)
}
pub(crate) fn pad(&self, packet: Bytes) -> Bytes {
self.config().prepare_transport_plaintext(packet, 1200)
}
/// Generates structured signature packets with fresh fields, never static captures.
/// These target the other Tsunagi peer, not DNS or SIP services.
pub fn signature_packets(&self) -> Vec<Vec<u8>> {
if self.signature == 0 {
// A complete DNS query. Counts, label lengths and qtype remain valid.
let label = format!("n{:016x}", rand::random::<u64>());
let mut packet = Vec::from(rand::random::<u16>().to_be_bytes());
packet.extend_from_slice(&[1, 0, 0, 1, 0, 0, 0, 0, 0, 0]);
packet.push(label.len() as u8);
packet.extend_from_slice(label.as_bytes());
packet.extend_from_slice(b"\x07example\x03net\0\0");
packet.push(if rand::random() { 1 } else { 28 });
packet.extend_from_slice(&[0, 1]);
vec![packet]
} else {
let id = rand::random::<u64>();
vec![format!("OPTIONS sip:n{id:x}@example.net SIP/2.0\r\nVia: SIP/2.0/UDP example.net;branch=z9hG4bK{id:x}\r\nMax-Forwards: 70\r\nFrom: <sip:a@example.net>;tag={id:x}\r\nTo: <sip:n{id:x}@example.net>\r\nCall-ID: {id:x}@example.net\r\nCSeq: 1 OPTIONS\r\nContent-Length: 0\r\n\r\n").into_bytes()]
}
}
pub(crate) fn junk_packets(&self) -> Vec<Vec<u8>> {
(0..self.junk_count)
.map(|_| {
let mut packet = vec![0; rand::random_range(self.junk_min..=self.junk_max).into()];
rand::fill(&mut packet[..]);
packet
})
.collect()
}
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
#[test]
fn unsafe_peer_profiles_are_rejected_before_the_codec() {
let valid = PairProfile::generate();
let mut bad = valid.clone();
bad.prefix = 11;
assert!(bad.validate().is_err());
let mut bad = valid.clone();
bad.headers[1] = bad.headers[0];
assert!(bad.validate().is_err());
let mut bad = valid.clone();
bad.junk_max = u16::MAX;
assert!(bad.validate().is_err());
let mut bad = valid.clone();
bad.padding = u16::MAX;
assert!(bad.validate().is_err());
let mut bad = valid.clone();
bad.psk = [0; 32];
assert!(bad.validate().is_err());
let mut bad = valid.clone();
bad.signature = 2;
assert!(bad.validate().is_err());
}
#[test]
fn signature_templates_have_consistent_framing_and_fresh_fields() {
let mut profile = PairProfile::generate();
profile.signature = 0;
let dns = profile.signature_packets().remove(0);
assert_eq!(&dns[2..12], &[1, 0, 0, 1, 0, 0, 0, 0, 0, 0]);
let mut at = 12;
while dns[at] != 0 {
at += 1 + usize::from(dns[at]);
}
assert_eq!(at + 5, dns.len());
assert_eq!(&dns[dns.len() - 2..], &[0, 1]);
profile.signature = 1;
let sip = profile.signature_packets().remove(0);
assert!(sip.starts_with(b"OPTIONS sip:"));
assert!(sip.ends_with(b"Content-Length: 0\r\n\r\n"));
assert_ne!(sip, profile.signature_packets().remove(0));
}
}
+402
View File
@@ -0,0 +1,402 @@
//! One cancellable actor and UDP socket per pair. All payload and health frames
//! are inside the Amnezia-masked authenticated WireGuard tunnel.
use crate::profile::PairProfile;
use boringtun::noise::{Tunn, TunnResult};
use bytes::Bytes;
use iroh::EndpointId;
use std::collections::{HashMap, VecDeque};
use std::net::{Ipv4Addr, SocketAddr};
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};
use tokio::net::UdpSocket;
use tokio::sync::{mpsc, watch};
use tsunagi::dataplane::transport::{PacketLink, TransportError};
use tsunagi::{BoxFuture, NetworkId};
pub(crate) const MAX_PAYLOAD: usize = tsunagi::config::MAX_DATA_DATAGRAM;
const CHUNK: usize = 700;
const QUEUE: usize = 64;
const WINDOW: Duration = Duration::from_secs(20);
pub(crate) struct PairLink {
network: NetworkId,
peer: EndpointId,
pub(crate) attempt: [u8; 16],
tx: mpsc::Sender<Bytes>,
rx: tokio::sync::Mutex<mpsc::Receiver<Bytes>>,
stop: watch::Sender<bool>,
closed: AtomicBool,
remote: Mutex<Option<SocketAddr>>,
reason: Mutex<Option<String>>,
}
impl std::fmt::Debug for PairLink {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("AwgPair")
.field("network", &self.network)
.field("peer", &self.peer)
.finish_non_exhaustive()
}
}
impl PairLink {
pub(crate) fn has_authenticated_path(&self) -> bool {
match self.remote.lock() {
Ok(remote) => remote.is_some(),
Err(_) => false,
}
}
}
impl PacketLink for PairLink {
fn network(&self) -> NetworkId {
self.network
}
fn peer(&self) -> EndpointId {
self.peer
}
fn max_datagram_size(&self) -> usize {
MAX_PAYLOAD
}
fn send(&self, payload: Bytes) -> Result<(), TransportError> {
if self.is_closed() {
return Err(TransportError::Closed);
}
if payload.len() > MAX_PAYLOAD {
return Err(TransportError::TooLarge {
size: payload.len(),
limit: MAX_PAYLOAD,
});
}
self.tx
.try_send(payload)
.map_err(|_| TransportError::Other("AWG send queue full or closed".into()))
}
fn recv(&self) -> BoxFuture<'_, Option<Bytes>> {
Box::pin(async move {
let mut stop = self.stop.subscribe();
if *stop.borrow() {
return None;
}
let mut rx = self.rx.lock().await;
tokio::select! {biased; _=stop.changed()=>None,packet=rx.recv()=>packet}
})
}
fn closed(&self) -> BoxFuture<'_, ()> {
Box::pin(async move {
let mut stop = self.stop.subscribe();
if !*stop.borrow() {
let _ = stop.changed().await;
}
})
}
fn is_closed(&self) -> bool {
self.closed.load(Ordering::Acquire) || *self.stop.borrow()
}
fn close(&self) {
self.stop.send_replace(true);
}
fn failure_reason(&self) -> Option<String> {
match self.reason.lock() {
Ok(reason) => reason.clone(),
Err(_) => None,
}
}
fn path_description(&self) -> String {
match self.remote.lock() {
Ok(addr) => match *addr {
Some(addr) => format!("awg UDP via {addr}"),
None => "awg UDP, establishing".into(),
},
Err(_) => "awg UDP".into(),
}
}
}
impl Drop for PairLink {
fn drop(&mut self) {
self.stop.send_replace(true);
}
}
pub(crate) fn start(
network: NetworkId,
peer: EndpointId,
socket: UdpSocket,
tunn: Tunn,
profile: PairProfile,
candidates: Vec<SocketAddr>,
parent: watch::Receiver<bool>,
) -> Arc<PairLink> {
let (tx, rx) = mpsc::channel(QUEUE);
let (in_tx, in_rx) = mpsc::channel(QUEUE);
let (stop, _) = watch::channel(false);
let link = Arc::new(PairLink {
network,
peer,
attempt: profile.id(),
tx,
rx: tokio::sync::Mutex::new(in_rx),
stop: stop.clone(),
closed: AtomicBool::new(false),
remote: Mutex::new(None),
reason: Mutex::new(None),
});
let weak = Arc::downgrade(&link);
tokio::spawn(async move {
let mut engine = Engine {
stop: stop.subscribe(),
socket,
tunn,
profile,
candidates,
remote: None,
inbox: in_tx,
rx,
counter: rand::random(),
sequence: 0,
pending: VecDeque::new(),
samples: VecDeque::new(),
assemblies: HashMap::new(),
started: Instant::now(),
last_authenticated: Instant::now(),
};
engine.run(parent, &weak).await;
if let Some(link) = weak.upgrade() {
link.closed.store(true, Ordering::Release);
link.stop.send_replace(true);
}
});
link
}
struct Assembly {
created: Instant,
count: u16,
chunks: Vec<Option<Bytes>>,
}
struct Engine {
stop: watch::Receiver<bool>,
socket: UdpSocket,
tunn: Tunn,
profile: PairProfile,
candidates: Vec<SocketAddr>,
remote: Option<SocketAddr>,
inbox: mpsc::Sender<Bytes>,
rx: mpsc::Receiver<Bytes>,
counter: u64,
sequence: u64,
pending: VecDeque<(u64, Instant)>,
samples: VecDeque<bool>,
assemblies: HashMap<u64, Assembly>,
started: Instant,
last_authenticated: Instant,
}
enum ResultOwned {
Wire(Vec<u8>),
Plain(Vec<u8>),
Done,
}
fn own(result: TunnResult<'_>) -> ResultOwned {
match result {
TunnResult::WriteToNetwork(p) => ResultOwned::Wire(p.to_vec()),
TunnResult::WriteToTunnelV4(p, _) => ResultOwned::Plain(p.to_vec()),
_ => ResultOwned::Done,
}
}
fn ip(payload: &[u8]) -> Bytes {
let mut packet = vec![0; 20];
packet[0] = 0x45;
packet[8] = 64;
packet[9] = 253;
packet[12..16].copy_from_slice(&Ipv4Addr::new(192, 0, 2, 1).octets());
packet[16..20].copy_from_slice(&Ipv4Addr::new(192, 0, 2, 2).octets());
packet[2..4].copy_from_slice(&((20 + payload.len()) as u16).to_be_bytes());
let sum = packet
.as_chunks::<2>()
.0
.iter()
.map(|b| u32::from(u16::from_be_bytes([b[0], b[1]])))
.sum::<u32>();
let sum = (sum & 0xffff) + (sum >> 16);
packet[10..12].copy_from_slice(&(!(sum as u16)).to_be_bytes());
packet.extend_from_slice(payload);
Bytes::from(packet)
}
impl Engine {
async fn run(&mut self, mut stop: watch::Receiver<bool>, link: &std::sync::Weak<PairLink>) {
if *self.stop.borrow() || *stop.borrow() {
return;
}
let mut buffer = [0; 1201];
let mut scratch = [0; 4096];
let mut timer = tokio::time::interval(Duration::from_millis(250));
timer.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
let mut last_probe = Instant::now() - Duration::from_secs(2);
loop {
tokio::select! {
biased;
_=stop.changed()=>break,
_=self.stop.changed()=>break,
_=timer.tick()=>{
let result=own(self.tunn.update_timers(&mut scratch));self.process(result,None).await;
if last_probe.elapsed()>=Duration::from_secs(1) {
last_probe=Instant::now();self.sequence=self.sequence.wrapping_add(1);
self.pending.push_back((self.sequence,Instant::now()));
let mut probe=vec![1];probe.extend_from_slice(&self.sequence.to_be_bytes());
probe.resize(713,0);rand::fill(&mut probe[9..]);self.encrypt(&probe).await;
}
while self.pending.front().is_some_and(|(_,at)|at.elapsed()>=Duration::from_secs(3)) {
self.pending.pop_front();self.samples.push_back(false);
}
while self.samples.len()>20 {self.samples.pop_front();}
self.assemblies.retain(|_,a|a.created.elapsed()<Duration::from_secs(5));
if self.started.elapsed()>WINDOW && (self.last_authenticated.elapsed()>Duration::from_secs(10)
|| (self.samples.len()>=15&&self.samples.iter().filter(|ok|!**ok).count()*100>=self.samples.len()*60)) {
if let Some(link)=link.upgrade() && let Ok(mut reason)=link.reason.lock() {*reason=Some("AWG data delivery checks: persistent loss or timeout".into());}
break;
}
}
packet=self.socket.recv_from(&mut buffer)=>{
if let Ok((len,source))=packet && let Some(message)=self.profile.decode(&buffer[..len]) {
let result=own(self.tunn.decapsulate(None,&message,&mut scratch));
if matches!(result,ResultOwned::Plain(_)) {
self.remote=Some(source);self.last_authenticated=Instant::now();
if let Some(link)=link.upgrade() && let Ok(mut remote)=link.remote.lock(){*remote=Some(source);}
}
self.process(result,Some(source)).await;
for _ in 0..16 {
let result=own(self.tunn.decapsulate(None,&[],&mut scratch));
if matches!(result,ResultOwned::Done){break;}
self.process(result,Some(source)).await;
}
}
}
packet=self.rx.recv()=>{
let Some(packet)=packet else{break;};
let count=packet.len().max(1).div_ceil(CHUNK) as u16;
self.counter=self.counter.wrapping_add(1);
for (index,chunk) in packet.chunks(CHUNK).enumerate() {
let mut payload=vec![3];payload.extend_from_slice(&self.counter.to_be_bytes());
payload.extend_from_slice(&(index as u16).to_be_bytes());payload.extend_from_slice(&count.to_be_bytes());
payload.extend_from_slice(chunk);self.encrypt(&payload).await;
// Large logical datagrams must let local readers and other
// pairs drain their bounded socket/command queues.
if (index + 1).is_multiple_of(16) {tokio::task::yield_now().await;}
}
}
}
}
}
async fn encrypt(&mut self, payload: &[u8]) {
let packet = self.profile.pad(ip(payload));
let mut scratch = [0; 4096];
let result = own(self.tunn.encapsulate(&packet, &mut scratch));
self.process(result, None).await;
}
async fn wire(&self, raw: &[u8], target: Option<SocketAddr>) {
let Some(wire) = self.profile.encode(raw) else {
return;
};
let targets: Vec<_> = target
.or(self.remote)
.map(|a| vec![a])
.unwrap_or_else(|| self.candidates.clone());
for addr in targets {
if raw.first() == Some(&1) {
for signature in self.profile.signature_packets() {
let _ = self.send_wire(&signature, addr).await;
}
for junk in self.profile.junk_packets() {
let _ = self.send_wire(&junk, addr).await;
}
}
let _ = self.send_wire(&wire, addr).await;
}
}
async fn send_wire(&self, bytes: &[u8], address: SocketAddr) {
let mut stop = self.stop.clone();
if *stop.borrow() {
return;
}
tokio::select! {biased; _=stop.changed()=>{}, _=tokio::time::timeout(Duration::from_millis(100),self.socket.send_to(bytes,address))=>{}}
}
async fn process(&mut self, result: ResultOwned, target: Option<SocketAddr>) {
match result {
ResultOwned::Wire(raw) => self.wire(&raw, target).await,
ResultOwned::Plain(packet) => {
let Some(payload) = packet.get(20..) else {
return;
};
let Some(seq) = payload
.get(1..9)
.and_then(|b| b.try_into().ok())
.map(u64::from_be_bytes)
else {
return;
};
match payload.first() {
Some(1) => {
let mut pong = vec![2];
pong.extend_from_slice(&seq.to_be_bytes());
pong.resize(713, 0);
let packet = self.profile.pad(ip(&pong));
let mut scratch = [0; 4096];
if let ResultOwned::Wire(raw) =
own(self.tunn.encapsulate(&packet, &mut scratch))
{
self.wire(&raw, target).await;
}
}
Some(2) => {
if let Some(index) =
self.pending.iter().position(|(pending, _)| *pending == seq)
{
self.pending.remove(index);
self.samples.push_back(true);
}
}
Some(3) => self.fragment(seq, payload),
_ => {}
}
}
ResultOwned::Done => {}
}
}
fn fragment(&mut self, id: u64, payload: &[u8]) {
let Some(header) = payload.get(9..13) else {
return;
};
let index = u16::from_be_bytes([header[0], header[1]]);
let count = u16::from_be_bytes([header[2], header[3]]);
let chunk = &payload[13..];
if count == 0
|| usize::from(count) > MAX_PAYLOAD.div_ceil(CHUNK)
|| index >= count
|| chunk.len() > CHUNK
{
return;
}
if !self.assemblies.contains_key(&id) && self.assemblies.len() >= 8 {
return;
}
let assembly = self.assemblies.entry(id).or_insert_with(|| Assembly {
created: Instant::now(),
count,
chunks: vec![None; count.into()],
});
if assembly.count != count {
return;
}
assembly.chunks[usize::from(index)] = Some(Bytes::copy_from_slice(chunk));
if assembly.chunks.iter().all(Option::is_some)
&& let Some(assembly) = self.assemblies.remove(&id)
{
let mut packet = Vec::new();
for chunk in assembly.chunks.into_iter().flatten() {
packet.extend_from_slice(&chunk);
}
if packet.len() <= MAX_PAYLOAD {
let _ = self.inbox.try_send(Bytes::from(packet));
}
}
}
}
+101
View File
@@ -0,0 +1,101 @@
//! Optional, bounded server-reflexive discovery on the pair's own UDP socket.
//! A STUN answer is a candidate only; WireGuard authentication verifies a path.
use std::net::{IpAddr, Ipv4Addr, SocketAddr};
use std::time::Duration;
use tokio::net::UdpSocket;
const COOKIE: [u8; 4] = [0x21, 0x12, 0xa4, 0x42];
pub(crate) async fn mapped(socket: &UdpSocket, server: &str) -> Option<SocketAddr> {
tokio::time::timeout(Duration::from_millis(1200), async {
let address = tokio::net::lookup_host(server)
.await
.ok()?
.find(SocketAddr::is_ipv4)?;
let transaction: [u8; 12] = rand::random();
let mut request = [0; 20];
request[..2].copy_from_slice(&[0, 1]);
request[4..8].copy_from_slice(&COOKIE);
request[8..].copy_from_slice(&transaction);
socket.send_to(&request, address).await.ok()?;
let mut bytes = [0; 1024];
loop {
let (len, source) = socket.recv_from(&mut bytes).await.ok()?;
if source == address
&& let Some(mapped) = parse(&bytes[..len], &transaction)
{
return Some(mapped);
}
}
})
.await
.ok()
.flatten()
}
fn parse(bytes: &[u8], transaction: &[u8; 12]) -> Option<SocketAddr> {
if bytes.len() < 20
|| bytes[..2] != [1, 1]
|| bytes[4..8] != COOKIE
|| &bytes[8..20] != transaction
{
return None;
}
let declared = u16::from_be_bytes(bytes[2..4].try_into().ok()?) as usize;
if !declared.is_multiple_of(4) || declared != bytes.len() - 20 {
return None;
}
let mut at = 20;
while at + 4 <= bytes.len() {
let kind = u16::from_be_bytes(bytes[at..at + 2].try_into().ok()?);
let len = u16::from_be_bytes(bytes[at + 2..at + 4].try_into().ok()?) as usize;
at += 4;
let attribute = bytes.get(at..at.checked_add(len)?)?;
if kind == 0x0020 && attribute.len() == 8 && attribute[0..2] == [0, 1] {
let port = u16::from_be_bytes([attribute[2] ^ COOKIE[0], attribute[3] ^ COOKIE[1]]);
let ip = Ipv4Addr::new(
attribute[4] ^ COOKIE[0],
attribute[5] ^ COOKIE[1],
attribute[6] ^ COOKIE[2],
attribute[7] ^ COOKIE[3],
);
if port == 0 || ip.is_unspecified() || ip.is_multicast() || ip.is_broadcast() {
return None;
}
return Some(SocketAddr::new(IpAddr::V4(ip), port));
}
at = at.checked_add((len + 3) & !3)?;
}
None
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
#[tokio::test]
async fn mapping_uses_the_actual_socket_and_rejects_unrelated_or_truncated_answers() {
let server = UdpSocket::bind((Ipv4Addr::LOCALHOST, 0)).await.unwrap();
let address = server.local_addr().unwrap();
let socket = UdpSocket::bind((Ipv4Addr::LOCALHOST, 0)).await.unwrap();
let expected = socket.local_addr().unwrap();
let task = tokio::spawn(async move {
let mut request = [0; 128];
let (len, source) = server.recv_from(&mut request).await.unwrap();
assert_eq!(len, 20);
assert_eq!(source, expected);
let tx: [u8; 12] = request[8..20].try_into().unwrap();
let mut reply = request[..20].to_vec();
reply[..4].copy_from_slice(&[1, 1, 0, 12]);
reply.extend_from_slice(&[0, 0x20, 0, 8, 0, 1]);
let port = source.port().to_be_bytes();
reply.extend_from_slice(&[port[0] ^ COOKIE[0], port[1] ^ COOKIE[1]]);
for (i, b) in Ipv4Addr::LOCALHOST.octets().into_iter().enumerate() {
reply.push(b ^ COOKIE[i]);
}
assert!(parse(&reply, &[0; 12]).is_none());
for len in 0..reply.len() {
assert!(parse(&reply[..len], &tx).is_none());
}
server.send_to(&reply, source).await.unwrap();
});
assert_eq!(mapped(&socket, &address.to_string()).await, Some(expected));
task.await.unwrap();
}
}
+720
View File
@@ -0,0 +1,720 @@
//! Private pair negotiation. No secret profiles in capabilities or durable state.
use crate::{
AWG_PROTOCOL, AWG_WIRE_VERSION,
profile::PairProfile,
session::{self, PairLink},
};
use boringtun::{
noise::Tunn,
x25519::{PublicKey, StaticSecret},
};
use iroh::EndpointId;
use serde::{Deserialize, Serialize};
use std::collections::{HashMap, HashSet};
use std::net::{Ipv4Addr, SocketAddr};
use std::sync::{Arc, Mutex, OnceLock, Weak};
use std::time::Duration;
use tokio::net::UdpSocket;
use tokio::sync::{mpsc, oneshot, watch};
use tsunagi::dataplane::transport::TransportPayload;
use tsunagi::dataplane::transport::{
ControlSender, InboundLink, PacketLink, PacketTransport, SharedLink, TransportError,
};
use tsunagi::{BoxFuture, NetworkId};
type Pair = (NetworkId, EndpointId);
#[derive(Serialize, Deserialize)]
enum Negotiation {
Offer {
version: u16,
profile: PairProfile,
key: [u8; 32],
#[serde(deserialize_with = "bounded_addresses")]
addresses: Vec<SocketAddr>,
},
Answer {
attempt: [u8; 16],
key: [u8; 32],
#[serde(deserialize_with = "bounded_addresses")]
addresses: Vec<SocketAddr>,
},
}
fn bounded_addresses<'de, D: serde::Deserializer<'de>>(
deserializer: D,
) -> Result<Vec<SocketAddr>, D::Error> {
struct Visitor;
impl<'de> serde::de::Visitor<'de> for Visitor {
type Value = Vec<SocketAddr>;
fn expecting(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str("at most 32 UDP candidates")
}
fn visit_seq<A: serde::de::SeqAccess<'de>>(
self,
mut sequence: A,
) -> Result<Self::Value, A::Error> {
if sequence.size_hint().is_some_and(|n| n > 32) {
return Err(serde::de::Error::custom("too many UDP candidates"));
}
let mut addresses = Vec::with_capacity(sequence.size_hint().unwrap_or(0).min(32));
while let Some(address) = sequence.next_element()? {
if addresses.len() == 32 {
return Err(serde::de::Error::custom("too many UDP candidates"));
}
addresses.push(address);
}
Ok(addresses)
}
}
deserializer.deserialize_seq(Visitor)
}
struct Pending {
profile: PairProfile,
key: StaticSecret,
socket: UdpSocket,
reply: oneshot::Sender<Result<SharedLink, TransportError>>,
}
struct Inner {
stopped: watch::Sender<bool>,
local: EndpointId,
bind: Ipv4Addr,
stun: Option<String>,
busy: Mutex<HashSet<Pair>>,
control: OnceLock<Arc<dyn ControlSender>>,
allowed: Mutex<HashSet<Pair>>,
pending: Mutex<HashMap<Pair, Pending>>,
links: Mutex<HashMap<Pair, Arc<PairLink>>>,
inbound: mpsc::Sender<InboundLink>,
port_by_pair: Mutex<HashMap<Pair, u16>>,
}
/// Public diagnostics for one pair. No key or masking parameters are exposed.
#[derive(Debug, Clone)]
pub struct PairOverview {
/// Other endpoint's authenticated identity.
pub peer: EndpointId,
/// Fresh connection-attempt identifier.
pub attempt: [u8; 16],
/// Last known UDP port; None if diagnostics race with pair retirement.
pub local_port: Option<u16>,
/// Actual transport path, if established.
pub path: String,
/// Whether this attempt has been retired.
pub closed: bool,
}
/// A bounded set of independently configured pair engines.
pub struct AwgTransport {
inner: Arc<Inner>,
commands: mpsc::Sender<(NetworkId, EndpointId, Negotiation)>,
incoming: tokio::sync::Mutex<mpsc::Receiver<InboundLink>>,
}
impl std::fmt::Debug for AwgTransport {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("AwgTransport")
.field("local", &self.inner.local)
.finish_non_exhaustive()
}
}
fn lock<T>(mutex: &Mutex<T>) -> std::sync::MutexGuard<'_, T> {
match mutex.lock() {
Ok(g) => g,
Err(p) => p.into_inner(),
}
}
fn error(text: &str) -> TransportError {
TransportError::Other(text.into())
}
async fn addresses(inner: &Inner, socket: &UdpSocket) -> Result<Vec<SocketAddr>, TransportError> {
let local = socket
.local_addr()
.map_err(|_| error("AWG socket address unavailable"))?;
let mut candidates = if inner.bind.is_loopback() {
vec![local]
} else {
tsunagi_wireguard::local_ip_candidates()
.into_iter()
.filter(|ip| ip.is_ipv4())
.take(31)
.map(|ip| SocketAddr::new(ip, local.port()))
.collect()
};
if let Some(server) = &inner.stun
&& let Some(mapped) = crate::stun::mapped(socket, server).await
&& !candidates.contains(&mapped)
{
candidates.push(mapped);
}
if candidates.is_empty() {
return Err(error("AWG has no UDP address candidates"));
}
Ok(candidates)
}
fn validate_endpoint(key: &[u8; 32], addresses: &[SocketAddr]) -> Result<(), TransportError> {
if *key == [0; 32]
|| addresses.is_empty()
|| addresses.len() > 32
|| addresses.iter().any(|address| {
address.port() == 0
|| !address.is_ipv4()
|| address.ip().is_unspecified()
|| address.ip().is_multicast()
|| matches!(address.ip(), std::net::IpAddr::V4(ip) if ip.is_broadcast())
})
{
return Err(error("invalid AWG endpoint"));
}
Ok(())
}
impl AwgTransport {
/// Starts the coordinator. Production binds wildcard IPv4; tests use loopback.
pub fn new(local: EndpointId, bind: Ipv4Addr) -> Arc<Self> {
Self::new_with_stun(local, bind, None)
}
/// Enables an optional STUN server (host:port) for this pair's own socket.
/// Local-only callers pass None; failure still permits local UDP candidates.
pub fn new_with_stun(local: EndpointId, bind: Ipv4Addr, stun: Option<String>) -> Arc<Self> {
let (commands, mut rx) = mpsc::channel(64);
let (inbound, incoming) = mpsc::channel(64);
let (stopped, _) = watch::channel(false);
let inner = Arc::new(Inner {
stopped,
local,
bind,
stun,
busy: Mutex::default(),
control: OnceLock::new(),
allowed: Mutex::default(),
pending: Mutex::default(),
links: Mutex::default(),
inbound,
port_by_pair: Mutex::default(),
});
let weak = Arc::downgrade(&inner);
let mut stopped = inner.stopped.subscribe();
tokio::spawn(async move {
while let Some((network, peer, message)) =
tokio::select! { biased; _=stopped.changed()=>None, message=rx.recv()=>message }
{
let Some(inner) = weak.upgrade() else {
break;
};
if !lock(&inner.allowed).contains(&(network, peer))
|| !lock(&inner.busy).insert((network, peer))
{
continue;
}
tokio::spawn(async move {
let mut stop = inner.stopped.subscribe();
tokio::select! {biased; _=stop.changed()=>{}, _=async {
if let Err(err)=receive(&inner,(network,peer),message).await {tracing::debug!(%err,"AWG pair negotiation rejected");}
}=>{}}
lock(&inner.busy).remove(&(network, peer));
});
}
});
Arc::new(Self {
inner,
commands,
incoming: tokio::sync::Mutex::new(incoming),
})
}
/// Registers a peer after its authenticated capability has been validated.
pub fn allow_peer(&self, network: NetworkId, peer: EndpointId) {
let mut allowed = lock(&self.inner.allowed);
if !*self.inner.stopped.borrow() && allowed.len() < 128 {
allowed.insert((network, peer));
}
}
/// Retires a pair when its network membership session goes away.
pub fn forget_peer(&self, network: NetworkId, peer: EndpointId) {
let pair = (network, peer);
lock(&self.inner.allowed).remove(&pair);
lock(&self.inner.pending).remove(&pair);
lock(&self.inner.port_by_pair).remove(&pair);
if let Some(link) = lock(&self.inner.links).remove(&pair) {
link.close();
}
}
/// Stops all pair engines in this network.
pub fn stop_network(&self, network: NetworkId) {
let peers: Vec<_> = lock(&self.inner.allowed)
.iter()
.filter(|(n, _)| *n == network)
.map(|(_, p)| *p)
.collect();
for peer in peers {
self.forget_peer(network, peer);
}
}
/// Stops all sockets and coordinator work without waiting for remote peers.
pub fn shutdown(&self) {
self.inner.stopped.send_replace(true);
lock(&self.inner.allowed).clear();
lock(&self.inner.pending).clear();
for (_, link) in lock(&self.inner.links).drain() {
link.close();
}
lock(&self.inner.port_by_pair).clear();
}
/// Receives a successfully negotiated and authenticated inbound pair link.
pub async fn next_inbound(&self) -> Option<InboundLink> {
let mut stopped = self.inner.stopped.subscribe();
if *stopped.borrow() {
return None;
}
let mut incoming = self.incoming.lock().await;
tokio::select! {biased; _=stopped.changed()=>None, link=incoming.recv()=>link}
}
/// Takes a public snapshot of this network's pair attempts.
pub fn overview(&self, network: NetworkId) -> Vec<PairOverview> {
let links = lock(&self.inner.links);
let ports = lock(&self.inner.port_by_pair);
links
.iter()
.filter(|((n, _), _)| *n == network)
.map(|((_, peer), link)| PairOverview {
peer: *peer,
attempt: link.attempt,
local_port: ports.get(&(network, *peer)).copied(),
path: link.path_description(),
closed: link.is_closed(),
})
.collect()
}
}
impl PacketTransport for AwgTransport {
fn name(&self) -> &str {
AWG_PROTOCOL
}
fn as_any(&self) -> &dyn std::any::Any {
self
}
fn attach_control(&self, control: Arc<dyn ControlSender>) {
let _ = self.inner.control.set(control);
}
fn on_control(
&self,
network: NetworkId,
peer: EndpointId,
protocol: &str,
payload: &[u8],
) -> Result<(), TransportError> {
if protocol != AWG_PROTOCOL || !lock(&self.inner.allowed).contains(&(network, peer)) {
return Err(TransportError::Declined(protocol.into()));
}
if payload.len() > 4096 {
return Err(error("AWG negotiation too large"));
}
let message: Negotiation =
postcard::from_bytes(payload).map_err(|_| error("malformed AWG negotiation"))?;
match &message {
Negotiation::Offer {
version,
profile,
key,
addresses,
} => {
if *version != AWG_WIRE_VERSION || peer.as_bytes() >= self.inner.local.as_bytes() {
return Err(error("invalid AWG offer role/version"));
}
profile.validate().map_err(error)?;
validate_endpoint(key, addresses)?;
}
Negotiation::Answer { key, addresses, .. } => validate_endpoint(key, addresses)?,
}
self.commands
.try_send((network, peer, message))
.map_err(|_| error("AWG negotiation queue full"))
}
fn open<'a>(
&'a self,
network: NetworkId,
peer: EndpointId,
protocol: &'a str,
) -> BoxFuture<'a, Result<SharedLink, TransportError>> {
Box::pin(async move {
let pair = (network, peer);
if protocol != AWG_PROTOCOL
|| self.inner.local.as_bytes() >= peer.as_bytes()
|| !lock(&self.inner.allowed).contains(&pair)
{
return Err(TransportError::Declined(protocol.into()));
}
let control = self
.inner
.control
.get()
.ok_or_else(|| error("AWG control plane unavailable"))?;
let socket = UdpSocket::bind((self.inner.bind, 0))
.await
.map_err(|_| error("AWG UDP bind failed"))?;
let profile = PairProfile::generate();
let attempt = profile.id();
let key = StaticSecret::from(rand::random::<[u8; 32]>());
let public = PublicKey::from(&key).to_bytes();
let message = Negotiation::Offer {
version: AWG_WIRE_VERSION,
profile: profile.clone(),
key: public,
addresses: addresses(&self.inner, &socket).await?,
};
let (reply, rx) = oneshot::channel();
if lock(&self.inner.pending).len() >= 128 {
return Err(error("too many AWG pair negotiations"));
}
lock(&self.inner.pending).insert(
pair,
Pending {
profile,
key,
socket,
reply,
},
);
// A cancellation guard also retires partially started attempts.
let mut guard = AttemptGuard {
inner: Arc::downgrade(&self.inner),
pair,
attempt,
completed: false,
};
let bytes =
postcard::to_stdvec(&message).map_err(|_| error("AWG offer encoding failed"))?;
control
.send(network, peer, AWG_PROTOCOL, TransportPayload::new(bytes))
.await?;
let result = tokio::time::timeout(Duration::from_secs(15), rx)
.await
.map_err(|_| error("AWG negotiation/handshake timed out"))?
.map_err(|_| error("AWG attempt cancelled"))?;
guard.completed = result.is_ok();
result
})
}
}
struct AttemptGuard {
inner: Weak<Inner>,
pair: Pair,
attempt: [u8; 16],
completed: bool,
}
impl Drop for AttemptGuard {
fn drop(&mut self) {
if let Some(inner) = self.inner.upgrade() {
let mut pending = lock(&inner.pending);
if pending
.get(&self.pair)
.is_some_and(|p| p.profile.id() == self.attempt)
{
pending.remove(&self.pair);
}
drop(pending);
if !self.completed
&& let Some(link) = lock(&inner.links).get(&self.pair)
&& link.attempt == self.attempt
{
link.close();
}
}
}
}
async fn receive(
inner: &Arc<Inner>,
pair: Pair,
message: Negotiation,
) -> Result<(), TransportError> {
if !lock(&inner.allowed).contains(&pair) {
return Err(error("AWG peer no longer authenticated"));
}
match message {
Negotiation::Offer {
profile,
key,
addresses: candidates,
..
} => {
if let Some(link) = lock(&inner.links).get(&pair) {
if link.attempt == profile.id() {
return Err(error("duplicate AWG offer"));
}
link.close();
}
let socket = UdpSocket::bind((inner.bind, 0))
.await
.map_err(|_| error("AWG UDP bind failed"))?;
let local_port = socket
.local_addr()
.map_err(|_| error("AWG socket address unavailable"))?
.port();
let local_key = StaticSecret::from(rand::random::<[u8; 32]>());
let public = PublicKey::from(&local_key).to_bytes();
let answer = Negotiation::Answer {
attempt: profile.id(),
key: public,
addresses: addresses(inner, &socket).await?,
};
let tunn = Tunn::new(
local_key,
PublicKey::from(key),
Some(profile.psk),
Some(1),
rand::random(),
None,
);
let link = session::start(
pair.0,
pair.1,
socket,
tunn,
profile,
candidates,
inner.stopped.subscribe(),
);
if !lock(&inner.allowed).contains(&pair) {
link.close();
return Err(error("AWG membership ended"));
}
lock(&inner.links).insert(pair, link.clone());
lock(&inner.port_by_pair).insert(pair, local_port);
let control = inner
.control
.get()
.ok_or_else(|| error("AWG control unavailable"))?;
if let Err(err) = control
.send(
pair.0,
pair.1,
AWG_PROTOCOL,
TransportPayload::new(
postcard::to_stdvec(&answer)
.map_err(|_| error("AWG answer encoding failed"))?,
),
)
.await
{
link.close();
return Err(err);
}
publish_when_up(Arc::downgrade(inner), pair, link, None);
}
Negotiation::Answer {
attempt,
key,
addresses: candidates,
} => {
if !lock(&inner.pending)
.get(&pair)
.is_some_and(|p| p.profile.id() == attempt)
{
return Err(error("stale AWG answer"));
}
let pending = lock(&inner.pending)
.remove(&pair)
.ok_or_else(|| error("AWG attempt cancelled"))?;
let local_port = pending
.socket
.local_addr()
.map_err(|_| error("AWG socket address unavailable"))?
.port();
let tunn = Tunn::new(
pending.key,
PublicKey::from(key),
Some(pending.profile.psk),
Some(1),
rand::random(),
None,
);
let link = session::start(
pair.0,
pair.1,
pending.socket,
tunn,
pending.profile,
candidates,
inner.stopped.subscribe(),
);
if !lock(&inner.allowed).contains(&pair) {
link.close();
return Err(error("AWG membership ended"));
}
lock(&inner.links).insert(pair, link.clone());
lock(&inner.port_by_pair).insert(pair, local_port);
publish_when_up(Arc::downgrade(inner), pair, link, Some(pending.reply));
}
}
Ok(())
}
fn publish_when_up(
inner: Weak<Inner>,
pair: Pair,
link: Arc<PairLink>,
reply: Option<oneshot::Sender<Result<SharedLink, TransportError>>>,
) {
tokio::spawn(async move {
let mut reply = reply;
let result = tokio::time::timeout(Duration::from_secs(14), async {
loop {
if link.is_closed() || reply.as_ref().is_some_and(oneshot::Sender::is_closed) {
return Err(error("AWG pair closed during handshake"));
}
if link.has_authenticated_path() {
return Ok(());
}
tokio::time::sleep(Duration::from_millis(25)).await;
}
})
.await;
if !matches!(result, Ok(Ok(()))) {
link.close();
if let Some(reply) = reply.take() {
let _ = reply.send(Err(error("AWG authenticated data path timed out")));
}
return;
}
let Some(inner) = inner.upgrade() else {
link.close();
return;
};
if !lock(&inner.allowed).contains(&pair) {
link.close();
return;
}
if link.is_closed()
|| !lock(&inner.links)
.get(&pair)
.is_some_and(|current| current.attempt == link.attempt)
{
link.close();
return;
}
let shared: SharedLink = link.clone();
if let Some(reply) = reply {
if reply.send(Ok(shared)).is_err() {
link.close();
}
} else if inner
.inbound
.try_send(InboundLink {
network: pair.0,
peer: pair.1,
protocol: AWG_PROTOCOL.into(),
link: shared,
})
.is_err()
{
link.close();
}
});
}
impl Drop for AwgTransport {
fn drop(&mut self) {
self.shutdown();
}
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
#[tokio::test]
async fn concurrent_admission_is_bounded_across_networks_and_retirement_releases_slots() {
let identity = tsunagi::identity::DeviceIdentity::generate();
let transport = AwgTransport::new(identity.endpoint_id(), Ipv4Addr::LOCALHOST);
let peer = tsunagi::identity::DeviceIdentity::generate().endpoint_id();
std::thread::scope(|scope| {
for thread in 0..8u8 {
let transport = &transport;
scope.spawn(move || {
for item in 0..32u8 {
let mut id = [0; 32];
id[0] = thread;
id[1] = item;
transport.allow_peer(NetworkId::from_bytes(id), peer);
}
});
}
});
assert_eq!(lock(&transport.inner.allowed).len(), 128);
let network = lock(&transport.inner.allowed).iter().next().unwrap().0;
transport.stop_network(network);
let new = NetworkId::from_bytes([255; 32]);
transport.allow_peer(new, peer);
assert!(lock(&transport.inner.allowed).contains(&(new, peer)));
assert_eq!(lock(&transport.inner.allowed).len(), 128);
transport.shutdown();
assert!(lock(&transport.inner.allowed).is_empty());
}
#[tokio::test]
async fn private_negotiation_bounds_roles_and_membership_before_starting_engines() {
let mut ids = [
tsunagi::identity::DeviceIdentity::generate().endpoint_id(),
tsunagi::identity::DeviceIdentity::generate().endpoint_id(),
];
ids.sort_by_key(|id| *id.as_bytes());
let peer = ids[0];
let local = ids[1];
let network = NetworkId::from_bytes([7; 32]);
let transport = AwgTransport::new(local, Ipv4Addr::LOCALHOST);
let offer = |version, key, addresses| Negotiation::Offer {
version,
profile: PairProfile::generate(),
key,
addresses,
};
let address = SocketAddr::from((Ipv4Addr::LOCALHOST, 12345));
let encode = |message: &Negotiation| postcard::to_stdvec(message).unwrap();
let valid = encode(&offer(AWG_WIRE_VERSION, [1; 32], vec![address]));
assert!(
transport
.on_control(network, peer, AWG_PROTOCOL, &valid)
.is_err()
);
transport.allow_peer(network, peer);
assert!(
transport
.on_control(network, peer, "wrong-protocol", &valid)
.is_err()
);
assert!(
transport
.on_control(network, peer, AWG_PROTOCOL, &[0; 4097])
.is_err()
);
for bytes in [
encode(&offer(99, [1; 32], vec![address])),
encode(&offer(AWG_WIRE_VERSION, [0; 32], vec![address])),
encode(&offer(AWG_WIRE_VERSION, [1; 32], vec![])),
encode(&offer(AWG_WIRE_VERSION, [1; 32], vec![address; 33])),
] {
assert!(
transport
.on_control(network, peer, AWG_PROTOCOL, &bytes)
.is_err()
);
}
for len in 0..valid.len() {
assert!(
transport
.on_control(network, peer, AWG_PROTOCOL, &valid[..len])
.is_err()
);
}
let wrong_role = AwgTransport::new(peer, Ipv4Addr::LOCALHOST);
wrong_role.allow_peer(network, local);
assert!(
wrong_role
.on_control(network, local, AWG_PROTOCOL, &valid)
.is_err()
);
transport.forget_peer(network, peer);
assert!(
transport
.on_control(network, peer, AWG_PROTOCOL, &valid)
.is_err()
);
assert!(transport.overview(network).is_empty());
transport.shutdown();
wrong_role.shutdown();
}
}
+773
View File
@@ -0,0 +1,773 @@
//! Real membership, UDP sockets, pair negotiation and nested WireGuard encryption.
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use bytes::Bytes;
use std::{
net::Ipv4Addr,
path::Path,
sync::{
Arc,
atomic::{AtomicBool, AtomicU64, Ordering},
},
time::Duration,
};
use tempfile::TempDir;
use tsunagi::dataplane::IpPlugin;
use tsunagi::dataplane::transport::{
ControlSender, PacketLink, PacketTransport, SharedLink, TransportError,
};
use tsunagi::discovery::SharedMemoryDiscovery;
use tsunagi::overlay::{MemoryTun, MemoryTunFactory};
use tsunagi::testing::{config_with, network, wait_for_peers, wait_until};
use tsunagi::{Agent, BoxFuture, NetworkId};
use tsunagi_awg::{AWG_PROTOCOL, AwgPlugin, AwgTransport};
use tsunagi_wg_quic::WireguardConfig;
struct Node {
agent: Agent,
plugin: Arc<AwgPlugin>,
quic: Option<Arc<tsunagi_wg_quic::WireguardPlugin>>,
transport: Arc<AwgTransport>,
tuns: MemoryTunFactory,
loss: Arc<AtomicBool>,
}
impl Node {
async fn open(root: &Path, discovery: &SharedMemoryDiscovery, interface: &str) -> Self {
Self::open_with_fallback(root, discovery, interface, false).await
}
async fn open_with_fallback(
root: &Path,
discovery: &SharedMemoryDiscovery,
interface: &str,
fallback: bool,
) -> Self {
Self::open_config(root, discovery, interface, fallback, 1280).await
}
async fn open_config(
root: &Path,
discovery: &SharedMemoryDiscovery,
interface: &str,
fallback: bool,
mtu: u32,
) -> Self {
Self::open_selected(root, discovery, interface, fallback, mtu, None).await
}
async fn open_selected(
root: &Path,
discovery: &SharedMemoryDiscovery,
interface: &str,
fallback: bool,
mtu: u32,
default_protocols: Option<Vec<String>>,
) -> Self {
let identity =
tsunagi::storage::StateStore::open(tsunagi::StoragePaths::under(root).state_db())
.unwrap()
.load_or_create_device_identity()
.unwrap();
let transport = AwgTransport::new(identity.endpoint_id(), Ipv4Addr::LOCALHOST);
let plugin = AwgPlugin::open(
WireguardConfig::new(root.join("awg"))
.with_mtu(mtu)
.with_reconcile(Duration::from_millis(20), Duration::from_millis(250)),
transport.clone(),
)
.await
.unwrap();
let loss = Arc::new(AtomicBool::new(false));
let faults = Arc::new(FaultTransport {
inner: transport.clone(),
loss: loss.clone(),
});
let tuns = MemoryTunFactory::new();
let mut config = config_with(root, discovery)
.with_interface(Arc::new(tuns.clone()), interface, mtu)
.with_plugin(plugin.clone() as Arc<dyn IpPlugin>)
.with_custom_transport(AWG_PROTOCOL, faults);
let mut quic_plugin = None;
if fallback {
let quic = tsunagi_wg_quic::WireguardPlugin::open(
WireguardConfig::new(root.join("wg-quic"))
.with_mtu(mtu)
.with_reconcile(Duration::from_millis(20), Duration::from_millis(250)),
)
.await
.unwrap();
config = config.with_plugin(quic.clone());
quic_plugin = Some(quic);
}
config.default_protocols = default_protocols;
let agent = Agent::spawn(config).await.unwrap();
let inbound_agent = agent.clone();
let inbound_transport = transport.clone();
let inbound_loss = loss.clone();
tokio::spawn(async move {
while let Some(mut link) = inbound_transport.next_inbound().await {
link.link = impaired(link.link, inbound_loss.clone());
let _ = inbound_agent.install_inbound_link(link).await;
}
});
Self {
agent,
plugin,
quic: quic_plugin,
transport,
tuns,
loss,
}
}
async fn ready(&self, id: NetworkId, n: usize) {
wait_until("AWG outer paths and end-to-end handshakes", || async {
(self
.transport
.overview(id)
.iter()
.filter(|p| !p.closed && !p.path.contains("establishing"))
.count()
== n
&& self
.plugin
.overview(id)?
.peers
.iter()
.filter(|p| {
p.tunnel
.as_ref()
.is_some_and(|t| t.health.is_up() && t.path.contains("awg UDP via"))
})
.count()
== n)
.then_some(())
})
.await;
}
fn tun(&self) -> Arc<MemoryTun> {
self.tuns
.device(&self.agent.overlay().unwrap().interface)
.unwrap()
}
fn addr(&self, id: NetworkId) -> Ipv4Addr {
self.plugin
.overview(id)
.unwrap()
.overlay_address_v4
.unwrap()
}
}
fn packet(source: Ipv4Addr, dest: Ipv4Addr, len: usize) -> Bytes {
let mut bytes = vec![0; len];
bytes[0] = 0x45;
bytes[2..4].copy_from_slice(&(len as u16).to_be_bytes());
bytes[8] = 64;
bytes[9] = 253;
bytes[12..16].copy_from_slice(&source.octets());
bytes[16..20].copy_from_slice(&dest.octets());
rand::fill(&mut bytes[20..]);
Bytes::from(bytes)
}
async fn exchange(a: &Node, b: &Node, id: NetworkId) {
for len in [64, 1097, 1280] {
let bytes = packet(a.addr(id), b.addr(id), len);
a.tun().push_from_os(bytes.clone());
assert_eq!(
tokio::time::timeout(Duration::from_secs(5), b.tun().pop_to_os())
.await
.unwrap()
.unwrap(),
bytes
);
}
}
#[tokio::test]
async fn a_control_only_agent_enables_real_data_without_restart_and_cancels_stale_attempts() {
let discovery = SharedMemoryDiscovery::new();
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
let a = Node::open_selected(
roots[0].path(),
&discovery,
"initial-none-a",
true,
1280,
Some(vec![]),
)
.await;
let b = Node::open_selected(
roots[1].path(),
&discovery,
"initial-none-b",
true,
1280,
Some(vec![]),
)
.await;
let (name, secret) = network("enable-after-control-only");
let id = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
wait_for_peers(&a.agent, id, 1).await;
wait_for_peers(&b.agent, id, 1).await;
assert!(a.agent.overlay().is_none());
assert!(b.agent.overlay().is_none());
for node in [&a, &b] {
assert!(
node.agent
.network_status(id)
.await
.unwrap()
.protocols
.is_empty()
);
node.agent
.set_protocols(id, vec!["awg".into()])
.await
.unwrap();
assert!(node.agent.overlay().is_some());
}
a.ready(id, 1).await;
b.ready(id, 1).await;
exchange(&a, &b, id).await;
let before = a.agent.overlay().unwrap().interface;
// Changes race unfinished opens and prepared plugin callbacks; the last
// selection must win even if an earlier open has completed in its queue.
for selection in [
vec!["wg-quic".into()],
vec![],
vec!["awg".into()],
vec![],
vec!["awg".into()],
] {
a.agent.set_protocols(id, selection).await.unwrap();
}
a.ready(id, 1).await;
b.ready(id, 1).await;
exchange(&a, &b, id).await;
assert_eq!(a.agent.overlay().unwrap().interface, before);
for node in [&a, &b] {
let status = node.agent.network_status(id).await.unwrap();
assert_eq!(status.protocols, ["awg"]);
assert_eq!(status.metrics.sessions_established, 1);
assert_eq!(status.metrics.disconnects, 0);
}
a.agent.shutdown().await;
b.agent.shutdown().await;
}
#[tokio::test]
async fn runtime_protocol_selection_replaces_real_tunnels_and_survives_restart() {
let discovery = SharedMemoryDiscovery::new();
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
let a = Node::open_with_fallback(roots[0].path(), &discovery, "runtime-a", true).await;
let b = Node::open_with_fallback(roots[1].path(), &discovery, "runtime-b", true).await;
let (name, secret) = network("runtime-protocols");
let id = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
a.ready(id, 1).await;
b.ready(id, 1).await;
exchange(&a, &b, id).await;
let (other_name, other_secret, other_id) = loop {
let (name, secret) = network("unchanged-runtime-network");
let other_id = tsunagi::identity::NetworkKeys::derive(&name, &secret).network_id();
if tsunagi::state::derived_ipv4_range(id) != tsunagi::state::derived_ipv4_range(other_id) {
break (name, secret, other_id);
}
};
a.agent
.join_network(&other_name, &other_secret)
.await
.unwrap();
b.agent
.join_network(&other_name, &other_secret)
.await
.unwrap();
a.ready(other_id, 1).await;
b.ready(other_id, 1).await;
let other_before = a
.agent
.network_status(other_id)
.await
.unwrap()
.metrics
.data_links_established;
for node in [&a, &b] {
let before = node.agent.network_status(id).await.unwrap().protocols;
for invalid in [vec!["unknown".into()], vec!["awg".into(), "awg".into()]] {
assert!(node.agent.set_protocols(id, invalid).await.is_err());
assert_eq!(
node.agent.network_status(id).await.unwrap().protocols,
before
);
}
}
// Exercise updates initiated by either endpoint-id role, including an
// empty set. Membership and the control session stay alive throughout.
for node in [&a, &b] {
node.agent.set_protocols(id, Vec::new()).await.unwrap();
wait_until("peer accepts control-only selection", || async {
let other = if node.agent.endpoint_id() == a.agent.endpoint_id() {
&b
} else {
&a
};
let status = other.agent.network_status(id).await.ok()?;
(status.peers.len() == 1
&& status.peers[0].protocols.is_empty()
&& matches!(status.peers[0].data_path, tsunagi::agent::DataPath::None))
.then_some(())
})
.await;
assert_eq!(node.agent.network_status(id).await.unwrap().peers.len(), 1);
node.agent
.set_protocols(id, vec!["wg-quic".into(), "awg".into()])
.await
.unwrap();
}
a.agent
.set_protocols(id, vec!["wg-quic".into()])
.await
.unwrap();
wait_until("real QUIC replaces AWG", || async {
let a_status = a.agent.network_status(id).await.ok()?;
let b_status = b.agent.network_status(id).await.ok()?;
(a_status.peers[0].protocols == ["wg-quic"]
&& b_status.peers[0].protocols == ["wg-quic"]
&& !matches!(a_status.peers[0].data_path, tsunagi::agent::DataPath::None)
&& !matches!(b_status.peers[0].data_path, tsunagi::agent::DataPath::None)
&& a.transport.overview(id).iter().all(|p| p.closed)
&& b.transport.overview(id).iter().all(|p| p.closed))
.then_some(())
})
.await;
// Get addresses from the system allocation: the disabled AWG plugin no
// longer has a live view of this network.
for (source, dest) in [(&a, &b), (&b, &a)] {
wait_until("QUIC end-to-end handshake", || async {
source
.quic
.as_ref()?
.overview(id)?
.peers
.iter()
.any(|p| p.tunnel.as_ref().is_some_and(|t| t.health.is_up()))
.then_some(())
})
.await;
let addr = |n: &Node| {
n.quic
.as_ref()
.unwrap()
.overview(id)
.unwrap()
.overlay_address_v4
.unwrap()
};
let bytes = packet(addr(source), addr(dest), 1280);
source.tun().push_from_os(bytes.clone());
assert_eq!(
tokio::time::timeout(Duration::from_secs(5), dest.tun().pop_to_os())
.await
.unwrap()
.unwrap(),
bytes
);
}
assert_eq!(
a.agent
.network_status(other_id)
.await
.unwrap()
.metrics
.data_links_established,
other_before
);
exchange(&a, &b, other_id).await;
a.agent.shutdown().await;
let restarted = Node::open_with_fallback(roots[0].path(), &discovery, "runtime-a", true).await;
wait_for_peers(&restarted.agent, id, 1).await;
assert_eq!(
restarted.agent.network_status(id).await.unwrap().protocols,
["wg-quic"]
);
assert_eq!(
b.agent.network_status(other_id).await.unwrap().protocols,
["awg", "wg-quic"]
);
restarted.ready(other_id, 1).await;
b.ready(other_id, 1).await;
exchange(&restarted, &b, other_id).await;
restarted
.agent
.set_protocols(id, vec!["awg".into()])
.await
.unwrap();
restarted.ready(id, 1).await;
b.ready(id, 1).await;
exchange(&restarted, &b, id).await;
restarted.agent.shutdown().await;
b.agent.shutdown().await;
}
#[tokio::test]
async fn real_pairs_carry_full_size_packets_and_restart_with_fresh_profiles() {
tsunagi::testing::init_tracing();
let discovery = SharedMemoryDiscovery::new();
let roots = [
TempDir::new().unwrap(),
TempDir::new().unwrap(),
TempDir::new().unwrap(),
];
let a = Node::open(roots[0].path(), &discovery, "awga").await;
let b = Node::open(roots[1].path(), &discovery, "awgb").await;
let c = Node::open(roots[2].path(), &discovery, "awgc").await;
let (name, secret) = network("awg-pairs");
let id = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
c.agent.join_network(&name, &secret).await.unwrap();
for node in [&a, &b, &c] {
wait_for_peers(&node.agent, id, 2).await;
node.ready(id, 2).await;
}
let ids: std::collections::HashSet<_> = [&a, &b, &c]
.into_iter()
.flat_map(|n| n.transport.overview(id).into_iter().map(|p| p.attempt))
.collect();
assert_eq!(
ids.len(),
3,
"one independent profile for each of three pairs"
);
exchange(&a, &b, id).await;
exchange(&b, &a, id).await;
exchange(&a, &c, id).await;
let old_ports: Vec<_> = a
.transport
.overview(id)
.into_iter()
.filter_map(|p| p.local_port)
.collect();
a.agent.shutdown().await;
for port in old_ports {
wait_until("old pair socket released", || async {
tokio::net::UdpSocket::bind((Ipv4Addr::LOCALHOST, port))
.await
.ok()
})
.await;
}
let restarted = Node::open(roots[0].path(), &discovery, "awga").await;
restarted.ready(id, 2).await;
assert!(
restarted
.transport
.overview(id)
.iter()
.all(|p| !ids.contains(&p.attempt))
);
exchange(&restarted, &b, id).await;
exchange(&c, &restarted, id).await;
restarted.agent.shutdown().await;
b.agent.shutdown().await;
c.agent.shutdown().await;
assert!(
tokio::time::timeout(Duration::from_secs(1), restarted.transport.next_inbound())
.await
.unwrap()
.is_none()
);
}
// Drops datagrams at the link boundary. Membership, control, sockets, crypto,
// addressing and storage remain real. This substitutes a lossy physical path.
#[derive(Debug)]
struct FaultTransport {
inner: Arc<AwgTransport>,
loss: Arc<AtomicBool>,
}
impl PacketTransport for FaultTransport {
fn name(&self) -> &str {
AWG_PROTOCOL
}
fn as_any(&self) -> &dyn std::any::Any {
self
}
fn attach_control(&self, sender: Arc<dyn ControlSender>) {
self.inner.attach_control(sender);
}
fn on_control(
&self,
network: NetworkId,
peer: iroh::EndpointId,
protocol: &str,
bytes: &[u8],
) -> Result<(), TransportError> {
self.inner.on_control(network, peer, protocol, bytes)
}
fn open<'a>(
&'a self,
network: NetworkId,
peer: iroh::EndpointId,
protocol: &'a str,
) -> BoxFuture<'a, Result<SharedLink, TransportError>> {
Box::pin(async move {
Ok(impaired(
self.inner.open(network, peer, protocol).await?,
self.loss.clone(),
))
})
}
}
fn impaired(raw: SharedLink, loss: Arc<AtomicBool>) -> SharedLink {
Arc::new(FaultLink {
raw,
loss,
sequence: AtomicU64::new(0),
})
}
#[derive(Debug)]
struct FaultLink {
raw: SharedLink,
loss: Arc<AtomicBool>,
sequence: AtomicU64,
}
impl PacketLink for FaultLink {
fn network(&self) -> NetworkId {
self.raw.network()
}
fn peer(&self) -> iroh::EndpointId {
self.raw.peer()
}
fn max_datagram_size(&self) -> usize {
self.raw.max_datagram_size()
}
fn send(&self, bytes: Bytes) -> Result<(), TransportError> {
if self.loss.load(Ordering::Relaxed)
&& !self
.sequence
.fetch_add(1, Ordering::Relaxed)
.is_multiple_of(5)
{
Ok(())
} else {
self.raw.send(bytes)
}
}
fn recv(&self) -> BoxFuture<'_, Option<Bytes>> {
self.raw.recv()
}
fn closed(&self) -> BoxFuture<'_, ()> {
self.raw.closed()
}
fn is_closed(&self) -> bool {
self.raw.is_closed()
}
fn close(&self) {
self.raw.close();
}
fn path_description(&self) -> String {
self.raw.path_description()
}
}
#[tokio::test]
async fn persistent_runtime_loss_retires_the_pair_and_negotiates_fresh_parameters() {
let discovery = SharedMemoryDiscovery::new();
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
let a = Node::open(roots[0].path(), &discovery, "lossa").await;
let b = Node::open(roots[1].path(), &discovery, "lossb").await;
let (name, secret) = network("awg-loss");
let id = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
a.ready(id, 1).await;
b.ready(id, 1).await;
exchange(&a, &b, id).await;
let old = a.transport.overview(id)[0].attempt;
let mut events = a.agent.subscribe();
a.loss.store(true, Ordering::Relaxed);
tsunagi::testing::wait_event(&mut events, |event| match event {
tsunagi::Event::DataLinkDown {
network, reason, ..
} if *network == id && reason.contains("delivery checks") => Some(()),
_ => None,
})
.await;
a.loss.store(false, Ordering::Relaxed);
wait_until("fresh AWG profile after runtime loss", || async {
a.transport
.overview(id)
.iter()
.any(|p| p.attempt != old && !p.closed && !p.path.contains("establishing"))
.then_some(())
})
.await;
a.ready(id, 1).await;
b.ready(id, 1).await;
let initiator = if a.agent.endpoint_id().as_bytes() < b.agent.endpoint_id().as_bytes() {
&a
} else {
&b
};
let received = initiator.plugin.overview(id).unwrap().peers[0]
.tunnel
.as_ref()
.unwrap()
.stats
.rx_packets;
let attempt = initiator.transport.overview(id)[0].attempt;
wait_until(
"fresh pair delivery checks answered after installation",
|| async {
let view = initiator.plugin.overview(id)?;
(view.peers.first()?.tunnel.as_ref()?.stats.rx_packets >= received + 3).then_some(())
},
)
.await;
assert_eq!(initiator.transport.overview(id)[0].attempt, attempt);
wait_for_peers(&a.agent, id, 1).await;
exchange(&a, &b, id).await;
exchange(&b, &a, id).await;
a.agent.shutdown().await;
b.agent.shutdown().await;
}
#[tokio::test]
async fn runtime_loss_falls_back_to_real_iroh_without_interrupting_control_or_ip() {
let discovery = SharedMemoryDiscovery::new();
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
let a = Node::open_with_fallback(roots[0].path(), &discovery, "fallba", true).await;
let b = Node::open_with_fallback(roots[1].path(), &discovery, "fallbb", true).await;
let (name, secret) = network("awg-fallback");
let id = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
a.ready(id, 1).await;
b.ready(id, 1).await;
exchange(&a, &b, id).await;
let mut events = a.agent.subscribe();
a.loss.store(true, Ordering::Relaxed);
tsunagi::testing::wait_event(&mut events, |event| match event {
tsunagi::Event::DataLinkDown {
network, reason, ..
} if *network == id && reason.contains("delivery checks") => Some(()),
_ => None,
})
.await;
for node in [&a, &b] {
wait_until("fallback carrier is wg-quic",||async {
let status=node.agent.status().await.ok()?;
let network=status.networks.iter().find(|n|n.network_id==id)?;
(network.members.iter().any(|m|matches!(&m.data_path,tsunagi::agent::DataPath::Direct{transport} if transport=="wg-quic"))).then_some(())
}).await;
wait_for_peers(&node.agent, id, 1).await;
}
exchange(&a, &b, id).await;
exchange(&b, &a, id).await;
a.agent.shutdown().await;
b.agent.shutdown().await;
}
#[tokio::test]
async fn opaque_pair_carrier_fragments_its_full_advertised_capacity() {
let discovery = SharedMemoryDiscovery::new();
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
let mut nodes = Vec::new();
for root in &roots {
let identity = tsunagi::storage::StateStore::open(
tsunagi::StoragePaths::under(root.path()).state_db(),
)
.unwrap()
.load_or_create_device_identity()
.unwrap();
let transport = AwgTransport::new(identity.endpoint_id(), Ipv4Addr::LOCALHOST);
let agent = Agent::spawn(
config_with(root.path(), &discovery)
.with_custom_transport(AWG_PROTOCOL, transport.clone()),
)
.await
.unwrap();
nodes.push((agent, transport));
}
let (name, secret) = network("awg-capacity");
let id = nodes[0].0.join_network(&name, &secret).await.unwrap();
nodes[1].0.join_network(&name, &secret).await.unwrap();
for (i, (agent, transport)) in nodes.iter().enumerate() {
wait_for_peers(agent, id, 1).await;
// This fixture tests only the transport contract: register the member
// after the real core has authenticated it, without an IP plugin/TUN.
transport.allow_peer(id, nodes[1 - i].0.endpoint_id());
}
nodes.sort_by_key(|(agent, _)| *agent.endpoint_id().as_bytes());
let outgoing = nodes[0]
.1
.open(id, nodes[1].0.endpoint_id(), AWG_PROTOCOL)
.await
.unwrap();
let incoming = tokio::time::timeout(Duration::from_secs(5), nodes[1].1.next_inbound())
.await
.unwrap()
.unwrap()
.link;
for (from, to) in [(&outgoing, &incoming), (&incoming, &outgoing)] {
let mut bytes = vec![0; from.max_datagram_size()];
rand::fill(&mut bytes[..]);
let bytes = Bytes::from(bytes);
from.send(bytes.clone()).unwrap();
assert_eq!(
tokio::time::timeout(Duration::from_secs(5), to.recv())
.await
.unwrap()
.unwrap(),
bytes
);
assert!(matches!(
from.send(Bytes::from(vec![0; from.max_datagram_size() + 1])),
Err(TransportError::TooLarge { .. })
));
}
outgoing.close();
incoming.close();
for (agent, transport) in nodes {
transport.shutdown();
agent.shutdown().await;
}
}
#[tokio::test]
async fn encrypted_health_checks_fit_a_smaller_peers_configured_interface_mtu() {
let discovery = SharedMemoryDiscovery::new();
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
let a = Node::open_config(roots[0].path(), &discovery, "smalla", false, 1280).await;
let b = Node::open_config(roots[1].path(), &discovery, "smallb", false, 576).await;
let (name, secret) = network("awg-small-mtu");
let id = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
a.ready(id, 1).await;
b.ready(id, 1).await;
let bytes = packet(a.addr(id), b.addr(id), 576);
a.tun().push_from_os(bytes.clone());
assert_eq!(
tokio::time::timeout(Duration::from_secs(5), b.tun().pop_to_os())
.await
.unwrap()
.unwrap(),
bytes
);
wait_until(
"encrypted delivery checks answered with a small MTU",
|| async {
let view = a.plugin.overview(id)?;
let peer = view.peers.first()?.tunnel.as_ref()?;
(peer.stats.rx_packets >= 3).then_some(())
},
)
.await;
for node in [&a, &b] {
assert_eq!(
node.plugin.overview(id).unwrap().peers[0]
.tunnel
.as_ref()
.unwrap()
.stats
.dropped_oversize,
0
);
}
a.agent.shutdown().await;
b.agent.shutdown().await;
}
+36
View File
@@ -0,0 +1,36 @@
//! Fresh and bounded masking profiles, and rejection of foreign/truncated wire packets.
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use tsunagi_awg::profile::PairProfile;
#[test]
fn profiles_are_fresh_bounded_and_redacted() {
let profiles: Vec<_> = (0..128).map(|_| PairProfile::generate()).collect();
for (i, profile) in profiles.iter().enumerate() {
profile.validate().unwrap();
assert!(!format!("{profile:?}").contains(&format!("{:?}", profile.header_key())));
for other in &profiles[..i] {
assert_ne!(profile.header_key(), other.header_key());
assert_ne!(profile.id(), other.id());
}
for packet in profile.signature_packets() {
assert!(packet.len() < 1200);
}
}
}
#[test]
fn masked_packets_round_trip_and_reject_other_profiles_and_truncation() {
let profile = PairProfile::generate();
let other = PairProfile::generate();
for (kind, size) in [(1, 148), (2, 92), (3, 64), (4, 800)] {
let mut message = vec![0x7b; size];
message[..4].copy_from_slice(&(kind as u32).to_le_bytes());
let wire = profile.encode(&message).unwrap();
assert_ne!(&wire[..4], &message[..4]);
assert_eq!(profile.decode(&wire).unwrap(), message);
assert!(other.decode(&wire).is_none());
for length in 0..44 {
assert!(profile.decode(&wire[..length]).is_none());
}
}
}
+446 -161
View File
@@ -27,14 +27,14 @@ mod status;
pub use events::Event;
pub use status::{
AgentStatus, CandidateStatus, DataPath, HostnameConflict, MemberStatus, NetworkMetrics,
NetworkState, NetworkStatus, OverlayStatus, PeerStatus,
NetworkState, NetworkStatus, OverlayStatus, PeerStatus, RangeConflict,
};
use std::collections::HashMap;
use std::sync::{Arc, Weak};
use iroh::{EndpointAddr, EndpointId};
use tokio::sync::{RwLock, broadcast, mpsc, oneshot};
use tokio::sync::{Mutex, RwLock, broadcast, mpsc, oneshot, watch};
use tokio::task::JoinHandle;
use crate::BoxFuture;
@@ -58,6 +58,8 @@ use shutdown::Shutdown;
/// Summary of a configured network, whether or not it is running.
#[derive(Debug, Clone)]
pub struct ConfiguredNetwork {
/// Enabled local data-plane protocols, in preference order.
pub protocols: Vec<String>,
/// Saved local broadcast participation.
pub broadcast: bool,
/// Public network identifier.
@@ -70,6 +72,17 @@ pub struct ConfiguredNetwork {
pub active: bool,
}
/// A durable version-scoped cleanup, not an identity ban.
#[derive(Debug, Clone)]
pub struct ForgetNodeOutcome {
/// Identity whose observed claim was retired.
pub target: EndpointId,
/// The retired claim version; future signed claims remain admissible.
pub record_version: u64,
/// Authenticated peers to which cleanup was queued immediately.
pub peers_told: usize,
}
/// A running agent.
///
/// Cloning gives another handle to the same agent.
@@ -90,6 +103,10 @@ struct Inner {
events: broadcast::Sender<Event>,
networks: RwLock<HashMap<NetworkId, NetworkHandle>>,
shutdown: Shutdown,
/// Serialises start, stop, switching and conflict cleanup, including teardown.
lifecycle: Mutex<()>,
range_changed: watch::Sender<()>,
range_task: std::sync::Mutex<Option<JoinHandle<()>>>,
accept_task: std::sync::Mutex<Option<JoinHandle<()>>>,
plugin_task: std::sync::Mutex<Option<JoinHandle<()>>>,
transport: std::sync::OnceLock<Arc<dyn PacketTransport>>,
@@ -100,6 +117,37 @@ struct Inner {
}
impl Inner {
/// Called under the lifecycle lock after startup: an initially
/// control-only agent creates its one interface on first enablement.
async fn ensure_interface(self: &Arc<Self>) {
if self.interface.get().is_some() {
return;
}
if let Some(factory) = self.config.tun_factory.clone() {
let carrier = Arc::new(Carrier(Arc::downgrade(self))) as Arc<dyn PacketCarrier>;
match crate::overlay::Interface::start(
factory,
self.config.interface_name.clone(),
self.config.interface_mtu,
Arc::clone(&self.routes),
carrier,
)
.await
{
Ok(interface) => {
let _ = self.interface.set(Arc::new(interface));
}
Err(err) => {
let _ = self.events.send(Event::PluginError {
network: NetworkId::from_bytes([0u8; 32]),
protocol: "overlay".into(),
reason: err.to_string(),
});
}
}
}
}
/// The name this agent currently answers to.
fn read_hostname(&self) -> String {
match self.hostname.read() {
@@ -165,6 +213,21 @@ impl PacketSink for Sink {
let Some(inner) = self.0.upgrade() else {
return;
};
if let Some(payload) = crate::dataplane::transport::health::ip_payload(&packet) {
if let Some(header) = crate::overlay::IpHeader::parse(&packet)
&& let (std::net::IpAddr::V4(source), std::net::IpAddr::V4(destination)) =
(header.source(), header.destination())
&& inner.routes.may_send_from(network, peer, source)
&& inner.routes.is_local_destination(network, destination)
&& let Some(handle) = inner.networks.read().await.get(&network)
{
let _ = handle.commands.try_send(NetCommand::HealthProbe {
peer,
packet: bytes::Bytes::copy_from_slice(payload),
});
}
return;
}
let Some(interface) = inner.interface.get() else {
return;
};
@@ -183,6 +246,35 @@ impl PacketSink for Sink {
#[derive(Debug)]
struct TransportCtx(Weak<Inner>);
impl crate::dataplane::transport::ControlSender for TransportCtx {
fn send<'a>(
&'a self,
network: NetworkId,
peer: EndpointId,
protocol: &'a str,
payload: crate::proto::message::TransportPayload,
) -> crate::BoxFuture<'a, std::result::Result<(), crate::dataplane::transport::TransportError>>
{
Box::pin(async move {
let inner = self
.0
.upgrade()
.ok_or(crate::dataplane::transport::TransportError::Closed)?;
Agent { inner }
.send(
network,
peer,
crate::proto::message::ControlMessage::Transport {
protocol: protocol.into(),
payload,
},
)
.await
.map_err(|err| crate::dataplane::transport::TransportError::Other(err.to_string()))
})
}
}
impl TransportContext for TransportCtx {
fn snapshot(&self) -> crate::BoxFuture<'_, HashMap<NetworkId, NetworkKeys>> {
Box::pin(async move {
@@ -217,12 +309,111 @@ impl TransportContext for TransportCtx {
}
impl Agent {
/// Retires one offline member's observed claim without revoking membership.
/// Every member has this right. The version must still match the selection.
pub async fn forget_node(
&self,
network: NetworkId,
peer: EndpointId,
expected_version: u64,
) -> Result<ForgetNodeOutcome> {
let _lifecycle = self.inner.lifecycle.lock().await;
let (reply, receive) = oneshot::channel();
self.command(
network,
NetCommand::ForgetNode {
peer,
expected_version,
reply,
},
)
.await?;
receive.await.map_err(|_| Error::Stopped)?
}
fn effective_protocols(&self, saved: Option<&Vec<String>>) -> Vec<String> {
saved
.or(self.inner.config.default_protocols.as_ref())
.cloned()
.unwrap_or_else(|| {
self.inner
.config
.plugins
.iter()
.map(|p| p.protocol_id().to_owned())
.collect()
})
}
fn selected_plugins(&self, protocols: &[String]) -> Vec<crate::dataplane::SharedPlugin> {
protocols
.iter()
.filter_map(|name| {
self.inner
.config
.plugins
.iter()
.find(|p| p.protocol_id() == name)
.cloned()
})
.collect()
}
/// Saves and immediately applies a network's ordered local protocol set.
/// An empty set disables data traffic while retaining control sessions.
pub async fn set_protocols(&self, network_id: NetworkId, protocols: Vec<String>) -> Result<()> {
crate::dataplane::validate_protocols(&protocols, self.inner.limits.max_capabilities)?;
for name in &protocols {
if !self
.inner
.config
.plugins
.iter()
.any(|p| p.protocol_id() == name)
{
return Err(Error::InvalidProtocols(format!(
"this agent has no `{name}` plugin"
)));
}
}
let _lifecycle = self.inner.lifecycle.lock().await;
self.inner
.storage
.set_protocols(network_id, protocols.clone())
.await?;
if self.is_active(network_id).await {
let (reply, receive) = oneshot::channel();
if !protocols.is_empty() {
self.inner.ensure_interface().await;
}
self.command(
network_id,
NetCommand::SetProtocols {
protocols,
interface: self.inner.interface.get().cloned(),
reply,
},
)
.await?;
receive.await.map_err(|_| Error::Stopped)?;
}
Ok(())
}
/// Starts an agent.
///
/// Opens the state store (taking its ownership lock), restores the
/// persistent device identity, binds the iroh endpoint and activates every
/// configured network whose auto-start flag is set.
pub async fn spawn(config: AgentConfig) -> Result<Self> {
if let Some(protocols) = &config.default_protocols {
crate::dataplane::validate_protocols(protocols, config.limits.max_capabilities)?;
for name in protocols {
if !config.plugins.iter().any(|p| p.protocol_id() == name) {
return Err(Error::InvalidProtocols(format!(
"this agent has no `{name}` plugin"
)));
}
}
}
let storage = Storage::open(&config.paths)?;
let identity = storage.device_identity().await?;
let adapter = EndpointAdapter::bind(&config, &identity).await?;
@@ -234,6 +425,7 @@ impl Agent {
let (events, _) = broadcast::channel(config.limits.event_buffer);
let limits = Arc::new(config.limits.clone());
let (range_changed, range_updates) = watch::channel(());
let inner = Arc::new(Inner {
limits,
storage,
@@ -243,6 +435,9 @@ impl Agent {
events,
networks: RwLock::new(HashMap::new()),
shutdown: Shutdown::new(),
lifecycle: Mutex::new(()),
range_changed,
range_task: std::sync::Mutex::new(None),
accept_task: std::sync::Mutex::new(None),
plugin_task: std::sync::Mutex::new(None),
transport: std::sync::OnceLock::new(),
@@ -251,31 +446,13 @@ impl Agent {
config,
});
// One interface for the agent, if it was given a way to make one.
// Failing to is reported and not fatal: the control plane works, and
// so do the protocols, they just have nowhere to put packets.
if let Some(factory) = inner.config.tun_factory.clone() {
let carrier = Arc::new(Carrier(Arc::downgrade(&inner))) as Arc<dyn PacketCarrier>;
match crate::overlay::Interface::start(
factory,
inner.config.interface_name.clone(),
inner.config.interface_mtu,
Arc::clone(&inner.routes),
carrier,
)
.await
{
Ok(interface) => {
let _ = inner.interface.set(Arc::new(interface));
}
Err(err) => {
let _ = inner.events.send(Event::PluginError {
network: NetworkId::from_bytes([0u8; 32]),
protocol: "overlay".into(),
reason: err.to_string(),
});
}
}
if inner
.config
.default_protocols
.as_ref()
.is_none_or(|names| !names.is_empty())
{
inner.ensure_interface().await;
}
// The data plane rides on iroh too, which is where it gets hole
@@ -294,6 +471,7 @@ impl Agent {
inner.config.custom_transports.clone(),
))
};
transport.attach_control(Arc::new(TransportCtx(Arc::downgrade(&inner))));
let _ = inner.transport.set(transport);
if let CacheOutcome::Reset(reason) = inner.storage.cache_outcome().clone() {
@@ -311,8 +489,9 @@ impl Agent {
if !inner.config.plugins.is_empty() {
let (plugin_tx, plugin_rx) = mpsc::channel(64);
let sink = inner
.interface
.get()
.config
.tun_factory
.as_ref()
.map(|_| Arc::new(Sink(Arc::downgrade(&inner))) as Arc<dyn PacketSink>);
let context = PluginContext::new(plugin_tx, inner.identity.endpoint_id(), sink);
for plugin in &inner.config.plugins {
@@ -326,11 +505,16 @@ impl Agent {
let agent = Self { inner };
for stored in agent.inner.storage.list_networks().await? {
if stored.auto_start {
let keys = NetworkKeys::derive(&stored.name, &stored.secret);
agent.activate_with_keys(keys).await?;
}
{
let _guard = agent.inner.lifecycle.lock().await;
agent.resume_networks_locked().await?;
}
let task = tokio::spawn(range_conflict_loop(
Arc::downgrade(&agent.inner),
range_updates,
));
if let Ok(mut guard) = agent.inner.range_task.lock() {
*guard = Some(task);
}
Ok(agent)
@@ -374,100 +558,65 @@ impl Agent {
Ok(())
}
/// Reserves the configured overlay range for a network, if it can.
///
/// Done here, where activations are serialised, rather than inside the
/// runtime: whether a range is free depends on what the other networks
/// took, and deciding that in a task would make the answer depend on
/// which task ran first.
///
/// `None` means this agent will not propose a range for that network and
/// waits to adopt whatever it settles on. One agent has one interface, so
/// proposing a range it could not route would be worse than having none:
/// the lowest author's range wins, and the collision would spread.
async fn reserve_range(&self, network: NetworkId) -> RangePlan {
let reserve = |wanted: Ipv4Range| {
let reservation = crate::overlay::NetworkRoutes {
exit: Default::default(),
broadcast: Default::default(),
range: Some(wanted),
local: None,
peers: Vec::new(),
};
self.inner.routes.set_network(network, reservation).is_ok()
};
// If this network already has signed records in storage, check if its
// members have already agreed on a range. If so, preserve and reserve
// THAT range rather than preemptively seizing the default configured range.
let stored_records = self
.inner
.storage
.signed_records(network)
.await
.unwrap_or_default();
let mut state_set = crate::state::StateSet::new();
state_set.merge_all(network, stored_records);
if let Some(agreed) = state_set.agreed_range() {
if reserve(agreed) {
return RangePlan {
propose: if Some(agreed) == self.inner.config.overlay_ipv4_range {
Some(agreed)
} else {
None
},
..RangePlan::default()
};
/// Signed state always wins over a default or an explicit proposal.
async fn desired_range(&self, network: NetworkId) -> Result<Option<Ipv4Range>> {
let records = self.inner.storage.signed_records(network).await?;
let mut state = crate::state::StateSet::new();
state.merge_all(network, records);
Ok(state.agreed_range().or_else(|| {
if self.inner.config.overlay_ipv4_auto {
Some(crate::state::derived_ipv4_range(network))
} else {
tracing::warn!(
%network,
range = %agreed,
"agreed range for this network overlaps another network already active on this interface"
);
return RangePlan {
conflict: Some(agreed),
..RangePlan::default()
};
self.inner.config.overlay_ipv4_range
}
}))
}
async fn range_conflicts(
&self,
network: NetworkId,
range: Option<Ipv4Range>,
) -> Result<Vec<RangeConflict>> {
let Some(range) = range else {
return Ok(Vec::new());
};
let overlapping = self.inner.routes.overlapping_networks(network, range);
let stored = self.inner.storage.list_networks().await?;
Ok(overlapping
.into_iter()
.filter_map(|(id, range)| {
let name = stored.iter().find(|n| n.network_id == id)?.name.clone();
Some(RangeConflict {
network_id: id,
name,
range,
})
})
.collect())
}
/// Starts requested networks in saved preference order, skipping overlaps.
/// Called with the lifecycle lock held; skipped networks remain requested.
async fn resume_networks_locked(&self) -> Result<()> {
let order = self.inner.storage.network_start_order().await?;
let mut stored = self.inner.storage.list_networks().await?;
stored.sort_by_key(|n| {
order
.iter()
.position(|id| *id == n.network_id)
.unwrap_or(usize::MAX)
});
for network in stored {
if !network.auto_start || self.is_active(network.network_id).await {
continue;
}
let keys = NetworkKeys::derive(&network.name, &network.secret);
match self.activate_with_keys_locked(keys).await {
Ok(()) | Err(Error::NetworkRangeConflict { .. }) => {}
Err(err) => return Err(err),
}
}
let Some(configured) = self.inner.config.overlay_ipv4_range else {
return RangePlan::default();
};
// The configured range first, so a network a device has always had
// keeps the addresses it has always had.
if reserve(configured) {
return RangePlan {
propose: Some(configured),
..RangePlan::default()
};
}
// A second network on the same agent cannot have it — one agent,
// one interface — so it falls back to the range derived from its
// own id, which every one of its members derives identically
// without being told.
let derived = crate::state::derived_ipv4_range(network);
if reserve(derived) {
tracing::info!(
%network,
range = %derived,
"another network here holds the configured range; this one uses the \
range derived from its id unless its members settled another"
);
return RangePlan {
fallback: Some(derived),
conflict: Some(configured),
..RangePlan::default()
};
}
tracing::info!(%network, "not proposing a range for this network");
RangePlan {
conflict: Some(configured),
..RangePlan::default()
}
Ok(())
}
/// The overlay interface this agent owns, when it has one.
@@ -546,6 +695,9 @@ impl Agent {
/// The same `(name, secret)` always produces the same [`NetworkId`], on
/// every device.
///
/// An overlapping network stays configured and requested, but inactive;
/// use [`Agent::switch_network`] to suspend the blockers explicitly.
///
/// This is declarative and therefore **idempotent**: joining a network
/// that is already active succeeds and changes nothing. That matters
/// because a configured network is activated automatically at startup, so
@@ -599,7 +751,9 @@ impl Agent {
}
match self.activate_with_keys(keys).await {
// Already a member of exactly this network space: nothing to do.
Ok(()) | Err(Error::NetworkAlreadyActive(_)) => Ok(network_id),
Ok(())
| Err(Error::NetworkAlreadyActive(_))
| Err(Error::NetworkRangeConflict { .. }) => Ok(network_id),
Err(err) => Err(err),
}
}
@@ -703,7 +857,44 @@ impl Agent {
self.activate_with_keys(keys).await
}
/// Switches to a configured network, suspending every overlapping network.
/// Suspended networks retain their auto-start request and resume when free.
/// Returns the networks suspended by this request.
pub async fn switch_network(&self, network_id: NetworkId) -> Result<Vec<NetworkId>> {
let _guard = self.inner.lifecycle.lock().await;
if self.inner.shutdown.is_triggered() {
return Err(Error::Stopped);
}
let stored = self
.inner
.storage
.list_networks()
.await?
.into_iter()
.find(|n| n.network_id == network_id)
.ok_or(Error::NetworkUnknown(network_id))?;
let range = self.desired_range(network_id).await?;
let conflicts = self.range_conflicts(network_id, range).await?;
self.inner.storage.prefer_network(network_id).await?;
let mut stopped = Vec::new();
for conflict in conflicts {
self.stop_network_locked(conflict.network_id).await?;
stopped.push(conflict.network_id);
}
if !self.is_active(network_id).await {
self.activate_with_keys_locked(NetworkKeys::derive(&stored.name, &stored.secret))
.await?;
}
self.resume_networks_locked().await?;
Ok(stopped)
}
async fn activate_with_keys(&self, keys: NetworkKeys) -> Result<()> {
let _guard = self.inner.lifecycle.lock().await;
self.activate_with_keys_locked(keys).await
}
async fn activate_with_keys_locked(&self, keys: NetworkKeys) -> Result<()> {
if self.inner.shutdown.is_triggered() {
return Err(Error::Stopped);
}
@@ -712,8 +903,42 @@ impl Agent {
if networks.contains_key(&network_id) {
return Err(Error::NetworkAlreadyActive(network_id));
}
let range = self.reserve_range(network_id).await;
let stored = self
.inner
.storage
.list_networks()
.await?
.into_iter()
.find(|stored| stored.network_id == network_id);
// Mandatory cleanup proofs must never be silently dropped on startup.
let retirements = self
.inner
.storage
.retirements(network_id, self.inner.limits.max_retirements)
.await?;
let range = self.desired_range(network_id).await?;
if let Some(range) = range {
let conflicts = self.inner.routes.overlapping_networks(network_id, range);
if !conflicts.is_empty() {
return Err(Error::NetworkRangeConflict {
network: network_id,
range,
conflicts: conflicts.into_iter().map(|(id, _)| id).collect(),
});
}
self.inner
.routes
.set_network(
network_id,
crate::overlay::NetworkRoutes {
range: Some(range),
..Default::default()
},
)
.map_err(|err| Error::Rejected {
reason: err.to_string(),
})?;
}
let mut backends = Vec::new();
if let Some(discovery) = &self.inner.config.discovery {
backends.push(discovery.clone());
@@ -729,20 +954,21 @@ impl Agent {
as Arc<dyn crate::discovery::NetworkDiscovery>,
)
};
let stored = self
.inner
.storage
.list_networks()
.await?
.into_iter()
.find(|stored| stored.network_id == network_id);
let broadcast = stored.as_ref().is_none_or(|stored| stored.broadcast);
let protocols =
self.effective_protocols(stored.as_ref().and_then(|s| s.protocols.as_ref()));
let plugins = self.selected_plugins(&protocols);
if !plugins.is_empty() {
self.inner.ensure_interface().await;
}
let exit_offer = stored.as_ref().is_some_and(|stored| stored.exit_node);
let exit_via = stored
.as_ref()
.and_then(|stored| stored.exit_via)
.and_then(|bytes| EndpointId::from_bytes(&bytes).ok());
let handle = network::spawn(RuntimeParams {
retirements,
broadcast,
exit_offer,
exit_via,
@@ -755,7 +981,8 @@ impl Agent {
discovery,
discovery_interval: self.inner.config.discovery_interval,
discovery_policy: self.inner.config.discovery_policy.clone(),
plugins: self.inner.config.plugins.clone(),
available_plugins: self.inner.config.plugins.clone(),
plugins: plugins.clone(),
routes: Arc::clone(&self.inner.routes),
interface: self.inner.interface.get().cloned(),
hostname: self.inner.read_hostname(),
@@ -763,14 +990,13 @@ impl Agent {
device_secret: self.inner.identity.signing_key(),
#[cfg(feature = "testing")]
unreachable_data_peers: Arc::clone(&self.inner.config.unreachable_data_peers),
ipv4_range: range.propose,
ipv4_fallback: range.fallback,
range_conflict: range.conflict,
ipv4_range: range,
range_changed: self.inner.range_changed.clone(),
});
networks.insert(network_id, handle);
drop(networks);
for plugin in &self.inner.config.plugins {
for plugin in &plugins {
plugin.on_network_activated(network_id);
}
@@ -786,6 +1012,30 @@ impl Agent {
/// it does not remove this agent from anyone else's view of the network.
/// Other networks keep running.
pub async fn deactivate_network(&self, network_id: NetworkId) -> Result<()> {
let _guard = self.inner.lifecycle.lock().await;
if !self.is_active(network_id).await {
return Err(Error::NetworkNotActive(network_id));
}
self.inner.storage.set_auto_start(network_id, false).await?;
self.stop_network_locked(network_id).await?;
self.resume_networks_locked().await
}
/// Stops a configured network, including one already suspended by overlap.
/// Clears its auto-start request so it will not resume behind the user.
pub async fn pause_network(&self, network_id: NetworkId) -> Result<()> {
let _guard = self.inner.lifecycle.lock().await;
if self.inner.shutdown.is_triggered() {
return Err(Error::Stopped);
}
self.inner.storage.set_auto_start(network_id, false).await?;
if self.is_active(network_id).await {
self.stop_network_locked(network_id).await?;
}
self.resume_networks_locked().await
}
async fn stop_network_locked(&self, network_id: NetworkId) -> Result<()> {
let handle = {
let mut networks = self.inner.networks.write().await;
networks
@@ -814,7 +1064,6 @@ impl Agent {
interface.sync_broadcast_rules().await;
interface.sync_exit_rules().await;
}
self.inner.storage.set_auto_start(network_id, false).await?;
Ok(())
}
@@ -824,17 +1073,15 @@ impl Agent {
/// Local and silent: nobody is told. [`Agent::leave_network`] is the one
/// that says goodbye first, and is what a person means by leaving.
pub async fn forget_network(&self, network_id: NetworkId) -> Result<()> {
let _guard = self.inner.lifecycle.lock().await;
if self.is_active(network_id).await {
self.deactivate_network(network_id).await?;
self.stop_network_locked(network_id).await?;
}
self.inner.storage.remove_network(network_id).await?;
// Whatever a protocol kept for this network goes with it. The
// agent cannot know what that is, only that there is no longer any
// reason to hold it.
for plugin in &self.inner.config.plugins {
plugin.on_network_forgotten(network_id);
}
Ok(())
self.resume_networks_locked().await
}
/// Leaves a network: gives up what was claimed, then forgets it.
@@ -890,6 +1137,7 @@ impl Agent {
.await?
.into_iter()
.map(|stored| ConfiguredNetwork {
protocols: self.effective_protocols(stored.protocols.as_ref()),
broadcast: stored.broadcast,
network_id: stored.network_id,
name: stored.name,
@@ -958,13 +1206,18 @@ impl Agent {
let mut networks = Vec::new();
for stored in self.inner.storage.list_networks().await? {
if active.contains(&stored.network_id)
&& let Ok(status) = self.network_status(stored.network_id).await
&& let Ok(mut status) = self.network_status(stored.network_id).await
{
status.auto_start = stored.auto_start;
networks.push(status);
continue;
}
let keys = NetworkKeys::derive(&stored.name, &stored.secret);
let range = self.desired_range(stored.network_id).await?;
let range_conflicts = self.range_conflicts(stored.network_id, range).await?;
networks.push(NetworkStatus {
protocols: self.effective_protocols(stored.protocols.as_ref()),
auto_start: stored.auto_start,
broadcast: stored.broadcast,
exit_offer: stored.exit_node,
exit_via: stored
@@ -978,12 +1231,13 @@ impl Agent {
state: NetworkState::Inactive,
peers: Vec::new(),
candidates: Vec::new(),
// An inactive network has no runtime to ask; the roster and
// the range come from one. Empty, not invented.
// The range is available from signed state or the deterministic
// default even while the network has no runtime.
members: Vec::new(),
hostname_conflict: None,
range: None,
range_conflict: None,
range,
range_conflict: range_conflicts.first().map(|conflict| conflict.range),
range_conflicts,
metrics: NetworkMetrics::default(),
relay: Default::default(),
});
@@ -1039,6 +1293,7 @@ impl Agent {
/// After this returns, the state directory can be opened by another agent.
pub async fn shutdown(&self) {
self.inner.shutdown.trigger();
let lifecycle_guard = self.inner.lifecycle.lock().await;
let handles: Vec<NetworkHandle> = {
let mut networks = self.inner.networks.write().await;
@@ -1048,6 +1303,7 @@ impl Agent {
handle.stop().await;
}
drop(lifecycle_guard);
if let Some(dht) = &self.inner.config.dht {
dht.shutdown().await;
}
@@ -1064,6 +1320,7 @@ impl Agent {
for (what, handle) in [
("inbound connections", &self.inner.accept_task),
("plugin requests", &self.inner.plugin_task),
("range conflicts", &self.inner.range_task),
] {
let task = handle.lock().ok().and_then(|mut guard| guard.take());
if let Some(task) = task {
@@ -1107,16 +1364,44 @@ impl Agent {
}
}
/// What a network may propose as its overlay range, and what it cannot have.
#[derive(Debug, Clone, Copy, Default)]
struct RangePlan {
/// Proposed straight away: the configured range, when it is free here.
propose: Option<Ipv4Range>,
/// Proposed after a moment, when the configured range is another
/// network's: the range derived from this network's own id.
fallback: Option<Ipv4Range>,
/// The configured range, when this network cannot have it.
conflict: Option<Ipv4Range>,
/// Processes coalesced conflict notifications without blocking a network actor.
async fn range_conflict_loop(inner: Weak<Inner>, mut updates: watch::Receiver<()>) {
loop {
let Some(owner) = inner.upgrade() else {
return;
};
let shutdown = owner.shutdown.clone();
drop(owner);
tokio::select! {
_ = shutdown.wait() => return,
result = updates.changed() => if result.is_err() { return; },
}
let Some(inner) = inner.upgrade() else {
return;
};
let agent = Agent { inner };
let _guard = agent.inner.lifecycle.lock().await;
if agent.inner.shutdown.is_triggered() {
return;
}
let blocked: Vec<_> = agent
.inner
.networks
.read()
.await
.iter()
.filter(|(_, h)| h.range_blocked.load(std::sync::atomic::Ordering::Acquire))
.map(|(id, _)| *id)
.collect();
for id in blocked {
if let Err(err) = agent.stop_network_locked(id).await {
tracing::warn!(%err, "cannot suspend overlapping network");
}
}
if let Err(err) = agent.resume_networks_locked().await {
tracing::warn!(%err, "cannot resume suspended networks");
}
}
}
/// How long a release is given to reach the sessions it was queued on.
File diff suppressed because it is too large. Load diff
+3 -2
View File
@@ -70,7 +70,7 @@ pub(crate) struct Session {
pub(crate) conn: Connection,
/// Already encoded frame payloads, so the runtime can account for the exact
/// number of control bytes it queues.
pub(crate) outbound: mpsc::Sender<Vec<u8>>,
pub(crate) outbound: mpsc::Sender<zeroize::Zeroizing<Vec<u8>>>,
pub(crate) hostname: Option<String>,
pub(crate) capabilities: Vec<PluginCapability>,
pub(crate) broadcast: bool,
@@ -244,7 +244,7 @@ pub(crate) fn spawn(
async fn writer_task(
mut send: SendStream,
mut outbound: mpsc::Receiver<Vec<u8>>,
mut outbound: mpsc::Receiver<zeroize::Zeroizing<Vec<u8>>>,
limits: Arc<Limits>,
shutdown: Shutdown,
parent: Shutdown,
@@ -331,6 +331,7 @@ async fn reader_task(
}
};
let payload = zeroize::Zeroizing::new(payload);
let bytes = payload.len();
let envelope: Envelope = match decode(&payload) {
Ok(envelope) => envelope,
+20 -6
View File
@@ -108,6 +108,8 @@ pub struct HostnameConflict {
/// connected.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct MemberStatus {
/// Current signed claim version, used to bind cleanup confirmation.
pub record_version: u64,
/// The member's device identity.
pub endpoint_id: EndpointId,
/// The IPv4 overlay address it claimed and signed for.
@@ -209,9 +211,24 @@ pub struct NetworkMetrics {
pub data_link_failures: u64,
}
/// An active network preventing another network from starting locally.
#[derive(Debug, Clone)]
pub struct RangeConflict {
/// The blocking network's public identifier.
pub network_id: NetworkId,
/// Its configured name.
pub name: NetworkName,
/// Its overlapping overlay range.
pub range: crate::state::Ipv4Range,
}
/// Status of one network.
#[derive(Debug, Clone)]
pub struct NetworkStatus {
/// Enabled local data-plane protocols, in preference order.
pub protocols: Vec<String>,
/// Whether this network is requested to start or resume automatically.
pub auto_start: bool,
/// Local broadcast participation in this network.
pub broadcast: bool,
/// This agent offers itself as an exit node in this network.
@@ -242,13 +259,10 @@ pub struct NetworkStatus {
pub hostname_conflict: Option<HostnameConflict>,
/// The overlay range this network uses, once it has one.
pub range: Option<crate::state::Ipv4Range>,
/// The range it could not have, because another network on this agent
/// already holds it.
///
/// One agent has one interface, so an address belongs to one network.
/// This network waits to adopt whatever its members settle on instead of
/// proposing something it could not route.
/// The first overlapping active range; `range_conflicts` identifies all blockers.
pub range_conflict: Option<crate::state::Ipv4Range>,
/// Active networks that would be suspended by switching to this network.
pub range_conflicts: Vec<RangeConflict>,
/// Per-network counters.
pub metrics: NetworkMetrics,
/// What has gone through a peer in the middle, in both directions.
+18 -2
View File
@@ -166,12 +166,16 @@ pub struct Limits {
pub max_capabilities: usize,
/// Largest opaque plugin payload, in bytes.
pub max_capability_data_len: usize,
/// Largest private transport negotiation payload (also hard-capped at 4096 bytes).
pub max_transport_payload_len: usize,
/// Largest accepted echo payload in a ping/pong exchange, in bytes.
pub max_echo_payload_len: usize,
/// Largest accepted free-text reason string, in bytes.
pub max_reason_len: usize,
/// Largest number of signed records accepted in one snapshot.
pub max_state_records: usize,
/// Maximum durable retirement markers retained per network.
pub max_retirements: usize,
/// Deadline for the whole handshake.
pub handshake_timeout: Duration,
/// Deadline for one outbound dial attempt.
@@ -204,10 +208,12 @@ impl Default for Limits {
max_frame_len: 64 * 1024,
max_hostname_len: 255,
max_capabilities: 16,
max_transport_payload_len: 4096,
max_capability_data_len: 4 * 1024,
max_echo_payload_len: 4 * 1024,
max_reason_len: 256,
max_state_records: crate::state::MAX_RECORDS_PER_MESSAGE,
max_retirements: 1024,
handshake_timeout: Duration::from_secs(10),
dial_timeout: Duration::from_secs(10),
write_timeout: Duration::from_secs(30),
@@ -325,6 +331,9 @@ pub struct AgentConfig {
pub reconnect: ReconnectPolicy,
/// IP plugins whose capabilities are announced and dispatched.
pub plugins: Vec<SharedPlugin>,
/// Initial ordered selection for networks without a saved override.
/// `None` enables all registered plugins; an empty list is control-only.
pub default_protocols: Option<Vec<String>>,
/// Custom packet transports registered per protocol.
pub custom_transports:
std::collections::HashMap<String, Arc<dyn crate::dataplane::PacketTransport>>,
@@ -339,7 +348,11 @@ pub struct AgentConfig {
pub interface_name: String,
/// The interface MTU.
pub interface_mtu: u32,
/// The IPv4 overlay range this agent proposes.
/// Whether new networks derive their IPv4 range from their network id.
///
/// Enabled by default. Explicit ranges and IPv4 opt-out disable this.
pub overlay_ipv4_auto: bool,
/// The IPv4 overlay range this agent proposes when automatic selection is off.
///
/// Addresses are allocated from it and recorded in signed state, so a
/// participant keeps the same one across restarts. A network that has
@@ -369,11 +382,13 @@ impl AgentConfig {
limits: Limits::default(),
reconnect: ReconnectPolicy::default(),
plugins: Vec::new(),
default_protocols: None,
custom_transports: std::collections::HashMap::new(),
tun_factory: None,
interface_name: "tsun0".to_string(),
interface_mtu: 1280,
overlay_ipv4_range: Some(crate::state::DEFAULT_IPV4_RANGE),
overlay_ipv4_auto: true,
overlay_ipv4_range: None,
}
}
@@ -445,6 +460,7 @@ impl AgentConfig {
/// Sets the IPv4 overlay range this agent proposes, or disables IPv4.
pub fn with_overlay_ipv4_range(mut self, range: Option<crate::state::Ipv4Range>) -> Self {
self.overlay_ipv4_auto = false;
self.overlay_ipv4_range = range;
self
}
+27
View File
@@ -40,6 +40,33 @@ pub use tunnel::{GenericTunnelPlugin, TunnelCodec};
/// Maximum length of a plugin protocol identifier.
pub const MAX_PROTOCOL_ID_LEN: usize = 32;
/// Validates bounded local protocol preferences before storing or applying them.
pub(crate) fn validate_protocols(protocols: &[String], max: usize) -> crate::error::Result<()> {
if protocols.len() > max {
return Err(crate::Error::InvalidProtocols(format!(
"at most {max} protocols are allowed"
)));
}
for (index, name) in protocols.iter().enumerate() {
if name.is_empty()
|| name.len() > MAX_PROTOCOL_ID_LEN
|| !name
.bytes()
.all(|c| c.is_ascii_alphanumeric() || c == b'-' || c == b'_')
{
return Err(crate::Error::InvalidProtocols(
"names must be 1–32 ASCII letters, digits, hyphens or underscores".into(),
));
}
if protocols[..index].contains(name) {
return Err(crate::Error::InvalidProtocols(format!(
"`{name}` is listed more than once"
)));
}
}
Ok(())
}
/// An announcement of one IP plugin's capability.
///
/// `data` is opaque to the core. Nothing in it may be interpreted as a shell
+10 -2
View File
@@ -492,11 +492,12 @@ impl RelayHub {
}
}
/// Revokes all links and topology involving a departed authenticated peer.
pub fn remove_peer(&self, peer: EndpointId) {
/// Retires data channels without removing an authenticated peer's topology.
pub fn clear_peer_links(&self, peer: EndpointId) {
let mut state = self.state();
if let Some(links) = state.raw.remove(peer.as_bytes()) {
for raw in links {
raw.link.close();
raw.reader.abort();
}
}
@@ -511,6 +512,13 @@ impl RelayHub {
link.close();
}
}
state.publish();
}
/// Revokes all links and topology involving a departed authenticated peer.
pub fn remove_peer(&self, peer: EndpointId) {
self.clear_peer_links(peer);
let mut state = self.state();
state.members.remove(peer.as_bytes());
state.graph.remove(peer.as_bytes());
for neighbors in state.graph.values_mut() {
@@ -0,0 +1,350 @@
//! Authenticated probes of the actual data link, independent of control RTT.
//! Keys are negotiated privately for each physical link. No fixed wire magic.
use super::{PacketLink, SharedLink, TransportError};
use crate::dataplane::SharedPlugin;
use crate::overlay::RoutingTable;
use crate::{BoxFuture, NetworkId};
use bytes::Bytes;
use hmac::{Hmac, KeyInit, Mac};
use iroh::EndpointId;
use serde::{Deserialize, Serialize};
use sha2::Sha256;
use std::collections::VecDeque;
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};
use tokio::sync::{mpsc, watch};
#[derive(Clone, Serialize, Deserialize, zeroize::ZeroizeOnDrop)]
pub(crate) struct Settings {
marker: [u8; 16],
key: [u8; 32],
}
impl Settings {
pub(crate) fn fresh() -> Self {
Self {
marker: rand::random(),
key: rand::random(),
}
}
}
impl std::fmt::Debug for Settings {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str("HealthSettings([REDACTED])")
}
}
struct Samples {
pending: VecDeque<(u64, Instant)>,
recent: VecDeque<bool>,
}
impl Samples {
fn new() -> Self {
Self {
pending: VecDeque::new(),
recent: VecDeque::new(),
}
}
fn sent(&mut self, seq: u64, now: Instant) {
self.pending.push_back((seq, now));
}
fn acknowledge(&mut self, seq: u64) {
if let Some(i) = self.pending.iter().position(|(n, _)| *n == seq) {
self.pending.remove(i);
self.recent.push_back(true);
}
}
fn expire(&mut self, now: Instant) {
while self
.pending
.front()
.is_some_and(|(_, at)| now.duration_since(*at) >= Duration::from_secs(3))
{
self.pending.pop_front();
self.recent.push_back(false);
}
while self.recent.len() > 20 {
self.recent.pop_front();
}
}
fn degraded(&self) -> bool {
self.recent.len() >= 15
&& self.recent.iter().filter(|ok| !**ok).count() * 100 >= self.recent.len() * 60
}
}
// This prefix is inside end-to-end encryption. It never appears on the UDP wire.
const IP_MAGIC: &[u8; 16] = b"tsunagi-health01";
pub(crate) fn ip_payload(packet: &[u8]) -> Option<&[u8]> {
(packet.len() >= 36 && packet[0] == 0x45 && packet[9] == 253 && &packet[20..36] == IP_MAGIC)
.then(|| &packet[36..])
}
/// Inject probes through the same encrypted tunnel as application IP packets.
/// This matters for pure WG: raw UDP probes could pass DPI while WG is blocked.
pub(crate) struct ProbeRoute {
pub(crate) plugin: SharedPlugin,
pub(crate) routes: Arc<RoutingTable>,
pub(crate) network: NetworkId,
pub(crate) peer: EndpointId,
pub(crate) max_probe_size: usize,
}
impl ProbeRoute {
fn send(&self, payload: Bytes) -> bool {
let Some((source, destination)) = self.routes.probe_addresses(self.network, self.peer)
else {
return false;
};
let mut packet = vec![0; 36];
packet[0] = 0x45;
packet[8] = 64;
packet[9] = 253;
packet[12..16].copy_from_slice(&source.octets());
packet[16..20].copy_from_slice(&destination.octets());
packet[20..36].copy_from_slice(IP_MAGIC);
let length = (packet.len() + payload.len()) as u16;
packet[2..4].copy_from_slice(&length.to_be_bytes());
let sum = packet[..20]
.as_chunks::<2>()
.0
.iter()
.map(|b| u32::from(u16::from_be_bytes(*b)))
.sum::<u32>();
let sum = (sum & 0xffff) + (sum >> 16);
packet[10..12].copy_from_slice(&(!(sum as u16)).to_be_bytes());
packet.extend_from_slice(&payload);
self.plugin
.carry(self.network, self.peer, Bytes::from(packet))
}
}
/// A monitored physical link. It forwards application bytes unchanged.
pub(crate) struct HealthLink {
probes: mpsc::Sender<Bytes>,
raw: SharedLink,
settings: Mutex<Option<Settings>>,
rx: tokio::sync::Mutex<mpsc::Receiver<Bytes>>,
stop: watch::Sender<bool>,
reason: Mutex<Option<String>>,
}
impl std::fmt::Debug for HealthLink {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("HealthLink")
.field("raw", &self.raw)
.finish_non_exhaustive()
}
}
fn lock<T>(m: &Mutex<T>) -> std::sync::MutexGuard<'_, T> {
match m.lock() {
Ok(g) => g,
Err(p) => p.into_inner(),
}
}
impl HealthLink {
pub(crate) fn new(
raw: SharedLink,
local: EndpointId,
settings: Option<Settings>,
route: Option<ProbeRoute>,
) -> Arc<Self> {
let (tx, rx) = mpsc::channel(64);
let (probe_tx, mut probe_rx) = mpsc::channel(64);
let (stop, _) = watch::channel(false);
let link = Arc::new(Self {
probes: probe_tx,
raw: raw.clone(),
settings: Mutex::new(settings),
rx: tokio::sync::Mutex::new(rx),
stop,
reason: Mutex::new(None),
});
let weak = Arc::downgrade(&link);
let mut stopped = link.stop.subscribe();
tokio::spawn(async move {
let mut interval = tokio::time::interval(Duration::from_secs(1));
interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
let mut samples = Samples::new();
let mut sequence = rand::random::<u64>();
let mut since: Option<Instant> = None;
let mut marker = None;
loop {
tokio::select! {
biased;
_=stopped.changed()=>break,
_=interval.tick()=>{
let Some(link)=weak.upgrade()else{break;};let Some(settings)=lock(&link.settings).clone()else{continue;};
if marker != Some(settings.marker) { marker = Some(settings.marker); samples = Samples::new(); since = None; }
let start=*since.get_or_insert_with(Instant::now);let now=Instant::now();samples.expire(now);
if now.duration_since(start)>=Duration::from_secs(20) && samples.degraded() {
*lock(&link.reason)=Some("data delivery checks: at least 60% loss over the last 20 probes".into());break;
}
sequence=sequence.wrapping_add(1);
let size=route.as_ref().map_or(raw.max_datagram_size(),|route|route.max_probe_size);
let packet=probe(&settings,local,1,sequence,size.min(1000));
if let Some(route)=&route { if route.send(packet) {samples.sent(sequence,now);} }
else { let _=raw.send(packet); samples.sent(sequence,now); }
}
packet=raw.recv()=>{
let Some(packet)=packet else{break;};let Some(link)=weak.upgrade()else{break;};
let settings=lock(&link.settings).clone();
if route.is_none() && let Some(settings)=settings && packet.starts_with(&settings.marker) {
if let Some((kind,seq))=verify(&settings,raw.peer(),&packet) {
if kind==1 {let _=raw.send(probe(&settings,local,2,seq,packet.len()));}
else if kind==2 {samples.acknowledge(seq);}
}
} else {let _=tx.try_send(packet);}
}
packet=probe_rx.recv(),if route.is_some()=>{
let Some(packet)=packet else {break;};let Some(link)=weak.upgrade()else{break;};
let Some(settings)=lock(&link.settings).clone()else{continue;};
if let Some((kind,seq))=verify(&settings,raw.peer(),&packet) {
if kind==1 { if let Some(route)=&route { route.send(probe(&settings,local,2,seq,packet.len().min(route.max_probe_size))); } }
else if kind==2 {samples.acknowledge(seq);}
}
}
}
}
raw.close();
if let Some(link) = weak.upgrade() {
link.stop.send_replace(true);
}
});
link
}
pub(crate) fn receive_probe(&self, packet: Bytes) {
let _ = self.probes.try_send(packet);
}
pub(crate) fn configure(&self, settings: Settings) {
*lock(&self.settings) = Some(settings);
}
}
fn probe(settings: &Settings, sender: EndpointId, kind: u8, seq: u64, size: usize) -> Bytes {
let mut bytes = vec![0; size.max(57)];
bytes[..16].copy_from_slice(&settings.marker);
bytes[16] = kind;
bytes[17..25].copy_from_slice(&seq.to_be_bytes());
rand::fill(&mut bytes[25..size.max(57) - 32]);
let tail = bytes.len() - 32;
if let Ok(mut mac) = <Hmac<Sha256> as KeyInit>::new_from_slice(&settings.key) {
mac.update(sender.as_bytes());
mac.update(&bytes[..tail]);
bytes[tail..].copy_from_slice(&mac.finalize().into_bytes());
}
Bytes::from(bytes)
}
fn verify(settings: &Settings, sender: EndpointId, bytes: &[u8]) -> Option<(u8, u64)> {
if bytes.len() < 57 || !bytes.starts_with(&settings.marker) {
return None;
}
let mut mac = <Hmac<Sha256> as KeyInit>::new_from_slice(&settings.key).ok()?;
let tail = bytes.len() - 32;
mac.update(sender.as_bytes());
mac.update(&bytes[..tail]);
mac.verify_slice(&bytes[tail..]).ok()?;
Some((
bytes[16],
u64::from_be_bytes(bytes.get(17..25)?.try_into().ok()?),
))
}
impl PacketLink for HealthLink {
fn network(&self) -> NetworkId {
self.raw.network()
}
fn peer(&self) -> EndpointId {
self.raw.peer()
}
fn max_datagram_size(&self) -> usize {
self.raw.max_datagram_size()
}
fn send(&self, payload: Bytes) -> Result<(), TransportError> {
if self.is_closed() {
Err(TransportError::Closed)
} else {
self.raw.send(payload)
}
}
fn send_flow(
&self,
payload: Bytes,
flow: super::super::routing::FlowId,
) -> Result<(), TransportError> {
if self.is_closed() {
Err(TransportError::Closed)
} else {
self.raw.send_flow(payload, flow)
}
}
fn recv(&self) -> BoxFuture<'_, Option<Bytes>> {
Box::pin(async move {
let mut stop = self.stop.subscribe();
if *stop.borrow() {
return None;
}
let mut rx = self.rx.lock().await;
tokio::select! {biased;_=stop.changed()=>None,packet=rx.recv()=>packet}
})
}
fn closed(&self) -> BoxFuture<'_, ()> {
Box::pin(async move {
let mut stop = self.stop.subscribe();
if !*stop.borrow() {
let _ = stop.changed().await;
}
})
}
fn is_closed(&self) -> bool {
*self.stop.borrow() || self.raw.is_closed()
}
fn close(&self) {
self.stop.send_replace(true);
self.raw.close();
}
fn failure_reason(&self) -> Option<String> {
lock(&self.reason)
.clone()
.or_else(|| self.raw.failure_reason())
}
fn path_description(&self) -> String {
self.raw.path_description()
}
}
impl Drop for HealthLink {
fn drop(&mut self) {
self.stop.send_replace(true);
self.raw.close();
}
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
#[test]
fn no_false_failure_before_window_but_persistent_loss_is_detected() {
let mut samples = Samples::new();
let now = Instant::now();
for seq in 0..20 {
samples.sent(seq, now);
if seq % 5 == 0 {
samples.acknowledge(seq);
}
}
assert!(!samples.degraded());
samples.expire(now + Duration::from_secs(4));
assert!(samples.degraded());
for seq in 20..40 {
samples.sent(seq, now);
samples.acknowledge(seq);
}
samples.expire(now + Duration::from_secs(4));
assert!(!samples.degraded());
}
#[test]
fn a_forged_or_reflected_probe_cannot_claim_delivery() {
let a = crate::identity::DeviceIdentity::generate().endpoint_id();
let b = crate::identity::DeviceIdentity::generate().endpoint_id();
let settings = Settings::fresh();
let packet = probe(&settings, a, 1, 42, 1100);
assert_eq!(verify(&settings, a, &packet), Some((1, 42)));
assert!(verify(&settings, b, &packet).is_none());
let mut forged = packet.to_vec();
forged[16] = 2;
assert!(verify(&settings, a, &forged).is_none());
}
}
@@ -97,6 +97,9 @@ impl IrohLink {
}
impl PacketLink for IrohLink {
fn close(&self) {
self.conn.close(0u32.into(), b"data link retired");
}
fn network(&self) -> NetworkId {
self.network
}
@@ -30,12 +30,14 @@
//! one plugin protocol.
mod fragments;
pub(crate) mod health;
pub mod iroh_link;
use std::sync::Arc;
use bytes::Bytes;
use iroh::EndpointId;
use serde::{Deserialize, Serialize};
use crate::BoxFuture;
use crate::identity::NetworkId;
@@ -112,6 +114,14 @@ pub trait PacketLink: Send + Sync + std::fmt::Debug + 'static {
/// keeping a task parked on [`PacketLink::closed`].
fn is_closed(&self) -> bool;
/// Explicitly retires a physical link. Implementations should wake readers.
fn close(&self) {}
/// Observed reason for retiring this link, if available.
fn failure_reason(&self) -> Option<String> {
None
}
/// A short description of the path in use, for diagnostics.
///
/// Reports what the transport actually knows. It must not invent a value.
@@ -121,6 +131,76 @@ pub trait PacketLink: Send + Sync + std::fmt::Debug + 'static {
/// A shared handle to a link.
pub type SharedLink = std::sync::Arc<dyn PacketLink>;
/// Maximum opaque pair-negotiation payload.
pub const MAX_TRANSPORT_PAYLOAD: usize = 4096;
/// Private transport bytes, never included in Debug output.
#[derive(Clone, PartialEq, Eq, Serialize, zeroize::ZeroizeOnDrop)]
pub struct TransportPayload(Vec<u8>);
impl<'de> Deserialize<'de> for TransportPayload {
fn deserialize<D: serde::Deserializer<'de>>(
deserializer: D,
) -> std::result::Result<Self, D::Error> {
struct Visitor;
impl<'de> serde::de::Visitor<'de> for Visitor {
type Value = TransportPayload;
fn expecting(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "at most {MAX_TRANSPORT_PAYLOAD} private transport bytes")
}
fn visit_seq<A: serde::de::SeqAccess<'de>>(
self,
mut sequence: A,
) -> std::result::Result<Self::Value, A::Error> {
if sequence
.size_hint()
.is_some_and(|size| size > MAX_TRANSPORT_PAYLOAD)
{
return Err(serde::de::Error::custom("transport payload too large"));
}
let mut payload = TransportPayload(Vec::with_capacity(
sequence.size_hint().unwrap_or(0).min(MAX_TRANSPORT_PAYLOAD),
));
while let Some(byte) = sequence.next_element::<u8>()? {
if payload.0.len() == MAX_TRANSPORT_PAYLOAD {
return Err(serde::de::Error::custom("transport payload too large"));
}
payload.0.push(byte);
}
Ok(payload)
}
}
deserializer.deserialize_seq(Visitor)
}
}
impl TransportPayload {
/// Wraps bytes for one authenticated control session.
pub fn new(bytes: Vec<u8>) -> Self {
Self(bytes)
}
/// Borrows private bytes; callers must not log them.
pub fn as_bytes(&self) -> &[u8] {
&self.0
}
}
impl std::fmt::Debug for TransportPayload {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str("TransportPayload([REDACTED])")
}
}
/// Private control-plane sender for transports. Contains no network secret.
pub trait ControlSender: Send + Sync + std::fmt::Debug + 'static {
/// Queues a bounded message to one authenticated member of one network.
fn send<'a>(
&'a self,
network: NetworkId,
peer: EndpointId,
protocol: &'a str,
payload: TransportPayload,
) -> BoxFuture<'a, Result<(), TransportError>>;
}
/// An inbound link a peer opened towards us.
#[derive(Debug)]
pub struct InboundLink {
@@ -139,6 +219,18 @@ pub struct InboundLink {
/// The agent owns one of these and hands links to plugins; plugins never call
/// it directly.
pub trait PacketTransport: Send + Sync + std::fmt::Debug + 'static {
/// Supplies a private sender before network activation.
fn attach_control(&self, _control: Arc<dyn ControlSender>) {}
/// Validates and queues private negotiation received from an authenticated member.
fn on_control(
&self,
_network: NetworkId,
_peer: EndpointId,
_protocol: &str,
_payload: &[u8],
) -> Result<(), TransportError> {
Err(TransportError::Declined(self.name().into()))
}
/// A short name used in diagnostics.
fn name(&self) -> &str;
@@ -184,6 +276,24 @@ impl MultiPacketTransport {
}
impl PacketTransport for MultiPacketTransport {
fn attach_control(&self, control: Arc<dyn ControlSender>) {
self.default.attach_control(control.clone());
for transport in self.by_protocol.values() {
transport.attach_control(control.clone());
}
}
fn on_control(
&self,
network: NetworkId,
peer: EndpointId,
protocol: &str,
payload: &[u8],
) -> Result<(), TransportError> {
match self.by_protocol.get(protocol) {
Some(transport) => transport.on_control(network, peer, protocol, payload),
None => Err(TransportError::Declined(protocol.into())),
}
}
fn name(&self) -> &str {
"multi"
}
+19
View File
@@ -15,6 +15,12 @@ pub type Result<T, E = Error> = std::result::Result<T, E>;
#[derive(Debug, thiserror::Error)]
#[non_exhaustive]
pub enum Error {
/// A node-history cleanup is unsafe, stale or cannot be performed.
#[error("cannot forget node: {0}")]
ForgetNode(String),
/// An ordered local data-plane protocol selection is invalid.
#[error("invalid protocol selection: {0}")]
InvalidProtocols(String),
/// The supplied network name does not satisfy the documented rules.
#[error("invalid network name: {0}")]
InvalidNetworkName(&'static str),
@@ -97,6 +103,19 @@ pub enum Error {
#[error("network {0} is not configured")]
NetworkUnknown(NetworkId),
/// An overlay range overlaps networks already active on this agent.
#[error(
"network {network} cannot start: {range} overlaps active networks {conflicts:?}; switch to it to suspend those networks"
)]
NetworkRangeConflict {
/// Network that could not start.
network: NetworkId,
/// Its agreed or proposed range.
range: crate::state::Ipv4Range,
/// Active networks that must be suspended first.
conflicts: Vec<NetworkId>,
},
/// No session with that peer exists in the given network.
#[error("no authenticated session with peer {peer} in network {network}")]
NoSuchPeer {
+263 -2
View File
@@ -171,6 +171,22 @@ pub fn default_control_socket_path() -> PathBuf {
#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)]
#[non_exhaustive]
pub enum Request {
/// Retire the selected offline member's observed claim.
ForgetNode {
/// Exact public network id.
network_id: String,
/// Exact device identity selected by the client.
peer: String,
/// Claim version displayed when the user selected the node.
expected_version: u64,
},
/// Replace one network's ordered local data-plane selection.
SetProtocols {
/// Public network identifier.
network_id: String,
/// Protocol names, best first; empty selects control only.
protocols: Vec<String>,
},
/// Report what the agent is doing.
Status,
/// Set local broadcast participation in one configured network.
@@ -255,6 +271,21 @@ pub enum Request {
impl std::fmt::Debug for Request {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
Request::ForgetNode {
network_id,
peer,
expected_version,
} => write!(
f,
"ForgetNode {{ {network_id}, peer: {peer}, version: {expected_version} }}"
),
Request::SetProtocols {
network_id,
protocols,
} => write!(
f,
"SetProtocols {{ {network_id}, protocols: {protocols:?} }}"
),
Request::Status => f.write_str("Status"),
Request::SetBroadcast {
network_id,
@@ -286,6 +317,10 @@ impl std::fmt::Debug for Request {
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[non_exhaustive]
pub enum Response {
/// An observed offline claim was durably retired.
NodeForgotten(ForgottenNodeReport),
/// Accepted ordered protocol selection.
Protocols(Vec<String>),
/// A status report.
Status(Box<StatusReport>),
/// Accepted local broadcast participation.
@@ -312,6 +347,141 @@ pub enum Response {
Error(String),
}
/// The durable result of an offline-node cleanup.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct ForgottenNodeReport {
/// Selected device identity.
pub peer: String,
/// The retired claim version, not a future-version ban.
pub record_version: u64,
/// Peers to which cleanup was immediately queued.
pub peers_told: u32,
}
/// Resolves a node by exact hostname, full id, unique id prefix or overlay IPv4.
/// Ambiguity is reported with names and full ids and never guessed.
pub fn resolve_node<'a>(
network: &'a NetworkReport,
selector: &str,
) -> std::result::Result<&'a MemberReport, String> {
let selector = selector.trim();
if selector.is_empty() {
return Err("select a node by name, id or IPv4 address".into());
}
if let Some(member) = network.members.iter().find(|m| m.endpoint_id == selector) {
return Ok(member);
}
let prefix = selector.trim_end_matches('…');
let matches: Vec<_> = network
.members
.iter()
.filter(|m| {
m.hostname.as_deref() == Some(selector)
|| m.overlay_address_v4.as_deref() == Some(selector)
|| (!prefix.is_empty() && m.endpoint_id.starts_with(prefix))
})
.collect();
match matches.as_slice() {
[one] => Ok(one),
[] => Err(format!(
"no signed node matches `{selector}` in `{}`",
network.name
)),
many => Err(format!(
"`{selector}` is ambiguous; use a full id or unique id prefix:\n{}",
many.iter()
.map(|m| format!(
" {} {} {}",
m.hostname.as_deref().unwrap_or("unnamed"),
m.endpoint_id,
m.overlay_address_v4.as_deref().unwrap_or("no IPv4")
))
.collect::<Vec<_>>()
.join("\n")
)),
}
}
/// Retires an exact selected claim; an online or changed target is rejected.
pub async fn forget_node(
socket: &Path,
network: &str,
peer: &str,
expected_version: u64,
) -> Result<ForgottenNodeReport> {
let request = Request::ForgetNode {
network_id: network.into(),
peer: peer.into(),
expected_version,
};
match exchange(socket, &request, EXCHANGE_TIMEOUT).await? {
Response::NodeForgotten(report) => Ok(report),
Response::Error(reason) => Err(Error::ForgetNode(reason)),
other => Err(Error::Storage(format!("unexpected response: {other:?}"))),
}
}
#[cfg(test)]
mod node_selector_tests {
use super::*;
#[test]
fn exact_names_addresses_and_unique_prefixes_select_the_same_signed_node() {
let node = MemberReport {
endpoint_id: "abcdef01".into(),
hostname: Some("laptop".into()),
overlay_address_v4: Some("10.2.0.3".into()),
record_version: 42,
..Default::default()
};
let network = NetworkReport {
name: "home".into(),
members: vec![node.clone()],
..Default::default()
};
for selector in ["laptop", "10.2.0.3", "abcdef01", "abc…"] {
assert_eq!(resolve_node(&network, selector), Ok(&node));
}
for selector in ["", "LAPTOP", "missing"] {
assert!(resolve_node(&network, selector).is_err());
}
}
#[test]
fn collisions_between_names_addresses_and_prefixes_require_an_explicit_identity() {
let first = MemberReport {
endpoint_id: "abcdef01".into(),
hostname: Some("duplicate".into()),
overlay_address_v4: Some("10.2.0.3".into()),
..Default::default()
};
let second = MemberReport {
endpoint_id: "abcdef02".into(),
hostname: Some("duplicate".into()),
overlay_address_v4: Some("10.2.0.3".into()),
..Default::default()
};
let third = MemberReport {
endpoint_id: "12345678".into(),
hostname: Some("abc".into()),
..Default::default()
};
let network = NetworkReport {
name: "home".into(),
members: vec![first.clone(), second, third],
..Default::default()
};
for selector in ["duplicate", "10.2.0.3", "abc"] {
let result = resolve_node(&network, selector);
assert!(result.is_err_and(|error| error.contains("ambiguous")
&& error.contains("abcdef01")
&& error.contains("abcdef02")
&& error.contains("duplicate")));
}
assert_eq!(resolve_node(&network, "abcdef01"), Ok(&first));
}
}
/// What happened when a network was stopped or started.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct ActiveReport {
@@ -324,6 +494,9 @@ pub struct ActiveReport {
/// Stopping something already stopped is not an error — it is the state
/// asked for — but it is worth saying which happened.
pub changed: bool,
/// Networks suspended by switching to this network.
#[serde(default)]
pub suspended: Vec<RangeConflictReport>,
}
/// What happened when a network was joined.
@@ -345,6 +518,13 @@ pub struct JoinedReport {
/// Almost always a mistyped secret, and the one thing that makes two
/// sections of a report look like one network.
pub name_shared_with: Option<String>,
/// Whether it started; false means it is configured but blocked by overlap.
#[serde(default = "joined_active_default")]
pub active: bool,
}
fn joined_active_default() -> bool {
true
}
/// What happened when a network was left.
@@ -367,6 +547,8 @@ pub struct LeftReport {
/// Everything the agent is doing, in one snapshot.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct StatusReport {
/// Data-plane protocols this agent can enable at runtime.
pub available_protocols: Vec<ProtocolReport>,
/// This device's persistent endpoint id.
pub endpoint_id: String,
/// Hostname announced to peers.
@@ -437,9 +619,34 @@ pub struct DnsReport {
pub names: u32,
}
/// An active network with an overlapping overlay range.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct RangeConflictReport {
/// Public network id, for unambiguous selection.
pub network_id: String,
/// Configured network name.
pub name: String,
/// The overlapping range.
pub range: String,
}
/// A data-plane protocol available in the running agent.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct ProtocolReport {
/// Stable protocol name used by the CLI and network negotiation.
pub name: String,
/// A short description of encryption and transport.
pub summary: String,
}
/// One network.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct NetworkReport {
/// Enabled local data-plane protocols, in preference order.
pub protocols: Vec<String>,
/// Whether the network starts or resumes automatically when its range is free.
#[serde(default)]
pub auto_start: bool,
/// Whether this agent participates in LAN broadcasts in this network.
#[serde(default)]
pub broadcast: bool,
@@ -482,6 +689,9 @@ pub struct NetworkReport {
/// The range it could not have, because another network on this agent
/// already holds it.
pub range_conflict: Option<String>,
/// Networks that switching to this network would suspend.
#[serde(default)]
pub range_conflicts: Vec<RangeConflictReport>,
/// The overlay, when a protocol is running one.
pub overlay: Option<OverlayReport>,
/// This agent's own name, when another member claimed it first.
@@ -524,6 +734,8 @@ pub struct HostnameConflictReport {
/// One member of the network, from signed state.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct MemberReport {
/// Signed claim version used to reject stale cleanup requests.
pub record_version: u64,
/// The member's device identity.
pub endpoint_id: String,
/// The IPv4 overlay address it claimed and signed for.
@@ -693,6 +905,23 @@ impl OverlayPeerReport {
/// running and what they can report or change. That is what keeps this module
/// free of any knowledge of them.
pub trait ReportSource: Send + Sync + 'static {
/// Persists and announces a version-scoped node cleanup.
fn forget_node(
&self,
_network: String,
_peer: String,
_expected_version: u64,
) -> BoxFuture<'_, std::result::Result<ForgottenNodeReport, String>> {
Box::pin(async { Err("this agent cannot forget node history".into()) })
}
/// Saves and applies a local ordered protocol set for one network.
fn set_protocols(
&self,
_network_id: String,
_protocols: Vec<String>,
) -> BoxFuture<'_, std::result::Result<Vec<String>, String>> {
Box::pin(async { Err("this agent cannot change protocols".into()) })
}
/// Produces a fresh report.
fn report(&self) -> BoxFuture<'_, StatusReport>;
@@ -822,7 +1051,7 @@ pub const EXCHANGE_TIMEOUT: Duration = Duration::from_secs(5);
///
/// Bump it whenever [`Request`], [`Response`] or anything they contain
/// changes shape.
pub const CONTROL_PROTOCOL: u32 = u32::from_be_bytes([b'T', b'S', b'N', 18]);
pub const CONTROL_PROTOCOL: u32 = u32::from_be_bytes([b'T', b'S', b'N', 21]);
/// Reads one request off an accepted stream, answers it, writes the response.
///
@@ -846,6 +1075,21 @@ where
}
};
let response = match request {
Request::ForgetNode {
network_id,
peer,
expected_version,
} => match source.forget_node(network_id, peer, expected_version).await {
Ok(report) => Response::NodeForgotten(report),
Err(reason) => Response::Error(reason),
},
Request::SetProtocols {
network_id,
protocols,
} => match source.set_protocols(network_id, protocols).await {
Ok(protocols) => Response::Protocols(protocols),
Err(reason) => Response::Error(reason),
},
Request::Status => Response::Status(Box::new(source.report().await)),
Request::SetBroadcast {
network_id,
@@ -1001,7 +1245,24 @@ pub async fn join_network_with_broadcast(
}
}
/// Changes one network's broadcast participation while the agent runs.
/// Replaces the enabled data-plane protocols of one configured network.
pub async fn set_protocols(
socket: &Path,
network_id: &str,
protocols: Vec<String>,
) -> Result<Vec<String>> {
let request = Request::SetProtocols {
network_id: network_id.to_owned(),
protocols,
};
match exchange(socket, &request, EXCHANGE_TIMEOUT).await? {
Response::Protocols(protocols) => Ok(protocols),
Response::Error(reason) => Err(Error::Storage(reason)),
other => Err(Error::Storage(format!("unexpected response: {other:?}"))),
}
}
/// Changes local broadcast participation in a network.
pub async fn set_broadcast(
path: impl AsRef<Path>,
network_id: &str,
@@ -84,6 +84,11 @@ pub trait ExitHostRules: Send + Sync + std::fmt::Debug + 'static {
/// startup to take away what a crashed run left.
fn clear<'a>(&'a self, interface: &'a str) -> BoxFuture<'a, ()>;
/// Refreshes volatile local socket protection without replacing exit routes.
fn refresh<'a>(&'a self, _plan: &'a ExitHostPlan) -> BoxFuture<'a, Option<RuleOutcome>> {
Box::pin(async { None })
}
/// Whether using an exit node on this host needs the underlay addresses
/// the agent talks to, because the host cannot exempt the agent's own
/// traffic from the default route by user id (Windows). The default is no.
@@ -33,12 +33,13 @@ const CHAIN: &str = "INPUT";
/// Fixed places `iptables` is installed. Never `PATH`: the agent is
/// privileged and a directory a user controls must not decide what it runs.
const IPTABLES_DIRECTORIES: [&str; 5] = [
const IPTABLES_DIRECTORIES: [&str; 6] = [
"/usr/sbin/iptables",
"/sbin/iptables",
"/usr/bin/iptables",
"/bin/iptables",
"/usr/local/sbin/iptables",
"/run/current-system/sw/bin/iptables",
];
/// The first candidate that exists.
@@ -328,7 +329,23 @@ fn clear_firewall(run: Run<'_>, interface: &str) {
/// Runs the real `iptables`.
pub(super) fn run_iptables(args: &[String]) -> Ran {
let Some(program) = pick_iptables(|path| std::path::Path::new(path).is_file()) else {
run_firewall(false, args)
}
pub(super) fn run_firewall(ipv6: bool, args: &[String]) -> Ran {
let v6_paths: Vec<_> = IPTABLES_DIRECTORIES
.iter()
.map(|p| p.replace("iptables", "ip6tables"))
.collect();
let v6_program = v6_paths
.iter()
.find(|path| std::path::Path::new(path).is_file());
let program = if ipv6 {
v6_program.map(String::as_str)
} else {
pick_iptables(|path| std::path::Path::new(path).is_file())
};
let Some(program) = program else {
return Ran::Missing;
};
match std::process::Command::new(program).args(args).output() {
@@ -350,11 +367,17 @@ pub(super) fn run_iptables(args: &[String]) -> Ran {
#[derive(Debug)]
pub struct LinuxHostRules {
pub(super) route: RouteHandle,
/// Interface, sorted socket snapshot and whether exit routes committed.
pub(super) underlay_sockets:
std::sync::Mutex<Option<(String, Vec<super::linux_exit_mark::Listener>, bool)>>,
}
impl LinuxHostRules {
pub(crate) fn new(route: RouteHandle) -> Self {
Self { route }
Self {
route,
underlay_sockets: std::sync::Mutex::new(None),
}
}
async fn set_route(&self, interface: String, source: Ipv4Addr) -> RuleOutcome {
@@ -25,18 +25,19 @@
//!
//! ```text
//! table 0x7473: default dev <if>
//! 5280: from all uidrange <euid>-<euid> lookup main
//! 5205: from all fwmark 0x40000000/0x40000000 lookup main
//! 5206: from all uidrange <euid>-<euid> lookup main
//! 5290: from all lookup main suppress_prefixlength 0
//! 5300: from all lookup 0x7473
//! ```
//!
//! The first keeps the agent's *own* sockets off the tunnel: its QUIC, relay,
//! The first two keep the agent's *own* sockets and marked replies off the tunnel: its QUIC, relay,
//! DHT and DNS traffic would otherwise be routed into the interface it is
//! carrying, a loop. The second lets every specific route — the LAN, another
//! carrying, a loop. The specific-route rule lets the LAN, another
//! VPN, a tunnel to somewhere else — keep working, because only the default
//! route is suppressed. The priorities sit after Tailscale's 5270.
//! route is suppressed. Mark/UID bypass precedes Tailscale; application rules follow it.
//!
//! The first rule exempts a *user id*, so it is only right when the agent has
//! The UID rule exempts a *user id*, so it is only right when the agent has
//! one of its own, as the packaged service does. An agent started from a login
//! session would take that user's own applications out of the tunnel with it,
//! and the exit node would silently do nothing for them. That is refused, not
@@ -48,9 +49,10 @@ use crate::state::Ipv4Range;
use super::RuleOutcome;
use super::exit::{ExitHostPlan, ExitHostReport, ExitHostRules, exit_tag, exit_tag_prefix};
use super::linux::{
LinuxHostRules, Ran, Run, comment_supported, failure, locked, run_iptables, same_rule,
split_rule_line,
LinuxHostRules, Ran, Run, comment_supported, failure, locked, run_firewall, run_iptables,
same_rule, split_rule_line,
};
use super::linux_exit_mark;
/// The chains the offering rules live in: `(table, chain)`.
const CHAINS: [(&str, &str); 2] = [("nat", "POSTROUTING"), ("filter", "FORWARD")];
@@ -377,6 +379,32 @@ fn read_sysctl(path: &str) -> Option<String> {
std::fs::read_to_string(path).ok()
}
fn check_kernel_rpf(overlay: &str) -> Result<(), String> {
let directory = "/proc/sys/net/ipv4/conf";
let value = |name: &str, setting: &str| -> Result<u32, String> {
read_sysctl(&format!("{directory}/{name}/{setting}"))
.and_then(|s| s.trim().parse().ok())
.ok_or_else(|| format!("cannot check {name}/{setting} before installing exit routes"))
};
let mut interfaces = Vec::new();
for entry in std::fs::read_dir(directory).map_err(|err| err.to_string())? {
let entry = entry.map_err(|err| err.to_string())?;
let name = entry.file_name().to_string_lossy().into_owned();
if name != "all" && name != "default" && name != "lo" && name != overlay {
interfaces.push((
name.clone(),
value(&name, "rp_filter")?,
value(&name, "src_valid_mark")?,
));
}
}
linux_exit_mark::check_kernel_rpf(
value("all", "rp_filter")?,
value("all", "src_valid_mark")?,
&interfaces,
)
}
/// The effective user id from the text of `/proc/self/status`.
fn parse_euid(status: &str) -> Option<u32> {
status
@@ -417,7 +445,7 @@ impl LinuxHostRules {
let xdg = std::env::var("XDG_RUNTIME_DIR").ok();
if let Some(reason) = client_refusal(xdg.as_deref()) {
// Whatever an earlier run left must not stay in force.
self.clear_client().await;
let _ = self.clear_client(interface).await;
return RuleOutcome::Failed(reason);
}
let Some(euid) = std::fs::read_to_string("/proc/self/status")
@@ -426,24 +454,147 @@ impl LinuxHostRules {
else {
return RuleOutcome::Failed("cannot tell which user this agent runs as".to_string());
};
let installed = self
.underlay_sockets
.lock()
.unwrap_or_else(|p| p.into_inner())
.as_ref()
.is_some_and(|(name, _, routed)| name == interface && *routed);
if installed {
// A failed socket refresh must never remove an active exit route.
return self
.refresh_sockets(interface)
.await
.unwrap_or(RuleOutcome::Applied);
}
if let Err(reason) = self.clear_client(interface).await {
return RuleOutcome::Failed(reason);
}
let name = interface.to_string();
let prepared = Self::blocking(move || {
check_kernel_rpf(&name)?;
linux_exit_mark::install(
&run_firewall,
&name,
euid,
std::path::Path::new("/proc/sys/net/ipv6").is_dir(),
)
})
.await;
match prepared {
Ok(Ok(sockets)) => {
*self
.underlay_sockets
.lock()
.unwrap_or_else(|p| p.into_inner()) = Some((interface.into(), sockets, false));
}
Ok(Err(reason)) | Err(reason) => return RuleOutcome::Failed(reason),
}
let route = self.route.clone();
let interface = interface.to_string();
match Self::blocking(move || route.set_exit_client(&interface, euid)).await {
let name = interface.to_string();
let outcome = match Self::blocking(move || route.set_exit_client(&name, euid)).await {
Ok(Ok(())) => RuleOutcome::Applied,
Ok(Err(err)) => RuleOutcome::Failed(err.to_string()),
Err(reason) => RuleOutcome::Failed(reason),
};
if let RuleOutcome::Failed(reason) = outcome {
let reason = match self.clear_client(interface).await {
Ok(()) => reason,
Err(cleanup) => format!("{reason}; cannot roll back exit rules: {cleanup}"),
};
RuleOutcome::Failed(reason)
} else {
if let Some((_, _, routed)) = self
.underlay_sockets
.lock()
.unwrap_or_else(|p| p.into_inner())
.as_mut()
{
*routed = true;
}
outcome
}
}
async fn clear_client(&self) {
async fn clear_client(&self, interface: &str) -> Result<(), String> {
let route = self.route.clone();
if let Ok(Err(err)) = Self::blocking(move || route.clear_exit_client()).await {
tracing::warn!(%err, "cannot remove the exit node routing rules");
Self::blocking(move || route.clear_exit_client())
.await?
.map_err(|err| err.to_string())?;
// Route removal must precede removing reply protection.
let name = interface.to_string();
Self::blocking(move || {
linux_exit_mark::clear(
&run_firewall,
&name,
std::path::Path::new("/proc/sys/net/ipv6").is_dir(),
)
})
.await??;
*self
.underlay_sockets
.lock()
.unwrap_or_else(|p| p.into_inner()) = None;
Ok(())
}
async fn refresh_sockets(&self, interface: &str) -> Option<RuleOutcome> {
let previous = self
.underlay_sockets
.lock()
.unwrap_or_else(|p| p.into_inner())
.clone();
if previous
.as_ref()
.is_none_or(|(name, _, routed)| name != interface || !routed)
{
return None;
}
let name = interface.to_string();
let work_name = name.clone();
let result = Self::blocking(move || {
let uid = std::fs::read_to_string("/proc/self/status")
.ok()
.and_then(|s| parse_euid(&s))
.ok_or("cannot read agent UID")?;
check_kernel_rpf(&work_name)?;
let v6 = std::path::Path::new("/proc/sys/net/ipv6").is_dir();
let sockets = linux_exit_mark::local_sockets(uid, v6)?;
if previous
.as_ref()
.is_some_and(|(name, old, _)| name == &work_name && old == &sockets)
{
return Ok(None);
}
linux_exit_mark::refresh(&run_firewall, &work_name, v6, &sockets)?;
Ok(Some(sockets))
})
.await;
match result {
Ok(Ok(None)) => None,
Ok(Ok(Some(sockets))) => {
*self
.underlay_sockets
.lock()
.unwrap_or_else(|p| p.into_inner()) = Some((name, sockets, true));
Some(RuleOutcome::Applied)
}
Ok(Err(reason)) | Err(reason) => Some(RuleOutcome::Failed(reason)),
}
}
}
impl ExitHostRules for LinuxHostRules {
fn refresh<'a>(&'a self, plan: &'a ExitHostPlan) -> BoxFuture<'a, Option<RuleOutcome>> {
Box::pin(async move {
if plan.client {
self.refresh_sockets(&plan.interface).await
} else {
None
}
})
}
fn apply<'a>(&'a self, plan: &'a ExitHostPlan) -> BoxFuture<'a, ExitHostReport> {
Box::pin(async move {
let interface = plan.interface.clone();
@@ -467,7 +618,9 @@ impl ExitHostRules for LinuxHostRules {
let client = if plan.client {
Some(self.apply_client(&interface).await)
} else {
self.clear_client().await;
if let Err(reason) = self.clear_client(&interface).await {
tracing::warn!(%reason, "cannot remove exit-client rules");
}
None
};
ExitHostReport {
@@ -482,7 +635,9 @@ impl ExitHostRules for LinuxHostRules {
Box::pin(async move {
let name = interface.to_string();
let _ = Self::blocking(move || clear_offer(&run_iptables, &name)).await;
self.clear_client().await;
if let Err(reason) = self.clear_client(interface).await {
tracing::warn!(%reason, "cannot remove exit-client rules");
}
})
}
}
@@ -0,0 +1,439 @@
//! Privileged routing regression, isolated from the host in named namespaces.
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use std::io::{BufRead, BufReader, Read, Write};
use std::net::{SocketAddr, TcpListener, TcpStream, UdpSocket};
use std::os::unix::process::CommandExt;
use std::process::{Child, Command};
use std::time::Duration;
use super::linux::run_firewall;
use super::linux_exit_mark::{clear, install, local_sockets, refresh};
use crate::overlay::NetlinkProvisioner;
const TEST: &str = "overlay::hostrules::linux_exit_kernel::strict_firewall_preserves_underlay_and_application_exit_routing";
const ROLE: &str = "TSUNAGI_EXIT_KERNEL_TEST_ROLE";
fn ip(arguments: &[&str]) -> String {
let result = Command::new("ip").args(arguments).output().unwrap();
assert!(
result.status.success(),
"ip {arguments:?}: {}",
String::from_utf8_lossy(&result.stderr)
);
String::from_utf8_lossy(&result.stdout).into_owned()
}
struct Namespace(String);
impl Drop for Namespace {
fn drop(&mut self) {
let _ = Command::new("ip").args(["netns", "del", &self.0]).status();
}
}
struct Echo(Child);
impl Drop for Echo {
fn drop(&mut self) {
let _ = self.0.kill();
let _ = self.0.wait();
}
}
fn probes() {
let should_pass = std::env::var("TSUNAGI_EXIT_KERNEL_EXPECT").unwrap() == "pass";
for address in ["198.51.100.2:42000", "[2001:db8:2::2]:42000"] {
let address: SocketAddr = address.parse().unwrap();
let socket = UdpSocket::bind(if address.is_ipv4() {
"0.0.0.0:0"
} else {
"[::]:0"
})
.unwrap();
socket
.set_read_timeout(Some(Duration::from_secs(1)))
.unwrap();
socket.send_to(b"rpf", address).unwrap();
let mut packet = [0u8; 3];
assert_eq!(
socket.recv(&mut packet).is_ok(),
should_pass,
"UDP {address}"
);
if should_pass {
assert_eq!(&packet, b"rpf");
}
let connected = TcpStream::connect_timeout(&address, Duration::from_secs(1));
assert_eq!(connected.is_ok(), should_pass, "TCP {address}");
if let Ok(mut stream) = connected {
stream
.set_read_timeout(Some(Duration::from_secs(1)))
.unwrap();
stream.write_all(b"rpf").unwrap();
stream.read_exact(&mut packet).unwrap();
assert_eq!(&packet, b"rpf");
}
}
}
fn run_probes(expect: &str) {
let result = Command::new(std::env::current_exe().unwrap())
.args(["--ignored", "--exact", TEST, "--nocapture"])
.env(ROLE, "probe")
.env("TSUNAGI_EXIT_KERNEL_EXPECT", expect)
.uid(990)
.gid(990)
.output()
.unwrap();
assert!(
result.status.success(),
"{}\n{}",
String::from_utf8_lossy(&result.stdout),
String::from_utf8_lossy(&result.stderr)
);
}
fn passive_listener() {
let listeners: Vec<_> = ["192.0.2.1:42010", "[2001:db8:1::1]:42010"]
.into_iter()
.map(|address| TcpListener::bind(address).unwrap())
.collect();
println!("PASSIVE_READY");
let tasks: Vec<_> = listeners
.into_iter()
.map(|listener| {
std::thread::spawn(move || {
let (mut socket, _) = listener.accept().unwrap();
socket
.set_read_timeout(Some(Duration::from_secs(2)))
.unwrap();
let mut bytes = [0; 3];
socket.read_exact(&mut bytes).unwrap();
socket.write_all(&bytes).unwrap();
})
})
.collect();
for task in tasks {
task.join().unwrap();
}
}
fn udp_listener() {
let sockets: Vec<_> = ["192.0.2.1:42011", "[2001:db8:1::1]:42011"]
.into_iter()
.map(|a| UdpSocket::bind(a).unwrap())
.collect();
println!("PASSIVE_READY");
let tasks: Vec<_> = sockets
.into_iter()
.map(|socket| {
std::thread::spawn(move || {
socket
.set_read_timeout(Some(Duration::from_secs(5)))
.unwrap();
let mut bytes = [0; 3];
let (len, peer) = socket.recv_from(&mut bytes).unwrap();
socket.send_to(&bytes[..len], peer).unwrap();
})
})
.collect();
for task in tasks {
task.join().unwrap();
}
}
fn spawn_listener(role: &str) -> Echo {
let process = Command::new(std::env::current_exe().unwrap())
.args(["--ignored", "--exact", TEST, "--nocapture"])
.env(ROLE, role)
.uid(990)
.gid(990)
.stdout(std::process::Stdio::piped())
.spawn()
.unwrap();
let mut child = Echo(process);
let mut output = BufReader::new(child.0.stdout.take().unwrap());
loop {
let mut line = String::new();
assert_ne!(
output.read_line(&mut line).unwrap(),
0,
"listener exited before readiness"
);
if line.contains("PASSIVE_READY") {
// Keep the reader open for libtest's final status after the socket
// worker finishes; closing it here makes success a broken pipe.
child.0.stdout = Some(output.into_inner());
return child;
}
}
}
fn client() {
for setting in ["rp_filter", "src_valid_mark"] {
std::fs::write(format!("/proc/sys/net/ipv4/conf/all/{setting}"), b"1").unwrap();
}
std::fs::write("/proc/sys/net/ipv4/conf/rpf-c/rp_filter", b"1").unwrap();
for family in [false, true] {
let args: Vec<_> = [
"-t",
"mangle",
"-A",
"PREROUTING",
"-m",
"rpfilter",
"--invert",
"--validmark",
"-j",
"DROP",
]
.into_iter()
.map(str::to_string)
.collect();
let result = run_firewall(family, &args);
assert!(
matches!(result, super::linux::Ran::Exited { success: true, .. }),
"{result:?}"
);
}
let provisioner = NetlinkProvisioner::new().unwrap();
let route = provisioner.host_rules().route;
// Model Tailscale's earlier lookup, without running another VPN daemon.
ip(&["-4", "rule", "add", "priority", "5270", "table", "52"]);
ip(&[
"-4", "route", "add", "default", "dev", "tsun0", "table", "52",
]);
ip(&["-6", "rule", "add", "priority", "5270", "table", "52"]);
ip(&[
"-6",
"route",
"add",
"unreachable",
"default",
"table",
"52",
]);
// A foreign VPN must leave neighbour discovery and connected LAN paths
// usable; otherwise the fixture itself cannot resolve its uplink neighbour.
for prefix in ["fe80::/64", "2001:db8:1::/64"] {
ip(&["-6", "route", "add", "throw", prefix, "table", "52"]);
}
let mut passive = spawn_listener("listener");
route.set_exit_client("tsun0", 990).unwrap();
// uid bypass alone cannot protect reverse lookups, reproducing the bug.
run_probes("blocked");
install(&run_firewall, "tsun0", 990, true).unwrap();
run_probes("pass");
// Pair transports may bind fresh UDP ports after exit activation.
let mut dynamic_udp = spawn_listener("udp-listener");
refresh(
&run_firewall,
"tsun0",
true,
&local_sockets(990, true).unwrap(),
)
.unwrap();
let server = std::env::var("TSUNAGI_EXIT_KERNEL_SERVER").unwrap();
let incoming = Command::new("ip").args(["netns", "exec", &server, "python3", "-c", r#"import socket
for family,source,target in [(socket.AF_INET,'198.51.100.2','192.0.2.1'),(socket.AF_INET6,'2001:db8:2::2','2001:db8:1::1')]:
for kind,port in [(socket.SOCK_STREAM,42010),(socket.SOCK_DGRAM,42011)]:
s=socket.socket(family,kind); s.settimeout(2); s.bind((source,0)); s.connect((target,port)); s.sendall(b'rpf'); assert s.recv(3)==b'rpf'; s.close()
"#]).output().unwrap();
assert!(
incoming.status.success(),
"first inbound TCP/UDP must be protected: {}",
String::from_utf8_lossy(&incoming.stderr)
);
assert!(passive.0.wait().unwrap().success());
assert!(dynamic_udp.0.wait().unwrap().success());
// Repeated apply remains usable, and does not redirect application sockets.
install(&run_firewall, "tsun0", 990, true).unwrap();
run_probes("pass");
assert!(ip(&["-4", "route", "get", "198.51.100.2", "uid", "991"]).contains("dev tsun0"));
let blocked = Command::new("ip")
.args(["-6", "route", "get", "2001:db8:2::2", "uid", "991"])
.output()
.unwrap();
assert!(
!blocked.status.success(),
"application IPv6 must stay blocked"
);
route.clear_exit_client().unwrap();
clear(&run_firewall, "tsun0", true).unwrap();
for family in ["-4", "-6"] {
assert!(
ip(&[family, "rule", "show"]).contains("5270:"),
"foreign routing rules survive cleanup"
);
ip(&[family, "rule", "del", "priority", "5270"]);
}
assert!(ip(&["-4", "route", "get", "198.51.100.2", "uid", "991"]).contains("dev rpf-c"));
assert!(ip(&["-6", "route", "get", "2001:db8:2::2", "uid", "991"]).contains("dev rpf-c"));
for family in [false, true] {
let result = run_firewall(family, &["-t".into(), "mangle".into(), "-S".into()]);
if let super::linux::Ran::Exited { stdout, .. } = result {
assert!(!stdout.contains("tsunagi-exit-underlay:"));
assert!(!stdout.contains("tsunagi-exit-sockets:"));
assert!(
stdout.contains("rpfilter"),
"strict host filtering is retained"
);
}
}
}
#[test]
#[ignore = "needs root, iproute2, iptables/ip6tables and Python; changes only temporary namespaces"]
fn strict_firewall_preserves_underlay_and_application_exit_routing() {
match std::env::var(ROLE).as_deref() {
Ok("probe") => {
probes();
return;
}
Ok("client") => {
client();
return;
}
Ok("listener") => {
passive_listener();
return;
}
Ok("udp-listener") => {
udp_listener();
return;
}
_ => {}
}
let suffix = std::process::id();
let client_name = format!("tsng-rpf-c-{suffix}");
let server_name = format!("tsng-rpf-s-{suffix}");
ip(&["netns", "add", &client_name]);
let _client = Namespace(client_name.clone());
ip(&["netns", "add", &server_name]);
let _server = Namespace(server_name.clone());
ip(&[
"link",
"add",
"rpf-c",
"netns",
&client_name,
"type",
"veth",
"peer",
"name",
"rpf-s",
"netns",
&server_name,
]);
for (namespace, interface, v4, v6) in [
(&client_name, "rpf-c", "192.0.2.1/24", "2001:db8:1::1/64"),
(&server_name, "rpf-s", "192.0.2.2/24", "2001:db8:1::2/64"),
] {
ip(&["-n", namespace, "link", "set", "lo", "up"]);
ip(&["-n", namespace, "addr", "add", v4, "dev", interface]);
ip(&[
"-n", namespace, "-6", "addr", "add", v6, "dev", interface, "nodad",
]);
ip(&["-n", namespace, "link", "set", interface, "up"]);
}
ip(&[
"-n",
&server_name,
"addr",
"add",
"198.51.100.2/32",
"dev",
"lo",
]);
ip(&[
"-n",
&server_name,
"-6",
"addr",
"add",
"2001:db8:2::2/128",
"dev",
"lo",
"nodad",
]);
ip(&[
"-n",
&client_name,
"route",
"add",
"default",
"via",
"192.0.2.2",
]);
ip(&[
"-n",
&client_name,
"-6",
"route",
"add",
"default",
"via",
"2001:db8:1::2",
]);
ip(&["-n", &client_name, "link", "add", "tsun0", "type", "dummy"]);
ip(&[
"-n",
&client_name,
"addr",
"add",
"10.13.37.1/24",
"dev",
"tsun0",
]);
ip(&["-n", &client_name, "link", "set", "tsun0", "up"]);
let echo_code = r#"import socket,selectors,sys
selector=selectors.DefaultSelector()
for family in [socket.AF_INET,socket.AF_INET6]:
for kind in [socket.SOCK_DGRAM,socket.SOCK_STREAM]:
sock=socket.socket(family,kind)
if family==socket.AF_INET6: sock.setsockopt(socket.IPPROTO_IPV6,socket.IPV6_V6ONLY,1)
sock.bind(('198.51.100.2' if family==socket.AF_INET else '2001:db8:2::2',42000))
if kind==socket.SOCK_STREAM: sock.listen()
selector.register(sock,selectors.EVENT_READ,kind)
print('ready',flush=True)
while True:
for key,_ in selector.select():
if key.data==socket.SOCK_DGRAM:
data,address=key.fileobj.recvfrom(128); key.fileobj.sendto(data,address)
else:
conn,_=key.fileobj.accept(); conn.settimeout(2)
try: conn.sendall(conn.recv(128))
finally: conn.close()
"#;
let child = Command::new("ip")
.args([
"netns",
"exec",
&server_name,
"python3",
"-u",
"-c",
echo_code,
])
.stdout(std::process::Stdio::piped())
.spawn()
.unwrap();
let mut echo = Echo(child);
let mut ready = String::new();
BufReader::new(echo.0.stdout.take().unwrap())
.read_line(&mut ready)
.unwrap();
assert_eq!(ready.trim(), "ready");
let result = Command::new("ip")
.args(["netns", "exec", &client_name])
.arg(std::env::current_exe().unwrap())
.args(["--ignored", "--exact", TEST, "--nocapture"])
.env(ROLE, "client")
.env("TSUNAGI_EXIT_KERNEL_SERVER", &server_name)
.output()
.unwrap();
assert!(
result.status.success(),
"{}\n{}",
String::from_utf8_lossy(&result.stdout),
String::from_utf8_lossy(&result.stderr)
);
}
@@ -0,0 +1,535 @@
//! Own-bit connection marks for exit-client underlay traffic.
//! Reply marks are restored before host reverse-path filtering, never accepted
//! around it. No peer address becomes a host route or firewall argument.
use super::linux::{Ran, failure, locked, split_rule_line};
use std::net::{IpAddr, Ipv4Addr, Ipv6Addr};
/// Outside Tailscale's 0xff0000 mask and ordinary low-bit WireGuard marks.
pub(crate) const UNDERLAY_MARK: u32 = 0x4000_0000;
pub(crate) const UNDERLAY_MASK: u32 = UNDERLAY_MARK;
const MARK: &str = "0x40000000/0x40000000";
const MASK: &str = "0x40000000";
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord)]
pub(super) struct Listener {
address: IpAddr,
port: u16,
tcp: bool,
}
fn parse_listeners(listing: &str, uid: u32, v6: bool, tcp: bool) -> Result<Vec<Listener>, String> {
let mut result = Vec::new();
for line in listing.lines().skip(1) {
let fields: Vec<_> = line.split_whitespace().collect();
if (tcp && fields.get(3) != Some(&"0A"))
|| fields.get(7).and_then(|s| s.parse::<u32>().ok()) != Some(uid)
{
continue;
}
let (address, port) = fields
.get(1)
.and_then(|s| s.split_once(':'))
.ok_or("malformed local socket")?;
let port = u16::from_str_radix(port, 16).map_err(|e| e.to_string())?;
let address = if v6 {
let mut bytes = [0; 16];
for (word, chunk) in bytes.as_chunks_mut::<4>().0.iter_mut().enumerate() {
let hex = address
.get(word * 8..word * 8 + 8)
.ok_or("malformed local IPv6 listener")?;
chunk.copy_from_slice(
&u32::from_str_radix(hex, 16)
.map_err(|e| e.to_string())?
.to_ne_bytes(),
);
}
IpAddr::V6(Ipv6Addr::from(bytes))
} else {
IpAddr::V4(Ipv4Addr::from(
u32::from_str_radix(address, 16)
.map_err(|e| e.to_string())?
.to_ne_bytes(),
))
};
if result.len() == 128 {
return Err("too many local sockets for exit protection".into());
}
if port != 0 {
result.push(Listener { address, port, tcp });
}
}
Ok(result)
}
pub(super) fn local_sockets(uid: u32, v6: bool) -> Result<Vec<Listener>, String> {
let mut listeners = Vec::new();
for (protocol, tcp) in [("tcp", true), ("udp", false)] {
for family in [false, true].into_iter().filter(|f| !f || v6) {
let path = format!("/proc/net/{protocol}{}", if family { "6" } else { "" });
listeners.extend(parse_listeners(
&std::fs::read_to_string(path).map_err(|e| e.to_string())?,
uid,
family,
tcp,
)?);
}
}
listeners.sort();
listeners.dedup();
Ok(listeners)
}
fn listener_rules(interface: &str, listener: &Listener) -> Vec<Vec<String>> {
let mut base = words(&[
"!",
"-i",
interface,
"-p",
if listener.tcp { "tcp" } else { "udp" },
"--dport",
&listener.port.to_string(),
]);
if !listener.address.is_unspecified() {
base.extend(words(&["-d", &listener.address.to_string()]));
}
// Match an actual local socket, not just a destination port. Listening
// sockets need protection on the first packet, before any outbound connmark.
base.extend(words(&[
"-m",
"socket",
"--nowildcard",
"-m",
"comment",
"--comment",
&listener_tag(interface),
]));
let save = base
.iter()
.cloned()
.chain(words(&[
"-m",
"mark",
"--mark",
MARK,
"-j",
"CONNMARK",
"--save-mark",
"--nfmask",
MASK,
"--ctmask",
MASK,
]))
.collect();
let mark = base
.into_iter()
.chain(words(&["-j", "MARK", "--set-xmark", MARK]))
.collect();
vec![save, mark]
}
pub(super) type RunFamily<'a> = &'a dyn Fn(bool, &[String]) -> Ran;
fn tag(interface: &str) -> String {
format!("tsunagi-exit-underlay:{interface}")
}
fn listener_tag(interface: &str) -> String {
format!("tsunagi-exit-sockets:{interface}")
}
fn words(values: &[&str]) -> Vec<String> {
values.iter().map(|s| (*s).into()).collect()
}
fn rules(interface: &str, uid: u32) -> Vec<(&'static str, Vec<String>)> {
let tag = tag(interface);
let uid = uid.to_string();
// Insert at position one in this order: final OUTPUT order is MARK, SAVE.
vec![
(
"PREROUTING",
words(&[
"!",
"-i",
interface,
"-m",
"connmark",
"--mark",
MARK,
"-m",
"comment",
"--comment",
&tag,
"-j",
"CONNMARK",
"--restore-mark",
"--nfmask",
MASK,
"--ctmask",
MASK,
]),
),
(
"OUTPUT",
words(&[
"-m",
"owner",
"--uid-owner",
&uid,
"-m",
"mark",
"--mark",
MARK,
"-m",
"comment",
"--comment",
&tag,
"-j",
"CONNMARK",
"--save-mark",
"--nfmask",
MASK,
"--ctmask",
MASK,
]),
),
(
"OUTPUT",
words(&[
"-m",
"owner",
"--uid-owner",
&uid,
"-m",
"comment",
"--comment",
&tag,
"-j",
"MARK",
"--set-xmark",
MARK,
]),
),
]
}
fn execute(run: RunFamily<'_>, v6: bool, args: Vec<String>) -> Result<String, String> {
match run(v6, &locked(args)) {
Ran::Exited {
success: true,
stdout,
..
} => Ok(stdout),
Ran::Exited { stderr, .. } => Err(failure(&stderr)),
Ran::Missing => Err(format!(
"{} is required for exit-node connection marks",
if v6 { "ip6tables" } else { "iptables" }
)),
}
}
/// Removes only exact interface tags, retaining other agents and host rules.
pub(super) fn clear(run: RunFamily<'_>, interface: &str, v6: bool) -> Result<(), String> {
clear_tags(run, interface, v6, false)
}
fn clear_tags(
run: RunFamily<'_>,
interface: &str,
v6: bool,
sockets_only: bool,
) -> Result<(), String> {
let wanted = tag(interface);
let sockets = listener_tag(interface);
for family in [false, true].into_iter().filter(|family| !family || v6) {
let chains: &[&str] = if sockets_only {
&["PREROUTING"]
} else {
&["OUTPUT", "PREROUTING"]
};
for chain in chains {
let listing = execute(run, family, words(&["-t", "mangle", "-S", chain]))?;
for line in listing.lines() {
let tokens = split_rule_line(line);
let [append, listed_chain, rest @ ..] = tokens.as_slice() else {
continue;
};
if append == "-A"
&& listed_chain == *chain
&& rest.windows(2).any(|pair| {
pair[0] == "--comment"
&& (pair[1] == sockets || (!sockets_only && pair[1] == wanted))
})
{
let args = words(&["-t", "mangle", "-D", chain])
.into_iter()
.chain(rest.iter().cloned())
.collect();
execute(run, family, args)?;
}
}
}
}
Ok(())
}
/// Establishes protection before routing may redirect application traffic.
pub(super) fn install(
run: RunFamily<'_>,
interface: &str,
uid: u32,
v6: bool,
) -> Result<Vec<Listener>, String> {
let listeners = local_sockets(uid, v6)?;
install_with_listeners(run, interface, uid, v6, &listeners)?;
Ok(listeners)
}
fn install_with_listeners(
run: RunFamily<'_>,
interface: &str,
uid: u32,
v6: bool,
listeners: &[Listener],
) -> Result<(), String> {
clear(run, interface, v6)?;
let result = (|| {
for family in [false, true].into_iter().filter(|family| !family || v6) {
for (chain, spec) in rules(interface, uid) {
let args = words(&["-t", "mangle", "-I", chain, "1"])
.into_iter()
.chain(spec)
.collect();
execute(run, family, args)?;
}
for listener in listeners.iter().filter(|s| s.address.is_ipv6() == family) {
for spec in listener_rules(interface, listener) {
execute(
run,
family,
words(&["-t", "mangle", "-I", "PREROUTING", "1"])
.into_iter()
.chain(spec)
.collect(),
)?;
}
}
}
Ok(())
})();
if result.is_err()
&& let Err(cleanup) = clear(run, interface, v6)
{
return Err(format!(
"{}; cannot roll back connection marks: {cleanup}",
result.err().unwrap_or_default()
));
}
result
}
/// Updates passive protection without interrupting established connmark rules.
pub(super) fn refresh(
run: RunFamily<'_>,
interface: &str,
v6: bool,
listeners: &[Listener],
) -> Result<(), String> {
clear_tags(run, interface, v6, true)?;
for family in [false, true].into_iter().filter(|f| !f || v6) {
for listener in listeners.iter().filter(|s| s.address.is_ipv6() == family) {
for spec in listener_rules(interface, listener) {
execute(
run,
family,
words(&["-t", "mangle", "-I", "PREROUTING", "1"])
.into_iter()
.chain(spec)
.collect(),
)?;
}
}
}
Ok(())
}
/// Kernel strict RPF must include the restored mark, independently of iptables.
pub(super) fn check_kernel_rpf(
all_rpf: u32,
all_valid: u32,
interfaces: &[(String, u32, u32)],
) -> Result<(), String> {
for (interface, rpf, valid) in interfaces {
if all_rpf.max(*rpf) == 1 && all_valid.max(*valid) == 0 {
return Err(format!(
"strict rp_filter on `{interface}` ignores connection marks; enable net.ipv4.conf.all.src_valid_mark=1 before using an exit node (rp_filter stays enabled)"
));
}
}
Ok(())
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
use std::cell::RefCell;
#[test]
fn both_families_restore_before_firewall_and_preserve_foreign_mark_bits() {
let calls = RefCell::new(Vec::new());
let run = |v6, args: &[String]| {
calls.borrow_mut().push((v6, args.to_vec()));
Ran::Exited {
success: true,
stdout: String::new(),
stderr: String::new(),
}
};
install_with_listeners(&run, "tsun0", 990, true, &[]).unwrap();
let calls = calls.borrow();
for family in [false, true] {
let inserts: Vec<_> = calls
.iter()
.filter(|(v6, a)| *v6 == family && a.iter().any(|s| s == "-I"))
.map(|(_, a)| a.join(" "))
.collect();
assert_eq!(inserts.len(), 3);
assert!(inserts[0].contains("-I PREROUTING 1 ! -i tsun0 -m connmark"));
assert!(inserts[0].contains("--restore-mark --nfmask 0x40000000 --ctmask 0x40000000"));
assert!(inserts[1].contains("--uid-owner 990"));
assert!(inserts[1].contains("--save-mark"));
assert!(inserts[2].contains("--set-xmark 0x40000000/0x40000000"));
assert!(
!inserts
.iter()
.any(|s| s.contains("-j ACCEPT") || s.contains("-j RETURN"))
);
}
assert_eq!(UNDERLAY_MASK & 0x00ff_0000, 0);
}
#[test]
fn cleanup_retains_other_interfaces_and_rolls_back_partial_ipv6_installation() {
let calls = RefCell::new(Vec::new());
let run = |v6, args: &[String]| {
calls.borrow_mut().push((v6, args.to_vec()));
let listing = "-A OUTPUT -m comment --comment tsunagi-exit-underlay:tsun0 -j MARK --set-xmark 0x40000000/0x40000000\n-A OUTPUT -m comment --comment tsunagi-exit-underlay:tsun1 -j MARK --set-xmark 0x40000000/0x40000000\n-A OUTPUT -m comment --comment unrelated -j RETURN\n";
Ran::Exited {
success: !(v6 && args.iter().any(|s| s == "-I")),
stdout: if args.iter().any(|s| s == "-S") {
listing.into()
} else {
String::new()
},
stderr: "injected IPv6 failure".into(),
}
};
assert!(install_with_listeners(&run, "tsun0", 990, true, &[]).is_err());
let calls = calls.borrow();
let deletions: Vec<_> = calls
.iter()
.filter(|(_, a)| a.iter().any(|s| s == "-D"))
.collect();
assert!(
deletions.len() >= 4,
"both initial and rollback cleanup run"
);
assert!(
deletions
.iter()
.all(|(_, a)| a.iter().any(|s| s == "tsunagi-exit-underlay:tsun0"))
);
}
#[test]
fn kernel_strict_validation_requires_mark_awareness_without_weakening_rpf() {
let interfaces = vec![("eth0".into(), 1, 0)];
assert!(check_kernel_rpf(0, 0, &interfaces).is_err());
assert!(check_kernel_rpf(0, 1, &interfaces).is_ok());
assert!(check_kernel_rpf(2, 0, &interfaces).is_ok());
assert!(check_kernel_rpf(0, 0, &[("eth0".into(), 1, 1)]).is_ok());
}
#[test]
fn passive_tcp_protection_uses_only_locally_owned_listeners() {
let listing = "header\n0: 00000000:01BB 00000000:0000 0A 0:0 0:0 0 990 0 1\n1: 0100007F:1111 00000000:0000 0A 0:0 0:0 0 1000 0 2\n2: 0100007F:2222 00000000:0000 01 0:0 0:0 0 990 0 3\n";
let listeners = parse_listeners(listing, 990, false, true).unwrap();
assert_eq!(
listeners,
vec![Listener {
address: IpAddr::V4(Ipv4Addr::UNSPECIFIED),
port: 443,
tcp: true
}]
);
let rules = listener_rules("tsun0", &listeners[0]);
assert!(
rules[0]
.join(" ")
.contains("-p tcp --dport 443 -m socket --nowildcard")
);
assert!(rules[0].join(" ").contains("--save-mark"));
assert!(rules[1].join(" ").contains("--set-xmark"));
}
#[test]
fn listener_parser_checks_family_encoding_and_bounds() {
let loopback = Ipv6Addr::LOCALHOST.octets();
let hex: String = loopback
.as_chunks::<4>()
.0
.iter()
.map(|bytes| format!("{:08X}", u32::from_ne_bytes(*bytes)))
.collect();
let listing = format!("header\n0: {hex}:01BB 0:0 0A 0:0 0:0 0 990 0 1\n");
assert_eq!(
parse_listeners(&listing, 990, true, true).unwrap(),
vec![Listener {
address: IpAddr::V6(Ipv6Addr::LOCALHOST),
port: 443,
tcp: true
}]
);
assert!(
parse_listeners(
"header\n0: BAD:01BB 0:0 0A 0:0 0:0 0 990 0 1",
990,
true,
true
)
.is_err()
);
let huge = format!(
"header\n{}",
"0: 00000000:01BB 0:0 0A 0:0 0:0 0 990 0 1\n".repeat(129)
);
assert!(parse_listeners(&huge, 990, false, true).is_err());
}
#[test]
fn udp_refresh_preserves_general_connmark_protection_and_other_interfaces() {
let listing = "header\n0: 00000000:ABA0 0:0 07 0:0 0:0 0 990 0 1\n1: 00000000:0000 0:0 07 0:0 0:0 0 990 0 2\n";
let sockets = parse_listeners(listing, 990, false, false).unwrap();
assert_eq!(sockets.len(), 1);
assert!(!sockets[0].tcp);
let calls = RefCell::new(Vec::new());
let run = |v6, args: &[String]| {
calls.borrow_mut().push((v6, args.join(" ")));
Ran::Exited { success: true, stdout: "-A PREROUTING -m comment --comment tsunagi-exit-underlay:tsun0 -j CONNMARK --restore-mark\n-A PREROUTING -m comment --comment tsunagi-exit-sockets:tsun0 -j MARK\n-A PREROUTING -m comment --comment tsunagi-exit-sockets:tsun1 -j MARK\n".into(), stderr: String::new() }
};
refresh(&run, "tsun0", false, &sockets).unwrap();
let calls = calls.borrow();
assert!(
calls
.iter()
.any(|(_, a)| a.contains("-I PREROUTING 1") && a.contains("-p udp"))
);
assert!(
calls
.iter()
.filter(|(_, a)| a.contains("-D"))
.all(|(_, a)| a.contains("tsunagi-exit-sockets:tsun0"))
);
}
}
@@ -55,6 +55,10 @@ pub use exit::{
mod linux;
#[cfg(all(feature = "tun-device", target_os = "linux"))]
mod linux_exit;
#[cfg(all(test, feature = "tun-device", target_os = "linux"))]
mod linux_exit_kernel;
#[cfg(all(feature = "tun-device", target_os = "linux"))]
pub(in crate::overlay) mod linux_exit_mark;
#[cfg(all(feature = "tun-device", target_os = "linux"))]
pub use linux::LinuxHostRules;
@@ -397,6 +397,14 @@ impl Interface {
return;
}
if planned.as_ref() == Some(&wanted) && ok {
if let Some(outcome) = exit.refresh(&wanted).await {
if let RuleOutcome::Failed(reason) = &outcome {
tracing::warn!(interface = %self.name, %reason, "cannot refresh exit-client socket protection");
}
if let Some(report) = self.exit_state().report.as_mut() {
report.client = Some(outcome);
}
}
return;
}
let report = exit.apply(&wanted).await;
@@ -55,6 +55,7 @@ use crate::BoxFuture;
use crate::overlay::OverlayError;
use super::super::config::Cidr;
use super::super::hostrules::linux_exit_mark::{UNDERLAY_MARK, UNDERLAY_MASK};
use super::super::tun::{TunDevice, TunRequest};
use super::privilege::{NetAdmin, Privilege, probe_net_admin};
use super::{
@@ -641,8 +642,10 @@ async fn del_broadcast_route(name: &str) -> Result<(), OverlayError> {
const EXIT_TABLE: u32 = 0x7473;
/// The main table.
const MAIN_TABLE: u32 = 254;
/// Where the exit rules sit, after Tailscale's 5270.
const EXIT_BYPASS_PRIORITY: u32 = 5280;
/// Protect marked underlay and initial socket routing before Tailscale's rules.
const EXIT_MARK_PRIORITY: u32 = 5205;
const EXIT_BYPASS_PRIORITY: u32 = 5206;
const LEGACY_EXIT_BYPASS_PRIORITY: u32 = 5280;
const EXIT_SPECIFIC_PRIORITY: u32 = 5290;
const EXIT_DEFAULT_PRIORITY: u32 = 5300;
@@ -666,14 +669,23 @@ fn rule_priority(rule: &RuleMessage) -> Option<u32> {
})
}
/// Whether a rule is one of the three this agent adds for an exit node.
/// Whether a rule is one this agent adds for an exit node, including old UID priority.
///
/// Recognised by shape as well as priority: those numbers are not reserved,
/// and a rule somebody else put at one of them must not be removed.
fn is_exit_rule(rule: &RuleMessage) -> bool {
let table = rule_table(rule);
match rule_priority(rule) {
Some(EXIT_BYPASS_PRIORITY) => {
Some(EXIT_MARK_PRIORITY) => {
table == MAIN_TABLE
&& rule
.attributes
.contains(&RuleAttribute::FwMark(UNDERLAY_MARK))
&& rule
.attributes
.contains(&RuleAttribute::FwMask(UNDERLAY_MASK))
}
Some(EXIT_BYPASS_PRIORITY | LEGACY_EXIT_BYPASS_PRIORITY) => {
table == MAIN_TABLE
&& rule
.attributes
@@ -789,7 +801,8 @@ async fn remove_exit_client(handle: &rtnetlink::Handle) -> Result<(), OverlayErr
///
/// ```text
/// ip route replace default dev <if> table 0x7473
/// ip rule add priority 5280 uidrange <uid>-<uid> lookup main
/// ip rule add priority 5205 fwmark 0x40000000/0x40000000 lookup main
/// ip rule add priority 5206 uidrange <uid>-<uid> lookup main
/// ip rule add priority 5290 lookup main suppress_prefixlength 0
/// ip rule add priority 5300 lookup 0x7473
/// ```
@@ -819,6 +832,8 @@ async fn set_exit_client(name: &str, uid: u32) -> Result<(), OverlayError> {
.await
.map_err(|err| fail("default route", err))?;
add_mark_bypass(&handle, false).await?;
let mut bypass = handle
.rule()
.add()
@@ -891,6 +906,7 @@ async fn set_exit_block_v6(handle: &rtnetlink::Handle, uid: u32) -> Result<(), O
};
}
add_mark_bypass(handle, true).await?;
let mut bypass = handle
.rule()
.add()
@@ -938,6 +954,27 @@ async fn set_exit_block_v6(handle: &rtnetlink::Handle, uid: u32) -> Result<(), O
.map_err(|err| fail("rule for the default route", err))
}
async fn add_mark_bypass(handle: &rtnetlink::Handle, v6: bool) -> Result<(), OverlayError> {
let mut rule = handle
.rule()
.add()
.table_id(MAIN_TABLE)
.action(RuleAction::ToTable)
.priority(EXIT_MARK_PRIORITY)
.fw_mark(UNDERLAY_MARK);
rule.message_mut().header.family = if v6 {
rtnetlink::packet_route::AddressFamily::Inet6
} else {
rtnetlink::packet_route::AddressFamily::Inet
};
rule.message_mut()
.attributes
.push(RuleAttribute::FwMask(UNDERLAY_MASK));
rule.execute().await.map_err(|err| {
OverlayError::Unavailable(format!("cannot protect marked underlay traffic: {err}"))
})
}
async fn configure_link(configure: &Configure) -> Result<(), OverlayError> {
let name = configure.name.as_str();
with_netlink(|handle| async move {
@@ -1110,6 +1147,28 @@ mod tests {
assert!(!is_exit_rule(&rule(5280, 100, vec![uid])));
}
#[test]
fn underlay_bypass_recognises_only_our_exact_bit_and_mask() {
let rule = |table, mark, mask| {
let mut message = RuleMessage::default();
message.attributes.extend([
RuleAttribute::Table(table),
RuleAttribute::Priority(EXIT_MARK_PRIORITY),
RuleAttribute::FwMark(mark),
RuleAttribute::FwMask(mask),
]);
message
};
assert!(is_exit_rule(&rule(
MAIN_TABLE,
UNDERLAY_MARK,
UNDERLAY_MASK
)));
assert!(!is_exit_rule(&rule(MAIN_TABLE, 0x80000, 0xff0000)));
assert!(!is_exit_rule(&rule(MAIN_TABLE, UNDERLAY_MARK, u32::MAX)));
assert!(!is_exit_rule(&rule(52, UNDERLAY_MARK, UNDERLAY_MASK)));
}
#[test]
fn the_ipv6_default_of_the_exit_table_refuses_instead_of_routing() {
let route = exit_blocking_route_v6();
+39 -2
View File
@@ -72,7 +72,7 @@ pub enum RouteError {
#[error(
"network {network} uses {range}, which overlaps {other_range} already in use by \
{other}. One agent has one interface, so an address can belong to only one of \
them: give one network a different range."
them: stop the other network before starting this one."
)]
Overlap {
/// The network being added.
@@ -365,6 +365,43 @@ impl RoutingTable {
})
}
/// All active networks whose reserved ranges intersect this range.
pub fn overlapping_networks(
&self,
network: NetworkId,
range: Ipv4Range,
) -> Vec<(NetworkId, Ipv4Range)> {
let mut conflicts: Vec<_> = self
.read()
.iter()
.filter_map(|(other, routes)| {
let other_range = routes.range?;
(*other != network && ranges_overlap(range, other_range))
.then_some((*other, other_range))
})
.collect();
conflicts.sort_by_key(|(id, _)| *id);
conflicts
}
/// Current signed allocations used for encrypted data delivery checks.
pub(crate) fn probe_addresses(
&self,
network: NetworkId,
peer: EndpointId,
) -> Option<(Ipv4Addr, Ipv4Addr)> {
let table = self.read();
let routes = table.get(&network)?;
Some((
routes.local?,
routes
.peers
.iter()
.find(|(_, holder)| *holder == peer)
.map(|(address, _)| *address)?,
))
}
/// How many networks the table covers.
pub fn len(&self) -> usize {
self.read().len()
@@ -510,7 +547,7 @@ mod tests {
let err = table.set_network(second, routes()).unwrap_err();
let RouteError::Overlap { other, .. } = err;
assert_eq!(other, first);
assert!(err.to_string().contains("different range"), "{err}");
assert!(err.to_string().contains("stop the other network"), "{err}");
assert_eq!(table.len(), 1, "the overlapping network is not added");
// A range of its own is fine.
+51 -2
View File
@@ -20,7 +20,7 @@ use crate::error::ProtocolError;
/// The version in the ALPN is the wire-compatibility version of the control
/// protocol. It is independent of the network identity scheme version, so
/// bumping it must not change any existing [`crate::NetworkId`].
pub const ALPN: &[u8] = b"tsunagi/ctrl/5";
pub const ALPN: &[u8] = b"tsunagi/ctrl/7";
/// ALPN of the tsunagi data plane.
///
@@ -43,7 +43,9 @@ pub const MAX_DATA_PROTOCOL_LEN: usize = 32;
pub const MAX_SIGNATURE_LEN: usize = 64;
/// Control protocol version carried inside the handshake.
pub const PROTOCOL_VERSION: u16 = 5;
pub const PROTOCOL_VERSION: u16 = 7;
pub use crate::dataplane::transport::{MAX_TRANSPORT_PAYLOAD, TransportPayload};
/// First message of the handshake, sent by the initiator.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
@@ -114,6 +116,13 @@ pub struct DataOpenAck {
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[non_exhaustive]
pub enum ControlMessage {
/// Private transport negotiation; never persisted or rebroadcast.
Transport {
/// Transport protocol id; the payload stays opaque to the core.
protocol: String,
/// Secret negotiation data, redacted in diagnostics.
payload: TransportPayload,
},
/// Hostname and capability announcement.
Announce(Announcement),
/// A small request used to verify that the exchange works.
@@ -172,6 +181,11 @@ pub enum ControlMessage {
/// Short free-text reason.
reason: String,
},
/// Signed cleanup markers, each proving the retired claim version.
Retirements {
/// Bounded by the per-message signed-state limit.
markers: Vec<crate::state::SignedRetirement>,
},
}
/// Somewhere a member has been seen.
@@ -264,6 +278,39 @@ pub fn validate_capability(
/// network, the other networks or the agent.
pub fn validate(message: &ControlMessage, limits: &Limits) -> Result<(), ProtocolError> {
match message {
ControlMessage::Retirements { markers } => {
check_len("retirements", markers.len(), limits.max_state_records)?;
for marker in markers {
check_len(
"retirement.signature",
marker.signature.len(),
MAX_SIGNATURE_LEN,
)?;
check_len(
"retirement.target.signature",
marker.target.signature.len(),
MAX_SIGNATURE_LEN,
)?;
if let Some(name) = marker.target.body.hostname() {
check_len(
"retirement.target.hostname",
name.len(),
limits.max_hostname_len,
)?;
}
}
}
ControlMessage::Transport { protocol, payload } => {
if protocol.is_empty() {
return Err(ProtocolError::Malformed("empty transport protocol"));
}
check_len("transport.protocol", protocol.len(), MAX_PROTOCOL_ID_LEN)?;
check_len(
"transport.payload",
payload.as_bytes().len(),
limits.max_transport_payload_len.min(MAX_TRANSPORT_PAYLOAD),
)?;
}
ControlMessage::Peers { peers } => {
check_len("peers", peers.len(), limits.max_state_records)?;
for hint in peers {
@@ -320,10 +367,12 @@ pub fn validate(message: &ControlMessage, limits: &Limits) -> Result<(), Protoco
/// A short, stable label for a message kind, for metrics and diagnostics.
pub fn kind(message: &ControlMessage) -> &'static str {
match message {
ControlMessage::Transport { .. } => "transport",
ControlMessage::Announce(_) => "announce",
ControlMessage::Ping { .. } => "ping",
ControlMessage::Pong { .. } => "pong",
ControlMessage::State { .. } => "state",
ControlMessage::Retirements { .. } => "retirements",
ControlMessage::Reachable { .. } => "reachable",
ControlMessage::Peers { .. } => "peers",
ControlMessage::Bye { .. } => "bye",
+134 -26
View File
@@ -28,16 +28,23 @@
//! replica that has the record needs nothing else to interpret it, and
//! recovery never depends on replaying a chain from the beginning.
//!
//! Any member may also sign a separate retirement of another author's proved
//! claim version. It suppresses stale reservations without changing that
//! author's record, and a newer claim remains valid.
//!
//! # What this slice does not do yet
//!
//! Compaction, revocation of a whole author, and snapshots covering more than
//! one record type. See `docs/sync-model.md` for the shape those take.
//! Compaction and permanent revocation of a whole author. See
//! `docs/sync-model.md` for the implemented cleanup semantics.
pub mod allocator;
use std::collections::HashMap;
use std::net::Ipv4Addr;
mod retirement;
pub use retirement::{MAX_RETIREMENT_LEN, SignedRetirement};
use iroh::{EndpointId, SecretKey, Signature};
use serde::{Deserialize, Serialize};
@@ -109,12 +116,8 @@ impl std::str::FromStr for Ipv4Range {
}
}
/// The IPv4 overlay range used unless something else is configured or agreed.
///
/// A small, specific `/24`: memorable, and far less likely to overlap a
/// network the machine is already on than taking a whole `/8` or `/10` would
/// be. Because addresses are allocated rather than derived, 254 of them is
/// plenty for the size of network this is for.
/// The legacy default range, retained for existing networks and explicit use.
/// New networks use [`derived_ipv4_range`] by default.
pub const DEFAULT_IPV4_RANGE: Ipv4Range = Ipv4Range {
base: Ipv4Addr::new(10, 13, 37, 0),
prefix_len: 24,
@@ -122,15 +125,10 @@ pub const DEFAULT_IPV4_RANGE: Ipv4Range = Ipv4Range {
/// A range derived from a network's own identity.
///
/// One agent has one interface, so two of its networks cannot both use the
/// configured range. The second needs another one — and it cannot be picked
/// locally, because every member has to arrive at the same answer without
/// being told. So it comes from the network id, which every member already
/// has and nobody can choose: the same network derives the same range on
/// every device.
///
/// Inside 10/8 like the default, and never equal to it, so the first
/// network keeps what it has always had.
/// Every member independently derives the same pseudorandom `/24` in 10/8,
/// including isolated first members. Stored signed ranges take precedence.
/// The derivation is kept stable for networks already using it; the legacy
/// default is excluded. Distinct network ids can still produce collisions.
pub fn derived_ipv4_range(network: NetworkId) -> Ipv4Range {
let bytes = network.as_bytes();
let second = bytes[0];
@@ -497,6 +495,7 @@ pub enum Merged {
#[derive(Debug, Clone, Default)]
pub struct StateSet {
records: HashMap<EndpointId, SignedRecord>,
retirements: HashMap<EndpointId, SignedRetirement>,
}
impl StateSet {
@@ -509,6 +508,7 @@ impl StateSet {
pub fn from_verified(records: impl IntoIterator<Item = (EndpointId, SignedRecord)>) -> Self {
Self {
records: records.into_iter().collect(),
retirements: HashMap::new(),
}
}
@@ -537,6 +537,52 @@ impl StateSet {
.collect()
}
/// Visible records, excluding versions retired by authenticated members.
pub fn active_records(&self) -> Vec<SignedRecord> {
self.records()
.into_iter()
.filter(|r| r.author_id().is_ok_and(|id| !self.is_retired(&id)))
.collect()
}
/// Whether the current known version is at or below a proved retirement.
pub fn is_retired(&self, author: &EndpointId) -> bool {
self.retirements
.get(author)
.is_some_and(|m| self.version_of(author) <= m.target.version)
}
/// Strongest retained cleanup proof for an identity.
pub fn retirement_of(&self, target: &EndpointId) -> Option<&SignedRetirement> {
self.retirements.get(target)
}
/// Durable markers in target-id order, including markers surpassed by a fresh claim.
pub fn retirements(&self) -> Vec<SignedRetirement> {
let mut markers: Vec<_> = self.retirements.values().cloned().collect();
markers.sort_by_key(|m| m.target.author);
markers
}
/// Accepts any member's signed cleanup after checking both signatures.
pub fn merge_retirement(
&mut self,
network: NetworkId,
marker: SignedRetirement,
max: usize,
) -> Result<bool, StateError> {
let target = marker.verify(network)?;
if let Some(old) = self.retirements.get(&target) {
if !marker.supersedes(old) {
return Ok(false);
}
} else if self.retirements.len() >= max {
return Err(StateError::Malformed("retirement capacity reached"));
}
self.retirements.insert(target, marker);
Ok(true)
}
/// Merges one record, verifying it first.
///
/// Merging is into the existing set, never a wholesale replacement, and an
@@ -600,6 +646,9 @@ impl StateSet {
pub fn address_holders(&self) -> HashMap<Ipv4Addr, EndpointId> {
let mut holders: HashMap<Ipv4Addr, EndpointId> = HashMap::new();
for (author, record) in &self.records {
if self.is_retired(author) {
continue;
}
let Some(address) = record.body.claimed_address() else {
continue;
};
@@ -627,6 +676,9 @@ impl StateSet {
pub fn hostname_holders(&self) -> HashMap<String, EndpointId> {
let mut holders: HashMap<String, (u64, EndpointId)> = HashMap::new();
for (author, record) in &self.records {
if self.is_retired(author) {
continue;
}
let Some(hostname) = record.body.hostname() else {
continue;
};
@@ -655,6 +707,9 @@ impl StateSet {
/// The name an author claimed and the member that holds it instead, when
/// somebody who claimed it earlier does.
pub fn hostname_taken(&self, author: &EndpointId) -> Option<(&str, EndpointId)> {
if self.is_retired(author) {
return None;
}
let hostname = self.records.get(author)?.body.hostname()?;
let holder = *self.hostname_holders().get(hostname)?;
(holder != *author).then_some((hostname, holder))
@@ -673,21 +728,44 @@ impl StateSet {
/// The range the network settled on, if anybody has said.
///
/// When claims disagree, the one from the lowest author id wins, so every
/// replica reaches the same answer. A participant joining later therefore
/// adopts the range already in use instead of imposing its own.
/// A legacy or explicit range takes precedence over the derived default,
/// so a fresh member's default cannot renumber an established network.
/// Other disagreements use the lowest author id, independently of arrival
/// order. The signed record format and network identity remain unchanged.
pub fn agreed_range(&self) -> Option<Ipv4Range> {
let mut authors: Vec<&EndpointId> = self.records.keys().collect();
authors.sort_by_key(|author| *author.as_bytes());
authors
.into_iter()
.find_map(|author| self.records.get(author)?.body.range())
self.records
.iter()
.chain(
self.retirements
.iter()
.filter(|(author, _)| !self.records.contains_key(*author))
.map(|(author, marker)| (author, &marker.target)),
)
.filter_map(|(author, record)| {
let range = record.body.range()?;
let derived = derived_ipv4_range(NetworkId::from_bytes(record.network));
Some(((range == derived, *author.as_bytes()), range))
})
.min_by_key(|(priority, _)| *priority)
.map(|(_, range)| range)
}
/// The highest version this author has published, as far as is known.
pub fn version_of(&self, author: &EndpointId) -> u64 {
self.records.get(author).map_or(0, |record| record.version)
}
/// Highest witnessed author counter across claims and authored cleanup.
/// This differs from the claim version used to decide retirement.
pub fn authored_version(&self, author: &EndpointId) -> u64 {
self.retirements
.values()
.filter(|m| m.author == *author.as_bytes())
.map(|m| m.version)
.chain(std::iter::once(self.version_of(author)))
.max()
.unwrap_or(0)
}
}
#[cfg(test)]
@@ -1263,7 +1341,7 @@ mod tests {
assert_eq!(once.base.octets()[0], 10, "inside 10/8 like the default");
assert_ne!(
once, DEFAULT_IPV4_RANGE,
"the default belongs to whichever network asked first"
"the legacy default remains excluded"
);
// A different network derives a different range, which is the
@@ -1275,4 +1353,34 @@ mod tests {
.network_id();
assert_ne!(once, derived_ipv4_range(other));
}
#[test]
fn a_derived_default_cannot_displace_a_legacy_or_explicit_range() {
let id = network("range-migration");
let a = SecretKey::generate();
let b = SecretKey::generate();
let (low, high) = if a.public().as_bytes() < b.public().as_bytes() {
(a, b)
} else {
(b, a)
};
for established in [DEFAULT_IPV4_RANGE, "10.90.0.0/16".parse().unwrap()] {
let body = |range: Ipv4Range| RecordBody::Claim {
address: Some(Ipv4Addr::from(u32::from(range.base) + 1)),
range: Some(range),
hostname: None,
hostname_since: None,
};
let fresh = SignedRecord::sign(&low, id, 1, body(derived_ipv4_range(id)));
let existing = SignedRecord::sign(&high, id, 1, body(established));
for records in [
vec![fresh.clone(), existing.clone()],
vec![existing.clone(), fresh.clone()],
] {
let mut state = StateSet::new();
state.merge_all(id, records);
assert_eq!(state.agreed_range(), Some(established));
}
}
}
}
+219
View File
@@ -0,0 +1,219 @@
//! A member may retire an observed claim, never another identity's future.
use iroh::{EndpointId, SecretKey, Signature};
use serde::{Deserialize, Serialize};
use super::{RecordBody, SignedRecord, StateError, push_lp};
use crate::NetworkId;
/// Maximum encoded retirement size, including its bounded claim proof.
pub const MAX_RETIREMENT_LEN: usize = 1024;
/// A network member's signed retirement of a proved claim version.
///
/// The original author's signed claim proves the cutoff really existed.
/// Newer claims remain valid. This is neither a forged Release nor a ban.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct SignedRetirement {
/// Device identity of the member requesting cleanup.
pub author: [u8; 32],
/// Network in which cleanup applies.
pub network: [u8; 32],
/// Requester's durable author counter, committed with this marker.
pub version: u64,
/// The target's original signed claim; bounds the retired version.
pub target: SignedRecord,
/// Requester's Ed25519 signature over this marker and its proof.
pub signature: Vec<u8>,
}
impl SignedRetirement {
fn canonical_bytes(&self) -> Vec<u8> {
let mut out = Vec::with_capacity(768);
push_lp(&mut out, b"tsunagi-node-retirement/v1");
push_lp(&mut out, &self.network);
push_lp(&mut out, &self.author);
push_lp(&mut out, &self.version.to_be_bytes());
// Bind every field of the original signed claim, including its signature.
push_lp(&mut out, &self.target.network);
push_lp(&mut out, &self.target.author);
push_lp(&mut out, &self.target.version.to_be_bytes());
push_lp(&mut out, &self.target.body.canonical());
push_lp(&mut out, &self.target.signature);
out
}
/// Signs a marker with the requesting member's persistent device key.
pub fn sign(
secret: &SecretKey,
network: NetworkId,
version: u64,
target: SignedRecord,
) -> Self {
let mut marker = Self {
author: *secret.public().as_bytes(),
network: *network.as_bytes(),
version,
target,
signature: Vec::new(),
};
marker.signature = secret.sign(&marker.canonical_bytes()).to_bytes().to_vec();
marker
}
/// Verifies both authors, scope and a finite, genuinely signed cutoff.
pub fn verify(&self, network: NetworkId) -> Result<EndpointId, StateError> {
if self.network != *network.as_bytes() {
return Err(StateError::WrongNetwork);
}
if self.version == 0 || self.version > i64::MAX as u64 {
return Err(StateError::Malformed("retirement version is out of range"));
}
if !matches!(self.target.body, RecordBody::Claim { .. }) {
return Err(StateError::Malformed("only a claim can be retired"));
}
let target = self.target.verify(network)?;
let author = EndpointId::from_bytes(&self.author).map_err(|_| StateError::BadAuthor)?;
if author == target {
return Err(StateError::Malformed(
"use Release to retire your own claim",
));
}
let raw: [u8; Signature::LENGTH] =
self.signature
.as_slice()
.try_into()
.map_err(|_| StateError::BadSignatureLength {
expected: Signature::LENGTH,
})?;
author
.verify(&self.canonical_bytes(), &Signature::from_bytes(&raw))
.map_err(|_| StateError::BadSignature)?;
Ok(target)
}
/// A deterministic strongest marker per target, independent of arrival order.
pub(crate) fn supersedes(&self, other: &Self) -> bool {
self.target.version > other.target.version
|| (self.target.version == other.target.version
&& (self.canonical_bytes(), &self.signature)
< (other.canonical_bytes(), &other.signature))
}
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
use crate::state::{Ipv4Range, StateSet};
fn claim(key: &SecretKey, id: NetworkId, version: u64) -> SignedRecord {
SignedRecord::sign(
key,
id,
version,
RecordBody::Claim {
address: Some("10.23.0.2".parse().unwrap()),
range: Some("10.23.0.0/24".parse::<Ipv4Range>().unwrap()),
hostname: Some("retired".into()),
hostname_since: Some(123),
},
)
}
#[test]
fn markers_release_claims_in_any_merge_order_and_older_snapshots_cannot_restore_them() {
let id = NetworkId::from_bytes([7; 32]);
let target = SecretKey::generate();
let cleaner = SecretKey::generate();
let record = claim(&target, id, 5);
let marker = SignedRetirement::sign(&cleaner, id, 8, record.clone());
for first in [true, false] {
let mut state = StateSet::new();
if first {
state.merge_retirement(id, marker.clone(), 16).unwrap();
}
state.merge(id, record.clone()).unwrap();
if !first {
state.merge_retirement(id, marker.clone(), 16).unwrap();
}
assert!(state.address_holders().is_empty());
assert!(state.hostname_holders().is_empty());
state.merge(id, claim(&target, id, 4)).unwrap();
assert!(state.is_retired(&target.public()));
assert!(
state.agreed_range().is_some(),
"cleanup does not renumber the network"
);
state.merge(id, claim(&target, id, 6)).unwrap();
assert!(!state.is_retired(&target.public()));
assert_eq!(state.hostname_of(&target.public()), Some("retired"));
}
}
#[test]
fn proof_signature_scope_tampering_future_cutoffs_and_capacity_are_checked() {
let id = NetworkId::from_bytes([7; 32]);
let target = SecretKey::generate();
let cleaner = SecretKey::generate();
let marker = SignedRetirement::sign(&cleaner, id, 8, claim(&target, id, 5));
assert!(marker.verify(NetworkId::from_bytes([8; 32])).is_err());
let mut forged = marker.clone();
forged.target.version = 99;
assert!(forged.verify(id).is_err());
let mut forged = marker.clone();
forged.signature[0] ^= 1;
assert!(forged.verify(id).is_err());
let mut proof = marker.target.clone();
proof.version = 99;
assert!(
SignedRetirement::sign(&cleaner, id, 8, proof)
.verify(id)
.is_err()
);
let mut state = StateSet::new();
assert!(state.merge_retirement(id, marker.clone(), 0).is_err());
assert!(state.merge_retirement(id, marker.clone(), 1).unwrap());
assert!(!state.merge_retirement(id, marker.clone(), 1).unwrap());
assert!(postcard::to_stdvec(&marker).unwrap().len() <= MAX_RETIREMENT_LEN);
let another = SecretKey::generate();
assert!(
state
.merge_retirement(
id,
SignedRetirement::sign(&cleaner, id, 9, claim(&another, id, 1)),
1
)
.is_err()
);
}
#[test]
fn competing_cleanup_markers_converge_and_keep_the_highest_observed_cutoff() {
let id = NetworkId::from_bytes([7; 32]);
let target = SecretKey::generate();
let a = SecretKey::generate();
let b = SecretKey::generate();
let markers = [
SignedRetirement::sign(&a, id, 1, claim(&target, id, 4)),
SignedRetirement::sign(&a, id, 2, claim(&target, id, 5)),
SignedRetirement::sign(&b, id, 3, claim(&target, id, 5)),
];
let mut forward = StateSet::new();
let mut reverse = StateSet::new();
for marker in &markers {
forward.merge_retirement(id, marker.clone(), 16).unwrap();
}
for marker in markers.iter().rev() {
reverse.merge_retirement(id, marker.clone(), 16).unwrap();
}
assert_eq!(forward.retirements(), reverse.retirements());
assert_eq!(
forward
.retirement_of(&target.public())
.unwrap()
.target
.version,
5
);
}
}
+59
View File
@@ -254,6 +254,12 @@ impl Storage {
.await
}
/// Persists the ordered protocol selection off the async executor.
pub async fn set_protocols(&self, network_id: NetworkId, protocols: Vec<String>) -> Result<()> {
self.with_state(move |state| state.set_protocols(network_id, &protocols))
.await
}
/// Persists whether this agent offers itself as an exit node in a network.
pub async fn set_exit_node(&self, network_id: NetworkId, enabled: bool) -> Result<()> {
self.with_state(move |state| state.set_exit_node(network_id, enabled))
@@ -272,11 +278,39 @@ impl Storage {
.await
}
/// Configured networks in the user's most recently selected order.
pub async fn network_start_order(&self) -> Result<Vec<NetworkId>> {
self.with_state(|state| {
let configured = state.list_networks()?;
let text = state
.get_setting("network_start_order")?
.unwrap_or_default();
let mut order = Vec::new();
for word in text.split(',') {
if let Ok(id) = word.parse::<NetworkId>()
&& configured.iter().any(|n| n.network_id == id)
&& !order.contains(&id)
{
order.push(id);
}
}
Ok(order)
})
.await
}
/// Prefers this network at startup without disabling suspended networks.
pub async fn prefer_network(&self, network: NetworkId) -> Result<()> {
self.with_state(move |state| state.prefer_network(network))
.await
}
/// Removes a network configuration and its cached hints.
pub async fn remove_network(&self, network_id: NetworkId) -> Result<()> {
self.with_state(move |state| {
state.remove_network(network_id)?;
state.forget_signed_records(network_id)?;
state.forget_retirements(network_id)?;
state.forget_peer_hostnames(network_id)
})
.await?;
@@ -336,6 +370,31 @@ impl Storage {
.await
}
/// Loads verified durable cleanup proofs off the executor.
pub async fn retirements(
&self,
network: NetworkId,
max: usize,
) -> Result<Vec<crate::state::SignedRetirement>> {
self.with_state(move |state| state.retirements(network, max))
.await
}
/// Persists a received cleanup proof before applying it.
pub async fn put_retirement(&self, marker: crate::state::SignedRetirement) -> Result<()> {
self.with_state(move |state| state.put_retirement(&marker))
.await
}
/// Commits our cleanup and author counter together before announcement.
pub async fn publish_own_retirement(
&self,
marker: crate::state::SignedRetirement,
) -> Result<()> {
self.with_state(move |state| state.publish_own_retirement(&marker))
.await
}
/// Stores a record received from another replica.
pub async fn put_signed_record(&self, record: SignedRecord) -> Result<()> {
self.with_state(move |state| state.put_signed_record(&record))
+369 -5
View File
@@ -18,10 +18,10 @@ use rusqlite::{Connection, OptionalExtension, params};
use crate::error::{Error, Result};
use crate::identity::{DeviceIdentity, NetworkId, NetworkName, NetworkSecret};
use crate::state::{RecordBody, SignedRecord};
use crate::state::{MAX_RETIREMENT_LEN, RecordBody, SignedRecord, SignedRetirement};
/// Schema version written by this build.
pub const SCHEMA_VERSION: i64 = 7;
pub const SCHEMA_VERSION: i64 = 9;
/// Key of the stored hostname setting.
const SETTING_HOSTNAME: &str = "hostname";
@@ -32,6 +32,8 @@ const SETTING_HOSTNAME: &str = "hostname";
/// and zeroizes on drop.
#[derive(Debug, Clone)]
pub struct StoredNetwork {
/// Ordered local protocol override; absent inherits the agent's defaults.
pub protocols: Option<Vec<String>>,
/// Derived public network identifier.
pub network_id: NetworkId,
/// Network name.
@@ -238,6 +240,29 @@ impl StateStore {
)
.map_err(|err| self.corrupt(format!("cannot migrate schema to 7: {err}")))?;
}
if found < 8 {
self.conn
.execute_batch(
"BEGIN;
ALTER TABLE networks ADD COLUMN protocols TEXT;
PRAGMA user_version = 8;
COMMIT;",
)
.map_err(|err| self.corrupt(format!("cannot migrate schema to 8: {err}")))?;
}
if found < 9 {
self.conn
.execute_batch(
"BEGIN;
CREATE TABLE IF NOT EXISTS retirements (
network_id BLOB NOT NULL, target BLOB NOT NULL, data BLOB NOT NULL,
PRIMARY KEY (network_id, target)
);
PRAGMA user_version = 9;
COMMIT;",
)
.map_err(|err| self.corrupt(format!("cannot migrate schema to 9: {err}")))?;
}
Ok(())
}
@@ -250,6 +275,7 @@ impl StateStore {
"settings",
"signed_records",
"peer_hostnames",
"retirements",
] {
let present: Option<String> = self
.conn
@@ -448,12 +474,16 @@ impl StateStore {
/// Sets the auto-start flag of a configured network.
pub fn set_auto_start(&self, network_id: NetworkId, auto_start: bool) -> Result<()> {
self.conn
let changed = self
.conn
.execute(
"UPDATE networks SET auto_start = ?2 WHERE network_id = ?1",
params![network_id.as_bytes().as_slice(), auto_start as i64],
)
.map_err(|err| Error::Storage(format!("cannot update network: {err}")))?;
if changed == 0 {
return Err(Error::NetworkUnknown(network_id));
}
Ok(())
}
@@ -472,6 +502,25 @@ impl StateStore {
Ok(())
}
/// Saves a local ordered protocol selection without changing membership.
pub fn set_protocols(&self, network_id: NetworkId, protocols: &[String]) -> Result<()> {
crate::dataplane::validate_protocols(
protocols,
crate::config::Limits::default().max_capabilities,
)?;
let changed = self
.conn
.execute(
"UPDATE networks SET protocols = ?2 WHERE network_id = ?1",
params![network_id.as_bytes().as_slice(), protocols.join(",")],
)
.map_err(|err| Error::Storage(format!("cannot update protocols: {err}")))?;
if changed == 0 {
return Err(Error::NetworkUnknown(network_id));
}
Ok(())
}
/// Removes a network configuration entirely.
pub fn remove_network(&self, network_id: NetworkId) -> Result<()> {
self.conn
@@ -532,7 +581,7 @@ impl StateStore {
pub fn list_networks(&self) -> Result<Vec<StoredNetwork>> {
let mut stmt = self
.conn
.prepare("SELECT network_id, name, secret, auto_start, broadcast, exit_node, exit_via FROM networks ORDER BY name")
.prepare("SELECT network_id, name, secret, auto_start, broadcast, exit_node, exit_via, protocols FROM networks ORDER BY name")
.map_err(|err| Error::Storage(format!("cannot list networks: {err}")))?;
let rows = stmt
.query_map([], |row| {
@@ -548,19 +597,37 @@ impl StateStore {
row.get::<_, bool>(4)?,
row.get::<_, bool>(5)?,
row.get::<_, Option<Vec<u8>>>(6)?,
row.get::<_, Option<String>>(7)?,
))
})
.map_err(|err| Error::Storage(format!("cannot list networks: {err}")))?;
let mut out = Vec::new();
for row in rows {
let (id, name, secret, auto_start, broadcast, exit_node, exit_via) =
let (id, name, secret, auto_start, broadcast, exit_node, exit_via, protocols) =
row.map_err(|err| Error::Storage(format!("cannot read network row: {err}")))?;
let protocols = protocols
.map(|text| {
let max = crate::config::Limits::default().max_capabilities;
if text.len() > max * (crate::dataplane::MAX_PROTOCOL_ID_LEN + 1) {
return Err(self.corrupt("stored protocol selection is too long"));
}
let names = if text.is_empty() {
Vec::new()
} else {
text.split(',').map(str::to_owned).collect()
};
crate::dataplane::validate_protocols(&names, max)
.map_err(|err| self.corrupt(err))?;
Ok(names)
})
.transpose()?;
let id: [u8; 32] = id
.as_slice()
.try_into()
.map_err(|_| self.corrupt("stored network id is not 32 bytes"))?;
out.push(StoredNetwork {
protocols,
network_id: NetworkId::from_bytes(id),
name: NetworkName::new(name)?,
secret: NetworkSecret::from_bytes(secret)?,
@@ -732,6 +799,112 @@ impl StateStore {
Ok(version.unwrap_or(0).max(0) as u64)
}
/// Loads durable cleanup markers without allocating oversized stored blobs.
pub fn retirements(&self, network: NetworkId, max: usize) -> Result<Vec<SignedRetirement>> {
let mut stmt = self
.conn
.prepare(
"SELECT CASE WHEN length(data) <= ?2 THEN data ELSE NULL END
FROM retirements WHERE network_id = ?1 ORDER BY target",
)
.map_err(|err| self.corrupt(err))?;
let rows = stmt
.query_map(
params![network.as_bytes().as_slice(), MAX_RETIREMENT_LEN as i64],
|row| row.get::<_, Option<Vec<u8>>>(0),
)
.map_err(|err| self.corrupt(err))?;
let mut out = Vec::new();
for row in rows {
if out.len() >= max {
return Err(Error::Storage(
"stored retirements exceed the configured limit".into(),
));
}
let bytes = row
.map_err(|err| self.corrupt(err))?
.ok_or_else(|| self.corrupt("oversized retirement"))?;
let marker: SignedRetirement =
postcard::from_bytes(&bytes).map_err(|err| self.corrupt(err))?;
marker.verify(network).map_err(|err| self.corrupt(err))?;
out.push(marker);
}
Ok(out)
}
/// Persists a strongest verified marker received from an authenticated session.
pub fn put_retirement(&self, marker: &SignedRetirement) -> Result<()> {
let network = NetworkId::from_bytes(marker.network);
marker
.verify(network)
.map_err(|err| Error::Storage(err.to_string()))?;
let old: Option<Option<Vec<u8>>> = self.conn.query_row(
"SELECT CASE WHEN length(data) <= ?3 THEN data ELSE NULL END FROM retirements WHERE network_id = ?1 AND target = ?2",
params![marker.network.as_slice(), marker.target.author.as_slice(), MAX_RETIREMENT_LEN as i64], |row| row.get(0))
.optional().map_err(|err| self.corrupt(err))?;
if let Some(bytes) = old {
let bytes = bytes.ok_or_else(|| self.corrupt("oversized retirement"))?;
let old: SignedRetirement =
postcard::from_bytes(&bytes).map_err(|err| self.corrupt(err))?;
old.verify(network).map_err(|err| self.corrupt(err))?;
if !marker.supersedes(&old) {
return Ok(());
}
}
let bytes = postcard::to_stdvec(marker).map_err(|err| Error::Storage(err.to_string()))?;
if bytes.len() > MAX_RETIREMENT_LEN {
return Err(Error::Storage(
"retirement exceeds its encoded bound".into(),
));
}
self.conn
.execute(
"INSERT INTO retirements (network_id, target, data) VALUES (?1, ?2, ?3)
ON CONFLICT(network_id, target) DO UPDATE SET data = excluded.data",
params![
marker.network.as_slice(),
marker.target.author.as_slice(),
bytes
],
)
.map_err(|err| Error::Storage(format!("cannot store retirement: {err}")))?;
Ok(())
}
/// Commits our signed cleanup marker and author counter in one transaction.
pub fn publish_own_retirement(&self, marker: &SignedRetirement) -> Result<()> {
let transaction = self
.conn
.unchecked_transaction()
.map_err(|err| Error::Storage(err.to_string()))?;
self.put_retirement(marker)?;
transaction
.execute(
"INSERT INTO own_record_version (network_id, author, version) VALUES (?1, ?2, ?3)
ON CONFLICT(network_id, author) DO UPDATE SET version = max(version, excluded.version)",
params![
marker.network.as_slice(),
marker.author.as_slice(),
marker.version as i64
],
)
.map_err(|err| Error::Storage(format!("cannot store retirement counter: {err}")))?;
transaction
.commit()
.map_err(|err| Error::Storage(format!("cannot commit retirement: {err}")))
}
/// Drops a forgotten network's local cleanup proofs.
pub fn forget_retirements(&self, network: NetworkId) -> Result<()> {
self.conn
.execute(
"DELETE FROM retirements WHERE network_id = ?1",
params![network.as_bytes().as_slice()],
)
.map_err(|err| Error::Storage(err.to_string()))?;
Ok(())
}
/// Forgets every record of a network.
pub fn forget_signed_records(&self, network_id: NetworkId) -> Result<()> {
self.conn
@@ -798,6 +971,44 @@ impl StateStore {
Ok(())
}
/// Atomically enables a network and puts it first in the startup order.
pub fn prefer_network(&self, network: NetworkId) -> Result<()> {
let configured = self.list_networks()?;
if !configured.iter().any(|n| n.network_id == network) {
return Err(Error::NetworkUnknown(network));
}
let existing = self.get_setting("network_start_order")?.unwrap_or_default();
let mut order = vec![network];
for word in existing.split(',') {
if let Ok(id) = word.parse::<NetworkId>()
&& configured.iter().any(|n| n.network_id == id)
&& !order.contains(&id)
{
order.push(id);
}
}
let value = order
.iter()
.map(ToString::to_string)
.collect::<Vec<_>>()
.join(",");
let transaction = self
.conn
.unchecked_transaction()
.map_err(|err| Error::Storage(format!("cannot begin network preference: {err}")))?;
transaction
.execute(
"UPDATE networks SET auto_start = 1 WHERE network_id = ?1",
params![network.as_bytes().as_slice()],
)
.map_err(|err| Error::Storage(format!("cannot enable network: {err}")))?;
transaction.execute("INSERT INTO settings (key, value) VALUES ('network_start_order', ?1) ON CONFLICT(key) DO UPDATE SET value = excluded.value", params![value])
.map_err(|err| Error::Storage(format!("cannot store network preference: {err}")))?;
transaction
.commit()
.map_err(|err| Error::Storage(format!("cannot commit network preference: {err}")))
}
/// Reads an arbitrary setting.
pub fn get_setting(&self, key: &str) -> Result<Option<String>> {
self.conn
@@ -915,6 +1126,158 @@ mod broadcast_storage_tests {
assert!(store.peer_hostnames(id).unwrap().is_empty());
}
#[test]
fn retirement_migration_and_atomic_counter_commit_preserve_existing_state() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("state.sqlite");
let store = StateStore::open(&path).unwrap();
let identity = store.load_or_create_device_identity().unwrap();
let name = NetworkName::new("retirement-migration").unwrap();
let secret = NetworkSecret::from_bytes([6; 32]).unwrap();
let id = NetworkKeys::derive(&name, &secret).network_id();
store.upsert_network(id, &name, &secret, true).unwrap();
store.set_protocols(id, &["awg".into()]).unwrap();
let own = SignedRecord::sign(&identity.signing_key(), id, 8, RecordBody::Release);
store.publish_own_record(&own).unwrap();
store
.conn
.execute_batch("DROP TABLE retirements; PRAGMA user_version = 8;")
.unwrap();
drop(store);
let store = StateStore::open(&path).unwrap();
assert_eq!(
store.device_identity().unwrap().unwrap().endpoint_id(),
identity.endpoint_id()
);
assert_eq!(
store.list_networks().unwrap()[0].protocols,
Some(vec!["awg".into()])
);
assert_eq!(store.signed_records(id).unwrap(), vec![own]);
let target_key = iroh::SecretKey::generate();
let target = SignedRecord::sign(
&target_key,
id,
3,
RecordBody::Claim {
address: Some("10.13.37.2".parse().unwrap()),
range: Some(crate::state::DEFAULT_IPV4_RANGE),
hostname: Some("old-device".into()),
hostname_since: Some(123),
},
);
let marker = SignedRetirement::sign(&identity.signing_key(), id, 9, target);
store
.conn
.execute_batch(
"CREATE TRIGGER fail_retirement_counter BEFORE INSERT ON own_record_version
WHEN NEW.version = 9 BEGIN SELECT RAISE(ABORT, 'counter write refused'); END;",
)
.unwrap();
assert!(store.publish_own_retirement(&marker).is_err());
assert!(
store.retirements(id, 16).unwrap().is_empty(),
"marker must roll back with counter failure"
);
assert_eq!(
store
.own_record_version(id, identity.endpoint_id())
.unwrap(),
8
);
store
.conn
.execute_batch("DROP TRIGGER fail_retirement_counter;")
.unwrap();
store.publish_own_retirement(&marker).unwrap();
drop(store);
let store = StateStore::open(&path).unwrap();
assert_eq!(store.retirements(id, 16).unwrap(), vec![marker.clone()]);
assert_eq!(
store
.own_record_version(id, identity.endpoint_id())
.unwrap(),
9
);
assert!(
store.retirements(id, 0).is_err(),
"capacity changes must not silently drop markers"
);
store
.conn
.execute(
"UPDATE retirements SET data = ?1",
params![vec![0u8; MAX_RETIREMENT_LEN + 1]],
)
.unwrap();
assert!(matches!(
store.retirements(id, 16),
Err(Error::StateCorrupted { .. })
));
}
#[test]
fn v7_migration_keeps_signed_state_and_protocol_overrides_are_local_and_durable() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("state.sqlite");
let store = StateStore::open(&path).unwrap();
let identity = store.load_or_create_device_identity().unwrap();
let name = NetworkName::new("protocol-migration").unwrap();
let secret = NetworkSecret::from_bytes([4; 32]).unwrap();
let id = NetworkKeys::derive(&name, &secret).network_id();
store.upsert_network(id, &name, &secret, true).unwrap();
store.set_broadcast(id, false).unwrap();
let record = SignedRecord::sign(&identity.signing_key(), id, 17, RecordBody::Release);
store.publish_own_record(&record).unwrap();
store
.conn
.execute_batch("ALTER TABLE networks DROP COLUMN protocols; PRAGMA user_version=7;")
.unwrap();
drop(store);
let store = StateStore::open(&path).unwrap();
assert_eq!(
store
.load_or_create_device_identity()
.unwrap()
.endpoint_id(),
identity.endpoint_id()
);
assert_eq!(store.signed_records(id).unwrap(), vec![record]);
let saved = store.list_networks().unwrap();
assert!(saved[0].protocols.is_none());
assert!(!saved[0].broadcast);
assert!(saved[0].auto_start);
assert_eq!(saved[0].secret, secret);
let chosen = vec!["awg".into(), "wg-quic".into()];
store.set_protocols(id, &chosen).unwrap();
assert!(
store
.set_protocols(id, &["awg".into(), "awg".into()])
.is_err()
);
assert!(
store
.set_protocols(NetworkId::from_bytes([1; 32]), &[])
.is_err()
);
drop(store);
let store = StateStore::open(&path).unwrap();
assert_eq!(store.list_networks().unwrap()[0].protocols, Some(chosen));
store.upsert_network(id, &name, &secret, true).unwrap();
store.set_protocols(id, &[]).unwrap();
drop(store);
let store = StateStore::open(&path).unwrap();
assert_eq!(store.list_networks().unwrap()[0].protocols, Some(vec![]));
store
.conn
.execute("UPDATE networks SET protocols = 'awg,awg'", [])
.unwrap();
assert!(matches!(
store.list_networks(),
Err(Error::StateCorrupted { .. })
));
}
#[test]
fn v3_migration_preserves_identity_and_networks_and_defaults_broadcast_on() {
let dir = tempfile::tempdir().unwrap();
@@ -935,6 +1298,7 @@ mod broadcast_storage_tests {
"ALTER TABLE networks DROP COLUMN broadcast; \
ALTER TABLE networks DROP COLUMN exit_node; \
ALTER TABLE networks DROP COLUMN exit_via; \
ALTER TABLE networks DROP COLUMN protocols; \
PRAGMA user_version=3;",
)
.unwrap();
+358
View File
@@ -0,0 +1,358 @@
//! Equal-rights retirement uses real membership, signatures and persistent state.
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use tsunagi::discovery::SharedMemoryDiscovery;
use tsunagi::testing::{TestAgent, config_with, network, wait_for_peers, wait_until};
use tsunagi::{Agent, NetworkId};
#[tokio::test]
async fn bounded_marker_batches_eventually_reach_the_tail_of_a_small_send_queue() {
use tsunagi::config::StoragePaths;
use tsunagi::state::{RecordBody, SignedRecord, SignedRetirement};
use tsunagi::storage::StateStore;
let discovery = SharedMemoryDiscovery::new();
let cleaner = TestAgent::spawn(&discovery).await.unwrap();
let replica = TestAgent::spawn(&discovery).await.unwrap();
let (name, secret) = network("forget-batched-history");
let id = cleaner.agent.join_network(&name, &secret).await.unwrap();
replica.agent.join_network(&name, &secret).await.unwrap();
let cleaner_dir = cleaner.stop().await;
let replica_dir = replica.stop().await;
let cleaner_path = StoragePaths::under(cleaner_dir.path()).state_db();
let replica_path = StoragePaths::under(replica_dir.path()).state_db();
// Seed genuine signed offline history in closed stores. Delivery still uses
// real agents, authenticated loopback sessions and the persistent stores.
let targets = tokio::task::spawn_blocking(move || {
let cleaner = StateStore::open(&cleaner_path).unwrap();
let replica = StateStore::open(&replica_path).unwrap();
let author = cleaner.device_identity().unwrap().unwrap();
let signing_key = iroh::SecretKey::from_bytes(&author.secret_bytes());
let mut targets = Vec::new();
for index in 0..12 {
let target = iroh::SecretKey::generate();
let record = SignedRecord::sign(
&target,
id,
1,
RecordBody::Claim {
address: None,
range: None,
hostname: Some(format!("offline-{index}")),
hostname_since: Some(1),
},
);
cleaner.put_signed_record(&record).unwrap();
replica.put_signed_record(&record).unwrap();
cleaner
.publish_own_retirement(&SignedRetirement::sign(
&signing_key,
id,
100 + index,
record,
))
.unwrap();
targets.push(target.public());
}
targets
})
.await
.unwrap();
let replica = Agent::spawn(config_with(replica_dir.path(), &discovery))
.await
.unwrap();
assert_eq!(
replica
.network_status(id)
.await
.unwrap()
.members
.iter()
.filter(|m| targets.contains(&m.endpoint_id))
.count(),
12
);
let mut config = config_with(cleaner_dir.path(), &discovery);
config.limits.max_state_records = 1;
config.limits.session_send_queue = 2;
let cleaner = Agent::spawn(config).await.unwrap();
wait_for_peers(&cleaner, id, 1).await;
wait_until(
"all cleanup batches arrive despite send queue pressure",
|| async {
replica
.network_status(id)
.await
.ok()?
.members
.iter()
.all(|m| !targets.contains(&m.endpoint_id))
.then_some(())
},
)
.await;
cleaner.shutdown().await;
replica.shutdown().await;
let replica_path = StoragePaths::under(replica_dir.path()).state_db();
tokio::task::spawn_blocking(move || {
let replica = StateStore::open(&replica_path).unwrap();
assert_eq!(replica.retirements(id, 16).unwrap().len(), 12);
})
.await
.unwrap();
}
async fn member(
agent: &Agent,
network: NetworkId,
peer: iroh::EndpointId,
) -> tsunagi::agent::MemberStatus {
wait_until("signed member", || async {
agent
.network_status(network)
.await
.ok()?
.members
.into_iter()
.find(|m| m.endpoint_id == peer)
})
.await
}
#[tokio::test]
async fn retirement_survives_restart_and_stale_replicas_but_allows_a_new_claim() {
let discovery = SharedMemoryDiscovery::new();
let cleaner = TestAgent::spawn_with(|c| c.with_hostname("cleaner"), &discovery)
.await
.unwrap();
let target = TestAgent::spawn_with(|c| c.with_hostname("old-device"), &discovery)
.await
.unwrap();
let replica = TestAgent::spawn_with(|c| c.with_hostname("replica"), &discovery)
.await
.unwrap();
let (name, secret) = network("forget-propagation");
let id = cleaner.agent.join_network(&name, &secret).await.unwrap();
target.agent.join_network(&name, &secret).await.unwrap();
replica.agent.join_network(&name, &secret).await.unwrap();
wait_for_peers(&cleaner.agent, id, 2).await;
let peer = target.agent.endpoint_id();
let original = wait_until("all three agents agree on allocated claims", || async {
let a = cleaner.agent.network_status(id).await.ok()?;
let b = target.agent.network_status(id).await.ok()?;
let c = replica.agent.network_status(id).await.ok()?;
for status in [&a, &b, &c] {
if status.members.len() != 3 {
return None;
}
let addresses: std::collections::HashSet<_> = status
.members
.iter()
.filter_map(|m| m.overlay_address_v4)
.collect();
if addresses.len() != 3 {
return None;
}
}
let record = a.members.iter().find(|m| m.endpoint_id == peer)?;
(b.members
.iter()
.find(|m| m.endpoint_id == peer)?
.record_version
== record.record_version
&& c.members
.iter()
.find(|m| m.endpoint_id == peer)?
.record_version
== record.record_version)
.then(|| record.clone())
})
.await;
assert!(
cleaner
.agent
.forget_node(id, peer, original.record_version)
.await
.is_err(),
"online nodes are not history"
);
assert!(
cleaner
.agent
.forget_node(id, cleaner.agent.endpoint_id(), 1)
.await
.is_err()
);
let replica_dir = replica.stop().await;
let target_dir = target.stop().await;
wait_until("target offline", || async {
cleaner
.agent
.network_status(id)
.await
.ok()?
.members
.iter()
.find(|m| m.endpoint_id == peer)
.filter(|m| !m.online)
.map(|_| ())
})
.await;
assert!(
cleaner
.agent
.forget_node(id, peer, original.record_version + 1)
.await
.is_err(),
"stale confirmation changes nothing"
);
let result = cleaner
.agent
.forget_node(id, peer, original.record_version)
.await
.unwrap();
assert_eq!(result.target, peer);
assert_eq!(result.record_version, original.record_version);
assert_eq!(result.peers_told, 0);
assert!(
!cleaner
.agent
.network_status(id)
.await
.unwrap()
.members
.iter()
.any(|m| m.endpoint_id == peer)
);
let cleaner_dir = cleaner.stop().await;
let cleaner =
Agent::spawn(config_with(cleaner_dir.path(), &discovery).with_hostname("cleaner"))
.await
.unwrap();
let replica =
Agent::spawn(config_with(replica_dir.path(), &discovery).with_hostname("replica"))
.await
.unwrap();
wait_for_peers(&cleaner, id, 1).await;
wait_until("stale replica learns durable retirement", || async {
replica
.network_status(id)
.await
.ok()?
.members
.iter()
.all(|m| m.endpoint_id != peer)
.then_some(())
})
.await;
assert!(
!cleaner
.network_status(id)
.await
.unwrap()
.members
.iter()
.any(|m| m.endpoint_id == peer)
);
replica.set_hostname("old-device").await.unwrap();
let replica_id = replica.endpoint_id();
wait_until("released hostname is reclaimed", || async {
cleaner
.network_status(id)
.await
.ok()?
.members
.iter()
.find(|m| m.endpoint_id == replica_id)
.filter(|m| m.hostname.as_deref() == Some("old-device"))?
.hostname_conflict
.is_none()
.then_some(())
})
.await;
let returned =
Agent::spawn(config_with(target_dir.path(), &discovery).with_hostname("old-device"))
.await
.unwrap();
wait_for_peers(&returned, id, 2).await;
wait_until("returning identity publishes a fresh version", || async {
let m = cleaner
.network_status(id)
.await
.ok()?
.members
.into_iter()
.find(|m| m.endpoint_id == peer)?;
(m.online && m.record_version > original.record_version && m.hostname_conflict.is_some())
.then_some(())
})
.await;
cleaner.shutdown().await;
replica.shutdown().await;
returned.shutdown().await;
}
#[tokio::test]
async fn forgetting_is_network_scoped_and_every_member_has_the_same_right() {
let discovery = SharedMemoryDiscovery::new();
let a = TestAgent::spawn(&discovery).await.unwrap();
let b = TestAgent::spawn(&discovery).await.unwrap();
let target = TestAgent::spawn(&discovery).await.unwrap();
let (name, secret) = network("forget-one-network");
let (other_name, other_secret) = network("keep-other-network");
let id = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
target.agent.join_network(&name, &secret).await.unwrap();
let other = a
.agent
.join_network(&other_name, &other_secret)
.await
.unwrap();
target
.agent
.join_network(&other_name, &other_secret)
.await
.unwrap();
let peer = target.agent.endpoint_id();
let record = member(&b.agent, id, peer).await;
member(&a.agent, other, peer).await;
target.agent.shutdown().await;
wait_until("offline on B", || async {
b.agent
.network_status(id)
.await
.ok()?
.members
.iter()
.find(|m| m.endpoint_id == peer)
.filter(|m| !m.online)
.map(|_| ())
})
.await;
b.agent
.forget_node(id, peer, record.record_version)
.await
.unwrap();
wait_until("A learns B's retirement", || async {
a.agent
.network_status(id)
.await
.ok()?
.members
.iter()
.all(|m| m.endpoint_id != peer)
.then_some(())
})
.await;
assert!(
a.agent
.network_status(other)
.await
.unwrap()
.members
.iter()
.any(|m| m.endpoint_id == peer)
);
a.agent.shutdown().await;
b.agent.shutdown().await;
}
+1 -1
View File
@@ -162,7 +162,7 @@ async fn rejoining_after_leaving_is_not_taken_for_a_stale_record() {
})
.await;
assert!(
tsunagi::state::DEFAULT_IPV4_RANGE.contains(again),
tsunagi::state::derived_ipv4_range(network_id).contains(again),
"an address in the network's range: {again}"
);
@@ -265,10 +265,7 @@ async fn deactivating_one_network_leaves_the_agent_and_others_running() {
#[tokio::test]
async fn two_networks_on_one_agent_each_get_a_range_of_their_own() {
// One agent, one interface: the configured range can only belong to one
// of them. The second does not go without — it takes the range derived
// from its own id, which every one of its members derives identically,
// so it is an agreement and not a local invention.
// Each network derives its range independently of activation order.
let discovery = SharedMemoryDiscovery::new();
let (first, first_secret) = network("two-ranges-one");
let (second, second_secret) = network("two-ranges-two");
@@ -309,8 +306,8 @@ async fn two_networks_on_one_agent_each_get_a_range_of_their_own() {
"different networks, different addresses"
);
assert!(
tsunagi::state::DEFAULT_IPV4_RANGE.contains(addresses.0),
"the first keeps the configured range: {}",
tsunagi::state::derived_ipv4_range(one).contains(addresses.0),
"the first uses its derived range: {}",
addresses.0
);
assert!(
@@ -319,7 +316,7 @@ async fn two_networks_on_one_agent_each_get_a_range_of_their_own() {
addresses.1
);
assert!(
!tsunagi::state::DEFAULT_IPV4_RANGE.contains(addresses.1),
!tsunagi::state::derived_ipv4_range(one).contains(addresses.1),
"and not in the one the first network holds"
);
@@ -0,0 +1,336 @@
//! Derived defaults and local exclusion of overlapping overlay networks.
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use tsunagi::Agent;
use tsunagi::discovery::SharedMemoryDiscovery;
use tsunagi::state::{DEFAULT_IPV4_RANGE, Ipv4Range, derived_ipv4_range};
use tsunagi::testing::{TestAgent, config_with, network, wait_until};
// These tests need discovery to happen before a collision, so keep the initial
// default outside the established ranges. Derived collisions have their own test.
fn network_outside(
name: &str,
ranges: &[Ipv4Range],
) -> (
tsunagi::identity::NetworkName,
tsunagi::identity::NetworkSecret,
) {
use tsunagi::identity::{NetworkKeys, NetworkName, NetworkSecret};
let name = NetworkName::new(name).unwrap();
let secret = (0..=255u8)
.map(|seed| NetworkSecret::from_bytes([seed; 32]).unwrap())
.find(|secret| {
let derived = derived_ipv4_range(NetworkKeys::derive(&name, secret).network_id());
ranges.iter().all(|range| !range.contains(derived.base))
})
.expect("a deterministic test secret with a non-overlapping default");
(name, secret)
}
#[tokio::test]
async fn colliding_derived_defaults_are_blocked_instead_of_locally_renumbered() {
use std::collections::HashMap;
use tsunagi::identity::{NetworkKeys, NetworkName, NetworkSecret};
let secret = NetworkSecret::from_bytes([19; 32]).unwrap();
let mut seen = HashMap::new();
let (first, second, range) = (0..=65_536)
.find_map(|index| {
let name = NetworkName::new(format!("derived-collision-{index}")).unwrap();
let range = derived_ipv4_range(NetworkKeys::derive(&name, &secret).network_id());
seen.insert(range.base, name.clone())
.map(|first| (first, name, range))
})
.expect("more network names than possible /24 ranges");
let agent = TestAgent::spawn(&SharedMemoryDiscovery::new())
.await
.unwrap();
let first = agent.agent.join_network(&first, &secret).await.unwrap();
let second = agent.agent.join_network(&second, &secret).await.unwrap();
assert!(agent.agent.is_active(first).await);
assert!(!agent.agent.is_active(second).await);
let status = agent.agent.status().await.unwrap();
let blocked = status.network(&second).unwrap();
assert_eq!(blocked.range, Some(range));
assert_eq!(blocked.range_conflicts[0].network_id, first);
assert_eq!(
agent.agent.switch_network(second).await.unwrap(),
vec![first]
);
assert_eq!(
agent.agent.network_status(second).await.unwrap().range,
Some(range)
);
agent.agent.shutdown().await;
}
#[tokio::test]
async fn isolated_first_members_choose_the_same_range_and_keep_it_after_restart() {
let discovery = SharedMemoryDiscovery::new();
let (name, secret) = network("derived-first");
let a = TestAgent::spawn(&discovery).await.unwrap();
let b = TestAgent::spawn(&SharedMemoryDiscovery::new())
.await
.unwrap();
let id = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
let expected = derived_ipv4_range(id);
for agent in [&a.agent, &b.agent] {
wait_until("isolated member allocates in its derived range", || async {
let status = agent.network_status(id).await.ok()?;
(status.range == Some(expected) && !status.members.is_empty()).then_some(())
})
.await;
}
let dir = a.stop().await;
let a = Agent::spawn(config_with(dir.path(), &discovery))
.await
.unwrap();
assert_eq!(a.network_status(id).await.unwrap().range, Some(expected));
a.shutdown().await;
b.agent.shutdown().await;
}
#[tokio::test]
async fn overlap_blocks_activation_switches_and_resumes_across_restart() {
let discovery = SharedMemoryDiscovery::new();
let a = TestAgent::spawn_with(
|c| c.with_overlay_ipv4_range(Some(DEFAULT_IPV4_RANGE)),
&discovery,
)
.await
.unwrap();
let (one_name, one_secret) = network("conflict-one");
let (two_name, two_secret) = network("conflict-two");
let one = a.agent.join_network(&one_name, &one_secret).await.unwrap();
a.agent.network_status(one).await.unwrap();
let two = a.agent.join_network(&two_name, &two_secret).await.unwrap();
assert!(a.agent.is_active(one).await);
assert!(!a.agent.is_active(two).await);
assert!(matches!(
a.agent.activate_network(two).await,
Err(tsunagi::Error::NetworkRangeConflict { .. })
));
let status = a.agent.status().await.unwrap();
let blocked = status.network(&two).unwrap();
assert_eq!(blocked.range, Some(DEFAULT_IPV4_RANGE));
assert_eq!(blocked.range_conflicts[0].network_id, one);
assert_eq!(blocked.range_conflicts[0].name, one_name);
assert_eq!(a.agent.switch_network(two).await.unwrap(), vec![one]);
assert!(a.agent.is_active(two).await);
assert!(!a.agent.is_active(one).await);
a.agent.network_status(two).await.unwrap();
let dir = a.stop().await;
let a = Agent::spawn(config_with(dir.path(), &discovery))
.await
.unwrap();
assert!(a.is_active(two).await, "the user's choice survives restart");
assert!(!a.is_active(one).await);
a.deactivate_network(two).await.unwrap();
assert!(
a.is_active(one).await,
"suspended networks resume when the blocker stops"
);
assert_eq!(
a.network_status(one).await.unwrap().range,
Some(DEFAULT_IPV4_RANGE),
"legacy ranges survive the new default"
);
a.shutdown().await;
}
#[tokio::test]
async fn late_authenticated_state_with_a_nested_range_suspends_only_that_network() {
let discovery = SharedMemoryDiscovery::new();
let broad: Ipv4Range = "10.90.0.0/16".parse().unwrap();
let narrow: Ipv4Range = "10.90.12.0/24".parse().unwrap();
let (name, secret) = network_outside("late-conflict", &[broad]);
let remote = TestAgent::spawn_with(|c| c.with_overlay_ipv4_range(Some(narrow)), &discovery)
.await
.unwrap();
let target = remote.agent.join_network(&name, &secret).await.unwrap();
wait_until("remote range is persisted", || async {
remote
.agent
.network_status(target)
.await
.ok()?
.members
.first()
.map(|_| ())
})
.await;
let (other_name, other_secret) = network("existing-broad");
// Seed an existing network using a real agent and its signed claim.
let seeded = TestAgent::spawn_with(
|c| c.with_overlay_ipv4_range(Some(broad)),
&SharedMemoryDiscovery::new(),
)
.await
.unwrap();
let other = seeded
.agent
.join_network(&other_name, &other_secret)
.await
.unwrap();
wait_until("broad range is persisted", || async {
seeded
.agent
.network_status(other)
.await
.ok()?
.members
.first()
.map(|_| ())
})
.await;
let dir = seeded.stop().await;
let seeded = Agent::spawn(config_with(dir.path(), &discovery))
.await
.unwrap();
seeded.join_network(&name, &secret).await.unwrap();
wait_until("late overlap suspends the joining network", || async {
let status = seeded.status().await.ok()?;
let blocked = status.network(&target)?;
(!seeded.is_active(target).await
&& blocked
.range_conflicts
.iter()
.any(|c| c.network_id == other))
.then_some(())
})
.await;
assert!(seeded.is_active(other).await);
assert_eq!(seeded.switch_network(target).await.unwrap(), vec![other]);
assert_eq!(
seeded.network_status(target).await.unwrap().range,
Some(narrow)
);
seeded.shutdown().await;
remote.agent.shutdown().await;
}
#[tokio::test]
async fn a_wide_range_suspends_all_overlaps_and_manual_stop_cancels_resume() {
let discovery = SharedMemoryDiscovery::new();
let hub = TestAgent::spawn(&discovery).await.unwrap();
let mut members = Vec::new();
let mut ids = Vec::new();
for (name, range) in [
("narrow-one", "10.81.0.0/24"),
("narrow-two", "10.81.1.0/24"),
] {
let peer = TestAgent::spawn_with(
|c| c.with_overlay_ipv4_range(Some(range.parse().unwrap())),
&discovery,
)
.await
.unwrap();
let (name, secret) = network_outside(
name,
&[
"10.81.0.0/24".parse().unwrap(),
"10.81.1.0/24".parse().unwrap(),
],
);
let id = peer.agent.join_network(&name, &secret).await.unwrap();
peer.agent.network_status(id).await.unwrap();
hub.agent.join_network(&name, &secret).await.unwrap();
wait_until("hub adopts the established narrow range", || async {
(hub.agent.network_status(id).await.ok()?.range == Some(range.parse().unwrap()))
.then_some(())
})
.await;
members.push(peer);
ids.push(id);
}
let (name, secret) = network_outside(
"wide-overlap",
&[
"10.81.0.0/24".parse().unwrap(),
"10.81.1.0/24".parse().unwrap(),
],
);
let wide = TestAgent::spawn_with(
|c| c.with_overlay_ipv4_range(Some("10.0.0.0/8".parse().unwrap())),
&discovery,
)
.await
.unwrap();
let id = wide.agent.join_network(&name, &secret).await.unwrap();
wide.agent.network_status(id).await.unwrap();
hub.agent.join_network(&name, &secret).await.unwrap();
wait_until("wide range is blocked by both narrow networks", || async {
let status = hub.agent.status().await.ok()?;
let blocked = status.network(&id)?;
(!hub.agent.is_active(id).await && blocked.range_conflicts.len() == 2).then_some(())
})
.await;
let stopped = hub.agent.switch_network(id).await.unwrap();
assert_eq!(stopped.len(), 2);
assert!(ids.iter().all(|id| stopped.contains(id)));
assert_eq!(hub.agent.switch_network(ids[0]).await.unwrap(), vec![id]);
assert!(
hub.agent.is_active(ids[1]).await,
"switching away from the wide range frees the other narrow range too"
);
assert_eq!(hub.agent.switch_network(id).await.unwrap().len(), 2);
hub.agent.pause_network(ids[0]).await.unwrap();
hub.agent.deactivate_network(id).await.unwrap();
assert!(
!hub.agent.is_active(ids[0]).await,
"explicitly stopped networks do not resume"
);
assert!(hub.agent.is_active(ids[1]).await);
assert!(
hub.agent
.switch_network(tsunagi::NetworkId::from_bytes([0; 32]))
.await
.is_err()
);
assert!(
hub.agent.is_active(ids[1]).await,
"unknown selectors do not stop any network"
);
hub.agent.shutdown().await;
wide.agent.shutdown().await;
for peer in members {
peer.agent.shutdown().await;
}
}
#[tokio::test]
async fn simultaneous_switches_never_leave_overlapping_networks_active() {
let discovery = SharedMemoryDiscovery::new();
let agent = TestAgent::spawn_with(
|c| c.with_overlay_ipv4_range(Some(DEFAULT_IPV4_RANGE)),
&discovery,
)
.await
.unwrap();
let (a, a_secret) = network("switch-a");
let (b, b_secret) = network("switch-b");
let a = agent.agent.join_network(&a, &a_secret).await.unwrap();
agent.agent.network_status(a).await.unwrap();
let b = agent.agent.join_network(&b, &b_secret).await.unwrap();
let (left, right) = tokio::join!(agent.agent.switch_network(b), agent.agent.switch_network(a));
left.unwrap();
right.unwrap();
assert_ne!(
agent.agent.is_active(a).await,
agent.agent.is_active(b).await
);
assert_eq!(
agent
.agent
.list_networks()
.await
.unwrap()
.iter()
.filter(|n| n.active)
.count(),
1
);
agent.agent.shutdown().await;
}
@@ -0,0 +1,130 @@
//! Opaque transport negotiation stays inside authenticated network sessions.
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use std::sync::{Arc, Mutex};
use tsunagi::dataplane::transport::{ControlSender, PacketTransport, SharedLink, TransportError};
use tsunagi::discovery::SharedMemoryDiscovery;
use tsunagi::proto::message::{ControlMessage, TransportPayload};
use tsunagi::testing::{TestAgent, network, wait_for_peers, wait_until};
use tsunagi::{BoxFuture, NetworkId};
#[derive(Debug, Default)]
struct Negotiator {
control: Mutex<Option<Arc<dyn ControlSender>>>,
received: Mutex<Vec<(NetworkId, iroh::EndpointId, Vec<u8>)>>,
}
impl PacketTransport for Negotiator {
fn name(&self) -> &str {
"negotiator"
}
fn as_any(&self) -> &dyn std::any::Any {
self
}
fn attach_control(&self, control: Arc<dyn ControlSender>) {
*self.control.lock().unwrap() = Some(control);
}
fn on_control(
&self,
network: NetworkId,
peer: iroh::EndpointId,
_: &str,
payload: &[u8],
) -> Result<(), TransportError> {
self.received
.lock()
.unwrap()
.push((network, peer, payload.to_vec()));
Ok(())
}
fn open<'a>(
&'a self,
_: NetworkId,
_: iroh::EndpointId,
_: &'a str,
) -> BoxFuture<'a, Result<SharedLink, TransportError>> {
Box::pin(async { Err(TransportError::Closed) })
}
}
#[tokio::test]
async fn negotiated_secrets_are_redacted_scoped_and_delivered_over_real_control() {
let discovery = SharedMemoryDiscovery::new();
let a_transport = Arc::new(Negotiator::default());
let b_transport = Arc::new(Negotiator::default());
let a = TestAgent::spawn_with(
|c| c.with_custom_transport("negotiator", a_transport.clone()),
&discovery,
)
.await
.unwrap();
let b = TestAgent::spawn_with(
|c| c.with_custom_transport("negotiator", b_transport.clone()),
&discovery,
)
.await
.unwrap();
let (name, secret) = network("private-negotiation");
let id = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
wait_for_peers(&a.agent, id, 1).await;
let message = ControlMessage::Transport {
protocol: "negotiator".into(),
payload: TransportPayload::new(b"never-export-this-secret".to_vec()),
};
assert!(!format!("{message:?}").contains("never-export"));
let sender = a_transport.control.lock().unwrap().clone().unwrap();
sender
.send(
id,
b.agent.endpoint_id(),
"negotiator",
TransportPayload::new(b"never-export-this-secret".to_vec()),
)
.await
.unwrap();
wait_until("private transport message", || async {
(!b_transport.received.lock().unwrap().is_empty()).then_some(())
})
.await;
assert_eq!(
b_transport.received.lock().unwrap()[0],
(
id,
a.agent.endpoint_id(),
b"never-export-this-secret".to_vec()
)
);
let unknown = NetworkId::from_bytes([0xa5; 32]);
assert!(
sender
.send(
unknown,
b.agent.endpoint_id(),
"negotiator",
TransportPayload::new(vec![1])
)
.await
.is_err()
);
a.agent.shutdown().await;
b.agent.shutdown().await;
}
#[test]
fn private_bytes_are_bounded_on_decode_and_by_local_policy() {
let limits = tsunagi::Limits {
max_transport_payload_len: 4,
..Default::default()
};
let message = |len| ControlMessage::Transport {
protocol: "awg".into(),
payload: TransportPayload::new(vec![42; len]),
};
assert!(tsunagi::proto::message::validate(&message(4), &limits).is_ok());
assert!(tsunagi::proto::message::validate(&message(5), &limits).is_err());
let bytes = postcard::to_stdvec(&TransportPayload::new(vec![1; 4097])).unwrap();
assert!(postcard::from_bytes::<TransportPayload>(&bytes).is_err());
let bytes = postcard::to_stdvec(&message(4)).unwrap();
let decoded: ControlMessage = postcard::from_bytes(&bytes).unwrap();
assert_eq!(decoded, message(4));
}
+16 -2
View File
@@ -115,6 +115,7 @@ pub struct TcpTlsLink {
peer: EndpointId,
tx: mpsc::Sender<Bytes>,
rx: tokio::sync::Mutex<mpsc::Receiver<Bytes>>,
tasks: [tokio::task::AbortHandle; 2],
closed_notify: Arc<Notify>,
is_closed: Arc<AtomicBool>,
remote_addr: String,
@@ -136,7 +137,7 @@ impl TcpTlsLink {
let closed_flag_w = Arc::clone(&is_closed);
let closed_notify_w = Arc::clone(&closed_notify);
tokio::spawn(async move {
let writer_task = tokio::spawn(async move {
while let Some(msg) = rx_out.recv().await {
if msg.len() > MAX_FRAME_SIZE {
continue;
@@ -158,7 +159,7 @@ impl TcpTlsLink {
let closed_flag_r = Arc::clone(&is_closed);
let closed_notify_r = Arc::clone(&closed_notify);
tokio::spawn(async move {
let reader_task = tokio::spawn(async move {
let mut len_buf = [0u8; 2];
loop {
if reader.read_exact(&mut len_buf).await.is_err() {
@@ -185,6 +186,7 @@ impl TcpTlsLink {
peer,
tx: tx_out,
rx: tokio::sync::Mutex::new(rx_in),
tasks: [writer_task.abort_handle(), reader_task.abort_handle()],
closed_notify,
is_closed,
remote_addr,
@@ -192,7 +194,19 @@ impl TcpTlsLink {
}
}
impl Drop for TcpTlsLink {
fn drop(&mut self) {
self.close();
}
}
impl PacketLink for TcpTlsLink {
fn close(&self) {
self.is_closed.store(true, Ordering::Release);
self.closed_notify.notify_waiters();
for task in &self.tasks {
task.abort();
}
}
fn network(&self) -> NetworkId {
self.network
}
+75 -4
View File
@@ -21,6 +21,17 @@ const POLL_INTERVAL: Duration = Duration::from_secs(2);
/// A request from the UI to the agent.
pub(crate) enum Command {
/// Retire the exact offline-node claim selected in the devices window.
ForgetNode {
network_id: String,
peer: String,
expected_version: u64,
},
/// Replace the network's enabled protocols in preference order.
SetProtocols {
network_id: String,
protocols: Vec<String>,
},
/// Stop serving a network, or start serving it again.
SetActive { network_id: String, active: bool },
/// Turn local broadcast participation on or off for a network.
@@ -106,7 +117,13 @@ impl AgentClient {
/// Sends a command; a dropped worker simply means nothing happens.
pub(crate) fn send(&self, command: Command) {
let _ = self.tx.send(command);
// Disable protocol editing until the refreshed snapshot arrives;
// otherwise a second checkbox click could replace the first edit
// using the same stale network selection.
lock(&self.shared).busy = true;
if self.tx.send(command).is_err() {
lock(&self.shared).busy = false;
}
}
/// A copy of the latest snapshot.
@@ -142,6 +159,8 @@ async fn worker(
let Some(command) = command else { return };
run(&resolve_socket(), command, &shared, &pending_copy, &ctx).await;
refresh(&shared, &ctx).await;
lock(&shared).busy = false;
ctx.request_repaint();
}
}
}
@@ -179,11 +198,54 @@ async fn run(
ctx.request_repaint();
let outcome: Result<String, String> = match command {
Command::ForgetNode {
network_id,
peer,
expected_version,
} => ipc::forget_node(socket, &network_id, &peer, expected_version)
.await
.map(|report| {
format!(
"forgot node claim v{}; cleanup queued to {} peers",
report.record_version, report.peers_told
)
})
.map_err(|err| err.to_string()),
Command::SetProtocols {
network_id,
protocols,
} => ipc::set_protocols(socket, &network_id, protocols)
.await
.map(|names| {
format!(
"protocols: {}",
if names.is_empty() {
"none".into()
} else {
names.join(", ")
}
)
})
.map_err(|err| err.to_string()),
Command::SetActive { network_id, active } => ipc::set_active(socket, &network_id, active)
.await
.map(|report| {
let state = if report.active { "started" } else { "stopped" };
format!("{} {}", state, report.name)
if report.suspended.is_empty() {
format!("{} {}", state, report.name)
} else {
format!(
"{} {}; suspended {} until it stops",
state,
report.name,
report
.suspended
.iter()
.map(|n| format!("{} ({})", n.name, n.network_id))
.collect::<Vec<_>>()
.join(", ")
)
}
})
.map_err(|err| err.to_string()),
Command::SetBroadcast {
@@ -211,7 +273,17 @@ async fn run(
}
Command::Join { name, secret } => ipc::join_network(socket, &name, &secret)
.await
.map(|report| format!("joined {}", report.name))
.map(|report| {
format!(
"{} {}",
if report.active {
"joined"
} else {
"configured; stopped by range conflict:"
},
report.name
)
})
.map_err(|err| err.to_string()),
Command::Leave { network_id } => ipc::leave_network(socket, &network_id)
.await
@@ -236,7 +308,6 @@ async fn run(
{
let mut guard = lock(shared);
guard.busy = false;
guard.last_action = if matches!(&outcome, Ok(message) if message.is_empty()) {
None
} else {
+4 -3
View File
@@ -116,20 +116,21 @@ impl App {
.with_title(devices::title(network))
.with_app_id(crate::icon::APP_ID)
.with_icon(crate::icon::window())
.with_inner_size([940.0, 540.0])
.with_min_inner_size([480.0, 320.0]);
.with_inner_size([1440.0, 640.0])
.with_min_inner_size([800.0, 400.0]);
let agent = &self.agent;
let traffic = &self.traffic;
// The window shares the single unit setting, so its toggle and the
// main window's stay in step.
let unit = &mut self.state.unit;
let confirm = &mut self.state.exit_confirm;
let forget_confirm = &mut self.state.forget_confirm;
let keep = ctx.show_viewport_immediate(viewport_id, builder, |vctx, _class| {
devices::show(
vctx,
agent,
unit,
confirm,
(confirm, forget_confirm),
network,
&report.endpoint_id,
traffic,
+418 -79
View File
@@ -27,6 +27,8 @@ pub(crate) fn title(network: &NetworkReport) -> String {
/// One row of the device table, built from a tunnel and/or a signed member.
struct Device<'a> {
/// Observed claim version, if this row came from signed membership.
record_version: Option<u64>,
/// The agent's own answer to whether it can be reached by any path.
online: bool,
/// How packets reach it.
@@ -42,9 +44,11 @@ struct Device<'a> {
copy: Option<String>,
/// The full endpoint id, which clicking the short form copies.
id: &'a str,
proto: &'a str,
/// End-to-end tunnel protocol, independent of the path carrying it.
tunnel_protocol: &'a str,
address: Option<String>,
handshake_secs: Option<u64>,
control_rtt_ms: Option<u64>,
overlay: Option<&'a OverlayPeerReport>,
/// It is connected and offers to be an exit node.
exit_node: bool,
@@ -64,6 +68,15 @@ pub(crate) struct ExitConfirm {
main: bool,
}
/// An exact offline claim selected for durable cleanup.
pub(crate) struct ForgetConfirm {
network_id: String,
peer: String,
version: u64,
name: String,
address: Option<String>,
}
impl ExitConfirm {
/// A question about stopping the exit node a network's tile shows.
pub(crate) fn stop_from_tile(network: &NetworkReport, peer_id: &str) -> Self {
@@ -108,8 +121,7 @@ fn name_cell(ui: &mut egui::Ui, device: &Device<'_>) {
if let Some((name, holder)) = &device.name_taken {
ui.label(egui::RichText::new(egui_phosphor::regular::WARNING).color(AMBER))
.on_hover_text(format!(
"`{name}` was claimed first by {holder}. This device works by address, \
but nobody finds it by that name."
"Name `{name}` belongs to {holder}. Use this device's address."
));
}
});
@@ -120,11 +132,12 @@ pub(crate) fn show(
ctx: &egui::Context,
agent: &AgentClient,
unit: &mut Unit,
confirm: &mut Option<ExitConfirm>,
confirmations: (&mut Option<ExitConfirm>, &mut Option<ForgetConfirm>),
network: &NetworkReport,
own_id: &str,
traffic: &Traffic,
) {
let (confirm, forget_confirm) = confirmations;
egui::CentralPanel::default().show(ctx, |ui| {
ui.horizontal(|ui| {
ui.heading(&network.name);
@@ -157,10 +170,68 @@ pub(crate) fn show(
egui::ScrollArea::both()
.auto_shrink([false, false])
.show(ui, |ui| {
device_table(ui, unit, network, &devices, traffic, confirm);
device_table(
ui,
unit,
network,
&devices,
traffic,
confirm,
forget_confirm,
);
});
});
confirm_exit(ctx, agent, confirm, Some(&network.network_id));
confirm_forget(ctx, agent, forget_confirm, &network.network_id);
}
fn confirm_forget(
ctx: &egui::Context,
agent: &AgentClient,
confirm: &mut Option<ForgetConfirm>,
network: &str,
) {
let Some(pending) = confirm.as_ref().filter(|c| c.network_id == network) else {
return;
};
egui::Area::new(egui::Id::new("forget node backdrop"))
.order(egui::Order::Middle)
.fixed_pos(egui::Pos2::ZERO)
.show(ctx, |ui| {
let screen = ctx.screen_rect();
ui.allocate_response(screen.size(), egui::Sense::click_and_drag());
ui.painter()
.rect_filled(screen, 0.0, egui::Color32::from_black_alpha(140));
});
let (mut accepted, mut cancelled) = (false, false);
egui::Window::new("Forget offline node")
.order(egui::Order::Foreground)
.collapsible(false)
.resizable(false)
.anchor(egui::Align2::CENTER_CENTER, [0.0, 0.0])
.show(ctx, |ui| {
ui.set_max_width(440.0);
ui.label(format!("Forget {}?", pending.name));
ui.monospace(&pending.peer);
if let Some(address) = &pending.address {
ui.label(format!("IP: {address}"));
}
ui.label("The network will forget this claim and free its name and IP. A returning device can publish a new claim; this does not ban it.");
ui.horizontal(|ui| {
accepted = ui.add_enabled(!agent.snapshot().busy, egui::Button::new("Forget")).clicked();
cancelled = ui.button("Cancel").clicked();
});
});
if accepted {
agent.send(Command::ForgetNode {
network_id: pending.network_id.clone(),
peer: pending.peer.clone(),
expected_version: pending.version,
});
}
if accepted || cancelled || ctx.input(|i| i.key_pressed(egui::Key::Escape)) {
*confirm = None;
}
}
/// The confirmation for an exit-node choice, drawn in the window it was
@@ -353,6 +424,7 @@ fn devices<'a>(network: &'a NetworkReport, own_id: &str) -> Vec<Device<'a>> {
});
let hostname = hostnames.get(id).copied();
Device {
record_version: member.map(|m| m.record_version),
online,
path,
via_name,
@@ -360,11 +432,16 @@ fn devices<'a>(network: &'a NetworkReport, own_id: &str) -> Vec<Device<'a>> {
hostname,
copy: hostname.map(|h| format!("{h}.{}", network.name)),
id,
proto: overlay.map_or("—", |p| p.protocol.as_str()),
tunnel_protocol: overlay.map_or("—", |p| p.protocol.as_str()),
address: overlay
.and_then(|p| p.address.clone())
.or_else(|| member.and_then(|m| m.overlay_address_v4.clone())),
handshake_secs: overlay.and_then(|p| p.handshake_secs_ago),
control_rtt_ms: network
.peers
.iter()
.find(|p| p.endpoint_id == id)
.and_then(|p| p.rtt_ms),
overlay,
exit_node: online
&& network
@@ -386,16 +463,17 @@ fn devices<'a>(network: &'a NetworkReport, own_id: &str) -> Vec<Device<'a>> {
/// The widest value each column is expected to hold, so the table starts out
/// wide enough and does not reflow as numbers grow.
const COLUMN_SAMPLES: [&str; 9] = [
"",
const COLUMN_SAMPLES: [&str; 10] = [
"device",
"exit",
"tcp-tls",
"end-to-end",
"999d / 9999 ms",
"255.255.255.255",
"23h 59m",
"direct over tcp-tls",
"99999/99999",
"1023.9 MB / 1023.9 MB",
"1023.9 kB/s",
"",
"plot",
];
/// Column widths that only ever grow, kept across frames.
@@ -404,11 +482,11 @@ const COLUMN_SAMPLES: [&str; 9] = [
/// gets shorter — `14.4 MB/s` giving way to `382 pkt/s` — would pull the
/// column in and push every cell after it. Remembering the widest seen keeps
/// the table where it is.
fn column_widths(ui: &egui::Ui, id: egui::Id) -> [f32; 9] {
let stored: Option<[f32; 9]> = ui.memory(|m| m.data.get_temp(id));
fn column_widths(ui: &egui::Ui, id: egui::Id) -> [f32; 10] {
let stored: Option<[f32; 10]> = ui.memory(|m| m.data.get_temp(id));
stored.unwrap_or_else(|| {
let font = egui::TextStyle::Body.resolve(ui.style());
let mut widths = [0.0; 9];
let mut widths = [0.0; 10];
for (width, sample) in widths.iter_mut().zip(COLUMN_SAMPLES) {
if !sample.is_empty() {
*width = ui.fonts(|fonts| {
@@ -424,13 +502,24 @@ fn column_widths(ui: &egui::Ui, id: egui::Id) -> [f32; 9] {
}
/// One cell, at least as wide as its column has ever been.
fn cell(ui: &mut egui::Ui, column: usize, widths: &mut [f32; 9], add: impl FnOnce(&mut egui::Ui)) {
fn cell(ui: &mut egui::Ui, column: usize, widths: &[f32; 10], add: impl FnOnce(&mut egui::Ui)) {
let floor = widths[column];
let response = ui.scope(|ui| {
ui.set_min_width(floor);
add(ui);
});
widths[column] = widths[column].max(response.response.rect.width());
ui.allocate_ui_with_layout(
egui::vec2(floor, 0.0),
egui::Layout::top_down(egui::Align::Min),
|ui| {
ui.set_min_width(floor);
ui.set_max_width(floor);
add(ui);
},
);
}
fn stretch_columns(mut widths: [f32; 10], available: f32) -> [f32; 10] {
let gaps = 10.0 * (widths.len() - 1) as f32;
widths[9] = widths[9].max(100.0);
widths[9] += (available - gaps - widths.iter().sum::<f32>()).max(0.0);
widths
}
/// The exit-node mark of a device: shown when it offers to be one, bright
@@ -456,10 +545,7 @@ fn exit_cell(
};
(color, "in use as your exit node — click to stop")
} else {
(
ui.visuals().text_color(),
"offers to be an exit node — click to send all your internet traffic through it",
)
(ui.visuals().text_color(), "Use this exit node")
};
let response = ui
.add(
@@ -492,6 +578,7 @@ fn device_table(
devices: &[Device<'_>],
traffic: &Traffic,
confirm: &mut Option<ExitConfirm>,
forget_confirm: &mut Option<ForgetConfirm>,
) {
let network_id = network.network_id.as_str();
if devices.is_empty() {
@@ -500,69 +587,147 @@ fn device_table(
}
let widths_id = egui::Id::new(("devices column widths", network_id));
let mut widths = column_widths(ui, widths_id);
let font = egui::TextStyle::Body.resolve(ui.style());
for device in devices {
let name = format!(
"{} ({}){}",
device.name(),
short_id(device.id),
if !device.online { " Forget" } else { "" }
);
let natural = ui.fonts(|fonts| {
fonts
.layout_no_wrap(name, font.clone(), egui::Color32::WHITE)
.size()
.x
});
widths[0] = widths[0].max(natural + 24.0);
}
ui.memory_mut(|m| m.data.insert_temp(widths_id, widths));
let widths = stretch_columns(widths, ui.available_width());
egui::Grid::new("devices")
.num_columns(9)
.num_columns(10)
.min_col_width(0.0)
.striped(true)
.spacing([10.0, 4.0])
.show(ui, |ui| {
for (column, header) in [
"device", "exit", "proto", "address", "link", "pkts", "bytes", "rate", "plot",
"device",
"exit",
"end-to-end",
"handshake / RTT",
"address",
"data path",
"pkts",
"bytes",
"rate",
"plot",
]
.into_iter()
.enumerate()
{
cell(ui, column, &mut widths, |ui| {
ui.weak(header);
cell(ui, column, &widths, |ui| {
let response = ui.weak(header);
match column {
2 => {
response.on_hover_text("End-to-end encryption.");
}
3 => {
response.on_hover_text("Handshake age / control-connection RTT.");
}
5 => {
response.on_hover_text("Transport carrying the encrypted tunnel.");
}
_ => {}
}
});
}
ui.end_row();
for device in devices {
cell(ui, 0, &mut widths, |ui| name_cell(ui, device));
cell(ui, 1, &mut widths, |ui| {
cell(ui, 0, &widths, |ui| {
ui.horizontal(|ui| {
name_cell(ui, device);
if !device.online
&& !device.in_use
&& let Some(version) = device.record_version
&& ui
.add_enabled(network.active, egui::Button::new("Forget"))
.on_hover_text("Free this offline device's name and IPv4")
.clicked()
{
*forget_confirm = Some(ForgetConfirm {
network_id: network.network_id.clone(),
peer: device.id.into(),
version,
name: device.name(),
address: device.address.clone(),
});
}
});
});
cell(ui, 1, &widths, |ui| {
exit_cell(ui, device, network_id, confirm);
});
cell(ui, 2, &mut widths, |ui| {
ui.label(device.proto);
cell(ui, 2, &widths, |ui| {
ui.push_id((device.id, "end-to-end"), |ui| {
tunnel_cell(ui, device);
});
});
cell(ui, 3, &mut widths, |ui| match &device.address {
cell(ui, 3, &widths, |ui| {
ui.label(handshake_rtt_text(
device.handshake_secs,
device.control_rtt_ms,
displayed_protocol(device.tunnel_protocol) != "—",
))
.on_hover_text(if device.tunnel_protocol == "tcp-tls" {
"Session age / control RTT."
} else {
"Last handshake age / control RTT."
});
});
cell(ui, 4, &widths, |ui| match &device.address {
Some(address) => format::copy_field(ui, address, address),
None => {
ui.weak("—");
}
});
cell(ui, 4, &mut widths, |ui| link_cell(ui, device));
cell(ui, 5, &widths, |ui| {
ui.push_id((device.id, "data-path"), |ui| data_path_cell(ui, device));
});
match device.overlay.filter(|_| device.online) {
Some(peer) => {
let series = traffic.peer(network_id, &peer.public_key);
cell(ui, 5, &mut widths, |ui| {
cell(ui, 6, &widths, |ui| {
ui.label(format!("{}/{}", peer.tx_packets, peer.rx_packets));
});
cell(ui, 6, &mut widths, |ui| {
cell(ui, 7, &widths, |ui| {
ui.label(format!(
"{} / {}",
format::bytes(peer.tx_bytes as f64),
format::bytes(peer.rx_bytes as f64)
));
});
cell(ui, 7, &mut widths, |ui| {
cell(ui, 8, &widths, |ui| {
ui.add(
egui::Label::new(format::series_rate(series, unit))
.wrap_mode(egui::TextWrapMode::Extend),
);
});
ui.allocate_ui(egui::vec2(100.0, 20.0), |ui| {
ui.allocate_ui(egui::vec2(widths[9], 20.0), |ui| {
format::sparkline(ui, series, unit, 18.0);
});
}
None => {
for column in 5..8 {
cell(ui, column, &mut widths, |ui| {
for column in 6..9 {
cell(ui, column, &widths, |ui| {
ui.weak("—");
});
}
ui.weak("—");
cell(ui, 9, &widths, |ui| {
ui.weak("—");
});
}
}
ui.end_row();
@@ -577,25 +742,25 @@ fn device_table(
tx_bytes += peer.tx_bytes;
rx_bytes += peer.rx_bytes;
}
cell(ui, 0, &mut widths, |ui| {
cell(ui, 0, &widths, |ui| {
ui.strong("total");
});
for column in 1..5 {
cell(ui, column, &mut widths, |ui| {
for column in 1..6 {
cell(ui, column, &widths, |ui| {
ui.label("");
});
}
cell(ui, 5, &mut widths, |ui| {
cell(ui, 6, &widths, |ui| {
ui.strong(format!("{tx_packets}/{rx_packets}"));
});
cell(ui, 6, &mut widths, |ui| {
cell(ui, 7, &widths, |ui| {
ui.strong(format!(
"{} / {}",
format::bytes(tx_bytes as f64),
format::bytes(rx_bytes as f64)
));
});
cell(ui, 7, &mut widths, |ui| {
cell(ui, 8, &widths, |ui| {
ui.add(
egui::Label::new(
egui::RichText::new(format::series_rate(traffic.network(network_id), unit))
@@ -606,48 +771,83 @@ fn device_table(
});
ui.end_row();
});
ui.memory_mut(|m| m.data.insert_temp(widths_id, widths));
}
/// How a device is reached, and how long its tunnel has been up.
/// A protocol name as it should appear in the table.
fn displayed_protocol(protocol: &str) -> &str {
if protocol.is_empty() { "—" } else { protocol }
}
/// The end-to-end tunnel and its own health, without transport details.
fn handshake_rtt_text(handshake: Option<u64>, rtt: Option<u64>, has_tunnel: bool) -> String {
let age = handshake.map_or_else(
|| {
if has_tunnel {
"waiting".into()
} else {
"—".into()
}
},
format::age,
);
let rtt = rtt.map_or_else(|| "—".into(), |value| format!("{value} ms"));
format!("{age} / {rtt}")
}
fn tunnel_hint(protocol: &str) -> String {
let name = displayed_protocol(protocol);
if name == "tcp-tls" {
format!("{name}: Noise end-to-end encryption")
} else {
format!("{name}: WireGuard end-to-end encryption")
}
}
fn tunnel_cell(ui: &mut egui::Ui, device: &Device<'_>) {
let protocol = displayed_protocol(device.tunnel_protocol);
if protocol == "—" {
ui.weak(protocol);
return;
}
ui.label(protocol).on_hover_text(tunnel_hint(protocol));
}
/// Current route text and whether that route is the preferred direct form.
fn data_path_text(
path: Option<&DataPathReport>,
via_name: Option<&str>,
online: bool,
) -> (String, bool) {
if !online {
return ("offline".into(), false);
}
match path {
Some(DataPathReport::Direct { transport }) => (format!("direct over {transport}"), true),
Some(DataPathReport::Relayed { hops, .. }) => (
format!("via {} · {hops} hops", via_name.unwrap_or("another device")),
false,
),
Some(DataPathReport::None) | None => ("no data path yet".into(), false),
}
}
/// How a device is reached right now.
///
/// A device reached through others is online, over a longer path, and says
/// so: the colour is the only difference in how it is treated.
fn link_cell(ui: &mut egui::Ui, device: &Device<'_>) {
if !device.online {
ui.label("offline");
return;
}
let (path, good) = match device.path {
Some(DataPathReport::Direct { transport }) => (format!("direct · {transport}"), true),
Some(DataPathReport::Relayed { hops, .. }) => (
format!(
"via {} · {hops} hops",
device.via_name.as_deref().unwrap_or("another device")
),
false,
),
Some(DataPathReport::None) | None => ("no data path".to_string(), false),
};
let age = device.handshake_secs.map(format::age);
let text = match age {
Some(age) => format!("{path} · {age}"),
None => path,
};
fn data_path_cell(ui: &mut egui::Ui, device: &Device<'_>) {
let (text, good) = data_path_text(device.path, device.via_name.as_deref(), device.online);
let response = if good {
ui.label(text)
} else {
ui.label(egui::RichText::new(text).color(AMBER))
};
if !good {
response.on_hover_text(match device.path {
Some(DataPathReport::Relayed { .. }) => {
"no direct link yet: traffic goes through another device, and moves to a \
direct link by itself as soon as one comes up"
}
_ => "connected, but no path carries packets yet",
});
}
response.on_hover_text(match device.path {
Some(DataPathReport::Direct { .. }) => "Direct transport; no mesh relay.",
Some(DataPathReport::Relayed { .. }) => "Traffic forwarded through another device.",
_ if device.online => "Control connected; data transport unavailable.",
_ => "Unreachable.",
});
}
fn yes_no(value: bool) -> &'static str {
@@ -662,6 +862,7 @@ mod tests {
fn member(id: &str, online: bool, path: DataPathReport) -> MemberReport {
MemberReport {
record_version: 1,
endpoint_id: id.into(),
online,
path,
@@ -726,4 +927,142 @@ mod tests {
.is_none()
);
}
#[test]
fn tunnel_and_direct_transport_are_named_as_different_layers() {
assert_eq!(displayed_protocol("wg"), "wg");
assert_eq!(tunnel_hint("wg"), "wg: WireGuard end-to-end encryption");
assert_eq!(
data_path_text(
Some(&DataPathReport::Direct {
transport: "tcp-tls".into(),
}),
None,
true,
),
("direct over tcp-tls".into(), true)
);
assert_eq!(
tunnel_hint("tcp-tls"),
"tcp-tls: Noise end-to-end encryption"
);
}
#[test]
fn incomplete_and_longer_paths_stay_explicit() {
assert_eq!(handshake_rtt_text(None, Some(25), true), "waiting / 25 ms");
assert_eq!(
data_path_text(
Some(&DataPathReport::Relayed {
hops: 2,
via: "near".into(),
}),
Some("iris"),
true,
),
("via iris · 2 hops".into(), false)
);
assert_eq!(
data_path_text(Some(&DataPathReport::None), None, true),
("no data path yet".into(), false)
);
assert_eq!(data_path_text(None, None, false), ("offline".into(), false));
}
#[test]
fn handshake_rtt_has_no_invented_values_and_uses_control_measurements() {
use tsunagi::ipc::{OverlayReport, PeerReport};
let network = NetworkReport {
peers: vec![PeerReport {
endpoint_id: "peer".into(),
rtt_ms: Some(27),
..Default::default()
}],
overlay: Some(OverlayReport {
peers: vec![OverlayPeerReport {
endpoint_id: "peer".into(),
protocol: "wg".into(),
handshake_secs_ago: Some(5),
..Default::default()
}],
..Default::default()
}),
..Default::default()
};
let rows = devices(&network, "own");
assert_eq!(
handshake_rtt_text(rows[0].handshake_secs, rows[0].control_rtt_ms, true),
"5s / 27 ms"
);
assert_eq!(handshake_rtt_text(None, None, false), "— / —");
assert_eq!(rows[0].tunnel_protocol, "wg");
assert!(!tunnel_hint(rows[0].tunnel_protocol).contains("wg-quic"));
}
#[test]
fn resized_table_fills_the_window_without_growing_fixed_columns() {
let minimum = [100.0; 10];
let wide = stretch_columns(minimum, 1440.0);
assert!((wide.iter().sum::<f32>() + 90.0 - 1440.0).abs() < 0.1);
assert_eq!(&wide[..9], &minimum[..9]);
let narrow = stretch_columns(minimum, 800.0);
assert_eq!(
narrow, minimum,
"narrow windows scroll rather than clipping columns"
);
assert_eq!(
stretch_columns(minimum, 1440.0),
wide,
"a resize does not accumulate padding"
);
}
#[test]
fn rendered_table_uses_window_width_without_an_expanding_device_cell() {
let context = egui::Context::default();
let network = NetworkReport {
members: vec![MemberReport {
endpoint_id: "0123456789abcdef".into(),
hostname: Some("laptop".into()),
online: true,
..Default::default()
}],
..Default::default()
};
let rows = devices(&network, "own");
for width in [1440.0, 1200.0, 1440.0] {
for _ in 0..3 {
let input = egui::RawInput {
screen_rect: Some(egui::Rect::from_min_size(
egui::Pos2::ZERO,
egui::vec2(width, 640.0),
)),
..Default::default()
};
let _output = context.run(input, |ctx| {
egui::CentralPanel::default().show(ctx, |ui| {
let right = ui.max_rect().right();
egui::ScrollArea::both()
.auto_shrink([false, false])
.show(ui, |ui| {
device_table(
ui,
Unit::Bytes,
&network,
&rows,
&Traffic::default(),
&mut None,
&mut None,
);
});
assert!(
(ui.min_rect().right() - right).abs() < 2.0,
"table width after resize: {} vs {right}",
ui.min_rect().right()
);
});
});
}
}
}
}
+1
View File
@@ -78,6 +78,7 @@ fn colour_of(health: Health) -> [u8; 3] {
match health {
Health::Disconnected => DARK_RED,
Health::Connected => BLUE,
Health::RangeConflict => [0xd9, 0x91, 0x20],
Health::ExitNode => GREEN,
}
}
+112 -10
View File
@@ -45,6 +45,8 @@ pub(crate) struct UiState {
pub(crate) about_open: bool,
/// An exit-node choice waiting for the user to confirm it.
pub(crate) exit_confirm: Option<devices::ExitConfirm>,
/// A selected offline claim waiting for cleanup confirmation.
pub(crate) forget_confirm: Option<devices::ForgetConfirm>,
}
impl Default for UiState {
@@ -56,6 +58,7 @@ impl Default for UiState {
open_devices: BTreeSet::new(),
about_open: false,
exit_confirm: None,
forget_confirm: None,
}
}
}
@@ -123,7 +126,7 @@ pub(crate) fn draw(
ui.weak("No networks yet — join one below.");
}
for network in &report.networks {
draw_network(ui, agent, state, traffic, network, &report.endpoint_id);
draw_network(ui, agent, state, traffic, network, report);
}
});
}
@@ -202,7 +205,7 @@ fn draw_network(
state: &mut UiState,
traffic: &Traffic,
network: &NetworkReport,
own_id: &str,
report: &StatusReport,
) {
ui.add_space(6.0);
egui::Frame::group(ui.style()).show(ui, |ui| {
@@ -244,21 +247,40 @@ fn draw_network(
ui.weak("peers");
ui.label(format!(
"{} online · {} known",
devices::online_count(network, own_id),
devices::known_count(network, own_id)
devices::online_count(network, &report.endpoint_id),
devices::known_count(network, &report.endpoint_id)
));
ui.end_row();
});
draw_protocols(ui, agent, network, &report.available_protocols);
format::sparkline(ui, series, state.unit, 34.0);
if !network.range_conflicts.is_empty() {
ui.colored_label(BAD, "Stopped: overlapping network ranges");
for conflict in &network.range_conflicts {
ui.label(format!("{} ({}) uses {}", conflict.name, conflict.network_id, conflict.range));
}
ui.label("Switching suspends these networks until this network stops, then resumes them automatically.");
if network.auto_start { ui.weak("This network will resume automatically when its range is free."); }
}
ui.horizontal(|ui| {
let mut active = network.active;
if ui.checkbox(&mut active, "active").changed() {
agent.send(Command::SetActive {
network_id: network.network_id.clone(),
active,
});
if !network.active && !network.range_conflicts.is_empty() {
if ui.button("Switch to this network").clicked() {
agent.send(Command::SetActive { network_id: network.network_id.clone(), active: true });
}
if network.auto_start && ui.button("Keep stopped").clicked() {
agent.send(Command::SetActive { network_id: network.network_id.clone(), active: false });
}
} else {
let mut active = network.active;
if ui.checkbox(&mut active, "active").changed() {
agent.send(Command::SetActive {
network_id: network.network_id.clone(),
active,
});
}
}
let mut broadcast = network.broadcast;
if ui.checkbox(&mut broadcast, "broadcast").changed() {
@@ -328,6 +350,86 @@ fn draw_network(
});
}
fn draw_protocols(
ui: &mut egui::Ui,
agent: &AgentClient,
network: &NetworkReport,
available: &[tsunagi::ipc::ProtocolReport],
) {
let summary = if network.protocols.is_empty() {
"none".into()
} else {
network.protocols.join(", ")
};
egui::CollapsingHeader::new(format!("protocols {summary}"))
.id_salt(("protocols", &network.network_id))
.show(ui, |ui| {
ui.weak("Preference order; changes apply immediately and are saved.");
let mut selected = network.protocols.clone();
let mut changed = false;
ui.add_enabled_ui(!agent.snapshot().busy, |ui| {
// Enabled items follow the user's preference; unchecked items
// follow the agent's catalog. The worker refreshes after edits.
let ordered: Vec<_> = network
.protocols
.iter()
.filter_map(|name| available.iter().find(|p| &p.name == name))
.chain(
available
.iter()
.filter(|p| !network.protocols.contains(&p.name)),
)
.collect();
for protocol in ordered {
ui.horizontal(|ui| {
let mut on = selected.contains(&protocol.name);
if ui.checkbox(&mut on, &protocol.name).changed() {
if on {
selected.push(protocol.name.clone());
} else {
selected.retain(|p| p != &protocol.name);
}
changed = true;
}
if let Some(index) = selected.iter().position(|p| p == &protocol.name) {
if ui
.add_enabled(index > 0, egui::Button::new(icon::ARROW_UP))
.on_hover_text("prefer earlier")
.clicked()
{
selected.swap(index, index - 1);
changed = true;
}
if ui
.add_enabled(
index + 1 < selected.len(),
egui::Button::new(icon::ARROW_DOWN),
)
.on_hover_text("prefer later")
.clicked()
{
selected.swap(index, index + 1);
changed = true;
}
}
});
ui.indent(("description", &network.network_id, &protocol.name), |ui| {
ui.weak(&protocol.summary);
});
}
});
if selected.is_empty() {
ui.weak("Control only; no IP traffic in this network.");
}
if changed {
agent.send(Command::SetProtocols {
network_id: network.network_id.clone(),
protocols: selected,
});
}
});
}
/// A prominent line about an exit node. One that carries a command copies it
/// when clicked.
fn draw_exit_warning(ui: &mut egui::Ui, warning: &exit::Warning) {
+28
View File
@@ -22,6 +22,8 @@ pub(crate) enum Health {
Disconnected,
/// Connected, nothing out of the ordinary.
Connected,
/// At least one configured network is blocked by an overlapping active range.
RangeConflict,
/// This device's traffic leaves through an exit node, and that works.
ExitNode,
}
@@ -33,6 +35,13 @@ impl Health {
let Some(report) = report else {
return Self::Disconnected;
};
if report
.networks
.iter()
.any(|network| network.auto_start && !network.range_conflicts.is_empty())
{
return Self::RangeConflict;
}
if !report
.networks
.iter()
@@ -56,6 +65,7 @@ impl Health {
match self {
Self::Disconnected => "tsunagi: not connected",
Self::Connected => "tsunagi: connected",
Self::RangeConflict => "tsunagi: network range conflict; open to switch networks",
Self::ExitNode => "tsunagi: using an exit node",
}
}
@@ -108,6 +118,24 @@ mod tests {
}
}
#[test]
fn overlapping_networks_are_visible_even_while_other_peers_are_connected() {
let mut blocked = NetworkReport {
auto_start: true,
..Default::default()
};
blocked
.range_conflicts
.push(tsunagi::ipc::RangeConflictReport {
name: "other".into(),
network_id: "other-id".into(),
range: "10.90.0.0/16".into(),
});
let status = report(vec![network(2, ExitReport::default()), blocked]);
assert_eq!(Health::of(Some(&status)), Health::RangeConflict);
assert!(Health::RangeConflict.label().contains("switch networks"));
}
#[test]
fn no_agent_or_nobody_connected_is_the_red_state() {
assert_eq!(Health::of(None), Health::Disconnected);
+6 -3
View File
@@ -79,6 +79,8 @@ pub const WIREGUARD_OVERHEAD: u32 = 32;
/// Configuration of the WireGuard plugin.
#[derive(Debug, Clone)]
pub struct WireguardConfig {
/// Diagnostic protocol id when this end-to-end engine is reused by another carrier.
pub protocol: &'static str,
/// Directory for the plugin's own key store. Separate from agent state.
pub state_dir: PathBuf,
/// WireGuard keepalive, which keeps tunnels and their links warm.
@@ -100,6 +102,7 @@ impl WireguardConfig {
/// Creates a configuration rooted at `state_dir` with sensible defaults.
pub fn new(state_dir: impl Into<PathBuf>) -> Self {
Self {
protocol: WIREGUARD_PROTOCOL,
state_dir: state_dir.into(),
keepalive: Some(25),
mtu: DEFAULT_MTU,
@@ -394,7 +397,7 @@ impl Worker {
fn report(&self, network: NetworkId, reason: impl std::fmt::Display) {
tracing::warn!(network = %network.fmt_short(), %reason, "wireguard plugin error");
if let Some(context) = self.context.get() {
context.report_error(network, WIREGUARD_PROTOCOL, reason.to_string());
context.report_error(network, self.config.protocol, reason.to_string());
}
}
@@ -647,7 +650,7 @@ async fn run(worker: Arc<Worker>, mut commands: mpsc::Receiver<Command>) {
impl IpPlugin for WireguardPlugin {
fn protocol_id(&self) -> &str {
WIREGUARD_PROTOCOL
self.worker.config.protocol
}
fn protocol_version(&self) -> u16 {
@@ -686,7 +689,7 @@ impl IpPlugin for WireguardPlugin {
// Identity only. Where to send packets is the transport's business.
let announcement = WgAnnouncement::new(network, &state.key.public());
Ok(Some(PluginCapability {
protocol: WIREGUARD_PROTOCOL.to_string(),
protocol: self.worker.config.protocol.to_string(),
version: crate::announcement::ANNOUNCEMENT_VERSION,
enabled: true,
data: announcement.encode()?,
@@ -277,6 +277,7 @@ impl tsunagi::ipc::ReportSource for Control {
network_id: network_id.to_string(),
already_configured: already,
name_shared_with: None,
active: self.0.is_active(network_id).await,
})
})
}
+29 -3
View File
@@ -109,6 +109,30 @@ impl WgAgent {
.await
}
async fn spawn_auto(discovery: &SharedMemoryDiscovery, tag: &str) -> Self {
let dir = TempDir::new().unwrap();
let tuns = MemoryTunFactory::new();
let plugin = WireguardPlugin::open(
WireguardConfig::new(dir.path().join("wireguard"))
.with_reconcile(Duration::from_millis(20), Duration::from_millis(250)),
)
.await
.unwrap();
let agent = Agent::spawn(
config_with(dir.path(), discovery)
.with_interface(Arc::new(tuns.clone()), tag, 1280)
.with_plugin(plugin.clone() as Arc<dyn IpPlugin>),
)
.await
.unwrap();
Self {
dir,
agent,
plugin,
tuns,
}
}
/// Starts an agent with an explicit overlay range, or none at all.
async fn spawn_range(
discovery: &SharedMemoryDiscovery,
@@ -989,7 +1013,7 @@ async fn two_networks_share_one_interface_with_keys_and_ranges_of_their_own() {
let (name_b, secret_b) = network("wg-right");
let beta_range = Some("10.99.0.0/16".parse::<Ipv4Range>().unwrap());
let hub = WgAgent::spawn(&discovery, "th").await;
let hub = WgAgent::spawn_auto(&discovery, "th").await;
let left = WgAgent::spawn(&discovery, "tl").await;
let right = WgAgent::spawn_range(&discovery, "tr", beta_range).await;
@@ -1661,8 +1685,10 @@ async fn lan_broadcast_fanout_is_scoped_opt_in_and_does_not_reflood() {
bytes[28..].copy_from_slice(b"LAN GAME");
// UDP checksum zero is valid for IPv4.
let sum: u32 = bytes[..20]
.chunks_exact(2)
.map(|w| u16::from_be_bytes([w[0], w[1]]) as u32)
.as_chunks::<2>()
.0
.iter()
.map(|w| u16::from_be_bytes(*w) as u32)
.sum();
let checksum = !((sum & 0xffff) + (sum >> 16)) as u16;
bytes[10..12].copy_from_slice(&checksum.to_be_bytes());
@@ -231,6 +231,9 @@ impl WireguardLink {
}
impl PacketLink for WireguardLink {
fn close(&self) {
WireguardLink::close(self);
}
fn network(&self) -> NetworkId {
self.network
}
+1
View File
@@ -14,6 +14,7 @@ path = "src/main.rs"
[dependencies]
tsunagi-core = { path = "../tsunagi-core", version = "0.1.0", features = ["tun-device", "dns-publish"] }
tsunagi-wireguard = { path = "../tsunagi-wireguard", version = "0.1.0" }
tsunagi-awg = { path = "../tsunagi-awg", version = "0.1.0" }
tsunagi-wg-quic = { path = "../tsunagi-wg-quic", version = "0.1.0" }
tsunagi-tcp-tls = { path = "../tsunagi-tcp-tls", version = "0.1.0" }
iroh.workspace = true
+624 -139
View File
File diff suppressed because it is too large. Load diff
+435 -7
View File
@@ -6,7 +6,7 @@
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use std::time::Duration;
use std::time::{Duration, Instant};
use tempfile::TempDir;
@@ -50,8 +50,26 @@ fn read_networks(dir: &TempDir) -> Vec<tsunagi::storage::StoredNetwork> {
/// An agent with no network at all, the way a daemon is started before
/// anything has been decided.
fn start_bare(port: u16) -> Running {
start_bare_range(port, None)
}
fn start_bare_range(port: u16, range: Option<&str>) -> Running {
start_bare_config(port, range, None)
}
fn start_bare_config(port: u16, range: Option<&str>, protocols: Option<&str>) -> Running {
start_bare_peers(port, range, protocols, &[])
}
fn start_bare_peers(
port: u16,
range: Option<&str>,
protocols: Option<&str>,
peers: &[String],
) -> Running {
let dir = TempDir::new().unwrap();
let child = std::process::Command::new(env!("CARGO_BIN_EXE_tsng"))
let mut command = std::process::Command::new(env!("CARGO_BIN_EXE_tsng"));
command
.arg("up")
.arg("--state-dir")
.arg(dir.path().join("state"))
@@ -62,11 +80,37 @@ fn start_bare(port: u16) -> Running {
.arg(format!("127.0.0.1:{port}"))
.args(["--log", "error", "--status-interval", "0"])
.stdout(std::process::Stdio::null())
.stderr(std::process::Stdio::null())
.spawn()
.expect("the agent binary starts");
std::thread::sleep(Duration::from_secs(2));
Running { child, dir }
.stderr(std::process::Stdio::null());
if let Some(range) = range {
command.args(["--ipv4-range", range]);
}
if let Some(protocols) = protocols {
command.args(["--protocol", protocols]);
}
for peer in peers {
command.args(["--peer", peer]);
}
let child = command.spawn().expect("the agent binary starts");
let mut agent = Running { child, dir };
let runtime = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.unwrap();
let socket = tsunagi::ipc::control_socket_path(&agent.dir.path().join("state"));
runtime.block_on(async {
tokio::time::timeout(Duration::from_secs(30), async {
while !tsunagi::ipc::is_serving(&socket).await {
assert!(
agent.child.try_wait().unwrap().is_none(),
"agent exited before becoming ready"
);
tokio::time::sleep(Duration::from_millis(25)).await;
}
})
.await
.expect("agent becomes ready");
});
agent
}
/// An agent already in one network, the way most of these start.
@@ -87,6 +131,254 @@ fn start(network: &str, port: u16) -> Running {
agent
}
#[test]
fn cli_forgets_only_the_selected_offline_claim_over_real_control() {
let cleaner = start("forget-cli", 0);
let runtime = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.unwrap();
let socket = tsunagi::ipc::control_socket_path(&cleaner.dir.path().join("state"));
let read = || {
runtime
.block_on(tsunagi::ipc::request_status(&socket))
.unwrap()
};
let before = read();
let bootstrap = format!("{}@{}", before.endpoint_id, before.bound_sockets[0]);
let target = start_bare_peers(0, None, Some("none"), &[bootstrap]);
assert!(
target
.run(&["id", "hostname", "old-cli-node"])
.status
.success()
);
assert!(
target
.run(&[
"join",
"--network",
"forget-cli",
"--secret",
"a-secret-for-the-cli-test"
])
.status
.success()
);
runtime.block_on(async {
tokio::time::timeout(Duration::from_secs(30), async {
loop {
let status = tsunagi::ipc::request_status(&socket).await.unwrap();
if status.networks[0]
.members
.iter()
.any(|m| m.hostname.as_deref() == Some("old-cli-node") && m.online)
{
break;
}
tokio::time::sleep(Duration::from_millis(25)).await;
}
})
.await
.unwrap();
});
assert!(
!cleaner
.run(&["network", "forget-node", "forget-cli", "old-cli-node"])
.status
.success()
);
assert!(
!cleaner
.run(&["network", "forget-node", "forget-cli", &before.endpoint_id])
.status
.success()
);
assert!(
target
.run(&["network", "stop", "forget-cli"])
.status
.success()
);
runtime.block_on(async {
tokio::time::timeout(Duration::from_secs(30), async {
loop {
let status = tsunagi::ipc::request_status(&socket).await.unwrap();
if status.networks[0]
.members
.iter()
.any(|m| m.hostname.as_deref() == Some("old-cli-node") && !m.online)
{
break;
}
tokio::time::sleep(Duration::from_millis(25)).await;
}
})
.await
.unwrap();
});
let forgotten = cleaner.run(&["network", "forget-node", "forget-cli", "old-cli-node"]);
assert!(
forgotten.status.success(),
"{}",
String::from_utf8_lossy(&forgotten.stderr)
);
assert!(String::from_utf8_lossy(&forgotten.stdout).contains("newer signed claim can return"));
assert!(
!read().networks[0]
.members
.iter()
.any(|m| m.hostname.as_deref() == Some("old-cli-node"))
);
assert!(
!cleaner
.run(&["network", "forget-node", "forget-cli", "old-cli-node"])
.status
.success()
);
}
#[test]
fn protocols_absent_from_startup_can_be_enabled_and_changed_offline() {
let mut agent = start_bare_config(0, None, Some("none"));
assert!(
agent
.run(&[
"join",
"--network",
"from-control-only",
"--secret",
"a-control-only-test-secret"
])
.status
.success()
);
let initial = agent.run(&["protocols", "from-control-only"]);
assert!(initial.status.success());
assert!(String::from_utf8_lossy(&initial.stdout).contains("protocols none"));
let enabled = agent.run(&["protocols", "from-control-only", "awg"]);
assert!(
enabled.status.success(),
"{}",
String::from_utf8_lossy(&enabled.stderr)
);
assert_eq!(
read_networks(&agent.dir)[0].protocols,
Some(vec!["awg".into()])
);
agent.child.kill().unwrap();
agent.child.wait().unwrap();
let saved = agent.run(&["protocols", "from-control-only", "wg-quic,awg"]);
assert!(
saved.status.success(),
"{}",
String::from_utf8_lossy(&saved.stderr)
);
assert!(String::from_utf8_lossy(&saved.stdout).contains("agent stopped"));
assert_eq!(
read_networks(&agent.dir)[0].protocols,
Some(vec!["wg-quic".into(), "awg".into()])
);
}
#[test]
fn protocols_are_changed_live_per_network_and_saved_without_changing_the_service() {
let agent = start("protocol-choice", 0);
assert!(
agent
.run(&[
"join",
"--network",
"other-choice",
"--secret",
"other-secret-for-cli-test"
])
.status
.success()
);
let before = read_networks(&agent.dir);
let other_id = before
.iter()
.find(|n| n.name.as_str() == "other-choice")
.unwrap()
.network_id;
let changed = agent.run(&["protocols", "protocol-choice", "awg,wg-quic"]);
assert!(
changed.status.success(),
"{}",
String::from_utf8_lossy(&changed.stderr)
);
let shown = agent.run(&["protocols", "protocol-choice"]);
assert!(shown.status.success());
assert!(String::from_utf8_lossy(&shown.stdout).contains("awg,wg-quic"));
let saved = read_networks(&agent.dir);
assert_eq!(
saved
.iter()
.find(|n| n.name.as_str() == "protocol-choice")
.unwrap()
.protocols,
Some(vec!["awg".into(), "wg-quic".into()])
);
assert_eq!(
saved
.iter()
.find(|n| n.network_id == other_id)
.unwrap()
.protocols,
before
.iter()
.find(|n| n.network_id == other_id)
.unwrap()
.protocols
);
for invalid in ["unknown", "awg,awg", "none,awg"] {
assert!(
!agent
.run(&["protocols", "protocol-choice", invalid])
.status
.success()
);
}
assert!(!agent.run(&["protocols", "missing", "awg"]).status.success());
let disabled = agent.run(&["protocols", "protocol-choice", "none"]);
assert!(
disabled.status.success(),
"{}",
String::from_utf8_lossy(&disabled.stderr)
);
assert_eq!(
read_networks(&agent.dir)
.iter()
.find(|n| n.name.as_str() == "protocol-choice")
.unwrap()
.protocols,
Some(Vec::new())
);
assert!(
agent
.run(&["network", "stop", "protocol-choice"])
.status
.success()
);
assert!(
agent
.run(&["protocols", "protocol-choice", "wg-quic"])
.status
.success()
);
assert!(
agent
.run(&["network", "start", "protocol-choice"])
.status
.success()
);
assert!(
String::from_utf8_lossy(&agent.run(&["protocols", "protocol-choice"]).stdout)
.contains("wg-quic")
);
}
#[test]
fn joining_with_no_secret_makes_one_and_prints_what_to_send() {
// The hurry case: a network with somebody, for as long as it is needed,
@@ -461,3 +753,139 @@ fn an_exit_node_is_offered_per_network_only_when_asked_and_nobody_offers_by_defa
String::from_utf8_lossy(&unknown.stderr)
);
}
#[test]
fn overlapping_networks_are_reported_and_cli_start_switches_then_resumes() {
let agent = start_bare_range(0, Some("10.90.0.0/16"));
for name in ["resident", "games"] {
let joined = agent.run(&["join", "-n", name, "-s", "a-secret-for-the-conflict-test"]);
assert!(
joined.status.success(),
"{}",
String::from_utf8_lossy(&joined.stderr)
);
if name == "games" {
let text = String::from_utf8_lossy(&joined.stdout);
assert!(
text.contains("stopped because its range overlaps"),
"{text}"
);
assert!(!text.contains("it is running"), "{text}");
}
}
let runtime = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.unwrap();
let socket = tsunagi::ipc::control_socket_path(&agent.dir.path().join("state"));
let snapshot = || {
runtime
.block_on(tsunagi::ipc::request_status(&socket))
.unwrap()
};
let before = snapshot();
let resident = before
.networks
.iter()
.find(|n| n.name == "resident")
.unwrap();
let games = before.networks.iter().find(|n| n.name == "games").unwrap();
assert!(resident.active);
assert!(!games.active);
assert_eq!(games.range_conflicts[0].network_id, resident.network_id);
assert_eq!(games.range_conflicts[0].name, "resident");
for args in [vec!["status"], vec!["network"]] {
let result = agent.run(&args);
let text = String::from_utf8_lossy(&result.stdout);
assert!(text.contains("range conflict"), "{text}");
assert!(text.contains(&resident.network_id), "{text}");
assert!(text.contains("tsng network start"), "{text}");
}
let switched = agent.run(&["network", "start", "games"]);
assert!(
switched.status.success(),
"{}",
String::from_utf8_lossy(&switched.stderr)
);
let text = String::from_utf8_lossy(&switched.stdout);
assert!(
text.contains("started `games`") && text.contains("suspended `resident`"),
"{text}"
);
let after = snapshot();
assert!(
after
.networks
.iter()
.find(|n| n.name == "games")
.unwrap()
.active
);
assert!(
!after
.networks
.iter()
.find(|n| n.name == "resident")
.unwrap()
.active
);
assert!(agent.run(&["network", "stop", "games"]).status.success());
assert!(
snapshot()
.networks
.iter()
.find(|n| n.name == "resident")
.unwrap()
.active
);
assert!(agent.run(&["network", "start", "games"]).status.success());
let stop_suspended = agent.run(&["network", "stop", "resident"]);
assert!(stop_suspended.status.success());
assert!(String::from_utf8_lossy(&stop_suspended.stdout).contains("stopped `resident`"));
assert!(agent.run(&["network", "stop", "games"]).status.success());
assert!(
snapshot()
.networks
.iter()
.all(|n| !n.active && !n.auto_start)
);
}
#[test]
fn cli_defaults_derive_each_networks_range() {
let agent = start_bare(0);
for name in ["derived-one", "derived-two"] {
assert!(
agent
.run(&["join", "-n", name, "-s", "a-secret-for-the-derived-test"])
.status
.success()
);
}
let runtime = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.unwrap();
let socket = tsunagi::ipc::control_socket_path(&agent.dir.path().join("state"));
let deadline = Instant::now() + Duration::from_secs(30);
loop {
let report = runtime
.block_on(tsunagi::ipc::request_status(&socket))
.unwrap();
if report.networks.iter().all(|n| n.range.is_some()) {
assert_eq!(report.networks.len(), 2);
for network in report.networks {
let id = network.network_id.parse().unwrap();
assert!(network.active);
assert_eq!(
network.range,
Some(tsunagi::state::derived_ipv4_range(id).to_string())
);
}
break;
}
assert!(Instant::now() < deadline, "networks allocate ranges");
std::thread::sleep(Duration::from_millis(25));
}
}
+131
View File
@@ -0,0 +1,131 @@
# Pairwise Amnezia transport
The CLI's default carrier preference is `wg,awg,wg-quic,tcp-tls`. To select
AWG explicitly:
```sh
tsng up --protocol awg
tsng up --protocol wg,awg,wg-quic,tcp-tls
```
The startup list is the initial per-network preference. A saved network choice
overrides it and can enable any built-in protocol at runtime:
```sh
tsng protocols home awg,wg-quic
tsng protocols home awg
```
Changes are local to this device, persist across restarts, and retire existing
data links and end-to-end channels for that network before renegotiation.
Authenticated control sessions, membership, addresses and other networks stay
live. Removing every protocol (`none` in the CLI, all checkboxes off in the tray)
leaves the network control-only. This is different from automatic carrier
fallback, which preserves the selected end-to-end channel.
`awg` is Tsunagi AWG wire version 1. Its packet masking engine is adapted from
[RunaRoo/AmneziaWG-RS](https://github.com/RunaRoo/AmneziaWG-RS), with the pinned
source and license recorded in
[UPSTREAM.md](../crates/tsunagi-awg/UPSTREAM.md). It uses the existing boringtun
Noise/WireGuard implementation instead of upstream's Node, global executors
and OS interface machinery. Compatibility with ordinary Amnezia servers is
not implemented: this carrier connects authenticated Tsunagi members.
## Negotiation and lifetime
There is one independent UDP socket and cancellable packet-engine task per
`(network, peer)` pair. The smaller endpoint id initiates, following the same
rule as other data carriers. Both sides must announce enabled AWG wire version
1. An authenticated iroh control session is required throughout; AWG supplies
no alternate discovery or membership mechanism.
Every `open` generates new masking parameters, a header-protection key, an
outer WireGuard key pair and a PSK. The initiator sends a private `Offer` with
a random attempt id, its public key, candidate socket addresses and the shared
profile. The responder validates it, creates its own fresh key pair and socket,
and returns an `Answer` naming that attempt. Both send masked handshakes and
encrypted delivery probes to the candidates. Only authenticated WireGuard
**data**, not a STUN response or an incoming UDP address alone, verifies a path.
Negotiation payloads are bounded at 4096 bytes (a caller can lower the limit),
redacted in Debug, excluded from public message events and erased on drop.
They are never capabilities, signed records, cache entries or exported status.
At most 128 pairs are admitted per transport, with bounded command/data queues,
at most 32 candidates per attempt and finite establishment deadlines.
Stale answers and duplicate offers are rejected; a new attempt replaces and
closes an older pair engine. Cancellation, peer departure, network deactivation
and shutdown retire sockets without waiting for a remote reader.
A process restart, a failed setup, or a complete fallback cycle returning to AWG
starts fresh negotiation. Normal WireGuard rekeying within a live attempt keeps
that attempt's masking profile. A failed pair does not reconfigure other pairs
or other networks. A full mesh of 15 members uses 14 engines per member and 105
independent pairs in total; it does not create additional system interfaces.
## Masking
Each profile randomizes disjoint H1–H4 header ranges, a 12–48-byte S1–S4 prefix,
a ChaCha20 header-protection key, handshake trailers, junk packet count/lengths,
and extra content padding. Prefix lengths are equal within one profile so
header detection has one offset. A fresh random nonce is in every packet's
prefix; handshake headers and transport headers/counters are protected.
The standard WireGuard header is restored before boringtun authenticates a
message. This preserves boringtun's standard MAC calculations, which is another
reason this is a Tsunagi-specific AWG wire version.
Before a handshake, an I1 signature packet is generated from either a complete
DNS query or SIP OPTIONS template, with fresh identifiers and consistent length
fields, followed by random junk packets. I2–I5 are unused in this version.
The templates are structural generators, not replayed captures. QUIC imitation
is not selected. Masking can change DPI classification; it is not a promise of
indistinguishability or that any particular firewall will admit it.
## Packets, NAT and fallback
The pair engine carries opaque relay envelopes inside authenticated outer WG
packets. It fragments those envelopes into bounded chunks and expires incomplete
assemblies; a logical datagram is bounded at 64 KiB and an outer UDP packet is
at most 1200 bytes. The encapsulated IPv4 headers are internal framing only: no interface or OS address is created for a
pair. Overlay addresses, routing, ownership checks and end-to-end encryption
remain independent. A relay decrypts its own adjacent AWG carrier to forward
opaque end-to-end ciphertext, never the destination's IP packet.
Local IPv4 candidates and simultaneous encrypted probes provide direct UDP
connectivity. In `direct`/`relay` reach mode the CLI also uses
`stun.l.google.com:19302` to discover a server-reflexive address on **this pair's
socket**, with a 1.2-second bound. A STUN result is only a candidate. Local reach
mode makes no STUN request. Configure or disable it with
`-o awg:stun=HOST:PORT` or `-o awg:stun=off`. Failed discovery still permits local
candidates. IPv6-only paths, symmetric NAT and networks blocking arbitrary UDP
may fail AWG and fall back; there is no AWG-over-QUIC substitution or AWG TURN
relay hidden behind its name.
Data delivery checks go through the actual end-to-end encrypted tunnel, once
per second, independently of control-plane RTT. Requests and replies bind a
fresh private marker/key to the sender's endpoint id. Unanswered requests expire
after three seconds. After a 20-second warmup, at least 15 completed observations
and at least 60% loss in the last 20 observations retire a carrier. Probes are
consumed locally after source/destination admission and never enter the TUN.
AWG additionally checks encrypted outer delivery throughout the pair lifetime.
Setup failure or persistent runtime loss advances to the next mutually supported
carrier in the initiator's order. Earlier expired cooldowns do not jump ahead of
untried carriers. Retry cooldowns remain bounded; reaching AWG again negotiates
new parameters. Control sessions and the logical end-to-end tunnel survive a
carrier change. Status names the end-to-end protocol and the actual carrier
separately, so `wg` over `awg` is a valid result.
Control wire version 6 introduces the private transport message. All members
must upgrade together; device identities, network ids, handshake transcript
encoding, signed state and stored network configuration are unchanged. Data
ALPN remains `tsunagi/data/5`; the relay envelope is unchanged.
## Verification
The default rootless, offline tests use real iroh endpoints, membership proofs,
SQLite, UDP sockets, masked WireGuard handshakes, end-to-end encryption and
memory TUNs. They cover independent three-member pairs, full-size bidirectional
packets, restart freshness/socket release, 80% runtime datagram loss, fallback
to real iroh and fresh negotiation on retry, plus malformed profiles/messages,
role checks, probe forgery and STUN response parsing. These tests do not establish
DPI effectiveness, WAN throughput, public NAT traversal or per-pair memory cost.
+8 -2
View File
@@ -20,8 +20,9 @@ network secret. The core moves an opaque, bounded payload and never parses it.
a `PacketLink` — an authenticated, unreliable datagram channel to one peer for
one protocol — and never learn how it is carried.
The separation between the two planes is **logical, not physical**. Both ride
on iroh, on different ALPNs and different connections. That is deliberate:
The separation between the two planes is **logical, not physical**. The control plane uses
iroh; the data plane can use iroh on its own ALPN and connection, or independent
UDP/TLS carriers. That is deliberate:
iroh's whole value is hole punching a direct path between peers behind NAT,
with a relay as fallback, and a data plane that refused to use it would have to
reimplement all of it. What the separation buys is that `proto` knows nothing
@@ -32,6 +33,10 @@ A plugin talks to the core through three narrow hooks — `on_network_activated`
a `PluginContext` for re-announcements and error reports, and a bounded
`shutdown` — so the core never learns anything protocol-specific.
Private pair negotiation uses a bounded transport sender supplied by the core.
Protocol crates do not encode or parse control messages and never receive the
network secret. No pair profiles enter signed state.
A data plane failure never stops the daemon: the control plane keeps running
and the agent stays manageable.
@@ -68,6 +73,7 @@ And in crates of their own:
| crate | responsibility |
|---|---|
| `tsunagi-awg` | ephemeral pairwise Amnezia masking, private negotiation and independently encrypted UDP carriers; reuses the end-to-end WireGuard engine |
| `tsunagi-wg-quic` | the `wg-quic` protocol: its keys, its announcement, its tunnels |
| `tsunagi` | the command line agent, built as the `tsng` executable: the only place that owns a runtime, a logger and signals |
+30 -6
View File
@@ -43,12 +43,36 @@ be an exit node in one network and not in another.
`tsunagi-exit:<interface>:<range>` and are removed when the offer ends, the
network stops, and when the agent exits. Only networks that offer have
rules, so the others' traffic is not forwarded.
- **Client:** policy routing, with no change to the main table. Table `28787`
holds `default dev <interface>`, and three rules sit after Tailscale's:
`5280 uidrange <agent uid> lookup main` (the agent's own connections never
go through the tunnel, which would be a loop), `5290 lookup main
suppress_prefixlength 0` (LAN and other specific routes keep working) and
`5300 lookup 28787`.
- **Client:** policy routing, with no change to the main table. Table `29811`
holds `default dev <interface>`. Rules `5205 fwmark 0x40000000/0x40000000
lookup main` and `5206 uidrange <agent uid> lookup main` keep the agent's
sockets and their replies outside the tunnel, before Tailscale's rules.
`5290 lookup main suppress_prefixlength 0` preserves LAN and other specific
routes, and `5300 lookup 29811` directs other traffic through the exit.
The exit table is `29811` (`0x7473`).
- **Linux reverse-path filtering:** only the agent's dedicated UID is marked
in mangle `OUTPUT`. Its own bit, `0x40000000`, is saved to conntrack and
restored in `PREROUTING` before the host firewall checks reply routing.
Other mark bits, including Tailscale's `0xff0000` mask, are preserved. The
rules are tagged `tsunagi-exit-underlay:<interface>` and never accept packets
around the host firewall. IPv4 and IPv6 protection is installed before exit
routes; routes are removed before protection, including rollback on failure.
Locally owned TCP listeners and bound UDP sockets also protect the first
incoming packet, before any outgoing packet has created a connection mark.
Socket addresses and ports
come from local kernel socket tables and match an actual socket; remote
announcements never supply these firewall arguments.
The socket set is refreshed before private pair negotiation is sent and on
discovery ticks, so fresh AWG ports are included. An unchanged set causes no
firewall writes; a socket-refresh failure leaves active exit routes in place.
Linux needs both `iptables` and `ip6tables` when IPv6 is present.
- **Kernel strict `rp_filter`:** if an interface uses strict mode (`1`), the
kernel must include marks in reverse lookups: set
`net.ipv4.conf.all.src_valid_mark=1`, or enable it on those interfaces.
The agent checks this before installing exit routes and reports an actionable
error if marks would be ignored. It never disables `rp_filter` or changes
host sysctls. Netfilter strict reverse checks also need their mark-aware
form, such as `rpfilter --validmark`, which NixOS already uses.
- Packets for addresses outside every overlay range are sent to the exit node
by the router; the exit node's replies are accepted from any source. A member
that is not your exit node cannot do that.
+11 -1
View File
@@ -9,11 +9,19 @@ Two versions exist and are independent:
- **Identity scheme**, `tsunagi-network-id-v1`. Frozen. Changing it creates a
different network space for the same name and secret.
- **Control protocol**, ALPN `tsunagi/ctrl/5`, `PROTOCOL_VERSION = 5`.
- **Control protocol**, ALPN `tsunagi/ctrl/7`, `PROTOCOL_VERSION = 7`.
Upgrading the crate or bumping the control protocol must never change an
existing `NetworkId`.
Control version 7 adds signed offline-node retirement messages to version 6's
private AWG transport negotiation and delivery checks. Data ALPN remains
`tsunagi/data/5`. All members must upgrade together; network ids, handshake
encoding and existing record signatures are unchanged. Local CLI/tray IPC is
version 21, so restart the agent with matching clients. SQLite schema 9 adds
retirement storage while preserving existing identities, settings and records.
See [sync-model.md](sync-model.md#offline-node-cleanup) and [amnezia.md](amnezia.md).
## Canonical encoding
Everything that is hashed or MAC'd uses length-prefixed fields, written here as
@@ -230,9 +238,11 @@ not affect other networks.
| message | meaning |
|---|---|
| `Announce { hostname, capabilities, broadcast }` | this agent's hostname, IP-plugin capabilities and local broadcast participation |
| `Transport { protocol, payload }` | private, bounded per-peer transport negotiation; never state or a public message event |
| `Ping { seq, payload }` | small request used to verify the exchange |
| `Pong { seq, payload }` | the echoed reply |
| `State { records }` | a snapshot of signed records, merged into what the receiver holds |
| `Retirements { markers }` | signed cleanup markers with the target's original signed claim as proof |
| `Peers { peers }` | unverified member address hints, requiring their own handshake |
| `Reachable { links }` | sender's current direct data links, by kind of transport |
| `Bye { reason }` | graceful goodbye; not a revocation of anything |
+9 -1
View File
@@ -96,6 +96,14 @@ topology change can move a flow; datagrams remain unreliable and unordered,
and there is no promise to preserve order across a physical path failure.
Logical peer links and end-to-end encryption state survive these changes.
## Carrier health and switching
Carrier setup failures and persistent encrypted delivery loss advance the
initiator's ordered fallback cycle. Logical end-to-end links survive while
physical links are retired explicitly. The AWG carrier negotiates a fresh
private pair profile on every reentry, without changing topology's durable
state. See [amnezia.md](amnezia.md) for thresholds and bounds.
## Verification and measurement
The default offline suite exercises four real agents, iroh transports and
@@ -116,5 +124,5 @@ counts sends without storing frames. Results are CPU forwarding cost, **not**
end-to-end network latency or VPN throughput; encryption, fragmentation,
sockets, congestion and scheduling contribute separately.
Wire compatibility: control ALPN `tsunagi/ctrl/5`, data ALPN `tsunagi/data/5`.
Wire compatibility: control ALPN `tsunagi/ctrl/7`, data ALPN `tsunagi/data/5`.
Upgrade every participant together; saved identities and network state persist.
+46 -27
View File
@@ -1,18 +1,9 @@
# Planned state synchronisation
# State synchronisation
**The first slice of this model is now implemented**, in `crates/tsunagi-core/src/state/`, and is
used for one thing: IPv4 overlay addresses. What follows describes the whole
model; the section at the end says exactly which parts exist.
The proof of concept still exchanges hostname and capability announcements
over live sessions and keeps no replicated history for those, which is why
WireGuard peer *membership* remains session-scoped even though a peer's
*address* no longer is. This file records the intended direction so the module
boundaries in [architecture.md](architecture.md) stay compatible with it, and so
nobody mistakes the current announcements for synchronisation.
There is no fake "ready CRDT" here and no snapshots that are not actually
verified.
Signed claims in `crates/tsunagi-core/src/state/` preserve IPv4 reservations and
hostnames across an absence. Signed releases and offline-node cleanup markers
free those reservations durably. Capabilities, links and authenticated sessions
remain volatile; a stored claim never authenticates a live connection.
## The problem
@@ -75,6 +66,39 @@ Turning a computer off is not a revocation of its hostname and does not remove
the participant. Revocations are not dropped merely because they are old, and
no acknowledgement from offline peers is required to keep working.
## Offline-node cleanup
Any authenticated member can sign a `SignedRetirement` for another member's
observed `Claim`. It includes the original signed claim as proof and uses the
separate `tsunagi-node-retirement/v1` signing domain. Both signatures and both
network scopes are checked. No owner, vote or quorum is involved, and no member
can fabricate the target's release or invent a future version cutoff.
The strongest observed target version wins; ties use a deterministic canonical
byte order. Claims at or below that version do not occupy an address, hostname
or visible member row. Original signed records remain stored, including their
range, so cleanup does not renumber the network. Markers remain even after a
newer claim returns: absence from a snapshot is never deletion and replaying an
old snapshot cannot restore the retired claim.
The local `Agent::forget_node` API checks the exact observed version again and
refuses self, online nodes, the selected exit node and missing/changed claims.
CLI and tray share this API. A returning live node can sign a newer claim and
allocate against the current holders; it resets the old hostname priority, so
it cannot reclaim a reused name merely by keeping its old timestamp. Cleanup
is not a permanent identity ban or a change of secret.
The marker and requester's own author counter commit in one SQLite transaction
before publication. Receivers persist a verified marker before applying it.
Invalid proofs are rejected individually. Periodic bounded marker batches
rotate through each peer's send queue, including when a queue fills, so a
replica that missed an update can catch up without the requester being online.
One strongest marker per target is retained, with a default limit of 1024 per
network and 1024 bytes per encoded marker. New targets beyond that limit are
refused rather than evicting resurrection protection. Corrupt mandatory marker
storage or a configured limit smaller than the stored set is an explicit error.
## Storage requirement this creates
When signed records land, writing the event and bumping the author's own
@@ -104,9 +128,9 @@ Implemented, in `crates/tsunagi-core/src/state/`:
* the merge rules above: higher version wins, an older version never rolls
back a newer one, duplicates are idempotent, a same-version conflict is
resolved identically on every replica and reported;
* a release tombstone, which merges correctly and is not undone by a replica
that has not heard of it — though nothing emits one yet, so freeing an
address still means forgetting the network;
* a release tombstone, emitted when leaving or replacing the device key and
not undone by a replica that has not heard of it;
* durable hostnames and equal-rights offline-node retirement described above;
* persistence in `state.sqlite`, with the record and the author's own version
counter committed in **one transaction before the record is announced**;
* distribution as a `State` control message, merged into what the receiver
@@ -114,21 +138,16 @@ Implemented, in `crates/tsunagi-core/src/state/`:
* allocation of a free IPv4 address against what everybody else holds, which
is what makes an address stable across an absence.
Deliberately not implemented: compaction, revoking a whole author, record
types beyond addressing, and any bound on how large a snapshot may grow
beyond the per-message limit.
Deliberately not implemented: compaction and permanent author revocation.
Claims have a per-message bound; retirement history also has a per-network bound.
## Future tests
These are **not implemented and must not be reported as passing**:
- snapshot merge against the rules above, including conflicting same-version
records;
- revocation propagation and resistance to resurrection after compaction;
- resistance to resurrection after compaction (compaction is not implemented);
- long network partitions and rejoin after an extended absence;
- hostname rename with atomic revoke-and-announce;
- recovery of a returning participant from a replica that is not the author;
- NAT traversal and hole punching between real hosts;
- relay fallback behaviour against a self-hosted relay;
- multi-process and multi-host deployment, as opposed to several library
instances inside one test process.
- multi-host deployment. CLI tests do run separate local agent processes;
library tests run several independent instances inside one process.
+62 -4
View File
@@ -98,6 +98,13 @@ a batch does not stop the rest, and allocation is deterministic, spread out,
walks past everything taken and reports a full range instead of handing out a
duplicate.
`tests/range_conflicts.rs` checks isolated first members choosing the same
range, persistence across restart, preservation of legacy ranges, exclusion
of equal and nested ranges, late authenticated conflicts, switching all
blockers, automatic resume, cancelling a suspended network's resume and
concurrent switch requests. These use real agents, loopback handshakes and
SQLite; they do not change host network settings.
`tests/local_control.rs` covers the local control socket end to end: a client
asking a running agent for status over a real Unix socket, joining and
leaving a network through it, a leftover socket file being replaced while a
@@ -157,10 +164,9 @@ paused clock. The public Mainline/relay smoke test is ignored by default.
## Not covered, and not claimed to be
Listed in [sync-model.md](sync-model.md#future-tests): snapshots, revocations,
long partitions, hostname renames, recovery of a returning participant, NAT
traversal, relay fallback, and multi-process or multi-host deployment. None of
these are implemented, and none are marked as passing.
Listed in [sync-model.md](sync-model.md#future-tests): compaction, long partitions,
NAT traversal, relay fallback and multi-host deployment. These scenarios are
not established by the default loopback suite.
## Debugging a test
@@ -170,3 +176,55 @@ TSUNAGI_TEST_LOG=tsunagi=debug cargo test --test multi_peer -- --nocapture
The library never installs a global subscriber; the harness opts in only when
that variable is set.
The `tsunagi-awg` suite runs real pair negotiation and masked UDP WireGuard
engines with memory TUNs. A loss wrapper drops datagrams at the carrier boundary
without substituting iroh, membership, encryption or SQLite. It checks runtime
loss detection, fallback and fresh profiles, in addition to restart and input
validation. Tests are offline; public STUN/DPI/NAT effectiveness needs separate
real-network validation.
Runtime protocol selection tests switch real AWG and iroh/QUIC carriers and
end-to-end WireGuard tunnels from either endpoint-id role, carry full-size
packets, keep another active network's tunnels unchanged, disable and re-enable
the data plane without losing control sessions, and preserve selection after
restart. CLI tests use a real agent process and control socket to exercise
selection by network name, stopped networks, invalid lists, and persisted
control-only configuration, offline updates, and enabling a protocol absent
from the startup list. A real AWG test starts without a TUN, creates the single
memory interface on enablement, and races rapid changes against pending opens.
SQLite tests migrate schema 7 to 8 without changing
identity, secrets or signed records, preserve order and explicit empty choices,
and reject malformed stored selections. These checks use memory TUNs and do
not establish GUI rendering or behavior on an actual OS interface.
`tests/forget_node.rs` uses real agents, handshakes and SQLite to check equal
cleanup rights, network isolation, online/self/stale-version refusal, restart,
recovery of a stale replica, reuse of a hostname and a returning identity's
newer claim. Bounded batch delivery is checked with a two-message send queue
and twelve retained markers. Retirement unit tests check signatures, proof tampering, invented
future cutoffs, bounds, deterministic merge and resistance to stale snapshots.
SQLite tests migrate schema 8 to 9 and inject counter-write failure to prove
atomic rollback. The CLI test starts two real agent processes and exercises
offline cleanup by network and hostname through the control socket.
Native tray rendering and physical screen sizing are not exercised automatically.
Linux exit-client unit tests cover preservation of foreign mark bits, matching
only the agent's UID and locally owned TCP/UDP sockets, exact tagged cleanup,
partial IPv6 failure and mark-aware strict kernel RPF validation. The ignored
`overlay::hostrules::linux_exit_kernel::strict_firewall_preserves_underlay_and_application_exit_routing`
test creates two disposable network namespaces. It reproduces the UID-only
failure with real sockets, then runs TCP/UDP IPv4/IPv6 and first inbound TCP/UDP
packets, including a UDP port bound after activation, through real mark/conntrack
rules with both strict kernel and
Netfilter reverse-path checks. It verifies application exit routing, IPv6
blocking, repeated apply and cleanup with foreign rules at Tailscale's priority
retained. It requires root, iproute2,
iptables/ip6tables and Python; it does not run in the default suite or claim
to test the entire mesh inside these namespaces.
Devices-window tests render the egui grid at multiple widths with its real
scroll area. They check stretching and shrinking without an expanding first
column, separate handshake/control RTT values, unknown measurements and short
end-to-end hints that do not substitute the carrier protocol. This does not
establish native desktop rendering on every platform.
+25 -4
View File
@@ -26,8 +26,8 @@ Read this before relying on anything here. The protocol is in
neither receive nor forge another member's traffic. See
[wireguard.md](wireguard.md#address-ownership-is-enforced-not-announced).
- **Tunnelled traffic is end-to-end encrypted by WireGuard**, independently of
this crate. The transport underneath is also encrypted by iroh, but the
tunnel's confidentiality does not depend on that.
this crate. Carriers may be iroh, TLS, direct WG or
pairwise AWG; confidentiality does not depend on a particular carrier.
- **Resource bounds.** Frame lengths are validated before allocation; strings,
lists, queues, concurrent dials and in-flight handshakes are all bounded;
handshakes, dials and writes have timeouts.
@@ -38,6 +38,14 @@ Read this before relying on anything here. The protocol is in
arbitrarily many identities, flood the network with records and collide with
other participants' names. This is why a majority is not a root of trust.
Signatures protect authorship; they do not make a participant honest.
- **Cleanup rights are equal.** Any member can retire an observed signed claim
of another device, freeing its hostname and IPv4 reservation. A valid original
claim bounds the cutoff; a member cannot invent a future cutoff or forge the
target's release. The local API refuses online nodes, but a remote signature
cannot prove that a target was offline. A live target may publish a newer
claim; a malicious member can repeatedly disrupt reservations. Cleanup is
not expulsion or secret revocation. See
[sync-model.md](sync-model.md#offline-node-cleanup).
- **Weak secrets.** This targets high-entropy secrets. There is no PAKE, so a
short human passphrase can be guessed offline by anyone who can reach the
handshake. Use `NetworkSecret::generate()`.
@@ -55,8 +63,10 @@ Read this before relying on anything here. The protocol is in
having iroh under it does not make it so.
- **A cloned state directory is a cloned identity.** `state.sqlite` holds the
device secret key and the network secrets. Copying it copies the participant.
Restoring an old backup rolls the agent's state back, which — once signed
records exist — can resurrect revoked information or replay stale versions.
Restoring an old backup rolls the agent's state back and can replay stale
versions until it meets a replica holding the newer signed state. Retained
cleanup markers prevent old claims from resurrecting after that merge; they
cannot help if every copy of a marker has been lost.
- **A compromised host.** The secret is on disk to survive restarts. File
permissions are owner-only where the platform supports it, and the state
directory takes an ownership lock, but neither defends against a user who can
@@ -113,3 +123,14 @@ for key separation, HMAC-SHA256 for the membership proof, constant-time
verification, iroh's Ed25519 endpoint keys and QUIC/TLS for the transport, and
RFC 5705 TLS exporter output for channel binding. There is no custom encryption
layer and no custom PAKE.
## Pairwise AWG masking
AWG profiles, PSKs and outer keys are private to one authenticated pair attempt,
not shared network state. Address candidates are unverified until authenticated
outer WireGuard data arrives. Header protection and structured signature packets
reduce fixed wire markers; they do not guarantee admission by DPI or hide traffic
analysis. iroh control availability remains a prerequisite. Encrypted end-to-end
delivery probes detect sustained carrier loss, never confer membership and pass
the same signed source/destination admission as ordinary IP traffic. See
[amnezia.md](amnezia.md) for limits and tested scope.
+30 -7
View File
@@ -92,8 +92,8 @@ and then recorded as a signed fact**, using the model in
[sync-model.md](sync-model.md).
```text
default range 10.13.37.0/24 (override with --ipv4-range)
who decides the first member to claim; later ones adopt what they find
default range stable pseudorandom /24 in 10/8, derived from NetworkId
who decides all members derive the default; signed explicit ranges win
who signs the claiming member, with its persistent device key
where it is kept state.sqlite, and every replica that has seen it
what a return costs nothing: the old address is reclaimed
@@ -120,11 +120,34 @@ the same address at the same moment are resolved by a rule both compute
identically — the lower endpoint id keeps it — and the loser simply allocates
again with a higher version.
**The range is agreed, not configured per member.** `--ipv4-range` says what
this agent would use; a network that has already settled on something else
wins, and the agent adopts it. So the flag matters for whoever starts the
network and is harmless afterwards. Pass `--ipv4-range none` for an IPv6-only
overlay.
**The range is agreed, not configured per member.** New networks use the
same deterministic `/24` on every device, independently of which network a
member activates first or whether any peers are reachable. The derivation
is unchanged from the former fallback; network identity and signed record
encoding are unchanged. Distinct network ids can still derive the same range.
`--ipv4-range` proposes an explicit range; existing signed ranges take
precedence over the new derived default, including the former `10.13.37.0/24`.
Among disagreeing non-default ranges the lowest author id still wins. Updated
members propagate their adopted range in their signed claims; older releases
still apply their original selection rule until those claims converge.
`--ipv4-range none` disables local proposals, as before.
An agent reserves each active network's range before starting its runtime.
Overlapping ranges, including nested blocks such as a `/24` inside a `/16`,
cannot run simultaneously. A late authenticated state update that introduces
an overlap suspends that network before publishing new allocations to plugins
or the interface. Other networks continue running.
Status includes every blocking network's name, id and range. `tsng network
start <name-or-id>` and the tray's switch button stop all blockers before
starting the selected network. Suspended networks keep their startup request
and resume when the blocker stops or is forgotten. A manual stop cancels that
request even for a network already suspended. The user's preference order is
stored locally, so the same selection wins at the next startup; it is never
advertised or treated as a remote network setting. Local IPC uses wire version
19 for the new reports: restart the agent and tray together after upgrading.
The peer control protocol and signed record encoding do not change.
The agent puts the address on the interface itself, as soon as the network has
agreed on it — no restart, and the interface is not recreated, which would
+1
View File
@@ -102,6 +102,7 @@ in
# Exit nodes: the agent looks for iptables only in fixed FHS directories.
systemd.tmpfiles.rules = [
"L+ /usr/bin/iptables - - - - ${pkgs.iptables}/bin/iptables"
"L+ /usr/bin/ip6tables - - - - ${pkgs.iptables}/bin/ip6tables"
];
# Names of mesh members are served locally and handed to the system resolver.