Compare commits

...
10 Commits
Author SHA1 Message Date
Aleksandr Bogomiakov 8a6216bccb Expose tray runtime libraries in the Nix development shell
The development shell inherited build dependencies but Cargo-built tray
binaries lack the installed package's wrapper. AppIndicator was present in
the store yet its dlopen lookup failed because the runtime library path was
missing. The same problem affected dynamically loaded graphics libraries.

Expose the package's existing trayRuntimeLibs through passthru and reuse that
list in mkShell packages and its LD_LIBRARY_PATH hook. Include the existing
NixOS OpenGL driver path and preserve any caller-supplied library paths.
The development shell and installed tray now share one dependency list.
Document direct execution of Cargo-built tray binaries inside nix develop.
No agent, protocol, package version or system configuration changes.

Validation on Linux:
- Reproduced the missing AppIndicator dlopen failure before the change.
- Loaded libayatana-appindicator3.so.1, libxkbcommon-x11.so.0, libEGL.so.1,
  libGL.so.1 and libwayland-client.so.0 inside the updated shell.
- Verified AppIndicator still loads with an inherited LD_LIBRARY_PATH and
  that the existing value is preserved. The first probe used an older host
  Python with an incompatible glibc; probes using Python from the pinned
  nixpkgs input passed.
- Evaluated the x86_64-linux CLI package and shared runtime dependency list,
  and the aarch64-linux development shell; no aarch64 execution claimed.
- nixfmt --check flake.nix nix/package.nix
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets: 566 passed, 3 ignored
Cargo checks ran through nix develop, with TSUNAGI_CONTROL_SOCKET unset for
Clippy and tests. Existing agent and tray processes were not restarted.
2026-10-21 17:05:25 +01:00
Aleksandr Bogomiakov 2e7e238459 Improve tray VIP monitoring and table layout
The VIP editor put routing and traffic observations in a dense text block,
while nested layouts in the devices grid could let text escape its row.
Separate the VIP overview from configuration and draw both live tables with
explicit cell bounds, font-aware row heights, clipping and horizontal overflow.
Measure columns before painting and keep widths stable as live values change.

The overview reuses the existing traffic sparklines for service totals and
individual listener backends, with TX/RX rates, health, RTT, open flows and
counters. Retain bounded histories by service/listener/backend identity and
release them when their resources disappear. Recreated listeners start fresh.

Use egui 0.31 Scene and Painter for a pannable, zoomable routing diagram with
Fit, clickable backend details and linked table selection. Highlight recent
counter growth separately from the backend selected for new connections;
idle bindings alone do not imply traffic, and stale activity expires. Draw
legend markers directly rather than depending on optional font glyphs.
The overview only reads saved observations; edits and deletion still require
Save. No dependencies, wire formats, IPC or persistent schemas change.

Validation on Linux through nix develop:
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets: 566 passed, 3 ignored
- cargo build --locked -p tsunagi-tray
The inherited TSUNAGI_CONTROL_SOCKET was unset for Clippy and tests.
Seven new GUI tests cover enlarged fonts, mixed online/offline rows, long
names, large counters, narrow/wide layouts, scene clicks, traffic selection,
stale samples, counter resets, listener recreation and history cleanup.
The first enlarged-font regression exposed an override-font height mismatch;
row sizing now resolves the actual selected font and the regression passes.
A disposable fixture preview rendered successfully with software OpenGL on
an isolated Xvfb display. Initial Xvfb attempts needed explicit xkbcommon and
OpenGL runtime paths in the Nix environment. The preview never contacted an
agent; its temporary source was removed. macOS, Windows, Wayland and live host
TUN traffic were not exercised by this UI change.
2026-10-21 16:57:27 +01:00
Aleksandr Bogomiakov 23af1a2d9d Add shared TCP and UDP virtual services
Provide named VIPs with independent protocol/port listeners and node/port
backends so applications can use one DNS endpoint without a separate load
balancer. Every authenticated member may create, edit or delete any service.
Client agents select the eligible backend with the lowest measured encrypted
data-plane RTT, including relayed paths, and pin existing flows. UDP recovers
on another backend after failure; established TCP is never replayed or moved.
An unanswered empty TCP SYN may retry another backend before establishment.

Keep signed multi-writer field registers in state/vip.rs. Independent backend
edits merge, same-field conflicts use Lamport/author/canonical ordering, and
listener generations prevent old backend entries returning on port recreation.
Permanent service tombstones protect replay while allowing obsolete fields to
be compacted. Active-service and deletion-history limits are separate. Commit
complete snapshots and the author's durable counter atomically before local
routing changes or publication. Restore counters from newer genuine author
updates learned from replicas after an old backup is restored.

Implement bounded client-side address/port NAT at overlay ingress and reverse
translation only after authenticated source/destination admission. Reserve OS
source ports to avoid direct-flow collisions and quarantine retired ports for
late replies. Handle a backend on the client itself, scoped fragment assembly,
checksums, re-fragmentation and quoted ICMP/PMTU headers. Plugins and encrypted
transit retain their existing contracts. Data-path probes measure RTT; bounded
fair TCP connect checks are enabled by default and configurable per listener.
Arbitrary UDP application readiness remains unknown; path checks and explicit
port errors drive eligibility without protocol-specific probe payloads.

Expose the shared draft API through local IPC, simple vip/listener/backend CLI
commands, JSON, editor-based atomic updates and watch output. Add a VIP button
on each network tile and a separate tray editor. Edits and removals stay local
until Save; stale revisions retain the draft, and closing a dirty window offers
explicit discard. Report local routing choices, health reasons, RTT age, active
and draining bindings, IP counters, rates, drops and UDP failovers. Extend DNS
with service names; keep numeric addresses secondary and preserve existing
bindings while VIP addresses or listeners change.

This changes control wire ALPN/version to tsunagi/ctrl/8 and local IPC to 22;
all members and local clients must upgrade together. Data ALPN remains 5.
SQLite schema 10 migrates existing mandatory state without changing identity,
secrets, network ids, derivation labels, handshake encoding or node records.
Crate versions and release tags are unchanged. No installed agent or tray was
replaced. Commands, limits and operational semantics are in docs/vip.md.

Checks on Linux in nix develop:
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets: 559 passed, 3 ignored
- git diff --cached --check

The full suite includes real encrypted TCP/UDP VIP traffic, backend-stop UDP
failover, real loopback TCP check recovery, source admission, signed merges,
stale/deleted state, SQLite migration and rollback, restart/author recovery,
network isolation, actual CLI/DNS processes, and headless text-entry/Save-button
interaction. Discovery and memory TUNs are the existing test substitutions;
authentication, transport, encryption and SQLite remain real.

Initial checks outside Nix failed for missing GTK pkg-config metadata, and an
initial CLI test inherited TSUNAGI_CONTROL_SOCKET. Final checks use the Nix
environment with that variable unset so tests address their temporary sockets.
Root/internet-dependent tests remain ignored. No live host TUN, native desktop
session, macOS or Windows validation was performed.
2026-10-21 15:40:45 +01:00
Aleksandr Bogomiakov 94318e7c16 Keep tray device metrics on one line
The device table sized handshake/RTT and packet columns from fixed samples.
Its own header exceeded the handshake sample, and six-digit counters wrapped
at the slash, making device and total rows taller and misaligning the plots.
Measure actual headers, displayed values and totals before drawing, retain
the widest values across frames, and disable wrapping inside table cells.
When a column grows, request a second egui layout pass so the grid's cached
positions settle before larger text is painted over the next column.
Keep the plot as the stretch column and horizontal scrolling for narrow
windows. This changes only tray presentation.

A headless rendering regression checks the actual text shapes for the
header, handshake age/control RTT, per-device packet counts and totals as
counts grow past five digits, reach billions, shrink again, and the table
gets a narrower container. It checks one-line text and non-overlap with the
next column; it failed on the original wrapped header. Existing resize and
long-relay tests remain in place.

Checks on Linux in nix develop:
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets, with inherited
  TSUNAGI_CONTROL_SOCKET unset so tests use their temporary sockets
- git diff --cached --check

The suite's root/internet-dependent tests remain ignored. No native GUI
session or checks on macOS/Windows were run; rendering was checked through
egui's headless output. No installed tray binary was replaced.
2026-10-14 23:37:38 +01:00
Aleksandr Bogomiakov 2b0c938487 Bound iroh path retries to prevent multi-connection OOM
A long-running rc.33 daemon reached OOM with roughly 22 GiB resident and
24 GiB swapped. Live inspection found 40 GiB and 10/20 GiB anonymous
container mappings; the active worker stack and tracing metadata identified
iroh's scheduled path-open retry loop. In iroh 1.2.0 every failed connection
requeues the same path, while each retry visits every connection again.
Separate control and data connections multiply the queue geometrically.

Vendor the published iroh 1.2.0 source and select it with a workspace Cargo
patch. Deduplicate retry admission and cap distinct pending paths at 64,
evicting the oldest before inserting a new candidate. Keep the current and
next retry batches bounded without combining control/data connections or
changing wire protocols, identity derivation, authentication or state.
Document source provenance, the live evidence and removal conditions.
Upstream PR 4551 merged on October 6, but inspection of the official 1.3.0
crate still shows the unconditional enqueue, so that upgrade alone is not
a fix. No version bump, release tag or installed-service change.

The default workspace suite includes the exact vendored admission module.
Three regressions failed with the original enqueue and pass with the fix:
repeated failures across connections, duplicate admission at capacity, and
10,000 distinct candidates with bounded length/capacity and FIFO eviction.
Transport integration tests continue using real endpoints and encryption.

Checks on Linux in nix develop:
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets, with only the inherited
  TSUNAGI_CONTROL_SOCKET removed from the test environment
- git diff --cached --check

Initial full-suite runs failed six DNS readiness checks because the inherited
system socket path overrode their temporary state directories. Clearing that
environment variable resolves those failures without changing application code.
Root/network-dependent ignored tests and other operating systems were not run.
2026-10-14 23:28:53 +01:00
Ultradesu 4e235cbac6 Keep device data paths on one line
The tray used a fixed sample for the data-path column, so longer relay names wrapped and made device rows uneven. Measure each rendered path and retain the widest value, then render path labels without wrapping. Replace direct/via words with embedded Phosphor route icons while preserving transport, relay name, hop count and explanatory tooltips. The app already ships the font; keep its setup in one function and verify registration alongside a long-relay-width regression test. No wire or stored-state changes. Checks: cargo fmt --all -- --check; cargo clippy --locked --workspace --all-targets -- -D warnings; cargo test --locked --workspace --all-targets.
2026-10-14 13:50:05 +01:00
Aleksandr Bogomiakov 2631eb6975 Protect Linux exit-node connections and tidy the device table
Using an exit node could destroy direct peer connections: the agent's UID
bypass protected outbound routing, but reverse-path checks look up replies
without that UID and selected the overlay default. A real-host A/B check
confirmed that relaxing only strict reverse-path filtering restored direct
connections and exit HTTP traffic; the diagnostic host changes were removed.

Keep strict filtering and protect the Linux underlay with a dedicated masked
mark bit, 0x40000000. Mark the dedicated service UID in OUTPUT, save only that
bit in conntrack and restore it before PREROUTING reverse-path checks. Install
IPv4/IPv6 mark and UID main-table bypass rules before Tailscale's priorities;
leave application-specific and exit-default routing after them. Preserve
foreign mark bits, main-table routes and foreign firewall/policy rules.

Protect the first inbound TCP/UDP packet using bounded, locally owned socket
addresses/ports from /proc and an actual socket match. Refresh socket protection
before sending private transport negotiations and on discovery ticks, without
firewall writes for an unchanged socket set. Remote announcements never become
OS arguments. A failed refresh leaves active exit routing in place. Install
protection before routes, remove routes before protection, roll back partial
activation, and remove only exact interface tags, including legacy UID rules.

Reject strict kernel rp_filter without src_valid_mark with an actionable error;
never weaken filtering or change host sysctls. Document mark-aware Netfilter
requirements and provide the trusted ip6tables path in the NixOS module.
Other platforms retain their existing exit implementation. Peer wire formats,
identity derivation, local IPC and persistent storage are unchanged.

Make the tray devices grid fill and shrink with the window, constrain the first
cell and give remaining width to traffic plots. Keep displayed end-to-end
protocol and its short encryption tooltip consistent, independent of carrier
protocol. Move handshake/session age into its own column beside the existing
iroh control RTT. Use stable per-device hover identities and concise tooltips;
missing protocols/measurements remain explicit rather than invented.

Validation on Linux through nix develop:
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets: 529 passed, 3 ignored;
  inherited TSUNAGI_CONTROL_SOCKET and TSUNAGI_CONTROL_GROUP were unset
- explicit privileged kernel regression: 1 passed, using disposable namespaces,
  real TCP/UDP IPv4/IPv6, strict kernel and Netfilter RPF, first inbound packets,
  dynamic UDP binding, application exit routing/IPv6 blocking, repeated apply
  and exact cleanup with foreign Tailscale-priority rules retained
- headless egui grid/scroll-area rendering at multiple sizes and control RTT,
  mark masks/order, passive socket parsing, partial failure and cleanup tests
- git diff --check

Earlier iterations caught rtnetlink family API and nonexhaustive initializer
errors, Clippy warnings, egui's implicit minimum column width, and kernel-test
fixture issues (wildcard echo source, blocked IPv6 neighbour discovery and a
closed child stdout pipe). Corrected before final checks; the stdout-only test
fix was checked with fmt, Clippy and the explicit kernel regression afterward.
Native tray rendering and macOS/Windows checks were not run. The running host
agent was not replaced or restarted. Requested release tag: v0.1.0-rc.32.
2026-10-14 10:40:30 +01:00
Aleksandr Bogomiakov 90ef5cefec Retire offline node claims with equal network-wide cleanup rights
Offline devices reserved names and IPv4 addresses indefinitely, leaving stale
members in the device history. Give every authenticated network participant
the same right to retire another device's observed signed claim without an
owner, quorum, permanent identity ban or secret change.

Add a separately signed retirement containing the target's original signed
claim as proof. Verify both signatures and network scopes, retain the strongest
proved cutoff per target with deterministic ties, and suppress claim versions
at or below it in membership, hostname ownership and IPv4 allocation. Keep the
old signed records and range agreement, and retain markers after a newer claim
returns so old replicas cannot resurrect obsolete reservations. A returning
device signs a newer claim, allocates against current holders and resets its
previous hostname priority rather than taking a reused name with an old time.

Commit local retirement and the requester's author counter atomically before
announcing it; receivers persist proofs before applying them. Bound history to
1024 targets by default, reject additional targets without evicting protection,
and treat corrupt or over-limit mandatory storage as an explicit error. Repeat
bounded marker batches with a per-session rotating cursor so queue pressure
cannot indefinitely starve later markers. Exclude retired claims and cached
hints from history-derived discovery without preventing a fresh authenticated
device from returning.

Expose Agent::forget_node through the shared local control API. CLI command
tsng network forget-node accepts exact network names, full ids or unique id
prefixes, and exact node hostnames, ids, prefixes or overlay IPv4 addresses.
Ambiguous selections change nothing. Recheck the observed version and refuse
self, online nodes, the selected exit node and absent or changed claims.
Add Forget with an identity confirmation to offline tray device rows. Widen
Show devices to 1440x640 with an 800x400 minimum to fit its contents.

Control ALPN/version advances to tsunagi/ctrl/7 and local IPC to 21; upgrade
all peers and restart the agent with matching CLI/tray clients. Data ALPN 5,
network identity derivation, handshake encoding and existing record signing
bytes remain unchanged. SQLite schema 9 adds retirement storage while retaining
identity, secrets, settings and signed claims. Document behavior and limits.

Validation on Linux through nix develop:
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets: 519 passed, 2 existing ignored
  tests; inherited TSUNAGI_CONTROL_SOCKET and TSUNAGI_CONTROL_GROUP were unset
- git diff --check and the built CLI forget-node help

Tests use real iroh authentication and SQLite. Cover restart, stale-replica
recovery, equal rights and network isolation, hostname reuse and return, twelve
marker batches through a two-message queue, malformed signatures and proofs,
invented future cutoffs, deterministic merge, capacity, migration and injected
counter-write rollback, selectors and cleanup through two real CLI processes.
Earlier iterations caught a private test API, a collapsible tray conditional
and a crash-based CLI test waiting for QUIC's idle timeout; corrected before
the final successful suite. Manual tray rendering, macOS/Windows, privileged
TUNs and public-network traversal were not run. The requested RC tag is
v0.1.0-rc.31; no push was requested.
2026-10-12 18:04:29 +01:00
Aleksandr Bogomiakov 13bdcaf98d Render protocol priority buttons with Phosphor icons
The tray protocol selection used ordinary Unicode up/down arrows for its
priority buttons. The default UI fonts lack these glyphs, so both buttons
rendered missing-glyph boxes. Use the regular Phosphor ARROW_UP and ARROW_DOWN
icons already installed in the application's proportional font fallback.

Validation on Linux through nix develop:
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets, with inherited control socket
  and group overrides removed: 509 passed, 2 existing ignored tests

No new tests were added for the two glyph substitutions. Manual tray rendering
and macOS/Windows checks were not performed. No release tag was requested for
this fix.
2026-10-12 17:19:53 +01:00
Aleksandr Bogomiakov d941cdac7a Change network protocol selections at runtime
Protocol selection previously belonged to tsng up and therefore to the
service command line. A blocked or degraded carrier could only be removed
by editing the service and restarting the whole agent. Make the ordered
selection a local per-network setting instead.

Add Agent::set_protocols and a local control request shared by CLI and tray.
Keep tsng protocols without arguments as the protocol catalog; naming a
network shows its selection and supplying a comma-separated list replaces
it. Accept exact names, full ids and unique prefixes using shared selector
rules; ambiguous selectors and invalid lists change nothing. An explicit
none selects control-only operation. Offline edits take the directory lock
and apply on the next start; live edits never write behind the daemon.

Register every built-in plugin and transport in the CLI. Treat up --protocol
as ordered initial defaults for networks without an override, allowing a
protocol absent from the unit to be enabled later. Retain the targeting of
unqualified startup options and allow explicitly scoped options to configure
initially disabled protocols. Persist NULL for inherited preferences and an
empty selection distinctly in SQLite schema 8; migration preserves existing
identities, secrets, signed records and counters.

Serialize changes with network lifecycle operations and commit preferences
before applying or announcing them. Retire only the affected network's data
links and E2E channels, cancel pending opens, and reject obsolete completion
results by attempt id. Clear peer data channels when authenticated protocol
capabilities change while retaining control sessions and topology. Replay
allocations and peer identities after asynchronously prepared plugins become
ready. A control-only start defers creation of the agent's single interface
until data is enabled; plugins receive a sink that can use it later. Automatic
carrier fallback continues to preserve E2E state; explicit policy changes
renegotiate it. Identity derivation and peer wire formats are unchanged.

Add a collapsible protocols row to each tray network tile, with a catalog
provided by the daemon, descriptions, immediately applied checkboxes and
preference-order buttons. Keep editing disabled until the worker refreshes
status so a second click cannot overwrite an edit using a stale snapshot.
Local IPC wire version is 20 for the new request and report fields; upgrade
and restart agent, CLI and tray together. Crate release versions remain unchanged; the requested RC tag is v0.1.0-rc.30.

Validation on Linux through nix develop:
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets: 509 passed, 2 existing ignored
  tests, with inherited TSUNAGI_CONTROL_SOCKET and TSUNAGI_CONTROL_GROUP removed

Tests use real iroh membership, SQLite, UDP, AWG and WireGuard encryption,
QUIC carriers and memory TUNs. Cover either endpoint-id role, full-size packet
delivery after switching, an unaffected second active network, saved selection
after restart, empty/invalid selections, first enablement without restart,
rapid changes racing old opens, live and offline CLI changes, startup-absent
protocols and preservation of signed state through migration. Earlier compile
and focused-test iterations caught initializer mistakes, a short test secret,
asynchronous plugin allocation readiness, and an inherited system socket
path; corrected before the successful final checks. Plain cargo was unavailable
in the host shell, so validation used the Nix toolchain. Manual tray interaction,
macOS/Windows, privileged host TUNs and public-network traversal were not run.
2026-10-12 16:19:18 +01:00
144 changed files with 46852 additions and 437 deletions

No files matched your search

Generated
+1 -2
View File
@@ -2806,8 +2806,6 @@ checksum = "791930b43c0d5973160d90a8f3894509f2b273430f5c5c73b668636d0287c5c0"
[[package]]
name = "iroh"
version = "1.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b2f8d1cfffc83efe39a1031aab423ce09cb8048071baba550508931e9a81ce46"
dependencies = [
"backon",
"blake3",
@@ -6241,6 +6239,7 @@ dependencies = [
"iroh",
"netwatch",
"nix 0.31.3",
"serde_json",
"simple-dns",
"tempfile",
"tokio",
+3 -1
View File
@@ -4,12 +4,14 @@
[workspace]
resolver = "3"
members = ["crates/*"]
exclude = ["vendor/mainline"]
exclude = ["vendor/mainline", "vendor/iroh"]
# Mainline 8.0.0 treats Windows UDP read timeouts as socket failures.
# Keep the small receive-loop correction reproducible until an upstream fix.
[patch.crates-io]
mainline = { path = "vendor/mainline" }
# Backport bounded path retries until an iroh release includes the OOM fix.
iroh = { path = "vendor/iroh" }
[workspace.package]
edition = "2024"
+54
View File
@@ -94,6 +94,19 @@ stopping a suspended network cancels its automatic resume.
Machines are called `tsunagi-<id>` until you rename them: `tsng id hostname laptop`.
### Virtual services
```sh
tsng vip create home api
tsng vip port add home api tcp/443 --backend server1:6443 --backend server2:7443
tsng vip status home api --watch
```
Use `api.home:443`. Each client selects the available backend with the lowest
measured data-plane RTT. A VIP can have multiple independent TCP and UDP ports.
Every member may edit it; the tray's **VIP** button opens a draft editor with
explicit Save. [Commands, health, metrics and limits](docs/vip.md).
### Exit nodes
```sh
@@ -112,6 +125,41 @@ fresh masking profile and outer keys for each pair connection attempt, including
reentry after fallback. Sustained data loss triggers switching independently of
the iroh control connection. [Configuration and limits](docs/amnezia.md).
Change a network's protocol selection without editing its service or restarting:
```sh
tsng protocols # available protocols and settings
tsng protocols home # this device's choice for home
tsng protocols home awg,wg-quic # prefer AWG, retain QUIC as fallback
tsng protocols home none # control plane only
```
Exact network names, full ids and unique id prefixes work here. The choice is
local to this device, saved per network and applied immediately; an offline
change applies on the next start. In the tray, expand **protocols** on a network
tile to toggle protocols or change their preference order. Both endpoints need
a common protocol at the same wire version. `tsng up --protocol` supplies the
initial defaults for networks without a saved choice; it does not restrict
which built-in protocols can be enabled later.
Remove an obsolete offline device from a network's shared history:
```sh
tsng network forget-node home old-laptop
tsng network forget-node home 10.13.37.143
```
Networks accept exact names, full ids or unique id prefixes; nodes accept those
id selectors, exact hostnames or overlay IPv4 addresses. Ambiguous selectors
change nothing. Start the network first if it is stopped. In **Show devices**,
click **Forget** beside an offline node and confirm its identity.
Every member has the same cleanup right. The signed cleanup frees the observed
claim's name and IPv4 reservation, survives restarts and reaches offline replicas
when they reconnect. It does not ban the device: a newer signed claim can return.
Online nodes and the currently selected exit node cannot be forgotten.
[State and limits](docs/sync-model.md#offline-node-cleanup).
## Platforms
| Platform | Build | CLI `tsng` | Tray | Exit node: offer / use | Packages |
@@ -159,6 +207,11 @@ Just trying it: `nix run github:house-of-vanity/tsunagi#tsunagi-cli -- --help`.
`packages.<system>.tsunagi` (with the tray) and `tsunagi-cli` are also exported,
along with `overlays.default`.
For development from a checkout, `nix develop` also exposes the tray's native
runtime libraries, including AppIndicator and graphics/windowing libraries.
Cargo-built binaries such as `target/debug/tsunagi-tray` can run directly inside
this shell, without installing the packaged wrapper.
## Build from source
Rust 1.91 or newer.
@@ -192,6 +245,7 @@ The [threat model](docs/threat-model.md) says what this does and doesn't protect
[Pairwise Amnezia transport](docs/amnezia.md) ·
[Routing](docs/routing.md) ·
[Exit nodes](docs/exit-node.md) ·
[Virtual services](docs/vip.md) ·
[Threat model](docs/threat-model.md) ·
[Contributing](AGENTS.md)
+245 -2
View File
@@ -25,6 +25,7 @@ use tsunagi_wg_quic::WireguardConfig;
struct Node {
agent: Agent,
plugin: Arc<AwgPlugin>,
quic: Option<Arc<tsunagi_wg_quic::WireguardPlugin>>,
transport: Arc<AwgTransport>,
tuns: MemoryTunFactory,
loss: Arc<AtomicBool>,
@@ -47,6 +48,16 @@ impl Node {
interface: &str,
fallback: bool,
mtu: u32,
) -> Self {
Self::open_selected(root, discovery, interface, fallback, mtu, None).await
}
async fn open_selected(
root: &Path,
discovery: &SharedMemoryDiscovery,
interface: &str,
fallback: bool,
mtu: u32,
default_protocols: Option<Vec<String>>,
) -> Self {
let identity =
tsunagi::storage::StateStore::open(tsunagi::StoragePaths::under(root).state_db())
@@ -70,9 +81,9 @@ impl Node {
let tuns = MemoryTunFactory::new();
let mut config = config_with(root, discovery)
.with_interface(Arc::new(tuns.clone()), interface, mtu)
.with_overlay_ipv4_range(Some(tsunagi::state::DEFAULT_IPV4_RANGE))
.with_plugin(plugin.clone() as Arc<dyn IpPlugin>)
.with_custom_transport(AWG_PROTOCOL, faults);
let mut quic_plugin = None;
if fallback {
let quic = tsunagi_wg_quic::WireguardPlugin::open(
WireguardConfig::new(root.join("wg-quic"))
@@ -81,8 +92,10 @@ impl Node {
)
.await
.unwrap();
config = config.with_plugin(quic);
config = config.with_plugin(quic.clone());
quic_plugin = Some(quic);
}
config.default_protocols = default_protocols;
let agent = Agent::spawn(config).await.unwrap();
let inbound_agent = agent.clone();
let inbound_transport = transport.clone();
@@ -96,6 +109,7 @@ impl Node {
Self {
agent,
plugin,
quic: quic_plugin,
transport,
tuns,
loss,
@@ -163,6 +177,235 @@ async fn exchange(a: &Node, b: &Node, id: NetworkId) {
);
}
}
#[tokio::test]
async fn a_control_only_agent_enables_real_data_without_restart_and_cancels_stale_attempts() {
let discovery = SharedMemoryDiscovery::new();
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
let a = Node::open_selected(
roots[0].path(),
&discovery,
"initial-none-a",
true,
1280,
Some(vec![]),
)
.await;
let b = Node::open_selected(
roots[1].path(),
&discovery,
"initial-none-b",
true,
1280,
Some(vec![]),
)
.await;
let (name, secret) = network("enable-after-control-only");
let id = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
wait_for_peers(&a.agent, id, 1).await;
wait_for_peers(&b.agent, id, 1).await;
assert!(a.agent.overlay().is_none());
assert!(b.agent.overlay().is_none());
for node in [&a, &b] {
assert!(
node.agent
.network_status(id)
.await
.unwrap()
.protocols
.is_empty()
);
node.agent
.set_protocols(id, vec!["awg".into()])
.await
.unwrap();
assert!(node.agent.overlay().is_some());
}
a.ready(id, 1).await;
b.ready(id, 1).await;
exchange(&a, &b, id).await;
let before = a.agent.overlay().unwrap().interface;
// Changes race unfinished opens and prepared plugin callbacks; the last
// selection must win even if an earlier open has completed in its queue.
for selection in [
vec!["wg-quic".into()],
vec![],
vec!["awg".into()],
vec![],
vec!["awg".into()],
] {
a.agent.set_protocols(id, selection).await.unwrap();
}
a.ready(id, 1).await;
b.ready(id, 1).await;
exchange(&a, &b, id).await;
assert_eq!(a.agent.overlay().unwrap().interface, before);
for node in [&a, &b] {
let status = node.agent.network_status(id).await.unwrap();
assert_eq!(status.protocols, ["awg"]);
assert_eq!(status.metrics.sessions_established, 1);
assert_eq!(status.metrics.disconnects, 0);
}
a.agent.shutdown().await;
b.agent.shutdown().await;
}
#[tokio::test]
async fn runtime_protocol_selection_replaces_real_tunnels_and_survives_restart() {
let discovery = SharedMemoryDiscovery::new();
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
let a = Node::open_with_fallback(roots[0].path(), &discovery, "runtime-a", true).await;
let b = Node::open_with_fallback(roots[1].path(), &discovery, "runtime-b", true).await;
let (name, secret) = network("runtime-protocols");
let id = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
a.ready(id, 1).await;
b.ready(id, 1).await;
exchange(&a, &b, id).await;
let (other_name, other_secret, other_id) = loop {
let (name, secret) = network("unchanged-runtime-network");
let other_id = tsunagi::identity::NetworkKeys::derive(&name, &secret).network_id();
if tsunagi::state::derived_ipv4_range(id) != tsunagi::state::derived_ipv4_range(other_id) {
break (name, secret, other_id);
}
};
a.agent
.join_network(&other_name, &other_secret)
.await
.unwrap();
b.agent
.join_network(&other_name, &other_secret)
.await
.unwrap();
a.ready(other_id, 1).await;
b.ready(other_id, 1).await;
let other_before = a
.agent
.network_status(other_id)
.await
.unwrap()
.metrics
.data_links_established;
for node in [&a, &b] {
let before = node.agent.network_status(id).await.unwrap().protocols;
for invalid in [vec!["unknown".into()], vec!["awg".into(), "awg".into()]] {
assert!(node.agent.set_protocols(id, invalid).await.is_err());
assert_eq!(
node.agent.network_status(id).await.unwrap().protocols,
before
);
}
}
// Exercise updates initiated by either endpoint-id role, including an
// empty set. Membership and the control session stay alive throughout.
for node in [&a, &b] {
node.agent.set_protocols(id, Vec::new()).await.unwrap();
wait_until("peer accepts control-only selection", || async {
let other = if node.agent.endpoint_id() == a.agent.endpoint_id() {
&b
} else {
&a
};
let status = other.agent.network_status(id).await.ok()?;
(status.peers.len() == 1
&& status.peers[0].protocols.is_empty()
&& matches!(status.peers[0].data_path, tsunagi::agent::DataPath::None))
.then_some(())
})
.await;
assert_eq!(node.agent.network_status(id).await.unwrap().peers.len(), 1);
node.agent
.set_protocols(id, vec!["wg-quic".into(), "awg".into()])
.await
.unwrap();
}
a.agent
.set_protocols(id, vec!["wg-quic".into()])
.await
.unwrap();
wait_until("real QUIC replaces AWG", || async {
let a_status = a.agent.network_status(id).await.ok()?;
let b_status = b.agent.network_status(id).await.ok()?;
(a_status.peers[0].protocols == ["wg-quic"]
&& b_status.peers[0].protocols == ["wg-quic"]
&& !matches!(a_status.peers[0].data_path, tsunagi::agent::DataPath::None)
&& !matches!(b_status.peers[0].data_path, tsunagi::agent::DataPath::None)
&& a.transport.overview(id).iter().all(|p| p.closed)
&& b.transport.overview(id).iter().all(|p| p.closed))
.then_some(())
})
.await;
// Get addresses from the system allocation: the disabled AWG plugin no
// longer has a live view of this network.
for (source, dest) in [(&a, &b), (&b, &a)] {
wait_until("QUIC end-to-end handshake", || async {
source
.quic
.as_ref()?
.overview(id)?
.peers
.iter()
.any(|p| p.tunnel.as_ref().is_some_and(|t| t.health.is_up()))
.then_some(())
})
.await;
let addr = |n: &Node| {
n.quic
.as_ref()
.unwrap()
.overview(id)
.unwrap()
.overlay_address_v4
.unwrap()
};
let bytes = packet(addr(source), addr(dest), 1280);
source.tun().push_from_os(bytes.clone());
assert_eq!(
tokio::time::timeout(Duration::from_secs(5), dest.tun().pop_to_os())
.await
.unwrap()
.unwrap(),
bytes
);
}
assert_eq!(
a.agent
.network_status(other_id)
.await
.unwrap()
.metrics
.data_links_established,
other_before
);
exchange(&a, &b, other_id).await;
a.agent.shutdown().await;
let restarted = Node::open_with_fallback(roots[0].path(), &discovery, "runtime-a", true).await;
wait_for_peers(&restarted.agent, id, 1).await;
assert_eq!(
restarted.agent.network_status(id).await.unwrap().protocols,
["wg-quic"]
);
assert_eq!(
b.agent.network_status(other_id).await.unwrap().protocols,
["awg", "wg-quic"]
);
restarted.ready(other_id, 1).await;
b.ready(other_id, 1).await;
exchange(&restarted, &b, other_id).await;
restarted
.agent
.set_protocols(id, vec!["awg".into()])
.await
.unwrap();
restarted.ready(id, 1).await;
b.ready(id, 1).await;
exchange(&restarted, &b, id).await;
restarted.agent.shutdown().await;
b.agent.shutdown().await;
}
#[tokio::test]
async fn real_pairs_carry_full_size_packets_and_restart_with_fresh_profiles() {
tsunagi::testing::init_tracing();
+258 -29
View File
@@ -58,6 +58,8 @@ use shutdown::Shutdown;
/// Summary of a configured network, whether or not it is running.
#[derive(Debug, Clone)]
pub struct ConfiguredNetwork {
/// Enabled local data-plane protocols, in preference order.
pub protocols: Vec<String>,
/// Saved local broadcast participation.
pub broadcast: bool,
/// Public network identifier.
@@ -70,6 +72,17 @@ pub struct ConfiguredNetwork {
pub active: bool,
}
/// A durable version-scoped cleanup, not an identity ban.
#[derive(Debug, Clone)]
pub struct ForgetNodeOutcome {
/// Identity whose observed claim was retired.
pub target: EndpointId,
/// The retired claim version; future signed claims remain admissible.
pub record_version: u64,
/// Authenticated peers to which cleanup was queued immediately.
pub peers_told: usize,
}
/// A running agent.
///
/// Cloning gives another handle to the same agent.
@@ -104,6 +117,37 @@ struct Inner {
}
impl Inner {
/// Called under the lifecycle lock after startup: an initially
/// control-only agent creates its one interface on first enablement.
async fn ensure_interface(self: &Arc<Self>) {
if self.interface.get().is_some() {
return;
}
if let Some(factory) = self.config.tun_factory.clone() {
let carrier = Arc::new(Carrier(Arc::downgrade(self))) as Arc<dyn PacketCarrier>;
match crate::overlay::Interface::start(
factory,
self.config.interface_name.clone(),
self.config.interface_mtu,
Arc::clone(&self.routes),
carrier,
)
.await
{
Ok(interface) => {
let _ = self.interface.set(Arc::new(interface));
}
Err(err) => {
let _ = self.events.send(Event::PluginError {
network: NetworkId::from_bytes([0u8; 32]),
protocol: "overlay".into(),
reason: err.to_string(),
});
}
}
}
}
/// The name this agent currently answers to.
fn read_hostname(&self) -> String {
match self.hostname.read() {
@@ -265,12 +309,133 @@ impl TransportContext for TransportCtx {
}
impl Agent {
/// Atomically saves or removes a version-checked shared VIP draft.
/// Every authenticated network member may edit any service.
pub async fn save_vip(
&self,
network: NetworkId,
config: crate::vip::VipConfig,
expected: Option<String>,
delete: bool,
) -> Result<Vec<crate::vip::VipStatus>> {
let (reply, receive) = oneshot::channel();
self.command(
network,
NetCommand::SaveVip {
config,
expected,
delete,
reply,
},
)
.await?;
receive.await.map_err(|_| Error::Stopped)?
}
/// Retires one offline member's observed claim without revoking membership.
/// Every member has this right. The version must still match the selection.
pub async fn forget_node(
&self,
network: NetworkId,
peer: EndpointId,
expected_version: u64,
) -> Result<ForgetNodeOutcome> {
let _lifecycle = self.inner.lifecycle.lock().await;
let (reply, receive) = oneshot::channel();
self.command(
network,
NetCommand::ForgetNode {
peer,
expected_version,
reply,
},
)
.await?;
receive.await.map_err(|_| Error::Stopped)?
}
fn effective_protocols(&self, saved: Option<&Vec<String>>) -> Vec<String> {
saved
.or(self.inner.config.default_protocols.as_ref())
.cloned()
.unwrap_or_else(|| {
self.inner
.config
.plugins
.iter()
.map(|p| p.protocol_id().to_owned())
.collect()
})
}
fn selected_plugins(&self, protocols: &[String]) -> Vec<crate::dataplane::SharedPlugin> {
protocols
.iter()
.filter_map(|name| {
self.inner
.config
.plugins
.iter()
.find(|p| p.protocol_id() == name)
.cloned()
})
.collect()
}
/// Saves and immediately applies a network's ordered local protocol set.
/// An empty set disables data traffic while retaining control sessions.
pub async fn set_protocols(&self, network_id: NetworkId, protocols: Vec<String>) -> Result<()> {
crate::dataplane::validate_protocols(&protocols, self.inner.limits.max_capabilities)?;
for name in &protocols {
if !self
.inner
.config
.plugins
.iter()
.any(|p| p.protocol_id() == name)
{
return Err(Error::InvalidProtocols(format!(
"this agent has no `{name}` plugin"
)));
}
}
let _lifecycle = self.inner.lifecycle.lock().await;
self.inner
.storage
.set_protocols(network_id, protocols.clone())
.await?;
if self.is_active(network_id).await {
let (reply, receive) = oneshot::channel();
if !protocols.is_empty() {
self.inner.ensure_interface().await;
}
self.command(
network_id,
NetCommand::SetProtocols {
protocols,
interface: self.inner.interface.get().cloned(),
reply,
},
)
.await?;
receive.await.map_err(|_| Error::Stopped)?;
}
Ok(())
}
/// Starts an agent.
///
/// Opens the state store (taking its ownership lock), restores the
/// persistent device identity, binds the iroh endpoint and activates every
/// configured network whose auto-start flag is set.
pub async fn spawn(config: AgentConfig) -> Result<Self> {
if let Some(protocols) = &config.default_protocols {
crate::dataplane::validate_protocols(protocols, config.limits.max_capabilities)?;
for name in protocols {
if !config.plugins.iter().any(|p| p.protocol_id() == name) {
return Err(Error::InvalidProtocols(format!(
"this agent has no `{name}` plugin"
)));
}
}
}
let storage = Storage::open(&config.paths)?;
let identity = storage.device_identity().await?;
let adapter = EndpointAdapter::bind(&config, &identity).await?;
@@ -303,31 +468,13 @@ impl Agent {
config,
});
// One interface for the agent, if it was given a way to make one.
// Failing to is reported and not fatal: the control plane works, and
// so do the protocols, they just have nowhere to put packets.
if let Some(factory) = inner.config.tun_factory.clone() {
let carrier = Arc::new(Carrier(Arc::downgrade(&inner))) as Arc<dyn PacketCarrier>;
match crate::overlay::Interface::start(
factory,
inner.config.interface_name.clone(),
inner.config.interface_mtu,
Arc::clone(&inner.routes),
carrier,
)
.await
{
Ok(interface) => {
let _ = inner.interface.set(Arc::new(interface));
}
Err(err) => {
let _ = inner.events.send(Event::PluginError {
network: NetworkId::from_bytes([0u8; 32]),
protocol: "overlay".into(),
reason: err.to_string(),
});
}
}
if inner
.config
.default_protocols
.as_ref()
.is_none_or(|names| !names.is_empty())
{
inner.ensure_interface().await;
}
// The data plane rides on iroh too, which is where it gets hole
@@ -364,8 +511,9 @@ impl Agent {
if !inner.config.plugins.is_empty() {
let (plugin_tx, plugin_rx) = mpsc::channel(64);
let sink = inner
.interface
.get()
.config
.tun_factory
.as_ref()
.map(|_| Arc::new(Sink(Arc::downgrade(&inner))) as Arc<dyn PacketSink>);
let context = PluginContext::new(plugin_tx, inner.identity.endpoint_id(), sink);
for plugin in &inner.config.plugins {
@@ -784,6 +932,40 @@ impl Agent {
.await?
.into_iter()
.find(|stored| stored.network_id == network_id);
// Mandatory cleanup proofs must never be silently dropped on startup.
let retirements = self
.inner
.storage
.retirements(network_id, self.inner.limits.max_retirements)
.await?;
let vip_entries = self
.inner
.storage
.vip_fields(
network_id,
self.inner
.limits
.max_vips
.saturating_mul(self.inner.limits.max_vip_fields)
.saturating_add(self.inner.limits.max_vip_tombstones),
)
.await?;
let mut vips = crate::vip::VipSet::default();
for id in vip_entries
.iter()
.map(|e| e.vip.clone())
.collect::<std::collections::BTreeSet<_>>()
{
vips.merge(
network_id,
vip_entries
.iter()
.filter(|e| e.vip == id)
.cloned()
.collect(),
&self.inner.limits,
)?;
}
let range = self.desired_range(network_id).await?;
if let Some(range) = range {
let conflicts = self.inner.routes.overlapping_networks(network_id, range);
@@ -824,12 +1006,20 @@ impl Agent {
};
let broadcast = stored.as_ref().is_none_or(|stored| stored.broadcast);
let protocols =
self.effective_protocols(stored.as_ref().and_then(|s| s.protocols.as_ref()));
let plugins = self.selected_plugins(&protocols);
if !plugins.is_empty() {
self.inner.ensure_interface().await;
}
let exit_offer = stored.as_ref().is_some_and(|stored| stored.exit_node);
let exit_via = stored
.as_ref()
.and_then(|stored| stored.exit_via)
.and_then(|bytes| EndpointId::from_bytes(&bytes).ok());
let handle = network::spawn(RuntimeParams {
vips,
retirements,
broadcast,
exit_offer,
exit_via,
@@ -842,7 +1032,8 @@ impl Agent {
discovery,
discovery_interval: self.inner.config.discovery_interval,
discovery_policy: self.inner.config.discovery_policy.clone(),
plugins: self.inner.config.plugins.clone(),
available_plugins: self.inner.config.plugins.clone(),
plugins: plugins.clone(),
routes: Arc::clone(&self.inner.routes),
interface: self.inner.interface.get().cloned(),
hostname: self.inner.read_hostname(),
@@ -856,7 +1047,7 @@ impl Agent {
networks.insert(network_id, handle);
drop(networks);
for plugin in &self.inner.config.plugins {
for plugin in &plugins {
plugin.on_network_activated(network_id);
}
@@ -997,6 +1188,7 @@ impl Agent {
.await?
.into_iter()
.map(|stored| ConfiguredNetwork {
protocols: self.effective_protocols(stored.protocols.as_ref()),
broadcast: stored.broadcast,
network_id: stored.network_id,
name: stored.name,
@@ -1074,7 +1266,44 @@ impl Agent {
let keys = NetworkKeys::derive(&stored.name, &stored.secret);
let range = self.desired_range(stored.network_id).await?;
let range_conflicts = self.range_conflicts(stored.network_id, range).await?;
let fields = self
.inner
.storage
.vip_fields(
stored.network_id,
self.inner
.limits
.max_vips
.saturating_mul(self.inner.limits.max_vip_fields)
.saturating_add(self.inner.limits.max_vip_tombstones),
)
.await?;
let mut service_state = crate::vip::VipSet::default();
for id in fields
.iter()
.map(|e| e.vip.clone())
.collect::<std::collections::BTreeSet<_>>()
{
service_state.merge(
stored.network_id,
fields.iter().filter(|e| e.vip == id).cloned().collect(),
&self.inner.limits,
)?;
}
let vips = service_state
.configs()
.into_iter()
.map(|config| crate::vip::VipStatus {
revision: service_state.revision(&config.id),
config,
address: None,
error: Some("network is inactive".into()),
listeners: vec![],
})
.collect();
networks.push(NetworkStatus {
vips,
protocols: self.effective_protocols(stored.protocols.as_ref()),
auto_start: stored.auto_start,
broadcast: stored.broadcast,
exit_offer: stored.exit_node,
+625 -26
View File
@@ -26,7 +26,9 @@ use crate::net::{EndpointAdapter, PathAddr, snapshot_connection};
use crate::proto::handshake::{self, HandshakeOutcome, Role};
use crate::proto::message::{Announcement, ControlMessage, Envelope, encode, kind};
use crate::state::allocator::allocate;
use crate::state::{Ipv4Range, Merged, RecordBody, SignedRecord, StateSet};
use crate::state::{
Ipv4Range, MAX_RETIREMENT_LEN, Merged, RecordBody, SignedRecord, SignedRetirement, StateSet,
};
use crate::storage::Storage;
use super::events::Event;
@@ -48,6 +50,17 @@ pub(crate) struct InboundSession {
/// Commands accepted by a network runtime.
pub(crate) enum NetCommand {
SaveVip {
config: crate::vip::VipConfig,
expected: Option<String>,
delete: bool,
reply: oneshot::Sender<Result<Vec<crate::vip::VipStatus>>>,
},
ForgetNode {
peer: EndpointId,
expected_version: u64,
reply: oneshot::Sender<Result<super::ForgetNodeOutcome>>,
},
Inbound(Box<InboundSession>),
Send {
peer: EndpointId,
@@ -72,6 +85,11 @@ pub(crate) enum NetCommand {
enabled: bool,
reply: oneshot::Sender<()>,
},
SetProtocols {
interface: Option<Arc<crate::overlay::Interface>>,
protocols: Vec<String>,
reply: oneshot::Sender<()>,
},
/// Offer this agent as an exit node in this network, or stop.
SetExitOffer {
enabled: bool,
@@ -113,6 +131,8 @@ pub(crate) enum NetCommand {
impl std::fmt::Debug for NetCommand {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
NetCommand::SaveVip { .. } => f.write_str("SaveVip"),
NetCommand::ForgetNode { peer, .. } => write!(f, "ForgetNode({peer})"),
NetCommand::Inbound(_) => f.write_str("Inbound"),
NetCommand::InboundLink(link) => write!(f, "InboundLink({})", link.protocol),
NetCommand::Send { peer, message, .. } => {
@@ -122,6 +142,7 @@ impl std::fmt::Debug for NetCommand {
NetCommand::Status { .. } => f.write_str("Status"),
NetCommand::Recheck => f.write_str("Recheck"),
NetCommand::SetBroadcast { enabled, .. } => write!(f, "SetBroadcast({enabled})"),
NetCommand::SetProtocols { protocols, .. } => write!(f, "SetProtocols({protocols:?})"),
NetCommand::SetExitOffer { enabled, .. } => write!(f, "SetExitOffer({enabled})"),
NetCommand::SetExitVia { via, .. } => write!(f, "SetExitVia({})", via.is_some()),
NetCommand::Reannounce => f.write_str("Reannounce"),
@@ -156,6 +177,8 @@ impl NetworkHandle {
/// Everything a network runtime needs to run.
pub(crate) struct RuntimeParams {
pub(crate) vips: crate::vip::VipSet,
pub(crate) retirements: Vec<SignedRetirement>,
pub(crate) broadcast: bool,
/// This agent offers itself as an exit node here.
pub(crate) exit_offer: bool,
@@ -171,6 +194,7 @@ pub(crate) struct RuntimeParams {
pub(crate) discovery_interval: Duration,
pub(crate) discovery_policy: DiscoveryPolicy,
pub(crate) plugins: Vec<SharedPlugin>,
pub(crate) available_plugins: Vec<SharedPlugin>,
/// Who holds which overlay address, shared with every other network.
pub(crate) routes: Arc<crate::overlay::RoutingTable>,
/// The one interface, when the agent has one.
@@ -192,6 +216,7 @@ pub(crate) struct RuntimeParams {
/// Outcome of one attempt to open a data plane link.
struct LinkOutcome {
generation: u64,
peer: EndpointId,
protocol: String,
result: Result<SharedLink, String>,
@@ -332,6 +357,9 @@ struct Runtime {
announced_reach: HashSet<crate::proto::message::ReachableLink>,
/// Links currently being opened, so we do not start two.
opening: HashSet<(EndpointId, String)>,
link_tasks: HashMap<(EndpointId, String), tokio::task::AbortHandle>,
link_attempts: HashMap<(EndpointId, String), u64>,
next_link_attempt: u64,
link_results_tx: mpsc::Sender<LinkOutcome>,
link_results_rx: mpsc::Receiver<LinkOutcome>,
/// Protocol dial failure count and backoff cooldown per peer and protocol.
@@ -355,6 +383,11 @@ struct Runtime {
state: StateSet,
/// Snapshots received while dispatching, handled on the next loop pass.
pending_state: Vec<(EndpointId, Vec<SignedRecord>)>,
pending_retirements: Vec<(EndpointId, Vec<SignedRetirement>)>,
pending_vips: Vec<Vec<crate::vip::VipEntry>>,
vip_cursor: HashMap<EndpointId, usize>,
/// Next cleanup batch per live session; prevents queue pressure starving a suffix.
retirement_cursor: HashMap<EndpointId, usize>,
/// The highest version this agent has ever published for this network.
own_version: u64,
/// Whether the exit node in use was reachable when last looked at, so
@@ -425,6 +458,9 @@ impl Runtime {
local_capabilities: Vec::new(),
announced_reach: HashSet::new(),
opening: HashSet::new(),
link_tasks: HashMap::new(),
link_attempts: HashMap::new(),
next_link_attempt: 0,
link_results_tx,
link_results_rx,
protocol_dial_states: HashMap::new(),
@@ -434,6 +470,10 @@ impl Runtime {
reported_name_taken: None,
state: StateSet::new(),
pending_state: Vec::new(),
pending_retirements: Vec::new(),
pending_vips: Vec::new(),
vip_cursor: HashMap::new(),
retirement_cursor: HashMap::new(),
own_version: 0,
exit_via_was_online: false,
announced_exit: false,
@@ -511,6 +551,10 @@ impl Runtime {
link.close();
}
self.links.clear();
for (_, task) in self.link_tasks.drain() {
task.abort();
}
self.opening.clear();
self.pending_health.clear();
self.last_failed_protocol.clear();
self.protocol_dial_states.clear();
@@ -529,6 +573,23 @@ impl Runtime {
async fn handle_command(&mut self, command: NetCommand) {
match command {
NetCommand::SaveVip {
config,
expected,
delete,
reply,
} => {
let result = self.save_vip(config, expected, delete).await;
let _ = reply.send(result);
}
NetCommand::ForgetNode {
peer,
expected_version,
reply,
} => {
let result = self.forget_node(peer, expected_version).await;
let _ = reply.send(result);
}
NetCommand::Inbound(inbound) => {
self.install_session(*inbound).await;
}
@@ -538,6 +599,13 @@ impl Runtime {
message,
reply,
} => {
if matches!(&message, ControlMessage::Transport { .. })
&& let Some(interface) = self.params.interface.clone()
{
// Locally bound pair sockets must be protected before their
// private offer/accept invites the first inbound datagram.
interface.sync_exit_rules().await;
}
let _ = reply.send(self.send_to(peer, message));
}
NetCommand::Broadcast { message, reply } => {
@@ -553,6 +621,60 @@ impl Runtime {
NetCommand::Status { reply } => {
let _ = reply.send(Box::new(self.status()));
}
NetCommand::SetProtocols {
protocols,
interface,
reply,
} => {
self.params.interface = interface;
let current: Vec<_> = self
.params
.plugins
.iter()
.map(|p| p.protocol_id().to_owned())
.collect();
if current != protocols {
let peers: Vec<_> = self.sessions.keys().copied().collect();
for peer in &peers {
self.reset_peer_data(*peer, "local protocol selection changed");
}
for plugin in &self.params.plugins {
if !protocols.iter().any(|name| name == plugin.protocol_id()) {
plugin.on_network_deactivated(self.network_id);
}
}
self.params.plugins = protocols
.iter()
.filter_map(|name| {
self.params
.available_plugins
.iter()
.find(|p| p.protocol_id() == name)
.cloned()
})
.collect();
for plugin in &self.params.plugins {
if !current.iter().any(|name| name == plugin.protocol_id()) {
plugin.on_network_activated(self.network_id);
}
}
self.hub.set_priority(protocols);
self.publish_allocations().await;
for peer in peers {
let caps = self
.sessions
.get(&peer)
.map(|s| s.capabilities.clone())
.unwrap_or_default();
self.dispatch_capabilities(peer, &caps);
}
self.reannounce();
self.ensure_links();
self.update_paths();
self.announce_reach(true);
}
let _ = reply.send(());
}
NetCommand::SetBroadcast { enabled, reply } => {
self.params.broadcast = enabled;
self.update_broadcast();
@@ -584,7 +706,22 @@ impl Runtime {
let _ = reply.send(outcome);
}
NetCommand::Recheck => self.discovery_round().await,
NetCommand::Reannounce => self.reannounce(),
NetCommand::Reannounce => {
// A newly enabled plugin may prepare its key asynchronously.
// Replay allocations and peer identities once it is ready.
self.publish_allocations().await;
let peers: Vec<_> = self
.sessions
.iter()
.map(|(peer, s)| (*peer, s.capabilities.clone()))
.collect();
for (peer, caps) in peers {
self.dispatch_capabilities(peer, &caps);
}
self.reannounce();
self.ensure_links();
self.update_paths();
}
NetCommand::Release { reply } => {
// A tombstone, so the address and the name this agent held
// are freed for somebody else instead of staying reserved
@@ -714,6 +851,8 @@ impl Runtime {
// the check is a comparison when nothing has changed.
self.ensure_own_claim().await;
self.report_hostname_taken();
self.broadcast_vips();
self.broadcast_retirements();
// The same timer repairs the relay picture: reachability is soft
// state, repeated while it holds and forgotten when it stops.
self.update_paths();
@@ -727,7 +866,7 @@ impl Runtime {
// still dialable when the endpoint's own discovery can resolve
// one. This is what makes a network a mesh rather than a star
// around whoever was named on a command line.
for record in self.state.records() {
for record in self.state.active_records() {
if let Ok(author) = record.author_id()
&& author != self.local_id
{
@@ -744,6 +883,9 @@ impl Runtime {
let Ok(endpoint_id) = EndpointId::from_bytes(&hint.endpoint_id) else {
continue;
};
if self.state.is_retired(&endpoint_id) {
continue;
}
let Some(addr) = decode_hint(endpoint_id, &hint.addr) else {
continue;
};
@@ -758,6 +900,9 @@ impl Runtime {
// Last, so it carries what this round learned.
self.introduce_peers();
self.refresh_underlay().await;
if let Some(interface) = self.params.interface.clone() {
interface.sync_exit_rules().await;
}
}
/// Hands the interface the underlay destinations this network depends
@@ -949,6 +1094,22 @@ impl Runtime {
/// boundary, because a restored backup or a copied file could hold
/// anything.
async fn load_state(&mut self) {
for marker in std::mem::take(&mut self.params.retirements) {
// Storage verified these proofs and activation enforced capacity.
if let Err(err) = self.state.merge_retirement(
self.network_id,
marker,
self.params.limits.max_retirements,
) {
self.emit(Event::PluginError {
network: self.network_id,
protocol: "state".into(),
reason: err.to_string(),
});
self.shutdown.trigger();
return;
}
}
let stored = self
.params
.storage
@@ -964,7 +1125,9 @@ impl Runtime {
.storage
.own_record_version(self.network_id, self.local_id)
.await
.unwrap_or(0);
.unwrap_or(0)
.max(self.state.authored_version(&self.local_id))
.max(self.params.vips.authored_version(self.local_id));
for (id, hostname) in self
.params
.storage
@@ -1038,11 +1201,26 @@ impl Runtime {
match self.state.address_of(&self.local_id) {
Some(mine) if range.contains(mine) => Some(mine),
_ => {
let taken: std::collections::HashSet<std::net::Ipv4Addr> = holders
let mut taken: std::collections::HashSet<std::net::Ipv4Addr> = holders
.iter()
.filter(|(_, holder)| **holder != self.local_id)
.map(|(address, _)| *address)
.collect();
for vip in self.params.vips.views(
Some(range),
taken.clone(),
&std::collections::BTreeSet::new(),
&self.params.limits,
) {
if let Some(address) = vip.address {
taken.insert(address);
}
}
for vip in self.params.routes.vips.statuses(self.network_id) {
if let Some(address) = vip.address {
taken.insert(address);
}
}
match allocate(
self.network_id,
self.local_id,
@@ -1076,6 +1254,7 @@ impl Runtime {
let kept = self
.state
.get(&self.local_id)
.filter(|_| !self.state.is_retired(&self.local_id))
.filter(|record| record.body.hostname() == Some(name.as_str()))
.and_then(|record| record.body.hostname_since());
kept.unwrap_or_else(unix_millis)
@@ -1108,7 +1287,7 @@ impl Runtime {
if self
.state
.get(&self.local_id)
.is_some_and(|record| record.body == body)
.is_some_and(|record| record.body == body && !self.state.is_retired(&self.local_id))
{
return;
}
@@ -1116,13 +1295,187 @@ impl Runtime {
self.publish_record(body).await;
}
async fn forget_node(
&mut self,
peer: EndpointId,
expected_version: u64,
) -> Result<super::ForgetNodeOutcome> {
if peer == self.local_id {
return Err(Error::ForgetNode(
"use network leave for this device".into(),
));
}
if self.is_online(peer) {
return Err(Error::ForgetNode(
"the node is online; only offline history can be forgotten".into(),
));
}
if self.params.exit_via == Some(peer) {
return Err(Error::ForgetNode(
"stop using this exit node before forgetting it".into(),
));
}
let target = self
.state
.get(&peer)
.filter(|_| !self.state.is_retired(&peer))
.filter(|record| matches!(record.body, RecordBody::Claim { .. }))
.cloned()
.ok_or_else(|| Error::ForgetNode("no current signed claim for this node".into()))?;
if target.version != expected_version {
return Err(Error::ForgetNode(
"the node's claim changed; refresh and select it again".into(),
));
}
let version = self
.own_version
.max(self.state.authored_version(&self.local_id))
.checked_add(1)
.filter(|v| *v <= i64::MAX as u64)
.ok_or_else(|| Error::ForgetNode("author counter exhausted".into()))?;
let marker =
SignedRetirement::sign(&self.params.device_secret, self.network_id, version, target);
let mut candidate = self.state.clone();
candidate
.merge_retirement(
self.network_id,
marker.clone(),
self.params.limits.max_retirements,
)
.map_err(|err| Error::ForgetNode(err.to_string()))?;
// Signed cleanup and our author counter commit together before visibility.
self.params
.storage
.publish_own_retirement(marker.clone())
.await?;
self.own_version = version;
self.state = candidate;
self.known_hostnames.remove(&peer);
self.dial_states.remove(&peer);
self.candidate_addrs.remove(&peer);
self.publish_allocations().await;
self.report_hostname_taken();
let message = ControlMessage::Retirements {
markers: vec![marker],
};
let peers: Vec<_> = self.sessions.keys().copied().collect();
let peers_told = peers
.into_iter()
.filter(|peer| self.send_to(*peer, message.clone()).is_ok())
.count();
Ok(super::ForgetNodeOutcome {
target: peer,
record_version: expected_version,
peers_told,
})
}
fn broadcast_retirements(&mut self) -> usize {
let markers = self.state.retirements();
if markers.is_empty() {
return 0;
}
// Each proof has an encoded bound; reserve envelope/list overhead and
// chunk every marker, so a low-id prefix cannot starve later cleanup.
let batch = (self.params.limits.max_frame_len.saturating_sub(128) / MAX_RETIREMENT_LEN)
.max(1)
.min(self.params.limits.max_state_records.max(1));
let mut delivered = HashSet::new();
let peers: Vec<_> = self.sessions.keys().copied().collect();
let chunks: Vec<_> = markers.chunks(batch).collect();
for peer in peers {
let start = self.retirement_cursor.get(&peer).copied().unwrap_or(0) % chunks.len();
for offset in 0..chunks.len() {
let index = (start + offset) % chunks.len();
let message = ControlMessage::Retirements {
markers: chunks[index].to_vec(),
};
if self.send_to(peer, message).is_err() {
self.retirement_cursor.insert(peer, index);
break;
}
delivered.insert(peer);
self.retirement_cursor
.insert(peer, (index + 1) % chunks.len());
}
}
delivered.len()
}
async fn receive_retirements(&mut self, peer: EndpointId, markers: Vec<SignedRetirement>) {
let mut changed = false;
for marker in markers {
let mut candidate = self.state.clone();
match candidate.merge_retirement(
self.network_id,
marker.clone(),
self.params.limits.max_retirements,
) {
Ok(false) => continue,
Ok(true) => {}
Err(err) => {
self.metrics.protocol_violations += 1;
self.emit(Event::ProtocolViolation {
network: Some(self.network_id),
peer: Some(peer),
reason: format!("unusable cleanup proof: {err}"),
});
continue;
}
}
let target = marker.target.author_id().ok();
if let Err(err) = self.params.storage.put_retirement(marker).await {
self.emit(Event::PluginError {
network: self.network_id,
protocol: "state".into(),
reason: format!("cannot persist cleanup proof: {err}"),
});
continue;
}
self.state = candidate;
if let Some(target) = target
&& self.state.is_retired(&target)
&& !self.sessions.contains_key(&target)
{
self.known_hostnames.remove(&target);
self.dial_states.remove(&target);
self.candidate_addrs.remove(&target);
}
changed = true;
}
if changed {
// A live target answers cleanup with a fresh claim; its identity
// is still authenticated, and old hostname priority is not reused.
self.ensure_own_claim().await;
self.publish_allocations().await;
self.broadcast_retirements();
}
}
/// Signs, stores and announces one of this agent's own records.
///
/// Stored before it is announced, in one transaction with the version
/// counter, so a crash can never let us reuse a version we already put on
/// the wire.
async fn publish_record(&mut self, body: RecordBody) {
let version = self.own_version.saturating_add(1);
let observed = self.state.authored_version(&self.local_id).max(
self.state
.retirement_of(&self.local_id)
.map_or(0, |m| m.target.version),
);
let Some(version) = self
.own_version
.max(observed)
.checked_add(1)
.filter(|v| *v <= i64::MAX as u64)
else {
self.emit(Event::PluginError {
network: self.network_id,
protocol: "state".into(),
reason: "author counter exhausted".into(),
});
return;
};
let record = SignedRecord::sign(&self.params.device_secret, self.network_id, version, body);
if let Err(err) = self.params.storage.publish_own_record(record.clone()).await {
@@ -1145,8 +1498,191 @@ impl Runtime {
self.broadcast_state();
}
fn update_vips(&self) {
let taken = self.state.address_holders().into_keys().collect();
let names = self
.state
.active_records()
.into_iter()
.filter_map(|r| {
r.author_id()
.ok()
.and_then(|p| self.state.hostname_of(&p).map(str::to_owned))
})
.collect();
let views =
self.params
.vips
.views(self.effective_range(), taken, &names, &self.params.limits);
let peers = self
.state
.address_holders()
.into_iter()
.map(|(a, p)| (p, a, self.hub.reachable(p)))
.collect();
self.params.routes.vips.configure(
self.network_id,
self.local_id,
self.state.address_of(&self.local_id),
views,
peers,
&self.params.limits,
);
}
async fn save_vip(
&mut self,
config: crate::vip::VipConfig,
expected: Option<String>,
delete: bool,
) -> Result<Vec<crate::vip::VipStatus>> {
if self.released || self.shutdown.is_triggered() {
return Err(Error::Stopped);
}
let version = self
.own_version
.checked_add(1)
.filter(|n| *n <= i64::MAX as u64)
.ok_or_else(|| crate::vip::invalid("author counter exhausted"))?;
let entries = if delete {
self.params.vips.delete(
self.network_id,
&self.params.device_secret,
version,
&config.id,
expected.as_deref().unwrap_or(""),
)?
} else {
self.params.vips.edit(
self.network_id,
&self.params.device_secret,
version,
&config,
expected.as_deref(),
&self.params.limits,
)?
};
let mut candidate = self.params.vips.clone();
candidate.merge(self.network_id, entries, &self.params.limits)?;
if !delete {
let duplicate = candidate
.configs()
.iter()
.any(|v| v.id != config.id && v.name == config.name)
|| self.state.active_records().iter().any(|r| {
r.author_id().ok().and_then(|p| self.state.hostname_of(&p))
== Some(config.name.as_str())
});
if duplicate {
return Err(crate::vip::invalid(
"VIP name is already used by a node or another VIP",
));
}
}
let snapshot = candidate.entries(&config.id);
self.validate_vip_frame(&snapshot)?;
self.params
.storage
.put_vip_fields(
self.network_id,
snapshot.clone(),
Some((*self.local_id.as_bytes(), version)),
)
.await?;
self.own_version = version;
self.params.vips = candidate;
self.update_vips();
let peers: Vec<_> = self.sessions.keys().copied().collect();
for peer in peers {
let _ = self.send_to(
peer,
ControlMessage::Vips {
entries: snapshot.clone(),
},
);
}
Ok(self.params.routes.vips.statuses(self.network_id))
}
async fn receive_vips(&mut self, entries: Vec<crate::vip::VipEntry>) -> Result<()> {
let Some(id) = entries.first().map(|e| e.vip.clone()) else {
return Ok(());
};
if entries.iter().any(|e| e.vip != id) {
return Err(crate::vip::invalid("VIP snapshot mixes service identities"));
}
let own_seen = entries
.iter()
.filter(|e| e.author == *self.local_id.as_bytes())
.map(|e| e.version)
.max()
.unwrap_or(0);
let mut candidate = self.params.vips.clone();
let changed = candidate.merge(self.network_id, entries, &self.params.limits)?;
if !changed && own_seen <= self.own_version {
return Ok(());
}
let counter =
(own_seen > self.own_version).then_some((*self.local_id.as_bytes(), own_seen));
let snapshot = candidate.entries(&id);
self.validate_vip_frame(&snapshot)?;
self.params
.storage
.put_vip_fields(self.network_id, snapshot, counter)
.await?;
self.own_version = self.own_version.max(own_seen);
self.params.vips = candidate;
self.update_vips();
self.broadcast_vips();
Ok(())
}
fn validate_vip_frame(&self, entries: &[crate::vip::VipEntry]) -> Result<()> {
let frame = Envelope {
network_id: *self.network_id.as_bytes(),
message: ControlMessage::Vips {
entries: entries.to_vec(),
},
};
let bytes = encode(&frame).map_err(|e| crate::vip::invalid(e.to_string()))?;
if bytes.len() > self.params.limits.max_frame_len {
return Err(crate::vip::invalid(
"VIP snapshot exceeds the control frame limit",
));
}
Ok(())
}
fn broadcast_vips(&mut self) {
let ids = self.params.vips.ids();
if ids.is_empty() {
return;
}
let peers: Vec<_> = self.sessions.keys().copied().collect();
for peer in peers {
let start = self.vip_cursor.get(&peer).copied().unwrap_or(0) % ids.len();
for step in 0..ids.len() {
let index = (start + step) % ids.len();
let entries = self.params.vips.entries(&ids[index]);
if self
.send_to(peer, ControlMessage::Vips { entries })
.is_err()
{
self.vip_cursor.insert(peer, index);
break;
}
self.vip_cursor.insert(peer, (index + 1) % ids.len());
}
}
}
/// Handles snapshots collected while dispatching messages.
async fn drain_pending_state(&mut self) {
for entries in std::mem::take(&mut self.pending_vips) {
if let Err(err) = self.receive_vips(entries).await {
self.metrics.protocol_violations += 1;
tracing::warn!(%err,"rejecting VIP snapshot");
}
}
for (peer, markers) in std::mem::take(&mut self.pending_retirements) {
self.receive_retirements(peer, markers).await;
}
for (peer, records) in std::mem::take(&mut self.pending_state) {
self.receive_state(peer, records).await;
}
@@ -1155,6 +1691,8 @@ impl Runtime {
/// Sends everything we know to every peer.
fn broadcast_state(&mut self) {
self.broadcast_vips();
self.broadcast_retirements();
let mut records = self.state.records();
records.truncate(self.params.limits.max_state_records);
if records.is_empty() {
@@ -1352,6 +1890,7 @@ impl Runtime {
self.params.range_changed.send_replace(());
return;
}
self.update_vips();
for plugin in &self.params.plugins {
plugin.on_address_allocation(self.network_id, range, &allocations);
}
@@ -1578,13 +2117,16 @@ impl Runtime {
if self.links.contains_key(&key) || self.opening.contains(&key) {
continue;
}
self.opening.insert(key);
self.opening.insert(key.clone());
let results = self.link_results_tx.clone();
let transport = Arc::clone(&transport);
let network = self.network_id;
let shutdown = self.shutdown.clone();
tokio::spawn(async move {
self.next_link_attempt = self.next_link_attempt.wrapping_add(1);
let generation = self.next_link_attempt;
self.link_attempts.insert(key.clone(), generation);
let task = tokio::spawn(async move {
let result = tokio::select! {
biased;
_ = shutdown.wait() => Err("network deactivated".to_string()),
@@ -1594,12 +2136,14 @@ impl Runtime {
};
let _ = results
.send(LinkOutcome {
generation,
peer,
protocol,
result,
})
.await;
});
self.link_tasks.insert(key, task.abort_handle());
}
}
@@ -1830,11 +2374,20 @@ impl Runtime {
});
}
}
self.update_vips();
}
fn handle_link_result(&mut self, outcome: LinkOutcome) {
let key = (outcome.peer, outcome.protocol.clone());
if self.link_attempts.get(&key) != Some(&outcome.generation) {
if let Ok(link) = outcome.result {
link.close();
}
return;
}
self.link_attempts.remove(&key);
self.opening.remove(&key);
self.link_tasks.remove(&key);
match outcome.result {
Ok(link) => {
self.record_protocol_success(outcome.peer, &outcome.protocol);
@@ -1903,12 +2456,17 @@ impl Runtime {
link.close();
return;
}
if !self
.params
.plugins
.iter()
.any(|plugin| plugin.protocol_id() == protocol)
{
if !self.params.plugins.iter().any(|plugin| {
plugin.protocol_id() == protocol
&& self.sessions.get(&peer).is_some_and(|s| {
s.capabilities.iter().any(|c| {
c.enabled
&& c.protocol == protocol
&& c.version == plugin.protocol_version()
})
})
}) {
link.close();
return;
}
@@ -1966,14 +2524,27 @@ impl Runtime {
/// Drops every link to a peer.
fn drop_links_for(&mut self, peer: EndpointId) {
// The peer itself is going, not just a path to it, so the
// protocol's link goes with it.
self.retirement_cursor.remove(&peer);
self.vip_cursor.remove(&peer);
self.reset_peer_data(peer, "peer session ended");
self.hub.remove_peer(peer);
self.reachable.remove(&peer);
self.protocol_dial_states.retain(|(p, _), _| *p != peer);
self.last_failed_protocol.remove(&peer);
self.pending_health.retain(|(p, _), _| *p != peer);
let keys: Vec<(EndpointId, String)> = self
}
/// Retire only the data plane; authenticated membership and topology remain.
fn reset_peer_data(&mut self, peer: EndpointId, reason: &str) {
self.link_attempts.retain(|(id, _), _| *id != peer);
self.opening.retain(|(id, _)| *id != peer);
self.link_tasks.retain(|(id, _), task| {
if *id == peer {
task.abort();
false
} else {
true
}
});
self.hub.clear_peer_links(peer);
let keys: Vec<_> = self
.links
.keys()
.filter(|(id, _)| *id == peer)
@@ -1987,9 +2558,15 @@ impl Runtime {
network: self.network_id,
peer,
protocol: key.1,
reason: "peer session ended".into(),
reason: reason.into(),
});
}
self.protocol_dial_states.retain(|(id, _), _| *id != peer);
self.last_failed_protocol.remove(&peer);
self.pending_health.retain(|(id, _), _| *id != peer);
for plugin in &self.params.plugins {
plugin.on_peer_gone(self.network_id, peer);
}
}
// --------------------------------------------------------------- sessions
@@ -2182,9 +2759,6 @@ impl Runtime {
self.watch_exit_via();
self.update_paths();
self.announce_reach(false);
for plugin in &self.params.plugins {
plugin.on_peer_gone(self.network_id, peer);
}
// Retry promptly, then back off if it keeps failing.
let policy = &self.params.reconnect;
let state = self
@@ -2277,6 +2851,18 @@ impl Runtime {
}
ControlMessage::Announce(announcement) => {
let capabilities = announcement.capabilities.clone();
let changed = self.sessions.get(&peer).is_some_and(|session| {
let selection = |caps: &[PluginCapability]| {
caps.iter()
.filter(|c| c.enabled)
.map(|c| (c.protocol.clone(), c.version))
.collect::<Vec<_>>()
};
selection(&session.capabilities) != selection(&capabilities)
});
if changed {
self.reset_peer_data(peer, "peer protocol selection changed");
}
if let Some(session) = self.sessions.get_mut(&peer) {
session.hostname = Some(announcement.hostname.clone());
session.broadcast = announcement.broadcast;
@@ -2289,6 +2875,7 @@ impl Runtime {
self.dispatch_capabilities(peer, &capabilities);
self.ensure_links();
self.update_paths();
self.announce_reach(false);
}
ControlMessage::Ping { seq, payload } => {
let pong = ControlMessage::Pong {
@@ -2299,9 +2886,13 @@ impl Runtime {
tracing::debug!(%err, "could not queue pong");
}
}
ControlMessage::Vips { entries } => self.pending_vips.push(entries.clone()),
ControlMessage::State { records } => {
self.pending_state.push((peer, records.clone()));
}
ControlMessage::Retirements { markers } => {
self.pending_retirements.push((peer, markers.clone()));
}
// Members somebody else knows of. Candidates, nothing more:
// each still has to pass the handshake, and each is tried the
// same way as one from any other source.
@@ -2499,12 +3090,13 @@ impl Runtime {
// listing it as one turns leaving into a peer that looks broken.
let mut members: Vec<MemberStatus> = self
.state
.records()
.active_records()
.into_iter()
.filter(|record| !matches!(record.body, RecordBody::Release))
.filter_map(|record| {
let endpoint_id = record.author_id().ok()?;
Some(MemberStatus {
record_version: record.version,
endpoint_id,
// Only uncontested claims are reported as held: two
// members may have claimed the same thing, and saying
@@ -2522,6 +3114,13 @@ impl Runtime {
members.dedup_by(|a, b| a.endpoint_id == b.endpoint_id);
NetworkStatus {
vips: self.params.routes.vips.statuses(self.network_id),
protocols: self
.params
.plugins
.iter()
.map(|p| p.protocol_id().to_owned())
.collect(),
auto_start: true,
broadcast: self.params.broadcast,
exit_offer: self.params.exit_offer,
+6
View File
@@ -108,6 +108,8 @@ pub struct HostnameConflict {
/// connected.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct MemberStatus {
/// Current signed claim version, used to bind cleanup confirmation.
pub record_version: u64,
/// The member's device identity.
pub endpoint_id: EndpointId,
/// The IPv4 overlay address it claimed and signed for.
@@ -223,6 +225,10 @@ pub struct RangeConflict {
/// Status of one network.
#[derive(Debug, Clone)]
pub struct NetworkStatus {
/// Shared service configuration with this client's routing observations.
pub vips: Vec<crate::vip::VipStatus>,
/// Enabled local data-plane protocols, in preference order.
pub protocols: Vec<String>,
/// Whether this network is requested to start or resume automatically.
pub auto_start: bool,
/// Local broadcast participation in this network.
+31
View File
@@ -158,6 +158,22 @@ pub enum TransportPolicy {
/// object where that is possible (notably [`Limits::max_frame_len`]).
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Limits {
/// Maximum active VIP identities per network.
pub max_vips: usize,
/// Maximum permanent VIP deletion tombstones per network.
pub max_vip_tombstones: usize,
/// Maximum concurrent active TCP port checks across the agent.
pub max_vip_checks: usize,
/// Maximum signed fields per VIP, including removed fields.
pub max_vip_fields: usize,
/// Maximum listeners per VIP.
pub max_vip_listeners: usize,
/// Maximum backend endpoints per listener.
pub max_vip_backends: usize,
/// Maximum simultaneous local VIP flow bindings.
pub max_vip_flows: usize,
/// Maximum incomplete IPv4 datagrams per network in VIP reassembly.
pub max_vip_fragments: usize,
/// Largest accepted control frame payload, in bytes.
pub max_frame_len: usize,
/// Largest accepted hostname, in bytes.
@@ -174,6 +190,8 @@ pub struct Limits {
pub max_reason_len: usize,
/// Largest number of signed records accepted in one snapshot.
pub max_state_records: usize,
/// Maximum durable retirement markers retained per network.
pub max_retirements: usize,
/// Deadline for the whole handshake.
pub handshake_timeout: Duration,
/// Deadline for one outbound dial attempt.
@@ -203,6 +221,14 @@ pub struct Limits {
impl Default for Limits {
fn default() -> Self {
Self {
max_vips: 64,
max_vip_tombstones: 1024,
max_vip_checks: 32,
max_vip_fields: 192,
max_vip_listeners: 16,
max_vip_backends: 32,
max_vip_flows: 4096,
max_vip_fragments: 64,
max_frame_len: 64 * 1024,
max_hostname_len: 255,
max_capabilities: 16,
@@ -211,6 +237,7 @@ impl Default for Limits {
max_echo_payload_len: 4 * 1024,
max_reason_len: 256,
max_state_records: crate::state::MAX_RECORDS_PER_MESSAGE,
max_retirements: 1024,
handshake_timeout: Duration::from_secs(10),
dial_timeout: Duration::from_secs(10),
write_timeout: Duration::from_secs(30),
@@ -328,6 +355,9 @@ pub struct AgentConfig {
pub reconnect: ReconnectPolicy,
/// IP plugins whose capabilities are announced and dispatched.
pub plugins: Vec<SharedPlugin>,
/// Initial ordered selection for networks without a saved override.
/// `None` enables all registered plugins; an empty list is control-only.
pub default_protocols: Option<Vec<String>>,
/// Custom packet transports registered per protocol.
pub custom_transports:
std::collections::HashMap<String, Arc<dyn crate::dataplane::PacketTransport>>,
@@ -376,6 +406,7 @@ impl AgentConfig {
limits: Limits::default(),
reconnect: ReconnectPolicy::default(),
plugins: Vec::new(),
default_protocols: None,
custom_transports: std::collections::HashMap::new(),
tun_factory: None,
interface_name: "tsun0".to_string(),
+27
View File
@@ -40,6 +40,33 @@ pub use tunnel::{GenericTunnelPlugin, TunnelCodec};
/// Maximum length of a plugin protocol identifier.
pub const MAX_PROTOCOL_ID_LEN: usize = 32;
/// Validates bounded local protocol preferences before storing or applying them.
pub(crate) fn validate_protocols(protocols: &[String], max: usize) -> crate::error::Result<()> {
if protocols.len() > max {
return Err(crate::Error::InvalidProtocols(format!(
"at most {max} protocols are allowed"
)));
}
for (index, name) in protocols.iter().enumerate() {
if name.is_empty()
|| name.len() > MAX_PROTOCOL_ID_LEN
|| !name
.bytes()
.all(|c| c.is_ascii_alphanumeric() || c == b'-' || c == b'_')
{
return Err(crate::Error::InvalidProtocols(
"names must be 1–32 ASCII letters, digits, hyphens or underscores".into(),
));
}
if protocols[..index].contains(name) {
return Err(crate::Error::InvalidProtocols(format!(
"`{name}` is listed more than once"
)));
}
}
Ok(())
}
/// An announcement of one IP plugin's capability.
///
/// `data` is opaque to the core. Nothing in it may be interpreted as a shell
+10 -2
View File
@@ -492,11 +492,12 @@ impl RelayHub {
}
}
/// Revokes all links and topology involving a departed authenticated peer.
pub fn remove_peer(&self, peer: EndpointId) {
/// Retires data channels without removing an authenticated peer's topology.
pub fn clear_peer_links(&self, peer: EndpointId) {
let mut state = self.state();
if let Some(links) = state.raw.remove(peer.as_bytes()) {
for raw in links {
raw.link.close();
raw.reader.abort();
}
}
@@ -511,6 +512,13 @@ impl RelayHub {
link.close();
}
}
state.publish();
}
/// Revokes all links and topology involving a departed authenticated peer.
pub fn remove_peer(&self, peer: EndpointId) {
self.clear_peer_links(peer);
let mut state = self.state();
state.members.remove(peer.as_bytes());
state.graph.remove(peer.as_bytes());
for neighbors in state.graph.values_mut() {
+6
View File
@@ -15,6 +15,12 @@ pub type Result<T, E = Error> = std::result::Result<T, E>;
#[derive(Debug, thiserror::Error)]
#[non_exhaustive]
pub enum Error {
/// A node-history cleanup is unsafe, stale or cannot be performed.
#[error("cannot forget node: {0}")]
ForgetNode(String),
/// An ordered local data-plane protocol selection is invalid.
#[error("invalid protocol selection: {0}")]
InvalidProtocols(String),
/// The supplied network name does not satisfy the documented rules.
#[error("invalid network name: {0}")]
InvalidNetworkName(&'static str),
+325 -2
View File
@@ -171,6 +171,33 @@ pub fn default_control_socket_path() -> PathBuf {
#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)]
#[non_exhaustive]
pub enum Request {
/// Save a complete version-checked service draft or its deletion.
SaveVip {
/// Exact network id.
network_id: String,
/// Entire desired configuration.
config: crate::vip::VipConfig,
/// Revision originally displayed; absent for creation.
expected: Option<String>,
/// Remove the service permanently.
delete: bool,
},
/// Retire the selected offline member's observed claim.
ForgetNode {
/// Exact public network id.
network_id: String,
/// Exact device identity selected by the client.
peer: String,
/// Claim version displayed when the user selected the node.
expected_version: u64,
},
/// Replace one network's ordered local data-plane selection.
SetProtocols {
/// Public network identifier.
network_id: String,
/// Protocol names, best first; empty selects control only.
protocols: Vec<String>,
},
/// Report what the agent is doing.
Status,
/// Set local broadcast participation in one configured network.
@@ -255,6 +282,24 @@ pub enum Request {
impl std::fmt::Debug for Request {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
Request::SaveVip {
network_id, config, ..
} => write!(f, "SaveVip {{ {network_id}, {} }}", config.id),
Request::ForgetNode {
network_id,
peer,
expected_version,
} => write!(
f,
"ForgetNode {{ {network_id}, peer: {peer}, version: {expected_version} }}"
),
Request::SetProtocols {
network_id,
protocols,
} => write!(
f,
"SetProtocols {{ {network_id}, protocols: {protocols:?} }}"
),
Request::Status => f.write_str("Status"),
Request::SetBroadcast {
network_id,
@@ -286,6 +331,12 @@ impl std::fmt::Debug for Request {
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[non_exhaustive]
pub enum Response {
/// Accepted configuration with fresh local routing observations.
Vips(Vec<crate::vip::VipStatus>),
/// An observed offline claim was durably retired.
NodeForgotten(ForgottenNodeReport),
/// Accepted ordered protocol selection.
Protocols(Vec<String>),
/// A status report.
Status(Box<StatusReport>),
/// Accepted local broadcast participation.
@@ -312,6 +363,192 @@ pub enum Response {
Error(String),
}
/// The durable result of an offline-node cleanup.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct ForgottenNodeReport {
/// Selected device identity.
pub peer: String,
/// The retired claim version, not a future-version ban.
pub record_version: u64,
/// Peers to which cleanup was immediately queued.
pub peers_told: u32,
}
/// Saves a whole VIP draft through the running agent.
pub async fn save_vip(
socket: &Path,
network_id: &str,
config: crate::vip::VipConfig,
expected: Option<String>,
delete: bool,
) -> Result<Vec<crate::vip::VipStatus>> {
match exchange(
socket,
&Request::SaveVip {
network_id: network_id.into(),
config,
expected,
delete,
},
EXCHANGE_TIMEOUT,
)
.await?
{
Response::Vips(vips) => Ok(vips),
Response::Error(reason) => Err(Error::Rejected { reason }),
_ => Err(Error::Rejected {
reason: "unexpected VIP response".into(),
}),
}
}
/// Resolves a service by exact name, full id or unique id prefix.
pub fn resolve_vip<'a>(
network: &'a NetworkReport,
selector: &str,
) -> std::result::Result<&'a crate::vip::VipStatus, String> {
let matches: Vec<_> = network
.vips
.iter()
.filter(|v| v.config.name == selector || v.config.id.starts_with(selector))
.collect();
match matches.as_slice() {
[vip] => Ok(vip),
[] => Err(format!("no VIP matches `{selector}`")),
_ => Err(format!(
"ambiguous VIP `{selector}`: {}",
matches
.iter()
.map(|v| format!("{} ({})", v.config.name, v.config.id))
.collect::<Vec<_>>()
.join(", ")
)),
}
}
/// Resolves a node by exact hostname, full id, unique id prefix or overlay IPv4.
/// Ambiguity is reported with names and full ids and never guessed.
pub fn resolve_node<'a>(
network: &'a NetworkReport,
selector: &str,
) -> std::result::Result<&'a MemberReport, String> {
let selector = selector.trim();
if selector.is_empty() {
return Err("select a node by name, id or IPv4 address".into());
}
if let Some(member) = network.members.iter().find(|m| m.endpoint_id == selector) {
return Ok(member);
}
let prefix = selector.trim_end_matches('…');
let matches: Vec<_> = network
.members
.iter()
.filter(|m| {
m.hostname.as_deref() == Some(selector)
|| m.overlay_address_v4.as_deref() == Some(selector)
|| (!prefix.is_empty() && m.endpoint_id.starts_with(prefix))
})
.collect();
match matches.as_slice() {
[one] => Ok(one),
[] => Err(format!(
"no signed node matches `{selector}` in `{}`",
network.name
)),
many => Err(format!(
"`{selector}` is ambiguous; use a full id or unique id prefix:\n{}",
many.iter()
.map(|m| format!(
" {} {} {}",
m.hostname.as_deref().unwrap_or("unnamed"),
m.endpoint_id,
m.overlay_address_v4.as_deref().unwrap_or("no IPv4")
))
.collect::<Vec<_>>()
.join("\n")
)),
}
}
/// Retires an exact selected claim; an online or changed target is rejected.
pub async fn forget_node(
socket: &Path,
network: &str,
peer: &str,
expected_version: u64,
) -> Result<ForgottenNodeReport> {
let request = Request::ForgetNode {
network_id: network.into(),
peer: peer.into(),
expected_version,
};
match exchange(socket, &request, EXCHANGE_TIMEOUT).await? {
Response::NodeForgotten(report) => Ok(report),
Response::Error(reason) => Err(Error::ForgetNode(reason)),
other => Err(Error::Storage(format!("unexpected response: {other:?}"))),
}
}
#[cfg(test)]
mod node_selector_tests {
use super::*;
#[test]
fn exact_names_addresses_and_unique_prefixes_select_the_same_signed_node() {
let node = MemberReport {
endpoint_id: "abcdef01".into(),
hostname: Some("laptop".into()),
overlay_address_v4: Some("10.2.0.3".into()),
record_version: 42,
..Default::default()
};
let network = NetworkReport {
name: "home".into(),
members: vec![node.clone()],
..Default::default()
};
for selector in ["laptop", "10.2.0.3", "abcdef01", "abc…"] {
assert_eq!(resolve_node(&network, selector), Ok(&node));
}
for selector in ["", "LAPTOP", "missing"] {
assert!(resolve_node(&network, selector).is_err());
}
}
#[test]
fn collisions_between_names_addresses_and_prefixes_require_an_explicit_identity() {
let first = MemberReport {
endpoint_id: "abcdef01".into(),
hostname: Some("duplicate".into()),
overlay_address_v4: Some("10.2.0.3".into()),
..Default::default()
};
let second = MemberReport {
endpoint_id: "abcdef02".into(),
hostname: Some("duplicate".into()),
overlay_address_v4: Some("10.2.0.3".into()),
..Default::default()
};
let third = MemberReport {
endpoint_id: "12345678".into(),
hostname: Some("abc".into()),
..Default::default()
};
let network = NetworkReport {
name: "home".into(),
members: vec![first.clone(), second, third],
..Default::default()
};
for selector in ["duplicate", "10.2.0.3", "abc"] {
let result = resolve_node(&network, selector);
assert!(result.is_err_and(|error| error.contains("ambiguous")
&& error.contains("abcdef01")
&& error.contains("abcdef02")
&& error.contains("duplicate")));
}
assert_eq!(resolve_node(&network, "abcdef01"), Ok(&first));
}
}
/// What happened when a network was stopped or started.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct ActiveReport {
@@ -377,6 +614,8 @@ pub struct LeftReport {
/// Everything the agent is doing, in one snapshot.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct StatusReport {
/// Data-plane protocols this agent can enable at runtime.
pub available_protocols: Vec<ProtocolReport>,
/// This device's persistent endpoint id.
pub endpoint_id: String,
/// Hostname announced to peers.
@@ -458,9 +697,23 @@ pub struct RangeConflictReport {
pub range: String,
}
/// A data-plane protocol available in the running agent.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct ProtocolReport {
/// Stable protocol name used by the CLI and network negotiation.
pub name: String,
/// A short description of encryption and transport.
pub summary: String,
}
/// One network.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct NetworkReport {
/// Shared VIPs and this client's routing observations.
#[serde(default)]
pub vips: Vec<crate::vip::VipStatus>,
/// Enabled local data-plane protocols, in preference order.
pub protocols: Vec<String>,
/// Whether the network starts or resumes automatically when its range is free.
#[serde(default)]
pub auto_start: bool,
@@ -551,6 +804,8 @@ pub struct HostnameConflictReport {
/// One member of the network, from signed state.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct MemberReport {
/// Signed claim version used to reject stale cleanup requests.
pub record_version: u64,
/// The member's device identity.
pub endpoint_id: String,
/// The IPv4 overlay address it claimed and signed for.
@@ -720,6 +975,33 @@ impl OverlayPeerReport {
/// running and what they can report or change. That is what keeps this module
/// free of any knowledge of them.
pub trait ReportSource: Send + Sync + 'static {
/// Atomically saves a complete VIP draft using its original revision.
fn save_vip(
&self,
_network: String,
_config: crate::vip::VipConfig,
_expected: Option<String>,
_delete: bool,
) -> BoxFuture<'_, std::result::Result<Vec<crate::vip::VipStatus>, String>> {
Box::pin(async { Err("this agent cannot manage VIPs".into()) })
}
/// Persists and announces a version-scoped node cleanup.
fn forget_node(
&self,
_network: String,
_peer: String,
_expected_version: u64,
) -> BoxFuture<'_, std::result::Result<ForgottenNodeReport, String>> {
Box::pin(async { Err("this agent cannot forget node history".into()) })
}
/// Saves and applies a local ordered protocol set for one network.
fn set_protocols(
&self,
_network_id: String,
_protocols: Vec<String>,
) -> BoxFuture<'_, std::result::Result<Vec<String>, String>> {
Box::pin(async { Err("this agent cannot change protocols".into()) })
}
/// Produces a fresh report.
fn report(&self) -> BoxFuture<'_, StatusReport>;
@@ -849,7 +1131,7 @@ pub const EXCHANGE_TIMEOUT: Duration = Duration::from_secs(5);
///
/// Bump it whenever [`Request`], [`Response`] or anything they contain
/// changes shape.
pub const CONTROL_PROTOCOL: u32 = u32::from_be_bytes([b'T', b'S', b'N', 19]);
pub const CONTROL_PROTOCOL: u32 = u32::from_be_bytes([b'T', b'S', b'N', 22]);
/// Reads one request off an accepted stream, answers it, writes the response.
///
@@ -873,6 +1155,30 @@ where
}
};
let response = match request {
Request::SaveVip {
network_id,
config,
expected,
delete,
} => match source.save_vip(network_id, config, expected, delete).await {
Ok(vips) => Response::Vips(vips),
Err(reason) => Response::Error(reason),
},
Request::ForgetNode {
network_id,
peer,
expected_version,
} => match source.forget_node(network_id, peer, expected_version).await {
Ok(report) => Response::NodeForgotten(report),
Err(reason) => Response::Error(reason),
},
Request::SetProtocols {
network_id,
protocols,
} => match source.set_protocols(network_id, protocols).await {
Ok(protocols) => Response::Protocols(protocols),
Err(reason) => Response::Error(reason),
},
Request::Status => Response::Status(Box::new(source.report().await)),
Request::SetBroadcast {
network_id,
@@ -1028,7 +1334,24 @@ pub async fn join_network_with_broadcast(
}
}
/// Changes one network's broadcast participation while the agent runs.
/// Replaces the enabled data-plane protocols of one configured network.
pub async fn set_protocols(
socket: &Path,
network_id: &str,
protocols: Vec<String>,
) -> Result<Vec<String>> {
let request = Request::SetProtocols {
network_id: network_id.to_owned(),
protocols,
};
match exchange(socket, &request, EXCHANGE_TIMEOUT).await? {
Response::Protocols(protocols) => Ok(protocols),
Response::Error(reason) => Err(Error::Storage(reason)),
other => Err(Error::Storage(format!("unexpected response: {other:?}"))),
}
}
/// Changes local broadcast participation in a network.
pub async fn set_broadcast(
path: impl AsRef<Path>,
network_id: &str,
+1
View File
@@ -51,6 +51,7 @@ pub mod storage;
pub mod task;
#[cfg(feature = "testing")]
pub mod testing;
pub mod vip;
pub use agent::{Agent, AgentStatus, Event, NetworkStatus, PeerStatus};
pub use config::{AgentConfig, Limits, ReconnectPolicy, StoragePaths, TransportPolicy};
@@ -84,6 +84,11 @@ pub trait ExitHostRules: Send + Sync + std::fmt::Debug + 'static {
/// startup to take away what a crashed run left.
fn clear<'a>(&'a self, interface: &'a str) -> BoxFuture<'a, ()>;
/// Refreshes volatile local socket protection without replacing exit routes.
fn refresh<'a>(&'a self, _plan: &'a ExitHostPlan) -> BoxFuture<'a, Option<RuleOutcome>> {
Box::pin(async { None })
}
/// Whether using an exit node on this host needs the underlay addresses
/// the agent talks to, because the host cannot exempt the agent's own
/// traffic from the default route by user id (Windows). The default is no.
@@ -33,12 +33,13 @@ const CHAIN: &str = "INPUT";
/// Fixed places `iptables` is installed. Never `PATH`: the agent is
/// privileged and a directory a user controls must not decide what it runs.
const IPTABLES_DIRECTORIES: [&str; 5] = [
const IPTABLES_DIRECTORIES: [&str; 6] = [
"/usr/sbin/iptables",
"/sbin/iptables",
"/usr/bin/iptables",
"/bin/iptables",
"/usr/local/sbin/iptables",
"/run/current-system/sw/bin/iptables",
];
/// The first candidate that exists.
@@ -328,7 +329,23 @@ fn clear_firewall(run: Run<'_>, interface: &str) {
/// Runs the real `iptables`.
pub(super) fn run_iptables(args: &[String]) -> Ran {
let Some(program) = pick_iptables(|path| std::path::Path::new(path).is_file()) else {
run_firewall(false, args)
}
pub(super) fn run_firewall(ipv6: bool, args: &[String]) -> Ran {
let v6_paths: Vec<_> = IPTABLES_DIRECTORIES
.iter()
.map(|p| p.replace("iptables", "ip6tables"))
.collect();
let v6_program = v6_paths
.iter()
.find(|path| std::path::Path::new(path).is_file());
let program = if ipv6 {
v6_program.map(String::as_str)
} else {
pick_iptables(|path| std::path::Path::new(path).is_file())
};
let Some(program) = program else {
return Ran::Missing;
};
match std::process::Command::new(program).args(args).output() {
@@ -350,11 +367,17 @@ pub(super) fn run_iptables(args: &[String]) -> Ran {
#[derive(Debug)]
pub struct LinuxHostRules {
pub(super) route: RouteHandle,
/// Interface, sorted socket snapshot and whether exit routes committed.
pub(super) underlay_sockets:
std::sync::Mutex<Option<(String, Vec<super::linux_exit_mark::Listener>, bool)>>,
}
impl LinuxHostRules {
pub(crate) fn new(route: RouteHandle) -> Self {
Self { route }
Self {
route,
underlay_sockets: std::sync::Mutex::new(None),
}
}
async fn set_route(&self, interface: String, source: Ipv4Addr) -> RuleOutcome {
@@ -25,18 +25,19 @@
//!
//! ```text
//! table 0x7473: default dev <if>
//! 5280: from all uidrange <euid>-<euid> lookup main
//! 5205: from all fwmark 0x40000000/0x40000000 lookup main
//! 5206: from all uidrange <euid>-<euid> lookup main
//! 5290: from all lookup main suppress_prefixlength 0
//! 5300: from all lookup 0x7473
//! ```
//!
//! The first keeps the agent's *own* sockets off the tunnel: its QUIC, relay,
//! The first two keep the agent's *own* sockets and marked replies off the tunnel: its QUIC, relay,
//! DHT and DNS traffic would otherwise be routed into the interface it is
//! carrying, a loop. The second lets every specific route — the LAN, another
//! carrying, a loop. The specific-route rule lets the LAN, another
//! VPN, a tunnel to somewhere else — keep working, because only the default
//! route is suppressed. The priorities sit after Tailscale's 5270.
//! route is suppressed. Mark/UID bypass precedes Tailscale; application rules follow it.
//!
//! The first rule exempts a *user id*, so it is only right when the agent has
//! The UID rule exempts a *user id*, so it is only right when the agent has
//! one of its own, as the packaged service does. An agent started from a login
//! session would take that user's own applications out of the tunnel with it,
//! and the exit node would silently do nothing for them. That is refused, not
@@ -48,9 +49,10 @@ use crate::state::Ipv4Range;
use super::RuleOutcome;
use super::exit::{ExitHostPlan, ExitHostReport, ExitHostRules, exit_tag, exit_tag_prefix};
use super::linux::{
LinuxHostRules, Ran, Run, comment_supported, failure, locked, run_iptables, same_rule,
split_rule_line,
LinuxHostRules, Ran, Run, comment_supported, failure, locked, run_firewall, run_iptables,
same_rule, split_rule_line,
};
use super::linux_exit_mark;
/// The chains the offering rules live in: `(table, chain)`.
const CHAINS: [(&str, &str); 2] = [("nat", "POSTROUTING"), ("filter", "FORWARD")];
@@ -377,6 +379,32 @@ fn read_sysctl(path: &str) -> Option<String> {
std::fs::read_to_string(path).ok()
}
fn check_kernel_rpf(overlay: &str) -> Result<(), String> {
let directory = "/proc/sys/net/ipv4/conf";
let value = |name: &str, setting: &str| -> Result<u32, String> {
read_sysctl(&format!("{directory}/{name}/{setting}"))
.and_then(|s| s.trim().parse().ok())
.ok_or_else(|| format!("cannot check {name}/{setting} before installing exit routes"))
};
let mut interfaces = Vec::new();
for entry in std::fs::read_dir(directory).map_err(|err| err.to_string())? {
let entry = entry.map_err(|err| err.to_string())?;
let name = entry.file_name().to_string_lossy().into_owned();
if name != "all" && name != "default" && name != "lo" && name != overlay {
interfaces.push((
name.clone(),
value(&name, "rp_filter")?,
value(&name, "src_valid_mark")?,
));
}
}
linux_exit_mark::check_kernel_rpf(
value("all", "rp_filter")?,
value("all", "src_valid_mark")?,
&interfaces,
)
}
/// The effective user id from the text of `/proc/self/status`.
fn parse_euid(status: &str) -> Option<u32> {
status
@@ -417,7 +445,7 @@ impl LinuxHostRules {
let xdg = std::env::var("XDG_RUNTIME_DIR").ok();
if let Some(reason) = client_refusal(xdg.as_deref()) {
// Whatever an earlier run left must not stay in force.
self.clear_client().await;
let _ = self.clear_client(interface).await;
return RuleOutcome::Failed(reason);
}
let Some(euid) = std::fs::read_to_string("/proc/self/status")
@@ -426,24 +454,147 @@ impl LinuxHostRules {
else {
return RuleOutcome::Failed("cannot tell which user this agent runs as".to_string());
};
let installed = self
.underlay_sockets
.lock()
.unwrap_or_else(|p| p.into_inner())
.as_ref()
.is_some_and(|(name, _, routed)| name == interface && *routed);
if installed {
// A failed socket refresh must never remove an active exit route.
return self
.refresh_sockets(interface)
.await
.unwrap_or(RuleOutcome::Applied);
}
if let Err(reason) = self.clear_client(interface).await {
return RuleOutcome::Failed(reason);
}
let name = interface.to_string();
let prepared = Self::blocking(move || {
check_kernel_rpf(&name)?;
linux_exit_mark::install(
&run_firewall,
&name,
euid,
std::path::Path::new("/proc/sys/net/ipv6").is_dir(),
)
})
.await;
match prepared {
Ok(Ok(sockets)) => {
*self
.underlay_sockets
.lock()
.unwrap_or_else(|p| p.into_inner()) = Some((interface.into(), sockets, false));
}
Ok(Err(reason)) | Err(reason) => return RuleOutcome::Failed(reason),
}
let route = self.route.clone();
let interface = interface.to_string();
match Self::blocking(move || route.set_exit_client(&interface, euid)).await {
let name = interface.to_string();
let outcome = match Self::blocking(move || route.set_exit_client(&name, euid)).await {
Ok(Ok(())) => RuleOutcome::Applied,
Ok(Err(err)) => RuleOutcome::Failed(err.to_string()),
Err(reason) => RuleOutcome::Failed(reason),
};
if let RuleOutcome::Failed(reason) = outcome {
let reason = match self.clear_client(interface).await {
Ok(()) => reason,
Err(cleanup) => format!("{reason}; cannot roll back exit rules: {cleanup}"),
};
RuleOutcome::Failed(reason)
} else {
if let Some((_, _, routed)) = self
.underlay_sockets
.lock()
.unwrap_or_else(|p| p.into_inner())
.as_mut()
{
*routed = true;
}
outcome
}
}
async fn clear_client(&self) {
async fn clear_client(&self, interface: &str) -> Result<(), String> {
let route = self.route.clone();
if let Ok(Err(err)) = Self::blocking(move || route.clear_exit_client()).await {
tracing::warn!(%err, "cannot remove the exit node routing rules");
Self::blocking(move || route.clear_exit_client())
.await?
.map_err(|err| err.to_string())?;
// Route removal must precede removing reply protection.
let name = interface.to_string();
Self::blocking(move || {
linux_exit_mark::clear(
&run_firewall,
&name,
std::path::Path::new("/proc/sys/net/ipv6").is_dir(),
)
})
.await??;
*self
.underlay_sockets
.lock()
.unwrap_or_else(|p| p.into_inner()) = None;
Ok(())
}
async fn refresh_sockets(&self, interface: &str) -> Option<RuleOutcome> {
let previous = self
.underlay_sockets
.lock()
.unwrap_or_else(|p| p.into_inner())
.clone();
if previous
.as_ref()
.is_none_or(|(name, _, routed)| name != interface || !routed)
{
return None;
}
let name = interface.to_string();
let work_name = name.clone();
let result = Self::blocking(move || {
let uid = std::fs::read_to_string("/proc/self/status")
.ok()
.and_then(|s| parse_euid(&s))
.ok_or("cannot read agent UID")?;
check_kernel_rpf(&work_name)?;
let v6 = std::path::Path::new("/proc/sys/net/ipv6").is_dir();
let sockets = linux_exit_mark::local_sockets(uid, v6)?;
if previous
.as_ref()
.is_some_and(|(name, old, _)| name == &work_name && old == &sockets)
{
return Ok(None);
}
linux_exit_mark::refresh(&run_firewall, &work_name, v6, &sockets)?;
Ok(Some(sockets))
})
.await;
match result {
Ok(Ok(None)) => None,
Ok(Ok(Some(sockets))) => {
*self
.underlay_sockets
.lock()
.unwrap_or_else(|p| p.into_inner()) = Some((name, sockets, true));
Some(RuleOutcome::Applied)
}
Ok(Err(reason)) | Err(reason) => Some(RuleOutcome::Failed(reason)),
}
}
}
impl ExitHostRules for LinuxHostRules {
fn refresh<'a>(&'a self, plan: &'a ExitHostPlan) -> BoxFuture<'a, Option<RuleOutcome>> {
Box::pin(async move {
if plan.client {
self.refresh_sockets(&plan.interface).await
} else {
None
}
})
}
fn apply<'a>(&'a self, plan: &'a ExitHostPlan) -> BoxFuture<'a, ExitHostReport> {
Box::pin(async move {
let interface = plan.interface.clone();
@@ -467,7 +618,9 @@ impl ExitHostRules for LinuxHostRules {
let client = if plan.client {
Some(self.apply_client(&interface).await)
} else {
self.clear_client().await;
if let Err(reason) = self.clear_client(&interface).await {
tracing::warn!(%reason, "cannot remove exit-client rules");
}
None
};
ExitHostReport {
@@ -482,7 +635,9 @@ impl ExitHostRules for LinuxHostRules {
Box::pin(async move {
let name = interface.to_string();
let _ = Self::blocking(move || clear_offer(&run_iptables, &name)).await;
self.clear_client().await;
if let Err(reason) = self.clear_client(interface).await {
tracing::warn!(%reason, "cannot remove exit-client rules");
}
})
}
}
@@ -0,0 +1,439 @@
//! Privileged routing regression, isolated from the host in named namespaces.
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use std::io::{BufRead, BufReader, Read, Write};
use std::net::{SocketAddr, TcpListener, TcpStream, UdpSocket};
use std::os::unix::process::CommandExt;
use std::process::{Child, Command};
use std::time::Duration;
use super::linux::run_firewall;
use super::linux_exit_mark::{clear, install, local_sockets, refresh};
use crate::overlay::NetlinkProvisioner;
const TEST: &str = "overlay::hostrules::linux_exit_kernel::strict_firewall_preserves_underlay_and_application_exit_routing";
const ROLE: &str = "TSUNAGI_EXIT_KERNEL_TEST_ROLE";
fn ip(arguments: &[&str]) -> String {
let result = Command::new("ip").args(arguments).output().unwrap();
assert!(
result.status.success(),
"ip {arguments:?}: {}",
String::from_utf8_lossy(&result.stderr)
);
String::from_utf8_lossy(&result.stdout).into_owned()
}
struct Namespace(String);
impl Drop for Namespace {
fn drop(&mut self) {
let _ = Command::new("ip").args(["netns", "del", &self.0]).status();
}
}
struct Echo(Child);
impl Drop for Echo {
fn drop(&mut self) {
let _ = self.0.kill();
let _ = self.0.wait();
}
}
fn probes() {
let should_pass = std::env::var("TSUNAGI_EXIT_KERNEL_EXPECT").unwrap() == "pass";
for address in ["198.51.100.2:42000", "[2001:db8:2::2]:42000"] {
let address: SocketAddr = address.parse().unwrap();
let socket = UdpSocket::bind(if address.is_ipv4() {
"0.0.0.0:0"
} else {
"[::]:0"
})
.unwrap();
socket
.set_read_timeout(Some(Duration::from_secs(1)))
.unwrap();
socket.send_to(b"rpf", address).unwrap();
let mut packet = [0u8; 3];
assert_eq!(
socket.recv(&mut packet).is_ok(),
should_pass,
"UDP {address}"
);
if should_pass {
assert_eq!(&packet, b"rpf");
}
let connected = TcpStream::connect_timeout(&address, Duration::from_secs(1));
assert_eq!(connected.is_ok(), should_pass, "TCP {address}");
if let Ok(mut stream) = connected {
stream
.set_read_timeout(Some(Duration::from_secs(1)))
.unwrap();
stream.write_all(b"rpf").unwrap();
stream.read_exact(&mut packet).unwrap();
assert_eq!(&packet, b"rpf");
}
}
}
fn run_probes(expect: &str) {
let result = Command::new(std::env::current_exe().unwrap())
.args(["--ignored", "--exact", TEST, "--nocapture"])
.env(ROLE, "probe")
.env("TSUNAGI_EXIT_KERNEL_EXPECT", expect)
.uid(990)
.gid(990)
.output()
.unwrap();
assert!(
result.status.success(),
"{}\n{}",
String::from_utf8_lossy(&result.stdout),
String::from_utf8_lossy(&result.stderr)
);
}
fn passive_listener() {
let listeners: Vec<_> = ["192.0.2.1:42010", "[2001:db8:1::1]:42010"]
.into_iter()
.map(|address| TcpListener::bind(address).unwrap())
.collect();
println!("PASSIVE_READY");
let tasks: Vec<_> = listeners
.into_iter()
.map(|listener| {
std::thread::spawn(move || {
let (mut socket, _) = listener.accept().unwrap();
socket
.set_read_timeout(Some(Duration::from_secs(2)))
.unwrap();
let mut bytes = [0; 3];
socket.read_exact(&mut bytes).unwrap();
socket.write_all(&bytes).unwrap();
})
})
.collect();
for task in tasks {
task.join().unwrap();
}
}
fn udp_listener() {
let sockets: Vec<_> = ["192.0.2.1:42011", "[2001:db8:1::1]:42011"]
.into_iter()
.map(|a| UdpSocket::bind(a).unwrap())
.collect();
println!("PASSIVE_READY");
let tasks: Vec<_> = sockets
.into_iter()
.map(|socket| {
std::thread::spawn(move || {
socket
.set_read_timeout(Some(Duration::from_secs(5)))
.unwrap();
let mut bytes = [0; 3];
let (len, peer) = socket.recv_from(&mut bytes).unwrap();
socket.send_to(&bytes[..len], peer).unwrap();
})
})
.collect();
for task in tasks {
task.join().unwrap();
}
}
fn spawn_listener(role: &str) -> Echo {
let process = Command::new(std::env::current_exe().unwrap())
.args(["--ignored", "--exact", TEST, "--nocapture"])
.env(ROLE, role)
.uid(990)
.gid(990)
.stdout(std::process::Stdio::piped())
.spawn()
.unwrap();
let mut child = Echo(process);
let mut output = BufReader::new(child.0.stdout.take().unwrap());
loop {
let mut line = String::new();
assert_ne!(
output.read_line(&mut line).unwrap(),
0,
"listener exited before readiness"
);
if line.contains("PASSIVE_READY") {
// Keep the reader open for libtest's final status after the socket
// worker finishes; closing it here makes success a broken pipe.
child.0.stdout = Some(output.into_inner());
return child;
}
}
}
fn client() {
for setting in ["rp_filter", "src_valid_mark"] {
std::fs::write(format!("/proc/sys/net/ipv4/conf/all/{setting}"), b"1").unwrap();
}
std::fs::write("/proc/sys/net/ipv4/conf/rpf-c/rp_filter", b"1").unwrap();
for family in [false, true] {
let args: Vec<_> = [
"-t",
"mangle",
"-A",
"PREROUTING",
"-m",
"rpfilter",
"--invert",
"--validmark",
"-j",
"DROP",
]
.into_iter()
.map(str::to_string)
.collect();
let result = run_firewall(family, &args);
assert!(
matches!(result, super::linux::Ran::Exited { success: true, .. }),
"{result:?}"
);
}
let provisioner = NetlinkProvisioner::new().unwrap();
let route = provisioner.host_rules().route;
// Model Tailscale's earlier lookup, without running another VPN daemon.
ip(&["-4", "rule", "add", "priority", "5270", "table", "52"]);
ip(&[
"-4", "route", "add", "default", "dev", "tsun0", "table", "52",
]);
ip(&["-6", "rule", "add", "priority", "5270", "table", "52"]);
ip(&[
"-6",
"route",
"add",
"unreachable",
"default",
"table",
"52",
]);
// A foreign VPN must leave neighbour discovery and connected LAN paths
// usable; otherwise the fixture itself cannot resolve its uplink neighbour.
for prefix in ["fe80::/64", "2001:db8:1::/64"] {
ip(&["-6", "route", "add", "throw", prefix, "table", "52"]);
}
let mut passive = spawn_listener("listener");
route.set_exit_client("tsun0", 990).unwrap();
// uid bypass alone cannot protect reverse lookups, reproducing the bug.
run_probes("blocked");
install(&run_firewall, "tsun0", 990, true).unwrap();
run_probes("pass");
// Pair transports may bind fresh UDP ports after exit activation.
let mut dynamic_udp = spawn_listener("udp-listener");
refresh(
&run_firewall,
"tsun0",
true,
&local_sockets(990, true).unwrap(),
)
.unwrap();
let server = std::env::var("TSUNAGI_EXIT_KERNEL_SERVER").unwrap();
let incoming = Command::new("ip").args(["netns", "exec", &server, "python3", "-c", r#"import socket
for family,source,target in [(socket.AF_INET,'198.51.100.2','192.0.2.1'),(socket.AF_INET6,'2001:db8:2::2','2001:db8:1::1')]:
for kind,port in [(socket.SOCK_STREAM,42010),(socket.SOCK_DGRAM,42011)]:
s=socket.socket(family,kind); s.settimeout(2); s.bind((source,0)); s.connect((target,port)); s.sendall(b'rpf'); assert s.recv(3)==b'rpf'; s.close()
"#]).output().unwrap();
assert!(
incoming.status.success(),
"first inbound TCP/UDP must be protected: {}",
String::from_utf8_lossy(&incoming.stderr)
);
assert!(passive.0.wait().unwrap().success());
assert!(dynamic_udp.0.wait().unwrap().success());
// Repeated apply remains usable, and does not redirect application sockets.
install(&run_firewall, "tsun0", 990, true).unwrap();
run_probes("pass");
assert!(ip(&["-4", "route", "get", "198.51.100.2", "uid", "991"]).contains("dev tsun0"));
let blocked = Command::new("ip")
.args(["-6", "route", "get", "2001:db8:2::2", "uid", "991"])
.output()
.unwrap();
assert!(
!blocked.status.success(),
"application IPv6 must stay blocked"
);
route.clear_exit_client().unwrap();
clear(&run_firewall, "tsun0", true).unwrap();
for family in ["-4", "-6"] {
assert!(
ip(&[family, "rule", "show"]).contains("5270:"),
"foreign routing rules survive cleanup"
);
ip(&[family, "rule", "del", "priority", "5270"]);
}
assert!(ip(&["-4", "route", "get", "198.51.100.2", "uid", "991"]).contains("dev rpf-c"));
assert!(ip(&["-6", "route", "get", "2001:db8:2::2", "uid", "991"]).contains("dev rpf-c"));
for family in [false, true] {
let result = run_firewall(family, &["-t".into(), "mangle".into(), "-S".into()]);
if let super::linux::Ran::Exited { stdout, .. } = result {
assert!(!stdout.contains("tsunagi-exit-underlay:"));
assert!(!stdout.contains("tsunagi-exit-sockets:"));
assert!(
stdout.contains("rpfilter"),
"strict host filtering is retained"
);
}
}
}
#[test]
#[ignore = "needs root, iproute2, iptables/ip6tables and Python; changes only temporary namespaces"]
fn strict_firewall_preserves_underlay_and_application_exit_routing() {
match std::env::var(ROLE).as_deref() {
Ok("probe") => {
probes();
return;
}
Ok("client") => {
client();
return;
}
Ok("listener") => {
passive_listener();
return;
}
Ok("udp-listener") => {
udp_listener();
return;
}
_ => {}
}
let suffix = std::process::id();
let client_name = format!("tsng-rpf-c-{suffix}");
let server_name = format!("tsng-rpf-s-{suffix}");
ip(&["netns", "add", &client_name]);
let _client = Namespace(client_name.clone());
ip(&["netns", "add", &server_name]);
let _server = Namespace(server_name.clone());
ip(&[
"link",
"add",
"rpf-c",
"netns",
&client_name,
"type",
"veth",
"peer",
"name",
"rpf-s",
"netns",
&server_name,
]);
for (namespace, interface, v4, v6) in [
(&client_name, "rpf-c", "192.0.2.1/24", "2001:db8:1::1/64"),
(&server_name, "rpf-s", "192.0.2.2/24", "2001:db8:1::2/64"),
] {
ip(&["-n", namespace, "link", "set", "lo", "up"]);
ip(&["-n", namespace, "addr", "add", v4, "dev", interface]);
ip(&[
"-n", namespace, "-6", "addr", "add", v6, "dev", interface, "nodad",
]);
ip(&["-n", namespace, "link", "set", interface, "up"]);
}
ip(&[
"-n",
&server_name,
"addr",
"add",
"198.51.100.2/32",
"dev",
"lo",
]);
ip(&[
"-n",
&server_name,
"-6",
"addr",
"add",
"2001:db8:2::2/128",
"dev",
"lo",
"nodad",
]);
ip(&[
"-n",
&client_name,
"route",
"add",
"default",
"via",
"192.0.2.2",
]);
ip(&[
"-n",
&client_name,
"-6",
"route",
"add",
"default",
"via",
"2001:db8:1::2",
]);
ip(&["-n", &client_name, "link", "add", "tsun0", "type", "dummy"]);
ip(&[
"-n",
&client_name,
"addr",
"add",
"10.13.37.1/24",
"dev",
"tsun0",
]);
ip(&["-n", &client_name, "link", "set", "tsun0", "up"]);
let echo_code = r#"import socket,selectors,sys
selector=selectors.DefaultSelector()
for family in [socket.AF_INET,socket.AF_INET6]:
for kind in [socket.SOCK_DGRAM,socket.SOCK_STREAM]:
sock=socket.socket(family,kind)
if family==socket.AF_INET6: sock.setsockopt(socket.IPPROTO_IPV6,socket.IPV6_V6ONLY,1)
sock.bind(('198.51.100.2' if family==socket.AF_INET else '2001:db8:2::2',42000))
if kind==socket.SOCK_STREAM: sock.listen()
selector.register(sock,selectors.EVENT_READ,kind)
print('ready',flush=True)
while True:
for key,_ in selector.select():
if key.data==socket.SOCK_DGRAM:
data,address=key.fileobj.recvfrom(128); key.fileobj.sendto(data,address)
else:
conn,_=key.fileobj.accept(); conn.settimeout(2)
try: conn.sendall(conn.recv(128))
finally: conn.close()
"#;
let child = Command::new("ip")
.args([
"netns",
"exec",
&server_name,
"python3",
"-u",
"-c",
echo_code,
])
.stdout(std::process::Stdio::piped())
.spawn()
.unwrap();
let mut echo = Echo(child);
let mut ready = String::new();
BufReader::new(echo.0.stdout.take().unwrap())
.read_line(&mut ready)
.unwrap();
assert_eq!(ready.trim(), "ready");
let result = Command::new("ip")
.args(["netns", "exec", &client_name])
.arg(std::env::current_exe().unwrap())
.args(["--ignored", "--exact", TEST, "--nocapture"])
.env(ROLE, "client")
.env("TSUNAGI_EXIT_KERNEL_SERVER", &server_name)
.output()
.unwrap();
assert!(
result.status.success(),
"{}\n{}",
String::from_utf8_lossy(&result.stdout),
String::from_utf8_lossy(&result.stderr)
);
}
@@ -0,0 +1,535 @@
//! Own-bit connection marks for exit-client underlay traffic.
//! Reply marks are restored before host reverse-path filtering, never accepted
//! around it. No peer address becomes a host route or firewall argument.
use super::linux::{Ran, failure, locked, split_rule_line};
use std::net::{IpAddr, Ipv4Addr, Ipv6Addr};
/// Outside Tailscale's 0xff0000 mask and ordinary low-bit WireGuard marks.
pub(crate) const UNDERLAY_MARK: u32 = 0x4000_0000;
pub(crate) const UNDERLAY_MASK: u32 = UNDERLAY_MARK;
const MARK: &str = "0x40000000/0x40000000";
const MASK: &str = "0x40000000";
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord)]
pub(super) struct Listener {
address: IpAddr,
port: u16,
tcp: bool,
}
fn parse_listeners(listing: &str, uid: u32, v6: bool, tcp: bool) -> Result<Vec<Listener>, String> {
let mut result = Vec::new();
for line in listing.lines().skip(1) {
let fields: Vec<_> = line.split_whitespace().collect();
if (tcp && fields.get(3) != Some(&"0A"))
|| fields.get(7).and_then(|s| s.parse::<u32>().ok()) != Some(uid)
{
continue;
}
let (address, port) = fields
.get(1)
.and_then(|s| s.split_once(':'))
.ok_or("malformed local socket")?;
let port = u16::from_str_radix(port, 16).map_err(|e| e.to_string())?;
let address = if v6 {
let mut bytes = [0; 16];
for (word, chunk) in bytes.as_chunks_mut::<4>().0.iter_mut().enumerate() {
let hex = address
.get(word * 8..word * 8 + 8)
.ok_or("malformed local IPv6 listener")?;
chunk.copy_from_slice(
&u32::from_str_radix(hex, 16)
.map_err(|e| e.to_string())?
.to_ne_bytes(),
);
}
IpAddr::V6(Ipv6Addr::from(bytes))
} else {
IpAddr::V4(Ipv4Addr::from(
u32::from_str_radix(address, 16)
.map_err(|e| e.to_string())?
.to_ne_bytes(),
))
};
if result.len() == 128 {
return Err("too many local sockets for exit protection".into());
}
if port != 0 {
result.push(Listener { address, port, tcp });
}
}
Ok(result)
}
pub(super) fn local_sockets(uid: u32, v6: bool) -> Result<Vec<Listener>, String> {
let mut listeners = Vec::new();
for (protocol, tcp) in [("tcp", true), ("udp", false)] {
for family in [false, true].into_iter().filter(|f| !f || v6) {
let path = format!("/proc/net/{protocol}{}", if family { "6" } else { "" });
listeners.extend(parse_listeners(
&std::fs::read_to_string(path).map_err(|e| e.to_string())?,
uid,
family,
tcp,
)?);
}
}
listeners.sort();
listeners.dedup();
Ok(listeners)
}
fn listener_rules(interface: &str, listener: &Listener) -> Vec<Vec<String>> {
let mut base = words(&[
"!",
"-i",
interface,
"-p",
if listener.tcp { "tcp" } else { "udp" },
"--dport",
&listener.port.to_string(),
]);
if !listener.address.is_unspecified() {
base.extend(words(&["-d", &listener.address.to_string()]));
}
// Match an actual local socket, not just a destination port. Listening
// sockets need protection on the first packet, before any outbound connmark.
base.extend(words(&[
"-m",
"socket",
"--nowildcard",
"-m",
"comment",
"--comment",
&listener_tag(interface),
]));
let save = base
.iter()
.cloned()
.chain(words(&[
"-m",
"mark",
"--mark",
MARK,
"-j",
"CONNMARK",
"--save-mark",
"--nfmask",
MASK,
"--ctmask",
MASK,
]))
.collect();
let mark = base
.into_iter()
.chain(words(&["-j", "MARK", "--set-xmark", MARK]))
.collect();
vec![save, mark]
}
pub(super) type RunFamily<'a> = &'a dyn Fn(bool, &[String]) -> Ran;
fn tag(interface: &str) -> String {
format!("tsunagi-exit-underlay:{interface}")
}
fn listener_tag(interface: &str) -> String {
format!("tsunagi-exit-sockets:{interface}")
}
fn words(values: &[&str]) -> Vec<String> {
values.iter().map(|s| (*s).into()).collect()
}
fn rules(interface: &str, uid: u32) -> Vec<(&'static str, Vec<String>)> {
let tag = tag(interface);
let uid = uid.to_string();
// Insert at position one in this order: final OUTPUT order is MARK, SAVE.
vec![
(
"PREROUTING",
words(&[
"!",
"-i",
interface,
"-m",
"connmark",
"--mark",
MARK,
"-m",
"comment",
"--comment",
&tag,
"-j",
"CONNMARK",
"--restore-mark",
"--nfmask",
MASK,
"--ctmask",
MASK,
]),
),
(
"OUTPUT",
words(&[
"-m",
"owner",
"--uid-owner",
&uid,
"-m",
"mark",
"--mark",
MARK,
"-m",
"comment",
"--comment",
&tag,
"-j",
"CONNMARK",
"--save-mark",
"--nfmask",
MASK,
"--ctmask",
MASK,
]),
),
(
"OUTPUT",
words(&[
"-m",
"owner",
"--uid-owner",
&uid,
"-m",
"comment",
"--comment",
&tag,
"-j",
"MARK",
"--set-xmark",
MARK,
]),
),
]
}
fn execute(run: RunFamily<'_>, v6: bool, args: Vec<String>) -> Result<String, String> {
match run(v6, &locked(args)) {
Ran::Exited {
success: true,
stdout,
..
} => Ok(stdout),
Ran::Exited { stderr, .. } => Err(failure(&stderr)),
Ran::Missing => Err(format!(
"{} is required for exit-node connection marks",
if v6 { "ip6tables" } else { "iptables" }
)),
}
}
/// Removes only exact interface tags, retaining other agents and host rules.
pub(super) fn clear(run: RunFamily<'_>, interface: &str, v6: bool) -> Result<(), String> {
clear_tags(run, interface, v6, false)
}
fn clear_tags(
run: RunFamily<'_>,
interface: &str,
v6: bool,
sockets_only: bool,
) -> Result<(), String> {
let wanted = tag(interface);
let sockets = listener_tag(interface);
for family in [false, true].into_iter().filter(|family| !family || v6) {
let chains: &[&str] = if sockets_only {
&["PREROUTING"]
} else {
&["OUTPUT", "PREROUTING"]
};
for chain in chains {
let listing = execute(run, family, words(&["-t", "mangle", "-S", chain]))?;
for line in listing.lines() {
let tokens = split_rule_line(line);
let [append, listed_chain, rest @ ..] = tokens.as_slice() else {
continue;
};
if append == "-A"
&& listed_chain == *chain
&& rest.windows(2).any(|pair| {
pair[0] == "--comment"
&& (pair[1] == sockets || (!sockets_only && pair[1] == wanted))
})
{
let args = words(&["-t", "mangle", "-D", chain])
.into_iter()
.chain(rest.iter().cloned())
.collect();
execute(run, family, args)?;
}
}
}
}
Ok(())
}
/// Establishes protection before routing may redirect application traffic.
pub(super) fn install(
run: RunFamily<'_>,
interface: &str,
uid: u32,
v6: bool,
) -> Result<Vec<Listener>, String> {
let listeners = local_sockets(uid, v6)?;
install_with_listeners(run, interface, uid, v6, &listeners)?;
Ok(listeners)
}
fn install_with_listeners(
run: RunFamily<'_>,
interface: &str,
uid: u32,
v6: bool,
listeners: &[Listener],
) -> Result<(), String> {
clear(run, interface, v6)?;
let result = (|| {
for family in [false, true].into_iter().filter(|family| !family || v6) {
for (chain, spec) in rules(interface, uid) {
let args = words(&["-t", "mangle", "-I", chain, "1"])
.into_iter()
.chain(spec)
.collect();
execute(run, family, args)?;
}
for listener in listeners.iter().filter(|s| s.address.is_ipv6() == family) {
for spec in listener_rules(interface, listener) {
execute(
run,
family,
words(&["-t", "mangle", "-I", "PREROUTING", "1"])
.into_iter()
.chain(spec)
.collect(),
)?;
}
}
}
Ok(())
})();
if result.is_err()
&& let Err(cleanup) = clear(run, interface, v6)
{
return Err(format!(
"{}; cannot roll back connection marks: {cleanup}",
result.err().unwrap_or_default()
));
}
result
}
/// Updates passive protection without interrupting established connmark rules.
pub(super) fn refresh(
run: RunFamily<'_>,
interface: &str,
v6: bool,
listeners: &[Listener],
) -> Result<(), String> {
clear_tags(run, interface, v6, true)?;
for family in [false, true].into_iter().filter(|f| !f || v6) {
for listener in listeners.iter().filter(|s| s.address.is_ipv6() == family) {
for spec in listener_rules(interface, listener) {
execute(
run,
family,
words(&["-t", "mangle", "-I", "PREROUTING", "1"])
.into_iter()
.chain(spec)
.collect(),
)?;
}
}
}
Ok(())
}
/// Kernel strict RPF must include the restored mark, independently of iptables.
pub(super) fn check_kernel_rpf(
all_rpf: u32,
all_valid: u32,
interfaces: &[(String, u32, u32)],
) -> Result<(), String> {
for (interface, rpf, valid) in interfaces {
if all_rpf.max(*rpf) == 1 && all_valid.max(*valid) == 0 {
return Err(format!(
"strict rp_filter on `{interface}` ignores connection marks; enable net.ipv4.conf.all.src_valid_mark=1 before using an exit node (rp_filter stays enabled)"
));
}
}
Ok(())
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
use std::cell::RefCell;
#[test]
fn both_families_restore_before_firewall_and_preserve_foreign_mark_bits() {
let calls = RefCell::new(Vec::new());
let run = |v6, args: &[String]| {
calls.borrow_mut().push((v6, args.to_vec()));
Ran::Exited {
success: true,
stdout: String::new(),
stderr: String::new(),
}
};
install_with_listeners(&run, "tsun0", 990, true, &[]).unwrap();
let calls = calls.borrow();
for family in [false, true] {
let inserts: Vec<_> = calls
.iter()
.filter(|(v6, a)| *v6 == family && a.iter().any(|s| s == "-I"))
.map(|(_, a)| a.join(" "))
.collect();
assert_eq!(inserts.len(), 3);
assert!(inserts[0].contains("-I PREROUTING 1 ! -i tsun0 -m connmark"));
assert!(inserts[0].contains("--restore-mark --nfmask 0x40000000 --ctmask 0x40000000"));
assert!(inserts[1].contains("--uid-owner 990"));
assert!(inserts[1].contains("--save-mark"));
assert!(inserts[2].contains("--set-xmark 0x40000000/0x40000000"));
assert!(
!inserts
.iter()
.any(|s| s.contains("-j ACCEPT") || s.contains("-j RETURN"))
);
}
assert_eq!(UNDERLAY_MASK & 0x00ff_0000, 0);
}
#[test]
fn cleanup_retains_other_interfaces_and_rolls_back_partial_ipv6_installation() {
let calls = RefCell::new(Vec::new());
let run = |v6, args: &[String]| {
calls.borrow_mut().push((v6, args.to_vec()));
let listing = "-A OUTPUT -m comment --comment tsunagi-exit-underlay:tsun0 -j MARK --set-xmark 0x40000000/0x40000000\n-A OUTPUT -m comment --comment tsunagi-exit-underlay:tsun1 -j MARK --set-xmark 0x40000000/0x40000000\n-A OUTPUT -m comment --comment unrelated -j RETURN\n";
Ran::Exited {
success: !(v6 && args.iter().any(|s| s == "-I")),
stdout: if args.iter().any(|s| s == "-S") {
listing.into()
} else {
String::new()
},
stderr: "injected IPv6 failure".into(),
}
};
assert!(install_with_listeners(&run, "tsun0", 990, true, &[]).is_err());
let calls = calls.borrow();
let deletions: Vec<_> = calls
.iter()
.filter(|(_, a)| a.iter().any(|s| s == "-D"))
.collect();
assert!(
deletions.len() >= 4,
"both initial and rollback cleanup run"
);
assert!(
deletions
.iter()
.all(|(_, a)| a.iter().any(|s| s == "tsunagi-exit-underlay:tsun0"))
);
}
#[test]
fn kernel_strict_validation_requires_mark_awareness_without_weakening_rpf() {
let interfaces = vec![("eth0".into(), 1, 0)];
assert!(check_kernel_rpf(0, 0, &interfaces).is_err());
assert!(check_kernel_rpf(0, 1, &interfaces).is_ok());
assert!(check_kernel_rpf(2, 0, &interfaces).is_ok());
assert!(check_kernel_rpf(0, 0, &[("eth0".into(), 1, 1)]).is_ok());
}
#[test]
fn passive_tcp_protection_uses_only_locally_owned_listeners() {
let listing = "header\n0: 00000000:01BB 00000000:0000 0A 0:0 0:0 0 990 0 1\n1: 0100007F:1111 00000000:0000 0A 0:0 0:0 0 1000 0 2\n2: 0100007F:2222 00000000:0000 01 0:0 0:0 0 990 0 3\n";
let listeners = parse_listeners(listing, 990, false, true).unwrap();
assert_eq!(
listeners,
vec![Listener {
address: IpAddr::V4(Ipv4Addr::UNSPECIFIED),
port: 443,
tcp: true
}]
);
let rules = listener_rules("tsun0", &listeners[0]);
assert!(
rules[0]
.join(" ")
.contains("-p tcp --dport 443 -m socket --nowildcard")
);
assert!(rules[0].join(" ").contains("--save-mark"));
assert!(rules[1].join(" ").contains("--set-xmark"));
}
#[test]
fn listener_parser_checks_family_encoding_and_bounds() {
let loopback = Ipv6Addr::LOCALHOST.octets();
let hex: String = loopback
.as_chunks::<4>()
.0
.iter()
.map(|bytes| format!("{:08X}", u32::from_ne_bytes(*bytes)))
.collect();
let listing = format!("header\n0: {hex}:01BB 0:0 0A 0:0 0:0 0 990 0 1\n");
assert_eq!(
parse_listeners(&listing, 990, true, true).unwrap(),
vec![Listener {
address: IpAddr::V6(Ipv6Addr::LOCALHOST),
port: 443,
tcp: true
}]
);
assert!(
parse_listeners(
"header\n0: BAD:01BB 0:0 0A 0:0 0:0 0 990 0 1",
990,
true,
true
)
.is_err()
);
let huge = format!(
"header\n{}",
"0: 00000000:01BB 0:0 0A 0:0 0:0 0 990 0 1\n".repeat(129)
);
assert!(parse_listeners(&huge, 990, false, true).is_err());
}
#[test]
fn udp_refresh_preserves_general_connmark_protection_and_other_interfaces() {
let listing = "header\n0: 00000000:ABA0 0:0 07 0:0 0:0 0 990 0 1\n1: 00000000:0000 0:0 07 0:0 0:0 0 990 0 2\n";
let sockets = parse_listeners(listing, 990, false, false).unwrap();
assert_eq!(sockets.len(), 1);
assert!(!sockets[0].tcp);
let calls = RefCell::new(Vec::new());
let run = |v6, args: &[String]| {
calls.borrow_mut().push((v6, args.join(" ")));
Ran::Exited { success: true, stdout: "-A PREROUTING -m comment --comment tsunagi-exit-underlay:tsun0 -j CONNMARK --restore-mark\n-A PREROUTING -m comment --comment tsunagi-exit-sockets:tsun0 -j MARK\n-A PREROUTING -m comment --comment tsunagi-exit-sockets:tsun1 -j MARK\n".into(), stderr: String::new() }
};
refresh(&run, "tsun0", false, &sockets).unwrap();
let calls = calls.borrow();
assert!(
calls
.iter()
.any(|(_, a)| a.contains("-I PREROUTING 1") && a.contains("-p udp"))
);
assert!(
calls
.iter()
.filter(|(_, a)| a.contains("-D"))
.all(|(_, a)| a.contains("tsunagi-exit-sockets:tsun0"))
);
}
}
@@ -55,6 +55,10 @@ pub use exit::{
mod linux;
#[cfg(all(feature = "tun-device", target_os = "linux"))]
mod linux_exit;
#[cfg(all(test, feature = "tun-device", target_os = "linux"))]
mod linux_exit_kernel;
#[cfg(all(feature = "tun-device", target_os = "linux"))]
pub(in crate::overlay) mod linux_exit_mark;
#[cfg(all(feature = "tun-device", target_os = "linux"))]
pub use linux::LinuxHostRules;
+97 -1
View File
@@ -145,18 +145,19 @@ impl Tally {
}
/// The overlay interface.
#[derive(Debug)]
pub struct Interface {
device: Arc<dyn TunDevice>,
factory: Arc<dyn TunFactory>,
name: String,
mtu: u32,
routes: Arc<RoutingTable>,
carrier: Arc<dyn PacketCarrier>,
tally: Arc<Tally>,
/// What was last applied to the host, so an unchanged table is not
/// re-applied on every pass.
applied: std::sync::Mutex<Vec<Cidr>>,
task: std::sync::Mutex<Option<JoinHandle<()>>>,
vip_task: std::sync::Mutex<Option<JoinHandle<()>>>,
sync_lock: Arc<tokio::sync::Mutex<()>>,
/// The host rules for this interface, when the factory manages the host.
host_rules: Option<Arc<dyn BroadcastHostRules>>,
@@ -200,6 +201,14 @@ impl ExitState {
}
}
impl std::fmt::Debug for Interface {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("Interface")
.field("name", &self.name)
.finish_non_exhaustive()
}
}
impl Interface {
/// Creates the interface and starts moving packets.
pub async fn start(
@@ -221,7 +230,29 @@ impl Interface {
let name = device.name().to_string();
let tally = Arc::new(Tally::default());
let vip_task = {
let routes = Arc::clone(&routes);
let carrier = Arc::clone(&carrier);
tokio::spawn(async move {
let mut interval = tokio::time::interval(std::time::Duration::from_millis(250));
interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
let mut checks = tokio::task::JoinSet::new();
loop {
tokio::select! {
_=interval.tick()=>{
let (probes,jobs)=routes.vips.tick(routes.vips.check_capacity().saturating_sub(checks.len()));
for (route,packet) in probes {carrier.carry(route,packet);}
for job in jobs {checks.spawn(job.run());}
},
Some(result)=checks.join_next(),if !checks.is_empty()=>{
if let Ok((job,ok))=result {routes.vips.checked(job,ok);}
}
}
}
})
};
let task = {
let carrier = Arc::clone(&carrier);
let device = Arc::clone(&device);
let routes = Arc::clone(&routes);
let tally = Arc::clone(&tally);
@@ -231,6 +262,26 @@ impl Interface {
tally.malformed.fetch_add(1, Ordering::Relaxed);
continue;
};
match routes.vips.outgoing(packet.clone()) {
crate::vip::router::Decision::Send(route, packet) => {
for part in crate::vip::packet::fragments(packet, mtu) {
if carrier.carry(route, part) {
tally.sent.fetch_add(1, Ordering::Relaxed);
} else {
tally.undeliverable.fetch_add(1, Ordering::Relaxed);
}
}
continue;
}
crate::vip::router::Decision::Local(packet) => {
for part in crate::vip::packet::fragments(packet, mtu) {
let _ = device.send(part).await;
}
continue;
}
crate::vip::router::Decision::Drop => continue,
crate::vip::router::Decision::Pass => {}
}
let destination = header.destination();
if let (IpAddr::V4(source), IpAddr::V4(destination)) =
(header.source(), destination)
@@ -279,9 +330,11 @@ impl Interface {
name,
mtu,
routes,
carrier,
tally,
applied: std::sync::Mutex::new(Vec::new()),
task: std::sync::Mutex::new(Some(task)),
vip_task: std::sync::Mutex::new(Some(vip_task)),
sync_lock: Arc::new(tokio::sync::Mutex::new(())),
host_rules,
rules_report: std::sync::Mutex::new(None),
@@ -397,6 +450,14 @@ impl Interface {
return;
}
if planned.as_ref() == Some(&wanted) && ok {
if let Some(outcome) = exit.refresh(&wanted).await {
if let RuleOutcome::Failed(reason) = &outcome {
tracing::warn!(interface = %self.name, %reason, "cannot refresh exit-client socket protection");
}
if let Some(report) = self.exit_state().report.as_mut() {
report.client = Some(outcome);
}
}
return;
}
let report = exit.apply(&wanted).await;
@@ -708,6 +769,15 @@ impl Interface {
task.abort();
let _ = task.await;
}
let vip_task = self
.vip_task
.lock()
.unwrap_or_else(|e| e.into_inner())
.take();
if let Some(task) = vip_task {
task.abort();
let _ = task.await;
}
self.factory.destroy(self.device.name()).await;
}
@@ -762,6 +832,24 @@ impl Interface {
return Err(Rejected::WrongDestination);
}
};
// Source ownership was checked before any VIP reverse translation.
let packet = match self.routes.vips.incoming(network, peer, packet.clone()) {
crate::vip::router::Decision::Send(route, packet) => {
// Probe replies use the ordinary peer carrier, never the TUN.
self.carrier.carry(route, packet);
return Ok(());
}
crate::vip::router::Decision::Local(packet) => {
for part in crate::vip::packet::fragments(packet, self.mtu) {
if self.device.send(part).await.is_ok() {
self.tally.received.fetch_add(1, Ordering::Relaxed);
}
}
return Ok(());
}
crate::vip::router::Decision::Drop => return Ok(()),
crate::vip::router::Decision::Pass => packet,
};
// Remote broadcasts terminate here. Only local TUN ingress can fan out.
if self.device.send(packet).await.is_ok() {
if broadcast {
@@ -786,6 +874,14 @@ impl Interface {
impl Drop for Interface {
fn drop(&mut self) {
if let Some(task) = self
.vip_task
.lock()
.unwrap_or_else(|e| e.into_inner())
.take()
{
task.abort();
}
if let Ok(mut guard) = self.task.lock()
&& let Some(task) = guard.take()
{
@@ -55,6 +55,7 @@ use crate::BoxFuture;
use crate::overlay::OverlayError;
use super::super::config::Cidr;
use super::super::hostrules::linux_exit_mark::{UNDERLAY_MARK, UNDERLAY_MASK};
use super::super::tun::{TunDevice, TunRequest};
use super::privilege::{NetAdmin, Privilege, probe_net_admin};
use super::{
@@ -641,8 +642,10 @@ async fn del_broadcast_route(name: &str) -> Result<(), OverlayError> {
const EXIT_TABLE: u32 = 0x7473;
/// The main table.
const MAIN_TABLE: u32 = 254;
/// Where the exit rules sit, after Tailscale's 5270.
const EXIT_BYPASS_PRIORITY: u32 = 5280;
/// Protect marked underlay and initial socket routing before Tailscale's rules.
const EXIT_MARK_PRIORITY: u32 = 5205;
const EXIT_BYPASS_PRIORITY: u32 = 5206;
const LEGACY_EXIT_BYPASS_PRIORITY: u32 = 5280;
const EXIT_SPECIFIC_PRIORITY: u32 = 5290;
const EXIT_DEFAULT_PRIORITY: u32 = 5300;
@@ -666,14 +669,23 @@ fn rule_priority(rule: &RuleMessage) -> Option<u32> {
})
}
/// Whether a rule is one of the three this agent adds for an exit node.
/// Whether a rule is one this agent adds for an exit node, including old UID priority.
///
/// Recognised by shape as well as priority: those numbers are not reserved,
/// and a rule somebody else put at one of them must not be removed.
fn is_exit_rule(rule: &RuleMessage) -> bool {
let table = rule_table(rule);
match rule_priority(rule) {
Some(EXIT_BYPASS_PRIORITY) => {
Some(EXIT_MARK_PRIORITY) => {
table == MAIN_TABLE
&& rule
.attributes
.contains(&RuleAttribute::FwMark(UNDERLAY_MARK))
&& rule
.attributes
.contains(&RuleAttribute::FwMask(UNDERLAY_MASK))
}
Some(EXIT_BYPASS_PRIORITY | LEGACY_EXIT_BYPASS_PRIORITY) => {
table == MAIN_TABLE
&& rule
.attributes
@@ -789,7 +801,8 @@ async fn remove_exit_client(handle: &rtnetlink::Handle) -> Result<(), OverlayErr
///
/// ```text
/// ip route replace default dev <if> table 0x7473
/// ip rule add priority 5280 uidrange <uid>-<uid> lookup main
/// ip rule add priority 5205 fwmark 0x40000000/0x40000000 lookup main
/// ip rule add priority 5206 uidrange <uid>-<uid> lookup main
/// ip rule add priority 5290 lookup main suppress_prefixlength 0
/// ip rule add priority 5300 lookup 0x7473
/// ```
@@ -819,6 +832,8 @@ async fn set_exit_client(name: &str, uid: u32) -> Result<(), OverlayError> {
.await
.map_err(|err| fail("default route", err))?;
add_mark_bypass(&handle, false).await?;
let mut bypass = handle
.rule()
.add()
@@ -891,6 +906,7 @@ async fn set_exit_block_v6(handle: &rtnetlink::Handle, uid: u32) -> Result<(), O
};
}
add_mark_bypass(handle, true).await?;
let mut bypass = handle
.rule()
.add()
@@ -938,6 +954,27 @@ async fn set_exit_block_v6(handle: &rtnetlink::Handle, uid: u32) -> Result<(), O
.map_err(|err| fail("rule for the default route", err))
}
async fn add_mark_bypass(handle: &rtnetlink::Handle, v6: bool) -> Result<(), OverlayError> {
let mut rule = handle
.rule()
.add()
.table_id(MAIN_TABLE)
.action(RuleAction::ToTable)
.priority(EXIT_MARK_PRIORITY)
.fw_mark(UNDERLAY_MARK);
rule.message_mut().header.family = if v6 {
rtnetlink::packet_route::AddressFamily::Inet6
} else {
rtnetlink::packet_route::AddressFamily::Inet
};
rule.message_mut()
.attributes
.push(RuleAttribute::FwMask(UNDERLAY_MASK));
rule.execute().await.map_err(|err| {
OverlayError::Unavailable(format!("cannot protect marked underlay traffic: {err}"))
})
}
async fn configure_link(configure: &Configure) -> Result<(), OverlayError> {
let name = configure.name.as_str();
with_netlink(|handle| async move {
@@ -1110,6 +1147,28 @@ mod tests {
assert!(!is_exit_rule(&rule(5280, 100, vec![uid])));
}
#[test]
fn underlay_bypass_recognises_only_our_exact_bit_and_mask() {
let rule = |table, mark, mask| {
let mut message = RuleMessage::default();
message.attributes.extend([
RuleAttribute::Table(table),
RuleAttribute::Priority(EXIT_MARK_PRIORITY),
RuleAttribute::FwMark(mark),
RuleAttribute::FwMask(mask),
]);
message
};
assert!(is_exit_rule(&rule(
MAIN_TABLE,
UNDERLAY_MARK,
UNDERLAY_MASK
)));
assert!(!is_exit_rule(&rule(MAIN_TABLE, 0x80000, 0xff0000)));
assert!(!is_exit_rule(&rule(MAIN_TABLE, UNDERLAY_MARK, u32::MAX)));
assert!(!is_exit_rule(&rule(52, UNDERLAY_MARK, UNDERLAY_MASK)));
}
#[test]
fn the_ipv6_default_of_the_exit_table_refuses_instead_of_routing() {
let route = exit_blocking_route_v6();
@@ -89,6 +89,7 @@ pub enum RouteError {
/// Address ownership across every network this agent is in.
#[derive(Debug, Default)]
pub struct RoutingTable {
pub(crate) vips: Arc<crate::vip::VipRouter>,
broadcast: ArcSwap<BroadcastTable>,
networks: RwLock<HashMap<NetworkId, NetworkRoutes>>,
}
@@ -146,6 +147,7 @@ impl RoutingTable {
/// Forgets a network.
pub fn remove_network(&self, network: NetworkId) {
self.vips.remove_network(network);
let mut networks = self.write();
networks.remove(&network);
self.broadcast
+45 -2
View File
@@ -20,7 +20,7 @@ use crate::error::ProtocolError;
/// The version in the ALPN is the wire-compatibility version of the control
/// protocol. It is independent of the network identity scheme version, so
/// bumping it must not change any existing [`crate::NetworkId`].
pub const ALPN: &[u8] = b"tsunagi/ctrl/6";
pub const ALPN: &[u8] = b"tsunagi/ctrl/8";
/// ALPN of the tsunagi data plane.
///
@@ -43,7 +43,7 @@ pub const MAX_DATA_PROTOCOL_LEN: usize = 32;
pub const MAX_SIGNATURE_LEN: usize = 64;
/// Control protocol version carried inside the handshake.
pub const PROTOCOL_VERSION: u16 = 6;
pub const PROTOCOL_VERSION: u16 = 8;
pub use crate::dataplane::transport::{MAX_TRANSPORT_PAYLOAD, TransportPayload};
@@ -181,6 +181,16 @@ pub enum ControlMessage {
/// Short free-text reason.
reason: String,
},
/// One complete signed VIP field snapshot, merged atomically.
Vips {
/// Signed registers bounded by the per-VIP field limit.
entries: Vec<crate::vip::VipEntry>,
},
/// Signed cleanup markers, each proving the retired claim version.
Retirements {
/// Bounded by the per-message signed-state limit.
markers: Vec<crate::state::SignedRetirement>,
},
}
/// Somewhere a member has been seen.
@@ -273,6 +283,37 @@ pub fn validate_capability(
/// network, the other networks or the agent.
pub fn validate(message: &ControlMessage, limits: &Limits) -> Result<(), ProtocolError> {
match message {
ControlMessage::Vips { entries } => {
check_len("vip.fields", entries.len(), limits.max_vip_fields)?;
for e in entries {
check_len("vip.id", e.vip.len(), 32)?;
check_len("vip.key", e.key.len(), 160)?;
check_len("vip.value", e.value.as_ref().map_or(0, Vec::len), 256)?;
check_len("vip.signature", e.signature.len(), 64)?;
}
}
ControlMessage::Retirements { markers } => {
check_len("retirements", markers.len(), limits.max_state_records)?;
for marker in markers {
check_len(
"retirement.signature",
marker.signature.len(),
MAX_SIGNATURE_LEN,
)?;
check_len(
"retirement.target.signature",
marker.target.signature.len(),
MAX_SIGNATURE_LEN,
)?;
if let Some(name) = marker.target.body.hostname() {
check_len(
"retirement.target.hostname",
name.len(),
limits.max_hostname_len,
)?;
}
}
}
ControlMessage::Transport { protocol, payload } => {
if protocol.is_empty() {
return Err(ProtocolError::Malformed("empty transport protocol"));
@@ -345,6 +386,8 @@ pub fn kind(message: &ControlMessage) -> &'static str {
ControlMessage::Ping { .. } => "ping",
ControlMessage::Pong { .. } => "pong",
ControlMessage::State { .. } => "state",
ControlMessage::Retirements { .. } => "retirements",
ControlMessage::Vips { .. } => "vips",
ControlMessage::Reachable { .. } => "reachable",
ControlMessage::Peers { .. } => "peers",
ControlMessage::Bye { .. } => "bye",
+86 -2
View File
@@ -28,16 +28,25 @@
//! replica that has the record needs nothing else to interpret it, and
//! recovery never depends on replaying a chain from the beginning.
//!
//! Any member may also sign a separate retirement of another author's proved
//! claim version. It suppresses stale reservations without changing that
//! author's record, and a newer claim remains valid.
//!
//! # What this slice does not do yet
//!
//! Compaction, revocation of a whole author, and snapshots covering more than
//! one record type. See `docs/sync-model.md` for the shape those take.
//! Compaction and permanent revocation of a whole author. See
//! `docs/sync-model.md` for the implemented cleanup semantics.
pub mod allocator;
/// Signed shared virtual service state and its merge rules.
pub mod vip;
use std::collections::HashMap;
use std::net::Ipv4Addr;
mod retirement;
pub use retirement::{MAX_RETIREMENT_LEN, SignedRetirement};
use iroh::{EndpointId, SecretKey, Signature};
use serde::{Deserialize, Serialize};
@@ -488,6 +497,7 @@ pub enum Merged {
#[derive(Debug, Clone, Default)]
pub struct StateSet {
records: HashMap<EndpointId, SignedRecord>,
retirements: HashMap<EndpointId, SignedRetirement>,
}
impl StateSet {
@@ -500,6 +510,7 @@ impl StateSet {
pub fn from_verified(records: impl IntoIterator<Item = (EndpointId, SignedRecord)>) -> Self {
Self {
records: records.into_iter().collect(),
retirements: HashMap::new(),
}
}
@@ -528,6 +539,52 @@ impl StateSet {
.collect()
}
/// Visible records, excluding versions retired by authenticated members.
pub fn active_records(&self) -> Vec<SignedRecord> {
self.records()
.into_iter()
.filter(|r| r.author_id().is_ok_and(|id| !self.is_retired(&id)))
.collect()
}
/// Whether the current known version is at or below a proved retirement.
pub fn is_retired(&self, author: &EndpointId) -> bool {
self.retirements
.get(author)
.is_some_and(|m| self.version_of(author) <= m.target.version)
}
/// Strongest retained cleanup proof for an identity.
pub fn retirement_of(&self, target: &EndpointId) -> Option<&SignedRetirement> {
self.retirements.get(target)
}
/// Durable markers in target-id order, including markers surpassed by a fresh claim.
pub fn retirements(&self) -> Vec<SignedRetirement> {
let mut markers: Vec<_> = self.retirements.values().cloned().collect();
markers.sort_by_key(|m| m.target.author);
markers
}
/// Accepts any member's signed cleanup after checking both signatures.
pub fn merge_retirement(
&mut self,
network: NetworkId,
marker: SignedRetirement,
max: usize,
) -> Result<bool, StateError> {
let target = marker.verify(network)?;
if let Some(old) = self.retirements.get(&target) {
if !marker.supersedes(old) {
return Ok(false);
}
} else if self.retirements.len() >= max {
return Err(StateError::Malformed("retirement capacity reached"));
}
self.retirements.insert(target, marker);
Ok(true)
}
/// Merges one record, verifying it first.
///
/// Merging is into the existing set, never a wholesale replacement, and an
@@ -591,6 +648,9 @@ impl StateSet {
pub fn address_holders(&self) -> HashMap<Ipv4Addr, EndpointId> {
let mut holders: HashMap<Ipv4Addr, EndpointId> = HashMap::new();
for (author, record) in &self.records {
if self.is_retired(author) {
continue;
}
let Some(address) = record.body.claimed_address() else {
continue;
};
@@ -618,6 +678,9 @@ impl StateSet {
pub fn hostname_holders(&self) -> HashMap<String, EndpointId> {
let mut holders: HashMap<String, (u64, EndpointId)> = HashMap::new();
for (author, record) in &self.records {
if self.is_retired(author) {
continue;
}
let Some(hostname) = record.body.hostname() else {
continue;
};
@@ -646,6 +709,9 @@ impl StateSet {
/// The name an author claimed and the member that holds it instead, when
/// somebody who claimed it earlier does.
pub fn hostname_taken(&self, author: &EndpointId) -> Option<(&str, EndpointId)> {
if self.is_retired(author) {
return None;
}
let hostname = self.records.get(author)?.body.hostname()?;
let holder = *self.hostname_holders().get(hostname)?;
(holder != *author).then_some((hostname, holder))
@@ -671,6 +737,12 @@ impl StateSet {
pub fn agreed_range(&self) -> Option<Ipv4Range> {
self.records
.iter()
.chain(
self.retirements
.iter()
.filter(|(author, _)| !self.records.contains_key(*author))
.map(|(author, marker)| (author, &marker.target)),
)
.filter_map(|(author, record)| {
let range = record.body.range()?;
let derived = derived_ipv4_range(NetworkId::from_bytes(record.network));
@@ -684,6 +756,18 @@ impl StateSet {
pub fn version_of(&self, author: &EndpointId) -> u64 {
self.records.get(author).map_or(0, |record| record.version)
}
/// Highest witnessed author counter across claims and authored cleanup.
/// This differs from the claim version used to decide retirement.
pub fn authored_version(&self, author: &EndpointId) -> u64 {
self.retirements
.values()
.filter(|m| m.author == *author.as_bytes())
.map(|m| m.version)
.chain(std::iter::once(self.version_of(author)))
.max()
.unwrap_or(0)
}
}
#[cfg(test)]
+219
View File
@@ -0,0 +1,219 @@
//! A member may retire an observed claim, never another identity's future.
use iroh::{EndpointId, SecretKey, Signature};
use serde::{Deserialize, Serialize};
use super::{RecordBody, SignedRecord, StateError, push_lp};
use crate::NetworkId;
/// Maximum encoded retirement size, including its bounded claim proof.
pub const MAX_RETIREMENT_LEN: usize = 1024;
/// A network member's signed retirement of a proved claim version.
///
/// The original author's signed claim proves the cutoff really existed.
/// Newer claims remain valid. This is neither a forged Release nor a ban.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct SignedRetirement {
/// Device identity of the member requesting cleanup.
pub author: [u8; 32],
/// Network in which cleanup applies.
pub network: [u8; 32],
/// Requester's durable author counter, committed with this marker.
pub version: u64,
/// The target's original signed claim; bounds the retired version.
pub target: SignedRecord,
/// Requester's Ed25519 signature over this marker and its proof.
pub signature: Vec<u8>,
}
impl SignedRetirement {
fn canonical_bytes(&self) -> Vec<u8> {
let mut out = Vec::with_capacity(768);
push_lp(&mut out, b"tsunagi-node-retirement/v1");
push_lp(&mut out, &self.network);
push_lp(&mut out, &self.author);
push_lp(&mut out, &self.version.to_be_bytes());
// Bind every field of the original signed claim, including its signature.
push_lp(&mut out, &self.target.network);
push_lp(&mut out, &self.target.author);
push_lp(&mut out, &self.target.version.to_be_bytes());
push_lp(&mut out, &self.target.body.canonical());
push_lp(&mut out, &self.target.signature);
out
}
/// Signs a marker with the requesting member's persistent device key.
pub fn sign(
secret: &SecretKey,
network: NetworkId,
version: u64,
target: SignedRecord,
) -> Self {
let mut marker = Self {
author: *secret.public().as_bytes(),
network: *network.as_bytes(),
version,
target,
signature: Vec::new(),
};
marker.signature = secret.sign(&marker.canonical_bytes()).to_bytes().to_vec();
marker
}
/// Verifies both authors, scope and a finite, genuinely signed cutoff.
pub fn verify(&self, network: NetworkId) -> Result<EndpointId, StateError> {
if self.network != *network.as_bytes() {
return Err(StateError::WrongNetwork);
}
if self.version == 0 || self.version > i64::MAX as u64 {
return Err(StateError::Malformed("retirement version is out of range"));
}
if !matches!(self.target.body, RecordBody::Claim { .. }) {
return Err(StateError::Malformed("only a claim can be retired"));
}
let target = self.target.verify(network)?;
let author = EndpointId::from_bytes(&self.author).map_err(|_| StateError::BadAuthor)?;
if author == target {
return Err(StateError::Malformed(
"use Release to retire your own claim",
));
}
let raw: [u8; Signature::LENGTH] =
self.signature
.as_slice()
.try_into()
.map_err(|_| StateError::BadSignatureLength {
expected: Signature::LENGTH,
})?;
author
.verify(&self.canonical_bytes(), &Signature::from_bytes(&raw))
.map_err(|_| StateError::BadSignature)?;
Ok(target)
}
/// A deterministic strongest marker per target, independent of arrival order.
pub(crate) fn supersedes(&self, other: &Self) -> bool {
self.target.version > other.target.version
|| (self.target.version == other.target.version
&& (self.canonical_bytes(), &self.signature)
< (other.canonical_bytes(), &other.signature))
}
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
use crate::state::{Ipv4Range, StateSet};
fn claim(key: &SecretKey, id: NetworkId, version: u64) -> SignedRecord {
SignedRecord::sign(
key,
id,
version,
RecordBody::Claim {
address: Some("10.23.0.2".parse().unwrap()),
range: Some("10.23.0.0/24".parse::<Ipv4Range>().unwrap()),
hostname: Some("retired".into()),
hostname_since: Some(123),
},
)
}
#[test]
fn markers_release_claims_in_any_merge_order_and_older_snapshots_cannot_restore_them() {
let id = NetworkId::from_bytes([7; 32]);
let target = SecretKey::generate();
let cleaner = SecretKey::generate();
let record = claim(&target, id, 5);
let marker = SignedRetirement::sign(&cleaner, id, 8, record.clone());
for first in [true, false] {
let mut state = StateSet::new();
if first {
state.merge_retirement(id, marker.clone(), 16).unwrap();
}
state.merge(id, record.clone()).unwrap();
if !first {
state.merge_retirement(id, marker.clone(), 16).unwrap();
}
assert!(state.address_holders().is_empty());
assert!(state.hostname_holders().is_empty());
state.merge(id, claim(&target, id, 4)).unwrap();
assert!(state.is_retired(&target.public()));
assert!(
state.agreed_range().is_some(),
"cleanup does not renumber the network"
);
state.merge(id, claim(&target, id, 6)).unwrap();
assert!(!state.is_retired(&target.public()));
assert_eq!(state.hostname_of(&target.public()), Some("retired"));
}
}
#[test]
fn proof_signature_scope_tampering_future_cutoffs_and_capacity_are_checked() {
let id = NetworkId::from_bytes([7; 32]);
let target = SecretKey::generate();
let cleaner = SecretKey::generate();
let marker = SignedRetirement::sign(&cleaner, id, 8, claim(&target, id, 5));
assert!(marker.verify(NetworkId::from_bytes([8; 32])).is_err());
let mut forged = marker.clone();
forged.target.version = 99;
assert!(forged.verify(id).is_err());
let mut forged = marker.clone();
forged.signature[0] ^= 1;
assert!(forged.verify(id).is_err());
let mut proof = marker.target.clone();
proof.version = 99;
assert!(
SignedRetirement::sign(&cleaner, id, 8, proof)
.verify(id)
.is_err()
);
let mut state = StateSet::new();
assert!(state.merge_retirement(id, marker.clone(), 0).is_err());
assert!(state.merge_retirement(id, marker.clone(), 1).unwrap());
assert!(!state.merge_retirement(id, marker.clone(), 1).unwrap());
assert!(postcard::to_stdvec(&marker).unwrap().len() <= MAX_RETIREMENT_LEN);
let another = SecretKey::generate();
assert!(
state
.merge_retirement(
id,
SignedRetirement::sign(&cleaner, id, 9, claim(&another, id, 1)),
1
)
.is_err()
);
}
#[test]
fn competing_cleanup_markers_converge_and_keep_the_highest_observed_cutoff() {
let id = NetworkId::from_bytes([7; 32]);
let target = SecretKey::generate();
let a = SecretKey::generate();
let b = SecretKey::generate();
let markers = [
SignedRetirement::sign(&a, id, 1, claim(&target, id, 4)),
SignedRetirement::sign(&a, id, 2, claim(&target, id, 5)),
SignedRetirement::sign(&b, id, 3, claim(&target, id, 5)),
];
let mut forward = StateSet::new();
let mut reverse = StateSet::new();
for marker in &markers {
forward.merge_retirement(id, marker.clone(), 16).unwrap();
}
for marker in markers.iter().rev() {
reverse.merge_retirement(id, marker.clone(), 16).unwrap();
}
assert_eq!(forward.retirements(), reverse.retirements());
assert_eq!(
forward
.retirement_of(&target.public())
.unwrap()
.target
.version,
5
);
}
}
+439
View File
@@ -0,0 +1,439 @@
//! Signed multi-writer virtual service registers and deterministic merge rules.
use crate::error::Result;
use crate::vip::{Backend, Listener, Protocol, VipConfig, VipStatus, invalid};
use crate::{Limits, NetworkId};
use iroh::{EndpointId, SecretKey, Signature};
use serde::{Deserialize, Serialize};
use sha2::{Digest, Sha256};
use std::collections::{BTreeMap, BTreeSet};
use std::net::Ipv4Addr;
/// A signed, independently mergeable field. A removed field keeps its tombstone.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct VipEntry {
/// Network scope.
pub network: [u8; 32],
/// Stable service id.
pub vip: String,
/// Canonical field key.
pub key: String,
/// Encoded field value; absent means deletion.
pub value: Option<Vec<u8>>,
/// Lamport clock, independent of wall time.
pub clock: u64,
/// Persistent signing identity.
pub author: [u8; 32],
/// Author's durable network counter.
pub version: u64,
/// Ed25519 signature.
pub signature: Vec<u8>,
}
impl VipEntry {
fn bytes(&self) -> Vec<u8> {
let mut e = self.clone();
e.signature.clear();
let mut out = b"tsunagi-vip-field/v1\0".to_vec();
if let Ok(bytes) = postcard::to_stdvec(&e) {
out.extend(bytes);
}
out
}
/// Verifies bounds, network and authorship before the entry is merged.
pub fn verify(&self, network: NetworkId) -> Result<()> {
if self.network != *network.as_bytes()
|| self.vip.len() != 32
|| hex::decode(&self.vip).is_err()
|| self.key.len() > 160
|| self.value.as_ref().is_some_and(|v| v.len() > 256)
|| self.version == 0
|| self.version > i64::MAX as u64
|| self.clock == 0
|| self.clock > i64::MAX as u64
{
return Err(invalid("invalid VIP field bounds or network"));
}
let author =
EndpointId::from_bytes(&self.author).map_err(|_| invalid("invalid VIP author"))?;
let raw: [u8; 64] = self
.signature
.as_slice()
.try_into()
.map_err(|_| invalid("invalid VIP signature length"))?;
author
.verify(&self.bytes(), &Signature::from_bytes(&raw))
.map_err(|_| invalid("invalid VIP signature"))?;
if self.key != "deleted"
&& self.key != "name"
&& !self.key.starts_with("l/")
&& !self.key.starts_with("b/")
{
return Err(invalid("unknown VIP field"));
}
match self.key.as_str() {
"deleted" => {
if self.value.as_deref() != Some(&[1][..]) {
return Err(invalid("invalid VIP deletion tombstone"));
}
}
"name" => {
let value = self
.value
.as_ref()
.ok_or_else(|| invalid("VIP name cannot be absent"))?;
let name: String =
postcard::from_bytes(value).map_err(|_| invalid("invalid VIP name field"))?;
VipConfig {
id: self.vip.clone(),
name,
listeners: vec![],
}
.validate(&Limits::default())?;
}
key if key.starts_with("l/") => {
let parts: Vec<_> = key.split('/').collect();
if parts.len() != 3
|| parts[1].parse::<Protocol>().is_err()
|| parts[2].parse::<u16>().ok().is_none_or(|p| p == 0)
{
return Err(invalid("invalid listener field key"));
}
if let Some(value) = &self.value {
let l: Listener = postcard::from_bytes(value)
.map_err(|_| invalid("invalid listener field"))?;
if !l.backends.is_empty() || key != format!("l/{}/{}", l.protocol, l.port) {
return Err(invalid("listener key does not match its value"));
}
VipConfig {
id: self.vip.clone(),
name: "validation".into(),
listeners: vec![l],
}
.validate(&Limits::default())?;
}
}
key if key.starts_with("b/") => {
let parts: Vec<_> = key.split('/').collect();
if parts.len() != 6
|| parts[1].parse::<Protocol>().is_err()
|| parts[2].parse::<u16>().ok().is_none_or(|p| p == 0)
|| parts[3].len() != 32
|| hex::decode(parts[3]).is_err()
|| parts[4].parse::<EndpointId>().is_err()
|| parts[5].parse::<u16>().ok().is_none_or(|p| p == 0)
{
return Err(invalid("invalid backend field key"));
}
if let Some(value) = &self.value {
let b: Backend = postcard::from_bytes(value)
.map_err(|_| invalid("invalid backend field"))?;
if b.peer != parts[4] || b.port.to_string() != parts[5] {
return Err(invalid("backend key does not match its value"));
}
}
}
_ => return Err(invalid("unknown VIP field")),
}
Ok(())
}
fn rank(&self) -> (u64, [u8; 32], Vec<u8>) {
(self.clock, self.author, self.bytes())
}
}
/// Signed registers retained across absences, including deletion tombstones.
#[derive(Debug, Clone, Default)]
pub struct VipSet {
entries: BTreeMap<(String, String), VipEntry>,
}
impl VipSet {
/// Highest retained version genuinely signed by this author.
pub fn authored_version(&self, author: EndpointId) -> u64 {
self.entries
.values()
.filter(|e| e.author == *author.as_bytes())
.map(|e| e.version)
.max()
.unwrap_or(0)
}
/// Complete independently verifiable state of one service, including tombstones.
pub fn entries(&self, id: &str) -> Vec<VipEntry> {
self.entries
.values()
.filter(|e| e.vip == id)
.cloned()
.collect()
}
/// All retained service ids, including deleted services.
pub fn ids(&self) -> Vec<String> {
self.entries
.keys()
.map(|(id, _)| id.clone())
.collect::<BTreeSet<_>>()
.into_iter()
.collect()
}
/// Opaque optimistic concurrency token for a whole service.
pub fn revision(&self, id: &str) -> String {
let mut hash = Sha256::new();
for e in self.entries(id) {
hash.update(e.bytes());
hash.update(&e.signature);
}
hex::encode(hash.finalize())
}
/// Atomically verifies and merges one complete service snapshot.
pub fn merge(
&mut self,
network: NetworkId,
entries: Vec<VipEntry>,
limits: &Limits,
) -> Result<bool> {
if entries.len() > limits.max_vip_fields {
return Err(invalid("too many VIP fields"));
}
let mut next = self.clone();
for e in entries {
e.verify(network)?;
let key = (e.vip.clone(), e.key.clone());
if next
.entries
.get(&key)
.is_none_or(|old| e.rank() > old.rank())
{
next.entries.insert(key, e);
}
}
let deleted: BTreeSet<_> = next
.entries
.values()
.filter(|e| e.key == "deleted" && e.value.is_some())
.map(|e| e.vip.clone())
.collect();
// A permanent service tombstone alone prevents every former field's replay.
next.entries
.retain(|(id, key), _| !deleted.contains(id) || key == "deleted");
let ids = next.ids();
if ids.len().saturating_sub(deleted.len()) > limits.max_vips
|| deleted.len() > limits.max_vip_tombstones
|| ids
.iter()
.any(|id| next.entries(id).len() > limits.max_vip_fields)
{
return Err(invalid(
"VIP state capacity exceeded; deletion protection is retained",
));
}
let changed = self.entries != next.entries;
*self = next;
Ok(changed)
}
/// Materialized configuration. A service tombstone permanently retires its id.
pub fn configs(&self) -> Vec<VipConfig> {
self.ids()
.into_iter()
.filter_map(|id| self.config(&id))
.collect()
}
fn config(&self, id: &str) -> Option<VipConfig> {
let fields = self.entries(id);
if fields
.iter()
.any(|e| e.key == "deleted" && e.value.is_some())
{
return None;
}
let name = fields
.iter()
.find(|e| e.key == "name")?
.value
.as_ref()
.and_then(|v| postcard::from_bytes::<String>(v).ok())?;
let mut listeners: Vec<Listener> = fields
.iter()
.filter(|e| e.key.starts_with("l/"))
.filter_map(|e| e.value.as_ref().and_then(|v| postcard::from_bytes(v).ok()))
.collect();
for l in &mut listeners {
l.backends = fields
.iter()
.filter(|e| {
e.key
.starts_with(&format!("b/{}/{}/{}/", l.protocol, l.port, l.id))
})
.filter_map(|e| e.value.as_ref().and_then(|v| postcard::from_bytes(v).ok()))
.collect();
l.backends.sort();
}
listeners.sort_by_key(|l| (l.protocol, l.port));
Some(VipConfig {
id: id.into(),
name,
listeners,
})
}
/// Builds a signed patch from the user's complete, version-checked draft.
pub fn edit(
&self,
network: NetworkId,
secret: &SecretKey,
version: u64,
config: &VipConfig,
expected: Option<&str>,
limits: &Limits,
) -> Result<Vec<VipEntry>> {
config.validate(limits)?;
if let Some(revision) = expected {
if revision != self.revision(&config.id) {
return Err(invalid("VIP changed while editing; reload before saving"));
}
} else if !self.entries(&config.id).is_empty() {
return Err(invalid("VIP id already exists"));
}
if self
.entries
.get(&(config.id.clone(), "deleted".into()))
.is_some_and(|e| e.value.is_some())
{
return Err(invalid("VIP was deleted; create a new VIP"));
}
let before = self
.config(&config.id)
.map(|c| flatten(&c))
.transpose()?
.unwrap_or_default();
let after = flatten(config)?;
let keys: BTreeSet<_> = before.keys().chain(after.keys()).cloned().collect();
let changes = keys
.into_iter()
.filter_map(|key| {
let value = after.get(&key).cloned();
(before.get(&key) != value.as_ref()).then_some((key, value))
})
.collect();
self.sign(network, secret, version, &config.id, changes)
}
/// Builds a permanent service tombstone after checking the selected revision.
pub fn delete(
&self,
network: NetworkId,
secret: &SecretKey,
version: u64,
id: &str,
expected: &str,
) -> Result<Vec<VipEntry>> {
if self.config(id).is_none() || self.revision(id) != expected {
return Err(invalid("VIP missing or changed while editing"));
}
self.sign(
network,
secret,
version,
id,
vec![("deleted".into(), Some(vec![1]))],
)
}
fn sign(
&self,
network: NetworkId,
secret: &SecretKey,
version: u64,
id: &str,
changes: Vec<(String, Option<Vec<u8>>)>,
) -> Result<Vec<VipEntry>> {
let clock = self
.entries
.values()
.map(|e| e.clock)
.max()
.unwrap_or(0)
.checked_add(1)
.filter(|n| *n <= i64::MAX as u64)
.ok_or_else(|| invalid("VIP clock exhausted"))?;
Ok(changes
.into_iter()
.map(|(key, value)| {
let mut e = VipEntry {
network: *network.as_bytes(),
vip: id.into(),
key,
value,
clock,
author: *secret.public().as_bytes(),
version,
signature: vec![],
};
e.signature = secret.sign(&e.bytes()).to_bytes().to_vec();
e
})
.collect())
}
}
pub(crate) fn flatten(cfg: &VipConfig) -> Result<BTreeMap<String, Vec<u8>>> {
let mut fields = BTreeMap::new();
fields.insert(
"name".into(),
postcard::to_stdvec(&cfg.name).map_err(|e| invalid(e.to_string()))?,
);
for l in &cfg.listeners {
let mut settings = l.clone();
settings.backends.clear();
fields.insert(
format!("l/{}/{}", l.protocol, l.port),
postcard::to_stdvec(&settings).map_err(|e| invalid(e.to_string()))?,
);
for b in &l.backends {
fields.insert(
format!("b/{}/{}/{}/{}/{}", l.protocol, l.port, l.id, b.peer, b.port),
postcard::to_stdvec(b).map_err(|e| invalid(e.to_string()))?,
);
}
}
Ok(fields)
}
impl VipSet {
/// Allocates internal addresses deterministically against durable node claims.
pub(crate) fn views(
&self,
range: Option<crate::state::Ipv4Range>,
mut taken: std::collections::HashSet<Ipv4Addr>,
hostnames: &BTreeSet<String>,
limits: &Limits,
) -> Vec<VipStatus> {
let mut names = hostnames.clone();
let mut result = Vec::new();
for config in self.configs() {
let mut error = config.validate(limits).err().map(|e| e.to_string());
if !names.insert(config.name.clone()) {
error = Some("name conflicts with another VIP or node".into());
}
let address = range.and_then(|r| {
let usable = crate::state::allocator::usable_addresses(r);
if usable == 0 {
return None;
}
let digest = Sha256::digest(config.id.as_bytes());
let seed = u64::from_be_bytes([
digest[0], digest[1], digest[2], digest[3], digest[4], digest[5], digest[6],
digest[7],
]);
let bits = 32 - u32::from(r.prefix_len);
let mask = if bits == 32 { 0 } else { u32::MAX << bits };
(0..usable).find_map(|step| {
let offset = (seed.wrapping_add(step) % usable + 1) as u32;
let ip = Ipv4Addr::from((u32::from(r.base) & mask) | offset);
taken.insert(ip).then_some(ip)
})
});
if address.is_none() {
error = Some("no free overlay address".into());
}
result.push(VipStatus {
revision: self.revision(&config.id),
config,
address,
error,
listeners: vec![],
});
}
result
}
}
+51
View File
@@ -254,6 +254,12 @@ impl Storage {
.await
}
/// Persists the ordered protocol selection off the async executor.
pub async fn set_protocols(&self, network_id: NetworkId, protocols: Vec<String>) -> Result<()> {
self.with_state(move |state| state.set_protocols(network_id, &protocols))
.await
}
/// Persists whether this agent offers itself as an exit node in a network.
pub async fn set_exit_node(&self, network_id: NetworkId, enabled: bool) -> Result<()> {
self.with_state(move |state| state.set_exit_node(network_id, enabled))
@@ -304,6 +310,7 @@ impl Storage {
self.with_state(move |state| {
state.remove_network(network_id)?;
state.forget_signed_records(network_id)?;
state.forget_retirements(network_id)?;
state.forget_peer_hostnames(network_id)
})
.await?;
@@ -363,6 +370,50 @@ impl Storage {
.await
}
/// Loads mandatory signed VIP fields with a strict capacity bound.
pub async fn vip_fields(
&self,
network: NetworkId,
max: usize,
) -> Result<Vec<crate::vip::VipEntry>> {
self.with_state(move |state| state.vip_fields(network, max))
.await
}
/// Stores a complete merged service snapshot and optional own counter.
pub async fn put_vip_fields(
&self,
network: NetworkId,
entries: Vec<crate::vip::VipEntry>,
own: Option<([u8; 32], u64)>,
) -> Result<()> {
self.with_state(move |state| state.put_vip_fields(network, &entries, own))
.await
}
/// Loads verified durable cleanup proofs off the executor.
pub async fn retirements(
&self,
network: NetworkId,
max: usize,
) -> Result<Vec<crate::state::SignedRetirement>> {
self.with_state(move |state| state.retirements(network, max))
.await
}
/// Persists a received cleanup proof before applying it.
pub async fn put_retirement(&self, marker: crate::state::SignedRetirement) -> Result<()> {
self.with_state(move |state| state.put_retirement(&marker))
.await
}
/// Commits our cleanup and author counter together before announcement.
pub async fn publish_own_retirement(
&self,
marker: crate::state::SignedRetirement,
) -> Result<()> {
self.with_state(move |state| state.publish_own_retirement(&marker))
.await
}
/// Stores a record received from another replica.
pub async fn put_signed_record(&self, record: SignedRecord) -> Result<()> {
self.with_state(move |state| state.put_signed_record(&record))
+458 -4
View File
@@ -18,10 +18,10 @@ use rusqlite::{Connection, OptionalExtension, params};
use crate::error::{Error, Result};
use crate::identity::{DeviceIdentity, NetworkId, NetworkName, NetworkSecret};
use crate::state::{RecordBody, SignedRecord};
use crate::state::{MAX_RETIREMENT_LEN, RecordBody, SignedRecord, SignedRetirement};
/// Schema version written by this build.
pub const SCHEMA_VERSION: i64 = 7;
pub const SCHEMA_VERSION: i64 = 10;
/// Key of the stored hostname setting.
const SETTING_HOSTNAME: &str = "hostname";
@@ -32,6 +32,8 @@ const SETTING_HOSTNAME: &str = "hostname";
/// and zeroizes on drop.
#[derive(Debug, Clone)]
pub struct StoredNetwork {
/// Ordered local protocol override; absent inherits the agent's defaults.
pub protocols: Option<Vec<String>>,
/// Derived public network identifier.
pub network_id: NetworkId,
/// Network name.
@@ -238,6 +240,35 @@ impl StateStore {
)
.map_err(|err| self.corrupt(format!("cannot migrate schema to 7: {err}")))?;
}
if found < 8 {
self.conn
.execute_batch(
"BEGIN;
ALTER TABLE networks ADD COLUMN protocols TEXT;
PRAGMA user_version = 8;
COMMIT;",
)
.map_err(|err| self.corrupt(format!("cannot migrate schema to 8: {err}")))?;
}
if found < 9 {
self.conn
.execute_batch(
"BEGIN;
CREATE TABLE IF NOT EXISTS retirements (
network_id BLOB NOT NULL, target BLOB NOT NULL, data BLOB NOT NULL,
PRIMARY KEY (network_id, target)
);
PRAGMA user_version = 9;
COMMIT;",
)
.map_err(|err| self.corrupt(format!("cannot migrate schema to 9: {err}")))?;
}
if found < 10 {
self.conn.execute_batch("BEGIN; CREATE TABLE IF NOT EXISTS vip_fields (
network_id BLOB NOT NULL, vip TEXT NOT NULL, field TEXT NOT NULL, data BLOB NOT NULL,
PRIMARY KEY(network_id, vip, field)); PRAGMA user_version = 10; COMMIT;")
.map_err(|err| self.corrupt(format!("cannot migrate VIP state: {err}")))?;
}
Ok(())
}
@@ -250,6 +281,8 @@ impl StateStore {
"settings",
"signed_records",
"peer_hostnames",
"retirements",
"vip_fields",
] {
let present: Option<String> = self
.conn
@@ -476,6 +509,25 @@ impl StateStore {
Ok(())
}
/// Saves a local ordered protocol selection without changing membership.
pub fn set_protocols(&self, network_id: NetworkId, protocols: &[String]) -> Result<()> {
crate::dataplane::validate_protocols(
protocols,
crate::config::Limits::default().max_capabilities,
)?;
let changed = self
.conn
.execute(
"UPDATE networks SET protocols = ?2 WHERE network_id = ?1",
params![network_id.as_bytes().as_slice(), protocols.join(",")],
)
.map_err(|err| Error::Storage(format!("cannot update protocols: {err}")))?;
if changed == 0 {
return Err(Error::NetworkUnknown(network_id));
}
Ok(())
}
/// Removes a network configuration entirely.
pub fn remove_network(&self, network_id: NetworkId) -> Result<()> {
self.conn
@@ -536,7 +588,7 @@ impl StateStore {
pub fn list_networks(&self) -> Result<Vec<StoredNetwork>> {
let mut stmt = self
.conn
.prepare("SELECT network_id, name, secret, auto_start, broadcast, exit_node, exit_via FROM networks ORDER BY name")
.prepare("SELECT network_id, name, secret, auto_start, broadcast, exit_node, exit_via, protocols FROM networks ORDER BY name")
.map_err(|err| Error::Storage(format!("cannot list networks: {err}")))?;
let rows = stmt
.query_map([], |row| {
@@ -552,19 +604,37 @@ impl StateStore {
row.get::<_, bool>(4)?,
row.get::<_, bool>(5)?,
row.get::<_, Option<Vec<u8>>>(6)?,
row.get::<_, Option<String>>(7)?,
))
})
.map_err(|err| Error::Storage(format!("cannot list networks: {err}")))?;
let mut out = Vec::new();
for row in rows {
let (id, name, secret, auto_start, broadcast, exit_node, exit_via) =
let (id, name, secret, auto_start, broadcast, exit_node, exit_via, protocols) =
row.map_err(|err| Error::Storage(format!("cannot read network row: {err}")))?;
let protocols = protocols
.map(|text| {
let max = crate::config::Limits::default().max_capabilities;
if text.len() > max * (crate::dataplane::MAX_PROTOCOL_ID_LEN + 1) {
return Err(self.corrupt("stored protocol selection is too long"));
}
let names = if text.is_empty() {
Vec::new()
} else {
text.split(',').map(str::to_owned).collect()
};
crate::dataplane::validate_protocols(&names, max)
.map_err(|err| self.corrupt(err))?;
Ok(names)
})
.transpose()?;
let id: [u8; 32] = id
.as_slice()
.try_into()
.map_err(|_| self.corrupt("stored network id is not 32 bytes"))?;
out.push(StoredNetwork {
protocols,
network_id: NetworkId::from_bytes(id),
name: NetworkName::new(name)?,
secret: NetworkSecret::from_bytes(secret)?,
@@ -736,6 +806,181 @@ impl StateStore {
Ok(version.unwrap_or(0).max(0) as u64)
}
/// Loads durable cleanup markers without allocating oversized stored blobs.
pub fn retirements(&self, network: NetworkId, max: usize) -> Result<Vec<SignedRetirement>> {
let mut stmt = self
.conn
.prepare(
"SELECT CASE WHEN length(data) <= ?2 THEN data ELSE NULL END
FROM retirements WHERE network_id = ?1 ORDER BY target",
)
.map_err(|err| self.corrupt(err))?;
let rows = stmt
.query_map(
params![network.as_bytes().as_slice(), MAX_RETIREMENT_LEN as i64],
|row| row.get::<_, Option<Vec<u8>>>(0),
)
.map_err(|err| self.corrupt(err))?;
let mut out = Vec::new();
for row in rows {
if out.len() >= max {
return Err(Error::Storage(
"stored retirements exceed the configured limit".into(),
));
}
let bytes = row
.map_err(|err| self.corrupt(err))?
.ok_or_else(|| self.corrupt("oversized retirement"))?;
let marker: SignedRetirement =
postcard::from_bytes(&bytes).map_err(|err| self.corrupt(err))?;
marker.verify(network).map_err(|err| self.corrupt(err))?;
out.push(marker);
}
Ok(out)
}
/// Persists a strongest verified marker received from an authenticated session.
pub fn put_retirement(&self, marker: &SignedRetirement) -> Result<()> {
let network = NetworkId::from_bytes(marker.network);
marker
.verify(network)
.map_err(|err| Error::Storage(err.to_string()))?;
let old: Option<Option<Vec<u8>>> = self.conn.query_row(
"SELECT CASE WHEN length(data) <= ?3 THEN data ELSE NULL END FROM retirements WHERE network_id = ?1 AND target = ?2",
params![marker.network.as_slice(), marker.target.author.as_slice(), MAX_RETIREMENT_LEN as i64], |row| row.get(0))
.optional().map_err(|err| self.corrupt(err))?;
if let Some(bytes) = old {
let bytes = bytes.ok_or_else(|| self.corrupt("oversized retirement"))?;
let old: SignedRetirement =
postcard::from_bytes(&bytes).map_err(|err| self.corrupt(err))?;
old.verify(network).map_err(|err| self.corrupt(err))?;
if !marker.supersedes(&old) {
return Ok(());
}
}
let bytes = postcard::to_stdvec(marker).map_err(|err| Error::Storage(err.to_string()))?;
if bytes.len() > MAX_RETIREMENT_LEN {
return Err(Error::Storage(
"retirement exceeds its encoded bound".into(),
));
}
self.conn
.execute(
"INSERT INTO retirements (network_id, target, data) VALUES (?1, ?2, ?3)
ON CONFLICT(network_id, target) DO UPDATE SET data = excluded.data",
params![
marker.network.as_slice(),
marker.target.author.as_slice(),
bytes
],
)
.map_err(|err| Error::Storage(format!("cannot store retirement: {err}")))?;
Ok(())
}
/// Commits our signed cleanup marker and author counter in one transaction.
pub fn publish_own_retirement(&self, marker: &SignedRetirement) -> Result<()> {
let transaction = self
.conn
.unchecked_transaction()
.map_err(|err| Error::Storage(err.to_string()))?;
self.put_retirement(marker)?;
transaction
.execute(
"INSERT INTO own_record_version (network_id, author, version) VALUES (?1, ?2, ?3)
ON CONFLICT(network_id, author) DO UPDATE SET version = max(version, excluded.version)",
params![
marker.network.as_slice(),
marker.author.as_slice(),
marker.version as i64
],
)
.map_err(|err| Error::Storage(format!("cannot store retirement counter: {err}")))?;
transaction
.commit()
.map_err(|err| Error::Storage(format!("cannot commit retirement: {err}")))
}
/// Reads bounded mandatory VIP state; corrupt records fail activation.
pub fn vip_fields(&self, network: NetworkId, max: usize) -> Result<Vec<crate::vip::VipEntry>> {
let mut stmt = self.conn.prepare("SELECT CASE WHEN length(data) <= 1024 THEN data ELSE NULL END FROM vip_fields WHERE network_id=?1 ORDER BY vip, field")
.map_err(|e| self.corrupt(e.to_string()))?;
let mut rows = stmt
.query(params![network.as_bytes().as_slice()])
.map_err(|e| self.corrupt(e.to_string()))?;
let mut entries = Vec::new();
while let Some(row) = rows.next().map_err(|e| self.corrupt(e.to_string()))? {
if entries.len() >= max {
return Err(self.corrupt("stored VIP state exceeds configured limits"));
}
let bytes: Option<Vec<u8>> = row.get(0).map_err(|e| self.corrupt(e.to_string()))?;
let bytes = bytes.ok_or_else(|| self.corrupt("oversized VIP field"))?;
let entry: crate::vip::VipEntry =
postcard::from_bytes(&bytes).map_err(|e| self.corrupt(e.to_string()))?;
entry
.verify(network)
.map_err(|e| self.corrupt(e.to_string()))?;
entries.push(entry);
}
Ok(entries)
}
/// Commits a validated VIP snapshot and optional author counter atomically.
pub fn put_vip_fields(
&self,
network: NetworkId,
entries: &[crate::vip::VipEntry],
own: Option<([u8; 32], u64)>,
) -> Result<()> {
let tx = self
.conn
.unchecked_transaction()
.map_err(|e| Error::Storage(e.to_string()))?;
if let Some(first) = entries.first() {
if entries.iter().any(|e| e.vip != first.vip) {
return Err(Error::Storage(
"VIP snapshot mixes service identities".into(),
));
}
tx.execute(
"DELETE FROM vip_fields WHERE network_id=?1 AND vip=?2",
params![network.as_bytes().as_slice(), first.vip],
)
.map_err(|e| Error::Storage(e.to_string()))?;
}
for e in entries {
e.verify(network)?;
let bytes = postcard::to_stdvec(e).map_err(|e| Error::Storage(e.to_string()))?;
if bytes.len() > 1024 {
return Err(Error::Storage("oversized VIP field".into()));
}
tx.execute("INSERT INTO vip_fields(network_id,vip,field,data) VALUES(?1,?2,?3,?4) ON CONFLICT(network_id,vip,field) DO UPDATE SET data=excluded.data",
params![network.as_bytes().as_slice(),e.vip,e.key,bytes]).map_err(|e|Error::Storage(e.to_string()))?;
}
if let Some((author, version)) = own {
tx.execute("INSERT INTO own_record_version(network_id,author,version) VALUES(?1,?2,?3) ON CONFLICT(network_id,author) DO UPDATE SET version=max(version,excluded.version)",
params![network.as_bytes().as_slice(),author.as_slice(),version as i64]).map_err(|e|Error::Storage(e.to_string()))?;
}
tx.commit().map_err(|e| Error::Storage(e.to_string()))
}
/// Drops a forgotten network's local cleanup proofs.
pub fn forget_retirements(&self, network: NetworkId) -> Result<()> {
self.conn
.execute(
"DELETE FROM vip_fields WHERE network_id = ?1",
params![network.as_bytes().as_slice()],
)
.map_err(|e| Error::Storage(e.to_string()))?;
self.conn
.execute(
"DELETE FROM retirements WHERE network_id = ?1",
params![network.as_bytes().as_slice()],
)
.map_err(|err| Error::Storage(err.to_string()))?;
Ok(())
}
/// Forgets every record of a network.
pub fn forget_signed_records(&self, network_id: NetworkId) -> Result<()> {
self.conn
@@ -957,6 +1202,158 @@ mod broadcast_storage_tests {
assert!(store.peer_hostnames(id).unwrap().is_empty());
}
#[test]
fn retirement_migration_and_atomic_counter_commit_preserve_existing_state() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("state.sqlite");
let store = StateStore::open(&path).unwrap();
let identity = store.load_or_create_device_identity().unwrap();
let name = NetworkName::new("retirement-migration").unwrap();
let secret = NetworkSecret::from_bytes([6; 32]).unwrap();
let id = NetworkKeys::derive(&name, &secret).network_id();
store.upsert_network(id, &name, &secret, true).unwrap();
store.set_protocols(id, &["awg".into()]).unwrap();
let own = SignedRecord::sign(&identity.signing_key(), id, 8, RecordBody::Release);
store.publish_own_record(&own).unwrap();
store
.conn
.execute_batch("DROP TABLE retirements; PRAGMA user_version = 8;")
.unwrap();
drop(store);
let store = StateStore::open(&path).unwrap();
assert_eq!(
store.device_identity().unwrap().unwrap().endpoint_id(),
identity.endpoint_id()
);
assert_eq!(
store.list_networks().unwrap()[0].protocols,
Some(vec!["awg".into()])
);
assert_eq!(store.signed_records(id).unwrap(), vec![own]);
let target_key = iroh::SecretKey::generate();
let target = SignedRecord::sign(
&target_key,
id,
3,
RecordBody::Claim {
address: Some("10.13.37.2".parse().unwrap()),
range: Some(crate::state::DEFAULT_IPV4_RANGE),
hostname: Some("old-device".into()),
hostname_since: Some(123),
},
);
let marker = SignedRetirement::sign(&identity.signing_key(), id, 9, target);
store
.conn
.execute_batch(
"CREATE TRIGGER fail_retirement_counter BEFORE INSERT ON own_record_version
WHEN NEW.version = 9 BEGIN SELECT RAISE(ABORT, 'counter write refused'); END;",
)
.unwrap();
assert!(store.publish_own_retirement(&marker).is_err());
assert!(
store.retirements(id, 16).unwrap().is_empty(),
"marker must roll back with counter failure"
);
assert_eq!(
store
.own_record_version(id, identity.endpoint_id())
.unwrap(),
8
);
store
.conn
.execute_batch("DROP TRIGGER fail_retirement_counter;")
.unwrap();
store.publish_own_retirement(&marker).unwrap();
drop(store);
let store = StateStore::open(&path).unwrap();
assert_eq!(store.retirements(id, 16).unwrap(), vec![marker.clone()]);
assert_eq!(
store
.own_record_version(id, identity.endpoint_id())
.unwrap(),
9
);
assert!(
store.retirements(id, 0).is_err(),
"capacity changes must not silently drop markers"
);
store
.conn
.execute(
"UPDATE retirements SET data = ?1",
params![vec![0u8; MAX_RETIREMENT_LEN + 1]],
)
.unwrap();
assert!(matches!(
store.retirements(id, 16),
Err(Error::StateCorrupted { .. })
));
}
#[test]
fn v7_migration_keeps_signed_state_and_protocol_overrides_are_local_and_durable() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("state.sqlite");
let store = StateStore::open(&path).unwrap();
let identity = store.load_or_create_device_identity().unwrap();
let name = NetworkName::new("protocol-migration").unwrap();
let secret = NetworkSecret::from_bytes([4; 32]).unwrap();
let id = NetworkKeys::derive(&name, &secret).network_id();
store.upsert_network(id, &name, &secret, true).unwrap();
store.set_broadcast(id, false).unwrap();
let record = SignedRecord::sign(&identity.signing_key(), id, 17, RecordBody::Release);
store.publish_own_record(&record).unwrap();
store
.conn
.execute_batch("ALTER TABLE networks DROP COLUMN protocols; PRAGMA user_version=7;")
.unwrap();
drop(store);
let store = StateStore::open(&path).unwrap();
assert_eq!(
store
.load_or_create_device_identity()
.unwrap()
.endpoint_id(),
identity.endpoint_id()
);
assert_eq!(store.signed_records(id).unwrap(), vec![record]);
let saved = store.list_networks().unwrap();
assert!(saved[0].protocols.is_none());
assert!(!saved[0].broadcast);
assert!(saved[0].auto_start);
assert_eq!(saved[0].secret, secret);
let chosen = vec!["awg".into(), "wg-quic".into()];
store.set_protocols(id, &chosen).unwrap();
assert!(
store
.set_protocols(id, &["awg".into(), "awg".into()])
.is_err()
);
assert!(
store
.set_protocols(NetworkId::from_bytes([1; 32]), &[])
.is_err()
);
drop(store);
let store = StateStore::open(&path).unwrap();
assert_eq!(store.list_networks().unwrap()[0].protocols, Some(chosen));
store.upsert_network(id, &name, &secret, true).unwrap();
store.set_protocols(id, &[]).unwrap();
drop(store);
let store = StateStore::open(&path).unwrap();
assert_eq!(store.list_networks().unwrap()[0].protocols, Some(vec![]));
store
.conn
.execute("UPDATE networks SET protocols = 'awg,awg'", [])
.unwrap();
assert!(matches!(
store.list_networks(),
Err(Error::StateCorrupted { .. })
));
}
#[test]
fn v3_migration_preserves_identity_and_networks_and_defaults_broadcast_on() {
let dir = tempfile::tempdir().unwrap();
@@ -977,6 +1374,7 @@ mod broadcast_storage_tests {
"ALTER TABLE networks DROP COLUMN broadcast; \
ALTER TABLE networks DROP COLUMN exit_node; \
ALTER TABLE networks DROP COLUMN exit_via; \
ALTER TABLE networks DROP COLUMN protocols; \
PRAGMA user_version=3;",
)
.unwrap();
@@ -1011,3 +1409,59 @@ mod broadcast_storage_tests {
);
}
}
#[cfg(test)]
mod vip_tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
#[test]
fn vip_migration_and_counter_failure_preserve_identity_and_rollback_fields() {
let dir = tempfile::TempDir::new().unwrap();
let path = dir.path().join("state.sqlite");
let store = StateStore::open(&path).unwrap();
let identity = store.load_or_create_device_identity().unwrap();
store
.conn
.execute_batch("DROP TABLE vip_fields; PRAGMA user_version=9;")
.unwrap();
drop(store);
let store = StateStore::open(&path).unwrap();
assert_eq!(
store.device_identity().unwrap().unwrap().endpoint_id(),
identity.endpoint_id()
);
let net = NetworkId::from_bytes([7; 32]);
let key = identity.signing_key();
let config = crate::vip::VipConfig::new("api");
let fields = crate::vip::VipSet::default()
.edit(net, &key, 7, &config, None, &crate::Limits::default())
.unwrap();
store.conn.execute_batch("CREATE TRIGGER fail_vip_counter BEFORE INSERT ON own_record_version BEGIN SELECT RAISE(ABORT,'counter unavailable'); END;").unwrap();
assert!(
store
.put_vip_fields(net, &fields, Some((*key.public().as_bytes(), 7)))
.is_err()
);
assert!(store.vip_fields(net, 192).unwrap().is_empty());
store
.conn
.execute_batch("DROP TRIGGER fail_vip_counter;")
.unwrap();
store
.put_vip_fields(net, &fields, Some((*key.public().as_bytes(), 7)))
.unwrap();
assert_eq!(store.vip_fields(net, 192).unwrap(), fields);
assert_eq!(store.own_record_version(net, key.public()).unwrap(), 7);
drop(store);
let reopened = StateStore::open(path).unwrap();
assert_eq!(reopened.vip_fields(net, 192).unwrap(), fields);
reopened
.conn
.execute("UPDATE vip_fields SET data=?1", params![vec![0xffu8; 1025]])
.unwrap();
assert!(matches!(
reopened.vip_fields(net, 192),
Err(Error::StateCorrupted { .. })
));
}
}
+239
View File
@@ -0,0 +1,239 @@
//! Named, shared virtual services. Signed field registers converge without an
//! owner; routing, health and connection state remain local to each client.
use crate::Limits;
use crate::error::{Error, Result};
use iroh::EndpointId;
use serde::{Deserialize, Serialize};
use std::collections::BTreeSet;
use std::net::Ipv4Addr;
pub(crate) mod packet;
pub(crate) mod router;
pub use crate::state::vip::{VipEntry, VipSet};
pub(crate) use router::VipRouter;
#[cfg(test)]
mod tests;
/// IP transport carried by a listener.
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum Protocol {
/// TCP.
Tcp,
/// UDP.
Udp,
}
impl std::fmt::Display for Protocol {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str(match self {
Self::Tcp => "tcp",
Self::Udp => "udp",
})
}
}
impl std::str::FromStr for Protocol {
type Err = String;
fn from_str(s: &str) -> std::result::Result<Self, String> {
match s {
"tcp" => Ok(Self::Tcp),
"udp" => Ok(Self::Udp),
_ => Err("protocol must be tcp or udp".into()),
}
}
}
/// A node identity and its application port.
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct Backend {
/// Full device identity, resolved when editing.
pub peer: String,
/// Destination port.
pub port: u16,
}
/// One independently routed port of a VIP.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct Listener {
/// Stable listener generation; prevents removed backend fields resurfacing on re-creation.
pub id: String,
/// TCP or UDP.
pub protocol: Protocol,
/// Client-facing port.
pub port: u16,
/// Enable active TCP connect checks; UDP uses path checks and ICMP errors.
pub port_check: bool,
/// Period between checks in milliseconds.
pub check_interval_ms: u32,
/// Deadline of a port check in milliseconds.
pub check_timeout_ms: u32,
/// Consecutive failed checks before exclusion.
pub failure_threshold: u8,
/// Consecutive successes before recovery.
pub recovery_threshold: u8,
/// Idle expiry for this listener's flows, in seconds.
pub idle_timeout_secs: u32,
/// Allowed destination nodes and ports.
pub backends: Vec<Backend>,
}
impl Listener {
/// Creates a listener with automatic recovery and bounded checks.
pub fn new(protocol: Protocol, port: u16) -> Self {
Self {
id: hex::encode(rand::random::<[u8; 16]>()),
protocol,
port,
port_check: true,
check_interval_ms: 3000,
check_timeout_ms: 2000,
failure_threshold: 2,
recovery_threshold: 1,
idle_timeout_secs: if protocol == Protocol::Tcp { 7200 } else { 120 },
backends: vec![],
}
}
}
/// Entire editable service; its address is an internal derived reservation.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct VipConfig {
/// Stable random 128-bit id in hexadecimal.
pub id: String,
/// One DNS label, unique within the network.
pub name: String,
/// Independent listeners.
pub listeners: Vec<Listener>,
}
impl VipConfig {
/// Creates an empty named service with a fresh stable identity.
pub fn new(name: impl Into<String>) -> Self {
Self {
id: hex::encode(rand::random::<[u8; 16]>()),
name: name.into(),
listeners: vec![],
}
}
/// Validates a complete draft before any state is changed.
pub fn validate(&self, limits: &Limits) -> Result<()> {
if self.id.len() != 32 || hex::decode(&self.id).is_err() {
return Err(invalid("invalid VIP id"));
}
if self.name.is_empty()
|| self.name.len() > 63
|| self.name.starts_with('-')
|| self.name.ends_with('-')
|| !self
.name
.bytes()
.all(|b| b.is_ascii_lowercase() || b.is_ascii_digit() || b == b'-')
{
return Err(invalid(
"VIP name must be a lowercase DNS label (letters, digits and hyphens)",
));
}
if self.listeners.len() > limits.max_vip_listeners {
return Err(invalid("too many VIP listeners"));
}
let mut seen = BTreeSet::new();
for l in &self.listeners {
if l.id.len() != 32 || hex::decode(&l.id).is_err() {
return Err(invalid("invalid listener identity"));
}
if l.port == 0 || !seen.insert((l.protocol, l.port)) {
return Err(invalid(
"listener ports must be nonzero and unique per protocol",
));
}
if !(250..=3_600_000).contains(&l.check_interval_ms)
|| !(100..=60_000).contains(&l.check_timeout_ms)
|| l.failure_threshold == 0
|| l.recovery_threshold == 0
|| !(1..=604800).contains(&l.idle_timeout_secs)
{
return Err(invalid(
"invalid check interval, timeout, thresholds or flow idle timeout",
));
}
if l.backends.len() > limits.max_vip_backends {
return Err(invalid("too many listener backends"));
}
let mut backends = BTreeSet::new();
for b in &l.backends {
if b.port == 0 || b.peer.parse::<EndpointId>().is_err() || !backends.insert(b) {
return Err(invalid(
"backend must have a unique node identity and a nonzero port",
));
}
}
}
if crate::state::vip::flatten(self)?.len() > limits.max_vip_fields {
return Err(invalid("VIP configuration exceeds signed field capacity"));
}
Ok(())
}
}
pub(crate) fn invalid(reason: impl Into<String>) -> Error {
Error::Rejected {
reason: reason.into(),
}
}
/// Cumulative IP traffic, excluding probes. Reset on agent restart.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct Counters {
/// Outbound packets.
pub tx_packets: u64,
/// Outbound bytes.
pub tx_bytes: u64,
/// Inbound packets.
pub rx_packets: u64,
/// Inbound bytes.
pub rx_bytes: u64,
/// Created flow bindings.
pub flows: u64,
/// UDP failovers.
pub failovers: u64,
/// Packets rejected or lacking an eligible backend.
pub dropped: u64,
}
/// One backend, as seen by this client.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct BackendStatus {
/// Configured endpoint.
pub backend: Backend,
/// Eligibility and reason.
pub state: String,
/// Smoothed full data-plane RTT, microseconds; unknown stays absent.
pub rtt_us: Option<u64>,
/// Last successful path measurement age.
pub measurement_age_ms: Option<u64>,
/// Current bindings.
pub active_flows: u32,
/// Local traffic counters.
pub counters: Counters,
}
/// Runtime view of one listener.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct ListenerStatus {
/// Protocol.
pub protocol: Protocol,
/// Frontend port.
pub port: u16,
/// Best backend for a new flow, if measured and eligible.
pub selected: Option<Backend>,
/// Configured and draining endpoints.
pub backends: Vec<BackendStatus>,
/// Local counters.
pub counters: Counters,
}
/// Configuration and local diagnostics returned through the same API to CLI and GUI.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct VipStatus {
/// Persisted configuration.
pub config: VipConfig,
/// Opaque revision used when saving a draft.
pub revision: String,
/// Internal service address.
pub address: Option<Ipv4Addr>,
/// Conflict or allocation issue, if any.
pub error: Option<String>,
/// Local listener status.
pub listeners: Vec<ListenerStatus>,
}
+566
View File
@@ -0,0 +1,566 @@
//! Bounded IPv4 parsing, NAT checksums and fragment reassembly at local ingress.
use super::Protocol;
use bytes::Bytes;
use std::collections::HashMap;
use std::net::Ipv4Addr;
use std::time::{Duration, Instant};
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub(crate) struct Tuple {
pub src: Ipv4Addr,
pub dst: Ipv4Addr,
pub sport: u16,
pub dport: u16,
pub protocol: Protocol,
}
impl Tuple {
pub fn reverse(self) -> Self {
Self {
src: self.dst,
dst: self.src,
sport: self.dport,
dport: self.sport,
protocol: self.protocol,
}
}
}
pub(crate) fn header(p: &[u8]) -> Option<usize> {
if p.len() < 20 || p[0] >> 4 != 4 {
return None;
}
let n = usize::from(p[0] & 15) * 4;
let total = usize::from(u16::from_be_bytes([p[2], p[3]]));
(n >= 20 && n <= p.len() && total == p.len() && checksum(&p[..n]) == 0).then_some(n)
}
pub(crate) fn addresses(p: &[u8]) -> Option<(Ipv4Addr, Ipv4Addr)> {
header(p)?;
Some((
Ipv4Addr::new(p[12], p[13], p[14], p[15]),
Ipv4Addr::new(p[16], p[17], p[18], p[19]),
))
}
pub(crate) fn tuple(p: &[u8]) -> Option<Tuple> {
let n = header(p)?;
if u16::from_be_bytes([p[6], p[7]]) & 0x3fff != 0 {
return None;
}
let protocol = match p[9] {
6 => Protocol::Tcp,
17 => Protocol::Udp,
_ => return None,
};
let min = if protocol == Protocol::Tcp { 20 } else { 8 };
if p.len() < n + min {
return None;
}
if protocol == Protocol::Tcp {
let h = usize::from(p[n + 12] >> 4) * 4;
if h < 20 || n + h > p.len() {
return None;
}
} else if usize::from(u16::from_be_bytes([p[n + 4], p[n + 5]])) != p.len() - n {
return None;
}
let (src, dst) = addresses(p)?;
Some(Tuple {
src,
dst,
sport: u16::from_be_bytes([p[n], p[n + 1]]),
dport: u16::from_be_bytes([p[n + 2], p[n + 3]]),
protocol,
})
}
pub(crate) fn rewrite(p: &[u8], t: Tuple) -> Option<Bytes> {
let n = header(p)?;
if p.len() < n + if t.protocol == Protocol::Tcp { 20 } else { 8 } {
return None;
}
let mut out = p.to_vec();
out[12..16].copy_from_slice(&t.src.octets());
out[16..20].copy_from_slice(&t.dst.octets());
out[n..n + 2].copy_from_slice(&t.sport.to_be_bytes());
out[n + 2..n + 4].copy_from_slice(&t.dport.to_be_bytes());
finish(&mut out, n, t.protocol);
Some(Bytes::from(out))
}
fn finish(p: &mut [u8], n: usize, protocol: Protocol) {
p[10..12].fill(0);
let c = checksum(&p[..n]);
p[10..12].copy_from_slice(&c.to_be_bytes());
let offset = n + if protocol == Protocol::Tcp { 16 } else { 6 };
p[offset..offset + 2].fill(0);
let mut sum = 0u32;
for pair in p[12..20].as_chunks::<2>().0 {
sum += u32::from(u16::from_be_bytes([pair[0], pair[1]]));
}
sum += u32::from(p[9]) + (p.len() - n) as u32;
let c = checksum_seed(&p[n..], sum);
let c = if c == 0 && protocol == Protocol::Udp {
0xffff
} else {
c
};
p[offset..offset + 2].copy_from_slice(&c.to_be_bytes());
}
pub(crate) fn checksum(p: &[u8]) -> u16 {
checksum_seed(p, 0)
}
fn checksum_seed(p: &[u8], mut sum: u32) -> u16 {
for pair in p.chunks(2) {
sum += u32::from(u16::from_be_bytes([pair[0], *pair.get(1).unwrap_or(&0)]));
}
while sum >> 16 != 0 {
sum = (sum & 0xffff) + (sum >> 16);
}
!(sum as u16)
}
pub(crate) fn tcp_syn(p: &[u8]) -> bool {
header(p).is_some_and(|n| p[9] == 6 && p.get(n + 13).is_some_and(|f| f & 0x12 == 2))
}
pub(crate) fn syn_sequence(p: &[u8]) -> Option<u32> {
if !tcp_syn(p) {
return None;
}
let n = header(p)?;
let bytes = p.get(n + 4..n + 8)?;
Some(u32::from_be_bytes(bytes.try_into().ok()?))
}
pub(crate) fn empty_syn(p: &[u8]) -> bool {
header(p).is_some_and(|n| {
tcp_syn(p)
&& p.get(n + 12)
.is_some_and(|h| n + usize::from(h >> 4) * 4 == p.len())
})
}
pub(crate) fn syn_ack(p: &[u8]) -> bool {
header(p).is_some_and(|n| p[9] == 6 && p.get(n + 13).is_some_and(|f| f & 0x12 == 0x12))
}
pub(crate) fn tcp_reset(p: &[u8]) -> bool {
header(p).is_some_and(|n| p[9] == 6 && p.get(n + 13).is_some_and(|f| f & 4 != 0))
}
/// Rewrites an ICMP error and its quoted original header after NAT lookup.
pub(crate) fn icmp_quote(p: &[u8]) -> Option<Tuple> {
let n = header(p)?;
if p[9] != 1 || p.len() < n + 8 + 28 || ![3, 11, 12].contains(&p[n]) {
return None;
}
let q = &p[n + 8..];
let h = usize::from(q[0] & 15) * 4;
if q[0] >> 4 != 4 || h < 20 || q.len() < h + 8 {
return None;
}
let protocol = match q[9] {
6 => Protocol::Tcp,
17 => Protocol::Udp,
_ => return None,
};
Some(Tuple {
src: Ipv4Addr::new(q[12], q[13], q[14], q[15]),
dst: Ipv4Addr::new(q[16], q[17], q[18], q[19]),
sport: u16::from_be_bytes([q[h], q[h + 1]]),
dport: u16::from_be_bytes([q[h + 2], q[h + 3]]),
protocol,
})
}
pub(crate) fn rewrite_icmp(p: &[u8], original: Tuple) -> Option<Bytes> {
let n = header(p)?;
icmp_quote(p)?;
let mut out = p.to_vec();
out[12..16].copy_from_slice(&original.dst.octets());
out[16..20].copy_from_slice(&original.src.octets());
let q = &mut out[n + 8..];
let h = usize::from(q[0] & 15) * 4;
// Incrementally update the quoted transport checksum when it is present.
let off = h + if original.protocol == Protocol::Tcp {
16
} else {
6
};
if q.len() >= off + 2 {
let old = u16::from_be_bytes([q[off], q[off + 1]]);
if old != 0 || original.protocol == Protocol::Tcp {
let mut sum = u32::from(!old);
let mut replacement = Vec::from(original.src.octets());
replacement.extend(original.dst.octets());
replacement.extend(original.sport.to_be_bytes());
replacement.extend(original.dport.to_be_bytes());
let previous = [&q[12..20], &q[h..h + 4]].concat();
for (a, b) in previous
.as_chunks::<2>()
.0
.iter()
.zip(replacement.as_chunks::<2>().0)
{
sum += u32::from(!u16::from_be_bytes([a[0], a[1]]))
+ u32::from(u16::from_be_bytes([b[0], b[1]]));
}
while sum >> 16 != 0 {
sum = (sum & 0xffff) + (sum >> 16);
}
let c = !(sum as u16);
q[off..off + 2].copy_from_slice(
&(if c == 0 && original.protocol == Protocol::Udp {
0xffff
} else {
c
})
.to_be_bytes(),
);
}
}
q[12..16].copy_from_slice(&original.src.octets());
q[16..20].copy_from_slice(&original.dst.octets());
q[h..h + 2].copy_from_slice(&original.sport.to_be_bytes());
q[h + 2..h + 4].copy_from_slice(&original.dport.to_be_bytes());
q[10..12].fill(0);
let c = checksum(&q[..h]);
q[10..12].copy_from_slice(&c.to_be_bytes());
out[n + 2..n + 4].fill(0);
let c = checksum(&out[n..]);
out[n + 2..n + 4].copy_from_slice(&c.to_be_bytes());
out[10..12].fill(0);
let c = checksum(&out[..n]);
out[10..12].copy_from_slice(&c.to_be_bytes());
Some(Bytes::from(out))
}
pub(crate) fn reject(p: &[u8]) -> Option<Bytes> {
let t = tuple(p)?;
let n = header(p)?;
if t.protocol == Protocol::Tcp {
if !tcp_syn(p) {
return None;
}
let mut out = vec![0; 40];
out[0] = 0x45;
out[2..4].copy_from_slice(&40u16.to_be_bytes());
out[8] = 64;
out[9] = 6;
out[12..16].copy_from_slice(&t.dst.octets());
out[16..20].copy_from_slice(&t.src.octets());
out[20..22].copy_from_slice(&t.dport.to_be_bytes());
out[22..24].copy_from_slice(&t.sport.to_be_bytes());
let seq = u32::from_be_bytes(p[n + 4..n + 8].try_into().ok()?);
out[28..32].copy_from_slice(&seq.wrapping_add(1).to_be_bytes());
out[32] = 0x50;
out[33] = 0x14;
finish(&mut out, 20, Protocol::Tcp);
Some(Bytes::from(out))
} else {
let q = p.len().min(n + 8);
let mut out = vec![0; 28 + q];
out[0] = 0x45;
let len = out.len() as u16;
out[2..4].copy_from_slice(&len.to_be_bytes());
out[8] = 64;
out[9] = 1;
out[12..16].copy_from_slice(&t.dst.octets());
out[16..20].copy_from_slice(&t.src.octets());
out[20] = 3;
out[21] = 3;
out[28..].copy_from_slice(&p[..q]);
let c = checksum(&out[20..]);
out[22..24].copy_from_slice(&c.to_be_bytes());
let c = checksum(&out[..20]);
out[10..12].copy_from_slice(&c.to_be_bytes());
Some(Bytes::from(out))
}
}
#[derive(Debug)]
struct Datagram {
at: Instant,
header: Option<Vec<u8>>,
parts: std::collections::BTreeMap<usize, Vec<u8>>,
end: Option<usize>,
bytes: usize,
}
type FragmentKey = (Option<[u8; 32]>, Ipv4Addr, Ipv4Addr, u16, u8);
#[derive(Debug, Default)]
pub(crate) struct Reassembly {
pending: HashMap<FragmentKey, Datagram>,
}
impl Reassembly {
pub fn expire(&mut self, now: Instant) {
self.pending
.retain(|_, d| now.duration_since(d.at) < Duration::from_secs(5));
}
pub fn accept(&mut self, p: Bytes, max: usize, scope: Option<[u8; 32]>) -> Option<Bytes> {
let n = header(&p)?;
let flags = u16::from_be_bytes([p[6], p[7]]);
if flags & 0x3fff == 0 {
return Some(p);
}
let now = Instant::now();
self.pending
.retain(|_, d| now.duration_since(d.at) < Duration::from_secs(5));
let (src, dst) = addresses(&p)?;
let key = (scope, src, dst, u16::from_be_bytes([p[4], p[5]]), p[9]);
let start = usize::from(flags & 0x1fff) * 8;
let payload = &p[n..];
let end = start.checked_add(payload.len())?;
if payload.is_empty()
|| end > 65515
|| flags & 0x2000 != 0 && !payload.len().is_multiple_of(8)
{
self.pending.remove(&key);
return None;
}
if !self.pending.contains_key(&key) && self.pending.len() >= max {
return None;
}
let d = self.pending.entry(key).or_insert_with(|| Datagram {
at: now,
header: None,
parts: Default::default(),
end: None,
bytes: 0,
});
if d.parts
.get(&start)
.is_some_and(|bytes| bytes.as_slice() == payload)
{
return None;
}
if d.parts.iter().any(|(s, b)| *s < end && start < s + b.len()) {
self.pending.remove(&key);
return None;
}
if flags & 0x2000 == 0 {
if d.end.is_some_and(|e| e != end) {
self.pending.remove(&key);
return None;
}
d.end = Some(end);
}
if d.end.is_some_and(|e| end > e) {
self.pending.remove(&key);
return None;
}
if start == 0 {
d.header = Some(p[..n].to_vec());
}
d.bytes += payload.len();
d.parts.insert(start, payload.to_vec());
let complete = d.end.is_some_and(|e| e == d.bytes) && d.header.is_some();
if !complete {
return None;
}
let d = self.pending.remove(&key)?;
let mut out = d.header?;
for part in d.parts.values() {
out.extend(part);
}
if out.len() > 65535 {
return None;
}
let len = out.len() as u16;
out[2..4].copy_from_slice(&len.to_be_bytes());
out[6..8].fill(0);
out[10..12].fill(0);
let h = usize::from(out[0] & 15) * 4;
let c = checksum(&out[..h]);
out[10..12].copy_from_slice(&c.to_be_bytes());
Some(Bytes::from(out))
}
}
/// Re-fragments an assembled NAT datagram for the unchanged overlay MTU.
pub(crate) fn fragments(packet: Bytes, mtu: u32) -> Vec<Bytes> {
if packet.len() <= mtu as usize {
return vec![packet];
}
let Some(n) = header(&packet) else {
return vec![];
};
if u16::from_be_bytes([packet[6], packet[7]]) & 0x4000 != 0 {
return vec![];
}
let size = (mtu as usize).saturating_sub(n) / 8 * 8;
if size == 0 {
return vec![];
}
let mut copied = Vec::new();
let mut offset = 20;
while offset < n {
let kind = packet[offset];
if kind == 0 {
break;
}
if kind == 1 {
offset += 1;
continue;
}
let Some(&len) = packet.get(offset + 1) else {
return vec![];
};
let len = usize::from(len);
if len < 2 || offset + len > n {
return vec![];
}
if kind & 0x80 != 0 {
copied.extend_from_slice(&packet[offset..offset + len]);
}
offset += len;
}
while !copied.len().is_multiple_of(4) {
copied.push(0);
}
let mut result = Vec::new();
let payload = &packet[n..];
for (start, part) in payload.chunks(size).enumerate() {
let offset = start * size;
let mut out = if offset == 0 {
packet[..n].to_vec()
} else {
let mut h = packet[..20].to_vec();
h.extend_from_slice(&copied);
h[0] = 0x40 | (h.len() / 4) as u8;
h
};
let header_len = out.len();
out.extend(part);
let len = out.len() as u16;
out[2..4].copy_from_slice(&len.to_be_bytes());
let flags = (offset / 8) as u16
| if offset + part.len() < payload.len() {
0x2000
} else {
0
};
out[6..8].copy_from_slice(&flags.to_be_bytes());
out[10..12].fill(0);
let c = checksum(&out[..header_len]);
out[10..12].copy_from_slice(&c.to_be_bytes());
result.push(Bytes::from(out));
}
result
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
fn datagram() -> Bytes {
let mut p = vec![0; 3028];
p[0] = 0x45;
p[2..4].copy_from_slice(&3028u16.to_be_bytes());
p[8] = 64;
p[9] = 17;
p[12..16].copy_from_slice(&[10, 0, 0, 1]);
p[16..20].copy_from_slice(&[10, 0, 0, 2]);
p[20..22].copy_from_slice(&1000u16.to_be_bytes());
p[22..24].copy_from_slice(&2000u16.to_be_bytes());
p[24..26].copy_from_slice(&3008u16.to_be_bytes());
finish(&mut p, 20, Protocol::Udp);
Bytes::from(p)
}
#[test]
fn fragmented_udp_reassembles_out_of_order_and_nat_checksums_are_valid() {
let p = datagram();
let parts = fragments(p.clone(), 1280);
assert_eq!(parts.len(), 3);
let mut reassembly = Reassembly::default();
let mut result = None;
for part in parts.into_iter().rev() {
if let Some(p) = reassembly.accept(part, 8, None) {
result = Some(p);
}
}
assert_eq!(result, Some(p.clone()));
let mut t = tuple(&p).unwrap();
t.dst = Ipv4Addr::new(10, 0, 0, 240);
t.dport = 53;
let out = rewrite(&p, t).unwrap();
assert_eq!(header(&out), Some(20));
let mut sum = 0u32;
for pair in out[12..20].as_chunks::<2>().0 {
sum += u32::from(u16::from_be_bytes([pair[0], pair[1]]));
}
sum += 17 + (out.len() - 20) as u32;
assert_eq!(checksum_seed(&out[20..], sum), 0);
}
#[test]
fn overlaps_and_capacity_are_rejected_without_allocating_unbounded_state() {
let parts = fragments(datagram(), 1280);
let mut r = Reassembly::default();
assert!(r.accept(parts[0].clone(), 0, None).is_none());
assert!(r.pending.is_empty());
assert!(r.accept(parts[0].clone(), 1, None).is_none());
assert!(r.accept(parts[0].clone(), 1, None).is_none());
assert_eq!(r.pending.len(), 1);
let mut conflict = parts[0].to_vec();
conflict[20] ^= 1;
assert!(r.accept(Bytes::from(conflict), 1, None).is_none());
assert!(r.pending.is_empty());
for size in 0..80 {
let p = vec![0xff; size];
assert!(tuple(&p).is_none());
assert!(icmp_quote(&p).is_none());
}
}
#[test]
fn fragment_assembly_is_scoped_to_authenticated_sender_and_expires_when_idle() {
let packet = datagram();
let parts = fragments(packet.clone(), 1280);
let mut r = Reassembly::default();
assert!(r.accept(parts[0].clone(), 8, Some([1; 32])).is_none());
assert!(r.accept(parts[1].clone(), 8, Some([2; 32])).is_none());
assert!(r.accept(parts[2].clone(), 8, Some([1; 32])).is_none());
assert_eq!(r.accept(parts[1].clone(), 8, Some([1; 32])), Some(packet));
assert_eq!(r.pending.len(), 1);
r.expire(Instant::now() + Duration::from_secs(6));
assert!(r.pending.is_empty());
}
}
#[cfg(test)]
mod icmp_tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
#[test]
fn pmtu_errors_restore_quoted_addresses_ports_and_checksums() {
let original = Tuple {
src: Ipv4Addr::new(10, 0, 0, 1),
dst: Ipv4Addr::new(10, 0, 0, 240),
sport: 40000,
dport: 53,
protocol: Protocol::Udp,
};
let mut p = vec![0; 36];
p[0] = 0x45;
p[2..4].copy_from_slice(&36u16.to_be_bytes());
p[8] = 64;
p[9] = 17;
p[24..26].copy_from_slice(&16u16.to_be_bytes());
let c = checksum(&p[..20]);
p[10..12].copy_from_slice(&c.to_be_bytes());
let p = rewrite(&p, original).unwrap();
let wire = Tuple {
dst: Ipv4Addr::new(10, 0, 0, 2),
sport: 50000,
dport: 5353,
..original
};
let translated = rewrite(&p, wire).unwrap();
let mut error = vec![0; 56];
error[0] = 0x45;
error[2..4].copy_from_slice(&56u16.to_be_bytes());
error[8] = 64;
error[9] = 1;
error[12..16].copy_from_slice(&wire.dst.octets());
error[16..20].copy_from_slice(&wire.src.octets());
error[20] = 3;
error[21] = 4;
error[26..28].copy_from_slice(&1200u16.to_be_bytes());
error[28..].copy_from_slice(&translated[..28]);
let c = checksum(&error[20..]);
error[22..24].copy_from_slice(&c.to_be_bytes());
let c = checksum(&error[..20]);
error[10..12].copy_from_slice(&c.to_be_bytes());
let restored = rewrite_icmp(&error, original).unwrap();
assert_eq!(icmp_quote(&restored), Some(original));
assert_eq!(checksum(&restored[20..]), 0);
assert_eq!(checksum(&restored[28..48]), 0);
assert_eq!(&restored[54..56], &p[26..28]);
assert_eq!(&restored[26..28], &1200u16.to_be_bytes());
assert_eq!(&restored[12..16], &original.dst.octets());
}
}
File diff suppressed because it is too large. Load diff
+144
View File
@@ -0,0 +1,144 @@
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
use crate::NetworkId;
#[test]
fn concurrent_backend_edits_merge_and_deletion_cannot_resurrect() {
let net = NetworkId::from_bytes([7; 32]);
let a = iroh::SecretKey::generate();
let b = iroh::SecretKey::generate();
let limits = Limits::default();
let mut state = VipSet::default();
let id = hex::encode([9; 16]);
let initial = VipConfig {
id: id.clone(),
name: "api".into(),
listeners: vec![Listener::new(Protocol::Tcp, 443)],
};
let changes = state.edit(net, &a, 1, &initial, None, &limits).unwrap();
state.merge(net, changes, &limits).unwrap();
let revision = state.revision(&id);
let mut first = initial.clone();
first.listeners[0].backends.push(Backend {
peer: a.public().to_string(),
port: 6443,
});
let mut second = initial.clone();
second.listeners[0].backends.push(Backend {
peer: b.public().to_string(),
port: 7443,
});
let x = state
.edit(net, &a, 2, &first, Some(&revision), &limits)
.unwrap();
let y = state
.edit(net, &b, 1, &second, Some(&revision), &limits)
.unwrap();
let mut reverse = state.clone();
state.merge(net, x.clone(), &limits).unwrap();
state.merge(net, y.clone(), &limits).unwrap();
reverse.merge(net, y, &limits).unwrap();
reverse.merge(net, x, &limits).unwrap();
assert_eq!(state.configs(), reverse.configs());
assert_eq!(state.configs()[0].listeners[0].backends.len(), 2);
assert!(
state
.edit(net, &a, 3, &initial, Some(&revision), &limits)
.is_err()
);
let removed = state.delete(net, &a, 3, &id, &state.revision(&id)).unwrap();
let old = state.entries(&id);
state.merge(net, removed, &limits).unwrap();
state.merge(net, old, &limits).unwrap();
assert!(state.configs().is_empty());
}
#[test]
fn signed_fields_reject_tampering_wrong_network_and_limits() {
let net = NetworkId::from_bytes([2; 32]);
let key = iroh::SecretKey::generate();
let limits = Limits::default();
let state = VipSet::default();
let cfg = VipConfig {
id: hex::encode([3; 16]),
name: "dns".into(),
..Default::default()
};
let entries = state.edit(net, &key, 1, &cfg, None, &limits).unwrap();
let mut bad = entries.clone();
bad[0].clock += 1;
assert!(VipSet::default().merge(net, bad, &limits).is_err());
assert!(
VipSet::default()
.merge(NetworkId::from_bytes([4; 32]), entries.clone(), &limits)
.is_err()
);
let mut small = limits.clone();
small.max_vips = 0;
assert!(VipSet::default().merge(net, entries, &small).is_err());
}
#[test]
fn deletion_history_does_not_exhaust_active_capacity_or_accept_stale_fields() {
let network = NetworkId::from_bytes([9; 32]);
let key = iroh::SecretKey::generate();
let limits = Limits {
max_vips: 1,
max_vip_tombstones: 4,
..Default::default()
};
let mut state = VipSet::default();
for index in 0..4 {
let config = VipConfig::new("reused-name");
let fields = state
.edit(network, &key, index * 2 + 1, &config, None, &limits)
.unwrap();
state.merge(network, fields.clone(), &limits).unwrap();
let deletion = state
.delete(
network,
&key,
index * 2 + 2,
&config.id,
&state.revision(&config.id),
)
.unwrap();
state.merge(network, deletion, &limits).unwrap();
assert_eq!(state.entries(&config.id).len(), 1);
assert!(!state.merge(network, fields, &limits).unwrap());
}
let next = VipConfig::new("next");
let fields = state.edit(network, &key, 9, &next, None, &limits).unwrap();
state.merge(network, fields, &limits).unwrap();
assert_eq!(state.configs().len(), 1);
}
#[test]
fn reservations_avoid_nodes_and_name_conflicts_have_a_deterministic_holder() {
let network = NetworkId::from_bytes([4; 32]);
let key = iroh::SecretKey::generate();
let limits = Limits::default();
let mut state = VipSet::default();
for version in 1..=2 {
let config = VipConfig::new("shared-name");
let fields = state
.edit(network, &key, version, &config, None, &limits)
.unwrap();
state.merge(network, fields, &limits).unwrap();
}
let range = "10.0.0.0/29".parse().unwrap();
let taken = ["10.0.0.1".parse().unwrap()].into_iter().collect();
let views = state.views(Some(range), taken, &Default::default(), &limits);
assert!(views[0].error.is_none());
assert!(
views[1]
.error
.as_deref()
.unwrap()
.contains("name conflicts")
);
assert_ne!(views[0].address, views[1].address);
for v in views {
assert_ne!(v.address, Some("10.0.0.1".parse().unwrap()));
}
}
+358
View File
@@ -0,0 +1,358 @@
//! Equal-rights retirement uses real membership, signatures and persistent state.
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use tsunagi::discovery::SharedMemoryDiscovery;
use tsunagi::testing::{TestAgent, config_with, network, wait_for_peers, wait_until};
use tsunagi::{Agent, NetworkId};
#[tokio::test]
async fn bounded_marker_batches_eventually_reach_the_tail_of_a_small_send_queue() {
use tsunagi::config::StoragePaths;
use tsunagi::state::{RecordBody, SignedRecord, SignedRetirement};
use tsunagi::storage::StateStore;
let discovery = SharedMemoryDiscovery::new();
let cleaner = TestAgent::spawn(&discovery).await.unwrap();
let replica = TestAgent::spawn(&discovery).await.unwrap();
let (name, secret) = network("forget-batched-history");
let id = cleaner.agent.join_network(&name, &secret).await.unwrap();
replica.agent.join_network(&name, &secret).await.unwrap();
let cleaner_dir = cleaner.stop().await;
let replica_dir = replica.stop().await;
let cleaner_path = StoragePaths::under(cleaner_dir.path()).state_db();
let replica_path = StoragePaths::under(replica_dir.path()).state_db();
// Seed genuine signed offline history in closed stores. Delivery still uses
// real agents, authenticated loopback sessions and the persistent stores.
let targets = tokio::task::spawn_blocking(move || {
let cleaner = StateStore::open(&cleaner_path).unwrap();
let replica = StateStore::open(&replica_path).unwrap();
let author = cleaner.device_identity().unwrap().unwrap();
let signing_key = iroh::SecretKey::from_bytes(&author.secret_bytes());
let mut targets = Vec::new();
for index in 0..12 {
let target = iroh::SecretKey::generate();
let record = SignedRecord::sign(
&target,
id,
1,
RecordBody::Claim {
address: None,
range: None,
hostname: Some(format!("offline-{index}")),
hostname_since: Some(1),
},
);
cleaner.put_signed_record(&record).unwrap();
replica.put_signed_record(&record).unwrap();
cleaner
.publish_own_retirement(&SignedRetirement::sign(
&signing_key,
id,
100 + index,
record,
))
.unwrap();
targets.push(target.public());
}
targets
})
.await
.unwrap();
let replica = Agent::spawn(config_with(replica_dir.path(), &discovery))
.await
.unwrap();
assert_eq!(
replica
.network_status(id)
.await
.unwrap()
.members
.iter()
.filter(|m| targets.contains(&m.endpoint_id))
.count(),
12
);
let mut config = config_with(cleaner_dir.path(), &discovery);
config.limits.max_state_records = 1;
config.limits.session_send_queue = 2;
let cleaner = Agent::spawn(config).await.unwrap();
wait_for_peers(&cleaner, id, 1).await;
wait_until(
"all cleanup batches arrive despite send queue pressure",
|| async {
replica
.network_status(id)
.await
.ok()?
.members
.iter()
.all(|m| !targets.contains(&m.endpoint_id))
.then_some(())
},
)
.await;
cleaner.shutdown().await;
replica.shutdown().await;
let replica_path = StoragePaths::under(replica_dir.path()).state_db();
tokio::task::spawn_blocking(move || {
let replica = StateStore::open(&replica_path).unwrap();
assert_eq!(replica.retirements(id, 16).unwrap().len(), 12);
})
.await
.unwrap();
}
async fn member(
agent: &Agent,
network: NetworkId,
peer: iroh::EndpointId,
) -> tsunagi::agent::MemberStatus {
wait_until("signed member", || async {
agent
.network_status(network)
.await
.ok()?
.members
.into_iter()
.find(|m| m.endpoint_id == peer)
})
.await
}
#[tokio::test]
async fn retirement_survives_restart_and_stale_replicas_but_allows_a_new_claim() {
let discovery = SharedMemoryDiscovery::new();
let cleaner = TestAgent::spawn_with(|c| c.with_hostname("cleaner"), &discovery)
.await
.unwrap();
let target = TestAgent::spawn_with(|c| c.with_hostname("old-device"), &discovery)
.await
.unwrap();
let replica = TestAgent::spawn_with(|c| c.with_hostname("replica"), &discovery)
.await
.unwrap();
let (name, secret) = network("forget-propagation");
let id = cleaner.agent.join_network(&name, &secret).await.unwrap();
target.agent.join_network(&name, &secret).await.unwrap();
replica.agent.join_network(&name, &secret).await.unwrap();
wait_for_peers(&cleaner.agent, id, 2).await;
let peer = target.agent.endpoint_id();
let original = wait_until("all three agents agree on allocated claims", || async {
let a = cleaner.agent.network_status(id).await.ok()?;
let b = target.agent.network_status(id).await.ok()?;
let c = replica.agent.network_status(id).await.ok()?;
for status in [&a, &b, &c] {
if status.members.len() != 3 {
return None;
}
let addresses: std::collections::HashSet<_> = status
.members
.iter()
.filter_map(|m| m.overlay_address_v4)
.collect();
if addresses.len() != 3 {
return None;
}
}
let record = a.members.iter().find(|m| m.endpoint_id == peer)?;
(b.members
.iter()
.find(|m| m.endpoint_id == peer)?
.record_version
== record.record_version
&& c.members
.iter()
.find(|m| m.endpoint_id == peer)?
.record_version
== record.record_version)
.then(|| record.clone())
})
.await;
assert!(
cleaner
.agent
.forget_node(id, peer, original.record_version)
.await
.is_err(),
"online nodes are not history"
);
assert!(
cleaner
.agent
.forget_node(id, cleaner.agent.endpoint_id(), 1)
.await
.is_err()
);
let replica_dir = replica.stop().await;
let target_dir = target.stop().await;
wait_until("target offline", || async {
cleaner
.agent
.network_status(id)
.await
.ok()?
.members
.iter()
.find(|m| m.endpoint_id == peer)
.filter(|m| !m.online)
.map(|_| ())
})
.await;
assert!(
cleaner
.agent
.forget_node(id, peer, original.record_version + 1)
.await
.is_err(),
"stale confirmation changes nothing"
);
let result = cleaner
.agent
.forget_node(id, peer, original.record_version)
.await
.unwrap();
assert_eq!(result.target, peer);
assert_eq!(result.record_version, original.record_version);
assert_eq!(result.peers_told, 0);
assert!(
!cleaner
.agent
.network_status(id)
.await
.unwrap()
.members
.iter()
.any(|m| m.endpoint_id == peer)
);
let cleaner_dir = cleaner.stop().await;
let cleaner =
Agent::spawn(config_with(cleaner_dir.path(), &discovery).with_hostname("cleaner"))
.await
.unwrap();
let replica =
Agent::spawn(config_with(replica_dir.path(), &discovery).with_hostname("replica"))
.await
.unwrap();
wait_for_peers(&cleaner, id, 1).await;
wait_until("stale replica learns durable retirement", || async {
replica
.network_status(id)
.await
.ok()?
.members
.iter()
.all(|m| m.endpoint_id != peer)
.then_some(())
})
.await;
assert!(
!cleaner
.network_status(id)
.await
.unwrap()
.members
.iter()
.any(|m| m.endpoint_id == peer)
);
replica.set_hostname("old-device").await.unwrap();
let replica_id = replica.endpoint_id();
wait_until("released hostname is reclaimed", || async {
cleaner
.network_status(id)
.await
.ok()?
.members
.iter()
.find(|m| m.endpoint_id == replica_id)
.filter(|m| m.hostname.as_deref() == Some("old-device"))?
.hostname_conflict
.is_none()
.then_some(())
})
.await;
let returned =
Agent::spawn(config_with(target_dir.path(), &discovery).with_hostname("old-device"))
.await
.unwrap();
wait_for_peers(&returned, id, 2).await;
wait_until("returning identity publishes a fresh version", || async {
let m = cleaner
.network_status(id)
.await
.ok()?
.members
.into_iter()
.find(|m| m.endpoint_id == peer)?;
(m.online && m.record_version > original.record_version && m.hostname_conflict.is_some())
.then_some(())
})
.await;
cleaner.shutdown().await;
replica.shutdown().await;
returned.shutdown().await;
}
#[tokio::test]
async fn forgetting_is_network_scoped_and_every_member_has_the_same_right() {
let discovery = SharedMemoryDiscovery::new();
let a = TestAgent::spawn(&discovery).await.unwrap();
let b = TestAgent::spawn(&discovery).await.unwrap();
let target = TestAgent::spawn(&discovery).await.unwrap();
let (name, secret) = network("forget-one-network");
let (other_name, other_secret) = network("keep-other-network");
let id = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
target.agent.join_network(&name, &secret).await.unwrap();
let other = a
.agent
.join_network(&other_name, &other_secret)
.await
.unwrap();
target
.agent
.join_network(&other_name, &other_secret)
.await
.unwrap();
let peer = target.agent.endpoint_id();
let record = member(&b.agent, id, peer).await;
member(&a.agent, other, peer).await;
target.agent.shutdown().await;
wait_until("offline on B", || async {
b.agent
.network_status(id)
.await
.ok()?
.members
.iter()
.find(|m| m.endpoint_id == peer)
.filter(|m| !m.online)
.map(|_| ())
})
.await;
b.agent
.forget_node(id, peer, record.record_version)
.await
.unwrap();
wait_until("A learns B's retirement", || async {
a.agent
.network_status(id)
.await
.ok()?
.members
.iter()
.all(|m| m.endpoint_id != peer)
.then_some(())
})
.await;
assert!(
a.agent
.network_status(other)
.await
.unwrap()
.members
.iter()
.any(|m| m.endpoint_id == peer)
);
a.agent.shutdown().await;
b.agent.shutdown().await;
}
@@ -0,0 +1,4 @@
//! Exercise the vendored retry admission in the default workspace suite.
//! Transport integration tests still use real iroh endpoints.
#[path = "../../../vendor/iroh/src/socket/remote_map/remote_state/pending_paths.rs"]
mod pending_paths;
+207
View File
@@ -0,0 +1,207 @@
//! Shared services use real membership, signed state and mandatory SQLite.
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use tsunagi::discovery::SharedMemoryDiscovery;
use tsunagi::testing::{TestAgent, config_with, network, wait_for_peers, wait_until};
use tsunagi::vip::{Backend, Listener, Protocol, VipConfig, VipStatus};
use tsunagi::{Agent, NetworkId};
async fn view(agent: &Agent, network: NetworkId, id: &str) -> VipStatus {
wait_until("VIP configuration reaches the replica", || async {
agent
.network_status(network)
.await
.ok()?
.vips
.into_iter()
.find(|v| v.config.id == id)
})
.await
}
#[tokio::test]
async fn any_member_can_edit_services_and_an_offline_creator_is_not_required() {
let discovery = SharedMemoryDiscovery::new();
let a = TestAgent::spawn(&discovery).await.unwrap();
let b = TestAgent::spawn(&discovery).await.unwrap();
let (name, secret) = network("shared-vips");
let net = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
wait_for_peers(&a.agent, net, 1).await;
let cfg = VipConfig::new("api");
a.agent
.save_vip(net, cfg.clone(), None, false)
.await
.unwrap();
let before = view(&b.agent, net, &cfg.id).await;
let mut edited = before.config.clone();
let mut listener = Listener::new(Protocol::Tcp, 443);
listener.backends.push(Backend {
peer: b.agent.endpoint_id().to_string(),
port: 6443,
});
edited.listeners.push(listener);
b.agent
.save_vip(net, edited.clone(), Some(before.revision.clone()), false)
.await
.unwrap();
assert!(
b.agent
.save_vip(net, cfg.clone(), Some(before.revision), false)
.await
.is_err()
);
wait_until("creator observes another member's edit", || async {
(view(&a.agent, net, &cfg.id).await.config == edited).then_some(())
})
.await;
let a_dir = a.stop().await;
let c = TestAgent::spawn(&discovery).await.unwrap();
c.agent.join_network(&name, &secret).await.unwrap();
assert_eq!(view(&c.agent, net, &cfg.id).await.config, edited);
let a = Agent::spawn(config_with(a_dir.path(), &discovery))
.await
.unwrap();
assert_eq!(view(&a, net, &cfg.id).await.config, edited);
let current = view(&c.agent, net, &cfg.id).await;
c.agent
.save_vip(net, current.config, Some(current.revision), true)
.await
.unwrap();
wait_until("deletion reaches the original creator", || async {
a.network_status(net)
.await
.ok()?
.vips
.is_empty()
.then_some(())
})
.await;
a.shutdown().await;
b.stop().await;
c.stop().await;
}
#[tokio::test]
async fn invalid_drafts_are_atomic_and_services_do_not_cross_networks() {
let discovery = SharedMemoryDiscovery::new();
let a = TestAgent::spawn(&discovery).await.unwrap();
let (name, secret) = network("vip-one");
let net = a.agent.join_network(&name, &secret).await.unwrap();
let (other, secret) = network("vip-two");
let second = a.agent.join_network(&other, &secret).await.unwrap();
let mut cfg = VipConfig::new("api");
cfg.listeners.push(Listener::new(Protocol::Udp, 53));
a.agent
.save_vip(net, cfg.clone(), None, false)
.await
.unwrap();
let before = view(&a.agent, net, &cfg.id).await;
let mut invalid = before.config.clone();
invalid.name = "renamed".into();
invalid.listeners.push(invalid.listeners[0].clone());
assert!(
a.agent
.save_vip(net, invalid, Some(before.revision.clone()), false)
.await
.is_err()
);
let after = view(&a.agent, net, &cfg.id).await;
assert_eq!(after.config, before.config);
assert_eq!(after.revision, before.revision);
assert!(
a.agent
.network_status(second)
.await
.unwrap()
.vips
.is_empty()
);
a.stop().await;
}
#[tokio::test]
async fn a_restored_author_advances_its_counter_past_its_own_updates_held_by_a_replica() {
use tsunagi::config::StoragePaths;
use tsunagi::storage::StateStore;
use tsunagi::vip::VipSet;
let discovery = SharedMemoryDiscovery::new();
let a = TestAgent::spawn(&discovery).await.unwrap();
let b = TestAgent::spawn(&discovery).await.unwrap();
let (name, secret) = network("vip-author-recovery");
let net = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
wait_for_peers(&a.agent, net, 1).await;
let config = VipConfig::new("api");
a.agent
.save_vip(net, config.clone(), None, false)
.await
.unwrap();
view(&b.agent, net, &config.id).await;
let author = a.agent.endpoint_id();
let adir = a.stop().await;
let bdir = b.stop().await;
let apath = StoragePaths::under(adir.path()).state_db();
let bpath = StoragePaths::under(bdir.path()).state_db();
// Model an old author backup with a newer, genuinely signed update retained
// by another offline replica. Connections and recovery below remain real.
tokio::task::spawn_blocking(move || {
let a = StateStore::open(apath).unwrap();
let b = StateStore::open(bpath).unwrap();
let mut set = VipSet::default();
set.merge(
net,
a.vip_fields(net, 192).unwrap(),
&tsunagi::Limits::default(),
)
.unwrap();
let mut cfg = set.configs().remove(0);
cfg.name = "newer".into();
let key =
iroh::SecretKey::from_bytes(&a.device_identity().unwrap().unwrap().secret_bytes());
let fields = set
.edit(
net,
&key,
100,
&cfg,
Some(&set.revision(&cfg.id)),
&tsunagi::Limits::default(),
)
.unwrap();
set.merge(net, fields, &tsunagi::Limits::default()).unwrap();
b.put_vip_fields(net, &set.entries(&cfg.id), None).unwrap();
})
.await
.unwrap();
let b = Agent::spawn(config_with(bdir.path(), &discovery))
.await
.unwrap();
let a = Agent::spawn(config_with(adir.path(), &discovery))
.await
.unwrap();
let learned = wait_until(
"the restored author learns its newer signed service",
|| async {
let v = view(&a, net, &config.id).await;
(v.config.name == "newer").then_some(v)
},
)
.await;
let mut next = learned.config;
next.name = "latest".into();
a.save_vip(net, next, Some(learned.revision), false)
.await
.unwrap();
a.shutdown().await;
let path = StoragePaths::under(adir.path()).state_db();
let version = tokio::task::spawn_blocking(move || {
StateStore::open(path)
.unwrap()
.own_record_version(net, author)
.unwrap()
})
.await
.unwrap();
assert!(
version > 100,
"the observed author counter must not be reused"
);
b.shutdown().await;
}
+90 -2
View File
@@ -21,6 +21,24 @@ const POLL_INTERVAL: Duration = Duration::from_secs(2);
/// A request from the UI to the agent.
pub(crate) enum Command {
SaveVip {
token: String,
network_id: String,
config: tsunagi::vip::VipConfig,
expected: Option<String>,
delete: bool,
},
/// Retire the exact offline-node claim selected in the devices window.
ForgetNode {
network_id: String,
peer: String,
expected_version: u64,
},
/// Replace the network's enabled protocols in preference order.
SetProtocols {
network_id: String,
protocols: Vec<String>,
},
/// Stop serving a network, or start serving it again.
SetActive { network_id: String, active: bool },
/// Turn local broadcast participation on or off for a network.
@@ -44,8 +62,14 @@ pub(crate) enum Command {
}
/// The latest the worker knows, read by the UI each frame.
#[derive(Clone)]
pub(crate) struct VipSaveResult {
pub token: String,
pub result: Result<Vec<tsunagi::vip::VipStatus>, String>,
}
#[derive(Clone, Default)]
pub(crate) struct Snapshot {
pub vip_result: Option<VipSaveResult>,
/// The last status, or the reason it could not be read (agent down, no
/// permission on the socket, …).
pub status: Option<Result<StatusReport, String>>,
@@ -73,6 +97,18 @@ pub(crate) struct AgentClient {
}
impl AgentClient {
#[cfg(test)]
pub(crate) fn test_client() -> (Self, mpsc::UnboundedReceiver<Command>) {
let (tx, rx) = mpsc::unbounded_channel();
(
Self {
tx,
shared: Arc::new(Mutex::new(Snapshot::default())),
pending_copy: Arc::new(Mutex::new(None)),
},
rx,
)
}
/// Starts the worker on `runtime`, reporting to `ctx` (for repaints).
pub(crate) fn spawn(runtime: &tokio::runtime::Handle, ctx: egui::Context) -> Self {
let (tx, rx) = mpsc::unbounded_channel();
@@ -106,7 +142,13 @@ impl AgentClient {
/// Sends a command; a dropped worker simply means nothing happens.
pub(crate) fn send(&self, command: Command) {
let _ = self.tx.send(command);
// Disable protocol editing until the refreshed snapshot arrives;
// otherwise a second checkbox click could replace the first edit
// using the same stale network selection.
lock(&self.shared).busy = true;
if self.tx.send(command).is_err() {
lock(&self.shared).busy = false;
}
}
/// A copy of the latest snapshot.
@@ -142,6 +184,8 @@ async fn worker(
let Some(command) = command else { return };
run(&resolve_socket(), command, &shared, &pending_copy, &ctx).await;
refresh(&shared, &ctx).await;
lock(&shared).busy = false;
ctx.request_repaint();
}
}
}
@@ -179,6 +223,51 @@ async fn run(
ctx.request_repaint();
let outcome: Result<String, String> = match command {
Command::SaveVip {
token,
network_id,
config,
expected,
delete,
} => {
let result = ipc::save_vip(socket, &network_id, config, expected, delete)
.await
.map_err(|e| e.to_string());
lock(shared).vip_result = Some(VipSaveResult {
token,
result: result.clone(),
});
result.map(|_| "VIP configuration saved".into())
}
Command::ForgetNode {
network_id,
peer,
expected_version,
} => ipc::forget_node(socket, &network_id, &peer, expected_version)
.await
.map(|report| {
format!(
"forgot node claim v{}; cleanup queued to {} peers",
report.record_version, report.peers_told
)
})
.map_err(|err| err.to_string()),
Command::SetProtocols {
network_id,
protocols,
} => ipc::set_protocols(socket, &network_id, protocols)
.await
.map(|names| {
format!(
"protocols: {}",
if names.is_empty() {
"none".into()
} else {
names.join(", ")
}
)
})
.map_err(|err| err.to_string()),
Command::SetActive { network_id, active } => ipc::set_active(socket, &network_id, active)
.await
.map(|report| {
@@ -260,7 +349,6 @@ async fn run(
{
let mut guard = lock(shared);
guard.busy = false;
guard.last_action = if matches!(&outcome, Ok(message) if message.is_empty()) {
None
} else {
+74 -6
View File
@@ -15,6 +15,12 @@ use crate::devices;
use crate::stats::Traffic;
use crate::ui::{self, UiState};
fn app_fonts() -> egui::FontDefinitions {
let mut fonts = egui::FontDefinitions::default();
egui_phosphor::add_to_fonts(&mut fonts, egui_phosphor::Variant::Regular);
fonts
}
/// The window's state.
pub(crate) struct App {
agent: AgentClient,
@@ -34,9 +40,7 @@ impl App {
let ctx = cc.egui_ctx.clone();
// Add the Phosphor icon glyphs to the font set so the icon buttons
// render instead of showing as tofu.
let mut fonts = egui::FontDefinitions::default();
egui_phosphor::add_to_fonts(&mut fonts, egui_phosphor::Variant::Regular);
ctx.set_fonts(fonts);
ctx.set_fonts(app_fonts());
let agent = AgentClient::spawn(&runtime, ctx);
Ok(Self {
@@ -60,6 +64,7 @@ impl eframe::App for App {
ui::draw(ctx, &self.agent, &mut self.state, &snapshot, &self.traffic);
self.show_devices_windows(ctx, &snapshot);
self.show_vip_windows(ctx, &snapshot);
self.show_about_window(ctx, &snapshot);
// A secret the worker fetched for the clipboard: copy it here, on the
@@ -74,6 +79,51 @@ impl eframe::App for App {
}
impl App {
fn show_vip_windows(&mut self, ctx: &egui::Context, snapshot: &crate::agent::Snapshot) {
let Some(Ok(report)) = &snapshot.status else {
return;
};
let ids: Vec<_> = self.state.vip_windows.keys().cloned().collect();
for id in ids {
let Some(network) = report.networks.iter().find(|n| n.network_id == id) else {
self.state.vip_windows.remove(&id);
continue;
};
let Some(state) = self.state.vip_windows.get_mut(&id) else {
continue;
};
let builder = egui::ViewportBuilder::default()
.with_title(format!("tsunagi · VIP · {}", network.name))
.with_app_id(crate::icon::APP_ID)
.with_icon(crate::icon::window())
.with_inner_size([1280., 860.])
.with_min_inner_size([650., 400.]);
let keep = ctx.show_viewport_immediate(
egui::ViewportId::from_hash_of(("vip", &id)),
builder,
|vctx, _| {
let close = crate::vip::show(vctx, &self.agent, state, network, snapshot);
if close {
return false;
}
if vctx.input(|i| i.viewport().close_requested()) {
if state.dirty() {
state.request_close();
vctx.send_viewport_cmd(egui::ViewportCommand::CancelClose);
true
} else {
false
}
} else {
true
}
},
);
if !keep {
self.state.vip_windows.remove(&id);
}
}
}
/// Renders the About window while it is open.
fn show_about_window(&mut self, ctx: &egui::Context, snapshot: &crate::agent::Snapshot) {
if !self.state.about_open {
@@ -116,20 +166,21 @@ impl App {
.with_title(devices::title(network))
.with_app_id(crate::icon::APP_ID)
.with_icon(crate::icon::window())
.with_inner_size([940.0, 540.0])
.with_min_inner_size([480.0, 320.0]);
.with_inner_size([1440.0, 640.0])
.with_min_inner_size([800.0, 400.0]);
let agent = &self.agent;
let traffic = &self.traffic;
// The window shares the single unit setting, so its toggle and the
// main window's stay in step.
let unit = &mut self.state.unit;
let confirm = &mut self.state.exit_confirm;
let forget_confirm = &mut self.state.forget_confirm;
let keep = ctx.show_viewport_immediate(viewport_id, builder, |vctx, _class| {
devices::show(
vctx,
agent,
unit,
confirm,
(confirm, forget_confirm),
network,
&report.endpoint_id,
traffic,
@@ -142,3 +193,20 @@ impl App {
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn route_icons_have_an_embedded_font_in_the_app() {
let fonts = app_fonts();
assert!(fonts.font_data.contains_key("phosphor"));
assert!(
fonts
.families
.get(&egui::FontFamily::Proportional)
.is_some_and(|family| family.iter().any(|name| name == "phosphor"))
);
}
}
File diff suppressed because it is too large. Load diff
+3
View File
@@ -17,8 +17,11 @@ mod exit;
mod format;
mod icon;
mod stats;
mod table;
mod tray;
mod ui;
mod vip;
mod vip_status;
mod watch;
use eframe::egui;
+1 -1
View File
@@ -34,7 +34,7 @@ pub(crate) struct Series {
}
impl Series {
fn sample(&mut self, now: Instant, packets: u64, bytes: u64) {
pub(crate) fn sample(&mut self, now: Instant, packets: u64, bytes: u64) {
if let Some((then, prev_packets, prev_bytes)) = self.last {
let dt = now.saturating_duration_since(then).as_secs_f32();
if dt > 0.0 {
+88
View File
@@ -0,0 +1,88 @@
//! Fixed row geometry for dense live tables. Children cannot resize a row or
//! paint into its neighbours; the enclosing scroll area owns overflow.
use eframe::egui;
pub(crate) const GAP: f32 = 10.0;
pub(crate) fn row(
ui: &mut egui::Ui,
id: impl std::hash::Hash,
widths: &[f32],
stripe: bool,
mut cell: impl FnMut(usize, &mut egui::Ui),
) {
let font = egui::FontSelection::Default.resolve(ui.style());
let text_height = ui.fonts(|fonts| fonts.row_height(&font));
let height = ui.spacing().interact_size.y.max(text_height) + 10.0;
let width = widths.iter().sum::<f32>() + GAP * widths.len().saturating_sub(1) as f32;
let (rect, _) = ui.allocate_exact_size(egui::vec2(width, height), egui::Sense::hover());
if stripe {
ui.painter()
.rect_filled(rect, 2.0, ui.visuals().faint_bg_color);
}
let mut x = rect.left();
for (column, width) in widths.iter().enumerate() {
let bounds = egui::Rect::from_min_size(
egui::pos2(x, rect.top() + 3.0),
egui::vec2(*width, height - 6.0),
);
let mut child = ui.new_child(
egui::UiBuilder::new()
.id_salt((&id, column))
.max_rect(bounds)
.layout(egui::Layout::left_to_right(egui::Align::Center)),
);
child.set_clip_rect(ui.clip_rect().intersect(bounds));
child.style_mut().wrap_mode = Some(egui::TextWrapMode::Truncate);
cell(column, &mut child);
x += width + GAP;
}
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
#[test]
fn oversized_content_is_clipped_to_its_cell_at_every_scale() {
for size in [13.0, 20.0, 28.0] {
let ctx = egui::Context::default();
ctx.style_mut(|style| {
style.override_font_id = Some(egui::FontId::proportional(size));
});
let output = ctx.run(egui::RawInput::default(), |ctx| {
egui::CentralPanel::default().show(ctx, |ui| {
for index in 0..4 {
row(ui, index, &[90.0, 120.0], index % 2 == 0, |column, ui| {
if column == 0 {
ui.horizontal(|ui| {
ui.label("very long device name that cannot fit");
ui.button("Forget").on_hover_text("Forget");
});
} else {
ui.label(format!("row {index}"));
}
});
}
});
});
let mut previous_bottom = 0.0;
for index in 0..4 {
let shape = output.shapes.iter().find(|shape| matches!(&shape.shape, egui::Shape::Text(t) if t.galley.text() == format!("row {index}"))).unwrap();
let egui::Shape::Text(text) = &shape.shape else {
unreachable!()
};
assert!(text.visual_bounding_rect().top() >= previous_bottom);
assert!(
shape.clip_rect.contains_rect(text.visual_bounding_rect()),
"font {size}: clip {:?}, text {:?}",
shape.clip_rect,
text.visual_bounding_rect()
);
assert!(shape.clip_rect.left() >= 108.0);
previous_bottom = shape.clip_rect.bottom();
}
}
}
}
+92 -4
View File
@@ -41,10 +41,13 @@ pub(crate) struct UiState {
pub(crate) host_edit: Option<String>,
/// Network ids whose devices window is open.
pub(crate) open_devices: BTreeSet<String>,
pub(crate) vip_windows: std::collections::BTreeMap<String, crate::vip::Window>,
/// Whether the About window is open.
pub(crate) about_open: bool,
/// An exit-node choice waiting for the user to confirm it.
pub(crate) exit_confirm: Option<devices::ExitConfirm>,
/// A selected offline claim waiting for cleanup confirmation.
pub(crate) forget_confirm: Option<devices::ForgetConfirm>,
}
impl Default for UiState {
@@ -54,8 +57,10 @@ impl Default for UiState {
unit: Unit::Packets,
host_edit: None,
open_devices: BTreeSet::new(),
vip_windows: Default::default(),
about_open: false,
exit_confirm: None,
forget_confirm: None,
}
}
}
@@ -123,7 +128,7 @@ pub(crate) fn draw(
ui.weak("No networks yet — join one below.");
}
for network in &report.networks {
draw_network(ui, agent, state, traffic, network, &report.endpoint_id);
draw_network(ui, agent, state, traffic, network, report);
}
});
}
@@ -202,12 +207,13 @@ fn draw_network(
state: &mut UiState,
traffic: &Traffic,
network: &NetworkReport,
own_id: &str,
report: &StatusReport,
) {
ui.add_space(6.0);
egui::Frame::group(ui.style()).show(ui, |ui| {
ui.horizontal(|ui| {
format::copy_field(ui, network.name.as_str(), &network.name);
if ui.button("VIP").clicked(){state.vip_windows.entry(network.network_id.clone()).or_default();}
ui.with_layout(egui::Layout::right_to_left(egui::Align::Center), |ui| {
let leave =
egui::Button::new(egui::RichText::new("Leave").color(egui::Color32::WHITE))
@@ -244,12 +250,14 @@ fn draw_network(
ui.weak("peers");
ui.label(format!(
"{} online · {} known",
devices::online_count(network, own_id),
devices::known_count(network, own_id)
devices::online_count(network, &report.endpoint_id),
devices::known_count(network, &report.endpoint_id)
));
ui.end_row();
});
draw_protocols(ui, agent, network, &report.available_protocols);
format::sparkline(ui, series, state.unit, 34.0);
if !network.range_conflicts.is_empty() {
@@ -345,6 +353,86 @@ fn draw_network(
});
}
fn draw_protocols(
ui: &mut egui::Ui,
agent: &AgentClient,
network: &NetworkReport,
available: &[tsunagi::ipc::ProtocolReport],
) {
let summary = if network.protocols.is_empty() {
"none".into()
} else {
network.protocols.join(", ")
};
egui::CollapsingHeader::new(format!("protocols {summary}"))
.id_salt(("protocols", &network.network_id))
.show(ui, |ui| {
ui.weak("Preference order; changes apply immediately and are saved.");
let mut selected = network.protocols.clone();
let mut changed = false;
ui.add_enabled_ui(!agent.snapshot().busy, |ui| {
// Enabled items follow the user's preference; unchecked items
// follow the agent's catalog. The worker refreshes after edits.
let ordered: Vec<_> = network
.protocols
.iter()
.filter_map(|name| available.iter().find(|p| &p.name == name))
.chain(
available
.iter()
.filter(|p| !network.protocols.contains(&p.name)),
)
.collect();
for protocol in ordered {
ui.horizontal(|ui| {
let mut on = selected.contains(&protocol.name);
if ui.checkbox(&mut on, &protocol.name).changed() {
if on {
selected.push(protocol.name.clone());
} else {
selected.retain(|p| p != &protocol.name);
}
changed = true;
}
if let Some(index) = selected.iter().position(|p| p == &protocol.name) {
if ui
.add_enabled(index > 0, egui::Button::new(icon::ARROW_UP))
.on_hover_text("prefer earlier")
.clicked()
{
selected.swap(index, index - 1);
changed = true;
}
if ui
.add_enabled(
index + 1 < selected.len(),
egui::Button::new(icon::ARROW_DOWN),
)
.on_hover_text("prefer later")
.clicked()
{
selected.swap(index, index + 1);
changed = true;
}
}
});
ui.indent(("description", &network.network_id, &protocol.name), |ui| {
ui.weak(&protocol.summary);
});
}
});
if selected.is_empty() {
ui.weak("Control only; no IP traffic in this network.");
}
if changed {
agent.send(Command::SetProtocols {
network_id: network.network_id.clone(),
protocols: selected,
});
}
});
}
/// A prominent line about an exit node. One that carries a command copies it
/// when clicked.
fn draw_exit_warning(ui: &mut egui::Ui, warning: &exit::Warning) {
+402
View File
@@ -0,0 +1,402 @@
//! Per-network VIP editor. Drafts never issue commands until Save is pressed.
use crate::agent::{AgentClient, Command, Snapshot};
use crate::vip_status;
use eframe::egui;
use tsunagi::ipc::NetworkReport;
use tsunagi::vip::{Backend, Listener, Protocol, VipConfig};
#[derive(Clone)]
struct Draft {
config: VipConfig,
original: VipConfig,
revision: Option<String>,
delete: bool,
}
impl Draft {
fn new(config: VipConfig, revision: Option<String>) -> Self {
Self {
original: config.clone(),
config,
revision,
delete: false,
}
}
fn dirty(&self) -> bool {
self.revision.is_none() || self.delete || self.config != self.original
}
}
#[derive(Default)]
pub(crate) struct Window {
draft: Option<Draft>,
pending: Option<String>,
error: Option<String>,
close_question: bool,
close_after_save: bool,
live: vip_status::View,
configure: bool,
generation: u64,
acknowledged_generation: Option<u64>,
}
impl Window {
pub fn request_close(&mut self) {
self.close_question = true;
self.close_after_save = true;
}
pub fn dirty(&self) -> bool {
self.draft.as_ref().is_some_and(Draft::dirty) || self.pending.is_some()
}
}
fn peer_name<'a>(n: &'a NetworkReport, id: &'a str) -> &'a str {
n.members
.iter()
.find(|m| m.endpoint_id == id)
.and_then(|m| m.hostname.as_deref())
.unwrap_or(id)
}
pub(crate) fn show(
ctx: &egui::Context,
agent: &AgentClient,
state: &mut Window,
n: &NetworkReport,
snapshot: &Snapshot,
) -> bool {
if state.generation != snapshot.generation {
state.generation = snapshot.generation;
if let Some(at) = snapshot.at {
state.live.observe(n, at);
}
}
if let Some(result) = &snapshot.vip_result
&& state.pending.as_deref() == Some(result.token.as_str())
{
state.pending = None;
state.acknowledged_generation = Some(snapshot.generation);
match &result.result {
Ok(vips) => {
if let Some(draft) = &state.draft {
state.draft = vips
.iter()
.find(|v| v.config.id == draft.config.id)
.map(|v| Draft::new(v.config.clone(), Some(v.revision.clone())));
}
state.error = None;
}
Err(error) => {
state.error = Some(error.clone());
state.close_after_save = false;
}
}
}
let mut close = state.close_after_save && !state.dirty();
let mut discard_new = false;
egui::CentralPanel::default().show(ctx, |ui| {
ui.heading(format!("VIP · {}", n.name));
ui.weak("Configuration is shared with every member. Changes take effect only after Save.");
if state.close_question {
ui.horizontal(|ui| {
ui.label("Unsaved changes.");
if ui
.add_enabled(
state.pending.is_none(),
egui::Button::new("Discard and close"),
)
.clicked()
{
state.draft = None;
state.close_question = false;
close = true;
}
if ui.button("Keep editing").clicked() {
state.close_question = false;
state.close_after_save = false;
}
});
}
if let Some(error) = &state.error {
ui.colored_label(egui::Color32::LIGHT_RED, error);
}
ui.separator();
ui.horizontal_wrapped(|ui| {
for vip in &n.vips {
let selected = state
.draft
.as_ref()
.is_some_and(|d| d.config.id == vip.config.id);
if ui.selectable_label(selected, &vip.config.name).clicked() {
if state.dirty() {
state.error = Some(
"Save or discard the current draft before selecting another VIP."
.into(),
);
} else {
state.draft =
Some(Draft::new(vip.config.clone(), Some(vip.revision.clone())));
state.error = None;
}
}
}
if ui.button("+ New VIP").clicked() {
if state.dirty() {
state.error = Some("Save or discard the current draft first.".into());
} else {
state.draft = Some(Draft::new(VipConfig::new(""), None));
state.configure = true;
state.error = None;
}
}
});
let Some(draft) = &mut state.draft else {
ui.label("Select a VIP or create one.");
return;
};
let current = n.vips.iter().find(|v| v.config.id == draft.config.id);
if state.pending.is_none()
&& state.acknowledged_generation != Some(snapshot.generation)
&& !draft.dirty()
&& let Some(current) = current
&& draft.revision.as_ref() != Some(&current.revision)
{
*draft = Draft::new(current.config.clone(), Some(current.revision.clone()));
}
let changed =
current.is_some_and(|v| draft.revision.as_ref().is_some_and(|r| r != &v.revision));
if changed {
ui.colored_label(
egui::Color32::YELLOW,
"This VIP changed on the network. Reload before saving; your draft is retained.",
);
}
if draft.revision.is_some() && current.is_none() {
ui.colored_label(
egui::Color32::YELLOW,
"This VIP was removed on the network. The draft cannot overwrite its deletion.",
);
}
let busy = state.pending.is_some();
ui.horizontal(|ui| {
let valid = draft.config.validate(&tsunagi::Limits::default());
if !n.active {
ui.weak("Start the network to save changes.");
}
if ui
.add_enabled(
n.active
&& !busy
&& draft.dirty()
&& !changed
&& (valid.is_ok() || draft.delete),
egui::Button::new("Save"),
)
.clicked()
{
let token = VipConfig::new("").id;
state.pending = Some(token.clone());
agent.send(Command::SaveVip {
token,
network_id: n.network_id.clone(),
config: draft.config.clone(),
expected: draft.revision.clone(),
delete: draft.delete,
});
}
if ui
.add_enabled(!busy, egui::Button::new("Discard / reload"))
.clicked()
{
if let Some(v) = current {
*draft = Draft::new(v.config.clone(), Some(v.revision.clone()));
} else {
discard_new = true;
}
state.error = None;
}
if busy {
ui.spinner();
ui.label("Saving…");
} else if draft.dirty() {
ui.weak("Unsaved changes");
}
if let Err(error) = valid {
ui.colored_label(egui::Color32::LIGHT_RED, error.to_string());
}
});
ui.separator();
ui.horizontal(|ui| {
ui.selectable_value(&mut state.configure, false, "Overview");
ui.selectable_value(&mut state.configure, true, "Configuration");
});
egui::ScrollArea::vertical()
.auto_shrink([false, false])
.show(ui, |ui| {
if state.configure {
ui.add_enabled_ui(!busy, |ui| {
ui.horizontal(|ui| {
ui.label("Name");
ui.add(
egui::TextEdit::singleline(&mut draft.config.name)
.id(egui::Id::new(("vip-name", &draft.config.id))),
);
ui.weak(format!(".{}", n.name));
});
if draft.revision.is_some() {
ui.checkbox(&mut draft.delete, "Delete this VIP when saved");
}
if !draft.delete {
edit_listeners(ui, &mut draft.config, n);
}
});
} else if let Some(current) = current {
vip_status::show(ui, current, n, &mut state.live);
} else {
ui.weak("Save this VIP to see its live status.");
}
});
});
if discard_new {
state.draft = None;
state.error = None;
}
close
}
fn edit_listeners(ui: &mut egui::Ui, config: &mut VipConfig, n: &NetworkReport) {
let mut remove = None;
for (index, l) in config.listeners.iter_mut().enumerate() {
ui.push_id(index,|ui|{egui::Frame::group(ui.style()).show(ui,|ui|{
ui.horizontal(|ui|{egui::ComboBox::from_id_salt("protocol").selected_text(l.protocol.to_string()).show_ui(ui,|ui|{ui.selectable_value(&mut l.protocol,Protocol::Tcp,"TCP");ui.selectable_value(&mut l.protocol,Protocol::Udp,"UDP");});ui.label("Port");ui.add(egui::DragValue::new(&mut l.port).range(1..=65535));if ui.button("Remove listener").clicked(){remove=Some(index);}});
if l.protocol==Protocol::Tcp{ui.checkbox(&mut l.port_check,"Active TCP port checks");}else{ui.weak("UDP health: encrypted path probes and explicit port errors; application readiness is unknown.");}
ui.collapsing("Checks and flow expiry",|ui|{egui::Grid::new("settings").show(ui,|ui|{
ui.label("Check interval, ms");ui.add(egui::DragValue::new(&mut l.check_interval_ms).range(250..=3_600_000));ui.end_row();
ui.label("Check timeout, ms");ui.add(egui::DragValue::new(&mut l.check_timeout_ms).range(100..=60_000));ui.end_row();
ui.label("Failures before exclusion");ui.add(egui::DragValue::new(&mut l.failure_threshold).range(1..=255));ui.end_row();
ui.label("Successes before recovery");ui.add(egui::DragValue::new(&mut l.recovery_threshold).range(1..=255));ui.end_row();
ui.label("Flow idle timeout, seconds");ui.add(egui::DragValue::new(&mut l.idle_timeout_secs).range(1..=604800));ui.end_row();
});});
let mut remove_backend=None;for(i,b)in l.backends.iter_mut().enumerate(){ui.push_id(i,|ui|{ui.horizontal(|ui|{
egui::ComboBox::from_id_salt("node").width(240.).selected_text(peer_name(n,&b.peer)).show_ui(ui,|ui|{for m in &n.members{let label=format!("{} ({})",m.hostname.as_deref().unwrap_or("unnamed"),m.endpoint_id.chars().take(8).collect::<String>());ui.selectable_value(&mut b.peer,m.endpoint_id.clone(),label);}});
ui.label("Target port");ui.add(egui::DragValue::new(&mut b.port).range(1..=65535));if ui.button("Remove").clicked(){remove_backend=Some(i);}
});});}if let Some(i)=remove_backend{l.backends.remove(i);}
if ui.button("+ Backend").clicked(){l.backends.push(Backend{peer:n.members.first().map(|m|m.endpoint_id.clone()).unwrap_or_default(),port:l.port});}
});});
}
if let Some(index) = remove {
config.listeners.remove(index);
}
if ui.button("+ Listener").clicked() {
config.listeners.push(Listener::new(Protocol::Tcp, 443));
}
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
use tsunagi::vip::VipStatus;
#[test]
fn editing_the_actual_form_sends_nothing_until_save_is_clicked() {
let (agent, mut commands) = AgentClient::test_client();
let ctx = egui::Context::default();
let mut config = VipConfig::new("api");
config.listeners.push(Listener::new(Protocol::Tcp, 443));
let mut state = Window {
draft: Some(Draft::new(config.clone(), Some("revision".into()))),
configure: true,
..Default::default()
};
let network = NetworkReport {
active: true,
name: "lab".into(),
vips: vec![VipStatus {
config: config.clone(),
revision: "revision".into(),
address: None,
error: None,
listeners: vec![],
}],
..Default::default()
};
let snapshot = Snapshot::default();
let frame = |events: Vec<egui::Event>, state: &mut Window| {
ctx.run(
egui::RawInput {
screen_rect: Some(egui::Rect::from_min_size(
egui::Pos2::ZERO,
egui::vec2(1000., 800.),
)),
events,
..Default::default()
},
|ctx| {
show(ctx, &agent, state, &network, &snapshot);
},
)
};
frame(vec![], &mut state);
ctx.memory_mut(|m| m.request_focus(egui::Id::new(("vip-name", &config.id))));
frame(vec![egui::Event::Text("new".into())], &mut state);
assert!(state.dirty());
assert_ne!(state.draft.as_ref().unwrap().config.name, config.name);
assert!(commands.try_recv().is_err());
assert_eq!(network.vips[0].config, config);
fn save_position(shape: &egui::Shape) -> Option<egui::Pos2> {
match shape {
egui::Shape::Text(text) if text.galley.text() == "Save" => {
Some(text.pos + text.galley.rect.center().to_vec2())
}
egui::Shape::Vec(shapes) => shapes.iter().find_map(save_position),
_ => None,
}
}
// Save was disabled before this edit; allow the next layout to register
// its now-enabled hit target before dispatching the pointer press.
let ready = frame(vec![], &mut state);
let position = ready
.shapes
.iter()
.find_map(|s| save_position(&s.shape))
.unwrap();
frame(
vec![
egui::Event::PointerMoved(position),
egui::Event::PointerButton {
pos: position,
button: egui::PointerButton::Primary,
pressed: true,
modifiers: egui::Modifiers::NONE,
},
],
&mut state,
);
frame(
vec![egui::Event::PointerButton {
pos: position,
button: egui::PointerButton::Primary,
pressed: false,
modifiers: egui::Modifiers::NONE,
}],
&mut state,
);
let Command::SaveVip {
config: saved,
expected,
delete,
..
} = commands.try_recv().unwrap()
else {
panic!("Save must submit one complete VIP draft");
};
assert_ne!(saved.name, config.name);
assert_eq!(expected.as_deref(), Some("revision"));
assert!(!delete);
assert!(commands.try_recv().is_err());
}
#[test]
fn all_listener_settings_render_without_a_live_agent() {
let ctx = egui::Context::default();
let mut cfg = VipConfig::new("api");
cfg.listeners.push(Listener::new(Protocol::Tcp, 443));
cfg.listeners.push(Listener::new(Protocol::Udp, 53));
let n = NetworkReport::default();
let _ = ctx.run(egui::RawInput::default(), |ctx| {
egui::CentralPanel::default().show(ctx, |ui| edit_listeners(ui, &mut cfg, &n));
});
assert_eq!(cfg.listeners.len(), 2);
}
}
+885
View File
@@ -0,0 +1,885 @@
//! Local VIP observations, kept apart from editable configuration drafts.
use std::collections::BTreeMap;
use std::time::{Duration, Instant};
use eframe::egui;
use tsunagi::ipc::NetworkReport;
use tsunagi::vip::{Backend, BackendStatus, Counters, ListenerStatus, Protocol, VipStatus};
use crate::{
format,
stats::{Series, Unit},
table,
};
const ACTIVE: egui::Color32 = egui::Color32::from_rgb(73, 190, 133);
const NEXT: egui::Color32 = egui::Color32::from_rgb(78, 157, 237);
const WARN: egui::Color32 = egui::Color32::from_rgb(225, 170, 73);
const BAD: egui::Color32 = egui::Color32::from_rgb(228, 112, 112);
#[derive(Default)]
struct Rates {
tx: Series,
rx: Series,
total: Series,
at: Option<Instant>,
}
impl Rates {
fn sample(&mut self, at: Instant, c: &Counters) {
self.tx.sample(at, c.tx_packets, c.tx_bytes);
self.rx.sample(at, c.rx_packets, c.rx_bytes);
self.total.sample(
at,
c.tx_packets.saturating_add(c.rx_packets),
c.tx_bytes.saturating_add(c.rx_bytes),
);
self.at = Some(at);
}
fn active(&self, now: Instant) -> bool {
self.at
.is_some_and(|at| now.saturating_duration_since(at) <= Duration::from_secs(5))
&& self.total.rate(Unit::Packets) > 0.0
}
}
#[derive(Clone, PartialEq, Eq, PartialOrd, Ord, Hash)]
struct Key(String, String, u16, String, u16);
fn key(v: &VipStatus, l: &ListenerStatus, b: &Backend) -> Key {
// Listener identity separates counter histories when a port is recreated.
let listener = v
.config
.listeners
.iter()
.find(|entry| entry.protocol == l.protocol && entry.port == l.port)
.map(|entry| entry.id.clone())
.unwrap_or_else(|| l.protocol.to_string());
Key(
v.config.id.clone(),
listener,
l.port,
b.peer.clone(),
b.port,
)
}
struct Camera {
rect: egui::Rect,
members: Vec<Backend>,
}
impl Default for Camera {
fn default() -> Self {
Self {
rect: egui::Rect::NOTHING,
members: Vec::new(),
}
}
}
pub(crate) struct View {
totals: BTreeMap<String, Rates>,
backends: BTreeMap<Key, Rates>,
cameras: BTreeMap<(String, String, u16), Camera>,
selected: Option<Key>,
unit: Unit,
}
impl Default for View {
fn default() -> Self {
Self {
totals: BTreeMap::new(),
backends: BTreeMap::new(),
cameras: BTreeMap::new(),
selected: None,
unit: Unit::Bytes,
}
}
}
impl View {
pub(crate) fn observe(&mut self, network: &NetworkReport, at: Instant) {
self.totals
.retain(|id, _| network.vips.iter().any(|v| &v.config.id == id));
let mut present = std::collections::BTreeSet::new();
for v in &network.vips {
let mut counters = Counters::default();
for l in &v.listeners {
counters.tx_packets = counters.tx_packets.saturating_add(l.counters.tx_packets);
counters.rx_packets = counters.rx_packets.saturating_add(l.counters.rx_packets);
counters.tx_bytes = counters.tx_bytes.saturating_add(l.counters.tx_bytes);
counters.rx_bytes = counters.rx_bytes.saturating_add(l.counters.rx_bytes);
for b in &l.backends {
let key = key(v, l, &b.backend);
self.backends
.entry(key.clone())
.or_default()
.sample(at, &b.counters);
present.insert(key);
}
}
self.totals
.entry(v.config.id.clone())
.or_default()
.sample(at, &counters);
}
self.backends.retain(|key, _| present.contains(key));
if self
.selected
.as_ref()
.is_some_and(|key| !present.contains(key))
{
self.selected = None;
}
self.cameras.retain(|(id, protocol, port), _| {
network.vips.iter().any(|v| {
&v.config.id == id
&& v.listeners
.iter()
.any(|l| l.protocol.to_string() == *protocol && l.port == *port)
})
});
}
}
fn name<'a>(n: &'a NetworkReport, id: &'a str) -> &'a str {
n.members
.iter()
.find(|m| m.endpoint_id == id)
.and_then(|m| m.hostname.as_deref())
.unwrap_or(id)
}
fn backend_name(n: &NetworkReport, b: &Backend) -> String {
format!("{}:{}", name(n, &b.peer), b.port)
}
fn rtt(b: &BackendStatus) -> String {
b.rtt_us
.map_or_else(|| "—".into(), |us| format!("{:.2} ms", us as f64 / 1000.0))
}
fn health_color(state: &str) -> egui::Color32 {
match state {
"ready" => ACTIVE,
"checking port" | "measuring RTT" | "draining" => WARN,
_ => BAD,
}
}
pub(crate) fn show(ui: &mut egui::Ui, v: &VipStatus, n: &NetworkReport, view: &mut View) {
ui.horizontal_wrapped(|ui| {
ui.heading(format!("{}.{}", v.config.name, n.name));
ui.weak("Traffic seen by this device");
ui.selectable_value(&mut view.unit, Unit::Bytes, "bytes");
ui.selectable_value(&mut view.unit, Unit::Packets, "pkts");
});
if let Some(error) = &v.error {
ui.colored_label(BAD, error);
}
let total = view.totals.get(&v.config.id);
ui.horizontal_wrapped(|ui| {
metric(
ui,
"TX",
&format::series_rate(total.map(|r| &r.tx), view.unit),
);
metric(
ui,
"RX",
&format::series_rate(total.map(|r| &r.rx), view.unit),
);
let flows: u64 = v
.listeners
.iter()
.flat_map(|l| &l.backends)
.map(|b| u64::from(b.active_flows))
.sum();
metric(ui, "Open flows", &flows.to_string());
});
format::sparkline(ui, total.map(|r| &r.total), view.unit, 64.0);
ui.add_space(8.0);
if v.listeners.is_empty() {
ui.weak("No listeners configured. Add one in Configuration.");
}
for l in &v.listeners {
ui.push_id(
(
"vip live listener",
&v.config.id,
l.protocol.to_string(),
l.port,
),
|ui| {
egui::Frame::group(ui.style()).show(ui, |ui| {
ui.set_min_width((ui.available_width() - 2.0).max(0.0));
ui.horizontal_wrapped(|ui| {
ui.heading(format!("{}/{}", l.protocol, l.port));
ui.weak("New connections:");
if let Some(b) = &l.selected {
ui.colored_label(NEXT, backend_name(n, b));
} else {
ui.colored_label(WARN, "No eligible backend");
}
});
ui.horizontal_wrapped(|ui| {
metric(ui, "TX", &format::bytes(l.counters.tx_bytes as f64));
metric(ui, "RX", &format::bytes(l.counters.rx_bytes as f64));
metric(
ui,
"Packets TX / RX",
&format!("{} / {}", l.counters.tx_packets, l.counters.rx_packets),
);
metric(ui, "Drops", &l.counters.dropped.to_string());
if l.protocol == Protocol::Udp {
metric(ui, "Failovers", &l.counters.failovers.to_string());
}
});
ui.separator();
diagram(ui, v, l, n, view);
ui.add_space(6.0);
egui::ScrollArea::horizontal()
.id_salt("backend table scroll")
.auto_shrink([false, true])
.show(ui, |ui| {
backend_table(ui, v, l, n, view);
});
if let Some(b) = l
.backends
.iter()
.find(|b| view.selected.as_ref() == Some(&key(v, l, &b.backend)))
{
egui::Frame::group(ui.style()).show(ui, |ui| {
ui.horizontal_wrapped(|ui| {
ui.strong(backend_name(n, &b.backend));
ui.colored_label(health_color(&b.state), &b.state);
ui.weak(format!("{} open flows", b.active_flows));
if let Some(age) = b.measurement_age_ms {
ui.weak(format!(
"RTT measured {:.1}s ago",
age as f64 / 1000.0
));
} else {
ui.weak("No RTT measurement yet");
}
});
ui.horizontal_wrapped(|ui| {
ui.weak("Node");
format::copy_field(
ui,
&format::short(&b.backend.peer),
&b.backend.peer,
);
if let Some(address) = n
.members
.iter()
.find(|m| m.endpoint_id == b.backend.peer)
.and_then(|m| m.overlay_address_v4.as_deref())
{
ui.weak("Address");
format::copy_field(ui, address, address);
}
});
});
}
});
},
);
ui.add_space(8.0);
}
ui.collapsing("Internal address and identity", |ui| {
egui::Grid::new(("vip identity", &v.config.id)).show(ui, |ui| {
ui.weak("Address");
if let Some(address) = v.address {
format::copy_field(ui, &address.to_string(), &address.to_string());
} else {
ui.weak("Not allocated");
}
ui.end_row();
ui.weak("ID");
format::copy_field(ui, &v.config.id, &v.config.id);
ui.end_row();
});
});
}
fn metric(ui: &mut egui::Ui, title: &str, value: &str) {
ui.horizontal(|ui| {
ui.weak(title);
ui.strong(value);
});
ui.add_space(8.0);
}
fn backend_table(
ui: &mut egui::Ui,
v: &VipStatus,
l: &ListenerStatus,
n: &NetworkReport,
view: &mut View,
) {
let headers = [
"Backend",
"Health",
"Route",
"RTT",
"Flows",
"TX / RX",
"TX / RX rate",
"Traffic",
];
let mut widths = [170.0_f32, 110.0, 100.0, 90.0, 50.0, 160.0, 160.0, 160.0];
// Measure every value before layout, including long identities and counters.
let values: Vec<_> = l
.backends
.iter()
.map(|b| {
let rates = view.backends.get(&key(v, l, &b.backend));
let route = if rates.is_some_and(|r| r.active(Instant::now())) {
"Traffic"
} else if l.selected.as_ref() == Some(&b.backend) {
"Next flow"
} else {
"—"
};
[
backend_name(n, &b.backend),
b.state.clone(),
route.into(),
rtt(b),
b.active_flows.to_string(),
format!(
"{} / {}",
format::bytes(b.counters.tx_bytes as f64),
format::bytes(b.counters.rx_bytes as f64)
),
format!(
"{} / {}",
format::series_rate(rates.map(|r| &r.tx), view.unit),
format::series_rate(rates.map(|r| &r.rx), view.unit)
),
]
})
.collect();
for (column, width) in widths.iter_mut().enumerate().take(7) {
let font = egui::FontSelection::Default.resolve(ui.style());
for text in
std::iter::once(headers[column]).chain(values.iter().map(|row| row[column].as_str()))
{
*width = width.max(
ui.fonts(|f| {
f.layout_no_wrap(text.to_owned(), font.clone(), egui::Color32::WHITE)
.size()
.x
}) + 12.0,
);
}
}
let widths_id = ui.id().with("backend column widths");
if let Some(previous) = ui.memory(|memory| memory.data.get_temp::<[f32; 8]>(widths_id)) {
for (width, previous) in widths.iter_mut().zip(previous) {
*width = width.max(previous);
}
}
ui.memory_mut(|memory| memory.data.insert_temp(widths_id, widths));
let fixed = widths.iter().sum::<f32>() + table::GAP * 7.0;
widths[7] += (ui.available_width() - fixed).max(0.0);
table::row(ui, "backend headers", &widths, false, |column, ui| {
ui.weak(headers[column]);
});
for (index, b) in l.backends.iter().enumerate() {
let key = key(v, l, &b.backend);
let rates = view.backends.get(&key);
table::row(
ui,
&key,
&widths,
index % 2 == 0,
|column, ui| match column {
0 => {
if ui
.selectable_label(view.selected.as_ref() == Some(&key), &values[index][0])
.clicked()
{
view.selected = Some(key.clone());
}
}
1 => {
ui.colored_label(health_color(&b.state), &values[index][1])
.on_hover_text(&b.state);
}
2 => {
ui.colored_label(
if values[index][2] == "Traffic" {
ACTIVE
} else {
NEXT
},
&values[index][2],
)
.on_hover_text(
if l.selected.as_ref() == Some(&b.backend) {
"Selected for new connections. Existing flows can use other backends."
} else {
"Traffic during the latest sample interval."
},
);
}
3..=6 => {
ui.label(&values[index][column]);
}
_ => format::sparkline(ui, rates.map(|r| &r.total), view.unit, 24.0),
},
);
}
if l.backends.is_empty() {
ui.weak("No backends. Add a node and target port in Configuration.");
}
}
#[derive(Clone, Copy)]
struct Branch {
active: bool,
next: bool,
}
fn branch(l: &ListenerStatus, b: &BackendStatus, rates: Option<&Rates>, now: Instant) -> Branch {
Branch {
active: rates.is_some_and(|r| r.active(now)),
next: l.selected.as_ref() == Some(&b.backend),
}
}
fn diagram(
ui: &mut egui::Ui,
v: &VipStatus,
l: &ListenerStatus,
n: &NetworkReport,
view: &mut View,
) {
let camera = view
.cameras
.entry((v.config.id.clone(), l.protocol.to_string(), l.port))
.or_default();
let members: Vec<_> = l.backends.iter().map(|b| b.backend.clone()).collect();
let height = (members.len().max(1) as f32 * 92.0 + 32.0).max(180.0);
let bounds = egui::Rect::from_min_size(egui::Pos2::ZERO, egui::vec2(860.0, height));
if camera.members != members || !camera.rect.is_positive() {
camera.rect = bounds;
camera.members = members;
}
ui.horizontal_wrapped(|ui| {
legend(ui, "Traffic", ACTIVE, true);
legend(ui, "Next new flow", NEXT, false);
ui.weak("Click a backend · drag to pan · Ctrl+scroll to zoom");
if ui.small_button("Fit").clicked() {
camera.rect = bounds;
}
});
let size = egui::vec2(ui.available_width(), height.clamp(180.0, 360.0));
let (outer, _) = ui.allocate_exact_size(size, egui::Sense::hover());
ui.painter()
.rect_filled(outer, 6.0, ui.visuals().extreme_bg_color);
let mut child = ui.new_child(egui::UiBuilder::new().id_salt("vip graph").max_rect(outer));
egui::Scene::new()
.zoom_range(0.05..=1.5)
.max_inner_size(bounds.size())
.show(&mut child, &mut camera.rect, |ui| {
ui.set_min_size(bounds.size());
let center = height / 2.0;
let source =
egui::Rect::from_min_size(egui::pos2(18.0, center - 32.0), egui::vec2(204.0, 64.0));
let listener = egui::Rect::from_min_size(
egui::pos2(290.0, center - 32.0),
egui::vec2(146.0, 64.0),
);
let active = l.backends.iter().any(|b| {
view.backends
.get(&key(v, l, &b.backend))
.is_some_and(|r| r.active(Instant::now()))
});
edge(
ui,
source.right_center(),
listener.left_center(),
if active { ACTIVE } else { NEXT },
active,
);
node(
ui,
source,
&format!("{}.{}", v.config.name, n.name),
"This device",
NEXT,
false,
);
node(
ui,
listener,
&format!("{}/{}", l.protocol, l.port),
"Listener",
NEXT,
false,
);
for (index, b) in l.backends.iter().enumerate() {
let key = key(v, l, &b.backend);
let rates = view.backends.get(&key);
let state = branch(l, b, rates, Instant::now());
let rect = egui::Rect::from_min_size(
egui::pos2(558.0, 24.0 + index as f32 * 92.0),
egui::vec2(278.0, 68.0),
);
let color = if state.active {
ACTIVE
} else if state.next {
NEXT
} else if b.state == "ready" {
ui.visuals().weak_text_color()
} else {
health_color(&b.state)
};
edge(
ui,
listener.right_center(),
rect.left_center(),
color,
state.active,
);
if state.next {
ui.painter().circle_stroke(
rect.left_center(),
5.0,
egui::Stroke::new(2.0_f32, NEXT),
);
}
let response = ui
.interact(rect, ui.id().with((&key.3, key.4)), egui::Sense::click())
.on_hover_cursor(egui::CursorIcon::PointingHand)
.on_hover_text(format!(
"{}\n{} · RTT {}\n{} open flows\n{}{}",
backend_name(n, &b.backend),
b.state,
rtt(b),
b.active_flows,
if state.active {
"Traffic in latest sample. "
} else {
""
},
if state.next {
"Selected for new connections."
} else {
""
}
));
if response.clicked() {
view.selected = Some(key.clone());
}
let detail = format!("{} · {} · {} flows", b.state, rtt(b), b.active_flows);
node(
ui,
rect,
&backend_name(n, &b.backend),
&detail,
color,
view.selected.as_ref() == Some(&key) || response.hovered(),
);
}
if l.backends.is_empty() {
ui.painter().text(
egui::pos2(570.0, center),
egui::Align2::LEFT_CENTER,
"No backends",
egui::FontId::proportional(15.0),
ui.visuals().weak_text_color(),
);
}
});
}
fn legend(ui: &mut egui::Ui, text: &str, color: egui::Color32, filled: bool) {
ui.horizontal(|ui| {
let (rect, _) = ui.allocate_exact_size(egui::vec2(12.0, 12.0), egui::Sense::hover());
if filled {
ui.painter().circle_filled(rect.center(), 4.0, color);
} else {
ui.painter()
.circle_stroke(rect.center(), 4.0, egui::Stroke::new(1.5_f32, color));
}
ui.colored_label(color, text);
});
}
fn node(
ui: &egui::Ui,
rect: egui::Rect,
title: &str,
detail: &str,
color: egui::Color32,
selected: bool,
) {
let painter = ui.painter_at(rect);
painter.rect_filled(rect, 8.0, ui.visuals().widgets.inactive.bg_fill);
painter.rect_stroke(
rect.shrink(1.0),
8.0,
egui::Stroke::new(if selected { 3.0_f32 } else { 1.5_f32 }, color),
egui::StrokeKind::Inside,
);
text_line(
&painter,
rect.min + egui::vec2(12.0, 9.0),
title,
16.0,
ui.visuals().strong_text_color(),
rect.width() - 24.0,
);
text_line(
&painter,
rect.min + egui::vec2(12.0, 34.0),
detail,
12.0,
ui.visuals().text_color(),
rect.width() - 24.0,
);
}
fn text_line(
painter: &egui::Painter,
position: egui::Pos2,
text: &str,
size: f32,
color: egui::Color32,
width: f32,
) {
let mut job = egui::text::LayoutJob::simple_singleline(
text.into(),
egui::FontId::proportional(size),
color,
);
job.wrap.max_width = width;
job.wrap.max_rows = 1;
job.wrap.break_anywhere = true;
painter.galley(position, painter.layout_job(job), color);
}
fn edge(ui: &egui::Ui, start: egui::Pos2, end: egui::Pos2, color: egui::Color32, active: bool) {
let bend = (end.x - start.x) * 0.5;
let curve = egui::epaint::CubicBezierShape::from_points_stroke(
[
start,
start + egui::vec2(bend, 0.0),
end - egui::vec2(bend, 0.0),
end,
],
false,
egui::Color32::TRANSPARENT,
egui::Stroke::new(if active { 3.0_f32 } else { 1.5_f32 }, color),
);
ui.painter().add(curve.clone());
if active {
let t = ui.input(|i| i.time) as f32 * 0.5 % 1.0;
let position = curve.sample(t);
ui.painter().circle_filled(position, 4.0, color);
ui.ctx().request_repaint_after(Duration::from_millis(40));
}
ui.painter().arrow(
end - egui::vec2(12.0, 0.0),
egui::vec2(10.0, 0.0),
egui::Stroke::new(1.5_f32, color),
);
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
use tsunagi::vip::{Listener, VipConfig};
fn fixture() -> (NetworkReport, VipStatus) {
let mut config = VipConfig::new("k8s");
config.listeners.push(Listener::new(Protocol::Tcp, 6443));
let backends: Vec<_> = ["ai", "master", "music"]
.into_iter()
.map(|peer| BackendStatus {
backend: Backend {
peer: peer.into(),
port: 7443,
},
state: "ready".into(),
rtt_us: Some(4500),
measurement_age_ms: Some(500),
active_flows: 5,
counters: Counters::default(),
})
.collect();
let v = VipStatus {
config,
revision: "r1".into(),
address: None,
error: None,
listeners: vec![ListenerStatus {
protocol: Protocol::Tcp,
port: 6443,
selected: Some(backends[0].backend.clone()),
backends,
counters: Counters::default(),
}],
};
let n = NetworkReport {
name: "test".into(),
vips: vec![v.clone()],
..Default::default()
};
(n, v)
}
#[test]
fn traffic_branches_are_sampled_independently_of_new_flow_selection_and_open_flows() {
let (mut n, _) = fixture();
let now = Instant::now();
let mut view = View::default();
view.observe(&n, now);
n.vips[0].listeners[0].backends[2].counters.tx_packets = 20;
n.vips[0].listeners[0].backends[2].counters.tx_bytes = 4000;
n.vips[0].listeners[0].counters.tx_packets = 20;
n.vips[0].listeners[0].counters.tx_bytes = 4000;
view.observe(&n, now + Duration::from_secs(2));
let v = &n.vips[0];
let l = &v.listeners[0];
let selected = branch(
l,
&l.backends[0],
view.backends.get(&key(v, l, &l.backends[0].backend)),
now + Duration::from_secs(2),
);
let carrying = branch(
l,
&l.backends[2],
view.backends.get(&key(v, l, &l.backends[2].backend)),
now + Duration::from_secs(2),
);
assert!(selected.next && !selected.active);
assert!(carrying.active && !carrying.next);
assert_eq!(view.totals[&v.config.id].total.rate(Unit::Bytes), 2000.0);
assert!(
!view.backends[&key(v, l, &l.backends[2].backend)].active(now + Duration::from_secs(8))
);
view.observe(&n, now + Duration::from_secs(4));
assert!(
!view.backends[&key(v, l, &l.backends[2].backend)].active(now + Duration::from_secs(4))
);
}
#[test]
fn removed_services_and_backends_release_history_selection_and_cameras() {
let (mut n, v) = fixture();
let mut view = View::default();
view.observe(&n, Instant::now());
view.selected = Some(key(
&v,
&v.listeners[0],
&v.listeners[0].backends[0].backend,
));
n.vips.clear();
view.observe(&n, Instant::now());
assert!(
view.backends.is_empty()
&& view.totals.is_empty()
&& view.selected.is_none()
&& view.cameras.is_empty()
);
}
#[test]
fn clicking_a_scene_node_selects_its_backend_without_changing_configuration() {
let (n, v) = fixture();
let ctx = egui::Context::default();
let mut view = View::default();
let frame = |view: &mut View, events| {
ctx.run(
egui::RawInput {
screen_rect: Some(egui::Rect::from_min_size(
egui::Pos2::ZERO,
egui::vec2(1000.0, 600.0),
)),
events,
..Default::default()
},
|ctx| {
egui::CentralPanel::default()
.show(ctx, |ui| diagram(ui, &v, &v.listeners[0], &n, view));
},
)
};
frame(&mut view, vec![]);
let output = frame(&mut view, vec![]);
let pos = output
.shapes
.iter()
.find_map(|s| match &s.shape {
egui::Shape::Text(t) if t.galley.text() == "music:7443" => {
Some(t.pos + egui::vec2(20.0, 10.0))
}
_ => None,
})
.expect("scene draws the backend node");
for pressed in [true, false] {
frame(
&mut view,
vec![
egui::Event::PointerMoved(pos),
egui::Event::PointerButton {
pos,
button: egui::PointerButton::Primary,
pressed,
modifiers: egui::Modifiers::NONE,
},
],
);
}
assert!(
view.selected
== Some(key(
&v,
&v.listeners[0],
&v.listeners[0].backends[2].backend
))
);
assert_eq!(n.vips[0].config, v.config);
}
#[test]
fn reset_counters_and_recreated_listeners_do_not_inherit_active_traffic() {
let (mut n, _) = fixture();
let mut view = View::default();
let at = Instant::now();
n.vips[0].listeners[0].backends[0].counters.tx_packets = 100;
view.observe(&n, at);
n.vips[0].listeners[0].backends[0].counters.tx_packets = 0;
view.observe(&n, at + Duration::from_secs(2));
let v = &n.vips[0];
let l = &v.listeners[0];
assert!(
!view.backends[&key(v, l, &l.backends[0].backend)].active(at + Duration::from_secs(2))
);
n.vips[0].config.listeners[0].id = "new listener".into();
n.vips[0].listeners[0].backends[0].counters.tx_packets = 200;
view.observe(&n, at + Duration::from_secs(4));
let v = &n.vips[0];
let l = &v.listeners[0];
assert!(
!view.backends[&key(v, l, &l.backends[0].backend)].active(at + Duration::from_secs(4))
);
assert_eq!(view.backends.len(), 3);
}
#[test]
fn live_tables_and_scenes_render_at_narrow_and_wide_sizes() {
let (n, v) = fixture();
let ctx = egui::Context::default();
let mut view = View::default();
for width in [650.0, 1000.0, 1800.0, 650.0] {
for _ in 0..2 {
let _ = ctx.run(
egui::RawInput {
screen_rect: Some(egui::Rect::from_min_size(
egui::Pos2::ZERO,
egui::vec2(width, 1000.0),
)),
..Default::default()
},
|ctx| {
egui::CentralPanel::default().show(ctx, |ui| {
egui::ScrollArea::vertical().show(ui, |ui| show(ui, &v, &n, &mut view));
assert!(
ui.min_rect().right() <= width + 1.0,
"VIP content expands the window at {width}"
);
});
},
);
}
}
}
}
+185
View File
@@ -1770,3 +1770,188 @@ async fn lan_broadcast_fanout_is_scoped_opt_in_and_does_not_reflood() {
agent.shutdown().await;
}
}
fn vip_segment(
source: Ipv4Addr,
destination: Ipv4Addr,
sport: u16,
dport: u16,
udp: bool,
flags: u8,
) -> Bytes {
fn sum(bytes: &[u8]) -> u16 {
let mut s: u32 = bytes
.chunks(2)
.map(|b| u32::from(u16::from_be_bytes([b[0], *b.get(1).unwrap_or(&0)])))
.sum();
while s >> 16 != 0 {
s = (s & 0xffff) + (s >> 16);
}
!(s as u16)
}
let mut p = vec![0; if udp { 36 } else { 40 }];
p[0] = 0x45;
let len = p.len() as u16;
p[2..4].copy_from_slice(&len.to_be_bytes());
p[8] = 64;
p[9] = if udp { 17 } else { 6 };
p[12..16].copy_from_slice(&source.octets());
p[16..20].copy_from_slice(&destination.octets());
p[20..22].copy_from_slice(&sport.to_be_bytes());
p[22..24].copy_from_slice(&dport.to_be_bytes());
if udp {
p[24..26].copy_from_slice(&16u16.to_be_bytes());
} else {
p[32] = 0x50;
p[33] = flags;
p[24..28].copy_from_slice(&123u32.to_be_bytes());
}
let mut pseudo = p[12..20].to_vec();
pseudo.extend([0, p[9]]);
pseudo.extend(((p.len() - 20) as u16).to_be_bytes());
pseudo.extend(&p[20..]);
let c = sum(&pseudo);
let offset = if udp { 26 } else { 36 };
p[offset..offset + 2].copy_from_slice(&c.to_be_bytes());
let c = sum(&p[..20]);
p[10..12].copy_from_slice(&c.to_be_bytes());
Bytes::from(p)
}
#[tokio::test]
async fn vip_tcp_and_udp_remap_ports_over_real_encryption_and_preserve_source_admission() {
use tsunagi::vip::{Backend, Listener, Protocol, VipConfig};
let discovery = SharedMemoryDiscovery::new();
let (name, secret) = network("vip-encrypted");
let a = WgAgent::spawn(&discovery, "vipa").await;
let b = WgAgent::spawn(&discovery, "vipb").await;
let net = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
a.wait_for_tunnels(net, 1).await;
b.wait_for_tunnels(net, 1).await;
let aa = a.overlay(net).await;
let ab = b.overlay(net).await;
let ta = a.tun(net).await;
let tb = b.tun(net).await;
let mut cfg = VipConfig::new("service");
for (protocol, port, backend) in [(Protocol::Tcp, 443, 6443), (Protocol::Udp, 53, 5353)] {
let mut l = Listener::new(protocol, port);
l.port_check = false;
l.backends.push(Backend {
peer: b.endpoint_id().to_string(),
port: backend,
});
cfg.listeners.push(l);
}
a.agent
.save_vip(net, cfg.clone(), None, false)
.await
.unwrap();
let vip = wait_until("data-plane probes measure the backend RTT", || async {
let status = a.agent.network_status(net).await.ok()?;
let v = status.vips.iter().find(|v| v.config.id == cfg.id)?;
(v.listeners.iter().all(|l| l.selected.is_some()))
.then_some(v.address?)
.filter(|_| v.listeners.len() == 2)
})
.await;
for (udp, port, target) in [(false, 443, 6443), (true, 53, 5353)] {
ta.push_from_os(vip_segment(aa, vip, 40000, port, udp, 2));
let received = tokio::time::timeout(Duration::from_secs(5), tb.pop_to_os())
.await
.unwrap()
.unwrap();
assert_eq!(&received[12..16], &aa.octets());
assert_eq!(&received[16..20], &ab.octets());
assert_eq!(u16::from_be_bytes([received[22], received[23]]), target);
let translated = u16::from_be_bytes([received[20], received[21]]);
tb.push_from_os(vip_segment(ab, aa, target, translated, udp, 0x12));
let reply = tokio::time::timeout(Duration::from_secs(5), ta.pop_to_os())
.await
.unwrap()
.unwrap();
assert_eq!(&reply[12..16], &vip.octets());
assert_eq!(u16::from_be_bytes([reply[20], reply[21]]), port);
assert_eq!(u16::from_be_bytes([reply[22], reply[23]]), 40000);
// A backend cannot send a VIP source over its authenticated tunnel.
let forged = vip_segment(vip, aa, port, 40000, udp, 0x12);
tb.push_from_os(forged);
}
wait_until("forged VIP sources are rejected before NAT", || async {
(a.agent.overlay()?.counters.wrong_source >= 2).then_some(())
})
.await;
a.shutdown().await;
b.shutdown().await;
}
#[tokio::test]
async fn vip_udp_reaches_another_backend_after_the_selected_agent_stops() {
use tsunagi::vip::{Backend, Listener, Protocol, VipConfig};
let discovery = SharedMemoryDiscovery::new();
let (name, secret) = network("vip-failover");
let a = WgAgent::spawn(&discovery, "vipfa").await;
let b = WgAgent::spawn(&discovery, "vipfb").await;
let c = WgAgent::spawn(&discovery, "vipfc").await;
let net = a.agent.join_network(&name, &secret).await.unwrap();
for node in [&b, &c] {
node.agent.join_network(&name, &secret).await.unwrap();
}
for node in [&a, &b, &c] {
node.wait_for_tunnels(net, 2).await;
}
let mut config = VipConfig::new("datagrams");
let mut listener = Listener::new(Protocol::Udp, 8888);
listener.backends = vec![
Backend {
peer: b.endpoint_id().to_string(),
port: 9000,
},
Backend {
peer: c.endpoint_id().to_string(),
port: 9001,
},
];
config.listeners.push(listener);
a.agent
.save_vip(net, config.clone(), None, false)
.await
.unwrap();
let first = wait_until("both UDP backends have measured paths", || async {
let s = a.agent.network_status(net).await.ok()?;
let v = s.vips.iter().find(|v| v.config.id == config.id)?;
let l = v.listeners.first()?;
(l.backends.iter().all(|b| b.rtt_us.is_some())).then_some((v.address?, l.selected.clone()?))
})
.await;
let (dead, alive) = if first.1.peer == b.endpoint_id().to_string() {
(b, c)
} else {
(c, b)
};
let address = a.overlay(net).await;
let tun = a.tun(net).await;
let dest = dead.tun(net).await;
tun.push_from_os(vip_segment(address, first.0, 41000, 8888, true, 0));
tokio::time::timeout(Duration::from_secs(5), dest.pop_to_os())
.await
.unwrap()
.unwrap();
dead.shutdown().await;
wait_until("the remaining backend becomes the next choice", || async {
let s = a.agent.network_status(net).await.ok()?;
let v = s.vips.iter().find(|v| v.config.id == config.id)?;
(v.listeners.first()?.selected.as_ref()?.peer == alive.endpoint_id().to_string())
.then_some(())
})
.await;
tun.push_from_os(vip_segment(address, first.0, 41000, 8888, true, 0));
let packet = tokio::time::timeout(Duration::from_secs(5), alive.tun(net).await.pop_to_os())
.await
.unwrap()
.unwrap();
assert_eq!(&packet[16..20], &alive.overlay(net).await.octets());
let view = a.agent.network_status(net).await.unwrap();
assert_eq!(view.vips[0].listeners[0].counters.failovers, 1);
a.shutdown().await;
alive.shutdown().await;
}
+2
View File
@@ -27,6 +27,8 @@ clap = { version = "4.5", features = ["derive", "env"] }
# colour is never written where it would show up as rubbish.
anstream = "1.0"
anstyle = "1.0"
serde_json = "1.0"
tempfile.workspace = true
# Local interface addresses, for the diagnostics in `status`.
netwatch = "0.19.3"
+303 -62
View File
@@ -14,6 +14,7 @@ use std::time::Duration;
#[cfg(windows)]
mod winservice;
mod vip_cli;
use clap::{Args, Parser, Subcommand, ValueEnum};
use tsunagi::agent::Event;
use tsunagi::config::{AgentConfig, StoragePaths, TransportPolicy};
@@ -52,6 +53,8 @@ struct Cli {
#[derive(Debug, Subcommand)]
enum Command {
/// Manage named shared TCP/UDP virtual services.
Vip(vip_cli::VipArgs),
/// Shows this device's identity, and changes it.
///
/// Every item follows the same shape: name it to see it, name it with a
@@ -69,8 +72,8 @@ enum Command {
Up(Box<UpArgs>),
/// Reports this device, what the agent is doing, and what this host can do.
Status(StatusArgs),
/// Shows the protocols this build can carry packets with.
Protocols,
/// Lists protocols, or shows and changes one network's selection at runtime.
Protocols(ProtocolsArgs),
/// Makes a network, or joins one — the command that does the work.
///
/// The same as `tsng network join`, at the top level because it is
@@ -107,6 +110,23 @@ struct DnsArgs {
action: Option<DnsAction>,
}
#[derive(Debug, Args)]
#[command(
after_help = "Examples:\n tsng protocols\n tsng protocols home\n tsng protocols home awg,wg-quic\n tsng protocols home none\n\nThe selection is local to this device, saved per network and applied immediately.\nBoth endpoints need a common protocol at the same wire version."
)]
struct ProtocolsArgs {
#[command(flatten)]
paths: PathArgs,
/// Control socket to talk to; uses the running service by default.
#[arg(long, env = "TSUNAGI_CONTROL_SOCKET")]
control_socket: Option<PathBuf>,
/// Network's exact name, id or unique id prefix.
network: Option<String>,
/// Replace the selection, best first: awg,wg-quic; none disables data traffic.
#[arg(value_name = "LIST", requires = "network")]
protocols: Option<String>,
}
#[derive(Debug, Subcommand)]
enum DnsAction {
/// Starts serving, now and after every restart.
@@ -187,6 +207,13 @@ struct JoinArgs {
#[derive(Debug, Subcommand)]
enum NetworkAction {
/// Forgets an offline node's observed claim on every replica; newer claims may return.
ForgetNode {
/// Exact network name, full id or unique id prefix.
network: String,
/// Exact node hostname, full endpoint id, unique id prefix or overlay IPv4.
node: String,
},
/// Shows or changes this network's local LAN broadcast participation.
Broadcast {
/// Exact network name, full id or unique id prefix.
@@ -465,11 +492,13 @@ struct UpArgs {
#[arg(long, help_heading = "System")]
no_tun: bool,
/// Protocols to carry packets with, best first.
/// Initial protocol selection for networks without a saved choice, best first.
///
/// A pair of peers uses one they both have at the same wire version. A
/// peer with none in common keeps its control plane and gets no data
/// plane. `none` runs the control plane alone.
/// plane. `none` runs the control plane alone. `tsng protocols NETWORK LIST`
/// changes and saves each network's choice while this agent runs; any
/// built-in protocol can be enabled, even if absent from this initial list.
#[arg(
long = "protocol",
value_name = "LIST",
@@ -897,10 +926,11 @@ async fn run(command: Command) -> Result<(), Box<dyn std::error::Error>> {
Command::Id(args) => id(args).await,
Command::Up(args) => up(*args).await,
Command::Status(args) => status(args).await,
Command::Protocols => show_protocols(),
Command::Protocols(args) => protocols_command(args).await,
Command::Join(args) => join_command(args).await,
Command::Network(args) => network_command(args).await,
Command::Dns(args) => dns_command(args).await,
Command::Vip(args) => vip_cli::run(args).await,
Command::ExitNode(args) => exit_node_command(args).await,
Command::Wipe(args) => wipe(args).await,
// Taken in `main`, before there is a runtime: the service manager
@@ -1291,7 +1321,12 @@ fn spawn_dns(agent: Agent, port: u16) -> DnsService {
let members = network.members.iter().filter_map(|member| {
Some((member.hostname.clone()?, member.overlay_address_v4?))
});
zones.push(Zone::new(zone, members));
let services = network
.vips
.iter()
.filter(|v| v.error.is_none())
.filter_map(|v| Some((v.config.name.clone(), v.address?)));
zones.push(Zone::new(zone, members.chain(services)));
}
let zones = Zones::new(zones);
let names = zones.names() as u32;
@@ -1454,6 +1489,7 @@ const PROTOCOLS: &[ProtocolSpec] = &[
];
/// One `-o` setting, and the protocol it was aimed at.
#[derive(Clone)]
struct Setting {
/// `Some` when written as `protocol:key=value`.
protocol: Option<String>,
@@ -1550,6 +1586,48 @@ struct AgentControl {
}
impl tsunagi::ipc::ReportSource for AgentControl {
fn save_vip(
&self,
network: String,
config: tsunagi::vip::VipConfig,
expected: Option<String>,
delete: bool,
) -> tsunagi::BoxFuture<'_, Result<Vec<tsunagi::vip::VipStatus>, String>> {
Box::pin(async move {
let network = network
.parse()
.map_err(|e| format!("invalid network id: {e}"))?;
self.agent
.save_vip(network, config, expected, delete)
.await
.map_err(|e| e.to_string())
})
}
fn forget_node(
&self,
network: String,
peer: String,
expected_version: u64,
) -> tsunagi::BoxFuture<'_, Result<tsunagi::ipc::ForgottenNodeReport, String>> {
Box::pin(async move {
let network = network
.parse()
.map_err(|err| format!("invalid network id: {err}"))?;
let peer = peer
.parse()
.map_err(|err| format!("invalid endpoint id: {err}"))?;
let result = self
.agent
.forget_node(network, peer, expected_version)
.await
.map_err(|err| err.to_string())?;
Ok(tsunagi::ipc::ForgottenNodeReport {
peer: result.target.to_string(),
record_version: result.record_version,
peers_told: result.peers_told as u32,
})
})
}
fn report(&self) -> tsunagi::BoxFuture<'_, tsunagi::ipc::StatusReport> {
Box::pin(async move {
let dns = self.dns_state().await;
@@ -1704,6 +1782,23 @@ impl tsunagi::ipc::ReportSource for AgentControl {
})
}
fn set_protocols(
&self,
network_id: String,
protocols: Vec<String>,
) -> tsunagi::BoxFuture<'_, Result<Vec<String>, String>> {
Box::pin(async move {
let id = network_id
.parse()
.map_err(|err| format!("invalid network id: {err}"))?;
self.agent
.set_protocols(id, protocols.clone())
.await
.map_err(|err| err.to_string())?;
Ok(protocols)
})
}
fn set_exit_offer(
&self,
network_id: String,
@@ -1866,45 +1961,178 @@ fn resolve_network<'a>(
networks: &'a [tsunagi::storage::StoredNetwork],
wanted: &str,
) -> Result<&'a tsunagi::storage::StoredNetwork, String> {
let entries: Vec<_> = networks
.iter()
.map(|n| (n.network_id.to_string(), n.name.as_str().to_owned()))
.collect();
resolve_network_index(&entries, wanted).map(|i| &networks[i])
}
/// Shared selector rules for stored networks and secret-free live reports.
fn resolve_network_index(networks: &[(String, String)], wanted: &str) -> Result<usize, String> {
let wanted = wanted.trim_matches(|c: char| c.is_ascii_whitespace());
if wanted.is_empty() {
return Err("select a network by name or id; `tsng network` lists them".to_string());
return Err("select a network by name or id; `tsng network` lists them".into());
}
if let Some(network) = networks
.iter()
.find(|network| network.network_id.to_string() == wanted)
{
return Ok(network);
if let Some(index) = networks.iter().position(|(id, _)| id == wanted) {
return Ok(index);
}
// Only id prefixes lose a copied report ellipsis; names are used verbatim.
let prefix = wanted.trim_end_matches('…');
let matched: Vec<&tsunagi::storage::StoredNetwork> = networks
let matches: Vec<_> = networks
.iter()
.filter(|network| {
network.name.as_str() == wanted
|| (!prefix.is_empty() && network.network_id.to_string().starts_with(prefix))
})
.enumerate()
.filter(|(_, (id, name))| name == wanted || (!prefix.is_empty() && id.starts_with(prefix)))
.collect();
match matched.as_slice() {
[one] => Ok(one),
match matches.as_slice() {
[(index, _)] => Ok(*index),
[] => Err(format!(
"no configured network matches name or id `{wanted}`; \
`tsng network` lists them"
"no configured network matches name or id `{wanted}`; `tsng network` lists them"
)),
several => {
let candidates = several
several => Err(format!(
"`{wanted}` matches {} networks; use the full id or a unique id prefix:\n{}",
several.len(),
several
.iter()
.map(|network| format!(" `{}` {}", network.name, network.network_id))
.map(|(_, (id, name))| format!(" `{name}` {id}"))
.collect::<Vec<_>>()
.join("\n");
Err(format!(
"`{wanted}` matches {} networks; use the full id or a unique id prefix:\n{candidates}",
several.len()
))
}
.join("\n")
)),
}
}
fn parse_protocol_list(text: &str) -> Result<Vec<String>, String> {
if text.eq_ignore_ascii_case("none") {
return Ok(Vec::new());
}
let mut names = Vec::new();
for name in text.split(',') {
let name = name.trim();
let name = if name.eq_ignore_ascii_case("wireguard") {
WG_PROTOCOL
} else {
name
};
if !PROTOCOLS.iter().any(|spec| spec.name == name) {
return Err(format!(
"unknown protocol `{name}`; use wg,awg,wg-quic,tcp-tls or `none` alone. Run `tsng protocols` for details"
));
}
if names.iter().any(|n| n == name) {
return Err(format!("protocol `{name}` is listed more than once"));
}
names.push(name.to_owned());
}
Ok(names)
}
async fn protocols_command(args: ProtocolsArgs) -> Result<(), Box<dyn std::error::Error>> {
let Some(selector) = args.network else {
return show_protocols();
};
let requested = args
.protocols
.as_deref()
.map(parse_protocol_list)
.transpose()?;
let custom = args.paths.state_dir.is_some();
let paths = args.paths.resolve()?;
let socket = control_socket(&paths, args.control_socket.as_ref(), custom);
if tsunagi::ipc::is_serving(&socket).await {
let report = tsunagi::ipc::request_status(&socket).await?;
let entries: Vec<_> = report
.networks
.iter()
.map(|n| (n.network_id.clone(), n.name.clone()))
.collect();
let network = &report.networks[resolve_network_index(&entries, &selector)?];
let protocols = match requested {
Some(protocols) => {
tsunagi::ipc::set_protocols(&socket, &network.network_id, protocols).await?
}
None => network.protocols.clone(),
};
println!(
"protocols {} for `{}` ({}) · preference order",
if protocols.is_empty() {
"none".into()
} else {
protocols.join(",")
},
network.name,
network.network_id
);
} else {
// Take the directory ownership lock even for an offline edit. Never
// write behind a live agent whose socket cannot be reached.
let storage = tsunagi::storage::Storage::open(&paths)?;
let configured = storage.list_networks().await?;
let network = resolve_network(&configured, &selector)?;
let protocols = match requested {
Some(protocols) => {
storage
.set_protocols(network.network_id, protocols.clone())
.await?;
Some(protocols)
}
None => network.protocols.clone(),
};
let shown = protocols
.map(|names| {
if names.is_empty() {
"none".into()
} else {
names.join(",")
}
})
.unwrap_or_else(|| "defaults from tsng up --protocol".into());
println!(
"protocols {shown} for `{}` ({}) · agent stopped; applies on next start",
network.name, network.network_id
);
}
Ok(())
}
async fn forget_node_command(
socket: &std::path::Path,
network: &str,
node: &str,
) -> Result<(), Box<dyn std::error::Error>> {
let report = tsunagi::ipc::request_status(socket)
.await
.map_err(|err| format!("a reachable running agent is required to forget nodes: {err}"))?;
let entries: Vec<_> = report
.networks
.iter()
.map(|n| (n.network_id.clone(), n.name.clone()))
.collect();
let network = &report.networks[resolve_network_index(&entries, network)?];
if !network.active {
return Err(format!(
"start `{}` first: tsng network start {}",
network.name, network.network_id
)
.into());
}
let member = tsunagi::ipc::resolve_node(network, node)?;
let result = tsunagi::ipc::forget_node(
socket,
&network.network_id,
&member.endpoint_id,
member.record_version,
)
.await?;
println!(
"forgot {} ({}) claim v{} in `{}`; cleanup queued to {} peers. A newer signed claim can return.",
member.hostname.as_deref().unwrap_or("unnamed"),
result.peer,
result.record_version,
network.name,
result.peers_told
);
Ok(())
}
/// `tsng network`: what this device belongs to, and leaving it.
async fn network_command(args: NetworkArgs) -> Result<(), Box<dyn std::error::Error>> {
let paths = args.paths.resolve()?;
@@ -1912,6 +2140,9 @@ async fn network_command(args: NetworkArgs) -> Result<(), Box<dyn std::error::Er
let socket = control_socket(&paths, args.control_socket.as_ref(), is_custom_state);
let paths = agent_paths(paths, &socket, is_custom_state).await;
match args.action {
Some(NetworkAction::ForgetNode { network, node }) => {
forget_node_command(&socket, &network, &node).await
}
None => show_networks(&paths, &socket).await,
Some(NetworkAction::Broadcast { network, choice }) => {
broadcast_command(&paths, &socket, &network, choice).await
@@ -4445,43 +4676,24 @@ async fn up(args: UpArgs) -> Result<(), Box<dyn std::error::Error>> {
// What the user asked for is checked first, before anything that could
// fail for a reason of its own: a misspelled protocol or setting is
// their mistake to see, not something to bury under a privilege error.
let wanted: Vec<&ProtocolSpec> = {
let mut wanted = Vec::new();
for name in args
.protocols
.iter()
.filter(|name| !name.eq_ignore_ascii_case("none"))
{
// Accept "wireguard" as a convenience alias for the canonical "wg".
let canonical = if name.eq_ignore_ascii_case("wireguard") {
WG_PROTOCOL
} else {
name.as_str()
};
let Some(spec) = PROTOCOLS.iter().find(|spec| spec.name == canonical) else {
let known: Vec<&str> = PROTOCOLS.iter().map(|spec| spec.name).collect();
return Err(format!(
"this build has no protocol called `{name}`; it has {}. \
Run `tsng protocols` to see what each one takes.",
known.join(", ")
)
.into());
};
wanted.push(spec);
}
wanted
};
let initial = parse_protocol_list(&args.protocols.join(","))?;
let wanted: Vec<&ProtocolSpec> = initial
.iter()
.filter_map(|name| PROTOCOLS.iter().find(|spec| spec.name == name))
.collect();
config.default_protocols = Some(initial);
let settings = parse_settings(&args.protocol_options)?;
// A setting nobody takes is a mistake, not a preference: one that was
// silently dropped looks exactly like one that did not work.
for setting in &settings {
if !wanted
if !PROTOCOLS
.iter()
.filter(|spec| setting.protocol.is_some() || wanted.iter().any(|p| p.name == spec.name))
.any(|spec| !settings_for(spec, std::slice::from_ref(setting)).is_empty())
{
return Err(format!(
"no selected protocol takes `{}`; run `tsng protocols` to see what they do",
"no configured protocol takes `{}`; qualify disabled protocol settings as protocol:key=value. Run `tsng protocols` for details",
setting.key
)
.into());
@@ -4497,7 +4709,7 @@ async fn up(args: UpArgs) -> Result<(), Box<dyn std::error::Error>> {
let mut tcp_tls_plugin = None;
let mut pure_wg_plugin: Option<Arc<PureWgPlugin>> = None;
let mut pure_wg_transport: Option<Arc<PureWgTransport>> = None;
if !wanted.is_empty() {
if !PROTOCOLS.is_empty() {
let tun_factory: Arc<dyn TunFactory> = if args.no_tun {
Arc::new(MemoryTunFactory::new())
} else {
@@ -4507,8 +4719,17 @@ async fn up(args: UpArgs) -> Result<(), Box<dyn std::error::Error>> {
config = config.with_interface(tun_factory, args.interface.clone(), mtu);
}
for spec in &wanted {
let options = settings_for(spec, &settings);
for spec in PROTOCOLS {
// Unqualified options keep targeting the initial selection; explicitly
// scoped options also configure protocols that will be enabled later.
let scoped: Vec<_> = settings
.iter()
.filter(|setting| {
setting.protocol.is_some() || wanted.iter().any(|p| p.name == spec.name)
})
.cloned()
.collect();
let options = settings_for(spec, &scoped);
match spec.name {
AWG_PROTOCOL => {
let mut wg = WireguardConfig::new(paths.state_dir.join("awg"));
@@ -5112,6 +5333,8 @@ async fn build_report(
};
NetworkReport {
vips: network.vips.clone(),
protocols: network.protocols.clone(),
auto_start: network.auto_start,
exit: exit_report(network),
broadcast: network.broadcast,
@@ -5138,6 +5361,7 @@ async fn build_report(
.members
.iter()
.map(|member| MemberReport {
record_version: member.record_version,
endpoint_id: member.endpoint_id.to_string(),
overlay_address_v4: member.overlay_address_v4.map(|addr| addr.to_string()),
// What this agent is currently experiencing trying to
@@ -5182,6 +5406,13 @@ async fn build_report(
.collect();
StatusReport {
available_protocols: PROTOCOLS
.iter()
.map(|p| tsunagi::ipc::ProtocolReport {
name: p.name.into(),
summary: p.summary.into(),
})
.collect(),
endpoint_id: status.endpoint_id.to_string(),
hostname: status.hostname.clone(),
bound_sockets: status
@@ -5536,6 +5767,8 @@ mod status_tests {
/// The situation that prompted this: one peer left and came back.
fn network_after_a_peer_returned() -> NetworkReport {
NetworkReport {
vips: Vec::new(),
protocols: vec!["wg-quic".into()],
auto_start: true,
exit: Default::default(),
hostname_conflict: None,
@@ -5559,6 +5792,7 @@ mod status_tests {
}],
members: vec![
MemberReport {
record_version: 1,
endpoint_id: OWN.into(),
overlay_address_v4: Some("10.13.37.69".into()),
failed_dials: 0,
@@ -5566,6 +5800,7 @@ mod status_tests {
..Default::default()
},
MemberReport {
record_version: 1,
endpoint_id: ONLINE.into(),
overlay_address_v4: Some("10.13.37.237".into()),
failed_dials: 0,
@@ -5842,6 +6077,7 @@ mod status_tests {
fn offline_members_are_listed_after_online_ones() {
let mut network = network_after_a_peer_returned();
network.members.push(MemberReport {
record_version: 1,
endpoint_id: AWAY.into(),
overlay_address_v4: Some("10.13.37.99".into()),
failed_dials: 9,
@@ -5963,6 +6199,7 @@ mod status_tests {
fn an_offline_member_keeps_the_last_name_it_announced() {
let mut network = network_after_a_peer_returned();
network.members.push(MemberReport {
record_version: 1,
endpoint_id: AWAY.into(),
overlay_address_v4: Some("10.13.37.99".into()),
failed_dials: 0,
@@ -5984,6 +6221,7 @@ mod status_tests {
// are attributed to it instead of floating free as a counter.
let mut network = network_after_a_peer_returned();
network.members.push(MemberReport {
record_version: 1,
endpoint_id: AWAY.into(),
overlay_address_v4: Some("10.13.37.99".into()),
failed_dials: 9,
@@ -6008,6 +6246,7 @@ mod status_tests {
// and said to be, rather than being shown as away or as a fault.
let mut network = network_after_a_peer_returned();
network.members.push(MemberReport {
record_version: 1,
endpoint_id: AWAY.into(),
overlay_address_v4: Some("10.13.37.99".into()),
hostname: Some("laptop".into()),
@@ -6219,6 +6458,7 @@ mod network_tests {
NetworkSecret::from_bytes(&[secret.as_bytes(), &[0u8; 32]].concat()[..32]).unwrap();
let keys = NetworkKeys::derive(&name, &secret);
StoredNetwork {
protocols: None,
exit_node: false,
exit_via: None,
broadcast: true,
@@ -6442,6 +6682,7 @@ mod network_context_tests {
let secret = NetworkSecret::from_bytes([seed; 32]).unwrap();
let keys = NetworkKeys::derive(&name, &secret);
StoredNetwork {
protocols: None,
exit_node: false,
exit_via: None,
broadcast: true,
+506
View File
@@ -0,0 +1,506 @@
//! Named virtual service administration; every mutation submits one complete draft.
use super::{PathArgs, control_socket, resolve_network_index};
use clap::{Args, Subcommand};
use std::path::PathBuf;
use tsunagi::ipc::{self, NetworkReport};
use tsunagi::vip::{Backend, Listener, Protocol, VipConfig, VipStatus};
type Result<T> = std::result::Result<T, Box<dyn std::error::Error>>;
#[derive(Debug, Args)]
#[command(
after_help = "Examples:\n tsng vip create lab k8s\n tsng vip port add lab k8s tcp/443 --backend cp1:6443 --backend cp2:7443\n tsng vip backend add lab k8s tcp/443 cp3:6443\n tsng vip status lab k8s --watch\n tsng vip edit lab k8s\n\nNames, full ids and unique id prefixes are accepted. Changes are shared with the network.\nTCP port checks default to on; UDP uses data-path probes and explicit ICMP errors."
)]
pub(crate) struct VipArgs {
#[command(flatten)]
paths: PathArgs,
/// Agent control socket; defaults to the running service.
#[arg(long, global = true, env = "TSUNAGI_CONTROL_SOCKET")]
control_socket: Option<PathBuf>,
/// Machine-readable configuration and local observations.
#[arg(long, global = true)]
json: bool,
#[command(subcommand)]
action: Action,
}
#[derive(Debug, Args)]
struct Target {
/// Network exact name, id or unique id prefix.
network: String,
/// VIP exact name, id or unique id prefix.
vip: String,
}
#[derive(Debug, Args)]
struct PortTarget {
#[command(flatten)]
target: Target,
/// Listener as tcp/443 or udp/53.
listener: String,
}
#[derive(Debug, Subcommand)]
enum Action {
/// List services in one network, or every configured network.
List { network: Option<String> },
/// Create a named VIP with an automatically reserved internal address.
Create { network: String, name: String },
/// Show the complete saved configuration and local status.
Show(Target),
/// Show local health, routing choices and cumulative traffic counters.
Status {
#[command(flatten)]
target: Target,
/// Refresh once a second until interrupted.
#[arg(long)]
watch: bool,
},
/// Edit the entire configuration in VISUAL/EDITOR and save it atomically.
Edit(Target),
/// Remove a VIP; existing flows drain until completion or idle expiry.
Remove(Target),
/// Add, configure or remove listeners independently of the VIP.
Port {
#[command(subcommand)]
action: PortAction,
},
/// Change a listener's destination nodes and application ports.
Backend {
#[command(subcommand)]
action: BackendAction,
},
}
#[derive(Debug, Subcommand)]
enum PortAction {
/// Add a listener, optionally with its complete initial backend set.
Add {
#[command(flatten)]
target: PortTarget,
/// Node name/id followed by :port. Repeat for multiple destinations.
#[arg(long)]
backend: Vec<String>,
},
/// Change check settings and flow idle expiry.
Set {
#[command(flatten)]
target: PortTarget,
/// Active TCP connect checks: on or off.
#[arg(long,value_parser=["on","off"])]
port_check: Option<String>,
/// Milliseconds between active checks (250..3600000).
#[arg(long)]
check_interval_ms: Option<u32>,
/// TCP check deadline in milliseconds (100..60000).
#[arg(long)]
check_timeout_ms: Option<u32>,
/// Consecutive failures before exclusion.
#[arg(long)]
failures: Option<u8>,
/// Consecutive successes before recovery.
#[arg(long)]
successes: Option<u8>,
/// Flow idle expiry in seconds (1..604800).
#[arg(long)]
idle_timeout: Option<u32>,
},
/// Remove a listener from new-flow selection; existing flows drain.
Remove(PortTarget),
}
#[derive(Debug, Subcommand)]
enum BackendAction {
/// Add a destination node and port to a listener.
Add {
#[command(flatten)]
target: PortTarget,
backend: String,
},
/// Remove a destination from new-flow selection.
Remove {
#[command(flatten)]
target: PortTarget,
backend: String,
},
}
impl Action {
fn network(&self) -> Option<&str> {
match self {
Self::List { network } => network.as_deref(),
Self::Create { network, .. } => Some(network),
Self::Show(t) | Self::Edit(t) | Self::Remove(t) | Self::Status { target: t, .. } => {
Some(&t.network)
}
Self::Port { action } => Some(
&match action {
PortAction::Add { target, .. }
| PortAction::Set { target, .. }
| PortAction::Remove(target) => target,
}
.target
.network,
),
Self::Backend { action } => Some(
&match action {
BackendAction::Add { target, .. } | BackendAction::Remove { target, .. } => {
target
}
}
.target
.network,
),
}
}
}
fn parse_listener(s: &str) -> Result<(Protocol, u16)> {
let (p, n) = s
.split_once('/')
.ok_or("listener must be tcp/PORT or udp/PORT")?;
let port = n.parse::<u16>()?;
if port == 0 {
return Err("port must not be zero".into());
}
Ok((p.parse()?, port))
}
fn backend(n: &NetworkReport, s: &str) -> Result<Backend> {
let (node, port) = s.rsplit_once(':').ok_or("backend must be NODE:PORT")?;
let peer = match ipc::resolve_node(n, node) {
Ok(member) => member.endpoint_id.clone(),
Err(error) => {
let matches = n
.members
.iter()
.any(|m| m.hostname.as_deref() == Some(node) || m.endpoint_id.starts_with(node));
if matches {
return Err(error.into());
}
node.parse::<iroh::EndpointId>()
.map_err(|_| error)?
.to_string()
}
};
let port = port.parse()?;
if port == 0 {
return Err("backend port must not be zero".into());
}
Ok(Backend { peer, port })
}
fn selected(n: &NetworkReport, t: &Target) -> Result<VipStatus> {
Ok(ipc::resolve_vip(n, &t.vip)?.clone())
}
fn listener<'a>(c: &'a mut VipConfig, target: &PortTarget) -> Result<&'a mut Listener> {
let (p, n) = parse_listener(&target.listener)?;
c.listeners
.iter_mut()
.find(|l| l.protocol == p && l.port == n)
.ok_or_else(|| format!("no listener {} on {}", target.listener, c.name).into())
}
fn name_of<'a>(n: &'a NetworkReport, peer: &'a str) -> &'a str {
n.members
.iter()
.find(|m| m.endpoint_id == peer)
.and_then(|m| m.hostname.as_deref())
.unwrap_or(peer)
}
fn show(n: &NetworkReport, v: &VipStatus, json: bool) -> Result<()> {
if json {
println!("{}", serde_json::to_string(v)?);
return Ok(());
}
println!(
"{}.{} {}",
v.config.name,
n.name,
v.error.as_deref().unwrap_or("configured")
);
for l in &v.listeners {
let choice = l
.selected
.as_ref()
.map(|b| format!("{}:{}", name_of(n, &b.peer), b.port))
.unwrap_or_else(|| "none".into());
println!(
" {}/{} next: {} flows: {} tx/rx: {}/{} bytes drops: {} UDP failovers: {}",
l.protocol,
l.port,
choice,
l.backends.iter().map(|b| b.active_flows).sum::<u32>(),
l.counters.tx_bytes,
l.counters.rx_bytes,
l.counters.dropped,
l.counters.failovers
);
for b in &l.backends {
let rtt = b
.rtt_us
.map(|n| format!("{:.2} ms", n as f64 / 1000.))
.unwrap_or_else(|| "unknown".into());
println!(
" {}:{} {} RTT {} active {}",
name_of(n, &b.backend.peer),
b.backend.port,
b.state,
rtt,
b.active_flows
);
}
}
Ok(())
}
pub(crate) async fn run(args: VipArgs) -> Result<()> {
let custom = args.paths.state_dir.is_some();
let paths = args.paths.resolve()?;
let socket = control_socket(&paths, args.control_socket.as_ref(), custom);
let report = ipc::request_status(&socket).await?;
if let Action::List { network: None } = &args.action {
for n in &report.networks {
for v in &n.vips {
show(n, v, args.json)?;
}
}
return Ok(());
}
let selector = args.action.network().ok_or("network is required")?;
let names: Vec<_> = report
.networks
.iter()
.map(|n| (n.network_id.clone(), n.name.clone()))
.collect();
let network = &report.networks[resolve_network_index(&names, selector)?];
if let Action::Status {
target,
watch: true,
} = &args.action
{
let mut previous: Option<(std::time::Instant, u64, u64)> = None;
loop {
let report = ipc::request_status(&socket).await?;
let n = report
.networks
.iter()
.find(|n| n.network_id == network.network_id)
.ok_or("network was removed")?;
let vip = selected(n, target)?;
let (tx, rx) = vip.listeners.iter().fold((0u64, 0u64), |a, l| {
(
a.0.saturating_add(l.counters.tx_bytes),
a.1.saturating_add(l.counters.rx_bytes),
)
});
let now = std::time::Instant::now();
if let Some((at, old_tx, old_rx)) = previous
&& !args.json
{
let secs = now.duration_since(at).as_secs_f64();
if secs > 0. {
println!(
"TX {:.1} KiB/s · RX {:.1} KiB/s",
tx.saturating_sub(old_tx) as f64 / secs / 1024.,
rx.saturating_sub(old_rx) as f64 / secs / 1024.
);
}
}
previous = Some((now, tx, rx));
show(n, &vip, args.json)?;
tokio::select! {_=tokio::signal::ctrl_c()=>break,_=tokio::time::sleep(std::time::Duration::from_secs(1))=>{}}
}
return Ok(());
}
let mut edited_file = None;
let (mut config, expected, delete) = match &args.action {
Action::List { .. } => {
for v in &network.vips {
show(network, v, args.json)?;
}
return Ok(());
}
Action::Show(t) | Action::Status { target: t, .. } => {
let v = selected(network, t)?;
show(network, &v, args.json)?;
if matches!(args.action, Action::Show(_)) && !args.json {
println!("{}", serde_json::to_string_pretty(&v.config)?);
}
return Ok(());
}
Action::Create { name, .. } => (VipConfig::new(name), None, false),
Action::Remove(t) => {
let v = selected(network, t)?;
(v.config, Some(v.revision), true)
}
Action::Edit(t) => {
let v = selected(network, t)?;
let mut file = tempfile::Builder::new()
.prefix("tsunagi-vip-")
.suffix(".json")
.tempfile()?;
use std::io::Write;
file.write_all(serde_json::to_string_pretty(&v.config)?.as_bytes())?;
file.flush()?;
let editor = std::env::var("VISUAL")
.or_else(|_| std::env::var("EDITOR"))
.unwrap_or_else(|_| "vi".into());
let mut words = editor.split_whitespace();
let program = words.next().ok_or("empty EDITOR")?;
let status = std::process::Command::new(program)
.args(words)
.arg(file.path())
.status()?;
if !status.success() {
return Err("editor failed; VIP unchanged".into());
}
let bytes = std::fs::read(file.path())?;
let cfg: VipConfig = match serde_json::from_slice(&bytes) {
Ok(c) => c,
Err(e) => {
let (_, path) = file.keep()?;
return Err(
format!("invalid draft: {e}; retained at {}", path.display()).into(),
);
}
};
if cfg.id != v.config.id {
return Err("VIP id cannot be edited".into());
}
edited_file = Some(file);
(cfg, Some(v.revision), false)
}
Action::Port { action } => {
let target = match action {
PortAction::Add { target, .. }
| PortAction::Set { target, .. }
| PortAction::Remove(target) => target,
};
let v = selected(network, &target.target)?;
let mut cfg = v.config;
match action {
PortAction::Add {
target,
backend: destinations,
} => {
let (p, port) = parse_listener(&target.listener)?;
let mut l = Listener::new(p, port);
l.backends = destinations
.iter()
.map(|b| backend(network, b))
.collect::<Result<_>>()?;
cfg.listeners.push(l);
}
PortAction::Remove(target) => {
let (p, port) = parse_listener(&target.listener)?;
listener(&mut cfg, target)?;
cfg.listeners.retain(|l| l.protocol != p || l.port != port);
}
PortAction::Set {
target,
port_check,
check_interval_ms,
check_timeout_ms,
failures,
successes,
idle_timeout,
} => {
let l = listener(&mut cfg, target)?;
if let Some(v) = port_check {
l.port_check = v == "on";
}
if let Some(v) = check_interval_ms {
l.check_interval_ms = *v;
}
if let Some(v) = check_timeout_ms {
l.check_timeout_ms = *v;
}
if let Some(v) = failures {
l.failure_threshold = *v;
}
if let Some(v) = successes {
l.recovery_threshold = *v;
}
if let Some(v) = idle_timeout {
l.idle_timeout_secs = *v;
}
}
}
(cfg, Some(v.revision), false)
}
Action::Backend { action } => {
let (target, destination, remove) = match action {
BackendAction::Add { target, backend } => (target, backend, false),
BackendAction::Remove { target, backend } => (target, backend, true),
};
let v = selected(network, &target.target)?;
let mut cfg = v.config;
let b = backend(network, destination)?;
let l = listener(&mut cfg, target)?;
if remove {
if !l.backends.contains(&b) {
return Err("backend is not configured".into());
}
l.backends.retain(|x| x != &b);
} else {
l.backends.push(b);
}
(cfg, Some(v.revision), false)
}
};
// Canonical order makes displayed drafts and shared state deterministic.
config.listeners.sort_by_key(|l| (l.protocol, l.port));
for l in &mut config.listeners {
l.backends.sort();
}
let id = config.id.clone();
let vips = match ipc::save_vip(&socket, &network.network_id, config, expected, delete).await {
Ok(vips) => vips,
Err(error) => {
if let Some(file) = edited_file {
let (_, path) = file.keep()?;
return Err(format!("{error}; draft retained at {}", path.display()).into());
}
return Err(error.into());
}
};
if delete {
if args.json {
println!("{{\"removed\":\"{id}\"}}");
} else {
println!("VIP removed; existing flows drain.");
}
} else if let Some(v) = vips.iter().find(|v| v.config.id == id) {
show(network, v, args.json)?;
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
use clap::Parser;
#[test]
fn vip_commands_parse_complete_listener_and_settings() {
let cli = super::super::Cli::try_parse_from([
"tsng",
"vip",
"port",
"add",
"lab",
"k8s",
"tcp/443",
"--backend",
"cp1:6443",
"--backend",
"cp2:7443",
]);
assert!(cli.is_ok(), "{cli:?}");
assert!(
super::super::Cli::try_parse_from([
"tsng",
"vip",
"port",
"set",
"lab",
"k8s",
"tcp/443",
"--port-check",
"off",
"--failures",
"3"
])
.is_ok()
);
assert!(parse_listener("tcp/0").is_err());
assert!(parse_listener("sctp/53").is_err());
}
}
+92
View File
@@ -345,3 +345,95 @@ fn opposing_dns_flags_are_rejected() {
assert!(!result.status.success());
assert!(String::from_utf8_lossy(&result.stderr).contains("cannot be used with"));
}
#[test]
fn vip_cli_creates_multiple_ports_updates_dns_and_rejects_partial_invalid_changes() {
let agent = start("vip-test", 15367);
let created = agent.run(&["vip", "create", "vip-test", "api", "--json"]);
assert!(
created.status.success(),
"{}",
String::from_utf8_lossy(&created.stderr)
);
let vip: tsunagi::vip::VipStatus = serde_json::from_slice(&created.stdout).unwrap();
assert!(vip.config.listeners.is_empty());
let reply = wait_for_answer("127.0.0.1:15367".parse().unwrap(), "api.vip-test");
assert!(has_answer(&reply));
let socket = tsunagi::ipc::control_socket_path(&agent.dir.path().join("state"));
let runtime = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.unwrap();
let status = runtime
.block_on(tsunagi::ipc::request_status(socket))
.unwrap();
let backend = format!("{}:6443", status.endpoint_id);
let added = agent.run(&[
"vip",
"port",
"add",
"vip-test",
"api",
"tcp/443",
"--backend",
&backend,
]);
assert!(
added.status.success(),
"{}",
String::from_utf8_lossy(&added.stderr)
);
assert!(
agent
.run(&["vip", "port", "add", "vip-test", "api", "udp/53"])
.status
.success()
);
assert!(
!agent
.run(&["vip", "port", "add", "vip-test", "api", "tcp/443"])
.status
.success()
);
assert!(
agent
.run(&[
"vip",
"port",
"set",
"vip-test",
"api",
"tcp/443",
"--port-check",
"off"
])
.status
.success()
);
let shown = agent.run(&["vip", "show", "vip-test", "api", "--json"]);
assert!(shown.status.success());
let shown: tsunagi::vip::VipStatus = serde_json::from_slice(&shown.stdout).unwrap();
assert_eq!(shown.config.listeners.len(), 2);
assert!(
!shown
.config
.listeners
.iter()
.find(|l| l.protocol == tsunagi::vip::Protocol::Tcp)
.unwrap()
.port_check
);
assert_eq!(shown.config.listeners[0].backends[0].port, 6443);
let removed = agent.run(&["vip", "remove", "vip-test", "api"]);
assert!(removed.status.success());
let deadline = Instant::now() + Duration::from_secs(10);
loop {
if let Some(reply) = query("127.0.0.1:15367".parse().unwrap(), "api.vip-test", TYPE::A)
&& !has_answer(&reply)
{
break;
}
assert!(Instant::now() < deadline);
std::thread::sleep(Duration::from_millis(25));
}
}
+267
View File
@@ -54,6 +54,19 @@ fn start_bare(port: u16) -> Running {
}
fn start_bare_range(port: u16, range: Option<&str>) -> Running {
start_bare_config(port, range, None)
}
fn start_bare_config(port: u16, range: Option<&str>, protocols: Option<&str>) -> Running {
start_bare_peers(port, range, protocols, &[])
}
fn start_bare_peers(
port: u16,
range: Option<&str>,
protocols: Option<&str>,
peers: &[String],
) -> Running {
let dir = TempDir::new().unwrap();
let mut command = std::process::Command::new(env!("CARGO_BIN_EXE_tsng"));
command
@@ -71,6 +84,12 @@ fn start_bare_range(port: u16, range: Option<&str>) -> Running {
if let Some(range) = range {
command.args(["--ipv4-range", range]);
}
if let Some(protocols) = protocols {
command.args(["--protocol", protocols]);
}
for peer in peers {
command.args(["--peer", peer]);
}
let child = command.spawn().expect("the agent binary starts");
let mut agent = Running { child, dir };
let runtime = tokio::runtime::Builder::new_current_thread()
@@ -112,6 +131,254 @@ fn start(network: &str, port: u16) -> Running {
agent
}
#[test]
fn cli_forgets_only_the_selected_offline_claim_over_real_control() {
let cleaner = start("forget-cli", 0);
let runtime = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.unwrap();
let socket = tsunagi::ipc::control_socket_path(&cleaner.dir.path().join("state"));
let read = || {
runtime
.block_on(tsunagi::ipc::request_status(&socket))
.unwrap()
};
let before = read();
let bootstrap = format!("{}@{}", before.endpoint_id, before.bound_sockets[0]);
let target = start_bare_peers(0, None, Some("none"), &[bootstrap]);
assert!(
target
.run(&["id", "hostname", "old-cli-node"])
.status
.success()
);
assert!(
target
.run(&[
"join",
"--network",
"forget-cli",
"--secret",
"a-secret-for-the-cli-test"
])
.status
.success()
);
runtime.block_on(async {
tokio::time::timeout(Duration::from_secs(30), async {
loop {
let status = tsunagi::ipc::request_status(&socket).await.unwrap();
if status.networks[0]
.members
.iter()
.any(|m| m.hostname.as_deref() == Some("old-cli-node") && m.online)
{
break;
}
tokio::time::sleep(Duration::from_millis(25)).await;
}
})
.await
.unwrap();
});
assert!(
!cleaner
.run(&["network", "forget-node", "forget-cli", "old-cli-node"])
.status
.success()
);
assert!(
!cleaner
.run(&["network", "forget-node", "forget-cli", &before.endpoint_id])
.status
.success()
);
assert!(
target
.run(&["network", "stop", "forget-cli"])
.status
.success()
);
runtime.block_on(async {
tokio::time::timeout(Duration::from_secs(30), async {
loop {
let status = tsunagi::ipc::request_status(&socket).await.unwrap();
if status.networks[0]
.members
.iter()
.any(|m| m.hostname.as_deref() == Some("old-cli-node") && !m.online)
{
break;
}
tokio::time::sleep(Duration::from_millis(25)).await;
}
})
.await
.unwrap();
});
let forgotten = cleaner.run(&["network", "forget-node", "forget-cli", "old-cli-node"]);
assert!(
forgotten.status.success(),
"{}",
String::from_utf8_lossy(&forgotten.stderr)
);
assert!(String::from_utf8_lossy(&forgotten.stdout).contains("newer signed claim can return"));
assert!(
!read().networks[0]
.members
.iter()
.any(|m| m.hostname.as_deref() == Some("old-cli-node"))
);
assert!(
!cleaner
.run(&["network", "forget-node", "forget-cli", "old-cli-node"])
.status
.success()
);
}
#[test]
fn protocols_absent_from_startup_can_be_enabled_and_changed_offline() {
let mut agent = start_bare_config(0, None, Some("none"));
assert!(
agent
.run(&[
"join",
"--network",
"from-control-only",
"--secret",
"a-control-only-test-secret"
])
.status
.success()
);
let initial = agent.run(&["protocols", "from-control-only"]);
assert!(initial.status.success());
assert!(String::from_utf8_lossy(&initial.stdout).contains("protocols none"));
let enabled = agent.run(&["protocols", "from-control-only", "awg"]);
assert!(
enabled.status.success(),
"{}",
String::from_utf8_lossy(&enabled.stderr)
);
assert_eq!(
read_networks(&agent.dir)[0].protocols,
Some(vec!["awg".into()])
);
agent.child.kill().unwrap();
agent.child.wait().unwrap();
let saved = agent.run(&["protocols", "from-control-only", "wg-quic,awg"]);
assert!(
saved.status.success(),
"{}",
String::from_utf8_lossy(&saved.stderr)
);
assert!(String::from_utf8_lossy(&saved.stdout).contains("agent stopped"));
assert_eq!(
read_networks(&agent.dir)[0].protocols,
Some(vec!["wg-quic".into(), "awg".into()])
);
}
#[test]
fn protocols_are_changed_live_per_network_and_saved_without_changing_the_service() {
let agent = start("protocol-choice", 0);
assert!(
agent
.run(&[
"join",
"--network",
"other-choice",
"--secret",
"other-secret-for-cli-test"
])
.status
.success()
);
let before = read_networks(&agent.dir);
let other_id = before
.iter()
.find(|n| n.name.as_str() == "other-choice")
.unwrap()
.network_id;
let changed = agent.run(&["protocols", "protocol-choice", "awg,wg-quic"]);
assert!(
changed.status.success(),
"{}",
String::from_utf8_lossy(&changed.stderr)
);
let shown = agent.run(&["protocols", "protocol-choice"]);
assert!(shown.status.success());
assert!(String::from_utf8_lossy(&shown.stdout).contains("awg,wg-quic"));
let saved = read_networks(&agent.dir);
assert_eq!(
saved
.iter()
.find(|n| n.name.as_str() == "protocol-choice")
.unwrap()
.protocols,
Some(vec!["awg".into(), "wg-quic".into()])
);
assert_eq!(
saved
.iter()
.find(|n| n.network_id == other_id)
.unwrap()
.protocols,
before
.iter()
.find(|n| n.network_id == other_id)
.unwrap()
.protocols
);
for invalid in ["unknown", "awg,awg", "none,awg"] {
assert!(
!agent
.run(&["protocols", "protocol-choice", invalid])
.status
.success()
);
}
assert!(!agent.run(&["protocols", "missing", "awg"]).status.success());
let disabled = agent.run(&["protocols", "protocol-choice", "none"]);
assert!(
disabled.status.success(),
"{}",
String::from_utf8_lossy(&disabled.stderr)
);
assert_eq!(
read_networks(&agent.dir)
.iter()
.find(|n| n.name.as_str() == "protocol-choice")
.unwrap()
.protocols,
Some(Vec::new())
);
assert!(
agent
.run(&["network", "stop", "protocol-choice"])
.status
.success()
);
assert!(
agent
.run(&["protocols", "protocol-choice", "wg-quic"])
.status
.success()
);
assert!(
agent
.run(&["network", "start", "protocol-choice"])
.status
.success()
);
assert!(
String::from_utf8_lossy(&agent.run(&["protocols", "protocol-choice"]).stdout)
.contains("wg-quic")
);
}
#[test]
fn joining_with_no_secret_makes_one_and_prints_what_to_send() {
// The hurry case: a network with somebody, for as long as it is needed,
+15
View File
@@ -8,6 +8,21 @@ tsng up --protocol awg
tsng up --protocol wg,awg,wg-quic,tcp-tls
```
The startup list is the initial per-network preference. A saved network choice
overrides it and can enable any built-in protocol at runtime:
```sh
tsng protocols home awg,wg-quic
tsng protocols home awg
```
Changes are local to this device, persist across restarts, and retire existing
data links and end-to-end channels for that network before renegotiation.
Authenticated control sessions, membership, addresses and other networks stay
live. Removing every protocol (`none` in the CLI, all checkboxes off in the tray)
leaves the network control-only. This is different from automatic carrier
fallback, which preserves the selected end-to-end channel.
`awg` is Tsunagi AWG wire version 1. Its packet masking engine is adapted from
[RunaRoo/AmneziaWG-RS](https://github.com/RunaRoo/AmneziaWG-RS), with the pinned
source and license recorded in
+30 -6
View File
@@ -43,12 +43,36 @@ be an exit node in one network and not in another.
`tsunagi-exit:<interface>:<range>` and are removed when the offer ends, the
network stops, and when the agent exits. Only networks that offer have
rules, so the others' traffic is not forwarded.
- **Client:** policy routing, with no change to the main table. Table `28787`
holds `default dev <interface>`, and three rules sit after Tailscale's:
`5280 uidrange <agent uid> lookup main` (the agent's own connections never
go through the tunnel, which would be a loop), `5290 lookup main
suppress_prefixlength 0` (LAN and other specific routes keep working) and
`5300 lookup 28787`.
- **Client:** policy routing, with no change to the main table. Table `29811`
holds `default dev <interface>`. Rules `5205 fwmark 0x40000000/0x40000000
lookup main` and `5206 uidrange <agent uid> lookup main` keep the agent's
sockets and their replies outside the tunnel, before Tailscale's rules.
`5290 lookup main suppress_prefixlength 0` preserves LAN and other specific
routes, and `5300 lookup 29811` directs other traffic through the exit.
The exit table is `29811` (`0x7473`).
- **Linux reverse-path filtering:** only the agent's dedicated UID is marked
in mangle `OUTPUT`. Its own bit, `0x40000000`, is saved to conntrack and
restored in `PREROUTING` before the host firewall checks reply routing.
Other mark bits, including Tailscale's `0xff0000` mask, are preserved. The
rules are tagged `tsunagi-exit-underlay:<interface>` and never accept packets
around the host firewall. IPv4 and IPv6 protection is installed before exit
routes; routes are removed before protection, including rollback on failure.
Locally owned TCP listeners and bound UDP sockets also protect the first
incoming packet, before any outgoing packet has created a connection mark.
Socket addresses and ports
come from local kernel socket tables and match an actual socket; remote
announcements never supply these firewall arguments.
The socket set is refreshed before private pair negotiation is sent and on
discovery ticks, so fresh AWG ports are included. An unchanged set causes no
firewall writes; a socket-refresh failure leaves active exit routes in place.
Linux needs both `iptables` and `ip6tables` when IPv6 is present.
- **Kernel strict `rp_filter`:** if an interface uses strict mode (`1`), the
kernel must include marks in reverse lookups: set
`net.ipv4.conf.all.src_valid_mark=1`, or enable it on those interfaces.
The agent checks this before installing exit routes and reports an actionable
error if marks would be ignored. It never disables `rp_filter` or changes
host sysctls. Netfilter strict reverse checks also need their mark-aware
form, such as `rpfilter --validmark`, which NixOS already uses.
- Packets for addresses outside every overlay range are sent to the exit node
by the router; the exit node's replies are accepted from any source. A member
that is not your exit node cannot do that.
+14 -7
View File
@@ -9,20 +9,26 @@ Two versions exist and are independent:
- **Identity scheme**, `tsunagi-network-id-v1`. Frozen. Changing it creates a
different network space for the same name and secret.
- **Control protocol**, ALPN `tsunagi/ctrl/6`, `PROTOCOL_VERSION = 6`.
- **Control protocol**, ALPN `tsunagi/ctrl/8`, `PROTOCOL_VERSION = 8`.
Upgrading the crate or bumping the control protocol must never change an
existing `NetworkId`.
Control version 6 adds private transport negotiation for pairwise AWG and data
delivery checks. Data ALPN remains `tsunagi/data/5`. All members must upgrade
together; network ids and stored state are unchanged. See [amnezia.md](amnezia.md).
Control version 8 adds signed, multi-writer VIP field snapshots to version 7's
node retirements. Data ALPN remains `tsunagi/data/5`. All members must upgrade
together. Network ids, derivation labels, handshake encoding and existing record
signatures are unchanged. Local CLI/tray IPC is version 22; restart the agent
with matching clients. SQLite schema 10 adds `vip_fields`, preserving identity,
network configuration, signed node records and retirement proofs.
See [virtual services](vip.md), [state sync](sync-model.md) and [AWG](amnezia.md).
## Canonical encoding
Everything that is hashed or MAC'd uses length-prefixed fields, written here as
`LP(x) = u32_be(len(x)) || x`. String concatenation is never used, so no two
different field splits can produce the same bytes.
Network identity derivation and the membership transcript use length-prefixed
fields, written here as `LP(x) = u32_be(len(x)) || x`. String concatenation is
never used, so different field splits cannot produce the same bytes. VIP field
signatures use their separate domain and canonical postcard encoding described
in [virtual services](vip.md#shared-state-and-addresses).
## Network space identity
@@ -238,6 +244,7 @@ not affect other networks.
| `Ping { seq, payload }` | small request used to verify the exchange |
| `Pong { seq, payload }` | the echoed reply |
| `State { records }` | a snapshot of signed records, merged into what the receiver holds |
| `Retirements { markers }` | signed cleanup markers with the target's original signed claim as proof |
| `Peers { peers }` | unverified member address hints, requiring their own handshake |
| `Reachable { links }` | sender's current direct data links, by kind of transport |
| `Bye { reason }` | graceful goodbye; not a revocation of anything |
+1 -1
View File
@@ -124,5 +124,5 @@ counts sends without storing frames. Results are CPU forwarding cost, **not**
end-to-end network latency or VPN throughput; encryption, fragmentation,
sockets, congestion and scheduling contribute separately.
Wire compatibility: control ALPN `tsunagi/ctrl/6`, data ALPN `tsunagi/data/5`.
Wire compatibility: control ALPN `tsunagi/ctrl/8`, data ALPN `tsunagi/data/5`.
Upgrade every participant together; saved identities and network state persist.
+46 -27
View File
@@ -1,18 +1,9 @@
# Planned state synchronisation
# State synchronisation
**The first slice of this model is now implemented**, in `crates/tsunagi-core/src/state/`, and is
used for one thing: IPv4 overlay addresses. What follows describes the whole
model; the section at the end says exactly which parts exist.
The proof of concept still exchanges hostname and capability announcements
over live sessions and keeps no replicated history for those, which is why
WireGuard peer *membership* remains session-scoped even though a peer's
*address* no longer is. This file records the intended direction so the module
boundaries in [architecture.md](architecture.md) stay compatible with it, and so
nobody mistakes the current announcements for synchronisation.
There is no fake "ready CRDT" here and no snapshots that are not actually
verified.
Signed claims in `crates/tsunagi-core/src/state/` preserve IPv4 reservations and
hostnames across an absence. Signed releases and offline-node cleanup markers
free those reservations durably. Capabilities, links and authenticated sessions
remain volatile; a stored claim never authenticates a live connection.
## The problem
@@ -75,6 +66,39 @@ Turning a computer off is not a revocation of its hostname and does not remove
the participant. Revocations are not dropped merely because they are old, and
no acknowledgement from offline peers is required to keep working.
## Offline-node cleanup
Any authenticated member can sign a `SignedRetirement` for another member's
observed `Claim`. It includes the original signed claim as proof and uses the
separate `tsunagi-node-retirement/v1` signing domain. Both signatures and both
network scopes are checked. No owner, vote or quorum is involved, and no member
can fabricate the target's release or invent a future version cutoff.
The strongest observed target version wins; ties use a deterministic canonical
byte order. Claims at or below that version do not occupy an address, hostname
or visible member row. Original signed records remain stored, including their
range, so cleanup does not renumber the network. Markers remain even after a
newer claim returns: absence from a snapshot is never deletion and replaying an
old snapshot cannot restore the retired claim.
The local `Agent::forget_node` API checks the exact observed version again and
refuses self, online nodes, the selected exit node and missing/changed claims.
CLI and tray share this API. A returning live node can sign a newer claim and
allocate against the current holders; it resets the old hostname priority, so
it cannot reclaim a reused name merely by keeping its old timestamp. Cleanup
is not a permanent identity ban or a change of secret.
The marker and requester's own author counter commit in one SQLite transaction
before publication. Receivers persist a verified marker before applying it.
Invalid proofs are rejected individually. Periodic bounded marker batches
rotate through each peer's send queue, including when a queue fills, so a
replica that missed an update can catch up without the requester being online.
One strongest marker per target is retained, with a default limit of 1024 per
network and 1024 bytes per encoded marker. New targets beyond that limit are
refused rather than evicting resurrection protection. Corrupt mandatory marker
storage or a configured limit smaller than the stored set is an explicit error.
## Storage requirement this creates
When signed records land, writing the event and bumping the author's own
@@ -104,9 +128,9 @@ Implemented, in `crates/tsunagi-core/src/state/`:
* the merge rules above: higher version wins, an older version never rolls
back a newer one, duplicates are idempotent, a same-version conflict is
resolved identically on every replica and reported;
* a release tombstone, which merges correctly and is not undone by a replica
that has not heard of it — though nothing emits one yet, so freeing an
address still means forgetting the network;
* a release tombstone, emitted when leaving or replacing the device key and
not undone by a replica that has not heard of it;
* durable hostnames and equal-rights offline-node retirement described above;
* persistence in `state.sqlite`, with the record and the author's own version
counter committed in **one transaction before the record is announced**;
* distribution as a `State` control message, merged into what the receiver
@@ -114,21 +138,16 @@ Implemented, in `crates/tsunagi-core/src/state/`:
* allocation of a free IPv4 address against what everybody else holds, which
is what makes an address stable across an absence.
Deliberately not implemented: compaction, revoking a whole author, record
types beyond addressing, and any bound on how large a snapshot may grow
beyond the per-message limit.
Deliberately not implemented: compaction and permanent author revocation.
Claims have a per-message bound; retirement history also has a per-network bound.
## Future tests
These are **not implemented and must not be reported as passing**:
- snapshot merge against the rules above, including conflicting same-version
records;
- revocation propagation and resistance to resurrection after compaction;
- resistance to resurrection after compaction (compaction is not implemented);
- long network partitions and rejoin after an extended absence;
- hostname rename with atomic revoke-and-announce;
- recovery of a returning participant from a replica that is not the author;
- NAT traversal and hole punching between real hosts;
- relay fallback behaviour against a self-hosted relay;
- multi-process and multi-host deployment, as opposed to several library
instances inside one test process.
- multi-host deployment. CLI tests do run separate local agent processes;
library tests run several independent instances inside one process.
+72 -4
View File
@@ -164,10 +164,9 @@ paused clock. The public Mainline/relay smoke test is ignored by default.
## Not covered, and not claimed to be
Listed in [sync-model.md](sync-model.md#future-tests): snapshots, revocations,
long partitions, hostname renames, recovery of a returning participant, NAT
traversal, relay fallback, and multi-process or multi-host deployment. None of
these are implemented, and none are marked as passing.
Listed in [sync-model.md](sync-model.md#future-tests): compaction, long partitions,
NAT traversal, relay fallback and multi-host deployment. These scenarios are
not established by the default loopback suite.
## Debugging a test
@@ -184,3 +183,72 @@ without substituting iroh, membership, encryption or SQLite. It checks runtime
loss detection, fallback and fresh profiles, in addition to restart and input
validation. Tests are offline; public STUN/DPI/NAT effectiveness needs separate
real-network validation.
Runtime protocol selection tests switch real AWG and iroh/QUIC carriers and
end-to-end WireGuard tunnels from either endpoint-id role, carry full-size
packets, keep another active network's tunnels unchanged, disable and re-enable
the data plane without losing control sessions, and preserve selection after
restart. CLI tests use a real agent process and control socket to exercise
selection by network name, stopped networks, invalid lists, and persisted
control-only configuration, offline updates, and enabling a protocol absent
from the startup list. A real AWG test starts without a TUN, creates the single
memory interface on enablement, and races rapid changes against pending opens.
SQLite tests migrate schema 7 to 8 without changing
identity, secrets or signed records, preserve order and explicit empty choices,
and reject malformed stored selections. These checks use memory TUNs and do
not establish GUI rendering or behavior on an actual OS interface.
`tests/forget_node.rs` uses real agents, handshakes and SQLite to check equal
cleanup rights, network isolation, online/self/stale-version refusal, restart,
recovery of a stale replica, reuse of a hostname and a returning identity's
newer claim. Bounded batch delivery is checked with a two-message send queue
and twelve retained markers. Retirement unit tests check signatures, proof tampering, invented
future cutoffs, bounds, deterministic merge and resistance to stale snapshots.
SQLite tests migrate schema 8 to 9 and inject counter-write failure to prove
atomic rollback. The CLI test starts two real agent processes and exercises
offline cleanup by network and hostname through the control socket.
Native tray rendering and physical screen sizing are not exercised automatically.
Linux exit-client unit tests cover preservation of foreign mark bits, matching
only the agent's UID and locally owned TCP/UDP sockets, exact tagged cleanup,
partial IPv6 failure and mark-aware strict kernel RPF validation. The ignored
`overlay::hostrules::linux_exit_kernel::strict_firewall_preserves_underlay_and_application_exit_routing`
test creates two disposable network namespaces. It reproduces the UID-only
failure with real sockets, then runs TCP/UDP IPv4/IPv6 and first inbound TCP/UDP
packets, including a UDP port bound after activation, through real mark/conntrack
rules with both strict kernel and
Netfilter reverse-path checks. It verifies application exit routing, IPv6
blocking, repeated apply and cleanup with foreign rules at Tailscale's priority
retained. It requires root, iproute2,
iptables/ip6tables and Python; it does not run in the default suite or claim
to test the entire mesh inside these namespaces.
Devices-window tests render the egui grid at multiple widths with its real
scroll area. They check stretching and shrinking without an expanding first
column, separate handshake/control RTT values, unknown measurements and short
end-to-end hints that do not substitute the carrier protocol. This does not
establish native desktop rendering on every platform.
VIP tests cover signed multi-writer backend merging, stale draft rejection,
permanent deletion and replay, separate active/deleted capacity, and mandatory
SQLite migration/counter rollback. Real agent instances verify equal editing
rights, network isolation, restart and recovery without the creator. Packet
checks exercise RTT selection, port translation, UDP failover, a local backend,
fragment reassembly/checksums and ICMP quoted-header/PMTU translation. TCP check
recovery uses real loopback listening sockets. The WireGuard suite moves VIP
TCP/UDP packets through real encrypted tunnels, rejects forged VIP sources and
checks UDP failover after the selected agent stops. CLI/DNS tests run the real
binary, configure independent listeners, disable checks, reject an invalid
change atomically and observe DNS creation/deletion. GUI tests cover draft
isolation and form rendering. These do not establish live host TUN behavior or
native desktop behavior on Linux, macOS or Windows. See [VIP semantics](vip.md).
Tray layout tests render bounded table rows with long names, online/offline
members, Forget buttons, large counters and enlarged fonts. They assert text
stays within its own row and that narrow windows scroll without expanding the
viewport. VIP overview tests render live tables and egui scenes at multiple
widths, click a diagram backend, and distinguish the new-flow selection from
traffic on an existing binding. Sampling checks cover idle open flows, stale
observations, counter resets, recreated listeners and removal of cached
histories. These are headless egui checks, not native desktop screenshots.
+12 -2
View File
@@ -38,6 +38,14 @@ Read this before relying on anything here. The protocol is in
arbitrarily many identities, flood the network with records and collide with
other participants' names. This is why a majority is not a root of trust.
Signatures protect authorship; they do not make a participant honest.
- **Cleanup rights are equal.** Any member can retire an observed signed claim
of another device, freeing its hostname and IPv4 reservation. A valid original
claim bounds the cutoff; a member cannot invent a future cutoff or forge the
target's release. The local API refuses online nodes, but a remote signature
cannot prove that a target was offline. A live target may publish a newer
claim; a malicious member can repeatedly disrupt reservations. Cleanup is
not expulsion or secret revocation. See
[sync-model.md](sync-model.md#offline-node-cleanup).
- **Weak secrets.** This targets high-entropy secrets. There is no PAKE, so a
short human passphrase can be guessed offline by anyone who can reach the
handshake. Use `NetworkSecret::generate()`.
@@ -55,8 +63,10 @@ Read this before relying on anything here. The protocol is in
having iroh under it does not make it so.
- **A cloned state directory is a cloned identity.** `state.sqlite` holds the
device secret key and the network secrets. Copying it copies the participant.
Restoring an old backup rolls the agent's state back, which — once signed
records exist — can resurrect revoked information or replay stale versions.
Restoring an old backup rolls the agent's state back and can replay stale
versions until it meets a replica holding the newer signed state. Retained
cleanup markers prevent old claims from resurrecting after that merge; they
cannot help if every copy of a marker has been lost.
- **A compromised host.** The secret is on disk to survive restarts. File
permissions are owner-only where the platform supports it, and the state
directory takes an ownership lock, but neither defends against a user who can
+187
View File
@@ -0,0 +1,187 @@
# Named virtual services
A VIP is a shared name such as `api.home`, with independent TCP/UDP listeners.
Each listener maps its port to a set of node identities and application ports.
The application uses the name; the automatically allocated IPv4 address is an
internal reservation. The current overlay carries IPv4 only.
## CLI and tray
```sh
tsng vip list home
tsng vip create home api
tsng vip port add home api tcp/443 --backend server1:6443 --backend server2:7443
tsng vip port add home api udp/53 --backend server1:5353
tsng vip backend add home api tcp/443 server3:6443
tsng vip backend remove home api tcp/443 server2:7443
tsng vip port set home api tcp/443 --port-check off
tsng vip port set home api tcp/443 --check-interval-ms 3000 --check-timeout-ms 2000 --failures 2 --successes 1 --idle-timeout 7200
tsng vip show home api
tsng vip status home api --watch
tsng vip edit home api
tsng vip port remove home api udp/53
tsng vip remove home api
```
`list` without a network lists all configured networks. Network and VIP
selectors accept exact names, full ids and unique id prefixes; backend node
selectors also accept stored node names. Ambiguous selections change nothing.
Names resolve to identities when saved, so a renamed node remains a backend.
`--json` provides configuration, revision, counters and observations;
`list`/`watch` emit one JSON object per service/sample. `edit`
opens JSON in `VISUAL`/`EDITOR` and submits one complete version-checked draft.
An invalid or rejected edited file is retained with its path reported.
Each mutating command is atomic. A listener can be added with its full initial
backend set, and `edit` can replace the entire configuration at once. Empty
VIPs/listeners are valid and resolve, but cannot accept traffic. Configuration
updates currently require the network to be active locally. Saved configuration remains visible while a
network is stopped.
The tray's network tile has a **VIP** button. Its separate window edits names,
listeners, backend nodes/ports, check settings and flow timeouts. All edits,
including deletion, remain drafts until **Save**. Status updates do not
replace a dirty draft. A save conflict retains the draft; **Discard / reload**
loads the latest configuration. Closing a dirty window offers explicit discard
or continued editing.
The **Overview** tab shows a total VIP traffic plot and one backend table per
listener, with health, RTT, open flows, totals, TX/RX rates and individual
traffic plots. Plots switch between bytes and packets. Observations describe
traffic seen by this device, not a network-wide aggregate.
An interactive diagram connects the service name through each listener to its
node and target port. Drag to pan, Ctrl+scroll/pinch to zoom, and use **Fit**
to restore the view. Click a backend in the diagram or table to highlight it
and see its identity, address and measurement age. Green branches with moving
dots indicate counter growth during the latest status sample; a blue ring
marks the backend selected for new connections. Existing flows can therefore
keep a different branch active. Idle open flows alone do not light up a branch;
activity expires after five seconds without a fresh sample. Graph interaction
never edits or saves configuration. The **Configuration** tab retains the
explicit Save workflow. Addresses and ids are available in a collapsed detail.
## Delivery and recovery
The client agent selects the eligible backend with the lowest smoothed full
round-trip latency through the actual encrypted data plane, including relays.
Unknown or stale RTT is not treated as zero. Local backends have zero path RTT.
A new flow is pinned to its selected backend; a faster RTT does not move it.
This can concentrate new traffic on the closest node.
At local ingress, NAT replaces destination address/port and reserves an OS
source port for an unambiguous reverse mapping. Return traffic passes normal
network/peer source and destination admission **before** reverse NAT restores
the virtual source address/port. Plugins and encrypted transit stay unchanged.
A peer never gains permission to claim a VIP source. Fragment assembly is
bounded, checksums are updated, packets are fragmented back to the overlay MTU,
and ICMP errors have their quoted NAT headers restored. Fragment caches are
scoped to network and authenticated sender, with idle expiry independent of
further traffic.
For a backend on the client itself, the request is injected into the same TUN
with the nonlocal VIP as source, and the host reply is reverse-translated at
local ingress. The local backend therefore sees the VIP as source; remote
backends see the client's ordinary overlay address. Firewalls and application
bind addresses remain the administrator's responsibility.
Removing a backend/listener/VIP prevents new bindings while existing bindings
continue until idle expiry. Established TCP connections are not migrated. A retransmitted empty SYN may
choose another backend before a SYN-ACK has arrived; SYNs carrying application
data are not retargeted. A new SYN sequence creates a new binding even if the
client reuses its address/port tuple. TCP RST shortens binding retention. An unavailable UDP backend is replaced for subsequent
datagrams; already sent datagrams are never replayed. Stateful UDP applications
may need to re-establish their application session. Protocols that embed IP
addresses or negotiate additional ports require their own configuration; no
application payloads are parsed or rewritten.
Idle expiry defaults to 7200 seconds for TCP and 120 seconds for UDP. Shadow
ports are quarantined for 120 seconds after retirement to reject late replies
and avoid immediate reuse. Active and quarantined bindings share a bounded
capacity. State and sockets are released on network stop or agent shutdown.
A restart drops bindings; shared service configuration remains intact.
## Health
One probe exchange per second measures each used peer through the encrypted
IP tunnel. The EWMA uses 7/8 of the previous sample and 1/8 of the new one.
A probe expires after three seconds; two consecutive missing responses or a
missing data path exclude the peer. Measurements expire after ten seconds.
Probes continue during failure so recovery is automatic.
TCP connect checks default to on. Defaults are a three-second interval,
two-second timeout, two failures to exclude and one success to recover.
Checks are shared for the same network/node/address/TCP-port, run off the
packet path with at most 32 concurrent connects, and are scheduled fairly.
A 250ms worker tick services timers. Connect checks send no application data;
they confirm port acceptance and may appear in server logs. They cannot prove
application readiness after a successful handshake. Disabling port checks
retains data-plane probes.
UDP uses encrypted path checks and explicit ICMP port errors. No arbitrary
probe datagrams are injected into an unknown application protocol. Silence
cannot prove failure of an arbitrary UDP application. An explicit unreachable
port is excluded for ten seconds and then becomes eligible to retry.
## Shared state and addresses
Any authenticated member can create, edit or remove any VIP. Each field is a
self-contained signed register: network, service id, field key, optional encoded
value, Lamport clock, author, durable author counter and signature. Signing uses
the separate `tsunagi-vip-field/v1\0` domain followed by postcard encoding of
those fields with the signature vector empty. The encoding/order is a wire
contract; the identity scheme and handshake transcript are unchanged.
Concurrent edits to different backend fields merge. A conflict on one field
uses `(Lamport clock, author id, canonical bytes)` ordering, independent of wall
time and arrival order. Listener generations prevent backend fields from an
old removed listener reappearing when the port is recreated. Service deletion
is a permanent signed tombstone for its id; recreation gets a fresh id.
Missing snapshot fields are never deletion.
Saving validates the entire draft and checks the revision shown by the editor.
The merged snapshot and author's counter commit together in SQLite before local
routing changes or publication. Receivers verify and persist complete service
snapshots before exposing them. Periodic rotating snapshots repair queue loss
and returning replicas without requiring the creator online. Concurrent edits
made before participants have seen each other can still merge after a save.
A merged configuration exceeding listener/backend limits is reported as an
error and excluded from routing rather than selected according to arrival order.
VIP allocation walks the agreed overlay range from a service-id-derived seed,
excluding known node claims and other allocated VIPs. Nodes avoid known VIP
reservations when choosing a new address. Names colliding with node names or
another service are reported; node names take precedence and service ties use
id order. A partition can temporarily have incomplete state; after merge an
address or conflicting name's effective holder can change. Use DNS names.
DNS TTL is 30 seconds; DNS does not migrate established connections or force
applications to discard cached addresses. Existing NAT bindings retain their
original VIP address while draining.
`Limits` bounds active services (64 by default), permanent deletion markers
(1024), signed fields per active service (192), listeners (16), backends per
listener (32), active TCP checks (32), flow/port reservations (4096) and incomplete
IPv4 datagrams (64). A deleted service retains only its signed permanent
marker; obsolete fields can be dropped because replay cannot override deletion.
A full deletion history is an explicit error; protection is never silently
evicted. Deleted services do not consume the active-service limit. Whole service
snapshots must also fit the configured control frame limit. Corrupt mandatory VIP storage
is a hard error on activation, never reset as a cache.
## Observations and verification
Status is this client's observation. Per VIP/listener/backend it reports the
next selected backend, health reason, RTT and sample age, active bindings,
user IP packet/byte counters, drops and UDP failovers. Probe bytes do not enter
user traffic counters. The GUI and CLI watch derive rates from cumulative
counters; counters reset on restart. These are IP forwarding counters, not
application acknowledgements or guaranteed delivered-byte totals.
Automated tests cover field merging, stale drafts, signed deletion, tampering,
network isolation, SQLite migration and transactional rollback, real port
connect failure/recovery, RTT selection, UDP failover, local hairpin NAT,
fragmentation/checksums, real encrypted TCP/UDP packet delivery and source
admission, real CLI/DNS wiring and draft form rendering. In-memory TUN tests
exercise packets, not an operating system TCP stack. Native desktop behavior,
physical interfaces and cross-platform kernel/firewall behavior need live-host
validation; automated tests do not establish those.
+21 -9
View File
@@ -26,14 +26,26 @@
nixosModules.default = import ./nix/module.nix self;
devShells = forAllSystems (pkgs: {
default = pkgs.mkShell {
inputsFrom = [ self.packages.${pkgs.stdenv.hostPlatform.system}.tsunagi ];
packages = [
pkgs.rustfmt
pkgs.clippy
];
};
});
devShells = forAllSystems (
pkgs:
let
trayPackage = self.packages.${pkgs.stdenv.hostPlatform.system}.tsunagi;
in
{
default = pkgs.mkShell {
inputsFrom = [ trayPackage ];
packages = [
pkgs.rustfmt
pkgs.clippy
]
++ trayPackage.trayRuntimeLibs;
# Cargo binaries have no installed wrapper. Expose runtime-loaded
# libraries here too, while preserving the caller's library path.
shellHook = ''
export LD_LIBRARY_PATH="${pkgs.lib.makeLibraryPath trayPackage.trayRuntimeLibs}:/run/opengl-driver/lib''${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
'';
};
}
);
};
}
+1
View File
@@ -102,6 +102,7 @@ in
# Exit nodes: the agent looks for iptables only in fixed FHS directories.
systemd.tmpfiles.rules = [
"L+ /usr/bin/iptables - - - - ${pkgs.iptables}/bin/iptables"
"L+ /usr/bin/ip6tables - - - - ${pkgs.iptables}/bin/ip6tables"
];
# Names of mesh members are served locally and handed to the system resolver.
+8 -1
View File
@@ -52,7 +52,10 @@ rustPlatform.buildRustPackage {
# The suite opens real sockets and interfaces; it runs in CI instead.
doCheck = false;
nativeBuildInputs = [ pkg-config ] ++ lib.optionals withTray [
nativeBuildInputs = [
pkg-config
]
++ lib.optionals withTray [
copyDesktopItems
wrapGAppsHook3
];
@@ -64,6 +67,10 @@ rustPlatform.buildRustPackage {
dontWrapGApps = true;
# The development shell runs unwrapped Cargo binaries and needs the same
# dynamically loaded libraries as the installed tray wrapper.
passthru = { inherit trayRuntimeLibs; };
postInstall = ''
install -Dm644 dist/linux/50-tsunagi-resolved.rules \
"$out/share/doc/tsunagi/50-tsunagi-resolved.rules"
+539
View File
@@ -0,0 +1,539 @@
# THIS FILE IS AUTOMATICALLY GENERATED BY CARGO
#
# When uploading crates to the registry Cargo will automatically
# "normalize" Cargo.toml files for maximal compatibility
# with all versions of Cargo and also rewrite `path` dependencies
# to registry (e.g., crates.io) dependencies.
#
# If you are reading this file be aware that the original Cargo.toml
# will likely look very different (and much more reasonable).
# See Cargo.toml.orig for the original contents.
[package]
edition = "2024"
rust-version = "1.91"
name = "iroh"
version = "1.2.0"
authors = [
"dignifiedquire <me@dignifiedquire.com>",
"n0 team",
]
build = "build.rs"
autolib = false
autobins = false
autoexamples = false
autotests = false
autobenches = false
description = "p2p quic connections dialed by public key"
readme = "README.md"
keywords = [
"quic",
"networking",
"holepunching",
"p2p",
]
license = "MIT OR Apache-2.0"
repository = "https://github.com/n0-computer/iroh"
resolver = "2"
[package.metadata.docs.rs]
all-features = true
rustdoc-args = [
"--cfg",
"iroh_docsrs",
]
[package.metadata.cargo_check_external_types]
allowed_external_types = [
"iroh_base::*",
"iroh_dns::*",
"iroh_relay::*",
"iroh_metrics::*",
"n0_error::*",
"n0_watcher::*",
"noq::*",
"noq_proto::*",
"noq_udp::*",
"bytes::*",
"http::*",
"serde_core::*",
"tokio::*",
"url::*",
"rustls::*",
"futures_core::stream::Stream",
"futures_lite::stream::Boxed",
]
[features]
default = [
"metrics",
"fast-apple-datapath",
"portmapper",
"tls-ring",
]
fast-apple-datapath = ["noq/fast-apple-datapath"]
metrics = [
"iroh-metrics/metrics",
"iroh-relay/metrics",
]
platform-verifier = ["iroh-relay/platform-verifier"]
portmapper = ["dep:portmapper"]
qlog = ["noq/qlog"]
test-utils = [
"iroh-relay/test-utils",
"iroh-relay/server",
"dep:axum",
"dep:simple-dns",
]
tls-aws-lc-rs = [
"noq/aws-lc-rs",
"iroh-relay/tls-aws-lc-rs",
"iroh-dns/tls-aws-lc-rs",
]
tls-ring = [
"noq/ring",
"iroh-relay/tls-ring",
"iroh-dns/tls-ring",
]
unstable-custom-transports = []
unstable-net-report = []
[lib]
name = "iroh"
crate-type = [
"lib",
"cdylib",
]
path = "src/lib.rs"
[[example]]
name = "0rtt"
path = "examples/0rtt.rs"
required-features = []
[[example]]
name = "auth-hook"
path = "examples/auth-hook.rs"
[[example]]
name = "connect"
path = "examples/connect.rs"
required-features = []
[[example]]
name = "connect-unreliable"
path = "examples/connect-unreliable.rs"
required-features = []
[[example]]
name = "custom-transport"
path = "examples/custom-transport.rs"
required-features = [
"test-utils",
"unstable-custom-transports",
]
[[example]]
name = "echo"
path = "examples/echo.rs"
required-features = []
[[example]]
name = "echo-no-router"
path = "examples/echo-no-router.rs"
required-features = []
[[example]]
name = "home-relay-status"
path = "examples/home-relay-status.rs"
required-features = []
[[example]]
name = "incoming-filter"
path = "examples/incoming-filter.rs"
required-features = []
[[example]]
name = "listen"
path = "examples/listen.rs"
required-features = []
[[example]]
name = "listen-unreliable"
path = "examples/listen-unreliable.rs"
required-features = []
[[example]]
name = "monitor-connections"
path = "examples/monitor-connections.rs"
[[example]]
name = "pq-only-key-exchange"
path = "examples/pq-only-key-exchange.rs"
required-features = ["tls-aws-lc-rs"]
[[example]]
name = "prefer-pq-key-exchange"
path = "examples/prefer-pq-key-exchange.rs"
required-features = ["tls-aws-lc-rs"]
[[example]]
name = "remote-info"
path = "examples/remote-info.rs"
[[example]]
name = "screening-connection"
path = "examples/screening-connection.rs"
[[example]]
name = "search"
path = "examples/search.rs"
required-features = []
[[example]]
name = "transfer"
path = "examples/transfer.rs"
required-features = []
[[test]]
name = "integration"
path = "tests/integration.rs"
[[test]]
name = "patchbay"
path = "tests/patchbay.rs"
[dependencies.axum]
version = "0.8"
optional = true
[dependencies.backon]
version = "1.4"
[dependencies.blake3]
version = "1.8.3"
default-features = false
[dependencies.bytes]
version = "1.11"
[dependencies.ctutils]
version = "0.4.0"
default-features = false
[dependencies.data-encoding]
version = "2.2"
[dependencies.derive_more]
version = "2.0.1"
features = [
"debug",
"display",
"from",
"try_into",
"deref",
"from_str",
"into_iterator",
]
[dependencies.ed25519-dalek]
version = ">=3.0.0-rc.0,<4.0.0"
features = [
"serde",
"rand_core",
"zeroize",
"pkcs8",
"pem",
]
[dependencies.futures-util]
version = "0.3"
[dependencies.http]
version = "1"
[dependencies.ipnet]
version = "2"
[dependencies.iroh-base]
version = "1.2.0"
features = [
"key",
"relay",
]
default-features = false
[dependencies.iroh-dns]
version = "1.3.0"
[dependencies.iroh-metrics]
version = "1.0.1"
default-features = false
[dependencies.iroh-relay]
version = "1.2.0"
default-features = false
[dependencies.n0-error]
version = "1.0.0"
[dependencies.n0-future]
version = "0.3"
[dependencies.n0-watcher]
version = "1.0.0"
[dependencies.netwatch]
version = "0.19.3"
[dependencies.noq]
version = "1.3.0"
features = ["rustls"]
default-features = false
[dependencies.noq-proto]
version = "1.3.0"
default-features = false
[dependencies.noq-udp]
version = "1.3.0"
default-features = false
[dependencies.papaya]
version = "0.2.3"
default-features = false
[dependencies.pin-project]
version = "1"
[dependencies.portable-atomic]
version = "1"
[dependencies.rand]
version = "0.10"
[dependencies.reqwest]
version = "0.13"
features = [
"rustls-no-provider",
"stream",
]
default-features = false
[dependencies.rustc-hash]
version = "2"
[dependencies.rustls]
version = "0.23.33"
default-features = false
[dependencies.serde]
version = "1.0.219"
features = [
"derive",
"rc",
]
[dependencies.simple-dns]
version = "0.12"
optional = true
[dependencies.smallvec]
version = "1.11.1"
[dependencies.strum]
version = "0.28"
features = ["derive"]
[dependencies.tokio]
version = "1.44.1"
features = [
"io-util",
"macros",
"sync",
"rt",
]
[dependencies.tokio-stream]
version = "0.1.15"
features = ["sync"]
[dependencies.tokio-util]
version = "0.7"
features = [
"io-util",
"io",
"rt",
]
[dependencies.tracing]
version = "0.1"
[dependencies.url]
version = "2.5"
features = ["serde"]
[dependencies.webpki_types]
version = "1.12"
package = "rustls-pki-types"
[dev-dependencies.chrono]
version = "0.4.43"
[dev-dependencies.console_error_panic_hook]
version = "0.1"
[dev-dependencies.n0-error]
version = "1.0.0"
features = ["anyhow"]
[dev-dependencies.postcard]
version = "1.1.1"
features = ["use-std"]
[dev-dependencies.rand_chacha]
version = "0.10"
[dev-dependencies.simple-dns]
version = "0.12"
[dev-dependencies.tracing-subscriber]
version = "0.3"
features = ["env-filter"]
[build-dependencies.cfg_aliases]
version = "0.2.2"
[target.'cfg(all(target_family = "wasm", target_os = "unknown"))'.dependencies.getrandom]
version = "0.4"
features = ["wasm_js"]
[target.'cfg(all(target_family = "wasm", target_os = "unknown"))'.dependencies.time]
version = "0.3"
features = ["wasm-bindgen"]
[target.'cfg(all(target_family = "wasm", target_os = "unknown"))'.dependencies.wasm-bindgen-futures]
version = "0.4"
[target.'cfg(all(target_family = "wasm", target_os = "unknown"))'.dev-dependencies.wasm-bindgen-test]
version = "0.3.62"
[target.'cfg(all(target_family = "wasm", target_os = "unknown"))'.dev-dependencies.wasm-tracing]
version = "2.1.0"
[target.'cfg(not(all(target_family = "wasm", target_os = "unknown")))'.dependencies.noq]
version = "1.3.0"
features = [
"runtime-tokio",
"rustls",
]
default-features = false
[target.'cfg(not(all(target_family = "wasm", target_os = "unknown")))'.dependencies.portmapper]
version = "0.19.3"
optional = true
default-features = false
[target.'cfg(not(all(target_family = "wasm", target_os = "unknown")))'.dependencies.tokio]
version = "1"
features = [
"io-util",
"macros",
"sync",
"rt",
"net",
"fs",
"io-std",
]
[target.'cfg(not(all(target_family = "wasm", target_os = "unknown")))'.dev-dependencies.assert_matches]
version = "1.5.0"
[target.'cfg(not(all(target_family = "wasm", target_os = "unknown")))'.dev-dependencies.axum]
version = "0.8"
[target.'cfg(not(all(target_family = "wasm", target_os = "unknown")))'.dev-dependencies.clap]
version = "4"
features = ["derive"]
[target.'cfg(not(all(target_family = "wasm", target_os = "unknown")))'.dev-dependencies.console]
version = "0.16"
[target.'cfg(not(all(target_family = "wasm", target_os = "unknown")))'.dev-dependencies.indicatif]
version = "0.18"
features = ["tokio"]
[target.'cfg(not(all(target_family = "wasm", target_os = "unknown")))'.dev-dependencies.iroh-base]
version = "1.2.0"
features = [
"key",
"relay",
]
default-features = false
[target.'cfg(not(all(target_family = "wasm", target_os = "unknown")))'.dev-dependencies.iroh-relay]
version = "1.2.0"
features = [
"test-utils",
"server",
]
default-features = false
[target.'cfg(not(all(target_family = "wasm", target_os = "unknown")))'.dev-dependencies.n0-tracing-test]
version = "0.3"
[target.'cfg(not(all(target_family = "wasm", target_os = "unknown")))'.dev-dependencies.parse-size]
version = "1.1.0"
features = ["std"]
[target.'cfg(not(all(target_family = "wasm", target_os = "unknown")))'.dev-dependencies.pretty_assertions]
version = "1.4"
[target.'cfg(not(all(target_family = "wasm", target_os = "unknown")))'.dev-dependencies.rand_chacha]
version = "0.10"
[target.'cfg(not(all(target_family = "wasm", target_os = "unknown")))'.dev-dependencies.serde_json]
version = "1"
[target.'cfg(not(all(target_family = "wasm", target_os = "unknown")))'.dev-dependencies.tokio]
version = "1"
features = [
"io-util",
"sync",
"rt",
"rt-multi-thread",
"net",
"fs",
"macros",
"time",
"test-util",
]
[target.'cfg(not(all(target_family = "wasm", target_os = "unknown")))'.dev-dependencies.tracing-subscriber]
version = "0.3"
features = ["env-filter"]
[target.'cfg(target_os = "linux")'.dev-dependencies.ctor]
version = "1.0"
[target.'cfg(target_os = "linux")'.dev-dependencies.patchbay]
version = "0.7"
features = ["iroh-metrics"]
[target.'cfg(target_os = "linux")'.dev-dependencies.testdir]
version = "0.10"
[lints.clippy]
unused-async = "warn"
[lints.rust]
missing_debug_implementations = "warn"
[lints.rust.unexpected_cfgs]
level = "warn"
priority = 0
check-cfg = [
"cfg(iroh_docsrs)",
"cfg(iroh_loom)",
"cfg(skip_patchbay)",
]
+256
View File
@@ -0,0 +1,256 @@
[package]
name = "iroh"
version = "1.2.0"
edition = "2024"
readme = "README.md"
description = "p2p quic connections dialed by public key"
license = "MIT OR Apache-2.0"
authors = ["dignifiedquire <me@dignifiedquire.com>", "n0 team"]
repository = "https://github.com/n0-computer/iroh"
keywords = ["quic", "networking", "holepunching", "p2p"]
# Sadly this also needs to be updated in .github/workflows/ci.yml
rust-version = "1.91"
[lib]
# We need "cdylib" to actually generate .wasm files when we run with --target=wasm32-unknown-unknown.
# It would be nice if we could make this target-dependent, but we can't (yet): https://github.com/rust-lang/cargo/issues/12260
crate-type = ["lib", "cdylib"]
[lints]
workspace = true
[dependencies]
backon = { version = "1.4" }
blake3 = { version = "1.8.3", default-features = false }
bytes = "1.11"
ctutils = { version = "0.4.0", default-features = false }
data-encoding = "2.2"
derive_more = { version = "2.0.1", features = ["debug", "display", "from", "try_into", "deref", "from_str", "into_iterator"] }
ed25519-dalek = { version = ">=3.0.0-rc.0,<4.0.0", features = ["serde", "rand_core", "zeroize", "pkcs8", "pem"] }
http = "1"
ipnet = "2"
iroh-base = { version = "1.2.0", default-features = false, features = ["key", "relay"], path = "../iroh-base" }
iroh-dns = { version = "1.3.0", path = "../iroh-dns" }
iroh-relay = { version = "1.2.0", path = "../iroh-relay", default-features = false }
n0-future = "0.3"
n0-error = "1.0.0"
n0-watcher = "1.0.0"
netwatch = "0.19.3"
papaya = { version = "0.2.3", default-features = false }
pin-project = "1"
portable-atomic = "1"
noq = { version = "1.3.0", default-features = false, features = ["rustls"] }
noq-proto = { version = "1.3.0", default-features = false }
noq-udp = { version = "1.3.0", default-features = false }
rand = "0.10"
reqwest = { version = "0.13", default-features = false, features = ["rustls-no-provider", "stream"] }
rustc-hash = "2"
rustls = { version = "0.23.33", default-features = false }
serde = { version = "1.0.219", features = ["derive", "rc"] }
simple-dns = { version = "0.12", optional = true }
smallvec = "1.11.1"
strum = { version = "0.28", features = ["derive"] }
tokio = { version = "1.44.1", features = [
"io-util",
"macros",
"sync",
"rt",
] }
tokio-stream = { version = "0.1.15", features = ["sync"] }
tokio-util = { version = "0.7", features = ["io-util", "io", "rt"] }
tracing = "0.1"
url = { version = "2.5", features = ["serde"] }
webpki_types = { package = "rustls-pki-types", version = "1.12" }
# metrics
iroh-metrics = { version = "1.0.1", default-features = false }
futures-util = "0.3"
# test_utils
axum = { version = "0.8", optional = true }
# non-wasm-in-browser dependencies
[target.'cfg(not(all(target_family = "wasm", target_os = "unknown")))'.dependencies]
portmapper = { version = "0.19.3", optional = true, default-features = false }
noq = { version = "1.3.0", default-features = false, features = ["runtime-tokio", "rustls"] }
tokio = { version = "1", features = [
"io-util",
"macros",
"sync",
"rt",
"net",
"fs",
"io-std",
] }
# wasm-in-browser dependencies
[target.'cfg(all(target_family = "wasm", target_os = "unknown"))'.dependencies]
wasm-bindgen-futures = "0.4"
# we don't use time directly, but need to enable it because x509_parser uses these in browsers and we need to enable some feature flags for that to work
time = { version = "0.3", features = ["wasm-bindgen"] }
getrandom = { version = "0.4", features = ["wasm_js"] }
# target-common test/dev dependencies
[dev-dependencies]
console_error_panic_hook = "0.1"
n0-error = { version = "1.0.0", features = ["anyhow"] }
postcard = { version = "1.1.1", features = ["use-std"] }
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
rand_chacha = "0.10"
chrono = "0.4.43"
simple-dns = "0.12"
# *non*-wasm-in-browser test/dev dependencies
[target.'cfg(not(all(target_family = "wasm", target_os = "unknown")))'.dev-dependencies]
assert_matches = "1.5.0"
axum = { version = "0.8" }
pretty_assertions = "1.4"
rand_chacha = "0.10"
tokio = { version = "1", features = [
"io-util",
"sync",
"rt",
"rt-multi-thread",
"net",
"fs",
"macros",
"time",
"test-util",
] }
serde_json = "1"
iroh-relay = { version = "1.2.0", path = "../iroh-relay", default-features = false, features = ["test-utils", "server"] }
n0-tracing-test = "0.3"
clap = { version = "4", features = ["derive"] }
tracing-subscriber = { version = "0.3", features = [
"env-filter",
] }
indicatif = { version = "0.18", features = ["tokio"] }
parse-size = { version = "1.1.0", features = ['std'] }
iroh-base = { version = "1.2.0", default-features = false, features = ["key", "relay"], path = "../iroh-base" }
console = { version = "0.16" }
# wasm-in-browser test/dev dependencies
[target.'cfg(all(target_family = "wasm", target_os = "unknown"))'.dev-dependencies]
wasm-tracing = "2.1.0"
wasm-bindgen-test = "0.3.62"
# patchbay netsim test dependencies (linux only)
[target.'cfg(target_os = "linux")'.dev-dependencies]
ctor = "1.0"
patchbay = { version = "0.7", features = ["iroh-metrics"] }
testdir = "0.10"
[build-dependencies]
cfg_aliases = { version = "0.2.2" }
[features]
default = ["metrics", "fast-apple-datapath", "portmapper", "tls-ring"]
portmapper = ["dep:portmapper"]
metrics = ["iroh-metrics/metrics", "iroh-relay/metrics"]
test-utils = ["iroh-relay/test-utils", "iroh-relay/server", "dep:axum", "dep:simple-dns"]
# Enables fetching TLS trust anchors from the operating system
platform-verifier = ["iroh-relay/platform-verifier"]
qlog = ["noq/qlog"]
# Use private Apple APIs to send multiple packets in a single syscall.
fast-apple-datapath = ["noq/fast-apple-datapath"]
# Use ring as the crypto backend.
tls-ring = ["noq/ring", "iroh-relay/tls-ring", "iroh-dns/tls-ring"]
# Use aws-lc-rs as the crypto backend, unless `ring` is also enabled.
tls-aws-lc-rs = ["noq/aws-lc-rs", "iroh-relay/tls-aws-lc-rs", "iroh-dns/tls-aws-lc-rs"]
# Unstable: Custom transport API (may change without notice)
unstable-custom-transports = []
# Unstable: API to access an endpoint's NetReport (may change without notice)
unstable-net-report = []
[package.metadata.docs.rs]
all-features = true
rustdoc-args = ["--cfg", "iroh_docsrs"]
[package.metadata.cargo_check_external_types]
allowed_external_types = [
# workspace crates
"iroh_base::*",
"iroh_dns::*",
"iroh_relay::*",
# crates owned by us that will move to 1.0 as well
"iroh_metrics::*",
"n0_error::*",
"n0_watcher::*",
"noq::*",
"noq_proto::*",
"noq_udp::*",
# 1.0 crates that we deem fine to be part of the public API
"bytes::*",
"http::*",
"serde_core::*",
"tokio::*",
"url::*",
# non-1.0 crates that we decided to accept in the public API
"rustls::*",
"futures_core::stream::Stream",
# only type alias
"futures_lite::stream::Boxed",
]
[[test]]
name = "integration"
path = "tests/integration.rs"
[[example]]
name = "listen"
required-features = []
[[example]]
name = "connect"
required-features = []
[[example]]
name = "custom-transport"
required-features = ["test-utils", "unstable-custom-transports"]
[[example]]
name = "listen-unreliable"
required-features = []
[[example]]
name = "connect-unreliable"
required-features = []
[[example]]
name = "search"
required-features = []
[[example]]
name = "echo"
required-features = []
[[example]]
name = "echo-no-router"
required-features = []
[[example]]
name = "transfer"
required-features = []
[[example]]
name = "0rtt"
required-features = []
[[example]]
name = "incoming-filter"
required-features = []
[[example]]
name = "pq-only-key-exchange"
required-features = ["tls-aws-lc-rs"]
[[example]]
name = "prefer-pq-key-exchange"
required-features = ["tls-aws-lc-rs"]
[[example]]
name = "home-relay-status"
required-features = []
+49
View File
@@ -0,0 +1,49 @@
# Developing iroh
## Structured events
The library uses [tracing] both for logging and for *structured events*.
Events differ from normal logging by convention:
- The [target] is prefixed with `iroh::_events::`, with `::`-separated names.
- There is **no message**; the unique target indicates the meaning.
- The [fields] carry exclusively structured data.
- The [Level] is always `DEBUG`.
This lets automated tooling process events through custom subscribers while
still producing distinct output under the default tracing formatters, and makes
them unlikely to conflict with real modules.
An application can subscribe to the `iroh::_events` target separately. With the
default file logging it is also easy to grep for all events:
```sh
rg 'events::[a-z_\-:]+' path/to/iroh/logs/iroh.YYYY-MM-DD-NN.log
```
When adding events, aim for a high signal-to-noise ratio and design them to be
extracted automatically. To keep them distinct from normal logging, write them
with the `event!()` macro:
```rust,ignore
event!(
target: "iroh::_events::subject",
Level::DEBUG,
field = value,
);
```
## Building documentation
Building the documentation is only supported with `--all-features`. To also
document the cargo features required for certain APIs, pass the `iroh_docsrs`
cfg to rustdoc, which requires nightly Rust:
```sh
RUSTDOCFLAGS="--cfg iroh_docsrs" cargo +nightly doc --workspace --no-deps --all-features
```
[target]: https://docs.rs/tracing/latest/tracing/struct.Metadata.html#method.target
[fields]: https://docs.rs/tracing/latest/tracing/#recording-fields
[Level]: https://docs.rs/tracing/latest/tracing/struct.Level.html
[tracing]: https://docs.rs/tracing
+33
View File
@@ -0,0 +1,33 @@
Parts of the code has been derived from tailscale, which is under the following license.
Specifically the following files are most relevant
- ./src/socket**
BSD 3-Clause License
Copyright (c) 2020 Tailscale Inc & AUTHORS.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
3. Neither the name of the copyright holder nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+183
View File
@@ -0,0 +1,183 @@
<h1 align="center">
<a href="https://iroh.computer">
<img alt="iroh" src="https://raw.githubusercontent.com/n0-computer/iroh/main/.img/iroh_wordmark.svg" width="100" />
</a>
</h1>
<h3 align="center">
less net work for networks
</h3>
[![Documentation](https://img.shields.io/badge/docs-latest-blue.svg?style=flat-square)](https://docs.rs/iroh/)
[![Crates.io](https://img.shields.io/crates/v/iroh.svg?style=flat-square)](https://crates.io/crates/iroh)
[![Chat](https://img.shields.io/discord/1161119546170687619?logo=discord&style=flat-square)](https://discord.com/invite/DpmJgtU7cW)
[![Youtube](https://img.shields.io/badge/YouTube-red?logo=youtube&logoColor=white&style=flat-square)](https://www.youtube.com/@n0computer)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg?style=flat-square)](LICENSE-MIT)
[![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg?style=flat-square)](LICENSE-APACHE)
<div align="center">
<h3>
<a href="https://iroh.computer/docs">
Docs Site
</a>
<span> | </span>
<a href="https://docs.rs/iroh">
Rust Docs
</a>
</h3>
</div>
<br/>
Iroh is a Rust library to establish direct connections between endpoints.
It gives you an API for dialing by public key. You say "connect to that
endpoint", and iroh finds and maintains the best connection for you.
Under the hood iroh establishes peer-to-peer [QUIC] connections between
endpoints. The fastest route is a direct connection, so iroh tries to
[hole-punch] one whenever it can. If that fails it falls back to using
relay servers.
Because iroh is built on [QUIC], all connections are end-to-end encrypted and may
carry any number of concurrent streams. Dialing by public key also makes them mutually
authenticated, because each endpoint's public key is its TLS identity.
## Overview
An iroh endpoint is created and controlled by the [`Endpoint`]. Each endpoint
has a unique [`SecretKey`], whose public key is the endpoint's identity, the
[`EndpointId`]. Connections are authenticated against this key, which means an
[`EndpointId`] can't be impersonated.
A connection is usually established with the help of a *relay server*. When an
endpoint is created it connects to the closest relay and designates it as its
*home relay*. Other endpoints reach it first through this relay, then both
sides use QUIC NAT traversal to establish a direct connection. In the rare
cases where a direct connection is not possible, traffic keeps flowing over the
relay.
Relay servers only forward encrypted packets addressed to Endpoint IDs, they
cannot read any traffic between endpoints.
Endpoints can also connect directly without a relay, as long as the accepting
endpoint is directly reachable at one of its addresses.
To discover addressing information for an endpoint, iroh uses
[address lookup services]. With address lookup, you can connect to other
endpoints with only their [`EndpointId`]. Addressing information will then
be resolved on-demand.
The [`N0` preset] installs the DNS/Pkarr address lookup service, which uses
servers hosted by [n0] to provide global lookup for endpoints.
## Example
This is an echo protocol: the accepting side copies back whatever it receives.
The full, commented version is in [`echo.rs`](examples/echo.rs).
```rust
use iroh::{
Endpoint,
endpoint::{Connection, presets},
protocol::{AcceptError, ProtocolHandler, Router},
};
/// Each protocol is identified by its ALPN, exchanged during the handshake.
const ALPN: &[u8] = b"iroh-example/echo/0";
#[tokio::main]
async fn main() -> anyhow::Result<()> {
// The accepting side: bind an endpoint and route the ALPN to a handler.
let endpoint = Endpoint::bind(presets::N0).await?;
let router = Router::builder(endpoint.clone()).accept(ALPN, Echo).spawn();
endpoint.online().await;
// Get the endpoint's address so that we can connect to it.
let addr = endpoint.addr();
// The connecting side: dial the accepting endpoint by its address.
{
let other_endpoint = Endpoint::bind(presets::N0).await?;
let conn = other_endpoint.connect(addr, ALPN).await?;
let (mut send, mut recv) = conn.open_bi().await?;
send.write_all(b"Hello, world!").await?;
send.finish()?;
let response = recv.read_to_end(1000).await?;
assert_eq!(&response, b"Hello, world!");
conn.close(0u32.into(), b"bye!");
other_endpoint.close().await;
}
router.shutdown().await?;
Ok(())
}
#[derive(Debug, Clone)]
struct Echo;
impl ProtocolHandler for Echo {
async fn accept(&self, connection: Connection) -> Result<(), AcceptError> {
let (mut send, mut recv) = connection.accept_bi().await?;
// Echo bytes back until the sender signals the end of data.
tokio::io::copy(&mut recv, &mut send).await?;
send.finish()?;
// Wait until the other endpoint closes the connection.
connection.closed().await;
Ok(())
}
}
```
More examples live in [`iroh/examples`](examples). Run them with
`cargo run --example NAME`. Details for each are in the file itself.
## Compose protocols
Instead of writing your own, you can build on protocols that already exist on
top of iroh:
- [iroh-blobs] for [BLAKE3]-based content-addressed blob transfer, scaling from
kilobytes to terabytes.
- [iroh-gossip] for publish-subscribe overlay networks that scale down to what
an average phone can handle.
- and many more.
To use iroh from other languages, see [iroh-ffi].
## Development
For notes on iroh's structured events and how to build the documentation, see
[DEVELOPMENT.md](DEVELOPMENT.md).
# License
This project is licensed under either of
* Apache License, Version 2.0, ([LICENSE-APACHE](LICENSE-APACHE) or
https://www.apache.org/licenses/LICENSE-2.0)
* MIT license ([LICENSE-MIT](LICENSE-MIT) or
https://opensource.org/licenses/MIT)
at your option.
### Contribution
See [CONTRIBUTING.md](https://github.com/n0-computer/iroh/blob/main/CONTRIBUTING.md)
for how to get involved.
Unless you explicitly state otherwise, any contribution intentionally submitted
for inclusion in this project by you, as defined in the Apache-2.0 license,
shall be dual licensed as above, without any additional terms or conditions.
[QUIC]: https://en.wikipedia.org/wiki/QUIC
[hole-punch]: https://en.wikipedia.org/wiki/Hole_punching_(networking)
[BLAKE3]: https://github.com/BLAKE3-team/BLAKE3
[`Endpoint`]: https://docs.rs/iroh/latest/iroh/struct.Endpoint.html
[`SecretKey`]: https://docs.rs/iroh/latest/iroh/struct.SecretKey.html
[`EndpointId`]: https://docs.rs/iroh/latest/iroh/struct.EndpointId.html
[address lookup services]: https://docs.rs/iroh/latest/iroh/address_lookup/index.html
[`N0` preset]: https://docs.rs/iroh/latest/iroh/endpoint/presets/struct.N0.html
[iroh-blobs]: https://github.com/n0-computer/iroh-blobs
[iroh-gossip]: https://github.com/n0-computer/iroh-gossip
[iroh-ffi]: https://github.com/n0-computer/iroh-ffi
[n0]: https://n0.computer
+43
View File
@@ -0,0 +1,43 @@
# Bounded path retries
This is the crates.io iroh 1.2.0 source, with registry bookkeeping and its
independent Cargo.lock omitted. The workspace selects it with `[patch.crates-io]`.
- Published source commit: `17c0612f80f78f5288e97b818b1360ae6ea0a51a`.
- Crates.io archive SHA-256:
`b2f8d1cfffc83efe39a1031aab423ce09cb8048071baba550508931e9a81ce46`.
- Upstream source, manifests, examples, tests and license are otherwise retained.
`State::open_path_on_conn` appended a failed path once per connection. The actor
takes the queue every 333 ms and retries every entry on every connection to that
remote. Two failing connections therefore double duplicate work each round.
Tsunagi deliberately keeps control and data on separate QUIC connections, so
merging those connections would violate its architecture and conceal the bug.
On 2026-10-14 a Linux rc.33 daemon reached OOM after roughly five hours. Before
the kill, `/proc` showed anonymous mappings of 40 GiB and 10 GiB, later 20 GiB.
Pointers and container capacities were consistent with the retry queue and its
drained batch. The active worker's stack was in iroh's retry loop,
`open_path_on_all_conns` and `open_path_on_conn`, identified through executable
addresses and tracing metadata in the stripped static binary. This matches
[upstream issue 4509](https://github.com/n0-computer/iroh/issues/4509).
The local change extracts the enqueue operation into `pending_paths.rs`, rejects
duplicates, and limits pending candidates to 64, evicting the oldest before
admitting a new distinct path. The current retry batch and next batch are each
bounded; no wire formats, identity derivation, timers or authentication change.
The upstream deduplication fix was merged in
[PR 4551](https://github.com/n0-computer/iroh/pull/4551) on October 6. Both the
published 1.2.0 source used here and the 1.3.0 crates.io archive (released
September 28) still have the unconditional enqueue. Upgrading to 1.3.0 alone
does not fix this OOM. Remove this override when adopting a release that
includes the fix, retaining a retry regression check.
`crates/tsunagi-core/tests/iroh_path_retries.rs` includes the exact admission
module so its tests run in the default workspace suite, even though vendored
dependencies are excluded from workspace membership. The tests reproduce
multi-connection retry multiplication, preserve FIFO order under duplicate
failures, exercise success draining, and bound length and capacity under 10,000
distinct candidates. They fail with the original unconditional enqueue.
These are queue algorithm tests; the normal integration suite continues to use
real iroh endpoints, membership handshakes, SQLite and encryption.
+10
View File
@@ -0,0 +1,10 @@
use cfg_aliases::cfg_aliases;
fn main() {
// Setup cfg aliases
cfg_aliases! {
// Convenience aliases
wasm_browser: { all(target_family = "wasm", target_os = "unknown") },
with_crypto_provider: { any(feature = "tls-ring", feature = "tls-aws-lc-rs") }
}
}
+36
View File
@@ -0,0 +1,36 @@
# Using a local iroh-relay
It's easy to set up a iroh-relay that runs locally on your machine.
Using cargo:
```shell
$ cargo run --bin iroh-relay --features="iroh-relay" -- --dev
```
This will bind the iroh-relay to `[::]3340` and run it over HTTP.
To connect to this iroh-relay when doing your normal iroh commands, adjust the iroh configuration file to read:
```toml
# iroh.config.toml:
[[relays]]
url = "http://localhost:3340"
```
If you want to give a specific port for the iroh-relay to bind to, you can create a iroh-relay config file and pass that file in using the `--config_path` flag. You need to retain a `secret_key`, so it is recommended to run `iroh-relay --config-path [PATH]` once to generate a secret key and save it to the config file before doing further edits to the file.
To change the port you want to listen on, change the port in the `addr` field:
```
# iroh-relay.toml
secret_key = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
addr = "[::]:12345"
hostname = "my.relay.network"
enable_relay = true
```
Check [the iroh-relay file's](../src/bin/iroh-relay.rs) `Config` struct for documentation on each configuration field.
If you change the local iroh-relay server's configuration, however, be sure to adjust the associated fields in your iroh config as well.
+9
View File
@@ -0,0 +1,9 @@
# Relays
When an Iroh endpoint starts up, it does a latency test to see which known relay endpoint it is “closest to”. That relay server is considered the Iroh endpoint's home relay server.
An endpoint may be connected to multiple relay servers, but it will advertise its home relay endpoint as the one best used to hole-punch or relay packets through.
You do not need to know an endpoint's relay server in order to connect to them directly, if there are no firewalls or NATs between the two endpoints trying to connect. However, to have any hole punching, you must know at least one relay server to which that endpoint is connected.
We currently run 3 relays.
+181
View File
@@ -0,0 +1,181 @@
use std::{env, str::FromStr, time::Instant};
use clap::Parser;
use data_encoding::HEXLOWER;
use iroh::{
EndpointId, SecretKey,
endpoint::{RecvStream, SendStream, ZeroRttStatus, presets},
};
use n0_error::{Result, StackResultExt, StdResultExt};
use n0_future::StreamExt;
use tracing::{info, trace};
const PINGPONG_ALPN: &[u8] = b"0rtt-pingpong";
#[derive(Parser)]
struct Args {
/// The endpoint id to connect to. If not set, the program will start a server.
endpoint_id: Option<EndpointId>,
/// Number of rounds to run.
#[clap(long, default_value = "100")]
rounds: u64,
/// Run without 0-RTT for comparison.
#[clap(long)]
disable_0rtt: bool,
}
/// Gets a secret key from the IROH_SECRET environment variable or generates a new random one.
/// If the environment variable is set, it must be a valid string representation of a secret key.
pub fn get_or_generate_secret_key() -> Result<SecretKey> {
if let Ok(secret) = env::var("IROH_SECRET") {
// Parse the secret key from string
SecretKey::from_str(&secret).context("Invalid secret key format")
} else {
// Generate a new random key
let secret_key = SecretKey::generate();
println!(
"Generated new secret key: {}",
HEXLOWER.encode(&secret_key.to_bytes())
);
println!("To reuse this key, set the IROH_SECRET environment variable to this value");
Ok(secret_key)
}
}
/// Do a simple ping-pong with the given connection.
async fn pingpong(send: SendStream, recv: RecvStream, x: u64) -> Result<()> {
ping(send, x).await?;
pong(recv, x).await
}
async fn ping(mut send: SendStream, x: u64) -> Result<()> {
let data = x.to_be_bytes();
send.write_all(&data).await.anyerr()?;
send.finish().anyerr()
}
async fn pong(mut recv: RecvStream, x: u64) -> Result<()> {
let data = x.to_be_bytes();
let echo = recv.read_to_end(8).await.anyerr()?;
assert!(echo == data);
Ok(())
}
async fn connect(args: Args) -> Result<()> {
let remote_id = args.endpoint_id.unwrap();
let endpoint = iroh::Endpoint::builder(presets::N0)
.relay_mode(iroh::RelayMode::Disabled)
.keylog(true)
.bind()
.await?;
// ensure we have resolved the remote_id before connecting
// so we get a more accurate connection timing
let mut address_lookup_stream = endpoint.address_lookup()?.resolve(remote_id);
let _item = address_lookup_stream
.next()
.await
.context("failed to lookup remote")?;
let t0 = Instant::now();
for i in 0..args.rounds {
let t0 = Instant::now();
let connecting = endpoint
.connect_with_opts(remote_id, PINGPONG_ALPN, Default::default())
.await?;
let connection = if args.disable_0rtt {
let connection = connecting.await.anyerr()?;
trace!("connecting without 0-RTT");
let (send, recv) = connection.open_bi().await.anyerr()?;
pingpong(send, recv, i).await?;
connection
} else {
match connecting.into_0rtt() {
Ok(zrtt_connection) => {
trace!("0-RTT possible from our side");
let (send, recv) = zrtt_connection.open_bi().await.anyerr()?;
// before we get the full handshake, attempt to send 0-RTT data
let zrtt_task = tokio::spawn(ping(send, i));
match zrtt_connection.handshake_completed().await? {
ZeroRttStatus::Accepted(conn) => {
let _ = zrtt_task.await.anyerr()?;
pong(recv, i).await?;
conn
}
ZeroRttStatus::Rejected(conn) => {
zrtt_task.abort();
let (send, recv) = conn.open_bi().await.anyerr()?;
pingpong(send, recv, i).await?;
conn
}
}
}
Err(connecting) => {
trace!("0-RTT not possible from our side");
let conn = connecting.await.anyerr()?;
let (send, recv) = conn.open_bi().await.anyerr()?;
pingpong(send, recv, i).await?;
conn
}
}
};
connection.close(0u8.into(), b"");
let elapsed = t0.elapsed();
println!("round {i}: {} us", elapsed.as_micros());
}
let elapsed = t0.elapsed();
println!("total time: {} us", elapsed.as_micros());
println!(
"time per round: {} us",
elapsed.as_micros() / (args.rounds as u128)
);
Ok(())
}
async fn accept(_args: Args) -> Result<()> {
let secret_key = get_or_generate_secret_key()?;
let endpoint = iroh::Endpoint::builder(presets::N0)
.alpns(vec![PINGPONG_ALPN.to_vec()])
.secret_key(secret_key)
.relay_mode(iroh::RelayMode::Disabled)
.bind()
.await?;
println!("endpoint id: {}", endpoint.id());
let accept = async move {
while let Some(incoming) = endpoint.accept().await {
tokio::spawn(async move {
let accepting = incoming.accept().anyerr()?;
let connection = accepting.into_0rtt();
let (mut send, mut recv) = connection.accept_bi().await.anyerr()?;
trace!("recv.is_0rtt: {}", recv.is_0rtt());
let data = recv.read_to_end(8).await.anyerr()?;
trace!("recv: {}", data.len());
send.write_all(&data).await.anyerr()?;
send.finish().anyerr()?;
connection.closed().await;
n0_error::Ok(())
});
}
};
tokio::select! {
_ = accept => {
info!("accept finished, shutting down");
},
_ = tokio::signal::ctrl_c()=> {
info!("Ctrl-C received, shutting down");
}
}
Ok(())
}
#[tokio::main]
async fn main() -> Result<()> {
tracing_subscriber::fmt::init();
let args = Args::parse();
if args.endpoint_id.is_some() {
connect(args).await?;
} else {
accept(args).await?;
};
Ok(())
}
+355
View File
@@ -0,0 +1,355 @@
//! Implementation of authentication using iroh hooks
//!
//! This implements an auth protocol that works with iroh hooks.
//! It allows to put authentication in front of iroh protocols. The protocols don't need any special support.
//! Authentication is handled prior to establishing the connections, over a separate connection.
use iroh::{Endpoint, EndpointAddr, endpoint::presets, protocol::Router};
use n0_error::{Result, StdResultExt};
use crate::echo::Echo;
#[tokio::main]
async fn main() -> Result<()> {
tracing_subscriber::fmt::init();
let server_router = accept_side(b"secret!!").await?;
server_router.endpoint().online().await;
let server_addr = server_router.endpoint().addr();
println!("-- no --");
let res = connect_side_no_auth(server_addr.clone()).await;
println!("echo without auth: {:#}", res.unwrap_err());
println!("-- wrong --");
let res = connect_side(server_addr.clone(), b"dunno").await;
println!("echo with wrong auth: {:#}", res.unwrap_err());
println!("-- correct --");
let res = connect_side(server_addr.clone(), b"secret!!").await;
println!("echo with correct auth: {res:?}");
server_router.shutdown().await.anyerr()?;
Ok(())
}
async fn connect_side(remote_addr: EndpointAddr, token: &[u8]) -> Result<()> {
let (auth_hook, auth_task) = auth::outgoing(token.to_vec());
let endpoint = Endpoint::builder(presets::N0)
.hooks(auth_hook)
.bind()
.await?;
let _guard = auth_task.spawn(endpoint.clone());
Echo::connect(&endpoint, remote_addr, b"hello there!").await
}
async fn connect_side_no_auth(remote_addr: EndpointAddr) -> Result<()> {
let endpoint = Endpoint::bind(presets::N0).await?;
Echo::connect(&endpoint, remote_addr, b"hello there!").await
}
async fn accept_side(token: &[u8]) -> Result<Router> {
let (auth_hook, auth_protocol) = auth::incoming(token.to_vec());
let endpoint = Endpoint::builder(presets::N0)
.hooks(auth_hook)
.bind()
.await?;
let router = Router::builder(endpoint)
.accept(auth::ALPN, auth_protocol)
.accept(echo::ALPN, Echo)
.spawn();
Ok(router)
}
mod echo {
//! A bare-bones protocol with no knowledge of auth whatsoever.
use iroh::{
Endpoint, EndpointAddr,
endpoint::Connection,
protocol::{AcceptError, ProtocolHandler},
};
use n0_error::{Result, StdResultExt, anyerr};
#[derive(Debug, Clone)]
pub struct Echo;
pub const ALPN: &[u8] = b"iroh-example/echo/0";
impl Echo {
pub async fn connect(
endpoint: &Endpoint,
remote: impl Into<EndpointAddr>,
message: &[u8],
) -> Result<()> {
let conn = endpoint.connect(remote, ALPN).await?;
let (mut send, mut recv) = conn.open_bi().await.anyerr()?;
send.write_all(message).await.anyerr()?;
send.finish().anyerr()?;
let response = recv.read_to_end(1000).await.anyerr()?;
conn.close(0u32.into(), b"bye!");
if response == message {
Ok(())
} else {
Err(anyerr!("Received invalid response"))
}
}
}
impl ProtocolHandler for Echo {
async fn accept(&self, connection: Connection) -> Result<(), AcceptError> {
let (mut send, mut recv) = connection.accept_bi().await?;
tokio::io::copy(&mut recv, &mut send).await?;
send.finish()?;
connection.closed().await;
Ok(())
}
}
}
mod auth {
//! Authentication hook
use std::{
collections::{HashMap, HashSet, hash_map},
sync::{Arc, Mutex},
};
use iroh::{
Endpoint, EndpointAddr, EndpointId,
endpoint::{
AfterHandshakeOutcome, BeforeConnectOutcome, Connection, ConnectionError, EndpointHooks,
},
protocol::{AcceptError, ProtocolHandler},
};
use n0_error::{AnyError, Result, StackResultExt, StdResultExt, anyerr};
use n0_future::task::AbortOnDropHandle;
use tokio::{
sync::{mpsc, oneshot},
task::JoinSet,
};
use tracing::debug;
pub const ALPN: &[u8] = b"iroh-example/auth/0";
const CLOSE_ACCEPTED: u32 = 1;
const CLOSE_DENIED: u32 = 403;
/// Outgoing side: Use this if you want to pre-auth outgoing connections.
pub fn outgoing(token: Vec<u8>) -> (OutgoingAuthHook, OutgoingAuthTask) {
let (tx, rx) = mpsc::channel(16);
let hook = OutgoingAuthHook { tx };
let connector = OutgoingAuthTask {
token,
rx,
allowed_remotes: Default::default(),
pending_remotes: Default::default(),
tasks: JoinSet::new(),
};
(hook, connector)
}
type AuthResult = Result<(), Arc<AnyError>>;
/// Hook to mount on the endpoint builder.
#[derive(Debug)]
pub struct OutgoingAuthHook {
tx: mpsc::Sender<(EndpointId, oneshot::Sender<AuthResult>)>,
}
impl OutgoingAuthHook {
async fn authenticate(&self, remote_id: EndpointId) -> Result<()> {
let (tx, rx) = oneshot::channel();
self.tx
.send((remote_id, tx))
.await
.std_context("authenticator stopped")?;
rx.await
.std_context("authenticator stopped")?
.context("failed to authenticate")
}
}
impl EndpointHooks for OutgoingAuthHook {
async fn before_connect<'a>(
&'a self,
remote_addr: &'a EndpointAddr,
alpn: &'a [u8],
) -> BeforeConnectOutcome {
// Don't intercept auth request themsevles
if alpn == ALPN {
BeforeConnectOutcome::Accept
} else {
match self.authenticate(remote_addr.id).await {
Ok(()) => BeforeConnectOutcome::Accept,
Err(err) => {
debug!("authentication denied: {err:#}");
BeforeConnectOutcome::Reject
}
}
}
}
}
/// Connector task that initiates pre-auth request. Call [`Self::spawn`] once the endpoint is built.
pub struct OutgoingAuthTask {
token: Vec<u8>,
rx: mpsc::Receiver<(EndpointId, oneshot::Sender<AuthResult>)>,
allowed_remotes: HashSet<EndpointId>,
pending_remotes: HashMap<EndpointId, Vec<oneshot::Sender<AuthResult>>>,
tasks: JoinSet<(EndpointId, Result<()>)>,
}
impl OutgoingAuthTask {
pub fn spawn(self, endpoint: Endpoint) -> AbortOnDropHandle<()> {
AbortOnDropHandle::new(tokio::spawn(self.run(endpoint)))
}
async fn run(mut self, endpoint: Endpoint) {
loop {
tokio::select! {
msg = self.rx.recv() => {
let Some((remote_id, tx)) = msg else {
break;
};
self.handle_msg(&endpoint, remote_id, tx);
}
Some(res) = self.tasks.join_next(), if !self.tasks.is_empty() => {
let (remote_id, res) = res.expect("connect task panicked");
let res = res.map_err(Arc::new);
self.handle_task(remote_id, res);
}
}
}
}
fn handle_msg(
&mut self,
endpoint: &Endpoint,
remote_id: EndpointId,
tx: oneshot::Sender<Result<(), Arc<AnyError>>>,
) {
if self.allowed_remotes.contains(&remote_id) {
tx.send(Ok(())).ok();
} else {
match self.pending_remotes.entry(remote_id) {
hash_map::Entry::Occupied(mut entry) => {
entry.get_mut().push(tx);
}
hash_map::Entry::Vacant(entry) => {
let endpoint = endpoint.clone();
let token = self.token.clone();
self.tasks.spawn(async move {
let res = Self::connect(endpoint, remote_id, token).await;
(remote_id, res)
});
entry.insert(vec![tx]);
}
}
}
}
fn handle_task(&mut self, remote_id: EndpointId, res: Result<(), Arc<AnyError>>) {
if res.is_ok() {
self.allowed_remotes.insert(remote_id);
}
let senders = self.pending_remotes.remove(&remote_id);
for tx in senders.into_iter().flatten() {
tx.send(res.clone()).ok();
}
}
async fn connect(endpoint: Endpoint, remote_id: EndpointId, token: Vec<u8>) -> Result<()> {
let conn = endpoint.connect(remote_id, ALPN).await?;
let mut stream = conn.open_uni().await.anyerr()?;
stream.write_all(&token).await.anyerr()?;
stream.finish().anyerr()?;
let reason = conn.closed().await;
if let ConnectionError::ApplicationClosed(code) = &reason
&& code.error_code.into_inner() as u32 == CLOSE_ACCEPTED
{
Ok(())
} else if let ConnectionError::ApplicationClosed(code) = &reason
&& code.error_code.into_inner() as u32 == CLOSE_DENIED
{
Err(anyerr!("authentication denied by remote"))
} else {
Err(AnyError::from_std(reason))
}
}
}
/// Incoming side: Use this if you want to only accept connections from peers with successful pre-auth requests.
pub fn incoming(token: Vec<u8>) -> (IncomingAuthHook, AuthProtocol) {
let allowed_remotes: Arc<Mutex<HashSet<EndpointId>>> = Default::default();
let hook = IncomingAuthHook {
allowed_remotes: allowed_remotes.clone(),
};
let protocol = AuthProtocol {
allowed_remotes,
token,
};
(hook, protocol)
}
/// Accept-side auth hook: Mount this onto the endpoint.
///
/// This will reject incoming connections if the remote did not successfully authenticate before.
#[derive(Debug)]
pub struct IncomingAuthHook {
allowed_remotes: Arc<Mutex<HashSet<EndpointId>>>,
}
impl EndpointHooks for IncomingAuthHook {
async fn after_handshake<'a>(
&'a self,
conn: &'a iroh::endpoint::Connection,
) -> AfterHandshakeOutcome {
if conn.alpn() == ALPN
|| self
.allowed_remotes
.lock()
.expect("poisoned")
.contains(&conn.remote_id())
{
AfterHandshakeOutcome::Accept
} else {
AfterHandshakeOutcome::Reject {
error_code: 403u32.into(),
reason: b"not authenticated".to_vec(),
}
}
}
}
/// Accept-side auth protocol. Mount this on the router to accept authentication requests.
#[derive(Debug, Clone)]
pub struct AuthProtocol {
token: Vec<u8>,
allowed_remotes: Arc<Mutex<HashSet<EndpointId>>>,
}
impl ProtocolHandler for AuthProtocol {
/// The `accept` method is called for each incoming connection for our ALPN.
///
/// The returned future runs on a newly spawned tokio task, so it can run as long as
/// the connection lasts.
async fn accept(&self, connection: Connection) -> Result<(), AcceptError> {
let mut stream = connection.accept_uni().await?;
let token = stream.read_to_end(256).await.anyerr()?;
let remote_id = connection.remote_id();
if token == self.token {
self.allowed_remotes
.lock()
.expect("poisoned")
.insert(remote_id);
connection.close(CLOSE_ACCEPTED.into(), b"accepted");
} else {
connection.close(CLOSE_DENIED.into(), b"rejected");
}
Ok(())
}
}
}
+95
View File
@@ -0,0 +1,95 @@
//! The smallest example showing how to use iroh and [`iroh::Endpoint`] to connect to a remote endpoint and pass bytes using unreliable datagrams.
//!
//! We use the endpoint ID (the PublicKey of the remote endpoint), the direct UDP addresses, and the relay url to achieve a connection.
//!
//! This example uses the default relay servers to attempt to holepunch, and will use that relay server to relay packets if the two devices cannot establish a direct UDP connection.
//!
//! Run the `listen-unreliable` example first (`iroh/examples/listen-unreliable.rs`), which will give you instructions on how to run this example to watch two endpoints connect and exchange bytes.
use std::net::SocketAddr;
use clap::Parser;
use iroh::{Endpoint, EndpointAddr, RelayMode, RelayUrl, SecretKey, endpoint::presets};
use iroh_base::TransportAddr;
use n0_error::{Result, StdResultExt};
use tracing::info;
// An example ALPN that we are using to communicate over the `Endpoint`
const EXAMPLE_ALPN: &[u8] = b"n0/iroh/examples/0";
#[derive(Debug, Parser)]
struct Cli {
/// The id of the remote endpoint.
#[clap(long)]
endpoint_id: iroh::EndpointId,
/// The list of direct UDP addresses for the remote endpoint.
#[clap(long, value_parser, num_args = 1.., value_delimiter = ' ')]
addrs: Vec<SocketAddr>,
/// The url of the relay server the remote endpoint can also be reached at.
#[clap(long)]
relay_url: RelayUrl,
}
#[tokio::main]
async fn main() -> Result<()> {
tracing_subscriber::fmt::init();
println!("\nconnect (unreliable) example!\n");
let args = Cli::parse();
let secret_key = SecretKey::from_bytes(&rand::random());
println!("public key: {}", secret_key.public());
// Build a `Endpoint`, which uses PublicKeys as endpoint identifiers, uses QUIC for directly connecting to other endpoints, and uses the relay protocol and relay servers to holepunch direct connections between endpoints when there are NATs or firewalls preventing direct connections. If no direct connection can be made, packets are relayed over the relay servers.
let endpoint = Endpoint::builder(presets::N0)
// The secret key is used to authenticate with other endpoints. The PublicKey portion of this secret key is how we identify endpoints, often referred to as the `endpoint_id` in our codebase.
.secret_key(secret_key)
// Set the ALPN protocols this endpoint will accept on incoming connections
.alpns(vec![EXAMPLE_ALPN.to_vec()])
// `RelayMode::Default` means that we will use the default relay servers to holepunch and relay.
// Use `RelayMode::Custom` to pass in a `RelayMap` with custom relay urls.
// Use `RelayMode::Disable` to disable holepunching and relaying over HTTPS
// If you want to experiment with relaying using your own relay server, you must pass in the same custom relay url to both the `listen` code AND the `connect` code
.relay_mode(RelayMode::Default)
// You can choose an address to bind to, but passing in `None` will bind the socket to a random available port
.bind()
.await?;
// wait for the endpoint to be online
endpoint.online().await;
let endpoint_addr = endpoint.addr();
let me = endpoint_addr.id;
println!("endpoint id: {me}");
println!("endpoint listening addresses:");
endpoint_addr
.ip_addrs()
.for_each(|addr| println!("\t{addr}"));
let relay_url = endpoint_addr
.relay_urls()
.next()
.expect("Should have a relay URL, assuming a default endpoint setup.");
println!("endpoint relay server url: {relay_url}\n");
// Build a `EndpointAddr` from the endpoint_id, relay url, and UDP addresses.
let addrs = args
.addrs
.into_iter()
.map(TransportAddr::Ip)
.chain(std::iter::once(TransportAddr::Relay(args.relay_url)));
let addr = EndpointAddr::from_parts(args.endpoint_id, addrs);
// Attempt to connect, over the given ALPN.
// Returns a QUIC connection.
let conn = endpoint.connect(addr, EXAMPLE_ALPN).await?;
info!("connected");
// Send a datagram over the connection.
let message = format!("{me} is saying 'hello!'");
conn.send_datagram(message.as_bytes().to_vec().into())
.anyerr()?;
// Read a datagram over the connection.
let message = conn.read_datagram().await.anyerr()?;
let message = String::from_utf8(message.into()).anyerr()?;
println!("received: {message}");
Ok(())
}
+101
View File
@@ -0,0 +1,101 @@
//! The smallest example showing how to use iroh and [`iroh::Endpoint`] to connect to a remote endpoint.
//!
//! We use the endpoint ID (the PublicKey of the remote endpoint), the direct UDP addresses, and the relay url to achieve a connection.
//!
//! This example uses the default relay servers to attempt to holepunch, and will use that relay server to relay packets if the two devices cannot establish a direct UDP connection.
//!
//! Run the `listen` example first (`iroh/examples/listen.rs`), which will give you instructions on how to run this example to watch two endpoints connect and exchange bytes.
use std::net::SocketAddr;
use clap::Parser;
use iroh::{
Endpoint, EndpointAddr, RelayMode, RelayUrl, SecretKey, TransportAddr, endpoint::presets,
};
use n0_error::{Result, StdResultExt};
use tracing::info;
// An example ALPN that we are using to communicate over the `Endpoint`
const EXAMPLE_ALPN: &[u8] = b"n0/iroh/examples/0";
#[derive(Debug, Parser)]
struct Cli {
/// The id of the remote endpoint.
#[clap(long)]
endpoint_id: iroh::EndpointId,
/// The list of direct UDP addresses for the remote endpoint.
#[clap(long, value_parser, num_args = 1.., value_delimiter = ' ')]
addrs: Vec<SocketAddr>,
/// The url of the relay server the remote endpoint can also be reached at.
#[clap(long)]
relay_url: RelayUrl,
}
#[tokio::main]
async fn main() -> Result<()> {
tracing_subscriber::fmt::init();
println!("\nconnect example!\n");
let args = Cli::parse();
let secret_key = SecretKey::generate();
println!("public key: {}", secret_key.public());
// Build a `Endpoint`, which uses PublicKeys as endpoint identifiers, uses QUIC for directly connecting to other endpoints, and uses the relay protocol and relay servers to holepunch direct connections between endpoints when there are NATs or firewalls preventing direct connections. If no direct connection can be made, packets are relayed over the relay servers.
let endpoint = Endpoint::builder(presets::N0)
// The secret key is used to authenticate with other endpoints. The PublicKey portion of this secret key is how we identify endpoints, often referred to as the `endpoint_id` in our codebase.
.secret_key(secret_key)
// Set the ALPN protocols this endpoint will accept on incoming connections
.alpns(vec![EXAMPLE_ALPN.to_vec()])
// `RelayMode::Default` means that we will use the default relay servers to holepunch and relay.
// Use `RelayMode::Custom` to pass in a `RelayMap` with custom relay urls.
// Use `RelayMode::Disable` to disable holepunching and relaying over HTTPS
// If you want to experiment with relaying using your own relay server, you must pass in the same custom relay url to both the `listen` code AND the `connect` code
.relay_mode(RelayMode::Default)
// You can choose an address to bind to, but passing in `None` will bind the socket to a random available port
.bind()
.await?;
// wait for the endpoint to be online
endpoint.online().await;
let endpoint_addr = endpoint.addr();
let me = endpoint.id();
println!("endpoint id: {me}");
println!("endpoint listening addresses:");
for addr in endpoint_addr.ip_addrs() {
println!("\t{addr}")
}
let relay_url = endpoint_addr
.relay_urls()
.next()
.expect("should be connected to a relay server");
println!("endpoint relay server url: {relay_url}\n");
// Build a `EndpointAddr` from the endpoint_id, relay url, and UDP addresses.
let addrs = args
.addrs
.into_iter()
.map(TransportAddr::Ip)
.chain(std::iter::once(TransportAddr::Relay(args.relay_url)));
let addr = EndpointAddr::from_parts(args.endpoint_id, addrs);
// Attempt to connect, over the given ALPN.
// Returns a Noq connection.
let conn = endpoint.connect(addr, EXAMPLE_ALPN).await?;
info!("connected");
// Use the Noq API to send and recv content.
let (mut send, mut recv) = conn.open_bi().await.anyerr()?;
let message = format!("{me} is saying 'hello!'");
send.write_all(message.as_bytes()).await.anyerr()?;
// Call `finish` to close the send side of the connection gracefully.
send.finish().anyerr()?;
let message = recv.read_to_end(100).await.anyerr()?;
let message = String::from_utf8(message).anyerr()?;
println!("received: {message}");
// We received the last message: close all connections and allow for the close
// message to be sent.
endpoint.close().await;
Ok(())
}
+205
View File
@@ -0,0 +1,205 @@
use std::{sync::Arc, time::Duration};
use clap::Parser;
use iroh::{
Endpoint, SecretKey, TransportAddr,
endpoint::{
Builder, Connection, presets,
transports::{Addr, PathSelection, PathSelectionContext, PathSelector},
},
protocol::{AcceptError, ProtocolHandler, Router},
test_utils::test_transport::{TEST_TRANSPORT_ID, TestNetwork, TestTransport},
};
use n0_error::{Result, StdResultExt};
/// Each protocol is identified by its ALPN string.
///
/// The ALPN, or application-layer protocol negotiation, is exchanged in the connection handshake,
/// and the connection is aborted unless both endpoints pass the same bytestring.
const ALPN: &[u8] = b"iroh-example/echo/0";
/// Example demonstrating custom transport usage.
#[derive(Parser, Debug, Clone)]
struct Args {
/// Keep IP transports enabled (in addition to custom transport)
#[arg(long)]
keep_ip: bool,
/// Keep relay transports enabled (in addition to custom transport)
#[arg(long)]
keep_relay: bool,
/// Delay in seconds to wait after connecting before re-checking the selected transport
#[arg(long, default_value = "0")]
delay: u64,
}
/// A [`PathSelector`] that prefers the test custom transport whenever a candidate path on
/// it exists, falling back to the lowest-RTT candidate otherwise.
#[derive(Debug)]
struct PreferTestTransport;
impl PathSelector for PreferTestTransport {
fn select(&self, ctx: &PathSelectionContext<'_>) -> PathSelection {
tracing::debug!("dumping path RTTs");
for p in ctx.paths() {
let network_path = p.network_path();
let rtt = p.stats().map(|s| s.rtt);
tracing::debug!(%network_path, ?rtt);
}
let mut selection = PathSelection::none();
// First preference: any path on our test custom transport.
if let Some(p) = ctx.paths().find(
|p| matches!(p.network_path().remote(), Addr::Custom(c) if c.id() == TEST_TRANSPORT_ID),
) {
selection.set(&p);
return selection;
}
// Otherwise: lowest RTT wins. Paths whose stats can't be read (closed
// concurrently with selection) are skipped entirely.
if let Some(p) = ctx
.paths()
.filter_map(|p| p.stats().map(|s| (p, s.rtt)))
.min_by_key(|(_, rtt)| *rtt)
.map(|(p, _)| p)
{
selection.set(&p);
}
selection
}
}
impl Args {
/// Configure an endpoint builder with the custom transport and optional IP/relay transports.
fn configure(&self, secret_key: SecretKey, transport: Arc<TestTransport>) -> Builder {
let mut builder = Endpoint::builder(presets::N0)
.secret_key(secret_key)
.preset(transport)
// Always prefer the custom transport when it has a working path.
.path_selector(Arc::new(PreferTestTransport));
if !self.keep_ip {
builder = builder.clear_ip_transports();
}
if !self.keep_relay {
builder = builder.clear_relay_transports();
}
builder
}
}
#[derive(Debug, Clone)]
struct Echo;
impl ProtocolHandler for Echo {
/// The `accept` method is called for each incoming connection for our ALPN.
///
/// The returned future runs on a newly spawned tokio task, so it can run as long as
/// the connection lasts.
async fn accept(&self, connection: Connection) -> Result<(), AcceptError> {
// We can get the remote's endpoint id from the connection.
let endpoint_id = connection.remote_id();
println!("accepted connection from {endpoint_id}");
// Our protocol is a simple request-response protocol, so we expect the
// connecting peer to open a single bi-directional stream.
let (mut send, mut recv) = connection.accept_bi().await?;
// Echo any bytes received back directly.
// This will keep copying until the sender signals the end of data on the stream.
let bytes_sent = tokio::io::copy(&mut recv, &mut send).await?;
println!("Copied over {bytes_sent} byte(s)");
// By calling `finish` on the send stream we signal that we will not send anything
// further, which makes the receive stream on the other end terminate.
send.finish()?;
// Wait until the remote closes the connection, which it does once it
// received the response.
connection.closed().await;
Ok(())
}
}
#[tokio::main]
async fn main() -> Result<()> {
tracing_subscriber::fmt::init();
let args = Args::parse();
println!(
"Config: keep_ip={}, keep_relay={}, delay={}s",
args.keep_ip, args.keep_relay, args.delay
);
let network = TestNetwork::new();
let s1 = SecretKey::from([0u8; 32]);
let s2 = SecretKey::from([1u8; 32]);
// Create transports and configure builders with transport + address lookup
let t1 = network.create_transport(s1.public())?;
let ep1 = args.configure(s1.clone(), t1).bind().await?;
let t2 = network.create_transport(s2.public())?;
let ep2 = args.configure(s2.clone(), t2).bind().await?;
println!("ep2 addr: {:?}", ep2.addr());
let server = Router::builder(ep2).accept(ALPN, Echo).spawn();
// Connect using just the endpoint ID - discovery will resolve addresses
// Note: The test network's discovery is very fast (in-memory), so the custom
// transport address is available immediately and wins before IP discovery runs.
println!("Connecting to: {:?}", s2.public());
let conn = ep1.connect(s2.public(), ALPN).await?;
// Helper to print paths and verify test transport is selected
let verify_test_transport = |label: &str| {
let paths = conn.paths();
println!("Paths {}:", label);
for path in paths.iter() {
println!(
" {} selected={} rtt={:?}",
path.remote_addr(),
path.is_selected(),
path.rtt()
);
}
let selected_path = paths.iter().find(|p| p.is_selected());
let is_test_transport = selected_path.as_ref().is_some_and(|p| {
matches!(p.remote_addr(), TransportAddr::Custom(addr) if addr.id() == TEST_TRANSPORT_ID)
});
assert!(
is_test_transport,
"Expected test transport (id={}) to be selected {}, got: {:?}",
TEST_TRANSPORT_ID,
label,
selected_path.as_ref().map(|p| p.remote_addr())
);
println!(
"Verified: test transport (id={}) is selected {}",
TEST_TRANSPORT_ID, label
);
};
// Verify test transport is selected immediately after connecting
verify_test_transport("immediately after connecting");
// If a delay is specified, wait and then re-check to see if the transport is still selected
// after other discovery mechanisms have had time to run.
if args.delay > 0 {
println!(
"Waiting {}s to let other discovery mechanisms run...",
args.delay
);
tokio::time::sleep(Duration::from_secs(args.delay)).await;
verify_test_transport(&format!("after {}s delay", args.delay));
}
let (mut send, mut recv) = conn.open_bi().await.anyerr()?;
send.write_all(b"Hello custom transport!").await.anyerr()?;
send.finish().anyerr()?;
let response = recv.read_to_end(1000).await.anyerr()?;
assert_eq!(&response, b"Hello custom transport!");
conn.close(0u32.into(), b"bye!");
server.shutdown().await.anyerr()?;
drop(server);
Ok(())
}
+121
View File
@@ -0,0 +1,121 @@
//! Very basic example showing how to implement a basic echo protocol,
//! without using the `Router` API. (For the router version, check out the echo.rs example.)
//!
//! The echo protocol echos any data sent to it in the first stream.
//!
//! ## Running the Example
//!
//! cargo run --example echo-no-router
use iroh::{Endpoint, EndpointAddr, endpoint::presets};
use n0_error::{AnyError as Error, Result, StdResultExt};
/// Each protocol is identified by its ALPN string.
///
/// The ALPN, or application-layer protocol negotiation, is exchanged in the connection handshake,
/// and the connection is aborted unless both endpoints pass the same bytestring.
const ALPN: &[u8] = b"iroh-example/echo/0";
#[tokio::main]
async fn main() -> Result<()> {
let endpoint = start_accept_side().await?;
// wait for the endpoint to be online
endpoint.online().await;
connect_side(endpoint.addr()).await?;
// This makes sure the endpoint is closed properly and connections close gracefully
// and will indirectly close the tasks spawned by `start_accept_side`.
endpoint.close().await;
Ok(())
}
async fn connect_side(addr: EndpointAddr) -> Result<()> {
let endpoint = Endpoint::bind(presets::N0).await?;
// Open a connection to the accepting endpoint
let conn = endpoint.connect(addr, ALPN).await?;
// Open a bidirectional QUIC stream
let (mut send, mut recv) = conn.open_bi().await.anyerr()?;
// Send some data to be echoed
send.write_all(b"Hello, world!").await.anyerr()?;
// Signal the end of data for this particular stream
send.finish().anyerr()?;
// Receive the echo, but limit reading up to maximum 1000 bytes
let response = recv.read_to_end(1000).await.anyerr()?;
assert_eq!(&response, b"Hello, world!");
// Explicitly close the whole connection.
conn.close(0u32.into(), b"bye!");
// The above call only queues a close message to be sent (see how it's not async!).
// We need to actually call this to make sure this message is sent out.
endpoint.close().await;
// If we don't call this, but continue using the endpoint, then the queued
// close call will eventually be picked up and sent.
// But always try to wait for endpoint.close().await to go through before dropping
// the endpoint to ensure any queued messages are sent through and connections are
// closed gracefully.
Ok(())
}
async fn start_accept_side() -> Result<Endpoint> {
let endpoint = Endpoint::builder(presets::N0)
// The accept side needs to opt-in to the protocols it accepts,
// as any connection attempts that can't be found with a matching ALPN
// will be rejected.
.alpns(vec![ALPN.to_vec()])
.bind()
.await?;
// spawn a task so that `start_accept_side` returns immediately and we can continue in main().
tokio::spawn({
let endpoint = endpoint.clone();
async move {
// This task won't leak, because we call `endpoint.close()` in `main()`,
// which causes `endpoint.accept().await` to return `None`.
// In a more serious environment, we recommend avoiding `tokio::spawn` and use either a `TaskTracker` or
// `JoinSet` instead to make sure you're not accidentally leaking tasks.
while let Some(incoming) = endpoint.accept().await {
// spawn a task for each incoming connection, so we can serve multiple connections asynchronously
tokio::spawn(async move {
let connection = incoming.await.anyerr()?;
// We can get the remote's endpoint id from the connection.
let endpoint_id = connection.remote_id();
println!("accepted connection from {endpoint_id}");
// Our protocol is a simple request-response protocol, so we expect the
// connecting peer to open a single bi-directional stream.
let (mut send, mut recv) = connection.accept_bi().await.anyerr()?;
// Echo any bytes received back directly.
// This will keep copying until the sender signals the end of data on the stream.
let bytes_sent = tokio::io::copy(&mut recv, &mut send).await.anyerr()?;
println!("Copied over {bytes_sent} byte(s)");
// By calling `finish` on the send stream we signal that we will not send anything
// further, which makes the receive stream on the other end terminate.
send.finish().anyerr()?;
// Wait until the remote closes the connection, which it does once it
// received the response.
connection.closed().await;
Ok::<_, Error>(())
});
}
Ok::<_, Error>(())
}
});
Ok(endpoint)
}
+112
View File
@@ -0,0 +1,112 @@
//! Very basic example to showcase how to use iroh's APIs.
//!
//! This example implements a simple protocol that echos any data sent to it in the first stream.
//!
//! ## Usage
//!
//! cargo run --example echo
use iroh::{
Endpoint, EndpointAddr,
endpoint::{Connection, presets},
protocol::{AcceptError, ProtocolHandler, Router},
};
use n0_error::{Result, StdResultExt};
/// Each protocol is identified by its ALPN string.
///
/// The ALPN, or application-layer protocol negotiation, is exchanged in the connection handshake,
/// and the connection is aborted unless both endpoints pass the same bytestring.
const ALPN: &[u8] = b"iroh-example/echo/0";
#[tokio::main]
async fn main() -> Result<()> {
tracing_subscriber::fmt::init();
let router = start_accept_side().await?;
// wait for the endpoint to be online
router.endpoint().online().await;
connect_side(router.endpoint().addr()).await?;
// This makes sure the endpoint in the router is closed properly and connections close gracefully
router.shutdown().await.anyerr()?;
Ok(())
}
async fn connect_side(addr: EndpointAddr) -> Result<()> {
let endpoint = Endpoint::bind(presets::N0).await?;
// Open a connection to the accepting endpoint
let conn = endpoint.connect(addr, ALPN).await?;
// Open a bidirectional QUIC stream
let (mut send, mut recv) = conn.open_bi().await.anyerr()?;
// Send some data to be echoed
send.write_all(b"Hello, world!").await.anyerr()?;
// Signal the end of data for this particular stream
send.finish().anyerr()?;
// Receive the echo, but limit reading up to maximum 1000 bytes
let response = recv.read_to_end(1000).await.anyerr()?;
assert_eq!(&response, b"Hello, world!");
// Explicitly close the whole connection.
conn.close(0u32.into(), b"bye!");
// The above call only queues a close message to be sent (see how it's not async!).
// We need to actually call this to make sure this message is sent out.
endpoint.close().await;
// If we don't call this, but continue using the endpoint, we then the queued
// close call will eventually be picked up and sent.
// But always try to wait for endpoint.close().await to go through before dropping
// the endpoint to ensure any queued messages are sent through and connections are
// closed gracefully.
Ok(())
}
async fn start_accept_side() -> Result<Router> {
let endpoint = Endpoint::bind(presets::N0).await?;
// Build our protocol handler and add our protocol, identified by its ALPN, and spawn the endpoint.
let router = Router::builder(endpoint).accept(ALPN, Echo).spawn();
Ok(router)
}
#[derive(Debug, Clone)]
struct Echo;
impl ProtocolHandler for Echo {
/// The `accept` method is called for each incoming connection for our ALPN.
///
/// The returned future runs on a newly spawned tokio task, so it can run as long as
/// the connection lasts.
async fn accept(&self, connection: Connection) -> Result<(), AcceptError> {
// We can get the remote's endpoint id from the connection.
let endpoint_id = connection.remote_id();
println!("accepted connection from {endpoint_id}");
// Our protocol is a simple request-response protocol, so we expect the
// connecting peer to open a single bi-directional stream.
let (mut send, mut recv) = connection.accept_bi().await?;
// Echo any bytes received back directly.
// This will keep copying until the sender signals the end of data on the stream.
let bytes_sent = tokio::io::copy(&mut recv, &mut send).await?;
println!("Copied over {bytes_sent} byte(s)");
// By calling `finish` on the send stream we signal that we will not send anything
// further, which makes the receive stream on the other end terminate.
send.finish()?;
// Wait until the remote closes the connection, which it does once it
// received the response.
connection.closed().await;
Ok(())
}
}
+89
View File
@@ -0,0 +1,89 @@
//! Tests connection to a home relay server and reports the status.
//!
//! Also shows how to specifically test if a connection fails due to authentication.
use clap::Parser;
use iroh::{Endpoint, RelayMap, RelayUrl, Watcher, endpoint::presets};
use n0_future::StreamExt;
#[derive(clap::Parser, Debug)]
struct Args {
/// Pass one or more relay URLs.
///
/// If unset will use the public default relays.
relays: Vec<RelayUrl>,
}
#[tokio::main]
async fn main() -> n0_error::Result<()> {
tracing_subscriber::fmt::init();
let args = Args::parse();
let relay_map = if args.relays.is_empty() {
iroh::defaults::prod::default_relay_map()
} else {
RelayMap::from_iter(args.relays)
};
let endpoint = Endpoint::builder(presets::Minimal)
.relay_mode(iroh::RelayMode::Custom(relay_map))
.bind()
.await?;
println!("endpoint bound");
// Spawn a task to report the home relay status.
//
// You could pass a channel here if you wanted to abort something in case the
// home relay connection fails.
tokio::spawn(report_home_relay_status(endpoint.clone()));
// Wait for the endpoint to be online, i.e. connected to a home relay.
//
// Internally, this works like simplified version of the loop in `report_home_relay_status`,
// only reporting about whether a connection was established and not exposing any
// details in case of failure.
endpoint.online().await;
println!(
"Endpoint is online. Home relay: {}",
endpoint.addr().relay_urls().next().expect("has relay")
);
tokio::signal::ctrl_c().await.ok();
endpoint.close().await;
Ok(())
}
async fn report_home_relay_status(endpoint: Endpoint) {
let mut home_relay_status = endpoint.home_relay_status().stream();
// The stream moves to the next item whenever the home relay status changes in any way.
while let Some(status_list) = home_relay_status.next().await {
// The stream's item is a list of status entries, one per home relay. Today the list
// never holds more than one entry, because iroh only connects to a single home
// relay. It is a list so that iroh could support multiple home relays without an API
// change, so we may as well iterate it.
for status in status_list {
let url = status.url();
if status.is_connected() {
// We are connected!
// When we reach this line, `Endpoint::online` also resolves.
println!("Relay {url}: Connected");
} else if let Some(reason) = status.auth_denied_reason() {
// The relay server denied our authentication. Retrying will not help: we
// would present the same credentials again. Report this prominently in
// our app instead of waiting to come online.
println!("Relay {url}: Authentication denied ({reason})");
} else if let Some(error) = status.last_error() {
// The connection failed for another reason, for example a network issue.
// iroh keeps retrying with a backoff, so let's just print it.
println!("Relay {url}: Connection failed ({error:#})");
} else {
// We are not connected, and there is no error to report: either no
// connection has been attempted yet, or one is in progress. You should
// usually just loop over this and wait for the next update.
println!("Relay {url}: Disconnected")
}
}
}
}
+76
View File
@@ -0,0 +1,76 @@
//! Example demonstrating the [`IncomingFilter`] hook.
//!
//! This example requires all direct (UDP) connections to pass QUIC address
//! validation via a retry token before being accepted.
//!
//! QUIC address validation ensures that a client truly owns its claimed IP
//! address before proceeding with the more expensive part of the handshake,
//! preventing denial-of-service attacks via spoofed source IPs.
//!
//! Relay connections are
//! accepted without validation since the relay already vouches for the source.
//!
//! ## Usage
//!
//! ```sh
//! cargo run --example incoming-filter
//! ```
//!
//! To test, connect from another process using:
//! ```sh
//! cargo run --example connect -- <NODE_ID>
//! ```
use std::sync::Arc;
use iroh::{
Endpoint,
endpoint::{Connection, Incoming, IncomingAddr, presets},
protocol::{AcceptError, IncomingFilterOutcome, ProtocolHandler, Router},
};
use n0_error::{Result, StdResultExt};
const ALPN: &[u8] = b"iroh-example/incoming-filter/0";
#[derive(Debug, Clone)]
struct Echo;
impl ProtocolHandler for Echo {
async fn accept(&self, connection: Connection) -> Result<(), AcceptError> {
let (mut send, mut recv) = connection.accept_bi().await?;
tokio::io::copy(&mut recv, &mut send).await?;
send.finish()?;
Ok(())
}
}
/// Require address validation for direct connections.
///
/// If the address is not yet validated, return `Retry` so the client has to
/// prove it owns the source address before we do any further work. Validated
/// connections and relay connections are accepted.
fn filter(incoming: &Incoming) -> IncomingFilterOutcome {
match incoming.remote_addr() {
IncomingAddr::Ip(_) if !incoming.remote_addr_validated() => IncomingFilterOutcome::Retry,
_ => IncomingFilterOutcome::Accept,
}
}
#[tokio::main]
async fn main() -> Result<()> {
let endpoint = Endpoint::bind(presets::N0).await?;
endpoint.online().await;
let addr = endpoint.addr();
println!("Node ID: {}", endpoint.id());
println!("Listening on: {addr:?}");
println!("All direct connections require address validation via retry.\n");
let router = Router::builder(endpoint)
.incoming_filter(Arc::new(filter))
.accept(ALPN, Echo)
.spawn();
tokio::signal::ctrl_c().await.anyerr()?;
router.shutdown().await.anyerr()?;
Ok(())
}
+99
View File
@@ -0,0 +1,99 @@
//! The smallest example showing how to use iroh and [`iroh::Endpoint`] to connect two devices and pass bytes using unreliable datagrams.
//!
//! This example uses the default relay servers to attempt to holepunch, and will use that relay server to relay packets if the two devices cannot establish a direct UDP connection.
//! run this example from the project root:
//! $ cargo run --example listen-unreliable
use iroh::{Endpoint, RelayMode, SecretKey, endpoint::presets};
use n0_error::{AnyError as Error, Result, StdResultExt};
use tracing::{info, warn};
// An example ALPN that we are using to communicate over the `Endpoint`
const EXAMPLE_ALPN: &[u8] = b"n0/iroh/examples/0";
#[tokio::main]
async fn main() -> Result<()> {
tracing_subscriber::fmt::init();
println!("\nlisten (unreliable) example!\n");
let secret_key = SecretKey::generate();
println!("public key: {}", secret_key.public());
// Build a `Endpoint`, which uses PublicKeys as endpoint identifiers, uses QUIC for directly connecting to other endpoints, and uses the relay servers to holepunch direct connections between endpoints when there are NATs or firewalls preventing direct connections. If no direct connection can be made, packets are relayed over the relay servers.
let endpoint = Endpoint::builder(presets::N0)
// The secret key is used to authenticate with other endpoints. The PublicKey portion of this secret key is how we identify endpoints, often referred to as the `endpoint_id` in our codebase.
.secret_key(secret_key)
// set the ALPN protocols this endpoint will accept on incoming connections
.alpns(vec![EXAMPLE_ALPN.to_vec()])
// `RelayMode::Default` means that we will use the default relay servers to holepunch and relay.
// Use `RelayMode::Custom` to pass in a `RelayMap` with custom relay urls.
// Use `RelayMode::Disable` to disable holepunching and relaying over HTTPS
// If you want to experiment with relaying using your own relay server, you must pass in the same custom relay url to both the `listen` code AND the `connect` code
.relay_mode(RelayMode::Default)
// you can choose a port to bind to, but passing in `0` will bind the socket to a random available port
.bind()
.await?;
let me = endpoint.id();
println!("endpoint id: {me}");
println!("endpoint listening addresses:");
// wait for the endpoint to be online
endpoint.online().await;
let endpoint_addr = endpoint.addr();
let local_addrs = endpoint_addr
.ip_addrs()
.map(|addr| {
let addr = addr.to_string();
println!("\t{addr}");
addr
})
.collect::<Vec<_>>()
.join(" ");
let relay_url = endpoint_addr
.relay_urls()
.next()
.expect("Should have a relay URL, assuming a default endpoint setup.");
println!("endpoint relay server url: {relay_url}");
println!("\nin a separate terminal run:");
println!(
"\tcargo run --example connect-unreliable -- --endpoint-id {me} --addrs \"{local_addrs}\" --relay-url {relay_url}\n"
);
// accept incoming connections, returns a normal QUIC connection
while let Some(incoming) = endpoint.accept().await {
let mut accepting = match incoming.accept() {
Ok(accepting) => accepting,
Err(err) => {
warn!("incoming connection failed: {err:#}");
// we can carry on in these cases:
// this can be caused by retransmitted datagrams
continue;
}
};
let alpn = accepting.alpn().await?;
let conn = accepting.await?;
let endpoint_id = conn.remote_id();
info!(
"new (unreliable) connection from {endpoint_id} with ALPN {}",
String::from_utf8_lossy(&alpn),
);
// spawn a task to handle reading and writing off of the connection
tokio::spawn(async move {
// use the `noq` API to read a datagram off the connection, and send a datagra, in return
while let Ok(message) = conn.read_datagram().await {
let message = String::from_utf8(message.into()).anyerr()?;
println!("received: {message}");
let message = format!("hi! you connected to {me}. bye bye");
conn.send_datagram(message.as_bytes().to_vec().into())
.anyerr()?;
}
Ok::<_, Error>(())
});
}
// stop with SIGINT (ctrl-c)
Ok(())
}
+116
View File
@@ -0,0 +1,116 @@
//! The smallest example showing how to use iroh and [`iroh::Endpoint`] to connect two devices.
//!
//! This example uses the default relay servers to attempt to holepunch, and will use that relay server to relay packets if the two devices cannot establish a direct UDP connection.
//! run this example from the project root:
//! $ cargo run --example listen
use std::time::Duration;
use iroh::{
Endpoint, RelayMode, SecretKey,
endpoint::{ConnectionError, presets},
};
use n0_error::{Result, StdResultExt};
use tracing::{debug, info, warn};
// An example ALPN that we are using to communicate over the `Endpoint`
const EXAMPLE_ALPN: &[u8] = b"n0/iroh/examples/0";
#[tokio::main]
async fn main() -> Result<()> {
tracing_subscriber::fmt::init();
println!("\nlisten example!\n");
let secret_key = SecretKey::generate();
println!("public key: {}", secret_key.public());
// Build a `Endpoint`, which uses PublicKeys as endpoint identifiers, uses QUIC for directly connecting to other endpoints, and uses the relay protocol and relay servers to holepunch direct connections between endpoints when there are NATs or firewalls preventing direct connections. If no direct connection can be made, packets are relayed over the relay servers.
let endpoint = Endpoint::builder(presets::N0)
// The secret key is used to authenticate with other endpoints. The PublicKey portion of this secret key is how we identify endpoints, often referred to as the `endpoint_id` in our codebase.
.secret_key(secret_key)
// set the ALPN protocols this endpoint will accept on incoming connections
.alpns(vec![EXAMPLE_ALPN.to_vec()])
// `RelayMode::Default` means that we will use the default relay servers to holepunch and relay.
// Use `RelayMode::Custom` to pass in a `RelayMap` with custom relay urls.
// Use `RelayMode::Disable` to disable holepunching and relaying over HTTPS
// If you want to experiment with relaying using your own relay server, you must pass in the same custom relay url to both the `listen` code AND the `connect` code
.relay_mode(RelayMode::Default)
// you can choose a port to bind to, but passing in `0` will bind the socket to a random available port
.bind()
.await?;
let me = endpoint.id();
println!("endpoint id: {me}");
println!("endpoint listening addresses:");
// wait for the endpoint to be online
endpoint.online().await;
let endpoint_addr = endpoint.addr();
let local_addrs = endpoint_addr
.ip_addrs()
.map(|addr| {
let addr = addr.to_string();
println!("\t{addr}");
addr
})
.collect::<Vec<_>>()
.join(" ");
let relay_url = endpoint_addr.relay_urls().next().expect("missing relay");
println!("endpoint relay server url: {relay_url}");
println!("\nin a separate terminal run:");
println!(
"\tcargo run --example connect -- --endpoint-id {me} --addrs \"{local_addrs}\" --relay-url {relay_url}\n"
);
// accept incoming connections, returns a normal QUIC connection
while let Some(incoming) = endpoint.accept().await {
let mut accepting = match incoming.accept() {
Ok(accepting) => accepting,
Err(err) => {
warn!("incoming connection failed: {err:#}");
// we can carry on in these cases:
// this can be caused by retransmitted datagrams
continue;
}
};
let alpn = accepting.alpn().await?;
let conn = accepting.await?;
let endpoint_id = conn.remote_id();
info!(
"new connection from {endpoint_id} with ALPN {}",
String::from_utf8_lossy(&alpn),
);
// spawn a task to handle reading and writing off of the connection
tokio::spawn(async move {
// accept a bi-directional QUIC connection
// use the `noq` APIs to send and recv content
let (mut send, mut recv) = conn.accept_bi().await.anyerr()?;
debug!("accepted bi stream, waiting for data...");
let message = recv.read_to_end(100).await.anyerr()?;
let message = String::from_utf8(message).anyerr()?;
println!("received: {message}");
let message = format!("hi! you connected to {me}. bye bye");
send.write_all(message.as_bytes()).await.anyerr()?;
// call `finish` to close the connection gracefully
send.finish().anyerr()?;
// We sent the last message, so wait for the client to close the connection once
// it received this message.
let res = tokio::time::timeout(Duration::from_secs(3), async move {
let closed = conn.closed().await;
if !matches!(closed, ConnectionError::ApplicationClosed(_)) {
println!("endpoint {endpoint_id} disconnected with an error: {closed:#}");
}
})
.await;
if res.is_err() {
println!("endpoint {endpoint_id} did not disconnect within 3 seconds");
}
n0_error::Ok(())
});
}
// stop with SIGINT (ctrl-c)
Ok(())
}
+159
View File
@@ -0,0 +1,159 @@
use std::{sync::Arc, time::Duration};
use iroh::{
Endpoint,
endpoint::{
AfterHandshakeOutcome, Closed, Connection, EndpointHooks, WeakConnectionHandle, presets,
},
};
use n0_error::{Result, StackResultExt, StdResultExt, ensure_any};
use n0_future::task::AbortOnDropHandle;
use tokio::{
sync::mpsc::{UnboundedReceiver, UnboundedSender},
task::JoinSet,
};
use tracing::{Instrument, info, info_span};
const ALPN: &[u8] = b"iroh/test";
#[tokio::main]
async fn main() -> Result {
tracing_subscriber::fmt()
.with_env_filter(
tracing_subscriber::EnvFilter::try_from_default_env().unwrap_or_else(|_| "info".into()),
)
.init();
let monitor = Monitor::new();
let server = Endpoint::builder(presets::Minimal)
.alpns(vec![ALPN.to_vec()])
.hooks(monitor.clone())
.bind()
.instrument(info_span!("server"))
.await?;
let server_addr = server.addr();
let count = 2;
let client_task = tokio::spawn(
async move {
let client = Endpoint::builder(presets::Minimal)
.bind()
.instrument(info_span!("client"))
.await?;
for _i in 0..count {
let conn = client.connect(server_addr.clone(), ALPN).await?;
let mut s = conn.accept_uni().await.anyerr()?;
let data = s.read_to_end(2).await.anyerr()?;
ensure_any!(data == b"hi", "unexpected data");
conn.close(23u32.into(), b"bye");
}
client.close().await;
n0_error::Ok(client)
}
.instrument(info_span!("client")),
);
let server_task = tokio::spawn(
async move {
for _i in 0..count {
let conn = server
.accept()
.await
.context("server endpoint closed")?
.await?;
let mut s = conn.open_uni().await.anyerr()?;
s.write_all(b"hi").await.anyerr()?;
s.finish().anyerr()?;
conn.closed().await;
}
server.close().await;
n0_error::Ok(())
}
.instrument(info_span!("server")),
);
client_task.await.std_context("client")?.context("client")?;
server_task.await.std_context("server")?.context("server")?;
tokio::time::sleep(Duration::from_secs(1)).await;
drop(monitor);
Ok(())
}
/// Our connection monitor impl.
///
/// This here only logs connection open and close events via tracing.
/// It could also maintain a datastructure of all connections, or send the stats to some metrics service.
#[derive(Clone, Debug)]
struct Monitor {
tx: UnboundedSender<MonitoredConnection>,
_task: Arc<AbortOnDropHandle<()>>,
}
/// Static info captured at handshake time, paired with a weak handle to the connection.
///
/// We capture `alpn` and `remote_id` at hook time because [`WeakConnectionHandle`] only
/// exposes [`upgrade`] and [`closed`], so reading these fields after the connection has
/// been dropped would otherwise be impossible.
///
/// [`upgrade`]: WeakConnectionHandle::upgrade
/// [`closed`]: WeakConnectionHandle::closed
#[derive(Debug)]
struct MonitoredConnection {
alpn: Vec<u8>,
remote_id: iroh::EndpointId,
handle: WeakConnectionHandle,
}
impl EndpointHooks for Monitor {
async fn after_handshake(&self, conn: &Connection) -> AfterHandshakeOutcome {
let info = MonitoredConnection {
alpn: conn.alpn().to_vec(),
remote_id: conn.remote_id(),
handle: conn.weak_handle(),
};
self.tx.send(info).ok();
AfterHandshakeOutcome::Accept
}
}
impl Monitor {
fn new() -> Self {
let (tx, rx) = tokio::sync::mpsc::unbounded_channel();
let task = tokio::spawn(Self::run(rx).instrument(info_span!("watcher")));
Self {
tx,
_task: Arc::new(AbortOnDropHandle::new(task)),
}
}
async fn run(mut rx: UnboundedReceiver<MonitoredConnection>) {
let mut tasks = JoinSet::new();
loop {
tokio::select! {
Some(MonitoredConnection { alpn, remote_id, handle }) = rx.recv() => {
let alpn = String::from_utf8_lossy(&alpn).to_string();
let remote = remote_id.fmt_short();
let rtt = handle.upgrade().and_then(|c| c.paths().iter().map(|p| p.rtt()).min());
info!(%remote, %alpn, ?rtt, "new connection");
tasks.spawn(async move {
match handle.closed().await {
Some(Closed { reason, stats, .. }) => {
// We have access to the final stats of the connection!
info!(%remote, %alpn, ?reason, udp_rx=stats.udp_rx.bytes, udp_tx=stats.udp_tx.bytes, "connection closed");
}
None => {
// The connection was closed before we could register our stats-on-close listener.
info!(%remote, %alpn, "connection closed before tracking started");
}
}
}.instrument(tracing::Span::current()));
}
Some(res) = tasks.join_next(), if !tasks.is_empty() => res.expect("conn close task panicked"),
else => break,
}
while let Some(res) = tasks.join_next().await {
res.expect("conn close task panicked");
}
}
}
}
+94
View File
@@ -0,0 +1,94 @@
//! Force iroh to negotiate ONLY a post-quantum key exchange (X25519MLKEM768).
//!
//! Requires `tls-aws-lc-rs` (the only rustls backend with ML-KEM today).
//! Stripping `kx_groups` to the PQ group makes PQ *required* rather than
//! *preferred*: peers without it fail the TLS handshake.
//!
//! Note: iroh's `crypto_provider` is shared with relay/discovery TLS, and
//! n0's infra does not support PQ key exchange yet — so a PQ-only endpoint
//! can't use n0 public relays or discovery servers.
//!
//! ## Usage
//!
//! `tls-aws-lc-rs` is required (the example explicitly constructs an
//! aws-lc-rs `CryptoProvider`, so the backend must be linked):
//!
//! cargo run --example pq-key-exchange --features=tls-aws-lc-rs
//!
//! With iroh's default features still on, both `ring` and `aws-lc-rs` get
//! linked. That's harmless — we wire the aws-lc-rs provider in directly via
//! `Builder::crypto_provider`.
use std::sync::Arc;
use iroh::{
RelayMode,
endpoint::{Endpoint, presets::Empty},
};
use n0_error::{Result, StdResultExt};
use rustls::crypto::aws_lc_rs;
const ALPN: &[u8] = b"iroh-example/pq-key-exchange/0";
#[tokio::main]
async fn main() -> Result<()> {
tracing_subscriber::fmt::init();
let pq = pq_only_provider();
let server = Endpoint::builder(Empty)
.crypto_provider(pq.clone())
.alpns(vec![ALPN.to_vec()])
.relay_mode(RelayMode::Disabled)
.bind()
.await?;
let server_addr = server.addr();
let server_task = tokio::spawn({
let server = server.clone();
async move {
let conn = server
.accept()
.await
.expect("incoming")
.accept()
.anyerr()?
.await
.anyerr()?;
let mut recv = conn.accept_uni().await.anyerr()?;
let msg = recv.read_to_end(1024).await.anyerr()?;
println!("server received: {:?}", String::from_utf8_lossy(&msg));
let mut send = conn.open_uni().await.anyerr()?;
send.write_all(b"pong over PQ").await.anyerr()?;
send.finish().anyerr()?;
conn.closed().await;
n0_error::Ok(())
}
});
let client = Endpoint::builder(Empty)
.crypto_provider(pq)
.relay_mode(RelayMode::Disabled)
.bind()
.await?;
let conn = client.connect(server_addr, ALPN).await.anyerr()?;
println!("client handshake done (X25519MLKEM768 was the only kx offered)");
let mut send = conn.open_uni().await.anyerr()?;
send.write_all(b"ping over PQ").await.anyerr()?;
send.finish().anyerr()?;
let mut recv = conn.accept_uni().await.anyerr()?;
let msg = recv.read_to_end(1024).await.anyerr()?;
println!("client received: {:?}", String::from_utf8_lossy(&msg));
conn.close(0u32.into(), b"done");
server_task.await.anyerr()?.anyerr()?;
client.close().await;
server.close().await;
Ok(())
}
fn pq_only_provider() -> Arc<rustls::crypto::CryptoProvider> {
let mut p = aws_lc_rs::default_provider();
p.kx_groups = vec![aws_lc_rs::kx_group::X25519MLKEM768];
Arc::new(p)
}
+90
View File
@@ -0,0 +1,90 @@
//! Prefer post-quantum key exchange when available, fall back to classical.
//!
//! Unlike `pq-only-key-exchange`, this example keeps classical kx groups in
//! the list so the endpoint can still talk to peers that don't support
//! ML-KEM, *and* so n0's relay/discovery TLS (classical kx) keeps working.
//! Putting `X25519MLKEM768` first in `kx_groups` means rustls negotiates PQ
//! whenever both peers support it, classical otherwise.
//!
//! Note: rustls' `aws_lc_rs::default_provider()` only puts `X25519MLKEM768`
//! first when rustls is built with its `prefer-post-quantum` feature; without
//! it, PQ is offered last. We override `kx_groups` here so the policy is
//! independent of how rustls was compiled, and print the list at startup.
//!
//! ## Usage
//!
//! `tls-aws-lc-rs` is required:
//!
//! cargo run --example prefer-pq-key-exchange --features=tls-aws-lc-rs
use std::sync::Arc;
use iroh::endpoint::{Endpoint, presets::N0};
use n0_error::{Result, StdResultExt};
use rustls::crypto::aws_lc_rs::{self, kx_group};
const ALPN: &[u8] = b"iroh-example/prefer-pq-key-exchange/0";
#[tokio::main]
async fn main() -> Result<()> {
tracing_subscriber::fmt::init();
let mut provider = aws_lc_rs::default_provider();
provider.kx_groups = vec![
kx_group::X25519MLKEM768,
kx_group::X25519,
kx_group::SECP256R1,
kx_group::SECP384R1,
];
let kx_names: Vec<_> = provider.kx_groups.iter().map(|g| g.name()).collect();
println!("kx_groups (in offer order): {kx_names:?}");
let pq = Arc::new(provider);
let server = Endpoint::builder(N0)
.crypto_provider(pq.clone())
.alpns(vec![ALPN.to_vec()])
.bind()
.await?;
server.online().await;
let server_addr = server.addr();
println!("server addr: {server_addr:?}");
let server_task = tokio::spawn({
let server = server.clone();
async move {
let conn = server
.accept()
.await
.expect("incoming")
.accept()
.anyerr()?
.await
.anyerr()?;
let mut recv = conn.accept_uni().await.anyerr()?;
let msg = recv.read_to_end(1024).await.anyerr()?;
println!("server received: {:?}", String::from_utf8_lossy(&msg));
let mut send = conn.open_uni().await.anyerr()?;
send.write_all(b"pong over PQ-preferred").await.anyerr()?;
send.finish().anyerr()?;
conn.closed().await;
n0_error::Ok(())
}
});
let client = Endpoint::builder(N0).crypto_provider(pq).bind().await?;
client.online().await;
let conn = client.connect(server_addr, ALPN).await.anyerr()?;
println!("client handshake done (X25519MLKEM768 preferred, classical kx kept as fallback)");
let mut send = conn.open_uni().await.anyerr()?;
send.write_all(b"ping over PQ-preferred").await.anyerr()?;
send.finish().anyerr()?;
let mut recv = conn.accept_uni().await.anyerr()?;
let msg = recv.read_to_end(1024).await.anyerr()?;
println!("client received: {:?}", String::from_utf8_lossy(&msg));
conn.close(0u32.into(), b"done");
server_task.await.anyerr()?.anyerr()?;
client.close().await;
server.close().await;
Ok(())
}
+447
View File
@@ -0,0 +1,447 @@
//! Example for using an iroh hook to collect information about remote endpoints.
//!
//! This implements a [`RemoteMap`] which collects information about all connections and paths from an iroh endpoint.
//! The remote map can be cloned and inspected from other tasks at any time. It contains both data about all
//! currently active connections, and an aggregate status for each remote that remains available even after
//! all connections to the endpoint have been closed.
use std::time::{Duration, SystemTime};
use iroh::{Endpoint, EndpointAddr, endpoint::presets};
use n0_error::{Result, StackResultExt, StdResultExt, ensure_any};
use n0_future::IterExt;
use tracing::{Instrument, info, info_span};
use crate::remote_map::RemoteMap;
const ALPN: &[u8] = b"iroh/test";
#[tokio::main(flavor = "multi_thread")]
async fn main() -> Result {
tracing_subscriber::fmt()
.with_env_filter(
tracing_subscriber::EnvFilter::try_from_default_env().unwrap_or_else(|_| "info".into()),
)
.init();
// Create the remote map and hook.
let (hook, remote_map) = RemoteMap::new();
// Bind our endpoint and install the remote map hook.
let server = Endpoint::builder(presets::N0)
.alpns(vec![ALPN.to_vec()])
.hooks(hook)
.bind()
.instrument(info_span!("server"))
.await?;
// Wait for our endpoint to be fully online.
server.online().await;
let server_addr = server.addr();
// Spawn a task that creates `count` client endpoints that each connect to our server.
let count = 3;
let client_task = tokio::spawn(run_clients(server_addr, count));
// Spawn a task that prints info from the remote map while some connections are active.
// You can use this info to make decisions about remotes.
let _inspect_task = tokio::task::spawn({
let remote_map = remote_map.clone();
async move {
// Wait a bit.
tokio::time::sleep(Duration::from_millis(500)).await;
println!("== while connections are active == ");
log_active(&remote_map);
log_aggregate(&remote_map);
println!();
}
});
// Let the server accept `count` connections in parallel.
// The server keeps all connections open for at least 500 milliseconds.
std::iter::repeat_with(async || {
let conn = server
.accept()
.await
.context("server endpoint closed")?
.await?;
info!("accepted");
let mut s = conn.open_uni().await.anyerr()?;
// wait a bit.
tokio::time::sleep(Duration::from_millis(500)).await;
s.write_all(b"hi").await.anyerr()?;
s.finish().anyerr()?;
conn.closed().await;
info!("closed");
n0_error::Ok(())
})
.take(count)
.enumerate()
.map(|(i, fut)| fut.instrument(info_span!("server-conn", %i)))
.try_join_all()
.await?;
// Print the remote map again.
println!("== all connections closed ==");
log_active(&remote_map);
log_aggregate(&remote_map);
server.close().await;
client_task.await.std_context("client")?.context("client")?;
Ok(())
}
/// Uses the current connection info to print info about a remote.
///
/// Uses the info about *currently active* connections, which return `None` if no connections are active.
fn log_active(remote_map: &RemoteMap) {
println!("current remote state:");
for (id, info) in remote_map.read().iter() {
println!(
"[{}] is_active {}, connections {}, ip_path {:?}, relay_path {:?}, current_min_rtt {:?}",
id.fmt_short(),
info.is_active(),
info.active_connections(),
info.has_ip_path(),
info.has_relay_path(),
info.current_min_rtt(),
);
}
}
/// Uses the aggregated info to print info about a remote.
///
/// The aggregated info is updated for all connection and path changes, and stays at the latest values
/// even if all connections are closed.
fn log_aggregate(remote_map: &RemoteMap) {
println!("aggregate remote state:");
for (id, info) in remote_map.read().iter() {
let aggregate = info.aggregate();
println!(
"[{}] min_rtt {:?}, max_rtt {:?}, ip_path {:?}, relay_path {}, last_update {:?} ago",
id.fmt_short(),
aggregate.rtt_min,
aggregate.rtt_max,
aggregate.ip_path,
aggregate.relay_path,
SystemTime::now()
.duration_since(aggregate.last_update)
.unwrap_or_default()
);
}
}
async fn run_clients(server_addr: EndpointAddr, count: usize) -> Result {
std::iter::repeat_with(async || {
let client = Endpoint::builder(presets::N0)
.bind()
.instrument(info_span!("client"))
.await?;
let conn = client.connect(server_addr.clone(), ALPN).await?;
info!("connected");
let mut s = conn.accept_uni().await.anyerr()?;
let data = s.read_to_end(2).await.anyerr()?;
ensure_any!(data == b"hi", "unexpected data");
conn.close(23u32.into(), b"bye");
info!("closed");
client.close().await;
n0_error::Ok(())
})
.take(count)
.enumerate()
.map(|(i, fut)| fut.instrument(info_span!("client", %i)))
.try_join_all()
.await?;
Ok(())
}
mod remote_map {
//! Implementation of a remote map and hook to track information about all remote endpoints to which an iroh endpoint
//! has connections with.
use std::{
collections::HashMap,
sync::{Arc, RwLock, RwLockReadGuard},
time::{Duration, SystemTime},
};
use iroh::{
EndpointId, TransportAddr,
endpoint::{
AfterHandshakeOutcome, Connection, EndpointHooks, PathEvent, WeakConnectionHandle,
},
};
use n0_future::{StreamExt, task::AbortOnDropHandle};
use tokio::{sync::mpsc, task::JoinSet};
use tracing::{Instrument, debug, info, info_span};
/// Information about a remote info.
#[derive(Debug, Default)]
pub struct RemoteInfo {
aggregate: Aggregate,
connections: HashMap<u64, WeakConnectionHandle>,
}
/// Aggregate information about a remote info.
#[derive(Debug)]
pub struct Aggregate {
/// Minimal RTT observed over all paths to this remote.
pub rtt_min: Duration,
/// Maximal RTT observed over all paths to this remote.
pub rtt_max: Duration,
/// Whether we ever had an IP path to this remote.
pub ip_path: bool,
/// Whether we ever had a relay path to this remote.
pub relay_path: bool,
/// Time this aggregate was last updated.
pub last_update: SystemTime,
}
impl Default for Aggregate {
fn default() -> Self {
Self {
rtt_min: Duration::MAX,
rtt_max: Duration::ZERO,
ip_path: false,
relay_path: false,
last_update: SystemTime::UNIX_EPOCH,
}
}
}
impl Aggregate {
fn update(&mut self, addr: TransportAddr, stats: iroh::endpoint::PathStats) {
self.last_update = SystemTime::now();
self.ip_path |= addr.is_ip();
self.relay_path |= addr.is_relay();
debug!("path update {addr} {stats:?}");
self.rtt_min = self.rtt_min.min(stats.rtt);
self.rtt_max = self.rtt_max.max(stats.rtt);
}
}
impl RemoteInfo {
/// Returns an aggregate of stats for this remote.
///
/// This includes info from closed connections.
pub fn aggregate(&self) -> &Aggregate {
&self.aggregate
}
/// Returns the minimal RTT of all currently active paths.
///
/// Returns `None` if there are no active connections.
pub fn current_min_rtt(&self) -> Option<Duration> {
self.upgraded()
.filter_map(|c| c.paths().iter().map(|p| p.rtt()).min())
.min()
}
/// Returns whether any active connection to the remote has an active IP path.
pub fn has_ip_path(&self) -> bool {
self.upgraded().any(|c| c.paths().iter().any(|p| p.is_ip()))
}
/// Returns whether any active connection to the remote has an active relay path.
pub fn has_relay_path(&self) -> bool {
self.upgraded()
.any(|c| c.paths().iter().any(|p| p.is_relay()))
}
/// Returns `true` if there are active connections to this remote.
pub fn is_active(&self) -> bool {
self.active_connections() > 0
}
/// Returns the number of active connections to this remote.
pub fn active_connections(&self) -> usize {
self.upgraded().count()
}
/// Returns an iterator over all active handles upgraded to a [`Connection`].
fn upgraded(&self) -> impl Iterator<Item = Connection> {
self.connections
.values()
.filter_map(WeakConnectionHandle::upgrade)
}
}
type RemoteMapInner = Arc<RwLock<HashMap<EndpointId, RemoteInfo>>>;
/// Contains information about remote nodes our endpoint has or had connections with.
#[derive(Clone, Debug)]
pub struct RemoteMap {
map: RemoteMapInner,
_task: Arc<AbortOnDropHandle<()>>,
}
/// Hook to collect information about remote endpoints from an endpoint.
#[derive(Debug)]
pub struct RemoteMapHook {
tx: mpsc::Sender<TrackedConnection>,
}
/// Pairing of a connection's remote endpoint id, captured at handshake time, with a
/// weak handle to the connection. The `remote_id` is captured eagerly because
/// [`WeakConnectionHandle`] only exposes [`upgrade`] and [`closed`].
///
/// [`upgrade`]: WeakConnectionHandle::upgrade
/// [`closed`]: WeakConnectionHandle::closed
#[derive(Debug, Clone)]
pub struct TrackedConnection {
pub remote_id: EndpointId,
pub handle: WeakConnectionHandle,
}
impl EndpointHooks for RemoteMapHook {
async fn after_handshake(&self, conn: &Connection) -> AfterHandshakeOutcome {
info!(remote=%conn.remote_id().fmt_short(), "after_handshake");
// We track the connection via a weak handle so this map does not keep the
// connection alive past the application's last reference to it.
let tracked = TrackedConnection {
remote_id: conn.remote_id(),
handle: conn.weak_handle(),
};
self.tx.send(tracked).await.ok();
AfterHandshakeOutcome::Accept
}
}
impl RemoteMap {
/// Creates a new [`RemoteMapHook`] and [`RemoteMap`].
pub fn new() -> (RemoteMapHook, Self) {
Self::with_max_retention(Duration::from_secs(60 * 5))
}
/// Creates a new [`RemoteMapHook`] and [`RemoteMap`] and configure the retention time.
///
/// `retention_time` is the time entries for remote endpoints remain in the map after the last connection has closed.
pub fn with_max_retention(retention_time: Duration) -> (RemoteMapHook, Self) {
let (tx, rx) = mpsc::channel(8);
let map = RemoteMapInner::default();
let task = tokio::spawn(
Self::run(rx, map.clone(), retention_time)
.instrument(info_span!("remote-map-task")),
);
let map = Self {
map,
_task: Arc::new(AbortOnDropHandle::new(task)),
};
let hook = RemoteMapHook { tx };
(hook, map)
}
/// Read the current state of the remote map.
///
/// Returns a [`RwLockReadGuard`] with the actual remote map. Don't hold over await points!
pub fn read(&self) -> RwLockReadGuard<'_, HashMap<EndpointId, RemoteInfo>> {
self.map.read().expect("poisoned")
}
async fn run(
mut rx: mpsc::Receiver<TrackedConnection>,
map: RemoteMapInner,
retention_time: Duration,
) {
let mut tasks = JoinSet::new();
let mut conn_id = 0;
// Spawn a task to clear expired entries.
let expiry_task = tasks.spawn(Self::clear_expired(retention_time, map.clone()));
// Main loop
loop {
tokio::select! {
Some(conn) = rx.recv() => {
conn_id += 1;
Self::on_connection(&mut tasks, map.clone(), conn_id, conn);
}
Some(res) = tasks.join_next(), if !tasks.is_empty() => {
res.expect("conn close task panicked");
}
else => break,
}
}
// Abort expiry task and join remaining tasks.
expiry_task.abort();
while let Some(res) = tasks.join_next().await {
if let Err(err) = &res
&& !err.is_cancelled()
{
res.expect("conn close task panicked");
}
}
}
fn on_connection(
tasks: &mut JoinSet<()>,
map: RemoteMapInner,
conn_id: u64,
conn: TrackedConnection,
) {
// Store conn info for full introspection possibility.
{
let mut inner = map.write().expect("poisoned");
inner
.entry(conn.remote_id)
.or_default()
.connections
.insert(conn_id, conn.handle.clone());
}
// Track path changes to update stats aggregate.
if let Some(mut path_events) = conn.handle.upgrade().map(|conn| conn.path_events()) {
tasks.spawn(
async move {
while let Some(event) = path_events.next().await {
let mut inner = map.write().expect("poisoned");
let info = inner.entry(conn.remote_id).or_default();
match event {
PathEvent::Closed {
remote_addr,
last_stats,
..
} => {
info.aggregate.update(remote_addr, *last_stats);
}
_ => {
if let Some(conn) = conn.handle.upgrade() {
for path in conn.paths().iter() {
info.aggregate
.update(path.remote_addr().clone(), path.stats());
}
}
}
}
}
let mut inner = map.write().expect("poisoned");
let info = inner.entry(conn.remote_id).or_default();
info.connections.remove(&conn_id);
info.aggregate.last_update = SystemTime::now();
}
.instrument(tracing::Span::current()),
);
}
}
async fn clear_expired(
retention_time: Duration,
map: Arc<RwLock<HashMap<iroh::PublicKey, RemoteInfo>>>,
) {
let mut interval = tokio::time::interval(retention_time);
loop {
interval.tick().await;
let now = SystemTime::now();
let mut inner = map.write().expect("poisoned");
inner.retain(|_remote, info| {
info.is_active()
|| now
.duration_since(info.aggregate().last_update)
.map(|age| age < retention_time)
.unwrap_or(true)
});
}
}
}
}
+150
View File
@@ -0,0 +1,150 @@
//! Very basic example to showcase how to write a protocol that rejects new
//! connections based on internal state. Useful when you want an endpoint to
//! stop accepting new connections for some reason only known to the endpoint. Maybe
//! it's doing a migration, starting up, in a "maintenance mode", or serving
//! too many connections.
//!
//! ## Usage
//!
//! cargo run --example screening-connection
use std::sync::{
Arc,
atomic::{AtomicU64, Ordering},
};
use iroh::{
Endpoint, EndpointAddr,
endpoint::{Accepting, Connection, presets},
protocol::{AcceptError, ProtocolHandler, Router},
};
use n0_error::{Result, StdResultExt, e};
/// Each protocol is identified by its ALPN string.
///
/// The ALPN, or application-layer protocol negotiation, is exchanged in the connection handshake,
/// and the connection is aborted unless both endpoints pass the same bytestring.
const ALPN: &[u8] = b"iroh-example/screening-connection/0";
#[tokio::main]
async fn main() -> Result<()> {
let router = start_accept_side().await?;
// Wait for the endpoint to be reachable
router.endpoint().online().await;
let endpoint_addr = router.endpoint().addr();
// call connect three times. connection index 1 will be an odd number, and rejected.
connect_side(&endpoint_addr).await?;
if let Err(err) = connect_side(&endpoint_addr).await {
println!("Error connecting: {}", err);
}
connect_side(&endpoint_addr).await?;
// This makes sure the endpoint in the router is closed properly and connections close gracefully
router.shutdown().await.anyerr()?;
Ok(())
}
async fn connect_side(addr: &EndpointAddr) -> Result<()> {
let endpoint = Endpoint::bind(presets::N0).await?;
// Open a connection to the accepting endpoint
let conn = endpoint.connect(addr.clone(), ALPN).await?;
// Open a bidirectional QUIC stream
let (mut send, mut recv) = conn.open_bi().await.anyerr()?;
// Send some data to be echoed
send.write_all(b"Hello, world!").await.anyerr()?;
// Signal the end of data for this particular stream
send.finish().anyerr()?;
// Receive the echo, but limit reading up to maximum 1000 bytes
let response = recv.read_to_end(1000).await.anyerr()?;
assert_eq!(&response, b"Hello, world!");
// Explicitly close the whole connection.
conn.close(0u32.into(), b"bye!");
// The above call only queues a close message to be sent (see how it's not async!).
// We need to actually call this to make sure this message is sent out.
endpoint.close().await;
// If we don't call this, but continue using the endpoint, we then the queued
// close call will eventually be picked up and sent.
// But always try to wait for endpoint.close().await to go through before dropping
// the endpoint to ensure any queued messages are sent through and connections are
// closed gracefully.
Ok(())
}
async fn start_accept_side() -> Result<Router> {
let endpoint = Endpoint::bind(presets::N0).await?;
let echo = ScreenedEcho {
conn_attempt_count: Arc::new(AtomicU64::new(0)),
};
// Build our protocol handler and add our protocol, identified by its ALPN, and spawn the endpoint.
let router = Router::builder(endpoint).accept(ALPN, echo).spawn();
Ok(router)
}
/// This is the same as the echo example, but keeps an internal count of the
/// number of connections that have been attempted. This is to demonstrate how
/// to plumb state into the protocol handler
#[derive(Debug, Clone)]
struct ScreenedEcho {
conn_attempt_count: Arc<AtomicU64>,
}
impl ProtocolHandler for ScreenedEcho {
/// `on_accepting` allows us to intercept a connection as it's being formed,
/// which is the right place to cut off a connection as early as possible.
/// This is an optional method on the ProtocolHandler trait.
async fn on_accepting(&self, accepting: Accepting) -> Result<Connection, AcceptError> {
self.conn_attempt_count.fetch_add(1, Ordering::Relaxed);
let count = self.conn_attempt_count.load(Ordering::Relaxed);
// reject every other connection
if count.is_multiple_of(2) {
println!("rejecting connection");
return Err(e!(AcceptError::NotAllowed));
}
// To allow normal connection construction, await the accepting future & return
let conn = accepting.await?;
Ok(conn)
}
/// The `accept` method is called for each incoming connection for our ALPN.
/// This is the primary place to kick off work in response to a new connection.
///
/// The returned future runs on a newly spawned tokio task, so it can run as long as
/// the connection lasts.
async fn accept(&self, connection: Connection) -> Result<(), AcceptError> {
// We can get the remote's endpoint id from the connection.
let endpoint_id = connection.remote_id();
println!("accepted connection from {endpoint_id}");
// Our protocol is a simple request-response protocol, so we expect the
// connecting peer to open a single bi-directional stream.
let (mut send, mut recv) = connection.accept_bi().await?;
// Echo any bytes received back directly.
// This will keep copying until the sender signals the end of data on the stream.
let bytes_sent = tokio::io::copy(&mut recv, &mut send).await?;
println!("Copied over {bytes_sent} byte(s)");
// By calling `finish` on the send stream we signal that we will not send anything
// further, which makes the receive stream on the other end terminate.
send.finish()?;
// Wait until the remote closes the connection, which it does once it
// received the response.
connection.closed().await;
Ok(())
}
}
+228
View File
@@ -0,0 +1,228 @@
//! Example protocol for running search on a remote endpoint.
//!
//! We are building a very simple protocol here.
//!
//! Our protocol allows querying the text stored on the other endpoint.
//!
//! The example is contrived - we only use memory endpoints, and our database is a hashmap in a mutex,
//! and our queries just match if the query string appears as-is.
//!
//! ## Usage
//!
//! In one terminal, run
//!
//! cargo run --example search -- listen "hello-world" "foo-bar" "hello-moon"
//!
//! This spawns an iroh endpoint with three blobs. It will print the endpoint's endpoint id.
//!
//! In another terminal, run
//!
//! cargo run --example search -- query <endpoint-id> hello
//!
//! Replace <endpoint-id> with the endpoint id from above. This will connect to the listening endpoint with our
//! protocol and query for the string `hello`. The listening endpoint will return a number of how many
//! strings match the query.
//!
//! For this example, this will print:
//!
//! Found 2 matches
//!
//! That's it! Follow along in the code below, we added a bunch of comments to explain things.
use std::{collections::BTreeSet, sync::Arc};
use clap::Parser;
use iroh::{
Endpoint, EndpointId,
endpoint::{Connection, presets},
protocol::{AcceptError, ProtocolHandler, Router},
};
use n0_error::{Result, StdResultExt};
use tokio::sync::Mutex;
use tracing_subscriber::{EnvFilter, prelude::*};
#[derive(Debug, Parser)]
pub struct Cli {
#[clap(subcommand)]
command: Command,
}
#[derive(Debug, Parser)]
pub enum Command {
/// Spawn an endpoint in listening mode.
Listen {
/// Each text string will be imported as a blob and inserted into the search database.
text: Vec<String>,
},
/// Query a remote endpoint for data and print the results.
Query {
/// The endpoint id of the endpoint we want to query.
endpoint_id: EndpointId,
/// The text we want to match.
query: String,
},
}
/// Each protocol is identified by its ALPN string.
///
/// The ALPN, or application-layer protocol negotiation, is exchanged in the connection handshake,
/// and the connection is aborted unless both endpoints pass the same bytestring.
const ALPN: &[u8] = b"iroh-example/text-search/0";
#[tokio::main]
async fn main() -> Result<()> {
setup_logging();
let args = Cli::parse();
// Build an endpoint
let endpoint = Endpoint::bind(presets::N0).await?;
// Build our protocol handler. The `builder` exposes access to various subsystems in the
// iroh endpoint. In our case, we need a blobs client and the endpoint.
let proto = BlobSearch::new(endpoint.clone());
let builder = Router::builder(endpoint);
// Add our protocol, identified by our ALPN, to the endpoint, and spawn the endpoint.
let router = builder.accept(ALPN, proto.clone()).spawn();
match args.command {
Command::Listen { text } => {
let endpoint_id = router.endpoint().id();
println!("our endpoint id: {endpoint_id}");
// Insert the text strings as blobs and index them.
for text in text.into_iter() {
proto.insert(text).await?;
}
// Wait for Ctrl-C to be pressed.
tokio::signal::ctrl_c().await.anyerr()?;
}
Command::Query { endpoint_id, query } => {
// Query the remote endpoint.
// This will send the query over our protocol, read hashes on the reply stream,
// and download each hash over iroh-blobs.
let num_matches = proto.query_remote(endpoint_id, &query).await?;
// Print out our query results.
println!("Found {num_matches} matches");
}
}
router.shutdown().await.anyerr()?;
Ok(())
}
#[derive(Debug, Clone)]
struct BlobSearch {
endpoint: Endpoint,
blobs: Arc<Mutex<BTreeSet<String>>>,
}
impl ProtocolHandler for BlobSearch {
/// The `accept` method is called for each incoming connection for our ALPN.
///
/// The returned future runs on a newly spawned tokio task, so it can run as long as
/// the connection lasts.
async fn accept(&self, connection: Connection) -> Result<(), AcceptError> {
// We can get the remote's endpoint id from the connection.
let endpoint_id = connection.remote_id();
println!("accepted connection from {endpoint_id}");
// Our protocol is a simple request-response protocol, so we expect the
// connecting peer to open a single bi-directional stream.
let (mut send, mut recv) = connection.accept_bi().await?;
// We read the query from the receive stream, while enforcing a max query length.
let query_bytes = recv.read_to_end(64).await.map_err(AcceptError::from_err)?;
// Now, we can perform the actual query on our local database.
let query = String::from_utf8(query_bytes).map_err(AcceptError::from_err)?;
let num_matches = self.query_local(&query).await;
// We want to return a list of hashes. We do the simplest thing possible, and just send
// one hash after the other. Because the hashes have a fixed size of 32 bytes, this is
// very easy to parse on the other end.
send.write_all(&num_matches.to_le_bytes())
.await
.map_err(AcceptError::from_err)?;
// By calling `finish` on the send stream we signal that we will not send anything
// further, which makes the receive stream on the other end terminate.
send.finish()?;
// Wait until the remote closes the connection, which it does once it
// received the response.
connection.closed().await;
Ok(())
}
}
impl BlobSearch {
/// Create a new protocol handler.
pub fn new(endpoint: Endpoint) -> Self {
Self {
endpoint,
blobs: Default::default(),
}
}
/// Query a remote endpoint, download all matching blobs and print the results.
pub async fn query_remote(&self, endpoint_id: EndpointId, query: &str) -> Result<u64> {
// Establish a connection to our endpoint.
// We use the default address lookup in iroh, so we can connect by endpoint id without
// providing further information.
let conn = self.endpoint.connect(endpoint_id, ALPN).await?;
// Open a bi-directional in our connection.
let (mut send, mut recv) = conn.open_bi().await.anyerr()?;
// Send our query.
send.write_all(query.as_bytes()).await.anyerr()?;
// Finish the send stream, signalling that no further data will be sent.
// This makes the `read_to_end` call on the accepting side terminate.
send.finish().anyerr()?;
// The response is a 64 bit integer
// We simply read it into a byte buffer.
let mut num_matches = [0u8; 8];
// Read 8 bytes from the stream.
recv.read_exact(&mut num_matches).await.anyerr()?;
let num_matches = u64::from_le_bytes(num_matches);
// Dropping the connection here will close it.
Ok(num_matches)
}
/// Query the local database.
///
/// Returns how many matches were found.
pub async fn query_local(&self, query: &str) -> u64 {
let guard = self.blobs.lock().await;
let count: usize = guard.iter().filter(|text| text.contains(query)).count();
count as u64
}
/// Insert a text string into the database.
pub async fn insert(&self, text: String) -> Result<()> {
let mut guard = self.blobs.lock().await;
guard.insert(text);
Ok(())
}
}
/// Set the RUST_LOG env var to one of {debug,info,warn} to see logging.
fn setup_logging() {
tracing_subscriber::registry()
.with(tracing_subscriber::fmt::layer().with_writer(std::io::stderr))
.with(EnvFilter::from_default_env())
.try_init()
.ok();
}
+1372
View File
File diff suppressed because it is too large. Load diff
+1
View File
@@ -0,0 +1 @@
pre-release-hook = ["git", "cliff", "--repository", "../", "--unreleased", "--prepend", "../CHANGELOG.md", "--tag", "{{version}}" ]
+1401
View File
File diff suppressed because it is too large. Load diff
+136
View File
@@ -0,0 +1,136 @@
//! DNS endpoint discovery for iroh
use iroh_base::EndpointId;
use iroh_dns::dns::DnsResolver;
pub use iroh_dns::dns::{N0_DNS_ENDPOINT_ORIGIN_PROD, N0_DNS_ENDPOINT_ORIGIN_STAGING};
use n0_future::boxed::BoxStream;
use tracing::{Instrument, debug, debug_span, trace};
use crate::{
Endpoint,
address_lookup::{
AddressLookup, AddressLookupBuilder, AddressLookupBuilderError,
Error as AddressLookupError, Item as AddressLookupItem,
},
endpoint::force_staging_infra,
};
/// Delays after which additional DNS lookup calls are issued.
///
/// Each query has its own timeout of 3s. This means that a lookup will finally
/// abort after 6 seconds.
pub(crate) const DNS_STAGGERING_MS: &[u64] = &[200, 300, 600, 1000, 2000, 3000];
/// DNS endpoint discovery
///
/// When asked to resolve a [`EndpointId`], this service performs a lookup in the Domain Name System (DNS).
///
/// It uses the [`Endpoint`]'s DNS resolver to query for `TXT` records under the domain
/// `_iroh.<z32-endpoint-id>.<origin-domain>`:
///
/// * `_iroh`: is the record name
/// * `<z32-endpoint-id>` is the [`EndpointId`] encoded in [`z-base-32`] format
/// * `<origin-domain>` is the endpoint origin domain as set in [`DnsAddressLookup::builder`].
///
/// Each TXT record returned from the query is expected to contain a string in the format `<name>=<value>`.
/// If a TXT record contains multiple character strings, they are concatenated first.
/// The supported attributes are:
/// * `relay=<url>`: The URL of the home relay server of the endpoint
///
/// The DNS resolver defaults to using the nameservers configured on the host system, but can be changed
/// with [`crate::endpoint::Builder::dns_resolver`].
///
/// [`z-base-32`]: https://philzimmermann.com/docs/human-oriented-base-32-encoding.txt
/// [`Endpoint`]: crate::Endpoint
#[derive(Debug)]
pub struct DnsAddressLookup {
origin_domain: String,
dns_resolver: DnsResolver,
}
/// Builder for [`DnsAddressLookup`].
///
/// See [`DnsAddressLookup::builder`].
#[derive(Debug)]
pub struct DnsAddressLookupBuilder {
origin_domain: String,
dns_resolver: Option<DnsResolver>,
}
impl DnsAddressLookupBuilder {
/// Sets the DNS resolver to use.
pub fn dns_resolver(mut self, dns_resolver: DnsResolver) -> Self {
self.dns_resolver = Some(dns_resolver);
self
}
/// Builds a [`DnsAddressLookup`] with the passed [`DnsResolver`].
pub fn build(self) -> DnsAddressLookup {
DnsAddressLookup {
dns_resolver: self.dns_resolver.unwrap_or_default(),
origin_domain: self.origin_domain,
}
}
}
impl DnsAddressLookup {
/// Creates a [`DnsAddressLookupBuilder`] that implements [`AddressLookupBuilder`].
pub fn builder(origin_domain: String) -> DnsAddressLookupBuilder {
DnsAddressLookupBuilder {
origin_domain,
dns_resolver: None,
}
}
/// Creates a new DNS address lookup using the `iroh.link` domain.
///
/// This uses the [`N0_DNS_ENDPOINT_ORIGIN_PROD`] domain.
///
/// When running with the environment variable `IROH_FORCE_STAGING_RELAYS`
/// set to any non empty value the [`N0_DNS_ENDPOINT_ORIGIN_STAGING`] domain
/// is used instead.
pub fn n0_dns() -> DnsAddressLookupBuilder {
if force_staging_infra() {
Self::builder(N0_DNS_ENDPOINT_ORIGIN_STAGING.to_string())
} else {
Self::builder(N0_DNS_ENDPOINT_ORIGIN_PROD.to_string())
}
}
}
impl AddressLookupBuilder for DnsAddressLookupBuilder {
fn into_address_lookup(
mut self,
endpoint: &Endpoint,
) -> Result<impl AddressLookup, AddressLookupBuilderError> {
if self.dns_resolver.is_none() {
self.dns_resolver = Some(endpoint.dns_resolver()?.clone());
}
Ok(self.build())
}
}
impl AddressLookup for DnsAddressLookup {
fn resolve(
&self,
endpoint_id: EndpointId,
) -> Option<BoxStream<Result<AddressLookupItem, AddressLookupError>>> {
let resolver = self.dns_resolver.clone();
let origin_domain = self.origin_domain.clone();
let span =
debug_span!("DnsAddressLookup", lookup_id=%endpoint_id.fmt_short(), %origin_domain);
let fut = async move {
trace!("starting DNS lookup");
let endpoint_info = resolver
.lookup_endpoint_by_id_staggered(&endpoint_id, &origin_domain, DNS_STAGGERING_MS)
.await
.inspect_err(|err| debug!("DNS lookup failed: {err:#}"))
.map_err(|e| AddressLookupError::from_err_any("dns", e))?;
debug!(info=?endpoint_info, "DNS lookup success");
Ok(AddressLookupItem::new(endpoint_info, "dns", None))
}
.instrument(span);
let stream = n0_future::stream::once_future(fut);
Some(Box::pin(stream))
}
}
+306
View File
@@ -0,0 +1,306 @@
//! An in-memory address lookup system to manually add endpoint addressing information.
//!
//! Often an application might get endpoint addressing information out-of-band in an
//! application-specific way. [`EndpointTicket`]'s are one common way used to achieve this.
//! This addressing information is often only usable for a limited time so needs to
//! be able to be removed again once you know it is no longer useful.
//!
//! This is where the [`MemoryLookup`] is useful: it allows applications to add and
//! retract endpoint addressing information that is otherwise out-of-band to iroh.
//!
//! [`EndpointTicket`]: https://docs.rs/iroh-tickets/latest/iroh_tickets/endpoint/struct.EndpointTicket.html
use std::{
collections::{BTreeMap, btree_map::Entry},
sync::{Arc, RwLock},
};
use iroh_base::EndpointId;
use n0_future::{
boxed::BoxStream,
stream::{self, StreamExt},
time::SystemTime,
};
use super::{AddressLookup, EndpointData, EndpointInfo, Error, Item};
/// An in-memory address lookup system to manually add endpoint addressing information.
///
/// Often an application might get endpoint addressing information out-of-band in an
/// application-specific way. [`EndpointTicket`]'s are one common way used to achieve this.
/// This addressing information is often only usable for a limited time so needs to
/// be able to be removed again once you know it is no longer useful.
///
/// This is where the [`MemoryLookup`] is useful: it allows applications to add and
/// retract endpoint addressing information that is otherwise out-of-band to iroh.
///
/// # Examples
///
/// ```no_run
/// # #[cfg(with_crypto_provider)] // Endpoint::bind needs a crypto provider
/// # {
/// use iroh::{
/// Endpoint, EndpointAddr, TransportAddr, address_lookup::memory::MemoryLookup,
/// endpoint::presets,
/// };
/// use iroh_base::SecretKey;
///
/// # #[tokio::main]
/// # async fn wrapper() -> n0_error::Result<()> {
/// // Create the Address Lookup and endpoint.
/// let address_lookup = MemoryLookup::new();
///
/// let _ep = Endpoint::builder(presets::N0)
/// .address_lookup(address_lookup.clone())
/// .bind()
/// .await?;
///
/// // Sometime later add a RelayUrl for our endpoint.
/// let id = SecretKey::generate().public();
/// // You can pass either `EndpointInfo` or `EndpointAddr` to `add_endpoint_info`.
/// address_lookup.add_endpoint_info(EndpointAddr {
/// id,
/// addrs: [TransportAddr::Relay("https://example.com".parse()?)]
/// .into_iter()
/// .collect(),
/// });
///
/// # Ok(())
/// # }
/// # }
/// ```
///
/// [`EndpointTicket`]: https://docs.rs/iroh-tickets/latest/iroh_tickets/endpoint/struct.EndpointTicket.html
#[derive(Debug, Clone)]
pub struct MemoryLookup {
endpoints: Arc<RwLock<BTreeMap<EndpointId, StoredEndpointInfo>>>,
provenance: &'static str,
}
impl Default for MemoryLookup {
fn default() -> Self {
Self {
endpoints: Default::default(),
provenance: Self::PROVENANCE,
}
}
}
#[derive(Debug)]
struct StoredEndpointInfo {
data: EndpointData,
last_updated: SystemTime,
}
impl MemoryLookup {
/// The provenance string for this Address Lookup implementation.
///
/// This is mostly used for debugging information and allows understanding the origin of
/// addressing information used by an iroh [`Endpoint`].
///
/// [`Endpoint`]: crate::Endpoint
pub const PROVENANCE: &'static str = "memory_lookup";
/// Creates a new empty Memory Lookup instance.
pub fn new() -> Self {
Self::default()
}
/// Creates a new Memory Lookup instance with the provided `provenance`.
///
/// The provenance is part of [`address_lookup::Item`]s returned from [`Self::resolve`].
/// It is mostly used for debugging information and allows understanding the origin of
/// addressing information used by an iroh [`Endpoint`].
///
/// [`Endpoint`]: crate::Endpoint
/// [`address_lookup::Item`]: crate::address_lookup::Item
pub fn with_provenance(provenance: &'static str) -> Self {
Self {
endpoints: Default::default(),
provenance,
}
}
/// Creates a Memory Lookup instance from endpoint addresses.
///
/// # Examples
///
/// ```no_run
/// # #[cfg(with_crypto_provider)] // Endpoint::bind needs a crypto provider
/// # {
/// use std::{net::SocketAddr, str::FromStr};
///
/// use iroh::{Endpoint, EndpointAddr, address_lookup::memory::MemoryLookup, endpoint::presets};
///
/// # fn get_addrs() -> Vec<EndpointAddr> {
/// # Vec::new()
/// # }
/// # #[tokio::main]
/// # async fn wrapper() -> n0_error::Result<()> {
/// // get addrs from somewhere
/// let addrs = get_addrs();
///
/// // create a MemoryLookup from the list of addrs.
/// let address_lookup = MemoryLookup::from_endpoint_info(addrs);
/// // create an endpoint with the memory lookup address_lookup
/// let endpoint = Endpoint::builder(presets::N0)
/// .address_lookup(address_lookup)
/// .bind()
/// .await?;
/// # Ok(())
/// # }
/// # }
/// ```
pub fn from_endpoint_info(infos: impl IntoIterator<Item = impl Into<EndpointInfo>>) -> Self {
let res = Self::default();
for info in infos {
res.add_endpoint_info(info);
}
res
}
/// Sets endpoint addressing information for the given endpoint ID.
///
/// This will completely overwrite any existing info for the endpoint.
///
/// Returns the [`EndpointData`] of the previous entry, or `None` if there was no previous
/// entry for this endpoint ID.
pub fn set_endpoint_info(
&self,
endpoint_info: impl Into<EndpointInfo>,
) -> Option<EndpointData> {
let last_updated = SystemTime::now();
let EndpointInfo { endpoint_id, data } = endpoint_info.into();
let mut guard = self.endpoints.write().expect("poisoned");
let previous = guard.insert(endpoint_id, StoredEndpointInfo { data, last_updated });
previous.map(|x| x.data)
}
/// Augments endpoint addressing information for the given endpoint ID.
///
/// The provided addressing information is combined with the existing info in the memory
/// lookup. Any new direct addresses are added to those already present while the
/// relay URL is overwritten.
pub fn add_endpoint_info(&self, endpoint_info: impl Into<EndpointInfo>) {
let last_updated = SystemTime::now();
let EndpointInfo { endpoint_id, data } = endpoint_info.into();
let mut guard = self.endpoints.write().expect("poisoned");
match guard.entry(endpoint_id) {
Entry::Occupied(mut entry) => {
let existing = entry.get_mut();
existing.data.add_addrs(data.addrs().cloned());
existing.data.set_user_data(data.user_data().cloned());
existing.last_updated = last_updated;
}
Entry::Vacant(entry) => {
entry.insert(StoredEndpointInfo { data, last_updated });
}
}
}
/// Returns endpoint addressing information for the given endpoint ID.
pub fn get_endpoint_info(&self, endpoint_id: EndpointId) -> Option<EndpointInfo> {
let guard = self.endpoints.read().expect("poisoned");
let info = guard.get(&endpoint_id)?;
Some(EndpointInfo::from_parts(endpoint_id, info.data.clone()))
}
/// Removes all endpoint addressing information for the given endpoint ID.
///
/// Any removed information is returned.
pub fn remove_endpoint_info(&self, endpoint_id: EndpointId) -> Option<EndpointInfo> {
let mut guard = self.endpoints.write().expect("poisoned");
let info = guard.remove(&endpoint_id)?;
Some(EndpointInfo::from_parts(endpoint_id, info.data))
}
}
impl AddressLookup for MemoryLookup {
fn publish(&self, _data: &EndpointData) {}
fn resolve(&self, endpoint_id: EndpointId) -> Option<BoxStream<Result<super::Item, Error>>> {
let guard = self.endpoints.read().expect("poisoned");
let info = guard.get(&endpoint_id);
match info {
Some(endpoint_info) => {
let last_updated = endpoint_info
.last_updated
.duration_since(SystemTime::UNIX_EPOCH)
.expect("time drift")
.as_micros() as u64;
let item = Item::new(
EndpointInfo::from_parts(endpoint_id, endpoint_info.data.clone()),
self.provenance,
Some(last_updated),
);
Some(stream::iter(Some(Ok(item))).boxed())
}
None => None,
}
}
}
#[cfg(all(test, with_crypto_provider))]
mod tests {
use iroh_base::{EndpointAddr, SecretKey, TransportAddr};
use n0_error::{Result, StackResultExt};
use super::*;
use crate::{Endpoint, endpoint::presets};
#[tokio::test]
async fn test_basic() -> Result {
let address_lookup = MemoryLookup::new();
let _ep = Endpoint::builder(presets::Minimal)
.address_lookup(address_lookup.clone())
.bind()
.await?;
let key = SecretKey::from_bytes(&[0u8; 32]);
let addr = EndpointAddr::from_parts(
key.public(),
[TransportAddr::Relay("https://example.com".parse()?)],
);
let user_data = Some("foobar".parse().unwrap());
let endpoint_info = EndpointInfo::from(addr.clone()).with_user_data(user_data.clone());
address_lookup.add_endpoint_info(endpoint_info.clone());
let back = address_lookup
.get_endpoint_info(key.public())
.context("no addr")?;
assert_eq!(back, endpoint_info);
assert_eq!(back.user_data(), user_data.as_ref());
assert_eq!(back.into_endpoint_addr(), addr);
let removed = address_lookup
.remove_endpoint_info(key.public())
.context("nothing removed")?;
assert_eq!(removed, endpoint_info);
let res = address_lookup.get_endpoint_info(key.public());
assert!(res.is_none());
Ok(())
}
#[tokio::test]
async fn test_provenance() -> Result {
let address_lookup = MemoryLookup::with_provenance("foo");
let key = SecretKey::from_bytes(&[0u8; 32]);
let addr = EndpointAddr::from_parts(
key.public(),
[TransportAddr::Relay("https://example.com".parse()?)],
);
address_lookup.add_endpoint_info(addr);
let mut stream = address_lookup.resolve(key.public()).unwrap();
let item = stream.next().await.unwrap()?;
assert_eq!(item.provenance(), "foo");
assert_eq!(
item.relay_urls().next(),
Some(&("https://example.com".parse()?))
);
Ok(())
}
}
+683
View File
@@ -0,0 +1,683 @@
//! An address lookup service which publishes and resolves endpoint information using a [pkarr] relay.
//!
//! Public-Key Addressable Resource Records, [pkarr], is a system which allows publishing
//! [DNS Resource Records] owned by a particular [`SecretKey`] under a name derived from its
//! corresponding [`PublicKey`], also known as the [`EndpointId`]. Additionally this pkarr
//! Resource Record is signed using the same [`SecretKey`], ensuring authenticity of the
//! record.
//!
//! Pkarr normally stores these records on the [Mainline DHT], but also provides two bridges
//! that do not require clients to directly interact with the DHT:
//!
//! - Resolvers are servers which expose the pkarr Resource Record under a domain name,
//! e.g. `o3dks..6uyy.dns.iroh.link`. This allows looking up the pkarr Resource Records
//! using normal DNS clients. These resolvers would normally perform lookups on the
//! Mainline DHT augmented with a local cache to improve performance.
//!
//! - Relays are servers which allow both publishing and looking up of the pkarr Resource
//! Records using HTTP PUT and GET requests. They will usually perform the publishing to
//! the Mainline DHT on behalf on the client as well as cache lookups performed on the DHT
//! to improve performance.
//!
//! [`PkarrPublisher`] filters published addresses: only relay addresses are published by default.
//! To change this behavior, use [`PkarrPublisherBuilder::addr_filter`] and set it to e.g. [`AddrFilter::unfiltered`].
//! This can be useful to enable publishing IP addresses if the iroh endpoint is reachable via public
//! IP addresses.
//!
//! For address lookup in iroh the pkarr Resource Records contain the addressing information,
//! providing endpoints which retrieve the pkarr Resource Record with enough detail
//! to contact the iroh endpoint.
//!
//! There are several Address Lookup's built on top of pkarr, which can be composed
//! to the application's needs:
//!
//! - [`PkarrPublisher`], which publishes to a pkarr relay server using HTTP.
//!
//! - [`PkarrResolver`], which resolves from a pkarr relay server using HTTP.
//!
//! - [`address_lookup::DnsAddressLookup`], which resolves from a DNS server.
//!
//! Mainline-DHT-based pkarr publishing/lookup lives in the
//! [`iroh-mainline-address-lookup`] crate.
//!
//! [`iroh-mainline-address-lookup`]: https://docs.rs/iroh-mainline-address-lookup
//! [pkarr]: https://pkarr.org
//! [DNS Resource Records]: https://en.wikipedia.org/wiki/Domain_Name_System#Resource_records
//! [Mainline DHT]: https://en.wikipedia.org/wiki/Mainline_DHT
//! [`SecretKey`]: crate::SecretKey
//! [`PublicKey`]: crate::PublicKey
//! [`EndpointId`]: crate::EndpointId
//! [`address_lookup::DnsAddressLookup`]: crate::address_lookup::DnsAddressLookup
//! [`N0` preset]: crate::endpoint::presets::N0
//! [`AddrFilter`]: crate::address_lookup::AddrFilter
//! [`AddrFilter::relay_only`]: crate::address_lookup::AddrFilter::relay_only
//! [`AddrFilter::unfiltered`]: crate::address_lookup::AddrFilter::unfiltered
//! [`PkarrPublisherBuilder::addr_filter`]: PkarrPublisherBuilder::addr_filter
use std::sync::Arc;
use iroh_base::{EndpointId, RelayUrl, SecretKey};
use iroh_dns::{
EncodingError,
endpoint_info::{AddrFilter, EndpointInfo},
pkarr::{SignedPacket, SignedPacketVerifyError},
};
use n0_error::{AnyError, anyerr, e, stack_error};
use n0_future::{
boxed::BoxStream,
task::{self, AbortOnDropHandle},
time::{self, Duration, Instant},
};
use n0_watcher::{Disconnected, Watchable, Watcher as _};
use tracing::{Instrument, debug, info_span, trace, warn};
use url::Url;
#[cfg(not(wasm_browser))]
use crate::dns::DnsResolver;
use crate::{
Endpoint,
address_lookup::{
AddressLookup, AddressLookupBuilder, AddressLookupBuilderError, EndpointData,
Error as AddressLookupError, Item as AddressLookupItem,
},
endpoint::force_staging_infra,
util::reqwest_client_builder,
};
#[allow(missing_docs)]
#[stack_error(derive, add_meta)]
#[non_exhaustive]
pub enum PkarrError {
#[error("Invalid public key")]
PublicKey {
#[error(std_err)]
source: iroh_base::KeyParsingError,
},
#[error("Packet failed to verify")]
Verify {
#[error(std_err)]
source: SignedPacketVerifyError,
},
#[error("Invalid relay URL")]
InvalidRelayUrl { url: RelayUrl },
#[error("Error sending http request")]
HttpSend { source: AnyError },
#[error("Error resolving http request")]
HttpRequest { status: http::StatusCode },
#[error("Http payload error")]
HttpPayload { source: AnyError },
#[error("EncodingError")]
Encoding { source: EncodingError },
}
impl From<PkarrError> for AddressLookupError {
fn from(err: PkarrError) -> Self {
AddressLookupError::from_err_any("pkarr", err)
}
}
/// The production pkarr relay run by [number 0].
///
/// This server is both a pkarr relay server as well as a DNS resolver, see the [module
/// documentation]. However it does not interact with the Mainline DHT, so is a more
/// central service. It is a reliable service to use for address lookup.
///
/// [number 0]: https://n0.computer
/// [module documentation]: crate::address_lookup::pkarr
pub const N0_DNS_PKARR_RELAY_PROD: &str = "https://dns.iroh.link/pkarr";
/// The testing pkarr relay run by [number 0].
///
/// This server operates similarly to [`N0_DNS_PKARR_RELAY_PROD`] but is not as reliable.
/// It is meant for more experimental use and testing purposes.
///
/// [number 0]: https://n0.computer
pub const N0_DNS_PKARR_RELAY_STAGING: &str = "https://staging-dns.iroh.link/pkarr";
/// Default TTL for the records in the pkarr signed packet.
///
/// The Time To Live (TTL) tells DNS caches how long to store a record. It is ignored by the
/// `iroh-dns-server`, e.g. as running on [`N0_DNS_PKARR_RELAY_PROD`], as the home server
/// keeps the records for the domain. When using the pkarr relay no DNS is involved and the
/// setting is ignored.
// TODO(flub): huh?
pub const DEFAULT_PKARR_TTL: u32 = 30;
/// Interval in which to republish the endpoint info even if unchanged: 5 minutes.
pub const DEFAULT_REPUBLISH_INTERVAL: Duration = Duration::from_secs(60 * 5);
/// Builder for [`PkarrPublisher`].
///
/// See [`PkarrPublisher::builder`].
#[derive(Debug)]
pub struct PkarrPublisherBuilder {
pkarr_relay: Url,
ttl: u32,
republish_interval: Duration,
filter: AddrFilter,
#[cfg(not(wasm_browser))]
dns_resolver: Option<DnsResolver>,
}
impl PkarrPublisherBuilder {
/// See [`PkarrPublisher::builder`].
fn new(pkarr_relay: Url) -> Self {
Self {
pkarr_relay,
ttl: DEFAULT_PKARR_TTL,
republish_interval: DEFAULT_REPUBLISH_INTERVAL,
filter: AddrFilter::relay_only(),
#[cfg(not(wasm_browser))]
dns_resolver: None,
}
}
/// See [`PkarrPublisher::n0_dns`].
fn n0_dns() -> Self {
let pkarr_relay = match force_staging_infra() {
true => N0_DNS_PKARR_RELAY_STAGING,
false => N0_DNS_PKARR_RELAY_PROD,
};
let pkarr_relay: Url = pkarr_relay.parse().expect("url is valid");
Self::new(pkarr_relay)
}
/// Sets the TTL (time-to-live) for published packets.
///
/// Default is [`DEFAULT_PKARR_TTL`].
pub fn ttl(mut self, ttl: u32) -> Self {
self.ttl = ttl;
self
}
/// Sets the interval after which packets are republished even if our endpoint info did not change.
///
/// Default is [`DEFAULT_REPUBLISH_INTERVAL`].
pub fn republish_interval(mut self, republish_interval: Duration) -> Self {
self.republish_interval = republish_interval;
self
}
/// Sets the DNS resolver to use for resolving the pkarr relay URL.
#[cfg(not(wasm_browser))]
pub fn dns_resolver(mut self, dns_resolver: DnsResolver) -> Self {
self.dns_resolver = Some(dns_resolver);
self
}
/// Sets the address filter to control which addresses are published to the pkarr server.
///
/// By default [`AddrFilter::relay_only`] is used. This avoids leaking IP addresses to the
/// public pkarr server.
///
/// However, enabling IP address publishing can be useful, e.g. when iroh runs on a machine
/// connected to the internet via public IP addresses without a firewall.
/// In such cases, publishing them can make dialing such endpoints via DNS or Pkarr lookup
/// faster, potentially skipping a relay connection altogether.
pub fn addr_filter(mut self, filter: AddrFilter) -> Self {
self.filter = filter;
self
}
/// Builds the [`PkarrPublisher`] with the passed secret key for signing packets.
///
/// This publisher will be able to publish [pkarr](https://pkarr.org) records for [`SecretKey`].
pub fn build(self, secret_key: SecretKey, tls_config: rustls::ClientConfig) -> PkarrPublisher {
PkarrPublisher::new(
secret_key,
self.pkarr_relay,
self.ttl,
self.republish_interval,
#[cfg(not(wasm_browser))]
self.dns_resolver.unwrap_or_default(),
tls_config,
self.filter,
)
}
}
impl AddressLookupBuilder for PkarrPublisherBuilder {
fn into_address_lookup(
mut self,
endpoint: &Endpoint,
) -> Result<impl AddressLookup, AddressLookupBuilderError> {
#[cfg(not(wasm_browser))]
if self.dns_resolver.is_none() {
self.dns_resolver = Some(endpoint.dns_resolver()?.clone());
}
let tls_config = endpoint.tls_config().clone();
Ok(self.build(endpoint.secret_key().clone(), tls_config))
}
}
/// Publisher of address lookup information to a [pkarr] relay.
///
/// This publisher uses HTTP to publish address lookup information to a pkarr relay
/// server, see the [module docs] for details.
///
/// This implements the [`AddressLookup`] trait to be used as an address lookup service. Note
/// that it only publishes address lookup information, for the corresponding resolver use
/// the [`PkarrResolver`] together with [`AddressLookupServices`].
///
/// By default this publisher only publishes the [`RelayUrl`], to avoid leaking IP addresses to
/// the public pkarr server. Which addresses are published is controlled by the [`AddrFilter`]
/// set via [`PkarrPublisherBuilder::addr_filter`].
///
/// [pkarr]: https://pkarr.org
/// [module docs]: crate::address_lookup::pkarr
/// [`RelayUrl`]: crate::RelayUrl
/// [`AddressLookupServices`]: super::AddressLookupServices
#[derive(derive_more::Debug, Clone)]
pub struct PkarrPublisher {
endpoint_id: EndpointId,
watchable: Watchable<Option<EndpointInfo>>,
addr_filter: AddrFilter,
_drop_guard: Arc<AbortOnDropHandle<()>>,
}
impl PkarrPublisher {
/// Returns a [`PkarrPublisherBuilder`] that publishes endpoint info to a [pkarr] relay at `pkarr_relay`.
///
/// If no further options are set, the pkarr publisher will use [`DEFAULT_PKARR_TTL`] as the
/// time-to-live value for the published packets, and it will republish Address Lookup information
/// every [`DEFAULT_REPUBLISH_INTERVAL`], even if the information is unchanged.
///
/// [`PkarrPublisherBuilder`] implements [`AddressLookupBuilder`], so it can be passed to [`address_lookup`].
/// It will then use the endpoint's secret key to sign published packets.
///
/// [`address_lookup`]: crate::endpoint::Builder::address_lookup
/// [pkarr]: https://pkarr.org
pub fn builder(pkarr_relay: Url) -> PkarrPublisherBuilder {
PkarrPublisherBuilder::new(pkarr_relay)
}
/// Creates a new [`PkarrPublisher`] with a custom TTL and republish intervals.
///
/// This allows creating the publisher with custom time-to-live values of the
/// [`SignedPacket`]s as well as a custom republish interval.
fn new(
secret_key: SecretKey,
pkarr_relay: Url,
ttl: u32,
republish_interval: Duration,
#[cfg(not(wasm_browser))] dns_resolver: DnsResolver,
tls_config: rustls::ClientConfig,
addr_filter: AddrFilter,
) -> Self {
debug!("creating pkarr publisher that publishes to {pkarr_relay}");
let endpoint_id = secret_key.public();
#[cfg(wasm_browser)]
let pkarr_client = PkarrRelayClient::new(pkarr_relay);
#[cfg(not(wasm_browser))]
let pkarr_client = PkarrRelayClient::new(pkarr_relay, tls_config, dns_resolver);
let watchable = Watchable::default();
let service = PublisherService {
ttl,
watcher: watchable.watch(),
secret_key,
pkarr_client,
republish_interval,
};
let join_handle = task::spawn(service.run().instrument(info_span!("pkarr_publish")));
Self {
watchable,
endpoint_id,
addr_filter,
_drop_guard: Arc::new(AbortOnDropHandle::new(join_handle)),
}
}
/// Creates a pkarr publisher which uses the [number 0] pkarr relay server.
///
/// This uses the pkarr relay server operated by [number 0], at
/// [`N0_DNS_PKARR_RELAY_PROD`].
///
/// When running with the environment variable
/// `IROH_FORCE_STAGING_RELAYS` set to any non empty value [`N0_DNS_PKARR_RELAY_STAGING`]
/// server is used instead.
///
/// [number 0]: https://n0.computer
pub fn n0_dns() -> PkarrPublisherBuilder {
PkarrPublisherBuilder::n0_dns()
}
/// Publishes the addressing information about this endpoint to a pkarr relay.
///
/// This is a nonblocking function, the actual update is performed in the background.
pub fn update_endpoint_data(&self, data: &EndpointData) {
let data = data.apply_filter(&self.addr_filter).into_owned();
let info = EndpointInfo::from_parts(self.endpoint_id, data);
self.watchable.set(Some(info)).ok();
}
}
impl AddressLookup for PkarrPublisher {
fn publish(&self, data: &EndpointData) {
self.update_endpoint_data(data);
}
}
/// Publish endpoint info to a pkarr relay.
#[derive(derive_more::Debug, Clone)]
struct PublisherService {
#[debug("SecretKey")]
secret_key: SecretKey,
#[debug("PkarrClient")]
pkarr_client: PkarrRelayClient,
watcher: n0_watcher::Direct<Option<EndpointInfo>>,
ttl: u32,
republish_interval: Duration,
}
impl PublisherService {
async fn run(mut self) {
let mut failed_attempts = 0;
let republish = time::sleep(Duration::MAX);
tokio::pin!(republish);
loop {
if !self.watcher.is_connected() {
break;
}
if let Some(info) = self.watcher.get() {
match self.publish_current(info).await {
Err(err) => {
failed_attempts += 1;
// Retry after increasing timeout
let retry_after = Duration::from_secs(failed_attempts);
republish.as_mut().reset(Instant::now() + retry_after);
warn!(
err = %format!("{err:#}"),
url = %self.pkarr_client.pkarr_relay_url ,
?retry_after,
%failed_attempts,
"Failed to publish to pkarr",
);
}
Ok(()) => {
failed_attempts = 0;
// Republish after fixed interval
republish
.as_mut()
.reset(Instant::now() + self.republish_interval);
}
}
}
// Wait until either the retry/republish timeout is reached, or the endpoint info changed.
tokio::select! {
res = self.watcher.updated() => match res {
Ok(_) => debug!("Publish endpoint info to pkarr (info changed)"),
Err(Disconnected { .. }) => break,
},
_ = &mut republish => debug!("Publish endpoint info to pkarr (interval elapsed)"),
}
}
}
async fn publish_current(&self, info: EndpointInfo) -> Result<(), PkarrError> {
debug!(
data = ?info.data,
pkarr_relay = %self.pkarr_client.pkarr_relay_url,
"Publishing endpoint info to pkarr"
);
let signed_packet = info
.to_pkarr_signed_packet(&self.secret_key, self.ttl)
.map_err(|err| e!(PkarrError::Encoding, err))?;
self.pkarr_client.publish(&signed_packet).await?;
trace!(
data = ?info.data,
pkarr_relay = %self.pkarr_client.pkarr_relay_url,
"Published endpoint info to pkarr"
);
Ok(())
}
}
/// Builder for [`PkarrResolver`].
///
/// See [`PkarrResolver::builder`].
#[derive(Debug)]
pub struct PkarrResolverBuilder {
pkarr_relay: Url,
#[cfg(not(wasm_browser))]
dns_resolver: Option<DnsResolver>,
}
impl PkarrResolverBuilder {
/// Sets the DNS resolver to use for resolving the pkarr relay URL.
#[cfg(not(wasm_browser))]
pub fn dns_resolver(mut self, dns_resolver: DnsResolver) -> Self {
self.dns_resolver = Some(dns_resolver);
self
}
/// Creates a [`PkarrResolver`] from this builder.
pub fn build(self, tls_config: rustls::ClientConfig) -> PkarrResolver {
#[cfg(wasm_browser)]
let pkarr_client = PkarrRelayClient::new(self.pkarr_relay);
#[cfg(not(wasm_browser))]
let pkarr_client = PkarrRelayClient::new(
self.pkarr_relay,
tls_config,
self.dns_resolver.unwrap_or_default(),
);
PkarrResolver { pkarr_client }
}
}
impl AddressLookupBuilder for PkarrResolverBuilder {
fn into_address_lookup(
mut self,
endpoint: &Endpoint,
) -> Result<impl AddressLookup, AddressLookupBuilderError> {
#[cfg(not(wasm_browser))]
if self.dns_resolver.is_none() {
self.dns_resolver = Some(endpoint.dns_resolver()?.clone());
}
let tls_config = endpoint.tls_config().clone();
Ok(self.build(tls_config))
}
}
/// Resolver of address lookup information from a [pkarr] relay.
///
/// The resolver uses HTTP to query address lookup information from a pkarr relay server,
/// see the [module docs] for details.
///
/// This implements the [`AddressLookup`] trait to be used as an address lookup service. Note
/// that it only resolves address lookup information, for the corresponding publisher use
/// the [`PkarrPublisher`] together with [`AddressLookupServices`].
///
/// [pkarr]: https://pkarr.org
/// [module docs]: crate::address_lookup::pkarr
/// [`AddressLookupServices`]: super::AddressLookupServices
#[derive(derive_more::Debug, Clone)]
pub struct PkarrResolver {
pkarr_client: PkarrRelayClient,
}
impl PkarrResolver {
/// Creates a new resolver builder using the pkarr relay server at the URL.
///
/// The builder implements [`AddressLookupBuilder`].
pub fn builder(pkarr_relay: Url) -> PkarrResolverBuilder {
PkarrResolverBuilder {
pkarr_relay,
#[cfg(not(wasm_browser))]
dns_resolver: None,
}
}
/// Creates a pkarr resolver builder which uses the [number 0] pkarr relay server.
///
/// This uses the pkarr relay server operated by [number 0] at
/// [`N0_DNS_PKARR_RELAY_PROD`].
///
/// When running with the environment variable `IROH_FORCE_STAGING_RELAYS`
/// set to any non empty value [`N0_DNS_PKARR_RELAY_STAGING`]
/// server is used instead.
///
/// [number 0]: https://n0.computer
pub fn n0_dns() -> PkarrResolverBuilder {
let pkarr_relay = match force_staging_infra() {
true => N0_DNS_PKARR_RELAY_STAGING,
false => N0_DNS_PKARR_RELAY_PROD,
};
let pkarr_relay: Url = pkarr_relay.parse().expect("url is valid");
Self::builder(pkarr_relay)
}
}
impl AddressLookup for PkarrResolver {
fn resolve(
&self,
endpoint_id: EndpointId,
) -> Option<BoxStream<Result<AddressLookupItem, AddressLookupError>>> {
let pkarr_client = self.pkarr_client.clone();
let fut = async move {
let signed_packet = pkarr_client.resolve(endpoint_id).await?;
let info = EndpointInfo::from_pkarr_signed_packet(&signed_packet)
.map_err(|err| AddressLookupError::from_err_any("pkarr", err))?;
let item = AddressLookupItem::new(info, "pkarr", None);
Ok(item)
};
let stream = n0_future::stream::once_future(fut);
Some(Box::pin(stream))
}
}
/// A [pkarr](https://pkarr.org) client to publish [`SignedPacket`]s to a pkarr relay.
///
/// [pkarr]: https://pkarr.org
#[derive(Debug, Clone)]
pub struct PkarrRelayClient {
http_client: reqwest::Client,
pkarr_relay_url: Url,
}
/// A builder for the [`PkarrRelayClient`]
#[derive(Debug, Clone)]
pub struct PkarrRelayClientBuilder {
pkarr_relay_url: Url,
#[cfg(not(wasm_browser))]
dns_resolver: DnsResolver,
#[cfg(not(wasm_browser))]
tls_config: rustls::ClientConfig,
}
impl PkarrRelayClientBuilder {
/// Build a [`PkarrRelayClient`].
pub fn build(self) -> PkarrRelayClient {
#[cfg(not(wasm_browser))]
let builder = reqwest_client_builder(self.tls_config, self.dns_resolver);
#[cfg(wasm_browser)]
let builder = reqwest_client_builder();
let http_client = builder.build().expect("failed to create reqwest client");
PkarrRelayClient {
http_client,
pkarr_relay_url: self.pkarr_relay_url,
}
}
}
impl PkarrRelayClient {
/// Creates a [`PkarrRelayClient`].
#[cfg(not(wasm_browser))]
pub fn new(
pkarr_relay_url: Url,
tls_config: rustls::ClientConfig,
dns_resolver: DnsResolver,
) -> Self {
let http_client = reqwest_client_builder(tls_config, dns_resolver)
.build()
.expect("failed to create reqwest client");
Self {
http_client,
pkarr_relay_url,
}
}
/// Creates a [`PkarrRelayClient`].
#[cfg(wasm_browser)]
pub fn new(pkarr_relay_url: Url) -> Self {
let http_client = reqwest_client_builder()
.build()
.expect("failed to create reqwest client");
Self {
http_client,
pkarr_relay_url,
}
}
/// Resolves a [`SignedPacket`] for the given [`EndpointId`].
pub async fn resolve(
&self,
endpoint_id: EndpointId,
) -> Result<SignedPacket, AddressLookupError> {
let mut url = self.pkarr_relay_url.clone();
url.path_segments_mut()
.map_err(|_| {
e!(PkarrError::InvalidRelayUrl {
url: self.pkarr_relay_url.clone().into()
})
})?
.push(&endpoint_id.to_z32());
let response = self
.http_client
.get(url)
.send()
.await
.map_err(|err| e!(PkarrError::HttpSend, anyerr!(err)))?;
if !response.status().is_success() {
return Err(e!(PkarrError::HttpRequest {
status: response.status()
})
.into());
}
let payload = response
.bytes()
.await
.map_err(|err| e!(PkarrError::HttpPayload, anyerr!(err)))?;
let packet = SignedPacket::from_relay_payload(&endpoint_id, &payload)
.map_err(|err| e!(PkarrError::Verify, err))?;
Ok(packet)
}
/// Publishes a [`SignedPacket`].
pub async fn publish(&self, signed_packet: &SignedPacket) -> Result<(), PkarrError> {
let mut url = self.pkarr_relay_url.clone();
url.path_segments_mut()
.map_err(|_| {
e!(PkarrError::InvalidRelayUrl {
url: self.pkarr_relay_url.clone().into()
})
})?
.push(&signed_packet.public_key().to_z32());
let response = self
.http_client
.put(url)
.body(signed_packet.to_relay_payload())
.send()
.await
.map_err(|err| e!(PkarrError::HttpSend, anyerr!(err)))?;
if !response.status().is_success() {
return Err(e!(PkarrError::HttpRequest {
status: response.status()
}));
}
Ok(())
}
}
+156
View File
@@ -0,0 +1,156 @@
//! Default values used in [`iroh`][`crate`]
/// The default QUIC port used by the Relay server to accept QUIC connections
/// for QUIC address discovery
///
/// The port is "QUIC" typed on a phone keypad.
pub use iroh_relay::defaults::DEFAULT_RELAY_QUIC_PORT;
use url::Url;
/// The default HTTP port used by the Relay server.
pub const DEFAULT_HTTP_PORT: u16 = 80;
/// The default HTTPS port used by the Relay server.
pub const DEFAULT_HTTPS_PORT: u16 = 443;
/// The default metrics port used by the Relay server.
pub const DEFAULT_METRICS_PORT: u16 = 9090;
/// Production configuration.
pub mod prod {
use iroh_relay::{RelayConfig, RelayMap};
use super::*;
use crate::RelayUrl;
/// Hostname of the default NA east relay.
pub const NA_EAST_RELAY_HOSTNAME: &str = "use1-1.relay.n0.iroh.link.";
/// Hostname of the default NA west relay.
pub const NA_WEST_RELAY_HOSTNAME: &str = "usw1-1.relay.n0.iroh.link.";
/// Hostname of the default EU relay.
pub const EU_RELAY_HOSTNAME: &str = "euc1-1.relay.n0.iroh.link.";
/// Hostname of the default Asia-Pacific relay.
pub const AP_RELAY_HOSTNAME: &str = "aps1-1.relay.n0.iroh.link.";
/// Get the default [`RelayMap`].
pub fn default_relay_map() -> RelayMap {
RelayMap::from_iter([
default_na_east_relay(),
default_na_west_relay(),
default_eu_relay(),
default_ap_relay(),
])
}
/// Get the default [`RelayConfig`] for NA east.
pub fn default_na_east_relay() -> RelayConfig {
// The default NA east relay server run by number0.
let url: Url = format!("https://{NA_EAST_RELAY_HOSTNAME}")
.parse()
.expect("default url");
RelayConfig::from(RelayUrl::from(url))
}
/// Get the default [`RelayConfig`] for NA west.
pub fn default_na_west_relay() -> RelayConfig {
// The default NA west relay server run by number0.
let url: Url = format!("https://{NA_WEST_RELAY_HOSTNAME}")
.parse()
.expect("default url");
RelayConfig::from(RelayUrl::from(url))
}
/// Get the default [`RelayConfig`] for EU.
pub fn default_eu_relay() -> RelayConfig {
// The default EU relay server run by number0.
let url: Url = format!("https://{EU_RELAY_HOSTNAME}")
.parse()
.expect("default_url");
RelayConfig::from(RelayUrl::from(url))
}
/// Get the default [`RelayConfig`] for Asia-Pacific.
pub fn default_ap_relay() -> RelayConfig {
// The default Asia-Pacific relay server run by number0.
let url: Url = format!("https://{AP_RELAY_HOSTNAME}")
.parse()
.expect("default_url");
RelayConfig::from(RelayUrl::from(url))
}
}
/// Staging configuration.
///
/// Used by tests and might have incompatible changes deployed
///
/// Note: we have staging servers in EU and NA, but no corresponding staging server for AP at this time.
pub mod staging {
use iroh_relay::{RelayConfig, RelayMap};
use super::*;
use crate::RelayUrl;
/// Hostname of the default NA relay.
pub const NA_EAST_RELAY_HOSTNAME: &str = "use1-1.staging-relay.n0.iroh.link.";
/// Hostname of the default EU relay.
pub const EU_RELAY_HOSTNAME: &str = "euc1-1.staging-relay.n0.iroh.link.";
/// Get the default [`RelayMap`].
pub fn default_relay_map() -> RelayMap {
RelayMap::from_iter([default_na_east_relay(), default_eu_relay()])
}
/// Get the default [`RelayConfig`] for NA east.
pub fn default_na_east_relay() -> RelayConfig {
// The default NA east relay server run by number0.
let url: Url = format!("https://{NA_EAST_RELAY_HOSTNAME}")
.parse()
.expect("default url");
RelayConfig::from(RelayUrl::from(url))
}
/// Get the default [`RelayConfig`] for EU.
pub fn default_eu_relay() -> RelayConfig {
// The default EU relay server run by number0.
let url: Url = format!("https://{EU_RELAY_HOSTNAME}")
.parse()
.expect("default_url");
RelayConfig::from(RelayUrl::from(url))
}
}
/// Contains all timeouts that we use in `iroh`.
pub(crate) mod timeouts {
use n0_future::time::Duration;
// Timeouts for net_report
/// Maximum duration to wait for a net_report.
pub(crate) const NET_REPORT_TIMEOUT: Duration = Duration::from_secs(10);
}
#[cfg(test)]
pub(crate) mod tests {
use std::time::Duration;
use n0_tracing_test::traced_test;
use super::staging::NA_EAST_RELAY_HOSTNAME;
use crate::dns::DnsResolver;
const TIMEOUT: Duration = Duration::from_secs(5);
const STAGGERING_DELAYS: &[u64] = &[200, 300];
#[tokio::test]
#[traced_test]
async fn test_dns_lookup_ipv4_ipv6() {
let resolver = DnsResolver::new();
let res: Vec<_> = resolver
.lookup_ipv4_ipv6_staggered(NA_EAST_RELAY_HOSTNAME, TIMEOUT, STAGGERING_DELAYS)
.await
.unwrap()
.collect();
assert!(!res.is_empty());
dbg!(res);
}
}
+4224
View File
File diff suppressed because it is too large. Load diff
+252
View File
@@ -0,0 +1,252 @@
use std::{
convert::Infallible,
net::{IpAddr, Ipv4Addr, Ipv6Addr, SocketAddr, SocketAddrV4, SocketAddrV6},
};
use n0_error::stack_error;
/// Options when configuring binding an IP socket.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct BindOpts {
/// Sets the network prefix length of the subnet for this interface.
///
/// The prefix length is used in the routing table that is built to decide where
/// datagrams without a specific source address are sent. If these number of leading
/// bits in the destination IP address match the same number of leading bits in this
/// bound socket address, then it matches the subnet and the datagram will be sent
/// here. Otherwise the next bound sockets will be checked for a subnet match. Sockets
/// are ordered from longest prefix to shortest prefix.
///
/// If no bound address has a matching subnet, the bind marked with
/// [`Self::is_default_route`] will be used.
///
/// Note that most datagrams belonging to a traffic flow are in response to an incoming
/// datagram. Those are usually sent on the same bound socket as they were received and
/// will not consult the routing table derived from these bound sockets to select the
/// socket on which they will be sent.
prefix_len: u8,
/// If set, binding this interface is required and any errors will abort the
/// initialization of the endpoint.
///
/// Defaults to `true`.
is_required: bool,
/// Whether this socket should be used as default route.
///
/// The default route is used for outgoing datagrams not belonging to an existing
/// traffic flow, which does not fit in any subnet of the bound sockets. It is assumed
/// this subnet has a gateway router to route such packets.
///
/// See [`Self::prefix_len`] for details of how such routing works.
is_default_route: Option<bool>,
}
impl Default for BindOpts {
fn default() -> Self {
Self {
prefix_len: 0,
is_required: true,
is_default_route: None,
}
}
}
impl BindOpts {
/// Sets the network prefix length of the subnet this interface is in.
///
///
/// The subnets of bound sockets are used to route outgoing datagrams not belonging to
/// an existing traffic flow to the socket they should be sent on. Subnets are ordered
/// from longest prefix length to shortest prefix length and the first subnet which
/// contains the destination IP address will be chosen. If no subnet matches but there
/// is a bound socket marked with [`Self::set_is_default_route`] then this socket will
/// be used. In this case it is assumed the attached subnet has a gateway router to
/// forward the datagram.
///
/// Defaults to `0`, which means *all* IP addresses will belong to the subnet of this
/// socket's address. If multiple sockets of the same address family (IPv4 or IPv6) are
/// bound with such a `/0` prefix the socket which will be chosen is undefined.
///
/// For IPv4 sockets the maximum prefix is `32`. For IPv6 the maximum prefix is `128`.
pub fn set_prefix_len(mut self, prefix_len: u8) -> Self {
self.prefix_len = prefix_len;
self
}
/// Returns the `prefix_len`, see [`Self::set_prefix_len`].
pub fn prefix_len(&self) -> u8 {
self.prefix_len
}
/// Sets whether bind errors are fatal for this socket.
///
/// If `false` and this socket fails to bind, the error will be silently ignored and the
/// endpoint will still be created.
///
/// Defaults to `true`.
pub fn set_is_required(mut self, is_required: bool) -> Self {
self.is_required = is_required;
self
}
/// Returns the value set by [`Self::set_is_required`].
pub fn is_required(&self) -> bool {
self.is_required
}
/// Sets whether this is a default route.
///
/// The default route is used for outgoing datagrams not belonging to an existing
/// traffic flow, which does not fit in any subnet of the bound sockets. It is assumed
/// this subnet has a gateway router to route such packets.
///
/// See [`Self::set_prefix_len`] for details on how this routing works.
///
/// If not set explicitly, then [`Self::is_default_route`] will return `true`
/// if the prefix length is set to `0` (the default) and `false` otherwise.
pub fn set_is_default_route(mut self, is_default_route: bool) -> Self {
self.is_default_route = Some(is_default_route);
self
}
/// Returns whether this is a default route.
///
/// If [`Self::set_is_default_route`] has been called then that value is returned.
/// Otherwise, returns `true` if the [`prefix_len`][`Self::prefix_len`] is `0` and
/// `false` otherwise.
pub fn is_default_route(&self) -> bool {
match self.is_default_route {
Some(is_default) => is_default,
None => self.prefix_len() == 0,
}
}
}
/// A simpler version of [`ToSocketAddrs`], that does not do any DNS resolution.
///
/// [`ToSocketAddrs`]: std::net::ToSocketAddrs
pub trait ToSocketAddr {
/// Error type on failed conversion.
type Err: std::error::Error;
/// Tries to convert this type to a [`SocketAddr`].
fn to_socket_addr(&self) -> Result<SocketAddr, Self::Err>;
}
impl ToSocketAddr for SocketAddr {
type Err = Infallible;
fn to_socket_addr(&self) -> Result<SocketAddr, Self::Err> {
Ok(*self)
}
}
impl ToSocketAddr for SocketAddrV4 {
type Err = Infallible;
fn to_socket_addr(&self) -> Result<SocketAddr, Self::Err> {
Ok(SocketAddr::V4(*self))
}
}
impl ToSocketAddr for SocketAddrV6 {
type Err = Infallible;
fn to_socket_addr(&self) -> Result<SocketAddr, Self::Err> {
Ok(SocketAddr::V6(*self))
}
}
impl ToSocketAddr for (IpAddr, u16) {
type Err = Infallible;
fn to_socket_addr(&self) -> Result<SocketAddr, Self::Err> {
let (ip, port) = *self;
match ip {
IpAddr::V4(ref a) => (*a, port).to_socket_addr(),
IpAddr::V6(ref a) => (*a, port).to_socket_addr(),
}
}
}
impl ToSocketAddr for (Ipv4Addr, u16) {
type Err = Infallible;
fn to_socket_addr(&self) -> Result<SocketAddr, Self::Err> {
let (ip, port) = *self;
SocketAddrV4::new(ip, port).to_socket_addr()
}
}
impl ToSocketAddr for (Ipv6Addr, u16) {
type Err = Infallible;
fn to_socket_addr(&self) -> Result<SocketAddr, Self::Err> {
let (ip, port) = *self;
SocketAddrV6::new(ip, port, 0, 0).to_socket_addr()
}
}
impl ToSocketAddr for (&str, u16) {
type Err = std::net::AddrParseError;
fn to_socket_addr(&self) -> Result<SocketAddr, Self::Err> {
let (host, port) = *self;
let addr = host.parse::<IpAddr>()?;
let addr = SocketAddr::new(addr, port);
Ok(addr)
}
}
impl ToSocketAddr for (String, u16) {
type Err = std::net::AddrParseError;
fn to_socket_addr(&self) -> Result<SocketAddr, Self::Err> {
(&*self.0, self.1).to_socket_addr()
}
}
impl ToSocketAddr for str {
type Err = std::net::AddrParseError;
fn to_socket_addr(&self) -> Result<SocketAddr, Self::Err> {
let addr = self.parse()?;
Ok(addr)
}
}
impl<T: ToSocketAddr + ?Sized> ToSocketAddr for &T {
type Err = T::Err;
fn to_socket_addr(&self) -> Result<SocketAddr, Self::Err> {
(**self).to_socket_addr()
}
}
impl ToSocketAddr for String {
type Err = std::net::AddrParseError;
fn to_socket_addr(&self) -> Result<SocketAddr, Self::Err> {
(**self).to_socket_addr()
}
}
#[allow(missing_docs)]
#[stack_error(derive, add_meta, from_sources)]
#[non_exhaustive]
pub enum InvalidSocketAddr {
#[error(transparent)]
AddrParse {
#[error(std_err)]
source: std::net::AddrParseError,
},
#[error("Invalid IP prefix length")]
InvalidPrefixLength {},
#[error(transparent)]
Infallible {
#[error(std_err)]
source: Infallible,
},
#[error("Only a single default address can be set per IP family")]
DuplicateDefaultAddr,
}
File diff suppressed because it is too large. Load diff
+171
View File
@@ -0,0 +1,171 @@
use std::pin::Pin;
use iroh_base::EndpointAddr;
use crate::endpoint::{connection::Connection, quic::VarInt};
type BoxFuture<'a, T> = Pin<Box<dyn Future<Output = T> + Send + 'a>>;
/// Outcome of [`EndpointHooks::before_connect`]
#[derive(Debug)]
pub enum BeforeConnectOutcome {
/// Accept the connect attempt.
Accept,
/// Reject the connect attempt.
Reject,
}
/// Outcome of [`EndpointHooks::after_handshake`]
#[derive(Debug)]
pub enum AfterHandshakeOutcome {
/// Accept the connection.
Accept,
/// Reject and close the connection.
///
/// See [`Connection::close`] for details on `error_code` and `reason`.
///
/// [`Connection::close`]: crate::endpoint::Connection::close
Reject {
/// Error code to send with the connection close frame.
error_code: VarInt,
/// Close reason to send with the connection close frame.
reason: Vec<u8>,
},
}
impl AfterHandshakeOutcome {
/// Returns [`Self::Accept`].
pub fn accept() -> Self {
Self::Accept
}
/// Returns [`Self::Reject`].
pub fn reject(&self, error_code: VarInt, reason: &[u8]) -> Self {
Self::Reject {
error_code,
reason: reason.to_vec(),
}
}
}
/// EndpointHooks intercept the connection establishment process of an [`Endpoint`].
///
/// Use [`Builder::hooks`] to install hooks onto an endpoint.
///
/// For each hook, all installed hooks are invoked in the order they were installed on
/// the endpoint builder. If a hook returns `Accept`, processing continues with the next
/// hook. If a hook returns `Reject`, processing is aborted and further hooks
/// are not invoked for this hook point.
///
/// ## Notes to implementers
///
/// As hooks are stored on the endpoint, you must make sure to never store an [`Endpoint`]
/// on the hook struct itself, as this would create reference counting loop and cause the
/// endpoint to never be dropped, leaking memory.
///
/// [`Endpoint`]: crate::Endpoint
/// [`Builder::hooks`]: crate::endpoint::Builder::hooks
pub trait EndpointHooks: std::fmt::Debug + Send + Sync {
/// Intercept outgoing connections before they are started.
///
/// This is called whenever a new outgoing connection is initiated via [`Endpoint::connect`]
/// or [`Endpoint::connect_with_opts`].
///
/// If any hook returns [`BeforeConnectOutcome::Reject`], the connection attempt is aborted
/// before any packets are sent to the remote.
///
/// [`Endpoint::connect`]: crate::Endpoint::connect
/// [`Endpoint::connect_with_opts`]: crate::Endpoint::connect_with_opts
fn before_connect<'a>(
&'a self,
_remote_addr: &'a EndpointAddr,
_alpn: &'a [u8],
) -> impl Future<Output = BeforeConnectOutcome> + Send + 'a {
async { BeforeConnectOutcome::Accept }
}
/// Intercept both incoming and outgoing connections once the TLS handshake has completed.
///
/// At this point in time, we know the remote's endpoint id and ALPN. If any hook returns
/// [`AfterHandshakeOutcome::Reject`], the connection is closed with the provided error code
/// and reason.
///
/// The hook receives the [`Connection`] by reference so that implementations can read any
/// information they need synchronously. Do not clone the [`Connection`] out of the hook:
/// holding a strong [`Connection`] handle keeps the connection alive and disables
/// close-on-drop, which the primary use sites of the connection might rely on. If you need to
/// keep a reference for later (for example to wait for the connection to close, or to
/// look up the connection in a map), call [`Connection::weak_handle`] and store the
/// resulting [`WeakConnectionHandle`] instead.
///
/// [`WeakConnectionHandle`]: crate::endpoint::WeakConnectionHandle
fn after_handshake<'a>(
&'a self,
_conn: &'a Connection,
) -> impl Future<Output = AfterHandshakeOutcome> + Send + 'a {
async { AfterHandshakeOutcome::accept() }
}
}
pub(crate) trait DynEndpointHooks: std::fmt::Debug + Send + Sync {
fn before_connect<'a>(
&'a self,
remote_addr: &'a EndpointAddr,
alpn: &'a [u8],
) -> BoxFuture<'a, BeforeConnectOutcome>;
fn after_handshake<'a>(&'a self, conn: &'a Connection) -> BoxFuture<'a, AfterHandshakeOutcome>;
}
impl<T: EndpointHooks> DynEndpointHooks for T {
fn before_connect<'a>(
&'a self,
remote_addr: &'a EndpointAddr,
alpn: &'a [u8],
) -> BoxFuture<'a, BeforeConnectOutcome> {
Box::pin(EndpointHooks::before_connect(self, remote_addr, alpn))
}
fn after_handshake<'a>(&'a self, conn: &'a Connection) -> BoxFuture<'a, AfterHandshakeOutcome> {
Box::pin(EndpointHooks::after_handshake(self, conn))
}
}
#[derive(Debug, Default)]
pub(crate) struct EndpointHooksList {
inner: Vec<Box<dyn DynEndpointHooks>>,
}
impl EndpointHooksList {
pub(super) fn push(&mut self, hook: impl EndpointHooks + 'static) {
let hook: Box<dyn DynEndpointHooks> = Box::new(hook);
self.inner.push(hook);
}
pub(super) async fn before_connect(
&self,
remote_addr: &EndpointAddr,
alpn: &[u8],
) -> BeforeConnectOutcome {
for hook in self.inner.iter() {
match hook.before_connect(remote_addr, alpn).await {
BeforeConnectOutcome::Accept => continue,
reject @ BeforeConnectOutcome::Reject => {
return reject;
}
}
}
BeforeConnectOutcome::Accept
}
pub(super) async fn after_handshake(&self, conn: &Connection) -> AfterHandshakeOutcome {
for hook in self.inner.iter() {
match hook.after_handshake(conn).await {
AfterHandshakeOutcome::Accept => continue,
reject @ AfterHandshakeOutcome::Reject { .. } => {
return reject;
}
}
}
AfterHandshakeOutcome::Accept
}
}
+184
View File
@@ -0,0 +1,184 @@
//! Presets allow configuring an endpoint quickly with a chosen set of defaults.
//!
//! # Example
//!
//! ```no_run
//! # #[cfg(with_crypto_provider)]
//! # {
//! # async fn wrapper() -> n0_error::Result {
//! use iroh::{Endpoint, RelayMode, Watcher, endpoint::presets};
//!
//! let endpoint = Endpoint::builder(presets::N0).bind().await?;
//! # let _ = endpoint;
//! # Ok(())
//! # }
//! # }
//! ```
use crate::endpoint::Builder;
/// A reusable bundle of endpoint [`Builder`] configuration.
pub trait Preset {
/// Applies the configuration to the passed in [`Builder`].
fn apply(self, builder: Builder) -> Builder;
}
/// An empty preset that doesn't set anything on the builder.
///
/// This doesn't set mandatory builder options, so using this in
/// `Endpoint::bind(presets::Empty)` will always fail.
///
/// However, it can be useful, if you want control over all mandatory options
/// yourself, by using `Endpoint::builder(presets::Empty)`.
///
/// If you prefer a minimal version that is guaranteed to work, see the
/// [`Minimal`] preset.
#[derive(Debug, Copy, Clone, Default)]
pub struct Empty;
impl Preset for Empty {
fn apply(self, builder: Builder) -> Builder {
builder
}
}
/// A preset that is almost empty, besides setting mandatory options.
///
/// At the moment the only mandatory option to set on the endpoint builder is
/// [`Builder::crypto_provider`]. This preset makes a choice for that based on
/// the current set of enabled features in iroh, which is why it's only available
/// with the `tls-ring` or `tls-aws-lc-rs` feature flag.
///
/// It uses either [ring] or [aws-lc-rs], depending on which feature is enabled
/// on iroh (preferring ring if both are enabled).
///
/// [ring]: rustls::crypto::ring::default_provider
/// [aws-lc-rs]: rustls::crypto::aws_lc_rs::default_provider
#[cfg(with_crypto_provider)]
#[derive(Debug, Copy, Clone, Default)]
pub struct Minimal;
#[cfg(with_crypto_provider)]
impl Preset for Minimal {
fn apply(self, mut builder: Builder) -> Builder {
use std::sync::Arc;
#[cfg(feature = "tls-ring")]
{
builder = builder.crypto_provider(Arc::new(rustls::crypto::ring::default_provider()));
}
#[cfg(all(feature = "tls-aws-lc-rs", not(feature = "tls-ring")))]
{
builder =
builder.crypto_provider(Arc::new(rustls::crypto::aws_lc_rs::default_provider()));
}
builder
}
}
/// Configures the endpoint to use the n0 defaults
///
/// Currently this consists of
/// - the DNS Address Lookup service.
/// - the default relay servers provided by Number 0.
/// - setting the [`rustls::crypto::CryptoProvider`] to [ring] or [aws-lc-rs], depending
/// on which feature is enabled in iroh (preferring ring if both are enabled).
///
/// Due to the last point, this preset is only available with the `tls-ring` or
/// `tls-aws-lc-rs` feature enabled.
/// If you want to set your own crypto provider, we recommend copying the
/// implementation of this preset into your own and setting the appropriate crypto
/// provider there.
///
/// The default address lookup service publishes to and resolves from the
/// n0.computer dns server `iroh.link`.
///
/// This is equivalent to adding a [`crate::address_lookup::PkarrPublisher`],
/// a [`crate::address_lookup::PkarrResolver`], and (outside browsers) a
/// [`crate::address_lookup::DnsAddressLookup`], all configured to use the
/// n0.computer dns server.
///
/// This will by default use [`N0_DNS_PKARR_RELAY_PROD`].
/// When in tests, or when the `test-utils` feature is enabled, this will use the
/// [`N0_DNS_PKARR_RELAY_STAGING`].
///
/// [ring]: rustls::crypto::ring::default_provider
/// [aws-lc-rs]: rustls::crypto::aws_lc_rs::default_provider
/// [`N0_DNS_PKARR_RELAY_PROD`]: crate::address_lookup::N0_DNS_PKARR_RELAY_PROD
/// [`N0_DNS_PKARR_RELAY_STAGING`]: crate::address_lookup::N0_DNS_PKARR_RELAY_STAGING
#[cfg(with_crypto_provider)]
#[derive(Debug, Copy, Clone, Default)]
pub struct N0;
#[cfg(with_crypto_provider)]
impl Preset for N0 {
fn apply(self, mut builder: Builder) -> Builder {
use crate::{
address_lookup::{PkarrPublisher, PkarrResolver},
endpoint::default_relay_mode,
};
builder = Minimal.apply(builder);
builder = builder.address_lookup(PkarrPublisher::n0_dns());
// Resolve using HTTPS requests to our DNS server's /pkarr path.
builder = builder.address_lookup(PkarrResolver::n0_dns());
// Additionally resolve using DNS queries outside browsers.
#[cfg(not(wasm_browser))]
{
builder = builder.address_lookup(crate::address_lookup::DnsAddressLookup::n0_dns());
}
builder = builder.relay_mode(default_relay_mode());
builder
}
}
/// Configures the endpoint to use the n0 defaults, but with relay mode disabled.
///
/// Currently this consists of
/// - setting `RelayMode::Disabled`
/// - the DNS Address Lookup service, that publishes IP addresses rather than
/// relay urls.
/// - setting the [`rustls::crypto::CryptoProvider`] to [ring] or [aws-lc-rs], depending
/// on which feature is enabled in iroh (preferring ring if both are enabled).
///
/// Due to the last point, this preset is only available with the `tls-ring` or
/// `tls-aws-lc-rs` feature enabled.
/// If you want to set your own crypto provider, we recommend copying the
/// implementation of this preset into your own and setting the appropriate crypto
/// provider there.
///
/// The default address lookup service publishes to and resolves from the
/// n0.computer dns server `iroh.link`.
///
/// This is equivalent to adding a [`crate::address_lookup::PkarrPublisher`],
/// a [`crate::address_lookup::PkarrResolver`], and (outside browsers) a
/// [`crate::address_lookup::DnsAddressLookup`], all configured to use the
/// n0.computer dns server.
///
/// This will by default use [`N0_DNS_PKARR_RELAY_PROD`].
/// When in tests, or when the `test-utils` feature is enabled, this will use the
/// [`N0_DNS_PKARR_RELAY_STAGING`].
///
/// [ring]: rustls::crypto::ring::default_provider
/// [aws-lc-rs]: rustls::crypto::aws_lc_rs::default_provider
/// [`N0_DNS_PKARR_RELAY_PROD`]: crate::address_lookup::N0_DNS_PKARR_RELAY_PROD
/// [`N0_DNS_PKARR_RELAY_STAGING`]: crate::address_lookup::N0_DNS_PKARR_RELAY_STAGING
#[cfg(with_crypto_provider)]
#[derive(Debug, Copy, Clone, Default)]
pub struct N0DisableRelay;
#[cfg(with_crypto_provider)]
impl Preset for N0DisableRelay {
fn apply(self, builder: Builder) -> Builder {
use crate::RelayMode;
N0.apply(builder).relay_mode(RelayMode::Disabled)
}
}
+716
View File
@@ -0,0 +1,716 @@
//! Exporting and encapsulating structs from noq
//!
//! Co-locates all iroh-noq exports.
//!
//! There are some structs that we use in particular ways, where we would like
//! to limit or expand how those structs are used in iroh. By encapsulating them
//! we can ensure the functionality needed to make iroh work.
#[cfg(feature = "qlog")]
use std::path::Path;
use std::{sync::Arc, time::Duration};
/// `noq` types that are used in the public iroh API.
// Each type is notated with the iroh type or noq type that uses it.
pub use noq::{
AcceptBi, // iroh::endpoint::Connection
AcceptUni, // iroh::endpoint::Connection
AckFrequencyConfig, // iroh::endpoint::quic::QuicTransportConfig
Closed, // iroh::endpoint::WeakConnectionHandle
ClosedStream, // iroh::protocol::AcceptError, noq::RecvStream, noq::SendStream
ConnectionError, // iroh::endpoint::ConnectError
ConnectionStats, // iroh::endpoint::Connection
Dir, // noq::StreamId
IdleTimeout, // iroh::endpoint::quic::QuicTransportConfig
MtuDiscoveryConfig, // iroh::endpoint::quic::QuicTransportConfig
OpenBi, // iroh::endpoint::Connection
OpenUni, // iroh::endpoint::Connection
PathStats, // iroh::socket::remote_map::remote_state::PathInfo
ReadDatagram, // iroh::endpoint::Connection
ReadError, // noq::RecvStream
ReadExactError, // noq::RecvStream
ReadToEndError, // noq::RecvStream
RecvStream, // noq::AcceptBi, noq::AcceptUni, noq::OpenBi, noq::OpenUni
ResetError, // noq::RecvStream
SendDatagram, // iroh::endpoint::Connection
SendDatagramError, // iroh::endpoint::Connection
SendStream, // noq::AcceptBi, noq::OpenUni
Side, // iroh::endpoint::Connection, noq::StreamId,
StoppedError, // noq::SendStream
StreamId, // noq::RecvStream
UnorderedRecvStream, // noq::RecvStream
VarInt, // various
VarIntBoundsExceeded, // noq::VarInt, noq::IdleTimeout
WriteError, // noq::SendStream
};
#[cfg(feature = "qlog")]
pub use noq::{QlogConfig, QlogFactory, QlogFileFactory};
/// `noq_proto` types that are used in the public iroh API.
// Each type is notated with the iroh type or noq type that uses it.
pub use noq_proto::{
ApplicationClose, // noq::ConnectionError
Chunk, // noq::RecvStream
ConnectError as QuicConnectError, // iroh::endpoint::ConnectWithOptsError
ConnectionClose, // noq::ConnectionError
DecryptedInitial, // iroh::endpoint::connection::Incoming
FrameStats, // noq::ConnectionStats
FrameType, // noq_proto::TransportError
IncomingAlpns, // iroh::endpoint::DecryptedInitial
PathId, // noq_proto::crypto::PacketKey
RttEstimator, // noq_proto::congestion::Controller
TimeSource, // iroh::endpoint::quic::ServerConfig
TokenLog, // noq::ValidationTokenConfig
TokenReuseError, // noq::TokenLog
TransportError, // noq::ConnectionError
TransportErrorCode, // noq_proto::TransportError
UdpStats, // noq::ConnectionStats
ValidationTokenConfig, // iroh::endpoint::quic::::ServerConfig
congestion::{
Controller, // iroh::endpoint::Connection
ControllerFactory, // iroh::endpoint::quic::QuicTransportConfig
ControllerMetrics, // noq_proto::congestion::Controller
},
crypto::{
CryptoError, // noq_proto::crypto::CryptoError, noq_proto::crypto::PacketKey
ExportKeyingMaterialError, // iroh::endpoint::Connection
HandshakeTokenKey, // iroh::endpoint::quic::ServerConfig
HeaderKey, // noq_proto::crypto::Keys
Keys, // noq_proto::crypto::Session
PacketKey, // noq_proto::crypto::Keys
UnsupportedVersion, // noq_proto::ConnectError
},
transport_parameters::TransportParameters, // noq_proto::crypot::ServerConfig
};
use tracing::warn;
use crate::socket::{
HEARTBEAT_INTERVAL, MAX_MULTIPATH_PATHS, MAX_QNT_ADDRESSES, PATH_MAX_IDLE_TIMEOUT,
};
/// Builder for a [`QuicTransportConfig`].
#[derive(Debug, Clone)]
pub struct QuicTransportConfigBuilder(noq::TransportConfig);
/// Parameters governing the core QUIC state machine
///
/// Default values should be suitable for most internet applications. Applications protocols which
/// forbid remotely-initiated streams should set `max_concurrent_bidi_streams` and
/// `max_concurrent_uni_streams` to zero.
///
/// In some cases, performance or resource requirements can be improved by tuning these values to
/// suit a particular application and/or network connection. In particular, data window sizes can be
/// tuned for a particular expected round trip time, link capacity, and memory availability. Tuning
/// for higher bandwidths and latencies increases worst-case memory consumption, but does not impair
/// performance at lower bandwidths and latencies. The default configuration is tuned for a 100Mbps
/// link with a 100ms round trip time.
///
/// Use the [`QuicTransportConfigBuilder`] to customize these tunable fields.
///
/// In iroh, the config has some specific default values that make iroh's holepunching work
/// well with QUIC multipath. Adjusting those settings may cause suboptimal usage.
///
/// Look at the following methods for more details:
/// - [`QuicTransportConfigBuilder::default_path_keep_alive_interval`]
/// - [`QuicTransportConfigBuilder::default_path_max_idle_timeout`]
/// - [`QuicTransportConfigBuilder::max_concurrent_multipath_paths`]
/// - [`QuicTransportConfigBuilder::max_remote_nat_traversal_addresses`]
///
/// # Examples
/// ```
/// use std::time::Duration;
///
/// use iroh::endpoint::QuicTransportConfig;
///
/// let _cfg = QuicTransportConfig::builder()
/// .send_observed_address_reports(true)
/// .build();
/// ```
#[derive(Debug, Clone)]
pub struct QuicTransportConfig(Arc<noq::TransportConfig>);
impl QuicTransportConfig {
/// Returns a default [`QuicTransportConfigBuilder`] that allows customizing
/// a [`QuicTransportConfig`].
pub fn builder() -> QuicTransportConfigBuilder {
QuicTransportConfigBuilder::new()
}
}
impl Default for QuicTransportConfig {
fn default() -> Self {
QuicTransportConfigBuilder::new().build()
}
}
impl QuicTransportConfig {
pub(crate) fn to_inner_arc(&self) -> Arc<noq::TransportConfig> {
self.0.clone()
}
}
impl QuicTransportConfigBuilder {
/// Creates a default [`QuicTransportConfigBuilder`].
fn new() -> Self {
let mut cfg = noq::TransportConfig::default();
// Override some transport config settings.
cfg.keep_alive_interval(Some(HEARTBEAT_INTERVAL));
cfg.default_path_keep_alive_interval(Some(HEARTBEAT_INTERVAL));
cfg.default_path_max_idle_timeout(Some(PATH_MAX_IDLE_TIMEOUT));
cfg.max_concurrent_multipath_paths(MAX_MULTIPATH_PATHS);
cfg.max_remote_nat_traversal_addresses(MAX_QNT_ADDRESSES);
cfg.server_handshake_migration(true);
Self(cfg)
}
/// Builds a [`QuicTransportConfig`] from the builder.
pub fn build(self) -> QuicTransportConfig {
QuicTransportConfig(Arc::new(self.0))
}
/// Maximum number of incoming bidirectional streams that may be open concurrently.
///
/// Must be nonzero for the peer to open any bidirectional streams.
///
/// Worst-case memory use is directly proportional to `max_concurrent_bidi_streams *
/// stream_receive_window`, with an upper bound proportional to `receive_window`.
pub fn max_concurrent_bidi_streams(mut self, value: VarInt) -> Self {
self.0.max_concurrent_bidi_streams(value);
self
}
/// Variant of `max_concurrent_bidi_streams` affecting unidirectional streams.
pub fn max_concurrent_uni_streams(mut self, value: VarInt) -> Self {
self.0.max_concurrent_uni_streams(value);
self
}
/// Maximum duration of inactivity to accept before timing out the connection.
///
/// The true idle timeout is the minimum of this and the peer's own max idle timeout. `None`
/// represents an infinite timeout. Defaults to 30 seconds.
///
/// **WARNING**: If a peer or its network path malfunctions or acts maliciously, an infinite
/// idle timeout can result in permanently hung futures!
///
/// ```
/// # use std::{convert::TryInto, time::Duration};
/// # use iroh::endpoint::{QuicTransportConfig, VarInt, VarIntBoundsExceeded};
/// # fn main() -> Result<(), VarIntBoundsExceeded> {
/// let mut builder = QuicTransportConfig::builder()
/// // Set the idle timeout as `VarInt`-encoded milliseconds
/// .max_idle_timeout(Some(VarInt::from_u32(10_000).into()));
///
/// // Set the idle timeout as a `Duration`
/// builder = builder.max_idle_timeout(Some(Duration::from_secs(10).try_into()?));
///
/// let _cfg = builder.build();
///
/// # Ok(())
/// # }
/// ```
pub fn max_idle_timeout(mut self, value: Option<IdleTimeout>) -> Self {
self.0.max_idle_timeout(value);
self
}
/// Maximum number of bytes the peer may transmit without acknowledgement on any one stream
/// before becoming blocked.
///
/// This should be set to at least the expected connection latency multiplied by the maximum
/// desired throughput. Setting this smaller than `receive_window` helps ensure that a single
/// stream doesn't monopolize receive buffers, which may otherwise occur if the application
/// chooses not to read from a large stream for a time while still requiring data on other
/// streams.
pub fn stream_receive_window(mut self, value: VarInt) -> Self {
self.0.stream_receive_window(value);
self
}
/// Maximum number of bytes the peer may transmit across all streams of a connection before
/// becoming blocked.
///
/// This should be set to at least the expected connection latency multiplied by the maximum
/// desired throughput. Larger values can be useful to allow maximum throughput within a
/// stream while another is blocked.
pub fn receive_window(mut self, value: VarInt) -> Self {
self.0.receive_window(value);
self
}
/// Maximum number of bytes to transmit to a peer without acknowledgment.
///
/// Provides an upper bound on memory when communicating with peers that issue large amounts of
/// flow control credit. Endpoints that wish to handle large numbers of connections robustly
/// should take care to set this low enough to guarantee memory exhaustion does not occur if
/// every connection uses the entire window.
pub fn send_window(mut self, value: u64) -> Self {
self.0.send_window(value);
self
}
/// Whether to implement fair queuing for send streams having the same priority.
///
/// When enabled, connections schedule data from outgoing streams having the same priority in a
/// round-robin fashion. When disabled, streams are scheduled in the order they are written to.
///
/// Note that this only affects streams with the same priority. Higher priority streams always
/// take precedence over lower priority streams.
///
/// Disabling fairness can reduce fragmentation and protocol overhead for workloads that use
/// many small streams.
pub fn send_fairness(mut self, value: bool) -> Self {
self.0.send_fairness(value);
self
}
/// Maximum reordering in packet number space before FACK style loss detection considers a
/// packet lost. Should not be less than 3, per RFC5681.
pub fn packet_threshold(mut self, value: u32) -> Self {
self.0.packet_threshold(value);
self
}
/// Maximum reordering in time space before time based loss detection considers a packet lost,
/// as a factor of RTT.
pub fn time_threshold(mut self, value: f32) -> Self {
self.0.time_threshold(value);
self
}
/// The RTT used before an RTT sample is taken.
pub fn initial_rtt(mut self, value: Duration) -> Self {
self.0.initial_rtt(value);
self
}
/// The initial value to be used as the maximum UDP payload size before running MTU discovery
/// (see [`QuicTransportConfigBuilder::mtu_discovery_config`]).
///
/// Must be at least 1200, which is the default, and known to be safe for typical internet
/// applications. Larger values are more efficient, but increase the risk of packet loss due to
/// exceeding the network path's IP MTU. If the provided value is higher than what the network
/// path actually supports, packet loss will eventually trigger black hole detection and bring
/// it down to [`QuicTransportConfigBuilder::min_mtu`].
pub fn initial_mtu(mut self, value: u16) -> Self {
self.0.initial_mtu(value);
self
}
/// The maximum UDP payload size guaranteed to be supported by the network.
///
/// Must be at least 1200, which is the default, and lower than or equal to
/// [`QuicTransportConfigBuilder::initial_mtu`].
///
/// Real-world MTUs can vary according to ISP, VPN, and properties of intermediate network links
/// outside of either endpoint's control. Extreme care should be used when raising this value
/// outside of private networks where these factors are fully controlled. If the provided value
/// is higher than what the network path actually supports, the result will be unpredictable and
/// catastrophic packet loss, without a possibility of repair. Prefer
/// [`QuicTransportConfigBuilder::initial_mtu`] together with
/// [`QuicTransportConfigBuilder::mtu_discovery_config`] to set a maximum UDP payload size that robustly
/// adapts to the network.
pub fn min_mtu(mut self, value: u16) -> Self {
self.0.min_mtu(value);
self
}
/// Specifies the MTU discovery config (see [`MtuDiscoveryConfig`] for details).
///
/// Enabled by default.
pub fn mtu_discovery_config(mut self, value: Option<MtuDiscoveryConfig>) -> Self {
self.0.mtu_discovery_config(value);
self
}
/// Pad UDP datagrams carrying application data to current maximum UDP payload size.
///
/// Disabled by default. UDP datagrams containing loss probes are exempt from padding.
///
/// Enabling this helps mitigate traffic analysis by network observers, but it increases
/// bandwidth usage. Without this mitigation precise plain text size of application datagrams as
/// well as the total size of stream write bursts can be inferred by observers under certain
/// conditions. This analysis requires either an uncongested connection or application datagrams
/// too large to be coalesced.
pub fn pad_to_mtu(mut self, value: bool) -> Self {
self.0.pad_to_mtu(value);
self
}
/// Specifies the ACK frequency config (see [`AckFrequencyConfig`] for details).
///
/// The provided configuration will be ignored if the peer does not support the acknowledgement
/// frequency QUIC extension.
///
/// Defaults to `None`, which disables controlling the peer's acknowledgement frequency. Even
/// if set to `None`, the local side still supports the acknowledgement frequency QUIC
/// extension and may use it in other ways.
pub fn ack_frequency_config(mut self, value: Option<AckFrequencyConfig>) -> Self {
self.0.ack_frequency_config(value);
self
}
/// Number of consecutive PTOs after which network is considered to be experiencing persistent congestion.
pub fn persistent_congestion_threshold(mut self, value: u32) -> Self {
self.0.persistent_congestion_threshold(value);
self
}
/// Period of inactivity before sending a keep-alive packet.
///
/// Keep-alive packets prevent an inactive but otherwise healthy connection from timing
/// out. They are important to keep NAT bindings alive and firewalls open.
///
/// The default is 5s, please be careful when modifying this as it may affect connection
/// stability. Must be set lower than the idle_timeout of both peers to be effective.
pub fn keep_alive_interval(mut self, value: Duration) -> Self {
self.0.keep_alive_interval(Some(value));
self
}
/// Maximum quantity of out-of-order crypto layer data to buffer.
pub fn crypto_buffer_size(mut self, value: usize) -> Self {
self.0.crypto_buffer_size(value);
self
}
/// Whether the implementation is permitted to set the spin bit on this connection.
///
/// This allows passive observers to easily judge the round trip time of a connection, which can
/// be useful for network administration but sacrifices a small amount of privacy.
pub fn allow_spin(mut self, value: bool) -> Self {
self.0.allow_spin(value);
self
}
/// Maximum number of incoming application datagram bytes to buffer, or None to disable
/// incoming datagrams.
///
/// The peer is forbidden to send single datagrams larger than this size. If the aggregate size
/// of all datagrams that have been received from the peer but not consumed by the application
/// exceeds this value, old datagrams are dropped until it is no longer exceeded.
pub fn datagram_receive_buffer_size(mut self, value: Option<usize>) -> Self {
self.0.datagram_receive_buffer_size(value);
self
}
/// Maximum number of outgoing application datagram bytes to buffer.
///
/// While datagrams are sent ASAP, it is possible for an application to generate data faster
/// than the link, or even the underlying hardware, can transmit them. This limits the amount of
/// memory that may be consumed in that case. When the send buffer is full and a new datagram is
/// sent, older datagrams are dropped until sufficient space is available.
pub fn datagram_send_buffer_size(mut self, value: usize) -> Self {
self.0.datagram_send_buffer_size(value);
self
}
/// How to construct new `congestion::Controller`s.
///
/// Typically the refcounted configuration of a `congestion::Controller`,
/// e.g. a `congestion::NewRenoConfig`.
///
/// # Example
/// ```
/// # use iroh::endpoint::QuicTransportConfig; use noq_proto::congestion; use std::sync::Arc;
/// let config = QuicTransportConfig::builder()
/// .congestion_controller_factory(Arc::new(congestion::NewRenoConfig::default()))
/// .build();
/// ```
pub fn congestion_controller_factory(
mut self,
factory: Arc<dyn ControllerFactory + Send + Sync + 'static>,
) -> Self {
self.0.congestion_controller_factory(factory);
self
}
/// Whether to use "Generic Segmentation Offload" to accelerate transmits, when supported by the
/// environment.
///
/// Defaults to `true`.
///
/// GSO dramatically reduces CPU consumption when sending large numbers of packets with the same
/// headers, such as when transmitting bulk data on a connection. However, it is not supported
/// by all network interface drivers or packet inspection tools. `noq-udp` will attempt to
/// disable GSO automatically when unavailable, but this can lead to spurious packet loss at
/// startup, temporarily degrading performance.
pub fn enable_segmentation_offload(mut self, enabled: bool) -> Self {
self.0.enable_segmentation_offload(enabled);
self
}
/// Whether to send observed address reports to peers.
///
/// This will aid peers in inferring their reachable address, which in most NATd networks
/// will not be easily available to them.
pub fn send_observed_address_reports(mut self, enabled: bool) -> Self {
self.0.send_observed_address_reports(enabled);
self
}
/// Whether to receive observed address reports from other peers.
///
/// Peers with the address discovery extension enabled that are willing to provide observed
/// address reports will do so if this transport parameter is set. In general, observed address
/// reports cannot be trusted. This, however, can aid the current endpoint in inferring its
/// reachable address, which in most NATd networks will not be easily available.
pub fn receive_observed_address_reports(mut self, enabled: bool) -> Self {
self.0.receive_observed_address_reports(enabled);
self
}
/// Enables the Multipath Extension for QUIC.
///
/// Setting this to any nonzero value will enable the Multipath Extension for QUIC,
/// <https://datatracker.ietf.org/doc/draft-ietf-quic-multipath/>.
///
/// The value provided specifies the number maximum number of paths this endpoint may open
/// concurrently when multipath is negotiated. For any path to be opened, the remote must
/// enable multipath as well.
///
/// Note: this method will ignore values less than the recommended 13 and will log a warning.
pub fn max_concurrent_multipath_paths(mut self, max_concurrent: u32) -> Self {
if max_concurrent < MAX_MULTIPATH_PATHS + 1 {
warn!(
"QuicTransportConfig::max_concurrent_multipath_paths must be at minimum {}, ignoring user supplied value",
MAX_MULTIPATH_PATHS + 1
);
return self;
}
self.0.max_concurrent_multipath_paths(max_concurrent);
self
}
/// Sets a default per-path maximum idle timeout.
///
/// If the path is idle for this long the path will be abandoned. Bear in mind this will
/// interact with the [`QuicTransportConfigBuilder::max_idle_timeout`], if the last path is
/// abandoned the entire connection will be closed.
///
/// Note: values higher than `PATH_MAX_IDLE_TIMEOUT` (15 seconds) are clamped and a warning is logged.
pub fn default_path_max_idle_timeout(mut self, timeout: Duration) -> Self {
if timeout > PATH_MAX_IDLE_TIMEOUT {
warn!(
"QuicTransportConfig::default_path_max_idle must be at most {:?}, clamping",
PATH_MAX_IDLE_TIMEOUT
);
self.0
.default_path_max_idle_timeout(Some(PATH_MAX_IDLE_TIMEOUT));
return self;
}
self.0.default_path_max_idle_timeout(Some(timeout));
self
}
/// Sets a default per-path keep alive interval.
///
/// Note that this does not interact with the connection-wide
/// [`QuicTransportConfigBuilder::keep_alive_interval`]. This setting will keep this path active,
/// [`QuicTransportConfigBuilder::keep_alive_interval`] will keep the connection active, with no
/// control over which path is used for this.
///
/// Note: this method will ignore values higher than the recommended 5 seconds and will log a warning.
pub fn default_path_keep_alive_interval(mut self, interval: Duration) -> Self {
if interval > HEARTBEAT_INTERVAL {
warn!(
"QuicTransportConfig::default_path_keep_alive must be at most {:?}, ignoring user supplied value",
HEARTBEAT_INTERVAL
);
return self;
}
self.0.default_path_keep_alive_interval(Some(interval));
self
}
/// Sets the maximum number of nat traversal addresses this endpoint allows the remote to
/// advertise.
///
/// Setting this to any nonzero value will enable Iroh's holepunching, loosely based in the Nat
/// Traversal Extension for QUIC, see
/// <https://www.ietf.org/archive/id/draft-seemann-quic-nat-traversal-02.html>
///
/// This implementation expects the multipath extension to be enabled as well. If not yet
/// enabled via [`Self::max_concurrent_multipath_paths`], a default value of
/// 8 will be used.
///
/// Note: this method will ignore values less than the recommended 8 and will log a warning.
pub fn max_remote_nat_traversal_addresses(mut self, max_addresses: u8) -> Self {
if max_addresses < MAX_MULTIPATH_PATHS as u8 {
warn!(
"QuicTransportConfig::max_remote_nat_traversal_addresses must be at least {}, ignoring user supplied value",
MAX_MULTIPATH_PATHS
);
return self;
}
self.0.max_remote_nat_traversal_addresses(max_addresses);
self
}
/// Configures qlog capturing by setting a [`QlogFactory`].
///
/// This assigns a [`QlogFactory`] that produces qlog capture configurations for
/// individual connections.
#[cfg(feature = "qlog")]
pub fn qlog_factory(mut self, factory: Arc<dyn QlogFactory>) -> Self {
self.0.qlog_factory(factory);
self
}
/// Configures qlog capturing through the `QLOGDIR` environment variable.
///
/// This uses [`QlogFileFactory::from_env`] to create a factory to write qlog traces
/// into the directory set through the `QLOGDIR` environment variable.
///
/// If `QLOGDIR` is not set, no traces will be written. If `QLOGDIR` is set to a path
/// that does not exist, it will be created.
///
/// The files will be prefixed with `prefix`.
#[cfg(feature = "qlog")]
pub fn qlog_from_env(mut self, prefix: &str) -> Self {
self.0.qlog_from_env(prefix);
self
}
/// Configures qlog capturing into a directory.
///
/// This uses [`QlogFileFactory`] to create a factory to write qlog traces into
/// the specified directory. The files will be prefixed with `prefix`.
#[cfg(feature = "qlog")]
pub fn qlog_from_path(mut self, path: impl AsRef<Path>, prefix: &str) -> Self {
self.0.qlog_from_path(path, prefix);
self
}
}
/// A builder for a [`ServerConfig`].
#[derive(Debug, Clone)]
pub struct ServerConfigBuilder {
inner: noq::ServerConfig,
transport: QuicTransportConfig,
}
/// Parameters governing incoming connections
///
/// Default values should be suitable for most internet applications.
///
/// Use a [`ServerConfigBuilder`] to adjust the default values.
///
/// To create a [`ServerConfig`] compatible with your [`Endpoint`] identity, use the [`Endpoint::create_server_config_builder`] method.
///
/// [`Endpoint`]: crate::Endpoint
/// [`Endpoint::create_server_config_builder`]: crate::Endpoint::create_server_config_builder
// Note: used in `iroh::endpoint::connection::Incoming::accept_with`
// This is new-typed since `noq::ServerConfig` takes a `TransportConfig`, which we new-type as a `QuicTransportConfig`
#[derive(Debug, Clone)]
pub struct ServerConfig(Arc<noq::ServerConfig>);
impl ServerConfig {
pub(crate) fn to_inner_arc(&self) -> Arc<noq::ServerConfig> {
self.0.clone()
}
}
impl ServerConfigBuilder {
/// Build a [`ServerConfig`] from a [`ServerConfigBuilder`].
pub fn build(self) -> ServerConfig {
ServerConfig(Arc::new(self.inner))
}
pub(crate) fn new(inner: noq::ServerConfig, transport: QuicTransportConfig) -> Self {
Self { inner, transport }
}
/// Sets a custom [`QuicTransportConfig`].
pub fn set_transport_config(mut self, transport: QuicTransportConfig) -> Self {
self.inner.transport_config(transport.to_inner_arc());
self.transport = transport;
self
}
/// Sets a custom [`ValidationTokenConfig`].
pub fn set_validation_token_config(mut self, validation_token: ValidationTokenConfig) -> Self {
self.inner.validation_token_config(validation_token);
self
}
/// Private key used to authenticate data included in handshake tokens
pub fn set_token_key(mut self, value: Arc<dyn HandshakeTokenKey>) -> Self {
self.inner.token_key(value);
self
}
/// Duration after a retry token was issued for which it's considered valid
///
/// Defaults to 15 seconds.
pub fn set_retry_token_lifetime(mut self, value: Duration) -> Self {
self.inner.retry_token_lifetime(value);
self
}
/// Maximum number of [`Incoming`] to allow to exist at a time.
///
/// An [`Incoming`] comes into existence when an incoming connection attempt
/// is received and stops existing when the application either accepts it or otherwise disposes
/// of it. While this limit is reached, new incoming connection attempts are immediately
/// refused. Larger values have greater worst-case memory consumption, but accommodate greater
/// application latency in handling incoming connection attempts.
///
/// The default value is set to 65536. With a typical Ethernet MTU of 1500 bytes, this limits
/// memory consumption from this to under 100 MiB--a generous amount that still prevents memory
/// exhaustion in most contexts.
///
/// [`Incoming`]: crate::endpoint::Incoming
pub fn set_max_incoming(mut self, max_incoming: usize) -> Self {
self.inner.max_incoming(max_incoming);
self
}
/// Maximum number of received bytes to buffer for each [`Incoming`].
///
/// An [`Incoming`] comes into existence when an incoming connection attempt
/// is received and stops existing when the application either accepts it or otherwise disposes
/// of it. This limit governs only packets received within that period, and does not include
/// the first packet. Packets received in excess of this limit are dropped, which may cause
/// 0-RTT or handshake data to have to be retransmitted.
///
/// The default value is set to 10 MiB--an amount such that in most situations a client would
/// not transmit that much 0-RTT data faster than the server handles the corresponding
/// [`Incoming`].
///
/// [`Incoming`]: crate::endpoint::Incoming
pub fn set_incoming_buffer_size(mut self, incoming_buffer_size: u64) -> Self {
self.inner.incoming_buffer_size(incoming_buffer_size);
self
}
/// Maximum number of received bytes to buffer for all [`Incoming`]
/// collectively.
///
/// An [`Incoming`] comes into existence when an incoming connection attempt
/// is received and stops existing when the application either accepts it or otherwise disposes
/// of it. This limit governs only packets received within that period, and does not include
/// the first packet. Packets received in excess of this limit are dropped, which may cause
/// 0-RTT or handshake data to have to be retransmitted.
///
/// The default value is set to 100 MiB--a generous amount that still prevents memory
/// exhaustion in most contexts.
///
/// [`Incoming`]: crate::endpoint::Incoming
pub fn set_incoming_buffer_size_total(mut self, incoming_buffer_size_total: u64) -> Self {
self.inner
.incoming_buffer_size_total(incoming_buffer_size_total);
self
}
/// Object to get current [`SystemTime`].
///
/// This exists to allow system time to be mocked in tests, or wherever else desired.
///
/// Defaults to [`noq::StdSystemTime`], which simply calls [`SystemTime::now()`](std::time::SystemTime::now).
///
/// [`SystemTime`]: std::time::SystemTime
pub fn set_time_source(mut self, time_source: Arc<dyn TimeSource>) -> Self {
self.inner.time_source(time_source);
self
}
}
Loaded 100 of 144 files, more files were not shown because too many files have changed in this diff. Show more