Compare commits

...
Author SHA1 Message Date
Aleksandr Bogomiakov d941cdac7a Change network protocol selections at runtime
Protocol selection previously belonged to tsng up and therefore to the
service command line. A blocked or degraded carrier could only be removed
by editing the service and restarting the whole agent. Make the ordered
selection a local per-network setting instead.

Add Agent::set_protocols and a local control request shared by CLI and tray.
Keep tsng protocols without arguments as the protocol catalog; naming a
network shows its selection and supplying a comma-separated list replaces
it. Accept exact names, full ids and unique prefixes using shared selector
rules; ambiguous selectors and invalid lists change nothing. An explicit
none selects control-only operation. Offline edits take the directory lock
and apply on the next start; live edits never write behind the daemon.

Register every built-in plugin and transport in the CLI. Treat up --protocol
as ordered initial defaults for networks without an override, allowing a
protocol absent from the unit to be enabled later. Retain the targeting of
unqualified startup options and allow explicitly scoped options to configure
initially disabled protocols. Persist NULL for inherited preferences and an
empty selection distinctly in SQLite schema 8; migration preserves existing
identities, secrets, signed records and counters.

Serialize changes with network lifecycle operations and commit preferences
before applying or announcing them. Retire only the affected network's data
links and E2E channels, cancel pending opens, and reject obsolete completion
results by attempt id. Clear peer data channels when authenticated protocol
capabilities change while retaining control sessions and topology. Replay
allocations and peer identities after asynchronously prepared plugins become
ready. A control-only start defers creation of the agent's single interface
until data is enabled; plugins receive a sink that can use it later. Automatic
carrier fallback continues to preserve E2E state; explicit policy changes
renegotiate it. Identity derivation and peer wire formats are unchanged.

Add a collapsible protocols row to each tray network tile, with a catalog
provided by the daemon, descriptions, immediately applied checkboxes and
preference-order buttons. Keep editing disabled until the worker refreshes
status so a second click cannot overwrite an edit using a stale snapshot.
Local IPC wire version is 20 for the new request and report fields; upgrade
and restart agent, CLI and tray together. Crate release versions remain unchanged; the requested RC tag is v0.1.0-rc.30.

Validation on Linux through nix develop:
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets: 509 passed, 2 existing ignored
  tests, with inherited TSUNAGI_CONTROL_SOCKET and TSUNAGI_CONTROL_GROUP removed

Tests use real iroh membership, SQLite, UDP, AWG and WireGuard encryption,
QUIC carriers and memory TUNs. Cover either endpoint-id role, full-size packet
delivery after switching, an unaffected second active network, saved selection
after restart, empty/invalid selections, first enablement without restart,
rapid changes racing old opens, live and offline CLI changes, startup-absent
protocols and preservation of signed state through migration. Earlier compile
and focused-test iterations caught initializer mistakes, a short test secret,
asynchronous plugin allocation readiness, and an inherited system socket
path; corrected before the successful final checks. Plain cargo was unavailable
in the host shell, so validation used the Nix toolchain. Manual tray interaction,
macOS/Windows, privileged host TUNs and public-network traversal were not run.
2026-10-12 16:19:18 +01:00
Aleksandr Bogomiakov e701986a39 Add ephemeral pairwise Amnezia transport and runtime fallback
Plain WG and QUIC can be blocked or severely degraded while the iroh control
connection still works. Add an awg carrier with one independent UDP socket,
WireGuard engine and randomized masking profile per network/peer attempt.
The smaller endpoint id negotiates fresh outer keys, PSK, header protection,
header ranges, padding and signature templates through private authenticated
control messages. Restart or fallback reentry generates a new profile;
ordinary rekeying preserves the current profile. No pair parameters enter
signed state, capabilities, persistent storage or public message events.
Private payloads and serialized control buffers redact and zeroize themselves.

Adapt the bounded packet masking engine from RunaRoo/AmneziaWG-RS at commit
51579a7d6e8882d1d8d6d00822f09e18a9423b73, retaining its Apache license and
provenance. Declare WTFPL AND Apache-2.0 in package metadata to cover both
new glue and the adapted source. Reuse boringtun instead of upstream's Node/global executors and
interface machinery. This is Tsunagi AWG wire version 1, with standard WG
headers restored before authentication; external Amnezia server compatibility
is not implemented. DNS/SIP signature generators vary fields instead of
replaying captures. Each attempt closes independently, bounds allocation,
queues, candidates and fragment assemblies, and supports the full 64 KiB
carrier contract using UDP packets at most 1200 bytes. Yield between fragment
batches so large messages do not starve local socket readers.

Keep carrier encryption separate from end-to-end tunnels so multihop relays
still forward opaque ciphertext. The system owns the sole interface, signed
addresses and admission. Plugins receive neither network secrets nor paths.
Expose a bounded opaque control sender at the transport boundary; core control
messages remain outside protocol crates. Optional STUN on the pair's actual
socket adds IPv4 public candidates in CLI direct/relay mode, with a finite
query deadline and an off/alternate-server option. Local mode remains offline.
Symmetric NAT, IPv6-only paths and arbitrary UDP blocking can still require
fallback; no AWG-over-QUIC or hidden AWG relay is introduced.

Default carrier order is wg, awg, wg-quic, tcp-tls. Advance the fallback cycle
after setup failures and sustained runtime loss, even after earlier cooldowns
expire. Authenticated delivery probes traverse the actual encrypted tunnel,
fit the interface MTU and never enter the TUN. Retire after a 20-second warmup
and at least 60% loss over at least 15 completed recent observations. Preserve
logical tunnels and control sessions while explicitly closing physical links,
including TCP reader/writer tasks. Retain the newest private probe settings
across independently queued control negotiation and inbound installation.
CLI/tray reports keep end-to-end protocol and actual carrier separate.

Control ALPN/version becomes tsunagi/ctrl/6 for the new private message;
upgrade all members together. Data ALPN/envelopes remain version 5. Network
identity derivation, transcript encoding, device identities, signed records,
saved network configuration and release versions are unchanged.

Validation on Linux through nix develop:
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets, with the inherited
  TSUNAGI_CONTROL_SOCKET override removed: 504 passed, 2 pre-existing ignored

Tests use real iroh membership, SQLite, UDP sockets and WireGuard encryption
with memory TUNs. Cover three independent pairs, full-size IP and carrier
frames, restart freshness/socket release, 80% carrier datagram loss, fallback
to real iroh, fresh retries with working delivery checks, unequal small MTUs,
malformed negotiations/profiles, role and resource bounds, probe forgery and
STUN parsing. Earlier compile/lint iterations and focused tests caught RNG
API/initializer mistakes, a retry assertion racing path installation, and a
large-fragment burst overflowing loopback buffers; corrected before these
successful checks. Normalized upstream license line endings for diff checks.
Public NAT/STUN connectivity, actual DPI effectiveness, WAN performance,
per-pair memory measurement, macOS/Windows, privileged TUN operations and
manual tray interaction were not exercised.
2026-10-11 13:52:10 +01:00
Aleksandr Bogomiakov ed28e86748 Derive default network ranges and switch overlapping networks
Every first network previously proposed 10.13.37.0/24. Independently created
networks therefore collided when a device joined more than one of them.
Use the existing stable NetworkId-derived /24 in 10/8 for every new network,
independent of activation order or peer availability. Legacy and explicit
signed ranges outrank the derived default; remaining range disagreements
still use author-id ordering. The derivation, identity labels, handshake and
signed record encoding stay unchanged. There is no creator, timestamp or vote.

Reserve ranges before starting network actors and serialise start, stop and
switching through the agent lifecycle lock. Refuse equal or nested overlaps
without changing a network's subnet locally. A late authenticated range that
collides suspends only its network before updating plugins or the interface.
Coalesced watch notifications let the agent clean up outside the network actor
and reconsider suspended networks when ranges change.

An explicit switch stops every blocker before activating the chosen network.
Suspended networks retain their startup request and resume whenever their
ranges become free, including switching from a wide range to a narrow one.
A manual stop cancels that request even for an already suspended network.
Persist preference order and enabling together in an SQLite transaction so
the user's selection survives restart; no database schema migration is needed.

CLI status and network listing show blocker names, ids, ranges and the start
command. Network start switches explicitly and reports suspended networks;
joining an overlapping network reports it as configured but inactive. The
tray adds conflict details, an amber icon, Switch to this network and Keep
stopped actions. Local IPC wire version is 19 for the extended reports, so
restart the agent and tray together. Release versions and peer wire formats
are unchanged. Update the user documentation and compatibility tests.

Validation on Linux through nix develop:
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets with TSUNAGI_CONTROL_SOCKET
  removed from the test environment: 486 passed, 2 pre-existing ignored tests
- final focused range_conflicts integration suite after making discovery
  fixtures deterministic: 6 passed

Earlier validation attempts caught outdated diagnostic assertions, a new
test helper scope error, and the inherited system control socket override.
Those were corrected before the successful runs. Plain cargo was unavailable
in the host shell, so validation used the Nix development toolchain. macOS,
Windows, privileged host TUN operations and manual tray interaction were not
exercised.
2026-10-11 11:58:27 +01:00
Aleksandr Bogomiakov 781e359775 Make tunnel and transport layers explicit in status
The tray called the end-to-end tunnel proto and the current route link, then put the tunnel handshake age beside the route. A valid wg tunnel over a tcp-tls transport consequently looked like contradictory state. The terminal status used similarly terse wording.\n\nThe tray now separates end-to-end from data path, renders direct paths as direct over TRANSPORT, keeps handshake or uptime beside the tunnel, and explains both layers in hover text. Terminal status uses the same end-to-end tunnel, data path, transport detail, and control path vocabulary. Routing, protocol selection, IPC, and wire behavior are unchanged.\n\nPresentation tests cover wg over tcp-tls, tcp-tls uptime, relayed and missing paths, a missing handshake, and offline devices. The WireGuard broadcast checksum test uses the current slice-array iterator spelling required by Clippy; its behavior is unchanged.\n\nChecks: cargo fmt --all -- --check; cargo clippy --locked --workspace --all-targets -- -D warnings; cargo test --locked --workspace --all-targets with TSUNAGI_CONTROL_SOCKET removed from the test environment so integration agents use their temporary control sockets. All passed on Linux.
2026-10-08 16:03:20 +01:00
Aleksandr Bogomiakov d5e02ec256 fixed gitignore 2026-10-08 15:35:38 +01:00
56 changed files with 6739 additions and 536 deletions

No files matched your search

+1
View File
@@ -1 +1,2 @@
/target
/.claude
Generated
+21
View File
@@ -6246,6 +6246,7 @@ dependencies = [
"tokio",
"tracing",
"tracing-subscriber",
"tsunagi-awg",
"tsunagi-core",
"tsunagi-tcp-tls",
"tsunagi-wg-quic",
@@ -6253,6 +6254,26 @@ dependencies = [
"windows-service",
]
[[package]]
name = "tsunagi-awg"
version = "0.1.0"
dependencies = [
"boringtun",
"bytes",
"chacha20 0.9.1",
"iroh",
"postcard",
"rand 0.10.3",
"serde",
"tempfile",
"tokio",
"tracing",
"tsunagi-core",
"tsunagi-wg-quic",
"tsunagi-wireguard",
"zeroize",
]
[[package]]
name = "tsunagi-core"
version = "0.1.0"
+34 -1
View File
@@ -81,7 +81,17 @@ tsng join -n home -s tsn1… # every other machine
tsng status # who is online, over which path
```
Every member gets an address in `10.13.37.0/24` and a name like `<host>.home`.
Every new network gets a stable pseudorandom `/24` in `10.0.0.0/8`, derived
from its network id. Its members agree on that range even when starting offline,
and each gets a name like `<host>.home`. Existing signed ranges are preserved.
If two configured networks have overlapping ranges, the agent runs only one.
`tsng status`, `tsng network` and the tray show which network blocks the
other. `tsng network start <name-or-id>` or the tray’s “Switch to this network”
button suspends the conflicting networks. They resume automatically when the
selected network stops; the selection survives restarting the agent. Explicitly
stopping a suspended network cancels its automatic resume.
Machines are called `tsunagi-<id>` until you rename them: `tsng id hostname laptop`.
### Exit nodes
@@ -97,6 +107,28 @@ The tray does the same with a checkbox. An exit node sees everything you send
through it, so use only members you trust.
[Details and per-OS notes](docs/exit-node.md).
The default data transport order is `wg,awg,wg-quic,tcp-tls`. AWG negotiates a
fresh masking profile and outer keys for each pair connection attempt, including
reentry after fallback. Sustained data loss triggers switching independently of
the iroh control connection. [Configuration and limits](docs/amnezia.md).
Change a network's protocol selection without editing its service or restarting:
```sh
tsng protocols # available protocols and settings
tsng protocols home # this device's choice for home
tsng protocols home awg,wg-quic # prefer AWG, retain QUIC as fallback
tsng protocols home none # control plane only
```
Exact network names, full ids and unique id prefixes work here. The choice is
local to this device, saved per network and applied immediately; an offline
change applies on the next start. In the tray, expand **protocols** on a network
tile to toggle protocols or change their preference order. Both endpoints need
a common protocol at the same wire version. `tsng up --protocol` supplies the
initial defaults for networks without a saved choice; it does not restrict
which built-in protocols can be enabled later.
## Platforms
| Platform | Build | CLI `tsng` | Tray | Exit node: offer / use | Packages |
@@ -174,6 +206,7 @@ The [threat model](docs/threat-model.md) says what this does and doesn't protect
[Architecture](docs/architecture.md) ·
[Protocol](docs/protocol.md) ·
[WireGuard data plane](docs/wireguard.md) ·
[Pairwise Amnezia transport](docs/amnezia.md) ·
[Routing](docs/routing.md) ·
[Exit nodes](docs/exit-node.md) ·
[Threat model](docs/threat-model.md) ·
+30
View File
@@ -0,0 +1,30 @@
[package]
name = "tsunagi-awg"
version = "0.1.0"
description = "Ephemeral pairwise Amnezia transport for Tsunagi."
edition.workspace = true
rust-version.workspace = true
license = "WTFPL AND Apache-2.0"
repository.workspace = true
[dependencies]
tsunagi-core = { path = "../tsunagi-core" }
tsunagi-wg-quic = { path = "../tsunagi-wg-quic" }
tsunagi-wireguard = { path = "../tsunagi-wireguard" }
iroh.workspace = true
tokio = { workspace = true, features = ["net", "time", "sync", "macros"] }
serde.workspace = true
postcard.workspace = true
bytes.workspace = true
tracing.workspace = true
rand = "0.10"
zeroize = { version = "1.9", features = ["derive"] }
boringtun = { version = "0.7.1", default-features = false }
chacha20 = "0.9"
[dev-dependencies]
tsunagi-core = { path = "../tsunagi-core", features = ["testing"] }
tempfile.workspace = true
[lints]
workspace = true
+201
View File
@@ -0,0 +1,201 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright 2026 Friderika Coulson (NanoBee)
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
+21
View File
@@ -0,0 +1,21 @@
# Amnezia packet engine
`src/masking.rs` is adapted from `src/core/amnezia.rs` in
https://github.com/RunaRoo/AmneziaWG-RS at
`51579a7d6e8882d1d8d6d00822f09e18a9423b73`, under Apache-2.0.
The upstream license is retained in `LICENSE-AmneziaWG-RS`.
The adaptation replaces configuration-file loading with bounded typed pair
profiles, updates RNG calls to rand 0.10, removes an unnecessary unwrap, and
does not embed upstream's Node, global executors, OS interface operations or
detached tasks. Noise/WireGuard is supplied by boringtun, already used by
Tsunagi. Each pair owns one socket and one cancellable packet-engine task.
This is Tsunagi AWG wire version 1, not a claim of compatibility with external
Amnezia servers. The masking engine substitutes message headers outside the
standard WireGuard state machine and restores them before Noise validation.
There are no remote file paths, shell hooks, or persisted pair profiles.
The adapted masking source remains Apache-2.0. Newly written Tsunagi code
follows the repository's WTFPL. This package declares both licenses in Cargo
metadata; the workspace license alone does not describe its copied source.
+16
View File
@@ -0,0 +1,16 @@
//! Pairwise, ephemeral Amnezia packet transport with independent end-to-end tunnels.
//! All negotiation travels over authenticated Tsunagi control sessions.
mod masking;
mod plugin;
pub mod profile;
mod session;
mod stun;
mod transport;
pub use plugin::AwgPlugin;
pub use transport::{AwgTransport, PairOverview};
/// Protocol identifier and default second-choice direct transport.
pub const AWG_PROTOCOL: &str = "awg";
/// Tsunagi AWG wire version, independent of software releases.
pub const AWG_WIRE_VERSION: u16 = 1;
+336
View File
@@ -0,0 +1,336 @@
// Adapted from RunaRoo/AmneziaWG-RS, commit 51579a7d6e8882d1d8d6d00822f09e18a9423b73.
// Apache-2.0; see LICENSE-AmneziaWG-RS and UPSTREAM.md.
#![allow(dead_code, missing_docs)]
// AmneziaWG configuration, header protection, and traffic shaping.
use bytes::{Bytes, BytesMut};
use chacha20::{
ChaCha20,
cipher::{KeyIvInit, StreamCipher},
};
use rand::RngExt;
pub const HEADER_PROTECTION_NONCE_SIZE: usize = 12;
pub const DEFAULT_UDP_WINDOW: usize = 500;
#[derive(Clone, Copy, Debug, Default, Eq, PartialEq)]
pub struct Range {
pub min: u32,
pub max: u32,
}
impl Range {
pub fn contains(&self, val: u32) -> bool {
val >= self.min && val <= self.max
}
pub fn random(&self) -> u32 {
if self.min >= self.max {
self.min
} else {
rand::rng().random_range(self.min..=self.max)
}
}
pub fn is_zero(&self) -> bool {
self.min == 0 && self.max == 0
}
}
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub enum AwgKind {
Initiation,
Response,
Cookie,
Transport,
}
impl AwgKind {
pub fn fixed_size(self) -> usize {
match self {
Self::Initiation => 148,
Self::Response => 92,
Self::Cookie => 64,
Self::Transport => 32,
}
}
fn standard_type(self) -> u32 {
match self {
Self::Initiation => 1,
Self::Response => 2,
Self::Cookie => 3,
Self::Transport => 4,
}
}
}
#[derive(Debug)]
pub struct DecodedPacket {
pub kind: AwgKind,
pub message: Bytes,
pub wire_len: usize,
}
pub struct AmneziaConfig {
pub jc: u32,
pub jmin: usize,
pub jmax: usize,
pub s1: usize,
pub s2: usize,
pub s3: usize,
pub s4: usize,
pub h1: Range,
pub h2: Range,
pub h3: Range,
pub h4: Range,
pub i1: String,
pub i2: String,
pub i3: String,
pub i4: String,
pub i5: String,
pub header_protection_key: Option<[u8; 32]>,
pub content_padding_addition: Range,
pub random_trailers: bool,
pub mtu: usize,
}
impl AmneziaConfig {
pub fn random_header(&self, range: &Range) -> u32 {
range.random()
}
pub fn random_padding(&self, len: usize) -> Vec<u8> {
let mut padding = vec![0u8; len];
if len > 0 {
rand::fill(&mut padding);
}
padding
}
fn shape(&self, kind: AwgKind) -> (usize, Range) {
match kind {
AwgKind::Initiation => (self.s1, self.h1),
AwgKind::Response => (self.s2, self.h2),
AwgKind::Cookie => (self.s3, self.h3),
AwgKind::Transport => (self.s4, self.h4),
}
}
pub fn encode(&self, kind: AwgKind, payload: Bytes, udp_window: usize) -> Bytes {
let (padding_len, _) = self.shape(kind);
let trailer_len = if kind != AwgKind::Transport && self.random_trailers {
let base = padding_len.saturating_add(payload.len());
let room = udp_window.max(DEFAULT_UDP_WINDOW).saturating_sub(base);
if room == 0 {
0
} else {
rand::rng().random_range(0..room)
}
} else {
0
};
let mut out = BytesMut::zeroed(padding_len + payload.len() + trailer_len);
if padding_len > 0 {
rand::fill(&mut out[..padding_len]);
}
out[padding_len..padding_len + payload.len()].copy_from_slice(&payload);
if trailer_len > 0 {
rand::fill(&mut out[padding_len + payload.len()..]);
}
if self.header_protection_key.is_some() {
let protected_len = if kind == AwgKind::Transport {
16
} else {
kind.fixed_size()
};
self.apply_header_cipher(&mut out, padding_len, protected_len);
}
out.freeze()
}
pub fn decode(&self, datagram: &Bytes) -> Option<DecodedPacket> {
if datagram.len() < 4 {
return None;
}
for kind in [
AwgKind::Initiation,
AwgKind::Response,
AwgKind::Cookie,
AwgKind::Transport,
] {
let (padding_len, headers) = self.shape(kind);
let minimum = padding_len + kind.fixed_size();
let size_ok = match kind {
AwgKind::Transport | AwgKind::Initiation => datagram.len() >= minimum,
AwgKind::Response | AwgKind::Cookie if self.random_trailers => {
datagram.len() >= minimum
}
AwgKind::Response | AwgKind::Cookie => datagram.len() == minimum,
};
if !size_ok || datagram.len() < padding_len + 4 {
continue;
}
let protected_header =
u32::from_le_bytes(datagram[padding_len..padding_len + 4].try_into().ok()?);
let candidate = if self.header_protection_key.is_some() {
protected_header ^ self.header_type_hash(datagram)?
} else {
protected_header
};
if !headers.contains(candidate) {
continue;
}
let message_len = match kind {
AwgKind::Transport | AwgKind::Initiation => datagram.len() - padding_len,
AwgKind::Response | AwgKind::Cookie => kind.fixed_size(),
};
let mut payload = BytesMut::from(&datagram[padding_len..padding_len + message_len]);
if self.header_protection_key.is_some() {
let protected_len = if kind == AwgKind::Transport {
16
} else {
kind.fixed_size()
};
self.apply_header_cipher_with_nonce(
&mut payload,
0,
protected_len,
&datagram[..HEADER_PROTECTION_NONCE_SIZE],
);
}
return Some(DecodedPacket {
kind,
message: payload.freeze(),
wire_len: datagram.len(),
});
}
None
}
pub fn prepare_transport_plaintext(&self, packet: Bytes, udp_window: usize) -> Bytes {
if packet.is_empty() {
return packet;
}
let requested = if !self.content_padding_addition.is_zero() {
self.content_padding_addition.random() as usize
} else if self.random_trailers {
let wire_without_padding = self.s4 + AwgKind::Transport.fixed_size() + packet.len();
let room = udp_window
.max(DEFAULT_UDP_WINDOW)
.saturating_sub(wire_without_padding);
if room == 0 {
0
} else {
rand::rng().random_range(0..room)
}
} else {
let aligned = (packet.len() + 15) & !15;
aligned.saturating_sub(packet.len())
};
let padding = self.clamp_padding_to_mtu(packet.len(), requested);
if padding == 0 {
return packet;
}
let mut out = BytesMut::with_capacity(packet.len() + padding);
out.extend_from_slice(&packet);
out.resize(packet.len() + padding, 0);
out.freeze()
}
pub fn transport_content_length(&self, plaintext: &[u8]) -> Option<usize> {
if plaintext.is_empty() || plaintext[0] == 0 {
return Some(0);
}
match plaintext[0] >> 4 {
4 if plaintext.len() >= 20 => {
let total = u16::from_be_bytes([plaintext[2], plaintext[3]]) as usize;
(total >= 20 && total <= plaintext.len()).then_some(total)
}
6 if plaintext.len() >= 40 => {
let payload = u16::from_be_bytes([plaintext[4], plaintext[5]]) as usize;
let total = 40 + payload;
(total <= plaintext.len()).then_some(total)
}
_ => None,
}
}
fn clamp_padding_to_mtu(&self, packet_len: usize, requested: usize) -> usize {
if self.mtu == 0 {
return requested;
}
let normalized = if packet_len > self.mtu {
packet_len % self.mtu
} else {
packet_len
};
requested.min(self.mtu.saturating_sub(normalized))
}
fn header_type_hash(&self, datagram: &Bytes) -> Option<u32> {
if datagram.len() < HEADER_PROTECTION_NONCE_SIZE {
return None;
}
let mut hash = [0u8; 4];
self.apply_keystream(&mut hash, &datagram[..HEADER_PROTECTION_NONCE_SIZE]);
Some(u32::from_le_bytes(hash))
}
fn apply_header_cipher(&self, packet: &mut [u8], offset: usize, len: usize) {
if packet.len() < HEADER_PROTECTION_NONCE_SIZE || packet.len() < offset + len {
return;
}
let nonce: [u8; HEADER_PROTECTION_NONCE_SIZE] =
match packet[..HEADER_PROTECTION_NONCE_SIZE].try_into() {
Ok(nonce) => nonce,
Err(_) => return,
};
self.apply_header_cipher_with_nonce(packet, offset, len, &nonce);
}
fn apply_header_cipher_with_nonce(
&self,
packet: &mut [u8],
offset: usize,
len: usize,
nonce: &[u8],
) {
if packet.len() < offset + len {
return;
}
self.apply_keystream(&mut packet[offset..offset + len], nonce);
}
fn apply_keystream(&self, data: &mut [u8], nonce: &[u8]) {
let Some(key) = self.header_protection_key.as_ref() else {
return;
};
if nonce.len() != HEADER_PROTECTION_NONCE_SIZE {
return;
}
let Ok(nonce) = <&[u8; HEADER_PROTECTION_NONCE_SIZE]>::try_from(nonce) else {
return;
};
let mut cipher = ChaCha20::new(key.into(), nonce.into());
cipher.apply_keystream(data);
}
}
impl Drop for AmneziaConfig {
fn drop(&mut self) {
use zeroize::Zeroize;
self.header_protection_key.zeroize();
}
}
+132
View File
@@ -0,0 +1,132 @@
//! End-to-end encryption remains independent of physical Amnezia pair links.
use crate::{AWG_PROTOCOL, AWG_WIRE_VERSION, AwgTransport};
use iroh::EndpointId;
use std::sync::Arc;
use tsunagi::dataplane::transport::SharedLink;
use tsunagi::dataplane::{IpPlugin, PluginCapability, PluginContext, PluginError, ProtocolOption};
use tsunagi::state::Ipv4Range;
use tsunagi::{BoxFuture, NetworkId};
use tsunagi_wg_quic::{NetworkOverview, WireguardConfig, WireguardPlugin};
/// AWG capability and an independent end-to-end WireGuard engine.
/// Pair transport keys/profiles never enter its durable end-to-end key store.
#[derive(Debug)]
pub struct AwgPlugin {
inner: Arc<WireguardPlugin>,
transport: Arc<AwgTransport>,
}
impl AwgPlugin {
/// Uses standard tunnel MTU/keepalive options; pair ports and profiles are ephemeral.
pub const OPTIONS: &'static [ProtocolOption] = &[
ProtocolOption {
key: "keepalive",
value: "SECONDS",
help: "end-to-end tunnel keepalive; 0 disables it",
default: Some("25"),
},
ProtocolOption {
key: "mtu",
value: "BYTES",
help: "maximum overlay packet size",
default: Some("1280"),
},
ProtocolOption {
key: "stun",
value: "HOST:PORT|off",
help: "UDP public mapping discovery; disabled in local reach mode",
default: Some("stun.l.google.com:19302"),
},
];
/// Opens the end-to-end engine, retaining separate pair transport state.
pub async fn open(
mut config: WireguardConfig,
transport: Arc<AwgTransport>,
) -> Result<Arc<Self>, PluginError> {
config.protocol = AWG_PROTOCOL;
Ok(Arc::new(Self {
inner: WireguardPlugin::open(config).await?,
transport,
}))
}
/// Reports end-to-end tunnel health and traffic counters.
pub fn overview(&self, network: NetworkId) -> Option<NetworkOverview> {
self.inner.overview(network)
}
}
impl IpPlugin for AwgPlugin {
fn protocol_id(&self) -> &str {
AWG_PROTOCOL
}
fn protocol_version(&self) -> u16 {
AWG_WIRE_VERSION
}
fn options(&self) -> &'static [ProtocolOption] {
Self::OPTIONS
}
fn attach(&self, context: PluginContext) {
self.inner.attach(context);
}
fn local_capability(
&self,
network: NetworkId,
) -> Result<Option<PluginCapability>, PluginError> {
self.inner.local_capability(network).map(|cap| {
cap.map(|mut cap| {
cap.protocol = AWG_PROTOCOL.into();
cap.version = AWG_WIRE_VERSION;
cap
})
})
}
fn on_network_activated(&self, network: NetworkId) {
self.inner.on_network_activated(network);
}
fn on_peer_capability(
&self,
network: NetworkId,
peer: EndpointId,
capability: &PluginCapability,
) -> Result<(), PluginError> {
if capability.protocol != AWG_PROTOCOL
|| capability.version != AWG_WIRE_VERSION
|| !capability.enabled
{
return Err(PluginError::Rejected("unsupported AWG capability".into()));
}
self.inner.on_peer_capability(network, peer, capability)?;
self.transport.allow_peer(network, peer);
Ok(())
}
fn on_address_allocation(
&self,
network: NetworkId,
range: Ipv4Range,
allocations: &[(EndpointId, std::net::Ipv4Addr)],
) {
self.inner
.on_address_allocation(network, range, allocations);
}
fn carry(&self, network: NetworkId, peer: EndpointId, packet: bytes::Bytes) -> bool {
self.inner.carry(network, peer, packet)
}
fn on_peer_link(&self, network: NetworkId, peer: EndpointId, link: SharedLink) {
self.inner.on_peer_link(network, peer, link);
}
fn on_peer_gone(&self, network: NetworkId, peer: EndpointId) {
self.transport.forget_peer(network, peer);
self.inner.on_peer_gone(network, peer);
}
fn on_network_deactivated(&self, network: NetworkId) {
self.transport.stop_network(network);
self.inner.on_network_deactivated(network);
}
fn on_network_forgotten(&self, network: NetworkId) {
self.inner.on_network_forgotten(network);
}
fn shutdown(&self) -> BoxFuture<'_, ()> {
Box::pin(async move {
self.transport.shutdown();
self.inner.shutdown().await;
})
}
}
+226
View File
@@ -0,0 +1,226 @@
//! Bounded profiles, regenerated for every pair connection attempt.
use crate::masking::{AmneziaConfig, AwgKind, Range};
use bytes::Bytes;
use serde::{Deserialize, Serialize};
/// An ephemeral masking profile. Secret fields are redacted and erased on drop.
#[derive(Clone, Serialize, Deserialize, zeroize::ZeroizeOnDrop)]
pub struct PairProfile {
id: [u8; 16],
prefix: u16,
headers: [[u32; 2]; 4],
header_key: [u8; 32],
pub(crate) psk: [u8; 32],
junk_count: u8,
junk_min: u16,
junk_max: u16,
padding: u16,
signature: u8,
}
impl std::fmt::Debug for PairProfile {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str("PairProfile([REDACTED])")
}
}
impl PairProfile {
/// Generates a new safe profile; no deterministic seed or durable state.
pub fn generate() -> Self {
let headers = std::array::from_fn(|i| {
let base = (i as u32) * 0x4000_0000 + rand::random_range(1024..0x3fff_0000);
[base, base + rand::random_range(64..1024)]
});
Self {
id: rand::random(),
prefix: rand::random_range(12..49),
headers,
header_key: rand::random(),
psk: rand::random(),
junk_count: rand::random_range(4..9),
junk_min: rand::random_range(24..65),
junk_max: rand::random_range(80..161),
padding: rand::random_range(16..65),
signature: rand::random_range(0..2),
}
}
/// Public attempt identifier, useful for distinguishing retries without revealing secrets.
pub fn id(&self) -> [u8; 16] {
self.id
}
/// Header key, for packet engines and protocol tests; never log this value.
pub fn header_key(&self) -> &[u8; 32] {
&self.header_key
}
/// Rejects invalid remote profiles before allocating packets or entering the codec.
pub fn validate(&self) -> Result<(), &'static str> {
if !(12..=48).contains(&self.prefix)
|| !(4..=8).contains(&self.junk_count)
|| self.junk_min < 24
|| self.junk_min > self.junk_max
|| self.junk_max > 160
|| self.padding > 64
|| self.signature > 1
|| self.header_key == [0; 32]
|| self.psk == [0; 32]
{
return Err("invalid AWG profile bounds");
}
for (i, [lo, hi]) in self.headers.iter().copied().enumerate() {
if lo < 5 || lo > hi || hi - lo > 1024 {
return Err("invalid AWG header range");
}
for [other_lo, other_hi] in &self.headers[..i] {
if lo <= *other_hi && *other_lo <= hi {
return Err("overlapping AWG header ranges");
}
}
}
Ok(())
}
fn config(&self) -> AmneziaConfig {
let range = |i: usize| Range {
min: self.headers[i][0],
max: self.headers[i][1],
};
AmneziaConfig {
jc: self.junk_count.into(),
jmin: self.junk_min.into(),
jmax: self.junk_max.into(),
s1: self.prefix.into(),
s2: self.prefix.into(),
s3: self.prefix.into(),
s4: self.prefix.into(),
h1: range(0),
h2: range(1),
h3: range(2),
h4: range(3),
i1: String::new(),
i2: String::new(),
i3: String::new(),
i4: String::new(),
i5: String::new(),
header_protection_key: Some(self.header_key),
content_padding_addition: Range {
min: 0,
max: self.padding.into(),
},
random_trailers: true,
mtu: 1050,
}
}
/// Masks one standard WireGuard message; invalid types and oversized packets are refused.
pub fn encode(&self, raw: &[u8]) -> Option<Bytes> {
let kind = match raw.get(..4)? {
[1, 0, 0, 0] => AwgKind::Initiation,
[2, 0, 0, 0] => AwgKind::Response,
[3, 0, 0, 0] => AwgKind::Cookie,
[4, 0, 0, 0] => AwgKind::Transport,
_ => return None,
};
if raw.len() < kind.fixed_size() || raw.len() > 1050 {
return None;
}
let config = self.config();
let range = match kind {
AwgKind::Initiation => config.h1,
AwgKind::Response => config.h2,
AwgKind::Cookie => config.h3,
AwgKind::Transport => config.h4,
};
let mut payload = raw.to_vec();
payload[..4].copy_from_slice(&range.random().to_le_bytes());
Some(config.encode(kind, Bytes::from(payload), 1200))
}
/// Unmasks a packet and restores standard headers before WireGuard validation.
pub fn decode(&self, wire: &[u8]) -> Option<Vec<u8>> {
if wire.len() > 1200 {
return None;
}
let decoded = self.config().decode(&Bytes::copy_from_slice(wire))?;
let (kind, len) = match decoded.kind {
AwgKind::Initiation => (1, 148),
AwgKind::Response => (2, 92),
AwgKind::Cookie => (3, 64),
AwgKind::Transport => (4, decoded.message.len()),
};
let mut message = decoded.message.get(..len)?.to_vec();
message[..4].copy_from_slice(&(kind as u32).to_le_bytes());
Some(message)
}
pub(crate) fn pad(&self, packet: Bytes) -> Bytes {
self.config().prepare_transport_plaintext(packet, 1200)
}
/// Generates structured signature packets with fresh fields, never static captures.
/// These target the other Tsunagi peer, not DNS or SIP services.
pub fn signature_packets(&self) -> Vec<Vec<u8>> {
if self.signature == 0 {
// A complete DNS query. Counts, label lengths and qtype remain valid.
let label = format!("n{:016x}", rand::random::<u64>());
let mut packet = Vec::from(rand::random::<u16>().to_be_bytes());
packet.extend_from_slice(&[1, 0, 0, 1, 0, 0, 0, 0, 0, 0]);
packet.push(label.len() as u8);
packet.extend_from_slice(label.as_bytes());
packet.extend_from_slice(b"\x07example\x03net\0\0");
packet.push(if rand::random() { 1 } else { 28 });
packet.extend_from_slice(&[0, 1]);
vec![packet]
} else {
let id = rand::random::<u64>();
vec![format!("OPTIONS sip:n{id:x}@example.net SIP/2.0\r\nVia: SIP/2.0/UDP example.net;branch=z9hG4bK{id:x}\r\nMax-Forwards: 70\r\nFrom: <sip:a@example.net>;tag={id:x}\r\nTo: <sip:n{id:x}@example.net>\r\nCall-ID: {id:x}@example.net\r\nCSeq: 1 OPTIONS\r\nContent-Length: 0\r\n\r\n").into_bytes()]
}
}
pub(crate) fn junk_packets(&self) -> Vec<Vec<u8>> {
(0..self.junk_count)
.map(|_| {
let mut packet = vec![0; rand::random_range(self.junk_min..=self.junk_max).into()];
rand::fill(&mut packet[..]);
packet
})
.collect()
}
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
#[test]
fn unsafe_peer_profiles_are_rejected_before_the_codec() {
let valid = PairProfile::generate();
let mut bad = valid.clone();
bad.prefix = 11;
assert!(bad.validate().is_err());
let mut bad = valid.clone();
bad.headers[1] = bad.headers[0];
assert!(bad.validate().is_err());
let mut bad = valid.clone();
bad.junk_max = u16::MAX;
assert!(bad.validate().is_err());
let mut bad = valid.clone();
bad.padding = u16::MAX;
assert!(bad.validate().is_err());
let mut bad = valid.clone();
bad.psk = [0; 32];
assert!(bad.validate().is_err());
let mut bad = valid.clone();
bad.signature = 2;
assert!(bad.validate().is_err());
}
#[test]
fn signature_templates_have_consistent_framing_and_fresh_fields() {
let mut profile = PairProfile::generate();
profile.signature = 0;
let dns = profile.signature_packets().remove(0);
assert_eq!(&dns[2..12], &[1, 0, 0, 1, 0, 0, 0, 0, 0, 0]);
let mut at = 12;
while dns[at] != 0 {
at += 1 + usize::from(dns[at]);
}
assert_eq!(at + 5, dns.len());
assert_eq!(&dns[dns.len() - 2..], &[0, 1]);
profile.signature = 1;
let sip = profile.signature_packets().remove(0);
assert!(sip.starts_with(b"OPTIONS sip:"));
assert!(sip.ends_with(b"Content-Length: 0\r\n\r\n"));
assert_ne!(sip, profile.signature_packets().remove(0));
}
}
+402
View File
@@ -0,0 +1,402 @@
//! One cancellable actor and UDP socket per pair. All payload and health frames
//! are inside the Amnezia-masked authenticated WireGuard tunnel.
use crate::profile::PairProfile;
use boringtun::noise::{Tunn, TunnResult};
use bytes::Bytes;
use iroh::EndpointId;
use std::collections::{HashMap, VecDeque};
use std::net::{Ipv4Addr, SocketAddr};
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};
use tokio::net::UdpSocket;
use tokio::sync::{mpsc, watch};
use tsunagi::dataplane::transport::{PacketLink, TransportError};
use tsunagi::{BoxFuture, NetworkId};
pub(crate) const MAX_PAYLOAD: usize = tsunagi::config::MAX_DATA_DATAGRAM;
const CHUNK: usize = 700;
const QUEUE: usize = 64;
const WINDOW: Duration = Duration::from_secs(20);
pub(crate) struct PairLink {
network: NetworkId,
peer: EndpointId,
pub(crate) attempt: [u8; 16],
tx: mpsc::Sender<Bytes>,
rx: tokio::sync::Mutex<mpsc::Receiver<Bytes>>,
stop: watch::Sender<bool>,
closed: AtomicBool,
remote: Mutex<Option<SocketAddr>>,
reason: Mutex<Option<String>>,
}
impl std::fmt::Debug for PairLink {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("AwgPair")
.field("network", &self.network)
.field("peer", &self.peer)
.finish_non_exhaustive()
}
}
impl PairLink {
pub(crate) fn has_authenticated_path(&self) -> bool {
match self.remote.lock() {
Ok(remote) => remote.is_some(),
Err(_) => false,
}
}
}
impl PacketLink for PairLink {
fn network(&self) -> NetworkId {
self.network
}
fn peer(&self) -> EndpointId {
self.peer
}
fn max_datagram_size(&self) -> usize {
MAX_PAYLOAD
}
fn send(&self, payload: Bytes) -> Result<(), TransportError> {
if self.is_closed() {
return Err(TransportError::Closed);
}
if payload.len() > MAX_PAYLOAD {
return Err(TransportError::TooLarge {
size: payload.len(),
limit: MAX_PAYLOAD,
});
}
self.tx
.try_send(payload)
.map_err(|_| TransportError::Other("AWG send queue full or closed".into()))
}
fn recv(&self) -> BoxFuture<'_, Option<Bytes>> {
Box::pin(async move {
let mut stop = self.stop.subscribe();
if *stop.borrow() {
return None;
}
let mut rx = self.rx.lock().await;
tokio::select! {biased; _=stop.changed()=>None,packet=rx.recv()=>packet}
})
}
fn closed(&self) -> BoxFuture<'_, ()> {
Box::pin(async move {
let mut stop = self.stop.subscribe();
if !*stop.borrow() {
let _ = stop.changed().await;
}
})
}
fn is_closed(&self) -> bool {
self.closed.load(Ordering::Acquire) || *self.stop.borrow()
}
fn close(&self) {
self.stop.send_replace(true);
}
fn failure_reason(&self) -> Option<String> {
match self.reason.lock() {
Ok(reason) => reason.clone(),
Err(_) => None,
}
}
fn path_description(&self) -> String {
match self.remote.lock() {
Ok(addr) => match *addr {
Some(addr) => format!("awg UDP via {addr}"),
None => "awg UDP, establishing".into(),
},
Err(_) => "awg UDP".into(),
}
}
}
impl Drop for PairLink {
fn drop(&mut self) {
self.stop.send_replace(true);
}
}
pub(crate) fn start(
network: NetworkId,
peer: EndpointId,
socket: UdpSocket,
tunn: Tunn,
profile: PairProfile,
candidates: Vec<SocketAddr>,
parent: watch::Receiver<bool>,
) -> Arc<PairLink> {
let (tx, rx) = mpsc::channel(QUEUE);
let (in_tx, in_rx) = mpsc::channel(QUEUE);
let (stop, _) = watch::channel(false);
let link = Arc::new(PairLink {
network,
peer,
attempt: profile.id(),
tx,
rx: tokio::sync::Mutex::new(in_rx),
stop: stop.clone(),
closed: AtomicBool::new(false),
remote: Mutex::new(None),
reason: Mutex::new(None),
});
let weak = Arc::downgrade(&link);
tokio::spawn(async move {
let mut engine = Engine {
stop: stop.subscribe(),
socket,
tunn,
profile,
candidates,
remote: None,
inbox: in_tx,
rx,
counter: rand::random(),
sequence: 0,
pending: VecDeque::new(),
samples: VecDeque::new(),
assemblies: HashMap::new(),
started: Instant::now(),
last_authenticated: Instant::now(),
};
engine.run(parent, &weak).await;
if let Some(link) = weak.upgrade() {
link.closed.store(true, Ordering::Release);
link.stop.send_replace(true);
}
});
link
}
struct Assembly {
created: Instant,
count: u16,
chunks: Vec<Option<Bytes>>,
}
struct Engine {
stop: watch::Receiver<bool>,
socket: UdpSocket,
tunn: Tunn,
profile: PairProfile,
candidates: Vec<SocketAddr>,
remote: Option<SocketAddr>,
inbox: mpsc::Sender<Bytes>,
rx: mpsc::Receiver<Bytes>,
counter: u64,
sequence: u64,
pending: VecDeque<(u64, Instant)>,
samples: VecDeque<bool>,
assemblies: HashMap<u64, Assembly>,
started: Instant,
last_authenticated: Instant,
}
enum ResultOwned {
Wire(Vec<u8>),
Plain(Vec<u8>),
Done,
}
fn own(result: TunnResult<'_>) -> ResultOwned {
match result {
TunnResult::WriteToNetwork(p) => ResultOwned::Wire(p.to_vec()),
TunnResult::WriteToTunnelV4(p, _) => ResultOwned::Plain(p.to_vec()),
_ => ResultOwned::Done,
}
}
fn ip(payload: &[u8]) -> Bytes {
let mut packet = vec![0; 20];
packet[0] = 0x45;
packet[8] = 64;
packet[9] = 253;
packet[12..16].copy_from_slice(&Ipv4Addr::new(192, 0, 2, 1).octets());
packet[16..20].copy_from_slice(&Ipv4Addr::new(192, 0, 2, 2).octets());
packet[2..4].copy_from_slice(&((20 + payload.len()) as u16).to_be_bytes());
let sum = packet
.as_chunks::<2>()
.0
.iter()
.map(|b| u32::from(u16::from_be_bytes([b[0], b[1]])))
.sum::<u32>();
let sum = (sum & 0xffff) + (sum >> 16);
packet[10..12].copy_from_slice(&(!(sum as u16)).to_be_bytes());
packet.extend_from_slice(payload);
Bytes::from(packet)
}
impl Engine {
async fn run(&mut self, mut stop: watch::Receiver<bool>, link: &std::sync::Weak<PairLink>) {
if *self.stop.borrow() || *stop.borrow() {
return;
}
let mut buffer = [0; 1201];
let mut scratch = [0; 4096];
let mut timer = tokio::time::interval(Duration::from_millis(250));
timer.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
let mut last_probe = Instant::now() - Duration::from_secs(2);
loop {
tokio::select! {
biased;
_=stop.changed()=>break,
_=self.stop.changed()=>break,
_=timer.tick()=>{
let result=own(self.tunn.update_timers(&mut scratch));self.process(result,None).await;
if last_probe.elapsed()>=Duration::from_secs(1) {
last_probe=Instant::now();self.sequence=self.sequence.wrapping_add(1);
self.pending.push_back((self.sequence,Instant::now()));
let mut probe=vec![1];probe.extend_from_slice(&self.sequence.to_be_bytes());
probe.resize(713,0);rand::fill(&mut probe[9..]);self.encrypt(&probe).await;
}
while self.pending.front().is_some_and(|(_,at)|at.elapsed()>=Duration::from_secs(3)) {
self.pending.pop_front();self.samples.push_back(false);
}
while self.samples.len()>20 {self.samples.pop_front();}
self.assemblies.retain(|_,a|a.created.elapsed()<Duration::from_secs(5));
if self.started.elapsed()>WINDOW && (self.last_authenticated.elapsed()>Duration::from_secs(10)
|| (self.samples.len()>=15&&self.samples.iter().filter(|ok|!**ok).count()*100>=self.samples.len()*60)) {
if let Some(link)=link.upgrade() && let Ok(mut reason)=link.reason.lock() {*reason=Some("AWG data delivery checks: persistent loss or timeout".into());}
break;
}
}
packet=self.socket.recv_from(&mut buffer)=>{
if let Ok((len,source))=packet && let Some(message)=self.profile.decode(&buffer[..len]) {
let result=own(self.tunn.decapsulate(None,&message,&mut scratch));
if matches!(result,ResultOwned::Plain(_)) {
self.remote=Some(source);self.last_authenticated=Instant::now();
if let Some(link)=link.upgrade() && let Ok(mut remote)=link.remote.lock(){*remote=Some(source);}
}
self.process(result,Some(source)).await;
for _ in 0..16 {
let result=own(self.tunn.decapsulate(None,&[],&mut scratch));
if matches!(result,ResultOwned::Done){break;}
self.process(result,Some(source)).await;
}
}
}
packet=self.rx.recv()=>{
let Some(packet)=packet else{break;};
let count=packet.len().max(1).div_ceil(CHUNK) as u16;
self.counter=self.counter.wrapping_add(1);
for (index,chunk) in packet.chunks(CHUNK).enumerate() {
let mut payload=vec![3];payload.extend_from_slice(&self.counter.to_be_bytes());
payload.extend_from_slice(&(index as u16).to_be_bytes());payload.extend_from_slice(&count.to_be_bytes());
payload.extend_from_slice(chunk);self.encrypt(&payload).await;
// Large logical datagrams must let local readers and other
// pairs drain their bounded socket/command queues.
if (index + 1).is_multiple_of(16) {tokio::task::yield_now().await;}
}
}
}
}
}
async fn encrypt(&mut self, payload: &[u8]) {
let packet = self.profile.pad(ip(payload));
let mut scratch = [0; 4096];
let result = own(self.tunn.encapsulate(&packet, &mut scratch));
self.process(result, None).await;
}
async fn wire(&self, raw: &[u8], target: Option<SocketAddr>) {
let Some(wire) = self.profile.encode(raw) else {
return;
};
let targets: Vec<_> = target
.or(self.remote)
.map(|a| vec![a])
.unwrap_or_else(|| self.candidates.clone());
for addr in targets {
if raw.first() == Some(&1) {
for signature in self.profile.signature_packets() {
let _ = self.send_wire(&signature, addr).await;
}
for junk in self.profile.junk_packets() {
let _ = self.send_wire(&junk, addr).await;
}
}
let _ = self.send_wire(&wire, addr).await;
}
}
async fn send_wire(&self, bytes: &[u8], address: SocketAddr) {
let mut stop = self.stop.clone();
if *stop.borrow() {
return;
}
tokio::select! {biased; _=stop.changed()=>{}, _=tokio::time::timeout(Duration::from_millis(100),self.socket.send_to(bytes,address))=>{}}
}
async fn process(&mut self, result: ResultOwned, target: Option<SocketAddr>) {
match result {
ResultOwned::Wire(raw) => self.wire(&raw, target).await,
ResultOwned::Plain(packet) => {
let Some(payload) = packet.get(20..) else {
return;
};
let Some(seq) = payload
.get(1..9)
.and_then(|b| b.try_into().ok())
.map(u64::from_be_bytes)
else {
return;
};
match payload.first() {
Some(1) => {
let mut pong = vec![2];
pong.extend_from_slice(&seq.to_be_bytes());
pong.resize(713, 0);
let packet = self.profile.pad(ip(&pong));
let mut scratch = [0; 4096];
if let ResultOwned::Wire(raw) =
own(self.tunn.encapsulate(&packet, &mut scratch))
{
self.wire(&raw, target).await;
}
}
Some(2) => {
if let Some(index) =
self.pending.iter().position(|(pending, _)| *pending == seq)
{
self.pending.remove(index);
self.samples.push_back(true);
}
}
Some(3) => self.fragment(seq, payload),
_ => {}
}
}
ResultOwned::Done => {}
}
}
fn fragment(&mut self, id: u64, payload: &[u8]) {
let Some(header) = payload.get(9..13) else {
return;
};
let index = u16::from_be_bytes([header[0], header[1]]);
let count = u16::from_be_bytes([header[2], header[3]]);
let chunk = &payload[13..];
if count == 0
|| usize::from(count) > MAX_PAYLOAD.div_ceil(CHUNK)
|| index >= count
|| chunk.len() > CHUNK
{
return;
}
if !self.assemblies.contains_key(&id) && self.assemblies.len() >= 8 {
return;
}
let assembly = self.assemblies.entry(id).or_insert_with(|| Assembly {
created: Instant::now(),
count,
chunks: vec![None; count.into()],
});
if assembly.count != count {
return;
}
assembly.chunks[usize::from(index)] = Some(Bytes::copy_from_slice(chunk));
if assembly.chunks.iter().all(Option::is_some)
&& let Some(assembly) = self.assemblies.remove(&id)
{
let mut packet = Vec::new();
for chunk in assembly.chunks.into_iter().flatten() {
packet.extend_from_slice(&chunk);
}
if packet.len() <= MAX_PAYLOAD {
let _ = self.inbox.try_send(Bytes::from(packet));
}
}
}
}
+101
View File
@@ -0,0 +1,101 @@
//! Optional, bounded server-reflexive discovery on the pair's own UDP socket.
//! A STUN answer is a candidate only; WireGuard authentication verifies a path.
use std::net::{IpAddr, Ipv4Addr, SocketAddr};
use std::time::Duration;
use tokio::net::UdpSocket;
const COOKIE: [u8; 4] = [0x21, 0x12, 0xa4, 0x42];
pub(crate) async fn mapped(socket: &UdpSocket, server: &str) -> Option<SocketAddr> {
tokio::time::timeout(Duration::from_millis(1200), async {
let address = tokio::net::lookup_host(server)
.await
.ok()?
.find(SocketAddr::is_ipv4)?;
let transaction: [u8; 12] = rand::random();
let mut request = [0; 20];
request[..2].copy_from_slice(&[0, 1]);
request[4..8].copy_from_slice(&COOKIE);
request[8..].copy_from_slice(&transaction);
socket.send_to(&request, address).await.ok()?;
let mut bytes = [0; 1024];
loop {
let (len, source) = socket.recv_from(&mut bytes).await.ok()?;
if source == address
&& let Some(mapped) = parse(&bytes[..len], &transaction)
{
return Some(mapped);
}
}
})
.await
.ok()
.flatten()
}
fn parse(bytes: &[u8], transaction: &[u8; 12]) -> Option<SocketAddr> {
if bytes.len() < 20
|| bytes[..2] != [1, 1]
|| bytes[4..8] != COOKIE
|| &bytes[8..20] != transaction
{
return None;
}
let declared = u16::from_be_bytes(bytes[2..4].try_into().ok()?) as usize;
if !declared.is_multiple_of(4) || declared != bytes.len() - 20 {
return None;
}
let mut at = 20;
while at + 4 <= bytes.len() {
let kind = u16::from_be_bytes(bytes[at..at + 2].try_into().ok()?);
let len = u16::from_be_bytes(bytes[at + 2..at + 4].try_into().ok()?) as usize;
at += 4;
let attribute = bytes.get(at..at.checked_add(len)?)?;
if kind == 0x0020 && attribute.len() == 8 && attribute[0..2] == [0, 1] {
let port = u16::from_be_bytes([attribute[2] ^ COOKIE[0], attribute[3] ^ COOKIE[1]]);
let ip = Ipv4Addr::new(
attribute[4] ^ COOKIE[0],
attribute[5] ^ COOKIE[1],
attribute[6] ^ COOKIE[2],
attribute[7] ^ COOKIE[3],
);
if port == 0 || ip.is_unspecified() || ip.is_multicast() || ip.is_broadcast() {
return None;
}
return Some(SocketAddr::new(IpAddr::V4(ip), port));
}
at = at.checked_add((len + 3) & !3)?;
}
None
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
#[tokio::test]
async fn mapping_uses_the_actual_socket_and_rejects_unrelated_or_truncated_answers() {
let server = UdpSocket::bind((Ipv4Addr::LOCALHOST, 0)).await.unwrap();
let address = server.local_addr().unwrap();
let socket = UdpSocket::bind((Ipv4Addr::LOCALHOST, 0)).await.unwrap();
let expected = socket.local_addr().unwrap();
let task = tokio::spawn(async move {
let mut request = [0; 128];
let (len, source) = server.recv_from(&mut request).await.unwrap();
assert_eq!(len, 20);
assert_eq!(source, expected);
let tx: [u8; 12] = request[8..20].try_into().unwrap();
let mut reply = request[..20].to_vec();
reply[..4].copy_from_slice(&[1, 1, 0, 12]);
reply.extend_from_slice(&[0, 0x20, 0, 8, 0, 1]);
let port = source.port().to_be_bytes();
reply.extend_from_slice(&[port[0] ^ COOKIE[0], port[1] ^ COOKIE[1]]);
for (i, b) in Ipv4Addr::LOCALHOST.octets().into_iter().enumerate() {
reply.push(b ^ COOKIE[i]);
}
assert!(parse(&reply, &[0; 12]).is_none());
for len in 0..reply.len() {
assert!(parse(&reply[..len], &tx).is_none());
}
server.send_to(&reply, source).await.unwrap();
});
assert_eq!(mapped(&socket, &address.to_string()).await, Some(expected));
task.await.unwrap();
}
}
+720
View File
@@ -0,0 +1,720 @@
//! Private pair negotiation. No secret profiles in capabilities or durable state.
use crate::{
AWG_PROTOCOL, AWG_WIRE_VERSION,
profile::PairProfile,
session::{self, PairLink},
};
use boringtun::{
noise::Tunn,
x25519::{PublicKey, StaticSecret},
};
use iroh::EndpointId;
use serde::{Deserialize, Serialize};
use std::collections::{HashMap, HashSet};
use std::net::{Ipv4Addr, SocketAddr};
use std::sync::{Arc, Mutex, OnceLock, Weak};
use std::time::Duration;
use tokio::net::UdpSocket;
use tokio::sync::{mpsc, oneshot, watch};
use tsunagi::dataplane::transport::TransportPayload;
use tsunagi::dataplane::transport::{
ControlSender, InboundLink, PacketLink, PacketTransport, SharedLink, TransportError,
};
use tsunagi::{BoxFuture, NetworkId};
type Pair = (NetworkId, EndpointId);
#[derive(Serialize, Deserialize)]
enum Negotiation {
Offer {
version: u16,
profile: PairProfile,
key: [u8; 32],
#[serde(deserialize_with = "bounded_addresses")]
addresses: Vec<SocketAddr>,
},
Answer {
attempt: [u8; 16],
key: [u8; 32],
#[serde(deserialize_with = "bounded_addresses")]
addresses: Vec<SocketAddr>,
},
}
fn bounded_addresses<'de, D: serde::Deserializer<'de>>(
deserializer: D,
) -> Result<Vec<SocketAddr>, D::Error> {
struct Visitor;
impl<'de> serde::de::Visitor<'de> for Visitor {
type Value = Vec<SocketAddr>;
fn expecting(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str("at most 32 UDP candidates")
}
fn visit_seq<A: serde::de::SeqAccess<'de>>(
self,
mut sequence: A,
) -> Result<Self::Value, A::Error> {
if sequence.size_hint().is_some_and(|n| n > 32) {
return Err(serde::de::Error::custom("too many UDP candidates"));
}
let mut addresses = Vec::with_capacity(sequence.size_hint().unwrap_or(0).min(32));
while let Some(address) = sequence.next_element()? {
if addresses.len() == 32 {
return Err(serde::de::Error::custom("too many UDP candidates"));
}
addresses.push(address);
}
Ok(addresses)
}
}
deserializer.deserialize_seq(Visitor)
}
struct Pending {
profile: PairProfile,
key: StaticSecret,
socket: UdpSocket,
reply: oneshot::Sender<Result<SharedLink, TransportError>>,
}
struct Inner {
stopped: watch::Sender<bool>,
local: EndpointId,
bind: Ipv4Addr,
stun: Option<String>,
busy: Mutex<HashSet<Pair>>,
control: OnceLock<Arc<dyn ControlSender>>,
allowed: Mutex<HashSet<Pair>>,
pending: Mutex<HashMap<Pair, Pending>>,
links: Mutex<HashMap<Pair, Arc<PairLink>>>,
inbound: mpsc::Sender<InboundLink>,
port_by_pair: Mutex<HashMap<Pair, u16>>,
}
/// Public diagnostics for one pair. No key or masking parameters are exposed.
#[derive(Debug, Clone)]
pub struct PairOverview {
/// Other endpoint's authenticated identity.
pub peer: EndpointId,
/// Fresh connection-attempt identifier.
pub attempt: [u8; 16],
/// Last known UDP port; None if diagnostics race with pair retirement.
pub local_port: Option<u16>,
/// Actual transport path, if established.
pub path: String,
/// Whether this attempt has been retired.
pub closed: bool,
}
/// A bounded set of independently configured pair engines.
pub struct AwgTransport {
inner: Arc<Inner>,
commands: mpsc::Sender<(NetworkId, EndpointId, Negotiation)>,
incoming: tokio::sync::Mutex<mpsc::Receiver<InboundLink>>,
}
impl std::fmt::Debug for AwgTransport {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("AwgTransport")
.field("local", &self.inner.local)
.finish_non_exhaustive()
}
}
fn lock<T>(mutex: &Mutex<T>) -> std::sync::MutexGuard<'_, T> {
match mutex.lock() {
Ok(g) => g,
Err(p) => p.into_inner(),
}
}
fn error(text: &str) -> TransportError {
TransportError::Other(text.into())
}
async fn addresses(inner: &Inner, socket: &UdpSocket) -> Result<Vec<SocketAddr>, TransportError> {
let local = socket
.local_addr()
.map_err(|_| error("AWG socket address unavailable"))?;
let mut candidates = if inner.bind.is_loopback() {
vec![local]
} else {
tsunagi_wireguard::local_ip_candidates()
.into_iter()
.filter(|ip| ip.is_ipv4())
.take(31)
.map(|ip| SocketAddr::new(ip, local.port()))
.collect()
};
if let Some(server) = &inner.stun
&& let Some(mapped) = crate::stun::mapped(socket, server).await
&& !candidates.contains(&mapped)
{
candidates.push(mapped);
}
if candidates.is_empty() {
return Err(error("AWG has no UDP address candidates"));
}
Ok(candidates)
}
fn validate_endpoint(key: &[u8; 32], addresses: &[SocketAddr]) -> Result<(), TransportError> {
if *key == [0; 32]
|| addresses.is_empty()
|| addresses.len() > 32
|| addresses.iter().any(|address| {
address.port() == 0
|| !address.is_ipv4()
|| address.ip().is_unspecified()
|| address.ip().is_multicast()
|| matches!(address.ip(), std::net::IpAddr::V4(ip) if ip.is_broadcast())
})
{
return Err(error("invalid AWG endpoint"));
}
Ok(())
}
impl AwgTransport {
/// Starts the coordinator. Production binds wildcard IPv4; tests use loopback.
pub fn new(local: EndpointId, bind: Ipv4Addr) -> Arc<Self> {
Self::new_with_stun(local, bind, None)
}
/// Enables an optional STUN server (host:port) for this pair's own socket.
/// Local-only callers pass None; failure still permits local UDP candidates.
pub fn new_with_stun(local: EndpointId, bind: Ipv4Addr, stun: Option<String>) -> Arc<Self> {
let (commands, mut rx) = mpsc::channel(64);
let (inbound, incoming) = mpsc::channel(64);
let (stopped, _) = watch::channel(false);
let inner = Arc::new(Inner {
stopped,
local,
bind,
stun,
busy: Mutex::default(),
control: OnceLock::new(),
allowed: Mutex::default(),
pending: Mutex::default(),
links: Mutex::default(),
inbound,
port_by_pair: Mutex::default(),
});
let weak = Arc::downgrade(&inner);
let mut stopped = inner.stopped.subscribe();
tokio::spawn(async move {
while let Some((network, peer, message)) =
tokio::select! { biased; _=stopped.changed()=>None, message=rx.recv()=>message }
{
let Some(inner) = weak.upgrade() else {
break;
};
if !lock(&inner.allowed).contains(&(network, peer))
|| !lock(&inner.busy).insert((network, peer))
{
continue;
}
tokio::spawn(async move {
let mut stop = inner.stopped.subscribe();
tokio::select! {biased; _=stop.changed()=>{}, _=async {
if let Err(err)=receive(&inner,(network,peer),message).await {tracing::debug!(%err,"AWG pair negotiation rejected");}
}=>{}}
lock(&inner.busy).remove(&(network, peer));
});
}
});
Arc::new(Self {
inner,
commands,
incoming: tokio::sync::Mutex::new(incoming),
})
}
/// Registers a peer after its authenticated capability has been validated.
pub fn allow_peer(&self, network: NetworkId, peer: EndpointId) {
let mut allowed = lock(&self.inner.allowed);
if !*self.inner.stopped.borrow() && allowed.len() < 128 {
allowed.insert((network, peer));
}
}
/// Retires a pair when its network membership session goes away.
pub fn forget_peer(&self, network: NetworkId, peer: EndpointId) {
let pair = (network, peer);
lock(&self.inner.allowed).remove(&pair);
lock(&self.inner.pending).remove(&pair);
lock(&self.inner.port_by_pair).remove(&pair);
if let Some(link) = lock(&self.inner.links).remove(&pair) {
link.close();
}
}
/// Stops all pair engines in this network.
pub fn stop_network(&self, network: NetworkId) {
let peers: Vec<_> = lock(&self.inner.allowed)
.iter()
.filter(|(n, _)| *n == network)
.map(|(_, p)| *p)
.collect();
for peer in peers {
self.forget_peer(network, peer);
}
}
/// Stops all sockets and coordinator work without waiting for remote peers.
pub fn shutdown(&self) {
self.inner.stopped.send_replace(true);
lock(&self.inner.allowed).clear();
lock(&self.inner.pending).clear();
for (_, link) in lock(&self.inner.links).drain() {
link.close();
}
lock(&self.inner.port_by_pair).clear();
}
/// Receives a successfully negotiated and authenticated inbound pair link.
pub async fn next_inbound(&self) -> Option<InboundLink> {
let mut stopped = self.inner.stopped.subscribe();
if *stopped.borrow() {
return None;
}
let mut incoming = self.incoming.lock().await;
tokio::select! {biased; _=stopped.changed()=>None, link=incoming.recv()=>link}
}
/// Takes a public snapshot of this network's pair attempts.
pub fn overview(&self, network: NetworkId) -> Vec<PairOverview> {
let links = lock(&self.inner.links);
let ports = lock(&self.inner.port_by_pair);
links
.iter()
.filter(|((n, _), _)| *n == network)
.map(|((_, peer), link)| PairOverview {
peer: *peer,
attempt: link.attempt,
local_port: ports.get(&(network, *peer)).copied(),
path: link.path_description(),
closed: link.is_closed(),
})
.collect()
}
}
impl PacketTransport for AwgTransport {
fn name(&self) -> &str {
AWG_PROTOCOL
}
fn as_any(&self) -> &dyn std::any::Any {
self
}
fn attach_control(&self, control: Arc<dyn ControlSender>) {
let _ = self.inner.control.set(control);
}
fn on_control(
&self,
network: NetworkId,
peer: EndpointId,
protocol: &str,
payload: &[u8],
) -> Result<(), TransportError> {
if protocol != AWG_PROTOCOL || !lock(&self.inner.allowed).contains(&(network, peer)) {
return Err(TransportError::Declined(protocol.into()));
}
if payload.len() > 4096 {
return Err(error("AWG negotiation too large"));
}
let message: Negotiation =
postcard::from_bytes(payload).map_err(|_| error("malformed AWG negotiation"))?;
match &message {
Negotiation::Offer {
version,
profile,
key,
addresses,
} => {
if *version != AWG_WIRE_VERSION || peer.as_bytes() >= self.inner.local.as_bytes() {
return Err(error("invalid AWG offer role/version"));
}
profile.validate().map_err(error)?;
validate_endpoint(key, addresses)?;
}
Negotiation::Answer { key, addresses, .. } => validate_endpoint(key, addresses)?,
}
self.commands
.try_send((network, peer, message))
.map_err(|_| error("AWG negotiation queue full"))
}
fn open<'a>(
&'a self,
network: NetworkId,
peer: EndpointId,
protocol: &'a str,
) -> BoxFuture<'a, Result<SharedLink, TransportError>> {
Box::pin(async move {
let pair = (network, peer);
if protocol != AWG_PROTOCOL
|| self.inner.local.as_bytes() >= peer.as_bytes()
|| !lock(&self.inner.allowed).contains(&pair)
{
return Err(TransportError::Declined(protocol.into()));
}
let control = self
.inner
.control
.get()
.ok_or_else(|| error("AWG control plane unavailable"))?;
let socket = UdpSocket::bind((self.inner.bind, 0))
.await
.map_err(|_| error("AWG UDP bind failed"))?;
let profile = PairProfile::generate();
let attempt = profile.id();
let key = StaticSecret::from(rand::random::<[u8; 32]>());
let public = PublicKey::from(&key).to_bytes();
let message = Negotiation::Offer {
version: AWG_WIRE_VERSION,
profile: profile.clone(),
key: public,
addresses: addresses(&self.inner, &socket).await?,
};
let (reply, rx) = oneshot::channel();
if lock(&self.inner.pending).len() >= 128 {
return Err(error("too many AWG pair negotiations"));
}
lock(&self.inner.pending).insert(
pair,
Pending {
profile,
key,
socket,
reply,
},
);
// A cancellation guard also retires partially started attempts.
let mut guard = AttemptGuard {
inner: Arc::downgrade(&self.inner),
pair,
attempt,
completed: false,
};
let bytes =
postcard::to_stdvec(&message).map_err(|_| error("AWG offer encoding failed"))?;
control
.send(network, peer, AWG_PROTOCOL, TransportPayload::new(bytes))
.await?;
let result = tokio::time::timeout(Duration::from_secs(15), rx)
.await
.map_err(|_| error("AWG negotiation/handshake timed out"))?
.map_err(|_| error("AWG attempt cancelled"))?;
guard.completed = result.is_ok();
result
})
}
}
struct AttemptGuard {
inner: Weak<Inner>,
pair: Pair,
attempt: [u8; 16],
completed: bool,
}
impl Drop for AttemptGuard {
fn drop(&mut self) {
if let Some(inner) = self.inner.upgrade() {
let mut pending = lock(&inner.pending);
if pending
.get(&self.pair)
.is_some_and(|p| p.profile.id() == self.attempt)
{
pending.remove(&self.pair);
}
drop(pending);
if !self.completed
&& let Some(link) = lock(&inner.links).get(&self.pair)
&& link.attempt == self.attempt
{
link.close();
}
}
}
}
async fn receive(
inner: &Arc<Inner>,
pair: Pair,
message: Negotiation,
) -> Result<(), TransportError> {
if !lock(&inner.allowed).contains(&pair) {
return Err(error("AWG peer no longer authenticated"));
}
match message {
Negotiation::Offer {
profile,
key,
addresses: candidates,
..
} => {
if let Some(link) = lock(&inner.links).get(&pair) {
if link.attempt == profile.id() {
return Err(error("duplicate AWG offer"));
}
link.close();
}
let socket = UdpSocket::bind((inner.bind, 0))
.await
.map_err(|_| error("AWG UDP bind failed"))?;
let local_port = socket
.local_addr()
.map_err(|_| error("AWG socket address unavailable"))?
.port();
let local_key = StaticSecret::from(rand::random::<[u8; 32]>());
let public = PublicKey::from(&local_key).to_bytes();
let answer = Negotiation::Answer {
attempt: profile.id(),
key: public,
addresses: addresses(inner, &socket).await?,
};
let tunn = Tunn::new(
local_key,
PublicKey::from(key),
Some(profile.psk),
Some(1),
rand::random(),
None,
);
let link = session::start(
pair.0,
pair.1,
socket,
tunn,
profile,
candidates,
inner.stopped.subscribe(),
);
if !lock(&inner.allowed).contains(&pair) {
link.close();
return Err(error("AWG membership ended"));
}
lock(&inner.links).insert(pair, link.clone());
lock(&inner.port_by_pair).insert(pair, local_port);
let control = inner
.control
.get()
.ok_or_else(|| error("AWG control unavailable"))?;
if let Err(err) = control
.send(
pair.0,
pair.1,
AWG_PROTOCOL,
TransportPayload::new(
postcard::to_stdvec(&answer)
.map_err(|_| error("AWG answer encoding failed"))?,
),
)
.await
{
link.close();
return Err(err);
}
publish_when_up(Arc::downgrade(inner), pair, link, None);
}
Negotiation::Answer {
attempt,
key,
addresses: candidates,
} => {
if !lock(&inner.pending)
.get(&pair)
.is_some_and(|p| p.profile.id() == attempt)
{
return Err(error("stale AWG answer"));
}
let pending = lock(&inner.pending)
.remove(&pair)
.ok_or_else(|| error("AWG attempt cancelled"))?;
let local_port = pending
.socket
.local_addr()
.map_err(|_| error("AWG socket address unavailable"))?
.port();
let tunn = Tunn::new(
pending.key,
PublicKey::from(key),
Some(pending.profile.psk),
Some(1),
rand::random(),
None,
);
let link = session::start(
pair.0,
pair.1,
pending.socket,
tunn,
pending.profile,
candidates,
inner.stopped.subscribe(),
);
if !lock(&inner.allowed).contains(&pair) {
link.close();
return Err(error("AWG membership ended"));
}
lock(&inner.links).insert(pair, link.clone());
lock(&inner.port_by_pair).insert(pair, local_port);
publish_when_up(Arc::downgrade(inner), pair, link, Some(pending.reply));
}
}
Ok(())
}
fn publish_when_up(
inner: Weak<Inner>,
pair: Pair,
link: Arc<PairLink>,
reply: Option<oneshot::Sender<Result<SharedLink, TransportError>>>,
) {
tokio::spawn(async move {
let mut reply = reply;
let result = tokio::time::timeout(Duration::from_secs(14), async {
loop {
if link.is_closed() || reply.as_ref().is_some_and(oneshot::Sender::is_closed) {
return Err(error("AWG pair closed during handshake"));
}
if link.has_authenticated_path() {
return Ok(());
}
tokio::time::sleep(Duration::from_millis(25)).await;
}
})
.await;
if !matches!(result, Ok(Ok(()))) {
link.close();
if let Some(reply) = reply.take() {
let _ = reply.send(Err(error("AWG authenticated data path timed out")));
}
return;
}
let Some(inner) = inner.upgrade() else {
link.close();
return;
};
if !lock(&inner.allowed).contains(&pair) {
link.close();
return;
}
if link.is_closed()
|| !lock(&inner.links)
.get(&pair)
.is_some_and(|current| current.attempt == link.attempt)
{
link.close();
return;
}
let shared: SharedLink = link.clone();
if let Some(reply) = reply {
if reply.send(Ok(shared)).is_err() {
link.close();
}
} else if inner
.inbound
.try_send(InboundLink {
network: pair.0,
peer: pair.1,
protocol: AWG_PROTOCOL.into(),
link: shared,
})
.is_err()
{
link.close();
}
});
}
impl Drop for AwgTransport {
fn drop(&mut self) {
self.shutdown();
}
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
#[tokio::test]
async fn concurrent_admission_is_bounded_across_networks_and_retirement_releases_slots() {
let identity = tsunagi::identity::DeviceIdentity::generate();
let transport = AwgTransport::new(identity.endpoint_id(), Ipv4Addr::LOCALHOST);
let peer = tsunagi::identity::DeviceIdentity::generate().endpoint_id();
std::thread::scope(|scope| {
for thread in 0..8u8 {
let transport = &transport;
scope.spawn(move || {
for item in 0..32u8 {
let mut id = [0; 32];
id[0] = thread;
id[1] = item;
transport.allow_peer(NetworkId::from_bytes(id), peer);
}
});
}
});
assert_eq!(lock(&transport.inner.allowed).len(), 128);
let network = lock(&transport.inner.allowed).iter().next().unwrap().0;
transport.stop_network(network);
let new = NetworkId::from_bytes([255; 32]);
transport.allow_peer(new, peer);
assert!(lock(&transport.inner.allowed).contains(&(new, peer)));
assert_eq!(lock(&transport.inner.allowed).len(), 128);
transport.shutdown();
assert!(lock(&transport.inner.allowed).is_empty());
}
#[tokio::test]
async fn private_negotiation_bounds_roles_and_membership_before_starting_engines() {
let mut ids = [
tsunagi::identity::DeviceIdentity::generate().endpoint_id(),
tsunagi::identity::DeviceIdentity::generate().endpoint_id(),
];
ids.sort_by_key(|id| *id.as_bytes());
let peer = ids[0];
let local = ids[1];
let network = NetworkId::from_bytes([7; 32]);
let transport = AwgTransport::new(local, Ipv4Addr::LOCALHOST);
let offer = |version, key, addresses| Negotiation::Offer {
version,
profile: PairProfile::generate(),
key,
addresses,
};
let address = SocketAddr::from((Ipv4Addr::LOCALHOST, 12345));
let encode = |message: &Negotiation| postcard::to_stdvec(message).unwrap();
let valid = encode(&offer(AWG_WIRE_VERSION, [1; 32], vec![address]));
assert!(
transport
.on_control(network, peer, AWG_PROTOCOL, &valid)
.is_err()
);
transport.allow_peer(network, peer);
assert!(
transport
.on_control(network, peer, "wrong-protocol", &valid)
.is_err()
);
assert!(
transport
.on_control(network, peer, AWG_PROTOCOL, &[0; 4097])
.is_err()
);
for bytes in [
encode(&offer(99, [1; 32], vec![address])),
encode(&offer(AWG_WIRE_VERSION, [0; 32], vec![address])),
encode(&offer(AWG_WIRE_VERSION, [1; 32], vec![])),
encode(&offer(AWG_WIRE_VERSION, [1; 32], vec![address; 33])),
] {
assert!(
transport
.on_control(network, peer, AWG_PROTOCOL, &bytes)
.is_err()
);
}
for len in 0..valid.len() {
assert!(
transport
.on_control(network, peer, AWG_PROTOCOL, &valid[..len])
.is_err()
);
}
let wrong_role = AwgTransport::new(peer, Ipv4Addr::LOCALHOST);
wrong_role.allow_peer(network, local);
assert!(
wrong_role
.on_control(network, local, AWG_PROTOCOL, &valid)
.is_err()
);
transport.forget_peer(network, peer);
assert!(
transport
.on_control(network, peer, AWG_PROTOCOL, &valid)
.is_err()
);
assert!(transport.overview(network).is_empty());
transport.shutdown();
wrong_role.shutdown();
}
}
+773
View File
@@ -0,0 +1,773 @@
//! Real membership, UDP sockets, pair negotiation and nested WireGuard encryption.
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use bytes::Bytes;
use std::{
net::Ipv4Addr,
path::Path,
sync::{
Arc,
atomic::{AtomicBool, AtomicU64, Ordering},
},
time::Duration,
};
use tempfile::TempDir;
use tsunagi::dataplane::IpPlugin;
use tsunagi::dataplane::transport::{
ControlSender, PacketLink, PacketTransport, SharedLink, TransportError,
};
use tsunagi::discovery::SharedMemoryDiscovery;
use tsunagi::overlay::{MemoryTun, MemoryTunFactory};
use tsunagi::testing::{config_with, network, wait_for_peers, wait_until};
use tsunagi::{Agent, BoxFuture, NetworkId};
use tsunagi_awg::{AWG_PROTOCOL, AwgPlugin, AwgTransport};
use tsunagi_wg_quic::WireguardConfig;
struct Node {
agent: Agent,
plugin: Arc<AwgPlugin>,
quic: Option<Arc<tsunagi_wg_quic::WireguardPlugin>>,
transport: Arc<AwgTransport>,
tuns: MemoryTunFactory,
loss: Arc<AtomicBool>,
}
impl Node {
async fn open(root: &Path, discovery: &SharedMemoryDiscovery, interface: &str) -> Self {
Self::open_with_fallback(root, discovery, interface, false).await
}
async fn open_with_fallback(
root: &Path,
discovery: &SharedMemoryDiscovery,
interface: &str,
fallback: bool,
) -> Self {
Self::open_config(root, discovery, interface, fallback, 1280).await
}
async fn open_config(
root: &Path,
discovery: &SharedMemoryDiscovery,
interface: &str,
fallback: bool,
mtu: u32,
) -> Self {
Self::open_selected(root, discovery, interface, fallback, mtu, None).await
}
async fn open_selected(
root: &Path,
discovery: &SharedMemoryDiscovery,
interface: &str,
fallback: bool,
mtu: u32,
default_protocols: Option<Vec<String>>,
) -> Self {
let identity =
tsunagi::storage::StateStore::open(tsunagi::StoragePaths::under(root).state_db())
.unwrap()
.load_or_create_device_identity()
.unwrap();
let transport = AwgTransport::new(identity.endpoint_id(), Ipv4Addr::LOCALHOST);
let plugin = AwgPlugin::open(
WireguardConfig::new(root.join("awg"))
.with_mtu(mtu)
.with_reconcile(Duration::from_millis(20), Duration::from_millis(250)),
transport.clone(),
)
.await
.unwrap();
let loss = Arc::new(AtomicBool::new(false));
let faults = Arc::new(FaultTransport {
inner: transport.clone(),
loss: loss.clone(),
});
let tuns = MemoryTunFactory::new();
let mut config = config_with(root, discovery)
.with_interface(Arc::new(tuns.clone()), interface, mtu)
.with_plugin(plugin.clone() as Arc<dyn IpPlugin>)
.with_custom_transport(AWG_PROTOCOL, faults);
let mut quic_plugin = None;
if fallback {
let quic = tsunagi_wg_quic::WireguardPlugin::open(
WireguardConfig::new(root.join("wg-quic"))
.with_mtu(mtu)
.with_reconcile(Duration::from_millis(20), Duration::from_millis(250)),
)
.await
.unwrap();
config = config.with_plugin(quic.clone());
quic_plugin = Some(quic);
}
config.default_protocols = default_protocols;
let agent = Agent::spawn(config).await.unwrap();
let inbound_agent = agent.clone();
let inbound_transport = transport.clone();
let inbound_loss = loss.clone();
tokio::spawn(async move {
while let Some(mut link) = inbound_transport.next_inbound().await {
link.link = impaired(link.link, inbound_loss.clone());
let _ = inbound_agent.install_inbound_link(link).await;
}
});
Self {
agent,
plugin,
quic: quic_plugin,
transport,
tuns,
loss,
}
}
async fn ready(&self, id: NetworkId, n: usize) {
wait_until("AWG outer paths and end-to-end handshakes", || async {
(self
.transport
.overview(id)
.iter()
.filter(|p| !p.closed && !p.path.contains("establishing"))
.count()
== n
&& self
.plugin
.overview(id)?
.peers
.iter()
.filter(|p| {
p.tunnel
.as_ref()
.is_some_and(|t| t.health.is_up() && t.path.contains("awg UDP via"))
})
.count()
== n)
.then_some(())
})
.await;
}
fn tun(&self) -> Arc<MemoryTun> {
self.tuns
.device(&self.agent.overlay().unwrap().interface)
.unwrap()
}
fn addr(&self, id: NetworkId) -> Ipv4Addr {
self.plugin
.overview(id)
.unwrap()
.overlay_address_v4
.unwrap()
}
}
fn packet(source: Ipv4Addr, dest: Ipv4Addr, len: usize) -> Bytes {
let mut bytes = vec![0; len];
bytes[0] = 0x45;
bytes[2..4].copy_from_slice(&(len as u16).to_be_bytes());
bytes[8] = 64;
bytes[9] = 253;
bytes[12..16].copy_from_slice(&source.octets());
bytes[16..20].copy_from_slice(&dest.octets());
rand::fill(&mut bytes[20..]);
Bytes::from(bytes)
}
async fn exchange(a: &Node, b: &Node, id: NetworkId) {
for len in [64, 1097, 1280] {
let bytes = packet(a.addr(id), b.addr(id), len);
a.tun().push_from_os(bytes.clone());
assert_eq!(
tokio::time::timeout(Duration::from_secs(5), b.tun().pop_to_os())
.await
.unwrap()
.unwrap(),
bytes
);
}
}
#[tokio::test]
async fn a_control_only_agent_enables_real_data_without_restart_and_cancels_stale_attempts() {
let discovery = SharedMemoryDiscovery::new();
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
let a = Node::open_selected(
roots[0].path(),
&discovery,
"initial-none-a",
true,
1280,
Some(vec![]),
)
.await;
let b = Node::open_selected(
roots[1].path(),
&discovery,
"initial-none-b",
true,
1280,
Some(vec![]),
)
.await;
let (name, secret) = network("enable-after-control-only");
let id = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
wait_for_peers(&a.agent, id, 1).await;
wait_for_peers(&b.agent, id, 1).await;
assert!(a.agent.overlay().is_none());
assert!(b.agent.overlay().is_none());
for node in [&a, &b] {
assert!(
node.agent
.network_status(id)
.await
.unwrap()
.protocols
.is_empty()
);
node.agent
.set_protocols(id, vec!["awg".into()])
.await
.unwrap();
assert!(node.agent.overlay().is_some());
}
a.ready(id, 1).await;
b.ready(id, 1).await;
exchange(&a, &b, id).await;
let before = a.agent.overlay().unwrap().interface;
// Changes race unfinished opens and prepared plugin callbacks; the last
// selection must win even if an earlier open has completed in its queue.
for selection in [
vec!["wg-quic".into()],
vec![],
vec!["awg".into()],
vec![],
vec!["awg".into()],
] {
a.agent.set_protocols(id, selection).await.unwrap();
}
a.ready(id, 1).await;
b.ready(id, 1).await;
exchange(&a, &b, id).await;
assert_eq!(a.agent.overlay().unwrap().interface, before);
for node in [&a, &b] {
let status = node.agent.network_status(id).await.unwrap();
assert_eq!(status.protocols, ["awg"]);
assert_eq!(status.metrics.sessions_established, 1);
assert_eq!(status.metrics.disconnects, 0);
}
a.agent.shutdown().await;
b.agent.shutdown().await;
}
#[tokio::test]
async fn runtime_protocol_selection_replaces_real_tunnels_and_survives_restart() {
let discovery = SharedMemoryDiscovery::new();
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
let a = Node::open_with_fallback(roots[0].path(), &discovery, "runtime-a", true).await;
let b = Node::open_with_fallback(roots[1].path(), &discovery, "runtime-b", true).await;
let (name, secret) = network("runtime-protocols");
let id = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
a.ready(id, 1).await;
b.ready(id, 1).await;
exchange(&a, &b, id).await;
let (other_name, other_secret, other_id) = loop {
let (name, secret) = network("unchanged-runtime-network");
let other_id = tsunagi::identity::NetworkKeys::derive(&name, &secret).network_id();
if tsunagi::state::derived_ipv4_range(id) != tsunagi::state::derived_ipv4_range(other_id) {
break (name, secret, other_id);
}
};
a.agent
.join_network(&other_name, &other_secret)
.await
.unwrap();
b.agent
.join_network(&other_name, &other_secret)
.await
.unwrap();
a.ready(other_id, 1).await;
b.ready(other_id, 1).await;
let other_before = a
.agent
.network_status(other_id)
.await
.unwrap()
.metrics
.data_links_established;
for node in [&a, &b] {
let before = node.agent.network_status(id).await.unwrap().protocols;
for invalid in [vec!["unknown".into()], vec!["awg".into(), "awg".into()]] {
assert!(node.agent.set_protocols(id, invalid).await.is_err());
assert_eq!(
node.agent.network_status(id).await.unwrap().protocols,
before
);
}
}
// Exercise updates initiated by either endpoint-id role, including an
// empty set. Membership and the control session stay alive throughout.
for node in [&a, &b] {
node.agent.set_protocols(id, Vec::new()).await.unwrap();
wait_until("peer accepts control-only selection", || async {
let other = if node.agent.endpoint_id() == a.agent.endpoint_id() {
&b
} else {
&a
};
let status = other.agent.network_status(id).await.ok()?;
(status.peers.len() == 1
&& status.peers[0].protocols.is_empty()
&& matches!(status.peers[0].data_path, tsunagi::agent::DataPath::None))
.then_some(())
})
.await;
assert_eq!(node.agent.network_status(id).await.unwrap().peers.len(), 1);
node.agent
.set_protocols(id, vec!["wg-quic".into(), "awg".into()])
.await
.unwrap();
}
a.agent
.set_protocols(id, vec!["wg-quic".into()])
.await
.unwrap();
wait_until("real QUIC replaces AWG", || async {
let a_status = a.agent.network_status(id).await.ok()?;
let b_status = b.agent.network_status(id).await.ok()?;
(a_status.peers[0].protocols == ["wg-quic"]
&& b_status.peers[0].protocols == ["wg-quic"]
&& !matches!(a_status.peers[0].data_path, tsunagi::agent::DataPath::None)
&& !matches!(b_status.peers[0].data_path, tsunagi::agent::DataPath::None)
&& a.transport.overview(id).iter().all(|p| p.closed)
&& b.transport.overview(id).iter().all(|p| p.closed))
.then_some(())
})
.await;
// Get addresses from the system allocation: the disabled AWG plugin no
// longer has a live view of this network.
for (source, dest) in [(&a, &b), (&b, &a)] {
wait_until("QUIC end-to-end handshake", || async {
source
.quic
.as_ref()?
.overview(id)?
.peers
.iter()
.any(|p| p.tunnel.as_ref().is_some_and(|t| t.health.is_up()))
.then_some(())
})
.await;
let addr = |n: &Node| {
n.quic
.as_ref()
.unwrap()
.overview(id)
.unwrap()
.overlay_address_v4
.unwrap()
};
let bytes = packet(addr(source), addr(dest), 1280);
source.tun().push_from_os(bytes.clone());
assert_eq!(
tokio::time::timeout(Duration::from_secs(5), dest.tun().pop_to_os())
.await
.unwrap()
.unwrap(),
bytes
);
}
assert_eq!(
a.agent
.network_status(other_id)
.await
.unwrap()
.metrics
.data_links_established,
other_before
);
exchange(&a, &b, other_id).await;
a.agent.shutdown().await;
let restarted = Node::open_with_fallback(roots[0].path(), &discovery, "runtime-a", true).await;
wait_for_peers(&restarted.agent, id, 1).await;
assert_eq!(
restarted.agent.network_status(id).await.unwrap().protocols,
["wg-quic"]
);
assert_eq!(
b.agent.network_status(other_id).await.unwrap().protocols,
["awg", "wg-quic"]
);
restarted.ready(other_id, 1).await;
b.ready(other_id, 1).await;
exchange(&restarted, &b, other_id).await;
restarted
.agent
.set_protocols(id, vec!["awg".into()])
.await
.unwrap();
restarted.ready(id, 1).await;
b.ready(id, 1).await;
exchange(&restarted, &b, id).await;
restarted.agent.shutdown().await;
b.agent.shutdown().await;
}
#[tokio::test]
async fn real_pairs_carry_full_size_packets_and_restart_with_fresh_profiles() {
tsunagi::testing::init_tracing();
let discovery = SharedMemoryDiscovery::new();
let roots = [
TempDir::new().unwrap(),
TempDir::new().unwrap(),
TempDir::new().unwrap(),
];
let a = Node::open(roots[0].path(), &discovery, "awga").await;
let b = Node::open(roots[1].path(), &discovery, "awgb").await;
let c = Node::open(roots[2].path(), &discovery, "awgc").await;
let (name, secret) = network("awg-pairs");
let id = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
c.agent.join_network(&name, &secret).await.unwrap();
for node in [&a, &b, &c] {
wait_for_peers(&node.agent, id, 2).await;
node.ready(id, 2).await;
}
let ids: std::collections::HashSet<_> = [&a, &b, &c]
.into_iter()
.flat_map(|n| n.transport.overview(id).into_iter().map(|p| p.attempt))
.collect();
assert_eq!(
ids.len(),
3,
"one independent profile for each of three pairs"
);
exchange(&a, &b, id).await;
exchange(&b, &a, id).await;
exchange(&a, &c, id).await;
let old_ports: Vec<_> = a
.transport
.overview(id)
.into_iter()
.filter_map(|p| p.local_port)
.collect();
a.agent.shutdown().await;
for port in old_ports {
wait_until("old pair socket released", || async {
tokio::net::UdpSocket::bind((Ipv4Addr::LOCALHOST, port))
.await
.ok()
})
.await;
}
let restarted = Node::open(roots[0].path(), &discovery, "awga").await;
restarted.ready(id, 2).await;
assert!(
restarted
.transport
.overview(id)
.iter()
.all(|p| !ids.contains(&p.attempt))
);
exchange(&restarted, &b, id).await;
exchange(&c, &restarted, id).await;
restarted.agent.shutdown().await;
b.agent.shutdown().await;
c.agent.shutdown().await;
assert!(
tokio::time::timeout(Duration::from_secs(1), restarted.transport.next_inbound())
.await
.unwrap()
.is_none()
);
}
// Drops datagrams at the link boundary. Membership, control, sockets, crypto,
// addressing and storage remain real. This substitutes a lossy physical path.
#[derive(Debug)]
struct FaultTransport {
inner: Arc<AwgTransport>,
loss: Arc<AtomicBool>,
}
impl PacketTransport for FaultTransport {
fn name(&self) -> &str {
AWG_PROTOCOL
}
fn as_any(&self) -> &dyn std::any::Any {
self
}
fn attach_control(&self, sender: Arc<dyn ControlSender>) {
self.inner.attach_control(sender);
}
fn on_control(
&self,
network: NetworkId,
peer: iroh::EndpointId,
protocol: &str,
bytes: &[u8],
) -> Result<(), TransportError> {
self.inner.on_control(network, peer, protocol, bytes)
}
fn open<'a>(
&'a self,
network: NetworkId,
peer: iroh::EndpointId,
protocol: &'a str,
) -> BoxFuture<'a, Result<SharedLink, TransportError>> {
Box::pin(async move {
Ok(impaired(
self.inner.open(network, peer, protocol).await?,
self.loss.clone(),
))
})
}
}
fn impaired(raw: SharedLink, loss: Arc<AtomicBool>) -> SharedLink {
Arc::new(FaultLink {
raw,
loss,
sequence: AtomicU64::new(0),
})
}
#[derive(Debug)]
struct FaultLink {
raw: SharedLink,
loss: Arc<AtomicBool>,
sequence: AtomicU64,
}
impl PacketLink for FaultLink {
fn network(&self) -> NetworkId {
self.raw.network()
}
fn peer(&self) -> iroh::EndpointId {
self.raw.peer()
}
fn max_datagram_size(&self) -> usize {
self.raw.max_datagram_size()
}
fn send(&self, bytes: Bytes) -> Result<(), TransportError> {
if self.loss.load(Ordering::Relaxed)
&& !self
.sequence
.fetch_add(1, Ordering::Relaxed)
.is_multiple_of(5)
{
Ok(())
} else {
self.raw.send(bytes)
}
}
fn recv(&self) -> BoxFuture<'_, Option<Bytes>> {
self.raw.recv()
}
fn closed(&self) -> BoxFuture<'_, ()> {
self.raw.closed()
}
fn is_closed(&self) -> bool {
self.raw.is_closed()
}
fn close(&self) {
self.raw.close();
}
fn path_description(&self) -> String {
self.raw.path_description()
}
}
#[tokio::test]
async fn persistent_runtime_loss_retires_the_pair_and_negotiates_fresh_parameters() {
let discovery = SharedMemoryDiscovery::new();
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
let a = Node::open(roots[0].path(), &discovery, "lossa").await;
let b = Node::open(roots[1].path(), &discovery, "lossb").await;
let (name, secret) = network("awg-loss");
let id = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
a.ready(id, 1).await;
b.ready(id, 1).await;
exchange(&a, &b, id).await;
let old = a.transport.overview(id)[0].attempt;
let mut events = a.agent.subscribe();
a.loss.store(true, Ordering::Relaxed);
tsunagi::testing::wait_event(&mut events, |event| match event {
tsunagi::Event::DataLinkDown {
network, reason, ..
} if *network == id && reason.contains("delivery checks") => Some(()),
_ => None,
})
.await;
a.loss.store(false, Ordering::Relaxed);
wait_until("fresh AWG profile after runtime loss", || async {
a.transport
.overview(id)
.iter()
.any(|p| p.attempt != old && !p.closed && !p.path.contains("establishing"))
.then_some(())
})
.await;
a.ready(id, 1).await;
b.ready(id, 1).await;
let initiator = if a.agent.endpoint_id().as_bytes() < b.agent.endpoint_id().as_bytes() {
&a
} else {
&b
};
let received = initiator.plugin.overview(id).unwrap().peers[0]
.tunnel
.as_ref()
.unwrap()
.stats
.rx_packets;
let attempt = initiator.transport.overview(id)[0].attempt;
wait_until(
"fresh pair delivery checks answered after installation",
|| async {
let view = initiator.plugin.overview(id)?;
(view.peers.first()?.tunnel.as_ref()?.stats.rx_packets >= received + 3).then_some(())
},
)
.await;
assert_eq!(initiator.transport.overview(id)[0].attempt, attempt);
wait_for_peers(&a.agent, id, 1).await;
exchange(&a, &b, id).await;
exchange(&b, &a, id).await;
a.agent.shutdown().await;
b.agent.shutdown().await;
}
#[tokio::test]
async fn runtime_loss_falls_back_to_real_iroh_without_interrupting_control_or_ip() {
let discovery = SharedMemoryDiscovery::new();
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
let a = Node::open_with_fallback(roots[0].path(), &discovery, "fallba", true).await;
let b = Node::open_with_fallback(roots[1].path(), &discovery, "fallbb", true).await;
let (name, secret) = network("awg-fallback");
let id = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
a.ready(id, 1).await;
b.ready(id, 1).await;
exchange(&a, &b, id).await;
let mut events = a.agent.subscribe();
a.loss.store(true, Ordering::Relaxed);
tsunagi::testing::wait_event(&mut events, |event| match event {
tsunagi::Event::DataLinkDown {
network, reason, ..
} if *network == id && reason.contains("delivery checks") => Some(()),
_ => None,
})
.await;
for node in [&a, &b] {
wait_until("fallback carrier is wg-quic",||async {
let status=node.agent.status().await.ok()?;
let network=status.networks.iter().find(|n|n.network_id==id)?;
(network.members.iter().any(|m|matches!(&m.data_path,tsunagi::agent::DataPath::Direct{transport} if transport=="wg-quic"))).then_some(())
}).await;
wait_for_peers(&node.agent, id, 1).await;
}
exchange(&a, &b, id).await;
exchange(&b, &a, id).await;
a.agent.shutdown().await;
b.agent.shutdown().await;
}
#[tokio::test]
async fn opaque_pair_carrier_fragments_its_full_advertised_capacity() {
let discovery = SharedMemoryDiscovery::new();
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
let mut nodes = Vec::new();
for root in &roots {
let identity = tsunagi::storage::StateStore::open(
tsunagi::StoragePaths::under(root.path()).state_db(),
)
.unwrap()
.load_or_create_device_identity()
.unwrap();
let transport = AwgTransport::new(identity.endpoint_id(), Ipv4Addr::LOCALHOST);
let agent = Agent::spawn(
config_with(root.path(), &discovery)
.with_custom_transport(AWG_PROTOCOL, transport.clone()),
)
.await
.unwrap();
nodes.push((agent, transport));
}
let (name, secret) = network("awg-capacity");
let id = nodes[0].0.join_network(&name, &secret).await.unwrap();
nodes[1].0.join_network(&name, &secret).await.unwrap();
for (i, (agent, transport)) in nodes.iter().enumerate() {
wait_for_peers(agent, id, 1).await;
// This fixture tests only the transport contract: register the member
// after the real core has authenticated it, without an IP plugin/TUN.
transport.allow_peer(id, nodes[1 - i].0.endpoint_id());
}
nodes.sort_by_key(|(agent, _)| *agent.endpoint_id().as_bytes());
let outgoing = nodes[0]
.1
.open(id, nodes[1].0.endpoint_id(), AWG_PROTOCOL)
.await
.unwrap();
let incoming = tokio::time::timeout(Duration::from_secs(5), nodes[1].1.next_inbound())
.await
.unwrap()
.unwrap()
.link;
for (from, to) in [(&outgoing, &incoming), (&incoming, &outgoing)] {
let mut bytes = vec![0; from.max_datagram_size()];
rand::fill(&mut bytes[..]);
let bytes = Bytes::from(bytes);
from.send(bytes.clone()).unwrap();
assert_eq!(
tokio::time::timeout(Duration::from_secs(5), to.recv())
.await
.unwrap()
.unwrap(),
bytes
);
assert!(matches!(
from.send(Bytes::from(vec![0; from.max_datagram_size() + 1])),
Err(TransportError::TooLarge { .. })
));
}
outgoing.close();
incoming.close();
for (agent, transport) in nodes {
transport.shutdown();
agent.shutdown().await;
}
}
#[tokio::test]
async fn encrypted_health_checks_fit_a_smaller_peers_configured_interface_mtu() {
let discovery = SharedMemoryDiscovery::new();
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
let a = Node::open_config(roots[0].path(), &discovery, "smalla", false, 1280).await;
let b = Node::open_config(roots[1].path(), &discovery, "smallb", false, 576).await;
let (name, secret) = network("awg-small-mtu");
let id = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
a.ready(id, 1).await;
b.ready(id, 1).await;
let bytes = packet(a.addr(id), b.addr(id), 576);
a.tun().push_from_os(bytes.clone());
assert_eq!(
tokio::time::timeout(Duration::from_secs(5), b.tun().pop_to_os())
.await
.unwrap()
.unwrap(),
bytes
);
wait_until(
"encrypted delivery checks answered with a small MTU",
|| async {
let view = a.plugin.overview(id)?;
let peer = view.peers.first()?.tunnel.as_ref()?;
(peer.stats.rx_packets >= 3).then_some(())
},
)
.await;
for node in [&a, &b] {
assert_eq!(
node.plugin.overview(id).unwrap().peers[0]
.tunnel
.as_ref()
.unwrap()
.stats
.dropped_oversize,
0
);
}
a.agent.shutdown().await;
b.agent.shutdown().await;
}
+36
View File
@@ -0,0 +1,36 @@
//! Fresh and bounded masking profiles, and rejection of foreign/truncated wire packets.
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use tsunagi_awg::profile::PairProfile;
#[test]
fn profiles_are_fresh_bounded_and_redacted() {
let profiles: Vec<_> = (0..128).map(|_| PairProfile::generate()).collect();
for (i, profile) in profiles.iter().enumerate() {
profile.validate().unwrap();
assert!(!format!("{profile:?}").contains(&format!("{:?}", profile.header_key())));
for other in &profiles[..i] {
assert_ne!(profile.header_key(), other.header_key());
assert_ne!(profile.id(), other.id());
}
for packet in profile.signature_packets() {
assert!(packet.len() < 1200);
}
}
}
#[test]
fn masked_packets_round_trip_and_reject_other_profiles_and_truncation() {
let profile = PairProfile::generate();
let other = PairProfile::generate();
for (kind, size) in [(1, 148), (2, 92), (3, 64), (4, 800)] {
let mut message = vec![0x7b; size];
message[..4].copy_from_slice(&(kind as u32).to_le_bytes());
let wire = profile.encode(&message).unwrap();
assert_ne!(&wire[..4], &message[..4]);
assert_eq!(profile.decode(&wire).unwrap(), message);
assert!(other.decode(&wire).is_none());
for length in 0..44 {
assert!(profile.decode(&wire[..length]).is_none());
}
}
}
+407 -161
View File
@@ -27,14 +27,14 @@ mod status;
pub use events::Event;
pub use status::{
AgentStatus, CandidateStatus, DataPath, HostnameConflict, MemberStatus, NetworkMetrics,
NetworkState, NetworkStatus, OverlayStatus, PeerStatus,
NetworkState, NetworkStatus, OverlayStatus, PeerStatus, RangeConflict,
};
use std::collections::HashMap;
use std::sync::{Arc, Weak};
use iroh::{EndpointAddr, EndpointId};
use tokio::sync::{RwLock, broadcast, mpsc, oneshot};
use tokio::sync::{Mutex, RwLock, broadcast, mpsc, oneshot, watch};
use tokio::task::JoinHandle;
use crate::BoxFuture;
@@ -58,6 +58,8 @@ use shutdown::Shutdown;
/// Summary of a configured network, whether or not it is running.
#[derive(Debug, Clone)]
pub struct ConfiguredNetwork {
/// Enabled local data-plane protocols, in preference order.
pub protocols: Vec<String>,
/// Saved local broadcast participation.
pub broadcast: bool,
/// Public network identifier.
@@ -90,6 +92,10 @@ struct Inner {
events: broadcast::Sender<Event>,
networks: RwLock<HashMap<NetworkId, NetworkHandle>>,
shutdown: Shutdown,
/// Serialises start, stop, switching and conflict cleanup, including teardown.
lifecycle: Mutex<()>,
range_changed: watch::Sender<()>,
range_task: std::sync::Mutex<Option<JoinHandle<()>>>,
accept_task: std::sync::Mutex<Option<JoinHandle<()>>>,
plugin_task: std::sync::Mutex<Option<JoinHandle<()>>>,
transport: std::sync::OnceLock<Arc<dyn PacketTransport>>,
@@ -100,6 +106,37 @@ struct Inner {
}
impl Inner {
/// Called under the lifecycle lock after startup: an initially
/// control-only agent creates its one interface on first enablement.
async fn ensure_interface(self: &Arc<Self>) {
if self.interface.get().is_some() {
return;
}
if let Some(factory) = self.config.tun_factory.clone() {
let carrier = Arc::new(Carrier(Arc::downgrade(self))) as Arc<dyn PacketCarrier>;
match crate::overlay::Interface::start(
factory,
self.config.interface_name.clone(),
self.config.interface_mtu,
Arc::clone(&self.routes),
carrier,
)
.await
{
Ok(interface) => {
let _ = self.interface.set(Arc::new(interface));
}
Err(err) => {
let _ = self.events.send(Event::PluginError {
network: NetworkId::from_bytes([0u8; 32]),
protocol: "overlay".into(),
reason: err.to_string(),
});
}
}
}
}
/// The name this agent currently answers to.
fn read_hostname(&self) -> String {
match self.hostname.read() {
@@ -165,6 +202,21 @@ impl PacketSink for Sink {
let Some(inner) = self.0.upgrade() else {
return;
};
if let Some(payload) = crate::dataplane::transport::health::ip_payload(&packet) {
if let Some(header) = crate::overlay::IpHeader::parse(&packet)
&& let (std::net::IpAddr::V4(source), std::net::IpAddr::V4(destination)) =
(header.source(), header.destination())
&& inner.routes.may_send_from(network, peer, source)
&& inner.routes.is_local_destination(network, destination)
&& let Some(handle) = inner.networks.read().await.get(&network)
{
let _ = handle.commands.try_send(NetCommand::HealthProbe {
peer,
packet: bytes::Bytes::copy_from_slice(payload),
});
}
return;
}
let Some(interface) = inner.interface.get() else {
return;
};
@@ -183,6 +235,35 @@ impl PacketSink for Sink {
#[derive(Debug)]
struct TransportCtx(Weak<Inner>);
impl crate::dataplane::transport::ControlSender for TransportCtx {
fn send<'a>(
&'a self,
network: NetworkId,
peer: EndpointId,
protocol: &'a str,
payload: crate::proto::message::TransportPayload,
) -> crate::BoxFuture<'a, std::result::Result<(), crate::dataplane::transport::TransportError>>
{
Box::pin(async move {
let inner = self
.0
.upgrade()
.ok_or(crate::dataplane::transport::TransportError::Closed)?;
Agent { inner }
.send(
network,
peer,
crate::proto::message::ControlMessage::Transport {
protocol: protocol.into(),
payload,
},
)
.await
.map_err(|err| crate::dataplane::transport::TransportError::Other(err.to_string()))
})
}
}
impl TransportContext for TransportCtx {
fn snapshot(&self) -> crate::BoxFuture<'_, HashMap<NetworkId, NetworkKeys>> {
Box::pin(async move {
@@ -217,12 +298,90 @@ impl TransportContext for TransportCtx {
}
impl Agent {
fn effective_protocols(&self, saved: Option<&Vec<String>>) -> Vec<String> {
saved
.or(self.inner.config.default_protocols.as_ref())
.cloned()
.unwrap_or_else(|| {
self.inner
.config
.plugins
.iter()
.map(|p| p.protocol_id().to_owned())
.collect()
})
}
fn selected_plugins(&self, protocols: &[String]) -> Vec<crate::dataplane::SharedPlugin> {
protocols
.iter()
.filter_map(|name| {
self.inner
.config
.plugins
.iter()
.find(|p| p.protocol_id() == name)
.cloned()
})
.collect()
}
/// Saves and immediately applies a network's ordered local protocol set.
/// An empty set disables data traffic while retaining control sessions.
pub async fn set_protocols(&self, network_id: NetworkId, protocols: Vec<String>) -> Result<()> {
crate::dataplane::validate_protocols(&protocols, self.inner.limits.max_capabilities)?;
for name in &protocols {
if !self
.inner
.config
.plugins
.iter()
.any(|p| p.protocol_id() == name)
{
return Err(Error::InvalidProtocols(format!(
"this agent has no `{name}` plugin"
)));
}
}
let _lifecycle = self.inner.lifecycle.lock().await;
self.inner
.storage
.set_protocols(network_id, protocols.clone())
.await?;
if self.is_active(network_id).await {
let (reply, receive) = oneshot::channel();
if !protocols.is_empty() {
self.inner.ensure_interface().await;
}
self.command(
network_id,
NetCommand::SetProtocols {
protocols,
interface: self.inner.interface.get().cloned(),
reply,
},
)
.await?;
receive.await.map_err(|_| Error::Stopped)?;
}
Ok(())
}
/// Starts an agent.
///
/// Opens the state store (taking its ownership lock), restores the
/// persistent device identity, binds the iroh endpoint and activates every
/// configured network whose auto-start flag is set.
pub async fn spawn(config: AgentConfig) -> Result<Self> {
if let Some(protocols) = &config.default_protocols {
crate::dataplane::validate_protocols(protocols, config.limits.max_capabilities)?;
for name in protocols {
if !config.plugins.iter().any(|p| p.protocol_id() == name) {
return Err(Error::InvalidProtocols(format!(
"this agent has no `{name}` plugin"
)));
}
}
}
let storage = Storage::open(&config.paths)?;
let identity = storage.device_identity().await?;
let adapter = EndpointAdapter::bind(&config, &identity).await?;
@@ -234,6 +393,7 @@ impl Agent {
let (events, _) = broadcast::channel(config.limits.event_buffer);
let limits = Arc::new(config.limits.clone());
let (range_changed, range_updates) = watch::channel(());
let inner = Arc::new(Inner {
limits,
storage,
@@ -243,6 +403,9 @@ impl Agent {
events,
networks: RwLock::new(HashMap::new()),
shutdown: Shutdown::new(),
lifecycle: Mutex::new(()),
range_changed,
range_task: std::sync::Mutex::new(None),
accept_task: std::sync::Mutex::new(None),
plugin_task: std::sync::Mutex::new(None),
transport: std::sync::OnceLock::new(),
@@ -251,31 +414,13 @@ impl Agent {
config,
});
// One interface for the agent, if it was given a way to make one.
// Failing to is reported and not fatal: the control plane works, and
// so do the protocols, they just have nowhere to put packets.
if let Some(factory) = inner.config.tun_factory.clone() {
let carrier = Arc::new(Carrier(Arc::downgrade(&inner))) as Arc<dyn PacketCarrier>;
match crate::overlay::Interface::start(
factory,
inner.config.interface_name.clone(),
inner.config.interface_mtu,
Arc::clone(&inner.routes),
carrier,
)
.await
{
Ok(interface) => {
let _ = inner.interface.set(Arc::new(interface));
}
Err(err) => {
let _ = inner.events.send(Event::PluginError {
network: NetworkId::from_bytes([0u8; 32]),
protocol: "overlay".into(),
reason: err.to_string(),
});
}
}
if inner
.config
.default_protocols
.as_ref()
.is_none_or(|names| !names.is_empty())
{
inner.ensure_interface().await;
}
// The data plane rides on iroh too, which is where it gets hole
@@ -294,6 +439,7 @@ impl Agent {
inner.config.custom_transports.clone(),
))
};
transport.attach_control(Arc::new(TransportCtx(Arc::downgrade(&inner))));
let _ = inner.transport.set(transport);
if let CacheOutcome::Reset(reason) = inner.storage.cache_outcome().clone() {
@@ -311,8 +457,9 @@ impl Agent {
if !inner.config.plugins.is_empty() {
let (plugin_tx, plugin_rx) = mpsc::channel(64);
let sink = inner
.interface
.get()
.config
.tun_factory
.as_ref()
.map(|_| Arc::new(Sink(Arc::downgrade(&inner))) as Arc<dyn PacketSink>);
let context = PluginContext::new(plugin_tx, inner.identity.endpoint_id(), sink);
for plugin in &inner.config.plugins {
@@ -326,11 +473,16 @@ impl Agent {
let agent = Self { inner };
for stored in agent.inner.storage.list_networks().await? {
if stored.auto_start {
let keys = NetworkKeys::derive(&stored.name, &stored.secret);
agent.activate_with_keys(keys).await?;
}
{
let _guard = agent.inner.lifecycle.lock().await;
agent.resume_networks_locked().await?;
}
let task = tokio::spawn(range_conflict_loop(
Arc::downgrade(&agent.inner),
range_updates,
));
if let Ok(mut guard) = agent.inner.range_task.lock() {
*guard = Some(task);
}
Ok(agent)
@@ -374,100 +526,65 @@ impl Agent {
Ok(())
}
/// Reserves the configured overlay range for a network, if it can.
///
/// Done here, where activations are serialised, rather than inside the
/// runtime: whether a range is free depends on what the other networks
/// took, and deciding that in a task would make the answer depend on
/// which task ran first.
///
/// `None` means this agent will not propose a range for that network and
/// waits to adopt whatever it settles on. One agent has one interface, so
/// proposing a range it could not route would be worse than having none:
/// the lowest author's range wins, and the collision would spread.
async fn reserve_range(&self, network: NetworkId) -> RangePlan {
let reserve = |wanted: Ipv4Range| {
let reservation = crate::overlay::NetworkRoutes {
exit: Default::default(),
broadcast: Default::default(),
range: Some(wanted),
local: None,
peers: Vec::new(),
};
self.inner.routes.set_network(network, reservation).is_ok()
};
// If this network already has signed records in storage, check if its
// members have already agreed on a range. If so, preserve and reserve
// THAT range rather than preemptively seizing the default configured range.
let stored_records = self
.inner
.storage
.signed_records(network)
.await
.unwrap_or_default();
let mut state_set = crate::state::StateSet::new();
state_set.merge_all(network, stored_records);
if let Some(agreed) = state_set.agreed_range() {
if reserve(agreed) {
return RangePlan {
propose: if Some(agreed) == self.inner.config.overlay_ipv4_range {
Some(agreed)
} else {
None
},
..RangePlan::default()
};
/// Signed state always wins over a default or an explicit proposal.
async fn desired_range(&self, network: NetworkId) -> Result<Option<Ipv4Range>> {
let records = self.inner.storage.signed_records(network).await?;
let mut state = crate::state::StateSet::new();
state.merge_all(network, records);
Ok(state.agreed_range().or_else(|| {
if self.inner.config.overlay_ipv4_auto {
Some(crate::state::derived_ipv4_range(network))
} else {
tracing::warn!(
%network,
range = %agreed,
"agreed range for this network overlaps another network already active on this interface"
);
return RangePlan {
conflict: Some(agreed),
..RangePlan::default()
};
self.inner.config.overlay_ipv4_range
}
}))
}
async fn range_conflicts(
&self,
network: NetworkId,
range: Option<Ipv4Range>,
) -> Result<Vec<RangeConflict>> {
let Some(range) = range else {
return Ok(Vec::new());
};
let overlapping = self.inner.routes.overlapping_networks(network, range);
let stored = self.inner.storage.list_networks().await?;
Ok(overlapping
.into_iter()
.filter_map(|(id, range)| {
let name = stored.iter().find(|n| n.network_id == id)?.name.clone();
Some(RangeConflict {
network_id: id,
name,
range,
})
})
.collect())
}
/// Starts requested networks in saved preference order, skipping overlaps.
/// Called with the lifecycle lock held; skipped networks remain requested.
async fn resume_networks_locked(&self) -> Result<()> {
let order = self.inner.storage.network_start_order().await?;
let mut stored = self.inner.storage.list_networks().await?;
stored.sort_by_key(|n| {
order
.iter()
.position(|id| *id == n.network_id)
.unwrap_or(usize::MAX)
});
for network in stored {
if !network.auto_start || self.is_active(network.network_id).await {
continue;
}
let keys = NetworkKeys::derive(&network.name, &network.secret);
match self.activate_with_keys_locked(keys).await {
Ok(()) | Err(Error::NetworkRangeConflict { .. }) => {}
Err(err) => return Err(err),
}
}
let Some(configured) = self.inner.config.overlay_ipv4_range else {
return RangePlan::default();
};
// The configured range first, so a network a device has always had
// keeps the addresses it has always had.
if reserve(configured) {
return RangePlan {
propose: Some(configured),
..RangePlan::default()
};
}
// A second network on the same agent cannot have it — one agent,
// one interface — so it falls back to the range derived from its
// own id, which every one of its members derives identically
// without being told.
let derived = crate::state::derived_ipv4_range(network);
if reserve(derived) {
tracing::info!(
%network,
range = %derived,
"another network here holds the configured range; this one uses the \
range derived from its id unless its members settled another"
);
return RangePlan {
fallback: Some(derived),
conflict: Some(configured),
..RangePlan::default()
};
}
tracing::info!(%network, "not proposing a range for this network");
RangePlan {
conflict: Some(configured),
..RangePlan::default()
}
Ok(())
}
/// The overlay interface this agent owns, when it has one.
@@ -546,6 +663,9 @@ impl Agent {
/// The same `(name, secret)` always produces the same [`NetworkId`], on
/// every device.
///
/// An overlapping network stays configured and requested, but inactive;
/// use [`Agent::switch_network`] to suspend the blockers explicitly.
///
/// This is declarative and therefore **idempotent**: joining a network
/// that is already active succeeds and changes nothing. That matters
/// because a configured network is activated automatically at startup, so
@@ -599,7 +719,9 @@ impl Agent {
}
match self.activate_with_keys(keys).await {
// Already a member of exactly this network space: nothing to do.
Ok(()) | Err(Error::NetworkAlreadyActive(_)) => Ok(network_id),
Ok(())
| Err(Error::NetworkAlreadyActive(_))
| Err(Error::NetworkRangeConflict { .. }) => Ok(network_id),
Err(err) => Err(err),
}
}
@@ -703,7 +825,44 @@ impl Agent {
self.activate_with_keys(keys).await
}
/// Switches to a configured network, suspending every overlapping network.
/// Suspended networks retain their auto-start request and resume when free.
/// Returns the networks suspended by this request.
pub async fn switch_network(&self, network_id: NetworkId) -> Result<Vec<NetworkId>> {
let _guard = self.inner.lifecycle.lock().await;
if self.inner.shutdown.is_triggered() {
return Err(Error::Stopped);
}
let stored = self
.inner
.storage
.list_networks()
.await?
.into_iter()
.find(|n| n.network_id == network_id)
.ok_or(Error::NetworkUnknown(network_id))?;
let range = self.desired_range(network_id).await?;
let conflicts = self.range_conflicts(network_id, range).await?;
self.inner.storage.prefer_network(network_id).await?;
let mut stopped = Vec::new();
for conflict in conflicts {
self.stop_network_locked(conflict.network_id).await?;
stopped.push(conflict.network_id);
}
if !self.is_active(network_id).await {
self.activate_with_keys_locked(NetworkKeys::derive(&stored.name, &stored.secret))
.await?;
}
self.resume_networks_locked().await?;
Ok(stopped)
}
async fn activate_with_keys(&self, keys: NetworkKeys) -> Result<()> {
let _guard = self.inner.lifecycle.lock().await;
self.activate_with_keys_locked(keys).await
}
async fn activate_with_keys_locked(&self, keys: NetworkKeys) -> Result<()> {
if self.inner.shutdown.is_triggered() {
return Err(Error::Stopped);
}
@@ -712,8 +871,36 @@ impl Agent {
if networks.contains_key(&network_id) {
return Err(Error::NetworkAlreadyActive(network_id));
}
let range = self.reserve_range(network_id).await;
let stored = self
.inner
.storage
.list_networks()
.await?
.into_iter()
.find(|stored| stored.network_id == network_id);
let range = self.desired_range(network_id).await?;
if let Some(range) = range {
let conflicts = self.inner.routes.overlapping_networks(network_id, range);
if !conflicts.is_empty() {
return Err(Error::NetworkRangeConflict {
network: network_id,
range,
conflicts: conflicts.into_iter().map(|(id, _)| id).collect(),
});
}
self.inner
.routes
.set_network(
network_id,
crate::overlay::NetworkRoutes {
range: Some(range),
..Default::default()
},
)
.map_err(|err| Error::Rejected {
reason: err.to_string(),
})?;
}
let mut backends = Vec::new();
if let Some(discovery) = &self.inner.config.discovery {
backends.push(discovery.clone());
@@ -729,14 +916,14 @@ impl Agent {
as Arc<dyn crate::discovery::NetworkDiscovery>,
)
};
let stored = self
.inner
.storage
.list_networks()
.await?
.into_iter()
.find(|stored| stored.network_id == network_id);
let broadcast = stored.as_ref().is_none_or(|stored| stored.broadcast);
let protocols =
self.effective_protocols(stored.as_ref().and_then(|s| s.protocols.as_ref()));
let plugins = self.selected_plugins(&protocols);
if !plugins.is_empty() {
self.inner.ensure_interface().await;
}
let exit_offer = stored.as_ref().is_some_and(|stored| stored.exit_node);
let exit_via = stored
.as_ref()
@@ -755,7 +942,8 @@ impl Agent {
discovery,
discovery_interval: self.inner.config.discovery_interval,
discovery_policy: self.inner.config.discovery_policy.clone(),
plugins: self.inner.config.plugins.clone(),
available_plugins: self.inner.config.plugins.clone(),
plugins: plugins.clone(),
routes: Arc::clone(&self.inner.routes),
interface: self.inner.interface.get().cloned(),
hostname: self.inner.read_hostname(),
@@ -763,14 +951,13 @@ impl Agent {
device_secret: self.inner.identity.signing_key(),
#[cfg(feature = "testing")]
unreachable_data_peers: Arc::clone(&self.inner.config.unreachable_data_peers),
ipv4_range: range.propose,
ipv4_fallback: range.fallback,
range_conflict: range.conflict,
ipv4_range: range,
range_changed: self.inner.range_changed.clone(),
});
networks.insert(network_id, handle);
drop(networks);
for plugin in &self.inner.config.plugins {
for plugin in &plugins {
plugin.on_network_activated(network_id);
}
@@ -786,6 +973,30 @@ impl Agent {
/// it does not remove this agent from anyone else's view of the network.
/// Other networks keep running.
pub async fn deactivate_network(&self, network_id: NetworkId) -> Result<()> {
let _guard = self.inner.lifecycle.lock().await;
if !self.is_active(network_id).await {
return Err(Error::NetworkNotActive(network_id));
}
self.inner.storage.set_auto_start(network_id, false).await?;
self.stop_network_locked(network_id).await?;
self.resume_networks_locked().await
}
/// Stops a configured network, including one already suspended by overlap.
/// Clears its auto-start request so it will not resume behind the user.
pub async fn pause_network(&self, network_id: NetworkId) -> Result<()> {
let _guard = self.inner.lifecycle.lock().await;
if self.inner.shutdown.is_triggered() {
return Err(Error::Stopped);
}
self.inner.storage.set_auto_start(network_id, false).await?;
if self.is_active(network_id).await {
self.stop_network_locked(network_id).await?;
}
self.resume_networks_locked().await
}
async fn stop_network_locked(&self, network_id: NetworkId) -> Result<()> {
let handle = {
let mut networks = self.inner.networks.write().await;
networks
@@ -814,7 +1025,6 @@ impl Agent {
interface.sync_broadcast_rules().await;
interface.sync_exit_rules().await;
}
self.inner.storage.set_auto_start(network_id, false).await?;
Ok(())
}
@@ -824,17 +1034,15 @@ impl Agent {
/// Local and silent: nobody is told. [`Agent::leave_network`] is the one
/// that says goodbye first, and is what a person means by leaving.
pub async fn forget_network(&self, network_id: NetworkId) -> Result<()> {
let _guard = self.inner.lifecycle.lock().await;
if self.is_active(network_id).await {
self.deactivate_network(network_id).await?;
self.stop_network_locked(network_id).await?;
}
self.inner.storage.remove_network(network_id).await?;
// Whatever a protocol kept for this network goes with it. The
// agent cannot know what that is, only that there is no longer any
// reason to hold it.
for plugin in &self.inner.config.plugins {
plugin.on_network_forgotten(network_id);
}
Ok(())
self.resume_networks_locked().await
}
/// Leaves a network: gives up what was claimed, then forgets it.
@@ -890,6 +1098,7 @@ impl Agent {
.await?
.into_iter()
.map(|stored| ConfiguredNetwork {
protocols: self.effective_protocols(stored.protocols.as_ref()),
broadcast: stored.broadcast,
network_id: stored.network_id,
name: stored.name,
@@ -958,13 +1167,18 @@ impl Agent {
let mut networks = Vec::new();
for stored in self.inner.storage.list_networks().await? {
if active.contains(&stored.network_id)
&& let Ok(status) = self.network_status(stored.network_id).await
&& let Ok(mut status) = self.network_status(stored.network_id).await
{
status.auto_start = stored.auto_start;
networks.push(status);
continue;
}
let keys = NetworkKeys::derive(&stored.name, &stored.secret);
let range = self.desired_range(stored.network_id).await?;
let range_conflicts = self.range_conflicts(stored.network_id, range).await?;
networks.push(NetworkStatus {
protocols: self.effective_protocols(stored.protocols.as_ref()),
auto_start: stored.auto_start,
broadcast: stored.broadcast,
exit_offer: stored.exit_node,
exit_via: stored
@@ -978,12 +1192,13 @@ impl Agent {
state: NetworkState::Inactive,
peers: Vec::new(),
candidates: Vec::new(),
// An inactive network has no runtime to ask; the roster and
// the range come from one. Empty, not invented.
// The range is available from signed state or the deterministic
// default even while the network has no runtime.
members: Vec::new(),
hostname_conflict: None,
range: None,
range_conflict: None,
range,
range_conflict: range_conflicts.first().map(|conflict| conflict.range),
range_conflicts,
metrics: NetworkMetrics::default(),
relay: Default::default(),
});
@@ -1039,6 +1254,7 @@ impl Agent {
/// After this returns, the state directory can be opened by another agent.
pub async fn shutdown(&self) {
self.inner.shutdown.trigger();
let lifecycle_guard = self.inner.lifecycle.lock().await;
let handles: Vec<NetworkHandle> = {
let mut networks = self.inner.networks.write().await;
@@ -1048,6 +1264,7 @@ impl Agent {
handle.stop().await;
}
drop(lifecycle_guard);
if let Some(dht) = &self.inner.config.dht {
dht.shutdown().await;
}
@@ -1064,6 +1281,7 @@ impl Agent {
for (what, handle) in [
("inbound connections", &self.inner.accept_task),
("plugin requests", &self.inner.plugin_task),
("range conflicts", &self.inner.range_task),
] {
let task = handle.lock().ok().and_then(|mut guard| guard.take());
if let Some(task) = task {
@@ -1107,16 +1325,44 @@ impl Agent {
}
}
/// What a network may propose as its overlay range, and what it cannot have.
#[derive(Debug, Clone, Copy, Default)]
struct RangePlan {
/// Proposed straight away: the configured range, when it is free here.
propose: Option<Ipv4Range>,
/// Proposed after a moment, when the configured range is another
/// network's: the range derived from this network's own id.
fallback: Option<Ipv4Range>,
/// The configured range, when this network cannot have it.
conflict: Option<Ipv4Range>,
/// Processes coalesced conflict notifications without blocking a network actor.
async fn range_conflict_loop(inner: Weak<Inner>, mut updates: watch::Receiver<()>) {
loop {
let Some(owner) = inner.upgrade() else {
return;
};
let shutdown = owner.shutdown.clone();
drop(owner);
tokio::select! {
_ = shutdown.wait() => return,
result = updates.changed() => if result.is_err() { return; },
}
let Some(inner) = inner.upgrade() else {
return;
};
let agent = Agent { inner };
let _guard = agent.inner.lifecycle.lock().await;
if agent.inner.shutdown.is_triggered() {
return;
}
let blocked: Vec<_> = agent
.inner
.networks
.read()
.await
.iter()
.filter(|(_, h)| h.range_blocked.load(std::sync::atomic::Ordering::Acquire))
.map(|(id, _)| *id)
.collect();
for id in blocked {
if let Err(err) = agent.stop_network_locked(id).await {
tracing::warn!(%err, "cannot suspend overlapping network");
}
}
if let Err(err) = agent.resume_networks_locked().await {
tracing::warn!(%err, "cannot resume suspended networks");
}
}
}
/// How long a release is given to reach the sessions it was queued on.
+383 -100
View File
@@ -15,6 +15,7 @@ use tokio::sync::{broadcast, mpsc, oneshot};
use tokio::task::JoinHandle;
use crate::config::{DiscoveryPolicy, Limits, ReconnectPolicy};
use crate::dataplane::transport::health::{HealthLink, ProbeRoute, Settings as HealthSettings};
use crate::dataplane::transport::{InboundLink, PacketLink, PacketTransport, SharedLink};
use crate::dataplane::{PluginCapability, SharedPlugin};
use crate::discovery::worker::DiscoveryWorker;
@@ -59,6 +60,10 @@ pub(crate) enum NetCommand {
},
/// A peer opened a data plane link towards us.
InboundLink(Box<InboundLink>),
HealthProbe {
peer: EndpointId,
packet: bytes::Bytes,
},
Status {
reply: oneshot::Sender<Box<NetworkStatus>>,
},
@@ -67,6 +72,11 @@ pub(crate) enum NetCommand {
enabled: bool,
reply: oneshot::Sender<()>,
},
SetProtocols {
interface: Option<Arc<crate::overlay::Interface>>,
protocols: Vec<String>,
reply: oneshot::Sender<()>,
},
/// Offer this agent as an exit node in this network, or stop.
SetExitOffer {
enabled: bool,
@@ -117,12 +127,14 @@ impl std::fmt::Debug for NetCommand {
NetCommand::Status { .. } => f.write_str("Status"),
NetCommand::Recheck => f.write_str("Recheck"),
NetCommand::SetBroadcast { enabled, .. } => write!(f, "SetBroadcast({enabled})"),
NetCommand::SetProtocols { protocols, .. } => write!(f, "SetProtocols({protocols:?})"),
NetCommand::SetExitOffer { enabled, .. } => write!(f, "SetExitOffer({enabled})"),
NetCommand::SetExitVia { via, .. } => write!(f, "SetExitVia({})", via.is_some()),
NetCommand::Reannounce => f.write_str("Reannounce"),
NetCommand::SetHostname(_) => f.write_str("SetHostname"),
NetCommand::Release { .. } => f.write_str("Release"),
NetCommand::PluginError { protocol, .. } => write!(f, "PluginError({protocol})"),
NetCommand::HealthProbe { peer, .. } => write!(f, "HealthProbe({})", peer.fmt_short()),
NetCommand::ProtocolFailure { peer, protocol, .. } => {
write!(f, "ProtocolFailure({}, {protocol})", peer.fmt_short())
}
@@ -137,6 +149,7 @@ pub(crate) struct NetworkHandle {
pub(crate) commands: mpsc::Sender<NetCommand>,
shutdown: Shutdown,
task: JoinHandle<()>,
pub(crate) range_blocked: Arc<std::sync::atomic::AtomicBool>,
}
impl NetworkHandle {
@@ -164,6 +177,7 @@ pub(crate) struct RuntimeParams {
pub(crate) discovery_interval: Duration,
pub(crate) discovery_policy: DiscoveryPolicy,
pub(crate) plugins: Vec<SharedPlugin>,
pub(crate) available_plugins: Vec<SharedPlugin>,
/// Who holds which overlay address, shared with every other network.
pub(crate) routes: Arc<crate::overlay::RoutingTable>,
/// The one interface, when the agent has one.
@@ -177,23 +191,15 @@ pub(crate) struct RuntimeParams {
/// The IPv4 overlay range this agent would use, if the network has not
/// already settled on another one.
pub(crate) ipv4_range: Option<Ipv4Range>,
/// The range it was configured with but cannot have, because another
/// network on this agent holds it.
pub(crate) range_conflict: Option<Ipv4Range>,
/// The range derived from this network's own id, proposed when the
/// configured one belongs to another network here.
///
/// Held back at first: a network that already exists has a range of its
/// own, and a member that proposed before hearing anything would be
/// arguing with it instead of adopting it. Every member derives the
/// same one, so once the wait is over there is nothing to argue about.
pub(crate) ipv4_fallback: Option<Ipv4Range>,
/// Coalesced notifications to the agent's lifecycle manager.
pub(crate) range_changed: tokio::sync::watch::Sender<()>,
/// How data plane links are opened. `None` disables the data plane.
pub(crate) transport: Option<Arc<dyn PacketTransport>>,
}
/// Outcome of one attempt to open a data plane link.
struct LinkOutcome {
generation: u64,
peer: EndpointId,
protocol: String,
result: Result<SharedLink, String>,
@@ -236,8 +242,10 @@ pub(crate) fn spawn(params: RuntimeParams) -> NetworkHandle {
let (commands_tx, commands_rx) = mpsc::channel(64);
let runtime_shutdown = shutdown.clone();
let range_blocked = Arc::new(std::sync::atomic::AtomicBool::new(false));
let runtime_blocked = Arc::clone(&range_blocked);
let task = tokio::spawn(async move {
let mut runtime = Runtime::new(params, runtime_shutdown);
let mut runtime = Runtime::new(params, runtime_shutdown, runtime_blocked);
runtime.run(commands_rx).await;
});
@@ -246,6 +254,7 @@ pub(crate) fn spawn(params: RuntimeParams) -> NetworkHandle {
commands: commands_tx,
shutdown,
task,
range_blocked,
}
}
@@ -257,14 +266,6 @@ pub(crate) fn spawn(params: RuntimeParams) -> NetworkHandle {
/// itself, a few announcements later.
const REACH_EXPIRY: Duration = Duration::from_secs(90);
/// How long a network waits to be told its range before proposing the one
/// derived from its id.
///
/// Only for that fallback: the configured range is proposed at once, as it
/// always was. This is the window in which an existing network's records
/// can arrive and be adopted instead.
const RANGE_PROPOSAL_GRACE: Duration = Duration::from_secs(3);
#[derive(Debug, Clone)]
struct ProtocolDialState {
failures: u32,
@@ -274,6 +275,12 @@ struct ProtocolDialState {
const DATA_LINK_BASE_BACKOFF: Duration = Duration::from_secs(3);
const DATA_LINK_MAX_BACKOFF: Duration = Duration::from_secs(60);
fn next_protocol_index(supported: &[String], last_failed: Option<&str>) -> usize {
last_failed
.and_then(|failed| supported.iter().position(|p| p == failed))
.map_or(0, |i| (i + 1) % supported.len().max(1))
}
fn data_link_backoff(failures: u32) -> Duration {
if failures == 0 {
return Duration::ZERO;
@@ -287,8 +294,9 @@ struct Runtime {
discovery_worker: Option<DiscoveryWorker>,
discovery_candidates: mpsc::Receiver<Candidate>,
params: RuntimeParams,
/// When this runtime started, for the fallback range's grace period.
activated: std::time::Instant,
range_blocked: Arc<std::sync::atomic::AtomicBool>,
/// Last range successfully published to the agent's routing table.
published_range: Option<Ipv4Range>,
network_id: NetworkId,
local_id: EndpointId,
shutdown: Shutdown,
@@ -304,7 +312,9 @@ struct Runtime {
///
/// The direct ones. What a protocol holds is the hub's link for that
/// peer, which outlives any particular path.
links: HashMap<(EndpointId, String), SharedLink>,
links: HashMap<(EndpointId, String), Arc<HealthLink>>,
pending_health: HashMap<(EndpointId, String), HealthSettings>,
last_failed_protocol: HashMap<EndpointId, String>,
/// Every data link of this network, and what is reached through what.
hub: Arc<crate::dataplane::relay::RelayHub>,
/// Which peers each peer says it has a live link with, and when it
@@ -330,6 +340,9 @@ struct Runtime {
announced_reach: HashSet<crate::proto::message::ReachableLink>,
/// Links currently being opened, so we do not start two.
opening: HashSet<(EndpointId, String)>,
link_tasks: HashMap<(EndpointId, String), tokio::task::AbortHandle>,
link_attempts: HashMap<(EndpointId, String), u64>,
next_link_attempt: u64,
link_results_tx: mpsc::Sender<LinkOutcome>,
link_results_rx: mpsc::Receiver<LinkOutcome>,
/// Protocol dial failure count and backoff cooldown per peer and protocol.
@@ -366,7 +379,11 @@ struct Runtime {
}
impl Runtime {
fn new(params: RuntimeParams, shutdown: Shutdown) -> Self {
fn new(
params: RuntimeParams,
shutdown: Shutdown,
range_blocked: Arc<std::sync::atomic::AtomicBool>,
) -> Self {
let network_id = params.keys.network_id();
let local_id = params.adapter.endpoint_id();
let (session_events_tx, session_events_rx) = mpsc::channel(256);
@@ -398,7 +415,8 @@ impl Runtime {
discovery_worker,
discovery_candidates,
params,
activated: std::time::Instant::now(),
range_blocked,
published_range: None,
network_id,
local_id,
shutdown,
@@ -411,11 +429,16 @@ impl Runtime {
dial_results_tx,
dial_results_rx,
links: HashMap::new(),
pending_health: HashMap::new(),
last_failed_protocol: HashMap::new(),
hub,
reachable: HashMap::new(),
local_capabilities: Vec::new(),
announced_reach: HashSet::new(),
opening: HashSet::new(),
link_tasks: HashMap::new(),
link_attempts: HashMap::new(),
next_link_attempt: 0,
link_results_tx,
link_results_rx,
protocol_dial_states: HashMap::new(),
@@ -498,7 +521,16 @@ impl Runtime {
// any more, and a sender blocked on a full queue would stall shutdown.
self.session_events_rx.close();
self.hub.close();
for link in self.links.values() {
link.close();
}
self.links.clear();
for (_, task) in self.link_tasks.drain() {
task.abort();
}
self.opening.clear();
self.pending_health.clear();
self.last_failed_protocol.clear();
self.protocol_dial_states.clear();
let peers: Vec<EndpointId> = self.sessions.keys().copied().collect();
for peer in peers {
@@ -539,6 +571,60 @@ impl Runtime {
NetCommand::Status { reply } => {
let _ = reply.send(Box::new(self.status()));
}
NetCommand::SetProtocols {
protocols,
interface,
reply,
} => {
self.params.interface = interface;
let current: Vec<_> = self
.params
.plugins
.iter()
.map(|p| p.protocol_id().to_owned())
.collect();
if current != protocols {
let peers: Vec<_> = self.sessions.keys().copied().collect();
for peer in &peers {
self.reset_peer_data(*peer, "local protocol selection changed");
}
for plugin in &self.params.plugins {
if !protocols.iter().any(|name| name == plugin.protocol_id()) {
plugin.on_network_deactivated(self.network_id);
}
}
self.params.plugins = protocols
.iter()
.filter_map(|name| {
self.params
.available_plugins
.iter()
.find(|p| p.protocol_id() == name)
.cloned()
})
.collect();
for plugin in &self.params.plugins {
if !current.iter().any(|name| name == plugin.protocol_id()) {
plugin.on_network_activated(self.network_id);
}
}
self.hub.set_priority(protocols);
self.publish_allocations().await;
for peer in peers {
let caps = self
.sessions
.get(&peer)
.map(|s| s.capabilities.clone())
.unwrap_or_default();
self.dispatch_capabilities(peer, &caps);
}
self.reannounce();
self.ensure_links();
self.update_paths();
self.announce_reach(true);
}
let _ = reply.send(());
}
NetCommand::SetBroadcast { enabled, reply } => {
self.params.broadcast = enabled;
self.update_broadcast();
@@ -570,7 +656,22 @@ impl Runtime {
let _ = reply.send(outcome);
}
NetCommand::Recheck => self.discovery_round().await,
NetCommand::Reannounce => self.reannounce(),
NetCommand::Reannounce => {
// A newly enabled plugin may prepare its key asynchronously.
// Replay allocations and peer identities once it is ready.
self.publish_allocations().await;
let peers: Vec<_> = self
.sessions
.iter()
.map(|(peer, s)| (*peer, s.capabilities.clone()))
.collect();
for (peer, caps) in peers {
self.dispatch_capabilities(peer, &caps);
}
self.reannounce();
self.ensure_links();
self.update_paths();
}
NetCommand::Release { reply } => {
// A tombstone, so the address and the name this agent held
// are freed for somebody else instead of staying reserved
@@ -602,12 +703,20 @@ impl Runtime {
reason,
});
}
NetCommand::HealthProbe { peer, packet } => {
for ((p, _), link) in &self.links {
if *p == peer {
link.receive_probe(packet.clone());
}
}
}
NetCommand::ProtocolFailure {
peer,
protocol,
reason,
} => {
if self.links.remove(&(peer, protocol.clone())).is_some() {
if let Some(link) = self.links.remove(&(peer, protocol.clone())) {
link.close();
self.hub.clear_direct(peer, &protocol);
}
self.metrics.data_link_failures += 1;
@@ -662,7 +771,7 @@ impl Runtime {
peer: peer.fmt_short().to_string(),
})?;
match session.outbound.try_send(encoded) {
match session.outbound.try_send(zeroize::Zeroizing::new(encoded)) {
Ok(()) => {
session.messages_sent += 1;
session.bytes_sent += bytes;
@@ -968,35 +1077,27 @@ impl Runtime {
if let Some(agreed) = self.state.agreed_range() {
return Some(agreed);
}
// Nobody has settled one yet, so this agent would be proposing its
// own. It must not propose a range it cannot route: one agent has
// one interface, and claiming an address in a range another network
// already owns would spread the collision rather than contain it —
// "the lowest author's range wins" would carry it to everybody.
if let Some(wanted) = self.params.ipv4_range
self.params.ipv4_range
}
/// A late signed range can conflict even though activation was safe.
/// Fail closed before touching claims, plugins or the host interface.
fn suspend_if_overlapping(&self) -> bool {
if let Some(range) = self.effective_range()
&& self
.params
.routes
.would_overlap(self.network_id, wanted)
.is_none()
.would_overlap(self.network_id, range)
.is_some()
{
return Some(wanted);
self.range_blocked
.store(true, std::sync::atomic::Ordering::Release);
self.params.routes.remove_network(self.network_id);
self.shutdown.trigger();
self.params.range_changed.send_replace(());
return true;
}
// The configured range is another network's here, so this one falls
// back to the range derived from its id — after a moment's wait, so
// that a network which already exists gets to say what it uses
// first. Every member derives the same one, so members that reach
// this point agree without negotiating.
let fallback = self.params.ipv4_fallback?;
if self.activated.elapsed() < RANGE_PROPOSAL_GRACE {
return None;
}
self.params
.routes
.would_overlap(self.network_id, fallback)
.is_none()
.then_some(fallback)
false
}
/// Makes sure this agent holds an address, claiming one if it does not.
@@ -1006,7 +1107,7 @@ impl Runtime {
async fn ensure_own_claim(&mut self) {
// On the way out. Claiming again here is how a goodbye becomes a
// hello nobody asked for.
if self.released {
if self.released || self.suspend_if_overlapping() {
return;
}
let wanted_hostname = {
@@ -1296,6 +1397,9 @@ impl Runtime {
/// Tells the plugins who holds which overlay address.
async fn publish_allocations(&mut self) {
if self.shutdown.is_triggered() || self.suspend_if_overlapping() {
return;
}
let Some(range) = self.effective_range() else {
return;
};
@@ -1307,10 +1411,6 @@ impl Runtime {
.collect();
allocations.sort_by_key(|(holder, _)| *holder.as_bytes());
for plugin in &self.params.plugins {
plugin.on_address_allocation(self.network_id, range, &allocations);
}
// And the system level's own view, which is what decides whose
// packet is whose. The local address is kept out of the peer list:
// a packet for ourselves does not go over a tunnel.
@@ -1326,17 +1426,27 @@ impl Runtime {
.map(|(holder, address)| (*address, *holder))
.collect(),
};
if let Err(err) = self.params.routes.set_network(self.network_id, routes) {
// Reported once per change rather than swallowed: two networks
// wanting the same addresses is a thing the user has to settle.
self.metrics.plugin_errors += 1;
self.emit(Event::PluginError {
network: self.network_id,
protocol: "overlay".into(),
reason: err.to_string(),
});
if self
.params
.routes
.set_network(self.network_id, routes)
.is_err()
{
self.range_blocked
.store(true, std::sync::atomic::Ordering::Release);
self.params.routes.remove_network(self.network_id);
self.shutdown.trigger();
self.params.range_changed.send_replace(());
return;
}
for plugin in &self.params.plugins {
plugin.on_address_allocation(self.network_id, range, &allocations);
}
if self.published_range != Some(range) {
self.published_range = Some(range);
self.params.range_changed.send_replace(());
}
let Some(interface) = self.params.interface.clone() else {
return;
};
@@ -1429,6 +1539,7 @@ impl Runtime {
}
fn record_protocol_failure(&mut self, peer: EndpointId, protocol: &str) {
self.last_failed_protocol.insert(peer, protocol.to_string());
let key = (peer, protocol.to_string());
let entry = self
.protocol_dial_states
@@ -1467,7 +1578,17 @@ impl Runtime {
}
}
for (peer, protocol) in dead {
self.links.remove(&(peer, protocol.clone()));
let reason = self
.links
.remove(&(peer, protocol.clone()))
.map(|link| {
let reason = link
.failure_reason()
.unwrap_or_else(|| "link closed".into());
link.close();
reason
})
.unwrap_or_else(|| "link closed".into());
// Only the direct path is gone. Whether the peer is still
// reachable through somebody is decided below; the protocol's
// link stays either way.
@@ -1477,7 +1598,7 @@ impl Runtime {
network: self.network_id,
peer,
protocol,
reason: "link closed".into(),
reason,
});
}
@@ -1518,12 +1639,21 @@ impl Runtime {
continue;
}
let chosen = supported.iter().find(|proto| {
match self.protocol_dial_states.get(&(peer, (*proto).clone())) {
Some(state) => state.cooldown_until <= now,
None => true,
}
});
let start = next_protocol_index(
&supported,
self.last_failed_protocol.get(&peer).map(String::as_str),
);
let chosen = supported
.iter()
.cycle()
.skip(start)
.take(supported.len())
.find(
|proto| match self.protocol_dial_states.get(&(peer, (*proto).clone())) {
Some(state) => state.cooldown_until <= now,
None => true,
},
);
if let Some(proto) = chosen {
wanted.push((peer, proto.clone()));
@@ -1535,13 +1665,16 @@ impl Runtime {
if self.links.contains_key(&key) || self.opening.contains(&key) {
continue;
}
self.opening.insert(key);
self.opening.insert(key.clone());
let results = self.link_results_tx.clone();
let transport = Arc::clone(&transport);
let network = self.network_id;
let shutdown = self.shutdown.clone();
tokio::spawn(async move {
self.next_link_attempt = self.next_link_attempt.wrapping_add(1);
let generation = self.next_link_attempt;
self.link_attempts.insert(key.clone(), generation);
let task = tokio::spawn(async move {
let result = tokio::select! {
biased;
_ = shutdown.wait() => Err("network deactivated".to_string()),
@@ -1551,12 +1684,14 @@ impl Runtime {
};
let _ = results
.send(LinkOutcome {
generation,
peer,
protocol,
result,
})
.await;
});
self.link_tasks.insert(key, task.abort_handle());
}
}
@@ -1791,7 +1926,15 @@ impl Runtime {
fn handle_link_result(&mut self, outcome: LinkOutcome) {
let key = (outcome.peer, outcome.protocol.clone());
if self.link_attempts.get(&key) != Some(&outcome.generation) {
if let Ok(link) = outcome.result {
link.close();
}
return;
}
self.link_attempts.remove(&key);
self.opening.remove(&key);
self.link_tasks.remove(&key);
match outcome.result {
Ok(link) => {
self.record_protocol_success(outcome.peer, &outcome.protocol);
@@ -1837,7 +1980,11 @@ impl Runtime {
}
fn install_link(&mut self, inbound: InboundLink) {
if self.no_direct_data(inbound.peer) {
if self.no_direct_data(inbound.peer)
|| inbound.link.is_closed()
|| !self.sessions.contains_key(&inbound.peer)
{
inbound.link.close();
return;
}
self.record_protocol_success(inbound.peer, &inbound.protocol);
@@ -1848,19 +1995,64 @@ impl Runtime {
/// Hands a link to the plugin that owns its protocol.
fn adopt_link(&mut self, peer: EndpointId, protocol: String, link: SharedLink) {
if !self
.params
.plugins
.iter()
.any(|plugin| plugin.protocol_id() == protocol)
if link.is_closed()
|| link.network() != self.network_id
|| link.peer() != peer
|| !self.sessions.contains_key(&peer)
{
link.close();
return;
}
if !self.params.plugins.iter().any(|plugin| {
plugin.protocol_id() == protocol
&& self.sessions.get(&peer).is_some_and(|s| {
s.capabilities.iter().any(|c| {
c.enabled
&& c.protocol == protocol
&& c.version == plugin.protocol_version()
})
})
}) {
link.close();
return;
}
let key = (peer, protocol.clone());
let settings = if self.local_id.as_bytes() < peer.as_bytes() {
let settings = HealthSettings::fresh();
if let Ok(bytes) = postcard::to_stdvec(&(protocol.clone(), settings.clone())) {
let _ = self.send_to(
peer,
ControlMessage::Transport {
protocol: "link-health".into(),
payload: crate::proto::message::TransportPayload::new(bytes),
},
);
}
Some(settings)
} else {
self.pending_health.remove(&key)
};
let route = self
.sessions
.get(&peer)
.and_then(|s| self.tunnel_protocol(s))
.and_then(|name| self.params.plugins.iter().find(|p| p.protocol_id() == name))
.map(|plugin| ProbeRoute {
plugin: plugin.clone(),
routes: self.params.routes.clone(),
network: self.network_id,
peer,
max_probe_size: self.params.interface.as_ref().map_or(540, |interface| {
(interface.mtu() as usize).saturating_sub(36)
}),
});
let link = HealthLink::new(link, self.local_id, settings, route);
let path = link.path_description();
let max_datagram = link.max_datagram_size();
self.links
.insert((peer, protocol.clone()), Arc::clone(&link));
if let Some(old) = self.links.insert(key, Arc::clone(&link)) {
old.close();
}
// What this is, is a way to reach the peer. Which protocol's
// tunnel runs over it is decided in `update_paths`, and that tunnel
// is handed the hub's link, not this one: the same object for as
@@ -1879,26 +2071,47 @@ impl Runtime {
/// Drops every link to a peer.
fn drop_links_for(&mut self, peer: EndpointId) {
// The peer itself is going, not just a path to it, so the
// protocol's link goes with it.
self.reset_peer_data(peer, "peer session ended");
self.hub.remove_peer(peer);
self.reachable.remove(&peer);
self.protocol_dial_states.retain(|(p, _), _| *p != peer);
let keys: Vec<(EndpointId, String)> = self
}
/// Retire only the data plane; authenticated membership and topology remain.
fn reset_peer_data(&mut self, peer: EndpointId, reason: &str) {
self.link_attempts.retain(|(id, _), _| *id != peer);
self.opening.retain(|(id, _)| *id != peer);
self.link_tasks.retain(|(id, _), task| {
if *id == peer {
task.abort();
false
} else {
true
}
});
self.hub.clear_peer_links(peer);
let keys: Vec<_> = self
.links
.keys()
.filter(|(id, _)| *id == peer)
.cloned()
.collect();
for key in keys {
self.links.remove(&key);
if let Some(link) = self.links.remove(&key) {
link.close();
}
self.emit(Event::DataLinkDown {
network: self.network_id,
peer,
protocol: key.1,
reason: "peer session ended".into(),
reason: reason.into(),
});
}
self.protocol_dial_states.retain(|(id, _), _| *id != peer);
self.last_failed_protocol.remove(&peer);
self.pending_health.retain(|(id, _), _| *id != peer);
for plugin in &self.params.plugins {
plugin.on_peer_gone(self.network_id, peer);
}
}
// --------------------------------------------------------------- sessions
@@ -2091,9 +2304,6 @@ impl Runtime {
self.watch_exit_via();
self.update_paths();
self.announce_reach(false);
for plugin in &self.params.plugins {
plugin.on_peer_gone(self.network_id, peer);
}
// Retry promptly, then back off if it keeps failing.
let policy = &self.params.reconnect;
let state = self
@@ -2140,8 +2350,64 @@ impl Runtime {
fn dispatch_message(&mut self, peer: EndpointId, message: ControlMessage) {
match &message {
ControlMessage::Transport { protocol, payload } => {
if protocol == "link-health" {
// Only the elected initiator supplies a profile, for an enabled,
// mutually supported physical protocol in this authenticated session.
if peer.as_bytes() >= self.local_id.as_bytes() {
return;
}
let Ok((name, settings)) =
postcard::from_bytes::<(String, HealthSettings)>(payload.as_bytes())
else {
return;
};
if name.len() > crate::dataplane::MAX_PROTOCOL_ID_LEN
|| !self.served_protocols().iter().any(|(p, v)| {
p == &name
&& self.sessions.get(&peer).is_some_and(|s| {
s.capabilities
.iter()
.any(|c| c.enabled && c.protocol == name && c.version == *v)
})
})
{
return;
}
let key = (peer, name);
// Negotiation and inbound installation are separate queues.
// Keep the newest settings even if the previous physical
// link is still in the map when this control frame arrives.
if let Some(link) = self.links.get(&key)
&& !link.is_closed()
{
link.configure(settings.clone());
}
self.pending_health.insert(key, settings);
return;
}
if let Some(transport) = &self.params.transport
&& let Err(err) =
transport.on_control(self.network_id, peer, protocol, payload.as_bytes())
{
tracing::debug!(%err, "transport negotiation rejected");
}
return;
}
ControlMessage::Announce(announcement) => {
let capabilities = announcement.capabilities.clone();
let changed = self.sessions.get(&peer).is_some_and(|session| {
let selection = |caps: &[PluginCapability]| {
caps.iter()
.filter(|c| c.enabled)
.map(|c| (c.protocol.clone(), c.version))
.collect::<Vec<_>>()
};
selection(&session.capabilities) != selection(&capabilities)
});
if changed {
self.reset_peer_data(peer, "peer protocol selection changed");
}
if let Some(session) = self.sessions.get_mut(&peer) {
session.hostname = Some(announcement.hostname.clone());
session.broadcast = announcement.broadcast;
@@ -2154,6 +2420,7 @@ impl Runtime {
self.dispatch_capabilities(peer, &capabilities);
self.ensure_links();
self.update_paths();
self.announce_reach(false);
}
ControlMessage::Ping { seq, payload } => {
let pong = ControlMessage::Pong {
@@ -2387,6 +2654,13 @@ impl Runtime {
members.dedup_by(|a, b| a.endpoint_id == b.endpoint_id);
NetworkStatus {
protocols: self
.params
.plugins
.iter()
.map(|p| p.protocol_id().to_owned())
.collect(),
auto_start: true,
broadcast: self.params.broadcast,
exit_offer: self.params.exit_offer,
exit_via: self.params.exit_via,
@@ -2409,13 +2683,8 @@ impl Runtime {
members,
hostname_conflict: self.hostname_conflict_of(&self.local_id),
range: self.effective_range(),
// Only worth reporting while it is actually stuck: with a
// fallback in play the configured range being another
// network's is how it is meant to work, not a fault.
range_conflict: self
.params
.range_conflict
.filter(|_| self.effective_range().is_none()),
range_conflict: None,
range_conflicts: Vec::new(),
metrics: self.metrics.clone(),
relay: self.hub.counters(),
}
@@ -2662,3 +2931,17 @@ mod relay_host_tests {
assert_eq!(relay_host("https://bad host/").as_deref(), None);
}
}
#[cfg(test)]
mod fallback_order_tests {
use super::next_protocol_index;
#[test]
fn degradation_advances_the_cycle_even_after_earlier_cooldowns_expire() {
let protocols = ["wg", "awg", "wg-quic", "tcp-tls"].map(str::to_owned);
assert_eq!(next_protocol_index(&protocols, None), 0);
for (i, p) in protocols.iter().enumerate() {
assert_eq!(next_protocol_index(&protocols, Some(p)), (i + 1) % 4);
}
assert_eq!(next_protocol_index(&protocols, Some("removed")), 0);
}
}
+3 -2
View File
@@ -70,7 +70,7 @@ pub(crate) struct Session {
pub(crate) conn: Connection,
/// Already encoded frame payloads, so the runtime can account for the exact
/// number of control bytes it queues.
pub(crate) outbound: mpsc::Sender<Vec<u8>>,
pub(crate) outbound: mpsc::Sender<zeroize::Zeroizing<Vec<u8>>>,
pub(crate) hostname: Option<String>,
pub(crate) capabilities: Vec<PluginCapability>,
pub(crate) broadcast: bool,
@@ -244,7 +244,7 @@ pub(crate) fn spawn(
async fn writer_task(
mut send: SendStream,
mut outbound: mpsc::Receiver<Vec<u8>>,
mut outbound: mpsc::Receiver<zeroize::Zeroizing<Vec<u8>>>,
limits: Arc<Limits>,
shutdown: Shutdown,
parent: Shutdown,
@@ -331,6 +331,7 @@ async fn reader_task(
}
};
let payload = zeroize::Zeroizing::new(payload);
let bytes = payload.len();
let envelope: Envelope = match decode(&payload) {
Ok(envelope) => envelope,
+18 -6
View File
@@ -209,9 +209,24 @@ pub struct NetworkMetrics {
pub data_link_failures: u64,
}
/// An active network preventing another network from starting locally.
#[derive(Debug, Clone)]
pub struct RangeConflict {
/// The blocking network's public identifier.
pub network_id: NetworkId,
/// Its configured name.
pub name: NetworkName,
/// Its overlapping overlay range.
pub range: crate::state::Ipv4Range,
}
/// Status of one network.
#[derive(Debug, Clone)]
pub struct NetworkStatus {
/// Enabled local data-plane protocols, in preference order.
pub protocols: Vec<String>,
/// Whether this network is requested to start or resume automatically.
pub auto_start: bool,
/// Local broadcast participation in this network.
pub broadcast: bool,
/// This agent offers itself as an exit node in this network.
@@ -242,13 +257,10 @@ pub struct NetworkStatus {
pub hostname_conflict: Option<HostnameConflict>,
/// The overlay range this network uses, once it has one.
pub range: Option<crate::state::Ipv4Range>,
/// The range it could not have, because another network on this agent
/// already holds it.
///
/// One agent has one interface, so an address belongs to one network.
/// This network waits to adopt whatever its members settle on instead of
/// proposing something it could not route.
/// The first overlapping active range; `range_conflicts` identifies all blockers.
pub range_conflict: Option<crate::state::Ipv4Range>,
/// Active networks that would be suspended by switching to this network.
pub range_conflicts: Vec<RangeConflict>,
/// Per-network counters.
pub metrics: NetworkMetrics,
/// What has gone through a peer in the middle, in both directions.
+15 -2
View File
@@ -166,6 +166,8 @@ pub struct Limits {
pub max_capabilities: usize,
/// Largest opaque plugin payload, in bytes.
pub max_capability_data_len: usize,
/// Largest private transport negotiation payload (also hard-capped at 4096 bytes).
pub max_transport_payload_len: usize,
/// Largest accepted echo payload in a ping/pong exchange, in bytes.
pub max_echo_payload_len: usize,
/// Largest accepted free-text reason string, in bytes.
@@ -204,6 +206,7 @@ impl Default for Limits {
max_frame_len: 64 * 1024,
max_hostname_len: 255,
max_capabilities: 16,
max_transport_payload_len: 4096,
max_capability_data_len: 4 * 1024,
max_echo_payload_len: 4 * 1024,
max_reason_len: 256,
@@ -325,6 +328,9 @@ pub struct AgentConfig {
pub reconnect: ReconnectPolicy,
/// IP plugins whose capabilities are announced and dispatched.
pub plugins: Vec<SharedPlugin>,
/// Initial ordered selection for networks without a saved override.
/// `None` enables all registered plugins; an empty list is control-only.
pub default_protocols: Option<Vec<String>>,
/// Custom packet transports registered per protocol.
pub custom_transports:
std::collections::HashMap<String, Arc<dyn crate::dataplane::PacketTransport>>,
@@ -339,7 +345,11 @@ pub struct AgentConfig {
pub interface_name: String,
/// The interface MTU.
pub interface_mtu: u32,
/// The IPv4 overlay range this agent proposes.
/// Whether new networks derive their IPv4 range from their network id.
///
/// Enabled by default. Explicit ranges and IPv4 opt-out disable this.
pub overlay_ipv4_auto: bool,
/// The IPv4 overlay range this agent proposes when automatic selection is off.
///
/// Addresses are allocated from it and recorded in signed state, so a
/// participant keeps the same one across restarts. A network that has
@@ -369,11 +379,13 @@ impl AgentConfig {
limits: Limits::default(),
reconnect: ReconnectPolicy::default(),
plugins: Vec::new(),
default_protocols: None,
custom_transports: std::collections::HashMap::new(),
tun_factory: None,
interface_name: "tsun0".to_string(),
interface_mtu: 1280,
overlay_ipv4_range: Some(crate::state::DEFAULT_IPV4_RANGE),
overlay_ipv4_auto: true,
overlay_ipv4_range: None,
}
}
@@ -445,6 +457,7 @@ impl AgentConfig {
/// Sets the IPv4 overlay range this agent proposes, or disables IPv4.
pub fn with_overlay_ipv4_range(mut self, range: Option<crate::state::Ipv4Range>) -> Self {
self.overlay_ipv4_auto = false;
self.overlay_ipv4_range = range;
self
}
+27
View File
@@ -40,6 +40,33 @@ pub use tunnel::{GenericTunnelPlugin, TunnelCodec};
/// Maximum length of a plugin protocol identifier.
pub const MAX_PROTOCOL_ID_LEN: usize = 32;
/// Validates bounded local protocol preferences before storing or applying them.
pub(crate) fn validate_protocols(protocols: &[String], max: usize) -> crate::error::Result<()> {
if protocols.len() > max {
return Err(crate::Error::InvalidProtocols(format!(
"at most {max} protocols are allowed"
)));
}
for (index, name) in protocols.iter().enumerate() {
if name.is_empty()
|| name.len() > MAX_PROTOCOL_ID_LEN
|| !name
.bytes()
.all(|c| c.is_ascii_alphanumeric() || c == b'-' || c == b'_')
{
return Err(crate::Error::InvalidProtocols(
"names must be 1–32 ASCII letters, digits, hyphens or underscores".into(),
));
}
if protocols[..index].contains(name) {
return Err(crate::Error::InvalidProtocols(format!(
"`{name}` is listed more than once"
)));
}
}
Ok(())
}
/// An announcement of one IP plugin's capability.
///
/// `data` is opaque to the core. Nothing in it may be interpreted as a shell
+10 -2
View File
@@ -492,11 +492,12 @@ impl RelayHub {
}
}
/// Revokes all links and topology involving a departed authenticated peer.
pub fn remove_peer(&self, peer: EndpointId) {
/// Retires data channels without removing an authenticated peer's topology.
pub fn clear_peer_links(&self, peer: EndpointId) {
let mut state = self.state();
if let Some(links) = state.raw.remove(peer.as_bytes()) {
for raw in links {
raw.link.close();
raw.reader.abort();
}
}
@@ -511,6 +512,13 @@ impl RelayHub {
link.close();
}
}
state.publish();
}
/// Revokes all links and topology involving a departed authenticated peer.
pub fn remove_peer(&self, peer: EndpointId) {
self.clear_peer_links(peer);
let mut state = self.state();
state.members.remove(peer.as_bytes());
state.graph.remove(peer.as_bytes());
for neighbors in state.graph.values_mut() {
@@ -0,0 +1,350 @@
//! Authenticated probes of the actual data link, independent of control RTT.
//! Keys are negotiated privately for each physical link. No fixed wire magic.
use super::{PacketLink, SharedLink, TransportError};
use crate::dataplane::SharedPlugin;
use crate::overlay::RoutingTable;
use crate::{BoxFuture, NetworkId};
use bytes::Bytes;
use hmac::{Hmac, KeyInit, Mac};
use iroh::EndpointId;
use serde::{Deserialize, Serialize};
use sha2::Sha256;
use std::collections::VecDeque;
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};
use tokio::sync::{mpsc, watch};
#[derive(Clone, Serialize, Deserialize, zeroize::ZeroizeOnDrop)]
pub(crate) struct Settings {
marker: [u8; 16],
key: [u8; 32],
}
impl Settings {
pub(crate) fn fresh() -> Self {
Self {
marker: rand::random(),
key: rand::random(),
}
}
}
impl std::fmt::Debug for Settings {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str("HealthSettings([REDACTED])")
}
}
struct Samples {
pending: VecDeque<(u64, Instant)>,
recent: VecDeque<bool>,
}
impl Samples {
fn new() -> Self {
Self {
pending: VecDeque::new(),
recent: VecDeque::new(),
}
}
fn sent(&mut self, seq: u64, now: Instant) {
self.pending.push_back((seq, now));
}
fn acknowledge(&mut self, seq: u64) {
if let Some(i) = self.pending.iter().position(|(n, _)| *n == seq) {
self.pending.remove(i);
self.recent.push_back(true);
}
}
fn expire(&mut self, now: Instant) {
while self
.pending
.front()
.is_some_and(|(_, at)| now.duration_since(*at) >= Duration::from_secs(3))
{
self.pending.pop_front();
self.recent.push_back(false);
}
while self.recent.len() > 20 {
self.recent.pop_front();
}
}
fn degraded(&self) -> bool {
self.recent.len() >= 15
&& self.recent.iter().filter(|ok| !**ok).count() * 100 >= self.recent.len() * 60
}
}
// This prefix is inside end-to-end encryption. It never appears on the UDP wire.
const IP_MAGIC: &[u8; 16] = b"tsunagi-health01";
pub(crate) fn ip_payload(packet: &[u8]) -> Option<&[u8]> {
(packet.len() >= 36 && packet[0] == 0x45 && packet[9] == 253 && &packet[20..36] == IP_MAGIC)
.then(|| &packet[36..])
}
/// Inject probes through the same encrypted tunnel as application IP packets.
/// This matters for pure WG: raw UDP probes could pass DPI while WG is blocked.
pub(crate) struct ProbeRoute {
pub(crate) plugin: SharedPlugin,
pub(crate) routes: Arc<RoutingTable>,
pub(crate) network: NetworkId,
pub(crate) peer: EndpointId,
pub(crate) max_probe_size: usize,
}
impl ProbeRoute {
fn send(&self, payload: Bytes) -> bool {
let Some((source, destination)) = self.routes.probe_addresses(self.network, self.peer)
else {
return false;
};
let mut packet = vec![0; 36];
packet[0] = 0x45;
packet[8] = 64;
packet[9] = 253;
packet[12..16].copy_from_slice(&source.octets());
packet[16..20].copy_from_slice(&destination.octets());
packet[20..36].copy_from_slice(IP_MAGIC);
let length = (packet.len() + payload.len()) as u16;
packet[2..4].copy_from_slice(&length.to_be_bytes());
let sum = packet[..20]
.as_chunks::<2>()
.0
.iter()
.map(|b| u32::from(u16::from_be_bytes(*b)))
.sum::<u32>();
let sum = (sum & 0xffff) + (sum >> 16);
packet[10..12].copy_from_slice(&(!(sum as u16)).to_be_bytes());
packet.extend_from_slice(&payload);
self.plugin
.carry(self.network, self.peer, Bytes::from(packet))
}
}
/// A monitored physical link. It forwards application bytes unchanged.
pub(crate) struct HealthLink {
probes: mpsc::Sender<Bytes>,
raw: SharedLink,
settings: Mutex<Option<Settings>>,
rx: tokio::sync::Mutex<mpsc::Receiver<Bytes>>,
stop: watch::Sender<bool>,
reason: Mutex<Option<String>>,
}
impl std::fmt::Debug for HealthLink {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("HealthLink")
.field("raw", &self.raw)
.finish_non_exhaustive()
}
}
fn lock<T>(m: &Mutex<T>) -> std::sync::MutexGuard<'_, T> {
match m.lock() {
Ok(g) => g,
Err(p) => p.into_inner(),
}
}
impl HealthLink {
pub(crate) fn new(
raw: SharedLink,
local: EndpointId,
settings: Option<Settings>,
route: Option<ProbeRoute>,
) -> Arc<Self> {
let (tx, rx) = mpsc::channel(64);
let (probe_tx, mut probe_rx) = mpsc::channel(64);
let (stop, _) = watch::channel(false);
let link = Arc::new(Self {
probes: probe_tx,
raw: raw.clone(),
settings: Mutex::new(settings),
rx: tokio::sync::Mutex::new(rx),
stop,
reason: Mutex::new(None),
});
let weak = Arc::downgrade(&link);
let mut stopped = link.stop.subscribe();
tokio::spawn(async move {
let mut interval = tokio::time::interval(Duration::from_secs(1));
interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
let mut samples = Samples::new();
let mut sequence = rand::random::<u64>();
let mut since: Option<Instant> = None;
let mut marker = None;
loop {
tokio::select! {
biased;
_=stopped.changed()=>break,
_=interval.tick()=>{
let Some(link)=weak.upgrade()else{break;};let Some(settings)=lock(&link.settings).clone()else{continue;};
if marker != Some(settings.marker) { marker = Some(settings.marker); samples = Samples::new(); since = None; }
let start=*since.get_or_insert_with(Instant::now);let now=Instant::now();samples.expire(now);
if now.duration_since(start)>=Duration::from_secs(20) && samples.degraded() {
*lock(&link.reason)=Some("data delivery checks: at least 60% loss over the last 20 probes".into());break;
}
sequence=sequence.wrapping_add(1);
let size=route.as_ref().map_or(raw.max_datagram_size(),|route|route.max_probe_size);
let packet=probe(&settings,local,1,sequence,size.min(1000));
if let Some(route)=&route { if route.send(packet) {samples.sent(sequence,now);} }
else { let _=raw.send(packet); samples.sent(sequence,now); }
}
packet=raw.recv()=>{
let Some(packet)=packet else{break;};let Some(link)=weak.upgrade()else{break;};
let settings=lock(&link.settings).clone();
if route.is_none() && let Some(settings)=settings && packet.starts_with(&settings.marker) {
if let Some((kind,seq))=verify(&settings,raw.peer(),&packet) {
if kind==1 {let _=raw.send(probe(&settings,local,2,seq,packet.len()));}
else if kind==2 {samples.acknowledge(seq);}
}
} else {let _=tx.try_send(packet);}
}
packet=probe_rx.recv(),if route.is_some()=>{
let Some(packet)=packet else {break;};let Some(link)=weak.upgrade()else{break;};
let Some(settings)=lock(&link.settings).clone()else{continue;};
if let Some((kind,seq))=verify(&settings,raw.peer(),&packet) {
if kind==1 { if let Some(route)=&route { route.send(probe(&settings,local,2,seq,packet.len().min(route.max_probe_size))); } }
else if kind==2 {samples.acknowledge(seq);}
}
}
}
}
raw.close();
if let Some(link) = weak.upgrade() {
link.stop.send_replace(true);
}
});
link
}
pub(crate) fn receive_probe(&self, packet: Bytes) {
let _ = self.probes.try_send(packet);
}
pub(crate) fn configure(&self, settings: Settings) {
*lock(&self.settings) = Some(settings);
}
}
fn probe(settings: &Settings, sender: EndpointId, kind: u8, seq: u64, size: usize) -> Bytes {
let mut bytes = vec![0; size.max(57)];
bytes[..16].copy_from_slice(&settings.marker);
bytes[16] = kind;
bytes[17..25].copy_from_slice(&seq.to_be_bytes());
rand::fill(&mut bytes[25..size.max(57) - 32]);
let tail = bytes.len() - 32;
if let Ok(mut mac) = <Hmac<Sha256> as KeyInit>::new_from_slice(&settings.key) {
mac.update(sender.as_bytes());
mac.update(&bytes[..tail]);
bytes[tail..].copy_from_slice(&mac.finalize().into_bytes());
}
Bytes::from(bytes)
}
fn verify(settings: &Settings, sender: EndpointId, bytes: &[u8]) -> Option<(u8, u64)> {
if bytes.len() < 57 || !bytes.starts_with(&settings.marker) {
return None;
}
let mut mac = <Hmac<Sha256> as KeyInit>::new_from_slice(&settings.key).ok()?;
let tail = bytes.len() - 32;
mac.update(sender.as_bytes());
mac.update(&bytes[..tail]);
mac.verify_slice(&bytes[tail..]).ok()?;
Some((
bytes[16],
u64::from_be_bytes(bytes.get(17..25)?.try_into().ok()?),
))
}
impl PacketLink for HealthLink {
fn network(&self) -> NetworkId {
self.raw.network()
}
fn peer(&self) -> EndpointId {
self.raw.peer()
}
fn max_datagram_size(&self) -> usize {
self.raw.max_datagram_size()
}
fn send(&self, payload: Bytes) -> Result<(), TransportError> {
if self.is_closed() {
Err(TransportError::Closed)
} else {
self.raw.send(payload)
}
}
fn send_flow(
&self,
payload: Bytes,
flow: super::super::routing::FlowId,
) -> Result<(), TransportError> {
if self.is_closed() {
Err(TransportError::Closed)
} else {
self.raw.send_flow(payload, flow)
}
}
fn recv(&self) -> BoxFuture<'_, Option<Bytes>> {
Box::pin(async move {
let mut stop = self.stop.subscribe();
if *stop.borrow() {
return None;
}
let mut rx = self.rx.lock().await;
tokio::select! {biased;_=stop.changed()=>None,packet=rx.recv()=>packet}
})
}
fn closed(&self) -> BoxFuture<'_, ()> {
Box::pin(async move {
let mut stop = self.stop.subscribe();
if !*stop.borrow() {
let _ = stop.changed().await;
}
})
}
fn is_closed(&self) -> bool {
*self.stop.borrow() || self.raw.is_closed()
}
fn close(&self) {
self.stop.send_replace(true);
self.raw.close();
}
fn failure_reason(&self) -> Option<String> {
lock(&self.reason)
.clone()
.or_else(|| self.raw.failure_reason())
}
fn path_description(&self) -> String {
self.raw.path_description()
}
}
impl Drop for HealthLink {
fn drop(&mut self) {
self.stop.send_replace(true);
self.raw.close();
}
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
#[test]
fn no_false_failure_before_window_but_persistent_loss_is_detected() {
let mut samples = Samples::new();
let now = Instant::now();
for seq in 0..20 {
samples.sent(seq, now);
if seq % 5 == 0 {
samples.acknowledge(seq);
}
}
assert!(!samples.degraded());
samples.expire(now + Duration::from_secs(4));
assert!(samples.degraded());
for seq in 20..40 {
samples.sent(seq, now);
samples.acknowledge(seq);
}
samples.expire(now + Duration::from_secs(4));
assert!(!samples.degraded());
}
#[test]
fn a_forged_or_reflected_probe_cannot_claim_delivery() {
let a = crate::identity::DeviceIdentity::generate().endpoint_id();
let b = crate::identity::DeviceIdentity::generate().endpoint_id();
let settings = Settings::fresh();
let packet = probe(&settings, a, 1, 42, 1100);
assert_eq!(verify(&settings, a, &packet), Some((1, 42)));
assert!(verify(&settings, b, &packet).is_none());
let mut forged = packet.to_vec();
forged[16] = 2;
assert!(verify(&settings, a, &forged).is_none());
}
}
@@ -97,6 +97,9 @@ impl IrohLink {
}
impl PacketLink for IrohLink {
fn close(&self) {
self.conn.close(0u32.into(), b"data link retired");
}
fn network(&self) -> NetworkId {
self.network
}
@@ -30,12 +30,14 @@
//! one plugin protocol.
mod fragments;
pub(crate) mod health;
pub mod iroh_link;
use std::sync::Arc;
use bytes::Bytes;
use iroh::EndpointId;
use serde::{Deserialize, Serialize};
use crate::BoxFuture;
use crate::identity::NetworkId;
@@ -112,6 +114,14 @@ pub trait PacketLink: Send + Sync + std::fmt::Debug + 'static {
/// keeping a task parked on [`PacketLink::closed`].
fn is_closed(&self) -> bool;
/// Explicitly retires a physical link. Implementations should wake readers.
fn close(&self) {}
/// Observed reason for retiring this link, if available.
fn failure_reason(&self) -> Option<String> {
None
}
/// A short description of the path in use, for diagnostics.
///
/// Reports what the transport actually knows. It must not invent a value.
@@ -121,6 +131,76 @@ pub trait PacketLink: Send + Sync + std::fmt::Debug + 'static {
/// A shared handle to a link.
pub type SharedLink = std::sync::Arc<dyn PacketLink>;
/// Maximum opaque pair-negotiation payload.
pub const MAX_TRANSPORT_PAYLOAD: usize = 4096;
/// Private transport bytes, never included in Debug output.
#[derive(Clone, PartialEq, Eq, Serialize, zeroize::ZeroizeOnDrop)]
pub struct TransportPayload(Vec<u8>);
impl<'de> Deserialize<'de> for TransportPayload {
fn deserialize<D: serde::Deserializer<'de>>(
deserializer: D,
) -> std::result::Result<Self, D::Error> {
struct Visitor;
impl<'de> serde::de::Visitor<'de> for Visitor {
type Value = TransportPayload;
fn expecting(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "at most {MAX_TRANSPORT_PAYLOAD} private transport bytes")
}
fn visit_seq<A: serde::de::SeqAccess<'de>>(
self,
mut sequence: A,
) -> std::result::Result<Self::Value, A::Error> {
if sequence
.size_hint()
.is_some_and(|size| size > MAX_TRANSPORT_PAYLOAD)
{
return Err(serde::de::Error::custom("transport payload too large"));
}
let mut payload = TransportPayload(Vec::with_capacity(
sequence.size_hint().unwrap_or(0).min(MAX_TRANSPORT_PAYLOAD),
));
while let Some(byte) = sequence.next_element::<u8>()? {
if payload.0.len() == MAX_TRANSPORT_PAYLOAD {
return Err(serde::de::Error::custom("transport payload too large"));
}
payload.0.push(byte);
}
Ok(payload)
}
}
deserializer.deserialize_seq(Visitor)
}
}
impl TransportPayload {
/// Wraps bytes for one authenticated control session.
pub fn new(bytes: Vec<u8>) -> Self {
Self(bytes)
}
/// Borrows private bytes; callers must not log them.
pub fn as_bytes(&self) -> &[u8] {
&self.0
}
}
impl std::fmt::Debug for TransportPayload {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str("TransportPayload([REDACTED])")
}
}
/// Private control-plane sender for transports. Contains no network secret.
pub trait ControlSender: Send + Sync + std::fmt::Debug + 'static {
/// Queues a bounded message to one authenticated member of one network.
fn send<'a>(
&'a self,
network: NetworkId,
peer: EndpointId,
protocol: &'a str,
payload: TransportPayload,
) -> BoxFuture<'a, Result<(), TransportError>>;
}
/// An inbound link a peer opened towards us.
#[derive(Debug)]
pub struct InboundLink {
@@ -139,6 +219,18 @@ pub struct InboundLink {
/// The agent owns one of these and hands links to plugins; plugins never call
/// it directly.
pub trait PacketTransport: Send + Sync + std::fmt::Debug + 'static {
/// Supplies a private sender before network activation.
fn attach_control(&self, _control: Arc<dyn ControlSender>) {}
/// Validates and queues private negotiation received from an authenticated member.
fn on_control(
&self,
_network: NetworkId,
_peer: EndpointId,
_protocol: &str,
_payload: &[u8],
) -> Result<(), TransportError> {
Err(TransportError::Declined(self.name().into()))
}
/// A short name used in diagnostics.
fn name(&self) -> &str;
@@ -184,6 +276,24 @@ impl MultiPacketTransport {
}
impl PacketTransport for MultiPacketTransport {
fn attach_control(&self, control: Arc<dyn ControlSender>) {
self.default.attach_control(control.clone());
for transport in self.by_protocol.values() {
transport.attach_control(control.clone());
}
}
fn on_control(
&self,
network: NetworkId,
peer: EndpointId,
protocol: &str,
payload: &[u8],
) -> Result<(), TransportError> {
match self.by_protocol.get(protocol) {
Some(transport) => transport.on_control(network, peer, protocol, payload),
None => Err(TransportError::Declined(protocol.into())),
}
}
fn name(&self) -> &str {
"multi"
}
+16
View File
@@ -15,6 +15,9 @@ pub type Result<T, E = Error> = std::result::Result<T, E>;
#[derive(Debug, thiserror::Error)]
#[non_exhaustive]
pub enum Error {
/// An ordered local data-plane protocol selection is invalid.
#[error("invalid protocol selection: {0}")]
InvalidProtocols(String),
/// The supplied network name does not satisfy the documented rules.
#[error("invalid network name: {0}")]
InvalidNetworkName(&'static str),
@@ -97,6 +100,19 @@ pub enum Error {
#[error("network {0} is not configured")]
NetworkUnknown(NetworkId),
/// An overlay range overlaps networks already active on this agent.
#[error(
"network {network} cannot start: {range} overlaps active networks {conflicts:?}; switch to it to suspend those networks"
)]
NetworkRangeConflict {
/// Network that could not start.
network: NetworkId,
/// Its agreed or proposed range.
range: crate::state::Ipv4Range,
/// Active networks that must be suspended first.
conflicts: Vec<NetworkId>,
},
/// No session with that peer exists in the given network.
#[error("no authenticated session with peer {peer} in network {network}")]
NoSuchPeer {
+90 -2
View File
@@ -171,6 +171,13 @@ pub fn default_control_socket_path() -> PathBuf {
#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)]
#[non_exhaustive]
pub enum Request {
/// Replace one network's ordered local data-plane selection.
SetProtocols {
/// Public network identifier.
network_id: String,
/// Protocol names, best first; empty selects control only.
protocols: Vec<String>,
},
/// Report what the agent is doing.
Status,
/// Set local broadcast participation in one configured network.
@@ -255,6 +262,13 @@ pub enum Request {
impl std::fmt::Debug for Request {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
Request::SetProtocols {
network_id,
protocols,
} => write!(
f,
"SetProtocols {{ {network_id}, protocols: {protocols:?} }}"
),
Request::Status => f.write_str("Status"),
Request::SetBroadcast {
network_id,
@@ -286,6 +300,8 @@ impl std::fmt::Debug for Request {
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[non_exhaustive]
pub enum Response {
/// Accepted ordered protocol selection.
Protocols(Vec<String>),
/// A status report.
Status(Box<StatusReport>),
/// Accepted local broadcast participation.
@@ -324,6 +340,9 @@ pub struct ActiveReport {
/// Stopping something already stopped is not an error — it is the state
/// asked for — but it is worth saying which happened.
pub changed: bool,
/// Networks suspended by switching to this network.
#[serde(default)]
pub suspended: Vec<RangeConflictReport>,
}
/// What happened when a network was joined.
@@ -345,6 +364,13 @@ pub struct JoinedReport {
/// Almost always a mistyped secret, and the one thing that makes two
/// sections of a report look like one network.
pub name_shared_with: Option<String>,
/// Whether it started; false means it is configured but blocked by overlap.
#[serde(default = "joined_active_default")]
pub active: bool,
}
fn joined_active_default() -> bool {
true
}
/// What happened when a network was left.
@@ -367,6 +393,8 @@ pub struct LeftReport {
/// Everything the agent is doing, in one snapshot.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct StatusReport {
/// Data-plane protocols this agent can enable at runtime.
pub available_protocols: Vec<ProtocolReport>,
/// This device's persistent endpoint id.
pub endpoint_id: String,
/// Hostname announced to peers.
@@ -437,9 +465,34 @@ pub struct DnsReport {
pub names: u32,
}
/// An active network with an overlapping overlay range.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct RangeConflictReport {
/// Public network id, for unambiguous selection.
pub network_id: String,
/// Configured network name.
pub name: String,
/// The overlapping range.
pub range: String,
}
/// A data-plane protocol available in the running agent.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct ProtocolReport {
/// Stable protocol name used by the CLI and network negotiation.
pub name: String,
/// A short description of encryption and transport.
pub summary: String,
}
/// One network.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct NetworkReport {
/// Enabled local data-plane protocols, in preference order.
pub protocols: Vec<String>,
/// Whether the network starts or resumes automatically when its range is free.
#[serde(default)]
pub auto_start: bool,
/// Whether this agent participates in LAN broadcasts in this network.
#[serde(default)]
pub broadcast: bool,
@@ -482,6 +535,9 @@ pub struct NetworkReport {
/// The range it could not have, because another network on this agent
/// already holds it.
pub range_conflict: Option<String>,
/// Networks that switching to this network would suspend.
#[serde(default)]
pub range_conflicts: Vec<RangeConflictReport>,
/// The overlay, when a protocol is running one.
pub overlay: Option<OverlayReport>,
/// This agent's own name, when another member claimed it first.
@@ -693,6 +749,14 @@ impl OverlayPeerReport {
/// running and what they can report or change. That is what keeps this module
/// free of any knowledge of them.
pub trait ReportSource: Send + Sync + 'static {
/// Saves and applies a local ordered protocol set for one network.
fn set_protocols(
&self,
_network_id: String,
_protocols: Vec<String>,
) -> BoxFuture<'_, std::result::Result<Vec<String>, String>> {
Box::pin(async { Err("this agent cannot change protocols".into()) })
}
/// Produces a fresh report.
fn report(&self) -> BoxFuture<'_, StatusReport>;
@@ -822,7 +886,7 @@ pub const EXCHANGE_TIMEOUT: Duration = Duration::from_secs(5);
///
/// Bump it whenever [`Request`], [`Response`] or anything they contain
/// changes shape.
pub const CONTROL_PROTOCOL: u32 = u32::from_be_bytes([b'T', b'S', b'N', 18]);
pub const CONTROL_PROTOCOL: u32 = u32::from_be_bytes([b'T', b'S', b'N', 20]);
/// Reads one request off an accepted stream, answers it, writes the response.
///
@@ -846,6 +910,13 @@ where
}
};
let response = match request {
Request::SetProtocols {
network_id,
protocols,
} => match source.set_protocols(network_id, protocols).await {
Ok(protocols) => Response::Protocols(protocols),
Err(reason) => Response::Error(reason),
},
Request::Status => Response::Status(Box::new(source.report().await)),
Request::SetBroadcast {
network_id,
@@ -1001,7 +1072,24 @@ pub async fn join_network_with_broadcast(
}
}
/// Changes one network's broadcast participation while the agent runs.
/// Replaces the enabled data-plane protocols of one configured network.
pub async fn set_protocols(
socket: &Path,
network_id: &str,
protocols: Vec<String>,
) -> Result<Vec<String>> {
let request = Request::SetProtocols {
network_id: network_id.to_owned(),
protocols,
};
match exchange(socket, &request, EXCHANGE_TIMEOUT).await? {
Response::Protocols(protocols) => Ok(protocols),
Response::Error(reason) => Err(Error::Storage(reason)),
other => Err(Error::Storage(format!("unexpected response: {other:?}"))),
}
}
/// Changes local broadcast participation in a network.
pub async fn set_broadcast(
path: impl AsRef<Path>,
network_id: &str,
+39 -2
View File
@@ -72,7 +72,7 @@ pub enum RouteError {
#[error(
"network {network} uses {range}, which overlaps {other_range} already in use by \
{other}. One agent has one interface, so an address can belong to only one of \
them: give one network a different range."
them: stop the other network before starting this one."
)]
Overlap {
/// The network being added.
@@ -365,6 +365,43 @@ impl RoutingTable {
})
}
/// All active networks whose reserved ranges intersect this range.
pub fn overlapping_networks(
&self,
network: NetworkId,
range: Ipv4Range,
) -> Vec<(NetworkId, Ipv4Range)> {
let mut conflicts: Vec<_> = self
.read()
.iter()
.filter_map(|(other, routes)| {
let other_range = routes.range?;
(*other != network && ranges_overlap(range, other_range))
.then_some((*other, other_range))
})
.collect();
conflicts.sort_by_key(|(id, _)| *id);
conflicts
}
/// Current signed allocations used for encrypted data delivery checks.
pub(crate) fn probe_addresses(
&self,
network: NetworkId,
peer: EndpointId,
) -> Option<(Ipv4Addr, Ipv4Addr)> {
let table = self.read();
let routes = table.get(&network)?;
Some((
routes.local?,
routes
.peers
.iter()
.find(|(_, holder)| *holder == peer)
.map(|(address, _)| *address)?,
))
}
/// How many networks the table covers.
pub fn len(&self) -> usize {
self.read().len()
@@ -510,7 +547,7 @@ mod tests {
let err = table.set_network(second, routes()).unwrap_err();
let RouteError::Overlap { other, .. } = err;
assert_eq!(other, first);
assert!(err.to_string().contains("different range"), "{err}");
assert!(err.to_string().contains("stop the other network"), "{err}");
assert_eq!(table.len(), 1, "the overlapping network is not added");
// A range of its own is fine.
+23 -2
View File
@@ -20,7 +20,7 @@ use crate::error::ProtocolError;
/// The version in the ALPN is the wire-compatibility version of the control
/// protocol. It is independent of the network identity scheme version, so
/// bumping it must not change any existing [`crate::NetworkId`].
pub const ALPN: &[u8] = b"tsunagi/ctrl/5";
pub const ALPN: &[u8] = b"tsunagi/ctrl/6";
/// ALPN of the tsunagi data plane.
///
@@ -43,7 +43,9 @@ pub const MAX_DATA_PROTOCOL_LEN: usize = 32;
pub const MAX_SIGNATURE_LEN: usize = 64;
/// Control protocol version carried inside the handshake.
pub const PROTOCOL_VERSION: u16 = 5;
pub const PROTOCOL_VERSION: u16 = 6;
pub use crate::dataplane::transport::{MAX_TRANSPORT_PAYLOAD, TransportPayload};
/// First message of the handshake, sent by the initiator.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
@@ -114,6 +116,13 @@ pub struct DataOpenAck {
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[non_exhaustive]
pub enum ControlMessage {
/// Private transport negotiation; never persisted or rebroadcast.
Transport {
/// Transport protocol id; the payload stays opaque to the core.
protocol: String,
/// Secret negotiation data, redacted in diagnostics.
payload: TransportPayload,
},
/// Hostname and capability announcement.
Announce(Announcement),
/// A small request used to verify that the exchange works.
@@ -264,6 +273,17 @@ pub fn validate_capability(
/// network, the other networks or the agent.
pub fn validate(message: &ControlMessage, limits: &Limits) -> Result<(), ProtocolError> {
match message {
ControlMessage::Transport { protocol, payload } => {
if protocol.is_empty() {
return Err(ProtocolError::Malformed("empty transport protocol"));
}
check_len("transport.protocol", protocol.len(), MAX_PROTOCOL_ID_LEN)?;
check_len(
"transport.payload",
payload.as_bytes().len(),
limits.max_transport_payload_len.min(MAX_TRANSPORT_PAYLOAD),
)?;
}
ControlMessage::Peers { peers } => {
check_len("peers", peers.len(), limits.max_state_records)?;
for hint in peers {
@@ -320,6 +340,7 @@ pub fn validate(message: &ControlMessage, limits: &Limits) -> Result<(), Protoco
/// A short, stable label for a message kind, for metrics and diagnostics.
pub fn kind(message: &ControlMessage) -> &'static str {
match message {
ControlMessage::Transport { .. } => "transport",
ControlMessage::Announce(_) => "announce",
ControlMessage::Ping { .. } => "ping",
ControlMessage::Pong { .. } => "pong",
+50 -24
View File
@@ -109,12 +109,8 @@ impl std::str::FromStr for Ipv4Range {
}
}
/// The IPv4 overlay range used unless something else is configured or agreed.
///
/// A small, specific `/24`: memorable, and far less likely to overlap a
/// network the machine is already on than taking a whole `/8` or `/10` would
/// be. Because addresses are allocated rather than derived, 254 of them is
/// plenty for the size of network this is for.
/// The legacy default range, retained for existing networks and explicit use.
/// New networks use [`derived_ipv4_range`] by default.
pub const DEFAULT_IPV4_RANGE: Ipv4Range = Ipv4Range {
base: Ipv4Addr::new(10, 13, 37, 0),
prefix_len: 24,
@@ -122,15 +118,10 @@ pub const DEFAULT_IPV4_RANGE: Ipv4Range = Ipv4Range {
/// A range derived from a network's own identity.
///
/// One agent has one interface, so two of its networks cannot both use the
/// configured range. The second needs another one — and it cannot be picked
/// locally, because every member has to arrive at the same answer without
/// being told. So it comes from the network id, which every member already
/// has and nobody can choose: the same network derives the same range on
/// every device.
///
/// Inside 10/8 like the default, and never equal to it, so the first
/// network keeps what it has always had.
/// Every member independently derives the same pseudorandom `/24` in 10/8,
/// including isolated first members. Stored signed ranges take precedence.
/// The derivation is kept stable for networks already using it; the legacy
/// default is excluded. Distinct network ids can still produce collisions.
pub fn derived_ipv4_range(network: NetworkId) -> Ipv4Range {
let bytes = network.as_bytes();
let second = bytes[0];
@@ -673,15 +664,20 @@ impl StateSet {
/// The range the network settled on, if anybody has said.
///
/// When claims disagree, the one from the lowest author id wins, so every
/// replica reaches the same answer. A participant joining later therefore
/// adopts the range already in use instead of imposing its own.
/// A legacy or explicit range takes precedence over the derived default,
/// so a fresh member's default cannot renumber an established network.
/// Other disagreements use the lowest author id, independently of arrival
/// order. The signed record format and network identity remain unchanged.
pub fn agreed_range(&self) -> Option<Ipv4Range> {
let mut authors: Vec<&EndpointId> = self.records.keys().collect();
authors.sort_by_key(|author| *author.as_bytes());
authors
.into_iter()
.find_map(|author| self.records.get(author)?.body.range())
self.records
.iter()
.filter_map(|(author, record)| {
let range = record.body.range()?;
let derived = derived_ipv4_range(NetworkId::from_bytes(record.network));
Some(((range == derived, *author.as_bytes()), range))
})
.min_by_key(|(priority, _)| *priority)
.map(|(_, range)| range)
}
/// The highest version this author has published, as far as is known.
@@ -1263,7 +1259,7 @@ mod tests {
assert_eq!(once.base.octets()[0], 10, "inside 10/8 like the default");
assert_ne!(
once, DEFAULT_IPV4_RANGE,
"the default belongs to whichever network asked first"
"the legacy default remains excluded"
);
// A different network derives a different range, which is the
@@ -1275,4 +1271,34 @@ mod tests {
.network_id();
assert_ne!(once, derived_ipv4_range(other));
}
#[test]
fn a_derived_default_cannot_displace_a_legacy_or_explicit_range() {
let id = network("range-migration");
let a = SecretKey::generate();
let b = SecretKey::generate();
let (low, high) = if a.public().as_bytes() < b.public().as_bytes() {
(a, b)
} else {
(b, a)
};
for established in [DEFAULT_IPV4_RANGE, "10.90.0.0/16".parse().unwrap()] {
let body = |range: Ipv4Range| RecordBody::Claim {
address: Some(Ipv4Addr::from(u32::from(range.base) + 1)),
range: Some(range),
hostname: None,
hostname_since: None,
};
let fresh = SignedRecord::sign(&low, id, 1, body(derived_ipv4_range(id)));
let existing = SignedRecord::sign(&high, id, 1, body(established));
for records in [
vec![fresh.clone(), existing.clone()],
vec![existing.clone(), fresh.clone()],
] {
let mut state = StateSet::new();
state.merge_all(id, records);
assert_eq!(state.agreed_range(), Some(established));
}
}
}
}
+33
View File
@@ -254,6 +254,12 @@ impl Storage {
.await
}
/// Persists the ordered protocol selection off the async executor.
pub async fn set_protocols(&self, network_id: NetworkId, protocols: Vec<String>) -> Result<()> {
self.with_state(move |state| state.set_protocols(network_id, &protocols))
.await
}
/// Persists whether this agent offers itself as an exit node in a network.
pub async fn set_exit_node(&self, network_id: NetworkId, enabled: bool) -> Result<()> {
self.with_state(move |state| state.set_exit_node(network_id, enabled))
@@ -272,6 +278,33 @@ impl Storage {
.await
}
/// Configured networks in the user's most recently selected order.
pub async fn network_start_order(&self) -> Result<Vec<NetworkId>> {
self.with_state(|state| {
let configured = state.list_networks()?;
let text = state
.get_setting("network_start_order")?
.unwrap_or_default();
let mut order = Vec::new();
for word in text.split(',') {
if let Ok(id) = word.parse::<NetworkId>()
&& configured.iter().any(|n| n.network_id == id)
&& !order.contains(&id)
{
order.push(id);
}
}
Ok(order)
})
.await
}
/// Prefers this network at startup without disabling suspended networks.
pub async fn prefer_network(&self, network: NetworkId) -> Result<()> {
self.with_state(move |state| state.prefer_network(network))
.await
}
/// Removes a network configuration and its cached hints.
pub async fn remove_network(&self, network_id: NetworkId) -> Result<()> {
self.with_state(move |state| {
+158 -4
View File
@@ -21,7 +21,7 @@ use crate::identity::{DeviceIdentity, NetworkId, NetworkName, NetworkSecret};
use crate::state::{RecordBody, SignedRecord};
/// Schema version written by this build.
pub const SCHEMA_VERSION: i64 = 7;
pub const SCHEMA_VERSION: i64 = 8;
/// Key of the stored hostname setting.
const SETTING_HOSTNAME: &str = "hostname";
@@ -32,6 +32,8 @@ const SETTING_HOSTNAME: &str = "hostname";
/// and zeroizes on drop.
#[derive(Debug, Clone)]
pub struct StoredNetwork {
/// Ordered local protocol override; absent inherits the agent's defaults.
pub protocols: Option<Vec<String>>,
/// Derived public network identifier.
pub network_id: NetworkId,
/// Network name.
@@ -238,6 +240,16 @@ impl StateStore {
)
.map_err(|err| self.corrupt(format!("cannot migrate schema to 7: {err}")))?;
}
if found < 8 {
self.conn
.execute_batch(
"BEGIN;
ALTER TABLE networks ADD COLUMN protocols TEXT;
PRAGMA user_version = 8;
COMMIT;",
)
.map_err(|err| self.corrupt(format!("cannot migrate schema to 8: {err}")))?;
}
Ok(())
}
@@ -448,12 +460,16 @@ impl StateStore {
/// Sets the auto-start flag of a configured network.
pub fn set_auto_start(&self, network_id: NetworkId, auto_start: bool) -> Result<()> {
self.conn
let changed = self
.conn
.execute(
"UPDATE networks SET auto_start = ?2 WHERE network_id = ?1",
params![network_id.as_bytes().as_slice(), auto_start as i64],
)
.map_err(|err| Error::Storage(format!("cannot update network: {err}")))?;
if changed == 0 {
return Err(Error::NetworkUnknown(network_id));
}
Ok(())
}
@@ -472,6 +488,25 @@ impl StateStore {
Ok(())
}
/// Saves a local ordered protocol selection without changing membership.
pub fn set_protocols(&self, network_id: NetworkId, protocols: &[String]) -> Result<()> {
crate::dataplane::validate_protocols(
protocols,
crate::config::Limits::default().max_capabilities,
)?;
let changed = self
.conn
.execute(
"UPDATE networks SET protocols = ?2 WHERE network_id = ?1",
params![network_id.as_bytes().as_slice(), protocols.join(",")],
)
.map_err(|err| Error::Storage(format!("cannot update protocols: {err}")))?;
if changed == 0 {
return Err(Error::NetworkUnknown(network_id));
}
Ok(())
}
/// Removes a network configuration entirely.
pub fn remove_network(&self, network_id: NetworkId) -> Result<()> {
self.conn
@@ -532,7 +567,7 @@ impl StateStore {
pub fn list_networks(&self) -> Result<Vec<StoredNetwork>> {
let mut stmt = self
.conn
.prepare("SELECT network_id, name, secret, auto_start, broadcast, exit_node, exit_via FROM networks ORDER BY name")
.prepare("SELECT network_id, name, secret, auto_start, broadcast, exit_node, exit_via, protocols FROM networks ORDER BY name")
.map_err(|err| Error::Storage(format!("cannot list networks: {err}")))?;
let rows = stmt
.query_map([], |row| {
@@ -548,19 +583,37 @@ impl StateStore {
row.get::<_, bool>(4)?,
row.get::<_, bool>(5)?,
row.get::<_, Option<Vec<u8>>>(6)?,
row.get::<_, Option<String>>(7)?,
))
})
.map_err(|err| Error::Storage(format!("cannot list networks: {err}")))?;
let mut out = Vec::new();
for row in rows {
let (id, name, secret, auto_start, broadcast, exit_node, exit_via) =
let (id, name, secret, auto_start, broadcast, exit_node, exit_via, protocols) =
row.map_err(|err| Error::Storage(format!("cannot read network row: {err}")))?;
let protocols = protocols
.map(|text| {
let max = crate::config::Limits::default().max_capabilities;
if text.len() > max * (crate::dataplane::MAX_PROTOCOL_ID_LEN + 1) {
return Err(self.corrupt("stored protocol selection is too long"));
}
let names = if text.is_empty() {
Vec::new()
} else {
text.split(',').map(str::to_owned).collect()
};
crate::dataplane::validate_protocols(&names, max)
.map_err(|err| self.corrupt(err))?;
Ok(names)
})
.transpose()?;
let id: [u8; 32] = id
.as_slice()
.try_into()
.map_err(|_| self.corrupt("stored network id is not 32 bytes"))?;
out.push(StoredNetwork {
protocols,
network_id: NetworkId::from_bytes(id),
name: NetworkName::new(name)?,
secret: NetworkSecret::from_bytes(secret)?,
@@ -798,6 +851,44 @@ impl StateStore {
Ok(())
}
/// Atomically enables a network and puts it first in the startup order.
pub fn prefer_network(&self, network: NetworkId) -> Result<()> {
let configured = self.list_networks()?;
if !configured.iter().any(|n| n.network_id == network) {
return Err(Error::NetworkUnknown(network));
}
let existing = self.get_setting("network_start_order")?.unwrap_or_default();
let mut order = vec![network];
for word in existing.split(',') {
if let Ok(id) = word.parse::<NetworkId>()
&& configured.iter().any(|n| n.network_id == id)
&& !order.contains(&id)
{
order.push(id);
}
}
let value = order
.iter()
.map(ToString::to_string)
.collect::<Vec<_>>()
.join(",");
let transaction = self
.conn
.unchecked_transaction()
.map_err(|err| Error::Storage(format!("cannot begin network preference: {err}")))?;
transaction
.execute(
"UPDATE networks SET auto_start = 1 WHERE network_id = ?1",
params![network.as_bytes().as_slice()],
)
.map_err(|err| Error::Storage(format!("cannot enable network: {err}")))?;
transaction.execute("INSERT INTO settings (key, value) VALUES ('network_start_order', ?1) ON CONFLICT(key) DO UPDATE SET value = excluded.value", params![value])
.map_err(|err| Error::Storage(format!("cannot store network preference: {err}")))?;
transaction
.commit()
.map_err(|err| Error::Storage(format!("cannot commit network preference: {err}")))
}
/// Reads an arbitrary setting.
pub fn get_setting(&self, key: &str) -> Result<Option<String>> {
self.conn
@@ -915,6 +1006,68 @@ mod broadcast_storage_tests {
assert!(store.peer_hostnames(id).unwrap().is_empty());
}
#[test]
fn v7_migration_keeps_signed_state_and_protocol_overrides_are_local_and_durable() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("state.sqlite");
let store = StateStore::open(&path).unwrap();
let identity = store.load_or_create_device_identity().unwrap();
let name = NetworkName::new("protocol-migration").unwrap();
let secret = NetworkSecret::from_bytes([4; 32]).unwrap();
let id = NetworkKeys::derive(&name, &secret).network_id();
store.upsert_network(id, &name, &secret, true).unwrap();
store.set_broadcast(id, false).unwrap();
let record = SignedRecord::sign(&identity.signing_key(), id, 17, RecordBody::Release);
store.publish_own_record(&record).unwrap();
store
.conn
.execute_batch("ALTER TABLE networks DROP COLUMN protocols; PRAGMA user_version=7;")
.unwrap();
drop(store);
let store = StateStore::open(&path).unwrap();
assert_eq!(
store
.load_or_create_device_identity()
.unwrap()
.endpoint_id(),
identity.endpoint_id()
);
assert_eq!(store.signed_records(id).unwrap(), vec![record]);
let saved = store.list_networks().unwrap();
assert!(saved[0].protocols.is_none());
assert!(!saved[0].broadcast);
assert!(saved[0].auto_start);
assert_eq!(saved[0].secret, secret);
let chosen = vec!["awg".into(), "wg-quic".into()];
store.set_protocols(id, &chosen).unwrap();
assert!(
store
.set_protocols(id, &["awg".into(), "awg".into()])
.is_err()
);
assert!(
store
.set_protocols(NetworkId::from_bytes([1; 32]), &[])
.is_err()
);
drop(store);
let store = StateStore::open(&path).unwrap();
assert_eq!(store.list_networks().unwrap()[0].protocols, Some(chosen));
store.upsert_network(id, &name, &secret, true).unwrap();
store.set_protocols(id, &[]).unwrap();
drop(store);
let store = StateStore::open(&path).unwrap();
assert_eq!(store.list_networks().unwrap()[0].protocols, Some(vec![]));
store
.conn
.execute("UPDATE networks SET protocols = 'awg,awg'", [])
.unwrap();
assert!(matches!(
store.list_networks(),
Err(Error::StateCorrupted { .. })
));
}
#[test]
fn v3_migration_preserves_identity_and_networks_and_defaults_broadcast_on() {
let dir = tempfile::tempdir().unwrap();
@@ -935,6 +1088,7 @@ mod broadcast_storage_tests {
"ALTER TABLE networks DROP COLUMN broadcast; \
ALTER TABLE networks DROP COLUMN exit_node; \
ALTER TABLE networks DROP COLUMN exit_via; \
ALTER TABLE networks DROP COLUMN protocols; \
PRAGMA user_version=3;",
)
.unwrap();
+1 -1
View File
@@ -162,7 +162,7 @@ async fn rejoining_after_leaving_is_not_taken_for_a_stale_record() {
})
.await;
assert!(
tsunagi::state::DEFAULT_IPV4_RANGE.contains(again),
tsunagi::state::derived_ipv4_range(network_id).contains(again),
"an address in the network's range: {again}"
);
@@ -265,10 +265,7 @@ async fn deactivating_one_network_leaves_the_agent_and_others_running() {
#[tokio::test]
async fn two_networks_on_one_agent_each_get_a_range_of_their_own() {
// One agent, one interface: the configured range can only belong to one
// of them. The second does not go without — it takes the range derived
// from its own id, which every one of its members derives identically,
// so it is an agreement and not a local invention.
// Each network derives its range independently of activation order.
let discovery = SharedMemoryDiscovery::new();
let (first, first_secret) = network("two-ranges-one");
let (second, second_secret) = network("two-ranges-two");
@@ -309,8 +306,8 @@ async fn two_networks_on_one_agent_each_get_a_range_of_their_own() {
"different networks, different addresses"
);
assert!(
tsunagi::state::DEFAULT_IPV4_RANGE.contains(addresses.0),
"the first keeps the configured range: {}",
tsunagi::state::derived_ipv4_range(one).contains(addresses.0),
"the first uses its derived range: {}",
addresses.0
);
assert!(
@@ -319,7 +316,7 @@ async fn two_networks_on_one_agent_each_get_a_range_of_their_own() {
addresses.1
);
assert!(
!tsunagi::state::DEFAULT_IPV4_RANGE.contains(addresses.1),
!tsunagi::state::derived_ipv4_range(one).contains(addresses.1),
"and not in the one the first network holds"
);
@@ -0,0 +1,336 @@
//! Derived defaults and local exclusion of overlapping overlay networks.
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use tsunagi::Agent;
use tsunagi::discovery::SharedMemoryDiscovery;
use tsunagi::state::{DEFAULT_IPV4_RANGE, Ipv4Range, derived_ipv4_range};
use tsunagi::testing::{TestAgent, config_with, network, wait_until};
// These tests need discovery to happen before a collision, so keep the initial
// default outside the established ranges. Derived collisions have their own test.
fn network_outside(
name: &str,
ranges: &[Ipv4Range],
) -> (
tsunagi::identity::NetworkName,
tsunagi::identity::NetworkSecret,
) {
use tsunagi::identity::{NetworkKeys, NetworkName, NetworkSecret};
let name = NetworkName::new(name).unwrap();
let secret = (0..=255u8)
.map(|seed| NetworkSecret::from_bytes([seed; 32]).unwrap())
.find(|secret| {
let derived = derived_ipv4_range(NetworkKeys::derive(&name, secret).network_id());
ranges.iter().all(|range| !range.contains(derived.base))
})
.expect("a deterministic test secret with a non-overlapping default");
(name, secret)
}
#[tokio::test]
async fn colliding_derived_defaults_are_blocked_instead_of_locally_renumbered() {
use std::collections::HashMap;
use tsunagi::identity::{NetworkKeys, NetworkName, NetworkSecret};
let secret = NetworkSecret::from_bytes([19; 32]).unwrap();
let mut seen = HashMap::new();
let (first, second, range) = (0..=65_536)
.find_map(|index| {
let name = NetworkName::new(format!("derived-collision-{index}")).unwrap();
let range = derived_ipv4_range(NetworkKeys::derive(&name, &secret).network_id());
seen.insert(range.base, name.clone())
.map(|first| (first, name, range))
})
.expect("more network names than possible /24 ranges");
let agent = TestAgent::spawn(&SharedMemoryDiscovery::new())
.await
.unwrap();
let first = agent.agent.join_network(&first, &secret).await.unwrap();
let second = agent.agent.join_network(&second, &secret).await.unwrap();
assert!(agent.agent.is_active(first).await);
assert!(!agent.agent.is_active(second).await);
let status = agent.agent.status().await.unwrap();
let blocked = status.network(&second).unwrap();
assert_eq!(blocked.range, Some(range));
assert_eq!(blocked.range_conflicts[0].network_id, first);
assert_eq!(
agent.agent.switch_network(second).await.unwrap(),
vec![first]
);
assert_eq!(
agent.agent.network_status(second).await.unwrap().range,
Some(range)
);
agent.agent.shutdown().await;
}
#[tokio::test]
async fn isolated_first_members_choose_the_same_range_and_keep_it_after_restart() {
let discovery = SharedMemoryDiscovery::new();
let (name, secret) = network("derived-first");
let a = TestAgent::spawn(&discovery).await.unwrap();
let b = TestAgent::spawn(&SharedMemoryDiscovery::new())
.await
.unwrap();
let id = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
let expected = derived_ipv4_range(id);
for agent in [&a.agent, &b.agent] {
wait_until("isolated member allocates in its derived range", || async {
let status = agent.network_status(id).await.ok()?;
(status.range == Some(expected) && !status.members.is_empty()).then_some(())
})
.await;
}
let dir = a.stop().await;
let a = Agent::spawn(config_with(dir.path(), &discovery))
.await
.unwrap();
assert_eq!(a.network_status(id).await.unwrap().range, Some(expected));
a.shutdown().await;
b.agent.shutdown().await;
}
#[tokio::test]
async fn overlap_blocks_activation_switches_and_resumes_across_restart() {
let discovery = SharedMemoryDiscovery::new();
let a = TestAgent::spawn_with(
|c| c.with_overlay_ipv4_range(Some(DEFAULT_IPV4_RANGE)),
&discovery,
)
.await
.unwrap();
let (one_name, one_secret) = network("conflict-one");
let (two_name, two_secret) = network("conflict-two");
let one = a.agent.join_network(&one_name, &one_secret).await.unwrap();
a.agent.network_status(one).await.unwrap();
let two = a.agent.join_network(&two_name, &two_secret).await.unwrap();
assert!(a.agent.is_active(one).await);
assert!(!a.agent.is_active(two).await);
assert!(matches!(
a.agent.activate_network(two).await,
Err(tsunagi::Error::NetworkRangeConflict { .. })
));
let status = a.agent.status().await.unwrap();
let blocked = status.network(&two).unwrap();
assert_eq!(blocked.range, Some(DEFAULT_IPV4_RANGE));
assert_eq!(blocked.range_conflicts[0].network_id, one);
assert_eq!(blocked.range_conflicts[0].name, one_name);
assert_eq!(a.agent.switch_network(two).await.unwrap(), vec![one]);
assert!(a.agent.is_active(two).await);
assert!(!a.agent.is_active(one).await);
a.agent.network_status(two).await.unwrap();
let dir = a.stop().await;
let a = Agent::spawn(config_with(dir.path(), &discovery))
.await
.unwrap();
assert!(a.is_active(two).await, "the user's choice survives restart");
assert!(!a.is_active(one).await);
a.deactivate_network(two).await.unwrap();
assert!(
a.is_active(one).await,
"suspended networks resume when the blocker stops"
);
assert_eq!(
a.network_status(one).await.unwrap().range,
Some(DEFAULT_IPV4_RANGE),
"legacy ranges survive the new default"
);
a.shutdown().await;
}
#[tokio::test]
async fn late_authenticated_state_with_a_nested_range_suspends_only_that_network() {
let discovery = SharedMemoryDiscovery::new();
let broad: Ipv4Range = "10.90.0.0/16".parse().unwrap();
let narrow: Ipv4Range = "10.90.12.0/24".parse().unwrap();
let (name, secret) = network_outside("late-conflict", &[broad]);
let remote = TestAgent::spawn_with(|c| c.with_overlay_ipv4_range(Some(narrow)), &discovery)
.await
.unwrap();
let target = remote.agent.join_network(&name, &secret).await.unwrap();
wait_until("remote range is persisted", || async {
remote
.agent
.network_status(target)
.await
.ok()?
.members
.first()
.map(|_| ())
})
.await;
let (other_name, other_secret) = network("existing-broad");
// Seed an existing network using a real agent and its signed claim.
let seeded = TestAgent::spawn_with(
|c| c.with_overlay_ipv4_range(Some(broad)),
&SharedMemoryDiscovery::new(),
)
.await
.unwrap();
let other = seeded
.agent
.join_network(&other_name, &other_secret)
.await
.unwrap();
wait_until("broad range is persisted", || async {
seeded
.agent
.network_status(other)
.await
.ok()?
.members
.first()
.map(|_| ())
})
.await;
let dir = seeded.stop().await;
let seeded = Agent::spawn(config_with(dir.path(), &discovery))
.await
.unwrap();
seeded.join_network(&name, &secret).await.unwrap();
wait_until("late overlap suspends the joining network", || async {
let status = seeded.status().await.ok()?;
let blocked = status.network(&target)?;
(!seeded.is_active(target).await
&& blocked
.range_conflicts
.iter()
.any(|c| c.network_id == other))
.then_some(())
})
.await;
assert!(seeded.is_active(other).await);
assert_eq!(seeded.switch_network(target).await.unwrap(), vec![other]);
assert_eq!(
seeded.network_status(target).await.unwrap().range,
Some(narrow)
);
seeded.shutdown().await;
remote.agent.shutdown().await;
}
#[tokio::test]
async fn a_wide_range_suspends_all_overlaps_and_manual_stop_cancels_resume() {
let discovery = SharedMemoryDiscovery::new();
let hub = TestAgent::spawn(&discovery).await.unwrap();
let mut members = Vec::new();
let mut ids = Vec::new();
for (name, range) in [
("narrow-one", "10.81.0.0/24"),
("narrow-two", "10.81.1.0/24"),
] {
let peer = TestAgent::spawn_with(
|c| c.with_overlay_ipv4_range(Some(range.parse().unwrap())),
&discovery,
)
.await
.unwrap();
let (name, secret) = network_outside(
name,
&[
"10.81.0.0/24".parse().unwrap(),
"10.81.1.0/24".parse().unwrap(),
],
);
let id = peer.agent.join_network(&name, &secret).await.unwrap();
peer.agent.network_status(id).await.unwrap();
hub.agent.join_network(&name, &secret).await.unwrap();
wait_until("hub adopts the established narrow range", || async {
(hub.agent.network_status(id).await.ok()?.range == Some(range.parse().unwrap()))
.then_some(())
})
.await;
members.push(peer);
ids.push(id);
}
let (name, secret) = network_outside(
"wide-overlap",
&[
"10.81.0.0/24".parse().unwrap(),
"10.81.1.0/24".parse().unwrap(),
],
);
let wide = TestAgent::spawn_with(
|c| c.with_overlay_ipv4_range(Some("10.0.0.0/8".parse().unwrap())),
&discovery,
)
.await
.unwrap();
let id = wide.agent.join_network(&name, &secret).await.unwrap();
wide.agent.network_status(id).await.unwrap();
hub.agent.join_network(&name, &secret).await.unwrap();
wait_until("wide range is blocked by both narrow networks", || async {
let status = hub.agent.status().await.ok()?;
let blocked = status.network(&id)?;
(!hub.agent.is_active(id).await && blocked.range_conflicts.len() == 2).then_some(())
})
.await;
let stopped = hub.agent.switch_network(id).await.unwrap();
assert_eq!(stopped.len(), 2);
assert!(ids.iter().all(|id| stopped.contains(id)));
assert_eq!(hub.agent.switch_network(ids[0]).await.unwrap(), vec![id]);
assert!(
hub.agent.is_active(ids[1]).await,
"switching away from the wide range frees the other narrow range too"
);
assert_eq!(hub.agent.switch_network(id).await.unwrap().len(), 2);
hub.agent.pause_network(ids[0]).await.unwrap();
hub.agent.deactivate_network(id).await.unwrap();
assert!(
!hub.agent.is_active(ids[0]).await,
"explicitly stopped networks do not resume"
);
assert!(hub.agent.is_active(ids[1]).await);
assert!(
hub.agent
.switch_network(tsunagi::NetworkId::from_bytes([0; 32]))
.await
.is_err()
);
assert!(
hub.agent.is_active(ids[1]).await,
"unknown selectors do not stop any network"
);
hub.agent.shutdown().await;
wide.agent.shutdown().await;
for peer in members {
peer.agent.shutdown().await;
}
}
#[tokio::test]
async fn simultaneous_switches_never_leave_overlapping_networks_active() {
let discovery = SharedMemoryDiscovery::new();
let agent = TestAgent::spawn_with(
|c| c.with_overlay_ipv4_range(Some(DEFAULT_IPV4_RANGE)),
&discovery,
)
.await
.unwrap();
let (a, a_secret) = network("switch-a");
let (b, b_secret) = network("switch-b");
let a = agent.agent.join_network(&a, &a_secret).await.unwrap();
agent.agent.network_status(a).await.unwrap();
let b = agent.agent.join_network(&b, &b_secret).await.unwrap();
let (left, right) = tokio::join!(agent.agent.switch_network(b), agent.agent.switch_network(a));
left.unwrap();
right.unwrap();
assert_ne!(
agent.agent.is_active(a).await,
agent.agent.is_active(b).await
);
assert_eq!(
agent
.agent
.list_networks()
.await
.unwrap()
.iter()
.filter(|n| n.active)
.count(),
1
);
agent.agent.shutdown().await;
}
@@ -0,0 +1,130 @@
//! Opaque transport negotiation stays inside authenticated network sessions.
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use std::sync::{Arc, Mutex};
use tsunagi::dataplane::transport::{ControlSender, PacketTransport, SharedLink, TransportError};
use tsunagi::discovery::SharedMemoryDiscovery;
use tsunagi::proto::message::{ControlMessage, TransportPayload};
use tsunagi::testing::{TestAgent, network, wait_for_peers, wait_until};
use tsunagi::{BoxFuture, NetworkId};
#[derive(Debug, Default)]
struct Negotiator {
control: Mutex<Option<Arc<dyn ControlSender>>>,
received: Mutex<Vec<(NetworkId, iroh::EndpointId, Vec<u8>)>>,
}
impl PacketTransport for Negotiator {
fn name(&self) -> &str {
"negotiator"
}
fn as_any(&self) -> &dyn std::any::Any {
self
}
fn attach_control(&self, control: Arc<dyn ControlSender>) {
*self.control.lock().unwrap() = Some(control);
}
fn on_control(
&self,
network: NetworkId,
peer: iroh::EndpointId,
_: &str,
payload: &[u8],
) -> Result<(), TransportError> {
self.received
.lock()
.unwrap()
.push((network, peer, payload.to_vec()));
Ok(())
}
fn open<'a>(
&'a self,
_: NetworkId,
_: iroh::EndpointId,
_: &'a str,
) -> BoxFuture<'a, Result<SharedLink, TransportError>> {
Box::pin(async { Err(TransportError::Closed) })
}
}
#[tokio::test]
async fn negotiated_secrets_are_redacted_scoped_and_delivered_over_real_control() {
let discovery = SharedMemoryDiscovery::new();
let a_transport = Arc::new(Negotiator::default());
let b_transport = Arc::new(Negotiator::default());
let a = TestAgent::spawn_with(
|c| c.with_custom_transport("negotiator", a_transport.clone()),
&discovery,
)
.await
.unwrap();
let b = TestAgent::spawn_with(
|c| c.with_custom_transport("negotiator", b_transport.clone()),
&discovery,
)
.await
.unwrap();
let (name, secret) = network("private-negotiation");
let id = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
wait_for_peers(&a.agent, id, 1).await;
let message = ControlMessage::Transport {
protocol: "negotiator".into(),
payload: TransportPayload::new(b"never-export-this-secret".to_vec()),
};
assert!(!format!("{message:?}").contains("never-export"));
let sender = a_transport.control.lock().unwrap().clone().unwrap();
sender
.send(
id,
b.agent.endpoint_id(),
"negotiator",
TransportPayload::new(b"never-export-this-secret".to_vec()),
)
.await
.unwrap();
wait_until("private transport message", || async {
(!b_transport.received.lock().unwrap().is_empty()).then_some(())
})
.await;
assert_eq!(
b_transport.received.lock().unwrap()[0],
(
id,
a.agent.endpoint_id(),
b"never-export-this-secret".to_vec()
)
);
let unknown = NetworkId::from_bytes([0xa5; 32]);
assert!(
sender
.send(
unknown,
b.agent.endpoint_id(),
"negotiator",
TransportPayload::new(vec![1])
)
.await
.is_err()
);
a.agent.shutdown().await;
b.agent.shutdown().await;
}
#[test]
fn private_bytes_are_bounded_on_decode_and_by_local_policy() {
let limits = tsunagi::Limits {
max_transport_payload_len: 4,
..Default::default()
};
let message = |len| ControlMessage::Transport {
protocol: "awg".into(),
payload: TransportPayload::new(vec![42; len]),
};
assert!(tsunagi::proto::message::validate(&message(4), &limits).is_ok());
assert!(tsunagi::proto::message::validate(&message(5), &limits).is_err());
let bytes = postcard::to_stdvec(&TransportPayload::new(vec![1; 4097])).unwrap();
assert!(postcard::from_bytes::<TransportPayload>(&bytes).is_err());
let bytes = postcard::to_stdvec(&message(4)).unwrap();
let decoded: ControlMessage = postcard::from_bytes(&bytes).unwrap();
assert_eq!(decoded, message(4));
}
+16 -2
View File
@@ -115,6 +115,7 @@ pub struct TcpTlsLink {
peer: EndpointId,
tx: mpsc::Sender<Bytes>,
rx: tokio::sync::Mutex<mpsc::Receiver<Bytes>>,
tasks: [tokio::task::AbortHandle; 2],
closed_notify: Arc<Notify>,
is_closed: Arc<AtomicBool>,
remote_addr: String,
@@ -136,7 +137,7 @@ impl TcpTlsLink {
let closed_flag_w = Arc::clone(&is_closed);
let closed_notify_w = Arc::clone(&closed_notify);
tokio::spawn(async move {
let writer_task = tokio::spawn(async move {
while let Some(msg) = rx_out.recv().await {
if msg.len() > MAX_FRAME_SIZE {
continue;
@@ -158,7 +159,7 @@ impl TcpTlsLink {
let closed_flag_r = Arc::clone(&is_closed);
let closed_notify_r = Arc::clone(&closed_notify);
tokio::spawn(async move {
let reader_task = tokio::spawn(async move {
let mut len_buf = [0u8; 2];
loop {
if reader.read_exact(&mut len_buf).await.is_err() {
@@ -185,6 +186,7 @@ impl TcpTlsLink {
peer,
tx: tx_out,
rx: tokio::sync::Mutex::new(rx_in),
tasks: [writer_task.abort_handle(), reader_task.abort_handle()],
closed_notify,
is_closed,
remote_addr,
@@ -192,7 +194,19 @@ impl TcpTlsLink {
}
}
impl Drop for TcpTlsLink {
fn drop(&mut self) {
self.close();
}
}
impl PacketLink for TcpTlsLink {
fn close(&self) {
self.is_closed.store(true, Ordering::Release);
self.closed_notify.notify_waiters();
for task in &self.tasks {
task.abort();
}
}
fn network(&self) -> NetworkId {
self.network
}
+56 -4
View File
@@ -21,6 +21,11 @@ const POLL_INTERVAL: Duration = Duration::from_secs(2);
/// A request from the UI to the agent.
pub(crate) enum Command {
/// Replace the network's enabled protocols in preference order.
SetProtocols {
network_id: String,
protocols: Vec<String>,
},
/// Stop serving a network, or start serving it again.
SetActive { network_id: String, active: bool },
/// Turn local broadcast participation on or off for a network.
@@ -106,7 +111,13 @@ impl AgentClient {
/// Sends a command; a dropped worker simply means nothing happens.
pub(crate) fn send(&self, command: Command) {
let _ = self.tx.send(command);
// Disable protocol editing until the refreshed snapshot arrives;
// otherwise a second checkbox click could replace the first edit
// using the same stale network selection.
lock(&self.shared).busy = true;
if self.tx.send(command).is_err() {
lock(&self.shared).busy = false;
}
}
/// A copy of the latest snapshot.
@@ -142,6 +153,8 @@ async fn worker(
let Some(command) = command else { return };
run(&resolve_socket(), command, &shared, &pending_copy, &ctx).await;
refresh(&shared, &ctx).await;
lock(&shared).busy = false;
ctx.request_repaint();
}
}
}
@@ -179,11 +192,41 @@ async fn run(
ctx.request_repaint();
let outcome: Result<String, String> = match command {
Command::SetProtocols {
network_id,
protocols,
} => ipc::set_protocols(socket, &network_id, protocols)
.await
.map(|names| {
format!(
"protocols: {}",
if names.is_empty() {
"none".into()
} else {
names.join(", ")
}
)
})
.map_err(|err| err.to_string()),
Command::SetActive { network_id, active } => ipc::set_active(socket, &network_id, active)
.await
.map(|report| {
let state = if report.active { "started" } else { "stopped" };
format!("{} {}", state, report.name)
if report.suspended.is_empty() {
format!("{} {}", state, report.name)
} else {
format!(
"{} {}; suspended {} until it stops",
state,
report.name,
report
.suspended
.iter()
.map(|n| format!("{} ({})", n.name, n.network_id))
.collect::<Vec<_>>()
.join(", ")
)
}
})
.map_err(|err| err.to_string()),
Command::SetBroadcast {
@@ -211,7 +254,17 @@ async fn run(
}
Command::Join { name, secret } => ipc::join_network(socket, &name, &secret)
.await
.map(|report| format!("joined {}", report.name))
.map(|report| {
format!(
"{} {}",
if report.active {
"joined"
} else {
"configured; stopped by range conflict:"
},
report.name
)
})
.map_err(|err| err.to_string()),
Command::Leave { network_id } => ipc::leave_network(socket, &network_id)
.await
@@ -236,7 +289,6 @@ async fn run(
{
let mut guard = lock(shared);
guard.busy = false;
guard.last_action = if matches!(&outcome, Ok(message) if message.is_empty()) {
None
} else {
+153 -39
View File
@@ -42,7 +42,8 @@ struct Device<'a> {
copy: Option<String>,
/// The full endpoint id, which clicking the short form copies.
id: &'a str,
proto: &'a str,
/// End-to-end tunnel protocol, independent of the path carrying it.
tunnel_protocol: &'a str,
address: Option<String>,
handshake_secs: Option<u64>,
overlay: Option<&'a OverlayPeerReport>,
@@ -360,7 +361,7 @@ fn devices<'a>(network: &'a NetworkReport, own_id: &str) -> Vec<Device<'a>> {
hostname,
copy: hostname.map(|h| format!("{h}.{}", network.name)),
id,
proto: overlay.map_or("—", |p| p.protocol.as_str()),
tunnel_protocol: overlay.map_or("—", |p| p.protocol.as_str()),
address: overlay
.and_then(|p| p.address.clone())
.or_else(|| member.and_then(|m| m.overlay_address_v4.clone())),
@@ -389,9 +390,9 @@ fn devices<'a>(network: &'a NetworkReport, own_id: &str) -> Vec<Device<'a>> {
const COLUMN_SAMPLES: [&str; 9] = [
"",
"exit",
"tcp-tls",
"wg-quic · waiting for handshake",
"255.255.255.255",
"23h 59m",
"direct over tcp-tls",
"99999/99999",
"1023.9 MB / 1023.9 MB",
"1023.9 kB/s",
@@ -506,13 +507,36 @@ fn device_table(
.spacing([10.0, 4.0])
.show(ui, |ui| {
for (column, header) in [
"device", "exit", "proto", "address", "link", "pkts", "bytes", "rate", "plot",
"device",
"exit",
"end-to-end",
"address",
"data path",
"pkts",
"bytes",
"rate",
"plot",
]
.into_iter()
.enumerate()
{
cell(ui, column, &mut widths, |ui| {
ui.weak(header);
let response = ui.weak(header);
match column {
2 => {
response.on_hover_text(
"Protocol that encrypts traffic between this device and the \
destination. It is independent of the route carrying it.",
);
}
4 => {
response.on_hover_text(
"Current route carrying the encrypted tunnel traffic. `direct \
over tcp-tls` means one Tsunagi hop with no mesh relay.",
);
}
_ => {}
}
});
}
ui.end_row();
@@ -523,7 +547,7 @@ fn device_table(
exit_cell(ui, device, network_id, confirm);
});
cell(ui, 2, &mut widths, |ui| {
ui.label(device.proto);
tunnel_cell(ui, device);
});
cell(ui, 3, &mut widths, |ui| match &device.address {
Some(address) => format::copy_field(ui, address, address),
@@ -531,7 +555,7 @@ fn device_table(
ui.weak("—");
}
});
cell(ui, 4, &mut widths, |ui| link_cell(ui, device));
cell(ui, 4, &mut widths, |ui| data_path_cell(ui, device));
match device.overlay.filter(|_| device.online) {
Some(peer) => {
@@ -609,45 +633,95 @@ fn device_table(
ui.memory_mut(|m| m.data.insert_temp(widths_id, widths));
}
/// How a device is reached, and how long its tunnel has been up.
/// A protocol name as it should appear in the table.
fn displayed_protocol(protocol: &str) -> &str {
if protocol.is_empty() {
"wg-quic"
} else {
protocol
}
}
/// The end-to-end tunnel and its own health, without transport details.
fn tunnel_text(protocol: &str, handshake_secs: Option<u64>) -> String {
let protocol = displayed_protocol(protocol);
if protocol == "—" {
return protocol.to_string();
}
match handshake_secs {
Some(secs) if protocol == "tcp-tls" => {
format!("{protocol} · up {}", format::age(secs))
}
Some(secs) => format!("{protocol} · handshake {} ago", format::age(secs)),
None => format!("{protocol} · waiting for handshake"),
}
}
fn tunnel_cell(ui: &mut egui::Ui, device: &Device<'_>) {
let protocol = displayed_protocol(device.tunnel_protocol);
let text = tunnel_text(protocol, device.handshake_secs);
if protocol == "—" {
ui.weak(text);
return;
}
let hint = match device.path {
Some(DataPathReport::Direct { transport }) => format!(
"The end-to-end {protocol} tunnel is currently carried over a direct {transport} \
transport."
),
Some(DataPathReport::Relayed { hops, .. }) => format!(
"The end-to-end {protocol} tunnel is currently carried through other devices over \
{hops} transport hops."
),
Some(DataPathReport::None) | None => {
format!("The end-to-end {protocol} tunnel does not have a data path carrying it yet.")
}
};
ui.label(text).on_hover_text(hint);
}
/// Current route text and whether that route is the preferred direct form.
fn data_path_text(
path: Option<&DataPathReport>,
via_name: Option<&str>,
online: bool,
) -> (String, bool) {
if !online {
return ("offline".into(), false);
}
match path {
Some(DataPathReport::Direct { transport }) => (format!("direct over {transport}"), true),
Some(DataPathReport::Relayed { hops, .. }) => (
format!("via {} · {hops} hops", via_name.unwrap_or("another device")),
false,
),
Some(DataPathReport::None) | None => ("no data path yet".into(), false),
}
}
/// How a device is reached right now.
///
/// A device reached through others is online, over a longer path, and says
/// so: the colour is the only difference in how it is treated.
fn link_cell(ui: &mut egui::Ui, device: &Device<'_>) {
if !device.online {
ui.label("offline");
return;
}
let (path, good) = match device.path {
Some(DataPathReport::Direct { transport }) => (format!("direct · {transport}"), true),
Some(DataPathReport::Relayed { hops, .. }) => (
format!(
"via {} · {hops} hops",
device.via_name.as_deref().unwrap_or("another device")
),
false,
),
Some(DataPathReport::None) | None => ("no data path".to_string(), false),
};
let age = device.handshake_secs.map(format::age);
let text = match age {
Some(age) => format!("{path} · {age}"),
None => path,
};
fn data_path_cell(ui: &mut egui::Ui, device: &Device<'_>) {
let (text, good) = data_path_text(device.path, device.via_name.as_deref(), device.online);
let response = if good {
ui.label(text)
} else {
ui.label(egui::RichText::new(text).color(AMBER))
};
if !good {
response.on_hover_text(match device.path {
Some(DataPathReport::Relayed { .. }) => {
"no direct link yet: traffic goes through another device, and moves to a \
direct link by itself as soon as one comes up"
}
_ => "connected, but no path carries packets yet",
});
}
response.on_hover_text(match device.path {
Some(DataPathReport::Direct { .. }) => {
"One Tsunagi hop with no mesh relay. The end-to-end tunnel may use a different \
protocol from this transport."
}
Some(DataPathReport::Relayed { .. }) => {
"No direct transport yet: traffic goes through another device, and moves to a \
direct path by itself as soon as one comes up."
}
_ if device.online => "Connected, but no data path carries packets yet.",
_ => "This device is not reachable now.",
});
}
fn yes_no(value: bool) -> &'static str {
@@ -726,4 +800,44 @@ mod tests {
.is_none()
);
}
#[test]
fn tunnel_and_direct_transport_are_named_as_different_layers() {
assert_eq!(tunnel_text("wg", Some(1)), "wg · handshake 1s ago");
assert_eq!(
data_path_text(
Some(&DataPathReport::Direct {
transport: "tcp-tls".into(),
}),
None,
true,
),
("direct over tcp-tls".into(), true)
);
assert_eq!(tunnel_text("tcp-tls", Some(42)), "tcp-tls · up 42s");
}
#[test]
fn incomplete_and_longer_paths_stay_explicit() {
assert_eq!(
tunnel_text("wg-quic", None),
"wg-quic · waiting for handshake"
);
assert_eq!(
data_path_text(
Some(&DataPathReport::Relayed {
hops: 2,
via: "near".into(),
}),
Some("iris"),
true,
),
("via iris · 2 hops".into(), false)
);
assert_eq!(
data_path_text(Some(&DataPathReport::None), None, true),
("no data path yet".into(), false)
);
assert_eq!(data_path_text(None, None, false), ("offline".into(), false));
}
}
+1
View File
@@ -78,6 +78,7 @@ fn colour_of(health: Health) -> [u8; 3] {
match health {
Health::Disconnected => DARK_RED,
Health::Connected => BLUE,
Health::RangeConflict => [0xd9, 0x91, 0x20],
Health::ExitNode => GREEN,
}
}
+106 -10
View File
@@ -123,7 +123,7 @@ pub(crate) fn draw(
ui.weak("No networks yet — join one below.");
}
for network in &report.networks {
draw_network(ui, agent, state, traffic, network, &report.endpoint_id);
draw_network(ui, agent, state, traffic, network, report);
}
});
}
@@ -202,7 +202,7 @@ fn draw_network(
state: &mut UiState,
traffic: &Traffic,
network: &NetworkReport,
own_id: &str,
report: &StatusReport,
) {
ui.add_space(6.0);
egui::Frame::group(ui.style()).show(ui, |ui| {
@@ -244,21 +244,40 @@ fn draw_network(
ui.weak("peers");
ui.label(format!(
"{} online · {} known",
devices::online_count(network, own_id),
devices::known_count(network, own_id)
devices::online_count(network, &report.endpoint_id),
devices::known_count(network, &report.endpoint_id)
));
ui.end_row();
});
draw_protocols(ui, agent, network, &report.available_protocols);
format::sparkline(ui, series, state.unit, 34.0);
if !network.range_conflicts.is_empty() {
ui.colored_label(BAD, "Stopped: overlapping network ranges");
for conflict in &network.range_conflicts {
ui.label(format!("{} ({}) uses {}", conflict.name, conflict.network_id, conflict.range));
}
ui.label("Switching suspends these networks until this network stops, then resumes them automatically.");
if network.auto_start { ui.weak("This network will resume automatically when its range is free."); }
}
ui.horizontal(|ui| {
let mut active = network.active;
if ui.checkbox(&mut active, "active").changed() {
agent.send(Command::SetActive {
network_id: network.network_id.clone(),
active,
});
if !network.active && !network.range_conflicts.is_empty() {
if ui.button("Switch to this network").clicked() {
agent.send(Command::SetActive { network_id: network.network_id.clone(), active: true });
}
if network.auto_start && ui.button("Keep stopped").clicked() {
agent.send(Command::SetActive { network_id: network.network_id.clone(), active: false });
}
} else {
let mut active = network.active;
if ui.checkbox(&mut active, "active").changed() {
agent.send(Command::SetActive {
network_id: network.network_id.clone(),
active,
});
}
}
let mut broadcast = network.broadcast;
if ui.checkbox(&mut broadcast, "broadcast").changed() {
@@ -328,6 +347,83 @@ fn draw_network(
});
}
fn draw_protocols(
ui: &mut egui::Ui,
agent: &AgentClient,
network: &NetworkReport,
available: &[tsunagi::ipc::ProtocolReport],
) {
let summary = if network.protocols.is_empty() {
"none".into()
} else {
network.protocols.join(", ")
};
egui::CollapsingHeader::new(format!("protocols {summary}"))
.id_salt(("protocols", &network.network_id))
.show(ui, |ui| {
ui.weak("Preference order; changes apply immediately and are saved.");
let mut selected = network.protocols.clone();
let mut changed = false;
ui.add_enabled_ui(!agent.snapshot().busy, |ui| {
// Enabled items follow the user's preference; unchecked items
// follow the agent's catalog. The worker refreshes after edits.
let ordered: Vec<_> = network
.protocols
.iter()
.filter_map(|name| available.iter().find(|p| &p.name == name))
.chain(
available
.iter()
.filter(|p| !network.protocols.contains(&p.name)),
)
.collect();
for protocol in ordered {
ui.horizontal(|ui| {
let mut on = selected.contains(&protocol.name);
if ui.checkbox(&mut on, &protocol.name).changed() {
if on {
selected.push(protocol.name.clone());
} else {
selected.retain(|p| p != &protocol.name);
}
changed = true;
}
if let Some(index) = selected.iter().position(|p| p == &protocol.name) {
if ui
.add_enabled(index > 0, egui::Button::new("↑"))
.on_hover_text("prefer earlier")
.clicked()
{
selected.swap(index, index - 1);
changed = true;
}
if ui
.add_enabled(index + 1 < selected.len(), egui::Button::new("↓"))
.on_hover_text("prefer later")
.clicked()
{
selected.swap(index, index + 1);
changed = true;
}
}
});
ui.indent(("description", &network.network_id, &protocol.name), |ui| {
ui.weak(&protocol.summary);
});
}
});
if selected.is_empty() {
ui.weak("Control only; no IP traffic in this network.");
}
if changed {
agent.send(Command::SetProtocols {
network_id: network.network_id.clone(),
protocols: selected,
});
}
});
}
/// A prominent line about an exit node. One that carries a command copies it
/// when clicked.
fn draw_exit_warning(ui: &mut egui::Ui, warning: &exit::Warning) {
+28
View File
@@ -22,6 +22,8 @@ pub(crate) enum Health {
Disconnected,
/// Connected, nothing out of the ordinary.
Connected,
/// At least one configured network is blocked by an overlapping active range.
RangeConflict,
/// This device's traffic leaves through an exit node, and that works.
ExitNode,
}
@@ -33,6 +35,13 @@ impl Health {
let Some(report) = report else {
return Self::Disconnected;
};
if report
.networks
.iter()
.any(|network| network.auto_start && !network.range_conflicts.is_empty())
{
return Self::RangeConflict;
}
if !report
.networks
.iter()
@@ -56,6 +65,7 @@ impl Health {
match self {
Self::Disconnected => "tsunagi: not connected",
Self::Connected => "tsunagi: connected",
Self::RangeConflict => "tsunagi: network range conflict; open to switch networks",
Self::ExitNode => "tsunagi: using an exit node",
}
}
@@ -108,6 +118,24 @@ mod tests {
}
}
#[test]
fn overlapping_networks_are_visible_even_while_other_peers_are_connected() {
let mut blocked = NetworkReport {
auto_start: true,
..Default::default()
};
blocked
.range_conflicts
.push(tsunagi::ipc::RangeConflictReport {
name: "other".into(),
network_id: "other-id".into(),
range: "10.90.0.0/16".into(),
});
let status = report(vec![network(2, ExitReport::default()), blocked]);
assert_eq!(Health::of(Some(&status)), Health::RangeConflict);
assert!(Health::RangeConflict.label().contains("switch networks"));
}
#[test]
fn no_agent_or_nobody_connected_is_the_red_state() {
assert_eq!(Health::of(None), Health::Disconnected);
+6 -3
View File
@@ -79,6 +79,8 @@ pub const WIREGUARD_OVERHEAD: u32 = 32;
/// Configuration of the WireGuard plugin.
#[derive(Debug, Clone)]
pub struct WireguardConfig {
/// Diagnostic protocol id when this end-to-end engine is reused by another carrier.
pub protocol: &'static str,
/// Directory for the plugin's own key store. Separate from agent state.
pub state_dir: PathBuf,
/// WireGuard keepalive, which keeps tunnels and their links warm.
@@ -100,6 +102,7 @@ impl WireguardConfig {
/// Creates a configuration rooted at `state_dir` with sensible defaults.
pub fn new(state_dir: impl Into<PathBuf>) -> Self {
Self {
protocol: WIREGUARD_PROTOCOL,
state_dir: state_dir.into(),
keepalive: Some(25),
mtu: DEFAULT_MTU,
@@ -394,7 +397,7 @@ impl Worker {
fn report(&self, network: NetworkId, reason: impl std::fmt::Display) {
tracing::warn!(network = %network.fmt_short(), %reason, "wireguard plugin error");
if let Some(context) = self.context.get() {
context.report_error(network, WIREGUARD_PROTOCOL, reason.to_string());
context.report_error(network, self.config.protocol, reason.to_string());
}
}
@@ -647,7 +650,7 @@ async fn run(worker: Arc<Worker>, mut commands: mpsc::Receiver<Command>) {
impl IpPlugin for WireguardPlugin {
fn protocol_id(&self) -> &str {
WIREGUARD_PROTOCOL
self.worker.config.protocol
}
fn protocol_version(&self) -> u16 {
@@ -686,7 +689,7 @@ impl IpPlugin for WireguardPlugin {
// Identity only. Where to send packets is the transport's business.
let announcement = WgAnnouncement::new(network, &state.key.public());
Ok(Some(PluginCapability {
protocol: WIREGUARD_PROTOCOL.to_string(),
protocol: self.worker.config.protocol.to_string(),
version: crate::announcement::ANNOUNCEMENT_VERSION,
enabled: true,
data: announcement.encode()?,
@@ -277,6 +277,7 @@ impl tsunagi::ipc::ReportSource for Control {
network_id: network_id.to_string(),
already_configured: already,
name_shared_with: None,
active: self.0.is_active(network_id).await,
})
})
}
+29 -3
View File
@@ -109,6 +109,30 @@ impl WgAgent {
.await
}
async fn spawn_auto(discovery: &SharedMemoryDiscovery, tag: &str) -> Self {
let dir = TempDir::new().unwrap();
let tuns = MemoryTunFactory::new();
let plugin = WireguardPlugin::open(
WireguardConfig::new(dir.path().join("wireguard"))
.with_reconcile(Duration::from_millis(20), Duration::from_millis(250)),
)
.await
.unwrap();
let agent = Agent::spawn(
config_with(dir.path(), discovery)
.with_interface(Arc::new(tuns.clone()), tag, 1280)
.with_plugin(plugin.clone() as Arc<dyn IpPlugin>),
)
.await
.unwrap();
Self {
dir,
agent,
plugin,
tuns,
}
}
/// Starts an agent with an explicit overlay range, or none at all.
async fn spawn_range(
discovery: &SharedMemoryDiscovery,
@@ -989,7 +1013,7 @@ async fn two_networks_share_one_interface_with_keys_and_ranges_of_their_own() {
let (name_b, secret_b) = network("wg-right");
let beta_range = Some("10.99.0.0/16".parse::<Ipv4Range>().unwrap());
let hub = WgAgent::spawn(&discovery, "th").await;
let hub = WgAgent::spawn_auto(&discovery, "th").await;
let left = WgAgent::spawn(&discovery, "tl").await;
let right = WgAgent::spawn_range(&discovery, "tr", beta_range).await;
@@ -1661,8 +1685,10 @@ async fn lan_broadcast_fanout_is_scoped_opt_in_and_does_not_reflood() {
bytes[28..].copy_from_slice(b"LAN GAME");
// UDP checksum zero is valid for IPv4.
let sum: u32 = bytes[..20]
.chunks_exact(2)
.map(|w| u16::from_be_bytes([w[0], w[1]]) as u32)
.as_chunks::<2>()
.0
.iter()
.map(|w| u16::from_be_bytes(*w) as u32)
.sum();
let checksum = !((sum & 0xffff) + (sum >> 16)) as u16;
bytes[10..12].copy_from_slice(&checksum.to_be_bytes());
@@ -231,6 +231,9 @@ impl WireguardLink {
}
impl PacketLink for WireguardLink {
fn close(&self) {
WireguardLink::close(self);
}
fn network(&self) -> NetworkId {
self.network
}
+1
View File
@@ -14,6 +14,7 @@ path = "src/main.rs"
[dependencies]
tsunagi-core = { path = "../tsunagi-core", version = "0.1.0", features = ["tun-device", "dns-publish"] }
tsunagi-wireguard = { path = "../tsunagi-wireguard", version = "0.1.0" }
tsunagi-awg = { path = "../tsunagi-awg", version = "0.1.0" }
tsunagi-wg-quic = { path = "../tsunagi-wg-quic", version = "0.1.0" }
tsunagi-tcp-tls = { path = "../tsunagi-tcp-tls", version = "0.1.0" }
iroh.workspace = true
+542 -139
View File
File diff suppressed because it is too large. Load diff
+316 -7
View File
@@ -6,7 +6,7 @@
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use std::time::Duration;
use std::time::{Duration, Instant};
use tempfile::TempDir;
@@ -50,8 +50,17 @@ fn read_networks(dir: &TempDir) -> Vec<tsunagi::storage::StoredNetwork> {
/// An agent with no network at all, the way a daemon is started before
/// anything has been decided.
fn start_bare(port: u16) -> Running {
start_bare_range(port, None)
}
fn start_bare_range(port: u16, range: Option<&str>) -> Running {
start_bare_config(port, range, None)
}
fn start_bare_config(port: u16, range: Option<&str>, protocols: Option<&str>) -> Running {
let dir = TempDir::new().unwrap();
let child = std::process::Command::new(env!("CARGO_BIN_EXE_tsng"))
let mut command = std::process::Command::new(env!("CARGO_BIN_EXE_tsng"));
command
.arg("up")
.arg("--state-dir")
.arg(dir.path().join("state"))
@@ -62,11 +71,34 @@ fn start_bare(port: u16) -> Running {
.arg(format!("127.0.0.1:{port}"))
.args(["--log", "error", "--status-interval", "0"])
.stdout(std::process::Stdio::null())
.stderr(std::process::Stdio::null())
.spawn()
.expect("the agent binary starts");
std::thread::sleep(Duration::from_secs(2));
Running { child, dir }
.stderr(std::process::Stdio::null());
if let Some(range) = range {
command.args(["--ipv4-range", range]);
}
if let Some(protocols) = protocols {
command.args(["--protocol", protocols]);
}
let child = command.spawn().expect("the agent binary starts");
let mut agent = Running { child, dir };
let runtime = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.unwrap();
let socket = tsunagi::ipc::control_socket_path(&agent.dir.path().join("state"));
runtime.block_on(async {
tokio::time::timeout(Duration::from_secs(30), async {
while !tsunagi::ipc::is_serving(&socket).await {
assert!(
agent.child.try_wait().unwrap().is_none(),
"agent exited before becoming ready"
);
tokio::time::sleep(Duration::from_millis(25)).await;
}
})
.await
.expect("agent becomes ready");
});
agent
}
/// An agent already in one network, the way most of these start.
@@ -87,6 +119,147 @@ fn start(network: &str, port: u16) -> Running {
agent
}
#[test]
fn protocols_absent_from_startup_can_be_enabled_and_changed_offline() {
let mut agent = start_bare_config(0, None, Some("none"));
assert!(
agent
.run(&[
"join",
"--network",
"from-control-only",
"--secret",
"a-control-only-test-secret"
])
.status
.success()
);
let initial = agent.run(&["protocols", "from-control-only"]);
assert!(initial.status.success());
assert!(String::from_utf8_lossy(&initial.stdout).contains("protocols none"));
let enabled = agent.run(&["protocols", "from-control-only", "awg"]);
assert!(
enabled.status.success(),
"{}",
String::from_utf8_lossy(&enabled.stderr)
);
assert_eq!(
read_networks(&agent.dir)[0].protocols,
Some(vec!["awg".into()])
);
agent.child.kill().unwrap();
agent.child.wait().unwrap();
let saved = agent.run(&["protocols", "from-control-only", "wg-quic,awg"]);
assert!(
saved.status.success(),
"{}",
String::from_utf8_lossy(&saved.stderr)
);
assert!(String::from_utf8_lossy(&saved.stdout).contains("agent stopped"));
assert_eq!(
read_networks(&agent.dir)[0].protocols,
Some(vec!["wg-quic".into(), "awg".into()])
);
}
#[test]
fn protocols_are_changed_live_per_network_and_saved_without_changing_the_service() {
let agent = start("protocol-choice", 0);
assert!(
agent
.run(&[
"join",
"--network",
"other-choice",
"--secret",
"other-secret-for-cli-test"
])
.status
.success()
);
let before = read_networks(&agent.dir);
let other_id = before
.iter()
.find(|n| n.name.as_str() == "other-choice")
.unwrap()
.network_id;
let changed = agent.run(&["protocols", "protocol-choice", "awg,wg-quic"]);
assert!(
changed.status.success(),
"{}",
String::from_utf8_lossy(&changed.stderr)
);
let shown = agent.run(&["protocols", "protocol-choice"]);
assert!(shown.status.success());
assert!(String::from_utf8_lossy(&shown.stdout).contains("awg,wg-quic"));
let saved = read_networks(&agent.dir);
assert_eq!(
saved
.iter()
.find(|n| n.name.as_str() == "protocol-choice")
.unwrap()
.protocols,
Some(vec!["awg".into(), "wg-quic".into()])
);
assert_eq!(
saved
.iter()
.find(|n| n.network_id == other_id)
.unwrap()
.protocols,
before
.iter()
.find(|n| n.network_id == other_id)
.unwrap()
.protocols
);
for invalid in ["unknown", "awg,awg", "none,awg"] {
assert!(
!agent
.run(&["protocols", "protocol-choice", invalid])
.status
.success()
);
}
assert!(!agent.run(&["protocols", "missing", "awg"]).status.success());
let disabled = agent.run(&["protocols", "protocol-choice", "none"]);
assert!(
disabled.status.success(),
"{}",
String::from_utf8_lossy(&disabled.stderr)
);
assert_eq!(
read_networks(&agent.dir)
.iter()
.find(|n| n.name.as_str() == "protocol-choice")
.unwrap()
.protocols,
Some(Vec::new())
);
assert!(
agent
.run(&["network", "stop", "protocol-choice"])
.status
.success()
);
assert!(
agent
.run(&["protocols", "protocol-choice", "wg-quic"])
.status
.success()
);
assert!(
agent
.run(&["network", "start", "protocol-choice"])
.status
.success()
);
assert!(
String::from_utf8_lossy(&agent.run(&["protocols", "protocol-choice"]).stdout)
.contains("wg-quic")
);
}
#[test]
fn joining_with_no_secret_makes_one_and_prints_what_to_send() {
// The hurry case: a network with somebody, for as long as it is needed,
@@ -461,3 +634,139 @@ fn an_exit_node_is_offered_per_network_only_when_asked_and_nobody_offers_by_defa
String::from_utf8_lossy(&unknown.stderr)
);
}
#[test]
fn overlapping_networks_are_reported_and_cli_start_switches_then_resumes() {
let agent = start_bare_range(0, Some("10.90.0.0/16"));
for name in ["resident", "games"] {
let joined = agent.run(&["join", "-n", name, "-s", "a-secret-for-the-conflict-test"]);
assert!(
joined.status.success(),
"{}",
String::from_utf8_lossy(&joined.stderr)
);
if name == "games" {
let text = String::from_utf8_lossy(&joined.stdout);
assert!(
text.contains("stopped because its range overlaps"),
"{text}"
);
assert!(!text.contains("it is running"), "{text}");
}
}
let runtime = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.unwrap();
let socket = tsunagi::ipc::control_socket_path(&agent.dir.path().join("state"));
let snapshot = || {
runtime
.block_on(tsunagi::ipc::request_status(&socket))
.unwrap()
};
let before = snapshot();
let resident = before
.networks
.iter()
.find(|n| n.name == "resident")
.unwrap();
let games = before.networks.iter().find(|n| n.name == "games").unwrap();
assert!(resident.active);
assert!(!games.active);
assert_eq!(games.range_conflicts[0].network_id, resident.network_id);
assert_eq!(games.range_conflicts[0].name, "resident");
for args in [vec!["status"], vec!["network"]] {
let result = agent.run(&args);
let text = String::from_utf8_lossy(&result.stdout);
assert!(text.contains("range conflict"), "{text}");
assert!(text.contains(&resident.network_id), "{text}");
assert!(text.contains("tsng network start"), "{text}");
}
let switched = agent.run(&["network", "start", "games"]);
assert!(
switched.status.success(),
"{}",
String::from_utf8_lossy(&switched.stderr)
);
let text = String::from_utf8_lossy(&switched.stdout);
assert!(
text.contains("started `games`") && text.contains("suspended `resident`"),
"{text}"
);
let after = snapshot();
assert!(
after
.networks
.iter()
.find(|n| n.name == "games")
.unwrap()
.active
);
assert!(
!after
.networks
.iter()
.find(|n| n.name == "resident")
.unwrap()
.active
);
assert!(agent.run(&["network", "stop", "games"]).status.success());
assert!(
snapshot()
.networks
.iter()
.find(|n| n.name == "resident")
.unwrap()
.active
);
assert!(agent.run(&["network", "start", "games"]).status.success());
let stop_suspended = agent.run(&["network", "stop", "resident"]);
assert!(stop_suspended.status.success());
assert!(String::from_utf8_lossy(&stop_suspended.stdout).contains("stopped `resident`"));
assert!(agent.run(&["network", "stop", "games"]).status.success());
assert!(
snapshot()
.networks
.iter()
.all(|n| !n.active && !n.auto_start)
);
}
#[test]
fn cli_defaults_derive_each_networks_range() {
let agent = start_bare(0);
for name in ["derived-one", "derived-two"] {
assert!(
agent
.run(&["join", "-n", name, "-s", "a-secret-for-the-derived-test"])
.status
.success()
);
}
let runtime = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.unwrap();
let socket = tsunagi::ipc::control_socket_path(&agent.dir.path().join("state"));
let deadline = Instant::now() + Duration::from_secs(30);
loop {
let report = runtime
.block_on(tsunagi::ipc::request_status(&socket))
.unwrap();
if report.networks.iter().all(|n| n.range.is_some()) {
assert_eq!(report.networks.len(), 2);
for network in report.networks {
let id = network.network_id.parse().unwrap();
assert!(network.active);
assert_eq!(
network.range,
Some(tsunagi::state::derived_ipv4_range(id).to_string())
);
}
break;
}
assert!(Instant::now() < deadline, "networks allocate ranges");
std::thread::sleep(Duration::from_millis(25));
}
}
+131
View File
@@ -0,0 +1,131 @@
# Pairwise Amnezia transport
The CLI's default carrier preference is `wg,awg,wg-quic,tcp-tls`. To select
AWG explicitly:
```sh
tsng up --protocol awg
tsng up --protocol wg,awg,wg-quic,tcp-tls
```
The startup list is the initial per-network preference. A saved network choice
overrides it and can enable any built-in protocol at runtime:
```sh
tsng protocols home awg,wg-quic
tsng protocols home awg
```
Changes are local to this device, persist across restarts, and retire existing
data links and end-to-end channels for that network before renegotiation.
Authenticated control sessions, membership, addresses and other networks stay
live. Removing every protocol (`none` in the CLI, all checkboxes off in the tray)
leaves the network control-only. This is different from automatic carrier
fallback, which preserves the selected end-to-end channel.
`awg` is Tsunagi AWG wire version 1. Its packet masking engine is adapted from
[RunaRoo/AmneziaWG-RS](https://github.com/RunaRoo/AmneziaWG-RS), with the pinned
source and license recorded in
[UPSTREAM.md](../crates/tsunagi-awg/UPSTREAM.md). It uses the existing boringtun
Noise/WireGuard implementation instead of upstream's Node, global executors
and OS interface machinery. Compatibility with ordinary Amnezia servers is
not implemented: this carrier connects authenticated Tsunagi members.
## Negotiation and lifetime
There is one independent UDP socket and cancellable packet-engine task per
`(network, peer)` pair. The smaller endpoint id initiates, following the same
rule as other data carriers. Both sides must announce enabled AWG wire version
1. An authenticated iroh control session is required throughout; AWG supplies
no alternate discovery or membership mechanism.
Every `open` generates new masking parameters, a header-protection key, an
outer WireGuard key pair and a PSK. The initiator sends a private `Offer` with
a random attempt id, its public key, candidate socket addresses and the shared
profile. The responder validates it, creates its own fresh key pair and socket,
and returns an `Answer` naming that attempt. Both send masked handshakes and
encrypted delivery probes to the candidates. Only authenticated WireGuard
**data**, not a STUN response or an incoming UDP address alone, verifies a path.
Negotiation payloads are bounded at 4096 bytes (a caller can lower the limit),
redacted in Debug, excluded from public message events and erased on drop.
They are never capabilities, signed records, cache entries or exported status.
At most 128 pairs are admitted per transport, with bounded command/data queues,
at most 32 candidates per attempt and finite establishment deadlines.
Stale answers and duplicate offers are rejected; a new attempt replaces and
closes an older pair engine. Cancellation, peer departure, network deactivation
and shutdown retire sockets without waiting for a remote reader.
A process restart, a failed setup, or a complete fallback cycle returning to AWG
starts fresh negotiation. Normal WireGuard rekeying within a live attempt keeps
that attempt's masking profile. A failed pair does not reconfigure other pairs
or other networks. A full mesh of 15 members uses 14 engines per member and 105
independent pairs in total; it does not create additional system interfaces.
## Masking
Each profile randomizes disjoint H1–H4 header ranges, a 12–48-byte S1–S4 prefix,
a ChaCha20 header-protection key, handshake trailers, junk packet count/lengths,
and extra content padding. Prefix lengths are equal within one profile so
header detection has one offset. A fresh random nonce is in every packet's
prefix; handshake headers and transport headers/counters are protected.
The standard WireGuard header is restored before boringtun authenticates a
message. This preserves boringtun's standard MAC calculations, which is another
reason this is a Tsunagi-specific AWG wire version.
Before a handshake, an I1 signature packet is generated from either a complete
DNS query or SIP OPTIONS template, with fresh identifiers and consistent length
fields, followed by random junk packets. I2–I5 are unused in this version.
The templates are structural generators, not replayed captures. QUIC imitation
is not selected. Masking can change DPI classification; it is not a promise of
indistinguishability or that any particular firewall will admit it.
## Packets, NAT and fallback
The pair engine carries opaque relay envelopes inside authenticated outer WG
packets. It fragments those envelopes into bounded chunks and expires incomplete
assemblies; a logical datagram is bounded at 64 KiB and an outer UDP packet is
at most 1200 bytes. The encapsulated IPv4 headers are internal framing only: no interface or OS address is created for a
pair. Overlay addresses, routing, ownership checks and end-to-end encryption
remain independent. A relay decrypts its own adjacent AWG carrier to forward
opaque end-to-end ciphertext, never the destination's IP packet.
Local IPv4 candidates and simultaneous encrypted probes provide direct UDP
connectivity. In `direct`/`relay` reach mode the CLI also uses
`stun.l.google.com:19302` to discover a server-reflexive address on **this pair's
socket**, with a 1.2-second bound. A STUN result is only a candidate. Local reach
mode makes no STUN request. Configure or disable it with
`-o awg:stun=HOST:PORT` or `-o awg:stun=off`. Failed discovery still permits local
candidates. IPv6-only paths, symmetric NAT and networks blocking arbitrary UDP
may fail AWG and fall back; there is no AWG-over-QUIC substitution or AWG TURN
relay hidden behind its name.
Data delivery checks go through the actual end-to-end encrypted tunnel, once
per second, independently of control-plane RTT. Requests and replies bind a
fresh private marker/key to the sender's endpoint id. Unanswered requests expire
after three seconds. After a 20-second warmup, at least 15 completed observations
and at least 60% loss in the last 20 observations retire a carrier. Probes are
consumed locally after source/destination admission and never enter the TUN.
AWG additionally checks encrypted outer delivery throughout the pair lifetime.
Setup failure or persistent runtime loss advances to the next mutually supported
carrier in the initiator's order. Earlier expired cooldowns do not jump ahead of
untried carriers. Retry cooldowns remain bounded; reaching AWG again negotiates
new parameters. Control sessions and the logical end-to-end tunnel survive a
carrier change. Status names the end-to-end protocol and the actual carrier
separately, so `wg` over `awg` is a valid result.
Control wire version 6 introduces the private transport message. All members
must upgrade together; device identities, network ids, handshake transcript
encoding, signed state and stored network configuration are unchanged. Data
ALPN remains `tsunagi/data/5`; the relay envelope is unchanged.
## Verification
The default rootless, offline tests use real iroh endpoints, membership proofs,
SQLite, UDP sockets, masked WireGuard handshakes, end-to-end encryption and
memory TUNs. They cover independent three-member pairs, full-size bidirectional
packets, restart freshness/socket release, 80% runtime datagram loss, fallback
to real iroh and fresh negotiation on retry, plus malformed profiles/messages,
role checks, probe forgery and STUN response parsing. These tests do not establish
DPI effectiveness, WAN throughput, public NAT traversal or per-pair memory cost.
+8 -2
View File
@@ -20,8 +20,9 @@ network secret. The core moves an opaque, bounded payload and never parses it.
a `PacketLink` — an authenticated, unreliable datagram channel to one peer for
one protocol — and never learn how it is carried.
The separation between the two planes is **logical, not physical**. Both ride
on iroh, on different ALPNs and different connections. That is deliberate:
The separation between the two planes is **logical, not physical**. The control plane uses
iroh; the data plane can use iroh on its own ALPN and connection, or independent
UDP/TLS carriers. That is deliberate:
iroh's whole value is hole punching a direct path between peers behind NAT,
with a relay as fallback, and a data plane that refused to use it would have to
reimplement all of it. What the separation buys is that `proto` knows nothing
@@ -32,6 +33,10 @@ A plugin talks to the core through three narrow hooks — `on_network_activated`
a `PluginContext` for re-announcements and error reports, and a bounded
`shutdown` — so the core never learns anything protocol-specific.
Private pair negotiation uses a bounded transport sender supplied by the core.
Protocol crates do not encode or parse control messages and never receive the
network secret. No pair profiles enter signed state.
A data plane failure never stops the daemon: the control plane keeps running
and the agent stays manageable.
@@ -68,6 +73,7 @@ And in crates of their own:
| crate | responsibility |
|---|---|
| `tsunagi-awg` | ephemeral pairwise Amnezia masking, private negotiation and independently encrypted UDP carriers; reuses the end-to-end WireGuard engine |
| `tsunagi-wg-quic` | the `wg-quic` protocol: its keys, its announcement, its tunnels |
| `tsunagi` | the command line agent, built as the `tsng` executable: the only place that owns a runtime, a logger and signals |
+6 -1
View File
@@ -9,11 +9,15 @@ Two versions exist and are independent:
- **Identity scheme**, `tsunagi-network-id-v1`. Frozen. Changing it creates a
different network space for the same name and secret.
- **Control protocol**, ALPN `tsunagi/ctrl/5`, `PROTOCOL_VERSION = 5`.
- **Control protocol**, ALPN `tsunagi/ctrl/6`, `PROTOCOL_VERSION = 6`.
Upgrading the crate or bumping the control protocol must never change an
existing `NetworkId`.
Control version 6 adds private transport negotiation for pairwise AWG and data
delivery checks. Data ALPN remains `tsunagi/data/5`. All members must upgrade
together; network ids and stored state are unchanged. See [amnezia.md](amnezia.md).
## Canonical encoding
Everything that is hashed or MAC'd uses length-prefixed fields, written here as
@@ -230,6 +234,7 @@ not affect other networks.
| message | meaning |
|---|---|
| `Announce { hostname, capabilities, broadcast }` | this agent's hostname, IP-plugin capabilities and local broadcast participation |
| `Transport { protocol, payload }` | private, bounded per-peer transport negotiation; never state or a public message event |
| `Ping { seq, payload }` | small request used to verify the exchange |
| `Pong { seq, payload }` | the echoed reply |
| `State { records }` | a snapshot of signed records, merged into what the receiver holds |
+9 -1
View File
@@ -96,6 +96,14 @@ topology change can move a flow; datagrams remain unreliable and unordered,
and there is no promise to preserve order across a physical path failure.
Logical peer links and end-to-end encryption state survive these changes.
## Carrier health and switching
Carrier setup failures and persistent encrypted delivery loss advance the
initiator's ordered fallback cycle. Logical end-to-end links survive while
physical links are retired explicitly. The AWG carrier negotiates a fresh
private pair profile on every reentry, without changing topology's durable
state. See [amnezia.md](amnezia.md) for thresholds and bounds.
## Verification and measurement
The default offline suite exercises four real agents, iroh transports and
@@ -116,5 +124,5 @@ counts sends without storing frames. Results are CPU forwarding cost, **not**
end-to-end network latency or VPN throughput; encryption, fragmentation,
sockets, congestion and scheduling contribute separately.
Wire compatibility: control ALPN `tsunagi/ctrl/5`, data ALPN `tsunagi/data/5`.
Wire compatibility: control ALPN `tsunagi/ctrl/6`, data ALPN `tsunagi/data/5`.
Upgrade every participant together; saved identities and network state persist.
+28
View File
@@ -98,6 +98,13 @@ a batch does not stop the rest, and allocation is deterministic, spread out,
walks past everything taken and reports a full range instead of handing out a
duplicate.
`tests/range_conflicts.rs` checks isolated first members choosing the same
range, persistence across restart, preservation of legacy ranges, exclusion
of equal and nested ranges, late authenticated conflicts, switching all
blockers, automatic resume, cancelling a suspended network's resume and
concurrent switch requests. These use real agents, loopback handshakes and
SQLite; they do not change host network settings.
`tests/local_control.rs` covers the local control socket end to end: a client
asking a running agent for status over a real Unix socket, joining and
leaving a network through it, a leftover socket file being replaced while a
@@ -170,3 +177,24 @@ TSUNAGI_TEST_LOG=tsunagi=debug cargo test --test multi_peer -- --nocapture
The library never installs a global subscriber; the harness opts in only when
that variable is set.
The `tsunagi-awg` suite runs real pair negotiation and masked UDP WireGuard
engines with memory TUNs. A loss wrapper drops datagrams at the carrier boundary
without substituting iroh, membership, encryption or SQLite. It checks runtime
loss detection, fallback and fresh profiles, in addition to restart and input
validation. Tests are offline; public STUN/DPI/NAT effectiveness needs separate
real-network validation.
Runtime protocol selection tests switch real AWG and iroh/QUIC carriers and
end-to-end WireGuard tunnels from either endpoint-id role, carry full-size
packets, keep another active network's tunnels unchanged, disable and re-enable
the data plane without losing control sessions, and preserve selection after
restart. CLI tests use a real agent process and control socket to exercise
selection by network name, stopped networks, invalid lists, and persisted
control-only configuration, offline updates, and enabling a protocol absent
from the startup list. A real AWG test starts without a TUN, creates the single
memory interface on enablement, and races rapid changes against pending opens.
SQLite tests migrate schema 7 to 8 without changing
identity, secrets or signed records, preserve order and explicit empty choices,
and reject malformed stored selections. These checks use memory TUNs and do
not establish GUI rendering or behavior on an actual OS interface.
+13 -2
View File
@@ -26,8 +26,8 @@ Read this before relying on anything here. The protocol is in
neither receive nor forge another member's traffic. See
[wireguard.md](wireguard.md#address-ownership-is-enforced-not-announced).
- **Tunnelled traffic is end-to-end encrypted by WireGuard**, independently of
this crate. The transport underneath is also encrypted by iroh, but the
tunnel's confidentiality does not depend on that.
this crate. Carriers may be iroh, TLS, direct WG or
pairwise AWG; confidentiality does not depend on a particular carrier.
- **Resource bounds.** Frame lengths are validated before allocation; strings,
lists, queues, concurrent dials and in-flight handshakes are all bounded;
handshakes, dials and writes have timeouts.
@@ -113,3 +113,14 @@ for key separation, HMAC-SHA256 for the membership proof, constant-time
verification, iroh's Ed25519 endpoint keys and QUIC/TLS for the transport, and
RFC 5705 TLS exporter output for channel binding. There is no custom encryption
layer and no custom PAKE.
## Pairwise AWG masking
AWG profiles, PSKs and outer keys are private to one authenticated pair attempt,
not shared network state. Address candidates are unverified until authenticated
outer WireGuard data arrives. Header protection and structured signature packets
reduce fixed wire markers; they do not guarantee admission by DPI or hide traffic
analysis. iroh control availability remains a prerequisite. Encrypted end-to-end
delivery probes detect sustained carrier loss, never confer membership and pass
the same signed source/destination admission as ordinary IP traffic. See
[amnezia.md](amnezia.md) for limits and tested scope.
+30 -7
View File
@@ -92,8 +92,8 @@ and then recorded as a signed fact**, using the model in
[sync-model.md](sync-model.md).
```text
default range 10.13.37.0/24 (override with --ipv4-range)
who decides the first member to claim; later ones adopt what they find
default range stable pseudorandom /24 in 10/8, derived from NetworkId
who decides all members derive the default; signed explicit ranges win
who signs the claiming member, with its persistent device key
where it is kept state.sqlite, and every replica that has seen it
what a return costs nothing: the old address is reclaimed
@@ -120,11 +120,34 @@ the same address at the same moment are resolved by a rule both compute
identically — the lower endpoint id keeps it — and the loser simply allocates
again with a higher version.
**The range is agreed, not configured per member.** `--ipv4-range` says what
this agent would use; a network that has already settled on something else
wins, and the agent adopts it. So the flag matters for whoever starts the
network and is harmless afterwards. Pass `--ipv4-range none` for an IPv6-only
overlay.
**The range is agreed, not configured per member.** New networks use the
same deterministic `/24` on every device, independently of which network a
member activates first or whether any peers are reachable. The derivation
is unchanged from the former fallback; network identity and signed record
encoding are unchanged. Distinct network ids can still derive the same range.
`--ipv4-range` proposes an explicit range; existing signed ranges take
precedence over the new derived default, including the former `10.13.37.0/24`.
Among disagreeing non-default ranges the lowest author id still wins. Updated
members propagate their adopted range in their signed claims; older releases
still apply their original selection rule until those claims converge.
`--ipv4-range none` disables local proposals, as before.
An agent reserves each active network's range before starting its runtime.
Overlapping ranges, including nested blocks such as a `/24` inside a `/16`,
cannot run simultaneously. A late authenticated state update that introduces
an overlap suspends that network before publishing new allocations to plugins
or the interface. Other networks continue running.
Status includes every blocking network's name, id and range. `tsng network
start <name-or-id>` and the tray's switch button stop all blockers before
starting the selected network. Suspended networks keep their startup request
and resume when the blocker stops or is forgotten. A manual stop cancels that
request even for a network already suspended. The user's preference order is
stored locally, so the same selection wins at the next startup; it is never
advertised or treated as a remote network setting. Local IPC uses wire version
19 for the new reports: restart the agent and tray together after upgrading.
The peer control protocol and signed record encoding do not change.
The agent puts the address on the interface itself, as soon as the network has
agreed on it — no restart, and the interface is not recreated, which would