Files
tsunagi/crates/tsunagi-core/src/dataplane/mod.rs
T
Aleksandr Bogomiakov 2166ba4800 Separate internal WireGuard encryption from data carriers
AWG-only policy also selected a competing end-to-end plugin, so compatible
external links could still have no compatible encrypted data channel. Carrier
changes replaced tunnel state and status selected counters from different
engines. Use one internal WireGuard plugin and a separate DataCarrier contract
for external capabilities/lifecycle. Keep the logical link, keys, handshake,
interface and counter epoch across policy, fallback and direct/relay changes.
Control/discovery transport selection and membership remain independent.

Share the bounded per-neighbour UDP WireGuard engine between ordinary WG and
Amnezia masking. Encrypt the relay envelope inside the outer hop channel with
fresh pair keys/PSK. Remove duplicate WG key/device code and the TCP/TLS inner
cipher. TLS carries opaque internal WireGuard frames. Refuse missing native
transports instead of letting default iroh impersonate their protocol names.
Probe the actual authenticated carrier, independently of the shared IP tunnel;
failed local probe submissions count as loss too.

Apply the vendored iroh interface exclusion to actual direct candidates, not
just public discovery records. Exclude overlay subnets from introductions and
propagate the actual owned interface name, including kernel-assigned utun names,
to every carrier. Production logs showed paths through the overlay itself.
This removes a recursive transport path; it does not establish the historical
cause of the reported 100 GB absolute counter. Exit-node traffic is also real
peer IP traffic even when local applications are idle.

Hide inner cipher choices/handshake clutter in CLI and tray. Report external
routes, readiness and actionable policy/version failures. Count original IP
packets only when their ciphertext is accepted by the logical transport,
including queued packets' exact unpadded lengths. Exclude physical probes and
keepalives. Export counter epochs; sum per-peer rate deltas and discard stale
histories so joins/restarts cannot turn old totals into fresh traffic. Status
also prints exact IP byte totals for comparison with the tray.

Coordinated upgrade: data ALPN 6, internal wireguard-e2e capability v4,
wg/awg v2, QUIC carrier v1, TLS v2, local IPC v23. Control ALPN 8, device/network
identity derivation, signed records and saved network choices stay unchanged.
Keep the historical wg-quic key store; transactionally import missing keys from
wg then awg without replacing canonical keys. Forget removes legacy copies too,
on the blocking pool, to prevent resurrection after restart. Preserve historic
cipher options as shared settings, rejecting contradictory values.

Validation on Linux in the Nix development shell:
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets (562 passed, 3 ignored)
The inherited service socket/secret/group environment overrides were removed
for isolated CLI process tests. A new carrier test initially raced the first
peer announcement; its deadline polling now waits for a peer row explicitly.
Real loopback endpoints, membership, SQLite, WG/AWG/TLS encryption and memory
TUNs cover mixed-carrier relaying and one smoltcp TCP connection surviving
relay/direct/policy changes and a dropped transmission. Counter tests check
exact plaintext lengths, refused sends, epochs, joins and stale histories.
Public DHT, privileged kernel and manual benchmark tests remain ignored;
macOS/Windows native TUNs, desktop GUI and WAN/DPI behavior were not run.
2026-10-22 01:07:52 +01:00

516 lines
18 KiB
Rust
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
//! Boundaries between the agent, one IP encryption tunnel, and external carriers.
//!
//! The core moves bounded opaque capabilities without parsing cipher payloads.
//! An [`IpPlugin`] sees routed IP packets and a logical [`PacketLink`] per peer;
//! it never learns reachability or the network secret. The agent configures one
//! such plugin. Built-in agents use the same internal WireGuard cipher for every
//! peer, regardless of which external transports carry the encrypted bytes.
//!
//! [`transport::DataCarrier`] owns external capability negotiation and lifecycle.
//! It sees reachability but never plaintext overlay packets. A relay forwards
//! the encrypted inner bytes without entering an IP plugin or the TUN.
//! Data-plane failure never stops control sessions or discovery.
pub mod relay;
pub mod routing;
pub mod transport;
use std::sync::Arc;
use iroh::EndpointId;
use tokio::sync::mpsc;
use crate::BoxFuture;
use crate::identity::NetworkId;
pub use transport::{
MultiPacketTransport, PacketLink, PacketTransport, SharedLink, TransportError,
};
/// Maximum length of a plugin protocol identifier.
pub const MAX_PROTOCOL_ID_LEN: usize = 32;
/// Validates bounded local protocol preferences before storing or applying them.
pub(crate) fn validate_protocols(protocols: &[String], max: usize) -> crate::error::Result<()> {
if protocols.len() > max {
return Err(crate::Error::InvalidProtocols(format!(
"at most {max} protocols are allowed"
)));
}
for (index, name) in protocols.iter().enumerate() {
if name.is_empty()
|| name.len() > MAX_PROTOCOL_ID_LEN
|| !name
.bytes()
.all(|c| c.is_ascii_alphanumeric() || c == b'-' || c == b'_')
{
return Err(crate::Error::InvalidProtocols(
"names must be 1–32 ASCII letters, digits, hyphens or underscores".into(),
));
}
if protocols[..index].contains(name) {
return Err(crate::Error::InvalidProtocols(format!(
"`{name}` is listed more than once"
)));
}
}
Ok(())
}
/// An announcement of one IP plugin's capability.
///
/// `data` is opaque to the core. Nothing in it may be interpreted as a shell
/// command, a filesystem path or an OS setting by the core; a plugin that
/// chooses to do so must validate it itself.
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
pub struct PluginCapability {
/// Protocol identifier, e.g. `wg-quic`. Bounded by [`MAX_PROTOCOL_ID_LEN`].
pub protocol: String,
/// Version of the plugin's announcement format.
pub version: u16,
/// Whether the peer currently has this plugin enabled.
pub enabled: bool,
/// Opaque, bounded, plugin-defined payload.
pub data: Vec<u8>,
}
/// Where a protocol hands the packets it has decrypted.
///
/// A protocol proves *who* sent a packet; it does not know what that member
/// is entitled to say, because entitlement is an address claim the system
/// level holds. So a decrypted packet goes here rather than straight to an
/// interface, and is checked on the way.
pub trait PacketSink: Send + Sync + 'static {
/// Hands over one packet, attributed to the peer whose tunnel decrypted
/// it.
fn deliver<'a>(
&'a self,
network: NetworkId,
peer: EndpointId,
packet: bytes::Bytes,
) -> BoxFuture<'a, ()>;
}
/// A sink that drops everything, for a protocol running without an interface.
#[derive(Debug, Clone, Copy, Default)]
pub struct DiscardPackets;
impl PacketSink for DiscardPackets {
fn deliver<'a>(
&'a self,
_network: NetworkId,
_peer: EndpointId,
_packet: bytes::Bytes,
) -> BoxFuture<'a, ()> {
Box::pin(async move {})
}
}
/// One setting a protocol accepts.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct ProtocolOption {
/// The key, as written in `key=value`.
pub key: &'static str,
/// A word for the value, for the help line: `BYTES`, `NAME`.
pub value: &'static str,
/// What it does.
pub help: &'static str,
/// What happens when it is not given.
pub default: Option<&'static str>,
}
/// Errors a plugin may return. They are recorded, never fatal for the agent.
#[derive(Debug, thiserror::Error)]
#[non_exhaustive]
pub enum PluginError {
/// The plugin is not currently able to produce or apply configuration.
#[error("plugin unavailable: {0}")]
Unavailable(String),
/// A peer announcement was not acceptable to the plugin.
#[error("rejected peer announcement: {0}")]
Rejected(String),
/// Anything else.
#[error("plugin error: {0}")]
Other(String),
}
/// A request a plugin makes of the agent that owns it.
#[derive(Debug)]
pub enum PluginRequest {
/// Re-send this agent's announcement to every peer of a network.
Reannounce(NetworkId),
/// Surface a plugin error on the agent's event stream.
Error {
/// Network the error is scoped to.
network: NetworkId,
/// Plugin protocol id.
protocol: String,
/// Human readable reason, free of secrets.
reason: String,
},
/// A plugin reported that a data link or tunnel for a peer failed.
ProtocolFailure {
/// Network the failure occurred in.
network: NetworkId,
/// Peer whose protocol failed.
peer: EndpointId,
/// Protocol that failed.
protocol: String,
/// Human readable reason.
reason: String,
},
}
/// The agent-side handle a plugin is given when it is attached.
///
/// It is deliberately tiny: a plugin may ask for its announcement to be resent
/// and may report an error. It cannot reach into agent state, cannot send
/// arbitrary control messages and knows nothing about sessions.
///
/// All calls are non-blocking. If the agent is gone or its queue is full the
/// request is dropped rather than stalling the plugin.
#[derive(Clone)]
pub struct PluginContext {
sender: Option<mpsc::Sender<PluginRequest>>,
local: Option<EndpointId>,
/// Where decrypted packets go. Absent when nothing is carrying traffic,
/// in which case a protocol still runs and its packets are discarded.
sink: Option<Arc<dyn PacketSink>>,
}
impl PluginContext {
pub(crate) fn new(
sender: mpsc::Sender<PluginRequest>,
local: EndpointId,
sink: Option<Arc<dyn PacketSink>>,
) -> Self {
Self {
sender: Some(sender),
local: Some(local),
sink,
}
}
/// Where to hand a decrypted packet.
///
/// Always answers: with no interface to write to, the packets are
/// discarded, which is what `--no-tun` means and is not an error.
pub fn packet_sink(&self) -> Arc<dyn PacketSink> {
match &self.sink {
Some(sink) => Arc::clone(sink),
None => Arc::new(DiscardPackets),
}
}
/// A context that discards everything, for plugins used outside an agent.
pub fn detached() -> Self {
Self {
sender: None,
local: None,
sink: None,
}
}
/// A context configured with a custom packet sink.
pub fn with_sink(sink: Arc<dyn PacketSink>) -> Self {
Self {
sender: None,
local: None,
sink: Some(sink),
}
}
/// A context configured with a channel for plugin requests and custom sink, for tests.
#[cfg(feature = "testing")]
pub fn for_test(
sender: mpsc::Sender<PluginRequest>,
local: EndpointId,
sink: Option<Arc<dyn PacketSink>>,
) -> Self {
Self {
sender: Some(sender),
local: Some(local),
sink,
}
}
/// This agent's own endpoint id, when the context is attached.
///
/// A plugin needs it to find itself in the agreed allocation.
pub fn local_endpoint_id(&self) -> Option<EndpointId> {
self.local
}
fn send(&self, request: PluginRequest) {
let Some(sender) = &self.sender else {
return;
};
if let Err(err) = sender.try_send(request) {
tracing::debug!(%err, "dropping plugin request");
}
}
/// Asks the agent to resend this agent's announcement in `network`.
///
/// A plugin calls this when its own capability changed — it finished
/// starting up, its keys or reachability changed — so that peers learn the
/// new value without waiting for a reconnect.
pub fn request_reannounce(&self, network: NetworkId) {
self.send(PluginRequest::Reannounce(network));
}
/// Reports a plugin error on the agent's event stream.
///
/// Plugin work happens in the plugin's own tasks, so errors cannot always
/// be returned from a trait call. They are never fatal for the agent.
pub fn report_error(
&self,
network: NetworkId,
protocol: impl Into<String>,
reason: impl Into<String>,
) {
self.send(PluginRequest::Error {
network,
protocol: protocol.into(),
reason: reason.into(),
});
}
/// Reports that data plane connectivity or a tunnel for a peer has failed.
pub fn report_peer_protocol_failure(
&self,
network: NetworkId,
peer: EndpointId,
protocol: impl Into<String>,
reason: impl Into<String>,
) {
self.send(PluginRequest::ProtocolFailure {
network,
peer,
protocol: protocol.into(),
reason: reason.into(),
});
}
}
impl std::fmt::Debug for PluginContext {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("PluginContext")
.field("attached", &self.sender.is_some())
.field("local", &self.local.map(|id| id.fmt_short().to_string()))
.finish()
}
}
/// The contract an IP plugin implements.
///
/// Implementations must be cheap and non-blocking: the agent calls them from
/// its runtime tasks. Anything slow belongs in the plugin's own tasks.
pub trait IpPlugin: Send + Sync + std::fmt::Debug + 'static {
/// Stable protocol identifier, e.g. `wg-quic`.
///
/// Must be non-empty and at most [`MAX_PROTOCOL_ID_LEN`] bytes.
fn protocol_id(&self) -> &str;
/// The wire version of this protocol.
///
/// Two peers carry traffic for each other only when they have the same
/// protocol at the same version. There is no negotiating a middle
/// ground: a protocol either speaks the same words at both ends or it
/// does not, and pretending otherwise produces a link that fails later
/// and less clearly.
///
/// **Not the software version.** A plugin crate has its own version and
/// it is nobody else's business: two peers on different builds work
/// together for as long as the wire between them has not changed. This
/// number moves only when the bytes do, so it must never be derived from
/// `CARGO_PKG_VERSION` or anything else that moves with a release.
fn protocol_version(&self) -> u16;
/// The settings this protocol accepts, and what they mean.
///
/// Declared rather than documented elsewhere, so the agent can list them
/// without knowing anything about the protocol.
fn options(&self) -> &'static [ProtocolOption] {
&[]
}
/// Called once, when the agent starts, before any network is activated.
///
/// The plugin keeps the context to ask for re-announcements and to report
/// errors that happen in its own tasks.
fn attach(&self, context: PluginContext) {
let _ = context;
}
/// Produces this agent's announcement for a given network.
///
/// Returning `Ok(None)` means "nothing to announce right now", which is
/// different from an error.
fn local_capability(
&self,
network: NetworkId,
) -> std::result::Result<Option<PluginCapability>, PluginError>;
/// Called when a network is activated locally, before any peer appears.
///
/// A plugin uses it to get its per-network state ready, so that the first
/// announcement already carries its capability.
fn on_network_activated(&self, network: NetworkId) {
let _ = network;
}
/// Called when a peer announces a capability for this plugin's protocol.
///
/// The core has already bounded the payload size but has not interpreted it.
fn on_peer_capability(
&self,
network: NetworkId,
peer: EndpointId,
capability: &PluginCapability,
) -> std::result::Result<(), PluginError>;
/// The overlay addresses the network has agreed on.
///
/// Allocated rather than derived, and backed by the signed records in
/// [`crate::state`], so a participant keeps its address across restarts
/// and long absences. Called whenever the agreed picture changes.
fn on_address_allocation(
&self,
network: NetworkId,
range: crate::state::Ipv4Range,
allocations: &[(EndpointId, std::net::Ipv4Addr)],
) {
let _ = (network, range, allocations);
}
/// Carries one packet to a peer, encrypting it however this protocol
/// does.
///
/// `false` when it cannot right now — no link, no tunnel, not this
/// protocol's peer — which the caller reports rather than treats as an
/// error. The packet came off the one interface the agent owns, and
/// which protocol takes it is settled by asking.
///
/// The default carries nothing, which is right for a plugin that only
/// announces something.
fn carry(&self, _network: NetworkId, _peer: EndpointId, _packet: bytes::Bytes) -> bool {
false
}
/// A data plane link to a peer is available for this plugin's protocol.
///
/// The plugin moves its packets over this link and never learns how the
/// link is carried. A new link for a peer replaces any previous one.
fn on_peer_link(&self, network: NetworkId, peer: EndpointId, link: SharedLink) {
let _ = (network, peer, link);
}
/// Called when a peer's session in a network goes away.
///
/// Any link handed to the plugin for that peer must be dropped here.
fn on_peer_gone(&self, network: NetworkId, peer: EndpointId);
/// Called when a network is deactivated locally.
///
/// This is a local deactivation, not a signed revocation of membership.
/// The plugin is expected to remove whatever it created for that network.
fn on_network_deactivated(&self, network: NetworkId);
/// Called when this agent has left a network for good.
///
/// Deactivation is temporary and keeps everything ready for next time;
/// this is the other one. Whatever the plugin holds *durably* for that
/// network — a key of its own, a file, a record — goes now, because the
/// agent is no longer a member and keeping it is keeping a secret for a
/// network it cannot rejoin without being told the secret again.
///
/// Always preceded by [`IpPlugin::on_network_deactivated`], so this is
/// only about what outlives a session. Defaulted to nothing, for a
/// plugin that stores nothing.
fn on_network_forgotten(&self, _network: NetworkId) -> BoxFuture<'_, ()> {
Box::pin(async {})
}
/// Called once when the agent shuts down.
///
/// The plugin removes the system objects it created and stops its tasks.
/// It must be bounded: the agent awaits it during shutdown.
fn shutdown<'a>(&'a self) -> BoxFuture<'a, ()> {
Box::pin(async {})
}
}
/// A shared handle to a plugin.
pub type SharedPlugin = Arc<dyn IpPlugin>;
/// A plugin used in tests and examples.
///
/// It announces an explicitly test-only protocol id, so nothing in this crate
/// ever advertises WireGuard as an available transport before it exists.
#[derive(Debug)]
pub struct TestCapabilityPlugin {
protocol: String,
payload: Vec<u8>,
seen: std::sync::Mutex<Vec<(NetworkId, EndpointId, PluginCapability)>>,
}
impl TestCapabilityPlugin {
/// Creates a plugin announcing `protocol` with a fixed opaque payload.
pub fn new(protocol: impl Into<String>, payload: impl Into<Vec<u8>>) -> Self {
Self {
protocol: protocol.into(),
payload: payload.into(),
seen: std::sync::Mutex::new(Vec::new()),
}
}
/// Returns everything this plugin was handed so far.
pub fn observed(&self) -> Vec<(NetworkId, EndpointId, PluginCapability)> {
match self.seen.lock() {
Ok(guard) => guard.clone(),
Err(poisoned) => poisoned.into_inner().clone(),
}
}
}
impl IpPlugin for TestCapabilityPlugin {
fn protocol_id(&self) -> &str {
&self.protocol
}
fn protocol_version(&self) -> u16 {
1
}
fn local_capability(
&self,
_network: NetworkId,
) -> std::result::Result<Option<PluginCapability>, PluginError> {
Ok(Some(PluginCapability {
protocol: self.protocol.clone(),
version: 1,
enabled: true,
data: self.payload.clone(),
}))
}
fn on_peer_capability(
&self,
network: NetworkId,
peer: EndpointId,
capability: &PluginCapability,
) -> std::result::Result<(), PluginError> {
let mut guard = match self.seen.lock() {
Ok(guard) => guard,
Err(poisoned) => poisoned.into_inner(),
};
guard.push((network, peer, capability.clone()));
Ok(())
}
fn on_peer_gone(&self, _network: NetworkId, _peer: EndpointId) {}
fn on_network_deactivated(&self, _network: NetworkId) {}
}