Make the protocol a crate of its own

tsunagi-wg-quic. The line between a protocol and the system level is now
drawn by the compiler: nothing in it can reach into tsunagi beyond what
tsunagi makes public, and it carries its own version — which is not the
version peers compare.

Two things the compiler found the moment the boundary was real. The key
store was reaching into the core's `pub(crate)` file-permission helpers;
those are a legitimate service of the system level, because a protocol
keeping keys on disk has the same obligation the agent does, so they are
public now with that said. And the test harness was about to be copied
into a second crate, which is how two copies start to drift; it is a
`testing` feature of the core instead, which is also what anybody writing
a protocol would need.

The bridges put up while things were moving are gone: the error
conversion between the two levels, and the re-exports of the system
level's types from the protocol crate. Imports now say which level they
come from, which is the point.

One deliberate deviation, stated rather than hidden. The authenticated
transport stayed in the core. Moving it would have meant handing a
protocol the network's keys so it could prove membership itself, and a
plugin that can authenticate on the control plane is a worse trade than
a module boundary is worth. So the core proves who is at the other end
and the protocol owns what is said over it — the same separation, without
the secret crossing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
tsunagi
2026-09-21 19:55:30 +01:00
co-authored by Claude Opus 5
parent ff7e235414
commit 142fdf995c
31 changed files with 289 additions and 234 deletions
+39
View File
@@ -0,0 +1,39 @@
[package]
name = "tsunagi-wg-quic"
# Its own version, and deliberately separate from the wire version it
# negotiates: a release here does not stop it talking to a peer on an older
# build, because what peers compare is `ANNOUNCEMENT_VERSION`.
version = "0.1.0"
description = "The wg-quic protocol for tsunagi: WireGuard's cryptography carried in iroh's QUIC datagrams."
edition.workspace = true
rust-version.workspace = true
license.workspace = true
repository.workspace = true
[dependencies]
tsunagi = { path = "../tsunagi", version = "0.1.0" }
iroh.workspace = true
tokio.workspace = true
serde.workspace = true
postcard.workspace = true
bytes.workspace = true
thiserror.workspace = true
tracing.workspace = true
data-encoding.workspace = true
hex.workspace = true
rusqlite = { version = "0.40", features = ["bundled"] }
sha2 = "0.11"
hkdf = "0.13"
subtle = "2.6"
zeroize = { version = "1.9", features = ["derive"] }
rand = "0.10"
boringtun = { version = "0.7.1", default-features = false }
[dev-dependencies]
tsunagi = { path = "../tsunagi", features = ["testing"] }
tokio = { version = "1.53", features = ["rt", "rt-multi-thread", "sync", "time", "macros"] }
tempfile.workspace = true
tracing-subscriber.workspace = true
[lints]
workspace = true
+245
View File
@@ -0,0 +1,245 @@
//! What a WireGuard peer tells the network about itself.
//!
//! This is the opaque payload the control plane carries in a
//! [`tsunagi::dataplane::PluginCapability`]. The agent core never parses it —
//! only this module does, and only after bounding every field.
//!
//! The announcement is deliberately tiny: a participant says **who it is**,
//! not **where it is**. Reachability is the data plane transport's job, and
//! the transport already solves it — see
//! [`tsunagi::dataplane::transport`]. A plugin that also tried to advertise
//! addresses would be reimplementing NAT traversal badly.
use serde::{Deserialize, Serialize};
use tsunagi::dataplane::PluginError;
use tsunagi::identity::NetworkId;
use crate::keys::WgPublicKey;
/// Version of the announcement format.
///
/// Version 3 dropped the IPv4 range again: overlay addressing moved to the
/// signed records in [`tsunagi::state`], which carry the range and survive a
/// participant being away. postcard is not self-describing, so an older peer
/// cannot read a newer announcement; the mismatch is reported, not misparsed.
pub const ANNOUNCEMENT_VERSION: u16 = 4;
/// What one participant advertises for the WireGuard data plane.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct WgAnnouncement {
/// Announcement format version.
pub version: u16,
/// The peer's WireGuard public key.
///
/// The whole announcement, now that addresses belong to the system
/// level: this says *who* is at the other end of a tunnel, and nothing
/// about where.
pub public_key: [u8; 32],
/// The network this key is for.
///
/// Strictly redundant — a capability arrives on a session that already
/// proved membership of one network — and kept anyway, because the
/// binding used to be a side effect of checking a derived address and
/// losing it silently when that check went would be the wrong way to
/// lose it.
pub network: [u8; 32],
}
/// A peer announcement that has been validated against a specific network.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ValidatedAnnouncement {
/// The peer's WireGuard public key.
pub public_key: WgPublicKey,
}
impl WgAnnouncement {
/// Builds this agent's announcement.
pub fn new(network: NetworkId, public_key: &WgPublicKey) -> Self {
Self {
version: ANNOUNCEMENT_VERSION,
public_key: *public_key.as_bytes(),
network: *network.as_bytes(),
}
}
/// Encodes the announcement into the opaque capability payload.
pub fn encode(&self) -> Result<Vec<u8>, PluginError> {
postcard::to_stdvec(self)
.map_err(|err| PluginError::Other(format!("cannot encode announcement: {err}")))
}
/// Decodes and validates a payload received from a peer.
///
/// `network` and `local_key` scope the checks: an announcement is only
/// meaningful inside one network, and a peer must not claim our own key.
pub fn decode_and_validate(
payload: &[u8],
network: NetworkId,
local_key: &WgPublicKey,
) -> Result<ValidatedAnnouncement, PluginError> {
let announcement: Self = postcard::from_bytes(payload)
.map_err(|_| PluginError::Rejected("malformed WireGuard announcement".into()))?;
announcement.validate(network, local_key)
}
fn validate(
self,
network: NetworkId,
local_key: &WgPublicKey,
) -> Result<ValidatedAnnouncement, PluginError> {
if self.version != ANNOUNCEMENT_VERSION {
return Err(PluginError::Rejected(format!(
"peer speaks WireGuard announcement version {} but this build speaks \
{ANNOUNCEMENT_VERSION}; one of the two needs updating",
self.version
)));
}
let public_key = WgPublicKey::from_bytes(self.public_key);
if public_key.is_zero() {
return Err(PluginError::Rejected(
"WireGuard public key is all zeroes".into(),
));
}
if &public_key == local_key {
return Err(PluginError::Rejected(
"peer announced this agent's own WireGuard key".into(),
));
}
if self.network != *network.as_bytes() {
return Err(PluginError::Rejected(
"announcement is for a different network".into(),
));
}
Ok(ValidatedAnnouncement { public_key })
}
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
use tsunagi::identity::{NetworkKeys, NetworkName, NetworkSecret};
use super::super::keys::WgSecretKey;
fn network(name: &str) -> NetworkId {
NetworkKeys::derive(
&NetworkName::new(name).unwrap(),
&NetworkSecret::from_bytes(vec![3u8; 32]).unwrap(),
)
.network_id()
}
#[test]
fn a_well_formed_announcement_round_trips() {
let id = network("round-trip");
let peer = WgSecretKey::generate().public();
let local = WgSecretKey::generate().public();
let payload = WgAnnouncement::new(id, &peer).encode().unwrap();
let validated = WgAnnouncement::decode_and_validate(&payload, id, &local).unwrap();
assert_eq!(validated.public_key, peer);
}
#[test]
fn the_announcement_says_who_not_where() {
// Reachability belongs to the transport. Nothing address-like is
// carried here, so there is nothing for a peer to lie about.
let id = network("identity-only");
let peer = WgSecretKey::generate().public();
let payload = WgAnnouncement::new(id, &peer).encode().unwrap();
assert!(
payload.len() < 80,
"the announcement should stay tiny, got {} bytes",
payload.len()
);
}
#[test]
fn there_is_nothing_address_like_to_forge() {
// Addresses belong to the system level, are allocated there and are
// signed by the member that holds one. A protocol announcement
// carries no address at all, so this is not a thing a peer can lie
// about here — and a peer sending traffic from an address it does
// not hold is rejected by the agreed address, not by anything it
// said in this message.
let id = network("no-hijack");
let peer = WgSecretKey::generate().public();
let local = WgSecretKey::generate().public();
let announcement = WgAnnouncement::new(id, &peer);
let validated =
WgAnnouncement::decode_and_validate(&announcement.encode().unwrap(), id, &local)
.unwrap();
assert_eq!(validated.public_key, peer);
// The validated form has one field, and it is an identity.
assert_eq!(
std::mem::size_of_val(&validated),
std::mem::size_of::<WgPublicKey>()
);
}
#[test]
fn an_announcement_from_another_network_does_not_validate() {
let here = network("here");
let there = network("there");
let peer = WgSecretKey::generate().public();
let local = WgSecretKey::generate().public();
let payload = WgAnnouncement::new(there, &peer).encode().unwrap();
assert!(WgAnnouncement::decode_and_validate(&payload, here, &local).is_err());
}
#[test]
fn hostile_payloads_are_rejected_without_panicking() {
let id = network("hostile");
let local = WgSecretKey::generate().public();
let peer = WgSecretKey::generate().public();
assert!(WgAnnouncement::decode_and_validate(&[0xff; 64], id, &local).is_err());
assert!(WgAnnouncement::decode_and_validate(&[], id, &local).is_err());
let wrong_version = WgAnnouncement {
version: ANNOUNCEMENT_VERSION + 1,
..WgAnnouncement::new(id, &peer)
};
assert!(
WgAnnouncement::decode_and_validate(&wrong_version.encode().unwrap(), id, &local)
.is_err()
);
let zero_key = WgAnnouncement {
public_key: [0u8; 32],
..WgAnnouncement::new(id, &peer)
};
assert!(
WgAnnouncement::decode_and_validate(&zero_key.encode().unwrap(), id, &local).is_err()
);
}
#[test]
fn a_peer_cannot_claim_our_own_key() {
let id = network("self");
let local = WgSecretKey::generate().public();
let payload = WgAnnouncement::new(id, &local).encode().unwrap();
assert!(WgAnnouncement::decode_and_validate(&payload, id, &local).is_err());
}
#[test]
fn announcements_stay_well_under_the_capability_payload_limit() {
let id = network("size");
let peer = WgSecretKey::generate().public();
let payload = WgAnnouncement::new(id, &peer).encode().unwrap();
assert!(
payload.len() < tsunagi::config::Limits::default().max_capability_data_len,
"announcement is {} bytes",
payload.len()
);
}
}
+552
View File
@@ -0,0 +1,552 @@
//! Userspace WireGuard.
//!
//! The protocol itself is [`boringtun::noise::Tunn`], which is pure state
//! machine: no sockets, no TUN, no kernel module. That is what lets this work
//! the same way on any platform and be tested end to end without privileges.
//!
//! ```text
//! TunDevice (IP packets) PacketLink per peer
//! | |
//! v v
//! destination address -> peer --Tunn.encapsulate--> ciphertext
//! source address checked <--Tunn.decapsulate-- ciphertext
//! ```
//!
//! # Address ownership is enforced here
//!
//! Kernel WireGuard enforces `AllowedIPs`; in userspace we must do it
//! ourselves, and we do:
//!
//! * outbound, a packet is routed to the peer that **owns** its destination
//! address, where ownership is the signed claim the system level agreed;
//! * inbound, a decrypted packet is dropped unless its **source** is exactly
//! the address that peer holds.
//!
//! So a participant cannot receive traffic addressed to someone else, and
//! cannot forge traffic that appears to come from someone else. Neither
//! check consults anything the peer said here: an address is claimed at the
//! system level and signed by its holder, and that is what is compared
//! against.
use std::collections::HashMap;
use std::net::Ipv4Addr;
use std::sync::atomic::{AtomicU32, AtomicU64, Ordering};
use std::sync::{Arc, Mutex, RwLock};
use std::time::Duration;
use boringtun::noise::{Tunn, TunnResult};
use bytes::Bytes;
use iroh::EndpointId;
use tokio::task::JoinHandle;
use tsunagi::dataplane::transport::{SharedLink, TransportError};
use tsunagi::dataplane::{PacketSink, PluginError};
use tsunagi::identity::NetworkId;
use crate::keys::{WgPublicKey, WgSecretKey};
/// How often WireGuard's own timers are driven.
///
/// boringtun expects this at least every few hundred milliseconds; it is what
/// drives handshakes, rekeying and keepalives.
const TIMER_INTERVAL: Duration = Duration::from_millis(250);
/// Scratch space for one encapsulate or decapsulate call.
const SCRATCH: usize = 4096;
/// Counters for one peer's tunnel.
#[derive(Debug, Default)]
struct PeerCounters {
tx_packets: AtomicU64,
tx_bytes: AtomicU64,
rx_packets: AtomicU64,
rx_bytes: AtomicU64,
dropped_wrong_source: AtomicU64,
dropped_oversize: AtomicU64,
/// Packets there was no session to encrypt with yet.
dropped_no_session: AtomicU64,
protocol_errors: AtomicU64,
}
/// A snapshot of one peer's tunnel counters.
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub struct PeerStats {
/// Plaintext packets encrypted and sent to this peer.
pub tx_packets: u64,
/// Plaintext bytes encrypted and sent to this peer.
pub tx_bytes: u64,
/// Plaintext packets decrypted from this peer and given to the OS.
pub rx_packets: u64,
/// Plaintext bytes decrypted from this peer and given to the OS.
pub rx_bytes: u64,
/// Packets dropped because their source was not this peer's address.
///
/// A non-zero value means a peer tried to use an address it does not own.
pub dropped_wrong_source: u64,
/// Packets dropped because they did not fit in one link datagram.
pub dropped_oversize: u64,
/// Packets dropped because there was no session to encrypt with yet.
///
/// A handful while a tunnel comes up is normal; a number that keeps
/// climbing means the handshake is not completing.
pub dropped_no_session: u64,
/// WireGuard protocol errors, including packets that failed to decrypt.
pub protocol_errors: u64,
}
/// Whether a peer's tunnel has completed a handshake.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct PeerHealth {
/// Time since the last successful WireGuard handshake.
///
/// `None` means no handshake has completed yet, so the tunnel is not
/// carrying traffic. This is reported as it is, never guessed.
pub since_handshake: Option<Duration>,
}
impl PeerHealth {
/// Whether the tunnel has ever completed a handshake.
pub fn is_up(&self) -> bool {
self.since_handshake.is_some()
}
}
struct Peer {
endpoint_id: EndpointId,
public_key: WgPublicKey,
/// The overlay address this peer holds, as the control plane agreed it.
///
/// Not derived here and not taken from the peer: the system level
/// allocates it, the peer signs the claim, and every protocol carries
/// traffic for the same address.
overlay_v4: Mutex<Option<Ipv4Addr>>,
tunn: Mutex<Tunn>,
link: SharedLink,
counters: Arc<PeerCounters>,
task: Mutex<Option<JoinHandle<()>>>,
}
impl std::fmt::Debug for Peer {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("Peer")
.field("peer", &self.endpoint_id.fmt_short().to_string())
.field("public_key", &self.public_key)
.finish()
}
}
impl Drop for Peer {
fn drop(&mut self) {
if let Ok(mut guard) = self.task.lock()
&& let Some(task) = guard.take()
{
task.abort();
}
}
}
impl Peer {
fn stats(&self) -> PeerStats {
PeerStats {
tx_packets: self.counters.tx_packets.load(Ordering::Relaxed),
tx_bytes: self.counters.tx_bytes.load(Ordering::Relaxed),
rx_packets: self.counters.rx_packets.load(Ordering::Relaxed),
rx_bytes: self.counters.rx_bytes.load(Ordering::Relaxed),
dropped_wrong_source: self.counters.dropped_wrong_source.load(Ordering::Relaxed),
dropped_oversize: self.counters.dropped_oversize.load(Ordering::Relaxed),
dropped_no_session: self.counters.dropped_no_session.load(Ordering::Relaxed),
protocol_errors: self.counters.protocol_errors.load(Ordering::Relaxed),
}
}
fn health(&self) -> PeerHealth {
let guard = match self.tunn.lock() {
Ok(guard) => guard,
Err(poisoned) => poisoned.into_inner(),
};
PeerHealth {
since_handshake: guard.time_since_last_handshake(),
}
}
}
/// What a peer's tunnel looks like from outside.
#[derive(Debug, Clone)]
pub struct PeerSummary {
/// The peer's control plane identity.
pub endpoint_id: EndpointId,
/// The peer's WireGuard public key.
pub public_key: WgPublicKey,
/// The overlay address it holds, once the control plane has agreed one.
pub overlay_address_v4: Option<Ipv4Addr>,
/// Whether the tunnel has handshaken.
pub health: PeerHealth,
/// Traffic counters.
pub stats: PeerStats,
/// What the transport reports about the path carrying this tunnel.
pub path: String,
/// Largest datagram the link accepts.
pub max_datagram: usize,
}
struct Inner {
network: NetworkId,
private_key: WgSecretKey,
peers: RwLock<HashMap<WgPublicKey, Arc<Peer>>>,
/// Where decrypted packets go.
///
/// The interface belongs to the system level, so this hands a packet up
/// rather than writing it out: only that level knows which addresses the
/// sending member is entitled to use.
sink: Arc<dyn PacketSink>,
next_index: AtomicU32,
}
impl std::fmt::Debug for Inner {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("Inner")
.field("network", &self.network.fmt_short())
.finish()
}
}
/// The userspace WireGuard tunnels of one network.
#[derive(Debug)]
pub struct WireguardDevice {
inner: Arc<Inner>,
tasks: Vec<JoinHandle<()>>,
}
impl WireguardDevice {
/// Starts the tunnels for one network.
///
/// No interface is involved: packets arrive through [`Self::carry`] and
/// leave through the sink. Which address belongs to whom, and therefore
/// where a packet should go, is decided above this.
pub fn start(network: NetworkId, private_key: WgSecretKey, sink: Arc<dyn PacketSink>) -> Self {
let inner = Arc::new(Inner {
network,
private_key,
peers: RwLock::new(HashMap::new()),
sink,
next_index: AtomicU32::new(1),
});
let timers = tokio::spawn(drive_timers(Arc::clone(&inner)));
Self {
inner,
tasks: vec![timers],
}
}
/// Encrypts a packet and sends it to a peer.
///
/// `false` when there is no tunnel for that peer, which is a state the
/// caller reports rather than an error: a peer whose link has not come
/// up yet is normal.
pub fn carry(&self, peer: iroh::EndpointId, packet: &[u8]) -> bool {
let found = read_lock(&self.inner.peers)
.values()
.find(|candidate| candidate.endpoint_id == peer)
.map(Arc::clone);
let Some(peer) = found else {
return false;
};
let mut scratch = vec![0u8; SCRATCH];
// The encryption is the whole of what this protocol contributes, so
// it happens here rather than anywhere the packet passes through.
// The lock is released before the send: a slow link must not hold up
// the tunnel's timers.
let len = {
let mut tunn = match peer.tunn.lock() {
Ok(guard) => guard,
Err(poisoned) => poisoned.into_inner(),
};
match tunn.encapsulate(packet, &mut scratch) {
TunnResult::WriteToNetwork(out) => Some(out.len()),
// No session yet, so nothing to send. Counted as dropped
// rather than reported: the handshake is in flight and the
// next packet will go.
_ => None,
}
};
let Some(len) = len else {
peer.counters
.dropped_no_session
.fetch_add(1, Ordering::Relaxed);
return false;
};
send_to_peer(&peer, &scratch[..len]);
peer.counters.tx_packets.fetch_add(1, Ordering::Relaxed);
peer.counters
.tx_bytes
.fetch_add(packet.len() as u64, Ordering::Relaxed);
true
}
/// Adds or replaces a peer and starts its tunnel.
///
/// `overlay_v4` is decided by the caller, because only it knows whether
/// both sides agree on an IPv4 range. `None` means this peer is reachable
/// over IPv6 only.
pub fn add_peer(
&self,
endpoint_id: EndpointId,
public_key: WgPublicKey,
overlay_v4: Option<Ipv4Addr>,
link: SharedLink,
keepalive: Option<u16>,
) -> Result<(), PluginError> {
if public_key == self.inner.private_key.public() {
return Err(PluginError::Rejected(
"refusing to add ourselves as a WireGuard peer".into(),
));
}
let index = self.inner.next_index.fetch_add(1, Ordering::Relaxed);
let tunn = Tunn::new(
self.inner.private_key.to_static_secret(),
public_key.into_x25519(),
None,
keepalive,
index,
None,
);
let peer = Arc::new(Peer {
endpoint_id,
public_key,
overlay_v4: Mutex::new(overlay_v4),
tunn: Mutex::new(tunn),
link,
counters: Arc::new(PeerCounters::default()),
task: Mutex::new(None),
});
let task = tokio::spawn(read_from_link(Arc::clone(&self.inner), Arc::clone(&peer)));
if let Ok(mut guard) = peer.task.lock() {
*guard = Some(task);
}
write_lock(&self.inner.peers).insert(public_key, Arc::clone(&peer));
// Start the handshake now instead of waiting for the next timer tick,
// so the tunnel is usable as soon as the link exists.
kick_handshake(&peer);
Ok(())
}
/// Removes a peer and stops its tunnel.
pub fn remove_peer(&self, public_key: &WgPublicKey) {
// Dropping it stops the task and closes the link; there is no route
// to withdraw, because routes are not kept here.
write_lock(&self.inner.peers).remove(public_key);
}
/// Removes every peer whose key is not in `keep`.
pub fn retain_peers(&self, keep: &[WgPublicKey]) {
let stale: Vec<WgPublicKey> = read_lock(&self.inner.peers)
.keys()
.filter(|key| !keep.contains(key))
.copied()
.collect();
for key in stale {
self.remove_peer(&key);
}
}
/// Whether a peer's tunnel exists.
pub fn has_peer(&self, public_key: &WgPublicKey) -> bool {
read_lock(&self.inner.peers).contains_key(public_key)
}
/// A snapshot of every peer.
pub fn peers(&self) -> Vec<PeerSummary> {
let mut peers: Vec<PeerSummary> = read_lock(&self.inner.peers)
.values()
.map(|peer| {
let overlay_v4 = match peer.overlay_v4.lock() {
Ok(guard) => *guard,
Err(poisoned) => *poisoned.into_inner(),
};
PeerSummary {
endpoint_id: peer.endpoint_id,
public_key: peer.public_key,
overlay_address_v4: overlay_v4,
health: peer.health(),
stats: peer.stats(),
path: peer.link.path_description(),
max_datagram: peer.link.max_datagram_size(),
}
})
.collect();
peers.sort_by_key(|peer| peer.public_key);
peers
}
}
impl Drop for WireguardDevice {
fn drop(&mut self) {
for task in &self.tasks {
task.abort();
}
}
}
fn read_lock<T>(lock: &RwLock<T>) -> std::sync::RwLockReadGuard<'_, T> {
match lock.read() {
Ok(guard) => guard,
Err(poisoned) => poisoned.into_inner(),
}
}
fn write_lock<T>(lock: &RwLock<T>) -> std::sync::RwLockWriteGuard<'_, T> {
match lock.write() {
Ok(guard) => guard,
Err(poisoned) => poisoned.into_inner(),
}
}
/// Asks boringtun for a handshake initiation and sends it.
///
/// Encapsulating an empty packet is how the protocol state machine is told
/// "there is something to say"; with no session yet it answers with the
/// handshake initiation.
fn kick_handshake(peer: &Peer) {
let mut scratch = vec![0u8; SCRATCH];
let len = {
let mut tunn = match peer.tunn.lock() {
Ok(guard) => guard,
Err(poisoned) => poisoned.into_inner(),
};
match tunn.encapsulate(&[], &mut scratch) {
TunnResult::WriteToNetwork(out) => Some(out.len()),
_ => None,
}
};
if let Some(len) = len {
send_to_peer(peer, &scratch[..len]);
}
}
/// Sends whatever boringtun produced, without holding the tunnel lock.
fn send_to_peer(peer: &Peer, payload: &[u8]) {
match peer.link.send(Bytes::copy_from_slice(payload)) {
Ok(()) => {}
Err(TransportError::TooLarge { .. }) => {
peer.counters
.dropped_oversize
.fetch_add(1, Ordering::Relaxed);
}
Err(TransportError::Closed) => {}
Err(err) => {
tracing::trace!(%err, "dropping a WireGuard packet the link refused");
}
}
}
async fn read_from_link(inner: Arc<Inner>, peer: Arc<Peer>) {
loop {
let Some(datagram) = peer.link.recv().await else {
return;
};
let mut scratch = vec![0u8; SCRATCH];
// boringtun may need several passes: a handshake reply first, then
// any packets that were queued while the session was coming up.
let mut input: Option<&[u8]> = Some(&datagram);
loop {
let outcome = {
let mut tunn = match peer.tunn.lock() {
Ok(guard) => guard,
Err(poisoned) => poisoned.into_inner(),
};
match tunn.decapsulate(None, input.unwrap_or(&[]), &mut scratch) {
TunnResult::WriteToNetwork(out) => Outcome::ToNetwork(out.len()),
// The source boringtun reports is not consulted here:
// whether the peer may use it is checked where the
// claims live.
TunnResult::WriteToTunnelV6(out, _) => Outcome::ToTunnel(out.len()),
TunnResult::WriteToTunnelV4(out, _) => Outcome::ToTunnel(out.len()),
TunnResult::Done => Outcome::Done,
TunnResult::Err(err) => {
tracing::trace!(?err, "wireguard decapsulation failed");
Outcome::Failed
}
}
};
match outcome {
Outcome::ToNetwork(len) => {
send_to_peer(&peer, &scratch[..len]);
// Keep draining with an empty datagram, as boringtun asks.
input = None;
continue;
}
Outcome::ToTunnel(len) => {
// Handed up, not written out. This end has proved *who*
// sent the packet; whether that member may use the source
// address it chose is a question about a signed claim,
// and only the system level holds those.
let payload = Bytes::copy_from_slice(&scratch[..len]);
inner
.sink
.deliver(inner.network, peer.endpoint_id, payload)
.await;
peer.counters.rx_packets.fetch_add(1, Ordering::Relaxed);
peer.counters
.rx_bytes
.fetch_add(len as u64, Ordering::Relaxed);
break;
}
Outcome::Failed => {
peer.counters
.protocol_errors
.fetch_add(1, Ordering::Relaxed);
break;
}
Outcome::Done => break,
}
}
}
}
enum Outcome {
ToNetwork(usize),
ToTunnel(usize),
Done,
Failed,
}
/// Drives WireGuard's handshake, rekey and keepalive timers.
async fn drive_timers(inner: Arc<Inner>) {
let mut ticker = tokio::time::interval(TIMER_INTERVAL);
ticker.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay);
loop {
ticker.tick().await;
let peers: Vec<Arc<Peer>> = read_lock(&inner.peers).values().cloned().collect();
for peer in peers {
let mut scratch = vec![0u8; SCRATCH];
let len = {
let mut tunn = match peer.tunn.lock() {
Ok(guard) => guard,
Err(poisoned) => poisoned.into_inner(),
};
match tunn.update_timers(&mut scratch) {
TunnResult::WriteToNetwork(out) => Some(out.len()),
TunnResult::Err(err) => {
tracing::trace!(?err, "wireguard timer produced an error");
None
}
_ => None,
}
};
if let Some(len) = len {
send_to_peer(&peer, &scratch[..len]);
}
}
}
}
+224
View File
@@ -0,0 +1,224 @@
//! WireGuard key material.
//!
//! These keys belong to the plugin and to nothing else. They are **not**
//! derived from the iroh device key and **not** derived from the network
//! secret, so compromising or rotating one does not affect the others.
//!
//! Keys are X25519, encoded the way WireGuard encodes them: standard base64
//! with padding, 44 characters.
use boringtun::x25519;
use data_encoding::BASE64;
use zeroize::{Zeroize, Zeroizing};
use tsunagi::dataplane::PluginError;
/// Length of a raw WireGuard key, in bytes.
pub const KEY_LEN: usize = 32;
/// Length of the base64 text form of a key.
pub const KEY_TEXT_LEN: usize = 44;
/// Applies the X25519 clamping WireGuard applies to private keys.
///
/// `wg genkey` clamps, so clamping here keeps the printed private key and the
/// derived public key byte-identical to what the WireGuard tools produce.
fn clamp(bytes: &mut [u8; KEY_LEN]) {
bytes[0] &= 248;
bytes[31] &= 127;
bytes[31] |= 64;
}
/// A WireGuard public key.
///
/// Public, safe to log, and the identity a peer is known by inside the
/// overlay.
#[derive(Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub struct WgPublicKey([u8; KEY_LEN]);
impl WgPublicKey {
/// Wraps raw key bytes.
pub fn from_bytes(bytes: [u8; KEY_LEN]) -> Self {
Self(bytes)
}
/// The raw key bytes.
pub fn as_bytes(&self) -> &[u8; KEY_LEN] {
&self.0
}
/// The key in the form the WireGuard implementation expects.
pub(crate) fn into_x25519(self) -> x25519::PublicKey {
x25519::PublicKey::from(self.0)
}
/// Whether this is the all-zero key, which is never a valid peer.
pub fn is_zero(&self) -> bool {
self.0 == [0u8; KEY_LEN]
}
/// The base64 form WireGuard uses.
pub fn encode(&self) -> String {
BASE64.encode(&self.0)
}
/// Parses the base64 form WireGuard uses.
pub fn decode(text: &str) -> Result<Self, PluginError> {
if text.len() != KEY_TEXT_LEN {
return Err(PluginError::Rejected(format!(
"a WireGuard key is {KEY_TEXT_LEN} base64 characters, got {}",
text.len()
)));
}
let raw = BASE64
.decode(text.as_bytes())
.map_err(|_| PluginError::Rejected("key is not valid base64".into()))?;
let bytes = <[u8; KEY_LEN]>::try_from(raw.as_slice())
.map_err(|_| PluginError::Rejected("key is not 32 bytes".into()))?;
Ok(Self(bytes))
}
/// A short prefix for logs and diagnostics.
pub fn fmt_short(&self) -> String {
self.encode().chars().take(8).collect()
}
}
impl std::fmt::Display for WgPublicKey {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str(&self.encode())
}
}
impl std::fmt::Debug for WgPublicKey {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "WgPublicKey({})", self.fmt_short())
}
}
/// A WireGuard private key.
///
/// Zeroized on drop and redacted from [`Debug`]. Its base64 form is only ever
/// produced for the configuration handed to the WireGuard backend, and that
/// value is itself zeroized.
#[derive(Clone)]
pub struct WgSecretKey(Zeroizing<[u8; KEY_LEN]>);
impl WgSecretKey {
/// Generates a fresh clamped private key.
pub fn generate() -> Self {
let mut bytes = Zeroizing::new([0u8; KEY_LEN]);
rand::fill(bytes.as_mut());
clamp(&mut bytes);
Self(bytes)
}
/// Wraps stored key bytes, clamping them.
pub fn from_bytes(bytes: &[u8; KEY_LEN]) -> Self {
let mut owned = Zeroizing::new(*bytes);
clamp(&mut owned);
Self(owned)
}
/// The raw key bytes, for persistence only.
pub(crate) fn expose(&self) -> &[u8; KEY_LEN] {
&self.0
}
/// The matching public key.
pub fn public(&self) -> WgPublicKey {
WgPublicKey(x25519::PublicKey::from(&self.to_static_secret()).to_bytes())
}
/// The key in the form the WireGuard implementation expects.
pub(crate) fn to_static_secret(&self) -> x25519::StaticSecret {
x25519::StaticSecret::from(*self.0)
}
/// The base64 form, for the WireGuard configuration. Zeroized on drop.
pub fn encode(&self) -> Zeroizing<String> {
let mut encoded = BASE64.encode(self.0.as_ref());
let out = Zeroizing::new(encoded.clone());
encoded.zeroize();
out
}
}
impl std::fmt::Debug for WgSecretKey {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("WgSecretKey")
.field("public", &self.public().fmt_short())
.field("secret", &"<redacted>")
.finish()
}
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
#[test]
fn generated_keys_are_clamped_and_round_trip() {
let secret = WgSecretKey::generate();
let raw = *secret.expose();
assert_eq!(raw[0] & 7, 0, "low three bits must be cleared");
assert_eq!(raw[31] & 128, 0, "top bit must be cleared");
assert_eq!(raw[31] & 64, 64, "second-highest bit must be set");
let text = secret.encode();
assert_eq!(text.len(), KEY_TEXT_LEN);
let public = secret.public();
let parsed = WgPublicKey::decode(&public.encode()).unwrap();
assert_eq!(parsed, public);
}
#[test]
fn reloading_a_stored_key_gives_the_same_public_key() {
let secret = WgSecretKey::generate();
let reloaded = WgSecretKey::from_bytes(secret.expose());
assert_eq!(secret.public(), reloaded.public());
}
#[test]
fn the_rfc_7748_test_vector_derives_the_expected_public_key() {
// RFC 7748 section 6.1. WireGuard keys are plain X25519 keys, and
// X25519 clamps internally, so storing the clamped form must not
// change the derived public key.
let private =
hex_to_key("77076d0a7318a57d3c16c17251b26645df4c2f87ebc0992ab177fba51db92c2a");
let expected =
hex_to_key("8520f0098930a754748b7ddcb43ef75a0dbf3a0d26381af4eba4a98eaa9b4e6a");
let secret = WgSecretKey::from_bytes(&private);
assert_eq!(secret.public(), WgPublicKey::from_bytes(expected));
assert_eq!(
secret.public().encode(),
"hSDwCYkwp1R0i33ctD73Wg2/Og0mOBr066SpjqqbTmo="
);
}
fn hex_to_key(text: &str) -> [u8; KEY_LEN] {
let raw = hex::decode(text).unwrap();
<[u8; KEY_LEN]>::try_from(raw.as_slice()).unwrap()
}
#[test]
fn malformed_keys_are_rejected_without_panicking() {
assert!(WgPublicKey::decode("").is_err());
assert!(WgPublicKey::decode("not base64 at all!!!").is_err());
assert!(WgPublicKey::decode(&"A".repeat(KEY_TEXT_LEN)).is_err());
assert!(WgPublicKey::decode(&BASE64.encode(&[0u8; 16])).is_err());
assert!(WgPublicKey::from_bytes([0u8; KEY_LEN]).is_zero());
}
#[test]
fn secrets_are_redacted_in_debug_output() {
let secret = WgSecretKey::generate();
let rendered = format!("{secret:?}");
assert!(rendered.contains("<redacted>"));
assert!(!rendered.contains(secret.encode().as_str()));
}
}
+50
View File
@@ -0,0 +1,50 @@
//! The `wg-quic` protocol: WireGuard's cryptography in iroh's QUIC datagrams.
//!
//! A crate of its own, so the line between a protocol and the system level
//! is drawn by the compiler rather than by discipline: nothing here can
//! reach into `tsunagi` beyond what it makes public. It also carries its own
//! version, which is **not** the version peers compare — that is
//! [`ANNOUNCEMENT_VERSION`], and it moves only when the bytes on the wire
//! do, so two peers on different releases keep working.
//!
//! That it uses iroh is a convenience, not a requirement of the design: the
//! system level already has an iroh endpoint that crosses NAT, and plain
//! WireGuard is blocked on some networks where this is not.
//!
//! Where the line falls:
//!
//! * **It knows nothing about reachability.** It is handed a
//! [`PacketLink`](tsunagi::dataplane::transport::PacketLink) per peer and
//! moves datagrams over it. Hole punching and relaying are the transport's.
//! * **It knows nothing about addresses, and owns no interface.** One agent
//! has one interface at the system level, and every protocol carries
//! traffic for the same addresses on it. A packet arrives here already
//! routed and leaves here already decrypted, to be checked against the
//! signed claim by the level that holds those.
//! * **Its announcement says who, not where.** A public key and the network
//! it is for, so there is nothing about reachability to lie about.
//! * **The core never parses that announcement.** It moves a bounded opaque
//! blob; only [`announcement`] reads it.
//! * **Its keys are its own.** One per network, in its own store, unrelated
//! to the iroh device key and to the network secret — which it never sees.
//! * **WireGuard's own crypto is untouched.** The handshake and encryption
//! run end to end between the two ends of a tunnel, via [`boringtun`]'s
//! state machine in this process: no kernel module, no `wg` tool, the same
//! code on every platform.
//!
//! See `docs/wireguard.md` for the full picture.
pub mod announcement;
pub mod device;
pub mod keys;
pub mod plugin;
pub mod store;
pub use announcement::{ANNOUNCEMENT_VERSION, ValidatedAnnouncement, WgAnnouncement};
pub use device::{PeerHealth, PeerStats, PeerSummary, WireguardDevice};
pub use keys::{WgPublicKey, WgSecretKey};
pub use plugin::{
DEFAULT_MTU, MIN_MTU, NetworkOverview, PeerOverview, WIREGUARD_OVERHEAD, WIREGUARD_PROTOCOL,
WireguardConfig, WireguardPlugin,
};
pub use store::WgKeyStore;
+816
View File
@@ -0,0 +1,816 @@
//! The WireGuard protocol plugin.
//!
//! Each agent builds its own view of the overlay from the set of participants
//! the control plane agreed on. For a full mesh of `N` members that is `N - 1`
//! tunnels locally. Nobody is handed a configuration by anybody else, and no
//! participant is authoritative.
//!
//! # What this plugin does and does not know
//!
//! * It does **not** know where a peer is. It is handed a
//! [`PacketLink`](tsunagi::dataplane::transport::PacketLink) per peer and runs
//! a WireGuard tunnel over it. Reachability, hole punching and relaying are
//! the transport's problem.
//! * It does **not** know which addresses anybody holds, and owns no
//! interface. One agent has one interface, at the system level, and every
//! protocol carries traffic for the same addresses on it. A packet arrives
//! here already routed and leaves here already decrypted.
//! * It owns one WireGuard key per network, in its own store, unrelated to the
//! iroh device key and to the network secret.
//!
//! WireGuard runs in userspace via [`boringtun`], so there is no kernel module
//! and no `wg` tool to depend on, and nothing this plugin does needs
//! privileges: creating the interface is somebody else's job now.
//!
//! A failure here is reported and retried. It never stops the control plane.
use std::collections::{BTreeSet, HashMap};
use std::net::Ipv4Addr;
use std::path::PathBuf;
use std::sync::{Arc, Mutex, OnceLock};
use std::time::Duration;
use iroh::EndpointId;
use tokio::sync::{mpsc, oneshot};
use tokio::task::JoinHandle;
use tsunagi::BoxFuture;
use tsunagi::dataplane::PacketSink;
use tsunagi::dataplane::transport::SharedLink;
use tsunagi::dataplane::{IpPlugin, PluginCapability, PluginContext, PluginError};
use tsunagi::identity::NetworkId;
use crate::announcement::{ValidatedAnnouncement, WgAnnouncement};
use crate::device::{PeerSummary, WireguardDevice};
use crate::keys::{WgPublicKey, WgSecretKey};
use crate::store::WgKeyStore;
use tsunagi::state::Ipv4Range;
/// The protocol identifier this plugin announces.
/// The protocol id of this plugin.
///
/// `wg-quic`, because that is what it is: WireGuard's cryptography carried
/// in QUIC datagrams. The name is on the wire, so it is a protocol name and
/// not a description of the implementation.
pub const WIREGUARD_PROTOCOL: &str = "wg-quic";
/// Smallest interface MTU the overlay accepts.
///
/// 576 bytes is what IPv4 guarantees every host can reassemble (RFC 1122),
/// so nothing below it is worth offering. The floor used to be 1280 because
/// Linux tears IPv6 down on an interface below that; the overlay is IPv4
/// now, so that constraint is gone and a path with small datagrams — a
/// relay, typically — can be matched instead of warned about.
pub const MIN_MTU: u32 = 576;
/// Default interface MTU.
///
/// Comfortably under what a direct path carries, and the same number the
/// overlay used before, so an existing network does not have to change.
pub const DEFAULT_MTU: u32 = 1280;
/// Bytes WireGuard adds to a packet: type and reserved, receiver index,
/// counter and the Poly1305 tag.
///
/// A link therefore has to carry `mtu + WIREGUARD_OVERHEAD` bytes in one
/// datagram for a full-size packet to get through.
pub const WIREGUARD_OVERHEAD: u32 = 32;
/// Configuration of the WireGuard plugin.
#[derive(Debug, Clone)]
pub struct WireguardConfig {
/// Directory for the plugin's own key store. Separate from agent state.
pub state_dir: PathBuf,
/// WireGuard keepalive, which keeps tunnels and their links warm.
pub keepalive: Option<u16>,
/// The largest packet a tunnel will carry.
///
/// Not the interface MTU, which belongs to the agent: this is what this
/// protocol refuses to encrypt because it would not fit one datagram.
pub mtu: u32,
/// How long to coalesce changes before reconciling.
pub reconcile_debounce: Duration,
/// How often to reconcile anyway, which is also when a packet interface
/// that could not be created before is retried.
pub reconcile_interval: Duration,
}
impl WireguardConfig {
/// Creates a configuration rooted at `state_dir` with sensible defaults.
pub fn new(state_dir: impl Into<PathBuf>) -> Self {
Self {
state_dir: state_dir.into(),
keepalive: Some(25),
mtu: DEFAULT_MTU,
reconcile_debounce: Duration::from_millis(200),
reconcile_interval: Duration::from_secs(15),
}
}
/// Sets the interface MTU.
///
/// Validated when the plugin is opened; see [`MIN_MTU`].
pub fn with_mtu(mut self, mtu: u32) -> Self {
self.mtu = mtu;
self
}
/// Sets the reconciliation timings.
pub fn with_reconcile(mut self, debounce: Duration, interval: Duration) -> Self {
self.reconcile_debounce = debounce;
self.reconcile_interval = interval;
self
}
/// Path of the plugin's key store.
pub fn key_store_path(&self) -> PathBuf {
self.state_dir.join("wireguard.sqlite")
}
}
/// What this agent has set up for one network.
#[derive(Debug, Clone)]
pub struct NetworkOverview {
/// The network.
pub network: NetworkId,
/// The largest packet a tunnel in this network will carry.
pub mtu: u32,
/// This agent's WireGuard public key in this network.
pub public_key: WgPublicKey,
/// This agent's overlay address, once the network has agreed one.
pub overlay_address_v4: Option<Ipv4Addr>,
/// The IPv4 overlay range in use.
pub ipv4_range: Option<Ipv4Range>,
/// Peers this agent knows about.
pub peers: Vec<PeerOverview>,
}
impl NetworkOverview {
/// Peers whose tunnel has completed a handshake.
pub fn established_peers(&self) -> usize {
self.peers.iter().filter(|peer| peer.is_up()).count()
}
}
/// One peer of the overlay.
#[derive(Debug, Clone)]
pub struct PeerOverview {
/// The peer's control plane identity.
pub endpoint_id: EndpointId,
/// The peer's WireGuard public key.
pub public_key: WgPublicKey,
/// The overlay address the network agreed it holds.
pub overlay_address_v4: Option<Ipv4Addr>,
/// Whether a data plane link to it exists.
pub has_link: bool,
/// The running tunnel, once there is a link.
pub tunnel: Option<PeerSummary>,
}
impl PeerOverview {
/// Whether the tunnel to this peer has handshaken and can carry traffic.
pub fn is_up(&self) -> bool {
self.tunnel
.as_ref()
.is_some_and(|tunnel| tunnel.health.is_up())
}
}
#[derive(Debug)]
struct NetworkState {
key: WgSecretKey,
device: Option<Arc<WireguardDevice>>,
announcements: HashMap<EndpointId, ValidatedAnnouncement>,
links: HashMap<EndpointId, SharedLink>,
/// What the network agreed, pushed in by the agent. Authoritative.
allocations: HashMap<EndpointId, Ipv4Addr>,
/// The range those allocations came from.
ipv4_range: Option<Ipv4Range>,
}
#[derive(Debug, Default)]
struct Shared {
networks: HashMap<NetworkId, NetworkState>,
}
#[derive(Debug)]
enum Command {
Prepare(NetworkId),
Sync(NetworkId),
Link {
network: NetworkId,
peer: EndpointId,
link: SharedLink,
},
Teardown(NetworkId),
Stop(oneshot::Sender<()>),
}
struct Worker {
config: WireguardConfig,
/// This agent's endpoint id, learned when the plugin is attached.
local_id: OnceLock<EndpointId>,
store: WgKeyStore,
shared: Mutex<Shared>,
context: OnceLock<PluginContext>,
/// Where decrypted packets go, once the agent has attached one.
sink: OnceLock<Arc<dyn PacketSink>>,
}
impl std::fmt::Debug for Worker {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("Worker")
.field("store", &self.store.path())
.finish()
}
}
/// The WireGuard data plane plugin.
#[derive(Debug)]
pub struct WireguardPlugin {
worker: Arc<Worker>,
commands: mpsc::Sender<Command>,
task: Mutex<Option<JoinHandle<()>>>,
}
impl WireguardPlugin {
/// The settings this protocol accepts.
pub const OPTIONS: &'static [tsunagi::dataplane::ProtocolOption] = &[
tsunagi::dataplane::ProtocolOption {
key: "keepalive",
value: "SECONDS",
help: "keeps a tunnel and its link warm through a NAT; 0 turns it off",
default: Some("25"),
},
tsunagi::dataplane::ProtocolOption {
key: "mtu",
value: "BYTES",
help: "largest packet a tunnel will carry, at least 576",
default: Some("1280"),
},
];
/// Applies `key=value` settings to a configuration.
///
/// An unknown key is refused rather than ignored: a setting that was
/// silently dropped looks exactly like one that did not work.
pub fn configure(
mut config: WireguardConfig,
options: &[(String, String)],
) -> Result<WireguardConfig, PluginError> {
for (key, value) in options {
match key.as_str() {
"keepalive" => {
let seconds: u16 = value.parse().map_err(|_| {
PluginError::Other(format!("keepalive={value} is not a number of seconds"))
})?;
config.keepalive = (seconds > 0).then_some(seconds);
}
"mtu" => {
let mtu: u32 = value.parse().map_err(|_| {
PluginError::Other(format!("mtu={value} is not a number of bytes"))
})?;
config = config.with_mtu(mtu);
}
other => {
let known: Vec<&str> = Self::OPTIONS.iter().map(|spec| spec.key).collect();
return Err(PluginError::Other(format!(
"`{other}` is not a setting of {WIREGUARD_PROTOCOL}; it takes {}",
known.join(", ")
)));
}
}
}
Ok(config)
}
/// Opens the plugin's key store and starts its reconciliation task.
///
/// Must be called from inside a tokio runtime; the plugin starts no
/// runtime of its own.
pub async fn open(config: WireguardConfig) -> Result<Arc<Self>, PluginError> {
if config.mtu < MIN_MTU {
return Err(PluginError::Other(format!(
"an MTU of {} is below the {MIN_MTU} bytes every IPv4 host must be able \
to reassemble (RFC 1122)",
config.mtu
)));
}
let path = config.key_store_path();
let store = tokio::task::spawn_blocking(move || WgKeyStore::open(path))
.await
.map_err(|err| PluginError::Other(format!("key store task failed: {err}")))??;
let worker = Arc::new(Worker {
config,
local_id: OnceLock::new(),
store,
shared: Mutex::new(Shared::default()),
context: OnceLock::new(),
sink: OnceLock::new(),
});
let (commands, receiver) = mpsc::channel(64);
let task = tokio::spawn(run(Arc::clone(&worker), receiver));
Ok(Arc::new(Self {
worker,
commands,
task: Mutex::new(Some(task)),
}))
}
/// What this agent has set up for a network, if anything yet.
pub fn overview(&self, network: NetworkId) -> Option<NetworkOverview> {
let shared = self.worker.lock_shared();
let state = shared.networks.get(&network)?;
let tunnels: HashMap<WgPublicKey, PeerSummary> = state
.device
.as_ref()
.map(|device| {
device
.peers()
.into_iter()
.map(|summary| (summary.public_key, summary))
.collect()
})
.unwrap_or_default();
let mut peers: Vec<PeerOverview> = state
.announcements
.iter()
.map(|(endpoint_id, announcement)| PeerOverview {
endpoint_id: *endpoint_id,
public_key: announcement.public_key,
overlay_address_v4: state.allocations.get(endpoint_id).copied(),
has_link: state.links.contains_key(endpoint_id),
tunnel: tunnels.get(&announcement.public_key).cloned(),
})
.collect();
peers.sort_by_key(|peer| peer.public_key);
Some(NetworkOverview {
network,
mtu: self.worker.config.mtu,
public_key: state.key.public(),
overlay_address_v4: state.allocations.get(&self.worker.local_id()).copied(),
ipv4_range: state.ipv4_range,
peers,
})
}
/// Asks the reconciliation task to run now.
pub async fn reconcile_now(&self, network: NetworkId) {
let _ = self.commands.send(Command::Sync(network)).await;
}
fn nudge(&self, command: Command) {
if let Err(err) = self.commands.try_send(command) {
// A full queue means work is already scheduled; the periodic
// reconcile picks up anything that was missed.
tracing::debug!(%err, "wireguard command queue is busy");
}
}
}
impl Worker {
/// This agent's endpoint id, or a placeholder before it is attached.
fn local_id(&self) -> EndpointId {
self.local_id.get().copied().unwrap_or_else(|| {
EndpointId::from_bytes(&[1u8; 32]).unwrap_or_else(|_| unreachable!("a fixed valid key"))
})
}
fn lock_shared(&self) -> std::sync::MutexGuard<'_, Shared> {
match self.shared.lock() {
Ok(guard) => guard,
Err(poisoned) => poisoned.into_inner(),
}
}
fn report(&self, network: NetworkId, reason: impl std::fmt::Display) {
tracing::warn!(network = %network.fmt_short(), %reason, "wireguard plugin error");
if let Some(context) = self.context.get() {
context.report_error(network, WIREGUARD_PROTOCOL, reason.to_string());
}
}
fn request_reannounce(&self, network: NetworkId) {
if let Some(context) = self.context.get() {
context.request_reannounce(network);
}
}
/// Makes sure a network has a key, a name and a running packet interface.
///
/// Returns `true` when the key became available now, so peers should be
/// told. Creating the interface may fail without privileges; the key and
/// the announcement still work, and the interface is retried.
async fn prepare(self: &Arc<Self>, network: NetworkId) -> Result<bool, PluginError> {
let existing = {
let shared = self.lock_shared();
shared
.networks
.get(&network)
.map(|state| state.device.is_some())
};
if let Some(has_device) = existing {
if has_device {
return Ok(false);
}
// The key is there but the interface is not. Try again.
self.ensure_device(network).await?;
return Ok(false);
}
let worker = Arc::clone(self);
let key = tokio::task::spawn_blocking(move || worker.store.load_or_create(network))
.await
.map_err(|err| PluginError::Other(format!("key store task failed: {err}")))??;
{
let mut shared = self.lock_shared();
shared.networks.entry(network).or_insert(NetworkState {
key,
device: None,
announcements: HashMap::new(),
links: HashMap::new(),
allocations: HashMap::new(),
ipv4_range: None,
});
}
// The announcement only needs the key, so peers can be told even if
// the interface is not up yet.
let device = self.ensure_device(network).await;
if let Err(err) = device {
self.report(network, err);
}
Ok(true)
}
/// Starts this network's tunnels.
///
/// No interface is created: one agent has one, it belongs to the system
/// level, and decrypted packets are handed there rather than written
/// out.
async fn ensure_device(&self, network: NetworkId) -> Result<(), PluginError> {
let key = {
let shared = self.lock_shared();
match shared.networks.get(&network) {
Some(state) if state.device.is_none() => state.key.clone(),
_ => return Ok(()),
}
};
let sink = match self.sink.get() {
Some(sink) => Arc::clone(sink),
// Not attached to an agent: the protocol still runs, and its
// packets have nowhere to go.
None => Arc::new(tsunagi::dataplane::DiscardPackets) as Arc<dyn PacketSink>,
};
let device = Arc::new(WireguardDevice::start(network, key, sink));
let mut shared = self.lock_shared();
if let Some(state) = shared.networks.get_mut(&network) {
state.device = Some(device);
}
Ok(())
}
/// Brings the running tunnels in line with what is known.
///
/// A peer gets a tunnel once both halves have arrived: its announcement,
/// which says who it is, and a link, which says packets can reach it.
fn sync(&self, network: NetworkId) {
let mut shared = self.lock_shared();
let Some(state) = shared.networks.get_mut(&network) else {
return;
};
let Some(device) = state.device.clone() else {
return;
};
let allocations = state.allocations.clone();
let mut wanted: Vec<WgPublicKey> = Vec::new();
let mut too_small: Vec<(usize, usize)> = Vec::new();
for (endpoint_id, announcement) in &state.announcements {
let Some(link) = state.links.get(endpoint_id) else {
continue;
};
if link.is_closed() {
continue;
}
wanted.push(announcement.public_key);
if device.has_peer(&announcement.public_key) {
continue;
}
// A link that cannot carry a full-size packet will silently drop
// the large ones, which looks like a broken network rather than a
// configuration problem. Say so when the tunnel is set up.
let needed = self.config.mtu.saturating_add(WIREGUARD_OVERHEAD) as usize;
let available = link.max_datagram_size();
if available < needed {
too_small.push((available, needed));
}
// The address comes from the agreed signed state, not from
// anything this peer said and not from a derivation: that is what
// makes it survive the peer being away.
let peer_v4 = allocations.get(endpoint_id).copied();
if let Err(err) = device.add_peer(
*endpoint_id,
announcement.public_key,
peer_v4,
Arc::clone(link),
self.config.keepalive,
) {
tracing::debug!(%err, "cannot start a WireGuard tunnel");
}
}
device.retain_peers(&wanted);
drop(shared);
for (available, needed) in too_small {
self.report(
network,
format!(
"this path carries only {available} byte datagrams but a {} byte MTU needs \
{needed}; packets larger than {} bytes will be dropped. The floor is \
{MIN_MTU} bytes, what every IPv4 host must be able to reassemble.",
self.config.mtu,
available.saturating_sub(WIREGUARD_OVERHEAD as usize)
),
);
}
}
/// Removes a network's interface and tunnels, keeping its key.
async fn teardown(&self, network: NetworkId) {
// Dropping the state drops the device, which stops its tasks and
// closes the packet interface. Closing it is already enough for the
// kernel to remove an interface this agent created; the explicit
// destroy makes that immediate and definite rather than dependent on
// the last reader letting go.
// Dropping the state drops the tunnels, which stops their tasks and
// closes their links. There is no interface to remove: the agent owns
// it, and it outlives any one network.
self.lock_shared().networks.remove(&network);
}
fn known_networks(&self) -> Vec<NetworkId> {
self.lock_shared().networks.keys().copied().collect()
}
}
/// The reconciliation task.
async fn run(worker: Arc<Worker>, mut commands: mpsc::Receiver<Command>) {
let mut pending: BTreeSet<NetworkId> = BTreeSet::new();
let mut deadline: Option<tokio::time::Instant> = None;
let mut ticker = tokio::time::interval(worker.config.reconcile_interval);
ticker.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay);
ticker.tick().await;
loop {
let wait_until = deadline;
tokio::select! {
biased;
command = commands.recv() => {
let Some(command) = command else { break };
match command {
Command::Prepare(network) => {
match worker.prepare(network).await {
Ok(true) => worker.request_reannounce(network),
Ok(false) => {}
Err(err) => worker.report(network, err),
}
pending.insert(network);
}
Command::Sync(network) => {
pending.insert(network);
}
Command::Link { network, peer, link } => {
{
let mut shared = worker.lock_shared();
if let Some(state) = shared.networks.get_mut(&network) {
state.links.insert(peer, link);
}
}
pending.insert(network);
}
Command::Teardown(network) => {
pending.remove(&network);
worker.teardown(network).await;
continue;
}
Command::Stop(reply) => {
for network in worker.known_networks() {
worker.teardown(network).await;
}
let _ = reply.send(());
return;
}
}
deadline = Some(tokio::time::Instant::now() + worker.config.reconcile_debounce);
}
_ = async {
match wait_until {
Some(at) => tokio::time::sleep_until(at).await,
None => std::future::pending::<()>().await,
}
}, if wait_until.is_some() => {
deadline = None;
for network in std::mem::take(&mut pending) {
worker.sync(network);
}
}
_ = ticker.tick() => {
for network in worker.known_networks() {
// Also the retry for an interface that could not be
// created earlier.
if let Err(err) = worker.ensure_device(network).await {
tracing::debug!(%err, "packet interface still unavailable");
}
worker.sync(network);
}
}
}
}
}
impl IpPlugin for WireguardPlugin {
fn protocol_id(&self) -> &str {
WIREGUARD_PROTOCOL
}
fn protocol_version(&self) -> u16 {
crate::announcement::ANNOUNCEMENT_VERSION
}
fn options(&self) -> &'static [tsunagi::dataplane::ProtocolOption] {
Self::OPTIONS
}
fn attach(&self, context: PluginContext) {
if let Some(local) = context.local_endpoint_id() {
let _ = self.worker.local_id.set(local);
}
let _ = self.worker.sink.set(context.packet_sink());
let _ = self.worker.context.set(context);
}
fn on_network_activated(&self, network: NetworkId) {
self.nudge(Command::Prepare(network));
}
fn local_capability(
&self,
network: NetworkId,
) -> Result<Option<PluginCapability>, PluginError> {
let shared = self.worker.lock_shared();
let Some(state) = shared.networks.get(&network) else {
// Not ready yet. Ask for preparation; once the key exists the
// plugin asks the agent to re-announce.
drop(shared);
self.nudge(Command::Prepare(network));
return Ok(None);
};
// Identity only. Where to send packets is the transport's business.
let announcement = WgAnnouncement::new(network, &state.key.public());
Ok(Some(PluginCapability {
protocol: WIREGUARD_PROTOCOL.to_string(),
version: crate::announcement::ANNOUNCEMENT_VERSION,
enabled: true,
data: announcement.encode()?,
}))
}
fn on_peer_capability(
&self,
network: NetworkId,
peer: EndpointId,
capability: &PluginCapability,
) -> Result<(), PluginError> {
let local_key = {
let shared = self.worker.lock_shared();
match shared.networks.get(&network) {
Some(state) => state.key.public(),
None => {
drop(shared);
self.nudge(Command::Prepare(network));
return Err(PluginError::Unavailable(
"WireGuard is not ready for this network yet".into(),
));
}
}
};
let validated = WgAnnouncement::decode_and_validate(&capability.data, network, &local_key)?;
let changed = {
let mut shared = self.worker.lock_shared();
match shared.networks.get_mut(&network) {
Some(state) => {
state.announcements.insert(peer, validated.clone()) != Some(validated)
}
None => false,
}
};
if changed {
self.nudge(Command::Sync(network));
}
Ok(())
}
fn on_address_allocation(
&self,
network: NetworkId,
range: Ipv4Range,
allocations: &[(EndpointId, Ipv4Addr)],
) {
let changed = {
let mut shared = self.worker.lock_shared();
match shared.networks.get_mut(&network) {
Some(state) => {
let fresh: HashMap<EndpointId, Ipv4Addr> =
allocations.iter().copied().collect();
let changed = state.allocations != fresh || state.ipv4_range != Some(range);
state.allocations = fresh;
state.ipv4_range = Some(range);
changed
}
None => false,
}
};
if changed {
self.nudge(Command::Sync(network));
}
}
fn carry(&self, network: NetworkId, peer: EndpointId, packet: bytes::Bytes) -> bool {
let shared = self.worker.lock_shared();
shared
.networks
.get(&network)
.and_then(|state| state.device.as_ref())
.is_some_and(|device| device.carry(peer, &packet))
}
fn on_peer_link(&self, network: NetworkId, peer: EndpointId, link: SharedLink) {
self.nudge(Command::Link {
network,
peer,
link,
});
}
fn on_peer_gone(&self, network: NetworkId, peer: EndpointId) {
let removed = {
let mut shared = self.worker.lock_shared();
match shared.networks.get_mut(&network) {
Some(state) => {
let had_link = state.links.remove(&peer).is_some();
state.announcements.remove(&peer).is_some() || had_link
}
None => false,
}
};
if removed {
self.nudge(Command::Sync(network));
}
}
fn on_network_deactivated(&self, network: NetworkId) {
self.nudge(Command::Teardown(network));
}
fn shutdown<'a>(&'a self) -> BoxFuture<'a, ()> {
Box::pin(async move {
let (reply_tx, reply_rx) = oneshot::channel();
if self.commands.send(Command::Stop(reply_tx)).await.is_ok() {
let _ = reply_rx.await;
}
let task = self.task.lock().ok().and_then(|mut guard| guard.take());
if let Some(task) = task {
let _ = task.await;
}
})
}
}
impl Drop for WireguardPlugin {
fn drop(&mut self) {
if let Ok(mut guard) = self.task.lock()
&& let Some(task) = guard.take()
{
task.abort();
}
}
}
+222
View File
@@ -0,0 +1,222 @@
//! The plugin's own key store.
//!
//! Deliberately a separate SQLite file from the agent's `state.sqlite`: plugin
//! keys are not the iroh identity and not the network secret, and their
//! lifecycle is the plugin's business alone.
//!
//! One key per network, so a participant presents a different WireGuard
//! identity — and therefore a different overlay address — in each network it
//! belongs to.
//!
//! A damaged key store is an error, never a silent regeneration: a new key
//! would silently move this agent to a different overlay address and orphan
//! every peer's configuration.
use std::path::{Path, PathBuf};
use std::sync::Mutex;
use rusqlite::{Connection, OptionalExtension, params};
use tsunagi::dataplane::PluginError;
use tsunagi::identity::NetworkId;
use crate::keys::{KEY_LEN, WgSecretKey};
/// Schema version written by this build.
pub const SCHEMA_VERSION: i64 = 1;
/// Per-network WireGuard private keys.
#[derive(Debug)]
pub struct WgKeyStore {
conn: Mutex<Connection>,
path: PathBuf,
}
impl WgKeyStore {
/// Opens, creating the file and its directory if needed.
pub fn open(path: impl AsRef<Path>) -> Result<Self, PluginError> {
let path = path.as_ref().to_path_buf();
if let Some(parent) = path.parent() {
tsunagi::storage::create_dir(parent)
.map_err(|err| PluginError::Other(format!("cannot create {parent:?}: {err}")))?;
}
let existed = path.exists();
let conn = Connection::open(&path).map_err(|err| {
PluginError::Other(format!("cannot open the WireGuard key store: {err}"))
})?;
tsunagi::storage::restrict_path_permissions(&path)
.map_err(|err| PluginError::Other(format!("cannot secure the key store: {err}")))?;
conn.busy_timeout(std::time::Duration::from_secs(5))
.and_then(|()| conn.pragma_update(None, "journal_mode", "WAL"))
.and_then(|()| conn.pragma_update(None, "synchronous", "NORMAL"))
.map_err(|err| PluginError::Other(format!("cannot configure the key store: {err}")))?;
if existed {
let integrity: String = conn
.query_row("PRAGMA integrity_check", [], |row| row.get(0))
.map_err(|err| {
PluginError::Other(format!("WireGuard key store is unusable: {err}"))
})?;
if integrity != "ok" {
return Err(PluginError::Other(format!(
"WireGuard key store at {} is corrupt and will not be recreated: {integrity}",
path.display()
)));
}
}
let found: i64 = conn
.query_row("PRAGMA user_version", [], |row| row.get(0))
.map_err(|err| PluginError::Other(format!("cannot read the schema version: {err}")))?;
if found > SCHEMA_VERSION {
return Err(PluginError::Other(format!(
"WireGuard key store schema {found} is newer than {SCHEMA_VERSION}"
)));
}
if found < SCHEMA_VERSION {
conn.execute_batch(
"BEGIN;
CREATE TABLE IF NOT EXISTS network_keys (
network_id BLOB PRIMARY KEY,
secret BLOB NOT NULL,
created_at INTEGER NOT NULL
);
PRAGMA user_version = 1;
COMMIT;",
)
.map_err(|err| PluginError::Other(format!("cannot create the schema: {err}")))?;
}
Ok(Self {
conn: Mutex::new(conn),
path,
})
}
/// Path of the underlying file.
pub fn path(&self) -> &Path {
&self.path
}
fn lock(&self) -> std::sync::MutexGuard<'_, Connection> {
match self.conn.lock() {
Ok(guard) => guard,
Err(poisoned) => poisoned.into_inner(),
}
}
/// Returns this agent's key for a network, creating it on first use.
pub fn load_or_create(&self, network: NetworkId) -> Result<WgSecretKey, PluginError> {
let conn = self.lock();
let stored: Option<Vec<u8>> = conn
.query_row(
"SELECT secret FROM network_keys WHERE network_id = ?1",
params![network.as_bytes().as_slice()],
|row| row.get(0),
)
.optional()
.map_err(|err| PluginError::Other(format!("cannot read the WireGuard key: {err}")))?;
if let Some(bytes) = stored {
let bytes = <[u8; KEY_LEN]>::try_from(bytes.as_slice()).map_err(|_| {
PluginError::Other(format!(
"the stored WireGuard key for network {} is not {KEY_LEN} bytes; \
refusing to replace it",
network.fmt_short()
))
})?;
return Ok(WgSecretKey::from_bytes(&bytes));
}
let key = WgSecretKey::generate();
conn.execute(
"INSERT INTO network_keys (network_id, secret, created_at) VALUES (?1, ?2, ?3)",
params![
network.as_bytes().as_slice(),
key.expose().as_slice(),
now_unix()
],
)
.map_err(|err| PluginError::Other(format!("cannot store the WireGuard key: {err}")))?;
Ok(key)
}
/// Deletes the key for a network.
///
/// Not called when a network is merely deactivated: coming back should
/// keep the same overlay address.
pub fn forget(&self, network: NetworkId) -> Result<(), PluginError> {
self.lock()
.execute(
"DELETE FROM network_keys WHERE network_id = ?1",
params![network.as_bytes().as_slice()],
)
.map_err(|err| PluginError::Other(format!("cannot remove the WireGuard key: {err}")))?;
Ok(())
}
}
fn now_unix() -> i64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs() as i64)
.unwrap_or(0)
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
use tsunagi::identity::{NetworkKeys, NetworkName, NetworkSecret};
fn network(name: &str) -> NetworkId {
NetworkKeys::derive(
&NetworkName::new(name).unwrap(),
&NetworkSecret::from_bytes(vec![8u8; 32]).unwrap(),
)
.network_id()
}
#[test]
fn keys_are_per_network_and_survive_reopening() {
let dir = tempfile::TempDir::new().unwrap();
let path = dir.path().join("wireguard.sqlite");
let first = network("one");
let second = network("two");
let (key_one, key_two) = {
let store = WgKeyStore::open(&path).unwrap();
let a = store.load_or_create(first).unwrap();
let b = store.load_or_create(second).unwrap();
assert_ne!(a.public(), b.public(), "networks get separate identities");
assert_eq!(a.public(), store.load_or_create(first).unwrap().public());
(a.public(), b.public())
};
let reopened = WgKeyStore::open(&path).unwrap();
assert_eq!(reopened.load_or_create(first).unwrap().public(), key_one);
assert_eq!(reopened.load_or_create(second).unwrap().public(), key_two);
reopened.forget(first).unwrap();
assert_ne!(reopened.load_or_create(first).unwrap().public(), key_one);
}
#[test]
fn a_corrupt_key_store_is_an_error_not_a_new_key() {
let dir = tempfile::TempDir::new().unwrap();
let path = dir.path().join("wireguard.sqlite");
let original = {
let store = WgKeyStore::open(&path).unwrap();
store.load_or_create(network("keep")).unwrap().public()
};
std::fs::write(&path, [0x5a; 4096]).unwrap();
let result = WgKeyStore::open(&path);
assert!(
result.is_err(),
"a damaged key store must not silently mint a new identity (was {original})"
);
}
}
@@ -0,0 +1,251 @@
//! The agent managing its own overlay interface.
//!
//! Everything here is real except the host: real agents, real control plane,
//! real iroh links, the real plugin lifecycle and the real reconciliation
//! rules. The host itself is a [`MockHost`], so what the agent would have
//! done to a machine's interfaces is asserted instead of done — which is how
//! this runs with no privileges and without touching the machine it is on.
//!
//! What the real provisioner adds on top of this is the netlink calls, and
//! only those.
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use std::net::{IpAddr, Ipv4Addr};
use std::sync::Arc;
use std::time::Duration;
use tempfile::TempDir;
use tsunagi::dataplane::IpPlugin;
use tsunagi::discovery::SharedMemoryDiscovery;
use tsunagi::identity::NetworkId;
use tsunagi::overlay::{
Cidr, InterfaceState, LinkKind, ManagedTunFactory, MockHost, MockProvisioner,
};
use tsunagi::state::DEFAULT_IPV4_RANGE;
use tsunagi::testing::{config_with, network, wait_until};
use tsunagi::{Agent, NetworkStatus};
use tsunagi_wg_quic::{WireguardConfig, WireguardPlugin};
/// An agent whose overlay interface is applied to a pretend host.
struct HostedAgent {
_dir: TempDir,
agent: Agent,
_plugin: Arc<WireguardPlugin>,
host: MockHost,
}
impl HostedAgent {
async fn spawn(discovery: &SharedMemoryDiscovery, tag: &str, host: MockHost) -> Self {
let dir = TempDir::new().unwrap();
let provisioner = Arc::new(MockProvisioner::new(host.clone()));
let factory = Arc::new(ManagedTunFactory::new(provisioner));
let config = WireguardConfig::new(dir.path().join("wireguard"))
.with_reconcile(Duration::from_millis(20), Duration::from_millis(100));
let plugin = WireguardPlugin::open(config).await.unwrap();
let agent = Agent::spawn(
config_with(dir.path(), discovery)
.with_overlay_ipv4_range(Some(DEFAULT_IPV4_RANGE))
.with_interface(factory, tag, 1280)
.with_plugin(plugin.clone() as Arc<dyn IpPlugin>),
)
.await
.unwrap();
Self {
_dir: dir,
agent,
_plugin: plugin,
host,
}
}
/// The interface name the plugin settled on for a network.
/// The one interface this agent owns. Not per network.
async fn interface(&self, _network: NetworkId) -> String {
wait_until("the agent named its interface", || async {
self.agent
.overlay()
.map(|overlay| overlay.interface)
.filter(|name| !name.is_empty())
})
.await
}
/// Waits until the pretend host shows an interface in the given state.
async fn wait_for_host<T>(
&self,
what: &str,
name: &str,
probe: impl Fn(Option<InterfaceState>) -> Option<T>,
) -> T {
wait_until(what, || async { probe(self.host.get(name)) }).await
}
}
fn v4(state: &InterfaceState) -> Vec<Ipv4Addr> {
state
.addresses
.iter()
.filter_map(|cidr| match cidr.addr {
IpAddr::V4(addr) => Some(addr),
IpAddr::V6(_) => None,
})
.collect()
}
/// Whether the overlay address has been put on the interface yet.
///
/// It is allocated and signed at the system level, so it arrives on a later
/// reconciliation than the interface itself rather than with it.
fn addressed(state: &InterfaceState) -> bool {
!state.addresses.is_empty()
}
#[tokio::test]
async fn an_agent_creates_and_configures_its_own_overlay_interface() {
let discovery = SharedMemoryDiscovery::new();
let (name, secret) = network("provision-create");
let agent = HostedAgent::spawn(&discovery, "tsunp", MockHost::new()).await;
let network_id = agent.agent.join_network(&name, &secret).await.unwrap();
let interface = agent.interface(network_id).await;
let state = agent
.wait_for_host("the interface to be created", &interface, |state| state)
.await;
assert_eq!(state.kind, LinkKind::Tun);
assert!(state.up, "the agent brought the link up itself");
assert_eq!(state.mtu, 1280);
let addresses = agent
.wait_for_host("the allocated IPv4 address", &interface, |state| {
state.map(|state| v4(&state)).filter(|v4| !v4.is_empty())
})
.await;
assert_eq!(addresses.len(), 1);
assert!(
DEFAULT_IPV4_RANGE.contains(addresses[0]),
"{:?} is outside {DEFAULT_IPV4_RANGE}",
addresses[0]
);
agent.agent.shutdown().await;
}
#[tokio::test]
async fn an_interface_left_by_a_crashed_run_is_replaced_rather_than_tripped_over() {
let discovery = SharedMemoryDiscovery::new();
let (name, secret) = network("provision-crash");
// Work out the name this agent will use, then seed the host with what a
// run that died would have left there: the interface, still carrying an
// address from an allocation that no longer applies, with nothing holding
// it open.
let probe = HostedAgent::spawn(&discovery, "tsunc", MockHost::new()).await;
let network_id = probe.agent.join_network(&name, &secret).await.unwrap();
let interface = probe.interface(network_id).await;
probe.agent.shutdown().await;
let host = MockHost::new();
let stale = Cidr::new(IpAddr::V4(Ipv4Addr::new(10, 13, 37, 178)), 24).unwrap();
host.insert_stale_tun(&interface, vec![stale]);
let agent = HostedAgent::spawn(&discovery, "tsunc", host).await;
let network_id = agent.agent.join_network(&name, &secret).await.unwrap();
assert_eq!(agent.interface(network_id).await, interface);
// Waited on directly rather than on "it has some address": the address
// this agent was allocated arrives a reconciliation after the interface
// does, and in between the leftover is still the only one there.
let state = agent
.wait_for_host("the stale address to be replaced", &interface, |state| {
state.filter(|state| {
state.attached && addressed(state) && !state.addresses.contains(&stale)
})
})
.await;
assert!(
!state.addresses.contains(&stale),
"the stale address is gone: {:?}",
state.addresses
);
agent.agent.shutdown().await;
}
#[tokio::test]
async fn an_interface_belonging_to_something_else_is_left_alone() {
let discovery = SharedMemoryDiscovery::new();
let (name, secret) = network("provision-foreign");
let probe = HostedAgent::spawn(&discovery, "tsunf", MockHost::new()).await;
let network_id = probe.agent.join_network(&name, &secret).await.unwrap();
let interface = probe.interface(network_id).await;
probe.agent.shutdown().await;
// Somebody else's bridge happens to hold the name.
let host = MockHost::new();
let theirs = InterfaceState {
kind: LinkKind::Foreign("bridge".into()),
attached: true,
up: true,
mtu: 1500,
addresses: vec![Cidr::new(IpAddr::V4(Ipv4Addr::new(192, 168, 9, 1)), 24).unwrap()],
};
host.insert(&interface, theirs.clone());
let agent = HostedAgent::spawn(&discovery, "tsunf", host).await;
agent.agent.join_network(&name, &secret).await.unwrap();
// Give the plugin several reconciliation rounds to do the wrong thing.
tokio::time::sleep(Duration::from_millis(300)).await;
assert_eq!(
agent.host.get(&interface),
Some(theirs),
"the foreign interface must be untouched"
);
// The control plane is unaffected by the data plane refusing.
assert!(matches!(
agent.agent.network_status(network_id).await,
Ok(NetworkStatus { .. })
));
agent.agent.shutdown().await;
}
#[tokio::test]
async fn leaving_a_network_takes_its_address_off_the_interface_but_not_the_interface() {
// The interface belongs to the agent, so it outlives any one network:
// another may still be using it. What a network takes with it is its own
// address.
let discovery = SharedMemoryDiscovery::new();
let (name, secret) = network("provision-cleanup");
let agent = HostedAgent::spawn(&discovery, "tsunx", MockHost::new()).await;
let network_id = agent.agent.join_network(&name, &secret).await.unwrap();
let interface = agent.interface(network_id).await;
agent
.wait_for_host("the address to be assigned", &interface, |state| {
state.filter(addressed)
})
.await;
agent.agent.deactivate_network(network_id).await.unwrap();
let state = agent
.wait_for_host("the address to be withdrawn", &interface, |state| {
state.filter(|state| !addressed(state))
})
.await;
assert_eq!(state.kind, LinkKind::Tun, "the interface is still there");
// And it goes when the agent does.
agent.agent.shutdown().await;
assert!(
agent.host.names().is_empty(),
"nothing is left behind: {:?}",
agent.host.names()
);
}
@@ -0,0 +1,235 @@
//! The local control interface: a client asking a running agent for status.
//!
//! Uses a real Unix socket on a temporary path, the real agent and the real
//! WireGuard data plane, so what a `tsunagi status` client would see is what
//! is checked here.
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use std::sync::Arc;
use std::time::Duration;
use tempfile::TempDir;
use tsunagi::dataplane::IpPlugin;
use tsunagi::discovery::SharedMemoryDiscovery;
use tsunagi::ipc::unix::{ControlSocket, request_status};
use tsunagi::ipc::{StatusReport, control_socket_path};
use tsunagi::overlay::MemoryTunFactory;
use tsunagi::testing::{config_with, network, wait_for_peers, wait_until};
use tsunagi::{Agent, BoxFuture};
use tsunagi_wg_quic::{WireguardConfig, WireguardPlugin};
/// Builds the report the way the binary does, from the agent plus the plugin.
fn source(agent: Agent, plugin: Arc<WireguardPlugin>) -> Arc<dyn tsunagi::ipc::unix::ReportSource> {
Arc::new(move || -> BoxFuture<'static, StatusReport> {
let agent = agent.clone();
let plugin = Arc::clone(&plugin);
Box::pin(async move {
let status = agent.status().await.unwrap();
let networks = status
.networks
.iter()
.map(|net| tsunagi::ipc::NetworkReport {
name: net.name.to_string(),
network_id: net.network_id.to_string(),
active: true,
peers: net
.peers
.iter()
.map(|peer| tsunagi::ipc::PeerReport {
endpoint_id: peer.endpoint_id.to_string(),
hostname: peer.hostname.clone(),
transport: format!("{:?}", peer.transport),
rtt_ms: peer.rtt.map(|rtt| rtt.as_millis() as u64),
})
.collect(),
overlay: plugin.overview(net.network_id).map(|view| {
tsunagi::ipc::OverlayReport {
// The interface belongs to the agent now.
interface: agent
.overlay()
.map_or_else(String::new, |overlay| overlay.interface),
mtu: agent.overlay().map_or(0, |overlay| overlay.mtu),
address: view.overlay_address_v4.map(|a| a.to_string()),
prefix_len: view.ipv4_range.map_or(0, |range| range.prefix_len),
peers: view
.peers
.iter()
.map(|peer| tsunagi::ipc::OverlayPeerReport {
public_key: peer.public_key.to_string(),
address: peer.overlay_address_v4.map(|a| a.to_string()),
handshake_secs_ago: peer
.tunnel
.as_ref()
.and_then(|t| t.health.since_handshake)
.map(|since| since.as_secs()),
..Default::default()
})
.collect(),
..Default::default()
}
}),
..Default::default()
})
.collect();
StatusReport {
endpoint_id: status.endpoint_id.to_string(),
hostname: status.hostname.clone(),
bound_sockets: status
.bound_sockets
.iter()
.map(ToString::to_string)
.collect(),
cache_healthy: status.cache_healthy,
networks,
dns: None,
}
})
})
}
#[tokio::test]
async fn a_client_sees_the_agent_and_its_overlay() {
let discovery = SharedMemoryDiscovery::new();
let (name, secret) = network("control-socket");
let dir_a = TempDir::new().unwrap();
let tuns = MemoryTunFactory::new();
let plugin = WireguardPlugin::open(
WireguardConfig::new(dir_a.path().join("wg"))
.with_reconcile(Duration::from_millis(20), Duration::from_millis(250)),
)
.await
.unwrap();
let agent = Agent::spawn(
config_with(dir_a.path(), &discovery)
.with_interface(Arc::new(tuns), "tca0", 1280)
.with_plugin(plugin.clone() as Arc<dyn IpPlugin>),
)
.await
.unwrap();
let dir_b = TempDir::new().unwrap();
let plugin_b = WireguardPlugin::open(
WireguardConfig::new(dir_b.path().join("wg"))
.with_reconcile(Duration::from_millis(20), Duration::from_millis(250)),
)
.await
.unwrap();
let agent_b = Agent::spawn(
config_with(dir_b.path(), &discovery)
.with_interface(Arc::new(MemoryTunFactory::new()), "tcb0", 1280)
.with_plugin(plugin_b.clone() as Arc<dyn IpPlugin>),
)
.await
.unwrap();
let network_id = agent.join_network(&name, &secret).await.unwrap();
agent_b.join_network(&name, &secret).await.unwrap();
wait_for_peers(&agent, network_id, 1).await;
// A short path: a Unix socket address is limited to about 100 bytes.
let socket_path = dir_a.path().join("agent.sock");
let control = ControlSocket::bind(&socket_path, source(agent.clone(), plugin.clone()))
.await
.unwrap();
let report = wait_until("the overlay is reported as up", || {
let socket_path = socket_path.clone();
async move {
let report = request_status(&socket_path).await.ok()?;
let overlay = report.networks.first()?.overlay.as_ref()?;
overlay
.peers
.iter()
.any(|peer| peer.is_up())
.then_some(report)
}
})
.await;
assert_eq!(report.endpoint_id, agent.endpoint_id().to_string());
assert_eq!(report.networks.len(), 1);
let net = &report.networks[0];
assert_eq!(net.network_id, network_id.to_string());
assert_eq!(net.peers.len(), 1);
assert_eq!(net.peers[0].endpoint_id, agent_b.endpoint_id().to_string());
let overlay = net.overlay.as_ref().unwrap();
assert!(overlay.interface.starts_with("tca"));
assert_eq!(overlay.mtu, 1280);
assert_eq!(overlay.peers.len(), 1);
// The report carries what a reader needs, in structured form: how it is
// laid out is the CLI's business and is tested there.
assert_eq!(report.endpoint_id, agent.endpoint_id().to_string());
assert_eq!(
overlay.peers.iter().filter(|peer| peer.is_up()).count(),
1,
"the tunnel is up: {:?}",
overlay.peers
);
assert!(overlay.address.is_some());
control.shutdown().await;
assert!(!socket_path.exists(), "the socket is removed on shutdown");
// With nothing listening, a client gets an error rather than hanging.
assert!(request_status(&socket_path).await.is_err());
agent.shutdown().await;
agent_b.shutdown().await;
}
#[tokio::test]
async fn a_leftover_socket_file_is_replaced_but_a_live_one_is_not() {
let dir = TempDir::new().unwrap();
let path = dir.path().join("agent.sock");
let empty: Arc<dyn tsunagi::ipc::unix::ReportSource> =
Arc::new(|| -> BoxFuture<'static, StatusReport> {
Box::pin(async { StatusReport::default() })
});
// A file with nobody listening is a leftover from a crash.
std::fs::write(&path, b"stale").unwrap();
let first = ControlSocket::bind(&path, Arc::clone(&empty))
.await
.unwrap();
assert!(request_status(&path).await.is_ok());
// A live socket is not stolen from the agent that owns it.
let second = ControlSocket::bind(&path, Arc::clone(&empty)).await;
assert!(
matches!(second, Err(tsunagi::Error::StateLocked { .. })),
"a second agent must not take over a live control socket"
);
first.shutdown().await;
}
#[test]
fn the_socket_path_is_derived_and_short_enough() {
let deep = std::path::PathBuf::from(
"/home/someone/.local/share/with/a/very/deeply/nested/directory/that/goes/on/and/on/and/on/tsunagi/state",
);
let path = control_socket_path(&deep);
// A Unix socket address is limited to roughly 100 bytes, so a deep state
// directory must not produce a path that cannot be bound.
if std::env::var_os("XDG_RUNTIME_DIR").is_some() {
assert!(
path.as_os_str().len() < 100,
"derived path is {} bytes: {}",
path.as_os_str().len(),
path.display()
);
}
// Deterministic, and different state directories never share a socket.
assert_eq!(path, control_socket_path(&deep));
assert_ne!(
path,
control_socket_path(&std::path::PathBuf::from("/somewhere/else"))
);
}
File diff suppressed because it is too large Load Diff