diff --git a/README.md b/README.md index 5c20227..6aa34ca 100644 --- a/README.md +++ b/README.md @@ -31,10 +31,10 @@ A working library with **real iroh connections** and integration tests: - status snapshots, an event stream and honest diagnostics; - configuration restored after a restart; - correct behaviour when the disposable cache is missing or corrupt; -- a **WireGuard data plane**, in userspace: its own key per network, - deterministic IPv6 overlay addressing, real tunnels carried over iroh, and - address ownership enforced rather than believed; -- a **command line agent**, `tsunagi`. +- a **WireGuard data plane**, in userspace: its own key per network, a dual + stack overlay with deterministically derived addresses, real tunnels carried + over iroh, and address ownership enforced rather than believed; +- a **command line agent**, `tsunagi`, with a local control socket. ### What it deliberately does **not** do @@ -94,8 +94,41 @@ wireguard: tsunkkcp43lmdje on fd15:1d9e:fa21:f201:…/64 mtu 1280, 1/1 tunnel(s) 4jO4kx9Z fd15:1d9e:fa21:f201:… handshake 3s ago tx=0 rx=0 dropped=0 path=Direct via Ip(…) ``` -`1/1 tunnel(s) established` means a real WireGuard handshake completed. Then -`ping6` the peer's overlay address. +`1/1 tunnel(s) established` means a real WireGuard handshake completed. + +## Checking that it works + +From another shell on either machine: + +```bash +tsunagi status +``` + +```text +endpoint 7d76ccbbc21bf30767e14422c0494740a2cecc02aa9f82d5b8d57bdae350e7fc +hostname tsunagi-7d76ccbbc2 +bound 0.0.0.0:41641 + +network lab (z2o4qwrvnj3zb6st2aoqg4abf342j662q2ujttqrsmz22argk2ba) active + peer b345d5271b tsunagi-b345d5271b Direct rtt 24ms + overlay tsunz2o4qwrvnj3 fd09:…:c1c6/64 and 100.110.49.177 mtu 1280 1/1 tunnel(s) up + SDsEb/WF fd09:…:c4b / 100.65.243.53 handshake 4s ago tx 0 rx 0 Direct via Ip(…) +``` + +`1/1 tunnel(s) up` and a recent handshake mean the tunnel is live. Then send +real traffic to the peer's overlay address: + +```bash +ping6 fd09:…:c4b # or +ping 100.65.243.53 +``` + +`tx` and `rx` in the status should start moving. + +The overlay is dual stack: every member derives both an IPv6 address, which +can never collide, and an IPv4 one in `100.64.0.0/10`, which very rarely can — +see [docs/wireguard.md](docs/wireguard.md#ipv4-alongside-ipv6). `--no-ipv4` +runs IPv6 only, `--ipv4-range` moves the range. Notes: @@ -130,6 +163,7 @@ sudo ip tuntap add dev tsunjwc6dcrtmo5 mode tun user ab sudo ip link set dev tsunjwc6dcrtmo5 mtu 1280 up sudo sysctl -qw net.ipv6.conf.tsunjwc6dcrtmo5.keep_addr_on_down=1 sudo ip -6 address add fd80:1210:f724:f620:d1bb:f982:3b6e:19bd/64 dev tsunjwc6dcrtmo5 nodad +sudo ip address add 100.110.49.177/10 dev tsunjwc6dcrtmo5 ``` The MTU is 1280 because that is the minimum IPv6 requires (RFC 8200). Linux diff --git a/docs/testing.md b/docs/testing.md index 20de3da..dd2646c 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -50,6 +50,11 @@ keeping the WireGuard identity, shutdown removing every interface, a forged overlay claim being rejected, and the core carrying the payload without interpreting it. +`tests/local_control.rs` covers the local control socket end to end: a client +asking a running agent for status over a real Unix socket, a leftover socket +file being replaced while a live one is not, and the derived socket path +staying short enough to bind. + `tests/discovery.rs` covers the discovery contract itself: a static bootstrap candidate is enough to join, several backends compose, entries are withdrawn when a network stops, and a forgotten network stays forgotten across a restart. diff --git a/docs/wireguard.md b/docs/wireguard.md index 04a4cee..999e675 100644 --- a/docs/wireguard.md +++ b/docs/wireguard.md @@ -56,6 +56,12 @@ direct path when it can and falls back to a relay when it cannot; the tunnel rides on whichever it got. There is no separate STUN, no separate hole punching and no second set of NAT problems to solve for WireGuard. +## Checking it from outside + +`tsunagi status` asks a running agent over its local control socket and prints +what it sees, including whether each tunnel has actually handshaken. See +[../README.md](../README.md#checking-that-it-works). + ## Deterministic overlay addressing A mesh with no coordinator cannot hand out addresses, so everyone derives their @@ -77,6 +83,29 @@ Two consequences matter: * a member's address is bound to its WireGuard public key, so address ownership can be checked locally rather than believed. +## IPv4 alongside IPv6 + +The overlay is dual stack by default: every member also derives an IPv4 +address, from the same inputs, into `100.64.0.0/10` (RFC 6598 shared address +space — deliberately not RFC 1918, so it rarely clashes with the network the +machine is already on). The range is configurable, and IPv4 can be turned off. + +**IPv4 is weaker than IPv6 here, and the difference is not cosmetic.** A 64 bit +interface identifier makes an IPv6 collision impossible in practice. IPv4 has +nothing like that much room: in a `/10` with 50 members the chance that two +derive the same address is roughly 0.03%. Small, but not zero, and a mesh with +no coordinator cannot simply allocate around it. + +So a collision is detected and resolved rather than assumed away: the member +whose WireGuard public key sorts lower keeps the address, a rule every member +computes identically and therefore agrees on without exchanging anything. The +other member ends up with **no IPv4 address** and is still fully reachable over +IPv6. The status output flags it. + +That is the honest summary: **IPv6 always works; IPv4 almost always works and +degrades predictably when it does not.** Allocating IPv4 properly needs the +agreed state described in [sync-model.md](sync-model.md). + ## Address ownership is enforced, not announced Kernel WireGuard enforces `AllowedIPs`. In userspace that is our job, and @@ -87,6 +116,8 @@ Kernel WireGuard enforces `AllowedIPs`. In userspace that is our job, and * **inbound**, a decrypted packet is dropped unless its *source* is exactly the address derived for the peer whose tunnel decrypted it. +Both apply to IPv4 and IPv6 alike. + So a participant cannot receive traffic addressed to somebody else and cannot forge traffic that appears to come from somebody else. A participant who knows the network secret can mint many keys and therefore occupy many addresses, but @@ -192,9 +223,8 @@ async fn main() -> Result<()> { * **Full mesh only.** Every member runs a tunnel to every other member. Routing through an intermediate participant is not implemented. -* **IPv6 overlay only.** Addressing is IPv6 ULA because it can be derived - collision-free. An IPv4 overlay would need an allocator, which needs the - agreed state described in [sync-model.md](sync-model.md). +* **IPv4 addressing can collide.** See above: it is resolved deterministically + and the loser keeps IPv6, but a proper allocator needs agreed state. * **No routes, DNS or firewall rules.** The plugin creates its interface and nothing else. Anything beyond the overlay `/64` is the operator's business. * **Membership is session-scoped.** A peer leaves the overlay when its control diff --git a/src/bin/tsunagi.rs b/src/bin/tsunagi.rs index f3993b4..2942034 100644 --- a/src/bin/tsunagi.rs +++ b/src/bin/tsunagi.rs @@ -98,6 +98,45 @@ struct TunSetupArgs { /// Interface MTU, matching `tsunagi up --wg-mtu`. At least 1280. #[arg(long)] wg_mtu: Option, + + /// Match `tsunagi up --no-ipv4`. + #[arg(long)] + no_ipv4: bool, + + /// Match `tsunagi up --ipv4-range`. + #[arg(long, value_name = "CIDR", conflicts_with = "no_ipv4")] + ipv4_range: Option, +} + +/// Parses `address/prefix` into an IPv4 range. +fn parse_ipv4_range(text: &str) -> Result<(std::net::Ipv4Addr, u8), String> { + let (address, prefix) = text + .split_once('/') + .ok_or_else(|| format!("`{text}` is not an address with a prefix, e.g. 100.64.0.0/10"))?; + let address = address + .parse() + .map_err(|err| format!("`{address}` is not an IPv4 address: {err}"))?; + let prefix: u8 = prefix + .parse() + .map_err(|err| format!("`{prefix}` is not a prefix length: {err}"))?; + if prefix > 30 { + return Err(format!("a /{prefix} has no room for hosts")); + } + Ok((address, prefix)) +} + +/// Resolves the IPv4 overlay range from the flags. +fn resolve_ipv4_range( + no_ipv4: bool, + range: Option<&String>, +) -> Result, Box> { + if no_ipv4 { + return Ok(None); + } + match range { + Some(text) => Ok(Some(parse_ipv4_range(text)?)), + None => Ok(Some(tsunagi::dataplane::wireguard::DEFAULT_IPV4_RANGE)), + } } #[derive(Debug, Args, Clone)] @@ -212,6 +251,14 @@ struct UpArgs { #[arg(long)] wg_mtu: Option, + /// Run an IPv6-only overlay instead of dual stack. + #[arg(long)] + no_ipv4: bool, + + /// IPv4 overlay range, as `address/prefix`. Defaults to 100.64.0.0/10. + #[arg(long, value_name = "CIDR", conflicts_with = "no_ipv4")] + ipv4_range: Option, + /// How often to print a status summary, in seconds. Zero disables it. #[arg(long, default_value_t = 15)] status_interval: u64, @@ -346,6 +393,7 @@ async fn status(args: StatusArgs) -> Result<(), Box> { async fn tun_setup(args: TunSetupArgs) -> Result<(), Box> { use tsunagi::dataplane::wireguard::{ DEFAULT_MTU, OVERLAY_PREFIX_LEN, WgKeyStore, interface_name, overlay_address, + overlay_address_v4, }; use tsunagi::identity::NetworkKeys; @@ -364,6 +412,7 @@ async fn tun_setup(args: TunSetupArgs) -> Result<(), Box> let interface = interface_name(&args.wg_prefix, network)?; let address = overlay_address(network, &key.public()); + let ipv4_range = resolve_ipv4_range(args.no_ipv4, args.ipv4_range.as_ref())?; let mtu = args.wg_mtu.unwrap_or(DEFAULT_MTU); let user = args.user.unwrap_or_else(|| { std::env::var("SUDO_USER") @@ -373,6 +422,11 @@ async fn tun_setup(args: TunSetupArgs) -> Result<(), Box> println!("# Network {name} ({network})"); println!("# Interface {interface}, address {address}/{OVERLAY_PREFIX_LEN}, mtu {mtu}"); + if let Some((base, prefix)) = ipv4_range + && let Some(v4) = overlay_address_v4(network, &key.public(), (base, prefix)) + { + println!("# IPv4 overlay address {v4}/{prefix}"); + } println!("# Run once as root; then run `tsunagi up` as {user}."); println!( "#\n\ @@ -386,6 +440,11 @@ async fn tun_setup(args: TunSetupArgs) -> Result<(), Box> println!("sudo ip link set dev {interface} mtu {mtu} up"); println!("sudo sysctl -qw net.ipv6.conf.{interface}.keep_addr_on_down=1"); println!("sudo ip -6 address add {address}/{OVERLAY_PREFIX_LEN} dev {interface} nodad"); + if let Some((base, prefix)) = ipv4_range + && let Some(v4) = overlay_address_v4(network, &key.public(), (base, prefix)) + { + println!("sudo ip address add {v4}/{prefix} dev {interface}"); + } println!("\n# To check it afterwards:"); println!("ip -6 addr show dev {interface}"); println!("\n# To remove it again:"); @@ -516,7 +575,8 @@ async fn up(args: UpArgs) -> Result<(), Box> { system_tun_factory()? }; let mut wg = WireguardConfig::new(paths.state_dir.join("wireguard")) - .with_interface_prefix(args.wg_prefix.clone()); + .with_interface_prefix(args.wg_prefix.clone()) + .with_ipv4_range(resolve_ipv4_range(args.no_ipv4, args.ipv4_range.as_ref())?); if let Some(mtu) = args.wg_mtu { wg = wg.with_mtu(mtu); } @@ -637,6 +697,7 @@ async fn build_report( interface: view.interface.clone(), mtu: view.mtu, address: view.overlay_address.to_string(), + address_v4: view.overlay_address_v4.map(|addr| addr.to_string()), prefix: view.overlay_prefix.to_string(), prefix_len: view.overlay_prefix_len, peers: view @@ -645,6 +706,7 @@ async fn build_report( .map(|peer| OverlayPeerReport { public_key: peer.public_key.to_string(), address: peer.overlay_address.to_string(), + address_v4: peer.overlay_address_v4.map(|addr| addr.to_string()), handshake_secs_ago: peer .tunnel .as_ref() diff --git a/src/dataplane/wireguard/device.rs b/src/dataplane/wireguard/device.rs index a51e6fd..e0d9994 100644 --- a/src/dataplane/wireguard/device.rs +++ b/src/dataplane/wireguard/device.rs @@ -27,7 +27,7 @@ //! what it announced. use std::collections::HashMap; -use std::net::Ipv6Addr; +use std::net::{IpAddr, Ipv4Addr, Ipv6Addr}; use std::sync::atomic::{AtomicU32, AtomicU64, Ordering}; use std::sync::{Arc, Mutex, RwLock}; use std::time::Duration; @@ -42,7 +42,7 @@ use crate::dataplane::transport::{SharedLink, TransportError}; use crate::identity::NetworkId; use super::keys::{WgPublicKey, WgSecretKey}; -use super::overlay::overlay_address; +use super::overlay::{overlay_address, overlay_address_v4}; use super::packet::IpHeader; use super::tun::TunDevice; @@ -109,6 +109,9 @@ struct Peer { endpoint_id: EndpointId, public_key: WgPublicKey, overlay: Ipv6Addr, + /// The IPv4 address this peer owns, when the overlay is dual stack and + /// nobody else derived the same one. + overlay_v4: Mutex>, tunn: Mutex, link: SharedLink, counters: Arc, @@ -168,6 +171,13 @@ pub struct PeerSummary { pub public_key: WgPublicKey, /// The overlay address this agent derived for it. pub overlay_address: Ipv6Addr, + /// Its IPv4 overlay address, when the overlay is dual stack. + /// + /// `None` with `ipv4_conflict` set means another member derived the same + /// address and won it; that peer is still fully reachable over IPv6. + pub overlay_address_v4: Option, + /// Whether this peer lost an IPv4 address to a derivation collision. + pub ipv4_conflict: bool, /// Whether the tunnel has handshaken. pub health: PeerHealth, /// Traffic counters. @@ -182,11 +192,15 @@ struct Inner { network: NetworkId, private_key: WgSecretKey, tun: Arc, + /// The IPv4 overlay range, when the overlay is dual stack. + ipv4_range: Option<(Ipv4Addr, u8)>, peers: RwLock>>, - routes: RwLock>, + /// Both families, so one lookup routes any packet. + routes: RwLock>, next_index: AtomicU32, unroutable: AtomicU64, multicast: AtomicU64, + ipv4_conflicts: AtomicU64, } impl std::fmt::Debug for Inner { @@ -207,16 +221,23 @@ pub struct WireguardDevice { impl WireguardDevice { /// Starts a device on top of `tun`. - pub fn start(network: NetworkId, private_key: WgSecretKey, tun: Arc) -> Self { + pub fn start( + network: NetworkId, + private_key: WgSecretKey, + tun: Arc, + ipv4_range: Option<(Ipv4Addr, u8)>, + ) -> Self { let inner = Arc::new(Inner { network, private_key, tun, + ipv4_range, peers: RwLock::new(HashMap::new()), routes: RwLock::new(HashMap::new()), next_index: AtomicU32::new(1), unroutable: AtomicU64::new(0), multicast: AtomicU64::new(0), + ipv4_conflicts: AtomicU64::new(0), }); let reader = tokio::spawn(read_from_os(Arc::clone(&inner))); @@ -263,10 +284,12 @@ impl WireguardDevice { ); let overlay = overlay_address(self.inner.network, &public_key); + let overlay_v4 = self.claim_ipv4(&public_key); let peer = Arc::new(Peer { endpoint_id, public_key, overlay, + overlay_v4: Mutex::new(overlay_v4), tunn: Mutex::new(tunn), link, counters: Arc::new(PeerCounters::default()), @@ -279,7 +302,10 @@ impl WireguardDevice { } write_lock(&self.inner.peers).insert(public_key, Arc::clone(&peer)); - write_lock(&self.inner.routes).insert(overlay, public_key); + write_lock(&self.inner.routes).insert(IpAddr::V6(overlay), public_key); + if let Some(v4) = overlay_v4 { + write_lock(&self.inner.routes).insert(IpAddr::V4(v4), public_key); + } // Start the handshake now instead of waiting for the next timer tick, // so the tunnel is usable as soon as the link exists. @@ -287,13 +313,64 @@ impl WireguardDevice { Ok(()) } + /// Decides which IPv4 address a new peer gets, if any. + /// + /// IPv4 has far too little room for a derived address to be collision + /// free. When two members derive the same one, the member whose public + /// key sorts lower keeps it — a rule every member computes identically, + /// so they all agree on the outcome without talking about it. The other + /// member simply has no IPv4 address; it is still fully reachable over + /// IPv6, which never collides. + fn claim_ipv4(&self, public_key: &WgPublicKey) -> Option { + let range = self.inner.ipv4_range?; + let wanted = overlay_address_v4(self.inner.network, public_key, range)?; + + let holder = read_lock(&self.inner.routes) + .get(&IpAddr::V4(wanted)) + .copied(); + let Some(holder) = holder else { + return Some(wanted); + }; + if holder == *public_key { + return Some(wanted); + } + + self.inner.ipv4_conflicts.fetch_add(1, Ordering::Relaxed); + if holder.as_bytes() <= public_key.as_bytes() { + // The peer already holding it wins. + return None; + } + // The newcomer wins; take the address away from the other peer. + if let Some(loser) = read_lock(&self.inner.peers).get(&holder).cloned() { + let mut slot = match loser.overlay_v4.lock() { + Ok(guard) => guard, + Err(poisoned) => poisoned.into_inner(), + }; + *slot = None; + } + Some(wanted) + } + /// Removes a peer and stops its tunnel. pub fn remove_peer(&self, public_key: &WgPublicKey) { if let Some(peer) = write_lock(&self.inner.peers).remove(public_key) { - write_lock(&self.inner.routes).remove(&peer.overlay); + let mut routes = write_lock(&self.inner.routes); + routes.remove(&IpAddr::V6(peer.overlay)); + let v4 = match peer.overlay_v4.lock() { + Ok(guard) => *guard, + Err(poisoned) => *poisoned.into_inner(), + }; + if let Some(v4) = v4 { + routes.remove(&IpAddr::V4(v4)); + } } } + /// How many IPv4 derivation collisions have been resolved. + pub fn ipv4_conflicts(&self) -> u64 { + self.inner.ipv4_conflicts.load(Ordering::Relaxed) + } + /// Removes every peer whose key is not in `keep`. pub fn retain_peers(&self, keep: &[WgPublicKey]) { let stale: Vec = read_lock(&self.inner.peers) @@ -315,14 +392,22 @@ impl WireguardDevice { pub fn peers(&self) -> Vec { let mut peers: Vec = read_lock(&self.inner.peers) .values() - .map(|peer| PeerSummary { - endpoint_id: peer.endpoint_id, - public_key: peer.public_key, - overlay_address: peer.overlay, - health: peer.health(), - stats: peer.stats(), - path: peer.link.path_description(), - max_datagram: peer.link.max_datagram_size(), + .map(|peer| { + let overlay_v4 = match peer.overlay_v4.lock() { + Ok(guard) => *guard, + Err(poisoned) => *poisoned.into_inner(), + }; + PeerSummary { + endpoint_id: peer.endpoint_id, + public_key: peer.public_key, + overlay_address: peer.overlay, + overlay_address_v4: overlay_v4, + ipv4_conflict: overlay_v4.is_none() && self.inner.ipv4_range.is_some(), + health: peer.health(), + stats: peer.stats(), + path: peer.link.path_description(), + max_datagram: peer.link.max_datagram_size(), + } }) .collect(); peers.sort_by_key(|peer| peer.public_key); @@ -415,9 +500,8 @@ async fn read_from_os(inner: Arc) { }; // Route by destination: only the peer that owns that overlay address - // may receive it. - let Some(destination) = IpHeader::parse(&packet).and_then(|header| header.v6_destination()) - else { + // may receive it. Both families go through the same table. + let Some(destination) = IpHeader::parse(&packet).map(|header| header.destination()) else { inner.unroutable.fetch_add(1, Ordering::Relaxed); continue; }; @@ -488,9 +572,11 @@ async fn read_from_link(inner: Arc, peer: Arc) { match tunn.decapsulate(None, input.unwrap_or(&[]), &mut scratch) { TunnResult::WriteToNetwork(out) => Outcome::ToNetwork(out.len()), TunnResult::WriteToTunnelV6(out, source) => { - Outcome::ToTunnel(out.len(), Some(source)) + Outcome::ToTunnel(out.len(), IpAddr::V6(source)) + } + TunnResult::WriteToTunnelV4(out, source) => { + Outcome::ToTunnel(out.len(), IpAddr::V4(source)) } - TunnResult::WriteToTunnelV4(out, _) => Outcome::ToTunnel(out.len(), None), TunnResult::Done => Outcome::Done, TunnResult::Err(err) => { tracing::trace!(?err, "wireguard decapsulation failed"); @@ -508,9 +594,19 @@ async fn read_from_link(inner: Arc, peer: Arc) { } Outcome::ToTunnel(len, source) => { let payload = Bytes::copy_from_slice(&scratch[..len]); - // Enforce address ownership: a peer may only send from the - // address derived for its own key. - if source != Some(peer.overlay) { + // Enforce address ownership: a peer may only send from an + // address derived for its own key, in either family. + let owned = match source { + IpAddr::V6(addr) => addr == peer.overlay, + IpAddr::V4(addr) => { + let held = match peer.overlay_v4.lock() { + Ok(guard) => *guard, + Err(poisoned) => *poisoned.into_inner(), + }; + held == Some(addr) + } + }; + if !owned { peer.counters .dropped_wrong_source .fetch_add(1, Ordering::Relaxed); @@ -538,7 +634,7 @@ async fn read_from_link(inner: Arc, peer: Arc) { enum Outcome { ToNetwork(usize), - ToTunnel(usize, Option), + ToTunnel(usize, IpAddr), Done, Failed, } diff --git a/src/dataplane/wireguard/mod.rs b/src/dataplane/wireguard/mod.rs index ec9b928..4ce9a77 100644 --- a/src/dataplane/wireguard/mod.rs +++ b/src/dataplane/wireguard/mod.rs @@ -54,7 +54,9 @@ pub use announcement::{ValidatedAnnouncement, WgAnnouncement}; pub use config::{Cidr, DEFAULT_INTERFACE_PREFIX, MAX_INTERFACE_NAME_LEN, interface_name}; pub use device::{PeerHealth, PeerStats, PeerSummary, WireguardDevice}; pub use keys::{WgPublicKey, WgSecretKey}; -pub use overlay::{OVERLAY_PREFIX_LEN, overlay_address, overlay_prefix}; +pub use overlay::{ + DEFAULT_IPV4_RANGE, OVERLAY_PREFIX_LEN, overlay_address, overlay_address_v4, overlay_prefix, +}; pub use packet::IpHeader; pub use plugin::{ DEFAULT_MTU, MIN_MTU, NetworkOverview, PeerOverview, WIREGUARD_OVERHEAD, WIREGUARD_PROTOCOL, diff --git a/src/dataplane/wireguard/overlay.rs b/src/dataplane/wireguard/overlay.rs index 33b05bf..d68a151 100644 --- a/src/dataplane/wireguard/overlay.rs +++ b/src/dataplane/wireguard/overlay.rs @@ -20,7 +20,7 @@ //! but it cannot choose to collide with an existing member's address without //! finding a hash preimage. -use std::net::Ipv6Addr; +use std::net::{Ipv4Addr, Ipv6Addr}; use sha2::{Digest, Sha256}; @@ -37,6 +37,13 @@ pub const OVERLAY_PREFIX_LEN: u8 = 64; /// Prefix length of one member's address inside the overlay. pub const OVERLAY_HOST_PREFIX_LEN: u8 = 128; +/// Default IPv4 overlay range: RFC 6598 shared address space. +/// +/// Deliberately not RFC 1918, so it rarely collides with the home or office +/// network the machine is already on. It can collide with a carrier-grade NAT +/// that uses the same range, which is why it is configurable. +pub const DEFAULT_IPV4_RANGE: (Ipv4Addr, u8) = (Ipv4Addr::new(100, 64, 0, 0), 10); + fn push_lp(out: &mut Vec, bytes: &[u8]) { let len = u32::try_from(bytes.len()).unwrap_or(u32::MAX); out.extend_from_slice(&len.to_be_bytes()); @@ -85,6 +92,49 @@ pub fn overlay_address(network: NetworkId, key: &WgPublicKey) -> Ipv6Addr { Ipv6Addr::from(octets) } +/// The IPv4 address a member with `key` has in `network`. +/// +/// # Why this is weaker than the IPv6 derivation +/// +/// A 64 bit interface identifier makes an IPv6 collision impossible in +/// practice. IPv4 has nothing like that much room, so two members *can* derive +/// the same address. In a `/10` with 50 members the chance is roughly 0.03%, +/// which is small but real, so it is detected and resolved rather than +/// assumed away — see [`super::device`]. IPv6 remains the address that always +/// works. +/// +/// Returns `None` when the range has no room for hosts. +pub fn overlay_address_v4( + network: NetworkId, + key: &WgPublicKey, + range: (Ipv4Addr, u8), +) -> Option { + let (base, prefix_len) = range; + if prefix_len > 32 { + return None; + } + let host_bits = 32 - u32::from(prefix_len); + // A usable range needs a network address, a broadcast address and at + // least one host between them. + if host_bits < 2 { + return None; + } + + let hash = digest("ipv4", network, Some(key)); + let raw = u32::from_be_bytes([hash[0], hash[1], hash[2], hash[3]]); + + let usable = (1u64 << host_bits) - 2; + let offset = (u64::from(raw) % usable) + 1; + + let mask = if host_bits == 32 { + 0 + } else { + u32::MAX << host_bits + }; + let network_part = u32::from(base) & mask; + Some(Ipv4Addr::from(network_part | offset as u32)) +} + #[cfg(test)] mod tests { #![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)] @@ -132,6 +182,65 @@ mod tests { assert_ne!(overlay_prefix(first), overlay_prefix(second)); } + #[test] + fn ipv4_addresses_land_inside_the_range_and_avoid_its_edges() { + let id = network("v4"); + let range = DEFAULT_IPV4_RANGE; + for byte in 0..64u8 { + let key = WgPublicKey::from_bytes([byte; 32]); + let addr = overlay_address_v4(id, &key, range).unwrap(); + let raw = u32::from(addr); + assert_eq!( + raw & 0xffc0_0000, + u32::from(range.0), + "outside 100.64.0.0/10" + ); + // Never the network address and never the broadcast address. + assert_ne!(raw & 0x003f_ffff, 0); + assert_ne!(raw & 0x003f_ffff, 0x003f_ffff); + } + } + + #[test] + fn ipv4_derivation_is_deterministic_and_scoped_like_ipv6() { + let key = WgPublicKey::from_bytes([9u8; 32]); + let first = network("one"); + let second = network("two"); + let range = DEFAULT_IPV4_RANGE; + + assert_eq!( + overlay_address_v4(first, &key, range), + overlay_address_v4(first, &key, range) + ); + assert_ne!( + overlay_address_v4(first, &key, range), + overlay_address_v4(second, &key, range) + ); + assert_ne!( + overlay_address_v4(first, &key, range), + overlay_address_v4(first, &WgPublicKey::from_bytes([10u8; 32]), range) + ); + // A different range moves everybody. + assert_ne!( + overlay_address_v4(first, &key, range), + overlay_address_v4(first, &key, (Ipv4Addr::new(10, 0, 0, 0), 8)) + ); + } + + #[test] + fn a_range_with_no_room_yields_nothing() { + let id = network("tiny"); + let key = WgPublicKey::from_bytes([1u8; 32]); + // /31 and /32 have no usable host addresses. + assert!(overlay_address_v4(id, &key, (Ipv4Addr::new(10, 0, 0, 0), 31)).is_none()); + assert!(overlay_address_v4(id, &key, (Ipv4Addr::new(10, 0, 0, 1), 32)).is_none()); + assert!(overlay_address_v4(id, &key, (Ipv4Addr::new(10, 0, 0, 0), 33)).is_none()); + // A /30 has two usable addresses. + assert!(overlay_address_v4(id, &key, (Ipv4Addr::new(10, 0, 0, 0), 30)).is_some()); + // A /0 must not overflow. + assert!(overlay_address_v4(id, &key, (Ipv4Addr::UNSPECIFIED, 0)).is_some()); + } + #[test] fn addresses_are_never_the_subnet_router_anycast_address() { let id = network("anycast"); diff --git a/src/dataplane/wireguard/packet.rs b/src/dataplane/wireguard/packet.rs index 60ae5ab..e6c36ea 100644 --- a/src/dataplane/wireguard/packet.rs +++ b/src/dataplane/wireguard/packet.rs @@ -65,6 +65,22 @@ impl IpHeader { IpHeader::V4 { .. } => None, } } + + /// The destination, whichever family it is. + pub fn destination(&self) -> std::net::IpAddr { + match self { + IpHeader::V4 { destination, .. } => std::net::IpAddr::V4(*destination), + IpHeader::V6 { destination, .. } => std::net::IpAddr::V6(*destination), + } + } + + /// The source, whichever family it is. + pub fn source(&self) -> std::net::IpAddr { + match self { + IpHeader::V4 { source, .. } => std::net::IpAddr::V4(*source), + IpHeader::V6 { source, .. } => std::net::IpAddr::V6(*source), + } + } } #[cfg(test)] diff --git a/src/dataplane/wireguard/plugin.rs b/src/dataplane/wireguard/plugin.rs index 735c11e..08211d9 100644 --- a/src/dataplane/wireguard/plugin.rs +++ b/src/dataplane/wireguard/plugin.rs @@ -25,7 +25,7 @@ //! A failure here is reported and retried. It never stops the control plane. use std::collections::{BTreeSet, HashMap}; -use std::net::IpAddr; +use std::net::{IpAddr, Ipv4Addr}; use std::path::PathBuf; use std::sync::{Arc, Mutex, OnceLock}; use std::time::Duration; @@ -43,7 +43,9 @@ use super::announcement::{ValidatedAnnouncement, WgAnnouncement}; use super::config::{DEFAULT_INTERFACE_PREFIX, interface_name}; use super::device::{PeerSummary, WireguardDevice}; use super::keys::{WgPublicKey, WgSecretKey}; -use super::overlay::{OVERLAY_PREFIX_LEN, overlay_address, overlay_prefix}; +use super::overlay::{ + DEFAULT_IPV4_RANGE, OVERLAY_PREFIX_LEN, overlay_address, overlay_address_v4, overlay_prefix, +}; use super::store::WgKeyStore; use super::tun::{TunFactory, TunRequest}; @@ -86,6 +88,12 @@ pub struct WireguardConfig { pub keepalive: Option, /// Interface MTU. See [`DEFAULT_MTU`]. pub mtu: u32, + /// IPv4 overlay range, or `None` for an IPv6-only overlay. + /// + /// IPv6 addresses are derived collision-free; IPv4 ones cannot be, so a + /// collision is detected and resolved deterministically instead. See + /// `docs/wireguard.md`. + pub ipv4_range: Option<(Ipv4Addr, u8)>, /// How long to coalesce changes before reconciling. pub reconcile_debounce: Duration, /// How often to reconcile anyway, which is also when a packet interface @@ -101,6 +109,7 @@ impl WireguardConfig { interface_prefix: DEFAULT_INTERFACE_PREFIX.to_string(), keepalive: Some(25), mtu: DEFAULT_MTU, + ipv4_range: Some(DEFAULT_IPV4_RANGE), reconcile_debounce: Duration::from_millis(200), reconcile_interval: Duration::from_secs(15), } @@ -120,6 +129,12 @@ impl WireguardConfig { self } + /// Sets the IPv4 overlay range, or disables IPv4 with `None`. + pub fn with_ipv4_range(mut self, range: Option<(Ipv4Addr, u8)>) -> Self { + self.ipv4_range = range; + self + } + /// Sets the reconciliation timings. pub fn with_reconcile(mut self, debounce: Duration, interval: Duration) -> Self { self.reconcile_debounce = debounce; @@ -150,6 +165,10 @@ pub struct NetworkOverview { pub overlay_prefix: IpAddr, /// Prefix length of the overlay subnet. pub overlay_prefix_len: u8, + /// This agent's IPv4 overlay address, when the overlay is dual stack. + pub overlay_address_v4: Option, + /// The IPv4 overlay range in use. + pub ipv4_range: Option<(Ipv4Addr, u8)>, /// Peers this agent knows about. pub peers: Vec, /// Unicast packets the operating system sent to an address no peer owns. @@ -174,6 +193,8 @@ pub struct PeerOverview { pub public_key: WgPublicKey, /// The overlay address derived for it locally. pub overlay_address: IpAddr, + /// Its IPv4 overlay address, once a tunnel exists and it won the address. + pub overlay_address_v4: Option, /// Whether a data plane link to it exists. pub has_link: bool, /// The running tunnel, once there is a link. @@ -309,6 +330,9 @@ impl WireguardPlugin { endpoint_id: *endpoint_id, public_key: announcement.public_key, overlay_address: IpAddr::V6(announcement.overlay_address), + overlay_address_v4: tunnels + .get(&announcement.public_key) + .and_then(|tunnel| tunnel.overlay_address_v4), has_link: state.links.contains_key(endpoint_id), tunnel: tunnels.get(&announcement.public_key).cloned(), }) @@ -323,6 +347,12 @@ impl WireguardPlugin { overlay_address: IpAddr::V6(overlay_address(network, &state.key.public())), overlay_prefix: IpAddr::V6(overlay_prefix(network)), overlay_prefix_len: OVERLAY_PREFIX_LEN, + overlay_address_v4: self + .worker + .config + .ipv4_range + .and_then(|range| overlay_address_v4(network, &state.key.public(), range)), + ipv4_range: self.worker.config.ipv4_range, peers, unroutable_packets: state .device @@ -437,10 +467,18 @@ impl Worker { name: name.clone(), address: overlay_address(network, &key.public()), prefix_len: OVERLAY_PREFIX_LEN, + address_v4: self.config.ipv4_range.and_then(|range| { + overlay_address_v4(network, &key.public(), range).map(|address| (address, range.1)) + }), mtu: self.config.mtu, }; let tun = self.tun_factory.create(request).await?; - let device = Arc::new(WireguardDevice::start(network, key, tun)); + let device = Arc::new(WireguardDevice::start( + network, + key, + tun, + self.config.ipv4_range, + )); let mut shared = self.lock_shared(); if let Some(state) = shared.networks.get_mut(&network) { diff --git a/src/dataplane/wireguard/tun.rs b/src/dataplane/wireguard/tun.rs index f2fa481..b78578f 100644 --- a/src/dataplane/wireguard/tun.rs +++ b/src/dataplane/wireguard/tun.rs @@ -12,7 +12,7 @@ //! * `SystemTun`, behind the `tun-device` feature, is a real TUN interface. //! Creating one needs `CAP_NET_ADMIN` on Linux or the equivalent elsewhere. -use std::net::Ipv6Addr; +use std::net::{Ipv4Addr, Ipv6Addr}; use std::sync::Arc; use bytes::Bytes; @@ -29,6 +29,8 @@ pub struct TunRequest { pub address: Ipv6Addr, /// Prefix length of the overlay subnet, so the OS routes it here. pub prefix_len: u8, + /// The IPv4 overlay address and its prefix length, when dual stack. + pub address_v4: Option<(Ipv4Addr, u8)>, /// Interface MTU. pub mtu: u32, } @@ -381,7 +383,7 @@ mod system { /// reason: duplicate address detection can never finish with no carrier, /// and the address would stay tentative and unusable. pub fn setup_commands(request: &TunRequest, user: &str) -> Vec { - vec![ + let mut commands = vec![ format!( "sudo ip tuntap add dev {} mode tun user {user}", request.name @@ -398,7 +400,16 @@ mod system { "sudo ip -6 address add {}/{} dev {} nodad", request.address, request.prefix_len, request.name ), - ] + ]; + if let Some((address, prefix_len)) = request.address_v4 { + // IPv4 is not sensitive to carrier the way IPv6 is, so it needs + // no extra settings. + commands.push(format!( + "sudo ip address add {address}/{prefix_len} dev {}", + request.name + )); + } + commands } fn current_user() -> String { @@ -578,7 +589,8 @@ fd559caf9652cb86321feac65c73bd84 05 40 00 08 tsun0 name: "tsun0".into(), address: "fd00::1".parse().unwrap(), prefix_len: 64, - mtu: 1100, + address_v4: Some(("100.64.1.2".parse().unwrap(), 10)), + mtu: 1280, }; let commands = setup_commands(&request, "someone"); @@ -586,11 +598,24 @@ fd559caf9652cb86321feac65c73bd84 05 40 00 08 tsun0 // must survive losing carrier, and it must not wait for duplicate // address detection that can never complete. let joined = commands.join("\n"); - let up = joined.find("link set dev tsun0 mtu 1100 up").unwrap(); + let up = joined.find("link set dev tsun0 mtu 1280 up").unwrap(); let keep = joined.find("keep_addr_on_down=1").unwrap(); let add = joined.find("address add fd00::1/64").unwrap(); assert!(up < keep && keep < add, "wrong order:\n{joined}"); assert!(joined.contains("nodad")); assert!(joined.contains("user someone")); + // IPv4 needs no carrier tricks, just the address. + assert!(joined.contains("ip address add 100.64.1.2/10 dev tsun0")); + + // An IPv6-only overlay says nothing about IPv4. + let v6_only = TunRequest { + address_v4: None, + ..request + }; + assert!( + !setup_commands(&v6_only, "someone") + .join("\n") + .contains("100.64") + ); } } diff --git a/src/ipc/mod.rs b/src/ipc/mod.rs index ef27bed..196b8c1 100644 --- a/src/ipc/mod.rs +++ b/src/ipc/mod.rs @@ -136,6 +136,8 @@ pub struct OverlayReport { pub mtu: u32, /// This agent's overlay address. pub address: String, + /// This agent's IPv4 overlay address, when the overlay is dual stack. + pub address_v4: Option, /// The subnet every member shares. pub prefix: String, /// Prefix length of that subnet. @@ -155,6 +157,8 @@ pub struct OverlayPeerReport { pub public_key: String, /// Its overlay address. pub address: String, + /// Its IPv4 overlay address, when it has one. + pub address_v4: Option, /// Seconds since the last WireGuard handshake. /// /// `None` means the tunnel has never handshaken and cannot carry traffic. @@ -229,10 +233,14 @@ impl StatusReport { let up = overlay.peers.iter().filter(|peer| peer.is_up()).count(); let _ = writeln!( out, - " overlay {} {}/{} mtu {} {}/{} tunnel(s) up", + " overlay {} {}/{}{} mtu {} {}/{} tunnel(s) up", overlay.interface, overlay.address, overlay.prefix_len, + match &overlay.address_v4 { + Some(v4) => format!(" and {v4}"), + None => String::new(), + }, overlay.mtu, up, overlay.peers.len() @@ -240,9 +248,13 @@ impl StatusReport { for peer in &overlay.peers { let _ = writeln!( out, - " {} {} {} tx {} rx {}{} {}", + " {} {}{} {} tx {} rx {}{} {}", &peer.public_key[..8.min(peer.public_key.len())], peer.address, + match &peer.address_v4 { + Some(v4) => format!(" / {v4}"), + None => String::new(), + }, match peer.handshake_secs_ago { Some(secs) => format!("handshake {secs}s ago"), None => "NOT HANDSHAKEN".to_string(), diff --git a/tests/wireguard.rs b/tests/wireguard.rs index 3a81ac0..fc37099 100644 --- a/tests/wireguard.rs +++ b/tests/wireguard.rs @@ -10,7 +10,7 @@ mod common; -use std::net::{IpAddr, Ipv6Addr}; +use std::net::{IpAddr, Ipv4Addr, Ipv6Addr}; use std::sync::Arc; use std::time::Duration; @@ -123,6 +123,24 @@ impl WgAgent { } } +/// Builds a minimal well-formed IPv4 packet. +fn ipv4_packet(source: Ipv4Addr, destination: Ipv4Addr, payload: &[u8]) -> Bytes { + let total = 20 + payload.len(); + let mut packet = Vec::with_capacity(total); + packet.push((4 << 4) | 5); // version 4, header length 5 words + packet.push(0); // dscp/ecn + packet.extend_from_slice(&(total as u16).to_be_bytes()); + packet.extend_from_slice(&[0, 0]); // identification + packet.extend_from_slice(&[0, 0]); // flags and fragment offset + packet.push(64); // ttl + packet.push(253); // an experimental protocol number + packet.extend_from_slice(&[0, 0]); // checksum, not verified by the overlay + packet.extend_from_slice(&source.octets()); + packet.extend_from_slice(&destination.octets()); + packet.extend_from_slice(payload); + Bytes::from(packet) +} + /// Builds a minimal well-formed IPv6 packet. fn ipv6_packet(source: Ipv6Addr, destination: Ipv6Addr, payload: &[u8]) -> Bytes { let mut packet = Vec::with_capacity(40 + payload.len()); @@ -244,6 +262,151 @@ async fn a_peer_cannot_send_from_an_address_it_does_not_own() { b.shutdown().await; } +#[tokio::test] +async fn the_overlay_carries_ipv4_alongside_ipv6() { + let discovery = SharedMemoryDiscovery::new(); + let (name, secret) = network("wg-dual-stack"); + + let a = WgAgent::spawn(&discovery, "ta").await; + let b = WgAgent::spawn(&discovery, "tb").await; + + let network_id = a.agent.join_network(&name, &secret).await.unwrap(); + b.agent.join_network(&name, &secret).await.unwrap(); + a.wait_for_tunnels(network_id, 1).await; + b.wait_for_tunnels(network_id, 1).await; + + let view_a = a.plugin.overview(network_id).unwrap(); + let view_b = b.plugin.overview(network_id).unwrap(); + + let v4_a = view_a.overlay_address_v4.expect("dual stack by default"); + let v4_b = view_b.overlay_address_v4.expect("dual stack by default"); + assert_ne!(v4_a, v4_b); + // Both inside the configured range. + for addr in [v4_a, v4_b] { + assert_eq!( + u32::from(addr) & 0xffc0_0000, + u32::from(Ipv4Addr::new(100, 64, 0, 0)) + ); + } + // Each side derived the other's address identically. + assert_eq!(view_a.peers[0].overlay_address_v4, Some(v4_b)); + assert_eq!(view_b.peers[0].overlay_address_v4, Some(v4_a)); + assert!(!view_a.peers[0].tunnel.as_ref().unwrap().ipv4_conflict); + + let tun_a = a.tun(network_id).await; + let tun_b = b.tun(network_id).await; + + // A real IPv4 packet through the same tunnel. + tun_a.push_from_os(ipv4_packet(v4_a, v4_b, b"ipv4 over the overlay")); + let received = tokio::time::timeout(common::DEADLINE, tun_b.pop_to_os()) + .await + .expect("the IPv4 packet should arrive") + .unwrap(); + assert_eq!(received[0] >> 4, 4, "still an IPv4 packet"); + assert_eq!(&received[12..16], &v4_a.octets()); + assert_eq!(&received[16..20], &v4_b.octets()); + assert_eq!(&received[20..], b"ipv4 over the overlay"); + + // IPv6 keeps working on the same tunnel. + let addr_a = a.overlay(network_id).await; + let addr_b = b.overlay(network_id).await; + tun_a.push_from_os(ipv6_packet(addr_a, addr_b, b"and ipv6 too")); + let received = tokio::time::timeout(common::DEADLINE, tun_b.pop_to_os()) + .await + .expect("the IPv6 packet should arrive") + .unwrap(); + assert_eq!(received[0] >> 4, 6); + assert_eq!(&received[40..], b"and ipv6 too"); + + a.shutdown().await; + b.shutdown().await; +} + +#[tokio::test] +async fn an_ipv4_source_a_peer_does_not_own_is_dropped() { + let discovery = SharedMemoryDiscovery::new(); + let (name, secret) = network("wg-v4-spoof"); + + let a = WgAgent::spawn(&discovery, "ta").await; + let b = WgAgent::spawn(&discovery, "tb").await; + let network_id = a.agent.join_network(&name, &secret).await.unwrap(); + b.agent.join_network(&name, &secret).await.unwrap(); + a.wait_for_tunnels(network_id, 1).await; + b.wait_for_tunnels(network_id, 1).await; + + let v4_a = a + .plugin + .overview(network_id) + .unwrap() + .overlay_address_v4 + .unwrap(); + let v4_b = b + .plugin + .overview(network_id) + .unwrap() + .overlay_address_v4 + .unwrap(); + let tun_a = a.tun(network_id).await; + let tun_b = b.tun(network_id).await; + + // A claims an IPv4 address that is not the one derived from its key. + let forged = Ipv4Addr::from(u32::from(v4_a) ^ 0x0000_00ff); + tun_a.push_from_os(ipv4_packet(forged, v4_b, b"spoofed v4")); + + wait_until("the spoofed IPv4 packet is dropped", || async { + let view = b.plugin.overview(network_id)?; + let tunnel = view.peers.first()?.tunnel.as_ref()?; + (tunnel.stats.dropped_wrong_source >= 1).then_some(()) + }) + .await; + + // The honest one still gets through. + tun_a.push_from_os(ipv4_packet(v4_a, v4_b, b"honest v4")); + let received = tokio::time::timeout(common::DEADLINE, tun_b.pop_to_os()) + .await + .expect("the honest packet should arrive") + .unwrap(); + assert_eq!(&received[20..], b"honest v4"); + + a.shutdown().await; + b.shutdown().await; +} + +#[tokio::test] +async fn an_ipv6_only_overlay_can_be_asked_for() { + let discovery = SharedMemoryDiscovery::new(); + let (name, secret) = network("wg-v6-only"); + + let a = WgAgent::spawn_with(&discovery, "ta", |config| config.with_ipv4_range(None)).await; + let b = WgAgent::spawn_with(&discovery, "tb", |config| config.with_ipv4_range(None)).await; + + let network_id = a.agent.join_network(&name, &secret).await.unwrap(); + b.agent.join_network(&name, &secret).await.unwrap(); + a.wait_for_tunnels(network_id, 1).await; + + let view = a.plugin.overview(network_id).unwrap(); + assert_eq!(view.overlay_address_v4, None); + assert_eq!(view.ipv4_range, None); + assert_eq!(view.peers[0].overlay_address_v4, None); + // No IPv4 configured is not a conflict. + assert!(!view.peers[0].tunnel.as_ref().unwrap().ipv4_conflict); + + // IPv6 is unaffected. + let addr_a = a.overlay(network_id).await; + let addr_b = b.overlay(network_id).await; + a.tun(network_id) + .await + .push_from_os(ipv6_packet(addr_a, addr_b, b"v6 only")); + let received = tokio::time::timeout(common::DEADLINE, b.tun(network_id).await.pop_to_os()) + .await + .expect("the packet should arrive") + .unwrap(); + assert_eq!(&received[40..], b"v6 only"); + + a.shutdown().await; + b.shutdown().await; +} + #[tokio::test] async fn packets_for_an_unknown_address_are_counted_not_broadcast() { let discovery = SharedMemoryDiscovery::new();