diff --git a/README.md b/README.md index 6aa34ca..f60204a 100644 --- a/README.md +++ b/README.md @@ -31,9 +31,10 @@ A working library with **real iroh connections** and integration tests: - status snapshots, an event stream and honest diagnostics; - configuration restored after a restart; - correct behaviour when the disposable cache is missing or corrupt; -- a **WireGuard data plane**, in userspace: its own key per network, a dual - stack overlay with deterministically derived addresses, real tunnels carried - over iroh, and address ownership enforced rather than believed; +- a **WireGuard data plane**, in userspace: its own key per network, an IPv6 + overlay with deterministically derived addresses and optional IPv4, real + tunnels carried over iroh, and address ownership enforced rather than + believed; - a **command line agent**, `tsunagi`, with a local control socket. ### What it deliberately does **not** do @@ -125,10 +126,23 @@ ping 100.65.243.53 `tx` and `rx` in the status should start moving. -The overlay is dual stack: every member derives both an IPv6 address, which -can never collide, and an IPv4 one in `100.64.0.0/10`, which very rarely can — -see [docs/wireguard.md](docs/wireguard.md#ipv4-alongside-ipv6). `--no-ipv4` -runs IPv6 only, `--ipv4-range` moves the range. +IPv6 works out of the box: each member's address is derived from the network +id and collides with essentially nothing. + +**IPv4 is opt-in**, because no IPv4 range is free on every host — +`100.64.0.0/10` is Tailscale's, `10.0.0.0/8` and `192.168.0.0/16` are +everywhere, `172.17.0.0/16` is Docker. Name one you know is unused, the same +one on every member: + +```bash +tsunagi up --network lab --secret "$SECRET" --wireguard --ipv4-range 10.77.0.0/16 +``` + +The range is part of how addresses are derived, so members configured +differently would misroute. It travels in the announcement purely so a +mismatch is reported instead: the offending peer gets no IPv4 and keeps +working over IPv6. See +[docs/wireguard.md](docs/wireguard.md#ipv4-alongside-ipv6). Notes: @@ -163,9 +177,11 @@ sudo ip tuntap add dev tsunjwc6dcrtmo5 mode tun user ab sudo ip link set dev tsunjwc6dcrtmo5 mtu 1280 up sudo sysctl -qw net.ipv6.conf.tsunjwc6dcrtmo5.keep_addr_on_down=1 sudo ip -6 address add fd80:1210:f724:f620:d1bb:f982:3b6e:19bd/64 dev tsunjwc6dcrtmo5 nodad -sudo ip address add 100.110.49.177/10 dev tsunjwc6dcrtmo5 ``` +If you asked for IPv4 with `--ipv4-range`, `tun-setup` adds an `ip address add` +line for it too. + The MTU is 1280 because that is the minimum IPv6 requires (RFC 8200). Linux disables IPv6 entirely on an interface below it — the per-device `/proc/sys/net/ipv6` entries vanish and `ip -6 address add` fails with diff --git a/docs/wireguard.md b/docs/wireguard.md index 999e675..b5913f9 100644 --- a/docs/wireguard.md +++ b/docs/wireguard.md @@ -85,26 +85,49 @@ Two consequences matter: ## IPv4 alongside IPv6 -The overlay is dual stack by default: every member also derives an IPv4 -address, from the same inputs, into `100.64.0.0/10` (RFC 6598 shared address -space — deliberately not RFC 1918, so it rarely clashes with the network the -machine is already on). The range is configurable, and IPv4 can be turned off. +The overlay carries IPv4 as well, but **it is off unless you name a range**, +and every member must name the same one: -**IPv4 is weaker than IPv6 here, and the difference is not cosmetic.** A 64 bit -interface identifier makes an IPv6 collision impossible in practice. IPv4 has -nothing like that much room: in a `/10` with 50 members the chance that two -derive the same address is roughly 0.03%. Small, but not zero, and a mesh with -no coordinator cannot simply allocate around it. +```bash +tsunagi up --network lab --secret "$SECRET" --wireguard --ipv4-range 10.77.0.0/16 +``` -So a collision is detected and resolved rather than assumed away: the member -whose WireGuard public key sorts lower keeps the address, a rule every member -computes identically and therefore agrees on without exchanging anything. The -other member ends up with **no IPv4 address** and is still fully reachable over -IPv6. The status output flags it. +Two reasons it has no default. -That is the honest summary: **IPv6 always works; IPv4 almost always works and -degrades predictably when it does not.** Allocating IPv4 properly needs the -agreed state described in [sync-model.md](sync-model.md). +**There is no IPv4 range that is free everywhere.** `100.64.0.0/10` is +Tailscale's and carrier-grade NAT's, `10.0.0.0/8` and `192.168.0.0/16` are on +half the networks in the world, `172.17.0.0/16` is Docker. Picking one +requires knowing what is already in use on every machine that will join, which +is the operator's knowledge, not ours. IPv6 needs none of this: a ULA derived +from the network id collides with essentially nothing. + +**The range is an input to the derivation.** Each agent computes every peer's +address itself, so two members configured with different ranges would derive +different addresses for each other and IPv4 would silently misroute. The range +therefore travels in the announcement — not as a request, and never trusted, +but so that a mismatch is *detected*. When it happens, the offending peer gets +no IPv4 address here, keeps working over IPv6, and the reason is reported: + +```text +! wireguard: peer SDsEb/WF is configured with the IPv4 overlay range + 10.81.0.0/16 but this agent uses 10.80.0.0/16; every member must use the + same one. That peer has no IPv4 address here and is reachable over IPv6 only. +``` + +**IPv4 addresses can also collide with each other.** A 64 bit interface +identifier makes an IPv6 collision impossible in practice; IPv4 has nothing +like that room. In a `/16` with 50 members the chance that two members derive +the same address is roughly 2%. A mesh with no coordinator cannot allocate +around it, so the collision is resolved instead: the member whose WireGuard +public key sorts lower keeps the address, a rule every member computes +identically and therefore agrees on without exchanging anything. The other +member has no IPv4 address and remains reachable over IPv6. Pick a roomy +range — a `/16` for a handful of machines, larger for more — and the odds stay +small. + +The honest summary: **IPv6 always works. IPv4 is opt-in, needs agreement, and +degrades predictably when it does not get it.** Allocating IPv4 properly needs +the agreed state described in [sync-model.md](sync-model.md). ## Address ownership is enforced, not announced @@ -223,8 +246,8 @@ async fn main() -> Result<()> { * **Full mesh only.** Every member runs a tunnel to every other member. Routing through an intermediate participant is not implemented. -* **IPv4 addressing can collide.** See above: it is resolved deterministically - and the loser keeps IPv6, but a proper allocator needs agreed state. +* **IPv4 is opt-in, must be agreed, and can collide.** See above. A proper + allocator needs agreed state. * **No routes, DNS or firewall rules.** The plugin creates its interface and nothing else. Anything beyond the overlay `/64` is the operator's business. * **Membership is session-scoped.** A peer leaves the overlay when its control diff --git a/src/bin/tsunagi.rs b/src/bin/tsunagi.rs index 2942034..f995822 100644 --- a/src/bin/tsunagi.rs +++ b/src/bin/tsunagi.rs @@ -16,7 +16,7 @@ use tsunagi::agent::Event; use tsunagi::config::{AgentConfig, StoragePaths, TransportPolicy}; use tsunagi::dataplane::IpPlugin; use tsunagi::dataplane::wireguard::{ - MemoryTunFactory, TunFactory, WireguardConfig, WireguardPlugin, + Ipv4Range, MemoryTunFactory, TunFactory, WireguardConfig, WireguardPlugin, }; use tsunagi::discovery::{CompositeDiscovery, NetworkDiscovery, StaticBootstrap}; use tsunagi::identity::{NetworkName, NetworkSecret}; @@ -99,43 +99,29 @@ struct TunSetupArgs { #[arg(long)] wg_mtu: Option, - /// Match `tsunagi up --no-ipv4`. - #[arg(long)] - no_ipv4: bool, - /// Match `tsunagi up --ipv4-range`. - #[arg(long, value_name = "CIDR", conflicts_with = "no_ipv4")] + #[arg(long, value_name = "CIDR")] ipv4_range: Option, } -/// Parses `address/prefix` into an IPv4 range. -fn parse_ipv4_range(text: &str) -> Result<(std::net::Ipv4Addr, u8), String> { - let (address, prefix) = text - .split_once('/') - .ok_or_else(|| format!("`{text}` is not an address with a prefix, e.g. 100.64.0.0/10"))?; - let address = address - .parse() - .map_err(|err| format!("`{address}` is not an IPv4 address: {err}"))?; - let prefix: u8 = prefix - .parse() - .map_err(|err| format!("`{prefix}` is not a prefix length: {err}"))?; - if prefix > 30 { - return Err(format!("a /{prefix} has no room for hosts")); - } - Ok((address, prefix)) +/// Strips the error type's own prefix, which is about peers rather than flags. +fn plain_reason(err: &tsunagi::dataplane::PluginError) -> String { + let text = err.to_string(); + text.split_once(": ") + .map(|(_, rest)| rest.to_string()) + .unwrap_or(text) } -/// Resolves the IPv4 overlay range from the flags. +/// Resolves the IPv4 overlay range from the flag. fn resolve_ipv4_range( - no_ipv4: bool, range: Option<&String>, -) -> Result, Box> { - if no_ipv4 { - return Ok(None); - } +) -> Result, Box> { match range { - Some(text) => Ok(Some(parse_ipv4_range(text)?)), - None => Ok(Some(tsunagi::dataplane::wireguard::DEFAULT_IPV4_RANGE)), + Some(text) => Ok(Some(text.parse::().map_err(|err| { + // The underlying error type is about peers; reword it for a flag. + format!("--ipv4-range {text}: {}", plain_reason(&err)) + })?)), + None => Ok(None), } } @@ -251,12 +237,14 @@ struct UpArgs { #[arg(long)] wg_mtu: Option, - /// Run an IPv6-only overlay instead of dual stack. - #[arg(long)] - no_ipv4: bool, - - /// IPv4 overlay range, as `address/prefix`. Defaults to 100.64.0.0/10. - #[arg(long, value_name = "CIDR", conflicts_with = "no_ipv4")] + /// Also run an IPv4 overlay in this range, as `address/prefix`. + /// + /// Off unless given: no IPv4 range is free on every host. Pick one you + /// know is unused everywhere — not 100.64.0.0/10, which is Tailscale's + /// and carrier-grade NAT's. Every member must pass the same range; a + /// mismatch is detected and reported rather than silently misrouted. + /// IPv6 needs none of this and is always on. + #[arg(long, value_name = "CIDR")] ipv4_range: Option, /// How often to print a status summary, in seconds. Zero disables it. @@ -412,7 +400,7 @@ async fn tun_setup(args: TunSetupArgs) -> Result<(), Box> let interface = interface_name(&args.wg_prefix, network)?; let address = overlay_address(network, &key.public()); - let ipv4_range = resolve_ipv4_range(args.no_ipv4, args.ipv4_range.as_ref())?; + let ipv4_range = resolve_ipv4_range(args.ipv4_range.as_ref())?; let mtu = args.wg_mtu.unwrap_or(DEFAULT_MTU); let user = args.user.unwrap_or_else(|| { std::env::var("SUDO_USER") @@ -422,10 +410,10 @@ async fn tun_setup(args: TunSetupArgs) -> Result<(), Box> println!("# Network {name} ({network})"); println!("# Interface {interface}, address {address}/{OVERLAY_PREFIX_LEN}, mtu {mtu}"); - if let Some((base, prefix)) = ipv4_range - && let Some(v4) = overlay_address_v4(network, &key.public(), (base, prefix)) + if let Some(range) = ipv4_range + && let Some(v4) = overlay_address_v4(network, &key.public(), range) { - println!("# IPv4 overlay address {v4}/{prefix}"); + println!("# IPv4 overlay address {v4}/{}", range.prefix_len); } println!("# Run once as root; then run `tsunagi up` as {user}."); println!( @@ -440,10 +428,13 @@ async fn tun_setup(args: TunSetupArgs) -> Result<(), Box> println!("sudo ip link set dev {interface} mtu {mtu} up"); println!("sudo sysctl -qw net.ipv6.conf.{interface}.keep_addr_on_down=1"); println!("sudo ip -6 address add {address}/{OVERLAY_PREFIX_LEN} dev {interface} nodad"); - if let Some((base, prefix)) = ipv4_range - && let Some(v4) = overlay_address_v4(network, &key.public(), (base, prefix)) + if let Some(range) = ipv4_range + && let Some(v4) = overlay_address_v4(network, &key.public(), range) { - println!("sudo ip address add {v4}/{prefix} dev {interface}"); + println!( + "sudo ip address add {v4}/{} dev {interface}", + range.prefix_len + ); } println!("\n# To check it afterwards:"); println!("ip -6 addr show dev {interface}"); @@ -547,6 +538,13 @@ async fn up(args: UpArgs) -> Result<(), Box> { let secret = load_secret(args.secret.as_deref(), args.secret_file.as_deref())?; let paths = args.paths.resolve()?; + // Parsed up front so a typo is reported immediately, and so the option is + // never silently ignored when the data plane is off. + let ipv4_range = resolve_ipv4_range(args.ipv4_range.as_ref())?; + if ipv4_range.is_some() && !args.wireguard { + return Err("--ipv4-range only applies together with --wireguard".into()); + } + let mut bootstrap: Vec = Vec::new(); for peer in &args.peers { bootstrap.push(parse_peer(peer)?); @@ -576,7 +574,7 @@ async fn up(args: UpArgs) -> Result<(), Box> { }; let mut wg = WireguardConfig::new(paths.state_dir.join("wireguard")) .with_interface_prefix(args.wg_prefix.clone()) - .with_ipv4_range(resolve_ipv4_range(args.no_ipv4, args.ipv4_range.as_ref())?); + .with_ipv4_range(ipv4_range); if let Some(mtu) = args.wg_mtu { wg = wg.with_mtu(mtu); } diff --git a/src/dataplane/wireguard/announcement.rs b/src/dataplane/wireguard/announcement.rs index c8a5a27..e28f73c 100644 --- a/src/dataplane/wireguard/announcement.rs +++ b/src/dataplane/wireguard/announcement.rs @@ -18,10 +18,14 @@ use crate::dataplane::PluginError; use crate::identity::NetworkId; use super::keys::WgPublicKey; -use super::overlay::overlay_address; +use super::overlay::{Ipv4Range, overlay_address}; /// Version of the announcement format. -pub const ANNOUNCEMENT_VERSION: u16 = 1; +/// +/// Bumped to 2 when the IPv4 overlay range was added. postcard is not +/// self-describing, so an older peer cannot read a newer announcement; the +/// mismatch is reported rather than misparsed. +pub const ANNOUNCEMENT_VERSION: u16 = 2; /// What one participant advertises for the WireGuard data plane. #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] @@ -32,9 +36,15 @@ pub struct WgAnnouncement { pub public_key: [u8; 32], /// The overlay address the peer believes it has. /// - /// Carried for diagnostics and cross-checking only. `AllowedIPs` are - /// always derived locally, never taken from this field. + /// Carried for diagnostics and cross-checking only. Addresses are always + /// derived locally, never taken from this field. pub overlay_address: Ipv6Addr, + /// The IPv4 overlay range this peer is configured with, if any. + /// + /// Not a request and not trusted: it exists so that two members who were + /// configured differently find out, instead of silently deriving + /// different addresses for each other and misrouting IPv4. + pub ipv4_range: Option, } /// A peer announcement that has been validated against a specific network. @@ -44,15 +54,22 @@ pub struct ValidatedAnnouncement { pub public_key: WgPublicKey, /// The overlay address derived locally for this key. Authoritative. pub overlay_address: Ipv6Addr, + /// The IPv4 overlay range the peer is configured with. + pub ipv4_range: Option, } impl WgAnnouncement { /// Builds this agent's announcement. - pub fn new(network: NetworkId, public_key: &WgPublicKey) -> Self { + pub fn new( + network: NetworkId, + public_key: &WgPublicKey, + ipv4_range: Option, + ) -> Self { Self { version: ANNOUNCEMENT_VERSION, public_key: *public_key.as_bytes(), overlay_address: overlay_address(network, public_key), + ipv4_range, } } @@ -83,7 +100,8 @@ impl WgAnnouncement { ) -> Result { if self.version != ANNOUNCEMENT_VERSION { return Err(PluginError::Rejected(format!( - "unsupported WireGuard announcement version {} (this build speaks {ANNOUNCEMENT_VERSION})", + "peer speaks WireGuard announcement version {} but this build speaks \ + {ANNOUNCEMENT_VERSION}; one of the two needs updating", self.version ))); } @@ -109,9 +127,18 @@ impl WgAnnouncement { )); } + if let Some(range) = self.ipv4_range + && range.prefix_len > 30 + { + return Err(PluginError::Rejected(format!( + "announced IPv4 range {range} has no room for hosts" + ))); + } + Ok(ValidatedAnnouncement { public_key, overlay_address: derived, + ipv4_range: self.ipv4_range, }) } } @@ -139,7 +166,7 @@ mod tests { let peer = WgSecretKey::generate().public(); let local = WgSecretKey::generate().public(); - let payload = WgAnnouncement::new(id, &peer).encode().unwrap(); + let payload = WgAnnouncement::new(id, &peer, None).encode().unwrap(); let validated = WgAnnouncement::decode_and_validate(&payload, id, &local).unwrap(); assert_eq!(validated.public_key, peer); @@ -152,7 +179,7 @@ mod tests { // carried here, so there is nothing for a peer to lie about. let id = network("identity-only"); let peer = WgSecretKey::generate().public(); - let payload = WgAnnouncement::new(id, &peer).encode().unwrap(); + let payload = WgAnnouncement::new(id, &peer, None).encode().unwrap(); assert!( payload.len() < 80, "the announcement should stay tiny, got {} bytes", @@ -168,7 +195,7 @@ mod tests { let local = WgSecretKey::generate().public(); // An attacker claims the victim's overlay address with its own key. - let mut forged = WgAnnouncement::new(id, &attacker); + let mut forged = WgAnnouncement::new(id, &attacker, None); forged.overlay_address = overlay_address(id, &victim); let result = WgAnnouncement::decode_and_validate(&forged.encode().unwrap(), id, &local); @@ -185,7 +212,7 @@ mod tests { let peer = WgSecretKey::generate().public(); let local = WgSecretKey::generate().public(); - let payload = WgAnnouncement::new(there, &peer).encode().unwrap(); + let payload = WgAnnouncement::new(there, &peer, None).encode().unwrap(); assert!(WgAnnouncement::decode_and_validate(&payload, here, &local).is_err()); } @@ -200,7 +227,7 @@ mod tests { let wrong_version = WgAnnouncement { version: ANNOUNCEMENT_VERSION + 1, - ..WgAnnouncement::new(id, &peer) + ..WgAnnouncement::new(id, &peer, None) }; assert!( WgAnnouncement::decode_and_validate(&wrong_version.encode().unwrap(), id, &local) @@ -209,18 +236,38 @@ mod tests { let zero_key = WgAnnouncement { public_key: [0u8; 32], - ..WgAnnouncement::new(id, &peer) + ..WgAnnouncement::new(id, &peer, None) }; assert!( WgAnnouncement::decode_and_validate(&zero_key.encode().unwrap(), id, &local).is_err() ); } + #[test] + fn the_ipv4_range_travels_so_a_mismatch_can_be_seen() { + let id = network("ranges"); + let peer = WgSecretKey::generate().public(); + let local = WgSecretKey::generate().public(); + let range = Some(Ipv4Range::new("10.9.0.0".parse().unwrap(), 16).unwrap()); + + let payload = WgAnnouncement::new(id, &peer, range).encode().unwrap(); + let validated = WgAnnouncement::decode_and_validate(&payload, id, &local).unwrap(); + assert_eq!(validated.ipv4_range, range); + + // A range with no usable hosts is nonsense and is refused. + let mut bad = WgAnnouncement::new(id, &peer, range); + bad.ipv4_range = Some(Ipv4Range { + base: "10.9.0.0".parse().unwrap(), + prefix_len: 31, + }); + assert!(WgAnnouncement::decode_and_validate(&bad.encode().unwrap(), id, &local).is_err()); + } + #[test] fn a_peer_cannot_claim_our_own_key() { let id = network("self"); let local = WgSecretKey::generate().public(); - let payload = WgAnnouncement::new(id, &local).encode().unwrap(); + let payload = WgAnnouncement::new(id, &local, None).encode().unwrap(); assert!(WgAnnouncement::decode_and_validate(&payload, id, &local).is_err()); } @@ -228,7 +275,7 @@ mod tests { fn announcements_stay_well_under_the_capability_payload_limit() { let id = network("size"); let peer = WgSecretKey::generate().public(); - let payload = WgAnnouncement::new(id, &peer).encode().unwrap(); + let payload = WgAnnouncement::new(id, &peer, None).encode().unwrap(); assert!( payload.len() < crate::config::Limits::default().max_capability_data_len, "announcement is {} bytes", diff --git a/src/dataplane/wireguard/device.rs b/src/dataplane/wireguard/device.rs index e0d9994..d48b107 100644 --- a/src/dataplane/wireguard/device.rs +++ b/src/dataplane/wireguard/device.rs @@ -42,7 +42,7 @@ use crate::dataplane::transport::{SharedLink, TransportError}; use crate::identity::NetworkId; use super::keys::{WgPublicKey, WgSecretKey}; -use super::overlay::{overlay_address, overlay_address_v4}; +use super::overlay::{Ipv4Range, overlay_address}; use super::packet::IpHeader; use super::tun::TunDevice; @@ -193,7 +193,7 @@ struct Inner { private_key: WgSecretKey, tun: Arc, /// The IPv4 overlay range, when the overlay is dual stack. - ipv4_range: Option<(Ipv4Addr, u8)>, + ipv4_range: Option, peers: RwLock>>, /// Both families, so one lookup routes any packet. routes: RwLock>, @@ -225,7 +225,7 @@ impl WireguardDevice { network: NetworkId, private_key: WgSecretKey, tun: Arc, - ipv4_range: Option<(Ipv4Addr, u8)>, + ipv4_range: Option, ) -> Self { let inner = Arc::new(Inner { network, @@ -260,10 +260,15 @@ impl WireguardDevice { } /// Adds or replaces a peer and starts its tunnel. + /// + /// `overlay_v4` is decided by the caller, because only it knows whether + /// both sides agree on an IPv4 range. `None` means this peer is reachable + /// over IPv6 only. pub fn add_peer( &self, endpoint_id: EndpointId, public_key: WgPublicKey, + overlay_v4: Option, link: SharedLink, keepalive: Option, ) -> Result<(), PluginError> { @@ -284,7 +289,7 @@ impl WireguardDevice { ); let overlay = overlay_address(self.inner.network, &public_key); - let overlay_v4 = self.claim_ipv4(&public_key); + let overlay_v4 = self.claim_ipv4(&public_key, overlay_v4); let peer = Arc::new(Peer { endpoint_id, public_key, @@ -321,9 +326,8 @@ impl WireguardDevice { /// so they all agree on the outcome without talking about it. The other /// member simply has no IPv4 address; it is still fully reachable over /// IPv6, which never collides. - fn claim_ipv4(&self, public_key: &WgPublicKey) -> Option { - let range = self.inner.ipv4_range?; - let wanted = overlay_address_v4(self.inner.network, public_key, range)?; + fn claim_ipv4(&self, public_key: &WgPublicKey, wanted: Option) -> Option { + let wanted = wanted?; let holder = read_lock(&self.inner.routes) .get(&IpAddr::V4(wanted)) diff --git a/src/dataplane/wireguard/mod.rs b/src/dataplane/wireguard/mod.rs index 4ce9a77..243a9d4 100644 --- a/src/dataplane/wireguard/mod.rs +++ b/src/dataplane/wireguard/mod.rs @@ -55,7 +55,8 @@ pub use config::{Cidr, DEFAULT_INTERFACE_PREFIX, MAX_INTERFACE_NAME_LEN, interfa pub use device::{PeerHealth, PeerStats, PeerSummary, WireguardDevice}; pub use keys::{WgPublicKey, WgSecretKey}; pub use overlay::{ - DEFAULT_IPV4_RANGE, OVERLAY_PREFIX_LEN, overlay_address, overlay_address_v4, overlay_prefix, + Ipv4Range, OVERLAY_PREFIX_LEN, RFC6598_SHARED_RANGE, overlay_address, overlay_address_v4, + overlay_prefix, }; pub use packet::IpHeader; pub use plugin::{ diff --git a/src/dataplane/wireguard/overlay.rs b/src/dataplane/wireguard/overlay.rs index d68a151..8a6942a 100644 --- a/src/dataplane/wireguard/overlay.rs +++ b/src/dataplane/wireguard/overlay.rs @@ -22,8 +22,11 @@ use std::net::{Ipv4Addr, Ipv6Addr}; +use serde::{Deserialize, Serialize}; use sha2::{Digest, Sha256}; +use crate::dataplane::PluginError; + use crate::identity::NetworkId; use super::keys::WgPublicKey; @@ -37,12 +40,76 @@ pub const OVERLAY_PREFIX_LEN: u8 = 64; /// Prefix length of one member's address inside the overlay. pub const OVERLAY_HOST_PREFIX_LEN: u8 = 128; -/// Default IPv4 overlay range: RFC 6598 shared address space. +/// An IPv4 range the overlay can be derived into. /// -/// Deliberately not RFC 1918, so it rarely collides with the home or office -/// network the machine is already on. It can collide with a carrier-grade NAT -/// that uses the same range, which is why it is configurable. -pub const DEFAULT_IPV4_RANGE: (Ipv4Addr, u8) = (Ipv4Addr::new(100, 64, 0, 0), 10); +/// Every member of a network must be configured with the same one, because +/// addresses are derived from it. See [`crate::dataplane::wireguard::plugin::WireguardConfig::ipv4_range`]. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +pub struct Ipv4Range { + /// Base address of the range. + pub base: Ipv4Addr, + /// Prefix length, at most 30 so there is room for hosts. + pub prefix_len: u8, +} + +impl Ipv4Range { + /// Builds a range, rejecting one with no room for hosts. + pub fn new(base: Ipv4Addr, prefix_len: u8) -> Result { + if prefix_len > 30 { + return Err(PluginError::Rejected(format!( + "a /{prefix_len} has no room for hosts; use /30 or larger" + ))); + } + Ok(Self { base, prefix_len }) + } + + /// Whether an address falls inside the range. + pub fn contains(&self, address: Ipv4Addr) -> bool { + let host_bits = 32 - u32::from(self.prefix_len); + let mask = if host_bits >= 32 { + 0 + } else { + u32::MAX << host_bits + }; + u32::from(address) & mask == u32::from(self.base) & mask + } +} + +impl std::fmt::Display for Ipv4Range { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "{}/{}", self.base, self.prefix_len) + } +} + +impl std::str::FromStr for Ipv4Range { + type Err = PluginError; + + fn from_str(text: &str) -> Result { + let (base, prefix) = text.split_once('/').ok_or_else(|| { + PluginError::Rejected(format!( + "`{text}` is not an address with a prefix, for example 10.77.0.0/16" + )) + })?; + let base = base.parse().map_err(|err| { + PluginError::Rejected(format!("`{base}` is not an IPv4 address: {err}")) + })?; + let prefix_len = prefix.parse().map_err(|err| { + PluginError::Rejected(format!("`{prefix}` is not a prefix length: {err}")) + })?; + Self::new(base, prefix_len) + } +} + +/// RFC 6598 shared address space, offered only as a reference point. +/// +/// **Not a default, and usually a bad choice.** Tailscale uses exactly this +/// range, and so does carrier-grade NAT, so a machine running either will +/// collide with it. There is no IPv4 range that is free on every host, which +/// is why the IPv4 overlay has no default at all and must be configured. +pub const RFC6598_SHARED_RANGE: Ipv4Range = Ipv4Range { + base: Ipv4Addr::new(100, 64, 0, 0), + prefix_len: 10, +}; fn push_lp(out: &mut Vec, bytes: &[u8]) { let len = u32::try_from(bytes.len()).unwrap_or(u32::MAX); @@ -107,9 +174,9 @@ pub fn overlay_address(network: NetworkId, key: &WgPublicKey) -> Ipv6Addr { pub fn overlay_address_v4( network: NetworkId, key: &WgPublicKey, - range: (Ipv4Addr, u8), + range: Ipv4Range, ) -> Option { - let (base, prefix_len) = range; + let Ipv4Range { base, prefix_len } = range; if prefix_len > 32 { return None; } @@ -185,14 +252,14 @@ mod tests { #[test] fn ipv4_addresses_land_inside_the_range_and_avoid_its_edges() { let id = network("v4"); - let range = DEFAULT_IPV4_RANGE; + let range = RFC6598_SHARED_RANGE; for byte in 0..64u8 { let key = WgPublicKey::from_bytes([byte; 32]); let addr = overlay_address_v4(id, &key, range).unwrap(); let raw = u32::from(addr); assert_eq!( raw & 0xffc0_0000, - u32::from(range.0), + u32::from(range.base), "outside 100.64.0.0/10" ); // Never the network address and never the broadcast address. @@ -201,12 +268,25 @@ mod tests { } } + #[test] + fn a_range_parses_and_prints_round_trip() { + let range: Ipv4Range = "10.77.0.0/16".parse().unwrap(); + assert_eq!(range.to_string(), "10.77.0.0/16"); + assert!(range.contains("10.77.3.4".parse().unwrap())); + assert!(!range.contains("10.78.3.4".parse().unwrap())); + + assert!("10.77.0.0".parse::().is_err()); + assert!("nonsense/16".parse::().is_err()); + assert!("10.77.0.0/zz".parse::().is_err()); + assert!("10.77.0.0/31".parse::().is_err()); + } + #[test] fn ipv4_derivation_is_deterministic_and_scoped_like_ipv6() { let key = WgPublicKey::from_bytes([9u8; 32]); let first = network("one"); let second = network("two"); - let range = DEFAULT_IPV4_RANGE; + let range = RFC6598_SHARED_RANGE; assert_eq!( overlay_address_v4(first, &key, range), @@ -223,7 +303,11 @@ mod tests { // A different range moves everybody. assert_ne!( overlay_address_v4(first, &key, range), - overlay_address_v4(first, &key, (Ipv4Addr::new(10, 0, 0, 0), 8)) + overlay_address_v4( + first, + &key, + Ipv4Range::new(Ipv4Addr::new(10, 0, 0, 0), 8).unwrap() + ) ); } @@ -231,14 +315,16 @@ mod tests { fn a_range_with_no_room_yields_nothing() { let id = network("tiny"); let key = WgPublicKey::from_bytes([1u8; 32]); - // /31 and /32 have no usable host addresses. - assert!(overlay_address_v4(id, &key, (Ipv4Addr::new(10, 0, 0, 0), 31)).is_none()); - assert!(overlay_address_v4(id, &key, (Ipv4Addr::new(10, 0, 0, 1), 32)).is_none()); - assert!(overlay_address_v4(id, &key, (Ipv4Addr::new(10, 0, 0, 0), 33)).is_none()); + // /31 and /32 have no usable host addresses, so they are refused. + assert!(Ipv4Range::new(Ipv4Addr::new(10, 0, 0, 0), 31).is_err()); + assert!(Ipv4Range::new(Ipv4Addr::new(10, 0, 0, 1), 32).is_err()); + assert!(Ipv4Range::new(Ipv4Addr::new(10, 0, 0, 0), 33).is_err()); // A /30 has two usable addresses. - assert!(overlay_address_v4(id, &key, (Ipv4Addr::new(10, 0, 0, 0), 30)).is_some()); + let small = Ipv4Range::new(Ipv4Addr::new(10, 0, 0, 0), 30).unwrap(); + assert!(overlay_address_v4(id, &key, small).is_some()); // A /0 must not overflow. - assert!(overlay_address_v4(id, &key, (Ipv4Addr::UNSPECIFIED, 0)).is_some()); + let everything = Ipv4Range::new(Ipv4Addr::UNSPECIFIED, 0).unwrap(); + assert!(overlay_address_v4(id, &key, everything).is_some()); } #[test] diff --git a/src/dataplane/wireguard/plugin.rs b/src/dataplane/wireguard/plugin.rs index 08211d9..f52a063 100644 --- a/src/dataplane/wireguard/plugin.rs +++ b/src/dataplane/wireguard/plugin.rs @@ -44,7 +44,7 @@ use super::config::{DEFAULT_INTERFACE_PREFIX, interface_name}; use super::device::{PeerSummary, WireguardDevice}; use super::keys::{WgPublicKey, WgSecretKey}; use super::overlay::{ - DEFAULT_IPV4_RANGE, OVERLAY_PREFIX_LEN, overlay_address, overlay_address_v4, overlay_prefix, + Ipv4Range, OVERLAY_PREFIX_LEN, overlay_address, overlay_address_v4, overlay_prefix, }; use super::store::WgKeyStore; use super::tun::{TunFactory, TunRequest}; @@ -90,10 +90,19 @@ pub struct WireguardConfig { pub mtu: u32, /// IPv4 overlay range, or `None` for an IPv6-only overlay. /// - /// IPv6 addresses are derived collision-free; IPv4 ones cannot be, so a - /// collision is detected and resolved deterministically instead. See - /// `docs/wireguard.md`. - pub ipv4_range: Option<(Ipv4Addr, u8)>, + /// **Every member of a network must configure the same range.** Addresses + /// are derived from it, so two members configured differently would + /// derive different addresses for each other. The range travels in the + /// announcement purely so that such a mismatch is detected and reported + /// instead of silently misrouting. + /// + /// There is no default, because no IPv4 range is free on every host: + /// `100.64.0.0/10` belongs to Tailscale and to carrier-grade NAT, + /// `10.0.0.0/8` and `192.168.0.0/16` are everywhere, `172.17.0.0/16` is + /// Docker. Pick one you know is unused on every machine that will join. + /// IPv6 needs none of this: its addresses are derived from the network + /// id and never collide. + pub ipv4_range: Option, /// How long to coalesce changes before reconciling. pub reconcile_debounce: Duration, /// How often to reconcile anyway, which is also when a packet interface @@ -109,7 +118,9 @@ impl WireguardConfig { interface_prefix: DEFAULT_INTERFACE_PREFIX.to_string(), keepalive: Some(25), mtu: DEFAULT_MTU, - ipv4_range: Some(DEFAULT_IPV4_RANGE), + // Off by default: no IPv4 range is free on every host. See + // `with_ipv4_range`. + ipv4_range: None, reconcile_debounce: Duration::from_millis(200), reconcile_interval: Duration::from_secs(15), } @@ -130,7 +141,9 @@ impl WireguardConfig { } /// Sets the IPv4 overlay range, or disables IPv4 with `None`. - pub fn with_ipv4_range(mut self, range: Option<(Ipv4Addr, u8)>) -> Self { + /// + /// Must match on every member; see the field documentation. + pub fn with_ipv4_range(mut self, range: Option) -> Self { self.ipv4_range = range; self } @@ -168,7 +181,7 @@ pub struct NetworkOverview { /// This agent's IPv4 overlay address, when the overlay is dual stack. pub overlay_address_v4: Option, /// The IPv4 overlay range in use. - pub ipv4_range: Option<(Ipv4Addr, u8)>, + pub ipv4_range: Option, /// Peers this agent knows about. pub peers: Vec, /// Unicast packets the operating system sent to an address no peer owns. @@ -467,9 +480,11 @@ impl Worker { name: name.clone(), address: overlay_address(network, &key.public()), prefix_len: OVERLAY_PREFIX_LEN, - address_v4: self.config.ipv4_range.and_then(|range| { - overlay_address_v4(network, &key.public(), range).map(|address| (address, range.1)) - }), + address_v4: self + .config + .ipv4_range + .and_then(|range| overlay_address_v4(network, &key.public(), range)), + prefix_len_v4: self.config.ipv4_range.map_or(0, |range| range.prefix_len), mtu: self.config.mtu, }; let tun = self.tun_factory.create(request).await?; @@ -503,6 +518,7 @@ impl Worker { let mut wanted: Vec = Vec::new(); let mut too_small: Vec<(usize, usize)> = Vec::new(); + let mut mismatched: Vec<(WgPublicKey, Ipv4Range, Ipv4Range)> = Vec::new(); for (endpoint_id, announcement) in &state.announcements { let Some(link) = state.links.get(endpoint_id) else { continue; @@ -524,9 +540,24 @@ impl Worker { too_small.push((available, needed)); } + // A peer only gets an IPv4 address if both sides were configured + // with the same range. Otherwise the two would derive different + // addresses for each other and IPv4 would silently misroute. + let peer_v4 = match (self.config.ipv4_range, announcement.ipv4_range) { + (Some(ours), Some(theirs)) if ours == theirs => { + overlay_address_v4(network, &announcement.public_key, ours) + } + (Some(ours), Some(theirs)) => { + mismatched.push((announcement.public_key, ours, theirs)); + None + } + (Some(_), None) | (None, Some(_)) | (None, None) => None, + }; + if let Err(err) = device.add_peer( *endpoint_id, announcement.public_key, + peer_v4, Arc::clone(link), self.config.keepalive, ) { @@ -536,6 +567,18 @@ impl Worker { device.retain_peers(&wanted); drop(shared); + for (key, ours, theirs) in mismatched { + self.report( + network, + format!( + "peer {} is configured with the IPv4 overlay range {theirs} but this agent \ + uses {ours}; every member must use the same one. That peer has no IPv4 \ + address here and is reachable over IPv6 only.", + key.fmt_short() + ), + ); + } + for (available, needed) in too_small { self.report( network, @@ -664,7 +707,8 @@ impl IpPlugin for WireguardPlugin { }; // Identity only. Where to send packets is the transport's business. - let announcement = WgAnnouncement::new(network, &state.key.public()); + let announcement = + WgAnnouncement::new(network, &state.key.public(), self.worker.config.ipv4_range); Ok(Some(PluginCapability { protocol: WIREGUARD_PROTOCOL.to_string(), version: super::announcement::ANNOUNCEMENT_VERSION, diff --git a/src/dataplane/wireguard/tun.rs b/src/dataplane/wireguard/tun.rs index b78578f..a7078a9 100644 --- a/src/dataplane/wireguard/tun.rs +++ b/src/dataplane/wireguard/tun.rs @@ -12,7 +12,7 @@ //! * `SystemTun`, behind the `tun-device` feature, is a real TUN interface. //! Creating one needs `CAP_NET_ADMIN` on Linux or the equivalent elsewhere. -use std::net::{Ipv4Addr, Ipv6Addr}; +use std::net::Ipv6Addr; use std::sync::Arc; use bytes::Bytes; @@ -29,8 +29,10 @@ pub struct TunRequest { pub address: Ipv6Addr, /// Prefix length of the overlay subnet, so the OS routes it here. pub prefix_len: u8, - /// The IPv4 overlay address and its prefix length, when dual stack. - pub address_v4: Option<(Ipv4Addr, u8)>, + /// The IPv4 overlay address this host answers to, when dual stack. + pub address_v4: Option, + /// Prefix length of the IPv4 overlay range. + pub prefix_len_v4: u8, /// Interface MTU. pub mtu: u32, } @@ -401,7 +403,8 @@ mod system { request.address, request.prefix_len, request.name ), ]; - if let Some((address, prefix_len)) = request.address_v4 { + if let Some(address) = request.address_v4 { + let prefix_len = request.prefix_len_v4; // IPv4 is not sensitive to carrier the way IPv6 is, so it needs // no extra settings. commands.push(format!( @@ -589,7 +592,8 @@ fd559caf9652cb86321feac65c73bd84 05 40 00 08 tsun0 name: "tsun0".into(), address: "fd00::1".parse().unwrap(), prefix_len: 64, - address_v4: Some(("100.64.1.2".parse().unwrap(), 10)), + address_v4: Some("100.64.1.2".parse().unwrap()), + prefix_len_v4: 10, mtu: 1280, }; let commands = setup_commands(&request, "someone"); diff --git a/tests/wireguard.rs b/tests/wireguard.rs index fc37099..eb0bc6f 100644 --- a/tests/wireguard.rs +++ b/tests/wireguard.rs @@ -21,8 +21,8 @@ use tempfile::TempDir; use tsunagi::agent::Event; use tsunagi::dataplane::IpPlugin; use tsunagi::dataplane::wireguard::{ - MemoryTun, MemoryTunFactory, WIREGUARD_PROTOCOL, WgAnnouncement, WgSecretKey, WireguardConfig, - WireguardPlugin, overlay_address, overlay_prefix, + Ipv4Range, MemoryTun, MemoryTunFactory, WIREGUARD_PROTOCOL, WgAnnouncement, WgSecretKey, + WireguardConfig, WireguardPlugin, overlay_address, overlay_prefix, }; use tsunagi::discovery::SharedMemoryDiscovery; use tsunagi::identity::{NetworkId, NetworkName, NetworkSecret}; @@ -267,8 +267,9 @@ async fn the_overlay_carries_ipv4_alongside_ipv6() { let discovery = SharedMemoryDiscovery::new(); let (name, secret) = network("wg-dual-stack"); - let a = WgAgent::spawn(&discovery, "ta").await; - let b = WgAgent::spawn(&discovery, "tb").await; + let range = Some("10.77.0.0/16".parse::().unwrap()); + let a = WgAgent::spawn_with(&discovery, "ta", |c| c.with_ipv4_range(range)).await; + let b = WgAgent::spawn_with(&discovery, "tb", |c| c.with_ipv4_range(range)).await; let network_id = a.agent.join_network(&name, &secret).await.unwrap(); b.agent.join_network(&name, &secret).await.unwrap(); @@ -278,14 +279,14 @@ async fn the_overlay_carries_ipv4_alongside_ipv6() { let view_a = a.plugin.overview(network_id).unwrap(); let view_b = b.plugin.overview(network_id).unwrap(); - let v4_a = view_a.overlay_address_v4.expect("dual stack by default"); - let v4_b = view_b.overlay_address_v4.expect("dual stack by default"); + let v4_a = view_a.overlay_address_v4.expect("ipv4 was configured"); + let v4_b = view_b.overlay_address_v4.expect("ipv4 was configured"); assert_ne!(v4_a, v4_b); // Both inside the configured range. for addr in [v4_a, v4_b] { assert_eq!( - u32::from(addr) & 0xffc0_0000, - u32::from(Ipv4Addr::new(100, 64, 0, 0)) + u32::from(addr) & 0xffff_0000, + u32::from(Ipv4Addr::new(10, 77, 0, 0)) ); } // Each side derived the other's address identically. @@ -327,8 +328,9 @@ async fn an_ipv4_source_a_peer_does_not_own_is_dropped() { let discovery = SharedMemoryDiscovery::new(); let (name, secret) = network("wg-v4-spoof"); - let a = WgAgent::spawn(&discovery, "ta").await; - let b = WgAgent::spawn(&discovery, "tb").await; + let range = Some("10.78.0.0/16".parse::().unwrap()); + let a = WgAgent::spawn_with(&discovery, "ta", |c| c.with_ipv4_range(range)).await; + let b = WgAgent::spawn_with(&discovery, "tb", |c| c.with_ipv4_range(range)).await; let network_id = a.agent.join_network(&name, &secret).await.unwrap(); b.agent.join_network(&name, &secret).await.unwrap(); a.wait_for_tunnels(network_id, 1).await; @@ -373,12 +375,13 @@ async fn an_ipv4_source_a_peer_does_not_own_is_dropped() { } #[tokio::test] -async fn an_ipv6_only_overlay_can_be_asked_for() { +async fn an_ipv6_only_overlay_is_the_default() { let discovery = SharedMemoryDiscovery::new(); let (name, secret) = network("wg-v6-only"); - let a = WgAgent::spawn_with(&discovery, "ta", |config| config.with_ipv4_range(None)).await; - let b = WgAgent::spawn_with(&discovery, "tb", |config| config.with_ipv4_range(None)).await; + // No IPv4 range is configured, which is the default. + let a = WgAgent::spawn(&discovery, "ta").await; + let b = WgAgent::spawn(&discovery, "tb").await; let network_id = a.agent.join_network(&name, &secret).await.unwrap(); b.agent.join_network(&name, &secret).await.unwrap(); @@ -407,6 +410,58 @@ async fn an_ipv6_only_overlay_can_be_asked_for() { b.shutdown().await; } +#[tokio::test] +async fn members_configured_with_different_ipv4_ranges_are_told_so() { + let discovery = SharedMemoryDiscovery::new(); + let (name, secret) = network("wg-range-mismatch"); + + // Two members configured differently. Deriving addresses from the range + // means they would disagree about each other, so IPv4 must be withheld + // and the mismatch reported rather than silently misrouting. + let a = WgAgent::spawn_with(&discovery, "ta", |c| { + c.with_ipv4_range(Some("10.80.0.0/16".parse().unwrap())) + }) + .await; + let b = WgAgent::spawn_with(&discovery, "tb", |c| { + c.with_ipv4_range(Some("10.81.0.0/16".parse().unwrap())) + }) + .await; + + let mut events = a.agent.subscribe(); + let network_id = a.agent.join_network(&name, &secret).await.unwrap(); + b.agent.join_network(&name, &secret).await.unwrap(); + a.wait_for_tunnels(network_id, 1).await; + + let reason = wait_event(&mut events, |event| match event { + Event::PluginError { reason, .. } if reason.contains("IPv4 overlay range") => { + Some(reason.clone()) + } + _ => None, + }) + .await; + assert!(reason.contains("10.81.0.0/16"), "unexpected: {reason}"); + assert!(reason.contains("10.80.0.0/16"), "unexpected: {reason}"); + + // The peer has no IPv4 here, but IPv6 is unaffected. + let view = a.plugin.overview(network_id).unwrap(); + assert_eq!(view.peers[0].overlay_address_v4, None); + assert!(view.peers[0].is_up(), "the tunnel itself still works"); + + let addr_a = a.overlay(network_id).await; + let addr_b = b.overlay(network_id).await; + a.tun(network_id) + .await + .push_from_os(ipv6_packet(addr_a, addr_b, b"v6 still fine")); + let received = tokio::time::timeout(common::DEADLINE, b.tun(network_id).await.pop_to_os()) + .await + .expect("IPv6 should be unaffected") + .unwrap(); + assert_eq!(&received[40..], b"v6 still fine"); + + a.shutdown().await; + b.shutdown().await; +} + #[tokio::test] async fn packets_for_an_unknown_address_are_counted_not_broadcast() { let discovery = SharedMemoryDiscovery::new(); @@ -672,7 +727,7 @@ async fn the_core_carries_the_payload_without_interpreting_it() { assert_eq!(capability.protocol, WIREGUARD_PROTOCOL); let view_b = b.plugin.overview(network_id).unwrap(); - let expected = WgAnnouncement::new(network_id, &view_b.public_key) + let expected = WgAnnouncement::new(network_id, &view_b.public_key, None) .encode() .unwrap(); assert_eq!(capability.data, expected); @@ -695,7 +750,7 @@ async fn a_forged_overlay_claim_is_rejected_and_never_reaches_a_tunnel() { // A legitimate member — it knows the secret — claims the victim's overlay // address with its own WireGuard key. let attacker_key = WgSecretKey::generate().public(); - let mut forged = WgAnnouncement::new(network_id, &attacker_key); + let mut forged = WgAnnouncement::new(network_id, &attacker_key, None); forged.overlay_address = victim_address; let forger = Arc::new(ForgingPlugin {