Make the overlay dual stack

Every member now also derives an IPv4 address, from the same inputs as its
IPv6 one, into 100.64.0.0/10 by default. The range is configurable and IPv4
can be turned off with --no-ipv4.

IPv4 is honestly weaker than IPv6 here and the code says so. A 64 bit
interface identifier makes an IPv6 collision impossible in practice; IPv4
has nothing like that room, and in a /10 with 50 members two will derive the
same address about 0.03% of the time. A mesh with no coordinator cannot
allocate around that, so a collision is detected and resolved instead: the
member whose public key sorts lower keeps the address, a rule every member
computes identically and therefore agrees on. The other keeps IPv6 and is
flagged in the status. IPv6 always works; IPv4 almost always works and
degrades predictably.

Routing and address-ownership enforcement now cover both families: a packet
goes to the peer that owns its destination, and a decrypted packet is
dropped unless its source is an address derived for the peer that sent it,
IPv4 included.

Six new tests, among them a real IPv4 packet crossing a tunnel next to an
IPv6 one, a spoofed IPv4 source being dropped, and an IPv6-only overlay.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
tsunagi
2026-09-21 13:00:35 +01:00
co-authored by Claude Opus 5
parent be459e5bd0
commit cfab38824d
12 changed files with 638 additions and 46 deletions
+40 -6
View File
@@ -31,10 +31,10 @@ A working library with **real iroh connections** and integration tests:
- status snapshots, an event stream and honest diagnostics;
- configuration restored after a restart;
- correct behaviour when the disposable cache is missing or corrupt;
- a **WireGuard data plane**, in userspace: its own key per network,
deterministic IPv6 overlay addressing, real tunnels carried over iroh, and
address ownership enforced rather than believed;
- a **command line agent**, `tsunagi`.
- a **WireGuard data plane**, in userspace: its own key per network, a dual
stack overlay with deterministically derived addresses, real tunnels carried
over iroh, and address ownership enforced rather than believed;
- a **command line agent**, `tsunagi`, with a local control socket.
### What it deliberately does **not** do
@@ -94,8 +94,41 @@ wireguard: tsunkkcp43lmdje on fd15:1d9e:fa21:f201:…/64 mtu 1280, 1/1 tunnel(s)
4jO4kx9Z fd15:1d9e:fa21:f201:… handshake 3s ago tx=0 rx=0 dropped=0 path=Direct via Ip(…)
```
`1/1 tunnel(s) established` means a real WireGuard handshake completed. Then
`ping6` the peer's overlay address.
`1/1 tunnel(s) established` means a real WireGuard handshake completed.
## Checking that it works
From another shell on either machine:
```bash
tsunagi status
```
```text
endpoint 7d76ccbbc21bf30767e14422c0494740a2cecc02aa9f82d5b8d57bdae350e7fc
hostname tsunagi-7d76ccbbc2
bound 0.0.0.0:41641
network lab (z2o4qwrvnj3zb6st2aoqg4abf342j662q2ujttqrsmz22argk2ba) active
peer b345d5271b tsunagi-b345d5271b Direct rtt 24ms
overlay tsunz2o4qwrvnj3 fd09:…:c1c6/64 and 100.110.49.177 mtu 1280 1/1 tunnel(s) up
SDsEb/WF fd09:…:c4b / 100.65.243.53 handshake 4s ago tx 0 rx 0 Direct via Ip(…)
```
`1/1 tunnel(s) up` and a recent handshake mean the tunnel is live. Then send
real traffic to the peer's overlay address:
```bash
ping6 fd09:…:c4b # or
ping 100.65.243.53
```
`tx` and `rx` in the status should start moving.
The overlay is dual stack: every member derives both an IPv6 address, which
can never collide, and an IPv4 one in `100.64.0.0/10`, which very rarely can —
see [docs/wireguard.md](docs/wireguard.md#ipv4-alongside-ipv6). `--no-ipv4`
runs IPv6 only, `--ipv4-range` moves the range.
Notes:
@@ -130,6 +163,7 @@ sudo ip tuntap add dev tsunjwc6dcrtmo5 mode tun user ab
sudo ip link set dev tsunjwc6dcrtmo5 mtu 1280 up
sudo sysctl -qw net.ipv6.conf.tsunjwc6dcrtmo5.keep_addr_on_down=1
sudo ip -6 address add fd80:1210:f724:f620:d1bb:f982:3b6e:19bd/64 dev tsunjwc6dcrtmo5 nodad
sudo ip address add 100.110.49.177/10 dev tsunjwc6dcrtmo5
```
The MTU is 1280 because that is the minimum IPv6 requires (RFC 8200). Linux
+5
View File
@@ -50,6 +50,11 @@ keeping the WireGuard identity, shutdown removing every interface, a forged
overlay claim being rejected, and the core carrying the payload without
interpreting it.
`tests/local_control.rs` covers the local control socket end to end: a client
asking a running agent for status over a real Unix socket, a leftover socket
file being replaced while a live one is not, and the derived socket path
staying short enough to bind.
`tests/discovery.rs` covers the discovery contract itself: a static bootstrap
candidate is enough to join, several backends compose, entries are withdrawn
when a network stops, and a forgotten network stays forgotten across a restart.
+33 -3
View File
@@ -56,6 +56,12 @@ direct path when it can and falls back to a relay when it cannot; the tunnel
rides on whichever it got. There is no separate STUN, no separate hole punching
and no second set of NAT problems to solve for WireGuard.
## Checking it from outside
`tsunagi status` asks a running agent over its local control socket and prints
what it sees, including whether each tunnel has actually handshaken. See
[../README.md](../README.md#checking-that-it-works).
## Deterministic overlay addressing
A mesh with no coordinator cannot hand out addresses, so everyone derives their
@@ -77,6 +83,29 @@ Two consequences matter:
* a member's address is bound to its WireGuard public key, so address
ownership can be checked locally rather than believed.
## IPv4 alongside IPv6
The overlay is dual stack by default: every member also derives an IPv4
address, from the same inputs, into `100.64.0.0/10` (RFC 6598 shared address
space — deliberately not RFC 1918, so it rarely clashes with the network the
machine is already on). The range is configurable, and IPv4 can be turned off.
**IPv4 is weaker than IPv6 here, and the difference is not cosmetic.** A 64 bit
interface identifier makes an IPv6 collision impossible in practice. IPv4 has
nothing like that much room: in a `/10` with 50 members the chance that two
derive the same address is roughly 0.03%. Small, but not zero, and a mesh with
no coordinator cannot simply allocate around it.
So a collision is detected and resolved rather than assumed away: the member
whose WireGuard public key sorts lower keeps the address, a rule every member
computes identically and therefore agrees on without exchanging anything. The
other member ends up with **no IPv4 address** and is still fully reachable over
IPv6. The status output flags it.
That is the honest summary: **IPv6 always works; IPv4 almost always works and
degrades predictably when it does not.** Allocating IPv4 properly needs the
agreed state described in [sync-model.md](sync-model.md).
## Address ownership is enforced, not announced
Kernel WireGuard enforces `AllowedIPs`. In userspace that is our job, and
@@ -87,6 +116,8 @@ Kernel WireGuard enforces `AllowedIPs`. In userspace that is our job, and
* **inbound**, a decrypted packet is dropped unless its *source* is exactly the
address derived for the peer whose tunnel decrypted it.
Both apply to IPv4 and IPv6 alike.
So a participant cannot receive traffic addressed to somebody else and cannot
forge traffic that appears to come from somebody else. A participant who knows
the network secret can mint many keys and therefore occupy many addresses, but
@@ -192,9 +223,8 @@ async fn main() -> Result<()> {
* **Full mesh only.** Every member runs a tunnel to every other member.
Routing through an intermediate participant is not implemented.
* **IPv6 overlay only.** Addressing is IPv6 ULA because it can be derived
collision-free. An IPv4 overlay would need an allocator, which needs the
agreed state described in [sync-model.md](sync-model.md).
* **IPv4 addressing can collide.** See above: it is resolved deterministically
and the loser keeps IPv6, but a proper allocator needs agreed state.
* **No routes, DNS or firewall rules.** The plugin creates its interface and
nothing else. Anything beyond the overlay `/64` is the operator's business.
* **Membership is session-scoped.** A peer leaves the overlay when its control
+63 -1
View File
@@ -98,6 +98,45 @@ struct TunSetupArgs {
/// Interface MTU, matching `tsunagi up --wg-mtu`. At least 1280.
#[arg(long)]
wg_mtu: Option<u32>,
/// Match `tsunagi up --no-ipv4`.
#[arg(long)]
no_ipv4: bool,
/// Match `tsunagi up --ipv4-range`.
#[arg(long, value_name = "CIDR", conflicts_with = "no_ipv4")]
ipv4_range: Option<String>,
}
/// Parses `address/prefix` into an IPv4 range.
fn parse_ipv4_range(text: &str) -> Result<(std::net::Ipv4Addr, u8), String> {
let (address, prefix) = text
.split_once('/')
.ok_or_else(|| format!("`{text}` is not an address with a prefix, e.g. 100.64.0.0/10"))?;
let address = address
.parse()
.map_err(|err| format!("`{address}` is not an IPv4 address: {err}"))?;
let prefix: u8 = prefix
.parse()
.map_err(|err| format!("`{prefix}` is not a prefix length: {err}"))?;
if prefix > 30 {
return Err(format!("a /{prefix} has no room for hosts"));
}
Ok((address, prefix))
}
/// Resolves the IPv4 overlay range from the flags.
fn resolve_ipv4_range(
no_ipv4: bool,
range: Option<&String>,
) -> Result<Option<(std::net::Ipv4Addr, u8)>, Box<dyn std::error::Error>> {
if no_ipv4 {
return Ok(None);
}
match range {
Some(text) => Ok(Some(parse_ipv4_range(text)?)),
None => Ok(Some(tsunagi::dataplane::wireguard::DEFAULT_IPV4_RANGE)),
}
}
#[derive(Debug, Args, Clone)]
@@ -212,6 +251,14 @@ struct UpArgs {
#[arg(long)]
wg_mtu: Option<u32>,
/// Run an IPv6-only overlay instead of dual stack.
#[arg(long)]
no_ipv4: bool,
/// IPv4 overlay range, as `address/prefix`. Defaults to 100.64.0.0/10.
#[arg(long, value_name = "CIDR", conflicts_with = "no_ipv4")]
ipv4_range: Option<String>,
/// How often to print a status summary, in seconds. Zero disables it.
#[arg(long, default_value_t = 15)]
status_interval: u64,
@@ -346,6 +393,7 @@ async fn status(args: StatusArgs) -> Result<(), Box<dyn std::error::Error>> {
async fn tun_setup(args: TunSetupArgs) -> Result<(), Box<dyn std::error::Error>> {
use tsunagi::dataplane::wireguard::{
DEFAULT_MTU, OVERLAY_PREFIX_LEN, WgKeyStore, interface_name, overlay_address,
overlay_address_v4,
};
use tsunagi::identity::NetworkKeys;
@@ -364,6 +412,7 @@ async fn tun_setup(args: TunSetupArgs) -> Result<(), Box<dyn std::error::Error>>
let interface = interface_name(&args.wg_prefix, network)?;
let address = overlay_address(network, &key.public());
let ipv4_range = resolve_ipv4_range(args.no_ipv4, args.ipv4_range.as_ref())?;
let mtu = args.wg_mtu.unwrap_or(DEFAULT_MTU);
let user = args.user.unwrap_or_else(|| {
std::env::var("SUDO_USER")
@@ -373,6 +422,11 @@ async fn tun_setup(args: TunSetupArgs) -> Result<(), Box<dyn std::error::Error>>
println!("# Network {name} ({network})");
println!("# Interface {interface}, address {address}/{OVERLAY_PREFIX_LEN}, mtu {mtu}");
if let Some((base, prefix)) = ipv4_range
&& let Some(v4) = overlay_address_v4(network, &key.public(), (base, prefix))
{
println!("# IPv4 overlay address {v4}/{prefix}");
}
println!("# Run once as root; then run `tsunagi up` as {user}.");
println!(
"#\n\
@@ -386,6 +440,11 @@ async fn tun_setup(args: TunSetupArgs) -> Result<(), Box<dyn std::error::Error>>
println!("sudo ip link set dev {interface} mtu {mtu} up");
println!("sudo sysctl -qw net.ipv6.conf.{interface}.keep_addr_on_down=1");
println!("sudo ip -6 address add {address}/{OVERLAY_PREFIX_LEN} dev {interface} nodad");
if let Some((base, prefix)) = ipv4_range
&& let Some(v4) = overlay_address_v4(network, &key.public(), (base, prefix))
{
println!("sudo ip address add {v4}/{prefix} dev {interface}");
}
println!("\n# To check it afterwards:");
println!("ip -6 addr show dev {interface}");
println!("\n# To remove it again:");
@@ -516,7 +575,8 @@ async fn up(args: UpArgs) -> Result<(), Box<dyn std::error::Error>> {
system_tun_factory()?
};
let mut wg = WireguardConfig::new(paths.state_dir.join("wireguard"))
.with_interface_prefix(args.wg_prefix.clone());
.with_interface_prefix(args.wg_prefix.clone())
.with_ipv4_range(resolve_ipv4_range(args.no_ipv4, args.ipv4_range.as_ref())?);
if let Some(mtu) = args.wg_mtu {
wg = wg.with_mtu(mtu);
}
@@ -637,6 +697,7 @@ async fn build_report(
interface: view.interface.clone(),
mtu: view.mtu,
address: view.overlay_address.to_string(),
address_v4: view.overlay_address_v4.map(|addr| addr.to_string()),
prefix: view.overlay_prefix.to_string(),
prefix_len: view.overlay_prefix_len,
peers: view
@@ -645,6 +706,7 @@ async fn build_report(
.map(|peer| OverlayPeerReport {
public_key: peer.public_key.to_string(),
address: peer.overlay_address.to_string(),
address_v4: peer.overlay_address_v4.map(|addr| addr.to_string()),
handshake_secs_ago: peer
.tunnel
.as_ref()
+119 -23
View File
@@ -27,7 +27,7 @@
//! what it announced.
use std::collections::HashMap;
use std::net::Ipv6Addr;
use std::net::{IpAddr, Ipv4Addr, Ipv6Addr};
use std::sync::atomic::{AtomicU32, AtomicU64, Ordering};
use std::sync::{Arc, Mutex, RwLock};
use std::time::Duration;
@@ -42,7 +42,7 @@ use crate::dataplane::transport::{SharedLink, TransportError};
use crate::identity::NetworkId;
use super::keys::{WgPublicKey, WgSecretKey};
use super::overlay::overlay_address;
use super::overlay::{overlay_address, overlay_address_v4};
use super::packet::IpHeader;
use super::tun::TunDevice;
@@ -109,6 +109,9 @@ struct Peer {
endpoint_id: EndpointId,
public_key: WgPublicKey,
overlay: Ipv6Addr,
/// The IPv4 address this peer owns, when the overlay is dual stack and
/// nobody else derived the same one.
overlay_v4: Mutex<Option<Ipv4Addr>>,
tunn: Mutex<Tunn>,
link: SharedLink,
counters: Arc<PeerCounters>,
@@ -168,6 +171,13 @@ pub struct PeerSummary {
pub public_key: WgPublicKey,
/// The overlay address this agent derived for it.
pub overlay_address: Ipv6Addr,
/// Its IPv4 overlay address, when the overlay is dual stack.
///
/// `None` with `ipv4_conflict` set means another member derived the same
/// address and won it; that peer is still fully reachable over IPv6.
pub overlay_address_v4: Option<Ipv4Addr>,
/// Whether this peer lost an IPv4 address to a derivation collision.
pub ipv4_conflict: bool,
/// Whether the tunnel has handshaken.
pub health: PeerHealth,
/// Traffic counters.
@@ -182,11 +192,15 @@ struct Inner {
network: NetworkId,
private_key: WgSecretKey,
tun: Arc<dyn TunDevice>,
/// The IPv4 overlay range, when the overlay is dual stack.
ipv4_range: Option<(Ipv4Addr, u8)>,
peers: RwLock<HashMap<WgPublicKey, Arc<Peer>>>,
routes: RwLock<HashMap<Ipv6Addr, WgPublicKey>>,
/// Both families, so one lookup routes any packet.
routes: RwLock<HashMap<IpAddr, WgPublicKey>>,
next_index: AtomicU32,
unroutable: AtomicU64,
multicast: AtomicU64,
ipv4_conflicts: AtomicU64,
}
impl std::fmt::Debug for Inner {
@@ -207,16 +221,23 @@ pub struct WireguardDevice {
impl WireguardDevice {
/// Starts a device on top of `tun`.
pub fn start(network: NetworkId, private_key: WgSecretKey, tun: Arc<dyn TunDevice>) -> Self {
pub fn start(
network: NetworkId,
private_key: WgSecretKey,
tun: Arc<dyn TunDevice>,
ipv4_range: Option<(Ipv4Addr, u8)>,
) -> Self {
let inner = Arc::new(Inner {
network,
private_key,
tun,
ipv4_range,
peers: RwLock::new(HashMap::new()),
routes: RwLock::new(HashMap::new()),
next_index: AtomicU32::new(1),
unroutable: AtomicU64::new(0),
multicast: AtomicU64::new(0),
ipv4_conflicts: AtomicU64::new(0),
});
let reader = tokio::spawn(read_from_os(Arc::clone(&inner)));
@@ -263,10 +284,12 @@ impl WireguardDevice {
);
let overlay = overlay_address(self.inner.network, &public_key);
let overlay_v4 = self.claim_ipv4(&public_key);
let peer = Arc::new(Peer {
endpoint_id,
public_key,
overlay,
overlay_v4: Mutex::new(overlay_v4),
tunn: Mutex::new(tunn),
link,
counters: Arc::new(PeerCounters::default()),
@@ -279,7 +302,10 @@ impl WireguardDevice {
}
write_lock(&self.inner.peers).insert(public_key, Arc::clone(&peer));
write_lock(&self.inner.routes).insert(overlay, public_key);
write_lock(&self.inner.routes).insert(IpAddr::V6(overlay), public_key);
if let Some(v4) = overlay_v4 {
write_lock(&self.inner.routes).insert(IpAddr::V4(v4), public_key);
}
// Start the handshake now instead of waiting for the next timer tick,
// so the tunnel is usable as soon as the link exists.
@@ -287,13 +313,64 @@ impl WireguardDevice {
Ok(())
}
/// Decides which IPv4 address a new peer gets, if any.
///
/// IPv4 has far too little room for a derived address to be collision
/// free. When two members derive the same one, the member whose public
/// key sorts lower keeps it — a rule every member computes identically,
/// so they all agree on the outcome without talking about it. The other
/// member simply has no IPv4 address; it is still fully reachable over
/// IPv6, which never collides.
fn claim_ipv4(&self, public_key: &WgPublicKey) -> Option<Ipv4Addr> {
let range = self.inner.ipv4_range?;
let wanted = overlay_address_v4(self.inner.network, public_key, range)?;
let holder = read_lock(&self.inner.routes)
.get(&IpAddr::V4(wanted))
.copied();
let Some(holder) = holder else {
return Some(wanted);
};
if holder == *public_key {
return Some(wanted);
}
self.inner.ipv4_conflicts.fetch_add(1, Ordering::Relaxed);
if holder.as_bytes() <= public_key.as_bytes() {
// The peer already holding it wins.
return None;
}
// The newcomer wins; take the address away from the other peer.
if let Some(loser) = read_lock(&self.inner.peers).get(&holder).cloned() {
let mut slot = match loser.overlay_v4.lock() {
Ok(guard) => guard,
Err(poisoned) => poisoned.into_inner(),
};
*slot = None;
}
Some(wanted)
}
/// Removes a peer and stops its tunnel.
pub fn remove_peer(&self, public_key: &WgPublicKey) {
if let Some(peer) = write_lock(&self.inner.peers).remove(public_key) {
write_lock(&self.inner.routes).remove(&peer.overlay);
let mut routes = write_lock(&self.inner.routes);
routes.remove(&IpAddr::V6(peer.overlay));
let v4 = match peer.overlay_v4.lock() {
Ok(guard) => *guard,
Err(poisoned) => *poisoned.into_inner(),
};
if let Some(v4) = v4 {
routes.remove(&IpAddr::V4(v4));
}
}
}
/// How many IPv4 derivation collisions have been resolved.
pub fn ipv4_conflicts(&self) -> u64 {
self.inner.ipv4_conflicts.load(Ordering::Relaxed)
}
/// Removes every peer whose key is not in `keep`.
pub fn retain_peers(&self, keep: &[WgPublicKey]) {
let stale: Vec<WgPublicKey> = read_lock(&self.inner.peers)
@@ -315,14 +392,22 @@ impl WireguardDevice {
pub fn peers(&self) -> Vec<PeerSummary> {
let mut peers: Vec<PeerSummary> = read_lock(&self.inner.peers)
.values()
.map(|peer| PeerSummary {
endpoint_id: peer.endpoint_id,
public_key: peer.public_key,
overlay_address: peer.overlay,
health: peer.health(),
stats: peer.stats(),
path: peer.link.path_description(),
max_datagram: peer.link.max_datagram_size(),
.map(|peer| {
let overlay_v4 = match peer.overlay_v4.lock() {
Ok(guard) => *guard,
Err(poisoned) => *poisoned.into_inner(),
};
PeerSummary {
endpoint_id: peer.endpoint_id,
public_key: peer.public_key,
overlay_address: peer.overlay,
overlay_address_v4: overlay_v4,
ipv4_conflict: overlay_v4.is_none() && self.inner.ipv4_range.is_some(),
health: peer.health(),
stats: peer.stats(),
path: peer.link.path_description(),
max_datagram: peer.link.max_datagram_size(),
}
})
.collect();
peers.sort_by_key(|peer| peer.public_key);
@@ -415,9 +500,8 @@ async fn read_from_os(inner: Arc<Inner>) {
};
// Route by destination: only the peer that owns that overlay address
// may receive it.
let Some(destination) = IpHeader::parse(&packet).and_then(|header| header.v6_destination())
else {
// may receive it. Both families go through the same table.
let Some(destination) = IpHeader::parse(&packet).map(|header| header.destination()) else {
inner.unroutable.fetch_add(1, Ordering::Relaxed);
continue;
};
@@ -488,9 +572,11 @@ async fn read_from_link(inner: Arc<Inner>, peer: Arc<Peer>) {
match tunn.decapsulate(None, input.unwrap_or(&[]), &mut scratch) {
TunnResult::WriteToNetwork(out) => Outcome::ToNetwork(out.len()),
TunnResult::WriteToTunnelV6(out, source) => {
Outcome::ToTunnel(out.len(), Some(source))
Outcome::ToTunnel(out.len(), IpAddr::V6(source))
}
TunnResult::WriteToTunnelV4(out, source) => {
Outcome::ToTunnel(out.len(), IpAddr::V4(source))
}
TunnResult::WriteToTunnelV4(out, _) => Outcome::ToTunnel(out.len(), None),
TunnResult::Done => Outcome::Done,
TunnResult::Err(err) => {
tracing::trace!(?err, "wireguard decapsulation failed");
@@ -508,9 +594,19 @@ async fn read_from_link(inner: Arc<Inner>, peer: Arc<Peer>) {
}
Outcome::ToTunnel(len, source) => {
let payload = Bytes::copy_from_slice(&scratch[..len]);
// Enforce address ownership: a peer may only send from the
// address derived for its own key.
if source != Some(peer.overlay) {
// Enforce address ownership: a peer may only send from an
// address derived for its own key, in either family.
let owned = match source {
IpAddr::V6(addr) => addr == peer.overlay,
IpAddr::V4(addr) => {
let held = match peer.overlay_v4.lock() {
Ok(guard) => *guard,
Err(poisoned) => *poisoned.into_inner(),
};
held == Some(addr)
}
};
if !owned {
peer.counters
.dropped_wrong_source
.fetch_add(1, Ordering::Relaxed);
@@ -538,7 +634,7 @@ async fn read_from_link(inner: Arc<Inner>, peer: Arc<Peer>) {
enum Outcome {
ToNetwork(usize),
ToTunnel(usize, Option<Ipv6Addr>),
ToTunnel(usize, IpAddr),
Done,
Failed,
}
+3 -1
View File
@@ -54,7 +54,9 @@ pub use announcement::{ValidatedAnnouncement, WgAnnouncement};
pub use config::{Cidr, DEFAULT_INTERFACE_PREFIX, MAX_INTERFACE_NAME_LEN, interface_name};
pub use device::{PeerHealth, PeerStats, PeerSummary, WireguardDevice};
pub use keys::{WgPublicKey, WgSecretKey};
pub use overlay::{OVERLAY_PREFIX_LEN, overlay_address, overlay_prefix};
pub use overlay::{
DEFAULT_IPV4_RANGE, OVERLAY_PREFIX_LEN, overlay_address, overlay_address_v4, overlay_prefix,
};
pub use packet::IpHeader;
pub use plugin::{
DEFAULT_MTU, MIN_MTU, NetworkOverview, PeerOverview, WIREGUARD_OVERHEAD, WIREGUARD_PROTOCOL,
+110 -1
View File
@@ -20,7 +20,7 @@
//! but it cannot choose to collide with an existing member's address without
//! finding a hash preimage.
use std::net::Ipv6Addr;
use std::net::{Ipv4Addr, Ipv6Addr};
use sha2::{Digest, Sha256};
@@ -37,6 +37,13 @@ pub const OVERLAY_PREFIX_LEN: u8 = 64;
/// Prefix length of one member's address inside the overlay.
pub const OVERLAY_HOST_PREFIX_LEN: u8 = 128;
/// Default IPv4 overlay range: RFC 6598 shared address space.
///
/// Deliberately not RFC 1918, so it rarely collides with the home or office
/// network the machine is already on. It can collide with a carrier-grade NAT
/// that uses the same range, which is why it is configurable.
pub const DEFAULT_IPV4_RANGE: (Ipv4Addr, u8) = (Ipv4Addr::new(100, 64, 0, 0), 10);
fn push_lp(out: &mut Vec<u8>, bytes: &[u8]) {
let len = u32::try_from(bytes.len()).unwrap_or(u32::MAX);
out.extend_from_slice(&len.to_be_bytes());
@@ -85,6 +92,49 @@ pub fn overlay_address(network: NetworkId, key: &WgPublicKey) -> Ipv6Addr {
Ipv6Addr::from(octets)
}
/// The IPv4 address a member with `key` has in `network`.
///
/// # Why this is weaker than the IPv6 derivation
///
/// A 64 bit interface identifier makes an IPv6 collision impossible in
/// practice. IPv4 has nothing like that much room, so two members *can* derive
/// the same address. In a `/10` with 50 members the chance is roughly 0.03%,
/// which is small but real, so it is detected and resolved rather than
/// assumed away — see [`super::device`]. IPv6 remains the address that always
/// works.
///
/// Returns `None` when the range has no room for hosts.
pub fn overlay_address_v4(
network: NetworkId,
key: &WgPublicKey,
range: (Ipv4Addr, u8),
) -> Option<Ipv4Addr> {
let (base, prefix_len) = range;
if prefix_len > 32 {
return None;
}
let host_bits = 32 - u32::from(prefix_len);
// A usable range needs a network address, a broadcast address and at
// least one host between them.
if host_bits < 2 {
return None;
}
let hash = digest("ipv4", network, Some(key));
let raw = u32::from_be_bytes([hash[0], hash[1], hash[2], hash[3]]);
let usable = (1u64 << host_bits) - 2;
let offset = (u64::from(raw) % usable) + 1;
let mask = if host_bits == 32 {
0
} else {
u32::MAX << host_bits
};
let network_part = u32::from(base) & mask;
Some(Ipv4Addr::from(network_part | offset as u32))
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
@@ -132,6 +182,65 @@ mod tests {
assert_ne!(overlay_prefix(first), overlay_prefix(second));
}
#[test]
fn ipv4_addresses_land_inside_the_range_and_avoid_its_edges() {
let id = network("v4");
let range = DEFAULT_IPV4_RANGE;
for byte in 0..64u8 {
let key = WgPublicKey::from_bytes([byte; 32]);
let addr = overlay_address_v4(id, &key, range).unwrap();
let raw = u32::from(addr);
assert_eq!(
raw & 0xffc0_0000,
u32::from(range.0),
"outside 100.64.0.0/10"
);
// Never the network address and never the broadcast address.
assert_ne!(raw & 0x003f_ffff, 0);
assert_ne!(raw & 0x003f_ffff, 0x003f_ffff);
}
}
#[test]
fn ipv4_derivation_is_deterministic_and_scoped_like_ipv6() {
let key = WgPublicKey::from_bytes([9u8; 32]);
let first = network("one");
let second = network("two");
let range = DEFAULT_IPV4_RANGE;
assert_eq!(
overlay_address_v4(first, &key, range),
overlay_address_v4(first, &key, range)
);
assert_ne!(
overlay_address_v4(first, &key, range),
overlay_address_v4(second, &key, range)
);
assert_ne!(
overlay_address_v4(first, &key, range),
overlay_address_v4(first, &WgPublicKey::from_bytes([10u8; 32]), range)
);
// A different range moves everybody.
assert_ne!(
overlay_address_v4(first, &key, range),
overlay_address_v4(first, &key, (Ipv4Addr::new(10, 0, 0, 0), 8))
);
}
#[test]
fn a_range_with_no_room_yields_nothing() {
let id = network("tiny");
let key = WgPublicKey::from_bytes([1u8; 32]);
// /31 and /32 have no usable host addresses.
assert!(overlay_address_v4(id, &key, (Ipv4Addr::new(10, 0, 0, 0), 31)).is_none());
assert!(overlay_address_v4(id, &key, (Ipv4Addr::new(10, 0, 0, 1), 32)).is_none());
assert!(overlay_address_v4(id, &key, (Ipv4Addr::new(10, 0, 0, 0), 33)).is_none());
// A /30 has two usable addresses.
assert!(overlay_address_v4(id, &key, (Ipv4Addr::new(10, 0, 0, 0), 30)).is_some());
// A /0 must not overflow.
assert!(overlay_address_v4(id, &key, (Ipv4Addr::UNSPECIFIED, 0)).is_some());
}
#[test]
fn addresses_are_never_the_subnet_router_anycast_address() {
let id = network("anycast");
+16
View File
@@ -65,6 +65,22 @@ impl IpHeader {
IpHeader::V4 { .. } => None,
}
}
/// The destination, whichever family it is.
pub fn destination(&self) -> std::net::IpAddr {
match self {
IpHeader::V4 { destination, .. } => std::net::IpAddr::V4(*destination),
IpHeader::V6 { destination, .. } => std::net::IpAddr::V6(*destination),
}
}
/// The source, whichever family it is.
pub fn source(&self) -> std::net::IpAddr {
match self {
IpHeader::V4 { source, .. } => std::net::IpAddr::V4(*source),
IpHeader::V6 { source, .. } => std::net::IpAddr::V6(*source),
}
}
}
#[cfg(test)]
+41 -3
View File
@@ -25,7 +25,7 @@
//! A failure here is reported and retried. It never stops the control plane.
use std::collections::{BTreeSet, HashMap};
use std::net::IpAddr;
use std::net::{IpAddr, Ipv4Addr};
use std::path::PathBuf;
use std::sync::{Arc, Mutex, OnceLock};
use std::time::Duration;
@@ -43,7 +43,9 @@ use super::announcement::{ValidatedAnnouncement, WgAnnouncement};
use super::config::{DEFAULT_INTERFACE_PREFIX, interface_name};
use super::device::{PeerSummary, WireguardDevice};
use super::keys::{WgPublicKey, WgSecretKey};
use super::overlay::{OVERLAY_PREFIX_LEN, overlay_address, overlay_prefix};
use super::overlay::{
DEFAULT_IPV4_RANGE, OVERLAY_PREFIX_LEN, overlay_address, overlay_address_v4, overlay_prefix,
};
use super::store::WgKeyStore;
use super::tun::{TunFactory, TunRequest};
@@ -86,6 +88,12 @@ pub struct WireguardConfig {
pub keepalive: Option<u16>,
/// Interface MTU. See [`DEFAULT_MTU`].
pub mtu: u32,
/// IPv4 overlay range, or `None` for an IPv6-only overlay.
///
/// IPv6 addresses are derived collision-free; IPv4 ones cannot be, so a
/// collision is detected and resolved deterministically instead. See
/// `docs/wireguard.md`.
pub ipv4_range: Option<(Ipv4Addr, u8)>,
/// How long to coalesce changes before reconciling.
pub reconcile_debounce: Duration,
/// How often to reconcile anyway, which is also when a packet interface
@@ -101,6 +109,7 @@ impl WireguardConfig {
interface_prefix: DEFAULT_INTERFACE_PREFIX.to_string(),
keepalive: Some(25),
mtu: DEFAULT_MTU,
ipv4_range: Some(DEFAULT_IPV4_RANGE),
reconcile_debounce: Duration::from_millis(200),
reconcile_interval: Duration::from_secs(15),
}
@@ -120,6 +129,12 @@ impl WireguardConfig {
self
}
/// Sets the IPv4 overlay range, or disables IPv4 with `None`.
pub fn with_ipv4_range(mut self, range: Option<(Ipv4Addr, u8)>) -> Self {
self.ipv4_range = range;
self
}
/// Sets the reconciliation timings.
pub fn with_reconcile(mut self, debounce: Duration, interval: Duration) -> Self {
self.reconcile_debounce = debounce;
@@ -150,6 +165,10 @@ pub struct NetworkOverview {
pub overlay_prefix: IpAddr,
/// Prefix length of the overlay subnet.
pub overlay_prefix_len: u8,
/// This agent's IPv4 overlay address, when the overlay is dual stack.
pub overlay_address_v4: Option<Ipv4Addr>,
/// The IPv4 overlay range in use.
pub ipv4_range: Option<(Ipv4Addr, u8)>,
/// Peers this agent knows about.
pub peers: Vec<PeerOverview>,
/// Unicast packets the operating system sent to an address no peer owns.
@@ -174,6 +193,8 @@ pub struct PeerOverview {
pub public_key: WgPublicKey,
/// The overlay address derived for it locally.
pub overlay_address: IpAddr,
/// Its IPv4 overlay address, once a tunnel exists and it won the address.
pub overlay_address_v4: Option<Ipv4Addr>,
/// Whether a data plane link to it exists.
pub has_link: bool,
/// The running tunnel, once there is a link.
@@ -309,6 +330,9 @@ impl WireguardPlugin {
endpoint_id: *endpoint_id,
public_key: announcement.public_key,
overlay_address: IpAddr::V6(announcement.overlay_address),
overlay_address_v4: tunnels
.get(&announcement.public_key)
.and_then(|tunnel| tunnel.overlay_address_v4),
has_link: state.links.contains_key(endpoint_id),
tunnel: tunnels.get(&announcement.public_key).cloned(),
})
@@ -323,6 +347,12 @@ impl WireguardPlugin {
overlay_address: IpAddr::V6(overlay_address(network, &state.key.public())),
overlay_prefix: IpAddr::V6(overlay_prefix(network)),
overlay_prefix_len: OVERLAY_PREFIX_LEN,
overlay_address_v4: self
.worker
.config
.ipv4_range
.and_then(|range| overlay_address_v4(network, &state.key.public(), range)),
ipv4_range: self.worker.config.ipv4_range,
peers,
unroutable_packets: state
.device
@@ -437,10 +467,18 @@ impl Worker {
name: name.clone(),
address: overlay_address(network, &key.public()),
prefix_len: OVERLAY_PREFIX_LEN,
address_v4: self.config.ipv4_range.and_then(|range| {
overlay_address_v4(network, &key.public(), range).map(|address| (address, range.1))
}),
mtu: self.config.mtu,
};
let tun = self.tun_factory.create(request).await?;
let device = Arc::new(WireguardDevice::start(network, key, tun));
let device = Arc::new(WireguardDevice::start(
network,
key,
tun,
self.config.ipv4_range,
));
let mut shared = self.lock_shared();
if let Some(state) = shared.networks.get_mut(&network) {
+30 -5
View File
@@ -12,7 +12,7 @@
//! * `SystemTun`, behind the `tun-device` feature, is a real TUN interface.
//! Creating one needs `CAP_NET_ADMIN` on Linux or the equivalent elsewhere.
use std::net::Ipv6Addr;
use std::net::{Ipv4Addr, Ipv6Addr};
use std::sync::Arc;
use bytes::Bytes;
@@ -29,6 +29,8 @@ pub struct TunRequest {
pub address: Ipv6Addr,
/// Prefix length of the overlay subnet, so the OS routes it here.
pub prefix_len: u8,
/// The IPv4 overlay address and its prefix length, when dual stack.
pub address_v4: Option<(Ipv4Addr, u8)>,
/// Interface MTU.
pub mtu: u32,
}
@@ -381,7 +383,7 @@ mod system {
/// reason: duplicate address detection can never finish with no carrier,
/// and the address would stay tentative and unusable.
pub fn setup_commands(request: &TunRequest, user: &str) -> Vec<String> {
vec![
let mut commands = vec![
format!(
"sudo ip tuntap add dev {} mode tun user {user}",
request.name
@@ -398,7 +400,16 @@ mod system {
"sudo ip -6 address add {}/{} dev {} nodad",
request.address, request.prefix_len, request.name
),
]
];
if let Some((address, prefix_len)) = request.address_v4 {
// IPv4 is not sensitive to carrier the way IPv6 is, so it needs
// no extra settings.
commands.push(format!(
"sudo ip address add {address}/{prefix_len} dev {}",
request.name
));
}
commands
}
fn current_user() -> String {
@@ -578,7 +589,8 @@ fd559caf9652cb86321feac65c73bd84 05 40 00 08 tsun0
name: "tsun0".into(),
address: "fd00::1".parse().unwrap(),
prefix_len: 64,
mtu: 1100,
address_v4: Some(("100.64.1.2".parse().unwrap(), 10)),
mtu: 1280,
};
let commands = setup_commands(&request, "someone");
@@ -586,11 +598,24 @@ fd559caf9652cb86321feac65c73bd84 05 40 00 08 tsun0
// must survive losing carrier, and it must not wait for duplicate
// address detection that can never complete.
let joined = commands.join("\n");
let up = joined.find("link set dev tsun0 mtu 1100 up").unwrap();
let up = joined.find("link set dev tsun0 mtu 1280 up").unwrap();
let keep = joined.find("keep_addr_on_down=1").unwrap();
let add = joined.find("address add fd00::1/64").unwrap();
assert!(up < keep && keep < add, "wrong order:\n{joined}");
assert!(joined.contains("nodad"));
assert!(joined.contains("user someone"));
// IPv4 needs no carrier tricks, just the address.
assert!(joined.contains("ip address add 100.64.1.2/10 dev tsun0"));
// An IPv6-only overlay says nothing about IPv4.
let v6_only = TunRequest {
address_v4: None,
..request
};
assert!(
!setup_commands(&v6_only, "someone")
.join("\n")
.contains("100.64")
);
}
}
+14 -2
View File
@@ -136,6 +136,8 @@ pub struct OverlayReport {
pub mtu: u32,
/// This agent's overlay address.
pub address: String,
/// This agent's IPv4 overlay address, when the overlay is dual stack.
pub address_v4: Option<String>,
/// The subnet every member shares.
pub prefix: String,
/// Prefix length of that subnet.
@@ -155,6 +157,8 @@ pub struct OverlayPeerReport {
pub public_key: String,
/// Its overlay address.
pub address: String,
/// Its IPv4 overlay address, when it has one.
pub address_v4: Option<String>,
/// Seconds since the last WireGuard handshake.
///
/// `None` means the tunnel has never handshaken and cannot carry traffic.
@@ -229,10 +233,14 @@ impl StatusReport {
let up = overlay.peers.iter().filter(|peer| peer.is_up()).count();
let _ = writeln!(
out,
" overlay {} {}/{} mtu {} {}/{} tunnel(s) up",
" overlay {} {}/{}{} mtu {} {}/{} tunnel(s) up",
overlay.interface,
overlay.address,
overlay.prefix_len,
match &overlay.address_v4 {
Some(v4) => format!(" and {v4}"),
None => String::new(),
},
overlay.mtu,
up,
overlay.peers.len()
@@ -240,9 +248,13 @@ impl StatusReport {
for peer in &overlay.peers {
let _ = writeln!(
out,
" {} {} {} tx {} rx {}{} {}",
" {} {}{} {} tx {} rx {}{} {}",
&peer.public_key[..8.min(peer.public_key.len())],
peer.address,
match &peer.address_v4 {
Some(v4) => format!(" / {v4}"),
None => String::new(),
},
match peer.handshake_secs_ago {
Some(secs) => format!("handshake {secs}s ago"),
None => "NOT HANDSHAKEN".to_string(),
+164 -1
View File
@@ -10,7 +10,7 @@
mod common;
use std::net::{IpAddr, Ipv6Addr};
use std::net::{IpAddr, Ipv4Addr, Ipv6Addr};
use std::sync::Arc;
use std::time::Duration;
@@ -123,6 +123,24 @@ impl WgAgent {
}
}
/// Builds a minimal well-formed IPv4 packet.
fn ipv4_packet(source: Ipv4Addr, destination: Ipv4Addr, payload: &[u8]) -> Bytes {
let total = 20 + payload.len();
let mut packet = Vec::with_capacity(total);
packet.push((4 << 4) | 5); // version 4, header length 5 words
packet.push(0); // dscp/ecn
packet.extend_from_slice(&(total as u16).to_be_bytes());
packet.extend_from_slice(&[0, 0]); // identification
packet.extend_from_slice(&[0, 0]); // flags and fragment offset
packet.push(64); // ttl
packet.push(253); // an experimental protocol number
packet.extend_from_slice(&[0, 0]); // checksum, not verified by the overlay
packet.extend_from_slice(&source.octets());
packet.extend_from_slice(&destination.octets());
packet.extend_from_slice(payload);
Bytes::from(packet)
}
/// Builds a minimal well-formed IPv6 packet.
fn ipv6_packet(source: Ipv6Addr, destination: Ipv6Addr, payload: &[u8]) -> Bytes {
let mut packet = Vec::with_capacity(40 + payload.len());
@@ -244,6 +262,151 @@ async fn a_peer_cannot_send_from_an_address_it_does_not_own() {
b.shutdown().await;
}
#[tokio::test]
async fn the_overlay_carries_ipv4_alongside_ipv6() {
let discovery = SharedMemoryDiscovery::new();
let (name, secret) = network("wg-dual-stack");
let a = WgAgent::spawn(&discovery, "ta").await;
let b = WgAgent::spawn(&discovery, "tb").await;
let network_id = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
a.wait_for_tunnels(network_id, 1).await;
b.wait_for_tunnels(network_id, 1).await;
let view_a = a.plugin.overview(network_id).unwrap();
let view_b = b.plugin.overview(network_id).unwrap();
let v4_a = view_a.overlay_address_v4.expect("dual stack by default");
let v4_b = view_b.overlay_address_v4.expect("dual stack by default");
assert_ne!(v4_a, v4_b);
// Both inside the configured range.
for addr in [v4_a, v4_b] {
assert_eq!(
u32::from(addr) & 0xffc0_0000,
u32::from(Ipv4Addr::new(100, 64, 0, 0))
);
}
// Each side derived the other's address identically.
assert_eq!(view_a.peers[0].overlay_address_v4, Some(v4_b));
assert_eq!(view_b.peers[0].overlay_address_v4, Some(v4_a));
assert!(!view_a.peers[0].tunnel.as_ref().unwrap().ipv4_conflict);
let tun_a = a.tun(network_id).await;
let tun_b = b.tun(network_id).await;
// A real IPv4 packet through the same tunnel.
tun_a.push_from_os(ipv4_packet(v4_a, v4_b, b"ipv4 over the overlay"));
let received = tokio::time::timeout(common::DEADLINE, tun_b.pop_to_os())
.await
.expect("the IPv4 packet should arrive")
.unwrap();
assert_eq!(received[0] >> 4, 4, "still an IPv4 packet");
assert_eq!(&received[12..16], &v4_a.octets());
assert_eq!(&received[16..20], &v4_b.octets());
assert_eq!(&received[20..], b"ipv4 over the overlay");
// IPv6 keeps working on the same tunnel.
let addr_a = a.overlay(network_id).await;
let addr_b = b.overlay(network_id).await;
tun_a.push_from_os(ipv6_packet(addr_a, addr_b, b"and ipv6 too"));
let received = tokio::time::timeout(common::DEADLINE, tun_b.pop_to_os())
.await
.expect("the IPv6 packet should arrive")
.unwrap();
assert_eq!(received[0] >> 4, 6);
assert_eq!(&received[40..], b"and ipv6 too");
a.shutdown().await;
b.shutdown().await;
}
#[tokio::test]
async fn an_ipv4_source_a_peer_does_not_own_is_dropped() {
let discovery = SharedMemoryDiscovery::new();
let (name, secret) = network("wg-v4-spoof");
let a = WgAgent::spawn(&discovery, "ta").await;
let b = WgAgent::spawn(&discovery, "tb").await;
let network_id = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
a.wait_for_tunnels(network_id, 1).await;
b.wait_for_tunnels(network_id, 1).await;
let v4_a = a
.plugin
.overview(network_id)
.unwrap()
.overlay_address_v4
.unwrap();
let v4_b = b
.plugin
.overview(network_id)
.unwrap()
.overlay_address_v4
.unwrap();
let tun_a = a.tun(network_id).await;
let tun_b = b.tun(network_id).await;
// A claims an IPv4 address that is not the one derived from its key.
let forged = Ipv4Addr::from(u32::from(v4_a) ^ 0x0000_00ff);
tun_a.push_from_os(ipv4_packet(forged, v4_b, b"spoofed v4"));
wait_until("the spoofed IPv4 packet is dropped", || async {
let view = b.plugin.overview(network_id)?;
let tunnel = view.peers.first()?.tunnel.as_ref()?;
(tunnel.stats.dropped_wrong_source >= 1).then_some(())
})
.await;
// The honest one still gets through.
tun_a.push_from_os(ipv4_packet(v4_a, v4_b, b"honest v4"));
let received = tokio::time::timeout(common::DEADLINE, tun_b.pop_to_os())
.await
.expect("the honest packet should arrive")
.unwrap();
assert_eq!(&received[20..], b"honest v4");
a.shutdown().await;
b.shutdown().await;
}
#[tokio::test]
async fn an_ipv6_only_overlay_can_be_asked_for() {
let discovery = SharedMemoryDiscovery::new();
let (name, secret) = network("wg-v6-only");
let a = WgAgent::spawn_with(&discovery, "ta", |config| config.with_ipv4_range(None)).await;
let b = WgAgent::spawn_with(&discovery, "tb", |config| config.with_ipv4_range(None)).await;
let network_id = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
a.wait_for_tunnels(network_id, 1).await;
let view = a.plugin.overview(network_id).unwrap();
assert_eq!(view.overlay_address_v4, None);
assert_eq!(view.ipv4_range, None);
assert_eq!(view.peers[0].overlay_address_v4, None);
// No IPv4 configured is not a conflict.
assert!(!view.peers[0].tunnel.as_ref().unwrap().ipv4_conflict);
// IPv6 is unaffected.
let addr_a = a.overlay(network_id).await;
let addr_b = b.overlay(network_id).await;
a.tun(network_id)
.await
.push_from_os(ipv6_packet(addr_a, addr_b, b"v6 only"));
let received = tokio::time::timeout(common::DEADLINE, b.tun(network_id).await.pop_to_os())
.await
.expect("the packet should arrive")
.unwrap();
assert_eq!(&received[40..], b"v6 only");
a.shutdown().await;
b.shutdown().await;
}
#[tokio::test]
async fn packets_for_an_unknown_address_are_counted_not_broadcast() {
let discovery = SharedMemoryDiscovery::new();