Allocate IPv4 addresses and keep them, as signed state

Derived IPv4 addresses could not survive anything: they changed with the
range, and there was no way for a member to come back to the one it had.
Addresses are now allocated and recorded as signed facts, which is the
first slice of the model in docs/sync-model.md.

src/state/ holds one record per author per network, carrying that author's
complete current statement, signed with its persistent device key over a
length-prefixed canonical encoding. Merging follows the model's rules: a
higher version wins, an older one never rolls back a newer, duplicates are
idempotent, absence from a snapshot is not deletion, and a same-version
conflict is resolved identically on every replica and reported rather than
letting replicas diverge. Records are persisted in state.sqlite, with the
record and the author's version counter committed in one transaction
before anything is announced, and distributed as a State control message
that is merged into what the receiver already holds.

No vote, deliberately, despite the request. A majority is not a trust root
here — anyone with the secret can mint identities — and a quorum would
stall with one peer online and diverge across a partition. Signatures plus
a deterministic merge converge without either failure mode: two members
claiming one address at once are resolved by the lower endpoint id, and
the loser allocates again with a higher version.

The range moved from the plugin to the agent, defaults to 10.13.37.0/24,
and is now agreed rather than configured per member: a joining agent
adopts what the network already uses, so --ipv4-range only matters for
whoever starts it. The announcement went back to identity only (version 3)
since the range travels in signed records now.

A release tombstone exists and merges correctly, but nothing emits one
yet.

116 tests. The headline ones: an address survives restarting both agents,
three members get three distinct addresses, and a member started with a
different range adopts the one in use. Confirmed by hand with two CLI
agents restarted end to end.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
tsunagi
2026-09-21 13:43:01 +01:00
co-authored by Claude Opus 5
parent ce64264027
commit 84c06c6cac
25 changed files with 1967 additions and 365 deletions
+15 -56
View File
@@ -18,14 +18,15 @@ use crate::dataplane::PluginError;
use crate::identity::NetworkId;
use super::keys::WgPublicKey;
use super::overlay::{Ipv4Range, overlay_address};
use super::overlay::overlay_address;
/// Version of the announcement format.
///
/// Bumped to 2 when the IPv4 overlay range was added. postcard is not
/// self-describing, so an older peer cannot read a newer announcement; the
/// mismatch is reported rather than misparsed.
pub const ANNOUNCEMENT_VERSION: u16 = 2;
/// Version 3 dropped the IPv4 range again: overlay addressing moved to the
/// signed records in [`crate::state`], which carry the range and survive a
/// participant being away. postcard is not self-describing, so an older peer
/// cannot read a newer announcement; the mismatch is reported, not misparsed.
pub const ANNOUNCEMENT_VERSION: u16 = 3;
/// What one participant advertises for the WireGuard data plane.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
@@ -39,12 +40,6 @@ pub struct WgAnnouncement {
/// Carried for diagnostics and cross-checking only. Addresses are always
/// derived locally, never taken from this field.
pub overlay_address: Ipv6Addr,
/// The IPv4 overlay range this peer is configured with, if any.
///
/// Not a request and not trusted: it exists so that two members who were
/// configured differently find out, instead of silently deriving
/// different addresses for each other and misrouting IPv4.
pub ipv4_range: Option<Ipv4Range>,
}
/// A peer announcement that has been validated against a specific network.
@@ -54,22 +49,15 @@ pub struct ValidatedAnnouncement {
pub public_key: WgPublicKey,
/// The overlay address derived locally for this key. Authoritative.
pub overlay_address: Ipv6Addr,
/// The IPv4 overlay range the peer is configured with.
pub ipv4_range: Option<Ipv4Range>,
}
impl WgAnnouncement {
/// Builds this agent's announcement.
pub fn new(
network: NetworkId,
public_key: &WgPublicKey,
ipv4_range: Option<Ipv4Range>,
) -> Self {
pub fn new(network: NetworkId, public_key: &WgPublicKey) -> Self {
Self {
version: ANNOUNCEMENT_VERSION,
public_key: *public_key.as_bytes(),
overlay_address: overlay_address(network, public_key),
ipv4_range,
}
}
@@ -127,18 +115,9 @@ impl WgAnnouncement {
));
}
if let Some(range) = self.ipv4_range
&& range.prefix_len > 30
{
return Err(PluginError::Rejected(format!(
"announced IPv4 range {range} has no room for hosts"
)));
}
Ok(ValidatedAnnouncement {
public_key,
overlay_address: derived,
ipv4_range: self.ipv4_range,
})
}
}
@@ -166,7 +145,7 @@ mod tests {
let peer = WgSecretKey::generate().public();
let local = WgSecretKey::generate().public();
let payload = WgAnnouncement::new(id, &peer, None).encode().unwrap();
let payload = WgAnnouncement::new(id, &peer).encode().unwrap();
let validated = WgAnnouncement::decode_and_validate(&payload, id, &local).unwrap();
assert_eq!(validated.public_key, peer);
@@ -179,7 +158,7 @@ mod tests {
// carried here, so there is nothing for a peer to lie about.
let id = network("identity-only");
let peer = WgSecretKey::generate().public();
let payload = WgAnnouncement::new(id, &peer, None).encode().unwrap();
let payload = WgAnnouncement::new(id, &peer).encode().unwrap();
assert!(
payload.len() < 80,
"the announcement should stay tiny, got {} bytes",
@@ -195,7 +174,7 @@ mod tests {
let local = WgSecretKey::generate().public();
// An attacker claims the victim's overlay address with its own key.
let mut forged = WgAnnouncement::new(id, &attacker, None);
let mut forged = WgAnnouncement::new(id, &attacker);
forged.overlay_address = overlay_address(id, &victim);
let result = WgAnnouncement::decode_and_validate(&forged.encode().unwrap(), id, &local);
@@ -212,7 +191,7 @@ mod tests {
let peer = WgSecretKey::generate().public();
let local = WgSecretKey::generate().public();
let payload = WgAnnouncement::new(there, &peer, None).encode().unwrap();
let payload = WgAnnouncement::new(there, &peer).encode().unwrap();
assert!(WgAnnouncement::decode_and_validate(&payload, here, &local).is_err());
}
@@ -227,7 +206,7 @@ mod tests {
let wrong_version = WgAnnouncement {
version: ANNOUNCEMENT_VERSION + 1,
..WgAnnouncement::new(id, &peer, None)
..WgAnnouncement::new(id, &peer)
};
assert!(
WgAnnouncement::decode_and_validate(&wrong_version.encode().unwrap(), id, &local)
@@ -236,38 +215,18 @@ mod tests {
let zero_key = WgAnnouncement {
public_key: [0u8; 32],
..WgAnnouncement::new(id, &peer, None)
..WgAnnouncement::new(id, &peer)
};
assert!(
WgAnnouncement::decode_and_validate(&zero_key.encode().unwrap(), id, &local).is_err()
);
}
#[test]
fn the_ipv4_range_travels_so_a_mismatch_can_be_seen() {
let id = network("ranges");
let peer = WgSecretKey::generate().public();
let local = WgSecretKey::generate().public();
let range = Some(Ipv4Range::new("10.9.0.0".parse().unwrap(), 16).unwrap());
let payload = WgAnnouncement::new(id, &peer, range).encode().unwrap();
let validated = WgAnnouncement::decode_and_validate(&payload, id, &local).unwrap();
assert_eq!(validated.ipv4_range, range);
// A range with no usable hosts is nonsense and is refused.
let mut bad = WgAnnouncement::new(id, &peer, range);
bad.ipv4_range = Some(Ipv4Range {
base: "10.9.0.0".parse().unwrap(),
prefix_len: 31,
});
assert!(WgAnnouncement::decode_and_validate(&bad.encode().unwrap(), id, &local).is_err());
}
#[test]
fn a_peer_cannot_claim_our_own_key() {
let id = network("self");
let local = WgSecretKey::generate().public();
let payload = WgAnnouncement::new(id, &local, None).encode().unwrap();
let payload = WgAnnouncement::new(id, &local).encode().unwrap();
assert!(WgAnnouncement::decode_and_validate(&payload, id, &local).is_err());
}
@@ -275,7 +234,7 @@ mod tests {
fn announcements_stay_well_under_the_capability_payload_limit() {
let id = network("size");
let peer = WgSecretKey::generate().public();
let payload = WgAnnouncement::new(id, &peer, None).encode().unwrap();
let payload = WgAnnouncement::new(id, &peer).encode().unwrap();
assert!(
payload.len() < crate::config::Limits::default().max_capability_data_len,
"announcement is {} bytes",
+2 -1
View File
@@ -42,9 +42,10 @@ use crate::dataplane::transport::{SharedLink, TransportError};
use crate::identity::NetworkId;
use super::keys::{WgPublicKey, WgSecretKey};
use super::overlay::{Ipv4Range, overlay_address};
use super::overlay::overlay_address;
use super::packet::IpHeader;
use super::tun::TunDevice;
use crate::state::Ipv4Range;
/// How often WireGuard's own timers are driven.
///
+2 -4
View File
@@ -50,14 +50,12 @@ pub mod plugin;
pub mod store;
pub mod tun;
pub use crate::state::Ipv4Range;
pub use announcement::{ValidatedAnnouncement, WgAnnouncement};
pub use config::{Cidr, DEFAULT_INTERFACE_PREFIX, MAX_INTERFACE_NAME_LEN, interface_name};
pub use device::{PeerHealth, PeerStats, PeerSummary, WireguardDevice};
pub use keys::{WgPublicKey, WgSecretKey};
pub use overlay::{
Ipv4Range, OVERLAY_PREFIX_LEN, RFC6598_SHARED_RANGE, overlay_address, overlay_address_v4,
overlay_prefix,
};
pub use overlay::{OVERLAY_PREFIX_LEN, overlay_address, overlay_address_v4, overlay_prefix};
pub use packet::IpHeader;
pub use plugin::{
DEFAULT_MTU, MIN_MTU, NetworkOverview, PeerOverview, WIREGUARD_OVERHEAD, WIREGUARD_PROTOCOL,
+3 -75
View File
@@ -22,10 +22,9 @@
use std::net::{Ipv4Addr, Ipv6Addr};
use serde::{Deserialize, Serialize};
use sha2::{Digest, Sha256};
use crate::dataplane::PluginError;
use crate::state::Ipv4Range;
use crate::identity::NetworkId;
@@ -40,77 +39,6 @@ pub const OVERLAY_PREFIX_LEN: u8 = 64;
/// Prefix length of one member's address inside the overlay.
pub const OVERLAY_HOST_PREFIX_LEN: u8 = 128;
/// An IPv4 range the overlay can be derived into.
///
/// Every member of a network must be configured with the same one, because
/// addresses are derived from it. See [`crate::dataplane::wireguard::plugin::WireguardConfig::ipv4_range`].
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
pub struct Ipv4Range {
/// Base address of the range.
pub base: Ipv4Addr,
/// Prefix length, at most 30 so there is room for hosts.
pub prefix_len: u8,
}
impl Ipv4Range {
/// Builds a range, rejecting one with no room for hosts.
pub fn new(base: Ipv4Addr, prefix_len: u8) -> Result<Self, PluginError> {
if prefix_len > 30 {
return Err(PluginError::Rejected(format!(
"a /{prefix_len} has no room for hosts; use /30 or larger"
)));
}
Ok(Self { base, prefix_len })
}
/// Whether an address falls inside the range.
pub fn contains(&self, address: Ipv4Addr) -> bool {
let host_bits = 32 - u32::from(self.prefix_len);
let mask = if host_bits >= 32 {
0
} else {
u32::MAX << host_bits
};
u32::from(address) & mask == u32::from(self.base) & mask
}
}
impl std::fmt::Display for Ipv4Range {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "{}/{}", self.base, self.prefix_len)
}
}
impl std::str::FromStr for Ipv4Range {
type Err = PluginError;
fn from_str(text: &str) -> Result<Self, Self::Err> {
let (base, prefix) = text.split_once('/').ok_or_else(|| {
PluginError::Rejected(format!(
"`{text}` is not an address with a prefix, for example 10.77.0.0/16"
))
})?;
let base = base.parse().map_err(|err| {
PluginError::Rejected(format!("`{base}` is not an IPv4 address: {err}"))
})?;
let prefix_len = prefix.parse().map_err(|err| {
PluginError::Rejected(format!("`{prefix}` is not a prefix length: {err}"))
})?;
Self::new(base, prefix_len)
}
}
/// RFC 6598 shared address space, offered only as a reference point.
///
/// **Not a default, and usually a bad choice.** Tailscale uses exactly this
/// range, and so does carrier-grade NAT, so a machine running either will
/// collide with it. There is no IPv4 range that is free on every host, which
/// is why the IPv4 overlay has no default at all and must be configured.
pub const RFC6598_SHARED_RANGE: Ipv4Range = Ipv4Range {
base: Ipv4Addr::new(100, 64, 0, 0),
prefix_len: 10,
};
fn push_lp(out: &mut Vec<u8>, bytes: &[u8]) {
let len = u32::try_from(bytes.len()).unwrap_or(u32::MAX);
out.extend_from_slice(&len.to_be_bytes());
@@ -252,7 +180,7 @@ mod tests {
#[test]
fn ipv4_addresses_land_inside_the_range_and_avoid_its_edges() {
let id = network("v4");
let range = RFC6598_SHARED_RANGE;
let range: Ipv4Range = "100.64.0.0/10".parse().unwrap();
for byte in 0..64u8 {
let key = WgPublicKey::from_bytes([byte; 32]);
let addr = overlay_address_v4(id, &key, range).unwrap();
@@ -286,7 +214,7 @@ mod tests {
let key = WgPublicKey::from_bytes([9u8; 32]);
let first = network("one");
let second = network("two");
let range = RFC6598_SHARED_RANGE;
let range: Ipv4Range = "100.64.0.0/10".parse().unwrap();
assert_eq!(
overlay_address_v4(first, &key, range),
+65 -81
View File
@@ -43,11 +43,10 @@ use super::announcement::{ValidatedAnnouncement, WgAnnouncement};
use super::config::{DEFAULT_INTERFACE_PREFIX, interface_name};
use super::device::{PeerSummary, WireguardDevice};
use super::keys::{WgPublicKey, WgSecretKey};
use super::overlay::{
Ipv4Range, OVERLAY_PREFIX_LEN, overlay_address, overlay_address_v4, overlay_prefix,
};
use super::overlay::{OVERLAY_PREFIX_LEN, overlay_address, overlay_prefix};
use super::store::WgKeyStore;
use super::tun::{TunFactory, TunRequest};
use crate::state::Ipv4Range;
/// The protocol identifier this plugin announces.
pub const WIREGUARD_PROTOCOL: &str = "wireguard";
@@ -88,21 +87,6 @@ pub struct WireguardConfig {
pub keepalive: Option<u16>,
/// Interface MTU. See [`DEFAULT_MTU`].
pub mtu: u32,
/// IPv4 overlay range, or `None` for an IPv6-only overlay.
///
/// **Every member of a network must configure the same range.** Addresses
/// are derived from it, so two members configured differently would
/// derive different addresses for each other. The range travels in the
/// announcement purely so that such a mismatch is detected and reported
/// instead of silently misrouting.
///
/// There is no default, because no IPv4 range is free on every host:
/// `100.64.0.0/10` belongs to Tailscale and to carrier-grade NAT,
/// `10.0.0.0/8` and `192.168.0.0/16` are everywhere, `172.17.0.0/16` is
/// Docker. Pick one you know is unused on every machine that will join.
/// IPv6 needs none of this: its addresses are derived from the network
/// id and never collide.
pub ipv4_range: Option<Ipv4Range>,
/// How long to coalesce changes before reconciling.
pub reconcile_debounce: Duration,
/// How often to reconcile anyway, which is also when a packet interface
@@ -118,9 +102,6 @@ impl WireguardConfig {
interface_prefix: DEFAULT_INTERFACE_PREFIX.to_string(),
keepalive: Some(25),
mtu: DEFAULT_MTU,
// Off by default: no IPv4 range is free on every host. See
// `with_ipv4_range`.
ipv4_range: None,
reconcile_debounce: Duration::from_millis(200),
reconcile_interval: Duration::from_secs(15),
}
@@ -140,14 +121,6 @@ impl WireguardConfig {
self
}
/// Sets the IPv4 overlay range, or disables IPv4 with `None`.
///
/// Must match on every member; see the field documentation.
pub fn with_ipv4_range(mut self, range: Option<Ipv4Range>) -> Self {
self.ipv4_range = range;
self
}
/// Sets the reconciliation timings.
pub fn with_reconcile(mut self, debounce: Duration, interval: Duration) -> Self {
self.reconcile_debounce = debounce;
@@ -230,6 +203,10 @@ struct NetworkState {
device: Option<Arc<WireguardDevice>>,
announcements: HashMap<EndpointId, ValidatedAnnouncement>,
links: HashMap<EndpointId, SharedLink>,
/// What the network agreed, pushed in by the agent. Authoritative.
allocations: HashMap<EndpointId, Ipv4Addr>,
/// The range those allocations came from.
ipv4_range: Option<Ipv4Range>,
}
#[derive(Debug, Default)]
@@ -252,6 +229,8 @@ enum Command {
struct Worker {
config: WireguardConfig,
/// This agent's endpoint id, learned when the plugin is attached.
local_id: OnceLock<EndpointId>,
tun_factory: Arc<dyn TunFactory>,
store: WgKeyStore,
shared: Mutex<Shared>,
@@ -303,6 +282,7 @@ impl WireguardPlugin {
let worker = Arc::new(Worker {
config,
local_id: OnceLock::new(),
tun_factory,
store,
shared: Mutex::new(Shared::default()),
@@ -343,9 +323,7 @@ impl WireguardPlugin {
endpoint_id: *endpoint_id,
public_key: announcement.public_key,
overlay_address: IpAddr::V6(announcement.overlay_address),
overlay_address_v4: tunnels
.get(&announcement.public_key)
.and_then(|tunnel| tunnel.overlay_address_v4),
overlay_address_v4: state.allocations.get(endpoint_id).copied(),
has_link: state.links.contains_key(endpoint_id),
tunnel: tunnels.get(&announcement.public_key).cloned(),
})
@@ -360,12 +338,8 @@ impl WireguardPlugin {
overlay_address: IpAddr::V6(overlay_address(network, &state.key.public())),
overlay_prefix: IpAddr::V6(overlay_prefix(network)),
overlay_prefix_len: OVERLAY_PREFIX_LEN,
overlay_address_v4: self
.worker
.config
.ipv4_range
.and_then(|range| overlay_address_v4(network, &state.key.public(), range)),
ipv4_range: self.worker.config.ipv4_range,
overlay_address_v4: state.allocations.get(&self.worker.local_id()).copied(),
ipv4_range: state.ipv4_range,
peers,
unroutable_packets: state
.device
@@ -395,6 +369,13 @@ impl WireguardPlugin {
}
impl Worker {
/// This agent's endpoint id, or a placeholder before it is attached.
fn local_id(&self) -> EndpointId {
self.local_id.get().copied().unwrap_or_else(|| {
EndpointId::from_bytes(&[1u8; 32]).unwrap_or_else(|_| unreachable!("a fixed valid key"))
})
}
fn lock_shared(&self) -> std::sync::MutexGuard<'_, Shared> {
match self.shared.lock() {
Ok(guard) => guard,
@@ -452,6 +433,8 @@ impl Worker {
device: None,
announcements: HashMap::new(),
links: HashMap::new(),
allocations: HashMap::new(),
ipv4_range: None,
});
}
@@ -466,12 +449,15 @@ impl Worker {
/// Creates the packet interface and starts the WireGuard device.
async fn ensure_device(&self, network: NetworkId) -> Result<(), PluginError> {
let (name, key) = {
let (name, key, own_v4, own_range) = {
let shared = self.lock_shared();
match shared.networks.get(&network) {
Some(state) if state.device.is_none() => {
(state.interface.clone(), state.key.clone())
}
Some(state) if state.device.is_none() => (
state.interface.clone(),
state.key.clone(),
state.allocations.get(&self.local_id()).copied(),
state.ipv4_range,
),
_ => return Ok(()),
}
};
@@ -480,20 +466,12 @@ impl Worker {
name: name.clone(),
address: overlay_address(network, &key.public()),
prefix_len: OVERLAY_PREFIX_LEN,
address_v4: self
.config
.ipv4_range
.and_then(|range| overlay_address_v4(network, &key.public(), range)),
prefix_len_v4: self.config.ipv4_range.map_or(0, |range| range.prefix_len),
address_v4: own_v4,
prefix_len_v4: own_range.map_or(0, |range| range.prefix_len),
mtu: self.config.mtu,
};
let tun = self.tun_factory.create(request).await?;
let device = Arc::new(WireguardDevice::start(
network,
key,
tun,
self.config.ipv4_range,
));
let device = Arc::new(WireguardDevice::start(network, key, tun, own_range));
let mut shared = self.lock_shared();
if let Some(state) = shared.networks.get_mut(&network) {
@@ -516,9 +494,9 @@ impl Worker {
return;
};
let allocations = state.allocations.clone();
let mut wanted: Vec<WgPublicKey> = Vec::new();
let mut too_small: Vec<(usize, usize)> = Vec::new();
let mut mismatched: Vec<(WgPublicKey, Ipv4Range, Ipv4Range)> = Vec::new();
for (endpoint_id, announcement) in &state.announcements {
let Some(link) = state.links.get(endpoint_id) else {
continue;
@@ -540,19 +518,10 @@ impl Worker {
too_small.push((available, needed));
}
// A peer only gets an IPv4 address if both sides were configured
// with the same range. Otherwise the two would derive different
// addresses for each other and IPv4 would silently misroute.
let peer_v4 = match (self.config.ipv4_range, announcement.ipv4_range) {
(Some(ours), Some(theirs)) if ours == theirs => {
overlay_address_v4(network, &announcement.public_key, ours)
}
(Some(ours), Some(theirs)) => {
mismatched.push((announcement.public_key, ours, theirs));
None
}
(Some(_), None) | (None, Some(_)) | (None, None) => None,
};
// The address comes from the agreed signed state, not from
// anything this peer said and not from a derivation: that is what
// makes it survive the peer being away.
let peer_v4 = allocations.get(endpoint_id).copied();
if let Err(err) = device.add_peer(
*endpoint_id,
@@ -567,18 +536,6 @@ impl Worker {
device.retain_peers(&wanted);
drop(shared);
for (key, ours, theirs) in mismatched {
self.report(
network,
format!(
"peer {} is configured with the IPv4 overlay range {theirs} but this agent \
uses {ours}; every member must use the same one. That peer has no IPv4 \
address here and is reachable over IPv6 only.",
key.fmt_short()
),
);
}
for (available, needed) in too_small {
self.report(
network,
@@ -686,6 +643,9 @@ impl IpPlugin for WireguardPlugin {
}
fn attach(&self, context: PluginContext) {
if let Some(local) = context.local_endpoint_id() {
let _ = self.worker.local_id.set(local);
}
let _ = self.worker.context.set(context);
}
@@ -707,8 +667,7 @@ impl IpPlugin for WireguardPlugin {
};
// Identity only. Where to send packets is the transport's business.
let announcement =
WgAnnouncement::new(network, &state.key.public(), self.worker.config.ipv4_range);
let announcement = WgAnnouncement::new(network, &state.key.public());
Ok(Some(PluginCapability {
protocol: WIREGUARD_PROTOCOL.to_string(),
version: super::announcement::ANNOUNCEMENT_VERSION,
@@ -754,6 +713,31 @@ impl IpPlugin for WireguardPlugin {
Ok(())
}
fn on_address_allocation(
&self,
network: NetworkId,
range: Ipv4Range,
allocations: &[(EndpointId, Ipv4Addr)],
) {
let changed = {
let mut shared = self.worker.lock_shared();
match shared.networks.get_mut(&network) {
Some(state) => {
let fresh: HashMap<EndpointId, Ipv4Addr> =
allocations.iter().copied().collect();
let changed = state.allocations != fresh || state.ipv4_range != Some(range);
state.allocations = fresh;
state.ipv4_range = Some(range);
changed
}
None => false,
}
};
if changed {
self.nudge(Command::Sync(network));
}
}
fn on_peer_link(&self, network: NetworkId, peer: EndpointId, link: SharedLink) {
self.nudge(Command::Link {
network,