Allocate IPv4 addresses and keep them, as signed state
Derived IPv4 addresses could not survive anything: they changed with the range, and there was no way for a member to come back to the one it had. Addresses are now allocated and recorded as signed facts, which is the first slice of the model in docs/sync-model.md. src/state/ holds one record per author per network, carrying that author's complete current statement, signed with its persistent device key over a length-prefixed canonical encoding. Merging follows the model's rules: a higher version wins, an older one never rolls back a newer, duplicates are idempotent, absence from a snapshot is not deletion, and a same-version conflict is resolved identically on every replica and reported rather than letting replicas diverge. Records are persisted in state.sqlite, with the record and the author's version counter committed in one transaction before anything is announced, and distributed as a State control message that is merged into what the receiver already holds. No vote, deliberately, despite the request. A majority is not a trust root here — anyone with the secret can mint identities — and a quorum would stall with one peer online and diverge across a partition. Signatures plus a deterministic merge converge without either failure mode: two members claiming one address at once are resolved by the lower endpoint id, and the loser allocates again with a higher version. The range moved from the plugin to the agent, defaults to 10.13.37.0/24, and is now agreed rather than configured per member: a joining agent adopts what the network already uses, so --ipv4-range only matters for whoever starts it. The announcement went back to identity only (version 3) since the range travels in signed records now. A release tombstone exists and merges correctly, but nothing emits one yet. 116 tests. The headline ones: an address survives restarting both agents, three members get three distinct addresses, and a member started with a different range adopts the one in use. Confirmed by hand with two CLI agents restarted end to end. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
tsunagi
Claude Opus 5
@@ -45,6 +45,7 @@ and the agent stays manageable.
|
||||
| `proto` | message format, handshake, membership proof, protocol limits |
|
||||
| `agent` | agent and per-network lifecycle, reconnect, in-process message routing |
|
||||
| `storage` | mandatory state and the separately recoverable cache |
|
||||
| `state` | signed records that outlive a session, merged between replicas |
|
||||
| `dataplane::transport` | authenticated datagram links to peers; where reachability lives |
|
||||
| `dataplane` | the contract IP plugins implement, plus the WireGuard plugin |
|
||||
|
||||
|
||||
@@ -172,8 +172,15 @@ not affect other networks.
|
||||
| `Announce { hostname, capabilities }` | this agent's hostname and IP-plugin capabilities |
|
||||
| `Ping { seq, payload }` | small request used to verify the exchange |
|
||||
| `Pong { seq, payload }` | the echoed reply |
|
||||
| `State { records }` | a snapshot of signed records, merged into what the receiver holds |
|
||||
| `Bye { reason }` | graceful goodbye; not a revocation of anything |
|
||||
|
||||
A `State` snapshot is merged, never substituted: an author missing from it is
|
||||
left untouched. Each record carries its own signature, so a peer forwarding
|
||||
somebody else's record cannot alter it, and a record that fails verification
|
||||
is dropped without affecting the rest of the batch. See
|
||||
[sync-model.md](sync-model.md).
|
||||
|
||||
`PluginCapability { protocol, version, enabled, data }` is opaque to the core:
|
||||
`data` is bounded and handed to the matching plugin unparsed. Nothing in it is
|
||||
ever treated as a shell command, filesystem path or OS setting.
|
||||
|
||||
+33
-5
@@ -1,10 +1,13 @@
|
||||
# Planned state synchronisation
|
||||
|
||||
**Nothing in this document is implemented.** The proof of concept exchanges
|
||||
hostname and capability announcements over live sessions and keeps no
|
||||
replicated history. That is also why WireGuard peer membership is
|
||||
session-scoped today: a peer leaves the overlay when its control session ends,
|
||||
because there is no agreed durable state to keep it. This file records the intended direction so the module
|
||||
**The first slice of this model is now implemented**, in `src/state/`, and is
|
||||
used for one thing: IPv4 overlay addresses. What follows describes the whole
|
||||
model; the section at the end says exactly which parts exist.
|
||||
|
||||
The proof of concept still exchanges hostname and capability announcements
|
||||
over live sessions and keeps no replicated history for those, which is why
|
||||
WireGuard peer *membership* remains session-scoped even though a peer's
|
||||
*address* no longer is. This file records the intended direction so the module
|
||||
boundaries in [architecture.md](architecture.md) stay compatible with it, and so
|
||||
nobody mistakes the current announcements for synchronisation.
|
||||
|
||||
@@ -72,6 +75,31 @@ migrations for this.
|
||||
- Anyone who knows the secret can author records, so a majority of records is
|
||||
not evidence of anything.
|
||||
|
||||
## What exists today
|
||||
|
||||
Implemented, in `src/state/`:
|
||||
|
||||
* signed records, one per author per network, each holding that author's
|
||||
complete current statement rather than a delta;
|
||||
* signing and verification with the persistent iroh device key, over a
|
||||
length-prefixed canonical encoding;
|
||||
* the merge rules above: higher version wins, an older version never rolls
|
||||
back a newer one, duplicates are idempotent, a same-version conflict is
|
||||
resolved identically on every replica and reported;
|
||||
* a release tombstone, which merges correctly and is not undone by a replica
|
||||
that has not heard of it — though nothing emits one yet, so freeing an
|
||||
address still means forgetting the network;
|
||||
* persistence in `state.sqlite`, with the record and the author's own version
|
||||
counter committed in **one transaction before the record is announced**;
|
||||
* distribution as a `State` control message, merged into what the receiver
|
||||
already holds rather than replacing it;
|
||||
* allocation of a free IPv4 address against what everybody else holds, which
|
||||
is what makes an address stable across an absence.
|
||||
|
||||
Deliberately not implemented: compaction, revoking a whole author, record
|
||||
types beyond addressing, and any bound on how large a snapshot may grow
|
||||
beyond the per-message limit.
|
||||
|
||||
## Future tests
|
||||
|
||||
These are **not implemented and must not be reported as passing**:
|
||||
|
||||
@@ -50,6 +50,15 @@ keeping the WireGuard identity, shutdown removing every interface, a forged
|
||||
overlay claim being rejected, and the core carrying the payload without
|
||||
interpreting it.
|
||||
|
||||
Unit tests in `src/state/` cover the signed record model directly: tampering
|
||||
with any field breaks verification, a newer version wins while an older one
|
||||
never rolls back, two authors claiming one address resolve the same way no
|
||||
matter the merge order, one key used in two places is reported rather than
|
||||
silently merged, a release survives a late-arriving old claim, a bad record in
|
||||
a batch does not stop the rest, and allocation is deterministic, spread out,
|
||||
walks past everything taken and reports a full range instead of handing out a
|
||||
duplicate.
|
||||
|
||||
`tests/local_control.rs` covers the local control socket end to end: a client
|
||||
asking a running agent for status over a real Unix socket, a leftover socket
|
||||
file being replaced while a live one is not, and the derived socket path
|
||||
|
||||
+44
-41
@@ -83,51 +83,52 @@ Two consequences matter:
|
||||
* a member's address is bound to its WireGuard public key, so address
|
||||
ownership can be checked locally rather than believed.
|
||||
|
||||
## IPv4 alongside IPv6
|
||||
## IPv4: allocated, signed, and kept
|
||||
|
||||
The overlay carries IPv4 as well, but **it is off unless you name a range**,
|
||||
and every member must name the same one:
|
||||
|
||||
```bash
|
||||
tsunagi up --network lab --secret "$SECRET" --wireguard --ipv4-range 10.77.0.0/16
|
||||
```
|
||||
|
||||
Two reasons it has no default.
|
||||
|
||||
**There is no IPv4 range that is free everywhere.** `100.64.0.0/10` is
|
||||
Tailscale's and carrier-grade NAT's, `10.0.0.0/8` and `192.168.0.0/16` are on
|
||||
half the networks in the world, `172.17.0.0/16` is Docker. Picking one
|
||||
requires knowing what is already in use on every machine that will join, which
|
||||
is the operator's knowledge, not ours. IPv6 needs none of this: a ULA derived
|
||||
from the network id collides with essentially nothing.
|
||||
|
||||
**The range is an input to the derivation.** Each agent computes every peer's
|
||||
address itself, so two members configured with different ranges would derive
|
||||
different addresses for each other and IPv4 would silently misroute. The range
|
||||
therefore travels in the announcement — not as a request, and never trusted,
|
||||
but so that a mismatch is *detected*. When it happens, the offending peer gets
|
||||
no IPv4 address here, keeps working over IPv6, and the reason is reported:
|
||||
IPv6 addresses are *derived*: a 64 bit interface identifier makes a collision
|
||||
impossible in practice, so nobody has to agree on anything. IPv4 has nothing
|
||||
like that room, so deriving would collide. Instead an address is **allocated
|
||||
and then recorded as a signed fact**, using the model in
|
||||
[sync-model.md](sync-model.md).
|
||||
|
||||
```text
|
||||
! wireguard: peer SDsEb/WF is configured with the IPv4 overlay range
|
||||
10.81.0.0/16 but this agent uses 10.80.0.0/16; every member must use the
|
||||
same one. That peer has no IPv4 address here and is reachable over IPv6 only.
|
||||
default range 10.13.37.0/24 (override with --ipv4-range)
|
||||
who decides the first member to claim; later ones adopt what they find
|
||||
who signs the claiming member, with its persistent device key
|
||||
where it is kept state.sqlite, and every replica that has seen it
|
||||
what a return costs nothing: the old address is reclaimed
|
||||
```
|
||||
|
||||
**IPv4 addresses can also collide with each other.** A 64 bit interface
|
||||
identifier makes an IPv6 collision impossible in practice; IPv4 has nothing
|
||||
like that room. In a `/16` with 50 members the chance that two members derive
|
||||
the same address is roughly 2%. A mesh with no coordinator cannot allocate
|
||||
around it, so the collision is resolved instead: the member whose WireGuard
|
||||
public key sorts lower keeps the address, a rule every member computes
|
||||
identically and therefore agrees on without exchanging anything. The other
|
||||
member has no IPv4 address and remains reachable over IPv6. Pick a roomy
|
||||
range — a `/16` for a handful of machines, larger for more — and the odds stay
|
||||
small.
|
||||
How it works:
|
||||
|
||||
The honest summary: **IPv6 always works. IPv4 is opt-in, needs agreement, and
|
||||
degrades predictably when it does not get it.** Allocating IPv4 properly needs
|
||||
the agreed state described in [sync-model.md](sync-model.md).
|
||||
1. On joining, an agent reads back the records it already had and learns more
|
||||
from its peers.
|
||||
2. If it already holds an address, it keeps it. **That is the whole point**: a
|
||||
participant that was away for a month comes back to the address it signed
|
||||
for, because the claim outlived the session.
|
||||
3. Otherwise it picks a free one — starting from a position derived from its
|
||||
own identity, so two newcomers rarely start in the same place — signs the
|
||||
claim, commits it together with its version counter, and only then
|
||||
announces it.
|
||||
4. Every replica merges what it receives into what it has. An author missing
|
||||
from a snapshot is left alone: absence is not deletion.
|
||||
|
||||
**No vote is involved, deliberately.** Anyone who knows the network secret can
|
||||
mint identities, so a majority proves nothing, and a quorum would stall with
|
||||
one participant online and diverge across a partition. Two members who claim
|
||||
the same address at the same moment are resolved by a rule both compute
|
||||
identically — the lower endpoint id keeps it — and the loser simply allocates
|
||||
again with a higher version.
|
||||
|
||||
**The range is agreed, not configured per member.** `--ipv4-range` says what
|
||||
this agent would use; a network that has already settled on something else
|
||||
wins, and the agent adopts it. So the flag matters for whoever starts the
|
||||
network and is harmless afterwards. Pass `--ipv4-range none` for an IPv6-only
|
||||
overlay.
|
||||
|
||||
A release tombstone exists in the record type and merges correctly, but
|
||||
nothing emits one yet, so an address stays claimed until the network is
|
||||
forgotten.
|
||||
|
||||
## Address ownership is enforced, not announced
|
||||
|
||||
@@ -246,8 +247,10 @@ async fn main() -> Result<()> {
|
||||
|
||||
* **Full mesh only.** Every member runs a tunnel to every other member.
|
||||
Routing through an intermediate participant is not implemented.
|
||||
* **IPv4 is opt-in, must be agreed, and can collide.** See above. A proper
|
||||
allocator needs agreed state.
|
||||
* **Nothing frees an address yet.** The release record exists and merges, but
|
||||
no command emits one.
|
||||
* **A snapshot grows with the number of members ever seen**, and is capped per
|
||||
message rather than compacted.
|
||||
* **No routes, DNS or firewall rules.** The plugin creates its interface and
|
||||
nothing else. Anything beyond the overlay `/64` is the operator's business.
|
||||
* **Membership is session-scoped.** A peer leaves the overlay when its control
|
||||
|
||||
Reference in New Issue
Block a user