Aleksandr BogomiakovandClaude
97377e3703
A tunnel no longer depends on the kind of link under it. The relay hub keeps one forwarding table per network instead of one per protocol, the envelope names the end-to-end protocol, and a neighbour reached over several kinds of transport is sent to over the best open one. A member in the middle needs no protocol in common with either end. Which protocol a pair runs is settled by the lower endpoint id's announced order, so both ends agree without asking. A direct link that comes up later is shorter than the path through others, so traffic moves to it with the same logical link. The agent now answers "is it online" once: a live control session or any data path. Members and peers report how they are reached (direct over a named transport, relayed through a named member, or no data path yet), and `tsng status` and the tray both read that instead of a session list and a WireGuard handshake respectively. A hostname belongs to whoever claimed it first. A claim carries the time its author first claimed the name, signed with the record, and replicas order by (time, id) so every member names the same holder. A later claimant keeps working by address, is not found by the name, and is told so in its status, `tsng status`, the tray and a HostnameTaken event. Wire change: control and data ALPN 5, control PROTOCOL_VERSION 5, envelope version 2, signing domain v3, state schema 7, local control protocol 18. Upgrade every member together. Co-Authored-By: Claude <noreply@anthropic.com>
135 lines
6.4 KiB
Markdown
135 lines
6.4 KiB
Markdown
# Planned state synchronisation
|
|
|
|
**The first slice of this model is now implemented**, in `crates/tsunagi-core/src/state/`, and is
|
|
used for one thing: IPv4 overlay addresses. What follows describes the whole
|
|
model; the section at the end says exactly which parts exist.
|
|
|
|
The proof of concept still exchanges hostname and capability announcements
|
|
over live sessions and keeps no replicated history for those, which is why
|
|
WireGuard peer *membership* remains session-scoped even though a peer's
|
|
*address* no longer is. This file records the intended direction so the module
|
|
boundaries in [architecture.md](architecture.md) stay compatible with it, and so
|
|
nobody mistakes the current announcements for synchronisation.
|
|
|
|
There is no fake "ready CRDT" here and no snapshots that are not actually
|
|
verified.
|
|
|
|
## The problem
|
|
|
|
The network may be unstable. A participant can come back after months. So:
|
|
|
|
- no dependence on the author of a change being online;
|
|
- no dependence on acknowledgements from every participant ever seen;
|
|
- any available replica holding the signed data must be able to hand it to a
|
|
returning participant without the original author present.
|
|
|
|
## The model
|
|
|
|
**Signed self-contained state per author, merged between replicas.**
|
|
|
|
A record contains: the network, the author, the author's own retained version,
|
|
the full current content, and a signature. A change log may speed delivery up,
|
|
but recovery must never require the entire chain from the first event.
|
|
|
|
A network snapshot is a set of verifiable authored records plus the revocations
|
|
needed to interpret them. It is **not** a SQLite dump, and **not** a single
|
|
document trusted merely because the neighbour who forwarded it signed it.
|
|
|
|
## Merge rules
|
|
|
|
- A snapshot is merged into local state, never substituted for it wholesale.
|
|
- An older version never rolls back a newer known one.
|
|
- Absence from a snapshot does not mean deletion.
|
|
- Duplicates do not change the result.
|
|
- Two conflicting signed records at the same version from the same author need
|
|
explicit handling; they are not resolved by luck.
|
|
- Neither arrival order nor the receiving replica's clock decides a winner. The
|
|
one thing an author's own clock is used for is the time it first claimed a
|
|
hostname, signed into its record (see below).
|
|
- Compaction must not drop what is needed to stop revoked records being
|
|
resurrected.
|
|
|
|
## Hostnames
|
|
|
|
A hostname is a mutable binding to a persistent author, not an identity. A
|
|
rename must be a signed record that revokes the specific old binding and
|
|
announces the new one, ideally atomically in one record.
|
|
|
|
A hostname belongs to the member that claimed it **first**. Each claim carries
|
|
`hostname_since`, the author's own time in milliseconds when it first claimed
|
|
that exact name, signed with the rest of the record and kept across every later
|
|
version that keeps the name; renaming starts it afresh. Replicas order claims
|
|
by `(hostname_since, author id)`, a pure function of the records, so every
|
|
replica names the same holder whatever order the records arrived in and there
|
|
is no race to lose. A member that arrives later with a name already held keeps
|
|
working by address and keeps its claim, but is not found by that name on any
|
|
member; it is told so in its status and in `tsng status`, and finds the name
|
|
again if the holder renames or releases it.
|
|
|
|
It is the author's word, so a member with a clock set far back can claim an
|
|
older time. That can reorder who holds a *name* and nothing else — addresses,
|
|
keys and everything the data plane trusts are decided without it — and anybody
|
|
who knows the secret could take a name by other means anyway.
|
|
|
|
Turning a computer off is not a revocation of its hostname and does not remove
|
|
the participant. Revocations are not dropped merely because they are old, and
|
|
no acknowledgement from offline peers is required to keep working.
|
|
|
|
## Storage requirement this creates
|
|
|
|
When signed records land, writing the event and bumping the author's own
|
|
counter must happen in **one SQLite transaction, committed before the change is
|
|
published to the network**. SQLite gives atomic commit; use it instead of
|
|
separate, inconsistent writes. `state.sqlite` already has a schema version and
|
|
migrations for this.
|
|
|
|
## Limits to state honestly
|
|
|
|
- Data that every copy has lost is not recoverable from the secret.
|
|
- A signature proves authorship, not global freshness: a replica can be
|
|
behind, and you cannot tell from the signature alone.
|
|
- An isolated new client can end up with incomplete state and has no way to
|
|
know what it is missing.
|
|
- Anyone who knows the secret can author records, so a majority of records is
|
|
not evidence of anything.
|
|
|
|
## What exists today
|
|
|
|
Implemented, in `crates/tsunagi-core/src/state/`:
|
|
|
|
* signed records, one per author per network, each holding that author's
|
|
complete current statement rather than a delta;
|
|
* signing and verification with the persistent iroh device key, over a
|
|
length-prefixed canonical encoding;
|
|
* the merge rules above: higher version wins, an older version never rolls
|
|
back a newer one, duplicates are idempotent, a same-version conflict is
|
|
resolved identically on every replica and reported;
|
|
* a release tombstone, which merges correctly and is not undone by a replica
|
|
that has not heard of it — though nothing emits one yet, so freeing an
|
|
address still means forgetting the network;
|
|
* persistence in `state.sqlite`, with the record and the author's own version
|
|
counter committed in **one transaction before the record is announced**;
|
|
* distribution as a `State` control message, merged into what the receiver
|
|
already holds rather than replacing it;
|
|
* allocation of a free IPv4 address against what everybody else holds, which
|
|
is what makes an address stable across an absence.
|
|
|
|
Deliberately not implemented: compaction, revoking a whole author, record
|
|
types beyond addressing, and any bound on how large a snapshot may grow
|
|
beyond the per-message limit.
|
|
|
|
## Future tests
|
|
|
|
These are **not implemented and must not be reported as passing**:
|
|
|
|
- snapshot merge against the rules above, including conflicting same-version
|
|
records;
|
|
- revocation propagation and resistance to resurrection after compaction;
|
|
- long network partitions and rejoin after an extended absence;
|
|
- hostname rename with atomic revoke-and-announce;
|
|
- recovery of a returning participant from a replica that is not the author;
|
|
- NAT traversal and hole punching between real hosts;
|
|
- relay fallback behaviour against a self-hosted relay;
|
|
- multi-process and multi-host deployment, as opposed to several library
|
|
instances inside one test process.
|