Relay across any mix of links, one online answer for every client, first claim owns a hostname

A tunnel no longer depends on the kind of link under it. The relay hub keeps
one forwarding table per network instead of one per protocol, the envelope
names the end-to-end protocol, and a neighbour reached over several kinds of
transport is sent to over the best open one. A member in the middle needs no
protocol in common with either end. Which protocol a pair runs is settled by
the lower endpoint id's announced order, so both ends agree without asking.
A direct link that comes up later is shorter than the path through others, so
traffic moves to it with the same logical link.

The agent now answers "is it online" once: a live control session or any data
path. Members and peers report how they are reached (direct over a named
transport, relayed through a named member, or no data path yet), and `tsng
status` and the tray both read that instead of a session list and a WireGuard
handshake respectively.

A hostname belongs to whoever claimed it first. A claim carries the time its
author first claimed the name, signed with the record, and replicas order by
(time, id) so every member names the same holder. A later claimant keeps
working by address, is not found by the name, and is told so in its status,
`tsng status`, the tray and a HostnameTaken event.

Wire change: control and data ALPN 5, control PROTOCOL_VERSION 5, envelope
version 2, signing domain v3, state schema 7, local control protocol 18.
Upgrade every member together.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Aleksandr BogomiakovandClaude committed 2026-10-05 11:39:33 +01:00
1 parent c184ef0914
commit 97377e3703
22 files changed
+1699 -292

No files matched your search

+29 -15
View File
@@ -9,7 +9,7 @@ Two versions exist and are independent:
- **Identity scheme**, `tsunagi-network-id-v1`. Frozen. Changing it creates a
different network space for the same name and secret.
- **Control protocol**, ALPN `tsunagi/ctrl/4`, `PROTOCOL_VERSION = 4`.
- **Control protocol**, ALPN `tsunagi/ctrl/5`, `PROTOCOL_VERSION = 5`.
Upgrading the crate or bumping the control protocol must never change an
existing `NetworkId`.
@@ -135,7 +135,7 @@ event does not report a network id.
## The data plane protocol
IP plugin packets never travel on a control connection. They use their own
ALPN, `tsunagi/data/4`, on their own iroh connection:
ALPN, `tsunagi/data/5`, on their own iroh connection:
```text
initiator -> responder : (the same membership handshake as above)
@@ -172,32 +172,46 @@ holding up subsequent packets. Packet IDs are scoped to a QUIC connection;
reassembly is created only after the membership handshake. All limits live in
`config.rs`. There is no transport-layer retransmission added by this framing.
Version 4 adds a routing envelope inside the transport fragmentation framing:
Version 4 added a routing envelope inside the transport fragmentation framing,
and version 5 added the end-to-end protocol to it:
| field | bytes | encoding |
|---|---:|---|
| envelope version | 1 | `1` |
| envelope version | 1 | `2` |
| remaining hops | 1 | initially `16`; transit decrements, drops at `1` |
| source peer | 32 | original endpoint public key |
| destination peer | 32 | final endpoint public key |
| flow id | 8 | big endian opaque stable hash supplied by the plugin |
| protocol | 8 | big endian FNV-1a of the end-to-end protocol id (`wg`, `tcp-tls`, ...) |
| payload | remaining | end-to-end encrypted plugin bytes |
The fixed header is 74 bytes. Zero/oversized hop limits, unknown versions,
The fixed header is 82 bytes. Zero/oversized hop limits, unknown versions,
short frames, oversized datagrams and unknown sources are rejected. A frame
addressed here is delivered to its source's protocol inbox; transit goes straight
to its next transport. The network and protocol are bound to the authenticated
transport connection, not supplied by the frame. Intermediate nodes cannot
decrypt or authenticate the inner WireGuard payload; the destination does that.
Flow ids are routing hints, not authorization proofs.
addressed here is delivered to the inbox of its source and protocol; transit
goes straight to its next transport, whatever kind of transport that is. The
network is bound to the authenticated transport connection, not supplied by
the frame. The protocol names which end-to-end tunnel the payload belongs to
and is independent of the links it crosses: a datagram of one protocol can
travel over a link opened for another, and a middle member needs no protocol
in common with either end. Intermediate nodes cannot decrypt or authenticate
the inner payload; the destination's tunnel does that, so a frame naming the
wrong protocol or a forged source is simply discarded there. Flow ids are
routing hints, not authorization proofs.
Control ALPN 3 carries `Reachable { links: [{ peer, protocol }] }`. Each row
belongs to the authenticated sender and is replaced atomically, expires after
90 seconds, and is withdrawn on session closure. Only compatible authenticated
members enter a protocol's graph; local edges always come from actual links.
`Reachable { links: [{ peer, protocol }] }` lists the sender's direct links;
`protocol` is the kind of transport the link is, for diagnostics, and routing
ignores it. Each row belongs to the authenticated sender and is replaced
atomically, expires after 90 seconds, and is withdrawn on session closure. Only
authenticated members enter the graph; local edges always come from actual
links.
Announcements refresh on the existing control maintenance interval and are
sent immediately on link changes. Topology stays volatile, outside signed state.
Version 5 also adds `hostname_since` to a signed hostname claim (signing domain
`tsunagi-signed-record-v3`): a name belongs to whoever claimed it first. Stored
records signed before are discarded on upgrade and re-published by their
authors.
Upgrade both endpoints and intermediate peers together. Older control and data
ALPNs cannot interoperate. The network secret, device identity, identity
derivation, transcript encoding and stored network configuration are unchanged.
@@ -220,7 +234,7 @@ not affect other networks.
| `Pong { seq, payload }` | the echoed reply |
| `State { records }` | a snapshot of signed records, merged into what the receiver holds |
| `Peers { peers }` | unverified member address hints, requiring their own handshake |
| `Reachable { links }` | sender's current direct data links, scoped by protocol |
| `Reachable { links }` | sender's current direct data links, by kind of transport |
| `Bye { reason }` | graceful goodbye; not a revocation of anything |
`broadcast` is a per-network local opt-in, enabled by default and treated as
+39 -9
View File
@@ -15,12 +15,27 @@ the default 1280-byte interface MTU also works on small QUIC paths.
## Graph and lifecycle
Each authenticated control session advertises that peer's own direct data
links, including the plugin protocol. Tables are separate for each network
and protocol; only members offering a matching enabled protocol version enter
the graph. A snapshot refreshes on the maintenance interval, expires after
90 seconds, and is withdrawn when the session ends. Link arrival, closure and
changed announcements rebuild routes immediately. Unchanged announcements
refresh their age without republishing the forwarding snapshot.
links. There is one table for each network, whatever protocols run over it: a
link is a way to reach a neighbour, and a protocol is an end-to-end tunnel to a
peer. The envelope names the protocol, so the two are independent. A neighbour
reached over several kinds of transport (UDP, TCP, QUIC datagrams) is sent to
over the best open one, in the order the protocols were configured, and over
the next if that one fails or cannot carry the frame. Every authenticated
member enters the graph; a member in the middle needs no protocol in common
with either end, only a link to each. A snapshot refreshes on the maintenance
interval, expires after 90 seconds, and is withdrawn when the session ends.
Link arrival, closure and changed announcements rebuild routes immediately.
Unchanged announcements refresh their age without republishing the forwarding
snapshot.
Which protocol a pair of members runs their tunnel in is settled without
asking: the member with the lower endpoint id decides, by the order it
announces its protocols in, among those both have enabled at the same wire
version. Both ends read the same answer from what they already hold. The
tunnel is created as soon as the peer is reachable by any path, and it does not
change when a direct link comes or goes underneath it: as soon as a direct
link comes up it is shorter than the path through others, and traffic moves to
it with nothing to redo. The user sees only a lower round-trip time.
Breadth-first search computes the shortest directed paths and every equal-cost
first hop in sorted order. An actual direct link always wins. The first row is
@@ -30,14 +45,29 @@ loops if different nodes have not yet received the same topology update.
The existing control plane still forms authenticated pairwise sessions.
Routing provides multihop **data** paths among these members; it does not add
control-plane flooding or carry control sessions through the data plane.
control-plane flooding or carry control sessions through the data plane, so a
member that has no control session with another has no data path to it either.
Announcements are volatile, not durable membership or availability guarantees.
## What "online" means
One definition, computed by the agent and reported identically by `tsng status`
and the tray: a member is online when this agent can reach it by any path — a
live authenticated control session (QUIC keep-alive every 5 seconds, so a
vanished peer is noticed within about 15), or a data path to it through other
members. It is never inferred from ping, which a firewall may block, or from a
tunnel's last handshake, which says only when traffic last flowed.
How it is reached is reported separately: `direct` over a named transport,
`relayed` through a named member over N hops, or no data path yet while one is
being set up. Being relayed is not being degraded in any way but distance, and
is shown so it is not a surprise.
## Packet path
The control loop binds next hops to transport handles in an immutable table
and publishes it through `ArcSwap`. Readers do not acquire the topology mutex.
The transit routine validates the fixed 74-byte envelope, looks up the source
The transit routine validates the fixed 82-byte envelope, looks up the source
and destination, selects a cached next hop, decrements one byte, and calls the
transport's synchronous datagram send. It never searches the graph, formats an
endpoint string, validates an Ed25519 key, decrypts, or touches TUN.
@@ -86,5 +116,5 @@ counts sends without storing frames. Results are CPU forwarding cost, **not**
end-to-end network latency or VPN throughput; encryption, fragmentation,
sockets, congestion and scheduling contribute separately.
Wire compatibility: control ALPN `tsunagi/ctrl/4`, data ALPN `tsunagi/data/4`.
Wire compatibility: control ALPN `tsunagi/ctrl/5`, data ALPN `tsunagi/data/5`.
Upgrade every participant together; saved identities and network state persist.
+19 -1
View File
@@ -43,7 +43,9 @@ document trusted merely because the neighbour who forwarded it signed it.
- Duplicates do not change the result.
- Two conflicting signed records at the same version from the same author need
explicit handling; they are not resolved by luck.
- Neither arrival order nor system clocks decide a winner.
- Neither arrival order nor the receiving replica's clock decides a winner. The
one thing an author's own clock is used for is the time it first claimed a
hostname, signed into its record (see below).
- Compaction must not drop what is needed to stop revoked records being
resurrected.
@@ -53,6 +55,22 @@ A hostname is a mutable binding to a persistent author, not an identity. A
rename must be a signed record that revokes the specific old binding and
announces the new one, ideally atomically in one record.
A hostname belongs to the member that claimed it **first**. Each claim carries
`hostname_since`, the author's own time in milliseconds when it first claimed
that exact name, signed with the rest of the record and kept across every later
version that keeps the name; renaming starts it afresh. Replicas order claims
by `(hostname_since, author id)`, a pure function of the records, so every
replica names the same holder whatever order the records arrived in and there
is no race to lose. A member that arrives later with a name already held keeps
working by address and keeps its claim, but is not found by that name on any
member; it is told so in its status and in `tsng status`, and finds the name
again if the holder renames or releases it.
It is the author's word, so a member with a clock set far back can claim an
older time. That can reorder who holds a *name* and nothing else — addresses,
keys and everything the data plane trusts are decided without it — and anybody
who knows the secret could take a name by other means anyway.
Turning a computer off is not a revocation of its hostname and does not remove
the participant. Revocations are not dropped merely because they are old, and
no acknowledgement from offline peers is required to keep working.
+1 -1
View File
@@ -184,7 +184,7 @@ QUIC DATAGRAM frames themselves cannot fragment.
Reassembly is bounded and expires incomplete packets; one lost fragment loses
one packet, without blocking unrelated traffic. The logical data payload limit
is 64 KiB, with the relay envelope subtracted before it reaches the plugin.
The new framing uses data ALPN `tsunagi/data/4`; both ends and intermediate
The new framing uses data ALPN `tsunagi/data/5`; both ends and intermediate
peers need the updated binary. Network identities and saved state do not change.
See [protocol.md](protocol.md#the-data-plane-protocol) for the wire format.