Relay across any mix of links, one online answer for every client, first claim owns a hostname
A tunnel no longer depends on the kind of link under it. The relay hub keeps one forwarding table per network instead of one per protocol, the envelope names the end-to-end protocol, and a neighbour reached over several kinds of transport is sent to over the best open one. A member in the middle needs no protocol in common with either end. Which protocol a pair runs is settled by the lower endpoint id's announced order, so both ends agree without asking. A direct link that comes up later is shorter than the path through others, so traffic moves to it with the same logical link. The agent now answers "is it online" once: a live control session or any data path. Members and peers report how they are reached (direct over a named transport, relayed through a named member, or no data path yet), and `tsng status` and the tray both read that instead of a session list and a WireGuard handshake respectively. A hostname belongs to whoever claimed it first. A claim carries the time its author first claimed the name, signed with the record, and replicas order by (time, id) so every member names the same holder. A later claimant keeps working by address, is not found by the name, and is told so in its status, `tsng status`, the tray and a HostnameTaken event. Wire change: control and data ALPN 5, control PROTOCOL_VERSION 5, envelope version 2, signing domain v3, state schema 7, local control protocol 18. Upgrade every member together. Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
1 parent
c184ef0914
commit
97377e3703
22 files changed
+1699
-292
No files matched your search
+29
-15
@@ -9,7 +9,7 @@ Two versions exist and are independent:
|
||||
|
||||
- **Identity scheme**, `tsunagi-network-id-v1`. Frozen. Changing it creates a
|
||||
different network space for the same name and secret.
|
||||
- **Control protocol**, ALPN `tsunagi/ctrl/4`, `PROTOCOL_VERSION = 4`.
|
||||
- **Control protocol**, ALPN `tsunagi/ctrl/5`, `PROTOCOL_VERSION = 5`.
|
||||
|
||||
Upgrading the crate or bumping the control protocol must never change an
|
||||
existing `NetworkId`.
|
||||
@@ -135,7 +135,7 @@ event does not report a network id.
|
||||
## The data plane protocol
|
||||
|
||||
IP plugin packets never travel on a control connection. They use their own
|
||||
ALPN, `tsunagi/data/4`, on their own iroh connection:
|
||||
ALPN, `tsunagi/data/5`, on their own iroh connection:
|
||||
|
||||
```text
|
||||
initiator -> responder : (the same membership handshake as above)
|
||||
@@ -172,32 +172,46 @@ holding up subsequent packets. Packet IDs are scoped to a QUIC connection;
|
||||
reassembly is created only after the membership handshake. All limits live in
|
||||
`config.rs`. There is no transport-layer retransmission added by this framing.
|
||||
|
||||
Version 4 adds a routing envelope inside the transport fragmentation framing:
|
||||
Version 4 added a routing envelope inside the transport fragmentation framing,
|
||||
and version 5 added the end-to-end protocol to it:
|
||||
|
||||
| field | bytes | encoding |
|
||||
|---|---:|---|
|
||||
| envelope version | 1 | `1` |
|
||||
| envelope version | 1 | `2` |
|
||||
| remaining hops | 1 | initially `16`; transit decrements, drops at `1` |
|
||||
| source peer | 32 | original endpoint public key |
|
||||
| destination peer | 32 | final endpoint public key |
|
||||
| flow id | 8 | big endian opaque stable hash supplied by the plugin |
|
||||
| protocol | 8 | big endian FNV-1a of the end-to-end protocol id (`wg`, `tcp-tls`, ...) |
|
||||
| payload | remaining | end-to-end encrypted plugin bytes |
|
||||
|
||||
The fixed header is 74 bytes. Zero/oversized hop limits, unknown versions,
|
||||
The fixed header is 82 bytes. Zero/oversized hop limits, unknown versions,
|
||||
short frames, oversized datagrams and unknown sources are rejected. A frame
|
||||
addressed here is delivered to its source's protocol inbox; transit goes straight
|
||||
to its next transport. The network and protocol are bound to the authenticated
|
||||
transport connection, not supplied by the frame. Intermediate nodes cannot
|
||||
decrypt or authenticate the inner WireGuard payload; the destination does that.
|
||||
Flow ids are routing hints, not authorization proofs.
|
||||
addressed here is delivered to the inbox of its source and protocol; transit
|
||||
goes straight to its next transport, whatever kind of transport that is. The
|
||||
network is bound to the authenticated transport connection, not supplied by
|
||||
the frame. The protocol names which end-to-end tunnel the payload belongs to
|
||||
and is independent of the links it crosses: a datagram of one protocol can
|
||||
travel over a link opened for another, and a middle member needs no protocol
|
||||
in common with either end. Intermediate nodes cannot decrypt or authenticate
|
||||
the inner payload; the destination's tunnel does that, so a frame naming the
|
||||
wrong protocol or a forged source is simply discarded there. Flow ids are
|
||||
routing hints, not authorization proofs.
|
||||
|
||||
Control ALPN 3 carries `Reachable { links: [{ peer, protocol }] }`. Each row
|
||||
belongs to the authenticated sender and is replaced atomically, expires after
|
||||
90 seconds, and is withdrawn on session closure. Only compatible authenticated
|
||||
members enter a protocol's graph; local edges always come from actual links.
|
||||
`Reachable { links: [{ peer, protocol }] }` lists the sender's direct links;
|
||||
`protocol` is the kind of transport the link is, for diagnostics, and routing
|
||||
ignores it. Each row belongs to the authenticated sender and is replaced
|
||||
atomically, expires after 90 seconds, and is withdrawn on session closure. Only
|
||||
authenticated members enter the graph; local edges always come from actual
|
||||
links.
|
||||
Announcements refresh on the existing control maintenance interval and are
|
||||
sent immediately on link changes. Topology stays volatile, outside signed state.
|
||||
|
||||
Version 5 also adds `hostname_since` to a signed hostname claim (signing domain
|
||||
`tsunagi-signed-record-v3`): a name belongs to whoever claimed it first. Stored
|
||||
records signed before are discarded on upgrade and re-published by their
|
||||
authors.
|
||||
|
||||
Upgrade both endpoints and intermediate peers together. Older control and data
|
||||
ALPNs cannot interoperate. The network secret, device identity, identity
|
||||
derivation, transcript encoding and stored network configuration are unchanged.
|
||||
@@ -220,7 +234,7 @@ not affect other networks.
|
||||
| `Pong { seq, payload }` | the echoed reply |
|
||||
| `State { records }` | a snapshot of signed records, merged into what the receiver holds |
|
||||
| `Peers { peers }` | unverified member address hints, requiring their own handshake |
|
||||
| `Reachable { links }` | sender's current direct data links, scoped by protocol |
|
||||
| `Reachable { links }` | sender's current direct data links, by kind of transport |
|
||||
| `Bye { reason }` | graceful goodbye; not a revocation of anything |
|
||||
|
||||
`broadcast` is a per-network local opt-in, enabled by default and treated as
|
||||
|
||||
+39
-9
@@ -15,12 +15,27 @@ the default 1280-byte interface MTU also works on small QUIC paths.
|
||||
## Graph and lifecycle
|
||||
|
||||
Each authenticated control session advertises that peer's own direct data
|
||||
links, including the plugin protocol. Tables are separate for each network
|
||||
and protocol; only members offering a matching enabled protocol version enter
|
||||
the graph. A snapshot refreshes on the maintenance interval, expires after
|
||||
90 seconds, and is withdrawn when the session ends. Link arrival, closure and
|
||||
changed announcements rebuild routes immediately. Unchanged announcements
|
||||
refresh their age without republishing the forwarding snapshot.
|
||||
links. There is one table for each network, whatever protocols run over it: a
|
||||
link is a way to reach a neighbour, and a protocol is an end-to-end tunnel to a
|
||||
peer. The envelope names the protocol, so the two are independent. A neighbour
|
||||
reached over several kinds of transport (UDP, TCP, QUIC datagrams) is sent to
|
||||
over the best open one, in the order the protocols were configured, and over
|
||||
the next if that one fails or cannot carry the frame. Every authenticated
|
||||
member enters the graph; a member in the middle needs no protocol in common
|
||||
with either end, only a link to each. A snapshot refreshes on the maintenance
|
||||
interval, expires after 90 seconds, and is withdrawn when the session ends.
|
||||
Link arrival, closure and changed announcements rebuild routes immediately.
|
||||
Unchanged announcements refresh their age without republishing the forwarding
|
||||
snapshot.
|
||||
|
||||
Which protocol a pair of members runs their tunnel in is settled without
|
||||
asking: the member with the lower endpoint id decides, by the order it
|
||||
announces its protocols in, among those both have enabled at the same wire
|
||||
version. Both ends read the same answer from what they already hold. The
|
||||
tunnel is created as soon as the peer is reachable by any path, and it does not
|
||||
change when a direct link comes or goes underneath it: as soon as a direct
|
||||
link comes up it is shorter than the path through others, and traffic moves to
|
||||
it with nothing to redo. The user sees only a lower round-trip time.
|
||||
|
||||
Breadth-first search computes the shortest directed paths and every equal-cost
|
||||
first hop in sorted order. An actual direct link always wins. The first row is
|
||||
@@ -30,14 +45,29 @@ loops if different nodes have not yet received the same topology update.
|
||||
|
||||
The existing control plane still forms authenticated pairwise sessions.
|
||||
Routing provides multihop **data** paths among these members; it does not add
|
||||
control-plane flooding or carry control sessions through the data plane.
|
||||
control-plane flooding or carry control sessions through the data plane, so a
|
||||
member that has no control session with another has no data path to it either.
|
||||
Announcements are volatile, not durable membership or availability guarantees.
|
||||
|
||||
## What "online" means
|
||||
|
||||
One definition, computed by the agent and reported identically by `tsng status`
|
||||
and the tray: a member is online when this agent can reach it by any path — a
|
||||
live authenticated control session (QUIC keep-alive every 5 seconds, so a
|
||||
vanished peer is noticed within about 15), or a data path to it through other
|
||||
members. It is never inferred from ping, which a firewall may block, or from a
|
||||
tunnel's last handshake, which says only when traffic last flowed.
|
||||
|
||||
How it is reached is reported separately: `direct` over a named transport,
|
||||
`relayed` through a named member over N hops, or no data path yet while one is
|
||||
being set up. Being relayed is not being degraded in any way but distance, and
|
||||
is shown so it is not a surprise.
|
||||
|
||||
## Packet path
|
||||
|
||||
The control loop binds next hops to transport handles in an immutable table
|
||||
and publishes it through `ArcSwap`. Readers do not acquire the topology mutex.
|
||||
The transit routine validates the fixed 74-byte envelope, looks up the source
|
||||
The transit routine validates the fixed 82-byte envelope, looks up the source
|
||||
and destination, selects a cached next hop, decrements one byte, and calls the
|
||||
transport's synchronous datagram send. It never searches the graph, formats an
|
||||
endpoint string, validates an Ed25519 key, decrypts, or touches TUN.
|
||||
@@ -86,5 +116,5 @@ counts sends without storing frames. Results are CPU forwarding cost, **not**
|
||||
end-to-end network latency or VPN throughput; encryption, fragmentation,
|
||||
sockets, congestion and scheduling contribute separately.
|
||||
|
||||
Wire compatibility: control ALPN `tsunagi/ctrl/4`, data ALPN `tsunagi/data/4`.
|
||||
Wire compatibility: control ALPN `tsunagi/ctrl/5`, data ALPN `tsunagi/data/5`.
|
||||
Upgrade every participant together; saved identities and network state persist.
|
||||
+19
-1
@@ -43,7 +43,9 @@ document trusted merely because the neighbour who forwarded it signed it.
|
||||
- Duplicates do not change the result.
|
||||
- Two conflicting signed records at the same version from the same author need
|
||||
explicit handling; they are not resolved by luck.
|
||||
- Neither arrival order nor system clocks decide a winner.
|
||||
- Neither arrival order nor the receiving replica's clock decides a winner. The
|
||||
one thing an author's own clock is used for is the time it first claimed a
|
||||
hostname, signed into its record (see below).
|
||||
- Compaction must not drop what is needed to stop revoked records being
|
||||
resurrected.
|
||||
|
||||
@@ -53,6 +55,22 @@ A hostname is a mutable binding to a persistent author, not an identity. A
|
||||
rename must be a signed record that revokes the specific old binding and
|
||||
announces the new one, ideally atomically in one record.
|
||||
|
||||
A hostname belongs to the member that claimed it **first**. Each claim carries
|
||||
`hostname_since`, the author's own time in milliseconds when it first claimed
|
||||
that exact name, signed with the rest of the record and kept across every later
|
||||
version that keeps the name; renaming starts it afresh. Replicas order claims
|
||||
by `(hostname_since, author id)`, a pure function of the records, so every
|
||||
replica names the same holder whatever order the records arrived in and there
|
||||
is no race to lose. A member that arrives later with a name already held keeps
|
||||
working by address and keeps its claim, but is not found by that name on any
|
||||
member; it is told so in its status and in `tsng status`, and finds the name
|
||||
again if the holder renames or releases it.
|
||||
|
||||
It is the author's word, so a member with a clock set far back can claim an
|
||||
older time. That can reorder who holds a *name* and nothing else — addresses,
|
||||
keys and everything the data plane trusts are decided without it — and anybody
|
||||
who knows the secret could take a name by other means anyway.
|
||||
|
||||
Turning a computer off is not a revocation of its hostname and does not remove
|
||||
the participant. Revocations are not dropped merely because they are old, and
|
||||
no acknowledgement from offline peers is required to keep working.
|
||||
|
||||
+1
-1
@@ -184,7 +184,7 @@ QUIC DATAGRAM frames themselves cannot fragment.
|
||||
Reassembly is bounded and expires incomplete packets; one lost fragment loses
|
||||
one packet, without blocking unrelated traffic. The logical data payload limit
|
||||
is 64 KiB, with the relay envelope subtracted before it reaches the plugin.
|
||||
The new framing uses data ALPN `tsunagi/data/4`; both ends and intermediate
|
||||
The new framing uses data ALPN `tsunagi/data/5`; both ends and intermediate
|
||||
peers need the updated binary. Network identities and saved state do not change.
|
||||
See [protocol.md](protocol.md#the-data-plane-protocol) for the wire format.
|
||||
|
||||
|
||||
Reference in new issue
Block a user