`status` asked a running agent and failed without one; `doctor` checked
the host and ignored the agent. Between them they answered one question
in two halves. They are now one command: device, agent, networks, host
capability, local addresses, all graded and aligned the same way.
`id` was worse than either. It spawned a whole agent to print three
facts, which took the directory lock and so failed with "owned by
another running agent instance" exactly when the answer was most wanted.
The lock exists to keep one writer over the mandatory state; reading who
this device is needs no such thing.
So both commands now prefer the running agent, which is live and
authoritative, and fall back to the state store, which takes no lock.
StateStore gains device_identity(), which reads and never writes:
load_or_create had the side effect of deciding an identity as a
consequence of asking about one.
Presentation moved out of the library. StatusReport::render is gone and
the CLI renders the structured data, so there is one renderer rather than
two that would drift. Health gains an Info level for rows that are facts
rather than checks — an endpoint id is neither good nor bad, and a column
of green next to plain data teaches the eye to ignore the column. A
report with no checks in it now ends without a summary instead of
claiming that everything checked out.
PathAddr and TransportKind grew Display impls; both were reaching the
user through {:?}, which is how "Direct via Ip(88.198.17.44:49792)" got
printed. The transport grading compares without regard to case, because
the agent answering can be a different build from the client asking and
this field's spelling has now changed once.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Findings now carry a level: ok, warn or FAIL. The distinction between the
middle two is the part worth getting right — warn is something the agent
runs without and that the user can fix from the line printed beneath it,
FAIL is something it cannot work around. A diagnostic that grades those
the wrong way round is worse than an ungraded one, so each check states
which it is.
The clearest case is storage: the same failure on the state directory is
FAIL and on the cache directory is warn, because one is mandatory and the
other is disposable. That asymmetry is central to the design and the
report now shows it. The control plane check also became real — it binds
a UDP socket rather than asserting that it could.
Colour is redundant by construction. Every row carries its grade as a
word, so the report reads identically when the escapes are gone: piped to
a file, on a dumb terminal, under NO_COLOR, or to someone who cannot
distinguish the colours. anstream decides whether they survive, which
also gets virtual terminal processing right on Windows; it and anstyle
were already in the tree through clap.
What is reported and how it looks are separated, so the rendering is
tested without a terminal: that a plain render contains no escapes, that
a styled one says the same thing once they are stripped, that columns
line up across sections whose labels differ in length, and that the
summary names the worst thing found.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The managed interface supersedes both. They go together because apart
they are useless: attaching needs an interface somebody prepared, and
tun-setup existed only to say how to prepare one.
This also corrects what the last commit's README claimed. It said the
manual route was needed on macOS and Windows; it was not, and could not
be. The recipe printed Linux `ip` commands, and a persistent TUN that a
second process can attach to is a Linux concept — macOS creates a utun
by opening a control socket and there is nothing to hand over. So those
platforms were never served by this path, and their honest state is that
a real interface waits on a provisioner, with --no-tun meanwhile.
Gone with it: the interface-existence check, the /proc/net/if_inet6
address inspection and its DAD flag decoding, and the --interface flag,
which had one mode left.
Kept: the check that the allocated IPv4 address is really on a local
interface. The agent now assigns that address itself, so the check is no
longer telling a user what to run — it verifies the outcome instead of
trusting it, which is worth keeping precisely because the assumptions
around Linux address behaviour have been wrong here more than once. Its
message says which interface should have had the address rather than a
command to run.
Boxing Up(UpArgs) is fallout: TunSetupArgs had been masking how much
larger that variant is than its siblings.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The agent printed a list of `ip` commands and asked a human to run them.
That is fragile in the way hand-held setup always is: a persistent TUN
does not survive a reboot, a changed address allocation needs another
manual round, and a run that died leaves a half-configured interface the
next run trips over.
On Linux the agent now creates the interface, sets the MTU, brings it up
and assigns both overlay addresses itself, over netlink in process. No
`ip` is invoked, so nothing this path does can be influenced by PATH, a
shell, or anything a remote peer said.
Cleanup stops being an action. The interface is tied to an open file
descriptor and is deliberately not persistent, so the kernel removes it
when the agent goes — cleanly, by panic, by SIGKILL or by power loss
alike. That also retires `keep_addr_on_down` and `nodad`, which existed
only because an interface nobody held open lost carrier.
Anything still left behind is repaired rather than tripped over: an
abandoned TUN is replaced along with its stale addresses. Two cases
refuse instead of guessing — a link that is not a TUN, because a name
collision is no reason to destroy somebody's bridge, and a TUN another
process holds open, because that is a working overlay belonging to
someone else.
CAP_NET_ADMIN is kept out of the effective set except around the calls
that use it. Two facts shape how: capabilities are per thread, and
netlink checks the credentials of whichever thread calls sendmsg, which
with an async client is the connection task rather than the caller. So
netlink runs on one dedicated thread with a current-thread runtime where
nothing is polled outside a block_on, and opening the TUN descriptor is
synchronous with no await between the guard and its release.
The decision of what to change is a pure function, tested on every
platform; only the execution is behind the provisioner trait. macOS and
Windows get an implementation that refuses with an explanation and falls
back to attaching to a prepared interface, plus a mock host the tests
drive the whole plugin lifecycle against.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The overlay IPv4 address is not derived from the keys: it is allocated at
run time and signed, so on a fresh state directory there is nothing for
tun-setup to print. Once the agent has run, the claim is in state.sqlite,
and reading it back takes no directory lock, so tun-setup can show the
`ip address add` line while the agent is running. Records are verified on
the way out; the database is not a trust boundary.
The line needs no keep_addr_on_down and no nodad, unlike its IPv6
counterpart: Linux keeps IPv4 addresses on an interface that has lost
carrier, and IPv4 has no duplicate address detection to stall.
Also fix a race in the four-agent test. A peer counts as connected once
its session authenticates, which can precede the announcement carrying
its hostname, so reading the hostnames straight away occasionally saw
only two. It now waits for them like every other success condition.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
IPv6 works end to end between two machines; IPv4 silently did not, and the
agent said nothing useful about why.
Allocation moved the address from something derivable before startup to
something agreed at run time, so an interface configured by an earlier
`tun-setup` carries a different address than the one allocated. The kernel
then sends packets with that stale source and every peer drops them as not
belonging to us — correct behaviour, invisible cause. Meanwhile pings to
our own allocated address fall into the tunnel and land in the "nobody
owns this" counter.
The agent now checks whether its allocated address is assigned anywhere on
the host — by binding a UDP socket to it, which needs no privileges and no
platform code — and reports the exact `ip address add` command until it
is, mentioning that another address of the range has to go.
`tun-setup` no longer prints a derived IPv4 address, because that number
is now wrong by construction. It says the agent will print the real one.
The unroutable counter keeps one destination as a sample, in status output
too. A bare count says something is wrong; the address says what.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Derived IPv4 addresses could not survive anything: they changed with the
range, and there was no way for a member to come back to the one it had.
Addresses are now allocated and recorded as signed facts, which is the
first slice of the model in docs/sync-model.md.
src/state/ holds one record per author per network, carrying that author's
complete current statement, signed with its persistent device key over a
length-prefixed canonical encoding. Merging follows the model's rules: a
higher version wins, an older one never rolls back a newer, duplicates are
idempotent, absence from a snapshot is not deletion, and a same-version
conflict is resolved identically on every replica and reported rather than
letting replicas diverge. Records are persisted in state.sqlite, with the
record and the author's version counter committed in one transaction
before anything is announced, and distributed as a State control message
that is merged into what the receiver already holds.
No vote, deliberately, despite the request. A majority is not a trust root
here — anyone with the secret can mint identities — and a quorum would
stall with one peer online and diverge across a partition. Signatures plus
a deterministic merge converge without either failure mode: two members
claiming one address at once are resolved by the lower endpoint id, and
the loser allocates again with a higher version.
The range moved from the plugin to the agent, defaults to 10.13.37.0/24,
and is now agreed rather than configured per member: a joining agent
adopts what the network already uses, so --ipv4-range only matters for
whoever starts it. The announcement went back to identity only (version 3)
since the range travels in signed records now.
A release tombstone exists and merges correctly, but nothing emits one
yet.
116 tests. The headline ones: an address survives restarting both agents,
three members get three distinct addresses, and a member started with a
different range adopts the one in use. Confirmed by hand with two CLI
agents restarted end to end.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
100.64.0.0/10 was a bad default: it is exactly Tailscale's range, and
carrier-grade NAT's. There is no IPv4 range that is free on every host, so
there is now no default at all — IPv4 is off until --ipv4-range names one.
IPv6 is unaffected and still works out of the box, because a ULA derived
from the network id collides with essentially nothing.
The more serious problem this exposed: the range is an input to the address
derivation, and each agent derives every peer's address itself. Two members
configured with different ranges would therefore derive different addresses
for each other and IPv4 would silently misroute. So the range now travels
in the announcement — not as a request and never trusted, only so the
mismatch is seen. A peer whose range disagrees gets no IPv4 address here,
keeps working over IPv6, and the reason is reported with both ranges named.
The announcement format goes to version 2. postcard is not
self-describing, so an older peer cannot read it; the version check already
catches that and now says which side needs updating.
The (Ipv4Addr, u8) tuple that had spread across six modules is now an
Ipv4Range with validation, Display and FromStr, so a bad --ipv4-range is
refused with a reason instead of being accepted and misbehaving later. It
is also rejected when passed without --wireguard rather than ignored.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every member now also derives an IPv4 address, from the same inputs as its
IPv6 one, into 100.64.0.0/10 by default. The range is configurable and IPv4
can be turned off with --no-ipv4.
IPv4 is honestly weaker than IPv6 here and the code says so. A 64 bit
interface identifier makes an IPv6 collision impossible in practice; IPv4
has nothing like that room, and in a /10 with 50 members two will derive the
same address about 0.03% of the time. A mesh with no coordinator cannot
allocate around that, so a collision is detected and resolved instead: the
member whose public key sorts lower keeps the address, a rule every member
computes identically and therefore agrees on. The other keeps IPv6 and is
flagged in the status. IPv6 always works; IPv4 almost always works and
degrades predictably.
Routing and address-ownership enforcement now cover both families: a packet
goes to the peer that owns its destination, and a decrypted packet is
dropped unless its source is an address derived for the peer that sent it,
IPv4 included.
Six new tests, among them a real IPv4 packet crossing a tunnel next to an
IPv6 one, a spoofed IPv4 source being dropped, and an IPv6-only overlay.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was no way to ask a running agent what it was doing; the only status
came from the periodic print of the `up` process itself.
The new ipc module is an adapter over the public API: nothing in the agent
core knows it exists, so a Windows named pipe or an authenticated loopback
socket can be added beside it. It is also a different interface from the
peer-to-peer protocol — between processes on one machine, authorised by
filesystem permissions rather than the network secret. The socket is 0600
inside an owner-only directory, the wire format is length-prefixed postcard
with the same bounds the network protocol uses, and the report types are
their own stable format rather than the crate's internals.
The socket path is derived from the state directory into XDG_RUNTIME_DIR
when there is one. A Unix socket address is limited to about 100 bytes, and
a deeply nested state directory overflows it — which is exactly what
happened on the first attempt.
Two presentation fixes while here. Multicast is counted separately from
unroutable traffic, because Linux emits multicast on every IPv6 interface
and it was showing up as "packets for unknown addresses" on a healthy
agent. And WireGuard protocol errors are no longer added into the dropped
counter: a few are normal while both ends start a handshake at once, and a
working tunnel was reporting "dropped 3" with no traffic at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The setup recipe failed with a missing sysctl directory and "RTNETLINK
answers: Invalid argument". The cause was the default MTU of 1100.
IPv6 requires a minimum MTU of 1280 (RFC 8200) and Linux enforces it by
tearing IPv6 down on any interface below it: the per-device
/proc/sys/net/ipv6/conf entries disappear and an address can no longer be
assigned. Evidence on the test host: every interface at 1280 or above has
an IPv6 conf directory, every interface below it (1230, 1100) has none.
So the overlay MTU is now 1280, which is also the floor. A smaller value is
refused when the plugin opens, naming the reason, rather than surfacing as
an obscure netlink error after the user has already run four commands.
That leaves no slack against the other constraint: a packet needs mtu + 32
bytes of transport datagram, so 1312. A direct QUIC path offers roughly
1380 and fits; a relayed path may not, so the plugin now reports the exact
numbers when a link cannot carry a full-size packet, instead of only
counting silent drops.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The setup this tool printed did not work, and the agent then correctly
refused to start. A persistent TUN interface has no carrier until a process
attaches to it, and Linux flushes IPv6 addresses from an interface that
loses carrier unless net.ipv6.conf.<dev>.keep_addr_on_down is set, which it
is not by default. So `ip -6 address add` on a freshly created interface
silently lost the address before the agent ever ran.
The recipe now brings the link up first, sets keep_addr_on_down, and adds
the address with `nodad` — without which duplicate address detection can
never finish on an interface with no carrier and the address stays
tentative and unusable.
The agent's own retry loop made this worse: it attached, failed the address
check, dropped the device and toggled the carrier, which flushed the
address again. The check now runs before attaching to an existing
interface, so looking is not destructive.
Failures are self-diagnosing now: the check parses the IFA_F_* flags, tells
tentative and DAD-failed apart from missing, and lists the addresses the
interface actually has.
Four new tests, including one that reads this host's real /proc/net/if_inet6
and one that pins the ordering of the setup commands.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The tun crate runs configure() by default (ensure_root_privileges is true),
which is harmless today because it only acts on fields that were set, and
the attach path sets none. Saying so explicitly documents the intent and
keeps it true if the crate changes.
Also records why packet information stays off: `ip tuntap add ... mode tun`
defaults to no packet information too, so the TUNSETIFF flags match when
attaching and reads and writes stay raw IP packets.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Creating a network interface needs CAP_NET_ADMIN, but that is a one-time
setup step rather than something the agent must hold for its whole life.
SystemTunFactory now attaches to an interface that already exists and only
creates one when it does not. A persistent interface created by root and
owned by the user therefore lets the agent run with no privileges and no
capabilities at all. When attaching, nothing is reconfigured, since doing so
would need exactly the privileges we are avoiding.
New `tsunagi tun-setup` prints the three commands to run once as root,
resolving the derived interface name and overlay address for the network.
This also fixes a real gap: the overlay address was passed to the factory
and thrown away, so an interface the agent created had no address and could
never have received anything. The `tun` crate sets addresses through an
IPv4-only ioctl and cannot assign an IPv6 one at all, so the agent now
verifies the address is present via /proc/net/if_inet6 and refuses with the
exact command to run instead of coming up broken. Doing it in-process would
mean speaking netlink, which is not implemented and is recorded as such.
Not verified on this machine: no sudo is available here, so the privileged
setup and the attach path were not executed end to end.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
"n0" is Number 0, the company behind iroh, and the name leaked from iroh's
own preset into this project's user interface, where it explains nothing.
The value is now --transport relay, which says what it does; n0 stays as an
accepted alias.
Also spells out, in the CLI help, the README, the threat model and the
TransportPolicy docs, whose infrastructure is involved: address records are
published to and resolved from dns.iroh.link, and the fallback relays are
Number 0's, in the US, EU and Asia-Pacific.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Running `tsunagi up` twice with the same arguments failed with "network ...
is already active", and then dropped the iroh endpoint without closing it.
A configured network is activated automatically at startup, so the second
run found it already up. `join_network` is declarative — "be a member of
this network" — so joining one that is already active now succeeds and
changes nothing. `activate_network` stays strict for callers that
specifically want to know whether an inactive network was started.
The CLI now closes the agent on the error path too, and handles SIGTERM as
well as Ctrl-C, so a service manager stopping the agent gets the same clean
shutdown an interactive user does.
Also documents the two lookups people conflate: resolving one endpoint's
address is iroh's public pkarr/DNS service and works today, which is why
`--peer <endpoint-id>` needs no address; finding who is in a network is this
project's `NetworkDiscovery` and is still static bootstrap only. Notes in
the README and the threat model that `n0` and `direct` publish this
endpoint's addresses to a public third-party service.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Corrects the architecture on two points raised in review, while the project
is still small enough to change cheaply.
1. Control and data are separated *logically*, not physically.
The old reading — "nothing but control may ride on iroh" — threw away iroh's
whole value and would have forced the data plane to reimplement STUN, ICE and
a relay. Now both planes ride on iroh with different ALPNs and different
connections, so the data plane inherits hole punching and relay fallback,
while proto/ still knows nothing about packets and dataplane/ knows nothing
about the control protocol.
New boundary: PacketTransport / PacketLink, an authenticated unreliable
datagram channel per (network, peer, protocol). tsunagi/data/1 runs the same
membership handshake, then DataOpen/DataOpenAck, then QUIC datagrams. Only
the smaller endpoint id dials, so exactly one link exists per pair.
A plugin is handed links and never learns reachability, so the WireGuard
announcement shrank to a public key: there is no address left to lie about.
2. WireGuard now runs in userspace, on boringtun's protocol state machine.
No kernel module, no wg tool, no ip shell-out, no loopback proxy: the wgtool,
backend and bridge modules are gone. Only creating a TUN device needs
privileges, and that sits behind TunFactory, so the entire data plane —
handshake, encryption, routing, address ownership — is tested with none.
Address ownership is enforced rather than believed: outbound packets go to
the owner of the destination address, inbound packets are dropped unless
their source is the address derived for the peer that sent them.
3. A `tsunagi` binary: secret, doctor, id, up. It owns the runtime, the
logging subscriber and Ctrl-C, which the library still refuses to.
Also fixes a reference cycle where IrohTransport held Arc<Inner>, which kept
the databases open and the directory lock held after shutdown; two storage
tests caught it once the cycle existed.
81 tests pass offline with no privileges, including real IPv6 packets
crossing a real WireGuard tunnel over real iroh connections. Verified by
hand: two CLI processes forming a mesh both on loopback and via n0 discovery
using only an endpoint id.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The first IP plugin, built on the data plane boundary the core already had.
Plugin:
- one X25519 key per network in the plugin's own wireguard.sqlite, separate
from the iroh identity and from the network secret; a damaged store is an
error, never a silently regenerated identity
- deterministic IPv6 ULA overlay: every member derives the same /64 from the
network id and its own /128 from its WireGuard public key, so no
coordinator allocates addresses
- AllowedIPs are derived locally, never taken from a peer's announcement, so
a member cannot claim another member's overlay address; a mismatched claim
is rejected
- bounded, versioned, validated announcement carried as the existing opaque
capability payload, which the core still never parses
- each agent builds its own full-mesh configuration (N-1 peers) and
reconciles on every change and on a timer, repairing drift
- WireguardBackend abstraction: RecordingBackend in memory, and WgToolBackend
driving real wg/ip on Linux, split into a pure planner plus parsers and a
thin executor so everything interesting is testable without root
Core, three generic additions the plugin needed:
- IpPlugin::on_network_activated, so per-network state is ready before peers
- PluginContext for re-announcements and error reports from plugin tasks,
with errors counted by the owning network runtime
- IpPlugin::shutdown, awaited with a grace period, so system objects go away
94 tests pass offline with no privileges: 35 new WireGuard unit tests and 12
integration tests over real iroh connections. The real wg/ip backend needs
root and is behind --ignored in tests/wireguard_system.rs; it was not run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Working library with real iroh connections, not an interface sketch:
- persistent device identity in state.sqlite, stable across restarts
- deterministic network space derived from name + secret via HKDF-SHA256,
with frozen labels and unambiguous length-prefixed encoding
- replaceable discovery returning unverified candidates only; static
bootstrap, in-memory test backend and a composite
- real iroh connections plus an explicit mutual membership proof:
HMAC-SHA256 over a role-separated transcript bound to the TLS exporter,
the network id and both endpoint identities
- small versioned control protocol: handshake, announcement, ping/pong
- multiple networks per agent with enforced isolation
- automatic reconnect with bounded backoff and jitter
- mandatory state vs disposable cache, with a real directory ownership lock
- status snapshots, event stream and honest diagnostics
47 integration and unit tests cover the required scenarios offline on
loopback. Snapshots, revocations and WireGuard are designed for and
documented, not implemented.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
tsunagiandClaude Opus 5