Relay across any mix of links, one online answer for every client, first claim owns a hostname

A tunnel no longer depends on the kind of link under it. The relay hub keeps
one forwarding table per network instead of one per protocol, the envelope
names the end-to-end protocol, and a neighbour reached over several kinds of
transport is sent to over the best open one. A member in the middle needs no
protocol in common with either end. Which protocol a pair runs is settled by
the lower endpoint id's announced order, so both ends agree without asking.
A direct link that comes up later is shorter than the path through others, so
traffic moves to it with the same logical link.

The agent now answers "is it online" once: a live control session or any data
path. Members and peers report how they are reached (direct over a named
transport, relayed through a named member, or no data path yet), and `tsng
status` and the tray both read that instead of a session list and a WireGuard
handshake respectively.

A hostname belongs to whoever claimed it first. A claim carries the time its
author first claimed the name, signed with the record, and replicas order by
(time, id) so every member names the same holder. A later claimant keeps
working by address, is not found by the name, and is told so in its status,
`tsng status`, the tray and a HostnameTaken event.

Wire change: control and data ALPN 5, control PROTOCOL_VERSION 5, envelope
version 2, signing domain v3, state schema 7, local control protocol 18.
Upgrade every member together.

Co-Authored-By: Claude <noreply@anthropic.com>

crates/tsunagi-core/src/agent/events.rs

@@ -129,6 +129,19 @@ pub enum Event {
         /// Why it is not up.
         reason: String,
     },
+    /// The name this agent claimed was claimed first by another member.
+    ///
+    /// Nothing stops working: this agent keeps its address and every
+    /// connection. It just is not found by that name until it is renamed or
+    /// the member that holds the name lets it go.
+    HostnameTaken {
+        /// The network.
+        network: NetworkId,
+        /// The name that is taken.
+        name: String,
+        /// The member that holds it.
+        holder: EndpointId,
+    },
     /// An IP plugin reported an error. Never fatal.
     PluginError {
         /// The network the call was scoped to.

crates/tsunagi-core/src/agent/mod.rs

@@ -26,8 +26,8 @@ mod status;
 
 pub use events::Event;
 pub use status::{
-    AgentStatus, CandidateStatus, MemberStatus, NetworkMetrics, NetworkState, NetworkStatus,
-    OverlayStatus, PeerStatus,
+    AgentStatus, CandidateStatus, DataPath, HostnameConflict, MemberStatus, NetworkMetrics,
+    NetworkState, NetworkStatus, OverlayStatus, PeerStatus,
 };
 
 use std::collections::HashMap;
@@ -981,6 +981,7 @@ impl Agent {
                 // An inactive network has no runtime to ask; the roster and
                 // the range come from one. Empty, not invented.
                 members: Vec::new(),
+                hostname_conflict: None,
                 range: None,
                 range_conflict: None,
                 metrics: NetworkMetrics::default(),

crates/tsunagi-core/src/agent/network.rs

@@ -32,7 +32,8 @@ use super::events::Event;
 use super::session::{self, Session, SessionEvent};
 use super::shutdown::Shutdown;
 use super::status::{
-    CandidateStatus, MemberStatus, NetworkMetrics, NetworkState, NetworkStatus, PeerStatus,
+    CandidateStatus, DataPath, HostnameConflict, MemberStatus, NetworkMetrics, NetworkState,
+    NetworkStatus, PeerStatus,
 };
 
 /// An inbound connection that already passed the handshake.
@@ -320,6 +321,10 @@ struct Runtime {
             std::time::Instant,
         ),
     >,
+    /// The protocols this agent last announced, in the order it announced
+    /// them. The order is its preference, and the lower of two ids settles
+    /// which protocol a pair runs.
+    local_capabilities: Vec<PluginCapability>,
     /// What this agent last told its peers it could reach, so it is only
     /// said again when it changed.
     announced_reach: HashSet<crate::proto::message::ReachableLink>,
@@ -338,6 +343,9 @@ struct Runtime {
     /// publish a claim afterwards — the periodic check, a peer's records
     /// arriving — would silently take it back.
     released: bool,
+    /// The name this agent last said was taken, so it is said once per
+    /// change rather than on every round.
+    reported_name_taken: Option<HostnameConflict>,
     /// The address last reported as absent from the interface, so it is said
     /// once rather than for ever.
     reported_missing: Option<std::net::Ipv4Addr>,
@@ -366,6 +374,16 @@ impl Runtime {
         let (link_results_tx, link_results_rx) = mpsc::channel(64);
         let (candidate_tx, discovery_candidates) =
             mpsc::channel(params.limits.max_discovery_candidates.max(1));
+        let hub = crate::dataplane::relay::RelayHub::new(network_id, local_id);
+        // The order the protocols were configured in is also how links are
+        // preferred when a neighbour can be reached over more than one.
+        hub.set_priority(
+            params
+                .plugins
+                .iter()
+                .map(|plugin| plugin.protocol_id().to_string())
+                .collect(),
+        );
         let discovery_worker = params.discovery.as_ref().map(|backend| {
             DiscoveryWorker::spawn(
                 backend.clone(),
@@ -393,8 +411,9 @@ impl Runtime {
             dial_results_tx,
             dial_results_rx,
             links: HashMap::new(),
-            hub: crate::dataplane::relay::RelayHub::new(network_id, local_id),
+            hub,
             reachable: HashMap::new(),
+            local_capabilities: Vec::new(),
             announced_reach: HashSet::new(),
             opening: HashSet::new(),
             link_results_tx,
@@ -403,6 +422,7 @@ impl Runtime {
             reported_mismatch: HashSet::new(),
             released: false,
             reported_missing: None,
+            reported_name_taken: None,
             state: StateSet::new(),
             pending_state: Vec::new(),
             own_version: 0,
@@ -617,6 +637,8 @@ impl Runtime {
                 tracing::debug!(%err, "could not queue re-announcement");
             }
         }
+        // What was announced decides which protocol each pair runs.
+        self.update_paths();
     }
 
     /// Queues a message without blocking the runtime loop.
@@ -669,6 +691,7 @@ impl Runtime {
         // alone, so something has to look again; this runs on a timer and
         // the check is a comparison when nothing has changed.
         self.ensure_own_claim().await;
+        self.report_hostname_taken();
         // The same timer repairs the relay picture: reachability is soft
         // state, repeated while it holds and forgotten when it stops.
         self.update_paths();
@@ -1031,12 +1054,25 @@ impl Runtime {
             }
         };
 
+        // A name keeps the time it was first claimed for as long as this
+        // member keeps the name, so a later claimant cannot overtake it by
+        // the passage of time, and a new or changed name starts the clock
+        // afresh.
+        let hostname_since = wanted_hostname.as_ref().map(|name| {
+            let kept = self
+                .state
+                .get(&self.local_id)
+                .filter(|record| record.body.hostname() == Some(name.as_str()))
+                .and_then(|record| record.body.hostname_since());
+            kept.unwrap_or_else(unix_millis)
+        });
         let body = RecordBody::Claim {
             address: wanted_address,
             // The range only travels with an address, so a member of an
             // IPv6-only network does not assert one.
             range: wanted_address.and(range),
             hostname: wanted_hostname,
+            hostname_since,
         };
 
         // Nothing to say is not the same as saying nothing changed: a member
@@ -1100,6 +1136,7 @@ impl Runtime {
         for (peer, records) in std::mem::take(&mut self.pending_state) {
             self.receive_state(peer, records).await;
         }
+        self.report_hostname_taken();
     }
 
     /// Sends everything we know to every peer.
@@ -1658,68 +1695,93 @@ impl Runtime {
         }
     }
 
+    /// The protocol this agent and a member run their tunnel in, if they have
+    /// one in common.
+    ///
+    /// Both ends must name the same one without asking each other, and what
+    /// each side was configured to prefer can differ. So the member with the
+    /// lower id decides, by the order it announces its protocols in, and the
+    /// other side reads that order from the announcement it already has.
+    fn tunnel_protocol(&self, session: &Session) -> Option<String> {
+        let usable = |ours: &[PluginCapability], theirs: &[PluginCapability], name: &str| {
+            ours.iter().any(|capability| {
+                capability.enabled
+                    && capability.protocol == name
+                    && theirs.iter().any(|other| {
+                        other.enabled
+                            && other.protocol == name
+                            && other.version == capability.version
+                    })
+            })
+        };
+        let (decides, other) = if self.local_id.as_bytes() < session.peer.as_bytes() {
+            (&self.local_capabilities, &session.capabilities)
+        } else {
+            (&session.capabilities, &self.local_capabilities)
+        };
+        decides
+            .iter()
+            .find(|capability| usable(decides, other, &capability.protocol))
+            .map(|capability| capability.protocol.clone())
+    }
+
     /// Builds the graph off the packet path. Every remote row comes from that
-    /// member's authenticated control session; stale or incompatible rows never
-    /// enter the graph. The hub replaces an immutable table only on changes.
+    /// member's authenticated control session, and a row says only who that
+    /// member has a link to, never what kind of link: a member in the middle
+    /// needs no protocol in common with either end. Stale rows never enter the
+    /// graph. The hub replaces an immutable table only on changes.
     fn update_paths(&mut self) {
         self.reachable.retain(|peer, (_, heard)| {
             heard.elapsed() < REACH_EXPIRY && self.sessions.contains_key(peer)
         });
-        for (protocol, version) in self.served_protocols() {
-            let peers: Vec<_> = self
-                .sessions
-                .values()
-                .filter(|session| {
-                    session.capabilities.iter().any(|capability| {
-                        capability.enabled
-                            && capability.protocol == protocol
-                            && capability.version == version
-                    })
-                })
-                .map(|session| session.peer)
-                .collect();
-            let mut members: HashSet<_> = peers.iter().map(|peer| *peer.as_bytes()).collect();
-            members.insert(*self.local_id.as_bytes());
-            let graph = self
-                .reachable
-                .iter()
-                .filter(|(peer, _)| members.contains(peer.as_bytes()))
-                .map(|(peer, (links, _))| {
-                    let mut neighbors: Vec<_> = links
-                        .iter()
-                        .filter(|link| {
-                            link.protocol == protocol
-                                && members.contains(&link.peer)
-                                && link.peer != *peer.as_bytes()
-                        })
-                        .map(|link| link.peer)
-                        .collect();
-                    neighbors.sort_unstable();
-                    (*peer.as_bytes(), neighbors)
-                })
-                .collect();
-            self.hub.set_topology(&protocol, graph, members);
-            for peer in peers {
-                if self.hub.has_link(peer, &protocol) || !self.hub.reachable(peer, &protocol) {
-                    continue;
-                }
-                let Some(plugin) = self
-                    .params
-                    .plugins
+        let mut members: HashSet<_> = self.sessions.keys().map(|peer| *peer.as_bytes()).collect();
+        members.insert(*self.local_id.as_bytes());
+        let graph = self
+            .reachable
+            .iter()
+            .filter(|(peer, _)| members.contains(peer.as_bytes()))
+            .map(|(peer, (links, _))| {
+                let mut neighbors: Vec<_> = links
                     .iter()
-                    .find(|plugin| plugin.protocol_id() == protocol)
-                    .cloned()
-                else {
-                    continue;
-                };
-                let link = self.hub.link(peer, &protocol);
-                let path = link.path_description();
-                let max_datagram = link.max_datagram_size();
-                plugin.on_peer_link(self.network_id, peer, link);
+                    .filter(|link| members.contains(&link.peer) && link.peer != *peer.as_bytes())
+                    .map(|link| link.peer)
+                    .collect();
+                neighbors.sort_unstable();
+                neighbors.dedup();
+                (*peer.as_bytes(), neighbors)
+            })
+            .collect();
+        self.hub.set_topology(graph, members);
+
+        let wanted: Vec<(EndpointId, String)> = self
+            .sessions
+            .values()
+            .filter_map(|session| Some((session.peer, self.tunnel_protocol(session)?)))
+            .collect();
+        for (peer, protocol) in wanted {
+            if self.hub.has_link(peer, &protocol) || !self.hub.reachable(peer) {
+                continue;
+            }
+            let Some(plugin) = self
+                .params
+                .plugins
+                .iter()
+                .find(|plugin| plugin.protocol_id() == protocol)
+                .cloned()
+            else {
+                continue;
+            };
+            let link = self.hub.link(peer, &protocol);
+            let relayed = self.hub.route(peer).is_some_and(|route| route.hops > 1);
+            let path = link.path_description();
+            let max_datagram = link.max_datagram_size();
+            plugin.on_peer_link(self.network_id, peer, link);
+            // A direct link has already been reported as one.
+            if relayed {
                 self.emit(Event::DataLinkUp {
                     network: self.network_id,
                     peer,
-                    protocol: protocol.clone(),
+                    protocol,
                     path,
                     max_datagram,
                 });
@@ -1786,30 +1848,25 @@ impl Runtime {
 
     /// Hands a link to the plugin that owns its protocol.
     fn adopt_link(&mut self, peer: EndpointId, protocol: String, link: SharedLink) {
-        let Some(plugin) = self
+        if !self
             .params
             .plugins
             .iter()
-            .find(|plugin| plugin.protocol_id() == protocol)
-            .cloned()
-        else {
+            .any(|plugin| plugin.protocol_id() == protocol)
+        {
             return;
-        };
+        }
 
         let path = link.path_description();
+        let max_datagram = link.max_datagram_size();
         self.links
             .insert((peer, protocol.clone()), Arc::clone(&link));
-        // The protocol is handed the hub's link, not this one: the same
-        // object for as long as the peer is a peer, whatever happens to
-        // the path under it. A tunnel is not torn down because a direct
-        // link came or went.
-        let first_time = !self.hub.has_link(peer, &protocol);
+        // What this is, is a way to reach the peer. Which protocol's
+        // tunnel runs over it is decided in `update_paths`, and that tunnel
+        // is handed the hub's link, not this one: the same object for as
+        // long as the peer is a peer, whatever happens to the path under
+        // it. A tunnel is not torn down because a direct link came or went.
         self.hub.set_direct(peer, &protocol, link);
-        let shared = self.hub.link(peer, &protocol);
-        let max_datagram = shared.max_datagram_size();
-        if first_time {
-            plugin.on_peer_link(self.network_id, peer, shared);
-        }
         self.metrics.data_links_established += 1;
         self.emit(Event::DataLinkUp {
             network: self.network_id,
@@ -1941,6 +1998,13 @@ impl Runtime {
             });
         }
         capabilities.truncate(self.params.limits.max_capabilities);
+        self.local_capabilities = capabilities
+            .iter()
+            .map(|capability| PluginCapability {
+                data: Vec::new(),
+                ..capability.clone()
+            })
+            .collect();
         let exit_node = self.exit_ready();
         self.announced_exit = exit_node;
         Announcement {
@@ -2175,6 +2239,69 @@ impl Runtime {
 
     // ----------------------------------------------------------------- status
 
+    /// Whether a member can be reached now, by any path.
+    ///
+    /// The single definition every client reports: a live control session,
+    /// or a data path to it through other members. Being reached over a long
+    /// path is being online; how well is `data_path`'s to say.
+    fn is_online(&self, peer: EndpointId) -> bool {
+        peer == self.local_id || self.sessions.contains_key(&peer) || self.hub.reachable(peer)
+    }
+
+    /// How packets reach a member, which says nothing about whether it is
+    /// there.
+    fn data_path(&self, peer: EndpointId) -> DataPath {
+        match self.hub.route(peer) {
+            None => DataPath::None,
+            Some(route) if route.hops <= 1 => DataPath::Direct {
+                transport: self
+                    .hub
+                    .direct_kinds(peer)
+                    .into_iter()
+                    .next()
+                    .unwrap_or_default(),
+            },
+            Some(route) => DataPath::Relayed {
+                hops: route.hops,
+                via: route.via,
+            },
+        }
+    }
+
+    /// Says so when this agent's own name is held by someone else, and
+    /// again if that changes. Never anything but a statement: the agent keeps
+    /// working by address either way.
+    fn report_hostname_taken(&mut self) {
+        let taken = self.hostname_conflict_of(&self.local_id);
+        if taken == self.reported_name_taken {
+            return;
+        }
+        if let Some(conflict) = &taken {
+            tracing::warn!(
+                network = %self.network_id,
+                name = %conflict.name,
+                holder = %conflict.holder.fmt_short(),
+                "this name was claimed first by another member; it stays reachable by \
+                 address but is not found by this name until it is renamed"
+            );
+            self.emit(Event::HostnameTaken {
+                network: self.network_id,
+                name: conflict.name.clone(),
+                holder: conflict.holder,
+            });
+        }
+        self.reported_name_taken = taken;
+    }
+
+    fn hostname_conflict_of(&self, author: &EndpointId) -> Option<HostnameConflict> {
+        self.state
+            .hostname_taken(author)
+            .map(|(name, holder)| HostnameConflict {
+                name: name.to_string(),
+                holder,
+            })
+    }
+
     fn status(&self) -> NetworkStatus {
         let served = self.served_protocols();
         let mut peers: Vec<PeerStatus> = self
@@ -2212,6 +2339,7 @@ impl Runtime {
                     control_messages_received: session.messages_received,
                     control_bytes_sent: session.bytes_sent,
                     control_bytes_received: session.bytes_received,
+                    data_path: self.data_path(session.peer),
                 }
             })
             .collect();
@@ -2249,6 +2377,9 @@ impl Runtime {
                     overlay_address_v4: self.state.address_of(&endpoint_id),
                     hostname: self.state.hostname_of(&endpoint_id).map(str::to_string),
                     last_hostname: self.known_hostnames.get(&endpoint_id).cloned(),
+                    hostname_conflict: self.hostname_conflict_of(&endpoint_id),
+                    online: self.is_online(endpoint_id),
+                    data_path: self.data_path(endpoint_id),
                 })
             })
             .collect();
@@ -2276,6 +2407,7 @@ impl Runtime {
             peers,
             candidates,
             members,
+            hostname_conflict: self.hostname_conflict_of(&self.local_id),
             range: self.effective_range(),
             // Only worth reporting while it is actually stuck: with a
             // fallback in play the configured range being another
@@ -2290,6 +2422,15 @@ impl Runtime {
     }
 }
 
+/// Milliseconds since the Unix epoch, for the time a name was first claimed.
+fn unix_millis() -> u64 {
+    std::time::SystemTime::now()
+        .duration_since(std::time::UNIX_EPOCH)
+        .map_or(u64::MAX, |elapsed| {
+            u64::try_from(elapsed.as_millis()).unwrap_or(u64::MAX)
+        })
+}
+
 /// Which side dialled, given a role and the two identities.
 fn initiator_of(role: Role, local: EndpointId, peer: EndpointId) -> EndpointId {
     match role {

crates/tsunagi-core/src/agent/status.rs

@@ -59,6 +59,43 @@ pub struct OverlayStatus {
     pub broadcast_rules: Option<crate::overlay::BroadcastRulesReport>,
 }
 
+/// How packets reach a member right now.
+///
+/// Being reachable is one thing and being reachable well is another, and
+/// the second is never a reason to call a member away: a member that can only
+/// be reached through somebody else is online, over a longer path.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum DataPath {
+    /// Nothing carries packets to it yet. The member is connected for
+    /// control only, which is the state of a data path being set up.
+    None,
+    /// A direct link, of the named kind of transport.
+    Direct {
+        /// The kind of transport carrying it, e.g. `wg`.
+        transport: String,
+    },
+    /// Through other members.
+    Relayed {
+        /// Number of links on the way, at least two.
+        hops: u8,
+        /// The member the first of them goes to.
+        via: EndpointId,
+    },
+}
+
+/// A name that a member claimed and somebody else holds.
+///
+/// A name belongs to whoever claimed it first. The member that claimed it
+/// later is still a member and still reachable by its address; it is just
+/// not found by that name, on any member of the network.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct HostnameConflict {
+    /// The name that is taken.
+    pub name: String,
+    /// The member that holds it.
+    pub holder: EndpointId,
+}
+
 /// A member the signed state knows about, connected or not.
 ///
 /// This is the durable roster: it comes from signed records, so a member that
@@ -85,6 +122,15 @@ pub struct MemberStatus {
     /// Not signed, so it is a convenience for naming a member that is away
     /// and never a claim: the zone is served from `hostname`.
     pub last_hostname: Option<String>,
+    /// The name it claimed, when somebody else claimed it first.
+    pub hostname_conflict: Option<HostnameConflict>,
+    /// Whether this agent can reach it now, by any path: a live control
+    /// session, or a data path through other members. The one answer to "is
+    /// it online" for every client, so that none of them has to work it out
+    /// from a different part of the picture. This agent counts as online.
+    pub online: bool,
+    /// How packets reach it. [`DataPath::None`] while it is offline.
+    pub data_path: DataPath,
 }
 
 /// Status of one authenticated session.
@@ -128,6 +174,8 @@ pub struct PeerStatus {
     pub control_bytes_sent: u64,
     /// Control payload bytes read from this session.
     pub control_bytes_received: u64,
+    /// How packets reach this peer right now.
+    pub data_path: DataPath,
 }
 
 /// Counters scoped to one logical network.
@@ -188,6 +236,10 @@ pub struct NetworkStatus {
     pub candidates: Vec<CandidateStatus>,
     /// Members the signed state knows about, whether connected or not.
     pub members: Vec<MemberStatus>,
+    /// This agent's own name, when another member claimed it first and so
+    /// holds it. This agent keeps working by address; nobody finds it by
+    /// this name until it is renamed or the other member lets it go.
+    pub hostname_conflict: Option<HostnameConflict>,
     /// The overlay range this network uses, once it has one.
     pub range: Option<crate::state::Ipv4Range>,
     /// The range it could not have, because another network on this agent