Compare commits

..
Author SHA1 Message Date
Aleksandr Bogomiakov 23af1a2d9d Add shared TCP and UDP virtual services
Provide named VIPs with independent protocol/port listeners and node/port
backends so applications can use one DNS endpoint without a separate load
balancer. Every authenticated member may create, edit or delete any service.
Client agents select the eligible backend with the lowest measured encrypted
data-plane RTT, including relayed paths, and pin existing flows. UDP recovers
on another backend after failure; established TCP is never replayed or moved.
An unanswered empty TCP SYN may retry another backend before establishment.

Keep signed multi-writer field registers in state/vip.rs. Independent backend
edits merge, same-field conflicts use Lamport/author/canonical ordering, and
listener generations prevent old backend entries returning on port recreation.
Permanent service tombstones protect replay while allowing obsolete fields to
be compacted. Active-service and deletion-history limits are separate. Commit
complete snapshots and the author's durable counter atomically before local
routing changes or publication. Restore counters from newer genuine author
updates learned from replicas after an old backup is restored.

Implement bounded client-side address/port NAT at overlay ingress and reverse
translation only after authenticated source/destination admission. Reserve OS
source ports to avoid direct-flow collisions and quarantine retired ports for
late replies. Handle a backend on the client itself, scoped fragment assembly,
checksums, re-fragmentation and quoted ICMP/PMTU headers. Plugins and encrypted
transit retain their existing contracts. Data-path probes measure RTT; bounded
fair TCP connect checks are enabled by default and configurable per listener.
Arbitrary UDP application readiness remains unknown; path checks and explicit
port errors drive eligibility without protocol-specific probe payloads.

Expose the shared draft API through local IPC, simple vip/listener/backend CLI
commands, JSON, editor-based atomic updates and watch output. Add a VIP button
on each network tile and a separate tray editor. Edits and removals stay local
until Save; stale revisions retain the draft, and closing a dirty window offers
explicit discard. Report local routing choices, health reasons, RTT age, active
and draining bindings, IP counters, rates, drops and UDP failovers. Extend DNS
with service names; keep numeric addresses secondary and preserve existing
bindings while VIP addresses or listeners change.

This changes control wire ALPN/version to tsunagi/ctrl/8 and local IPC to 22;
all members and local clients must upgrade together. Data ALPN remains 5.
SQLite schema 10 migrates existing mandatory state without changing identity,
secrets, network ids, derivation labels, handshake encoding or node records.
Crate versions and release tags are unchanged. No installed agent or tray was
replaced. Commands, limits and operational semantics are in docs/vip.md.

Checks on Linux in nix develop:
- cargo fmt --all -- --check
- cargo clippy --locked --workspace --all-targets -- -D warnings
- cargo test --locked --workspace --all-targets: 559 passed, 3 ignored
- git diff --cached --check

The full suite includes real encrypted TCP/UDP VIP traffic, backend-stop UDP
failover, real loopback TCP check recovery, source admission, signed merges,
stale/deleted state, SQLite migration and rollback, restart/author recovery,
network isolation, actual CLI/DNS processes, and headless text-entry/Save-button
interaction. Discovery and memory TUNs are the existing test substitutions;
authentication, transport, encryption and SQLite remain real.

Initial checks outside Nix failed for missing GTK pkg-config metadata, and an
initial CLI test inherited TSUNAGI_CONTROL_SOCKET. Final checks use the Nix
environment with that variable unset so tests address their temporary sockets.
Root/internet-dependent tests remain ignored. No live host TUN, native desktop
session, macOS or Windows validation was performed.
2026-10-21 15:40:45 +01:00
35 changed files with 5055 additions and 21 deletions

No files matched your search

Generated
+1
View File
@@ -6239,6 +6239,7 @@ dependencies = [
"iroh",
"netwatch",
"nix 0.31.3",
"serde_json",
"simple-dns",
"tempfile",
"tokio",
+14
View File
@@ -94,6 +94,19 @@ stopping a suspended network cancels its automatic resume.
Machines are called `tsunagi-<id>` until you rename them: `tsng id hostname laptop`.
### Virtual services
```sh
tsng vip create home api
tsng vip port add home api tcp/443 --backend server1:6443 --backend server2:7443
tsng vip status home api --watch
```
Use `api.home:443`. Each client selects the available backend with the lowest
measured data-plane RTT. A VIP can have multiple independent TCP and UDP ports.
Every member may edit it; the tray's **VIP** button opens a draft editor with
explicit Save. [Commands, health, metrics and limits](docs/vip.md).
### Exit nodes
```sh
@@ -227,6 +240,7 @@ The [threat model](docs/threat-model.md) says what this does and doesn't protect
[Pairwise Amnezia transport](docs/amnezia.md) ·
[Routing](docs/routing.md) ·
[Exit nodes](docs/exit-node.md) ·
[Virtual services](docs/vip.md) ·
[Threat model](docs/threat-model.md) ·
[Contributing](AGENTS.md)
+87
View File
@@ -309,6 +309,28 @@ impl TransportContext for TransportCtx {
}
impl Agent {
/// Atomically saves or removes a version-checked shared VIP draft.
/// Every authenticated network member may edit any service.
pub async fn save_vip(
&self,
network: NetworkId,
config: crate::vip::VipConfig,
expected: Option<String>,
delete: bool,
) -> Result<Vec<crate::vip::VipStatus>> {
let (reply, receive) = oneshot::channel();
self.command(
network,
NetCommand::SaveVip {
config,
expected,
delete,
reply,
},
)
.await?;
receive.await.map_err(|_| Error::Stopped)?
}
/// Retires one offline member's observed claim without revoking membership.
/// Every member has this right. The version must still match the selection.
pub async fn forget_node(
@@ -916,6 +938,34 @@ impl Agent {
.storage
.retirements(network_id, self.inner.limits.max_retirements)
.await?;
let vip_entries = self
.inner
.storage
.vip_fields(
network_id,
self.inner
.limits
.max_vips
.saturating_mul(self.inner.limits.max_vip_fields)
.saturating_add(self.inner.limits.max_vip_tombstones),
)
.await?;
let mut vips = crate::vip::VipSet::default();
for id in vip_entries
.iter()
.map(|e| e.vip.clone())
.collect::<std::collections::BTreeSet<_>>()
{
vips.merge(
network_id,
vip_entries
.iter()
.filter(|e| e.vip == id)
.cloned()
.collect(),
&self.inner.limits,
)?;
}
let range = self.desired_range(network_id).await?;
if let Some(range) = range {
let conflicts = self.inner.routes.overlapping_networks(network_id, range);
@@ -968,6 +1018,7 @@ impl Agent {
.and_then(|stored| stored.exit_via)
.and_then(|bytes| EndpointId::from_bytes(&bytes).ok());
let handle = network::spawn(RuntimeParams {
vips,
retirements,
broadcast,
exit_offer,
@@ -1215,7 +1266,43 @@ impl Agent {
let keys = NetworkKeys::derive(&stored.name, &stored.secret);
let range = self.desired_range(stored.network_id).await?;
let range_conflicts = self.range_conflicts(stored.network_id, range).await?;
let fields = self
.inner
.storage
.vip_fields(
stored.network_id,
self.inner
.limits
.max_vips
.saturating_mul(self.inner.limits.max_vip_fields)
.saturating_add(self.inner.limits.max_vip_tombstones),
)
.await?;
let mut service_state = crate::vip::VipSet::default();
for id in fields
.iter()
.map(|e| e.vip.clone())
.collect::<std::collections::BTreeSet<_>>()
{
service_state.merge(
stored.network_id,
fields.iter().filter(|e| e.vip == id).cloned().collect(),
&self.inner.limits,
)?;
}
let vips = service_state
.configs()
.into_iter()
.map(|config| crate::vip::VipStatus {
revision: service_state.revision(&config.id),
config,
address: None,
error: Some("network is inactive".into()),
listeners: vec![],
})
.collect();
networks.push(NetworkStatus {
vips,
protocols: self.effective_protocols(stored.protocols.as_ref()),
auto_start: stored.auto_start,
broadcast: stored.broadcast,
+226 -2
View File
@@ -50,6 +50,12 @@ pub(crate) struct InboundSession {
/// Commands accepted by a network runtime.
pub(crate) enum NetCommand {
SaveVip {
config: crate::vip::VipConfig,
expected: Option<String>,
delete: bool,
reply: oneshot::Sender<Result<Vec<crate::vip::VipStatus>>>,
},
ForgetNode {
peer: EndpointId,
expected_version: u64,
@@ -125,6 +131,7 @@ pub(crate) enum NetCommand {
impl std::fmt::Debug for NetCommand {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
NetCommand::SaveVip { .. } => f.write_str("SaveVip"),
NetCommand::ForgetNode { peer, .. } => write!(f, "ForgetNode({peer})"),
NetCommand::Inbound(_) => f.write_str("Inbound"),
NetCommand::InboundLink(link) => write!(f, "InboundLink({})", link.protocol),
@@ -170,6 +177,7 @@ impl NetworkHandle {
/// Everything a network runtime needs to run.
pub(crate) struct RuntimeParams {
pub(crate) vips: crate::vip::VipSet,
pub(crate) retirements: Vec<SignedRetirement>,
pub(crate) broadcast: bool,
/// This agent offers itself as an exit node here.
@@ -376,6 +384,8 @@ struct Runtime {
/// Snapshots received while dispatching, handled on the next loop pass.
pending_state: Vec<(EndpointId, Vec<SignedRecord>)>,
pending_retirements: Vec<(EndpointId, Vec<SignedRetirement>)>,
pending_vips: Vec<Vec<crate::vip::VipEntry>>,
vip_cursor: HashMap<EndpointId, usize>,
/// Next cleanup batch per live session; prevents queue pressure starving a suffix.
retirement_cursor: HashMap<EndpointId, usize>,
/// The highest version this agent has ever published for this network.
@@ -461,6 +471,8 @@ impl Runtime {
state: StateSet::new(),
pending_state: Vec::new(),
pending_retirements: Vec::new(),
pending_vips: Vec::new(),
vip_cursor: HashMap::new(),
retirement_cursor: HashMap::new(),
own_version: 0,
exit_via_was_online: false,
@@ -561,6 +573,15 @@ impl Runtime {
async fn handle_command(&mut self, command: NetCommand) {
match command {
NetCommand::SaveVip {
config,
expected,
delete,
reply,
} => {
let result = self.save_vip(config, expected, delete).await;
let _ = reply.send(result);
}
NetCommand::ForgetNode {
peer,
expected_version,
@@ -830,6 +851,7 @@ impl Runtime {
// the check is a comparison when nothing has changed.
self.ensure_own_claim().await;
self.report_hostname_taken();
self.broadcast_vips();
self.broadcast_retirements();
// The same timer repairs the relay picture: reachability is soft
// state, repeated while it holds and forgotten when it stops.
@@ -1104,7 +1126,8 @@ impl Runtime {
.own_record_version(self.network_id, self.local_id)
.await
.unwrap_or(0)
.max(self.state.authored_version(&self.local_id));
.max(self.state.authored_version(&self.local_id))
.max(self.params.vips.authored_version(self.local_id));
for (id, hostname) in self
.params
.storage
@@ -1178,11 +1201,26 @@ impl Runtime {
match self.state.address_of(&self.local_id) {
Some(mine) if range.contains(mine) => Some(mine),
_ => {
let taken: std::collections::HashSet<std::net::Ipv4Addr> = holders
let mut taken: std::collections::HashSet<std::net::Ipv4Addr> = holders
.iter()
.filter(|(_, holder)| **holder != self.local_id)
.map(|(address, _)| *address)
.collect();
for vip in self.params.vips.views(
Some(range),
taken.clone(),
&std::collections::BTreeSet::new(),
&self.params.limits,
) {
if let Some(address) = vip.address {
taken.insert(address);
}
}
for vip in self.params.routes.vips.statuses(self.network_id) {
if let Some(address) = vip.address {
taken.insert(address);
}
}
match allocate(
self.network_id,
self.local_id,
@@ -1460,8 +1498,188 @@ impl Runtime {
self.broadcast_state();
}
fn update_vips(&self) {
let taken = self.state.address_holders().into_keys().collect();
let names = self
.state
.active_records()
.into_iter()
.filter_map(|r| {
r.author_id()
.ok()
.and_then(|p| self.state.hostname_of(&p).map(str::to_owned))
})
.collect();
let views =
self.params
.vips
.views(self.effective_range(), taken, &names, &self.params.limits);
let peers = self
.state
.address_holders()
.into_iter()
.map(|(a, p)| (p, a, self.hub.reachable(p)))
.collect();
self.params.routes.vips.configure(
self.network_id,
self.local_id,
self.state.address_of(&self.local_id),
views,
peers,
&self.params.limits,
);
}
async fn save_vip(
&mut self,
config: crate::vip::VipConfig,
expected: Option<String>,
delete: bool,
) -> Result<Vec<crate::vip::VipStatus>> {
if self.released || self.shutdown.is_triggered() {
return Err(Error::Stopped);
}
let version = self
.own_version
.checked_add(1)
.filter(|n| *n <= i64::MAX as u64)
.ok_or_else(|| crate::vip::invalid("author counter exhausted"))?;
let entries = if delete {
self.params.vips.delete(
self.network_id,
&self.params.device_secret,
version,
&config.id,
expected.as_deref().unwrap_or(""),
)?
} else {
self.params.vips.edit(
self.network_id,
&self.params.device_secret,
version,
&config,
expected.as_deref(),
&self.params.limits,
)?
};
let mut candidate = self.params.vips.clone();
candidate.merge(self.network_id, entries, &self.params.limits)?;
if !delete {
let duplicate = candidate
.configs()
.iter()
.any(|v| v.id != config.id && v.name == config.name)
|| self.state.active_records().iter().any(|r| {
r.author_id().ok().and_then(|p| self.state.hostname_of(&p))
== Some(config.name.as_str())
});
if duplicate {
return Err(crate::vip::invalid(
"VIP name is already used by a node or another VIP",
));
}
}
let snapshot = candidate.entries(&config.id);
self.validate_vip_frame(&snapshot)?;
self.params
.storage
.put_vip_fields(
self.network_id,
snapshot.clone(),
Some((*self.local_id.as_bytes(), version)),
)
.await?;
self.own_version = version;
self.params.vips = candidate;
self.update_vips();
let peers: Vec<_> = self.sessions.keys().copied().collect();
for peer in peers {
let _ = self.send_to(
peer,
ControlMessage::Vips {
entries: snapshot.clone(),
},
);
}
Ok(self.params.routes.vips.statuses(self.network_id))
}
async fn receive_vips(&mut self, entries: Vec<crate::vip::VipEntry>) -> Result<()> {
let Some(id) = entries.first().map(|e| e.vip.clone()) else {
return Ok(());
};
if entries.iter().any(|e| e.vip != id) {
return Err(crate::vip::invalid("VIP snapshot mixes service identities"));
}
let own_seen = entries
.iter()
.filter(|e| e.author == *self.local_id.as_bytes())
.map(|e| e.version)
.max()
.unwrap_or(0);
let mut candidate = self.params.vips.clone();
let changed = candidate.merge(self.network_id, entries, &self.params.limits)?;
if !changed && own_seen <= self.own_version {
return Ok(());
}
let counter =
(own_seen > self.own_version).then_some((*self.local_id.as_bytes(), own_seen));
let snapshot = candidate.entries(&id);
self.validate_vip_frame(&snapshot)?;
self.params
.storage
.put_vip_fields(self.network_id, snapshot, counter)
.await?;
self.own_version = self.own_version.max(own_seen);
self.params.vips = candidate;
self.update_vips();
self.broadcast_vips();
Ok(())
}
fn validate_vip_frame(&self, entries: &[crate::vip::VipEntry]) -> Result<()> {
let frame = Envelope {
network_id: *self.network_id.as_bytes(),
message: ControlMessage::Vips {
entries: entries.to_vec(),
},
};
let bytes = encode(&frame).map_err(|e| crate::vip::invalid(e.to_string()))?;
if bytes.len() > self.params.limits.max_frame_len {
return Err(crate::vip::invalid(
"VIP snapshot exceeds the control frame limit",
));
}
Ok(())
}
fn broadcast_vips(&mut self) {
let ids = self.params.vips.ids();
if ids.is_empty() {
return;
}
let peers: Vec<_> = self.sessions.keys().copied().collect();
for peer in peers {
let start = self.vip_cursor.get(&peer).copied().unwrap_or(0) % ids.len();
for step in 0..ids.len() {
let index = (start + step) % ids.len();
let entries = self.params.vips.entries(&ids[index]);
if self
.send_to(peer, ControlMessage::Vips { entries })
.is_err()
{
self.vip_cursor.insert(peer, index);
break;
}
self.vip_cursor.insert(peer, (index + 1) % ids.len());
}
}
}
/// Handles snapshots collected while dispatching messages.
async fn drain_pending_state(&mut self) {
for entries in std::mem::take(&mut self.pending_vips) {
if let Err(err) = self.receive_vips(entries).await {
self.metrics.protocol_violations += 1;
tracing::warn!(%err,"rejecting VIP snapshot");
}
}
for (peer, markers) in std::mem::take(&mut self.pending_retirements) {
self.receive_retirements(peer, markers).await;
}
@@ -1473,6 +1691,7 @@ impl Runtime {
/// Sends everything we know to every peer.
fn broadcast_state(&mut self) {
self.broadcast_vips();
self.broadcast_retirements();
let mut records = self.state.records();
records.truncate(self.params.limits.max_state_records);
@@ -1671,6 +1890,7 @@ impl Runtime {
self.params.range_changed.send_replace(());
return;
}
self.update_vips();
for plugin in &self.params.plugins {
plugin.on_address_allocation(self.network_id, range, &allocations);
}
@@ -2154,6 +2374,7 @@ impl Runtime {
});
}
}
self.update_vips();
}
fn handle_link_result(&mut self, outcome: LinkOutcome) {
@@ -2304,6 +2525,7 @@ impl Runtime {
/// Drops every link to a peer.
fn drop_links_for(&mut self, peer: EndpointId) {
self.retirement_cursor.remove(&peer);
self.vip_cursor.remove(&peer);
self.reset_peer_data(peer, "peer session ended");
self.hub.remove_peer(peer);
self.reachable.remove(&peer);
@@ -2664,6 +2886,7 @@ impl Runtime {
tracing::debug!(%err, "could not queue pong");
}
}
ControlMessage::Vips { entries } => self.pending_vips.push(entries.clone()),
ControlMessage::State { records } => {
self.pending_state.push((peer, records.clone()));
}
@@ -2891,6 +3114,7 @@ impl Runtime {
members.dedup_by(|a, b| a.endpoint_id == b.endpoint_id);
NetworkStatus {
vips: self.params.routes.vips.statuses(self.network_id),
protocols: self
.params
.plugins
+2
View File
@@ -225,6 +225,8 @@ pub struct RangeConflict {
/// Status of one network.
#[derive(Debug, Clone)]
pub struct NetworkStatus {
/// Shared service configuration with this client's routing observations.
pub vips: Vec<crate::vip::VipStatus>,
/// Enabled local data-plane protocols, in preference order.
pub protocols: Vec<String>,
/// Whether this network is requested to start or resume automatically.
+24
View File
@@ -158,6 +158,22 @@ pub enum TransportPolicy {
/// object where that is possible (notably [`Limits::max_frame_len`]).
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Limits {
/// Maximum active VIP identities per network.
pub max_vips: usize,
/// Maximum permanent VIP deletion tombstones per network.
pub max_vip_tombstones: usize,
/// Maximum concurrent active TCP port checks across the agent.
pub max_vip_checks: usize,
/// Maximum signed fields per VIP, including removed fields.
pub max_vip_fields: usize,
/// Maximum listeners per VIP.
pub max_vip_listeners: usize,
/// Maximum backend endpoints per listener.
pub max_vip_backends: usize,
/// Maximum simultaneous local VIP flow bindings.
pub max_vip_flows: usize,
/// Maximum incomplete IPv4 datagrams per network in VIP reassembly.
pub max_vip_fragments: usize,
/// Largest accepted control frame payload, in bytes.
pub max_frame_len: usize,
/// Largest accepted hostname, in bytes.
@@ -205,6 +221,14 @@ pub struct Limits {
impl Default for Limits {
fn default() -> Self {
Self {
max_vips: 64,
max_vip_tombstones: 1024,
max_vip_checks: 32,
max_vip_fields: 192,
max_vip_listeners: 16,
max_vip_backends: 32,
max_vip_flows: 4096,
max_vip_fragments: 64,
max_frame_len: 64 * 1024,
max_hostname_len: 255,
max_capabilities: 16,
+90 -1
View File
@@ -171,6 +171,17 @@ pub fn default_control_socket_path() -> PathBuf {
#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)]
#[non_exhaustive]
pub enum Request {
/// Save a complete version-checked service draft or its deletion.
SaveVip {
/// Exact network id.
network_id: String,
/// Entire desired configuration.
config: crate::vip::VipConfig,
/// Revision originally displayed; absent for creation.
expected: Option<String>,
/// Remove the service permanently.
delete: bool,
},
/// Retire the selected offline member's observed claim.
ForgetNode {
/// Exact public network id.
@@ -271,6 +282,9 @@ pub enum Request {
impl std::fmt::Debug for Request {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
Request::SaveVip {
network_id, config, ..
} => write!(f, "SaveVip {{ {network_id}, {} }}", config.id),
Request::ForgetNode {
network_id,
peer,
@@ -317,6 +331,8 @@ impl std::fmt::Debug for Request {
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[non_exhaustive]
pub enum Response {
/// Accepted configuration with fresh local routing observations.
Vips(Vec<crate::vip::VipStatus>),
/// An observed offline claim was durably retired.
NodeForgotten(ForgottenNodeReport),
/// Accepted ordered protocol selection.
@@ -358,6 +374,57 @@ pub struct ForgottenNodeReport {
pub peers_told: u32,
}
/// Saves a whole VIP draft through the running agent.
pub async fn save_vip(
socket: &Path,
network_id: &str,
config: crate::vip::VipConfig,
expected: Option<String>,
delete: bool,
) -> Result<Vec<crate::vip::VipStatus>> {
match exchange(
socket,
&Request::SaveVip {
network_id: network_id.into(),
config,
expected,
delete,
},
EXCHANGE_TIMEOUT,
)
.await?
{
Response::Vips(vips) => Ok(vips),
Response::Error(reason) => Err(Error::Rejected { reason }),
_ => Err(Error::Rejected {
reason: "unexpected VIP response".into(),
}),
}
}
/// Resolves a service by exact name, full id or unique id prefix.
pub fn resolve_vip<'a>(
network: &'a NetworkReport,
selector: &str,
) -> std::result::Result<&'a crate::vip::VipStatus, String> {
let matches: Vec<_> = network
.vips
.iter()
.filter(|v| v.config.name == selector || v.config.id.starts_with(selector))
.collect();
match matches.as_slice() {
[vip] => Ok(vip),
[] => Err(format!("no VIP matches `{selector}`")),
_ => Err(format!(
"ambiguous VIP `{selector}`: {}",
matches
.iter()
.map(|v| format!("{} ({})", v.config.name, v.config.id))
.collect::<Vec<_>>()
.join(", ")
)),
}
}
/// Resolves a node by exact hostname, full id, unique id prefix or overlay IPv4.
/// Ambiguity is reported with names and full ids and never guessed.
pub fn resolve_node<'a>(
@@ -642,6 +709,9 @@ pub struct ProtocolReport {
/// One network.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct NetworkReport {
/// Shared VIPs and this client's routing observations.
#[serde(default)]
pub vips: Vec<crate::vip::VipStatus>,
/// Enabled local data-plane protocols, in preference order.
pub protocols: Vec<String>,
/// Whether the network starts or resumes automatically when its range is free.
@@ -905,6 +975,16 @@ impl OverlayPeerReport {
/// running and what they can report or change. That is what keeps this module
/// free of any knowledge of them.
pub trait ReportSource: Send + Sync + 'static {
/// Atomically saves a complete VIP draft using its original revision.
fn save_vip(
&self,
_network: String,
_config: crate::vip::VipConfig,
_expected: Option<String>,
_delete: bool,
) -> BoxFuture<'_, std::result::Result<Vec<crate::vip::VipStatus>, String>> {
Box::pin(async { Err("this agent cannot manage VIPs".into()) })
}
/// Persists and announces a version-scoped node cleanup.
fn forget_node(
&self,
@@ -1051,7 +1131,7 @@ pub const EXCHANGE_TIMEOUT: Duration = Duration::from_secs(5);
///
/// Bump it whenever [`Request`], [`Response`] or anything they contain
/// changes shape.
pub const CONTROL_PROTOCOL: u32 = u32::from_be_bytes([b'T', b'S', b'N', 21]);
pub const CONTROL_PROTOCOL: u32 = u32::from_be_bytes([b'T', b'S', b'N', 22]);
/// Reads one request off an accepted stream, answers it, writes the response.
///
@@ -1075,6 +1155,15 @@ where
}
};
let response = match request {
Request::SaveVip {
network_id,
config,
expected,
delete,
} => match source.save_vip(network_id, config, expected, delete).await {
Ok(vips) => Response::Vips(vips),
Err(reason) => Response::Error(reason),
},
Request::ForgetNode {
network_id,
peer,
+1
View File
@@ -51,6 +51,7 @@ pub mod storage;
pub mod task;
#[cfg(feature = "testing")]
pub mod testing;
pub mod vip;
pub use agent::{Agent, AgentStatus, Event, NetworkStatus, PeerStatus};
pub use config::{AgentConfig, Limits, ReconnectPolicy, StoragePaths, TransportPolicy};
+89 -1
View File
@@ -145,18 +145,19 @@ impl Tally {
}
/// The overlay interface.
#[derive(Debug)]
pub struct Interface {
device: Arc<dyn TunDevice>,
factory: Arc<dyn TunFactory>,
name: String,
mtu: u32,
routes: Arc<RoutingTable>,
carrier: Arc<dyn PacketCarrier>,
tally: Arc<Tally>,
/// What was last applied to the host, so an unchanged table is not
/// re-applied on every pass.
applied: std::sync::Mutex<Vec<Cidr>>,
task: std::sync::Mutex<Option<JoinHandle<()>>>,
vip_task: std::sync::Mutex<Option<JoinHandle<()>>>,
sync_lock: Arc<tokio::sync::Mutex<()>>,
/// The host rules for this interface, when the factory manages the host.
host_rules: Option<Arc<dyn BroadcastHostRules>>,
@@ -200,6 +201,14 @@ impl ExitState {
}
}
impl std::fmt::Debug for Interface {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("Interface")
.field("name", &self.name)
.finish_non_exhaustive()
}
}
impl Interface {
/// Creates the interface and starts moving packets.
pub async fn start(
@@ -221,7 +230,29 @@ impl Interface {
let name = device.name().to_string();
let tally = Arc::new(Tally::default());
let vip_task = {
let routes = Arc::clone(&routes);
let carrier = Arc::clone(&carrier);
tokio::spawn(async move {
let mut interval = tokio::time::interval(std::time::Duration::from_millis(250));
interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
let mut checks = tokio::task::JoinSet::new();
loop {
tokio::select! {
_=interval.tick()=>{
let (probes,jobs)=routes.vips.tick(routes.vips.check_capacity().saturating_sub(checks.len()));
for (route,packet) in probes {carrier.carry(route,packet);}
for job in jobs {checks.spawn(job.run());}
},
Some(result)=checks.join_next(),if !checks.is_empty()=>{
if let Ok((job,ok))=result {routes.vips.checked(job,ok);}
}
}
}
})
};
let task = {
let carrier = Arc::clone(&carrier);
let device = Arc::clone(&device);
let routes = Arc::clone(&routes);
let tally = Arc::clone(&tally);
@@ -231,6 +262,26 @@ impl Interface {
tally.malformed.fetch_add(1, Ordering::Relaxed);
continue;
};
match routes.vips.outgoing(packet.clone()) {
crate::vip::router::Decision::Send(route, packet) => {
for part in crate::vip::packet::fragments(packet, mtu) {
if carrier.carry(route, part) {
tally.sent.fetch_add(1, Ordering::Relaxed);
} else {
tally.undeliverable.fetch_add(1, Ordering::Relaxed);
}
}
continue;
}
crate::vip::router::Decision::Local(packet) => {
for part in crate::vip::packet::fragments(packet, mtu) {
let _ = device.send(part).await;
}
continue;
}
crate::vip::router::Decision::Drop => continue,
crate::vip::router::Decision::Pass => {}
}
let destination = header.destination();
if let (IpAddr::V4(source), IpAddr::V4(destination)) =
(header.source(), destination)
@@ -279,9 +330,11 @@ impl Interface {
name,
mtu,
routes,
carrier,
tally,
applied: std::sync::Mutex::new(Vec::new()),
task: std::sync::Mutex::new(Some(task)),
vip_task: std::sync::Mutex::new(Some(vip_task)),
sync_lock: Arc::new(tokio::sync::Mutex::new(())),
host_rules,
rules_report: std::sync::Mutex::new(None),
@@ -716,6 +769,15 @@ impl Interface {
task.abort();
let _ = task.await;
}
let vip_task = self
.vip_task
.lock()
.unwrap_or_else(|e| e.into_inner())
.take();
if let Some(task) = vip_task {
task.abort();
let _ = task.await;
}
self.factory.destroy(self.device.name()).await;
}
@@ -770,6 +832,24 @@ impl Interface {
return Err(Rejected::WrongDestination);
}
};
// Source ownership was checked before any VIP reverse translation.
let packet = match self.routes.vips.incoming(network, peer, packet.clone()) {
crate::vip::router::Decision::Send(route, packet) => {
// Probe replies use the ordinary peer carrier, never the TUN.
self.carrier.carry(route, packet);
return Ok(());
}
crate::vip::router::Decision::Local(packet) => {
for part in crate::vip::packet::fragments(packet, self.mtu) {
if self.device.send(part).await.is_ok() {
self.tally.received.fetch_add(1, Ordering::Relaxed);
}
}
return Ok(());
}
crate::vip::router::Decision::Drop => return Ok(()),
crate::vip::router::Decision::Pass => packet,
};
// Remote broadcasts terminate here. Only local TUN ingress can fan out.
if self.device.send(packet).await.is_ok() {
if broadcast {
@@ -794,6 +874,14 @@ impl Interface {
impl Drop for Interface {
fn drop(&mut self) {
if let Some(task) = self
.vip_task
.lock()
.unwrap_or_else(|e| e.into_inner())
.take()
{
task.abort();
}
if let Ok(mut guard) = self.task.lock()
&& let Some(task) = guard.take()
{
@@ -89,6 +89,7 @@ pub enum RouteError {
/// Address ownership across every network this agent is in.
#[derive(Debug, Default)]
pub struct RoutingTable {
pub(crate) vips: Arc<crate::vip::VipRouter>,
broadcast: ArcSwap<BroadcastTable>,
networks: RwLock<HashMap<NetworkId, NetworkRoutes>>,
}
@@ -146,6 +147,7 @@ impl RoutingTable {
/// Forgets a network.
pub fn remove_network(&self, network: NetworkId) {
self.vips.remove_network(network);
let mut networks = self.write();
networks.remove(&network);
self.broadcast
+17 -2
View File
@@ -20,7 +20,7 @@ use crate::error::ProtocolError;
/// The version in the ALPN is the wire-compatibility version of the control
/// protocol. It is independent of the network identity scheme version, so
/// bumping it must not change any existing [`crate::NetworkId`].
pub const ALPN: &[u8] = b"tsunagi/ctrl/7";
pub const ALPN: &[u8] = b"tsunagi/ctrl/8";
/// ALPN of the tsunagi data plane.
///
@@ -43,7 +43,7 @@ pub const MAX_DATA_PROTOCOL_LEN: usize = 32;
pub const MAX_SIGNATURE_LEN: usize = 64;
/// Control protocol version carried inside the handshake.
pub const PROTOCOL_VERSION: u16 = 7;
pub const PROTOCOL_VERSION: u16 = 8;
pub use crate::dataplane::transport::{MAX_TRANSPORT_PAYLOAD, TransportPayload};
@@ -181,6 +181,11 @@ pub enum ControlMessage {
/// Short free-text reason.
reason: String,
},
/// One complete signed VIP field snapshot, merged atomically.
Vips {
/// Signed registers bounded by the per-VIP field limit.
entries: Vec<crate::vip::VipEntry>,
},
/// Signed cleanup markers, each proving the retired claim version.
Retirements {
/// Bounded by the per-message signed-state limit.
@@ -278,6 +283,15 @@ pub fn validate_capability(
/// network, the other networks or the agent.
pub fn validate(message: &ControlMessage, limits: &Limits) -> Result<(), ProtocolError> {
match message {
ControlMessage::Vips { entries } => {
check_len("vip.fields", entries.len(), limits.max_vip_fields)?;
for e in entries {
check_len("vip.id", e.vip.len(), 32)?;
check_len("vip.key", e.key.len(), 160)?;
check_len("vip.value", e.value.as_ref().map_or(0, Vec::len), 256)?;
check_len("vip.signature", e.signature.len(), 64)?;
}
}
ControlMessage::Retirements { markers } => {
check_len("retirements", markers.len(), limits.max_state_records)?;
for marker in markers {
@@ -373,6 +387,7 @@ pub fn kind(message: &ControlMessage) -> &'static str {
ControlMessage::Pong { .. } => "pong",
ControlMessage::State { .. } => "state",
ControlMessage::Retirements { .. } => "retirements",
ControlMessage::Vips { .. } => "vips",
ControlMessage::Reachable { .. } => "reachable",
ControlMessage::Peers { .. } => "peers",
ControlMessage::Bye { .. } => "bye",
+2
View File
@@ -38,6 +38,8 @@
//! `docs/sync-model.md` for the implemented cleanup semantics.
pub mod allocator;
/// Signed shared virtual service state and its merge rules.
pub mod vip;
use std::collections::HashMap;
use std::net::Ipv4Addr;
+439
View File
@@ -0,0 +1,439 @@
//! Signed multi-writer virtual service registers and deterministic merge rules.
use crate::error::Result;
use crate::vip::{Backend, Listener, Protocol, VipConfig, VipStatus, invalid};
use crate::{Limits, NetworkId};
use iroh::{EndpointId, SecretKey, Signature};
use serde::{Deserialize, Serialize};
use sha2::{Digest, Sha256};
use std::collections::{BTreeMap, BTreeSet};
use std::net::Ipv4Addr;
/// A signed, independently mergeable field. A removed field keeps its tombstone.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct VipEntry {
/// Network scope.
pub network: [u8; 32],
/// Stable service id.
pub vip: String,
/// Canonical field key.
pub key: String,
/// Encoded field value; absent means deletion.
pub value: Option<Vec<u8>>,
/// Lamport clock, independent of wall time.
pub clock: u64,
/// Persistent signing identity.
pub author: [u8; 32],
/// Author's durable network counter.
pub version: u64,
/// Ed25519 signature.
pub signature: Vec<u8>,
}
impl VipEntry {
fn bytes(&self) -> Vec<u8> {
let mut e = self.clone();
e.signature.clear();
let mut out = b"tsunagi-vip-field/v1\0".to_vec();
if let Ok(bytes) = postcard::to_stdvec(&e) {
out.extend(bytes);
}
out
}
/// Verifies bounds, network and authorship before the entry is merged.
pub fn verify(&self, network: NetworkId) -> Result<()> {
if self.network != *network.as_bytes()
|| self.vip.len() != 32
|| hex::decode(&self.vip).is_err()
|| self.key.len() > 160
|| self.value.as_ref().is_some_and(|v| v.len() > 256)
|| self.version == 0
|| self.version > i64::MAX as u64
|| self.clock == 0
|| self.clock > i64::MAX as u64
{
return Err(invalid("invalid VIP field bounds or network"));
}
let author =
EndpointId::from_bytes(&self.author).map_err(|_| invalid("invalid VIP author"))?;
let raw: [u8; 64] = self
.signature
.as_slice()
.try_into()
.map_err(|_| invalid("invalid VIP signature length"))?;
author
.verify(&self.bytes(), &Signature::from_bytes(&raw))
.map_err(|_| invalid("invalid VIP signature"))?;
if self.key != "deleted"
&& self.key != "name"
&& !self.key.starts_with("l/")
&& !self.key.starts_with("b/")
{
return Err(invalid("unknown VIP field"));
}
match self.key.as_str() {
"deleted" => {
if self.value.as_deref() != Some(&[1][..]) {
return Err(invalid("invalid VIP deletion tombstone"));
}
}
"name" => {
let value = self
.value
.as_ref()
.ok_or_else(|| invalid("VIP name cannot be absent"))?;
let name: String =
postcard::from_bytes(value).map_err(|_| invalid("invalid VIP name field"))?;
VipConfig {
id: self.vip.clone(),
name,
listeners: vec![],
}
.validate(&Limits::default())?;
}
key if key.starts_with("l/") => {
let parts: Vec<_> = key.split('/').collect();
if parts.len() != 3
|| parts[1].parse::<Protocol>().is_err()
|| parts[2].parse::<u16>().ok().is_none_or(|p| p == 0)
{
return Err(invalid("invalid listener field key"));
}
if let Some(value) = &self.value {
let l: Listener = postcard::from_bytes(value)
.map_err(|_| invalid("invalid listener field"))?;
if !l.backends.is_empty() || key != format!("l/{}/{}", l.protocol, l.port) {
return Err(invalid("listener key does not match its value"));
}
VipConfig {
id: self.vip.clone(),
name: "validation".into(),
listeners: vec![l],
}
.validate(&Limits::default())?;
}
}
key if key.starts_with("b/") => {
let parts: Vec<_> = key.split('/').collect();
if parts.len() != 6
|| parts[1].parse::<Protocol>().is_err()
|| parts[2].parse::<u16>().ok().is_none_or(|p| p == 0)
|| parts[3].len() != 32
|| hex::decode(parts[3]).is_err()
|| parts[4].parse::<EndpointId>().is_err()
|| parts[5].parse::<u16>().ok().is_none_or(|p| p == 0)
{
return Err(invalid("invalid backend field key"));
}
if let Some(value) = &self.value {
let b: Backend = postcard::from_bytes(value)
.map_err(|_| invalid("invalid backend field"))?;
if b.peer != parts[4] || b.port.to_string() != parts[5] {
return Err(invalid("backend key does not match its value"));
}
}
}
_ => return Err(invalid("unknown VIP field")),
}
Ok(())
}
fn rank(&self) -> (u64, [u8; 32], Vec<u8>) {
(self.clock, self.author, self.bytes())
}
}
/// Signed registers retained across absences, including deletion tombstones.
#[derive(Debug, Clone, Default)]
pub struct VipSet {
entries: BTreeMap<(String, String), VipEntry>,
}
impl VipSet {
/// Highest retained version genuinely signed by this author.
pub fn authored_version(&self, author: EndpointId) -> u64 {
self.entries
.values()
.filter(|e| e.author == *author.as_bytes())
.map(|e| e.version)
.max()
.unwrap_or(0)
}
/// Complete independently verifiable state of one service, including tombstones.
pub fn entries(&self, id: &str) -> Vec<VipEntry> {
self.entries
.values()
.filter(|e| e.vip == id)
.cloned()
.collect()
}
/// All retained service ids, including deleted services.
pub fn ids(&self) -> Vec<String> {
self.entries
.keys()
.map(|(id, _)| id.clone())
.collect::<BTreeSet<_>>()
.into_iter()
.collect()
}
/// Opaque optimistic concurrency token for a whole service.
pub fn revision(&self, id: &str) -> String {
let mut hash = Sha256::new();
for e in self.entries(id) {
hash.update(e.bytes());
hash.update(&e.signature);
}
hex::encode(hash.finalize())
}
/// Atomically verifies and merges one complete service snapshot.
pub fn merge(
&mut self,
network: NetworkId,
entries: Vec<VipEntry>,
limits: &Limits,
) -> Result<bool> {
if entries.len() > limits.max_vip_fields {
return Err(invalid("too many VIP fields"));
}
let mut next = self.clone();
for e in entries {
e.verify(network)?;
let key = (e.vip.clone(), e.key.clone());
if next
.entries
.get(&key)
.is_none_or(|old| e.rank() > old.rank())
{
next.entries.insert(key, e);
}
}
let deleted: BTreeSet<_> = next
.entries
.values()
.filter(|e| e.key == "deleted" && e.value.is_some())
.map(|e| e.vip.clone())
.collect();
// A permanent service tombstone alone prevents every former field's replay.
next.entries
.retain(|(id, key), _| !deleted.contains(id) || key == "deleted");
let ids = next.ids();
if ids.len().saturating_sub(deleted.len()) > limits.max_vips
|| deleted.len() > limits.max_vip_tombstones
|| ids
.iter()
.any(|id| next.entries(id).len() > limits.max_vip_fields)
{
return Err(invalid(
"VIP state capacity exceeded; deletion protection is retained",
));
}
let changed = self.entries != next.entries;
*self = next;
Ok(changed)
}
/// Materialized configuration. A service tombstone permanently retires its id.
pub fn configs(&self) -> Vec<VipConfig> {
self.ids()
.into_iter()
.filter_map(|id| self.config(&id))
.collect()
}
fn config(&self, id: &str) -> Option<VipConfig> {
let fields = self.entries(id);
if fields
.iter()
.any(|e| e.key == "deleted" && e.value.is_some())
{
return None;
}
let name = fields
.iter()
.find(|e| e.key == "name")?
.value
.as_ref()
.and_then(|v| postcard::from_bytes::<String>(v).ok())?;
let mut listeners: Vec<Listener> = fields
.iter()
.filter(|e| e.key.starts_with("l/"))
.filter_map(|e| e.value.as_ref().and_then(|v| postcard::from_bytes(v).ok()))
.collect();
for l in &mut listeners {
l.backends = fields
.iter()
.filter(|e| {
e.key
.starts_with(&format!("b/{}/{}/{}/", l.protocol, l.port, l.id))
})
.filter_map(|e| e.value.as_ref().and_then(|v| postcard::from_bytes(v).ok()))
.collect();
l.backends.sort();
}
listeners.sort_by_key(|l| (l.protocol, l.port));
Some(VipConfig {
id: id.into(),
name,
listeners,
})
}
/// Builds a signed patch from the user's complete, version-checked draft.
pub fn edit(
&self,
network: NetworkId,
secret: &SecretKey,
version: u64,
config: &VipConfig,
expected: Option<&str>,
limits: &Limits,
) -> Result<Vec<VipEntry>> {
config.validate(limits)?;
if let Some(revision) = expected {
if revision != self.revision(&config.id) {
return Err(invalid("VIP changed while editing; reload before saving"));
}
} else if !self.entries(&config.id).is_empty() {
return Err(invalid("VIP id already exists"));
}
if self
.entries
.get(&(config.id.clone(), "deleted".into()))
.is_some_and(|e| e.value.is_some())
{
return Err(invalid("VIP was deleted; create a new VIP"));
}
let before = self
.config(&config.id)
.map(|c| flatten(&c))
.transpose()?
.unwrap_or_default();
let after = flatten(config)?;
let keys: BTreeSet<_> = before.keys().chain(after.keys()).cloned().collect();
let changes = keys
.into_iter()
.filter_map(|key| {
let value = after.get(&key).cloned();
(before.get(&key) != value.as_ref()).then_some((key, value))
})
.collect();
self.sign(network, secret, version, &config.id, changes)
}
/// Builds a permanent service tombstone after checking the selected revision.
pub fn delete(
&self,
network: NetworkId,
secret: &SecretKey,
version: u64,
id: &str,
expected: &str,
) -> Result<Vec<VipEntry>> {
if self.config(id).is_none() || self.revision(id) != expected {
return Err(invalid("VIP missing or changed while editing"));
}
self.sign(
network,
secret,
version,
id,
vec![("deleted".into(), Some(vec![1]))],
)
}
fn sign(
&self,
network: NetworkId,
secret: &SecretKey,
version: u64,
id: &str,
changes: Vec<(String, Option<Vec<u8>>)>,
) -> Result<Vec<VipEntry>> {
let clock = self
.entries
.values()
.map(|e| e.clock)
.max()
.unwrap_or(0)
.checked_add(1)
.filter(|n| *n <= i64::MAX as u64)
.ok_or_else(|| invalid("VIP clock exhausted"))?;
Ok(changes
.into_iter()
.map(|(key, value)| {
let mut e = VipEntry {
network: *network.as_bytes(),
vip: id.into(),
key,
value,
clock,
author: *secret.public().as_bytes(),
version,
signature: vec![],
};
e.signature = secret.sign(&e.bytes()).to_bytes().to_vec();
e
})
.collect())
}
}
pub(crate) fn flatten(cfg: &VipConfig) -> Result<BTreeMap<String, Vec<u8>>> {
let mut fields = BTreeMap::new();
fields.insert(
"name".into(),
postcard::to_stdvec(&cfg.name).map_err(|e| invalid(e.to_string()))?,
);
for l in &cfg.listeners {
let mut settings = l.clone();
settings.backends.clear();
fields.insert(
format!("l/{}/{}", l.protocol, l.port),
postcard::to_stdvec(&settings).map_err(|e| invalid(e.to_string()))?,
);
for b in &l.backends {
fields.insert(
format!("b/{}/{}/{}/{}/{}", l.protocol, l.port, l.id, b.peer, b.port),
postcard::to_stdvec(b).map_err(|e| invalid(e.to_string()))?,
);
}
}
Ok(fields)
}
impl VipSet {
/// Allocates internal addresses deterministically against durable node claims.
pub(crate) fn views(
&self,
range: Option<crate::state::Ipv4Range>,
mut taken: std::collections::HashSet<Ipv4Addr>,
hostnames: &BTreeSet<String>,
limits: &Limits,
) -> Vec<VipStatus> {
let mut names = hostnames.clone();
let mut result = Vec::new();
for config in self.configs() {
let mut error = config.validate(limits).err().map(|e| e.to_string());
if !names.insert(config.name.clone()) {
error = Some("name conflicts with another VIP or node".into());
}
let address = range.and_then(|r| {
let usable = crate::state::allocator::usable_addresses(r);
if usable == 0 {
return None;
}
let digest = Sha256::digest(config.id.as_bytes());
let seed = u64::from_be_bytes([
digest[0], digest[1], digest[2], digest[3], digest[4], digest[5], digest[6],
digest[7],
]);
let bits = 32 - u32::from(r.prefix_len);
let mask = if bits == 32 { 0 } else { u32::MAX << bits };
(0..usable).find_map(|step| {
let offset = (seed.wrapping_add(step) % usable + 1) as u32;
let ip = Ipv4Addr::from((u32::from(r.base) & mask) | offset);
taken.insert(ip).then_some(ip)
})
});
if address.is_none() {
error = Some("no free overlay address".into());
}
result.push(VipStatus {
revision: self.revision(&config.id),
config,
address,
error,
listeners: vec![],
});
}
result
}
}
+19
View File
@@ -370,6 +370,25 @@ impl Storage {
.await
}
/// Loads mandatory signed VIP fields with a strict capacity bound.
pub async fn vip_fields(
&self,
network: NetworkId,
max: usize,
) -> Result<Vec<crate::vip::VipEntry>> {
self.with_state(move |state| state.vip_fields(network, max))
.await
}
/// Stores a complete merged service snapshot and optional own counter.
pub async fn put_vip_fields(
&self,
network: NetworkId,
entries: Vec<crate::vip::VipEntry>,
own: Option<([u8; 32], u64)>,
) -> Result<()> {
self.with_state(move |state| state.put_vip_fields(network, &entries, own))
.await
}
/// Loads verified durable cleanup proofs off the executor.
pub async fn retirements(
&self,
+133 -1
View File
@@ -21,7 +21,7 @@ use crate::identity::{DeviceIdentity, NetworkId, NetworkName, NetworkSecret};
use crate::state::{MAX_RETIREMENT_LEN, RecordBody, SignedRecord, SignedRetirement};
/// Schema version written by this build.
pub const SCHEMA_VERSION: i64 = 9;
pub const SCHEMA_VERSION: i64 = 10;
/// Key of the stored hostname setting.
const SETTING_HOSTNAME: &str = "hostname";
@@ -263,6 +263,12 @@ impl StateStore {
)
.map_err(|err| self.corrupt(format!("cannot migrate schema to 9: {err}")))?;
}
if found < 10 {
self.conn.execute_batch("BEGIN; CREATE TABLE IF NOT EXISTS vip_fields (
network_id BLOB NOT NULL, vip TEXT NOT NULL, field TEXT NOT NULL, data BLOB NOT NULL,
PRIMARY KEY(network_id, vip, field)); PRAGMA user_version = 10; COMMIT;")
.map_err(|err| self.corrupt(format!("cannot migrate VIP state: {err}")))?;
}
Ok(())
}
@@ -276,6 +282,7 @@ impl StateStore {
"signed_records",
"peer_hostnames",
"retirements",
"vip_fields",
] {
let present: Option<String> = self
.conn
@@ -894,8 +901,77 @@ impl StateStore {
.map_err(|err| Error::Storage(format!("cannot commit retirement: {err}")))
}
/// Reads bounded mandatory VIP state; corrupt records fail activation.
pub fn vip_fields(&self, network: NetworkId, max: usize) -> Result<Vec<crate::vip::VipEntry>> {
let mut stmt = self.conn.prepare("SELECT CASE WHEN length(data) <= 1024 THEN data ELSE NULL END FROM vip_fields WHERE network_id=?1 ORDER BY vip, field")
.map_err(|e| self.corrupt(e.to_string()))?;
let mut rows = stmt
.query(params![network.as_bytes().as_slice()])
.map_err(|e| self.corrupt(e.to_string()))?;
let mut entries = Vec::new();
while let Some(row) = rows.next().map_err(|e| self.corrupt(e.to_string()))? {
if entries.len() >= max {
return Err(self.corrupt("stored VIP state exceeds configured limits"));
}
let bytes: Option<Vec<u8>> = row.get(0).map_err(|e| self.corrupt(e.to_string()))?;
let bytes = bytes.ok_or_else(|| self.corrupt("oversized VIP field"))?;
let entry: crate::vip::VipEntry =
postcard::from_bytes(&bytes).map_err(|e| self.corrupt(e.to_string()))?;
entry
.verify(network)
.map_err(|e| self.corrupt(e.to_string()))?;
entries.push(entry);
}
Ok(entries)
}
/// Commits a validated VIP snapshot and optional author counter atomically.
pub fn put_vip_fields(
&self,
network: NetworkId,
entries: &[crate::vip::VipEntry],
own: Option<([u8; 32], u64)>,
) -> Result<()> {
let tx = self
.conn
.unchecked_transaction()
.map_err(|e| Error::Storage(e.to_string()))?;
if let Some(first) = entries.first() {
if entries.iter().any(|e| e.vip != first.vip) {
return Err(Error::Storage(
"VIP snapshot mixes service identities".into(),
));
}
tx.execute(
"DELETE FROM vip_fields WHERE network_id=?1 AND vip=?2",
params![network.as_bytes().as_slice(), first.vip],
)
.map_err(|e| Error::Storage(e.to_string()))?;
}
for e in entries {
e.verify(network)?;
let bytes = postcard::to_stdvec(e).map_err(|e| Error::Storage(e.to_string()))?;
if bytes.len() > 1024 {
return Err(Error::Storage("oversized VIP field".into()));
}
tx.execute("INSERT INTO vip_fields(network_id,vip,field,data) VALUES(?1,?2,?3,?4) ON CONFLICT(network_id,vip,field) DO UPDATE SET data=excluded.data",
params![network.as_bytes().as_slice(),e.vip,e.key,bytes]).map_err(|e|Error::Storage(e.to_string()))?;
}
if let Some((author, version)) = own {
tx.execute("INSERT INTO own_record_version(network_id,author,version) VALUES(?1,?2,?3) ON CONFLICT(network_id,author) DO UPDATE SET version=max(version,excluded.version)",
params![network.as_bytes().as_slice(),author.as_slice(),version as i64]).map_err(|e|Error::Storage(e.to_string()))?;
}
tx.commit().map_err(|e| Error::Storage(e.to_string()))
}
/// Drops a forgotten network's local cleanup proofs.
pub fn forget_retirements(&self, network: NetworkId) -> Result<()> {
self.conn
.execute(
"DELETE FROM vip_fields WHERE network_id = ?1",
params![network.as_bytes().as_slice()],
)
.map_err(|e| Error::Storage(e.to_string()))?;
self.conn
.execute(
"DELETE FROM retirements WHERE network_id = ?1",
@@ -1333,3 +1409,59 @@ mod broadcast_storage_tests {
);
}
}
#[cfg(test)]
mod vip_tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
#[test]
fn vip_migration_and_counter_failure_preserve_identity_and_rollback_fields() {
let dir = tempfile::TempDir::new().unwrap();
let path = dir.path().join("state.sqlite");
let store = StateStore::open(&path).unwrap();
let identity = store.load_or_create_device_identity().unwrap();
store
.conn
.execute_batch("DROP TABLE vip_fields; PRAGMA user_version=9;")
.unwrap();
drop(store);
let store = StateStore::open(&path).unwrap();
assert_eq!(
store.device_identity().unwrap().unwrap().endpoint_id(),
identity.endpoint_id()
);
let net = NetworkId::from_bytes([7; 32]);
let key = identity.signing_key();
let config = crate::vip::VipConfig::new("api");
let fields = crate::vip::VipSet::default()
.edit(net, &key, 7, &config, None, &crate::Limits::default())
.unwrap();
store.conn.execute_batch("CREATE TRIGGER fail_vip_counter BEFORE INSERT ON own_record_version BEGIN SELECT RAISE(ABORT,'counter unavailable'); END;").unwrap();
assert!(
store
.put_vip_fields(net, &fields, Some((*key.public().as_bytes(), 7)))
.is_err()
);
assert!(store.vip_fields(net, 192).unwrap().is_empty());
store
.conn
.execute_batch("DROP TRIGGER fail_vip_counter;")
.unwrap();
store
.put_vip_fields(net, &fields, Some((*key.public().as_bytes(), 7)))
.unwrap();
assert_eq!(store.vip_fields(net, 192).unwrap(), fields);
assert_eq!(store.own_record_version(net, key.public()).unwrap(), 7);
drop(store);
let reopened = StateStore::open(path).unwrap();
assert_eq!(reopened.vip_fields(net, 192).unwrap(), fields);
reopened
.conn
.execute("UPDATE vip_fields SET data=?1", params![vec![0xffu8; 1025]])
.unwrap();
assert!(matches!(
reopened.vip_fields(net, 192),
Err(Error::StateCorrupted { .. })
));
}
}
+239
View File
@@ -0,0 +1,239 @@
//! Named, shared virtual services. Signed field registers converge without an
//! owner; routing, health and connection state remain local to each client.
use crate::Limits;
use crate::error::{Error, Result};
use iroh::EndpointId;
use serde::{Deserialize, Serialize};
use std::collections::BTreeSet;
use std::net::Ipv4Addr;
pub(crate) mod packet;
pub(crate) mod router;
pub use crate::state::vip::{VipEntry, VipSet};
pub(crate) use router::VipRouter;
#[cfg(test)]
mod tests;
/// IP transport carried by a listener.
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum Protocol {
/// TCP.
Tcp,
/// UDP.
Udp,
}
impl std::fmt::Display for Protocol {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str(match self {
Self::Tcp => "tcp",
Self::Udp => "udp",
})
}
}
impl std::str::FromStr for Protocol {
type Err = String;
fn from_str(s: &str) -> std::result::Result<Self, String> {
match s {
"tcp" => Ok(Self::Tcp),
"udp" => Ok(Self::Udp),
_ => Err("protocol must be tcp or udp".into()),
}
}
}
/// A node identity and its application port.
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct Backend {
/// Full device identity, resolved when editing.
pub peer: String,
/// Destination port.
pub port: u16,
}
/// One independently routed port of a VIP.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct Listener {
/// Stable listener generation; prevents removed backend fields resurfacing on re-creation.
pub id: String,
/// TCP or UDP.
pub protocol: Protocol,
/// Client-facing port.
pub port: u16,
/// Enable active TCP connect checks; UDP uses path checks and ICMP errors.
pub port_check: bool,
/// Period between checks in milliseconds.
pub check_interval_ms: u32,
/// Deadline of a port check in milliseconds.
pub check_timeout_ms: u32,
/// Consecutive failed checks before exclusion.
pub failure_threshold: u8,
/// Consecutive successes before recovery.
pub recovery_threshold: u8,
/// Idle expiry for this listener's flows, in seconds.
pub idle_timeout_secs: u32,
/// Allowed destination nodes and ports.
pub backends: Vec<Backend>,
}
impl Listener {
/// Creates a listener with automatic recovery and bounded checks.
pub fn new(protocol: Protocol, port: u16) -> Self {
Self {
id: hex::encode(rand::random::<[u8; 16]>()),
protocol,
port,
port_check: true,
check_interval_ms: 3000,
check_timeout_ms: 2000,
failure_threshold: 2,
recovery_threshold: 1,
idle_timeout_secs: if protocol == Protocol::Tcp { 7200 } else { 120 },
backends: vec![],
}
}
}
/// Entire editable service; its address is an internal derived reservation.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct VipConfig {
/// Stable random 128-bit id in hexadecimal.
pub id: String,
/// One DNS label, unique within the network.
pub name: String,
/// Independent listeners.
pub listeners: Vec<Listener>,
}
impl VipConfig {
/// Creates an empty named service with a fresh stable identity.
pub fn new(name: impl Into<String>) -> Self {
Self {
id: hex::encode(rand::random::<[u8; 16]>()),
name: name.into(),
listeners: vec![],
}
}
/// Validates a complete draft before any state is changed.
pub fn validate(&self, limits: &Limits) -> Result<()> {
if self.id.len() != 32 || hex::decode(&self.id).is_err() {
return Err(invalid("invalid VIP id"));
}
if self.name.is_empty()
|| self.name.len() > 63
|| self.name.starts_with('-')
|| self.name.ends_with('-')
|| !self
.name
.bytes()
.all(|b| b.is_ascii_lowercase() || b.is_ascii_digit() || b == b'-')
{
return Err(invalid(
"VIP name must be a lowercase DNS label (letters, digits and hyphens)",
));
}
if self.listeners.len() > limits.max_vip_listeners {
return Err(invalid("too many VIP listeners"));
}
let mut seen = BTreeSet::new();
for l in &self.listeners {
if l.id.len() != 32 || hex::decode(&l.id).is_err() {
return Err(invalid("invalid listener identity"));
}
if l.port == 0 || !seen.insert((l.protocol, l.port)) {
return Err(invalid(
"listener ports must be nonzero and unique per protocol",
));
}
if !(250..=3_600_000).contains(&l.check_interval_ms)
|| !(100..=60_000).contains(&l.check_timeout_ms)
|| l.failure_threshold == 0
|| l.recovery_threshold == 0
|| !(1..=604800).contains(&l.idle_timeout_secs)
{
return Err(invalid(
"invalid check interval, timeout, thresholds or flow idle timeout",
));
}
if l.backends.len() > limits.max_vip_backends {
return Err(invalid("too many listener backends"));
}
let mut backends = BTreeSet::new();
for b in &l.backends {
if b.port == 0 || b.peer.parse::<EndpointId>().is_err() || !backends.insert(b) {
return Err(invalid(
"backend must have a unique node identity and a nonzero port",
));
}
}
}
if crate::state::vip::flatten(self)?.len() > limits.max_vip_fields {
return Err(invalid("VIP configuration exceeds signed field capacity"));
}
Ok(())
}
}
pub(crate) fn invalid(reason: impl Into<String>) -> Error {
Error::Rejected {
reason: reason.into(),
}
}
/// Cumulative IP traffic, excluding probes. Reset on agent restart.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct Counters {
/// Outbound packets.
pub tx_packets: u64,
/// Outbound bytes.
pub tx_bytes: u64,
/// Inbound packets.
pub rx_packets: u64,
/// Inbound bytes.
pub rx_bytes: u64,
/// Created flow bindings.
pub flows: u64,
/// UDP failovers.
pub failovers: u64,
/// Packets rejected or lacking an eligible backend.
pub dropped: u64,
}
/// One backend, as seen by this client.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct BackendStatus {
/// Configured endpoint.
pub backend: Backend,
/// Eligibility and reason.
pub state: String,
/// Smoothed full data-plane RTT, microseconds; unknown stays absent.
pub rtt_us: Option<u64>,
/// Last successful path measurement age.
pub measurement_age_ms: Option<u64>,
/// Current bindings.
pub active_flows: u32,
/// Local traffic counters.
pub counters: Counters,
}
/// Runtime view of one listener.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct ListenerStatus {
/// Protocol.
pub protocol: Protocol,
/// Frontend port.
pub port: u16,
/// Best backend for a new flow, if measured and eligible.
pub selected: Option<Backend>,
/// Configured and draining endpoints.
pub backends: Vec<BackendStatus>,
/// Local counters.
pub counters: Counters,
}
/// Configuration and local diagnostics returned through the same API to CLI and GUI.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct VipStatus {
/// Persisted configuration.
pub config: VipConfig,
/// Opaque revision used when saving a draft.
pub revision: String,
/// Internal service address.
pub address: Option<Ipv4Addr>,
/// Conflict or allocation issue, if any.
pub error: Option<String>,
/// Local listener status.
pub listeners: Vec<ListenerStatus>,
}
+566
View File
@@ -0,0 +1,566 @@
//! Bounded IPv4 parsing, NAT checksums and fragment reassembly at local ingress.
use super::Protocol;
use bytes::Bytes;
use std::collections::HashMap;
use std::net::Ipv4Addr;
use std::time::{Duration, Instant};
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub(crate) struct Tuple {
pub src: Ipv4Addr,
pub dst: Ipv4Addr,
pub sport: u16,
pub dport: u16,
pub protocol: Protocol,
}
impl Tuple {
pub fn reverse(self) -> Self {
Self {
src: self.dst,
dst: self.src,
sport: self.dport,
dport: self.sport,
protocol: self.protocol,
}
}
}
pub(crate) fn header(p: &[u8]) -> Option<usize> {
if p.len() < 20 || p[0] >> 4 != 4 {
return None;
}
let n = usize::from(p[0] & 15) * 4;
let total = usize::from(u16::from_be_bytes([p[2], p[3]]));
(n >= 20 && n <= p.len() && total == p.len() && checksum(&p[..n]) == 0).then_some(n)
}
pub(crate) fn addresses(p: &[u8]) -> Option<(Ipv4Addr, Ipv4Addr)> {
header(p)?;
Some((
Ipv4Addr::new(p[12], p[13], p[14], p[15]),
Ipv4Addr::new(p[16], p[17], p[18], p[19]),
))
}
pub(crate) fn tuple(p: &[u8]) -> Option<Tuple> {
let n = header(p)?;
if u16::from_be_bytes([p[6], p[7]]) & 0x3fff != 0 {
return None;
}
let protocol = match p[9] {
6 => Protocol::Tcp,
17 => Protocol::Udp,
_ => return None,
};
let min = if protocol == Protocol::Tcp { 20 } else { 8 };
if p.len() < n + min {
return None;
}
if protocol == Protocol::Tcp {
let h = usize::from(p[n + 12] >> 4) * 4;
if h < 20 || n + h > p.len() {
return None;
}
} else if usize::from(u16::from_be_bytes([p[n + 4], p[n + 5]])) != p.len() - n {
return None;
}
let (src, dst) = addresses(p)?;
Some(Tuple {
src,
dst,
sport: u16::from_be_bytes([p[n], p[n + 1]]),
dport: u16::from_be_bytes([p[n + 2], p[n + 3]]),
protocol,
})
}
pub(crate) fn rewrite(p: &[u8], t: Tuple) -> Option<Bytes> {
let n = header(p)?;
if p.len() < n + if t.protocol == Protocol::Tcp { 20 } else { 8 } {
return None;
}
let mut out = p.to_vec();
out[12..16].copy_from_slice(&t.src.octets());
out[16..20].copy_from_slice(&t.dst.octets());
out[n..n + 2].copy_from_slice(&t.sport.to_be_bytes());
out[n + 2..n + 4].copy_from_slice(&t.dport.to_be_bytes());
finish(&mut out, n, t.protocol);
Some(Bytes::from(out))
}
fn finish(p: &mut [u8], n: usize, protocol: Protocol) {
p[10..12].fill(0);
let c = checksum(&p[..n]);
p[10..12].copy_from_slice(&c.to_be_bytes());
let offset = n + if protocol == Protocol::Tcp { 16 } else { 6 };
p[offset..offset + 2].fill(0);
let mut sum = 0u32;
for pair in p[12..20].as_chunks::<2>().0 {
sum += u32::from(u16::from_be_bytes([pair[0], pair[1]]));
}
sum += u32::from(p[9]) + (p.len() - n) as u32;
let c = checksum_seed(&p[n..], sum);
let c = if c == 0 && protocol == Protocol::Udp {
0xffff
} else {
c
};
p[offset..offset + 2].copy_from_slice(&c.to_be_bytes());
}
pub(crate) fn checksum(p: &[u8]) -> u16 {
checksum_seed(p, 0)
}
fn checksum_seed(p: &[u8], mut sum: u32) -> u16 {
for pair in p.chunks(2) {
sum += u32::from(u16::from_be_bytes([pair[0], *pair.get(1).unwrap_or(&0)]));
}
while sum >> 16 != 0 {
sum = (sum & 0xffff) + (sum >> 16);
}
!(sum as u16)
}
pub(crate) fn tcp_syn(p: &[u8]) -> bool {
header(p).is_some_and(|n| p[9] == 6 && p.get(n + 13).is_some_and(|f| f & 0x12 == 2))
}
pub(crate) fn syn_sequence(p: &[u8]) -> Option<u32> {
if !tcp_syn(p) {
return None;
}
let n = header(p)?;
let bytes = p.get(n + 4..n + 8)?;
Some(u32::from_be_bytes(bytes.try_into().ok()?))
}
pub(crate) fn empty_syn(p: &[u8]) -> bool {
header(p).is_some_and(|n| {
tcp_syn(p)
&& p.get(n + 12)
.is_some_and(|h| n + usize::from(h >> 4) * 4 == p.len())
})
}
pub(crate) fn syn_ack(p: &[u8]) -> bool {
header(p).is_some_and(|n| p[9] == 6 && p.get(n + 13).is_some_and(|f| f & 0x12 == 0x12))
}
pub(crate) fn tcp_reset(p: &[u8]) -> bool {
header(p).is_some_and(|n| p[9] == 6 && p.get(n + 13).is_some_and(|f| f & 4 != 0))
}
/// Rewrites an ICMP error and its quoted original header after NAT lookup.
pub(crate) fn icmp_quote(p: &[u8]) -> Option<Tuple> {
let n = header(p)?;
if p[9] != 1 || p.len() < n + 8 + 28 || ![3, 11, 12].contains(&p[n]) {
return None;
}
let q = &p[n + 8..];
let h = usize::from(q[0] & 15) * 4;
if q[0] >> 4 != 4 || h < 20 || q.len() < h + 8 {
return None;
}
let protocol = match q[9] {
6 => Protocol::Tcp,
17 => Protocol::Udp,
_ => return None,
};
Some(Tuple {
src: Ipv4Addr::new(q[12], q[13], q[14], q[15]),
dst: Ipv4Addr::new(q[16], q[17], q[18], q[19]),
sport: u16::from_be_bytes([q[h], q[h + 1]]),
dport: u16::from_be_bytes([q[h + 2], q[h + 3]]),
protocol,
})
}
pub(crate) fn rewrite_icmp(p: &[u8], original: Tuple) -> Option<Bytes> {
let n = header(p)?;
icmp_quote(p)?;
let mut out = p.to_vec();
out[12..16].copy_from_slice(&original.dst.octets());
out[16..20].copy_from_slice(&original.src.octets());
let q = &mut out[n + 8..];
let h = usize::from(q[0] & 15) * 4;
// Incrementally update the quoted transport checksum when it is present.
let off = h + if original.protocol == Protocol::Tcp {
16
} else {
6
};
if q.len() >= off + 2 {
let old = u16::from_be_bytes([q[off], q[off + 1]]);
if old != 0 || original.protocol == Protocol::Tcp {
let mut sum = u32::from(!old);
let mut replacement = Vec::from(original.src.octets());
replacement.extend(original.dst.octets());
replacement.extend(original.sport.to_be_bytes());
replacement.extend(original.dport.to_be_bytes());
let previous = [&q[12..20], &q[h..h + 4]].concat();
for (a, b) in previous
.as_chunks::<2>()
.0
.iter()
.zip(replacement.as_chunks::<2>().0)
{
sum += u32::from(!u16::from_be_bytes([a[0], a[1]]))
+ u32::from(u16::from_be_bytes([b[0], b[1]]));
}
while sum >> 16 != 0 {
sum = (sum & 0xffff) + (sum >> 16);
}
let c = !(sum as u16);
q[off..off + 2].copy_from_slice(
&(if c == 0 && original.protocol == Protocol::Udp {
0xffff
} else {
c
})
.to_be_bytes(),
);
}
}
q[12..16].copy_from_slice(&original.src.octets());
q[16..20].copy_from_slice(&original.dst.octets());
q[h..h + 2].copy_from_slice(&original.sport.to_be_bytes());
q[h + 2..h + 4].copy_from_slice(&original.dport.to_be_bytes());
q[10..12].fill(0);
let c = checksum(&q[..h]);
q[10..12].copy_from_slice(&c.to_be_bytes());
out[n + 2..n + 4].fill(0);
let c = checksum(&out[n..]);
out[n + 2..n + 4].copy_from_slice(&c.to_be_bytes());
out[10..12].fill(0);
let c = checksum(&out[..n]);
out[10..12].copy_from_slice(&c.to_be_bytes());
Some(Bytes::from(out))
}
pub(crate) fn reject(p: &[u8]) -> Option<Bytes> {
let t = tuple(p)?;
let n = header(p)?;
if t.protocol == Protocol::Tcp {
if !tcp_syn(p) {
return None;
}
let mut out = vec![0; 40];
out[0] = 0x45;
out[2..4].copy_from_slice(&40u16.to_be_bytes());
out[8] = 64;
out[9] = 6;
out[12..16].copy_from_slice(&t.dst.octets());
out[16..20].copy_from_slice(&t.src.octets());
out[20..22].copy_from_slice(&t.dport.to_be_bytes());
out[22..24].copy_from_slice(&t.sport.to_be_bytes());
let seq = u32::from_be_bytes(p[n + 4..n + 8].try_into().ok()?);
out[28..32].copy_from_slice(&seq.wrapping_add(1).to_be_bytes());
out[32] = 0x50;
out[33] = 0x14;
finish(&mut out, 20, Protocol::Tcp);
Some(Bytes::from(out))
} else {
let q = p.len().min(n + 8);
let mut out = vec![0; 28 + q];
out[0] = 0x45;
let len = out.len() as u16;
out[2..4].copy_from_slice(&len.to_be_bytes());
out[8] = 64;
out[9] = 1;
out[12..16].copy_from_slice(&t.dst.octets());
out[16..20].copy_from_slice(&t.src.octets());
out[20] = 3;
out[21] = 3;
out[28..].copy_from_slice(&p[..q]);
let c = checksum(&out[20..]);
out[22..24].copy_from_slice(&c.to_be_bytes());
let c = checksum(&out[..20]);
out[10..12].copy_from_slice(&c.to_be_bytes());
Some(Bytes::from(out))
}
}
#[derive(Debug)]
struct Datagram {
at: Instant,
header: Option<Vec<u8>>,
parts: std::collections::BTreeMap<usize, Vec<u8>>,
end: Option<usize>,
bytes: usize,
}
type FragmentKey = (Option<[u8; 32]>, Ipv4Addr, Ipv4Addr, u16, u8);
#[derive(Debug, Default)]
pub(crate) struct Reassembly {
pending: HashMap<FragmentKey, Datagram>,
}
impl Reassembly {
pub fn expire(&mut self, now: Instant) {
self.pending
.retain(|_, d| now.duration_since(d.at) < Duration::from_secs(5));
}
pub fn accept(&mut self, p: Bytes, max: usize, scope: Option<[u8; 32]>) -> Option<Bytes> {
let n = header(&p)?;
let flags = u16::from_be_bytes([p[6], p[7]]);
if flags & 0x3fff == 0 {
return Some(p);
}
let now = Instant::now();
self.pending
.retain(|_, d| now.duration_since(d.at) < Duration::from_secs(5));
let (src, dst) = addresses(&p)?;
let key = (scope, src, dst, u16::from_be_bytes([p[4], p[5]]), p[9]);
let start = usize::from(flags & 0x1fff) * 8;
let payload = &p[n..];
let end = start.checked_add(payload.len())?;
if payload.is_empty()
|| end > 65515
|| flags & 0x2000 != 0 && !payload.len().is_multiple_of(8)
{
self.pending.remove(&key);
return None;
}
if !self.pending.contains_key(&key) && self.pending.len() >= max {
return None;
}
let d = self.pending.entry(key).or_insert_with(|| Datagram {
at: now,
header: None,
parts: Default::default(),
end: None,
bytes: 0,
});
if d.parts
.get(&start)
.is_some_and(|bytes| bytes.as_slice() == payload)
{
return None;
}
if d.parts.iter().any(|(s, b)| *s < end && start < s + b.len()) {
self.pending.remove(&key);
return None;
}
if flags & 0x2000 == 0 {
if d.end.is_some_and(|e| e != end) {
self.pending.remove(&key);
return None;
}
d.end = Some(end);
}
if d.end.is_some_and(|e| end > e) {
self.pending.remove(&key);
return None;
}
if start == 0 {
d.header = Some(p[..n].to_vec());
}
d.bytes += payload.len();
d.parts.insert(start, payload.to_vec());
let complete = d.end.is_some_and(|e| e == d.bytes) && d.header.is_some();
if !complete {
return None;
}
let d = self.pending.remove(&key)?;
let mut out = d.header?;
for part in d.parts.values() {
out.extend(part);
}
if out.len() > 65535 {
return None;
}
let len = out.len() as u16;
out[2..4].copy_from_slice(&len.to_be_bytes());
out[6..8].fill(0);
out[10..12].fill(0);
let h = usize::from(out[0] & 15) * 4;
let c = checksum(&out[..h]);
out[10..12].copy_from_slice(&c.to_be_bytes());
Some(Bytes::from(out))
}
}
/// Re-fragments an assembled NAT datagram for the unchanged overlay MTU.
pub(crate) fn fragments(packet: Bytes, mtu: u32) -> Vec<Bytes> {
if packet.len() <= mtu as usize {
return vec![packet];
}
let Some(n) = header(&packet) else {
return vec![];
};
if u16::from_be_bytes([packet[6], packet[7]]) & 0x4000 != 0 {
return vec![];
}
let size = (mtu as usize).saturating_sub(n) / 8 * 8;
if size == 0 {
return vec![];
}
let mut copied = Vec::new();
let mut offset = 20;
while offset < n {
let kind = packet[offset];
if kind == 0 {
break;
}
if kind == 1 {
offset += 1;
continue;
}
let Some(&len) = packet.get(offset + 1) else {
return vec![];
};
let len = usize::from(len);
if len < 2 || offset + len > n {
return vec![];
}
if kind & 0x80 != 0 {
copied.extend_from_slice(&packet[offset..offset + len]);
}
offset += len;
}
while !copied.len().is_multiple_of(4) {
copied.push(0);
}
let mut result = Vec::new();
let payload = &packet[n..];
for (start, part) in payload.chunks(size).enumerate() {
let offset = start * size;
let mut out = if offset == 0 {
packet[..n].to_vec()
} else {
let mut h = packet[..20].to_vec();
h.extend_from_slice(&copied);
h[0] = 0x40 | (h.len() / 4) as u8;
h
};
let header_len = out.len();
out.extend(part);
let len = out.len() as u16;
out[2..4].copy_from_slice(&len.to_be_bytes());
let flags = (offset / 8) as u16
| if offset + part.len() < payload.len() {
0x2000
} else {
0
};
out[6..8].copy_from_slice(&flags.to_be_bytes());
out[10..12].fill(0);
let c = checksum(&out[..header_len]);
out[10..12].copy_from_slice(&c.to_be_bytes());
result.push(Bytes::from(out));
}
result
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
fn datagram() -> Bytes {
let mut p = vec![0; 3028];
p[0] = 0x45;
p[2..4].copy_from_slice(&3028u16.to_be_bytes());
p[8] = 64;
p[9] = 17;
p[12..16].copy_from_slice(&[10, 0, 0, 1]);
p[16..20].copy_from_slice(&[10, 0, 0, 2]);
p[20..22].copy_from_slice(&1000u16.to_be_bytes());
p[22..24].copy_from_slice(&2000u16.to_be_bytes());
p[24..26].copy_from_slice(&3008u16.to_be_bytes());
finish(&mut p, 20, Protocol::Udp);
Bytes::from(p)
}
#[test]
fn fragmented_udp_reassembles_out_of_order_and_nat_checksums_are_valid() {
let p = datagram();
let parts = fragments(p.clone(), 1280);
assert_eq!(parts.len(), 3);
let mut reassembly = Reassembly::default();
let mut result = None;
for part in parts.into_iter().rev() {
if let Some(p) = reassembly.accept(part, 8, None) {
result = Some(p);
}
}
assert_eq!(result, Some(p.clone()));
let mut t = tuple(&p).unwrap();
t.dst = Ipv4Addr::new(10, 0, 0, 240);
t.dport = 53;
let out = rewrite(&p, t).unwrap();
assert_eq!(header(&out), Some(20));
let mut sum = 0u32;
for pair in out[12..20].as_chunks::<2>().0 {
sum += u32::from(u16::from_be_bytes([pair[0], pair[1]]));
}
sum += 17 + (out.len() - 20) as u32;
assert_eq!(checksum_seed(&out[20..], sum), 0);
}
#[test]
fn overlaps_and_capacity_are_rejected_without_allocating_unbounded_state() {
let parts = fragments(datagram(), 1280);
let mut r = Reassembly::default();
assert!(r.accept(parts[0].clone(), 0, None).is_none());
assert!(r.pending.is_empty());
assert!(r.accept(parts[0].clone(), 1, None).is_none());
assert!(r.accept(parts[0].clone(), 1, None).is_none());
assert_eq!(r.pending.len(), 1);
let mut conflict = parts[0].to_vec();
conflict[20] ^= 1;
assert!(r.accept(Bytes::from(conflict), 1, None).is_none());
assert!(r.pending.is_empty());
for size in 0..80 {
let p = vec![0xff; size];
assert!(tuple(&p).is_none());
assert!(icmp_quote(&p).is_none());
}
}
#[test]
fn fragment_assembly_is_scoped_to_authenticated_sender_and_expires_when_idle() {
let packet = datagram();
let parts = fragments(packet.clone(), 1280);
let mut r = Reassembly::default();
assert!(r.accept(parts[0].clone(), 8, Some([1; 32])).is_none());
assert!(r.accept(parts[1].clone(), 8, Some([2; 32])).is_none());
assert!(r.accept(parts[2].clone(), 8, Some([1; 32])).is_none());
assert_eq!(r.accept(parts[1].clone(), 8, Some([1; 32])), Some(packet));
assert_eq!(r.pending.len(), 1);
r.expire(Instant::now() + Duration::from_secs(6));
assert!(r.pending.is_empty());
}
}
#[cfg(test)]
mod icmp_tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
#[test]
fn pmtu_errors_restore_quoted_addresses_ports_and_checksums() {
let original = Tuple {
src: Ipv4Addr::new(10, 0, 0, 1),
dst: Ipv4Addr::new(10, 0, 0, 240),
sport: 40000,
dport: 53,
protocol: Protocol::Udp,
};
let mut p = vec![0; 36];
p[0] = 0x45;
p[2..4].copy_from_slice(&36u16.to_be_bytes());
p[8] = 64;
p[9] = 17;
p[24..26].copy_from_slice(&16u16.to_be_bytes());
let c = checksum(&p[..20]);
p[10..12].copy_from_slice(&c.to_be_bytes());
let p = rewrite(&p, original).unwrap();
let wire = Tuple {
dst: Ipv4Addr::new(10, 0, 0, 2),
sport: 50000,
dport: 5353,
..original
};
let translated = rewrite(&p, wire).unwrap();
let mut error = vec![0; 56];
error[0] = 0x45;
error[2..4].copy_from_slice(&56u16.to_be_bytes());
error[8] = 64;
error[9] = 1;
error[12..16].copy_from_slice(&wire.dst.octets());
error[16..20].copy_from_slice(&wire.src.octets());
error[20] = 3;
error[21] = 4;
error[26..28].copy_from_slice(&1200u16.to_be_bytes());
error[28..].copy_from_slice(&translated[..28]);
let c = checksum(&error[20..]);
error[22..24].copy_from_slice(&c.to_be_bytes());
let c = checksum(&error[..20]);
error[10..12].copy_from_slice(&c.to_be_bytes());
let restored = rewrite_icmp(&error, original).unwrap();
assert_eq!(icmp_quote(&restored), Some(original));
assert_eq!(checksum(&restored[20..]), 0);
assert_eq!(checksum(&restored[28..48]), 0);
assert_eq!(&restored[54..56], &p[26..28]);
assert_eq!(&restored[26..28], &1200u16.to_be_bytes());
assert_eq!(&restored[12..16], &original.dst.octets());
}
}
File diff suppressed because it is too large. Load diff
+144
View File
@@ -0,0 +1,144 @@
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
use crate::NetworkId;
#[test]
fn concurrent_backend_edits_merge_and_deletion_cannot_resurrect() {
let net = NetworkId::from_bytes([7; 32]);
let a = iroh::SecretKey::generate();
let b = iroh::SecretKey::generate();
let limits = Limits::default();
let mut state = VipSet::default();
let id = hex::encode([9; 16]);
let initial = VipConfig {
id: id.clone(),
name: "api".into(),
listeners: vec![Listener::new(Protocol::Tcp, 443)],
};
let changes = state.edit(net, &a, 1, &initial, None, &limits).unwrap();
state.merge(net, changes, &limits).unwrap();
let revision = state.revision(&id);
let mut first = initial.clone();
first.listeners[0].backends.push(Backend {
peer: a.public().to_string(),
port: 6443,
});
let mut second = initial.clone();
second.listeners[0].backends.push(Backend {
peer: b.public().to_string(),
port: 7443,
});
let x = state
.edit(net, &a, 2, &first, Some(&revision), &limits)
.unwrap();
let y = state
.edit(net, &b, 1, &second, Some(&revision), &limits)
.unwrap();
let mut reverse = state.clone();
state.merge(net, x.clone(), &limits).unwrap();
state.merge(net, y.clone(), &limits).unwrap();
reverse.merge(net, y, &limits).unwrap();
reverse.merge(net, x, &limits).unwrap();
assert_eq!(state.configs(), reverse.configs());
assert_eq!(state.configs()[0].listeners[0].backends.len(), 2);
assert!(
state
.edit(net, &a, 3, &initial, Some(&revision), &limits)
.is_err()
);
let removed = state.delete(net, &a, 3, &id, &state.revision(&id)).unwrap();
let old = state.entries(&id);
state.merge(net, removed, &limits).unwrap();
state.merge(net, old, &limits).unwrap();
assert!(state.configs().is_empty());
}
#[test]
fn signed_fields_reject_tampering_wrong_network_and_limits() {
let net = NetworkId::from_bytes([2; 32]);
let key = iroh::SecretKey::generate();
let limits = Limits::default();
let state = VipSet::default();
let cfg = VipConfig {
id: hex::encode([3; 16]),
name: "dns".into(),
..Default::default()
};
let entries = state.edit(net, &key, 1, &cfg, None, &limits).unwrap();
let mut bad = entries.clone();
bad[0].clock += 1;
assert!(VipSet::default().merge(net, bad, &limits).is_err());
assert!(
VipSet::default()
.merge(NetworkId::from_bytes([4; 32]), entries.clone(), &limits)
.is_err()
);
let mut small = limits.clone();
small.max_vips = 0;
assert!(VipSet::default().merge(net, entries, &small).is_err());
}
#[test]
fn deletion_history_does_not_exhaust_active_capacity_or_accept_stale_fields() {
let network = NetworkId::from_bytes([9; 32]);
let key = iroh::SecretKey::generate();
let limits = Limits {
max_vips: 1,
max_vip_tombstones: 4,
..Default::default()
};
let mut state = VipSet::default();
for index in 0..4 {
let config = VipConfig::new("reused-name");
let fields = state
.edit(network, &key, index * 2 + 1, &config, None, &limits)
.unwrap();
state.merge(network, fields.clone(), &limits).unwrap();
let deletion = state
.delete(
network,
&key,
index * 2 + 2,
&config.id,
&state.revision(&config.id),
)
.unwrap();
state.merge(network, deletion, &limits).unwrap();
assert_eq!(state.entries(&config.id).len(), 1);
assert!(!state.merge(network, fields, &limits).unwrap());
}
let next = VipConfig::new("next");
let fields = state.edit(network, &key, 9, &next, None, &limits).unwrap();
state.merge(network, fields, &limits).unwrap();
assert_eq!(state.configs().len(), 1);
}
#[test]
fn reservations_avoid_nodes_and_name_conflicts_have_a_deterministic_holder() {
let network = NetworkId::from_bytes([4; 32]);
let key = iroh::SecretKey::generate();
let limits = Limits::default();
let mut state = VipSet::default();
for version in 1..=2 {
let config = VipConfig::new("shared-name");
let fields = state
.edit(network, &key, version, &config, None, &limits)
.unwrap();
state.merge(network, fields, &limits).unwrap();
}
let range = "10.0.0.0/29".parse().unwrap();
let taken = ["10.0.0.1".parse().unwrap()].into_iter().collect();
let views = state.views(Some(range), taken, &Default::default(), &limits);
assert!(views[0].error.is_none());
assert!(
views[1]
.error
.as_deref()
.unwrap()
.contains("name conflicts")
);
assert_ne!(views[0].address, views[1].address);
for v in views {
assert_ne!(v.address, Some("10.0.0.1".parse().unwrap()));
}
}
+207
View File
@@ -0,0 +1,207 @@
//! Shared services use real membership, signed state and mandatory SQLite.
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use tsunagi::discovery::SharedMemoryDiscovery;
use tsunagi::testing::{TestAgent, config_with, network, wait_for_peers, wait_until};
use tsunagi::vip::{Backend, Listener, Protocol, VipConfig, VipStatus};
use tsunagi::{Agent, NetworkId};
async fn view(agent: &Agent, network: NetworkId, id: &str) -> VipStatus {
wait_until("VIP configuration reaches the replica", || async {
agent
.network_status(network)
.await
.ok()?
.vips
.into_iter()
.find(|v| v.config.id == id)
})
.await
}
#[tokio::test]
async fn any_member_can_edit_services_and_an_offline_creator_is_not_required() {
let discovery = SharedMemoryDiscovery::new();
let a = TestAgent::spawn(&discovery).await.unwrap();
let b = TestAgent::spawn(&discovery).await.unwrap();
let (name, secret) = network("shared-vips");
let net = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
wait_for_peers(&a.agent, net, 1).await;
let cfg = VipConfig::new("api");
a.agent
.save_vip(net, cfg.clone(), None, false)
.await
.unwrap();
let before = view(&b.agent, net, &cfg.id).await;
let mut edited = before.config.clone();
let mut listener = Listener::new(Protocol::Tcp, 443);
listener.backends.push(Backend {
peer: b.agent.endpoint_id().to_string(),
port: 6443,
});
edited.listeners.push(listener);
b.agent
.save_vip(net, edited.clone(), Some(before.revision.clone()), false)
.await
.unwrap();
assert!(
b.agent
.save_vip(net, cfg.clone(), Some(before.revision), false)
.await
.is_err()
);
wait_until("creator observes another member's edit", || async {
(view(&a.agent, net, &cfg.id).await.config == edited).then_some(())
})
.await;
let a_dir = a.stop().await;
let c = TestAgent::spawn(&discovery).await.unwrap();
c.agent.join_network(&name, &secret).await.unwrap();
assert_eq!(view(&c.agent, net, &cfg.id).await.config, edited);
let a = Agent::spawn(config_with(a_dir.path(), &discovery))
.await
.unwrap();
assert_eq!(view(&a, net, &cfg.id).await.config, edited);
let current = view(&c.agent, net, &cfg.id).await;
c.agent
.save_vip(net, current.config, Some(current.revision), true)
.await
.unwrap();
wait_until("deletion reaches the original creator", || async {
a.network_status(net)
.await
.ok()?
.vips
.is_empty()
.then_some(())
})
.await;
a.shutdown().await;
b.stop().await;
c.stop().await;
}
#[tokio::test]
async fn invalid_drafts_are_atomic_and_services_do_not_cross_networks() {
let discovery = SharedMemoryDiscovery::new();
let a = TestAgent::spawn(&discovery).await.unwrap();
let (name, secret) = network("vip-one");
let net = a.agent.join_network(&name, &secret).await.unwrap();
let (other, secret) = network("vip-two");
let second = a.agent.join_network(&other, &secret).await.unwrap();
let mut cfg = VipConfig::new("api");
cfg.listeners.push(Listener::new(Protocol::Udp, 53));
a.agent
.save_vip(net, cfg.clone(), None, false)
.await
.unwrap();
let before = view(&a.agent, net, &cfg.id).await;
let mut invalid = before.config.clone();
invalid.name = "renamed".into();
invalid.listeners.push(invalid.listeners[0].clone());
assert!(
a.agent
.save_vip(net, invalid, Some(before.revision.clone()), false)
.await
.is_err()
);
let after = view(&a.agent, net, &cfg.id).await;
assert_eq!(after.config, before.config);
assert_eq!(after.revision, before.revision);
assert!(
a.agent
.network_status(second)
.await
.unwrap()
.vips
.is_empty()
);
a.stop().await;
}
#[tokio::test]
async fn a_restored_author_advances_its_counter_past_its_own_updates_held_by_a_replica() {
use tsunagi::config::StoragePaths;
use tsunagi::storage::StateStore;
use tsunagi::vip::VipSet;
let discovery = SharedMemoryDiscovery::new();
let a = TestAgent::spawn(&discovery).await.unwrap();
let b = TestAgent::spawn(&discovery).await.unwrap();
let (name, secret) = network("vip-author-recovery");
let net = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
wait_for_peers(&a.agent, net, 1).await;
let config = VipConfig::new("api");
a.agent
.save_vip(net, config.clone(), None, false)
.await
.unwrap();
view(&b.agent, net, &config.id).await;
let author = a.agent.endpoint_id();
let adir = a.stop().await;
let bdir = b.stop().await;
let apath = StoragePaths::under(adir.path()).state_db();
let bpath = StoragePaths::under(bdir.path()).state_db();
// Model an old author backup with a newer, genuinely signed update retained
// by another offline replica. Connections and recovery below remain real.
tokio::task::spawn_blocking(move || {
let a = StateStore::open(apath).unwrap();
let b = StateStore::open(bpath).unwrap();
let mut set = VipSet::default();
set.merge(
net,
a.vip_fields(net, 192).unwrap(),
&tsunagi::Limits::default(),
)
.unwrap();
let mut cfg = set.configs().remove(0);
cfg.name = "newer".into();
let key =
iroh::SecretKey::from_bytes(&a.device_identity().unwrap().unwrap().secret_bytes());
let fields = set
.edit(
net,
&key,
100,
&cfg,
Some(&set.revision(&cfg.id)),
&tsunagi::Limits::default(),
)
.unwrap();
set.merge(net, fields, &tsunagi::Limits::default()).unwrap();
b.put_vip_fields(net, &set.entries(&cfg.id), None).unwrap();
})
.await
.unwrap();
let b = Agent::spawn(config_with(bdir.path(), &discovery))
.await
.unwrap();
let a = Agent::spawn(config_with(adir.path(), &discovery))
.await
.unwrap();
let learned = wait_until(
"the restored author learns its newer signed service",
|| async {
let v = view(&a, net, &config.id).await;
(v.config.name == "newer").then_some(v)
},
)
.await;
let mut next = learned.config;
next.name = "latest".into();
a.save_vip(net, next, Some(learned.revision), false)
.await
.unwrap();
a.shutdown().await;
let path = StoragePaths::under(adir.path()).state_db();
let version = tokio::task::spawn_blocking(move || {
StateStore::open(path)
.unwrap()
.own_record_version(net, author)
.unwrap()
})
.await
.unwrap();
assert!(
version > 100,
"the observed author counter must not be reused"
);
b.shutdown().await;
}
+41
View File
@@ -21,6 +21,13 @@ const POLL_INTERVAL: Duration = Duration::from_secs(2);
/// A request from the UI to the agent.
pub(crate) enum Command {
SaveVip {
token: String,
network_id: String,
config: tsunagi::vip::VipConfig,
expected: Option<String>,
delete: bool,
},
/// Retire the exact offline-node claim selected in the devices window.
ForgetNode {
network_id: String,
@@ -55,8 +62,14 @@ pub(crate) enum Command {
}
/// The latest the worker knows, read by the UI each frame.
#[derive(Clone)]
pub(crate) struct VipSaveResult {
pub token: String,
pub result: Result<Vec<tsunagi::vip::VipStatus>, String>,
}
#[derive(Clone, Default)]
pub(crate) struct Snapshot {
pub vip_result: Option<VipSaveResult>,
/// The last status, or the reason it could not be read (agent down, no
/// permission on the socket, …).
pub status: Option<Result<StatusReport, String>>,
@@ -84,6 +97,18 @@ pub(crate) struct AgentClient {
}
impl AgentClient {
#[cfg(test)]
pub(crate) fn test_client() -> (Self, mpsc::UnboundedReceiver<Command>) {
let (tx, rx) = mpsc::unbounded_channel();
(
Self {
tx,
shared: Arc::new(Mutex::new(Snapshot::default())),
pending_copy: Arc::new(Mutex::new(None)),
},
rx,
)
}
/// Starts the worker on `runtime`, reporting to `ctx` (for repaints).
pub(crate) fn spawn(runtime: &tokio::runtime::Handle, ctx: egui::Context) -> Self {
let (tx, rx) = mpsc::unbounded_channel();
@@ -198,6 +223,22 @@ async fn run(
ctx.request_repaint();
let outcome: Result<String, String> = match command {
Command::SaveVip {
token,
network_id,
config,
expected,
delete,
} => {
let result = ipc::save_vip(socket, &network_id, config, expected, delete)
.await
.map_err(|e| e.to_string());
lock(shared).vip_result = Some(VipSaveResult {
token,
result: result.clone(),
});
result.map(|_| "VIP configuration saved".into())
}
Command::ForgetNode {
network_id,
peer,
+46
View File
@@ -64,6 +64,7 @@ impl eframe::App for App {
ui::draw(ctx, &self.agent, &mut self.state, &snapshot, &self.traffic);
self.show_devices_windows(ctx, &snapshot);
self.show_vip_windows(ctx, &snapshot);
self.show_about_window(ctx, &snapshot);
// A secret the worker fetched for the clipboard: copy it here, on the
@@ -78,6 +79,51 @@ impl eframe::App for App {
}
impl App {
fn show_vip_windows(&mut self, ctx: &egui::Context, snapshot: &crate::agent::Snapshot) {
let Some(Ok(report)) = &snapshot.status else {
return;
};
let ids: Vec<_> = self.state.vip_windows.keys().cloned().collect();
for id in ids {
let Some(network) = report.networks.iter().find(|n| n.network_id == id) else {
self.state.vip_windows.remove(&id);
continue;
};
let Some(state) = self.state.vip_windows.get_mut(&id) else {
continue;
};
let builder = egui::ViewportBuilder::default()
.with_title(format!("tsunagi · VIP · {}", network.name))
.with_app_id(crate::icon::APP_ID)
.with_icon(crate::icon::window())
.with_inner_size([1000., 720.])
.with_min_inner_size([650., 400.]);
let keep = ctx.show_viewport_immediate(
egui::ViewportId::from_hash_of(("vip", &id)),
builder,
|vctx, _| {
let close = crate::vip::show(vctx, &self.agent, state, network, snapshot);
if close {
return false;
}
if vctx.input(|i| i.viewport().close_requested()) {
if state.dirty() {
state.request_close();
vctx.send_viewport_cmd(egui::ViewportCommand::CancelClose);
true
} else {
false
}
} else {
true
}
},
);
if !keep {
self.state.vip_windows.remove(&id);
}
}
}
/// Renders the About window while it is open.
fn show_about_window(&mut self, ctx: &egui::Context, snapshot: &crate::agent::Snapshot) {
if !self.state.about_open {
+1
View File
@@ -19,6 +19,7 @@ mod icon;
mod stats;
mod tray;
mod ui;
mod vip;
mod watch;
use eframe::egui;
+1 -1
View File
@@ -34,7 +34,7 @@ pub(crate) struct Series {
}
impl Series {
fn sample(&mut self, now: Instant, packets: u64, bytes: u64) {
pub(crate) fn sample(&mut self, now: Instant, packets: u64, bytes: u64) {
if let Some((then, prev_packets, prev_bytes)) = self.last {
let dt = now.saturating_duration_since(then).as_secs_f32();
if dt > 0.0 {
+3
View File
@@ -41,6 +41,7 @@ pub(crate) struct UiState {
pub(crate) host_edit: Option<String>,
/// Network ids whose devices window is open.
pub(crate) open_devices: BTreeSet<String>,
pub(crate) vip_windows: std::collections::BTreeMap<String, crate::vip::Window>,
/// Whether the About window is open.
pub(crate) about_open: bool,
/// An exit-node choice waiting for the user to confirm it.
@@ -56,6 +57,7 @@ impl Default for UiState {
unit: Unit::Packets,
host_edit: None,
open_devices: BTreeSet::new(),
vip_windows: Default::default(),
about_open: false,
exit_confirm: None,
forget_confirm: None,
@@ -211,6 +213,7 @@ fn draw_network(
egui::Frame::group(ui.style()).show(ui, |ui| {
ui.horizontal(|ui| {
format::copy_field(ui, network.name.as_str(), &network.name);
if ui.button("VIP").clicked(){state.vip_windows.entry(network.network_id.clone()).or_default();}
ui.with_layout(egui::Layout::right_to_left(egui::Align::Center), |ui| {
let leave =
egui::Button::new(egui::RichText::new("Leave").color(egui::Color32::WHITE))
+461
View File
@@ -0,0 +1,461 @@
//! Per-network VIP editor. Drafts never issue commands until Save is pressed.
use crate::agent::{AgentClient, Command, Snapshot};
use crate::stats::{Series, Unit};
use eframe::egui;
use std::collections::BTreeMap;
use tsunagi::ipc::NetworkReport;
use tsunagi::vip::{Backend, Listener, Protocol, VipConfig, VipStatus};
#[derive(Clone)]
struct Draft {
config: VipConfig,
original: VipConfig,
revision: Option<String>,
delete: bool,
}
impl Draft {
fn new(config: VipConfig, revision: Option<String>) -> Self {
Self {
original: config.clone(),
config,
revision,
delete: false,
}
}
fn dirty(&self) -> bool {
self.revision.is_none() || self.delete || self.config != self.original
}
}
#[derive(Default)]
pub(crate) struct Window {
draft: Option<Draft>,
pending: Option<String>,
error: Option<String>,
close_question: bool,
close_after_save: bool,
rates: BTreeMap<String, (Series, Series)>,
generation: u64,
acknowledged_generation: Option<u64>,
}
impl Window {
pub fn request_close(&mut self) {
self.close_question = true;
self.close_after_save = true;
}
pub fn dirty(&self) -> bool {
self.draft.as_ref().is_some_and(Draft::dirty) || self.pending.is_some()
}
}
fn peer_name<'a>(n: &'a NetworkReport, id: &'a str) -> &'a str {
n.members
.iter()
.find(|m| m.endpoint_id == id)
.and_then(|m| m.hostname.as_deref())
.unwrap_or(id)
}
pub(crate) fn show(
ctx: &egui::Context,
agent: &AgentClient,
state: &mut Window,
n: &NetworkReport,
snapshot: &Snapshot,
) -> bool {
if state.generation != snapshot.generation {
state.generation = snapshot.generation;
if let Some(at) = snapshot.at {
for v in &n.vips {
let counters = v.listeners.iter().fold((0, 0, 0, 0), |a, l| {
(
a.0 + l.counters.tx_packets,
a.1 + l.counters.tx_bytes,
a.2 + l.counters.rx_packets,
a.3 + l.counters.rx_bytes,
)
});
let (tx, rx) = state.rates.entry(v.config.id.clone()).or_default();
tx.sample(at, counters.0, counters.1);
rx.sample(at, counters.2, counters.3);
}
}
}
if let Some(result) = &snapshot.vip_result
&& state.pending.as_deref() == Some(result.token.as_str())
{
state.pending = None;
state.acknowledged_generation = Some(snapshot.generation);
match &result.result {
Ok(vips) => {
if let Some(draft) = &state.draft {
state.draft = vips
.iter()
.find(|v| v.config.id == draft.config.id)
.map(|v| Draft::new(v.config.clone(), Some(v.revision.clone())));
}
state.error = None;
}
Err(error) => {
state.error = Some(error.clone());
state.close_after_save = false;
}
}
}
let mut close = state.close_after_save && !state.dirty();
let mut discard_new = false;
egui::CentralPanel::default().show(ctx, |ui| {
ui.heading(format!("VIP · {}", n.name));
ui.weak("Configuration is shared with every member. Changes take effect only after Save.");
if state.close_question {
ui.horizontal(|ui| {
ui.label("Unsaved changes.");
if ui
.add_enabled(
state.pending.is_none(),
egui::Button::new("Discard and close"),
)
.clicked()
{
state.draft = None;
state.close_question = false;
close = true;
}
if ui.button("Keep editing").clicked() {
state.close_question = false;
state.close_after_save = false;
}
});
}
if let Some(error) = &state.error {
ui.colored_label(egui::Color32::LIGHT_RED, error);
}
ui.separator();
ui.horizontal_wrapped(|ui| {
for vip in &n.vips {
let selected = state
.draft
.as_ref()
.is_some_and(|d| d.config.id == vip.config.id);
if ui.selectable_label(selected, &vip.config.name).clicked() {
if state.dirty() {
state.error = Some(
"Save or discard the current draft before selecting another VIP."
.into(),
);
} else {
state.draft =
Some(Draft::new(vip.config.clone(), Some(vip.revision.clone())));
state.error = None;
}
}
}
if ui.button("+ New VIP").clicked() {
if state.dirty() {
state.error = Some("Save or discard the current draft first.".into());
} else {
state.draft = Some(Draft::new(VipConfig::new(""), None));
state.error = None;
}
}
});
let Some(draft) = &mut state.draft else {
ui.label("Select a VIP or create one.");
return;
};
let current = n.vips.iter().find(|v| v.config.id == draft.config.id);
if state.pending.is_none()
&& state.acknowledged_generation != Some(snapshot.generation)
&& !draft.dirty()
&& let Some(current) = current
&& draft.revision.as_ref() != Some(&current.revision)
{
*draft = Draft::new(current.config.clone(), Some(current.revision.clone()));
}
let changed =
current.is_some_and(|v| draft.revision.as_ref().is_some_and(|r| r != &v.revision));
if changed {
ui.colored_label(
egui::Color32::YELLOW,
"This VIP changed on the network. Reload before saving; your draft is retained.",
);
}
if draft.revision.is_some() && current.is_none() {
ui.colored_label(
egui::Color32::YELLOW,
"This VIP was removed on the network. The draft cannot overwrite its deletion.",
);
}
let busy = state.pending.is_some();
ui.horizontal(|ui| {
let valid = draft.config.validate(&tsunagi::Limits::default());
if !n.active {
ui.weak("Start the network to save changes.");
}
if ui
.add_enabled(
n.active
&& !busy
&& draft.dirty()
&& !changed
&& (valid.is_ok() || draft.delete),
egui::Button::new("Save"),
)
.clicked()
{
let token = VipConfig::new("").id;
state.pending = Some(token.clone());
agent.send(Command::SaveVip {
token,
network_id: n.network_id.clone(),
config: draft.config.clone(),
expected: draft.revision.clone(),
delete: draft.delete,
});
}
if ui
.add_enabled(!busy, egui::Button::new("Discard / reload"))
.clicked()
{
if let Some(v) = current {
*draft = Draft::new(v.config.clone(), Some(v.revision.clone()));
} else {
discard_new = true;
}
state.error = None;
}
if busy {
ui.spinner();
ui.label("Saving…");
} else if draft.dirty() {
ui.weak("Unsaved changes");
}
if let Err(error) = valid {
ui.colored_label(egui::Color32::LIGHT_RED, error.to_string());
}
});
ui.separator();
egui::ScrollArea::vertical().show(ui, |ui| {
ui.add_enabled_ui(!busy, |ui| {
ui.horizontal(|ui| {
ui.label("Name");
ui.add(
egui::TextEdit::singleline(&mut draft.config.name)
.id(egui::Id::new(("vip-name", &draft.config.id))),
);
ui.weak(format!(".{}", n.name));
});
if draft.revision.is_some() {
ui.checkbox(&mut draft.delete, "Delete this VIP when saved");
}
if !draft.delete {
edit_listeners(ui, &mut draft.config, n);
}
});
if let Some(current) = current {
ui.separator();
status(ui, current, n, state.rates.get(&current.config.id));
}
});
});
if discard_new {
state.draft = None;
state.error = None;
}
close
}
fn edit_listeners(ui: &mut egui::Ui, config: &mut VipConfig, n: &NetworkReport) {
let mut remove = None;
for (index, l) in config.listeners.iter_mut().enumerate() {
ui.push_id(index,|ui|{egui::Frame::group(ui.style()).show(ui,|ui|{
ui.horizontal(|ui|{egui::ComboBox::from_id_salt("protocol").selected_text(l.protocol.to_string()).show_ui(ui,|ui|{ui.selectable_value(&mut l.protocol,Protocol::Tcp,"TCP");ui.selectable_value(&mut l.protocol,Protocol::Udp,"UDP");});ui.label("Port");ui.add(egui::DragValue::new(&mut l.port).range(1..=65535));if ui.button("Remove listener").clicked(){remove=Some(index);}});
if l.protocol==Protocol::Tcp{ui.checkbox(&mut l.port_check,"Active TCP port checks");}else{ui.weak("UDP health: encrypted path probes and explicit port errors; application readiness is unknown.");}
ui.collapsing("Checks and flow expiry",|ui|{egui::Grid::new("settings").show(ui,|ui|{
ui.label("Check interval, ms");ui.add(egui::DragValue::new(&mut l.check_interval_ms).range(250..=3_600_000));ui.end_row();
ui.label("Check timeout, ms");ui.add(egui::DragValue::new(&mut l.check_timeout_ms).range(100..=60_000));ui.end_row();
ui.label("Failures before exclusion");ui.add(egui::DragValue::new(&mut l.failure_threshold).range(1..=255));ui.end_row();
ui.label("Successes before recovery");ui.add(egui::DragValue::new(&mut l.recovery_threshold).range(1..=255));ui.end_row();
ui.label("Flow idle timeout, seconds");ui.add(egui::DragValue::new(&mut l.idle_timeout_secs).range(1..=604800));ui.end_row();
});});
let mut remove_backend=None;for(i,b)in l.backends.iter_mut().enumerate(){ui.push_id(i,|ui|{ui.horizontal(|ui|{
egui::ComboBox::from_id_salt("node").width(240.).selected_text(peer_name(n,&b.peer)).show_ui(ui,|ui|{for m in &n.members{let label=format!("{} ({})",m.hostname.as_deref().unwrap_or("unnamed"),m.endpoint_id.chars().take(8).collect::<String>());ui.selectable_value(&mut b.peer,m.endpoint_id.clone(),label);}});
ui.label("Target port");ui.add(egui::DragValue::new(&mut b.port).range(1..=65535));if ui.button("Remove").clicked(){remove_backend=Some(i);}
});});}if let Some(i)=remove_backend{l.backends.remove(i);}
if ui.button("+ Backend").clicked(){l.backends.push(Backend{peer:n.members.first().map(|m|m.endpoint_id.clone()).unwrap_or_default(),port:l.port});}
});});
}
if let Some(index) = remove {
config.listeners.remove(index);
}
if ui.button("+ Listener").clicked() {
config.listeners.push(Listener::new(Protocol::Tcp, 443));
}
}
fn status(ui: &mut egui::Ui, v: &VipStatus, n: &NetworkReport, rates: Option<&(Series, Series)>) {
ui.heading("Status · this device");
if let Some(error) = &v.error {
ui.colored_label(egui::Color32::LIGHT_RED, error);
}
if let Some((tx, rx)) = rates {
ui.label(format!(
"TX {:.1} KiB/s · RX {:.1} KiB/s",
tx.rate(Unit::Bytes) / 1024.,
rx.rate(Unit::Bytes) / 1024.
));
}
for l in &v.listeners {
ui.strong(format!("{}/{}", l.protocol, l.port));
let selected = l
.selected
.as_ref()
.map(|b| format!("{}:{}", peer_name(n, &b.peer), b.port))
.unwrap_or_else(|| "none".into());
ui.label(format!("Next new flow: {selected}"));
ui.label(format!(
"TX {} bytes / {} packets · RX {} bytes / {} packets · drops {} · UDP failovers {}",
l.counters.tx_bytes,
l.counters.tx_packets,
l.counters.rx_bytes,
l.counters.rx_packets,
l.counters.dropped,
l.counters.failovers
));
for b in &l.backends {
let rtt = b
.rtt_us
.map(|v| format!("{:.2} ms", v as f64 / 1000.))
.unwrap_or_else(|| "unknown".into());
let age = b
.measurement_age_ms
.map(|v| format!("{:.1}s", v as f64 / 1000.))
.unwrap_or_else(|| "unknown".into());
ui.label(format!(
"{}:{} · {} · RTT {} (age {}) · {} active · TX/RX {}/{} bytes",
peer_name(n, &b.backend.peer),
b.backend.port,
b.state,
rtt,
age,
b.active_flows,
b.counters.tx_bytes,
b.counters.rx_bytes
));
}
}
ui.collapsing("Internal address and identity", |ui| {
ui.monospace(&v.config.id);
if let Some(address) = v.address {
ui.monospace(address.to_string());
}
});
}
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use super::*;
#[test]
fn editing_the_actual_form_sends_nothing_until_save_is_clicked() {
let (agent, mut commands) = AgentClient::test_client();
let ctx = egui::Context::default();
let mut config = VipConfig::new("api");
config.listeners.push(Listener::new(Protocol::Tcp, 443));
let mut state = Window {
draft: Some(Draft::new(config.clone(), Some("revision".into()))),
..Default::default()
};
let network = NetworkReport {
active: true,
name: "lab".into(),
vips: vec![VipStatus {
config: config.clone(),
revision: "revision".into(),
address: None,
error: None,
listeners: vec![],
}],
..Default::default()
};
let snapshot = Snapshot::default();
let frame = |events: Vec<egui::Event>, state: &mut Window| {
ctx.run(
egui::RawInput {
screen_rect: Some(egui::Rect::from_min_size(
egui::Pos2::ZERO,
egui::vec2(1000., 800.),
)),
events,
..Default::default()
},
|ctx| {
show(ctx, &agent, state, &network, &snapshot);
},
)
};
frame(vec![], &mut state);
ctx.memory_mut(|m| m.request_focus(egui::Id::new(("vip-name", &config.id))));
frame(vec![egui::Event::Text("new".into())], &mut state);
assert!(state.dirty());
assert_ne!(state.draft.as_ref().unwrap().config.name, config.name);
assert!(commands.try_recv().is_err());
assert_eq!(network.vips[0].config, config);
fn save_position(shape: &egui::Shape) -> Option<egui::Pos2> {
match shape {
egui::Shape::Text(text) if text.galley.text() == "Save" => {
Some(text.pos + text.galley.rect.center().to_vec2())
}
egui::Shape::Vec(shapes) => shapes.iter().find_map(save_position),
_ => None,
}
}
// Save was disabled before this edit; allow the next layout to register
// its now-enabled hit target before dispatching the pointer press.
let ready = frame(vec![], &mut state);
let position = ready
.shapes
.iter()
.find_map(|s| save_position(&s.shape))
.unwrap();
frame(
vec![
egui::Event::PointerMoved(position),
egui::Event::PointerButton {
pos: position,
button: egui::PointerButton::Primary,
pressed: true,
modifiers: egui::Modifiers::NONE,
},
],
&mut state,
);
frame(
vec![egui::Event::PointerButton {
pos: position,
button: egui::PointerButton::Primary,
pressed: false,
modifiers: egui::Modifiers::NONE,
}],
&mut state,
);
let Command::SaveVip {
config: saved,
expected,
delete,
..
} = commands.try_recv().unwrap()
else {
panic!("Save must submit one complete VIP draft");
};
assert_ne!(saved.name, config.name);
assert_eq!(expected.as_deref(), Some("revision"));
assert!(!delete);
assert!(commands.try_recv().is_err());
}
#[test]
fn all_listener_settings_render_without_a_live_agent() {
let ctx = egui::Context::default();
let mut cfg = VipConfig::new("api");
cfg.listeners.push(Listener::new(Protocol::Tcp, 443));
cfg.listeners.push(Listener::new(Protocol::Udp, 53));
let n = NetworkReport::default();
let _ = ctx.run(egui::RawInput::default(), |ctx| {
egui::CentralPanel::default().show(ctx, |ui| edit_listeners(ui, &mut cfg, &n));
});
assert_eq!(cfg.listeners.len(), 2);
}
}
+185
View File
@@ -1770,3 +1770,188 @@ async fn lan_broadcast_fanout_is_scoped_opt_in_and_does_not_reflood() {
agent.shutdown().await;
}
}
fn vip_segment(
source: Ipv4Addr,
destination: Ipv4Addr,
sport: u16,
dport: u16,
udp: bool,
flags: u8,
) -> Bytes {
fn sum(bytes: &[u8]) -> u16 {
let mut s: u32 = bytes
.chunks(2)
.map(|b| u32::from(u16::from_be_bytes([b[0], *b.get(1).unwrap_or(&0)])))
.sum();
while s >> 16 != 0 {
s = (s & 0xffff) + (s >> 16);
}
!(s as u16)
}
let mut p = vec![0; if udp { 36 } else { 40 }];
p[0] = 0x45;
let len = p.len() as u16;
p[2..4].copy_from_slice(&len.to_be_bytes());
p[8] = 64;
p[9] = if udp { 17 } else { 6 };
p[12..16].copy_from_slice(&source.octets());
p[16..20].copy_from_slice(&destination.octets());
p[20..22].copy_from_slice(&sport.to_be_bytes());
p[22..24].copy_from_slice(&dport.to_be_bytes());
if udp {
p[24..26].copy_from_slice(&16u16.to_be_bytes());
} else {
p[32] = 0x50;
p[33] = flags;
p[24..28].copy_from_slice(&123u32.to_be_bytes());
}
let mut pseudo = p[12..20].to_vec();
pseudo.extend([0, p[9]]);
pseudo.extend(((p.len() - 20) as u16).to_be_bytes());
pseudo.extend(&p[20..]);
let c = sum(&pseudo);
let offset = if udp { 26 } else { 36 };
p[offset..offset + 2].copy_from_slice(&c.to_be_bytes());
let c = sum(&p[..20]);
p[10..12].copy_from_slice(&c.to_be_bytes());
Bytes::from(p)
}
#[tokio::test]
async fn vip_tcp_and_udp_remap_ports_over_real_encryption_and_preserve_source_admission() {
use tsunagi::vip::{Backend, Listener, Protocol, VipConfig};
let discovery = SharedMemoryDiscovery::new();
let (name, secret) = network("vip-encrypted");
let a = WgAgent::spawn(&discovery, "vipa").await;
let b = WgAgent::spawn(&discovery, "vipb").await;
let net = a.agent.join_network(&name, &secret).await.unwrap();
b.agent.join_network(&name, &secret).await.unwrap();
a.wait_for_tunnels(net, 1).await;
b.wait_for_tunnels(net, 1).await;
let aa = a.overlay(net).await;
let ab = b.overlay(net).await;
let ta = a.tun(net).await;
let tb = b.tun(net).await;
let mut cfg = VipConfig::new("service");
for (protocol, port, backend) in [(Protocol::Tcp, 443, 6443), (Protocol::Udp, 53, 5353)] {
let mut l = Listener::new(protocol, port);
l.port_check = false;
l.backends.push(Backend {
peer: b.endpoint_id().to_string(),
port: backend,
});
cfg.listeners.push(l);
}
a.agent
.save_vip(net, cfg.clone(), None, false)
.await
.unwrap();
let vip = wait_until("data-plane probes measure the backend RTT", || async {
let status = a.agent.network_status(net).await.ok()?;
let v = status.vips.iter().find(|v| v.config.id == cfg.id)?;
(v.listeners.iter().all(|l| l.selected.is_some()))
.then_some(v.address?)
.filter(|_| v.listeners.len() == 2)
})
.await;
for (udp, port, target) in [(false, 443, 6443), (true, 53, 5353)] {
ta.push_from_os(vip_segment(aa, vip, 40000, port, udp, 2));
let received = tokio::time::timeout(Duration::from_secs(5), tb.pop_to_os())
.await
.unwrap()
.unwrap();
assert_eq!(&received[12..16], &aa.octets());
assert_eq!(&received[16..20], &ab.octets());
assert_eq!(u16::from_be_bytes([received[22], received[23]]), target);
let translated = u16::from_be_bytes([received[20], received[21]]);
tb.push_from_os(vip_segment(ab, aa, target, translated, udp, 0x12));
let reply = tokio::time::timeout(Duration::from_secs(5), ta.pop_to_os())
.await
.unwrap()
.unwrap();
assert_eq!(&reply[12..16], &vip.octets());
assert_eq!(u16::from_be_bytes([reply[20], reply[21]]), port);
assert_eq!(u16::from_be_bytes([reply[22], reply[23]]), 40000);
// A backend cannot send a VIP source over its authenticated tunnel.
let forged = vip_segment(vip, aa, port, 40000, udp, 0x12);
tb.push_from_os(forged);
}
wait_until("forged VIP sources are rejected before NAT", || async {
(a.agent.overlay()?.counters.wrong_source >= 2).then_some(())
})
.await;
a.shutdown().await;
b.shutdown().await;
}
#[tokio::test]
async fn vip_udp_reaches_another_backend_after_the_selected_agent_stops() {
use tsunagi::vip::{Backend, Listener, Protocol, VipConfig};
let discovery = SharedMemoryDiscovery::new();
let (name, secret) = network("vip-failover");
let a = WgAgent::spawn(&discovery, "vipfa").await;
let b = WgAgent::spawn(&discovery, "vipfb").await;
let c = WgAgent::spawn(&discovery, "vipfc").await;
let net = a.agent.join_network(&name, &secret).await.unwrap();
for node in [&b, &c] {
node.agent.join_network(&name, &secret).await.unwrap();
}
for node in [&a, &b, &c] {
node.wait_for_tunnels(net, 2).await;
}
let mut config = VipConfig::new("datagrams");
let mut listener = Listener::new(Protocol::Udp, 8888);
listener.backends = vec![
Backend {
peer: b.endpoint_id().to_string(),
port: 9000,
},
Backend {
peer: c.endpoint_id().to_string(),
port: 9001,
},
];
config.listeners.push(listener);
a.agent
.save_vip(net, config.clone(), None, false)
.await
.unwrap();
let first = wait_until("both UDP backends have measured paths", || async {
let s = a.agent.network_status(net).await.ok()?;
let v = s.vips.iter().find(|v| v.config.id == config.id)?;
let l = v.listeners.first()?;
(l.backends.iter().all(|b| b.rtt_us.is_some())).then_some((v.address?, l.selected.clone()?))
})
.await;
let (dead, alive) = if first.1.peer == b.endpoint_id().to_string() {
(b, c)
} else {
(c, b)
};
let address = a.overlay(net).await;
let tun = a.tun(net).await;
let dest = dead.tun(net).await;
tun.push_from_os(vip_segment(address, first.0, 41000, 8888, true, 0));
tokio::time::timeout(Duration::from_secs(5), dest.pop_to_os())
.await
.unwrap()
.unwrap();
dead.shutdown().await;
wait_until("the remaining backend becomes the next choice", || async {
let s = a.agent.network_status(net).await.ok()?;
let v = s.vips.iter().find(|v| v.config.id == config.id)?;
(v.listeners.first()?.selected.as_ref()?.peer == alive.endpoint_id().to_string())
.then_some(())
})
.await;
tun.push_from_os(vip_segment(address, first.0, 41000, 8888, true, 0));
let packet = tokio::time::timeout(Duration::from_secs(5), alive.tun(net).await.pop_to_os())
.await
.unwrap()
.unwrap();
assert_eq!(&packet[16..20], &alive.overlay(net).await.octets());
let view = a.agent.network_status(net).await.unwrap();
assert_eq!(view.vips[0].listeners[0].counters.failovers, 1);
a.shutdown().await;
alive.shutdown().await;
}
+2
View File
@@ -27,6 +27,8 @@ clap = { version = "4.5", features = ["derive", "env"] }
# colour is never written where it would show up as rubbish.
anstream = "1.0"
anstyle = "1.0"
serde_json = "1.0"
tempfile.workspace = true
# Local interface addresses, for the diagnostics in `status`.
netwatch = "0.19.3"
+29 -1
View File
@@ -14,6 +14,7 @@ use std::time::Duration;
#[cfg(windows)]
mod winservice;
mod vip_cli;
use clap::{Args, Parser, Subcommand, ValueEnum};
use tsunagi::agent::Event;
use tsunagi::config::{AgentConfig, StoragePaths, TransportPolicy};
@@ -52,6 +53,8 @@ struct Cli {
#[derive(Debug, Subcommand)]
enum Command {
/// Manage named shared TCP/UDP virtual services.
Vip(vip_cli::VipArgs),
/// Shows this device's identity, and changes it.
///
/// Every item follows the same shape: name it to see it, name it with a
@@ -927,6 +930,7 @@ async fn run(command: Command) -> Result<(), Box<dyn std::error::Error>> {
Command::Join(args) => join_command(args).await,
Command::Network(args) => network_command(args).await,
Command::Dns(args) => dns_command(args).await,
Command::Vip(args) => vip_cli::run(args).await,
Command::ExitNode(args) => exit_node_command(args).await,
Command::Wipe(args) => wipe(args).await,
// Taken in `main`, before there is a runtime: the service manager
@@ -1317,7 +1321,12 @@ fn spawn_dns(agent: Agent, port: u16) -> DnsService {
let members = network.members.iter().filter_map(|member| {
Some((member.hostname.clone()?, member.overlay_address_v4?))
});
zones.push(Zone::new(zone, members));
let services = network
.vips
.iter()
.filter(|v| v.error.is_none())
.filter_map(|v| Some((v.config.name.clone(), v.address?)));
zones.push(Zone::new(zone, members.chain(services)));
}
let zones = Zones::new(zones);
let names = zones.names() as u32;
@@ -1577,6 +1586,23 @@ struct AgentControl {
}
impl tsunagi::ipc::ReportSource for AgentControl {
fn save_vip(
&self,
network: String,
config: tsunagi::vip::VipConfig,
expected: Option<String>,
delete: bool,
) -> tsunagi::BoxFuture<'_, Result<Vec<tsunagi::vip::VipStatus>, String>> {
Box::pin(async move {
let network = network
.parse()
.map_err(|e| format!("invalid network id: {e}"))?;
self.agent
.save_vip(network, config, expected, delete)
.await
.map_err(|e| e.to_string())
})
}
fn forget_node(
&self,
network: String,
@@ -5307,6 +5333,7 @@ async fn build_report(
};
NetworkReport {
vips: network.vips.clone(),
protocols: network.protocols.clone(),
auto_start: network.auto_start,
exit: exit_report(network),
@@ -5740,6 +5767,7 @@ mod status_tests {
/// The situation that prompted this: one peer left and came back.
fn network_after_a_peer_returned() -> NetworkReport {
NetworkReport {
vips: Vec::new(),
protocols: vec!["wg-quic".into()],
auto_start: true,
exit: Default::default(),
+506
View File
@@ -0,0 +1,506 @@
//! Named virtual service administration; every mutation submits one complete draft.
use super::{PathArgs, control_socket, resolve_network_index};
use clap::{Args, Subcommand};
use std::path::PathBuf;
use tsunagi::ipc::{self, NetworkReport};
use tsunagi::vip::{Backend, Listener, Protocol, VipConfig, VipStatus};
type Result<T> = std::result::Result<T, Box<dyn std::error::Error>>;
#[derive(Debug, Args)]
#[command(
after_help = "Examples:\n tsng vip create lab k8s\n tsng vip port add lab k8s tcp/443 --backend cp1:6443 --backend cp2:7443\n tsng vip backend add lab k8s tcp/443 cp3:6443\n tsng vip status lab k8s --watch\n tsng vip edit lab k8s\n\nNames, full ids and unique id prefixes are accepted. Changes are shared with the network.\nTCP port checks default to on; UDP uses data-path probes and explicit ICMP errors."
)]
pub(crate) struct VipArgs {
#[command(flatten)]
paths: PathArgs,
/// Agent control socket; defaults to the running service.
#[arg(long, global = true, env = "TSUNAGI_CONTROL_SOCKET")]
control_socket: Option<PathBuf>,
/// Machine-readable configuration and local observations.
#[arg(long, global = true)]
json: bool,
#[command(subcommand)]
action: Action,
}
#[derive(Debug, Args)]
struct Target {
/// Network exact name, id or unique id prefix.
network: String,
/// VIP exact name, id or unique id prefix.
vip: String,
}
#[derive(Debug, Args)]
struct PortTarget {
#[command(flatten)]
target: Target,
/// Listener as tcp/443 or udp/53.
listener: String,
}
#[derive(Debug, Subcommand)]
enum Action {
/// List services in one network, or every configured network.
List { network: Option<String> },
/// Create a named VIP with an automatically reserved internal address.
Create { network: String, name: String },
/// Show the complete saved configuration and local status.
Show(Target),
/// Show local health, routing choices and cumulative traffic counters.
Status {
#[command(flatten)]
target: Target,
/// Refresh once a second until interrupted.
#[arg(long)]
watch: bool,
},
/// Edit the entire configuration in VISUAL/EDITOR and save it atomically.
Edit(Target),
/// Remove a VIP; existing flows drain until completion or idle expiry.
Remove(Target),
/// Add, configure or remove listeners independently of the VIP.
Port {
#[command(subcommand)]
action: PortAction,
},
/// Change a listener's destination nodes and application ports.
Backend {
#[command(subcommand)]
action: BackendAction,
},
}
#[derive(Debug, Subcommand)]
enum PortAction {
/// Add a listener, optionally with its complete initial backend set.
Add {
#[command(flatten)]
target: PortTarget,
/// Node name/id followed by :port. Repeat for multiple destinations.
#[arg(long)]
backend: Vec<String>,
},
/// Change check settings and flow idle expiry.
Set {
#[command(flatten)]
target: PortTarget,
/// Active TCP connect checks: on or off.
#[arg(long,value_parser=["on","off"])]
port_check: Option<String>,
/// Milliseconds between active checks (250..3600000).
#[arg(long)]
check_interval_ms: Option<u32>,
/// TCP check deadline in milliseconds (100..60000).
#[arg(long)]
check_timeout_ms: Option<u32>,
/// Consecutive failures before exclusion.
#[arg(long)]
failures: Option<u8>,
/// Consecutive successes before recovery.
#[arg(long)]
successes: Option<u8>,
/// Flow idle expiry in seconds (1..604800).
#[arg(long)]
idle_timeout: Option<u32>,
},
/// Remove a listener from new-flow selection; existing flows drain.
Remove(PortTarget),
}
#[derive(Debug, Subcommand)]
enum BackendAction {
/// Add a destination node and port to a listener.
Add {
#[command(flatten)]
target: PortTarget,
backend: String,
},
/// Remove a destination from new-flow selection.
Remove {
#[command(flatten)]
target: PortTarget,
backend: String,
},
}
impl Action {
fn network(&self) -> Option<&str> {
match self {
Self::List { network } => network.as_deref(),
Self::Create { network, .. } => Some(network),
Self::Show(t) | Self::Edit(t) | Self::Remove(t) | Self::Status { target: t, .. } => {
Some(&t.network)
}
Self::Port { action } => Some(
&match action {
PortAction::Add { target, .. }
| PortAction::Set { target, .. }
| PortAction::Remove(target) => target,
}
.target
.network,
),
Self::Backend { action } => Some(
&match action {
BackendAction::Add { target, .. } | BackendAction::Remove { target, .. } => {
target
}
}
.target
.network,
),
}
}
}
fn parse_listener(s: &str) -> Result<(Protocol, u16)> {
let (p, n) = s
.split_once('/')
.ok_or("listener must be tcp/PORT or udp/PORT")?;
let port = n.parse::<u16>()?;
if port == 0 {
return Err("port must not be zero".into());
}
Ok((p.parse()?, port))
}
fn backend(n: &NetworkReport, s: &str) -> Result<Backend> {
let (node, port) = s.rsplit_once(':').ok_or("backend must be NODE:PORT")?;
let peer = match ipc::resolve_node(n, node) {
Ok(member) => member.endpoint_id.clone(),
Err(error) => {
let matches = n
.members
.iter()
.any(|m| m.hostname.as_deref() == Some(node) || m.endpoint_id.starts_with(node));
if matches {
return Err(error.into());
}
node.parse::<iroh::EndpointId>()
.map_err(|_| error)?
.to_string()
}
};
let port = port.parse()?;
if port == 0 {
return Err("backend port must not be zero".into());
}
Ok(Backend { peer, port })
}
fn selected(n: &NetworkReport, t: &Target) -> Result<VipStatus> {
Ok(ipc::resolve_vip(n, &t.vip)?.clone())
}
fn listener<'a>(c: &'a mut VipConfig, target: &PortTarget) -> Result<&'a mut Listener> {
let (p, n) = parse_listener(&target.listener)?;
c.listeners
.iter_mut()
.find(|l| l.protocol == p && l.port == n)
.ok_or_else(|| format!("no listener {} on {}", target.listener, c.name).into())
}
fn name_of<'a>(n: &'a NetworkReport, peer: &'a str) -> &'a str {
n.members
.iter()
.find(|m| m.endpoint_id == peer)
.and_then(|m| m.hostname.as_deref())
.unwrap_or(peer)
}
fn show(n: &NetworkReport, v: &VipStatus, json: bool) -> Result<()> {
if json {
println!("{}", serde_json::to_string(v)?);
return Ok(());
}
println!(
"{}.{} {}",
v.config.name,
n.name,
v.error.as_deref().unwrap_or("configured")
);
for l in &v.listeners {
let choice = l
.selected
.as_ref()
.map(|b| format!("{}:{}", name_of(n, &b.peer), b.port))
.unwrap_or_else(|| "none".into());
println!(
" {}/{} next: {} flows: {} tx/rx: {}/{} bytes drops: {} UDP failovers: {}",
l.protocol,
l.port,
choice,
l.backends.iter().map(|b| b.active_flows).sum::<u32>(),
l.counters.tx_bytes,
l.counters.rx_bytes,
l.counters.dropped,
l.counters.failovers
);
for b in &l.backends {
let rtt = b
.rtt_us
.map(|n| format!("{:.2} ms", n as f64 / 1000.))
.unwrap_or_else(|| "unknown".into());
println!(
" {}:{} {} RTT {} active {}",
name_of(n, &b.backend.peer),
b.backend.port,
b.state,
rtt,
b.active_flows
);
}
}
Ok(())
}
pub(crate) async fn run(args: VipArgs) -> Result<()> {
let custom = args.paths.state_dir.is_some();
let paths = args.paths.resolve()?;
let socket = control_socket(&paths, args.control_socket.as_ref(), custom);
let report = ipc::request_status(&socket).await?;
if let Action::List { network: None } = &args.action {
for n in &report.networks {
for v in &n.vips {
show(n, v, args.json)?;
}
}
return Ok(());
}
let selector = args.action.network().ok_or("network is required")?;
let names: Vec<_> = report
.networks
.iter()
.map(|n| (n.network_id.clone(), n.name.clone()))
.collect();
let network = &report.networks[resolve_network_index(&names, selector)?];
if let Action::Status {
target,
watch: true,
} = &args.action
{
let mut previous: Option<(std::time::Instant, u64, u64)> = None;
loop {
let report = ipc::request_status(&socket).await?;
let n = report
.networks
.iter()
.find(|n| n.network_id == network.network_id)
.ok_or("network was removed")?;
let vip = selected(n, target)?;
let (tx, rx) = vip.listeners.iter().fold((0u64, 0u64), |a, l| {
(
a.0.saturating_add(l.counters.tx_bytes),
a.1.saturating_add(l.counters.rx_bytes),
)
});
let now = std::time::Instant::now();
if let Some((at, old_tx, old_rx)) = previous
&& !args.json
{
let secs = now.duration_since(at).as_secs_f64();
if secs > 0. {
println!(
"TX {:.1} KiB/s · RX {:.1} KiB/s",
tx.saturating_sub(old_tx) as f64 / secs / 1024.,
rx.saturating_sub(old_rx) as f64 / secs / 1024.
);
}
}
previous = Some((now, tx, rx));
show(n, &vip, args.json)?;
tokio::select! {_=tokio::signal::ctrl_c()=>break,_=tokio::time::sleep(std::time::Duration::from_secs(1))=>{}}
}
return Ok(());
}
let mut edited_file = None;
let (mut config, expected, delete) = match &args.action {
Action::List { .. } => {
for v in &network.vips {
show(network, v, args.json)?;
}
return Ok(());
}
Action::Show(t) | Action::Status { target: t, .. } => {
let v = selected(network, t)?;
show(network, &v, args.json)?;
if matches!(args.action, Action::Show(_)) && !args.json {
println!("{}", serde_json::to_string_pretty(&v.config)?);
}
return Ok(());
}
Action::Create { name, .. } => (VipConfig::new(name), None, false),
Action::Remove(t) => {
let v = selected(network, t)?;
(v.config, Some(v.revision), true)
}
Action::Edit(t) => {
let v = selected(network, t)?;
let mut file = tempfile::Builder::new()
.prefix("tsunagi-vip-")
.suffix(".json")
.tempfile()?;
use std::io::Write;
file.write_all(serde_json::to_string_pretty(&v.config)?.as_bytes())?;
file.flush()?;
let editor = std::env::var("VISUAL")
.or_else(|_| std::env::var("EDITOR"))
.unwrap_or_else(|_| "vi".into());
let mut words = editor.split_whitespace();
let program = words.next().ok_or("empty EDITOR")?;
let status = std::process::Command::new(program)
.args(words)
.arg(file.path())
.status()?;
if !status.success() {
return Err("editor failed; VIP unchanged".into());
}
let bytes = std::fs::read(file.path())?;
let cfg: VipConfig = match serde_json::from_slice(&bytes) {
Ok(c) => c,
Err(e) => {
let (_, path) = file.keep()?;
return Err(
format!("invalid draft: {e}; retained at {}", path.display()).into(),
);
}
};
if cfg.id != v.config.id {
return Err("VIP id cannot be edited".into());
}
edited_file = Some(file);
(cfg, Some(v.revision), false)
}
Action::Port { action } => {
let target = match action {
PortAction::Add { target, .. }
| PortAction::Set { target, .. }
| PortAction::Remove(target) => target,
};
let v = selected(network, &target.target)?;
let mut cfg = v.config;
match action {
PortAction::Add {
target,
backend: destinations,
} => {
let (p, port) = parse_listener(&target.listener)?;
let mut l = Listener::new(p, port);
l.backends = destinations
.iter()
.map(|b| backend(network, b))
.collect::<Result<_>>()?;
cfg.listeners.push(l);
}
PortAction::Remove(target) => {
let (p, port) = parse_listener(&target.listener)?;
listener(&mut cfg, target)?;
cfg.listeners.retain(|l| l.protocol != p || l.port != port);
}
PortAction::Set {
target,
port_check,
check_interval_ms,
check_timeout_ms,
failures,
successes,
idle_timeout,
} => {
let l = listener(&mut cfg, target)?;
if let Some(v) = port_check {
l.port_check = v == "on";
}
if let Some(v) = check_interval_ms {
l.check_interval_ms = *v;
}
if let Some(v) = check_timeout_ms {
l.check_timeout_ms = *v;
}
if let Some(v) = failures {
l.failure_threshold = *v;
}
if let Some(v) = successes {
l.recovery_threshold = *v;
}
if let Some(v) = idle_timeout {
l.idle_timeout_secs = *v;
}
}
}
(cfg, Some(v.revision), false)
}
Action::Backend { action } => {
let (target, destination, remove) = match action {
BackendAction::Add { target, backend } => (target, backend, false),
BackendAction::Remove { target, backend } => (target, backend, true),
};
let v = selected(network, &target.target)?;
let mut cfg = v.config;
let b = backend(network, destination)?;
let l = listener(&mut cfg, target)?;
if remove {
if !l.backends.contains(&b) {
return Err("backend is not configured".into());
}
l.backends.retain(|x| x != &b);
} else {
l.backends.push(b);
}
(cfg, Some(v.revision), false)
}
};
// Canonical order makes displayed drafts and shared state deterministic.
config.listeners.sort_by_key(|l| (l.protocol, l.port));
for l in &mut config.listeners {
l.backends.sort();
}
let id = config.id.clone();
let vips = match ipc::save_vip(&socket, &network.network_id, config, expected, delete).await {
Ok(vips) => vips,
Err(error) => {
if let Some(file) = edited_file {
let (_, path) = file.keep()?;
return Err(format!("{error}; draft retained at {}", path.display()).into());
}
return Err(error.into());
}
};
if delete {
if args.json {
println!("{{\"removed\":\"{id}\"}}");
} else {
println!("VIP removed; existing flows drain.");
}
} else if let Some(v) = vips.iter().find(|v| v.config.id == id) {
show(network, v, args.json)?;
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
use clap::Parser;
#[test]
fn vip_commands_parse_complete_listener_and_settings() {
let cli = super::super::Cli::try_parse_from([
"tsng",
"vip",
"port",
"add",
"lab",
"k8s",
"tcp/443",
"--backend",
"cp1:6443",
"--backend",
"cp2:7443",
]);
assert!(cli.is_ok(), "{cli:?}");
assert!(
super::super::Cli::try_parse_from([
"tsng",
"vip",
"port",
"set",
"lab",
"k8s",
"tcp/443",
"--port-check",
"off",
"--failures",
"3"
])
.is_ok()
);
assert!(parse_listener("tcp/0").is_err());
assert!(parse_listener("sctp/53").is_err());
}
}
+92
View File
@@ -345,3 +345,95 @@ fn opposing_dns_flags_are_rejected() {
assert!(!result.status.success());
assert!(String::from_utf8_lossy(&result.stderr).contains("cannot be used with"));
}
#[test]
fn vip_cli_creates_multiple_ports_updates_dns_and_rejects_partial_invalid_changes() {
let agent = start("vip-test", 15367);
let created = agent.run(&["vip", "create", "vip-test", "api", "--json"]);
assert!(
created.status.success(),
"{}",
String::from_utf8_lossy(&created.stderr)
);
let vip: tsunagi::vip::VipStatus = serde_json::from_slice(&created.stdout).unwrap();
assert!(vip.config.listeners.is_empty());
let reply = wait_for_answer("127.0.0.1:15367".parse().unwrap(), "api.vip-test");
assert!(has_answer(&reply));
let socket = tsunagi::ipc::control_socket_path(&agent.dir.path().join("state"));
let runtime = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.unwrap();
let status = runtime
.block_on(tsunagi::ipc::request_status(socket))
.unwrap();
let backend = format!("{}:6443", status.endpoint_id);
let added = agent.run(&[
"vip",
"port",
"add",
"vip-test",
"api",
"tcp/443",
"--backend",
&backend,
]);
assert!(
added.status.success(),
"{}",
String::from_utf8_lossy(&added.stderr)
);
assert!(
agent
.run(&["vip", "port", "add", "vip-test", "api", "udp/53"])
.status
.success()
);
assert!(
!agent
.run(&["vip", "port", "add", "vip-test", "api", "tcp/443"])
.status
.success()
);
assert!(
agent
.run(&[
"vip",
"port",
"set",
"vip-test",
"api",
"tcp/443",
"--port-check",
"off"
])
.status
.success()
);
let shown = agent.run(&["vip", "show", "vip-test", "api", "--json"]);
assert!(shown.status.success());
let shown: tsunagi::vip::VipStatus = serde_json::from_slice(&shown.stdout).unwrap();
assert_eq!(shown.config.listeners.len(), 2);
assert!(
!shown
.config
.listeners
.iter()
.find(|l| l.protocol == tsunagi::vip::Protocol::Tcp)
.unwrap()
.port_check
);
assert_eq!(shown.config.listeners[0].backends[0].port, 6443);
let removed = agent.run(&["vip", "remove", "vip-test", "api"]);
assert!(removed.status.success());
let deadline = Instant::now() + Duration::from_secs(10);
loop {
if let Some(reply) = query("127.0.0.1:15367".parse().unwrap(), "api.vip-test", TYPE::A)
&& !has_answer(&reply)
{
break;
}
assert!(Instant::now() < deadline);
std::thread::sleep(Duration::from_millis(25));
}
}
+13 -11
View File
@@ -9,24 +9,26 @@ Two versions exist and are independent:
- **Identity scheme**, `tsunagi-network-id-v1`. Frozen. Changing it creates a
different network space for the same name and secret.
- **Control protocol**, ALPN `tsunagi/ctrl/7`, `PROTOCOL_VERSION = 7`.
- **Control protocol**, ALPN `tsunagi/ctrl/8`, `PROTOCOL_VERSION = 8`.
Upgrading the crate or bumping the control protocol must never change an
existing `NetworkId`.
Control version 7 adds signed offline-node retirement messages to version 6's
private AWG transport negotiation and delivery checks. Data ALPN remains
`tsunagi/data/5`. All members must upgrade together; network ids, handshake
encoding and existing record signatures are unchanged. Local CLI/tray IPC is
version 21, so restart the agent with matching clients. SQLite schema 9 adds
retirement storage while preserving existing identities, settings and records.
See [sync-model.md](sync-model.md#offline-node-cleanup) and [amnezia.md](amnezia.md).
Control version 8 adds signed, multi-writer VIP field snapshots to version 7's
node retirements. Data ALPN remains `tsunagi/data/5`. All members must upgrade
together. Network ids, derivation labels, handshake encoding and existing record
signatures are unchanged. Local CLI/tray IPC is version 22; restart the agent
with matching clients. SQLite schema 10 adds `vip_fields`, preserving identity,
network configuration, signed node records and retirement proofs.
See [virtual services](vip.md), [state sync](sync-model.md) and [AWG](amnezia.md).
## Canonical encoding
Everything that is hashed or MAC'd uses length-prefixed fields, written here as
`LP(x) = u32_be(len(x)) || x`. String concatenation is never used, so no two
different field splits can produce the same bytes.
Network identity derivation and the membership transcript use length-prefixed
fields, written here as `LP(x) = u32_be(len(x)) || x`. String concatenation is
never used, so different field splits cannot produce the same bytes. VIP field
signatures use their separate domain and canonical postcard encoding described
in [virtual services](vip.md#shared-state-and-addresses).
## Network space identity
+1 -1
View File
@@ -124,5 +124,5 @@ counts sends without storing frames. Results are CPU forwarding cost, **not**
end-to-end network latency or VPN throughput; encryption, fragmentation,
sockets, congestion and scheduling contribute separately.
Wire compatibility: control ALPN `tsunagi/ctrl/7`, data ALPN `tsunagi/data/5`.
Wire compatibility: control ALPN `tsunagi/ctrl/8`, data ALPN `tsunagi/data/5`.
Upgrade every participant together; saved identities and network state persist.
+15
View File
@@ -228,3 +228,18 @@ scroll area. They check stretching and shrinking without an expanding first
column, separate handshake/control RTT values, unknown measurements and short
end-to-end hints that do not substitute the carrier protocol. This does not
establish native desktop rendering on every platform.
VIP tests cover signed multi-writer backend merging, stale draft rejection,
permanent deletion and replay, separate active/deleted capacity, and mandatory
SQLite migration/counter rollback. Real agent instances verify equal editing
rights, network isolation, restart and recovery without the creator. Packet
checks exercise RTT selection, port translation, UDP failover, a local backend,
fragment reassembly/checksums and ICMP quoted-header/PMTU translation. TCP check
recovery uses real loopback listening sockets. The WireGuard suite moves VIP
TCP/UDP packets through real encrypted tunnels, rejects forged VIP sources and
checks UDP failover after the selected agent stops. CLI/DNS tests run the real
binary, configure independent listeners, disable checks, reject an invalid
change atomically and observe DNS creation/deletion. GUI tests cover draft
isolation and form rendering. These do not establish live host TUN behavior or
native desktop behavior on Linux, macOS or Windows. See [VIP semantics](vip.md).
+171
View File
@@ -0,0 +1,171 @@
# Named virtual services
A VIP is a shared name such as `api.home`, with independent TCP/UDP listeners.
Each listener maps its port to a set of node identities and application ports.
The application uses the name; the automatically allocated IPv4 address is an
internal reservation. The current overlay carries IPv4 only.
## CLI and tray
```sh
tsng vip list home
tsng vip create home api
tsng vip port add home api tcp/443 --backend server1:6443 --backend server2:7443
tsng vip port add home api udp/53 --backend server1:5353
tsng vip backend add home api tcp/443 server3:6443
tsng vip backend remove home api tcp/443 server2:7443
tsng vip port set home api tcp/443 --port-check off
tsng vip port set home api tcp/443 --check-interval-ms 3000 --check-timeout-ms 2000 --failures 2 --successes 1 --idle-timeout 7200
tsng vip show home api
tsng vip status home api --watch
tsng vip edit home api
tsng vip port remove home api udp/53
tsng vip remove home api
```
`list` without a network lists all configured networks. Network and VIP
selectors accept exact names, full ids and unique id prefixes; backend node
selectors also accept stored node names. Ambiguous selections change nothing.
Names resolve to identities when saved, so a renamed node remains a backend.
`--json` provides configuration, revision, counters and observations;
`list`/`watch` emit one JSON object per service/sample. `edit`
opens JSON in `VISUAL`/`EDITOR` and submits one complete version-checked draft.
An invalid or rejected edited file is retained with its path reported.
Each mutating command is atomic. A listener can be added with its full initial
backend set, and `edit` can replace the entire configuration at once. Empty
VIPs/listeners are valid and resolve, but cannot accept traffic. Configuration
updates currently require the network to be active locally. Saved configuration remains visible while a
network is stopped.
The tray's network tile has a **VIP** button. Its separate window edits names,
listeners, backend nodes/ports, check settings and flow timeouts. All edits,
including deletion, remain drafts until **Save**. Status updates do not
replace a dirty draft. A save conflict retains the draft; **Discard / reload**
loads the latest configuration. Closing a dirty window offers explicit discard
or continued editing. Addresses and ids are available in a collapsed detail.
## Delivery and recovery
The client agent selects the eligible backend with the lowest smoothed full
round-trip latency through the actual encrypted data plane, including relays.
Unknown or stale RTT is not treated as zero. Local backends have zero path RTT.
A new flow is pinned to its selected backend; a faster RTT does not move it.
This can concentrate new traffic on the closest node.
At local ingress, NAT replaces destination address/port and reserves an OS
source port for an unambiguous reverse mapping. Return traffic passes normal
network/peer source and destination admission **before** reverse NAT restores
the virtual source address/port. Plugins and encrypted transit stay unchanged.
A peer never gains permission to claim a VIP source. Fragment assembly is
bounded, checksums are updated, packets are fragmented back to the overlay MTU,
and ICMP errors have their quoted NAT headers restored. Fragment caches are
scoped to network and authenticated sender, with idle expiry independent of
further traffic.
For a backend on the client itself, the request is injected into the same TUN
with the nonlocal VIP as source, and the host reply is reverse-translated at
local ingress. The local backend therefore sees the VIP as source; remote
backends see the client's ordinary overlay address. Firewalls and application
bind addresses remain the administrator's responsibility.
Removing a backend/listener/VIP prevents new bindings while existing bindings
continue until idle expiry. Established TCP connections are not migrated. A retransmitted empty SYN may
choose another backend before a SYN-ACK has arrived; SYNs carrying application
data are not retargeted. A new SYN sequence creates a new binding even if the
client reuses its address/port tuple. TCP RST shortens binding retention. An unavailable UDP backend is replaced for subsequent
datagrams; already sent datagrams are never replayed. Stateful UDP applications
may need to re-establish their application session. Protocols that embed IP
addresses or negotiate additional ports require their own configuration; no
application payloads are parsed or rewritten.
Idle expiry defaults to 7200 seconds for TCP and 120 seconds for UDP. Shadow
ports are quarantined for 120 seconds after retirement to reject late replies
and avoid immediate reuse. Active and quarantined bindings share a bounded
capacity. State and sockets are released on network stop or agent shutdown.
A restart drops bindings; shared service configuration remains intact.
## Health
One probe exchange per second measures each used peer through the encrypted
IP tunnel. The EWMA uses 7/8 of the previous sample and 1/8 of the new one.
A probe expires after three seconds; two consecutive missing responses or a
missing data path exclude the peer. Measurements expire after ten seconds.
Probes continue during failure so recovery is automatic.
TCP connect checks default to on. Defaults are a three-second interval,
two-second timeout, two failures to exclude and one success to recover.
Checks are shared for the same network/node/address/TCP-port, run off the
packet path with at most 32 concurrent connects, and are scheduled fairly.
A 250ms worker tick services timers. Connect checks send no application data;
they confirm port acceptance and may appear in server logs. They cannot prove
application readiness after a successful handshake. Disabling port checks
retains data-plane probes.
UDP uses encrypted path checks and explicit ICMP port errors. No arbitrary
probe datagrams are injected into an unknown application protocol. Silence
cannot prove failure of an arbitrary UDP application. An explicit unreachable
port is excluded for ten seconds and then becomes eligible to retry.
## Shared state and addresses
Any authenticated member can create, edit or remove any VIP. Each field is a
self-contained signed register: network, service id, field key, optional encoded
value, Lamport clock, author, durable author counter and signature. Signing uses
the separate `tsunagi-vip-field/v1\0` domain followed by postcard encoding of
those fields with the signature vector empty. The encoding/order is a wire
contract; the identity scheme and handshake transcript are unchanged.
Concurrent edits to different backend fields merge. A conflict on one field
uses `(Lamport clock, author id, canonical bytes)` ordering, independent of wall
time and arrival order. Listener generations prevent backend fields from an
old removed listener reappearing when the port is recreated. Service deletion
is a permanent signed tombstone for its id; recreation gets a fresh id.
Missing snapshot fields are never deletion.
Saving validates the entire draft and checks the revision shown by the editor.
The merged snapshot and author's counter commit together in SQLite before local
routing changes or publication. Receivers verify and persist complete service
snapshots before exposing them. Periodic rotating snapshots repair queue loss
and returning replicas without requiring the creator online. Concurrent edits
made before participants have seen each other can still merge after a save.
A merged configuration exceeding listener/backend limits is reported as an
error and excluded from routing rather than selected according to arrival order.
VIP allocation walks the agreed overlay range from a service-id-derived seed,
excluding known node claims and other allocated VIPs. Nodes avoid known VIP
reservations when choosing a new address. Names colliding with node names or
another service are reported; node names take precedence and service ties use
id order. A partition can temporarily have incomplete state; after merge an
address or conflicting name's effective holder can change. Use DNS names.
DNS TTL is 30 seconds; DNS does not migrate established connections or force
applications to discard cached addresses. Existing NAT bindings retain their
original VIP address while draining.
`Limits` bounds active services (64 by default), permanent deletion markers
(1024), signed fields per active service (192), listeners (16), backends per
listener (32), active TCP checks (32), flow/port reservations (4096) and incomplete
IPv4 datagrams (64). A deleted service retains only its signed permanent
marker; obsolete fields can be dropped because replay cannot override deletion.
A full deletion history is an explicit error; protection is never silently
evicted. Deleted services do not consume the active-service limit. Whole service
snapshots must also fit the configured control frame limit. Corrupt mandatory VIP storage
is a hard error on activation, never reset as a cache.
## Observations and verification
Status is this client's observation. Per VIP/listener/backend it reports the
next selected backend, health reason, RTT and sample age, active bindings,
user IP packet/byte counters, drops and UDP failovers. Probe bytes do not enter
user traffic counters. The GUI and CLI watch derive rates from cumulative
counters; counters reset on restart. These are IP forwarding counters, not
application acknowledgements or guaranteed delivered-byte totals.
Automated tests cover field merging, stale drafts, signed deletion, tampering,
network isolation, SQLite migration and transactional rollback, real port
connect failure/recovery, RTT selection, UDP failover, local hairpin NAT,
fragmentation/checksums, real encrypted TCP/UDP packet delivery and source
admission, real CLI/DNS wiring and draft form rendering. In-memory TUN tests
exercise packets, not an operating system TCP stack. Native desktop behavior,
physical interfaces and cross-platform kernel/firewall behavior need live-host
validation; automated tests do not establish those.