Aleksandr Bogomiakov
23af1a2d9d
Provide named VIPs with independent protocol/port listeners and node/port backends so applications can use one DNS endpoint without a separate load balancer. Every authenticated member may create, edit or delete any service. Client agents select the eligible backend with the lowest measured encrypted data-plane RTT, including relayed paths, and pin existing flows. UDP recovers on another backend after failure; established TCP is never replayed or moved. An unanswered empty TCP SYN may retry another backend before establishment. Keep signed multi-writer field registers in state/vip.rs. Independent backend edits merge, same-field conflicts use Lamport/author/canonical ordering, and listener generations prevent old backend entries returning on port recreation. Permanent service tombstones protect replay while allowing obsolete fields to be compacted. Active-service and deletion-history limits are separate. Commit complete snapshots and the author's durable counter atomically before local routing changes or publication. Restore counters from newer genuine author updates learned from replicas after an old backup is restored. Implement bounded client-side address/port NAT at overlay ingress and reverse translation only after authenticated source/destination admission. Reserve OS source ports to avoid direct-flow collisions and quarantine retired ports for late replies. Handle a backend on the client itself, scoped fragment assembly, checksums, re-fragmentation and quoted ICMP/PMTU headers. Plugins and encrypted transit retain their existing contracts. Data-path probes measure RTT; bounded fair TCP connect checks are enabled by default and configurable per listener. Arbitrary UDP application readiness remains unknown; path checks and explicit port errors drive eligibility without protocol-specific probe payloads. Expose the shared draft API through local IPC, simple vip/listener/backend CLI commands, JSON, editor-based atomic updates and watch output. Add a VIP button on each network tile and a separate tray editor. Edits and removals stay local until Save; stale revisions retain the draft, and closing a dirty window offers explicit discard. Report local routing choices, health reasons, RTT age, active and draining bindings, IP counters, rates, drops and UDP failovers. Extend DNS with service names; keep numeric addresses secondary and preserve existing bindings while VIP addresses or listeners change. This changes control wire ALPN/version to tsunagi/ctrl/8 and local IPC to 22; all members and local clients must upgrade together. Data ALPN remains 5. SQLite schema 10 migrates existing mandatory state without changing identity, secrets, network ids, derivation labels, handshake encoding or node records. Crate versions and release tags are unchanged. No installed agent or tray was replaced. Commands, limits and operational semantics are in docs/vip.md. Checks on Linux in nix develop: - cargo fmt --all -- --check - cargo clippy --locked --workspace --all-targets -- -D warnings - cargo test --locked --workspace --all-targets: 559 passed, 3 ignored - git diff --cached --check The full suite includes real encrypted TCP/UDP VIP traffic, backend-stop UDP failover, real loopback TCP check recovery, source admission, signed merges, stale/deleted state, SQLite migration and rollback, restart/author recovery, network isolation, actual CLI/DNS processes, and headless text-entry/Save-button interaction. Discovery and memory TUNs are the existing test substitutions; authentication, transport, encryption and SQLite remain real. Initial checks outside Nix failed for missing GTK pkg-config metadata, and an initial CLI test inherited TSUNAGI_CONTROL_SOCKET. Final checks use the Nix environment with that variable unset so tests address their temporary sockets. Root/internet-dependent tests remain ignored. No live host TUN, native desktop session, macOS or Windows validation was performed.
172 lines
9.8 KiB
Markdown
172 lines
9.8 KiB
Markdown
# Named virtual services
|
|
|
|
A VIP is a shared name such as `api.home`, with independent TCP/UDP listeners.
|
|
Each listener maps its port to a set of node identities and application ports.
|
|
The application uses the name; the automatically allocated IPv4 address is an
|
|
internal reservation. The current overlay carries IPv4 only.
|
|
|
|
## CLI and tray
|
|
|
|
```sh
|
|
tsng vip list home
|
|
tsng vip create home api
|
|
tsng vip port add home api tcp/443 --backend server1:6443 --backend server2:7443
|
|
tsng vip port add home api udp/53 --backend server1:5353
|
|
tsng vip backend add home api tcp/443 server3:6443
|
|
tsng vip backend remove home api tcp/443 server2:7443
|
|
tsng vip port set home api tcp/443 --port-check off
|
|
tsng vip port set home api tcp/443 --check-interval-ms 3000 --check-timeout-ms 2000 --failures 2 --successes 1 --idle-timeout 7200
|
|
tsng vip show home api
|
|
tsng vip status home api --watch
|
|
tsng vip edit home api
|
|
tsng vip port remove home api udp/53
|
|
tsng vip remove home api
|
|
```
|
|
|
|
`list` without a network lists all configured networks. Network and VIP
|
|
selectors accept exact names, full ids and unique id prefixes; backend node
|
|
selectors also accept stored node names. Ambiguous selections change nothing.
|
|
Names resolve to identities when saved, so a renamed node remains a backend.
|
|
`--json` provides configuration, revision, counters and observations;
|
|
`list`/`watch` emit one JSON object per service/sample. `edit`
|
|
opens JSON in `VISUAL`/`EDITOR` and submits one complete version-checked draft.
|
|
An invalid or rejected edited file is retained with its path reported.
|
|
|
|
Each mutating command is atomic. A listener can be added with its full initial
|
|
backend set, and `edit` can replace the entire configuration at once. Empty
|
|
VIPs/listeners are valid and resolve, but cannot accept traffic. Configuration
|
|
updates currently require the network to be active locally. Saved configuration remains visible while a
|
|
network is stopped.
|
|
|
|
The tray's network tile has a **VIP** button. Its separate window edits names,
|
|
listeners, backend nodes/ports, check settings and flow timeouts. All edits,
|
|
including deletion, remain drafts until **Save**. Status updates do not
|
|
replace a dirty draft. A save conflict retains the draft; **Discard / reload**
|
|
loads the latest configuration. Closing a dirty window offers explicit discard
|
|
or continued editing. Addresses and ids are available in a collapsed detail.
|
|
|
|
## Delivery and recovery
|
|
|
|
The client agent selects the eligible backend with the lowest smoothed full
|
|
round-trip latency through the actual encrypted data plane, including relays.
|
|
Unknown or stale RTT is not treated as zero. Local backends have zero path RTT.
|
|
A new flow is pinned to its selected backend; a faster RTT does not move it.
|
|
This can concentrate new traffic on the closest node.
|
|
|
|
At local ingress, NAT replaces destination address/port and reserves an OS
|
|
source port for an unambiguous reverse mapping. Return traffic passes normal
|
|
network/peer source and destination admission **before** reverse NAT restores
|
|
the virtual source address/port. Plugins and encrypted transit stay unchanged.
|
|
A peer never gains permission to claim a VIP source. Fragment assembly is
|
|
bounded, checksums are updated, packets are fragmented back to the overlay MTU,
|
|
and ICMP errors have their quoted NAT headers restored. Fragment caches are
|
|
scoped to network and authenticated sender, with idle expiry independent of
|
|
further traffic.
|
|
|
|
For a backend on the client itself, the request is injected into the same TUN
|
|
with the nonlocal VIP as source, and the host reply is reverse-translated at
|
|
local ingress. The local backend therefore sees the VIP as source; remote
|
|
backends see the client's ordinary overlay address. Firewalls and application
|
|
bind addresses remain the administrator's responsibility.
|
|
|
|
Removing a backend/listener/VIP prevents new bindings while existing bindings
|
|
continue until idle expiry. Established TCP connections are not migrated. A retransmitted empty SYN may
|
|
choose another backend before a SYN-ACK has arrived; SYNs carrying application
|
|
data are not retargeted. A new SYN sequence creates a new binding even if the
|
|
client reuses its address/port tuple. TCP RST shortens binding retention. An unavailable UDP backend is replaced for subsequent
|
|
datagrams; already sent datagrams are never replayed. Stateful UDP applications
|
|
may need to re-establish their application session. Protocols that embed IP
|
|
addresses or negotiate additional ports require their own configuration; no
|
|
application payloads are parsed or rewritten.
|
|
|
|
Idle expiry defaults to 7200 seconds for TCP and 120 seconds for UDP. Shadow
|
|
ports are quarantined for 120 seconds after retirement to reject late replies
|
|
and avoid immediate reuse. Active and quarantined bindings share a bounded
|
|
capacity. State and sockets are released on network stop or agent shutdown.
|
|
A restart drops bindings; shared service configuration remains intact.
|
|
|
|
## Health
|
|
|
|
One probe exchange per second measures each used peer through the encrypted
|
|
IP tunnel. The EWMA uses 7/8 of the previous sample and 1/8 of the new one.
|
|
A probe expires after three seconds; two consecutive missing responses or a
|
|
missing data path exclude the peer. Measurements expire after ten seconds.
|
|
Probes continue during failure so recovery is automatic.
|
|
|
|
TCP connect checks default to on. Defaults are a three-second interval,
|
|
two-second timeout, two failures to exclude and one success to recover.
|
|
Checks are shared for the same network/node/address/TCP-port, run off the
|
|
packet path with at most 32 concurrent connects, and are scheduled fairly.
|
|
A 250ms worker tick services timers. Connect checks send no application data;
|
|
they confirm port acceptance and may appear in server logs. They cannot prove
|
|
application readiness after a successful handshake. Disabling port checks
|
|
retains data-plane probes.
|
|
|
|
UDP uses encrypted path checks and explicit ICMP port errors. No arbitrary
|
|
probe datagrams are injected into an unknown application protocol. Silence
|
|
cannot prove failure of an arbitrary UDP application. An explicit unreachable
|
|
port is excluded for ten seconds and then becomes eligible to retry.
|
|
|
|
## Shared state and addresses
|
|
|
|
Any authenticated member can create, edit or remove any VIP. Each field is a
|
|
self-contained signed register: network, service id, field key, optional encoded
|
|
value, Lamport clock, author, durable author counter and signature. Signing uses
|
|
the separate `tsunagi-vip-field/v1\0` domain followed by postcard encoding of
|
|
those fields with the signature vector empty. The encoding/order is a wire
|
|
contract; the identity scheme and handshake transcript are unchanged.
|
|
|
|
Concurrent edits to different backend fields merge. A conflict on one field
|
|
uses `(Lamport clock, author id, canonical bytes)` ordering, independent of wall
|
|
time and arrival order. Listener generations prevent backend fields from an
|
|
old removed listener reappearing when the port is recreated. Service deletion
|
|
is a permanent signed tombstone for its id; recreation gets a fresh id.
|
|
Missing snapshot fields are never deletion.
|
|
|
|
Saving validates the entire draft and checks the revision shown by the editor.
|
|
The merged snapshot and author's counter commit together in SQLite before local
|
|
routing changes or publication. Receivers verify and persist complete service
|
|
snapshots before exposing them. Periodic rotating snapshots repair queue loss
|
|
and returning replicas without requiring the creator online. Concurrent edits
|
|
made before participants have seen each other can still merge after a save.
|
|
A merged configuration exceeding listener/backend limits is reported as an
|
|
error and excluded from routing rather than selected according to arrival order.
|
|
|
|
VIP allocation walks the agreed overlay range from a service-id-derived seed,
|
|
excluding known node claims and other allocated VIPs. Nodes avoid known VIP
|
|
reservations when choosing a new address. Names colliding with node names or
|
|
another service are reported; node names take precedence and service ties use
|
|
id order. A partition can temporarily have incomplete state; after merge an
|
|
address or conflicting name's effective holder can change. Use DNS names.
|
|
DNS TTL is 30 seconds; DNS does not migrate established connections or force
|
|
applications to discard cached addresses. Existing NAT bindings retain their
|
|
original VIP address while draining.
|
|
|
|
`Limits` bounds active services (64 by default), permanent deletion markers
|
|
(1024), signed fields per active service (192), listeners (16), backends per
|
|
listener (32), active TCP checks (32), flow/port reservations (4096) and incomplete
|
|
IPv4 datagrams (64). A deleted service retains only its signed permanent
|
|
marker; obsolete fields can be dropped because replay cannot override deletion.
|
|
A full deletion history is an explicit error; protection is never silently
|
|
evicted. Deleted services do not consume the active-service limit. Whole service
|
|
snapshots must also fit the configured control frame limit. Corrupt mandatory VIP storage
|
|
is a hard error on activation, never reset as a cache.
|
|
|
|
## Observations and verification
|
|
|
|
Status is this client's observation. Per VIP/listener/backend it reports the
|
|
next selected backend, health reason, RTT and sample age, active bindings,
|
|
user IP packet/byte counters, drops and UDP failovers. Probe bytes do not enter
|
|
user traffic counters. The GUI and CLI watch derive rates from cumulative
|
|
counters; counters reset on restart. These are IP forwarding counters, not
|
|
application acknowledgements or guaranteed delivered-byte totals.
|
|
|
|
Automated tests cover field merging, stale drafts, signed deletion, tampering,
|
|
network isolation, SQLite migration and transactional rollback, real port
|
|
connect failure/recovery, RTT selection, UDP failover, local hairpin NAT,
|
|
fragmentation/checksums, real encrypted TCP/UDP packet delivery and source
|
|
admission, real CLI/DNS wiring and draft form rendering. In-memory TUN tests
|
|
exercise packets, not an operating system TCP stack. Native desktop behavior,
|
|
physical interfaces and cross-platform kernel/firewall behavior need live-host
|
|
validation; automated tests do not establish those.
|