Aleksandr Bogomiakov
2166ba4800
AWG-only policy also selected a competing end-to-end plugin, so compatible external links could still have no compatible encrypted data channel. Carrier changes replaced tunnel state and status selected counters from different engines. Use one internal WireGuard plugin and a separate DataCarrier contract for external capabilities/lifecycle. Keep the logical link, keys, handshake, interface and counter epoch across policy, fallback and direct/relay changes. Control/discovery transport selection and membership remain independent. Share the bounded per-neighbour UDP WireGuard engine between ordinary WG and Amnezia masking. Encrypt the relay envelope inside the outer hop channel with fresh pair keys/PSK. Remove duplicate WG key/device code and the TCP/TLS inner cipher. TLS carries opaque internal WireGuard frames. Refuse missing native transports instead of letting default iroh impersonate their protocol names. Probe the actual authenticated carrier, independently of the shared IP tunnel; failed local probe submissions count as loss too. Apply the vendored iroh interface exclusion to actual direct candidates, not just public discovery records. Exclude overlay subnets from introductions and propagate the actual owned interface name, including kernel-assigned utun names, to every carrier. Production logs showed paths through the overlay itself. This removes a recursive transport path; it does not establish the historical cause of the reported 100 GB absolute counter. Exit-node traffic is also real peer IP traffic even when local applications are idle. Hide inner cipher choices/handshake clutter in CLI and tray. Report external routes, readiness and actionable policy/version failures. Count original IP packets only when their ciphertext is accepted by the logical transport, including queued packets' exact unpadded lengths. Exclude physical probes and keepalives. Export counter epochs; sum per-peer rate deltas and discard stale histories so joins/restarts cannot turn old totals into fresh traffic. Status also prints exact IP byte totals for comparison with the tray. Coordinated upgrade: data ALPN 6, internal wireguard-e2e capability v4, wg/awg v2, QUIC carrier v1, TLS v2, local IPC v23. Control ALPN 8, device/network identity derivation, signed records and saved network choices stay unchanged. Keep the historical wg-quic key store; transactionally import missing keys from wg then awg without replacing canonical keys. Forget removes legacy copies too, on the blocking pool, to prevent resurrection after restart. Preserve historic cipher options as shared settings, rejecting contradictory values. Validation on Linux in the Nix development shell: - cargo fmt --all -- --check - cargo clippy --locked --workspace --all-targets -- -D warnings - cargo test --locked --workspace --all-targets (562 passed, 3 ignored) The inherited service socket/secret/group environment overrides were removed for isolated CLI process tests. A new carrier test initially raced the first peer announcement; its deadline polling now waits for a peer row explicitly. Real loopback endpoints, membership, SQLite, WG/AWG/TLS encryption and memory TUNs cover mixed-carrier relaying and one smoltcp TCP connection surviving relay/direct/policy changes and a dropped transmission. Counter tests check exact plaintext lengths, refused sends, epochs, joins and stale histories. Public DHT, privileged kernel and manual benchmark tests remain ignored; macOS/Windows native TUNs, desktop GUI and WAN/DPI behavior were not run.
948 lines
32 KiB
Rust
948 lines
32 KiB
Rust
//! Real membership, UDP sockets, pair negotiation and nested WireGuard encryption.
|
|
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
|
|
#[path = "common/tcp.rs"]
|
|
mod tcp;
|
|
use bytes::Bytes;
|
|
use std::{
|
|
net::Ipv4Addr,
|
|
path::Path,
|
|
sync::{
|
|
Arc,
|
|
atomic::{AtomicBool, AtomicU64, Ordering},
|
|
},
|
|
time::Duration,
|
|
};
|
|
use tempfile::TempDir;
|
|
use tsunagi::dataplane::IpPlugin;
|
|
use tsunagi::dataplane::transport::{
|
|
ControlSender, PacketLink, PacketTransport, SharedLink, TransportError,
|
|
};
|
|
use tsunagi::discovery::SharedMemoryDiscovery;
|
|
use tsunagi::overlay::{MemoryTun, MemoryTunFactory};
|
|
use tsunagi::testing::{config_with, network, wait_for_peers, wait_until};
|
|
use tsunagi::{Agent, BoxFuture, NetworkId};
|
|
use tsunagi_awg::{AWG_PROTOCOL, AwgPlugin, AwgTransport};
|
|
use tsunagi_wg_quic::WireguardConfig;
|
|
|
|
struct Node {
|
|
agent: Agent,
|
|
plugin: Arc<tsunagi_wg_quic::WireguardPlugin>,
|
|
transport: Arc<AwgTransport>,
|
|
tuns: MemoryTunFactory,
|
|
loss: Arc<AtomicBool>,
|
|
}
|
|
impl Node {
|
|
async fn open(root: &Path, discovery: &SharedMemoryDiscovery, interface: &str) -> Self {
|
|
Self::open_with_fallback(root, discovery, interface, false).await
|
|
}
|
|
async fn open_with_fallback(
|
|
root: &Path,
|
|
discovery: &SharedMemoryDiscovery,
|
|
interface: &str,
|
|
fallback: bool,
|
|
) -> Self {
|
|
Self::open_config(root, discovery, interface, fallback, 1280).await
|
|
}
|
|
async fn open_config(
|
|
root: &Path,
|
|
discovery: &SharedMemoryDiscovery,
|
|
interface: &str,
|
|
fallback: bool,
|
|
mtu: u32,
|
|
) -> Self {
|
|
Self::open_selected(root, discovery, interface, fallback, mtu, None).await
|
|
}
|
|
async fn open_selected(
|
|
root: &Path,
|
|
discovery: &SharedMemoryDiscovery,
|
|
interface: &str,
|
|
fallback: bool,
|
|
mtu: u32,
|
|
default_protocols: Option<Vec<String>>,
|
|
) -> Self {
|
|
let identity =
|
|
tsunagi::storage::StateStore::open(tsunagi::StoragePaths::under(root).state_db())
|
|
.unwrap()
|
|
.load_or_create_device_identity()
|
|
.unwrap();
|
|
let transport = AwgTransport::new(identity.endpoint_id(), Ipv4Addr::LOCALHOST);
|
|
let plugin = tsunagi_wg_quic::WireguardPlugin::open(
|
|
WireguardConfig::new(root.join("awg"))
|
|
.with_mtu(mtu)
|
|
.with_reconcile(Duration::from_millis(20), Duration::from_millis(250)),
|
|
)
|
|
.await
|
|
.unwrap();
|
|
let loss = Arc::new(AtomicBool::new(false));
|
|
let faults = Arc::new(FaultTransport {
|
|
inner: transport.clone(),
|
|
loss: loss.clone(),
|
|
});
|
|
let tuns = MemoryTunFactory::new();
|
|
let mut config = config_with(root, discovery)
|
|
.with_interface(Arc::new(tuns.clone()), interface, mtu)
|
|
.with_tunnel(plugin.clone() as Arc<dyn IpPlugin>)
|
|
.with_carrier(AwgPlugin::new(transport.clone()))
|
|
.with_custom_transport(AWG_PROTOCOL, faults);
|
|
if fallback {
|
|
config = config.with_carrier(Arc::new(
|
|
tsunagi::dataplane::transport::iroh_link::IrohCarrier,
|
|
));
|
|
}
|
|
config.default_protocols = default_protocols;
|
|
let agent = Agent::spawn(config).await.unwrap();
|
|
let inbound_agent = agent.clone();
|
|
let inbound_transport = transport.clone();
|
|
let inbound_loss = loss.clone();
|
|
tokio::spawn(async move {
|
|
while let Some(mut link) = inbound_transport.next_inbound().await {
|
|
link.link = impaired(link.link, inbound_loss.clone());
|
|
let _ = inbound_agent.install_inbound_link(link).await;
|
|
}
|
|
});
|
|
Self {
|
|
agent,
|
|
plugin,
|
|
transport,
|
|
tuns,
|
|
loss,
|
|
}
|
|
}
|
|
async fn ready(&self, id: NetworkId, n: usize) {
|
|
wait_until("AWG outer paths and end-to-end handshakes", || async {
|
|
(self
|
|
.transport
|
|
.overview(id)
|
|
.iter()
|
|
.filter(|p| !p.closed && !p.path.contains("establishing"))
|
|
.count()
|
|
== n
|
|
&& self
|
|
.plugin
|
|
.overview(id)?
|
|
.peers
|
|
.iter()
|
|
.filter(|p| {
|
|
p.tunnel
|
|
.as_ref()
|
|
.is_some_and(|t| t.health.is_up() && t.path.contains("awg UDP via"))
|
|
})
|
|
.count()
|
|
== n)
|
|
.then_some(())
|
|
})
|
|
.await;
|
|
}
|
|
fn tun(&self) -> Arc<MemoryTun> {
|
|
self.tuns
|
|
.device(&self.agent.overlay().unwrap().interface)
|
|
.unwrap()
|
|
}
|
|
fn addr(&self, id: NetworkId) -> Ipv4Addr {
|
|
self.plugin
|
|
.overview(id)
|
|
.unwrap()
|
|
.overlay_address_v4
|
|
.unwrap()
|
|
}
|
|
}
|
|
fn packet(source: Ipv4Addr, dest: Ipv4Addr, len: usize) -> Bytes {
|
|
let mut bytes = vec![0; len];
|
|
bytes[0] = 0x45;
|
|
bytes[2..4].copy_from_slice(&(len as u16).to_be_bytes());
|
|
bytes[8] = 64;
|
|
bytes[9] = 253;
|
|
bytes[12..16].copy_from_slice(&source.octets());
|
|
bytes[16..20].copy_from_slice(&dest.octets());
|
|
rand::fill(&mut bytes[20..]);
|
|
Bytes::from(bytes)
|
|
}
|
|
async fn exchange(a: &Node, b: &Node, id: NetworkId) {
|
|
for len in [64, 1097, 1280] {
|
|
let bytes = packet(a.addr(id), b.addr(id), len);
|
|
a.tun().push_from_os(bytes.clone());
|
|
assert_eq!(
|
|
tokio::time::timeout(Duration::from_secs(5), b.tun().pop_to_os())
|
|
.await
|
|
.unwrap()
|
|
.unwrap(),
|
|
bytes
|
|
);
|
|
}
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn a_control_only_agent_enables_real_data_without_restart_and_cancels_stale_attempts() {
|
|
let discovery = SharedMemoryDiscovery::new();
|
|
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
|
|
let a = Node::open_selected(
|
|
roots[0].path(),
|
|
&discovery,
|
|
"initial-none-a",
|
|
true,
|
|
1280,
|
|
Some(vec![]),
|
|
)
|
|
.await;
|
|
let b = Node::open_selected(
|
|
roots[1].path(),
|
|
&discovery,
|
|
"initial-none-b",
|
|
true,
|
|
1280,
|
|
Some(vec![]),
|
|
)
|
|
.await;
|
|
let (name, secret) = network("enable-after-control-only");
|
|
let id = a.agent.join_network(&name, &secret).await.unwrap();
|
|
b.agent.join_network(&name, &secret).await.unwrap();
|
|
wait_for_peers(&a.agent, id, 1).await;
|
|
wait_for_peers(&b.agent, id, 1).await;
|
|
assert!(a.agent.overlay().is_none());
|
|
assert!(b.agent.overlay().is_none());
|
|
for node in [&a, &b] {
|
|
assert!(
|
|
node.agent
|
|
.network_status(id)
|
|
.await
|
|
.unwrap()
|
|
.protocols
|
|
.is_empty()
|
|
);
|
|
node.agent
|
|
.set_protocols(id, vec!["awg".into()])
|
|
.await
|
|
.unwrap();
|
|
assert!(node.agent.overlay().is_some());
|
|
}
|
|
a.ready(id, 1).await;
|
|
b.ready(id, 1).await;
|
|
exchange(&a, &b, id).await;
|
|
let before = a.agent.overlay().unwrap().interface;
|
|
// Changes race unfinished opens and prepared plugin callbacks; the last
|
|
// selection must win even if an earlier open has completed in its queue.
|
|
for selection in [
|
|
vec!["wg-quic".into()],
|
|
vec![],
|
|
vec!["awg".into()],
|
|
vec![],
|
|
vec!["awg".into()],
|
|
] {
|
|
a.agent.set_protocols(id, selection).await.unwrap();
|
|
}
|
|
a.ready(id, 1).await;
|
|
b.ready(id, 1).await;
|
|
exchange(&a, &b, id).await;
|
|
assert_eq!(a.agent.overlay().unwrap().interface, before);
|
|
for node in [&a, &b] {
|
|
let status = node.agent.network_status(id).await.unwrap();
|
|
assert_eq!(status.protocols, ["awg"]);
|
|
assert_eq!(status.metrics.sessions_established, 1);
|
|
assert_eq!(status.metrics.disconnects, 0);
|
|
}
|
|
a.agent.shutdown().await;
|
|
b.agent.shutdown().await;
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn runtime_protocol_selection_preserves_real_tunnels_and_survives_restart() {
|
|
let discovery = SharedMemoryDiscovery::new();
|
|
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
|
|
let a = Node::open_with_fallback(roots[0].path(), &discovery, "runtime-a", true).await;
|
|
let b = Node::open_with_fallback(roots[1].path(), &discovery, "runtime-b", true).await;
|
|
let (name, secret) = network("runtime-protocols");
|
|
let id = a.agent.join_network(&name, &secret).await.unwrap();
|
|
b.agent.join_network(&name, &secret).await.unwrap();
|
|
a.ready(id, 1).await;
|
|
b.ready(id, 1).await;
|
|
exchange(&a, &b, id).await;
|
|
let original_key = a.plugin.overview(id).unwrap().public_key;
|
|
let original_stats = a.plugin.overview(id).unwrap().peers[0]
|
|
.tunnel
|
|
.as_ref()
|
|
.unwrap()
|
|
.stats;
|
|
|
|
let (other_name, other_secret, other_id) = loop {
|
|
let (name, secret) = network("unchanged-runtime-network");
|
|
let other_id = tsunagi::identity::NetworkKeys::derive(&name, &secret).network_id();
|
|
if tsunagi::state::derived_ipv4_range(id) != tsunagi::state::derived_ipv4_range(other_id) {
|
|
break (name, secret, other_id);
|
|
}
|
|
};
|
|
a.agent
|
|
.join_network(&other_name, &other_secret)
|
|
.await
|
|
.unwrap();
|
|
b.agent
|
|
.join_network(&other_name, &other_secret)
|
|
.await
|
|
.unwrap();
|
|
a.ready(other_id, 1).await;
|
|
b.ready(other_id, 1).await;
|
|
let other_before = a
|
|
.agent
|
|
.network_status(other_id)
|
|
.await
|
|
.unwrap()
|
|
.metrics
|
|
.data_links_established;
|
|
|
|
for node in [&a, &b] {
|
|
let before = node.agent.network_status(id).await.unwrap().protocols;
|
|
for invalid in [vec!["unknown".into()], vec!["awg".into(), "awg".into()]] {
|
|
assert!(node.agent.set_protocols(id, invalid).await.is_err());
|
|
assert_eq!(
|
|
node.agent.network_status(id).await.unwrap().protocols,
|
|
before
|
|
);
|
|
}
|
|
}
|
|
// Exercise updates initiated by either endpoint-id role, including an
|
|
// empty set. Membership and the control session stay alive throughout.
|
|
for node in [&a, &b] {
|
|
node.agent.set_protocols(id, Vec::new()).await.unwrap();
|
|
wait_until("peer accepts control-only selection", || async {
|
|
let other = if node.agent.endpoint_id() == a.agent.endpoint_id() {
|
|
&b
|
|
} else {
|
|
&a
|
|
};
|
|
let status = other.agent.network_status(id).await.ok()?;
|
|
(status.peers.len() == 1
|
|
&& status.peers[0].protocols.is_empty()
|
|
&& matches!(status.peers[0].data_path, tsunagi::agent::DataPath::None))
|
|
.then_some(())
|
|
})
|
|
.await;
|
|
assert_eq!(node.agent.network_status(id).await.unwrap().peers.len(), 1);
|
|
node.agent
|
|
.set_protocols(id, vec!["wg-quic".into(), "awg".into()])
|
|
.await
|
|
.unwrap();
|
|
}
|
|
a.agent
|
|
.set_protocols(id, vec!["wg-quic".into()])
|
|
.await
|
|
.unwrap();
|
|
wait_until("real QUIC replaces AWG", || async {
|
|
let a_status = a.agent.network_status(id).await.ok()?;
|
|
let b_status = b.agent.network_status(id).await.ok()?;
|
|
(a_status.peers[0].protocols == ["wg-quic"]
|
|
&& b_status.peers[0].protocols == ["wg-quic"]
|
|
&& !matches!(a_status.peers[0].data_path, tsunagi::agent::DataPath::None)
|
|
&& !matches!(b_status.peers[0].data_path, tsunagi::agent::DataPath::None)
|
|
&& a.transport.overview(id).iter().all(|p| p.closed)
|
|
&& b.transport.overview(id).iter().all(|p| p.closed))
|
|
.then_some(())
|
|
})
|
|
.await;
|
|
assert_eq!(a.plugin.overview(id).unwrap().public_key, original_key);
|
|
let stats = a.plugin.overview(id).unwrap().peers[0]
|
|
.tunnel
|
|
.as_ref()
|
|
.unwrap()
|
|
.stats;
|
|
assert_eq!(stats.counter_epoch, original_stats.counter_epoch);
|
|
assert!(stats.tx_bytes >= original_stats.tx_bytes);
|
|
assert!(stats.rx_bytes >= original_stats.rx_bytes);
|
|
assert!(a.plugin.overview(id).unwrap().peers.iter().any(|peer| {
|
|
peer.tunnel
|
|
.as_ref()
|
|
.is_some_and(|tunnel| tunnel.health.is_up())
|
|
}));
|
|
// The same internal plugin supplies addresses and counters on the new carrier.
|
|
for (source, dest) in [(&a, &b), (&b, &a)] {
|
|
wait_until("QUIC end-to-end handshake", || async {
|
|
source
|
|
.plugin
|
|
.overview(id)?
|
|
.peers
|
|
.iter()
|
|
.any(|p| p.tunnel.as_ref().is_some_and(|t| t.health.is_up()))
|
|
.then_some(())
|
|
})
|
|
.await;
|
|
let addr = |n: &Node| n.plugin.overview(id).unwrap().overlay_address_v4.unwrap();
|
|
let bytes = packet(addr(source), addr(dest), 1280);
|
|
source.tun().push_from_os(bytes.clone());
|
|
assert_eq!(
|
|
tokio::time::timeout(Duration::from_secs(5), dest.tun().pop_to_os())
|
|
.await
|
|
.unwrap()
|
|
.unwrap(),
|
|
bytes
|
|
);
|
|
}
|
|
assert_eq!(
|
|
a.agent
|
|
.network_status(other_id)
|
|
.await
|
|
.unwrap()
|
|
.metrics
|
|
.data_links_established,
|
|
other_before
|
|
);
|
|
exchange(&a, &b, other_id).await;
|
|
a.agent.shutdown().await;
|
|
let restarted = Node::open_with_fallback(roots[0].path(), &discovery, "runtime-a", true).await;
|
|
wait_for_peers(&restarted.agent, id, 1).await;
|
|
assert_eq!(
|
|
restarted.agent.network_status(id).await.unwrap().protocols,
|
|
["wg-quic"]
|
|
);
|
|
assert_eq!(
|
|
b.agent.network_status(other_id).await.unwrap().protocols,
|
|
["awg", "wg-quic"]
|
|
);
|
|
restarted.ready(other_id, 1).await;
|
|
b.ready(other_id, 1).await;
|
|
exchange(&restarted, &b, other_id).await;
|
|
restarted
|
|
.agent
|
|
.set_protocols(id, vec!["awg".into()])
|
|
.await
|
|
.unwrap();
|
|
restarted.ready(id, 1).await;
|
|
b.ready(id, 1).await;
|
|
exchange(&restarted, &b, id).await;
|
|
restarted.agent.shutdown().await;
|
|
b.agent.shutdown().await;
|
|
}
|
|
#[tokio::test]
|
|
async fn real_pairs_carry_full_size_packets_and_restart_with_fresh_profiles() {
|
|
tsunagi::testing::init_tracing();
|
|
let discovery = SharedMemoryDiscovery::new();
|
|
let roots = [
|
|
TempDir::new().unwrap(),
|
|
TempDir::new().unwrap(),
|
|
TempDir::new().unwrap(),
|
|
];
|
|
let a = Node::open(roots[0].path(), &discovery, "awga").await;
|
|
let b = Node::open(roots[1].path(), &discovery, "awgb").await;
|
|
let c = Node::open(roots[2].path(), &discovery, "awgc").await;
|
|
let (name, secret) = network("awg-pairs");
|
|
let id = a.agent.join_network(&name, &secret).await.unwrap();
|
|
b.agent.join_network(&name, &secret).await.unwrap();
|
|
c.agent.join_network(&name, &secret).await.unwrap();
|
|
for node in [&a, &b, &c] {
|
|
wait_for_peers(&node.agent, id, 2).await;
|
|
node.ready(id, 2).await;
|
|
}
|
|
let ids: std::collections::HashSet<_> = [&a, &b, &c]
|
|
.into_iter()
|
|
.flat_map(|n| n.transport.overview(id).into_iter().map(|p| p.attempt))
|
|
.collect();
|
|
assert_eq!(
|
|
ids.len(),
|
|
3,
|
|
"one independent profile for each of three pairs"
|
|
);
|
|
exchange(&a, &b, id).await;
|
|
exchange(&b, &a, id).await;
|
|
exchange(&a, &c, id).await;
|
|
let old_ports: Vec<_> = a
|
|
.transport
|
|
.overview(id)
|
|
.into_iter()
|
|
.filter_map(|p| p.local_port)
|
|
.collect();
|
|
a.agent.shutdown().await;
|
|
for port in old_ports {
|
|
wait_until("old pair socket released", || async {
|
|
tokio::net::UdpSocket::bind((Ipv4Addr::LOCALHOST, port))
|
|
.await
|
|
.ok()
|
|
})
|
|
.await;
|
|
}
|
|
let restarted = Node::open(roots[0].path(), &discovery, "awga").await;
|
|
restarted.ready(id, 2).await;
|
|
assert!(
|
|
restarted
|
|
.transport
|
|
.overview(id)
|
|
.iter()
|
|
.all(|p| !ids.contains(&p.attempt))
|
|
);
|
|
exchange(&restarted, &b, id).await;
|
|
exchange(&c, &restarted, id).await;
|
|
restarted.agent.shutdown().await;
|
|
b.agent.shutdown().await;
|
|
c.agent.shutdown().await;
|
|
assert!(
|
|
tokio::time::timeout(Duration::from_secs(1), restarted.transport.next_inbound())
|
|
.await
|
|
.unwrap()
|
|
.is_none()
|
|
);
|
|
}
|
|
|
|
// Drops datagrams at the link boundary. Membership, control, sockets, crypto,
|
|
// addressing and storage remain real. This substitutes a lossy physical path.
|
|
#[derive(Debug)]
|
|
struct FaultTransport {
|
|
inner: Arc<AwgTransport>,
|
|
loss: Arc<AtomicBool>,
|
|
}
|
|
impl PacketTransport for FaultTransport {
|
|
fn name(&self) -> &str {
|
|
AWG_PROTOCOL
|
|
}
|
|
fn as_any(&self) -> &dyn std::any::Any {
|
|
self
|
|
}
|
|
fn attach_control(&self, sender: Arc<dyn ControlSender>) {
|
|
self.inner.attach_control(sender);
|
|
}
|
|
fn on_control(
|
|
&self,
|
|
network: NetworkId,
|
|
peer: iroh::EndpointId,
|
|
protocol: &str,
|
|
bytes: &[u8],
|
|
) -> Result<(), TransportError> {
|
|
self.inner.on_control(network, peer, protocol, bytes)
|
|
}
|
|
fn open<'a>(
|
|
&'a self,
|
|
network: NetworkId,
|
|
peer: iroh::EndpointId,
|
|
protocol: &'a str,
|
|
) -> BoxFuture<'a, Result<SharedLink, TransportError>> {
|
|
Box::pin(async move {
|
|
Ok(impaired(
|
|
self.inner.open(network, peer, protocol).await?,
|
|
self.loss.clone(),
|
|
))
|
|
})
|
|
}
|
|
}
|
|
fn impaired(raw: SharedLink, loss: Arc<AtomicBool>) -> SharedLink {
|
|
Arc::new(FaultLink {
|
|
raw,
|
|
loss,
|
|
sequence: AtomicU64::new(0),
|
|
})
|
|
}
|
|
#[derive(Debug)]
|
|
struct FaultLink {
|
|
raw: SharedLink,
|
|
loss: Arc<AtomicBool>,
|
|
sequence: AtomicU64,
|
|
}
|
|
impl PacketLink for FaultLink {
|
|
fn network(&self) -> NetworkId {
|
|
self.raw.network()
|
|
}
|
|
fn peer(&self) -> iroh::EndpointId {
|
|
self.raw.peer()
|
|
}
|
|
fn max_datagram_size(&self) -> usize {
|
|
self.raw.max_datagram_size()
|
|
}
|
|
fn send(&self, bytes: Bytes) -> Result<(), TransportError> {
|
|
if self.loss.load(Ordering::Relaxed)
|
|
&& !self
|
|
.sequence
|
|
.fetch_add(1, Ordering::Relaxed)
|
|
.is_multiple_of(5)
|
|
{
|
|
Ok(())
|
|
} else {
|
|
self.raw.send(bytes)
|
|
}
|
|
}
|
|
fn recv(&self) -> BoxFuture<'_, Option<Bytes>> {
|
|
self.raw.recv()
|
|
}
|
|
fn closed(&self) -> BoxFuture<'_, ()> {
|
|
self.raw.closed()
|
|
}
|
|
fn is_closed(&self) -> bool {
|
|
self.raw.is_closed()
|
|
}
|
|
fn close(&self) {
|
|
self.raw.close();
|
|
}
|
|
fn path_description(&self) -> String {
|
|
self.raw.path_description()
|
|
}
|
|
}
|
|
#[tokio::test]
|
|
async fn persistent_runtime_loss_retires_the_pair_and_negotiates_fresh_parameters() {
|
|
let discovery = SharedMemoryDiscovery::new();
|
|
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
|
|
let a = Node::open(roots[0].path(), &discovery, "lossa").await;
|
|
let b = Node::open(roots[1].path(), &discovery, "lossb").await;
|
|
let (name, secret) = network("awg-loss");
|
|
let id = a.agent.join_network(&name, &secret).await.unwrap();
|
|
b.agent.join_network(&name, &secret).await.unwrap();
|
|
a.ready(id, 1).await;
|
|
b.ready(id, 1).await;
|
|
exchange(&a, &b, id).await;
|
|
let old = a.transport.overview(id)[0].attempt;
|
|
let mut events = a.agent.subscribe();
|
|
a.loss.store(true, Ordering::Relaxed);
|
|
tsunagi::testing::wait_event(&mut events, |event| match event {
|
|
tsunagi::Event::DataLinkDown {
|
|
network, reason, ..
|
|
} if *network == id && reason.contains("delivery checks") => Some(()),
|
|
_ => None,
|
|
})
|
|
.await;
|
|
a.loss.store(false, Ordering::Relaxed);
|
|
wait_until("fresh AWG profile after runtime loss", || async {
|
|
a.transport
|
|
.overview(id)
|
|
.iter()
|
|
.any(|p| p.attempt != old && !p.closed && !p.path.contains("establishing"))
|
|
.then_some(())
|
|
})
|
|
.await;
|
|
a.ready(id, 1).await;
|
|
b.ready(id, 1).await;
|
|
let initiator = if a.agent.endpoint_id().as_bytes() < b.agent.endpoint_id().as_bytes() {
|
|
&a
|
|
} else {
|
|
&b
|
|
};
|
|
let attempt = initiator.transport.overview(id)[0].attempt;
|
|
let received = initiator.transport.overview(id)[0].delivery_checks;
|
|
wait_until(
|
|
"fresh physical delivery checks answered after installation",
|
|
|| async {
|
|
(initiator.transport.overview(id).first()?.delivery_checks >= received + 3)
|
|
.then_some(())
|
|
},
|
|
)
|
|
.await;
|
|
assert_eq!(initiator.transport.overview(id)[0].attempt, attempt);
|
|
wait_for_peers(&a.agent, id, 1).await;
|
|
exchange(&a, &b, id).await;
|
|
exchange(&b, &a, id).await;
|
|
a.agent.shutdown().await;
|
|
b.agent.shutdown().await;
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn runtime_loss_falls_back_to_real_iroh_without_interrupting_control_or_ip() {
|
|
let discovery = SharedMemoryDiscovery::new();
|
|
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
|
|
let a = Node::open_with_fallback(roots[0].path(), &discovery, "fallba", true).await;
|
|
let b = Node::open_with_fallback(roots[1].path(), &discovery, "fallbb", true).await;
|
|
let (name, secret) = network("awg-fallback");
|
|
let id = a.agent.join_network(&name, &secret).await.unwrap();
|
|
b.agent.join_network(&name, &secret).await.unwrap();
|
|
a.ready(id, 1).await;
|
|
b.ready(id, 1).await;
|
|
exchange(&a, &b, id).await;
|
|
let mut events = a.agent.subscribe();
|
|
a.loss.store(true, Ordering::Relaxed);
|
|
tsunagi::testing::wait_event(&mut events, |event| match event {
|
|
tsunagi::Event::DataLinkDown {
|
|
network, reason, ..
|
|
} if *network == id && reason.contains("delivery checks") => Some(()),
|
|
_ => None,
|
|
})
|
|
.await;
|
|
for node in [&a, &b] {
|
|
wait_until("fallback carrier is wg-quic",||async {
|
|
let status=node.agent.status().await.ok()?;
|
|
let network=status.networks.iter().find(|n|n.network_id==id)?;
|
|
(network.members.iter().any(|m|matches!(&m.data_path,tsunagi::agent::DataPath::Direct{transport} if transport=="wg-quic"))).then_some(())
|
|
}).await;
|
|
wait_for_peers(&node.agent, id, 1).await;
|
|
}
|
|
exchange(&a, &b, id).await;
|
|
exchange(&b, &a, id).await;
|
|
a.agent.shutdown().await;
|
|
b.agent.shutdown().await;
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn opaque_pair_carrier_fragments_its_full_advertised_capacity() {
|
|
let discovery = SharedMemoryDiscovery::new();
|
|
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
|
|
let mut nodes = Vec::new();
|
|
for root in &roots {
|
|
let identity = tsunagi::storage::StateStore::open(
|
|
tsunagi::StoragePaths::under(root.path()).state_db(),
|
|
)
|
|
.unwrap()
|
|
.load_or_create_device_identity()
|
|
.unwrap();
|
|
let transport = AwgTransport::new(identity.endpoint_id(), Ipv4Addr::LOCALHOST);
|
|
let agent = Agent::spawn(
|
|
config_with(root.path(), &discovery)
|
|
.with_custom_transport(AWG_PROTOCOL, transport.clone()),
|
|
)
|
|
.await
|
|
.unwrap();
|
|
nodes.push((agent, transport));
|
|
}
|
|
let (name, secret) = network("awg-capacity");
|
|
let id = nodes[0].0.join_network(&name, &secret).await.unwrap();
|
|
nodes[1].0.join_network(&name, &secret).await.unwrap();
|
|
for (i, (agent, transport)) in nodes.iter().enumerate() {
|
|
wait_for_peers(agent, id, 1).await;
|
|
// This fixture tests only the transport contract: register the member
|
|
// after the real core has authenticated it, without an IP plugin/TUN.
|
|
transport.allow_peer(id, nodes[1 - i].0.endpoint_id());
|
|
}
|
|
nodes.sort_by_key(|(agent, _)| *agent.endpoint_id().as_bytes());
|
|
let outgoing = nodes[0]
|
|
.1
|
|
.open(id, nodes[1].0.endpoint_id(), AWG_PROTOCOL)
|
|
.await
|
|
.unwrap();
|
|
let incoming = tokio::time::timeout(Duration::from_secs(5), nodes[1].1.next_inbound())
|
|
.await
|
|
.unwrap()
|
|
.unwrap()
|
|
.link;
|
|
for (from, to) in [(&outgoing, &incoming), (&incoming, &outgoing)] {
|
|
let mut bytes = vec![0; from.max_datagram_size()];
|
|
rand::fill(&mut bytes[..]);
|
|
let bytes = Bytes::from(bytes);
|
|
from.send(bytes.clone()).unwrap();
|
|
assert_eq!(
|
|
tokio::time::timeout(Duration::from_secs(5), to.recv())
|
|
.await
|
|
.unwrap()
|
|
.unwrap(),
|
|
bytes
|
|
);
|
|
assert!(matches!(
|
|
from.send(Bytes::from(vec![0; from.max_datagram_size() + 1])),
|
|
Err(TransportError::TooLarge { .. })
|
|
));
|
|
}
|
|
outgoing.close();
|
|
incoming.close();
|
|
for (agent, transport) in nodes {
|
|
transport.shutdown();
|
|
agent.shutdown().await;
|
|
}
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn encrypted_health_checks_fit_a_smaller_peers_configured_interface_mtu() {
|
|
let discovery = SharedMemoryDiscovery::new();
|
|
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
|
|
let a = Node::open_config(roots[0].path(), &discovery, "smalla", false, 1280).await;
|
|
let b = Node::open_config(roots[1].path(), &discovery, "smallb", false, 576).await;
|
|
let (name, secret) = network("awg-small-mtu");
|
|
let id = a.agent.join_network(&name, &secret).await.unwrap();
|
|
b.agent.join_network(&name, &secret).await.unwrap();
|
|
a.ready(id, 1).await;
|
|
b.ready(id, 1).await;
|
|
let bytes = packet(a.addr(id), b.addr(id), 576);
|
|
a.tun().push_from_os(bytes.clone());
|
|
assert_eq!(
|
|
tokio::time::timeout(Duration::from_secs(5), b.tun().pop_to_os())
|
|
.await
|
|
.unwrap()
|
|
.unwrap(),
|
|
bytes
|
|
);
|
|
wait_until(
|
|
"physical delivery checks do not depend on overlay MTU",
|
|
|| async {
|
|
(a.transport.overview(id).first()?.delivery_checks >= 4
|
|
&& b.transport.overview(id).first()?.delivery_checks >= 4)
|
|
.then_some(())
|
|
},
|
|
)
|
|
.await;
|
|
for node in [&a, &b] {
|
|
assert_eq!(
|
|
node.plugin.overview(id).unwrap().peers[0]
|
|
.tunnel
|
|
.as_ref()
|
|
.unwrap()
|
|
.stats
|
|
.dropped_oversize,
|
|
0
|
|
);
|
|
}
|
|
a.agent.shutdown().await;
|
|
b.agent.shutdown().await;
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn disjoint_external_policies_relay_the_same_internal_tunnel_and_keep_it_when_routes_change()
|
|
{
|
|
let discovery = SharedMemoryDiscovery::new();
|
|
let roots = [
|
|
TempDir::new().unwrap(),
|
|
TempDir::new().unwrap(),
|
|
TempDir::new().unwrap(),
|
|
];
|
|
let a = Node::open_selected(
|
|
roots[0].path(),
|
|
&discovery,
|
|
"only-awg",
|
|
true,
|
|
1280,
|
|
Some(vec!["awg".into()]),
|
|
)
|
|
.await;
|
|
let b = Node::open_with_fallback(roots[1].path(), &discovery, "mixed-relay", true).await;
|
|
let c = Node::open_selected(
|
|
roots[2].path(),
|
|
&discovery,
|
|
"only-quic",
|
|
true,
|
|
1280,
|
|
Some(vec!["wg-quic".into()]),
|
|
)
|
|
.await;
|
|
let (name, secret) = network("independent-internal-encryption");
|
|
let id = a.agent.join_network(&name, &secret).await.unwrap();
|
|
b.agent.join_network(&name, &secret).await.unwrap();
|
|
c.agent.join_network(&name, &secret).await.unwrap();
|
|
for node in [&a, &b, &c] {
|
|
wait_for_peers(&node.agent, id, 2).await;
|
|
}
|
|
wait_until("AWG-only to QUIC-only through an authenticated mixed relay", || async {
|
|
let status = a.agent.network_status(id).await.ok()?;
|
|
let peer = status.peers.iter().find(|p| p.endpoint_id == c.agent.endpoint_id())?;
|
|
(matches!(peer.data_path, tsunagi::agent::DataPath::Relayed { via, .. } if via == b.agent.endpoint_id()) && a.plugin.overview(id)?.established_peers() == 2 && c.plugin.overview(id)?.established_peers() == 2).then_some(())
|
|
}).await;
|
|
exchange(&a, &c, id).await;
|
|
exchange(&c, &a, id).await;
|
|
let mut tcp = tcp::TcpPair::connect(a.tun(), a.addr(id), c.tun(), c.addr(id)).await;
|
|
tcp.exchange(b"SSH-like TCP stream over the initial relay")
|
|
.await;
|
|
let key = a.plugin.overview(id).unwrap().public_key;
|
|
let before = a
|
|
.plugin
|
|
.overview(id)
|
|
.unwrap()
|
|
.peers
|
|
.iter()
|
|
.find(|p| p.endpoint_id == c.agent.endpoint_id())
|
|
.unwrap()
|
|
.tunnel
|
|
.as_ref()
|
|
.unwrap()
|
|
.stats
|
|
.tx_packets;
|
|
// A's additional permission creates a direct QUIC route without replacing its IP tunnel.
|
|
a.agent
|
|
.set_protocols(id, vec!["wg-quic".into(), "awg".into()])
|
|
.await
|
|
.unwrap();
|
|
wait_until("direct carrier replaces the relay", || async {
|
|
let status = a.agent.network_status(id).await.ok()?;
|
|
matches!(
|
|
status
|
|
.peers
|
|
.iter()
|
|
.find(|p| p.endpoint_id == c.agent.endpoint_id())?
|
|
.data_path,
|
|
tsunagi::agent::DataPath::Direct { .. }
|
|
)
|
|
.then_some(())
|
|
})
|
|
.await;
|
|
tcp.exchange(b"same TCP socket after changing the carrier and next hop")
|
|
.await;
|
|
assert_eq!(a.plugin.overview(id).unwrap().public_key, key);
|
|
assert!(
|
|
a.plugin
|
|
.overview(id)
|
|
.unwrap()
|
|
.peers
|
|
.iter()
|
|
.find(|p| p.endpoint_id == c.agent.endpoint_id())
|
|
.unwrap()
|
|
.tunnel
|
|
.as_ref()
|
|
.unwrap()
|
|
.stats
|
|
.tx_packets
|
|
> before
|
|
);
|
|
a.agent.set_protocols(id, Vec::new()).await.unwrap();
|
|
// Poll the actual TCP stack while the route is absent: its first transmission is lost.
|
|
let pending = b"TCP retransmission survives the temporary absence of every data carrier";
|
|
tcp.queue_while_unreachable(pending);
|
|
a.agent.set_protocols(id, vec!["awg".into()]).await.unwrap();
|
|
wait_until("forbidden QUIC removed and relay restored", || async {
|
|
let status = a.agent.network_status(id).await.ok()?;
|
|
matches!(
|
|
status
|
|
.peers
|
|
.iter()
|
|
.find(|p| p.endpoint_id == c.agent.endpoint_id())?
|
|
.data_path,
|
|
tsunagi::agent::DataPath::Relayed { .. }
|
|
)
|
|
.then_some(())
|
|
})
|
|
.await;
|
|
assert_eq!(a.plugin.overview(id).unwrap().public_key, key);
|
|
tcp.receive_queued(pending).await;
|
|
tcp.exchange(b"same TCP connection after returning to AWG and the relay")
|
|
.await;
|
|
drop(tcp);
|
|
// Transit ciphertext must never be delivered to the middle agent's IP interface.
|
|
assert!(
|
|
tokio::time::timeout(Duration::from_millis(150), b.tun().pop_to_os())
|
|
.await
|
|
.is_err()
|
|
);
|
|
for node in [&a, &b, &c] {
|
|
node.agent.shutdown().await;
|
|
}
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn no_common_carrier_is_an_actionable_policy_error_without_disabling_control() {
|
|
let discovery = SharedMemoryDiscovery::new();
|
|
let roots = [TempDir::new().unwrap(), TempDir::new().unwrap()];
|
|
let a = Node::open_selected(
|
|
roots[0].path(),
|
|
&discovery,
|
|
"awg-policy",
|
|
true,
|
|
1280,
|
|
Some(vec!["awg".into()]),
|
|
)
|
|
.await;
|
|
let b = Node::open_selected(
|
|
roots[1].path(),
|
|
&discovery,
|
|
"quic-policy",
|
|
true,
|
|
1280,
|
|
Some(vec!["wg-quic".into()]),
|
|
)
|
|
.await;
|
|
let (name, secret) = network("no-common-carrier");
|
|
let id = a.agent.join_network(&name, &secret).await.unwrap();
|
|
b.agent.join_network(&name, &secret).await.unwrap();
|
|
wait_until("explain independent external policies", || async {
|
|
let status = a.agent.network_status(id).await.ok()?;
|
|
let peer = status.peers.first()?;
|
|
peer.data_error
|
|
.as_ref()?
|
|
.contains("no common external transport")
|
|
.then_some(())
|
|
})
|
|
.await;
|
|
assert_eq!(
|
|
a.agent
|
|
.network_status(id)
|
|
.await
|
|
.unwrap()
|
|
.metrics
|
|
.data_links_established,
|
|
0
|
|
);
|
|
a.agent.shutdown().await;
|
|
b.agent.shutdown().await;
|
|
}
|