Separate control and data logically, move WireGuard into userspace, add a CLI
Corrects the architecture on two points raised in review, while the project is still small enough to change cheaply. 1. Control and data are separated *logically*, not physically. The old reading — "nothing but control may ride on iroh" — threw away iroh's whole value and would have forced the data plane to reimplement STUN, ICE and a relay. Now both planes ride on iroh with different ALPNs and different connections, so the data plane inherits hole punching and relay fallback, while proto/ still knows nothing about packets and dataplane/ knows nothing about the control protocol. New boundary: PacketTransport / PacketLink, an authenticated unreliable datagram channel per (network, peer, protocol). tsunagi/data/1 runs the same membership handshake, then DataOpen/DataOpenAck, then QUIC datagrams. Only the smaller endpoint id dials, so exactly one link exists per pair. A plugin is handed links and never learns reachability, so the WireGuard announcement shrank to a public key: there is no address left to lie about. 2. WireGuard now runs in userspace, on boringtun's protocol state machine. No kernel module, no wg tool, no ip shell-out, no loopback proxy: the wgtool, backend and bridge modules are gone. Only creating a TUN device needs privileges, and that sits behind TunFactory, so the entire data plane — handshake, encryption, routing, address ownership — is tested with none. Address ownership is enforced rather than believed: outbound packets go to the owner of the destination address, inbound packets are dropped unless their source is the address derived for the peer that sent them. 3. A `tsunagi` binary: secret, doctor, id, up. It owns the runtime, the logging subscriber and Ctrl-C, which the library still refuses to. Also fixes a reference cycle where IrohTransport held Arc<Inner>, which kept the databases open and the directory lock held after shutdown; two storage tests caught it once the cycle existed. 81 tests pass offline with no privileges, including real IPv6 packets crossing a real WireGuard tunnel over real iroh connections. Verified by hand: two CLI processes forming a mesh both on loopback and via n0 discovery using only an endpoint id. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
tsunagi
Claude Opus 5
+94
-98
@@ -1,23 +1,22 @@
|
||||
//! Two agents forming a WireGuard overlay, printed step by step.
|
||||
//! Two agents forming a WireGuard overlay and exchanging a real IP packet.
|
||||
//!
|
||||
//! ```text
|
||||
//! cargo run --example wireguard_mesh
|
||||
//! ```
|
||||
//!
|
||||
//! By default it uses the in-memory backend, so it needs no privileges and
|
||||
//! changes nothing on the host: it shows the configuration each agent *would*
|
||||
//! apply. Pass `--real` to drive the actual `wg` and `ip` tools instead, which
|
||||
//! needs Linux and `CAP_NET_ADMIN`.
|
||||
//! It uses an in-memory packet interface, so it needs no privileges and
|
||||
//! changes nothing on the host: the WireGuard handshake, the encryption and
|
||||
//! the transport over iroh are all real, only the TUN device is simulated.
|
||||
|
||||
use std::net::IpAddr;
|
||||
use std::net::{IpAddr, Ipv6Addr};
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use bytes::Bytes;
|
||||
use tsunagi::config::{AgentConfig, StoragePaths, TransportPolicy};
|
||||
use tsunagi::dataplane::IpPlugin;
|
||||
use tsunagi::dataplane::wireguard::{
|
||||
AdvertisePolicy, PortPolicy, RecordingBackend, WireguardBackend, WireguardConfig,
|
||||
WireguardPlugin,
|
||||
MemoryTun, MemoryTunFactory, WireguardConfig, WireguardPlugin,
|
||||
};
|
||||
use tsunagi::discovery::SharedMemoryDiscovery;
|
||||
use tsunagi::identity::{NetworkName, NetworkSecret};
|
||||
@@ -26,32 +25,21 @@ use tsunagi::{Agent, NetworkId, Result};
|
||||
struct Node {
|
||||
agent: Agent,
|
||||
plugin: Arc<WireguardPlugin>,
|
||||
backend: Option<RecordingBackend>,
|
||||
tuns: MemoryTunFactory,
|
||||
}
|
||||
|
||||
async fn start(
|
||||
root: &std::path::Path,
|
||||
discovery: &SharedMemoryDiscovery,
|
||||
prefix: &str,
|
||||
advertise: IpAddr,
|
||||
real: bool,
|
||||
) -> Result<Node> {
|
||||
let recording = (!real).then(RecordingBackend::new);
|
||||
let backend: Arc<dyn WireguardBackend> = match &recording {
|
||||
Some(backend) => Arc::new(backend.clone()),
|
||||
None => Arc::new(
|
||||
tsunagi::dataplane::wireguard::WgToolBackend::new()
|
||||
.map_err(|err| tsunagi::Error::Discovery(err.to_string()))?,
|
||||
),
|
||||
};
|
||||
|
||||
let wireguard = WireguardConfig::new(root.join("wireguard"))
|
||||
.with_interface_prefix(prefix)
|
||||
.with_advertise(AdvertisePolicy::Explicit(vec![advertise]))
|
||||
.with_ports(PortPolicy::Fixed(if real { 51820 } else { 51821 }));
|
||||
let plugin = WireguardPlugin::open(wireguard, backend)
|
||||
.await
|
||||
.map_err(|err| tsunagi::Error::Discovery(err.to_string()))?;
|
||||
let tuns = MemoryTunFactory::new();
|
||||
let plugin = WireguardPlugin::open(
|
||||
WireguardConfig::new(root.join("wireguard")).with_interface_prefix(prefix),
|
||||
Arc::new(tuns.clone()),
|
||||
)
|
||||
.await
|
||||
.map_err(|err| tsunagi::Error::Discovery(err.to_string()))?;
|
||||
|
||||
let agent = Agent::spawn(
|
||||
AgentConfig::new(StoragePaths::under(root))
|
||||
@@ -66,49 +54,67 @@ async fn start(
|
||||
Ok(Node {
|
||||
agent,
|
||||
plugin,
|
||||
backend: recording,
|
||||
tuns,
|
||||
})
|
||||
}
|
||||
|
||||
fn report(label: &str, node: &Node, network: NetworkId) {
|
||||
let Some(view) = node.plugin.overview(network) else {
|
||||
println!("{label}: the plugin has not prepared this network yet");
|
||||
println!("{label}: not prepared yet");
|
||||
return;
|
||||
};
|
||||
println!("\n{label}");
|
||||
println!(" interface {}", view.interface);
|
||||
println!(" public key {}", view.public_key);
|
||||
println!(" interface {} (mtu {})", view.interface, view.mtu);
|
||||
println!(" public key {}", view.public_key);
|
||||
println!(
|
||||
" overlay {} in {}",
|
||||
view.overlay_address, view.overlay_prefix
|
||||
" overlay {} in {}/{}",
|
||||
view.overlay_address, view.overlay_prefix, view.overlay_prefix_len
|
||||
);
|
||||
println!(" listening on :{}", view.listen_port);
|
||||
println!(" advertising {:?}", view.advertised);
|
||||
for peer in &view.peers {
|
||||
println!(
|
||||
" peer {} -> {} via {:?}",
|
||||
peer.public_key.fmt_short(),
|
||||
peer.overlay_address,
|
||||
peer.endpoint
|
||||
);
|
||||
}
|
||||
if let Some(backend) = &node.backend
|
||||
&& let Some(state) = backend.state(&view.interface)
|
||||
{
|
||||
println!(" applied {} peer(s)", state.peers.len());
|
||||
for peer in &state.peers {
|
||||
println!(
|
||||
" AllowedIPs for {} = {:?}",
|
||||
match &peer.tunnel {
|
||||
Some(tunnel) => println!(
|
||||
" peer {} at {} — handshake {:?}, tx {} rx {}, path {}",
|
||||
peer.public_key.fmt_short(),
|
||||
peer.allowed_ips
|
||||
.iter()
|
||||
.map(ToString::to_string)
|
||||
.collect::<Vec<_>>()
|
||||
);
|
||||
peer.overlay_address,
|
||||
tunnel.health.since_handshake,
|
||||
tunnel.stats.tx_packets,
|
||||
tunnel.stats.rx_packets,
|
||||
tunnel.path
|
||||
),
|
||||
None => println!(
|
||||
" peer {} at {} — no data link yet",
|
||||
peer.public_key.fmt_short(),
|
||||
peer.overlay_address
|
||||
),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn ipv6_packet(source: Ipv6Addr, destination: Ipv6Addr, payload: &[u8]) -> Bytes {
|
||||
let mut packet = Vec::with_capacity(40 + payload.len());
|
||||
packet.push(6 << 4);
|
||||
packet.extend_from_slice(&[0, 0, 0]);
|
||||
packet.extend_from_slice(&(payload.len() as u16).to_be_bytes());
|
||||
packet.push(59);
|
||||
packet.push(64);
|
||||
packet.extend_from_slice(&source.octets());
|
||||
packet.extend_from_slice(&destination.octets());
|
||||
packet.extend_from_slice(payload);
|
||||
Bytes::from(packet)
|
||||
}
|
||||
|
||||
fn overlay_of(node: &Node, network: NetworkId) -> Option<Ipv6Addr> {
|
||||
match node.plugin.overview(network)?.overlay_address {
|
||||
IpAddr::V6(addr) => Some(addr),
|
||||
IpAddr::V4(_) => None,
|
||||
}
|
||||
}
|
||||
|
||||
fn tun_of(node: &Node, network: NetworkId) -> Option<Arc<MemoryTun>> {
|
||||
let view = node.plugin.overview(network)?;
|
||||
node.tuns.device(&view.interface)
|
||||
}
|
||||
|
||||
#[tokio::main]
|
||||
async fn main() -> Result<()> {
|
||||
tracing_subscriber::fmt()
|
||||
@@ -118,12 +124,7 @@ async fn main() -> Result<()> {
|
||||
)
|
||||
.init();
|
||||
|
||||
let real = std::env::args().any(|arg| arg == "--real");
|
||||
if real {
|
||||
println!("driving the real wg/ip tools; this needs Linux and CAP_NET_ADMIN\n");
|
||||
} else {
|
||||
println!("using the in-memory backend; nothing on this host is changed\n");
|
||||
}
|
||||
println!("in-memory packet interface: nothing on this host is changed\n");
|
||||
|
||||
let root = match tempfile::TempDir::new() {
|
||||
Ok(root) => root,
|
||||
@@ -134,22 +135,8 @@ async fn main() -> Result<()> {
|
||||
};
|
||||
let discovery = SharedMemoryDiscovery::new();
|
||||
|
||||
let alice = start(
|
||||
&root.path().join("alice"),
|
||||
&discovery,
|
||||
"wga",
|
||||
"10.88.0.1".parse().unwrap_or(IpAddr::from([10, 88, 0, 1])),
|
||||
real,
|
||||
)
|
||||
.await?;
|
||||
let bob = start(
|
||||
&root.path().join("bob"),
|
||||
&discovery,
|
||||
"wgb",
|
||||
"10.88.0.2".parse().unwrap_or(IpAddr::from([10, 88, 0, 2])),
|
||||
real,
|
||||
)
|
||||
.await?;
|
||||
let alice = start(&root.path().join("alice"), &discovery, "wga").await?;
|
||||
let bob = start(&root.path().join("bob"), &discovery, "wgb").await?;
|
||||
|
||||
let name = NetworkName::new("wireguard-demo")?;
|
||||
let secret = NetworkSecret::generate();
|
||||
@@ -162,34 +149,43 @@ async fn main() -> Result<()> {
|
||||
bob.agent.join_network(&name, &secret).await?;
|
||||
println!("network id: {network}");
|
||||
|
||||
// Wait until both sides configured one peer, under a bounded deadline.
|
||||
let deadline = std::time::Instant::now() + Duration::from_secs(20);
|
||||
let deadline = Instant::now() + Duration::from_secs(20);
|
||||
loop {
|
||||
let ready = [&alice, &bob].iter().all(|node| {
|
||||
let Some(view) = node.plugin.overview(network) else {
|
||||
return false;
|
||||
};
|
||||
if view.peers.len() != 1 {
|
||||
return false;
|
||||
}
|
||||
// With the in-memory backend we can also wait for the
|
||||
// configuration to actually be applied.
|
||||
match &node.backend {
|
||||
Some(backend) => backend
|
||||
.state(&view.interface)
|
||||
.map(|state| state.peers.len() == 1)
|
||||
.unwrap_or(false),
|
||||
None => true,
|
||||
}
|
||||
node.plugin
|
||||
.overview(network)
|
||||
.map(|view| view.established_peers() == 1)
|
||||
.unwrap_or(false)
|
||||
});
|
||||
if ready {
|
||||
break;
|
||||
}
|
||||
if std::time::Instant::now() > deadline {
|
||||
println!("\nthe overlay did not converge in time");
|
||||
break;
|
||||
if Instant::now() > deadline {
|
||||
println!("\nthe overlay did not come up in time");
|
||||
report("alice", &alice, network);
|
||||
report("bob", &bob, network);
|
||||
return Ok(());
|
||||
}
|
||||
tokio::time::sleep(Duration::from_millis(50)).await;
|
||||
}
|
||||
|
||||
// A real IP packet, encrypted by WireGuard and carried over iroh.
|
||||
if let (Some(from), Some(to), Some(tun_a), Some(tun_b)) = (
|
||||
overlay_of(&alice, network),
|
||||
overlay_of(&bob, network),
|
||||
tun_of(&alice, network),
|
||||
tun_of(&bob, network),
|
||||
) {
|
||||
tun_a.push_from_os(ipv6_packet(from, to, b"hello over the overlay"));
|
||||
match tokio::time::timeout(Duration::from_secs(5), tun_b.pop_to_os()).await {
|
||||
Ok(Some(packet)) => println!(
|
||||
"\nbob received {} bytes from {}: {:?}",
|
||||
packet.len(),
|
||||
from,
|
||||
String::from_utf8_lossy(&packet[40..])
|
||||
),
|
||||
_ => println!("\nthe packet did not arrive"),
|
||||
}
|
||||
tokio::time::sleep(Duration::from_millis(100)).await;
|
||||
}
|
||||
|
||||
report("alice", &alice, network);
|
||||
|
||||
Reference in New Issue
Block a user