Separate control and data logically, move WireGuard into userspace, add a CLI
Corrects the architecture on two points raised in review, while the project is still small enough to change cheaply. 1. Control and data are separated *logically*, not physically. The old reading — "nothing but control may ride on iroh" — threw away iroh's whole value and would have forced the data plane to reimplement STUN, ICE and a relay. Now both planes ride on iroh with different ALPNs and different connections, so the data plane inherits hole punching and relay fallback, while proto/ still knows nothing about packets and dataplane/ knows nothing about the control protocol. New boundary: PacketTransport / PacketLink, an authenticated unreliable datagram channel per (network, peer, protocol). tsunagi/data/1 runs the same membership handshake, then DataOpen/DataOpenAck, then QUIC datagrams. Only the smaller endpoint id dials, so exactly one link exists per pair. A plugin is handed links and never learns reachability, so the WireGuard announcement shrank to a public key: there is no address left to lie about. 2. WireGuard now runs in userspace, on boringtun's protocol state machine. No kernel module, no wg tool, no ip shell-out, no loopback proxy: the wgtool, backend and bridge modules are gone. Only creating a TUN device needs privileges, and that sits behind TunFactory, so the entire data plane — handshake, encryption, routing, address ownership — is tested with none. Address ownership is enforced rather than believed: outbound packets go to the owner of the destination address, inbound packets are dropped unless their source is the address derived for the peer that sent them. 3. A `tsunagi` binary: secret, doctor, id, up. It owns the runtime, the logging subscriber and Ctrl-C, which the library still refuses to. Also fixes a reference cycle where IrohTransport held Arc<Inner>, which kept the databases open and the directory lock held after shutdown; two storage tests caught it once the cycle existed. 81 tests pass offline with no privileges, including real IPv6 packets crossing a real WireGuard tunnel over real iroh connections. Verified by hand: two CLI processes forming a mesh both on loopback and via n0 discovery using only an endpoint id. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
ea7aaa2b69
commit
21be7e9b44
35 files changed
+3987
-2664
No files matched your search
+97
-11
@@ -37,6 +37,8 @@ use tokio::sync::{RwLock, broadcast, mpsc, oneshot};
|
||||
use tokio::task::JoinHandle;
|
||||
|
||||
use crate::config::{AgentConfig, Limits};
|
||||
use crate::dataplane::transport::PacketTransport;
|
||||
use crate::dataplane::transport::iroh_link::{IrohTransport, TransportContext};
|
||||
use crate::dataplane::{PluginContext, PluginRequest};
|
||||
use crate::error::{Error, Result};
|
||||
use crate::identity::{DeviceIdentity, NetworkId, NetworkKeys, NetworkName, NetworkSecret};
|
||||
@@ -82,20 +84,47 @@ struct Inner {
|
||||
shutdown: Shutdown,
|
||||
accept_task: std::sync::Mutex<Option<JoinHandle<()>>>,
|
||||
plugin_task: std::sync::Mutex<Option<JoinHandle<()>>>,
|
||||
transport: std::sync::OnceLock<Arc<dyn PacketTransport>>,
|
||||
}
|
||||
|
||||
impl Drop for Inner {
|
||||
fn drop(&mut self) {
|
||||
// Nothing here awaits; this is only a safety net for a handle that was
|
||||
// dropped without an explicit shutdown.
|
||||
self.shutdown.trigger();
|
||||
for guard in [&self.accept_task, &self.plugin_task] {
|
||||
if let Ok(mut guard) = guard.lock()
|
||||
&& let Some(task) = guard.take()
|
||||
{
|
||||
task.abort();
|
||||
/// Answers the data plane transport's questions about the agent.
|
||||
///
|
||||
/// Holds a weak reference on purpose: the transport lives inside the agent, so
|
||||
/// a strong one would be a cycle and the agent — with its open databases and
|
||||
/// its directory lock — would never be released.
|
||||
#[derive(Debug)]
|
||||
struct TransportCtx(Weak<Inner>);
|
||||
|
||||
impl TransportContext for TransportCtx {
|
||||
fn snapshot(&self) -> crate::BoxFuture<'_, HashMap<NetworkId, NetworkKeys>> {
|
||||
Box::pin(async move {
|
||||
let Some(inner) = self.0.upgrade() else {
|
||||
return HashMap::new();
|
||||
};
|
||||
inner
|
||||
.networks
|
||||
.read()
|
||||
.await
|
||||
.iter()
|
||||
.map(|(id, handle)| (*id, handle.keys.clone()))
|
||||
.collect()
|
||||
})
|
||||
}
|
||||
|
||||
fn serves<'a>(&'a self, network: NetworkId, protocol: &'a str) -> crate::BoxFuture<'a, bool> {
|
||||
Box::pin(async move {
|
||||
let Some(inner) = self.0.upgrade() else {
|
||||
return false;
|
||||
};
|
||||
if !inner.networks.read().await.contains_key(&network) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
inner
|
||||
.config
|
||||
.plugins
|
||||
.iter()
|
||||
.any(|plugin| plugin.protocol_id() == protocol)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
@@ -127,9 +156,20 @@ impl Agent {
|
||||
shutdown: Shutdown::new(),
|
||||
accept_task: std::sync::Mutex::new(None),
|
||||
plugin_task: std::sync::Mutex::new(None),
|
||||
transport: std::sync::OnceLock::new(),
|
||||
config,
|
||||
});
|
||||
|
||||
// The data plane rides on iroh too, which is where it gets hole
|
||||
// punching and relay fallback from. It is a separate ALPN and a
|
||||
// separate connection, so the two planes stay independent.
|
||||
let transport: Arc<dyn PacketTransport> = Arc::new(IrohTransport::new(
|
||||
inner.adapter.clone(),
|
||||
Arc::clone(&inner.limits),
|
||||
Arc::new(TransportCtx(Arc::downgrade(&inner))) as Arc<dyn TransportContext>,
|
||||
));
|
||||
let _ = inner.transport.set(transport);
|
||||
|
||||
if let CacheOutcome::Reset(reason) = inner.storage.cache_outcome().clone() {
|
||||
let _ = inner.events.send(Event::CacheReset { reason });
|
||||
}
|
||||
@@ -257,6 +297,7 @@ impl Agent {
|
||||
discovery_interval: self.inner.config.discovery_interval,
|
||||
plugins: self.inner.config.plugins.clone(),
|
||||
hostname: self.inner.hostname.clone(),
|
||||
transport: self.inner.transport.get().cloned(),
|
||||
});
|
||||
networks.insert(network_id, handle);
|
||||
drop(networks);
|
||||
@@ -625,6 +666,12 @@ async fn handle_incoming(inner: Arc<Inner>, incoming: iroh::endpoint::Incoming)
|
||||
};
|
||||
let peer = conn.remote_id();
|
||||
|
||||
// Two protocols share the endpoint; they are told apart here and never mix.
|
||||
if conn.alpn() == crate::proto::message::DATA_ALPN {
|
||||
handle_inbound_data(inner, conn).await;
|
||||
return;
|
||||
}
|
||||
|
||||
let (mut send, mut recv) = match conn.accept_bi().await {
|
||||
Ok(streams) => streams,
|
||||
Err(err) => {
|
||||
@@ -696,6 +743,45 @@ async fn handle_incoming(inner: Arc<Inner>, incoming: iroh::endpoint::Incoming)
|
||||
}
|
||||
}
|
||||
|
||||
/// Completes an inbound data plane connection and routes it to its network.
|
||||
async fn handle_inbound_data(inner: Arc<Inner>, conn: iroh::endpoint::Connection) {
|
||||
let Some(transport) = inner.transport.get().cloned() else {
|
||||
conn.close(5u32.into(), b"data plane not ready");
|
||||
return;
|
||||
};
|
||||
// Downcasting is avoided by keeping the accept side on the concrete type.
|
||||
let Some(iroh_transport) = transport.as_ref().as_any().downcast_ref::<IrohTransport>() else {
|
||||
conn.close(5u32.into(), b"unsupported data transport");
|
||||
return;
|
||||
};
|
||||
|
||||
let inbound = match iroh_transport.accept(conn).await {
|
||||
Ok(inbound) => inbound,
|
||||
Err(err) => {
|
||||
tracing::debug!(%err, "inbound data channel rejected");
|
||||
return;
|
||||
}
|
||||
};
|
||||
|
||||
let sender = {
|
||||
let networks = inner.networks.read().await;
|
||||
networks
|
||||
.get(&inbound.network)
|
||||
.map(|handle| handle.commands.clone())
|
||||
};
|
||||
let Some(sender) = sender else {
|
||||
// The network went away while the channel was being set up.
|
||||
return;
|
||||
};
|
||||
if sender
|
||||
.send(NetCommand::InboundLink(Box::new(inbound)))
|
||||
.await
|
||||
.is_err()
|
||||
{
|
||||
tracing::debug!("network runtime stopped before the data link was installed");
|
||||
}
|
||||
}
|
||||
|
||||
/// Picks the hostname to announce.
|
||||
///
|
||||
/// Order: explicit configuration, then what the state store already holds, then
|
||||
|
||||
Reference in new issue
Block a user