Exit nodes on Windows: use a member as exit node via destination bypass routes

Choosing an exit node on Windows did nothing: the using side was refused,
because Windows has no per-user routing and the agent's own traffic would loop
into the tunnel it carries.

Now the overlay interface gets 0.0.0.0/1 and 128.0.0.0/1 routes (and ::/1,
8000::/1 as an IPv6 blackhole), and every underlay address the agent talks to
directly gets a /32 route through the physical default gateway, marked with
route metric 7373 so stale ones are found and removed. Bypass routes are
written before the halves and removed after them; a failed step removes what
was written. All routes are in the active store.

The addresses come from the runtime: each network runtime hands the interface
the verified direct IPv4 paths of its control connections and the relay hosts
in use (Interface::set_underlay, refreshed every dial round); relay names are
resolved when the rules are applied. ExitHostRules::needs_bypass (true only on
Windows) gates it, so Linux and macOS plans never carry or re-apply on path
changes. ExitHostPlan gains `bypass` and `bypass_hosts`; networks leaving
forget their entry.

Limits: a path that appears after the last apply goes through the tunnel until
the next apply; data-plane-only addresses are not collected separately (the
data links ride the same iroh paths). Nothing was run on Windows.

Checks: cargo fmt --check, cargo clippy --locked --workspace --all-targets
-D warnings and cargo test --locked --workspace --all-targets passed on macOS;
scripts and ordering are unit-tested with an injected runner and resolver, and
Interface::set_underlay with the mock host.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

crates/tsunagi-core/src/agent/mod.rs

@@ -800,6 +800,11 @@ impl Agent {
         // itself stays, because the agent owns it and other networks may
         // still be using it.
         self.inner.routes.remove_network(network_id);
+        if let Some(interface) = self.inner.interface.get() {
+            interface
+                .set_underlay(network_id, Vec::new(), Vec::new())
+                .await;
+        }
         if let Some(interface) = self.inner.interface.get()
             && let Err(err) = interface.sync_addresses().await
         {

crates/tsunagi-core/src/agent/network.rs

@@ -712,6 +712,34 @@ impl Runtime {
         self.start_dials();
         // Last, so it carries what this round learned.
         self.introduce_peers();
+        self.refresh_underlay().await;
+    }
+
+    /// Hands the interface the underlay destinations this network depends
+    /// on: the verified direct IPv4 paths of every control connection, and
+    /// the relays in use. Only hosts that cannot exempt the agent from an
+    /// exit node's default route read it.
+    async fn refresh_underlay(&self) {
+        let Some(interface) = self.params.interface.clone() else {
+            return;
+        };
+        let mut addresses = Vec::new();
+        let mut hosts = Vec::new();
+        for session in self.sessions.values() {
+            for path in snapshot_connection(&session.conn).paths {
+                match path.remote {
+                    PathAddr::Ip(std::net::SocketAddr::V4(socket)) => addresses.push(*socket.ip()),
+                    PathAddr::Relay(url) => hosts.extend(relay_host(&url)),
+                    _ => {}
+                }
+            }
+        }
+        for url in self.params.adapter.snapshot().relay_urls {
+            hosts.extend(relay_host(&url));
+        }
+        interface
+            .set_underlay(self.network_id, addresses, hosts)
+            .await;
     }
 
     fn add_candidate(&mut self, candidate: Candidate) {
@@ -2448,3 +2476,48 @@ mod tests {
         assert_eq!(chosen_restored.copied(), Some("wg"));
     }
 }
+
+/// The host name of a relay URL such as `https://relay.example:443/`.
+fn relay_host(url: &str) -> Option<String> {
+    let rest = url.split_once("://").map_or(url, |(_, rest)| rest);
+    let authority = rest.split(['/', '?', '#']).next()?;
+    let host = authority.rsplit_once('@').map_or(authority, |(_, h)| h);
+    let host = if host.starts_with('[') {
+        host.split_once(']')?.0.trim_start_matches('[')
+    } else {
+        host.split(':').next()?
+    };
+    (!host.is_empty()
+        && host.len() <= 253
+        && host
+            .chars()
+            .all(|c| c.is_ascii_alphanumeric() || c == '.' || c == '-' || c == ':'))
+    .then(|| host.to_ascii_lowercase())
+}
+
+#[cfg(test)]
+mod relay_host_tests {
+    use super::relay_host;
+
+    #[test]
+    fn the_host_is_taken_out_of_a_relay_url() {
+        assert_eq!(
+            relay_host("https://relay.example.com./").as_deref(),
+            Some("relay.example.com.")
+        );
+        assert_eq!(
+            relay_host("https://Euw1-1.relay.iroh.network:443/x").as_deref(),
+            Some("euw1-1.relay.iroh.network")
+        );
+        assert_eq!(
+            relay_host("http://192.0.2.7:3340").as_deref(),
+            Some("192.0.2.7")
+        );
+        assert_eq!(
+            relay_host("https://[2001:db8::1]:443/").as_deref(),
+            Some("2001:db8::1")
+        );
+        assert_eq!(relay_host("https:///").as_deref(), None);
+        assert_eq!(relay_host("https://bad host/").as_deref(), None);
+    }
+}

crates/tsunagi-core/src/overlay/hostrules/exit.rs

@@ -35,6 +35,17 @@ pub struct ExitHostPlan {
     pub offer: Vec<Ipv4Range>,
     /// Whether this device sends its internet traffic through a member.
     pub client: bool,
+    /// The range of every network on the interface. Only the using side
+    /// reads it, to keep this agent's own traffic *to* the overlay inside it
+    /// where the host has no per-user routing (macOS).
+    pub overlay: Vec<Ipv4Range>,
+    /// Underlay addresses this agent talks to directly and must keep reaching
+    /// outside the overlay's default route. Only filled while `client` is set
+    /// and only for hosts that report [`ExitHostRules::needs_bypass`].
+    pub bypass: Vec<std::net::Ipv4Addr>,
+    /// Host names (relay servers) that must stay reachable the same way; the
+    /// host rules resolve them.
+    pub bypass_hosts: Vec<String>,
 }
 
 impl ExitHostPlan {
@@ -72,6 +83,13 @@ pub trait ExitHostRules: Send + Sync + std::fmt::Debug + 'static {
     /// when nothing is there: it runs on the shutdown path, and once at
     /// startup to take away what a crashed run left.
     fn clear<'a>(&'a self, interface: &'a str) -> BoxFuture<'a, ()>;
+
+    /// Whether using an exit node on this host needs the underlay addresses
+    /// the agent talks to, because the host cannot exempt the agent's own
+    /// traffic from the default route by user id (Windows). The default is no.
+    fn needs_bypass(&self) -> bool {
+        false
+    }
 }
 
 /// The tag every firewall object of one offered range carries.
@@ -99,6 +117,7 @@ pub struct MockExitRules {
     calls: Arc<Mutex<Vec<String>>>,
     forwarding: Arc<Mutex<Option<bool>>>,
     failure: Arc<Mutex<Option<String>>>,
+    needs_bypass: Arc<Mutex<bool>>,
 }
 
 impl MockExitRules {
@@ -114,6 +133,11 @@ impl MockExitRules {
         *lock(&self.forwarding) = forwarding;
     }
 
+    /// Whether this host asks for the underlay addresses, like Windows.
+    pub fn set_needs_bypass(&self, needs: bool) {
+        *lock(&self.needs_bypass) = needs;
+    }
+
     /// Makes every step fail with this reason from now on.
     pub fn fail_with(&self, reason: Option<String>) {
         *lock(&self.failure) = reason;
@@ -138,6 +162,10 @@ fn lock<T>(mutex: &Mutex<T>) -> std::sync::MutexGuard<'_, T> {
 }
 
 impl ExitHostRules for MockExitRules {
+    fn needs_bypass(&self) -> bool {
+        *lock(&self.needs_bypass)
+    }
+
     fn apply<'a>(&'a self, plan: &'a ExitHostPlan) -> BoxFuture<'a, ExitHostReport> {
         Box::pin(async move {
             lock(&self.calls).push(format!("apply:{}", plan.interface));
@@ -204,6 +232,8 @@ mod tests {
             interface: "tsun0".into(),
             offer: vec![range()],
             client: true,
+            overlay: vec![range()],
+            ..Default::default()
         };
         let report = rules.apply(&plan).await;
         assert_eq!(report.offer[0].1, RuleOutcome::Applied);
@@ -218,6 +248,8 @@ mod tests {
                 interface: "tsun0".into(),
                 offer: vec![range()],
                 client: false,
+                overlay: Vec::new(),
+                ..Default::default()
             })
             .await;
         assert_eq!(report.forwarding, Some(false));

crates/tsunagi-core/src/overlay/hostrules/macos.rs

@@ -19,6 +19,10 @@
 
 use crate::BoxFuture;
 
+use std::sync::{Arc, Mutex};
+
+use super::exit::{ExitHostPlan, ExitHostReport, ExitHostRules};
+use super::macos_exit::{apply_plan, clear_all, run_tool};
 use super::{BroadcastHostRules, BroadcastRulesPlan, BroadcastRulesReport, RuleOutcome};
 
 /// The limited-broadcast destination routed through the overlay interface.
@@ -60,14 +64,19 @@ fn firewall_todo(plan: &BroadcastRulesPlan) -> String {
     )
 }
 
-/// The macOS implementation: `route` for the route, no firewall yet.
+/// The macOS implementation: `route` for the route, no firewall yet, and
+/// `pf` plus `route` for exit nodes (see [`super::macos_exit`]).
 #[derive(Debug, Default)]
-pub struct MacosHostRules;
+pub struct MacosHostRules {
+    /// The reference `pfctl -E` handed out, held while exit-node rules need
+    /// `pf` on and released when they are removed.
+    pub(super) pf_token: Arc<Mutex<Option<String>>>,
+}
 
 impl MacosHostRules {
     /// Creates the host rules.
     pub fn new() -> Self {
-        Self
+        Self::default()
     }
 
     /// Installs the route, replacing one this agent added earlier so the call
@@ -82,11 +91,46 @@ impl MacosHostRules {
     }
 }
 
+impl ExitHostRules for MacosHostRules {
+    fn apply<'a>(&'a self, plan: &'a ExitHostPlan) -> BoxFuture<'a, ExitHostReport> {
+        Box::pin(async move {
+            let job_plan = plan.clone();
+            let token = Arc::clone(&self.pf_token);
+            tokio::task::spawn_blocking(move || apply_plan(&run_tool, &token, &job_plan))
+                .await
+                .unwrap_or_else(|err| {
+                    let failed = RuleOutcome::Failed(format!("the task failed: {err}"));
+                    ExitHostReport {
+                        offer: plan
+                            .offer
+                            .iter()
+                            .map(|range| (*range, failed.clone()))
+                            .collect(),
+                        forwarding: None,
+                        client: plan.client.then_some(failed),
+                    }
+                })
+        })
+    }
+
+    fn clear<'a>(&'a self, interface: &'a str) -> BoxFuture<'a, ()> {
+        Box::pin(async move {
+            let name = interface.to_string();
+            let token = Arc::clone(&self.pf_token);
+            let _ = tokio::task::spawn_blocking(move || clear_all(&run_tool, &token, &name)).await;
+        })
+    }
+}
+
 impl BroadcastHostRules for MacosHostRules {
     fn name(&self) -> &str {
         "route"
     }
 
+    fn exit_rules(&self) -> Option<&dyn ExitHostRules> {
+        Some(self)
+    }
+
     fn apply<'a>(&'a self, plan: &'a BroadcastRulesPlan) -> BoxFuture<'a, BroadcastRulesReport> {
         Box::pin(async move {
             let route = self.set_route(plan.interface.clone()).await;

crates/tsunagi-core/src/overlay/hostrules/macos_exit.rs

@@ -0,0 +1,915 @@
+//! Exit-node host rules on macOS: a `pf` anchor and `route` entries.
+//!
+//! Compiled on every platform so the rule text and the parsers are tested
+//! everywhere; only `MacosHostRules` runs them, and only on macOS.
+//!
+//! # Offering
+//!
+//! Per offered range, in the anchor `com.apple/tsunagi-exit-<if>`:
+//!
+//! ```text
+//! nat on <egress> inet from <range> to any -> (<egress>)
+//! pass in quick on <if> inet from <range> to ! <range>
+//! ```
+//!
+//! The anchor lives under `com.apple/` on purpose: the stock `/etc/pf.conf`
+//! evaluates `nat-anchor "com.apple/*"` and `anchor "com.apple/*"`, so the
+//! rules take effect without editing the user's ruleset. `pf` cannot say
+//! "every interface but one" in a translation address, so the egress is the
+//! interface of the default route **when the rules are applied**; after the
+//! default route moves (Wi-Fi to Ethernet) the rules are re-applied by
+//! switching the exit node off and on, or restarting the agent.
+//!
+//! The kernel's `net.inet.ip.forwarding` is only read, never written: it
+//! changes how the whole host behaves, which is the owner's decision.
+//!
+//! # Using
+//!
+//! macOS has no `uidrange` policy routing, and the agent is root, so the
+//! Linux scheme does not carry over. Instead:
+//!
+//! ```text
+//! route add -net 0.0.0.0/1   -interface <if>
+//! route add -net 128.0.0.0/1 -interface <if>
+//! pass out quick on <if> route-to (<egress> <gateway>) inet proto { tcp udp }
+//!     from any to ! <overlay ranges> user <agent uid>
+//! nat on <egress> inet from (<if>) to any -> (<egress>)
+//! ```
+//!
+//! The two halves outrank the default route without replacing it, and every
+//! more specific route (LAN, another VPN) keeps working. The `route-to` rule
+//! sends the *agent's own* traffic — QUIC, relay, DHT, which would otherwise
+//! loop into the tunnel it carries — back out the physical interface, and the
+//! translation gives it that interface's address, since an unbound socket
+//! picked the overlay address when the route pointed at the overlay. Like the
+//! Linux exemption this is by user id, so anything else running as root also
+//! bypasses the tunnel.
+//!
+//! IPv6 is blocked while an exit node is in use, as on Linux: `::/1` and
+//! `8000::/1` are rejecting routes, so applications fall back to IPv4 at once
+//! instead of leaking.
+//!
+//! Every object is derived from local state and the interface the agent owns;
+//! none of it comes from a peer.
+
+use std::net::Ipv4Addr;
+use std::sync::Mutex;
+
+use crate::state::Ipv4Range;
+
+use super::RuleOutcome;
+use super::exit::{ExitHostPlan, ExitHostReport};
+
+/// The programs this module runs, always by absolute path.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub(super) enum Tool {
+    Route,
+    Pfctl,
+    Sysctl,
+    Id,
+}
+
+impl Tool {
+    fn path(self) -> &'static str {
+        match self {
+            Self::Route => "/sbin/route",
+            Self::Pfctl => "/sbin/pfctl",
+            Self::Sysctl => "/usr/sbin/sysctl",
+            Self::Id => "/usr/bin/id",
+        }
+    }
+}
+
+/// What running a tool came to.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub(super) enum Ran {
+    /// The program is not there.
+    Missing,
+    /// It ran.
+    Exited {
+        success: bool,
+        stdout: String,
+        stderr: String,
+    },
+}
+
+/// Runs a tool with optional standard input; injectable so the logic is
+/// tested without touching the host.
+pub(super) type Run<'a> = &'a dyn Fn(Tool, &[String], Option<&str>) -> Ran;
+
+/// Runs a real tool. Blocking: callers use `spawn_blocking`.
+pub(super) fn run_tool(tool: Tool, args: &[String], stdin: Option<&str>) -> Ran {
+    use std::io::Write;
+    use std::process::{Command, Stdio};
+
+    let mut command = Command::new(tool.path());
+    command
+        .args(args)
+        .stdin(if stdin.is_some() {
+            Stdio::piped()
+        } else {
+            Stdio::null()
+        })
+        .stdout(Stdio::piped())
+        .stderr(Stdio::piped());
+    let mut child = match command.spawn() {
+        Ok(child) => child,
+        Err(err) if err.kind() == std::io::ErrorKind::NotFound => return Ran::Missing,
+        Err(err) => {
+            return Ran::Exited {
+                success: false,
+                stdout: String::new(),
+                stderr: format!("could not run {}: {err}", tool.path()),
+            };
+        }
+    };
+    if let (Some(text), Some(mut pipe)) = (stdin, child.stdin.take()) {
+        // A write error shows up as the tool's own failure below.
+        let _ = pipe.write_all(text.as_bytes());
+    }
+    match child.wait_with_output() {
+        Ok(output) => Ran::Exited {
+            success: output.status.success(),
+            stdout: String::from_utf8_lossy(&output.stdout).into_owned(),
+            stderr: String::from_utf8_lossy(&output.stderr).into_owned(),
+        },
+        Err(err) => Ran::Exited {
+            success: false,
+            stdout: String::new(),
+            stderr: format!("could not run {}: {err}", tool.path()),
+        },
+    }
+}
+
+fn owned(words: &[&str]) -> Vec<String> {
+    words.iter().map(|word| word.to_string()).collect()
+}
+
+/// A name that is safe in a `pf` rule, an anchor path and a table name.
+fn safe_name(name: &str) -> bool {
+    !name.is_empty() && name.len() <= 32 && name.chars().all(|c| c.is_ascii_alphanumeric())
+}
+
+/// The anchor holding every rule of one interface.
+fn anchor_name(interface: &str) -> Result<String, String> {
+    if safe_name(interface) {
+        Ok(format!("com.apple/tsunagi-exit-{interface}"))
+    } else {
+        Err(format!(
+            "`{interface}` is not an interface name this agent can write a pf rule for"
+        ))
+    }
+}
+
+/// The default route as `route -n get default` reports it.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub(super) struct DefaultRoute {
+    pub interface: String,
+    pub gateway: Option<Ipv4Addr>,
+}
+
+/// Reads `interface:` and `gateway:` out of `route -n get default`.
+fn parse_default_route(text: &str) -> Option<DefaultRoute> {
+    let field = |name: &str| {
+        text.lines().find_map(|line| {
+            let (key, value) = line.split_once(':')?;
+            (key.trim() == name).then(|| value.trim().to_string())
+        })
+    };
+    let interface = field("interface").filter(|name| safe_name(name))?;
+    let gateway = field("gateway").and_then(|value| value.parse().ok());
+    Some(DefaultRoute { interface, gateway })
+}
+
+fn failure(stderr: &str, stdout: &str) -> String {
+    let text = if stderr.trim().is_empty() {
+        stdout.trim()
+    } else {
+        stderr.trim()
+    };
+    if text.is_empty() {
+        "the command failed without saying why".to_string()
+    } else {
+        text.to_string()
+    }
+}
+
+/// Runs a tool and turns a failure into a sentence a person can act on.
+fn run_ok(
+    run: Run<'_>,
+    tool: Tool,
+    args: &[String],
+    stdin: Option<&str>,
+) -> Result<String, String> {
+    match run(tool, args, stdin) {
+        Ran::Missing => Err(format!("{} not found", tool.path())),
+        Ran::Exited {
+            success: true,
+            stdout,
+            ..
+        } => Ok(stdout),
+        Ran::Exited { stderr, stdout, .. } => Err(format!(
+            "{} {} failed: {}",
+            tool.path(),
+            args.join(" "),
+            failure(&stderr, &stdout)
+        )),
+    }
+}
+
+/// The route traffic leaves by now, which is the egress for translation and
+/// for the agent's own sockets. Read *before* the overlay routes are added.
+fn default_route(run: Run<'_>, overlay: &str) -> Result<DefaultRoute, String> {
+    let out = run_ok(run, Tool::Route, &owned(&["-n", "get", "default"]), None)
+        .map_err(|why| format!("cannot find the default route: {why}"))?;
+    let route = parse_default_route(&out)
+        .ok_or_else(|| "the host has no default route to send traffic out of".to_string())?;
+    if route.interface == overlay {
+        return Err(format!(
+            "the default route already goes through `{overlay}`"
+        ));
+    }
+    Ok(route)
+}
+
+/// Whether the kernel forwards IPv4. Read only.
+fn forwarding_state(run: Run<'_>) -> Option<bool> {
+    let out = run_ok(
+        run,
+        Tool::Sysctl,
+        &owned(&["-n", "net.inet.ip.forwarding"]),
+        None,
+    )
+    .ok()?;
+    match out.trim() {
+        "1" => Some(true),
+        "0" => Some(false),
+        _ => None,
+    }
+}
+
+/// The effective user id of this process, from `id -u`.
+fn agent_uid(run: Run<'_>) -> Result<u32, String> {
+    run_ok(run, Tool::Id, &owned(&["-u"]), None)
+        .ok()
+        .and_then(|out| out.trim().parse().ok())
+        .ok_or_else(|| "cannot tell which user this agent runs as".to_string())
+}
+
+/// What the using side adds to the rules.
+#[derive(Debug, Clone, PartialEq, Eq)]
+struct ClientRules<'a> {
+    uid: u32,
+    /// Every overlay range: the agent's own traffic *to* those still goes
+    /// through the interface.
+    overlay: &'a [Ipv4Range],
+}
+
+/// The text of the anchor: translation first, filtering after, which is the
+/// order `pf` insists on.
+fn pf_rules(
+    interface: &str,
+    egress: &DefaultRoute,
+    offer: &[Ipv4Range],
+    client: Option<&ClientRules<'_>>,
+) -> String {
+    let out = &egress.interface;
+    let mut nat = Vec::new();
+    let mut filter = Vec::new();
+    for range in offer {
+        nat.push(format!("nat on {out} inet from {range} to any -> ({out})"));
+        filter.push(format!(
+            "pass in quick on {interface} inet from {range} to ! {range}"
+        ));
+    }
+    let mut tables = Vec::new();
+    if let Some(client) = client {
+        nat.push(format!(
+            "nat on {out} inet from ({interface}) to any -> ({out})"
+        ));
+        let target = if client.overlay.is_empty() {
+            "any".to_string()
+        } else {
+            let ranges: Vec<String> = client.overlay.iter().map(|r| r.to_string()).collect();
+            // A table, because `! { a, b }` expands to one rule per element
+            // in older `pf`, which matches everything.
+            tables.push(format!(
+                "table <tsunagi_overlay_{interface}> const {{ {} }}",
+                ranges.join(", ")
+            ));
+            format!("! <tsunagi_overlay_{interface}>")
+        };
+        let via = match egress.gateway {
+            Some(gateway) => format!("({out} {gateway})"),
+            None => out.clone(),
+        };
+        filter.push(format!(
+            "pass out quick on {interface} route-to {via} inet proto {{ tcp udp }} \
+             from any to {target} user {}",
+            client.uid
+        ));
+    }
+    tables
+        .into_iter()
+        .chain(nat)
+        .chain(filter)
+        .collect::<Vec<_>>()
+        .join("\n")
+        + "\n"
+}
+
+/// Turns `pf` on if nobody has, holding the reference `pfctl` hands out so it
+/// can be released when the agent is done. One reference at most.
+fn acquire_pf(run: Run<'_>, slot: &Mutex<Option<String>>) -> Result<(), String> {
+    if lock(slot).is_some() {
+        return Ok(());
+    }
+    match run(Tool::Pfctl, &owned(&["-E"]), None) {
+        Ran::Missing => Err(format!("{} not found", Tool::Pfctl.path())),
+        Ran::Exited {
+            success,
+            stdout,
+            stderr,
+        } => {
+            // `pfctl -E` reports `Token : <n>` on stderr, and also says so
+            // when pf is already enabled.
+            let token = stdout.lines().chain(stderr.lines()).find_map(parse_token);
+            if !success && token.is_none() {
+                return Err(format!("cannot enable pf: {}", failure(&stderr, &stdout)));
+            }
+            *lock(slot) = token;
+            Ok(())
+        }
+    }
+}
+
+fn parse_token(line: &str) -> Option<String> {
+    let (key, value) = line.split_once(':')?;
+    let value = value.trim();
+    (key.trim().eq_ignore_ascii_case("token")
+        && !value.is_empty()
+        && value.chars().all(|c| c.is_ascii_digit()))
+    .then(|| value.to_string())
+}
+
+fn release_pf(run: Run<'_>, slot: &Mutex<Option<String>>) {
+    if let Some(token) = lock(slot).take()
+        && let Err(why) = run_ok(run, Tool::Pfctl, &["-X".to_string(), token], None)
+    {
+        tracing::debug!(%why, "cannot release the pf reference");
+    }
+}
+
+fn lock<T>(mutex: &Mutex<T>) -> std::sync::MutexGuard<'_, T> {
+    match mutex.lock() {
+        Ok(guard) => guard,
+        Err(poisoned) => poisoned.into_inner(),
+    }
+}
+
+fn flush_args(anchor: &str) -> Vec<String> {
+    owned(&["-q", "-a", anchor, "-F", "all"])
+}
+
+fn load_args(anchor: &str) -> Vec<String> {
+    owned(&["-q", "-a", anchor, "-f", "-"])
+}
+
+/// The two halves that out-rank the default route.
+const HALVES: [&str; 2] = ["0.0.0.0/1", "128.0.0.0/1"];
+/// The IPv6 halves, made unreachable.
+const HALVES_V6: [&str; 2] = ["::/1", "8000::/1"];
+
+fn route_add_args(half: &str, interface: &str) -> Vec<String> {
+    owned(&["-q", "-n", "add", "-net", half, "-interface", interface])
+}
+
+fn route_del_args(half: &str, interface: &str) -> Vec<String> {
+    owned(&["-q", "-n", "delete", "-net", half, "-interface", interface])
+}
+
+fn route_add_v6_args(half: &str) -> Vec<String> {
+    owned(&["-q", "-n", "add", "-inet6", "-net", half, "::1", "-reject"])
+}
+
+fn route_del_v6_args(half: &str) -> Vec<String> {
+    owned(&["-q", "-n", "delete", "-inet6", "-net", half, "::1"])
+}
+
+/// Takes away the overlay default routes. Best effort: what is not there is
+/// the state wanted.
+fn clear_routes(run: Run<'_>, interface: &str) {
+    for half in HALVES {
+        let _ = run(Tool::Route, &route_del_args(half, interface), None);
+    }
+    for half in HALVES_V6 {
+        let _ = run(Tool::Route, &route_del_v6_args(half), None);
+    }
+}
+
+fn set_routes(run: Run<'_>, interface: &str) -> Result<(), String> {
+    clear_routes(run, interface);
+    for half in HALVES {
+        if let Err(why) = run_ok(run, Tool::Route, &route_add_args(half, interface), None) {
+            clear_routes(run, interface);
+            return Err(why);
+        }
+    }
+    for half in HALVES_V6 {
+        // A host without IPv6 has nothing to block; not a failure.
+        if let Err(why) = run_ok(run, Tool::Route, &route_add_v6_args(half), None) {
+            tracing::debug!(%why, "cannot block IPv6 for the exit node");
+        }
+    }
+    Ok(())
+}
+
+/// Makes the host match the plan.
+pub(super) fn apply_plan(
+    run: Run<'_>,
+    token: &Mutex<Option<String>>,
+    plan: &ExitHostPlan,
+) -> ExitHostReport {
+    let forwarding = if plan.offer.is_empty() {
+        None
+    } else {
+        forwarding_state(run)
+    };
+    let outcome = apply_rules(run, token, plan);
+    let verdict = |ok: bool, why: &str| {
+        if ok {
+            RuleOutcome::Applied
+        } else {
+            RuleOutcome::Failed(why.to_string())
+        }
+    };
+    match outcome {
+        Ok(()) => ExitHostReport {
+            offer: plan
+                .offer
+                .iter()
+                .map(|range| (*range, RuleOutcome::Applied))
+                .collect(),
+            forwarding,
+            client: plan.client.then_some(RuleOutcome::Applied),
+        },
+        Err(why) => ExitHostReport {
+            offer: plan
+                .offer
+                .iter()
+                .map(|range| (*range, verdict(false, &why)))
+                .collect(),
+            forwarding,
+            client: plan.client.then(|| verdict(false, &why)),
+        },
+    }
+}
+
+fn apply_rules(
+    run: Run<'_>,
+    token: &Mutex<Option<String>>,
+    plan: &ExitHostPlan,
+) -> Result<(), String> {
+    let interface = plan.interface.as_str();
+    let anchor = anchor_name(interface)?;
+    let egress = default_route(run, interface)?;
+    let uid = if plan.client {
+        Some(agent_uid(run)?)
+    } else {
+        None
+    };
+    let client = uid.map(|uid| ClientRules {
+        uid,
+        overlay: &plan.overlay,
+    });
+    let rules = pf_rules(interface, &egress, &plan.offer, client.as_ref());
+
+    acquire_pf(run, token)?;
+    run_ok(run, Tool::Pfctl, &load_args(&anchor), Some(&rules))
+        .map_err(|why| format!("cannot load the pf rules: {why}"))?;
+    // The routes come after the rules that keep the agent's own traffic off
+    // them: the other order would loop for as long as it takes.
+    if plan.client {
+        set_routes(run, interface)?;
+    } else {
+        clear_routes(run, interface);
+    }
+    Ok(())
+}
+
+/// Removes everything this agent put there for the interface.
+pub(super) fn clear_all(run: Run<'_>, token: &Mutex<Option<String>>, interface: &str) {
+    clear_routes(run, interface);
+    if let Ok(anchor) = anchor_name(interface) {
+        // Flushing an anchor nothing was ever loaded into is not an error
+        // worth reporting.
+        if let Err(why) = run_ok(run, Tool::Pfctl, &flush_args(&anchor), None) {
+            tracing::debug!(%why, "cannot flush the exit node pf anchor");
+        }
+    }
+    release_pf(run, token);
+}
+
+#[cfg(test)]
+mod tests {
+    #![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
+
+    use std::cell::RefCell;
+
+    use super::*;
+
+    fn range() -> Ipv4Range {
+        "10.13.37.0/24".parse().unwrap()
+    }
+
+    const DEFAULT_ROUTE: &str = "   route to: default\n\
+destination: default\n\
+       mask: default\n\
+    gateway: 192.168.1.1\n\
+  interface: en0\n\
+      flags: <UP,GATEWAY,DONE,STATIC,PRCLONING,GLOBAL>\n";
+
+    fn egress() -> DefaultRoute {
+        DefaultRoute {
+            interface: "en0".into(),
+            gateway: Some(Ipv4Addr::new(192, 168, 1, 1)),
+        }
+    }
+
+    /// A scripted host that records every call.
+    struct Fake {
+        calls: RefCell<Vec<String>>,
+        stdins: RefCell<Vec<String>>,
+        default_route: Option<String>,
+        forwarding: &'static str,
+        pf_missing: bool,
+        pf_load_fails: bool,
+        route_add_fails: bool,
+        pf_enable_output: &'static str,
+    }
+
+    impl Fake {
+        fn new() -> Self {
+            Self {
+                calls: RefCell::new(Vec::new()),
+                stdins: RefCell::new(Vec::new()),
+                default_route: Some(DEFAULT_ROUTE.to_string()),
+                forwarding: "1\n",
+                pf_missing: false,
+                pf_load_fails: false,
+                route_add_fails: false,
+                pf_enable_output: "pf enabled\nToken : 4242\n",
+            }
+        }
+
+        fn run(&self, tool: Tool, args: &[String], stdin: Option<&str>) -> Ran {
+            let line = format!("{:?} {}", tool, args.join(" "));
+            self.calls.borrow_mut().push(line);
+            if let Some(text) = stdin {
+                self.stdins.borrow_mut().push(text.to_string());
+            }
+            let ok = |stdout: &str, stderr: &str| Ran::Exited {
+                success: true,
+                stdout: stdout.to_string(),
+                stderr: stderr.to_string(),
+            };
+            let fail = |why: &str| Ran::Exited {
+                success: false,
+                stdout: String::new(),
+                stderr: why.to_string(),
+            };
+            match tool {
+                Tool::Id => ok("0\n", ""),
+                Tool::Sysctl => ok(self.forwarding, ""),
+                Tool::Route if args.first().map(String::as_str) == Some("-n") => {
+                    match &self.default_route {
+                        Some(text) => ok(text, ""),
+                        None => fail("route: writing to routing socket: not in table"),
+                    }
+                }
+                Tool::Route if args.contains(&"add".to_string()) && self.route_add_fails => {
+                    fail("route: writing to routing socket: File exists")
+                }
+                Tool::Route => ok("", ""),
+                Tool::Pfctl if self.pf_missing => Ran::Missing,
+                Tool::Pfctl if args == ["-E"] => ok("", self.pf_enable_output),
+                Tool::Pfctl if args.contains(&"-f".to_string()) && self.pf_load_fails => {
+                    fail("pfctl: Syntax error")
+                }
+                Tool::Pfctl => ok("", ""),
+            }
+        }
+
+        fn calls(&self) -> Vec<String> {
+            self.calls.borrow().clone()
+        }
+
+        fn position(&self, needle: &str) -> Option<usize> {
+            self.calls.borrow().iter().position(|c| c.contains(needle))
+        }
+
+        fn count(&self, needle: &str) -> usize {
+            self.calls
+                .borrow()
+                .iter()
+                .filter(|c| c.contains(needle))
+                .count()
+        }
+    }
+
+    fn plan(offer: bool, client: bool) -> ExitHostPlan {
+        ExitHostPlan {
+            interface: "utun5".into(),
+            offer: if offer { vec![range()] } else { Vec::new() },
+            client,
+            overlay: vec![range()],
+            ..Default::default()
+        }
+    }
+
+    #[test]
+    fn the_default_route_is_read_from_route_output() {
+        assert_eq!(parse_default_route(DEFAULT_ROUTE), Some(egress()));
+        // A point-to-point default has no gateway line to use.
+        let ppp = "destination: default\n  interface: ppp0\n";
+        assert_eq!(
+            parse_default_route(ppp),
+            Some(DefaultRoute {
+                interface: "ppp0".into(),
+                gateway: None
+            })
+        );
+        assert_eq!(parse_default_route("route: not in table"), None);
+        // An interface name that is not plain letters and digits never ends
+        // up in a rule.
+        assert_eq!(parse_default_route("interface: en0; pass all\n"), None);
+    }
+
+    #[test]
+    fn the_anchor_is_under_com_apple_and_only_for_plain_names() {
+        assert_eq!(
+            anchor_name("utun5").unwrap(),
+            "com.apple/tsunagi-exit-utun5"
+        );
+        assert!(anchor_name("").is_err());
+        assert!(anchor_name("utun5 pass all").is_err());
+        assert!(anchor_name("../x").is_err());
+    }
+
+    #[test]
+    fn offering_translates_the_range_and_admits_it_in() {
+        let text = pf_rules("utun5", &egress(), &[range()], None);
+        assert_eq!(
+            text,
+            "nat on en0 inet from 10.13.37.0/24 to any -> (en0)\n\
+             pass in quick on utun5 inet from 10.13.37.0/24 to ! 10.13.37.0/24\n"
+        );
+        // Translation precedes filtering, as pf requires.
+        assert!(text.find("nat on").unwrap() < text.find("pass in").unwrap());
+    }
+
+    #[test]
+    fn using_sends_the_agents_own_traffic_back_out_the_physical_interface() {
+        let other: Ipv4Range = "10.99.0.0/16".parse().unwrap();
+        let overlay = [range(), other];
+        let client = ClientRules {
+            uid: 0,
+            overlay: &overlay,
+        };
+        let text = pf_rules("utun5", &egress(), &[], Some(&client));
+        assert!(
+            text.starts_with(
+                "table <tsunagi_overlay_utun5> const { 10.13.37.0/24, 10.99.0.0/16 }\n"
+            ),
+            "{text}"
+        );
+        assert!(
+            text.contains("nat on en0 inet from (utun5) to any -> (en0)\n"),
+            "{text}"
+        );
+        assert!(
+            text.contains(
+                "pass out quick on utun5 route-to (en0 192.168.1.1) inet proto { tcp udp } \
+                 from any to ! <tsunagi_overlay_utun5> user 0\n"
+            ),
+            "{text}"
+        );
+        // Only the agent's user, and only what is not the overlay.
+        assert!(!text.contains("from any to any"));
+
+        let direct = DefaultRoute {
+            interface: "ppp0".into(),
+            gateway: None,
+        };
+        let text = pf_rules(
+            "utun5",
+            &direct,
+            &[],
+            Some(&ClientRules {
+                uid: 501,
+                overlay: &[],
+            }),
+        );
+        assert!(text.contains("route-to ppp0 inet"), "{text}");
+        assert!(text.contains("to any user 501"), "{text}");
+        assert!(!text.contains("table"), "{text}");
+    }
+
+    #[test]
+    fn the_overlay_default_routes_are_halves_and_ipv6_is_rejected() {
+        assert_eq!(
+            route_add_args("0.0.0.0/1", "utun5").join(" "),
+            "-q -n add -net 0.0.0.0/1 -interface utun5"
+        );
+        assert_eq!(
+            route_del_args("128.0.0.0/1", "utun5").join(" "),
+            "-q -n delete -net 128.0.0.0/1 -interface utun5"
+        );
+        assert_eq!(
+            route_add_v6_args("8000::/1").join(" "),
+            "-q -n add -inet6 -net 8000::/1 ::1 -reject"
+        );
+        assert_eq!(
+            route_del_v6_args("::/1").join(" "),
+            "-q -n delete -inet6 -net ::/1 ::1"
+        );
+    }
+
+    #[test]
+    fn the_pf_token_is_found_in_either_stream() {
+        assert_eq!(parse_token("Token : 1234"), Some("1234".into()));
+        assert_eq!(parse_token("token: 9"), Some("9".into()));
+        assert_eq!(parse_token("pf enabled"), None);
+        assert_eq!(parse_token("Token : x; rm"), None);
+    }
+
+    #[test]
+    fn offering_loads_the_anchor_and_reads_but_never_writes_forwarding() {
+        let fake = Fake::new();
+        let token = Mutex::new(None);
+        let report = apply_plan(&|t, a, s| fake.run(t, a, s), &token, &plan(true, false));
+        assert_eq!(report.offer, vec![(range(), RuleOutcome::Applied)]);
+        assert_eq!(report.forwarding, Some(true));
+        assert_eq!(report.client, None);
+        let calls = fake.calls();
+        assert!(
+            calls
+                .iter()
+                .any(|c| c == "Pfctl -q -a com.apple/tsunagi-exit-utun5 -f -"),
+            "{calls:?}"
+        );
+        assert_eq!(
+            fake.stdins.borrow()[0],
+            "nat on en0 inet from 10.13.37.0/24 to any -> (en0)\n\
+             pass in quick on utun5 inet from 10.13.37.0/24 to ! 10.13.37.0/24\n"
+        );
+        // sysctl is only ever asked, never told.
+        assert!(
+            calls
+                .iter()
+                .filter(|c| c.starts_with("Sysctl"))
+                .all(|c| c == "Sysctl -n net.inet.ip.forwarding"),
+            "{calls:?}"
+        );
+        // No client routes were added for an offer alone.
+        assert_eq!(fake.count(" add "), 0);
+        assert_eq!(token.lock().unwrap().as_deref(), Some("4242"));
+    }
+
+    #[test]
+    fn forwarding_off_is_reported_and_unreadable_is_none() {
+        let mut fake = Fake::new();
+        fake.forwarding = "0\n";
+        let token = Mutex::new(None);
+        let report = apply_plan(&|t, a, s| fake.run(t, a, s), &token, &plan(true, false));
+        assert_eq!(report.forwarding, Some(false));
+        fake.forwarding = "junk";
+        let report = apply_plan(&|t, a, s| fake.run(t, a, s), &token, &plan(true, false));
+        assert_eq!(report.forwarding, None);
+    }
+
+    #[test]
+    fn the_client_rules_are_loaded_before_the_routes_that_need_them() {
+        let fake = Fake::new();
+        let token = Mutex::new(None);
+        let report = apply_plan(&|t, a, s| fake.run(t, a, s), &token, &plan(false, true));
+        assert_eq!(report.client, Some(RuleOutcome::Applied));
+        assert!(report.offer.is_empty());
+        let load = fake.position("-f -").unwrap();
+        let add = fake.position(" add -net 0.0.0.0/1").unwrap();
+        assert!(load < add, "{:?}", fake.calls());
+        assert!(fake.position("add -net 128.0.0.0/1").is_some());
+        assert!(fake.position("-inet6 -net ::/1 ::1 -reject").is_some());
+        assert!(fake.stdins.borrow()[0].contains("user 0"));
+        // The default route was read before any overlay route existed.
+        assert!(fake.position("-n get default").unwrap() < add);
+    }
+
+    #[test]
+    fn no_routes_are_added_when_the_rules_did_not_load() {
+        let mut fake = Fake::new();
+        fake.pf_load_fails = true;
+        let token = Mutex::new(None);
+        let report = apply_plan(&|t, a, s| fake.run(t, a, s), &token, &plan(true, true));
+        let Some(RuleOutcome::Failed(why)) = report.client else {
+            panic!("{report:?}");
+        };
+        assert!(why.contains("cannot load the pf rules"), "{why}");
+        assert!(why.contains("Syntax error"), "{why}");
+        assert!(matches!(report.offer[0].1, RuleOutcome::Failed(_)));
+        // Nothing sent the host's traffic into a tunnel it could not exempt
+        // the agent from.
+        assert_eq!(fake.count(" add "), 0);
+    }
+
+    #[test]
+    fn a_failing_route_is_a_failure_and_leaves_nothing_behind() {
+        let mut fake = Fake::new();
+        fake.route_add_fails = true;
+        let token = Mutex::new(None);
+        let report = apply_plan(&|t, a, s| fake.run(t, a, s), &token, &plan(false, true));
+        let Some(RuleOutcome::Failed(why)) = report.client else {
+            panic!("{report:?}");
+        };
+        assert!(why.contains("File exists"), "{why}");
+        // The cleanup after the failed add removed whatever did land.
+        let last_add = fake
+            .calls()
+            .iter()
+            .rposition(|c| c.contains(" add "))
+            .unwrap();
+        assert!(
+            fake.calls()[last_add..]
+                .iter()
+                .any(|c| c.contains("delete"))
+        );
+    }
+
+    #[test]
+    fn a_missing_pfctl_or_default_route_says_what_is_wrong() {
+        let mut fake = Fake::new();
+        fake.pf_missing = true;
+        let token = Mutex::new(None);
+        let report = apply_plan(&|t, a, s| fake.run(t, a, s), &token, &plan(true, false));
+        let RuleOutcome::Failed(why) = &report.offer[0].1 else {
+            panic!("{report:?}");
+        };
+        assert!(why.contains("/sbin/pfctl not found"), "{why}");
+
+        let mut fake = Fake::new();
+        fake.default_route = None;
+        let report = apply_plan(&|t, a, s| fake.run(t, a, s), &token, &plan(true, false));
+        let RuleOutcome::Failed(why) = &report.offer[0].1 else {
+            panic!("{report:?}");
+        };
+        assert!(why.contains("default route"), "{why}");
+        assert_eq!(fake.count("-f -"), 0);
+    }
+
+    #[test]
+    fn a_default_route_through_the_overlay_is_refused() {
+        let mut fake = Fake::new();
+        fake.default_route = Some("gateway: 10.13.37.1\ninterface: utun5\n".into());
+        let token = Mutex::new(None);
+        let report = apply_plan(&|t, a, s| fake.run(t, a, s), &token, &plan(true, false));
+        let RuleOutcome::Failed(why) = &report.offer[0].1 else {
+            panic!("{report:?}");
+        };
+        assert!(why.contains("already goes through"), "{why}");
+    }
+
+    #[test]
+    fn pf_is_enabled_once_and_released_once_on_clear() {
+        let fake = Fake::new();
+        let token = Mutex::new(None);
+        let run = |t, a: &[String], s: Option<&str>| fake.run(t, a, s);
+        apply_plan(&run, &token, &plan(true, false));
+        apply_plan(&run, &token, &plan(true, true));
+        assert_eq!(fake.count("Pfctl -E"), 1, "{:?}", fake.calls());
+
+        clear_all(&run, &token, "utun5");
+        assert_eq!(fake.count("Pfctl -X 4242"), 1);
+        assert!(
+            fake.position("-a com.apple/tsunagi-exit-utun5 -F all")
+                .is_some()
+        );
+        assert!(
+            fake.position("delete -net 0.0.0.0/1 -interface utun5")
+                .is_some()
+        );
+        assert_eq!(*token.lock().unwrap(), None);
+        // A second clear has nothing to release and still succeeds.
+        clear_all(&run, &token, "utun5");
+        assert_eq!(fake.count("Pfctl -X"), 1);
+    }
+
+    #[test]
+    fn dropping_the_client_removes_its_routes() {
+        let fake = Fake::new();
+        let token = Mutex::new(None);
+        let run = |t, a: &[String], s: Option<&str>| fake.run(t, a, s);
+        apply_plan(&run, &token, &plan(true, false));
+        assert!(fake.position("delete -net 0.0.0.0/1").is_some());
+        assert_eq!(fake.count(" add "), 0);
+    }
+}

crates/tsunagi-core/src/overlay/hostrules/mod.rs

@@ -61,11 +61,16 @@ pub use linux::LinuxHostRules;
 // Compiled everywhere so the scripts it builds are tested everywhere; only
 // the type that runs them is exported on Windows.
 mod windows;
+#[cfg_attr(not(target_os = "windows"), allow(dead_code))]
+mod windows_exit;
 #[cfg(all(feature = "tun-device", target_os = "windows"))]
 pub use windows::WindowsHostRules;
 
+// Compiled everywhere so its rule text and parsers are tested everywhere.
 #[cfg(all(feature = "tun-device", target_os = "macos"))]
 mod macos;
+#[cfg_attr(not(target_os = "macos"), allow(dead_code))]
+mod macos_exit;
 #[cfg(all(feature = "tun-device", target_os = "macos"))]
 pub use macos::MacosHostRules;
 

crates/tsunagi-core/src/overlay/hostrules/windows.rs

@@ -29,6 +29,8 @@
 
 use crate::BoxFuture;
 
+use super::exit::{ExitHostPlan, ExitHostReport, ExitHostRules};
+use super::windows_exit;
 use super::{BroadcastHostRules, BroadcastRulesPlan, BroadcastRulesReport, RuleOutcome, tag_for};
 
 /// A PowerShell single-quoted literal, with any embedded quote doubled.
@@ -113,6 +115,11 @@ fn powershell_path() -> std::path::PathBuf {
 
 /// Runs a script and returns what it said on failure.
 fn run_powershell(script: &str) -> Result<(), String> {
+    run_powershell_output(script).map(|_| ())
+}
+
+/// Runs a script and returns what it printed, or what it said on failure.
+fn run_powershell_output(script: &str) -> Result<String, String> {
     let output = std::process::Command::new(powershell_path())
         .args([
             "-NoProfile",
@@ -125,7 +132,7 @@ fn run_powershell(script: &str) -> Result<(), String> {
         .output()
         .map_err(|err| format!("could not run PowerShell: {err}"))?;
     if output.status.success() {
-        return Ok(());
+        return Ok(String::from_utf8_lossy(&output.stdout).into_owned());
     }
     let stderr = String::from_utf8_lossy(&output.stderr);
     let text = if stderr.trim().is_empty() {
@@ -155,11 +162,73 @@ impl WindowsHostRules {
     }
 }
 
+/// The IPv4 addresses a host name resolves to; empty when it does not.
+fn resolve_host(host: &str) -> Vec<std::net::Ipv4Addr> {
+    use std::net::{IpAddr, ToSocketAddrs};
+    (host, 443)
+        .to_socket_addrs()
+        .map(|addrs| {
+            addrs
+                .filter_map(|addr| match addr.ip() {
+                    IpAddr::V4(ip) => Some(ip),
+                    IpAddr::V6(_) => None,
+                })
+                .collect()
+        })
+        .unwrap_or_default()
+}
+
+impl ExitHostRules for WindowsHostRules {
+    fn needs_bypass(&self) -> bool {
+        true
+    }
+
+    fn apply<'a>(&'a self, plan: &'a ExitHostPlan) -> BoxFuture<'a, ExitHostReport> {
+        Box::pin(async move {
+            let job = plan.clone();
+            tokio::task::spawn_blocking(move || {
+                windows_exit::apply_plan(
+                    &|script| run_powershell_output(script),
+                    &resolve_host,
+                    &job,
+                )
+            })
+            .await
+            .unwrap_or_else(|err| {
+                let failed = RuleOutcome::Failed(format!("the task failed: {err}"));
+                ExitHostReport {
+                    offer: plan
+                        .offer
+                        .iter()
+                        .map(|range| (*range, failed.clone()))
+                        .collect(),
+                    forwarding: None,
+                    client: plan.client.then_some(failed),
+                }
+            })
+        })
+    }
+
+    fn clear<'a>(&'a self, interface: &'a str) -> BoxFuture<'a, ()> {
+        Box::pin(async move {
+            let name = interface.to_string();
+            let _ = tokio::task::spawn_blocking(move || {
+                windows_exit::clear_all(&|script| run_powershell_output(script), &name)
+            })
+            .await;
+        })
+    }
+}
+
 impl BroadcastHostRules for WindowsHostRules {
     fn name(&self) -> &str {
         "windows-netsecurity"
     }
 
+    fn exit_rules(&self) -> Option<&dyn ExitHostRules> {
+        Some(self)
+    }
+
     fn apply<'a>(&'a self, plan: &'a BroadcastRulesPlan) -> BoxFuture<'a, BroadcastRulesReport> {
         Box::pin(async move {
             let route = step("route", route_script(plan)).await;

crates/tsunagi-core/src/overlay/hostrules/windows_exit.rs

@@ -0,0 +1,603 @@
+//! Exit-node host rules on Windows: WinNAT through PowerShell.
+//!
+//! Compiled on every platform so the scripts and the plan logic are tested
+//! everywhere; only `WindowsHostRules` runs them, and only on Windows. It
+//! could not be run against a real Windows host while it was written.
+//!
+//! # Offering
+//!
+//! One NAT object per offered range, named with the usual tag
+//! `tsunagi-exit:<if>:<range>`:
+//!
+//! ```text
+//! New-NetNat -Name <tag> -InternalIPInterfaceAddressPrefix <range>
+//! ```
+//!
+//! WinNAT translates that range out of whichever interface the traffic leaves
+//! by, which is the Windows counterpart of the Linux masquerade. Windows
+//! forwards per interface, and the egress is not known here, so forwarding is
+//! only *read* for the overlay interface; turning it on for the interfaces
+//! involved is the owner's decision (`Set-NetIPInterface -Forwarding Enabled`).
+//! Some Windows versions allow a single NAT per host: a second offered range
+//! then reports its own failure.
+//!
+//! # Using
+//!
+//! Windows has no per-user or per-process routing, so the Linux scheme (an
+//! exemption for the agent's user) does not carry over. Instead the agent's
+//! own underlay traffic is exempted by *destination*:
+//!
+//! ```text
+//! New-NetRoute 0.0.0.0/1   on the overlay interface, metric 1
+//! New-NetRoute 128.0.0.0/1 on the overlay interface, metric 1
+//! New-NetRoute <ip>/32 via the physical default gateway, metric 7373
+//! ```
+//!
+//! The halves out-rank the default route without replacing it, so every more
+//! specific route (LAN, another VPN) keeps working. A `/32` per address the
+//! agent talks to directly (the verified direct paths of its connections, and
+//! the relays it uses, resolved when the rules are applied) keeps that traffic
+//! on the physical network instead of looping into the tunnel it carries. The
+//! metric 7373 marks those routes as ours, which is how stale ones are found
+//! and removed. Everything is written to the active store, so a reboot clears
+//! it. IPv6 is blackholed with `::/1` and `8000::/1` on the overlay interface,
+//! which has no IPv6: applications fall back to IPv4 after a delay, where
+//! Linux and macOS answer at once. The bypass follows the agent's paths as
+//! they change; a path that appears after the last apply is reached through
+//! the tunnel until the next one.
+//!
+//! The bypass routes are written before the halves, and removed after them.
+
+use super::RuleOutcome;
+use std::net::Ipv4Addr;
+
+use super::exit::{ExitHostPlan, ExitHostReport, exit_tag, exit_tag_prefix};
+use crate::state::Ipv4Range;
+
+/// The route metric that marks a bypass route as this agent's.
+const BYPASS_METRIC: u32 = 7373;
+
+/// Runs a PowerShell script and returns what it printed.
+pub(super) type RunScript<'a> = &'a dyn Fn(&str) -> Result<String, String>;
+
+/// Resolves a host name to its IPv4 addresses.
+pub(super) type Resolve<'a> = &'a dyn Fn(&str) -> Vec<Ipv4Addr>;
+
+/// A PowerShell single-quoted literal, with any embedded quote doubled.
+fn ps_literal(value: &str) -> String {
+    format!("'{}'", value.replace('\'', "''"))
+}
+
+/// Creates the NAT of one range, replacing one this agent made before.
+fn nat_script(interface: &str, range: &Ipv4Range) -> String {
+    let name = ps_literal(&exit_tag(interface, range));
+    let prefix = ps_literal(&range.to_string());
+    format!(
+        "$ErrorActionPreference = 'Stop'\n\
+         $name = {name}\n\
+         Get-NetNat -Name $name -ErrorAction SilentlyContinue | \
+         Remove-NetNat -Confirm:$false -ErrorAction SilentlyContinue\n\
+         New-NetNat -Name $name -InternalIPInterfaceAddressPrefix {prefix} | Out-Null\n"
+    )
+}
+
+/// Removes this interface's NAT objects except those named in `keep`.
+fn prune_script(interface: &str, keep: &[Ipv4Range]) -> String {
+    let prefix = ps_literal(&exit_tag_prefix(interface));
+    let keep: Vec<String> = keep
+        .iter()
+        .map(|range| ps_literal(&exit_tag(interface, range)))
+        .collect();
+    format!(
+        "$ErrorActionPreference = 'SilentlyContinue'\n\
+         $prefix = {prefix}\n\
+         $keep = @({})\n\
+         Get-NetNat | Where-Object {{ $_.Name.StartsWith($prefix) -and \
+         ($keep -notcontains $_.Name) }} | Remove-NetNat -Confirm:$false\n\
+         exit 0\n",
+        keep.join(", ")
+    )
+}
+
+/// Prints `Enabled` or `Disabled` for the interface's IPv4 forwarding.
+fn forwarding_script(interface: &str) -> String {
+    let alias = ps_literal(interface);
+    format!(
+        "$ErrorActionPreference = 'Stop'\n\
+         (Get-NetIPInterface -InterfaceAlias {alias} -AddressFamily IPv4).Forwarding\n"
+    )
+}
+
+/// Prints `<interface index> <next hop>` of the best default route that is
+/// not through the overlay interface.
+fn gateway_script(interface: &str) -> String {
+    let alias = ps_literal(interface);
+    format!(
+        "$ErrorActionPreference = 'Stop'\n\
+         $alias = {alias}\n\
+         $best = Get-NetRoute -DestinationPrefix '0.0.0.0/0' -AddressFamily IPv4 \
+         -PolicyStore ActiveStore | Where-Object {{ $_.InterfaceAlias -ne $alias -and \
+         $_.NextHop -ne '0.0.0.0' }} | Sort-Object {{ $_.RouteMetric + \
+         (Get-NetIPInterface -InterfaceIndex $_.InterfaceIndex -AddressFamily IPv4).InterfaceMetric }} | \
+         Select-Object -First 1\n\
+         if (-not $best) {{ throw 'the host has no default gateway outside the overlay' }}\n\
+         \"$($best.InterfaceIndex) $($best.NextHop)\"\n"
+    )
+}
+
+/// The default gateway as the script above printed it.
+fn parse_gateway(text: &str) -> Option<(u32, Ipv4Addr)> {
+    let mut words = text.split_whitespace();
+    let index = words.next()?.parse().ok()?;
+    let hop = words.next()?.parse().ok()?;
+    words.next().is_none().then_some((index, hop))
+}
+
+/// Replaces the bypass routes with one per address, through the gateway.
+fn bypass_script(index: u32, gateway: Ipv4Addr, addresses: &[Ipv4Addr]) -> String {
+    let list: Vec<String> = addresses
+        .iter()
+        .map(|address| ps_literal(&address.to_string()))
+        .collect();
+    format!(
+        "$ErrorActionPreference = 'Stop'\n\
+         Get-NetRoute -AddressFamily IPv4 -PolicyStore ActiveStore -ErrorAction SilentlyContinue | \
+         Where-Object {{ $_.RouteMetric -eq {BYPASS_METRIC} -and $_.DestinationPrefix.EndsWith('/32') }} | \
+         Remove-NetRoute -Confirm:$false -ErrorAction SilentlyContinue\n\
+         foreach ($ip in @({})) {{\n\
+         New-NetRoute -DestinationPrefix \"$ip/32\" -InterfaceIndex {index} -NextHop '{gateway}' \
+         -RouteMetric {BYPASS_METRIC} -PolicyStore ActiveStore -ErrorAction SilentlyContinue | Out-Null\n\
+         }}\n",
+        list.join(", ")
+    )
+}
+
+/// Sends everything else through the overlay interface.
+fn halves_script(interface: &str) -> String {
+    let alias = ps_literal(interface);
+    format!(
+        "$ErrorActionPreference = 'Stop'\n\
+         $alias = {alias}\n\
+         foreach ($p in '0.0.0.0/1', '128.0.0.0/1') {{\n\
+         Get-NetRoute -DestinationPrefix $p -InterfaceAlias $alias -PolicyStore ActiveStore \
+         -ErrorAction SilentlyContinue | Remove-NetRoute -Confirm:$false -ErrorAction SilentlyContinue\n\
+         New-NetRoute -DestinationPrefix $p -InterfaceAlias $alias -NextHop '0.0.0.0' \
+         -RouteMetric 1 -PolicyStore ActiveStore | Out-Null\n\
+         }}\n\
+         foreach ($p in '::/1', '8000::/1') {{\n\
+         Get-NetRoute -DestinationPrefix $p -InterfaceAlias $alias -PolicyStore ActiveStore \
+         -ErrorAction SilentlyContinue | Remove-NetRoute -Confirm:$false -ErrorAction SilentlyContinue\n\
+         New-NetRoute -DestinationPrefix $p -InterfaceAlias $alias -NextHop '::' \
+         -RouteMetric 1 -PolicyStore ActiveStore -ErrorAction SilentlyContinue | Out-Null\n\
+         }}\n"
+    )
+}
+
+/// Removes the halves and every bypass route. What is not there is the state
+/// wanted.
+fn remove_client_script(interface: &str) -> String {
+    let alias = ps_literal(interface);
+    format!(
+        "$ErrorActionPreference = 'SilentlyContinue'\n\
+         $alias = {alias}\n\
+         foreach ($p in '0.0.0.0/1', '128.0.0.0/1', '::/1', '8000::/1') {{\n\
+         Get-NetRoute -DestinationPrefix $p -InterfaceAlias $alias -PolicyStore ActiveStore | \
+         Remove-NetRoute -Confirm:$false\n\
+         }}\n\
+         Get-NetRoute -AddressFamily IPv4 -PolicyStore ActiveStore | \
+         Where-Object {{ $_.RouteMetric -eq {BYPASS_METRIC} -and $_.DestinationPrefix.EndsWith('/32') }} | \
+         Remove-NetRoute -Confirm:$false\n\
+         exit 0\n"
+    )
+}
+
+/// Every address to keep direct: the ones given and what the names resolve
+/// to, sorted and without duplicates.
+fn bypass_addresses(
+    resolve: Resolve<'_>,
+    addresses: &[Ipv4Addr],
+    hosts: &[String],
+) -> Vec<Ipv4Addr> {
+    let mut all: std::collections::BTreeSet<Ipv4Addr> = addresses.iter().copied().collect();
+    for host in hosts {
+        all.extend(resolve(host));
+    }
+    all.into_iter()
+        .filter(|a| !a.is_loopback() && !a.is_unspecified() && !a.is_multicast())
+        .collect()
+}
+
+fn apply_client(
+    run: RunScript<'_>,
+    resolve: Resolve<'_>,
+    plan: &ExitHostPlan,
+) -> Result<(), String> {
+    let interface = plan.interface.as_str();
+    let gateway = run(&gateway_script(interface))?;
+    let (index, hop) = parse_gateway(&gateway)
+        .ok_or_else(|| format!("cannot read the default gateway from `{}`", gateway.trim()))?;
+    let addresses = bypass_addresses(resolve, &plan.bypass, &plan.bypass_hosts);
+    run(&bypass_script(index, hop, &addresses))?;
+    if let Err(why) = run(&halves_script(interface)) {
+        // Do not leave a half-installed set: bypass routes alone are harmless
+        // but the next apply starts clean.
+        let _ = run(&remove_client_script(interface));
+        return Err(why);
+    }
+    Ok(())
+}
+
+fn parse_forwarding(text: &str) -> Option<bool> {
+    match text.trim().to_ascii_lowercase().as_str() {
+        "enabled" | "1" => Some(true),
+        "disabled" | "0" => Some(false),
+        _ => None,
+    }
+}
+
+/// Makes the host match the plan.
+pub(super) fn apply_plan(
+    run: RunScript<'_>,
+    resolve: Resolve<'_>,
+    plan: &ExitHostPlan,
+) -> ExitHostReport {
+    let interface = plan.interface.as_str();
+    // What an earlier plan left and this one no longer wants.
+    let pruned = run(&prune_script(interface, &plan.offer));
+    let offer = plan
+        .offer
+        .iter()
+        .map(|range| {
+            let outcome = match &pruned {
+                Err(why) => RuleOutcome::Failed(why.clone()),
+                Ok(_) => match run(&nat_script(interface, range)) {
+                    Ok(_) => RuleOutcome::Applied,
+                    Err(why) => RuleOutcome::Failed(why),
+                },
+            };
+            (*range, outcome)
+        })
+        .collect();
+    let client = if plan.client {
+        Some(match apply_client(run, resolve, plan) {
+            Ok(()) => RuleOutcome::Applied,
+            Err(why) => RuleOutcome::Failed(why),
+        })
+    } else {
+        // Whatever an earlier plan left must not stay in force.
+        let _ = run(&remove_client_script(interface));
+        None
+    };
+    let forwarding = if plan.offer.is_empty() {
+        None
+    } else {
+        run(&forwarding_script(interface))
+            .ok()
+            .and_then(|text| parse_forwarding(&text))
+    };
+    ExitHostReport {
+        offer,
+        forwarding,
+        client,
+    }
+}
+
+/// Removes every NAT object of this agent's for the interface.
+pub(super) fn clear_all(run: RunScript<'_>, interface: &str) {
+    if let Err(why) = run(&prune_script(interface, &[])) {
+        tracing::warn!(interface, "cannot remove the exit node NAT: {why}");
+    }
+    if let Err(why) = run(&remove_client_script(interface)) {
+        tracing::warn!(interface, "cannot remove the exit node routes: {why}");
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    #![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
+
+    use std::cell::RefCell;
+
+    use super::*;
+
+    fn range() -> Ipv4Range {
+        "10.13.37.0/24".parse().unwrap()
+    }
+
+    fn plan(client: bool) -> ExitHostPlan {
+        ExitHostPlan {
+            interface: "tsun0".into(),
+            offer: vec![range()],
+            client,
+            overlay: vec![range()],
+            ..Default::default()
+        }
+    }
+
+    #[test]
+    fn the_nat_is_named_with_the_tag_and_replaces_its_own() {
+        let script = nat_script("tsun0", &range());
+        assert!(
+            script.contains("$name = 'tsunagi-exit:tsun0:10.13.37.0/24'"),
+            "{script}"
+        );
+        assert!(script.contains("-InternalIPInterfaceAddressPrefix '10.13.37.0/24'"));
+        assert!(
+            script.find("Remove-NetNat").unwrap() < script.find("New-NetNat").unwrap(),
+            "{script}"
+        );
+    }
+
+    #[test]
+    fn a_quote_in_a_value_cannot_escape_the_literal() {
+        let script = nat_script("x'; Remove-Item C:\\ #", &range());
+        assert!(script.contains("'tsunagi-exit:x''; Remove-Item C:\\ #:10.13.37.0/24'"));
+        assert!(prune_script("x'y", &[]).contains("'tsunagi-exit:x''y:'"));
+        assert!(forwarding_script("x'y").contains("'x''y'"));
+    }
+
+    #[test]
+    fn pruning_keeps_wanted_ranges_and_only_selects_this_interfaces_nats() {
+        let script = prune_script("tsun0", &[range()]);
+        assert!(script.contains("$prefix = 'tsunagi-exit:tsun0:'"));
+        assert!(script.contains("$keep = @('tsunagi-exit:tsun0:10.13.37.0/24')"));
+        assert!(script.contains("StartsWith($prefix)"));
+        assert!(script.contains("exit 0"));
+        assert!(prune_script("tsun0", &[]).contains("$keep = @()"));
+    }
+
+    #[test]
+    fn forwarding_text_is_read_never_written() {
+        assert_eq!(parse_forwarding("Enabled\r\n"), Some(true));
+        assert_eq!(parse_forwarding("Disabled"), Some(false));
+        assert_eq!(parse_forwarding("what"), None);
+        let script = forwarding_script("tsun0");
+        assert!(script.contains("Get-NetIPInterface"));
+        assert!(!script.contains("Set-NetIPInterface"));
+    }
+
+    #[test]
+    fn offering_prunes_then_creates_and_reads_forwarding() {
+        let calls = RefCell::new(Vec::new());
+        let run = |script: &str| -> Result<String, String> {
+            calls.borrow_mut().push(script.to_string());
+            Ok(if script.contains("Get-NetIPInterface") {
+                "Disabled\n".to_string()
+            } else {
+                String::new()
+            })
+        };
+        let report = apply_plan(&run, &no_names, &plan(false));
+        assert_eq!(report.offer, vec![(range(), RuleOutcome::Applied)]);
+        assert_eq!(report.forwarding, Some(false));
+        assert_eq!(report.client, None);
+        let calls = calls.borrow();
+        assert!(calls[0].contains("StartsWith"), "{calls:?}");
+        assert!(calls[1].contains("New-NetNat"), "{calls:?}");
+        // Without a client, the routes of an earlier plan are taken away.
+        assert!(calls[2].contains("RouteMetric -eq 7373"), "{calls:?}");
+        assert!(calls[3].contains("Get-NetIPInterface"), "{calls:?}");
+    }
+
+    #[test]
+    fn a_failing_nat_is_that_ranges_failure() {
+        let run = |script: &str| -> Result<String, String> {
+            if script.contains("New-NetNat") {
+                Err("A NAT with this prefix already exists".into())
+            } else {
+                Ok(String::new())
+            }
+        };
+        let report = apply_plan(&run, &no_names, &plan(false));
+        let RuleOutcome::Failed(why) = &report.offer[0].1 else {
+            panic!("{report:?}");
+        };
+        assert!(why.contains("already exists"), "{why}");
+    }
+
+    fn no_names(_: &str) -> Vec<Ipv4Addr> {
+        Vec::new()
+    }
+
+    fn client_plan() -> ExitHostPlan {
+        ExitHostPlan {
+            interface: "tsun0".into(),
+            offer: Vec::new(),
+            client: true,
+            overlay: vec![range()],
+            bypass: vec![Ipv4Addr::new(138, 201, 61, 182)],
+            bypass_hosts: vec!["relay.example".into()],
+        }
+    }
+
+    #[test]
+    fn the_gateway_is_read_from_the_script_output() {
+        assert_eq!(
+            parse_gateway("12 192.168.1.1\r\n"),
+            Some((12, Ipv4Addr::new(192, 168, 1, 1)))
+        );
+        assert_eq!(parse_gateway(""), None);
+        assert_eq!(parse_gateway("12"), None);
+        assert_eq!(parse_gateway("12 gw"), None);
+        assert_eq!(parse_gateway("12 1.1.1.1 extra"), None);
+        let script = gateway_script("tsun0");
+        assert!(script.contains("-ne $alias"));
+        assert!(script.contains("-ne '0.0.0.0'"));
+    }
+
+    #[test]
+    fn bypass_routes_are_marked_host_routes_through_the_physical_gateway() {
+        let script = bypass_script(
+            12,
+            Ipv4Addr::new(192, 168, 1, 1),
+            &[Ipv4Addr::new(1, 2, 3, 4), Ipv4Addr::new(5, 6, 7, 8)],
+        );
+        assert!(script.contains("@('1.2.3.4', '5.6.7.8')"), "{script}");
+        assert!(script.contains("-InterfaceIndex 12 -NextHop '192.168.1.1'"));
+        assert!(script.contains("-RouteMetric 7373"));
+        assert!(script.contains("-PolicyStore ActiveStore"));
+        // Stale marked routes go first, and only /32 ones with our metric.
+        assert!(script.find("Remove-NetRoute").unwrap() < script.find("New-NetRoute").unwrap());
+        assert!(script.contains("RouteMetric -eq 7373"));
+        assert!(script.contains("EndsWith('/32')"));
+    }
+
+    #[test]
+    fn the_halves_go_through_the_overlay_and_ipv6_is_blackholed() {
+        let script = halves_script("tsun0");
+        assert!(script.contains("'0.0.0.0/1', '128.0.0.0/1'"));
+        assert!(script.contains("-InterfaceAlias $alias -NextHop '0.0.0.0'"));
+        assert!(script.contains("'::/1', '8000::/1'"));
+        assert!(script.contains("-PolicyStore ActiveStore"));
+        assert!(
+            !script.contains("0.0.0.0/0"),
+            "the default route is left alone"
+        );
+        assert!(halves_script("x'y").contains("'x''y'"));
+    }
+
+    #[test]
+    fn addresses_are_resolved_merged_and_cleaned() {
+        let resolve = |host: &str| -> Vec<Ipv4Addr> {
+            match host {
+                "relay.example" => vec![Ipv4Addr::new(9, 9, 9, 9), Ipv4Addr::new(1, 2, 3, 4)],
+                _ => Vec::new(),
+            }
+        };
+        let all = bypass_addresses(
+            &resolve,
+            &[
+                Ipv4Addr::new(1, 2, 3, 4),
+                Ipv4Addr::LOCALHOST,
+                Ipv4Addr::UNSPECIFIED,
+            ],
+            &["relay.example".into(), "gone.example".into()],
+        );
+        assert_eq!(
+            all,
+            vec![Ipv4Addr::new(1, 2, 3, 4), Ipv4Addr::new(9, 9, 9, 9)]
+        );
+    }
+
+    #[test]
+    fn the_bypass_is_written_before_the_halves_that_need_it() {
+        let calls = RefCell::new(Vec::new());
+        let run = |script: &str| -> Result<String, String> {
+            calls.borrow_mut().push(script.to_string());
+            Ok(if script.contains("Select-Object -First 1") {
+                "12 192.168.1.1\n".to_string()
+            } else {
+                String::new()
+            })
+        };
+        let resolve = |_: &str| vec![Ipv4Addr::new(9, 9, 9, 9)];
+        let report = apply_plan(&run, &resolve, &client_plan());
+        assert_eq!(report.client, Some(RuleOutcome::Applied));
+        let calls = calls.borrow();
+        let at = |needle: &str| calls.iter().position(|c| c.contains(needle)).unwrap();
+        assert!(at("Select-Object -First 1") < at("-RouteMetric 7373"));
+        assert!(at("-RouteMetric 7373") < at("'0.0.0.0/1', '128.0.0.0/1'"));
+        let bypass = &calls[at("-RouteMetric 7373")];
+        assert!(
+            bypass.contains("'9.9.9.9'") && bypass.contains("'138.201.61.182'"),
+            "{bypass}"
+        );
+    }
+
+    #[test]
+    fn nothing_goes_through_the_overlay_when_the_bypass_cannot_be_written() {
+        let run = |script: &str| -> Result<String, String> {
+            if script.contains("Select-Object -First 1") {
+                Ok("12 192.168.1.1".to_string())
+            } else if script.contains("-RouteMetric 7373") && script.contains("foreach ($ip") {
+                Err("Access is denied".to_string())
+            } else {
+                assert!(!script.contains("-NextHop '0.0.0.0'"), "{script}");
+                Ok(String::new())
+            }
+        };
+        let report = apply_plan(&run, &no_names, &client_plan());
+        assert_eq!(
+            report.client,
+            Some(RuleOutcome::Failed("Access is denied".into()))
+        );
+    }
+
+    #[test]
+    fn a_missing_gateway_is_a_failure_that_installs_no_route() {
+        let run = |script: &str| -> Result<String, String> {
+            assert!(!script.contains("New-NetRoute"), "{script}");
+            if script.contains("Select-Object -First 1") {
+                Err("the host has no default gateway outside the overlay".into())
+            } else {
+                Ok(String::new())
+            }
+        };
+        let report = apply_plan(&run, &no_names, &client_plan());
+        let Some(RuleOutcome::Failed(why)) = report.client else {
+            panic!("{report:?}");
+        };
+        assert!(why.contains("no default gateway"), "{why}");
+    }
+
+    #[test]
+    fn a_failed_halves_step_removes_what_was_written() {
+        let calls = RefCell::new(Vec::new());
+        let run = |script: &str| -> Result<String, String> {
+            calls.borrow_mut().push(script.to_string());
+            if script.contains("Select-Object -First 1") {
+                Ok("12 192.168.1.1".to_string())
+            } else if script.contains("'128.0.0.0/1'") && script.contains("New-NetRoute") {
+                Err("boom".to_string())
+            } else {
+                Ok(String::new())
+            }
+        };
+        let report = apply_plan(&run, &no_names, &client_plan());
+        assert!(matches!(report.client, Some(RuleOutcome::Failed(_))));
+        assert!(calls.borrow().last().unwrap().contains("Remove-NetRoute"));
+    }
+
+    #[test]
+    fn dropping_the_client_removes_its_routes_and_clearing_does_too() {
+        let calls = RefCell::new(Vec::new());
+        let run = |script: &str| -> Result<String, String> {
+            calls.borrow_mut().push(script.to_string());
+            Ok(String::new())
+        };
+        let report = apply_plan(&run, &no_names, &plan(false));
+        assert_eq!(report.client, None);
+        assert!(
+            calls
+                .borrow()
+                .iter()
+                .any(|c| c.contains("RouteMetric -eq 7373"))
+        );
+        let script = remove_client_script("tsun0");
+        assert!(script.contains("'0.0.0.0/1', '128.0.0.0/1', '::/1', '8000::/1'"));
+        assert!(script.contains("exit 0"));
+        // Only routes on our interface, or marked ours, are touched.
+        assert!(script.contains("-InterfaceAlias $alias"));
+        calls.borrow_mut().clear();
+        clear_all(&run, "tsun0");
+        assert!(
+            calls
+                .borrow()
+                .iter()
+                .any(|c| c.contains("RouteMetric -eq 7373"))
+        );
+    }
+
+    #[test]
+    fn clearing_prunes_everything_for_the_interface() {
+        let seen = RefCell::new(Vec::<String>::new());
+        let run = |script: &str| -> Result<String, String> {
+            seen.borrow_mut().push(script.to_string());
+            Ok(String::new())
+        };
+        clear_all(&run, "tsun0");
+        let seen = seen.borrow();
+        assert!(seen[0].contains("$keep = @()"));
+        assert!(seen[0].contains("'tsunagi-exit:tsun0:'"));
+    }
+}

crates/tsunagi-core/src/overlay/interface.rs

@@ -164,6 +164,17 @@ pub struct Interface {
     rules_report: std::sync::Mutex<Option<BroadcastRulesReport>>,
     /// What the exit-node rules were last asked for, and what came of it.
     exit_state: std::sync::Mutex<ExitState>,
+    /// Where each network's runtime reaches its peers and relays outside the
+    /// overlay; read only by hosts that cannot exempt the agent from the
+    /// exit node's default route.
+    underlay: std::sync::Mutex<std::collections::HashMap<NetworkId, Underlay>>,
+}
+
+/// Underlay destinations one network's runtime currently depends on.
+#[derive(Debug, Default, Clone, PartialEq, Eq)]
+struct Underlay {
+    addresses: std::collections::BTreeSet<std::net::Ipv4Addr>,
+    hosts: std::collections::BTreeSet<String>,
 }
 
 /// The exit-node host rules as this interface last left them.
@@ -275,6 +286,7 @@ impl Interface {
             host_rules,
             rules_report: std::sync::Mutex::new(None),
             exit_state: std::sync::Mutex::new(ExitState::default()),
+            underlay: std::sync::Mutex::new(std::collections::HashMap::new()),
         })
     }
 
@@ -350,10 +362,20 @@ impl Interface {
             .collect();
         ranges.sort_by_key(|range| (range.base, range.prefix_len));
         ranges.dedup();
+        let client = self.routes.exit_via().is_some();
+        let overlay = self.routes.overlay_ranges();
+        let (bypass, bypass_hosts) = if client && exit.needs_bypass() {
+            self.underlay_targets(&overlay)
+        } else {
+            (Vec::new(), Vec::new())
+        };
         let wanted = ExitHostPlan {
             interface: self.name.clone(),
             offer: ranges,
-            client: self.routes.exit_via().is_some(),
+            client,
+            overlay,
+            bypass,
+            bypass_hosts,
         };
 
         let first = !std::mem::replace(&mut self.exit_state().cleaned, true);
@@ -398,9 +420,15 @@ impl Interface {
             tracing::warn!(
                 interface = %self.name,
                 "kernel forwarding is off for this interface, so nothing is forwarded for the \
-                 exit node yet: `sysctl -w net.ipv4.conf.{0}.forwarding=1` (or \
-                 `net.ipv4.ip_forward=1`), and set it in sysctl.d to keep it",
-                self.name
+                 exit node yet: {}",
+                if cfg!(target_os = "windows") {
+                    "`Set-NetIPInterface -Forwarding Enabled` on the interfaces involved"
+                } else if cfg!(target_os = "macos") {
+                    "`sysctl -w net.inet.ip.forwarding=1`, set again after every reboot"
+                } else {
+                    "`sysctl -w net.ipv4.ip_forward=1` (or `net.ipv4.conf.<interface>.forwarding=1`), \
+                     and set it in sysctl.d to keep it"
+                }
             );
         }
         match &report.client {
@@ -419,6 +447,68 @@ impl Interface {
         state.report = Some(report);
     }
 
+    /// Tells the interface where a network's runtime reaches peers and
+    /// relays outside the overlay. When the host needs those kept out of the
+    /// exit node's default route and the set changed, the rules follow.
+    pub async fn set_underlay(
+        &self,
+        network: NetworkId,
+        addresses: impl IntoIterator<Item = std::net::Ipv4Addr>,
+        hosts: impl IntoIterator<Item = String>,
+    ) {
+        let new = Underlay {
+            addresses: addresses.into_iter().collect(),
+            hosts: hosts.into_iter().collect(),
+        };
+        let changed = {
+            let mut map = match self.underlay.lock() {
+                Ok(guard) => guard,
+                Err(poisoned) => poisoned.into_inner(),
+            };
+            if new == Underlay::default() {
+                map.remove(&network).is_some()
+            } else {
+                map.insert(network, new.clone()) != Some(new)
+            }
+        };
+        let needed = changed
+            && self.routes.exit_via().is_some()
+            && self
+                .host_rules
+                .as_ref()
+                .and_then(|rules| rules.exit_rules())
+                .is_some_and(|exit| exit.needs_bypass());
+        if needed {
+            self.sync_exit_rules().await;
+        }
+    }
+
+    /// Every underlay destination worth keeping direct, minus anything that
+    /// is the overlay itself, loopback or link-local.
+    fn underlay_targets(
+        &self,
+        overlay: &[crate::state::Ipv4Range],
+    ) -> (Vec<std::net::Ipv4Addr>, Vec<String>) {
+        let map = match self.underlay.lock() {
+            Ok(guard) => guard,
+            Err(poisoned) => poisoned.into_inner(),
+        };
+        let mut addresses = std::collections::BTreeSet::new();
+        let mut hosts = std::collections::BTreeSet::new();
+        for underlay in map.values() {
+            addresses.extend(underlay.addresses.iter().copied().filter(|address| {
+                !address.is_loopback()
+                    && !address.is_link_local()
+                    && !address.is_multicast()
+                    && !address.is_unspecified()
+                    && !address.is_broadcast()
+                    && !overlay.iter().any(|range| range.contains(*address))
+            }));
+            hosts.extend(underlay.hosts.iter().cloned());
+        }
+        (addresses.into_iter().collect(), hosts.into_iter().collect())
+    }
+
     fn exit_state(&self) -> std::sync::MutexGuard<'_, ExitState> {
         match self.exit_state.lock() {
             Ok(guard) => guard,
@@ -443,7 +533,7 @@ impl Interface {
                 "this overlay interface is not on the host (no TUN device), so there is \
                  nothing to forward to",
             ),
-            Some(false) => Some("exit nodes are only supported on Linux yet"),
+            Some(false) => Some("exit nodes are not supported on this platform yet"),
             Some(true) => None,
         };
         let failed = |detail: &str| RuleSetReport {
@@ -1336,6 +1426,65 @@ mod tests {
         assert_eq!(interface.counters().received, 1);
     }
 
+    #[tokio::test]
+    async fn the_underlay_reaches_the_rules_only_for_hosts_that_need_it_and_only_while_using_one() {
+        let rules = MockHostRules::new();
+        let (interface, routes, id) = managed_interface(&rules, false).await;
+        let relay = || vec!["relay.example".to_string()];
+        let direct = || {
+            vec![
+                Ipv4Addr::new(138, 201, 61, 182),
+                Ipv4Addr::new(10, 13, 37, 9), // the overlay itself
+                Ipv4Addr::LOCALHOST,
+                Ipv4Addr::new(169, 254, 1, 1),
+            ]
+        };
+
+        // A host that does not ask never gets them, so Linux and macOS do not
+        // re-apply their rules whenever a path moves.
+        routes.set_exit(id, using(peer(2)));
+        interface.set_underlay(id, direct(), relay()).await;
+        interface.sync_exit_rules().await;
+        let plan = rules.exit().installed().unwrap();
+        assert!(plan.client);
+        assert!(plan.bypass.is_empty() && plan.bypass_hosts.is_empty());
+
+        // A host that does gets the usable ones, and follows changes.
+        rules.exit().set_needs_bypass(true);
+        interface.set_underlay(id, direct(), relay()).await;
+        interface.sync_exit_rules().await;
+        let plan = rules.exit().installed().unwrap();
+        assert_eq!(plan.bypass, vec![Ipv4Addr::new(138, 201, 61, 182)]);
+        assert_eq!(plan.bypass_hosts, relay());
+        let applies = || {
+            rules
+                .exit()
+                .calls()
+                .iter()
+                .filter(|c| c.starts_with("apply"))
+                .count()
+        };
+        let before = applies();
+        interface.set_underlay(id, direct(), relay()).await;
+        assert_eq!(applies(), before, "an unchanged set is not applied again");
+        interface
+            .set_underlay(id, vec![Ipv4Addr::new(5, 6, 7, 8)], relay())
+            .await;
+        assert_eq!(applies(), before + 1);
+        assert_eq!(
+            rules.exit().installed().unwrap().bypass,
+            vec![Ipv4Addr::new(5, 6, 7, 8)]
+        );
+
+        // Not using an exit node: nothing to keep direct.
+        routes.set_exit(id, Default::default());
+        interface.sync_exit_rules().await;
+        interface
+            .set_underlay(id, vec![Ipv4Addr::new(1, 1, 1, 1)], relay())
+            .await;
+        assert!(rules.exit().installed().is_none());
+    }
+
     #[tokio::test]
     async fn the_exit_rules_follow_the_routing_table_and_leave_with_the_interface() {
         let rules = MockHostRules::new();

crates/tsunagi-core/src/overlay/router.rs

@@ -293,6 +293,18 @@ impl RoutingTable {
         offers
     }
 
+    /// The range of every network in the table that has agreed one, sorted.
+    pub fn overlay_ranges(&self) -> Vec<Ipv4Range> {
+        let mut ranges: Vec<Ipv4Range> = self
+            .read()
+            .values()
+            .filter_map(|routes| routes.range)
+            .collect();
+        ranges.sort_by_key(|range| (range.base, range.prefix_len));
+        ranges.dedup();
+        ranges
+    }
+
     /// One network's exit-node settings and its range, if it is in the table.
     pub fn exit_settings(&self, network: NetworkId) -> Option<(ExitPolicy, Option<Ipv4Range>)> {
         self.read()

crates/tsunagi-tray/src/exit.rs

@@ -13,6 +13,12 @@ pub(crate) const GOOD: egui::Color32 = egui::Color32::from_rgb(0x3c, 0xb0, 0x4a)
 pub(crate) const LOUD: egui::Color32 = egui::Color32::from_rgb(0xe0, 0x4a, 0x3c);
 
 /// What to run to turn kernel forwarding on, which the agent never does.
+#[cfg(target_os = "macos")]
+pub(crate) const FORWARDING_COMMAND: &str = "sudo sysctl -w net.inet.ip.forwarding=1";
+#[cfg(target_os = "windows")]
+pub(crate) const FORWARDING_COMMAND: &str =
+    "Get-NetIPInterface | Set-NetIPInterface -Forwarding Enabled";
+#[cfg(not(any(target_os = "macos", target_os = "windows")))]
 pub(crate) const FORWARDING_COMMAND: &str = "sudo sysctl -w net.ipv4.ip_forward=1";
 
 /// One thing wrong, loud enough to read at a glance.

crates/tsunagi/src/main.rs

@@ -2085,15 +2085,29 @@ async fn broadcast_command(
 }
 
 /// What to run to turn kernel forwarding on, which the agent never does.
+#[cfg(target_os = "macos")]
+const FORWARDING_COMMAND: &str = "sudo sysctl -w net.inet.ip.forwarding=1";
+#[cfg(target_os = "windows")]
+const FORWARDING_COMMAND: &str = "Get-NetIPInterface | Set-NetIPInterface -Forwarding Enabled   # in an administrator PowerShell";
+#[cfg(not(any(target_os = "macos", target_os = "windows")))]
 const FORWARDING_COMMAND: &str = "sudo sysctl -w net.ipv4.ip_forward=1";
 
+/// How to keep forwarding on across reboots.
+#[cfg(target_os = "macos")]
+const FORWARDING_PERSIST: &str = "macOS forgets it at reboot; set it again from a boot script \
+                                  such as a LaunchDaemon";
+#[cfg(target_os = "windows")]
+const FORWARDING_PERSIST: &str = "the setting is kept across reboots";
+#[cfg(not(any(target_os = "macos", target_os = "windows")))]
+const FORWARDING_PERSIST: &str =
+    "keep it across reboots with `net.ipv4.ip_forward = 1` in /etc/sysctl.d/99-tsunagi.conf";
+
 /// The warning for an exit node whose rules are in place while the kernel is
 /// not forwarding, so that nothing passes.
 fn forwarding_off_note() -> String {
     format!(
         "kernel forwarding is OFF, so nothing passes yet: run `{FORWARDING_COMMAND}` \
-         (and keep it across reboots with `net.ipv4.ip_forward = 1` in \
-         /etc/sysctl.d/99-tsunagi.conf)"
+         ({FORWARDING_PERSIST})"
     )
 }
 
@@ -5435,10 +5449,7 @@ mod status_tests {
         out.push(network_section(&network, OWN, false));
         let text = out.render(false);
         assert_eq!(out.worst(), Health::Degraded, "{text}");
-        assert!(
-            text.contains("sudo sysctl -w net.ipv4.ip_forward=1"),
-            "{text}"
-        );
+        assert!(text.contains(FORWARDING_COMMAND), "{text}");
         assert!(text.contains("ok (nat + forward)"), "{text}");
 
         network.exit.forwarding = Some(true);

docs/exit-node.md

@@ -59,6 +59,49 @@ be an exit node in one network and not in another.
   specific IPv6 routes keep working, and so does the agent's own IPv6. A host
   with no IPv6 has nothing to block and is left alone.
 
+## macOS
+
+macOS has no `iptables` and no per-user policy routing, so the same behaviour
+is built from `pf` and `route`. The agent runs as root there.
+
+- **Offering:** a `pf` anchor `com.apple/tsunagi-exit-<interface>` (the stock
+  `/etc/pf.conf` already evaluates `com.apple/*`) holds
+  `nat on <egress> from <range> to any -> (<egress>)` and a `pass in` rule.
+  `pf` is enabled if it was off, and that reference is released when the rules
+  go. Forwarding is only read: `sudo sysctl -w net.inet.ip.forwarding=1`, which
+  macOS forgets at reboot. The egress is the default route's interface **at the
+  time the rules are applied**; after it changes (Wi-Fi to Ethernet) switch the
+  offer off and on.
+- **Using:** `0.0.0.0/1` and `128.0.0.0/1` routes through the overlay interface,
+  and a `pf` `route-to` rule that sends the agent's own traffic (matched by its
+  user id, root) back out the physical interface, with translation to that
+  interface's address. Anything else you run as root also bypasses the tunnel.
+  IPv6 is blocked with rejecting `::/1` and `8000::/1` routes.
+- Not verified on a live host when this was written: run `tsng exit-node` and
+  check `sudo pfctl -a com.apple/tsunagi-exit-utunN -s rules -s nat`.
+
+## Windows
+
+- **Offering:** each offered range gets a WinNAT object
+  `tsunagi-exit:<interface>:<range>` (`New-NetNat`), removed with the offer, the
+  network and the agent. Forwarding is read for the overlay interface and never
+  changed; Windows forwards per interface, so enable it on the ones involved
+  (`Get-NetIPInterface | Set-NetIPInterface -Forwarding Enabled`, administrator
+  PowerShell). Some Windows versions allow one NAT per host; a second range then
+  reports its own failure.
+- **Using:** Windows has no per-user routing, so the agent's own traffic is kept
+  off the tunnel *by destination*. `0.0.0.0/1` and `128.0.0.0/1` go through the
+  overlay interface; a `/32` route with metric 7373 (the marker for "ours")
+  through the physical default gateway is added for every address the agent
+  talks to directly: verified direct paths of its connections, plus the relays
+  it uses, resolved when the rules are applied. The set follows path changes. A
+  path that appears after the last apply goes through the tunnel until the next
+  one. IPv6 is blackholed with `::/1` and `8000::/1` on the overlay interface,
+  so applications fall back to IPv4 after a delay rather than at once.
+  Everything is in the active store: a reboot clears it.
+- Not verified on a live host when this was written. To inspect:
+  `Get-NetRoute -PolicyStore ActiveStore | ? { $_.RouteMetric -eq 7373 -or $_.InterfaceAlias -like 'tsun*' }`.
+
 ## What it does not do
 
 - **The agent must run as a system service** (its own user). The uid exemption