Withhold persisted podcast cards until the first share sync
CI / check (pull_request) Successful in 3m37s

After a restart, PodcastDht::maintenance republished every locally active
card as soon as its first bootstrap found peers, before the client could
reconcile with sync_shares, so a share revoked on another trusted device
while this one was offline was re-sent from the stale cache. Own cards are
now neither republished nor served to FindValue until the first
sync_shares of the process (or an explicit publish of that podcast);
tombstones are still sent. The first sync_shares re-sends unchanged cards.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
sanapachandClaude Opus 5.5 committed 2026-11-20 01:03:47 +02:00
1 parent 912527a671
commit 9c82f6d916
6 files changed
+215 -13

No files matched your search

+4 -1
View File
@@ -241,7 +241,10 @@ capability `podcast_dht`), built like the similarity-routing overlay. A
device publishes a card only for podcasts whose merged registers satisfy
`effective_share` (SQLite clients read the merged registers from
`PodcastStore`); `sync_shares` reconciles publications declaratively and
nothing is sent for a podcast the device never published. A card is the
nothing is sent for a podcast the device never published. Own cards persisted
by an earlier run are neither republished nor served until the first
`sync_shares` of the process, so a stale cache cannot revive a share revoked
elsewhere while the device was offline. A card is the
public projection of a podcast (feed URL and channel metadata, no personal
state), signed by the device's iroh identity over a domain-separated
canonical encoding. Only the origin URL the id hashes from is bound to the
+5
View File
@@ -23,6 +23,11 @@
- `sync_shares` is fail-safe for revocation (per-podcast `failed`) and holds
back cards with privacy warnings until confirmed (`held_back`,
`PodcastShareState::privacy_confirmed`).
- Own cards persisted by an earlier run are neither republished (by
`maintenance` or `republish`) nor served to lookups until the first
`sync_shares` of the process, so a share revoked on another trusted device
while this one was offline is not re-sent on startup. That first call also
re-sends unchanged cards; tombstones are always re-sent.
- `music-dht 0.6.0`: capability manifests can advertise `podcast_dht = 1`
(`capabilities::PODCAST_DHT_ID`, `CapabilityManifest::with_podcast_dht`,
`CapabilityManifest::protocol_version`,
+18 -2
View File
@@ -747,8 +747,22 @@ Fields are in postcard order. `PodcastDhtKey` is 32 raw bytes.
confirmation. See [SQLite reference store](#sqlite-reference-store).
* `PodcastDht::unpublish(podcast_id)` writes the tombstone. Nothing is ever
sent for a podcast this device never published: no tombstone, no id.
* `PodcastDht::republish` (run by `maintenance`) re-signs active cards and
re-sends unexpired tombstones every 10 minutes.
* `PodcastDht::republish` (run by `maintenance` every 10 minutes and after
a bootstrap that found new peers) re-signs active cards and re-sends
unexpired tombstones.
* Own cards persisted by an earlier run are not sent again until the client
calls `sync_shares` in this process: before that, `republish` sends only
tombstones and cards published with `publish` in this process, and
`FindValue` answers leave those cards out (they stay in the cache). A share
revoked on another trusted device while this one was offline is therefore
never re-sent on startup from the stale cache. Clients call `sync_shares`
on start, with the registers they hold, and again after device sync
applies newer operations; the first call also re-sends the unchanged cards
as they are (`PodcastDhtSyncStats::unchanged`, counted in `push`).
`sync_shares` itself still trusts the registers it is given: an unchanged
card is only re-signed when it used half its lifetime, so clients that may
hold stale registers on start lose nothing by reconciling again after
device sync.
* The privacy heuristics (`feed_url_privacy_warning`) are advisory: query
parameter names containing words such as `token`, `key`, `auth`, `secret`,
`sig`/`signature`, `uid`, `user`, `pass`, `session`, `access`, `private`;
@@ -823,6 +837,8 @@ is not needed afterwards, and the episode catalog comes from RSS.
* Only podcasts with `effective_share` are published; never-shared podcasts
produce no network traffic at all.
* After a restart nothing is re-sent from the persisted cards until the
client reconciled its registers with `sync_shares` (see Publication).
* Cards carry the public projection only (see Card); tombstones carry key
hashes only.
* Records are signed by the publisher device and verified on every receive
+6 -4
View File
@@ -264,10 +264,12 @@ explicitly shared. Wiring, as for `SimilarityDht`:
simply does not add it);
3. pass every peer manifest received to `note_peer_capabilities`, so peers
without the overlay are not dialed;
4. call `sync_shares` with the merged subscription and share registers after
applying sync operations: it publishes cards only where `effective_share`
holds, tombstones every other own publication, and reports cards it could
not build (`failed`) or held back for privacy (`held_back`);
4. call `sync_shares` with the merged subscription and share registers on
start and after applying sync operations (until the first call, own cards
persisted by an earlier run are neither republished nor served): it
publishes cards only where `effective_share` holds, tombstones every other
own publication, and reports cards it could not build (`failed`) or held
back for privacy (`held_back`);
5. ask the user before publishing when `PodcastCardSpec::privacy_warnings`
(or `feed_url_privacy_warning`) reports a URL that looks private, then
call `publish` or set `PodcastShareState::privacy_confirmed`.
+77 -6
View File
@@ -22,6 +22,11 @@
//! * Every record is verified (bounds, network, key placement, lifetime and
//! signature) when it is received and again whenever it is read from the
//! cache.
//! * Own cards persisted by an earlier run are not sent again, neither
//! republished nor served to lookups, until the client reconciles with
//! [`PodcastDht::sync_shares`] in this process: a share revoked on another
//! trusted device while this one was offline is never re-sent from the
//! stale cache.
//!
//! Applications own the serving and maintenance tasks, exactly as for
//! [`crate::similarity_dht::SimilarityDht`]: declare [`PODCAST_DHT_ALPN`] with
@@ -46,6 +51,7 @@ mod tests;
use std::collections::{BTreeMap, HashMap, HashSet};
use std::path::Path;
use std::str::FromStr;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, Mutex, MutexGuard, PoisonError};
use std::time::Duration;
@@ -264,7 +270,10 @@ pub struct PodcastDhtSyncStats {
/// Cards published with a new revision, or refreshed because they used
/// half their lifetime.
pub published: usize,
/// Shared podcasts whose current card was already published.
/// Shared podcasts whose current card was already published. The first
/// call in a process re-sends these cards as they are (counted in
/// [`Self::push`]), since [`PodcastDht::republish`] withholds them until
/// then.
pub unchanged: usize,
/// Podcasts unpublished with a tombstone.
pub unpublished: usize,
@@ -364,6 +373,13 @@ pub struct PodcastDht {
failed: Mutex<HashMap<EndpointId, u64>>,
failures: Mutex<HashMap<EndpointId, u32>>,
publish_gate: Semaphore,
/// Set by the first [`PodcastDht::sync_shares`] of this process. Until
/// then own cards from the persisted cache are neither republished nor
/// served: the registers they were derived from may be stale.
synced: AtomicBool,
/// Podcasts published with [`PodcastDht::publish`] in this process; their
/// cards are sent even before the first `sync_shares`.
published_here: Mutex<HashSet<PodcastId>>,
serve_limit: Arc<Semaphore>,
inflight: Arc<Mutex<HashMap<EndpointId, usize>>>,
}
@@ -412,6 +428,8 @@ impl PodcastDht {
failed: Mutex::new(HashMap::new()),
failures: Mutex::new(HashMap::new()),
publish_gate: Semaphore::new(1),
synced: AtomicBool::new(false),
published_here: Mutex::new(HashSet::new()),
serve_limit: Arc::new(Semaphore::new(MAX_CONCURRENT_PODCAST_DHT_REQUESTS)),
inflight: Arc::new(Mutex::new(HashMap::new())),
}))
@@ -449,7 +467,9 @@ impl PodcastDht {
/// Periodically discovers overlay peers, republishes own cards and
/// tombstones every [`PODCAST_REPUBLISH_INTERVAL`] and removes expired
/// replicas. The first bootstrap runs immediately.
/// replicas. The first bootstrap runs immediately. Own cards are only
/// republished once the client called [`Self::sync_shares`] in this
/// process (see [`Self::republish`]).
pub async fn maintenance(self: Arc<Self>) {
let mut bootstrap = tokio::time::interval(PODCAST_BOOTSTRAP_INTERVAL);
let mut republish = tokio::time::interval(PODCAST_REPUBLISH_INTERVAL);
@@ -621,7 +641,9 @@ impl PodcastDht {
let card = card.bounded();
card.validate()?;
let _permit = self.publish_permit().await?;
let podcast_id = card.podcast_id.clone();
let record = self.sign_card_locked(card, true).await?;
lock(&self.published_here).insert(podcast_id);
self.push(record.into_iter().collect()).await
}
@@ -647,6 +669,11 @@ impl PodcastDht {
/// including podcasts absent from `states`, is unpublished. The call is
/// idempotent: repeating it publishes nothing new.
///
/// The first successful call in a process arms [`Self::republish`] and
/// serving of own cards: clients call it on start, after loading their
/// merged registers, and again whenever they change. That call also
/// re-sends the unchanged cards to the overlay as they are.
///
/// Each podcast is handled on its own, so revocation is fail-safe: a
/// shared podcast whose card cannot be built is reported in
/// [`PodcastDhtSyncStats::failed`] and unpublished, and never prevents
@@ -657,6 +684,7 @@ impl PodcastDht {
/// publishes the same feed URLs.
pub async fn sync_shares(&self, states: Vec<PodcastShareState>) -> Result<PodcastDhtSyncStats> {
let _permit = self.publish_permit().await?;
let first_sync = !self.synced.load(Ordering::SeqCst);
let local = self.database.local_publications().await?;
let active: HashMap<&PodcastId, &PodcastCardSpec> = local
.iter()
@@ -727,33 +755,61 @@ impl PodcastDht {
records.push(record);
stats.published += 1;
}
Ok(None) => stats.unchanged += 1,
Ok(None) => {
stats.unchanged += 1;
// Withheld by `republish` until now: send it as it is.
if first_sync
&& let Some(publication) = local
.iter()
.find(|publication| publication.podcast_id == podcast_id)
{
records.push(publication.record.clone());
}
}
Err(error) => stats.failed.push(PodcastSyncFailure {
podcast_id,
error: error.to_string(),
}),
}
}
// Own publications now reflect the client's registers.
self.synced.store(true, Ordering::SeqCst);
stats.push = self.push(records).await?;
Ok(stats)
}
/// Re-signs every published card with a fresh lifetime (same revision)
/// and re-sends every unexpired tombstone. Called by
/// [`Self::maintenance`] every [`PODCAST_REPUBLISH_INTERVAL`].
/// [`Self::maintenance`] every [`PODCAST_REPUBLISH_INTERVAL`] and after a
/// bootstrap that found new peers.
///
/// Until [`Self::sync_shares`] has run in this process, only tombstones
/// and cards published with [`Self::publish`] in this process are sent:
/// the persisted cards were derived from registers that may have been
/// revoked on another trusted device meanwhile.
pub async fn republish(&self) -> Result<PodcastDhtPublishStats> {
let _permit = self.publish_permit().await?;
let now = now_ms();
let mut records = Vec::new();
let mut withheld = 0usize;
for publication in self.database.local_publications().await? {
match publication.spec {
Some(spec) => records.extend(self.sign_card_locked(spec, true).await?),
Some(spec) if self.may_send_card(&publication.podcast_id) => {
records.extend(self.sign_card_locked(spec, true).await?);
}
Some(_) => withheld += 1,
None if publication.record.expires_at_ms() > now => {
records.push(publication.record);
}
None => {}
}
}
if withheld > 0 {
debug!(
withheld,
"own podcast cards wait for the first share reconciliation"
);
}
if records.is_empty() {
return Ok(PodcastDhtPublishStats::default());
}
@@ -837,6 +893,13 @@ impl PodcastDht {
.find(|hit| hit.podcast_id == *podcast_id))
}
/// Whether an own card of `podcast_id` may leave this device: after the
/// first [`Self::sync_shares`] of this process, or when it was published
/// with [`Self::publish`] in this process.
fn may_send_card(&self, podcast_id: &PodcastId) -> bool {
self.synced.load(Ordering::SeqCst) || lock(&self.published_here).contains(podcast_id)
}
async fn publish_permit(&self) -> Result<tokio::sync::SemaphorePermit<'_>> {
self.publish_gate
.acquire()
@@ -1174,10 +1237,18 @@ impl PodcastDht {
)),
PodcastDhtOperation::FindValue { key, filter_tokens } => {
validate_filter_tokens(&filter_tokens)?;
let records = self
let mut records = self
.database
.records_by_key(key, filter_tokens, now_ms(), MAX_PODCAST_DHT_RECORDS)
.await?;
// Own cards persisted by an earlier run wait for the first
// share reconciliation, like republishing.
let own = self.own_contact.peer_id;
records.retain(|record| {
record.publisher() != own
|| record.is_tombstone()
|| self.may_send_card(record.podcast_id())
});
Ok(PodcastDhtResponse::success(
self.closest_contacts(key.as_bytes(), K),
records,
+105
View File
@@ -20,6 +20,9 @@
//! look private (single node).
//! * Contacts a peer returns are never trusted: fabricated, unreachable
//! contacts do not enter the routing table (two nodes).
//! * After a restart, own cards persisted by the previous run are neither
//! republished nor served until the first `sync_shares`, so a share
//! revoked on another device meanwhile is never re-sent (two nodes).
use std::path::Path;
use std::sync::Arc;
@@ -656,3 +659,105 @@ async fn fabricated_contacts_scenario() {
m_events.abort();
m.shutdown().await.unwrap();
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn restarted_device_waits_for_sync_shares_before_resending_cards() {
tokio::time::timeout(TEST_TIMEOUT, restart_scenario())
.await
.unwrap();
}
async fn restart_scenario() {
let temp = tempfile::tempdir().unwrap();
let network = NetworkId::from_name("podcast-dht-restart");
let state =
|subscription: &SubscriptionRecord, shared: bool, generation: u64| PodcastShareState {
subscription: subscription.clone(),
share: Some(share(subscription, shared, generation)),
card: None,
privacy_confirmed: false,
};
let kept = subscription("kept.xml", "Kept Show");
let revoked = subscription("revoked.xml", "Revoked Show");
// A publishes both podcasts while alone, then goes offline.
let a = Node::start(&temp.path().join("a"), network).await;
let stats = a
.dht
.sync_shares(vec![state(&kept, true, 2), state(&revoked, true, 2)])
.await
.unwrap();
assert_eq!((stats.published, stats.push.remote_nodes), (2, 0));
a.stop().await;
// Meanwhile another trusted device revokes "Revoked Show". A comes back
// with its persisted cards and runs maintenance before its client
// reconciled: the first bootstrap finds B.
let b = Node::start(&temp.path().join("b"), network).await;
let mut a = Node::start(&temp.path().join("a"), network).await;
b.service
.connect(a.service.ticket().await.unwrap())
.await
.unwrap();
wait_for("music DHT contacts", async || {
(!a.service.known_peers().is_empty()).then_some(())
})
.await;
a.tasks.push(tokio::spawn(Arc::clone(&a.dht).maintenance()));
wait_for("overlay contacts", async || {
(a.dht.known_peers() >= 1 && b.dht.known_peers() >= 1).then_some(())
})
.await;
// Neither maintenance nor an explicit republish sends a stale card, and
// A does not serve them to lookups either.
let stats = a.dht.republish().await.unwrap();
assert_eq!(stats.records, 0, "stale cards are withheld");
for podcast in [&kept, &revoked] {
assert!(
b.dht
.lookup_podcast(&podcast.podcast_id)
.await
.unwrap()
.is_none()
);
}
assert!(search(&b, "show").await.is_empty());
// The client's first reconciliation revokes the stale share and
// re-sends the unchanged card.
let stats = a
.dht
.sync_shares(vec![state(&kept, true, 2), state(&revoked, false, 3)])
.await
.unwrap();
assert_eq!(
(stats.published, stats.unchanged, stats.unpublished),
(0, 1, 1)
);
assert_eq!(stats.push.records, 2, "the kept card and the tombstone");
assert!(stats.push.remote_nodes >= 1);
let a_only = vec![a.service.endpoint_id()];
let hit = b
.dht
.lookup_podcast(&kept.podcast_id)
.await
.unwrap()
.expect("the kept card is found");
assert_eq!(hit.publishers, a_only);
assert!(
b.dht
.lookup_podcast(&revoked.podcast_id)
.await
.unwrap()
.is_none()
);
// From now on republishing covers the kept card and the tombstone.
let stats = a.dht.republish().await.unwrap();
assert_eq!(stats.records, 2);
let hits = search(&b, "show").await;
assert_eq!(hits.len(), 1);
assert_eq!(hits[0].podcast_id, kept.podcast_id);
a.stop().await;
b.stop().await;
}