Document podcast discovery wire protocol and replica limits

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
sanapachandClaude Opus 5.5 committed 2026-11-19 18:32:40 +02:00
1 parent 81ccd924d0
commit 5a4d418d2b
4 files changed
+215 -74

No files matched your search

+19 -10
View File
@@ -243,16 +243,25 @@ device publishes a card only for podcasts whose merged registers satisfy
nothing is sent for a podcast the device never published. A card is the
public projection of a podcast (feed URL and channel metadata, no personal
state), signed by the device's iroh identity over a domain-separated
canonical encoding and bound to its id by the URL the id hashes from. Cards
live under the podcast-id key and one key per normalized title/author token;
unpublishing writes a signed tombstone of the next revision that supersedes
lower and equal revisions under every key and outlives every older card.
Records use the music DHT's active/tombstone TTLs and republish cadence, are
verified on every receive and cache read, and are kept in a separate,
replaceable SQLite cache. Search filters on every replica by all query tokens,
merges cards of one podcast across publisher devices, and returns typed
`podcast` hits that clients merge with music results. Capability data only
helps skip peers; it never authorizes. See the "Public discovery" section of
canonical encoding. Only the origin URL the id hashes from is bound to the
podcast; a moved `feed_url` is an unverified publisher claim that search never
offers as the podcast's feed. Cards live under the podcast-id key and one key
per normalized title/author token; unpublishing writes a signed tombstone of
the next revision that supersedes lower and equal revisions on replicas that
hold it, for the keys it lists, and outlives every older card. Only a record's
publisher may store it; replicas keep one payload per publisher and podcast,
admit placements only for keys they are close to, within per-key,
per-publisher and total budgets, and never evict existing records for
newcomers (tombstones cannot displace cards). Records use the music DHT's
active/tombstone TTLs and republish cadence, are verified on every receive and
cache read, and are kept in a separate, replaceable SQLite cache. Search
filters on every replica by all query tokens, merges cards of one podcast
across publisher devices, and returns typed `podcast` hits that clients merge
with music results; publisher counts are not a trust signal. The publisher key
is the device's transport identity, so a card is linkable to the device's
network address and music-DHT records. Applications advertise the
`podcast_dht` capability only when they run the overlay; capability data only
helps skip peers, it never authorizes. See the "Public discovery" section of
[PODCASTS.md](PODCASTS.md).
Music discovery identifies an owner and content; it does not force all domain
+17 -7
View File
@@ -4,17 +4,27 @@
- `music-dht 0.6.0`: new `music_dht::podcast_dht` overlay for public podcast
discovery on its own ALPN `furumi-fd/podcast-dht/1` (version 1) and its own
SQLite cache: signed `PodcastCard`s and `PodcastTombstone`s
SQLite cache (schema 2): signed `PodcastCard`s and `PodcastTombstone`s
(`furumi-podcast-card:v1` / `furumi-podcast-card-tombstone:v1` canonical
signing input), monotonic revisions with a total tie-break, text and id
keys, `PodcastDht::{publish, unpublish, sync_shares, republish, search,
lookup_podcast}` gated by `effective_share`, typed search hits
(`PodcastSearchHit`, `FederatedSearchHit`, `SearchHitKind`,
`merge_search_results`) and the advisory `feed_url_privacy_warning`.
- `music-dht 0.6.0`: capability manifests advertise `podcast_dht = 1`
(`capabilities::PODCAST_DHT_ID`, `CapabilityManifest::protocol_version`,
`CapabilityManifest::supports_podcast_dht`); applications that do not run
the overlay must remove it with `without_protocol`.
(`PodcastSearchHit`, `ClaimedFeedUrl`, `FederatedSearchHit`,
`SearchHitKind`, `merge_search_results`) and the advisory
`feed_url_privacy_warning`.
- Replicas accept records only from their publisher, store one payload per
`(podcast_id, publisher)`, admit placements only for keys they are close to
and within per-key, per-publisher and total budgets, and never evict
existing records for newcomers. Search offers only the URL the podcast id
hashes from; moved-feed URLs are reported as unverified claims.
- `sync_shares` is fail-safe for revocation (per-podcast `failed`) and holds
back cards with privacy warnings until confirmed (`held_back`,
`PodcastShareState::privacy_confirmed`).
- `music-dht 0.6.0`: capability manifests can advertise `podcast_dht = 1`
(`capabilities::PODCAST_DHT_ID`, `CapabilityManifest::with_podcast_dht`,
`CapabilityManifest::protocol_version`,
`CapabilityManifest::supports_podcast_dht`). It is opt-in:
`CapabilityManifest::frid()` does not include it.
- `music-dht 0.6.0`: new `MusicDhtError::PodcastPublication` variant (the enum
is `#[non_exhaustive]`).
- The music DHT (`music-dht-v5`), `federation-net` and device-sync protocol v4
+158 -48
View File
@@ -514,8 +514,8 @@ of one podcast by `podcast_id`.
| --- | --- |
| ALPN | `furumi-fd/podcast-dht/1` (schema-independent stream protocol) |
| Wire and record version | `PODCAST_DHT_PROTOCOL_VERSION = 1` |
| Capability manifest | `podcast_dht = 1` (`CapabilityManifest::supports_podcast_dht`) |
| Cache | separate SQLite file, `PRAGMA user_version = 1`; a newer schema is refused |
| Capability manifest | `podcast_dht = 1`, opt-in (`CapabilityManifest::with_podcast_dht`, `supports_podcast_dht`) |
| Cache | separate SQLite file, `PRAGMA user_version = 2`; a newer schema is refused, the pre-release version 1 replica table is dropped |
| Card TTL | `ACTIVE_RECORD_TTL` of the music DHT (12 h) |
| Tombstone TTL | `TOMBSTONE_TTL` of the music DHT (24 h) |
| Republish | `DEFAULT_REPUBLISH_INTERVAL` (10 min), expiry sweep `DEFAULT_EXPIRE_INTERVAL` (1 min) |
@@ -534,7 +534,7 @@ HTTP validators or episodes.
| --- | --- |
| `version`, `network_id`, `publisher` | 1, the federation network, the device's iroh endpoint key |
| `podcast_id` | must equal `PodcastId::from_feed_url(id_feed_url or feed_url)` |
| `feed_url` | canonical `FeedUrl` the feed is fetched from |
| `feed_url` | canonical `FeedUrl` the publisher fetches the feed from |
| `id_feed_url` | the URL the id was derived from, present only when it differs from `feed_url` (feed moved) |
| `title` | 1..300 characters, no control characters |
| `author` | optional, ≤ 300 characters, no control characters |
@@ -550,10 +550,14 @@ HTTP validators or episodes.
whole signed record is at most 24 KiB postcard-encoded; the field bounds keep
every valid card below it.
Binding the id to a URL in the card prevents claiming an existing podcast id
for an arbitrary feed. When a feed moved, `feed_url` is the publisher's
claim; clients should prefer their own subscription's URLs for a podcast
they already know.
Only the **origin URL** (`PodcastCard::origin_feed_url`: `id_feed_url`, or
`feed_url` when the feed has not moved) is bound to the podcast, because the
id is its hash. When `id_feed_url` is present, `feed_url` is an unverified
claim of the publisher that the feed moved: anyone can sign a card that
carries an existing podcast's origin URL and an arbitrary `feed_url`. Search
therefore never offers a claimed URL as the podcast's feed (see Search), and
clients must not fetch it as that podcast or record it as an alias unless the
feed at the origin URL itself redirects there.
### Tombstone
@@ -566,6 +570,10 @@ lower or equal revision and is itself retained and served until it expires.
Because the tombstone TTL exceeds the card TTL, every card issued before a
tombstone expires before it.
`index_keys` is self-asserted, so it never creates a placement by itself: a
replica indexes a tombstone under a text key only in place of a record of the
same `(podcast_id, publisher)` it already holds there (see Replica storage).
### Signing input
Both records are signed with the publisher's Ed25519 iroh identity over a
@@ -594,9 +602,10 @@ postcard `SignedPodcastRecord { body: Card | Tombstone, signature }`.
material)` with kind `0x01` and the podcast id string for the id key, or
kind `0x02` and a token for a text key. Tokens are the distinct
`normalize_name` tokens of the title, then of the author, skipping tokens
longer than 64 bytes, at most 32. A card is stored under the id key and its
longer than 64 bytes, at most 32. A card belongs under the id key and its
text keys (at most 33); a tombstone under the id key and its `index_keys`.
Replicas accept a record only under a key derived from it.
The publisher sends each record to the `K` (8) overlay nodes closest to each
of its keys, one `StoreBatch` per node listing the keys that node should hold.
### Ordering
@@ -604,10 +613,10 @@ For one `(podcast_id, publisher)` the greater record wins: higher
`revision`; at an equal revision a tombstone beats a card; then later
`issued_at_ms` (a republish); then later `expires_at_ms`; then the
lexicographically greater signing input. This is a total order, so every
replica converges whatever the delivery order. A replica that holds a
tombstone, or a card of a higher revision, under any key drops older cards of
that publisher under every key and refuses them later; a replayed old
revision is therefore ignored.
replica and every lookup converge whatever the delivery order. A replica
keeps only the winning record of each `(podcast_id, publisher)`; a replayed
old revision is ignored. These fields only order one publisher's own records:
they never rank or select records of different publishers.
Revisions: a changed card gets `max(previous + 1, now_ms)` (time-seeded, so
revisions stay monotonic even if the local cache is lost); an unchanged card
@@ -615,12 +624,80 @@ is re-signed with the same revision and a fresh lifetime on every republish;
unpublishing writes a tombstone with `previous + 1`; publishing again later
gets `max(tombstone + 1, now_ms)`.
### Wire protocol
One request per bidirectional stream on `furumi-fd/podcast-dht/1`: the
requester writes one postcard `PodcastDhtRequest` and finishes its send side;
the responder writes one postcard `PodcastDhtResponse` and finishes. Messages
are not length-prefixed; the stream end delimits them. Requests above
512 KiB and responses above 4 MiB are refused.
```text
PodcastDhtRequest { version: u16, requester: NodeContact, operation: PodcastDhtOperation }
PodcastDhtOperation = 0: FindNode { target: [u8; 32] }
| 1: FindValue { key: PodcastDhtKey, filter_tokens: Vec<String> }
| 2: StoreBatch { records: Vec<PodcastStoreEntry> }
PodcastStoreEntry { keys: Vec<PodcastDhtKey>, record: SignedPodcastRecord }
PodcastDhtResponse { version: u16, ok: bool, error: Option<String>,
nodes: Vec<NodeContact>, records: Vec<SignedPodcastRecord>, stored: u32 }
NodeContact { node_id: [u8; 32], peer_id: EndpointId, ticket: String, last_seen_ms: u64 }
PodcastRecordBody = 0: Card(PodcastCard) | 1: Tombstone(PodcastTombstone)
```
Fields are in postcard order. `PodcastDhtKey` is 32 raw bytes.
* `version` must be 1 in both directions. `requester.peer_id` must equal the
authenticated transport peer, `node_id` must be `NodeId::from_endpoint`
of it and its ticket must name the same endpoint and network; otherwise the
request is refused. The responder learns the requester as an overlay
contact.
* `FindNode` returns the `K` closest overlay contacts (the responder
included).
* `FindValue` returns the records under `key` (see Replica storage) and the
`K` closest contacts. `filter_tokens` (≤ 8, each 1..64 bytes) are
normalized tokens every returned card must be indexed under; tombstones are
returned regardless. Requesters verify every record again and drop records
that do not belong under `key` or miss a filter token.
* `StoreBatch` carries 1..16 entries, each with 1..33 distinct keys. Every
record must be published by the requester itself (replicas never forward
records), pass `verify_at` (bounds, network, lifetime, clock skew,
signature, size) and belong under each listed key; otherwise the whole
batch is refused. Keys the responder is not responsible for are skipped.
`stored` counts the `(key, record)` placements written.
* A refused request gets `ok = false` with an `error` of at most 1 KiB and
empty `nodes`, `records` and `stored = 0`; an accepted one has `ok = true`
and no `error`. Requesters reject responses that violate this or exceed
`K` nodes or 128 records.
### Replica storage
* A replica stores one payload per `(podcast_id, publisher)` (the winning
record) and indexes it under the keys it holds. A newer version replaces
it in place; placements the new version no longer belongs under (an old
title's tokens, keys a tombstone does not list) are dropped, so on one
replica a tombstone supersedes the card under every key it held.
* A node accepts a placement only when fewer than `2 * K` of its known
overlay contacts are closer to the key than itself.
* Admission, never eviction: a key holds at most 256 records. A full key
admits no new `(podcast_id, publisher)` until one expires; existing records
keep their place and their publishers refresh them in place. A tombstone
is indexed under a text key only in place of its owner's record there; new
tombstones under a key (in practice the id key, for cards this replica
never saw) are limited to 32.
* A replica holds at most 256 records per publisher, 4 096 records and
32 MiB of payload; records beyond that are refused, not evicted. At most
1 024 overlay contacts are persisted.
* `FindValue` answers with at most 128 records: at most 32 tombstones and
the cards indexed longest first. Every record is verified again on every
read.
### Publication
* `PodcastDht::publish(subscription, share, card)` refuses
(`MusicDhtError::PodcastPublication`) unless `effective_share` holds, so a
new or resumed subscription is never published without a fresh share
decision.
decision. The UI calls it only after asking the user about any privacy
warning, so the call is the user's consent.
* `PodcastDht::sync_shares(states)` is declarative and idempotent: podcasts
for which `effective_share` holds are published (when changed, new or past
half their lifetime), every other podcast this device published is
@@ -628,17 +705,27 @@ gets `max(tombstone + 1, now_ms)`.
operations, so unsharing or unsubscribing on any trusted device revokes
every device's own card once the registers converge, and a stale device
cannot republish (its registers lose to the newer share decision).
* Revocation is fail-safe: each podcast is handled on its own. A shared
podcast whose card cannot be built or signed is reported in
`PodcastDhtSyncStats::failed` and treated as not shared (an earlier
publication is tombstoned); it never prevents other podcasts from being
unpublished.
* A card whose `PodcastCardSpec::privacy_warnings` is not empty is held back
(`PodcastDhtSyncStats::held_back`, with the warnings) unless
`PodcastShareState::privacy_confirmed` is set or this device already
publishes the same `feed_url` and `id_feed_url`. A feed move to a URL that
looks private therefore asks again; an earlier publication of a held-back
podcast stays as it was.
* `PodcastDht::unpublish(podcast_id)` writes the tombstone. Nothing is ever
sent for a podcast this device never published: no tombstone, no id.
* `PodcastDht::republish` (run by `maintenance`) re-signs active cards and
re-sends unexpired tombstones every 10 minutes.
* Before publishing, the UI must call `feed_url_privacy_warning` (or
`PodcastCardSpec::privacy_warnings`, which also checks `id_feed_url`) and
ask the user when it returns a warning. The heuristics are advisory: query
* The privacy heuristics (`feed_url_privacy_warning`) are advisory: query
parameter names containing words such as `token`, `key`, `auth`, `secret`,
`sig`/`signature`, `uid`, `user`, `pass`, `session`, `access`, `private`;
known member-feed hosts; and long high-entropy query values (≥ 16
characters) or path segments (≥ 32 characters). False positives and misses
characters) or path segments (≥ 32 characters). They check `feed_url` and
`id_feed_url`, not artwork or web-page links. False positives and misses
are possible.
### Search
@@ -649,39 +736,54 @@ gets `max(tombstone + 1, now_ms)`.
cards indexed under **all** query tokens (tombstones are always returned).
Results are verified again, the winning record per publisher is kept,
tombstoned publishers are dropped and cards are merged by `podcast_id`
into `PodcastSearchHit { podcast_id, card, publishers }`. Ranking: exact
normalized title first, then more publishers.
into `PodcastSearchHit { podcast_id, feed_url, card, publishers,
claimed_feed_urls }`.
* `PodcastDht::lookup_podcast(podcast_id)` reads the id key.
* `publishers` lists distinct publisher **devices**; it is not a subscriber
* `feed_url` is the origin URL; it is the only URL a client subscribes with.
`claimed_feed_urls` lists other `feed_url`s that publishers claim (at most
8, with their publishers), for display only.
* The representative `card` is a card whose `feed_url` is the origin URL
when one exists, then the title most publishers agree on, then the
smallest publisher key. Publisher-controlled times and revisions never
break ties. Its metadata remains a publisher's claim.
* `publishers` lists distinct publisher **devices**. It is not a subscriber
count (one user may publish from several devices, and subscribers who did
not share are invisible), so no subscriber count is offered.
* The representative card is the one whose `feed_url` most publishers agree
on, then the most recently issued, then the greatest publisher key.
not share are invisible) and not a trust signal: device identities are
free. No subscriber count is offered and ranking ignores it.
* Ranking: exact normalized title first, then hits that have a card under
the origin URL, then title and id.
* `FederatedSearchHit` (`{"type": "music" | "podcast", "hit": ...}`),
`SearchHitKind` (`artist`, `release`, `track`, `podcast`) and
`merge_search_results` merge podcast hits with `SearchOutcome` music
results; `SearchOutcome` itself is unchanged.
* Lookups query the 8 closest overlay nodes (α = 3, at most 32 requests,
15 s deadline) and return partial results when peers fail. A peer that
fails a request is skipped for 30 s.
15 s deadline), keep at most 512 `(podcast_id, publisher)` records each and
return partial results when peers fail. A peer that fails a request is
skipped for 30 s.
Subscribing from a hit creates the user's own private subscription: resolve
the id with `resolve_podcast_id` on the card's `id_feed_url` (or `feed_url`),
which yields the card's `podcast_id` unless the user already knows that URL,
fetch the feed from `feed_url` and record it as an alias when it differs. The
new subscription is private (see `effective_share`), the publisher's
availability is not needed afterwards, and the episode catalog comes from RSS.
Subscribing from a hit creates the user's own private subscription with the
hit's `feed_url` (the origin URL): resolve the id with `resolve_podcast_id`,
which yields the hit's `podcast_id` unless the user already knows that URL,
fetch the feed from that URL and follow its own HTTP redirects and
`itunes:new-feed-url`, recording those as aliases as usual. A URL from
`claimed_feed_urls` becomes an alias only when that fetch leads there. The new
subscription is private (see `effective_share`), the publisher's availability
is not needed afterwards, and the episode catalog comes from RSS.
### Limits
| Limit | Value |
| --- | --- |
| Request / response | 512 KiB / 4 MiB postcard |
| Store batch | 1..16 entries, each 1..33 distinct keys |
| Records per `FindValue` response | 128 |
| Store batch | 1..16 entries, each 1..33 distinct keys, records of the requester only |
| Records per `FindValue` response | 128, at most 32 of them tombstones |
| Filter tokens | ≤ 8, each 1..64 bytes |
| Records cached per key | 256; the record expiring first is evicted for a longer-lived one |
| Concurrent inbound requests | 64 (more are dropped); 10 s per request |
| Records per key | 256 (new tombstones 32); admission only, no eviction |
| Records per replica | 256 per publisher, 4 096 in all, 32 MiB of payload |
| Store responsibility | fewer than `2 * K` = 16 known contacts closer to the key |
| Records kept per lookup | 512 `(podcast_id, publisher)` |
| Persisted overlay contacts | 1 024 |
| Inbound requests | 64 concurrent, 8 per peer (more are dropped); 10 s each |
| Error message | ≤ 1 KiB |
| Contact ticket | ≤ 16 KiB |
@@ -692,20 +794,28 @@ availability is not needed afterwards, and the episode catalog comes from RSS.
* Cards carry the public projection only (see Card); tombstones carry key
hashes only.
* Records are signed by the publisher device and verified on every receive
and on every cache read; a forwarder cannot alter or impersonate them, and
records from another network are rejected.
* Cards do not carry a connection ticket: discovering a card does not reveal
how to reach the publisher, and subscribing never contacts it.
* The publisher's device key is public in the card. Devices of one user are
not linked by the overlay, but publishing the same podcasts from several
devices makes them correlatable.
and on every cache read; a forwarder cannot alter or impersonate them,
only the publisher can store them, and records from another network are
rejected.
* The card's `publisher` is the device's persistent iroh endpoint key, which
is also its overlay and music-DHT identity. It is **linkable**: anyone can
look the key up in the overlay (`FindNode` for
`NodeId::from_endpoint(publisher)`) and obtain the device's contact ticket
with its relay and direct IP addresses, because every overlay request
carries the requester's ticket. The same key owns the device's music-DHT
records, so a device's shared podcasts can be linked to its public music
library and its network address. Subscribing from a card never contacts
the publisher.
* Devices of one user are not linked by the overlay, but publishing the same
podcasts from several devices makes them correlatable.
* Unpublishing is eventually consistent: replicas that missed the tombstone
keep serving the old card until it expires (at most 12 h). A replica that
only held text keys of an older revision (after a title change) may serve
that revision until it expires.
keep serving the old card until it expires (at most 12 h), including
replicas that only held an older title's text keys.
* Capability manifests are informational. A peer without `podcast_dht` never
enters the overlay routing table (it cannot answer on the ALPN), and
`PodcastDht::note_peer_capabilities` evicts a peer whose manifest lacks it.
`PodcastDht::note_peer_capabilities` evicts a peer whose manifest lacks it
and stops dialing it. Applications advertise the overlay only when they
run it (`with_podcast_dht`).
## Feed parser
@@ -879,5 +989,5 @@ must keep the same invariants.
* Public podcast cards use the separate signed overlay `podcast_dht` with
its own ALPN (`furumi-fd/podcast-dht/1`), wire/record version 1, signing
domains `furumi-podcast-card:v1` / `furumi-podcast-card-tombstone:v1`, key
domain `furumi-podcast-dht:key:v1` and cache schema 1, all frozen; the music
domain `furumi-podcast-dht:key:v1`, all frozen, and cache schema 2; the music
DHT schema and `federation-net` do not change.
+21 -9
View File
@@ -254,15 +254,27 @@ reported itself (`profile_protocol_version`; relayed profiles go through
`music_dht::podcast_dht::PodcastDht` is a separate signed overlay (ALPN
`furumi-fd/podcast-dht/1`, capability `podcast_dht`) for podcasts the user
explicitly shared. Declare the ALPN with
`schema_independent_stream_protocol`, open the overlay with its own cache file,
and spawn `serve` and `maintenance`, as for `SimilarityDht`. Call
`sync_shares` with the merged subscription and share registers after applying
sync operations: it publishes cards only where `effective_share` holds and
tombstones every other own publication. Ask the user before publishing when
`PodcastCardSpec::privacy_warnings` (or `feed_url_privacy_warning`) reports a
URL that looks private. `search` and `lookup_podcast` return
`PodcastSearchHit`s; `merge_search_results` combines them with
explicitly shared. Wiring, as for `SimilarityDht`:
1. declare the ALPN with `schema_independent_stream_protocol(PODCAST_DHT_ALPN)`,
open the overlay with its own cache file, take
`stream_acceptor(PODCAST_DHT_ALPN)` and spawn `serve` and `maintenance`;
2. advertise it with `CapabilityManifest::with_podcast_dht()` (it is not in
`CapabilityManifest::frid()`; an application that does not run the overlay
simply does not add it);
3. pass every peer manifest received to `note_peer_capabilities`, so peers
without the overlay are not dialed;
4. call `sync_shares` with the merged subscription and share registers after
applying sync operations: it publishes cards only where `effective_share`
holds, tombstones every other own publication, and reports cards it could
not build (`failed`) or held back for privacy (`held_back`);
5. ask the user before publishing when `PodcastCardSpec::privacy_warnings`
(or `feed_url_privacy_warning`) reports a URL that looks private, then
call `publish` or set `PodcastShareState::privacy_confirmed`.
`search` and `lookup_podcast` return `PodcastSearchHit`s; subscribe with the
hit's `feed_url` (the URL the id hashes from), never with a
`claimed_feed_urls` entry. `merge_search_results` combines hits with
`SearchOutcome::network_results` as typed `FederatedSearchHit`s.
## Verification