From 5a4d418d2b3585a990c3174dd1c31607a38d38c1 Mon Sep 17 00:00:00 2001 From: Sanapach Date: Thu, 19 Nov 2026 17:11:50 +0200 Subject: [PATCH] Document podcast discovery wire protocol and replica limits Co-Authored-By: Claude Opus 5.5 --- ARCHITECTURE.md | 29 ++++-- CHANGELOG.md | 24 +++-- PODCASTS.md | 206 ++++++++++++++++++++++++++++--------- crates/music-dht/README.md | 30 ++++-- 4 files changed, 215 insertions(+), 74 deletions(-) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 23a53ef..d508928 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -243,16 +243,25 @@ device publishes a card only for podcasts whose merged registers satisfy nothing is sent for a podcast the device never published. A card is the public projection of a podcast (feed URL and channel metadata, no personal state), signed by the device's iroh identity over a domain-separated -canonical encoding and bound to its id by the URL the id hashes from. Cards -live under the podcast-id key and one key per normalized title/author token; -unpublishing writes a signed tombstone of the next revision that supersedes -lower and equal revisions under every key and outlives every older card. -Records use the music DHT's active/tombstone TTLs and republish cadence, are -verified on every receive and cache read, and are kept in a separate, -replaceable SQLite cache. Search filters on every replica by all query tokens, -merges cards of one podcast across publisher devices, and returns typed -`podcast` hits that clients merge with music results. Capability data only -helps skip peers; it never authorizes. See the "Public discovery" section of +canonical encoding. Only the origin URL the id hashes from is bound to the +podcast; a moved `feed_url` is an unverified publisher claim that search never +offers as the podcast's feed. Cards live under the podcast-id key and one key +per normalized title/author token; unpublishing writes a signed tombstone of +the next revision that supersedes lower and equal revisions on replicas that +hold it, for the keys it lists, and outlives every older card. Only a record's +publisher may store it; replicas keep one payload per publisher and podcast, +admit placements only for keys they are close to, within per-key, +per-publisher and total budgets, and never evict existing records for +newcomers (tombstones cannot displace cards). Records use the music DHT's +active/tombstone TTLs and republish cadence, are verified on every receive and +cache read, and are kept in a separate, replaceable SQLite cache. Search +filters on every replica by all query tokens, merges cards of one podcast +across publisher devices, and returns typed `podcast` hits that clients merge +with music results; publisher counts are not a trust signal. The publisher key +is the device's transport identity, so a card is linkable to the device's +network address and music-DHT records. Applications advertise the +`podcast_dht` capability only when they run the overlay; capability data only +helps skip peers, it never authorizes. See the "Public discovery" section of [PODCASTS.md](PODCASTS.md). Music discovery identifies an owner and content; it does not force all domain diff --git a/CHANGELOG.md b/CHANGELOG.md index cb64153..4b14174 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,17 +4,27 @@ - `music-dht 0.6.0`: new `music_dht::podcast_dht` overlay for public podcast discovery on its own ALPN `furumi-fd/podcast-dht/1` (version 1) and its own - SQLite cache: signed `PodcastCard`s and `PodcastTombstone`s + SQLite cache (schema 2): signed `PodcastCard`s and `PodcastTombstone`s (`furumi-podcast-card:v1` / `furumi-podcast-card-tombstone:v1` canonical signing input), monotonic revisions with a total tie-break, text and id keys, `PodcastDht::{publish, unpublish, sync_shares, republish, search, lookup_podcast}` gated by `effective_share`, typed search hits - (`PodcastSearchHit`, `FederatedSearchHit`, `SearchHitKind`, - `merge_search_results`) and the advisory `feed_url_privacy_warning`. -- `music-dht 0.6.0`: capability manifests advertise `podcast_dht = 1` - (`capabilities::PODCAST_DHT_ID`, `CapabilityManifest::protocol_version`, - `CapabilityManifest::supports_podcast_dht`); applications that do not run - the overlay must remove it with `without_protocol`. + (`PodcastSearchHit`, `ClaimedFeedUrl`, `FederatedSearchHit`, + `SearchHitKind`, `merge_search_results`) and the advisory + `feed_url_privacy_warning`. +- Replicas accept records only from their publisher, store one payload per + `(podcast_id, publisher)`, admit placements only for keys they are close to + and within per-key, per-publisher and total budgets, and never evict + existing records for newcomers. Search offers only the URL the podcast id + hashes from; moved-feed URLs are reported as unverified claims. +- `sync_shares` is fail-safe for revocation (per-podcast `failed`) and holds + back cards with privacy warnings until confirmed (`held_back`, + `PodcastShareState::privacy_confirmed`). +- `music-dht 0.6.0`: capability manifests can advertise `podcast_dht = 1` + (`capabilities::PODCAST_DHT_ID`, `CapabilityManifest::with_podcast_dht`, + `CapabilityManifest::protocol_version`, + `CapabilityManifest::supports_podcast_dht`). It is opt-in: + `CapabilityManifest::frid()` does not include it. - `music-dht 0.6.0`: new `MusicDhtError::PodcastPublication` variant (the enum is `#[non_exhaustive]`). - The music DHT (`music-dht-v5`), `federation-net` and device-sync protocol v4 diff --git a/PODCASTS.md b/PODCASTS.md index dcd9653..4d4b9bd 100644 --- a/PODCASTS.md +++ b/PODCASTS.md @@ -514,8 +514,8 @@ of one podcast by `podcast_id`. | --- | --- | | ALPN | `furumi-fd/podcast-dht/1` (schema-independent stream protocol) | | Wire and record version | `PODCAST_DHT_PROTOCOL_VERSION = 1` | -| Capability manifest | `podcast_dht = 1` (`CapabilityManifest::supports_podcast_dht`) | -| Cache | separate SQLite file, `PRAGMA user_version = 1`; a newer schema is refused | +| Capability manifest | `podcast_dht = 1`, opt-in (`CapabilityManifest::with_podcast_dht`, `supports_podcast_dht`) | +| Cache | separate SQLite file, `PRAGMA user_version = 2`; a newer schema is refused, the pre-release version 1 replica table is dropped | | Card TTL | `ACTIVE_RECORD_TTL` of the music DHT (12 h) | | Tombstone TTL | `TOMBSTONE_TTL` of the music DHT (24 h) | | Republish | `DEFAULT_REPUBLISH_INTERVAL` (10 min), expiry sweep `DEFAULT_EXPIRE_INTERVAL` (1 min) | @@ -534,7 +534,7 @@ HTTP validators or episodes. | --- | --- | | `version`, `network_id`, `publisher` | 1, the federation network, the device's iroh endpoint key | | `podcast_id` | must equal `PodcastId::from_feed_url(id_feed_url or feed_url)` | -| `feed_url` | canonical `FeedUrl` the feed is fetched from | +| `feed_url` | canonical `FeedUrl` the publisher fetches the feed from | | `id_feed_url` | the URL the id was derived from, present only when it differs from `feed_url` (feed moved) | | `title` | 1..300 characters, no control characters | | `author` | optional, ≤ 300 characters, no control characters | @@ -550,10 +550,14 @@ HTTP validators or episodes. whole signed record is at most 24 KiB postcard-encoded; the field bounds keep every valid card below it. -Binding the id to a URL in the card prevents claiming an existing podcast id -for an arbitrary feed. When a feed moved, `feed_url` is the publisher's -claim; clients should prefer their own subscription's URLs for a podcast -they already know. +Only the **origin URL** (`PodcastCard::origin_feed_url`: `id_feed_url`, or +`feed_url` when the feed has not moved) is bound to the podcast, because the +id is its hash. When `id_feed_url` is present, `feed_url` is an unverified +claim of the publisher that the feed moved: anyone can sign a card that +carries an existing podcast's origin URL and an arbitrary `feed_url`. Search +therefore never offers a claimed URL as the podcast's feed (see Search), and +clients must not fetch it as that podcast or record it as an alias unless the +feed at the origin URL itself redirects there. ### Tombstone @@ -566,6 +570,10 @@ lower or equal revision and is itself retained and served until it expires. Because the tombstone TTL exceeds the card TTL, every card issued before a tombstone expires before it. +`index_keys` is self-asserted, so it never creates a placement by itself: a +replica indexes a tombstone under a text key only in place of a record of the +same `(podcast_id, publisher)` it already holds there (see Replica storage). + ### Signing input Both records are signed with the publisher's Ed25519 iroh identity over a @@ -594,9 +602,10 @@ postcard `SignedPodcastRecord { body: Card | Tombstone, signature }`. material)` with kind `0x01` and the podcast id string for the id key, or kind `0x02` and a token for a text key. Tokens are the distinct `normalize_name` tokens of the title, then of the author, skipping tokens -longer than 64 bytes, at most 32. A card is stored under the id key and its +longer than 64 bytes, at most 32. A card belongs under the id key and its text keys (at most 33); a tombstone under the id key and its `index_keys`. -Replicas accept a record only under a key derived from it. +The publisher sends each record to the `K` (8) overlay nodes closest to each +of its keys, one `StoreBatch` per node listing the keys that node should hold. ### Ordering @@ -604,10 +613,10 @@ For one `(podcast_id, publisher)` the greater record wins: higher `revision`; at an equal revision a tombstone beats a card; then later `issued_at_ms` (a republish); then later `expires_at_ms`; then the lexicographically greater signing input. This is a total order, so every -replica converges whatever the delivery order. A replica that holds a -tombstone, or a card of a higher revision, under any key drops older cards of -that publisher under every key and refuses them later; a replayed old -revision is therefore ignored. +replica and every lookup converge whatever the delivery order. A replica +keeps only the winning record of each `(podcast_id, publisher)`; a replayed +old revision is ignored. These fields only order one publisher's own records: +they never rank or select records of different publishers. Revisions: a changed card gets `max(previous + 1, now_ms)` (time-seeded, so revisions stay monotonic even if the local cache is lost); an unchanged card @@ -615,12 +624,80 @@ is re-signed with the same revision and a fresh lifetime on every republish; unpublishing writes a tombstone with `previous + 1`; publishing again later gets `max(tombstone + 1, now_ms)`. +### Wire protocol + +One request per bidirectional stream on `furumi-fd/podcast-dht/1`: the +requester writes one postcard `PodcastDhtRequest` and finishes its send side; +the responder writes one postcard `PodcastDhtResponse` and finishes. Messages +are not length-prefixed; the stream end delimits them. Requests above +512 KiB and responses above 4 MiB are refused. + +```text +PodcastDhtRequest { version: u16, requester: NodeContact, operation: PodcastDhtOperation } +PodcastDhtOperation = 0: FindNode { target: [u8; 32] } + | 1: FindValue { key: PodcastDhtKey, filter_tokens: Vec } + | 2: StoreBatch { records: Vec } +PodcastStoreEntry { keys: Vec, record: SignedPodcastRecord } +PodcastDhtResponse { version: u16, ok: bool, error: Option, + nodes: Vec, records: Vec, stored: u32 } +NodeContact { node_id: [u8; 32], peer_id: EndpointId, ticket: String, last_seen_ms: u64 } +PodcastRecordBody = 0: Card(PodcastCard) | 1: Tombstone(PodcastTombstone) +``` + +Fields are in postcard order. `PodcastDhtKey` is 32 raw bytes. + +* `version` must be 1 in both directions. `requester.peer_id` must equal the + authenticated transport peer, `node_id` must be `NodeId::from_endpoint` + of it and its ticket must name the same endpoint and network; otherwise the + request is refused. The responder learns the requester as an overlay + contact. +* `FindNode` returns the `K` closest overlay contacts (the responder + included). +* `FindValue` returns the records under `key` (see Replica storage) and the + `K` closest contacts. `filter_tokens` (≤ 8, each 1..64 bytes) are + normalized tokens every returned card must be indexed under; tombstones are + returned regardless. Requesters verify every record again and drop records + that do not belong under `key` or miss a filter token. +* `StoreBatch` carries 1..16 entries, each with 1..33 distinct keys. Every + record must be published by the requester itself (replicas never forward + records), pass `verify_at` (bounds, network, lifetime, clock skew, + signature, size) and belong under each listed key; otherwise the whole + batch is refused. Keys the responder is not responsible for are skipped. + `stored` counts the `(key, record)` placements written. +* A refused request gets `ok = false` with an `error` of at most 1 KiB and + empty `nodes`, `records` and `stored = 0`; an accepted one has `ok = true` + and no `error`. Requesters reject responses that violate this or exceed + `K` nodes or 128 records. + +### Replica storage + +* A replica stores one payload per `(podcast_id, publisher)` (the winning + record) and indexes it under the keys it holds. A newer version replaces + it in place; placements the new version no longer belongs under (an old + title's tokens, keys a tombstone does not list) are dropped, so on one + replica a tombstone supersedes the card under every key it held. +* A node accepts a placement only when fewer than `2 * K` of its known + overlay contacts are closer to the key than itself. +* Admission, never eviction: a key holds at most 256 records. A full key + admits no new `(podcast_id, publisher)` until one expires; existing records + keep their place and their publishers refresh them in place. A tombstone + is indexed under a text key only in place of its owner's record there; new + tombstones under a key (in practice the id key, for cards this replica + never saw) are limited to 32. +* A replica holds at most 256 records per publisher, 4 096 records and + 32 MiB of payload; records beyond that are refused, not evicted. At most + 1 024 overlay contacts are persisted. +* `FindValue` answers with at most 128 records: at most 32 tombstones and + the cards indexed longest first. Every record is verified again on every + read. + ### Publication * `PodcastDht::publish(subscription, share, card)` refuses (`MusicDhtError::PodcastPublication`) unless `effective_share` holds, so a new or resumed subscription is never published without a fresh share - decision. + decision. The UI calls it only after asking the user about any privacy + warning, so the call is the user's consent. * `PodcastDht::sync_shares(states)` is declarative and idempotent: podcasts for which `effective_share` holds are published (when changed, new or past half their lifetime), every other podcast this device published is @@ -628,17 +705,27 @@ gets `max(tombstone + 1, now_ms)`. operations, so unsharing or unsubscribing on any trusted device revokes every device's own card once the registers converge, and a stale device cannot republish (its registers lose to the newer share decision). +* Revocation is fail-safe: each podcast is handled on its own. A shared + podcast whose card cannot be built or signed is reported in + `PodcastDhtSyncStats::failed` and treated as not shared (an earlier + publication is tombstoned); it never prevents other podcasts from being + unpublished. +* A card whose `PodcastCardSpec::privacy_warnings` is not empty is held back + (`PodcastDhtSyncStats::held_back`, with the warnings) unless + `PodcastShareState::privacy_confirmed` is set or this device already + publishes the same `feed_url` and `id_feed_url`. A feed move to a URL that + looks private therefore asks again; an earlier publication of a held-back + podcast stays as it was. * `PodcastDht::unpublish(podcast_id)` writes the tombstone. Nothing is ever sent for a podcast this device never published: no tombstone, no id. * `PodcastDht::republish` (run by `maintenance`) re-signs active cards and re-sends unexpired tombstones every 10 minutes. -* Before publishing, the UI must call `feed_url_privacy_warning` (or - `PodcastCardSpec::privacy_warnings`, which also checks `id_feed_url`) and - ask the user when it returns a warning. The heuristics are advisory: query +* The privacy heuristics (`feed_url_privacy_warning`) are advisory: query parameter names containing words such as `token`, `key`, `auth`, `secret`, `sig`/`signature`, `uid`, `user`, `pass`, `session`, `access`, `private`; known member-feed hosts; and long high-entropy query values (≥ 16 - characters) or path segments (≥ 32 characters). False positives and misses + characters) or path segments (≥ 32 characters). They check `feed_url` and + `id_feed_url`, not artwork or web-page links. False positives and misses are possible. ### Search @@ -649,39 +736,54 @@ gets `max(tombstone + 1, now_ms)`. cards indexed under **all** query tokens (tombstones are always returned). Results are verified again, the winning record per publisher is kept, tombstoned publishers are dropped and cards are merged by `podcast_id` - into `PodcastSearchHit { podcast_id, card, publishers }`. Ranking: exact - normalized title first, then more publishers. + into `PodcastSearchHit { podcast_id, feed_url, card, publishers, + claimed_feed_urls }`. * `PodcastDht::lookup_podcast(podcast_id)` reads the id key. -* `publishers` lists distinct publisher **devices**; it is not a subscriber +* `feed_url` is the origin URL; it is the only URL a client subscribes with. + `claimed_feed_urls` lists other `feed_url`s that publishers claim (at most + 8, with their publishers), for display only. +* The representative `card` is a card whose `feed_url` is the origin URL + when one exists, then the title most publishers agree on, then the + smallest publisher key. Publisher-controlled times and revisions never + break ties. Its metadata remains a publisher's claim. +* `publishers` lists distinct publisher **devices**. It is not a subscriber count (one user may publish from several devices, and subscribers who did - not share are invisible), so no subscriber count is offered. -* The representative card is the one whose `feed_url` most publishers agree - on, then the most recently issued, then the greatest publisher key. + not share are invisible) and not a trust signal: device identities are + free. No subscriber count is offered and ranking ignores it. +* Ranking: exact normalized title first, then hits that have a card under + the origin URL, then title and id. * `FederatedSearchHit` (`{"type": "music" | "podcast", "hit": ...}`), `SearchHitKind` (`artist`, `release`, `track`, `podcast`) and `merge_search_results` merge podcast hits with `SearchOutcome` music results; `SearchOutcome` itself is unchanged. * Lookups query the 8 closest overlay nodes (α = 3, at most 32 requests, - 15 s deadline) and return partial results when peers fail. A peer that - fails a request is skipped for 30 s. + 15 s deadline), keep at most 512 `(podcast_id, publisher)` records each and + return partial results when peers fail. A peer that fails a request is + skipped for 30 s. -Subscribing from a hit creates the user's own private subscription: resolve -the id with `resolve_podcast_id` on the card's `id_feed_url` (or `feed_url`), -which yields the card's `podcast_id` unless the user already knows that URL, -fetch the feed from `feed_url` and record it as an alias when it differs. The -new subscription is private (see `effective_share`), the publisher's -availability is not needed afterwards, and the episode catalog comes from RSS. +Subscribing from a hit creates the user's own private subscription with the +hit's `feed_url` (the origin URL): resolve the id with `resolve_podcast_id`, +which yields the hit's `podcast_id` unless the user already knows that URL, +fetch the feed from that URL and follow its own HTTP redirects and +`itunes:new-feed-url`, recording those as aliases as usual. A URL from +`claimed_feed_urls` becomes an alias only when that fetch leads there. The new +subscription is private (see `effective_share`), the publisher's availability +is not needed afterwards, and the episode catalog comes from RSS. ### Limits | Limit | Value | | --- | --- | | Request / response | 512 KiB / 4 MiB postcard | -| Store batch | 1..16 entries, each 1..33 distinct keys | -| Records per `FindValue` response | 128 | +| Store batch | 1..16 entries, each 1..33 distinct keys, records of the requester only | +| Records per `FindValue` response | 128, at most 32 of them tombstones | | Filter tokens | ≤ 8, each 1..64 bytes | -| Records cached per key | 256; the record expiring first is evicted for a longer-lived one | -| Concurrent inbound requests | 64 (more are dropped); 10 s per request | +| Records per key | 256 (new tombstones 32); admission only, no eviction | +| Records per replica | 256 per publisher, 4 096 in all, 32 MiB of payload | +| Store responsibility | fewer than `2 * K` = 16 known contacts closer to the key | +| Records kept per lookup | 512 `(podcast_id, publisher)` | +| Persisted overlay contacts | 1 024 | +| Inbound requests | 64 concurrent, 8 per peer (more are dropped); 10 s each | | Error message | ≤ 1 KiB | | Contact ticket | ≤ 16 KiB | @@ -692,20 +794,28 @@ availability is not needed afterwards, and the episode catalog comes from RSS. * Cards carry the public projection only (see Card); tombstones carry key hashes only. * Records are signed by the publisher device and verified on every receive - and on every cache read; a forwarder cannot alter or impersonate them, and - records from another network are rejected. -* Cards do not carry a connection ticket: discovering a card does not reveal - how to reach the publisher, and subscribing never contacts it. -* The publisher's device key is public in the card. Devices of one user are - not linked by the overlay, but publishing the same podcasts from several - devices makes them correlatable. + and on every cache read; a forwarder cannot alter or impersonate them, + only the publisher can store them, and records from another network are + rejected. +* The card's `publisher` is the device's persistent iroh endpoint key, which + is also its overlay and music-DHT identity. It is **linkable**: anyone can + look the key up in the overlay (`FindNode` for + `NodeId::from_endpoint(publisher)`) and obtain the device's contact ticket + with its relay and direct IP addresses, because every overlay request + carries the requester's ticket. The same key owns the device's music-DHT + records, so a device's shared podcasts can be linked to its public music + library and its network address. Subscribing from a card never contacts + the publisher. +* Devices of one user are not linked by the overlay, but publishing the same + podcasts from several devices makes them correlatable. * Unpublishing is eventually consistent: replicas that missed the tombstone - keep serving the old card until it expires (at most 12 h). A replica that - only held text keys of an older revision (after a title change) may serve - that revision until it expires. + keep serving the old card until it expires (at most 12 h), including + replicas that only held an older title's text keys. * Capability manifests are informational. A peer without `podcast_dht` never enters the overlay routing table (it cannot answer on the ALPN), and - `PodcastDht::note_peer_capabilities` evicts a peer whose manifest lacks it. + `PodcastDht::note_peer_capabilities` evicts a peer whose manifest lacks it + and stops dialing it. Applications advertise the overlay only when they + run it (`with_podcast_dht`). ## Feed parser @@ -879,5 +989,5 @@ must keep the same invariants. * Public podcast cards use the separate signed overlay `podcast_dht` with its own ALPN (`furumi-fd/podcast-dht/1`), wire/record version 1, signing domains `furumi-podcast-card:v1` / `furumi-podcast-card-tombstone:v1`, key - domain `furumi-podcast-dht:key:v1` and cache schema 1, all frozen; the music + domain `furumi-podcast-dht:key:v1`, all frozen, and cache schema 2; the music DHT schema and `federation-net` do not change. diff --git a/crates/music-dht/README.md b/crates/music-dht/README.md index f5cb318..e34c778 100644 --- a/crates/music-dht/README.md +++ b/crates/music-dht/README.md @@ -254,15 +254,27 @@ reported itself (`profile_protocol_version`; relayed profiles go through `music_dht::podcast_dht::PodcastDht` is a separate signed overlay (ALPN `furumi-fd/podcast-dht/1`, capability `podcast_dht`) for podcasts the user -explicitly shared. Declare the ALPN with -`schema_independent_stream_protocol`, open the overlay with its own cache file, -and spawn `serve` and `maintenance`, as for `SimilarityDht`. Call -`sync_shares` with the merged subscription and share registers after applying -sync operations: it publishes cards only where `effective_share` holds and -tombstones every other own publication. Ask the user before publishing when -`PodcastCardSpec::privacy_warnings` (or `feed_url_privacy_warning`) reports a -URL that looks private. `search` and `lookup_podcast` return -`PodcastSearchHit`s; `merge_search_results` combines them with +explicitly shared. Wiring, as for `SimilarityDht`: + +1. declare the ALPN with `schema_independent_stream_protocol(PODCAST_DHT_ALPN)`, + open the overlay with its own cache file, take + `stream_acceptor(PODCAST_DHT_ALPN)` and spawn `serve` and `maintenance`; +2. advertise it with `CapabilityManifest::with_podcast_dht()` (it is not in + `CapabilityManifest::frid()`; an application that does not run the overlay + simply does not add it); +3. pass every peer manifest received to `note_peer_capabilities`, so peers + without the overlay are not dialed; +4. call `sync_shares` with the merged subscription and share registers after + applying sync operations: it publishes cards only where `effective_share` + holds, tombstones every other own publication, and reports cards it could + not build (`failed`) or held back for privacy (`held_back`); +5. ask the user before publishing when `PodcastCardSpec::privacy_warnings` + (or `feed_url_privacy_warning`) reports a URL that looks private, then + call `publish` or set `PodcastShareState::privacy_confirmed`. + +`search` and `lookup_podcast` return `PodcastSearchHit`s; subscribe with the +hit's `feed_url` (the URL the id hashes from), never with a +`claimed_feed_urls` entry. `merge_search_results` combines hits with `SearchOutcome::network_results` as typed `FederatedSearchHit`s. ## Verification