Document podcast replica eviction and contact verification

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
sanapachandClaude Opus 5.5 committed 2026-11-19 18:50:22 +02:00
1 parent c0def74115
commit 912527a671
3 files changed
+57 -26

No files matched your search

+6 -3
View File
@@ -251,9 +251,12 @@ per normalized title/author token; unpublishing writes a signed tombstone of
the next revision that supersedes lower and equal revisions on replicas that the next revision that supersedes lower and equal revisions on replicas that
hold it, for the keys it lists, and outlives every older card. Only a record's hold it, for the keys it lists, and outlives every older card. Only a record's
publisher may store it; replicas keep one payload per publisher and podcast, publisher may store it; replicas keep one payload per publisher and podcast,
admit placements only for keys they are close to, within per-key, admit placements only for keys they are close to (counting only contacts that
per-publisher and total budgets, and never evict existing records for answered or authenticated, never ones another peer suggested), within per-key,
newcomers (tombstones cannot displace cards). Records use the music DHT's per-publisher and total budgets. A key holds no more records than one response
carries; a full key or cache admits a newcomer by evicting a random record from
its younger half, so the older half survives floods and capacity is never
first-come-forever (over their budget, tombstones only displace other tombstones). Records use the music DHT's
active/tombstone TTLs and republish cadence, are verified on every receive and active/tombstone TTLs and republish cadence, are verified on every receive and
cache read, and are kept in a separate, replaceable SQLite cache. Search cache read, and are kept in a separate, replaceable SQLite cache. Search
filters on every replica by all query tokens, merges cards of one podcast filters on every replica by all query tokens, merges cards of one podcast
+6 -3
View File
@@ -14,9 +14,12 @@
`feed_url_privacy_warning`. `feed_url_privacy_warning`.
- Replicas accept records only from their publisher, store one payload per - Replicas accept records only from their publisher, store one payload per
`(podcast_id, publisher)`, admit placements only for keys they are close to `(podcast_id, publisher)`, admit placements only for keys they are close to
and within per-key, per-publisher and total budgets, and never evict (counting verified contacts only) and within per-key, per-publisher and
existing records for newcomers. Search offers only the URL the podcast id total budgets. A key never holds more than one response carries; full keys
hashes from; moved-feed URLs are reported as unverified claims. and caches admit newcomers by evicting random records from their younger
half. Lookups give each response a bounded share of owners. Search offers
only the URL the podcast id hashes from; moved-feed URLs are reported as
unverified claims.
- `sync_shares` is fail-safe for revocation (per-podcast `failed`) and holds - `sync_shares` is fail-safe for revocation (per-podcast `failed`) and holds
back cards with privacy warnings until confirmed (`held_back`, back cards with privacy warnings until confirmed (`held_back`,
`PodcastShareState::privacy_confirmed`). `PodcastShareState::privacy_confirmed`).
+45 -20
View File
@@ -650,7 +650,10 @@ Fields are in postcard order. `PodcastDhtKey` is 32 raw bytes.
authenticated transport peer, `node_id` must be `NodeId::from_endpoint` authenticated transport peer, `node_id` must be `NodeId::from_endpoint`
of it and its ticket must name the same endpoint and network; otherwise the of it and its ticket must name the same endpoint and network; otherwise the
request is refused. The responder learns the requester as an overlay request is refused. The responder learns the requester as an overlay
contact. contact (it is authenticated).
* `nodes` in a response are unverified suggestions. A requester treats them
as lookup or bootstrap candidates only and adds one to its routing table
(and persisted contacts) only after it answered a request itself.
* `FindNode` returns the `K` closest overlay contacts (the responder * `FindNode` returns the `K` closest overlay contacts (the responder
included). included).
* `FindValue` returns the records under `key` (see Replica storage) and the * `FindValue` returns the records under `key` (see Replica storage) and the
@@ -663,7 +666,9 @@ Fields are in postcard order. `PodcastDhtKey` is 32 raw bytes.
records), pass `verify_at` (bounds, network, lifetime, clock skew, records), pass `verify_at` (bounds, network, lifetime, clock skew,
signature, size) and belong under each listed key; otherwise the whole signature, size) and belong under each listed key; otherwise the whole
batch is refused. Keys the responder is not responsible for are skipped. batch is refused. Keys the responder is not responsible for are skipped.
`stored` counts the `(key, record)` placements written. `stored` counts the `(key, record)` placements written. A publisher
offers the placements of a batch answered with `stored = 0` (or that
failed) once to the next closest node of each key.
* A refused request gets `ok = false` with an `error` of at most 1 KiB and * A refused request gets `ok = false` with an `error` of at most 1 KiB and
empty `nodes`, `records` and `stored = 0`; an accepted one has `ok = true` empty `nodes`, `records` and `stored = 0`; an accepted one has `ok = true`
and no `error`. Requesters reject responses that violate this or exceed and no `error`. Requesters reject responses that violate this or exceed
@@ -676,17 +681,34 @@ Fields are in postcard order. `PodcastDhtKey` is 32 raw bytes.
it in place; placements the new version no longer belongs under (an old it in place; placements the new version no longer belongs under (an old
title's tokens, keys a tombstone does not list) are dropped, so on one title's tokens, keys a tombstone does not list) are dropped, so on one
replica a tombstone supersedes the card under every key it held. replica a tombstone supersedes the card under every key it held.
* A node accepts a placement only when fewer than `2 * K` of its known * A node accepts a placement only when fewer than `2 * K` of its verified
overlay contacts are closer to the key than itself. overlay contacts that are not backed off are closer to the key than
* Admission, never eviction: a key holds at most 256 records. A full key itself. Contacts only enter the routing table by answering a request or
admits no new `(podcast_id, publisher)` until one expires; existing records by sending one, so a peer cannot make a replica refuse keys by returning
keep their place and their publishers refresh them in place. A tombstone made-up contacts. A contact that fails three times in a row (each after
is indexed under a text key only in place of its owner's record there; new its 30 s backoff) is removed.
tombstones under a key (in practice the id key, for cards this replica * A key holds at most 128 records, at most 32 of them tombstones: exactly
never saw) are limited to 32. what one `FindValue` response carries, so every admitted record is served.
* A replica holds at most 256 records per publisher, 4 096 records and A tombstone is indexed under a text key only in place of its owner's
32 MiB of payload; records beyond that are refused, not evicted. At most record there, and such an in-place conversion counts against the
1 024 overlay contacts are persisted. tombstone budget like a new tombstone.
* Full key: a new placement evicts a random placement of another owner from
the younger half of the key (by first indexing time; only tombstones when
the tombstone budget is full). The older half is never evicted for a
newcomer, so records placed before a flood keep their place and their
publishers refresh them in place. Capacity is never first-come-forever:
a newcomer always gets a place, though placements in the younger half can
be evicted again by later newcomers.
* A replica holds at most 256 records per publisher (a publisher over it is
refused), 4 096 records and 32 MiB of payload. A full cache admits a new
record by evicting random records of other owners from its younger half,
at most 16; when that is not enough room, nothing is evicted and the record
is refused. At most 1 024 overlay contacts are persisted.
* Device identities cost nothing, so these rules bound flooding but cannot
prevent it: an attacker with many identities can keep part of the younger
half of a key or cache busy, and a newcomer's place there can be evicted
again until its publisher's next republish (every 10 min) puts it back.
Protocol 1 has no proof of work or other identity cost.
* `FindValue` answers with at most 128 records: at most 32 tombstones and * `FindValue` answers with at most 128 records: at most 32 tombstones and
the cards indexed longest first. Every record is verified again on every the cards indexed longest first. Every record is verified again on every
read. read.
@@ -764,9 +786,12 @@ Fields are in postcard order. `PodcastDhtKey` is 32 raw bytes.
`merge_search_results` merge podcast hits with `SearchOutcome` music `merge_search_results` merge podcast hits with `SearchOutcome` music
results; `SearchOutcome` itself is unchanged. results; `SearchOutcome` itself is unchanged.
* Lookups query the 8 closest overlay nodes (α = 3, at most 32 requests, * Lookups query the 8 closest overlay nodes (α = 3, at most 32 requests,
15 s deadline), keep at most 512 `(podcast_id, publisher)` records each and 15 s deadline) and return partial results when peers fail. They keep at
return partial results when peers fail. A peer that fails a request is most 512 `(podcast_id, publisher)` owners; each response (and the local
skipped for 30 s. cache) may add at most 64 new owners (`512 / K`), so the responders that
answer first cannot fill a lookup and crowd out the rest. A newer record
of a known owner always replaces its winner. A peer that fails a request
is skipped for 30 s.
Subscribing from a hit creates the user's own private subscription with the Subscribing from a hit creates the user's own private subscription with the
hit's `feed_url` (the origin URL): resolve the id with `resolve_podcast_id`, hit's `feed_url` (the origin URL): resolve the id with `resolve_podcast_id`,
@@ -785,10 +810,10 @@ is not needed afterwards, and the episode catalog comes from RSS.
| Store batch | 1..16 entries, each 1..33 distinct keys, records of the requester only | | Store batch | 1..16 entries, each 1..33 distinct keys, records of the requester only |
| Records per `FindValue` response | 128, at most 32 of them tombstones | | Records per `FindValue` response | 128, at most 32 of them tombstones |
| Filter tokens | ≤ 8, each 1..64 bytes | | Filter tokens | ≤ 8, each 1..64 bytes |
| Records per key | 256 (new tombstones 32); admission only, no eviction | | Records per key | 128, at most 32 of them tombstones; random eviction from the younger half |
| Records per replica | 256 per publisher, 4 096 in all, 32 MiB of payload | | Records per replica | 256 per publisher (refused beyond), 4 096 in all, 32 MiB of payload (random eviction from the younger half, at most 16 per record) |
| Store responsibility | fewer than `2 * K` = 16 known contacts closer to the key | | Store responsibility | fewer than `2 * K` = 16 verified, not backed-off contacts closer to the key |
| Records kept per lookup | 512 `(podcast_id, publisher)` | | Records kept per lookup | 512 `(podcast_id, publisher)`, at most 64 new ones per response |
| Persisted overlay contacts | 1 024 | | Persisted overlay contacts | 1 024 |
| Inbound requests | 64 concurrent, 8 per peer (more are dropped); 10 s each | | Inbound requests | 64 concurrent, 8 per peer (more are dropped); 10 s each |
| Error message | ≤ 1 KiB | | Error message | ≤ 1 KiB |