Подкасты в FRID: публичное обнаружение podcast_dht — подписанные карточки, tombstone и поиск (MR-F3) #3

Merged
ab merged 9 commits from sanapach/frid:podcasts/discovery into main 2026-11-20 13:51:54 +00:00
16 changed files with 7077 additions and 10 deletions

No files matched your search

+38 -2
View File
@@ -232,8 +232,44 @@ refused, and their playback commands are ignored while the queue contains
episodes. Jam snapshots replace episodes with opaque placeholders. Progress,
subscription and share registers order by a Lamport generation before
wall-clock `hlc_ms`, so a device with a fast clock cannot undo an observed
seek, restart, unsubscribe or share decision. Public podcast discovery is a future
separate overlay; the music DHT schema and `federation-net` are unchanged.
seek, restart, unsubscribe or share decision. The music DHT schema and
`federation-net` are unchanged.
Public podcast discovery is a third schema-independent Kademlia overlay,
`music_dht::podcast_dht` (ALPN `furumi-fd/podcast-dht/1`, version 1,
capability `podcast_dht`), built like the similarity-routing overlay. A
device publishes a card only for podcasts whose merged registers satisfy
`effective_share` (SQLite clients read the merged registers from
`PodcastStore`); `sync_shares` reconciles publications declaratively and
nothing is sent for a podcast the device never published. Own cards persisted
by an earlier run are neither republished nor served until the first
`sync_shares` of the process, so a stale cache cannot revive a share revoked
elsewhere while the device was offline. A card is the
public projection of a podcast (feed URL and channel metadata, no personal
state), signed by the device's iroh identity over a domain-separated
canonical encoding. Only the origin URL the id hashes from is bound to the
podcast; a moved `feed_url` is an unverified publisher claim that search never
offers as the podcast's feed. Cards live under the podcast-id key and one key
per normalized title/author token; unpublishing writes a signed tombstone of
the next revision that supersedes lower and equal revisions on replicas that
hold it, for the keys it lists, and outlives every older card. Only a record's
publisher may store it; replicas keep one payload per publisher and podcast,
admit placements only for keys they are close to (counting only contacts that
answered or authenticated, never ones another peer suggested), within per-key,
per-publisher and total budgets. A key holds no more records than one response
carries; a full key or cache admits a newcomer by evicting a random record from
its younger half, so the older half survives floods and capacity is never
first-come-forever (over their budget, tombstones only displace other tombstones). Records use the music DHT's
active/tombstone TTLs and republish cadence, are verified on every receive and
cache read, and are kept in a separate, replaceable SQLite cache. Search
filters on every replica by all query tokens, merges cards of one podcast
across publisher devices, and returns typed `podcast` hits that clients merge
with music results; publisher counts are not a trust signal. The publisher key
is the device's transport identity, so a card is linkable to the device's
network address and music-DHT records. Applications advertise the
`podcast_dht` capability only when they run the overlay; capability data only
helps skip peers, it never authorizes. See the "Public discovery" section of
[PODCASTS.md](PODCASTS.md).
Music discovery identifies an owner and content; it does not force all domain
traffic through DHT messages. `music-dht` exposes the underlying authenticated
+38
View File
@@ -1,5 +1,43 @@
# Changelog
## Unreleased — podcast discovery (`podcast_dht`)
- `music-dht 0.6.0`: new `music_dht::podcast_dht` overlay for public podcast
discovery on its own ALPN `furumi-fd/podcast-dht/1` (version 1) and its own
SQLite cache (schema 2): signed `PodcastCard`s and `PodcastTombstone`s
(`furumi-podcast-card:v1` / `furumi-podcast-card-tombstone:v1` canonical
signing input), monotonic revisions with a total tie-break, text and id
keys, `PodcastDht::{publish, unpublish, sync_shares, republish, search,
lookup_podcast}` gated by `effective_share`, typed search hits
(`PodcastSearchHit`, `ClaimedFeedUrl`, `FederatedSearchHit`,
`SearchHitKind`, `merge_search_results`) and the advisory
`feed_url_privacy_warning`.
- Replicas accept records only from their publisher, store one payload per
`(podcast_id, publisher)`, admit placements only for keys they are close to
(counting verified contacts only) and within per-key, per-publisher and
total budgets. A key never holds more than one response carries; full keys
and caches admit newcomers by evicting random records from their younger
half. Lookups give each response a bounded share of owners. Search offers
only the URL the podcast id hashes from; moved-feed URLs are reported as
unverified claims.
- `sync_shares` is fail-safe for revocation (per-podcast `failed`) and holds
back cards with privacy warnings until confirmed (`held_back`,
`PodcastShareState::privacy_confirmed`).
- Own cards persisted by an earlier run are neither republished (by
`maintenance` or `republish`) nor served to lookups until the first
`sync_shares` of the process, so a share revoked on another trusted device
while this one was offline is not re-sent on startup. That first call also
re-sends unchanged cards; tombstones are always re-sent.
- `music-dht 0.6.0`: capability manifests can advertise `podcast_dht = 1`
(`capabilities::PODCAST_DHT_ID`, `CapabilityManifest::with_podcast_dht`,
`CapabilityManifest::protocol_version`,
`CapabilityManifest::supports_podcast_dht`). It is opt-in:
`CapabilityManifest::frid()` does not include it.
- `music-dht 0.6.0`: new `MusicDhtError::PodcastPublication` variant (the enum
is `#[non_exhaustive]`).
- The music DHT (`music-dht-v5`), `federation-net` and device-sync protocol v4
are unchanged. See "Public discovery (podcast_dht)" in `PODCASTS.md`.
## Unreleased — podcast contract
- `music-dht 0.6.0` (breaking): new `music_dht::podcast` module with the shared
+379 -7
View File
@@ -10,8 +10,9 @@ live in `music_dht::podcast` (pure types and functions) and
`music_dht::device_sync` (wire DTOs and gating helpers). Clients must use
them instead of local copies.
Public discovery (signed podcast cards in a separate DHT overlay) is not part
of this version of the contract; it will be specified with the overlay.
Public discovery of podcasts a user explicitly shared uses signed podcast
cards in a separate DHT overlay, `music_dht::podcast_dht`; see
[Public discovery (podcast_dht)](#public-discovery-podcast_dht).
## Responsibilities
@@ -19,7 +20,8 @@ of this version of the contract; it will be specified with the overlay.
| --- | --- |
| `music_dht::podcast` | ids, URL canonicalization, no-GUID rule, metadata DTOs and bounds, merge functions, RSS/Atom parser (feature `podcast-feed`) |
| `music_dht::device_sync` | sync operation kinds, snapshot sections, `PlaybackSource`, version gating, Jam redaction |
| Client storage (`furumi_library::podcasts::PodcastStore`, web PostgreSQL) | persistence and migrations implementing the rules below (SQLite reference: see "SQLite reference store") |
| `music_dht::podcast_dht` | public podcast cards and tombstones, their signatures, replication, cache and search, privacy warnings for feed URLs |
| Client storage (`furumi_library::podcasts::PodcastStore`, web PostgreSQL) | persistence and migrations implementing the rules below (SQLite reference: [SQLite reference store](#sqlite-reference-store)) |
| Client runtime | HTTP fetching, conditional requests, scheduling, audio buffering and decoding, UI |
Audio is always fetched from the publisher. It is never imported into the
@@ -32,7 +34,7 @@ offered for P2P transfer. RSS refreshes fetch metadata only.
| --- | --- | --- | --- |
| Subscription state, channel metadata | `podcast_subscription_set` + snapshot | never | |
| Feed aliases | `podcast_feed_aliases_added`, `podcast_subscription_set` + snapshot | never | |
| Share decision | `podcast_share_set` + snapshot | only the published card (future overlay) | |
| Share decision | `podcast_share_set` + snapshot | only the signed card while `effective_share` holds (`podcast_dht`) | |
| Episode progress | `podcast_progress_set` + snapshot | never | |
| Listening sessions, play counts | `podcast_listen_recorded` + snapshot | never | |
| Episode catalog | each device parses RSS itself; `PodcastEpisodeRef` travels with progress and queue items | never | |
@@ -498,6 +500,371 @@ Jam participants are outside the trusted-device group.
private item. States that drop, duplicate or inject private items are
rejected. Participants may still pause, seek and reorder music.
## Public discovery (podcast_dht)
`music_dht::podcast_dht` publishes podcasts the user explicitly shared so
that peers outside the user's trusted-device group can find and subscribe to
them. It is a separate, schema-independent Kademlia overlay built like
`similarity_dht`: its own ALPN, version, routing contacts and SQLite cache.
The music DHT (`music-dht-v5`) and `federation-net` are unchanged. Every
device of a user publishes under its own identity; search merges the cards
of one podcast by `podcast_id`.
| Item | Value |
| --- | --- |
| ALPN | `furumi-fd/podcast-dht/1` (schema-independent stream protocol) |
| Wire and record version | `PODCAST_DHT_PROTOCOL_VERSION = 1` |
| Capability manifest | `podcast_dht = 1`, opt-in (`CapabilityManifest::with_podcast_dht`, `supports_podcast_dht`) |
| Cache | separate SQLite file, `PRAGMA user_version = 2`; a newer schema is refused, the pre-release version 1 replica table is dropped |
| Card TTL | `ACTIVE_RECORD_TTL` of the music DHT (12 h) |
| Tombstone TTL | `TOMBSTONE_TTL` of the music DHT (24 h) |
| Republish | `DEFAULT_REPUBLISH_INTERVAL` (10 min), expiry sweep `DEFAULT_EXPIRE_INTERVAL` (1 min) |
| Clock skew | `issued_at_ms` at most 5 min in the future |
### Card
A card is the public projection of one podcast by one publisher device. It
is built from the subscription with `PodcastCardSpec::from_subscription`,
which reads only the podcast id, the feed URLs and the channel metadata. It
never contains progress, listening history, play counts, subscription or
share registers (`generation`, `hlc_ms`, `op_id`), device names, local paths,
HTTP validators or episodes.
| Field | Bound |
| --- | --- |
| `version`, `network_id`, `publisher` | 1, the federation network, the device's iroh endpoint key |
| `podcast_id` | must equal `PodcastId::from_feed_url(id_feed_url or feed_url)` |
| `feed_url` | canonical `FeedUrl` the publisher fetches the feed from |
| `id_feed_url` | the URL the id was derived from, present only when it differs from `feed_url` (feed moved) |
| `title` | 1..300 characters, no control characters |
| `author` | optional, ≤ 300 characters, no control characters |
| `description` | optional, ≤ 1 000 characters, only `\n`/`\t` controls |
| `image_url`, `link` | optional absolute `http(s)` URLs ≤ 2 048 bytes |
| `language` | optional, ≤ 35 bytes of ASCII letters, digits and `-` |
| `categories` | ≤ 8, each 1..64 characters, no control characters |
| `episode_count`, `latest_episode_at_ms` | optional; the time is ≥ 0 |
| `revision` | ≥ 1, monotonic per `(podcast_id, publisher)` |
| `issued_at_ms`, `expires_at_ms` | `0 < issued < expires ≤ issued + 12 h` |
`PodcastCardSpec::bounded` truncates publisher data to these bounds. The
whole signed record is at most 24 KiB postcard-encoded; the field bounds keep
every valid card below it.
Only the **origin URL** (`PodcastCard::origin_feed_url`: `id_feed_url`, or
`feed_url` when the feed has not moved) is bound to the podcast, because the
id is its hash. When `id_feed_url` is present, `feed_url` is an unverified
claim of the publisher that the feed moved: anyone can sign a card that
carries an existing podcast's origin URL and an arbitrary `feed_url`. Search
therefore never offers a claimed URL as the podcast's feed (see Search), and
clients must not fetch it as that podcast or record it as an alias unless the
feed at the origin URL itself redirects there.
### Tombstone
`(version, network_id, publisher, podcast_id, revision, issued_at_ms,
expires_at_ms, index_keys)` with `expires_at_ms ≤ issued_at_ms + 24 h` and
`index_keys` the sorted, distinct text keys of the retracted card (at most
32, without the podcast-id key). It carries key hashes, not the title. A
tombstone supersedes every card of the same `(podcast_id, publisher)` with a
lower or equal revision and is itself retained and served until it expires.
Because the tombstone TTL exceeds the card TTL, every card issued before a
tombstone expires before it.
`index_keys` is self-asserted, so it never creates a placement by itself: a
replica indexes a tombstone under a text key only in place of a record of the
same `(podcast_id, publisher)` it already holds there (see Replica storage).
### Signing input
Both records are signed with the publisher's Ed25519 iroh identity over a
canonical, language-independent encoding: the domain string, a `0x00` byte,
then the fields in the order listed. Integers are little-endian (`u16`,
`u32`, `u64`, `i64`); strings are a `u32` byte length followed by UTF-8;
`Option` is `0x00`, or `0x01` followed by the value; lists are a `u32` count
followed by the items; `network_id`, `publisher` and keys are their raw 32
bytes.
* Card, domain `furumi-podcast-card:v1`: `version u16, network_id, publisher,
podcast_id str, feed_url str, id_feed_url opt str, title str, author opt
str, description opt str, image_url opt str, link opt str, language opt
str, categories list of str, episode_count opt u32, latest_episode_at_ms
opt i64, revision u64, issued_at_ms u64, expires_at_ms u64`.
* Tombstone, domain `furumi-podcast-card-tombstone:v1`: `version u16,
network_id, publisher, podcast_id str, revision u64, issued_at_ms u64,
expires_at_ms u64, index_keys list of 32-byte keys`.
Unit tests pin BLAKE3 vectors of both encodings. On the wire, records are
postcard `SignedPodcastRecord { body: Card | Tombstone, signature }`.
### Keys and placement
`key = BLAKE3("furumi-podcast-dht:key:v1" || 0x00 || network_id || kind ||
material)` with kind `0x01` and the podcast id string for the id key, or
kind `0x02` and a token for a text key. Tokens are the distinct
`normalize_name` tokens of the title, then of the author, skipping tokens
longer than 64 bytes, at most 32. A card belongs under the id key and its
text keys (at most 33); a tombstone under the id key and its `index_keys`.
The publisher sends each record to the `K` (8) overlay nodes closest to each
of its keys, one `StoreBatch` per node listing the keys that node should hold.
### Ordering
For one `(podcast_id, publisher)` the greater record wins: higher
`revision`; at an equal revision a tombstone beats a card; then later
`issued_at_ms` (a republish); then later `expires_at_ms`; then the
lexicographically greater signing input. This is a total order, so every
replica and every lookup converge whatever the delivery order. A replica
keeps only the winning record of each `(podcast_id, publisher)`; a replayed
old revision is ignored. These fields only order one publisher's own records:
they never rank or select records of different publishers.
Revisions: a changed card gets `max(previous + 1, now_ms)` (time-seeded, so
revisions stay monotonic even if the local cache is lost); an unchanged card
is re-signed with the same revision and a fresh lifetime on every republish;
unpublishing writes a tombstone with `previous + 1`; publishing again later
gets `max(tombstone + 1, now_ms)`.
### Wire protocol
One request per bidirectional stream on `furumi-fd/podcast-dht/1`: the
requester writes one postcard `PodcastDhtRequest` and finishes its send side;
the responder writes one postcard `PodcastDhtResponse` and finishes. Messages
are not length-prefixed; the stream end delimits them. Requests above
512 KiB and responses above 4 MiB are refused.
```text
PodcastDhtRequest { version: u16, requester: NodeContact, operation: PodcastDhtOperation }
PodcastDhtOperation = 0: FindNode { target: [u8; 32] }
| 1: FindValue { key: PodcastDhtKey, filter_tokens: Vec<String> }
| 2: StoreBatch { records: Vec<PodcastStoreEntry> }
PodcastStoreEntry { keys: Vec<PodcastDhtKey>, record: SignedPodcastRecord }
PodcastDhtResponse { version: u16, ok: bool, error: Option<String>,
nodes: Vec<NodeContact>, records: Vec<SignedPodcastRecord>, stored: u32 }
NodeContact { node_id: [u8; 32], peer_id: EndpointId, ticket: String, last_seen_ms: u64 }
PodcastRecordBody = 0: Card(PodcastCard) | 1: Tombstone(PodcastTombstone)
```
Fields are in postcard order. `PodcastDhtKey` is 32 raw bytes.
* `version` must be 1 in both directions. `requester.peer_id` must equal the
authenticated transport peer, `node_id` must be `NodeId::from_endpoint`
of it and its ticket must name the same endpoint and network; otherwise the
request is refused. The responder learns the requester as an overlay
contact (it is authenticated).
* `nodes` in a response are unverified suggestions. A requester treats them
as lookup or bootstrap candidates only and adds one to its routing table
(and persisted contacts) only after it answered a request itself.
* `FindNode` returns the `K` closest overlay contacts (the responder
included).
* `FindValue` returns the records under `key` (see Replica storage) and the
`K` closest contacts. `filter_tokens` (≤ 8, each 1..64 bytes) are
normalized tokens every returned card must be indexed under; tombstones are
returned regardless. Requesters verify every record again and drop records
that do not belong under `key` or miss a filter token.
* `StoreBatch` carries 1..16 entries, each with 1..33 distinct keys. Every
record must be published by the requester itself (replicas never forward
records), pass `verify_at` (bounds, network, lifetime, clock skew,
signature, size) and belong under each listed key; otherwise the whole
batch is refused. Keys the responder is not responsible for are skipped.
`stored` counts the `(key, record)` placements written. A publisher
offers the placements of a batch answered with `stored = 0` (or that
failed) once to the next closest node of each key.
* A refused request gets `ok = false` with an `error` of at most 1 KiB and
empty `nodes`, `records` and `stored = 0`; an accepted one has `ok = true`
and no `error`. Requesters reject responses that violate this or exceed
`K` nodes or 128 records.
### Replica storage
* A replica stores one payload per `(podcast_id, publisher)` (the winning
record) and indexes it under the keys it holds. A newer version replaces
it in place; placements the new version no longer belongs under (an old
title's tokens, keys a tombstone does not list) are dropped, so on one
replica a tombstone supersedes the card under every key it held.
* A node accepts a placement only when fewer than `2 * K` of its verified
overlay contacts that are not backed off are closer to the key than
itself. Contacts only enter the routing table by answering a request or
by sending one, so a peer cannot make a replica refuse keys by returning
made-up contacts. A contact that fails three times in a row (each after
its 30 s backoff) is removed.
* A key holds at most 128 records, at most 32 of them tombstones: exactly
what one `FindValue` response carries, so every admitted record is served.
A tombstone is indexed under a text key only in place of its owner's
record there, and such an in-place conversion counts against the
tombstone budget like a new tombstone.
* Full key: a new placement evicts a random placement of another owner from
the younger half of the key (by first indexing time; only tombstones when
the tombstone budget is full). The older half is never evicted for a
newcomer, so records placed before a flood keep their place and their
publishers refresh them in place. Capacity is never first-come-forever:
a newcomer always gets a place, though placements in the younger half can
be evicted again by later newcomers.
* A replica holds at most 256 records per publisher (a publisher over it is
refused), 4 096 records and 32 MiB of payload. A full cache admits a new
record by evicting random records of other owners from its younger half,
at most 16; when that is not enough room, nothing is evicted and the record
is refused. At most 1 024 overlay contacts are persisted.
* Device identities cost nothing, so these rules bound flooding but cannot
prevent it: an attacker with many identities can keep part of the younger
half of a key or cache busy, and a newcomer's place there can be evicted
again until its publisher's next republish (every 10 min) puts it back.
Protocol 1 has no proof of work or other identity cost.
* `FindValue` answers with at most 128 records: at most 32 tombstones and
the cards indexed longest first. Every record is verified again on every
read.
### Publication
* `PodcastDht::publish(subscription, share, card)` refuses
(`MusicDhtError::PodcastPublication`) unless `effective_share` holds, so a
new or resumed subscription is never published without a fresh share
decision. The UI calls it only after asking the user about any privacy
warning, so the call is the user's consent.
* `PodcastDht::sync_shares(states)` is declarative and idempotent: podcasts
for which `effective_share` holds are published (when changed, new or past
half their lifetime), every other podcast this device published is
unpublished. Clients call it after applying share or subscription
operations, so unsharing or unsubscribing on any trusted device revokes
every device's own card once the registers converge, and a stale device
cannot republish (its registers lose to the newer share decision).
* Revocation is fail-safe: each podcast is handled on its own. A shared
podcast whose card cannot be built or signed is reported in
`PodcastDhtSyncStats::failed` and treated as not shared (an earlier
publication is tombstoned); it never prevents other podcasts from being
unpublished.
* A card whose `PodcastCardSpec::privacy_warnings` is not empty is held back
(`PodcastDhtSyncStats::held_back`, with the warnings) unless
`PodcastShareState::privacy_confirmed` is set or this device already
publishes the same `feed_url` and `id_feed_url`. A feed move to a URL that
looks private therefore asks again; an earlier publication of a held-back
podcast stays as it was.
* SQLite clients build each `PodcastShareState` from
`PodcastStore::subscription` (the merged register with its alias set, which
always contains the origin URL) and `PodcastStore::share`; its `is_shared`
equals `PodcastSummary::shared`. A card for the URL the device now fetches
sets `feed_url` from the catalog and `id_feed_url` to the origin URL. The
store does not persist `privacy_confirmed`; the client keeps that
confirmation. See [SQLite reference store](#sqlite-reference-store).
* `PodcastDht::unpublish(podcast_id)` writes the tombstone. Nothing is ever
sent for a podcast this device never published: no tombstone, no id.
* `PodcastDht::republish` (run by `maintenance` every 10 minutes and after
a bootstrap that found new peers) re-signs active cards and re-sends
unexpired tombstones.
* Own cards persisted by an earlier run are not sent again until the client
calls `sync_shares` in this process: before that, `republish` sends only
tombstones and cards published with `publish` in this process, and
`FindValue` answers leave those cards out (they stay in the cache). A share
revoked on another trusted device while this one was offline is therefore
never re-sent on startup from the stale cache. Clients call `sync_shares`
on start, with the registers they hold, and again after device sync
applies newer operations; the first call also re-sends the unchanged cards
as they are (`PodcastDhtSyncStats::unchanged`, counted in `push`).
`sync_shares` itself still trusts the registers it is given: an unchanged
card is only re-signed when it used half its lifetime, so clients that may
hold stale registers on start lose nothing by reconciling again after
device sync.
* The privacy heuristics (`feed_url_privacy_warning`) are advisory: query
parameter names containing words such as `token`, `key`, `auth`, `secret`,
`sig`/`signature`, `uid`, `user`, `pass`, `session`, `access`, `private`;
known member-feed hosts; and long high-entropy query values (≥ 16
characters) or path segments (≥ 32 characters). They check `feed_url` and
`id_feed_url`, not artwork or web-page links. False positives and misses
are possible.
### Search
* `PodcastDht::search(query, limit)` normalizes the query
(`podcast_query_tokens`: distinct tokens, at most 8), looks up the text
keys of the three longest tokens and asks every replica to return only
cards indexed under **all** query tokens (tombstones are always returned).
Results are verified again, the winning record per publisher is kept,
tombstoned publishers are dropped and cards are merged by `podcast_id`
into `PodcastSearchHit { podcast_id, feed_url, card, publishers,
claimed_feed_urls }`.
* `PodcastDht::lookup_podcast(podcast_id)` reads the id key.
* `feed_url` is the origin URL; it is the only URL a client subscribes with.
`claimed_feed_urls` lists other `feed_url`s that publishers claim (at most
8, with their publishers), for display only.
* The representative `card` is a card whose `feed_url` is the origin URL
when one exists, then the title most publishers agree on, then the
smallest publisher key. Publisher-controlled times and revisions never
break ties. Its metadata remains a publisher's claim.
* `publishers` lists distinct publisher **devices**. It is not a subscriber
count (one user may publish from several devices, and subscribers who did
not share are invisible) and not a trust signal: device identities are
free. No subscriber count is offered and ranking ignores it.
* Ranking: exact normalized title first, then hits that have a card under
the origin URL, then title and id.
* `FederatedSearchHit` (`{"type": "music" | "podcast", "hit": ...}`),
`SearchHitKind` (`artist`, `release`, `track`, `podcast`) and
`merge_search_results` merge podcast hits with `SearchOutcome` music
results; `SearchOutcome` itself is unchanged.
* Lookups query the 8 closest overlay nodes (α = 3, at most 32 requests,
15 s deadline) and return partial results when peers fail. They keep at
most 512 `(podcast_id, publisher)` owners; each response (and the local
cache) may add at most 64 new owners (`512 / K`), so the responders that
answer first cannot fill a lookup and crowd out the rest. A newer record
of a known owner always replaces its winner. A peer that fails a request
is skipped for 30 s.
Subscribing from a hit creates the user's own private subscription with the
hit's `feed_url` (the origin URL): resolve the id with `resolve_podcast_id`,
which yields the hit's `podcast_id` unless the user already knows that URL,
fetch the feed from that URL and follow its own HTTP redirects and
`itunes:new-feed-url`, recording those as aliases as usual. A URL from
`claimed_feed_urls` becomes an alias only when that fetch leads there. The new
subscription is private (see `effective_share`), the publisher's availability
is not needed afterwards, and the episode catalog comes from RSS.
### Limits
| Limit | Value |
| --- | --- |
| Request / response | 512 KiB / 4 MiB postcard |
| Store batch | 1..16 entries, each 1..33 distinct keys, records of the requester only |
| Records per `FindValue` response | 128, at most 32 of them tombstones |
| Filter tokens | ≤ 8, each 1..64 bytes |
| Records per key | 128, at most 32 of them tombstones; random eviction from the younger half |
| Records per replica | 256 per publisher (refused beyond), 4 096 in all, 32 MiB of payload (random eviction from the younger half, at most 16 per record) |
| Store responsibility | fewer than `2 * K` = 16 verified, not backed-off contacts closer to the key |
| Records kept per lookup | 512 `(podcast_id, publisher)`, at most 64 new ones per response |
| Persisted overlay contacts | 1 024 |
| Inbound requests | 64 concurrent, 8 per peer (more are dropped); 10 s each |
| Error message | ≤ 1 KiB |
| Contact ticket | ≤ 16 KiB |
### Privacy guarantees
* Only podcasts with `effective_share` are published; never-shared podcasts
produce no network traffic at all.
* After a restart nothing is re-sent from the persisted cards until the
client reconciled its registers with `sync_shares` (see Publication).
* Cards carry the public projection only (see Card); tombstones carry key
hashes only.
* Records are signed by the publisher device and verified on every receive
and on every cache read; a forwarder cannot alter or impersonate them,
only the publisher can store them, and records from another network are
rejected.
* The card's `publisher` is the device's persistent iroh endpoint key, which
is also its overlay and music-DHT identity. It is **linkable**: anyone can
look the key up in the overlay (`FindNode` for
`NodeId::from_endpoint(publisher)`) and obtain the device's contact ticket
with its relay and direct IP addresses, because every overlay request
carries the requester's ticket. The same key owns the device's music-DHT
records, so a device's shared podcasts can be linked to its public music
library and its network address. Subscribing from a card never contacts
the publisher.
* Devices of one user are not linked by the overlay, but publishing the same
podcasts from several devices makes them correlatable.
* Unpublishing is eventually consistent: replicas that missed the tombstone
keep serving the old card until it expires (at most 12 h), including
replicas that only held an older title's text keys.
* Capability manifests are informational. A peer without `podcast_dht` never
enters the overlay routing table (it cannot answer on the ALPN), and
`PodcastDht::note_peer_capabilities` evicts a peer whose manifest lacks it
and stops dialing it. Applications advertise the overlay only when they
run it (`with_podcast_dht`).
## Feed parser
With the `podcast-feed` feature, `music_dht::podcast::feed` parses RSS 2.0,
@@ -609,7 +976,9 @@ must keep the same invariants.
`set_shared`/`unsubscribe` the share register, with
`next_podcast_generation`. No-op actions draw no stamp and emit nothing.
`unsubscribe` emits `subscribed = false` and `shared = false` and keeps
progress, sessions and the catalog.
progress, sessions and the catalog. The store only records the decision;
publishing and revoking the public card is `PodcastDht::sync_shares` (see
[Public discovery](#public-discovery-podcast_dht)).
* Explicit progress actions use `explicit_generation(max(local, carried))`;
checkpoints keep the session generation and are refused (`Superseded`)
once the register holds a greater one, or after the session completed.
@@ -667,5 +1036,8 @@ must keep the same invariants.
* Feed URL canonicalization, the identity bounds (`MAX_URL_BYTES`,
`MAX_GUID_BYTES`) and the no-GUID rule are frozen with the `v1` domains:
receivers reject non-canonical ids and URLs while parsing.
* Public podcast cards will use a separate signed overlay with its own ALPN
and version; the music DHT schema and `federation-net` do not change.
* Public podcast cards use the separate signed overlay `podcast_dht` with
its own ALPN (`furumi-fd/podcast-dht/1`), wire/record version 1, signing
domains `furumi-podcast-card:v1` / `furumi-podcast-card-tombstone:v1`, key
domain `furumi-podcast-dht:key:v1`, all frozen, and cache schema 2; the music
DHT schema and `federation-net` do not change.
+4
View File
@@ -57,6 +57,10 @@ It lives in its own file, conventionally `podcasts.sqlite3`
- **Queries**: `list_podcasts`, `seasons`, `episodes` (season, played-state
and text filters, date or episode-number order), `resume_position_ms`,
`episode_ref` / `episode_track` for queueing, `listen_history`.
- **Public discovery**: the store records share decisions only. Build each
`music_dht::podcast_dht::PodcastShareState` for `PodcastDht::sync_shares`
from `subscription` and `share` (workspace `PODCASTS.md`, section "Public
discovery (podcast_dht)"); the client keeps the user's `privacy_confirmed` decision.
The store is synchronous: methods take `&self` and lock one connection, so
`Arc<PodcastStore>` can be shared; run calls on a blocking pool.
@@ -2242,3 +2242,82 @@ fn concurrent_fetch_state_writers_lose_no_update() {
assert_eq!(state.consecutive_failures as usize, THREADS * FAILURES);
assert_eq!(state.cover_path.as_deref(), Some("/covers/c.jpg"));
}
// ---------------------------------------------------------------------------
// Public discovery (`music_dht::podcast_dht`)
/// The share state `PodcastDht::sync_shares` takes, built from the store the
/// way clients do it.
fn discovery_state(
store: &PodcastStore,
podcast_id: &PodcastId,
) -> music_dht::podcast_dht::PodcastShareState {
music_dht::podcast_dht::PodcastShareState {
subscription: store.subscription(podcast_id).unwrap().unwrap(),
share: store.share(podcast_id).unwrap(),
card: None,
privacy_confirmed: false,
}
}
#[test]
fn store_registers_drive_discovery_publication() {
let store = PodcastStore::open_in_memory().unwrap();
let mut stamps = TestStamps::new("device-a", 1_000);
let podcast_id = store
.subscribe(&mut stamps, &meta(FEED, "Show"))
.unwrap()
.podcast_id;
// Subscribed but private: nothing to publish.
let state = discovery_state(&store, &podcast_id);
assert!(!state.is_shared());
assert!(!store.podcast(&podcast_id).unwrap().unwrap().shared);
store.set_shared(&mut stamps, &podcast_id, true).unwrap();
let state = discovery_state(&store, &podcast_id);
assert!(state.is_shared());
assert!(store.podcast(&podcast_id).unwrap().unwrap().shared);
let card = state.card_spec().unwrap();
assert_eq!(card.podcast_id, podcast_id);
assert_eq!(card.feed_url, url(FEED));
assert_eq!(card.id_feed_url, None);
assert_eq!(card.title, "Show");
assert!(card.privacy_warnings().is_empty());
// After a permanent move the register's alias set still holds the origin
// URL, so the card stays bound to the podcast id.
let items = [Item::new(
Some("g1"),
"One",
"https://cdn.example.test/1.mp3",
)];
let moved = url("https://new.example.test/feed.xml");
let feed = parse(&rss("Show", None, &items), &moved);
refresh(&store, &mut stamps, &podcast_id, &feed, &moved, true);
let state = discovery_state(&store, &podcast_id);
assert!(state.subscription.feed_urls().contains(&moved));
let card = state.card_spec().unwrap();
assert_eq!(card.podcast_id, podcast_id);
// A card for the URL this device now fetches names the origin URL.
let summary = store.podcast(&podcast_id).unwrap().unwrap();
assert_eq!(summary.meta.feed_url, moved);
let fetched = music_dht::podcast_dht::PodcastShareState {
card: Some(music_dht::podcast_dht::PodcastCardSpec {
feed_url: summary.meta.feed_url.clone(),
id_feed_url: Some(url(FEED)),
..card
}),
..state
};
let card = fetched.card_spec().unwrap();
assert_eq!(card.feed_url, moved);
assert_eq!(card.id_feed_url, Some(url(FEED)));
card.validate().unwrap();
// Unsubscribing revokes the share decision: the card must be unpublished.
store.unsubscribe(&mut stamps, &podcast_id).unwrap();
let state = discovery_state(&store, &podcast_id);
assert!(!state.is_shared());
assert!(!store.podcast(&podcast_id).unwrap().unwrap().shared);
}
+29
View File
@@ -250,6 +250,35 @@ reported itself (`profile_protocol_version`; relayed profiles go through
`jam::redact_jam_snapshot`, `jam_command_allowed` and
`restore_jam_private_items`.
### Public podcast discovery
`music_dht::podcast_dht::PodcastDht` is a separate signed overlay (ALPN
`furumi-fd/podcast-dht/1`, capability `podcast_dht`) for podcasts the user
explicitly shared. Wiring, as for `SimilarityDht`:
1. declare the ALPN with `schema_independent_stream_protocol(PODCAST_DHT_ALPN)`,
open the overlay with its own cache file, take
`stream_acceptor(PODCAST_DHT_ALPN)` and spawn `serve` and `maintenance`;
2. advertise it with `CapabilityManifest::with_podcast_dht()` (it is not in
`CapabilityManifest::frid()`; an application that does not run the overlay
simply does not add it);
3. pass every peer manifest received to `note_peer_capabilities`, so peers
without the overlay are not dialed;
4. call `sync_shares` with the merged subscription and share registers on
start and after applying sync operations (until the first call, own cards
persisted by an earlier run are neither republished nor served): it
publishes cards only where `effective_share` holds, tombstones every other
own publication, and reports cards it could not build (`failed`) or held
back for privacy (`held_back`);
5. ask the user before publishing when `PodcastCardSpec::privacy_warnings`
(or `feed_url_privacy_warning`) reports a URL that looks private, then
call `publish` or set `PodcastShareState::privacy_confirmed`.
`search` and `lookup_podcast` return `PodcastSearchHit`s; subscribe with the
hit's `feed_url` (the URL the id hashes from), never with a
`claimed_feed_urls` entry. `merge_search_results` combines hits with
`SearchOutcome::network_results` as typed `FederatedSearchHit`s.
## Verification
```bash
+50 -1
View File
@@ -35,6 +35,8 @@ pub const CATALOG_ID: &str = "catalog";
pub const SIMILARITY_ID: &str = "similarity";
/// Stable protocol identifier for DHT-routed similarity discovery.
pub const SIMILARITY_DHT_ID: &str = "similarity_dht";
/// Stable protocol identifier for the public podcast-discovery overlay.
pub const PODCAST_DHT_ID: &str = "podcast_dht";
/// Stable protocol identifier for personal-device synchronization.
pub const DEVICE_SYNC_ID: &str = "device_sync";
/// Stable protocol identifier for Jam playback control.
@@ -54,7 +56,13 @@ pub struct CapabilityManifest {
}
impl CapabilityManifest {
/// Creates a manifest containing every protocol owned by Frid.
/// Creates a manifest containing every protocol owned by Frid that a
/// music-DHT application serves.
///
/// The public podcast-discovery overlay is opt-in: applications that run
/// [`crate::podcast_dht::PodcastDht`] add it with
/// [`Self::with_podcast_dht`], so an application that does not serve the
/// overlay never advertises it.
pub fn frid(application: impl Into<String>, application_version: impl Into<String>) -> Self {
let mut protocols = BTreeMap::new();
protocols.insert(
@@ -105,6 +113,29 @@ impl CapabilityManifest {
self
}
/// Advertises the public podcast-discovery overlay
/// ([`PODCAST_DHT_ID`] at [`crate::podcast_dht::PODCAST_DHT_PROTOCOL_VERSION`]).
/// Call it only when the application serves
/// [`crate::podcast_dht::PODCAST_DHT_ALPN`].
pub fn with_podcast_dht(self) -> Self {
self.with_protocol(
PODCAST_DHT_ID,
crate::podcast_dht::PODCAST_DHT_PROTOCOL_VERSION,
)
}
/// Version of protocol `id` advertised by the peer, if any.
pub fn protocol_version(&self, id: &str) -> Option<u16> {
self.protocols.get(id).copied()
}
/// Whether the peer advertises the public podcast-discovery overlay at
/// the version this crate speaks. Peers without it are never queried.
pub fn supports_podcast_dht(&self) -> bool {
self.protocol_version(PODCAST_DHT_ID)
== Some(crate::podcast_dht::PODCAST_DHT_PROTOCOL_VERSION)
}
/// Validates bounds and identifiers received from a peer.
pub fn validate(&self) -> Result<()> {
if self.manifest_version != CAPABILITIES_PROTOCOL_VERSION {
@@ -227,6 +258,24 @@ mod tests {
manifest.validate().unwrap();
}
#[test]
fn podcast_dht_capability_is_opt_in_and_removable() {
let plain = CapabilityManifest::frid("furumi", "1.2.3");
assert_eq!(plain.protocol_version(PODCAST_DHT_ID), None);
assert!(!plain.supports_podcast_dht());
let manifest = plain.with_podcast_dht();
assert_eq!(
manifest.protocol_version(PODCAST_DHT_ID),
Some(crate::podcast_dht::PODCAST_DHT_PROTOCOL_VERSION)
);
assert!(manifest.supports_podcast_dht());
manifest.validate().unwrap();
let without = manifest.clone().without_protocol(PODCAST_DHT_ID);
assert!(!without.supports_podcast_dht());
let other_version = manifest.with_protocol(PODCAST_DHT_ID, 2);
assert!(!other_version.supports_podcast_dht());
}
#[test]
fn application_protocol_is_additive() {
let manifest = CapabilityManifest::frid("furumi", "1.2.3").with_protocol("audio", 1);
+5
View File
@@ -50,6 +50,11 @@ pub enum MusicDhtError {
/// The service is shutting down and no longer accepts operations.
#[error("service is shutting down")]
ShuttingDown,
/// A podcast card cannot be published, for example because the user did
/// not share the podcast (see `music_dht::podcast::effective_share`).
#[error("podcast publication refused: {0}")]
PodcastPublication(String),
}
impl From<federation_net::NetworkError> for MusicDhtError {
+4
View File
@@ -27,6 +27,9 @@
//! podcast/episode ids, metadata DTOs and convergent merge rules), with an
//! optional RSS/Atom parser behind the `podcast-feed` feature;
//! [`device_sync`] protocol v4 carries it between trusted devices.
//! * [`podcast_dht`] is a separate signed overlay for public podcast
//! discovery: cards a user explicitly shared, found by title/author tokens
//! or podcast id, retracted by signed tombstones.
//! * Records are replicated to the `K` nodes whose ids are XOR-closest to
//! each key. Publishers pick the targets from their routing table and send
//! batched store requests (one pipeline per peer), so even a large library
@@ -97,6 +100,7 @@ mod message;
mod node;
mod normalization;
pub mod podcast;
pub mod podcast_dht;
mod record;
mod request;
mod routing;
+899
View File
@@ -0,0 +1,899 @@
//! Signed public podcast cards, tombstones, DHT keys and their canonical
//! signing encoding.
//!
//! Everything here is pure and deterministic: no clock, no I/O. Callers pass
//! the current time to the validation functions.
use std::cmp::Ordering;
use std::collections::HashSet;
use std::time::Duration;
use federation_net::{EndpointId, NetworkId, Signature};
use serde::{Deserialize, Serialize};
use crate::error::{MusicDhtError, Result};
use crate::normalization::{normalize_name, tokenize};
use crate::podcast::{
FeedUrl, MAX_URL_BYTES, PodcastId, SubscriptionRecord, bounded_text, is_http_url,
};
use super::privacy::{FeedUrlPrivacyWarning, feed_url_privacy_warning};
/// Current card, tombstone and overlay wire version.
pub const PODCAST_DHT_PROTOCOL_VERSION: u16 = 1;
/// Signing domain of a [`PodcastCard`]. Changing it invalidates every card.
pub const PODCAST_CARD_SIGNING_DOMAIN: &str = "furumi-podcast-card:v1";
/// Signing domain of a [`PodcastTombstone`].
pub const PODCAST_TOMBSTONE_SIGNING_DOMAIN: &str = "furumi-podcast-card-tombstone:v1";
/// BLAKE3 derivation domain of [`PodcastDhtKey`].
pub const PODCAST_DHT_KEY_DOMAIN: &str = "furumi-podcast-dht:key:v1";
/// Maximum lifetime of a card: the main DHT's active-record TTL.
pub const PODCAST_CARD_TTL: Duration = crate::record::ACTIVE_RECORD_TTL;
/// Maximum lifetime of a tombstone: the main DHT's tombstone TTL. It exceeds
/// [`PODCAST_CARD_TTL`], so a card issued before a tombstone always expires
/// before the tombstone does.
pub const PODCAST_TOMBSTONE_TTL: Duration = crate::record::TOMBSTONE_TTL;
/// Tolerated clock skew for `issued_at_ms` in the future.
pub const PODCAST_DHT_MAX_CLOCK_SKEW: Duration = Duration::from_secs(5 * 60);
/// Maximum characters of a card title.
pub const MAX_CARD_TITLE_CHARS: usize = 300;
/// Maximum characters of a card author.
pub const MAX_CARD_AUTHOR_CHARS: usize = 300;
/// Maximum characters of a card description.
pub const MAX_CARD_DESCRIPTION_CHARS: usize = 1_000;
/// Maximum bytes of a card language tag (BCP 47 shape: ASCII letters,
/// digits and `-`).
pub const MAX_CARD_LANGUAGE_BYTES: usize = 35;
/// Maximum categories on a card.
pub const MAX_CARD_CATEGORIES: usize = 8;
/// Maximum characters of one category.
pub const MAX_CARD_CATEGORY_CHARS: usize = 64;
/// Maximum indexed search tokens of a card (title tokens, then author tokens).
pub const MAX_CARD_INDEX_TOKENS: usize = 32;
/// Maximum bytes of one indexed or query token. Longer tokens are not indexed.
pub const MAX_INDEX_TOKEN_BYTES: usize = 64;
/// Maximum DHT keys a record is stored under: the podcast-id key plus one
/// key per indexed token.
pub const MAX_RECORD_DHT_KEYS: usize = MAX_CARD_INDEX_TOKENS + 1;
/// Maximum postcard-encoded size of one signed record.
pub const MAX_PODCAST_RECORD_BYTES: usize = 24 * 1024;
const KEY_KIND_PODCAST: u8 = 1;
const KEY_KIND_TOKEN: u8 = 2;
/// A 256-bit point in the podcast-discovery key space.
///
/// `BLAKE3("furumi-podcast-dht:key:v1" || 0x00 || network_id || kind || material)`
/// with kind `0x01` and the podcast id string for the id key, or kind `0x02`
/// and a normalized token for a text key.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord, Serialize, Deserialize)]
pub struct PodcastDhtKey([u8; 32]);
impl PodcastDhtKey {
/// Key under which every card and tombstone of `podcast_id` is stored.
pub fn podcast(network_id: &NetworkId, podcast_id: &PodcastId) -> Self {
Self::derive(network_id, KEY_KIND_PODCAST, podcast_id.as_str())
}
/// Key of one normalized search token (see [`card_index_tokens`]).
pub fn token(network_id: &NetworkId, token: &str) -> Self {
Self::derive(network_id, KEY_KIND_TOKEN, token)
}
fn derive(network_id: &NetworkId, kind: u8, material: &str) -> Self {
let mut hasher = blake3::Hasher::new();
hasher.update(PODCAST_DHT_KEY_DOMAIN.as_bytes());
hasher.update(&[0]);
hasher.update(network_id.as_bytes());
hasher.update(&[kind]);
hasher.update(material.as_bytes());
Self(*hasher.finalize().as_bytes())
}
/// Creates a key from raw bytes received on the wire.
pub fn from_bytes(bytes: [u8; 32]) -> Self {
Self(bytes)
}
/// Raw key bytes.
pub fn as_bytes(&self) -> &[u8; 32] {
&self.0
}
}
/// Indexed search tokens of a card: distinct [`normalize_name`] tokens of the
/// title, then of the author, in order, skipping tokens longer than
/// [`MAX_INDEX_TOKEN_BYTES`], at most [`MAX_CARD_INDEX_TOKENS`].
pub fn card_index_tokens(title: &str, author: Option<&str>) -> Vec<String> {
let mut seen = HashSet::new();
let mut tokens = Vec::new();
let sources = std::iter::once(title).chain(author);
for source in sources {
for token in tokenize(&normalize_name(source)) {
if tokens.len() == MAX_CARD_INDEX_TOKENS {
return tokens;
}
if token.len() <= MAX_INDEX_TOKEN_BYTES && seen.insert(token.clone()) {
tokens.push(token);
}
}
}
tokens
}
/// Unsigned public projection of a podcast prepared by its publisher.
///
/// It is built from the user's subscription ([`PodcastCardSpec::from_subscription`])
/// and contains only publisher metadata: never progress, listening history,
/// subscription times, operation ids, device names or episodes.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct PodcastCardSpec {
/// Podcast.
pub podcast_id: PodcastId,
/// Canonical URL the feed is currently fetched from.
pub feed_url: FeedUrl,
/// URL `podcast_id` was derived from, when it differs from `feed_url`.
pub id_feed_url: Option<FeedUrl>,
/// Podcast title.
pub title: String,
/// Author.
pub author: Option<String>,
/// Plain-text description.
pub description: Option<String>,
/// Artwork URL.
pub image_url: Option<String>,
/// Publisher web page.
pub link: Option<String>,
/// Feed language tag, for example `en` or `pt-BR`.
pub language: Option<String>,
/// Feed categories.
pub categories: Vec<String>,
/// Number of episodes in the feed, when known.
pub episode_count: Option<u32>,
/// Publication time of the newest episode in unix milliseconds.
pub latest_episode_at_ms: Option<i64>,
}
impl PodcastCardSpec {
/// Builds the bounded public projection of a subscription.
///
/// Only `podcast_id`, the feed URLs and the channel metadata are read.
/// `id_feed_url` is the alias that hashes to `podcast_id`
/// ([`crate::podcast::merge_feed_aliases`] always keeps it). Fails when no
/// such URL is known. Callers may then set `language`, `categories`,
/// `episode_count` and `latest_episode_at_ms` from their feed cache.
pub fn from_subscription(subscription: &SubscriptionRecord) -> Result<Self> {
let origin = subscription
.feed_urls()
.into_iter()
.find(|url| PodcastId::from_feed_url(url) == subscription.podcast_id)
.ok_or_else(|| {
publication_error("the feed URL the podcast id was derived from is unknown")
})?;
let feed_url = subscription.podcast.feed_url.clone();
let meta = &subscription.podcast;
Ok(Self {
podcast_id: subscription.podcast_id.clone(),
id_feed_url: (origin != feed_url).then_some(origin),
feed_url,
title: meta.title.clone(),
author: meta.author.clone(),
description: meta.description.clone(),
image_url: meta.image_url.clone(),
link: meta.link.clone(),
language: None,
categories: Vec::new(),
episode_count: None,
latest_episode_at_ms: None,
}
.bounded())
}
/// Returns a copy that fits the card bounds: text is truncated with
/// [`bounded_text`], control characters are removed, invalid URLs and
/// language tags are dropped, and categories are trimmed, deduplicated and
/// bounded. An empty title becomes `"Untitled podcast"`.
pub fn bounded(&self) -> Self {
let title = clean_line(&self.title);
let mut seen = HashSet::new();
let categories = self
.categories
.iter()
.map(|category| bounded_text(&clean_line(category), MAX_CARD_CATEGORY_CHARS))
.filter(|category| !category.is_empty() && seen.insert(category.to_lowercase()))
.take(MAX_CARD_CATEGORIES)
.collect();
Self {
podcast_id: self.podcast_id.clone(),
feed_url: self.feed_url.clone(),
id_feed_url: self
.id_feed_url
.clone()
.filter(|origin| *origin != self.feed_url),
title: if title.is_empty() {
"Untitled podcast".to_string()
} else {
bounded_text(&title, MAX_CARD_TITLE_CHARS)
},
author: bounded_line(self.author.as_deref(), MAX_CARD_AUTHOR_CHARS),
description: self
.description
.as_deref()
.map(|text| bounded_text(clean_block(text).trim(), MAX_CARD_DESCRIPTION_CHARS))
.filter(|text| !text.is_empty()),
image_url: self.image_url.clone().filter(|url| is_http_url(url)),
link: self.link.clone().filter(|url| is_http_url(url)),
language: self
.language
.as_deref()
.map(str::trim)
.filter(|language| valid_language(language))
.map(str::to_string),
categories,
episode_count: self.episode_count,
latest_episode_at_ms: self.latest_episode_at_ms.filter(|at| *at >= 0),
}
}
/// Validates the card bounds without bounding anything.
pub fn validate(&self) -> Result<()> {
let id_url = self.id_feed_url.as_ref().unwrap_or(&self.feed_url);
if PodcastId::from_feed_url(id_url) != self.podcast_id {
return Err(invalid("podcast id does not match its feed URL"));
}
if self.id_feed_url.as_ref() == Some(&self.feed_url) {
return Err(invalid("id_feed_url must differ from feed_url"));
}
check_line("title", &self.title, MAX_CARD_TITLE_CHARS, true)?;
if let Some(author) = &self.author {
check_line("author", author, MAX_CARD_AUTHOR_CHARS, false)?;
}
if let Some(description) = &self.description {
check_chars("description", description, MAX_CARD_DESCRIPTION_CHARS)?;
if description
.chars()
.any(|c| c.is_control() && c != '\n' && c != '\t')
{
return Err(invalid("description contains control characters"));
}
}
for url in [&self.image_url, &self.link].into_iter().flatten() {
if !is_http_url(url) {
return Err(invalid("card URLs must be absolute http(s) URLs"));
}
}
if self
.language
.as_deref()
.is_some_and(|language| !valid_language(language))
{
return Err(invalid("invalid language tag"));
}
if self.categories.len() > MAX_CARD_CATEGORIES {
return Err(invalid("too many categories"));
}
for category in &self.categories {
check_line("category", category, MAX_CARD_CATEGORY_CHARS, true)?;
}
if self.latest_episode_at_ms.is_some_and(|at| at < 0) {
return Err(invalid("negative latest_episode_at_ms"));
}
Ok(())
}
/// Advisory privacy warnings for both feed URLs of the card (see
/// [`feed_url_privacy_warning`]). The UI must ask before publishing a
/// card with warnings.
pub fn privacy_warnings(&self) -> Vec<FeedUrlPrivacyWarning> {
std::iter::once(&self.feed_url)
.chain(self.id_feed_url.as_ref())
.filter_map(feed_url_privacy_warning)
.collect()
}
/// Indexed search tokens of this card.
pub fn index_tokens(&self) -> Vec<String> {
card_index_tokens(&self.title, self.author.as_deref())
}
}
/// A signed public podcast card: one publisher device's projection of one
/// podcast. See `PODCASTS.md`, "Public discovery (podcast_dht)".
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct PodcastCard {
/// Record format version ([`PODCAST_DHT_PROTOCOL_VERSION`]).
pub version: u16,
/// Network scope; prevents replay into another federation.
pub network_id: NetworkId,
/// Publisher device (its persistent transport identity); signs the card.
pub publisher: EndpointId,
/// Podcast.
pub podcast_id: PodcastId,
/// Canonical URL the feed is currently fetched from.
pub feed_url: FeedUrl,
/// URL `podcast_id` was derived from, when it differs from `feed_url`.
pub id_feed_url: Option<FeedUrl>,
/// Podcast title.
pub title: String,
/// Author.
pub author: Option<String>,
/// Plain-text description.
pub description: Option<String>,
/// Artwork URL.
pub image_url: Option<String>,
/// Publisher web page.
pub link: Option<String>,
/// Feed language tag.
pub language: Option<String>,
/// Feed categories.
pub categories: Vec<String>,
/// Number of episodes in the feed, when known.
pub episode_count: Option<u32>,
/// Publication time of the newest episode in unix milliseconds.
pub latest_episode_at_ms: Option<i64>,
/// Monotonic per `(podcast_id, publisher)` revision.
pub revision: u64,
/// Unix time at which this version was signed.
pub issued_at_ms: u64,
/// Unix time after which the card must be discarded.
pub expires_at_ms: u64,
}
impl PodcastCard {
/// Builds a card from a spec. `expires_at_ms` is
/// `issued_at_ms + PODCAST_CARD_TTL`.
pub fn from_spec(
spec: PodcastCardSpec,
network_id: NetworkId,
publisher: EndpointId,
revision: u64,
issued_at_ms: u64,
) -> Self {
Self {
version: PODCAST_DHT_PROTOCOL_VERSION,
network_id,
publisher,
podcast_id: spec.podcast_id,
feed_url: spec.feed_url,
id_feed_url: spec.id_feed_url,
title: spec.title,
author: spec.author,
description: spec.description,
image_url: spec.image_url,
link: spec.link,
language: spec.language,
categories: spec.categories,
episode_count: spec.episode_count,
latest_episode_at_ms: spec.latest_episode_at_ms,
revision,
issued_at_ms,
expires_at_ms: issued_at_ms.saturating_add(PODCAST_CARD_TTL.as_millis() as u64),
}
}
/// The publisher-provided part of the card.
pub fn spec(&self) -> PodcastCardSpec {
PodcastCardSpec {
podcast_id: self.podcast_id.clone(),
feed_url: self.feed_url.clone(),
id_feed_url: self.id_feed_url.clone(),
title: self.title.clone(),
author: self.author.clone(),
description: self.description.clone(),
image_url: self.image_url.clone(),
link: self.link.clone(),
language: self.language.clone(),
categories: self.categories.clone(),
episode_count: self.episode_count,
latest_episode_at_ms: self.latest_episode_at_ms,
}
}
/// Indexed search tokens.
pub fn index_tokens(&self) -> Vec<String> {
card_index_tokens(&self.title, self.author.as_deref())
}
/// The URL `podcast_id` is derived from (`id_feed_url`, or `feed_url`
/// when the feed has not moved). A valid card's id hashes from it, so it
/// is the only URL of the card bound to the podcast.
pub fn origin_feed_url(&self) -> &FeedUrl {
self.id_feed_url.as_ref().unwrap_or(&self.feed_url)
}
/// Whether `feed_url` is the URL the id is derived from. Otherwise
/// `feed_url` is an unverified claim of the publisher that the feed moved.
pub fn feed_url_is_origin(&self) -> bool {
self.id_feed_url.is_none()
}
/// Canonical signing input (see `PODCASTS.md`).
pub fn signing_bytes(&self) -> Vec<u8> {
let mut out = Canonical::new(PODCAST_CARD_SIGNING_DOMAIN);
out.u16(self.version);
out.raw(self.network_id.as_bytes());
out.raw(self.publisher.as_bytes());
out.str(self.podcast_id.as_str());
out.str(self.feed_url.as_str());
out.opt_str(self.id_feed_url.as_ref().map(FeedUrl::as_str));
out.str(&self.title);
out.opt_str(self.author.as_deref());
out.opt_str(self.description.as_deref());
out.opt_str(self.image_url.as_deref());
out.opt_str(self.link.as_deref());
out.opt_str(self.language.as_deref());
out.u32(self.categories.len() as u32);
for category in &self.categories {
out.str(category);
}
match self.episode_count {
Some(count) => {
out.u8(1);
out.u32(count);
}
None => out.u8(0),
}
match self.latest_episode_at_ms {
Some(at) => {
out.u8(1);
out.i64(at);
}
None => out.u8(0),
}
out.u64(self.revision);
out.u64(self.issued_at_ms);
out.u64(self.expires_at_ms);
out.finish()
}
fn validate(&self, expected_network: &NetworkId) -> Result<()> {
check_header(self.version, &self.network_id, expected_network)?;
self.spec().validate()?;
check_lifetime(
self.revision,
self.issued_at_ms,
self.expires_at_ms,
PODCAST_CARD_TTL,
)
}
}
/// A signed retraction of one publisher's card of one podcast.
///
/// It supersedes every card of the same `(podcast_id, publisher)` with a
/// revision lower than or equal to its own and is retained until it expires.
/// `index_keys` lists the text keys of the retracted cards so the tombstone
/// can be stored next to them; it carries hashes, not the tokens.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct PodcastTombstone {
/// Record format version ([`PODCAST_DHT_PROTOCOL_VERSION`]).
pub version: u16,
/// Network scope.
pub network_id: NetworkId,
/// Publisher device; signs the tombstone.
pub publisher: EndpointId,
/// Podcast.
pub podcast_id: PodcastId,
/// Revision; greater than that of every retracted card.
pub revision: u64,
/// Unix time at which the tombstone was signed.
pub issued_at_ms: u64,
/// Unix time after which the tombstone may be discarded.
pub expires_at_ms: u64,
/// Sorted, distinct text keys besides the podcast-id key, at most
/// [`MAX_CARD_INDEX_TOKENS`].
pub index_keys: Vec<PodcastDhtKey>,
}
impl PodcastTombstone {
/// Builds a tombstone. `expires_at_ms` is `issued_at_ms + PODCAST_TOMBSTONE_TTL`;
/// `index_keys` is sorted, deduplicated and stripped of the podcast-id key.
pub fn new(
network_id: NetworkId,
publisher: EndpointId,
podcast_id: PodcastId,
revision: u64,
issued_at_ms: u64,
index_keys: impl IntoIterator<Item = PodcastDhtKey>,
) -> Self {
let id_key = PodcastDhtKey::podcast(&network_id, &podcast_id);
let mut index_keys: Vec<_> = index_keys
.into_iter()
.filter(|key| *key != id_key)
.collect();
index_keys.sort();
index_keys.dedup();
index_keys.truncate(MAX_CARD_INDEX_TOKENS);
Self {
version: PODCAST_DHT_PROTOCOL_VERSION,
network_id,
publisher,
podcast_id,
revision,
issued_at_ms,
expires_at_ms: issued_at_ms.saturating_add(PODCAST_TOMBSTONE_TTL.as_millis() as u64),
index_keys,
}
}
/// Canonical signing input (see `PODCASTS.md`).
pub fn signing_bytes(&self) -> Vec<u8> {
let mut out = Canonical::new(PODCAST_TOMBSTONE_SIGNING_DOMAIN);
out.u16(self.version);
out.raw(self.network_id.as_bytes());
out.raw(self.publisher.as_bytes());
out.str(self.podcast_id.as_str());
out.u64(self.revision);
out.u64(self.issued_at_ms);
out.u64(self.expires_at_ms);
out.u32(self.index_keys.len() as u32);
for key in &self.index_keys {
out.raw(key.as_bytes());
}
out.finish()
}
fn validate(&self, expected_network: &NetworkId) -> Result<()> {
check_header(self.version, &self.network_id, expected_network)?;
if self.index_keys.len() > MAX_CARD_INDEX_TOKENS {
return Err(invalid("tombstone has too many keys"));
}
let id_key = PodcastDhtKey::podcast(&self.network_id, &self.podcast_id);
if self.index_keys.contains(&id_key)
|| self.index_keys.windows(2).any(|pair| pair[0] >= pair[1])
{
return Err(invalid("tombstone keys are not canonical"));
}
check_lifetime(
self.revision,
self.issued_at_ms,
self.expires_at_ms,
PODCAST_TOMBSTONE_TTL,
)
}
}
/// Body of a signed podcast-discovery record.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub enum PodcastRecordBody {
/// An active public card.
Card(Box<PodcastCard>),
/// A retraction.
Tombstone(PodcastTombstone),
}
/// End-to-end signed podcast-discovery record.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct SignedPodcastRecord {
/// Signed body.
pub body: PodcastRecordBody,
/// Ed25519 signature by `publisher` over [`Self::signing_bytes`].
pub signature: Signature,
}
impl SignedPodcastRecord {
/// Validates and signs a card.
pub fn sign_card(card: PodcastCard, signer: impl FnOnce(&[u8]) -> Signature) -> Result<Self> {
card.validate(&card.network_id)?;
let signature = signer(&card.signing_bytes());
let record = Self {
body: PodcastRecordBody::Card(Box::new(card)),
signature,
};
record.check_size()?;
Ok(record)
}
/// Validates and signs a tombstone.
pub fn sign_tombstone(
tombstone: PodcastTombstone,
signer: impl FnOnce(&[u8]) -> Signature,
) -> Result<Self> {
tombstone.validate(&tombstone.network_id)?;
let signature = signer(&tombstone.signing_bytes());
let record = Self {
body: PodcastRecordBody::Tombstone(tombstone),
signature,
};
record.check_size()?;
Ok(record)
}
/// Canonical signing input of the body.
pub fn signing_bytes(&self) -> Vec<u8> {
match &self.body {
PodcastRecordBody::Card(card) => card.signing_bytes(),
PodcastRecordBody::Tombstone(tombstone) => tombstone.signing_bytes(),
}
}
/// Validates bounds, network, lifetime shape, encoded size and the
/// publisher's signature. Does not look at the clock; see
/// [`Self::verify_at`].
pub fn verify(&self, expected_network: &NetworkId) -> Result<()> {
match &self.body {
PodcastRecordBody::Card(card) => card.validate(expected_network)?,
PodcastRecordBody::Tombstone(tombstone) => tombstone.validate(expected_network)?,
}
self.check_size()?;
self.publisher()
.verify(&self.signing_bytes(), &self.signature)
.map_err(|_| invalid("invalid podcast record signature"))
}
/// [`Self::verify`] plus time checks: not issued more than
/// [`PODCAST_DHT_MAX_CLOCK_SKEW`] in the future and not expired at `now_ms`.
pub fn verify_at(&self, expected_network: &NetworkId, now_ms: u64) -> Result<()> {
self.verify(expected_network)?;
let future_limit = now_ms.saturating_add(PODCAST_DHT_MAX_CLOCK_SKEW.as_millis() as u64);
if self.issued_at_ms() > future_limit || self.expires_at_ms() <= now_ms {
return Err(invalid("podcast record is expired or from the future"));
}
Ok(())
}
/// The card, when this is not a tombstone.
pub fn card(&self) -> Option<&PodcastCard> {
match &self.body {
PodcastRecordBody::Card(card) => Some(card.as_ref()),
PodcastRecordBody::Tombstone(_) => None,
}
}
/// Whether this is a tombstone.
pub fn is_tombstone(&self) -> bool {
matches!(self.body, PodcastRecordBody::Tombstone(_))
}
/// Podcast.
pub fn podcast_id(&self) -> &PodcastId {
match &self.body {
PodcastRecordBody::Card(card) => &card.podcast_id,
PodcastRecordBody::Tombstone(tombstone) => &tombstone.podcast_id,
}
}
/// Publisher device.
pub fn publisher(&self) -> EndpointId {
match &self.body {
PodcastRecordBody::Card(card) => card.publisher,
PodcastRecordBody::Tombstone(tombstone) => tombstone.publisher,
}
}
/// Network scope.
pub fn network_id(&self) -> NetworkId {
match &self.body {
PodcastRecordBody::Card(card) => card.network_id,
PodcastRecordBody::Tombstone(tombstone) => tombstone.network_id,
}
}
/// Revision.
pub fn revision(&self) -> u64 {
match &self.body {
PodcastRecordBody::Card(card) => card.revision,
PodcastRecordBody::Tombstone(tombstone) => tombstone.revision,
}
}
/// Signing time.
pub fn issued_at_ms(&self) -> u64 {
match &self.body {
PodcastRecordBody::Card(card) => card.issued_at_ms,
PodcastRecordBody::Tombstone(tombstone) => tombstone.issued_at_ms,
}
}
/// Expiry time.
pub fn expires_at_ms(&self) -> u64 {
match &self.body {
PodcastRecordBody::Card(card) => card.expires_at_ms,
PodcastRecordBody::Tombstone(tombstone) => tombstone.expires_at_ms,
}
}
/// Every DHT key this record belongs under: the podcast-id key, then the
/// text keys (card tokens or the tombstone's `index_keys`).
pub fn dht_keys(&self) -> Vec<PodcastDhtKey> {
let network_id = self.network_id();
let mut keys = vec![PodcastDhtKey::podcast(&network_id, self.podcast_id())];
match &self.body {
PodcastRecordBody::Card(card) => keys.extend(
card.index_tokens()
.iter()
.map(|token| PodcastDhtKey::token(&network_id, token)),
),
PodcastRecordBody::Tombstone(tombstone) => {
keys.extend(tombstone.index_keys.iter().copied());
}
}
keys
}
/// Whether the record may be stored under `key`.
pub fn belongs_under(&self, key: &PodcastDhtKey) -> bool {
self.dht_keys().contains(key)
}
/// postcard-encoded size.
pub fn encoded_len(&self) -> usize {
postcard::to_stdvec(self).map_or(usize::MAX, |bytes| bytes.len())
}
fn check_size(&self) -> Result<()> {
if self.encoded_len() > MAX_PODCAST_RECORD_BYTES {
Err(invalid("podcast record is too large"))
} else {
Ok(())
}
}
}
/// Total order of two records of the same `(podcast_id, publisher)`.
///
/// Greater wins: higher `revision`; at an equal revision a tombstone beats a
/// card; then later `issued_at_ms` (a republish); then later
/// `expires_at_ms`; then the lexicographically greater canonical signing
/// input. Every node therefore keeps the same winner whatever the delivery
/// order.
pub fn compare_records(left: &SignedPodcastRecord, right: &SignedPodcastRecord) -> Ordering {
left.revision()
.cmp(&right.revision())
.then_with(|| left.is_tombstone().cmp(&right.is_tombstone()))
.then_with(|| left.issued_at_ms().cmp(&right.issued_at_ms()))
.then_with(|| left.expires_at_ms().cmp(&right.expires_at_ms()))
.then_with(|| left.signing_bytes().cmp(&right.signing_bytes()))
}
struct Canonical(Vec<u8>);
impl Canonical {
fn new(domain: &str) -> Self {
let mut bytes = Vec::with_capacity(512);
bytes.extend_from_slice(domain.as_bytes());
bytes.push(0);
Self(bytes)
}
fn raw(&mut self, bytes: &[u8]) {
self.0.extend_from_slice(bytes);
}
fn u8(&mut self, value: u8) {
self.0.push(value);
}
fn u16(&mut self, value: u16) {
self.raw(&value.to_le_bytes());
}
fn u32(&mut self, value: u32) {
self.raw(&value.to_le_bytes());
}
fn u64(&mut self, value: u64) {
self.raw(&value.to_le_bytes());
}
fn i64(&mut self, value: i64) {
self.raw(&value.to_le_bytes());
}
fn str(&mut self, value: &str) {
self.u32(value.len() as u32);
self.raw(value.as_bytes());
}
fn opt_str(&mut self, value: Option<&str>) {
match value {
Some(value) => {
self.u8(1);
self.str(value);
}
None => self.u8(0),
}
}
fn finish(self) -> Vec<u8> {
self.0
}
}
fn check_header(version: u16, network_id: &NetworkId, expected: &NetworkId) -> Result<()> {
if version != PODCAST_DHT_PROTOCOL_VERSION {
return Err(invalid("unsupported podcast record version"));
}
if network_id != expected {
return Err(invalid("podcast record belongs to another network"));
}
Ok(())
}
fn check_lifetime(
revision: u64,
issued_at_ms: u64,
expires_at_ms: u64,
ttl: Duration,
) -> Result<()> {
if revision == 0
|| issued_at_ms == 0
|| expires_at_ms <= issued_at_ms
|| expires_at_ms - issued_at_ms > ttl.as_millis() as u64
{
return Err(invalid("invalid podcast record revision or lifetime"));
}
Ok(())
}
fn check_chars(field: &'static str, value: &str, max_chars: usize) -> Result<()> {
if value.len() > max_chars * 4 || value.chars().count() > max_chars {
return Err(MusicDhtError::Protocol(format!("card {field} is too long")));
}
Ok(())
}
fn check_line(field: &'static str, value: &str, max_chars: usize, required: bool) -> Result<()> {
if required && value.trim().is_empty() {
return Err(MusicDhtError::Protocol(format!("card {field} is empty")));
}
check_chars(field, value, max_chars)?;
if value.chars().any(char::is_control) {
return Err(MusicDhtError::Protocol(format!(
"card {field} contains control characters"
)));
}
Ok(())
}
fn valid_language(language: &str) -> bool {
!language.is_empty()
&& language.len() <= MAX_CARD_LANGUAGE_BYTES
&& language
.bytes()
.all(|byte| byte.is_ascii_alphanumeric() || byte == b'-')
}
fn clean_line(text: &str) -> String {
text.split(char::is_control)
.filter(|part| !part.is_empty())
.collect::<Vec<_>>()
.join(" ")
.trim()
.to_string()
}
fn clean_block(text: &str) -> String {
text.chars()
.filter(|c| !c.is_control() || *c == '\n' || *c == '\t')
.collect()
}
fn bounded_line(text: Option<&str>, max_chars: usize) -> Option<String> {
text.map(clean_line)
.filter(|text| !text.is_empty())
.map(|text| bounded_text(&text, max_chars))
}
fn invalid(reason: &str) -> MusicDhtError {
MusicDhtError::Protocol(reason.to_string())
}
pub(crate) fn publication_error(reason: &str) -> MusicDhtError {
MusicDhtError::PodcastPublication(reason.to_string())
}
// Statically bound the worst-case card so `MAX_PODCAST_RECORD_BYTES` is a
// defensive limit, not the one users hit first.
const _: () = assert!(
MAX_CARD_TITLE_CHARS * 4
+ MAX_CARD_AUTHOR_CHARS * 4
+ MAX_CARD_DESCRIPTION_CHARS * 4
+ MAX_CARD_CATEGORIES * MAX_CARD_CATEGORY_CHARS * 4
+ MAX_CARD_LANGUAGE_BYTES
+ 4 * MAX_URL_BYTES
+ 1024
<= MAX_PODCAST_RECORD_BYTES
);
File diff suppressed because it is too large. Load diff
+339
View File
@@ -0,0 +1,339 @@
//! Advisory detection of feed URLs that look private.
//!
//! Premium and member feeds usually carry a personal secret in the URL. A
//! card publishes its feed URL verbatim, so the UI must ask the user before
//! publishing a URL for which [`feed_url_privacy_warning`] returns a warning.
//! The heuristics are deliberately broad and can produce false positives
//! (and miss secrets); they never block publication by themselves.
use std::collections::HashMap;
use std::fmt;
use serde::{Deserialize, Serialize};
use crate::podcast::FeedUrl;
/// Query-parameter name words that usually carry a credential. A name is
/// split into lowercase words at non-alphanumeric characters and camelCase
/// boundaries; any listed word triggers a warning.
pub const SECRET_QUERY_WORDS: &[&str] = &[
"token",
"key",
"auth",
"secret",
"sig",
"signature",
"uid",
"user",
"pass",
"password",
"passwd",
"pwd",
"session",
"sess",
"sid",
"access",
"private",
"credential",
"credentials",
"apikey",
"jwt",
"hmac",
];
/// Substrings that trigger a warning anywhere in a query-parameter name,
/// for names such as `apitoken` or `privatefeed` that do not split.
const SECRET_QUERY_SUBSTRINGS: &[&str] = &[
"token",
"secret",
"signature",
"password",
"passwd",
"session",
"private",
"credential",
"apikey",
];
/// Hosts (and their subdomains) known to serve personal member feeds.
pub const PRIVATE_FEED_HOSTS: &[&str] = &[
"patreon.com",
"supercast.com",
"supportingcast.fm",
"glow.fm",
];
/// Minimum length of a query value considered a possible secret.
const MIN_SECRET_VALUE_CHARS: usize = 16;
/// Minimum length of a path segment considered a possible secret.
const MIN_SECRET_SEGMENT_CHARS: usize = 32;
/// Minimum Shannon entropy, in bits per character, of a possible secret.
const MIN_SECRET_ENTROPY_BITS: f64 = 3.0;
/// Why a feed URL may contain a personal secret.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(tag = "reason", rename_all = "snake_case")]
#[non_exhaustive]
pub enum FeedUrlPrivacyWarning {
/// A query parameter whose name suggests a credential.
SecretQueryParameter {
/// Parameter name as it appears in the URL.
name: String,
},
/// The host is known to serve personal member feeds.
PrivateFeedHost {
/// Lowercase host.
host: String,
},
/// A long, random-looking query value.
HighEntropyQueryValue {
/// Name of the parameter carrying the value.
name: String,
},
/// A long, random-looking path segment.
HighEntropyPathSegment {
/// Zero-based index of the segment after the host.
index: usize,
},
}
impl fmt::Display for FeedUrlPrivacyWarning {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
Self::SecretQueryParameter { name } => write!(
f,
"the feed URL has a query parameter \"{name}\" that looks like a credential"
),
Self::PrivateFeedHost { host } => {
write!(f, "{host} usually serves personal member feeds")
}
Self::HighEntropyQueryValue { name } => write!(
f,
"the value of the query parameter \"{name}\" looks like a secret token"
),
Self::HighEntropyPathSegment { index } => write!(
f,
"path segment {} of the feed URL looks like a secret token",
index + 1
),
}
}
}
/// Returns why `url` may contain a personal secret, or `None`.
///
/// Advisory only (see the module documentation). Checks, in order: query
/// parameter names ([`SECRET_QUERY_WORDS`]), known member-feed hosts
/// ([`PRIVATE_FEED_HOSTS`]), long high-entropy query values, and long
/// high-entropy path segments.
pub fn feed_url_privacy_warning(url: &FeedUrl) -> Option<FeedUrlPrivacyWarning> {
let value = url.as_str();
let rest = value.split_once("://").map_or(value, |(_, rest)| rest);
let (authority, path_and_query) = match rest.find('/') {
Some(index) => rest.split_at(index),
None => (rest, "/"),
};
let host = if authority.starts_with('[') {
authority
.find(']')
.map_or(authority, |end| &authority[..=end])
} else {
authority.split(':').next().unwrap_or(authority)
};
let (path, query) = path_and_query
.split_once('?')
.unwrap_or((path_and_query, ""));
let parameters: Vec<(&str, &str)> = query
.split('&')
.filter(|pair| !pair.is_empty())
.map(|pair| pair.split_once('=').unwrap_or((pair, "")))
.collect();
if let Some((name, _)) = parameters
.iter()
.find(|(name, _)| is_secret_parameter_name(name))
{
return Some(FeedUrlPrivacyWarning::SecretQueryParameter {
name: (*name).to_string(),
});
}
if PRIVATE_FEED_HOSTS
.iter()
.any(|known| host == *known || host.ends_with(&format!(".{known}")))
{
return Some(FeedUrlPrivacyWarning::PrivateFeedHost {
host: host.to_string(),
});
}
if let Some((name, _)) = parameters
.iter()
.find(|(_, value)| looks_random(value, MIN_SECRET_VALUE_CHARS))
{
return Some(FeedUrlPrivacyWarning::HighEntropyQueryValue {
name: (*name).to_string(),
});
}
path.split('/')
.filter(|segment| !segment.is_empty())
.position(|segment| {
let stem = segment.rsplit_once('.').map_or(segment, |(stem, _)| stem);
looks_random(stem, MIN_SECRET_SEGMENT_CHARS)
})
.map(|index| FeedUrlPrivacyWarning::HighEntropyPathSegment { index })
}
fn is_secret_parameter_name(name: &str) -> bool {
let words = split_words(name);
if words
.iter()
.any(|word| SECRET_QUERY_WORDS.contains(&word.as_str()))
{
return true;
}
let compact: String = name
.chars()
.filter(char::is_ascii_alphanumeric)
.collect::<String>()
.to_ascii_lowercase();
SECRET_QUERY_SUBSTRINGS
.iter()
.any(|needle| compact.contains(needle))
}
/// Splits at non-alphanumeric characters and lower-to-upper case changes.
fn split_words(name: &str) -> Vec<String> {
let mut words = Vec::new();
let mut current = String::new();
let mut previous_lower = false;
for c in name.chars() {
if !c.is_ascii_alphanumeric() {
if !current.is_empty() {
words.push(std::mem::take(&mut current));
}
previous_lower = false;
continue;
}
if c.is_ascii_uppercase() && previous_lower && !current.is_empty() {
words.push(std::mem::take(&mut current));
}
previous_lower = c.is_ascii_lowercase() || c.is_ascii_digit();
current.push(c.to_ascii_lowercase());
}
if !current.is_empty() {
words.push(current);
}
words
}
/// A long token-alphabet string mixing letters and digits (or both letter
/// cases) with high per-character entropy.
fn looks_random(value: &str, min_chars: usize) -> bool {
if value.chars().count() < min_chars
|| !value
.chars()
.all(|c| c.is_ascii_alphanumeric() || "-_.~%=+".contains(c))
{
return false;
}
let has_digit = value.chars().any(|c| c.is_ascii_digit());
let has_lower = value.chars().any(|c| c.is_ascii_lowercase());
let has_upper = value.chars().any(|c| c.is_ascii_uppercase());
let classes = [has_digit, has_lower, has_upper]
.into_iter()
.filter(|present| *present)
.count();
classes >= 2 && shannon_entropy(value) >= MIN_SECRET_ENTROPY_BITS
}
fn shannon_entropy(value: &str) -> f64 {
let mut counts: HashMap<char, usize> = HashMap::new();
for c in value.chars() {
*counts.entry(c).or_default() += 1;
}
let total = value.chars().count() as f64;
counts
.values()
.map(|count| {
let p = *count as f64 / total;
-p * p.log2()
})
.sum()
}
#[cfg(test)]
mod tests {
use super::*;
fn warn(url: &str) -> Option<FeedUrlPrivacyWarning> {
feed_url_privacy_warning(&FeedUrl::parse(url).unwrap())
}
#[test]
fn public_feeds_have_no_warning() {
for url in [
"https://feeds.example.com/podcast.xml",
"https://example.com/feed?format=rss&page=2",
"https://anchor.example/s/abc123de/podcast/rss",
"https://feeds.example.com/12345/rss",
"https://example.com/feed?author=jane",
"https://example.com:8443/shows/my-great-show/feed.xml",
] {
assert_eq!(warn(url), None, "{url}");
}
}
#[test]
fn secret_query_names_warn() {
for (url, name) in [
("https://example.com/feed?token=abc", "token"),
("https://example.com/feed?auth_token=abc", "auth_token"),
(
"https://example.com/feed?X-Amz-Signature=abc",
"X-Amz-Signature",
),
("https://example.com/feed?accessKey=abc", "accessKey"),
("https://example.com/feed?a=1&uid=42", "uid"),
("https://example.com/feed?apitoken=x", "apitoken"),
("https://example.com/feed?PRIVATE=1", "PRIVATE"),
("https://example.com/feed?session_id=1", "session_id"),
] {
assert_eq!(
warn(url),
Some(FeedUrlPrivacyWarning::SecretQueryParameter { name: name.into() }),
"{url}"
);
}
}
#[test]
fn known_member_hosts_warn() {
assert_eq!(
warn("https://www.patreon.com/rss/show"),
Some(FeedUrlPrivacyWarning::PrivateFeedHost {
host: "www.patreon.com".into()
})
);
assert_eq!(warn("https://notpatreon.com/rss"), None);
}
#[test]
fn high_entropy_values_and_segments_warn() {
assert_eq!(
warn("https://example.com/feed?c=Zx81kQ2mP0aLr7Vb"),
Some(FeedUrlPrivacyWarning::HighEntropyQueryValue { name: "c".into() })
);
assert_eq!(warn("https://example.com/feed?c=aaaaaaaaaaaaaaaa1"), None);
assert_eq!(
warn("https://example.com/feeds/q8Zr2LmK0pXv7TnB4sYc1HdWe9GuJa3F.rss"),
Some(FeedUrlPrivacyWarning::HighEntropyPathSegment { index: 1 })
);
// Below the segment length: an ordinary short id.
assert_eq!(warn("https://example.com/feeds/q8Zr2LmK0pXv7TnB.rss"), None);
}
#[test]
fn warnings_render_without_the_secret() {
let warning = warn("https://example.com/feed?c=Zx81kQ2mP0aLr7Vb").unwrap();
assert!(!warning.to_string().contains("Zx81"));
}
}
+296
View File
@@ -0,0 +1,296 @@
//! Podcast search hits and their merge with music search results.
use std::cmp::Reverse;
use std::collections::{BTreeMap, HashMap, HashSet};
use std::fmt;
use federation_net::EndpointId;
use serde::{Deserialize, Serialize};
use crate::normalization::{normalize_name, tokenize};
use crate::podcast::{FeedUrl, PodcastId};
use crate::record::{ItemKind, LibraryItem};
use super::card::{MAX_INDEX_TOKEN_BYTES, PodcastCard, SignedPodcastRecord, compare_records};
/// Maximum distinct query tokens used by a podcast text search.
pub const MAX_PODCAST_QUERY_TOKENS: usize = 8;
/// Maximum publishers reported per hit and per claimed feed URL.
pub const MAX_HIT_PUBLISHERS: usize = 64;
/// Maximum claimed moved feed URLs reported per hit.
pub const MAX_HIT_FEED_URL_CLAIMS: usize = 8;
/// Kind of a federated search hit.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum SearchHitKind {
/// A musical artist.
Artist,
/// A release.
Release,
/// A track.
Track,
/// A public podcast card.
Podcast,
}
impl SearchHitKind {
/// Stable lowercase name.
pub fn as_str(&self) -> &'static str {
match self {
Self::Artist => "artist",
Self::Release => "release",
Self::Track => "track",
Self::Podcast => "podcast",
}
}
}
impl fmt::Display for SearchHitKind {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.write_str(self.as_str())
}
}
impl From<ItemKind> for SearchHitKind {
fn from(kind: ItemKind) -> Self {
match kind {
ItemKind::Artist => Self::Artist,
ItemKind::Release => Self::Release,
ItemKind::Track => Self::Track,
}
}
}
/// One public podcast found in the discovery overlay.
///
/// Cards of the same podcast published by several devices are merged by
/// `podcast_id`. The hit carries no personal state of any publisher.
///
/// Only [`Self::feed_url`] is bound to the podcast: the id is its hash. Every
/// other field is a claim of some publisher device, and device identities are
/// free, so neither the metadata nor the number of publishers is a trust
/// signal.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct PodcastSearchHit {
/// Podcast.
pub podcast_id: PodcastId,
/// The URL `podcast_id` is derived from
/// (`PodcastId::from_feed_url(feed_url) == podcast_id`). Subscribe with
/// this URL; follow the feed's own HTTP redirects and
/// `itunes:new-feed-url` from there.
pub feed_url: FeedUrl,
/// Representative card for display: a card whose `feed_url` is the
/// origin URL when one exists, then the title most publishers agree on,
/// then the smallest publisher key. Its metadata is a publisher's claim.
pub card: PodcastCard,
/// Sorted distinct publisher devices with a current valid card, at most
/// [`MAX_HIT_PUBLISHERS`]. These are devices, not users or subscribers:
/// one user may publish from several devices, and anyone can mint
/// devices.
pub publishers: Vec<EndpointId>,
/// Feed URLs other than [`Self::feed_url`] that publishers claim the
/// podcast moved to, most claimed first, at most
/// [`MAX_HIT_FEED_URL_CLAIMS`]. Unverified: never fetch one as this
/// podcast or record it as an alias unless the feed at
/// [`Self::feed_url`] itself redirects there.
pub claimed_feed_urls: Vec<ClaimedFeedUrl>,
}
impl PodcastSearchHit {
/// Always [`SearchHitKind::Podcast`].
pub fn kind(&self) -> SearchHitKind {
SearchHitKind::Podcast
}
}
/// An unverified moved-feed URL claimed by some publishers of a podcast.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct ClaimedFeedUrl {
/// Claimed current feed URL.
pub feed_url: FeedUrl,
/// Sorted distinct publisher devices claiming it, at most
/// [`MAX_HIT_PUBLISHERS`].
pub publishers: Vec<EndpointId>,
}
/// A music or podcast result of a federated search.
///
/// Serialized as `{"type": "music" | "podcast", "hit": {...}}`; use
/// [`FederatedSearchHit::kind`] for the fine-grained kind.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
#[serde(tag = "type", content = "hit", rename_all = "snake_case")]
pub enum FederatedSearchHit {
/// A music-DHT record (artist, release or track).
Music(LibraryItem),
/// A public podcast.
Podcast(PodcastSearchHit),
}
impl FederatedSearchHit {
/// Kind of the hit: `artist`, `release`, `track` or `podcast`.
pub fn kind(&self) -> SearchHitKind {
match self {
Self::Music(item) => item.kind.into(),
Self::Podcast(_) => SearchHitKind::Podcast,
}
}
/// Display name: the item name or the podcast title.
pub fn title(&self) -> &str {
match self {
Self::Music(item) => &item.name,
Self::Podcast(hit) => &hit.card.title,
}
}
}
/// Merges music results (as returned by `MusicDhtService::search_network`)
/// with podcast hits for the same query.
///
/// Results whose normalized name equals the normalized query come first;
/// within each tier music keeps its order and precedes podcasts, which keep
/// theirs. Existing music consumers that only read `SearchOutcome` are
/// unaffected.
pub fn merge_search_results(
query: &str,
music: Vec<LibraryItem>,
podcasts: Vec<PodcastSearchHit>,
) -> Vec<FederatedSearchHit> {
let normalized = normalize_name(query);
let mut exact = Vec::new();
let mut rest = Vec::new();
let hits = music
.into_iter()
.map(FederatedSearchHit::Music)
.chain(podcasts.into_iter().map(FederatedSearchHit::Podcast));
for hit in hits {
let is_exact = !normalized.is_empty() && normalize_name(hit.title()) == normalized;
if is_exact {
exact.push(hit);
} else {
rest.push(hit);
}
}
exact.extend(rest);
exact
}
/// Distinct normalized query tokens, at most [`MAX_PODCAST_QUERY_TOKENS`].
pub fn podcast_query_tokens(query: &str) -> Vec<String> {
let mut seen = HashSet::new();
tokenize(&normalize_name(query))
.into_iter()
.filter(|token| token.len() <= MAX_INDEX_TOKEN_BYTES && seen.insert(token.clone()))
.take(MAX_PODCAST_QUERY_TOKENS)
.collect()
}
/// Keeps the winning record per `(podcast_id, publisher)` and groups the
/// cards that are still active by podcast. Records must be verified.
pub(crate) fn aggregate_hits(
records: impl IntoIterator<Item = SignedPodcastRecord>,
) -> Vec<PodcastSearchHit> {
let mut winners: HashMap<(PodcastId, EndpointId), SignedPodcastRecord> = HashMap::new();
for record in records {
let owner = (record.podcast_id().clone(), record.publisher());
match winners.get(&owner) {
Some(current) if compare_records(current, &record).is_ge() => {}
_ => {
winners.insert(owner, record);
}
}
}
let mut by_podcast: BTreeMap<PodcastId, Vec<PodcastCard>> = BTreeMap::new();
for record in winners.into_values() {
if let Some(card) = record.card() {
by_podcast
.entry(card.podcast_id.clone())
.or_default()
.push(card.clone());
}
}
by_podcast
.into_iter()
.map(|(podcast_id, cards)| {
let mut title_votes: HashMap<String, usize> = HashMap::new();
for card in &cards {
*title_votes.entry(normalize_name(&card.title)).or_default() += 1;
}
// Publisher-controlled times and revisions never break ties.
let card = cards
.iter()
.max_by_key(|card| {
(
card.feed_url_is_origin(),
title_votes[&normalize_name(&card.title)],
Reverse(*card.publisher.as_bytes()),
)
})
.cloned()
.expect("a group has at least one card");
let mut claims: BTreeMap<&FeedUrl, Vec<EndpointId>> = BTreeMap::new();
for card in cards.iter().filter(|card| !card.feed_url_is_origin()) {
claims
.entry(&card.feed_url)
.or_default()
.push(card.publisher);
}
let mut claimed_feed_urls: Vec<ClaimedFeedUrl> = claims
.into_iter()
.map(|(feed_url, publishers)| ClaimedFeedUrl {
feed_url: feed_url.clone(),
publishers: sorted_publishers(publishers),
})
.collect();
claimed_feed_urls.sort_by(|left, right| {
right
.publishers
.len()
.cmp(&left.publishers.len())
.then_with(|| left.feed_url.cmp(&right.feed_url))
});
claimed_feed_urls.truncate(MAX_HIT_FEED_URL_CLAIMS);
PodcastSearchHit {
podcast_id,
feed_url: card.origin_feed_url().clone(),
publishers: sorted_publishers(cards.iter().map(|card| card.publisher).collect()),
card,
claimed_feed_urls,
}
})
.collect()
}
fn sorted_publishers(mut publishers: Vec<EndpointId>) -> Vec<EndpointId> {
publishers.sort_by_key(|publisher| *publisher.as_bytes());
publishers.dedup();
publishers.truncate(MAX_HIT_PUBLISHERS);
publishers
}
/// Keeps hits whose card is indexed under every query token and ranks
/// them: exact normalized title first, then hits with a card published under
/// the origin feed URL, then title and id. The publisher count is not used:
/// anyone can inflate it with new device identities.
pub(crate) fn rank_hits(
mut hits: Vec<PodcastSearchHit>,
query: &str,
tokens: &[String],
) -> Vec<PodcastSearchHit> {
let normalized_query = normalize_name(query);
hits.retain(|hit| {
let indexed = hit.card.index_tokens();
tokens.iter().all(|token| indexed.contains(token))
});
hits.sort_by_cached_key(|hit| {
let title = normalize_name(&hit.card.title);
(
title != normalized_query,
!hit.card.feed_url_is_origin(),
title,
hit.podcast_id.clone(),
)
});
hits
}
+891
View File
@@ -0,0 +1,891 @@
//! SQLite cache of the podcast-discovery overlay: replicas, routing contacts
//! and this device's own publications.
//!
//! The cache is replaceable federation state, kept in its own database file
//! (like the similarity-routing cache). Every record read from it is decoded
//! and verified again before it is used or served.
use std::collections::HashSet;
use std::path::Path;
use std::str::FromStr;
use std::sync::{Arc, Mutex, MutexGuard, PoisonError};
use federation_net::{EndpointId, NetworkId};
use rusqlite::{Connection, OptionalExtension, params};
use crate::error::{MusicDhtError, Result};
use crate::podcast::PodcastId;
use crate::routing::{NodeContact, NodeId};
use super::MAX_PODCAST_DHT_RECORDS;
use super::card::{PodcastCardSpec, PodcastDhtKey, SignedPodcastRecord, compare_records};
/// `PRAGMA user_version` of the podcast-discovery cache schema. Version 1
/// (pre-release) stored one payload per key; its replica table is dropped on
/// open because the cache is replaceable. Own publications are kept.
pub const PODCAST_DHT_CACHE_SCHEMA_VERSION: i32 = 2;
/// Maximum records (cards and tombstones) indexed under one key. It equals
/// [`MAX_PODCAST_DHT_RECORDS`], so one `FindValue` response carries every
/// record of a key: nothing admitted is unreachable.
///
/// A full key admits a newcomer by evicting a random placement from its
/// younger half (by first indexing time). The older half is never evicted
/// for newcomers, so cards that were there before a flood stay, and the
/// younger half stays contestable, so nobody who stored first holds the key
/// for good.
pub const MAX_PODCAST_RECORDS_PER_KEY: usize = MAX_PODCAST_DHT_RECORDS;
/// Maximum tombstones indexed under one key, including tombstones that
/// replaced their owner's card in place. Over budget, a random tombstone of
/// another owner from the younger half of the key's tombstones is evicted.
/// It equals [`MAX_PODCAST_RESPONSE_TOMBSTONES`], so every tombstone of a key
/// is served.
pub const MAX_PODCAST_TOMBSTONES_PER_KEY: usize = MAX_PODCAST_RESPONSE_TOMBSTONES;
/// Maximum tombstones in one `FindValue` response; cards fill the rest.
pub const MAX_PODCAST_RESPONSE_TOMBSTONES: usize = 32;
/// Maximum `(podcast_id, publisher)` records one publisher device keeps in a
/// replica. A publisher over it is refused (nothing is evicted for it).
pub const MAX_PODCAST_RECORDS_PER_PUBLISHER: usize = 256;
/// Maximum `(podcast_id, publisher)` records in a replica cache. A full cache
/// admits a new record by evicting random records from the younger half of
/// the cache (by first indexing time), like a full key.
pub const MAX_PODCAST_CACHE_RECORDS: usize = 4_096;
/// Maximum total encoded size of the records in a replica cache, enforced
/// like [`MAX_PODCAST_CACHE_RECORDS`].
pub const MAX_PODCAST_CACHE_BYTES: usize = 32 * 1024 * 1024;
/// Maximum persisted overlay contacts; the least recently seen are dropped.
pub const MAX_PERSISTED_PODCAST_DHT_PEERS: usize = 1_024;
/// Maximum records evicted from the cache to admit one record; when that is
/// not enough room, the record is refused and nothing is evicted.
const MAX_EVICTIONS_PER_RECORD: usize = 16;
const _: () = assert!(
MAX_PODCAST_RECORDS_PER_KEY <= MAX_PODCAST_DHT_RECORDS
&& MAX_PODCAST_TOMBSTONES_PER_KEY <= MAX_PODCAST_RESPONSE_TOMBSTONES
&& MAX_PODCAST_RESPONSE_TOMBSTONES <= MAX_PODCAST_DHT_RECORDS,
"every record admitted under a key must fit one FindValue response"
);
const SCHEMA: &str = "
CREATE TABLE IF NOT EXISTS podcast_dht_records (
podcast_id TEXT NOT NULL,
publisher TEXT NOT NULL,
tombstone INTEGER NOT NULL,
expires_at_ms INTEGER NOT NULL,
payload_bytes INTEGER NOT NULL,
payload BLOB NOT NULL,
PRIMARY KEY (podcast_id, publisher)
);
CREATE INDEX IF NOT EXISTS idx_podcast_dht_records_expiry
ON podcast_dht_records(expires_at_ms);
CREATE INDEX IF NOT EXISTS idx_podcast_dht_records_publisher
ON podcast_dht_records(publisher);
CREATE TABLE IF NOT EXISTS podcast_dht_keys (
dht_key BLOB NOT NULL,
podcast_id TEXT NOT NULL,
publisher TEXT NOT NULL,
tombstone INTEGER NOT NULL,
expires_at_ms INTEGER NOT NULL,
first_stored_at_ms INTEGER NOT NULL,
PRIMARY KEY (dht_key, podcast_id, publisher)
);
CREATE INDEX IF NOT EXISTS idx_podcast_dht_keys_owner
ON podcast_dht_keys(podcast_id, publisher);
CREATE INDEX IF NOT EXISTS idx_podcast_dht_keys_expiry
ON podcast_dht_keys(expires_at_ms);
CREATE TABLE IF NOT EXISTS podcast_dht_peers (
peer_id TEXT PRIMARY KEY,
node_id BLOB NOT NULL,
ticket TEXT NOT NULL,
last_seen_ms INTEGER NOT NULL
);
CREATE TABLE IF NOT EXISTS podcast_dht_local (
podcast_id TEXT PRIMARY KEY,
revision INTEGER NOT NULL,
spec BLOB,
record BLOB NOT NULL,
updated_at_ms INTEGER NOT NULL
);
";
/// Admission limits of a replica cache.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(crate) struct StoreLimits {
pub per_key: usize,
pub tombstones_per_key: usize,
pub per_publisher: usize,
pub records: usize,
pub bytes: usize,
}
impl Default for StoreLimits {
fn default() -> Self {
Self {
per_key: MAX_PODCAST_RECORDS_PER_KEY,
tombstones_per_key: MAX_PODCAST_TOMBSTONES_PER_KEY,
per_publisher: MAX_PODCAST_RECORDS_PER_PUBLISHER,
records: MAX_PODCAST_CACHE_RECORDS,
bytes: MAX_PODCAST_CACHE_BYTES,
}
}
}
/// This device's latest publication of one podcast.
#[derive(Debug, Clone, PartialEq, Eq)]
pub(crate) struct LocalPublication {
pub podcast_id: PodcastId,
/// Greatest revision ever used for this podcast by this device.
pub revision: u64,
/// Published card content; `None` once unpublished.
pub spec: Option<PodcastCardSpec>,
/// Latest signed record (card or tombstone).
pub record: SignedPodcastRecord,
}
#[derive(Clone)]
pub(crate) struct PodcastDhtDatabase {
conn: Arc<Mutex<Connection>>,
network_id: NetworkId,
limits: StoreLimits,
}
impl std::fmt::Debug for PodcastDhtDatabase {
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
formatter
.debug_struct("PodcastDhtDatabase")
.finish_non_exhaustive()
}
}
impl PodcastDhtDatabase {
pub async fn open(path: &Path, network_id: NetworkId) -> Result<Self> {
Self::open_with_limits(path, network_id, StoreLimits::default()).await
}
pub async fn open_with_limits(
path: &Path,
network_id: NetworkId,
limits: StoreLimits,
) -> Result<Self> {
let path = path.to_path_buf();
let conn = tokio::task::spawn_blocking(move || -> Result<Connection> {
let conn = Connection::open(&path).map_err(|error| {
MusicDhtError::Database(format!("failed to open {}: {error}", path.display()))
})?;
let version: i32 = conn
.query_row("PRAGMA user_version", [], |row| row.get(0))
.map_err(database_error)?;
if version > PODCAST_DHT_CACHE_SCHEMA_VERSION {
return Err(MusicDhtError::Database(format!(
"podcast DHT cache schema {version} is newer than supported {PODCAST_DHT_CACHE_SCHEMA_VERSION}"
)));
}
if version == 1 {
// Replicas are replaceable; the old layout is dropped.
conn.execute_batch("DROP TABLE IF EXISTS podcast_dht_records;")
.map_err(database_error)?;
}
conn.execute_batch(SCHEMA).map_err(|error| {
MusicDhtError::Database(format!("failed to initialize podcast DHT cache: {error}"))
})?;
conn.pragma_update(None, "user_version", PODCAST_DHT_CACHE_SCHEMA_VERSION)
.map_err(database_error)?;
Ok(conn)
})
.await
.map_err(|error| MusicDhtError::Database(format!("database task panicked: {error}")))??;
Ok(Self {
conn: Arc::new(Mutex::new(conn)),
network_id,
limits,
})
}
async fn call<F, T>(&self, operation: F) -> Result<T>
where
F: FnOnce(&Connection, &NetworkId, &StoreLimits) -> rusqlite::Result<T> + Send + 'static,
T: Send + 'static,
{
let conn = Arc::clone(&self.conn);
let network_id = self.network_id;
let limits = self.limits;
tokio::task::spawn_blocking(move || {
let guard = lock(&conn);
operation(&guard, &network_id, &limits).map_err(database_error)
})
.await
.map_err(|error| MusicDhtError::Database(format!("database task panicked: {error}")))?
}
/// Stores records under the given keys. The caller has verified every
/// record, checked that each key belongs to it and that the sender is its
/// publisher. Returns the number of `(key, record)` placements written.
pub async fn store(
&self,
entries: Vec<(Vec<PodcastDhtKey>, SignedPodcastRecord)>,
now_ms: u64,
) -> Result<usize> {
self.call(move |conn, network_id, limits| {
let transaction = conn.unchecked_transaction()?;
let mut written = 0;
for (keys, record) in entries {
written += store_one(&transaction, network_id, limits, &keys, &record, now_ms)?;
}
transaction.commit()?;
Ok(written)
})
.await
}
/// Valid, unexpired records under `key`: at most
/// [`MAX_PODCAST_RESPONSE_TOMBSTONES`] tombstones (always returned so stale
/// replicas elsewhere can be overruled) and cards containing every token
/// in `filter_tokens` up to `limit` records in all. Records indexed longest
/// come first. Cards precede tombstones. With the default limits and
/// `limit` = [`MAX_PODCAST_DHT_RECORDS`], every record of the key fits.
pub async fn records_by_key(
&self,
key: PodcastDhtKey,
filter_tokens: Vec<String>,
now_ms: u64,
limit: usize,
) -> Result<Vec<SignedPodcastRecord>> {
self.call(move |conn, network_id, _| {
let mut statement = conn.prepare(
"SELECT r.payload FROM podcast_dht_keys k
JOIN podcast_dht_records r
ON r.podcast_id = k.podcast_id AND r.publisher = k.publisher
WHERE k.dht_key = ?1 AND k.tombstone = ?2 AND k.expires_at_ms > ?3
ORDER BY k.first_stored_at_ms, k.podcast_id, k.publisher",
)?;
let mut select = |tombstone: bool, limit: usize| -> rusqlite::Result<Vec<_>> {
let mut records = Vec::new();
if limit == 0 {
return Ok(records);
}
let rows = statement.query_map(
params![key.as_bytes().as_slice(), tombstone, clamp_i64(now_ms)],
|row| row.get::<_, Vec<u8>>(0),
)?;
for row in rows {
let Some(record) = decode_valid(&row?, network_id, now_ms) else {
continue;
};
if record.is_tombstone() != tombstone || !record.belongs_under(&key) {
continue;
}
let matches = record.card().is_none_or(|card| {
let tokens = card.index_tokens();
filter_tokens.iter().all(|token| tokens.contains(token))
});
if matches {
records.push(record);
if records.len() == limit {
break;
}
}
}
Ok(records)
};
let tombstones = select(true, MAX_PODCAST_RESPONSE_TOMBSTONES.min(limit))?;
let mut records = select(false, limit - tombstones.len())?;
records.extend(tombstones);
Ok(records)
})
.await
}
/// Removes expired records and their key placements. Returns the number
/// of records removed.
pub async fn delete_expired(&self, now_ms: u64) -> Result<usize> {
self.call(move |conn, _, _| {
let now = clamp_i64(now_ms);
conn.execute(
"DELETE FROM podcast_dht_keys WHERE expires_at_ms <= ?1",
[now],
)?;
conn.execute(
"DELETE FROM podcast_dht_records WHERE expires_at_ms <= ?1",
[now],
)
})
.await
}
pub async fn upsert_peer(&self, contact: NodeContact) -> Result<()> {
self.call(move |conn, _, _| {
conn.execute(
"INSERT OR REPLACE INTO podcast_dht_peers
(peer_id, node_id, ticket, last_seen_ms)
VALUES (?1, ?2, ?3, ?4)",
params![
contact.peer_id.to_string(),
contact.node_id.as_bytes().as_slice(),
contact.ticket,
clamp_i64(contact.last_seen_ms),
],
)?;
conn.execute(
"DELETE FROM podcast_dht_peers WHERE peer_id IN (
SELECT peer_id FROM podcast_dht_peers
ORDER BY last_seen_ms DESC, peer_id
LIMIT -1 OFFSET ?1)",
[MAX_PERSISTED_PODCAST_DHT_PEERS as i64],
)?;
Ok(())
})
.await
}
pub async fn remove_peer(&self, peer: EndpointId) -> Result<()> {
self.call(move |conn, _, _| {
conn.execute(
"DELETE FROM podcast_dht_peers WHERE peer_id = ?1",
[peer.to_string()],
)?;
Ok(())
})
.await
}
pub async fn load_peers(&self) -> Result<Vec<NodeContact>> {
self.call(move |conn, _, _| {
let mut statement =
conn.prepare("SELECT peer_id, ticket, last_seen_ms FROM podcast_dht_peers")?;
let rows = statement.query_map([], |row| {
Ok((
row.get::<_, String>(0)?,
row.get::<_, String>(1)?,
row.get::<_, i64>(2)?,
))
})?;
let mut peers = Vec::new();
for row in rows {
let (peer, ticket, last_seen_ms) = row?;
let Ok(peer_id) = EndpointId::from_str(&peer) else {
continue;
};
peers.push(NodeContact {
node_id: NodeId::from_endpoint(&peer_id),
peer_id,
ticket,
last_seen_ms: last_seen_ms.max(0) as u64,
});
}
Ok(peers)
})
.await
}
pub async fn local_publication(
&self,
podcast_id: PodcastId,
) -> Result<Option<LocalPublication>> {
self.call(move |conn, _, _| {
conn.query_row(
"SELECT podcast_id, revision, spec, record FROM podcast_dht_local
WHERE podcast_id = ?1",
[podcast_id.as_str()],
local_from_row,
)
.optional()
.map(Option::flatten)
})
.await
}
pub async fn local_publications(&self) -> Result<Vec<LocalPublication>> {
self.call(move |conn, _, _| {
let mut statement = conn.prepare(
"SELECT podcast_id, revision, spec, record FROM podcast_dht_local
ORDER BY podcast_id",
)?;
let rows = statement.query_map([], local_from_row)?;
let mut out = Vec::new();
for row in rows {
if let Some(publication) = row? {
out.push(publication);
}
}
Ok(out)
})
.await
}
pub async fn put_local_publication(
&self,
publication: LocalPublication,
now_ms: u64,
) -> Result<()> {
self.call(move |conn, _, _| {
let spec = publication
.spec
.as_ref()
.map(postcard::to_stdvec)
.transpose()
.map_err(to_sql_error)?;
let record = postcard::to_stdvec(&publication.record).map_err(to_sql_error)?;
conn.execute(
"INSERT OR REPLACE INTO podcast_dht_local
(podcast_id, revision, spec, record, updated_at_ms)
VALUES (?1, ?2, ?3, ?4, ?5)",
params![
publication.podcast_id.as_str(),
publication.revision as i64,
spec,
record,
clamp_i64(now_ms),
],
)?;
Ok(())
})
.await
}
}
fn store_one(
conn: &Connection,
network_id: &NetworkId,
limits: &StoreLimits,
keys: &[PodcastDhtKey],
record: &SignedPodcastRecord,
now_ms: u64,
) -> rusqlite::Result<usize> {
let podcast_id = record.podcast_id().as_str().to_string();
let publisher = record.publisher().to_string();
let now = clamp_i64(now_ms);
// The cached record of this (podcast, publisher), re-verified.
let cached: Option<(Vec<u8>, i64)> = conn
.query_row(
"SELECT payload, payload_bytes FROM podcast_dht_records
WHERE podcast_id = ?1 AND publisher = ?2",
params![podcast_id, publisher],
|row| Ok((row.get(0)?, row.get(1)?)),
)
.optional()?;
let (existing, existing_bytes) = match cached {
Some((payload, bytes)) => match decode_valid(&payload, network_id, now_ms) {
Some(existing) => (Some(existing), bytes.max(0) as usize),
None => {
delete_owner(conn, &podcast_id, &publisher)?;
(None, 0)
}
},
None => {
// Stray placements without a record.
delete_owner(conn, &podcast_id, &publisher)?;
(None, 0)
}
};
// Higher revision, tombstone at an equal revision, later republish...
let replace = match &existing {
Some(existing) => match compare_records(existing, record) {
std::cmp::Ordering::Greater => return Ok(0),
std::cmp::Ordering::Equal => false,
std::cmp::Ordering::Less => true,
},
None => true,
};
let payload = postcard::to_stdvec(record).map_err(to_sql_error)?;
if replace
&& !make_room_for_record(
conn,
limits,
(&podcast_id, &publisher),
existing.is_none(),
existing_bytes,
payload.len(),
now,
)?
{
return Ok(0);
}
let mut indexed: HashSet<[u8; 32]> = HashSet::new();
{
let mut statement = conn.prepare(
"SELECT dht_key FROM podcast_dht_keys WHERE podcast_id = ?1 AND publisher = ?2",
)?;
let rows = statement.query_map(params![podcast_id, publisher], |row| {
row.get::<_, Vec<u8>>(0)
})?;
for row in rows {
if let Ok(key) = <[u8; 32]>::try_from(row?.as_slice()) {
indexed.insert(key);
}
}
}
let mut written = 0;
if replace {
// Placements the new version no longer belongs under are dropped (a
// renamed card, or a tombstone that does not list the key); the others
// now serve the new version.
let belongs: HashSet<[u8; 32]> = record
.dht_keys()
.iter()
.map(|key| *key.as_bytes())
.collect();
let stale: Vec<[u8; 32]> = indexed
.iter()
.filter(|key| !belongs.contains(*key))
.copied()
.collect();
for key in stale {
conn.execute(
"DELETE FROM podcast_dht_keys
WHERE dht_key = ?1 AND podcast_id = ?2 AND publisher = ?3",
params![key.as_slice(), podcast_id, publisher],
)?;
indexed.remove(&key);
}
written += conn.execute(
"UPDATE podcast_dht_keys SET tombstone = ?3, expires_at_ms = ?4
WHERE podcast_id = ?1 AND publisher = ?2",
params![
podcast_id,
publisher,
record.is_tombstone(),
clamp_i64(record.expires_at_ms()),
],
)?;
// A card converted to a tombstone in place counts against the
// tombstone budget of each key it stays under, like a new one.
if record.is_tombstone() {
for key in &indexed {
let key = PodcastDhtKey::from_bytes(*key);
make_room_under_key(conn, limits, &key, true, (&podcast_id, &publisher), now, 0)?;
}
}
}
// New placements. A tombstone only takes a new place under the podcast-id
// key: under a text key it may only replace a record of its owner, so
// nobody can index tombstones under someone else's search terms.
let id_key = PodcastDhtKey::podcast(network_id, record.podcast_id());
for key in keys {
if indexed.contains(key.as_bytes()) {
continue;
}
if record.is_tombstone() && *key != id_key {
continue;
}
if !make_room_under_key(
conn,
limits,
key,
record.is_tombstone(),
(&podcast_id, &publisher),
now,
1,
)? {
continue;
}
conn.execute(
"INSERT INTO podcast_dht_keys
(dht_key, podcast_id, publisher, tombstone, expires_at_ms, first_stored_at_ms)
VALUES (?1, ?2, ?3, ?4, ?5, ?6)",
params![
key.as_bytes().as_slice(),
podcast_id,
publisher,
record.is_tombstone(),
clamp_i64(record.expires_at_ms()),
now,
],
)?;
indexed.insert(*key.as_bytes());
written += 1;
}
if indexed.is_empty() {
delete_owner(conn, &podcast_id, &publisher)?;
return Ok(0);
}
if replace {
conn.execute(
"INSERT INTO podcast_dht_records
(podcast_id, publisher, tombstone, expires_at_ms, payload_bytes, payload)
VALUES (?1, ?2, ?3, ?4, ?5, ?6)
ON CONFLICT(podcast_id, publisher) DO UPDATE SET
tombstone = excluded.tombstone,
expires_at_ms = excluded.expires_at_ms,
payload_bytes = excluded.payload_bytes,
payload = excluded.payload",
params![
podcast_id,
publisher,
record.is_tombstone(),
clamp_i64(record.expires_at_ms()),
payload.len() as i64,
payload,
],
)?;
}
Ok(written)
}
/// Makes room in the cache for the new version of a record of `owner`.
///
/// * A publisher at [`StoreLimits::per_publisher`] gets no new record.
/// * When the record count (for a new owner) or the byte budget would be
/// exceeded, random records of other owners from the younger half of the
/// cache (by first indexing time) are evicted, at most
/// [`MAX_EVICTIONS_PER_RECORD`]. If that is not enough, nothing is evicted
/// and the record is refused.
fn make_room_for_record(
conn: &Connection,
limits: &StoreLimits,
owner: (&str, &str),
new_owner: bool,
previous_bytes: usize,
new_bytes: usize,
now: i64,
) -> rusqlite::Result<bool> {
if new_bytes > limits.bytes {
return Ok(false);
}
if new_owner {
let own: i64 = conn.query_row(
"SELECT COUNT(*) FROM podcast_dht_records WHERE publisher = ?1 AND expires_at_ms > ?2",
params![owner.1, now],
|row| row.get(0),
)?;
if own.max(0) as usize >= limits.per_publisher {
return Ok(false);
}
}
let (records, bytes): (i64, i64) = conn.query_row(
"SELECT COUNT(*), COALESCE(SUM(payload_bytes), 0) FROM podcast_dht_records
WHERE expires_at_ms > ?1",
[now],
|row| Ok((row.get(0)?, row.get(1)?)),
)?;
let mut records = records.max(0) as usize;
let mut bytes = (bytes.max(0) as usize).saturating_sub(previous_bytes);
let fits = |records: usize, bytes: usize| {
bytes.saturating_add(new_bytes) <= limits.bytes && (!new_owner || records < limits.records)
};
if fits(records, bytes) {
return Ok(true);
}
// Candidates: the younger half of the other owners' records, shuffled.
let others = records.saturating_sub(usize::from(!new_owner));
let mut statement = conn.prepare(
"SELECT podcast_id, publisher, payload_bytes FROM (
SELECT r.podcast_id, r.publisher, r.payload_bytes,
(SELECT MIN(k.first_stored_at_ms) FROM podcast_dht_keys k
WHERE k.podcast_id = r.podcast_id AND k.publisher = r.publisher) AS since
FROM podcast_dht_records r
WHERE r.expires_at_ms > ?1 AND NOT (r.podcast_id = ?2 AND r.publisher = ?3)
ORDER BY since DESC, random()
LIMIT ?4)
ORDER BY random() LIMIT ?5",
)?;
let candidates = statement
.query_map(
params![
now,
owner.0,
owner.1,
others.div_ceil(2) as i64,
MAX_EVICTIONS_PER_RECORD as i64
],
|row| {
Ok((
row.get::<_, String>(0)?,
row.get::<_, String>(1)?,
row.get::<_, i64>(2)?,
))
},
)?
.collect::<rusqlite::Result<Vec<_>>>()?;
let mut evict = Vec::new();
for (podcast_id, publisher, payload_bytes) in candidates {
if fits(records, bytes) {
break;
}
records = records.saturating_sub(1);
bytes = bytes.saturating_sub(payload_bytes.max(0) as usize);
evict.push((podcast_id, publisher));
}
if !fits(records, bytes) {
return Ok(false);
}
for (podcast_id, publisher) in evict {
delete_owner(conn, &podcast_id, &publisher)?;
}
Ok(true)
}
/// Makes room under `key` for `incoming` more placements (0 when an existing
/// placement of `owner` was converted to a tombstone in place) and returns
/// whether the key has room.
///
/// Over [`StoreLimits::per_key`], or over [`StoreLimits::tombstones_per_key`]
/// for a tombstone, a random placement of another owner from the younger
/// half of the key (by first indexing time; only tombstones for the
/// tombstone budget) is evicted. The older half is never evicted for a
/// newcomer.
fn make_room_under_key(
conn: &Connection,
limits: &StoreLimits,
key: &PodcastDhtKey,
tombstone: bool,
owner: (&str, &str),
now: i64,
incoming: usize,
) -> rusqlite::Result<bool> {
for _ in 0..=MAX_EVICTIONS_PER_RECORD {
let (total, tombstones): (i64, i64) = conn.query_row(
"SELECT COUNT(*), COALESCE(SUM(tombstone), 0) FROM podcast_dht_keys
WHERE dht_key = ?1 AND expires_at_ms > ?2",
params![key.as_bytes().as_slice(), now],
|row| Ok((row.get(0)?, row.get(1)?)),
)?;
let over_tombstones =
tombstone && tombstones.max(0) as usize + incoming > limits.tombstones_per_key;
let over_total = total.max(0) as usize + incoming > limits.per_key;
if !over_tombstones && !over_total {
return Ok(true);
}
let Some((podcast_id, publisher)) =
young_placement(conn, key, over_tombstones, owner, now)?
else {
return Ok(false);
};
conn.execute(
"DELETE FROM podcast_dht_keys
WHERE dht_key = ?1 AND podcast_id = ?2 AND publisher = ?3",
params![key.as_bytes().as_slice(), podcast_id, publisher],
)?;
let placed: bool = conn.query_row(
"SELECT EXISTS (SELECT 1 FROM podcast_dht_keys
WHERE podcast_id = ?1 AND publisher = ?2)",
params![podcast_id, publisher],
|row| row.get(0),
)?;
if !placed {
delete_owner(conn, &podcast_id, &publisher)?;
}
}
Ok(false)
}
/// A random unexpired placement of another owner than `owner` from the
/// younger half of `key` (only tombstones with `tombstones_only`).
fn young_placement(
conn: &Connection,
key: &PodcastDhtKey,
tombstones_only: bool,
owner: (&str, &str),
now: i64,
) -> rusqlite::Result<Option<(String, String)>> {
let filter = "dht_key = ?1 AND expires_at_ms > ?2 AND (?3 = 0 OR tombstone = 1)
AND NOT (podcast_id = ?4 AND publisher = ?5)";
let count: i64 = conn.query_row(
&format!("SELECT COUNT(*) FROM podcast_dht_keys WHERE {filter}"),
params![
key.as_bytes().as_slice(),
now,
tombstones_only,
owner.0,
owner.1
],
|row| row.get(0),
)?;
if count <= 0 {
return Ok(None);
}
conn.query_row(
&format!(
"SELECT podcast_id, publisher FROM (
SELECT podcast_id, publisher FROM podcast_dht_keys WHERE {filter}
ORDER BY first_stored_at_ms DESC, random() LIMIT ?6)
ORDER BY random() LIMIT 1"
),
params![
key.as_bytes().as_slice(),
now,
tombstones_only,
owner.0,
owner.1,
(count as usize).div_ceil(2) as i64
],
|row| Ok((row.get(0)?, row.get(1)?)),
)
.optional()
}
fn delete_owner(conn: &Connection, podcast_id: &str, publisher: &str) -> rusqlite::Result<()> {
conn.execute(
"DELETE FROM podcast_dht_keys WHERE podcast_id = ?1 AND publisher = ?2",
params![podcast_id, publisher],
)?;
conn.execute(
"DELETE FROM podcast_dht_records WHERE podcast_id = ?1 AND publisher = ?2",
params![podcast_id, publisher],
)?;
Ok(())
}
fn decode_valid(
payload: &[u8],
network_id: &NetworkId,
now_ms: u64,
) -> Option<SignedPodcastRecord> {
let record: SignedPodcastRecord = postcard::from_bytes(payload).ok()?;
record.verify_at(network_id, now_ms).ok()?;
Some(record)
}
fn local_from_row(row: &rusqlite::Row<'_>) -> rusqlite::Result<Option<LocalPublication>> {
let podcast_id: String = row.get(0)?;
let revision: i64 = row.get(1)?;
let spec: Option<Vec<u8>> = row.get(2)?;
let record: Vec<u8> = row.get(3)?;
let Ok(podcast_id) = PodcastId::parse(&podcast_id) else {
return Ok(None);
};
let spec = match spec {
Some(bytes) => match postcard::from_bytes::<PodcastCardSpec>(&bytes) {
Ok(spec) => Some(spec),
Err(_) => return Ok(None),
},
None => None,
};
let Ok(record) = postcard::from_bytes::<SignedPodcastRecord>(&record) else {
return Ok(None);
};
Ok(Some(LocalPublication {
podcast_id,
revision: revision as u64,
spec,
record,
}))
}
fn clamp_i64(value: u64) -> i64 {
value.min(i64::MAX as u64) as i64
}
fn to_sql_error(error: impl std::error::Error + Send + Sync + 'static) -> rusqlite::Error {
rusqlite::Error::ToSqlConversionFailure(Box::new(error))
}
fn database_error(error: rusqlite::Error) -> MusicDhtError {
MusicDhtError::Database(error.to_string())
}
fn lock<T>(mutex: &Mutex<T>) -> MutexGuard<'_, T> {
mutex.lock().unwrap_or_else(PoisonError::into_inner)
}
File diff suppressed because it is too large. Load diff
+763
View File
@@ -0,0 +1,763 @@
//! Integration test: public podcast discovery on three real loopback iroh
//! nodes.
//!
//! * A publishes a podcast its user shared. B is another device of the same
//! user: it applies the same merged registers (as device sync would deliver
//! them) and publishes the same podcast under its own key. C is an
//! unrelated peer outside the user's trusted-device group, with no
//! device-sync pairing.
//! * C finds the podcast by text and by id, merged over publishers {A, B};
//! once B unshares and goes offline, C still finds A's card, so B is
//! irrelevant to discovery. A podcast the user never shared (held by both
//! A and B) is never discoverable; after A unpublishes, the tombstone hides
//! the card from C.
//! * Forged, foreign, tampered, replayed, malformed, oversized and
//! wrong-version requests are rejected by a real peer.
//! * D, a music-DHT peer without the overlay, is dialed by bootstrap but
//! never answers on the overlay ALPN, so it never becomes an overlay contact
//! and is never asked for records or to store them.
//! * `sync_shares` stays fail-safe for revocation and holds back URLs that
//! look private (single node).
//! * Contacts a peer returns are never trusted: fabricated, unreachable
//! contacts do not enter the routing table (two nodes).
//! * After a restart, own cards persisted by the previous run are neither
//! republished nor served until the first `sync_shares`, so a share
//! revoked on another device meanwhile is never re-sent (two nodes).
use std::path::Path;
use std::sync::Arc;
use std::time::Duration;
use music_dht::capabilities::{CapabilityManifest, PODCAST_DHT_ID};
use music_dht::podcast::{FeedUrl, PodcastId, PodcastMeta, ShareRecord, SubscriptionRecord};
use music_dht::podcast_dht::{
MAX_PODCAST_DHT_REQUEST_BYTES, PODCAST_DHT_ALPN, PODCAST_DHT_PROTOCOL_VERSION, PodcastCard,
PodcastCardSpec, PodcastDht, PodcastDhtOperation, PodcastDhtRequest, PodcastDhtResponse,
PodcastRecordBody, PodcastSearchHit, PodcastShareState, PodcastStoreEntry, SignedPodcastRecord,
};
use music_dht::{MusicDhtConfig, MusicDhtError, MusicDhtService, NetworkId, NodeContact, NodeId};
use tokio::io::AsyncReadExt;
const TEST_TIMEOUT: Duration = Duration::from_secs(240);
struct Node {
service: Arc<MusicDhtService>,
dht: Arc<PodcastDht>,
tasks: Vec<tokio::task::JoinHandle<()>>,
}
impl Node {
async fn start(directory: &Path, network: NetworkId) -> Self {
let (service, events) = start_service(directory, network, true).await;
let acceptor = service.stream_acceptor(PODCAST_DHT_ALPN).unwrap();
let dht = PodcastDht::open(Arc::clone(&service), directory.join("podcast-dht.sqlite3"))
.await
.unwrap();
let tasks = vec![events, tokio::spawn(Arc::clone(&dht).serve(acceptor))];
Self {
service,
dht,
tasks,
}
}
async fn stop(self) {
for task in &self.tasks {
task.abort();
}
self.service.shutdown().await.unwrap();
}
async fn contact(&self) -> NodeContact {
let peer_id = self.service.endpoint_id();
NodeContact {
node_id: NodeId::from_endpoint(&peer_id),
peer_id,
ticket: self.service.ticket().await.unwrap().to_string(),
last_seen_ms: 1,
}
}
}
async fn start_service(
directory: &Path,
network: NetworkId,
overlay: bool,
) -> (Arc<MusicDhtService>, tokio::task::JoinHandle<()>) {
std::fs::create_dir_all(directory).unwrap();
let mut config = MusicDhtConfig::builder()
.data_dir(directory)
.network_id(network)
.request_timeout(Duration::from_secs(2))
.lookup_timeout(Duration::from_secs(5))
.transport_timeout(Duration::from_secs(5))
.dial_timeout(Duration::from_secs(2));
if overlay {
config = config.schema_independent_stream_protocol(PODCAST_DHT_ALPN);
}
let (service, mut events) = MusicDhtService::start(config.build().unwrap())
.await
.unwrap();
let events = tokio::spawn(async move { while events.recv().await.is_some() {} });
(Arc::new(service), events)
}
fn feed(path: &str) -> FeedUrl {
FeedUrl::parse(&format!("https://feeds.example.test/{path}")).unwrap()
}
/// A subscription of A's user (generation 1).
fn subscription(path: &str, title: &str) -> SubscriptionRecord {
let feed_url = feed(path);
SubscriptionRecord {
podcast_id: PodcastId::from_feed_url(&feed_url),
subscribed: true,
podcast: PodcastMeta {
feed_url,
title: title.to_string(),
author: Some("Ferris Crab".to_string()),
description: Some("Weekly news.".to_string()),
link: None,
image_url: None,
},
feed_aliases: Vec::new(),
generation: 1,
hlc_ms: 1_000,
op_id: "device-a-op-1".to_string(),
}
}
fn share(subscription: &SubscriptionRecord, shared: bool, generation: u64) -> ShareRecord {
ShareRecord {
podcast_id: subscription.podcast_id.clone(),
shared,
generation,
hlc_ms: 2_000 + generation as i64,
op_id: format!("device-a-share-{generation}"),
}
}
async fn wait_for<T>(what: &str, mut check: impl AsyncFnMut() -> Option<T>) -> T {
let started = std::time::Instant::now();
loop {
if let Some(value) = check().await {
return value;
}
assert!(
started.elapsed() < Duration::from_secs(60),
"timed out waiting for {what}"
);
tokio::time::sleep(Duration::from_millis(300)).await;
}
}
/// Sends raw bytes as one overlay request and decodes the response.
async fn raw_request(from: &Node, to: &Node, payload: &[u8]) -> PodcastDhtResponse {
let ticket = to.service.ticket().await.unwrap();
let mut stream = from
.service
.open_stream_to(&ticket, PODCAST_DHT_ALPN)
.await
.unwrap();
stream.send.write_all(payload).await.unwrap();
stream.send.finish().unwrap();
let mut bytes = Vec::new();
AsyncReadExt::read_to_end(&mut stream.recv, &mut bytes)
.await
.unwrap();
postcard::from_bytes(&bytes).unwrap()
}
async fn store_request(from: &Node, to: &Node, record: SignedPodcastRecord) -> PodcastDhtResponse {
let request = PodcastDhtRequest {
version: PODCAST_DHT_PROTOCOL_VERSION,
requester: from.contact().await,
operation: PodcastDhtOperation::StoreBatch {
records: vec![PodcastStoreEntry {
keys: record.dht_keys(),
record,
}],
},
};
raw_request(from, to, &postcard::to_stdvec(&request).unwrap()).await
}
async fn search(node: &Node, query: &str) -> Vec<PodcastSearchHit> {
node.dht.search(query, 10).await.unwrap()
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn shared_podcast_is_discoverable_by_strangers_and_retractable() {
tokio::time::timeout(TEST_TIMEOUT, scenario())
.await
.unwrap();
}
async fn scenario() {
let temp = tempfile::tempdir().unwrap();
let network = NetworkId::from_name("podcast-dht-integration");
let a = Node::start(&temp.path().join("a"), network).await;
let b = Node::start(&temp.path().join("b"), network).await;
let c = Node::start(&temp.path().join("c"), network).await;
// D runs the music DHT only.
let (d, d_events) = start_service(&temp.path().join("d"), network, false).await;
// B and C reach A through the ordinary music-DHT transport; C has no
// device-sync pairing with A or B.
b.service
.connect(a.service.ticket().await.unwrap())
.await
.unwrap();
c.service
.connect(a.service.ticket().await.unwrap())
.await
.unwrap();
d.connect(a.service.ticket().await.unwrap()).await.unwrap();
wait_for("music DHT contacts", async || {
(a.service.known_peers().len() >= 3
&& !b.service.known_peers().is_empty()
&& !c.service.known_peers().is_empty())
.then_some(())
})
.await;
for node in [&a, &b, &c] {
node.dht.bootstrap().await.unwrap();
}
wait_for("podcast overlay contacts", async || {
(a.dht.known_peers() >= 2 && c.dht.known_peers() >= 1).then_some(())
})
.await;
// D is dialed by bootstrap but never answers on the overlay ALPN, so it
// never becomes an overlay contact: it is never asked for records or to
// store them.
a.dht.bootstrap().await.unwrap();
assert_eq!(a.dht.known_peers(), 2);
// A capability manifest without podcast_dht evicts a peer; one with it
// lets the peer be learned again.
let manifest = CapabilityManifest::frid("furumi", "test").with_podcast_dht();
a.dht
.note_peer_capabilities(
c.service.endpoint_id(),
&manifest.clone().without_protocol(PODCAST_DHT_ID),
)
.await
.unwrap();
assert_eq!(a.dht.known_peers(), 1);
a.dht.bootstrap().await.unwrap();
assert_eq!(
a.dht.known_peers(),
1,
"the evicted peer is not bootstrapped"
);
a.dht
.note_peer_capabilities(c.service.endpoint_id(), &manifest)
.await
.unwrap();
a.dht.bootstrap().await.unwrap();
assert_eq!(a.dht.known_peers(), 2);
// A request with another overlay version is refused.
let request = PodcastDhtRequest {
version: PODCAST_DHT_PROTOCOL_VERSION + 1,
requester: c.contact().await,
operation: PodcastDhtOperation::FindNode { target: [0; 32] },
};
let response = raw_request(&c, &a, &postcard::to_stdvec(&request).unwrap()).await;
assert!(!response.ok);
// A's user shares "Rust Radio Weekly" but never shares "Secret Garden".
let shared = subscription("rust-radio.xml", "Rust Radio Weekly");
let private = subscription("secret-garden.xml", "Secret Garden Diaries");
let states = vec![
PodcastShareState {
subscription: shared.clone(),
share: Some(share(&shared, true, 2)),
card: None,
privacy_confirmed: false,
},
PodcastShareState {
subscription: private.clone(),
share: None,
card: None,
privacy_confirmed: false,
},
];
let stats = a.dht.sync_shares(states.clone()).await.unwrap();
assert_eq!(
(stats.published, stats.unchanged, stats.unpublished),
(1, 0, 0)
);
assert!(stats.push.local_replica);
assert!(stats.push.remote_nodes >= 1);
let error = a
.dht
.publish(
&private,
&share(&private, false, 2),
PodcastCardSpec::from_subscription(&private).unwrap(),
)
.await
.unwrap_err();
assert!(matches!(error, MusicDhtError::PodcastPublication(_)));
// Idempotent: nothing new is published.
let again = a.dht.sync_shares(states).await.unwrap();
assert_eq!(
(again.published, again.unchanged, again.unpublished),
(0, 1, 0)
);
let published = a.dht.published().await.unwrap();
assert_eq!(published.len(), 1);
let first_card = published[0].record.clone();
// B, the same user's other device, holds the same merged registers and
// publishes the same podcast under its own key; the private podcast
// stays private there too.
let b_states = vec![
PodcastShareState {
subscription: shared.clone(),
share: Some(share(&shared, true, 2)),
card: None,
privacy_confirmed: false,
},
PodcastShareState {
subscription: private.clone(),
share: None,
card: None,
privacy_confirmed: false,
},
];
let stats = b.dht.sync_shares(b_states).await.unwrap();
assert_eq!((stats.published, stats.unpublished), (1, 0));
let mut both = vec![a.service.endpoint_id(), b.service.endpoint_id()];
both.sort_by_key(|key| *key.as_bytes());
let hits = wait_for("C sees both publishers", async || {
let hits = search(&c, "rust radio").await;
(hits.len() == 1 && hits[0].publishers == both).then_some(hits)
})
.await;
assert_eq!(hits[0].feed_url, shared.podcast.feed_url);
assert!(hits[0].claimed_feed_urls.is_empty());
// B unshares (for example a stale register on B) and goes offline: B is
// not needed for discovery, A's card is still found.
let stats = b
.dht
.sync_shares(vec![PodcastShareState {
subscription: shared.clone(),
share: Some(share(&shared, false, 3)),
card: None,
privacy_confirmed: false,
}])
.await
.unwrap();
assert_eq!(stats.unpublished, 1);
let a_only = vec![a.service.endpoint_id()];
wait_for("B's tombstone reaches C's lookup", async || {
let hits = search(&c, "rust radio").await;
(hits.len() == 1 && hits[0].publishers == a_only).then_some(())
})
.await;
b.stop().await;
let hits = wait_for("C finds the card by text", async || {
let hits = search(&c, "rust radio").await;
(!hits.is_empty()).then_some(hits)
})
.await;
assert_eq!(hits.len(), 1);
assert_eq!(hits[0].podcast_id, shared.podcast_id);
assert_eq!(hits[0].publishers, vec![a.service.endpoint_id()]);
assert_eq!(hits[0].feed_url, shared.podcast.feed_url);
assert_eq!(hits[0].card.feed_url, shared.podcast.feed_url);
assert_eq!(hits[0].card.title, "Rust Radio Weekly");
assert_eq!(search(&c, "ferris weekly").await.len(), 1, "author tokens");
assert!(
search(&c, "rust jazz").await.is_empty(),
"every token must match"
);
let by_id = c.dht.lookup_podcast(&shared.podcast_id).await.unwrap();
assert_eq!(by_id.unwrap().card.podcast_id, shared.podcast_id);
// The never-shared podcast leaves no trace anywhere.
assert!(search(&c, "secret garden").await.is_empty());
assert!(search(&a, "secret garden").await.is_empty());
assert!(
c.dht
.lookup_podcast(&private.podcast_id)
.await
.unwrap()
.is_none()
);
assert!(
a.dht
.lookup_podcast(&private.podcast_id)
.await
.unwrap()
.is_none()
);
// Forged: C signs a card that claims A as publisher.
let forged_spec =
PodcastCardSpec::from_subscription(&subscription("forged.xml", "Forged Show")).unwrap();
let forged_card = PodcastCard::from_spec(
forged_spec.clone(),
network,
a.service.endpoint_id(),
u64::MAX - 1,
first_card.issued_at_ms(),
);
let forged = SignedPodcastRecord {
signature: c.service.sign_identity(&forged_card.signing_bytes()),
body: PodcastRecordBody::Card(Box::new(forged_card)),
};
let response = store_request(&c, &a, forged).await;
assert!(!response.ok && response.stored == 0, "{response:?}");
// Foreign: A's genuine card stored by C. Only publishers store records.
let response = store_request(&c, &a, first_card.clone()).await;
assert!(!response.ok && response.stored == 0, "{response:?}");
// Tampered: A's genuine card with a different description.
let mut tampered = first_card.clone();
if let PodcastRecordBody::Card(card) = &mut tampered.body {
card.description = Some("Now with malware".into());
}
let response = store_request(&c, &a, tampered).await;
assert!(!response.ok, "{response:?}");
// Malformed and oversized requests.
let response = raw_request(&c, &a, b"definitely not postcard").await;
assert!(!response.ok);
let response = raw_request(&c, &a, &vec![0u8; MAX_PODCAST_DHT_REQUEST_BYTES + 1]).await;
assert!(!response.ok);
assert!(response.error.unwrap().contains("size"));
// A valid card signed by C is accepted: the overlay is open to anyone
// who signs their own cards.
let own_card = PodcastCard::from_spec(
forged_spec,
network,
c.service.endpoint_id(),
1,
first_card.issued_at_ms(),
);
let own =
SignedPodcastRecord::sign_card(own_card, |bytes| c.service.sign_identity(bytes)).unwrap();
let response = store_request(&c, &a, own).await;
assert!(response.ok && response.stored > 0, "{response:?}");
let forged_hits = search(&a, "forged show").await;
assert_eq!(forged_hits.len(), 1);
assert_eq!(forged_hits[0].publishers, vec![c.service.endpoint_id()]);
// A's user stops sharing: a tombstone with the next revision replaces
// the card and C stops seeing it.
let unshare = vec![PodcastShareState {
subscription: shared.clone(),
share: Some(share(&shared, false, 3)),
card: None,
privacy_confirmed: false,
}];
let stats = a.dht.sync_shares(unshare).await.unwrap();
assert_eq!(stats.unpublished, 1);
let published = a.dht.published().await.unwrap();
assert!(!published[0].active);
assert!(published[0].record.is_tombstone());
assert_eq!(published[0].revision, first_card.revision() + 1);
wait_for("tombstone hides the card from C", async || {
search(&c, "rust radio").await.is_empty().then_some(())
})
.await;
assert!(
c.dht
.lookup_podcast(&shared.podcast_id)
.await
.unwrap()
.is_none()
);
// A replaying its own old card to a replica that stores the tombstone
// does not bring it back; nobody else may replay it at all.
let response = store_request(&a, &c, first_card.clone()).await;
assert!(response.ok && response.stored == 0, "{response:?}");
let response = store_request(&c, &a, first_card).await;
assert!(!response.ok, "{response:?}");
assert!(search(&c, "rust radio").await.is_empty());
assert!(search(&a, "rust radio").await.is_empty());
a.stop().await;
c.stop().await;
d_events.abort();
d.shutdown().await.unwrap();
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn sync_shares_is_fail_safe_and_holds_back_private_urls() {
tokio::time::timeout(TEST_TIMEOUT, single_node_scenario())
.await
.unwrap();
}
async fn single_node_scenario() {
let temp = tempfile::tempdir().unwrap();
let network = NetworkId::from_name("podcast-dht-sync-shares");
let a = Node::start(&temp.path().join("a"), network).await;
let state =
|subscription: &SubscriptionRecord, shared: bool, generation: u64| PodcastShareState {
subscription: subscription.clone(),
share: Some(share(subscription, shared, generation)),
card: None,
privacy_confirmed: false,
};
let first = subscription("first.xml", "First Show");
let second = subscription("second.xml", "Second Show");
let stats = a
.dht
.sync_shares(vec![state(&first, true, 2), state(&second, true, 2)])
.await
.unwrap();
assert_eq!(stats.published, 2);
assert!(stats.failed.is_empty() && stats.held_back.is_empty());
// In one call the user unshares the first podcast while the second
// one's card cannot be built (the URL its id derives from is unknown).
let mut broken = second.clone();
broken.podcast.feed_url = feed("elsewhere.xml");
let stats = a
.dht
.sync_shares(vec![state(&first, false, 3), state(&broken, true, 2)])
.await
.unwrap();
assert_eq!(stats.failed.len(), 1);
assert_eq!(stats.failed[0].podcast_id, second.podcast_id);
assert_eq!(stats.unpublished, 2, "the unshared and the broken podcast");
let published = a.dht.published().await.unwrap();
assert!(published.iter().all(|publication| !publication.active));
assert!(
published
.iter()
.all(|publication| publication.record.is_tombstone())
);
// Republishing re-sends tombstones only; nothing comes back.
a.dht.republish().await.unwrap();
assert!(search(&a, "first show").await.is_empty());
assert!(search(&a, "second show").await.is_empty());
// A feed URL that looks private is held back until the user confirms.
let mut member = subscription("member.xml?token=abc", "Member Show");
member.podcast_id = PodcastId::from_feed_url(&member.podcast.feed_url);
let stats = a
.dht
.sync_shares(vec![state(&member, true, 2)])
.await
.unwrap();
assert_eq!(stats.published, 0);
assert_eq!(stats.held_back.len(), 1);
assert_eq!(stats.held_back[0].podcast_id, member.podcast_id);
assert!(!stats.held_back[0].warnings.is_empty());
assert!(search(&a, "member show").await.is_empty());
let mut confirmed = state(&member, true, 2);
confirmed.privacy_confirmed = true;
let stats = a.dht.sync_shares(vec![confirmed]).await.unwrap();
assert_eq!(stats.published, 1);
assert_eq!(search(&a, "member show").await.len(), 1);
// Once published with the user's consent, the same URLs are kept.
let stats = a
.dht
.sync_shares(vec![state(&member, true, 2)])
.await
.unwrap();
assert_eq!((stats.unchanged, stats.held_back.len()), (1, 0));
a.stop().await;
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn fabricated_contacts_never_join_the_overlay() {
tokio::time::timeout(TEST_TIMEOUT, fabricated_contacts_scenario())
.await
.unwrap();
}
/// K contacts whose node ids are genuine hashes of their endpoint ids but
/// whose tickets carry no address, as a malicious peer could return them.
fn fabricated_contacts(network: NetworkId) -> Vec<NodeContact> {
(0..music_dht::K)
.map(|index| {
let peer = federation_net::SecretKey::from_bytes(&[index as u8 + 101; 32]).public();
let ticket = federation_net::PeerTicket {
ticket_version: federation_net::TICKET_VERSION,
protocol_version: 1,
network_id: network,
schema_id: federation_net::SchemaId::from_bytes([0; 32]),
endpoint_addr: federation_net::EndpointAddr::new(peer),
};
NodeContact {
node_id: NodeId::from_endpoint(&peer),
peer_id: peer,
ticket: ticket.to_string(),
last_seen_ms: u64::MAX,
}
})
.collect()
}
async fn fabricated_contacts_scenario() {
let temp = tempfile::tempdir().unwrap();
let network = NetworkId::from_name("podcast-dht-fabricated-contacts");
let a = Node::start(&temp.path().join("a"), network).await;
// M speaks the overlay ALPN but answers every request with fabricated
// contacts.
let (m, m_events) = start_service(&temp.path().join("m"), network, true).await;
let mut acceptor = m.stream_acceptor(PODCAST_DHT_ALPN).unwrap();
let fabricated = fabricated_contacts(network);
let response = PodcastDhtResponse {
version: PODCAST_DHT_PROTOCOL_VERSION,
ok: true,
error: None,
nodes: fabricated.clone(),
records: Vec::new(),
stored: 0,
};
let payload = postcard::to_stdvec(&response).unwrap();
let server = tokio::spawn(async move {
while let Some(mut stream) = acceptor.accept().await {
let mut request = Vec::new();
let _ = AsyncReadExt::read_to_end(
&mut (&mut stream.recv).take(MAX_PODCAST_DHT_REQUEST_BYTES as u64),
&mut request,
)
.await;
let _ = stream.send.write_all(&payload).await;
let _ = stream.send.finish();
let _ = tokio::time::timeout(Duration::from_secs(2), stream.send.stopped()).await;
}
});
m.connect(a.service.ticket().await.unwrap()).await.unwrap();
wait_for("music DHT contact", async || {
(!a.service.known_peers().is_empty()).then_some(())
})
.await;
// Bootstrap learns M, which answered, and none of the contacts it made
// up: they are tried once and never answer.
a.dht.bootstrap().await.unwrap();
assert_eq!(a.dht.known_peers(), 1);
// A lookup treats them as candidates only.
let podcast_id = PodcastId::from_feed_url(&feed("nothing.xml"));
assert!(a.dht.lookup_podcast(&podcast_id).await.unwrap().is_none());
assert_eq!(a.dht.known_peers(), 1);
a.dht.bootstrap().await.unwrap();
assert_eq!(
a.dht.known_peers(),
1,
"fabricated contacts never count for replica responsibility"
);
server.abort();
a.stop().await;
m_events.abort();
m.shutdown().await.unwrap();
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn restarted_device_waits_for_sync_shares_before_resending_cards() {
tokio::time::timeout(TEST_TIMEOUT, restart_scenario())
.await
.unwrap();
}
async fn restart_scenario() {
let temp = tempfile::tempdir().unwrap();
let network = NetworkId::from_name("podcast-dht-restart");
let state =
|subscription: &SubscriptionRecord, shared: bool, generation: u64| PodcastShareState {
subscription: subscription.clone(),
share: Some(share(subscription, shared, generation)),
card: None,
privacy_confirmed: false,
};
let kept = subscription("kept.xml", "Kept Show");
let revoked = subscription("revoked.xml", "Revoked Show");
// A publishes both podcasts while alone, then goes offline.
let a = Node::start(&temp.path().join("a"), network).await;
let stats = a
.dht
.sync_shares(vec![state(&kept, true, 2), state(&revoked, true, 2)])
.await
.unwrap();
assert_eq!((stats.published, stats.push.remote_nodes), (2, 0));
a.stop().await;
// Meanwhile another trusted device revokes "Revoked Show". A comes back
// with its persisted cards and runs maintenance before its client
// reconciled: the first bootstrap finds B.
let b = Node::start(&temp.path().join("b"), network).await;
let mut a = Node::start(&temp.path().join("a"), network).await;
b.service
.connect(a.service.ticket().await.unwrap())
.await
.unwrap();
wait_for("music DHT contacts", async || {
(!a.service.known_peers().is_empty()).then_some(())
})
.await;
a.tasks.push(tokio::spawn(Arc::clone(&a.dht).maintenance()));
wait_for("overlay contacts", async || {
(a.dht.known_peers() >= 1 && b.dht.known_peers() >= 1).then_some(())
})
.await;
// Neither maintenance nor an explicit republish sends a stale card, and
// A does not serve them to lookups either.
let stats = a.dht.republish().await.unwrap();
assert_eq!(stats.records, 0, "stale cards are withheld");
for podcast in [&kept, &revoked] {
assert!(
b.dht
.lookup_podcast(&podcast.podcast_id)
.await
.unwrap()
.is_none()
);
}
assert!(search(&b, "show").await.is_empty());
// The client's first reconciliation revokes the stale share and
// re-sends the unchanged card.
let stats = a
.dht
.sync_shares(vec![state(&kept, true, 2), state(&revoked, false, 3)])
.await
.unwrap();
assert_eq!(
(stats.published, stats.unchanged, stats.unpublished),
(0, 1, 1)
);
assert_eq!(stats.push.records, 2, "the kept card and the tombstone");
assert!(stats.push.remote_nodes >= 1);
let a_only = vec![a.service.endpoint_id()];
let hit = b
.dht
.lookup_podcast(&kept.podcast_id)
.await
.unwrap()
.expect("the kept card is found");
assert_eq!(hit.publishers, a_only);
assert!(
b.dht
.lookup_podcast(&revoked.podcast_id)
.await
.unwrap()
.is_none()
);
// From now on republishing covers the kept card and the tombstone.
let stats = a.dht.republish().await.unwrap();
assert_eq!(stats.records, 2);
let hits = search(&b, "show").await;
assert_eq!(hits.len(), 1);
assert_eq!(hits[0].podcast_id, kept.podcast_id);
a.stop().await;
b.stop().await;
}