Подкасты в FRID: публичное обнаружение podcast_dht — подписанные карточки, tombstone и поиск (MR-F3) #3
No files matched your search
+38
-2
@@ -232,8 +232,44 @@ refused, and their playback commands are ignored while the queue contains
|
||||
episodes. Jam snapshots replace episodes with opaque placeholders. Progress,
|
||||
subscription and share registers order by a Lamport generation before
|
||||
wall-clock `hlc_ms`, so a device with a fast clock cannot undo an observed
|
||||
seek, restart, unsubscribe or share decision. Public podcast discovery is a future
|
||||
separate overlay; the music DHT schema and `federation-net` are unchanged.
|
||||
seek, restart, unsubscribe or share decision. The music DHT schema and
|
||||
`federation-net` are unchanged.
|
||||
|
||||
Public podcast discovery is a third schema-independent Kademlia overlay,
|
||||
`music_dht::podcast_dht` (ALPN `furumi-fd/podcast-dht/1`, version 1,
|
||||
capability `podcast_dht`), built like the similarity-routing overlay. A
|
||||
device publishes a card only for podcasts whose merged registers satisfy
|
||||
`effective_share` (SQLite clients read the merged registers from
|
||||
`PodcastStore`); `sync_shares` reconciles publications declaratively and
|
||||
nothing is sent for a podcast the device never published. Own cards persisted
|
||||
by an earlier run are neither republished nor served until the first
|
||||
`sync_shares` of the process, so a stale cache cannot revive a share revoked
|
||||
elsewhere while the device was offline. A card is the
|
||||
public projection of a podcast (feed URL and channel metadata, no personal
|
||||
state), signed by the device's iroh identity over a domain-separated
|
||||
canonical encoding. Only the origin URL the id hashes from is bound to the
|
||||
podcast; a moved `feed_url` is an unverified publisher claim that search never
|
||||
offers as the podcast's feed. Cards live under the podcast-id key and one key
|
||||
per normalized title/author token; unpublishing writes a signed tombstone of
|
||||
the next revision that supersedes lower and equal revisions on replicas that
|
||||
hold it, for the keys it lists, and outlives every older card. Only a record's
|
||||
publisher may store it; replicas keep one payload per publisher and podcast,
|
||||
admit placements only for keys they are close to (counting only contacts that
|
||||
answered or authenticated, never ones another peer suggested), within per-key,
|
||||
per-publisher and total budgets. A key holds no more records than one response
|
||||
carries; a full key or cache admits a newcomer by evicting a random record from
|
||||
its younger half, so the older half survives floods and capacity is never
|
||||
first-come-forever (over their budget, tombstones only displace other tombstones). Records use the music DHT's
|
||||
active/tombstone TTLs and republish cadence, are verified on every receive and
|
||||
cache read, and are kept in a separate, replaceable SQLite cache. Search
|
||||
filters on every replica by all query tokens, merges cards of one podcast
|
||||
across publisher devices, and returns typed `podcast` hits that clients merge
|
||||
with music results; publisher counts are not a trust signal. The publisher key
|
||||
is the device's transport identity, so a card is linkable to the device's
|
||||
network address and music-DHT records. Applications advertise the
|
||||
`podcast_dht` capability only when they run the overlay; capability data only
|
||||
helps skip peers, it never authorizes. See the "Public discovery" section of
|
||||
[PODCASTS.md](PODCASTS.md).
|
||||
|
||||
Music discovery identifies an owner and content; it does not force all domain
|
||||
traffic through DHT messages. `music-dht` exposes the underlying authenticated
|
||||
|
||||
@@ -1,5 +1,43 @@
|
||||
# Changelog
|
||||
|
||||
## Unreleased — podcast discovery (`podcast_dht`)
|
||||
|
||||
- `music-dht 0.6.0`: new `music_dht::podcast_dht` overlay for public podcast
|
||||
discovery on its own ALPN `furumi-fd/podcast-dht/1` (version 1) and its own
|
||||
SQLite cache (schema 2): signed `PodcastCard`s and `PodcastTombstone`s
|
||||
(`furumi-podcast-card:v1` / `furumi-podcast-card-tombstone:v1` canonical
|
||||
signing input), monotonic revisions with a total tie-break, text and id
|
||||
keys, `PodcastDht::{publish, unpublish, sync_shares, republish, search,
|
||||
lookup_podcast}` gated by `effective_share`, typed search hits
|
||||
(`PodcastSearchHit`, `ClaimedFeedUrl`, `FederatedSearchHit`,
|
||||
`SearchHitKind`, `merge_search_results`) and the advisory
|
||||
`feed_url_privacy_warning`.
|
||||
- Replicas accept records only from their publisher, store one payload per
|
||||
`(podcast_id, publisher)`, admit placements only for keys they are close to
|
||||
(counting verified contacts only) and within per-key, per-publisher and
|
||||
total budgets. A key never holds more than one response carries; full keys
|
||||
and caches admit newcomers by evicting random records from their younger
|
||||
half. Lookups give each response a bounded share of owners. Search offers
|
||||
only the URL the podcast id hashes from; moved-feed URLs are reported as
|
||||
unverified claims.
|
||||
- `sync_shares` is fail-safe for revocation (per-podcast `failed`) and holds
|
||||
back cards with privacy warnings until confirmed (`held_back`,
|
||||
`PodcastShareState::privacy_confirmed`).
|
||||
- Own cards persisted by an earlier run are neither republished (by
|
||||
`maintenance` or `republish`) nor served to lookups until the first
|
||||
`sync_shares` of the process, so a share revoked on another trusted device
|
||||
while this one was offline is not re-sent on startup. That first call also
|
||||
re-sends unchanged cards; tombstones are always re-sent.
|
||||
- `music-dht 0.6.0`: capability manifests can advertise `podcast_dht = 1`
|
||||
(`capabilities::PODCAST_DHT_ID`, `CapabilityManifest::with_podcast_dht`,
|
||||
`CapabilityManifest::protocol_version`,
|
||||
`CapabilityManifest::supports_podcast_dht`). It is opt-in:
|
||||
`CapabilityManifest::frid()` does not include it.
|
||||
- `music-dht 0.6.0`: new `MusicDhtError::PodcastPublication` variant (the enum
|
||||
is `#[non_exhaustive]`).
|
||||
- The music DHT (`music-dht-v5`), `federation-net` and device-sync protocol v4
|
||||
are unchanged. See "Public discovery (podcast_dht)" in `PODCASTS.md`.
|
||||
|
||||
## Unreleased — podcast contract
|
||||
|
||||
- `music-dht 0.6.0` (breaking): new `music_dht::podcast` module with the shared
|
||||
|
||||
+379
-7
@@ -10,8 +10,9 @@ live in `music_dht::podcast` (pure types and functions) and
|
||||
`music_dht::device_sync` (wire DTOs and gating helpers). Clients must use
|
||||
them instead of local copies.
|
||||
|
||||
Public discovery (signed podcast cards in a separate DHT overlay) is not part
|
||||
of this version of the contract; it will be specified with the overlay.
|
||||
Public discovery of podcasts a user explicitly shared uses signed podcast
|
||||
cards in a separate DHT overlay, `music_dht::podcast_dht`; see
|
||||
[Public discovery (podcast_dht)](#public-discovery-podcast_dht).
|
||||
|
||||
## Responsibilities
|
||||
|
||||
@@ -19,7 +20,8 @@ of this version of the contract; it will be specified with the overlay.
|
||||
| --- | --- |
|
||||
| `music_dht::podcast` | ids, URL canonicalization, no-GUID rule, metadata DTOs and bounds, merge functions, RSS/Atom parser (feature `podcast-feed`) |
|
||||
| `music_dht::device_sync` | sync operation kinds, snapshot sections, `PlaybackSource`, version gating, Jam redaction |
|
||||
| Client storage (`furumi_library::podcasts::PodcastStore`, web PostgreSQL) | persistence and migrations implementing the rules below (SQLite reference: see "SQLite reference store") |
|
||||
| `music_dht::podcast_dht` | public podcast cards and tombstones, their signatures, replication, cache and search, privacy warnings for feed URLs |
|
||||
| Client storage (`furumi_library::podcasts::PodcastStore`, web PostgreSQL) | persistence and migrations implementing the rules below (SQLite reference: [SQLite reference store](#sqlite-reference-store)) |
|
||||
| Client runtime | HTTP fetching, conditional requests, scheduling, audio buffering and decoding, UI |
|
||||
|
||||
Audio is always fetched from the publisher. It is never imported into the
|
||||
@@ -32,7 +34,7 @@ offered for P2P transfer. RSS refreshes fetch metadata only.
|
||||
| --- | --- | --- | --- |
|
||||
| Subscription state, channel metadata | `podcast_subscription_set` + snapshot | never | |
|
||||
| Feed aliases | `podcast_feed_aliases_added`, `podcast_subscription_set` + snapshot | never | |
|
||||
| Share decision | `podcast_share_set` + snapshot | only the published card (future overlay) | |
|
||||
| Share decision | `podcast_share_set` + snapshot | only the signed card while `effective_share` holds (`podcast_dht`) | |
|
||||
| Episode progress | `podcast_progress_set` + snapshot | never | |
|
||||
| Listening sessions, play counts | `podcast_listen_recorded` + snapshot | never | |
|
||||
| Episode catalog | each device parses RSS itself; `PodcastEpisodeRef` travels with progress and queue items | never | |
|
||||
@@ -498,6 +500,371 @@ Jam participants are outside the trusted-device group.
|
||||
private item. States that drop, duplicate or inject private items are
|
||||
rejected. Participants may still pause, seek and reorder music.
|
||||
|
||||
## Public discovery (podcast_dht)
|
||||
|
||||
`music_dht::podcast_dht` publishes podcasts the user explicitly shared so
|
||||
that peers outside the user's trusted-device group can find and subscribe to
|
||||
them. It is a separate, schema-independent Kademlia overlay built like
|
||||
`similarity_dht`: its own ALPN, version, routing contacts and SQLite cache.
|
||||
The music DHT (`music-dht-v5`) and `federation-net` are unchanged. Every
|
||||
device of a user publishes under its own identity; search merges the cards
|
||||
of one podcast by `podcast_id`.
|
||||
|
||||
| Item | Value |
|
||||
| --- | --- |
|
||||
| ALPN | `furumi-fd/podcast-dht/1` (schema-independent stream protocol) |
|
||||
| Wire and record version | `PODCAST_DHT_PROTOCOL_VERSION = 1` |
|
||||
| Capability manifest | `podcast_dht = 1`, opt-in (`CapabilityManifest::with_podcast_dht`, `supports_podcast_dht`) |
|
||||
| Cache | separate SQLite file, `PRAGMA user_version = 2`; a newer schema is refused, the pre-release version 1 replica table is dropped |
|
||||
| Card TTL | `ACTIVE_RECORD_TTL` of the music DHT (12 h) |
|
||||
| Tombstone TTL | `TOMBSTONE_TTL` of the music DHT (24 h) |
|
||||
| Republish | `DEFAULT_REPUBLISH_INTERVAL` (10 min), expiry sweep `DEFAULT_EXPIRE_INTERVAL` (1 min) |
|
||||
| Clock skew | `issued_at_ms` at most 5 min in the future |
|
||||
|
||||
### Card
|
||||
|
||||
A card is the public projection of one podcast by one publisher device. It
|
||||
is built from the subscription with `PodcastCardSpec::from_subscription`,
|
||||
which reads only the podcast id, the feed URLs and the channel metadata. It
|
||||
never contains progress, listening history, play counts, subscription or
|
||||
share registers (`generation`, `hlc_ms`, `op_id`), device names, local paths,
|
||||
HTTP validators or episodes.
|
||||
|
||||
| Field | Bound |
|
||||
| --- | --- |
|
||||
| `version`, `network_id`, `publisher` | 1, the federation network, the device's iroh endpoint key |
|
||||
| `podcast_id` | must equal `PodcastId::from_feed_url(id_feed_url or feed_url)` |
|
||||
| `feed_url` | canonical `FeedUrl` the publisher fetches the feed from |
|
||||
| `id_feed_url` | the URL the id was derived from, present only when it differs from `feed_url` (feed moved) |
|
||||
| `title` | 1..300 characters, no control characters |
|
||||
| `author` | optional, ≤ 300 characters, no control characters |
|
||||
| `description` | optional, ≤ 1 000 characters, only `\n`/`\t` controls |
|
||||
| `image_url`, `link` | optional absolute `http(s)` URLs ≤ 2 048 bytes |
|
||||
| `language` | optional, ≤ 35 bytes of ASCII letters, digits and `-` |
|
||||
| `categories` | ≤ 8, each 1..64 characters, no control characters |
|
||||
| `episode_count`, `latest_episode_at_ms` | optional; the time is ≥ 0 |
|
||||
| `revision` | ≥ 1, monotonic per `(podcast_id, publisher)` |
|
||||
| `issued_at_ms`, `expires_at_ms` | `0 < issued < expires ≤ issued + 12 h` |
|
||||
|
||||
`PodcastCardSpec::bounded` truncates publisher data to these bounds. The
|
||||
whole signed record is at most 24 KiB postcard-encoded; the field bounds keep
|
||||
every valid card below it.
|
||||
|
||||
Only the **origin URL** (`PodcastCard::origin_feed_url`: `id_feed_url`, or
|
||||
`feed_url` when the feed has not moved) is bound to the podcast, because the
|
||||
id is its hash. When `id_feed_url` is present, `feed_url` is an unverified
|
||||
claim of the publisher that the feed moved: anyone can sign a card that
|
||||
carries an existing podcast's origin URL and an arbitrary `feed_url`. Search
|
||||
therefore never offers a claimed URL as the podcast's feed (see Search), and
|
||||
clients must not fetch it as that podcast or record it as an alias unless the
|
||||
feed at the origin URL itself redirects there.
|
||||
|
||||
### Tombstone
|
||||
|
||||
`(version, network_id, publisher, podcast_id, revision, issued_at_ms,
|
||||
expires_at_ms, index_keys)` with `expires_at_ms ≤ issued_at_ms + 24 h` and
|
||||
`index_keys` the sorted, distinct text keys of the retracted card (at most
|
||||
32, without the podcast-id key). It carries key hashes, not the title. A
|
||||
tombstone supersedes every card of the same `(podcast_id, publisher)` with a
|
||||
lower or equal revision and is itself retained and served until it expires.
|
||||
Because the tombstone TTL exceeds the card TTL, every card issued before a
|
||||
tombstone expires before it.
|
||||
|
||||
`index_keys` is self-asserted, so it never creates a placement by itself: a
|
||||
replica indexes a tombstone under a text key only in place of a record of the
|
||||
same `(podcast_id, publisher)` it already holds there (see Replica storage).
|
||||
|
||||
### Signing input
|
||||
|
||||
Both records are signed with the publisher's Ed25519 iroh identity over a
|
||||
canonical, language-independent encoding: the domain string, a `0x00` byte,
|
||||
then the fields in the order listed. Integers are little-endian (`u16`,
|
||||
`u32`, `u64`, `i64`); strings are a `u32` byte length followed by UTF-8;
|
||||
`Option` is `0x00`, or `0x01` followed by the value; lists are a `u32` count
|
||||
followed by the items; `network_id`, `publisher` and keys are their raw 32
|
||||
bytes.
|
||||
|
||||
* Card, domain `furumi-podcast-card:v1`: `version u16, network_id, publisher,
|
||||
podcast_id str, feed_url str, id_feed_url opt str, title str, author opt
|
||||
str, description opt str, image_url opt str, link opt str, language opt
|
||||
str, categories list of str, episode_count opt u32, latest_episode_at_ms
|
||||
opt i64, revision u64, issued_at_ms u64, expires_at_ms u64`.
|
||||
* Tombstone, domain `furumi-podcast-card-tombstone:v1`: `version u16,
|
||||
network_id, publisher, podcast_id str, revision u64, issued_at_ms u64,
|
||||
expires_at_ms u64, index_keys list of 32-byte keys`.
|
||||
|
||||
Unit tests pin BLAKE3 vectors of both encodings. On the wire, records are
|
||||
postcard `SignedPodcastRecord { body: Card | Tombstone, signature }`.
|
||||
|
||||
### Keys and placement
|
||||
|
||||
`key = BLAKE3("furumi-podcast-dht:key:v1" || 0x00 || network_id || kind ||
|
||||
material)` with kind `0x01` and the podcast id string for the id key, or
|
||||
kind `0x02` and a token for a text key. Tokens are the distinct
|
||||
`normalize_name` tokens of the title, then of the author, skipping tokens
|
||||
longer than 64 bytes, at most 32. A card belongs under the id key and its
|
||||
text keys (at most 33); a tombstone under the id key and its `index_keys`.
|
||||
The publisher sends each record to the `K` (8) overlay nodes closest to each
|
||||
of its keys, one `StoreBatch` per node listing the keys that node should hold.
|
||||
|
||||
### Ordering
|
||||
|
||||
For one `(podcast_id, publisher)` the greater record wins: higher
|
||||
`revision`; at an equal revision a tombstone beats a card; then later
|
||||
`issued_at_ms` (a republish); then later `expires_at_ms`; then the
|
||||
lexicographically greater signing input. This is a total order, so every
|
||||
replica and every lookup converge whatever the delivery order. A replica
|
||||
keeps only the winning record of each `(podcast_id, publisher)`; a replayed
|
||||
old revision is ignored. These fields only order one publisher's own records:
|
||||
they never rank or select records of different publishers.
|
||||
|
||||
Revisions: a changed card gets `max(previous + 1, now_ms)` (time-seeded, so
|
||||
revisions stay monotonic even if the local cache is lost); an unchanged card
|
||||
is re-signed with the same revision and a fresh lifetime on every republish;
|
||||
unpublishing writes a tombstone with `previous + 1`; publishing again later
|
||||
gets `max(tombstone + 1, now_ms)`.
|
||||
|
||||
### Wire protocol
|
||||
|
||||
One request per bidirectional stream on `furumi-fd/podcast-dht/1`: the
|
||||
requester writes one postcard `PodcastDhtRequest` and finishes its send side;
|
||||
the responder writes one postcard `PodcastDhtResponse` and finishes. Messages
|
||||
are not length-prefixed; the stream end delimits them. Requests above
|
||||
512 KiB and responses above 4 MiB are refused.
|
||||
|
||||
```text
|
||||
PodcastDhtRequest { version: u16, requester: NodeContact, operation: PodcastDhtOperation }
|
||||
PodcastDhtOperation = 0: FindNode { target: [u8; 32] }
|
||||
| 1: FindValue { key: PodcastDhtKey, filter_tokens: Vec<String> }
|
||||
| 2: StoreBatch { records: Vec<PodcastStoreEntry> }
|
||||
PodcastStoreEntry { keys: Vec<PodcastDhtKey>, record: SignedPodcastRecord }
|
||||
PodcastDhtResponse { version: u16, ok: bool, error: Option<String>,
|
||||
nodes: Vec<NodeContact>, records: Vec<SignedPodcastRecord>, stored: u32 }
|
||||
NodeContact { node_id: [u8; 32], peer_id: EndpointId, ticket: String, last_seen_ms: u64 }
|
||||
PodcastRecordBody = 0: Card(PodcastCard) | 1: Tombstone(PodcastTombstone)
|
||||
```
|
||||
|
||||
Fields are in postcard order. `PodcastDhtKey` is 32 raw bytes.
|
||||
|
||||
* `version` must be 1 in both directions. `requester.peer_id` must equal the
|
||||
authenticated transport peer, `node_id` must be `NodeId::from_endpoint`
|
||||
of it and its ticket must name the same endpoint and network; otherwise the
|
||||
request is refused. The responder learns the requester as an overlay
|
||||
contact (it is authenticated).
|
||||
* `nodes` in a response are unverified suggestions. A requester treats them
|
||||
as lookup or bootstrap candidates only and adds one to its routing table
|
||||
(and persisted contacts) only after it answered a request itself.
|
||||
* `FindNode` returns the `K` closest overlay contacts (the responder
|
||||
included).
|
||||
* `FindValue` returns the records under `key` (see Replica storage) and the
|
||||
`K` closest contacts. `filter_tokens` (≤ 8, each 1..64 bytes) are
|
||||
normalized tokens every returned card must be indexed under; tombstones are
|
||||
returned regardless. Requesters verify every record again and drop records
|
||||
that do not belong under `key` or miss a filter token.
|
||||
* `StoreBatch` carries 1..16 entries, each with 1..33 distinct keys. Every
|
||||
record must be published by the requester itself (replicas never forward
|
||||
records), pass `verify_at` (bounds, network, lifetime, clock skew,
|
||||
signature, size) and belong under each listed key; otherwise the whole
|
||||
batch is refused. Keys the responder is not responsible for are skipped.
|
||||
`stored` counts the `(key, record)` placements written. A publisher
|
||||
offers the placements of a batch answered with `stored = 0` (or that
|
||||
failed) once to the next closest node of each key.
|
||||
* A refused request gets `ok = false` with an `error` of at most 1 KiB and
|
||||
empty `nodes`, `records` and `stored = 0`; an accepted one has `ok = true`
|
||||
and no `error`. Requesters reject responses that violate this or exceed
|
||||
`K` nodes or 128 records.
|
||||
|
||||
### Replica storage
|
||||
|
||||
* A replica stores one payload per `(podcast_id, publisher)` (the winning
|
||||
record) and indexes it under the keys it holds. A newer version replaces
|
||||
it in place; placements the new version no longer belongs under (an old
|
||||
title's tokens, keys a tombstone does not list) are dropped, so on one
|
||||
replica a tombstone supersedes the card under every key it held.
|
||||
* A node accepts a placement only when fewer than `2 * K` of its verified
|
||||
overlay contacts that are not backed off are closer to the key than
|
||||
itself. Contacts only enter the routing table by answering a request or
|
||||
by sending one, so a peer cannot make a replica refuse keys by returning
|
||||
made-up contacts. A contact that fails three times in a row (each after
|
||||
its 30 s backoff) is removed.
|
||||
* A key holds at most 128 records, at most 32 of them tombstones: exactly
|
||||
what one `FindValue` response carries, so every admitted record is served.
|
||||
A tombstone is indexed under a text key only in place of its owner's
|
||||
record there, and such an in-place conversion counts against the
|
||||
tombstone budget like a new tombstone.
|
||||
* Full key: a new placement evicts a random placement of another owner from
|
||||
the younger half of the key (by first indexing time; only tombstones when
|
||||
the tombstone budget is full). The older half is never evicted for a
|
||||
newcomer, so records placed before a flood keep their place and their
|
||||
publishers refresh them in place. Capacity is never first-come-forever:
|
||||
a newcomer always gets a place, though placements in the younger half can
|
||||
be evicted again by later newcomers.
|
||||
* A replica holds at most 256 records per publisher (a publisher over it is
|
||||
refused), 4 096 records and 32 MiB of payload. A full cache admits a new
|
||||
record by evicting random records of other owners from its younger half,
|
||||
at most 16; when that is not enough room, nothing is evicted and the record
|
||||
is refused. At most 1 024 overlay contacts are persisted.
|
||||
* Device identities cost nothing, so these rules bound flooding but cannot
|
||||
prevent it: an attacker with many identities can keep part of the younger
|
||||
half of a key or cache busy, and a newcomer's place there can be evicted
|
||||
again until its publisher's next republish (every 10 min) puts it back.
|
||||
Protocol 1 has no proof of work or other identity cost.
|
||||
* `FindValue` answers with at most 128 records: at most 32 tombstones and
|
||||
the cards indexed longest first. Every record is verified again on every
|
||||
read.
|
||||
|
||||
### Publication
|
||||
|
||||
* `PodcastDht::publish(subscription, share, card)` refuses
|
||||
(`MusicDhtError::PodcastPublication`) unless `effective_share` holds, so a
|
||||
new or resumed subscription is never published without a fresh share
|
||||
decision. The UI calls it only after asking the user about any privacy
|
||||
warning, so the call is the user's consent.
|
||||
* `PodcastDht::sync_shares(states)` is declarative and idempotent: podcasts
|
||||
for which `effective_share` holds are published (when changed, new or past
|
||||
half their lifetime), every other podcast this device published is
|
||||
unpublished. Clients call it after applying share or subscription
|
||||
operations, so unsharing or unsubscribing on any trusted device revokes
|
||||
every device's own card once the registers converge, and a stale device
|
||||
cannot republish (its registers lose to the newer share decision).
|
||||
* Revocation is fail-safe: each podcast is handled on its own. A shared
|
||||
podcast whose card cannot be built or signed is reported in
|
||||
`PodcastDhtSyncStats::failed` and treated as not shared (an earlier
|
||||
publication is tombstoned); it never prevents other podcasts from being
|
||||
unpublished.
|
||||
* A card whose `PodcastCardSpec::privacy_warnings` is not empty is held back
|
||||
(`PodcastDhtSyncStats::held_back`, with the warnings) unless
|
||||
`PodcastShareState::privacy_confirmed` is set or this device already
|
||||
publishes the same `feed_url` and `id_feed_url`. A feed move to a URL that
|
||||
looks private therefore asks again; an earlier publication of a held-back
|
||||
podcast stays as it was.
|
||||
* SQLite clients build each `PodcastShareState` from
|
||||
`PodcastStore::subscription` (the merged register with its alias set, which
|
||||
always contains the origin URL) and `PodcastStore::share`; its `is_shared`
|
||||
equals `PodcastSummary::shared`. A card for the URL the device now fetches
|
||||
sets `feed_url` from the catalog and `id_feed_url` to the origin URL. The
|
||||
store does not persist `privacy_confirmed`; the client keeps that
|
||||
confirmation. See [SQLite reference store](#sqlite-reference-store).
|
||||
* `PodcastDht::unpublish(podcast_id)` writes the tombstone. Nothing is ever
|
||||
sent for a podcast this device never published: no tombstone, no id.
|
||||
* `PodcastDht::republish` (run by `maintenance` every 10 minutes and after
|
||||
a bootstrap that found new peers) re-signs active cards and re-sends
|
||||
unexpired tombstones.
|
||||
* Own cards persisted by an earlier run are not sent again until the client
|
||||
calls `sync_shares` in this process: before that, `republish` sends only
|
||||
tombstones and cards published with `publish` in this process, and
|
||||
`FindValue` answers leave those cards out (they stay in the cache). A share
|
||||
revoked on another trusted device while this one was offline is therefore
|
||||
never re-sent on startup from the stale cache. Clients call `sync_shares`
|
||||
on start, with the registers they hold, and again after device sync
|
||||
applies newer operations; the first call also re-sends the unchanged cards
|
||||
as they are (`PodcastDhtSyncStats::unchanged`, counted in `push`).
|
||||
`sync_shares` itself still trusts the registers it is given: an unchanged
|
||||
card is only re-signed when it used half its lifetime, so clients that may
|
||||
hold stale registers on start lose nothing by reconciling again after
|
||||
device sync.
|
||||
* The privacy heuristics (`feed_url_privacy_warning`) are advisory: query
|
||||
parameter names containing words such as `token`, `key`, `auth`, `secret`,
|
||||
`sig`/`signature`, `uid`, `user`, `pass`, `session`, `access`, `private`;
|
||||
known member-feed hosts; and long high-entropy query values (≥ 16
|
||||
characters) or path segments (≥ 32 characters). They check `feed_url` and
|
||||
`id_feed_url`, not artwork or web-page links. False positives and misses
|
||||
are possible.
|
||||
|
||||
### Search
|
||||
|
||||
* `PodcastDht::search(query, limit)` normalizes the query
|
||||
(`podcast_query_tokens`: distinct tokens, at most 8), looks up the text
|
||||
keys of the three longest tokens and asks every replica to return only
|
||||
cards indexed under **all** query tokens (tombstones are always returned).
|
||||
Results are verified again, the winning record per publisher is kept,
|
||||
tombstoned publishers are dropped and cards are merged by `podcast_id`
|
||||
into `PodcastSearchHit { podcast_id, feed_url, card, publishers,
|
||||
claimed_feed_urls }`.
|
||||
* `PodcastDht::lookup_podcast(podcast_id)` reads the id key.
|
||||
* `feed_url` is the origin URL; it is the only URL a client subscribes with.
|
||||
`claimed_feed_urls` lists other `feed_url`s that publishers claim (at most
|
||||
8, with their publishers), for display only.
|
||||
* The representative `card` is a card whose `feed_url` is the origin URL
|
||||
when one exists, then the title most publishers agree on, then the
|
||||
smallest publisher key. Publisher-controlled times and revisions never
|
||||
break ties. Its metadata remains a publisher's claim.
|
||||
* `publishers` lists distinct publisher **devices**. It is not a subscriber
|
||||
count (one user may publish from several devices, and subscribers who did
|
||||
not share are invisible) and not a trust signal: device identities are
|
||||
free. No subscriber count is offered and ranking ignores it.
|
||||
* Ranking: exact normalized title first, then hits that have a card under
|
||||
the origin URL, then title and id.
|
||||
* `FederatedSearchHit` (`{"type": "music" | "podcast", "hit": ...}`),
|
||||
`SearchHitKind` (`artist`, `release`, `track`, `podcast`) and
|
||||
`merge_search_results` merge podcast hits with `SearchOutcome` music
|
||||
results; `SearchOutcome` itself is unchanged.
|
||||
* Lookups query the 8 closest overlay nodes (α = 3, at most 32 requests,
|
||||
15 s deadline) and return partial results when peers fail. They keep at
|
||||
most 512 `(podcast_id, publisher)` owners; each response (and the local
|
||||
cache) may add at most 64 new owners (`512 / K`), so the responders that
|
||||
answer first cannot fill a lookup and crowd out the rest. A newer record
|
||||
of a known owner always replaces its winner. A peer that fails a request
|
||||
is skipped for 30 s.
|
||||
|
||||
Subscribing from a hit creates the user's own private subscription with the
|
||||
hit's `feed_url` (the origin URL): resolve the id with `resolve_podcast_id`,
|
||||
which yields the hit's `podcast_id` unless the user already knows that URL,
|
||||
fetch the feed from that URL and follow its own HTTP redirects and
|
||||
`itunes:new-feed-url`, recording those as aliases as usual. A URL from
|
||||
`claimed_feed_urls` becomes an alias only when that fetch leads there. The new
|
||||
subscription is private (see `effective_share`), the publisher's availability
|
||||
is not needed afterwards, and the episode catalog comes from RSS.
|
||||
|
||||
### Limits
|
||||
|
||||
| Limit | Value |
|
||||
| --- | --- |
|
||||
| Request / response | 512 KiB / 4 MiB postcard |
|
||||
| Store batch | 1..16 entries, each 1..33 distinct keys, records of the requester only |
|
||||
| Records per `FindValue` response | 128, at most 32 of them tombstones |
|
||||
| Filter tokens | ≤ 8, each 1..64 bytes |
|
||||
| Records per key | 128, at most 32 of them tombstones; random eviction from the younger half |
|
||||
| Records per replica | 256 per publisher (refused beyond), 4 096 in all, 32 MiB of payload (random eviction from the younger half, at most 16 per record) |
|
||||
| Store responsibility | fewer than `2 * K` = 16 verified, not backed-off contacts closer to the key |
|
||||
| Records kept per lookup | 512 `(podcast_id, publisher)`, at most 64 new ones per response |
|
||||
| Persisted overlay contacts | 1 024 |
|
||||
| Inbound requests | 64 concurrent, 8 per peer (more are dropped); 10 s each |
|
||||
| Error message | ≤ 1 KiB |
|
||||
| Contact ticket | ≤ 16 KiB |
|
||||
|
||||
### Privacy guarantees
|
||||
|
||||
* Only podcasts with `effective_share` are published; never-shared podcasts
|
||||
produce no network traffic at all.
|
||||
* After a restart nothing is re-sent from the persisted cards until the
|
||||
client reconciled its registers with `sync_shares` (see Publication).
|
||||
* Cards carry the public projection only (see Card); tombstones carry key
|
||||
hashes only.
|
||||
* Records are signed by the publisher device and verified on every receive
|
||||
and on every cache read; a forwarder cannot alter or impersonate them,
|
||||
only the publisher can store them, and records from another network are
|
||||
rejected.
|
||||
* The card's `publisher` is the device's persistent iroh endpoint key, which
|
||||
is also its overlay and music-DHT identity. It is **linkable**: anyone can
|
||||
look the key up in the overlay (`FindNode` for
|
||||
`NodeId::from_endpoint(publisher)`) and obtain the device's contact ticket
|
||||
with its relay and direct IP addresses, because every overlay request
|
||||
carries the requester's ticket. The same key owns the device's music-DHT
|
||||
records, so a device's shared podcasts can be linked to its public music
|
||||
library and its network address. Subscribing from a card never contacts
|
||||
the publisher.
|
||||
* Devices of one user are not linked by the overlay, but publishing the same
|
||||
podcasts from several devices makes them correlatable.
|
||||
* Unpublishing is eventually consistent: replicas that missed the tombstone
|
||||
keep serving the old card until it expires (at most 12 h), including
|
||||
replicas that only held an older title's text keys.
|
||||
* Capability manifests are informational. A peer without `podcast_dht` never
|
||||
enters the overlay routing table (it cannot answer on the ALPN), and
|
||||
`PodcastDht::note_peer_capabilities` evicts a peer whose manifest lacks it
|
||||
and stops dialing it. Applications advertise the overlay only when they
|
||||
run it (`with_podcast_dht`).
|
||||
|
||||
## Feed parser
|
||||
|
||||
With the `podcast-feed` feature, `music_dht::podcast::feed` parses RSS 2.0,
|
||||
@@ -609,7 +976,9 @@ must keep the same invariants.
|
||||
`set_shared`/`unsubscribe` the share register, with
|
||||
`next_podcast_generation`. No-op actions draw no stamp and emit nothing.
|
||||
`unsubscribe` emits `subscribed = false` and `shared = false` and keeps
|
||||
progress, sessions and the catalog.
|
||||
progress, sessions and the catalog. The store only records the decision;
|
||||
publishing and revoking the public card is `PodcastDht::sync_shares` (see
|
||||
[Public discovery](#public-discovery-podcast_dht)).
|
||||
* Explicit progress actions use `explicit_generation(max(local, carried))`;
|
||||
checkpoints keep the session generation and are refused (`Superseded`)
|
||||
once the register holds a greater one, or after the session completed.
|
||||
@@ -667,5 +1036,8 @@ must keep the same invariants.
|
||||
* Feed URL canonicalization, the identity bounds (`MAX_URL_BYTES`,
|
||||
`MAX_GUID_BYTES`) and the no-GUID rule are frozen with the `v1` domains:
|
||||
receivers reject non-canonical ids and URLs while parsing.
|
||||
* Public podcast cards will use a separate signed overlay with its own ALPN
|
||||
and version; the music DHT schema and `federation-net` do not change.
|
||||
* Public podcast cards use the separate signed overlay `podcast_dht` with
|
||||
its own ALPN (`furumi-fd/podcast-dht/1`), wire/record version 1, signing
|
||||
domains `furumi-podcast-card:v1` / `furumi-podcast-card-tombstone:v1`, key
|
||||
domain `furumi-podcast-dht:key:v1`, all frozen, and cache schema 2; the music
|
||||
DHT schema and `federation-net` do not change.
|
||||
@@ -57,6 +57,10 @@ It lives in its own file, conventionally `podcasts.sqlite3`
|
||||
- **Queries**: `list_podcasts`, `seasons`, `episodes` (season, played-state
|
||||
and text filters, date or episode-number order), `resume_position_ms`,
|
||||
`episode_ref` / `episode_track` for queueing, `listen_history`.
|
||||
- **Public discovery**: the store records share decisions only. Build each
|
||||
`music_dht::podcast_dht::PodcastShareState` for `PodcastDht::sync_shares`
|
||||
from `subscription` and `share` (workspace `PODCASTS.md`, section "Public
|
||||
discovery (podcast_dht)"); the client keeps the user's `privacy_confirmed` decision.
|
||||
|
||||
The store is synchronous: methods take `&self` and lock one connection, so
|
||||
`Arc<PodcastStore>` can be shared; run calls on a blocking pool.
|
||||
|
||||
@@ -2242,3 +2242,82 @@ fn concurrent_fetch_state_writers_lose_no_update() {
|
||||
assert_eq!(state.consecutive_failures as usize, THREADS * FAILURES);
|
||||
assert_eq!(state.cover_path.as_deref(), Some("/covers/c.jpg"));
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Public discovery (`music_dht::podcast_dht`)
|
||||
|
||||
/// The share state `PodcastDht::sync_shares` takes, built from the store the
|
||||
/// way clients do it.
|
||||
fn discovery_state(
|
||||
store: &PodcastStore,
|
||||
podcast_id: &PodcastId,
|
||||
) -> music_dht::podcast_dht::PodcastShareState {
|
||||
music_dht::podcast_dht::PodcastShareState {
|
||||
subscription: store.subscription(podcast_id).unwrap().unwrap(),
|
||||
share: store.share(podcast_id).unwrap(),
|
||||
card: None,
|
||||
privacy_confirmed: false,
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn store_registers_drive_discovery_publication() {
|
||||
let store = PodcastStore::open_in_memory().unwrap();
|
||||
let mut stamps = TestStamps::new("device-a", 1_000);
|
||||
let podcast_id = store
|
||||
.subscribe(&mut stamps, &meta(FEED, "Show"))
|
||||
.unwrap()
|
||||
.podcast_id;
|
||||
|
||||
// Subscribed but private: nothing to publish.
|
||||
let state = discovery_state(&store, &podcast_id);
|
||||
assert!(!state.is_shared());
|
||||
assert!(!store.podcast(&podcast_id).unwrap().unwrap().shared);
|
||||
|
||||
store.set_shared(&mut stamps, &podcast_id, true).unwrap();
|
||||
let state = discovery_state(&store, &podcast_id);
|
||||
assert!(state.is_shared());
|
||||
assert!(store.podcast(&podcast_id).unwrap().unwrap().shared);
|
||||
let card = state.card_spec().unwrap();
|
||||
assert_eq!(card.podcast_id, podcast_id);
|
||||
assert_eq!(card.feed_url, url(FEED));
|
||||
assert_eq!(card.id_feed_url, None);
|
||||
assert_eq!(card.title, "Show");
|
||||
assert!(card.privacy_warnings().is_empty());
|
||||
|
||||
// After a permanent move the register's alias set still holds the origin
|
||||
// URL, so the card stays bound to the podcast id.
|
||||
let items = [Item::new(
|
||||
Some("g1"),
|
||||
"One",
|
||||
"https://cdn.example.test/1.mp3",
|
||||
)];
|
||||
let moved = url("https://new.example.test/feed.xml");
|
||||
let feed = parse(&rss("Show", None, &items), &moved);
|
||||
refresh(&store, &mut stamps, &podcast_id, &feed, &moved, true);
|
||||
let state = discovery_state(&store, &podcast_id);
|
||||
assert!(state.subscription.feed_urls().contains(&moved));
|
||||
let card = state.card_spec().unwrap();
|
||||
assert_eq!(card.podcast_id, podcast_id);
|
||||
// A card for the URL this device now fetches names the origin URL.
|
||||
let summary = store.podcast(&podcast_id).unwrap().unwrap();
|
||||
assert_eq!(summary.meta.feed_url, moved);
|
||||
let fetched = music_dht::podcast_dht::PodcastShareState {
|
||||
card: Some(music_dht::podcast_dht::PodcastCardSpec {
|
||||
feed_url: summary.meta.feed_url.clone(),
|
||||
id_feed_url: Some(url(FEED)),
|
||||
..card
|
||||
}),
|
||||
..state
|
||||
};
|
||||
let card = fetched.card_spec().unwrap();
|
||||
assert_eq!(card.feed_url, moved);
|
||||
assert_eq!(card.id_feed_url, Some(url(FEED)));
|
||||
card.validate().unwrap();
|
||||
|
||||
// Unsubscribing revokes the share decision: the card must be unpublished.
|
||||
store.unsubscribe(&mut stamps, &podcast_id).unwrap();
|
||||
let state = discovery_state(&store, &podcast_id);
|
||||
assert!(!state.is_shared());
|
||||
assert!(!store.podcast(&podcast_id).unwrap().unwrap().shared);
|
||||
}
|
||||
@@ -250,6 +250,35 @@ reported itself (`profile_protocol_version`; relayed profiles go through
|
||||
`jam::redact_jam_snapshot`, `jam_command_allowed` and
|
||||
`restore_jam_private_items`.
|
||||
|
||||
### Public podcast discovery
|
||||
|
||||
`music_dht::podcast_dht::PodcastDht` is a separate signed overlay (ALPN
|
||||
`furumi-fd/podcast-dht/1`, capability `podcast_dht`) for podcasts the user
|
||||
explicitly shared. Wiring, as for `SimilarityDht`:
|
||||
|
||||
1. declare the ALPN with `schema_independent_stream_protocol(PODCAST_DHT_ALPN)`,
|
||||
open the overlay with its own cache file, take
|
||||
`stream_acceptor(PODCAST_DHT_ALPN)` and spawn `serve` and `maintenance`;
|
||||
2. advertise it with `CapabilityManifest::with_podcast_dht()` (it is not in
|
||||
`CapabilityManifest::frid()`; an application that does not run the overlay
|
||||
simply does not add it);
|
||||
3. pass every peer manifest received to `note_peer_capabilities`, so peers
|
||||
without the overlay are not dialed;
|
||||
4. call `sync_shares` with the merged subscription and share registers on
|
||||
start and after applying sync operations (until the first call, own cards
|
||||
persisted by an earlier run are neither republished nor served): it
|
||||
publishes cards only where `effective_share` holds, tombstones every other
|
||||
own publication, and reports cards it could not build (`failed`) or held
|
||||
back for privacy (`held_back`);
|
||||
5. ask the user before publishing when `PodcastCardSpec::privacy_warnings`
|
||||
(or `feed_url_privacy_warning`) reports a URL that looks private, then
|
||||
call `publish` or set `PodcastShareState::privacy_confirmed`.
|
||||
|
||||
`search` and `lookup_podcast` return `PodcastSearchHit`s; subscribe with the
|
||||
hit's `feed_url` (the URL the id hashes from), never with a
|
||||
`claimed_feed_urls` entry. `merge_search_results` combines hits with
|
||||
`SearchOutcome::network_results` as typed `FederatedSearchHit`s.
|
||||
|
||||
## Verification
|
||||
|
||||
```bash
|
||||
|
||||
@@ -35,6 +35,8 @@ pub const CATALOG_ID: &str = "catalog";
|
||||
pub const SIMILARITY_ID: &str = "similarity";
|
||||
/// Stable protocol identifier for DHT-routed similarity discovery.
|
||||
pub const SIMILARITY_DHT_ID: &str = "similarity_dht";
|
||||
/// Stable protocol identifier for the public podcast-discovery overlay.
|
||||
pub const PODCAST_DHT_ID: &str = "podcast_dht";
|
||||
/// Stable protocol identifier for personal-device synchronization.
|
||||
pub const DEVICE_SYNC_ID: &str = "device_sync";
|
||||
/// Stable protocol identifier for Jam playback control.
|
||||
@@ -54,7 +56,13 @@ pub struct CapabilityManifest {
|
||||
}
|
||||
|
||||
impl CapabilityManifest {
|
||||
/// Creates a manifest containing every protocol owned by Frid.
|
||||
/// Creates a manifest containing every protocol owned by Frid that a
|
||||
/// music-DHT application serves.
|
||||
///
|
||||
/// The public podcast-discovery overlay is opt-in: applications that run
|
||||
/// [`crate::podcast_dht::PodcastDht`] add it with
|
||||
/// [`Self::with_podcast_dht`], so an application that does not serve the
|
||||
/// overlay never advertises it.
|
||||
pub fn frid(application: impl Into<String>, application_version: impl Into<String>) -> Self {
|
||||
let mut protocols = BTreeMap::new();
|
||||
protocols.insert(
|
||||
@@ -105,6 +113,29 @@ impl CapabilityManifest {
|
||||
self
|
||||
}
|
||||
|
||||
/// Advertises the public podcast-discovery overlay
|
||||
/// ([`PODCAST_DHT_ID`] at [`crate::podcast_dht::PODCAST_DHT_PROTOCOL_VERSION`]).
|
||||
/// Call it only when the application serves
|
||||
/// [`crate::podcast_dht::PODCAST_DHT_ALPN`].
|
||||
pub fn with_podcast_dht(self) -> Self {
|
||||
self.with_protocol(
|
||||
PODCAST_DHT_ID,
|
||||
crate::podcast_dht::PODCAST_DHT_PROTOCOL_VERSION,
|
||||
)
|
||||
}
|
||||
|
||||
/// Version of protocol `id` advertised by the peer, if any.
|
||||
pub fn protocol_version(&self, id: &str) -> Option<u16> {
|
||||
self.protocols.get(id).copied()
|
||||
}
|
||||
|
||||
/// Whether the peer advertises the public podcast-discovery overlay at
|
||||
/// the version this crate speaks. Peers without it are never queried.
|
||||
pub fn supports_podcast_dht(&self) -> bool {
|
||||
self.protocol_version(PODCAST_DHT_ID)
|
||||
== Some(crate::podcast_dht::PODCAST_DHT_PROTOCOL_VERSION)
|
||||
}
|
||||
|
||||
/// Validates bounds and identifiers received from a peer.
|
||||
pub fn validate(&self) -> Result<()> {
|
||||
if self.manifest_version != CAPABILITIES_PROTOCOL_VERSION {
|
||||
@@ -227,6 +258,24 @@ mod tests {
|
||||
manifest.validate().unwrap();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn podcast_dht_capability_is_opt_in_and_removable() {
|
||||
let plain = CapabilityManifest::frid("furumi", "1.2.3");
|
||||
assert_eq!(plain.protocol_version(PODCAST_DHT_ID), None);
|
||||
assert!(!plain.supports_podcast_dht());
|
||||
let manifest = plain.with_podcast_dht();
|
||||
assert_eq!(
|
||||
manifest.protocol_version(PODCAST_DHT_ID),
|
||||
Some(crate::podcast_dht::PODCAST_DHT_PROTOCOL_VERSION)
|
||||
);
|
||||
assert!(manifest.supports_podcast_dht());
|
||||
manifest.validate().unwrap();
|
||||
let without = manifest.clone().without_protocol(PODCAST_DHT_ID);
|
||||
assert!(!without.supports_podcast_dht());
|
||||
let other_version = manifest.with_protocol(PODCAST_DHT_ID, 2);
|
||||
assert!(!other_version.supports_podcast_dht());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn application_protocol_is_additive() {
|
||||
let manifest = CapabilityManifest::frid("furumi", "1.2.3").with_protocol("audio", 1);
|
||||
|
||||
@@ -50,6 +50,11 @@ pub enum MusicDhtError {
|
||||
/// The service is shutting down and no longer accepts operations.
|
||||
#[error("service is shutting down")]
|
||||
ShuttingDown,
|
||||
|
||||
/// A podcast card cannot be published, for example because the user did
|
||||
/// not share the podcast (see `music_dht::podcast::effective_share`).
|
||||
#[error("podcast publication refused: {0}")]
|
||||
PodcastPublication(String),
|
||||
}
|
||||
|
||||
impl From<federation_net::NetworkError> for MusicDhtError {
|
||||
|
||||
@@ -27,6 +27,9 @@
|
||||
//! podcast/episode ids, metadata DTOs and convergent merge rules), with an
|
||||
//! optional RSS/Atom parser behind the `podcast-feed` feature;
|
||||
//! [`device_sync`] protocol v4 carries it between trusted devices.
|
||||
//! * [`podcast_dht`] is a separate signed overlay for public podcast
|
||||
//! discovery: cards a user explicitly shared, found by title/author tokens
|
||||
//! or podcast id, retracted by signed tombstones.
|
||||
//! * Records are replicated to the `K` nodes whose ids are XOR-closest to
|
||||
//! each key. Publishers pick the targets from their routing table and send
|
||||
//! batched store requests (one pipeline per peer), so even a large library
|
||||
@@ -97,6 +100,7 @@ mod message;
|
||||
mod node;
|
||||
mod normalization;
|
||||
pub mod podcast;
|
||||
pub mod podcast_dht;
|
||||
mod record;
|
||||
mod request;
|
||||
mod routing;
|
||||
|
||||
@@ -0,0 +1,899 @@
|
||||
//! Signed public podcast cards, tombstones, DHT keys and their canonical
|
||||
//! signing encoding.
|
||||
//!
|
||||
//! Everything here is pure and deterministic: no clock, no I/O. Callers pass
|
||||
//! the current time to the validation functions.
|
||||
|
||||
use std::cmp::Ordering;
|
||||
use std::collections::HashSet;
|
||||
use std::time::Duration;
|
||||
|
||||
use federation_net::{EndpointId, NetworkId, Signature};
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
use crate::error::{MusicDhtError, Result};
|
||||
use crate::normalization::{normalize_name, tokenize};
|
||||
use crate::podcast::{
|
||||
FeedUrl, MAX_URL_BYTES, PodcastId, SubscriptionRecord, bounded_text, is_http_url,
|
||||
};
|
||||
|
||||
use super::privacy::{FeedUrlPrivacyWarning, feed_url_privacy_warning};
|
||||
|
||||
/// Current card, tombstone and overlay wire version.
|
||||
pub const PODCAST_DHT_PROTOCOL_VERSION: u16 = 1;
|
||||
/// Signing domain of a [`PodcastCard`]. Changing it invalidates every card.
|
||||
pub const PODCAST_CARD_SIGNING_DOMAIN: &str = "furumi-podcast-card:v1";
|
||||
/// Signing domain of a [`PodcastTombstone`].
|
||||
pub const PODCAST_TOMBSTONE_SIGNING_DOMAIN: &str = "furumi-podcast-card-tombstone:v1";
|
||||
/// BLAKE3 derivation domain of [`PodcastDhtKey`].
|
||||
pub const PODCAST_DHT_KEY_DOMAIN: &str = "furumi-podcast-dht:key:v1";
|
||||
|
||||
/// Maximum lifetime of a card: the main DHT's active-record TTL.
|
||||
pub const PODCAST_CARD_TTL: Duration = crate::record::ACTIVE_RECORD_TTL;
|
||||
/// Maximum lifetime of a tombstone: the main DHT's tombstone TTL. It exceeds
|
||||
/// [`PODCAST_CARD_TTL`], so a card issued before a tombstone always expires
|
||||
/// before the tombstone does.
|
||||
pub const PODCAST_TOMBSTONE_TTL: Duration = crate::record::TOMBSTONE_TTL;
|
||||
/// Tolerated clock skew for `issued_at_ms` in the future.
|
||||
pub const PODCAST_DHT_MAX_CLOCK_SKEW: Duration = Duration::from_secs(5 * 60);
|
||||
|
||||
/// Maximum characters of a card title.
|
||||
pub const MAX_CARD_TITLE_CHARS: usize = 300;
|
||||
/// Maximum characters of a card author.
|
||||
pub const MAX_CARD_AUTHOR_CHARS: usize = 300;
|
||||
/// Maximum characters of a card description.
|
||||
pub const MAX_CARD_DESCRIPTION_CHARS: usize = 1_000;
|
||||
/// Maximum bytes of a card language tag (BCP 47 shape: ASCII letters,
|
||||
/// digits and `-`).
|
||||
pub const MAX_CARD_LANGUAGE_BYTES: usize = 35;
|
||||
/// Maximum categories on a card.
|
||||
pub const MAX_CARD_CATEGORIES: usize = 8;
|
||||
/// Maximum characters of one category.
|
||||
pub const MAX_CARD_CATEGORY_CHARS: usize = 64;
|
||||
/// Maximum indexed search tokens of a card (title tokens, then author tokens).
|
||||
pub const MAX_CARD_INDEX_TOKENS: usize = 32;
|
||||
/// Maximum bytes of one indexed or query token. Longer tokens are not indexed.
|
||||
pub const MAX_INDEX_TOKEN_BYTES: usize = 64;
|
||||
/// Maximum DHT keys a record is stored under: the podcast-id key plus one
|
||||
/// key per indexed token.
|
||||
pub const MAX_RECORD_DHT_KEYS: usize = MAX_CARD_INDEX_TOKENS + 1;
|
||||
/// Maximum postcard-encoded size of one signed record.
|
||||
pub const MAX_PODCAST_RECORD_BYTES: usize = 24 * 1024;
|
||||
|
||||
const KEY_KIND_PODCAST: u8 = 1;
|
||||
const KEY_KIND_TOKEN: u8 = 2;
|
||||
|
||||
/// A 256-bit point in the podcast-discovery key space.
|
||||
///
|
||||
/// `BLAKE3("furumi-podcast-dht:key:v1" || 0x00 || network_id || kind || material)`
|
||||
/// with kind `0x01` and the podcast id string for the id key, or kind `0x02`
|
||||
/// and a normalized token for a text key.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord, Serialize, Deserialize)]
|
||||
pub struct PodcastDhtKey([u8; 32]);
|
||||
|
||||
impl PodcastDhtKey {
|
||||
/// Key under which every card and tombstone of `podcast_id` is stored.
|
||||
pub fn podcast(network_id: &NetworkId, podcast_id: &PodcastId) -> Self {
|
||||
Self::derive(network_id, KEY_KIND_PODCAST, podcast_id.as_str())
|
||||
}
|
||||
|
||||
/// Key of one normalized search token (see [`card_index_tokens`]).
|
||||
pub fn token(network_id: &NetworkId, token: &str) -> Self {
|
||||
Self::derive(network_id, KEY_KIND_TOKEN, token)
|
||||
}
|
||||
|
||||
fn derive(network_id: &NetworkId, kind: u8, material: &str) -> Self {
|
||||
let mut hasher = blake3::Hasher::new();
|
||||
hasher.update(PODCAST_DHT_KEY_DOMAIN.as_bytes());
|
||||
hasher.update(&[0]);
|
||||
hasher.update(network_id.as_bytes());
|
||||
hasher.update(&[kind]);
|
||||
hasher.update(material.as_bytes());
|
||||
Self(*hasher.finalize().as_bytes())
|
||||
}
|
||||
|
||||
/// Creates a key from raw bytes received on the wire.
|
||||
pub fn from_bytes(bytes: [u8; 32]) -> Self {
|
||||
Self(bytes)
|
||||
}
|
||||
|
||||
/// Raw key bytes.
|
||||
pub fn as_bytes(&self) -> &[u8; 32] {
|
||||
&self.0
|
||||
}
|
||||
}
|
||||
|
||||
/// Indexed search tokens of a card: distinct [`normalize_name`] tokens of the
|
||||
/// title, then of the author, in order, skipping tokens longer than
|
||||
/// [`MAX_INDEX_TOKEN_BYTES`], at most [`MAX_CARD_INDEX_TOKENS`].
|
||||
pub fn card_index_tokens(title: &str, author: Option<&str>) -> Vec<String> {
|
||||
let mut seen = HashSet::new();
|
||||
let mut tokens = Vec::new();
|
||||
let sources = std::iter::once(title).chain(author);
|
||||
for source in sources {
|
||||
for token in tokenize(&normalize_name(source)) {
|
||||
if tokens.len() == MAX_CARD_INDEX_TOKENS {
|
||||
return tokens;
|
||||
}
|
||||
if token.len() <= MAX_INDEX_TOKEN_BYTES && seen.insert(token.clone()) {
|
||||
tokens.push(token);
|
||||
}
|
||||
}
|
||||
}
|
||||
tokens
|
||||
}
|
||||
|
||||
/// Unsigned public projection of a podcast prepared by its publisher.
|
||||
///
|
||||
/// It is built from the user's subscription ([`PodcastCardSpec::from_subscription`])
|
||||
/// and contains only publisher metadata: never progress, listening history,
|
||||
/// subscription times, operation ids, device names or episodes.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct PodcastCardSpec {
|
||||
/// Podcast.
|
||||
pub podcast_id: PodcastId,
|
||||
/// Canonical URL the feed is currently fetched from.
|
||||
pub feed_url: FeedUrl,
|
||||
/// URL `podcast_id` was derived from, when it differs from `feed_url`.
|
||||
pub id_feed_url: Option<FeedUrl>,
|
||||
/// Podcast title.
|
||||
pub title: String,
|
||||
/// Author.
|
||||
pub author: Option<String>,
|
||||
/// Plain-text description.
|
||||
pub description: Option<String>,
|
||||
/// Artwork URL.
|
||||
pub image_url: Option<String>,
|
||||
/// Publisher web page.
|
||||
pub link: Option<String>,
|
||||
/// Feed language tag, for example `en` or `pt-BR`.
|
||||
pub language: Option<String>,
|
||||
/// Feed categories.
|
||||
pub categories: Vec<String>,
|
||||
/// Number of episodes in the feed, when known.
|
||||
pub episode_count: Option<u32>,
|
||||
/// Publication time of the newest episode in unix milliseconds.
|
||||
pub latest_episode_at_ms: Option<i64>,
|
||||
}
|
||||
|
||||
impl PodcastCardSpec {
|
||||
/// Builds the bounded public projection of a subscription.
|
||||
///
|
||||
/// Only `podcast_id`, the feed URLs and the channel metadata are read.
|
||||
/// `id_feed_url` is the alias that hashes to `podcast_id`
|
||||
/// ([`crate::podcast::merge_feed_aliases`] always keeps it). Fails when no
|
||||
/// such URL is known. Callers may then set `language`, `categories`,
|
||||
/// `episode_count` and `latest_episode_at_ms` from their feed cache.
|
||||
pub fn from_subscription(subscription: &SubscriptionRecord) -> Result<Self> {
|
||||
let origin = subscription
|
||||
.feed_urls()
|
||||
.into_iter()
|
||||
.find(|url| PodcastId::from_feed_url(url) == subscription.podcast_id)
|
||||
.ok_or_else(|| {
|
||||
publication_error("the feed URL the podcast id was derived from is unknown")
|
||||
})?;
|
||||
let feed_url = subscription.podcast.feed_url.clone();
|
||||
let meta = &subscription.podcast;
|
||||
Ok(Self {
|
||||
podcast_id: subscription.podcast_id.clone(),
|
||||
id_feed_url: (origin != feed_url).then_some(origin),
|
||||
feed_url,
|
||||
title: meta.title.clone(),
|
||||
author: meta.author.clone(),
|
||||
description: meta.description.clone(),
|
||||
image_url: meta.image_url.clone(),
|
||||
link: meta.link.clone(),
|
||||
language: None,
|
||||
categories: Vec::new(),
|
||||
episode_count: None,
|
||||
latest_episode_at_ms: None,
|
||||
}
|
||||
.bounded())
|
||||
}
|
||||
|
||||
/// Returns a copy that fits the card bounds: text is truncated with
|
||||
/// [`bounded_text`], control characters are removed, invalid URLs and
|
||||
/// language tags are dropped, and categories are trimmed, deduplicated and
|
||||
/// bounded. An empty title becomes `"Untitled podcast"`.
|
||||
pub fn bounded(&self) -> Self {
|
||||
let title = clean_line(&self.title);
|
||||
let mut seen = HashSet::new();
|
||||
let categories = self
|
||||
.categories
|
||||
.iter()
|
||||
.map(|category| bounded_text(&clean_line(category), MAX_CARD_CATEGORY_CHARS))
|
||||
.filter(|category| !category.is_empty() && seen.insert(category.to_lowercase()))
|
||||
.take(MAX_CARD_CATEGORIES)
|
||||
.collect();
|
||||
Self {
|
||||
podcast_id: self.podcast_id.clone(),
|
||||
feed_url: self.feed_url.clone(),
|
||||
id_feed_url: self
|
||||
.id_feed_url
|
||||
.clone()
|
||||
.filter(|origin| *origin != self.feed_url),
|
||||
title: if title.is_empty() {
|
||||
"Untitled podcast".to_string()
|
||||
} else {
|
||||
bounded_text(&title, MAX_CARD_TITLE_CHARS)
|
||||
},
|
||||
author: bounded_line(self.author.as_deref(), MAX_CARD_AUTHOR_CHARS),
|
||||
description: self
|
||||
.description
|
||||
.as_deref()
|
||||
.map(|text| bounded_text(clean_block(text).trim(), MAX_CARD_DESCRIPTION_CHARS))
|
||||
.filter(|text| !text.is_empty()),
|
||||
image_url: self.image_url.clone().filter(|url| is_http_url(url)),
|
||||
link: self.link.clone().filter(|url| is_http_url(url)),
|
||||
language: self
|
||||
.language
|
||||
.as_deref()
|
||||
.map(str::trim)
|
||||
.filter(|language| valid_language(language))
|
||||
.map(str::to_string),
|
||||
categories,
|
||||
episode_count: self.episode_count,
|
||||
latest_episode_at_ms: self.latest_episode_at_ms.filter(|at| *at >= 0),
|
||||
}
|
||||
}
|
||||
|
||||
/// Validates the card bounds without bounding anything.
|
||||
pub fn validate(&self) -> Result<()> {
|
||||
let id_url = self.id_feed_url.as_ref().unwrap_or(&self.feed_url);
|
||||
if PodcastId::from_feed_url(id_url) != self.podcast_id {
|
||||
return Err(invalid("podcast id does not match its feed URL"));
|
||||
}
|
||||
if self.id_feed_url.as_ref() == Some(&self.feed_url) {
|
||||
return Err(invalid("id_feed_url must differ from feed_url"));
|
||||
}
|
||||
check_line("title", &self.title, MAX_CARD_TITLE_CHARS, true)?;
|
||||
if let Some(author) = &self.author {
|
||||
check_line("author", author, MAX_CARD_AUTHOR_CHARS, false)?;
|
||||
}
|
||||
if let Some(description) = &self.description {
|
||||
check_chars("description", description, MAX_CARD_DESCRIPTION_CHARS)?;
|
||||
if description
|
||||
.chars()
|
||||
.any(|c| c.is_control() && c != '\n' && c != '\t')
|
||||
{
|
||||
return Err(invalid("description contains control characters"));
|
||||
}
|
||||
}
|
||||
for url in [&self.image_url, &self.link].into_iter().flatten() {
|
||||
if !is_http_url(url) {
|
||||
return Err(invalid("card URLs must be absolute http(s) URLs"));
|
||||
}
|
||||
}
|
||||
if self
|
||||
.language
|
||||
.as_deref()
|
||||
.is_some_and(|language| !valid_language(language))
|
||||
{
|
||||
return Err(invalid("invalid language tag"));
|
||||
}
|
||||
if self.categories.len() > MAX_CARD_CATEGORIES {
|
||||
return Err(invalid("too many categories"));
|
||||
}
|
||||
for category in &self.categories {
|
||||
check_line("category", category, MAX_CARD_CATEGORY_CHARS, true)?;
|
||||
}
|
||||
if self.latest_episode_at_ms.is_some_and(|at| at < 0) {
|
||||
return Err(invalid("negative latest_episode_at_ms"));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Advisory privacy warnings for both feed URLs of the card (see
|
||||
/// [`feed_url_privacy_warning`]). The UI must ask before publishing a
|
||||
/// card with warnings.
|
||||
pub fn privacy_warnings(&self) -> Vec<FeedUrlPrivacyWarning> {
|
||||
std::iter::once(&self.feed_url)
|
||||
.chain(self.id_feed_url.as_ref())
|
||||
.filter_map(feed_url_privacy_warning)
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Indexed search tokens of this card.
|
||||
pub fn index_tokens(&self) -> Vec<String> {
|
||||
card_index_tokens(&self.title, self.author.as_deref())
|
||||
}
|
||||
}
|
||||
|
||||
/// A signed public podcast card: one publisher device's projection of one
|
||||
/// podcast. See `PODCASTS.md`, "Public discovery (podcast_dht)".
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct PodcastCard {
|
||||
/// Record format version ([`PODCAST_DHT_PROTOCOL_VERSION`]).
|
||||
pub version: u16,
|
||||
/// Network scope; prevents replay into another federation.
|
||||
pub network_id: NetworkId,
|
||||
/// Publisher device (its persistent transport identity); signs the card.
|
||||
pub publisher: EndpointId,
|
||||
/// Podcast.
|
||||
pub podcast_id: PodcastId,
|
||||
/// Canonical URL the feed is currently fetched from.
|
||||
pub feed_url: FeedUrl,
|
||||
/// URL `podcast_id` was derived from, when it differs from `feed_url`.
|
||||
pub id_feed_url: Option<FeedUrl>,
|
||||
/// Podcast title.
|
||||
pub title: String,
|
||||
/// Author.
|
||||
pub author: Option<String>,
|
||||
/// Plain-text description.
|
||||
pub description: Option<String>,
|
||||
/// Artwork URL.
|
||||
pub image_url: Option<String>,
|
||||
/// Publisher web page.
|
||||
pub link: Option<String>,
|
||||
/// Feed language tag.
|
||||
pub language: Option<String>,
|
||||
/// Feed categories.
|
||||
pub categories: Vec<String>,
|
||||
/// Number of episodes in the feed, when known.
|
||||
pub episode_count: Option<u32>,
|
||||
/// Publication time of the newest episode in unix milliseconds.
|
||||
pub latest_episode_at_ms: Option<i64>,
|
||||
/// Monotonic per `(podcast_id, publisher)` revision.
|
||||
pub revision: u64,
|
||||
/// Unix time at which this version was signed.
|
||||
pub issued_at_ms: u64,
|
||||
/// Unix time after which the card must be discarded.
|
||||
pub expires_at_ms: u64,
|
||||
}
|
||||
|
||||
impl PodcastCard {
|
||||
/// Builds a card from a spec. `expires_at_ms` is
|
||||
/// `issued_at_ms + PODCAST_CARD_TTL`.
|
||||
pub fn from_spec(
|
||||
spec: PodcastCardSpec,
|
||||
network_id: NetworkId,
|
||||
publisher: EndpointId,
|
||||
revision: u64,
|
||||
issued_at_ms: u64,
|
||||
) -> Self {
|
||||
Self {
|
||||
version: PODCAST_DHT_PROTOCOL_VERSION,
|
||||
network_id,
|
||||
publisher,
|
||||
podcast_id: spec.podcast_id,
|
||||
feed_url: spec.feed_url,
|
||||
id_feed_url: spec.id_feed_url,
|
||||
title: spec.title,
|
||||
author: spec.author,
|
||||
description: spec.description,
|
||||
image_url: spec.image_url,
|
||||
link: spec.link,
|
||||
language: spec.language,
|
||||
categories: spec.categories,
|
||||
episode_count: spec.episode_count,
|
||||
latest_episode_at_ms: spec.latest_episode_at_ms,
|
||||
revision,
|
||||
issued_at_ms,
|
||||
expires_at_ms: issued_at_ms.saturating_add(PODCAST_CARD_TTL.as_millis() as u64),
|
||||
}
|
||||
}
|
||||
|
||||
/// The publisher-provided part of the card.
|
||||
pub fn spec(&self) -> PodcastCardSpec {
|
||||
PodcastCardSpec {
|
||||
podcast_id: self.podcast_id.clone(),
|
||||
feed_url: self.feed_url.clone(),
|
||||
id_feed_url: self.id_feed_url.clone(),
|
||||
title: self.title.clone(),
|
||||
author: self.author.clone(),
|
||||
description: self.description.clone(),
|
||||
image_url: self.image_url.clone(),
|
||||
link: self.link.clone(),
|
||||
language: self.language.clone(),
|
||||
categories: self.categories.clone(),
|
||||
episode_count: self.episode_count,
|
||||
latest_episode_at_ms: self.latest_episode_at_ms,
|
||||
}
|
||||
}
|
||||
|
||||
/// Indexed search tokens.
|
||||
pub fn index_tokens(&self) -> Vec<String> {
|
||||
card_index_tokens(&self.title, self.author.as_deref())
|
||||
}
|
||||
|
||||
/// The URL `podcast_id` is derived from (`id_feed_url`, or `feed_url`
|
||||
/// when the feed has not moved). A valid card's id hashes from it, so it
|
||||
/// is the only URL of the card bound to the podcast.
|
||||
pub fn origin_feed_url(&self) -> &FeedUrl {
|
||||
self.id_feed_url.as_ref().unwrap_or(&self.feed_url)
|
||||
}
|
||||
|
||||
/// Whether `feed_url` is the URL the id is derived from. Otherwise
|
||||
/// `feed_url` is an unverified claim of the publisher that the feed moved.
|
||||
pub fn feed_url_is_origin(&self) -> bool {
|
||||
self.id_feed_url.is_none()
|
||||
}
|
||||
|
||||
/// Canonical signing input (see `PODCASTS.md`).
|
||||
pub fn signing_bytes(&self) -> Vec<u8> {
|
||||
let mut out = Canonical::new(PODCAST_CARD_SIGNING_DOMAIN);
|
||||
out.u16(self.version);
|
||||
out.raw(self.network_id.as_bytes());
|
||||
out.raw(self.publisher.as_bytes());
|
||||
out.str(self.podcast_id.as_str());
|
||||
out.str(self.feed_url.as_str());
|
||||
out.opt_str(self.id_feed_url.as_ref().map(FeedUrl::as_str));
|
||||
out.str(&self.title);
|
||||
out.opt_str(self.author.as_deref());
|
||||
out.opt_str(self.description.as_deref());
|
||||
out.opt_str(self.image_url.as_deref());
|
||||
out.opt_str(self.link.as_deref());
|
||||
out.opt_str(self.language.as_deref());
|
||||
out.u32(self.categories.len() as u32);
|
||||
for category in &self.categories {
|
||||
out.str(category);
|
||||
}
|
||||
match self.episode_count {
|
||||
Some(count) => {
|
||||
out.u8(1);
|
||||
out.u32(count);
|
||||
}
|
||||
None => out.u8(0),
|
||||
}
|
||||
match self.latest_episode_at_ms {
|
||||
Some(at) => {
|
||||
out.u8(1);
|
||||
out.i64(at);
|
||||
}
|
||||
None => out.u8(0),
|
||||
}
|
||||
out.u64(self.revision);
|
||||
out.u64(self.issued_at_ms);
|
||||
out.u64(self.expires_at_ms);
|
||||
out.finish()
|
||||
}
|
||||
|
||||
fn validate(&self, expected_network: &NetworkId) -> Result<()> {
|
||||
check_header(self.version, &self.network_id, expected_network)?;
|
||||
self.spec().validate()?;
|
||||
check_lifetime(
|
||||
self.revision,
|
||||
self.issued_at_ms,
|
||||
self.expires_at_ms,
|
||||
PODCAST_CARD_TTL,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
/// A signed retraction of one publisher's card of one podcast.
|
||||
///
|
||||
/// It supersedes every card of the same `(podcast_id, publisher)` with a
|
||||
/// revision lower than or equal to its own and is retained until it expires.
|
||||
/// `index_keys` lists the text keys of the retracted cards so the tombstone
|
||||
/// can be stored next to them; it carries hashes, not the tokens.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct PodcastTombstone {
|
||||
/// Record format version ([`PODCAST_DHT_PROTOCOL_VERSION`]).
|
||||
pub version: u16,
|
||||
/// Network scope.
|
||||
pub network_id: NetworkId,
|
||||
/// Publisher device; signs the tombstone.
|
||||
pub publisher: EndpointId,
|
||||
/// Podcast.
|
||||
pub podcast_id: PodcastId,
|
||||
/// Revision; greater than that of every retracted card.
|
||||
pub revision: u64,
|
||||
/// Unix time at which the tombstone was signed.
|
||||
pub issued_at_ms: u64,
|
||||
/// Unix time after which the tombstone may be discarded.
|
||||
pub expires_at_ms: u64,
|
||||
/// Sorted, distinct text keys besides the podcast-id key, at most
|
||||
/// [`MAX_CARD_INDEX_TOKENS`].
|
||||
pub index_keys: Vec<PodcastDhtKey>,
|
||||
}
|
||||
|
||||
impl PodcastTombstone {
|
||||
/// Builds a tombstone. `expires_at_ms` is `issued_at_ms + PODCAST_TOMBSTONE_TTL`;
|
||||
/// `index_keys` is sorted, deduplicated and stripped of the podcast-id key.
|
||||
pub fn new(
|
||||
network_id: NetworkId,
|
||||
publisher: EndpointId,
|
||||
podcast_id: PodcastId,
|
||||
revision: u64,
|
||||
issued_at_ms: u64,
|
||||
index_keys: impl IntoIterator<Item = PodcastDhtKey>,
|
||||
) -> Self {
|
||||
let id_key = PodcastDhtKey::podcast(&network_id, &podcast_id);
|
||||
let mut index_keys: Vec<_> = index_keys
|
||||
.into_iter()
|
||||
.filter(|key| *key != id_key)
|
||||
.collect();
|
||||
index_keys.sort();
|
||||
index_keys.dedup();
|
||||
index_keys.truncate(MAX_CARD_INDEX_TOKENS);
|
||||
Self {
|
||||
version: PODCAST_DHT_PROTOCOL_VERSION,
|
||||
network_id,
|
||||
publisher,
|
||||
podcast_id,
|
||||
revision,
|
||||
issued_at_ms,
|
||||
expires_at_ms: issued_at_ms.saturating_add(PODCAST_TOMBSTONE_TTL.as_millis() as u64),
|
||||
index_keys,
|
||||
}
|
||||
}
|
||||
|
||||
/// Canonical signing input (see `PODCASTS.md`).
|
||||
pub fn signing_bytes(&self) -> Vec<u8> {
|
||||
let mut out = Canonical::new(PODCAST_TOMBSTONE_SIGNING_DOMAIN);
|
||||
out.u16(self.version);
|
||||
out.raw(self.network_id.as_bytes());
|
||||
out.raw(self.publisher.as_bytes());
|
||||
out.str(self.podcast_id.as_str());
|
||||
out.u64(self.revision);
|
||||
out.u64(self.issued_at_ms);
|
||||
out.u64(self.expires_at_ms);
|
||||
out.u32(self.index_keys.len() as u32);
|
||||
for key in &self.index_keys {
|
||||
out.raw(key.as_bytes());
|
||||
}
|
||||
out.finish()
|
||||
}
|
||||
|
||||
fn validate(&self, expected_network: &NetworkId) -> Result<()> {
|
||||
check_header(self.version, &self.network_id, expected_network)?;
|
||||
if self.index_keys.len() > MAX_CARD_INDEX_TOKENS {
|
||||
return Err(invalid("tombstone has too many keys"));
|
||||
}
|
||||
let id_key = PodcastDhtKey::podcast(&self.network_id, &self.podcast_id);
|
||||
if self.index_keys.contains(&id_key)
|
||||
|| self.index_keys.windows(2).any(|pair| pair[0] >= pair[1])
|
||||
{
|
||||
return Err(invalid("tombstone keys are not canonical"));
|
||||
}
|
||||
check_lifetime(
|
||||
self.revision,
|
||||
self.issued_at_ms,
|
||||
self.expires_at_ms,
|
||||
PODCAST_TOMBSTONE_TTL,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
/// Body of a signed podcast-discovery record.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub enum PodcastRecordBody {
|
||||
/// An active public card.
|
||||
Card(Box<PodcastCard>),
|
||||
/// A retraction.
|
||||
Tombstone(PodcastTombstone),
|
||||
}
|
||||
|
||||
/// End-to-end signed podcast-discovery record.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct SignedPodcastRecord {
|
||||
/// Signed body.
|
||||
pub body: PodcastRecordBody,
|
||||
/// Ed25519 signature by `publisher` over [`Self::signing_bytes`].
|
||||
pub signature: Signature,
|
||||
}
|
||||
|
||||
impl SignedPodcastRecord {
|
||||
/// Validates and signs a card.
|
||||
pub fn sign_card(card: PodcastCard, signer: impl FnOnce(&[u8]) -> Signature) -> Result<Self> {
|
||||
card.validate(&card.network_id)?;
|
||||
let signature = signer(&card.signing_bytes());
|
||||
let record = Self {
|
||||
body: PodcastRecordBody::Card(Box::new(card)),
|
||||
signature,
|
||||
};
|
||||
record.check_size()?;
|
||||
Ok(record)
|
||||
}
|
||||
|
||||
/// Validates and signs a tombstone.
|
||||
pub fn sign_tombstone(
|
||||
tombstone: PodcastTombstone,
|
||||
signer: impl FnOnce(&[u8]) -> Signature,
|
||||
) -> Result<Self> {
|
||||
tombstone.validate(&tombstone.network_id)?;
|
||||
let signature = signer(&tombstone.signing_bytes());
|
||||
let record = Self {
|
||||
body: PodcastRecordBody::Tombstone(tombstone),
|
||||
signature,
|
||||
};
|
||||
record.check_size()?;
|
||||
Ok(record)
|
||||
}
|
||||
|
||||
/// Canonical signing input of the body.
|
||||
pub fn signing_bytes(&self) -> Vec<u8> {
|
||||
match &self.body {
|
||||
PodcastRecordBody::Card(card) => card.signing_bytes(),
|
||||
PodcastRecordBody::Tombstone(tombstone) => tombstone.signing_bytes(),
|
||||
}
|
||||
}
|
||||
|
||||
/// Validates bounds, network, lifetime shape, encoded size and the
|
||||
/// publisher's signature. Does not look at the clock; see
|
||||
/// [`Self::verify_at`].
|
||||
pub fn verify(&self, expected_network: &NetworkId) -> Result<()> {
|
||||
match &self.body {
|
||||
PodcastRecordBody::Card(card) => card.validate(expected_network)?,
|
||||
PodcastRecordBody::Tombstone(tombstone) => tombstone.validate(expected_network)?,
|
||||
}
|
||||
self.check_size()?;
|
||||
self.publisher()
|
||||
.verify(&self.signing_bytes(), &self.signature)
|
||||
.map_err(|_| invalid("invalid podcast record signature"))
|
||||
}
|
||||
|
||||
/// [`Self::verify`] plus time checks: not issued more than
|
||||
/// [`PODCAST_DHT_MAX_CLOCK_SKEW`] in the future and not expired at `now_ms`.
|
||||
pub fn verify_at(&self, expected_network: &NetworkId, now_ms: u64) -> Result<()> {
|
||||
self.verify(expected_network)?;
|
||||
let future_limit = now_ms.saturating_add(PODCAST_DHT_MAX_CLOCK_SKEW.as_millis() as u64);
|
||||
if self.issued_at_ms() > future_limit || self.expires_at_ms() <= now_ms {
|
||||
return Err(invalid("podcast record is expired or from the future"));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// The card, when this is not a tombstone.
|
||||
pub fn card(&self) -> Option<&PodcastCard> {
|
||||
match &self.body {
|
||||
PodcastRecordBody::Card(card) => Some(card.as_ref()),
|
||||
PodcastRecordBody::Tombstone(_) => None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether this is a tombstone.
|
||||
pub fn is_tombstone(&self) -> bool {
|
||||
matches!(self.body, PodcastRecordBody::Tombstone(_))
|
||||
}
|
||||
|
||||
/// Podcast.
|
||||
pub fn podcast_id(&self) -> &PodcastId {
|
||||
match &self.body {
|
||||
PodcastRecordBody::Card(card) => &card.podcast_id,
|
||||
PodcastRecordBody::Tombstone(tombstone) => &tombstone.podcast_id,
|
||||
}
|
||||
}
|
||||
|
||||
/// Publisher device.
|
||||
pub fn publisher(&self) -> EndpointId {
|
||||
match &self.body {
|
||||
PodcastRecordBody::Card(card) => card.publisher,
|
||||
PodcastRecordBody::Tombstone(tombstone) => tombstone.publisher,
|
||||
}
|
||||
}
|
||||
|
||||
/// Network scope.
|
||||
pub fn network_id(&self) -> NetworkId {
|
||||
match &self.body {
|
||||
PodcastRecordBody::Card(card) => card.network_id,
|
||||
PodcastRecordBody::Tombstone(tombstone) => tombstone.network_id,
|
||||
}
|
||||
}
|
||||
|
||||
/// Revision.
|
||||
pub fn revision(&self) -> u64 {
|
||||
match &self.body {
|
||||
PodcastRecordBody::Card(card) => card.revision,
|
||||
PodcastRecordBody::Tombstone(tombstone) => tombstone.revision,
|
||||
}
|
||||
}
|
||||
|
||||
/// Signing time.
|
||||
pub fn issued_at_ms(&self) -> u64 {
|
||||
match &self.body {
|
||||
PodcastRecordBody::Card(card) => card.issued_at_ms,
|
||||
PodcastRecordBody::Tombstone(tombstone) => tombstone.issued_at_ms,
|
||||
}
|
||||
}
|
||||
|
||||
/// Expiry time.
|
||||
pub fn expires_at_ms(&self) -> u64 {
|
||||
match &self.body {
|
||||
PodcastRecordBody::Card(card) => card.expires_at_ms,
|
||||
PodcastRecordBody::Tombstone(tombstone) => tombstone.expires_at_ms,
|
||||
}
|
||||
}
|
||||
|
||||
/// Every DHT key this record belongs under: the podcast-id key, then the
|
||||
/// text keys (card tokens or the tombstone's `index_keys`).
|
||||
pub fn dht_keys(&self) -> Vec<PodcastDhtKey> {
|
||||
let network_id = self.network_id();
|
||||
let mut keys = vec![PodcastDhtKey::podcast(&network_id, self.podcast_id())];
|
||||
match &self.body {
|
||||
PodcastRecordBody::Card(card) => keys.extend(
|
||||
card.index_tokens()
|
||||
.iter()
|
||||
.map(|token| PodcastDhtKey::token(&network_id, token)),
|
||||
),
|
||||
PodcastRecordBody::Tombstone(tombstone) => {
|
||||
keys.extend(tombstone.index_keys.iter().copied());
|
||||
}
|
||||
}
|
||||
keys
|
||||
}
|
||||
|
||||
/// Whether the record may be stored under `key`.
|
||||
pub fn belongs_under(&self, key: &PodcastDhtKey) -> bool {
|
||||
self.dht_keys().contains(key)
|
||||
}
|
||||
|
||||
/// postcard-encoded size.
|
||||
pub fn encoded_len(&self) -> usize {
|
||||
postcard::to_stdvec(self).map_or(usize::MAX, |bytes| bytes.len())
|
||||
}
|
||||
|
||||
fn check_size(&self) -> Result<()> {
|
||||
if self.encoded_len() > MAX_PODCAST_RECORD_BYTES {
|
||||
Err(invalid("podcast record is too large"))
|
||||
} else {
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Total order of two records of the same `(podcast_id, publisher)`.
|
||||
///
|
||||
/// Greater wins: higher `revision`; at an equal revision a tombstone beats a
|
||||
/// card; then later `issued_at_ms` (a republish); then later
|
||||
/// `expires_at_ms`; then the lexicographically greater canonical signing
|
||||
/// input. Every node therefore keeps the same winner whatever the delivery
|
||||
/// order.
|
||||
pub fn compare_records(left: &SignedPodcastRecord, right: &SignedPodcastRecord) -> Ordering {
|
||||
left.revision()
|
||||
.cmp(&right.revision())
|
||||
.then_with(|| left.is_tombstone().cmp(&right.is_tombstone()))
|
||||
.then_with(|| left.issued_at_ms().cmp(&right.issued_at_ms()))
|
||||
.then_with(|| left.expires_at_ms().cmp(&right.expires_at_ms()))
|
||||
.then_with(|| left.signing_bytes().cmp(&right.signing_bytes()))
|
||||
}
|
||||
|
||||
struct Canonical(Vec<u8>);
|
||||
|
||||
impl Canonical {
|
||||
fn new(domain: &str) -> Self {
|
||||
let mut bytes = Vec::with_capacity(512);
|
||||
bytes.extend_from_slice(domain.as_bytes());
|
||||
bytes.push(0);
|
||||
Self(bytes)
|
||||
}
|
||||
|
||||
fn raw(&mut self, bytes: &[u8]) {
|
||||
self.0.extend_from_slice(bytes);
|
||||
}
|
||||
|
||||
fn u8(&mut self, value: u8) {
|
||||
self.0.push(value);
|
||||
}
|
||||
|
||||
fn u16(&mut self, value: u16) {
|
||||
self.raw(&value.to_le_bytes());
|
||||
}
|
||||
|
||||
fn u32(&mut self, value: u32) {
|
||||
self.raw(&value.to_le_bytes());
|
||||
}
|
||||
|
||||
fn u64(&mut self, value: u64) {
|
||||
self.raw(&value.to_le_bytes());
|
||||
}
|
||||
|
||||
fn i64(&mut self, value: i64) {
|
||||
self.raw(&value.to_le_bytes());
|
||||
}
|
||||
|
||||
fn str(&mut self, value: &str) {
|
||||
self.u32(value.len() as u32);
|
||||
self.raw(value.as_bytes());
|
||||
}
|
||||
|
||||
fn opt_str(&mut self, value: Option<&str>) {
|
||||
match value {
|
||||
Some(value) => {
|
||||
self.u8(1);
|
||||
self.str(value);
|
||||
}
|
||||
None => self.u8(0),
|
||||
}
|
||||
}
|
||||
|
||||
fn finish(self) -> Vec<u8> {
|
||||
self.0
|
||||
}
|
||||
}
|
||||
|
||||
fn check_header(version: u16, network_id: &NetworkId, expected: &NetworkId) -> Result<()> {
|
||||
if version != PODCAST_DHT_PROTOCOL_VERSION {
|
||||
return Err(invalid("unsupported podcast record version"));
|
||||
}
|
||||
if network_id != expected {
|
||||
return Err(invalid("podcast record belongs to another network"));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn check_lifetime(
|
||||
revision: u64,
|
||||
issued_at_ms: u64,
|
||||
expires_at_ms: u64,
|
||||
ttl: Duration,
|
||||
) -> Result<()> {
|
||||
if revision == 0
|
||||
|| issued_at_ms == 0
|
||||
|| expires_at_ms <= issued_at_ms
|
||||
|| expires_at_ms - issued_at_ms > ttl.as_millis() as u64
|
||||
{
|
||||
return Err(invalid("invalid podcast record revision or lifetime"));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn check_chars(field: &'static str, value: &str, max_chars: usize) -> Result<()> {
|
||||
if value.len() > max_chars * 4 || value.chars().count() > max_chars {
|
||||
return Err(MusicDhtError::Protocol(format!("card {field} is too long")));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn check_line(field: &'static str, value: &str, max_chars: usize, required: bool) -> Result<()> {
|
||||
if required && value.trim().is_empty() {
|
||||
return Err(MusicDhtError::Protocol(format!("card {field} is empty")));
|
||||
}
|
||||
check_chars(field, value, max_chars)?;
|
||||
if value.chars().any(char::is_control) {
|
||||
return Err(MusicDhtError::Protocol(format!(
|
||||
"card {field} contains control characters"
|
||||
)));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn valid_language(language: &str) -> bool {
|
||||
!language.is_empty()
|
||||
&& language.len() <= MAX_CARD_LANGUAGE_BYTES
|
||||
&& language
|
||||
.bytes()
|
||||
.all(|byte| byte.is_ascii_alphanumeric() || byte == b'-')
|
||||
}
|
||||
|
||||
fn clean_line(text: &str) -> String {
|
||||
text.split(char::is_control)
|
||||
.filter(|part| !part.is_empty())
|
||||
.collect::<Vec<_>>()
|
||||
.join(" ")
|
||||
.trim()
|
||||
.to_string()
|
||||
}
|
||||
|
||||
fn clean_block(text: &str) -> String {
|
||||
text.chars()
|
||||
.filter(|c| !c.is_control() || *c == '\n' || *c == '\t')
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn bounded_line(text: Option<&str>, max_chars: usize) -> Option<String> {
|
||||
text.map(clean_line)
|
||||
.filter(|text| !text.is_empty())
|
||||
.map(|text| bounded_text(&text, max_chars))
|
||||
}
|
||||
|
||||
fn invalid(reason: &str) -> MusicDhtError {
|
||||
MusicDhtError::Protocol(reason.to_string())
|
||||
}
|
||||
|
||||
pub(crate) fn publication_error(reason: &str) -> MusicDhtError {
|
||||
MusicDhtError::PodcastPublication(reason.to_string())
|
||||
}
|
||||
|
||||
// Statically bound the worst-case card so `MAX_PODCAST_RECORD_BYTES` is a
|
||||
// defensive limit, not the one users hit first.
|
||||
const _: () = assert!(
|
||||
MAX_CARD_TITLE_CHARS * 4
|
||||
+ MAX_CARD_AUTHOR_CHARS * 4
|
||||
+ MAX_CARD_DESCRIPTION_CHARS * 4
|
||||
+ MAX_CARD_CATEGORIES * MAX_CARD_CATEGORY_CHARS * 4
|
||||
+ MAX_CARD_LANGUAGE_BYTES
|
||||
+ 4 * MAX_URL_BYTES
|
||||
+ 1024
|
||||
<= MAX_PODCAST_RECORD_BYTES
|
||||
);
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,339 @@
|
||||
//! Advisory detection of feed URLs that look private.
|
||||
//!
|
||||
//! Premium and member feeds usually carry a personal secret in the URL. A
|
||||
//! card publishes its feed URL verbatim, so the UI must ask the user before
|
||||
//! publishing a URL for which [`feed_url_privacy_warning`] returns a warning.
|
||||
//! The heuristics are deliberately broad and can produce false positives
|
||||
//! (and miss secrets); they never block publication by themselves.
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::fmt;
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
use crate::podcast::FeedUrl;
|
||||
|
||||
/// Query-parameter name words that usually carry a credential. A name is
|
||||
/// split into lowercase words at non-alphanumeric characters and camelCase
|
||||
/// boundaries; any listed word triggers a warning.
|
||||
pub const SECRET_QUERY_WORDS: &[&str] = &[
|
||||
"token",
|
||||
"key",
|
||||
"auth",
|
||||
"secret",
|
||||
"sig",
|
||||
"signature",
|
||||
"uid",
|
||||
"user",
|
||||
"pass",
|
||||
"password",
|
||||
"passwd",
|
||||
"pwd",
|
||||
"session",
|
||||
"sess",
|
||||
"sid",
|
||||
"access",
|
||||
"private",
|
||||
"credential",
|
||||
"credentials",
|
||||
"apikey",
|
||||
"jwt",
|
||||
"hmac",
|
||||
];
|
||||
|
||||
/// Substrings that trigger a warning anywhere in a query-parameter name,
|
||||
/// for names such as `apitoken` or `privatefeed` that do not split.
|
||||
const SECRET_QUERY_SUBSTRINGS: &[&str] = &[
|
||||
"token",
|
||||
"secret",
|
||||
"signature",
|
||||
"password",
|
||||
"passwd",
|
||||
"session",
|
||||
"private",
|
||||
"credential",
|
||||
"apikey",
|
||||
];
|
||||
|
||||
/// Hosts (and their subdomains) known to serve personal member feeds.
|
||||
pub const PRIVATE_FEED_HOSTS: &[&str] = &[
|
||||
"patreon.com",
|
||||
"supercast.com",
|
||||
"supportingcast.fm",
|
||||
"glow.fm",
|
||||
];
|
||||
|
||||
/// Minimum length of a query value considered a possible secret.
|
||||
const MIN_SECRET_VALUE_CHARS: usize = 16;
|
||||
/// Minimum length of a path segment considered a possible secret.
|
||||
const MIN_SECRET_SEGMENT_CHARS: usize = 32;
|
||||
/// Minimum Shannon entropy, in bits per character, of a possible secret.
|
||||
const MIN_SECRET_ENTROPY_BITS: f64 = 3.0;
|
||||
|
||||
/// Why a feed URL may contain a personal secret.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(tag = "reason", rename_all = "snake_case")]
|
||||
#[non_exhaustive]
|
||||
pub enum FeedUrlPrivacyWarning {
|
||||
/// A query parameter whose name suggests a credential.
|
||||
SecretQueryParameter {
|
||||
/// Parameter name as it appears in the URL.
|
||||
name: String,
|
||||
},
|
||||
/// The host is known to serve personal member feeds.
|
||||
PrivateFeedHost {
|
||||
/// Lowercase host.
|
||||
host: String,
|
||||
},
|
||||
/// A long, random-looking query value.
|
||||
HighEntropyQueryValue {
|
||||
/// Name of the parameter carrying the value.
|
||||
name: String,
|
||||
},
|
||||
/// A long, random-looking path segment.
|
||||
HighEntropyPathSegment {
|
||||
/// Zero-based index of the segment after the host.
|
||||
index: usize,
|
||||
},
|
||||
}
|
||||
|
||||
impl fmt::Display for FeedUrlPrivacyWarning {
|
||||
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
|
||||
match self {
|
||||
Self::SecretQueryParameter { name } => write!(
|
||||
f,
|
||||
"the feed URL has a query parameter \"{name}\" that looks like a credential"
|
||||
),
|
||||
Self::PrivateFeedHost { host } => {
|
||||
write!(f, "{host} usually serves personal member feeds")
|
||||
}
|
||||
Self::HighEntropyQueryValue { name } => write!(
|
||||
f,
|
||||
"the value of the query parameter \"{name}\" looks like a secret token"
|
||||
),
|
||||
Self::HighEntropyPathSegment { index } => write!(
|
||||
f,
|
||||
"path segment {} of the feed URL looks like a secret token",
|
||||
index + 1
|
||||
),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Returns why `url` may contain a personal secret, or `None`.
|
||||
///
|
||||
/// Advisory only (see the module documentation). Checks, in order: query
|
||||
/// parameter names ([`SECRET_QUERY_WORDS`]), known member-feed hosts
|
||||
/// ([`PRIVATE_FEED_HOSTS`]), long high-entropy query values, and long
|
||||
/// high-entropy path segments.
|
||||
pub fn feed_url_privacy_warning(url: &FeedUrl) -> Option<FeedUrlPrivacyWarning> {
|
||||
let value = url.as_str();
|
||||
let rest = value.split_once("://").map_or(value, |(_, rest)| rest);
|
||||
let (authority, path_and_query) = match rest.find('/') {
|
||||
Some(index) => rest.split_at(index),
|
||||
None => (rest, "/"),
|
||||
};
|
||||
let host = if authority.starts_with('[') {
|
||||
authority
|
||||
.find(']')
|
||||
.map_or(authority, |end| &authority[..=end])
|
||||
} else {
|
||||
authority.split(':').next().unwrap_or(authority)
|
||||
};
|
||||
let (path, query) = path_and_query
|
||||
.split_once('?')
|
||||
.unwrap_or((path_and_query, ""));
|
||||
let parameters: Vec<(&str, &str)> = query
|
||||
.split('&')
|
||||
.filter(|pair| !pair.is_empty())
|
||||
.map(|pair| pair.split_once('=').unwrap_or((pair, "")))
|
||||
.collect();
|
||||
|
||||
if let Some((name, _)) = parameters
|
||||
.iter()
|
||||
.find(|(name, _)| is_secret_parameter_name(name))
|
||||
{
|
||||
return Some(FeedUrlPrivacyWarning::SecretQueryParameter {
|
||||
name: (*name).to_string(),
|
||||
});
|
||||
}
|
||||
if PRIVATE_FEED_HOSTS
|
||||
.iter()
|
||||
.any(|known| host == *known || host.ends_with(&format!(".{known}")))
|
||||
{
|
||||
return Some(FeedUrlPrivacyWarning::PrivateFeedHost {
|
||||
host: host.to_string(),
|
||||
});
|
||||
}
|
||||
if let Some((name, _)) = parameters
|
||||
.iter()
|
||||
.find(|(_, value)| looks_random(value, MIN_SECRET_VALUE_CHARS))
|
||||
{
|
||||
return Some(FeedUrlPrivacyWarning::HighEntropyQueryValue {
|
||||
name: (*name).to_string(),
|
||||
});
|
||||
}
|
||||
path.split('/')
|
||||
.filter(|segment| !segment.is_empty())
|
||||
.position(|segment| {
|
||||
let stem = segment.rsplit_once('.').map_or(segment, |(stem, _)| stem);
|
||||
looks_random(stem, MIN_SECRET_SEGMENT_CHARS)
|
||||
})
|
||||
.map(|index| FeedUrlPrivacyWarning::HighEntropyPathSegment { index })
|
||||
}
|
||||
|
||||
fn is_secret_parameter_name(name: &str) -> bool {
|
||||
let words = split_words(name);
|
||||
if words
|
||||
.iter()
|
||||
.any(|word| SECRET_QUERY_WORDS.contains(&word.as_str()))
|
||||
{
|
||||
return true;
|
||||
}
|
||||
let compact: String = name
|
||||
.chars()
|
||||
.filter(char::is_ascii_alphanumeric)
|
||||
.collect::<String>()
|
||||
.to_ascii_lowercase();
|
||||
SECRET_QUERY_SUBSTRINGS
|
||||
.iter()
|
||||
.any(|needle| compact.contains(needle))
|
||||
}
|
||||
|
||||
/// Splits at non-alphanumeric characters and lower-to-upper case changes.
|
||||
fn split_words(name: &str) -> Vec<String> {
|
||||
let mut words = Vec::new();
|
||||
let mut current = String::new();
|
||||
let mut previous_lower = false;
|
||||
for c in name.chars() {
|
||||
if !c.is_ascii_alphanumeric() {
|
||||
if !current.is_empty() {
|
||||
words.push(std::mem::take(&mut current));
|
||||
}
|
||||
previous_lower = false;
|
||||
continue;
|
||||
}
|
||||
if c.is_ascii_uppercase() && previous_lower && !current.is_empty() {
|
||||
words.push(std::mem::take(&mut current));
|
||||
}
|
||||
previous_lower = c.is_ascii_lowercase() || c.is_ascii_digit();
|
||||
current.push(c.to_ascii_lowercase());
|
||||
}
|
||||
if !current.is_empty() {
|
||||
words.push(current);
|
||||
}
|
||||
words
|
||||
}
|
||||
|
||||
/// A long token-alphabet string mixing letters and digits (or both letter
|
||||
/// cases) with high per-character entropy.
|
||||
fn looks_random(value: &str, min_chars: usize) -> bool {
|
||||
if value.chars().count() < min_chars
|
||||
|| !value
|
||||
.chars()
|
||||
.all(|c| c.is_ascii_alphanumeric() || "-_.~%=+".contains(c))
|
||||
{
|
||||
return false;
|
||||
}
|
||||
let has_digit = value.chars().any(|c| c.is_ascii_digit());
|
||||
let has_lower = value.chars().any(|c| c.is_ascii_lowercase());
|
||||
let has_upper = value.chars().any(|c| c.is_ascii_uppercase());
|
||||
let classes = [has_digit, has_lower, has_upper]
|
||||
.into_iter()
|
||||
.filter(|present| *present)
|
||||
.count();
|
||||
classes >= 2 && shannon_entropy(value) >= MIN_SECRET_ENTROPY_BITS
|
||||
}
|
||||
|
||||
fn shannon_entropy(value: &str) -> f64 {
|
||||
let mut counts: HashMap<char, usize> = HashMap::new();
|
||||
for c in value.chars() {
|
||||
*counts.entry(c).or_default() += 1;
|
||||
}
|
||||
let total = value.chars().count() as f64;
|
||||
counts
|
||||
.values()
|
||||
.map(|count| {
|
||||
let p = *count as f64 / total;
|
||||
-p * p.log2()
|
||||
})
|
||||
.sum()
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn warn(url: &str) -> Option<FeedUrlPrivacyWarning> {
|
||||
feed_url_privacy_warning(&FeedUrl::parse(url).unwrap())
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn public_feeds_have_no_warning() {
|
||||
for url in [
|
||||
"https://feeds.example.com/podcast.xml",
|
||||
"https://example.com/feed?format=rss&page=2",
|
||||
"https://anchor.example/s/abc123de/podcast/rss",
|
||||
"https://feeds.example.com/12345/rss",
|
||||
"https://example.com/feed?author=jane",
|
||||
"https://example.com:8443/shows/my-great-show/feed.xml",
|
||||
] {
|
||||
assert_eq!(warn(url), None, "{url}");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn secret_query_names_warn() {
|
||||
for (url, name) in [
|
||||
("https://example.com/feed?token=abc", "token"),
|
||||
("https://example.com/feed?auth_token=abc", "auth_token"),
|
||||
(
|
||||
"https://example.com/feed?X-Amz-Signature=abc",
|
||||
"X-Amz-Signature",
|
||||
),
|
||||
("https://example.com/feed?accessKey=abc", "accessKey"),
|
||||
("https://example.com/feed?a=1&uid=42", "uid"),
|
||||
("https://example.com/feed?apitoken=x", "apitoken"),
|
||||
("https://example.com/feed?PRIVATE=1", "PRIVATE"),
|
||||
("https://example.com/feed?session_id=1", "session_id"),
|
||||
] {
|
||||
assert_eq!(
|
||||
warn(url),
|
||||
Some(FeedUrlPrivacyWarning::SecretQueryParameter { name: name.into() }),
|
||||
"{url}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn known_member_hosts_warn() {
|
||||
assert_eq!(
|
||||
warn("https://www.patreon.com/rss/show"),
|
||||
Some(FeedUrlPrivacyWarning::PrivateFeedHost {
|
||||
host: "www.patreon.com".into()
|
||||
})
|
||||
);
|
||||
assert_eq!(warn("https://notpatreon.com/rss"), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn high_entropy_values_and_segments_warn() {
|
||||
assert_eq!(
|
||||
warn("https://example.com/feed?c=Zx81kQ2mP0aLr7Vb"),
|
||||
Some(FeedUrlPrivacyWarning::HighEntropyQueryValue { name: "c".into() })
|
||||
);
|
||||
assert_eq!(warn("https://example.com/feed?c=aaaaaaaaaaaaaaaa1"), None);
|
||||
assert_eq!(
|
||||
warn("https://example.com/feeds/q8Zr2LmK0pXv7TnB4sYc1HdWe9GuJa3F.rss"),
|
||||
Some(FeedUrlPrivacyWarning::HighEntropyPathSegment { index: 1 })
|
||||
);
|
||||
// Below the segment length: an ordinary short id.
|
||||
assert_eq!(warn("https://example.com/feeds/q8Zr2LmK0pXv7TnB.rss"), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn warnings_render_without_the_secret() {
|
||||
let warning = warn("https://example.com/feed?c=Zx81kQ2mP0aLr7Vb").unwrap();
|
||||
assert!(!warning.to_string().contains("Zx81"));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,296 @@
|
||||
//! Podcast search hits and their merge with music search results.
|
||||
|
||||
use std::cmp::Reverse;
|
||||
use std::collections::{BTreeMap, HashMap, HashSet};
|
||||
use std::fmt;
|
||||
|
||||
use federation_net::EndpointId;
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
use crate::normalization::{normalize_name, tokenize};
|
||||
use crate::podcast::{FeedUrl, PodcastId};
|
||||
use crate::record::{ItemKind, LibraryItem};
|
||||
|
||||
use super::card::{MAX_INDEX_TOKEN_BYTES, PodcastCard, SignedPodcastRecord, compare_records};
|
||||
|
||||
/// Maximum distinct query tokens used by a podcast text search.
|
||||
pub const MAX_PODCAST_QUERY_TOKENS: usize = 8;
|
||||
/// Maximum publishers reported per hit and per claimed feed URL.
|
||||
pub const MAX_HIT_PUBLISHERS: usize = 64;
|
||||
/// Maximum claimed moved feed URLs reported per hit.
|
||||
pub const MAX_HIT_FEED_URL_CLAIMS: usize = 8;
|
||||
|
||||
/// Kind of a federated search hit.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "lowercase")]
|
||||
pub enum SearchHitKind {
|
||||
/// A musical artist.
|
||||
Artist,
|
||||
/// A release.
|
||||
Release,
|
||||
/// A track.
|
||||
Track,
|
||||
/// A public podcast card.
|
||||
Podcast,
|
||||
}
|
||||
|
||||
impl SearchHitKind {
|
||||
/// Stable lowercase name.
|
||||
pub fn as_str(&self) -> &'static str {
|
||||
match self {
|
||||
Self::Artist => "artist",
|
||||
Self::Release => "release",
|
||||
Self::Track => "track",
|
||||
Self::Podcast => "podcast",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl fmt::Display for SearchHitKind {
|
||||
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
|
||||
f.write_str(self.as_str())
|
||||
}
|
||||
}
|
||||
|
||||
impl From<ItemKind> for SearchHitKind {
|
||||
fn from(kind: ItemKind) -> Self {
|
||||
match kind {
|
||||
ItemKind::Artist => Self::Artist,
|
||||
ItemKind::Release => Self::Release,
|
||||
ItemKind::Track => Self::Track,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// One public podcast found in the discovery overlay.
|
||||
///
|
||||
/// Cards of the same podcast published by several devices are merged by
|
||||
/// `podcast_id`. The hit carries no personal state of any publisher.
|
||||
///
|
||||
/// Only [`Self::feed_url`] is bound to the podcast: the id is its hash. Every
|
||||
/// other field is a claim of some publisher device, and device identities are
|
||||
/// free, so neither the metadata nor the number of publishers is a trust
|
||||
/// signal.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct PodcastSearchHit {
|
||||
/// Podcast.
|
||||
pub podcast_id: PodcastId,
|
||||
/// The URL `podcast_id` is derived from
|
||||
/// (`PodcastId::from_feed_url(feed_url) == podcast_id`). Subscribe with
|
||||
/// this URL; follow the feed's own HTTP redirects and
|
||||
/// `itunes:new-feed-url` from there.
|
||||
pub feed_url: FeedUrl,
|
||||
/// Representative card for display: a card whose `feed_url` is the
|
||||
/// origin URL when one exists, then the title most publishers agree on,
|
||||
/// then the smallest publisher key. Its metadata is a publisher's claim.
|
||||
pub card: PodcastCard,
|
||||
/// Sorted distinct publisher devices with a current valid card, at most
|
||||
/// [`MAX_HIT_PUBLISHERS`]. These are devices, not users or subscribers:
|
||||
/// one user may publish from several devices, and anyone can mint
|
||||
/// devices.
|
||||
pub publishers: Vec<EndpointId>,
|
||||
/// Feed URLs other than [`Self::feed_url`] that publishers claim the
|
||||
/// podcast moved to, most claimed first, at most
|
||||
/// [`MAX_HIT_FEED_URL_CLAIMS`]. Unverified: never fetch one as this
|
||||
/// podcast or record it as an alias unless the feed at
|
||||
/// [`Self::feed_url`] itself redirects there.
|
||||
pub claimed_feed_urls: Vec<ClaimedFeedUrl>,
|
||||
}
|
||||
|
||||
impl PodcastSearchHit {
|
||||
/// Always [`SearchHitKind::Podcast`].
|
||||
pub fn kind(&self) -> SearchHitKind {
|
||||
SearchHitKind::Podcast
|
||||
}
|
||||
}
|
||||
|
||||
/// An unverified moved-feed URL claimed by some publishers of a podcast.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct ClaimedFeedUrl {
|
||||
/// Claimed current feed URL.
|
||||
pub feed_url: FeedUrl,
|
||||
/// Sorted distinct publisher devices claiming it, at most
|
||||
/// [`MAX_HIT_PUBLISHERS`].
|
||||
pub publishers: Vec<EndpointId>,
|
||||
}
|
||||
|
||||
/// A music or podcast result of a federated search.
|
||||
///
|
||||
/// Serialized as `{"type": "music" | "podcast", "hit": {...}}`; use
|
||||
/// [`FederatedSearchHit::kind`] for the fine-grained kind.
|
||||
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
|
||||
#[serde(tag = "type", content = "hit", rename_all = "snake_case")]
|
||||
pub enum FederatedSearchHit {
|
||||
/// A music-DHT record (artist, release or track).
|
||||
Music(LibraryItem),
|
||||
/// A public podcast.
|
||||
Podcast(PodcastSearchHit),
|
||||
}
|
||||
|
||||
impl FederatedSearchHit {
|
||||
/// Kind of the hit: `artist`, `release`, `track` or `podcast`.
|
||||
pub fn kind(&self) -> SearchHitKind {
|
||||
match self {
|
||||
Self::Music(item) => item.kind.into(),
|
||||
Self::Podcast(_) => SearchHitKind::Podcast,
|
||||
}
|
||||
}
|
||||
|
||||
/// Display name: the item name or the podcast title.
|
||||
pub fn title(&self) -> &str {
|
||||
match self {
|
||||
Self::Music(item) => &item.name,
|
||||
Self::Podcast(hit) => &hit.card.title,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Merges music results (as returned by `MusicDhtService::search_network`)
|
||||
/// with podcast hits for the same query.
|
||||
///
|
||||
/// Results whose normalized name equals the normalized query come first;
|
||||
/// within each tier music keeps its order and precedes podcasts, which keep
|
||||
/// theirs. Existing music consumers that only read `SearchOutcome` are
|
||||
/// unaffected.
|
||||
pub fn merge_search_results(
|
||||
query: &str,
|
||||
music: Vec<LibraryItem>,
|
||||
podcasts: Vec<PodcastSearchHit>,
|
||||
) -> Vec<FederatedSearchHit> {
|
||||
let normalized = normalize_name(query);
|
||||
let mut exact = Vec::new();
|
||||
let mut rest = Vec::new();
|
||||
let hits = music
|
||||
.into_iter()
|
||||
.map(FederatedSearchHit::Music)
|
||||
.chain(podcasts.into_iter().map(FederatedSearchHit::Podcast));
|
||||
for hit in hits {
|
||||
let is_exact = !normalized.is_empty() && normalize_name(hit.title()) == normalized;
|
||||
if is_exact {
|
||||
exact.push(hit);
|
||||
} else {
|
||||
rest.push(hit);
|
||||
}
|
||||
}
|
||||
exact.extend(rest);
|
||||
exact
|
||||
}
|
||||
|
||||
/// Distinct normalized query tokens, at most [`MAX_PODCAST_QUERY_TOKENS`].
|
||||
pub fn podcast_query_tokens(query: &str) -> Vec<String> {
|
||||
let mut seen = HashSet::new();
|
||||
tokenize(&normalize_name(query))
|
||||
.into_iter()
|
||||
.filter(|token| token.len() <= MAX_INDEX_TOKEN_BYTES && seen.insert(token.clone()))
|
||||
.take(MAX_PODCAST_QUERY_TOKENS)
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Keeps the winning record per `(podcast_id, publisher)` and groups the
|
||||
/// cards that are still active by podcast. Records must be verified.
|
||||
pub(crate) fn aggregate_hits(
|
||||
records: impl IntoIterator<Item = SignedPodcastRecord>,
|
||||
) -> Vec<PodcastSearchHit> {
|
||||
let mut winners: HashMap<(PodcastId, EndpointId), SignedPodcastRecord> = HashMap::new();
|
||||
for record in records {
|
||||
let owner = (record.podcast_id().clone(), record.publisher());
|
||||
match winners.get(&owner) {
|
||||
Some(current) if compare_records(current, &record).is_ge() => {}
|
||||
_ => {
|
||||
winners.insert(owner, record);
|
||||
}
|
||||
}
|
||||
}
|
||||
let mut by_podcast: BTreeMap<PodcastId, Vec<PodcastCard>> = BTreeMap::new();
|
||||
for record in winners.into_values() {
|
||||
if let Some(card) = record.card() {
|
||||
by_podcast
|
||||
.entry(card.podcast_id.clone())
|
||||
.or_default()
|
||||
.push(card.clone());
|
||||
}
|
||||
}
|
||||
by_podcast
|
||||
.into_iter()
|
||||
.map(|(podcast_id, cards)| {
|
||||
let mut title_votes: HashMap<String, usize> = HashMap::new();
|
||||
for card in &cards {
|
||||
*title_votes.entry(normalize_name(&card.title)).or_default() += 1;
|
||||
}
|
||||
// Publisher-controlled times and revisions never break ties.
|
||||
let card = cards
|
||||
.iter()
|
||||
.max_by_key(|card| {
|
||||
(
|
||||
card.feed_url_is_origin(),
|
||||
title_votes[&normalize_name(&card.title)],
|
||||
Reverse(*card.publisher.as_bytes()),
|
||||
)
|
||||
})
|
||||
.cloned()
|
||||
.expect("a group has at least one card");
|
||||
let mut claims: BTreeMap<&FeedUrl, Vec<EndpointId>> = BTreeMap::new();
|
||||
for card in cards.iter().filter(|card| !card.feed_url_is_origin()) {
|
||||
claims
|
||||
.entry(&card.feed_url)
|
||||
.or_default()
|
||||
.push(card.publisher);
|
||||
}
|
||||
let mut claimed_feed_urls: Vec<ClaimedFeedUrl> = claims
|
||||
.into_iter()
|
||||
.map(|(feed_url, publishers)| ClaimedFeedUrl {
|
||||
feed_url: feed_url.clone(),
|
||||
publishers: sorted_publishers(publishers),
|
||||
})
|
||||
.collect();
|
||||
claimed_feed_urls.sort_by(|left, right| {
|
||||
right
|
||||
.publishers
|
||||
.len()
|
||||
.cmp(&left.publishers.len())
|
||||
.then_with(|| left.feed_url.cmp(&right.feed_url))
|
||||
});
|
||||
claimed_feed_urls.truncate(MAX_HIT_FEED_URL_CLAIMS);
|
||||
PodcastSearchHit {
|
||||
podcast_id,
|
||||
feed_url: card.origin_feed_url().clone(),
|
||||
publishers: sorted_publishers(cards.iter().map(|card| card.publisher).collect()),
|
||||
card,
|
||||
claimed_feed_urls,
|
||||
}
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn sorted_publishers(mut publishers: Vec<EndpointId>) -> Vec<EndpointId> {
|
||||
publishers.sort_by_key(|publisher| *publisher.as_bytes());
|
||||
publishers.dedup();
|
||||
publishers.truncate(MAX_HIT_PUBLISHERS);
|
||||
publishers
|
||||
}
|
||||
|
||||
/// Keeps hits whose card is indexed under every query token and ranks
|
||||
/// them: exact normalized title first, then hits with a card published under
|
||||
/// the origin feed URL, then title and id. The publisher count is not used:
|
||||
/// anyone can inflate it with new device identities.
|
||||
pub(crate) fn rank_hits(
|
||||
mut hits: Vec<PodcastSearchHit>,
|
||||
query: &str,
|
||||
tokens: &[String],
|
||||
) -> Vec<PodcastSearchHit> {
|
||||
let normalized_query = normalize_name(query);
|
||||
hits.retain(|hit| {
|
||||
let indexed = hit.card.index_tokens();
|
||||
tokens.iter().all(|token| indexed.contains(token))
|
||||
});
|
||||
hits.sort_by_cached_key(|hit| {
|
||||
let title = normalize_name(&hit.card.title);
|
||||
(
|
||||
title != normalized_query,
|
||||
!hit.card.feed_url_is_origin(),
|
||||
title,
|
||||
hit.podcast_id.clone(),
|
||||
)
|
||||
});
|
||||
hits
|
||||
}
|
||||
@@ -0,0 +1,891 @@
|
||||
//! SQLite cache of the podcast-discovery overlay: replicas, routing contacts
|
||||
//! and this device's own publications.
|
||||
//!
|
||||
//! The cache is replaceable federation state, kept in its own database file
|
||||
//! (like the similarity-routing cache). Every record read from it is decoded
|
||||
//! and verified again before it is used or served.
|
||||
|
||||
use std::collections::HashSet;
|
||||
use std::path::Path;
|
||||
use std::str::FromStr;
|
||||
use std::sync::{Arc, Mutex, MutexGuard, PoisonError};
|
||||
|
||||
use federation_net::{EndpointId, NetworkId};
|
||||
use rusqlite::{Connection, OptionalExtension, params};
|
||||
|
||||
use crate::error::{MusicDhtError, Result};
|
||||
use crate::podcast::PodcastId;
|
||||
use crate::routing::{NodeContact, NodeId};
|
||||
|
||||
use super::MAX_PODCAST_DHT_RECORDS;
|
||||
use super::card::{PodcastCardSpec, PodcastDhtKey, SignedPodcastRecord, compare_records};
|
||||
|
||||
/// `PRAGMA user_version` of the podcast-discovery cache schema. Version 1
|
||||
/// (pre-release) stored one payload per key; its replica table is dropped on
|
||||
/// open because the cache is replaceable. Own publications are kept.
|
||||
pub const PODCAST_DHT_CACHE_SCHEMA_VERSION: i32 = 2;
|
||||
/// Maximum records (cards and tombstones) indexed under one key. It equals
|
||||
/// [`MAX_PODCAST_DHT_RECORDS`], so one `FindValue` response carries every
|
||||
/// record of a key: nothing admitted is unreachable.
|
||||
///
|
||||
/// A full key admits a newcomer by evicting a random placement from its
|
||||
/// younger half (by first indexing time). The older half is never evicted
|
||||
/// for newcomers, so cards that were there before a flood stay, and the
|
||||
/// younger half stays contestable, so nobody who stored first holds the key
|
||||
/// for good.
|
||||
pub const MAX_PODCAST_RECORDS_PER_KEY: usize = MAX_PODCAST_DHT_RECORDS;
|
||||
/// Maximum tombstones indexed under one key, including tombstones that
|
||||
/// replaced their owner's card in place. Over budget, a random tombstone of
|
||||
/// another owner from the younger half of the key's tombstones is evicted.
|
||||
/// It equals [`MAX_PODCAST_RESPONSE_TOMBSTONES`], so every tombstone of a key
|
||||
/// is served.
|
||||
pub const MAX_PODCAST_TOMBSTONES_PER_KEY: usize = MAX_PODCAST_RESPONSE_TOMBSTONES;
|
||||
/// Maximum tombstones in one `FindValue` response; cards fill the rest.
|
||||
pub const MAX_PODCAST_RESPONSE_TOMBSTONES: usize = 32;
|
||||
/// Maximum `(podcast_id, publisher)` records one publisher device keeps in a
|
||||
/// replica. A publisher over it is refused (nothing is evicted for it).
|
||||
pub const MAX_PODCAST_RECORDS_PER_PUBLISHER: usize = 256;
|
||||
/// Maximum `(podcast_id, publisher)` records in a replica cache. A full cache
|
||||
/// admits a new record by evicting random records from the younger half of
|
||||
/// the cache (by first indexing time), like a full key.
|
||||
pub const MAX_PODCAST_CACHE_RECORDS: usize = 4_096;
|
||||
/// Maximum total encoded size of the records in a replica cache, enforced
|
||||
/// like [`MAX_PODCAST_CACHE_RECORDS`].
|
||||
pub const MAX_PODCAST_CACHE_BYTES: usize = 32 * 1024 * 1024;
|
||||
/// Maximum persisted overlay contacts; the least recently seen are dropped.
|
||||
pub const MAX_PERSISTED_PODCAST_DHT_PEERS: usize = 1_024;
|
||||
/// Maximum records evicted from the cache to admit one record; when that is
|
||||
/// not enough room, the record is refused and nothing is evicted.
|
||||
const MAX_EVICTIONS_PER_RECORD: usize = 16;
|
||||
|
||||
const _: () = assert!(
|
||||
MAX_PODCAST_RECORDS_PER_KEY <= MAX_PODCAST_DHT_RECORDS
|
||||
&& MAX_PODCAST_TOMBSTONES_PER_KEY <= MAX_PODCAST_RESPONSE_TOMBSTONES
|
||||
&& MAX_PODCAST_RESPONSE_TOMBSTONES <= MAX_PODCAST_DHT_RECORDS,
|
||||
"every record admitted under a key must fit one FindValue response"
|
||||
);
|
||||
|
||||
const SCHEMA: &str = "
|
||||
CREATE TABLE IF NOT EXISTS podcast_dht_records (
|
||||
podcast_id TEXT NOT NULL,
|
||||
publisher TEXT NOT NULL,
|
||||
tombstone INTEGER NOT NULL,
|
||||
expires_at_ms INTEGER NOT NULL,
|
||||
payload_bytes INTEGER NOT NULL,
|
||||
payload BLOB NOT NULL,
|
||||
PRIMARY KEY (podcast_id, publisher)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_podcast_dht_records_expiry
|
||||
ON podcast_dht_records(expires_at_ms);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_podcast_dht_records_publisher
|
||||
ON podcast_dht_records(publisher);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS podcast_dht_keys (
|
||||
dht_key BLOB NOT NULL,
|
||||
podcast_id TEXT NOT NULL,
|
||||
publisher TEXT NOT NULL,
|
||||
tombstone INTEGER NOT NULL,
|
||||
expires_at_ms INTEGER NOT NULL,
|
||||
first_stored_at_ms INTEGER NOT NULL,
|
||||
PRIMARY KEY (dht_key, podcast_id, publisher)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_podcast_dht_keys_owner
|
||||
ON podcast_dht_keys(podcast_id, publisher);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_podcast_dht_keys_expiry
|
||||
ON podcast_dht_keys(expires_at_ms);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS podcast_dht_peers (
|
||||
peer_id TEXT PRIMARY KEY,
|
||||
node_id BLOB NOT NULL,
|
||||
ticket TEXT NOT NULL,
|
||||
last_seen_ms INTEGER NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS podcast_dht_local (
|
||||
podcast_id TEXT PRIMARY KEY,
|
||||
revision INTEGER NOT NULL,
|
||||
spec BLOB,
|
||||
record BLOB NOT NULL,
|
||||
updated_at_ms INTEGER NOT NULL
|
||||
);
|
||||
";
|
||||
|
||||
/// Admission limits of a replica cache.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub(crate) struct StoreLimits {
|
||||
pub per_key: usize,
|
||||
pub tombstones_per_key: usize,
|
||||
pub per_publisher: usize,
|
||||
pub records: usize,
|
||||
pub bytes: usize,
|
||||
}
|
||||
|
||||
impl Default for StoreLimits {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
per_key: MAX_PODCAST_RECORDS_PER_KEY,
|
||||
tombstones_per_key: MAX_PODCAST_TOMBSTONES_PER_KEY,
|
||||
per_publisher: MAX_PODCAST_RECORDS_PER_PUBLISHER,
|
||||
records: MAX_PODCAST_CACHE_RECORDS,
|
||||
bytes: MAX_PODCAST_CACHE_BYTES,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// This device's latest publication of one podcast.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub(crate) struct LocalPublication {
|
||||
pub podcast_id: PodcastId,
|
||||
/// Greatest revision ever used for this podcast by this device.
|
||||
pub revision: u64,
|
||||
/// Published card content; `None` once unpublished.
|
||||
pub spec: Option<PodcastCardSpec>,
|
||||
/// Latest signed record (card or tombstone).
|
||||
pub record: SignedPodcastRecord,
|
||||
}
|
||||
|
||||
#[derive(Clone)]
|
||||
pub(crate) struct PodcastDhtDatabase {
|
||||
conn: Arc<Mutex<Connection>>,
|
||||
network_id: NetworkId,
|
||||
limits: StoreLimits,
|
||||
}
|
||||
|
||||
impl std::fmt::Debug for PodcastDhtDatabase {
|
||||
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
formatter
|
||||
.debug_struct("PodcastDhtDatabase")
|
||||
.finish_non_exhaustive()
|
||||
}
|
||||
}
|
||||
|
||||
impl PodcastDhtDatabase {
|
||||
pub async fn open(path: &Path, network_id: NetworkId) -> Result<Self> {
|
||||
Self::open_with_limits(path, network_id, StoreLimits::default()).await
|
||||
}
|
||||
|
||||
pub async fn open_with_limits(
|
||||
path: &Path,
|
||||
network_id: NetworkId,
|
||||
limits: StoreLimits,
|
||||
) -> Result<Self> {
|
||||
let path = path.to_path_buf();
|
||||
let conn = tokio::task::spawn_blocking(move || -> Result<Connection> {
|
||||
let conn = Connection::open(&path).map_err(|error| {
|
||||
MusicDhtError::Database(format!("failed to open {}: {error}", path.display()))
|
||||
})?;
|
||||
let version: i32 = conn
|
||||
.query_row("PRAGMA user_version", [], |row| row.get(0))
|
||||
.map_err(database_error)?;
|
||||
if version > PODCAST_DHT_CACHE_SCHEMA_VERSION {
|
||||
return Err(MusicDhtError::Database(format!(
|
||||
"podcast DHT cache schema {version} is newer than supported {PODCAST_DHT_CACHE_SCHEMA_VERSION}"
|
||||
)));
|
||||
}
|
||||
if version == 1 {
|
||||
// Replicas are replaceable; the old layout is dropped.
|
||||
conn.execute_batch("DROP TABLE IF EXISTS podcast_dht_records;")
|
||||
.map_err(database_error)?;
|
||||
}
|
||||
conn.execute_batch(SCHEMA).map_err(|error| {
|
||||
MusicDhtError::Database(format!("failed to initialize podcast DHT cache: {error}"))
|
||||
})?;
|
||||
conn.pragma_update(None, "user_version", PODCAST_DHT_CACHE_SCHEMA_VERSION)
|
||||
.map_err(database_error)?;
|
||||
Ok(conn)
|
||||
})
|
||||
.await
|
||||
.map_err(|error| MusicDhtError::Database(format!("database task panicked: {error}")))??;
|
||||
Ok(Self {
|
||||
conn: Arc::new(Mutex::new(conn)),
|
||||
network_id,
|
||||
limits,
|
||||
})
|
||||
}
|
||||
|
||||
async fn call<F, T>(&self, operation: F) -> Result<T>
|
||||
where
|
||||
F: FnOnce(&Connection, &NetworkId, &StoreLimits) -> rusqlite::Result<T> + Send + 'static,
|
||||
T: Send + 'static,
|
||||
{
|
||||
let conn = Arc::clone(&self.conn);
|
||||
let network_id = self.network_id;
|
||||
let limits = self.limits;
|
||||
tokio::task::spawn_blocking(move || {
|
||||
let guard = lock(&conn);
|
||||
operation(&guard, &network_id, &limits).map_err(database_error)
|
||||
})
|
||||
.await
|
||||
.map_err(|error| MusicDhtError::Database(format!("database task panicked: {error}")))?
|
||||
}
|
||||
|
||||
/// Stores records under the given keys. The caller has verified every
|
||||
/// record, checked that each key belongs to it and that the sender is its
|
||||
/// publisher. Returns the number of `(key, record)` placements written.
|
||||
pub async fn store(
|
||||
&self,
|
||||
entries: Vec<(Vec<PodcastDhtKey>, SignedPodcastRecord)>,
|
||||
now_ms: u64,
|
||||
) -> Result<usize> {
|
||||
self.call(move |conn, network_id, limits| {
|
||||
let transaction = conn.unchecked_transaction()?;
|
||||
let mut written = 0;
|
||||
for (keys, record) in entries {
|
||||
written += store_one(&transaction, network_id, limits, &keys, &record, now_ms)?;
|
||||
}
|
||||
transaction.commit()?;
|
||||
Ok(written)
|
||||
})
|
||||
.await
|
||||
}
|
||||
|
||||
/// Valid, unexpired records under `key`: at most
|
||||
/// [`MAX_PODCAST_RESPONSE_TOMBSTONES`] tombstones (always returned so stale
|
||||
/// replicas elsewhere can be overruled) and cards containing every token
|
||||
/// in `filter_tokens` up to `limit` records in all. Records indexed longest
|
||||
/// come first. Cards precede tombstones. With the default limits and
|
||||
/// `limit` = [`MAX_PODCAST_DHT_RECORDS`], every record of the key fits.
|
||||
pub async fn records_by_key(
|
||||
&self,
|
||||
key: PodcastDhtKey,
|
||||
filter_tokens: Vec<String>,
|
||||
now_ms: u64,
|
||||
limit: usize,
|
||||
) -> Result<Vec<SignedPodcastRecord>> {
|
||||
self.call(move |conn, network_id, _| {
|
||||
let mut statement = conn.prepare(
|
||||
"SELECT r.payload FROM podcast_dht_keys k
|
||||
JOIN podcast_dht_records r
|
||||
ON r.podcast_id = k.podcast_id AND r.publisher = k.publisher
|
||||
WHERE k.dht_key = ?1 AND k.tombstone = ?2 AND k.expires_at_ms > ?3
|
||||
ORDER BY k.first_stored_at_ms, k.podcast_id, k.publisher",
|
||||
)?;
|
||||
let mut select = |tombstone: bool, limit: usize| -> rusqlite::Result<Vec<_>> {
|
||||
let mut records = Vec::new();
|
||||
if limit == 0 {
|
||||
return Ok(records);
|
||||
}
|
||||
let rows = statement.query_map(
|
||||
params![key.as_bytes().as_slice(), tombstone, clamp_i64(now_ms)],
|
||||
|row| row.get::<_, Vec<u8>>(0),
|
||||
)?;
|
||||
for row in rows {
|
||||
let Some(record) = decode_valid(&row?, network_id, now_ms) else {
|
||||
continue;
|
||||
};
|
||||
if record.is_tombstone() != tombstone || !record.belongs_under(&key) {
|
||||
continue;
|
||||
}
|
||||
let matches = record.card().is_none_or(|card| {
|
||||
let tokens = card.index_tokens();
|
||||
filter_tokens.iter().all(|token| tokens.contains(token))
|
||||
});
|
||||
if matches {
|
||||
records.push(record);
|
||||
if records.len() == limit {
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(records)
|
||||
};
|
||||
let tombstones = select(true, MAX_PODCAST_RESPONSE_TOMBSTONES.min(limit))?;
|
||||
let mut records = select(false, limit - tombstones.len())?;
|
||||
records.extend(tombstones);
|
||||
Ok(records)
|
||||
})
|
||||
.await
|
||||
}
|
||||
|
||||
/// Removes expired records and their key placements. Returns the number
|
||||
/// of records removed.
|
||||
pub async fn delete_expired(&self, now_ms: u64) -> Result<usize> {
|
||||
self.call(move |conn, _, _| {
|
||||
let now = clamp_i64(now_ms);
|
||||
conn.execute(
|
||||
"DELETE FROM podcast_dht_keys WHERE expires_at_ms <= ?1",
|
||||
[now],
|
||||
)?;
|
||||
conn.execute(
|
||||
"DELETE FROM podcast_dht_records WHERE expires_at_ms <= ?1",
|
||||
[now],
|
||||
)
|
||||
})
|
||||
.await
|
||||
}
|
||||
|
||||
pub async fn upsert_peer(&self, contact: NodeContact) -> Result<()> {
|
||||
self.call(move |conn, _, _| {
|
||||
conn.execute(
|
||||
"INSERT OR REPLACE INTO podcast_dht_peers
|
||||
(peer_id, node_id, ticket, last_seen_ms)
|
||||
VALUES (?1, ?2, ?3, ?4)",
|
||||
params![
|
||||
contact.peer_id.to_string(),
|
||||
contact.node_id.as_bytes().as_slice(),
|
||||
contact.ticket,
|
||||
clamp_i64(contact.last_seen_ms),
|
||||
],
|
||||
)?;
|
||||
conn.execute(
|
||||
"DELETE FROM podcast_dht_peers WHERE peer_id IN (
|
||||
SELECT peer_id FROM podcast_dht_peers
|
||||
ORDER BY last_seen_ms DESC, peer_id
|
||||
LIMIT -1 OFFSET ?1)",
|
||||
[MAX_PERSISTED_PODCAST_DHT_PEERS as i64],
|
||||
)?;
|
||||
Ok(())
|
||||
})
|
||||
.await
|
||||
}
|
||||
|
||||
pub async fn remove_peer(&self, peer: EndpointId) -> Result<()> {
|
||||
self.call(move |conn, _, _| {
|
||||
conn.execute(
|
||||
"DELETE FROM podcast_dht_peers WHERE peer_id = ?1",
|
||||
[peer.to_string()],
|
||||
)?;
|
||||
Ok(())
|
||||
})
|
||||
.await
|
||||
}
|
||||
|
||||
pub async fn load_peers(&self) -> Result<Vec<NodeContact>> {
|
||||
self.call(move |conn, _, _| {
|
||||
let mut statement =
|
||||
conn.prepare("SELECT peer_id, ticket, last_seen_ms FROM podcast_dht_peers")?;
|
||||
let rows = statement.query_map([], |row| {
|
||||
Ok((
|
||||
row.get::<_, String>(0)?,
|
||||
row.get::<_, String>(1)?,
|
||||
row.get::<_, i64>(2)?,
|
||||
))
|
||||
})?;
|
||||
let mut peers = Vec::new();
|
||||
for row in rows {
|
||||
let (peer, ticket, last_seen_ms) = row?;
|
||||
let Ok(peer_id) = EndpointId::from_str(&peer) else {
|
||||
continue;
|
||||
};
|
||||
peers.push(NodeContact {
|
||||
node_id: NodeId::from_endpoint(&peer_id),
|
||||
peer_id,
|
||||
ticket,
|
||||
last_seen_ms: last_seen_ms.max(0) as u64,
|
||||
});
|
||||
}
|
||||
Ok(peers)
|
||||
})
|
||||
.await
|
||||
}
|
||||
|
||||
pub async fn local_publication(
|
||||
&self,
|
||||
podcast_id: PodcastId,
|
||||
) -> Result<Option<LocalPublication>> {
|
||||
self.call(move |conn, _, _| {
|
||||
conn.query_row(
|
||||
"SELECT podcast_id, revision, spec, record FROM podcast_dht_local
|
||||
WHERE podcast_id = ?1",
|
||||
[podcast_id.as_str()],
|
||||
local_from_row,
|
||||
)
|
||||
.optional()
|
||||
.map(Option::flatten)
|
||||
})
|
||||
.await
|
||||
}
|
||||
|
||||
pub async fn local_publications(&self) -> Result<Vec<LocalPublication>> {
|
||||
self.call(move |conn, _, _| {
|
||||
let mut statement = conn.prepare(
|
||||
"SELECT podcast_id, revision, spec, record FROM podcast_dht_local
|
||||
ORDER BY podcast_id",
|
||||
)?;
|
||||
let rows = statement.query_map([], local_from_row)?;
|
||||
let mut out = Vec::new();
|
||||
for row in rows {
|
||||
if let Some(publication) = row? {
|
||||
out.push(publication);
|
||||
}
|
||||
}
|
||||
Ok(out)
|
||||
})
|
||||
.await
|
||||
}
|
||||
|
||||
pub async fn put_local_publication(
|
||||
&self,
|
||||
publication: LocalPublication,
|
||||
now_ms: u64,
|
||||
) -> Result<()> {
|
||||
self.call(move |conn, _, _| {
|
||||
let spec = publication
|
||||
.spec
|
||||
.as_ref()
|
||||
.map(postcard::to_stdvec)
|
||||
.transpose()
|
||||
.map_err(to_sql_error)?;
|
||||
let record = postcard::to_stdvec(&publication.record).map_err(to_sql_error)?;
|
||||
conn.execute(
|
||||
"INSERT OR REPLACE INTO podcast_dht_local
|
||||
(podcast_id, revision, spec, record, updated_at_ms)
|
||||
VALUES (?1, ?2, ?3, ?4, ?5)",
|
||||
params![
|
||||
publication.podcast_id.as_str(),
|
||||
publication.revision as i64,
|
||||
spec,
|
||||
record,
|
||||
clamp_i64(now_ms),
|
||||
],
|
||||
)?;
|
||||
Ok(())
|
||||
})
|
||||
.await
|
||||
}
|
||||
}
|
||||
|
||||
fn store_one(
|
||||
conn: &Connection,
|
||||
network_id: &NetworkId,
|
||||
limits: &StoreLimits,
|
||||
keys: &[PodcastDhtKey],
|
||||
record: &SignedPodcastRecord,
|
||||
now_ms: u64,
|
||||
) -> rusqlite::Result<usize> {
|
||||
let podcast_id = record.podcast_id().as_str().to_string();
|
||||
let publisher = record.publisher().to_string();
|
||||
let now = clamp_i64(now_ms);
|
||||
|
||||
// The cached record of this (podcast, publisher), re-verified.
|
||||
let cached: Option<(Vec<u8>, i64)> = conn
|
||||
.query_row(
|
||||
"SELECT payload, payload_bytes FROM podcast_dht_records
|
||||
WHERE podcast_id = ?1 AND publisher = ?2",
|
||||
params![podcast_id, publisher],
|
||||
|row| Ok((row.get(0)?, row.get(1)?)),
|
||||
)
|
||||
.optional()?;
|
||||
let (existing, existing_bytes) = match cached {
|
||||
Some((payload, bytes)) => match decode_valid(&payload, network_id, now_ms) {
|
||||
Some(existing) => (Some(existing), bytes.max(0) as usize),
|
||||
None => {
|
||||
delete_owner(conn, &podcast_id, &publisher)?;
|
||||
(None, 0)
|
||||
}
|
||||
},
|
||||
None => {
|
||||
// Stray placements without a record.
|
||||
delete_owner(conn, &podcast_id, &publisher)?;
|
||||
(None, 0)
|
||||
}
|
||||
};
|
||||
|
||||
// Higher revision, tombstone at an equal revision, later republish...
|
||||
let replace = match &existing {
|
||||
Some(existing) => match compare_records(existing, record) {
|
||||
std::cmp::Ordering::Greater => return Ok(0),
|
||||
std::cmp::Ordering::Equal => false,
|
||||
std::cmp::Ordering::Less => true,
|
||||
},
|
||||
None => true,
|
||||
};
|
||||
let payload = postcard::to_stdvec(record).map_err(to_sql_error)?;
|
||||
if replace
|
||||
&& !make_room_for_record(
|
||||
conn,
|
||||
limits,
|
||||
(&podcast_id, &publisher),
|
||||
existing.is_none(),
|
||||
existing_bytes,
|
||||
payload.len(),
|
||||
now,
|
||||
)?
|
||||
{
|
||||
return Ok(0);
|
||||
}
|
||||
|
||||
let mut indexed: HashSet<[u8; 32]> = HashSet::new();
|
||||
{
|
||||
let mut statement = conn.prepare(
|
||||
"SELECT dht_key FROM podcast_dht_keys WHERE podcast_id = ?1 AND publisher = ?2",
|
||||
)?;
|
||||
let rows = statement.query_map(params![podcast_id, publisher], |row| {
|
||||
row.get::<_, Vec<u8>>(0)
|
||||
})?;
|
||||
for row in rows {
|
||||
if let Ok(key) = <[u8; 32]>::try_from(row?.as_slice()) {
|
||||
indexed.insert(key);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let mut written = 0;
|
||||
if replace {
|
||||
// Placements the new version no longer belongs under are dropped (a
|
||||
// renamed card, or a tombstone that does not list the key); the others
|
||||
// now serve the new version.
|
||||
let belongs: HashSet<[u8; 32]> = record
|
||||
.dht_keys()
|
||||
.iter()
|
||||
.map(|key| *key.as_bytes())
|
||||
.collect();
|
||||
let stale: Vec<[u8; 32]> = indexed
|
||||
.iter()
|
||||
.filter(|key| !belongs.contains(*key))
|
||||
.copied()
|
||||
.collect();
|
||||
for key in stale {
|
||||
conn.execute(
|
||||
"DELETE FROM podcast_dht_keys
|
||||
WHERE dht_key = ?1 AND podcast_id = ?2 AND publisher = ?3",
|
||||
params![key.as_slice(), podcast_id, publisher],
|
||||
)?;
|
||||
indexed.remove(&key);
|
||||
}
|
||||
written += conn.execute(
|
||||
"UPDATE podcast_dht_keys SET tombstone = ?3, expires_at_ms = ?4
|
||||
WHERE podcast_id = ?1 AND publisher = ?2",
|
||||
params![
|
||||
podcast_id,
|
||||
publisher,
|
||||
record.is_tombstone(),
|
||||
clamp_i64(record.expires_at_ms()),
|
||||
],
|
||||
)?;
|
||||
// A card converted to a tombstone in place counts against the
|
||||
// tombstone budget of each key it stays under, like a new one.
|
||||
if record.is_tombstone() {
|
||||
for key in &indexed {
|
||||
let key = PodcastDhtKey::from_bytes(*key);
|
||||
make_room_under_key(conn, limits, &key, true, (&podcast_id, &publisher), now, 0)?;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// New placements. A tombstone only takes a new place under the podcast-id
|
||||
// key: under a text key it may only replace a record of its owner, so
|
||||
// nobody can index tombstones under someone else's search terms.
|
||||
let id_key = PodcastDhtKey::podcast(network_id, record.podcast_id());
|
||||
for key in keys {
|
||||
if indexed.contains(key.as_bytes()) {
|
||||
continue;
|
||||
}
|
||||
if record.is_tombstone() && *key != id_key {
|
||||
continue;
|
||||
}
|
||||
if !make_room_under_key(
|
||||
conn,
|
||||
limits,
|
||||
key,
|
||||
record.is_tombstone(),
|
||||
(&podcast_id, &publisher),
|
||||
now,
|
||||
1,
|
||||
)? {
|
||||
continue;
|
||||
}
|
||||
conn.execute(
|
||||
"INSERT INTO podcast_dht_keys
|
||||
(dht_key, podcast_id, publisher, tombstone, expires_at_ms, first_stored_at_ms)
|
||||
VALUES (?1, ?2, ?3, ?4, ?5, ?6)",
|
||||
params![
|
||||
key.as_bytes().as_slice(),
|
||||
podcast_id,
|
||||
publisher,
|
||||
record.is_tombstone(),
|
||||
clamp_i64(record.expires_at_ms()),
|
||||
now,
|
||||
],
|
||||
)?;
|
||||
indexed.insert(*key.as_bytes());
|
||||
written += 1;
|
||||
}
|
||||
|
||||
if indexed.is_empty() {
|
||||
delete_owner(conn, &podcast_id, &publisher)?;
|
||||
return Ok(0);
|
||||
}
|
||||
if replace {
|
||||
conn.execute(
|
||||
"INSERT INTO podcast_dht_records
|
||||
(podcast_id, publisher, tombstone, expires_at_ms, payload_bytes, payload)
|
||||
VALUES (?1, ?2, ?3, ?4, ?5, ?6)
|
||||
ON CONFLICT(podcast_id, publisher) DO UPDATE SET
|
||||
tombstone = excluded.tombstone,
|
||||
expires_at_ms = excluded.expires_at_ms,
|
||||
payload_bytes = excluded.payload_bytes,
|
||||
payload = excluded.payload",
|
||||
params![
|
||||
podcast_id,
|
||||
publisher,
|
||||
record.is_tombstone(),
|
||||
clamp_i64(record.expires_at_ms()),
|
||||
payload.len() as i64,
|
||||
payload,
|
||||
],
|
||||
)?;
|
||||
}
|
||||
Ok(written)
|
||||
}
|
||||
|
||||
/// Makes room in the cache for the new version of a record of `owner`.
|
||||
///
|
||||
/// * A publisher at [`StoreLimits::per_publisher`] gets no new record.
|
||||
/// * When the record count (for a new owner) or the byte budget would be
|
||||
/// exceeded, random records of other owners from the younger half of the
|
||||
/// cache (by first indexing time) are evicted, at most
|
||||
/// [`MAX_EVICTIONS_PER_RECORD`]. If that is not enough, nothing is evicted
|
||||
/// and the record is refused.
|
||||
fn make_room_for_record(
|
||||
conn: &Connection,
|
||||
limits: &StoreLimits,
|
||||
owner: (&str, &str),
|
||||
new_owner: bool,
|
||||
previous_bytes: usize,
|
||||
new_bytes: usize,
|
||||
now: i64,
|
||||
) -> rusqlite::Result<bool> {
|
||||
if new_bytes > limits.bytes {
|
||||
return Ok(false);
|
||||
}
|
||||
if new_owner {
|
||||
let own: i64 = conn.query_row(
|
||||
"SELECT COUNT(*) FROM podcast_dht_records WHERE publisher = ?1 AND expires_at_ms > ?2",
|
||||
params![owner.1, now],
|
||||
|row| row.get(0),
|
||||
)?;
|
||||
if own.max(0) as usize >= limits.per_publisher {
|
||||
return Ok(false);
|
||||
}
|
||||
}
|
||||
let (records, bytes): (i64, i64) = conn.query_row(
|
||||
"SELECT COUNT(*), COALESCE(SUM(payload_bytes), 0) FROM podcast_dht_records
|
||||
WHERE expires_at_ms > ?1",
|
||||
[now],
|
||||
|row| Ok((row.get(0)?, row.get(1)?)),
|
||||
)?;
|
||||
let mut records = records.max(0) as usize;
|
||||
let mut bytes = (bytes.max(0) as usize).saturating_sub(previous_bytes);
|
||||
let fits = |records: usize, bytes: usize| {
|
||||
bytes.saturating_add(new_bytes) <= limits.bytes && (!new_owner || records < limits.records)
|
||||
};
|
||||
if fits(records, bytes) {
|
||||
return Ok(true);
|
||||
}
|
||||
|
||||
// Candidates: the younger half of the other owners' records, shuffled.
|
||||
let others = records.saturating_sub(usize::from(!new_owner));
|
||||
let mut statement = conn.prepare(
|
||||
"SELECT podcast_id, publisher, payload_bytes FROM (
|
||||
SELECT r.podcast_id, r.publisher, r.payload_bytes,
|
||||
(SELECT MIN(k.first_stored_at_ms) FROM podcast_dht_keys k
|
||||
WHERE k.podcast_id = r.podcast_id AND k.publisher = r.publisher) AS since
|
||||
FROM podcast_dht_records r
|
||||
WHERE r.expires_at_ms > ?1 AND NOT (r.podcast_id = ?2 AND r.publisher = ?3)
|
||||
ORDER BY since DESC, random()
|
||||
LIMIT ?4)
|
||||
ORDER BY random() LIMIT ?5",
|
||||
)?;
|
||||
let candidates = statement
|
||||
.query_map(
|
||||
params![
|
||||
now,
|
||||
owner.0,
|
||||
owner.1,
|
||||
others.div_ceil(2) as i64,
|
||||
MAX_EVICTIONS_PER_RECORD as i64
|
||||
],
|
||||
|row| {
|
||||
Ok((
|
||||
row.get::<_, String>(0)?,
|
||||
row.get::<_, String>(1)?,
|
||||
row.get::<_, i64>(2)?,
|
||||
))
|
||||
},
|
||||
)?
|
||||
.collect::<rusqlite::Result<Vec<_>>>()?;
|
||||
let mut evict = Vec::new();
|
||||
for (podcast_id, publisher, payload_bytes) in candidates {
|
||||
if fits(records, bytes) {
|
||||
break;
|
||||
}
|
||||
records = records.saturating_sub(1);
|
||||
bytes = bytes.saturating_sub(payload_bytes.max(0) as usize);
|
||||
evict.push((podcast_id, publisher));
|
||||
}
|
||||
if !fits(records, bytes) {
|
||||
return Ok(false);
|
||||
}
|
||||
for (podcast_id, publisher) in evict {
|
||||
delete_owner(conn, &podcast_id, &publisher)?;
|
||||
}
|
||||
Ok(true)
|
||||
}
|
||||
|
||||
/// Makes room under `key` for `incoming` more placements (0 when an existing
|
||||
/// placement of `owner` was converted to a tombstone in place) and returns
|
||||
/// whether the key has room.
|
||||
///
|
||||
/// Over [`StoreLimits::per_key`], or over [`StoreLimits::tombstones_per_key`]
|
||||
/// for a tombstone, a random placement of another owner from the younger
|
||||
/// half of the key (by first indexing time; only tombstones for the
|
||||
/// tombstone budget) is evicted. The older half is never evicted for a
|
||||
/// newcomer.
|
||||
fn make_room_under_key(
|
||||
conn: &Connection,
|
||||
limits: &StoreLimits,
|
||||
key: &PodcastDhtKey,
|
||||
tombstone: bool,
|
||||
owner: (&str, &str),
|
||||
now: i64,
|
||||
incoming: usize,
|
||||
) -> rusqlite::Result<bool> {
|
||||
for _ in 0..=MAX_EVICTIONS_PER_RECORD {
|
||||
let (total, tombstones): (i64, i64) = conn.query_row(
|
||||
"SELECT COUNT(*), COALESCE(SUM(tombstone), 0) FROM podcast_dht_keys
|
||||
WHERE dht_key = ?1 AND expires_at_ms > ?2",
|
||||
params![key.as_bytes().as_slice(), now],
|
||||
|row| Ok((row.get(0)?, row.get(1)?)),
|
||||
)?;
|
||||
let over_tombstones =
|
||||
tombstone && tombstones.max(0) as usize + incoming > limits.tombstones_per_key;
|
||||
let over_total = total.max(0) as usize + incoming > limits.per_key;
|
||||
if !over_tombstones && !over_total {
|
||||
return Ok(true);
|
||||
}
|
||||
let Some((podcast_id, publisher)) =
|
||||
young_placement(conn, key, over_tombstones, owner, now)?
|
||||
else {
|
||||
return Ok(false);
|
||||
};
|
||||
conn.execute(
|
||||
"DELETE FROM podcast_dht_keys
|
||||
WHERE dht_key = ?1 AND podcast_id = ?2 AND publisher = ?3",
|
||||
params![key.as_bytes().as_slice(), podcast_id, publisher],
|
||||
)?;
|
||||
let placed: bool = conn.query_row(
|
||||
"SELECT EXISTS (SELECT 1 FROM podcast_dht_keys
|
||||
WHERE podcast_id = ?1 AND publisher = ?2)",
|
||||
params![podcast_id, publisher],
|
||||
|row| row.get(0),
|
||||
)?;
|
||||
if !placed {
|
||||
delete_owner(conn, &podcast_id, &publisher)?;
|
||||
}
|
||||
}
|
||||
Ok(false)
|
||||
}
|
||||
|
||||
/// A random unexpired placement of another owner than `owner` from the
|
||||
/// younger half of `key` (only tombstones with `tombstones_only`).
|
||||
fn young_placement(
|
||||
conn: &Connection,
|
||||
key: &PodcastDhtKey,
|
||||
tombstones_only: bool,
|
||||
owner: (&str, &str),
|
||||
now: i64,
|
||||
) -> rusqlite::Result<Option<(String, String)>> {
|
||||
let filter = "dht_key = ?1 AND expires_at_ms > ?2 AND (?3 = 0 OR tombstone = 1)
|
||||
AND NOT (podcast_id = ?4 AND publisher = ?5)";
|
||||
let count: i64 = conn.query_row(
|
||||
&format!("SELECT COUNT(*) FROM podcast_dht_keys WHERE {filter}"),
|
||||
params![
|
||||
key.as_bytes().as_slice(),
|
||||
now,
|
||||
tombstones_only,
|
||||
owner.0,
|
||||
owner.1
|
||||
],
|
||||
|row| row.get(0),
|
||||
)?;
|
||||
if count <= 0 {
|
||||
return Ok(None);
|
||||
}
|
||||
conn.query_row(
|
||||
&format!(
|
||||
"SELECT podcast_id, publisher FROM (
|
||||
SELECT podcast_id, publisher FROM podcast_dht_keys WHERE {filter}
|
||||
ORDER BY first_stored_at_ms DESC, random() LIMIT ?6)
|
||||
ORDER BY random() LIMIT 1"
|
||||
),
|
||||
params![
|
||||
key.as_bytes().as_slice(),
|
||||
now,
|
||||
tombstones_only,
|
||||
owner.0,
|
||||
owner.1,
|
||||
(count as usize).div_ceil(2) as i64
|
||||
],
|
||||
|row| Ok((row.get(0)?, row.get(1)?)),
|
||||
)
|
||||
.optional()
|
||||
}
|
||||
|
||||
fn delete_owner(conn: &Connection, podcast_id: &str, publisher: &str) -> rusqlite::Result<()> {
|
||||
conn.execute(
|
||||
"DELETE FROM podcast_dht_keys WHERE podcast_id = ?1 AND publisher = ?2",
|
||||
params![podcast_id, publisher],
|
||||
)?;
|
||||
conn.execute(
|
||||
"DELETE FROM podcast_dht_records WHERE podcast_id = ?1 AND publisher = ?2",
|
||||
params![podcast_id, publisher],
|
||||
)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn decode_valid(
|
||||
payload: &[u8],
|
||||
network_id: &NetworkId,
|
||||
now_ms: u64,
|
||||
) -> Option<SignedPodcastRecord> {
|
||||
let record: SignedPodcastRecord = postcard::from_bytes(payload).ok()?;
|
||||
record.verify_at(network_id, now_ms).ok()?;
|
||||
Some(record)
|
||||
}
|
||||
|
||||
fn local_from_row(row: &rusqlite::Row<'_>) -> rusqlite::Result<Option<LocalPublication>> {
|
||||
let podcast_id: String = row.get(0)?;
|
||||
let revision: i64 = row.get(1)?;
|
||||
let spec: Option<Vec<u8>> = row.get(2)?;
|
||||
let record: Vec<u8> = row.get(3)?;
|
||||
let Ok(podcast_id) = PodcastId::parse(&podcast_id) else {
|
||||
return Ok(None);
|
||||
};
|
||||
let spec = match spec {
|
||||
Some(bytes) => match postcard::from_bytes::<PodcastCardSpec>(&bytes) {
|
||||
Ok(spec) => Some(spec),
|
||||
Err(_) => return Ok(None),
|
||||
},
|
||||
None => None,
|
||||
};
|
||||
let Ok(record) = postcard::from_bytes::<SignedPodcastRecord>(&record) else {
|
||||
return Ok(None);
|
||||
};
|
||||
Ok(Some(LocalPublication {
|
||||
podcast_id,
|
||||
revision: revision as u64,
|
||||
spec,
|
||||
record,
|
||||
}))
|
||||
}
|
||||
|
||||
fn clamp_i64(value: u64) -> i64 {
|
||||
value.min(i64::MAX as u64) as i64
|
||||
}
|
||||
|
||||
fn to_sql_error(error: impl std::error::Error + Send + Sync + 'static) -> rusqlite::Error {
|
||||
rusqlite::Error::ToSqlConversionFailure(Box::new(error))
|
||||
}
|
||||
|
||||
fn database_error(error: rusqlite::Error) -> MusicDhtError {
|
||||
MusicDhtError::Database(error.to_string())
|
||||
}
|
||||
|
||||
fn lock<T>(mutex: &Mutex<T>) -> MutexGuard<'_, T> {
|
||||
mutex.lock().unwrap_or_else(PoisonError::into_inner)
|
||||
}
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,763 @@
|
||||
//! Integration test: public podcast discovery on three real loopback iroh
|
||||
//! nodes.
|
||||
//!
|
||||
//! * A publishes a podcast its user shared. B is another device of the same
|
||||
//! user: it applies the same merged registers (as device sync would deliver
|
||||
//! them) and publishes the same podcast under its own key. C is an
|
||||
//! unrelated peer outside the user's trusted-device group, with no
|
||||
//! device-sync pairing.
|
||||
//! * C finds the podcast by text and by id, merged over publishers {A, B};
|
||||
//! once B unshares and goes offline, C still finds A's card, so B is
|
||||
//! irrelevant to discovery. A podcast the user never shared (held by both
|
||||
//! A and B) is never discoverable; after A unpublishes, the tombstone hides
|
||||
//! the card from C.
|
||||
//! * Forged, foreign, tampered, replayed, malformed, oversized and
|
||||
//! wrong-version requests are rejected by a real peer.
|
||||
//! * D, a music-DHT peer without the overlay, is dialed by bootstrap but
|
||||
//! never answers on the overlay ALPN, so it never becomes an overlay contact
|
||||
//! and is never asked for records or to store them.
|
||||
//! * `sync_shares` stays fail-safe for revocation and holds back URLs that
|
||||
//! look private (single node).
|
||||
//! * Contacts a peer returns are never trusted: fabricated, unreachable
|
||||
//! contacts do not enter the routing table (two nodes).
|
||||
//! * After a restart, own cards persisted by the previous run are neither
|
||||
//! republished nor served until the first `sync_shares`, so a share
|
||||
//! revoked on another device meanwhile is never re-sent (two nodes).
|
||||
|
||||
use std::path::Path;
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
|
||||
use music_dht::capabilities::{CapabilityManifest, PODCAST_DHT_ID};
|
||||
use music_dht::podcast::{FeedUrl, PodcastId, PodcastMeta, ShareRecord, SubscriptionRecord};
|
||||
use music_dht::podcast_dht::{
|
||||
MAX_PODCAST_DHT_REQUEST_BYTES, PODCAST_DHT_ALPN, PODCAST_DHT_PROTOCOL_VERSION, PodcastCard,
|
||||
PodcastCardSpec, PodcastDht, PodcastDhtOperation, PodcastDhtRequest, PodcastDhtResponse,
|
||||
PodcastRecordBody, PodcastSearchHit, PodcastShareState, PodcastStoreEntry, SignedPodcastRecord,
|
||||
};
|
||||
use music_dht::{MusicDhtConfig, MusicDhtError, MusicDhtService, NetworkId, NodeContact, NodeId};
|
||||
use tokio::io::AsyncReadExt;
|
||||
|
||||
const TEST_TIMEOUT: Duration = Duration::from_secs(240);
|
||||
|
||||
struct Node {
|
||||
service: Arc<MusicDhtService>,
|
||||
dht: Arc<PodcastDht>,
|
||||
tasks: Vec<tokio::task::JoinHandle<()>>,
|
||||
}
|
||||
|
||||
impl Node {
|
||||
async fn start(directory: &Path, network: NetworkId) -> Self {
|
||||
let (service, events) = start_service(directory, network, true).await;
|
||||
let acceptor = service.stream_acceptor(PODCAST_DHT_ALPN).unwrap();
|
||||
let dht = PodcastDht::open(Arc::clone(&service), directory.join("podcast-dht.sqlite3"))
|
||||
.await
|
||||
.unwrap();
|
||||
let tasks = vec![events, tokio::spawn(Arc::clone(&dht).serve(acceptor))];
|
||||
Self {
|
||||
service,
|
||||
dht,
|
||||
tasks,
|
||||
}
|
||||
}
|
||||
|
||||
async fn stop(self) {
|
||||
for task in &self.tasks {
|
||||
task.abort();
|
||||
}
|
||||
self.service.shutdown().await.unwrap();
|
||||
}
|
||||
|
||||
async fn contact(&self) -> NodeContact {
|
||||
let peer_id = self.service.endpoint_id();
|
||||
NodeContact {
|
||||
node_id: NodeId::from_endpoint(&peer_id),
|
||||
peer_id,
|
||||
ticket: self.service.ticket().await.unwrap().to_string(),
|
||||
last_seen_ms: 1,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async fn start_service(
|
||||
directory: &Path,
|
||||
network: NetworkId,
|
||||
overlay: bool,
|
||||
) -> (Arc<MusicDhtService>, tokio::task::JoinHandle<()>) {
|
||||
std::fs::create_dir_all(directory).unwrap();
|
||||
let mut config = MusicDhtConfig::builder()
|
||||
.data_dir(directory)
|
||||
.network_id(network)
|
||||
.request_timeout(Duration::from_secs(2))
|
||||
.lookup_timeout(Duration::from_secs(5))
|
||||
.transport_timeout(Duration::from_secs(5))
|
||||
.dial_timeout(Duration::from_secs(2));
|
||||
if overlay {
|
||||
config = config.schema_independent_stream_protocol(PODCAST_DHT_ALPN);
|
||||
}
|
||||
let (service, mut events) = MusicDhtService::start(config.build().unwrap())
|
||||
.await
|
||||
.unwrap();
|
||||
let events = tokio::spawn(async move { while events.recv().await.is_some() {} });
|
||||
(Arc::new(service), events)
|
||||
}
|
||||
|
||||
fn feed(path: &str) -> FeedUrl {
|
||||
FeedUrl::parse(&format!("https://feeds.example.test/{path}")).unwrap()
|
||||
}
|
||||
|
||||
/// A subscription of A's user (generation 1).
|
||||
fn subscription(path: &str, title: &str) -> SubscriptionRecord {
|
||||
let feed_url = feed(path);
|
||||
SubscriptionRecord {
|
||||
podcast_id: PodcastId::from_feed_url(&feed_url),
|
||||
subscribed: true,
|
||||
podcast: PodcastMeta {
|
||||
feed_url,
|
||||
title: title.to_string(),
|
||||
author: Some("Ferris Crab".to_string()),
|
||||
description: Some("Weekly news.".to_string()),
|
||||
link: None,
|
||||
image_url: None,
|
||||
},
|
||||
feed_aliases: Vec::new(),
|
||||
generation: 1,
|
||||
hlc_ms: 1_000,
|
||||
op_id: "device-a-op-1".to_string(),
|
||||
}
|
||||
}
|
||||
|
||||
fn share(subscription: &SubscriptionRecord, shared: bool, generation: u64) -> ShareRecord {
|
||||
ShareRecord {
|
||||
podcast_id: subscription.podcast_id.clone(),
|
||||
shared,
|
||||
generation,
|
||||
hlc_ms: 2_000 + generation as i64,
|
||||
op_id: format!("device-a-share-{generation}"),
|
||||
}
|
||||
}
|
||||
|
||||
async fn wait_for<T>(what: &str, mut check: impl AsyncFnMut() -> Option<T>) -> T {
|
||||
let started = std::time::Instant::now();
|
||||
loop {
|
||||
if let Some(value) = check().await {
|
||||
return value;
|
||||
}
|
||||
assert!(
|
||||
started.elapsed() < Duration::from_secs(60),
|
||||
"timed out waiting for {what}"
|
||||
);
|
||||
tokio::time::sleep(Duration::from_millis(300)).await;
|
||||
}
|
||||
}
|
||||
|
||||
/// Sends raw bytes as one overlay request and decodes the response.
|
||||
async fn raw_request(from: &Node, to: &Node, payload: &[u8]) -> PodcastDhtResponse {
|
||||
let ticket = to.service.ticket().await.unwrap();
|
||||
let mut stream = from
|
||||
.service
|
||||
.open_stream_to(&ticket, PODCAST_DHT_ALPN)
|
||||
.await
|
||||
.unwrap();
|
||||
stream.send.write_all(payload).await.unwrap();
|
||||
stream.send.finish().unwrap();
|
||||
let mut bytes = Vec::new();
|
||||
AsyncReadExt::read_to_end(&mut stream.recv, &mut bytes)
|
||||
.await
|
||||
.unwrap();
|
||||
postcard::from_bytes(&bytes).unwrap()
|
||||
}
|
||||
|
||||
async fn store_request(from: &Node, to: &Node, record: SignedPodcastRecord) -> PodcastDhtResponse {
|
||||
let request = PodcastDhtRequest {
|
||||
version: PODCAST_DHT_PROTOCOL_VERSION,
|
||||
requester: from.contact().await,
|
||||
operation: PodcastDhtOperation::StoreBatch {
|
||||
records: vec![PodcastStoreEntry {
|
||||
keys: record.dht_keys(),
|
||||
record,
|
||||
}],
|
||||
},
|
||||
};
|
||||
raw_request(from, to, &postcard::to_stdvec(&request).unwrap()).await
|
||||
}
|
||||
|
||||
async fn search(node: &Node, query: &str) -> Vec<PodcastSearchHit> {
|
||||
node.dht.search(query, 10).await.unwrap()
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||
async fn shared_podcast_is_discoverable_by_strangers_and_retractable() {
|
||||
tokio::time::timeout(TEST_TIMEOUT, scenario())
|
||||
.await
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
async fn scenario() {
|
||||
let temp = tempfile::tempdir().unwrap();
|
||||
let network = NetworkId::from_name("podcast-dht-integration");
|
||||
let a = Node::start(&temp.path().join("a"), network).await;
|
||||
let b = Node::start(&temp.path().join("b"), network).await;
|
||||
let c = Node::start(&temp.path().join("c"), network).await;
|
||||
// D runs the music DHT only.
|
||||
let (d, d_events) = start_service(&temp.path().join("d"), network, false).await;
|
||||
|
||||
// B and C reach A through the ordinary music-DHT transport; C has no
|
||||
// device-sync pairing with A or B.
|
||||
b.service
|
||||
.connect(a.service.ticket().await.unwrap())
|
||||
.await
|
||||
.unwrap();
|
||||
c.service
|
||||
.connect(a.service.ticket().await.unwrap())
|
||||
.await
|
||||
.unwrap();
|
||||
d.connect(a.service.ticket().await.unwrap()).await.unwrap();
|
||||
wait_for("music DHT contacts", async || {
|
||||
(a.service.known_peers().len() >= 3
|
||||
&& !b.service.known_peers().is_empty()
|
||||
&& !c.service.known_peers().is_empty())
|
||||
.then_some(())
|
||||
})
|
||||
.await;
|
||||
for node in [&a, &b, &c] {
|
||||
node.dht.bootstrap().await.unwrap();
|
||||
}
|
||||
wait_for("podcast overlay contacts", async || {
|
||||
(a.dht.known_peers() >= 2 && c.dht.known_peers() >= 1).then_some(())
|
||||
})
|
||||
.await;
|
||||
// D is dialed by bootstrap but never answers on the overlay ALPN, so it
|
||||
// never becomes an overlay contact: it is never asked for records or to
|
||||
// store them.
|
||||
a.dht.bootstrap().await.unwrap();
|
||||
assert_eq!(a.dht.known_peers(), 2);
|
||||
|
||||
// A capability manifest without podcast_dht evicts a peer; one with it
|
||||
// lets the peer be learned again.
|
||||
let manifest = CapabilityManifest::frid("furumi", "test").with_podcast_dht();
|
||||
a.dht
|
||||
.note_peer_capabilities(
|
||||
c.service.endpoint_id(),
|
||||
&manifest.clone().without_protocol(PODCAST_DHT_ID),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(a.dht.known_peers(), 1);
|
||||
a.dht.bootstrap().await.unwrap();
|
||||
assert_eq!(
|
||||
a.dht.known_peers(),
|
||||
1,
|
||||
"the evicted peer is not bootstrapped"
|
||||
);
|
||||
a.dht
|
||||
.note_peer_capabilities(c.service.endpoint_id(), &manifest)
|
||||
.await
|
||||
.unwrap();
|
||||
a.dht.bootstrap().await.unwrap();
|
||||
assert_eq!(a.dht.known_peers(), 2);
|
||||
|
||||
// A request with another overlay version is refused.
|
||||
let request = PodcastDhtRequest {
|
||||
version: PODCAST_DHT_PROTOCOL_VERSION + 1,
|
||||
requester: c.contact().await,
|
||||
operation: PodcastDhtOperation::FindNode { target: [0; 32] },
|
||||
};
|
||||
let response = raw_request(&c, &a, &postcard::to_stdvec(&request).unwrap()).await;
|
||||
assert!(!response.ok);
|
||||
|
||||
// A's user shares "Rust Radio Weekly" but never shares "Secret Garden".
|
||||
let shared = subscription("rust-radio.xml", "Rust Radio Weekly");
|
||||
let private = subscription("secret-garden.xml", "Secret Garden Diaries");
|
||||
let states = vec![
|
||||
PodcastShareState {
|
||||
subscription: shared.clone(),
|
||||
share: Some(share(&shared, true, 2)),
|
||||
card: None,
|
||||
privacy_confirmed: false,
|
||||
},
|
||||
PodcastShareState {
|
||||
subscription: private.clone(),
|
||||
share: None,
|
||||
card: None,
|
||||
privacy_confirmed: false,
|
||||
},
|
||||
];
|
||||
let stats = a.dht.sync_shares(states.clone()).await.unwrap();
|
||||
assert_eq!(
|
||||
(stats.published, stats.unchanged, stats.unpublished),
|
||||
(1, 0, 0)
|
||||
);
|
||||
assert!(stats.push.local_replica);
|
||||
assert!(stats.push.remote_nodes >= 1);
|
||||
let error = a
|
||||
.dht
|
||||
.publish(
|
||||
&private,
|
||||
&share(&private, false, 2),
|
||||
PodcastCardSpec::from_subscription(&private).unwrap(),
|
||||
)
|
||||
.await
|
||||
.unwrap_err();
|
||||
assert!(matches!(error, MusicDhtError::PodcastPublication(_)));
|
||||
// Idempotent: nothing new is published.
|
||||
let again = a.dht.sync_shares(states).await.unwrap();
|
||||
assert_eq!(
|
||||
(again.published, again.unchanged, again.unpublished),
|
||||
(0, 1, 0)
|
||||
);
|
||||
let published = a.dht.published().await.unwrap();
|
||||
assert_eq!(published.len(), 1);
|
||||
let first_card = published[0].record.clone();
|
||||
|
||||
// B, the same user's other device, holds the same merged registers and
|
||||
// publishes the same podcast under its own key; the private podcast
|
||||
// stays private there too.
|
||||
let b_states = vec![
|
||||
PodcastShareState {
|
||||
subscription: shared.clone(),
|
||||
share: Some(share(&shared, true, 2)),
|
||||
card: None,
|
||||
privacy_confirmed: false,
|
||||
},
|
||||
PodcastShareState {
|
||||
subscription: private.clone(),
|
||||
share: None,
|
||||
card: None,
|
||||
privacy_confirmed: false,
|
||||
},
|
||||
];
|
||||
let stats = b.dht.sync_shares(b_states).await.unwrap();
|
||||
assert_eq!((stats.published, stats.unpublished), (1, 0));
|
||||
let mut both = vec![a.service.endpoint_id(), b.service.endpoint_id()];
|
||||
both.sort_by_key(|key| *key.as_bytes());
|
||||
let hits = wait_for("C sees both publishers", async || {
|
||||
let hits = search(&c, "rust radio").await;
|
||||
(hits.len() == 1 && hits[0].publishers == both).then_some(hits)
|
||||
})
|
||||
.await;
|
||||
assert_eq!(hits[0].feed_url, shared.podcast.feed_url);
|
||||
assert!(hits[0].claimed_feed_urls.is_empty());
|
||||
|
||||
// B unshares (for example a stale register on B) and goes offline: B is
|
||||
// not needed for discovery, A's card is still found.
|
||||
let stats = b
|
||||
.dht
|
||||
.sync_shares(vec![PodcastShareState {
|
||||
subscription: shared.clone(),
|
||||
share: Some(share(&shared, false, 3)),
|
||||
card: None,
|
||||
privacy_confirmed: false,
|
||||
}])
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(stats.unpublished, 1);
|
||||
let a_only = vec![a.service.endpoint_id()];
|
||||
wait_for("B's tombstone reaches C's lookup", async || {
|
||||
let hits = search(&c, "rust radio").await;
|
||||
(hits.len() == 1 && hits[0].publishers == a_only).then_some(())
|
||||
})
|
||||
.await;
|
||||
b.stop().await;
|
||||
|
||||
let hits = wait_for("C finds the card by text", async || {
|
||||
let hits = search(&c, "rust radio").await;
|
||||
(!hits.is_empty()).then_some(hits)
|
||||
})
|
||||
.await;
|
||||
assert_eq!(hits.len(), 1);
|
||||
assert_eq!(hits[0].podcast_id, shared.podcast_id);
|
||||
assert_eq!(hits[0].publishers, vec![a.service.endpoint_id()]);
|
||||
assert_eq!(hits[0].feed_url, shared.podcast.feed_url);
|
||||
assert_eq!(hits[0].card.feed_url, shared.podcast.feed_url);
|
||||
assert_eq!(hits[0].card.title, "Rust Radio Weekly");
|
||||
assert_eq!(search(&c, "ferris weekly").await.len(), 1, "author tokens");
|
||||
assert!(
|
||||
search(&c, "rust jazz").await.is_empty(),
|
||||
"every token must match"
|
||||
);
|
||||
let by_id = c.dht.lookup_podcast(&shared.podcast_id).await.unwrap();
|
||||
assert_eq!(by_id.unwrap().card.podcast_id, shared.podcast_id);
|
||||
|
||||
// The never-shared podcast leaves no trace anywhere.
|
||||
assert!(search(&c, "secret garden").await.is_empty());
|
||||
assert!(search(&a, "secret garden").await.is_empty());
|
||||
assert!(
|
||||
c.dht
|
||||
.lookup_podcast(&private.podcast_id)
|
||||
.await
|
||||
.unwrap()
|
||||
.is_none()
|
||||
);
|
||||
assert!(
|
||||
a.dht
|
||||
.lookup_podcast(&private.podcast_id)
|
||||
.await
|
||||
.unwrap()
|
||||
.is_none()
|
||||
);
|
||||
|
||||
// Forged: C signs a card that claims A as publisher.
|
||||
let forged_spec =
|
||||
PodcastCardSpec::from_subscription(&subscription("forged.xml", "Forged Show")).unwrap();
|
||||
let forged_card = PodcastCard::from_spec(
|
||||
forged_spec.clone(),
|
||||
network,
|
||||
a.service.endpoint_id(),
|
||||
u64::MAX - 1,
|
||||
first_card.issued_at_ms(),
|
||||
);
|
||||
let forged = SignedPodcastRecord {
|
||||
signature: c.service.sign_identity(&forged_card.signing_bytes()),
|
||||
body: PodcastRecordBody::Card(Box::new(forged_card)),
|
||||
};
|
||||
let response = store_request(&c, &a, forged).await;
|
||||
assert!(!response.ok && response.stored == 0, "{response:?}");
|
||||
|
||||
// Foreign: A's genuine card stored by C. Only publishers store records.
|
||||
let response = store_request(&c, &a, first_card.clone()).await;
|
||||
assert!(!response.ok && response.stored == 0, "{response:?}");
|
||||
|
||||
// Tampered: A's genuine card with a different description.
|
||||
let mut tampered = first_card.clone();
|
||||
if let PodcastRecordBody::Card(card) = &mut tampered.body {
|
||||
card.description = Some("Now with malware".into());
|
||||
}
|
||||
let response = store_request(&c, &a, tampered).await;
|
||||
assert!(!response.ok, "{response:?}");
|
||||
|
||||
// Malformed and oversized requests.
|
||||
let response = raw_request(&c, &a, b"definitely not postcard").await;
|
||||
assert!(!response.ok);
|
||||
let response = raw_request(&c, &a, &vec![0u8; MAX_PODCAST_DHT_REQUEST_BYTES + 1]).await;
|
||||
assert!(!response.ok);
|
||||
assert!(response.error.unwrap().contains("size"));
|
||||
|
||||
// A valid card signed by C is accepted: the overlay is open to anyone
|
||||
// who signs their own cards.
|
||||
let own_card = PodcastCard::from_spec(
|
||||
forged_spec,
|
||||
network,
|
||||
c.service.endpoint_id(),
|
||||
1,
|
||||
first_card.issued_at_ms(),
|
||||
);
|
||||
let own =
|
||||
SignedPodcastRecord::sign_card(own_card, |bytes| c.service.sign_identity(bytes)).unwrap();
|
||||
let response = store_request(&c, &a, own).await;
|
||||
assert!(response.ok && response.stored > 0, "{response:?}");
|
||||
let forged_hits = search(&a, "forged show").await;
|
||||
assert_eq!(forged_hits.len(), 1);
|
||||
assert_eq!(forged_hits[0].publishers, vec![c.service.endpoint_id()]);
|
||||
|
||||
// A's user stops sharing: a tombstone with the next revision replaces
|
||||
// the card and C stops seeing it.
|
||||
let unshare = vec![PodcastShareState {
|
||||
subscription: shared.clone(),
|
||||
share: Some(share(&shared, false, 3)),
|
||||
card: None,
|
||||
privacy_confirmed: false,
|
||||
}];
|
||||
let stats = a.dht.sync_shares(unshare).await.unwrap();
|
||||
assert_eq!(stats.unpublished, 1);
|
||||
let published = a.dht.published().await.unwrap();
|
||||
assert!(!published[0].active);
|
||||
assert!(published[0].record.is_tombstone());
|
||||
assert_eq!(published[0].revision, first_card.revision() + 1);
|
||||
wait_for("tombstone hides the card from C", async || {
|
||||
search(&c, "rust radio").await.is_empty().then_some(())
|
||||
})
|
||||
.await;
|
||||
assert!(
|
||||
c.dht
|
||||
.lookup_podcast(&shared.podcast_id)
|
||||
.await
|
||||
.unwrap()
|
||||
.is_none()
|
||||
);
|
||||
|
||||
// A replaying its own old card to a replica that stores the tombstone
|
||||
// does not bring it back; nobody else may replay it at all.
|
||||
let response = store_request(&a, &c, first_card.clone()).await;
|
||||
assert!(response.ok && response.stored == 0, "{response:?}");
|
||||
let response = store_request(&c, &a, first_card).await;
|
||||
assert!(!response.ok, "{response:?}");
|
||||
assert!(search(&c, "rust radio").await.is_empty());
|
||||
assert!(search(&a, "rust radio").await.is_empty());
|
||||
|
||||
a.stop().await;
|
||||
c.stop().await;
|
||||
d_events.abort();
|
||||
d.shutdown().await.unwrap();
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn sync_shares_is_fail_safe_and_holds_back_private_urls() {
|
||||
tokio::time::timeout(TEST_TIMEOUT, single_node_scenario())
|
||||
.await
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
async fn single_node_scenario() {
|
||||
let temp = tempfile::tempdir().unwrap();
|
||||
let network = NetworkId::from_name("podcast-dht-sync-shares");
|
||||
let a = Node::start(&temp.path().join("a"), network).await;
|
||||
let state =
|
||||
|subscription: &SubscriptionRecord, shared: bool, generation: u64| PodcastShareState {
|
||||
subscription: subscription.clone(),
|
||||
share: Some(share(subscription, shared, generation)),
|
||||
card: None,
|
||||
privacy_confirmed: false,
|
||||
};
|
||||
let first = subscription("first.xml", "First Show");
|
||||
let second = subscription("second.xml", "Second Show");
|
||||
let stats = a
|
||||
.dht
|
||||
.sync_shares(vec![state(&first, true, 2), state(&second, true, 2)])
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(stats.published, 2);
|
||||
assert!(stats.failed.is_empty() && stats.held_back.is_empty());
|
||||
|
||||
// In one call the user unshares the first podcast while the second
|
||||
// one's card cannot be built (the URL its id derives from is unknown).
|
||||
let mut broken = second.clone();
|
||||
broken.podcast.feed_url = feed("elsewhere.xml");
|
||||
let stats = a
|
||||
.dht
|
||||
.sync_shares(vec![state(&first, false, 3), state(&broken, true, 2)])
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(stats.failed.len(), 1);
|
||||
assert_eq!(stats.failed[0].podcast_id, second.podcast_id);
|
||||
assert_eq!(stats.unpublished, 2, "the unshared and the broken podcast");
|
||||
let published = a.dht.published().await.unwrap();
|
||||
assert!(published.iter().all(|publication| !publication.active));
|
||||
assert!(
|
||||
published
|
||||
.iter()
|
||||
.all(|publication| publication.record.is_tombstone())
|
||||
);
|
||||
// Republishing re-sends tombstones only; nothing comes back.
|
||||
a.dht.republish().await.unwrap();
|
||||
assert!(search(&a, "first show").await.is_empty());
|
||||
assert!(search(&a, "second show").await.is_empty());
|
||||
|
||||
// A feed URL that looks private is held back until the user confirms.
|
||||
let mut member = subscription("member.xml?token=abc", "Member Show");
|
||||
member.podcast_id = PodcastId::from_feed_url(&member.podcast.feed_url);
|
||||
let stats = a
|
||||
.dht
|
||||
.sync_shares(vec![state(&member, true, 2)])
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(stats.published, 0);
|
||||
assert_eq!(stats.held_back.len(), 1);
|
||||
assert_eq!(stats.held_back[0].podcast_id, member.podcast_id);
|
||||
assert!(!stats.held_back[0].warnings.is_empty());
|
||||
assert!(search(&a, "member show").await.is_empty());
|
||||
let mut confirmed = state(&member, true, 2);
|
||||
confirmed.privacy_confirmed = true;
|
||||
let stats = a.dht.sync_shares(vec![confirmed]).await.unwrap();
|
||||
assert_eq!(stats.published, 1);
|
||||
assert_eq!(search(&a, "member show").await.len(), 1);
|
||||
// Once published with the user's consent, the same URLs are kept.
|
||||
let stats = a
|
||||
.dht
|
||||
.sync_shares(vec![state(&member, true, 2)])
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!((stats.unchanged, stats.held_back.len()), (1, 0));
|
||||
|
||||
a.stop().await;
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn fabricated_contacts_never_join_the_overlay() {
|
||||
tokio::time::timeout(TEST_TIMEOUT, fabricated_contacts_scenario())
|
||||
.await
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
/// K contacts whose node ids are genuine hashes of their endpoint ids but
|
||||
/// whose tickets carry no address, as a malicious peer could return them.
|
||||
fn fabricated_contacts(network: NetworkId) -> Vec<NodeContact> {
|
||||
(0..music_dht::K)
|
||||
.map(|index| {
|
||||
let peer = federation_net::SecretKey::from_bytes(&[index as u8 + 101; 32]).public();
|
||||
let ticket = federation_net::PeerTicket {
|
||||
ticket_version: federation_net::TICKET_VERSION,
|
||||
protocol_version: 1,
|
||||
network_id: network,
|
||||
schema_id: federation_net::SchemaId::from_bytes([0; 32]),
|
||||
endpoint_addr: federation_net::EndpointAddr::new(peer),
|
||||
};
|
||||
NodeContact {
|
||||
node_id: NodeId::from_endpoint(&peer),
|
||||
peer_id: peer,
|
||||
ticket: ticket.to_string(),
|
||||
last_seen_ms: u64::MAX,
|
||||
}
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
async fn fabricated_contacts_scenario() {
|
||||
let temp = tempfile::tempdir().unwrap();
|
||||
let network = NetworkId::from_name("podcast-dht-fabricated-contacts");
|
||||
let a = Node::start(&temp.path().join("a"), network).await;
|
||||
// M speaks the overlay ALPN but answers every request with fabricated
|
||||
// contacts.
|
||||
let (m, m_events) = start_service(&temp.path().join("m"), network, true).await;
|
||||
let mut acceptor = m.stream_acceptor(PODCAST_DHT_ALPN).unwrap();
|
||||
let fabricated = fabricated_contacts(network);
|
||||
let response = PodcastDhtResponse {
|
||||
version: PODCAST_DHT_PROTOCOL_VERSION,
|
||||
ok: true,
|
||||
error: None,
|
||||
nodes: fabricated.clone(),
|
||||
records: Vec::new(),
|
||||
stored: 0,
|
||||
};
|
||||
let payload = postcard::to_stdvec(&response).unwrap();
|
||||
let server = tokio::spawn(async move {
|
||||
while let Some(mut stream) = acceptor.accept().await {
|
||||
let mut request = Vec::new();
|
||||
let _ = AsyncReadExt::read_to_end(
|
||||
&mut (&mut stream.recv).take(MAX_PODCAST_DHT_REQUEST_BYTES as u64),
|
||||
&mut request,
|
||||
)
|
||||
.await;
|
||||
let _ = stream.send.write_all(&payload).await;
|
||||
let _ = stream.send.finish();
|
||||
let _ = tokio::time::timeout(Duration::from_secs(2), stream.send.stopped()).await;
|
||||
}
|
||||
});
|
||||
m.connect(a.service.ticket().await.unwrap()).await.unwrap();
|
||||
wait_for("music DHT contact", async || {
|
||||
(!a.service.known_peers().is_empty()).then_some(())
|
||||
})
|
||||
.await;
|
||||
|
||||
// Bootstrap learns M, which answered, and none of the contacts it made
|
||||
// up: they are tried once and never answer.
|
||||
a.dht.bootstrap().await.unwrap();
|
||||
assert_eq!(a.dht.known_peers(), 1);
|
||||
// A lookup treats them as candidates only.
|
||||
let podcast_id = PodcastId::from_feed_url(&feed("nothing.xml"));
|
||||
assert!(a.dht.lookup_podcast(&podcast_id).await.unwrap().is_none());
|
||||
assert_eq!(a.dht.known_peers(), 1);
|
||||
a.dht.bootstrap().await.unwrap();
|
||||
assert_eq!(
|
||||
a.dht.known_peers(),
|
||||
1,
|
||||
"fabricated contacts never count for replica responsibility"
|
||||
);
|
||||
|
||||
server.abort();
|
||||
a.stop().await;
|
||||
m_events.abort();
|
||||
m.shutdown().await.unwrap();
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||
async fn restarted_device_waits_for_sync_shares_before_resending_cards() {
|
||||
tokio::time::timeout(TEST_TIMEOUT, restart_scenario())
|
||||
.await
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
async fn restart_scenario() {
|
||||
let temp = tempfile::tempdir().unwrap();
|
||||
let network = NetworkId::from_name("podcast-dht-restart");
|
||||
let state =
|
||||
|subscription: &SubscriptionRecord, shared: bool, generation: u64| PodcastShareState {
|
||||
subscription: subscription.clone(),
|
||||
share: Some(share(subscription, shared, generation)),
|
||||
card: None,
|
||||
privacy_confirmed: false,
|
||||
};
|
||||
let kept = subscription("kept.xml", "Kept Show");
|
||||
let revoked = subscription("revoked.xml", "Revoked Show");
|
||||
|
||||
// A publishes both podcasts while alone, then goes offline.
|
||||
let a = Node::start(&temp.path().join("a"), network).await;
|
||||
let stats = a
|
||||
.dht
|
||||
.sync_shares(vec![state(&kept, true, 2), state(&revoked, true, 2)])
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!((stats.published, stats.push.remote_nodes), (2, 0));
|
||||
a.stop().await;
|
||||
|
||||
// Meanwhile another trusted device revokes "Revoked Show". A comes back
|
||||
// with its persisted cards and runs maintenance before its client
|
||||
// reconciled: the first bootstrap finds B.
|
||||
let b = Node::start(&temp.path().join("b"), network).await;
|
||||
let mut a = Node::start(&temp.path().join("a"), network).await;
|
||||
b.service
|
||||
.connect(a.service.ticket().await.unwrap())
|
||||
.await
|
||||
.unwrap();
|
||||
wait_for("music DHT contacts", async || {
|
||||
(!a.service.known_peers().is_empty()).then_some(())
|
||||
})
|
||||
.await;
|
||||
a.tasks.push(tokio::spawn(Arc::clone(&a.dht).maintenance()));
|
||||
wait_for("overlay contacts", async || {
|
||||
(a.dht.known_peers() >= 1 && b.dht.known_peers() >= 1).then_some(())
|
||||
})
|
||||
.await;
|
||||
// Neither maintenance nor an explicit republish sends a stale card, and
|
||||
// A does not serve them to lookups either.
|
||||
let stats = a.dht.republish().await.unwrap();
|
||||
assert_eq!(stats.records, 0, "stale cards are withheld");
|
||||
for podcast in [&kept, &revoked] {
|
||||
assert!(
|
||||
b.dht
|
||||
.lookup_podcast(&podcast.podcast_id)
|
||||
.await
|
||||
.unwrap()
|
||||
.is_none()
|
||||
);
|
||||
}
|
||||
assert!(search(&b, "show").await.is_empty());
|
||||
|
||||
// The client's first reconciliation revokes the stale share and
|
||||
// re-sends the unchanged card.
|
||||
let stats = a
|
||||
.dht
|
||||
.sync_shares(vec![state(&kept, true, 2), state(&revoked, false, 3)])
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
(stats.published, stats.unchanged, stats.unpublished),
|
||||
(0, 1, 1)
|
||||
);
|
||||
assert_eq!(stats.push.records, 2, "the kept card and the tombstone");
|
||||
assert!(stats.push.remote_nodes >= 1);
|
||||
let a_only = vec![a.service.endpoint_id()];
|
||||
let hit = b
|
||||
.dht
|
||||
.lookup_podcast(&kept.podcast_id)
|
||||
.await
|
||||
.unwrap()
|
||||
.expect("the kept card is found");
|
||||
assert_eq!(hit.publishers, a_only);
|
||||
assert!(
|
||||
b.dht
|
||||
.lookup_podcast(&revoked.podcast_id)
|
||||
.await
|
||||
.unwrap()
|
||||
.is_none()
|
||||
);
|
||||
// From now on republishing covers the kept card and the tombstone.
|
||||
let stats = a.dht.republish().await.unwrap();
|
||||
assert_eq!(stats.records, 2);
|
||||
let hits = search(&b, "show").await;
|
||||
assert_eq!(hits.len(), 1);
|
||||
assert_eq!(hits[0].podcast_id, kept.podcast_id);
|
||||
|
||||
a.stop().await;
|
||||
b.stop().await;
|
||||
}
|
||||
Reference in new issue
Block a user