15 Commits
Author SHA1 Message Date
sanapachandClaude Opus 5.5 d2530dca6c Convert received playback items without copying wire ids
Add TrackItem::from_playback_track and PlaybackItem: a received podcast
episode becomes TrackItem::podcast_episode (sentinel ids) and a Jam
private placeholder gets no TrackItem, so the wire ids (0 for both) are
never copied. Narrow the TrackItem::key guidance: it is the source
identity, episodes must never be keyed by id, and content-first music
matching stays as it is. Document how TrackItem::podcast_episode differs
from the wire form.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 18:32:38 +02:00
sanapachandClaude Opus 5.5 b8d8b532f2 Remove imported podcast episodes that duplicate a held id
A provisional imported GUID-less episode whose feed item id is already
held by an episode that stays (typically one a remote operation created
before this device's first refresh) is now deleted instead of kept as a
second, never re-keyed copy. The same applies when its legacy GUID turns
out to be the item's real GUID. Such episodes carry no personal state.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 18:29:19 +02:00
sanapachandClaude Opus 5.5 06ac3691d8 Key podcast queue items by source, not by a negative id range
The per-episode negative ids (TrackItem::podcast_track_id below
PODCAST_TRACK_ID_MAX) were a magic id range in shared code, which the
podcast requirements forbid. Podcast items now use the fixed sentinel
TrackItem::NOT_A_LIBRARY_ROW (-1) as id, release_id and artist id; it
carries no identity. Their identity is TrackItem.podcast alone.

Add TrackItem::key returning a typed TrackItemKey (library row, federated
owner and item id, podcast episode id, or transient), with the stable
string forms lib:<id>, fed:<owner>:<item_id>, ep:<episode id> and
tmp:<id>, so clients key queue items by source instead of by numeric id.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 18:04:37 +02:00
sanapachandClaude Opus 5.5 9709f5e2b5 Re-key imported GUID-less episodes and make fetch-state writes atomic
The prototype kept one row per enclosure URL for GUID-less items, so stale
rows could change which identity key is unique and give imported episodes
ids a fresh device never derives. The import now collapses rows of one item
seen under several enclosure URLs, and the first refresh re-keys imported
GUID-less episodes without progress or sessions to the ids the current feed
gives.

Fetch-state writes read and write the row under one lock and one immediate
transaction, so concurrent writers no longer lose columns or failure counts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 17:02:31 +02:00
sanapachandClaude Opus 5.5 e627be51d4 Give podcast queue items fail-safe negative ids
TrackItem::podcast_episode used id 0, release_id 0 and artist id 0, which
clients treat as library rows. Podcast items now get a negative id derived
from the episode id (TrackItem::podcast_track_id, at most
PODCAST_TRACK_ID_MAX, below pending federated ids) and release/artist id -1,
so existing id < 0 / release_id < 0 checks skip them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 17:02:31 +02:00
sanapachandClaude Opus 5.5 49ae6423f6 Document the SQLite podcast store
PODCASTS.md gains the "SQLite reference store" section (migrations,
prototype import, tables, invariants); ARCHITECTURE.md, the
furumi-library README and the changelog describe PodcastStore and
TrackItem.podcast.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 16:42:58 +02:00
sanapachandClaude Opus 5.5 e22c470ea7 Add SQLite podcast store with prototype import
New `furumi_library::podcasts::PodcastStore` in its own podcasts.sqlite3:

- user_version migrations in immediate transactions; version 1 is the
  prototype schema of the original podcast PR, version 2 renames its tables
  to legacy_* and imports feeds, episodes and fetch state with contract
  ids; unimportable rows go to legacy_quarantine; newer or foreign files
  are refused before anything is written;
- catalog (podcasts, feed aliases, seasons, episodes), subscription, share
  and progress registers, listening sessions and local-only fetch state;
- local actions that apply and return podcast sync operations, remote
  operation and snapshot application through the music-dht materializers
  and merge functions, bounded snapshot export;
- RSS refresh ingestion with stable episode ids and alias learning that
  never touches personal state, and queries for the UI.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 16:42:52 +02:00
sanapachandClaude Opus 5.5 35115ecc98 Add TrackItem.podcast for podcast queue items
TrackItem gains `podcast: Option<PodcastEpisodeRef>` with
`TrackItem::podcast_episode`, `is_podcast` and `podcast_episode_ref`.
Library queries keep returning music only (`podcast: None`); the library
database is unchanged. Music-only code paths must check `is_podcast`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 16:42:52 +02:00
sanapachandClaude Opus 5.5 65d6ca2f14 Document podcast register ordering, versions and snapshot limits
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 16:10:33 +02:00
sanapachandClaude Opus 5.5 085bff6191 Harden podcast sync registers, versions and snapshots
- Order subscription and share registers by a per-podcast Lamport
  generation before hlc_ms (next_podcast_generation), so clock skew
  cannot undo an observed unsubscribe or share decision.
- share=false is register state, not a compactable tombstone.
- Add podcast_feed_aliases_added so refreshes can sync aliases without
  touching the registers; SubscriptionRecord::feed_urls keeps the current
  feed URL resolvable beyond the alias bound.
- Carry progress_generation on queue items for handoff.
- Distinguish self-reported and relayed peer versions, and gate
  PairRequest by an optional inviter version in InviteWire.
- Give each podcast snapshot section its own byte budget, drop channel
  descriptions from snapshots and allow a caller-provided budget.
- Reject one-slash http(s) typos in FeedUrl::from_user_input.
- Pin that malformed ids and feed URLs reject the whole message.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 16:10:33 +02:00
sanapachandClaude Opus 5.5 629857dbfe Make podcast episode identity order-independent and linear
Run the no-GUID matching stages across the whole document so a weak
date+title match of an earlier item cannot take a known episode that a
later item matches by enclosure. Keep feed links verbatim (trimmed only)
because they are identity material, and stop html_to_text from scanning
to the end of the text at every '<' when no '>' follows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 16:10:33 +02:00
sanapachandClaude Opus 5.5 4f7e999280 Document podcast contract and bump crate versions
Add PODCASTS.md as the normative podcast contract: identity and URL
canonicalization, the no-GUID rule, synced versus local versus public
data, operation kinds and merge rules with examples, snapshot recovery,
version gating for peers below v4, Jam privacy and the reserved
playback_rate. Update PLAYBACK_PROTOCOL.md, ARCHITECTURE.md, the crate
READMEs and CHANGELOG.md.

Bump music-dht to 0.6.0 and furumi-library to 0.3.0: new public enum
variants and struct fields are breaking changes for 0.x, and
furumi-library re-exposes music-dht types.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 15:40:55 +02:00
sanapachandClaude Opus 5.5 2d4782819d Add podcasts to device-sync protocol v4
Bump DEVICE_SYNC_PROTOCOL_VERSION to 4 (ALPN unchanged) and carry the
podcast contract between trusted devices:

- podcast_subscription_set, podcast_share_set, podcast_progress_set and
  podcast_listen_recorded operations with record materializers and
  validation; unsubscribe is LWW state, share=false is a tombstone;
- bounded podcast sections in SyncSnapshot, the recovery path for peers
  that upgrade after operations were hidden from them;
- PlaybackTrack.source with PlaybackSource::{Music, PodcastEpisode,
  Private}, reserved PlaybackStateWire.playback_rate and an optional
  self-reported protocol_version in DeviceProfileSet;
- gating helpers for peers below v4 (op_allowed_for_peer,
  playback_allowed_for_peer, playback_command_allowed_from_peer,
  SyncSnapshot::for_peer) and Jam privacy (redact_for_jam,
  redact_jam_snapshot, jam_command_allowed, restore_jam_private_items).

Tests show that a v3-shaped enum rejects a whole batch containing a new
kind, which is why gating is mandatory, and that legacy JSON still
parses and music-only messages keep their v3 shape.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 15:40:43 +02:00
sanapachandClaude Opus 5.5 31d132bb3f Add podcast-feed RSS and Atom parser
Port the tolerant RSS 2.0 / RSS 1.0 / Atom parser of the original
furumi-tui podcast work into music_dht::podcast::feed behind the
optional podcast-feed feature (quick-xml, encoding_rs, url). Episode
identity depends on what the parser extracts, so it must be shared by
all clients.

The port keeps its robustness (BOM and declared encodings, entities,
mismatched tags, depth and size limits, lenient dates, http(s)-only
media) and adds itunes:season, itunes:episode, itunes:episodeType,
guid isPermaLink and itunes:new-feed-url. Missing GUIDs stay absent
instead of falling back to the enclosure URL, and parsed items produce
identity hints and contract metadata.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 15:40:19 +02:00
sanapachandClaude Opus 5.5 38e49c4926 Add shared podcast contract to music-dht
Introduce music_dht::podcast, a pure module (no SQLite, HTTP or clock)
that every Furumi client uses for podcast identity and personal state:

- FeedUrl canonicalization that keeps the path and every query
  parameter byte-for-byte, PodcastId/EpisodeId derivation with
  domain-separated BLAKE3, feed aliases and the documented no-GUID
  episode identity rule;
- bounded PodcastMeta, EpisodeMeta and the self-contained
  PodcastEpisodeRef queue item;
- convergent merge rules for subscriptions (LWW, unsubscribe is state),
  share decisions (false is a tombstone), progress registers ordered by
  a Lamport generation, and listening sessions (G-set, one completion
  per session).

Tests cover every delivery permutation of small operation sets, late
checkpoints of old sessions, concurrent offline sessions and RSS
metadata updates that must not touch progress.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 15:39:50 +02:00
32 changed files with 15202 additions and 20 deletions

No files matched your search

+58 -2
View File
@@ -12,6 +12,7 @@ application
furumi-library
├── local catalog, import, migrations
├── playlists, likes, history, similarity persistence
├── podcast store (podcasts.sqlite3)
│
music-dht
├── Kademlia routing and iterative lookup
@@ -41,9 +42,30 @@ metadata import, artwork resolution, playlists, likes, listening history, and
similarity index storage. Connections enable foreign keys, WAL, and a bounded
busy timeout so independently running clients can safely share the database.
Podcasts use a separate store, `furumi_library::podcasts::PodcastStore`, in
its own `podcasts.sqlite3` file next to the library database. It implements
the podcast contract of `music_dht::podcast` for SQLite clients: the episode
catalog refreshed from RSS, the subscription, share and progress registers,
grow-only listening sessions, feed aliases and local-only fetch state.
Local actions, received operations and snapshot sections share one apply
path built on the `music-dht` materializers and merge functions, so the store
converges independently of delivery order. Migrations follow
`PRAGMA user_version`; the prototype podcast database is imported and its
tables are kept as `legacy_*`; newer or foreign files are refused unchanged.
`TrackItem.podcast` marks episode queue items, which music-only code must
skip. An episode queue item is identified by its episode id
(`TrackItem::key`, a typed source identity: library row, federated owner and
item id, or episode id), never by the numeric id: episodes use the fixed
sentinel id, `release_id` and artist id -1, which carries no identity and
only keeps code written for library tracks (`id >= 0`) away from them.
Received playback items are converted with `TrackItem::from_playback_track`,
which never copies the wire ids (0 for episodes and private placeholders). See the "SQLite reference store" section
of [PODCASTS.md](PODCASTS.md).
Filesystem scanning schedules, playback, federation lifecycle, and UI state
remain application responsibilities. Database and metadata work must stay off
interactive and async reactor threads.
interactive and async reactor threads. RSS fetching and refresh scheduling are
application work as well.
## Transport: federation-net
@@ -177,13 +199,42 @@ their local output is playing. The web gateway uses a passive server profile;
browser startup and explicit selection are separate inputs.
This register provides eventual exclusivity after a partition heals, not a
quorum lock during a partition. Device-sync profiles now advertise version 3;
quorum lock during a partition. Device-sync profiles now advertise version 4
(version 3 introduced coordination, version 4 adds podcast queue items);
the additive coordination envelope is independently versioned and transported
over the existing JSON `furumi/sync/2` stream. Legacy library records remain
readable, but legacy unfenced personal playback messages cannot control an
upgraded output. See [PLAYBACK_PROTOCOL.md](PLAYBACK_PROTOCOL.md) for the
contract, adapter obligations, configuration and rollout rules.
Podcasts are a separate content type with a pure shared contract in
`music_dht::podcast` (normative description: [PODCASTS.md](PODCASTS.md)). It
owns canonical feed URLs, domain-separated BLAKE3 podcast and episode ids,
feed aliases, the no-GUID episode rule, bounded metadata DTOs, the
self-contained `PodcastEpisodeRef` queue item, and convergent merge functions
for subscriptions, share decisions, progress registers and listening
sessions. The tolerant RSS/Atom parser is shared behind the `podcast-feed`
feature because episode identity depends on it. HTTP fetching, scheduling,
buffering and decoding remain client runtime work, and persistence lives in
client storage that implements the same rules. Podcast audio is fetched from
the publisher: it is never imported, content-addressed or offered for P2P
transfer.
Device-sync protocol v4 carries podcast state between trusted devices only:
five `podcast_*` operations, bounded snapshot sections (the recovery path,
because sync vectors never re-request operations hidden from an older peer;
past the bounds an upgraded peer recovers only the newest records), and an
explicit `PlaybackTrack.source`. The operation enum is closed and v3 clients
ignore `source`, so senders gate on the version each peer reported itself
(relayed profiles only raise it; pairing requests use the version in the
invite); podcast state is never sent to peers below v4, handoff to them is
refused, and their playback commands are ignored while the queue contains
episodes. Jam snapshots replace episodes with opaque placeholders. Progress,
subscription and share registers order by a Lamport generation before
wall-clock `hlc_ms`, so a device with a fast clock cannot undo an observed
seek, restart, unsubscribe or share decision. Public podcast discovery is a future
separate overlay; the music DHT schema and `federation-net` are unchanged.
Music discovery identifies an owner and content; it does not force all domain
traffic through DHT messages. `music-dht` exposes the underlying authenticated
stream capability and defines shared wire models for Furumi catalog and device
@@ -283,3 +334,8 @@ architectural constraint:
- Local durable state survives ordinary peer and network failure.
- Additional stream protocols reuse authenticated connectivity without
coupling bulk data to the typed control channel.
- Private personal state (podcast subscriptions, progress, history) stays in
the trusted-device group; Jam and public overlays only see explicit public
projections or opaque placeholders.
- A device's protocol version is what that device reported; relays never
substitute their own.
+51
View File
@@ -1,5 +1,56 @@
# Changelog
## Unreleased — podcast contract
- `music-dht 0.6.0` (breaking): new `music_dht::podcast` module with the shared
podcast contract: canonical `FeedUrl`, `PodcastId`/`EpisodeId` derivation
(`furumi-podcast:*:v1`), feed aliases, the no-GUID episode identity rule,
`PodcastMeta`/`EpisodeMeta`/`PodcastEpisodeRef`, and convergent merge rules
for subscriptions, shares, progress registers and listening sessions.
- `music-dht 0.6.0`: optional `podcast-feed` feature with the shared tolerant
RSS/Atom parser (`quick-xml`, `encoding_rs`, `url`).
- `music-dht 0.6.0`: device-sync protocol version 4 (ALPN unchanged): five
`podcast_*` operation kinds (subscription, share, feed aliases, progress,
listening sessions), bounded podcast snapshot sections,
`PlaybackTrack.source`, reserved `PlaybackStateWire.playback_rate`,
`DeviceProfileSet.protocol_version`, optional `InviteWire` protocol version
(`p`), self-reported vs relayed version helpers, version-gating helpers and
Jam redaction. New struct fields and enum variants break exhaustive
construction and matching in clients.
- `furumi-library 0.3.0`: follows `music-dht 0.6.0`, whose types it re-exposes.
- `furumi-library 0.3.0`: new `furumi_library::podcasts::PodcastStore`, the
SQLite reference store of the podcast contract in its own
`podcasts.sqlite3`: `user_version` migrations that import the prototype
podcast database (tables kept as `legacy_*`, unimportable rows in
`legacy_quarantine`) and refuse newer or foreign files unchanged; podcasts,
feed aliases, seasons, episodes, subscription/share/progress registers,
listening sessions and local-only fetch state; local actions that return
the `podcast_*` operations for the client's sync log; `apply_remote_op`,
bounded snapshot export and merge; RSS refresh ingestion with stable
episode ids; UI queries. Enables the `music-dht` `podcast-feed` feature.
- `furumi-library 0.3.0` (breaking): `TrackItem.podcast:
Option<PodcastEpisodeRef>` and `TrackItem::podcast_episode`/`is_podcast`.
Exhaustive `TrackItem` literals in clients need the new field; music-only
code must skip podcast items. Podcast items use the fixed sentinel
`TrackItem::NOT_A_LIBRARY_ROW` (-1) as `id`, `release_id` and artist id,
so existing `id < 0` / `release_id < 0` checks already treat them as
non-library items; the sentinel carries no identity. New
`TrackItem::key` / `TrackItemKey` (`lib:<id>`, `fed:<owner>:<item_id>`,
`ep:<episode id>`), the source identity: episodes must be keyed by it (or
`TrackItem.podcast`), never by `id`; content-first music matching stays
as it is. New `TrackItem::from_playback_track` / `PlaybackItem` convert
received playback items without copying wire ids (0 for episodes and Jam
private placeholders). The library database is unchanged.
- `furumi-library 0.3.0`: prototype import collapses rows of one GUID-less
item under several enclosure URLs, and the first refresh re-keys imported
GUID-less episodes without personal state to the ids the current feed
gives, so they match devices that never had the prototype; when another
episode (for example one created by a remote operation) already holds
that id, the imported copy is deleted instead of kept as a duplicate.
Fetch-state
writes are atomic across threads and processes.
- See `PODCASTS.md` for the normative contract.
## v0.4.1 — 2026-09-02
- `federation-net 0.3.1`: recover timed-out or failed Mainline-DHT rendezvous clients and expose network health to applications.
Generated
+32 -2
View File
@@ -389,6 +389,12 @@ version = "0.8.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b"
[[package]]
name = "core_detect"
version = "1.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7f8f80099a98041a3d1622845c271458a2d73e688351bf3cb999266764b81d48"
[[package]]
name = "cpufeatures"
version = "0.2.17"
@@ -790,6 +796,20 @@ version = "0.6.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "edd0f118536f44f5ccd48bcb8b111bdc3de888b58c74639dfb034a357d0f206d"
[[package]]
name = "encoding_rs"
version = "0.8.44"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "89253175e704ab869b04f9db224d38630e0211a53841a79181deae0889e169f5"
dependencies = [
"cfg-if",
"core_detect",
"multiversion_no_op",
"rustversion",
"scopeguard",
"simdutf8",
]
[[package]]
name = "enum-assoc"
version = "1.3.0"
@@ -912,7 +932,7 @@ dependencies = [
[[package]]
name = "furumi-library"
version = "0.2.0"
version = "0.3.0"
dependencies = [
"anyhow",
"blake3",
@@ -922,6 +942,7 @@ dependencies = [
"rusqlite",
"serde",
"serde_json",
"tempfile",
"tracing",
]
@@ -2071,18 +2092,26 @@ dependencies = [
"uuid",
]
[[package]]
name = "multiversion_no_op"
version = "1.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "743fb55ba31b18fb1ecef6bdc9aa2743314978ac084044301a7eee33fb99a20d"
[[package]]
name = "music-dht"
version = "0.5.0"
version = "0.6.0"
dependencies = [
"anyhow",
"async-trait",
"blake3",
"data-encoding",
"encoding_rs",
"federation-net",
"futures",
"iroh",
"postcard",
"quick-xml",
"rand 0.9.5",
"rusqlite",
"serde",
@@ -2092,6 +2121,7 @@ dependencies = [
"tokio",
"tracing",
"unicode-normalization",
"url",
]
[[package]]
+4 -1
View File
@@ -14,7 +14,7 @@ repository = "https://gt.hexor.cy/ab/frid"
[workspace.dependencies]
federation-net = { path = "crates/federation-net", version = "0.3.1" }
music-dht = { path = "crates/music-dht", version = "0.5.0" }
music-dht = { path = "crates/music-dht", version = "0.6.0" }
iroh = "1"
iroh-base = "1"
iroh-tickets = "1"
@@ -38,3 +38,6 @@ lofty = "0.22"
data-encoding = "2"
rand = "0.9"
tempfile = "3"
quick-xml = "0.41"
encoding_rs = "0.8.35"
url = "2.5"
+55 -2
View File
@@ -97,8 +97,9 @@ is a runtime input, not a persisted setting.
## Versioning and rollout
The library-sync JSON ALPN remains `furumi/sync/2`; catalog, pairing and library
data do not require a new transport. Device profiles advertise protocol version
3. Coordination envelopes independently carry version 1. Snapshot coordination
data do not require a new transport. Device profiles advertised protocol
version 3 when coordination was introduced (version 4 adds podcast queue items,
see below). Coordination envelopes independently carry version 1. Snapshot coordination
and operation authority are additive optional JSON fields so library-sync
records from older clients still deserialize. Upgraded clients ignore legacy
personal playback snapshots/commands without the coordination envelope.
@@ -109,6 +110,58 @@ applications while preventing divergent control message definitions. These
envelopes use JSON, not a backwards-compatible promise for postcard layouts.
Do not change Jam's independent authority by interpreting personal stamps there.
## Protocol v4: podcast queue items
Device profiles advertise protocol version 4. The JSON ALPN stays
`furumi/sync/2` and the coordination envelope stays version 1. Version 4 adds
to the portable queue DTO:
- `PlaybackTrack.source`: `music`, `podcast_episode` with a self-contained
`PodcastEpisodeRef`, or `private` (Jam placeholder). It is omitted for
music, so music-only state serializes exactly as in v3. Podcast episodes are
never encoded as negative ids or URLs in `file_path`. Queue items carry
`PodcastEpisodeRef.progress_generation`, the greatest progress generation
the sender has seen for the episode.
- `PlaybackStateWire.playback_rate`: reserved, omitted when unset. Treat a
missing or unsupported value as 1.0.
A v3 client does not know `source` and ignores it: it would treat an episode
as a broken music item (id 0, no content id) and forward the queue that way,
losing the episode for every device. `PlaybackSource` is also a closed enum
for v4 clients, so a future variant needs a new protocol version. Adapters
therefore apply the shared helpers from `music_dht::device_sync`:
1. Send a playback snapshot to a peer only when
`playback_allowed_for_peer(state, peer_version)` holds; otherwise send
`None`.
2. Refuse an explicit transfer to a device below v4 while the queue contains
podcast items and show that the target needs an update. The decision uses
the version that device reported itself (`profile_protocol_version`;
profiles relayed by other devices only raise it, `relayed_protocol_version`),
never the local constant. Do not strip episodes from the queue.
3. Ignore `SetState`/`ActiveChanged` from peers below v4 while the local
queue contains podcast items (`playback_command_allowed_from_peer`).
4. A stamped `playback_command` operation with podcast items is itself gated
by `op_allowed_for_peer`.
5. A `PairRequest` is sent before the invite consumer hears from the inviter:
gate its playback snapshot by `InviteWire::inviter_protocol_version()`
(version 1 for invites without one).
A handoff of a podcast episode records the position and the play/pause intent
first, transfers the queue with its sources, and lets the receiver open the
publisher's HTTP source and seek. Ownership still changes only through the
claim, so a late response cannot make the previous owner resume audio. The
durable episode progress is a separate register; see `PODCASTS.md`. Only the
receiver writes the handoff's explicit progress action, with
`ProgressRecord::explicit_generation(max(local, progress_generation))`; the
sender writes a final checkpoint at its session generation.
Jam hosts send `music_dht::jam::redact_jam_snapshot`, which replaces podcast
items with opaque `private` placeholders and drops the personal coordination
envelope. They reject participant commands with real episodes
(`jam_command_allowed`) and map placeholders back with
`restore_jam_private_items` before applying a participant's state.
For local integration checks, patch the three frid crates with Cargo's external
`--config` mechanism. Keep released dependency declarations and lockfiles in
the applications; publish frid and then bump client dependencies for rollout.
+671
View File
@@ -0,0 +1,671 @@
# Podcast contract
Status: normative for `music-dht` 0.6 and device-sync protocol v4.
Podcasts are a separate content type shared by every Furumi client. This
document defines what all clients must agree on: identity, normalization,
private state and its merge rules, trusted-device sync operations, snapshot
recovery, queue items, version gating and Jam privacy. The Rust definitions
live in `music_dht::podcast` (pure types and functions) and
`music_dht::device_sync` (wire DTOs and gating helpers). Clients must use
them instead of local copies.
Public discovery (signed podcast cards in a separate DHT overlay) is not part
of this version of the contract; it will be specified with the overlay.
## Responsibilities
| Layer | Owns |
| --- | --- |
| `music_dht::podcast` | ids, URL canonicalization, no-GUID rule, metadata DTOs and bounds, merge functions, RSS/Atom parser (feature `podcast-feed`) |
| `music_dht::device_sync` | sync operation kinds, snapshot sections, `PlaybackSource`, version gating, Jam redaction |
| Client storage (`furumi_library::podcasts::PodcastStore`, web PostgreSQL) | persistence and migrations implementing the rules below (SQLite reference: see "SQLite reference store") |
| Client runtime | HTTP fetching, conditional requests, scheduling, audio buffering and decoding, UI |
Audio is always fetched from the publisher. It is never imported into the
music library, never content-addressed with a BLAKE3 content id and never
offered for P2P transfer. RSS refreshes fetch metadata only.
## Data classes
| Data | Trusted devices | Public federation | Local only |
| --- | --- | --- | --- |
| Subscription state, channel metadata | `podcast_subscription_set` + snapshot | never | |
| Feed aliases | `podcast_feed_aliases_added`, `podcast_subscription_set` + snapshot | never | |
| Share decision | `podcast_share_set` + snapshot | only the published card (future overlay) | |
| Episode progress | `podcast_progress_set` + snapshot | never | |
| Listening sessions, play counts | `podcast_listen_recorded` + snapshot | never | |
| Episode catalog | each device parses RSS itself; `PodcastEpisodeRef` travels with progress and queue items | never | |
| ETag/Last-Modified, last check time, fetch errors, artwork cache, audio buffer | | | yes |
Synchronizing to a trusted device is not permission to publish. Private
podcasts, their URLs, descriptions and episodes must not reach DHT records,
public catalogs, public search results or Jam participants.
## Identity
### Feed URL canonicalization
`FeedUrl::parse` produces the canonical form. Every client must use it (or
an implementation that passes the same tests) before deriving ids or
comparing URLs.
1. Only `http://` and `https://` are accepted (case-insensitive) and the
scheme is lowercased.
2. Userinfo (`user@`, `user:pass@`), whitespace, control characters and `\`
are rejected.
3. The host is lowercased. Non-ASCII hosts are rejected; pass the IDNA
(`xn--`) form as produced by WHATWG URL parsers. Bracketed IPv6 literals
are accepted.
4. An empty port and the default port (80 for http, 443 for https) are
removed; other ports lose leading zeros.
5. The fragment is removed.
6. An empty path becomes `/`. The rest of the path and the **entire** query
string are kept byte-for-byte: parameter order, case, percent-encoding and
dot segments are preserved. Query parameters can be part of the source
identity (for example a per-user token) and must never be dropped. Only
non-ASCII characters are percent-encoded (UTF-8, uppercase hex).
7. The result is at most 2048 bytes. Canonicalization is idempotent;
deserialization accepts only canonical strings.
| Input | Canonical |
| --- | --- |
| `HTTPS://Example.COM:443/Feed.xml?b=2&a=1#top` | `https://example.com/Feed.xml?b=2&a=1` |
| `http://example.com:80` | `http://example.com/` |
| `https://example.com/rss?token=XyZ&x=%2f` | unchanged |
| `https://user@example.com/rss` | rejected |
`FeedUrl::from_user_input` is for typed or pasted input only: it trims, maps
`feed:`, `podcast:`, `pcast:`, `itpc:` to `https://` and adds `https://` when
the scheme is missing, then canonicalizes.
### Podcast id
```text
podcast_id = "pod1:" + base32_lower_nopad(BLAKE3("furumi-podcast:podcast:v1" || 0x00 || canonical_feed_url))
```
The id is derived **once**, from the canonical URL of the first subscription.
It never changes afterwards.
### Feed aliases
Redirect targets (permanent redirects), `itunes:new-feed-url` and URLs used to
subscribe again are feed aliases of an existing podcast:
* aliases form a grow-only set per podcast, merged independently of the
subscription register (`merge_feed_aliases`). The URL that derives the
podcast id is always kept, plus the 8 lexicographically smallest other
URLs; this bounded union is commutative, associative and idempotent;
* aliases travel in their own operation, `podcast_feed_aliases_added`. A
background refresh that learns a URL emits it, but only when merging the
URL changes the stored set (a URL outside the bound is not sent again on
every refresh). The operation never touches the subscription or share
register, so it cannot resurrect a subscription or reset a share decision.
Subscribe and unsubscribe operations also carry the aliases known at that
time;
* `podcast.feed_url` of the subscription register is the URL used at the last
explicit subscribe or unsubscribe. It can fall outside the bounded alias
set, so the URLs that identify a podcast are
`SubscriptionRecord::feed_urls()`: the aliases plus `podcast.feed_url`;
* before deriving a new id for a URL, clients call `resolve_podcast_id` with
every known `(podcast_id, url)` pair, including `feed_urls()` and the URL
each device currently fetches. A match reuses the existing podcast, its
progress and its history. With several matches the smallest id wins.
Limitations: two devices that subscribe offline, through different URLs of
the same feed, before either learns the alias, create two podcast ids; the
contract does not merge podcast ids automatically in this version. A podcast
with more than 8 aliases can lose a larger URL from the synced set; a device
that only knows that URL derives a new id.
### Episode id
```text
episode_id = "ep1:" + base32_lower_nopad(BLAKE3("furumi-podcast:episode:v1" || 0x00 || podcast_id || key))
```
| Key | Material |
| --- | --- |
| GUID | `guid:` + RSS `<guid>` / Atom `<id>` trimmed, case-sensitive |
| link | `link:` + item `<link>` text (XML-unescaped once) or Atom `href`, trimmed of surrounding whitespace; no other cleanup |
| date + title | `date-title:` + publication unix seconds + `\0` + normalized title |
| enclosure | `enclosure:` + enclosure URL trimmed |
The normalized title is `music_dht::normalize_name` (NFKC, lowercase,
non-alphanumeric runs collapsed). GUIDs longer than 2048 bytes are treated as
absent. Changing the title, artwork, description or audio URL of an episode
never changes its id and never resets its progress.
### No-GUID rule
`resolve_episode_ids(podcast_id, channel_link, items, known)` is normative.
For one parsed feed document:
1. Items with a GUID get the GUID id. A GUID never matches another known
episode, so the result does not depend on local history.
2. Items without GUID then search the known episodes of the podcast in three
stages, each run over the **whole document** before the next one starts:
same enclosure URL, then same `<link>`, then same publication second and
normalized title. Among candidates the smallest id is reused. A known
episode can be claimed by only one item. Because a weak match of an
earlier item cannot take a known episode that a later item matches by
enclosure, the result does not depend on the order of items.
3. Otherwise a new id is derived from the first available key: `<link>`,
then date + title, then enclosure URL.
A `<link>` is identifying only if it is unique in the document and differs
from the channel `<link>` (many feeds repeat the homepage in every item).
Enclosure URLs and date + title pairs shared by several GUID-less items are
not identifying either. Items without any usable key, and duplicates of an
earlier item of the same document, get no id and are skipped.
The identity hints (GUID, enclosure, link, publication time, title) are the
ones `music_dht::podcast::feed` produces: which enclosure or `media:content`
is chosen, how relative URLs are resolved, which date field is used and how
it is parsed, how the title is cleaned. They are part of the rule; clients
must use that parser (or an implementation that passes its tests). Relative
media URLs are resolved against the URL the document was fetched from.
"Known episodes" include episodes from earlier refreshes **and** episodes
learned through sync (`PodcastEpisodeRef::identity_hints`). Feed GUID-less
episodes into it so a device that sees an episode first through sync keeps
the id used by the other devices.
Limitations, GUID-less episodes only:
* The id can depend on what a device saw first. If the publisher changes the
field the id was derived from (the `<link>`, or the title or date for a
date + title id), a device that saw the old version keeps the old id
through the enclosure match, while a device that never saw it derives a
new id. One changed field is enough.
* Feeds with relative enclosure URLs fetched through different aliases give
different enclosure URLs, so the enclosure match and enclosure-derived ids
can differ between such devices.
## Metadata and bounds
| Field | Bound |
| --- | --- |
| titles, author | 1000 characters, non-empty titles |
| descriptions | 4000 characters (plain text) |
| URLs | 2048 bytes, audio and artwork URLs absolute `http(s)` |
| GUID | 2048 bytes |
| MIME | 128 printable ASCII bytes |
| device, session, operation ids | 1..256 bytes, no control characters |
| feed aliases | 8 plus the origin URL |
`PodcastMeta` (channel), `EpisodeMeta` (episode, including season, episode
number, `itunes:episodeType`, publication time and duration in milliseconds,
audio URL, MIME, byte length, artwork and link) and `PodcastEpisodeRef`
(queue item) validate these bounds. `PodcastMeta::bounded` and
`EpisodeMeta::to_ref` truncate publisher data to fit. Receivers drop records
that fail validation.
Episodes without a season are shown in their own group. Unknown
`itunes:episodeType` values are `full`.
## Personal state and merge rules
All merge functions are commutative, associative and idempotent per entity;
ties are broken by a total order over all fields. The same set of operations
therefore yields the same state in any delivery order, with duplicates.
`hlc_ms` is the sync operation timestamp; clients advance it only past their
own operations, so it is not a hybrid logical clock and a device with a fast
clock stamps every write "later". Every register therefore orders by a
Lamport `generation` first and uses `hlc_ms` only between writes that did not
observe each other.
### Subscription
* Register: `subscribed` + `podcast` metadata, last-writer-wins by
`(generation, hlc_ms, op_id)`.
* `generation` is a per-podcast Lamport counter shared with the share
register: every subscribe, unsubscribe and share write uses
`next_podcast_generation(subscription, share)` = one more than the greatest
generation of either register (1 when there is none). An action taken after
observing another device's write orders after it whatever the clocks say.
* Unsubscribing writes `subscribed = false`. It is state, not a tombstone,
and is never compacted, so it beats a stale subscription from a device that
returns after being offline.
* Unsubscribing stops monitoring but keeps progress and history.
Subscribing again continues where the user stopped. Erasing history is a
separate explicit action (not part of this version).
* Only explicit subscribe and unsubscribe actions write the register. RSS
refreshes keep their metadata locally; a background refresh that wrote the
register could resurrect an unsubscribed podcast. Aliases learned by a
refresh use `podcast_feed_aliases_added`.
* Subscribing to a podcast already subscribed locally must not emit a new
operation.
| Events | Result |
| --- | --- |
| A (clock 10 min ahead) subscribes at g1; B sees it and unsubscribes at g2 | unsubscribed on every device |
| A and B, both offline after g1, unsubscribe and subscribe at g2 | the later `hlc_ms` (then `op_id`) wins |
### Share
* `ShareRecord`, last-writer-wins by `(generation, hlc_ms, op_id)`, with the
generation from `next_podcast_generation`.
* `shared = false` revokes the publication. Like an unsubscribe it is
privacy-protecting state, **not** a tombstone: it is never compacted and
its row is never deleted. Otherwise a stale `shared = true` still in some
device's log, or delivered again to a new device, would republish the
podcast. The register is one record per podcast, so keeping it is cheap.
* Unsubscribing also writes `shared = false`.
* `effective_share(subscription, share)` decides whether to publish: the user
is subscribed, chose to share, **and** the share decision orders after the
latest subscription write by `(generation, hlc_ms, op_id)`. Every new
subscription, including subscribing again or subscribing from someone
else's public card, is private until the user shares it again; a stale
share decision cannot republish; a share decision taken after observing the
subscription always applies.
### Progress
`ProgressRecord { episode_id, podcast_id, position_ms, status, generation,
session_id, hlc_ms, device_id, op_id }`, with `status` one of `not_started`,
`in_progress`, `completed`. The winner is the greatest
`(generation, hlc_ms, device_id, op_id)`. Progress is never the maximum
position: seeking back and "start over" are real changes.
* Explicit actions use `generation = max_seen + 1`
(`ProgressRecord::explicit_generation`): starting a listening session,
seeking, "start over", "mark played", "mark unplayed", taking over playback
through a handoff. `max_seen` is the generation of the current register,
including received records. An explicit action during a session becomes
the generation of that session: its later checkpoints use it.
* Periodic checkpoints and pause/stop saves within a session keep the
generation of that session's latest explicit action.
* Handoff: queue items carry `PodcastEpisodeRef.progress_generation`, the
greatest generation the sender has seen for the episode, because the
playback state can arrive before the sender's progress operations. The
sender writes a final checkpoint at its session generation and does not
bump. Only the receiver writes the explicit action, with
`explicit_generation(max(local, progress_generation))`, so its writes order
after the sender's whatever the clocks say.
* A device that observes a greater generation for the episode it is playing
stops writing at its old generation: either playback moved away (the
coordination claim says so), or the user keeps listening locally and the
device writes its next record as an explicit action
(`explicit_generation(max_seen)`).
* Write progress periodically (every 15–30 s) and on pause, seek, stop,
episode change and handoff, in the background. The outgoing log may drop a
pending progress operation only when another pending operation of the same
episode `supersedes` it (greater `(generation, hlc_ms, device_id, op_id)`).
Never coalesce by sequence order: a checkpoint of an older session queued
after "mark played" must not replace it.
| Events | Result |
| --- | --- |
| A checkpoints session s1 at g3; B seeks back at g4 with a slower clock; A's late g3 checkpoint arrives | B's seek (g4) wins |
| A restarts (g3, position 0); a queued g2 checkpoint of the old session arrives later | restart wins |
| A and B both saw g5 and listen offline; both start sessions at g6 | the later `hlc_ms` (then `device_id`, `op_id`) wins on every device; both sessions are kept |
| A plays at g5 with a fast clock and hands off to B, which has only seen g2 | B starts at g6 (`max(2, 5) + 1`) and its checkpoints win |
| A marks played (g5), then a queued g3 checkpoint of its session is written | the outgoing log keeps g5; g3 is dominated |
Ordinary Play resumes from `resume_position_ms()` (the position when in
progress, otherwise 0). "Start over" is a separate explicit action. Publisher
metadata changes never touch progress; if the duration or audio changed, the
client keeps the stored position and lets the user decide.
### Listening sessions and counts
`ListenSession { session_id, episode_id, podcast_id, device_id,
started_at_ms, ended_at_ms, start_position_ms, end_position_ms, listened_ms,
completed }` forms a grow-only set keyed by `session_id` (generated at
playback start). A device may resend its session with more listening time;
versions merge to the greatest `(completed, listened_ms, ended_at_ms, ...)`.
* A session is recorded in history when `completed` or
`listened_ms >= 30 000` (`should_record`).
* `completed` is set when the session reaches the end of the stream or
`reaches_completion(position_ms, duration_ms)`: within the smaller of 30 s
and 5 % of the duration from the end. Completing also writes progress with
status `completed` (explicit generation).
* The play count of an episode is the number of distinct completed sessions
(`completed_listen_count`). Each session counts at most once, so
re-delivery never double counts, and listening again is a new session.
* "Mark played" only writes progress; it does not create a session or invent
listening time.
* Concurrent offline sessions are all kept; there is no promise of a single
global session during a partition.
## Trusted-device sync (protocol v4)
The ALPN remains `furumi/sync/2`; device profiles advertise
`protocol_version = 4`. New operation kinds (none of them is a tombstone):
| Kind | Payload | Materialization |
| --- | --- | --- |
| `podcast_subscription_set` | `podcast_id`, `subscribed`, `podcast`, `feed_aliases`, `generation` | `SyncOpWire::podcast_subscription` |
| `podcast_share_set` | `podcast_id`, `shared`, `generation` | `SyncOpWire::podcast_share` |
| `podcast_feed_aliases_added` | `podcast_id`, `feed_aliases` (1..9 URLs) | `SyncOpWire::podcast_feed_aliases` |
| `podcast_progress_set` | `episode_id`, `podcast_id`, `position_ms`, `status`, `generation`, `session_id`, optional `episode` ref | `SyncOpWire::podcast_progress` |
| `podcast_listen_recorded` | `session`, optional `episode` ref | `SyncOpWire::podcast_listen_session` |
Register stamps come from the operation: `hlc_ms`, `op_id`, and for progress
`device_id = origin_device_id`. A session's `device_id` must equal the
operation origin. `SyncOpWire::validate_podcast` checks this and the bounds;
receivers skip operations that fail it but still acknowledge them, so the
log keeps moving.
Ids (`PodcastId`, `EpisodeId`) and feed URLs (`FeedUrl`) are validated while
the message is deserialized: a malformed id or a non-canonical feed URL is a
schema violation, like a wrong JSON type, and the **whole message** fails to
parse. Conforming clients cannot produce one, because these types serialize
only canonical values, and their rules are frozen with the `v1` derivation
domains (changing them needs a new protocol version, see Versioning).
### Peer versions
Each device reports its own version: in the `profile` of its `Hello`,
`PairRequest` and `PairResponse`, in its own entry of the `devices` list (a
`SyncResponse` has no `profile`, so an initiator learns the responder's
version from the entry whose `device_id` is the responder), and in
`DeviceProfileSet.protocol_version` of operations it originates. A client
emits a `DeviceProfileSet` with `protocol_version` on the first start of a
new protocol version.
* A self-report replaces the stored version:
`profile_protocol_version(reported, stored)` (`None` keeps the stored
value).
* Profiles of other devices relayed in `devices` only raise it:
`relayed_protocol_version(relayed, stored)` = `max(stored, relayed)`, or
`relayed` when nothing is stored. Deployed v1–v3 clients relay every
device with their own constant, so a relayed 3 must not downgrade a v4
peer.
* Relays forward the stored version of other devices in
`DeviceProfileWire.protocol_version` and must never substitute their own
constant: version gating would then send podcast state to an old client.
* A device that never reported a version is treated as version 1.
Downgrading a device below v4 is not supported while relays still hold
v4 for it.
### Pairing
The invite consumer sends its `PairRequest` (operations, snapshot, playback)
before it hears from the inviter. Invites created by v4 clients carry the
inviter's version in `InviteWire.protocol_version` (JSON key `p`, omitted by
older clients). The consumer gates the whole request by
`InviteWire::inviter_protocol_version()`, which is version 1 for invites
without it: no podcast operations, `snapshot.for_peer(version)`, and no
playback snapshot with podcast items. Podcast state then catches up through
the snapshot of the next regular sync. The inviter gates its `PairResponse`
by the version in the request's `profile`.
### Snapshot recovery
Sync vectors only grow, so operations hidden from a peer while it ran an
older version are never re-requested after it upgrades. Podcast state is
therefore part of `SyncSnapshot`:
| Section | Content | Limit | Budget share |
| --- | --- | --- | --- |
| `podcast_subscriptions` | subscription registers, including unsubscribed, with feed aliases; `podcast.description` is omitted | 1000 | 25 % |
| `podcast_shares` | share registers, including `shared = false` | 1000 | 5 % |
| `podcast_progress` | progress registers | 5000 | 45 % |
| `podcast_listen_sessions` | most recent sessions | 2000 | 25 % |
`SyncSnapshot::bound_podcast_state_to(budget)` orders each section newest
first (registers by `(generation, hlc_ms)`, sessions by `ended_at_ms`), caps
it by count and by its share of the JSON byte budget; unused budget passes on
to the following sections in table order, so large subscription records
cannot crowd out progress. `bound_podcast_state()` uses the default 4 MiB
budget. Senders whose other message parts are large pass
`podcast_snapshot_budget(other_bytes)`, which keeps the whole line under the
8 MiB limit. Channel descriptions are not snapshotted: every device refreshes
channel metadata from the feed.
Receivers ignore podcast sections that exceed the counts
(`podcast_state_within_limits`), validate each record and merge it with the
same functions as operations.
Limitations: records cut by the bounds reach a peer only through operations
it has not acknowledged yet. A peer that upgrades from v3, or that received
operations through a v3 relay, already skipped the podcast operations: it
recovers only the newest records that fit, so older progress registers and
listening sessions are missing on that device and its play counts
(`completed_listen_count`) can differ from other devices.
Empty sections are not serialized, so snapshots without podcast state are
byte-identical to v3.
## Queue and playback
`PlaybackTrack.source: Option<PlaybackSource>` makes the item source
explicit: absent or `music` for music, `podcast_episode` with a
`PodcastEpisodeRef`, or `private` for a Jam placeholder. Negative ids,
placeholder releases and URLs in `file_path` are not part of the contract.
`PlaybackTrack::podcast_episode` fills the legacy display fields (episode
title, podcast title as artist and release, `id = 0`, empty `file_path`, no
`content_id`); receivers read `source`.
`PodcastEpisodeRef` is self-contained: podcast and episode ids, canonical
feed URL, titles, audio URL, MIME, duration, publication time, season,
episode number, artwork. A device without the subscription can display,
resume and play the item; if the audio URL is stale it refreshes the feed and
looks the episode up by id. Mixed queues keep order, position and identity of
every item.
Handoff follows `PLAYBACK_PROTOCOL.md`: record position and play/pause
intent, transfer the queue with its sources, the receiver opens the HTTP
source and seeks, and ownership moves only through the coordination claim.
`PlaybackStateWire.playback_rate: Option<f32>` is reserved. No client
implements variable speed yet; receivers treat a missing or unsupported value
as 1.0. When implemented, position extrapolation and handoff must use it.
## Version gating for peers below v4
`SyncOpPayload` is a closed serde enum. A v3 client that receives an unknown
operation `kind` fails to parse the **whole** message, not one operation,
which would stall synchronization. A v3 client does not know
`PlaybackTrack.source` and ignores it: it would treat an episode as a broken
music item (id 0, no content id) and forward the queue that way, losing the
episode for every device. Additive `serde(default)` fields alone therefore do
not make podcasts compatible. The peer version comes from "Peer versions"
above, and from the invite while pairing. Therefore:
1. Never send podcast operations to a peer below v4
(`op_allowed_for_peer`).
2. Never send podcast snapshot sections (`SyncSnapshot::for_peer`) or a
playback snapshot whose queue contains podcast items
(`playback_allowed_for_peer`) to a peer below v4.
3. Refuse a handoff to a device below v4 while the queue contains podcast
items and tell the user that the target must be updated. Never strip the
episodes to make the queue fit.
4. Reject `SetState`/`ActiveChanged` from a peer below v4 while the local
queue contains podcast items (`playback_command_allowed_from_peer`): its
state would silently drop them.
5. Music operations and music-only playback keep working with v1–v3 peers.
`PlaybackSource` is closed for v4 clients as well: a v4 client fails to parse
a message with an unknown source `kind`.
## Jam privacy
Jam participants are outside the trusted-device group.
* Hosts send `jam::redact_jam_snapshot` (or `redact_for_jam` on the state):
every podcast item becomes a `private` placeholder titled "Private item"
without URL, feed, ids, titles or duration; music items, order and position
are kept.
* Commands containing real podcast episodes are rejected
(`jam_command_allowed`): participants cannot add episodes.
* Before applying a participant's state, hosts call
`restore_jam_private_items`: the k-th placeholder maps to the host's k-th
private item. States that drop, duplicate or inject private items are
rejected. Participants may still pause, seek and reorder music.
## Feed parser
With the `podcast-feed` feature, `music_dht::podcast::feed` parses RSS 2.0,
RSS 1.0 and Atom tolerantly: BOM and declared single-byte encodings, HTML
entities, CDATA, mismatched end tags, undeclared prefixes, depth and size
limits, lenient dates and durations, relative media URLs resolved against
the fetch URL. Items without an absolute `http(s)` audio URL are dropped. It
extracts `itunes:season`, `itunes:episode`, `itunes:episodeType`,
`guid isPermaLink` and `itunes:new-feed-url`, and produces identity hints for
`resolve_episode_ids`; those hints are normative (see the no-GUID rule).
Links are kept verbatim (trimmed only) because they are identity material.
Text cleanup is linear in the input size. It performs no I/O.
## SQLite reference store
`furumi_library::podcasts::PodcastStore` (`furumi-library` 0.3) implements
this contract for SQLite clients. Other stores (the web client's PostgreSQL)
must keep the same invariants.
### File and migrations
* Own file, conventionally `podcasts.sqlite3` next to `library.db`; the caller
passes the path. WAL, foreign keys, 5 s busy timeout.
* `PRAGMA user_version` drives migrations. Each runs in a `BEGIN IMMEDIATE`
transaction that re-reads the version, so concurrent openers and
interrupted upgrades are safe and reopening is idempotent.
| Version | Content |
| --- | --- |
| 0 | empty file |
| 1 | prototype schema of the original podcast PR (`podcasts`, `podcast_episodes`), verbatim |
| 2 | the model below; the prototype tables are renamed and imported |
* Before writing anything (including the WAL switch), the store refuses a
file with a `user_version` above its own, and a file whose tables are not
the expected ones for its version (for example `library.db` passed by
mistake). The file is left byte-for-byte unchanged.
* Prototype import (migration 2). Tables are renamed `legacy_podcasts` and
`legacy_podcast_episodes`, never dropped.
* Each feed becomes a podcast with `podcast_id` from the canonical feed URL
(`FeedUrl::parse`, then `FeedUrl::from_user_input`) and a subscription
register `subscribed = true`, generation 0, `hlc_ms` = `added_at`,
`op_id` = `legacy-import:<podcast_id>`, not shared. Every contract write
orders after it; other devices learn it through the snapshot sections.
Rows with the same canonical URL merge (first row wins).
* URLs that cannot be canonicalized, episodes with a non-`http(s)` audio URL,
invalid metadata or no identity are listed in `legacy_quarantine`; the
rows stay in the `legacy_*` tables.
* Episode ids come from `resolve_episode_ids` over the stored items. The
prototype stored the enclosure URL as `guid` when an item had none: such
rows are imported as GUID-less items and keep the value in
`episodes.legacy_guid`. The prototype kept one row per enclosure URL, so
rows with the same publication second, normalized title and `<link>`
(or none) are one item whose audio URL moved: the newest row
(`first_seen_at`, then `id`) is imported, the others are quarantined as
duplicates.
* Imported GUID-less episodes are **provisional**: stale rows can still
change which key is unique among the stored items, so their ids may
differ from what a device without the prototype derives. On a refresh,
item ids are resolved against the stored episodes except provisional
ones without progress or sessions; each such provisional episode is
matched to its item (enclosure, then link, then date and title, GUIDs
ignored) and re-keyed to that item's id. When another episode that
stays already holds that id (for example one a remote operation created
before this device's first refresh), it is the item's episode and the
provisional one is a duplicate: it is deleted rather than kept as a
second copy. The same applies when `legacy_guid` turns out to be the
item's real GUID. Listing an episode clears `legacy_guid`, which makes
its id final. Provisional episodes carry no personal state, so nothing
personal moves or is lost; an imported episode played before its first
refresh is not provisional and keeps its id (the feed item then becomes
a separate episode if the ids differ).
* ETag, Last-Modified, check time, last error and the cover cache move to the
fetch state. The prototype stores no progress or history, so none is
created.
### Tables (version 2)
| Table | Key | Content | Synced |
| --- | --- | --- | --- |
| `podcasts` | `podcast_id` | local channel metadata, `feed_url` = URL this device fetches, `meta_from_feed` | catalog, no |
| `podcast_feed_aliases` | `(podcast_id, feed_url)` | bounded grow-only alias set (`merge_feed_aliases`) | yes, inside subscription records and `podcast_feed_aliases_added` |
| `episodes` | `episode_id` | `EpisodeMeta` columns (season, number, type, published, duration, audio URL, MIME, length, artwork, link), `source` (`feed`/`sync`), `in_feed`, `legacy_guid` | catalog, no |
| `podcast_seasons` | `(podcast_id, season)` | derived from `episodes`: episode count, first/last publication; `season` NULL groups episodes without a season | no |
| `podcast_subscriptions` | `podcast_id` | subscription register: `subscribed`, `PodcastMeta` JSON, `generation`, `hlc_ms`, `op_id` | yes |
| `podcast_shares` | `podcast_id` | share register | yes |
| `episode_progress` | `episode_id` | progress register (`ProgressRecord`) | yes |
| `podcast_listen_sessions` | `session_id` | grow-only set of `ListenSession` | yes |
| `podcast_fetch_state` | `podcast_id` | ETag, Last-Modified, checked / last success / next check times, last error, consecutive failures, cover cache | **never** |
| `legacy_quarantine` | `(legacy_table, legacy_id)` | prototype rows that could not be imported | no |
### Invariants
* Registers and sessions have no foreign keys to the catalog: operations for
podcasts or episodes the device has not fetched are stored as they arrive,
in any order. An operation's `PodcastEpisodeRef` adds a `source = 'sync'`
catalog row only when the episode is unknown; feed metadata always wins.
* One apply path. Local actions validate the payload, draw a stamp from the
client's sync log, build the `SyncOpWire`, and apply it like a received
operation: `SyncOpWire::podcast_*` materializers, then `merge_*`. The
returned operation must be appended with exactly that stamp. Snapshot
records go through the same functions after validation. The result is
therefore commutative, associative and idempotent, and a snapshot never
replaces newer local state.
* Invalid operations (`validate_podcast`, a generation above `i64::MAX`, a
session whose identity differs from the stored one) are skipped and
reported as rejected, so the client can still acknowledge them.
* Only `subscribe`/`unsubscribe` write the subscription register and only
`set_shared`/`unsubscribe` the share register, with
`next_podcast_generation`. No-op actions draw no stamp and emit nothing.
`unsubscribe` emits `subscribed = false` and `shared = false` and keeps
progress, sessions and the catalog.
* Explicit progress actions use `explicit_generation(max(local, carried))`;
checkpoints keep the session generation and are refused (`Superseded`)
once the register holds a greater one, or after the session completed.
* RSS refreshes update the catalog (channel metadata, episode metadata and
audio URLs, `in_feed`) and the alias set only; episodes missing from the
feed are kept. Permanent redirects and `itunes:new-feed-url` become the
fetch URL and aliases, and emit `podcast_feed_aliases_added` only when the
stored alias set changes.
* Snapshot export reads each section newest first up to its count, then
applies `bound_podcast_state_to(budget)`. Fetch state and the catalog
(descriptions, episodes) are never exported.
* Feed URLs resolve through `resolve_podcast_id` over the alias set, the
catalog fetch URL and the register's `podcast.feed_url`.
* Fetch-state writes (`record_fetch_*`, `set_cover_cache`) read and write
the row in one immediate transaction, so concurrent writers on one or
several connections never lose a column or a failure increment.
* `TrackItem::podcast_episode` builds queue items whose identity is the
explicit source `TrackItem.podcast` alone. Their `id`, `release_id` and
artist id are the fixed sentinel `TrackItem::NOT_A_LIBRARY_ROW` (-1),
shared by every episode, the same "not a library row" convention as
pending federated tracks. The sentinel carries no identity and is never
decoded; it only keeps music-only code that predates podcasts
(`id >= 0`, `release_id < 0` checks) from taking an episode for library
track, release or artist 0. There is no reserved range of negative ids.
* `TrackItem::key` (a typed `TrackItemKey`; string form `lib:<id>`,
`fed:<owner>:<item_id>`, `ep:<episode id>`) is the source identity of a
queue item; on the wire it is `PlaybackSource`. An episode is identified
by `TrackItem.podcast` / `ep:` only and must never be keyed by its numeric
id: keying by id (for example a `local:{id}` string) would merge all
episodes into one. `TrackItemKey` has no content-id equivalence (a pending
federated item and its imported library copy differ), so clients that
match music content-first keep doing that and add the podcast branch
before any fallback to the id.
* Wire ids of podcast items and of Jam private placeholders are `0`
(`PlaybackTrack::podcast_episode`, `PlaybackTrack::private_placeholder`)
and must not be copied into a `TrackItem`.
`TrackItem::from_playback_track` classifies a received item as
`PlaybackItem::Music` (the client's music path), `PodcastEpisode` (built
with `TrackItem::podcast_episode`, sentinel ids) or `Private` (not
playable, no `TrackItem`). Besides the ids, `TrackItem::podcast_episode`
also differs from the wire form in `release_year` (publication year) and
`audio_format` (`None`; the MIME type stays in the reference).
## Versioning
* `music-dht` 0.6.0 and `furumi-library` 0.3.0: new public types, enum
variants and struct fields are breaking for 0.x.
* Device-sync protocol 4, ALPN unchanged. Podcast payloads are JSON; no
postcard layout changes.
* Id domains `furumi-podcast:podcast:v1` and `furumi-podcast:episode:v1` and
the `pod1:`/`ep1:` prefixes are frozen. A different derivation needs new
domains, new prefixes, a migration and a new protocol version.
* Adding an operation kind or a `PlaybackSource` variant requires a new
protocol version and gating as above.
* Feed URL canonicalization, the identity bounds (`MAX_URL_BYTES`,
`MAX_GUID_BYTES`) and the no-GUID rule are frozen with the `v1` domains:
receivers reject non-canonical ids and URLs while parsing.
* Public podcast cards will use a separate signed overlay with its own ALPN
and version; the music DHT schema and `federation-net` do not change.
+1 -1
View File
@@ -13,7 +13,7 @@ The workspace has three layers:
- [`music-dht`](crates/music-dht) builds a Kademlia-style distributed music
directory on top of that transport, including catalog discovery, content-id
lookup, signed similarity-LSH routing, direct byte streams, and shared Furumi
wire types.
wire types, including the shared podcast contract ([PODCASTS.md](PODCASTS.md)).
- [`furumi-library`](crates/furumi-library) owns the shared local `library.db`
contract, migrations, catalog operations, metadata import, artwork,
playlists, likes, history, and similarity persistence used by Furumi clients.
+5 -2
View File
@@ -1,6 +1,6 @@
[package]
name = "furumi-library"
version = "0.2.0"
version = "0.3.0"
description = "Client-independent SQLite music library for Furumi players"
readme = "README.md"
documentation = "https://docs.rs/furumi-library"
@@ -16,8 +16,11 @@ anyhow = { workspace = true }
blake3 = { workspace = true }
directories = { workspace = true }
lofty = { workspace = true }
music-dht = { workspace = true }
music-dht = { workspace = true, features = ["podcast-feed"] }
rusqlite = { workspace = true, features = ["functions"] }
serde = { workspace = true }
serde_json = { workspace = true }
tracing = { workspace = true }
[dev-dependencies]
tempfile = { workspace = true }
+62
View File
@@ -19,3 +19,65 @@ publication, clients can depend on the released crate version instead.
`Library::open(default_db_path()?)` opens the same platform-native Furumi
library used by every client. Connections use WAL, foreign keys, and a busy
timeout so independently running clients can safely share the database.
Version 0.3 follows `music-dht` 0.6, whose device-sync and podcast contract
types it re-exposes.
## Podcasts
`furumi_library::podcasts::PodcastStore` is the SQLite reference store of the
podcast contract (workspace `PODCASTS.md`, section "SQLite reference store").
It lives in its own file, conventionally `podcasts.sqlite3`
(`PODCAST_DB_FILE`) next to `library.db`; the caller passes the path.
- **Migrations** run on `PRAGMA user_version`, each in its own transaction.
A database of the original podcast prototype (`user_version` 1) is imported:
its tables are renamed `legacy_podcasts` / `legacy_podcast_episodes` (never
dropped), feeds become subscriptions with contract ids, rows that cannot be
imported are listed in `legacy_quarantine`. A database from a newer Furumi,
or a file that is not a podcast database, is refused without being touched.
- **Local actions** (`subscribe`, `unsubscribe`, `set_shared`,
`start_session`, `checkpoint`, `seek`, `restart`, `complete_session`,
`mark_played`, `mark_unplayed`, `record_listen_session`) draw stamps from a
`StampSource` (usually the client's sync log), apply the operation and
return `PodcastLocalOp`s to append to the log with exactly those stamps.
- **Sync**: `apply_remote_op` applies the five `podcast_*` operations,
`fill_snapshot` / `export_snapshot` produce the bounded snapshot sections
and `apply_snapshot` merges them. Everything goes through the `music-dht`
materializers and merge functions, so the result does not depend on
delivery order and never loses newer local state.
- **RSS**: `apply_feed` ingests a document parsed by
`music_dht::podcast::feed` (the crate enables the `podcast-feed` feature):
channel metadata, episodes with stable ids and current audio URLs, feed
aliases from permanent redirects and `itunes:new-feed-url`. It never
touches subscriptions, progress or history. HTTP, conditional requests and
scheduling stay in the client; `record_fetch_*`, `fetch_state` and
`podcasts_due_for_refresh` keep the local-only fetch state, which is never
synced.
- **Queries**: `list_podcasts`, `seasons`, `episodes` (season, played-state
and text filters, date or episode-number order), `resume_position_ms`,
`episode_ref` / `episode_track` for queueing, `listen_history`.
The store is synchronous: methods take `&self` and lock one connection, so
`Arc<PodcastStore>` can be shared; run calls on a blocking pool.
`TrackItem.podcast` marks a queue item as a podcast episode
(`TrackItem::podcast_episode`). Music-only code (likes, playlists, history,
content ids, federation downloads) must check `TrackItem::is_podcast`; the
music library never stores episodes.
`TrackItem::key` returns the typed source identity of a queue item
(`TrackItemKey`: `lib:<id>`, `fed:<owner>:<item_id>` or `ep:<episode id>` as
a string). A negative `TrackItem::id` only means "not a library row". Every
podcast item has the same sentinel `id`, `release_id` and artist id
(`TrackItem::NOT_A_LIBRARY_ROW` = -1). It carries no identity and must not
be decoded; it only keeps music-only code that predates podcasts from taking
an episode for library track, release or artist 0. The episode's identity is
`TrackItem.podcast` alone, so queue, current-track and position matching
must check it before any fallback to the id. `TrackItemKey` does not equate
a pending federated item with its library copy; clients that match music by
content id keep doing so.
Convert received `PlaybackTrack`s with `TrackItem::from_playback_track`
(`PlaybackItem::Music`, `PodcastEpisode` or `Private`): podcast items and
Jam private placeholders carry id 0 on the wire, which must never be copied.
+8 -2
View File
@@ -6,9 +6,13 @@
//!
//! All methods take `&self` and lock the connection internally, so an
//! `Arc<Library>` can be shared across blocking tasks.
//!
//! Podcasts live in a separate database, [`podcasts::PodcastStore`]
//! (`podcasts.sqlite3`); the music library never stores episodes.
pub mod import;
pub mod models;
pub mod podcasts;
use std::collections::{HashMap, HashSet};
use std::io::Write as _;
@@ -22,8 +26,8 @@ use rusqlite::{Connection, OptionalExtension as _, params};
pub use models::{
ArtistCard, ArtistDetail, ArtistRef, ArtistsPage, Availability, FederatedTrack, LibraryFilters,
LibrarySourceMode, PlaylistCard, PlaylistDetail, ReleaseCard, ReleaseDetail, ReleaseEdit,
SearchResults, TrackEdit, TrackItem,
LibrarySourceMode, PlaybackItem, PlaylistCard, PlaylistDetail, ReleaseCard, ReleaseDetail,
ReleaseEdit, SearchResults, TrackEdit, TrackItem, TrackItemKey,
};
/// The virtual "Liked tracks" playlist id, kept from the server API.
@@ -3283,6 +3287,7 @@ fn query_tracks(
artists: Vec::new(),
featured_artists: Vec::new(),
fed: None,
podcast: None,
})
})?
.collect::<rusqlite::Result<Vec<_>>>()?;
@@ -3348,6 +3353,7 @@ fn pending_track(fed: &FederatedTrack) -> TrackItem {
file_size_bytes: None,
play_count: 0,
fed: Some(fed.clone()),
podcast: None,
}
}
+465
View File
@@ -1,6 +1,8 @@
//! Data shapes the views render. They mirror what the furumusic API used to
//! return, but every field is now filled from the local SQLite library.
use music_dht::device_sync::{PlaybackSource, PlaybackTrack};
use music_dht::podcast::{EpisodeId, PodcastEpisodeRef};
use serde::{Deserialize, Serialize};
/// Which catalog sources a client wants library queries to include.
@@ -120,6 +122,56 @@ pub struct ArtistRef {
pub name: String,
}
/// A playable queue item: a library track, a pending federated track or a
/// podcast episode.
///
/// # Identity
///
/// What an item *is* comes from its explicit source fields, never from the
/// numeric `id` alone; [`TrackItem::key`] returns it as a typed
/// [`TrackItemKey`]:
///
/// * a library track is a library row: `id >= 0`, `fed` and `podcast` are
/// `None`;
/// * a federated track is identified by [`TrackItem::fed`] (owner and item
/// id). While it is not in the library its `id` is a negative,
/// process-local counter value (pending federated tracks count down from
/// -1) and `release_id` / artist ids are -1;
/// * a podcast episode is identified **only** by [`TrackItem::podcast`] (its
/// [`EpisodeId`]). Its `id`, `release_id` and artist id are the fixed
/// sentinel [`TrackItem::NOT_A_LIBRARY_ROW`] (-1), shared by every episode.
///
/// A negative `id` therefore means "not a library row" and nothing else: it
/// carries no identity and must never be decoded, compared across items or
/// used as a key. The -1 sentinel only keeps music-only code that predates
/// podcasts (`id >= 0` / `release_id < 0` checks) from treating an episode
/// as library track, release or artist 0.
///
/// [`TrackItem::key`] is the *source* identity. It does not merge a pending
/// federated item with its imported library copy, which share a content id
/// but have different keys (`fed:` and `lib:`). Clients that key playback
/// items content-first (content id, then federated source, then id) keep
/// doing so for music; what they must add is a podcast branch
/// ([`TrackItem::podcast`], or [`TrackItemKey::PodcastEpisode`]) checked
/// before any fallback to `id`. An episode must never be keyed by `id`: a
/// `local:{id}` key would merge all episodes into one. On the wire the item
/// kind is `music_dht::device_sync::PlaybackSource`; convert received items
/// with [`TrackItem::from_playback_track`], never by copying wire ids.
///
/// # Podcast episodes
///
/// Podcast episodes carry their [`PodcastEpisodeRef`] in
/// [`TrackItem::podcast`]: `file_path` is empty, `content_id`, `fed` and
/// `cover_path` are `None`, and the display fields mirror the reference
/// (episode title, podcast title as artist and release). The audio URL is
/// only in the reference: it is never a local path, never content-addressed
/// and never offered for P2P transfer.
///
/// **Music-only code paths must check [`TrackItem::is_podcast`]** (or
/// `podcast.is_none()`) before treating an item as music: likes, playlists,
/// listening history, content-id lookups, federation downloads, similarity
/// and file operations do not apply to episodes. Library queries never
/// return podcast items; the library database stores none.
#[derive(Debug, Clone)]
pub struct TrackItem {
pub id: i64,
@@ -149,9 +201,187 @@ pub struct TrackItem {
/// carries everything needed to download them from the owning peer.
/// With an empty `file_path` the player resolves the track on demand.
pub fed: Option<FederatedTrack>,
/// Set for podcast episodes (see the type documentation); `None` for music.
pub podcast: Option<PodcastEpisodeRef>,
}
/// Typed source identity of a [`TrackItem`], from [`TrackItem::key`].
///
/// Two items with equal keys are the same track or episode; the same track
/// may still appear more than once in a queue. The key has no content-id
/// equivalence: a pending federated item and its library copy differ. Use it
/// where the source matters (and always for podcast episodes, which have no
/// other identity); code that matches music by content id keeps doing that
/// and checks the podcast variant first (see [`TrackItem`]).
///
/// The [`Display`](std::fmt::Display) form is stable and suitable as a map
/// key or in logs:
///
/// | Variant | String form |
/// | --- | --- |
/// | `Library(123)` | `lib:123` |
/// | `Federated { owner, item_id }` | `fed:<owner>:<item_id>` |
/// | `PodcastEpisode(id)` | `ep:<episode id>`, e.g. `ep:ep1:...` |
/// | `Transient(-5)` | `tmp:-5` |
///
/// `owner` is a peer endpoint id and contains no `:`, so the federated form
/// is unambiguous. Keys are not persisted by this crate; a `Transient` key is
/// only meaningful inside the process that built the item.
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub enum TrackItemKey {
/// A row of the local music library (`TrackItem::id >= 0`).
Library(i64),
/// A federated track that is not a library row, identified by its owner
/// peer and DHT item id ([`FederatedTrack::owner`] /
/// [`FederatedTrack::item_id`]), not by its process-local negative `id`.
Federated { owner: String, item_id: String },
/// A podcast episode, identified by its [`EpisodeId`] only.
PodcastEpisode(EpisodeId),
/// A non-library item with no explicit source (negative `id`, no `fed`,
/// no `podcast`). furumi-library never builds one; clients that do must
/// keep its `id` unique within the process.
Transient(i64),
}
/// What a received `music_dht::device_sync::PlaybackTrack` is locally, from
/// [`TrackItem::from_playback_track`].
#[derive(Debug, Clone)]
pub enum PlaybackItem {
/// A music track. Convert it with the client's music path (content id,
/// then federated source); its wire ids are the sender's.
Music,
/// A podcast episode, built with [`TrackItem::podcast_episode`].
PodcastEpisode(Box<TrackItem>),
/// A Jam private placeholder: not playable and without identity. Show it
/// as `music_dht::device_sync::PRIVATE_ITEM_TITLE` where the queue is
/// displayed; never insert it as a library or queue track, never key it.
Private,
}
impl std::fmt::Display for TrackItemKey {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
Self::Library(id) => write!(f, "lib:{id}"),
Self::Federated { owner, item_id } => write!(f, "fed:{owner}:{item_id}"),
Self::PodcastEpisode(episode_id) => write!(f, "ep:{}", episode_id.as_str()),
Self::Transient(id) => write!(f, "tmp:{id}"),
}
}
}
impl TrackItem {
/// The `id`, `release_id` and artist id of items that are not library
/// rows and have no numeric identity: every podcast episode uses it.
///
/// It is a sentinel, not an identifier: it is shared by all episodes
/// (and can coincide with the first pending federated track's id), so it
/// must never be decoded or used as a key. See [`TrackItem::key`].
pub const NOT_A_LIBRARY_ROW: i64 = -1;
/// The typed identity of this item; see [`TrackItemKey`].
///
/// Precedence: [`TrackItem::podcast`] makes it a podcast episode; else a
/// non-negative `id` is a library row; else [`TrackItem::fed`] makes it
/// a federated track; anything else is [`TrackItemKey::Transient`].
pub fn key(&self) -> TrackItemKey {
if let Some(episode) = &self.podcast {
return TrackItemKey::PodcastEpisode(episode.episode_id.clone());
}
if self.id >= 0 {
return TrackItemKey::Library(self.id);
}
match &self.fed {
Some(fed) => TrackItemKey::Federated {
owner: fed.owner.clone(),
item_id: fed.item_id.clone(),
},
None => TrackItemKey::Transient(self.id),
}
}
/// Queue item for a podcast episode streamed from the publisher.
///
/// Title, artist and release follow `PlaybackTrack::podcast_episode` of
/// `music_dht::device_sync` (the episode title, the podcast title as
/// artist and release, the announced duration; no track or disc number).
/// The other fields differ from that wire form on purpose:
///
/// * `id`, `release_id` and the artist id are the sentinel
/// [`TrackItem::NOT_A_LIBRARY_ROW`] (-1) for every episode, where the
/// wire form carries `0`; the item's identity is [`TrackItem::podcast`]
/// alone ([`TrackItem::key`]);
/// * `release_year` is the publication year (`None` on the wire);
/// * `audio_format` is `None` (the wire form carries the MIME type, which
/// stays in the reference).
///
/// Season and episode number stay in the reference.
pub fn podcast_episode(episode: PodcastEpisodeRef) -> Self {
Self {
id: Self::NOT_A_LIBRARY_ROW,
title: episode.episode_title.clone(),
track_number: None,
disc_number: None,
duration_seconds: episode
.duration_ms
.map_or(0.0, |ms| ms.max(0) as f64 / 1_000.0),
artists: vec![ArtistRef {
id: Self::NOT_A_LIBRARY_ROW,
name: episode.podcast_title.clone(),
}],
featured_artists: Vec::new(),
release_id: Self::NOT_A_LIBRARY_ROW,
release_title: episode.podcast_title.clone(),
release_year: episode.published_at_ms.map(year_of_unix_ms),
file_path: String::new(),
content_id: None,
cover_path: None,
audio_format: None,
audio_bitrate: None,
audio_sample_rate: None,
audio_bit_depth: None,
file_size_bytes: None,
play_count: 0,
fed: None,
podcast: Some(episode),
}
}
/// Classifies a received `music_dht::device_sync::PlaybackTrack` and
/// builds the local queue item for non-music sources.
///
/// The wire `id` and `release_id` of podcast episodes and of Jam private
/// placeholders are `0`, and those of music items are the sender's local
/// ids. None of them identifies anything on this device, so they are
/// never copied:
///
/// * a podcast episode becomes [`TrackItem::podcast_episode`] of its
/// [`PlaybackSource::PodcastEpisode`] reference (sentinel ids, key
/// `ep:`), whatever the legacy display fields say;
/// * a private placeholder ([`PlaybackSource::Private`]) is not playable
/// and has no identity, so no `TrackItem` exists for it;
/// * music (no source, or [`PlaybackSource::Music`]) is left to the
/// client's music path, which resolves it by content id or federated
/// source.
pub fn from_playback_track(track: &PlaybackTrack) -> PlaybackItem {
match &track.source {
None | Some(PlaybackSource::Music) => PlaybackItem::Music,
Some(PlaybackSource::PodcastEpisode(episode)) => PlaybackItem::PodcastEpisode(
Box::new(Self::podcast_episode(episode.as_ref().clone())),
),
Some(PlaybackSource::Private) => PlaybackItem::Private,
}
}
/// Whether this item is a podcast episode (not music).
pub fn is_podcast(&self) -> bool {
self.podcast.is_some()
}
/// The episode reference of a podcast item.
pub fn podcast_episode_ref(&self) -> Option<&PodcastEpisodeRef> {
self.podcast.as_ref()
}
/// A federated track that still needs downloading before playback.
pub fn is_fed_pending(&self) -> bool {
self.fed.is_some() && self.file_path.is_empty()
@@ -202,6 +432,21 @@ impl TrackItem {
}
}
/// Proleptic Gregorian year of a unix timestamp in milliseconds (UTC).
fn year_of_unix_ms(ms: i64) -> i32 {
// Howard Hinnant's civil_from_days.
let days = ms.div_euclid(86_400_000);
let z = days + 719_468;
let era = z.div_euclid(146_097);
let doe = z - era * 146_097;
let yoe = (doe - doe / 1_460 + doe / 36_524 - doe / 146_096) / 365;
let doy = doe - (365 * yoe + yoe / 4 - yoe / 100);
let mp = (5 * doy + 2) / 153;
let month = if mp < 10 { mp + 3 } else { mp - 9 };
let year = yoe + era * 400 + i64::from(month <= 2);
i32::try_from(year).unwrap_or(if year < 0 { i32::MIN } else { i32::MAX })
}
#[derive(Debug, Clone)]
pub struct ReleaseCard {
pub id: i64,
@@ -340,8 +585,228 @@ mod tests {
file_size_bytes: None,
play_count: 0,
fed: None,
podcast: None,
};
assert_eq!(track.artist_line(), "Main feat. Guest, Other");
assert!(!track.is_podcast());
}
#[test]
fn year_of_unix_ms_handles_epoch_and_leap_years() {
assert_eq!(year_of_unix_ms(0), 1970);
assert_eq!(year_of_unix_ms(-1), 1969);
// 2024-02-29T12:00:00Z and 2024-12-31T23:59:59Z.
assert_eq!(year_of_unix_ms(1_709_208_000_000), 2024);
assert_eq!(year_of_unix_ms(1_735_689_599_000), 2024);
assert_eq!(year_of_unix_ms(1_735_689_600_000), 2025);
}
fn episode_ref(guid: &str) -> PodcastEpisodeRef {
use music_dht::podcast::{FeedUrl, PodcastId};
let feed_url = FeedUrl::parse("https://example.test/feed.xml").unwrap();
let podcast_id = PodcastId::from_feed_url(&feed_url);
PodcastEpisodeRef {
episode_id: EpisodeId::from_guid(&podcast_id, guid).unwrap(),
podcast_id,
feed_url,
podcast_title: "Show".into(),
episode_title: "Pilot".into(),
audio_url: "https://cdn.example.test/1.mp3".into(),
mime: Some("audio/mpeg".into()),
duration_ms: Some(90_500),
published_at_ms: Some(1_709_208_000_000),
season: Some(2),
episode_number: Some(7),
image_url: None,
progress_generation: None,
}
}
fn library_track(id: i64) -> TrackItem {
TrackItem {
id,
title: "Track".into(),
track_number: None,
disc_number: None,
duration_seconds: 1.0,
artists: vec![artist("Main")],
featured_artists: Vec::new(),
release_id: 1,
release_title: "Release".into(),
release_year: None,
file_path: "/tmp/track.mp3".into(),
content_id: None,
cover_path: None,
audio_format: None,
audio_bitrate: None,
audio_sample_rate: None,
audio_bit_depth: None,
file_size_bytes: None,
play_count: 0,
fed: None,
podcast: None,
}
}
fn federated(owner: &str, item_id: &str) -> FederatedTrack {
FederatedTrack {
item_id: item_id.into(),
owner: owner.into(),
own: false,
title: "Remote".into(),
artist_names: vec!["Someone".into()],
featured_artist_names: Vec::new(),
year: None,
duration_seconds: None,
content_id: None,
release_title: None,
track_number: None,
disc_number: None,
}
}
#[test]
fn podcast_track_mirrors_the_episode_reference() {
let episode = episode_ref("e1");
let track = TrackItem::podcast_episode(episode.clone());
assert!(track.is_podcast());
assert_eq!(track.podcast_episode_ref(), Some(&episode));
// The sentinel marks "not a library row, release or artist" and
// carries no identity.
assert_eq!(TrackItem::NOT_A_LIBRARY_ROW, -1);
assert_eq!(track.id, -1);
assert_eq!(track.release_id, -1);
assert!(track.artists.iter().all(|artist| artist.id == -1));
assert!(track.featured_artists.is_empty());
assert!(track.file_path.is_empty());
assert!(track.content_id.is_none() && track.fed.is_none());
assert!(!track.is_fed_pending());
assert_eq!(track.title, "Pilot");
assert_eq!(track.artist_line(), "Show");
assert_eq!(track.release_title, "Show");
assert_eq!(track.release_year, Some(2024));
assert_eq!(track.duration_label(), "1:31");
}
#[test]
fn key_is_typed_by_source() {
let library = library_track(123);
assert_eq!(library.key(), TrackItemKey::Library(123));
assert_eq!(library.key().to_string(), "lib:123");
assert_eq!(library_track(0).key(), TrackItemKey::Library(0));
let mut pending = library_track(-5);
pending.release_id = -1;
pending.file_path.clear();
pending.fed = Some(federated("peer1", "item-9"));
let fed_key = TrackItemKey::Federated {
owner: "peer1".into(),
item_id: "item-9".into(),
};
assert_eq!(pending.key(), fed_key);
assert_eq!(pending.key().to_string(), "fed:peer1:item-9");
// The process-local negative id is not part of a federated identity.
let mut renumbered = pending.clone();
renumbered.id = -6;
assert_eq!(renumbered.key(), pending.key());
let episode = episode_ref("e1");
let track = TrackItem::podcast_episode(episode.clone());
let ep_key = TrackItemKey::PodcastEpisode(episode.episode_id.clone());
assert_eq!(track.key(), ep_key);
assert_eq!(
track.key().to_string(),
format!("ep:{}", episode.episode_id.as_str())
);
assert!(track.key().to_string().starts_with("ep:ep1:"));
let transient = library_track(-7);
assert_eq!(transient.key(), TrackItemKey::Transient(-7));
assert_eq!(transient.key().to_string(), "tmp:-7");
// The source wins over the numeric id: a podcast item is never keyed
// as a library row, a federated item never by its counter value.
let mut odd = TrackItem::podcast_episode(episode);
odd.id = 42;
assert_eq!(odd.key(), ep_key);
let mut pending_minus_one = pending.clone();
pending_minus_one.id = TrackItem::NOT_A_LIBRARY_ROW;
assert_eq!(pending_minus_one.key(), fed_key);
assert_ne!(pending_minus_one.key(), track.key());
}
#[test]
fn received_playback_tracks_never_copy_wire_ids() {
// F1 wire form of an episode: id 0, release_id 0.
let episode = episode_ref("e1");
let wire = PlaybackTrack::podcast_episode(episode.clone());
assert_eq!((wire.id, wire.release_id), (0, 0));
let PlaybackItem::PodcastEpisode(item) = TrackItem::from_playback_track(&wire) else {
panic!("an episode is not music");
};
assert_eq!(item.id, TrackItem::NOT_A_LIBRARY_ROW);
assert_eq!(item.release_id, TrackItem::NOT_A_LIBRARY_ROW);
assert!(item.artists.iter().all(|artist| artist.id < 0));
assert_eq!(
item.key(),
TrackItemKey::PodcastEpisode(episode.episode_id.clone())
);
assert_eq!(item.podcast_episode_ref(), Some(&episode));
// Legacy display fields an older sender filled in do not matter.
let mut odd = wire.clone();
odd.id = 42;
odd.release_id = 7;
odd.title = "Something else".into();
let PlaybackItem::PodcastEpisode(odd) = TrackItem::from_playback_track(&odd) else {
panic!("the source decides");
};
assert_eq!(odd.key(), item.key());
assert_eq!(odd.id, TrackItem::NOT_A_LIBRARY_ROW);
// A private placeholder (also id 0) never becomes library row 0.
let private = PlaybackTrack::private_placeholder();
assert_eq!(private.id, 0);
assert!(matches!(
TrackItem::from_playback_track(&private),
PlaybackItem::Private
));
// Music is left to the client's music path.
let mut music = wire;
music.source = None;
assert!(matches!(
TrackItem::from_playback_track(&music),
PlaybackItem::Music
));
music.source = Some(PlaybackSource::Music);
assert!(matches!(
TrackItem::from_playback_track(&music),
PlaybackItem::Music
));
}
#[test]
fn episodes_share_the_sentinel_id_but_not_the_key() {
let first = TrackItem::podcast_episode(episode_ref("e1"));
let second = TrackItem::podcast_episode(episode_ref("e2"));
assert_eq!(first.id, TrackItem::NOT_A_LIBRARY_ROW);
assert_eq!(second.id, TrackItem::NOT_A_LIBRARY_ROW);
assert_eq!(first.id, second.id);
assert_eq!(first.release_id, second.release_id);
assert_ne!(first.key(), second.key());
assert_ne!(first.key().to_string(), second.key().to_string());
assert_eq!(
TrackItem::podcast_episode(episode_ref("e1")).key(),
first.key()
);
let keys: std::collections::HashSet<TrackItemKey> =
[first.key(), second.key(), first.key()]
.into_iter()
.collect();
assert_eq!(keys.len(), 2);
let mut ordered = [second.key(), first.key(), TrackItemKey::Library(1)];
ordered.sort();
assert_eq!(ordered[0], TrackItemKey::Library(1));
}
}
@@ -0,0 +1,544 @@
//! Local user actions: they build the device-sync operation, apply it through
//! the shared code path and return it for the client's sync log.
use anyhow::Result;
use music_dht::device_sync::SyncOpPayload;
use music_dht::podcast::{
EpisodeId, FeedUrl, ListenSession, PodcastEpisodeRef, PodcastId, PodcastMeta, ProgressRecord,
ProgressStatus, effective_share, merge_feed_aliases, merge_listen_session,
next_podcast_generation,
};
use rusqlite::{Connection, TransactionBehavior};
use super::apply::{self, ApplyOutcome};
use super::{OpStamp, PodcastLocalOp, PodcastStore, PodcastStoreError, StampSource};
/// Result of [`PodcastStore::subscribe`].
#[derive(Debug, Clone, PartialEq)]
pub struct SubscribeOutcome {
/// Podcast the URL resolved to (an existing one when the URL is a known
/// feed URL or alias).
pub podcast_id: PodcastId,
/// The podcast was already subscribed; no operation was emitted.
pub already_subscribed: bool,
/// Operations to append to the sync log.
pub ops: Vec<PodcastLocalOp>,
}
/// A listening session of this device as seen by the progress register.
///
/// Returned by explicit actions; periodic checkpoints of the session use its
/// `generation` ([`PodcastStore::checkpoint`]).
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ProgressSession {
/// Episode being played.
pub episode: PodcastEpisodeRef,
/// Session id (also the `ListenSession::session_id`).
pub session_id: String,
/// Generation of the session's latest explicit action.
pub generation: u64,
}
/// A progress register written by a local action.
#[derive(Debug, Clone, PartialEq)]
pub struct ProgressWrite {
/// The register as written.
pub record: ProgressRecord,
/// The session after the action (its generation for later checkpoints).
pub session: ProgressSession,
/// Operation to append to the sync log.
pub op: PodcastLocalOp,
}
/// Result of [`PodcastStore::checkpoint`].
#[derive(Debug, Clone, PartialEq)]
pub enum CheckpointOutcome {
/// The checkpoint was written.
Written(Box<ProgressWrite>),
/// The register holds a greater generation (another device or an
/// explicit action took over). Nothing was written: stop writing at the
/// old generation, or continue locally with an explicit action
/// ([`PodcastStore::start_session`] / [`PodcastStore::seek`]).
Superseded {
/// Current register.
current: ProgressRecord,
},
/// This session already completed the episode at its generation; a
/// checkpoint would turn it back into "in progress". Nothing was written.
AlreadyCompleted {
/// Current register.
current: ProgressRecord,
},
}
#[derive(Debug, Clone, Copy)]
enum ExplicitAction {
Start(i64),
Seek(i64),
Restart,
Complete(i64),
MarkPlayed,
MarkUnplayed,
}
fn invalid(reason: impl ToString) -> anyhow::Error {
PodcastStoreError::Invalid(reason.to_string()).into()
}
/// Validates the payload (with a provisional stamp), draws the real stamp,
/// and applies the operation through [`apply::apply_op`].
pub(crate) fn emit(
conn: &Connection,
stamps: &mut dyn StampSource,
make: impl Fn(&OpStamp) -> SyncOpPayload,
) -> Result<PodcastLocalOp> {
let device_id = stamps.device_id().to_string();
let provisional = OpStamp::new("provisional", device_id.clone(), 0);
provisional
.wire(make(&provisional))
.validate_podcast()
.map_err(invalid)?;
let stamp = stamps.next_stamp()?;
if stamp.origin_device_id != device_id {
return Err(invalid("stamp origin differs from the local device id"));
}
let payload = make(&stamp);
let op = stamp.wire(payload.clone());
match apply::apply_op(conn, &op)? {
ApplyOutcome::Rejected(reason) => Err(invalid(reason)),
_ => Ok(PodcastLocalOp { stamp, payload }),
}
}
/// Resolves a feed URL against every known URL of every podcast.
pub(crate) fn resolve_podcast(conn: &Connection, url: &FeedUrl) -> Result<PodcastId> {
let mut statement = conn.prepare_cached(
"SELECT podcast_id FROM podcast_feed_aliases WHERE feed_url = ?1
UNION SELECT podcast_id FROM podcasts WHERE feed_url = ?1
UNION SELECT podcast_id FROM podcast_subscriptions
WHERE json_extract(podcast_json, '$.feed_url') = ?1",
)?;
let ids = statement
.query_map([url.as_str()], |row| super::podcast_id_col(row.get(0)?))?
.collect::<rusqlite::Result<Vec<_>>>()?;
let pairs: Vec<(PodcastId, FeedUrl)> = ids.into_iter().map(|id| (id, url.clone())).collect();
Ok(music_dht::podcast::resolve_podcast_id(
url,
pairs.iter().map(|(id, url)| (id, url)),
))
}
fn catalog_meta(conn: &Connection, podcast_id: &PodcastId) -> Result<Option<PodcastMeta>> {
use rusqlite::OptionalExtension as _;
Ok(conn
.query_row(
"SELECT feed_url, title, author, description, link, image_url
FROM podcasts WHERE podcast_id = ?1",
[podcast_id.as_str()],
|row| {
Ok(PodcastMeta {
feed_url: super::feed_url_col(row.get(0)?)?,
title: row.get(1)?,
author: row.get(2)?,
description: row.get(3)?,
link: row.get(4)?,
image_url: row.get(5)?,
})
},
)
.optional()?)
}
fn share_payload(conn: &Connection, podcast_id: &PodcastId, shared: bool) -> Result<SyncOpPayload> {
let subscription = apply::load_subscription_register(conn, podcast_id)?;
let share = apply::load_share(conn, podcast_id)?;
Ok(SyncOpPayload::PodcastShareSet {
podcast_id: podcast_id.clone(),
shared,
generation: next_podcast_generation(subscription.as_ref(), share.as_ref()),
})
}
/// The reference carried by progress operations: identity and display only.
fn op_ref(episode: &PodcastEpisodeRef) -> PodcastEpisodeRef {
let mut episode = episode.clone();
episode.progress_generation = None;
episode
}
impl PodcastStore {
/// Subscribes to the podcast of `podcast.feed_url` (explicit user action).
///
/// The URL is resolved against known feed URLs and aliases first, so
/// subscribing again, through a redirect target or from someone else's
/// card reuses the existing podcast with its progress and history. When
/// the podcast is already subscribed nothing is emitted. Otherwise one
/// `podcast_subscription_set` with generation
/// `next_podcast_generation(..)` and the known aliases is applied and
/// returned. The new subscription is private until
/// [`PodcastStore::set_shared`].
///
/// Pass the channel metadata of the fetched feed
/// (`ParsedFeed::to_podcast_meta(url)`), then ingest the episodes with
/// [`PodcastStore::apply_feed`].
pub fn subscribe(
&self,
stamps: &mut dyn StampSource,
podcast: &PodcastMeta,
) -> Result<SubscribeOutcome> {
let meta = podcast.bounded();
let mut conn = self.lock();
let tx = conn.transaction_with_behavior(TransactionBehavior::Immediate)?;
let podcast_id = resolve_podcast(&tx, &meta.feed_url)?;
let current = apply::load_subscription_register(&tx, &podcast_id)?;
if current.as_ref().is_some_and(|record| record.subscribed) {
return Ok(SubscribeOutcome {
podcast_id,
already_subscribed: true,
ops: Vec::new(),
});
}
let share = apply::load_share(&tx, &podcast_id)?;
let generation = next_podcast_generation(current.as_ref(), share.as_ref());
let aliases = merge_feed_aliases(
&podcast_id,
&apply::load_aliases(&tx, &podcast_id)?,
std::slice::from_ref(&meta.feed_url),
);
let op = emit(&tx, stamps, |_| SyncOpPayload::PodcastSubscriptionSet {
podcast_id: podcast_id.clone(),
subscribed: true,
podcast: meta.clone(),
feed_aliases: aliases.clone(),
generation,
})?;
tx.commit()?;
Ok(SubscribeOutcome {
podcast_id,
already_subscribed: false,
ops: vec![op],
})
}
/// Unsubscribes (explicit user action).
///
/// Emits `podcast_subscription_set { subscribed: false }` followed by
/// `podcast_share_set { shared: false }`, each with the next podcast
/// generation. Progress, sessions and the catalog are kept, so
/// subscribing again continues where the user stopped. Does nothing when
/// the podcast is not subscribed.
pub fn unsubscribe(
&self,
stamps: &mut dyn StampSource,
podcast_id: &PodcastId,
) -> Result<Vec<PodcastLocalOp>> {
let mut conn = self.lock();
let tx = conn.transaction_with_behavior(TransactionBehavior::Immediate)?;
let Some(current) = apply::load_subscription_register(&tx, podcast_id)? else {
return Ok(Vec::new());
};
if !current.subscribed {
return Ok(Vec::new());
}
let share = apply::load_share(&tx, podcast_id)?;
let generation = next_podcast_generation(Some(&current), share.as_ref());
let mut meta = catalog_meta(&tx, podcast_id)?
.map(|meta| meta.bounded())
.unwrap_or_else(|| current.podcast.clone());
meta.feed_url = current.podcast.feed_url.clone();
let aliases = current.feed_aliases.clone();
let unsubscribe = emit(&tx, stamps, |_| SyncOpPayload::PodcastSubscriptionSet {
podcast_id: podcast_id.clone(),
subscribed: false,
podcast: meta.clone(),
feed_aliases: aliases.clone(),
generation,
})?;
let revoke_payload = share_payload(&tx, podcast_id, false)?;
let revoke = emit(&tx, stamps, |_| revoke_payload.clone())?;
tx.commit()?;
Ok(vec![unsubscribe, revoke])
}
/// Sets the decision to publish the podcast to the federation.
///
/// Sharing needs an active subscription and is a no-op when the podcast
/// is already effectively shared (`effective_share`). Revoking is a no-op
/// when no share decision says `true`. The write uses the next podcast
/// generation, so it orders after the subscription it observed.
pub fn set_shared(
&self,
stamps: &mut dyn StampSource,
podcast_id: &PodcastId,
shared: bool,
) -> Result<Vec<PodcastLocalOp>> {
let mut conn = self.lock();
let tx = conn.transaction_with_behavior(TransactionBehavior::Immediate)?;
let subscription = apply::load_subscription_register(&tx, podcast_id)?;
let share = apply::load_share(&tx, podcast_id)?;
if shared {
if !subscription
.as_ref()
.is_some_and(|record| record.subscribed)
{
return Err(PodcastStoreError::NotSubscribed(podcast_id.clone()).into());
}
if effective_share(subscription.as_ref(), share.as_ref()) {
return Ok(Vec::new());
}
} else if !share.as_ref().is_some_and(|record| record.shared) {
return Ok(Vec::new());
}
let payload = share_payload(&tx, podcast_id, shared)?;
let op = emit(&tx, stamps, |_| payload.clone())?;
tx.commit()?;
Ok(vec![op])
}
fn explicit_progress(
&self,
stamps: &mut dyn StampSource,
episode: &PodcastEpisodeRef,
session_id: Option<&str>,
carried_generation: Option<u64>,
action: ExplicitAction,
) -> Result<ProgressWrite> {
episode.validate().map_err(invalid)?;
let mut conn = self.lock();
let tx = conn.transaction_with_behavior(TransactionBehavior::Immediate)?;
let current = apply::load_progress(&tx, &episode.episode_id)?;
let max_seen = current
.as_ref()
.map(|record| record.generation)
.into_iter()
.chain(carried_generation)
.max();
let generation = ProgressRecord::explicit_generation(max_seen);
let current_position = current.as_ref().map_or(0, |record| record.position_ms);
let (position_ms, status) = match action {
ExplicitAction::Start(position) | ExplicitAction::Seek(position) => {
(position, ProgressStatus::InProgress)
}
ExplicitAction::Restart => (0, ProgressStatus::InProgress),
ExplicitAction::Complete(position) => (position, ProgressStatus::Completed),
ExplicitAction::MarkPlayed => (current_position, ProgressStatus::Completed),
ExplicitAction::MarkUnplayed => (0, ProgressStatus::NotStarted),
};
let reference = op_ref(episode);
let op = emit(&tx, stamps, |stamp| SyncOpPayload::PodcastProgressSet {
episode_id: episode.episode_id.clone(),
podcast_id: episode.podcast_id.clone(),
position_ms,
status,
generation,
session_id: session_id.unwrap_or(&stamp.op_id).to_string(),
episode: Some(reference.clone()),
})?;
tx.commit()?;
let wire = op.stamp.wire(op.payload.clone());
let record = wire.podcast_progress().expect("progress payload");
Ok(ProgressWrite {
session: ProgressSession {
episode: episode.clone(),
session_id: record.session_id.clone(),
generation,
},
record,
op,
})
}
/// Starts a listening session at `position_ms` (explicit action).
///
/// The generation is `explicit_generation(max(local, carried_generation))`;
/// pass `PodcastEpisodeRef::progress_generation` of a queue item received
/// through a handoff as `carried_generation`. Ordinary Play resumes from
/// [`PodcastStore::resume_position_ms`].
pub fn start_session(
&self,
stamps: &mut dyn StampSource,
episode: &PodcastEpisodeRef,
session_id: &str,
position_ms: i64,
carried_generation: Option<u64>,
) -> Result<ProgressWrite> {
self.explicit_progress(
stamps,
episode,
Some(session_id),
carried_generation,
ExplicitAction::Start(position_ms),
)
}
/// Periodic checkpoint (or pause/stop save) of a running session.
///
/// Keeps the session generation. When the register already holds a
/// greater generation nothing is written ([`CheckpointOutcome::Superseded`]).
pub fn checkpoint(
&self,
stamps: &mut dyn StampSource,
session: &ProgressSession,
position_ms: i64,
) -> Result<CheckpointOutcome> {
let episode = &session.episode;
let mut conn = self.lock();
let tx = conn.transaction_with_behavior(TransactionBehavior::Immediate)?;
if let Some(current) = apply::load_progress(&tx, &episode.episode_id)? {
if current.generation > session.generation {
return Ok(CheckpointOutcome::Superseded { current });
}
if current.generation == session.generation
&& current.session_id == session.session_id
&& current.status == ProgressStatus::Completed
{
return Ok(CheckpointOutcome::AlreadyCompleted { current });
}
}
let reference = op_ref(episode);
let op = emit(&tx, stamps, |_| SyncOpPayload::PodcastProgressSet {
episode_id: episode.episode_id.clone(),
podcast_id: episode.podcast_id.clone(),
position_ms,
status: ProgressStatus::InProgress,
generation: session.generation,
session_id: session.session_id.clone(),
episode: Some(reference.clone()),
})?;
tx.commit()?;
let record = op
.stamp
.wire(op.payload.clone())
.podcast_progress()
.expect("progress payload");
Ok(CheckpointOutcome::Written(Box::new(ProgressWrite {
record,
session: session.clone(),
op,
})))
}
/// User seek during a session (explicit action; the returned session
/// carries the new generation for later checkpoints).
pub fn seek(
&self,
stamps: &mut dyn StampSource,
session: &ProgressSession,
position_ms: i64,
) -> Result<ProgressWrite> {
self.explicit_progress(
stamps,
&session.episode,
Some(&session.session_id),
Some(session.generation),
ExplicitAction::Seek(position_ms),
)
}
/// The session reached the end (`reaches_completion` or end of stream):
/// writes `completed` with an explicit generation. Record the session
/// itself with [`PodcastStore::record_listen_session`].
pub fn complete_session(
&self,
stamps: &mut dyn StampSource,
session: &ProgressSession,
position_ms: i64,
) -> Result<ProgressWrite> {
self.explicit_progress(
stamps,
&session.episode,
Some(&session.session_id),
Some(session.generation),
ExplicitAction::Complete(position_ms),
)
}
/// "Start over": position 0, in progress, explicit generation.
pub fn restart(
&self,
stamps: &mut dyn StampSource,
episode: &PodcastEpisodeRef,
session_id: &str,
) -> Result<ProgressWrite> {
self.explicit_progress(
stamps,
episode,
Some(session_id),
None,
ExplicitAction::Restart,
)
}
/// "Mark played": status completed, explicit generation. Creates no
/// listening session and invents no listening time.
pub fn mark_played(
&self,
stamps: &mut dyn StampSource,
episode: &PodcastEpisodeRef,
) -> Result<ProgressWrite> {
self.explicit_progress(stamps, episode, None, None, ExplicitAction::MarkPlayed)
}
/// "Mark unplayed": status not started at position 0, explicit generation.
pub fn mark_unplayed(
&self,
stamps: &mut dyn StampSource,
episode: &PodcastEpisodeRef,
) -> Result<ProgressWrite> {
self.explicit_progress(stamps, episode, None, None, ExplicitAction::MarkUnplayed)
}
/// Records (or grows) a listening session of this device.
///
/// Sessions below the history threshold (`ListenSession::should_record`)
/// and versions that do not grow the stored session emit nothing.
/// `session.device_id` must be the local device id. The episode reference
/// defaults to the catalog's.
pub fn record_listen_session(
&self,
stamps: &mut dyn StampSource,
session: &ListenSession,
episode: Option<&PodcastEpisodeRef>,
) -> Result<Option<PodcastLocalOp>> {
if session.device_id != stamps.device_id() {
return Err(invalid(
"session device_id differs from the local device id",
));
}
session.validate().map_err(invalid)?;
if !session.should_record() {
return Ok(None);
}
let mut conn = self.lock();
let tx = conn.transaction_with_behavior(TransactionBehavior::Immediate)?;
if let Some(current) = apply::load_session(&tx, &session.session_id)? {
let merged = merge_listen_session(&current, session).map_err(invalid)?;
if merged == current {
return Ok(None);
}
}
let reference = match episode {
Some(episode) => Some(op_ref(episode)),
None => super::queries::episode_ref_in(&tx, &session.episode_id)?.map(|r| op_ref(&r)),
};
let op = emit(&tx, stamps, |_| SyncOpPayload::PodcastListenRecorded {
session: session.clone(),
episode: reference.clone(),
})?;
tx.commit()?;
Ok(Some(op))
}
/// Resolves a feed URL to a podcast id: the existing podcast when the
/// URL is a known feed URL or alias (`resolve_podcast_id`), otherwise
/// the id a first subscription through this URL would derive.
pub fn resolve_podcast_id(&self, url: &FeedUrl) -> Result<PodcastId> {
resolve_podcast(&self.lock(), url)
}
/// The episode reference for queueing or progress actions, or `None`
/// when the episode is unknown.
pub fn episode_ref(&self, episode_id: &EpisodeId) -> Result<Option<PodcastEpisodeRef>> {
super::queries::episode_ref_in(&self.lock(), episode_id)
}
}
+742
View File
@@ -0,0 +1,742 @@
//! The single code path that applies podcast records to the store.
//!
//! Remote operations, local actions and snapshot sections all end here.
//! Records come from the `SyncOpWire` materializers of
//! `music_dht::device_sync` and are merged with the functions of
//! `music_dht::podcast`, so applying is idempotent and order-independent.
use anyhow::Result;
use music_dht::device_sync::{SyncOpPayload, SyncOpWire, SyncSnapshot};
use music_dht::podcast::{
EpisodeIdentityHints, EpisodeMeta, FeedUrl, KnownEpisode, ListenSession, PodcastEpisodeRef,
PodcastError, PodcastId, ProgressRecord, ShareRecord, SubscriptionRecord, merge_feed_aliases,
merge_listen_session,
};
use rusqlite::{Connection, OptionalExtension as _, TransactionBehavior, params};
use super::{
PROGRESS_COLUMNS, PodcastStore, PodcastStoreError, SESSION_COLUMNS, episode_id_col,
feed_url_col, generation_to_sql, now_ms, podcast_meta_col, progress_from_row, session_from_row,
status_to_sql,
};
/// Result of applying one operation.
#[derive(Debug, Clone, PartialEq, Eq)]
#[non_exhaustive]
pub enum ApplyOutcome {
/// Stored state changed.
Changed,
/// The operation was already reflected (duplicate, or older than the
/// stored register).
Unchanged,
/// Not a podcast operation; nothing to do here.
Ignored,
/// The operation violates the contract and was skipped. Acknowledge it
/// anyway so the log keeps moving (`PODCASTS.md`).
Rejected(String),
}
/// Counters returned by [`PodcastStore::apply_snapshot`].
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub struct SnapshotApplyStats {
/// Records that changed stored state.
pub changed: usize,
/// Valid records already reflected locally.
pub unchanged: usize,
/// Records that failed validation and were skipped.
pub rejected: usize,
/// Whether the podcast sections were ignored for exceeding the counts.
pub ignored_over_limits: bool,
}
/// Where an episode row came from.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(crate) enum EpisodeSource {
/// Parsed from the podcast's feed on this device.
Feed,
/// Learned from a `PodcastEpisodeRef` (sync operation or queue item).
Sync,
}
impl EpisodeSource {
fn as_sql(self) -> &'static str {
match self {
Self::Feed => "feed",
Self::Sync => "sync",
}
}
}
fn invalid(error: impl ToString) -> anyhow::Error {
PodcastStoreError::Invalid(error.to_string()).into()
}
pub(crate) fn load_subscription_register(
conn: &Connection,
podcast_id: &PodcastId,
) -> Result<Option<SubscriptionRecord>> {
let row = conn
.query_row(
"SELECT subscribed, podcast_json, generation, hlc_ms, op_id
FROM podcast_subscriptions WHERE podcast_id = ?1",
[podcast_id.as_str()],
|row| {
let generation: i64 = row.get(2)?;
Ok(SubscriptionRecord {
podcast_id: podcast_id.clone(),
subscribed: row.get(0)?,
podcast: podcast_meta_col(&row.get::<_, String>(1)?)?,
feed_aliases: Vec::new(),
generation: u64::try_from(generation).unwrap_or(0),
hlc_ms: row.get(3)?,
op_id: row.get(4)?,
})
},
)
.optional()?;
let Some(mut record) = row else {
return Ok(None);
};
record.feed_aliases = load_aliases(conn, podcast_id)?;
Ok(Some(record))
}
pub(crate) fn load_aliases(conn: &Connection, podcast_id: &PodcastId) -> Result<Vec<FeedUrl>> {
let mut statement = conn.prepare_cached(
"SELECT feed_url FROM podcast_feed_aliases WHERE podcast_id = ?1 ORDER BY feed_url",
)?;
let urls = statement
.query_map([podcast_id.as_str()], |row| feed_url_col(row.get(0)?))?
.collect::<rusqlite::Result<Vec<_>>>()?;
Ok(urls)
}
pub(crate) fn load_share(conn: &Connection, podcast_id: &PodcastId) -> Result<Option<ShareRecord>> {
Ok(conn
.query_row(
"SELECT shared, generation, hlc_ms, op_id FROM podcast_shares WHERE podcast_id = ?1",
[podcast_id.as_str()],
|row| {
let generation: i64 = row.get(1)?;
Ok(ShareRecord {
podcast_id: podcast_id.clone(),
shared: row.get(0)?,
generation: u64::try_from(generation).unwrap_or(0),
hlc_ms: row.get(2)?,
op_id: row.get(3)?,
})
},
)
.optional()?)
}
pub(crate) fn load_progress(
conn: &Connection,
episode_id: &music_dht::podcast::EpisodeId,
) -> Result<Option<ProgressRecord>> {
Ok(conn
.query_row(
&format!("SELECT {PROGRESS_COLUMNS} FROM episode_progress p WHERE p.episode_id = ?1"),
[episode_id.as_str()],
|row| progress_from_row(row, 0),
)
.optional()?)
}
pub(crate) fn load_session(conn: &Connection, session_id: &str) -> Result<Option<ListenSession>> {
Ok(conn
.query_row(
&format!(
"SELECT {SESSION_COLUMNS} FROM podcast_listen_sessions s WHERE s.session_id = ?1"
),
[session_id],
|row| session_from_row(row, 0),
)
.optional()?)
}
/// Replaces the alias set of a podcast with `merged` when it differs.
fn store_aliases(conn: &Connection, podcast_id: &PodcastId, merged: &[FeedUrl]) -> Result<bool> {
let current = load_aliases(conn, podcast_id)?;
if current == merged {
return Ok(false);
}
conn.execute(
"DELETE FROM podcast_feed_aliases WHERE podcast_id = ?1",
[podcast_id.as_str()],
)?;
let mut insert = conn.prepare_cached(
"INSERT INTO podcast_feed_aliases (podcast_id, feed_url) VALUES (?1, ?2)",
)?;
for url in merged {
insert.execute(params![podcast_id.as_str(), url.as_str()])?;
}
Ok(true)
}
/// Merges feed URLs into the bounded grow-only alias set of a podcast.
///
/// Returns whether the stored set changed.
pub(crate) fn merge_aliases(
conn: &Connection,
podcast_id: &PodcastId,
urls: &[FeedUrl],
) -> Result<bool> {
let current = load_aliases(conn, podcast_id)?;
let merged = merge_feed_aliases(podcast_id, &current, urls);
store_aliases(conn, podcast_id, &merged)
}
/// Inserts the catalog row of a podcast if it does not exist yet.
fn ensure_podcast_row(
conn: &Connection,
podcast_id: &PodcastId,
feed_url: &FeedUrl,
title: &str,
now: i64,
) -> Result<()> {
conn.execute(
"INSERT OR IGNORE INTO podcasts
(podcast_id, feed_url, title, meta_from_feed, created_at_ms, updated_at_ms)
VALUES (?1, ?2, ?3, 0, ?4, ?4)",
params![podcast_id.as_str(), feed_url.as_str(), title, now],
)?;
Ok(())
}
/// Applies a subscription register record (operation or snapshot).
///
/// The register is last-writer-wins ([`SubscriptionRecord::supersedes`]);
/// the alias set is a bounded grow-only set merged independently, so the
/// stored aliases are the bounded union of every URL seen (equal to what
/// `merge_subscription` computes). The catalog row is created from the
/// register metadata and follows the winning register until this device has
/// refreshed the feed itself.
pub(crate) fn apply_subscription_record(
conn: &Connection,
record: &SubscriptionRecord,
now: i64,
) -> Result<bool> {
generation_to_sql(record.generation)?;
let podcast_id = &record.podcast_id;
let current = load_subscription_register(conn, podcast_id)?;
let wins = current
.as_ref()
.is_none_or(|current| record.supersedes(current));
let mut changed = false;
if wins {
conn.execute(
"INSERT INTO podcast_subscriptions
(podcast_id, subscribed, podcast_json, generation, hlc_ms, op_id)
VALUES (?1, ?2, ?3, ?4, ?5, ?6)
ON CONFLICT(podcast_id) DO UPDATE SET
subscribed = excluded.subscribed, podcast_json = excluded.podcast_json,
generation = excluded.generation, hlc_ms = excluded.hlc_ms,
op_id = excluded.op_id",
params![
podcast_id.as_str(),
record.subscribed,
serde_json::to_string(&record.podcast)?,
generation_to_sql(record.generation)?,
record.hlc_ms,
record.op_id,
],
)?;
changed = true;
}
let mut urls = record.feed_aliases.clone();
urls.push(record.podcast.feed_url.clone());
changed |= merge_aliases(conn, podcast_id, &urls)?;
ensure_podcast_row(
conn,
podcast_id,
&record.podcast.feed_url,
&record.podcast.title,
now,
)?;
if wins {
let meta = &record.podcast;
conn.execute(
"UPDATE podcasts
SET feed_url = ?2, title = ?3, author = ?4,
description = COALESCE(?5, description), link = ?6, image_url = ?7,
updated_at_ms = ?8
WHERE podcast_id = ?1 AND meta_from_feed = 0",
params![
podcast_id.as_str(),
meta.feed_url.as_str(),
meta.title,
meta.author,
meta.description,
meta.link,
meta.image_url,
now
],
)?;
}
Ok(changed)
}
/// Applies a share register record.
pub(crate) fn apply_share_record(conn: &Connection, record: &ShareRecord) -> Result<bool> {
let generation = generation_to_sql(record.generation)?;
let current = load_share(conn, &record.podcast_id)?;
if current
.as_ref()
.is_some_and(|current| !record.supersedes(current))
{
return Ok(false);
}
conn.execute(
"INSERT INTO podcast_shares (podcast_id, shared, generation, hlc_ms, op_id)
VALUES (?1, ?2, ?3, ?4, ?5)
ON CONFLICT(podcast_id) DO UPDATE SET
shared = excluded.shared, generation = excluded.generation,
hlc_ms = excluded.hlc_ms, op_id = excluded.op_id",
params![
record.podcast_id.as_str(),
record.shared,
generation,
record.hlc_ms,
record.op_id
],
)?;
Ok(true)
}
/// Applies a progress register record. Never touches the catalog.
pub(crate) fn apply_progress_record(conn: &Connection, record: &ProgressRecord) -> Result<bool> {
let generation = generation_to_sql(record.generation)?;
let current = load_progress(conn, &record.episode_id)?;
if current
.as_ref()
.is_some_and(|current| !record.supersedes(current))
{
return Ok(false);
}
conn.execute(
"INSERT INTO episode_progress
(episode_id, podcast_id, position_ms, status, generation, session_id, hlc_ms,
device_id, op_id)
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9)
ON CONFLICT(episode_id) DO UPDATE SET
podcast_id = excluded.podcast_id, position_ms = excluded.position_ms,
status = excluded.status, generation = excluded.generation,
session_id = excluded.session_id, hlc_ms = excluded.hlc_ms,
device_id = excluded.device_id, op_id = excluded.op_id",
params![
record.episode_id.as_str(),
record.podcast_id.as_str(),
record.position_ms,
status_to_sql(record.status),
generation,
record.session_id,
record.hlc_ms,
record.device_id,
record.op_id,
],
)?;
Ok(true)
}
/// Adds or grows a listening session (grow-only set keyed by `session_id`).
pub(crate) fn apply_listen_session(conn: &Connection, session: &ListenSession) -> Result<bool> {
let current = load_session(conn, &session.session_id)?;
let merged = match &current {
Some(current) => merge_listen_session(current, session).map_err(invalid)?,
None => session.clone(),
};
if current.as_ref() == Some(&merged) {
return Ok(false);
}
conn.execute(
"INSERT INTO podcast_listen_sessions
(session_id, episode_id, podcast_id, device_id, started_at_ms, ended_at_ms,
start_position_ms, end_position_ms, listened_ms, completed)
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10)
ON CONFLICT(session_id) DO UPDATE SET
started_at_ms = excluded.started_at_ms, ended_at_ms = excluded.ended_at_ms,
start_position_ms = excluded.start_position_ms,
end_position_ms = excluded.end_position_ms,
listened_ms = excluded.listened_ms, completed = excluded.completed",
params![
merged.session_id,
merged.episode_id.as_str(),
merged.podcast_id.as_str(),
merged.device_id,
merged.started_at_ms,
merged.ended_at_ms,
merged.start_position_ms,
merged.end_position_ms,
merged.listened_ms,
merged.completed,
],
)?;
Ok(true)
}
/// Inserts an episode row.
pub(crate) fn insert_episode(
conn: &Connection,
meta: &EpisodeMeta,
source: EpisodeSource,
in_feed: bool,
legacy_guid: Option<&str>,
first_seen_at_ms: i64,
now: i64,
) -> Result<()> {
conn.execute(
"INSERT INTO episodes
(episode_id, podcast_id, guid, title, description, season, episode_number,
episode_type, published_at_ms, duration_ms, audio_url, mime, byte_len,
image_url, link, source, in_feed, legacy_guid, first_seen_at_ms, updated_at_ms)
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11, ?12, ?13, ?14, ?15, ?16, ?17,
?18, ?19, ?20)",
params![
meta.episode_id.as_str(),
meta.podcast_id.as_str(),
meta.guid,
meta.title,
meta.description,
meta.season,
meta.episode_number,
episode_type_to_sql(meta.episode_type),
meta.published_at_ms,
meta.duration_ms,
meta.audio_url,
meta.mime,
meta.byte_len.and_then(|len| i64::try_from(len).ok()),
meta.image_url,
meta.link,
source.as_sql(),
in_feed,
legacy_guid,
first_seen_at_ms,
now,
],
)?;
Ok(())
}
pub(crate) fn episode_type_to_sql(kind: music_dht::podcast::EpisodeType) -> &'static str {
match kind {
music_dht::podcast::EpisodeType::Full => "full",
music_dht::podcast::EpisodeType::Trailer => "trailer",
music_dht::podcast::EpisodeType::Bonus => "bonus",
}
}
/// Records an episode learned through sync when the catalog does not know it.
///
/// Feed metadata always wins: an existing row is left untouched. The
/// podcast catalog row is created from the reference when missing (no
/// register is written).
pub(crate) fn ensure_episode_from_ref(
conn: &Connection,
episode: &PodcastEpisodeRef,
) -> Result<()> {
let exists = conn
.query_row(
"SELECT 1 FROM episodes WHERE episode_id = ?1",
[episode.episode_id.as_str()],
|_| Ok(()),
)
.optional()?
.is_some();
if exists {
return Ok(());
}
let now = now_ms();
ensure_podcast_row(
conn,
&episode.podcast_id,
&episode.feed_url,
&episode.podcast_title,
now,
)?;
let meta = EpisodeMeta {
episode_id: episode.episode_id.clone(),
podcast_id: episode.podcast_id.clone(),
guid: None,
title: episode.episode_title.clone(),
description: None,
season: episode.season,
episode_number: episode.episode_number,
episode_type: Default::default(),
published_at_ms: episode.published_at_ms,
duration_ms: episode.duration_ms,
audio_url: episode.audio_url.clone(),
mime: episode.mime.clone(),
byte_len: None,
image_url: episode.image_url.clone(),
link: None,
};
insert_episode(conn, &meta, EpisodeSource::Sync, false, None, now, now)?;
refresh_seasons(conn, &episode.podcast_id)?;
Ok(())
}
/// Recomputes the derived season rows of a podcast from its episodes.
pub(crate) fn refresh_seasons(conn: &Connection, podcast_id: &PodcastId) -> Result<()> {
conn.execute(
"DELETE FROM podcast_seasons WHERE podcast_id = ?1",
[podcast_id.as_str()],
)?;
conn.execute(
"INSERT INTO podcast_seasons
(podcast_id, season, episode_count, first_published_at_ms, last_published_at_ms)
SELECT podcast_id, season, COUNT(*), MIN(published_at_ms), MAX(published_at_ms)
FROM episodes WHERE podcast_id = ?1 GROUP BY season",
[podcast_id.as_str()],
)?;
Ok(())
}
/// Known episodes of a podcast with the identity hints they were last seen with.
pub(crate) fn known_episodes(
conn: &Connection,
podcast_id: &PodcastId,
) -> Result<Vec<KnownEpisode>> {
let mut statement = conn.prepare_cached(
"SELECT episode_id, guid, audio_url, link, published_at_ms, title
FROM episodes WHERE podcast_id = ?1 ORDER BY episode_id",
)?;
let known = statement
.query_map([podcast_id.as_str()], |row| {
Ok(KnownEpisode {
episode_id: episode_id_col(row.get(0)?)?,
hints: EpisodeIdentityHints {
guid: row.get(1)?,
enclosure_url: Some(row.get(2)?),
link: row.get(3)?,
published_at_ms: row.get(4)?,
title: Some(row.get(5)?),
},
})
})?
.collect::<rusqlite::Result<Vec<_>>>()?;
Ok(known)
}
/// Applies one podcast operation inside the caller's transaction.
pub(crate) fn apply_op(conn: &Connection, op: &SyncOpWire) -> Result<ApplyOutcome> {
if !op.payload.is_podcast() {
return Ok(ApplyOutcome::Ignored);
}
if let Err(error) = op.validate_podcast() {
return Ok(ApplyOutcome::Rejected(error.to_string()));
}
let result = match &op.payload {
SyncOpPayload::PodcastSubscriptionSet { .. } => {
let record = op.podcast_subscription().expect("subscription payload");
apply_subscription_record(conn, &record, now_ms())
}
SyncOpPayload::PodcastShareSet { .. } => {
apply_share_record(conn, &op.podcast_share().expect("share payload"))
}
SyncOpPayload::PodcastFeedAliasesAdded { .. } => {
let (podcast_id, urls) = op.podcast_feed_aliases().expect("alias payload");
merge_aliases(conn, podcast_id, urls)
}
SyncOpPayload::PodcastProgressSet { episode, .. } => {
let record = op.podcast_progress().expect("progress payload");
if let Err(error) = generation_to_sql(record.generation) {
return Ok(ApplyOutcome::Rejected(error.to_string()));
}
if let Some(episode) = episode {
ensure_episode_from_ref(conn, episode)?;
}
apply_progress_record(conn, &record)
}
SyncOpPayload::PodcastListenRecorded { session, episode } => {
if let Some(current) = load_session(conn, &session.session_id)?
&& let Err(error) = merge_listen_session(&current, session)
{
return Ok(ApplyOutcome::Rejected(error.to_string()));
}
if let Some(episode) = episode {
ensure_episode_from_ref(conn, episode)?;
}
apply_listen_session(conn, session)
}
_ => return Ok(ApplyOutcome::Ignored),
};
match result {
Ok(true) => Ok(ApplyOutcome::Changed),
Ok(false) => Ok(ApplyOutcome::Unchanged),
Err(error) => match error.downcast_ref::<PodcastStoreError>() {
Some(PodcastStoreError::Invalid(reason)) => Ok(ApplyOutcome::Rejected(reason.clone())),
_ => Err(error),
},
}
}
fn record_outcome(stats: &mut SnapshotApplyStats, result: Result<bool>) -> Result<()> {
match result {
Ok(true) => stats.changed += 1,
Ok(false) => stats.unchanged += 1,
Err(error) => match error.downcast_ref::<PodcastStoreError>() {
Some(PodcastStoreError::Invalid(_)) => stats.rejected += 1,
_ => return Err(error),
},
}
Ok(())
}
fn check(result: Result<(), PodcastError>) -> Result<()> {
result.map_err(invalid)
}
impl PodcastStore {
/// Applies a device-sync operation received from a trusted device.
///
/// Non-podcast operations return [`ApplyOutcome::Ignored`]. Podcast
/// operations are validated (`SyncOpWire::validate_podcast`) and merged
/// with the contract functions in one transaction; applying the same
/// operations in any order, with duplicates, gives the same state.
pub fn apply_remote_op(&self, op: &SyncOpWire) -> Result<ApplyOutcome> {
if !op.payload.is_podcast() {
return Ok(ApplyOutcome::Ignored);
}
let mut conn = self.lock();
let tx = conn.transaction_with_behavior(TransactionBehavior::Immediate)?;
let outcome = apply_op(&tx, op)?;
tx.commit()?;
Ok(outcome)
}
/// Merges the podcast sections of a sync snapshot.
///
/// Sections exceeding the contract counts are ignored
/// (`SyncSnapshot::podcast_state_within_limits`); every record is
/// validated and merged with the same functions as operations, so a
/// snapshot never replaces newer local state. One transaction.
pub fn apply_snapshot(&self, snapshot: &SyncSnapshot) -> Result<SnapshotApplyStats> {
let mut stats = SnapshotApplyStats::default();
if !snapshot.podcast_state_within_limits() {
stats.ignored_over_limits = true;
return Ok(stats);
}
let mut conn = self.lock();
let tx = conn.transaction_with_behavior(TransactionBehavior::Immediate)?;
let now = now_ms();
for record in &snapshot.podcast_subscriptions {
let mut record = record.clone();
record.normalize();
let result = check(record.validate())
.and_then(|()| apply_subscription_record(&tx, &record, now));
record_outcome(&mut stats, result)?;
}
for record in &snapshot.podcast_shares {
let result = check(record.validate()).and_then(|()| apply_share_record(&tx, record));
record_outcome(&mut stats, result)?;
}
for record in &snapshot.podcast_progress {
let result = check(record.validate()).and_then(|()| apply_progress_record(&tx, record));
record_outcome(&mut stats, result)?;
}
for session in &snapshot.podcast_listen_sessions {
let result =
check(session.validate()).and_then(|()| apply_listen_session(&tx, session));
record_outcome(&mut stats, result)?;
}
tx.commit()?;
Ok(stats)
}
/// Fills the podcast sections of `snapshot` with this store's registers
/// and sessions, bounded by `SyncSnapshot::bound_podcast_state_to`.
///
/// `budget_bytes` is usually `MAX_PODCAST_SNAPSHOT_BYTES` or
/// `device_sync::podcast_snapshot_budget(other_bytes)`. Records are read
/// newest first and capped by the per-section counts before the byte
/// budget is applied. Local fetch state and the catalog are never part
/// of the snapshot. Send it only to v4 peers (`SyncSnapshot::for_peer`).
pub fn fill_snapshot(&self, snapshot: &mut SyncSnapshot, budget_bytes: usize) -> Result<()> {
use music_dht::podcast::{
MAX_SNAPSHOT_LISTEN_SESSIONS, MAX_SNAPSHOT_PROGRESS, MAX_SNAPSHOT_SHARES,
MAX_SNAPSHOT_SUBSCRIPTIONS,
};
let conn = self.lock();
let ids: Vec<PodcastId> = {
let mut statement = conn.prepare(
"SELECT podcast_id FROM podcast_subscriptions
ORDER BY generation DESC, hlc_ms DESC, op_id DESC, podcast_id DESC
LIMIT ?1",
)?;
statement
.query_map([MAX_SNAPSHOT_SUBSCRIPTIONS as i64], |row| {
super::podcast_id_col(row.get(0)?)
})?
.collect::<rusqlite::Result<Vec<_>>>()?
};
let mut subscriptions = Vec::with_capacity(ids.len());
for id in &ids {
if let Some(record) = load_subscription_register(&conn, id)? {
subscriptions.push(record);
}
}
let shares = {
let mut statement = conn.prepare(
"SELECT podcast_id, shared, generation, hlc_ms, op_id FROM podcast_shares
ORDER BY generation DESC, hlc_ms DESC, op_id DESC, podcast_id DESC
LIMIT ?1",
)?;
statement
.query_map([MAX_SNAPSHOT_SHARES as i64], |row| {
let generation: i64 = row.get(2)?;
Ok(ShareRecord {
podcast_id: super::podcast_id_col(row.get(0)?)?,
shared: row.get(1)?,
generation: u64::try_from(generation).unwrap_or(0),
hlc_ms: row.get(3)?,
op_id: row.get(4)?,
})
})?
.collect::<rusqlite::Result<Vec<_>>>()?
};
let progress = {
let mut statement = conn.prepare(&format!(
"SELECT {PROGRESS_COLUMNS} FROM episode_progress p
ORDER BY p.generation DESC, p.hlc_ms DESC, p.op_id DESC, p.episode_id DESC
LIMIT ?1"
))?;
statement
.query_map([MAX_SNAPSHOT_PROGRESS as i64], |row| {
progress_from_row(row, 0)
})?
.collect::<rusqlite::Result<Vec<_>>>()?
};
let sessions = {
let mut statement = conn.prepare(&format!(
"SELECT {SESSION_COLUMNS} FROM podcast_listen_sessions s
ORDER BY s.ended_at_ms DESC, s.session_id DESC
LIMIT ?1"
))?;
statement
.query_map([MAX_SNAPSHOT_LISTEN_SESSIONS as i64], |row| {
session_from_row(row, 0)
})?
.collect::<rusqlite::Result<Vec<_>>>()?
};
drop(conn);
snapshot.podcast_subscriptions = subscriptions;
snapshot.podcast_shares = shares;
snapshot.podcast_progress = progress;
snapshot.podcast_listen_sessions = sessions;
snapshot.bound_podcast_state_to(budget_bytes);
Ok(())
}
/// A snapshot holding only this store's podcast sections, bounded with
/// the default budget (`MAX_PODCAST_SNAPSHOT_BYTES`).
pub fn export_snapshot(&self) -> Result<SyncSnapshot> {
let mut snapshot = SyncSnapshot::default();
self.fill_snapshot(
&mut snapshot,
music_dht::podcast::MAX_PODCAST_SNAPSHOT_BYTES,
)?;
Ok(snapshot)
}
}
+451
View File
@@ -0,0 +1,451 @@
//! SQLite reference store for podcasts (`podcasts.sqlite3`).
//!
//! [`PodcastStore`] persists the podcast contract of `music_dht::podcast`
//! for SQLite clients: the local episode catalog refreshed from RSS, the
//! synchronized personal registers (subscription, share, progress), the
//! grow-only listening sessions and the local-only fetch state. It owns no
//! networking, HTTP, scheduling or audio: clients fetch feeds, parse them
//! with `music_dht::podcast::feed`, append sync operations to their own log
//! and exchange snapshots. The normative rules live in the workspace
//! `PODCASTS.md` (section "SQLite reference store" describes this schema).
//!
//! # Files and migrations
//!
//! The store lives in its own file, conventionally [`PODCAST_DB_FILE`] next
//! to the library database; the caller passes the path. Connections use
//! WAL, foreign keys and a busy timeout. `PRAGMA user_version` drives the
//! migrations, each applied in its own `BEGIN IMMEDIATE` transaction that
//! re-checks the version, so concurrent openers and interrupted upgrades are
//! safe:
//!
//! | `user_version` | Schema |
//! | --- | --- |
//! | 0 | empty file |
//! | 1 | prototype schema of the original podcast PR (`podcasts`, `podcast_episodes`) |
//! | 2 | contract model; the prototype tables are renamed `legacy_*` and imported |
//!
//! A database written by a newer Furumi ([`PODCAST_SCHEMA_VERSION`] lower
//! than its `user_version`) or a file that is not a podcast database is
//! refused before anything is written ([`PodcastStoreError`]).
//!
//! # Personal state and sync
//!
//! Local actions ([`PodcastStore::subscribe`], [`PodcastStore::seek`], ...)
//! draw operation stamps from a [`StampSource`], build the device-sync
//! operation, apply it through the same code path as
//! [`PodcastStore::apply_remote_op`] (the `SyncOpWire` materializers of
//! `music_dht::device_sync` and the merge functions of `music_dht::podcast`)
//! and return the [`PodcastLocalOp`]s. The caller appends them to its sync
//! log **with exactly these stamps**; applying them again is a no-op.
//! Applying operations and snapshot sections is idempotent and independent
//! of delivery order.
//!
//! RSS refreshes ([`PodcastStore::apply_feed`]) only touch the catalog and
//! feed aliases: they never write a register, progress or a session.
//!
//! ```
//! use furumi_library::podcasts::{ApplyOutcome, OpStamp, PodcastStore, StampSource};
//! use music_dht::podcast::{FeedUrl, PodcastMeta};
//!
//! /// Stand-in for the client's sync log, which reserves seq and hlc.
//! struct Log {
//! device: String,
//! seq: i64,
//! }
//!
//! impl StampSource for Log {
//! fn device_id(&self) -> &str {
//! &self.device
//! }
//! fn next_stamp(&mut self) -> anyhow::Result<OpStamp> {
//! self.seq += 1;
//! let op_id = format!("{}:{}", self.device, self.seq);
//! Ok(OpStamp::new(op_id, self.device.clone(), 1_700_000_000_000 + self.seq))
//! }
//! }
//!
//! let store = PodcastStore::open_in_memory()?;
//! let mut log = Log { device: "device-a".into(), seq: 0 };
//! let meta = PodcastMeta {
//! feed_url: FeedUrl::parse("https://example.test/feed.xml")?,
//! title: "Show".into(),
//! author: None,
//! description: None,
//! link: None,
//! image_url: None,
//! };
//! let outcome = store.subscribe(&mut log, &meta)?;
//! assert_eq!(outcome.ops.len(), 1);
//! for op in outcome.ops {
//! // Append to the sync log with this stamp; re-applying is a no-op.
//! let op = op.into_wire(log.seq);
//! assert_eq!(store.apply_remote_op(&op)?, ApplyOutcome::Unchanged);
//! }
//! # Ok::<(), anyhow::Error>(())
//! ```
//!
//! # Thread safety
//!
//! All methods take `&self` and serialize on an internal mutex around one
//! synchronous `rusqlite` connection, so an `Arc<PodcastStore>` can be shared
//! between threads. Calls block on SQLite I/O: run them on a blocking pool
//! (for example `tokio::task::spawn_blocking`), never on an async reactor
//! thread. Several processes may open the same file; SQLite serializes the
//! writers.
mod actions;
mod apply;
mod queries;
mod refresh;
mod schema;
#[cfg(test)]
mod tests;
use std::fmt;
use std::path::{Path, PathBuf};
use std::sync::{Mutex, MutexGuard, PoisonError};
use std::time::Duration;
use anyhow::{Context as _, Result};
use music_dht::device_sync::{SyncOpPayload, SyncOpWire};
use music_dht::podcast::{
EpisodeId, FeedUrl, ListenSession, PodcastError, PodcastId, PodcastMeta, ProgressRecord,
ProgressStatus,
};
use rusqlite::Connection;
pub use actions::{CheckpointOutcome, ProgressSession, ProgressWrite, SubscribeOutcome};
pub use apply::{ApplyOutcome, SnapshotApplyStats};
pub use queries::{
EpisodeListItem, EpisodeQuery, EpisodeSort, LegacyQuarantineEntry, PlayedFilter,
PodcastListFilter, PodcastListenEntry, PodcastSummary, SeasonFilter, SeasonSummary,
};
pub use refresh::{FeedRefresh, FetchState, RefreshOutcome};
pub use schema::{LEGACY_PROTOTYPE_SCHEMA_VERSION, PODCAST_SCHEMA_VERSION};
/// Conventional file name of the podcast store, next to `library.db`.
pub const PODCAST_DB_FILE: &str = "podcasts.sqlite3";
/// `op_id` prefix of registers imported from the prototype database.
///
/// Imported subscriptions use generation 0, `hlc_ms` = the prototype
/// `added_at` and `op_id` = this prefix followed by the podcast id, so every
/// write made through the contract orders after them.
pub const LEGACY_IMPORT_OP_PREFIX: &str = "legacy-import:";
/// Typed errors of [`PodcastStore`]; other failures are SQLite or I/O errors.
///
/// They are returned inside `anyhow::Error`; use `downcast_ref` to match.
#[derive(Debug, Clone, PartialEq, Eq)]
#[non_exhaustive]
pub enum PodcastStoreError {
/// The database was written by a newer Furumi. It was not modified.
NewerSchema {
/// `user_version` found in the file.
found: i64,
/// Newest version this build understands.
supported: i64,
},
/// The file is not a podcast database (or has an unexpected shape). It
/// was not modified.
UnrecognizedDatabase(String),
/// The podcast is unknown to this store.
UnknownPodcast(PodcastId),
/// The action needs an active subscription.
NotSubscribed(PodcastId),
/// Input violates the podcast contract.
Invalid(String),
}
impl fmt::Display for PodcastStoreError {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
Self::NewerSchema { found, supported } => write!(
f,
"podcast database schema v{found} is newer than this Furumi supports (v{supported}); \
update Furumi, the database was left unchanged"
),
Self::UnrecognizedDatabase(reason) => {
write!(f, "not a Furumi podcast database: {reason}")
}
Self::UnknownPodcast(id) => write!(f, "unknown podcast {id}"),
Self::NotSubscribed(id) => write!(f, "not subscribed to podcast {id}"),
Self::Invalid(reason) => write!(f, "invalid podcast data: {reason}"),
}
}
}
impl std::error::Error for PodcastStoreError {}
impl From<PodcastError> for PodcastStoreError {
fn from(error: PodcastError) -> Self {
Self::Invalid(error.to_string())
}
}
/// Stamp of a device-sync operation: the fields the `SyncOpWire`
/// materializers turn into register stamps.
///
/// `seq` is not part of it: the client's sync log assigns it when it appends
/// the operation ([`PodcastLocalOp::into_wire`]).
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct OpStamp {
/// Unique operation id, usually `<origin_device_id>:<seq>`.
pub op_id: String,
/// Origin device id (the progress register `device_id`).
pub origin_device_id: String,
/// Operation timestamp (register `hlc_ms`).
pub hlc_ms: i64,
}
impl OpStamp {
/// Builds a stamp.
pub fn new(op_id: impl Into<String>, origin_device_id: impl Into<String>, hlc_ms: i64) -> Self {
Self {
op_id: op_id.into(),
origin_device_id: origin_device_id.into(),
hlc_ms,
}
}
/// The stamp of an existing operation.
pub fn of(op: &SyncOpWire) -> Self {
Self::new(op.op_id.clone(), op.origin_device_id.clone(), op.hlc_ms)
}
pub(crate) fn wire(&self, payload: SyncOpPayload) -> SyncOpWire {
SyncOpWire {
op_id: self.op_id.clone(),
origin_device_id: self.origin_device_id.clone(),
seq: 0,
hlc_ms: self.hlc_ms,
payload,
}
}
}
/// Source of operation stamps for local actions, usually the client's sync
/// log (which reserves the next sequence number and timestamp).
///
/// A stamp is drawn only when an action actually emits an operation.
pub trait StampSource {
/// The local device id; every stamp must use it as origin.
fn device_id(&self) -> &str;
/// Reserves the stamp of the next local operation.
fn next_stamp(&mut self) -> Result<OpStamp>;
}
/// A local operation produced by a [`PodcastStore`] action.
///
/// The store has already applied it. Append it to the sync log with exactly
/// this stamp: a different `op_id` or `hlc_ms` would make other devices
/// materialize a different register than this one.
#[derive(Debug, Clone, PartialEq)]
pub struct PodcastLocalOp {
/// Stamp drawn from the [`StampSource`].
pub stamp: OpStamp,
/// Operation payload (one of the `podcast_*` kinds).
pub payload: SyncOpPayload,
}
impl PodcastLocalOp {
/// The operation with the sequence number assigned by the sync log.
pub fn into_wire(self, seq: i64) -> SyncOpWire {
let mut op = self.stamp.wire(self.payload);
op.seq = seq;
op
}
}
/// SQLite store of podcast catalog, personal state and fetch state.
///
/// See the [module documentation](self) for the schema, the sync model and
/// thread safety.
pub struct PodcastStore {
conn: Mutex<Connection>,
path: Option<PathBuf>,
}
impl fmt::Debug for PodcastStore {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.debug_struct("PodcastStore")
.field("path", &self.path)
.finish_non_exhaustive()
}
}
impl PodcastStore {
/// Opens (creating and migrating if needed) the podcast store at `path`.
///
/// A prototype database (`user_version` 1) is imported; see the module
/// documentation. Fails without modifying the file when it was written
/// by a newer Furumi or is not a podcast database.
pub fn open(path: &Path) -> Result<Self> {
if let Some(parent) = path.parent()
&& !parent.as_os_str().is_empty()
{
std::fs::create_dir_all(parent)
.with_context(|| format!("creating {}", parent.display()))?;
}
let conn = Connection::open(path)
.with_context(|| format!("opening podcast database {}", path.display()))?;
Self::init(conn, Some(path.to_path_buf()))
}
/// Opens a private in-memory store (tests and previews).
pub fn open_in_memory() -> Result<Self> {
Self::init(Connection::open_in_memory()?, None)
}
fn init(mut conn: Connection, path: Option<PathBuf>) -> Result<Self> {
conn.busy_timeout(Duration::from_secs(5))?;
// Refuse newer or foreign files before anything (WAL included) is written.
schema::check_openable(&conn)?;
if path.is_some() {
let _: String = conn.query_row("PRAGMA journal_mode=WAL", [], |row| row.get(0))?;
}
conn.pragma_update(None, "foreign_keys", "ON")?;
register_norm_function(&conn)?;
schema::migrate(&mut conn)?;
Ok(Self {
conn: Mutex::new(conn),
path,
})
}
/// Path of the database file (`None` for in-memory stores).
pub fn path(&self) -> Option<&Path> {
self.path.as_deref()
}
/// Current `PRAGMA user_version` (always [`PODCAST_SCHEMA_VERSION`]
/// after a successful open).
pub fn schema_version(&self) -> Result<i64> {
Ok(self
.lock()
.query_row("PRAGMA user_version", [], |row| row.get(0))?)
}
pub(crate) fn lock(&self) -> MutexGuard<'_, Connection> {
self.conn.lock().unwrap_or_else(PoisonError::into_inner)
}
}
fn register_norm_function(conn: &Connection) -> Result<()> {
use rusqlite::functions::FunctionFlags;
conn.create_scalar_function(
"norm",
1,
FunctionFlags::SQLITE_UTF8 | FunctionFlags::SQLITE_DETERMINISTIC,
|ctx| {
let value: Option<String> = ctx.get(0)?;
Ok(music_dht::normalize_name(
value.as_deref().unwrap_or_default(),
))
},
)?;
Ok(())
}
pub(crate) fn now_ms() -> i64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|duration| i64::try_from(duration.as_millis()).unwrap_or(i64::MAX))
.unwrap_or(0)
}
/// Converts a contract generation to the SQLite column value.
///
/// Generations above `i64::MAX` cannot be stored; records carrying one are
/// rejected (no conforming client produces them).
pub(crate) fn generation_to_sql(generation: u64) -> Result<i64, PodcastStoreError> {
i64::try_from(generation)
.map_err(|_| PodcastStoreError::Invalid("generation exceeds i64::MAX".to_string()))
}
pub(crate) fn status_to_sql(status: ProgressStatus) -> &'static str {
match status {
ProgressStatus::NotStarted => "not_started",
ProgressStatus::InProgress => "in_progress",
ProgressStatus::Completed => "completed",
}
}
pub(crate) fn status_from_sql(value: &str) -> ProgressStatus {
match value {
"in_progress" => ProgressStatus::InProgress,
"completed" => ProgressStatus::Completed,
_ => ProgressStatus::NotStarted,
}
}
/// Parses a stored id; stored ids are validated on write.
pub(crate) fn podcast_id_col(value: String) -> rusqlite::Result<PodcastId> {
PodcastId::try_from(value).map_err(|error| conversion_error(error.to_string()))
}
pub(crate) fn episode_id_col(value: String) -> rusqlite::Result<EpisodeId> {
EpisodeId::try_from(value).map_err(|error| conversion_error(error.to_string()))
}
pub(crate) fn feed_url_col(value: String) -> rusqlite::Result<FeedUrl> {
FeedUrl::try_from(value).map_err(|error| conversion_error(error.to_string()))
}
pub(crate) fn podcast_meta_col(value: &str) -> rusqlite::Result<PodcastMeta> {
serde_json::from_str(value).map_err(|error| conversion_error(error.to_string()))
}
pub(crate) fn conversion_error(message: String) -> rusqlite::Error {
rusqlite::Error::FromSqlConversionFailure(
0,
rusqlite::types::Type::Text,
Box::new(PodcastStoreError::Invalid(message)),
)
}
pub(crate) fn progress_from_row(
row: &rusqlite::Row<'_>,
at: usize,
) -> rusqlite::Result<ProgressRecord> {
let generation: i64 = row.get(at + 4)?;
Ok(ProgressRecord {
episode_id: episode_id_col(row.get(at)?)?,
podcast_id: podcast_id_col(row.get(at + 1)?)?,
position_ms: row.get(at + 2)?,
status: status_from_sql(&row.get::<_, String>(at + 3)?),
generation: u64::try_from(generation).unwrap_or(0),
session_id: row.get(at + 5)?,
hlc_ms: row.get(at + 6)?,
device_id: row.get(at + 7)?,
op_id: row.get(at + 8)?,
})
}
/// Column list matching [`progress_from_row`], for table alias `p`.
pub(crate) const PROGRESS_COLUMNS: &str = "p.episode_id, p.podcast_id, p.position_ms, p.status, \
p.generation, p.session_id, p.hlc_ms, p.device_id, p.op_id";
pub(crate) fn session_from_row(
row: &rusqlite::Row<'_>,
at: usize,
) -> rusqlite::Result<ListenSession> {
Ok(ListenSession {
session_id: row.get(at)?,
episode_id: episode_id_col(row.get(at + 1)?)?,
podcast_id: podcast_id_col(row.get(at + 2)?)?,
device_id: row.get(at + 3)?,
started_at_ms: row.get(at + 4)?,
ended_at_ms: row.get(at + 5)?,
start_position_ms: row.get(at + 6)?,
end_position_ms: row.get(at + 7)?,
listened_ms: row.get(at + 8)?,
completed: row.get(at + 9)?,
})
}
/// Column list matching [`session_from_row`], for table alias `s`.
pub(crate) const SESSION_COLUMNS: &str = "s.session_id, s.episode_id, s.podcast_id, s.device_id, \
s.started_at_ms, s.ended_at_ms, s.start_position_ms, s.end_position_ms, s.listened_ms, \
s.completed";
@@ -0,0 +1,605 @@
//! Read-only queries for clients' podcast UI.
use anyhow::Result;
use music_dht::podcast::{
EpisodeId, EpisodeMeta, EpisodeType, FeedUrl, ListenSession, PodcastEpisodeRef, PodcastId,
PodcastMeta, ProgressRecord, ProgressStatus, ShareRecord, SubscriptionRecord, effective_share,
};
use rusqlite::types::Value;
use rusqlite::{Connection, OptionalExtension as _, params, params_from_iter};
use super::apply;
use super::{
PROGRESS_COLUMNS, PodcastStore, SESSION_COLUMNS, episode_id_col, feed_url_col, podcast_id_col,
progress_from_row, session_from_row,
};
use crate::TrackItem;
/// Which podcasts [`PodcastStore::list_podcasts`] returns.
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub enum PodcastListFilter {
/// Podcasts with an active subscription.
#[default]
Subscribed,
/// Every podcast in the catalog, including unsubscribed ones and
/// podcasts only known from synced progress or queue items.
All,
}
/// A podcast with its personal state and episode counters.
#[derive(Debug, Clone, PartialEq)]
pub struct PodcastSummary {
/// Podcast id.
pub podcast_id: PodcastId,
/// Local channel metadata; `feed_url` is the URL this device fetches.
pub meta: PodcastMeta,
/// Whether the subscription register says subscribed.
pub subscribed: bool,
/// Whether the podcast must currently be published (`effective_share`).
pub shared: bool,
/// Episodes in the catalog.
pub episode_count: u64,
/// Episodes not completed (no progress, not started or in progress).
pub unplayed_count: u64,
/// Episodes in progress.
pub in_progress_count: u64,
/// Newest publication time.
pub latest_published_at_ms: Option<i64>,
/// Numbered seasons (episodes without a season are not counted).
pub season_count: u64,
}
/// One season group of a podcast. `season == None` groups the episodes
/// without `itunes:season`.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct SeasonSummary {
/// Season number; `None` for episodes without a season.
pub season: Option<u32>,
/// Episodes in the group.
pub episode_count: u64,
/// Episodes not completed.
pub unplayed_count: u64,
/// Oldest publication time.
pub first_published_at_ms: Option<i64>,
/// Newest publication time.
pub last_published_at_ms: Option<i64>,
}
/// Season filter of an [`EpisodeQuery`].
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub enum SeasonFilter {
/// All episodes.
#[default]
Any,
/// Episodes of one season.
Season(u32),
/// Episodes without a season.
NoSeason,
}
/// Listening-state filter of an [`EpisodeQuery`].
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub enum PlayedFilter {
/// All episodes.
#[default]
Any,
/// Not completed (not started or in progress).
Unplayed,
/// Never started or marked unplayed.
NotStarted,
/// In progress.
InProgress,
/// Completed.
Played,
}
/// Episode ordering of an [`EpisodeQuery`].
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub enum EpisodeSort {
/// Newest publication first; undated episodes last.
#[default]
NewestFirst,
/// Oldest publication first; undated episodes last.
OldestFirst,
/// By season, then episode number, ascending; unnumbered last.
EpisodeNumber,
/// By season, then episode number, descending; unnumbered last.
EpisodeNumberDesc,
}
/// Parameters of [`PodcastStore::episodes`].
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct EpisodeQuery {
/// Podcast.
pub podcast_id: PodcastId,
/// Season filter.
pub season: SeasonFilter,
/// Listening-state filter.
pub played: PlayedFilter,
/// Text searched in titles and descriptions (Unicode-normalized,
/// case-insensitive substring match). Blank means no filter.
pub search: Option<String>,
/// Ordering.
pub sort: EpisodeSort,
/// Maximum rows (`None`: all).
pub limit: Option<usize>,
/// Rows to skip.
pub offset: usize,
}
impl EpisodeQuery {
/// All episodes of a podcast, newest first.
pub fn new(podcast_id: PodcastId) -> Self {
Self {
podcast_id,
season: SeasonFilter::Any,
played: PlayedFilter::Any,
search: None,
sort: EpisodeSort::NewestFirst,
limit: None,
offset: 0,
}
}
}
/// An episode with its personal state.
#[derive(Debug, Clone, PartialEq)]
pub struct EpisodeListItem {
/// Episode metadata (feed or synced reference).
pub episode: EpisodeMeta,
/// Progress register, if any.
pub progress: Option<ProgressRecord>,
/// Distinct completed listening sessions (`completed_listen_count`).
pub completed_count: u64,
/// Whether the latest refresh of the feed listed the episode.
pub in_feed: bool,
}
impl EpisodeListItem {
/// Listening status (`NotStarted` without a register).
pub fn status(&self) -> ProgressStatus {
self.progress
.as_ref()
.map_or(ProgressStatus::NotStarted, |progress| progress.status)
}
/// Position ordinary Play resumes from.
pub fn resume_position_ms(&self) -> i64 {
self.progress
.as_ref()
.map_or(0, ProgressRecord::resume_position_ms)
}
}
/// A listening session in history, with display titles when known.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct PodcastListenEntry {
/// The session.
pub session: ListenSession,
/// Episode title from the catalog.
pub episode_title: Option<String>,
/// Podcast title from the catalog.
pub podcast_title: Option<String>,
}
/// A prototype row that could not be imported (kept in its `legacy_*` table).
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct LegacyQuarantineEntry {
/// `legacy_podcasts` or `legacy_podcast_episodes`.
pub legacy_table: String,
/// Row id in that table.
pub legacy_id: i64,
/// Short reason.
pub reason: String,
/// Details (the offending value or error).
pub detail: Option<String>,
}
const EPISODE_COLUMNS: &str = "e.episode_id, e.podcast_id, e.guid, e.title, e.description, \
e.season, e.episode_number, e.episode_type, e.published_at_ms, e.duration_ms, \
e.audio_url, e.mime, e.byte_len, e.image_url, e.link";
const EPISODE_COLUMN_COUNT: usize = 15;
fn episode_from_row(row: &rusqlite::Row<'_>) -> rusqlite::Result<EpisodeMeta> {
Ok(EpisodeMeta {
episode_id: episode_id_col(row.get(0)?)?,
podcast_id: podcast_id_col(row.get(1)?)?,
guid: row.get(2)?,
title: row.get(3)?,
description: row.get(4)?,
season: row.get(5)?,
episode_number: row.get(6)?,
episode_type: EpisodeType::parse(&row.get::<_, String>(7)?),
published_at_ms: row.get(8)?,
duration_ms: row.get(9)?,
audio_url: row.get(10)?,
mime: row.get(11)?,
byte_len: row
.get::<_, Option<i64>>(12)?
.and_then(|len| u64::try_from(len).ok()),
image_url: row.get(13)?,
link: row.get(14)?,
})
}
fn optional_progress(
row: &rusqlite::Row<'_>,
at: usize,
) -> rusqlite::Result<Option<ProgressRecord>> {
if row.get::<_, Option<String>>(at)?.is_none() {
return Ok(None);
}
progress_from_row(row, at).map(Some)
}
pub(crate) fn episode_meta_in(
conn: &Connection,
episode_id: &EpisodeId,
) -> Result<Option<EpisodeMeta>> {
Ok(conn
.query_row(
&format!("SELECT {EPISODE_COLUMNS} FROM episodes e WHERE e.episode_id = ?1"),
[episode_id.as_str()],
episode_from_row,
)
.optional()?)
}
pub(crate) fn podcast_meta_in(
conn: &Connection,
podcast_id: &PodcastId,
) -> Result<Option<PodcastMeta>> {
Ok(conn
.query_row(
"SELECT feed_url, title, author, description, link, image_url
FROM podcasts WHERE podcast_id = ?1",
[podcast_id.as_str()],
|row| {
Ok(PodcastMeta {
feed_url: feed_url_col(row.get(0)?)?,
title: row.get(1)?,
author: row.get(2)?,
description: row.get(3)?,
link: row.get(4)?,
image_url: row.get(5)?,
})
},
)
.optional()?)
}
/// Queue reference of a catalog episode, with `progress_generation` set to
/// the generation of its progress register (handoff hint).
pub(crate) fn episode_ref_in(
conn: &Connection,
episode_id: &EpisodeId,
) -> Result<Option<PodcastEpisodeRef>> {
let Some(episode) = episode_meta_in(conn, episode_id)? else {
return Ok(None);
};
let Some(podcast) = podcast_meta_in(conn, &episode.podcast_id)? else {
return Ok(None);
};
let mut reference = episode.to_ref(&podcast);
reference.progress_generation =
apply::load_progress(conn, episode_id)?.map(|progress| progress.generation);
Ok(Some(reference))
}
const NOT_COMPLETED: &str = "IFNULL(g.status, 'not_started') <> 'completed'";
impl PodcastStore {
/// Podcasts with subscription state and counters, sorted by title.
pub fn list_podcasts(&self, filter: PodcastListFilter) -> Result<Vec<PodcastSummary>> {
let conn = self.lock();
let join = match filter {
PodcastListFilter::Subscribed => {
"JOIN podcast_subscriptions r ON r.podcast_id = p.podcast_id AND r.subscribed = 1"
}
PodcastListFilter::All => "",
};
let ids: Vec<PodcastId> = {
let mut statement = conn.prepare(&format!(
"SELECT p.podcast_id FROM podcasts p {join}
ORDER BY p.title COLLATE NOCASE, p.podcast_id"
))?;
statement
.query_map([], |row| podcast_id_col(row.get(0)?))?
.collect::<rusqlite::Result<Vec<_>>>()?
};
let mut out = Vec::with_capacity(ids.len());
for id in ids {
if let Some(summary) = summary_in(&conn, &id)? {
out.push(summary);
}
}
Ok(out)
}
/// One podcast with subscription state and counters.
pub fn podcast(&self, podcast_id: &PodcastId) -> Result<Option<PodcastSummary>> {
summary_in(&self.lock(), podcast_id)
}
/// The subscription register (with the stored alias set), if any.
pub fn subscription(&self, podcast_id: &PodcastId) -> Result<Option<SubscriptionRecord>> {
apply::load_subscription_register(&self.lock(), podcast_id)
}
/// The share register, if any.
pub fn share(&self, podcast_id: &PodcastId) -> Result<Option<ShareRecord>> {
apply::load_share(&self.lock(), podcast_id)
}
/// The bounded, synced feed alias set of a podcast.
pub fn feed_aliases(&self, podcast_id: &PodcastId) -> Result<Vec<FeedUrl>> {
apply::load_aliases(&self.lock(), podcast_id)
}
/// Season groups of a podcast: numbered seasons ascending, then the
/// group of episodes without a season.
pub fn seasons(&self, podcast_id: &PodcastId) -> Result<Vec<SeasonSummary>> {
let conn = self.lock();
let mut statement = conn.prepare(&format!(
"SELECT s.season, s.episode_count, s.first_published_at_ms, s.last_published_at_ms,
(SELECT COUNT(*) FROM episodes e
LEFT JOIN episode_progress g ON g.episode_id = e.episode_id
WHERE e.podcast_id = s.podcast_id AND e.season IS s.season
AND {NOT_COMPLETED})
FROM podcast_seasons s WHERE s.podcast_id = ?1
ORDER BY s.season IS NULL, s.season"
))?;
let seasons = statement
.query_map([podcast_id.as_str()], |row| {
Ok(SeasonSummary {
season: row.get(0)?,
episode_count: count(row.get(1)?),
first_published_at_ms: row.get(2)?,
last_published_at_ms: row.get(3)?,
unplayed_count: count(row.get(4)?),
})
})?
.collect::<rusqlite::Result<Vec<_>>>()?;
Ok(seasons)
}
/// Episodes of a podcast or season with their progress.
pub fn episodes(&self, query: &EpisodeQuery) -> Result<Vec<EpisodeListItem>> {
let mut conditions = vec!["e.podcast_id = ?".to_string()];
let mut values = vec![Value::Text(query.podcast_id.as_str().to_string())];
match query.season {
SeasonFilter::Any => {}
SeasonFilter::Season(season) => {
conditions.push("e.season = ?".to_string());
values.push(Value::Integer(i64::from(season)));
}
SeasonFilter::NoSeason => conditions.push("e.season IS NULL".to_string()),
}
match query.played {
PlayedFilter::Any => {}
PlayedFilter::Unplayed => conditions.push(NOT_COMPLETED.to_string()),
PlayedFilter::NotStarted => {
conditions.push("IFNULL(g.status, 'not_started') = 'not_started'".to_string());
}
PlayedFilter::InProgress => conditions.push("g.status = 'in_progress'".to_string()),
PlayedFilter::Played => conditions.push("g.status = 'completed'".to_string()),
}
if let Some(search) = query.search.as_deref() {
let needle = music_dht::normalize_name(search);
if !needle.is_empty() {
conditions.push(
"(instr(norm(e.title), ?) > 0 OR instr(norm(e.description), ?) > 0)"
.to_string(),
);
values.push(Value::Text(needle.clone()));
values.push(Value::Text(needle));
}
}
let order = match query.sort {
EpisodeSort::NewestFirst => {
"e.published_at_ms IS NULL, e.published_at_ms DESC, e.episode_id"
}
EpisodeSort::OldestFirst => {
"e.published_at_ms IS NULL, e.published_at_ms, e.episode_id"
}
EpisodeSort::EpisodeNumber => {
"e.season IS NULL, e.season, e.episode_number IS NULL, e.episode_number, \
e.published_at_ms IS NULL, e.published_at_ms, e.episode_id"
}
EpisodeSort::EpisodeNumberDesc => {
"e.season IS NULL, e.season DESC, e.episode_number IS NULL, \
e.episode_number DESC, e.published_at_ms IS NULL, e.published_at_ms DESC, \
e.episode_id"
}
};
values.push(Value::Integer(
query
.limit
.map_or(-1, |limit| i64::try_from(limit).unwrap_or(i64::MAX)),
));
values.push(Value::Integer(
i64::try_from(query.offset).unwrap_or(i64::MAX),
));
let progress_columns = PROGRESS_COLUMNS.replace("p.", "g.");
let sql = format!(
"SELECT {EPISODE_COLUMNS}, e.in_feed, {progress_columns},
(SELECT COUNT(*) FROM podcast_listen_sessions s
WHERE s.episode_id = e.episode_id AND s.completed = 1)
FROM episodes e
LEFT JOIN episode_progress g ON g.episode_id = e.episode_id
WHERE {}
ORDER BY {order}
LIMIT ? OFFSET ?",
conditions.join(" AND ")
);
let conn = self.lock();
let mut statement = conn.prepare(&sql)?;
let items = statement
.query_map(params_from_iter(values), list_item_from_row)?
.collect::<rusqlite::Result<Vec<_>>>()?;
Ok(items)
}
/// One episode with its progress.
pub fn episode(&self, episode_id: &EpisodeId) -> Result<Option<EpisodeListItem>> {
let progress_columns = PROGRESS_COLUMNS.replace("p.", "g.");
Ok(self
.lock()
.query_row(
&format!(
"SELECT {EPISODE_COLUMNS}, e.in_feed, {progress_columns},
(SELECT COUNT(*) FROM podcast_listen_sessions s
WHERE s.episode_id = e.episode_id AND s.completed = 1)
FROM episodes e
LEFT JOIN episode_progress g ON g.episode_id = e.episode_id
WHERE e.episode_id = ?1"
),
[episode_id.as_str()],
list_item_from_row,
)
.optional()?)
}
/// Progress register of an episode (also for episodes the catalog does
/// not know yet).
pub fn progress(&self, episode_id: &EpisodeId) -> Result<Option<ProgressRecord>> {
apply::load_progress(&self.lock(), episode_id)
}
/// Position ordinary Play resumes from (`ProgressRecord::resume_position_ms`, 0 without progress).
pub fn resume_position_ms(&self, episode_id: &EpisodeId) -> Result<i64> {
Ok(self
.progress(episode_id)?
.map_or(0, |progress| progress.resume_position_ms()))
}
/// A queue item for an episode (see [`TrackItem::podcast_episode`]).
pub fn episode_track(&self, episode_id: &EpisodeId) -> Result<Option<TrackItem>> {
Ok(self
.episode_ref(episode_id)?
.map(TrackItem::podcast_episode))
}
/// Listening history (sessions that `should_record`), newest first.
pub fn listen_history(
&self,
podcast_id: Option<&PodcastId>,
limit: usize,
) -> Result<Vec<PodcastListenEntry>> {
let conn = self.lock();
let mut statement = conn.prepare(&format!(
"SELECT {SESSION_COLUMNS}, e.title, p.title
FROM podcast_listen_sessions s
LEFT JOIN episodes e ON e.episode_id = s.episode_id
LEFT JOIN podcasts p ON p.podcast_id = s.podcast_id
WHERE (s.completed = 1 OR s.listened_ms >= ?1)
AND (?2 IS NULL OR s.podcast_id = ?2)
ORDER BY s.ended_at_ms DESC, s.session_id DESC
LIMIT ?3"
))?;
let entries = statement
.query_map(
params![
music_dht::podcast::MIN_RECORDED_EPISODE_LISTEN_MS,
podcast_id.map(PodcastId::as_str),
i64::try_from(limit).unwrap_or(i64::MAX)
],
|row| {
Ok(PodcastListenEntry {
session: session_from_row(row, 0)?,
episode_title: row.get(10)?,
podcast_title: row.get(11)?,
})
},
)?
.collect::<rusqlite::Result<Vec<_>>>()?;
Ok(entries)
}
/// Number of distinct completed sessions of an episode.
pub fn completed_listen_count(&self, episode_id: &EpisodeId) -> Result<u64> {
let total: i64 = self.lock().query_row(
"SELECT COUNT(*) FROM podcast_listen_sessions WHERE episode_id = ?1 AND completed = 1",
[episode_id.as_str()],
|row| row.get(0),
)?;
Ok(count(total))
}
/// Prototype rows that could not be imported.
pub fn legacy_quarantine(&self) -> Result<Vec<LegacyQuarantineEntry>> {
let conn = self.lock();
let mut statement = conn.prepare(
"SELECT legacy_table, legacy_id, reason, detail FROM legacy_quarantine
ORDER BY legacy_table, legacy_id",
)?;
let entries = statement
.query_map([], |row| {
Ok(LegacyQuarantineEntry {
legacy_table: row.get(0)?,
legacy_id: row.get(1)?,
reason: row.get(2)?,
detail: row.get(3)?,
})
})?
.collect::<rusqlite::Result<Vec<_>>>()?;
Ok(entries)
}
}
fn count(value: i64) -> u64 {
u64::try_from(value).unwrap_or(0)
}
fn list_item_from_row(row: &rusqlite::Row<'_>) -> rusqlite::Result<EpisodeListItem> {
Ok(EpisodeListItem {
episode: episode_from_row(row)?,
in_feed: row.get(EPISODE_COLUMN_COUNT)?,
progress: optional_progress(row, EPISODE_COLUMN_COUNT + 1)?,
completed_count: count(row.get(EPISODE_COLUMN_COUNT + 10)?),
})
}
fn summary_in(conn: &Connection, podcast_id: &PodcastId) -> Result<Option<PodcastSummary>> {
let Some(meta) = podcast_meta_in(conn, podcast_id)? else {
return Ok(None);
};
let (episode_count, unplayed_count, in_progress_count, latest, season_count) = conn.query_row(
&format!(
"SELECT COUNT(*),
COUNT(*) FILTER (WHERE {NOT_COMPLETED}),
COUNT(*) FILTER (WHERE g.status = 'in_progress'),
MAX(e.published_at_ms),
(SELECT COUNT(*) FROM podcast_seasons s
WHERE s.podcast_id = ?1 AND s.season IS NOT NULL)
FROM episodes e
LEFT JOIN episode_progress g ON g.episode_id = e.episode_id
WHERE e.podcast_id = ?1"
),
[podcast_id.as_str()],
|row| {
Ok((
row.get::<_, i64>(0)?,
row.get::<_, i64>(1)?,
row.get::<_, i64>(2)?,
row.get::<_, Option<i64>>(3)?,
row.get::<_, i64>(4)?,
))
},
)?;
let subscription = apply::load_subscription_register(conn, podcast_id)?;
let share = apply::load_share(conn, podcast_id)?;
Ok(Some(PodcastSummary {
podcast_id: podcast_id.clone(),
meta,
subscribed: subscription
.as_ref()
.is_some_and(|record| record.subscribed),
shared: effective_share(subscription.as_ref(), share.as_ref()),
episode_count: count(episode_count),
unplayed_count: count(unplayed_count),
in_progress_count: count(in_progress_count),
latest_published_at_ms: latest,
season_count: count(season_count),
}))
}
@@ -0,0 +1,618 @@
//! Feed refresh ingestion and the local-only fetch state.
use std::collections::BTreeSet;
use anyhow::Result;
use music_dht::device_sync::SyncOpPayload;
use music_dht::podcast::feed::ParsedFeed;
use music_dht::podcast::{
EpisodeId, EpisodeIdentityHints, EpisodeMeta, FeedUrl, KnownEpisode, PodcastId,
merge_feed_aliases, resolve_episode_ids,
};
use rusqlite::{Connection, OptionalExtension as _, TransactionBehavior, params};
use super::apply::{self, EpisodeSource};
use super::{PodcastLocalOp, PodcastStore, PodcastStoreError, StampSource, now_ms};
/// A fetched and parsed feed document for [`PodcastStore::apply_feed`].
#[derive(Debug, Clone, Copy)]
pub struct FeedRefresh<'a> {
/// Podcast the feed belongs to.
pub podcast_id: &'a PodcastId,
/// The parsed document (`music_dht::podcast::feed::parse_feed_with_base`
/// with `fetched_from` as base).
pub feed: &'a ParsedFeed,
/// URL the document was finally served from, after redirects.
pub fetched_from: &'a FeedUrl,
/// Whether every redirect leading to `fetched_from` was permanent
/// (301/308). Temporary redirects do not move the feed.
pub moved_permanently: bool,
}
/// Result of [`PodcastStore::apply_feed`].
#[derive(Debug, Clone, Default, PartialEq)]
pub struct RefreshOutcome {
/// Episodes seen for the first time.
pub new_episodes: usize,
/// Known episodes whose metadata changed (title, audio URL, ...).
pub updated_episodes: usize,
/// Known episodes without metadata changes.
pub unchanged_episodes: usize,
/// Items without identity or with invalid metadata.
pub skipped_items: usize,
/// Feed URLs learned from a permanent redirect or `itunes:new-feed-url`.
pub learned_aliases: Vec<FeedUrl>,
/// URL the podcast is fetched from after this refresh.
pub feed_url: Option<FeedUrl>,
/// `podcast_feed_aliases_added`, only when the learned URLs changed the
/// stored alias set. Append it to the sync log.
pub ops: Vec<PodcastLocalOp>,
}
/// Local-only fetch state of a podcast. Never part of sync output.
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct FetchState {
/// HTTP `ETag` validator of the last successful fetch.
pub etag: Option<String>,
/// HTTP `Last-Modified` validator of the last successful fetch.
pub last_modified: Option<String>,
/// Last fetch attempt, unix milliseconds.
pub checked_at_ms: Option<i64>,
/// Last successful fetch (including "not modified").
pub last_success_at_ms: Option<i64>,
/// When the scheduler should check again (`None`: due now).
pub next_check_at_ms: Option<i64>,
/// Error of the last attempt; `None` after a success.
pub last_error: Option<String>,
/// Failed attempts since the last success.
pub consecutive_failures: u32,
/// Image URL the cached cover was downloaded from.
pub cover_url: Option<String>,
/// Local path of the cached cover.
pub cover_path: Option<String>,
}
fn podcast_fetch_url(conn: &Connection, podcast_id: &PodcastId) -> Result<Option<FeedUrl>> {
Ok(conn
.query_row(
"SELECT feed_url FROM podcasts WHERE podcast_id = ?1",
[podcast_id.as_str()],
|row| super::feed_url_col(row.get(0)?),
)
.optional()?)
}
fn fetch_state_in(conn: &Connection, podcast_id: &PodcastId) -> Result<FetchState> {
Ok(conn
.query_row(
"SELECT etag, last_modified, checked_at_ms, last_success_at_ms,
next_check_at_ms, last_error, consecutive_failures, cover_url,
cover_path
FROM podcast_fetch_state WHERE podcast_id = ?1",
[podcast_id.as_str()],
|row| {
Ok(FetchState {
etag: row.get(0)?,
last_modified: row.get(1)?,
checked_at_ms: row.get(2)?,
last_success_at_ms: row.get(3)?,
next_check_at_ms: row.get(4)?,
last_error: row.get(5)?,
consecutive_failures: u32::try_from(row.get::<_, i64>(6)?).unwrap_or(u32::MAX),
cover_url: row.get(7)?,
cover_path: row.get(8)?,
})
},
)
.optional()?
.unwrap_or_default())
}
fn has_personal_state(conn: &Connection, episode_id: &EpisodeId) -> Result<bool> {
Ok(conn.query_row(
"SELECT EXISTS (SELECT 1 FROM episode_progress WHERE episode_id = ?1)
OR EXISTS (SELECT 1 FROM podcast_listen_sessions WHERE episode_id = ?1)",
[episode_id.as_str()],
|row| row.get(0),
)?)
}
/// Re-keys episodes imported from the prototype whose stored `guid` was the
/// enclosure URL (`legacy_guid`) when the feed shows it is a real GUID.
///
/// Only episodes without progress or sessions move, so no personal state is
/// touched; otherwise the GUID item becomes a separate episode. When another
/// episode already holds the GUID id (for example one a remote operation
/// created), the imported episode is a duplicate of it and is deleted.
fn rekey_legacy_guids(conn: &Connection, podcast_id: &PodcastId, feed: &ParsedFeed) -> Result<()> {
for item in &feed.episodes {
let Some(guid) = item.guid.as_deref() else {
continue;
};
let Some(guid_id) = EpisodeId::from_guid(podcast_id, guid) else {
continue;
};
let taken = conn
.query_row(
"SELECT 1 FROM episodes WHERE episode_id = ?1",
[guid_id.as_str()],
|_| Ok(()),
)
.optional()?
.is_some();
let legacy: Option<String> = conn
.query_row(
"SELECT episode_id FROM episodes
WHERE podcast_id = ?1 AND legacy_guid = ?2 ORDER BY episode_id LIMIT 1",
params![podcast_id.as_str(), guid],
|row| row.get(0),
)
.optional()?;
let Some(legacy) = legacy else {
continue;
};
let legacy = super::episode_id_col(legacy)?;
if legacy == guid_id || has_personal_state(conn, &legacy)? {
continue;
}
if taken {
conn.execute(
"DELETE FROM episodes WHERE episode_id = ?1",
[legacy.as_str()],
)?;
continue;
}
conn.execute(
"UPDATE episodes SET episode_id = ?2, guid = ?3, legacy_guid = NULL
WHERE episode_id = ?1",
params![legacy.as_str(), guid_id.as_str(), guid],
)?;
}
Ok(())
}
/// Episodes imported from the prototype without a GUID (`legacy_guid` set)
/// that carry no progress and no session.
///
/// Their ids were resolved over every stored prototype row at once, stale
/// rows included, so they may differ from what a fresh device derives from
/// the current feed. They are provisional until a refresh confirms them.
fn provisional_episode_ids(
conn: &Connection,
podcast_id: &PodcastId,
) -> Result<BTreeSet<EpisodeId>> {
let mut statement = conn.prepare(
"SELECT e.episode_id FROM episodes e
WHERE e.podcast_id = ?1 AND e.legacy_guid IS NOT NULL
AND NOT EXISTS (SELECT 1 FROM episode_progress p WHERE p.episode_id = e.episode_id)
AND NOT EXISTS (SELECT 1 FROM podcast_listen_sessions s
WHERE s.episode_id = e.episode_id)",
)?;
let ids = statement
.query_map([podcast_id.as_str()], |row| {
super::episode_id_col(row.get(0)?)
})?
.collect::<rusqlite::Result<BTreeSet<_>>>()?;
Ok(ids)
}
/// Resolves the ids of the feed items as a device without prototype history
/// would, and moves provisional imported episodes onto them.
///
/// * The ids come from the contract identity rule against every stored
/// episode except the provisional ones (episodes from earlier refreshes,
/// from sync, and imported ones that already carry personal state).
/// * Each provisional episode is matched to the item it was imported from
/// with the same rule (enclosure, then link, then date and title; GUIDs
/// ignored, since the prototype may have stored the enclosure URL as GUID)
/// and re-keyed to that item's id. When another stored episode that stays
/// already holds that id, it is the item's episode and the provisional one
/// is a duplicate: it is deleted. Unmatched provisional episodes stay as
/// they are (not in the feed); the next refresh that lists them re-keys
/// them.
///
/// Provisional episodes have no progress and no session, so no personal
/// state moves. The refresh then clears `legacy_guid` on every episode it
/// lists, which makes their ids final.
fn resolve_and_rekey_provisional(
conn: &Connection,
podcast_id: &PodcastId,
feed: &ParsedFeed,
) -> Result<Vec<Option<EpisodeId>>> {
let known = apply::known_episodes(conn, podcast_id)?;
let provisional = provisional_episode_ids(conn, podcast_id)?;
if provisional.is_empty() {
return Ok(feed.resolve_episode_ids(podcast_id, &known));
}
let (provisional_known, stable_known): (Vec<KnownEpisode>, Vec<KnownEpisode>) = known
.into_iter()
.partition(|episode| provisional.contains(&episode.episode_id));
let ids = feed.resolve_episode_ids(podcast_id, &stable_known);
let guidless: Vec<EpisodeIdentityHints> = feed
.identity_hints()
.into_iter()
.map(|hints| EpisodeIdentityHints {
guid: None,
..hints
})
.collect();
let matches = resolve_episode_ids(
podcast_id,
feed.link.as_deref(),
&guidless,
&provisional_known,
);
let mut moves: Vec<(EpisodeId, EpisodeId)> = matches
.into_iter()
.zip(&ids)
.filter_map(|(matched, target)| {
let matched = matched.filter(|id| provisional.contains(id))?;
let target = target.clone()?;
(matched != target).then_some((matched, target))
})
.collect();
// A target held by an episode that stays (for example a row a remote
// operation created before this first refresh) is the item's episode
// already: the provisional episode is a duplicate of it. It has no
// progress and no session, and its metadata would be overwritten by the
// item anyway, so it is removed instead of kept as a second copy. A
// removed duplicate frees its own id for other moves.
let mut occupied: BTreeSet<EpisodeId> = stable_known
.iter()
.chain(&provisional_known)
.map(|episode| episode.episode_id.clone())
.collect();
let mut duplicates: Vec<EpisodeId> = Vec::new();
loop {
let leaving: BTreeSet<&EpisodeId> = moves.iter().map(|(from, _)| from).collect();
let blocked: BTreeSet<EpisodeId> = moves
.iter()
.filter(|(_, to)| occupied.contains(to) && !leaving.contains(to))
.map(|(from, _)| from.clone())
.collect();
if blocked.is_empty() {
break;
}
moves.retain(|(from, _)| !blocked.contains(from));
for from in blocked {
occupied.remove(&from);
duplicates.push(from);
}
}
for duplicate in &duplicates {
// Guarded again: only a provisional episode without personal state.
conn.execute(
"DELETE FROM episodes
WHERE episode_id = ?1 AND legacy_guid IS NOT NULL
AND NOT EXISTS (SELECT 1 FROM episode_progress p
WHERE p.episode_id = episodes.episode_id)
AND NOT EXISTS (SELECT 1 FROM podcast_listen_sessions s
WHERE s.episode_id = episodes.episode_id)",
[duplicate.as_str()],
)?;
}
// Two phases, so a move may take an id another move vacates.
for (index, (from, _)) in moves.iter().enumerate() {
conn.execute(
"UPDATE episodes SET episode_id = ?2 WHERE episode_id = ?1",
params![from.as_str(), format!("rekey-pending:{index}")],
)?;
}
for (index, (_, to)) in moves.iter().enumerate() {
conn.execute(
"UPDATE episodes SET episode_id = ?2 WHERE episode_id = ?1",
params![format!("rekey-pending:{index}"), to.as_str()],
)?;
}
Ok(ids)
}
fn upsert_feed_episode(conn: &Connection, meta: &EpisodeMeta, now: i64) -> Result<Option<bool>> {
let Some(stored) = super::queries::episode_meta_in(conn, &meta.episode_id)? else {
apply::insert_episode(conn, meta, EpisodeSource::Feed, true, None, now, now)?;
return Ok(None);
};
let changed = stored != *meta;
conn.execute(
"UPDATE episodes
SET podcast_id = ?2, guid = ?3, title = ?4, description = ?5, season = ?6,
episode_number = ?7, episode_type = ?8, published_at_ms = ?9,
duration_ms = ?10, audio_url = ?11, mime = ?12, byte_len = ?13,
image_url = ?14, link = ?15, source = 'feed', in_feed = 1,
legacy_guid = NULL,
updated_at_ms = CASE WHEN ?16 THEN ?17 ELSE updated_at_ms END
WHERE episode_id = ?1",
params![
meta.episode_id.as_str(),
meta.podcast_id.as_str(),
meta.guid,
meta.title,
meta.description,
meta.season,
meta.episode_number,
apply::episode_type_to_sql(meta.episode_type),
meta.published_at_ms,
meta.duration_ms,
meta.audio_url,
meta.mime,
meta.byte_len.and_then(|len| i64::try_from(len).ok()),
meta.image_url,
meta.link,
changed,
now,
],
)?;
Ok(Some(changed))
}
impl PodcastStore {
/// Ingests a fetched feed document for a podcast.
///
/// * Channel metadata of the catalog row is replaced by the feed's.
/// * Episode ids are resolved with the contract identity rule
/// (`ParsedFeed::resolve_episode_ids`) against every stored episode of
/// the podcast, including episodes learned through sync, so a changed
/// title, audio URL or a GUID-less item keeps its id. Known episodes
/// get the new metadata (including the audio URL); new ones are added;
/// episodes missing from the document are kept (`in_feed = false`).
/// * A permanent redirect target and `itunes:new-feed-url` become feed
/// aliases and the new fetch URL. When they change the stored alias
/// set, one `podcast_feed_aliases_added` operation is emitted.
/// * Subscription, share and progress registers and listening sessions
/// are never touched, so a refresh cannot resurrect an unsubscribed
/// podcast or move anyone's position.
///
/// Record the HTTP outcome separately with
/// [`PodcastStore::record_fetch_success`] and friends.
pub fn apply_feed(
&self,
stamps: &mut dyn StampSource,
refresh: &FeedRefresh<'_>,
) -> Result<RefreshOutcome> {
let podcast_id = refresh.podcast_id;
let feed = refresh.feed;
let now = now_ms();
let mut conn = self.lock();
let tx = conn.transaction_with_behavior(TransactionBehavior::Immediate)?;
let current_url = match podcast_fetch_url(&tx, podcast_id)? {
Some(url) => url,
None => {
let Some(register) = apply::load_subscription_register(&tx, podcast_id)? else {
return Err(PodcastStoreError::UnknownPodcast(podcast_id.clone()).into());
};
apply::apply_subscription_record(&tx, &register, now)?;
register.podcast.feed_url
}
};
rekey_legacy_guids(&tx, podcast_id, feed)?;
let ids = resolve_and_rekey_provisional(&tx, podcast_id, feed)?;
tx.execute(
"UPDATE episodes SET in_feed = 0 WHERE podcast_id = ?1",
[podcast_id.as_str()],
)?;
let mut outcome = RefreshOutcome::default();
for (item, id) in feed.episodes.iter().zip(ids) {
let Some(id) = id else {
outcome.skipped_items += 1;
continue;
};
let meta = item.to_meta(podcast_id, id);
if meta.validate().is_err() {
outcome.skipped_items += 1;
continue;
}
match upsert_feed_episode(&tx, &meta, now)? {
None => outcome.new_episodes += 1,
Some(true) => outcome.updated_episodes += 1,
Some(false) => outcome.unchanged_episodes += 1,
}
}
let mut fetch_url = current_url.clone();
let mut learned = BTreeSet::new();
if refresh.moved_permanently && *refresh.fetched_from != fetch_url {
learned.insert(fetch_url.clone());
learned.insert(refresh.fetched_from.clone());
fetch_url = refresh.fetched_from.clone();
}
if let Some(new_url) = feed
.new_feed_url
.as_deref()
.and_then(|url| FeedUrl::parse(url).ok())
&& new_url != fetch_url
{
learned.insert(fetch_url.clone());
learned.insert(new_url.clone());
fetch_url = new_url;
}
let meta = feed.to_podcast_meta(fetch_url.clone());
tx.execute(
"UPDATE podcasts
SET feed_url = ?2, title = ?3, author = ?4, description = ?5, link = ?6,
image_url = ?7, meta_from_feed = 1, updated_at_ms = ?8
WHERE podcast_id = ?1",
params![
podcast_id.as_str(),
meta.feed_url.as_str(),
meta.title,
meta.author,
meta.description,
meta.link,
meta.image_url,
now
],
)?;
if !learned.is_empty() {
let learned: Vec<FeedUrl> = learned.into_iter().collect();
let current = apply::load_aliases(&tx, podcast_id)?;
if merge_feed_aliases(podcast_id, &current, &learned) != current {
let op = super::actions::emit(&tx, stamps, |_| {
SyncOpPayload::PodcastFeedAliasesAdded {
podcast_id: podcast_id.clone(),
feed_aliases: learned.clone(),
}
})?;
outcome.ops.push(op);
}
outcome.learned_aliases = learned;
}
apply::refresh_seasons(&tx, podcast_id)?;
tx.commit()?;
outcome.feed_url = Some(fetch_url);
Ok(outcome)
}
/// Local fetch state of a podcast (defaults when never fetched).
pub fn fetch_state(&self, podcast_id: &PodcastId) -> Result<FetchState> {
fetch_state_in(&self.lock(), podcast_id)
}
/// Read-modify-write of the fetch state under one lock and one immediate
/// transaction, so concurrent `record_*` / `set_cover_cache` calls (from
/// this or another process) never lose each other's columns or failure
/// increments.
fn update_fetch_state(
&self,
podcast_id: &PodcastId,
update: impl FnOnce(&mut FetchState),
) -> Result<FetchState> {
let mut conn = self.lock();
let tx = conn.transaction_with_behavior(TransactionBehavior::Immediate)?;
if podcast_fetch_url(&tx, podcast_id)?.is_none() {
return Err(PodcastStoreError::UnknownPodcast(podcast_id.clone()).into());
}
let mut state = fetch_state_in(&tx, podcast_id)?;
update(&mut state);
tx.execute(
"INSERT INTO podcast_fetch_state
(podcast_id, etag, last_modified, checked_at_ms, last_success_at_ms,
next_check_at_ms, last_error, consecutive_failures, cover_url, cover_path)
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10)
ON CONFLICT(podcast_id) DO UPDATE SET
etag = excluded.etag, last_modified = excluded.last_modified,
checked_at_ms = excluded.checked_at_ms,
last_success_at_ms = excluded.last_success_at_ms,
next_check_at_ms = excluded.next_check_at_ms,
last_error = excluded.last_error,
consecutive_failures = excluded.consecutive_failures,
cover_url = excluded.cover_url, cover_path = excluded.cover_path",
params![
podcast_id.as_str(),
state.etag,
state.last_modified,
state.checked_at_ms,
state.last_success_at_ms,
state.next_check_at_ms,
state.last_error,
i64::from(state.consecutive_failures),
state.cover_url,
state.cover_path,
],
)?;
tx.commit()?;
Ok(state)
}
/// Records a successful fetch (HTTP 200) with its new validators.
pub fn record_fetch_success(
&self,
podcast_id: &PodcastId,
now_ms: i64,
etag: Option<&str>,
last_modified: Option<&str>,
next_check_at_ms: Option<i64>,
) -> Result<()> {
self.update_fetch_state(podcast_id, |state| {
state.etag = etag.map(str::to_string);
state.last_modified = last_modified.map(str::to_string);
state.checked_at_ms = Some(now_ms);
state.last_success_at_ms = Some(now_ms);
state.next_check_at_ms = next_check_at_ms;
state.last_error = None;
state.consecutive_failures = 0;
})
.map(|_| ())
}
/// Records a conditional fetch answered with "304 Not Modified".
pub fn record_fetch_not_modified(
&self,
podcast_id: &PodcastId,
now_ms: i64,
next_check_at_ms: Option<i64>,
) -> Result<()> {
self.update_fetch_state(podcast_id, |state| {
state.checked_at_ms = Some(now_ms);
state.last_success_at_ms = Some(now_ms);
state.next_check_at_ms = next_check_at_ms;
state.last_error = None;
state.consecutive_failures = 0;
})
.map(|_| ())
}
/// Records a failed fetch; returns the consecutive failure count. The
/// stored catalog is kept.
pub fn record_fetch_failure(
&self,
podcast_id: &PodcastId,
now_ms: i64,
error: &str,
next_check_at_ms: Option<i64>,
) -> Result<u32> {
self.update_fetch_state(podcast_id, |state| {
state.checked_at_ms = Some(now_ms);
state.next_check_at_ms = next_check_at_ms;
state.last_error = Some(error.to_string());
state.consecutive_failures = state.consecutive_failures.saturating_add(1);
})
.map(|state| state.consecutive_failures)
}
/// Records the locally cached cover image of a podcast.
pub fn set_cover_cache(
&self,
podcast_id: &PodcastId,
cover_url: Option<&str>,
cover_path: Option<&str>,
) -> Result<()> {
self.update_fetch_state(podcast_id, |state| {
state.cover_url = cover_url.map(str::to_string);
state.cover_path = cover_path.map(str::to_string);
})
.map(|_| ())
}
/// Subscribed podcasts due for a refresh at `now_ms` (never checked or
/// `next_check_at_ms <= now_ms`), most overdue first, with their fetch URL.
pub fn podcasts_due_for_refresh(
&self,
now_ms: i64,
limit: usize,
) -> Result<Vec<(PodcastId, FeedUrl)>> {
let conn = self.lock();
let mut statement = conn.prepare(
"SELECT p.podcast_id, p.feed_url
FROM podcasts p
JOIN podcast_subscriptions s ON s.podcast_id = p.podcast_id AND s.subscribed = 1
LEFT JOIN podcast_fetch_state f ON f.podcast_id = p.podcast_id
WHERE f.next_check_at_ms IS NULL OR f.next_check_at_ms <= ?1
ORDER BY IFNULL(f.next_check_at_ms, 0), p.podcast_id
LIMIT ?2",
)?;
let due = statement
.query_map(
params![now_ms, i64::try_from(limit).unwrap_or(i64::MAX)],
|row| {
Ok((
super::podcast_id_col(row.get(0)?)?,
super::feed_url_col(row.get(1)?)?,
))
},
)?
.collect::<rusqlite::Result<Vec<_>>>()?;
Ok(due)
}
}
@@ -0,0 +1,796 @@
//! Schema, migrations and the import of the prototype database.
use std::collections::{BTreeMap, BTreeSet};
use anyhow::{Context as _, Result};
use music_dht::podcast::{
EpisodeId, EpisodeIdentityHints, EpisodeMeta, EpisodeType, FeedUrl, KnownEpisode,
MAX_DESCRIPTION_CHARS, MAX_TITLE_CHARS, MAX_URL_BYTES, PodcastId, PodcastMeta,
SubscriptionRecord, bounded_text, is_http_url, normalize_episode_title, resolve_episode_ids,
resolve_podcast_id,
};
use rusqlite::{Connection, OptionalExtension as _, Transaction, TransactionBehavior, params};
use super::apply::{self, EpisodeSource};
use super::{LEGACY_IMPORT_OP_PREFIX, PodcastStoreError, now_ms};
/// Newest schema version (`PRAGMA user_version`) this build writes.
pub const PODCAST_SCHEMA_VERSION: i64 = 2;
/// `user_version` of the prototype database of the original podcast PR.
pub const LEGACY_PROTOTYPE_SCHEMA_VERSION: i64 = 1;
/// Migration 1: the prototype schema of the original podcast PR, verbatim
/// (`furumi-tui` `src/podcasts/store.rs`, `MIGRATIONS[0]`). It versioned
/// itself with `PRAGMA user_version = 1` in the same transaction.
const PROTOTYPE_SCHEMA_V1: &str = "
CREATE TABLE IF NOT EXISTS podcasts (
id INTEGER PRIMARY KEY AUTOINCREMENT,
feed_url TEXT NOT NULL UNIQUE,
title TEXT NOT NULL,
author TEXT,
description TEXT,
link TEXT,
image_url TEXT,
cover_url TEXT,
cover_path TEXT,
etag TEXT,
last_modified TEXT,
added_at INTEGER NOT NULL,
last_checked_at INTEGER,
last_error TEXT
);
CREATE TABLE IF NOT EXISTS podcast_episodes (
id INTEGER PRIMARY KEY AUTOINCREMENT,
podcast_id INTEGER NOT NULL REFERENCES podcasts(id) ON DELETE CASCADE,
guid TEXT NOT NULL,
title TEXT NOT NULL,
description TEXT,
published_at INTEGER,
duration_seconds REAL,
audio_url TEXT NOT NULL,
mime_type TEXT,
byte_len INTEGER,
link TEXT,
image_url TEXT,
first_seen_at INTEGER NOT NULL,
UNIQUE (podcast_id, guid)
);
CREATE INDEX IF NOT EXISTS podcast_episodes_by_date
ON podcast_episodes (podcast_id, published_at DESC, id DESC);
";
#[cfg(test)]
pub(super) const PROTOTYPE_SCHEMA_V1_FOR_TESTS: &str = PROTOTYPE_SCHEMA_V1;
/// Columns every prototype `podcasts` table has.
const PROTOTYPE_PODCAST_COLUMNS: &[&str] = &[
"id",
"feed_url",
"title",
"author",
"description",
"link",
"image_url",
"cover_url",
"cover_path",
"etag",
"last_modified",
"added_at",
"last_checked_at",
"last_error",
];
/// Columns every prototype `podcast_episodes` table has.
const PROTOTYPE_EPISODE_COLUMNS: &[&str] = &[
"id",
"podcast_id",
"guid",
"title",
"description",
"published_at",
"duration_seconds",
"audio_url",
"mime_type",
"byte_len",
"link",
"image_url",
"first_seen_at",
];
/// Migration 2: the contract model. See `PODCASTS.md`, "SQLite reference store".
const SCHEMA_V2: &str = "
CREATE TABLE IF NOT EXISTS podcasts (
podcast_id TEXT PRIMARY KEY,
feed_url TEXT NOT NULL,
title TEXT NOT NULL,
author TEXT,
description TEXT,
link TEXT,
image_url TEXT,
meta_from_feed INTEGER NOT NULL DEFAULT 0,
created_at_ms INTEGER NOT NULL,
updated_at_ms INTEGER NOT NULL
);
CREATE TABLE IF NOT EXISTS podcast_feed_aliases (
podcast_id TEXT NOT NULL,
feed_url TEXT NOT NULL,
PRIMARY KEY (podcast_id, feed_url)
);
CREATE INDEX IF NOT EXISTS podcast_feed_aliases_by_url
ON podcast_feed_aliases (feed_url);
CREATE TABLE IF NOT EXISTS episodes (
episode_id TEXT PRIMARY KEY,
podcast_id TEXT NOT NULL REFERENCES podcasts(podcast_id),
guid TEXT,
title TEXT NOT NULL,
description TEXT,
season INTEGER,
episode_number INTEGER,
episode_type TEXT NOT NULL DEFAULT 'full',
published_at_ms INTEGER,
duration_ms INTEGER,
audio_url TEXT NOT NULL,
mime TEXT,
byte_len INTEGER,
image_url TEXT,
link TEXT,
source TEXT NOT NULL CHECK (source IN ('feed', 'sync')),
in_feed INTEGER NOT NULL DEFAULT 0,
legacy_guid TEXT,
first_seen_at_ms INTEGER NOT NULL,
updated_at_ms INTEGER NOT NULL
);
CREATE INDEX IF NOT EXISTS episodes_by_podcast_date
ON episodes (podcast_id, published_at_ms DESC);
CREATE INDEX IF NOT EXISTS episodes_by_podcast_season
ON episodes (podcast_id, season, episode_number);
CREATE TABLE IF NOT EXISTS podcast_seasons (
podcast_id TEXT NOT NULL REFERENCES podcasts(podcast_id),
season INTEGER,
episode_count INTEGER NOT NULL,
first_published_at_ms INTEGER,
last_published_at_ms INTEGER
);
CREATE UNIQUE INDEX IF NOT EXISTS podcast_seasons_key
ON podcast_seasons (podcast_id, IFNULL(season, -1));
CREATE TABLE IF NOT EXISTS podcast_subscriptions (
podcast_id TEXT PRIMARY KEY,
subscribed INTEGER NOT NULL,
podcast_json TEXT NOT NULL,
generation INTEGER NOT NULL CHECK (generation >= 0),
hlc_ms INTEGER NOT NULL,
op_id TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS podcast_shares (
podcast_id TEXT PRIMARY KEY,
shared INTEGER NOT NULL,
generation INTEGER NOT NULL CHECK (generation >= 0),
hlc_ms INTEGER NOT NULL,
op_id TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS episode_progress (
episode_id TEXT PRIMARY KEY,
podcast_id TEXT NOT NULL,
position_ms INTEGER NOT NULL,
status TEXT NOT NULL
CHECK (status IN ('not_started', 'in_progress', 'completed')),
generation INTEGER NOT NULL CHECK (generation >= 0),
session_id TEXT NOT NULL,
hlc_ms INTEGER NOT NULL,
device_id TEXT NOT NULL,
op_id TEXT NOT NULL
);
CREATE INDEX IF NOT EXISTS episode_progress_by_podcast
ON episode_progress (podcast_id);
CREATE TABLE IF NOT EXISTS podcast_listen_sessions (
session_id TEXT PRIMARY KEY,
episode_id TEXT NOT NULL,
podcast_id TEXT NOT NULL,
device_id TEXT NOT NULL,
started_at_ms INTEGER NOT NULL,
ended_at_ms INTEGER NOT NULL,
start_position_ms INTEGER NOT NULL,
end_position_ms INTEGER NOT NULL,
listened_ms INTEGER NOT NULL,
completed INTEGER NOT NULL
);
CREATE INDEX IF NOT EXISTS podcast_listen_sessions_by_episode
ON podcast_listen_sessions (episode_id);
CREATE INDEX IF NOT EXISTS podcast_listen_sessions_by_end
ON podcast_listen_sessions (ended_at_ms DESC);
CREATE TABLE IF NOT EXISTS podcast_fetch_state (
podcast_id TEXT PRIMARY KEY REFERENCES podcasts(podcast_id),
etag TEXT,
last_modified TEXT,
checked_at_ms INTEGER,
last_success_at_ms INTEGER,
next_check_at_ms INTEGER,
last_error TEXT,
consecutive_failures INTEGER NOT NULL DEFAULT 0,
cover_url TEXT,
cover_path TEXT
);
CREATE TABLE IF NOT EXISTS legacy_quarantine (
legacy_table TEXT NOT NULL,
legacy_id INTEGER NOT NULL,
reason TEXT NOT NULL,
detail TEXT,
PRIMARY KEY (legacy_table, legacy_id)
);
";
/// Tables a version-2 store owns (the `legacy_*` tables included).
const V2_TABLES: &[&str] = &[
"podcasts",
"podcast_feed_aliases",
"episodes",
"podcast_seasons",
"podcast_subscriptions",
"podcast_shares",
"episode_progress",
"podcast_listen_sessions",
"podcast_fetch_state",
"legacy_quarantine",
"legacy_podcasts",
"legacy_podcast_episodes",
];
fn user_version(conn: &Connection) -> Result<i64> {
Ok(conn.query_row("PRAGMA user_version", [], |row| row.get(0))?)
}
fn user_tables(conn: &Connection) -> Result<BTreeSet<String>> {
let mut statement = conn.prepare(
"SELECT name FROM sqlite_master WHERE type = 'table' AND name NOT LIKE 'sqlite_%'",
)?;
let names = statement
.query_map([], |row| row.get::<_, String>(0))?
.collect::<rusqlite::Result<BTreeSet<_>>>()?;
Ok(names)
}
pub(crate) fn table_columns(conn: &Connection, table: &str) -> Result<BTreeSet<String>> {
let mut statement = conn.prepare("SELECT name FROM pragma_table_info(?1)")?;
let names = statement
.query_map([table], |row| row.get::<_, String>(0))?
.collect::<rusqlite::Result<BTreeSet<_>>>()?;
Ok(names)
}
fn unrecognized(reason: impl Into<String>) -> anyhow::Error {
PodcastStoreError::UnrecognizedDatabase(reason.into()).into()
}
fn has_columns(conn: &Connection, table: &str, required: &[&str]) -> Result<bool> {
let columns = table_columns(conn, table)?;
Ok(required.iter().all(|column| columns.contains(*column)))
}
/// Checks the prototype tables that exist have the prototype shape.
fn check_prototype_shape(conn: &Connection, tables: &BTreeSet<String>) -> Result<()> {
if tables.contains("podcasts") && !has_columns(conn, "podcasts", PROTOTYPE_PODCAST_COLUMNS)? {
return Err(unrecognized(
"table `podcasts` does not have the prototype columns",
));
}
if tables.contains("podcast_episodes")
&& !has_columns(conn, "podcast_episodes", PROTOTYPE_EPISODE_COLUMNS)?
{
return Err(unrecognized(
"table `podcast_episodes` does not have the prototype columns",
));
}
Ok(())
}
/// Read-only check run before anything is written to the file.
///
/// * `user_version` above [`PODCAST_SCHEMA_VERSION`]: written by a newer
/// Furumi, refused.
/// * 0: must be empty, or hold only the prototype tables (the prototype
/// sets its version in the same transaction, so this is defensive).
/// * 1: the prototype database; its tables must have the prototype shape.
/// * 2: the current schema.
///
/// Any other table means the path points at a different database (for
/// example `library.db`), which is refused instead of being modified.
pub(super) fn check_openable(conn: &Connection) -> Result<()> {
let version = user_version(conn)?;
if version > PODCAST_SCHEMA_VERSION {
return Err(PodcastStoreError::NewerSchema {
found: version,
supported: PODCAST_SCHEMA_VERSION,
}
.into());
}
if version < 0 {
return Err(unrecognized(format!("negative user_version {version}")));
}
let tables = user_tables(conn)?;
let allowed: &[&str] = if version < PODCAST_SCHEMA_VERSION {
&["podcasts", "podcast_episodes"]
} else {
V2_TABLES
};
if let Some(table) = tables
.iter()
.find(|table| !allowed.contains(&table.as_str()))
{
return Err(unrecognized(format!(
"unexpected table `{table}` at schema version {version}"
)));
}
if version < PODCAST_SCHEMA_VERSION {
check_prototype_shape(conn, &tables)?;
} else if tables.contains("podcasts")
&& !has_columns(conn, "podcasts", &["podcast_id", "feed_url"])?
{
return Err(unrecognized("table `podcasts` is not the version 2 table"));
}
Ok(())
}
/// Applies every pending migration, each in its own immediate transaction
/// that re-reads the version (another process may have migrated meanwhile).
pub(super) fn migrate(conn: &mut Connection) -> Result<()> {
for target in 1..=PODCAST_SCHEMA_VERSION {
let tx = conn.transaction_with_behavior(TransactionBehavior::Immediate)?;
let version = user_version(&tx)?;
if version > PODCAST_SCHEMA_VERSION {
return Err(PodcastStoreError::NewerSchema {
found: version,
supported: PODCAST_SCHEMA_VERSION,
}
.into());
}
if version >= target {
continue;
}
match target {
1 => tx
.execute_batch(PROTOTYPE_SCHEMA_V1)
.context("creating the prototype podcast schema (v1)")?,
2 => migrate_to_v2(&tx).context("migrating the podcast database to v2")?,
_ => unreachable!("no migration for version {target}"),
}
tx.pragma_update(None, "user_version", target)?;
tx.commit()
.with_context(|| format!("committing podcast schema v{target}"))?;
}
Ok(())
}
fn migrate_to_v2(tx: &Transaction<'_>) -> Result<()> {
let tables = user_tables(tx)?;
// Rename, never drop: the prototype data stays available as `legacy_*`.
// Renaming `podcasts` also rewrites the foreign key of the episode table.
if tables.contains("podcasts") && !tables.contains("legacy_podcasts") {
tx.execute_batch("ALTER TABLE podcasts RENAME TO legacy_podcasts")?;
}
if tables.contains("podcast_episodes") && !tables.contains("legacy_podcast_episodes") {
tx.execute_batch("ALTER TABLE podcast_episodes RENAME TO legacy_podcast_episodes")?;
}
tx.execute_batch(SCHEMA_V2)?;
let tables = user_tables(tx)?;
if tables.contains("legacy_podcasts") {
import_prototype(tx, tables.contains("legacy_podcast_episodes"))?;
}
Ok(())
}
struct LegacyPodcast {
id: i64,
feed_url: String,
title: String,
author: Option<String>,
description: Option<String>,
link: Option<String>,
image_url: Option<String>,
cover_url: Option<String>,
cover_path: Option<String>,
etag: Option<String>,
last_modified: Option<String>,
added_at: i64,
last_checked_at: Option<i64>,
last_error: Option<String>,
}
struct LegacyEpisode {
id: i64,
guid: String,
title: String,
description: Option<String>,
published_at: Option<i64>,
duration_seconds: Option<f64>,
audio_url: String,
mime_type: Option<String>,
byte_len: Option<i64>,
link: Option<String>,
image_url: Option<String>,
first_seen_at: i64,
}
fn quarantine(
tx: &Transaction<'_>,
table: &str,
id: i64,
reason: &str,
detail: &str,
) -> Result<()> {
tx.execute(
"INSERT OR IGNORE INTO legacy_quarantine (legacy_table, legacy_id, reason, detail)
VALUES (?1, ?2, ?3, ?4)",
params![table, id, reason, detail],
)?;
Ok(())
}
fn secs_to_ms(secs: i64) -> i64 {
secs.saturating_mul(1_000)
}
/// Imports the prototype tables into the contract model.
///
/// * Every prototype row is an active subscription: it becomes a
/// subscription register with `subscribed = true`, generation 0,
/// `hlc_ms` = `added_at` and `op_id` = [`LEGACY_IMPORT_OP_PREFIX`] + id,
/// so any later contract write orders after it. Other devices learn it
/// through the snapshot sections. Nothing is shared (private by default).
/// * The podcast id is derived from the canonical feed URL
/// ([`FeedUrl::parse`], falling back to [`FeedUrl::from_user_input`]); rows
/// whose URL cannot be canonicalized are recorded in `legacy_quarantine`
/// and stay in `legacy_podcasts` with their episodes. Rows with the same
/// canonical URL merge into one podcast (the first row's subscription and
/// fetch state win; the episodes of all of them are imported).
/// * Episodes get ids from the contract identity rule over the stored rows.
/// The prototype stored the enclosure URL as `guid` when the item had
/// none: such rows are imported as GUID-less items and keep the stored
/// value in `legacy_guid`. Rows of one GUID-less item seen under several
/// enclosure URLs are collapsed into the newest (the others are
/// quarantined as duplicates). Because stale rows can still change which
/// key is unique, these ids are provisional: the first refresh that lists
/// the item re-keys an episode without progress or sessions to the id the
/// current feed gives (see `refresh.rs`), including its GUID id when the
/// stored value turns out to be a real GUID.
/// * ETag, Last-Modified, check time, last error and the cover cache move to
/// the local fetch state. The prototype has no progress or history, so no
/// progress register or session is created.
fn import_prototype(tx: &Transaction<'_>, with_episodes: bool) -> Result<()> {
let now = now_ms();
let legacy: Vec<LegacyPodcast> = {
let mut statement = tx.prepare(
"SELECT id, feed_url, title, author, description, link, image_url, cover_url,
cover_path, etag, last_modified, added_at, last_checked_at, last_error
FROM legacy_podcasts ORDER BY id",
)?;
statement
.query_map([], |row| {
Ok(LegacyPodcast {
id: row.get(0)?,
feed_url: row.get(1)?,
title: row.get(2)?,
author: row.get(3)?,
description: row.get(4)?,
link: row.get(5)?,
image_url: row.get(6)?,
cover_url: row.get(7)?,
cover_path: row.get(8)?,
etag: row.get(9)?,
last_modified: row.get(10)?,
added_at: row.get(11)?,
last_checked_at: row.get(12)?,
last_error: row.get(13)?,
})
})?
.collect::<rusqlite::Result<Vec<_>>>()?
};
let mut known_urls: Vec<(PodcastId, FeedUrl)> = Vec::new();
for podcast in legacy {
let url = match FeedUrl::parse(&podcast.feed_url)
.or_else(|_| FeedUrl::from_user_input(&podcast.feed_url))
{
Ok(url) => url,
Err(error) => {
quarantine(
tx,
"legacy_podcasts",
podcast.id,
"invalid feed URL",
&error.to_string(),
)?;
continue;
}
};
let podcast_id = resolve_podcast_id(&url, known_urls.iter().map(|(id, url)| (id, url)));
let duplicate = known_urls.iter().any(|(id, _)| *id == podcast_id);
known_urls.push((podcast_id.clone(), url.clone()));
if duplicate {
// Same canonical feed as an earlier row: its episodes join that
// podcast; the first row's subscription and fetch state are kept.
if with_episodes {
import_prototype_episodes(tx, &podcast, &podcast_id, now)?;
}
apply::refresh_seasons(tx, &podcast_id)?;
continue;
}
let meta = PodcastMeta {
feed_url: url.clone(),
title: podcast.title.clone(),
author: podcast.author.clone(),
description: podcast.description.clone(),
link: podcast.link.clone(),
image_url: podcast.image_url.clone(),
}
.bounded();
let added_ms = secs_to_ms(podcast.added_at);
let record = SubscriptionRecord {
podcast_id: podcast_id.clone(),
subscribed: true,
podcast: meta.clone(),
feed_aliases: vec![url.clone()],
generation: 0,
hlc_ms: added_ms,
op_id: format!("{LEGACY_IMPORT_OP_PREFIX}{podcast_id}"),
};
apply::apply_subscription_record(tx, &record, now)?;
// The prototype metadata came from the feed itself.
tx.execute(
"UPDATE podcasts
SET title = ?2, author = ?3, description = ?4, link = ?5, image_url = ?6,
meta_from_feed = 1, created_at_ms = MIN(created_at_ms, ?7)
WHERE podcast_id = ?1 AND meta_from_feed = 0",
params![
podcast_id.as_str(),
meta.title,
meta.author,
meta.description,
meta.link,
meta.image_url,
added_ms
],
)?;
tx.execute(
"INSERT OR IGNORE INTO podcast_fetch_state
(podcast_id, etag, last_modified, checked_at_ms, last_success_at_ms,
next_check_at_ms, last_error, consecutive_failures, cover_url, cover_path)
VALUES (?1, ?2, ?3, ?4, ?5, NULL, ?6, ?7, ?8, ?9)",
params![
podcast_id.as_str(),
podcast.etag,
podcast.last_modified,
podcast.last_checked_at.map(secs_to_ms),
podcast
.last_error
.is_none()
.then(|| podcast.last_checked_at.map(secs_to_ms))
.flatten(),
podcast.last_error,
i64::from(podcast.last_error.is_some()),
podcast.cover_url,
podcast.cover_path,
],
)?;
if with_episodes {
import_prototype_episodes(tx, &podcast, &podcast_id, now)?;
}
apply::refresh_seasons(tx, &podcast_id)?;
}
Ok(())
}
fn import_prototype_episodes(
tx: &Transaction<'_>,
podcast: &LegacyPodcast,
podcast_id: &PodcastId,
now: i64,
) -> Result<()> {
let rows: Vec<LegacyEpisode> = {
let mut statement = tx.prepare(
"SELECT id, guid, title, description, published_at, duration_seconds, audio_url,
mime_type, byte_len, link, image_url, first_seen_at
FROM legacy_podcast_episodes WHERE podcast_id = ?1 ORDER BY id",
)?;
statement
.query_map([podcast.id], |row| {
Ok(LegacyEpisode {
id: row.get(0)?,
guid: row.get(1)?,
title: row.get(2)?,
description: row.get(3)?,
published_at: row.get(4)?,
duration_seconds: row.get(5)?,
audio_url: row.get(6)?,
mime_type: row.get(7)?,
byte_len: row.get(8)?,
link: row.get(9)?,
image_url: row.get(10)?,
first_seen_at: row.get(11)?,
})
})?
.collect::<rusqlite::Result<Vec<_>>>()?
};
let mut usable = Vec::new();
for row in rows {
if !is_http_url(&row.audio_url) {
quarantine(
tx,
"legacy_podcast_episodes",
row.id,
"invalid audio URL",
&row.audio_url,
)?;
continue;
}
usable.push(row);
}
// The prototype used the enclosure URL as GUID when the item had none.
let real_guid = |row: &LegacyEpisode| {
let guid = row.guid.trim();
(!guid.is_empty() && guid != row.audio_url.trim()).then(|| guid.to_string())
};
let usable = collapse_moved_guidless_rows(tx, usable, real_guid)?;
let hints: Vec<EpisodeIdentityHints> = usable
.iter()
.map(|row| EpisodeIdentityHints {
guid: real_guid(row),
enclosure_url: Some(row.audio_url.clone()),
link: row.link.clone(),
published_at_ms: row.published_at.map(secs_to_ms),
title: Some(row.title.clone()),
})
.collect();
let known: Vec<KnownEpisode> = apply::known_episodes(tx, podcast_id)?;
let ids = resolve_episode_ids(podcast_id, podcast.link.as_deref(), &hints, &known);
let mut seen: BTreeMap<EpisodeId, i64> = BTreeMap::new();
for ((row, hint), id) in usable.iter().zip(&hints).zip(ids) {
let Some(episode_id) = id else {
quarantine(
tx,
"legacy_podcast_episodes",
row.id,
"episode cannot be identified",
"no usable identity key or duplicate item",
)?;
continue;
};
if let Some(first) = seen.insert(episode_id.clone(), row.id) {
quarantine(
tx,
"legacy_podcast_episodes",
row.id,
"duplicate episode",
&format!("same identity as legacy episode {first}"),
)?;
continue;
}
let meta = EpisodeMeta {
episode_id,
podcast_id: podcast_id.clone(),
guid: hint.guid.clone(),
title: bounded_title(&row.title),
description: row
.description
.as_deref()
.map(str::trim)
.filter(|text| !text.is_empty())
.map(|text| bounded_text(text, MAX_DESCRIPTION_CHARS)),
season: None,
episode_number: None,
episode_type: EpisodeType::Full,
published_at_ms: row.published_at.map(secs_to_ms),
duration_ms: row
.duration_seconds
.filter(|secs| secs.is_finite() && *secs > 0.0)
.map(|secs| (secs * 1_000.0).round() as i64),
audio_url: row.audio_url.clone(),
mime: row.mime_type.clone().filter(|mime| {
!mime.is_empty() && mime.len() <= 128 && mime.bytes().all(|b| b.is_ascii_graphic())
}),
byte_len: row.byte_len.and_then(|len| u64::try_from(len).ok()),
image_url: row.image_url.clone().filter(|url| is_http_url(url)),
link: row.link.clone().filter(|link| link.len() <= MAX_URL_BYTES),
};
if let Err(error) = meta.validate() {
quarantine(
tx,
"legacy_podcast_episodes",
row.id,
"invalid episode metadata",
&error.to_string(),
)?;
continue;
}
let legacy_guid = hint.guid.is_none().then(|| row.guid.clone());
let exists: bool = tx
.query_row(
"SELECT 1 FROM episodes WHERE episode_id = ?1",
[meta.episode_id.as_str()],
|_| Ok(true),
)
.optional()?
.unwrap_or(false);
if exists {
continue;
}
apply::insert_episode(
tx,
&meta,
EpisodeSource::Feed,
true,
legacy_guid.as_deref(),
secs_to_ms(row.first_seen_at),
now,
)?;
}
Ok(())
}
/// Collapses prototype rows that are one GUID-less item seen under several
/// enclosure URLs.
///
/// The prototype keyed GUID-less items by their enclosure URL, so an item
/// whose audio URL changed (new CDN, tracking prefix) left one row per URL.
/// Those stale rows are not part of the current feed; resolved together with
/// it they would make the item's `<link>` and `(date, title)` look
/// non-unique and give it an id a fresh device never derives. Rows with the
/// same publication second, normalized title and `<link>` (or none) are the
/// same item: the newest (`first_seen_at`, then `id`) is imported, the others
/// are quarantined as duplicates. The link must match too, so distinct
/// episodes that only share a title and second are never merged.
fn collapse_moved_guidless_rows(
tx: &Transaction<'_>,
rows: Vec<LegacyEpisode>,
real_guid: impl Fn(&LegacyEpisode) -> Option<String>,
) -> Result<Vec<LegacyEpisode>> {
type ItemKey = (i64, String, Option<String>);
let key = |row: &LegacyEpisode| -> Option<ItemKey> {
if real_guid(row).is_some() {
return None;
}
let title = normalize_episode_title(&row.title);
let link = row
.link
.as_deref()
.map(str::trim)
.filter(|link| !link.is_empty())
.map(str::to_string);
(!title.is_empty()).then_some((row.published_at?, title, link))
};
let mut newest: BTreeMap<ItemKey, (i64, i64)> = BTreeMap::new();
for row in &rows {
if let Some(key) = key(row) {
let candidate = (row.first_seen_at, row.id);
newest
.entry(key)
.and_modify(|best| *best = (*best).max(candidate))
.or_insert(candidate);
}
}
let mut kept = Vec::with_capacity(rows.len());
for row in rows {
let keeper = key(&row).and_then(|key| newest.get(&key).map(|(_, id)| *id));
match keeper {
Some(keeper) if keeper != row.id => quarantine(
tx,
"legacy_podcast_episodes",
row.id,
"duplicate episode",
&format!(
"older enclosure URL of the GUID-less item imported from legacy episode {keeper}"
),
)?,
_ => kept.push(row),
}
}
Ok(kept)
}
fn bounded_title(title: &str) -> String {
let title = title.trim();
if title.is_empty() {
"Untitled episode".to_string()
} else {
bounded_text(title, MAX_TITLE_CHARS)
}
}
File diff suppressed because it is too large. Load diff
+8 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "music-dht"
version = "0.5.0"
version = "0.6.0"
description = "Distributed music library search: a Kademlia-style DHT on top of federation-net"
readme = "README.md"
documentation = "https://docs.rs/music-dht"
@@ -26,6 +26,13 @@ thiserror = { workspace = true }
tracing = { workspace = true }
rand = { workspace = true }
data-encoding = { workspace = true }
quick-xml = { workspace = true, optional = true }
encoding_rs = { workspace = true, optional = true }
url = { workspace = true, optional = true }
[features]
# RSS 2.0 / RSS 1.0 / Atom podcast feed parser (`music_dht::podcast::feed`).
podcast-feed = ["dep:quick-xml", "dep:encoding_rs", "dep:url"]
[dev-dependencies]
anyhow = { workspace = true }
+59 -1
View File
@@ -185,7 +185,9 @@ new clients should accept both ALPNs and only send `ListenRecorded` when
`supports_listen_history(peer.protocol_version)` returns true. Likes,
playlists and membership remain compatible with v1. Profile version 3 adds
versioned personal playback coordination over the existing JSON sync stream;
legacy unfenced playback messages are ignored by upgraded clients.
legacy unfenced playback messages are ignored by upgraded clients. Profile
version 4 adds podcasts (see below); `op_allowed_for_peer` applies all of
these rules to one operation.
`music_dht::playback` provides the shared pure ownership engine and generic
snapshot/command envelopes. Configure startup and failover with `Config`, feed
@@ -193,12 +195,68 @@ actual output reports, and persist `Checkpoint` before applying decisions.
`Config::passive()` is intended for always-running gateways. See the workspace
`PLAYBACK_PROTOCOL.md` for invariants, configuration and migration details.
## Podcasts
`music_dht::podcast` is the shared, pure podcast contract (no SQLite, HTTP or
clock). Clients use it for identity, merge rules and wire DTOs; see the
workspace `PODCASTS.md` for the normative rules.
```rust
use music_dht::podcast::{
EpisodeIdentityHints, FeedUrl, PodcastId, ProgressRecord, resolve_episode_ids,
};
let url = FeedUrl::from_user_input("feed://example.com/rss?token=abc")?;
let podcast = PodcastId::from_feed_url(&url); // "pod1:..."
let hints = [EpisodeIdentityHints {
guid: Some("episode-1".into()),
..Default::default()
}];
let ids = resolve_episode_ids(&podcast, None, &hints, &[]); // "ep1:..."
// Explicit actions (seek, start over, new session) bump the generation.
let generation = ProgressRecord::explicit_generation(Some(4)); // 5
```
* `FeedUrl` canonicalizes feed URLs and keeps every query parameter.
* `PodcastId`, `EpisodeId`, `EpisodeKey`, `resolve_podcast_id`,
`merge_feed_aliases`, `resolve_episode_ids`: identity, aliases and the
no-GUID rule.
* `PodcastMeta`, `EpisodeMeta`, `PodcastEpisodeRef`: bounded metadata and the
self-contained queue item.
* `SubscriptionRecord`, `ShareRecord`, `ProgressRecord`, `ListenSession` with
`merge_subscription`, `merge_share`, `merge_progress`,
`merge_listen_session`, `next_podcast_generation`, `effective_share`,
`reaches_completion` and `completed_listen_count`.
Enable the `podcast-feed` feature for `music_dht::podcast::feed`, the shared
tolerant RSS 2.0 / RSS 1.0 / Atom parser:
```toml
music-dht = { version = "0.6", features = ["podcast-feed"] }
```
Device-sync protocol v4 adds `podcast_subscription_set`, `podcast_share_set`,
`podcast_feed_aliases_added`, `podcast_progress_set` and
`podcast_listen_recorded`, podcast snapshot sections, `PlaybackTrack.source`
and a reserved `playback_rate`. Old clients cannot parse the new operation
kinds and ignore `source`, so always filter with `op_allowed_for_peer`,
`playback_allowed_for_peer`, `SyncSnapshot::for_peer` and
`playback_command_allowed_from_peer`, using the protocol version each device
reported itself (`profile_protocol_version`; relayed profiles go through
`relayed_protocol_version`, pairing requests use
`InviteWire::inviter_protocol_version`). Bound snapshots with
`SyncSnapshot::bound_podcast_state_to(podcast_snapshot_budget(..))`. Jam hosts use
`jam::redact_jam_snapshot`, `jam_command_allowed` and
`restore_jam_private_items`.
## Verification
```bash
cargo fmt --check
cargo clippy --workspace --all-targets --all-features -- -D warnings
cargo test -p music-dht
cargo test -p music-dht --features podcast-feed
```
The integration test starts three real nodes in one process (they use Iroh's
File diff suppressed because it is too large. Load diff
+65
View File
@@ -3,6 +3,12 @@
//! Jam is separate from personal-device sync. Possession of a [`JamInvite`]
//! capability authorizes playback control for one host process, but never
//! grants likes, playlists, history, or trusted-device membership.
//!
//! Podcast episodes are private. Hosts send [`redact_jam_snapshot`] to
//! participants, reject participant commands for which
//! [`crate::device_sync::jam_command_allowed`] is false, and map placeholders
//! back with [`crate::device_sync::restore_jam_private_items`] before applying
//! a participant's state.
use serde::{Deserialize, Serialize};
use tokio::io::{AsyncRead, AsyncReadExt};
@@ -140,6 +146,17 @@ pub enum JamWireMessage {
},
}
/// Returns a playback snapshot safe to send to Jam participants.
///
/// Podcast episodes become opaque placeholders (see
/// [`crate::device_sync::redact_for_jam`]) and the personal coordination
/// envelope is removed, since it belongs to the trusted-device scope.
pub fn redact_jam_snapshot(mut snapshot: PlaybackSnapshot) -> PlaybackSnapshot {
snapshot.state = crate::device_sync::redact_for_jam(snapshot.state);
snapshot.coordination = None;
snapshot
}
/// Reads a bounded Jam message from an async reader.
pub async fn read_message_from<R: AsyncRead + Unpin>(reader: &mut R) -> Result<JamWireMessage> {
let mut line = Vec::new();
@@ -277,6 +294,54 @@ mod tests {
assert!(JamInvite::from_uri("frid://i/abcd").is_err());
}
#[test]
fn jam_snapshots_hide_podcast_episodes() {
use crate::device_sync::{PlaybackRepeat, PlaybackStateWire, PlaybackTrack};
use crate::podcast::{EpisodeId, FeedUrl, PodcastEpisodeRef, PodcastId};
let feed = FeedUrl::parse("https://private.test/feed").unwrap();
let podcast_id = PodcastId::from_feed_url(&feed);
let episode = PodcastEpisodeRef {
episode_id: EpisodeId::from_guid(&podcast_id, "e").unwrap(),
podcast_id,
feed_url: feed,
podcast_title: "Hidden show".to_string(),
episode_title: "Hidden episode".to_string(),
audio_url: "https://private.test/e.mp3".to_string(),
mime: None,
duration_ms: None,
published_at_ms: None,
season: None,
episode_number: None,
image_url: None,
progress_generation: Some(7),
};
let snapshot = PlaybackSnapshot {
device_id: "host".to_string(),
device_name: "Host".to_string(),
active: true,
updated_at_ms: 1,
state: PlaybackStateWire {
queue: vec![PlaybackTrack::podcast_episode(episode)],
queue_pos: 0,
playing: true,
paused: false,
idle_since_ms: None,
position_secs: 1.0,
volume: 50,
shuffle: false,
repeat: PlaybackRepeat::Off,
playback_rate: None,
},
coordination: None,
};
let redacted = redact_jam_snapshot(snapshot);
let json = serde_json::to_string(&redacted).unwrap();
assert!(!json.contains("Hidden"));
assert!(!json.contains("private.test"));
assert!(redacted.state.queue[0].is_private_placeholder());
}
#[test]
fn weak_capability_is_rejected() {
let mut invite = invite();
+5
View File
@@ -23,6 +23,10 @@
//! * [`similarity_lsh`] turns those embeddings into deterministic compact
//! routing summaries, while [`similarity_dht`] stores and discovers signed
//! peer summaries without owning model inference or an embedding database.
//! * [`podcast`] is the pure, shared podcast contract (canonical feed URLs,
//! podcast/episode ids, metadata DTOs and convergent merge rules), with an
//! optional RSS/Atom parser behind the `podcast-feed` feature;
//! [`device_sync`] protocol v4 carries it between trusted devices.
//! * Records are replicated to the `K` nodes whose ids are XOR-closest to
//! each key. Publishers pick the targets from their routing table and send
//! batched store requests (one pipeline per peer), so even a large library
@@ -92,6 +96,7 @@ pub mod jam;
mod message;
mod node;
mod normalization;
pub mod podcast;
mod record;
mod request;
mod routing;
File diff suppressed because it is too large. Load diff
+731
View File
@@ -0,0 +1,731 @@
use super::*;
const ITUNES_NS: &str = "http://www.itunes.com/dtds/podcast-1.0.dtd";
fn rss(channel_extra: &str, items: &str) -> String {
format!(
r#"<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:itunes="{ITUNES_NS}" xmlns:content="http://purl.org/rss/1.0/modules/content/">
<channel>
<title>Test Cast</title>
<link>https://example.test/cast</link>
{channel_extra}
{items}
</channel>
</rss>"#
)
}
fn item(guid: &str, title: &str, date: &str) -> String {
format!(
r#"<item>
<title>{title}</title>
<guid isPermaLink="false">{guid}</guid>
<pubDate>{date}</pubDate>
<enclosure url="https://cdn.example.test/{guid}.mp3" length="1234" type="audio/mpeg"/>
</item>"#
)
}
#[test]
fn rss_with_itunes_tags_cdata_and_fallbacks() {
let xml = rss(
r#"<description><![CDATA[<p>Weekly <b>talk</b> show.</p><script>evil()</script>]]></description>
<itunes:author>Jane Host</itunes:author>
<itunes:image href="https://example.test/cover.jpg"/>
<image><url>https://example.test/small.png</url><title>ignored</title></image>"#,
r#"<item>
<title> Episode one </title>
<itunes:title>One</itunes:title>
<description>&lt;p&gt;First &amp;amp; best&lt;/p&gt;</description>
<guid>ep-1</guid>
<pubDate>Tue, 02 Jan 2024 10:00:00 GMT</pubDate>
<itunes:duration>1:02:03</itunes:duration>
<itunes:image href="https://example.test/ep1.jpg"/>
<link>https://example.test/ep1</link>
<enclosure url="https://cdn.example.test/1.mp3" length="5000" type="audio/mpeg"/>
</item>
<item>
<title></title>
<itunes:title>Only itunes title</itunes:title>
<content:encoded><![CDATA[<div>Long <i>show</i> notes</div>]]></content:encoded>
<guid>ep-2</guid>
<itunes:duration>754</itunes:duration>
<enclosure url="https://cdn.example.test/2.m4a" type="audio/x-m4a"/>
</item>
<item>
<itunes:summary>Summary only</itunes:summary>
<guid>ep-3</guid>
<enclosure url="https://cdn.example.test/3.mp3"/>
</item>"#,
);
let feed = parse_feed(xml.as_bytes()).unwrap();
assert_eq!(feed.title, "Test Cast");
assert_eq!(feed.author.as_deref(), Some("Jane Host"));
assert_eq!(feed.description.as_deref(), Some("Weekly talk show."));
assert_eq!(feed.link.as_deref(), Some("https://example.test/cast"));
assert_eq!(
feed.image_url.as_deref(),
Some("https://example.test/cover.jpg")
);
assert_eq!(feed.episodes.len(), 3);
let first = &feed.episodes[0];
assert_eq!(first.guid.as_deref(), Some("ep-1"));
assert_eq!(first.title, "Episode one");
assert_eq!(first.description.as_deref(), Some("First & best"));
assert_eq!(first.published_at_ms, Some(1_704_189_600_000));
assert_eq!(first.duration_ms, Some(3_723_000));
assert_eq!(first.audio_url, "https://cdn.example.test/1.mp3");
assert_eq!(first.mime_type.as_deref(), Some("audio/mpeg"));
assert_eq!(first.byte_len, Some(5000));
assert_eq!(first.link.as_deref(), Some("https://example.test/ep1"));
assert_eq!(
first.image_url.as_deref(),
Some("https://example.test/ep1.jpg")
);
let second = &feed.episodes[1];
assert_eq!(second.title, "Only itunes title");
assert_eq!(second.description.as_deref(), Some("Long show notes"));
assert_eq!(second.duration_ms, Some(754_000));
assert_eq!(second.byte_len, None);
assert_eq!(second.published_at_ms, None);
let third = &feed.episodes[2];
assert_eq!(third.title, "Untitled episode");
assert_eq!(third.description.as_deref(), Some("Summary only"));
assert_eq!(third.mime_type, None);
}
#[test]
fn entities_numeric_refs_and_unknown_html_entities() {
let xml = rss(
"",
r#"<item>
<title>A&nbsp;&amp;&nbsp;B &#8212; &#x41;&#1055; &mdash; &bogus; R&amp;D &copy;</title>
<description>x &lt; y &amp;&amp; z</description>
<guid>e</guid>
<enclosure url="https://cdn.example.test/a.mp3?x=1&amp;y=2" type="audio/mpeg"/>
</item>"#,
);
let feed = parse_feed(xml.as_bytes()).unwrap();
let episode = &feed.episodes[0];
assert_eq!(episode.title, "A & B — AП — &bogus; R&D ©");
assert_eq!(episode.description.as_deref(), Some("x < y && z"));
assert_eq!(episode.audio_url, "https://cdn.example.test/a.mp3?x=1&y=2");
}
#[test]
fn atom_feed() {
let xml = r#"<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title type="text">Atom Cast</title>
<subtitle>About &lt;b&gt;things&lt;/b&gt;</subtitle>
<link rel="self" href="https://example.test/atom.xml"/>
<link rel="alternate" href="https://example.test/"/>
<logo>https://example.test/logo.png</logo>
<author><name>Atom Author</name></author>
<entry>
<title>Entry one</title>
<id>urn:uuid:1</id>
<published>2024-03-01T12:30:00+02:00</published>
<updated>2024-03-05T00:00:00Z</updated>
<summary type="html">&lt;p&gt;Entry summary&lt;/p&gt;</summary>
<link rel="alternate" href="https://example.test/e1"/>
<link rel="enclosure" href="https://cdn.example.test/e1.ogg" type="audio/ogg" length="42"/>
</entry>
<entry>
<title>Entry two</title>
<id>urn:uuid:2</id>
<updated>2024-03-06T00:00:00Z</updated>
<content type="xhtml"><div xmlns="http://www.w3.org/1999/xhtml"><p>Inline</p><p>xhtml</p></div></content>
<link rel="enclosure" href="https://cdn.example.test/e2.mp3"/>
</entry>
<entry>
<title>No audio</title>
<id>urn:uuid:3</id>
<link rel="alternate" href="https://example.test/e3"/>
</entry>
</feed>"#;
let feed = parse_feed(xml.as_bytes()).unwrap();
assert_eq!(feed.title, "Atom Cast");
assert_eq!(feed.description.as_deref(), Some("About things"));
assert_eq!(feed.link.as_deref(), Some("https://example.test/"));
assert_eq!(
feed.image_url.as_deref(),
Some("https://example.test/logo.png")
);
assert_eq!(feed.author.as_deref(), Some("Atom Author"));
assert_eq!(feed.episodes.len(), 2);
let first = &feed.episodes[0];
assert_eq!(first.guid.as_deref(), Some("urn:uuid:1"));
assert_eq!(first.published_at_ms, Some(1_709_289_000_000));
assert_eq!(first.description.as_deref(), Some("Entry summary"));
assert_eq!(first.link.as_deref(), Some("https://example.test/e1"));
assert_eq!(first.audio_url, "https://cdn.example.test/e1.ogg");
assert_eq!(first.byte_len, Some(42));
let second = &feed.episodes[1];
assert_eq!(second.published_at_ms, Some(1_709_683_200_000));
assert_eq!(second.description.as_deref(), Some("Inline xhtml"));
}
#[test]
fn items_without_audio_are_skipped_and_media_content_is_a_fallback() {
let xml = r#"<rss xmlns:media="http://search.yahoo.com/mrss/"><channel><title>T</title>
<item><title>No enclosure</title><guid>a</guid></item>
<item><title>Image enclosure</title><guid>b</guid><enclosure url="https://x.test/b.jpg" type="image/jpeg"/></item>
<item><title>Empty url</title><guid>c</guid><enclosure url="" type="audio/mpeg"/></item>
<item><title>Media</title><guid>d</guid>
<media:content url="https://x.test/d.png" medium="image"/>
<media:group><media:content url="https://x.test/d.mp3" type="audio/mpeg" fileSize="77"/></media:group>
<media:thumbnail url="https://x.test/d-thumb.jpg"/>
</item>
<item><title>Video</title><guid>e</guid><enclosure url="https://x.test/e.mp4" type="video/mp4"/></item>
</channel></rss>"#;
let feed = parse_feed(xml.as_bytes()).unwrap();
let titles: Vec<_> = feed.episodes.iter().map(|e| e.title.as_str()).collect();
assert_eq!(titles, ["Media", "Video"]);
assert_eq!(feed.episodes[0].audio_url, "https://x.test/d.mp3");
assert_eq!(feed.episodes[0].byte_len, Some(77));
assert_eq!(
feed.episodes[0].image_url.as_deref(),
Some("https://x.test/d-thumb.jpg")
);
}
#[test]
fn duplicate_guids_keep_the_first_and_missing_guids_stay_absent() {
let items = [
item("dup", "First", "Mon, 01 Jan 2024 00:00:00 +0000"),
item("dup", "Second", "Tue, 02 Jan 2024 00:00:00 +0000"),
r#"<item><title>No guid</title><enclosure url="https://cdn.example.test/noguid.mp3" type="audio/mpeg"/></item>"#.to_string(),
r#"<item><title>Blank guid</title><guid> </guid><enclosure url="https://cdn.example.test/blank.mp3" type="audio/mpeg"/></item>"#.to_string(),
]
.concat();
let feed = parse_feed(rss("", &items).as_bytes()).unwrap();
let pairs: Vec<_> = feed
.episodes
.iter()
.map(|e| (e.guid.as_deref(), e.title.as_str(), e.audio_url.as_str()))
.collect();
assert_eq!(
pairs,
[
(Some("dup"), "First", "https://cdn.example.test/dup.mp3"),
(None, "No guid", "https://cdn.example.test/noguid.mp3"),
(None, "Blank guid", "https://cdn.example.test/blank.mp3"),
]
);
}
#[test]
fn durations() {
let cases = [
("01:02:03", Some(3723.0)),
("1:02:03", Some(3723.0)),
("62:03", Some(3723.0)),
("3723", Some(3723.0)),
("3723.5", Some(3723.5)),
("00:00:01.25", Some(1.25)),
(" 45:00 ", Some(2700.0)),
("0", None),
("00:00:00", None),
("", None),
("abc", None),
("1:2:3:4", None),
("-5", None),
("1e3", None),
("NaN", None),
("12:", None),
];
for (input, expected) in cases {
assert_eq!(parse_duration(input), expected, "{input:?}");
}
}
#[test]
fn dates() {
let reference = Some(1_704_189_600); // 2024-01-02 10:00:00 UTC
let cases = [
("Tue, 02 Jan 2024 10:00:00 GMT", reference),
("Tue, 02 Jan 2024 10:00:00 +0000", reference),
("02 Jan 2024 10:00:00 UT", reference),
("Tue, 2 Jan 2024 13:00:00 +0300", reference),
("Tue, 02 Jan 2024 05:00:00 EST", reference),
("Tue, 02 Jan 2024 02:00:00 PST", reference),
("Tuesday, 02 January 2024 10:00 GMT", reference),
("Tue, 02 Jan 24 10:00:00 GMT", reference),
("Tue, 02 Jan 2024 13:00:00 +03:00", reference),
("Tue, 02 Jan 2024 10:00:00", reference),
("Вт, 02 янв 2024 13:00:00 +0300", reference),
("2024-01-02T10:00:00Z", reference),
("2024-01-02T13:00:00+03:00", reference),
("2024-01-02T10:00:00.123Z", reference),
("2024-01-02 10:00:00", reference),
("2024-01-02", Some(1_704_153_600)),
("not a date", None),
("Tue, 31 Feb 2024 10:00:00 GMT", None),
("Tue, 02 Foo 2024 10:00:00 GMT", None),
("2024-13-02T10:00:00Z", None),
("", None),
];
for (input, expected) in cases {
assert_eq!(parse_date(input), expected, "{input:?}");
}
}
#[test]
fn non_ascii_zones_do_not_panic() {
for input in [
"Mon, 07 Nov 2022 10:00:00 +й0",
"Mon, 07 Nov 2022 10:00:00 -йй",
"Mon, 07 Nov 2022 10:00+й0",
"Mon, 07 Nov 2022 10:00:00 +0й00",
"2022-11-07T10:00:00+0й0",
"2022-11-07T10:00:00+й",
"2022-11-07T10:00:00-€:00",
"2022-11-07T10:00:00+日本",
] {
assert_eq!(parse_date(input), None, "{input:?}");
}
// Exhaustive-ish: every short suffix of mixed ASCII / multi-byte
// characters after a sign.
let alphabet = ['0', '9', ':', 'й', '€', '日', '+', '-'];
for a in alphabet {
for b in alphabet {
for c in alphabet {
for prefix in ["Mon, 07 Nov 2022 10:00:00 ", "2022-11-07T10:00:00"] {
for sign in ['+', '-'] {
let _ = parse_date(&format!("{prefix}{sign}{a}{b}{c}"));
}
}
}
}
}
// In a feed, the bad date only loses the date.
let xml = rss("", &item("z", "Zone", "Mon, 07 Nov 2022 10:00:00 +й0"));
let feed = parse_feed(xml.as_bytes()).unwrap();
assert_eq!(feed.episodes.len(), 1);
assert_eq!(feed.episodes[0].published_at_ms, None);
}
#[test]
fn media_urls_are_resolved_and_must_be_http() {
let xml = rss(
r#"<itunes:image href="/img/cover.jpg"/>"#,
r#"<item><guid>rel</guid><title>Relative</title>
<itunes:image href="ep.png"/>
<enclosure url="/audio/1.mp3" type="audio/mpeg"/></item>
<item><guid>proto</guid><title>Protocol relative</title>
<enclosure url="//cdn.example.test/2.mp3" type="audio/mpeg"/></item>
<item><guid>file</guid><title>Local file</title>
<enclosure url="file:///etc/passwd" type="audio/mpeg"/></item>
<item><guid>ftp</guid><title>FTP</title>
<enclosure url="ftp://example.test/3.mp3" type="audio/mpeg"/></item>
<item><guid>fallback</guid><title>Media fallback</title>
<enclosure url="file:///tmp/x.mp3" type="audio/mpeg"/>
<media:content xmlns:media="http://search.yahoo.com/mrss/" url="https://cdn.example.test/4.mp3" type="audio/mpeg"/></item>
<item><guid>abs</guid><title>Absolute</title>
<enclosure url="HTTPS://cdn.example.test/5.mp3" type="audio/mpeg"/></item>"#,
);
let feed =
parse_feed_with_base(xml.as_bytes(), Some("https://host.test/feeds/rss.xml")).unwrap();
let urls: Vec<(&str, &str)> = feed
.episodes
.iter()
.map(|e| (e.guid.as_deref().unwrap_or_default(), e.audio_url.as_str()))
.collect();
assert_eq!(
urls,
[
("rel", "https://host.test/audio/1.mp3"),
("proto", "https://cdn.example.test/2.mp3"),
("fallback", "https://cdn.example.test/4.mp3"),
("abs", "HTTPS://cdn.example.test/5.mp3"),
]
);
assert_eq!(
feed.image_url.as_deref(),
Some("https://host.test/img/cover.jpg")
);
assert_eq!(
feed.episodes[0].image_url.as_deref(),
Some("https://host.test/feeds/ep.png")
);
for episode in &feed.episodes {
assert!(is_remote_url(&episode.audio_url), "{}", episode.audio_url);
}
// Without a base, relative media cannot be resolved and is dropped;
// protocol-relative URLs default to https.
let feed = parse_feed(xml.as_bytes()).unwrap();
let guids: Vec<&str> = feed
.episodes
.iter()
.map(|e| e.guid.as_deref().unwrap_or_default())
.collect();
assert_eq!(guids, ["proto", "fallback", "abs"]);
assert_eq!(feed.image_url, None);
}
#[test]
fn rss_with_an_unrelated_default_namespace() {
for ns in ["http://example.test/x", "http://www.w3.org/2005/Atom"] {
let xml = format!(
r#"<rss version="2.0" xmlns="{ns}" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>T</title>
<atom:link href="https://a.test/feed" rel="self"/>
<item><guid>1</guid><title>One</title><enclosure url="https://a.test/1.mp3" type="audio/mpeg"/></item>
</channel></rss>"#
);
let feed = parse_feed(xml.as_bytes()).unwrap();
assert_eq!(feed.title, "T", "{ns}");
assert_eq!(feed.episodes.len(), 1, "{ns}");
assert_eq!(feed.episodes[0].title, "One");
}
}
#[test]
fn html_pages_and_garbage_are_rejected() {
let html = br#"<!DOCTYPE html><html><head><title>Vivat</title></head><body><p>Hi &nbsp; there</p></body></html>"#;
let err = parse_feed(html).unwrap_err().to_string();
assert!(err.contains("not an RSS or Atom feed"), "{err}");
for garbage in [
&b""[..],
b"hello world",
b"\x00\xff\xfe garbage",
b"{\"json\": true}",
b"<",
b"<<<>>>",
] {
let err = parse_feed(garbage).unwrap_err().to_string();
assert!(
err.contains("not an RSS or Atom feed"),
"{garbage:?}: {err}"
);
}
}
#[test]
fn truncated_and_malformed_xml_never_panics() {
let items: String = (0..5)
.map(|i| item(&format!("g{i}"), &format!("T {i} &amp; &nbsp;"), "bad date"))
.collect();
let xml = rss(
"<description><![CDATA[cdata]]></description><itunes:image href=\"x\"/>",
&items,
);
let bytes = xml.as_bytes();
for len in 0..=bytes.len() {
let _ = parse_feed(&bytes[..len]);
}
// Mismatched end tags are tolerated.
let sloppy = r#"<rss><channel><title>S</title><item><title>A</b></title><guid>1</guid><enclosure url="https://x.test/1.mp3"/></item></channel></rss>"#;
let feed = parse_feed(sloppy.as_bytes()).unwrap();
assert_eq!(feed.episodes.len(), 1);
assert_eq!(feed.episodes[0].title, "A");
// Truncation after complete items keeps them.
let third = xml.find("<title>T 3").unwrap();
let cut = &bytes[..xml[..third].rfind("<item>").unwrap()];
let feed = parse_feed(cut).unwrap();
assert_eq!(feed.episodes.len(), 3);
}
#[test]
fn deep_nesting_does_not_overflow() {
let depth = 100_000;
let mut xml = String::from("<rss><channel><title>Deep</title>");
xml.push_str(&"<x>".repeat(depth));
xml.push_str(&"</x>".repeat(depth));
xml.push_str(&item("after", "After", ""));
xml.push_str("</channel></rss>");
let feed = parse_feed(xml.as_bytes()).unwrap();
assert_eq!(feed.episodes.len(), 1);
}
#[test]
fn cyrillic_titles_in_utf8_and_windows_1251() {
let xml = rss(
"<itunes:author>Иван Петров</itunes:author>",
&item(
"1",
"Выпуск №1: «Привет»",
"Mon, 01 Jan 2024 00:00:00 +0300",
),
)
.replace("Test Cast", "Подкаст");
let feed = parse_feed(xml.as_bytes()).unwrap();
assert_eq!(feed.title, "Подкаст");
assert_eq!(feed.author.as_deref(), Some("Иван Петров"));
assert_eq!(feed.episodes[0].title, "Выпуск №1: «Привет»");
let cp1251 = xml.replace("encoding=\"UTF-8\"", "encoding=\"windows-1251\"");
let (encoded, _, unmappable) = encoding_rs::WINDOWS_1251.encode(&cp1251);
assert!(!unmappable);
let feed = parse_feed(&encoded).unwrap();
assert_eq!(feed.title, "Подкаст");
assert_eq!(feed.episodes[0].title, "Выпуск №1: «Привет»");
// Invalid UTF-8 without a declaration is decoded lossily.
let mut broken = xml.into_bytes();
let at = broken
.windows(7)
.position(|w| w == b"Test Ca")
.unwrap_or(10);
broken.insert(at, 0xff);
assert!(parse_feed(&broken).is_ok());
}
#[test]
fn namespace_prefixes_are_matched_tolerantly() {
// Undeclared `itunes:` prefix and a custom prefix bound to the
// iTunes namespace.
let xml = format!(
r#"<rss><channel><title>P</title><itunes:author>Undeclared</itunes:author>
<item xmlns:it="{ITUNES_NS}"><guid>1</guid><it:title>Custom prefix</it:title><it:duration>90</it:duration>
<enclosure url="https://x.test/1.mp3" type="audio/mpeg"/></item></channel></rss>"#
);
let feed = parse_feed(xml.as_bytes()).unwrap();
assert_eq!(feed.author.as_deref(), Some("Undeclared"));
assert_eq!(feed.episodes[0].title, "Custom prefix");
assert_eq!(feed.episodes[0].duration_ms, Some(90_000));
}
#[test]
fn long_descriptions_are_truncated() {
let long = "word ".repeat(2000);
let text = html_to_text(&long, 4000);
assert!(text.chars().count() <= 4000);
assert!(text.ends_with('…'));
let xml = rss(&format!("<description>{long}</description>"), "");
let feed = parse_feed(xml.as_bytes()).unwrap();
assert!(feed.description.unwrap().chars().count() <= 4000);
}
#[test]
fn itunes_season_episode_and_type() {
let xml = rss(
"",
r#"<item><guid>a</guid><title>A</title>
<itunes:season>2</itunes:season><itunes:episode> 7 </itunes:episode>
<itunes:episodeType>Trailer</itunes:episodeType>
<enclosure url="https://cdn.example.test/a.mp3" type="audio/mpeg"/></item>
<item><guid>b</guid><title>B</title>
<itunes:season>two</itunes:season><itunes:episode>-1</itunes:episode>
<itunes:episodeType>mystery</itunes:episodeType>
<enclosure url="https://cdn.example.test/b.mp3" type="audio/mpeg"/></item>
<item><guid>c</guid><title>C</title><itunes:episodeType>bonus</itunes:episodeType>
<itunes:episode>99999999999</itunes:episode>
<enclosure url="https://cdn.example.test/c.mp3" type="audio/mpeg"/></item>"#,
);
let feed = parse_feed(xml.as_bytes()).unwrap();
let a = &feed.episodes[0];
assert_eq!((a.season, a.episode_number), (Some(2), Some(7)));
assert_eq!(a.episode_type, EpisodeType::Trailer);
let b = &feed.episodes[1];
assert_eq!((b.season, b.episode_number), (None, None));
assert_eq!(b.episode_type, EpisodeType::Full);
let c = &feed.episodes[2];
assert_eq!((c.season, c.episode_number), (None, None));
assert_eq!(c.episode_type, EpisodeType::Bonus);
}
#[test]
fn guid_permalink_flag() {
let xml = rss(
"",
r#"<item><guid isPermaLink="false">x</guid><enclosure url="https://c.test/x.mp3"/></item>
<item><guid>https://e.test/y</guid><enclosure url="https://c.test/y.mp3"/></item>
<item><guid isPermaLink=" FALSE ">z</guid><enclosure url="https://c.test/z.mp3"/></item>
<item><enclosure url="https://c.test/n.mp3"/></item>"#,
);
let feed = parse_feed(xml.as_bytes()).unwrap();
let flags: Vec<Option<bool>> = feed.episodes.iter().map(|e| e.guid_is_permalink).collect();
assert_eq!(flags, [Some(false), Some(true), Some(false), None]);
}
#[test]
fn new_feed_url_is_reported() {
let xml = rss(
"<itunes:new-feed-url> https://new.example.test/feed?x=1 </itunes:new-feed-url>",
"",
);
let feed = parse_feed(xml.as_bytes()).unwrap();
assert_eq!(
feed.new_feed_url.as_deref(),
Some("https://new.example.test/feed?x=1")
);
let alias = FeedUrl::parse(feed.new_feed_url.as_deref().unwrap()).unwrap();
assert_eq!(alias.as_str(), "https://new.example.test/feed?x=1");
}
#[test]
fn oversized_guid_is_treated_as_absent() {
let long = "g".repeat(MAX_GUID_BYTES + 1);
let xml = rss(
"",
&format!(r#"<item><guid>{long}</guid><enclosure url="https://c.test/1.mp3"/></item>"#),
);
let feed = parse_feed(xml.as_bytes()).unwrap();
assert_eq!(feed.episodes[0].guid, None);
}
#[test]
fn media_urls_with_spaces_are_percent_encoded() {
let xml = rss(
"",
r#"<item><guid>s</guid><enclosure url="https://c.test/my file.mp3" type="audio/mpeg"/></item>"#,
);
let feed = parse_feed(xml.as_bytes()).unwrap();
assert_eq!(feed.episodes[0].audio_url, "https://c.test/my%20file.mp3");
}
#[test]
fn parsed_feed_maps_to_valid_contract_metadata() {
let base = FeedUrl::parse("https://example.test/feed.xml").unwrap();
let items = [
item("g1", "One", "Tue, 02 Jan 2024 10:00:00 GMT"),
r#"<item><title>No guid</title><link>https://example.test/no-guid</link>
<pubDate>Wed, 03 Jan 2024 10:00:00 GMT</pubDate>
<enclosure url="https://cdn.example.test/ng.mp3" type="audio/mpeg"/></item>"#
.to_string(),
// Same page as the channel link: not identifying.
r#"<item><title>Home link</title><link>https://example.test/cast</link>
<enclosure url="https://cdn.example.test/home.mp3" type="audio/mpeg"/></item>"#
.to_string(),
]
.concat();
let feed = parse_feed_with_base(rss("", &items).as_bytes(), Some(base.as_str())).unwrap();
let podcast_id = PodcastId::from_feed_url(&base);
let ids = feed.resolve_episode_ids(&podcast_id, &[]);
assert_eq!(ids[0], EpisodeId::from_guid(&podcast_id, "g1"));
assert_eq!(
ids[1],
Some(EpisodeId::derive(
&podcast_id,
&crate::podcast::EpisodeKey::link("https://example.test/no-guid").unwrap()
))
);
assert_eq!(
ids[2],
Some(EpisodeId::derive(
&podcast_id,
&crate::podcast::EpisodeKey::enclosure("https://cdn.example.test/home.mp3").unwrap()
))
);
let podcast = feed.to_podcast_meta(base.clone());
podcast.validate().unwrap();
for (episode, id) in feed.episodes.iter().zip(ids) {
let meta = episode.to_meta(&podcast_id, id.unwrap());
meta.validate().unwrap();
meta.to_ref(&podcast).validate().unwrap();
}
// A later refresh with a changed title and audio URL keeps every id,
// using the stored hints of the previous refresh.
let known: Vec<KnownEpisode> = feed
.episodes
.iter()
.zip(feed.resolve_episode_ids(&podcast_id, &[]))
.map(|(episode, id)| KnownEpisode {
episode_id: id.unwrap(),
hints: episode.identity_hints(),
})
.collect();
let refreshed = items
.replace("Home link", "Home link (edited)")
.replace("cdn.example.test/ng.mp3", "cdn2.example.test/ng.mp3");
let feed2 = parse_feed_with_base(rss("", &refreshed).as_bytes(), Some(base.as_str())).unwrap();
let ids2 = feed2.resolve_episode_ids(&podcast_id, &known);
let expected: Vec<Option<EpisodeId>> =
known.iter().map(|k| Some(k.episode_id.clone())).collect();
assert_eq!(ids2, expected);
}
#[test]
fn hostile_sizes_stay_bounded() {
let huge_title = "t".repeat(MAX_CAPTURE_BYTES * 2);
let xml = rss(
"",
&format!(
r#"<item><guid>h</guid><title>{huge_title}</title><enclosure url="https://c.test/h.mp3"/></item>"#
),
);
let feed = parse_feed(xml.as_bytes()).unwrap();
assert!(feed.episodes[0].title.chars().count() <= MAX_TITLE_CHARS);
let duration = rss(
"",
r#"<item><guid>d</guid><itunes:duration>99999999999999999999</itunes:duration><enclosure url="https://c.test/d.mp3"/></item>"#,
);
assert_eq!(
parse_feed(duration.as_bytes()).unwrap().episodes[0].duration_ms,
None
);
}
#[test]
fn text_full_of_unclosed_angle_brackets_is_linear() {
// '<a' repeated up to the capture cap, with no '>' anywhere: the old
// scanner searched for '>' to the end of the text at every '<'.
let body = "&lt;a".repeat(MAX_CAPTURE_BYTES / 2);
let xml = rss(
"",
&format!(
r#"<item><guid>lt</guid><title>{body}</title><description>{body}</description><enclosure url="https://c.test/lt.mp3"/></item>"#
),
);
let started = std::time::Instant::now();
let feed = parse_feed(xml.as_bytes()).unwrap();
assert!(
started.elapsed() < std::time::Duration::from_secs(2),
"took {:?}",
started.elapsed()
);
let episode = &feed.episodes[0];
assert!(episode.title.starts_with("<a<a"));
assert!(episode.title.chars().count() <= MAX_TITLE_CHARS);
assert_eq!(
html_to_text("a < b and <b>bold</b> <c", 100),
"a < b and bold <c"
);
}
#[test]
fn links_are_identity_material_and_kept_verbatim() {
// Entities are decoded once by the XML reader and never again; tag-like
// text, inner whitespace and long links are not rewritten by the display
// cleaner.
let long_path = "p".repeat(1_500);
let items = format!(
r#"<item><title>A</title><link> https://e.test/a?x=1&amp;amp;y=&lt;b&gt; </link><enclosure url="https://c.test/a.mp3"/></item>
<item><title>B</title><link>https://e.test/{long_path}</link><enclosure url="https://c.test/b.mp3"/></item>"#
);
let feed = parse_feed(rss("", &items).as_bytes()).unwrap();
assert_eq!(
feed.episodes[0].link.as_deref(),
Some("https://e.test/a?x=1&amp;y=<b>")
);
let long = format!("https://e.test/{long_path}");
assert_eq!(feed.episodes[1].link.as_deref(), Some(long.as_str()));
let podcast_id =
PodcastId::from_feed_url(&FeedUrl::parse("https://example.test/feed").unwrap());
let ids = feed.resolve_episode_ids(&podcast_id, &[]);
assert_eq!(
ids[1],
Some(EpisodeId::derive(
&podcast_id,
&crate::podcast::EpisodeKey::link(&long).unwrap()
))
);
let atom = r#"<?xml version="1.0"?><feed xmlns="http://www.w3.org/2005/Atom"><title>T</title>
<link href=" https://e.test/home?a=1&amp;b=2 "/>
<entry><title>E</title><link rel="alternate" href="https://e.test/e?q=&lt;x&gt;"/>
<link rel="enclosure" href="https://c.test/e.mp3" type="audio/mpeg"/></entry></feed>"#;
let feed = parse_feed(atom.as_bytes()).unwrap();
assert_eq!(feed.link.as_deref(), Some("https://e.test/home?a=1&b=2"));
assert_eq!(
feed.episodes[0].link.as_deref(),
Some("https://e.test/e?q=<x>")
);
}
+530
View File
@@ -0,0 +1,530 @@
//! Feed URL canonicalization and feed aliases.
use std::collections::BTreeSet;
use std::fmt;
use serde::{Deserialize, Serialize};
use super::{MAX_FEED_ALIASES, MAX_URL_BYTES, PodcastError, PodcastId, PodcastResult};
/// Canonical podcast feed URL.
///
/// Canonicalization ([`FeedUrl::parse`]) is deliberately small so that every
/// client, whatever its URL library, produces the same bytes:
///
/// 1. Only `http://` and `https://` (any case) are accepted; the scheme is
/// lowercased.
/// 2. Userinfo (`user:pass@`), whitespace, control characters and `\` are
/// rejected.
/// 3. The host is lowercased. Non-ASCII hosts are rejected: pass the IDNA
/// (`xn--`) form, as produced by WHATWG URL parsers. IPv6 literals are
/// accepted in brackets.
/// 4. An empty port and the scheme's default port (80, 443) are dropped;
/// other ports lose leading zeros.
/// 5. The fragment is dropped.
/// 6. An empty path becomes `/`. The path and the **whole** query string are
/// otherwise kept byte-for-byte, including parameter order, case and
/// percent-encoding; only non-ASCII bytes are percent-encoded (`%XX`,
/// uppercase hex) so the result is ASCII.
///
/// The canonical form is at most [`MAX_URL_BYTES`] bytes and canonicalizing it
/// again is the identity. Deserialization accepts only canonical strings.
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)]
#[serde(try_from = "String", into = "String")]
pub struct FeedUrl(String);
impl FeedUrl {
/// Canonicalizes an absolute `http(s)` feed URL.
pub fn parse(input: &str) -> PodcastResult<Self> {
canonicalize(input).map(Self)
}
/// Canonicalizes a URL typed or pasted by a user.
///
/// Surrounding whitespace is trimmed, `feed:`, `podcast:`, `pcast:` and
/// `itpc:` prefixes map to `https://` (or to the embedded `http(s)` URL),
/// and input without a scheme defaults to `https://`. `http:` or `https:`
/// not followed by `//` (for example `https:/host/feed`) is rejected.
pub fn from_user_input(input: &str) -> PodcastResult<Self> {
let trimmed = input.trim();
if trimmed.is_empty() {
return Err(invalid("feed URL is empty"));
}
let lower = trimmed.to_ascii_lowercase();
if is_http_scheme_typo(&lower) {
return Err(invalid("feed URL must start with http:// or https://"));
}
let mut candidate = trimmed.to_string();
for scheme in ["feed:", "podcast:", "pcast:", "itpc:"] {
if lower.starts_with(scheme) {
let rest = &trimmed[scheme.len()..];
let rest_lower = rest.to_ascii_lowercase();
if is_http_scheme_typo(&rest_lower) {
return Err(invalid("feed URL must start with http:// or https://"));
}
candidate =
if rest_lower.starts_with("http://") || rest_lower.starts_with("https://") {
rest.to_string()
} else {
format!("https://{}", rest.trim_start_matches('/'))
};
break;
}
}
if !has_scheme(&candidate) {
candidate = format!("https://{candidate}");
}
Self::parse(&candidate)
}
/// Canonical string form.
pub fn as_str(&self) -> &str {
&self.0
}
}
impl fmt::Display for FeedUrl {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.write_str(&self.0)
}
}
impl TryFrom<String> for FeedUrl {
type Error = PodcastError;
fn try_from(value: String) -> PodcastResult<Self> {
if canonicalize(&value)? == value {
Ok(Self(value))
} else {
Err(invalid("feed URL is not in canonical form"))
}
}
}
impl From<FeedUrl> for String {
fn from(value: FeedUrl) -> String {
value.0
}
}
impl AsRef<str> for FeedUrl {
fn as_ref(&self) -> &str {
&self.0
}
}
fn invalid(reason: &str) -> PodcastError {
PodcastError::InvalidFeedUrl(reason.to_string())
}
/// `http:` or `https:` not followed by `//`, such as `https:/host/feed`.
fn is_http_scheme_typo(lower: &str) -> bool {
["http:", "https:"]
.iter()
.any(|scheme| lower.starts_with(scheme) && !lower[scheme.len()..].starts_with("//"))
}
fn has_scheme(text: &str) -> bool {
let Some((scheme, rest)) = text.split_once(':') else {
return false;
};
let mut chars = scheme.chars();
chars.next().is_some_and(|c| c.is_ascii_alphabetic())
&& chars.all(|c| c.is_ascii_alphanumeric() || matches!(c, '+' | '-' | '.'))
&& rest.starts_with("//")
}
fn canonicalize(input: &str) -> PodcastResult<String> {
if input.len() > MAX_URL_BYTES * 3 {
return Err(invalid("feed URL is too long"));
}
if input.chars().any(|c| c.is_whitespace() || c.is_control()) {
return Err(invalid("feed URL must not contain whitespace"));
}
if input.contains('\\') {
return Err(invalid("feed URL must not contain '\\'"));
}
let (scheme, rest) = input
.split_once("://")
.ok_or_else(|| invalid("feed URL must start with http:// or https://"))?;
let (scheme, default_port) = if scheme.eq_ignore_ascii_case("http") {
("http", 80)
} else if scheme.eq_ignore_ascii_case("https") {
("https", 443)
} else {
return Err(invalid("only http and https feed URLs are supported"));
};
let authority_end = rest.find(['/', '?', '#']).unwrap_or(rest.len());
let (authority, tail) = rest.split_at(authority_end);
if authority.contains('@') {
return Err(invalid("feed URL must not contain credentials"));
}
let (host, port) = split_host_port(authority)?;
let host = canonical_host(host)?;
let port = match port {
None | Some("") => None,
Some(port) => {
if port.len() > 5 || !port.bytes().all(|b| b.is_ascii_digit()) {
return Err(invalid("invalid feed URL port"));
}
match port.parse::<u32>() {
Ok(value) if (1..=65_535).contains(&value) => {
(value != default_port).then_some(value)
}
_ => return Err(invalid("invalid feed URL port")),
}
}
};
let tail = tail.split('#').next().unwrap_or_default();
let (path, query) = match tail.find('?') {
Some(at) => tail.split_at(at),
None => (tail, ""),
};
let mut out = String::with_capacity(input.len() + 8);
out.push_str(scheme);
out.push_str("://");
out.push_str(&host);
if let Some(port) = port {
out.push(':');
out.push_str(&port.to_string());
}
if path.is_empty() {
out.push('/');
} else {
push_ascii(&mut out, path);
}
push_ascii(&mut out, query);
if out.len() > MAX_URL_BYTES {
return Err(invalid("feed URL is too long"));
}
Ok(out)
}
fn split_host_port(authority: &str) -> PodcastResult<(&str, Option<&str>)> {
if authority.starts_with('[') {
let close = authority
.find(']')
.ok_or_else(|| invalid("unterminated IPv6 host"))?;
let (host, after) = authority.split_at(close + 1);
return match after {
"" => Ok((host, None)),
_ => after
.strip_prefix(':')
.map(|port| (host, Some(port)))
.ok_or_else(|| invalid("invalid feed URL host")),
};
}
Ok(match authority.rsplit_once(':') {
Some((host, port)) => (host, Some(port)),
None => (authority, None),
})
}
fn canonical_host(host: &str) -> PodcastResult<String> {
if host.is_empty() {
return Err(invalid("feed URL has no host"));
}
if !host.is_ascii() {
return Err(invalid(
"feed URL host must be ASCII (use the IDNA xn-- form)",
));
}
if let Some(inner) = host.strip_prefix('[').and_then(|h| h.strip_suffix(']')) {
if inner.is_empty()
|| !inner
.bytes()
.all(|b| b.is_ascii_hexdigit() || matches!(b, b':' | b'.'))
{
return Err(invalid("invalid IPv6 host"));
}
return Ok(host.to_ascii_lowercase());
}
if !host
.bytes()
.all(|b| b.is_ascii_alphanumeric() || matches!(b, b'-' | b'.' | b'_'))
|| !host.bytes().any(|b| b.is_ascii_alphanumeric())
{
return Err(invalid("invalid feed URL host"));
}
Ok(host.to_ascii_lowercase())
}
fn push_ascii(out: &mut String, value: &str) {
for ch in value.chars() {
if ch.is_ascii() {
out.push(ch);
} else {
let mut buf = [0u8; 4];
for byte in ch.encode_utf8(&mut buf).bytes() {
out.push_str(&format!("%{byte:02X}"));
}
}
}
}
/// Resolves the podcast a canonical feed URL belongs to.
///
/// `known` yields `(podcast_id, url)` pairs of every URL already associated
/// with a podcast: the URL of its first subscription, its current fetch URL
/// and its feed aliases ([`super::SubscriptionRecord::feed_urls`]; the bounded
/// alias set alone can miss the current fetch URL). When `url` matches, the existing id is returned (the
/// smallest id if, after a concurrent offline subscription, several podcasts
/// share the URL). Otherwise a new id is derived from `url`.
///
/// Re-subscribing through a redirect target or an `itunes:new-feed-url`
/// therefore reuses the original podcast, its progress and its history.
pub fn resolve_podcast_id<'a>(
url: &FeedUrl,
known: impl IntoIterator<Item = (&'a PodcastId, &'a FeedUrl)>,
) -> PodcastId {
known
.into_iter()
.filter(|(_, alias)| *alias == url)
.map(|(id, _)| id)
.min()
.cloned()
.unwrap_or_else(|| PodcastId::from_feed_url(url))
}
/// Merges two feed alias sets of `podcast` as a bounded grow-only set.
///
/// The result is sorted and duplicate-free. The URL from which `podcast` was
/// derived is always kept; of the other URLs the [`MAX_FEED_ALIASES`]
/// lexicographically smallest are kept. Taking the smallest `k` elements of a
/// union is commutative, associative and idempotent, so devices converge
/// regardless of delivery order. Other URLs, including a podcast's current
/// fetch URL, can fall outside the bound; the materialized subscription keeps
/// its `podcast.feed_url` separately.
pub fn merge_feed_aliases(podcast: &PodcastId, a: &[FeedUrl], b: &[FeedUrl]) -> Vec<FeedUrl> {
let mut origin = None;
let mut rest = BTreeSet::new();
for url in a.iter().chain(b) {
if PodcastId::from_feed_url(url) == *podcast {
origin = Some(url.clone());
} else {
rest.insert(url.clone());
}
}
let mut out: Vec<FeedUrl> = rest.into_iter().take(MAX_FEED_ALIASES).collect();
if let Some(origin) = origin {
out.push(origin);
out.sort();
}
out
}
#[cfg(test)]
mod tests {
use super::*;
fn canon(input: &str) -> String {
FeedUrl::parse(input).unwrap().as_str().to_string()
}
#[test]
fn canonicalization_rules() {
let cases = [
(
"https://example.com/feed.xml",
"https://example.com/feed.xml",
),
(
"HTTPS://Example.COM/Feed.XML",
"https://example.com/Feed.XML",
),
("http://example.com:80/rss", "http://example.com/rss"),
("https://example.com:443/rss", "https://example.com/rss"),
("http://example.com:443/rss", "http://example.com:443/rss"),
(
"https://example.com:08080/rss",
"https://example.com:8080/rss",
),
("https://example.com:/rss", "https://example.com/rss"),
("https://example.com", "https://example.com/"),
("https://example.com?x=1", "https://example.com/?x=1"),
("https://example.com/rss#latest", "https://example.com/rss"),
("https://example.com/rss?#x", "https://example.com/rss?"),
(
"https://example.com/a/../b/./c?b=2&a=1&A=%2f&token=XyZ",
"https://example.com/a/../b/./c?b=2&a=1&A=%2f&token=XyZ",
),
(
"https://example.com/rss?url=https://other.test/x?y#frag",
"https://example.com/rss?url=https://other.test/x?y",
),
(
"https://example.com/подкаст?q=ё",
"https://example.com/%D0%BF%D0%BE%D0%B4%D0%BA%D0%B0%D1%81%D1%82?q=%D1%91",
),
(
"https://[2001:DB8::1]:443/feed",
"https://[2001:db8::1]/feed",
),
("https://[::1]:8443/feed", "https://[::1]:8443/feed"),
(
"https://xn--d1acufc.xn--p1ai/rss",
"https://xn--d1acufc.xn--p1ai/rss",
),
(
"https://my_host.example./rss",
"https://my_host.example./rss",
),
];
for (input, expected) in cases {
assert_eq!(canon(input), expected, "{input:?}");
// Idempotent.
assert_eq!(canon(expected), expected, "{expected:?}");
}
}
#[test]
fn rejected_urls() {
for bad in [
"",
"example.com/feed",
"ftp://example.com/feed",
"file:///etc/passwd",
"javascript:alert(1)",
"https://",
"https:///path",
"https://user:pass@example.com/feed",
"https://token@example.com/feed",
"https://example.com/my feed",
" https://example.com/feed",
"https://example.com/feed\n",
"https://example.com\\feed",
"https://пример.рф/rss",
"https://example.com:0/rss",
"https://example.com:65536/rss",
"https://example.com:12a/rss",
"https://[::1/feed",
"https://[zz]/feed",
"https://exa%6dple.com/feed",
"https://.../feed",
] {
assert!(FeedUrl::parse(bad).is_err(), "{bad:?}");
}
}
#[test]
fn length_limit_is_inclusive() {
let prefix = "https://a.test/";
let at = format!("{prefix}{}", "x".repeat(MAX_URL_BYTES - prefix.len()));
assert_eq!(canon(&at).len(), MAX_URL_BYTES);
let over = format!("{at}x");
assert!(FeedUrl::parse(&over).is_err());
}
#[test]
fn user_input_forms() {
let ok = [
(
"https://podster.test/rss.xml?pid=778",
"https://podster.test/rss.xml?pid=778",
),
(" http://a.test/feed ", "http://a.test/feed"),
("a.test/feed.xml", "https://a.test/feed.xml"),
("feed://a.test/rss", "https://a.test/rss"),
("feed:https://a.test/rss", "https://a.test/rss"),
("podcast://a.test/rss", "https://a.test/rss"),
("itpc://a.test/rss", "https://a.test/rss"),
("HTTPS://A.TEST/Feed", "https://a.test/Feed"),
("a.test:8080/rss", "https://a.test:8080/rss"),
(
"example.test/rss?src=https://other.test",
"https://example.test/rss?src=https://other.test",
),
(
"localhost:8080/feed?u=http://x.test/",
"https://localhost:8080/feed?u=http://x.test/",
),
];
for (input, expected) in ok {
assert_eq!(
FeedUrl::from_user_input(input).unwrap().as_str(),
expected,
"{input:?}"
);
}
for bad in [
"",
" ",
"ftp://a.test/feed",
"https://a.test/my feed",
"https://",
"file:///etc/passwd",
"javascript:alert(1)",
// One-slash typos must not become a host named "https".
"https:/a.test/feed",
"http:/a.test/feed",
"HTTPS:a.test/feed",
"https:///a.test/feed",
"feed:https:/a.test/feed",
] {
assert!(FeedUrl::from_user_input(bad).is_err(), "{bad:?}");
}
}
#[test]
fn deserialization_requires_canonical_form() {
let url = FeedUrl::parse("https://example.com/rss").unwrap();
let json = serde_json::to_string(&url).unwrap();
assert_eq!(serde_json::from_str::<FeedUrl>(&json).unwrap(), url);
assert!(serde_json::from_str::<FeedUrl>("\"HTTPS://example.com/rss\"").is_err());
assert!(serde_json::from_str::<FeedUrl>("\"https://example.com/rss#x\"").is_err());
assert!(serde_json::from_str::<FeedUrl>("\"ftp://example.com/rss\"").is_err());
}
#[test]
fn query_parameters_change_identity() {
let a = PodcastId::from_feed_url(&FeedUrl::parse("https://a.test/rss?id=1").unwrap());
let b = PodcastId::from_feed_url(&FeedUrl::parse("https://a.test/rss?id=2").unwrap());
let c = PodcastId::from_feed_url(&FeedUrl::parse("HTTPS://A.test:443/rss?id=1#x").unwrap());
assert_ne!(a, b);
assert_eq!(a, c);
}
#[test]
fn aliases_resolve_to_the_original_podcast() {
let original = FeedUrl::parse("http://old.test/feed").unwrap();
let moved = FeedUrl::parse("https://new.test/feed").unwrap();
let id = PodcastId::from_feed_url(&original);
let known = [(&id, &original), (&id, &moved)];
assert_eq!(resolve_podcast_id(&moved, known), id);
assert_eq!(resolve_podcast_id(&original, known), id);
let unrelated = FeedUrl::parse("https://other.test/feed").unwrap();
assert_eq!(
resolve_podcast_id(&unrelated, known),
PodcastId::from_feed_url(&unrelated)
);
// Ambiguity after concurrent offline subscriptions resolves to the
// smallest id, independent of iteration order.
let other = PodcastId::from_feed_url(&moved);
let ambiguous = [(&id, &moved), (&other, &moved)];
let reversed = [(&other, &moved), (&id, &moved)];
let expected = (&id).min(&other).clone();
assert_eq!(resolve_podcast_id(&moved, ambiguous), expected);
assert_eq!(resolve_podcast_id(&moved, reversed), expected);
}
#[test]
fn alias_merge_is_a_bounded_semilattice() {
let origin = FeedUrl::parse("https://zzz.test/origin").unwrap();
let id = PodcastId::from_feed_url(&origin);
let urls: Vec<FeedUrl> = (0..(MAX_FEED_ALIASES + 4))
.map(|i| FeedUrl::parse(&format!("https://a.test/{i:02}")).unwrap())
.collect();
let a: Vec<FeedUrl> = urls[..5].to_vec();
let b: Vec<FeedUrl> = urls[3..].iter().cloned().chain([origin.clone()]).collect();
let c: Vec<FeedUrl> = urls[7..].to_vec();
let ab = merge_feed_aliases(&id, &a, &b);
assert_eq!(ab, merge_feed_aliases(&id, &b, &a));
assert_eq!(ab, merge_feed_aliases(&id, &ab, &ab));
let left = merge_feed_aliases(&id, &ab, &c);
let right = merge_feed_aliases(&id, &a, &merge_feed_aliases(&id, &b, &c));
assert_eq!(left, right);
assert_eq!(left.len(), MAX_FEED_ALIASES + 1);
assert!(left.contains(&origin));
assert!(left.windows(2).all(|w| w[0] < w[1]));
}
}
+343
View File
@@ -0,0 +1,343 @@
//! Cross-device podcast and episode identifiers.
use std::fmt;
use std::str::FromStr;
use serde::{Deserialize, Serialize};
use super::{FeedUrl, MAX_GUID_BYTES, PodcastError, PodcastResult};
/// BLAKE3 derivation domain of [`PodcastId`]. Changing it changes every id.
pub const PODCAST_ID_DOMAIN: &str = "furumi-podcast:podcast:v1";
/// BLAKE3 derivation domain of [`EpisodeId`]. Changing it changes every id.
pub const EPISODE_ID_DOMAIN: &str = "furumi-podcast:episode:v1";
/// String prefix of a [`PodcastId`].
pub const PODCAST_ID_PREFIX: &str = "pod1:";
/// String prefix of an [`EpisodeId`].
pub const EPISODE_ID_PREFIX: &str = "ep1:";
/// Length of the base32 body: 32 bytes, RFC 4648 lowercase, no padding.
const DIGEST_CHARS: usize = 52;
fn digest(domain: &str, parts: &[&[u8]]) -> [u8; 32] {
let mut hasher = blake3::Hasher::new();
hasher.update(domain.as_bytes());
hasher.update(&[0]);
for part in parts {
hasher.update(part);
}
*hasher.finalize().as_bytes()
}
fn encode(prefix: &str, digest: &[u8; 32]) -> String {
let mut body = data_encoding::BASE32_NOPAD.encode(digest);
body.make_ascii_lowercase();
format!("{prefix}{body}")
}
fn decode(prefix: &str, value: &str) -> Option<[u8; 32]> {
let body = value.strip_prefix(prefix)?;
if body.len() != DIGEST_CHARS
|| !body
.bytes()
.all(|b| b.is_ascii_lowercase() || (b'2'..=b'7').contains(&b))
{
return None;
}
let bytes = data_encoding::BASE32_NOPAD
.decode(body.to_ascii_uppercase().as_bytes())
.ok()?;
let digest: [u8; 32] = bytes.try_into().ok()?;
// Reject non-canonical trailing bits so every id has one spelling.
(encode(prefix, &digest) == value).then_some(digest)
}
macro_rules! string_id {
($name:ident, $prefix:expr, $kind:literal) => {
impl $name {
/// Parses and validates the canonical string form.
pub fn parse(value: &str) -> PodcastResult<Self> {
decode($prefix, value)
.map(|_| Self(value.to_string()))
.ok_or(PodcastError::InvalidId { kind: $kind })
}
/// Canonical string form.
pub fn as_str(&self) -> &str {
&self.0
}
/// The 32-byte BLAKE3 digest encoded in the id.
pub fn digest(&self) -> [u8; 32] {
decode($prefix, &self.0).expect("validated on construction")
}
}
impl fmt::Display for $name {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.write_str(&self.0)
}
}
impl FromStr for $name {
type Err = PodcastError;
fn from_str(value: &str) -> PodcastResult<Self> {
Self::parse(value)
}
}
impl TryFrom<String> for $name {
type Error = PodcastError;
fn try_from(value: String) -> PodcastResult<Self> {
decode($prefix, &value)
.map(|_| Self(value))
.ok_or(PodcastError::InvalidId { kind: $kind })
}
}
impl From<$name> for String {
fn from(value: $name) -> String {
value.0
}
}
impl AsRef<str> for $name {
fn as_ref(&self) -> &str {
&self.0
}
}
};
}
/// Stable cross-device podcast identity: `pod1:<52 lowercase base32 chars>`.
///
/// Derived as `BLAKE3("furumi-podcast:podcast:v1" || 0x00 || canonical_feed_url)`
/// from the URL used by the **first** subscription. The id never changes when
/// the feed moves; new URLs are recorded as aliases (see
/// [`super::resolve_podcast_id`]). Deserialization validates the format.
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)]
#[serde(try_from = "String", into = "String")]
pub struct PodcastId(String);
string_id!(PodcastId, PODCAST_ID_PREFIX, "podcast");
impl PodcastId {
/// Derives the id of a podcast first subscribed through `url`.
pub fn from_feed_url(url: &FeedUrl) -> Self {
Self(encode(
PODCAST_ID_PREFIX,
&digest(PODCAST_ID_DOMAIN, &[url.as_str().as_bytes()]),
))
}
}
/// Stable cross-device episode identity: `ep1:<52 lowercase base32 chars>`.
///
/// Derived as `BLAKE3("furumi-podcast:episode:v1" || 0x00 || podcast_id || key)`
/// where `key` is the material of an [`EpisodeKey`]. Title, artwork or audio
/// URL changes never change the id of an existing episode.
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)]
#[serde(try_from = "String", into = "String")]
pub struct EpisodeId(String);
string_id!(EpisodeId, EPISODE_ID_PREFIX, "episode");
impl EpisodeId {
/// Derives an episode id from a key inside `podcast`.
pub fn derive(podcast: &PodcastId, key: &EpisodeKey) -> Self {
let material = key.material();
Self(encode(
EPISODE_ID_PREFIX,
&digest(
EPISODE_ID_DOMAIN,
&[podcast.as_str().as_bytes(), material.as_bytes()],
),
))
}
/// Derives the id of an episode with an RSS GUID (or Atom `<id>`).
///
/// Returns `None` when the GUID is blank or longer than [`MAX_GUID_BYTES`].
pub fn from_guid(podcast: &PodcastId, guid: &str) -> Option<Self> {
EpisodeKey::guid(guid).map(|key| Self::derive(podcast, &key))
}
}
/// Key material from which an [`EpisodeId`] is derived.
///
/// | Variant | Material |
/// | --- | --- |
/// | `Guid` | `guid:` + GUID trimmed of surrounding whitespace |
/// | `Link` | `link:` + `<link>` trimmed |
/// | `DateTitle` | `date-title:` + unix seconds + `\0` + [`super::normalize_episode_title`] |
/// | `Enclosure` | `enclosure:` + enclosure URL trimmed |
///
/// GUIDs and URLs are otherwise byte-for-byte: GUIDs are case-sensitive.
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub enum EpisodeKey {
/// RSS `<guid>` or Atom `<id>`.
Guid(String),
/// Item `<link>`.
Link(String),
/// Publication time (whole unix seconds) and normalized title.
DateTitle {
/// Publication time in unix seconds.
published_at_secs: i64,
/// Title normalized by [`super::normalize_episode_title`].
normalized_title: String,
},
/// Enclosure (audio) URL.
Enclosure(String),
}
impl EpisodeKey {
/// GUID key; `None` for a blank or oversized GUID.
pub fn guid(guid: &str) -> Option<Self> {
let guid = guid.trim();
(!guid.is_empty() && guid.len() <= MAX_GUID_BYTES).then(|| Self::Guid(guid.to_string()))
}
/// Link key; `None` for a blank or oversized link.
pub fn link(link: &str) -> Option<Self> {
let link = link.trim();
(!link.is_empty() && link.len() <= super::MAX_URL_BYTES)
.then(|| Self::Link(link.to_string()))
}
/// Date + title key; `None` when the title normalizes to nothing.
pub fn date_title(published_at_ms: i64, title: &str) -> Option<Self> {
let normalized_title = super::normalize_episode_title(title);
(!normalized_title.is_empty()).then(|| Self::DateTitle {
published_at_secs: published_at_ms.div_euclid(1_000),
normalized_title,
})
}
/// Enclosure key; `None` for a blank or oversized URL.
pub fn enclosure(url: &str) -> Option<Self> {
let url = url.trim();
(!url.is_empty() && url.len() <= super::MAX_URL_BYTES)
.then(|| Self::Enclosure(url.to_string()))
}
/// Exact bytes hashed after the podcast id.
pub fn material(&self) -> String {
match self {
Self::Guid(guid) => format!("guid:{guid}"),
Self::Link(link) => format!("link:{link}"),
Self::DateTitle {
published_at_secs,
normalized_title,
} => format!("date-title:{published_at_secs}\0{normalized_title}"),
Self::Enclosure(url) => format!("enclosure:{url}"),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
fn podcast() -> PodcastId {
PodcastId::from_feed_url(&FeedUrl::parse("https://example.test/feed.xml").unwrap())
}
#[test]
fn podcast_id_is_domain_separated_blake3() {
let url = FeedUrl::parse("https://example.test/feed.xml").unwrap();
let mut input = b"furumi-podcast:podcast:v1\0".to_vec();
input.extend_from_slice(url.as_str().as_bytes());
let expected = blake3::hash(&input);
let id = PodcastId::from_feed_url(&url);
assert_eq!(&id.digest(), expected.as_bytes());
assert_eq!(id.as_str().len(), PODCAST_ID_PREFIX.len() + DIGEST_CHARS);
assert!(id.as_str()[5..].bytes().all(|b| !b.is_ascii_uppercase()));
}
#[test]
fn podcast_id_golden_value() {
// Golden vector: changing it breaks every stored and synced id.
let id = podcast();
let mut input = b"furumi-podcast:podcast:v1\0".to_vec();
input.extend_from_slice(b"https://example.test/feed.xml");
let mut body = data_encoding::BASE32_NOPAD.encode(blake3::hash(&input).as_bytes());
body.make_ascii_lowercase();
assert_eq!(id.as_str(), format!("pod1:{body}"));
}
#[test]
fn episode_id_follows_the_plan_formula() {
let podcast = podcast();
let id = EpisodeId::from_guid(&podcast, " abc-123 ").unwrap();
let mut input = b"furumi-podcast:episode:v1\0".to_vec();
input.extend_from_slice(podcast.as_str().as_bytes());
input.extend_from_slice(b"guid:abc-123");
assert_eq!(&id.digest(), blake3::hash(&input).as_bytes());
// GUIDs are case-sensitive and podcast-scoped.
assert_ne!(EpisodeId::from_guid(&podcast, "ABC-123"), Some(id.clone()));
let other = PodcastId::from_feed_url(&FeedUrl::parse("https://other.test/").unwrap());
assert_ne!(EpisodeId::from_guid(&other, "abc-123"), Some(id));
}
#[test]
fn keys_are_distinct_per_kind() {
let podcast = podcast();
let value = "https://example.test/e1";
let link = EpisodeId::derive(&podcast, &EpisodeKey::link(value).unwrap());
let guid = EpisodeId::derive(&podcast, &EpisodeKey::guid(value).unwrap());
let enclosure = EpisodeId::derive(&podcast, &EpisodeKey::enclosure(value).unwrap());
assert_ne!(link, guid);
assert_ne!(link, enclosure);
assert_ne!(guid, enclosure);
}
#[test]
fn blank_and_oversized_keys_are_rejected() {
assert!(EpisodeKey::guid(" ").is_none());
assert!(EpisodeKey::guid(&"g".repeat(MAX_GUID_BYTES)).is_some());
assert!(EpisodeKey::guid(&"g".repeat(MAX_GUID_BYTES + 1)).is_none());
assert!(EpisodeKey::date_title(0, " -- ").is_none());
assert!(EpisodeKey::link("").is_none());
}
#[test]
fn date_title_uses_whole_seconds_and_normalized_title() {
let a = EpisodeKey::date_title(1_700_000_000_123, "Hello, World!").unwrap();
let b = EpisodeKey::date_title(1_700_000_000_999, "hello world").unwrap();
assert_eq!(a, b);
let c = EpisodeKey::date_title(-1, "x").unwrap();
assert_eq!(
c,
EpisodeKey::DateTitle {
published_at_secs: -1,
normalized_title: "x".to_string()
}
);
}
#[test]
fn parsing_validates_and_round_trips() {
let id = podcast();
assert_eq!(PodcastId::parse(id.as_str()).unwrap(), id);
let json = serde_json::to_string(&id).unwrap();
assert_eq!(serde_json::from_str::<PodcastId>(&json).unwrap(), id);
let upper = id.as_str().to_ascii_uppercase().replace("POD1:", "pod1:");
assert!(PodcastId::parse(&upper).is_err());
assert!(PodcastId::parse(&id.as_str()[..id.as_str().len() - 1]).is_err());
assert!(PodcastId::parse(&format!("{id}a")).is_err());
assert!(PodcastId::parse("pod1:").is_err());
assert!(PodcastId::parse("").is_err());
assert!(EpisodeId::parse(id.as_str()).is_err());
assert!(serde_json::from_str::<PodcastId>("\"pod1:xyz\"").is_err());
// Non-canonical trailing bits: last char encodes 1 data bit + 4 zero bits.
let mut chars: Vec<char> = id.as_str().chars().collect();
let last = chars.len() - 1;
chars[last] = if chars[last] == 'b' { 'c' } else { 'b' };
let tampered: String = chars.into_iter().collect();
assert!(PodcastId::parse(&tampered).is_err());
}
}
+604
View File
@@ -0,0 +1,604 @@
//! Episode identity resolution, including the documented no-GUID rule.
use std::collections::{BTreeMap, BTreeSet};
use serde::{Deserialize, Serialize};
use super::{EpisodeId, EpisodeKey, PodcastId};
/// Normalizes an episode title for the `(date, title)` identity key.
///
/// This is [`crate::normalize_name`]: NFKC, lowercase, non-alphanumeric runs
/// collapsed to one space, trimmed. It is part of the episode-id derivation.
pub fn normalize_episode_title(title: &str) -> String {
crate::normalize_name(title)
}
/// Fields of a feed item (or of a stored episode) that identify it.
///
/// The parser in `podcast::feed` produces these hints; stores keep them for
/// known episodes so that a later refresh can recognize items without GUID.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct EpisodeIdentityHints {
/// RSS `<guid>` / Atom `<id>`, when present.
#[serde(default)]
pub guid: Option<String>,
/// Enclosure (audio) URL.
#[serde(default)]
pub enclosure_url: Option<String>,
/// Item `<link>`.
#[serde(default)]
pub link: Option<String>,
/// Publication time in unix milliseconds.
#[serde(default)]
pub published_at_ms: Option<i64>,
/// Item title.
#[serde(default)]
pub title: Option<String>,
}
impl EpisodeIdentityHints {
fn guid_key(&self) -> Option<EpisodeKey> {
self.guid.as_deref().and_then(EpisodeKey::guid)
}
fn enclosure(&self) -> Option<&str> {
trimmed(self.enclosure_url.as_deref())
}
fn link(&self) -> Option<&str> {
trimmed(self.link.as_deref())
}
fn date_title(&self) -> Option<(i64, String)> {
let published = self.published_at_ms?.div_euclid(1_000);
let title = normalize_episode_title(self.title.as_deref()?);
(!title.is_empty()).then_some((published, title))
}
}
/// One matching stage of the no-GUID rule: does `known` match `item`?
type StageMatcher<'f> = &'f dyn Fn(&Usable<'_>, &EpisodeIdentityHints) -> bool;
struct Usable<'a> {
link: Option<&'a str>,
enclosure: Option<&'a str>,
date_title: Option<(i64, String)>,
}
fn trimmed(value: Option<&str>) -> Option<&str> {
value.map(str::trim).filter(|value| !value.is_empty())
}
/// An episode already stored for the podcast, with the hints it was last seen with.
///
/// Include episodes learned through sync (progress operations, queue items)
/// as well as episodes from earlier RSS refreshes.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct KnownEpisode {
/// Existing stable id.
pub episode_id: EpisodeId,
/// Identity hints recorded for it.
pub hints: EpisodeIdentityHints,
}
/// Resolves the id of a single feed item. See [`resolve_episode_ids`].
///
/// A single item has no feed context, so its `<link>` is only used when it
/// differs from `channel_link`. Prefer [`resolve_episode_ids`] for a whole
/// feed refresh.
pub fn resolve_episode_id(
podcast: &PodcastId,
channel_link: Option<&str>,
item: &EpisodeIdentityHints,
known: &[KnownEpisode],
) -> Option<EpisodeId> {
resolve_episode_ids(podcast, channel_link, std::slice::from_ref(item), known)
.pop()
.flatten()
}
/// Resolves stable ids for the items of one feed document.
///
/// Returns one entry per item, in order; `None` marks an item that cannot be
/// identified (no usable key) or that duplicates an earlier item of the same
/// document. The rule (normative, see `PODCASTS.md`):
///
/// 1. An item with a GUID gets `EpisodeId::from_guid`. A GUID never matches
/// a different known episode, so the id is independent of local history.
/// 2. Items without a GUID then look for known episodes in three stages run
/// over the whole document: same enclosure URL, then same `<link>`, then
/// same publication second and [`normalize_episode_title`]. A stage
/// assigns matches for all remaining items before the next stage starts;
/// among candidates the smallest id is reused. A known episode already
/// claimed (by a GUID item or an earlier stage) is not a candidate, so the
/// result does not depend on item order.
/// 3. Otherwise a new id is derived from the first available key: `<link>`,
/// then `(date, title)`, then enclosure URL.
///
/// A `<link>` is only identifying when it is unique within the document and
/// differs from the channel link; many feeds repeat the show homepage in every
/// item. The same applies to enclosure URLs and `(date, title)` pairs shared
/// by several GUID-less items.
pub fn resolve_episode_ids(
podcast: &PodcastId,
channel_link: Option<&str>,
items: &[EpisodeIdentityHints],
known: &[KnownEpisode],
) -> Vec<Option<EpisodeId>> {
let channel_link = trimmed(channel_link);
let mut link_counts: BTreeMap<&str, usize> = BTreeMap::new();
let mut enclosure_counts: BTreeMap<&str, usize> = BTreeMap::new();
let mut date_title_counts: BTreeMap<(i64, String), usize> = BTreeMap::new();
for item in items {
if let Some(link) = item.link() {
*link_counts.entry(link).or_default() += 1;
}
if item.guid_key().is_none() {
if let Some(url) = item.enclosure() {
*enclosure_counts.entry(url).or_default() += 1;
}
if let Some(pair) = item.date_title() {
*date_title_counts.entry(pair).or_default() += 1;
}
}
}
let usable: Vec<Usable<'_>> = items
.iter()
.map(|item| Usable {
link: item
.link()
.filter(|link| Some(*link) != channel_link && link_counts.get(link) == Some(&1)),
enclosure: item
.enclosure()
.filter(|url| enclosure_counts.get(url) == Some(&1)),
date_title: item
.date_title()
.filter(|pair| date_title_counts.get(pair) == Some(&1)),
})
.collect();
let mut claimed: BTreeSet<EpisodeId> = BTreeSet::new();
let mut out: Vec<Option<EpisodeId>> = vec![None; items.len()];
// Pass 1: GUID items claim their derived ids first.
for (index, item) in items.iter().enumerate() {
if let Some(key) = item.guid_key() {
let id = EpisodeId::derive(podcast, &key);
if claimed.insert(id.clone()) {
out[index] = Some(id);
}
}
}
// Pass 2: GUID-less items reuse a matching known episode. Stages run
// across the whole document: every enclosure match is assigned before any
// link match, and every link match before any (date, title) match, so a
// weak match of an earlier item cannot steal the known episode that a
// later item matches strongly. Within one stage an item's key is unique in
// the document, so two items never compete for the same known episode
// and the result does not depend on item order.
let mut pending: Vec<usize> = (0..items.len())
.filter(|index| items[*index].guid_key().is_none())
.collect();
let stages: [StageMatcher<'_>; 3] = [
&|own, candidate| {
own.enclosure
.is_some_and(|url| candidate.enclosure() == Some(url))
},
&|own, candidate| own.link.is_some_and(|link| candidate.link() == Some(link)),
&|own, candidate| {
own.date_title
.as_ref()
.is_some_and(|pair| candidate.date_title().as_ref() == Some(pair))
},
];
for matches in stages {
let mut unmatched = Vec::with_capacity(pending.len());
for index in pending {
let own = &usable[index];
let reused = known
.iter()
.filter(|known| !claimed.contains(&known.episode_id) && matches(own, &known.hints))
.map(|known| &known.episode_id)
.min()
.cloned();
match reused {
Some(id) => {
claimed.insert(id.clone());
out[index] = Some(id);
}
None => unmatched.push(index),
}
}
pending = unmatched;
}
// Pass 3: derive new ids from the first available fallback key.
for index in pending {
let own = &usable[index];
let key = own
.link
.and_then(EpisodeKey::link)
.or_else(|| {
own.date_title
.as_ref()
.map(|(secs, title)| EpisodeKey::DateTitle {
published_at_secs: *secs,
normalized_title: title.clone(),
})
})
.or_else(|| own.enclosure.and_then(EpisodeKey::enclosure));
if let Some(key) = key {
let id = EpisodeId::derive(podcast, &key);
if claimed.insert(id.clone()) {
out[index] = Some(id);
}
}
}
out
}
#[cfg(test)]
mod tests {
use super::*;
use crate::podcast::FeedUrl;
fn podcast() -> PodcastId {
PodcastId::from_feed_url(&FeedUrl::parse("https://example.test/feed").unwrap())
}
fn hints(
guid: Option<&str>,
enclosure: Option<&str>,
link: Option<&str>,
date: Option<i64>,
title: Option<&str>,
) -> EpisodeIdentityHints {
EpisodeIdentityHints {
guid: guid.map(str::to_string),
enclosure_url: enclosure.map(str::to_string),
link: link.map(str::to_string),
published_at_ms: date,
title: title.map(str::to_string),
}
}
fn derived(key: EpisodeKey) -> Option<EpisodeId> {
Some(EpisodeId::derive(&podcast(), &key))
}
#[test]
fn guid_wins_and_ignores_history() {
let item = hints(Some("g1"), Some("https://cdn/1.mp3"), None, None, Some("T"));
let known = [KnownEpisode {
episode_id: EpisodeId::from_guid(&podcast(), "other").unwrap(),
hints: hints(None, Some("https://cdn/1.mp3"), None, None, None),
}];
assert_eq!(
resolve_episode_id(&podcast(), None, &item, &known),
derived(EpisodeKey::guid("g1").unwrap())
);
}
#[test]
fn title_cover_and_audio_url_changes_keep_the_guid_id() {
let before = hints(
Some("g1"),
Some("https://cdn/a.mp3"),
None,
Some(1),
Some("Old"),
);
let after = hints(
Some("g1"),
Some("https://cdn/b.mp3"),
None,
Some(2),
Some("New"),
);
assert_eq!(
resolve_episode_id(&podcast(), None, &before, &[]),
resolve_episode_id(&podcast(), None, &after, &[])
);
}
#[test]
fn fallback_key_order_for_new_items() {
let p = podcast();
let with_link = hints(
None,
Some("https://cdn/1.mp3"),
Some("https://e/1"),
Some(5_000),
Some("A"),
);
assert_eq!(
resolve_episode_id(&p, None, &with_link, &[]),
derived(EpisodeKey::link("https://e/1").unwrap())
);
let date_title = hints(
None,
Some("https://cdn/1.mp3"),
None,
Some(5_000),
Some("A"),
);
assert_eq!(
resolve_episode_id(&p, None, &date_title, &[]),
derived(EpisodeKey::date_title(5_000, "A").unwrap())
);
let enclosure_only = hints(None, Some("https://cdn/1.mp3"), None, None, Some("A"));
assert_eq!(
resolve_episode_id(&p, None, &enclosure_only, &[]),
derived(EpisodeKey::enclosure("https://cdn/1.mp3").unwrap())
);
assert_eq!(
resolve_episode_id(&p, None, &EpisodeIdentityHints::default(), &[]),
None
);
}
#[test]
fn guidless_episode_keeps_its_id_after_title_and_url_changes() {
let p = podcast();
// First seen: no link, so the id comes from (date, title).
let first = hints(
None,
Some("https://cdn/a.mp3"),
None,
Some(9_000),
Some("Pilot"),
);
let id = resolve_episode_id(&p, None, &first, &[]).unwrap();
let known = [KnownEpisode {
episode_id: id.clone(),
hints: first.clone(),
}];
// Title changed: matched by enclosure.
let retitled = hints(
None,
Some("https://cdn/a.mp3"),
None,
Some(9_000),
Some("Pilot (remastered)"),
);
assert_eq!(
resolve_episode_id(&p, None, &retitled, &known),
Some(id.clone())
);
// Audio URL changed: matched by (date, title).
let moved = hints(
None,
Some("https://cdn2/a.mp3"),
None,
Some(9_999),
Some("PILOT!"),
);
assert_eq!(
resolve_episode_id(&p, None, &moved, &known),
Some(id.clone())
);
// Link appeared later: the enclosure still matches first.
let linked = hints(
None,
Some("https://cdn/a.mp3"),
Some("https://e/pilot"),
None,
None,
);
assert_eq!(resolve_episode_id(&p, None, &linked, &known), Some(id));
}
#[test]
fn shared_channel_links_are_not_identifying() {
let p = podcast();
let home = "https://show.test/";
let items = [
hints(None, Some("https://cdn/1.mp3"), Some(home), None, None),
hints(None, Some("https://cdn/2.mp3"), Some(home), None, None),
];
let ids = resolve_episode_ids(&p, Some(home), &items, &[]);
assert_eq!(
ids[0],
derived(EpisodeKey::enclosure("https://cdn/1.mp3").unwrap())
);
assert_eq!(
ids[1],
derived(EpisodeKey::enclosure("https://cdn/2.mp3").unwrap())
);
// The same holds without the channel link: the link repeats.
assert_eq!(resolve_episode_ids(&p, None, &items, &[]), ids);
// A single item whose link equals the channel link.
assert_eq!(resolve_episode_id(&p, Some(home), &items[0], &[]), ids[0]);
}
#[test]
fn duplicates_in_one_document_are_dropped() {
let p = podcast();
let items = [
hints(Some("g"), Some("https://cdn/1.mp3"), None, None, None),
hints(Some("g"), Some("https://cdn/2.mp3"), None, None, None),
hints(None, Some("https://cdn/3.mp3"), None, None, None),
hints(None, Some("https://cdn/3.mp3"), None, None, None),
];
let ids = resolve_episode_ids(&p, None, &items, &[]);
assert!(ids[0].is_some());
assert_eq!(ids[1], None);
// The shared enclosure is not identifying: both GUID-less items have
// no usable key.
assert_eq!(ids[2], None);
assert_eq!(ids[3], None);
}
#[test]
fn a_known_episode_is_claimed_once() {
let p = podcast();
let known_id = derived(EpisodeKey::enclosure("https://cdn/old.mp3").unwrap()).unwrap();
let known = [KnownEpisode {
episode_id: known_id.clone(),
hints: hints(
None,
Some("https://cdn/old.mp3"),
None,
Some(1_000),
Some("Same"),
),
}];
// Two new items both match by (date, title) only if unique; here
// they share it, so neither matches; the enclosure one matches.
let items = [
hints(
None,
Some("https://cdn/x.mp3"),
None,
Some(1_000),
Some("Same"),
),
hints(
None,
Some("https://cdn/old.mp3"),
None,
Some(1_000),
Some("Same"),
),
];
let ids = resolve_episode_ids(&p, None, &items, &known);
assert_eq!(ids[1], Some(known_id.clone()));
assert_eq!(
ids[0],
derived(EpisodeKey::enclosure("https://cdn/x.mp3").unwrap())
);
assert_ne!(ids[0], ids[1]);
}
#[test]
fn ambiguous_known_matches_pick_the_smallest_id_in_any_order() {
let p = podcast();
let a = EpisodeId::from_guid(&p, "a").unwrap();
let b = EpisodeId::from_guid(&p, "b").unwrap();
let same = hints(None, Some("https://cdn/1.mp3"), None, None, None);
let k1 = KnownEpisode {
episode_id: a.clone(),
hints: same.clone(),
};
let k2 = KnownEpisode {
episode_id: b.clone(),
hints: same.clone(),
};
let expected = Some(a.clone().min(b.clone()));
assert_eq!(
resolve_episode_id(&p, None, &same, &[k1.clone(), k2.clone()]),
expected
);
assert_eq!(resolve_episode_id(&p, None, &same, &[k2, k1]), expected);
}
#[test]
fn result_is_independent_of_known_order() {
let p = podcast();
let items: Vec<EpisodeIdentityHints> = (0..6)
.map(|i| {
hints(
None,
Some(&format!("https://cdn/{i}.mp3")),
Some(&format!("https://e/{i}")),
Some(i * 1_000),
Some(&format!("Ep {i}")),
)
})
.collect();
let mut known: Vec<KnownEpisode> = items
.iter()
.enumerate()
.map(|(i, h)| KnownEpisode {
episode_id: EpisodeId::from_guid(&p, &format!("legacy-{i}")).unwrap(),
hints: h.clone(),
})
.collect();
let forward = resolve_episode_ids(&p, None, &items, &known);
known.reverse();
assert_eq!(resolve_episode_ids(&p, None, &items, &known), forward);
assert!(forward.iter().all(Option::is_some));
}
#[test]
fn strong_matches_beat_earlier_weak_matches_in_any_item_order() {
let p = podcast();
let k1 = EpisodeId::from_guid(&p, "legacy-k1").unwrap();
let known = [KnownEpisode {
episode_id: k1.clone(),
hints: hints(
None,
Some("https://cdn/e1.mp3"),
None,
Some(5_000),
Some("Same"),
),
}];
// A new item that only matches K1 by (date, title), and the real K1
// (same enclosure) renamed and re-dated by the publisher.
let weak = hints(
None,
Some("https://cdn/new.mp3"),
None,
Some(5_000),
Some("Same"),
);
let strong = hints(
None,
Some("https://cdn/e1.mp3"),
None,
Some(9_000),
Some("Renamed"),
);
let forward = resolve_episode_ids(&p, None, &[weak.clone(), strong.clone()], &known);
assert_eq!(forward[1].as_ref(), Some(&k1), "enclosure match keeps K1");
assert_ne!(forward[0].as_ref(), Some(&k1));
let reverse = resolve_episode_ids(&p, None, &[strong, weak], &known);
assert_eq!(reverse, vec![forward[1].clone(), forward[0].clone()]);
}
#[test]
fn result_is_independent_of_item_order() {
let p = podcast();
let items: Vec<EpisodeIdentityHints> = (0..5)
.map(|i| {
hints(
None,
Some(&format!("https://cdn/{i}.mp3")),
(i % 2 == 0).then(|| format!("https://e/{i}")).as_deref(),
Some((i % 3) * 1_000),
Some(&format!("Ep {}", i % 2)),
)
})
.collect();
// Known episodes whose hints overlap different items at different
// stages.
let known: Vec<KnownEpisode> = (0..5)
.map(|i| KnownEpisode {
episode_id: EpisodeId::from_guid(&p, &format!("legacy-{i}")).unwrap(),
hints: hints(
None,
Some(&format!("https://cdn/{}.mp3", (i + 2) % 7)),
Some(&format!("https://e/{}", (i + 1) % 5)),
Some((i % 3) * 1_000),
Some(&format!("Ep {}", i % 2)),
),
})
.collect();
let reference = resolve_episode_ids(&p, None, &items, &known);
let mut order: Vec<usize> = (0..items.len()).collect();
for _ in 0..20 {
order.rotate_left(1);
let middle = order.len() / 2;
order.swap(0, middle);
let shuffled: Vec<EpisodeIdentityHints> =
order.iter().map(|i| items[*i].clone()).collect();
let result = resolve_episode_ids(&p, None, &shuffled, &known);
for (position, index) in order.iter().enumerate() {
assert_eq!(result[position], reference[*index], "order {order:?}");
}
}
}
}
+493
View File
@@ -0,0 +1,493 @@
//! Podcast and episode metadata DTOs.
use serde::{Deserialize, Serialize};
use super::{
EpisodeId, EpisodeIdentityHints, FeedUrl, MAX_DESCRIPTION_CHARS, MAX_GUID_BYTES,
MAX_MIME_BYTES, MAX_TITLE_CHARS, MAX_URL_BYTES, PodcastError, PodcastId, PodcastResult,
bounded_text, check_opt_text, check_opt_url, check_text, check_url,
};
/// `itunes:episodeType`.
///
/// Unknown values deserialize (and parse) as [`EpisodeType::Full`], the
/// iTunes default, so publishers cannot break the wire format.
#[derive(
Debug, Clone, Copy, Default, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize,
)]
#[serde(rename_all = "snake_case", from = "String")]
pub enum EpisodeType {
/// Regular episode.
#[default]
Full,
/// Trailer or teaser.
Trailer,
/// Bonus content.
Bonus,
}
impl EpisodeType {
/// Parses an `itunes:episodeType` value, case-insensitively.
pub fn parse(value: &str) -> Self {
match value.trim().to_ascii_lowercase().as_str() {
"trailer" => Self::Trailer,
"bonus" => Self::Bonus,
_ => Self::Full,
}
}
}
impl From<String> for EpisodeType {
fn from(value: String) -> Self {
Self::parse(&value)
}
}
/// Channel metadata of a podcast, as last fetched from its feed.
///
/// Synced inside subscription operations so that another trusted device can
/// show and refresh the podcast without the user typing the URL again. It is
/// private user data: never publish it outside the trusted-device group.
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)]
pub struct PodcastMeta {
/// Canonical URL the feed is currently fetched from.
pub feed_url: FeedUrl,
/// Podcast title.
pub title: String,
/// Author.
#[serde(default)]
pub author: Option<String>,
/// Plain-text description, at most [`MAX_DESCRIPTION_CHARS`].
#[serde(default)]
pub description: Option<String>,
/// Publisher web page.
#[serde(default)]
pub link: Option<String>,
/// Artwork URL.
#[serde(default)]
pub image_url: Option<String>,
}
impl PodcastMeta {
/// Validates contract bounds.
pub fn validate(&self) -> PodcastResult<()> {
check_text("podcast title", &self.title, MAX_TITLE_CHARS, true)?;
check_opt_text("podcast author", self.author.as_deref(), MAX_TITLE_CHARS)?;
check_opt_text(
"podcast description",
self.description.as_deref(),
MAX_DESCRIPTION_CHARS,
)?;
if self
.link
.as_ref()
.is_some_and(|link| link.len() > MAX_URL_BYTES)
{
return Err(PodcastError::InvalidField {
field: "podcast link",
reason: "is too long",
});
}
check_opt_url("podcast image_url", self.image_url.as_deref())
}
/// Returns a copy that fits the contract bounds.
///
/// Text is truncated with [`bounded_text`]; invalid optional URLs are
/// dropped; an empty title becomes `"Untitled podcast"`.
pub fn bounded(&self) -> Self {
Self {
feed_url: self.feed_url.clone(),
title: bounded_title(&self.title, "Untitled podcast"),
author: bounded_opt(self.author.as_deref(), MAX_TITLE_CHARS),
description: bounded_opt(self.description.as_deref(), MAX_DESCRIPTION_CHARS),
link: self
.link
.clone()
.filter(|link| link.len() <= MAX_URL_BYTES && !link.trim().is_empty()),
image_url: valid_url(self.image_url.as_deref()),
}
}
}
/// Full episode metadata, as stored by clients and refreshed from RSS.
///
/// Publisher metadata never touches personal progress or history: those are
/// separate records keyed by [`EpisodeId`].
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct EpisodeMeta {
/// Stable episode id.
pub episode_id: EpisodeId,
/// Owning podcast.
pub podcast_id: PodcastId,
/// RSS GUID / Atom id, when present.
#[serde(default)]
pub guid: Option<String>,
/// Episode title.
pub title: String,
/// Plain-text description, at most [`MAX_DESCRIPTION_CHARS`].
#[serde(default)]
pub description: Option<String>,
/// `itunes:season`. Episodes without a season form their own group.
#[serde(default)]
pub season: Option<u32>,
/// `itunes:episode`.
#[serde(default)]
pub episode_number: Option<u32>,
/// `itunes:episodeType`.
#[serde(default)]
pub episode_type: EpisodeType,
/// Publication time in unix milliseconds.
#[serde(default)]
pub published_at_ms: Option<i64>,
/// Duration announced by the publisher, in milliseconds.
#[serde(default)]
pub duration_ms: Option<i64>,
/// Absolute http(s) audio URL at the publisher.
pub audio_url: String,
/// Enclosure MIME type.
#[serde(default)]
pub mime: Option<String>,
/// Enclosure length in bytes.
#[serde(default)]
pub byte_len: Option<u64>,
/// Episode artwork URL.
#[serde(default)]
pub image_url: Option<String>,
/// Episode web page.
#[serde(default)]
pub link: Option<String>,
}
impl EpisodeMeta {
/// Validates contract bounds.
pub fn validate(&self) -> PodcastResult<()> {
if self
.guid
.as_ref()
.is_some_and(|guid| guid.len() > MAX_GUID_BYTES)
{
return Err(PodcastError::InvalidField {
field: "episode guid",
reason: "is too long",
});
}
check_text("episode title", &self.title, MAX_TITLE_CHARS, true)?;
check_opt_text(
"episode description",
self.description.as_deref(),
MAX_DESCRIPTION_CHARS,
)?;
check_duration(self.duration_ms)?;
check_url("episode audio_url", &self.audio_url)?;
check_mime(self.mime.as_deref())?;
check_opt_url("episode image_url", self.image_url.as_deref())?;
if self
.link
.as_ref()
.is_some_and(|link| link.len() > MAX_URL_BYTES)
{
return Err(PodcastError::InvalidField {
field: "episode link",
reason: "is too long",
});
}
Ok(())
}
/// Identity hints to keep for the no-GUID matching rule.
pub fn identity_hints(&self) -> EpisodeIdentityHints {
EpisodeIdentityHints {
guid: self.guid.clone(),
enclosure_url: Some(self.audio_url.clone()),
link: self.link.clone(),
published_at_ms: self.published_at_ms,
title: Some(self.title.clone()),
}
}
/// Builds the self-contained queue reference for this episode.
///
/// The episode artwork falls back to the podcast artwork.
pub fn to_ref(&self, podcast: &PodcastMeta) -> PodcastEpisodeRef {
PodcastEpisodeRef {
podcast_id: self.podcast_id.clone(),
episode_id: self.episode_id.clone(),
feed_url: podcast.feed_url.clone(),
podcast_title: bounded_title(&podcast.title, "Untitled podcast"),
episode_title: bounded_title(&self.title, "Untitled episode"),
audio_url: self.audio_url.clone(),
mime: self
.mime
.clone()
.filter(|mime| check_mime(Some(mime)).is_ok()),
duration_ms: self.duration_ms.filter(|ms| *ms > 0),
published_at_ms: self.published_at_ms,
season: self.season,
episode_number: self.episode_number,
image_url: valid_url(self.image_url.as_deref())
.or_else(|| valid_url(podcast.image_url.as_deref())),
progress_generation: None,
}
}
}
/// Self-contained, bounded reference to a podcast episode.
///
/// Carried by playback queues (`PlaybackSource::PodcastEpisode`) and by
/// progress operations, so a receiving trusted device can display, resume and
/// play the episode without being subscribed and without its own copy of the
/// feed. Apart from the optional handoff hint
/// [`progress_generation`](Self::progress_generation) it contains no personal
/// state. It is private: Jam snapshots replace it with an opaque placeholder.
///
/// If the audio URL went stale, refresh the feed at `feed_url` and look the
/// episode up by `episode_id`.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct PodcastEpisodeRef {
/// Owning podcast.
pub podcast_id: PodcastId,
/// Episode.
pub episode_id: EpisodeId,
/// Canonical feed URL used to refresh the episode.
pub feed_url: FeedUrl,
/// Podcast title for display.
pub podcast_title: String,
/// Episode title for display.
pub episode_title: String,
/// Absolute http(s) audio URL at the publisher.
pub audio_url: String,
/// Enclosure MIME type.
#[serde(default)]
pub mime: Option<String>,
/// Announced duration in milliseconds.
#[serde(default)]
pub duration_ms: Option<i64>,
/// Publication time in unix milliseconds.
#[serde(default)]
pub published_at_ms: Option<i64>,
/// `itunes:season`.
#[serde(default)]
pub season: Option<u32>,
/// `itunes:episode`.
#[serde(default)]
pub episode_number: Option<u32>,
/// Episode or podcast artwork URL.
#[serde(default)]
pub image_url: Option<String>,
/// Greatest progress generation the sender has seen for this episode.
///
/// Set on queue items of a playback state (in particular on handoff),
/// because the playback state can arrive before the sender's progress
/// operations. A device that starts playing the item writes its explicit
/// action with
/// [`ProgressRecord::explicit_generation`](super::ProgressRecord::explicit_generation)`(max(local, progress_generation))`.
/// Ignored in progress and listening operations, whose own generation is
/// authoritative. Not part of the episode identity.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub progress_generation: Option<u64>,
}
impl PodcastEpisodeRef {
/// Validates contract bounds. Receivers must drop invalid references.
pub fn validate(&self) -> PodcastResult<()> {
check_text("podcast title", &self.podcast_title, MAX_TITLE_CHARS, true)?;
check_text("episode title", &self.episode_title, MAX_TITLE_CHARS, true)?;
check_url("episode audio_url", &self.audio_url)?;
check_mime(self.mime.as_deref())?;
check_duration(self.duration_ms)?;
check_opt_url("episode image_url", self.image_url.as_deref())
}
/// Identity hints known from this reference.
///
/// Add them to the known-episode set before resolving ids from a feed so
/// a GUID-less episode learned through sync keeps its id.
pub fn identity_hints(&self) -> EpisodeIdentityHints {
EpisodeIdentityHints {
guid: None,
enclosure_url: Some(self.audio_url.clone()),
link: None,
published_at_ms: self.published_at_ms,
title: Some(self.episode_title.clone()),
}
}
}
fn check_duration(duration_ms: Option<i64>) -> PodcastResult<()> {
if duration_ms.is_some_and(|ms| ms < 0) {
return Err(PodcastError::InvalidField {
field: "duration_ms",
reason: "must not be negative",
});
}
Ok(())
}
fn check_mime(mime: Option<&str>) -> PodcastResult<()> {
let Some(mime) = mime else { return Ok(()) };
if mime.is_empty() || mime.len() > MAX_MIME_BYTES || !mime.bytes().all(|b| b.is_ascii_graphic())
{
return Err(PodcastError::InvalidField {
field: "mime",
reason: "must be a short printable ASCII token",
});
}
Ok(())
}
fn bounded_title(title: &str, fallback: &str) -> String {
let title = title.trim();
if title.is_empty() {
fallback.to_string()
} else {
bounded_text(title, MAX_TITLE_CHARS)
}
}
fn bounded_opt(value: Option<&str>, max_chars: usize) -> Option<String> {
value
.map(str::trim)
.filter(|value| !value.is_empty())
.map(|value| bounded_text(value, max_chars))
}
fn valid_url(value: Option<&str>) -> Option<String> {
value
.filter(|value| super::is_http_url(value))
.map(str::to_string)
}
#[cfg(test)]
mod tests {
use super::*;
fn podcast_meta() -> PodcastMeta {
PodcastMeta {
feed_url: FeedUrl::parse("https://example.test/feed").unwrap(),
title: "Show".to_string(),
author: Some("Host".to_string()),
description: None,
link: Some("https://example.test/".to_string()),
image_url: Some("https://example.test/cover.jpg".to_string()),
}
}
fn episode() -> EpisodeMeta {
let podcast_id = PodcastId::from_feed_url(&podcast_meta().feed_url);
EpisodeMeta {
episode_id: EpisodeId::from_guid(&podcast_id, "g1").unwrap(),
podcast_id,
guid: Some("g1".to_string()),
title: "Pilot".to_string(),
description: Some("Notes".to_string()),
season: Some(2),
episode_number: Some(7),
episode_type: EpisodeType::Bonus,
published_at_ms: Some(1_700_000_000_000),
duration_ms: Some(3_600_000),
audio_url: "https://cdn.example.test/1.mp3?sig=abc".to_string(),
mime: Some("audio/mpeg".to_string()),
byte_len: Some(1234),
image_url: None,
link: Some("https://example.test/1".to_string()),
}
}
#[test]
fn episode_type_is_tolerant() {
assert_eq!(EpisodeType::parse(" Trailer "), EpisodeType::Trailer);
assert_eq!(EpisodeType::parse("BONUS"), EpisodeType::Bonus);
assert_eq!(EpisodeType::parse("full"), EpisodeType::Full);
assert_eq!(EpisodeType::parse("teaser"), EpisodeType::Full);
assert_eq!(
serde_json::to_string(&EpisodeType::Trailer).unwrap(),
"\"trailer\""
);
assert_eq!(
serde_json::from_str::<EpisodeType>("\"something-new\"").unwrap(),
EpisodeType::Full
);
}
#[test]
fn metadata_round_trips_and_validates() {
let episode = episode();
episode.validate().unwrap();
let json = serde_json::to_string(&episode).unwrap();
assert_eq!(serde_json::from_str::<EpisodeMeta>(&json).unwrap(), episode);
podcast_meta().validate().unwrap();
let mut bad = episode.clone();
bad.audio_url = "file:///tmp/x.mp3".to_string();
assert!(bad.validate().is_err());
let mut bad = episode.clone();
bad.description = Some("x".repeat(MAX_DESCRIPTION_CHARS + 1));
assert!(bad.validate().is_err());
let mut bad = episode;
bad.mime = Some("audio/mpeg; codecs=\"mp3\"".to_string());
assert!(bad.validate().is_err());
}
#[test]
fn minimal_legacy_json_uses_defaults() {
let episode = episode();
let json = format!(
r#"{{"episode_id":"{}","podcast_id":"{}","title":"T","audio_url":"https://a.test/1.mp3"}}"#,
episode.episode_id, episode.podcast_id
);
let parsed: EpisodeMeta = serde_json::from_str(&json).unwrap();
assert_eq!(parsed.episode_type, EpisodeType::Full);
assert_eq!(parsed.season, None);
}
#[test]
fn episode_ref_is_self_contained_and_bounded() {
let podcast = podcast_meta();
let mut episode = episode();
episode.title = "x".repeat(MAX_TITLE_CHARS + 10);
let reference = episode.to_ref(&podcast);
reference.validate().unwrap();
assert_eq!(reference.episode_title.chars().count(), MAX_TITLE_CHARS);
assert_eq!(reference.podcast_title, "Show");
assert_eq!(
reference.image_url.as_deref(),
Some("https://example.test/cover.jpg")
);
assert_eq!(reference.feed_url, podcast.feed_url);
let json = serde_json::to_string(&reference).unwrap();
assert!(!json.contains("description"));
assert_eq!(
serde_json::from_str::<PodcastEpisodeRef>(&json).unwrap(),
reference
);
let mut bad = reference.clone();
bad.episode_title = " ".to_string();
assert!(bad.validate().is_err());
let mut bad = reference.clone();
bad.audio_url = "javascript:alert(1)".to_string();
assert!(bad.validate().is_err());
let mut bad = reference;
bad.duration_ms = Some(-1);
assert!(bad.validate().is_err());
}
#[test]
fn refs_reject_invalid_ids_on_deserialization() {
let json = r#"{"podcast_id":"pod1:bad","episode_id":"ep1:bad","feed_url":"https://a.test/","podcast_title":"P","episode_title":"E","audio_url":"https://a.test/1.mp3"}"#;
assert!(serde_json::from_str::<PodcastEpisodeRef>(json).is_err());
}
#[test]
fn bounded_podcast_meta_fits_the_contract() {
let mut meta = podcast_meta();
meta.title = " ".to_string();
meta.description = Some("d".repeat(MAX_DESCRIPTION_CHARS * 2));
meta.image_url = Some("ftp://x".to_string());
let bounded = meta.bounded();
bounded.validate().unwrap();
assert_eq!(bounded.title, "Untitled podcast");
assert_eq!(bounded.image_url, None);
}
}
+310
View File
@@ -0,0 +1,310 @@
//! Shared podcast contract for Furumi clients.
//!
//! This module is pure: it has no SQLite, HTTP, clock, audio or UI
//! dependency. Every Furumi client (TUI, web, desktop) uses it so that
//! podcast identity, normalization, merge rules and wire DTOs are defined
//! exactly once. The normative description lives in the workspace
//! `PODCASTS.md`; this documentation summarizes the API.
//!
//! # Identity
//!
//! * [`FeedUrl`] is a canonical feed URL. [`FeedUrl::parse`] lowercases the
//! scheme and host, drops a default port and the fragment, and keeps the
//! path and **every** query parameter byte-for-byte (only non-ASCII bytes
//! are percent-encoded). Non-`http(s)` URLs, userinfo and whitespace are
//! rejected. [`FeedUrl::from_user_input`] additionally accepts the
//! `feed:`/`podcast:`/`pcast:`/`itpc:` forms and a missing scheme.
//! * [`PodcastId`] is `pod1:` followed by the lowercase base32 BLAKE3 hash of
//! `"furumi-podcast:podcast:v1\0"` and the canonical feed URL. It is
//! derived once, from the URL of the first subscription; later URLs
//! (redirects, `itunes:new-feed-url`) become feed aliases resolved by
//! [`resolve_podcast_id`].
//! * [`EpisodeId`] is `ep1:` followed by the lowercase base32 BLAKE3 hash of
//! `"furumi-podcast:episode:v1\0"`, the podcast id and a key. The key is
//! the RSS GUID when present. Without a GUID, [`resolve_episode_ids`] first
//! reuses the id of a known episode that matches by enclosure URL, `<link>`
//! or (date, normalized title) and only then derives a new id from the
//! first available fallback key.
//!
//! # Personal state
//!
//! * [`SubscriptionRecord`]: last-writer-wins by `(generation, hlc_ms, op_id)`
//! where `generation` is a per-podcast Lamport counter
//! ([`next_podcast_generation`]); unsubscribing is state, not a tombstone.
//! Feed aliases merge as a bounded grow-only set.
//! * [`ShareRecord`]: last-writer-wins with the same ordering; `shared = false`
//! is register state that is never compacted. See [`effective_share`] for
//! the rule that a (re)subscription is private.
//! * [`ProgressRecord`]: register ordered by
//! `(generation, hlc_ms, device_id, op_id)`. Explicit user actions bump the
//! generation to `max_seen + 1`; periodic checkpoints keep the generation of
//! their session, so a late checkpoint of an old session always loses.
//! * [`ListenSession`]: grow-only set keyed by `session_id`; one completion per
//! session, so re-delivery never double counts.
//!
//! # Queue items
//!
//! [`PodcastEpisodeRef`] is the bounded, self-contained reference carried by
//! playback queues and progress operations. A receiving device can show and
//! play the episode without being subscribed to the podcast.
//!
//! # Feed parser
//!
//! With the `podcast-feed` cargo feature, `feed` provides the shared tolerant
//! RSS 2.0 / RSS 1.0 / Atom parser that produces the identity hints consumed
//! by [`resolve_episode_ids`].
//!
//! ```
//! use music_dht::podcast::{EpisodeId, FeedUrl, PodcastId};
//!
//! let url = FeedUrl::parse("HTTPS://Example.COM:443/feed.xml?token=Ab#top").unwrap();
//! assert_eq!(url.as_str(), "https://example.com/feed.xml?token=Ab");
//! let podcast = PodcastId::from_feed_url(&url);
//! assert!(podcast.as_str().starts_with("pod1:"));
//! let episode = EpisodeId::from_guid(&podcast, "episode-42").unwrap();
//! assert!(episode.as_str().starts_with("ep1:"));
//! ```
mod feed_url;
mod id;
mod identity;
mod meta;
mod state;
#[cfg(feature = "podcast-feed")]
pub mod feed;
pub use feed_url::{FeedUrl, merge_feed_aliases, resolve_podcast_id};
pub use id::{
EPISODE_ID_DOMAIN, EPISODE_ID_PREFIX, EpisodeId, EpisodeKey, PODCAST_ID_DOMAIN,
PODCAST_ID_PREFIX, PodcastId,
};
pub use identity::{
EpisodeIdentityHints, KnownEpisode, normalize_episode_title, resolve_episode_id,
resolve_episode_ids,
};
pub use meta::{EpisodeMeta, EpisodeType, PodcastEpisodeRef, PodcastMeta};
pub use state::{
COMPLETION_TAIL_MAX_MS, ListenSession, MIN_RECORDED_EPISODE_LISTEN_MS, ProgressRecord,
ProgressStatus, ShareRecord, SubscriptionRecord, completed_listen_count, effective_share,
merge_listen_session, merge_progress, merge_share, merge_subscription, next_podcast_generation,
reaches_completion,
};
/// Maximum characters of a title (podcast, episode, author).
pub const MAX_TITLE_CHARS: usize = 1_000;
/// Maximum characters of a podcast or episode description.
pub const MAX_DESCRIPTION_CHARS: usize = 4_000;
/// Maximum bytes of any URL carried by the contract.
pub const MAX_URL_BYTES: usize = 2_048;
/// Maximum bytes of an RSS GUID. Longer GUIDs are treated as absent.
pub const MAX_GUID_BYTES: usize = 2_048;
/// Maximum bytes of a MIME type.
pub const MAX_MIME_BYTES: usize = 128;
/// Maximum bytes of a device, session or operation id.
pub const MAX_TOKEN_BYTES: usize = 256;
/// Maximum feed aliases kept per podcast, besides the URL the id was derived from.
pub const MAX_FEED_ALIASES: usize = 8;
/// Maximum subscription records in one sync snapshot.
pub const MAX_SNAPSHOT_SUBSCRIPTIONS: usize = 1_000;
/// Maximum share records in one sync snapshot.
pub const MAX_SNAPSHOT_SHARES: usize = 1_000;
/// Maximum progress registers in one sync snapshot.
pub const MAX_SNAPSHOT_PROGRESS: usize = 5_000;
/// Maximum listening sessions in one sync snapshot.
pub const MAX_SNAPSHOT_LISTEN_SESSIONS: usize = 2_000;
/// Serialized JSON budget shared by all podcast snapshot sections.
///
/// It leaves room for music state inside the 8 MiB sync line limit.
pub const MAX_PODCAST_SNAPSHOT_BYTES: usize = 4 * 1024 * 1024;
/// Errors produced by podcast contract validation and parsing.
#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)]
#[non_exhaustive]
pub enum PodcastError {
/// A feed URL cannot be canonicalized.
#[error("invalid feed URL: {0}")]
InvalidFeedUrl(String),
/// A string is not a valid podcast or episode id.
#[error("invalid {kind} id")]
InvalidId {
/// `podcast` or `episode`.
kind: &'static str,
},
/// A field violates the contract (missing, too long, malformed).
#[error("invalid {field}: {reason}")]
InvalidField {
/// Field name.
field: &'static str,
/// Human-readable reason.
reason: &'static str,
},
/// Two records that must describe the same entity do not.
#[error("cannot merge records of different {0}")]
Mismatch(&'static str),
/// The document is not an RSS or Atom feed.
#[error("not an RSS or Atom feed{0}")]
NotAFeed(String),
/// The feed is malformed and nothing usable could be parsed.
#[error("malformed feed: {0}")]
MalformedFeed(String),
}
/// Result alias for this module.
pub type PodcastResult<T> = std::result::Result<T, PodcastError>;
/// Truncates `text` to at most `max_chars` characters, ending with `…` when cut.
///
/// Clients use it to fit publisher metadata into the contract bounds.
pub fn bounded_text(text: &str, max_chars: usize) -> String {
if text.chars().count() <= max_chars {
return text.to_string();
}
if max_chars == 0 {
return String::new();
}
let mut out: String = text.chars().take(max_chars - 1).collect();
out.push('…');
out
}
pub(crate) fn check_text(
field: &'static str,
value: &str,
max_chars: usize,
required: bool,
) -> PodcastResult<()> {
if required && value.trim().is_empty() {
return Err(PodcastError::InvalidField {
field,
reason: "must not be empty",
});
}
if value.len() > max_chars * 4 || value.chars().count() > max_chars {
return Err(PodcastError::InvalidField {
field,
reason: "is too long",
});
}
Ok(())
}
pub(crate) fn check_opt_text(
field: &'static str,
value: Option<&str>,
max_chars: usize,
) -> PodcastResult<()> {
value.map_or(Ok(()), |value| check_text(field, value, max_chars, false))
}
pub(crate) fn check_token(field: &'static str, value: &str) -> PodcastResult<()> {
if value.is_empty() {
return Err(PodcastError::InvalidField {
field,
reason: "must not be empty",
});
}
if value.len() > MAX_TOKEN_BYTES {
return Err(PodcastError::InvalidField {
field,
reason: "is too long",
});
}
if value.chars().any(|c| c.is_control()) {
return Err(PodcastError::InvalidField {
field,
reason: "contains control characters",
});
}
Ok(())
}
/// Returns whether `value` is an absolute `http`/`https` URL with a host,
/// without whitespace or control characters and within [`MAX_URL_BYTES`].
///
/// Media URLs keep the publisher's spelling; only feed URLs are canonicalized.
pub fn is_http_url(value: &str) -> bool {
if value.is_empty()
|| value.len() > MAX_URL_BYTES
|| value.chars().any(|c| c.is_whitespace() || c.is_control())
{
return false;
}
let lower_prefix = |prefix: &str| {
value.len() >= prefix.len() && value[..prefix.len()].eq_ignore_ascii_case(prefix)
};
let rest = if lower_prefix("http://") {
&value[7..]
} else if lower_prefix("https://") {
&value[8..]
} else {
return false;
};
let authority = rest.split(['/', '?', '#']).next().unwrap_or_default();
let host = authority.rsplit('@').next().unwrap_or_default();
!host.is_empty() && !host.starts_with(':')
}
pub(crate) fn check_url(field: &'static str, value: &str) -> PodcastResult<()> {
if is_http_url(value) {
Ok(())
} else {
Err(PodcastError::InvalidField {
field,
reason: "must be an absolute http(s) URL",
})
}
}
pub(crate) fn check_opt_url(field: &'static str, value: Option<&str>) -> PodcastResult<()> {
value.map_or(Ok(()), |value| check_url(field, value))
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn bounded_text_cuts_on_char_boundaries() {
assert_eq!(bounded_text("hello", 5), "hello");
assert_eq!(bounded_text("hello!", 5), "hell…");
assert_eq!(bounded_text("привет", 3), "пр…");
assert_eq!(bounded_text("x", 0), "");
}
#[test]
fn text_limits_accept_the_limit_and_reject_one_more() {
let at = "я".repeat(MAX_TITLE_CHARS);
let over = "я".repeat(MAX_TITLE_CHARS + 1);
assert!(check_text("title", &at, MAX_TITLE_CHARS, true).is_ok());
assert!(check_text("title", &over, MAX_TITLE_CHARS, true).is_err());
assert!(check_text("title", " ", MAX_TITLE_CHARS, true).is_err());
assert!(check_text("title", "", MAX_TITLE_CHARS, false).is_ok());
}
#[test]
fn token_limits() {
assert!(check_token("op_id", &"a".repeat(MAX_TOKEN_BYTES)).is_ok());
assert!(check_token("op_id", &"a".repeat(MAX_TOKEN_BYTES + 1)).is_err());
assert!(check_token("op_id", "").is_err());
assert!(check_token("op_id", "a\nb").is_err());
}
#[test]
fn http_urls() {
assert!(is_http_url("https://cdn.example.test/a.mp3?x=1"));
assert!(is_http_url("HTTP://cdn.example.test"));
assert!(!is_http_url("ftp://cdn.example.test/a.mp3"));
assert!(!is_http_url("file:///etc/passwd"));
assert!(!is_http_url("https://"));
assert!(!is_http_url("https:///path"));
assert!(!is_http_url("https://a.test/my file.mp3"));
assert!(!is_http_url(""));
let long = format!("https://a.test/{}", "x".repeat(MAX_URL_BYTES));
assert!(!is_http_url(&long));
let at = format!("https://a.test/{}", "x".repeat(MAX_URL_BYTES - 15));
assert_eq!(at.len(), MAX_URL_BYTES);
assert!(is_http_url(&at));
}
}
+526
View File
@@ -0,0 +1,526 @@
//! Personal podcast state and its convergent merge rules.
//!
//! Every merge function here is commutative, associative and idempotent over
//! records of the same entity, so trusted devices that receive the same set of
//! operations converge to the same state regardless of delivery order and
//! duplication. All ties are broken by a total order over every field.
use std::cmp::Ordering;
use std::collections::BTreeSet;
use serde::{Deserialize, Serialize};
use super::{
EpisodeId, FeedUrl, MAX_FEED_ALIASES, PodcastError, PodcastId, PodcastMeta, PodcastResult,
check_token, merge_feed_aliases,
};
/// Largest tail of an episode that still counts as finished, in milliseconds.
///
/// See [`reaches_completion`].
pub const COMPLETION_TAIL_MAX_MS: i64 = 30_000;
/// Minimum actual listening time for a non-completed session to enter history.
pub const MIN_RECORDED_EPISODE_LISTEN_MS: i64 = 30_000;
/// Shared automatic completion criterion for every client.
///
/// A position completes an episode when it is within the smaller of 30 s and
/// 5 % of the duration from the end. Natural end of stream always completes,
/// whatever the announced duration. Without a known positive duration only the
/// end of stream completes.
pub fn reaches_completion(position_ms: i64, duration_ms: Option<i64>) -> bool {
let Some(duration) = duration_ms.filter(|duration| *duration > 0) else {
return false;
};
let tail = COMPLETION_TAIL_MAX_MS.min(duration / 20);
position_ms >= duration - tail
}
/// Listening status of one episode.
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum ProgressStatus {
/// Never started, or explicitly marked unplayed.
NotStarted,
/// Started and not finished.
InProgress,
/// Finished, automatically or by an explicit "mark played".
Completed,
}
/// Durable resume register of one episode.
///
/// The winner among records of the same episode is the greatest
/// `(generation, hlc_ms, device_id, op_id)`; remaining fields only break ties
/// between malformed duplicates. Progress is **never** the maximum position:
/// seeking backwards and "start over" are real changes.
///
/// * Explicit user actions (start of a new listening session, seek, "start
/// over", "mark played", "mark unplayed", taking over playback through a
/// handoff) use [`ProgressRecord::explicit_generation`]: one more than the
/// greatest generation the device has seen for the episode. This is a
/// Lamport clock, so wall-clock skew cannot undo an explicit action that was
/// observed. An explicit action during a session becomes the generation of
/// that session.
/// * Periodic checkpoints keep the generation of the latest explicit action of
/// their session. A late checkpoint of an older session therefore loses
/// against any newer explicit action, whatever its `hlc_ms`.
/// * Concurrent explicit actions on offline devices receive the same
/// generation and are ordered by `hlc_ms`, then `device_id`, then `op_id`.
/// * Handoff: only the receiving device writes the explicit action, with
/// `explicit_generation(max(local, carried))` where `carried` is
/// [`PodcastEpisodeRef::progress_generation`](super::PodcastEpisodeRef::progress_generation)
/// of the queue item. The sender writes a final checkpoint at its session
/// generation.
/// * A device that observes a greater generation for the episode it is
/// playing stops writing at its old generation: either playback moved away,
/// or the user keeps listening locally and the device writes its next
/// record as an explicit action.
/// * Outgoing-log coalescing may drop a pending progress operation only when
/// another pending operation of the same episode [supersedes](Self::supersedes)
/// it; dropping by sequence order could discard an explicit action.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct ProgressRecord {
/// Episode the register belongs to.
pub episode_id: EpisodeId,
/// Owning podcast.
pub podcast_id: PodcastId,
/// Resume position in milliseconds.
pub position_ms: i64,
/// Listening status.
pub status: ProgressStatus,
/// Lamport generation of the latest explicit action (see type docs).
pub generation: u64,
/// Listening session (or explicit action) that wrote the record.
pub session_id: String,
/// Logical write time, normally the sync operation `hlc_ms`.
pub hlc_ms: i64,
/// Writing device, normally the sync operation origin.
pub device_id: String,
/// Writing operation id.
pub op_id: String,
}
impl ProgressRecord {
/// Generation for an explicit user action, given the greatest generation
/// already seen for the episode (`None` when none was seen).
pub fn explicit_generation(max_seen: Option<u64>) -> u64 {
max_seen.map_or(1, |generation| generation.saturating_add(1))
}
/// Primary ordering key.
pub fn order_key(&self) -> (u64, i64, &str, &str) {
(
self.generation,
self.hlc_ms,
self.device_id.as_str(),
self.op_id.as_str(),
)
}
fn total_cmp(&self, other: &Self) -> Ordering {
self.order_key()
.cmp(&other.order_key())
.then_with(|| self.position_ms.cmp(&other.position_ms))
.then_with(|| self.status.cmp(&other.status))
.then_with(|| self.session_id.cmp(&other.session_id))
.then_with(|| self.podcast_id.cmp(&other.podcast_id))
}
/// Returns whether this record wins over `other` (same episode).
pub fn supersedes(&self, other: &Self) -> bool {
self.total_cmp(other) == Ordering::Greater
}
/// Position from which ordinary Play resumes.
///
/// Completed and not-started episodes start from the beginning; use the
/// explicit "start over" action to restart an in-progress episode.
pub fn resume_position_ms(&self) -> i64 {
match self.status {
ProgressStatus::InProgress => self.position_ms,
ProgressStatus::NotStarted | ProgressStatus::Completed => 0,
}
}
/// Validates contract bounds.
pub fn validate(&self) -> PodcastResult<()> {
if self.position_ms < 0 {
return Err(PodcastError::InvalidField {
field: "position_ms",
reason: "must not be negative",
});
}
check_token("session_id", &self.session_id)?;
check_token("device_id", &self.device_id)?;
check_token("op_id", &self.op_id)
}
}
/// Merges two progress registers of the same episode.
pub fn merge_progress(
current: &ProgressRecord,
incoming: &ProgressRecord,
) -> PodcastResult<ProgressRecord> {
if current.episode_id != incoming.episode_id {
return Err(PodcastError::Mismatch("episodes"));
}
Ok(if incoming.supersedes(current) {
incoming.clone()
} else {
current.clone()
})
}
/// Personal subscription state of one podcast.
///
/// `subscribed` and `podcast` form one last-writer-wins register ordered by
/// `(generation, hlc_ms, op_id)`. `generation` is a Lamport counter shared by
/// the subscription and share registers of the podcast (see
/// [`next_podcast_generation`]), so an action taken after observing another
/// device's write wins whatever the device clocks say. Unsubscribing writes
/// `subscribed = false`: it is state, not a tombstone, is never compacted, and
/// never deletes progress or history, so subscribing again continues where
/// the user stopped.
///
/// Only explicit user actions (subscribe, unsubscribe) write this register.
/// Background RSS refreshes keep their metadata locally and must not emit a
/// subscription operation; otherwise a stale device could resurrect an
/// unsubscribed podcast. Feed aliases learned by refreshes travel in their
/// own grow-only operation.
///
/// `feed_aliases` is a bounded grow-only set merged independently of the
/// register (see [`merge_feed_aliases`]): the URL the id was derived from
/// plus the [`MAX_FEED_ALIASES`] smallest other URLs. The current
/// `podcast.feed_url` may fall outside that bound, so look podcasts up by
/// [`SubscriptionRecord::feed_urls`].
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct SubscriptionRecord {
/// Podcast.
pub podcast_id: PodcastId,
/// Whether the user is subscribed.
pub subscribed: bool,
/// Channel metadata at the time of the write.
pub podcast: PodcastMeta,
/// Known feed URLs of the podcast.
#[serde(default)]
pub feed_aliases: Vec<FeedUrl>,
/// Lamport generation shared with the share register (see type docs).
#[serde(default)]
pub generation: u64,
/// Logical write time.
pub hlc_ms: i64,
/// Writing operation id.
pub op_id: String,
}
impl SubscriptionRecord {
/// Register ordering key: `(generation, hlc_ms, op_id)`.
pub fn order_key(&self) -> (u64, i64, &str) {
(self.generation, self.hlc_ms, self.op_id.as_str())
}
fn total_cmp(&self, other: &Self) -> Ordering {
self.order_key()
.cmp(&other.order_key())
.then_with(|| self.subscribed.cmp(&other.subscribed))
.then_with(|| self.podcast.cmp(&other.podcast))
}
/// Returns whether this record's register wins over `other`'s.
pub fn supersedes(&self, other: &Self) -> bool {
self.total_cmp(other) == Ordering::Greater
}
/// Merges `podcast.feed_url` into `feed_aliases` and bounds the set.
///
/// The result is sorted and bounded by [`merge_feed_aliases`]; it may not
/// contain `podcast.feed_url` when many smaller aliases are known.
pub fn normalize(&mut self) {
self.feed_aliases = merge_feed_aliases(
&self.podcast_id,
&self.feed_aliases,
std::slice::from_ref(&self.podcast.feed_url),
);
}
/// Adds aliases learned through a `podcast_feed_aliases_added` operation.
///
/// Only the grow-only alias set changes; the register is untouched.
pub fn add_feed_aliases(&mut self, aliases: &[FeedUrl]) {
self.feed_aliases = merge_feed_aliases(&self.podcast_id, &self.feed_aliases, aliases);
}
/// Every URL that identifies this podcast: the aliases and the current
/// `podcast.feed_url`, sorted and without duplicates.
///
/// Stores pass these pairs to [`super::resolve_podcast_id`].
pub fn feed_urls(&self) -> Vec<FeedUrl> {
let mut urls: BTreeSet<FeedUrl> = self.feed_aliases.iter().cloned().collect();
urls.insert(self.podcast.feed_url.clone());
urls.into_iter().collect()
}
/// Validates contract bounds.
pub fn validate(&self) -> PodcastResult<()> {
self.podcast.validate()?;
if self.feed_aliases.len() > MAX_FEED_ALIASES + 1 {
return Err(PodcastError::InvalidField {
field: "feed_aliases",
reason: "has too many entries",
});
}
check_token("op_id", &self.op_id)
}
}
/// Merges two subscription records of the same podcast.
///
/// The result is normalized (see [`SubscriptionRecord::normalize`]).
pub fn merge_subscription(
current: &SubscriptionRecord,
incoming: &SubscriptionRecord,
) -> PodcastResult<SubscriptionRecord> {
if current.podcast_id != incoming.podcast_id {
return Err(PodcastError::Mismatch("podcasts"));
}
let mut merged = if incoming.supersedes(current) {
incoming.clone()
} else {
current.clone()
};
let mut left = current.feed_aliases.clone();
left.push(current.podcast.feed_url.clone());
let mut right = incoming.feed_aliases.clone();
right.push(incoming.podcast.feed_url.clone());
merged.feed_aliases = merge_feed_aliases(&merged.podcast_id, &left, &right);
Ok(merged)
}
/// Decision to publish a podcast card to the federation.
///
/// Last-writer-wins by `(generation, hlc_ms, op_id)`, with `generation` the
/// per-podcast Lamport counter shared with [`SubscriptionRecord`] (see
/// [`next_podcast_generation`]). `shared = false` revokes the publication.
/// It is privacy-protecting register state, **not** a tombstone: like an
/// unsubscribe it is never compacted, otherwise a stale `shared = true`
/// delivered again would republish the podcast. Unsubscribing must also write
/// `shared = false`.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct ShareRecord {
/// Podcast.
pub podcast_id: PodcastId,
/// Whether the user chose to share the podcast.
pub shared: bool,
/// Lamport generation shared with the subscription register.
#[serde(default)]
pub generation: u64,
/// Logical write time.
pub hlc_ms: i64,
/// Writing operation id.
pub op_id: String,
}
impl ShareRecord {
/// Register ordering key: `(generation, hlc_ms, op_id)`.
pub fn order_key(&self) -> (u64, i64, &str) {
(self.generation, self.hlc_ms, self.op_id.as_str())
}
fn total_cmp(&self, other: &Self) -> Ordering {
self.order_key()
.cmp(&other.order_key())
.then_with(|| self.shared.cmp(&other.shared))
}
/// Returns whether this record wins over `other`.
pub fn supersedes(&self, other: &Self) -> bool {
self.total_cmp(other) == Ordering::Greater
}
/// Validates contract bounds.
pub fn validate(&self) -> PodcastResult<()> {
check_token("op_id", &self.op_id)
}
}
/// Merges two share records of the same podcast.
pub fn merge_share(current: &ShareRecord, incoming: &ShareRecord) -> PodcastResult<ShareRecord> {
if current.podcast_id != incoming.podcast_id {
return Err(PodcastError::Mismatch("podcasts"));
}
Ok(if incoming.supersedes(current) {
incoming.clone()
} else {
current.clone()
})
}
/// Generation for a new write of the subscription or share register.
///
/// One more than the greatest generation of either register of the podcast
/// (`None` when it has none), so a write made after observing another
/// device's subscription, unsubscription or share decision orders after it
/// even when that device's clock runs ahead. Concurrent writes on devices
/// that did not see each other receive the same generation and are ordered
/// by `hlc_ms`, then `op_id`.
pub fn next_podcast_generation(
subscription: Option<&SubscriptionRecord>,
share: Option<&ShareRecord>,
) -> u64 {
subscription
.map(|record| record.generation)
.into_iter()
.chain(share.map(|record| record.generation))
.max()
.map_or(1, |generation| generation.saturating_add(1))
}
/// Whether a podcast must currently be published.
///
/// True only when the user is subscribed, chose to share, and the share
/// decision orders after the latest subscription write (by
/// `(generation, hlc_ms, op_id)`). A new subscription (including
/// re-subscribing, or subscribing from someone else's public card) is
/// therefore always private until the user shares it again, and a stale
/// device cannot republish a podcast through an old share decision. A share
/// decision taken after observing the subscription always applies, because it
/// carries a greater generation.
pub fn effective_share(
subscription: Option<&SubscriptionRecord>,
share: Option<&ShareRecord>,
) -> bool {
match (subscription, share) {
(Some(subscription), Some(share)) => {
subscription.subscribed
&& share.shared
&& subscription.podcast_id == share.podcast_id
&& share.order_key() > subscription.order_key()
}
_ => false,
}
}
/// One listening session of an episode on one device.
///
/// Sessions form a grow-only set keyed by `session_id`, generated when
/// playback of the episode starts on a device. A device may re-send its
/// session with more listening time (for example after a crash); versions of
/// one session merge by [`merge_listen_session`].
///
/// * A session enters history when it completed or lasted at least
/// [`MIN_RECORDED_EPISODE_LISTEN_MS`] ([`ListenSession::should_record`]).
/// * `completed` is set when the session reached [`reaches_completion`] or the
/// natural end of the stream. Each session counts at most once, so the play
/// count of an episode is the number of distinct completed sessions
/// ([`completed_listen_count`]); duplicate delivery never double counts.
/// * "Mark played" changes [`ProgressRecord::status`] only. It does not create
/// a session and does not invent listening time.
/// * Sessions listened concurrently on offline devices are all kept.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct ListenSession {
/// Unique session id, normally a UUID.
pub session_id: String,
/// Episode.
pub episode_id: EpisodeId,
/// Owning podcast.
pub podcast_id: PodcastId,
/// Device that played the session.
pub device_id: String,
/// Session start in unix milliseconds.
pub started_at_ms: i64,
/// Latest update of the session in unix milliseconds.
pub ended_at_ms: i64,
/// Position when the session started.
pub start_position_ms: i64,
/// Position when the session ended (or was last reported).
pub end_position_ms: i64,
/// Actual accumulated listening time, excluding pauses and seeks.
pub listened_ms: i64,
/// Whether the session completed the episode.
pub completed: bool,
}
impl ListenSession {
fn total_cmp(&self, other: &Self) -> Ordering {
(
self.completed,
self.listened_ms,
self.ended_at_ms,
self.end_position_ms,
self.started_at_ms,
self.start_position_ms,
)
.cmp(&(
other.completed,
other.listened_ms,
other.ended_at_ms,
other.end_position_ms,
other.started_at_ms,
other.start_position_ms,
))
}
/// Whether the session belongs in listening history.
pub fn should_record(&self) -> bool {
self.completed || self.listened_ms >= MIN_RECORDED_EPISODE_LISTEN_MS
}
/// Validates contract bounds.
pub fn validate(&self) -> PodcastResult<()> {
check_token("session_id", &self.session_id)?;
check_token("device_id", &self.device_id)?;
if self.started_at_ms < 0 || self.ended_at_ms < self.started_at_ms {
return Err(PodcastError::InvalidField {
field: "ended_at_ms",
reason: "must not precede started_at_ms",
});
}
if self.start_position_ms < 0 || self.end_position_ms < 0 || self.listened_ms < 0 {
return Err(PodcastError::InvalidField {
field: "listened_ms",
reason: "positions and listening time must not be negative",
});
}
Ok(())
}
}
/// Merges two versions of the same listening session.
///
/// The version with the greatest `(completed, listened_ms, ended_at_ms, ...)`
/// wins, so a session can only grow.
pub fn merge_listen_session(
current: &ListenSession,
incoming: &ListenSession,
) -> PodcastResult<ListenSession> {
if current.session_id != incoming.session_id {
return Err(PodcastError::Mismatch("sessions"));
}
if current.episode_id != incoming.episode_id
|| current.podcast_id != incoming.podcast_id
|| current.device_id != incoming.device_id
{
return Err(PodcastError::Mismatch("session identities"));
}
Ok(if incoming.total_cmp(current) == Ordering::Greater {
incoming.clone()
} else {
current.clone()
})
}
/// Number of distinct sessions that completed an episode.
pub fn completed_listen_count<'a>(sessions: impl IntoIterator<Item = &'a ListenSession>) -> usize {
sessions
.into_iter()
.filter(|session| session.completed)
.map(|session| session.session_id.as_str())
.collect::<BTreeSet<_>>()
.len()
}
#[cfg(test)]
#[path = "state_tests.rs"]
mod tests;
+699
View File
@@ -0,0 +1,699 @@
use std::collections::BTreeMap;
use super::*;
use crate::podcast::{EpisodeIdentityHints, EpisodeMeta, EpisodeType, resolve_episode_ids};
fn permutations<T: Clone>(items: &[T]) -> Vec<Vec<T>> {
if items.len() <= 1 {
return vec![items.to_vec()];
}
let mut out = Vec::new();
for index in 0..items.len() {
let mut rest = items.to_vec();
let head = rest.remove(index);
for mut tail in permutations(&rest) {
tail.insert(0, head.clone());
out.push(tail);
}
}
out
}
fn feed_url(path: &str) -> FeedUrl {
FeedUrl::parse(&format!("https://example.test/{path}")).unwrap()
}
fn podcast_id() -> PodcastId {
PodcastId::from_feed_url(&feed_url("feed"))
}
fn episode_id(guid: &str) -> EpisodeId {
EpisodeId::from_guid(&podcast_id(), guid).unwrap()
}
fn progress(
generation: u64,
hlc_ms: i64,
device: &str,
op: &str,
position_ms: i64,
status: ProgressStatus,
session: &str,
) -> ProgressRecord {
ProgressRecord {
episode_id: episode_id("e1"),
podcast_id: podcast_id(),
position_ms,
status,
generation,
session_id: session.to_string(),
hlc_ms,
device_id: device.to_string(),
op_id: op.to_string(),
}
}
fn fold<T: Clone>(items: &[T], merge: impl Fn(&T, &T) -> PodcastResult<T>) -> T {
let mut iter = items.iter();
let first = iter.next().expect("non-empty").clone();
iter.fold(first, |acc, item| merge(&acc, item).unwrap())
}
fn assert_converges<T: Clone + PartialEq + std::fmt::Debug>(
items: &[T],
merge: impl Fn(&T, &T) -> PodcastResult<T> + Copy,
) -> T {
let reference = fold(items, merge);
for order in permutations(items) {
assert_eq!(fold(&order, merge), reference, "order: {order:?}");
}
// Merge laws on every pair and triple.
for a in items {
assert_eq!(
merge(&merge(a, a).unwrap(), a).unwrap(),
merge(a, a).unwrap()
);
for b in items {
assert_eq!(merge(a, b).unwrap(), merge(b, a).unwrap());
for c in items {
assert_eq!(
merge(&merge(a, b).unwrap(), c).unwrap(),
merge(a, &merge(b, c).unwrap()).unwrap()
);
}
}
}
reference
}
#[test]
fn progress_converges_for_every_permutation() {
use ProgressStatus::*;
let ops = [
progress(1, 100, "dev-a", "dev-a:1", 10_000, InProgress, "s1"),
progress(1, 160, "dev-a", "dev-a:2", 70_000, InProgress, "s1"),
progress(2, 150, "dev-b", "dev-b:1", 0, InProgress, "s2"),
progress(2, 150, "dev-c", "dev-c:1", 5_000, InProgress, "s3"),
progress(2, 140, "dev-b", "dev-b:2", 0, Completed, "s2"),
// Duplicate delivery of the same operation.
progress(1, 160, "dev-a", "dev-a:2", 70_000, InProgress, "s1"),
];
let winner = assert_converges(&ops, merge_progress);
assert_eq!(winner.op_id, "dev-c:1");
}
#[test]
fn late_old_generation_checkpoint_loses() {
use ProgressStatus::*;
// Device A keeps checkpointing session s1 (generation 3) with a clock far
// ahead; device B explicitly seeks back (generation 4) with a slow clock.
let seek_back = progress(4, 1_000, "dev-b", "dev-b:9", 60_000, InProgress, "s2");
let late = progress(
3, 9_000_000, "dev-a", "dev-a:50", 3_000_000, InProgress, "s1",
);
for order in [[&seek_back, &late], [&late, &seek_back]] {
let merged = merge_progress(order[0], order[1]).unwrap();
assert_eq!(merged, seek_back);
}
assert!(seek_back.supersedes(&late));
assert!(!late.supersedes(&seek_back));
}
#[test]
fn restart_beats_later_checkpoint_of_the_old_session() {
use ProgressStatus::*;
let checkpoint = progress(2, 100, "dev-a", "dev-a:1", 2_400_000, InProgress, "s1");
let generation = ProgressRecord::explicit_generation(Some(checkpoint.generation));
assert_eq!(generation, 3);
let restart = progress(generation, 110, "dev-a", "dev-a:2", 0, InProgress, "s2");
// A checkpoint of s1 that was queued before the restart and delivered
// later, even stamped after the restart.
let late = progress(2, 120, "dev-a", "dev-a:3", 2_460_000, InProgress, "s1");
let merged = assert_converges(&[checkpoint, restart.clone(), late], merge_progress);
assert_eq!(merged, restart);
assert_eq!(merged.resume_position_ms(), 0);
}
#[test]
fn progress_is_not_the_maximum_position() {
use ProgressStatus::*;
let ahead = progress(1, 100, "dev-a", "dev-a:1", 3_000_000, InProgress, "s1");
let back = progress(2, 101, "dev-a", "dev-a:2", 60_000, InProgress, "s1");
assert_eq!(merge_progress(&ahead, &back).unwrap().position_ms, 60_000);
}
#[test]
fn concurrent_offline_sessions_converge_and_keep_both_sessions() {
use ProgressStatus::*;
// Both devices last saw generation 5 and start listening offline.
let generation = ProgressRecord::explicit_generation(Some(5));
let a_start = progress(
generation, 200, "dev-a", "dev-a:1", 300_000, InProgress, "sa",
);
let a_cp = progress(
generation, 300, "dev-a", "dev-a:2", 900_000, InProgress, "sa",
);
let b_start = progress(
generation, 210, "dev-b", "dev-b:1", 300_000, InProgress, "sb",
);
let b_cp = progress(
generation, 280, "dev-b", "dev-b:2", 1_500_000, InProgress, "sb",
);
let merged = assert_converges(&[a_start, a_cp.clone(), b_start, b_cp], merge_progress);
// Deterministic: the latest write of the shared generation wins.
assert_eq!(merged, a_cp);
let session = |id: &str, device: &str, listened: i64, completed: bool| ListenSession {
session_id: id.to_string(),
episode_id: episode_id("e1"),
podcast_id: podcast_id(),
device_id: device.to_string(),
started_at_ms: 200,
ended_at_ms: 300,
start_position_ms: 300_000,
end_position_ms: 900_000,
listened_ms: listened,
completed,
};
let sessions = [
session("sa", "dev-a", 600_000, false),
session("sb", "dev-b", 1_200_000, true),
session("sb", "dev-b", 1_200_000, true),
];
let mut set: BTreeMap<String, ListenSession> = BTreeMap::new();
for order in permutations(&sessions) {
set.clear();
for item in order {
let merged = match set.get(&item.session_id) {
Some(current) => merge_listen_session(current, &item).unwrap(),
None => item.clone(),
};
set.insert(item.session_id.clone(), merged);
}
assert_eq!(set.len(), 2);
assert_eq!(completed_listen_count(set.values()), 1);
}
}
#[test]
fn listen_session_versions_only_grow() {
let base = ListenSession {
session_id: "s".to_string(),
episode_id: episode_id("e1"),
podcast_id: podcast_id(),
device_id: "dev-a".to_string(),
started_at_ms: 1_000,
ended_at_ms: 2_000,
start_position_ms: 0,
end_position_ms: 60_000,
listened_ms: 60_000,
completed: false,
};
let longer = ListenSession {
ended_at_ms: 3_000,
end_position_ms: 120_000,
listened_ms: 120_000,
..base.clone()
};
let finished = ListenSession {
ended_at_ms: 3_500,
end_position_ms: 3_600_000,
listened_ms: 125_000,
completed: true,
..base.clone()
};
let merged = assert_converges(
&[base.clone(), longer, finished.clone(), base.clone()],
merge_listen_session,
);
assert_eq!(merged, finished);
assert!(merged.should_record());
assert_eq!(completed_listen_count([&merged, &merged, &finished]), 1);
let mut other_episode = base.clone();
other_episode.episode_id = episode_id("e2");
assert!(merge_listen_session(&base, &other_episode).is_err());
let short = ListenSession {
listened_ms: MIN_RECORDED_EPISODE_LISTEN_MS - 1,
..base.clone()
};
assert!(!short.should_record());
assert!(
ListenSession {
listened_ms: MIN_RECORDED_EPISODE_LISTEN_MS,
..base
}
.should_record()
);
}
#[test]
fn completion_criterion() {
// 60 min episode: tail is 30 s.
assert!(!reaches_completion(3_569_999, Some(3_600_000)));
assert!(reaches_completion(3_570_000, Some(3_600_000)));
// 5 min episode: tail is 5 % = 15 s.
assert!(!reaches_completion(284_999, Some(300_000)));
assert!(reaches_completion(285_000, Some(300_000)));
assert!(!reaches_completion(1_000_000, None));
assert!(!reaches_completion(1_000_000, Some(0)));
}
fn podcast_meta(title: &str, url: &str) -> PodcastMeta {
PodcastMeta {
feed_url: feed_url(url),
title: title.to_string(),
author: None,
description: None,
link: None,
image_url: None,
}
}
fn subscription(
subscribed: bool,
hlc_ms: i64,
op: &str,
title: &str,
url: &str,
) -> SubscriptionRecord {
let mut record = SubscriptionRecord {
podcast_id: podcast_id(),
subscribed,
podcast: podcast_meta(title, url),
feed_aliases: Vec::new(),
generation: 1,
hlc_ms,
op_id: op.to_string(),
};
record.normalize();
record
}
fn with_generation<T: Clone>(record: &T, set: impl Fn(&mut T)) -> T {
let mut record = record.clone();
set(&mut record);
record
}
#[test]
fn subscriptions_converge_for_every_permutation() {
let ops = [
// Same podcast added on two devices.
subscription(true, 100, "dev-a:1", "Show", "feed"),
subscription(true, 101, "dev-b:1", "Show (B)", "feed"),
// Unsubscribed on A while B was offline.
subscription(false, 200, "dev-a:2", "Show", "feed"),
// B comes back and re-delivers its old state after the feed moved.
subscription(true, 150, "dev-b:2", "Show", "moved"),
subscription(true, 101, "dev-b:1", "Show (B)", "feed"),
];
let merged = assert_converges(&ops, merge_subscription);
assert!(
!merged.subscribed,
"unsubscribe survives the offline device"
);
assert_eq!(merged.op_id, "dev-a:2");
assert!(merged.feed_aliases.contains(&feed_url("moved")));
assert!(merged.feed_aliases.contains(&feed_url("feed")));
merged.validate().unwrap();
}
#[test]
fn resubscribing_keeps_progress_and_history() {
use ProgressStatus::*;
let mut progress_register = BTreeMap::new();
let saved = progress(4, 500, "dev-a", "dev-a:5", 1_234_000, InProgress, "s1");
progress_register.insert(saved.episode_id.clone(), saved.clone());
let subscribed = subscription(true, 100, "dev-a:1", "Show", "feed");
let unsubscribed = subscription(false, 600, "dev-a:6", "Show", "feed");
let again = subscription(true, 700, "dev-b:1", "Show", "feed");
let merged = fold(&[subscribed, unsubscribed, again], merge_subscription);
assert!(merged.subscribed);
assert_eq!(progress_register.get(&saved.episode_id), Some(&saved));
}
fn share(shared: bool, hlc_ms: i64, op: &str) -> ShareRecord {
ShareRecord {
podcast_id: podcast_id(),
shared,
generation: 1,
hlc_ms,
op_id: op.to_string(),
}
}
#[test]
fn shares_converge_and_false_is_never_beaten_by_a_stale_true() {
let ops = [
share(true, 100, "dev-a:1"),
share(false, 200, "dev-a:2"),
share(true, 150, "dev-b:1"),
share(false, 200, "dev-a:2"),
];
let merged = assert_converges(&ops, merge_share);
assert!(!merged.shared, "late stale share cannot republish");
}
#[test]
fn effective_share_rules() {
let subscribed = subscription(true, 100, "dev-a:1", "Show", "feed");
let shared = share(true, 150, "dev-a:2");
assert!(effective_share(Some(&subscribed), Some(&shared)));
assert!(!effective_share(Some(&subscribed), None));
assert!(!effective_share(None, Some(&shared)));
// New subscriptions are private: a share decision older than the
// latest subscription write does not apply.
let resubscribed = subscription(true, 300, "dev-a:4", "Show", "feed");
assert!(!effective_share(Some(&resubscribed), Some(&shared)));
// Unsubscribing revokes publication even if a stale share wins.
let unsubscribed = subscription(false, 200, "dev-a:3", "Show", "feed");
let stale = share(true, 250, "dev-b:1");
assert!(!effective_share(Some(&unsubscribed), Some(&stale)));
}
#[test]
fn clock_skew_cannot_undo_an_observed_unsubscribe() {
// Device A's clock runs ten minutes ahead. B sees A's subscription, then
// the user unsubscribes on B.
let sub_a = subscription(true, 1_000_600_000, "dev-a:1", "Show", "feed");
assert_eq!(next_podcast_generation(None, None), 1);
let generation = next_podcast_generation(Some(&sub_a), None);
assert_eq!(generation, 2);
let unsub_b = with_generation(
&subscription(false, 1_000_060_000, "dev-b:1", "Show", "feed"),
|r| r.generation = generation,
);
let merged = assert_converges(&[sub_a.clone(), unsub_b.clone()], merge_subscription);
assert!(!merged.subscribed, "the later unsubscribe wins");
// Concurrent writes (same generation) still fall back to the clock.
let concurrent = with_generation(&unsub_b, |r| r.generation = 1);
assert!(merge_subscription(&sub_a, &concurrent).unwrap().subscribed);
}
#[test]
fn share_after_subscribe_applies_despite_clock_skew() {
let sub_a = subscription(true, 1_000_600_000, "dev-a:1", "Show", "feed");
let share_b = with_generation(&share(true, 1_000_060_000, "dev-b:1"), |r| {
r.generation = next_podcast_generation(Some(&sub_a), None)
});
assert!(effective_share(Some(&sub_a), Some(&share_b)));
// Unsubscribe written after the share (higher generation) revokes it,
// and so does a later re-subscription until the user shares again.
let unsub = with_generation(&subscription(false, 1, "dev-a:2", "Show", "feed"), |r| {
r.generation = next_podcast_generation(Some(&sub_a), Some(&share_b))
});
assert!(!effective_share(Some(&unsub), Some(&share_b)));
let resub = with_generation(&subscription(true, 2, "dev-a:3", "Show", "feed"), |r| {
r.generation = next_podcast_generation(Some(&unsub), Some(&share_b))
});
assert!(!effective_share(Some(&resub), Some(&share_b)));
let reshare = with_generation(&share(true, 3, "dev-a:4"), |r| {
r.generation = next_podcast_generation(Some(&resub), Some(&share_b))
});
assert!(effective_share(Some(&resub), Some(&reshare)));
// Revoking (shared = false) with a higher generation wins over every
// stale true, in any order.
let revoke = with_generation(&share(false, 0, "dev-b:2"), |r| {
r.generation = next_podcast_generation(Some(&resub), Some(&reshare))
});
let merged = assert_converges(&[share_b, reshare, revoke.clone()], merge_share);
assert_eq!(merged, revoke);
}
#[test]
fn current_feed_url_stays_resolvable_beyond_the_alias_bound() {
let mut record = subscription(true, 1, "dev-a:1", "Show", "feed");
record.add_feed_aliases(
&(0..=crate::podcast::MAX_FEED_ALIASES)
.map(|i| feed_url(&format!("a{i}")))
.collect::<Vec<_>>(),
);
record.podcast.feed_url = FeedUrl::parse("https://z.test/current").unwrap();
record.normalize();
assert!(!record.feed_aliases.contains(&record.podcast.feed_url));
let urls = record.feed_urls();
assert!(urls.contains(&record.podcast.feed_url));
assert!(urls.contains(&feed_url("feed")), "origin is kept");
let known: Vec<(&PodcastId, &FeedUrl)> =
urls.iter().map(|url| (&record.podcast_id, url)).collect();
assert_eq!(
crate::podcast::resolve_podcast_id(&record.podcast.feed_url, known),
podcast_id()
);
}
#[test]
fn handoff_receiver_continues_with_the_carried_generation() {
use ProgressStatus::*;
// Sender A plays session s1 at generation 5 with a fast clock; its
// progress operations have not reached B when the handoff arrives.
let a_checkpoint = progress(5, 9_000_000, "dev-a", "dev-a:9", 600_000, InProgress, "s1");
let carried = Some(a_checkpoint.generation);
let b_local = Some(2);
let generation = ProgressRecord::explicit_generation(b_local.max(carried));
assert_eq!(generation, 6);
let b_start = progress(
generation, 1_000, "dev-b", "dev-b:1", 600_000, InProgress, "s2",
);
let b_pause = progress(
generation, 2_000, "dev-b", "dev-b:2", 660_000, InProgress, "s2",
);
// Without the carried generation B would write generation 3 and lose
// every write to A's older checkpoint.
let uninformed = progress(
ProgressRecord::explicit_generation(b_local),
2_000,
"dev-b",
"dev-b:2",
660_000,
InProgress,
"s2",
);
assert!(a_checkpoint.supersedes(&uninformed));
let merged = assert_converges(&[a_checkpoint, b_start, b_pause.clone()], merge_progress);
assert_eq!(merged, b_pause);
assert_eq!(merged.resume_position_ms(), 660_000);
}
#[test]
fn coalescing_keeps_the_superseding_record_not_the_last_queued() {
use ProgressStatus::*;
// "Mark played" (explicit, generation 5) during a session, followed in
// the outgoing log by a checkpoint of the old session (generation 3).
let mark_played = progress(5, 100, "dev-a", "dev-a:7", 0, Completed, "mark");
let late_checkpoint = progress(3, 101, "dev-a", "dev-a:8", 30_000, InProgress, "s1");
let pending = [mark_played.clone(), late_checkpoint.clone()];
// Keep only records that no other pending record supersedes.
let kept: Vec<&ProgressRecord> = pending
.iter()
.filter(|record| !pending.iter().any(|other| other.supersedes(record)))
.collect();
assert_eq!(kept, vec![&mark_played]);
// Dropping by sequence order would have published the wrong state.
assert_ne!(pending.last(), Some(&mark_played));
assert_eq!(fold(&pending, merge_progress), mark_played);
}
/// Minimal client store used to show that whole operation sets converge and
/// that publisher metadata never reaches personal progress.
#[derive(Debug, Default, Clone, PartialEq)]
struct Store {
subscriptions: BTreeMap<PodcastId, SubscriptionRecord>,
shares: BTreeMap<PodcastId, ShareRecord>,
progress: BTreeMap<EpisodeId, ProgressRecord>,
sessions: BTreeMap<String, ListenSession>,
catalog: BTreeMap<EpisodeId, EpisodeMeta>,
}
#[derive(Debug, Clone)]
enum Op {
Subscription(SubscriptionRecord),
Share(ShareRecord),
Progress(ProgressRecord),
Session(ListenSession),
/// Local RSS refresh: title and audio URL changed at the publisher.
Rss {
title: &'static str,
audio_url: &'static str,
},
}
fn upsert<K: Ord + Clone, V: Clone>(
map: &mut BTreeMap<K, V>,
key: K,
value: &V,
merge: impl Fn(&V, &V) -> PodcastResult<V>,
) {
let merged = match map.get(&key) {
Some(current) => merge(current, value).unwrap(),
None => value.clone(),
};
map.insert(key, merged);
}
impl Store {
fn apply(&mut self, op: &Op) {
match op {
Op::Subscription(record) => upsert(
&mut self.subscriptions,
record.podcast_id.clone(),
record,
merge_subscription,
),
Op::Share(record) => upsert(
&mut self.shares,
record.podcast_id.clone(),
record,
merge_share,
),
Op::Progress(record) => upsert(
&mut self.progress,
record.episode_id.clone(),
record,
merge_progress,
),
Op::Session(session) => upsert(
&mut self.sessions,
session.session_id.clone(),
session,
merge_listen_session,
),
Op::Rss { title, audio_url } => {
let hints = EpisodeIdentityHints {
guid: Some("e1".to_string()),
enclosure_url: Some(audio_url.to_string()),
link: None,
published_at_ms: Some(1_000),
title: Some(title.to_string()),
};
let id = resolve_episode_ids(&podcast_id(), None, &[hints], &[])[0]
.clone()
.unwrap();
// Metadata only: last refresh on this device wins locally.
let meta = EpisodeMeta {
episode_id: id.clone(),
podcast_id: podcast_id(),
guid: Some("e1".to_string()),
title: title.to_string(),
description: None,
season: None,
episode_number: None,
episode_type: EpisodeType::Full,
published_at_ms: Some(1_000),
duration_ms: None,
audio_url: audio_url.to_string(),
mime: None,
byte_len: None,
image_url: None,
link: None,
};
self.catalog.insert(id, meta);
}
}
}
fn personal(&self) -> Store {
Store {
catalog: BTreeMap::new(),
..self.clone()
}
}
}
#[test]
fn whole_operation_sets_converge_and_rss_never_touches_progress() {
use ProgressStatus::*;
let session = ListenSession {
session_id: "s2".to_string(),
episode_id: episode_id("e1"),
podcast_id: podcast_id(),
device_id: "dev-b".to_string(),
started_at_ms: 10,
ended_at_ms: 20,
start_position_ms: 0,
end_position_ms: 400_000,
listened_ms: 400_000,
completed: false,
};
let ops = [
Op::Subscription(subscription(true, 100, "dev-a:1", "Show", "feed")),
Op::Share(share(true, 120, "dev-a:2")),
Op::Progress(progress(
1, 130, "dev-a", "dev-a:3", 900_000, InProgress, "s1",
)),
Op::Progress(progress(
2, 125, "dev-b", "dev-b:1", 400_000, InProgress, "s2",
)),
Op::Session(session),
Op::Rss {
title: "Old title",
audio_url: "https://cdn.test/old.mp3",
},
Op::Rss {
title: "New title",
audio_url: "https://cdn.test/new.mp3",
},
];
let mut reference: Option<Store> = None;
for order in permutations(&ops) {
let mut store = Store::default();
for op in &order {
store.apply(op);
}
// One episode id across both metadata versions.
assert_eq!(store.catalog.len(), 1);
let progress = store.progress.values().next().unwrap();
assert_eq!(progress.op_id, "dev-b:1");
assert_eq!(progress.position_ms, 400_000);
let personal = store.personal();
match &reference {
Some(reference) => assert_eq!(&personal, reference),
None => reference = Some(personal),
}
}
}
#[test]
fn validation_bounds() {
use ProgressStatus::*;
let ok = progress(1, 1, "dev-a", "dev-a:1", 0, NotStarted, "s");
ok.validate().unwrap();
let mut bad = ok.clone();
bad.position_ms = -1;
assert!(bad.validate().is_err());
let mut bad = ok.clone();
bad.session_id = String::new();
assert!(bad.validate().is_err());
let mut bad = ok;
bad.device_id = "d".repeat(crate::podcast::MAX_TOKEN_BYTES + 1);
assert!(bad.validate().is_err());
let mut record = subscription(true, 1, "dev-a:1", "Show", "feed");
record.feed_aliases = (0..=crate::podcast::MAX_FEED_ALIASES + 1)
.map(|i| feed_url(&format!("alias-{i}")))
.collect();
assert!(record.validate().is_err());
record.normalize();
record.validate().unwrap();
}
#[test]
fn progress_json_shape() {
let record = progress(
3,
42,
"dev-a",
"dev-a:7",
1_000,
ProgressStatus::InProgress,
"s",
);
let json = serde_json::to_value(&record).unwrap();
assert_eq!(json["status"], "in_progress");
assert_eq!(json["generation"], 3);
assert_eq!(
serde_json::from_value::<ProgressRecord>(json).unwrap(),
record
);
}