Merge pull request 'Подкасты в FRID: SQLite-хранилище PodcastStore, импорт прототипа и TrackItem.podcast (MR-F2)' (#2) from sanapach/frid:podcasts/store into main
CI / check (push) Successful in 1m14s

This commit was merged in pull request #2.
This commit is contained in:
ab committed 2026-11-20 13:51:51 +00:00
commit 3046c3f924
15 files changed
+6735 -7

No files matched your search

+23 -1
View File
@@ -12,6 +12,7 @@ application
furumi-library
├── local catalog, import, migrations
├── playlists, likes, history, similarity persistence
├── podcast store (podcasts.sqlite3)
│
music-dht
├── Kademlia routing and iterative lookup
@@ -41,9 +42,30 @@ metadata import, artwork resolution, playlists, likes, listening history, and
similarity index storage. Connections enable foreign keys, WAL, and a bounded
busy timeout so independently running clients can safely share the database.
Podcasts use a separate store, `furumi_library::podcasts::PodcastStore`, in
its own `podcasts.sqlite3` file next to the library database. It implements
the podcast contract of `music_dht::podcast` for SQLite clients: the episode
catalog refreshed from RSS, the subscription, share and progress registers,
grow-only listening sessions, feed aliases and local-only fetch state.
Local actions, received operations and snapshot sections share one apply
path built on the `music-dht` materializers and merge functions, so the store
converges independently of delivery order. Migrations follow
`PRAGMA user_version`; the prototype podcast database is imported and its
tables are kept as `legacy_*`; newer or foreign files are refused unchanged.
`TrackItem.podcast` marks episode queue items, which music-only code must
skip. An episode queue item is identified by its episode id
(`TrackItem::key`, a typed source identity: library row, federated owner and
item id, or episode id), never by the numeric id: episodes use the fixed
sentinel id, `release_id` and artist id -1, which carries no identity and
only keeps code written for library tracks (`id >= 0`) away from them.
Received playback items are converted with `TrackItem::from_playback_track`,
which never copies the wire ids (0 for episodes and private placeholders). See the "SQLite reference store" section
of [PODCASTS.md](PODCASTS.md).
Filesystem scanning schedules, playback, federation lifecycle, and UI state
remain application responsibilities. Database and metadata work must stay off
interactive and async reactor threads.
interactive and async reactor threads. RSS fetching and refresh scheduling are
application work as well.
## Transport: federation-net
+31
View File
@@ -18,6 +18,37 @@
Jam redaction. New struct fields and enum variants break exhaustive
construction and matching in clients.
- `furumi-library 0.3.0`: follows `music-dht 0.6.0`, whose types it re-exposes.
- `furumi-library 0.3.0`: new `furumi_library::podcasts::PodcastStore`, the
SQLite reference store of the podcast contract in its own
`podcasts.sqlite3`: `user_version` migrations that import the prototype
podcast database (tables kept as `legacy_*`, unimportable rows in
`legacy_quarantine`) and refuse newer or foreign files unchanged; podcasts,
feed aliases, seasons, episodes, subscription/share/progress registers,
listening sessions and local-only fetch state; local actions that return
the `podcast_*` operations for the client's sync log; `apply_remote_op`,
bounded snapshot export and merge; RSS refresh ingestion with stable
episode ids; UI queries. Enables the `music-dht` `podcast-feed` feature.
- `furumi-library 0.3.0` (breaking): `TrackItem.podcast:
Option<PodcastEpisodeRef>` and `TrackItem::podcast_episode`/`is_podcast`.
Exhaustive `TrackItem` literals in clients need the new field; music-only
code must skip podcast items. Podcast items use the fixed sentinel
`TrackItem::NOT_A_LIBRARY_ROW` (-1) as `id`, `release_id` and artist id,
so existing `id < 0` / `release_id < 0` checks already treat them as
non-library items; the sentinel carries no identity. New
`TrackItem::key` / `TrackItemKey` (`lib:<id>`, `fed:<owner>:<item_id>`,
`ep:<episode id>`), the source identity: episodes must be keyed by it (or
`TrackItem.podcast`), never by `id`; content-first music matching stays
as it is. New `TrackItem::from_playback_track` / `PlaybackItem` convert
received playback items without copying wire ids (0 for episodes and Jam
private placeholders). The library database is unchanged.
- `furumi-library 0.3.0`: prototype import collapses rows of one GUID-less
item under several enclosure URLs, and the first refresh re-keys imported
GUID-less episodes without personal state to the ids the current feed
gives, so they match devices that never had the prototype; when another
episode (for example one created by a remote operation) already holds
that id, the imported copy is deleted instead of kept as a duplicate.
Fetch-state
writes are atomic across threads and processes.
- See `PODCASTS.md` for the normative contract.
## v0.4.1 — 2026-09-02
Generated
+1
View File
@@ -942,6 +942,7 @@ dependencies = [
"rusqlite",
"serde",
"serde_json",
"tempfile",
"tracing",
]
+143 -1
View File
@@ -19,7 +19,7 @@ of this version of the contract; it will be specified with the overlay.
| --- | --- |
| `music_dht::podcast` | ids, URL canonicalization, no-GUID rule, metadata DTOs and bounds, merge functions, RSS/Atom parser (feature `podcast-feed`) |
| `music_dht::device_sync` | sync operation kinds, snapshot sections, `PlaybackSource`, version gating, Jam redaction |
| Client storage (`furumi-library`, web PostgreSQL) | persistence and migrations implementing the rules below |
| Client storage (`furumi_library::podcasts::PodcastStore`, web PostgreSQL) | persistence and migrations implementing the rules below (SQLite reference: see "SQLite reference store") |
| Client runtime | HTTP fetching, conditional requests, scheduling, audio buffering and decoding, UI |
Audio is always fetched from the publisher. It is never imported into the
@@ -511,6 +511,148 @@ extracts `itunes:season`, `itunes:episode`, `itunes:episodeType`,
Links are kept verbatim (trimmed only) because they are identity material.
Text cleanup is linear in the input size. It performs no I/O.
## SQLite reference store
`furumi_library::podcasts::PodcastStore` (`furumi-library` 0.3) implements
this contract for SQLite clients. Other stores (the web client's PostgreSQL)
must keep the same invariants.
### File and migrations
* Own file, conventionally `podcasts.sqlite3` next to `library.db`; the caller
passes the path. WAL, foreign keys, 5 s busy timeout.
* `PRAGMA user_version` drives migrations. Each runs in a `BEGIN IMMEDIATE`
transaction that re-reads the version, so concurrent openers and
interrupted upgrades are safe and reopening is idempotent.
| Version | Content |
| --- | --- |
| 0 | empty file |
| 1 | prototype schema of the original podcast PR (`podcasts`, `podcast_episodes`), verbatim |
| 2 | the model below; the prototype tables are renamed and imported |
* Before writing anything (including the WAL switch), the store refuses a
file with a `user_version` above its own, and a file whose tables are not
the expected ones for its version (for example `library.db` passed by
mistake). The file is left byte-for-byte unchanged.
* Prototype import (migration 2). Tables are renamed `legacy_podcasts` and
`legacy_podcast_episodes`, never dropped.
* Each feed becomes a podcast with `podcast_id` from the canonical feed URL
(`FeedUrl::parse`, then `FeedUrl::from_user_input`) and a subscription
register `subscribed = true`, generation 0, `hlc_ms` = `added_at`,
`op_id` = `legacy-import:<podcast_id>`, not shared. Every contract write
orders after it; other devices learn it through the snapshot sections.
Rows with the same canonical URL merge (first row wins).
* URLs that cannot be canonicalized, episodes with a non-`http(s)` audio URL,
invalid metadata or no identity are listed in `legacy_quarantine`; the
rows stay in the `legacy_*` tables.
* Episode ids come from `resolve_episode_ids` over the stored items. The
prototype stored the enclosure URL as `guid` when an item had none: such
rows are imported as GUID-less items and keep the value in
`episodes.legacy_guid`. The prototype kept one row per enclosure URL, so
rows with the same publication second, normalized title and `<link>`
(or none) are one item whose audio URL moved: the newest row
(`first_seen_at`, then `id`) is imported, the others are quarantined as
duplicates.
* Imported GUID-less episodes are **provisional**: stale rows can still
change which key is unique among the stored items, so their ids may
differ from what a device without the prototype derives. On a refresh,
item ids are resolved against the stored episodes except provisional
ones without progress or sessions; each such provisional episode is
matched to its item (enclosure, then link, then date and title, GUIDs
ignored) and re-keyed to that item's id. When another episode that
stays already holds that id (for example one a remote operation created
before this device's first refresh), it is the item's episode and the
provisional one is a duplicate: it is deleted rather than kept as a
second copy. The same applies when `legacy_guid` turns out to be the
item's real GUID. Listing an episode clears `legacy_guid`, which makes
its id final. Provisional episodes carry no personal state, so nothing
personal moves or is lost; an imported episode played before its first
refresh is not provisional and keeps its id (the feed item then becomes
a separate episode if the ids differ).
* ETag, Last-Modified, check time, last error and the cover cache move to the
fetch state. The prototype stores no progress or history, so none is
created.
### Tables (version 2)
| Table | Key | Content | Synced |
| --- | --- | --- | --- |
| `podcasts` | `podcast_id` | local channel metadata, `feed_url` = URL this device fetches, `meta_from_feed` | catalog, no |
| `podcast_feed_aliases` | `(podcast_id, feed_url)` | bounded grow-only alias set (`merge_feed_aliases`) | yes, inside subscription records and `podcast_feed_aliases_added` |
| `episodes` | `episode_id` | `EpisodeMeta` columns (season, number, type, published, duration, audio URL, MIME, length, artwork, link), `source` (`feed`/`sync`), `in_feed`, `legacy_guid` | catalog, no |
| `podcast_seasons` | `(podcast_id, season)` | derived from `episodes`: episode count, first/last publication; `season` NULL groups episodes without a season | no |
| `podcast_subscriptions` | `podcast_id` | subscription register: `subscribed`, `PodcastMeta` JSON, `generation`, `hlc_ms`, `op_id` | yes |
| `podcast_shares` | `podcast_id` | share register | yes |
| `episode_progress` | `episode_id` | progress register (`ProgressRecord`) | yes |
| `podcast_listen_sessions` | `session_id` | grow-only set of `ListenSession` | yes |
| `podcast_fetch_state` | `podcast_id` | ETag, Last-Modified, checked / last success / next check times, last error, consecutive failures, cover cache | **never** |
| `legacy_quarantine` | `(legacy_table, legacy_id)` | prototype rows that could not be imported | no |
### Invariants
* Registers and sessions have no foreign keys to the catalog: operations for
podcasts or episodes the device has not fetched are stored as they arrive,
in any order. An operation's `PodcastEpisodeRef` adds a `source = 'sync'`
catalog row only when the episode is unknown; feed metadata always wins.
* One apply path. Local actions validate the payload, draw a stamp from the
client's sync log, build the `SyncOpWire`, and apply it like a received
operation: `SyncOpWire::podcast_*` materializers, then `merge_*`. The
returned operation must be appended with exactly that stamp. Snapshot
records go through the same functions after validation. The result is
therefore commutative, associative and idempotent, and a snapshot never
replaces newer local state.
* Invalid operations (`validate_podcast`, a generation above `i64::MAX`, a
session whose identity differs from the stored one) are skipped and
reported as rejected, so the client can still acknowledge them.
* Only `subscribe`/`unsubscribe` write the subscription register and only
`set_shared`/`unsubscribe` the share register, with
`next_podcast_generation`. No-op actions draw no stamp and emit nothing.
`unsubscribe` emits `subscribed = false` and `shared = false` and keeps
progress, sessions and the catalog.
* Explicit progress actions use `explicit_generation(max(local, carried))`;
checkpoints keep the session generation and are refused (`Superseded`)
once the register holds a greater one, or after the session completed.
* RSS refreshes update the catalog (channel metadata, episode metadata and
audio URLs, `in_feed`) and the alias set only; episodes missing from the
feed are kept. Permanent redirects and `itunes:new-feed-url` become the
fetch URL and aliases, and emit `podcast_feed_aliases_added` only when the
stored alias set changes.
* Snapshot export reads each section newest first up to its count, then
applies `bound_podcast_state_to(budget)`. Fetch state and the catalog
(descriptions, episodes) are never exported.
* Feed URLs resolve through `resolve_podcast_id` over the alias set, the
catalog fetch URL and the register's `podcast.feed_url`.
* Fetch-state writes (`record_fetch_*`, `set_cover_cache`) read and write
the row in one immediate transaction, so concurrent writers on one or
several connections never lose a column or a failure increment.
* `TrackItem::podcast_episode` builds queue items whose identity is the
explicit source `TrackItem.podcast` alone. Their `id`, `release_id` and
artist id are the fixed sentinel `TrackItem::NOT_A_LIBRARY_ROW` (-1),
shared by every episode, the same "not a library row" convention as
pending federated tracks. The sentinel carries no identity and is never
decoded; it only keeps music-only code that predates podcasts
(`id >= 0`, `release_id < 0` checks) from taking an episode for library
track, release or artist 0. There is no reserved range of negative ids.
* `TrackItem::key` (a typed `TrackItemKey`; string form `lib:<id>`,
`fed:<owner>:<item_id>`, `ep:<episode id>`) is the source identity of a
queue item; on the wire it is `PlaybackSource`. An episode is identified
by `TrackItem.podcast` / `ep:` only and must never be keyed by its numeric
id: keying by id (for example a `local:{id}` string) would merge all
episodes into one. `TrackItemKey` has no content-id equivalence (a pending
federated item and its imported library copy differ), so clients that
match music content-first keep doing that and add the podcast branch
before any fallback to the id.
* Wire ids of podcast items and of Jam private placeholders are `0`
(`PlaybackTrack::podcast_episode`, `PlaybackTrack::private_placeholder`)
and must not be copied into a `TrackItem`.
`TrackItem::from_playback_track` classifies a received item as
`PlaybackItem::Music` (the client's music path), `PodcastEpisode` (built
with `TrackItem::podcast_episode`, sentinel ids) or `Private` (not
playable, no `TrackItem`). Besides the ids, `TrackItem::podcast_episode`
also differs from the wire form in `release_year` (publication year) and
`audio_format` (`None`; the MIME type stays in the reference).
## Versioning
* `music-dht` 0.6.0 and `furumi-library` 0.3.0: new public types, enum
+4 -1
View File
@@ -16,8 +16,11 @@ anyhow = { workspace = true }
blake3 = { workspace = true }
directories = { workspace = true }
lofty = { workspace = true }
music-dht = { workspace = true }
music-dht = { workspace = true, features = ["podcast-feed"] }
rusqlite = { workspace = true, features = ["functions"] }
serde = { workspace = true }
serde_json = { workspace = true }
tracing = { workspace = true }
[dev-dependencies]
tempfile = { workspace = true }
+60 -2
View File
@@ -21,5 +21,63 @@ library used by every client. Connections use WAL, foreign keys, and a busy
timeout so independently running clients can safely share the database.
Version 0.3 follows `music-dht` 0.6, whose device-sync and podcast contract
types it re-exposes. Podcast persistence is not part of this release; it will
implement the rules of the workspace `PODCASTS.md`.
types it re-exposes.
## Podcasts
`furumi_library::podcasts::PodcastStore` is the SQLite reference store of the
podcast contract (workspace `PODCASTS.md`, section "SQLite reference store").
It lives in its own file, conventionally `podcasts.sqlite3`
(`PODCAST_DB_FILE`) next to `library.db`; the caller passes the path.
- **Migrations** run on `PRAGMA user_version`, each in its own transaction.
A database of the original podcast prototype (`user_version` 1) is imported:
its tables are renamed `legacy_podcasts` / `legacy_podcast_episodes` (never
dropped), feeds become subscriptions with contract ids, rows that cannot be
imported are listed in `legacy_quarantine`. A database from a newer Furumi,
or a file that is not a podcast database, is refused without being touched.
- **Local actions** (`subscribe`, `unsubscribe`, `set_shared`,
`start_session`, `checkpoint`, `seek`, `restart`, `complete_session`,
`mark_played`, `mark_unplayed`, `record_listen_session`) draw stamps from a
`StampSource` (usually the client's sync log), apply the operation and
return `PodcastLocalOp`s to append to the log with exactly those stamps.
- **Sync**: `apply_remote_op` applies the five `podcast_*` operations,
`fill_snapshot` / `export_snapshot` produce the bounded snapshot sections
and `apply_snapshot` merges them. Everything goes through the `music-dht`
materializers and merge functions, so the result does not depend on
delivery order and never loses newer local state.
- **RSS**: `apply_feed` ingests a document parsed by
`music_dht::podcast::feed` (the crate enables the `podcast-feed` feature):
channel metadata, episodes with stable ids and current audio URLs, feed
aliases from permanent redirects and `itunes:new-feed-url`. It never
touches subscriptions, progress or history. HTTP, conditional requests and
scheduling stay in the client; `record_fetch_*`, `fetch_state` and
`podcasts_due_for_refresh` keep the local-only fetch state, which is never
synced.
- **Queries**: `list_podcasts`, `seasons`, `episodes` (season, played-state
and text filters, date or episode-number order), `resume_position_ms`,
`episode_ref` / `episode_track` for queueing, `listen_history`.
The store is synchronous: methods take `&self` and lock one connection, so
`Arc<PodcastStore>` can be shared; run calls on a blocking pool.
`TrackItem.podcast` marks a queue item as a podcast episode
(`TrackItem::podcast_episode`). Music-only code (likes, playlists, history,
content ids, federation downloads) must check `TrackItem::is_podcast`; the
music library never stores episodes.
`TrackItem::key` returns the typed source identity of a queue item
(`TrackItemKey`: `lib:<id>`, `fed:<owner>:<item_id>` or `ep:<episode id>` as
a string). A negative `TrackItem::id` only means "not a library row". Every
podcast item has the same sentinel `id`, `release_id` and artist id
(`TrackItem::NOT_A_LIBRARY_ROW` = -1). It carries no identity and must not
be decoded; it only keeps music-only code that predates podcasts from taking
an episode for library track, release or artist 0. The episode's identity is
`TrackItem.podcast` alone, so queue, current-track and position matching
must check it before any fallback to the id. `TrackItemKey` does not equate
a pending federated item with its library copy; clients that match music by
content id keep doing so.
Convert received `PlaybackTrack`s with `TrackItem::from_playback_track`
(`PlaybackItem::Music`, `PodcastEpisode` or `Private`): podcast items and
Jam private placeholders carry id 0 on the wire, which must never be copied.
+8 -2
View File
@@ -6,9 +6,13 @@
//!
//! All methods take `&self` and lock the connection internally, so an
//! `Arc<Library>` can be shared across blocking tasks.
//!
//! Podcasts live in a separate database, [`podcasts::PodcastStore`]
//! (`podcasts.sqlite3`); the music library never stores episodes.
pub mod import;
pub mod models;
pub mod podcasts;
use std::collections::{HashMap, HashSet};
use std::io::Write as _;
@@ -22,8 +26,8 @@ use rusqlite::{Connection, OptionalExtension as _, params};
pub use models::{
ArtistCard, ArtistDetail, ArtistRef, ArtistsPage, Availability, FederatedTrack, LibraryFilters,
LibrarySourceMode, PlaylistCard, PlaylistDetail, ReleaseCard, ReleaseDetail, ReleaseEdit,
SearchResults, TrackEdit, TrackItem,
LibrarySourceMode, PlaybackItem, PlaylistCard, PlaylistDetail, ReleaseCard, ReleaseDetail,
ReleaseEdit, SearchResults, TrackEdit, TrackItem, TrackItemKey,
};
/// The virtual "Liked tracks" playlist id, kept from the server API.
@@ -3283,6 +3287,7 @@ fn query_tracks(
artists: Vec::new(),
featured_artists: Vec::new(),
fed: None,
podcast: None,
})
})?
.collect::<rusqlite::Result<Vec<_>>>()?;
@@ -3348,6 +3353,7 @@ fn pending_track(fed: &FederatedTrack) -> TrackItem {
file_size_bytes: None,
play_count: 0,
fed: Some(fed.clone()),
podcast: None,
}
}
+465
View File
@@ -1,6 +1,8 @@
//! Data shapes the views render. They mirror what the furumusic API used to
//! return, but every field is now filled from the local SQLite library.
use music_dht::device_sync::{PlaybackSource, PlaybackTrack};
use music_dht::podcast::{EpisodeId, PodcastEpisodeRef};
use serde::{Deserialize, Serialize};
/// Which catalog sources a client wants library queries to include.
@@ -120,6 +122,56 @@ pub struct ArtistRef {
pub name: String,
}
/// A playable queue item: a library track, a pending federated track or a
/// podcast episode.
///
/// # Identity
///
/// What an item *is* comes from its explicit source fields, never from the
/// numeric `id` alone; [`TrackItem::key`] returns it as a typed
/// [`TrackItemKey`]:
///
/// * a library track is a library row: `id >= 0`, `fed` and `podcast` are
/// `None`;
/// * a federated track is identified by [`TrackItem::fed`] (owner and item
/// id). While it is not in the library its `id` is a negative,
/// process-local counter value (pending federated tracks count down from
/// -1) and `release_id` / artist ids are -1;
/// * a podcast episode is identified **only** by [`TrackItem::podcast`] (its
/// [`EpisodeId`]). Its `id`, `release_id` and artist id are the fixed
/// sentinel [`TrackItem::NOT_A_LIBRARY_ROW`] (-1), shared by every episode.
///
/// A negative `id` therefore means "not a library row" and nothing else: it
/// carries no identity and must never be decoded, compared across items or
/// used as a key. The -1 sentinel only keeps music-only code that predates
/// podcasts (`id >= 0` / `release_id < 0` checks) from treating an episode
/// as library track, release or artist 0.
///
/// [`TrackItem::key`] is the *source* identity. It does not merge a pending
/// federated item with its imported library copy, which share a content id
/// but have different keys (`fed:` and `lib:`). Clients that key playback
/// items content-first (content id, then federated source, then id) keep
/// doing so for music; what they must add is a podcast branch
/// ([`TrackItem::podcast`], or [`TrackItemKey::PodcastEpisode`]) checked
/// before any fallback to `id`. An episode must never be keyed by `id`: a
/// `local:{id}` key would merge all episodes into one. On the wire the item
/// kind is `music_dht::device_sync::PlaybackSource`; convert received items
/// with [`TrackItem::from_playback_track`], never by copying wire ids.
///
/// # Podcast episodes
///
/// Podcast episodes carry their [`PodcastEpisodeRef`] in
/// [`TrackItem::podcast`]: `file_path` is empty, `content_id`, `fed` and
/// `cover_path` are `None`, and the display fields mirror the reference
/// (episode title, podcast title as artist and release). The audio URL is
/// only in the reference: it is never a local path, never content-addressed
/// and never offered for P2P transfer.
///
/// **Music-only code paths must check [`TrackItem::is_podcast`]** (or
/// `podcast.is_none()`) before treating an item as music: likes, playlists,
/// listening history, content-id lookups, federation downloads, similarity
/// and file operations do not apply to episodes. Library queries never
/// return podcast items; the library database stores none.
#[derive(Debug, Clone)]
pub struct TrackItem {
pub id: i64,
@@ -149,9 +201,187 @@ pub struct TrackItem {
/// carries everything needed to download them from the owning peer.
/// With an empty `file_path` the player resolves the track on demand.
pub fed: Option<FederatedTrack>,
/// Set for podcast episodes (see the type documentation); `None` for music.
pub podcast: Option<PodcastEpisodeRef>,
}
/// Typed source identity of a [`TrackItem`], from [`TrackItem::key`].
///
/// Two items with equal keys are the same track or episode; the same track
/// may still appear more than once in a queue. The key has no content-id
/// equivalence: a pending federated item and its library copy differ. Use it
/// where the source matters (and always for podcast episodes, which have no
/// other identity); code that matches music by content id keeps doing that
/// and checks the podcast variant first (see [`TrackItem`]).
///
/// The [`Display`](std::fmt::Display) form is stable and suitable as a map
/// key or in logs:
///
/// | Variant | String form |
/// | --- | --- |
/// | `Library(123)` | `lib:123` |
/// | `Federated { owner, item_id }` | `fed:<owner>:<item_id>` |
/// | `PodcastEpisode(id)` | `ep:<episode id>`, e.g. `ep:ep1:...` |
/// | `Transient(-5)` | `tmp:-5` |
///
/// `owner` is a peer endpoint id and contains no `:`, so the federated form
/// is unambiguous. Keys are not persisted by this crate; a `Transient` key is
/// only meaningful inside the process that built the item.
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub enum TrackItemKey {
/// A row of the local music library (`TrackItem::id >= 0`).
Library(i64),
/// A federated track that is not a library row, identified by its owner
/// peer and DHT item id ([`FederatedTrack::owner`] /
/// [`FederatedTrack::item_id`]), not by its process-local negative `id`.
Federated { owner: String, item_id: String },
/// A podcast episode, identified by its [`EpisodeId`] only.
PodcastEpisode(EpisodeId),
/// A non-library item with no explicit source (negative `id`, no `fed`,
/// no `podcast`). furumi-library never builds one; clients that do must
/// keep its `id` unique within the process.
Transient(i64),
}
/// What a received `music_dht::device_sync::PlaybackTrack` is locally, from
/// [`TrackItem::from_playback_track`].
#[derive(Debug, Clone)]
pub enum PlaybackItem {
/// A music track. Convert it with the client's music path (content id,
/// then federated source); its wire ids are the sender's.
Music,
/// A podcast episode, built with [`TrackItem::podcast_episode`].
PodcastEpisode(Box<TrackItem>),
/// A Jam private placeholder: not playable and without identity. Show it
/// as `music_dht::device_sync::PRIVATE_ITEM_TITLE` where the queue is
/// displayed; never insert it as a library or queue track, never key it.
Private,
}
impl std::fmt::Display for TrackItemKey {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
Self::Library(id) => write!(f, "lib:{id}"),
Self::Federated { owner, item_id } => write!(f, "fed:{owner}:{item_id}"),
Self::PodcastEpisode(episode_id) => write!(f, "ep:{}", episode_id.as_str()),
Self::Transient(id) => write!(f, "tmp:{id}"),
}
}
}
impl TrackItem {
/// The `id`, `release_id` and artist id of items that are not library
/// rows and have no numeric identity: every podcast episode uses it.
///
/// It is a sentinel, not an identifier: it is shared by all episodes
/// (and can coincide with the first pending federated track's id), so it
/// must never be decoded or used as a key. See [`TrackItem::key`].
pub const NOT_A_LIBRARY_ROW: i64 = -1;
/// The typed identity of this item; see [`TrackItemKey`].
///
/// Precedence: [`TrackItem::podcast`] makes it a podcast episode; else a
/// non-negative `id` is a library row; else [`TrackItem::fed`] makes it
/// a federated track; anything else is [`TrackItemKey::Transient`].
pub fn key(&self) -> TrackItemKey {
if let Some(episode) = &self.podcast {
return TrackItemKey::PodcastEpisode(episode.episode_id.clone());
}
if self.id >= 0 {
return TrackItemKey::Library(self.id);
}
match &self.fed {
Some(fed) => TrackItemKey::Federated {
owner: fed.owner.clone(),
item_id: fed.item_id.clone(),
},
None => TrackItemKey::Transient(self.id),
}
}
/// Queue item for a podcast episode streamed from the publisher.
///
/// Title, artist and release follow `PlaybackTrack::podcast_episode` of
/// `music_dht::device_sync` (the episode title, the podcast title as
/// artist and release, the announced duration; no track or disc number).
/// The other fields differ from that wire form on purpose:
///
/// * `id`, `release_id` and the artist id are the sentinel
/// [`TrackItem::NOT_A_LIBRARY_ROW`] (-1) for every episode, where the
/// wire form carries `0`; the item's identity is [`TrackItem::podcast`]
/// alone ([`TrackItem::key`]);
/// * `release_year` is the publication year (`None` on the wire);
/// * `audio_format` is `None` (the wire form carries the MIME type, which
/// stays in the reference).
///
/// Season and episode number stay in the reference.
pub fn podcast_episode(episode: PodcastEpisodeRef) -> Self {
Self {
id: Self::NOT_A_LIBRARY_ROW,
title: episode.episode_title.clone(),
track_number: None,
disc_number: None,
duration_seconds: episode
.duration_ms
.map_or(0.0, |ms| ms.max(0) as f64 / 1_000.0),
artists: vec![ArtistRef {
id: Self::NOT_A_LIBRARY_ROW,
name: episode.podcast_title.clone(),
}],
featured_artists: Vec::new(),
release_id: Self::NOT_A_LIBRARY_ROW,
release_title: episode.podcast_title.clone(),
release_year: episode.published_at_ms.map(year_of_unix_ms),
file_path: String::new(),
content_id: None,
cover_path: None,
audio_format: None,
audio_bitrate: None,
audio_sample_rate: None,
audio_bit_depth: None,
file_size_bytes: None,
play_count: 0,
fed: None,
podcast: Some(episode),
}
}
/// Classifies a received `music_dht::device_sync::PlaybackTrack` and
/// builds the local queue item for non-music sources.
///
/// The wire `id` and `release_id` of podcast episodes and of Jam private
/// placeholders are `0`, and those of music items are the sender's local
/// ids. None of them identifies anything on this device, so they are
/// never copied:
///
/// * a podcast episode becomes [`TrackItem::podcast_episode`] of its
/// [`PlaybackSource::PodcastEpisode`] reference (sentinel ids, key
/// `ep:`), whatever the legacy display fields say;
/// * a private placeholder ([`PlaybackSource::Private`]) is not playable
/// and has no identity, so no `TrackItem` exists for it;
/// * music (no source, or [`PlaybackSource::Music`]) is left to the
/// client's music path, which resolves it by content id or federated
/// source.
pub fn from_playback_track(track: &PlaybackTrack) -> PlaybackItem {
match &track.source {
None | Some(PlaybackSource::Music) => PlaybackItem::Music,
Some(PlaybackSource::PodcastEpisode(episode)) => PlaybackItem::PodcastEpisode(
Box::new(Self::podcast_episode(episode.as_ref().clone())),
),
Some(PlaybackSource::Private) => PlaybackItem::Private,
}
}
/// Whether this item is a podcast episode (not music).
pub fn is_podcast(&self) -> bool {
self.podcast.is_some()
}
/// The episode reference of a podcast item.
pub fn podcast_episode_ref(&self) -> Option<&PodcastEpisodeRef> {
self.podcast.as_ref()
}
/// A federated track that still needs downloading before playback.
pub fn is_fed_pending(&self) -> bool {
self.fed.is_some() && self.file_path.is_empty()
@@ -202,6 +432,21 @@ impl TrackItem {
}
}
/// Proleptic Gregorian year of a unix timestamp in milliseconds (UTC).
fn year_of_unix_ms(ms: i64) -> i32 {
// Howard Hinnant's civil_from_days.
let days = ms.div_euclid(86_400_000);
let z = days + 719_468;
let era = z.div_euclid(146_097);
let doe = z - era * 146_097;
let yoe = (doe - doe / 1_460 + doe / 36_524 - doe / 146_096) / 365;
let doy = doe - (365 * yoe + yoe / 4 - yoe / 100);
let mp = (5 * doy + 2) / 153;
let month = if mp < 10 { mp + 3 } else { mp - 9 };
let year = yoe + era * 400 + i64::from(month <= 2);
i32::try_from(year).unwrap_or(if year < 0 { i32::MIN } else { i32::MAX })
}
#[derive(Debug, Clone)]
pub struct ReleaseCard {
pub id: i64,
@@ -340,8 +585,228 @@ mod tests {
file_size_bytes: None,
play_count: 0,
fed: None,
podcast: None,
};
assert_eq!(track.artist_line(), "Main feat. Guest, Other");
assert!(!track.is_podcast());
}
#[test]
fn year_of_unix_ms_handles_epoch_and_leap_years() {
assert_eq!(year_of_unix_ms(0), 1970);
assert_eq!(year_of_unix_ms(-1), 1969);
// 2024-02-29T12:00:00Z and 2024-12-31T23:59:59Z.
assert_eq!(year_of_unix_ms(1_709_208_000_000), 2024);
assert_eq!(year_of_unix_ms(1_735_689_599_000), 2024);
assert_eq!(year_of_unix_ms(1_735_689_600_000), 2025);
}
fn episode_ref(guid: &str) -> PodcastEpisodeRef {
use music_dht::podcast::{FeedUrl, PodcastId};
let feed_url = FeedUrl::parse("https://example.test/feed.xml").unwrap();
let podcast_id = PodcastId::from_feed_url(&feed_url);
PodcastEpisodeRef {
episode_id: EpisodeId::from_guid(&podcast_id, guid).unwrap(),
podcast_id,
feed_url,
podcast_title: "Show".into(),
episode_title: "Pilot".into(),
audio_url: "https://cdn.example.test/1.mp3".into(),
mime: Some("audio/mpeg".into()),
duration_ms: Some(90_500),
published_at_ms: Some(1_709_208_000_000),
season: Some(2),
episode_number: Some(7),
image_url: None,
progress_generation: None,
}
}
fn library_track(id: i64) -> TrackItem {
TrackItem {
id,
title: "Track".into(),
track_number: None,
disc_number: None,
duration_seconds: 1.0,
artists: vec![artist("Main")],
featured_artists: Vec::new(),
release_id: 1,
release_title: "Release".into(),
release_year: None,
file_path: "/tmp/track.mp3".into(),
content_id: None,
cover_path: None,
audio_format: None,
audio_bitrate: None,
audio_sample_rate: None,
audio_bit_depth: None,
file_size_bytes: None,
play_count: 0,
fed: None,
podcast: None,
}
}
fn federated(owner: &str, item_id: &str) -> FederatedTrack {
FederatedTrack {
item_id: item_id.into(),
owner: owner.into(),
own: false,
title: "Remote".into(),
artist_names: vec!["Someone".into()],
featured_artist_names: Vec::new(),
year: None,
duration_seconds: None,
content_id: None,
release_title: None,
track_number: None,
disc_number: None,
}
}
#[test]
fn podcast_track_mirrors_the_episode_reference() {
let episode = episode_ref("e1");
let track = TrackItem::podcast_episode(episode.clone());
assert!(track.is_podcast());
assert_eq!(track.podcast_episode_ref(), Some(&episode));
// The sentinel marks "not a library row, release or artist" and
// carries no identity.
assert_eq!(TrackItem::NOT_A_LIBRARY_ROW, -1);
assert_eq!(track.id, -1);
assert_eq!(track.release_id, -1);
assert!(track.artists.iter().all(|artist| artist.id == -1));
assert!(track.featured_artists.is_empty());
assert!(track.file_path.is_empty());
assert!(track.content_id.is_none() && track.fed.is_none());
assert!(!track.is_fed_pending());
assert_eq!(track.title, "Pilot");
assert_eq!(track.artist_line(), "Show");
assert_eq!(track.release_title, "Show");
assert_eq!(track.release_year, Some(2024));
assert_eq!(track.duration_label(), "1:31");
}
#[test]
fn key_is_typed_by_source() {
let library = library_track(123);
assert_eq!(library.key(), TrackItemKey::Library(123));
assert_eq!(library.key().to_string(), "lib:123");
assert_eq!(library_track(0).key(), TrackItemKey::Library(0));
let mut pending = library_track(-5);
pending.release_id = -1;
pending.file_path.clear();
pending.fed = Some(federated("peer1", "item-9"));
let fed_key = TrackItemKey::Federated {
owner: "peer1".into(),
item_id: "item-9".into(),
};
assert_eq!(pending.key(), fed_key);
assert_eq!(pending.key().to_string(), "fed:peer1:item-9");
// The process-local negative id is not part of a federated identity.
let mut renumbered = pending.clone();
renumbered.id = -6;
assert_eq!(renumbered.key(), pending.key());
let episode = episode_ref("e1");
let track = TrackItem::podcast_episode(episode.clone());
let ep_key = TrackItemKey::PodcastEpisode(episode.episode_id.clone());
assert_eq!(track.key(), ep_key);
assert_eq!(
track.key().to_string(),
format!("ep:{}", episode.episode_id.as_str())
);
assert!(track.key().to_string().starts_with("ep:ep1:"));
let transient = library_track(-7);
assert_eq!(transient.key(), TrackItemKey::Transient(-7));
assert_eq!(transient.key().to_string(), "tmp:-7");
// The source wins over the numeric id: a podcast item is never keyed
// as a library row, a federated item never by its counter value.
let mut odd = TrackItem::podcast_episode(episode);
odd.id = 42;
assert_eq!(odd.key(), ep_key);
let mut pending_minus_one = pending.clone();
pending_minus_one.id = TrackItem::NOT_A_LIBRARY_ROW;
assert_eq!(pending_minus_one.key(), fed_key);
assert_ne!(pending_minus_one.key(), track.key());
}
#[test]
fn received_playback_tracks_never_copy_wire_ids() {
// F1 wire form of an episode: id 0, release_id 0.
let episode = episode_ref("e1");
let wire = PlaybackTrack::podcast_episode(episode.clone());
assert_eq!((wire.id, wire.release_id), (0, 0));
let PlaybackItem::PodcastEpisode(item) = TrackItem::from_playback_track(&wire) else {
panic!("an episode is not music");
};
assert_eq!(item.id, TrackItem::NOT_A_LIBRARY_ROW);
assert_eq!(item.release_id, TrackItem::NOT_A_LIBRARY_ROW);
assert!(item.artists.iter().all(|artist| artist.id < 0));
assert_eq!(
item.key(),
TrackItemKey::PodcastEpisode(episode.episode_id.clone())
);
assert_eq!(item.podcast_episode_ref(), Some(&episode));
// Legacy display fields an older sender filled in do not matter.
let mut odd = wire.clone();
odd.id = 42;
odd.release_id = 7;
odd.title = "Something else".into();
let PlaybackItem::PodcastEpisode(odd) = TrackItem::from_playback_track(&odd) else {
panic!("the source decides");
};
assert_eq!(odd.key(), item.key());
assert_eq!(odd.id, TrackItem::NOT_A_LIBRARY_ROW);
// A private placeholder (also id 0) never becomes library row 0.
let private = PlaybackTrack::private_placeholder();
assert_eq!(private.id, 0);
assert!(matches!(
TrackItem::from_playback_track(&private),
PlaybackItem::Private
));
// Music is left to the client's music path.
let mut music = wire;
music.source = None;
assert!(matches!(
TrackItem::from_playback_track(&music),
PlaybackItem::Music
));
music.source = Some(PlaybackSource::Music);
assert!(matches!(
TrackItem::from_playback_track(&music),
PlaybackItem::Music
));
}
#[test]
fn episodes_share_the_sentinel_id_but_not_the_key() {
let first = TrackItem::podcast_episode(episode_ref("e1"));
let second = TrackItem::podcast_episode(episode_ref("e2"));
assert_eq!(first.id, TrackItem::NOT_A_LIBRARY_ROW);
assert_eq!(second.id, TrackItem::NOT_A_LIBRARY_ROW);
assert_eq!(first.id, second.id);
assert_eq!(first.release_id, second.release_id);
assert_ne!(first.key(), second.key());
assert_ne!(first.key().to_string(), second.key().to_string());
assert_eq!(
TrackItem::podcast_episode(episode_ref("e1")).key(),
first.key()
);
let keys: std::collections::HashSet<TrackItemKey> =
[first.key(), second.key(), first.key()]
.into_iter()
.collect();
assert_eq!(keys.len(), 2);
let mut ordered = [second.key(), first.key(), TrackItemKey::Library(1)];
ordered.sort();
assert_eq!(ordered[0], TrackItemKey::Library(1));
}
}
@@ -0,0 +1,544 @@
//! Local user actions: they build the device-sync operation, apply it through
//! the shared code path and return it for the client's sync log.
use anyhow::Result;
use music_dht::device_sync::SyncOpPayload;
use music_dht::podcast::{
EpisodeId, FeedUrl, ListenSession, PodcastEpisodeRef, PodcastId, PodcastMeta, ProgressRecord,
ProgressStatus, effective_share, merge_feed_aliases, merge_listen_session,
next_podcast_generation,
};
use rusqlite::{Connection, TransactionBehavior};
use super::apply::{self, ApplyOutcome};
use super::{OpStamp, PodcastLocalOp, PodcastStore, PodcastStoreError, StampSource};
/// Result of [`PodcastStore::subscribe`].
#[derive(Debug, Clone, PartialEq)]
pub struct SubscribeOutcome {
/// Podcast the URL resolved to (an existing one when the URL is a known
/// feed URL or alias).
pub podcast_id: PodcastId,
/// The podcast was already subscribed; no operation was emitted.
pub already_subscribed: bool,
/// Operations to append to the sync log.
pub ops: Vec<PodcastLocalOp>,
}
/// A listening session of this device as seen by the progress register.
///
/// Returned by explicit actions; periodic checkpoints of the session use its
/// `generation` ([`PodcastStore::checkpoint`]).
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ProgressSession {
/// Episode being played.
pub episode: PodcastEpisodeRef,
/// Session id (also the `ListenSession::session_id`).
pub session_id: String,
/// Generation of the session's latest explicit action.
pub generation: u64,
}
/// A progress register written by a local action.
#[derive(Debug, Clone, PartialEq)]
pub struct ProgressWrite {
/// The register as written.
pub record: ProgressRecord,
/// The session after the action (its generation for later checkpoints).
pub session: ProgressSession,
/// Operation to append to the sync log.
pub op: PodcastLocalOp,
}
/// Result of [`PodcastStore::checkpoint`].
#[derive(Debug, Clone, PartialEq)]
pub enum CheckpointOutcome {
/// The checkpoint was written.
Written(Box<ProgressWrite>),
/// The register holds a greater generation (another device or an
/// explicit action took over). Nothing was written: stop writing at the
/// old generation, or continue locally with an explicit action
/// ([`PodcastStore::start_session`] / [`PodcastStore::seek`]).
Superseded {
/// Current register.
current: ProgressRecord,
},
/// This session already completed the episode at its generation; a
/// checkpoint would turn it back into "in progress". Nothing was written.
AlreadyCompleted {
/// Current register.
current: ProgressRecord,
},
}
#[derive(Debug, Clone, Copy)]
enum ExplicitAction {
Start(i64),
Seek(i64),
Restart,
Complete(i64),
MarkPlayed,
MarkUnplayed,
}
fn invalid(reason: impl ToString) -> anyhow::Error {
PodcastStoreError::Invalid(reason.to_string()).into()
}
/// Validates the payload (with a provisional stamp), draws the real stamp,
/// and applies the operation through [`apply::apply_op`].
pub(crate) fn emit(
conn: &Connection,
stamps: &mut dyn StampSource,
make: impl Fn(&OpStamp) -> SyncOpPayload,
) -> Result<PodcastLocalOp> {
let device_id = stamps.device_id().to_string();
let provisional = OpStamp::new("provisional", device_id.clone(), 0);
provisional
.wire(make(&provisional))
.validate_podcast()
.map_err(invalid)?;
let stamp = stamps.next_stamp()?;
if stamp.origin_device_id != device_id {
return Err(invalid("stamp origin differs from the local device id"));
}
let payload = make(&stamp);
let op = stamp.wire(payload.clone());
match apply::apply_op(conn, &op)? {
ApplyOutcome::Rejected(reason) => Err(invalid(reason)),
_ => Ok(PodcastLocalOp { stamp, payload }),
}
}
/// Resolves a feed URL against every known URL of every podcast.
pub(crate) fn resolve_podcast(conn: &Connection, url: &FeedUrl) -> Result<PodcastId> {
let mut statement = conn.prepare_cached(
"SELECT podcast_id FROM podcast_feed_aliases WHERE feed_url = ?1
UNION SELECT podcast_id FROM podcasts WHERE feed_url = ?1
UNION SELECT podcast_id FROM podcast_subscriptions
WHERE json_extract(podcast_json, '$.feed_url') = ?1",
)?;
let ids = statement
.query_map([url.as_str()], |row| super::podcast_id_col(row.get(0)?))?
.collect::<rusqlite::Result<Vec<_>>>()?;
let pairs: Vec<(PodcastId, FeedUrl)> = ids.into_iter().map(|id| (id, url.clone())).collect();
Ok(music_dht::podcast::resolve_podcast_id(
url,
pairs.iter().map(|(id, url)| (id, url)),
))
}
fn catalog_meta(conn: &Connection, podcast_id: &PodcastId) -> Result<Option<PodcastMeta>> {
use rusqlite::OptionalExtension as _;
Ok(conn
.query_row(
"SELECT feed_url, title, author, description, link, image_url
FROM podcasts WHERE podcast_id = ?1",
[podcast_id.as_str()],
|row| {
Ok(PodcastMeta {
feed_url: super::feed_url_col(row.get(0)?)?,
title: row.get(1)?,
author: row.get(2)?,
description: row.get(3)?,
link: row.get(4)?,
image_url: row.get(5)?,
})
},
)
.optional()?)
}
fn share_payload(conn: &Connection, podcast_id: &PodcastId, shared: bool) -> Result<SyncOpPayload> {
let subscription = apply::load_subscription_register(conn, podcast_id)?;
let share = apply::load_share(conn, podcast_id)?;
Ok(SyncOpPayload::PodcastShareSet {
podcast_id: podcast_id.clone(),
shared,
generation: next_podcast_generation(subscription.as_ref(), share.as_ref()),
})
}
/// The reference carried by progress operations: identity and display only.
fn op_ref(episode: &PodcastEpisodeRef) -> PodcastEpisodeRef {
let mut episode = episode.clone();
episode.progress_generation = None;
episode
}
impl PodcastStore {
/// Subscribes to the podcast of `podcast.feed_url` (explicit user action).
///
/// The URL is resolved against known feed URLs and aliases first, so
/// subscribing again, through a redirect target or from someone else's
/// card reuses the existing podcast with its progress and history. When
/// the podcast is already subscribed nothing is emitted. Otherwise one
/// `podcast_subscription_set` with generation
/// `next_podcast_generation(..)` and the known aliases is applied and
/// returned. The new subscription is private until
/// [`PodcastStore::set_shared`].
///
/// Pass the channel metadata of the fetched feed
/// (`ParsedFeed::to_podcast_meta(url)`), then ingest the episodes with
/// [`PodcastStore::apply_feed`].
pub fn subscribe(
&self,
stamps: &mut dyn StampSource,
podcast: &PodcastMeta,
) -> Result<SubscribeOutcome> {
let meta = podcast.bounded();
let mut conn = self.lock();
let tx = conn.transaction_with_behavior(TransactionBehavior::Immediate)?;
let podcast_id = resolve_podcast(&tx, &meta.feed_url)?;
let current = apply::load_subscription_register(&tx, &podcast_id)?;
if current.as_ref().is_some_and(|record| record.subscribed) {
return Ok(SubscribeOutcome {
podcast_id,
already_subscribed: true,
ops: Vec::new(),
});
}
let share = apply::load_share(&tx, &podcast_id)?;
let generation = next_podcast_generation(current.as_ref(), share.as_ref());
let aliases = merge_feed_aliases(
&podcast_id,
&apply::load_aliases(&tx, &podcast_id)?,
std::slice::from_ref(&meta.feed_url),
);
let op = emit(&tx, stamps, |_| SyncOpPayload::PodcastSubscriptionSet {
podcast_id: podcast_id.clone(),
subscribed: true,
podcast: meta.clone(),
feed_aliases: aliases.clone(),
generation,
})?;
tx.commit()?;
Ok(SubscribeOutcome {
podcast_id,
already_subscribed: false,
ops: vec![op],
})
}
/// Unsubscribes (explicit user action).
///
/// Emits `podcast_subscription_set { subscribed: false }` followed by
/// `podcast_share_set { shared: false }`, each with the next podcast
/// generation. Progress, sessions and the catalog are kept, so
/// subscribing again continues where the user stopped. Does nothing when
/// the podcast is not subscribed.
pub fn unsubscribe(
&self,
stamps: &mut dyn StampSource,
podcast_id: &PodcastId,
) -> Result<Vec<PodcastLocalOp>> {
let mut conn = self.lock();
let tx = conn.transaction_with_behavior(TransactionBehavior::Immediate)?;
let Some(current) = apply::load_subscription_register(&tx, podcast_id)? else {
return Ok(Vec::new());
};
if !current.subscribed {
return Ok(Vec::new());
}
let share = apply::load_share(&tx, podcast_id)?;
let generation = next_podcast_generation(Some(&current), share.as_ref());
let mut meta = catalog_meta(&tx, podcast_id)?
.map(|meta| meta.bounded())
.unwrap_or_else(|| current.podcast.clone());
meta.feed_url = current.podcast.feed_url.clone();
let aliases = current.feed_aliases.clone();
let unsubscribe = emit(&tx, stamps, |_| SyncOpPayload::PodcastSubscriptionSet {
podcast_id: podcast_id.clone(),
subscribed: false,
podcast: meta.clone(),
feed_aliases: aliases.clone(),
generation,
})?;
let revoke_payload = share_payload(&tx, podcast_id, false)?;
let revoke = emit(&tx, stamps, |_| revoke_payload.clone())?;
tx.commit()?;
Ok(vec![unsubscribe, revoke])
}
/// Sets the decision to publish the podcast to the federation.
///
/// Sharing needs an active subscription and is a no-op when the podcast
/// is already effectively shared (`effective_share`). Revoking is a no-op
/// when no share decision says `true`. The write uses the next podcast
/// generation, so it orders after the subscription it observed.
pub fn set_shared(
&self,
stamps: &mut dyn StampSource,
podcast_id: &PodcastId,
shared: bool,
) -> Result<Vec<PodcastLocalOp>> {
let mut conn = self.lock();
let tx = conn.transaction_with_behavior(TransactionBehavior::Immediate)?;
let subscription = apply::load_subscription_register(&tx, podcast_id)?;
let share = apply::load_share(&tx, podcast_id)?;
if shared {
if !subscription
.as_ref()
.is_some_and(|record| record.subscribed)
{
return Err(PodcastStoreError::NotSubscribed(podcast_id.clone()).into());
}
if effective_share(subscription.as_ref(), share.as_ref()) {
return Ok(Vec::new());
}
} else if !share.as_ref().is_some_and(|record| record.shared) {
return Ok(Vec::new());
}
let payload = share_payload(&tx, podcast_id, shared)?;
let op = emit(&tx, stamps, |_| payload.clone())?;
tx.commit()?;
Ok(vec![op])
}
fn explicit_progress(
&self,
stamps: &mut dyn StampSource,
episode: &PodcastEpisodeRef,
session_id: Option<&str>,
carried_generation: Option<u64>,
action: ExplicitAction,
) -> Result<ProgressWrite> {
episode.validate().map_err(invalid)?;
let mut conn = self.lock();
let tx = conn.transaction_with_behavior(TransactionBehavior::Immediate)?;
let current = apply::load_progress(&tx, &episode.episode_id)?;
let max_seen = current
.as_ref()
.map(|record| record.generation)
.into_iter()
.chain(carried_generation)
.max();
let generation = ProgressRecord::explicit_generation(max_seen);
let current_position = current.as_ref().map_or(0, |record| record.position_ms);
let (position_ms, status) = match action {
ExplicitAction::Start(position) | ExplicitAction::Seek(position) => {
(position, ProgressStatus::InProgress)
}
ExplicitAction::Restart => (0, ProgressStatus::InProgress),
ExplicitAction::Complete(position) => (position, ProgressStatus::Completed),
ExplicitAction::MarkPlayed => (current_position, ProgressStatus::Completed),
ExplicitAction::MarkUnplayed => (0, ProgressStatus::NotStarted),
};
let reference = op_ref(episode);
let op = emit(&tx, stamps, |stamp| SyncOpPayload::PodcastProgressSet {
episode_id: episode.episode_id.clone(),
podcast_id: episode.podcast_id.clone(),
position_ms,
status,
generation,
session_id: session_id.unwrap_or(&stamp.op_id).to_string(),
episode: Some(reference.clone()),
})?;
tx.commit()?;
let wire = op.stamp.wire(op.payload.clone());
let record = wire.podcast_progress().expect("progress payload");
Ok(ProgressWrite {
session: ProgressSession {
episode: episode.clone(),
session_id: record.session_id.clone(),
generation,
},
record,
op,
})
}
/// Starts a listening session at `position_ms` (explicit action).
///
/// The generation is `explicit_generation(max(local, carried_generation))`;
/// pass `PodcastEpisodeRef::progress_generation` of a queue item received
/// through a handoff as `carried_generation`. Ordinary Play resumes from
/// [`PodcastStore::resume_position_ms`].
pub fn start_session(
&self,
stamps: &mut dyn StampSource,
episode: &PodcastEpisodeRef,
session_id: &str,
position_ms: i64,
carried_generation: Option<u64>,
) -> Result<ProgressWrite> {
self.explicit_progress(
stamps,
episode,
Some(session_id),
carried_generation,
ExplicitAction::Start(position_ms),
)
}
/// Periodic checkpoint (or pause/stop save) of a running session.
///
/// Keeps the session generation. When the register already holds a
/// greater generation nothing is written ([`CheckpointOutcome::Superseded`]).
pub fn checkpoint(
&self,
stamps: &mut dyn StampSource,
session: &ProgressSession,
position_ms: i64,
) -> Result<CheckpointOutcome> {
let episode = &session.episode;
let mut conn = self.lock();
let tx = conn.transaction_with_behavior(TransactionBehavior::Immediate)?;
if let Some(current) = apply::load_progress(&tx, &episode.episode_id)? {
if current.generation > session.generation {
return Ok(CheckpointOutcome::Superseded { current });
}
if current.generation == session.generation
&& current.session_id == session.session_id
&& current.status == ProgressStatus::Completed
{
return Ok(CheckpointOutcome::AlreadyCompleted { current });
}
}
let reference = op_ref(episode);
let op = emit(&tx, stamps, |_| SyncOpPayload::PodcastProgressSet {
episode_id: episode.episode_id.clone(),
podcast_id: episode.podcast_id.clone(),
position_ms,
status: ProgressStatus::InProgress,
generation: session.generation,
session_id: session.session_id.clone(),
episode: Some(reference.clone()),
})?;
tx.commit()?;
let record = op
.stamp
.wire(op.payload.clone())
.podcast_progress()
.expect("progress payload");
Ok(CheckpointOutcome::Written(Box::new(ProgressWrite {
record,
session: session.clone(),
op,
})))
}
/// User seek during a session (explicit action; the returned session
/// carries the new generation for later checkpoints).
pub fn seek(
&self,
stamps: &mut dyn StampSource,
session: &ProgressSession,
position_ms: i64,
) -> Result<ProgressWrite> {
self.explicit_progress(
stamps,
&session.episode,
Some(&session.session_id),
Some(session.generation),
ExplicitAction::Seek(position_ms),
)
}
/// The session reached the end (`reaches_completion` or end of stream):
/// writes `completed` with an explicit generation. Record the session
/// itself with [`PodcastStore::record_listen_session`].
pub fn complete_session(
&self,
stamps: &mut dyn StampSource,
session: &ProgressSession,
position_ms: i64,
) -> Result<ProgressWrite> {
self.explicit_progress(
stamps,
&session.episode,
Some(&session.session_id),
Some(session.generation),
ExplicitAction::Complete(position_ms),
)
}
/// "Start over": position 0, in progress, explicit generation.
pub fn restart(
&self,
stamps: &mut dyn StampSource,
episode: &PodcastEpisodeRef,
session_id: &str,
) -> Result<ProgressWrite> {
self.explicit_progress(
stamps,
episode,
Some(session_id),
None,
ExplicitAction::Restart,
)
}
/// "Mark played": status completed, explicit generation. Creates no
/// listening session and invents no listening time.
pub fn mark_played(
&self,
stamps: &mut dyn StampSource,
episode: &PodcastEpisodeRef,
) -> Result<ProgressWrite> {
self.explicit_progress(stamps, episode, None, None, ExplicitAction::MarkPlayed)
}
/// "Mark unplayed": status not started at position 0, explicit generation.
pub fn mark_unplayed(
&self,
stamps: &mut dyn StampSource,
episode: &PodcastEpisodeRef,
) -> Result<ProgressWrite> {
self.explicit_progress(stamps, episode, None, None, ExplicitAction::MarkUnplayed)
}
/// Records (or grows) a listening session of this device.
///
/// Sessions below the history threshold (`ListenSession::should_record`)
/// and versions that do not grow the stored session emit nothing.
/// `session.device_id` must be the local device id. The episode reference
/// defaults to the catalog's.
pub fn record_listen_session(
&self,
stamps: &mut dyn StampSource,
session: &ListenSession,
episode: Option<&PodcastEpisodeRef>,
) -> Result<Option<PodcastLocalOp>> {
if session.device_id != stamps.device_id() {
return Err(invalid(
"session device_id differs from the local device id",
));
}
session.validate().map_err(invalid)?;
if !session.should_record() {
return Ok(None);
}
let mut conn = self.lock();
let tx = conn.transaction_with_behavior(TransactionBehavior::Immediate)?;
if let Some(current) = apply::load_session(&tx, &session.session_id)? {
let merged = merge_listen_session(&current, session).map_err(invalid)?;
if merged == current {
return Ok(None);
}
}
let reference = match episode {
Some(episode) => Some(op_ref(episode)),
None => super::queries::episode_ref_in(&tx, &session.episode_id)?.map(|r| op_ref(&r)),
};
let op = emit(&tx, stamps, |_| SyncOpPayload::PodcastListenRecorded {
session: session.clone(),
episode: reference.clone(),
})?;
tx.commit()?;
Ok(Some(op))
}
/// Resolves a feed URL to a podcast id: the existing podcast when the
/// URL is a known feed URL or alias (`resolve_podcast_id`), otherwise
/// the id a first subscription through this URL would derive.
pub fn resolve_podcast_id(&self, url: &FeedUrl) -> Result<PodcastId> {
resolve_podcast(&self.lock(), url)
}
/// The episode reference for queueing or progress actions, or `None`
/// when the episode is unknown.
pub fn episode_ref(&self, episode_id: &EpisodeId) -> Result<Option<PodcastEpisodeRef>> {
super::queries::episode_ref_in(&self.lock(), episode_id)
}
}
+742
View File
@@ -0,0 +1,742 @@
//! The single code path that applies podcast records to the store.
//!
//! Remote operations, local actions and snapshot sections all end here.
//! Records come from the `SyncOpWire` materializers of
//! `music_dht::device_sync` and are merged with the functions of
//! `music_dht::podcast`, so applying is idempotent and order-independent.
use anyhow::Result;
use music_dht::device_sync::{SyncOpPayload, SyncOpWire, SyncSnapshot};
use music_dht::podcast::{
EpisodeIdentityHints, EpisodeMeta, FeedUrl, KnownEpisode, ListenSession, PodcastEpisodeRef,
PodcastError, PodcastId, ProgressRecord, ShareRecord, SubscriptionRecord, merge_feed_aliases,
merge_listen_session,
};
use rusqlite::{Connection, OptionalExtension as _, TransactionBehavior, params};
use super::{
PROGRESS_COLUMNS, PodcastStore, PodcastStoreError, SESSION_COLUMNS, episode_id_col,
feed_url_col, generation_to_sql, now_ms, podcast_meta_col, progress_from_row, session_from_row,
status_to_sql,
};
/// Result of applying one operation.
#[derive(Debug, Clone, PartialEq, Eq)]
#[non_exhaustive]
pub enum ApplyOutcome {
/// Stored state changed.
Changed,
/// The operation was already reflected (duplicate, or older than the
/// stored register).
Unchanged,
/// Not a podcast operation; nothing to do here.
Ignored,
/// The operation violates the contract and was skipped. Acknowledge it
/// anyway so the log keeps moving (`PODCASTS.md`).
Rejected(String),
}
/// Counters returned by [`PodcastStore::apply_snapshot`].
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub struct SnapshotApplyStats {
/// Records that changed stored state.
pub changed: usize,
/// Valid records already reflected locally.
pub unchanged: usize,
/// Records that failed validation and were skipped.
pub rejected: usize,
/// Whether the podcast sections were ignored for exceeding the counts.
pub ignored_over_limits: bool,
}
/// Where an episode row came from.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(crate) enum EpisodeSource {
/// Parsed from the podcast's feed on this device.
Feed,
/// Learned from a `PodcastEpisodeRef` (sync operation or queue item).
Sync,
}
impl EpisodeSource {
fn as_sql(self) -> &'static str {
match self {
Self::Feed => "feed",
Self::Sync => "sync",
}
}
}
fn invalid(error: impl ToString) -> anyhow::Error {
PodcastStoreError::Invalid(error.to_string()).into()
}
pub(crate) fn load_subscription_register(
conn: &Connection,
podcast_id: &PodcastId,
) -> Result<Option<SubscriptionRecord>> {
let row = conn
.query_row(
"SELECT subscribed, podcast_json, generation, hlc_ms, op_id
FROM podcast_subscriptions WHERE podcast_id = ?1",
[podcast_id.as_str()],
|row| {
let generation: i64 = row.get(2)?;
Ok(SubscriptionRecord {
podcast_id: podcast_id.clone(),
subscribed: row.get(0)?,
podcast: podcast_meta_col(&row.get::<_, String>(1)?)?,
feed_aliases: Vec::new(),
generation: u64::try_from(generation).unwrap_or(0),
hlc_ms: row.get(3)?,
op_id: row.get(4)?,
})
},
)
.optional()?;
let Some(mut record) = row else {
return Ok(None);
};
record.feed_aliases = load_aliases(conn, podcast_id)?;
Ok(Some(record))
}
pub(crate) fn load_aliases(conn: &Connection, podcast_id: &PodcastId) -> Result<Vec<FeedUrl>> {
let mut statement = conn.prepare_cached(
"SELECT feed_url FROM podcast_feed_aliases WHERE podcast_id = ?1 ORDER BY feed_url",
)?;
let urls = statement
.query_map([podcast_id.as_str()], |row| feed_url_col(row.get(0)?))?
.collect::<rusqlite::Result<Vec<_>>>()?;
Ok(urls)
}
pub(crate) fn load_share(conn: &Connection, podcast_id: &PodcastId) -> Result<Option<ShareRecord>> {
Ok(conn
.query_row(
"SELECT shared, generation, hlc_ms, op_id FROM podcast_shares WHERE podcast_id = ?1",
[podcast_id.as_str()],
|row| {
let generation: i64 = row.get(1)?;
Ok(ShareRecord {
podcast_id: podcast_id.clone(),
shared: row.get(0)?,
generation: u64::try_from(generation).unwrap_or(0),
hlc_ms: row.get(2)?,
op_id: row.get(3)?,
})
},
)
.optional()?)
}
pub(crate) fn load_progress(
conn: &Connection,
episode_id: &music_dht::podcast::EpisodeId,
) -> Result<Option<ProgressRecord>> {
Ok(conn
.query_row(
&format!("SELECT {PROGRESS_COLUMNS} FROM episode_progress p WHERE p.episode_id = ?1"),
[episode_id.as_str()],
|row| progress_from_row(row, 0),
)
.optional()?)
}
pub(crate) fn load_session(conn: &Connection, session_id: &str) -> Result<Option<ListenSession>> {
Ok(conn
.query_row(
&format!(
"SELECT {SESSION_COLUMNS} FROM podcast_listen_sessions s WHERE s.session_id = ?1"
),
[session_id],
|row| session_from_row(row, 0),
)
.optional()?)
}
/// Replaces the alias set of a podcast with `merged` when it differs.
fn store_aliases(conn: &Connection, podcast_id: &PodcastId, merged: &[FeedUrl]) -> Result<bool> {
let current = load_aliases(conn, podcast_id)?;
if current == merged {
return Ok(false);
}
conn.execute(
"DELETE FROM podcast_feed_aliases WHERE podcast_id = ?1",
[podcast_id.as_str()],
)?;
let mut insert = conn.prepare_cached(
"INSERT INTO podcast_feed_aliases (podcast_id, feed_url) VALUES (?1, ?2)",
)?;
for url in merged {
insert.execute(params![podcast_id.as_str(), url.as_str()])?;
}
Ok(true)
}
/// Merges feed URLs into the bounded grow-only alias set of a podcast.
///
/// Returns whether the stored set changed.
pub(crate) fn merge_aliases(
conn: &Connection,
podcast_id: &PodcastId,
urls: &[FeedUrl],
) -> Result<bool> {
let current = load_aliases(conn, podcast_id)?;
let merged = merge_feed_aliases(podcast_id, &current, urls);
store_aliases(conn, podcast_id, &merged)
}
/// Inserts the catalog row of a podcast if it does not exist yet.
fn ensure_podcast_row(
conn: &Connection,
podcast_id: &PodcastId,
feed_url: &FeedUrl,
title: &str,
now: i64,
) -> Result<()> {
conn.execute(
"INSERT OR IGNORE INTO podcasts
(podcast_id, feed_url, title, meta_from_feed, created_at_ms, updated_at_ms)
VALUES (?1, ?2, ?3, 0, ?4, ?4)",
params![podcast_id.as_str(), feed_url.as_str(), title, now],
)?;
Ok(())
}
/// Applies a subscription register record (operation or snapshot).
///
/// The register is last-writer-wins ([`SubscriptionRecord::supersedes`]);
/// the alias set is a bounded grow-only set merged independently, so the
/// stored aliases are the bounded union of every URL seen (equal to what
/// `merge_subscription` computes). The catalog row is created from the
/// register metadata and follows the winning register until this device has
/// refreshed the feed itself.
pub(crate) fn apply_subscription_record(
conn: &Connection,
record: &SubscriptionRecord,
now: i64,
) -> Result<bool> {
generation_to_sql(record.generation)?;
let podcast_id = &record.podcast_id;
let current = load_subscription_register(conn, podcast_id)?;
let wins = current
.as_ref()
.is_none_or(|current| record.supersedes(current));
let mut changed = false;
if wins {
conn.execute(
"INSERT INTO podcast_subscriptions
(podcast_id, subscribed, podcast_json, generation, hlc_ms, op_id)
VALUES (?1, ?2, ?3, ?4, ?5, ?6)
ON CONFLICT(podcast_id) DO UPDATE SET
subscribed = excluded.subscribed, podcast_json = excluded.podcast_json,
generation = excluded.generation, hlc_ms = excluded.hlc_ms,
op_id = excluded.op_id",
params![
podcast_id.as_str(),
record.subscribed,
serde_json::to_string(&record.podcast)?,
generation_to_sql(record.generation)?,
record.hlc_ms,
record.op_id,
],
)?;
changed = true;
}
let mut urls = record.feed_aliases.clone();
urls.push(record.podcast.feed_url.clone());
changed |= merge_aliases(conn, podcast_id, &urls)?;
ensure_podcast_row(
conn,
podcast_id,
&record.podcast.feed_url,
&record.podcast.title,
now,
)?;
if wins {
let meta = &record.podcast;
conn.execute(
"UPDATE podcasts
SET feed_url = ?2, title = ?3, author = ?4,
description = COALESCE(?5, description), link = ?6, image_url = ?7,
updated_at_ms = ?8
WHERE podcast_id = ?1 AND meta_from_feed = 0",
params![
podcast_id.as_str(),
meta.feed_url.as_str(),
meta.title,
meta.author,
meta.description,
meta.link,
meta.image_url,
now
],
)?;
}
Ok(changed)
}
/// Applies a share register record.
pub(crate) fn apply_share_record(conn: &Connection, record: &ShareRecord) -> Result<bool> {
let generation = generation_to_sql(record.generation)?;
let current = load_share(conn, &record.podcast_id)?;
if current
.as_ref()
.is_some_and(|current| !record.supersedes(current))
{
return Ok(false);
}
conn.execute(
"INSERT INTO podcast_shares (podcast_id, shared, generation, hlc_ms, op_id)
VALUES (?1, ?2, ?3, ?4, ?5)
ON CONFLICT(podcast_id) DO UPDATE SET
shared = excluded.shared, generation = excluded.generation,
hlc_ms = excluded.hlc_ms, op_id = excluded.op_id",
params![
record.podcast_id.as_str(),
record.shared,
generation,
record.hlc_ms,
record.op_id
],
)?;
Ok(true)
}
/// Applies a progress register record. Never touches the catalog.
pub(crate) fn apply_progress_record(conn: &Connection, record: &ProgressRecord) -> Result<bool> {
let generation = generation_to_sql(record.generation)?;
let current = load_progress(conn, &record.episode_id)?;
if current
.as_ref()
.is_some_and(|current| !record.supersedes(current))
{
return Ok(false);
}
conn.execute(
"INSERT INTO episode_progress
(episode_id, podcast_id, position_ms, status, generation, session_id, hlc_ms,
device_id, op_id)
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9)
ON CONFLICT(episode_id) DO UPDATE SET
podcast_id = excluded.podcast_id, position_ms = excluded.position_ms,
status = excluded.status, generation = excluded.generation,
session_id = excluded.session_id, hlc_ms = excluded.hlc_ms,
device_id = excluded.device_id, op_id = excluded.op_id",
params![
record.episode_id.as_str(),
record.podcast_id.as_str(),
record.position_ms,
status_to_sql(record.status),
generation,
record.session_id,
record.hlc_ms,
record.device_id,
record.op_id,
],
)?;
Ok(true)
}
/// Adds or grows a listening session (grow-only set keyed by `session_id`).
pub(crate) fn apply_listen_session(conn: &Connection, session: &ListenSession) -> Result<bool> {
let current = load_session(conn, &session.session_id)?;
let merged = match &current {
Some(current) => merge_listen_session(current, session).map_err(invalid)?,
None => session.clone(),
};
if current.as_ref() == Some(&merged) {
return Ok(false);
}
conn.execute(
"INSERT INTO podcast_listen_sessions
(session_id, episode_id, podcast_id, device_id, started_at_ms, ended_at_ms,
start_position_ms, end_position_ms, listened_ms, completed)
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10)
ON CONFLICT(session_id) DO UPDATE SET
started_at_ms = excluded.started_at_ms, ended_at_ms = excluded.ended_at_ms,
start_position_ms = excluded.start_position_ms,
end_position_ms = excluded.end_position_ms,
listened_ms = excluded.listened_ms, completed = excluded.completed",
params![
merged.session_id,
merged.episode_id.as_str(),
merged.podcast_id.as_str(),
merged.device_id,
merged.started_at_ms,
merged.ended_at_ms,
merged.start_position_ms,
merged.end_position_ms,
merged.listened_ms,
merged.completed,
],
)?;
Ok(true)
}
/// Inserts an episode row.
pub(crate) fn insert_episode(
conn: &Connection,
meta: &EpisodeMeta,
source: EpisodeSource,
in_feed: bool,
legacy_guid: Option<&str>,
first_seen_at_ms: i64,
now: i64,
) -> Result<()> {
conn.execute(
"INSERT INTO episodes
(episode_id, podcast_id, guid, title, description, season, episode_number,
episode_type, published_at_ms, duration_ms, audio_url, mime, byte_len,
image_url, link, source, in_feed, legacy_guid, first_seen_at_ms, updated_at_ms)
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11, ?12, ?13, ?14, ?15, ?16, ?17,
?18, ?19, ?20)",
params![
meta.episode_id.as_str(),
meta.podcast_id.as_str(),
meta.guid,
meta.title,
meta.description,
meta.season,
meta.episode_number,
episode_type_to_sql(meta.episode_type),
meta.published_at_ms,
meta.duration_ms,
meta.audio_url,
meta.mime,
meta.byte_len.and_then(|len| i64::try_from(len).ok()),
meta.image_url,
meta.link,
source.as_sql(),
in_feed,
legacy_guid,
first_seen_at_ms,
now,
],
)?;
Ok(())
}
pub(crate) fn episode_type_to_sql(kind: music_dht::podcast::EpisodeType) -> &'static str {
match kind {
music_dht::podcast::EpisodeType::Full => "full",
music_dht::podcast::EpisodeType::Trailer => "trailer",
music_dht::podcast::EpisodeType::Bonus => "bonus",
}
}
/// Records an episode learned through sync when the catalog does not know it.
///
/// Feed metadata always wins: an existing row is left untouched. The
/// podcast catalog row is created from the reference when missing (no
/// register is written).
pub(crate) fn ensure_episode_from_ref(
conn: &Connection,
episode: &PodcastEpisodeRef,
) -> Result<()> {
let exists = conn
.query_row(
"SELECT 1 FROM episodes WHERE episode_id = ?1",
[episode.episode_id.as_str()],
|_| Ok(()),
)
.optional()?
.is_some();
if exists {
return Ok(());
}
let now = now_ms();
ensure_podcast_row(
conn,
&episode.podcast_id,
&episode.feed_url,
&episode.podcast_title,
now,
)?;
let meta = EpisodeMeta {
episode_id: episode.episode_id.clone(),
podcast_id: episode.podcast_id.clone(),
guid: None,
title: episode.episode_title.clone(),
description: None,
season: episode.season,
episode_number: episode.episode_number,
episode_type: Default::default(),
published_at_ms: episode.published_at_ms,
duration_ms: episode.duration_ms,
audio_url: episode.audio_url.clone(),
mime: episode.mime.clone(),
byte_len: None,
image_url: episode.image_url.clone(),
link: None,
};
insert_episode(conn, &meta, EpisodeSource::Sync, false, None, now, now)?;
refresh_seasons(conn, &episode.podcast_id)?;
Ok(())
}
/// Recomputes the derived season rows of a podcast from its episodes.
pub(crate) fn refresh_seasons(conn: &Connection, podcast_id: &PodcastId) -> Result<()> {
conn.execute(
"DELETE FROM podcast_seasons WHERE podcast_id = ?1",
[podcast_id.as_str()],
)?;
conn.execute(
"INSERT INTO podcast_seasons
(podcast_id, season, episode_count, first_published_at_ms, last_published_at_ms)
SELECT podcast_id, season, COUNT(*), MIN(published_at_ms), MAX(published_at_ms)
FROM episodes WHERE podcast_id = ?1 GROUP BY season",
[podcast_id.as_str()],
)?;
Ok(())
}
/// Known episodes of a podcast with the identity hints they were last seen with.
pub(crate) fn known_episodes(
conn: &Connection,
podcast_id: &PodcastId,
) -> Result<Vec<KnownEpisode>> {
let mut statement = conn.prepare_cached(
"SELECT episode_id, guid, audio_url, link, published_at_ms, title
FROM episodes WHERE podcast_id = ?1 ORDER BY episode_id",
)?;
let known = statement
.query_map([podcast_id.as_str()], |row| {
Ok(KnownEpisode {
episode_id: episode_id_col(row.get(0)?)?,
hints: EpisodeIdentityHints {
guid: row.get(1)?,
enclosure_url: Some(row.get(2)?),
link: row.get(3)?,
published_at_ms: row.get(4)?,
title: Some(row.get(5)?),
},
})
})?
.collect::<rusqlite::Result<Vec<_>>>()?;
Ok(known)
}
/// Applies one podcast operation inside the caller's transaction.
pub(crate) fn apply_op(conn: &Connection, op: &SyncOpWire) -> Result<ApplyOutcome> {
if !op.payload.is_podcast() {
return Ok(ApplyOutcome::Ignored);
}
if let Err(error) = op.validate_podcast() {
return Ok(ApplyOutcome::Rejected(error.to_string()));
}
let result = match &op.payload {
SyncOpPayload::PodcastSubscriptionSet { .. } => {
let record = op.podcast_subscription().expect("subscription payload");
apply_subscription_record(conn, &record, now_ms())
}
SyncOpPayload::PodcastShareSet { .. } => {
apply_share_record(conn, &op.podcast_share().expect("share payload"))
}
SyncOpPayload::PodcastFeedAliasesAdded { .. } => {
let (podcast_id, urls) = op.podcast_feed_aliases().expect("alias payload");
merge_aliases(conn, podcast_id, urls)
}
SyncOpPayload::PodcastProgressSet { episode, .. } => {
let record = op.podcast_progress().expect("progress payload");
if let Err(error) = generation_to_sql(record.generation) {
return Ok(ApplyOutcome::Rejected(error.to_string()));
}
if let Some(episode) = episode {
ensure_episode_from_ref(conn, episode)?;
}
apply_progress_record(conn, &record)
}
SyncOpPayload::PodcastListenRecorded { session, episode } => {
if let Some(current) = load_session(conn, &session.session_id)?
&& let Err(error) = merge_listen_session(&current, session)
{
return Ok(ApplyOutcome::Rejected(error.to_string()));
}
if let Some(episode) = episode {
ensure_episode_from_ref(conn, episode)?;
}
apply_listen_session(conn, session)
}
_ => return Ok(ApplyOutcome::Ignored),
};
match result {
Ok(true) => Ok(ApplyOutcome::Changed),
Ok(false) => Ok(ApplyOutcome::Unchanged),
Err(error) => match error.downcast_ref::<PodcastStoreError>() {
Some(PodcastStoreError::Invalid(reason)) => Ok(ApplyOutcome::Rejected(reason.clone())),
_ => Err(error),
},
}
}
fn record_outcome(stats: &mut SnapshotApplyStats, result: Result<bool>) -> Result<()> {
match result {
Ok(true) => stats.changed += 1,
Ok(false) => stats.unchanged += 1,
Err(error) => match error.downcast_ref::<PodcastStoreError>() {
Some(PodcastStoreError::Invalid(_)) => stats.rejected += 1,
_ => return Err(error),
},
}
Ok(())
}
fn check(result: Result<(), PodcastError>) -> Result<()> {
result.map_err(invalid)
}
impl PodcastStore {
/// Applies a device-sync operation received from a trusted device.
///
/// Non-podcast operations return [`ApplyOutcome::Ignored`]. Podcast
/// operations are validated (`SyncOpWire::validate_podcast`) and merged
/// with the contract functions in one transaction; applying the same
/// operations in any order, with duplicates, gives the same state.
pub fn apply_remote_op(&self, op: &SyncOpWire) -> Result<ApplyOutcome> {
if !op.payload.is_podcast() {
return Ok(ApplyOutcome::Ignored);
}
let mut conn = self.lock();
let tx = conn.transaction_with_behavior(TransactionBehavior::Immediate)?;
let outcome = apply_op(&tx, op)?;
tx.commit()?;
Ok(outcome)
}
/// Merges the podcast sections of a sync snapshot.
///
/// Sections exceeding the contract counts are ignored
/// (`SyncSnapshot::podcast_state_within_limits`); every record is
/// validated and merged with the same functions as operations, so a
/// snapshot never replaces newer local state. One transaction.
pub fn apply_snapshot(&self, snapshot: &SyncSnapshot) -> Result<SnapshotApplyStats> {
let mut stats = SnapshotApplyStats::default();
if !snapshot.podcast_state_within_limits() {
stats.ignored_over_limits = true;
return Ok(stats);
}
let mut conn = self.lock();
let tx = conn.transaction_with_behavior(TransactionBehavior::Immediate)?;
let now = now_ms();
for record in &snapshot.podcast_subscriptions {
let mut record = record.clone();
record.normalize();
let result = check(record.validate())
.and_then(|()| apply_subscription_record(&tx, &record, now));
record_outcome(&mut stats, result)?;
}
for record in &snapshot.podcast_shares {
let result = check(record.validate()).and_then(|()| apply_share_record(&tx, record));
record_outcome(&mut stats, result)?;
}
for record in &snapshot.podcast_progress {
let result = check(record.validate()).and_then(|()| apply_progress_record(&tx, record));
record_outcome(&mut stats, result)?;
}
for session in &snapshot.podcast_listen_sessions {
let result =
check(session.validate()).and_then(|()| apply_listen_session(&tx, session));
record_outcome(&mut stats, result)?;
}
tx.commit()?;
Ok(stats)
}
/// Fills the podcast sections of `snapshot` with this store's registers
/// and sessions, bounded by `SyncSnapshot::bound_podcast_state_to`.
///
/// `budget_bytes` is usually `MAX_PODCAST_SNAPSHOT_BYTES` or
/// `device_sync::podcast_snapshot_budget(other_bytes)`. Records are read
/// newest first and capped by the per-section counts before the byte
/// budget is applied. Local fetch state and the catalog are never part
/// of the snapshot. Send it only to v4 peers (`SyncSnapshot::for_peer`).
pub fn fill_snapshot(&self, snapshot: &mut SyncSnapshot, budget_bytes: usize) -> Result<()> {
use music_dht::podcast::{
MAX_SNAPSHOT_LISTEN_SESSIONS, MAX_SNAPSHOT_PROGRESS, MAX_SNAPSHOT_SHARES,
MAX_SNAPSHOT_SUBSCRIPTIONS,
};
let conn = self.lock();
let ids: Vec<PodcastId> = {
let mut statement = conn.prepare(
"SELECT podcast_id FROM podcast_subscriptions
ORDER BY generation DESC, hlc_ms DESC, op_id DESC, podcast_id DESC
LIMIT ?1",
)?;
statement
.query_map([MAX_SNAPSHOT_SUBSCRIPTIONS as i64], |row| {
super::podcast_id_col(row.get(0)?)
})?
.collect::<rusqlite::Result<Vec<_>>>()?
};
let mut subscriptions = Vec::with_capacity(ids.len());
for id in &ids {
if let Some(record) = load_subscription_register(&conn, id)? {
subscriptions.push(record);
}
}
let shares = {
let mut statement = conn.prepare(
"SELECT podcast_id, shared, generation, hlc_ms, op_id FROM podcast_shares
ORDER BY generation DESC, hlc_ms DESC, op_id DESC, podcast_id DESC
LIMIT ?1",
)?;
statement
.query_map([MAX_SNAPSHOT_SHARES as i64], |row| {
let generation: i64 = row.get(2)?;
Ok(ShareRecord {
podcast_id: super::podcast_id_col(row.get(0)?)?,
shared: row.get(1)?,
generation: u64::try_from(generation).unwrap_or(0),
hlc_ms: row.get(3)?,
op_id: row.get(4)?,
})
})?
.collect::<rusqlite::Result<Vec<_>>>()?
};
let progress = {
let mut statement = conn.prepare(&format!(
"SELECT {PROGRESS_COLUMNS} FROM episode_progress p
ORDER BY p.generation DESC, p.hlc_ms DESC, p.op_id DESC, p.episode_id DESC
LIMIT ?1"
))?;
statement
.query_map([MAX_SNAPSHOT_PROGRESS as i64], |row| {
progress_from_row(row, 0)
})?
.collect::<rusqlite::Result<Vec<_>>>()?
};
let sessions = {
let mut statement = conn.prepare(&format!(
"SELECT {SESSION_COLUMNS} FROM podcast_listen_sessions s
ORDER BY s.ended_at_ms DESC, s.session_id DESC
LIMIT ?1"
))?;
statement
.query_map([MAX_SNAPSHOT_LISTEN_SESSIONS as i64], |row| {
session_from_row(row, 0)
})?
.collect::<rusqlite::Result<Vec<_>>>()?
};
drop(conn);
snapshot.podcast_subscriptions = subscriptions;
snapshot.podcast_shares = shares;
snapshot.podcast_progress = progress;
snapshot.podcast_listen_sessions = sessions;
snapshot.bound_podcast_state_to(budget_bytes);
Ok(())
}
/// A snapshot holding only this store's podcast sections, bounded with
/// the default budget (`MAX_PODCAST_SNAPSHOT_BYTES`).
pub fn export_snapshot(&self) -> Result<SyncSnapshot> {
let mut snapshot = SyncSnapshot::default();
self.fill_snapshot(
&mut snapshot,
music_dht::podcast::MAX_PODCAST_SNAPSHOT_BYTES,
)?;
Ok(snapshot)
}
}
+451
View File
@@ -0,0 +1,451 @@
//! SQLite reference store for podcasts (`podcasts.sqlite3`).
//!
//! [`PodcastStore`] persists the podcast contract of `music_dht::podcast`
//! for SQLite clients: the local episode catalog refreshed from RSS, the
//! synchronized personal registers (subscription, share, progress), the
//! grow-only listening sessions and the local-only fetch state. It owns no
//! networking, HTTP, scheduling or audio: clients fetch feeds, parse them
//! with `music_dht::podcast::feed`, append sync operations to their own log
//! and exchange snapshots. The normative rules live in the workspace
//! `PODCASTS.md` (section "SQLite reference store" describes this schema).
//!
//! # Files and migrations
//!
//! The store lives in its own file, conventionally [`PODCAST_DB_FILE`] next
//! to the library database; the caller passes the path. Connections use
//! WAL, foreign keys and a busy timeout. `PRAGMA user_version` drives the
//! migrations, each applied in its own `BEGIN IMMEDIATE` transaction that
//! re-checks the version, so concurrent openers and interrupted upgrades are
//! safe:
//!
//! | `user_version` | Schema |
//! | --- | --- |
//! | 0 | empty file |
//! | 1 | prototype schema of the original podcast PR (`podcasts`, `podcast_episodes`) |
//! | 2 | contract model; the prototype tables are renamed `legacy_*` and imported |
//!
//! A database written by a newer Furumi ([`PODCAST_SCHEMA_VERSION`] lower
//! than its `user_version`) or a file that is not a podcast database is
//! refused before anything is written ([`PodcastStoreError`]).
//!
//! # Personal state and sync
//!
//! Local actions ([`PodcastStore::subscribe`], [`PodcastStore::seek`], ...)
//! draw operation stamps from a [`StampSource`], build the device-sync
//! operation, apply it through the same code path as
//! [`PodcastStore::apply_remote_op`] (the `SyncOpWire` materializers of
//! `music_dht::device_sync` and the merge functions of `music_dht::podcast`)
//! and return the [`PodcastLocalOp`]s. The caller appends them to its sync
//! log **with exactly these stamps**; applying them again is a no-op.
//! Applying operations and snapshot sections is idempotent and independent
//! of delivery order.
//!
//! RSS refreshes ([`PodcastStore::apply_feed`]) only touch the catalog and
//! feed aliases: they never write a register, progress or a session.
//!
//! ```
//! use furumi_library::podcasts::{ApplyOutcome, OpStamp, PodcastStore, StampSource};
//! use music_dht::podcast::{FeedUrl, PodcastMeta};
//!
//! /// Stand-in for the client's sync log, which reserves seq and hlc.
//! struct Log {
//! device: String,
//! seq: i64,
//! }
//!
//! impl StampSource for Log {
//! fn device_id(&self) -> &str {
//! &self.device
//! }
//! fn next_stamp(&mut self) -> anyhow::Result<OpStamp> {
//! self.seq += 1;
//! let op_id = format!("{}:{}", self.device, self.seq);
//! Ok(OpStamp::new(op_id, self.device.clone(), 1_700_000_000_000 + self.seq))
//! }
//! }
//!
//! let store = PodcastStore::open_in_memory()?;
//! let mut log = Log { device: "device-a".into(), seq: 0 };
//! let meta = PodcastMeta {
//! feed_url: FeedUrl::parse("https://example.test/feed.xml")?,
//! title: "Show".into(),
//! author: None,
//! description: None,
//! link: None,
//! image_url: None,
//! };
//! let outcome = store.subscribe(&mut log, &meta)?;
//! assert_eq!(outcome.ops.len(), 1);
//! for op in outcome.ops {
//! // Append to the sync log with this stamp; re-applying is a no-op.
//! let op = op.into_wire(log.seq);
//! assert_eq!(store.apply_remote_op(&op)?, ApplyOutcome::Unchanged);
//! }
//! # Ok::<(), anyhow::Error>(())
//! ```
//!
//! # Thread safety
//!
//! All methods take `&self` and serialize on an internal mutex around one
//! synchronous `rusqlite` connection, so an `Arc<PodcastStore>` can be shared
//! between threads. Calls block on SQLite I/O: run them on a blocking pool
//! (for example `tokio::task::spawn_blocking`), never on an async reactor
//! thread. Several processes may open the same file; SQLite serializes the
//! writers.
mod actions;
mod apply;
mod queries;
mod refresh;
mod schema;
#[cfg(test)]
mod tests;
use std::fmt;
use std::path::{Path, PathBuf};
use std::sync::{Mutex, MutexGuard, PoisonError};
use std::time::Duration;
use anyhow::{Context as _, Result};
use music_dht::device_sync::{SyncOpPayload, SyncOpWire};
use music_dht::podcast::{
EpisodeId, FeedUrl, ListenSession, PodcastError, PodcastId, PodcastMeta, ProgressRecord,
ProgressStatus,
};
use rusqlite::Connection;
pub use actions::{CheckpointOutcome, ProgressSession, ProgressWrite, SubscribeOutcome};
pub use apply::{ApplyOutcome, SnapshotApplyStats};
pub use queries::{
EpisodeListItem, EpisodeQuery, EpisodeSort, LegacyQuarantineEntry, PlayedFilter,
PodcastListFilter, PodcastListenEntry, PodcastSummary, SeasonFilter, SeasonSummary,
};
pub use refresh::{FeedRefresh, FetchState, RefreshOutcome};
pub use schema::{LEGACY_PROTOTYPE_SCHEMA_VERSION, PODCAST_SCHEMA_VERSION};
/// Conventional file name of the podcast store, next to `library.db`.
pub const PODCAST_DB_FILE: &str = "podcasts.sqlite3";
/// `op_id` prefix of registers imported from the prototype database.
///
/// Imported subscriptions use generation 0, `hlc_ms` = the prototype
/// `added_at` and `op_id` = this prefix followed by the podcast id, so every
/// write made through the contract orders after them.
pub const LEGACY_IMPORT_OP_PREFIX: &str = "legacy-import:";
/// Typed errors of [`PodcastStore`]; other failures are SQLite or I/O errors.
///
/// They are returned inside `anyhow::Error`; use `downcast_ref` to match.
#[derive(Debug, Clone, PartialEq, Eq)]
#[non_exhaustive]
pub enum PodcastStoreError {
/// The database was written by a newer Furumi. It was not modified.
NewerSchema {
/// `user_version` found in the file.
found: i64,
/// Newest version this build understands.
supported: i64,
},
/// The file is not a podcast database (or has an unexpected shape). It
/// was not modified.
UnrecognizedDatabase(String),
/// The podcast is unknown to this store.
UnknownPodcast(PodcastId),
/// The action needs an active subscription.
NotSubscribed(PodcastId),
/// Input violates the podcast contract.
Invalid(String),
}
impl fmt::Display for PodcastStoreError {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
Self::NewerSchema { found, supported } => write!(
f,
"podcast database schema v{found} is newer than this Furumi supports (v{supported}); \
update Furumi, the database was left unchanged"
),
Self::UnrecognizedDatabase(reason) => {
write!(f, "not a Furumi podcast database: {reason}")
}
Self::UnknownPodcast(id) => write!(f, "unknown podcast {id}"),
Self::NotSubscribed(id) => write!(f, "not subscribed to podcast {id}"),
Self::Invalid(reason) => write!(f, "invalid podcast data: {reason}"),
}
}
}
impl std::error::Error for PodcastStoreError {}
impl From<PodcastError> for PodcastStoreError {
fn from(error: PodcastError) -> Self {
Self::Invalid(error.to_string())
}
}
/// Stamp of a device-sync operation: the fields the `SyncOpWire`
/// materializers turn into register stamps.
///
/// `seq` is not part of it: the client's sync log assigns it when it appends
/// the operation ([`PodcastLocalOp::into_wire`]).
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct OpStamp {
/// Unique operation id, usually `<origin_device_id>:<seq>`.
pub op_id: String,
/// Origin device id (the progress register `device_id`).
pub origin_device_id: String,
/// Operation timestamp (register `hlc_ms`).
pub hlc_ms: i64,
}
impl OpStamp {
/// Builds a stamp.
pub fn new(op_id: impl Into<String>, origin_device_id: impl Into<String>, hlc_ms: i64) -> Self {
Self {
op_id: op_id.into(),
origin_device_id: origin_device_id.into(),
hlc_ms,
}
}
/// The stamp of an existing operation.
pub fn of(op: &SyncOpWire) -> Self {
Self::new(op.op_id.clone(), op.origin_device_id.clone(), op.hlc_ms)
}
pub(crate) fn wire(&self, payload: SyncOpPayload) -> SyncOpWire {
SyncOpWire {
op_id: self.op_id.clone(),
origin_device_id: self.origin_device_id.clone(),
seq: 0,
hlc_ms: self.hlc_ms,
payload,
}
}
}
/// Source of operation stamps for local actions, usually the client's sync
/// log (which reserves the next sequence number and timestamp).
///
/// A stamp is drawn only when an action actually emits an operation.
pub trait StampSource {
/// The local device id; every stamp must use it as origin.
fn device_id(&self) -> &str;
/// Reserves the stamp of the next local operation.
fn next_stamp(&mut self) -> Result<OpStamp>;
}
/// A local operation produced by a [`PodcastStore`] action.
///
/// The store has already applied it. Append it to the sync log with exactly
/// this stamp: a different `op_id` or `hlc_ms` would make other devices
/// materialize a different register than this one.
#[derive(Debug, Clone, PartialEq)]
pub struct PodcastLocalOp {
/// Stamp drawn from the [`StampSource`].
pub stamp: OpStamp,
/// Operation payload (one of the `podcast_*` kinds).
pub payload: SyncOpPayload,
}
impl PodcastLocalOp {
/// The operation with the sequence number assigned by the sync log.
pub fn into_wire(self, seq: i64) -> SyncOpWire {
let mut op = self.stamp.wire(self.payload);
op.seq = seq;
op
}
}
/// SQLite store of podcast catalog, personal state and fetch state.
///
/// See the [module documentation](self) for the schema, the sync model and
/// thread safety.
pub struct PodcastStore {
conn: Mutex<Connection>,
path: Option<PathBuf>,
}
impl fmt::Debug for PodcastStore {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.debug_struct("PodcastStore")
.field("path", &self.path)
.finish_non_exhaustive()
}
}
impl PodcastStore {
/// Opens (creating and migrating if needed) the podcast store at `path`.
///
/// A prototype database (`user_version` 1) is imported; see the module
/// documentation. Fails without modifying the file when it was written
/// by a newer Furumi or is not a podcast database.
pub fn open(path: &Path) -> Result<Self> {
if let Some(parent) = path.parent()
&& !parent.as_os_str().is_empty()
{
std::fs::create_dir_all(parent)
.with_context(|| format!("creating {}", parent.display()))?;
}
let conn = Connection::open(path)
.with_context(|| format!("opening podcast database {}", path.display()))?;
Self::init(conn, Some(path.to_path_buf()))
}
/// Opens a private in-memory store (tests and previews).
pub fn open_in_memory() -> Result<Self> {
Self::init(Connection::open_in_memory()?, None)
}
fn init(mut conn: Connection, path: Option<PathBuf>) -> Result<Self> {
conn.busy_timeout(Duration::from_secs(5))?;
// Refuse newer or foreign files before anything (WAL included) is written.
schema::check_openable(&conn)?;
if path.is_some() {
let _: String = conn.query_row("PRAGMA journal_mode=WAL", [], |row| row.get(0))?;
}
conn.pragma_update(None, "foreign_keys", "ON")?;
register_norm_function(&conn)?;
schema::migrate(&mut conn)?;
Ok(Self {
conn: Mutex::new(conn),
path,
})
}
/// Path of the database file (`None` for in-memory stores).
pub fn path(&self) -> Option<&Path> {
self.path.as_deref()
}
/// Current `PRAGMA user_version` (always [`PODCAST_SCHEMA_VERSION`]
/// after a successful open).
pub fn schema_version(&self) -> Result<i64> {
Ok(self
.lock()
.query_row("PRAGMA user_version", [], |row| row.get(0))?)
}
pub(crate) fn lock(&self) -> MutexGuard<'_, Connection> {
self.conn.lock().unwrap_or_else(PoisonError::into_inner)
}
}
fn register_norm_function(conn: &Connection) -> Result<()> {
use rusqlite::functions::FunctionFlags;
conn.create_scalar_function(
"norm",
1,
FunctionFlags::SQLITE_UTF8 | FunctionFlags::SQLITE_DETERMINISTIC,
|ctx| {
let value: Option<String> = ctx.get(0)?;
Ok(music_dht::normalize_name(
value.as_deref().unwrap_or_default(),
))
},
)?;
Ok(())
}
pub(crate) fn now_ms() -> i64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|duration| i64::try_from(duration.as_millis()).unwrap_or(i64::MAX))
.unwrap_or(0)
}
/// Converts a contract generation to the SQLite column value.
///
/// Generations above `i64::MAX` cannot be stored; records carrying one are
/// rejected (no conforming client produces them).
pub(crate) fn generation_to_sql(generation: u64) -> Result<i64, PodcastStoreError> {
i64::try_from(generation)
.map_err(|_| PodcastStoreError::Invalid("generation exceeds i64::MAX".to_string()))
}
pub(crate) fn status_to_sql(status: ProgressStatus) -> &'static str {
match status {
ProgressStatus::NotStarted => "not_started",
ProgressStatus::InProgress => "in_progress",
ProgressStatus::Completed => "completed",
}
}
pub(crate) fn status_from_sql(value: &str) -> ProgressStatus {
match value {
"in_progress" => ProgressStatus::InProgress,
"completed" => ProgressStatus::Completed,
_ => ProgressStatus::NotStarted,
}
}
/// Parses a stored id; stored ids are validated on write.
pub(crate) fn podcast_id_col(value: String) -> rusqlite::Result<PodcastId> {
PodcastId::try_from(value).map_err(|error| conversion_error(error.to_string()))
}
pub(crate) fn episode_id_col(value: String) -> rusqlite::Result<EpisodeId> {
EpisodeId::try_from(value).map_err(|error| conversion_error(error.to_string()))
}
pub(crate) fn feed_url_col(value: String) -> rusqlite::Result<FeedUrl> {
FeedUrl::try_from(value).map_err(|error| conversion_error(error.to_string()))
}
pub(crate) fn podcast_meta_col(value: &str) -> rusqlite::Result<PodcastMeta> {
serde_json::from_str(value).map_err(|error| conversion_error(error.to_string()))
}
pub(crate) fn conversion_error(message: String) -> rusqlite::Error {
rusqlite::Error::FromSqlConversionFailure(
0,
rusqlite::types::Type::Text,
Box::new(PodcastStoreError::Invalid(message)),
)
}
pub(crate) fn progress_from_row(
row: &rusqlite::Row<'_>,
at: usize,
) -> rusqlite::Result<ProgressRecord> {
let generation: i64 = row.get(at + 4)?;
Ok(ProgressRecord {
episode_id: episode_id_col(row.get(at)?)?,
podcast_id: podcast_id_col(row.get(at + 1)?)?,
position_ms: row.get(at + 2)?,
status: status_from_sql(&row.get::<_, String>(at + 3)?),
generation: u64::try_from(generation).unwrap_or(0),
session_id: row.get(at + 5)?,
hlc_ms: row.get(at + 6)?,
device_id: row.get(at + 7)?,
op_id: row.get(at + 8)?,
})
}
/// Column list matching [`progress_from_row`], for table alias `p`.
pub(crate) const PROGRESS_COLUMNS: &str = "p.episode_id, p.podcast_id, p.position_ms, p.status, \
p.generation, p.session_id, p.hlc_ms, p.device_id, p.op_id";
pub(crate) fn session_from_row(
row: &rusqlite::Row<'_>,
at: usize,
) -> rusqlite::Result<ListenSession> {
Ok(ListenSession {
session_id: row.get(at)?,
episode_id: episode_id_col(row.get(at + 1)?)?,
podcast_id: podcast_id_col(row.get(at + 2)?)?,
device_id: row.get(at + 3)?,
started_at_ms: row.get(at + 4)?,
ended_at_ms: row.get(at + 5)?,
start_position_ms: row.get(at + 6)?,
end_position_ms: row.get(at + 7)?,
listened_ms: row.get(at + 8)?,
completed: row.get(at + 9)?,
})
}
/// Column list matching [`session_from_row`], for table alias `s`.
pub(crate) const SESSION_COLUMNS: &str = "s.session_id, s.episode_id, s.podcast_id, s.device_id, \
s.started_at_ms, s.ended_at_ms, s.start_position_ms, s.end_position_ms, s.listened_ms, \
s.completed";
@@ -0,0 +1,605 @@
//! Read-only queries for clients' podcast UI.
use anyhow::Result;
use music_dht::podcast::{
EpisodeId, EpisodeMeta, EpisodeType, FeedUrl, ListenSession, PodcastEpisodeRef, PodcastId,
PodcastMeta, ProgressRecord, ProgressStatus, ShareRecord, SubscriptionRecord, effective_share,
};
use rusqlite::types::Value;
use rusqlite::{Connection, OptionalExtension as _, params, params_from_iter};
use super::apply;
use super::{
PROGRESS_COLUMNS, PodcastStore, SESSION_COLUMNS, episode_id_col, feed_url_col, podcast_id_col,
progress_from_row, session_from_row,
};
use crate::TrackItem;
/// Which podcasts [`PodcastStore::list_podcasts`] returns.
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub enum PodcastListFilter {
/// Podcasts with an active subscription.
#[default]
Subscribed,
/// Every podcast in the catalog, including unsubscribed ones and
/// podcasts only known from synced progress or queue items.
All,
}
/// A podcast with its personal state and episode counters.
#[derive(Debug, Clone, PartialEq)]
pub struct PodcastSummary {
/// Podcast id.
pub podcast_id: PodcastId,
/// Local channel metadata; `feed_url` is the URL this device fetches.
pub meta: PodcastMeta,
/// Whether the subscription register says subscribed.
pub subscribed: bool,
/// Whether the podcast must currently be published (`effective_share`).
pub shared: bool,
/// Episodes in the catalog.
pub episode_count: u64,
/// Episodes not completed (no progress, not started or in progress).
pub unplayed_count: u64,
/// Episodes in progress.
pub in_progress_count: u64,
/// Newest publication time.
pub latest_published_at_ms: Option<i64>,
/// Numbered seasons (episodes without a season are not counted).
pub season_count: u64,
}
/// One season group of a podcast. `season == None` groups the episodes
/// without `itunes:season`.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct SeasonSummary {
/// Season number; `None` for episodes without a season.
pub season: Option<u32>,
/// Episodes in the group.
pub episode_count: u64,
/// Episodes not completed.
pub unplayed_count: u64,
/// Oldest publication time.
pub first_published_at_ms: Option<i64>,
/// Newest publication time.
pub last_published_at_ms: Option<i64>,
}
/// Season filter of an [`EpisodeQuery`].
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub enum SeasonFilter {
/// All episodes.
#[default]
Any,
/// Episodes of one season.
Season(u32),
/// Episodes without a season.
NoSeason,
}
/// Listening-state filter of an [`EpisodeQuery`].
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub enum PlayedFilter {
/// All episodes.
#[default]
Any,
/// Not completed (not started or in progress).
Unplayed,
/// Never started or marked unplayed.
NotStarted,
/// In progress.
InProgress,
/// Completed.
Played,
}
/// Episode ordering of an [`EpisodeQuery`].
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub enum EpisodeSort {
/// Newest publication first; undated episodes last.
#[default]
NewestFirst,
/// Oldest publication first; undated episodes last.
OldestFirst,
/// By season, then episode number, ascending; unnumbered last.
EpisodeNumber,
/// By season, then episode number, descending; unnumbered last.
EpisodeNumberDesc,
}
/// Parameters of [`PodcastStore::episodes`].
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct EpisodeQuery {
/// Podcast.
pub podcast_id: PodcastId,
/// Season filter.
pub season: SeasonFilter,
/// Listening-state filter.
pub played: PlayedFilter,
/// Text searched in titles and descriptions (Unicode-normalized,
/// case-insensitive substring match). Blank means no filter.
pub search: Option<String>,
/// Ordering.
pub sort: EpisodeSort,
/// Maximum rows (`None`: all).
pub limit: Option<usize>,
/// Rows to skip.
pub offset: usize,
}
impl EpisodeQuery {
/// All episodes of a podcast, newest first.
pub fn new(podcast_id: PodcastId) -> Self {
Self {
podcast_id,
season: SeasonFilter::Any,
played: PlayedFilter::Any,
search: None,
sort: EpisodeSort::NewestFirst,
limit: None,
offset: 0,
}
}
}
/// An episode with its personal state.
#[derive(Debug, Clone, PartialEq)]
pub struct EpisodeListItem {
/// Episode metadata (feed or synced reference).
pub episode: EpisodeMeta,
/// Progress register, if any.
pub progress: Option<ProgressRecord>,
/// Distinct completed listening sessions (`completed_listen_count`).
pub completed_count: u64,
/// Whether the latest refresh of the feed listed the episode.
pub in_feed: bool,
}
impl EpisodeListItem {
/// Listening status (`NotStarted` without a register).
pub fn status(&self) -> ProgressStatus {
self.progress
.as_ref()
.map_or(ProgressStatus::NotStarted, |progress| progress.status)
}
/// Position ordinary Play resumes from.
pub fn resume_position_ms(&self) -> i64 {
self.progress
.as_ref()
.map_or(0, ProgressRecord::resume_position_ms)
}
}
/// A listening session in history, with display titles when known.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct PodcastListenEntry {
/// The session.
pub session: ListenSession,
/// Episode title from the catalog.
pub episode_title: Option<String>,
/// Podcast title from the catalog.
pub podcast_title: Option<String>,
}
/// A prototype row that could not be imported (kept in its `legacy_*` table).
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct LegacyQuarantineEntry {
/// `legacy_podcasts` or `legacy_podcast_episodes`.
pub legacy_table: String,
/// Row id in that table.
pub legacy_id: i64,
/// Short reason.
pub reason: String,
/// Details (the offending value or error).
pub detail: Option<String>,
}
const EPISODE_COLUMNS: &str = "e.episode_id, e.podcast_id, e.guid, e.title, e.description, \
e.season, e.episode_number, e.episode_type, e.published_at_ms, e.duration_ms, \
e.audio_url, e.mime, e.byte_len, e.image_url, e.link";
const EPISODE_COLUMN_COUNT: usize = 15;
fn episode_from_row(row: &rusqlite::Row<'_>) -> rusqlite::Result<EpisodeMeta> {
Ok(EpisodeMeta {
episode_id: episode_id_col(row.get(0)?)?,
podcast_id: podcast_id_col(row.get(1)?)?,
guid: row.get(2)?,
title: row.get(3)?,
description: row.get(4)?,
season: row.get(5)?,
episode_number: row.get(6)?,
episode_type: EpisodeType::parse(&row.get::<_, String>(7)?),
published_at_ms: row.get(8)?,
duration_ms: row.get(9)?,
audio_url: row.get(10)?,
mime: row.get(11)?,
byte_len: row
.get::<_, Option<i64>>(12)?
.and_then(|len| u64::try_from(len).ok()),
image_url: row.get(13)?,
link: row.get(14)?,
})
}
fn optional_progress(
row: &rusqlite::Row<'_>,
at: usize,
) -> rusqlite::Result<Option<ProgressRecord>> {
if row.get::<_, Option<String>>(at)?.is_none() {
return Ok(None);
}
progress_from_row(row, at).map(Some)
}
pub(crate) fn episode_meta_in(
conn: &Connection,
episode_id: &EpisodeId,
) -> Result<Option<EpisodeMeta>> {
Ok(conn
.query_row(
&format!("SELECT {EPISODE_COLUMNS} FROM episodes e WHERE e.episode_id = ?1"),
[episode_id.as_str()],
episode_from_row,
)
.optional()?)
}
pub(crate) fn podcast_meta_in(
conn: &Connection,
podcast_id: &PodcastId,
) -> Result<Option<PodcastMeta>> {
Ok(conn
.query_row(
"SELECT feed_url, title, author, description, link, image_url
FROM podcasts WHERE podcast_id = ?1",
[podcast_id.as_str()],
|row| {
Ok(PodcastMeta {
feed_url: feed_url_col(row.get(0)?)?,
title: row.get(1)?,
author: row.get(2)?,
description: row.get(3)?,
link: row.get(4)?,
image_url: row.get(5)?,
})
},
)
.optional()?)
}
/// Queue reference of a catalog episode, with `progress_generation` set to
/// the generation of its progress register (handoff hint).
pub(crate) fn episode_ref_in(
conn: &Connection,
episode_id: &EpisodeId,
) -> Result<Option<PodcastEpisodeRef>> {
let Some(episode) = episode_meta_in(conn, episode_id)? else {
return Ok(None);
};
let Some(podcast) = podcast_meta_in(conn, &episode.podcast_id)? else {
return Ok(None);
};
let mut reference = episode.to_ref(&podcast);
reference.progress_generation =
apply::load_progress(conn, episode_id)?.map(|progress| progress.generation);
Ok(Some(reference))
}
const NOT_COMPLETED: &str = "IFNULL(g.status, 'not_started') <> 'completed'";
impl PodcastStore {
/// Podcasts with subscription state and counters, sorted by title.
pub fn list_podcasts(&self, filter: PodcastListFilter) -> Result<Vec<PodcastSummary>> {
let conn = self.lock();
let join = match filter {
PodcastListFilter::Subscribed => {
"JOIN podcast_subscriptions r ON r.podcast_id = p.podcast_id AND r.subscribed = 1"
}
PodcastListFilter::All => "",
};
let ids: Vec<PodcastId> = {
let mut statement = conn.prepare(&format!(
"SELECT p.podcast_id FROM podcasts p {join}
ORDER BY p.title COLLATE NOCASE, p.podcast_id"
))?;
statement
.query_map([], |row| podcast_id_col(row.get(0)?))?
.collect::<rusqlite::Result<Vec<_>>>()?
};
let mut out = Vec::with_capacity(ids.len());
for id in ids {
if let Some(summary) = summary_in(&conn, &id)? {
out.push(summary);
}
}
Ok(out)
}
/// One podcast with subscription state and counters.
pub fn podcast(&self, podcast_id: &PodcastId) -> Result<Option<PodcastSummary>> {
summary_in(&self.lock(), podcast_id)
}
/// The subscription register (with the stored alias set), if any.
pub fn subscription(&self, podcast_id: &PodcastId) -> Result<Option<SubscriptionRecord>> {
apply::load_subscription_register(&self.lock(), podcast_id)
}
/// The share register, if any.
pub fn share(&self, podcast_id: &PodcastId) -> Result<Option<ShareRecord>> {
apply::load_share(&self.lock(), podcast_id)
}
/// The bounded, synced feed alias set of a podcast.
pub fn feed_aliases(&self, podcast_id: &PodcastId) -> Result<Vec<FeedUrl>> {
apply::load_aliases(&self.lock(), podcast_id)
}
/// Season groups of a podcast: numbered seasons ascending, then the
/// group of episodes without a season.
pub fn seasons(&self, podcast_id: &PodcastId) -> Result<Vec<SeasonSummary>> {
let conn = self.lock();
let mut statement = conn.prepare(&format!(
"SELECT s.season, s.episode_count, s.first_published_at_ms, s.last_published_at_ms,
(SELECT COUNT(*) FROM episodes e
LEFT JOIN episode_progress g ON g.episode_id = e.episode_id
WHERE e.podcast_id = s.podcast_id AND e.season IS s.season
AND {NOT_COMPLETED})
FROM podcast_seasons s WHERE s.podcast_id = ?1
ORDER BY s.season IS NULL, s.season"
))?;
let seasons = statement
.query_map([podcast_id.as_str()], |row| {
Ok(SeasonSummary {
season: row.get(0)?,
episode_count: count(row.get(1)?),
first_published_at_ms: row.get(2)?,
last_published_at_ms: row.get(3)?,
unplayed_count: count(row.get(4)?),
})
})?
.collect::<rusqlite::Result<Vec<_>>>()?;
Ok(seasons)
}
/// Episodes of a podcast or season with their progress.
pub fn episodes(&self, query: &EpisodeQuery) -> Result<Vec<EpisodeListItem>> {
let mut conditions = vec!["e.podcast_id = ?".to_string()];
let mut values = vec![Value::Text(query.podcast_id.as_str().to_string())];
match query.season {
SeasonFilter::Any => {}
SeasonFilter::Season(season) => {
conditions.push("e.season = ?".to_string());
values.push(Value::Integer(i64::from(season)));
}
SeasonFilter::NoSeason => conditions.push("e.season IS NULL".to_string()),
}
match query.played {
PlayedFilter::Any => {}
PlayedFilter::Unplayed => conditions.push(NOT_COMPLETED.to_string()),
PlayedFilter::NotStarted => {
conditions.push("IFNULL(g.status, 'not_started') = 'not_started'".to_string());
}
PlayedFilter::InProgress => conditions.push("g.status = 'in_progress'".to_string()),
PlayedFilter::Played => conditions.push("g.status = 'completed'".to_string()),
}
if let Some(search) = query.search.as_deref() {
let needle = music_dht::normalize_name(search);
if !needle.is_empty() {
conditions.push(
"(instr(norm(e.title), ?) > 0 OR instr(norm(e.description), ?) > 0)"
.to_string(),
);
values.push(Value::Text(needle.clone()));
values.push(Value::Text(needle));
}
}
let order = match query.sort {
EpisodeSort::NewestFirst => {
"e.published_at_ms IS NULL, e.published_at_ms DESC, e.episode_id"
}
EpisodeSort::OldestFirst => {
"e.published_at_ms IS NULL, e.published_at_ms, e.episode_id"
}
EpisodeSort::EpisodeNumber => {
"e.season IS NULL, e.season, e.episode_number IS NULL, e.episode_number, \
e.published_at_ms IS NULL, e.published_at_ms, e.episode_id"
}
EpisodeSort::EpisodeNumberDesc => {
"e.season IS NULL, e.season DESC, e.episode_number IS NULL, \
e.episode_number DESC, e.published_at_ms IS NULL, e.published_at_ms DESC, \
e.episode_id"
}
};
values.push(Value::Integer(
query
.limit
.map_or(-1, |limit| i64::try_from(limit).unwrap_or(i64::MAX)),
));
values.push(Value::Integer(
i64::try_from(query.offset).unwrap_or(i64::MAX),
));
let progress_columns = PROGRESS_COLUMNS.replace("p.", "g.");
let sql = format!(
"SELECT {EPISODE_COLUMNS}, e.in_feed, {progress_columns},
(SELECT COUNT(*) FROM podcast_listen_sessions s
WHERE s.episode_id = e.episode_id AND s.completed = 1)
FROM episodes e
LEFT JOIN episode_progress g ON g.episode_id = e.episode_id
WHERE {}
ORDER BY {order}
LIMIT ? OFFSET ?",
conditions.join(" AND ")
);
let conn = self.lock();
let mut statement = conn.prepare(&sql)?;
let items = statement
.query_map(params_from_iter(values), list_item_from_row)?
.collect::<rusqlite::Result<Vec<_>>>()?;
Ok(items)
}
/// One episode with its progress.
pub fn episode(&self, episode_id: &EpisodeId) -> Result<Option<EpisodeListItem>> {
let progress_columns = PROGRESS_COLUMNS.replace("p.", "g.");
Ok(self
.lock()
.query_row(
&format!(
"SELECT {EPISODE_COLUMNS}, e.in_feed, {progress_columns},
(SELECT COUNT(*) FROM podcast_listen_sessions s
WHERE s.episode_id = e.episode_id AND s.completed = 1)
FROM episodes e
LEFT JOIN episode_progress g ON g.episode_id = e.episode_id
WHERE e.episode_id = ?1"
),
[episode_id.as_str()],
list_item_from_row,
)
.optional()?)
}
/// Progress register of an episode (also for episodes the catalog does
/// not know yet).
pub fn progress(&self, episode_id: &EpisodeId) -> Result<Option<ProgressRecord>> {
apply::load_progress(&self.lock(), episode_id)
}
/// Position ordinary Play resumes from (`ProgressRecord::resume_position_ms`, 0 without progress).
pub fn resume_position_ms(&self, episode_id: &EpisodeId) -> Result<i64> {
Ok(self
.progress(episode_id)?
.map_or(0, |progress| progress.resume_position_ms()))
}
/// A queue item for an episode (see [`TrackItem::podcast_episode`]).
pub fn episode_track(&self, episode_id: &EpisodeId) -> Result<Option<TrackItem>> {
Ok(self
.episode_ref(episode_id)?
.map(TrackItem::podcast_episode))
}
/// Listening history (sessions that `should_record`), newest first.
pub fn listen_history(
&self,
podcast_id: Option<&PodcastId>,
limit: usize,
) -> Result<Vec<PodcastListenEntry>> {
let conn = self.lock();
let mut statement = conn.prepare(&format!(
"SELECT {SESSION_COLUMNS}, e.title, p.title
FROM podcast_listen_sessions s
LEFT JOIN episodes e ON e.episode_id = s.episode_id
LEFT JOIN podcasts p ON p.podcast_id = s.podcast_id
WHERE (s.completed = 1 OR s.listened_ms >= ?1)
AND (?2 IS NULL OR s.podcast_id = ?2)
ORDER BY s.ended_at_ms DESC, s.session_id DESC
LIMIT ?3"
))?;
let entries = statement
.query_map(
params![
music_dht::podcast::MIN_RECORDED_EPISODE_LISTEN_MS,
podcast_id.map(PodcastId::as_str),
i64::try_from(limit).unwrap_or(i64::MAX)
],
|row| {
Ok(PodcastListenEntry {
session: session_from_row(row, 0)?,
episode_title: row.get(10)?,
podcast_title: row.get(11)?,
})
},
)?
.collect::<rusqlite::Result<Vec<_>>>()?;
Ok(entries)
}
/// Number of distinct completed sessions of an episode.
pub fn completed_listen_count(&self, episode_id: &EpisodeId) -> Result<u64> {
let total: i64 = self.lock().query_row(
"SELECT COUNT(*) FROM podcast_listen_sessions WHERE episode_id = ?1 AND completed = 1",
[episode_id.as_str()],
|row| row.get(0),
)?;
Ok(count(total))
}
/// Prototype rows that could not be imported.
pub fn legacy_quarantine(&self) -> Result<Vec<LegacyQuarantineEntry>> {
let conn = self.lock();
let mut statement = conn.prepare(
"SELECT legacy_table, legacy_id, reason, detail FROM legacy_quarantine
ORDER BY legacy_table, legacy_id",
)?;
let entries = statement
.query_map([], |row| {
Ok(LegacyQuarantineEntry {
legacy_table: row.get(0)?,
legacy_id: row.get(1)?,
reason: row.get(2)?,
detail: row.get(3)?,
})
})?
.collect::<rusqlite::Result<Vec<_>>>()?;
Ok(entries)
}
}
fn count(value: i64) -> u64 {
u64::try_from(value).unwrap_or(0)
}
fn list_item_from_row(row: &rusqlite::Row<'_>) -> rusqlite::Result<EpisodeListItem> {
Ok(EpisodeListItem {
episode: episode_from_row(row)?,
in_feed: row.get(EPISODE_COLUMN_COUNT)?,
progress: optional_progress(row, EPISODE_COLUMN_COUNT + 1)?,
completed_count: count(row.get(EPISODE_COLUMN_COUNT + 10)?),
})
}
fn summary_in(conn: &Connection, podcast_id: &PodcastId) -> Result<Option<PodcastSummary>> {
let Some(meta) = podcast_meta_in(conn, podcast_id)? else {
return Ok(None);
};
let (episode_count, unplayed_count, in_progress_count, latest, season_count) = conn.query_row(
&format!(
"SELECT COUNT(*),
COUNT(*) FILTER (WHERE {NOT_COMPLETED}),
COUNT(*) FILTER (WHERE g.status = 'in_progress'),
MAX(e.published_at_ms),
(SELECT COUNT(*) FROM podcast_seasons s
WHERE s.podcast_id = ?1 AND s.season IS NOT NULL)
FROM episodes e
LEFT JOIN episode_progress g ON g.episode_id = e.episode_id
WHERE e.podcast_id = ?1"
),
[podcast_id.as_str()],
|row| {
Ok((
row.get::<_, i64>(0)?,
row.get::<_, i64>(1)?,
row.get::<_, i64>(2)?,
row.get::<_, Option<i64>>(3)?,
row.get::<_, i64>(4)?,
))
},
)?;
let subscription = apply::load_subscription_register(conn, podcast_id)?;
let share = apply::load_share(conn, podcast_id)?;
Ok(Some(PodcastSummary {
podcast_id: podcast_id.clone(),
meta,
subscribed: subscription
.as_ref()
.is_some_and(|record| record.subscribed),
shared: effective_share(subscription.as_ref(), share.as_ref()),
episode_count: count(episode_count),
unplayed_count: count(unplayed_count),
in_progress_count: count(in_progress_count),
latest_published_at_ms: latest,
season_count: count(season_count),
}))
}
@@ -0,0 +1,618 @@
//! Feed refresh ingestion and the local-only fetch state.
use std::collections::BTreeSet;
use anyhow::Result;
use music_dht::device_sync::SyncOpPayload;
use music_dht::podcast::feed::ParsedFeed;
use music_dht::podcast::{
EpisodeId, EpisodeIdentityHints, EpisodeMeta, FeedUrl, KnownEpisode, PodcastId,
merge_feed_aliases, resolve_episode_ids,
};
use rusqlite::{Connection, OptionalExtension as _, TransactionBehavior, params};
use super::apply::{self, EpisodeSource};
use super::{PodcastLocalOp, PodcastStore, PodcastStoreError, StampSource, now_ms};
/// A fetched and parsed feed document for [`PodcastStore::apply_feed`].
#[derive(Debug, Clone, Copy)]
pub struct FeedRefresh<'a> {
/// Podcast the feed belongs to.
pub podcast_id: &'a PodcastId,
/// The parsed document (`music_dht::podcast::feed::parse_feed_with_base`
/// with `fetched_from` as base).
pub feed: &'a ParsedFeed,
/// URL the document was finally served from, after redirects.
pub fetched_from: &'a FeedUrl,
/// Whether every redirect leading to `fetched_from` was permanent
/// (301/308). Temporary redirects do not move the feed.
pub moved_permanently: bool,
}
/// Result of [`PodcastStore::apply_feed`].
#[derive(Debug, Clone, Default, PartialEq)]
pub struct RefreshOutcome {
/// Episodes seen for the first time.
pub new_episodes: usize,
/// Known episodes whose metadata changed (title, audio URL, ...).
pub updated_episodes: usize,
/// Known episodes without metadata changes.
pub unchanged_episodes: usize,
/// Items without identity or with invalid metadata.
pub skipped_items: usize,
/// Feed URLs learned from a permanent redirect or `itunes:new-feed-url`.
pub learned_aliases: Vec<FeedUrl>,
/// URL the podcast is fetched from after this refresh.
pub feed_url: Option<FeedUrl>,
/// `podcast_feed_aliases_added`, only when the learned URLs changed the
/// stored alias set. Append it to the sync log.
pub ops: Vec<PodcastLocalOp>,
}
/// Local-only fetch state of a podcast. Never part of sync output.
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct FetchState {
/// HTTP `ETag` validator of the last successful fetch.
pub etag: Option<String>,
/// HTTP `Last-Modified` validator of the last successful fetch.
pub last_modified: Option<String>,
/// Last fetch attempt, unix milliseconds.
pub checked_at_ms: Option<i64>,
/// Last successful fetch (including "not modified").
pub last_success_at_ms: Option<i64>,
/// When the scheduler should check again (`None`: due now).
pub next_check_at_ms: Option<i64>,
/// Error of the last attempt; `None` after a success.
pub last_error: Option<String>,
/// Failed attempts since the last success.
pub consecutive_failures: u32,
/// Image URL the cached cover was downloaded from.
pub cover_url: Option<String>,
/// Local path of the cached cover.
pub cover_path: Option<String>,
}
fn podcast_fetch_url(conn: &Connection, podcast_id: &PodcastId) -> Result<Option<FeedUrl>> {
Ok(conn
.query_row(
"SELECT feed_url FROM podcasts WHERE podcast_id = ?1",
[podcast_id.as_str()],
|row| super::feed_url_col(row.get(0)?),
)
.optional()?)
}
fn fetch_state_in(conn: &Connection, podcast_id: &PodcastId) -> Result<FetchState> {
Ok(conn
.query_row(
"SELECT etag, last_modified, checked_at_ms, last_success_at_ms,
next_check_at_ms, last_error, consecutive_failures, cover_url,
cover_path
FROM podcast_fetch_state WHERE podcast_id = ?1",
[podcast_id.as_str()],
|row| {
Ok(FetchState {
etag: row.get(0)?,
last_modified: row.get(1)?,
checked_at_ms: row.get(2)?,
last_success_at_ms: row.get(3)?,
next_check_at_ms: row.get(4)?,
last_error: row.get(5)?,
consecutive_failures: u32::try_from(row.get::<_, i64>(6)?).unwrap_or(u32::MAX),
cover_url: row.get(7)?,
cover_path: row.get(8)?,
})
},
)
.optional()?
.unwrap_or_default())
}
fn has_personal_state(conn: &Connection, episode_id: &EpisodeId) -> Result<bool> {
Ok(conn.query_row(
"SELECT EXISTS (SELECT 1 FROM episode_progress WHERE episode_id = ?1)
OR EXISTS (SELECT 1 FROM podcast_listen_sessions WHERE episode_id = ?1)",
[episode_id.as_str()],
|row| row.get(0),
)?)
}
/// Re-keys episodes imported from the prototype whose stored `guid` was the
/// enclosure URL (`legacy_guid`) when the feed shows it is a real GUID.
///
/// Only episodes without progress or sessions move, so no personal state is
/// touched; otherwise the GUID item becomes a separate episode. When another
/// episode already holds the GUID id (for example one a remote operation
/// created), the imported episode is a duplicate of it and is deleted.
fn rekey_legacy_guids(conn: &Connection, podcast_id: &PodcastId, feed: &ParsedFeed) -> Result<()> {
for item in &feed.episodes {
let Some(guid) = item.guid.as_deref() else {
continue;
};
let Some(guid_id) = EpisodeId::from_guid(podcast_id, guid) else {
continue;
};
let taken = conn
.query_row(
"SELECT 1 FROM episodes WHERE episode_id = ?1",
[guid_id.as_str()],
|_| Ok(()),
)
.optional()?
.is_some();
let legacy: Option<String> = conn
.query_row(
"SELECT episode_id FROM episodes
WHERE podcast_id = ?1 AND legacy_guid = ?2 ORDER BY episode_id LIMIT 1",
params![podcast_id.as_str(), guid],
|row| row.get(0),
)
.optional()?;
let Some(legacy) = legacy else {
continue;
};
let legacy = super::episode_id_col(legacy)?;
if legacy == guid_id || has_personal_state(conn, &legacy)? {
continue;
}
if taken {
conn.execute(
"DELETE FROM episodes WHERE episode_id = ?1",
[legacy.as_str()],
)?;
continue;
}
conn.execute(
"UPDATE episodes SET episode_id = ?2, guid = ?3, legacy_guid = NULL
WHERE episode_id = ?1",
params![legacy.as_str(), guid_id.as_str(), guid],
)?;
}
Ok(())
}
/// Episodes imported from the prototype without a GUID (`legacy_guid` set)
/// that carry no progress and no session.
///
/// Their ids were resolved over every stored prototype row at once, stale
/// rows included, so they may differ from what a fresh device derives from
/// the current feed. They are provisional until a refresh confirms them.
fn provisional_episode_ids(
conn: &Connection,
podcast_id: &PodcastId,
) -> Result<BTreeSet<EpisodeId>> {
let mut statement = conn.prepare(
"SELECT e.episode_id FROM episodes e
WHERE e.podcast_id = ?1 AND e.legacy_guid IS NOT NULL
AND NOT EXISTS (SELECT 1 FROM episode_progress p WHERE p.episode_id = e.episode_id)
AND NOT EXISTS (SELECT 1 FROM podcast_listen_sessions s
WHERE s.episode_id = e.episode_id)",
)?;
let ids = statement
.query_map([podcast_id.as_str()], |row| {
super::episode_id_col(row.get(0)?)
})?
.collect::<rusqlite::Result<BTreeSet<_>>>()?;
Ok(ids)
}
/// Resolves the ids of the feed items as a device without prototype history
/// would, and moves provisional imported episodes onto them.
///
/// * The ids come from the contract identity rule against every stored
/// episode except the provisional ones (episodes from earlier refreshes,
/// from sync, and imported ones that already carry personal state).
/// * Each provisional episode is matched to the item it was imported from
/// with the same rule (enclosure, then link, then date and title; GUIDs
/// ignored, since the prototype may have stored the enclosure URL as GUID)
/// and re-keyed to that item's id. When another stored episode that stays
/// already holds that id, it is the item's episode and the provisional one
/// is a duplicate: it is deleted. Unmatched provisional episodes stay as
/// they are (not in the feed); the next refresh that lists them re-keys
/// them.
///
/// Provisional episodes have no progress and no session, so no personal
/// state moves. The refresh then clears `legacy_guid` on every episode it
/// lists, which makes their ids final.
fn resolve_and_rekey_provisional(
conn: &Connection,
podcast_id: &PodcastId,
feed: &ParsedFeed,
) -> Result<Vec<Option<EpisodeId>>> {
let known = apply::known_episodes(conn, podcast_id)?;
let provisional = provisional_episode_ids(conn, podcast_id)?;
if provisional.is_empty() {
return Ok(feed.resolve_episode_ids(podcast_id, &known));
}
let (provisional_known, stable_known): (Vec<KnownEpisode>, Vec<KnownEpisode>) = known
.into_iter()
.partition(|episode| provisional.contains(&episode.episode_id));
let ids = feed.resolve_episode_ids(podcast_id, &stable_known);
let guidless: Vec<EpisodeIdentityHints> = feed
.identity_hints()
.into_iter()
.map(|hints| EpisodeIdentityHints {
guid: None,
..hints
})
.collect();
let matches = resolve_episode_ids(
podcast_id,
feed.link.as_deref(),
&guidless,
&provisional_known,
);
let mut moves: Vec<(EpisodeId, EpisodeId)> = matches
.into_iter()
.zip(&ids)
.filter_map(|(matched, target)| {
let matched = matched.filter(|id| provisional.contains(id))?;
let target = target.clone()?;
(matched != target).then_some((matched, target))
})
.collect();
// A target held by an episode that stays (for example a row a remote
// operation created before this first refresh) is the item's episode
// already: the provisional episode is a duplicate of it. It has no
// progress and no session, and its metadata would be overwritten by the
// item anyway, so it is removed instead of kept as a second copy. A
// removed duplicate frees its own id for other moves.
let mut occupied: BTreeSet<EpisodeId> = stable_known
.iter()
.chain(&provisional_known)
.map(|episode| episode.episode_id.clone())
.collect();
let mut duplicates: Vec<EpisodeId> = Vec::new();
loop {
let leaving: BTreeSet<&EpisodeId> = moves.iter().map(|(from, _)| from).collect();
let blocked: BTreeSet<EpisodeId> = moves
.iter()
.filter(|(_, to)| occupied.contains(to) && !leaving.contains(to))
.map(|(from, _)| from.clone())
.collect();
if blocked.is_empty() {
break;
}
moves.retain(|(from, _)| !blocked.contains(from));
for from in blocked {
occupied.remove(&from);
duplicates.push(from);
}
}
for duplicate in &duplicates {
// Guarded again: only a provisional episode without personal state.
conn.execute(
"DELETE FROM episodes
WHERE episode_id = ?1 AND legacy_guid IS NOT NULL
AND NOT EXISTS (SELECT 1 FROM episode_progress p
WHERE p.episode_id = episodes.episode_id)
AND NOT EXISTS (SELECT 1 FROM podcast_listen_sessions s
WHERE s.episode_id = episodes.episode_id)",
[duplicate.as_str()],
)?;
}
// Two phases, so a move may take an id another move vacates.
for (index, (from, _)) in moves.iter().enumerate() {
conn.execute(
"UPDATE episodes SET episode_id = ?2 WHERE episode_id = ?1",
params![from.as_str(), format!("rekey-pending:{index}")],
)?;
}
for (index, (_, to)) in moves.iter().enumerate() {
conn.execute(
"UPDATE episodes SET episode_id = ?2 WHERE episode_id = ?1",
params![format!("rekey-pending:{index}"), to.as_str()],
)?;
}
Ok(ids)
}
fn upsert_feed_episode(conn: &Connection, meta: &EpisodeMeta, now: i64) -> Result<Option<bool>> {
let Some(stored) = super::queries::episode_meta_in(conn, &meta.episode_id)? else {
apply::insert_episode(conn, meta, EpisodeSource::Feed, true, None, now, now)?;
return Ok(None);
};
let changed = stored != *meta;
conn.execute(
"UPDATE episodes
SET podcast_id = ?2, guid = ?3, title = ?4, description = ?5, season = ?6,
episode_number = ?7, episode_type = ?8, published_at_ms = ?9,
duration_ms = ?10, audio_url = ?11, mime = ?12, byte_len = ?13,
image_url = ?14, link = ?15, source = 'feed', in_feed = 1,
legacy_guid = NULL,
updated_at_ms = CASE WHEN ?16 THEN ?17 ELSE updated_at_ms END
WHERE episode_id = ?1",
params![
meta.episode_id.as_str(),
meta.podcast_id.as_str(),
meta.guid,
meta.title,
meta.description,
meta.season,
meta.episode_number,
apply::episode_type_to_sql(meta.episode_type),
meta.published_at_ms,
meta.duration_ms,
meta.audio_url,
meta.mime,
meta.byte_len.and_then(|len| i64::try_from(len).ok()),
meta.image_url,
meta.link,
changed,
now,
],
)?;
Ok(Some(changed))
}
impl PodcastStore {
/// Ingests a fetched feed document for a podcast.
///
/// * Channel metadata of the catalog row is replaced by the feed's.
/// * Episode ids are resolved with the contract identity rule
/// (`ParsedFeed::resolve_episode_ids`) against every stored episode of
/// the podcast, including episodes learned through sync, so a changed
/// title, audio URL or a GUID-less item keeps its id. Known episodes
/// get the new metadata (including the audio URL); new ones are added;
/// episodes missing from the document are kept (`in_feed = false`).
/// * A permanent redirect target and `itunes:new-feed-url` become feed
/// aliases and the new fetch URL. When they change the stored alias
/// set, one `podcast_feed_aliases_added` operation is emitted.
/// * Subscription, share and progress registers and listening sessions
/// are never touched, so a refresh cannot resurrect an unsubscribed
/// podcast or move anyone's position.
///
/// Record the HTTP outcome separately with
/// [`PodcastStore::record_fetch_success`] and friends.
pub fn apply_feed(
&self,
stamps: &mut dyn StampSource,
refresh: &FeedRefresh<'_>,
) -> Result<RefreshOutcome> {
let podcast_id = refresh.podcast_id;
let feed = refresh.feed;
let now = now_ms();
let mut conn = self.lock();
let tx = conn.transaction_with_behavior(TransactionBehavior::Immediate)?;
let current_url = match podcast_fetch_url(&tx, podcast_id)? {
Some(url) => url,
None => {
let Some(register) = apply::load_subscription_register(&tx, podcast_id)? else {
return Err(PodcastStoreError::UnknownPodcast(podcast_id.clone()).into());
};
apply::apply_subscription_record(&tx, &register, now)?;
register.podcast.feed_url
}
};
rekey_legacy_guids(&tx, podcast_id, feed)?;
let ids = resolve_and_rekey_provisional(&tx, podcast_id, feed)?;
tx.execute(
"UPDATE episodes SET in_feed = 0 WHERE podcast_id = ?1",
[podcast_id.as_str()],
)?;
let mut outcome = RefreshOutcome::default();
for (item, id) in feed.episodes.iter().zip(ids) {
let Some(id) = id else {
outcome.skipped_items += 1;
continue;
};
let meta = item.to_meta(podcast_id, id);
if meta.validate().is_err() {
outcome.skipped_items += 1;
continue;
}
match upsert_feed_episode(&tx, &meta, now)? {
None => outcome.new_episodes += 1,
Some(true) => outcome.updated_episodes += 1,
Some(false) => outcome.unchanged_episodes += 1,
}
}
let mut fetch_url = current_url.clone();
let mut learned = BTreeSet::new();
if refresh.moved_permanently && *refresh.fetched_from != fetch_url {
learned.insert(fetch_url.clone());
learned.insert(refresh.fetched_from.clone());
fetch_url = refresh.fetched_from.clone();
}
if let Some(new_url) = feed
.new_feed_url
.as_deref()
.and_then(|url| FeedUrl::parse(url).ok())
&& new_url != fetch_url
{
learned.insert(fetch_url.clone());
learned.insert(new_url.clone());
fetch_url = new_url;
}
let meta = feed.to_podcast_meta(fetch_url.clone());
tx.execute(
"UPDATE podcasts
SET feed_url = ?2, title = ?3, author = ?4, description = ?5, link = ?6,
image_url = ?7, meta_from_feed = 1, updated_at_ms = ?8
WHERE podcast_id = ?1",
params![
podcast_id.as_str(),
meta.feed_url.as_str(),
meta.title,
meta.author,
meta.description,
meta.link,
meta.image_url,
now
],
)?;
if !learned.is_empty() {
let learned: Vec<FeedUrl> = learned.into_iter().collect();
let current = apply::load_aliases(&tx, podcast_id)?;
if merge_feed_aliases(podcast_id, &current, &learned) != current {
let op = super::actions::emit(&tx, stamps, |_| {
SyncOpPayload::PodcastFeedAliasesAdded {
podcast_id: podcast_id.clone(),
feed_aliases: learned.clone(),
}
})?;
outcome.ops.push(op);
}
outcome.learned_aliases = learned;
}
apply::refresh_seasons(&tx, podcast_id)?;
tx.commit()?;
outcome.feed_url = Some(fetch_url);
Ok(outcome)
}
/// Local fetch state of a podcast (defaults when never fetched).
pub fn fetch_state(&self, podcast_id: &PodcastId) -> Result<FetchState> {
fetch_state_in(&self.lock(), podcast_id)
}
/// Read-modify-write of the fetch state under one lock and one immediate
/// transaction, so concurrent `record_*` / `set_cover_cache` calls (from
/// this or another process) never lose each other's columns or failure
/// increments.
fn update_fetch_state(
&self,
podcast_id: &PodcastId,
update: impl FnOnce(&mut FetchState),
) -> Result<FetchState> {
let mut conn = self.lock();
let tx = conn.transaction_with_behavior(TransactionBehavior::Immediate)?;
if podcast_fetch_url(&tx, podcast_id)?.is_none() {
return Err(PodcastStoreError::UnknownPodcast(podcast_id.clone()).into());
}
let mut state = fetch_state_in(&tx, podcast_id)?;
update(&mut state);
tx.execute(
"INSERT INTO podcast_fetch_state
(podcast_id, etag, last_modified, checked_at_ms, last_success_at_ms,
next_check_at_ms, last_error, consecutive_failures, cover_url, cover_path)
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10)
ON CONFLICT(podcast_id) DO UPDATE SET
etag = excluded.etag, last_modified = excluded.last_modified,
checked_at_ms = excluded.checked_at_ms,
last_success_at_ms = excluded.last_success_at_ms,
next_check_at_ms = excluded.next_check_at_ms,
last_error = excluded.last_error,
consecutive_failures = excluded.consecutive_failures,
cover_url = excluded.cover_url, cover_path = excluded.cover_path",
params![
podcast_id.as_str(),
state.etag,
state.last_modified,
state.checked_at_ms,
state.last_success_at_ms,
state.next_check_at_ms,
state.last_error,
i64::from(state.consecutive_failures),
state.cover_url,
state.cover_path,
],
)?;
tx.commit()?;
Ok(state)
}
/// Records a successful fetch (HTTP 200) with its new validators.
pub fn record_fetch_success(
&self,
podcast_id: &PodcastId,
now_ms: i64,
etag: Option<&str>,
last_modified: Option<&str>,
next_check_at_ms: Option<i64>,
) -> Result<()> {
self.update_fetch_state(podcast_id, |state| {
state.etag = etag.map(str::to_string);
state.last_modified = last_modified.map(str::to_string);
state.checked_at_ms = Some(now_ms);
state.last_success_at_ms = Some(now_ms);
state.next_check_at_ms = next_check_at_ms;
state.last_error = None;
state.consecutive_failures = 0;
})
.map(|_| ())
}
/// Records a conditional fetch answered with "304 Not Modified".
pub fn record_fetch_not_modified(
&self,
podcast_id: &PodcastId,
now_ms: i64,
next_check_at_ms: Option<i64>,
) -> Result<()> {
self.update_fetch_state(podcast_id, |state| {
state.checked_at_ms = Some(now_ms);
state.last_success_at_ms = Some(now_ms);
state.next_check_at_ms = next_check_at_ms;
state.last_error = None;
state.consecutive_failures = 0;
})
.map(|_| ())
}
/// Records a failed fetch; returns the consecutive failure count. The
/// stored catalog is kept.
pub fn record_fetch_failure(
&self,
podcast_id: &PodcastId,
now_ms: i64,
error: &str,
next_check_at_ms: Option<i64>,
) -> Result<u32> {
self.update_fetch_state(podcast_id, |state| {
state.checked_at_ms = Some(now_ms);
state.next_check_at_ms = next_check_at_ms;
state.last_error = Some(error.to_string());
state.consecutive_failures = state.consecutive_failures.saturating_add(1);
})
.map(|state| state.consecutive_failures)
}
/// Records the locally cached cover image of a podcast.
pub fn set_cover_cache(
&self,
podcast_id: &PodcastId,
cover_url: Option<&str>,
cover_path: Option<&str>,
) -> Result<()> {
self.update_fetch_state(podcast_id, |state| {
state.cover_url = cover_url.map(str::to_string);
state.cover_path = cover_path.map(str::to_string);
})
.map(|_| ())
}
/// Subscribed podcasts due for a refresh at `now_ms` (never checked or
/// `next_check_at_ms <= now_ms`), most overdue first, with their fetch URL.
pub fn podcasts_due_for_refresh(
&self,
now_ms: i64,
limit: usize,
) -> Result<Vec<(PodcastId, FeedUrl)>> {
let conn = self.lock();
let mut statement = conn.prepare(
"SELECT p.podcast_id, p.feed_url
FROM podcasts p
JOIN podcast_subscriptions s ON s.podcast_id = p.podcast_id AND s.subscribed = 1
LEFT JOIN podcast_fetch_state f ON f.podcast_id = p.podcast_id
WHERE f.next_check_at_ms IS NULL OR f.next_check_at_ms <= ?1
ORDER BY IFNULL(f.next_check_at_ms, 0), p.podcast_id
LIMIT ?2",
)?;
let due = statement
.query_map(
params![now_ms, i64::try_from(limit).unwrap_or(i64::MAX)],
|row| {
Ok((
super::podcast_id_col(row.get(0)?)?,
super::feed_url_col(row.get(1)?)?,
))
},
)?
.collect::<rusqlite::Result<Vec<_>>>()?;
Ok(due)
}
}
@@ -0,0 +1,796 @@
//! Schema, migrations and the import of the prototype database.
use std::collections::{BTreeMap, BTreeSet};
use anyhow::{Context as _, Result};
use music_dht::podcast::{
EpisodeId, EpisodeIdentityHints, EpisodeMeta, EpisodeType, FeedUrl, KnownEpisode,
MAX_DESCRIPTION_CHARS, MAX_TITLE_CHARS, MAX_URL_BYTES, PodcastId, PodcastMeta,
SubscriptionRecord, bounded_text, is_http_url, normalize_episode_title, resolve_episode_ids,
resolve_podcast_id,
};
use rusqlite::{Connection, OptionalExtension as _, Transaction, TransactionBehavior, params};
use super::apply::{self, EpisodeSource};
use super::{LEGACY_IMPORT_OP_PREFIX, PodcastStoreError, now_ms};
/// Newest schema version (`PRAGMA user_version`) this build writes.
pub const PODCAST_SCHEMA_VERSION: i64 = 2;
/// `user_version` of the prototype database of the original podcast PR.
pub const LEGACY_PROTOTYPE_SCHEMA_VERSION: i64 = 1;
/// Migration 1: the prototype schema of the original podcast PR, verbatim
/// (`furumi-tui` `src/podcasts/store.rs`, `MIGRATIONS[0]`). It versioned
/// itself with `PRAGMA user_version = 1` in the same transaction.
const PROTOTYPE_SCHEMA_V1: &str = "
CREATE TABLE IF NOT EXISTS podcasts (
id INTEGER PRIMARY KEY AUTOINCREMENT,
feed_url TEXT NOT NULL UNIQUE,
title TEXT NOT NULL,
author TEXT,
description TEXT,
link TEXT,
image_url TEXT,
cover_url TEXT,
cover_path TEXT,
etag TEXT,
last_modified TEXT,
added_at INTEGER NOT NULL,
last_checked_at INTEGER,
last_error TEXT
);
CREATE TABLE IF NOT EXISTS podcast_episodes (
id INTEGER PRIMARY KEY AUTOINCREMENT,
podcast_id INTEGER NOT NULL REFERENCES podcasts(id) ON DELETE CASCADE,
guid TEXT NOT NULL,
title TEXT NOT NULL,
description TEXT,
published_at INTEGER,
duration_seconds REAL,
audio_url TEXT NOT NULL,
mime_type TEXT,
byte_len INTEGER,
link TEXT,
image_url TEXT,
first_seen_at INTEGER NOT NULL,
UNIQUE (podcast_id, guid)
);
CREATE INDEX IF NOT EXISTS podcast_episodes_by_date
ON podcast_episodes (podcast_id, published_at DESC, id DESC);
";
#[cfg(test)]
pub(super) const PROTOTYPE_SCHEMA_V1_FOR_TESTS: &str = PROTOTYPE_SCHEMA_V1;
/// Columns every prototype `podcasts` table has.
const PROTOTYPE_PODCAST_COLUMNS: &[&str] = &[
"id",
"feed_url",
"title",
"author",
"description",
"link",
"image_url",
"cover_url",
"cover_path",
"etag",
"last_modified",
"added_at",
"last_checked_at",
"last_error",
];
/// Columns every prototype `podcast_episodes` table has.
const PROTOTYPE_EPISODE_COLUMNS: &[&str] = &[
"id",
"podcast_id",
"guid",
"title",
"description",
"published_at",
"duration_seconds",
"audio_url",
"mime_type",
"byte_len",
"link",
"image_url",
"first_seen_at",
];
/// Migration 2: the contract model. See `PODCASTS.md`, "SQLite reference store".
const SCHEMA_V2: &str = "
CREATE TABLE IF NOT EXISTS podcasts (
podcast_id TEXT PRIMARY KEY,
feed_url TEXT NOT NULL,
title TEXT NOT NULL,
author TEXT,
description TEXT,
link TEXT,
image_url TEXT,
meta_from_feed INTEGER NOT NULL DEFAULT 0,
created_at_ms INTEGER NOT NULL,
updated_at_ms INTEGER NOT NULL
);
CREATE TABLE IF NOT EXISTS podcast_feed_aliases (
podcast_id TEXT NOT NULL,
feed_url TEXT NOT NULL,
PRIMARY KEY (podcast_id, feed_url)
);
CREATE INDEX IF NOT EXISTS podcast_feed_aliases_by_url
ON podcast_feed_aliases (feed_url);
CREATE TABLE IF NOT EXISTS episodes (
episode_id TEXT PRIMARY KEY,
podcast_id TEXT NOT NULL REFERENCES podcasts(podcast_id),
guid TEXT,
title TEXT NOT NULL,
description TEXT,
season INTEGER,
episode_number INTEGER,
episode_type TEXT NOT NULL DEFAULT 'full',
published_at_ms INTEGER,
duration_ms INTEGER,
audio_url TEXT NOT NULL,
mime TEXT,
byte_len INTEGER,
image_url TEXT,
link TEXT,
source TEXT NOT NULL CHECK (source IN ('feed', 'sync')),
in_feed INTEGER NOT NULL DEFAULT 0,
legacy_guid TEXT,
first_seen_at_ms INTEGER NOT NULL,
updated_at_ms INTEGER NOT NULL
);
CREATE INDEX IF NOT EXISTS episodes_by_podcast_date
ON episodes (podcast_id, published_at_ms DESC);
CREATE INDEX IF NOT EXISTS episodes_by_podcast_season
ON episodes (podcast_id, season, episode_number);
CREATE TABLE IF NOT EXISTS podcast_seasons (
podcast_id TEXT NOT NULL REFERENCES podcasts(podcast_id),
season INTEGER,
episode_count INTEGER NOT NULL,
first_published_at_ms INTEGER,
last_published_at_ms INTEGER
);
CREATE UNIQUE INDEX IF NOT EXISTS podcast_seasons_key
ON podcast_seasons (podcast_id, IFNULL(season, -1));
CREATE TABLE IF NOT EXISTS podcast_subscriptions (
podcast_id TEXT PRIMARY KEY,
subscribed INTEGER NOT NULL,
podcast_json TEXT NOT NULL,
generation INTEGER NOT NULL CHECK (generation >= 0),
hlc_ms INTEGER NOT NULL,
op_id TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS podcast_shares (
podcast_id TEXT PRIMARY KEY,
shared INTEGER NOT NULL,
generation INTEGER NOT NULL CHECK (generation >= 0),
hlc_ms INTEGER NOT NULL,
op_id TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS episode_progress (
episode_id TEXT PRIMARY KEY,
podcast_id TEXT NOT NULL,
position_ms INTEGER NOT NULL,
status TEXT NOT NULL
CHECK (status IN ('not_started', 'in_progress', 'completed')),
generation INTEGER NOT NULL CHECK (generation >= 0),
session_id TEXT NOT NULL,
hlc_ms INTEGER NOT NULL,
device_id TEXT NOT NULL,
op_id TEXT NOT NULL
);
CREATE INDEX IF NOT EXISTS episode_progress_by_podcast
ON episode_progress (podcast_id);
CREATE TABLE IF NOT EXISTS podcast_listen_sessions (
session_id TEXT PRIMARY KEY,
episode_id TEXT NOT NULL,
podcast_id TEXT NOT NULL,
device_id TEXT NOT NULL,
started_at_ms INTEGER NOT NULL,
ended_at_ms INTEGER NOT NULL,
start_position_ms INTEGER NOT NULL,
end_position_ms INTEGER NOT NULL,
listened_ms INTEGER NOT NULL,
completed INTEGER NOT NULL
);
CREATE INDEX IF NOT EXISTS podcast_listen_sessions_by_episode
ON podcast_listen_sessions (episode_id);
CREATE INDEX IF NOT EXISTS podcast_listen_sessions_by_end
ON podcast_listen_sessions (ended_at_ms DESC);
CREATE TABLE IF NOT EXISTS podcast_fetch_state (
podcast_id TEXT PRIMARY KEY REFERENCES podcasts(podcast_id),
etag TEXT,
last_modified TEXT,
checked_at_ms INTEGER,
last_success_at_ms INTEGER,
next_check_at_ms INTEGER,
last_error TEXT,
consecutive_failures INTEGER NOT NULL DEFAULT 0,
cover_url TEXT,
cover_path TEXT
);
CREATE TABLE IF NOT EXISTS legacy_quarantine (
legacy_table TEXT NOT NULL,
legacy_id INTEGER NOT NULL,
reason TEXT NOT NULL,
detail TEXT,
PRIMARY KEY (legacy_table, legacy_id)
);
";
/// Tables a version-2 store owns (the `legacy_*` tables included).
const V2_TABLES: &[&str] = &[
"podcasts",
"podcast_feed_aliases",
"episodes",
"podcast_seasons",
"podcast_subscriptions",
"podcast_shares",
"episode_progress",
"podcast_listen_sessions",
"podcast_fetch_state",
"legacy_quarantine",
"legacy_podcasts",
"legacy_podcast_episodes",
];
fn user_version(conn: &Connection) -> Result<i64> {
Ok(conn.query_row("PRAGMA user_version", [], |row| row.get(0))?)
}
fn user_tables(conn: &Connection) -> Result<BTreeSet<String>> {
let mut statement = conn.prepare(
"SELECT name FROM sqlite_master WHERE type = 'table' AND name NOT LIKE 'sqlite_%'",
)?;
let names = statement
.query_map([], |row| row.get::<_, String>(0))?
.collect::<rusqlite::Result<BTreeSet<_>>>()?;
Ok(names)
}
pub(crate) fn table_columns(conn: &Connection, table: &str) -> Result<BTreeSet<String>> {
let mut statement = conn.prepare("SELECT name FROM pragma_table_info(?1)")?;
let names = statement
.query_map([table], |row| row.get::<_, String>(0))?
.collect::<rusqlite::Result<BTreeSet<_>>>()?;
Ok(names)
}
fn unrecognized(reason: impl Into<String>) -> anyhow::Error {
PodcastStoreError::UnrecognizedDatabase(reason.into()).into()
}
fn has_columns(conn: &Connection, table: &str, required: &[&str]) -> Result<bool> {
let columns = table_columns(conn, table)?;
Ok(required.iter().all(|column| columns.contains(*column)))
}
/// Checks the prototype tables that exist have the prototype shape.
fn check_prototype_shape(conn: &Connection, tables: &BTreeSet<String>) -> Result<()> {
if tables.contains("podcasts") && !has_columns(conn, "podcasts", PROTOTYPE_PODCAST_COLUMNS)? {
return Err(unrecognized(
"table `podcasts` does not have the prototype columns",
));
}
if tables.contains("podcast_episodes")
&& !has_columns(conn, "podcast_episodes", PROTOTYPE_EPISODE_COLUMNS)?
{
return Err(unrecognized(
"table `podcast_episodes` does not have the prototype columns",
));
}
Ok(())
}
/// Read-only check run before anything is written to the file.
///
/// * `user_version` above [`PODCAST_SCHEMA_VERSION`]: written by a newer
/// Furumi, refused.
/// * 0: must be empty, or hold only the prototype tables (the prototype
/// sets its version in the same transaction, so this is defensive).
/// * 1: the prototype database; its tables must have the prototype shape.
/// * 2: the current schema.
///
/// Any other table means the path points at a different database (for
/// example `library.db`), which is refused instead of being modified.
pub(super) fn check_openable(conn: &Connection) -> Result<()> {
let version = user_version(conn)?;
if version > PODCAST_SCHEMA_VERSION {
return Err(PodcastStoreError::NewerSchema {
found: version,
supported: PODCAST_SCHEMA_VERSION,
}
.into());
}
if version < 0 {
return Err(unrecognized(format!("negative user_version {version}")));
}
let tables = user_tables(conn)?;
let allowed: &[&str] = if version < PODCAST_SCHEMA_VERSION {
&["podcasts", "podcast_episodes"]
} else {
V2_TABLES
};
if let Some(table) = tables
.iter()
.find(|table| !allowed.contains(&table.as_str()))
{
return Err(unrecognized(format!(
"unexpected table `{table}` at schema version {version}"
)));
}
if version < PODCAST_SCHEMA_VERSION {
check_prototype_shape(conn, &tables)?;
} else if tables.contains("podcasts")
&& !has_columns(conn, "podcasts", &["podcast_id", "feed_url"])?
{
return Err(unrecognized("table `podcasts` is not the version 2 table"));
}
Ok(())
}
/// Applies every pending migration, each in its own immediate transaction
/// that re-reads the version (another process may have migrated meanwhile).
pub(super) fn migrate(conn: &mut Connection) -> Result<()> {
for target in 1..=PODCAST_SCHEMA_VERSION {
let tx = conn.transaction_with_behavior(TransactionBehavior::Immediate)?;
let version = user_version(&tx)?;
if version > PODCAST_SCHEMA_VERSION {
return Err(PodcastStoreError::NewerSchema {
found: version,
supported: PODCAST_SCHEMA_VERSION,
}
.into());
}
if version >= target {
continue;
}
match target {
1 => tx
.execute_batch(PROTOTYPE_SCHEMA_V1)
.context("creating the prototype podcast schema (v1)")?,
2 => migrate_to_v2(&tx).context("migrating the podcast database to v2")?,
_ => unreachable!("no migration for version {target}"),
}
tx.pragma_update(None, "user_version", target)?;
tx.commit()
.with_context(|| format!("committing podcast schema v{target}"))?;
}
Ok(())
}
fn migrate_to_v2(tx: &Transaction<'_>) -> Result<()> {
let tables = user_tables(tx)?;
// Rename, never drop: the prototype data stays available as `legacy_*`.
// Renaming `podcasts` also rewrites the foreign key of the episode table.
if tables.contains("podcasts") && !tables.contains("legacy_podcasts") {
tx.execute_batch("ALTER TABLE podcasts RENAME TO legacy_podcasts")?;
}
if tables.contains("podcast_episodes") && !tables.contains("legacy_podcast_episodes") {
tx.execute_batch("ALTER TABLE podcast_episodes RENAME TO legacy_podcast_episodes")?;
}
tx.execute_batch(SCHEMA_V2)?;
let tables = user_tables(tx)?;
if tables.contains("legacy_podcasts") {
import_prototype(tx, tables.contains("legacy_podcast_episodes"))?;
}
Ok(())
}
struct LegacyPodcast {
id: i64,
feed_url: String,
title: String,
author: Option<String>,
description: Option<String>,
link: Option<String>,
image_url: Option<String>,
cover_url: Option<String>,
cover_path: Option<String>,
etag: Option<String>,
last_modified: Option<String>,
added_at: i64,
last_checked_at: Option<i64>,
last_error: Option<String>,
}
struct LegacyEpisode {
id: i64,
guid: String,
title: String,
description: Option<String>,
published_at: Option<i64>,
duration_seconds: Option<f64>,
audio_url: String,
mime_type: Option<String>,
byte_len: Option<i64>,
link: Option<String>,
image_url: Option<String>,
first_seen_at: i64,
}
fn quarantine(
tx: &Transaction<'_>,
table: &str,
id: i64,
reason: &str,
detail: &str,
) -> Result<()> {
tx.execute(
"INSERT OR IGNORE INTO legacy_quarantine (legacy_table, legacy_id, reason, detail)
VALUES (?1, ?2, ?3, ?4)",
params![table, id, reason, detail],
)?;
Ok(())
}
fn secs_to_ms(secs: i64) -> i64 {
secs.saturating_mul(1_000)
}
/// Imports the prototype tables into the contract model.
///
/// * Every prototype row is an active subscription: it becomes a
/// subscription register with `subscribed = true`, generation 0,
/// `hlc_ms` = `added_at` and `op_id` = [`LEGACY_IMPORT_OP_PREFIX`] + id,
/// so any later contract write orders after it. Other devices learn it
/// through the snapshot sections. Nothing is shared (private by default).
/// * The podcast id is derived from the canonical feed URL
/// ([`FeedUrl::parse`], falling back to [`FeedUrl::from_user_input`]); rows
/// whose URL cannot be canonicalized are recorded in `legacy_quarantine`
/// and stay in `legacy_podcasts` with their episodes. Rows with the same
/// canonical URL merge into one podcast (the first row's subscription and
/// fetch state win; the episodes of all of them are imported).
/// * Episodes get ids from the contract identity rule over the stored rows.
/// The prototype stored the enclosure URL as `guid` when the item had
/// none: such rows are imported as GUID-less items and keep the stored
/// value in `legacy_guid`. Rows of one GUID-less item seen under several
/// enclosure URLs are collapsed into the newest (the others are
/// quarantined as duplicates). Because stale rows can still change which
/// key is unique, these ids are provisional: the first refresh that lists
/// the item re-keys an episode without progress or sessions to the id the
/// current feed gives (see `refresh.rs`), including its GUID id when the
/// stored value turns out to be a real GUID.
/// * ETag, Last-Modified, check time, last error and the cover cache move to
/// the local fetch state. The prototype has no progress or history, so no
/// progress register or session is created.
fn import_prototype(tx: &Transaction<'_>, with_episodes: bool) -> Result<()> {
let now = now_ms();
let legacy: Vec<LegacyPodcast> = {
let mut statement = tx.prepare(
"SELECT id, feed_url, title, author, description, link, image_url, cover_url,
cover_path, etag, last_modified, added_at, last_checked_at, last_error
FROM legacy_podcasts ORDER BY id",
)?;
statement
.query_map([], |row| {
Ok(LegacyPodcast {
id: row.get(0)?,
feed_url: row.get(1)?,
title: row.get(2)?,
author: row.get(3)?,
description: row.get(4)?,
link: row.get(5)?,
image_url: row.get(6)?,
cover_url: row.get(7)?,
cover_path: row.get(8)?,
etag: row.get(9)?,
last_modified: row.get(10)?,
added_at: row.get(11)?,
last_checked_at: row.get(12)?,
last_error: row.get(13)?,
})
})?
.collect::<rusqlite::Result<Vec<_>>>()?
};
let mut known_urls: Vec<(PodcastId, FeedUrl)> = Vec::new();
for podcast in legacy {
let url = match FeedUrl::parse(&podcast.feed_url)
.or_else(|_| FeedUrl::from_user_input(&podcast.feed_url))
{
Ok(url) => url,
Err(error) => {
quarantine(
tx,
"legacy_podcasts",
podcast.id,
"invalid feed URL",
&error.to_string(),
)?;
continue;
}
};
let podcast_id = resolve_podcast_id(&url, known_urls.iter().map(|(id, url)| (id, url)));
let duplicate = known_urls.iter().any(|(id, _)| *id == podcast_id);
known_urls.push((podcast_id.clone(), url.clone()));
if duplicate {
// Same canonical feed as an earlier row: its episodes join that
// podcast; the first row's subscription and fetch state are kept.
if with_episodes {
import_prototype_episodes(tx, &podcast, &podcast_id, now)?;
}
apply::refresh_seasons(tx, &podcast_id)?;
continue;
}
let meta = PodcastMeta {
feed_url: url.clone(),
title: podcast.title.clone(),
author: podcast.author.clone(),
description: podcast.description.clone(),
link: podcast.link.clone(),
image_url: podcast.image_url.clone(),
}
.bounded();
let added_ms = secs_to_ms(podcast.added_at);
let record = SubscriptionRecord {
podcast_id: podcast_id.clone(),
subscribed: true,
podcast: meta.clone(),
feed_aliases: vec![url.clone()],
generation: 0,
hlc_ms: added_ms,
op_id: format!("{LEGACY_IMPORT_OP_PREFIX}{podcast_id}"),
};
apply::apply_subscription_record(tx, &record, now)?;
// The prototype metadata came from the feed itself.
tx.execute(
"UPDATE podcasts
SET title = ?2, author = ?3, description = ?4, link = ?5, image_url = ?6,
meta_from_feed = 1, created_at_ms = MIN(created_at_ms, ?7)
WHERE podcast_id = ?1 AND meta_from_feed = 0",
params![
podcast_id.as_str(),
meta.title,
meta.author,
meta.description,
meta.link,
meta.image_url,
added_ms
],
)?;
tx.execute(
"INSERT OR IGNORE INTO podcast_fetch_state
(podcast_id, etag, last_modified, checked_at_ms, last_success_at_ms,
next_check_at_ms, last_error, consecutive_failures, cover_url, cover_path)
VALUES (?1, ?2, ?3, ?4, ?5, NULL, ?6, ?7, ?8, ?9)",
params![
podcast_id.as_str(),
podcast.etag,
podcast.last_modified,
podcast.last_checked_at.map(secs_to_ms),
podcast
.last_error
.is_none()
.then(|| podcast.last_checked_at.map(secs_to_ms))
.flatten(),
podcast.last_error,
i64::from(podcast.last_error.is_some()),
podcast.cover_url,
podcast.cover_path,
],
)?;
if with_episodes {
import_prototype_episodes(tx, &podcast, &podcast_id, now)?;
}
apply::refresh_seasons(tx, &podcast_id)?;
}
Ok(())
}
fn import_prototype_episodes(
tx: &Transaction<'_>,
podcast: &LegacyPodcast,
podcast_id: &PodcastId,
now: i64,
) -> Result<()> {
let rows: Vec<LegacyEpisode> = {
let mut statement = tx.prepare(
"SELECT id, guid, title, description, published_at, duration_seconds, audio_url,
mime_type, byte_len, link, image_url, first_seen_at
FROM legacy_podcast_episodes WHERE podcast_id = ?1 ORDER BY id",
)?;
statement
.query_map([podcast.id], |row| {
Ok(LegacyEpisode {
id: row.get(0)?,
guid: row.get(1)?,
title: row.get(2)?,
description: row.get(3)?,
published_at: row.get(4)?,
duration_seconds: row.get(5)?,
audio_url: row.get(6)?,
mime_type: row.get(7)?,
byte_len: row.get(8)?,
link: row.get(9)?,
image_url: row.get(10)?,
first_seen_at: row.get(11)?,
})
})?
.collect::<rusqlite::Result<Vec<_>>>()?
};
let mut usable = Vec::new();
for row in rows {
if !is_http_url(&row.audio_url) {
quarantine(
tx,
"legacy_podcast_episodes",
row.id,
"invalid audio URL",
&row.audio_url,
)?;
continue;
}
usable.push(row);
}
// The prototype used the enclosure URL as GUID when the item had none.
let real_guid = |row: &LegacyEpisode| {
let guid = row.guid.trim();
(!guid.is_empty() && guid != row.audio_url.trim()).then(|| guid.to_string())
};
let usable = collapse_moved_guidless_rows(tx, usable, real_guid)?;
let hints: Vec<EpisodeIdentityHints> = usable
.iter()
.map(|row| EpisodeIdentityHints {
guid: real_guid(row),
enclosure_url: Some(row.audio_url.clone()),
link: row.link.clone(),
published_at_ms: row.published_at.map(secs_to_ms),
title: Some(row.title.clone()),
})
.collect();
let known: Vec<KnownEpisode> = apply::known_episodes(tx, podcast_id)?;
let ids = resolve_episode_ids(podcast_id, podcast.link.as_deref(), &hints, &known);
let mut seen: BTreeMap<EpisodeId, i64> = BTreeMap::new();
for ((row, hint), id) in usable.iter().zip(&hints).zip(ids) {
let Some(episode_id) = id else {
quarantine(
tx,
"legacy_podcast_episodes",
row.id,
"episode cannot be identified",
"no usable identity key or duplicate item",
)?;
continue;
};
if let Some(first) = seen.insert(episode_id.clone(), row.id) {
quarantine(
tx,
"legacy_podcast_episodes",
row.id,
"duplicate episode",
&format!("same identity as legacy episode {first}"),
)?;
continue;
}
let meta = EpisodeMeta {
episode_id,
podcast_id: podcast_id.clone(),
guid: hint.guid.clone(),
title: bounded_title(&row.title),
description: row
.description
.as_deref()
.map(str::trim)
.filter(|text| !text.is_empty())
.map(|text| bounded_text(text, MAX_DESCRIPTION_CHARS)),
season: None,
episode_number: None,
episode_type: EpisodeType::Full,
published_at_ms: row.published_at.map(secs_to_ms),
duration_ms: row
.duration_seconds
.filter(|secs| secs.is_finite() && *secs > 0.0)
.map(|secs| (secs * 1_000.0).round() as i64),
audio_url: row.audio_url.clone(),
mime: row.mime_type.clone().filter(|mime| {
!mime.is_empty() && mime.len() <= 128 && mime.bytes().all(|b| b.is_ascii_graphic())
}),
byte_len: row.byte_len.and_then(|len| u64::try_from(len).ok()),
image_url: row.image_url.clone().filter(|url| is_http_url(url)),
link: row.link.clone().filter(|link| link.len() <= MAX_URL_BYTES),
};
if let Err(error) = meta.validate() {
quarantine(
tx,
"legacy_podcast_episodes",
row.id,
"invalid episode metadata",
&error.to_string(),
)?;
continue;
}
let legacy_guid = hint.guid.is_none().then(|| row.guid.clone());
let exists: bool = tx
.query_row(
"SELECT 1 FROM episodes WHERE episode_id = ?1",
[meta.episode_id.as_str()],
|_| Ok(true),
)
.optional()?
.unwrap_or(false);
if exists {
continue;
}
apply::insert_episode(
tx,
&meta,
EpisodeSource::Feed,
true,
legacy_guid.as_deref(),
secs_to_ms(row.first_seen_at),
now,
)?;
}
Ok(())
}
/// Collapses prototype rows that are one GUID-less item seen under several
/// enclosure URLs.
///
/// The prototype keyed GUID-less items by their enclosure URL, so an item
/// whose audio URL changed (new CDN, tracking prefix) left one row per URL.
/// Those stale rows are not part of the current feed; resolved together with
/// it they would make the item's `<link>` and `(date, title)` look
/// non-unique and give it an id a fresh device never derives. Rows with the
/// same publication second, normalized title and `<link>` (or none) are the
/// same item: the newest (`first_seen_at`, then `id`) is imported, the others
/// are quarantined as duplicates. The link must match too, so distinct
/// episodes that only share a title and second are never merged.
fn collapse_moved_guidless_rows(
tx: &Transaction<'_>,
rows: Vec<LegacyEpisode>,
real_guid: impl Fn(&LegacyEpisode) -> Option<String>,
) -> Result<Vec<LegacyEpisode>> {
type ItemKey = (i64, String, Option<String>);
let key = |row: &LegacyEpisode| -> Option<ItemKey> {
if real_guid(row).is_some() {
return None;
}
let title = normalize_episode_title(&row.title);
let link = row
.link
.as_deref()
.map(str::trim)
.filter(|link| !link.is_empty())
.map(str::to_string);
(!title.is_empty()).then_some((row.published_at?, title, link))
};
let mut newest: BTreeMap<ItemKey, (i64, i64)> = BTreeMap::new();
for row in &rows {
if let Some(key) = key(row) {
let candidate = (row.first_seen_at, row.id);
newest
.entry(key)
.and_modify(|best| *best = (*best).max(candidate))
.or_insert(candidate);
}
}
let mut kept = Vec::with_capacity(rows.len());
for row in rows {
let keeper = key(&row).and_then(|key| newest.get(&key).map(|(_, id)| *id));
match keeper {
Some(keeper) if keeper != row.id => quarantine(
tx,
"legacy_podcast_episodes",
row.id,
"duplicate episode",
&format!(
"older enclosure URL of the GUID-less item imported from legacy episode {keeper}"
),
)?,
_ => kept.push(row),
}
}
Ok(kept)
}
fn bounded_title(title: &str) -> String {
let title = title.trim();
if title.is_empty() {
"Untitled episode".to_string()
} else {
bounded_text(title, MAX_TITLE_CHARS)
}
}
File diff suppressed because it is too large. Load diff