Document the SQLite podcast store
PODCASTS.md gains the "SQLite reference store" section (migrations, prototype import, tables, invariants); ARCHITECTURE.md, the furumi-library README and the changelog describe PodcastStore and TrackItem.podcast. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
e22c470ea7
commit
49ae6423f6
4 files changed
+170
-4
No files matched your search
+16
-1
@@ -12,6 +12,7 @@ application
|
||||
furumi-library
|
||||
├── local catalog, import, migrations
|
||||
├── playlists, likes, history, similarity persistence
|
||||
├── podcast store (podcasts.sqlite3)
|
||||
│
|
||||
music-dht
|
||||
├── Kademlia routing and iterative lookup
|
||||
@@ -41,9 +42,23 @@ metadata import, artwork resolution, playlists, likes, listening history, and
|
||||
similarity index storage. Connections enable foreign keys, WAL, and a bounded
|
||||
busy timeout so independently running clients can safely share the database.
|
||||
|
||||
Podcasts use a separate store, `furumi_library::podcasts::PodcastStore`, in
|
||||
its own `podcasts.sqlite3` file next to the library database. It implements
|
||||
the podcast contract of `music_dht::podcast` for SQLite clients: the episode
|
||||
catalog refreshed from RSS, the subscription, share and progress registers,
|
||||
grow-only listening sessions, feed aliases and local-only fetch state.
|
||||
Local actions, received operations and snapshot sections share one apply
|
||||
path built on the `music-dht` materializers and merge functions, so the store
|
||||
converges independently of delivery order. Migrations follow
|
||||
`PRAGMA user_version`; the prototype podcast database is imported and its
|
||||
tables are kept as `legacy_*`; newer or foreign files are refused unchanged.
|
||||
`TrackItem.podcast` marks episode queue items, which music-only code must
|
||||
skip. See the "SQLite reference store" section of [PODCASTS.md](PODCASTS.md).
|
||||
|
||||
Filesystem scanning schedules, playback, federation lifecycle, and UI state
|
||||
remain application responsibilities. Database and metadata work must stay off
|
||||
interactive and async reactor threads.
|
||||
interactive and async reactor threads. RSS fetching and refresh scheduling are
|
||||
application work as well.
|
||||
|
||||
## Transport: federation-net
|
||||
|
||||
|
||||
@@ -18,6 +18,20 @@
|
||||
Jam redaction. New struct fields and enum variants break exhaustive
|
||||
construction and matching in clients.
|
||||
- `furumi-library 0.3.0`: follows `music-dht 0.6.0`, whose types it re-exposes.
|
||||
- `furumi-library 0.3.0`: new `furumi_library::podcasts::PodcastStore`, the
|
||||
SQLite reference store of the podcast contract in its own
|
||||
`podcasts.sqlite3`: `user_version` migrations that import the prototype
|
||||
podcast database (tables kept as `legacy_*`, unimportable rows in
|
||||
`legacy_quarantine`) and refuse newer or foreign files unchanged; podcasts,
|
||||
feed aliases, seasons, episodes, subscription/share/progress registers,
|
||||
listening sessions and local-only fetch state; local actions that return
|
||||
the `podcast_*` operations for the client's sync log; `apply_remote_op`,
|
||||
bounded snapshot export and merge; RSS refresh ingestion with stable
|
||||
episode ids; UI queries. Enables the `music-dht` `podcast-feed` feature.
|
||||
- `furumi-library 0.3.0` (breaking): `TrackItem.podcast:
|
||||
Option<PodcastEpisodeRef>` and `TrackItem::podcast_episode`/`is_podcast`.
|
||||
Exhaustive `TrackItem` literals in clients need the new field; music-only
|
||||
code must skip podcast items. The library database is unchanged.
|
||||
- See `PODCASTS.md` for the normative contract.
|
||||
|
||||
## v0.4.1 — 2026-09-02
|
||||
|
||||
+96
-1
@@ -19,7 +19,7 @@ of this version of the contract; it will be specified with the overlay.
|
||||
| --- | --- |
|
||||
| `music_dht::podcast` | ids, URL canonicalization, no-GUID rule, metadata DTOs and bounds, merge functions, RSS/Atom parser (feature `podcast-feed`) |
|
||||
| `music_dht::device_sync` | sync operation kinds, snapshot sections, `PlaybackSource`, version gating, Jam redaction |
|
||||
| Client storage (`furumi-library`, web PostgreSQL) | persistence and migrations implementing the rules below |
|
||||
| Client storage (`furumi_library::podcasts::PodcastStore`, web PostgreSQL) | persistence and migrations implementing the rules below (SQLite reference: see "SQLite reference store") |
|
||||
| Client runtime | HTTP fetching, conditional requests, scheduling, audio buffering and decoding, UI |
|
||||
|
||||
Audio is always fetched from the publisher. It is never imported into the
|
||||
@@ -511,6 +511,101 @@ extracts `itunes:season`, `itunes:episode`, `itunes:episodeType`,
|
||||
Links are kept verbatim (trimmed only) because they are identity material.
|
||||
Text cleanup is linear in the input size. It performs no I/O.
|
||||
|
||||
## SQLite reference store
|
||||
|
||||
`furumi_library::podcasts::PodcastStore` (`furumi-library` 0.3) implements
|
||||
this contract for SQLite clients. Other stores (the web client's PostgreSQL)
|
||||
must keep the same invariants.
|
||||
|
||||
### File and migrations
|
||||
|
||||
* Own file, conventionally `podcasts.sqlite3` next to `library.db`; the caller
|
||||
passes the path. WAL, foreign keys, 5 s busy timeout.
|
||||
* `PRAGMA user_version` drives migrations. Each runs in a `BEGIN IMMEDIATE`
|
||||
transaction that re-reads the version, so concurrent openers and
|
||||
interrupted upgrades are safe and reopening is idempotent.
|
||||
|
||||
| Version | Content |
|
||||
| --- | --- |
|
||||
| 0 | empty file |
|
||||
| 1 | prototype schema of the original podcast PR (`podcasts`, `podcast_episodes`), verbatim |
|
||||
| 2 | the model below; the prototype tables are renamed and imported |
|
||||
|
||||
* Before writing anything (including the WAL switch), the store refuses a
|
||||
file with a `user_version` above its own, and a file whose tables are not
|
||||
the expected ones for its version (for example `library.db` passed by
|
||||
mistake). The file is left byte-for-byte unchanged.
|
||||
* Prototype import (migration 2). Tables are renamed `legacy_podcasts` and
|
||||
`legacy_podcast_episodes`, never dropped.
|
||||
* Each feed becomes a podcast with `podcast_id` from the canonical feed URL
|
||||
(`FeedUrl::parse`, then `FeedUrl::from_user_input`) and a subscription
|
||||
register `subscribed = true`, generation 0, `hlc_ms` = `added_at`,
|
||||
`op_id` = `legacy-import:<podcast_id>`, not shared. Every contract write
|
||||
orders after it; other devices learn it through the snapshot sections.
|
||||
Rows with the same canonical URL merge (first row wins).
|
||||
* URLs that cannot be canonicalized, episodes with a non-`http(s)` audio URL,
|
||||
invalid metadata or no identity are listed in `legacy_quarantine`; the
|
||||
rows stay in the `legacy_*` tables.
|
||||
* Episode ids come from `resolve_episode_ids` over the stored items. The
|
||||
prototype stored the enclosure URL as `guid` when an item had none: such
|
||||
rows are imported as GUID-less items and keep the value in
|
||||
`episodes.legacy_guid`. When a later refresh shows that value as a real
|
||||
GUID, the episode is re-keyed to its GUID id if it has no progress and no
|
||||
session (otherwise the GUID item becomes a new episode).
|
||||
* ETag, Last-Modified, check time, last error and the cover cache move to the
|
||||
fetch state. The prototype stores no progress or history, so none is
|
||||
created.
|
||||
|
||||
### Tables (version 2)
|
||||
|
||||
| Table | Key | Content | Synced |
|
||||
| --- | --- | --- | --- |
|
||||
| `podcasts` | `podcast_id` | local channel metadata, `feed_url` = URL this device fetches, `meta_from_feed` | catalog, no |
|
||||
| `podcast_feed_aliases` | `(podcast_id, feed_url)` | bounded grow-only alias set (`merge_feed_aliases`) | yes, inside subscription records and `podcast_feed_aliases_added` |
|
||||
| `episodes` | `episode_id` | `EpisodeMeta` columns (season, number, type, published, duration, audio URL, MIME, length, artwork, link), `source` (`feed`/`sync`), `in_feed`, `legacy_guid` | catalog, no |
|
||||
| `podcast_seasons` | `(podcast_id, season)` | derived from `episodes`: episode count, first/last publication; `season` NULL groups episodes without a season | no |
|
||||
| `podcast_subscriptions` | `podcast_id` | subscription register: `subscribed`, `PodcastMeta` JSON, `generation`, `hlc_ms`, `op_id` | yes |
|
||||
| `podcast_shares` | `podcast_id` | share register | yes |
|
||||
| `episode_progress` | `episode_id` | progress register (`ProgressRecord`) | yes |
|
||||
| `podcast_listen_sessions` | `session_id` | grow-only set of `ListenSession` | yes |
|
||||
| `podcast_fetch_state` | `podcast_id` | ETag, Last-Modified, checked / last success / next check times, last error, consecutive failures, cover cache | **never** |
|
||||
| `legacy_quarantine` | `(legacy_table, legacy_id)` | prototype rows that could not be imported | no |
|
||||
|
||||
### Invariants
|
||||
|
||||
* Registers and sessions have no foreign keys to the catalog: operations for
|
||||
podcasts or episodes the device has not fetched are stored as they arrive,
|
||||
in any order. An operation's `PodcastEpisodeRef` adds a `source = 'sync'`
|
||||
catalog row only when the episode is unknown; feed metadata always wins.
|
||||
* One apply path. Local actions validate the payload, draw a stamp from the
|
||||
client's sync log, build the `SyncOpWire`, and apply it like a received
|
||||
operation: `SyncOpWire::podcast_*` materializers, then `merge_*`. The
|
||||
returned operation must be appended with exactly that stamp. Snapshot
|
||||
records go through the same functions after validation. The result is
|
||||
therefore commutative, associative and idempotent, and a snapshot never
|
||||
replaces newer local state.
|
||||
* Invalid operations (`validate_podcast`, a generation above `i64::MAX`, a
|
||||
session whose identity differs from the stored one) are skipped and
|
||||
reported as rejected, so the client can still acknowledge them.
|
||||
* Only `subscribe`/`unsubscribe` write the subscription register and only
|
||||
`set_shared`/`unsubscribe` the share register, with
|
||||
`next_podcast_generation`. No-op actions draw no stamp and emit nothing.
|
||||
`unsubscribe` emits `subscribed = false` and `shared = false` and keeps
|
||||
progress, sessions and the catalog.
|
||||
* Explicit progress actions use `explicit_generation(max(local, carried))`;
|
||||
checkpoints keep the session generation and are refused (`Superseded`)
|
||||
once the register holds a greater one, or after the session completed.
|
||||
* RSS refreshes update the catalog (channel metadata, episode metadata and
|
||||
audio URLs, `in_feed`) and the alias set only; episodes missing from the
|
||||
feed are kept. Permanent redirects and `itunes:new-feed-url` become the
|
||||
fetch URL and aliases, and emit `podcast_feed_aliases_added` only when the
|
||||
stored alias set changes.
|
||||
* Snapshot export reads each section newest first up to its count, then
|
||||
applies `bound_podcast_state_to(budget)`. Fetch state and the catalog
|
||||
(descriptions, episodes) are never exported.
|
||||
* Feed URLs resolve through `resolve_podcast_id` over the alias set, the
|
||||
catalog fetch URL and the register's `podcast.feed_url`.
|
||||
|
||||
## Versioning
|
||||
|
||||
* `music-dht` 0.6.0 and `furumi-library` 0.3.0: new public types, enum
|
||||
|
||||
@@ -21,5 +21,47 @@ library used by every client. Connections use WAL, foreign keys, and a busy
|
||||
timeout so independently running clients can safely share the database.
|
||||
|
||||
Version 0.3 follows `music-dht` 0.6, whose device-sync and podcast contract
|
||||
types it re-exposes. Podcast persistence is not part of this release; it will
|
||||
implement the rules of the workspace `PODCASTS.md`.
|
||||
types it re-exposes.
|
||||
|
||||
## Podcasts
|
||||
|
||||
`furumi_library::podcasts::PodcastStore` is the SQLite reference store of the
|
||||
podcast contract (workspace `PODCASTS.md`, section "SQLite reference store").
|
||||
It lives in its own file, conventionally `podcasts.sqlite3`
|
||||
(`PODCAST_DB_FILE`) next to `library.db`; the caller passes the path.
|
||||
|
||||
- **Migrations** run on `PRAGMA user_version`, each in its own transaction.
|
||||
A database of the original podcast prototype (`user_version` 1) is imported:
|
||||
its tables are renamed `legacy_podcasts` / `legacy_podcast_episodes` (never
|
||||
dropped), feeds become subscriptions with contract ids, rows that cannot be
|
||||
imported are listed in `legacy_quarantine`. A database from a newer Furumi,
|
||||
or a file that is not a podcast database, is refused without being touched.
|
||||
- **Local actions** (`subscribe`, `unsubscribe`, `set_shared`,
|
||||
`start_session`, `checkpoint`, `seek`, `restart`, `complete_session`,
|
||||
`mark_played`, `mark_unplayed`, `record_listen_session`) draw stamps from a
|
||||
`StampSource` (usually the client's sync log), apply the operation and
|
||||
return `PodcastLocalOp`s to append to the log with exactly those stamps.
|
||||
- **Sync**: `apply_remote_op` applies the five `podcast_*` operations,
|
||||
`fill_snapshot` / `export_snapshot` produce the bounded snapshot sections
|
||||
and `apply_snapshot` merges them. Everything goes through the `music-dht`
|
||||
materializers and merge functions, so the result does not depend on
|
||||
delivery order and never loses newer local state.
|
||||
- **RSS**: `apply_feed` ingests a document parsed by
|
||||
`music_dht::podcast::feed` (the crate enables the `podcast-feed` feature):
|
||||
channel metadata, episodes with stable ids and current audio URLs, feed
|
||||
aliases from permanent redirects and `itunes:new-feed-url`. It never
|
||||
touches subscriptions, progress or history. HTTP, conditional requests and
|
||||
scheduling stay in the client; `record_fetch_*`, `fetch_state` and
|
||||
`podcasts_due_for_refresh` keep the local-only fetch state, which is never
|
||||
synced.
|
||||
- **Queries**: `list_podcasts`, `seasons`, `episodes` (season, played-state
|
||||
and text filters, date or episode-number order), `resume_position_ms`,
|
||||
`episode_ref` / `episode_track` for queueing, `listen_history`.
|
||||
|
||||
The store is synchronous: methods take `&self` and lock one connection, so
|
||||
`Arc<PodcastStore>` can be shared; run calls on a blocking pool.
|
||||
|
||||
`TrackItem.podcast` marks a queue item as a podcast episode
|
||||
(`TrackItem::podcast_episode`). Music-only code (likes, playlists, history,
|
||||
content ids, federation downloads) must check `TrackItem::is_podcast`; the
|
||||
music library never stores episodes.
|
||||
Reference in new issue
Block a user