diff --git a/CHANGELOG.md b/CHANGELOG.md index c02cfba..3f351db 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,89 @@ # Changelog +## v0.14.0 — 2026-11-19 + +- Podcasts in the web player: a Podcasts section to add RSS feeds, browse + podcasts with unplayed counts and fetch errors, seasons, episode search, + filters and paging, refresh, unsubscribe (progress and history are kept) + and subscribe again, with Resume, Start over, Mark played/unplayed and the + recent listening history. +- The web player plays podcast episodes from the publisher (or the optional + proxy), resumes them from the synchronized position and saves progress in + the background: every 20 s and on pause, seek, item change, page hide and + handoff, with listening sessions and automatic completion by the shared + rule. A source that cannot seek to the saved position is shown as such + instead of playing from elsewhere. Handoffs to and from other devices keep + the episode, position and pause state; outgoing states carry the newest + progress generation. Play (also after a reload or a pause) continues from + the newest saved position of any device; a player that keeps listening + after another device wrote the episode saves again as a new session, and + one that loses the playback role records what it listened. Previous goes + back 15 s within an episode instead of resetting it. +- New endpoints under `/api/podcasts` (`/feeds`, `/feeds/{id}`, + `/feeds/{id}/episodes`, subscribe/unsubscribe, `/episodes/{id}/playback`, + `/progress`, `/restart`, `/played`, `/unplayed`). Request bodies must be + `application/json`. +- Jam guests still see the host's episodes only as private items and cannot + add episodes from the Podcasts section. +- Store podcast subscriptions, feed aliases, episodes, seasons, progress and + listening sessions of each account in PostgreSQL (`src/podcasts/`), with + the shared `music-dht` podcast rules. Received podcast operations and + snapshot sections are applied to this store, and snapshots are built from + it; the wire behavior towards other devices is unchanged. Migrations + `m_0056` and `m_0057` import the state that 0.13 relayed + (`furumusic__fed_podcast_relay`, kept but unused). +- Refresh subscribed podcast feeds in the background (`podcast_refresh` job, + every five minutes; each feed about hourly) with conditional requests, + permanent redirects learned as feed aliases, size and time limits and an + exponential backoff up to a day on errors. Accounts subscribed to the same + feed share one request; accounts take turns within a run of up to 200 + feeds. `POST /api/podcasts/{podcast_id}/refresh` refreshes one feed on + demand. Only metadata is fetched, never audio. An account can have up to + 1000 subscriptions and two user-started feed downloads at once. +- Optional podcast audio proxy (`podcast_audio_proxy_enabled`, off by + default; browsers stream from the publisher otherwise): + `GET /api/podcasts/episodes/{episode_id}/audio` streams an episode of the + signed-in account from its publisher with Range pass-through, audio only + (a generic binary type needs an audio file extension or audio content; the + feed's enclosure type is not trusted), size/time limits and four streams + per account. `GET /api/podcasts/episodes/{episode_id}/source` + tells the player which URLs to use. +- Feed and proxy requests only reach public addresses on ports 80, 443, 8080 + and 8443: names are resolved and checked once per connection (no DNS + rebinding), IP literals and every redirect hop are checked, and no cookies, + credentials or system proxies are used. Users see only the kind of a + failure; details go to the logs without URLs. A host name that resolves to + a blocked address gets the same answer as one that does not resolve. +- The browser can add at most 10 000 episodes outside the account's feeds + (handed-off queue items) through the playback endpoint (HTTP 422 beyond). + +## v0.13.0 — 2026-11-19 + +- Use the shared `music-dht` 0.6 device-sync types and announce device-sync + protocol 4. Podcast operations and snapshot sections from other devices are + kept and relayed; devices below protocol 4 never receive them. +- Keep podcast episodes and other non-library queue items through the web + player, with their explicit source. The player labels and skips them; the + typed queue is stored next to `queue_json` (migrations `m_0054`, `m_0055`). + A handoff through the web player keeps the position of the current episode. +- Refuse a playback transfer of a queue with podcast episodes to a device that + needs an update, and ignore its playback commands while the queue has them. +- Store the protocol version each device reports itself instead of this + server's version; invites carry the inviter's version. +- An episode that the web player selects without knowing its position (Next, + auto-advance, a click) resumes from its synchronized progress on the next + device instead of restarting at 0. Repeat all and shuffle continue past + episodes; Prev and seeking work on them. +- Jam guests see podcast episodes of the host as private items, without their + duration, and cannot add, remove or clear them; the host browser enforces + this on its own queue. Guests do not keep the placeholders after the Jam. +- Metadata Repair can still undo deletions recorded before the typed queue + column was added. +- Rolling back to 0.12 requires removing the relayed podcast operations first; + see "Rolling back to 0.12" in `docs/administration.md`. After a rollback, the + typed queue is used again only if 0.12 left the queue and its current track + unchanged. + ## v0.10.6 — 2026-09-02 - Recover federation automatically after sleep, prolonged idle, or a degraded rendezvous transport. diff --git a/Cargo.lock b/Cargo.lock index 68a93ca..8e8043f 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1843,8 +1843,7 @@ dependencies = [ [[package]] name = "federation-net" version = "0.3.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "01e5c2e6e5af020dc202d4924125ef57719aa21f8d2567e2d943914386bc8980" +source = "git+https://gt.hexor.cy/Sanapach/frid.git?branch=podcasts%2Fcontract#65d6ca2f14a1324dc129b90694da83ceb55c60dc" dependencies = [ "blake3", "data-encoding", @@ -1988,7 +1987,7 @@ checksum = "e6d5a32815ae3f33302d95fdcb2ce17862f8c65363dcfd29360480ba1001fc9c" [[package]] name = "furumusic" -version = "0.12.4" +version = "0.14.0" dependencies = [ "anyhow", "async-stream", @@ -3900,16 +3899,17 @@ dependencies = [ [[package]] name = "music-dht" -version = "0.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4c3f75d49d3a742a6777a1c4cc53f397399dfc9ddb70034c57aeb44d75d2ee4f" +version = "0.6.0" +source = "git+https://gt.hexor.cy/Sanapach/frid.git?branch=podcasts%2Fcontract#65d6ca2f14a1324dc129b90694da83ceb55c60dc" dependencies = [ "async-trait", "blake3", "data-encoding", + "encoding_rs", "federation-net", "futures", "postcard", + "quick-xml 0.41.0", "rand 0.9.5", "rusqlite", "serde", @@ -3918,6 +3918,7 @@ dependencies = [ "tokio", "tracing", "unicode-normalization", + "url", ] [[package]] diff --git a/Cargo.toml b/Cargo.toml index bbc8624..6d0c2c5 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "furumusic" -version = "0.12.4" +version = "0.14.0" edition = "2024" description = "Reusable web-app boilerplate: auth, OIDC/SSO, admin panel, user management, i18n, PostgreSQL" @@ -56,4 +56,10 @@ uuid = "1" librqbit = { version = "8.1.1", features = ["disable-upload"] } # P2P federation: publishes the library into a shared DHT and serves audio / # catalogs to furumi peers (TUI clients) over the frid stack. -music-dht = "0.5.0" +# `podcast-feed`: the shared RSS/Atom parser, whose identity hints are part of +# the podcast episode id contract. +music-dht = { version = "0.6.0", features = ["podcast-feed"] } + +[patch.crates-io] +# Unpublished frid (MR-F1); remove once music-dht 0.6.0 is on crates.io. +music-dht = { git = "https://gt.hexor.cy/Sanapach/frid.git", branch = "podcasts/contract" } diff --git a/README.md b/README.md index d5ecd5d..ccbdbc8 100644 --- a/README.md +++ b/README.md @@ -20,6 +20,10 @@ the Furumi federated network. without creating an account. - **Welcome your people.** Use password login or SSO. Instance admins can approve new users without needing access to the identity provider. +- **Follow podcasts.** Subscribe to RSS feeds, browse seasons and episodes, and + pick up where you stopped, also on your other Furumi players. Episodes stream + from the publisher and are never imported into the library; subscriptions + and listening progress stay private to your own devices. - **Discover more.** Find similar tracks and connect Last.fm for scrobbling. - **Connect your players.** Pair trusted Furumi devices to synchronize playlists, likes, playback state, and playback controls. diff --git a/docs/administration.md b/docs/administration.md index 0b13bc8..90e86c1 100644 --- a/docs/administration.md +++ b/docs/administration.md @@ -71,6 +71,146 @@ Trusted-device pairing is separate from public catalog discovery. It connects a user's own Furumi players so likes, playlists, playback state, and control can move between approved devices. +Queues synchronized from other Furumi players can contain podcast episodes; +the web player plays them (see [Podcasts](#podcasts)) and keeps them in order +with their position. A device that does not support podcasts yet (device-sync +protocol below 4, for example an older terminal player) cannot receive such a +queue: selecting it as the playback device shows that it needs an update, and +the episodes stay in the queue. Jam guests see podcast episodes of the host +only as private items they cannot play, add, remove or clear. + +## Podcasts + +**Podcasts** in the player sidebar lists your subscriptions with the number +of unplayed episodes and the result of the last feed check. Paste an RSS or +Atom feed URL to subscribe: the server loads the feed first and shows why when +it cannot (not a feed, not reachable, or an address that is not public). A +podcast page shows the description, the seasons (episodes without a season +form their own group), a search over episode titles and descriptions, filters +for unplayed, started and played episodes, and the recent listening history. + +- **Play / Resume** continues an episode where you stopped, on whichever of + your Furumi players last saved it. **Start over** plays it from the + beginning; **Mark played** and **Mark unplayed** change its status without + playing (marking does not count as a listen). An episode counts as listened + when playback reaches its end, or its last 30 seconds or last 5 %, whichever + is shorter. +- The web player saves the position about every 20 seconds and when you + pause, seek, switch items, close the page or hand playback to another + device, without interrupting the audio. Reloading the page, or pressing + Play after a pause while another player went on with the episode, + continues from the newest saved position. Previous on an episode goes + back 15 seconds instead of to the start. Marking the episode that is + playing played or unplayed pauses it. +- Handing playback to another device keeps the episode, its position and + whether it was paused. A later save of an older listening session never + moves the position back after you seek or start over elsewhere. +- **Unsubscribe** stops checking the feed and keeps your progress and history; + subscribing again continues where you stopped. **Refresh** checks the feed + now (at most every 30 seconds). + +Privacy: a new subscription is private. Subscriptions, progress and listening +history are synchronized only between your own paired Furumi players (trusted +devices), never published to the federation, and Jam guests cannot see your +episodes. Publishing a podcast for discovery is not available in the web +player yet. Episodes are not stored on the server or shared with peers: the +browser streams the audio and loads the artwork directly from the publisher, +so the publisher sees your browser's address. With the audio proxy (below) +the publisher sees the server instead. + +### Podcast feeds and audio + +The `podcast_refresh` job (every five minutes, listed under Jobs) checks the +feeds of subscribed podcasts: each feed about once an hour with conditional +requests, failing feeds less and less often (up to once a day). It downloads +feed metadata only, never episode audio. A failing feed keeps its known +episodes. Accounts subscribed to the same feed share one request. Each run +checks at most 200 feeds (2400 an hour), taking accounts in turns: first the +most overdue feed of every account, then the second, and so on. An account +with many or slow feeds therefore does not delay the others, but can use the +capacity nobody else needs. + +An account can add up to 1000 subscriptions and, from the browser, up to +10 000 episodes outside its feeds (handed-off queue items; episodes synced +from its own devices are not limited); beyond that the request is refused +with HTTP 422. It can have two feed downloads ("add RSS", Refresh) and four +proxied audio streams in progress at once (16 and 64 for the whole server); +beyond that the request is refused with HTTP 429. + +The server therefore makes outgoing HTTP(S) requests to podcast publishers. +It only connects to public addresses on the web ports 80, 443, 8080 and 8443: +feed or audio URLs that point to loopback, private, link-local, CGNAT or cloud +metadata addresses, directly, through DNS or through a redirect, or to any +other port, are refused. System proxy settings are not used for these +requests. Users only see the kind of failure (address not allowed, could not +connect, HTTP status, not a feed); the details, without the URL, are in the +server log and, for scheduled refreshes, in the job log with the podcast id. +"Address not allowed" is only said for what the URL itself shows (scheme, +credentials, port, an IP address). A host name that resolves to a blocked +address gets the same answer as a name that does not resolve or a server that +does not answer ("could not connect"), so users cannot find out which names +exist in the server's internal DNS. + +Episode audio is streamed by the browser directly from the publisher. Some +publishers cannot be played that way (plain HTTP while this server uses +HTTPS, or a server the browser refuses). For them, enable **Settings → +Podcasts → Podcast audio proxy** (`FURU_PODCAST_AUDIO_PROXY_ENABLED=true`): +the player then switches to the proxy when the direct URL fails or is plain +HTTP on an HTTPS page, and the server fetches the audio from the publisher +and passes it through without storing it. This uses the server's bandwidth. + +Security model of the proxy: + +- It takes an episode id, never a URL. The URL comes from the signed-in + account's own podcast catalog: its feeds, and episode references sent by + the account's own clients (its paired devices, or its browser when it plays + a handed-off episode the catalog does not know yet). A signed-in user + controls both (anyone can subscribe to a feed they host), so any signed-in + user can make the server fetch audio from any public URL, as with feed + refreshes. +- What limits that: the same public-address and port checks as for feeds + apply to every connection and redirect, and only audio is passed through. + The publisher's `Content-Type` must be audio; for a generic binary type + (`application/octet-stream` or none), the URL must have an audio file + extension or the file must start like an audio file (MP3, AAC, Ogg, FLAC, + WAV, M4A). The enclosure type written in the feed is never trusted. No + HTML, no video. Requests carry a single byte range, are limited to 1 GiB + and 4 hours each and to four streams per account (64 for the server), and + no cookies or credentials are sent. +- Responses are not cached or stored, and are marked `no-store`, `nosniff` + and sandboxed. + +Leave it off unless your users need it. + +### Upgrading from 0.13 + +Release 0.14 adds the podcast UI and playback described above; no setting is +needed besides the optional proxy. + +Podcast state synchronized from other Furumi players is kept in per-account +podcast tables (`furumusic__podcast*`). The upgrade imports what 0.13 kept in +`furumusic__fed_podcast_relay`; that table is no longer used and can be +dropped by a later release. Release 0.13 does not read the new tables: after +a rollback it would serve the relay table as it was at the upgrade, and +devices would send their newer podcast state again in their snapshots. + +### Rolling back to 0.12 + +Release 0.13 relays podcast synchronization operations from other Furumi +players. Release 0.12 cannot read them: device synchronization would fail for +every device that has not received them yet. Before you start a 0.12 release +on a database that 0.13 used, stop the server and remove those operations: + +```sql +DELETE FROM furumusic__fed_sync_ops WHERE kind LIKE 'podcast\_%'; +``` + +Devices that support podcasts keep their own podcast state. The relay table +`furumusic__fed_podcast_relay` and the typed queue column can stay: 0.12 +ignores them, and after the next upgrade 0.13 returns the relayed podcast state +in its snapshots again and uses the typed queue only while it matches the +queue, current track and position that 0.12 saved. + ## Admin statistics The only admin interface is `/admin/v2`, also reachable from `/admin/`. diff --git a/docs/development.md b/docs/development.md index bad10bc..fba2ab1 100644 --- a/docs/development.md +++ b/docs/development.md @@ -35,6 +35,183 @@ source code, templates or prompts reuse the dependency layer when its recipe and toolchain are unchanged. Both dependency and application builds use `--locked`. Update the pinned build tools in `Dockerfile`; changing them rebuilds the cache. +## Connected devices and podcast queue items + +Trusted-device sync uses the shared `music_dht::device_sync` types (no local +copies) and announces protocol version 4. Each account is one virtual device: + +- podcast operations stay in the operation log and are relayed to v4 devices + only. They are also applied to the account's podcast store + (`src/podcasts/`, `PodcastStore`), which also provides the podcast snapshot + sections; +- queue items that are not library tracks become external browser items + (`source` plus the verbatim `wire_track`), so podcast episodes and Jam + placeholders survive peer → server → browser → server → peer unchanged; +- `furumusic__playback_state.queue_items_json` stores the typed queue next to + `queue_json`. `queue_json` (library ids) is still written for rollback; the + typed value is used only while the legacy columns it recorded + (`queue_json`, `current_track_id`, `queue_position`) match the row. Jam + placeholders are never stored in a user's own queue. +- a browser that selects an item it cannot play (a Jam placeholder), or an + episode whose resume position the server has not sent yet, reports + `position_unknown`, and the server sends the resume position of the + episode's progress register instead of 0. Outgoing playback states raise + each episode's `progress_generation` to the register generation, so the + device that takes over writes above it. Jam participant commands carry + `jam_participant`; the host browser checks them against its own queue. + +Rolling back to 0.12 needs the podcast operations removed from the operation +log first; see "Rolling back to 0.12" in `administration.md`. + +### Podcast store + +`PodcastStore` implements the podcast contract of `music_dht::podcast` +(Frid `PODCASTS.md`) on PostgreSQL, with the invariants of the SQLite +reference store in `furumi-library`; every table has a `user_id` key. + +| Table | Content | Synced | +| --- | --- | --- | +| `furumusic__podcast` | channel metadata, fetch URL | no (catalog) | +| `furumusic__podcast_feed_alias` | bounded grow-only alias set | in subscription records and `podcast_feed_aliases_added` | +| `furumusic__podcast_episode` | `EpisodeMeta`, `source` `feed`/`sync`, `in_feed`, cached search text | no (catalog) | +| `furumusic__podcast_season` | view over the episodes | no | +| `furumusic__podcast_subscription`, `_share`, `_progress` | registers | yes | +| `furumusic__podcast_listen_session` | grow-only set of sessions | yes | +| `furumusic__podcast_fetch_state` | ETag, Last-Modified, check times, errors, cover cache | never | + +- One apply path: received operations, snapshot records and local actions go + through the `SyncOpWire` materializers and the `merge_*` functions of + `music_dht::podcast`. Each write transaction holds a per-account advisory + lock. +- Local actions (`subscribe`, `unsubscribe`, `set_shared`, `start_session`, + `checkpoint`, `seek`, `complete_if_current`, `restart`, `mark_played`, + `mark_unplayed`, `record_listen_session`) append their operation to the + account's sync log with the next stamp (`devices::append_local_op`: `seq`, + `hlc_ms`, `op_id = :`) and apply exactly that operation in + the same transaction. +- `apply_feed` resolves episode ids with the shared no-GUID rule against the + stored episodes, including those only known from synced references. It + updates the catalog and the alias set, never a register, progress or a + session. +- Snapshots export each section newest first up to its count and bound it + with `bound_podcast_state_to`. The catalog and the fetch state are never + exported. + +Migrations `m_0056_create_podcast_store` and `m_0057_import_podcast_relay` +create the tables and import what 0.13.0 kept in `furumusic__fed_podcast_relay` +(registers as contract JSON, already merged) plus the catalog rows of the +episode references in the operation log. The relay table is no longer written +or read; it stays until a later cleanup release. + +### Feed refresh and audio delivery + +The runtime around the store lives next to it in `src/podcasts/`: + +- `net.rs` — `SafeHttp`, the only HTTP client for podcast feeds and audio. + It accepts `http`/`https` URLs without userinfo, resolves names with + `GuardedResolver` and refuses the whole answer when any address is not + public (loopback, RFC 1918, CGNAT, link-local, cloud metadata, unique local, + multicast, unspecified, documentation, reserved; IPv4-mapped, NAT64 and + 6to4 addresses are judged by their IPv4 part). The connector connects to + exactly the checked addresses, so a second DNS answer cannot be swapped in. + IP literals are checked before connecting, redirects are followed manually + (each hop checked, at most 5), and system proxies, cookies and `Referer` + are disabled. `GuardError::is_refusal` is true only for what the URL + shows (scheme, credentials, port, a blocked IP literal); a name that + resolves to a blocked address is reported to users like a transport + failure, so the answers do not reveal internal DNS names. +- `fetch.rs` — `FeedFetcher`: conditional GET (`If-None-Match`, + `If-Modified-Since`), `Accept-Encoding: identity`, 32 MiB and 60 s limits, + `Retry-After` on 429/503, the shared `music_dht` parser. Only a chain of + permanent redirects (301/308) moves a feed. +- `runner.rs` — scheduling: `podcasts_due_for_refresh`, one request per feed + URL for all accounts subscribed to it (conditional only when they share the + same validators), `apply_feed`, then the fetch state. Healthy feeds are + checked hourly plus a stable spread of up to 10 minutes; failures back off + 15 min × 2ⁿ⁻¹ up to a day (a longer `Retry-After` wins). `refresh_now` is + the manual refresh: subscribed podcasts only, at most every 30 s. + `jobs/podcast_refresh.rs` runs `refresh_due` every five minutes (200 + podcasts taken in turns across accounts without a per-account cap, 4 + requests in flight, one run at a time via an advisory lock). +- `proxy.rs` and `http.rs` — the optional audio proxy and the endpoints under + `/api/podcasts`. The proxy takes an episode id, loads the URL from the + account's catalog (`PodcastStore::audio_source`), forwards a single + sanitized byte `Range`, passes 200/206/416 through, accepts audio content + types only (a generic binary type needs an audio URL extension or a file + that starts like audio, `sniff_audio`: read from the response when it + starts at byte 0 and has at least 16 bytes, otherwise with its own + `bytes=0-15` request; the feed's enclosure type is never trusted), and + stops at 1 GiB, 20 s without headers, 30 s + without data or 4 h per response. It is off unless + `podcast_audio_proxy_enabled` is set; by default the browser streams from + the publisher URL. + +### Podcast UI and playback + +`api.rs` holds what the endpoints do, independent of HTTP (`http.rs` only +authenticates and maps statuses), so the Postgres tests drive it directly: + +| Endpoint (`/api/podcasts`) | Does | +| --- | --- | +| `GET /feeds[?all=1]`, `POST /feeds {url}` | list; subscribe after fetching the feed (`runner::subscribe_url`) | +| `GET /feeds/{id}`, `GET /feeds/{id}/episodes?season=&played=&q=&sort=&limit=&offset=` | podcast, seasons, history; episode pages with the browser queue item | +| `POST /feeds/{id}/subscribe`, `/unsubscribe`, `POST /{id}/refresh` | subscription and manual refresh | +| `POST /episodes/{id}/restart`, `/played`, `/unplayed` | explicit progress actions | +| `POST /episodes/{id}/playback` | source, resume position; with `start` a new session (explicit, above `carried_generation`); records an unknown handed-off reference first | +| `POST /episodes/{id}/progress` | `checkpoint`/`pause`/`stop`/`handoff` at the session generation, `seek` explicit, completion once (`ended` or `reaches_completion`), listening session record | + +Request bodies must be `application/json` (415 otherwise, like cot's `Json` +extractor): other types need no CORS preflight and would let a same-site page +post with the session cookie. + +In the browser (`templates/player/scripts.html`), `_playEpisodeLocal` asks +for the playback, loads the publisher URL (or the proxy: mixed content, or a +failed direct load) and seeks before playing; when the source cannot seek to +the saved position it shows that instead of playing from elsewhere. Only a +handoff or the user's own seek sends a position (`explicitPosition`); an +ordinary Play, including after a reload, resumes from the register, and Play +after a pause first checks that the register is still this session's +(`register_session_id`). A seek made while the source is being prepared is +applied when it arrives. Writes of one session run in order +(`_reportEpisode`), read the session generation at send time and carry the +position of their own episode. After `superseded`, checkpoints stop; if the +user keeps listening here, the next write (one checkpoint interval later, or +at once on pause, stop or the end of the stream) records the old session and +starts a new explicit one (`_takeOverEpisode`). A browser that loses the +active role sends one final `stop` at its session generation (it records the +listening; the new device's start keeps the register), unless it handed off +itself and the transfer went through (a failed transfer clears `handedOff` in +the devices store's `select`). Preparing the same episode again (Play after +its source failed) waits for the old session's final report, so the server +resumes from where the audio stopped. Marking the loaded episode played or unplayed in the same tab saves +and closes its session and pauses it. Prev on an episode goes back 15 s +locally instead of resetting it. Jam guests never play or save episodes +(they only ever see placeholders). The `podcasts` store backs the Podcasts +views in `shell.html`. + +Tests never reach the internet: `test_server.rs` serves on 127.0.0.1 and +provides a scripted resolver. Loopback is reachable only through +`AddressPolicy::test_allow_loopback`, which exists only under `#[cfg(test)]`. + +Everything sent to a device is gated by the version that device reported itself +(`profile_protocol_version`; relayed profiles only raise it). See the podcast +contract (`PODCASTS.md`) and `PLAYBACK_PROTOCOL.md` in the Frid repository. + +```bash +cargo test devices:: queue_items podcasts:: queue_persistence +node --test tests/player_queue_sources.test.cjs tests/player_podcasts.test.cjs +# Disposable PostgreSQL; the tests run the production migrations on it, and +# the migration test creates and drops a database next to it: +FURU_DEVICE_SYNC_TEST_DATABASE_URL=postgresql://... \ + cargo test postgres_tests -- --ignored +FURU_METADATA_FIX_TEST_DATABASE_URL=postgresql://... \ + cargo test undo_accepts_playback_archives_written_before_the_typed_queue -- --ignored +``` + +`wire_compat_tests` compares the shared types with the private copies of +0.12.x byte for byte; `fixtures/v3_hello.golden.json` is the v3 shape. Rewrite +it only for a deliberate change, with `FURUMUSIC_BLESS_GOLDEN=1`. + ## Contributing Bug reports, design discussions, and patches are welcome. Before submitting a diff --git a/src/admin/metadata_fix.rs b/src/admin/metadata_fix.rs index 80c1b3b..33b4954 100644 --- a/src/admin/metadata_fix.rs +++ b/src/admin/metadata_fix.rs @@ -1438,18 +1438,28 @@ async fn delete_entities( } } // Playback queues contain local IDs rather than foreign keys. - let states: Vec<(i64, Option, String)> = sqlx::query_as( - "SELECT id,current_track_id,queue_json FROM furumusic__playback_state FOR UPDATE", + // (id, current_track_id, queue_json, queue_items_json, queue_position) + type QueueRow = (i64, Option, String, Option, i32); + let states: Vec = sqlx::query_as( + "SELECT id,current_track_id,queue_json,queue_items_json,queue_position FROM furumusic__playback_state FOR UPDATE", ) .fetch_all(&mut *conn) .await?; - for (id, current, queue_json) in states { + for (id, current, queue_json, queue_items_json, queue_position) in states { let mut queue: Vec = serde_json::from_str(&queue_json).unwrap_or_default(); if !current.is_some_and(|id| deleted.tracks.contains(&id)) && !queue.iter().any(|id| deleted.tracks.contains(id)) { continue; } + let mut typed = crate::player::queue_items::StoredQueue::from_columns( + queue_items_json.as_ref(), + &crate::player::queue_items::LegacyColumns { + queue: queue.clone(), + current_track_id: current, + queue_position, + }, + ); let before: String = sqlx::query_scalar( "SELECT to_jsonb(t)::text FROM furumusic__playback_state t WHERE id=$1", ) @@ -1461,8 +1471,19 @@ async fn delete_entities( let position = current .and_then(|id| queue.iter().position(|x| *x == id)) .unwrap_or(0) as i32; - let after: String = sqlx::query_scalar("UPDATE furumusic__playback_state SET current_track_id=$2,queue_json=$3,queue_position=$4,position_ms=CASE WHEN current_track_id IS DISTINCT FROM $2 THEN 0 ELSE position_ms END WHERE id=$1 RETURNING to_jsonb(furumusic__playback_state)::text") - .bind(id).bind(current).bind(serde_json::to_string(&queue)?).bind(position).fetch_one(&mut *conn).await?; + if let Some(typed) = &mut typed { + typed.retain_library( + |id| !deleted.tracks.contains(&id), + crate::player::queue_items::LegacyColumns { + queue: queue.clone(), + current_track_id: current, + queue_position: position, + }, + ); + } + let queue_items_json = typed.map(serde_json::to_value).transpose()?; + let after: String = sqlx::query_scalar("UPDATE furumusic__playback_state SET current_track_id=$2,queue_json=$3,queue_position=$4,queue_items_json=$5,position_ms=CASE WHEN current_track_id IS DISTINCT FROM $2 THEN 0 ELSE position_ms END WHERE id=$1 RETURNING to_jsonb(furumusic__playback_state)::text") + .bind(id).bind(current).bind(serde_json::to_string(&queue)?).bind(position).bind(queue_items_json).fetch_one(&mut *conn).await?; archived.push(ArchivedRows { table: "furumusic__playback_state".into(), rows: serde_json::json!([serde_json::from_str::(&before)?]), @@ -1502,6 +1523,7 @@ async fn restore_deleted(conn: &mut PgConnection, archived: &[ArchivedRows]) -> &[ "current_track_id", "queue_json", + "queue_items_json", "queue_position", "position_ms", ] @@ -1515,7 +1537,9 @@ async fn restore_deleted(conn: &mut PgConnection, archived: &[ArchivedRows]) -> }) .collect::>() .join(","); - let affected = sqlx::query(&format!("UPDATE {table} t SET {assignments} FROM jsonb_array_elements($1::jsonb) WITH ORDINALITY b(value,n) JOIN jsonb_array_elements($2::jsonb) WITH ORDINALITY a(value,n) USING(n) WHERE to_jsonb(t)=a.value")) + // Archives written before m_0054 have no `queue_items_json` key; + // compare them without the column added since. + let affected = sqlx::query(&format!("UPDATE {table} t SET {assignments} FROM jsonb_array_elements($1::jsonb) WITH ORDINALITY b(value,n) JOIN jsonb_array_elements($2::jsonb) WITH ORDINALITY a(value,n) USING(n) WHERE (CASE WHEN a.value ? 'queue_items_json' THEN to_jsonb(t) ELSE to_jsonb(t) - 'queue_items_json' END)=a.value")) .bind(entry.rows.to_string()).bind(after.to_string()).execute(&mut *conn).await?.rows_affected(); anyhow::ensure!( affected as usize == after.as_array().map_or(0, Vec::len), @@ -1804,6 +1828,51 @@ pub(super) async fn undo( mod tests { use super::*; + #[tokio::test] + #[ignore = "requires a migrated disposable database in FURU_METADATA_FIX_TEST_DATABASE_URL"] + async fn undo_accepts_playback_archives_written_before_the_typed_queue() { + let pool = + sqlx::PgPool::connect(&std::env::var("FURU_METADATA_FIX_TEST_DATABASE_URL").unwrap()) + .await + .unwrap(); + let mut tx = pool.begin().await.unwrap(); + let user_id = 9_000_000_000 + i64::from(std::process::id()); + // State after the deletion, read back like an archive written by 0.12 + // (`to_jsonb` of the row before m_0054 added `queue_items_json`). + let after: serde_json::Value = sqlx::query_scalar( + "INSERT INTO furumusic__playback_state (user_id,current_track_id,position_ms,queue_json,queue_position,shuffle,repeat_mode,volume,updated_at) VALUES ($1,NULL,0,'[]',0,false,'off',0.7,'now') RETURNING to_jsonb(furumusic__playback_state) - 'queue_items_json'", + ) + .bind(user_id) + .fetch_one(&mut *tx) + .await + .unwrap(); + let mut before = after.clone(); + before["queue_json"] = serde_json::json!("[123456789]"); + before["position_ms"] = serde_json::json!(5000); + let archive = ArchivedRows { + table: "furumusic__playback_state".into(), + rows: serde_json::json!([before]), + after: Some(serde_json::json!([after])), + }; + restore_deleted(&mut tx, std::slice::from_ref(&archive)) + .await + .unwrap(); + let (queue_json, position_ms, typed): (String, i32, Option) = + sqlx::query_as( + "SELECT queue_json,position_ms,queue_items_json FROM furumusic__playback_state WHERE user_id=$1", + ) + .bind(user_id) + .fetch_one(&mut *tx) + .await + .unwrap(); + assert_eq!(queue_json, "[123456789]"); + assert_eq!(position_ms, 5000); + assert_eq!(typed, None); + // A row changed since the deletion is still refused. + assert!(restore_deleted(&mut tx, &[archive]).await.is_err()); + tx.rollback().await.unwrap(); + } + fn snapshot() -> Snapshot { Snapshot { artists: vec![ArtistState { diff --git a/src/admin/v2.rs b/src/admin/v2.rs index 72a4ebc..aa2a9ce 100644 --- a/src/admin/v2.rs +++ b/src/admin/v2.rs @@ -545,6 +545,8 @@ struct AdminSettingsValues { youtube_proxy_id: String, #[serde(default)] youtube_cookie_id: String, + #[serde(default)] + podcast_audio_proxy_enabled: bool, } #[derive(Debug, Clone, Serialize, Deserialize, JsonSchema)] @@ -615,6 +617,7 @@ struct AdminSettingsSources { torrent_proxy_id: &'static str, youtube_proxy_id: &'static str, youtube_cookie_id: &'static str, + podcast_audio_proxy_enabled: &'static str, } #[derive(Debug, Deserialize)] @@ -667,6 +670,8 @@ pub(super) struct UpdateSettingsRequest { youtube_proxy_id: String, #[serde(default)] youtube_cookie_id: String, + #[serde(default)] + podcast_audio_proxy_enabled: bool, } fn default_similarity_model() -> String { @@ -1370,6 +1375,10 @@ pub async fn update_settings( ("torrent_proxy_id", torrent_proxy_id), ("youtube_proxy_id", youtube_proxy_id), ("youtube_cookie_id", youtube_cookie_id), + ( + "podcast_audio_proxy_enabled", + body.podcast_audio_proxy_enabled.to_string(), + ), ]; for (key, value) in fields { let mut entry = ConfigEntry::new(key.to_string(), value); @@ -1714,6 +1723,7 @@ fn settings_dto( torrent_proxy_id: config.torrent_proxy_id, youtube_proxy_id: config.youtube_proxy_id, youtube_cookie_id: config.youtube_cookie_id, + podcast_audio_proxy_enabled: config.podcast_audio_proxy_enabled, }, sources: AdminSettingsSources { auth_password_enabled: sources.auth_password_enabled.code(), @@ -1750,6 +1760,7 @@ fn settings_dto( torrent_proxy_id: sources.torrent_proxy_id.code(), youtube_proxy_id: sources.youtube_proxy_id.code(), youtube_cookie_id: sources.youtube_cookie_id.code(), + podcast_audio_proxy_enabled: sources.podcast_audio_proxy_enabled.code(), }, } } diff --git a/src/config.rs b/src/config.rs index e554667..30572ee 100644 --- a/src/config.rs +++ b/src/config.rs @@ -149,6 +149,7 @@ pub struct ConfigSources { pub torrent_proxy_id: ConfigSource, pub youtube_proxy_id: ConfigSource, pub youtube_cookie_id: ConfigSource, + pub podcast_audio_proxy_enabled: ConfigSource, } impl Default for ConfigSources { @@ -190,6 +191,7 @@ impl Default for ConfigSources { torrent_proxy_id: ConfigSource::Default, youtube_proxy_id: ConfigSource::Default, youtube_cookie_id: ConfigSource::Default, + podcast_audio_proxy_enabled: ConfigSource::Default, } } } @@ -407,6 +409,10 @@ pub struct AppConfig { pub youtube_proxy_id: String, /// Saved cookie-file id used by yt-dlp; empty means no cookies. pub youtube_cookie_id: String, + /// Whether the server proxies podcast episode audio from publishers for + /// browsers that cannot load it directly (CORS, mixed content). Off by + /// default: browsers stream from the publisher URL. + pub podcast_audio_proxy_enabled: bool, } impl Default for AppConfig { @@ -451,6 +457,7 @@ impl Default for AppConfig { torrent_proxy_id: String::new(), youtube_proxy_id: String::new(), youtube_cookie_id: String::new(), + podcast_audio_proxy_enabled: false, } } } @@ -493,6 +500,7 @@ impl_env_overrides!( torrent_proxy_id, youtube_proxy_id, youtube_cookie_id, + podcast_audio_proxy_enabled, ); impl AppConfig { @@ -636,6 +644,7 @@ impl AppConfig { apply_db_field!(torrent_proxy_id); apply_db_field!(youtube_proxy_id); apply_db_field!(youtube_cookie_id); + apply_db_field!(podcast_audio_proxy_enabled); } pub fn parsed_download_proxies(&self) -> anyhow::Result> { @@ -693,6 +702,7 @@ mod tests { assert!(cfg.torrent_downloads_enabled); assert!(cfg.youtube_downloads_enabled); assert!(cfg.parsed_download_proxies().unwrap().is_empty()); + assert!(!cfg.podcast_audio_proxy_enabled); } #[test] diff --git a/src/federation/devices.rs b/src/federation/devices.rs index 125aa4f..b73c9cf 100644 --- a/src/federation/devices.rs +++ b/src/federation/devices.rs @@ -1,40 +1,69 @@ //! Personal Fed Clients for the web player. //! //! The web service has one iroh endpoint, but each user owns one virtual -//! fed-device inside that endpoint. This module speaks the same JSON-lines -//! protocol as the TUI clients on `furumi/sync/1` and maps operations into -//! user-scoped Postgres state. +//! fed-device inside that endpoint. This module speaks the shared +//! `music_dht::device_sync` JSON-lines protocol of the TUI clients on +//! `furumi/sync/2` and maps operations into user-scoped Postgres state. +//! +//! The wire types are the shared definitions; there are no local copies. +//! Device-sync protocol v4 (podcasts) is announced: podcast operations stay +//! in the generic operation log and are applied to the account's podcast +//! store ([`crate::podcasts::PodcastStore`]), which also provides the +//! podcast snapshot sections, and queue items keep their explicit +//! `PlaybackTrack.source` through the browser bridge +//! ([`crate::player::queue_items`]). Everything sent to a peer is gated by +//! the version that peer reported itself. -use music_dht::playback::{ - Announcement, Checkpoint, CommandStamp, Config as PlaybackConfig, Engine, -}; +use music_dht::playback::{Announcement, Checkpoint, Config as PlaybackConfig, Engine}; use std::collections::BTreeMap; -use std::str::FromStr; use std::sync::Arc; use std::time::Duration; use anyhow::{Context, Result}; -use music_dht::device_sync::{ListenEndReason, ListenEvent, ListenTrackMetadata}; +use music_dht::device_sync::{ + DEVICE_SYNC_PROTOCOL_VERSION, DeviceProfileWire, InviteWire, ListenEndReason, ListenEvent, + ListenTrackMetadata, PlaybackCommand, PlaybackRepeat, PlaybackSnapshot, PlaybackSource, + PlaybackStateWire, PlaybackTrack, SnapshotLike, SnapshotLikeTombstone, SnapshotPlaylist, + SnapshotPlaylistItem, SnapshotPlaylistItemTombstone, SnapshotPlaylistTombstone, SyncOpPayload, + SyncOpWire, SyncSnapshot, SyncedFedTrack, WireMessage, encode_invite, hash_secret, + op_allowed_for_peer, parse_invite, playback_allowed_for_peer, + playback_command_allowed_from_peer, podcast_snapshot_budget, profile_protocol_version, + random_hex, relayed_protocol_version, supports_listen_history, supports_podcasts, + ticket_endpoint_id, +}; +use music_dht::podcast::{EpisodeId, ProgressRecord}; use music_dht::{ByteStream, MusicDhtService, NetworkId, PeerTicket, SecretKey, StreamAcceptor}; -use serde::{Deserialize, Serialize}; +use serde::Serialize; use sqlx::Row; -use tokio::io::{AsyncRead, AsyncReadExt}; use crate::player::PlayerDeviceHub; +use crate::player::queue_items; use super::{TransportStats, record_stream_transport}; +use crate::podcasts::{ApplyOutcome, PodcastStore}; pub const SYNC_ALPN: &[u8] = b"furumi/sync/2"; const CLIENT_VERSION: &str = env!("CARGO_PKG_VERSION"); -const PROTOCOL_VERSION: u16 = 3; +/// Device-sync protocol version announced by this server's virtual devices. +/// +/// Version 4 is announced only because podcast operations, podcast snapshot +/// sections and podcast queue items are kept and relayed losslessly (see +/// the module documentation). Review that support before following a newer +/// shared protocol version. +const PROTOCOL_VERSION: u16 = 4; +const _: () = assert!( + PROTOCOL_VERSION == DEVICE_SYNC_PROTOCOL_VERSION, + "review web device-sync support before announcing a new protocol version" +); +/// Lowest protocol version of a peer that speaks `furumi/sync/2`. +const SYNC_V2_PROTOCOL_VERSION: u16 = 2; const INVITE_TTL_MS: i64 = 10 * 60 * 1000; const PAIRING_WAIT_MS: i64 = 5 * 60 * 1000; const PAIRING_RETRY_DELAY: Duration = Duration::from_secs(1); const RESPONSE_DRAIN_TIMEOUT: Duration = Duration::from_secs(2); const DEVICE_SYNC_INTERVAL: Duration = Duration::from_secs(2); const LOCAL_SEED_RECHECK_MS: i64 = 60 * 1000; -const MAX_LINE: usize = 8 * 1024 * 1024; /// Playback commands carry queue state but are useful only briefly and only /// to their addressed device. They must not inflate a new peer's catch-up. const PLAYBACK_COMMAND_TTL_MS: i64 = 5 * 60 * 1_000; @@ -94,319 +123,6 @@ struct StoredDevice { endpoint_ticket: String, } -#[derive(Debug, Clone, Serialize, Deserialize)] -struct InviteWire { - v: u16, - #[serde(rename = "t")] - ticket: String, - #[serde(rename = "d")] - device_id: String, - #[serde(rename = "i")] - invite_id: String, - #[serde(rename = "s")] - secret: String, - #[serde(rename = "e")] - expires_at_ms: i64, -} - -#[derive(Debug, Clone, Serialize, Deserialize)] -struct DeviceProfileWire { - device_id: String, - name: String, - client_version: String, - protocol_version: u16, - endpoint_id: String, - endpoint_ticket: String, - #[serde(default)] - revoked: bool, - #[serde(default)] - revoke_cutoff_seq: Option, - #[serde(default)] - updated_at_ms: i64, -} - -#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] -struct SyncedFedTrack { - item_id: String, - owner: String, - title: String, - #[serde(default)] - artist_names: Vec, - #[serde(default)] - featured_artist_names: Vec, - year: Option, - duration_seconds: Option, - content_id: String, - release_title: Option, - track_number: Option, - disc_number: Option, -} - -#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] -struct PlaybackTrack { - id: i64, - title: String, - track_number: Option, - disc_number: Option, - duration_seconds: f64, - #[serde(default)] - artist_names: Vec, - #[serde(default)] - featured_artist_names: Vec, - release_id: i64, - release_title: String, - release_year: Option, - #[serde(default)] - file_path: String, - content_id: Option, - audio_format: Option, - audio_bitrate: Option, - audio_sample_rate: Option, - audio_bit_depth: Option, - file_size_bytes: Option, - #[serde(default)] - play_count: i64, - #[serde(default)] - fed: Option, -} - -#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Default)] -#[serde(rename_all = "snake_case")] -enum PlaybackRepeat { - #[default] - Off, - One, - All, -} - -#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] -struct PlaybackStateWire { - #[serde(default)] - queue: Vec, - #[serde(default)] - queue_pos: usize, - playing: bool, - paused: bool, - #[serde(default)] - idle_since_ms: Option, - position_secs: f64, - #[serde(default)] - volume: u8, - shuffle: bool, - repeat: PlaybackRepeat, -} - -type PlaybackSnapshot = music_dht::playback::Snapshot; -type PlaybackCommand = music_dht::playback::Command; - -#[derive(Debug, Clone, Serialize, Deserialize)] -struct SyncOpWire { - op_id: String, - origin_device_id: String, - seq: i64, - hlc_ms: i64, - payload: SyncOpPayload, -} - -#[derive(Debug, Clone, Serialize, Deserialize)] -#[serde(tag = "kind", rename_all = "snake_case")] -enum SyncOpPayload { - TrackLikeSet { - content_id: String, - liked: bool, - #[serde(default)] - fed: Option, - }, - PlaylistCreated { - playlist_id: String, - title: String, - }, - PlaylistRenamed { - playlist_id: String, - title: String, - }, - PlaylistDeleted { - playlist_id: String, - }, - PlaylistTrackAdded { - playlist_id: String, - content_id: String, - position: i64, - #[serde(default)] - fed: Option, - }, - PlaylistTrackRemoved { - playlist_id: String, - content_id: String, - }, - DeviceProfileSet { - name: String, - client_version: String, - endpoint_ticket: String, - endpoint_id: String, - }, - DeviceTrusted { - target_device_id: String, - }, - DeviceRevoked { - target_device_id: String, - target_max_seq_seen: i64, - }, - PlaybackCommand { - target_device_id: String, - command: PlaybackCommand, - #[serde(default)] - authority: Option, - }, - ListenRecorded { - event: ListenEvent, - }, -} - -impl SyncOpPayload { - fn is_tombstone(&self) -> bool { - matches!( - self, - SyncOpPayload::TrackLikeSet { liked: false, .. } - | SyncOpPayload::PlaylistDeleted { .. } - | SyncOpPayload::PlaylistTrackRemoved { .. } - | SyncOpPayload::DeviceRevoked { .. } - ) - } -} - -#[derive(Debug, Clone, Default, Serialize, Deserialize)] -struct SyncSnapshot { - #[serde(default)] - likes: Vec, - #[serde(default)] - unlikes: Vec, - #[serde(default)] - playlists: Vec, - #[serde(default)] - deleted_playlists: Vec, - #[serde(default)] - removed_playlist_items: Vec, -} - -#[derive(Debug, Clone, Serialize, Deserialize)] -struct SnapshotLike { - content_id: String, - hlc_ms: i64, - op_id: String, - #[serde(default)] - fed: Option, -} - -#[derive(Debug, Clone, Serialize, Deserialize)] -struct SnapshotLikeTombstone { - content_id: String, - hlc_ms: i64, - op_id: String, -} - -#[derive(Debug, Clone, Serialize, Deserialize)] -struct SnapshotPlaylist { - playlist_id: String, - title: String, - hlc_ms: i64, - op_id: String, - #[serde(default)] - items: Vec, -} - -#[derive(Debug, Clone, Serialize, Deserialize)] -struct SnapshotPlaylistTombstone { - playlist_id: String, - hlc_ms: i64, - op_id: String, -} - -#[derive(Debug, Clone, Serialize, Deserialize)] -struct SnapshotPlaylistItem { - content_id: String, - position: i64, - hlc_ms: i64, - op_id: String, - #[serde(default)] - fed: Option, -} - -#[derive(Debug, Clone, Serialize, Deserialize)] -struct SnapshotPlaylistItemTombstone { - playlist_id: String, - content_id: String, - hlc_ms: i64, - op_id: String, -} - -#[derive(Debug, Clone, Serialize, Deserialize)] -#[serde(tag = "type", rename_all = "snake_case")] -enum WireMessage { - PairRequest { - invite_id: String, - secret: String, - profile: DeviceProfileWire, - #[serde(default)] - group_id: Option, - #[serde(default)] - group_active_devices: usize, - #[serde(default)] - devices: Vec, - vector: BTreeMap, - ops: Vec, - snapshot: SyncSnapshot, - #[serde(default)] - playback: Option, - }, - PairResponse { - accepted: bool, - #[serde(default)] - pending: bool, - #[serde(default)] - error: Option, - #[serde(default)] - group_id: Option, - #[serde(default)] - profile: Option, - #[serde(default)] - devices: Vec, - #[serde(default)] - vector: BTreeMap, - #[serde(default)] - ops: Vec, - #[serde(default)] - snapshot: SyncSnapshot, - #[serde(default)] - playback: Option, - }, - Hello { - group_id: String, - profile: DeviceProfileWire, - devices: Vec, - vector: BTreeMap, - ops: Vec, - snapshot: SyncSnapshot, - #[serde(default)] - playback: Option, - }, - SyncResponse { - accepted: bool, - #[serde(default)] - error: Option, - #[serde(default)] - devices: Vec, - #[serde(default)] - vector: BTreeMap, - #[serde(default)] - ops: Vec, - #[serde(default)] - snapshot: SyncSnapshot, - #[serde(default)] - playback: Option, - }, -} - enum PairAttempt { Accepted(String), Pending, @@ -443,8 +159,9 @@ pub async fn create_invite( invite_id, secret, expires_at_ms, + protocol_version: Some(PROTOCOL_VERSION), }; - encode_invite(&payload) + Ok(encode_invite(&payload)?) } pub async fn connect_invite( @@ -528,11 +245,13 @@ pub async fn answer_pairing( if let Some(row) = pending { let device_id: String = row.get("device_id"); enforce_single_user_binding(pool, user_id, &device_id).await?; + // The pending row has no version; the requester reports it again in + // the PairRequest it repeats until the answer arrives. let profile = DeviceProfileWire { device_id, name: row.get("name"), client_version: row.get("client_version"), - protocol_version: PROTOCOL_VERSION, + protocol_version: music_dht::device_sync::UNREPORTED_PROTOCOL_VERSION, endpoint_id: row.get("endpoint_id"), endpoint_ticket: row.get("endpoint_ticket"), revoked: false, @@ -552,9 +271,16 @@ pub async fn answer_pairing( } let devices_json: String = row.get("requester_group_devices_json"); let devices: Vec = serde_json::from_str(&devices_json)?; - apply_device_profiles(pool, user_id, &devices).await?; + apply_device_profiles(pool, user_id, &devices, Some(&profile.device_id)).await?; } - apply_device_profile(pool, user_id, &profile, true).await?; + apply_device_profile( + pool, + user_id, + &profile, + VersionReport::SelfReported(None), + true, + ) + .await?; record_local_op( pool, user_id, @@ -1179,9 +905,13 @@ async fn try_connect_invite( let profile = own_profile(pool, user_id, user_name, &own_ticket).await?; let devices = device_profiles(pool, user_id).await?; let vector = vector(pool, user_id).await?; - let ops = ops_for_peer(pool, user_id, &invite.device_id).await?; - let snapshot = snapshot(pool, user_id).await?; - let playback = local_playback_snapshot(pool, Arc::clone(&hub), user_id, &identity).await; + // The request is sent before the inviter reports a version: gate it by + // the invite (version 1 for invites from older clients). + let inviter_version = invite.inviter_protocol_version(); + let ops = ops_for_peer(pool, user_id, &invite.device_id, inviter_version).await?; + let snapshot = snapshot(pool, user_id).await?.for_peer(inviter_version); + let playback = + local_playback_snapshot(pool, Arc::clone(&hub), user_id, &identity, inviter_version).await; let mut stream = service.open_stream(peer, SYNC_ALPN).await?; record_stream_transport(&transport_stats, "device-sync", "outbound", "open", &stream); write_msg( @@ -1220,7 +950,14 @@ async fn try_connect_invite( set_group_id(pool, user_id, &group_id).await?; if let Some(profile) = profile { enforce_single_user_binding(pool, user_id, &profile.device_id).await?; - apply_device_profile(pool, user_id, &profile, true).await?; + apply_device_profile( + pool, + user_id, + &profile, + VersionReport::SelfReported(Some(profile.protocol_version)), + true, + ) + .await?; if let Some(playback) = playback { apply_playback_snapshot( Arc::clone(&hub), @@ -1232,7 +969,7 @@ async fn try_connect_invite( .await?; } } - apply_device_profiles(pool, user_id, &devices).await?; + apply_device_profiles(pool, user_id, &devices, Some(&invite.device_id)).await?; apply_snapshot(pool, user_id, snapshot).await?; apply_ops(pool, Arc::clone(&hub), user_id, ops).await?; record_local_op( @@ -1446,7 +1183,14 @@ async fn handle_pair_request( let own_ticket = service.ticket().await?.to_string(); let own_profile = own_profile(pool, user_id, "", &own_ticket).await?; - apply_device_profile(pool, user_id, &profile, true).await?; + apply_device_profile( + pool, + user_id, + &profile, + VersionReport::SelfReported(Some(profile.protocol_version)), + true, + ) + .await?; if let Some(playback) = playback { apply_playback_snapshot( Arc::clone(&hub), @@ -1459,7 +1203,13 @@ async fn handle_pair_request( } apply_snapshot(pool, user_id, incoming_snapshot).await?; apply_ops(pool, Arc::clone(&hub), user_id, ops).await?; - apply_device_profiles(pool, user_id, &requester_group_devices).await?; + apply_device_profiles( + pool, + user_id, + &requester_group_devices, + Some(&profile.device_id), + ) + .await?; note_peer_vector(pool, user_id, &profile.device_id, &incoming_vector).await?; sqlx::query("UPDATE furumusic__fed_device_invite SET used_at_ms = $2 WHERE invite_id = $1") .bind(&invite_id) @@ -1473,11 +1223,12 @@ async fn handle_pair_request( ) .await?; + let peer_version = stored_protocol_version(pool, user_id, &profile.device_id).await?; let devices = device_profiles(pool, user_id).await?; let vector = vector(pool, user_id).await?; - let ops = ops_for_peer(pool, user_id, &profile.device_id).await?; - let snapshot = snapshot(pool, user_id).await?; - let playback = local_playback_snapshot(pool, hub, user_id, &identity).await; + let ops = ops_for_peer(pool, user_id, &profile.device_id, peer_version).await?; + let snapshot = snapshot(pool, user_id).await?.for_peer(peer_version); + let playback = local_playback_snapshot(pool, hub, user_id, &identity, peer_version).await; write_msg( &mut stream, &WireMessage::PairResponse { @@ -1522,8 +1273,15 @@ async fn handle_hello( return write_sync_reject(stream, "sync group mismatch").await; } profile.endpoint_id = stream.peer_id.to_string(); - apply_device_profile(pool, user_id, &profile, false).await?; - apply_device_profiles(pool, user_id, &devices).await?; + apply_device_profile( + pool, + user_id, + &profile, + VersionReport::SelfReported(Some(profile.protocol_version)), + false, + ) + .await?; + apply_device_profiles(pool, user_id, &devices, Some(&profile.device_id)).await?; if let Some(playback) = playback { apply_playback_snapshot( Arc::clone(&hub), @@ -1555,10 +1313,11 @@ async fn handle_hello( let own_profile = own_profile(pool, user_id, "", &own_ticket).await?; let mut devices = device_profiles(pool, user_id).await?; devices.push(own_profile); + let peer_version = stored_protocol_version(pool, user_id, &profile.device_id).await?; let vector = vector(pool, user_id).await?; - let ops = ops_for_peer(pool, user_id, &profile.device_id).await?; - let snapshot = snapshot(pool, user_id).await?; - let playback = local_playback_snapshot(pool, hub, user_id, &identity).await; + let ops = ops_for_peer(pool, user_id, &profile.device_id, peer_version).await?; + let snapshot = snapshot(pool, user_id).await?.for_peer(peer_version); + let playback = local_playback_snapshot(pool, hub, user_id, &identity, peer_version).await; write_msg( &mut stream, &WireMessage::SyncResponse { @@ -1589,11 +1348,13 @@ async fn sync_device( let own_ticket = service.ticket().await?.to_string(); let identity = ensure_identity(pool, user_id, "").await?; let profile = own_profile(pool, user_id, "", &own_ticket).await?; + let peer_version = stored_protocol_version(pool, user_id, &device.device_id).await?; let devices = device_profiles(pool, user_id).await?; let vector = vector(pool, user_id).await?; - let ops = ops_for_peer(pool, user_id, &device.device_id).await?; - let snapshot = snapshot(pool, user_id).await?; - let playback = local_playback_snapshot(pool, Arc::clone(&hub), user_id, &identity).await; + let ops = ops_for_peer(pool, user_id, &device.device_id, peer_version).await?; + let snapshot = snapshot(pool, user_id).await?.for_peer(peer_version); + let playback = + local_playback_snapshot(pool, Arc::clone(&hub), user_id, &identity, peer_version).await; let mut stream = service.open_stream(peer, SYNC_ALPN).await?; record_stream_transport(&transport_stats, "device-sync", "outbound", "open", &stream); write_msg( @@ -1624,7 +1385,9 @@ async fn sync_device( playback, .. } => { - apply_device_profiles(pool, user_id, &devices).await?; + // A SyncResponse has no profile: the responder's own entry in + // `devices` is its version report. + apply_device_profiles(pool, user_id, &devices, Some(&device.device_id)).await?; if let Some(playback) = playback { apply_playback_snapshot( Arc::clone(&hub), @@ -1930,6 +1693,14 @@ async fn own_profile( ) -> Result { let identity = ensure_identity(pool, user_id, user_name).await?; let endpoint_id = ticket_endpoint_id(endpoint_ticket).unwrap_or_default(); + let previous_version: Option = sqlx::query_scalar( + "SELECT protocol_version FROM furumusic__fed_device + WHERE user_id = $1 AND device_id = $2", + ) + .bind(user_id) + .bind(&identity.device_id) + .fetch_optional(pool) + .await?; let now = now_ms(); sqlx::query( "INSERT INTO furumusic__fed_device @@ -1954,6 +1725,23 @@ async fn own_profile( .bind(now) .execute(pool) .await?; + if previous_version.is_some_and(|version| version < i32::from(PROTOCOL_VERSION)) { + // First start of a new protocol version: announce it in the log too, + // so devices that only see this device's operations through relays + // learn the version (`profile_protocol_version`). + record_local_op( + pool, + user_id, + SyncOpPayload::DeviceProfileSet { + name: identity.name.clone(), + client_version: CLIENT_VERSION.to_string(), + endpoint_ticket: endpoint_ticket.to_string(), + endpoint_id: endpoint_id.clone(), + protocol_version: Some(PROTOCOL_VERSION), + }, + ) + .await?; + } Ok(DeviceProfileWire { device_id: identity.device_id, name: identity.name, @@ -1994,20 +1782,7 @@ async fn record_local_op( ); } - let now = now_ms(); - let row = sqlx::query( - "UPDATE furumusic__fed_device_identity - SET local_seq = local_seq + 1, - last_hlc_ms = GREATEST($2, last_hlc_ms + 1) - WHERE user_id = $1 - RETURNING local_seq, last_hlc_ms", - ) - .bind(user_id) - .bind(now) - .fetch_one(pool) - .await?; - let seq: i64 = row.get("local_seq"); - let hlc_ms: i64 = row.get("last_hlc_ms"); + let (seq, hlc_ms) = reserve_local_stamp(pool, user_id).await?; let op_id = format!("{}:{seq}", identity.device_id); let op = SyncOpWire { op_id, @@ -2016,7 +1791,122 @@ async fn record_local_op( hlc_ms, payload, }; + insert_op(pool, user_id, &op).await?; + advance_vector(pool, user_id, &op.origin_device_id, op.seq).await?; + let _ = apply_op(pool, Arc::new(PlayerDeviceHub::default()), user_id, &op).await?; + Ok(()) +} + +/// Reserves the next local `(seq, hlc_ms)` of the account's web device. +/// +/// `hlc_ms` is the wall clock, but always past the previous local +/// operation. Inside a transaction the identity row stays locked until it +/// ends, so local operations of one account are stamped in commit order. +async fn reserve_local_stamp<'e>( + db: impl sqlx::PgExecutor<'e>, + user_id: i64, +) -> Result<(i64, i64)> { + let row = sqlx::query( + "UPDATE furumusic__fed_device_identity + SET local_seq = local_seq + 1, + last_hlc_ms = GREATEST($2, last_hlc_ms + 1) + WHERE user_id = $1 + RETURNING local_seq, last_hlc_ms", + ) + .bind(user_id) + .bind(now_ms()) + .fetch_one(db) + .await?; + Ok((row.get("local_seq"), row.get("last_hlc_ms"))) +} + +async fn advance_vector<'e>( + db: impl sqlx::PgExecutor<'e>, + user_id: i64, + device_id: &str, + seq: i64, +) -> Result<()> { sqlx::query( + "INSERT INTO furumusic__fed_sync_vector (user_id, device_id, max_seq) + VALUES ($1, $2, $3) + ON CONFLICT (user_id, device_id) DO UPDATE SET + max_seq = GREATEST(furumusic__fed_sync_vector.max_seq, EXCLUDED.max_seq)", + ) + .bind(user_id) + .bind(device_id) + .bind(seq) + .execute(db) + .await?; + Ok(()) +} + +/// Device id of the account's web device, creating the identity on first +/// use. Local podcast actions stamp their operations with it. +pub(crate) async fn local_device_id(pool: &sqlx::PgPool, user_id: i64) -> Result { + Ok(ensure_identity(pool, user_id, "").await?.device_id) +} + +/// Appends a local operation of `device_id` (the account's web device, see +/// [`local_device_id`]) to the sync log inside the caller's transaction. +/// +/// The stamp (`seq`, `hlc_ms`, `op_id = :`) is reserved here; +/// `make` receives the `op_id`. The caller applies the returned operation +/// itself in the same transaction, so the log and the materialized state +/// change together (the podcast store does this for its local actions). +pub(crate) async fn append_local_op( + conn: &mut sqlx::PgConnection, + user_id: i64, + device_id: &str, + make: impl FnOnce(&str) -> SyncOpPayload, +) -> Result { + let (seq, hlc_ms) = reserve_local_stamp(&mut *conn, user_id).await?; + let op_id = format!("{device_id}:{seq}"); + let payload = make(&op_id); + let op = SyncOpWire { + op_id, + origin_device_id: device_id.to_string(), + seq, + hlc_ms, + payload, + }; + anyhow::ensure!( + insert_op(&mut *conn, user_id, &op).await?, + "local operation {} already exists", + op.op_id + ); + advance_vector(&mut *conn, user_id, device_id, seq).await?; + Ok(op) +} + +async fn apply_ops( + pool: &sqlx::PgPool, + hub: Arc, + user_id: i64, + ops: Vec, +) -> Result<()> { + for op in ops { + if !should_accept_op(pool, user_id, &op).await? { + continue; + } + // Podcast operations that break the contract are skipped but still + // acknowledged (the vector below advances), so the log keeps moving. + if let Err(error) = op.validate_podcast() { + tracing::debug!(op_id = %op.op_id, "skipped invalid podcast operation: {error}"); + } else if insert_op(pool, user_id, &op).await? { + let _ = apply_op(pool, Arc::clone(&hub), user_id, &op).await?; + } + advance_vector(pool, user_id, &op.origin_device_id, op.seq).await?; + } + Ok(()) +} + +/// Appends an operation to the log; `false` when it was already stored. +async fn insert_op<'e>( + db: impl sqlx::PgExecutor<'e>, + user_id: i64, + op: &SyncOpWire, +) -> Result { + let inserted = sqlx::query( "INSERT INTO furumusic__fed_sync_ops (user_id, op_id, origin_device_id, seq, kind, payload_json, hlc_ms, received_at_ms, tombstone) @@ -2032,68 +1922,10 @@ async fn record_local_op( .bind(op.hlc_ms) .bind(now_ms()) .bind(op.payload.is_tombstone()) - .execute(pool) - .await?; - sqlx::query( - "INSERT INTO furumusic__fed_sync_vector (user_id, device_id, max_seq) - VALUES ($1, $2, $3) - ON CONFLICT (user_id, device_id) DO UPDATE SET - max_seq = GREATEST(furumusic__fed_sync_vector.max_seq, EXCLUDED.max_seq)", - ) - .bind(user_id) - .bind(&op.origin_device_id) - .bind(op.seq) - .execute(pool) - .await?; - let _ = apply_op(pool, Arc::new(PlayerDeviceHub::default()), user_id, &op).await?; - Ok(()) -} - -async fn apply_ops( - pool: &sqlx::PgPool, - hub: Arc, - user_id: i64, - ops: Vec, -) -> Result<()> { - for op in ops { - if !should_accept_op(pool, user_id, &op).await? { - continue; - } - let inserted = sqlx::query( - "INSERT INTO furumusic__fed_sync_ops - (user_id, op_id, origin_device_id, seq, kind, payload_json, hlc_ms, - received_at_ms, tombstone) - VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9) - ON CONFLICT (user_id, op_id) DO NOTHING", - ) - .bind(user_id) - .bind(&op.op_id) - .bind(&op.origin_device_id) - .bind(op.seq) - .bind(payload_kind(&op.payload)) - .bind(serde_json::to_value(&op.payload)?) - .bind(op.hlc_ms) - .bind(now_ms()) - .bind(op.payload.is_tombstone()) - .execute(pool) - .await? - .rows_affected(); - if inserted > 0 { - let _ = apply_op(pool, Arc::clone(&hub), user_id, &op).await?; - } - sqlx::query( - "INSERT INTO furumusic__fed_sync_vector (user_id, device_id, max_seq) - VALUES ($1, $2, $3) - ON CONFLICT (user_id, device_id) DO UPDATE SET - max_seq = GREATEST(furumusic__fed_sync_vector.max_seq, EXCLUDED.max_seq)", - ) - .bind(user_id) - .bind(&op.origin_device_id) - .bind(op.seq) - .execute(pool) - .await?; - } - Ok(()) + .execute(db) + .await? + .rows_affected(); + Ok(inserted > 0) } async fn should_accept_op(pool: &sqlx::PgPool, user_id: i64, op: &SyncOpWire) -> Result { @@ -2201,19 +2033,29 @@ async fn apply_op( client_version, endpoint_ticket, endpoint_id, + protocol_version, } => { + // The origin's own report: never substitute this server's version. let profile = DeviceProfileWire { device_id: op.origin_device_id.clone(), name: name.clone(), client_version: client_version.clone(), - protocol_version: PROTOCOL_VERSION, + protocol_version: protocol_version + .unwrap_or(music_dht::device_sync::UNREPORTED_PROTOCOL_VERSION), endpoint_id: endpoint_id.clone(), endpoint_ticket: endpoint_ticket.clone(), revoked: false, revoke_cutoff_seq: None, updated_at_ms: op.hlc_ms, }; - apply_device_profile(pool, user_id, &profile, false).await?; + apply_device_profile( + pool, + user_id, + &profile, + VersionReport::SelfReported(*protocol_version), + false, + ) + .await?; Ok(false) } SyncOpPayload::DeviceTrusted { target_device_id } => { @@ -2240,6 +2082,25 @@ async fn apply_op( SyncOpPayload::ListenRecorded { event } => { apply_listen_event(pool, user_id, &op.origin_device_id, event).await } + SyncOpPayload::PodcastSubscriptionSet { .. } + | SyncOpPayload::PodcastShareSet { .. } + | SyncOpPayload::PodcastFeedAliasesAdded { .. } + | SyncOpPayload::PodcastProgressSet { .. } + | SyncOpPayload::PodcastListenRecorded { .. } => { + // The operation itself is relayed from the log; the store keeps + // the registers (and the catalog rows of its episode reference). + match PodcastStore::new(pool.clone()) + .apply_remote_op(user_id, op) + .await? + { + ApplyOutcome::Changed => Ok(true), + ApplyOutcome::Rejected(reason) => { + tracing::debug!(op_id = %op.op_id, "skipped invalid podcast operation: {reason}"); + Ok(false) + } + ApplyOutcome::Unchanged | ApplyOutcome::Ignored => Ok(false), + } + } } } @@ -2605,6 +2466,17 @@ async fn apply_playlist_item_state( } async fn apply_snapshot(pool: &sqlx::PgPool, user_id: i64, snapshot: SyncSnapshot) -> Result<()> { + let podcasts = PodcastStore::new(pool.clone()) + .apply_snapshot(user_id, &snapshot) + .await?; + if podcasts.ignored_over_limits { + tracing::debug!("ignored podcast snapshot sections above the contract limits"); + } else if podcasts.rejected > 0 { + tracing::debug!( + rejected = podcasts.rejected, + "skipped invalid podcast snapshot records" + ); + } for like in snapshot.likes { apply_like_state( pool, @@ -2684,21 +2556,75 @@ async fn apply_snapshot(pool: &sqlx::PgPool, user_id: i64, snapshot: SyncSnapsho Ok(()) } +/// How the protocol version in a device profile became known. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum VersionReport { + /// The device reported its own version (`None`: an older client left it + /// out). Replaces the stored version. + SelfReported(Option), + /// Another device relayed the profile. Only raises the stored version: + /// v1–v3 relays substitute their own constant. + Relayed(u16), +} + +impl VersionReport { + fn resolve(self, stored: Option) -> u16 { + match self { + VersionReport::SelfReported(reported) => profile_protocol_version(reported, stored), + VersionReport::Relayed(relayed) => relayed_protocol_version(relayed, stored), + } + } + + /// Report carried by an entry of a `devices` list sent by `sender`. + fn for_listed(profile: &DeviceProfileWire, sender: Option<&str>) -> Self { + if sender == Some(profile.device_id.as_str()) { + VersionReport::SelfReported(Some(profile.protocol_version)) + } else { + VersionReport::Relayed(profile.protocol_version) + } + } +} + +/// Stores the profiles of a `devices` list. The entry of the authenticated +/// `sender` is its own version report; the others are relayed. async fn apply_device_profiles( pool: &sqlx::PgPool, user_id: i64, devices: &[DeviceProfileWire], + sender: Option<&str>, ) -> Result<()> { for profile in devices { - apply_device_profile(pool, user_id, profile, false).await?; + let report = VersionReport::for_listed(profile, sender); + apply_device_profile(pool, user_id, profile, report, false).await?; } Ok(()) } +/// Version gating input for a peer: the version it reported itself. +async fn stored_protocol_version( + pool: &sqlx::PgPool, + user_id: i64, + device_id: &str, +) -> Result { + let stored: Option = sqlx::query_scalar( + "SELECT protocol_version FROM furumusic__fed_device + WHERE user_id = $1 AND device_id = $2", + ) + .bind(user_id) + .bind(device_id) + .fetch_optional(pool) + .await?; + Ok(profile_protocol_version( + None, + stored.and_then(|value| u16::try_from(value).ok()), + )) +} + async fn apply_device_profile( pool: &sqlx::PgPool, user_id: i64, profile: &DeviceProfileWire, + version: VersionReport, trusted: bool, ) -> Result<()> { let identity = ensure_identity(pool, user_id, "").await?; @@ -2706,6 +2632,15 @@ async fn apply_device_profile( return Ok(()); } enforce_single_user_binding(pool, user_id, &profile.device_id).await?; + let stored: Option = sqlx::query_scalar( + "SELECT protocol_version FROM furumusic__fed_device + WHERE user_id = $1 AND device_id = $2", + ) + .bind(user_id) + .bind(&profile.device_id) + .fetch_optional(pool) + .await?; + let protocol_version = version.resolve(stored.and_then(|value| u16::try_from(value).ok())); let trusted_at_ms = trusted.then_some(now_ms()).or(Some(profile.updated_at_ms)); sqlx::query( "INSERT INTO furumusic__fed_device @@ -2743,7 +2678,7 @@ async fn apply_device_profile( .bind(&profile.device_id) .bind(&profile.name) .bind(&profile.client_version) - .bind(profile.protocol_version as i32) + .bind(i32::from(protocol_version)) .bind(&profile.endpoint_id) .bind(&profile.endpoint_ticket) .bind(trusted_at_ms) @@ -2848,6 +2783,21 @@ async fn apply_playback_command( let origin = &op.origin_device_id; let op_id = &op.op_id; let identity = ensure_identity(pool, user_id, "").await?; + // A peer below v4 does not know podcast items: its state would silently + // drop them from the local queue. + let peer_version = stored_protocol_version(pool, user_id, origin).await?; + let local = hub + .playback_state_json_for_commands(user_id) + .map(|state| browser_state_shadow(&state)) + .unwrap_or_else(|| queue_items::shadow_wire_state(&[], 0)); + if !playback_command_allowed_from_peer(&local, peer_version) { + tracing::info!( + origin = %short_id(origin), + peer_version, + "ignored playback command from a device without podcast support" + ); + return Ok(()); + } let Some(authority) = authority else { return Ok(()); }; @@ -3007,8 +2957,15 @@ pub async fn record_web_playback_command( current_state: Option, ) -> Result { ensure_web_playback_target(pool, user_id, target_device_id).await?; - let next_state = browser_state_for_command(command, payload, current_state)?; - let wire = playback_state_from_browser_json(pool, next_state.clone()).await?; + let mut next_state = browser_state_for_command(command, payload, current_state)?; + let wire = playback_state_from_browser_json(pool, user_id, next_state.clone()).await?; + // Show the resume position that was sent, not the 0 the browser picked. + if let Some(object) = next_state.as_object_mut() + && object.remove(queue_items::POSITION_UNKNOWN_KEY).is_some() + { + object.insert("position_seconds".to_string(), json_f64(wire.position_secs)); + } + ensure_playback_supported_by(pool, user_id, target_device_id, &wire).await?; record_local_op( pool, user_id, @@ -3036,7 +2993,9 @@ pub async fn record_web_active_transfer( state: serde_json::Value, ) -> Result<()> { ensure_web_playback_target(pool, user_id, target_device_id).await?; - let wire = playback_state_from_browser_json(pool, state).await?; + let wire = playback_state_from_browser_json(pool, user_id, state).await?; + // Refuse instead of stripping podcast episodes from the queue. + ensure_playback_supported_by(pool, user_id, target_device_id, &wire).await?; let target_name = web_playback_target_name(pool, user_id, target_device_id).await?; let identity = ensure_identity(pool, user_id, "").await?; with_playback_engine(pool, user_id, &identity, |engine| { @@ -3081,6 +3040,17 @@ pub async fn record_web_active_transfer( Ok(()) } +/// Checks before a handoff that the target can receive the queue. +pub async fn check_web_transfer( + pool: &sqlx::PgPool, + user_id: i64, + target_device_id: &str, + state: serde_json::Value, +) -> Result<()> { + let wire = playback_state_from_browser_json(pool, user_id, state).await?; + ensure_playback_supported_by(pool, user_id, target_device_id, &wire).await +} + pub async fn record_web_active_takeover( pool: &sqlx::PgPool, user_id: i64, @@ -3089,7 +3059,7 @@ pub async fn record_web_active_takeover( ) -> Result<()> { ensure_web_playback_target(pool, user_id, previous_device_id).await?; let identity = ensure_identity(pool, user_id, "").await?; - let wire = playback_state_from_browser_json(pool, state).await?; + let wire = playback_state_from_browser_json(pool, user_id, state).await?; with_playback_engine(pool, user_id, &identity, |engine| { engine.transfer(&identity.device_id, playback_clock()) }) @@ -3111,6 +3081,28 @@ pub async fn record_web_active_takeover( Ok(()) } +/// Refuses to hand a queue with podcast items to a device below v4. +async fn ensure_playback_supported_by( + pool: &sqlx::PgPool, + user_id: i64, + device_id: &str, + state: &PlaybackStateWire, +) -> Result<()> { + let version = stored_protocol_version(pool, user_id, device_id).await?; + if !playback_allowed_for_peer(state, version) { + let name = web_playback_target_name(pool, user_id, device_id).await?; + anyhow::bail!( + "{name} needs an update to play podcast episodes. Update Furumi on that device, or remove the episodes from the queue." + ); + } + Ok(()) +} + +/// Wire-shaped view of a browser playback state for the shared gating helpers. +fn browser_state_shadow(state: &serde_json::Value) -> PlaybackStateWire { + queue_items::shadow_wire_state(&browser_tracks(state), browser_index(state)) +} + async fn ensure_web_playback_target( pool: &sqlx::PgPool, user_id: i64, @@ -3166,6 +3158,11 @@ fn browser_state_for_command( current_state.unwrap_or_else(empty_browser_playback_state) }; normalize_browser_playback_state(&mut state); + // Only a transfer carries the sender's knowledge of the position; the + // commands below decide for themselves. + if command != "transfer_state" { + browser_object_mut(&mut state).remove(queue_items::POSITION_UNKNOWN_KEY); + } match command { "play_track" | "play_from_index" | "transfer_state" => { @@ -3175,6 +3172,11 @@ fn browser_state_for_command( "paused", browser_bool(&payload, "paused").unwrap_or(false), ); + if command != "transfer_state" { + // The browser always starts the item at 0. For a podcast + // episode, ordinary Play resumes from its progress register. + mark_browser_position_unknown(&mut state); + } } "pause" => set_browser_bool(&mut state, "paused", true), "resume" => { @@ -3392,11 +3394,20 @@ fn advance_browser_state(state: &mut serde_json::Value, payload: &serde_json::Va set_browser_tracks(state, tracks, next); set_browser_number(state, "position_seconds", 0.0); set_browser_bool(state, "paused", false); + if next != current { + mark_browser_position_unknown(state); + } } else { set_browser_bool(state, "paused", true); } } +/// The current item changed without its position: a podcast episode resumes +/// from its progress register (see `playback_state_from_browser_json`). +fn mark_browser_position_unknown(state: &mut serde_json::Value) { + set_browser_bool(state, queue_items::POSITION_UNKNOWN_KEY, true); +} + fn rewind_browser_state(state: &mut serde_json::Value) { let tracks = browser_tracks(state); if tracks.is_empty() { @@ -3427,6 +3438,9 @@ fn rewind_browser_state(state: &mut serde_json::Value) { if let Some(previous) = previous { set_browser_tracks(state, tracks, previous); set_browser_bool(state, "paused", false); + if previous != current { + mark_browser_position_unknown(state); + } } set_browser_number(state, "position_seconds", 0.0); } @@ -3554,11 +3568,14 @@ async fn web_playback_payload( })) } +/// Playback snapshot for a peer, or `None` when the queue has podcast items +/// and the peer is below v4 (`playback_allowed_for_peer`): never strip them. async fn local_playback_snapshot( pool: &sqlx::PgPool, hub: Arc, user_id: i64, identity: &Identity, + peer_version: u16, ) -> Option { let coordination = coordinate_web_output(pool, Arc::clone(&hub), user_id, identity) .await @@ -3570,7 +3587,12 @@ async fn local_playback_snapshot( let state = hub .playback_state_json_for_commands(user_id) .unwrap_or_else(|| serde_json::json!({})); - let wire = playback_state_from_browser_json(pool, state).await.ok()?; + let wire = playback_state_from_browser_json(pool, user_id, state) + .await + .ok()?; + if !playback_allowed_for_peer(&wire, peer_version) { + return None; + } Some(PlaybackSnapshot { device_id: identity.device_id.clone(), device_name: identity.name.clone(), @@ -3694,7 +3716,101 @@ fn playback_clock() -> u64 { .as_millis() as u64 } +/// Wire playback state of a browser state. +/// +/// When the browser does not know the position of its current item +/// ([`queue_items::POSITION_UNKNOWN_KEY`]) and that item is a podcast +/// episode, the position comes from its synced progress register: handing +/// 0 to the next device would be written there as an explicit action and +/// restart the episode everywhere. async fn playback_state_from_browser_json( + pool: &sqlx::PgPool, + user_id: i64, + state: serde_json::Value, +) -> Result { + let unknown = browser_bool(&state, queue_items::POSITION_UNKNOWN_KEY) == Some(true); + let mut wire = browser_json_to_wire(pool, state).await?; + if let Some(episode_id) = episode_needing_resume_position(unknown, &wire) { + let record = PodcastStore::new(pool.clone()) + .progress(user_id, &episode_id) + .await?; + apply_resume_position(&mut wire, record.as_ref()); + } + let episodes: Vec = wire + .queue + .iter() + .filter_map(PlaybackTrack::podcast_episode_ref) + .map(|episode| episode.episode_id.clone()) + .collect(); + if !episodes.is_empty() { + let generations = PodcastStore::new(pool.clone()) + .progress_generations(user_id, &episodes) + .await?; + apply_progress_generations(&mut wire, &generations); + } + Ok(wire) +} + +/// Raises `PodcastEpisodeRef::progress_generation` of every episode in the +/// queue to the generation of its progress register here. +/// +/// The browser keeps the hint its queue item came with, which can be older +/// than what this account has seen since (its own checkpoints, operations +/// from other devices). The device taking over writes its explicit action +/// above this value (frid `PODCASTS.md`, "Handoff"). +fn apply_progress_generations( + state: &mut PlaybackStateWire, + generations: &std::collections::HashMap, +) { + for track in &mut state.queue { + if let Some(PlaybackSource::PodcastEpisode(episode)) = track.source.as_mut() + && let Some(generation) = generations.get(&episode.episode_id) + { + episode.progress_generation = Some( + episode + .progress_generation + .map_or(*generation, |seen| seen.max(*generation)), + ); + } + } +} + +/// The current episode whose position comes from its progress register: +/// only when the browser said it does not know the position. Known +/// positions (handoffs, seeks, playing) and music are never replaced. +fn episode_needing_resume_position(unknown: bool, wire: &PlaybackStateWire) -> Option { + if !unknown { + return None; + } + wire.queue + .get(wire.queue_pos) + .and_then(PlaybackTrack::podcast_episode_ref) + .map(|episode| episode.episode_id.clone()) +} + +/// Sets the position of the current podcast episode to where ordinary Play +/// resumes it (`ProgressRecord::resume_position_ms`, 0 without a register), +/// and records the register's generation as seen for the handoff. +fn apply_resume_position(state: &mut PlaybackStateWire, record: Option<&ProgressRecord>) { + let Some(track) = state.queue.get_mut(state.queue_pos) else { + return; + }; + let Some(PlaybackSource::PodcastEpisode(episode)) = track.source.as_mut() else { + return; + }; + let Some(record) = record.filter(|record| record.episode_id == episode.episode_id) else { + state.position_secs = 0.0; + return; + }; + state.position_secs = record.resume_position_ms().max(0) as f64 / 1000.0; + episode.progress_generation = Some( + episode + .progress_generation + .map_or(record.generation, |seen| seen.max(record.generation)), + ); +} + +async fn browser_json_to_wire( pool: &sqlx::PgPool, state: serde_json::Value, ) -> Result { @@ -3703,19 +3819,27 @@ async fn playback_state_from_browser_json( .and_then(serde_json::Value::as_array) .cloned() .unwrap_or_default(); + let mut queue_pos = state + .get("index") + .and_then(serde_json::Value::as_i64) + .unwrap_or(0) + .max(0) as usize; let mut queue = Vec::new(); - for track in tracks_value { - if let Some(track_id) = track.get("id").and_then(serde_json::Value::as_i64) { - queue.push(playback_track_from_track_json(pool, track_id, &track).await?); + for (index, track) in tracks_value.iter().enumerate() { + // Items that are not library tracks keep their verbatim wire item, + // including the explicit source (podcast episodes, placeholders). + if let Some(wire) = queue_items::wire_track_from_browser_json(track) { + queue.push(wire); + } else if let Some(track_id) = track.get("id").and_then(serde_json::Value::as_i64) { + queue.push(playback_track_from_track_json(pool, track_id, track).await?); + } else if index < queue_pos { + // A malformed entry is skipped; keep the position on its item. + queue_pos -= 1; } } Ok(PlaybackStateWire { queue, - queue_pos: state - .get("index") - .and_then(serde_json::Value::as_i64) - .unwrap_or(0) - .max(0) as usize, + queue_pos, playing: state.get("track").is_some_and(|track| !track.is_null()), paused: state .get("paused") @@ -3746,6 +3870,8 @@ async fn playback_state_from_browser_json( "all" => PlaybackRepeat::All, _ => PlaybackRepeat::Off, }, + // Reserved; the web player has no variable speed. + playback_rate: None, }) } @@ -3754,6 +3880,15 @@ async fn playback_track_from_track_json( track_id: i64, track: &serde_json::Value, ) -> Result { + let content_id = match track + .get("content_id") + .and_then(serde_json::Value::as_str) + .and_then(music_dht::normalize_content_id) + { + Some(content_id) => Some(content_id), + None if track_id > 0 => track_content_id(pool, track_id).await?, + None => None, + }; Ok(PlaybackTrack { id: track_id, title: track @@ -3789,15 +3924,7 @@ async fn playback_track_from_track_json( .and_then(serde_json::Value::as_i64) .map(|value| value as i32), file_path: String::new(), - content_id: track - .get("content_id") - .and_then(serde_json::Value::as_str) - .and_then(music_dht::normalize_content_id) - .or(if track_id > 0 { - track_content_id(pool, track_id).await? - } else { - None - }), + content_id, audio_format: track .get("audio_format") .and_then(serde_json::Value::as_str) @@ -3819,6 +3946,7 @@ async fn playback_track_from_track_json( .and_then(serde_json::Value::as_i64), play_count: 0, fed: None, + source: None, }) } @@ -3836,31 +3964,50 @@ fn names_from_track_json(track: &serde_json::Value, field: &str) -> Vec .unwrap_or_default() } -async fn resolve_playback_tracks( +/// Browser JSON of wire queue items, in order and without dropping any. +/// +/// Music resolved by content id becomes the library track; everything else +/// (unresolved music, podcast episodes, Jam placeholders) becomes an +/// external item that keeps the verbatim wire DTO. +pub(crate) async fn resolve_playback_tracks( pool: &sqlx::PgPool, tracks: &[PlaybackTrack], ) -> Result> { - let mut out = Vec::new(); + let mut out = Vec::with_capacity(tracks.len()); for (index, track) in tracks.iter().enumerate() { - let content_id = track - .content_id - .as_deref() - .or_else(|| track.fed.as_ref().map(|fed| fed.content_id.as_str())) - .and_then(music_dht::normalize_content_id); - let Some(content_id) = content_id else { - out.push(placeholder_playback_track(track, index, None)); - continue; - }; - if let Some(track_json) = web_track_json_by_content_id(pool, &content_id).await? { - out.push(track_json); - } else { - tracing::debug!(content_id, title = %track.title, "fed playback track is not available locally on web"); - out.push(placeholder_playback_track(track, index, Some(&content_id))); - } + out.push(resolve_playback_track(pool, track, index).await?); } Ok(out) } +/// Browser JSON of one wire queue item at queue position `index`. +pub(crate) async fn resolve_playback_track( + pool: &sqlx::PgPool, + track: &PlaybackTrack, + index: usize, +) -> Result { + if track.requires_podcast_support() { + return Ok(queue_items::external_track_json(track, index, None)); + } + let content_id = track + .content_id + .as_deref() + .or_else(|| track.fed.as_ref().map(|fed| fed.content_id.as_str())) + .and_then(music_dht::normalize_content_id); + let Some(content_id) = content_id else { + return Ok(queue_items::external_track_json(track, index, None)); + }; + if let Some(track_json) = web_track_json_by_content_id(pool, &content_id).await? { + return Ok(track_json); + } + tracing::debug!(content_id, title = %track.title, "fed playback track is not available locally on web"); + Ok(queue_items::external_track_json( + track, + index, + Some(&content_id), + )) +} + async fn web_track_json_by_content_id( pool: &sqlx::PgPool, content_id: &str, @@ -3962,83 +4109,6 @@ async fn track_artist_names(pool: &sqlx::PgPool, track_id: i64, role: &str) -> R Ok(rows.into_iter().map(|row| row.get("name")).collect()) } -fn placeholder_playback_track( - track: &PlaybackTrack, - index: usize, - content_id: Option<&str>, -) -> serde_json::Value { - let artists = if !track.artist_names.is_empty() { - track.artist_names.clone() - } else if let Some(fed) = &track.fed { - fed.artist_names.clone() - } else { - Vec::new() - }; - let featured = if !track.featured_artist_names.is_empty() { - track.featured_artist_names.clone() - } else if let Some(fed) = &track.fed { - fed.featured_artist_names.clone() - } else { - Vec::new() - }; - let title = track - .fed - .as_ref() - .map(|fed| fed.title.as_str()) - .filter(|title| !title.trim().is_empty()) - .unwrap_or(&track.title); - let release_title = track - .fed - .as_ref() - .and_then(|fed| fed.release_title.as_deref()) - .filter(|title| !title.trim().is_empty()) - .unwrap_or(&track.release_title); - let content_id = content_id - .map(ToString::to_string) - .or_else(|| track.content_id.clone()) - .or_else(|| track.fed.as_ref().map(|fed| fed.content_id.clone())); - serde_json::json!({ - "id": placeholder_track_id(content_id.as_deref(), index), - "title": title, - "track_number": track.track_number, - "disc_number": track.disc_number, - "duration_seconds": track.duration_seconds, - "artists": artist_names_to_json(&artists), - "featured_artists": artist_names_to_json(&featured), - "release_id": 0, - "release_title": release_title, - "release_year": track.release_year, - "cover_url": serde_json::Value::Null, - "stream_url": "", - "uploader_name": "Federation", - "audio_format": track.audio_format.clone(), - "audio_bitrate": track.audio_bitrate, - "audio_sample_rate": track.audio_sample_rate, - "audio_bit_depth": track.audio_bit_depth, - "file_size_bytes": track.file_size_bytes, - "content_id": content_id, - "unavailable": true, - "unavailable_reason": "Not available on this web player yet", - }) -} - -fn artist_names_to_json(names: &[String]) -> Vec { - names - .iter() - .enumerate() - .filter(|(_, name)| !name.trim().is_empty()) - .map(|(index, name)| serde_json::json!({"id": -((index as i64) + 1), "name": name})) - .collect() -} - -fn placeholder_track_id(content_id: Option<&str>, index: usize) -> i64 { - let mut bytes = [0u8; 8]; - let hash = blake3::hash(content_id.unwrap_or("").as_bytes()); - bytes.copy_from_slice(&hash.as_bytes()[..8]); - let value = i64::from_le_bytes(bytes).unsigned_abs() % 9_000_000_000_000; - -((value as i64) + index as i64 + 1) -} - async fn track_id_by_content_id(pool: &sqlx::PgPool, content_id: &str) -> Result> { let Some(content_id) = music_dht::normalize_content_id(content_id) else { return Ok(None); @@ -4471,11 +4541,23 @@ async fn note_peer_vector( Ok(()) } +/// Operations the peer has not acknowledged and can parse. +/// +/// Kinds the peer's version does not know are excluded in SQL; skipped +/// operations are never re-sent after an upgrade (vectors only grow), the +/// snapshot is the recovery path for them. async fn ops_for_peer( pool: &sqlx::PgPool, user_id: i64, peer_device_id: &str, + peer_version: u16, ) -> Result> { + // This server only speaks `furumi/sync/2`, so every peer it exchanges + // operations with implements v2 (listen history) even when it never + // reported a version (an invite without `p`, a device known only from + // `device_trusted`). Without this floor listen events would be skipped + // for good: they are not part of snapshots. + let peer_version = peer_version.max(SYNC_V2_PROTOCOL_VERSION); let rows = sqlx::query( "SELECT o.op_id, o.origin_device_id, o.seq, o.hlc_ms, o.payload_json FROM furumusic__fed_sync_ops o @@ -4492,6 +4574,7 @@ async fn ops_for_peer( AND o.payload_json->>'target_device_id' = $2 ) ) + AND NOT (o.kind = ANY($5)) ORDER BY o.hlc_ms, o.op_id LIMIT $4", ) @@ -4499,20 +4582,25 @@ async fn ops_for_peer( .bind(peer_device_id) .bind(now_ms().saturating_sub(PLAYBACK_COMMAND_TTL_MS)) .bind(MAX_OPS_PER_BATCH) + .bind(op_kinds_hidden_from(peer_version)) .fetch_all(pool) .await?; - rows.into_iter() - .map(|row| { - let payload_json: serde_json::Value = row.get("payload_json"); - Ok(SyncOpWire { - op_id: row.get("op_id"), - origin_device_id: row.get("origin_device_id"), - seq: row.get("seq"), - hlc_ms: row.get("hlc_ms"), - payload: serde_json::from_value(payload_json)?, - }) - }) - .collect() + let mut ops = Vec::with_capacity(rows.len()); + for row in rows { + let payload_json: serde_json::Value = row.get("payload_json"); + let op = SyncOpWire { + op_id: row.get("op_id"), + origin_device_id: row.get("origin_device_id"), + seq: row.get("seq"), + hlc_ms: row.get("hlc_ms"), + payload: serde_json::from_value(payload_json)?, + }; + // Playback commands carrying podcast items need v4 as well. + if op_allowed_for_peer(&op.payload, peer_version) { + ops.push(op); + } + } + Ok(ops) } async fn snapshot(pool: &sqlx::PgPool, user_id: i64) -> Result { @@ -4601,13 +4689,23 @@ async fn snapshot(pool: &sqlx::PgPool, user_id: i64) -> Result { op_id: row.get("op_id"), }) .collect(); - Ok(SyncSnapshot { + let mut snapshot = SyncSnapshot { likes, unlikes, playlists, deleted_playlists, removed_playlist_items, - }) + ..SyncSnapshot::default() + }; + // Bounded again per message by `fit_podcast_snapshot`. + PodcastStore::new(pool.clone()) + .fill_snapshot( + user_id, + &mut snapshot, + music_dht::podcast::MAX_PODCAST_SNAPSHOT_BYTES, + ) + .await?; + Ok(snapshot) } async fn snapshot_playlist_items( @@ -4839,6 +4937,7 @@ async fn write_sync_reject(mut stream: ByteStream, message: &str) -> Result<()> finish_response(&mut stream).await } +/// Operation `kind` tag as stored in `furumusic__fed_sync_ops.kind`. fn payload_kind(payload: &SyncOpPayload) -> &'static str { match payload { SyncOpPayload::TrackLikeSet { .. } => "track_like_set", @@ -4852,70 +4951,83 @@ fn payload_kind(payload: &SyncOpPayload) -> &'static str { SyncOpPayload::DeviceRevoked { .. } => "device_revoked", SyncOpPayload::PlaybackCommand { .. } => "playback_command", SyncOpPayload::ListenRecorded { .. } => "listen_recorded", + SyncOpPayload::PodcastSubscriptionSet { .. } => "podcast_subscription_set", + SyncOpPayload::PodcastShareSet { .. } => "podcast_share_set", + SyncOpPayload::PodcastFeedAliasesAdded { .. } => "podcast_feed_aliases_added", + SyncOpPayload::PodcastProgressSet { .. } => "podcast_progress_set", + SyncOpPayload::PodcastListenRecorded { .. } => "podcast_listen_recorded", } } +/// Stored operation kinds a peer of `peer_version` cannot parse. +/// +/// Filtered in SQL so a batch of hidden operations cannot stall the +/// per-peer cursor; `op_allowed_for_peer` stays the final check. +fn op_kinds_hidden_from(peer_version: u16) -> Vec<&'static str> { + let mut kinds = Vec::new(); + if !supports_listen_history(peer_version) { + kinds.push("listen_recorded"); + } + if !supports_podcasts(peer_version) { + kinds.extend([ + "podcast_subscription_set", + "podcast_share_set", + "podcast_feed_aliases_added", + "podcast_progress_set", + "podcast_listen_recorded", + ]); + } + kinds +} + +/// Bounds the podcast snapshot sections so the whole line stays under the +/// protocol limit (`podcast_snapshot_budget`). +fn fit_podcast_snapshot(message: &mut WireMessage) { + let snapshot = match message { + WireMessage::PairRequest { snapshot, .. } + | WireMessage::PairResponse { snapshot, .. } + | WireMessage::Hello { snapshot, .. } + | WireMessage::SyncResponse { snapshot, .. } => snapshot, + }; + if !snapshot.has_podcast_state() { + return; + } + let podcast = snapshot.clone(); + snapshot.strip_podcast_state(); + let other_bytes = serde_json::to_vec(message).map_or(usize::MAX, |bytes| bytes.len()); + let snapshot = match message { + WireMessage::PairRequest { snapshot, .. } + | WireMessage::PairResponse { snapshot, .. } + | WireMessage::Hello { snapshot, .. } + | WireMessage::SyncResponse { snapshot, .. } => snapshot, + }; + *snapshot = podcast; + snapshot.bound_podcast_state_to(podcast_snapshot_budget(other_bytes)); +} + async fn write_msg(stream: &mut ByteStream, message: &WireMessage) -> Result<()> { - let mut payload = serde_json::to_vec(message)?; - payload.push(b'\n'); - stream.send.write_all(&payload).await?; + let mut message = message.clone(); + fit_podcast_snapshot(&mut message); + music_dht::device_sync::write_msg(stream, &message).await?; Ok(()) } async fn finish_send(stream: &mut ByteStream) -> Result<()> { - stream.send.finish()?; + music_dht::device_sync::finish_send(stream).await?; Ok(()) } async fn finish_response(stream: &mut ByteStream) -> Result<()> { - stream.send.finish()?; - let _ = tokio::time::timeout(RESPONSE_DRAIN_TIMEOUT, stream.send.stopped()).await; + music_dht::device_sync::finish_response(stream, RESPONSE_DRAIN_TIMEOUT).await?; Ok(()) } async fn read_msg(stream: &mut ByteStream) -> Result { - let line = read_line(&mut stream.recv).await?; - Ok(serde_json::from_slice(&line)?) -} - -async fn read_line(reader: &mut R) -> Result> { - let mut out = Vec::new(); - loop { - let mut byte = [0u8; 1]; - let read = reader.read(&mut byte).await?; - if read == 0 { - break; - } - if byte[0] == b'\n' { - break; - } - out.push(byte[0]); - if out.len() > MAX_LINE { - anyhow::bail!("protocol line is too large"); - } - } - Ok(out) -} - -fn parse_invite(value: &str) -> Result { - let Some(token) = value.trim().strip_prefix("frid://i/") else { - anyhow::bail!("usage: frid://i/"); - }; - let bytes = base64url_decode(token)?; - let invite: InviteWire = serde_json::from_slice(&bytes)?; - anyhow::ensure!(invite.v == 1, "unsupported invite version"); - Ok(invite) + Ok(music_dht::device_sync::read_msg(stream).await?) } pub fn invite_network_id(value: &str) -> Result { - let invite = parse_invite(value)?; - let ticket: PeerTicket = invite.ticket.parse().context("malformed invite ticket")?; - Ok(ticket.network_id) -} - -fn encode_invite(invite: &InviteWire) -> Result { - let bytes = serde_json::to_vec(invite)?; - Ok(format!("frid://i/{}", base64url_encode(&bytes))) + Ok(music_dht::device_sync::invite_network_id(value)?) } fn pair_request_id(invite_id: &str, device_id: &str) -> String { @@ -4923,24 +5035,6 @@ fn pair_request_id(invite_id: &str, device_id: &str) -> String { format!("pair_{}", &digest.to_hex()[..16]) } -fn hash_secret(secret: &str) -> String { - blake3::hash(secret.as_bytes()).to_hex().to_string() -} - -fn random_hex(bytes: usize) -> String { - let key = SecretKey::generate(); - let mut seed = key.to_bytes().to_vec(); - while seed.len() < bytes { - seed.extend_from_slice(blake3::hash(&seed).as_bytes()); - } - hex_encode(&seed[..bytes]) -} - -fn ticket_endpoint_id(ticket: &str) -> Option { - let ticket = PeerTicket::from_str(ticket).ok()?; - Some(ticket.endpoint_id().to_string()) -} - fn repeat_label(repeat: PlaybackRepeat) -> &'static str { match repeat { PlaybackRepeat::Off => "off", @@ -4987,62 +5081,11 @@ fn short_id(value: &str) -> String { value.chars().take(10).collect() } -fn hex_encode(bytes: &[u8]) -> String { - bytes.iter().map(|byte| format!("{byte:02x}")).collect() -} - -fn base64url_encode(bytes: &[u8]) -> String { - const TABLE: &[u8; 64] = b"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_"; - let mut out = String::new(); - let mut i = 0; - while i < bytes.len() { - let b0 = bytes[i]; - let b1 = bytes.get(i + 1).copied().unwrap_or(0); - let b2 = bytes.get(i + 2).copied().unwrap_or(0); - out.push(TABLE[(b0 >> 2) as usize] as char); - out.push(TABLE[(((b0 & 0b0000_0011) << 4) | (b1 >> 4)) as usize] as char); - if i + 1 < bytes.len() { - out.push(TABLE[(((b1 & 0b0000_1111) << 2) | (b2 >> 6)) as usize] as char); - } - if i + 2 < bytes.len() { - out.push(TABLE[(b2 & 0b0011_1111) as usize] as char); - } - i += 3; - } - out -} - -fn base64url_decode(value: &str) -> Result> { - fn val(byte: u8) -> Option { - match byte { - b'A'..=b'Z' => Some(byte - b'A'), - b'a'..=b'z' => Some(byte - b'a' + 26), - b'0'..=b'9' => Some(byte - b'0' + 52), - b'-' => Some(62), - b'_' => Some(63), - _ => None, - } - } - let bytes = value.as_bytes(); - let mut out = Vec::with_capacity(bytes.len() * 3 / 4); - let mut i = 0; - while i < bytes.len() { - let a = val(bytes[i]).context("invalid base64url invite")?; - let b = val(*bytes.get(i + 1).context("truncated base64url invite")?) - .context("invalid base64url invite")?; - let c = bytes.get(i + 2).and_then(|byte| val(*byte)); - let d = bytes.get(i + 3).and_then(|byte| val(*byte)); - out.push((a << 2) | (b >> 4)); - if let Some(c) = c { - out.push((b << 4) | (c >> 2)); - if let Some(d) = d { - out.push((c << 6) | d); - } - } - i += 4; - } - Ok(out) -} - +#[cfg(test)] +mod gating_tests; #[cfg(test)] mod interop_tests; +#[cfg(test)] +mod postgres_tests; +#[cfg(test)] +mod wire_compat_tests; diff --git a/src/federation/devices/fixtures/v3_hello.golden.json b/src/federation/devices/fixtures/v3_hello.golden.json new file mode 100644 index 0000000..ad7622e --- /dev/null +++ b/src/federation/devices/fixtures/v3_hello.golden.json @@ -0,0 +1 @@ +{"type":"hello","group_id":"grp_1","profile":{"device_id":"tui_a","name":"TUI","client_version":"0.3.4","protocol_version":3,"endpoint_id":"e","endpoint_ticket":"t","revoked":false,"revoke_cutoff_seq":null,"updated_at_ms":5},"devices":[{"device_id":"tui_a","name":"TUI","client_version":"0.3.4","protocol_version":3,"endpoint_id":"e","endpoint_ticket":"t","revoked":false,"revoke_cutoff_seq":null,"updated_at_ms":5},{"device_id":"web_b","name":"TUI","client_version":"0.3.4","protocol_version":3,"endpoint_id":"e","endpoint_ticket":"t","revoked":false,"revoke_cutoff_seq":null,"updated_at_ms":5}],"vector":{"tui_a":12,"web_b":4},"ops":[{"op_id":"tui_a:1","origin_device_id":"tui_a","seq":1,"hlc_ms":1700000000001,"payload":{"kind":"track_like_set","content_id":"b3:1111111111111111111111111111111111111111111111111111111111111111","liked":true,"fed":{"item_id":"item","owner":"owner","title":"Song","artist_names":["A"],"featured_artist_names":[],"year":2001,"duration_seconds":180,"content_id":"b3:1111111111111111111111111111111111111111111111111111111111111111","release_title":"Album","track_number":1,"disc_number":1}}},{"op_id":"tui_a:2","origin_device_id":"tui_a","seq":2,"hlc_ms":1700000000002,"payload":{"kind":"playlist_created","playlist_id":"pl","title":"Mix"}},{"op_id":"tui_a:3","origin_device_id":"tui_a","seq":3,"hlc_ms":1700000000003,"payload":{"kind":"playlist_renamed","playlist_id":"pl","title":"Mix 2"}},{"op_id":"tui_a:4","origin_device_id":"tui_a","seq":4,"hlc_ms":1700000000004,"payload":{"kind":"playlist_track_added","playlist_id":"pl","content_id":"b3:1111111111111111111111111111111111111111111111111111111111111111","position":0,"fed":null}},{"op_id":"tui_a:5","origin_device_id":"tui_a","seq":5,"hlc_ms":1700000000005,"payload":{"kind":"playlist_track_removed","playlist_id":"pl","content_id":"b3:1111111111111111111111111111111111111111111111111111111111111111"}},{"op_id":"tui_a:6","origin_device_id":"tui_a","seq":6,"hlc_ms":1700000000006,"payload":{"kind":"playlist_deleted","playlist_id":"pl"}},{"op_id":"tui_a:7","origin_device_id":"tui_a","seq":7,"hlc_ms":1700000000007,"payload":{"kind":"device_profile_set","name":"TUI","client_version":"0.3.4","endpoint_ticket":"t","endpoint_id":"e"}},{"op_id":"tui_a:8","origin_device_id":"tui_a","seq":8,"hlc_ms":1700000000008,"payload":{"kind":"device_trusted","target_device_id":"web_b"}},{"op_id":"tui_a:9","origin_device_id":"tui_a","seq":9,"hlc_ms":1700000000009,"payload":{"kind":"device_revoked","target_device_id":"old","target_max_seq_seen":3}},{"op_id":"tui_a:10","origin_device_id":"tui_a","seq":10,"hlc_ms":1700000000010,"payload":{"kind":"playback_command","target_device_id":"web_b","command":{"kind":"set_state","state":{"queue":[{"id":7,"title":"Song","track_number":1,"disc_number":null,"duration_seconds":180.5,"artist_names":["A"],"featured_artist_names":["B"],"release_id":3,"release_title":"Album","release_year":2001,"file_path":"","content_id":"b3:1111111111111111111111111111111111111111111111111111111111111111","audio_format":"flac","audio_bitrate":900,"audio_sample_rate":44100,"audio_bit_depth":16,"file_size_bytes":12345,"play_count":2,"fed":{"item_id":"item","owner":"owner","title":"Song","artist_names":["A"],"featured_artist_names":[],"year":2001,"duration_seconds":180,"content_id":"b3:1111111111111111111111111111111111111111111111111111111111111111","release_title":"Album","track_number":1,"disc_number":1}},{"id":0,"title":"Remote","track_number":null,"disc_number":null,"duration_seconds":0.0,"artist_names":[],"featured_artist_names":[],"release_id":0,"release_title":"","release_year":null,"file_path":"","content_id":null,"audio_format":null,"audio_bitrate":null,"audio_sample_rate":null,"audio_bit_depth":null,"file_size_bytes":null,"play_count":0,"fed":null}],"queue_pos":1,"playing":true,"paused":false,"idle_since_ms":null,"position_secs":42.5,"volume":73,"shuffle":true,"repeat":"all"},"seek":true},"authority":{"version":1,"claim":{"counter":3,"issuer":"tui_a","owner":"tui_a"},"sequence":9}}},{"op_id":"tui_a:11","origin_device_id":"tui_a","seq":11,"hlc_ms":1700000000011,"payload":{"kind":"playback_command","target_device_id":"web_b","command":{"kind":"active_changed","active_device_id":"web_b","active_device_name":"WEB","state":{"queue":[{"id":7,"title":"Song","track_number":1,"disc_number":null,"duration_seconds":180.5,"artist_names":["A"],"featured_artist_names":["B"],"release_id":3,"release_title":"Album","release_year":2001,"file_path":"","content_id":"b3:1111111111111111111111111111111111111111111111111111111111111111","audio_format":"flac","audio_bitrate":900,"audio_sample_rate":44100,"audio_bit_depth":16,"file_size_bytes":12345,"play_count":2,"fed":{"item_id":"item","owner":"owner","title":"Song","artist_names":["A"],"featured_artist_names":[],"year":2001,"duration_seconds":180,"content_id":"b3:1111111111111111111111111111111111111111111111111111111111111111","release_title":"Album","track_number":1,"disc_number":1}},{"id":0,"title":"Remote","track_number":null,"disc_number":null,"duration_seconds":0.0,"artist_names":[],"featured_artist_names":[],"release_id":0,"release_title":"","release_year":null,"file_path":"","content_id":null,"audio_format":null,"audio_bitrate":null,"audio_sample_rate":null,"audio_bit_depth":null,"file_size_bytes":null,"play_count":0,"fed":null}],"queue_pos":1,"playing":true,"paused":false,"idle_since_ms":null,"position_secs":42.5,"volume":73,"shuffle":true,"repeat":"all"}},"authority":null}},{"op_id":"tui_a:12","origin_device_id":"tui_a","seq":12,"hlc_ms":1700000000012,"payload":{"kind":"listen_recorded","event":{"listen_id":"l1","content_id":"b3:1111111111111111111111111111111111111111111111111111111111111111","started_at_ms":1,"listened_ms":60000,"track_duration_ms":180000,"ended_reason":"finished","track":{"title":"Song","artist_names":["A"],"featured_artist_names":[],"release_title":null}}}}],"snapshot":{"likes":[{"content_id":"b3:1111111111111111111111111111111111111111111111111111111111111111","hlc_ms":1,"op_id":"tui_a:1","fed":{"item_id":"item","owner":"owner","title":"Song","artist_names":["A"],"featured_artist_names":[],"year":2001,"duration_seconds":180,"content_id":"b3:1111111111111111111111111111111111111111111111111111111111111111","release_title":"Album","track_number":1,"disc_number":1}}],"unlikes":[{"content_id":"b3:1111111111111111111111111111111111111111111111111111111111111111","hlc_ms":2,"op_id":"tui_a:2"}],"playlists":[{"playlist_id":"pl","title":"Mix","hlc_ms":3,"op_id":"tui_a:3","items":[{"content_id":"b3:1111111111111111111111111111111111111111111111111111111111111111","position":0,"hlc_ms":4,"op_id":"tui_a:4","fed":null}]}],"deleted_playlists":[{"playlist_id":"gone","hlc_ms":5,"op_id":"tui_a:5"}],"removed_playlist_items":[{"playlist_id":"pl","content_id":"b3:1111111111111111111111111111111111111111111111111111111111111111","hlc_ms":6,"op_id":"tui_a:6"}]},"playback":{"device_id":"tui_a","device_name":"TUI","active":true,"updated_at_ms":9,"state":{"queue":[{"id":7,"title":"Song","track_number":1,"disc_number":null,"duration_seconds":180.5,"artist_names":["A"],"featured_artist_names":["B"],"release_id":3,"release_title":"Album","release_year":2001,"file_path":"","content_id":"b3:1111111111111111111111111111111111111111111111111111111111111111","audio_format":"flac","audio_bitrate":900,"audio_sample_rate":44100,"audio_bit_depth":16,"file_size_bytes":12345,"play_count":2,"fed":{"item_id":"item","owner":"owner","title":"Song","artist_names":["A"],"featured_artist_names":[],"year":2001,"duration_seconds":180,"content_id":"b3:1111111111111111111111111111111111111111111111111111111111111111","release_title":"Album","track_number":1,"disc_number":1}},{"id":0,"title":"Remote","track_number":null,"disc_number":null,"duration_seconds":0.0,"artist_names":[],"featured_artist_names":[],"release_id":0,"release_title":"","release_year":null,"file_path":"","content_id":null,"audio_format":null,"audio_bitrate":null,"audio_sample_rate":null,"audio_bit_depth":null,"file_size_bytes":null,"play_count":0,"fed":null}],"queue_pos":1,"playing":true,"paused":false,"idle_since_ms":null,"position_secs":42.5,"volume":73,"shuffle":true,"repeat":"all"},"coordination":{"version":1,"claim":{"counter":3,"issuer":"tui_a","owner":"tui_a"},"heartbeat":4,"available":true,"playing":true}}} diff --git a/src/federation/devices/gating_tests.rs b/src/federation/devices/gating_tests.rs new file mode 100644 index 0000000..c12ac9f --- /dev/null +++ b/src/federation/devices/gating_tests.rs @@ -0,0 +1,621 @@ +//! Version gating and lossless queue bridge (device-sync protocol v4). + +use super::*; +use crate::player::queue_items::{self, test_episode}; +use music_dht::podcast::{ + EpisodeId, FeedUrl, ListenSession, PodcastId, PodcastMeta, ProgressStatus, +}; + +const CONTENT_ID: &str = "b3:2222222222222222222222222222222222222222222222222222222222222222"; + +fn podcast_id() -> PodcastId { + PodcastId::from_feed_url(&FeedUrl::parse("https://podcasts.example/feed.xml").unwrap()) +} + +fn episode_id() -> EpisodeId { + EpisodeId::from_guid(&podcast_id(), "ep-1").unwrap() +} + +fn music_track(id: i64) -> PlaybackTrack { + let mut track = PlaybackTrack::private_placeholder(); + track.source = None; + track.id = id; + track.title = format!("Song {id}"); + track +} + +fn state(queue: Vec) -> PlaybackStateWire { + PlaybackStateWire { + queue, + queue_pos: 0, + playing: true, + paused: false, + idle_since_ms: None, + position_secs: 12.5, + volume: 70, + shuffle: false, + repeat: PlaybackRepeat::Off, + playback_rate: None, + } +} + +fn podcast_state() -> PlaybackStateWire { + state(vec![ + music_track(1), + PlaybackTrack::podcast_episode(test_episode()), + ]) +} + +/// One payload of every operation kind (music-only playback commands). +fn every_payload(playback: PlaybackStateWire) -> Vec { + let session = ListenSession { + session_id: "s1".into(), + episode_id: episode_id(), + podcast_id: podcast_id(), + device_id: "tui_a".into(), + started_at_ms: 1, + ended_at_ms: 2, + start_position_ms: 0, + end_position_ms: 1, + listened_ms: 1, + completed: false, + }; + let meta: PodcastMeta = serde_json::from_value(serde_json::json!({ + "feed_url": "https://podcasts.example/feed.xml", "title": "Cast" + })) + .unwrap(); + vec![ + SyncOpPayload::TrackLikeSet { + content_id: CONTENT_ID.into(), + liked: true, + fed: None, + }, + SyncOpPayload::PlaylistCreated { + playlist_id: "p".into(), + title: "t".into(), + }, + SyncOpPayload::PlaylistRenamed { + playlist_id: "p".into(), + title: "t".into(), + }, + SyncOpPayload::PlaylistDeleted { + playlist_id: "p".into(), + }, + SyncOpPayload::PlaylistTrackAdded { + playlist_id: "p".into(), + content_id: CONTENT_ID.into(), + position: 0, + fed: None, + }, + SyncOpPayload::PlaylistTrackRemoved { + playlist_id: "p".into(), + content_id: CONTENT_ID.into(), + }, + SyncOpPayload::DeviceProfileSet { + name: "n".into(), + client_version: "v".into(), + endpoint_ticket: "t".into(), + endpoint_id: "e".into(), + protocol_version: Some(4), + }, + SyncOpPayload::DeviceTrusted { + target_device_id: "d".into(), + }, + SyncOpPayload::DeviceRevoked { + target_device_id: "d".into(), + target_max_seq_seen: 1, + }, + SyncOpPayload::PlaybackCommand { + target_device_id: "d".into(), + command: PlaybackCommand::SetState { + state: playback, + seek: false, + }, + authority: None, + }, + SyncOpPayload::ListenRecorded { + event: ListenEvent { + listen_id: "l".into(), + content_id: CONTENT_ID.into(), + started_at_ms: 1, + listened_ms: 1, + track_duration_ms: None, + ended_reason: ListenEndReason::Finished, + track: ListenTrackMetadata { + title: "t".into(), + artist_names: vec!["a".into()], + featured_artist_names: vec![], + release_title: None, + }, + }, + }, + SyncOpPayload::PodcastSubscriptionSet { + podcast_id: podcast_id(), + subscribed: true, + podcast: meta, + feed_aliases: vec![], + generation: 1, + }, + SyncOpPayload::PodcastShareSet { + podcast_id: podcast_id(), + shared: false, + generation: 1, + }, + SyncOpPayload::PodcastFeedAliasesAdded { + podcast_id: podcast_id(), + feed_aliases: vec![FeedUrl::parse("https://podcasts.example/new.xml").unwrap()], + }, + SyncOpPayload::PodcastProgressSet { + episode_id: episode_id(), + podcast_id: podcast_id(), + position_ms: 1, + status: ProgressStatus::InProgress, + generation: 1, + session_id: "s1".into(), + episode: None, + }, + SyncOpPayload::PodcastListenRecorded { + session, + episode: None, + }, + ] +} + +#[test] +fn stored_kind_is_the_wire_tag() { + for payload in every_payload(state(vec![])) { + let value = serde_json::to_value(&payload).unwrap(); + assert_eq!(payload_kind(&payload), value["kind"].as_str().unwrap()); + } +} + +#[test] +fn op_gating_matrix_matches_the_shared_helper() { + // The SQL kind filter must hide exactly what `op_allowed_for_peer` + // rejects (playback commands are content-dependent and checked in Rust). + for version in 1..=PROTOCOL_VERSION { + let hidden = op_kinds_hidden_from(version); + for payload in every_payload(state(vec![music_track(1)])) { + let kind = payload_kind(&payload); + assert_eq!( + hidden.contains(&kind), + !op_allowed_for_peer(&payload, version), + "{kind} for v{version}" + ); + } + } + assert!( + op_kinds_hidden_from(3) + .iter() + .all(|kind| kind.starts_with("podcast_")) + ); + assert_eq!(op_kinds_hidden_from(3).len(), 5); + assert!(op_kinds_hidden_from(4).is_empty()); +} + +#[test] +fn playback_gating_matrix() { + let music = state(vec![music_track(1), music_track(2)]); + let podcasts = podcast_state(); + let private = state(vec![PlaybackTrack::private_placeholder()]); + for version in 1..=PROTOCOL_VERSION { + let v4 = version >= 4; + // Snapshots and handoffs. + assert!(playback_allowed_for_peer(&music, version)); + assert_eq!(playback_allowed_for_peer(&podcasts, version), v4); + assert_eq!(playback_allowed_for_peer(&private, version), v4); + // Commands from the peer. + assert!(playback_command_allowed_from_peer(&music, version)); + assert_eq!(playback_command_allowed_from_peer(&podcasts, version), v4); + // Stamped playback command operations. + let command = |state: &PlaybackStateWire| SyncOpPayload::PlaybackCommand { + target_device_id: "d".into(), + command: PlaybackCommand::ActiveChanged { + active_device_id: "d".into(), + active_device_name: "D".into(), + state: state.clone(), + }, + authority: None, + }; + assert!(op_allowed_for_peer(&command(&music), version)); + assert_eq!(op_allowed_for_peer(&command(&podcasts), version), v4); + } +} + +#[test] +fn browser_state_gating_sees_podcast_items() { + let with_podcast = serde_json::json!({ + "tracks": [ + {"id": 5, "title": "Library song"}, + queue_items::external_track_json(&PlaybackTrack::podcast_episode(test_episode()), 1, None), + ], + "index": 0, + }); + let shadow = browser_state_shadow(&with_podcast); + assert!(!playback_command_allowed_from_peer(&shadow, 3)); + assert!(playback_command_allowed_from_peer(&shadow, 4)); + let music_only = serde_json::json!({"tracks": [{"id": 5}], "index": 0}); + assert!(playback_command_allowed_from_peer( + &browser_state_shadow(&music_only), + 3 + )); +} + +#[test] +fn relayed_versions_never_downgrade_and_self_reports_replace() { + let profile = |id: &str, version: u16| DeviceProfileWire { + device_id: id.into(), + name: "D".into(), + client_version: "x".into(), + protocol_version: version, + endpoint_id: String::new(), + endpoint_ticket: String::new(), + revoked: false, + revoke_cutoff_seq: None, + updated_at_ms: 0, + }; + // A v3 relay lists every device with its own constant. + let relayed = VersionReport::for_listed(&profile("tui_b", 3), Some("tui_a")); + assert_eq!(relayed, VersionReport::Relayed(3)); + assert_eq!(relayed.resolve(Some(4)), 4); + assert_eq!(relayed.resolve(None), 3); + assert_eq!(VersionReport::Relayed(4).resolve(Some(3)), 4); + // The sender's own entry is a self-report and may lower the version. + let own = VersionReport::for_listed(&profile("tui_a", 3), Some("tui_a")); + assert_eq!(own, VersionReport::SelfReported(Some(3))); + assert_eq!(own.resolve(Some(4)), 3); + // Profiles without a version (pending pairing, pre-v4 DeviceProfileSet) + // keep the stored value, or the unreported version 1. + assert_eq!(VersionReport::SelfReported(None).resolve(Some(4)), 4); + assert_eq!(VersionReport::SelfReported(None).resolve(None), 1); +} + +#[test] +fn podcast_snapshot_sections_are_bounded_per_message() { + let mut snapshot = SyncSnapshot::default(); + let progress = music_dht::podcast::ProgressRecord { + episode_id: episode_id(), + podcast_id: podcast_id(), + position_ms: 1, + status: ProgressStatus::InProgress, + generation: 1, + session_id: "s".into(), + hlc_ms: 1, + device_id: "tui_a".into(), + op_id: "tui_a:1".into(), + }; + snapshot.podcast_progress = vec![progress; 10]; + // Large other parts leave (almost) no budget for podcast sections. + let big_title = "x".repeat(music_dht::device_sync::MAX_SYNC_LINE - 16 * 1024); + let mut message = WireMessage::Hello { + group_id: "g".into(), + profile: DeviceProfileWire { + device_id: "web".into(), + name: big_title, + client_version: String::new(), + protocol_version: PROTOCOL_VERSION, + endpoint_id: String::new(), + endpoint_ticket: String::new(), + revoked: false, + revoke_cutoff_seq: None, + updated_at_ms: 0, + }, + devices: vec![], + vector: BTreeMap::new(), + ops: vec![], + snapshot: snapshot.clone(), + playback: None, + }; + fit_podcast_snapshot(&mut message); + let WireMessage::Hello { + snapshot: fitted, .. + } = &message + else { + unreachable!() + }; + assert!(fitted.podcast_progress.is_empty()); + + let mut small = WireMessage::SyncResponse { + accepted: true, + error: None, + devices: vec![], + vector: BTreeMap::new(), + ops: vec![], + snapshot, + playback: None, + }; + fit_podcast_snapshot(&mut small); + let WireMessage::SyncResponse { snapshot, .. } = &small else { + unreachable!() + }; + assert_eq!(snapshot.podcast_progress.len(), 10); +} + +/// Pool that is never connected: the bridge paths under test resolve +/// podcast items, placeholders and inline content ids without queries. +fn offline_pool() -> sqlx::PgPool { + sqlx::postgres::PgPoolOptions::new() + .connect_lazy("postgres://offline@127.0.0.1:1/offline") + .unwrap() +} + +#[tokio::test] +async fn podcast_queue_survives_server_browser_server_round_trip() { + let pool = offline_pool(); + let mut other = test_episode(); + other.episode_title = "Episode 43".into(); + other.episode_id = EpisodeId::from_guid(&other.podcast_id, "episode-43").unwrap(); + let mut unresolved = music_track(0); + unresolved.title = "Remote song".into(); + let queue = vec![ + PlaybackTrack::podcast_episode(test_episode()), + PlaybackTrack::private_placeholder(), + unresolved, + PlaybackTrack::podcast_episode(other), + ]; + let mut incoming = state(queue.clone()); + incoming.queue_pos = 3; + incoming.paused = true; + incoming.position_secs = 1200.0; + + // peer -> server -> browser + let payload = web_playback_payload(&pool, &incoming).await.unwrap(); + let tracks = payload["tracks"].as_array().unwrap(); + assert_eq!(tracks.len(), 4, "no item is dropped"); + assert_eq!(payload["index"], 3); + assert_eq!(payload["position_seconds"], 1200.0); + assert_eq!(tracks[0]["source"]["kind"], "podcast_episode"); + assert_eq!( + tracks[0]["unavailable"], false, + "episodes play on the web player" + ); + assert_eq!(tracks[1]["source"]["kind"], "private"); + + // The browser keeps its queue and sends it back (plus a library track + // it added at the end, with its content id inline). + let mut browser: serde_json::Value = serde_json::from_str(&payload.to_string()).unwrap(); + browser["tracks"] + .as_array_mut() + .unwrap() + .push(serde_json::json!({ + "id": 77, "title": "Library song", "content_id": CONTENT_ID, + "duration_seconds": 100.0, "artists": [{"id": 1, "name": "A"}], + "release_id": 9, "release_title": "Album", + })); + + // browser -> server -> peer (the bridge itself; the progress-register + // lookups of `playback_state_from_browser_json` need a database and + // are covered by `apply_progress_generations_*` and the Postgres tests) + let outgoing = browser_json_to_wire(&pool, browser).await.unwrap(); + assert_eq!(outgoing.queue[..4], queue[..]); + assert_eq!(outgoing.queue_pos, 3); + assert!(outgoing.paused); + // The episode position goes back to the peer (handoff web -> TUI). + assert_eq!(outgoing.position_secs, 1200.0); + let library = &outgoing.queue[4]; + assert_eq!(library.id, 77); + assert_eq!(library.content_id.as_deref(), Some(CONTENT_ID)); + assert_eq!(library.source, None); + // Library music keeps the v3 shape. + assert!(!serde_json::to_string(library).unwrap().contains("source")); + // Peers below v4 get no playback snapshot with these items. + assert!(!playback_allowed_for_peer(&outgoing, 3)); + assert!(playback_allowed_for_peer(&outgoing, 4)); +} + +#[test] +fn apply_progress_generations_raises_stale_handoff_hints_only() { + let mut first = test_episode(); + first.progress_generation = Some(9); + let mut second = test_episode(); + second.episode_id = EpisodeId::from_guid(&second.podcast_id, "episode-43").unwrap(); + second.progress_generation = None; + let mut third = test_episode(); + third.episode_id = EpisodeId::from_guid(&third.podcast_id, "episode-44").unwrap(); + third.progress_generation = Some(2); + let mut wire = state(vec![ + PlaybackTrack::podcast_episode(first.clone()), + PlaybackTrack::podcast_episode(second.clone()), + music_track(3), + PlaybackTrack::podcast_episode(third.clone()), + ]); + let generations = std::collections::HashMap::from([ + (first.episode_id.clone(), 4), + (second.episode_id.clone(), 6), + (third.episode_id.clone(), 5), + ]); + apply_progress_generations(&mut wire, &generations); + let hints: Vec> = wire + .queue + .iter() + .map(|track| { + track + .podcast_episode_ref() + .and_then(|e| e.progress_generation) + }) + .collect(); + // A newer carried hint stays; missing and older ones are raised. + assert_eq!(hints, vec![Some(9), Some(6), None, Some(5)]); + assert_eq!(wire.queue[2], music_track(3), "music is untouched"); +} + +#[tokio::test] +async fn malformed_browser_items_keep_the_position_on_its_item() { + let pool = offline_pool(); + let browser = serde_json::json!({ + "tracks": [ + {"title": "no id"}, + {"id": 5, "content_id": CONTENT_ID}, + {"id": 6, "content_id": CONTENT_ID}, + ], + "index": 2, + "track": {"id": 6}, + }); + let wire = playback_state_from_browser_json(&pool, 1, browser) + .await + .unwrap(); + assert_eq!(wire.queue.len(), 2); + assert_eq!(wire.queue[wire.queue_pos].id, 6); +} + +fn progress_record(position_ms: i64, status: ProgressStatus, generation: u64) -> ProgressRecord { + let episode = test_episode(); + ProgressRecord { + episode_id: episode.episode_id, + podcast_id: episode.podcast_id, + position_ms, + status, + generation, + session_id: "session-1".into(), + hlc_ms: 1_700_000_000_000, + device_id: "tui".into(), + op_id: "op-1".into(), + } +} + +/// Browser state of `[library song, episode]` with the song current at 200 s. +fn song_then_episode() -> serde_json::Value { + let episode = PlaybackTrack::podcast_episode(test_episode()); + let song = serde_json::json!({ + "id": 77, "title": "Library song", "content_id": CONTENT_ID, + "duration_seconds": 300.0, "stream_url": "/api/player/stream/77", + }); + serde_json::json!({ + "tracks": [song.clone(), queue_items::external_track_json(&episode, 1, None)], + "index": 0, + "track": song, + "position_seconds": 200.0, + "paused": false, + "repeat_mode": "off", + }) +} + +fn position_unknown(state: &serde_json::Value) -> bool { + state[queue_items::POSITION_UNKNOWN_KEY] == true +} + +#[test] +fn commands_that_select_an_item_do_not_know_its_position() { + let current = song_then_episode(); + // Next and Prev onto another item, and play commands (which the browser + // always sends with position 0), leave the position to the server. + let next = + browser_state_for_command("next", serde_json::json!({}), Some(current.clone())).unwrap(); + assert_eq!(next["index"], 1); + assert_eq!(next["position_seconds"], 0.0); + assert!(position_unknown(&next)); + let mut play = current.clone(); + play["index"] = serde_json::json!(1); + let played = browser_state_for_command("play_from_index", play.clone(), None).unwrap(); + assert!(position_unknown(&played)); + let prev = + browser_state_for_command("prev", serde_json::json!({}), Some(next.clone())).unwrap(); + assert_eq!(prev["index"], 0); + assert!(position_unknown(&prev)); + + // Restarting the same item and seeking are explicit positions. + let mut late = next.clone(); + late["position_seconds"] = serde_json::json!(900.0); + let restart = browser_state_for_command("prev", serde_json::json!({}), Some(late)).unwrap(); + assert_eq!(restart["index"], 1); + assert!(!position_unknown(&restart)); + let repeat = browser_state_for_command( + "next", + serde_json::json!({"repeat_mode": "one"}), + Some(next.clone()), + ) + .unwrap(); + assert!(!position_unknown(&repeat)); + let seek = + browser_state_for_command("seek", serde_json::json!({"time": 60.0}), Some(next)).unwrap(); + assert_eq!(seek["position_seconds"], 60.0); + assert!(!position_unknown(&seek)); + + // A transfer keeps what the sender knew. + let mut handed_off = play; + handed_off["position_seconds"] = serde_json::json!(1200.0); + let transfer = browser_state_for_command("transfer_state", handed_off.clone(), None).unwrap(); + assert_eq!(transfer["position_seconds"], 1200.0); + assert!(!position_unknown(&transfer)); + handed_off[queue_items::POSITION_UNKNOWN_KEY] = serde_json::json!(true); + let transfer = browser_state_for_command("transfer_state", handed_off, None).unwrap(); + assert!(position_unknown(&transfer)); +} + +#[tokio::test] +async fn unknown_episode_position_resumes_from_the_progress_register() { + let pool = offline_pool(); + let next = browser_state_for_command("next", serde_json::json!({}), Some(song_then_episode())) + .unwrap(); + let mut wire = browser_json_to_wire(&pool, next).await.unwrap(); + assert_eq!(wire.queue_pos, 1); + assert_eq!(wire.position_secs, 0.0); + + // In progress at 1200 s, generation 9 (the queue item has seen 7). + let mut resumed = wire.clone(); + apply_resume_position( + &mut resumed, + Some(&progress_record(1_200_000, ProgressStatus::InProgress, 9)), + ); + assert_eq!(resumed.position_secs, 1200.0); + let episode = resumed.queue[1].podcast_episode_ref().unwrap(); + assert_eq!(episode.progress_generation, Some(9)); + // The receiver's explicit write then orders after the register. + assert_eq!( + ProgressRecord::explicit_generation(episode.progress_generation), + 10 + ); + + // Completed: ordinary Play starts from the beginning; an older register + // does not lower the generation the item has seen. + let mut completed = wire.clone(); + apply_resume_position( + &mut completed, + Some(&progress_record(1_790_000, ProgressStatus::Completed, 3)), + ); + assert_eq!(completed.position_secs, 0.0); + assert_eq!( + completed.queue[1] + .podcast_episode_ref() + .unwrap() + .progress_generation, + Some(7) + ); + + // Music keeps the position it has. + wire.queue_pos = 0; + wire.position_secs = 5.0; + let before = wire.clone(); + apply_resume_position( + &mut wire, + Some(&progress_record(1_200_000, ProgressStatus::InProgress, 9)), + ); + assert_eq!(wire, before); +} + +#[tokio::test] +async fn known_positions_and_music_never_take_the_register_position() { + // The bridge itself runs without queries (offline pool); which state + // reads the episode's progress register is decided on the result. + let pool = offline_pool(); + let mut handed_off = song_then_episode(); + handed_off["index"] = serde_json::json!(1); + handed_off["position_seconds"] = serde_json::json!(1200.0); + let wire = browser_json_to_wire(&pool, handed_off).await.unwrap(); + assert_eq!(wire.position_secs, 1200.0); + assert_eq!(episode_needing_resume_position(false, &wire), None); + assert_eq!( + episode_needing_resume_position(true, &wire), + Some(test_episode().episode_id), + "an episode the browser did not know the position of" + ); + + let music = song_then_episode(); + let wire = browser_json_to_wire(&pool, music).await.unwrap(); + assert_eq!(wire.position_secs, 200.0); + assert_eq!( + episode_needing_resume_position(true, &wire), + None, + "music keeps its position even when flagged" + ); +} diff --git a/src/federation/devices/interop_tests.rs b/src/federation/devices/interop_tests.rs index 7c90357..3979462 100644 --- a/src/federation/devices/interop_tests.rs +++ b/src/federation/devices/interop_tests.rs @@ -1,5 +1,6 @@ //! Cross-binary protocol contract; the peer is the TUI's production adapter. use super::*; +use music_dht::device_sync::read_line; use tokio::io::AsyncWriteExt; #[tokio::test] @@ -34,7 +35,8 @@ async fn localhost_tui_peer() { else { panic!("expected TUI hello") }; - assert_eq!(profile.protocol_version, PROTOCOL_VERSION); + // TUI 0.3.x reports v3; podcast-capable TUI releases report v4. + assert!((3..=PROTOCOL_VERSION).contains(&profile.protocol_version)); assert_eq!(snapshot.device_id, profile.device_id); let announcement = snapshot .coordination diff --git a/src/federation/devices/postgres_tests.rs b/src/federation/devices/postgres_tests.rs new file mode 100644 index 0000000..d78cd9f --- /dev/null +++ b/src/federation/devices/postgres_tests.rs @@ -0,0 +1,604 @@ +//! Device-sync v4 storage paths against PostgreSQL. +//! +//! Requires a disposable database in `FURU_DEVICE_SYNC_TEST_DATABASE_URL`; +//! it is migrated with the production migrations on first use +//! (`crate::podcasts::postgres_tests::test_pool`). The tests write rows for +//! random user ids and remove them again. + +use super::*; +use crate::player::queue_items::test_episode; +use music_dht::podcast::{ + EpisodeId, FeedUrl, PodcastId, PodcastMeta, ProgressRecord, ProgressStatus, ShareRecord, +}; + +fn profile(device_id: &str, protocol_version: u16) -> DeviceProfileWire { + DeviceProfileWire { + device_id: device_id.into(), + name: device_id.to_uppercase(), + client_version: "test".into(), + protocol_version, + endpoint_id: String::new(), + endpoint_ticket: String::new(), + revoked: false, + revoke_cutoff_seq: None, + updated_at_ms: now_ms(), + } +} + +fn op(origin: &str, seq: i64, hlc_ms: i64, payload: SyncOpPayload) -> SyncOpWire { + SyncOpWire { + op_id: format!("{origin}:{seq}"), + origin_device_id: origin.into(), + seq, + hlc_ms, + payload, + } +} + +async fn cleanup(pool: &sqlx::PgPool, user_id: i64) { + for table in [ + "furumusic__fed_device_identity", + "furumusic__fed_device", + "furumusic__fed_device_invite", + "furumusic__fed_peer_ack", + "furumusic__fed_pending_pairing", + "furumusic__fed_playback_applied", + "furumusic__fed_podcast_relay", + "furumusic__fed_state_like", + "furumusic__fed_state_playlist", + "furumusic__fed_state_playlist_item", + "furumusic__fed_sync_ops", + "furumusic__fed_sync_vector", + "furumusic__playback_state", + "furumusic__podcast_fetch_state", + "furumusic__podcast_listen_session", + "furumusic__podcast_progress", + "furumusic__podcast_share", + "furumusic__podcast_subscription", + "furumusic__podcast_episode", + "furumusic__podcast_feed_alias", + "furumusic__podcast", + ] { + sqlx::query(&format!("DELETE FROM {table} WHERE user_id = $1")) + .bind(user_id) + .execute(pool) + .await + .unwrap(); + } +} + +#[tokio::test] +#[ignore = "requires a disposable database in FURU_DEVICE_SYNC_TEST_DATABASE_URL"] +async fn postgres_podcast_relay_gating_and_peer_versions() { + let pool = crate::podcasts::postgres_tests::test_pool().await; + let user_id = 8_000_000_000 + i64::from(rand_suffix()); + cleanup(&pool, user_id).await; + let new = format!("tui_new_{user_id}"); + let old = format!("tui_old_{user_id}"); + let other = format!("tui_other_{user_id}"); + let identity = ensure_identity(&pool, user_id, "tester").await.unwrap(); + + // Peer versions: self-reports replace, relayed profiles only raise. + for (device, version) in [(&new, 4), (&old, 3), (&other, 4)] { + apply_device_profile( + &pool, + user_id, + &profile(device, version), + VersionReport::SelfReported(Some(version)), + true, + ) + .await + .unwrap(); + } + // A v3 relay lists everyone with its own constant. + apply_device_profiles( + &pool, + user_id, + &[profile(&new, 3), profile(&old, 3)], + Some(old.as_str()), + ) + .await + .unwrap(); + assert_eq!( + stored_protocol_version(&pool, user_id, &new).await.unwrap(), + 4 + ); + assert_eq!( + stored_protocol_version(&pool, user_id, &old).await.unwrap(), + 3 + ); + + // Podcast and music operations from the v4 device. + let feed = FeedUrl::parse("https://podcasts.example/feed.xml").unwrap(); + let podcast_id = PodcastId::from_feed_url(&feed); + let episode_id = EpisodeId::from_guid(&podcast_id, "ep-1").unwrap(); + let meta: PodcastMeta = serde_json::from_value(serde_json::json!({ + "feed_url": feed.as_str(), "title": "Example Cast" + })) + .unwrap(); + let progress = |seq, generation, hlc_ms, position_ms| { + op( + &new, + seq, + hlc_ms, + SyncOpPayload::PodcastProgressSet { + episode_id: episode_id.clone(), + podcast_id: podcast_id.clone(), + position_ms, + status: ProgressStatus::InProgress, + generation, + session_id: format!("s{generation}"), + episode: None, + }, + ) + }; + let mut invalid = progress(5, 9, 50, 1); + if let SyncOpPayload::PodcastProgressSet { session_id, .. } = &mut invalid.payload { + session_id.clear(); + } + let hub = Arc::new(PlayerDeviceHub::default()); + apply_ops( + &pool, + Arc::clone(&hub), + user_id, + vec![ + op( + &new, + 1, + 10, + SyncOpPayload::PodcastSubscriptionSet { + podcast_id: podcast_id.clone(), + subscribed: true, + podcast: meta, + feed_aliases: vec![], + generation: 1, + }, + ), + // Newer generation first, then a late checkpoint of an older one. + progress(2, 4, 20, 10_000), + progress(3, 3, 90_000, 600_000), + op( + &new, + 4, + 40, + SyncOpPayload::TrackLikeSet { + content_id: format!("b3:{}", "3".repeat(64)), + liked: false, + fed: None, + }, + ), + invalid, + ], + ) + .await + .unwrap(); + // The invalid operation is not stored but acknowledged. + assert_eq!(vector(&pool, user_id).await.unwrap().get(&new), Some(&5)); + let stored: i64 = sqlx::query_scalar( + "SELECT COUNT(*) FROM furumusic__fed_sync_ops WHERE user_id = $1 AND origin_device_id = $2", + ) + .bind(user_id) + .bind(&new) + .fetch_one(&pool) + .await + .unwrap(); + assert_eq!(stored, 4); + + // Relay: the v3 peer gets only music, a v4 peer gets everything. + let kinds = |ops: Vec| { + ops.iter() + .map(|op| payload_kind(&op.payload)) + .collect::>() + }; + let to_old = kinds(ops_for_peer(&pool, user_id, &old, 3).await.unwrap()); + assert!(to_old.iter().all(|kind| !kind.starts_with("podcast_"))); + assert!(to_old.contains(&"track_like_set")); + let to_other = kinds(ops_for_peer(&pool, user_id, &other, 4).await.unwrap()); + assert_eq!( + to_other + .iter() + .filter(|kind| kind.starts_with("podcast_")) + .count(), + 3 + ); + + // Snapshot recovery: registers merged with the shared rules. + apply_snapshot( + &pool, + user_id, + SyncSnapshot { + podcast_shares: vec![ShareRecord { + podcast_id: podcast_id.clone(), + shared: false, + generation: 2, + hlc_ms: 60, + op_id: format!("{other}:1"), + }], + podcast_progress: vec![ProgressRecord { + episode_id: episode_id.clone(), + podcast_id: podcast_id.clone(), + position_ms: 1, + status: ProgressStatus::InProgress, + generation: 2, + session_id: "old".into(), + hlc_ms: 999_999, + device_id: other.clone(), + op_id: format!("{other}:2"), + }], + ..SyncSnapshot::default() + }, + ) + .await + .unwrap(); + let full = snapshot(&pool, user_id).await.unwrap(); + assert_eq!(full.podcast_subscriptions.len(), 1); + assert_eq!(full.podcast_shares.len(), 1); + assert_eq!(full.podcast_progress.len(), 1); + assert_eq!(full.podcast_progress[0].position_ms, 10_000); + assert_eq!(full.podcast_progress[0].generation, 4); + assert!(!full.for_peer(3).has_podcast_state()); + assert_eq!(full.for_peer(4), full); + + // A DeviceProfileSet with a version is a self-report of its origin. + apply_ops( + &pool, + Arc::clone(&hub), + user_id, + vec![op( + &old, + 1, + 70, + SyncOpPayload::DeviceProfileSet { + name: "OLD".into(), + client_version: "0.4.0".into(), + endpoint_ticket: String::new(), + endpoint_id: String::new(), + protocol_version: Some(4), + }, + )], + ) + .await + .unwrap(); + assert_eq!( + stored_protocol_version(&pool, user_id, &old).await.unwrap(), + 4 + ); + + // First start after the upgrade announces the new version once. + sqlx::query( + "UPDATE furumusic__fed_device SET protocol_version = 3 + WHERE user_id = $1 AND device_id = $2", + ) + .bind(user_id) + .bind(&identity.device_id) + .execute(&pool) + .await + .unwrap(); + own_profile(&pool, user_id, "", "").await.unwrap(); + own_profile(&pool, user_id, "", "").await.unwrap(); + let announced: Vec = sqlx::query_scalar( + "SELECT payload_json FROM furumusic__fed_sync_ops + WHERE user_id = $1 AND origin_device_id = $2 AND kind = 'device_profile_set'", + ) + .bind(user_id) + .bind(&identity.device_id) + .fetch_all(&pool) + .await + .unwrap(); + assert_eq!(announced.len(), 1); + assert_eq!(announced[0]["protocol_version"], 4); + + // Handoff gating: refused for the v3 device, allowed for v4. + let queue = PlaybackStateWire { + queue: vec![PlaybackTrack::podcast_episode(test_episode())], + queue_pos: 0, + playing: true, + paused: false, + idle_since_ms: None, + position_secs: 0.0, + volume: 50, + shuffle: false, + repeat: PlaybackRepeat::Off, + playback_rate: None, + }; + sqlx::query( + "UPDATE furumusic__fed_device SET protocol_version = 3 + WHERE user_id = $1 AND device_id = $2", + ) + .bind(user_id) + .bind(&old) + .execute(&pool) + .await + .unwrap(); + let refused = ensure_playback_supported_by(&pool, user_id, &old, &queue) + .await + .unwrap_err(); + assert!(refused.to_string().contains("needs an update")); + ensure_playback_supported_by(&pool, user_id, &other, &queue) + .await + .unwrap(); + + // The typed queue column exists next to queue_json. + sqlx::query("SELECT queue_json, queue_items_json FROM furumusic__playback_state LIMIT 0") + .execute(&pool) + .await + .unwrap(); + + cleanup(&pool, user_id).await; +} + +#[tokio::test] +#[ignore = "requires a disposable database in FURU_DEVICE_SYNC_TEST_DATABASE_URL"] +async fn postgres_unknown_episode_position_and_rollback_cleanup() { + let pool = crate::podcasts::postgres_tests::test_pool().await; + let user_id = 8_000_000_000 + i64::from(rand_suffix()); + cleanup(&pool, user_id).await; + let tui = format!("tui_{user_id}"); + ensure_identity(&pool, user_id, "tester").await.unwrap(); + apply_device_profile( + &pool, + user_id, + &profile(&tui, 4), + VersionReport::SelfReported(Some(4)), + true, + ) + .await + .unwrap(); + + // The TUI left the episode at 1200 s (generation 9; queue items saw 7). + let episode = test_episode(); + let hub = Arc::new(PlayerDeviceHub::default()); + apply_ops( + &pool, + Arc::clone(&hub), + user_id, + vec![op( + &tui, + 1, + 10, + SyncOpPayload::PodcastProgressSet { + episode_id: episode.episode_id.clone(), + podcast_id: episode.podcast_id.clone(), + position_ms: 1_200_000, + status: ProgressStatus::InProgress, + generation: 9, + session_id: "s9".into(), + episode: None, + }, + )], + ) + .await + .unwrap(); + + // The web player reached the episode by Next and does not know where it + // was left; it hands playback to the TUI. + let song = serde_json::json!({ + "id": 77, "title": "Library song", "content_id": format!("b3:{}", "4".repeat(64)), + "duration_seconds": 300.0, + }); + let item = crate::player::queue_items::external_track_json( + &PlaybackTrack::podcast_episode(episode.clone()), + 1, + None, + ); + let browser = serde_json::json!({ + "tracks": [song.clone(), item.clone()], "index": 1, "track": item.clone(), + "position_seconds": 0.0, "paused": true, "position_unknown": true, + }); + record_web_active_transfer(&pool, user_id, &tui, None, browser) + .await + .unwrap(); + let sent: serde_json::Value = sqlx::query_scalar( + "SELECT payload_json FROM furumusic__fed_sync_ops + WHERE user_id = $1 AND kind = 'playback_command' ORDER BY seq DESC LIMIT 1", + ) + .bind(user_id) + .fetch_one(&pool) + .await + .unwrap(); + let SyncOpPayload::PlaybackCommand { + command: PlaybackCommand::ActiveChanged { state, .. }, + .. + } = serde_json::from_value(sent).unwrap() + else { + panic!("expected ActiveChanged"); + }; + assert_eq!(state.position_secs, 1200.0, "never 0"); + assert_eq!( + state.queue[1] + .podcast_episode_ref() + .unwrap() + .progress_generation, + Some(9) + ); + + // A known position (the web player was playing the episode) is sent + // as is, but the stale handoff hint of the browser item (7) is raised + // to the register generation this account has seen (9). + let browser = serde_json::json!({ + "tracks": [song.clone(), item.clone()], "index": 1, "track": item.clone(), + "position_seconds": 1500.0, "paused": true, + }); + record_web_active_transfer(&pool, user_id, &tui, None, browser) + .await + .unwrap(); + let sent: serde_json::Value = sqlx::query_scalar( + "SELECT payload_json FROM furumusic__fed_sync_ops + WHERE user_id = $1 AND kind = 'playback_command' ORDER BY seq DESC LIMIT 1", + ) + .bind(user_id) + .fetch_one(&pool) + .await + .unwrap(); + let SyncOpPayload::PlaybackCommand { + command: PlaybackCommand::ActiveChanged { state, .. }, + .. + } = serde_json::from_value(sent).unwrap() + else { + panic!("expected ActiveChanged"); + }; + assert_eq!(state.position_secs, 1500.0); + assert_eq!( + state.queue[1] + .podcast_episode_ref() + .unwrap() + .progress_generation, + Some(9) + ); + + // Remote Next onto the episode while the TUI plays the song. + let current = serde_json::json!({ + "tracks": [song, item], "index": 0, "track": song, + "position_seconds": 200.0, "paused": false, + }); + let next = record_web_playback_command( + &pool, + user_id, + &tui, + "next", + serde_json::json!({}), + Some(current), + ) + .await + .unwrap(); + assert_eq!(next["index"], 1); + assert_eq!(next["position_seconds"], 1200.0); + assert!( + next.get(crate::player::queue_items::POSITION_UNKNOWN_KEY) + .is_none() + ); + + // Rollback to 0.12: its types fail on the relayed podcast operation, and + // parse everything once the documented cleanup ran. + let parse_like_0_12 = |rows: Vec| { + rows.into_iter() + .map(serde_json::from_value::) + .collect::, _>>() + }; + let rows = || async { + sqlx::query_scalar::<_, serde_json::Value>( + "SELECT payload_json FROM furumusic__fed_sync_ops WHERE user_id = $1", + ) + .bind(user_id) + .fetch_all(&pool) + .await + .unwrap() + }; + assert!(parse_like_0_12(rows().await).is_err()); + sqlx::query(&format!( + "{} AND user_id = $1", + crate::music::db_migrations::DELETE_PODCAST_SYNC_OPS_SQL + )) + .bind(user_id) + .execute(&pool) + .await + .unwrap(); + let remaining = parse_like_0_12(rows().await).unwrap(); + // The playback commands (with the episode's `source`) stay readable. + assert_eq!( + remaining + .iter() + .filter(|payload| matches!( + payload, + super::wire_compat_tests::legacy::SyncOpPayload::PlaybackCommand { .. } + )) + .count(), + 3 + ); + + cleanup(&pool, user_id).await; +} + +#[tokio::test] +#[ignore = "requires a disposable database in FURU_DEVICE_SYNC_TEST_DATABASE_URL"] +async fn postgres_local_podcast_actions_reach_v4_peers_only() { + let pool = crate::podcasts::postgres_tests::test_pool().await; + let user_id = 8_000_000_000 + i64::from(rand_suffix()); + cleanup(&pool, user_id).await; + let identity = ensure_identity(&pool, user_id, "tester").await.unwrap(); + let old = format!("tui_old_{user_id}"); + let new = format!("tui_new_{user_id}"); + for (device, version) in [(&old, 3), (&new, 4)] { + apply_device_profile( + &pool, + user_id, + &profile(device, version), + VersionReport::SelfReported(Some(version)), + true, + ) + .await + .unwrap(); + } + + // A local action lands in the log with this device's stamp and in the + // store, in one transaction. + let store = crate::podcasts::PodcastStore::new(pool.clone()); + let episode = test_episode(); + let meta: PodcastMeta = serde_json::from_value(serde_json::json!({ + "feed_url": episode.feed_url.as_str(), "title": episode.podcast_title, + })) + .unwrap(); + let subscribed = store.subscribe(user_id, &meta).await.unwrap(); + let played = store.mark_played(user_id, &episode).await.unwrap(); + assert!( + subscribed + .ops + .iter() + .chain([&played.op]) + .all(|op| op.origin_device_id == identity.device_id) + ); + // A music operation recorded afterwards keeps the same clock and sequence. + record_local_op( + &pool, + user_id, + SyncOpPayload::TrackLikeSet { + content_id: format!("b3:{}", "5".repeat(64)), + liked: false, + fed: None, + }, + ) + .await + .unwrap(); + let own: Vec = ops_for_peer(&pool, user_id, &new, 4) + .await + .unwrap() + .into_iter() + .filter(|op| op.origin_device_id == identity.device_id) + .collect(); + let kinds: Vec<_> = own.iter().map(|op| payload_kind(&op.payload)).collect(); + assert!(kinds.ends_with(&[ + "podcast_subscription_set", + "podcast_progress_set", + "track_like_set" + ])); + assert!( + own.windows(2) + .all(|pair| pair[1].seq > pair[0].seq && pair[1].hlc_ms > pair[0].hlc_ms) + ); + assert_eq!(own[own.len() - 3], subscribed.ops[0]); + assert_eq!(own[own.len() - 2], played.op); + // The v3 peer gets the music operation only. + let to_old = ops_for_peer(&pool, user_id, &old, 3).await.unwrap(); + assert!(to_old.iter().all(|op| !op.payload.is_podcast())); + assert!( + to_old + .iter() + .any(|op| payload_kind(&op.payload) == "track_like_set") + ); + // The snapshot carries the store's registers for v4 only. + let full = snapshot(&pool, user_id).await.unwrap(); + assert_eq!(full.podcast_subscriptions.len(), 1); + assert_eq!(full.podcast_progress.len(), 1); + assert_eq!(full.podcast_progress[0].status, ProgressStatus::Completed); + assert!(!full.for_peer(3).has_podcast_state()); + + cleanup(&pool, user_id).await; +} + +fn rand_suffix() -> u32 { + u32::from_le_bytes( + blake3::hash(random_hex(8).as_bytes()).as_bytes()[..4] + .try_into() + .unwrap(), + ) % 1_000_000_000 +} diff --git a/src/federation/devices/wire_compat_tests.rs b/src/federation/devices/wire_compat_tests.rs new file mode 100644 index 0000000..36f7ac8 --- /dev/null +++ b/src/federation/devices/wire_compat_tests.rs @@ -0,0 +1,524 @@ +//! Golden wire-compatibility tests: the shared `music_dht::device_sync` +//! types must serialize v3 content byte-for-byte like the private copies that +//! furumi-web 0.12.x used (kept verbatim in `legacy` below), so TUI 0.3.x and +//! older web servers keep interoperating. + +use super::*; + +/// Private wire types of furumi-web 0.12.4 (`src/federation/devices.rs`), +/// copied verbatim apart from visibility and `PartialEq`. +#[allow(dead_code)] +pub(super) mod legacy { + use music_dht::device_sync::ListenEvent; + use music_dht::playback::CommandStamp; + use serde::{Deserialize, Serialize}; + use std::collections::BTreeMap; + + #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] + pub struct InviteWire { + pub v: u16, + #[serde(rename = "t")] + pub ticket: String, + #[serde(rename = "d")] + pub device_id: String, + #[serde(rename = "i")] + pub invite_id: String, + #[serde(rename = "s")] + pub secret: String, + #[serde(rename = "e")] + pub expires_at_ms: i64, + } + + #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] + pub struct DeviceProfileWire { + pub device_id: String, + pub name: String, + pub client_version: String, + pub protocol_version: u16, + pub endpoint_id: String, + pub endpoint_ticket: String, + #[serde(default)] + pub revoked: bool, + #[serde(default)] + pub revoke_cutoff_seq: Option, + #[serde(default)] + pub updated_at_ms: i64, + } + + #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] + pub struct SyncedFedTrack { + pub item_id: String, + pub owner: String, + pub title: String, + #[serde(default)] + pub artist_names: Vec, + #[serde(default)] + pub featured_artist_names: Vec, + pub year: Option, + pub duration_seconds: Option, + pub content_id: String, + pub release_title: Option, + pub track_number: Option, + pub disc_number: Option, + } + + #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] + pub struct PlaybackTrack { + pub id: i64, + pub title: String, + pub track_number: Option, + pub disc_number: Option, + pub duration_seconds: f64, + #[serde(default)] + pub artist_names: Vec, + #[serde(default)] + pub featured_artist_names: Vec, + pub release_id: i64, + pub release_title: String, + pub release_year: Option, + #[serde(default)] + pub file_path: String, + pub content_id: Option, + pub audio_format: Option, + pub audio_bitrate: Option, + pub audio_sample_rate: Option, + pub audio_bit_depth: Option, + pub file_size_bytes: Option, + #[serde(default)] + pub play_count: i64, + #[serde(default)] + pub fed: Option, + } + + #[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Default)] + #[serde(rename_all = "snake_case")] + pub enum PlaybackRepeat { + #[default] + Off, + One, + All, + } + + #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] + pub struct PlaybackStateWire { + #[serde(default)] + pub queue: Vec, + #[serde(default)] + pub queue_pos: usize, + pub playing: bool, + pub paused: bool, + #[serde(default)] + pub idle_since_ms: Option, + pub position_secs: f64, + #[serde(default)] + pub volume: u8, + pub shuffle: bool, + pub repeat: PlaybackRepeat, + } + + pub type PlaybackSnapshot = music_dht::playback::Snapshot; + pub type PlaybackCommand = music_dht::playback::Command; + + #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] + pub struct SyncOpWire { + pub op_id: String, + pub origin_device_id: String, + pub seq: i64, + pub hlc_ms: i64, + pub payload: SyncOpPayload, + } + + #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] + #[serde(tag = "kind", rename_all = "snake_case")] + pub enum SyncOpPayload { + TrackLikeSet { + content_id: String, + liked: bool, + #[serde(default)] + fed: Option, + }, + PlaylistCreated { + playlist_id: String, + title: String, + }, + PlaylistRenamed { + playlist_id: String, + title: String, + }, + PlaylistDeleted { + playlist_id: String, + }, + PlaylistTrackAdded { + playlist_id: String, + content_id: String, + position: i64, + #[serde(default)] + fed: Option, + }, + PlaylistTrackRemoved { + playlist_id: String, + content_id: String, + }, + DeviceProfileSet { + name: String, + client_version: String, + endpoint_ticket: String, + endpoint_id: String, + }, + DeviceTrusted { + target_device_id: String, + }, + DeviceRevoked { + target_device_id: String, + target_max_seq_seen: i64, + }, + PlaybackCommand { + target_device_id: String, + command: PlaybackCommand, + #[serde(default)] + authority: Option, + }, + ListenRecorded { + event: ListenEvent, + }, + } + + #[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)] + pub struct SyncSnapshot { + #[serde(default)] + pub likes: Vec, + #[serde(default)] + pub unlikes: Vec, + #[serde(default)] + pub playlists: Vec, + #[serde(default)] + pub deleted_playlists: Vec, + #[serde(default)] + pub removed_playlist_items: Vec, + } + + #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] + pub struct SnapshotLike { + pub content_id: String, + pub hlc_ms: i64, + pub op_id: String, + #[serde(default)] + pub fed: Option, + } + + #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] + pub struct SnapshotLikeTombstone { + pub content_id: String, + pub hlc_ms: i64, + pub op_id: String, + } + + #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] + pub struct SnapshotPlaylist { + pub playlist_id: String, + pub title: String, + pub hlc_ms: i64, + pub op_id: String, + #[serde(default)] + pub items: Vec, + } + + #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] + pub struct SnapshotPlaylistTombstone { + pub playlist_id: String, + pub hlc_ms: i64, + pub op_id: String, + } + + #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] + pub struct SnapshotPlaylistItem { + pub content_id: String, + pub position: i64, + pub hlc_ms: i64, + pub op_id: String, + #[serde(default)] + pub fed: Option, + } + + #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] + pub struct SnapshotPlaylistItemTombstone { + pub playlist_id: String, + pub content_id: String, + pub hlc_ms: i64, + pub op_id: String, + } + + #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] + #[serde(tag = "type", rename_all = "snake_case")] + pub enum WireMessage { + PairRequest { + invite_id: String, + secret: String, + profile: DeviceProfileWire, + #[serde(default)] + group_id: Option, + #[serde(default)] + group_active_devices: usize, + #[serde(default)] + devices: Vec, + vector: BTreeMap, + ops: Vec, + snapshot: SyncSnapshot, + #[serde(default)] + playback: Option, + }, + PairResponse { + accepted: bool, + #[serde(default)] + pending: bool, + #[serde(default)] + error: Option, + #[serde(default)] + group_id: Option, + #[serde(default)] + profile: Option, + #[serde(default)] + devices: Vec, + #[serde(default)] + vector: BTreeMap, + #[serde(default)] + ops: Vec, + #[serde(default)] + snapshot: SyncSnapshot, + #[serde(default)] + playback: Option, + }, + Hello { + group_id: String, + profile: DeviceProfileWire, + devices: Vec, + vector: BTreeMap, + ops: Vec, + snapshot: SyncSnapshot, + #[serde(default)] + playback: Option, + }, + SyncResponse { + accepted: bool, + #[serde(default)] + error: Option, + #[serde(default)] + devices: Vec, + #[serde(default)] + vector: BTreeMap, + #[serde(default)] + ops: Vec, + #[serde(default)] + snapshot: SyncSnapshot, + #[serde(default)] + playback: Option, + }, + } +} + +const CONTENT_ID: &str = "b3:1111111111111111111111111111111111111111111111111111111111111111"; + +/// A v3 `hello` exercising every v3 operation kind, every snapshot section +/// and a coordinated playback snapshot (input order is irrelevant). +fn v3_hello_input() -> String { + let fed = serde_json::json!({ + "item_id": "item", "owner": "owner", "title": "Song", "artist_names": ["A"], + "featured_artist_names": [], "year": 2001, "duration_seconds": 180, + "content_id": CONTENT_ID, "release_title": "Album", "track_number": 1, "disc_number": 1, + }); + let track = serde_json::json!({ + "id": 7, "title": "Song", "track_number": 1, "disc_number": null, + "duration_seconds": 180.5, "artist_names": ["A"], "featured_artist_names": ["B"], + "release_id": 3, "release_title": "Album", "release_year": 2001, "file_path": "", + "content_id": CONTENT_ID, "audio_format": "flac", "audio_bitrate": 900, + "audio_sample_rate": 44100, "audio_bit_depth": 16, "file_size_bytes": 12345, + "play_count": 2, "fed": fed, + }); + let state = serde_json::json!({ + "queue": [track, {"id": 0, "title": "Remote", "track_number": null, "disc_number": null, + "duration_seconds": 0.0, "release_id": 0, "release_title": "", "release_year": null, + "content_id": null, "audio_format": null, "audio_bitrate": null, + "audio_sample_rate": null, "audio_bit_depth": null, "file_size_bytes": null}], + "queue_pos": 1, "playing": true, "paused": false, "idle_since_ms": null, + "position_secs": 42.5, "volume": 73, "shuffle": true, "repeat": "all", + }); + let claim = serde_json::json!({"counter": 3, "issuer": "tui_a", "owner": "tui_a"}); + let stamp = serde_json::json!({"version": 1, "claim": claim, "sequence": 9}); + let op = |seq: i64, payload: serde_json::Value| { + serde_json::json!({"op_id": format!("tui_a:{seq}"), "origin_device_id": "tui_a", + "seq": seq, "hlc_ms": 1_700_000_000_000i64 + seq, "payload": payload}) + }; + let profile = |id: &str| { + serde_json::json!({"device_id": id, "name": "TUI", "client_version": "0.3.4", + "protocol_version": 3, "endpoint_id": "e", "endpoint_ticket": "t", + "revoked": false, "revoke_cutoff_seq": null, "updated_at_ms": 5}) + }; + serde_json::json!({ + "type": "hello", + "group_id": "grp_1", + "profile": profile("tui_a"), + "devices": [profile("tui_a"), profile("web_b")], + "vector": {"tui_a": 12, "web_b": 4}, + "ops": [ + op(1, serde_json::json!({"kind": "track_like_set", "content_id": CONTENT_ID, "liked": true, "fed": fed})), + op(2, serde_json::json!({"kind": "playlist_created", "playlist_id": "pl", "title": "Mix"})), + op(3, serde_json::json!({"kind": "playlist_renamed", "playlist_id": "pl", "title": "Mix 2"})), + op(4, serde_json::json!({"kind": "playlist_track_added", "playlist_id": "pl", "content_id": CONTENT_ID, "position": 0, "fed": null})), + op(5, serde_json::json!({"kind": "playlist_track_removed", "playlist_id": "pl", "content_id": CONTENT_ID})), + op(6, serde_json::json!({"kind": "playlist_deleted", "playlist_id": "pl"})), + op(7, serde_json::json!({"kind": "device_profile_set", "name": "TUI", "client_version": "0.3.4", "endpoint_ticket": "t", "endpoint_id": "e"})), + op(8, serde_json::json!({"kind": "device_trusted", "target_device_id": "web_b"})), + op(9, serde_json::json!({"kind": "device_revoked", "target_device_id": "old", "target_max_seq_seen": 3})), + op(10, serde_json::json!({"kind": "playback_command", "target_device_id": "web_b", + "command": {"kind": "set_state", "state": state, "seek": true}, "authority": stamp})), + op(11, serde_json::json!({"kind": "playback_command", "target_device_id": "web_b", + "command": {"kind": "active_changed", "active_device_id": "web_b", "active_device_name": "WEB", "state": state}})), + op(12, serde_json::json!({"kind": "listen_recorded", "event": {"listen_id": "l1", + "content_id": CONTENT_ID, "started_at_ms": 1, "listened_ms": 60000, "track_duration_ms": 180000, + "ended_reason": "finished", "track": {"title": "Song", "artist_names": ["A"], "featured_artist_names": [], "release_title": null}}})), + ], + "snapshot": { + "likes": [{"content_id": CONTENT_ID, "hlc_ms": 1, "op_id": "tui_a:1", "fed": fed}], + "unlikes": [{"content_id": CONTENT_ID, "hlc_ms": 2, "op_id": "tui_a:2"}], + "playlists": [{"playlist_id": "pl", "title": "Mix", "hlc_ms": 3, "op_id": "tui_a:3", + "items": [{"content_id": CONTENT_ID, "position": 0, "hlc_ms": 4, "op_id": "tui_a:4", "fed": null}]}], + "deleted_playlists": [{"playlist_id": "gone", "hlc_ms": 5, "op_id": "tui_a:5"}], + "removed_playlist_items": [{"playlist_id": "pl", "content_id": CONTENT_ID, "hlc_ms": 6, "op_id": "tui_a:6"}], + }, + "playback": {"device_id": "tui_a", "device_name": "TUI", "active": true, "updated_at_ms": 9, + "state": state, "coordination": {"version": 1, "claim": claim, "heartbeat": 4, "available": true, "playing": true}}, + }) + .to_string() +} + +fn v3_other_inputs() -> Vec { + vec![ + r#"{"type":"sync_response","accepted":false,"error":"device is not trusted"}"#.into(), + r#"{"type":"pair_response","accepted":false,"pending":true,"error":"pairing pending"}"# + .into(), + serde_json::json!({"type": "pair_request", "invite_id": "i", "secret": "s", + "profile": {"device_id": "tui_a", "name": "TUI", "client_version": "0.3.4", + "protocol_version": 3, "endpoint_id": "e", "endpoint_ticket": "t"}, + "vector": {}, "ops": [], "snapshot": {}}) + .to_string(), + ] +} + +/// Parses `input` with the 0.12.x types and the shared types and checks +/// that both serialize to the same bytes, and that each side parses the +/// other's output. +fn assert_identical(input: &str) -> String { + let old: legacy::WireMessage = serde_json::from_str(input).unwrap(); + let new: WireMessage = serde_json::from_str(input).unwrap(); + let old_json = serde_json::to_string(&old).unwrap(); + let new_json = serde_json::to_string(&new).unwrap(); + assert_eq!(old_json, new_json); + assert_eq!( + serde_json::from_str::(&new_json).unwrap(), + old + ); + assert_eq!(serde_json::from_str::(&old_json).unwrap(), new); + new_json +} + +#[test] +fn v3_messages_serialize_like_furumi_web_0_12() { + let hello = assert_identical(&v3_hello_input()); + let golden = concat!( + env!("CARGO_MANIFEST_DIR"), + "/src/federation/devices/fixtures/v3_hello.golden.json" + ); + if std::env::var_os("FURUMUSIC_BLESS_GOLDEN").is_some() { + // Only for a deliberate change of the v3 shape. + std::fs::write(golden, format!("{hello}\n")).unwrap(); + } + assert_eq!(hello, std::fs::read_to_string(golden).unwrap().trim_end()); + for input in v3_other_inputs() { + assert_identical(&input); + } +} + +#[test] +fn invites_stay_readable_by_older_clients() { + let old = legacy::InviteWire { + v: 1, + ticket: "ticket".into(), + device_id: "web_a".into(), + invite_id: "inv".into(), + secret: "s3cr3t".into(), + expires_at_ms: 10, + }; + let old_json = serde_json::to_string(&old).unwrap(); + assert_eq!( + old_json, + r#"{"v":1,"t":"ticket","d":"web_a","i":"inv","s":"s3cr3t","e":10}"# + ); + let shared: InviteWire = serde_json::from_str(&old_json).unwrap(); + assert_eq!(shared.inviter_protocol_version(), 1); + assert_eq!(serde_json::to_string(&shared).unwrap(), old_json); + let link = encode_invite(&InviteWire { + protocol_version: Some(PROTOCOL_VERSION), + ..shared + }) + .unwrap(); + let parsed = parse_invite(&link).unwrap(); + assert_eq!(parsed.inviter_protocol_version(), PROTOCOL_VERSION); + // `p` is an unknown field for 0.12.x and is ignored there. + let token = link.strip_prefix("frid://i/").unwrap(); + assert!(!token.is_empty()); + let json = serde_json::to_string(&parsed).unwrap(); + assert!(json.ends_with(r#","p":4}"#)); + assert_eq!( + serde_json::from_str::(&json).unwrap(), + old + ); +} + +#[test] +fn v3_peers_cannot_parse_podcast_operations() { + // Why podcast operations must be gated: an unknown `kind` breaks the + // whole message for a 0.12.x/TUI 0.3.x peer, not one operation. + let mut hello: serde_json::Value = serde_json::from_str(&v3_hello_input()).unwrap(); + hello["ops"].as_array_mut().unwrap().push(serde_json::json!({ + "op_id": "tui_a:13", "origin_device_id": "tui_a", "seq": 13, "hlc_ms": 13, + "payload": {"kind": "podcast_share_set", + "podcast_id": music_dht::podcast::PodcastId::from_feed_url( + &music_dht::podcast::FeedUrl::parse("https://podcasts.example/feed.xml").unwrap()), + "shared": false, "generation": 1}, + })); + assert!(serde_json::from_value::(hello.clone()).is_err()); + assert!(serde_json::from_value::(hello).is_ok()); +} + +#[test] +fn v3_peers_would_drop_the_source_of_podcast_items() { + // Why podcast playback must be gated: older types ignore `source` and + // forward the episode as a broken music item. + let episode = PlaybackTrack::podcast_episode(crate::player::queue_items::test_episode()); + let json = serde_json::to_string(&episode).unwrap(); + assert!(json.contains(r#""source":{"kind":"podcast_episode""#)); + let old: legacy::PlaybackTrack = serde_json::from_str(&json).unwrap(); + let forwarded: PlaybackTrack = + serde_json::from_str(&serde_json::to_string(&old).unwrap()).unwrap(); + assert_eq!(forwarded.source, None); + assert_eq!(forwarded.id, 0); +} + +#[test] +fn v4_profile_announcements_stay_readable_by_older_clients() { + let announcement = SyncOpPayload::DeviceProfileSet { + name: "WEB".into(), + client_version: "0.13.0".into(), + endpoint_ticket: "t".into(), + endpoint_id: "e".into(), + protocol_version: Some(PROTOCOL_VERSION), + }; + let json = serde_json::to_string(&announcement).unwrap(); + assert!(json.ends_with(r#","protocol_version":4}"#)); + // 0.12.x and TUI 0.3.x ignore the new field. + let old: legacy::SyncOpPayload = serde_json::from_str(&json).unwrap(); + assert!(matches!( + old, + legacy::SyncOpPayload::DeviceProfileSet { .. } + )); +} diff --git a/src/federation/mod.rs b/src/federation/mod.rs index 435b019..88d084f 100644 --- a/src/federation/mod.rs +++ b/src/federation/mod.rs @@ -1192,6 +1192,18 @@ impl Federation { .await } + /// Refuses a transfer that the target device cannot receive (podcast + /// items for a device below protocol v4) before ownership changes. + pub async fn fed_device_web_transfer_check( + &self, + user_id: i64, + target_device_id: &str, + state: serde_json::Value, + ) -> Result<()> { + let pool = self.pool().await?; + devices::check_web_transfer(&pool, user_id, target_device_id, state).await + } + pub async fn fed_device_web_active_takeover( &self, user_id: i64, diff --git a/src/i18n/phrases.rs b/src/i18n/phrases.rs index 777e2d1..6e5855a 100644 --- a/src/i18n/phrases.rs +++ b/src/i18n/phrases.rs @@ -382,6 +382,64 @@ translations! { player_clear: "Clear" , "Очистить"; player_remove: "Remove" , "Удалить"; player_queue_empty: "Queue is empty" , "Очередь пуста"; + player_queue_podcast_item: "Podcast episode" , "Выпуск подкаста"; + player_queue_private_item: "Private item · hidden by its owner", "Приватный элемент · скрыт владельцем"; + player_podcasts: "Podcasts", "Подкасты"; + player_podcast_add_placeholder: "RSS feed URL", "Адрес RSS-ленты"; + player_podcast_add: "Add", "Добавить"; + player_podcast_adding: "Loading the feed…", "Загружаем ленту…"; + player_podcast_added: "Subscribed", "Подписка добавлена"; + player_podcast_already: "Already subscribed", "Вы уже подписаны"; + player_podcast_private_note: "New subscriptions are private: they sync only between your own devices.", "Новые подписки приватны: они синхронизируются только между вашими устройствами."; + player_podcasts_empty: "No podcasts yet. Add an RSS feed to subscribe.", "Подкастов пока нет. Добавьте RSS-ленту, чтобы подписаться."; + player_podcast_show_unsubscribed: "Show unsubscribed", "Показать без подписки"; + player_podcast_unplayed: "unplayed", "не прослушано"; + player_podcast_episodes: "episodes", "выпусков"; + player_podcast_seasons: "seasons", "сезонов"; + player_podcast_unsubscribed: "Not subscribed", "Нет подписки"; + player_podcast_refresh: "Refresh", "Обновить"; + player_podcast_refreshing: "Checking the feed…", "Проверяем ленту…"; + player_podcast_refresh_updated: "Feed updated", "Лента обновлена"; + player_podcast_refresh_not_modified: "No changes in the feed", "Изменений в ленте нет"; + player_podcast_refresh_failed: "Could not refresh the feed", "Не удалось обновить ленту"; + player_podcast_last_checked: "Checked", "Проверено"; + player_podcast_never_checked: "Not checked yet", "Ещё не проверялась"; + player_podcast_fetch_error: "Last check failed", "Последняя проверка не удалась"; + player_podcast_next_check: "next check", "следующая проверка"; + player_podcast_unsubscribe: "Unsubscribe", "Отписаться"; + player_podcast_unsubscribe_confirm: "Unsubscribe from “{title}”? The feed will no longer be checked. Your progress and listening history are kept: subscribing again continues where you stopped.", "Отписаться от «{title}»? Лента больше не будет проверяться. Прогресс и история прослушиваний сохранятся: при повторной подписке вы продолжите с того же места."; + player_podcast_subscribe: "Subscribe", "Подписаться"; + player_podcast_all_seasons: "All seasons", "Все сезоны"; + player_podcast_season: "Season", "Сезон"; + player_podcast_no_season: "No season", "Без сезона"; + player_podcast_filter_all: "All episodes", "Все выпуски"; + player_podcast_filter_unplayed: "Unplayed", "Не прослушано"; + player_podcast_filter_in_progress: "In progress", "Начато"; + player_podcast_filter_played: "Played", "Прослушано"; + player_podcast_sort_newest: "Newest first", "Сначала новые"; + player_podcast_sort_oldest: "Oldest first", "Сначала старые"; + player_podcast_sort_number: "By number", "По номеру"; + player_podcast_search: "Search episodes", "Поиск по выпускам"; + player_podcast_no_episodes: "No episodes match.", "Подходящих выпусков нет."; + player_podcast_play: "Play", "Слушать"; + player_podcast_resume: "Resume", "Продолжить"; + player_podcast_restart: "Start over", "Начать сначала"; + player_podcast_mark_played: "Mark played", "Прослушано"; + player_podcast_mark_unplayed: "Mark unplayed", "Не прослушано"; + player_podcast_add_queue: "Add to queue", "В очередь"; + player_podcast_left: "left", "осталось"; + player_podcast_saved_at: "Saved at", "Сохранено на"; + player_podcast_played_badge: "Played", "Прослушано"; + player_podcast_played_times: "Completed listens", "Полных прослушиваний"; + player_podcast_not_in_feed: "No longer in the feed", "Больше нет в ленте"; + player_podcast_load_more: "Load more", "Показать ещё"; + player_podcast_history: "Recently listened", "Недавно прослушано"; + player_podcast_jam_blocked: "Podcast episodes are private and cannot be added to a Jam.", "Выпуски подкастов приватны и не добавляются в Jam."; + player_podcast_source_failed: "Could not load this episode from the publisher.", "Не удалось загрузить выпуск у издателя."; + player_podcast_cannot_seek: "The publisher server cannot seek to the saved position. Press Play to listen from the start; until then the saved position is kept.", "Сервер издателя не умеет перематывать к сохранённой позиции. Нажмите Play, чтобы слушать с начала; до этого сохранённая позиция не меняется."; + player_podcast_loading: "Preparing the episode…", "Готовим выпуск…"; + player_podcast_feed: "Feed", "Лента"; + player_podcast_request_failed: "Request failed", "Запрос не выполнен"; player_shuffle: "Shuffle" , "Перемешать"; player_repeat: "Repeat" , "Повтор"; player_volume: "Volume" , "Громкость"; diff --git a/src/jobs/mod.rs b/src/jobs/mod.rs index 3e997d4..f9e50de 100644 --- a/src/jobs/mod.rs +++ b/src/jobs/mod.rs @@ -6,6 +6,7 @@ pub mod lastfm_popularity; pub mod lastfm_scrobble; pub mod metadata_backfill; pub mod musicbrainz; +pub mod podcast_refresh; use std::path::{Component, Path, PathBuf}; diff --git a/src/jobs/podcast_refresh.rs b/src/jobs/podcast_refresh.rs new file mode 100644 index 0000000..c330ed2 --- /dev/null +++ b/src/jobs/podcast_refresh.rs @@ -0,0 +1,73 @@ +//! Background refresh of subscribed podcast feeds. +//! +//! Every run takes the podcasts whose next check is due (see +//! [`crate::podcasts::runner`]): healthy feeds are checked about hourly with +//! conditional requests, failing ones back off exponentially up to a day. +//! One request per feed URL serves every account subscribed to it, a +//! bounded number of requests run concurrently, and a failing feed never +//! blocks the others or clears the stored catalog. + +use crate::podcasts::{FeedFetcher, PodcastStore, now_ms, runner}; +use crate::scheduler::{Job, JobContext, JobLog}; + +pub struct PodcastRefreshJob; + +/// Session-level advisory lock: one run at a time across processes. +const ADVISORY_LOCK_ID: i64 = 0x504f_4443_5246_5348; // "PODCRFSH" + +/// Failures listed individually in the run log. +const LOGGED_FAILURES: usize = 20; + +#[async_trait::async_trait] +impl Job for PodcastRefreshJob { + fn name(&self) -> &'static str { + "podcast_refresh" + } + + fn description(&self) -> &'static str { + "Check subscribed podcast feeds for new episodes (metadata only, no audio)" + } + + fn default_cron(&self) -> &'static str { + // Every five minutes; each feed is checked only when due. + "0 */5 * * * *" + } + + async fn run(&self, ctx: &JobContext, log: &mut JobLog) -> anyhow::Result<()> { + let mut conn = ctx.pool.acquire().await?; + // A session lock must never return to the pool, including on + // cancellation; the store below uses other pool connections. + conn.close_on_drop(); + let locked: bool = sqlx::query_scalar("SELECT pg_try_advisory_lock($1)") + .bind(ADVISORY_LOCK_ID) + .fetch_one(&mut *conn) + .await?; + if !locked { + log.info("Podcast refresh is already running; skipping this run"); + return Ok(()); + } + let store = PodcastStore::new(ctx.pool.clone()); + let fetcher = FeedFetcher::new()?; + let result = runner::refresh_due( + &store, + &fetcher, + now_ms(), + runner::RUN_LIMIT, + runner::RUN_CONCURRENCY, + ) + .await; + conn.close().await?; + let summary = result?; + for (podcast_id, error) in summary.failures.iter().take(LOGGED_FAILURES) { + log.warn(&format!("podcast {podcast_id}: {error}")); + } + if summary.failures.len() > LOGGED_FAILURES { + log.warn(&format!( + "... and {} more failed feeds", + summary.failures.len() - LOGGED_FAILURES + )); + } + log.info(&format!("Podcast refresh: {summary}")); + Ok(()) + } +} diff --git a/src/library_cleanup.rs b/src/library_cleanup.rs index 8f77a26..ea38851 100644 --- a/src/library_cleanup.rs +++ b/src/library_cleanup.rs @@ -20,6 +20,7 @@ struct PlaybackStateRow { current_track_id: Option, position_ms: i32, queue_json: String, + queue_items_json: Option, queue_position: i32, } @@ -738,7 +739,7 @@ async fn cleanup_playback_states( ) -> anyhow::Result<()> { let deleted: HashSet = track_ids.iter().copied().collect(); let states: Vec = sqlx::query_as( - "SELECT id, current_track_id, position_ms, queue_json, queue_position FROM furumusic__playback_state FOR UPDATE", + "SELECT id, current_track_id, position_ms, queue_json, queue_items_json, queue_position FROM furumusic__playback_state FOR UPDATE", ) .fetch_all(&mut **transaction) .await?; @@ -746,6 +747,14 @@ async fn cleanup_playback_states( for state in states { let mut queue: Vec = serde_json::from_str(&state.queue_json).unwrap_or_default(); let original_queue = queue.clone(); + let mut typed = crate::player::queue_items::StoredQueue::from_columns( + state.queue_items_json.as_ref(), + &crate::player::queue_items::LegacyColumns { + queue: original_queue.clone(), + current_track_id: state.current_track_id, + queue_position: state.queue_position, + }, + ); queue.retain(|track_id| !deleted.contains(track_id)); let current_track_id = state.current_track_id.filter(|id| !deleted.contains(id)); if queue == original_queue && current_track_id == state.current_track_id { @@ -763,16 +772,28 @@ async fn cleanup_playback_states( .clamp(0, i32::try_from(queue.len() - 1).unwrap_or(i32::MAX)) } }); + if let Some(typed) = &mut typed { + typed.retain_library( + |track_id| !deleted.contains(&track_id), + crate::player::queue_items::LegacyColumns { + queue: queue.clone(), + current_track_id, + queue_position, + }, + ); + } let position_ms = if current_track_id.is_some() { state.position_ms } else { 0 }; let queue_json = serde_json::to_string(&queue)?; + // The typed queue keeps its other items; a stale one is cleared. + let queue_items_json = typed.map(serde_json::to_value).transpose()?; sqlx::query( r#"UPDATE furumusic__playback_state SET current_track_id = $2, position_ms = $3, - queue_json = $4, queue_position = $5 + queue_json = $4, queue_position = $5, queue_items_json = $6 WHERE id = $1"#, ) .bind(state.id) @@ -780,6 +801,7 @@ async fn cleanup_playback_states( .bind(position_ms) .bind(queue_json) .bind(queue_position) + .bind(queue_items_json) .execute(&mut **transaction) .await?; } diff --git a/src/main.rs b/src/main.rs index d3a02c4..1918424 100644 --- a/src/main.rs +++ b/src/main.rs @@ -15,6 +15,7 @@ mod metrics; mod music; mod oidc; mod player; +mod podcasts; mod scheduler; mod similarity; mod torrents; @@ -63,6 +64,7 @@ fn build_registry() -> Arc { registry.register(jobs::metadata_backfill::MetadataBackfillJob); registry.register(jobs::lastfm_popularity::LastfmPopularityJob); registry.register(jobs::lastfm_scrobble::LastfmScrobbleJob); + registry.register(jobs::podcast_refresh::PodcastRefreshJob); Arc::new(registry) } @@ -475,6 +477,10 @@ impl Project for FuruProject { " API:\n", " FURU_SWAGGER_ENABLED Enable Swagger UI at /swagger/ (default: false)\n", "\n", + " Podcasts:\n", + " FURU_PODCAST_AUDIO_PROXY_ENABLED Stream episode audio through this\n", + " server (default: false)\n", + "\n", "QUICK START\n", " export FURU_DATABASE_URL=postgres://user:pass@localhost/furumusic\n", " furumusic run", @@ -579,6 +585,10 @@ impl Project for FuruProject { "/admin", ); apps.register_with_views(api::ApiApp, "/api"); + apps.register_with_views( + podcasts::http::PodcastsApp::new(Arc::clone(&self.app_config)), + "/api/podcasts", + ); apps.register_with_views( player::PlayerApp::new( Arc::clone(&self.app_config), diff --git a/src/music/mod.rs b/src/music/mod.rs index 2ec4b03..e8323e7 100644 --- a/src/music/mod.rs +++ b/src/music/mod.rs @@ -2838,6 +2838,180 @@ pub mod db_migrations { &[Operation::custom(add_import_playlist_destinations).build()]; } + // -- M0054: typed playback queue next to the legacy id list ------------ + + // (`m_0053` is taken by `src/user.rs` in the same app.) + + // `queue_json` keeps the library track ids and is still written, so an + // older release can run against the same database. The typed value adds + // the items that are not library tracks (podcast episodes, unresolved + // federated music, Jam placeholders) with their explicit source. + #[cot::db::migrations::migration_op] + async fn add_typed_playback_queue( + ctx: migrations::MigrationContext<'_>, + ) -> cot::db::Result<()> { + ctx.db + .raw( + "ALTER TABLE furumusic__playback_state + ADD COLUMN IF NOT EXISTS queue_items_json JSONB", + ) + .await?; + Ok(()) + } + + #[cot::db::migrations::migration_op] + async fn drop_typed_playback_queue( + ctx: migrations::MigrationContext<'_>, + ) -> cot::db::Result<()> { + ctx.db + .raw("ALTER TABLE furumusic__playback_state DROP COLUMN IF EXISTS queue_items_json") + .await?; + Ok(()) + } + + #[derive(Debug, Copy, Clone)] + pub struct M0054AddTypedPlaybackQueue; + impl migrations::Migration for M0054AddTypedPlaybackQueue { + const APP_NAME: &'static str = "furumusic"; + const MIGRATION_NAME: &'static str = "m_0054_add_typed_playback_queue"; + const DEPENDENCIES: &'static [migrations::MigrationDependency] = + &[migrations::MigrationDependency::migration( + "furumusic", + "m_0052_add_import_playlist_destinations", + )]; + const OPERATIONS: &'static [Operation] = &[Operation::custom(add_typed_playback_queue) + .backwards(drop_typed_playback_queue) + .build()]; + } + + // -- M0055: opaque podcast sync relay state (device-sync protocol v4) --- + + // Podcast registers received from trusted devices, kept as the shared + // contract JSON so this server can return them in its snapshots. This is + // not the podcast store: the web player has no podcast features yet. + #[cot::db::migrations::migration_op] + async fn create_fed_podcast_relay( + ctx: migrations::MigrationContext<'_>, + ) -> cot::db::Result<()> { + ctx.db + .raw( + "CREATE TABLE IF NOT EXISTS furumusic__fed_podcast_relay ( + user_id BIGINT NOT NULL, + section VARCHAR(16) NOT NULL, + record_key TEXT NOT NULL, + record_json JSONB NOT NULL, + updated_at_ms BIGINT NOT NULL, + PRIMARY KEY (user_id, section, record_key) + )", + ) + .await?; + Ok(()) + } + + /// Removes the operations a 0.12.x release cannot parse: the podcast + /// kinds relayed since device-sync protocol 4. Ends with its `WHERE` + /// clause, so tests can narrow it to one user. + pub(crate) const DELETE_PODCAST_SYNC_OPS_SQL: &str = + "DELETE FROM furumusic__fed_sync_ops WHERE kind LIKE 'podcast\\_%'"; + + // Release 0.12.x parses every unacknowledged operation it sends to a + // peer and fails the whole batch on an unknown kind, so the podcast + // operations relayed by 0.13 must go before it runs again (see + // docs/administration.md, "Rolling back to 0.12"). + #[cot::db::migrations::migration_op] + async fn drop_fed_podcast_relay(ctx: migrations::MigrationContext<'_>) -> cot::db::Result<()> { + ctx.db.raw(DELETE_PODCAST_SYNC_OPS_SQL).await?; + ctx.db + .raw("DROP TABLE IF EXISTS furumusic__fed_podcast_relay") + .await?; + Ok(()) + } + + #[derive(Debug, Copy, Clone)] + pub struct M0055CreateFedPodcastRelay; + impl migrations::Migration for M0055CreateFedPodcastRelay { + const APP_NAME: &'static str = "furumusic"; + const MIGRATION_NAME: &'static str = "m_0055_create_fed_podcast_relay"; + const DEPENDENCIES: &'static [migrations::MigrationDependency] = + &[migrations::MigrationDependency::migration( + "furumusic", + "m_0054_add_typed_playback_queue", + )]; + const OPERATIONS: &'static [Operation] = &[Operation::custom(create_fed_podcast_relay) + .backwards(drop_fed_podcast_relay) + .build()]; + } + + // -- M0056: PostgreSQL podcast store (device-sync protocol v4) --------- + + // Per-account podcast catalog, synced registers and local fetch state; + // see `crate::podcasts` and `crate::podcasts::schema`. + #[cot::db::migrations::migration_op] + async fn create_podcast_store(ctx: migrations::MigrationContext<'_>) -> cot::db::Result<()> { + for sql in crate::podcasts::schema::CREATE_PODCAST_STORE { + ctx.db.raw(sql).await?; + } + Ok(()) + } + + #[cot::db::migrations::migration_op] + async fn drop_podcast_store(ctx: migrations::MigrationContext<'_>) -> cot::db::Result<()> { + for sql in crate::podcasts::schema::DROP_PODCAST_STORE { + ctx.db.raw(sql).await?; + } + Ok(()) + } + + #[derive(Debug, Copy, Clone)] + pub struct M0056CreatePodcastStore; + impl migrations::Migration for M0056CreatePodcastStore { + const APP_NAME: &'static str = "furumusic"; + const MIGRATION_NAME: &'static str = "m_0056_create_podcast_store"; + const DEPENDENCIES: &'static [migrations::MigrationDependency] = + &[migrations::MigrationDependency::migration( + "furumusic", + "m_0055_create_fed_podcast_relay", + )]; + const OPERATIONS: &'static [Operation] = &[Operation::custom(create_podcast_store) + .backwards(drop_podcast_store) + .build()]; + } + + // -- M0057: import the 0.13.0 podcast relay state into the store -------- + + // The relay table stays (unused) until a later cleanup release, so a + // rollback to 0.13.0 still finds the state it relayed. + #[cot::db::migrations::migration_op] + async fn import_podcast_relay(ctx: migrations::MigrationContext<'_>) -> cot::db::Result<()> { + for sql in crate::podcasts::schema::import_statements("TRUE") { + ctx.db.raw(&sql).await?; + } + Ok(()) + } + + #[cot::db::migrations::migration_op] + async fn keep_podcast_store_rows( + _ctx: migrations::MigrationContext<'_>, + ) -> cot::db::Result<()> { + // The imported rows go with the tables (`m_0056` backwards). + Ok(()) + } + + #[derive(Debug, Copy, Clone)] + pub struct M0057ImportPodcastRelay; + impl migrations::Migration for M0057ImportPodcastRelay { + const APP_NAME: &'static str = "furumusic"; + const MIGRATION_NAME: &'static str = "m_0057_import_podcast_relay"; + const DEPENDENCIES: &'static [migrations::MigrationDependency] = + &[migrations::MigrationDependency::migration( + "furumusic", + "m_0056_create_podcast_store", + )]; + const OPERATIONS: &'static [Operation] = &[Operation::custom(import_podcast_relay) + .backwards(keep_podcast_store_rows) + .build()]; + } + pub const MIGRATIONS: &[&SyncDynMigration] = &[ &M0006CreateMediaFile, &M0007CreateArtist, @@ -2881,5 +3055,9 @@ pub mod db_migrations { &M0050CreateMetadataFixHistory, &M0051AddListenStatisticsIndexes, &M0052AddImportPlaylistDestinations, + &M0054AddTypedPlaybackQueue, + &M0055CreateFedPodcastRelay, + &M0056CreatePodcastStore, + &M0057ImportPodcastRelay, ]; } diff --git a/src/player/dto.rs b/src/player/dto.rs index 3c68d40..5e54bab 100644 --- a/src/player/dto.rs +++ b/src/player/dto.rs @@ -147,6 +147,17 @@ pub(super) struct PlaybackStateDto { pub(super) shuffle: bool, pub(super) repeat_mode: String, pub(super) volume: f64, + /// Typed queue (podcast episodes and other non-library items included). + /// + /// Requests send `{"kind": "library", "track_id"}` and + /// `{"kind": "external", "track": }` items; + /// responses return external items as browser track JSON. Clients that + /// omit it keep using `queue`, which is always written. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub(super) queue_items: Option, + /// Index of the current item in `queue_items`. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub(super) queue_items_position: Option, } #[derive(Debug, Deserialize, JsonSchema)] @@ -263,6 +274,10 @@ pub(super) struct PlayerDevicePlaybackStateDto { pub(super) volume: f64, #[serde(default)] pub(super) updated_at_ms: i64, + /// The browser does not know the position of its current item, a podcast + /// episode it cannot play (`queue_items::POSITION_UNKNOWN_KEY`). + #[serde(default, skip_serializing_if = "std::ops::Not::not")] + pub(super) position_unknown: bool, } #[derive(Debug, Serialize, JsonSchema)] diff --git a/src/player/mod.rs b/src/player/mod.rs index ac6e0bd..e4dbe71 100644 --- a/src/player/mod.rs +++ b/src/player/mod.rs @@ -29,6 +29,7 @@ use crate::youtube::{YouTubePreviewRequest, YouTubeService, YouTubeStartRequest} mod dto; mod helpers; mod queries; +pub(crate) mod queue_items; mod rows; mod sharing; @@ -606,7 +607,7 @@ impl PlayerDeviceHub { target_device_id: Option<&str>, jam_id: Option<&str>, command: &str, - payload: serde_json::Value, + mut payload: serde_json::Value, ) -> Result<(), &'static str> { let now = current_millis(); let mut state = self.state.lock().expect("player device hub lock"); @@ -621,6 +622,15 @@ impl PlayerDeviceHub { let target_id = self .jam_target_device_id_locked(&state, jam) .ok_or("jam playback device is offline")?; + if user_id != jam.host_user_id { + let host_tracks = state + .playback_state_by_user + .get(&jam.host_user_id) + .map(|playback| playback.tracks.clone()) + .unwrap_or_default(); + restore_jam_command_payload(command, &mut payload, &host_tracks)?; + stamp_jam_participant_command(&mut payload); + } (jam.host_user_id, target_id) } else { let target_id = match target_device_id { @@ -1042,11 +1052,17 @@ impl PlayerDeviceHub { }) }) .unwrap_or(user_id); - state + let playback = state .playback_state_by_user .get(&playback_user_id) .cloned() - .map(|playback_state| playback_state_at(playback_state, now)) + .map(|playback_state| playback_state_at(playback_state, now)); + if playback_user_id == user_id { + return playback; + } + // Jam participants are outside the host's trusted devices: private + // items (podcast episodes) become opaque placeholders. + playback.map(redact_playback_state_for_jam) } fn jam_dtos_locked( @@ -1457,6 +1473,7 @@ mod device_tests { repeat_mode: "off".to_string(), volume: 0.7, updated_at_ms: current_millis(), + position_unknown: false, }, ); } @@ -1646,6 +1663,246 @@ pub struct PlayerPageTemplate { pub youtube_downloads_enabled: bool, } +#[cfg(test)] +mod queue_persistence_tests { + use super::*; + use music_dht::device_sync::PlaybackTrack; + + fn offline_pool() -> sqlx::PgPool { + sqlx::postgres::PgPoolOptions::new() + .connect_lazy("postgres://offline@127.0.0.1:1/offline") + .unwrap() + } + + fn request(queue_items: serde_json::Value, position: usize) -> PlaybackStateDto { + PlaybackStateDto { + current_track_id: Some(3), + position_ms: 1_000, + queue: vec![3, 4], + queue_position: 1, + shuffle: false, + repeat_mode: "off".into(), + volume: 0.5, + queue_items: Some(queue_items), + queue_items_position: Some(position), + } + } + + #[tokio::test] + async fn podcast_items_survive_persistence_and_reload() { + let episode = PlaybackTrack::podcast_episode(queue_items::test_episode()); + let browser_track = queue_items::external_track_json(&episode, 1, None); + // The browser sends `_queueItemsPayload(queue.tracks)`. + let dto = request( + serde_json::json!([ + {"kind": "library", "track_id": 3}, + {"kind": "external", "track": browser_track["wire_track"]}, + {"kind": "library", "track_id": 4}, + ]), + 1, + ); + let stored = stored_queue_from_request(&dto).unwrap(); + // Written to queue_items_json and read back with the legacy ids. + let column = serde_json::to_value(&stored).unwrap(); + let row = queue_items::LegacyColumns { + queue: dto.queue.clone(), + current_track_id: dto.current_track_id, + queue_position: dto.queue_position, + }; + let loaded = queue_items::StoredQueue::from_columns(Some(&column), &row).unwrap(); + assert_eq!(loaded, stored); + assert_eq!(loaded.position, 1); + let items = browser_queue_items(&offline_pool(), &loaded).await.unwrap(); + assert_eq!( + items[0], + serde_json::json!({"kind": "library", "track_id": 3}) + ); + assert_eq!(items[1]["kind"], "external"); + assert_eq!(items[1]["track"]["source"]["kind"], "podcast_episode"); + assert_eq!( + queue_items::wire_track_from_browser_json(&items[1]["track"]), + Some(episode) + ); + } + + #[test] + fn malformed_typed_queues_keep_the_legacy_ids_only() { + let dto = request( + serde_json::json!([{"kind": "external", "track": {"id": "x"}}]), + 0, + ); + assert!(stored_queue_from_request(&dto).is_none()); + let legacy_only = PlaybackStateDto { + queue_items: None, + ..request(serde_json::json!([]), 0) + }; + assert!(stored_queue_from_request(&legacy_only).is_none()); + // Older clients do not send the new fields. + let old: PlaybackStateDto = serde_json::from_value(serde_json::json!({ + "current_track_id": 3, "position_ms": 0, "queue": [3], "queue_position": 0, + "shuffle": false, "repeat_mode": "off", "volume": 0.7, + })) + .unwrap(); + assert!(old.queue_items.is_none()); + assert!(!serde_json::to_string(&old).unwrap().contains("queue_items")); + } +} + +#[cfg(test)] +mod jam_command_tests { + use super::*; + use music_dht::device_sync::{PRIVATE_ITEM_TITLE, PlaybackTrack}; + + fn library(id: i64) -> serde_json::Value { + serde_json::json!({"id": id, "title": format!("Song {id}"), "duration_seconds": 100.0}) + } + + fn host_tracks() -> Vec { + let episode = PlaybackTrack::podcast_episode(queue_items::test_episode()); + vec![ + library(1), + queue_items::external_track_json(&episode, 1, None), + library(2), + ] + } + + fn check(command: &str, payload: serde_json::Value) -> Result<(), &'static str> { + let mut payload = payload; + restore_jam_command_payload(command, &mut payload, &host_tracks()) + } + + #[test] + fn participants_cannot_remove_or_clear_private_items() { + assert!(check("queue_remove", serde_json::json!({"index": 1})).is_err()); + assert!(check("queue_clear", serde_json::json!({})).is_err()); + // Music can still be removed, and private items moved. + assert!(check("queue_remove", serde_json::json!({"index": 0})).is_ok()); + assert!(check("queue_remove", serde_json::json!({"index": 2})).is_ok()); + assert!( + check( + "queue_move", + serde_json::json!({"from_index": 1, "to_index": 0}) + ) + .is_ok() + ); + // The host runs `Number(index)`: other spellings of an index are refused. + for index in [ + serde_json::json!("1"), + serde_json::json!(1.0), + serde_json::json!(-1), + ] { + assert!(check("queue_remove", serde_json::json!({"index": index})).is_err()); + } + assert!(check("queue_remove", serde_json::json!({})).is_err()); + // Without private items the host queue can be cleared as before. + let mut payload = serde_json::json!({}); + assert!(restore_jam_command_payload("queue_clear", &mut payload, &[library(1)]).is_ok()); + } + + #[test] + fn participant_commands_reach_the_host_stamped() { + let hub = PlayerDeviceHub::default(); + hub.heartbeat(1, "host", None, None, None); + let jam = hub + .create_jam(1, "Host", "host", vec![(2, "Guest".to_string())]) + .unwrap(); + let jam_id = jam.jams[0].id.clone(); + hub.heartbeat(2, "guest", None, None, None); + hub.join_jam(2, "Guest", "guest", &jam_id).unwrap(); + let tracks = host_tracks(); + hub.heartbeat( + 1, + "host", + None, + Some(&jam_id), + Some(PlayerDevicePlaybackStateDto { + track: Some(tracks[0].clone()), + tracks, + index: 0, + position_seconds: 0.0, + duration_seconds: 100.0, + paused: false, + shuffle: false, + repeat_mode: "off".into(), + volume: 0.5, + updated_at_ms: 0, + position_unknown: false, + }), + ); + // Move the episode to the end, then remove index 2: the cached host + // state still has music there, so only the host can refuse it. + for (command, payload) in [ + ( + "queue_move", + serde_json::json!({"from_index": 1, "to_index": 2}), + ), + ("queue_remove", serde_json::json!({"index": 2})), + ("pause", serde_json::Value::Null), + ] { + hub.enqueue_command(2, None, Some(&jam_id), command, payload) + .unwrap(); + } + hub.enqueue_command( + 1, + None, + Some(&jam_id), + "queue_remove", + serde_json::json!({"index": 0}), + ) + .unwrap(); + let commands = hub.poll(1, "host", None, Some(&jam_id), None).commands; + let stamped: Vec = commands + .iter() + .map(|command| command.payload[JAM_PARTICIPANT_KEY] == true) + .collect(); + assert_eq!(stamped, [true, true, true, false]); + assert_eq!(commands[1].payload["index"], 2); + } + + #[test] + fn redacted_current_episode_has_no_duration() { + let tracks = host_tracks(); + let playback = PlayerDevicePlaybackStateDto { + track: Some(tracks[1].clone()), + tracks: tracks.clone(), + index: 1, + position_seconds: 1200.0, + duration_seconds: 1800.0, + paused: false, + shuffle: false, + repeat_mode: "off".into(), + volume: 0.5, + updated_at_ms: 0, + position_unknown: false, + }; + let redacted = redact_playback_state_for_jam(playback.clone()); + assert_eq!(redacted.duration_seconds, 0.0); + assert_eq!( + redacted.track.as_ref().unwrap()["title"], + PRIVATE_ITEM_TITLE + ); + assert_eq!(redacted.position_seconds, 1200.0); + let leaked = serde_json::to_string(&redacted).unwrap(); + assert!(!leaked.contains("1800") && !leaked.contains("Episode")); + + // A music track keeps its duration. + let music = redact_playback_state_for_jam(PlayerDevicePlaybackStateDto { + track: Some(tracks[0].clone()), + index: 0, + duration_seconds: 100.0, + ..playback.clone() + }); + assert_eq!(music.duration_seconds, 100.0); + + // Without `track` the current item is taken from `tracks[index]`. + let untracked = redact_playback_state_for_jam(PlayerDevicePlaybackStateDto { + track: None, + ..playback + }); + assert_eq!(untracked.duration_seconds, 0.0); + } +} + #[cfg(test)] mod page_template_tests { use super::*; @@ -5755,6 +6012,16 @@ async fn devices_select_handler( .and_then(normalize_device_id) .unwrap_or_else(|| target_device_id.clone()); let previous_active_id = hub.active_device_id_for_commands(user.id); + // Refuse a handoff the target cannot receive before ownership moves. + if let Some(fed_device_id) = fed_device_id_from_virtual(&target_device_id) + && previous_active_id.as_deref() != Some(target_device_id.as_str()) + && let Some(state) = hub.playback_state_json_for_commands(user.id) + && let Err(err) = crate::federation::handle() + .fed_device_web_transfer_check(user.id, fed_device_id, state) + .await + { + return Ok(json_error(StatusCode::CONFLICT, &format!("{err}"))); + } let Some(response) = hub.select(user.id, ¤t_device_id, &target_device_id) else { return Ok(json_error( @@ -5992,6 +6259,158 @@ async fn fed_devices_sync_handler( } } +/// Host playback state as shown to Jam participants +/// (`music_dht::device_sync::redact_for_jam` over the browser queue). +fn redact_playback_state_for_jam( + mut playback: PlayerDevicePlaybackStateDto, +) -> PlayerDevicePlaybackStateDto { + let index = usize::try_from(playback.index).unwrap_or(0); + let current_is_private = playback + .track + .as_ref() + .or_else(|| playback.tracks.get(index)) + .is_some_and(queue_items::browser_track_requires_podcast_support); + playback.tracks = queue_items::redact_tracks_for_jam(&playback.tracks); + if current_is_private { + // Placeholders carry no duration; the host's position is kept like + // `redact_for_jam` keeps `position_secs`. + playback.duration_seconds = 0.0; + } + if playback + .track + .as_ref() + .is_some_and(queue_items::browser_track_requires_podcast_support) + { + playback.track = playback.tracks.get(index).cloned().or_else(|| { + Some( + queue_items::redact_tracks_for_jam(std::slice::from_ref(playback.track.as_ref()?)) + [0] + .clone(), + ) + }); + } + playback +} + +/// Applies the Jam privacy rules to a participant command. +/// +/// Participants cannot add podcast episodes. A full queue they send (play or +/// transfer) gets the host's private items back in place of the +/// placeholders; queues that drop, duplicate or inject private items are +/// rejected. Participants cannot remove a private item, nor clear a queue +/// that holds one; they may still move it (`queue_move`). +fn restore_jam_command_payload( + command: &str, + payload: &mut serde_json::Value, + host_tracks: &[serde_json::Value], +) -> Result<(), &'static str> { + const REJECTED: &str = "podcast episodes are private and cannot be changed in a jam"; + let tracks: Vec = payload + .get("tracks") + .and_then(serde_json::Value::as_array) + .cloned() + .unwrap_or_default(); + let current = payload + .get("track") + .filter(|track| !track.is_null()) + .cloned(); + match command { + "queue_add_end" | "queue_add_next" => { + if tracks + .iter() + .any(queue_items::browser_track_requires_podcast_support) + { + return Err(REJECTED); + } + } + "play_track" | "play_from_index" | "transfer_state" => { + if !queue_items::jam_tracks_allowed(&tracks) + || current.as_ref().is_some_and(|track| { + !queue_items::jam_tracks_allowed(std::slice::from_ref(track)) + }) + { + return Err(REJECTED); + } + if tracks.is_empty() { + if current + .as_ref() + .is_some_and(queue_items::browser_track_requires_podcast_support) + { + return Err(REJECTED); + } + return Ok(()); + } + let restored = queue_items::restore_jam_tracks(host_tracks, &tracks).ok_or(REJECTED)?; + if let Some(object) = payload.as_object_mut() { + if current + .as_ref() + .is_some_and(queue_items::browser_track_requires_podcast_support) + { + let index = object + .get("index") + .and_then(serde_json::Value::as_u64) + .unwrap_or(0) as usize; + let track = restored.get(index).cloned().ok_or(REJECTED)?; + object.insert("track".to_string(), track); + } + object.insert("tracks".to_string(), serde_json::Value::Array(restored)); + } + } + "queue_remove" => { + // The participant view has the host's length and order, so the + // index names the same item on the host. + let index = payload + .get("index") + .and_then(serde_json::Value::as_u64) + .and_then(|index| usize::try_from(index).ok()); + let removes_private = match index { + Some(index) => host_tracks + .get(index) + .is_some_and(queue_items::browser_track_requires_podcast_support), + // The host browser runs `Number(index)`, so "1" or 1.0 would + // still name an item: only plain indexes pass a private queue. + None => host_tracks + .iter() + .any(queue_items::browser_track_requires_podcast_support), + }; + if removes_private { + return Err(REJECTED); + } + } + "queue_clear" + if host_tracks + .iter() + .any(queue_items::browser_track_requires_podcast_support) => + { + return Err(REJECTED); + } + _ => {} + } + Ok(()) +} + +/// Marks a Jam participant command for the host browser. +/// +/// `restore_jam_command_payload` checks the command against the host state of +/// the host's last report. The host browser runs its pending commands in one +/// batch, so an earlier command (a move, an insert) can make a checked index +/// name a private item there. The host browser applies the same rules to its +/// own queue for stamped commands (`_jamParticipantCommandAllowed`). +fn stamp_jam_participant_command(payload: &mut serde_json::Value) { + if !payload.is_object() { + *payload = serde_json::json!({}); + } + if let Some(object) = payload.as_object_mut() { + object.insert( + JAM_PARTICIPANT_KEY.to_string(), + serde_json::Value::Bool(true), + ); + } +} + +/// Command payload key set by the server on Jam participant commands. +const JAM_PARTICIPANT_KEY: &str = "jam_participant"; + fn stamp_jam_queue_tracks(payload: &mut serde_json::Value, user_id: i64, user_name: &str) { let Some(tracks) = payload .get_mut("tracks") @@ -6320,7 +6739,7 @@ async fn get_state_handler( }; let state = sqlx::query_as::<_, PlaybackStateRow>( - r#"SELECT current_track_id, position_ms, queue_json, queue_position, shuffle, repeat_mode::text as repeat_mode, volume + r#"SELECT current_track_id, position_ms, queue_json, queue_items_json, queue_position, shuffle, repeat_mode::text as repeat_mode, volume FROM furumusic__playback_state WHERE user_id = $1"#, ) .bind(user.id) @@ -6331,6 +6750,26 @@ async fn get_state_handler( let dto = match state { Some(s) => { let queue: Vec = serde_json::from_str(&s.queue_json).unwrap_or_default(); + // Prefer the typed queue while it matches the legacy columns. + let typed = queue_items::StoredQueue::from_columns( + s.queue_items_json.as_ref(), + &queue_items::LegacyColumns { + queue: queue.clone(), + current_track_id: s.current_track_id, + queue_position: s.queue_position, + }, + ); + let (queue_items, queue_items_position) = match typed { + Some(stored) => ( + Some( + browser_queue_items(pool, &stored) + .await + .map_err(|e| cot::Error::internal(e.to_string()))?, + ), + Some(stored.position), + ), + None => (None, None), + }; PlaybackStateDto { current_track_id: s.current_track_id, position_ms: s.position_ms, @@ -6339,6 +6778,8 @@ async fn get_state_handler( shuffle: s.shuffle, repeat_mode: s.repeat_mode, volume: s.volume, + queue_items, + queue_items_position, } } None => PlaybackStateDto { @@ -6349,12 +6790,51 @@ async fn get_state_handler( shuffle: false, repeat_mode: "off".to_string(), volume: 0.7, + queue_items: None, + queue_items_position: None, }, }; Json(dto).into_response() } +/// Typed queue items for the browser: library ids to load, external items +/// as browser track JSON with their verbatim wire item. +async fn browser_queue_items( + pool: &sqlx::PgPool, + stored: &queue_items::StoredQueue, +) -> anyhow::Result { + let mut items = Vec::with_capacity(stored.items.len()); + for (index, item) in stored.items.iter().enumerate() { + items.push(match item { + queue_items::StoredQueueItem::Library { track_id } => { + serde_json::json!({"kind": "library", "track_id": track_id}) + } + queue_items::StoredQueueItem::External { track } => serde_json::json!({ + "kind": "external", + "track": crate::federation::devices::resolve_playback_track(pool, track, index).await?, + }), + }); + } + Ok(serde_json::Value::Array(items)) +} + +/// Typed queue to persist from a state request; `None` keeps only the legacy +/// ids (older clients, malformed or oversized queues). +fn stored_queue_from_request(dto: &PlaybackStateDto) -> Option { + let items: Vec = + serde_json::from_value(dto.queue_items.clone()?).ok()?; + queue_items::StoredQueue::new( + items, + dto.queue_items_position.unwrap_or(0), + queue_items::LegacyColumns { + queue: dto.queue.clone(), + current_track_id: dto.current_track_id, + queue_position: dto.queue_position, + }, + ) +} + // --------------------------------------------------------------------------- // PUT /api/player/state // --------------------------------------------------------------------------- @@ -6372,15 +6852,21 @@ async fn put_state_handler( let queue_json = serde_json::to_string(&dto.queue).map_err(|e| cot::Error::internal(e.to_string()))?; + // `queue_json` is still written so an older release keeps working. + let queue_items_json = stored_queue_from_request(&dto) + .map(serde_json::to_value) + .transpose() + .map_err(|e| cot::Error::internal(e.to_string()))?; let now = chrono::Utc::now().format("%Y-%m-%dT%H:%M:%SZ").to_string(); sqlx::query( - r#"INSERT INTO furumusic__playback_state (user_id, current_track_id, position_ms, queue_json, queue_position, shuffle, repeat_mode, volume, updated_at) - VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9) + r#"INSERT INTO furumusic__playback_state (user_id, current_track_id, position_ms, queue_json, queue_position, shuffle, repeat_mode, volume, updated_at, queue_items_json) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10) ON CONFLICT (user_id) DO UPDATE SET current_track_id = $2, position_ms = $3, queue_json = $4, - queue_position = $5, shuffle = $6, repeat_mode = $7, volume = $8, updated_at = $9"#, + queue_position = $5, shuffle = $6, repeat_mode = $7, volume = $8, updated_at = $9, + queue_items_json = $10"#, ) .bind(user.id) .bind(dto.current_track_id) @@ -6391,6 +6877,7 @@ async fn put_state_handler( .bind(&dto.repeat_mode) .bind(dto.volume) .bind(&now) + .bind(queue_items_json) .execute(pool) .await .map_err(|e| cot::Error::internal(e.to_string()))?; diff --git a/src/player/queue_items.rs b/src/player/queue_items.rs new file mode 100644 index 0000000..9e621fb --- /dev/null +++ b/src/player/queue_items.rs @@ -0,0 +1,710 @@ +//! Lossless queue items shared by the browser player, device sync and the +//! persisted playback state. +//! +//! The browser queue holds plain track JSON objects. Library tracks of this +//! server keep their usual shape. Every other item received from a trusted +//! device (unresolved federated music, podcast episodes, Jam private +//! placeholders) becomes an *external* item: display fields for the queue UI +//! plus the verbatim device-sync DTO under [`WIRE_TRACK_KEY`] and its explicit +//! [`PlaybackSource`] under [`SOURCE_KEY`]. Converting back to the wire uses +//! the verbatim DTO, so such items survive browser -> server -> peers -> +//! server -> browser unchanged. The web player plays podcast episodes +//! through `/api/podcasts/episodes/{id}/playback` (publisher URL or the +//! optional proxy); placeholders and unresolved federated music stay +//! unplayable here. +//! +//! Gating and Jam decisions are delegated to `music_dht::device_sync`; this +//! module only maps browser JSON onto the shared DTOs. + +use music_dht::device_sync::{ + PRIVATE_ITEM_TITLE, PlaybackCommand, PlaybackSource, PlaybackStateWire, PlaybackTrack, + jam_command_allowed, redact_for_jam, restore_jam_private_items, +}; +use serde::{Deserialize, Serialize}; +use serde_json::Value; + +/// Browser track key holding the verbatim [`PlaybackTrack`] of an external item. +pub(crate) const WIRE_TRACK_KEY: &str = "wire_track"; +/// Browser track key holding the explicit [`PlaybackSource`] of an external item. +pub(crate) const SOURCE_KEY: &str = "source"; +/// Browser playback state key: `true` when the browser does not know the +/// position of its current item. That happens when it selects a podcast +/// episode it cannot play (Next, auto-advance, a click or a play command): +/// it is never told where the episode was left, and inventing 0 would +/// restart the episode for every device. The server then takes the +/// position from the episode's progress register. +pub(crate) const POSITION_UNKNOWN_KEY: &str = "position_unknown"; + +/// Reason shown for federated music that is not in this library. +pub(crate) const FED_UNAVAILABLE_REASON: &str = "Not available on this web player yet"; +/// Reason shown for Jam placeholders of someone else's private item. +pub(crate) const PRIVATE_UNAVAILABLE_REASON: &str = "Private item of the Jam host"; + +/// Version of the typed queue stored in `furumusic__playback_state.queue_items_json`. +pub(crate) const STORED_QUEUE_VERSION: u16 = 1; +/// Upper bound of persisted queue items; larger queues keep only `queue_json`. +pub(crate) const MAX_STORED_QUEUE_ITEMS: usize = 10_000; + +/// Browser JSON for a queue item that is not a library track of this server. +/// +/// `content_id` is the normalized content id of a music item, when known. +/// The `id` is a negative placeholder that is stable for the item and its +/// queue position, so the browser can tell items apart; it never refers to a +/// library track. +pub(crate) fn external_track_json( + track: &PlaybackTrack, + index: usize, + content_id: Option<&str>, +) -> Value { + let mut json = match &track.source { + Some(PlaybackSource::PodcastEpisode(episode)) => serde_json::json!({ + "id": placeholder_track_id(Some(episode.episode_id.as_str()), index), + "title": episode.episode_title, + "track_number": serde_json::Value::Null, + "disc_number": serde_json::Value::Null, + "duration_seconds": episode.duration_ms.unwrap_or(0).max(0) as f64 / 1_000.0, + "artists": artist_names_to_json(std::slice::from_ref(&episode.podcast_title)), + "featured_artists": [], + "release_id": 0, + "release_title": episode.podcast_title, + "release_year": serde_json::Value::Null, + // Artwork is loaded from the publisher, like the audio. + "cover_url": episode.image_url, + // The player asks `/api/podcasts/episodes/{id}/playback` for the + // audio source when it starts the episode. + "stream_url": "", + "uploader_name": "Podcast", + "audio_format": episode.mime, + "content_id": serde_json::Value::Null, + "unavailable": false, + }), + Some(PlaybackSource::Private) => serde_json::json!({ + "id": placeholder_track_id(Some("private"), index), + "title": PRIVATE_ITEM_TITLE, + "track_number": serde_json::Value::Null, + "disc_number": serde_json::Value::Null, + "duration_seconds": 0.0, + "artists": [], + "featured_artists": [], + "release_id": 0, + "release_title": "", + "release_year": serde_json::Value::Null, + "cover_url": serde_json::Value::Null, + "stream_url": "", + "uploader_name": "", + "content_id": serde_json::Value::Null, + "unavailable": true, + "unavailable_reason": PRIVATE_UNAVAILABLE_REASON, + }), + Some(PlaybackSource::Music) | None => music_placeholder_json(track, index, content_id), + }; + if let Some(object) = json.as_object_mut() { + if let Some(source) = &track.source + && let Ok(source) = serde_json::to_value(source) + { + object.insert(SOURCE_KEY.to_string(), source); + } + if let Ok(wire) = serde_json::to_value(track) { + object.insert(WIRE_TRACK_KEY.to_string(), wire); + } + } + json +} + +/// Display fields of unresolved federated music (unchanged from earlier +/// releases, which had no verbatim wire item). +fn music_placeholder_json(track: &PlaybackTrack, index: usize, content_id: Option<&str>) -> Value { + let artists = if !track.artist_names.is_empty() { + track.artist_names.clone() + } else if let Some(fed) = &track.fed { + fed.artist_names.clone() + } else { + Vec::new() + }; + let featured = if !track.featured_artist_names.is_empty() { + track.featured_artist_names.clone() + } else if let Some(fed) = &track.fed { + fed.featured_artist_names.clone() + } else { + Vec::new() + }; + let title = track + .fed + .as_ref() + .map(|fed| fed.title.as_str()) + .filter(|title| !title.trim().is_empty()) + .unwrap_or(&track.title); + let release_title = track + .fed + .as_ref() + .and_then(|fed| fed.release_title.as_deref()) + .filter(|title| !title.trim().is_empty()) + .unwrap_or(&track.release_title); + let content_id = content_id + .map(ToString::to_string) + .or_else(|| track.content_id.clone()) + .or_else(|| track.fed.as_ref().map(|fed| fed.content_id.clone())); + serde_json::json!({ + "id": placeholder_track_id(content_id.as_deref(), index), + "title": title, + "track_number": track.track_number, + "disc_number": track.disc_number, + "duration_seconds": track.duration_seconds, + "artists": artist_names_to_json(&artists), + "featured_artists": artist_names_to_json(&featured), + "release_id": 0, + "release_title": release_title, + "release_year": track.release_year, + "cover_url": serde_json::Value::Null, + "stream_url": "", + "uploader_name": "Federation", + "audio_format": track.audio_format.clone(), + "audio_bitrate": track.audio_bitrate, + "audio_sample_rate": track.audio_sample_rate, + "audio_bit_depth": track.audio_bit_depth, + "file_size_bytes": track.file_size_bytes, + "content_id": content_id, + "unavailable": true, + "unavailable_reason": FED_UNAVAILABLE_REASON, + }) +} + +fn artist_names_to_json(names: &[String]) -> Vec { + names + .iter() + .enumerate() + .filter(|(_, name)| !name.trim().is_empty()) + .map(|(index, name)| serde_json::json!({"id": -((index as i64) + 1), "name": name})) + .collect() +} + +fn placeholder_track_id(seed: Option<&str>, index: usize) -> i64 { + let mut bytes = [0u8; 8]; + let hash = blake3::hash(seed.unwrap_or("").as_bytes()); + bytes.copy_from_slice(&hash.as_bytes()[..8]); + let value = i64::from_le_bytes(bytes).unsigned_abs() % 9_000_000_000_000; + -((value as i64) + index as i64 + 1) +} + +/// The device-sync item carried by an external browser track. +/// +/// Returns the verbatim [`WIRE_TRACK_KEY`] DTO when present. Otherwise a +/// browser item that only names a podcast or private [`SOURCE_KEY`] is +/// rebuilt with the shared constructors. Library tracks (and malformed +/// items) return `None` and keep the library mapping. +pub(crate) fn wire_track_from_browser_json(track: &Value) -> Option { + if let Some(wire) = track.get(WIRE_TRACK_KEY).filter(|wire| wire.is_object()) + && let Ok(wire) = serde_json::from_value::(wire.clone()) + { + return Some(wire); + } + match serde_json::from_value::(track.get(SOURCE_KEY)?.clone()).ok()? { + PlaybackSource::PodcastEpisode(episode) => Some(PlaybackTrack::podcast_episode(*episode)), + PlaybackSource::Private => Some(PlaybackTrack::private_placeholder()), + PlaybackSource::Music => None, + } +} + +/// Whether a browser item is a podcast episode or a Jam placeholder. +pub(crate) fn browser_track_requires_podcast_support(track: &Value) -> bool { + wire_track_from_browser_json(track).is_some_and(|track| track.requires_podcast_support()) +} + +/// Wire-shaped view of browser tracks for the shared gating and Jam helpers. +/// +/// External items are their verbatim DTOs. Library tracks become minimal +/// music items: the helpers only look at item sources and positions, so +/// this conversion needs no database lookups. +pub(crate) fn shadow_wire_state(tracks: &[Value], index: usize) -> PlaybackStateWire { + PlaybackStateWire { + queue: tracks + .iter() + .map(|track| wire_track_from_browser_json(track).unwrap_or_else(|| music_stub(track))) + .collect(), + queue_pos: index, + playing: false, + paused: true, + idle_since_ms: None, + position_secs: 0.0, + volume: 0, + shuffle: false, + repeat: Default::default(), + playback_rate: None, + } +} + +fn music_stub(track: &Value) -> PlaybackTrack { + PlaybackTrack { + id: track.get("id").and_then(Value::as_i64).unwrap_or(0), + title: String::new(), + track_number: None, + disc_number: None, + duration_seconds: 0.0, + artist_names: Vec::new(), + featured_artist_names: Vec::new(), + release_id: 0, + release_title: String::new(), + release_year: None, + file_path: String::new(), + content_id: None, + audio_format: None, + audio_bitrate: None, + audio_sample_rate: None, + audio_bit_depth: None, + file_size_bytes: None, + play_count: 0, + fed: None, + source: None, + } +} + +/// Browser tracks as shown to Jam participants. +/// +/// Uses [`redact_for_jam`]: every podcast episode (and existing placeholder) +/// becomes the opaque private placeholder; music, order and position stay. +pub(crate) fn redact_tracks_for_jam(tracks: &[Value]) -> Vec { + let redacted = redact_for_jam(shadow_wire_state(tracks, 0)); + tracks + .iter() + .zip(redacted.queue.iter()) + .enumerate() + .map(|(index, (original, redacted))| { + if redacted.is_private_placeholder() { + external_track_json(redacted, index, None) + } else { + original.clone() + } + }) + .collect() +} + +/// Maps the private placeholders of a Jam participant's full queue back to +/// the host's items ([`restore_jam_private_items`]). +/// +/// Returns `None` when the participant queue contains real podcast episodes +/// or drops, duplicates or injects private items; the command is then +/// rejected. +pub(crate) fn restore_jam_tracks(host: &[Value], participant: &[Value]) -> Option> { + let host_state = shadow_wire_state(host, 0); + let restored = restore_jam_private_items(&host_state, shadow_wire_state(participant, 0))?; + let mut host_private = host + .iter() + .zip(host_state.queue.iter()) + .filter(|(_, wire)| wire.requires_podcast_support()) + .map(|(json, _)| json); + let participant_state = shadow_wire_state(participant, 0); + let mut out = Vec::with_capacity(participant.len()); + for ((json, before), after) in participant + .iter() + .zip(participant_state.queue.iter()) + .zip(restored.queue.iter()) + { + if before.is_private_placeholder() { + out.push(host_private.next()?.clone()); + } else { + debug_assert_eq!(before, after); + out.push(json.clone()); + } + } + Some(out) +} + +/// Whether tracks sent by a Jam participant may enter the host queue at all +/// ([`jam_command_allowed`]): participants cannot add podcast episodes. +pub(crate) fn jam_tracks_allowed(tracks: &[Value]) -> bool { + jam_command_allowed(&PlaybackCommand::SetState { + state: shadow_wire_state(tracks, 0), + seek: false, + }) +} + +/// Typed playback queue persisted next to the legacy `queue_json`. +/// +/// `queue_json` (a list of library track ids) is still written for rollback +/// to older releases. `legacy_queue`, `legacy_current_track_id` and +/// `legacy_queue_position` record the legacy columns written alongside this +/// value: when they no longer match the row (an older release or a +/// legacy-only writer changed the queue or moved to another track since), +/// readers ignore the typed value and fall back to the legacy columns. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +pub(crate) struct StoredQueue { + /// Shape version, [`STORED_QUEUE_VERSION`]. + pub(crate) version: u16, + /// Index of the current item in `items`. + #[serde(default)] + pub(crate) position: usize, + /// Queue items in order. + #[serde(default)] + pub(crate) items: Vec, + /// `queue_json` ids written together with this value. + #[serde(default)] + pub(crate) legacy_queue: Vec, + /// `current_track_id` written together with this value. + #[serde(default)] + pub(crate) legacy_current_track_id: Option, + /// `queue_position` written together with this value (`None`: unknown, + /// the value is not used). + #[serde(default)] + pub(crate) legacy_queue_position: Option, +} + +/// Legacy queue columns of a `furumusic__playback_state` row. +#[derive(Debug, Clone, PartialEq, Eq, Default)] +pub(crate) struct LegacyColumns { + /// `queue_json`: library track ids. + pub(crate) queue: Vec, + /// `current_track_id`. + pub(crate) current_track_id: Option, + /// `queue_position`. + pub(crate) queue_position: i32, +} + +/// One persisted queue item. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[serde(tag = "kind", rename_all = "snake_case")] +pub(crate) enum StoredQueueItem { + /// Library track of this server. + Library { + /// Local track id. + track_id: i64, + }, + /// Any other item, kept verbatim with its explicit `source`. + External { + /// Device-sync queue item. + track: Box, + }, +} + +impl StoredQueue { + /// Builds the persisted value; `None` when the queue exceeds the bound. + /// + /// Jam private placeholders are dropped: they stand for the Jam host's + /// items, never for items of this user's own queue. + pub(crate) fn new( + items: Vec, + position: usize, + legacy: LegacyColumns, + ) -> Option { + if items.len() > MAX_STORED_QUEUE_ITEMS { + return None; + } + let mut stored = Self { + version: STORED_QUEUE_VERSION, + position, + items, + legacy_queue: Vec::new(), + legacy_current_track_id: None, + legacy_queue_position: None, + }; + stored.retain_items(|item| match item { + StoredQueueItem::Library { .. } => true, + StoredQueueItem::External { track } => !track.is_private_placeholder(), + }); + stored.set_legacy(legacy); + Some(stored) + } + + /// Reads the typed column, preferring it over the legacy columns only + /// when it has a known version and was written together with exactly + /// these legacy values. + pub(crate) fn from_columns(typed: Option<&Value>, legacy: &LegacyColumns) -> Option { + let stored = serde_json::from_value::(typed?.clone()).ok()?; + (stored.version == STORED_QUEUE_VERSION + && stored.legacy_queue == legacy.queue + && stored.legacy_current_track_id == legacy.current_track_id + && stored.legacy_queue_position == Some(legacy.queue_position) + && stored.items.len() <= MAX_STORED_QUEUE_ITEMS) + .then_some(stored) + } + + /// Removes library tracks rejected by `keep` and records the new legacy + /// columns. The current position follows its item, or the item after it + /// when the current item was removed. + pub(crate) fn retain_library(&mut self, keep: impl Fn(i64) -> bool, legacy: LegacyColumns) { + self.retain_items(|item| match item { + StoredQueueItem::Library { track_id } => keep(*track_id), + StoredQueueItem::External { .. } => true, + }); + self.set_legacy(legacy); + } + + fn retain_items(&mut self, keep: impl Fn(&StoredQueueItem) -> bool) { + let original_position = self.position; + let mut position = self.position; + let mut index = 0; + self.items.retain(|item| { + let retained = keep(item); + if !retained && index < original_position { + position = position.saturating_sub(1); + } + index += 1; + retained + }); + self.position = position.min(self.items.len().saturating_sub(1)); + } + + fn set_legacy(&mut self, legacy: LegacyColumns) { + self.legacy_queue = legacy.queue; + self.legacy_current_track_id = legacy.current_track_id; + self.legacy_queue_position = Some(legacy.queue_position); + } +} + +/// A valid podcast episode reference for tests. +#[cfg(test)] +pub(crate) fn test_episode() -> music_dht::podcast::PodcastEpisodeRef { + use music_dht::podcast::{EpisodeId, FeedUrl, PodcastEpisodeRef, PodcastId}; + let feed_url = FeedUrl::parse("https://podcasts.example/feed.xml?token=Ab").unwrap(); + let podcast_id = PodcastId::from_feed_url(&feed_url); + let episode_id = EpisodeId::from_guid(&podcast_id, "episode-42").unwrap(); + PodcastEpisodeRef { + podcast_id, + episode_id, + feed_url, + podcast_title: "Example Cast".into(), + episode_title: "Episode 42".into(), + audio_url: "https://cdn.example/ep42.mp3".into(), + mime: Some("audio/mpeg".into()), + duration_ms: Some(1_800_000), + published_at_ms: Some(1_700_000_000_000), + season: Some(2), + episode_number: Some(42), + image_url: None, + progress_generation: Some(7), + } +} + +#[cfg(test)] +mod tests { + use super::*; + use music_dht::podcast::EpisodeId; + + fn episode() -> music_dht::podcast::PodcastEpisodeRef { + test_episode() + } + + fn library(id: i64) -> Value { + serde_json::json!({"id": id, "title": format!("Track {id}"), "stream_url": format!("/api/player/stream/{id}")}) + } + + #[test] + fn podcast_item_round_trips_through_browser_json() { + let mut wire = PlaybackTrack::podcast_episode(episode()); + wire.play_count = 3; + let json = external_track_json(&wire, 4, None); + // Playable on the web player (the source is resolved on play). + assert_eq!(json["unavailable"], false); + assert!(json.get("unavailable_reason").is_none()); + assert_eq!(json["stream_url"], ""); + assert_eq!( + json["cover_url"], + Value::Null, + "the test episode has no artwork" + ); + assert_eq!(json["title"], "Episode 42"); + assert_eq!(json[SOURCE_KEY]["kind"], "podcast_episode"); + assert!(json["id"].as_i64().unwrap() < 0); + // The browser stores and echoes the object; the item is unchanged. + let echoed: Value = serde_json::from_str(&json.to_string()).unwrap(); + assert_eq!(wire_track_from_browser_json(&echoed), Some(wire)); + } + + #[test] + fn private_and_unresolved_music_keep_their_wire_item() { + let private = PlaybackTrack::private_placeholder(); + let json = external_track_json(&private, 0, None); + assert_eq!(json["title"], PRIVATE_ITEM_TITLE); + assert_eq!(json[SOURCE_KEY]["kind"], "private"); + assert_eq!(wire_track_from_browser_json(&json), Some(private)); + + let mut music = PlaybackTrack::private_placeholder(); + music.source = None; + music.title = "Remote song".into(); + music.content_id = Some(format!("b3:{}", "a".repeat(64))); + let json = external_track_json(&music, 1, music.content_id.as_deref()); + assert!(json.get(SOURCE_KEY).is_none()); + assert_eq!(json["unavailable_reason"], FED_UNAVAILABLE_REASON); + assert_eq!(wire_track_from_browser_json(&json), Some(music)); + } + + #[test] + fn library_tracks_are_not_external() { + assert_eq!(wire_track_from_browser_json(&library(5)), None); + let mut music_source = library(6); + music_source[SOURCE_KEY] = serde_json::json!({"kind": "music"}); + assert_eq!(wire_track_from_browser_json(&music_source), None); + } + + #[test] + fn source_only_items_use_shared_constructors() { + let json = serde_json::json!({ + "id": -1, + SOURCE_KEY: serde_json::to_value(PlaybackSource::PodcastEpisode(Box::new(episode()))).unwrap(), + }); + assert_eq!( + wire_track_from_browser_json(&json), + Some(PlaybackTrack::podcast_episode(episode())) + ); + // A malformed verbatim item falls back to the explicit source. + let mut tampered = json.clone(); + tampered[WIRE_TRACK_KEY] = serde_json::json!({"id": "x"}); + assert_eq!( + wire_track_from_browser_json(&tampered), + Some(PlaybackTrack::podcast_episode(episode())) + ); + } + + #[test] + fn jam_redaction_hides_episodes_and_restores_them_in_order() { + let first = external_track_json(&PlaybackTrack::podcast_episode(episode()), 1, None); + let mut other = episode(); + other.episode_title = "Episode 43".into(); + other.episode_id = EpisodeId::from_guid(&other.podcast_id, "episode-43").unwrap(); + let second = external_track_json(&PlaybackTrack::podcast_episode(other), 3, None); + let host = vec![library(1), first.clone(), library(2), second.clone()]; + + let redacted = redact_tracks_for_jam(&host); + assert_eq!(redacted[0], host[0]); + assert_eq!(redacted[1]["title"], PRIVATE_ITEM_TITLE); + let leaked = serde_json::to_string(&redacted).unwrap(); + assert!(!leaked.contains("Episode 4") && !leaked.contains("podcasts.example")); + assert!(!leaked.contains("cdn.example") && !leaked.contains("Example Cast")); + + // The participant reorders music and moves a placeholder. + let participant = vec![ + redacted[3].clone(), + library(2), + redacted[1].clone(), + library(1), + ]; + let restored = restore_jam_tracks(&host, &participant).unwrap(); + assert_eq!( + restored, + vec![first.clone(), library(2), second, library(1)] + ); + + // Dropping, duplicating or injecting private items is rejected. + assert!(restore_jam_tracks(&host, &[library(1), redacted[1].clone()]).is_none()); + assert!(restore_jam_tracks(&host, &[first.clone(), redacted[1].clone()]).is_none()); + assert!(!jam_tracks_allowed(std::slice::from_ref(&first))); + assert!(jam_tracks_allowed(&[library(9)])); + } + + fn legacy(queue: &[i64], current_track_id: Option, queue_position: i32) -> LegacyColumns { + LegacyColumns { + queue: queue.to_vec(), + current_track_id, + queue_position, + } + } + + #[test] + fn stored_queue_prefers_typed_value_only_while_legacy_ids_match() { + let items = vec![ + StoredQueueItem::Library { track_id: 3 }, + StoredQueueItem::External { + track: Box::new(PlaybackTrack::podcast_episode(episode())), + }, + StoredQueueItem::Library { track_id: 4 }, + ]; + // The episode is current: the browser wrote its placeholder id. + let written = legacy(&[3, -11, 4], Some(-11), 1); + let stored = StoredQueue::new(items, 1, written.clone()).unwrap(); + let value = serde_json::to_value(&stored).unwrap(); + assert_eq!(value["version"], 1); + assert_eq!(value["items"][1]["kind"], "external"); + assert_eq!( + value["items"][1]["track"]["source"]["kind"], + "podcast_episode" + ); + assert_eq!( + StoredQueue::from_columns(Some(&value), &written), + Some(stored.clone()) + ); + // A rollback release rewrote queue_json: fall back to it. + assert_eq!( + StoredQueue::from_columns(Some(&value), &legacy(&[3, 4], Some(4), 1)), + None + ); + // A rollback release kept the ids but moved to another track (Next + // or a click): the typed position is stale, fall back as well. + assert_eq!( + StoredQueue::from_columns(Some(&value), &legacy(&[3, -11, 4], Some(4), 2)), + None + ); + assert_eq!( + StoredQueue::from_columns(Some(&value), &legacy(&[3, -11, 4], Some(-11), 0)), + None + ); + // Values without the recorded current item are not trusted. + let mut unknown = value.clone(); + unknown + .as_object_mut() + .unwrap() + .remove("legacy_queue_position"); + assert_eq!(StoredQueue::from_columns(Some(&unknown), &written), None); + let mut future = value.clone(); + future["version"] = serde_json::json!(2); + assert_eq!(StoredQueue::from_columns(Some(&future), &written), None); + assert_eq!(StoredQueue::from_columns(None, &written), None); + } + + #[test] + fn stored_queue_never_keeps_jam_placeholders() { + let items = vec![ + StoredQueueItem::Library { track_id: 3 }, + StoredQueueItem::External { + track: Box::new(PlaybackTrack::private_placeholder()), + }, + StoredQueueItem::External { + track: Box::new(PlaybackTrack::podcast_episode(episode())), + }, + StoredQueueItem::Library { track_id: 4 }, + ]; + let stored = StoredQueue::new(items.clone(), 3, legacy(&[3, 4], Some(4), 1)).unwrap(); + assert_eq!( + stored.items, + vec![items[0].clone(), items[2].clone(), items[3].clone()] + ); + assert_eq!(stored.position, 2); + // A current placeholder hands the position to the next item. + let stored = StoredQueue::new(items.clone(), 1, legacy(&[3, 4], None, 0)).unwrap(); + assert_eq!(stored.position, 1); + assert_eq!(stored.items[1], items[2]); + } + + #[test] + fn stored_queue_cleanup_keeps_external_items_and_position() { + let items = vec![ + StoredQueueItem::Library { track_id: 3 }, + StoredQueueItem::External { + track: Box::new(PlaybackTrack::podcast_episode(episode())), + }, + StoredQueueItem::Library { track_id: 4 }, + ]; + let mut stored = StoredQueue::new(items.clone(), 1, legacy(&[3, 4], None, 0)).unwrap(); + stored.retain_library(|id| id != 3, legacy(&[4], None, 0)); + assert_eq!(stored.items, items[1..].to_vec()); + assert_eq!(stored.position, 0); + assert_eq!(stored.legacy_queue, vec![4]); + // The cleanup's new legacy columns are what readers compare with. + let value = serde_json::to_value(&stored).unwrap(); + assert_eq!( + StoredQueue::from_columns(Some(&value), &legacy(&[4], None, 0)), + Some(stored) + ); + + let mut current_removed = + StoredQueue::new(items.clone(), 2, legacy(&[3, 4], Some(4), 1)).unwrap(); + current_removed.retain_library(|id| id != 4, legacy(&[3], None, 0)); + assert_eq!(current_removed.position, 1); + assert!( + StoredQueue::new( + vec![items[0].clone(); MAX_STORED_QUEUE_ITEMS + 1], + 0, + LegacyColumns::default() + ) + .is_none() + ); + } +} diff --git a/src/player/rows.rs b/src/player/rows.rs index b016a4a..9756048 100644 --- a/src/player/rows.rs +++ b/src/player/rows.rs @@ -65,6 +65,7 @@ pub(super) struct PlaybackStateRow { pub(super) current_track_id: Option, pub(super) position_ms: i32, pub(super) queue_json: String, + pub(super) queue_items_json: Option, pub(super) queue_position: i32, pub(super) shuffle: bool, pub(super) repeat_mode: String, diff --git a/src/podcasts/actions.rs b/src/podcasts/actions.rs new file mode 100644 index 0000000..a15679e --- /dev/null +++ b/src/podcasts/actions.rs @@ -0,0 +1,702 @@ +//! Local user actions: they build the device-sync operation, append it to +//! the account's sync log with its stamp and apply it through the shared +//! code path, all in one transaction. + +use anyhow::Result; +use music_dht::device_sync::{SyncOpPayload, SyncOpWire}; +use music_dht::podcast::{ + EpisodeId, FeedUrl, ListenSession, PodcastEpisodeRef, PodcastId, PodcastMeta, ProgressRecord, + ProgressStatus, effective_share, merge_feed_aliases, merge_listen_session, + next_podcast_generation, resolve_podcast_id, +}; +use sqlx::{PgConnection, Row}; + +use super::apply::{self, ApplyOutcome}; +use super::{PodcastStore, PodcastStoreError, invalid}; +use crate::federation::devices; + +/// Result of [`PodcastStore::subscribe`]. +#[derive(Debug, Clone, PartialEq)] +pub(crate) struct SubscribeOutcome { + /// Podcast the URL resolved to (an existing one when the URL is a known + /// feed URL or alias). + pub(crate) podcast_id: PodcastId, + /// The podcast was already subscribed; no operation was emitted. + pub(crate) already_subscribed: bool, + /// Operations appended to the sync log (and applied). + pub(crate) ops: Vec, +} + +/// A listening session of this account as seen by the progress register. +/// +/// Returned by explicit actions; periodic checkpoints of the session use its +/// `generation` ([`PodcastStore::checkpoint`]). +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct ProgressSession { + /// Episode being played. + pub(crate) episode: PodcastEpisodeRef, + /// Session id (also the `ListenSession::session_id`). + pub(crate) session_id: String, + /// Generation of the session's latest explicit action. + pub(crate) generation: u64, +} + +/// A progress register written by a local action. +#[derive(Debug, Clone, PartialEq)] +pub(crate) struct ProgressWrite { + /// The register as written. + pub(crate) record: ProgressRecord, + /// The session after the action (its generation for later checkpoints). + pub(crate) session: ProgressSession, + /// Operation appended to the sync log. + pub(crate) op: SyncOpWire, +} + +/// Result of [`PodcastStore::checkpoint`]. +#[derive(Debug, Clone, PartialEq)] +pub(crate) enum CheckpointOutcome { + /// The checkpoint was written. + Written(Box), + /// The register holds a greater generation (another device or an + /// explicit action took over). Nothing was written: stop writing at the + /// old generation, or continue locally with an explicit action + /// ([`PodcastStore::start_session`] / [`PodcastStore::seek`]). + Superseded { + /// Current register. + current: ProgressRecord, + }, + /// This session already completed the episode at its generation; a + /// checkpoint would turn it back into "in progress". Nothing was written. + AlreadyCompleted { + /// Current register. + current: ProgressRecord, + }, +} + +#[derive(Debug, Clone, Copy)] +enum ExplicitAction { + Start(i64), + Seek(i64), + Restart, + MarkPlayed, + MarkUnplayed, +} + +/// Validates the payload (with a provisional stamp), appends it to the sync +/// log with the next stamp of `device_id` and applies exactly that operation +/// through [`apply::apply_op`]. `make` receives the operation id, which +/// explicit actions without a session use as their session id. +pub(crate) async fn emit( + conn: &mut PgConnection, + user_id: i64, + device_id: &str, + make: impl Fn(&str) -> SyncOpPayload, +) -> Result { + SyncOpWire { + op_id: "provisional".to_string(), + origin_device_id: device_id.to_string(), + seq: 0, + hlc_ms: 0, + payload: make("provisional"), + } + .validate_podcast() + .map_err(invalid)?; + let op = devices::append_local_op(conn, user_id, device_id, &make).await?; + match apply::apply_op(conn, user_id, &op).await? { + ApplyOutcome::Rejected(reason) => Err(invalid(reason)), + _ => Ok(op), + } +} + +/// Resolves a feed URL against every known URL of every podcast of the +/// account: the alias sets, the catalog fetch URLs and the registers' +/// `podcast.feed_url` (`resolve_podcast_id`). +pub(crate) async fn resolve_podcast( + conn: &mut PgConnection, + user_id: i64, + url: &FeedUrl, +) -> Result { + let rows = sqlx::query( + "SELECT podcast_id FROM furumusic__podcast_feed_alias + WHERE user_id = $1 AND feed_url = $2 + UNION SELECT podcast_id FROM furumusic__podcast + WHERE user_id = $1 AND feed_url = $2 + UNION SELECT podcast_id FROM furumusic__podcast_subscription + WHERE user_id = $1 AND podcast_json->>'feed_url' = $2", + ) + .bind(user_id) + .bind(url.as_str()) + .fetch_all(&mut *conn) + .await?; + let ids = rows + .iter() + .map(|row| super::podcast_id_col(row, "podcast_id")) + .collect::>>()?; + Ok(resolve_podcast_id(url, ids.iter().map(|id| (id, url)))) +} + +async fn catalog_meta( + conn: &mut PgConnection, + user_id: i64, + podcast_id: &PodcastId, +) -> Result> { + let row = sqlx::query( + "SELECT feed_url, title, author, description, link, image_url + FROM furumusic__podcast WHERE user_id = $1 AND podcast_id = $2", + ) + .bind(user_id) + .bind(podcast_id.as_str()) + .fetch_optional(&mut *conn) + .await?; + let Some(row) = row else { + return Ok(None); + }; + Ok(Some(PodcastMeta { + feed_url: super::feed_url_col(&row, "feed_url")?, + title: row.try_get("title")?, + author: row.try_get("author")?, + description: row.try_get("description")?, + link: row.try_get("link")?, + image_url: row.try_get("image_url")?, + })) +} + +async fn share_payload( + conn: &mut PgConnection, + user_id: i64, + podcast_id: &PodcastId, + shared: bool, +) -> Result { + let subscription = apply::load_subscription_register(conn, user_id, podcast_id).await?; + let share = apply::load_share(conn, user_id, podcast_id).await?; + Ok(SyncOpPayload::PodcastShareSet { + podcast_id: podcast_id.clone(), + shared, + generation: next_podcast_generation(subscription.as_ref(), share.as_ref()), + }) +} + +/// The reference carried by operations: identity and display only. +fn op_ref(episode: &PodcastEpisodeRef) -> PodcastEpisodeRef { + let mut episode = episode.clone(); + episode.progress_generation = None; + episode +} + +impl PodcastStore { + /// Subscribes to the podcast of `podcast.feed_url` (explicit user action). + /// + /// The URL is resolved against known feed URLs and aliases first, so + /// subscribing again, through a redirect target or from someone else's + /// card reuses the existing podcast with its progress and history. When + /// the podcast is already subscribed nothing is emitted. Otherwise one + /// `podcast_subscription_set` with generation + /// `next_podcast_generation(..)` and the known aliases is appended and + /// applied. The new subscription is private until + /// [`PodcastStore::set_shared`]. + /// + /// Pass the channel metadata of the fetched feed + /// (`ParsedFeed::to_podcast_meta(url)`), then ingest the episodes with + /// [`PodcastStore::apply_feed`]. + pub(crate) async fn subscribe( + &self, + user_id: i64, + podcast: &PodcastMeta, + ) -> Result { + let meta = podcast.bounded(); + meta.validate().map_err(invalid)?; + let device_id = devices::local_device_id(&self.pool, user_id).await?; + let mut tx = self.begin_write(user_id).await?; + let podcast_id = resolve_podcast(&mut tx, user_id, &meta.feed_url).await?; + let current = apply::load_subscription_register(&mut tx, user_id, &podcast_id).await?; + if current.as_ref().is_some_and(|record| record.subscribed) { + return Ok(SubscribeOutcome { + podcast_id, + already_subscribed: true, + ops: Vec::new(), + }); + } + let active: i64 = sqlx::query_scalar( + "SELECT COUNT(*) FROM furumusic__podcast_subscription + WHERE user_id = $1 AND subscribed", + ) + .bind(user_id) + .fetch_one(&mut *tx) + .await?; + if usize::try_from(active).unwrap_or(usize::MAX) >= super::MAX_SUBSCRIPTIONS_PER_ACCOUNT { + return Err( + PodcastStoreError::SubscriptionLimit(super::MAX_SUBSCRIPTIONS_PER_ACCOUNT).into(), + ); + } + let share = apply::load_share(&mut tx, user_id, &podcast_id).await?; + let generation = next_podcast_generation(current.as_ref(), share.as_ref()); + let aliases = merge_feed_aliases( + &podcast_id, + &apply::load_aliases(&mut tx, user_id, &podcast_id).await?, + std::slice::from_ref(&meta.feed_url), + ); + let op = emit(&mut tx, user_id, &device_id, |_| { + SyncOpPayload::PodcastSubscriptionSet { + podcast_id: podcast_id.clone(), + subscribed: true, + podcast: meta.clone(), + feed_aliases: aliases.clone(), + generation, + } + }) + .await?; + tx.commit().await?; + Ok(SubscribeOutcome { + podcast_id, + already_subscribed: false, + ops: vec![op], + }) + } + + /// Unsubscribes (explicit user action). + /// + /// Emits `podcast_subscription_set { subscribed: false }` followed by + /// `podcast_share_set { shared: false }`, each with the next podcast + /// generation. Progress, sessions and the catalog are kept, so + /// subscribing again continues where the user stopped. Does nothing when + /// the podcast is not subscribed. + pub(crate) async fn unsubscribe( + &self, + user_id: i64, + podcast_id: &PodcastId, + ) -> Result> { + let device_id = devices::local_device_id(&self.pool, user_id).await?; + let mut tx = self.begin_write(user_id).await?; + let Some(current) = apply::load_subscription_register(&mut tx, user_id, podcast_id).await? + else { + return Ok(Vec::new()); + }; + if !current.subscribed { + return Ok(Vec::new()); + } + let share = apply::load_share(&mut tx, user_id, podcast_id).await?; + let generation = next_podcast_generation(Some(¤t), share.as_ref()); + let mut meta = catalog_meta(&mut tx, user_id, podcast_id) + .await? + .map(|meta| meta.bounded()) + .unwrap_or_else(|| current.podcast.clone()); + meta.feed_url = current.podcast.feed_url.clone(); + let aliases = current.feed_aliases.clone(); + let unsubscribe = emit(&mut tx, user_id, &device_id, |_| { + SyncOpPayload::PodcastSubscriptionSet { + podcast_id: podcast_id.clone(), + subscribed: false, + podcast: meta.clone(), + feed_aliases: aliases.clone(), + generation, + } + }) + .await?; + let revoke_payload = share_payload(&mut tx, user_id, podcast_id, false).await?; + let revoke = emit(&mut tx, user_id, &device_id, |_| revoke_payload.clone()).await?; + tx.commit().await?; + Ok(vec![unsubscribe, revoke]) + } + + /// Sets the decision to publish the podcast to the federation. + /// + /// Sharing needs an active subscription and is a no-op when the podcast + /// is already effectively shared (`effective_share`). Revoking is a no-op + /// when no share decision says `true`. The write uses the next podcast + /// generation, so it orders after the subscription it observed. + // The publication toggle is part of public discovery (MR-W3); until + // then only the tests call it. + #[cfg_attr(not(test), allow(dead_code))] + pub(crate) async fn set_shared( + &self, + user_id: i64, + podcast_id: &PodcastId, + shared: bool, + ) -> Result> { + let device_id = devices::local_device_id(&self.pool, user_id).await?; + let mut tx = self.begin_write(user_id).await?; + let subscription = apply::load_subscription_register(&mut tx, user_id, podcast_id).await?; + let share = apply::load_share(&mut tx, user_id, podcast_id).await?; + if shared { + if !subscription + .as_ref() + .is_some_and(|record| record.subscribed) + { + return Err(PodcastStoreError::NotSubscribed(podcast_id.clone()).into()); + } + if effective_share(subscription.as_ref(), share.as_ref()) { + return Ok(Vec::new()); + } + } else if !share.as_ref().is_some_and(|record| record.shared) { + return Ok(Vec::new()); + } + let payload = share_payload(&mut tx, user_id, podcast_id, shared).await?; + let op = emit(&mut tx, user_id, &device_id, |_| payload.clone()).await?; + tx.commit().await?; + Ok(vec![op]) + } + + async fn explicit_progress( + &self, + user_id: i64, + episode: &PodcastEpisodeRef, + session_id: Option<&str>, + carried_generation: Option, + action: ExplicitAction, + ) -> Result { + episode.validate().map_err(invalid)?; + let device_id = devices::local_device_id(&self.pool, user_id).await?; + let mut tx = self.begin_write(user_id).await?; + let current = apply::load_progress(&mut tx, user_id, &episode.episode_id).await?; + let max_seen = current + .as_ref() + .map(|record| record.generation) + .into_iter() + .chain(carried_generation) + .max(); + let generation = ProgressRecord::explicit_generation(max_seen); + let current_position = current.as_ref().map_or(0, |record| record.position_ms); + let (position_ms, status) = match action { + ExplicitAction::Start(position) | ExplicitAction::Seek(position) => { + (position, ProgressStatus::InProgress) + } + ExplicitAction::Restart => (0, ProgressStatus::InProgress), + ExplicitAction::MarkPlayed => (current_position, ProgressStatus::Completed), + ExplicitAction::MarkUnplayed => (0, ProgressStatus::NotStarted), + }; + let reference = op_ref(episode); + let op = emit(&mut tx, user_id, &device_id, |op_id| { + SyncOpPayload::PodcastProgressSet { + episode_id: episode.episode_id.clone(), + podcast_id: episode.podcast_id.clone(), + position_ms, + status, + generation, + session_id: session_id.unwrap_or(op_id).to_string(), + episode: Some(reference.clone()), + } + }) + .await?; + tx.commit().await?; + let record = op.podcast_progress().expect("progress payload"); + Ok(ProgressWrite { + session: ProgressSession { + episode: episode.clone(), + session_id: record.session_id.clone(), + generation, + }, + record, + op, + }) + } + + /// Starts a listening session at `position_ms` (explicit action). + /// + /// The generation is `explicit_generation(max(local, carried_generation))`; + /// pass `PodcastEpisodeRef::progress_generation` of a queue item received + /// through a handoff as `carried_generation`. Ordinary Play resumes from + /// [`PodcastStore::resume_position_ms`]. + pub(crate) async fn start_session( + &self, + user_id: i64, + episode: &PodcastEpisodeRef, + session_id: &str, + position_ms: i64, + carried_generation: Option, + ) -> Result { + self.explicit_progress( + user_id, + episode, + Some(session_id), + carried_generation, + ExplicitAction::Start(position_ms), + ) + .await + } + + /// Periodic checkpoint (or pause/stop save) of a running session. + /// + /// Keeps the session generation. When the register already holds a + /// greater generation nothing is written ([`CheckpointOutcome::Superseded`]). + pub(crate) async fn checkpoint( + &self, + user_id: i64, + session: &ProgressSession, + position_ms: i64, + ) -> Result { + let episode = &session.episode; + let device_id = devices::local_device_id(&self.pool, user_id).await?; + let mut tx = self.begin_write(user_id).await?; + if let Some(current) = apply::load_progress(&mut tx, user_id, &episode.episode_id).await? { + if current.generation > session.generation { + return Ok(CheckpointOutcome::Superseded { current }); + } + if current.generation == session.generation + && current.session_id == session.session_id + && current.status == ProgressStatus::Completed + { + return Ok(CheckpointOutcome::AlreadyCompleted { current }); + } + } + let reference = op_ref(episode); + let op = emit(&mut tx, user_id, &device_id, |_| { + SyncOpPayload::PodcastProgressSet { + episode_id: episode.episode_id.clone(), + podcast_id: episode.podcast_id.clone(), + position_ms, + status: ProgressStatus::InProgress, + generation: session.generation, + session_id: session.session_id.clone(), + episode: Some(reference.clone()), + } + }) + .await?; + tx.commit().await?; + let record = op.podcast_progress().expect("progress payload"); + Ok(CheckpointOutcome::Written(Box::new(ProgressWrite { + record, + session: session.clone(), + op, + }))) + } + + /// User seek during a session (explicit action; the returned session + /// carries the new generation for later checkpoints). + pub(crate) async fn seek( + &self, + user_id: i64, + session: &ProgressSession, + position_ms: i64, + ) -> Result { + self.explicit_progress( + user_id, + &session.episode, + Some(&session.session_id), + Some(session.generation), + ExplicitAction::Seek(position_ms), + ) + .await + } + + /// Completes a running session (end of stream or `reaches_completion`) + /// unless another write took over. Record the session itself with + /// [`PodcastStore::record_listen_session`]. + /// + /// Like [`PodcastStore::checkpoint`], nothing is written when the + /// register holds a greater generation ([`CheckpointOutcome::Superseded`]) + /// or when this session already completed the episode + /// ([`CheckpointOutcome::AlreadyCompleted`]). Otherwise `completed` is + /// written with an explicit generation; the check and the write + /// happen in one transaction, so a periodic save after the end never + /// completes the episode twice. + pub(crate) async fn complete_if_current( + &self, + user_id: i64, + session: &ProgressSession, + position_ms: i64, + ) -> Result { + let episode = &session.episode; + episode.validate().map_err(invalid)?; + let device_id = devices::local_device_id(&self.pool, user_id).await?; + let mut tx = self.begin_write(user_id).await?; + let current = apply::load_progress(&mut tx, user_id, &episode.episode_id).await?; + if let Some(current) = current.as_ref() { + if current.generation > session.generation { + return Ok(CheckpointOutcome::Superseded { + current: current.clone(), + }); + } + if current.session_id == session.session_id + && current.status == ProgressStatus::Completed + { + return Ok(CheckpointOutcome::AlreadyCompleted { + current: current.clone(), + }); + } + } + let generation = ProgressRecord::explicit_generation( + current + .as_ref() + .map(|record| record.generation) + .into_iter() + .chain(Some(session.generation)) + .max(), + ); + let reference = op_ref(episode); + let op = emit(&mut tx, user_id, &device_id, |_| { + SyncOpPayload::PodcastProgressSet { + episode_id: episode.episode_id.clone(), + podcast_id: episode.podcast_id.clone(), + position_ms, + status: ProgressStatus::Completed, + generation, + session_id: session.session_id.clone(), + episode: Some(reference.clone()), + } + }) + .await?; + tx.commit().await?; + let record = op.podcast_progress().expect("progress payload"); + Ok(CheckpointOutcome::Written(Box::new(ProgressWrite { + record, + session: ProgressSession { + episode: episode.clone(), + session_id: session.session_id.clone(), + generation, + }, + op, + }))) + } + + /// "Start over": position 0, in progress, explicit generation. + pub(crate) async fn restart( + &self, + user_id: i64, + episode: &PodcastEpisodeRef, + session_id: &str, + ) -> Result { + self.explicit_progress( + user_id, + episode, + Some(session_id), + None, + ExplicitAction::Restart, + ) + .await + } + + /// "Mark played": status completed, explicit generation. Creates no + /// listening session and invents no listening time. + pub(crate) async fn mark_played( + &self, + user_id: i64, + episode: &PodcastEpisodeRef, + ) -> Result { + self.explicit_progress(user_id, episode, None, None, ExplicitAction::MarkPlayed) + .await + } + + /// "Mark unplayed": status not started at position 0, explicit generation. + pub(crate) async fn mark_unplayed( + &self, + user_id: i64, + episode: &PodcastEpisodeRef, + ) -> Result { + self.explicit_progress(user_id, episode, None, None, ExplicitAction::MarkUnplayed) + .await + } + + /// Records (or grows) a listening session of this account's device. + /// + /// Sessions below the history threshold (`ListenSession::should_record`) + /// and versions that do not grow the stored session emit nothing. + /// `session.device_id` must be the account's web device id. The episode + /// reference defaults to the catalog's. + pub(crate) async fn record_listen_session( + &self, + user_id: i64, + session: &ListenSession, + episode: Option<&PodcastEpisodeRef>, + ) -> Result> { + let device_id = devices::local_device_id(&self.pool, user_id).await?; + if session.device_id != device_id { + return Err(invalid( + "session device_id differs from the local device id", + )); + } + session.validate().map_err(invalid)?; + if !session.should_record() { + return Ok(None); + } + let mut tx = self.begin_write(user_id).await?; + if let Some(current) = apply::load_session(&mut tx, user_id, &session.session_id).await? { + let merged = merge_listen_session(¤t, session).map_err(invalid)?; + if merged == current { + return Ok(None); + } + } + let reference = match episode { + Some(episode) => Some(op_ref(episode)), + None => super::queries::episode_ref_in(&mut tx, user_id, &session.episode_id) + .await? + .map(|reference| op_ref(&reference)), + }; + let op = emit(&mut tx, user_id, &device_id, |_| { + SyncOpPayload::PodcastListenRecorded { + session: session.clone(), + episode: reference.clone(), + } + }) + .await?; + tx.commit().await?; + Ok(Some(op)) + } + + /// Makes sure the catalog knows an episode a queue item refers to. + /// + /// Queue items received through a handoff carry a self-contained + /// `PodcastEpisodeRef`; the account may not have refreshed that feed yet. + /// The reference is recorded like one learned through sync + /// (`source = 'sync'`, never replacing feed metadata). Returns whether a + /// row was added. No register is written. A new row is refused with + /// [`PodcastStoreError::ReferenceLimit`] once the account has + /// [`super::MAX_REFERENCE_EPISODES_PER_ACCOUNT`] such episodes. + pub(crate) async fn ensure_episode( + &self, + user_id: i64, + episode: &PodcastEpisodeRef, + ) -> Result { + episode.validate().map_err(invalid)?; + let mut tx = self.begin_write(user_id).await?; + let known = super::queries::episode_ref_in(&mut tx, user_id, &episode.episode_id) + .await? + .is_some(); + if !known { + // Counted under the account's write lock: concurrent requests + // cannot pass the limit together. + let references: i64 = sqlx::query_scalar( + "SELECT COUNT(*) FROM furumusic__podcast_episode + WHERE user_id = $1 AND source = 'sync'", + ) + .bind(user_id) + .fetch_one(&mut *tx) + .await?; + if usize::try_from(references).unwrap_or(usize::MAX) + >= super::MAX_REFERENCE_EPISODES_PER_ACCOUNT + { + return Err(PodcastStoreError::ReferenceLimit( + super::MAX_REFERENCE_EPISODES_PER_ACCOUNT, + ) + .into()); + } + apply::ensure_episode_from_ref(&mut tx, user_id, episode).await?; + } + tx.commit().await?; + Ok(!known) + } + + /// Resolves a feed URL to a podcast id: the existing podcast when the + /// URL is a known feed URL or alias (`resolve_podcast_id`), otherwise + /// the id a first subscription through this URL would derive. + #[cfg(test)] + pub(crate) async fn resolve_podcast_id( + &self, + user_id: i64, + url: &FeedUrl, + ) -> Result { + let mut conn = self.pool.acquire().await?; + resolve_podcast(&mut conn, user_id, url).await + } + + /// The episode reference for queueing or progress actions (with + /// `progress_generation` set to the stored register's generation), or + /// `None` when the episode is unknown. + pub(crate) async fn episode_ref( + &self, + user_id: i64, + episode_id: &EpisodeId, + ) -> Result> { + let mut conn = self.pool.acquire().await?; + super::queries::episode_ref_in(&mut conn, user_id, episode_id).await + } +} diff --git a/src/podcasts/api.rs b/src/podcasts/api.rs new file mode 100644 index 0000000..1251599 --- /dev/null +++ b/src/podcasts/api.rs @@ -0,0 +1,1038 @@ +//! Request handling of the podcast UI and the web player. +//! +//! [`super::http`] authenticates the request and passes the account here. +//! Every function takes the store, the account and the decoded input and +//! returns a JSON view or an [`ApiError`] with its HTTP status, so the +//! behavior of the endpoints is testable without an HTTP server +//! (`postgres_tests`). +//! +//! Playback follows the progress rules of the frid podcast contract +//! (`PODCASTS.md`, "Progress" and "Listening sessions and counts"): +//! +//! * [`playback`] prepares an episode for the player: the audio source, the +//! position ordinary Play resumes from and, when the browser starts +//! playing (`start`), a new listening session written as an explicit +//! action (`explicit_generation(max(local, carried))`). +//! * [`progress`] stores what the player reports for that session: periodic +//! checkpoints and pause/stop/handoff saves keep the session generation, a +//! user seek is an explicit action, the end of the stream or +//! `reaches_completion` completes the episode once, and the listening +//! session is recorded (history counts only completed sessions). + +use music_dht::device_sync::PlaybackTrack; +use music_dht::podcast::{ + EpisodeId, EpisodeType, FeedUrl, ListenSession, PodcastEpisodeRef, PodcastId, PodcastMeta, + ProgressRecord, ProgressStatus, reaches_completion, +}; +use serde::{Deserialize, Serialize}; +use serde_json::Value; + +use super::fetch::{FeedFetcher, FetchErrorKind}; +use super::runner::{self, SubscribeUrlError}; +use super::{ + CheckpointOutcome, EpisodeListItem, EpisodeQuery, EpisodeSort, FetchState, PlayedFilter, + PodcastListFilter, PodcastStore, PodcastStoreError, PodcastSummary, ProgressSession, + SeasonFilter, SeasonSummary, +}; +use crate::federation::devices; +use crate::player::queue_items; + +/// Episodes per page when the request does not say. +pub(crate) const DEFAULT_EPISODE_PAGE: usize = 50; +/// Largest episode page. +pub(crate) const MAX_EPISODE_PAGE: usize = 200; + +/// A failed request: HTTP status and a message for the user. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct ApiError { + pub(crate) status: u16, + pub(crate) message: String, +} + +impl ApiError { + pub(crate) fn new(status: u16, message: impl Into) -> Self { + Self { + status, + message: message.into(), + } + } + + fn bad_request(message: impl Into) -> Self { + Self::new(400, message) + } + + fn not_found(message: impl Into) -> Self { + Self::new(404, message) + } +} + +impl From for ApiError { + fn from(error: anyhow::Error) -> Self { + if let Some(error) = error.downcast_ref::() { + return match error { + PodcastStoreError::UnknownPodcast(_) => Self::not_found(error.to_string()), + PodcastStoreError::NotSubscribed(_) => Self::new(409, error.to_string()), + PodcastStoreError::Invalid(_) => Self::bad_request(error.to_string()), + PodcastStoreError::SubscriptionLimit(_) | PodcastStoreError::ReferenceLimit(_) => { + Self::new(422, error.to_string()) + } + }; + } + if let Some(SubscribeUrlError::Fetch(fetch)) = error.downcast_ref::() { + // The feed could not be used: the request cannot succeed as is. + let status = match fetch.kind { + FetchErrorKind::Timeout | FetchErrorKind::Transport => 502, + _ => 422, + }; + // The user sees the kind of failure only (no resolved + // addresses or transport details); the log keeps the details. + tracing::info!("podcast feed could not be added: {}", fetch.message); + return Self::new(status, error.to_string()); + } + tracing::warn!("podcast API request failed: {error:#}"); + Self::new(500, "internal error") + } +} + +pub(crate) type ApiResult = Result; + +pub(crate) fn parse_podcast_id(value: &str) -> ApiResult { + PodcastId::parse(value).map_err(|_| ApiError::bad_request("invalid podcast id")) +} + +pub(crate) fn parse_episode_id(value: &str) -> ApiResult { + EpisodeId::parse(value).map_err(|_| ApiError::bad_request("invalid episode id")) +} + +fn decode(body: Value) -> ApiResult { + serde_json::from_value(body).map_err(|error| ApiError::bad_request(error.to_string())) +} + +fn status_name(status: ProgressStatus) -> &'static str { + match status { + ProgressStatus::NotStarted => "not_started", + ProgressStatus::InProgress => "in_progress", + ProgressStatus::Completed => "completed", + } +} + +fn episode_type_name(kind: EpisodeType) -> &'static str { + match kind { + EpisodeType::Full => "full", + EpisodeType::Trailer => "trailer", + EpisodeType::Bonus => "bonus", + } +} + +// --------------------------------------------------------------------------- +// Views +// --------------------------------------------------------------------------- + +/// Local fetch state shown next to a podcast (never synced). +#[derive(Debug, Clone, Serialize)] +pub(crate) struct FetchView { + pub(crate) checked_at_ms: Option, + pub(crate) last_success_at_ms: Option, + pub(crate) next_check_at_ms: Option, + pub(crate) last_error: Option, + pub(crate) consecutive_failures: u32, +} + +impl From for FetchView { + fn from(state: FetchState) -> Self { + Self { + checked_at_ms: state.checked_at_ms, + last_success_at_ms: state.last_success_at_ms, + next_check_at_ms: state.next_check_at_ms, + last_error: state.last_error, + consecutive_failures: state.consecutive_failures, + } + } +} + +/// A podcast with its personal state and counters. +#[derive(Debug, Clone, Serialize)] +pub(crate) struct PodcastView { + pub(crate) podcast_id: String, + /// URL this account fetches. + pub(crate) feed_url: String, + pub(crate) title: String, + pub(crate) author: Option, + pub(crate) description: Option, + pub(crate) link: Option, + pub(crate) image_url: Option, + pub(crate) subscribed: bool, + /// `effective_share` (publication is MR-W3; shown read-only). + pub(crate) shared: bool, + pub(crate) episode_count: u64, + pub(crate) unplayed_count: u64, + pub(crate) in_progress_count: u64, + pub(crate) season_count: u64, + pub(crate) latest_published_at_ms: Option, + pub(crate) fetch: FetchView, +} + +impl PodcastView { + fn new(summary: PodcastSummary, fetch: FetchState) -> Self { + Self { + podcast_id: summary.podcast_id.to_string(), + feed_url: summary.meta.feed_url.to_string(), + title: summary.meta.title, + author: summary.meta.author, + description: summary.meta.description, + link: summary.meta.link, + image_url: summary.meta.image_url, + subscribed: summary.subscribed, + shared: summary.shared, + episode_count: summary.episode_count, + unplayed_count: summary.unplayed_count, + in_progress_count: summary.in_progress_count, + season_count: summary.season_count, + latest_published_at_ms: summary.latest_published_at_ms, + fetch: fetch.into(), + } + } +} + +/// One season group (`season: null` holds the episodes without a season). +#[derive(Debug, Clone, Serialize)] +pub(crate) struct SeasonView { + pub(crate) season: Option, + pub(crate) episode_count: u64, + pub(crate) unplayed_count: u64, + pub(crate) first_published_at_ms: Option, + pub(crate) last_published_at_ms: Option, +} + +impl From for SeasonView { + fn from(season: SeasonSummary) -> Self { + Self { + season: season.season, + episode_count: season.episode_count, + unplayed_count: season.unplayed_count, + first_published_at_ms: season.first_published_at_ms, + last_published_at_ms: season.last_published_at_ms, + } + } +} + +/// An episode with its progress and the queue item the player uses. +#[derive(Debug, Clone, Serialize)] +pub(crate) struct EpisodeView { + pub(crate) episode_id: String, + pub(crate) podcast_id: String, + pub(crate) title: String, + pub(crate) description: Option, + pub(crate) season: Option, + pub(crate) episode_number: Option, + pub(crate) episode_type: &'static str, + pub(crate) published_at_ms: Option, + pub(crate) duration_ms: Option, + pub(crate) image_url: Option, + pub(crate) link: Option, + /// Whether the latest refresh listed the episode (older episodes stay). + pub(crate) in_feed: bool, + /// `not_started`, `in_progress` or `completed`. + pub(crate) status: &'static str, + /// Stored position of the progress register. + pub(crate) position_ms: i64, + /// Where ordinary Play resumes (`ProgressRecord::resume_position_ms`). + pub(crate) resume_position_ms: i64, + /// Distinct completed listening sessions. + pub(crate) completed_count: u64, + /// Browser queue item: `PlaybackSource::PodcastEpisode` with the + /// self-contained reference (see `player::queue_items`). + pub(crate) track: Value, +} + +/// The queue reference of a catalog episode, with the register generation +/// as handoff hint. +fn episode_ref(item: &EpisodeListItem, podcast: &PodcastMeta) -> PodcastEpisodeRef { + let mut reference = item.episode.to_ref(podcast); + reference.progress_generation = item.progress.as_ref().map(|record| record.generation); + reference +} + +/// Browser queue item of an episode reference. +pub(crate) fn episode_track_json(reference: PodcastEpisodeRef) -> Value { + queue_items::external_track_json(&PlaybackTrack::podcast_episode(reference), 0, None) +} + +impl EpisodeView { + fn new(item: EpisodeListItem, podcast: &PodcastMeta) -> Self { + let track = episode_track_json(episode_ref(&item, podcast)); + let status = status_name(item.status()); + let resume_position_ms = item.resume_position_ms(); + let position_ms = item + .progress + .as_ref() + .map_or(0, |progress| progress.position_ms); + let episode = item.episode; + Self { + episode_id: episode.episode_id.to_string(), + podcast_id: episode.podcast_id.to_string(), + title: episode.title, + description: episode.description, + season: episode.season, + episode_number: episode.episode_number, + episode_type: episode_type_name(episode.episode_type), + published_at_ms: episode.published_at_ms, + duration_ms: episode.duration_ms, + image_url: episode.image_url, + link: episode.link, + in_feed: item.in_feed, + status, + position_ms, + resume_position_ms, + completed_count: item.completed_count, + track, + } + } +} + +async fn podcast_view( + store: &PodcastStore, + user_id: i64, + podcast_id: &PodcastId, +) -> ApiResult { + let summary = store + .podcast(user_id, podcast_id) + .await? + .ok_or_else(|| ApiError::not_found("unknown podcast"))?; + let fetch = store.fetch_state(user_id, podcast_id).await?; + Ok(PodcastView::new(summary, fetch)) +} + +async fn episode_view( + store: &PodcastStore, + user_id: i64, + episode_id: &EpisodeId, +) -> ApiResult { + let item = store + .episode(user_id, episode_id) + .await? + .ok_or_else(|| ApiError::not_found("unknown episode"))?; + let podcast = store + .podcast(user_id, &item.episode.podcast_id) + .await? + .ok_or_else(|| ApiError::not_found("unknown podcast"))?; + Ok(EpisodeView::new(item, &podcast.meta)) +} + +async fn catalog_ref( + store: &PodcastStore, + user_id: i64, + episode_id: &EpisodeId, +) -> ApiResult { + store + .episode_ref(user_id, episode_id) + .await? + .ok_or_else(|| ApiError::not_found("unknown episode")) +} + +// --------------------------------------------------------------------------- +// Podcasts +// --------------------------------------------------------------------------- + +#[derive(Debug, Serialize)] +pub(crate) struct PodcastList { + pub(crate) podcasts: Vec, +} + +/// Podcasts of the account: subscribed ones, or with `all` also +/// unsubscribed podcasts and podcasts only known from synced progress. +pub(crate) async fn list_podcasts( + store: &PodcastStore, + user_id: i64, + all: bool, +) -> ApiResult { + let filter = if all { + PodcastListFilter::All + } else { + PodcastListFilter::Subscribed + }; + let mut podcasts = Vec::new(); + for summary in store.list_podcasts(user_id, filter).await? { + let fetch = store.fetch_state(user_id, &summary.podcast_id).await?; + podcasts.push(PodcastView::new(summary, fetch)); + } + Ok(PodcastList { podcasts }) +} + +/// Listening sessions shown in a podcast's history. +pub(crate) const DETAIL_HISTORY_LIMIT: usize = 20; + +/// A listening session in the history (completed or at least 30 s). +#[derive(Debug, Clone, Serialize)] +pub(crate) struct HistoryView { + pub(crate) episode_id: String, + pub(crate) episode_title: Option, + pub(crate) started_at_ms: i64, + pub(crate) ended_at_ms: i64, + pub(crate) listened_ms: i64, + pub(crate) completed: bool, +} + +#[derive(Debug, Serialize)] +pub(crate) struct PodcastDetail { + pub(crate) podcast: PodcastView, + pub(crate) seasons: Vec, + /// Newest listening sessions of this podcast, from every device. + pub(crate) history: Vec, +} + +/// One podcast with its season groups. +pub(crate) async fn podcast_detail( + store: &PodcastStore, + user_id: i64, + podcast_id: &PodcastId, +) -> ApiResult { + let podcast = podcast_view(store, user_id, podcast_id).await?; + let seasons = store + .seasons(user_id, podcast_id) + .await? + .into_iter() + .map(SeasonView::from) + .collect(); + let history = store + .listen_history(user_id, Some(podcast_id), DETAIL_HISTORY_LIMIT) + .await? + .into_iter() + .map(|entry| HistoryView { + episode_id: entry.session.episode_id.to_string(), + episode_title: entry.episode_title, + started_at_ms: entry.session.started_at_ms, + ended_at_ms: entry.session.ended_at_ms, + listened_ms: entry.session.listened_ms, + completed: entry.session.completed, + }) + .collect(); + Ok(PodcastDetail { + podcast, + seasons, + history, + }) +} + +#[derive(Debug, Deserialize)] +struct SubscribeRequest { + url: String, +} + +#[derive(Debug, Serialize)] +pub(crate) struct SubscribeResponse { + pub(crate) podcast: PodcastView, + pub(crate) already_subscribed: bool, + pub(crate) new_episodes: usize, +} + +/// "Add RSS": `{ "url": "..." }`. The feed is fetched before anything is +/// stored; the new subscription is private. +pub(crate) async fn subscribe_url( + store: &PodcastStore, + fetcher: &FeedFetcher, + user_id: i64, + body: Value, + now_ms: i64, +) -> ApiResult { + let request: SubscribeRequest = decode(body)?; + let url = FeedUrl::from_user_input(&request.url) + .map_err(|error| ApiError::bad_request(format!("invalid feed URL: {error}")))?; + let subscription = runner::subscribe_url(store, fetcher, user_id, &url, now_ms).await?; + Ok(SubscribeResponse { + podcast: podcast_view(store, user_id, &subscription.podcast_id).await?, + already_subscribed: subscription.already_subscribed, + new_episodes: subscription.refresh.new_episodes, + }) +} + +/// Subscribes again to a known podcast with its stored channel metadata +/// (progress and history continue). The refresh job fetches it on its next +/// run. +pub(crate) async fn resubscribe( + store: &PodcastStore, + user_id: i64, + podcast_id: &PodcastId, +) -> ApiResult { + let summary = store + .podcast(user_id, podcast_id) + .await? + .ok_or_else(|| ApiError::not_found("unknown podcast"))?; + store.subscribe(user_id, &summary.meta).await?; + podcast_view(store, user_id, podcast_id).await +} + +/// Unsubscribes: stops monitoring, revokes a publication, keeps progress +/// and history. +pub(crate) async fn unsubscribe( + store: &PodcastStore, + user_id: i64, + podcast_id: &PodcastId, +) -> ApiResult { + if store.podcast(user_id, podcast_id).await?.is_none() { + return Err(ApiError::not_found("unknown podcast")); + } + store.unsubscribe(user_id, podcast_id).await?; + podcast_view(store, user_id, podcast_id).await +} + +// --------------------------------------------------------------------------- +// Episodes +// --------------------------------------------------------------------------- + +/// Query string of the episode list. +#[derive(Debug, Clone, Default, Deserialize)] +pub(crate) struct EpisodeListParams { + /// Season number, or `none` for the episodes without a season. + #[serde(default)] + pub(crate) season: Option, + /// `unplayed`, `not_started`, `in_progress` or `played`. + #[serde(default)] + pub(crate) played: Option, + /// Search text (titles and descriptions). + #[serde(default)] + pub(crate) q: Option, + /// `newest` (default), `oldest`, `number` or `number_desc`. + #[serde(default)] + pub(crate) sort: Option, + #[serde(default)] + pub(crate) limit: Option, + #[serde(default)] + pub(crate) offset: Option, +} + +impl EpisodeListParams { + /// The store query (one row more than the page, to tell whether more + /// rows follow). + pub(crate) fn query(&self, podcast_id: PodcastId) -> ApiResult<(EpisodeQuery, usize)> { + let blank = |value: &Option| { + value + .as_deref() + .map(str::trim) + .filter(|value| !value.is_empty()) + .map(str::to_string) + }; + let season = match blank(&self.season).as_deref() { + None | Some("all") => SeasonFilter::Any, + Some("none") => SeasonFilter::NoSeason, + Some(value) => SeasonFilter::Season( + value + .parse() + .map_err(|_| ApiError::bad_request("invalid season"))?, + ), + }; + let played = match blank(&self.played).as_deref() { + None | Some("all") => PlayedFilter::Any, + Some("unplayed") => PlayedFilter::Unplayed, + Some("not_started") => PlayedFilter::NotStarted, + Some("in_progress") => PlayedFilter::InProgress, + Some("played") => PlayedFilter::Played, + Some(_) => return Err(ApiError::bad_request("invalid played filter")), + }; + let sort = match blank(&self.sort).as_deref() { + None | Some("newest") => EpisodeSort::NewestFirst, + Some("oldest") => EpisodeSort::OldestFirst, + Some("number") => EpisodeSort::EpisodeNumber, + Some("number_desc") => EpisodeSort::EpisodeNumberDesc, + Some(_) => return Err(ApiError::bad_request("invalid sort")), + }; + let limit = self + .limit + .unwrap_or(DEFAULT_EPISODE_PAGE) + .clamp(1, MAX_EPISODE_PAGE); + let mut query = EpisodeQuery::new(podcast_id); + query.season = season; + query.played = played; + query.search = blank(&self.q); + query.sort = sort; + query.limit = Some(limit + 1); + query.offset = self.offset.unwrap_or(0); + Ok((query, limit)) + } +} + +#[derive(Debug, Serialize)] +pub(crate) struct EpisodePage { + pub(crate) episodes: Vec, + pub(crate) offset: usize, + pub(crate) has_more: bool, +} + +/// A page of episodes of one podcast. +pub(crate) async fn episodes( + store: &PodcastStore, + user_id: i64, + podcast_id: &PodcastId, + params: &EpisodeListParams, +) -> ApiResult { + let podcast = store + .podcast(user_id, podcast_id) + .await? + .ok_or_else(|| ApiError::not_found("unknown podcast"))?; + let (query, limit) = params.query(podcast_id.clone())?; + let mut rows = store.episodes(user_id, &query).await?; + let has_more = rows.len() > limit; + rows.truncate(limit); + Ok(EpisodePage { + episodes: rows + .into_iter() + .map(|item| EpisodeView::new(item, &podcast.meta)) + .collect(), + offset: query.offset, + has_more, + }) +} + +/// Explicit progress actions of the episode list. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum EpisodeAction { + /// "Start over": position 0, in progress. + Restart, + /// "Played": completed, no listening session, no invented time. + MarkPlayed, + /// "Unplayed": not started at 0. + MarkUnplayed, +} + +/// Runs an explicit progress action and returns the updated episode. +pub(crate) async fn episode_action( + store: &PodcastStore, + user_id: i64, + episode_id: &EpisodeId, + action: EpisodeAction, +) -> ApiResult { + let reference = catalog_ref(store, user_id, episode_id).await?; + match action { + EpisodeAction::Restart => { + let session_id = format!("web-{}", uuid::Uuid::new_v4().simple()); + store.restart(user_id, &reference, &session_id).await?; + } + EpisodeAction::MarkPlayed => { + store.mark_played(user_id, &reference).await?; + } + EpisodeAction::MarkUnplayed => { + store.mark_unplayed(user_id, &reference).await?; + } + } + episode_view(store, user_id, episode_id).await +} + +// --------------------------------------------------------------------------- +// Playback +// --------------------------------------------------------------------------- + +/// Body of `POST /episodes/{id}/playback`. +#[derive(Debug, Clone, Default, Deserialize)] +pub(crate) struct PlaybackRequest { + /// The queue item's reference. Used only when the catalog does not know + /// the episode yet (an item received through a handoff); it is recorded + /// like a reference learned through sync. + #[serde(default)] + pub(crate) episode: Option, + /// Start a listening session now (the browser is about to play). + #[serde(default)] + pub(crate) start: bool, + /// Session id generated by the browser (required with `start`). + #[serde(default)] + pub(crate) session_id: Option, + /// Position to start at: a handoff or the user's choice. Without it the + /// episode resumes from its progress register. + #[serde(default)] + pub(crate) position_ms: Option, + /// `PodcastEpisodeRef::progress_generation` of a handed-off queue item. + #[serde(default)] + pub(crate) carried_generation: Option, +} + +/// How the browser plays an episode. +#[derive(Debug, Clone, Serialize)] +pub(crate) struct PlaybackResponse { + pub(crate) episode_id: String, + pub(crate) podcast_id: String, + /// Session started by this request (`start`), with the generation its + /// checkpoints use. + pub(crate) session_id: Option, + pub(crate) generation: Option, + /// Where to start. + pub(crate) position_ms: i64, + /// Register status before this request. + pub(crate) status: &'static str, + pub(crate) duration_ms: Option, + /// Publisher URL (default: the browser streams it directly). + pub(crate) direct_url: String, + /// Same-origin proxy path when the administrator enabled the proxy. + pub(crate) proxy_url: Option, + pub(crate) mime: Option, + /// The progress register after this request: its generation and the + /// session (or action) that wrote it. Before continuing a paused + /// session the player checks that the register is still its own. + pub(crate) register_generation: Option, + pub(crate) register_session_id: Option, +} + +/// Prepares an episode for the player and, with `start`, begins a session. +pub(crate) async fn playback( + store: &PodcastStore, + user_id: i64, + episode_id: &EpisodeId, + body: Value, + proxy_enabled: bool, +) -> ApiResult { + let request: PlaybackRequest = decode(body)?; + if store.episode_ref(user_id, episode_id).await?.is_none() { + match request.episode.as_ref() { + Some(reference) if reference.episode_id == *episode_id => { + store.ensure_episode(user_id, reference).await?; + } + Some(_) => return Err(ApiError::bad_request("episode reference mismatch")), + None => return Err(ApiError::not_found("unknown episode")), + } + } + let reference = catalog_ref(store, user_id, episode_id).await?; + let current = store.progress(user_id, episode_id).await?; + let status = current + .as_ref() + .map_or(ProgressStatus::NotStarted, |record| record.status); + let resume = current + .as_ref() + .map_or(0, ProgressRecord::resume_position_ms); + let mut position_ms = request.position_ms.unwrap_or(resume).max(0); + let (mut session_id, mut generation) = (None, None); + let mut register = current; + if request.start { + let id = request + .session_id + .as_deref() + .map(str::trim) + .filter(|id| !id.is_empty()) + .ok_or_else(|| ApiError::bad_request("session_id is required to start"))?; + let write = store + .start_session( + user_id, + &reference, + id, + position_ms, + request.carried_generation, + ) + .await?; + position_ms = write.record.position_ms; + session_id = Some(write.session.session_id); + generation = Some(write.session.generation); + register = Some(write.record); + } + let (direct_url, mime) = store + .audio_source(user_id, episode_id) + .await? + .ok_or_else(|| ApiError::not_found("unknown episode"))?; + Ok(PlaybackResponse { + episode_id: episode_id.to_string(), + podcast_id: reference.podcast_id.to_string(), + session_id, + generation, + position_ms, + status: status_name(status), + duration_ms: reference.duration_ms, + direct_url, + proxy_url: proxy_enabled.then(|| super::http::proxy_path(episode_id)), + mime, + register_generation: register.as_ref().map(|record| record.generation), + register_session_id: register.map(|record| record.session_id), + }) +} + +/// What the player reports in `POST /episodes/{id}/progress`. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Deserialize)] +#[serde(rename_all = "snake_case")] +pub(crate) enum ProgressEvent { + /// Periodic save while playing (every 15–30 s). + Checkpoint, + /// The user paused. + Pause, + /// Playback stopped: another item, the page closed, the queue cleared. + Stop, + /// Playback moves to another device. + Handoff, + /// The user moved the position (explicit action). + Seek, + /// The audio reached the end of the stream. + Ended, +} + +impl ProgressEvent { + /// Whether the event ends a stretch of listening that belongs in the + /// session record. + fn records_session(self) -> bool { + matches!(self, Self::Pause | Self::Stop | Self::Handoff | Self::Ended) + } +} + +/// Body of `POST /episodes/{id}/progress`. +#[derive(Debug, Clone, Deserialize)] +pub(crate) struct ProgressRequest { + pub(crate) session_id: String, + /// Generation of the session's latest explicit action. + pub(crate) generation: u64, + pub(crate) event: ProgressEvent, + pub(crate) position_ms: i64, + /// Duration of the loaded audio, when the browser knows it. + #[serde(default)] + pub(crate) duration_ms: Option, + /// Time actually listened in this session (seeks excluded). + #[serde(default)] + pub(crate) listened_ms: Option, + /// When the session started (browser clock, unix ms). + #[serde(default)] + pub(crate) started_at_ms: Option, + #[serde(default)] + pub(crate) start_position_ms: Option, +} + +/// Result of a progress report. +#[derive(Debug, Clone, Serialize)] +pub(crate) struct ProgressResponse { + /// `written`, `superseded` (another device or action took over: stop + /// writing at this generation) or `already_completed`. + pub(crate) outcome: &'static str, + /// Generation to use for the next checkpoints of this session. + pub(crate) generation: u64, + /// Register after the report. + pub(crate) status: &'static str, + pub(crate) position_ms: i64, + /// Whether this report completed the episode. + pub(crate) completed: bool, + /// Whether the listening session was recorded (or grew). + pub(crate) session_recorded: bool, +} + +/// Stores a progress report of the player's current session. +pub(crate) async fn progress( + store: &PodcastStore, + user_id: i64, + episode_id: &EpisodeId, + body: Value, + now_ms: i64, +) -> ApiResult { + let request: ProgressRequest = decode(body)?; + let reference = catalog_ref(store, user_id, episode_id).await?; + let position_ms = request.position_ms.max(0); + let duration_ms = request + .duration_ms + .filter(|duration| *duration > 0) + .or(reference.duration_ms); + let session = ProgressSession { + episode: reference.clone(), + session_id: request.session_id.trim().to_string(), + generation: request.generation, + }; + let reached_end = + request.event == ProgressEvent::Ended || reaches_completion(position_ms, duration_ms); + let outcome = match request.event { + ProgressEvent::Seek => { + CheckpointOutcome::Written(Box::new(store.seek(user_id, &session, position_ms).await?)) + } + _ if reached_end => { + store + .complete_if_current(user_id, &session, position_ms) + .await? + } + _ => store.checkpoint(user_id, &session, position_ms).await?, + }; + let (name, record, completed) = match &outcome { + CheckpointOutcome::Written(write) => ( + "written", + write.record.clone(), + write.record.status == ProgressStatus::Completed && reached_end, + ), + CheckpointOutcome::Superseded { current } => ("superseded", current.clone(), false), + CheckpointOutcome::AlreadyCompleted { current } => { + ("already_completed", current.clone(), false) + } + }; + let mut session_recorded = false; + if request.event.records_session() + && let Some(started_at_ms) = request.started_at_ms + { + let started_at_ms = started_at_ms.clamp(0, now_ms); + let listened_ms = request + .listened_ms + .unwrap_or(0) + .clamp(0, now_ms - started_at_ms); + let listen = ListenSession { + session_id: session.session_id.clone(), + episode_id: episode_id.clone(), + podcast_id: reference.podcast_id.clone(), + device_id: devices::local_device_id(store.pool(), user_id).await?, + started_at_ms, + ended_at_ms: now_ms, + start_position_ms: request.start_position_ms.unwrap_or(0).max(0), + end_position_ms: position_ms, + listened_ms, + completed: reached_end, + }; + session_recorded = store + .record_listen_session(user_id, &listen, Some(&reference)) + .await? + .is_some(); + } + Ok(ProgressResponse { + outcome: name, + generation: record.generation, + status: status_name(record.status), + position_ms: record.position_ms, + completed, + session_recorded, + }) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn podcast() -> PodcastId { + PodcastId::from_feed_url(&FeedUrl::parse("https://podcasts.example/feed.xml").unwrap()) + } + + #[test] + fn episode_list_params_map_to_store_queries() { + let (query, limit) = EpisodeListParams::default().query(podcast()).unwrap(); + assert_eq!(limit, DEFAULT_EPISODE_PAGE); + assert_eq!(query.limit, Some(DEFAULT_EPISODE_PAGE + 1)); + assert_eq!(query.season, SeasonFilter::Any); + assert_eq!(query.played, PlayedFilter::Any); + assert_eq!(query.sort, EpisodeSort::NewestFirst); + assert_eq!(query.search, None); + + let params = EpisodeListParams { + season: Some("3".into()), + played: Some("in_progress".into()), + q: Some(" interview ".into()), + sort: Some("number_desc".into()), + limit: Some(10_000), + offset: Some(40), + }; + let (query, limit) = params.query(podcast()).unwrap(); + assert_eq!(query.season, SeasonFilter::Season(3)); + assert_eq!(query.played, PlayedFilter::InProgress); + assert_eq!(query.search.as_deref(), Some("interview")); + assert_eq!(query.sort, EpisodeSort::EpisodeNumberDesc); + assert_eq!(limit, MAX_EPISODE_PAGE); + assert_eq!(query.offset, 40); + + let none = EpisodeListParams { + season: Some("none".into()), + played: Some("played".into()), + q: Some(" ".into()), + sort: Some("oldest".into()), + ..Default::default() + }; + let (query, _) = none.query(podcast()).unwrap(); + assert_eq!(query.season, SeasonFilter::NoSeason); + assert_eq!(query.played, PlayedFilter::Played); + assert_eq!(query.search, None, "blank search is no filter"); + assert_eq!(query.sort, EpisodeSort::OldestFirst); + + for bad in [ + EpisodeListParams { + season: Some("first".into()), + ..Default::default() + }, + EpisodeListParams { + played: Some("sometimes".into()), + ..Default::default() + }, + EpisodeListParams { + sort: Some("random".into()), + ..Default::default() + }, + ] { + assert_eq!(bad.query(podcast()).unwrap_err().status, 400); + } + } + + #[test] + fn ids_and_bodies_are_validated_before_the_store() { + assert_eq!(parse_podcast_id("nope").unwrap_err().status, 400); + assert_eq!(parse_episode_id("pod1:abc").unwrap_err().status, 400); + assert!(parse_podcast_id(podcast().as_str()).is_ok()); + let error = decode::(serde_json::json!({ + "session_id": "s", "generation": 1, "event": "rewind", "position_ms": 0 + })) + .unwrap_err(); + assert_eq!(error.status, 400); + let request: ProgressRequest = decode(serde_json::json!({ + "session_id": "s", "generation": 3, "event": "handoff", "position_ms": 1200 + })) + .unwrap(); + assert_eq!(request.event, ProgressEvent::Handoff); + assert!(request.event.records_session()); + assert!(!ProgressEvent::Checkpoint.records_session()); + assert!(!ProgressEvent::Seek.records_session()); + // A malformed episode reference in a playback request is a 400. + let error = decode::(serde_json::json!({ + "episode": { "episode_id": "not an id" } + })) + .unwrap_err(); + assert_eq!(error.status, 400); + let empty: PlaybackRequest = decode(serde_json::json!({})).unwrap(); + assert!(!empty.start && empty.episode.is_none() && empty.position_ms.is_none()); + } + + #[test] + fn store_errors_map_to_http_statuses() { + let unknown: ApiError = + anyhow::Error::from(PodcastStoreError::UnknownPodcast(podcast())).into(); + assert_eq!(unknown.status, 404); + let not_subscribed: ApiError = + anyhow::Error::from(PodcastStoreError::NotSubscribed(podcast())).into(); + assert_eq!(not_subscribed.status, 409); + let invalid: ApiError = super::super::invalid("bad").into(); + assert_eq!(invalid.status, 400); + let limit: ApiError = + anyhow::Error::from(PodcastStoreError::SubscriptionLimit(1000)).into(); + assert_eq!(limit.status, 422); + assert_eq!(limit.message, "subscription limit reached (1000 podcasts)"); + let references: ApiError = + anyhow::Error::from(PodcastStoreError::ReferenceLimit(10_000)).into(); + assert_eq!(references.status, 422); + let refused: ApiError = + anyhow::Error::from(SubscribeUrlError::Fetch(super::super::fetch::FetchError { + kind: FetchErrorKind::Refused, + message: "10.1.2.3 is a private address".into(), + retry_after: None, + })) + .into(); + assert_eq!(refused.status, 422); + assert!( + refused.message.contains("only public"), + "{}", + refused.message + ); + assert!( + !refused.message.contains("10.1.2.3"), + "no resolved addresses" + ); + let timeout: ApiError = + anyhow::Error::from(SubscribeUrlError::Fetch(super::super::fetch::FetchError { + kind: FetchErrorKind::Timeout, + message: "timed out".into(), + retry_after: None, + })) + .into(); + assert_eq!(timeout.status, 502); + let transport: ApiError = + anyhow::Error::from(SubscribeUrlError::Fetch(super::super::fetch::FetchError { + kind: FetchErrorKind::Transport, + message: "error sending request: tcp connect error: Connection refused".into(), + retry_after: None, + })) + .into(); + assert_eq!(transport.status, 502); + assert_eq!( + transport.message, + "the feed could not be loaded: could not connect to the server" + ); + let other: ApiError = anyhow::anyhow!("connection reset").into(); + assert_eq!(other.status, 500); + assert_eq!(other.message, "internal error", "internals are not shown"); + } +} diff --git a/src/podcasts/apply.rs b/src/podcasts/apply.rs new file mode 100644 index 0000000..edcd294 --- /dev/null +++ b/src/podcasts/apply.rs @@ -0,0 +1,876 @@ +//! The single code path that applies podcast records to the store. +//! +//! Remote operations, local actions and snapshot sections all end here. +//! Records come from the `SyncOpWire` materializers of +//! `music_dht::device_sync` and are merged with the functions of +//! `music_dht::podcast`, so applying is idempotent and order-independent. + +use anyhow::Result; +use music_dht::device_sync::{SyncOpPayload, SyncOpWire, SyncSnapshot}; +use music_dht::podcast::{ + EpisodeId, EpisodeIdentityHints, EpisodeMeta, EpisodeType, FeedUrl, KnownEpisode, + ListenSession, PodcastEpisodeRef, PodcastError, PodcastId, ProgressRecord, ShareRecord, + SubscriptionRecord, merge_feed_aliases, merge_listen_session, +}; +use sqlx::{PgConnection, Row}; + +use super::{ + PodcastStore, PodcastStoreError, SESSION_COLUMNS, episode_id_col, episode_search_norm, + feed_url_col, generation_from_sql, generation_to_sql, invalid, now_ms, podcast_meta_col, + progress_columns, progress_from_row, session_from_row, status_to_sql, +}; + +/// Result of applying one operation. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) enum ApplyOutcome { + /// Stored state changed. + Changed, + /// The operation was already reflected (duplicate, or older than the + /// stored register). + Unchanged, + /// Not a podcast operation; nothing to do here. + Ignored, + /// The operation violates the contract and was skipped. Acknowledge it + /// anyway so the log keeps moving (`PODCASTS.md`). + Rejected(String), +} + +/// Counters returned by [`PodcastStore::apply_snapshot`]. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub(crate) struct SnapshotApplyStats { + /// Records that changed stored state. + pub(crate) changed: usize, + /// Valid records already reflected locally. + pub(crate) unchanged: usize, + /// Records that failed validation and were skipped. + pub(crate) rejected: usize, + /// Whether the podcast sections were ignored for exceeding the counts. + pub(crate) ignored_over_limits: bool, +} + +/// Where an episode row came from. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum EpisodeSource { + /// Parsed from the podcast's feed by this account. + Feed, + /// Learned from a `PodcastEpisodeRef` (sync operation or queue item). + Sync, +} + +impl EpisodeSource { + fn as_sql(self) -> &'static str { + match self { + Self::Feed => "feed", + Self::Sync => "sync", + } + } +} + +pub(crate) async fn load_subscription_register( + conn: &mut PgConnection, + user_id: i64, + podcast_id: &PodcastId, +) -> Result> { + let row = sqlx::query( + "SELECT subscribed, podcast_json, generation, hlc_ms, op_id + FROM furumusic__podcast_subscription WHERE user_id = $1 AND podcast_id = $2", + ) + .bind(user_id) + .bind(podcast_id.as_str()) + .fetch_optional(&mut *conn) + .await?; + let Some(row) = row else { + return Ok(None); + }; + let record = SubscriptionRecord { + podcast_id: podcast_id.clone(), + subscribed: row.try_get("subscribed")?, + podcast: podcast_meta_col(&row, "podcast_json")?, + feed_aliases: load_aliases(conn, user_id, podcast_id).await?, + generation: generation_from_sql(row.try_get("generation")?), + hlc_ms: row.try_get("hlc_ms")?, + op_id: row.try_get("op_id")?, + }; + Ok(Some(record)) +} + +pub(crate) async fn load_aliases( + conn: &mut PgConnection, + user_id: i64, + podcast_id: &PodcastId, +) -> Result> { + let rows = sqlx::query( + "SELECT feed_url FROM furumusic__podcast_feed_alias + WHERE user_id = $1 AND podcast_id = $2 ORDER BY feed_url COLLATE \"C\"", + ) + .bind(user_id) + .bind(podcast_id.as_str()) + .fetch_all(&mut *conn) + .await?; + let mut aliases = rows + .iter() + .map(|row| feed_url_col(row, "feed_url")) + .collect::>>()?; + // `merge_feed_aliases` returns byte order; the database collation may + // differ (ICU, glibc locales), and the comparisons below need equality. + aliases.sort(); + Ok(aliases) +} + +pub(crate) fn share_from_row(row: &sqlx::postgres::PgRow) -> sqlx::Result { + Ok(ShareRecord { + podcast_id: super::podcast_id_col(row, "podcast_id")?, + shared: row.try_get("shared")?, + generation: generation_from_sql(row.try_get("generation")?), + hlc_ms: row.try_get("hlc_ms")?, + op_id: row.try_get("op_id")?, + }) +} + +pub(crate) async fn load_share( + conn: &mut PgConnection, + user_id: i64, + podcast_id: &PodcastId, +) -> Result> { + let row = sqlx::query( + "SELECT podcast_id, shared, generation, hlc_ms, op_id + FROM furumusic__podcast_share WHERE user_id = $1 AND podcast_id = $2", + ) + .bind(user_id) + .bind(podcast_id.as_str()) + .fetch_optional(&mut *conn) + .await?; + Ok(row.as_ref().map(share_from_row).transpose()?) +} + +pub(crate) async fn load_progress( + conn: &mut PgConnection, + user_id: i64, + episode_id: &EpisodeId, +) -> Result> { + let row = sqlx::query(&format!( + "SELECT {} FROM furumusic__podcast_progress p + WHERE p.user_id = $1 AND p.episode_id = $2", + progress_columns("p", "") + )) + .bind(user_id) + .bind(episode_id.as_str()) + .fetch_optional(&mut *conn) + .await?; + Ok(row + .as_ref() + .map(|row| progress_from_row(row, "")) + .transpose()?) +} + +pub(crate) async fn load_session( + conn: &mut PgConnection, + user_id: i64, + session_id: &str, +) -> Result> { + let row = sqlx::query(&format!( + "SELECT {SESSION_COLUMNS} FROM furumusic__podcast_listen_session s + WHERE s.user_id = $1 AND s.session_id = $2" + )) + .bind(user_id) + .bind(session_id) + .fetch_optional(&mut *conn) + .await?; + Ok(row.as_ref().map(session_from_row).transpose()?) +} + +/// Replaces the alias set of a podcast with `merged` when it differs. +async fn store_aliases( + conn: &mut PgConnection, + user_id: i64, + podcast_id: &PodcastId, + merged: &[FeedUrl], +) -> Result { + let current = load_aliases(conn, user_id, podcast_id).await?; + if current == merged { + return Ok(false); + } + sqlx::query("DELETE FROM furumusic__podcast_feed_alias WHERE user_id = $1 AND podcast_id = $2") + .bind(user_id) + .bind(podcast_id.as_str()) + .execute(&mut *conn) + .await?; + let urls: Vec<&str> = merged.iter().map(FeedUrl::as_str).collect(); + sqlx::query( + "INSERT INTO furumusic__podcast_feed_alias (user_id, podcast_id, feed_url) + SELECT $1, $2, url FROM UNNEST($3::text[]) AS url", + ) + .bind(user_id) + .bind(podcast_id.as_str()) + .bind(&urls) + .execute(&mut *conn) + .await?; + Ok(true) +} + +/// Merges feed URLs into the bounded grow-only alias set of a podcast. +/// +/// Returns whether the stored set changed. +pub(crate) async fn merge_aliases( + conn: &mut PgConnection, + user_id: i64, + podcast_id: &PodcastId, + urls: &[FeedUrl], +) -> Result { + let current = load_aliases(conn, user_id, podcast_id).await?; + let merged = merge_feed_aliases(podcast_id, ¤t, urls); + store_aliases(conn, user_id, podcast_id, &merged).await +} + +/// Inserts the catalog row of a podcast if it does not exist yet. +pub(crate) async fn ensure_podcast_row( + conn: &mut PgConnection, + user_id: i64, + podcast_id: &PodcastId, + feed_url: &FeedUrl, + title: &str, + now: i64, +) -> Result<()> { + sqlx::query( + "INSERT INTO furumusic__podcast + (user_id, podcast_id, feed_url, title, meta_from_feed, created_at_ms, updated_at_ms) + VALUES ($1, $2, $3, $4, FALSE, $5, $5) + ON CONFLICT (user_id, podcast_id) DO NOTHING", + ) + .bind(user_id) + .bind(podcast_id.as_str()) + .bind(feed_url.as_str()) + .bind(title) + .bind(now) + .execute(&mut *conn) + .await?; + Ok(()) +} + +/// Applies a subscription register record (operation or snapshot). +/// +/// The register is last-writer-wins ([`SubscriptionRecord::supersedes`]); +/// the alias set is a bounded grow-only set merged independently, so the +/// stored aliases are the bounded union of every URL seen (what +/// `merge_subscription` computes). The catalog row is created from the +/// register metadata and follows the winning register until this account +/// has refreshed the feed itself. +pub(crate) async fn apply_subscription_record( + conn: &mut PgConnection, + user_id: i64, + record: &SubscriptionRecord, + now: i64, +) -> Result { + let generation = generation_to_sql(record.generation)?; + let podcast_id = &record.podcast_id; + let current = load_subscription_register(conn, user_id, podcast_id).await?; + let wins = current + .as_ref() + .is_none_or(|current| record.supersedes(current)); + let mut changed = false; + if wins { + sqlx::query( + "INSERT INTO furumusic__podcast_subscription + (user_id, podcast_id, subscribed, podcast_json, generation, hlc_ms, op_id) + VALUES ($1, $2, $3, $4, $5, $6, $7) + ON CONFLICT (user_id, podcast_id) DO UPDATE SET + subscribed = EXCLUDED.subscribed, podcast_json = EXCLUDED.podcast_json, + generation = EXCLUDED.generation, hlc_ms = EXCLUDED.hlc_ms, + op_id = EXCLUDED.op_id", + ) + .bind(user_id) + .bind(podcast_id.as_str()) + .bind(record.subscribed) + .bind(serde_json::to_value(&record.podcast)?) + .bind(generation) + .bind(record.hlc_ms) + .bind(&record.op_id) + .execute(&mut *conn) + .await?; + changed = true; + } + let mut urls = record.feed_aliases.clone(); + urls.push(record.podcast.feed_url.clone()); + changed |= merge_aliases(conn, user_id, podcast_id, &urls).await?; + + ensure_podcast_row( + conn, + user_id, + podcast_id, + &record.podcast.feed_url, + &record.podcast.title, + now, + ) + .await?; + if wins { + let meta = &record.podcast; + sqlx::query( + "UPDATE furumusic__podcast + SET feed_url = $3, title = $4, author = $5, + description = COALESCE($6, description), link = $7, image_url = $8, + updated_at_ms = $9 + WHERE user_id = $1 AND podcast_id = $2 AND NOT meta_from_feed", + ) + .bind(user_id) + .bind(podcast_id.as_str()) + .bind(meta.feed_url.as_str()) + .bind(&meta.title) + .bind(&meta.author) + .bind(&meta.description) + .bind(&meta.link) + .bind(&meta.image_url) + .bind(now) + .execute(&mut *conn) + .await?; + } + Ok(changed) +} + +/// Applies a share register record. +pub(crate) async fn apply_share_record( + conn: &mut PgConnection, + user_id: i64, + record: &ShareRecord, +) -> Result { + let generation = generation_to_sql(record.generation)?; + let current = load_share(conn, user_id, &record.podcast_id).await?; + if current + .as_ref() + .is_some_and(|current| !record.supersedes(current)) + { + return Ok(false); + } + sqlx::query( + "INSERT INTO furumusic__podcast_share + (user_id, podcast_id, shared, generation, hlc_ms, op_id) + VALUES ($1, $2, $3, $4, $5, $6) + ON CONFLICT (user_id, podcast_id) DO UPDATE SET + shared = EXCLUDED.shared, generation = EXCLUDED.generation, + hlc_ms = EXCLUDED.hlc_ms, op_id = EXCLUDED.op_id", + ) + .bind(user_id) + .bind(record.podcast_id.as_str()) + .bind(record.shared) + .bind(generation) + .bind(record.hlc_ms) + .bind(&record.op_id) + .execute(&mut *conn) + .await?; + Ok(true) +} + +/// Applies a progress register record. Never touches the catalog. +pub(crate) async fn apply_progress_record( + conn: &mut PgConnection, + user_id: i64, + record: &ProgressRecord, +) -> Result { + let generation = generation_to_sql(record.generation)?; + let current = load_progress(conn, user_id, &record.episode_id).await?; + if current + .as_ref() + .is_some_and(|current| !record.supersedes(current)) + { + return Ok(false); + } + sqlx::query( + "INSERT INTO furumusic__podcast_progress + (user_id, episode_id, podcast_id, position_ms, status, generation, session_id, + hlc_ms, device_id, op_id) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10) + ON CONFLICT (user_id, episode_id) DO UPDATE SET + podcast_id = EXCLUDED.podcast_id, position_ms = EXCLUDED.position_ms, + status = EXCLUDED.status, generation = EXCLUDED.generation, + session_id = EXCLUDED.session_id, hlc_ms = EXCLUDED.hlc_ms, + device_id = EXCLUDED.device_id, op_id = EXCLUDED.op_id", + ) + .bind(user_id) + .bind(record.episode_id.as_str()) + .bind(record.podcast_id.as_str()) + .bind(record.position_ms) + .bind(status_to_sql(record.status)) + .bind(generation) + .bind(&record.session_id) + .bind(record.hlc_ms) + .bind(&record.device_id) + .bind(&record.op_id) + .execute(&mut *conn) + .await?; + Ok(true) +} + +/// Adds or grows a listening session (grow-only set keyed by `session_id`). +pub(crate) async fn apply_listen_session( + conn: &mut PgConnection, + user_id: i64, + session: &ListenSession, +) -> Result { + let current = load_session(conn, user_id, &session.session_id).await?; + let merged = match ¤t { + Some(current) => merge_listen_session(current, session).map_err(invalid)?, + None => session.clone(), + }; + if current.as_ref() == Some(&merged) { + return Ok(false); + } + sqlx::query( + "INSERT INTO furumusic__podcast_listen_session + (user_id, session_id, episode_id, podcast_id, device_id, started_at_ms, + ended_at_ms, start_position_ms, end_position_ms, listened_ms, completed) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11) + ON CONFLICT (user_id, session_id) DO UPDATE SET + started_at_ms = EXCLUDED.started_at_ms, ended_at_ms = EXCLUDED.ended_at_ms, + start_position_ms = EXCLUDED.start_position_ms, + end_position_ms = EXCLUDED.end_position_ms, + listened_ms = EXCLUDED.listened_ms, completed = EXCLUDED.completed", + ) + .bind(user_id) + .bind(&merged.session_id) + .bind(merged.episode_id.as_str()) + .bind(merged.podcast_id.as_str()) + .bind(&merged.device_id) + .bind(merged.started_at_ms) + .bind(merged.ended_at_ms) + .bind(merged.start_position_ms) + .bind(merged.end_position_ms) + .bind(merged.listened_ms) + .bind(merged.completed) + .execute(&mut *conn) + .await?; + Ok(true) +} + +pub(crate) fn episode_type_to_sql(kind: EpisodeType) -> &'static str { + match kind { + EpisodeType::Full => "full", + EpisodeType::Trailer => "trailer", + EpisodeType::Bonus => "bonus", + } +} + +/// Inserts an episode row. +pub(crate) async fn insert_episode( + conn: &mut PgConnection, + user_id: i64, + meta: &EpisodeMeta, + source: EpisodeSource, + in_feed: bool, + now: i64, +) -> Result<()> { + sqlx::query( + "INSERT INTO furumusic__podcast_episode + (user_id, episode_id, podcast_id, guid, title, description, season, + episode_number, episode_type, published_at_ms, duration_ms, audio_url, mime, + byte_len, image_url, link, source, in_feed, search_norm, first_seen_at_ms, + updated_at_ms) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, + $18, $19, $20, $20)", + ) + .bind(user_id) + .bind(meta.episode_id.as_str()) + .bind(meta.podcast_id.as_str()) + .bind(&meta.guid) + .bind(&meta.title) + .bind(&meta.description) + .bind(meta.season.map(i64::from)) + .bind(meta.episode_number.map(i64::from)) + .bind(episode_type_to_sql(meta.episode_type)) + .bind(meta.published_at_ms) + .bind(meta.duration_ms) + .bind(&meta.audio_url) + .bind(&meta.mime) + .bind(meta.byte_len.and_then(|len| i64::try_from(len).ok())) + .bind(&meta.image_url) + .bind(&meta.link) + .bind(source.as_sql()) + .bind(in_feed) + .bind(episode_search_norm( + &meta.title, + meta.description.as_deref(), + )) + .bind(now) + .execute(&mut *conn) + .await?; + Ok(()) +} + +/// Episode metadata of a reference learned through sync. +pub(crate) fn meta_from_ref(episode: &PodcastEpisodeRef) -> EpisodeMeta { + EpisodeMeta { + episode_id: episode.episode_id.clone(), + podcast_id: episode.podcast_id.clone(), + guid: None, + title: episode.episode_title.clone(), + description: None, + season: episode.season, + episode_number: episode.episode_number, + episode_type: EpisodeType::default(), + published_at_ms: episode.published_at_ms, + duration_ms: episode.duration_ms, + audio_url: episode.audio_url.clone(), + mime: episode.mime.clone(), + byte_len: None, + image_url: episode.image_url.clone(), + link: None, + } +} + +/// Records an episode learned through sync when the catalog does not know it. +/// +/// Feed metadata always wins: an existing row is left untouched. The +/// podcast catalog row is created from the reference when missing (no +/// register is written). +pub(crate) async fn ensure_episode_from_ref( + conn: &mut PgConnection, + user_id: i64, + episode: &PodcastEpisodeRef, +) -> Result<()> { + let exists: bool = sqlx::query_scalar( + "SELECT EXISTS (SELECT 1 FROM furumusic__podcast_episode + WHERE user_id = $1 AND episode_id = $2)", + ) + .bind(user_id) + .bind(episode.episode_id.as_str()) + .fetch_one(&mut *conn) + .await?; + if exists { + return Ok(()); + } + let now = now_ms(); + ensure_podcast_row( + conn, + user_id, + &episode.podcast_id, + &episode.feed_url, + &episode.podcast_title, + now, + ) + .await?; + insert_episode( + conn, + user_id, + &meta_from_ref(episode), + EpisodeSource::Sync, + false, + now, + ) + .await +} + +/// Known episodes of a podcast with the identity hints they were last seen with. +pub(crate) async fn known_episodes( + conn: &mut PgConnection, + user_id: i64, + podcast_id: &PodcastId, +) -> Result> { + let rows = sqlx::query( + "SELECT episode_id, guid, audio_url, link, published_at_ms, title + FROM furumusic__podcast_episode + WHERE user_id = $1 AND podcast_id = $2 ORDER BY episode_id COLLATE \"C\"", + ) + .bind(user_id) + .bind(podcast_id.as_str()) + .fetch_all(&mut *conn) + .await?; + let known = rows + .iter() + .map(|row| { + Ok(KnownEpisode { + episode_id: episode_id_col(row, "episode_id")?, + hints: EpisodeIdentityHints { + guid: row.try_get("guid")?, + enclosure_url: Some(row.try_get("audio_url")?), + link: row.try_get("link")?, + published_at_ms: row.try_get("published_at_ms")?, + title: Some(row.try_get("title")?), + }, + }) + }) + .collect::>>()?; + Ok(known) +} + +/// Applies one podcast operation inside the caller's transaction (which +/// holds the account lock). +pub(crate) async fn apply_op( + conn: &mut PgConnection, + user_id: i64, + op: &SyncOpWire, +) -> Result { + if !op.payload.is_podcast() { + return Ok(ApplyOutcome::Ignored); + } + if let Err(error) = op.validate_podcast() { + return Ok(ApplyOutcome::Rejected(error.to_string())); + } + let result = match &op.payload { + SyncOpPayload::PodcastSubscriptionSet { .. } => { + let record = op.podcast_subscription().expect("subscription payload"); + apply_subscription_record(conn, user_id, &record, now_ms()).await + } + SyncOpPayload::PodcastShareSet { .. } => { + apply_share_record(conn, user_id, &op.podcast_share().expect("share payload")).await + } + SyncOpPayload::PodcastFeedAliasesAdded { .. } => { + let (podcast_id, urls) = op.podcast_feed_aliases().expect("alias payload"); + merge_aliases(conn, user_id, podcast_id, urls).await + } + SyncOpPayload::PodcastProgressSet { episode, .. } => { + let record = op.podcast_progress().expect("progress payload"); + if let Err(error) = generation_to_sql(record.generation) { + return Ok(ApplyOutcome::Rejected(error.to_string())); + } + if let Some(episode) = episode { + ensure_episode_from_ref(conn, user_id, episode).await?; + } + apply_progress_record(conn, user_id, &record).await + } + SyncOpPayload::PodcastListenRecorded { session, episode } => { + if let Some(current) = load_session(conn, user_id, &session.session_id).await? + && let Err(error) = merge_listen_session(¤t, session) + { + return Ok(ApplyOutcome::Rejected(error.to_string())); + } + if let Some(episode) = episode { + ensure_episode_from_ref(conn, user_id, episode).await?; + } + apply_listen_session(conn, user_id, session).await + } + _ => return Ok(ApplyOutcome::Ignored), + }; + match result { + Ok(true) => Ok(ApplyOutcome::Changed), + Ok(false) => Ok(ApplyOutcome::Unchanged), + Err(error) => match error.downcast_ref::() { + Some(PodcastStoreError::Invalid(reason)) => Ok(ApplyOutcome::Rejected(reason.clone())), + _ => Err(error), + }, + } +} + +fn record_outcome(stats: &mut SnapshotApplyStats, result: Result) -> Result<()> { + match result { + Ok(true) => stats.changed += 1, + Ok(false) => stats.unchanged += 1, + Err(error) => match error.downcast_ref::() { + Some(PodcastStoreError::Invalid(_)) => stats.rejected += 1, + _ => return Err(error), + }, + } + Ok(()) +} + +fn check(result: Result<(), PodcastError>) -> Result<()> { + result.map_err(invalid) +} + +impl PodcastStore { + /// Applies a device-sync operation received from a trusted device. + /// + /// Non-podcast operations return [`ApplyOutcome::Ignored`]. Podcast + /// operations are validated (`SyncOpWire::validate_podcast`) and merged + /// with the contract functions in one transaction; applying the same + /// operations in any order, with duplicates, gives the same state. + pub(crate) async fn apply_remote_op( + &self, + user_id: i64, + op: &SyncOpWire, + ) -> Result { + if !op.payload.is_podcast() { + return Ok(ApplyOutcome::Ignored); + } + let mut tx = self.begin_write(user_id).await?; + let outcome = apply_op(&mut tx, user_id, op).await?; + tx.commit().await?; + Ok(outcome) + } + + /// Merges the podcast sections of a sync snapshot. + /// + /// Sections exceeding the contract counts are ignored + /// (`SyncSnapshot::podcast_state_within_limits`); every record is + /// validated and merged with the same functions as operations, so a + /// snapshot never replaces newer local state. One transaction. + pub(crate) async fn apply_snapshot( + &self, + user_id: i64, + snapshot: &SyncSnapshot, + ) -> Result { + let mut stats = SnapshotApplyStats::default(); + if !snapshot.has_podcast_state() { + return Ok(stats); + } + if !snapshot.podcast_state_within_limits() { + stats.ignored_over_limits = true; + return Ok(stats); + } + let mut tx = self.begin_write(user_id).await?; + let now = now_ms(); + for record in &snapshot.podcast_subscriptions { + let mut record = record.clone(); + record.normalize(); + let result = match check(record.validate()) { + Ok(()) => apply_subscription_record(&mut tx, user_id, &record, now).await, + Err(error) => Err(error), + }; + record_outcome(&mut stats, result)?; + } + for record in &snapshot.podcast_shares { + let result = match check(record.validate()) { + Ok(()) => apply_share_record(&mut tx, user_id, record).await, + Err(error) => Err(error), + }; + record_outcome(&mut stats, result)?; + } + for record in &snapshot.podcast_progress { + let result = match check(record.validate()) { + Ok(()) => apply_progress_record(&mut tx, user_id, record).await, + Err(error) => Err(error), + }; + record_outcome(&mut stats, result)?; + } + for session in &snapshot.podcast_listen_sessions { + let result = match check(session.validate()) { + Ok(()) => apply_listen_session(&mut tx, user_id, session).await, + Err(error) => Err(error), + }; + record_outcome(&mut stats, result)?; + } + tx.commit().await?; + Ok(stats) + } + + /// Fills the podcast sections of `snapshot` with this account's + /// registers and sessions, bounded by + /// `SyncSnapshot::bound_podcast_state_to(budget_bytes)`. + /// + /// Records are read newest first and capped by the per-section counts + /// before the byte budget is applied. Fetch state and the catalog are + /// never part of the snapshot. Send it only to v4 peers + /// (`SyncSnapshot::for_peer`). + pub(crate) async fn fill_snapshot( + &self, + user_id: i64, + snapshot: &mut SyncSnapshot, + budget_bytes: usize, + ) -> Result<()> { + use music_dht::podcast::{ + MAX_SNAPSHOT_LISTEN_SESSIONS, MAX_SNAPSHOT_PROGRESS, MAX_SNAPSHOT_SHARES, + MAX_SNAPSHOT_SUBSCRIPTIONS, + }; + // One consistent view of every section. + let mut tx = self.pool.begin().await?; + sqlx::query("SET TRANSACTION ISOLATION LEVEL REPEATABLE READ READ ONLY") + .execute(&mut *tx) + .await?; + let rows = sqlx::query( + "SELECT s.podcast_id, s.subscribed, s.podcast_json, s.generation, s.hlc_ms, s.op_id, + COALESCE((SELECT array_agg(a.feed_url ORDER BY a.feed_url COLLATE \"C\") + FROM furumusic__podcast_feed_alias a + WHERE a.user_id = s.user_id AND a.podcast_id = s.podcast_id), + '{}') AS aliases + FROM furumusic__podcast_subscription s + WHERE s.user_id = $1 + ORDER BY s.generation DESC, s.hlc_ms DESC, s.op_id COLLATE \"C\" DESC, + s.podcast_id COLLATE \"C\" DESC + LIMIT $2", + ) + .bind(user_id) + .bind(MAX_SNAPSHOT_SUBSCRIPTIONS as i64) + .fetch_all(&mut *tx) + .await?; + let subscriptions = rows + .iter() + .map(|row| { + let aliases: Vec = row.try_get("aliases")?; + Ok(SubscriptionRecord { + podcast_id: super::podcast_id_col(row, "podcast_id")?, + subscribed: row.try_get("subscribed")?, + podcast: podcast_meta_col(row, "podcast_json")?, + feed_aliases: aliases + .into_iter() + .map(FeedUrl::try_from) + .collect::>() + .map_err(|error| sqlx::Error::Decode(Box::new(error)))?, + generation: generation_from_sql(row.try_get("generation")?), + hlc_ms: row.try_get("hlc_ms")?, + op_id: row.try_get("op_id")?, + }) + }) + .collect::>>()?; + let shares = sqlx::query( + "SELECT podcast_id, shared, generation, hlc_ms, op_id FROM furumusic__podcast_share + WHERE user_id = $1 + ORDER BY generation DESC, hlc_ms DESC, op_id COLLATE \"C\" DESC, + podcast_id COLLATE \"C\" DESC + LIMIT $2", + ) + .bind(user_id) + .bind(MAX_SNAPSHOT_SHARES as i64) + .fetch_all(&mut *tx) + .await? + .iter() + .map(share_from_row) + .collect::>>()?; + let progress = sqlx::query(&format!( + "SELECT {} FROM furumusic__podcast_progress p + WHERE p.user_id = $1 + ORDER BY p.generation DESC, p.hlc_ms DESC, p.op_id COLLATE \"C\" DESC, + p.episode_id COLLATE \"C\" DESC + LIMIT $2", + progress_columns("p", "") + )) + .bind(user_id) + .bind(MAX_SNAPSHOT_PROGRESS as i64) + .fetch_all(&mut *tx) + .await? + .iter() + .map(|row| progress_from_row(row, "")) + .collect::>>()?; + let sessions = sqlx::query(&format!( + "SELECT {SESSION_COLUMNS} FROM furumusic__podcast_listen_session s + WHERE s.user_id = $1 + ORDER BY s.ended_at_ms DESC, s.session_id COLLATE \"C\" DESC + LIMIT $2" + )) + .bind(user_id) + .bind(MAX_SNAPSHOT_LISTEN_SESSIONS as i64) + .fetch_all(&mut *tx) + .await? + .iter() + .map(session_from_row) + .collect::>>()?; + tx.commit().await?; + snapshot.podcast_subscriptions = subscriptions; + snapshot.podcast_shares = shares; + snapshot.podcast_progress = progress; + snapshot.podcast_listen_sessions = sessions; + snapshot.bound_podcast_state_to(budget_bytes); + Ok(()) + } + + /// A snapshot holding only this account's podcast sections, bounded + /// with the default budget (`MAX_PODCAST_SNAPSHOT_BYTES`). + #[cfg(test)] + pub(crate) async fn export_snapshot(&self, user_id: i64) -> Result { + let mut snapshot = SyncSnapshot::default(); + self.fill_snapshot( + user_id, + &mut snapshot, + music_dht::podcast::MAX_PODCAST_SNAPSHOT_BYTES, + ) + .await?; + Ok(snapshot) + } + + /// Progress register of an episode (also for episodes the catalog does + /// not know yet). + pub(crate) async fn progress( + &self, + user_id: i64, + episode_id: &EpisodeId, + ) -> Result> { + let mut conn = self.pool.acquire().await?; + load_progress(&mut conn, user_id, episode_id).await + } +} diff --git a/src/podcasts/fetch.rs b/src/podcasts/fetch.rs new file mode 100644 index 0000000..98b295a --- /dev/null +++ b/src/podcasts/fetch.rs @@ -0,0 +1,322 @@ +//! Conditional, size- and time-bounded feed downloads through [`SafeHttp`]. + +use std::fmt; +use std::time::Duration; + +use reqwest::header::{ + ACCEPT, ACCEPT_ENCODING, ETAG, HeaderMap, HeaderValue, IF_MODIFIED_SINCE, IF_NONE_MATCH, + LAST_MODIFIED, RETRY_AFTER, +}; +use reqwest::{StatusCode, Url}; + +use music_dht::podcast::FeedUrl; +use music_dht::podcast::feed::{ParsedFeed, parse_feed_with_base}; + +use super::net::{GuardError, REFUSED_MESSAGE, SafeHttp, SafeHttpConfig}; + +/// Hard cap on a feed document (large catalogs are a few MiB). +pub(crate) const MAX_FEED_BYTES: usize = 32 * 1024 * 1024; +/// Whole fetch, from connect to the last body byte. +pub(crate) const FEED_FETCH_TIMEOUT: Duration = Duration::from_secs(60); +/// Redirect hops followed for a feed. +pub(crate) const MAX_FEED_REDIRECTS: usize = 5; +/// `Retry-After` values above this are clamped. +const MAX_RETRY_AFTER: Duration = Duration::from_secs(24 * 60 * 60); + +const FEED_ACCEPT: &str = "application/rss+xml, application/atom+xml, application/xml;q=0.9, \ + text/xml;q=0.9, */*;q=0.5"; + +/// HTTP client settings of the feed fetcher. +pub(crate) const FEED_HTTP: SafeHttpConfig = SafeHttpConfig { + connect_timeout: Duration::from_secs(10), + read_timeout: Duration::from_secs(30), + total_timeout: Some(FEED_FETCH_TIMEOUT), + max_redirects: MAX_FEED_REDIRECTS, +}; + +/// Validators of the previous successful fetch. +#[derive(Debug, Clone, Default, PartialEq, Eq, Hash)] +pub(crate) struct Validators { + pub(crate) etag: Option, + pub(crate) last_modified: Option, +} + +/// A successfully fetched and parsed document. +#[derive(Debug, Clone)] +pub(crate) struct FetchedFeed { + pub(crate) feed: ParsedFeed, + /// Where the feed now lives: the final URL after a chain of permanent + /// redirects, otherwise the requested URL. + pub(crate) fetched_from: FeedUrl, + /// Whether the feed moved: at least one redirect, all permanent (see + /// [`super::net::Followed::moved_permanently`]), to a valid feed URL. + pub(crate) moved_permanently: bool, + /// New validators for the next conditional request. + pub(crate) validators: Validators, +} + +/// Outcome of [`FeedFetcher::fetch`]. +#[derive(Debug, Clone)] +pub(crate) enum FeedFetch { + /// `304 Not Modified`: the stored catalog is current. + NotModified, + /// A new document. + Fetched(Box), +} + +/// Kind of a failed fetch. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) enum FetchErrorKind { + /// Refused by the SSRF guard (blocked address or port, scheme, + /// credentials). + Refused, + /// Non-success HTTP status. + Status(u16), + /// Body over [`MAX_FEED_BYTES`]. + TooLarge, + /// The document is not a feed. + Parse, + /// Timed out. + Timeout, + /// DNS, connect, TLS or a dropped connection. + Transport, +} + +/// A failed fetch. The stored catalog is never touched on failure. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct FetchError { + pub(crate) kind: FetchErrorKind, + /// Details for the server logs (no URLs; see [`Self::public_message`] + /// for what users see). + pub(crate) message: String, + /// Server-requested delay (`Retry-After` on 429/503), clamped to a day. + pub(crate) retry_after: Option, +} + +impl FetchError { + fn new(kind: FetchErrorKind, message: impl Into) -> Self { + Self { + kind, + message: message.into(), + retry_after: None, + } + } + + /// What the signed-in user is told (API answers, the stored last + /// error): the kind of failure, never resolved addresses or transport + /// details, which would let users probe the server's network. + pub(crate) fn public_message(&self) -> String { + match &self.kind { + FetchErrorKind::Refused => REFUSED_MESSAGE.to_string(), + FetchErrorKind::Status(status) => format!("the server answered HTTP {status}"), + FetchErrorKind::TooLarge => format!( + "the feed is larger than {} MiB", + MAX_FEED_BYTES / (1024 * 1024) + ), + FetchErrorKind::Parse => "the document is not a podcast feed".to_string(), + FetchErrorKind::Timeout => "the request timed out".to_string(), + FetchErrorKind::Transport => "could not connect to the server".to_string(), + } + } +} + +impl fmt::Display for FetchError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str(&self.message) + } +} + +impl std::error::Error for FetchError {} + +impl From for FetchError { + fn from(error: GuardError) -> Self { + let kind = if error.is_refusal() { + FetchErrorKind::Refused + } else if error.is_timeout() { + FetchErrorKind::Timeout + } else { + FetchErrorKind::Transport + }; + Self::new(kind, error.to_string()) + } +} + +/// Parses `Retry-After` (delta seconds or an HTTP date). +pub(crate) fn parse_retry_after( + value: &str, + now: chrono::DateTime, +) -> Option { + let value = value.trim(); + let delay = if let Ok(seconds) = value.parse::() { + Duration::from_secs(seconds) + } else { + let at = chrono::DateTime::parse_from_rfc2822(value).ok()?; + (at.with_timezone(&chrono::Utc) - now) + .to_std() + .unwrap_or_default() + }; + Some(delay.min(MAX_RETRY_AFTER)) +} + +/// Downloads feeds with the SSRF guard, conditional requests, redirects +/// and hard size/time limits; parses them with the shared parser. +#[derive(Debug, Clone)] +pub(crate) struct FeedFetcher { + http: SafeHttp, + max_bytes: usize, +} + +impl FeedFetcher { + /// The production fetcher (public addresses only). + pub(crate) fn new() -> anyhow::Result { + Ok(Self::with_http(SafeHttp::public(FEED_HTTP)?)) + } + + pub(crate) fn with_http(http: SafeHttp) -> Self { + Self { + http, + max_bytes: MAX_FEED_BYTES, + } + } + + #[cfg(test)] + pub(crate) fn with_max_bytes(mut self, max_bytes: usize) -> Self { + self.max_bytes = max_bytes; + self + } + + /// Fetches `url`, conditionally when `validators` are known. + pub(crate) async fn fetch( + &self, + url: &FeedUrl, + validators: &Validators, + ) -> Result { + match tokio::time::timeout(FEED_FETCH_TIMEOUT, self.fetch_inner(url, validators)).await { + Ok(result) => result, + Err(_) => Err(FetchError::new( + FetchErrorKind::Timeout, + format!( + "feed fetch timed out after {} s", + FEED_FETCH_TIMEOUT.as_secs() + ), + )), + } + } + + async fn fetch_inner( + &self, + feed_url: &FeedUrl, + validators: &Validators, + ) -> Result { + let url = Url::parse(feed_url.as_str()) + .map_err(|error| FetchError::new(FetchErrorKind::Refused, error.to_string()))?; + let mut headers = HeaderMap::new(); + headers.insert(ACCEPT, HeaderValue::from_static(FEED_ACCEPT)); + // Compressed bodies are not negotiated (no decoder is enabled), so + // the size cap applies to what is parsed. + headers.insert(ACCEPT_ENCODING, HeaderValue::from_static("identity")); + if let Some(etag) = validators + .etag + .as_deref() + .and_then(|value| HeaderValue::from_str(value).ok()) + { + headers.insert(IF_NONE_MATCH, etag); + } + if let Some(date) = validators + .last_modified + .as_deref() + .and_then(|value| HeaderValue::from_str(value).ok()) + { + headers.insert(IF_MODIFIED_SINCE, date); + } + let followed = self.http.get(url, &headers).await?; + let mut moved_permanently = followed.moved_permanently(); + let final_url = followed.final_url; + let response = followed.response; + let status = response.status(); + if status == StatusCode::NOT_MODIFIED { + return Ok(FeedFetch::NotModified); + } + if !status.is_success() { + let mut error = FetchError::new( + FetchErrorKind::Status(status.as_u16()), + format!("feed request failed: HTTP {status}"), + ); + if matches!(status.as_u16(), 429 | 503) { + error.retry_after = response + .headers() + .get(RETRY_AFTER) + .and_then(|value| value.to_str().ok()) + .and_then(|value| parse_retry_after(value, chrono::Utc::now())); + } + return Err(error); + } + let header = |name| { + response + .headers() + .get(name) + .and_then(|value: &HeaderValue| value.to_str().ok()) + .map(str::to_string) + }; + let validators = Validators { + etag: header(ETAG), + last_modified: header(LAST_MODIFIED), + }; + let body = read_capped(response, self.max_bytes).await?; + // Relative enclosure URLs resolve against where the feed really is. + let base = final_url.to_string(); + let feed = tokio::task::spawn_blocking(move || parse_feed_with_base(&body, Some(&base))) + .await + .map_err(|error| { + FetchError::new( + FetchErrorKind::Parse, + format!("feed parser task failed: {error}"), + ) + })? + .map_err(|error| { + FetchError::new( + FetchErrorKind::Parse, + format!("not a podcast feed: {error}"), + ) + })?; + // Only a permanent move changes the feed URL; a target that is not a + // valid feed URL (over-long, ...) is not learned. + let moved_to = moved_permanently + .then(|| FeedUrl::parse(final_url.as_str()).ok()) + .flatten(); + moved_permanently = moved_to.is_some(); + let fetched_from = moved_to.unwrap_or_else(|| feed_url.clone()); + Ok(FeedFetch::Fetched(Box::new(FetchedFeed { + feed, + fetched_from, + moved_permanently, + validators, + }))) + } +} + +async fn read_capped(mut response: reqwest::Response, cap: usize) -> Result, FetchError> { + let too_large = |detail: String| { + FetchError::new( + FetchErrorKind::TooLarge, + format!("feed is too large ({detail}, limit {cap} bytes)"), + ) + }; + if let Some(len) = response.content_length() + && len > cap as u64 + { + return Err(too_large(format!("{len} bytes"))); + } + let mut body = Vec::new(); + while let Some(chunk) = response + .chunk() + .await + .map_err(|error| FetchError::from(GuardError::from(error)))? + { + if body.len() + chunk.len() > cap { + return Err(too_large("body exceeds the limit".to_string())); + } + body.extend_from_slice(&chunk); + } + Ok(body) +} diff --git a/src/podcasts/fetch_tests.rs b/src/podcasts/fetch_tests.rs new file mode 100644 index 0000000..b5be9eb --- /dev/null +++ b/src/podcasts/fetch_tests.rs @@ -0,0 +1,289 @@ +//! Feed fetching (conditional requests, redirects, limits) and the refresh +//! schedule, against the local test server. + +use std::time::Duration; + +use music_dht::podcast::{FeedUrl, PodcastId}; + +use super::fetch::{FeedFetch, FeedFetcher, FetchErrorKind, Validators, parse_retry_after}; +use super::runner::{ + BACKOFF_BASE_MS, BACKOFF_MAX_MS, REFRESH_INTERVAL_MS, failure_backoff_ms, next_regular_check_ms, +}; +use std::sync::Arc; + +use super::test_server::{Route, ScriptedLookup, TestServer, test_http}; + +// --------------------------------------------------------------------------- +// Feed fetching +// --------------------------------------------------------------------------- + +fn rss(items: &str) -> String { + format!( + r#"Show{items}"# + ) +} + +fn item(guid: &str) -> String { + format!( + r#"{guid}{guid}"# + ) +} + +fn fetcher(lookup: Arc) -> FeedFetcher { + FeedFetcher::with_http(test_http(lookup)) +} + +fn feed_url(text: &str) -> FeedUrl { + FeedUrl::parse(text).unwrap() +} + +#[tokio::test] +async fn feeds_are_fetched_conditionally() { + let server = TestServer::start(); + server.set( + "/feed.xml", + Route::new(rss(&item("a")), "application/rss+xml") + .etag("\"v1\"") + .last_modified("Wed, 21 Oct 2026 07:28:00 GMT"), + ); + let fetcher = fetcher(ScriptedLookup::with(&[])); + let feed = feed_url(&server.url("/feed.xml")); + let FeedFetch::Fetched(first) = fetcher.fetch(&feed, &Validators::default()).await.unwrap() + else { + panic!("expected a document"); + }; + assert_eq!(first.feed.episodes.len(), 1); + assert_eq!( + first.feed.episodes[0].audio_url, + server.url("/audio/a.mp3"), + "relative enclosures resolve against the feed URL" + ); + assert_eq!(first.fetched_from, feed); + assert!(!first.moved_permanently); + assert_eq!(first.validators.etag.as_deref(), Some("\"v1\"")); + let second = fetcher.fetch(&feed, &first.validators).await.unwrap(); + assert!(matches!(second, FeedFetch::NotModified)); + // Only Last-Modified known. + let by_date = Validators { + etag: None, + last_modified: first.validators.last_modified.clone(), + }; + assert!(matches!( + fetcher.fetch(&feed, &by_date).await.unwrap(), + FeedFetch::NotModified + )); + let requests = server.requests("/feed.xml"); + assert_eq!(requests.len(), 3); + assert!(!requests[0].contains_key("if-none-match")); + assert_eq!(requests[1]["if-none-match"], "\"v1\""); + assert_eq!( + requests[1]["if-modified-since"], + "Wed, 21 Oct 2026 07:28:00 GMT" + ); + assert!(!requests[2].contains_key("if-none-match")); + assert_eq!(requests[2]["accept-encoding"], "identity"); +} + +#[tokio::test] +async fn only_permanent_redirect_chains_move_a_feed() { + let server = TestServer::start(); + server.set( + "/new.xml", + Route::new(rss(&item("a")), "application/rss+xml"), + ); + server.set("/old.xml", Route::redirect(301, "/new.xml")); + server.set("/moved.xml", Route::redirect(308, "/old.xml")); + server.set("/temp.xml", Route::redirect(302, "/new.xml")); + server.set("/mixed.xml", Route::redirect(307, "/old.xml")); + let fetcher = fetcher(ScriptedLookup::with(&[])); + for (path, moved) in [ + ("/old.xml", true), + ("/moved.xml", true), + ("/temp.xml", false), + ("/mixed.xml", false), + ] { + let requested = feed_url(&server.url(path)); + let FeedFetch::Fetched(document) = fetcher + .fetch(&requested, &Validators::default()) + .await + .unwrap() + else { + panic!("{path}"); + }; + assert_eq!(document.moved_permanently, moved, "{path}"); + let expected = if moved { + feed_url(&server.url("/new.xml")) + } else { + requested + }; + assert_eq!(document.fetched_from, expected, "{path}"); + assert_eq!( + document.feed.episodes[0].audio_url, + server.url("/audio/a.mp3") + ); + } +} + +#[tokio::test] +async fn feed_fetch_failures_are_typed() { + let server = TestServer::start(); + let big = rss(&item("x").repeat(200)); + server.set("/big.xml", Route::new(big.clone(), "application/rss+xml")); + server.set( + "/big-chunked.xml", + Route::new(big.clone(), "application/rss+xml").no_length(), + ); + server.set( + "/page.html", + Route::new("hi", "text/html"), + ); + server.set( + "/busy.xml", + Route::new("busy", "text/plain") + .status(503) + .header("Retry-After", "120"), + ); + server.set("/gone.xml", Route::new("", "text/plain").status(410)); + server.set( + "/private.xml", + Route::redirect(301, "http://192.168.0.1/feed"), + ); + let fetcher = fetcher(ScriptedLookup::with(&[])).with_max_bytes(big.len() - 1); + let fetch = |path: &str| { + let feed = feed_url(&server.url(path)); + let fetcher = fetcher.clone(); + async move { + fetcher + .fetch(&feed, &Validators::default()) + .await + .unwrap_err() + } + }; + assert_eq!(fetch("/big.xml").await.kind, FetchErrorKind::TooLarge); + assert_eq!( + fetch("/big-chunked.xml").await.kind, + FetchErrorKind::TooLarge + ); + assert_eq!(fetch("/page.html").await.kind, FetchErrorKind::Parse); + let busy = fetch("/busy.xml").await; + assert_eq!(busy.kind, FetchErrorKind::Status(503)); + assert_eq!(busy.retry_after, Some(Duration::from_secs(120))); + let gone = fetch("/gone.xml").await; + assert_eq!(gone.kind, FetchErrorKind::Status(410)); + assert_eq!(gone.retry_after, None); + assert_eq!(fetch("/private.xml").await.kind, FetchErrorKind::Refused); + assert_eq!( + fetch("/missing.xml").await.kind, + FetchErrorKind::Status(404) + ); +} + +#[test] +fn retry_after_accepts_seconds_and_dates() { + let now = chrono::DateTime::parse_from_rfc3339("2026-11-19T10:00:00Z") + .unwrap() + .with_timezone(&chrono::Utc); + assert_eq!(parse_retry_after("90", now), Some(Duration::from_secs(90))); + assert_eq!( + parse_retry_after("Thu, 19 Nov 2026 10:05:00 GMT", now), + Some(Duration::from_secs(300)) + ); + assert_eq!( + parse_retry_after("Thu, 19 Nov 2026 09:00:00 GMT", now), + Some(Duration::ZERO) + ); + assert_eq!( + parse_retry_after("999999999", now), + Some(Duration::from_secs(24 * 60 * 60)) + ); + assert_eq!(parse_retry_after("soon", now), None); +} + +// --------------------------------------------------------------------------- +// Refresh schedule +// --------------------------------------------------------------------------- + +#[test] +fn failures_back_off_exponentially_up_to_a_day() { + assert_eq!(failure_backoff_ms(1, None), BACKOFF_BASE_MS); + assert_eq!(failure_backoff_ms(2, None), 2 * BACKOFF_BASE_MS); + assert_eq!(failure_backoff_ms(3, None), 4 * BACKOFF_BASE_MS); + assert_eq!(failure_backoff_ms(7, None), 64 * BACKOFF_BASE_MS); + assert_eq!(failure_backoff_ms(8, None), BACKOFF_MAX_MS); + assert_eq!(failure_backoff_ms(u32::MAX, None), BACKOFF_MAX_MS); + // A longer Retry-After wins, a shorter one does not shorten the backoff. + assert_eq!( + failure_backoff_ms(1, Some(Duration::from_secs(3 * 60 * 60))), + 3 * 60 * 60 * 1000 + ); + assert_eq!( + failure_backoff_ms(2, Some(Duration::from_secs(1))), + 2 * BACKOFF_BASE_MS + ); + assert_eq!( + failure_backoff_ms(1, Some(Duration::from_secs(u64::MAX))), + BACKOFF_MAX_MS + ); +} + +#[test] +fn regular_checks_are_hourly_with_a_stable_spread() { + let podcast = PodcastId::from_feed_url(&feed_url("https://feeds.example/a.xml")); + let other = PodcastId::from_feed_url(&feed_url("https://feeds.example/b.xml")); + let now = 1_000_000; + let next = next_regular_check_ms(now, 7, &podcast); + assert!( + (now + REFRESH_INTERVAL_MS..now + REFRESH_INTERVAL_MS + 10 * 60 * 1000).contains(&next) + ); + assert_eq!(next, next_regular_check_ms(now, 7, &podcast)); + let spread: std::collections::BTreeSet = (0..20) + .map(|user| next_regular_check_ms(now, user, &podcast)) + .chain([next_regular_check_ms(now, 7, &other)]) + .collect(); + assert!(spread.len() > 10); +} + +/// A port on which nothing listens. +fn closed_port() -> u16 { + let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap(); + listener.local_addr().unwrap().port() +} + +#[tokio::test] +async fn internal_names_cannot_be_told_from_unreachable_ones() { + // A name that resolves to a private address, a name that does not + // resolve and a server that refuses the connection look the same to + // the user: otherwise "add RSS" would tell which names exist in the + // server's internal DNS. Only what the URL itself says (an IP literal, + // the port, the scheme) is refused as such. + let port = closed_port(); + let lookup = ScriptedLookup::with(&[ + ("vault.internal.test", &[&["10.20.30.40"]]), + ("closed.test", &[&["127.0.0.1"]]), + ]); + let fetcher = fetcher(lookup); + let mut answers = Vec::new(); + for text in [ + format!("http://vault.internal.test:{port}/feed"), + format!("http://nowhere.internal.test:{port}/feed"), + format!("http://closed.test:{port}/feed"), + ] { + let error = fetcher + .fetch(&feed_url(&text), &Validators::default()) + .await + .unwrap_err(); + assert!(!error.public_message().contains("internal"), "{text}"); + answers.push((error.public_message(), error.kind)); + } + assert_eq!(answers[0].1, FetchErrorKind::Transport, "{answers:?}"); + assert!( + answers.iter().all(|answer| *answer == answers[0]), + "{answers:?}" + ); + let literal = fetcher + .fetch(&feed_url("http://10.20.30.40/feed"), &Validators::default()) + .await + .unwrap_err(); + assert_eq!(literal.kind, FetchErrorKind::Refused); + assert_ne!(literal.public_message(), answers[0].0); +} diff --git a/src/podcasts/http.rs b/src/podcasts/http.rs new file mode 100644 index 0000000..b57ac46 --- /dev/null +++ b/src/podcasts/http.rs @@ -0,0 +1,825 @@ +//! HTTP endpoints of the podcast runtime, mounted at `/api/podcasts`. The +//! request handling lives in [`super::api`]; this module authenticates (the +//! session or a bearer token of the signed-in account) and maps results to +//! JSON responses. +//! +//! Podcast UI: +//! +//! * `GET /feeds[?all=1]` — subscribed podcasts (with `all`, also +//! unsubscribed ones and podcasts only known from synced progress), with +//! counters and the local fetch state. +//! * `POST /feeds` `{url}` — "add RSS": fetch, then subscribe (private). +//! * `GET /feeds/{podcast_id}` — one podcast and its season groups. +//! * `GET /feeds/{podcast_id}/episodes?season=&played=&q=&sort=&limit=&offset=` +//! * `POST /feeds/{podcast_id}/subscribe`, `POST /feeds/{podcast_id}/unsubscribe` +//! (unsubscribing keeps progress and history). +//! * `POST /episodes/{episode_id}/restart|played|unplayed` — explicit +//! progress actions. +//! * `POST /{podcast_id}/refresh` — manual feed refresh of a subscribed +//! podcast (ignores the failure backoff, at most every 30 s). +//! +//! Web player: +//! +//! * `POST /episodes/{episode_id}/playback` — audio source, resume position +//! and (when playing starts) a new listening session. +//! * `POST /episodes/{episode_id}/progress` — checkpoints, pause, seek, +//! stop, handoff and end of stream of that session (also sent with +//! `navigator.sendBeacon` when the page is hidden). +//! * `GET /episodes/{episode_id}/source` — how the browser gets the audio: +//! the publisher URL (default, the browser streams it directly) and the +//! proxy URL when the administrator enabled the proxy. +//! * `GET /episodes/{episode_id}/audio` — the optional audio proxy +//! (`podcast_audio_proxy_enabled`). It takes an episode id, never a URL, +//! and only serves episodes of the signed-in account. + +use std::sync::Arc; + +use cot::db::Database; +use cot::http::StatusCode; +use cot::http::header::{ + ACCEPT_RANGES, CACHE_CONTROL, CONTENT_LENGTH, CONTENT_RANGE, CONTENT_TYPE, RANGE, RETRY_AFTER, +}; +use cot::json::Json; +use cot::request::extractors::{Path, UrlQuery}; +use cot::response::IntoResponse; +use cot::router::method::{get, post}; +use cot::router::{Route, Router}; +use cot::session::Session; +use cot::{App, Body}; +use futures_util::TryStreamExt; +use music_dht::podcast::{EpisodeId, PodcastId}; +use serde::{Deserialize, Serialize}; + +use super::api::{self, ApiError, EpisodeAction, EpisodeListParams}; +use super::fetch::FeedFetcher; +use super::limits::{self, ConcurrencyLimit, Permit}; +use super::proxy::AudioProxy; +use super::runner::{self, ManualRefreshError, TargetResult}; +use super::{PodcastStore, now_ms}; +use crate::auth; +use crate::config::AppConfig; + +fn json_error(status: StatusCode, message: &str) -> cot::response::Response { + let body = serde_json::json!({ "error": message }); + cot::http::Response::builder() + .status(status) + .header(CONTENT_TYPE, "application/json") + .header(CACHE_CONTROL, "private, no-store") + .body(Body::fixed(body.to_string())) + .expect("valid response") +} + +fn internal(error: impl std::fmt::Display) -> cot::Error { + cot::Error::internal(error.to_string()) +} + +fn api_error(error: ApiError) -> cot::response::Response { + json_error( + StatusCode::from_u16(error.status).unwrap_or(StatusCode::INTERNAL_SERVER_ERROR), + &error.message, + ) +} + +/// JSON response of an [`api`] result; personal data is never cached. +fn api_response(result: api::ApiResult) -> cot::Result { + match result { + Ok(value) => { + let mut response = Json(value).into_response()?; + response + .headers_mut() + .insert(CACHE_CONTROL, "private, no-store".parse().expect("static")); + Ok(response) + } + Err(error) => Ok(api_error(error)), + } +} + +/// The signed-in account (session or bearer token). +async fn signed_in(auth_ctx: &auth::AuthContext, session: &Session, db: &Database) -> Option { + auth::get_request_user(auth_ctx, session, db) + .await + .map(|user| user.id) +} + +fn unauthorized() -> cot::response::Response { + json_error(StatusCode::UNAUTHORIZED, "not authenticated") +} + +/// 429 when an account or the server has too many requests of a kind in +/// flight (see [`limits`]). +fn too_busy(message: &str) -> cot::response::Response { + let mut response = json_error(StatusCode::TOO_MANY_REQUESTS, message); + response.headers_mut().insert( + RETRY_AFTER, + limits::RETRY_AFTER_SECS + .to_string() + .parse() + .expect("digits"), + ); + response +} + +const FEEDS_BUSY: &str = "too many feed requests in progress; try again in a few seconds"; + +/// Largest JSON body accepted (a playback request carries one episode reference). +const MAX_JSON_BODY: usize = 64 * 1024; + +/// Whether a `Content-Type` value is `application/json` (parameters such +/// as `charset` allowed). +fn is_json_content_type(value: Option<&str>) -> bool { + value + .and_then(|value| value.split(';').next()) + .is_some_and(|essence| essence.trim().eq_ignore_ascii_case("application/json")) +} + +/// Request body as JSON (`{}` when empty); malformed JSON is a 400. +/// +/// A body must be sent as `application/json` (415 otherwise), like cot's +/// `Json` extractor requires on the other player endpoints: a cross-origin +/// page can only send that type after a CORS preflight, while `text/plain` +/// and form bodies need none and would carry the session cookie from a +/// same-site page. `navigator.sendBeacon` sends a Blob typed +/// `application/json`. +async fn json_body(request: cot::request::Request) -> Result { + let json = is_json_content_type( + request + .headers() + .get(CONTENT_TYPE) + .and_then(|value| value.to_str().ok()), + ); + let bytes = request + .into_body() + .into_bytes_limited(MAX_JSON_BODY) + .await + .map_err(|error| ApiError::new(400, format!("unreadable body: {error}")))?; + if bytes.iter().all(u8::is_ascii_whitespace) { + return Ok(serde_json::Value::Object(Default::default())); + } + if !json { + return Err(ApiError::new(415, "the body must be application/json")); + } + serde_json::from_slice(&bytes).map_err(|error| ApiError::new(400, error.to_string())) +} + +/// Shared state of the handlers. +#[derive(Clone)] +struct State { + config: Arc, + pool: Arc>, + fetcher: Arc>, + proxy: Arc>, + /// Feed downloads users started ("add RSS", manual refresh). + feed_fetches: ConcurrencyLimit, + /// Audio proxy responses being streamed. + proxy_streams: ConcurrencyLimit, +} + +impl State { + fn feed_fetch(&self, user_id: i64) -> Option { + self.feed_fetches.try_acquire(user_id) + } + + async fn store(&self) -> cot::Result { + let pool = self + .pool + .get_or_try_init(|| async { + sqlx::postgres::PgPoolOptions::new() + .max_connections(5) + .connect(&self.config.database_url) + .await + }) + .await + .map_err(internal)?; + Ok(PodcastStore::new(pool.clone())) + } + + async fn fetcher(&self) -> cot::Result<&FeedFetcher> { + self.fetcher + .get_or_try_init(|| async { FeedFetcher::new() }) + .await + .map_err(internal) + } + + async fn proxy(&self) -> cot::Result<&AudioProxy> { + self.proxy + .get_or_try_init(|| async { AudioProxy::new() }) + .await + .map_err(internal) + } +} + +/// The podcast runtime endpoints (see the module documentation). +pub(crate) struct PodcastsApp { + state: State, +} + +impl PodcastsApp { + pub(crate) fn new(config: Arc) -> Self { + Self { + state: State { + config, + pool: Arc::new(tokio::sync::OnceCell::new()), + fetcher: Arc::new(tokio::sync::OnceCell::new()), + proxy: Arc::new(tokio::sync::OnceCell::new()), + feed_fetches: ConcurrencyLimit::new( + limits::FEED_FETCHES_PER_ACCOUNT, + limits::FEED_FETCHES_TOTAL, + ), + proxy_streams: ConcurrencyLimit::new( + limits::PROXY_STREAMS_PER_ACCOUNT, + limits::PROXY_STREAMS_TOTAL, + ), + }, + } + } +} + +#[derive(Debug, Deserialize)] +struct PodcastPath { + podcast_id: String, +} + +#[derive(Debug, Deserialize)] +struct EpisodePath { + episode_id: String, +} + +/// Path of the proxy endpoint for an episode. +pub(crate) fn proxy_path(episode_id: &EpisodeId) -> String { + format!("/api/podcasts/episodes/{episode_id}/audio") +} + +#[derive(Debug, Serialize)] +struct RefreshResponse { + status: &'static str, + new_episodes: usize, + updated_episodes: usize, + unchanged_episodes: usize, + skipped_items: usize, + feed_url: Option, + error: Option, + next_check_at_ms: Option, +} + +impl RefreshResponse { + fn from_result(result: TargetResult) -> Self { + let mut response = Self { + status: "", + new_episodes: 0, + updated_episodes: 0, + unchanged_episodes: 0, + skipped_items: 0, + feed_url: None, + error: None, + next_check_at_ms: None, + }; + match result { + TargetResult::Updated(outcome) => { + response.status = "updated"; + response.new_episodes = outcome.new_episodes; + response.updated_episodes = outcome.updated_episodes; + response.unchanged_episodes = outcome.unchanged_episodes; + response.skipped_items = outcome.skipped_items; + response.feed_url = outcome.feed_url.map(String::from); + } + TargetResult::NotModified => response.status = "not_modified", + TargetResult::Failed { + error, + next_check_at_ms, + .. + } => { + response.status = "failed"; + response.error = Some(error); + response.next_check_at_ms = Some(next_check_at_ms); + } + TargetResult::Skipped(reason) => { + response.status = "skipped"; + response.error = Some(reason); + } + } + response + } +} + +async fn refresh_handler( + state: State, + auth_ctx: auth::AuthContext, + session: Session, + db: Database, + path: Path, +) -> cot::Result { + let Some(user) = auth::get_request_user(&auth_ctx, &session, &db).await else { + return Ok(json_error(StatusCode::UNAUTHORIZED, "not authenticated")); + }; + let Ok(podcast_id) = PodcastId::parse(&path.0.podcast_id) else { + return Ok(json_error(StatusCode::BAD_REQUEST, "invalid podcast id")); + }; + let Some(_permit) = state.feed_fetch(user.id) else { + return Ok(too_busy(FEEDS_BUSY)); + }; + let store = state.store().await?; + let fetcher = state.fetcher().await?; + match runner::refresh_now(&store, fetcher, user.id, &podcast_id, now_ms()).await { + Ok(result) => Json(RefreshResponse::from_result(result)).into_response(), + Err(error) => match error.downcast_ref::() { + Some(ManualRefreshError::UnknownPodcast) => { + Ok(json_error(StatusCode::NOT_FOUND, "unknown podcast")) + } + Some(ManualRefreshError::NotSubscribed) => { + Ok(json_error(StatusCode::CONFLICT, &error.to_string())) + } + Some(ManualRefreshError::TooSoon { retry_after_ms }) => { + let mut response = json_error(StatusCode::TOO_MANY_REQUESTS, &error.to_string()); + response.headers_mut().insert( + RETRY_AFTER, + ((retry_after_ms + 999) / 1000) + .to_string() + .parse() + .expect("digits"), + ); + Ok(response) + } + None => Err(internal(format!("{error:#}"))), + }, + } +} + +#[derive(Debug, Serialize)] +struct SourceResponse { + episode_id: String, + /// Publisher URL; the browser streams it directly by default. + direct_url: String, + /// Same-origin proxy path, only when the proxy is enabled. + proxy_url: Option, + mime: Option, +} + +async fn source_handler( + state: State, + auth_ctx: auth::AuthContext, + session: Session, + db: Database, + path: Path, +) -> cot::Result { + let Some(user) = auth::get_request_user(&auth_ctx, &session, &db).await else { + return Ok(json_error(StatusCode::UNAUTHORIZED, "not authenticated")); + }; + let Ok(episode_id) = EpisodeId::parse(&path.0.episode_id) else { + return Ok(json_error(StatusCode::BAD_REQUEST, "invalid episode id")); + }; + let store = state.store().await?; + let Some((audio_url, mime)) = store + .audio_source(user.id, &episode_id) + .await + .map_err(internal)? + else { + return Ok(json_error(StatusCode::NOT_FOUND, "unknown episode")); + }; + let (config, _) = AppConfig::load_with_db(&db).await; + let mut response = Json(SourceResponse { + proxy_url: config + .podcast_audio_proxy_enabled + .then(|| proxy_path(&episode_id)), + episode_id: episode_id.to_string(), + direct_url: audio_url, + mime, + }) + .into_response()?; + response + .headers_mut() + .insert(CACHE_CONTROL, "private, no-store".parse().expect("static")); + Ok(response) +} + +async fn audio_handler( + state: State, + auth_ctx: auth::AuthContext, + session: Session, + db: Database, + path: Path, + request: cot::request::Request, +) -> cot::Result { + let Some(user) = auth::get_request_user(&auth_ctx, &session, &db).await else { + return Ok(json_error(StatusCode::UNAUTHORIZED, "not authenticated")); + }; + let (config, _) = AppConfig::load_with_db(&db).await; + if !config.podcast_audio_proxy_enabled { + return Ok(json_error( + StatusCode::NOT_FOUND, + "the podcast audio proxy is disabled; stream from the publisher URL", + )); + } + let Ok(episode_id) = EpisodeId::parse(&path.0.episode_id) else { + return Ok(json_error(StatusCode::BAD_REQUEST, "invalid episode id")); + }; + let store = state.store().await?; + // The URL comes from this account's catalog, never from the request. + let Some((audio_url, _)) = store + .audio_source(user.id, &episode_id) + .await + .map_err(internal)? + else { + return Ok(json_error(StatusCode::NOT_FOUND, "unknown episode")); + }; + let range = request + .headers() + .get(RANGE) + .and_then(|value| value.to_str().ok()); + // Held until the body is fully sent or the browser goes away. + let Some(permit) = state.proxy_streams.try_acquire(user.id) else { + return Ok(too_busy("too many podcast audio streams in progress")); + }; + let proxy = state.proxy().await?; + let audio = match proxy.fetch(&audio_url, range).await { + Ok(audio) => audio, + Err(error) => { + // Neither the URL (access tokens) nor the account is logged. + tracing::info!(episode = %episode_id, "podcast proxy: {error}"); + let status = + StatusCode::from_u16(error.http_status()).unwrap_or(StatusCode::BAD_GATEWAY); + return Ok(json_error(status, &error.public_message())); + } + }; + let mut builder = cot::http::Response::builder() + .status(StatusCode::from_u16(audio.status).unwrap_or(StatusCode::BAD_GATEWAY)) + .header(CACHE_CONTROL, "private, no-store") + .header("X-Content-Type-Options", "nosniff") + .header("Content-Security-Policy", "default-src 'none'; sandbox") + .header("Content-Disposition", "inline"); + if let Some(content_type) = &audio.content_type { + builder = builder.header(CONTENT_TYPE, content_type.as_str()); + } + if let Some(length) = audio.content_length { + builder = builder.header(CONTENT_LENGTH, length.to_string()); + } + if let Some(content_range) = &audio.content_range { + builder = builder.header(CONTENT_RANGE, content_range.as_str()); + } + if audio.accept_ranges { + builder = builder.header(ACCEPT_RANGES, "bytes"); + } + let body = audio.body.map_err(move |error| { + let _held = &permit; + cot::Error::internal(error.to_string()) + }); + builder.body(Body::streaming(body)).map_err(internal) +} + +#[derive(Debug, Deserialize)] +struct ListQuery { + #[serde(default)] + all: Option, +} + +async fn list_handler( + state: State, + auth_ctx: auth::AuthContext, + session: Session, + db: Database, + query: UrlQuery, +) -> cot::Result { + let Some(user_id) = signed_in(&auth_ctx, &session, &db).await else { + return Ok(unauthorized()); + }; + let all = matches!(query.0.all.as_deref(), Some("1" | "true")); + let store = state.store().await?; + api_response(api::list_podcasts(&store, user_id, all).await) +} + +async fn subscribe_url_handler( + state: State, + auth_ctx: auth::AuthContext, + session: Session, + db: Database, + request: cot::request::Request, +) -> cot::Result { + let Some(user_id) = signed_in(&auth_ctx, &session, &db).await else { + return Ok(unauthorized()); + }; + let body = match json_body(request).await { + Ok(body) => body, + Err(error) => return Ok(api_error(error)), + }; + let Some(_permit) = state.feed_fetch(user_id) else { + return Ok(too_busy(FEEDS_BUSY)); + }; + let store = state.store().await?; + let fetcher = state.fetcher().await?; + api_response(api::subscribe_url(&store, fetcher, user_id, body, now_ms()).await) +} + +/// What a `/feeds/{podcast_id}...` route does. +#[derive(Debug, Clone, Copy)] +enum PodcastRoute { + Detail, + Subscribe, + Unsubscribe, +} + +async fn podcast_handler( + state: State, + route: PodcastRoute, + auth_ctx: auth::AuthContext, + session: Session, + db: Database, + path: Path, +) -> cot::Result { + let Some(user_id) = signed_in(&auth_ctx, &session, &db).await else { + return Ok(unauthorized()); + }; + let podcast_id = match api::parse_podcast_id(&path.0.podcast_id) { + Ok(id) => id, + Err(error) => return Ok(api_error(error)), + }; + let store = state.store().await?; + match route { + PodcastRoute::Detail => { + api_response(api::podcast_detail(&store, user_id, &podcast_id).await) + } + PodcastRoute::Subscribe => { + api_response(api::resubscribe(&store, user_id, &podcast_id).await) + } + PodcastRoute::Unsubscribe => { + api_response(api::unsubscribe(&store, user_id, &podcast_id).await) + } + } +} + +async fn episodes_handler( + state: State, + auth_ctx: auth::AuthContext, + session: Session, + db: Database, + path: Path, + query: UrlQuery, +) -> cot::Result { + let Some(user_id) = signed_in(&auth_ctx, &session, &db).await else { + return Ok(unauthorized()); + }; + let podcast_id = match api::parse_podcast_id(&path.0.podcast_id) { + Ok(id) => id, + Err(error) => return Ok(api_error(error)), + }; + let store = state.store().await?; + api_response(api::episodes(&store, user_id, &podcast_id, &query.0).await) +} + +/// What a `/episodes/{episode_id}/...` POST route does. +#[derive(Debug, Clone, Copy)] +enum EpisodeRoute { + Action(EpisodeAction), + Playback, + Progress, +} + +async fn episode_handler( + state: State, + route: EpisodeRoute, + auth_ctx: auth::AuthContext, + session: Session, + db: Database, + path: Path, + request: cot::request::Request, +) -> cot::Result { + let Some(user_id) = signed_in(&auth_ctx, &session, &db).await else { + return Ok(unauthorized()); + }; + let episode_id = match api::parse_episode_id(&path.0.episode_id) { + Ok(id) => id, + Err(error) => return Ok(api_error(error)), + }; + let store = state.store().await?; + match route { + EpisodeRoute::Action(action) => { + api_response(api::episode_action(&store, user_id, &episode_id, action).await) + } + EpisodeRoute::Playback => { + let body = match json_body(request).await { + Ok(body) => body, + Err(error) => return Ok(api_error(error)), + }; + let (config, _) = AppConfig::load_with_db(&db).await; + api_response( + api::playback( + &store, + user_id, + &episode_id, + body, + config.podcast_audio_proxy_enabled, + ) + .await, + ) + } + EpisodeRoute::Progress => { + let body = match json_body(request).await { + Ok(body) => body, + Err(error) => return Ok(api_error(error)), + }; + api_response(api::progress(&store, user_id, &episode_id, body, now_ms()).await) + } + } +} + +impl App for PodcastsApp { + fn name(&self) -> &'static str { + "podcasts" + } + + fn router(&self) -> Router { + let refresh_state = self.state.clone(); + let source_state = self.state.clone(); + let audio_state = self.state.clone(); + let list_state = self.state.clone(); + let subscribe_state = self.state.clone(); + let episodes_state = self.state.clone(); + let podcast_route = |route: PodcastRoute| { + let state = self.state.clone(); + move |auth_ctx: auth::AuthContext, + session: Session, + db: Database, + path: Path| { + podcast_handler(state.clone(), route, auth_ctx, session, db, path) + } + }; + let episode_route = |route: EpisodeRoute| { + let state = self.state.clone(); + move |auth_ctx: auth::AuthContext, + session: Session, + db: Database, + path: Path, + request: cot::request::Request| { + episode_handler(state.clone(), route, auth_ctx, session, db, path, request) + } + }; + // Routes are matched in order: literal prefixes come before + // `/{podcast_id}/refresh`. + Router::with_urls([ + Route::with_handler_and_name( + "/feeds", + get( + move |auth_ctx: auth::AuthContext, + session: Session, + db: Database, + query: UrlQuery| { + list_handler(list_state.clone(), auth_ctx, session, db, query) + }, + ) + .post( + move |auth_ctx: auth::AuthContext, + session: Session, + db: Database, + request: cot::request::Request| { + subscribe_url_handler( + subscribe_state.clone(), + auth_ctx, + session, + db, + request, + ) + }, + ), + "podcasts_feeds", + ), + Route::with_handler_and_name( + "/feeds/{podcast_id}/episodes", + get( + move |auth_ctx: auth::AuthContext, + session: Session, + db: Database, + path: Path, + query: UrlQuery| { + episodes_handler(episodes_state.clone(), auth_ctx, session, db, path, query) + }, + ), + "podcasts_episodes", + ), + Route::with_handler_and_name( + "/feeds/{podcast_id}/subscribe", + post(podcast_route(PodcastRoute::Subscribe)), + "podcasts_subscribe", + ), + Route::with_handler_and_name( + "/feeds/{podcast_id}/unsubscribe", + post(podcast_route(PodcastRoute::Unsubscribe)), + "podcasts_unsubscribe", + ), + Route::with_handler_and_name( + "/feeds/{podcast_id}", + get(podcast_route(PodcastRoute::Detail)), + "podcasts_detail", + ), + Route::with_handler_and_name( + "/episodes/{episode_id}/playback", + post(episode_route(EpisodeRoute::Playback)), + "podcasts_episode_playback", + ), + Route::with_handler_and_name( + "/episodes/{episode_id}/progress", + post(episode_route(EpisodeRoute::Progress)), + "podcasts_episode_progress", + ), + Route::with_handler_and_name( + "/episodes/{episode_id}/restart", + post(episode_route(EpisodeRoute::Action(EpisodeAction::Restart))), + "podcasts_episode_restart", + ), + Route::with_handler_and_name( + "/episodes/{episode_id}/played", + post(episode_route(EpisodeRoute::Action( + EpisodeAction::MarkPlayed, + ))), + "podcasts_episode_played", + ), + Route::with_handler_and_name( + "/episodes/{episode_id}/unplayed", + post(episode_route(EpisodeRoute::Action( + EpisodeAction::MarkUnplayed, + ))), + "podcasts_episode_unplayed", + ), + Route::with_handler_and_name( + "/episodes/{episode_id}/source", + get( + move |auth_ctx: auth::AuthContext, + session: Session, + db: Database, + path: Path| { + source_handler(source_state.clone(), auth_ctx, session, db, path) + }, + ), + "podcasts_episode_source", + ), + Route::with_handler_and_name( + "/episodes/{episode_id}/audio", + get( + move |auth_ctx: auth::AuthContext, + session: Session, + db: Database, + path: Path, + request: cot::request::Request| { + audio_handler(audio_state.clone(), auth_ctx, session, db, path, request) + }, + ), + "podcasts_episode_audio", + ), + Route::with_handler_and_name( + "/{podcast_id}/refresh", + post( + move |auth_ctx: auth::AuthContext, + session: Session, + db: Database, + path: Path| { + refresh_handler(refresh_state.clone(), auth_ctx, session, db, path) + }, + ), + "podcasts_refresh", + ), + ]) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn request(content_type: Option<&str>, body: &'static str) -> cot::request::Request { + let mut builder = cot::http::Request::builder() + .method("POST") + .uri("/api/podcasts/feeds"); + if let Some(value) = content_type { + builder = builder.header(CONTENT_TYPE, value); + } + builder.body(Body::fixed(body)).unwrap() + } + + #[tokio::test] + async fn json_bodies_need_the_json_content_type() { + let url = r#"{"url": "https://podcasts.example/feed.xml"}"#; + for content_type in [ + "application/json", + "application/json; charset=UTF-8", + "Application/JSON", + ] { + let body = json_body(request(Some(content_type), url)).await.unwrap(); + assert_eq!(body["url"], "https://podcasts.example/feed.xml"); + } + // What a page can send cross-origin without a CORS preflight. + for content_type in [ + Some("text/plain"), + Some("text/plain;charset=UTF-8"), + Some("application/x-www-form-urlencoded"), + Some("multipart/form-data; boundary=x"), + None, + ] { + let error = json_body(request(content_type, url)).await.unwrap_err(); + assert_eq!(error.status, 415, "{content_type:?}"); + } + // An empty body (a plain POST) is an empty object whatever the type. + let empty = json_body(request(None, "")).await.unwrap(); + assert_eq!(empty, serde_json::json!({})); + let malformed = json_body(request(Some("application/json"), "{")) + .await + .unwrap_err(); + assert_eq!(malformed.status, 400); + } +} diff --git a/src/podcasts/limits.rs b/src/podcasts/limits.rs new file mode 100644 index 0000000..d894a8b --- /dev/null +++ b/src/podcasts/limits.rs @@ -0,0 +1,123 @@ +//! Bounds on the outbound work signed-in users can start: feed downloads +//! they ask for ("add RSS", manual refresh) and audio proxy streams. Each +//! kind has a per-account and a server-wide limit on requests in flight, so +//! one account cannot tie up the server's memory, connections or bandwidth. +//! Scheduled refreshes are bounded by the job itself (`runner::RUN_LIMIT`, +//! `runner::RUN_CONCURRENCY`, accounts in turns). + +use std::collections::HashMap; +use std::sync::{Arc, Mutex}; + +/// Feed downloads one account may run at once ("add RSS", refresh). +pub(crate) const FEED_FETCHES_PER_ACCOUNT: usize = 2; +/// Feed downloads started by users, server-wide (each may buffer up to +/// `fetch::MAX_FEED_BYTES`). +pub(crate) const FEED_FETCHES_TOTAL: usize = 16; +/// Audio proxy responses one account may stream at once (a player opens a +/// new range request when seeking while the old one is still closing). +pub(crate) const PROXY_STREAMS_PER_ACCOUNT: usize = 4; +/// Audio proxy responses streamed at once, server-wide. +pub(crate) const PROXY_STREAMS_TOTAL: usize = 64; +/// Seconds a refused client is asked to wait (`Retry-After`). +pub(crate) const RETRY_AFTER_SECS: u64 = 5; + +#[derive(Debug, Default)] +struct Counts { + total: usize, + accounts: HashMap, +} + +/// Requests in flight per account and in total. +#[derive(Debug, Clone)] +pub(crate) struct ConcurrencyLimit { + per_account: usize, + total: usize, + counts: Arc>, +} + +impl ConcurrencyLimit { + pub(crate) fn new(per_account: usize, total: usize) -> Self { + Self { + per_account, + total, + counts: Arc::default(), + } + } + + /// A slot for one request of `user_id`, released when the permit is + /// dropped; `None` when the account or the server is at its limit. + pub(crate) fn try_acquire(&self, user_id: i64) -> Option { + let mut counts = self + .counts + .lock() + .unwrap_or_else(|error| error.into_inner()); + let account = counts.accounts.get(&user_id).copied().unwrap_or(0); + if account >= self.per_account || counts.total >= self.total { + return None; + } + counts.total += 1; + counts.accounts.insert(user_id, account + 1); + Some(Permit { + user_id, + counts: Arc::clone(&self.counts), + }) + } + + #[cfg(test)] + fn in_flight(&self, user_id: i64) -> (usize, usize) { + let counts = self.counts.lock().unwrap(); + ( + counts.accounts.get(&user_id).copied().unwrap_or(0), + counts.total, + ) + } +} + +/// One request in flight (see [`ConcurrencyLimit::try_acquire`]). +#[derive(Debug)] +pub(crate) struct Permit { + user_id: i64, + counts: Arc>, +} + +impl Drop for Permit { + fn drop(&mut self) { + let mut counts = self + .counts + .lock() + .unwrap_or_else(|error| error.into_inner()); + counts.total = counts.total.saturating_sub(1); + if let Some(account) = counts.accounts.get_mut(&self.user_id) { + *account = account.saturating_sub(1); + if *account == 0 { + counts.accounts.remove(&self.user_id); + } + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn accounts_and_the_server_are_bounded_until_permits_are_dropped() { + let limit = ConcurrencyLimit::new(2, 3); + let a1 = limit.try_acquire(1).unwrap(); + let a2 = limit.try_acquire(1).unwrap(); + assert!(limit.try_acquire(1).is_none(), "per account"); + let b1 = limit.try_acquire(2).unwrap(); + assert!(limit.try_acquire(3).is_none(), "server-wide"); + assert_eq!(limit.in_flight(1), (2, 3)); + drop(a1); + assert_eq!(limit.in_flight(1), (1, 2)); + let a3 = limit.try_acquire(1).unwrap(); + drop((a2, a3, b1)); + assert_eq!(limit.in_flight(1), (0, 0)); + assert!(limit.counts.lock().unwrap().accounts.is_empty()); + // Clones share the counts. + let clone = limit.clone(); + let _held = clone.try_acquire(4).unwrap(); + assert_eq!(limit.in_flight(4), (1, 1)); + } +} diff --git a/src/podcasts/mod.rs b/src/podcasts/mod.rs new file mode 100644 index 0000000..228e32c --- /dev/null +++ b/src/podcasts/mod.rs @@ -0,0 +1,327 @@ +//! PostgreSQL podcast store of the web server. +//! +//! [`PodcastStore`] implements the podcast contract of `music_dht::podcast` +//! (normative description: `PODCASTS.md` of frid) on PostgreSQL, with the +//! same invariants as the SQLite reference store of `furumi-library`. Each +//! web account is one virtual trusted device, so every table is keyed by +//! `user_id`. The store owns no HTTP, scheduling or UI: RSS fetching and the +//! player use it. +//! +//! # Tables +//! +//! | Table | Key | Content | Synced | +//! | --- | --- | --- | --- | +//! | `furumusic__podcast` | `podcast_id` | local channel metadata, `feed_url` = URL this account fetches | no (catalog) | +//! | `furumusic__podcast_feed_alias` | `(podcast_id, feed_url)` | bounded grow-only alias set | yes, inside subscription records and `podcast_feed_aliases_added` | +//! | `furumusic__podcast_episode` | `episode_id` | `EpisodeMeta`, `source` (`feed`/`sync`), `in_feed` | no (catalog) | +//! | `furumusic__podcast_season` (view) | `(podcast_id, season)` | derived from the episodes | no | +//! | `furumusic__podcast_subscription` | `podcast_id` | subscription register | yes | +//! | `furumusic__podcast_share` | `podcast_id` | share register | yes | +//! | `furumusic__podcast_progress` | `episode_id` | progress register | yes | +//! | `furumusic__podcast_listen_session` | `session_id` | grow-only set of sessions | yes | +//! | `furumusic__podcast_fetch_state` | `podcast_id` | ETag, Last-Modified, check times, errors, cover cache | **never** | +//! +//! # One apply path +//! +//! Remote operations ([`PodcastStore::apply_remote_op`]), snapshot sections +//! ([`PodcastStore::apply_snapshot`]) and local actions all end in the same +//! functions: the `SyncOpWire` materializers of `music_dht::device_sync` and +//! the `merge_*` rules of `music_dht::podcast`. Applying is therefore +//! idempotent and independent of delivery order, and a snapshot never +//! replaces newer state. +//! +//! Local actions ([`PodcastStore::subscribe`], [`PodcastStore::seek`], ...) +//! run in one transaction: they read the registers, append the operation to +//! the account's device-sync log with its stamp +//! ([`crate::federation::devices::append_local_op`]: next `seq`, `hlc_ms`, +//! `op_id = :`) and apply exactly that operation. Either both +//! the log and the registers change or neither does. Every write +//! transaction takes a per-account advisory lock, so concurrent writers of +//! one account serialize. +//! +//! RSS refreshes ([`PodcastStore::apply_feed`]) touch the catalog and the +//! alias set only; they never write a register, progress or a session. +//! +//! # Runtime +//! +//! * `net` — the SSRF-guarded HTTP client used for every feed and audio +//! request (public addresses only, checked per connection and per +//! redirect hop). +//! * `fetch` and [`runner`] — conditional feed downloads and their schedule +//! (hourly checks, exponential backoff, one request per feed URL), run by +//! the `podcast_refresh` job and the manual refresh endpoint. +//! * `proxy` and [`http`] — the optional audio proxy and the +//! `/api/podcasts` endpoints. Audio is streamed from the publisher, never +//! stored. +//! * [`api`] — what the endpoints of the podcast UI and the web player do +//! (subscribe by URL, lists, explicit progress actions, playback sessions +//! and progress reports), independent of HTTP. + +mod actions; +pub(crate) mod api; +mod apply; +mod fetch; +pub(crate) mod http; +mod limits; +mod net; +mod proxy; +mod queries; +mod refresh; +pub(crate) mod runner; +pub(crate) mod schema; + +#[cfg(test)] +mod fetch_tests; +#[cfg(test)] +mod net_tests; +#[cfg(test)] +pub(crate) mod postgres_tests; +#[cfg(test)] +mod proxy_tests; +#[cfg(test)] +pub(crate) mod test_server; +#[cfg(test)] +mod tests; + +use std::fmt; + +use anyhow::Result; +use music_dht::podcast::{ + EpisodeId, FeedUrl, ListenSession, PodcastError, PodcastId, PodcastMeta, ProgressRecord, + ProgressStatus, +}; +use sqlx::Row; +use sqlx::postgres::PgRow; + +pub(crate) use actions::{CheckpointOutcome, ProgressSession}; +pub(crate) use apply::ApplyOutcome; +pub(crate) use fetch::FeedFetcher; +pub(crate) use queries::{ + EpisodeListItem, EpisodeQuery, EpisodeSort, PlayedFilter, PodcastListFilter, PodcastSummary, + SeasonFilter, SeasonSummary, +}; +pub(crate) use refresh::{FeedRefresh, FetchState, RefreshOutcome}; + +/// Typed errors of [`PodcastStore`]; other failures are database errors. +/// +/// They are returned inside `anyhow::Error`; use `downcast_ref` to match. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) enum PodcastStoreError { + /// The podcast is unknown to this account. + UnknownPodcast(PodcastId), + /// The action needs an active subscription. + NotSubscribed(PodcastId), + /// Input violates the podcast contract. + Invalid(String), + /// A new subscription would exceed [`MAX_SUBSCRIPTIONS_PER_ACCOUNT`]. + SubscriptionLimit(usize), + /// A new episode reference from the browser would exceed + /// [`MAX_REFERENCE_EPISODES_PER_ACCOUNT`]. + ReferenceLimit(usize), +} + +/// Active subscriptions an account may add locally (subscriptions synced +/// from the account's own devices are always applied). Every subscription +/// is a feed the server downloads about once an hour. +pub(crate) const MAX_SUBSCRIPTIONS_PER_ACCOUNT: usize = 1000; + +/// Episodes known only from references (`source = 'sync'`: not from one of +/// the account's feeds) beyond which the browser cannot add another one +/// through `POST /api/podcasts/episodes/{id}/playback`. References in +/// operations of the account's own devices are always applied; this bounds +/// what a signed-in user can insert directly, one request at a time. +pub(crate) const MAX_REFERENCE_EPISODES_PER_ACCOUNT: usize = 10_000; + +impl fmt::Display for PodcastStoreError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::UnknownPodcast(id) => write!(f, "unknown podcast {id}"), + Self::NotSubscribed(id) => write!(f, "not subscribed to podcast {id}"), + Self::Invalid(reason) => write!(f, "invalid podcast data: {reason}"), + Self::SubscriptionLimit(limit) => { + write!(f, "subscription limit reached ({limit} podcasts)") + } + Self::ReferenceLimit(limit) => write!( + f, + "too many episodes from outside your feeds ({limit}); subscribe to the podcast to play it" + ), + } + } +} + +impl std::error::Error for PodcastStoreError {} + +impl From for PodcastStoreError { + fn from(error: PodcastError) -> Self { + Self::Invalid(error.to_string()) + } +} + +pub(crate) fn invalid(reason: impl ToString) -> anyhow::Error { + PodcastStoreError::Invalid(reason.to_string()).into() +} + +/// PostgreSQL store of podcast catalog, personal state and fetch state. +/// +/// Cheap to clone (the pool is shared). See the module documentation. +#[derive(Debug, Clone)] +pub(crate) struct PodcastStore { + pool: sqlx::PgPool, +} + +impl PodcastStore { + pub(crate) fn new(pool: sqlx::PgPool) -> Self { + Self { pool } + } + + /// The pool the store runs on. + pub(crate) fn pool(&self) -> &sqlx::PgPool { + &self.pool + } + + /// Starts a write transaction holding the account's podcast lock. + async fn begin_write( + &self, + user_id: i64, + ) -> Result> { + let mut tx = self.pool.begin().await?; + lock_account(&mut tx, user_id).await?; + Ok(tx) + } +} + +/// Serializes podcast writers of one account until the transaction ends. +pub(crate) async fn lock_account(conn: &mut sqlx::PgConnection, user_id: i64) -> Result<()> { + sqlx::query( + "SELECT pg_advisory_xact_lock(hashtextextended('furumusic:podcasts:' || $1::text, 0))", + ) + .bind(user_id) + .execute(conn) + .await?; + Ok(()) +} + +pub(crate) fn now_ms() -> i64 { + chrono::Utc::now().timestamp_millis() +} + +/// Converts a contract generation to the `BIGINT` column value. +/// +/// Generations above `i64::MAX` cannot be stored; records carrying one are +/// rejected (no conforming client produces them). +pub(crate) fn generation_to_sql(generation: u64) -> Result { + i64::try_from(generation) + .map_err(|_| PodcastStoreError::Invalid("generation exceeds i64::MAX".to_string())) +} + +pub(crate) fn generation_from_sql(value: i64) -> u64 { + u64::try_from(value).unwrap_or(0) +} + +pub(crate) fn status_to_sql(status: ProgressStatus) -> &'static str { + match status { + ProgressStatus::NotStarted => "not_started", + ProgressStatus::InProgress => "in_progress", + ProgressStatus::Completed => "completed", + } +} + +pub(crate) fn status_from_sql(value: &str) -> ProgressStatus { + match value { + "in_progress" => ProgressStatus::InProgress, + "completed" => ProgressStatus::Completed, + _ => ProgressStatus::NotStarted, + } +} + +/// `u32` contract numbers (season, episode number) from a `BIGINT` column. +pub(crate) fn u32_from_sql(value: Option) -> Option { + value.and_then(|value| u32::try_from(value).ok()) +} + +fn decode_error(message: impl ToString) -> sqlx::Error { + sqlx::Error::Decode(Box::new(PodcastStoreError::Invalid(message.to_string()))) +} + +/// Parses a stored id; stored ids are validated on write. +pub(crate) fn podcast_id_col(row: &PgRow, column: &str) -> sqlx::Result { + PodcastId::try_from(row.try_get::(column)?).map_err(decode_error) +} + +pub(crate) fn episode_id_col(row: &PgRow, column: &str) -> sqlx::Result { + EpisodeId::try_from(row.try_get::(column)?).map_err(decode_error) +} + +pub(crate) fn feed_url_col(row: &PgRow, column: &str) -> sqlx::Result { + FeedUrl::try_from(row.try_get::(column)?).map_err(decode_error) +} + +pub(crate) fn podcast_meta_col(row: &PgRow, column: &str) -> sqlx::Result { + serde_json::from_value(row.try_get::(column)?).map_err(decode_error) +} + +/// Progress register columns of table alias `alias`, named `{prefix}...`. +pub(crate) fn progress_columns(alias: &str, prefix: &str) -> String { + [ + "episode_id", + "podcast_id", + "position_ms", + "status", + "generation", + "session_id", + "hlc_ms", + "device_id", + "op_id", + ] + .iter() + .map(|column| format!("{alias}.{column} AS {prefix}{column}")) + .collect::>() + .join(", ") +} + +/// Progress register selected with [`progress_columns`]. +pub(crate) fn progress_from_row(row: &PgRow, prefix: &str) -> sqlx::Result { + let column = |name: &str| format!("{prefix}{name}"); + Ok(ProgressRecord { + episode_id: episode_id_col(row, &column("episode_id"))?, + podcast_id: podcast_id_col(row, &column("podcast_id"))?, + position_ms: row.try_get(column("position_ms").as_str())?, + status: status_from_sql(&row.try_get::(column("status").as_str())?), + generation: generation_from_sql(row.try_get(column("generation").as_str())?), + session_id: row.try_get(column("session_id").as_str())?, + hlc_ms: row.try_get(column("hlc_ms").as_str())?, + device_id: row.try_get(column("device_id").as_str())?, + op_id: row.try_get(column("op_id").as_str())?, + }) +} + +/// Listening session columns of table alias `s` (plain names). +pub(crate) const SESSION_COLUMNS: &str = "s.session_id, s.episode_id, s.podcast_id, s.device_id, \ + s.started_at_ms, s.ended_at_ms, s.start_position_ms, s.end_position_ms, s.listened_ms, \ + s.completed"; + +pub(crate) fn session_from_row(row: &PgRow) -> sqlx::Result { + Ok(ListenSession { + session_id: row.try_get("session_id")?, + episode_id: episode_id_col(row, "episode_id")?, + podcast_id: podcast_id_col(row, "podcast_id")?, + device_id: row.try_get("device_id")?, + started_at_ms: row.try_get("started_at_ms")?, + ended_at_ms: row.try_get("ended_at_ms")?, + start_position_ms: row.try_get("start_position_ms")?, + end_position_ms: row.try_get("end_position_ms")?, + listened_ms: row.try_get("listened_ms")?, + completed: row.try_get("completed")?, + }) +} + +/// Normalized search text of an episode: `normalize_name` of the title and +/// of the description, separated by a newline (the normalization never +/// produces one, so a needle cannot match across the two). +pub(crate) fn episode_search_norm(title: &str, description: Option<&str>) -> String { + format!( + "{}\n{}", + music_dht::normalize_name(title), + music_dht::normalize_name(description.unwrap_or_default()) + ) +} diff --git a/src/podcasts/net.rs b/src/podcasts/net.rs new file mode 100644 index 0000000..4bc7a3c --- /dev/null +++ b/src/podcasts/net.rs @@ -0,0 +1,629 @@ +//! Outbound HTTP for podcast feeds and the audio proxy, guarded against SSRF. +//! +//! Feed URLs and enclosure URLs come from users and from third-party feed +//! documents, so the server must never be tricked into requesting its own +//! network. Every request made through [`SafeHttp`]: +//! +//! * accepts only `http` and `https` URLs with a host and without userinfo; +//! * resolves host names with [`GuardedResolver`], which rejects the whole +//! answer when **any** address is not public (loopback, private, shared +//! CGNAT, link-local, cloud metadata, multicast, unspecified, documentation, +//! reserved; IPv4 and IPv6, including IPv4-mapped, NAT64 and 6to4 forms). +//! The connection is made to exactly the addresses that were checked: the +//! resolver *is* the connector's resolver, there is no separate pre-check +//! that a second DNS answer could bypass (DNS rebinding); +//! * checks IP-literal hosts the same way (the connector does not resolve +//! them); +//! * connects only to the web ports [`ALLOWED_PORTS`] (80, 443, 8080, 8443): +//! a public address can still be the server's own, and its other services +//! (databases, admin ports) are often reachable from the host itself; +//! * follows redirects itself, at most `max_redirects` hops, checking every +//! hop like the first URL; +//! * never uses a system proxy (it would resolve the target instead of the +//! guarded resolver), never stores or sends cookies and never forwards +//! credentials: callers pass the few headers they need explicitly. +//! +//! Loopback can be allowed only in tests (`AddressPolicy::test_allow_loopback` +//! is `#[cfg(test)]`), so the local test servers work and nothing else does. + +use std::fmt; +use std::future::Future; +use std::net::{IpAddr, Ipv4Addr, Ipv6Addr, SocketAddr}; +use std::pin::Pin; +use std::sync::Arc; +use std::time::Duration; + +use reqwest::header::{HeaderMap, LOCATION}; +use reqwest::{StatusCode, Url}; + +/// Why an address is not reachable through [`SafeHttp`]. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum AddressClass { + /// `0.0.0.0/8`, `::`. + Unspecified, + /// `127.0.0.0/8`, `::1`. + Loopback, + /// RFC 1918 and the deprecated IPv6 site-local range. + Private, + /// `100.64.0.0/10` (carrier-grade NAT). + SharedCgnat, + /// `169.254.0.0/16`, `fe80::/10`. + LinkLocal, + /// Cloud instance metadata endpoints. + CloudMetadata, + /// `fc00::/7`. + UniqueLocal, + /// `224.0.0.0/4`, `ff00::/8`. + Multicast, + /// `255.255.255.255`. + Broadcast, + /// Documentation ranges (`192.0.2.0/24`, `2001:db8::/32`, ...). + Documentation, + /// `198.18.0.0/15`, `2001:2::/48`. + Benchmarking, + /// Everything else outside global unicast (IETF protocol assignments, + /// `240.0.0.0/4`, IPv4-compatible IPv6, Teredo, local NAT64, ...). + Reserved, +} + +impl fmt::Display for AddressClass { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str(match self { + Self::Unspecified => "unspecified", + Self::Loopback => "loopback", + Self::Private => "private", + Self::SharedCgnat => "shared (CGNAT)", + Self::LinkLocal => "link-local", + Self::CloudMetadata => "cloud metadata", + Self::UniqueLocal => "unique local", + Self::Multicast => "multicast", + Self::Broadcast => "broadcast", + Self::Documentation => "documentation", + Self::Benchmarking => "benchmarking", + Self::Reserved => "reserved", + }) + } +} + +/// Well-known cloud metadata endpoints. They also fall into blocked ranges +/// (link-local, CGNAT, unique local); naming them makes errors clearer. +const METADATA_V4: [Ipv4Addr; 3] = [ + Ipv4Addr::new(169, 254, 169, 254), + Ipv4Addr::new(169, 254, 170, 2), + Ipv4Addr::new(100, 100, 100, 200), +]; +const METADATA_V6: [Ipv6Addr; 1] = [Ipv6Addr::new(0xfd00, 0x0ec2, 0, 0, 0, 0, 0, 0x0254)]; + +fn in_v4(ip: Ipv4Addr, net: [u8; 4], prefix: u32) -> bool { + let mask = if prefix == 0 { + 0 + } else { + u32::MAX << (32 - prefix) + }; + u32::from(ip) & mask == u32::from(Ipv4Addr::from(net)) & mask +} + +fn in_v6(ip: Ipv6Addr, net: [u16; 8], prefix: u32) -> bool { + let mask = if prefix == 0 { + 0 + } else { + u128::MAX << (128 - prefix) + }; + let [a, b, c, d, e, f, g, h] = net; + u128::from(ip) & mask == u128::from(Ipv6Addr::new(a, b, c, d, e, f, g, h)) & mask +} + +/// Class of a non-public IPv4 address; `None` for a public one. +fn classify_v4(ip: Ipv4Addr) -> Option { + use AddressClass::*; + if METADATA_V4.contains(&ip) { + return Some(CloudMetadata); + } + const RULES: &[([u8; 4], u32, AddressClass)] = &[ + ([0, 0, 0, 0], 8, Unspecified), + ([10, 0, 0, 0], 8, Private), + ([100, 64, 0, 0], 10, SharedCgnat), + ([127, 0, 0, 0], 8, Loopback), + ([169, 254, 0, 0], 16, LinkLocal), + ([172, 16, 0, 0], 12, Private), + ([192, 0, 0, 0], 24, Reserved), + ([192, 0, 2, 0], 24, Documentation), + ([192, 88, 99, 0], 24, Reserved), + ([192, 168, 0, 0], 16, Private), + ([198, 18, 0, 0], 15, Benchmarking), + ([198, 51, 100, 0], 24, Documentation), + ([203, 0, 113, 0], 24, Documentation), + ([224, 0, 0, 0], 4, Multicast), + ([255, 255, 255, 255], 32, Broadcast), + ([240, 0, 0, 0], 4, Reserved), + ]; + RULES + .iter() + .find(|(net, prefix, _)| in_v4(ip, *net, *prefix)) + .map(|(_, _, class)| *class) +} + +/// Class of a non-public IPv6 address; `None` for a public one. Forms that +/// embed an IPv4 address (mapped, NAT64, 6to4) are judged by that address. +fn classify_v6(ip: Ipv6Addr) -> Option { + use AddressClass::*; + if METADATA_V6.contains(&ip) { + return Some(CloudMetadata); + } + if ip.is_unspecified() { + return Some(Unspecified); + } + if ip.is_loopback() { + return Some(Loopback); + } + let bits = u128::from(ip); + let embedded_v4 = |shift: u32| Ipv4Addr::from((bits >> shift) as u32); + // ::ffff:a.b.c.d + if let Some(v4) = ip.to_ipv4_mapped() { + return classify_v4(v4); + } + // 64:ff9b::/96, the well-known NAT64 prefix. + if in_v6(ip, [0x64, 0xff9b, 0, 0, 0, 0, 0, 0], 96) { + return classify_v4(embedded_v4(0)); + } + // 2002::/16, 6to4: the IPv4 address follows the prefix. + if in_v6(ip, [0x2002, 0, 0, 0, 0, 0, 0, 0], 16) { + return classify_v4(embedded_v4(80)); + } + const RULES: &[([u16; 8], u32, AddressClass)] = &[ + // IPv4-compatible (deprecated) and the rest of ::/96. + ([0, 0, 0, 0, 0, 0, 0, 0], 96, Reserved), + ([0x64, 0xff9b, 1, 0, 0, 0, 0, 0], 48, Reserved), + ([0x100, 0, 0, 0, 0, 0, 0, 0], 64, Reserved), + ([0x2001, 0x2, 0, 0, 0, 0, 0, 0], 48, Benchmarking), + ([0x2001, 0xdb8, 0, 0, 0, 0, 0, 0], 32, Documentation), + // IETF protocol assignments, including Teredo (2001::/32). + ([0x2001, 0, 0, 0, 0, 0, 0, 0], 23, Reserved), + ([0x3fff, 0, 0, 0, 0, 0, 0, 0], 20, Documentation), + ([0xfc00, 0, 0, 0, 0, 0, 0, 0], 7, UniqueLocal), + ([0xfe80, 0, 0, 0, 0, 0, 0, 0], 10, LinkLocal), + ([0xfec0, 0, 0, 0, 0, 0, 0, 0], 10, Private), + ([0xff00, 0, 0, 0, 0, 0, 0, 0], 8, Multicast), + ]; + if let Some((_, _, class)) = RULES + .iter() + .find(|(net, prefix, _)| in_v6(ip, *net, *prefix)) + { + return Some(*class); + } + // Only global unicast (2000::/3) is public. + (!in_v6(ip, [0x2000, 0, 0, 0, 0, 0, 0, 0], 3)).then_some(Reserved) +} + +/// Class of a non-public address; `None` when it is public. +pub(crate) fn classify(ip: IpAddr) -> Option { + match ip { + IpAddr::V4(ip) => classify_v4(ip), + IpAddr::V6(ip) => classify_v6(ip), + } +} + +/// Ports feeds and enclosures may be fetched from. +pub(crate) const ALLOWED_PORTS: [u16; 4] = [80, 443, 8080, 8443]; + +/// Which addresses [`SafeHttp`] may connect to. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) struct AddressPolicy { + allow_loopback: bool, + any_port: bool, +} + +impl AddressPolicy { + /// Public addresses on [`ALLOWED_PORTS`] only. The only policy + /// available outside tests. + pub(crate) const PUBLIC_ONLY: Self = Self { + allow_loopback: false, + any_port: false, + }; + + /// Public addresses plus loopback on any port, for the local test + /// servers. + #[cfg(test)] + pub(crate) const fn test_allow_loopback() -> Self { + Self { + allow_loopback: true, + any_port: true, + } + } + + /// Whether a connection to `port` is allowed. + pub(crate) fn allows_port(&self, port: u16) -> bool { + self.any_port || ALLOWED_PORTS.contains(&port) + } + + /// Checks one address. + pub(crate) fn check(&self, ip: IpAddr) -> Result<(), AddressClass> { + match classify(ip) { + None => Ok(()), + Some(AddressClass::Loopback) if self.allow_loopback => Ok(()), + Some(class) => Err(class), + } + } +} + +/// A host name or literal resolved to an address that is not allowed. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct BlockedAddress { + /// Host as written in the URL. + pub(crate) host: String, + /// The offending address. + pub(crate) address: IpAddr, + /// Why it is blocked. + pub(crate) class: AddressClass, +} + +impl BlockedAddress { + /// Whether the URL named the address itself (an IP literal) rather + /// than a host name that resolved to it. + pub(crate) fn is_literal(&self) -> bool { + let literal = self.host.trim_start_matches('[').trim_end_matches(']'); + literal.parse::().is_ok() + } +} + +impl fmt::Display for BlockedAddress { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + if self.is_literal() { + write!(f, "{} is a {} address", self.host, self.class)?; + } else { + write!( + f, + "{} resolves to a {} address ({})", + self.host, self.class, self.address + )?; + } + f.write_str("; only public addresses are allowed") + } +} + +impl std::error::Error for BlockedAddress {} + +/// Name resolution used by [`GuardedResolver`]. +pub(crate) trait HostLookup: Send + Sync + 'static { + /// Every address of `host` (a DNS name, never an IP literal). + fn lookup(&self, host: &str) -> LookupFuture; +} + +pub(crate) type LookupFuture = + Pin>> + Send + 'static>>; + +/// The operating system resolver (`getaddrinfo` via tokio). +#[derive(Debug, Clone, Copy, Default)] +pub(crate) struct SystemLookup; + +impl HostLookup for SystemLookup { + fn lookup(&self, host: &str) -> LookupFuture { + let host = host.to_string(); + Box::pin(async move { + Ok(tokio::net::lookup_host((host.as_str(), 0)) + .await? + .map(|addr| addr.ip()) + .collect()) + }) + } +} + +const LOOKUP_TIMEOUT: Duration = Duration::from_secs(10); + +/// DNS resolver of the guarded client: resolves, then rejects the answer +/// unless every address is allowed. The connector connects to exactly the +/// returned addresses, so a checked answer cannot be swapped afterwards. +pub(crate) struct GuardedResolver { + policy: AddressPolicy, + lookup: Arc, +} + +impl GuardedResolver { + pub(crate) fn new(policy: AddressPolicy, lookup: Arc) -> Self { + Self { policy, lookup } + } + + /// Resolves and checks `host`; the addresses a connection may use. + pub(crate) async fn resolve_checked( + &self, + host: &str, + ) -> Result, Box> { + let ips = tokio::time::timeout(LOOKUP_TIMEOUT, self.lookup.lookup(host)) + .await + .map_err(|_| format!("resolving {host} timed out"))??; + if ips.is_empty() { + return Err(format!("{host} has no addresses").into()); + } + for ip in &ips { + if let Err(class) = self.policy.check(*ip) { + return Err(Box::new(BlockedAddress { + host: host.to_string(), + address: *ip, + class, + })); + } + } + Ok(ips) + } +} + +impl reqwest::dns::Resolve for GuardedResolver { + fn resolve(&self, name: reqwest::dns::Name) -> reqwest::dns::Resolving { + let resolver = Self { + policy: self.policy, + lookup: Arc::clone(&self.lookup), + }; + let host = name.as_str().to_string(); + Box::pin(async move { + let ips = resolver.resolve_checked(&host).await?; + // Port 0: the connector uses the URL's port. + let addrs: Vec = ips.into_iter().map(|ip| SocketAddr::new(ip, 0)).collect(); + Ok(Box::new(addrs.into_iter()) as reqwest::dns::Addrs) + }) + } +} + +/// Errors of [`SafeHttp`]. +#[derive(Debug)] +pub(crate) enum GuardError { + /// Not `http`/`https`. + UnsupportedScheme(String), + /// The URL carries `user:password@`. + Credentials, + /// The URL has no host. + MissingHost, + /// The host is (or resolves to) a blocked address. + Blocked(BlockedAddress), + /// The port is not one of [`ALLOWED_PORTS`]. + Port(u16), + /// More redirects than allowed. + TooManyRedirects(usize), + /// A redirect without a usable `Location`. + InvalidRedirect(String), + /// Any other transport failure (DNS, connect, TLS, timeout, ...). + Http(reqwest::Error), +} + +impl GuardError { + pub(crate) fn is_timeout(&self) -> bool { + matches!(self, Self::Http(error) if error.is_timeout()) + } + + /// Whether the guard refused the URL for what it says, before any name + /// resolution: scheme, credentials, port, a blocked IP literal. + /// + /// A host *name* that resolves to a blocked address is not a refusal in + /// this sense: users are told the same as for a name that does not + /// resolve or a server that does not answer, so the answers cannot be + /// used to learn which names exist in the server's internal DNS. The + /// details stay in the server log. + pub(crate) fn is_refusal(&self) -> bool { + match self { + Self::UnsupportedScheme(_) | Self::Credentials | Self::MissingHost | Self::Port(_) => { + true + } + Self::Blocked(blocked) => blocked.is_literal(), + _ => false, + } + } +} + +/// What a signed-in user is told about a refused or failed request. The +/// details ([`GuardError`]'s `Display`: resolved addresses, transport error +/// chains) stay in the server logs; showing them would turn feed and audio +/// requests into a scanner of the server's network. +pub(crate) const REFUSED_MESSAGE: &str = + "only public http and https addresses on ports 80, 443, 8080 and 8443 are allowed"; + +impl fmt::Display for GuardError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::UnsupportedScheme(scheme) => { + write!(f, "unsupported URL scheme {scheme:?} (only http and https)") + } + Self::Credentials => f.write_str("URLs with credentials are not allowed"), + Self::MissingHost => f.write_str("URL has no host"), + Self::Blocked(blocked) => blocked.fmt(f), + Self::Port(port) => write!(f, "port {port} is not allowed"), + Self::TooManyRedirects(limit) => write!(f, "more than {limit} redirects"), + Self::InvalidRedirect(reason) => write!(f, "invalid redirect: {reason}"), + Self::Http(error) => { + write!(f, "{error}")?; + let mut source = std::error::Error::source(error); + while let Some(inner) = source { + write!(f, ": {inner}")?; + source = inner.source(); + } + Ok(()) + } + } + } +} + +impl std::error::Error for GuardError {} + +impl From for GuardError { + /// Surfaces a resolver rejection buried in the transport error chain. + /// The URL is dropped: feed and enclosure URLs often carry access + /// tokens, and these errors end up in logs. + fn from(error: reqwest::Error) -> Self { + let error = error.without_url(); + let mut source = std::error::Error::source(&error); + while let Some(inner) = source { + if let Some(blocked) = inner.downcast_ref::() { + return Self::Blocked(blocked.clone()); + } + source = inner.source(); + } + Self::Http(error) + } +} + +/// Timeouts and limits of a [`SafeHttp`] client. +#[derive(Debug, Clone, Copy)] +pub(crate) struct SafeHttpConfig { + /// TCP + TLS connect timeout (name resolution has its own 10 s bound). + pub(crate) connect_timeout: Duration, + /// Maximum time between two reads of a response. + pub(crate) read_timeout: Duration, + /// Whole-request timeout including the body; `None` for streams. + pub(crate) total_timeout: Option, + /// Redirect hops followed before giving up. + pub(crate) max_redirects: usize, +} + +/// A response after following redirects. +#[derive(Debug)] +pub(crate) struct Followed { + /// The final response (never a followed redirect). + pub(crate) response: reqwest::Response, + /// URL of the final response. + pub(crate) final_url: Url, + /// Status of every redirect followed, in order. + pub(crate) redirects: Vec, +} + +impl Followed { + /// Whether at least one redirect was followed and all of them were + /// permanent (301/308): only then has the resource moved. + pub(crate) fn moved_permanently(&self) -> bool { + !self.redirects.is_empty() + && self + .redirects + .iter() + .all(|status| matches!(status.as_u16(), 301 | 308)) + } +} + +/// SSRF-guarded HTTP client (see the module documentation). +#[derive(Clone)] +pub(crate) struct SafeHttp { + client: reqwest::Client, + policy: AddressPolicy, + max_redirects: usize, +} + +impl fmt::Debug for SafeHttp { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("SafeHttp") + .field("policy", &self.policy) + .field("max_redirects", &self.max_redirects) + .finish_non_exhaustive() + } +} + +impl SafeHttp { + /// A production client: public addresses only, system DNS. + pub(crate) fn public(config: SafeHttpConfig) -> anyhow::Result { + Self::with_lookup(config, AddressPolicy::PUBLIC_ONLY, Arc::new(SystemLookup)) + } + + /// A client with an explicit policy and resolver (tests inject both). + pub(crate) fn with_lookup( + config: SafeHttpConfig, + policy: AddressPolicy, + lookup: Arc, + ) -> anyhow::Result { + let mut builder = reqwest::Client::builder() + .dns_resolver(Arc::new(GuardedResolver::new(policy, lookup))) + // A proxy would resolve the target itself, bypassing the guard. + .no_proxy() + .redirect(reqwest::redirect::Policy::none()) + .referer(false) + .connect_timeout(config.connect_timeout) + .read_timeout(config.read_timeout) + .user_agent(concat!( + "furumusic/", + env!("CARGO_PKG_VERSION"), + " (podcasts)" + )); + if let Some(timeout) = config.total_timeout { + builder = builder.timeout(timeout); + } + Ok(Self { + client: builder.build()?, + policy, + max_redirects: config.max_redirects, + }) + } + + /// Checks a URL before any connection: scheme, credentials, host, and + /// IP-literal hosts against the policy (names are checked by the + /// resolver at connect time). + pub(crate) fn check_url(&self, url: &Url) -> Result<(), GuardError> { + check_url(self.policy, url) + } + + /// `GET url` with `headers`, following redirects (each hop checked, the + /// same headers re-sent; nothing else is added). + pub(crate) async fn get(&self, url: Url, headers: &HeaderMap) -> Result { + let mut url = url; + let mut redirects = Vec::new(); + loop { + self.check_url(&url)?; + let response = self + .client + .get(url.clone()) + .headers(headers.clone()) + .send() + .await?; + let status = response.status(); + if !matches!(status.as_u16(), 301 | 302 | 303 | 307 | 308) { + return Ok(Followed { + response, + final_url: url, + redirects, + }); + } + if redirects.len() >= self.max_redirects { + return Err(GuardError::TooManyRedirects(self.max_redirects)); + } + let location = response + .headers() + .get(LOCATION) + .and_then(|value| value.to_str().ok()) + .ok_or_else(|| { + GuardError::InvalidRedirect(format!("HTTP {status} without Location")) + })?; + let next = url.join(location).map_err(|error| { + GuardError::InvalidRedirect(format!("unusable Location: {error}")) + })?; + redirects.push(status); + url = next; + } + } +} + +/// See [`SafeHttp::check_url`]. +pub(crate) fn check_url(policy: AddressPolicy, url: &Url) -> Result<(), GuardError> { + if !matches!(url.scheme(), "http" | "https") { + return Err(GuardError::UnsupportedScheme(url.scheme().to_string())); + } + if !url.username().is_empty() || url.password().is_some() { + return Err(GuardError::Credentials); + } + let host = url + .host_str() + .filter(|host| !host.is_empty()) + .ok_or(GuardError::MissingHost)?; + // The URL parser normalizes every IPv4 spelling (decimal, octal, hex, + // trailing dot) to dotted quads and keeps IPv6 literals in brackets. + let bare = host + .strip_prefix('[') + .and_then(|host| host.strip_suffix(']')) + .unwrap_or(host); + if let Ok(ip) = bare.parse::() { + policy.check(ip).map_err(|class| { + GuardError::Blocked(BlockedAddress { + host: host.to_string(), + address: ip, + class, + }) + })?; + } + let port = url.port_or_known_default().ok_or(GuardError::MissingHost)?; + if !policy.allows_port(port) { + return Err(GuardError::Port(port)); + } + Ok(()) +} diff --git a/src/podcasts/net_tests.rs b/src/podcasts/net_tests.rs new file mode 100644 index 0000000..78bf65f --- /dev/null +++ b/src/podcasts/net_tests.rs @@ -0,0 +1,438 @@ +//! SSRF guard: address classes, URL checks, the resolver, DNS rebinding, +//! redirects and headers. Network tests use the local test server; loopback +//! is allowed only through the test-only `AddressPolicy::test_allow_loopback`. + +use std::net::IpAddr; +use std::sync::Arc; +use std::sync::atomic::Ordering; + +use reqwest::Url; +use reqwest::header::HeaderMap; + +use super::net::{ + AddressClass, AddressPolicy, BlockedAddress, GuardError, GuardedResolver, SafeHttp, check_url, + classify, +}; +use super::test_server::{Route, ScriptedLookup, TEST_HTTP, TestServer, test_http}; + +// --------------------------------------------------------------------------- +// Address classes +// --------------------------------------------------------------------------- + +fn ip(text: &str) -> IpAddr { + text.parse().unwrap() +} + +#[test] +fn public_addresses_are_allowed() { + for address in [ + "8.8.8.8", + "1.1.1.1", + "93.184.215.14", + "172.32.0.1", + "100.128.0.1", + "192.0.3.1", + "2606:4700:4700::1111", + "2a00:1450:4001:80b::200e", + "::ffff:8.8.8.8", + "64:ff9b::808:808", + "2002:808:808::1", + ] { + assert_eq!(classify(ip(address)), None, "{address}"); + assert_eq!(AddressPolicy::PUBLIC_ONLY.check(ip(address)), Ok(())); + } +} + +#[test] +fn non_public_addresses_are_classified() { + use AddressClass::*; + let cases = [ + // IPv4 + ("0.0.0.0", Unspecified), + ("0.1.2.3", Unspecified), + ("10.1.2.3", Private), + ("100.64.0.1", SharedCgnat), + ("100.127.255.254", SharedCgnat), + ("127.0.0.1", Loopback), + ("127.255.0.9", Loopback), + ("169.254.1.1", LinkLocal), + ("169.254.169.254", CloudMetadata), + ("169.254.170.2", CloudMetadata), + ("100.100.100.200", CloudMetadata), + ("172.16.0.1", Private), + ("172.31.255.255", Private), + ("192.0.0.8", Reserved), + ("192.0.2.5", Documentation), + ("192.88.99.1", Reserved), + ("192.168.1.1", Private), + ("198.18.0.1", Benchmarking), + ("198.19.255.255", Benchmarking), + ("198.51.100.1", Documentation), + ("203.0.113.1", Documentation), + ("224.0.0.1", Multicast), + ("239.255.255.250", Multicast), + ("240.0.0.1", Reserved), + ("255.255.255.255", Broadcast), + // IPv6 + ("::", Unspecified), + ("::1", Loopback), + ("::ffff:127.0.0.1", Loopback), + ("::ffff:10.0.0.1", Private), + ("::ffff:169.254.169.254", CloudMetadata), + ("::ffff:0.0.0.0", Unspecified), + ("::127.0.0.1", Reserved), + ("::8.8.8.8", Reserved), + ("64:ff9b::a00:1", Private), + ("64:ff9b::7f00:1", Loopback), + ("64:ff9b:1::1", Reserved), + ("2002:7f00:1::1", Loopback), + ("2002:a9fe:a9fe::1", CloudMetadata), + ("2002:c0a8:101::1", Private), + ("100::1", Reserved), + ("2001::1", Reserved), + ("2001:2::1", Benchmarking), + ("2001:db8::1", Documentation), + ("3fff::1", Documentation), + ("fc00::1", UniqueLocal), + ("fd12:3456::1", UniqueLocal), + ("fd00:ec2::254", CloudMetadata), + ("fe80::1", LinkLocal), + ("febf::1", LinkLocal), + ("fec0::1", Private), + ("ff02::1", Multicast), + ("4000::1", Reserved), + ("::2", Reserved), + ]; + for (address, class) in cases { + assert_eq!(classify(ip(address)), Some(class), "{address}"); + assert_eq!( + AddressPolicy::PUBLIC_ONLY.check(ip(address)), + Err(class), + "{address}" + ); + } +} + +#[test] +fn only_the_test_policy_allows_loopback_and_nothing_else() { + let test = AddressPolicy::test_allow_loopback(); + assert_eq!(test.check(ip("127.0.0.1")), Ok(())); + assert_eq!(test.check(ip("::1")), Ok(())); + assert_eq!(test.check(ip("::ffff:127.0.0.1")), Ok(())); + assert_eq!(test.check(ip("10.0.0.1")), Err(AddressClass::Private)); + assert_eq!( + test.check(ip("169.254.169.254")), + Err(AddressClass::CloudMetadata) + ); + assert_eq!( + AddressPolicy::PUBLIC_ONLY.check(ip("127.0.0.1")), + Err(AddressClass::Loopback) + ); +} + +fn url(text: &str) -> Url { + Url::parse(text).unwrap() +} + +#[test] +fn urls_are_checked_before_connecting() { + let policy = AddressPolicy::PUBLIC_ONLY; + for (text, class) in [ + ("http://127.0.0.1/feed", AddressClass::Loopback), + // Every IPv4 spelling is normalized by the URL parser. + ("http://2130706433/feed", AddressClass::Loopback), + ("http://0x7f.1/feed", AddressClass::Loopback), + ("http://127.1/feed", AddressClass::Loopback), + ("http://0177.0.0.1/feed", AddressClass::Loopback), + ("http://127.0.0.1./feed", AddressClass::Loopback), + ("http://0/feed", AddressClass::Unspecified), + ("http://[::1]:8080/feed", AddressClass::Loopback), + ("http://[::ffff:7f00:1]/feed", AddressClass::Loopback), + ("http://[::ffff:a9fe:a9fe]/", AddressClass::CloudMetadata), + ( + "http://169.254.169.254/latest/meta-data/", + AddressClass::CloudMetadata, + ), + ("https://10.0.0.5/audio.mp3", AddressClass::Private), + ("https://[fd00::1]/audio.mp3", AddressClass::UniqueLocal), + ] { + match check_url(policy, &url(text)) { + Err(GuardError::Blocked(blocked)) => assert_eq!(blocked.class, class, "{text}"), + other => panic!("{text}: {other:?}"), + } + } + assert!(matches!( + check_url(policy, &url("ftp://example.com/feed")), + Err(GuardError::UnsupportedScheme(_)) + )); + assert!(matches!( + check_url(policy, &url("file:///etc/passwd")), + Err(GuardError::UnsupportedScheme(_)) + )); + assert!(matches!( + check_url(policy, &url("https://user:secret@example.com/feed")), + Err(GuardError::Credentials) + )); + assert!(matches!( + check_url(policy, &url("https://user@example.com/feed")), + Err(GuardError::Credentials) + )); + // Names are checked by the resolver at connect time. + assert!(check_url(policy, &url("https://feeds.example.com/rss")).is_ok()); + assert!(check_url(policy, &url("http://localhost/rss")).is_ok()); +} + +#[test] +fn only_web_ports_are_allowed() { + // A public address can be the server's own; its databases and admin + // ports must not be reachable through feed or audio requests. + let policy = AddressPolicy::PUBLIC_ONLY; + for text in [ + "https://feeds.example.com/rss", + "http://feeds.example.com/rss", + "https://feeds.example.com:443/rss", + "http://feeds.example.com:8080/rss", + "https://feeds.example.com:8443/rss", + "https://8.8.8.8/rss", + ] { + assert!(check_url(policy, &url(text)).is_ok(), "{text}"); + } + for (text, port) in [ + ("http://feeds.example.com:5432/", 5432), + ("https://feeds.example.com:22/", 22), + ("http://8.8.8.8:6379/feed", 6379), + ("https://feeds.example.com:3000/", 3000), + ] { + match check_url(policy, &url(text)) { + Err(error @ GuardError::Port(refused)) => { + assert_eq!(refused, port, "{text}"); + assert!(error.is_refusal()); + } + other => panic!("{text}: {other:?}"), + } + } + // The local test servers listen on random ports. + assert!( + check_url( + AddressPolicy::test_allow_loopback(), + &url("http://127.0.0.1:5432/") + ) + .is_ok() + ); +} + +// --------------------------------------------------------------------------- +// Resolver +// --------------------------------------------------------------------------- + +fn blocked_class(error: &(dyn std::error::Error + Send + Sync + 'static)) -> Option { + error + .downcast_ref::() + .map(|blocked| blocked.class) +} + +#[tokio::test] +async fn resolver_rejects_the_whole_answer_when_any_address_is_not_public() { + let lookup = ScriptedLookup::with(&[ + ("public.test", &[&["8.8.8.8", "2606:4700:4700::1111"]]), + ("mixed.test", &[&["8.8.8.8", "10.0.0.1"]]), + ("mapped.test", &[&["::ffff:192.168.0.10"]]), + ("localhost", &[&["127.0.0.1", "::1"]]), + ("empty.test", &[&[]]), + ]); + let resolver = GuardedResolver::new(AddressPolicy::PUBLIC_ONLY, lookup); + assert_eq!( + resolver.resolve_checked("public.test").await.unwrap(), + vec![ip("8.8.8.8"), ip("2606:4700:4700::1111")] + ); + let error = resolver.resolve_checked("mixed.test").await.unwrap_err(); + assert_eq!(blocked_class(error.as_ref()), Some(AddressClass::Private)); + let error = resolver.resolve_checked("mapped.test").await.unwrap_err(); + assert_eq!(blocked_class(error.as_ref()), Some(AddressClass::Private)); + let error = resolver.resolve_checked("localhost").await.unwrap_err(); + assert_eq!(blocked_class(error.as_ref()), Some(AddressClass::Loopback)); + assert!(resolver.resolve_checked("empty.test").await.is_err()); + assert!(resolver.resolve_checked("nowhere.test").await.is_err()); +} + +fn public_http(lookup: Arc) -> SafeHttp { + SafeHttp::with_lookup(TEST_HTTP, AddressPolicy::PUBLIC_ONLY, lookup).unwrap() +} + +#[tokio::test] +async fn production_policy_never_reaches_a_local_server() { + let server = TestServer::start(); + server.set("/feed", Route::new("ok", "text/plain")); + let lookup = ScriptedLookup::with(&[("loop.test", &[&["127.0.0.1"]])]); + let http = public_http(Arc::clone(&lookup)); + match http.get(url(&server.url("/feed")), &HeaderMap::new()).await { + Err(GuardError::Blocked(blocked)) => assert_eq!(blocked.class, AddressClass::Loopback), + other => panic!("{other:?}"), + } + // The test server's random port is refused before any lookup. + let named = url(&server.url_on("loop.test", "/feed")); + match http.get(named.clone(), &HeaderMap::new()).await { + Err(GuardError::Port(port)) => assert_eq!(Some(port), named.port()), + other => panic!("{other:?}"), + } + assert_eq!(lookup.calls.load(Ordering::SeqCst), 0); + // On an allowed port the resolver refuses the loopback answer. + let resolver = GuardedResolver::new(AddressPolicy::PUBLIC_ONLY, lookup); + let error = resolver.resolve_checked("loop.test").await.unwrap_err(); + assert_eq!(blocked_class(error.as_ref()), Some(AddressClass::Loopback)); + assert_eq!(server.request_count(), 0); +} + +#[tokio::test] +async fn every_connection_uses_the_addresses_it_checked() { + // DNS rebinding: the first answer is allowed, the next one points to a + // private address. Each connection resolves and checks once and + // connects to exactly those addresses, so the rebound answer is refused + // instead of being used after an earlier successful check. + let server = TestServer::start(); + server.set("/feed", Route::new("first", "text/plain")); + let lookup = ScriptedLookup::with(&[("rebind.test", &[&["127.0.0.1"], &["10.0.0.7"]])]); + let http = test_http(Arc::clone(&lookup)); + let target = url(&server.url_on("rebind.test", "/feed")); + let first = http.get(target.clone(), &HeaderMap::new()).await.unwrap(); + assert_eq!(first.response.text().await.unwrap(), "first"); + match http.get(target, &HeaderMap::new()).await { + Err(GuardError::Blocked(blocked)) => { + assert_eq!(blocked.class, AddressClass::Private); + assert_eq!(blocked.address, ip("10.0.0.7")); + } + other => panic!("{other:?}"), + } + assert_eq!(lookup.calls.load(Ordering::SeqCst), 2); + assert_eq!(server.requests("/feed").len(), 1); +} + +#[tokio::test] +async fn every_redirect_hop_is_checked() { + let server = TestServer::start(); + let lookup = ScriptedLookup::with(&[("intranet.test", &[&["192.168.1.20"]])]); + let http = test_http(lookup); + let cases = [ + ( + "/to-private", + "http://10.0.0.1/feed".to_string(), + AddressClass::Private, + ), + ( + "/to-metadata", + "http://169.254.169.254/latest/meta-data/".to_string(), + AddressClass::CloudMetadata, + ), + ( + "/to-mapped", + "http://[::ffff:10.1.1.1]/".to_string(), + AddressClass::Private, + ), + ( + "/to-name", + server.url_on("intranet.test", "/feed"), + AddressClass::Private, + ), + ]; + for (path, location, class) in &cases { + server.set(path, Route::redirect(302, location)); + match http.get(url(&server.url(path)), &HeaderMap::new()).await { + Err(GuardError::Blocked(blocked)) => assert_eq!(blocked.class, *class, "{path}"), + other => panic!("{path}: {other:?}"), + } + } + server.set("/to-file", Route::redirect(301, "file:///etc/passwd")); + assert!(matches!( + http.get(url(&server.url("/to-file")), &HeaderMap::new()) + .await, + Err(GuardError::UnsupportedScheme(_)) + )); + server.set("/no-location", Route::new("", "text/plain").status(302)); + assert!(matches!( + http.get(url(&server.url("/no-location")), &HeaderMap::new()) + .await, + Err(GuardError::InvalidRedirect(_)) + )); + // Hop limit (3 in the test client): /hop0 -> /hop1 -> ... -> /hop4. + for hop in 0..4 { + server.set( + &format!("/hop{hop}"), + Route::redirect(307, &format!("/hop{}", hop + 1)), + ); + } + server.set("/hop4", Route::new("end", "text/plain")); + assert!(matches!( + http.get(url(&server.url("/hop0")), &HeaderMap::new()).await, + Err(GuardError::TooManyRedirects(3)) + )); + let followed = http + .get(url(&server.url("/hop1")), &HeaderMap::new()) + .await + .unwrap(); + assert_eq!(followed.redirects.len(), 3); + assert_eq!(followed.final_url.path(), "/hop4"); +} + +#[tokio::test] +async fn transport_errors_never_carry_the_url() { + // Feed and enclosure URLs often carry access tokens; errors are logged. + let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap(); + let port = listener.local_addr().unwrap().port(); + drop(listener); + let http = test_http(ScriptedLookup::with(&[])); + let error = http + .get( + url(&format!( + "http://127.0.0.1:{port}/private/feed.xml?token=secret" + )), + &HeaderMap::new(), + ) + .await + .unwrap_err(); + assert!(matches!(error, GuardError::Http(_)), "{error:?}"); + let text = error.to_string(); + assert!( + !text.contains("secret") && !text.contains("feed.xml"), + "{text}" + ); + assert!(!format!("{error:?}").contains("secret")); + // A redirect to an unusable location does not echo it either. + let server = TestServer::start(); + server.set("/bad", Route::redirect(302, "http://[::1/?token=secret")); + let error = http + .get(url(&server.url("/bad")), &HeaderMap::new()) + .await + .unwrap_err(); + assert!(matches!(error, GuardError::InvalidRedirect(_)), "{error:?}"); + assert!(!error.to_string().contains("secret"), "{error}"); +} + +#[tokio::test] +async fn no_cookies_or_credentials_are_sent() { + let server = TestServer::start(); + server.set( + "/start", + Route::redirect(302, "/next").header("Set-Cookie", "session=leak; Path=/"), + ); + server.set( + "/next", + Route::new("ok", "text/plain").header("Set-Cookie", "other=leak; Path=/"), + ); + let http = test_http(ScriptedLookup::with(&[])); + for _ in 0..2 { + let followed = http + .get(url(&server.url("/start")), &HeaderMap::new()) + .await + .unwrap(); + assert_eq!(followed.response.text().await.unwrap(), "ok"); + } + for request in server + .requests("/start") + .iter() + .chain(&server.requests("/next")) + { + for header in ["cookie", "authorization", "referer", "proxy-authorization"] { + assert!(!request.contains_key(header), "{header} sent: {request:?}"); + } + } +} diff --git a/src/podcasts/postgres_tests.rs b/src/podcasts/postgres_tests.rs new file mode 100644 index 0000000..f3f91c9 --- /dev/null +++ b/src/podcasts/postgres_tests.rs @@ -0,0 +1,2544 @@ +//! Podcast store against PostgreSQL. +//! +//! Ignored by default: set `FURU_DEVICE_SYNC_TEST_DATABASE_URL` to a +//! disposable database (for example a throwaway `postgres:16-alpine` +//! container). The tests migrate it with the production migrations, use +//! random account ids and remove their rows again; the migration test +//! creates and drops its own database next to it. + +use std::collections::BTreeSet; + +use cot::db::migrations::{MigrationEngine, SyncDynMigration}; +use music_dht::device_sync::{SyncOpPayload, SyncOpWire, SyncSnapshot, random_hex}; +use music_dht::podcast::feed::{ParsedFeed, parse_feed_with_base}; +use music_dht::podcast::{ + EpisodeId, FeedUrl, ListenSession, PodcastEpisodeRef, PodcastId, PodcastMeta, ProgressRecord, + ProgressStatus, ShareRecord, SubscriptionRecord, +}; + +use super::*; +use crate::player::queue_items::test_episode; + +const DATABASE_URL: &str = "FURU_DEVICE_SYNC_TEST_DATABASE_URL"; + +/// Every production migration (the list of `admin::App::migrations`). +fn all_migrations() -> Vec<&'static SyncDynMigration> { + [ + crate::config::db_migrations::MIGRATIONS, + crate::user::db_migrations::MIGRATIONS, + crate::music::db_migrations::MIGRATIONS, + crate::scheduler::db_migrations::MIGRATIONS, + crate::auth::db_migrations::MIGRATIONS, + crate::youtube::db_migrations::MIGRATIONS, + ] + .into_iter() + .flatten() + .copied() + .collect() +} + +async fn migrate(url: &str, migrations: Vec<&'static SyncDynMigration>) { + let database = cot::db::Database::new(url).await.unwrap(); + MigrationEngine::new(migrations) + .unwrap() + .run(&database) + .await + .unwrap(); + database.close().await.unwrap(); +} + +/// Pool on the test database, migrated once per test binary with the +/// production migrations (also used by the device-sync Postgres tests). +pub(crate) async fn test_pool() -> sqlx::PgPool { + static MIGRATED: tokio::sync::Mutex = tokio::sync::Mutex::const_new(false); + let url = std::env::var(DATABASE_URL).unwrap(); + let mut migrated = MIGRATED.lock().await; + if !*migrated { + migrate(&url, all_migrations()).await; + *migrated = true; + } + sqlx::PgPool::connect(&url).await.unwrap() +} + +fn random_user() -> i64 { + let bytes = blake3::hash(random_hex(8).as_bytes()); + 9_000_000_000 + + i64::from(u32::from_le_bytes(bytes.as_bytes()[..4].try_into().unwrap()) % 1_000_000_000) +} + +const ACCOUNT_TABLES: &[&str] = &[ + "furumusic__podcast_fetch_state", + "furumusic__podcast_listen_session", + "furumusic__podcast_progress", + "furumusic__podcast_share", + "furumusic__podcast_subscription", + "furumusic__podcast_episode", + "furumusic__podcast_feed_alias", + "furumusic__podcast", + "furumusic__fed_podcast_relay", + "furumusic__fed_device_identity", + "furumusic__fed_device", + "furumusic__fed_sync_ops", + "furumusic__fed_sync_vector", + "furumusic__fed_state_like", + "furumusic__fed_state_playlist", + "furumusic__fed_state_playlist_item", +]; + +async fn cleanup(pool: &sqlx::PgPool, users: impl IntoIterator) { + let users: Vec = users.into_iter().collect(); + for table in ACCOUNT_TABLES { + sqlx::query(&format!("DELETE FROM {table} WHERE user_id = ANY($1)")) + .bind(&users) + .execute(pool) + .await + .unwrap(); + } +} + +fn feed_url(path: &str) -> FeedUrl { + FeedUrl::parse(&format!("https://podcasts.example/{path}")).unwrap() +} + +fn meta(path: &str, title: &str) -> PodcastMeta { + PodcastMeta { + feed_url: feed_url(path), + title: title.into(), + author: Some("Host".into()), + description: Some("Channel description".into()), + link: None, + image_url: None, + } +} + +fn op(origin: &str, seq: i64, hlc_ms: i64, payload: SyncOpPayload) -> SyncOpWire { + SyncOpWire { + op_id: format!("{origin}:{seq}"), + origin_device_id: origin.into(), + seq, + hlc_ms, + payload, + } +} + +fn session( + id: &str, + device: &str, + episode: &PodcastEpisodeRef, + listened_ms: i64, + completed: bool, +) -> ListenSession { + ListenSession { + session_id: id.into(), + episode_id: episode.episode_id.clone(), + podcast_id: episode.podcast_id.clone(), + device_id: device.into(), + started_at_ms: 1_000, + ended_at_ms: 1_000 + listened_ms, + start_position_ms: 0, + end_position_ms: listened_ms, + listened_ms, + completed, + } +} + +fn progress_op( + origin: &str, + seq: i64, + hlc_ms: i64, + episode: &PodcastEpisodeRef, + generation: u64, + position_ms: i64, +) -> SyncOpWire { + op( + origin, + seq, + hlc_ms, + SyncOpPayload::PodcastProgressSet { + episode_id: episode.episode_id.clone(), + podcast_id: episode.podcast_id.clone(), + position_ms, + status: ProgressStatus::InProgress, + generation, + session_id: format!("{origin}-g{generation}"), + episode: Some(PodcastEpisodeRef { + progress_generation: None, + ..episode.clone() + }), + }, + ) +} + +/// Every permutation of `0..n` (Heap's algorithm). +fn permutations(n: usize) -> Vec> { + fn heap(k: usize, items: &mut Vec, out: &mut Vec>) { + if k <= 1 { + out.push(items.clone()); + return; + } + for i in 0..k { + heap(k - 1, items, out); + let j = if k.is_multiple_of(2) { i } else { 0 }; + items.swap(j, k - 1); + } + } + let mut out = Vec::new(); + heap(n, &mut (0..n).collect(), &mut out); + out +} + +async fn apply_all(store: &PodcastStore, user_id: i64, ops: &[SyncOpWire]) { + for op in ops { + let outcome = store.apply_remote_op(user_id, op).await.unwrap(); + assert!( + matches!(outcome, ApplyOutcome::Changed | ApplyOutcome::Unchanged), + "{outcome:?}" + ); + } +} + +/// The registers and sessions of one podcast and episode from three +/// devices, with a duplicate, a late checkpoint of an older session, a +/// concurrent offline action and a growing session. +fn convergence_ops() -> Vec { + let episode = test_episode(); + let podcast_id = episode.podcast_id.clone(); + let podcast = PodcastMeta { + feed_url: episode.feed_url.clone(), + title: episode.podcast_title.clone(), + author: None, + description: Some("dropped from snapshots".into()), + link: None, + image_url: None, + }; + let subscribe = op( + "tui_a", + 1, + 100, + SyncOpPayload::PodcastSubscriptionSet { + podcast_id: podcast_id.clone(), + subscribed: true, + podcast: podcast.clone(), + feed_aliases: vec![], + generation: 1, + }, + ); + vec![ + subscribe.clone(), + // B saw g1 and unsubscribes with a slow clock: g2 still wins. + op( + "tui_b", + 1, + 50, + SyncOpPayload::PodcastSubscriptionSet { + podcast_id: podcast_id.clone(), + subscribed: false, + podcast, + feed_aliases: vec![], + generation: 2, + }, + ), + // A shared concurrently at g2: stored, but not effective. + op( + "tui_a", + 2, + 300, + SyncOpPayload::PodcastShareSet { + podcast_id: podcast_id.clone(), + shared: true, + generation: 2, + }, + ), + op( + "tui_c", + 1, + 400, + SyncOpPayload::PodcastFeedAliasesAdded { + podcast_id, + feed_aliases: vec![feed_url("moved.xml")], + }, + ), + subscribe, + // A late g3 checkpoint of an old session with a fast clock. + progress_op("tui_a", 3, 9_000, &episode, 3, 600_000), + // B and C both seek at g4 with the same clock: device id decides. + progress_op("tui_b", 2, 1_000, &episode, 4, 10_000), + progress_op("tui_c", 2, 1_000, &episode, 4, 20_000), + op( + "tui_a", + 4, + 9_100, + SyncOpPayload::PodcastListenRecorded { + session: session("s1", "tui_a", &episode, 40_000, false), + episode: None, + }, + ), + op( + "tui_a", + 5, + 9_200, + SyncOpPayload::PodcastListenRecorded { + session: session("s1", "tui_a", &episode, 90_000, true), + episode: None, + }, + ), + op( + "tui_b", + 3, + 1_100, + SyncOpPayload::PodcastListenRecorded { + session: session("s2", "tui_b", &episode, 31_000, false), + episode: Some(PodcastEpisodeRef { + progress_generation: None, + ..episode + }), + }, + ), + ] +} + +#[tokio::test] +#[ignore = "requires a disposable database in FURU_DEVICE_SYNC_TEST_DATABASE_URL"] +async fn postgres_podcast_ops_converge_in_any_order_and_match_snapshot_import() { + let pool = test_pool().await; + let store = PodcastStore::new(pool.clone()); + let ops = convergence_ops(); + let reference_user = random_user(); + let mut users = vec![reference_user]; + apply_all(&store, reference_user, &ops).await; + let reference = store.export_snapshot(reference_user).await.unwrap(); + + // The contract outcome, independently of the order. + let episode = test_episode(); + assert_eq!(reference.podcast_subscriptions.len(), 1); + let subscription = &reference.podcast_subscriptions[0]; + assert!(!subscription.subscribed); + assert_eq!(subscription.generation, 2); + assert!(subscription.podcast.description.is_none()); + assert_eq!( + subscription.feed_urls(), + vec![episode.feed_url.clone(), feed_url("moved.xml")] + ); + assert!(reference.podcast_shares[0].shared); + let summary = store + .podcast(reference_user, &episode.podcast_id) + .await + .unwrap() + .unwrap(); + assert!(!summary.subscribed && !summary.shared); + assert_eq!(reference.podcast_progress.len(), 1); + assert_eq!(reference.podcast_progress[0].position_ms, 20_000); + assert_eq!(reference.podcast_progress[0].device_id, "tui_c"); + let sessions: BTreeSet<_> = reference + .podcast_listen_sessions + .iter() + .map(|session| { + ( + session.session_id.as_str(), + session.listened_ms, + session.completed, + ) + }) + .collect(); + assert_eq!( + sessions, + BTreeSet::from([("s1", 90_000, true), ("s2", 31_000, false)]) + ); + assert_eq!( + store + .completed_listen_count(reference_user, &episode.episode_id) + .await + .unwrap(), + 1 + ); + // The episode reference created a catalog row (never touching progress). + let item = store + .episode(reference_user, &episode.episode_id) + .await + .unwrap() + .unwrap(); + assert!(!item.in_feed); + assert_eq!(item.resume_position_ms(), 20_000); + + // Register operations: all 120 orders of the first five operations + // (subscription, unsubscription, share, aliases, duplicate) with the + // rest appended; progress and session operations: all 720 orders of + // the last six with the first five prepended. + let (registers, personal) = ops.split_at(5); + for order in permutations(registers.len()) { + let user_id = random_user(); + users.push(user_id); + let mut shuffled: Vec = order.iter().map(|&i| registers[i].clone()).collect(); + shuffled.extend_from_slice(personal); + apply_all(&store, user_id, &shuffled).await; + assert_eq!( + store.export_snapshot(user_id).await.unwrap(), + reference, + "{order:?}" + ); + } + for order in permutations(personal.len()) { + let user_id = random_user(); + users.push(user_id); + let mut shuffled = registers.to_vec(); + shuffled.extend(order.iter().map(|&i| personal[i].clone())); + apply_all(&store, user_id, &shuffled).await; + assert_eq!( + store.export_snapshot(user_id).await.unwrap(), + reference, + "{order:?}" + ); + } + + // A new device recovering from the snapshot reaches the same state, and + // the operations delivered afterwards change nothing. (Snapshots omit + // the channel description, so the winning subscription operation only + // adds it back to the stored register: the contract's tie-break by + // metadata. Every other operation is already reflected.) + let recovered = random_user(); + users.push(recovered); + let stats = store.apply_snapshot(recovered, &reference).await.unwrap(); + assert_eq!(stats.rejected, 0); + assert!(!stats.ignored_over_limits); + assert_eq!(store.export_snapshot(recovered).await.unwrap(), reference); + for op in ops.iter().rev() { + let expected = if op.op_id == "tui_b:1" { + ApplyOutcome::Changed + } else { + ApplyOutcome::Unchanged + }; + assert_eq!( + store.apply_remote_op(recovered, op).await.unwrap(), + expected, + "{}", + op.op_id + ); + } + assert_eq!(store.export_snapshot(recovered).await.unwrap(), reference); + // A stale snapshot never replaces newer state. + let stale = SyncSnapshot { + podcast_progress: vec![ProgressRecord { + position_ms: 1, + generation: 3, + hlc_ms: i64::MAX, + ..reference.podcast_progress[0].clone() + }], + ..SyncSnapshot::default() + }; + let stats = store.apply_snapshot(recovered, &stale).await.unwrap(); + assert_eq!((stats.changed, stats.unchanged), (0, 1)); + assert_eq!(store.export_snapshot(recovered).await.unwrap(), reference); + // Invalid operations are rejected without touching state. + let mut invalid = progress_op("tui_a", 9, 1, &episode, 9, 1); + if let SyncOpPayload::PodcastProgressSet { session_id, .. } = &mut invalid.payload { + session_id.clear(); + } + assert!(matches!( + store.apply_remote_op(recovered, &invalid).await.unwrap(), + ApplyOutcome::Rejected(_) + )); + let mut huge = progress_op("tui_a", 10, 1, &episode, u64::MAX, 1); + huge.op_id = "tui_a:10".into(); + assert!(matches!( + store.apply_remote_op(recovered, &huge).await.unwrap(), + ApplyOutcome::Rejected(_) + )); + assert_eq!(store.export_snapshot(recovered).await.unwrap(), reference); + + cleanup(&pool, users).await; +} + +#[tokio::test] +#[ignore = "requires a disposable database in FURU_DEVICE_SYNC_TEST_DATABASE_URL"] +async fn postgres_podcast_local_actions_keep_progress_through_unsubscribe() { + let pool = test_pool().await; + let store = PodcastStore::new(pool.clone()); + let user_id = random_user(); + let device = crate::federation::devices::local_device_id(&pool, user_id) + .await + .unwrap(); + let episode = test_episode(); + let show = PodcastMeta { + feed_url: episode.feed_url.clone(), + ..meta("unused.xml", &episode.podcast_title) + }; + + let subscribed = store.subscribe(user_id, &show).await.unwrap(); + assert_eq!(subscribed.podcast_id, episode.podcast_id); + assert!(!subscribed.already_subscribed); + let [first] = subscribed.ops.as_slice() else { + panic!("one operation"); + }; + assert_eq!(first.origin_device_id, device); + assert_eq!(first.op_id, format!("{device}:{}", first.seq)); + // The operation is in the log with exactly this stamp; applying it again + // is a no-op. + let logged: serde_json::Value = sqlx::query_scalar( + "SELECT payload_json FROM furumusic__fed_sync_ops + WHERE user_id = $1 AND op_id = $2 AND hlc_ms = $3 AND kind = 'podcast_subscription_set'", + ) + .bind(user_id) + .bind(&first.op_id) + .bind(first.hlc_ms) + .fetch_one(&pool) + .await + .unwrap(); + assert_eq!(logged, serde_json::to_value(&first.payload).unwrap()); + assert_eq!( + store.apply_remote_op(user_id, first).await.unwrap(), + ApplyOutcome::Unchanged + ); + assert!( + store + .subscribe(user_id, &show) + .await + .unwrap() + .already_subscribed + ); + assert_eq!( + store + .set_shared(user_id, &episode.podcast_id, true) + .await + .unwrap() + .len(), + 1 + ); + assert!( + store + .set_shared(user_id, &episode.podcast_id, true) + .await + .unwrap() + .is_empty() + ); + assert!( + store + .podcast(user_id, &episode.podcast_id) + .await + .unwrap() + .unwrap() + .shared + ); + + // Listening: an explicit start, checkpoints keep the generation. + let start = store + .start_session(user_id, &episode, "web-s1", 0, episode.progress_generation) + .await + .unwrap(); + assert_eq!(start.record.generation, 8, "max(local, carried 7) + 1"); + let CheckpointOutcome::Written(checkpoint) = store + .checkpoint(user_id, &start.session, 30_000) + .await + .unwrap() + else { + panic!("checkpoint written"); + }; + assert_eq!(checkpoint.record.generation, 8); + assert_eq!( + store + .resume_position_ms(user_id, &episode.episode_id) + .await + .unwrap(), + 30_000 + ); + // A seek is explicit: later checkpoints use its generation. + let seek = store + .seek(user_id, &checkpoint.session, 12_000) + .await + .unwrap(); + assert_eq!(seek.record.generation, 9); + assert_eq!(seek.session.generation, 9); + let start = seek; + // Another device seeks back at a greater generation with a slow clock: + // later checkpoints of the old session are refused. + store + .apply_remote_op(user_id, &progress_op("tui_b", 1, 1, &episode, 10, 5_000)) + .await + .unwrap(); + assert!(matches!( + store + .checkpoint(user_id, &start.session, 900_000) + .await + .unwrap(), + CheckpointOutcome::Superseded { .. } + )); + assert_eq!( + store + .resume_position_ms(user_id, &episode.episode_id) + .await + .unwrap(), + 5_000 + ); + let listened = ListenSession { + device_id: device.clone(), + ..session("web-s1", &device, &episode, 45_000, false) + }; + assert!( + store + .record_listen_session(user_id, &listened, None) + .await + .unwrap() + .is_some() + ); + assert!( + store + .record_listen_session(user_id, &listened, None) + .await + .unwrap() + .is_none() + ); + assert!( + store + .record_listen_session( + user_id, + &session("x", "tui_b", &episode, 45_000, false), + None + ) + .await + .is_err() + ); + + // Unsubscribing keeps progress and history and revokes the share. + let ops = store + .unsubscribe(user_id, &episode.podcast_id) + .await + .unwrap(); + assert_eq!(ops.len(), 2); + assert!( + store + .unsubscribe(user_id, &episode.podcast_id) + .await + .unwrap() + .is_empty() + ); + let summary = store + .podcast(user_id, &episode.podcast_id) + .await + .unwrap() + .unwrap(); + assert!(!summary.subscribed && !summary.shared); + assert_eq!( + store + .resume_position_ms(user_id, &episode.episode_id) + .await + .unwrap(), + 5_000 + ); + assert_eq!( + store.listen_history(user_id, None, 10).await.unwrap().len(), + 1 + ); + assert!( + store + .list_podcasts(user_id, PodcastListFilter::Subscribed) + .await + .unwrap() + .is_empty() + ); + assert_eq!( + store + .list_podcasts(user_id, PodcastListFilter::All) + .await + .unwrap() + .len(), + 1 + ); + + // Subscribing again resolves to the same podcast, continues the + // progress and is private again. + let again = store.subscribe(user_id, &show).await.unwrap(); + assert_eq!(again.podcast_id, episode.podcast_id); + let register = store + .subscription(user_id, &episode.podcast_id) + .await + .unwrap() + .unwrap(); + let share = store + .share(user_id, &episode.podcast_id) + .await + .unwrap() + .unwrap(); + assert!(register.subscribed); + assert!(register.generation > share.generation); + let summary = store + .podcast(user_id, &episode.podcast_id) + .await + .unwrap() + .unwrap(); + assert!(summary.subscribed && !summary.shared); + assert_eq!( + store + .resume_position_ms(user_id, &episode.episode_id) + .await + .unwrap(), + 5_000 + ); + + // Mark played writes progress only; restart and unplayed reset it. + let played = store.mark_played(user_id, &episode).await.unwrap(); + assert_eq!(played.record.status, ProgressStatus::Completed); + assert_eq!(played.record.generation, 11); + assert_eq!( + store.listen_history(user_id, None, 10).await.unwrap().len(), + 1 + ); + assert_eq!( + store + .resume_position_ms(user_id, &episode.episode_id) + .await + .unwrap(), + 0 + ); + let restarted = store.restart(user_id, &episode, "web-s2").await.unwrap(); + assert_eq!( + ( + restarted.record.position_ms, + restarted.record.status, + restarted.record.generation + ), + (0, ProgressStatus::InProgress, 12) + ); + let CheckpointOutcome::Written(finished) = store + .complete_if_current(user_id, &restarted.session, 1_790_000) + .await + .unwrap() + else { + panic!("the running session completes the episode"); + }; + assert_eq!( + (finished.record.status, finished.record.generation), + (ProgressStatus::Completed, 13) + ); + assert!(matches!( + store + .checkpoint(user_id, &finished.session, 1_795_000) + .await + .unwrap(), + CheckpointOutcome::AlreadyCompleted { .. } + )); + let mut played_filter = EpisodeQuery::new(episode.podcast_id.clone()); + played_filter.played = PlayedFilter::Played; + assert_eq!( + store.episodes(user_id, &played_filter).await.unwrap().len(), + 1 + ); + played_filter.played = PlayedFilter::Unplayed; + assert!( + store + .episodes(user_id, &played_filter) + .await + .unwrap() + .is_empty() + ); + assert_eq!( + store + .mark_unplayed(user_id, &episode) + .await + .unwrap() + .record + .status, + ProgressStatus::NotStarted + ); + played_filter.played = PlayedFilter::NotStarted; + let items = store.episodes(user_id, &played_filter).await.unwrap(); + assert_eq!(items[0].status(), ProgressStatus::NotStarted); + + // Local stamps: consecutive sequence numbers, strictly growing clock. + let stamps: Vec<(i64, i64)> = sqlx::query_as( + "SELECT seq, hlc_ms FROM furumusic__fed_sync_ops + WHERE user_id = $1 AND origin_device_id = $2 ORDER BY seq", + ) + .bind(user_id) + .bind(&device) + .fetch_all(&pool) + .await + .unwrap(); + assert!( + stamps + .windows(2) + .all(|pair| pair[1].0 == pair[0].0 + 1 && pair[1].1 > pair[0].1) + ); + // Re-applying the whole local log changes nothing. + let before = store.export_snapshot(user_id).await.unwrap(); + let payloads: Vec<(String, i64, i64, serde_json::Value)> = sqlx::query_as( + "SELECT op_id, seq, hlc_ms, payload_json FROM furumusic__fed_sync_ops + WHERE user_id = $1 AND kind LIKE 'podcast\\_%' ORDER BY hlc_ms", + ) + .bind(user_id) + .fetch_all(&pool) + .await + .unwrap(); + for (op_id, seq, hlc_ms, payload) in payloads { + let origin = op_id.rsplit_once(':').unwrap().0.to_string(); + let op = SyncOpWire { + op_id, + origin_device_id: origin, + seq, + hlc_ms, + payload: serde_json::from_value(payload).unwrap(), + }; + assert_eq!( + store.apply_remote_op(user_id, &op).await.unwrap(), + ApplyOutcome::Unchanged + ); + } + assert_eq!(store.export_snapshot(user_id).await.unwrap(), before); + + // Sharing needs a subscription. + let unknown = PodcastId::from_feed_url(&feed_url("unknown.xml")); + let error = store.set_shared(user_id, &unknown, true).await.unwrap_err(); + assert!(matches!( + error.downcast_ref::(), + Some(PodcastStoreError::NotSubscribed(_)) + )); + + cleanup(&pool, [user_id]).await; +} + +fn rss(items: &str) -> String { + format!( + r#" + + + Refresh Show + https://show.example/ + Channel text + Host + {items} + +"# + ) +} + +fn parse(xml: &str, url: &FeedUrl) -> ParsedFeed { + parse_feed_with_base(xml.as_bytes(), Some(url.as_str())).unwrap() +} + +#[tokio::test] +#[ignore = "requires a disposable database in FURU_DEVICE_SYNC_TEST_DATABASE_URL"] +async fn postgres_podcast_feed_refresh_keeps_ids_and_progress() { + let pool = test_pool().await; + let store = PodcastStore::new(pool.clone()); + let user_id = random_user(); + let url = feed_url("refresh.xml"); + let podcast_id = PodcastId::from_feed_url(&url); + let v1 = parse( + &rss(r#" + g-1One1 + 1Mon, 01 Jan 2024 10:00:00 GMT + + Twohttps://show.example/ep2 + 12 + Mon, 08 Jan 2024 10:00:00 GMT + Talk about Ünïcode + + ThreeMon, 15 Jan 2024 10:00:00 GMT + "#), + &url, + ); + // What every device derives for this document. + let ids: Vec = v1 + .resolve_episode_ids(&podcast_id, &[]) + .into_iter() + .map(Option::unwrap) + .collect(); + + // Another device played the GUID-less "Two" before this account ever + // fetched the feed: its reference arrives with the progress. + let two = PodcastEpisodeRef { + podcast_id: podcast_id.clone(), + episode_id: ids[1].clone(), + feed_url: url.clone(), + podcast_title: "Refresh Show".into(), + episode_title: "Two".into(), + audio_url: "https://cdn.example/a2.mp3".into(), + mime: None, + duration_ms: None, + published_at_ms: None, + season: None, + episode_number: None, + image_url: None, + progress_generation: None, + }; + store + .apply_remote_op(user_id, &progress_op("tui_a", 1, 10, &two, 3, 61_000)) + .await + .unwrap(); + let subscribed = store + .subscribe(user_id, &v1.to_podcast_meta(url.clone())) + .await + .unwrap(); + assert_eq!(subscribed.podcast_id, podcast_id); + let first = store + .apply_feed( + user_id, + &FeedRefresh { + podcast_id: &podcast_id, + feed: &v1, + fetched_from: &url, + moved_permanently: false, + }, + ) + .await + .unwrap(); + assert_eq!((first.new_episodes, first.updated_episodes), (2, 1)); + assert!(first.ops.is_empty()); + let stored: BTreeSet = store + .episodes(user_id, &EpisodeQuery::new(podcast_id.clone())) + .await + .unwrap() + .into_iter() + .map(|item| item.episode.episode_id) + .collect(); + assert_eq!(stored, ids.iter().cloned().collect()); + let progress_before = store.progress(user_id, &ids[1]).await.unwrap().unwrap(); + + // The publisher renames episodes, moves their audio, drops "Three", + // adds "Four" and moves the feed permanently. + let moved = feed_url("moved-refresh.xml"); + let v2 = parse( + &rss(r#" + g-1One (remastered)1 + 1Mon, 01 Jan 2024 10:00:00 GMT + + Two, revisedhttps://show.example/ep2 + 12 + Mon, 08 Jan 2024 10:00:00 GMT + Talk about Ünïcode + + g-4Four2 + 1Mon, 22 Jan 2024 10:00:00 GMT + "#), + &moved, + ); + let refresh = FeedRefresh { + podcast_id: &podcast_id, + feed: &v2, + fetched_from: &moved, + moved_permanently: true, + }; + let second = store.apply_feed(user_id, &refresh).await.unwrap(); + assert_eq!( + ( + second.new_episodes, + second.updated_episodes, + second.unchanged_episodes + ), + (1, 2, 0) + ); + assert_eq!(second.ops.len(), 1, "aliases changed once"); + assert_eq!(second.feed_url.as_ref(), Some(&moved)); + let again = store.apply_feed(user_id, &refresh).await.unwrap(); + assert!(again.ops.is_empty()); + assert_eq!(again.unchanged_episodes, 3); + assert_eq!( + store.resolve_podcast_id(user_id, &moved).await.unwrap(), + podcast_id + ); + assert_eq!( + store.resolve_podcast_id(user_id, &url).await.unwrap(), + podcast_id + ); + + // Ids and progress survive; the dropped episode stays out of the feed. + let one = store.episode(user_id, &ids[0]).await.unwrap().unwrap(); + assert_eq!(one.episode.title, "One (remastered)"); + assert_eq!(one.episode.audio_url, "https://cdn2.example/a1.mp3"); + let two_item = store.episode(user_id, &ids[1]).await.unwrap().unwrap(); + assert_eq!(two_item.episode.audio_url, "https://cdn2.example/a2.mp3"); + assert_eq!(two_item.progress.as_ref(), Some(&progress_before)); + assert_eq!(two_item.resume_position_ms(), 61_000); + let three = store.episode(user_id, &ids[2]).await.unwrap().unwrap(); + assert!(!three.in_feed); + // The subscription register is untouched by refreshes. + let register = store + .subscription(user_id, &podcast_id) + .await + .unwrap() + .unwrap(); + assert_eq!(register.op_id, subscribed.ops[0].op_id); + assert!(register.feed_urls().contains(&moved)); + let reference = store.episode_ref(user_id, &ids[1]).await.unwrap().unwrap(); + assert_eq!(reference.progress_generation, Some(3)); + assert_eq!(reference.audio_url, "https://cdn2.example/a2.mp3"); + + // UI queries: counters, seasons, filters, search and sorting. + let summary = store.podcast(user_id, &podcast_id).await.unwrap().unwrap(); + assert_eq!( + ( + summary.episode_count, + summary.unplayed_count, + summary.in_progress_count, + summary.season_count + ), + (4, 4, 1, 2) + ); + assert_eq!(summary.meta.feed_url, moved); + let seasons = store.seasons(user_id, &podcast_id).await.unwrap(); + assert_eq!( + seasons + .iter() + .map(|season| (season.season, season.episode_count)) + .collect::>(), + vec![(Some(1), 2), (Some(2), 1), (None, 1)] + ); + let query = |configure: &dyn Fn(&mut EpisodeQuery)| { + let mut query = EpisodeQuery::new(podcast_id.clone()); + configure(&mut query); + query + }; + let titles = |items: Vec| { + items + .into_iter() + .map(|item| item.episode.title) + .collect::>() + }; + assert_eq!( + titles(store.episodes(user_id, &query(&|_| {})).await.unwrap()), + ["Four", "Three", "Two, revised", "One (remastered)"] + ); + assert_eq!( + titles( + store + .episodes(user_id, &query(&|q| q.sort = EpisodeSort::EpisodeNumber)) + .await + .unwrap() + ), + ["One (remastered)", "Two, revised", "Four", "Three"] + ); + assert_eq!( + titles( + store + .episodes(user_id, &query(&|q| q.season = SeasonFilter::Season(1))) + .await + .unwrap() + ), + ["Two, revised", "One (remastered)"] + ); + assert_eq!( + titles( + store + .episodes(user_id, &query(&|q| q.season = SeasonFilter::NoSeason)) + .await + .unwrap() + ), + ["Three"] + ); + assert_eq!( + titles( + store + .episodes(user_id, &query(&|q| q.played = PlayedFilter::InProgress)) + .await + .unwrap() + ), + ["Two, revised"] + ); + assert_eq!( + titles( + store + .episodes(user_id, &query(&|q| q.search = Some("ÜNÏCODE".into()))) + .await + .unwrap() + ), + ["Two, revised"] + ); + assert_eq!( + titles( + store + .episodes(user_id, &query(&|q| q.search = Some("remaster".into()))) + .await + .unwrap() + ), + ["One (remastered)"] + ); + assert_eq!( + titles( + store + .episodes( + user_id, + &query(&|q| { + q.limit = Some(1); + q.offset = 1; + }) + ) + .await + .unwrap() + ), + ["Three"] + ); + assert!( + store + .episodes(user_id, &query(&|q| q.search = Some("one two".into()))) + .await + .unwrap() + .is_empty() + ); + + // A refresh of a podcast this account does not know is refused. + let unknown = PodcastId::from_feed_url(&feed_url("never.xml")); + let error = store + .apply_feed( + user_id, + &FeedRefresh { + podcast_id: &unknown, + ..refresh + }, + ) + .await + .unwrap_err(); + assert!(matches!( + error.downcast_ref::(), + Some(PodcastStoreError::UnknownPodcast(_)) + )); + + cleanup(&pool, [user_id]).await; +} + +#[tokio::test] +#[ignore = "requires a disposable database in FURU_DEVICE_SYNC_TEST_DATABASE_URL"] +async fn postgres_podcast_fetch_state_is_local_only() { + let pool = test_pool().await; + let store = PodcastStore::new(pool.clone()); + let user_id = random_user(); + let due_show = meta("due.xml", "Due"); + let later_show = meta("later.xml", "Later"); + let gone_show = meta("gone.xml", "Gone"); + let mut ids = Vec::new(); + for show in [&due_show, &later_show, &gone_show] { + ids.push(store.subscribe(user_id, show).await.unwrap().podcast_id); + } + store.unsubscribe(user_id, &ids[2]).await.unwrap(); + let before = serde_json::to_string(&store.export_snapshot(user_id).await.unwrap()).unwrap(); + + let now = 1_800_000_000_000; + store + .record_fetch_success( + user_id, + &ids[1], + now, + Some("\"secret-etag\""), + Some("Tue"), + Some(now + 60_000), + ) + .await + .unwrap(); + assert_eq!( + store + .record_fetch_failure(user_id, &ids[0], now, "boom: 503", Some(now - 1)) + .await + .unwrap(), + 1 + ); + assert_eq!( + store + .record_fetch_failure(user_id, &ids[0], now, "boom: 503", Some(now - 1)) + .await + .unwrap(), + 2 + ); + store + .set_cover_cache( + user_id, + &ids[0], + Some("https://img.example/c.jpg"), + Some("/var/cache/cover-secret.jpg"), + ) + .await + .unwrap(); + let state = store.fetch_state(user_id, &ids[0]).await.unwrap(); + assert_eq!(state.consecutive_failures, 2); + assert_eq!(state.last_error.as_deref(), Some("boom: 503")); + assert_eq!( + state.cover_path.as_deref(), + Some("/var/cache/cover-secret.jpg") + ); + store + .record_fetch_not_modified(user_id, &ids[1], now + 1, Some(now + 60_000)) + .await + .unwrap(); + let later = store.fetch_state(user_id, &ids[1]).await.unwrap(); + assert_eq!(later.etag.as_deref(), Some("\"secret-etag\"")); + assert_eq!(later.last_success_at_ms, Some(now + 1)); + + // Fetch state changes nothing in the synced state. + let after = serde_json::to_string(&store.export_snapshot(user_id).await.unwrap()).unwrap(); + assert_eq!(after, before); + for secret in ["secret-etag", "boom", "cover-secret", "Channel description"] { + assert!(!after.contains(secret), "{secret}"); + } + let ops: Vec = sqlx::query_scalar( + "SELECT payload_json::text FROM furumusic__fed_sync_ops WHERE user_id = $1", + ) + .bind(user_id) + .fetch_all(&pool) + .await + .unwrap(); + assert!( + ops.iter() + .all(|op| !op.contains("secret-etag") && !op.contains("cover-secret")) + ); + + // Scheduling: overdue first, future and unsubscribed podcasts skipped. + let due: Vec = store + .podcasts_due_for_refresh(now, 10_000) + .await + .unwrap() + .into_iter() + .filter(|(user, ..)| *user == user_id) + .map(|(_, id, _)| id) + .collect(); + assert_eq!(due, vec![ids[0].clone()]); + let unknown = PodcastId::from_feed_url(&feed_url("unknown-fetch.xml")); + assert!( + store + .record_fetch_failure(user_id, &unknown, now, "x", None) + .await + .is_err() + ); + + cleanup(&pool, [user_id]).await; +} + +/// Relay rows exactly as 0.13.0 stored them (`podcast_relay`): the contract +/// records as JSON, keyed by section and entity. +async fn insert_relay( + pool: &sqlx::PgPool, + user_id: i64, + section: &str, + key: &str, + value: serde_json::Value, +) { + sqlx::query( + "INSERT INTO furumusic__fed_podcast_relay + (user_id, section, record_key, record_json, updated_at_ms) + VALUES ($1, $2, $3, $4, 1700000000000)", + ) + .bind(user_id) + .bind(section) + .bind(key) + .bind(value) + .execute(pool) + .await + .unwrap(); +} + +async fn insert_logged_op(pool: &sqlx::PgPool, user_id: i64, op: &SyncOpWire, kind: &str) { + sqlx::query( + "INSERT INTO furumusic__fed_sync_ops + (user_id, op_id, origin_device_id, seq, kind, payload_json, hlc_ms, + received_at_ms, tombstone) + VALUES ($1, $2, $3, $4, $5, $6, $7, 1700000000000, FALSE)", + ) + .bind(user_id) + .bind(&op.op_id) + .bind(&op.origin_device_id) + .bind(op.seq) + .bind(kind) + .bind(serde_json::to_value(&op.payload).unwrap()) + .bind(op.hlc_ms) + .execute(pool) + .await + .unwrap(); +} + +#[tokio::test] +#[ignore = "requires a disposable database in FURU_DEVICE_SYNC_TEST_DATABASE_URL"] +async fn postgres_podcast_migrations_import_the_relay_state() { + let admin_url = std::env::var(DATABASE_URL).unwrap(); + let admin = sqlx::PgPool::connect(&admin_url).await.unwrap(); + let name = format!("furu_podcast_migration_{}", random_hex(6)); + sqlx::query(&format!("CREATE DATABASE {name}")) + .execute(&admin) + .await + .unwrap(); + let (base, _) = admin_url.rsplit_once('/').unwrap(); + let url = format!("{base}/{name}"); + + // A 0.13.0 database: every migration up to the relay table. + let new_migrations = ["m_0056_create_podcast_store", "m_0057_import_podcast_relay"]; + let old: Vec<_> = all_migrations() + .into_iter() + .filter(|migration| { + !(migration.app_name() == "furumusic" && new_migrations.contains(&migration.name())) + }) + .collect(); + migrate(&url, old).await; + let pool = sqlx::PgPool::connect(&url).await.unwrap(); + + // State relayed by 0.13.0 for two accounts. + let episode = test_episode(); + let user_a = 1; + let user_b = 2; + let subscription = SubscriptionRecord { + podcast_id: episode.podcast_id.clone(), + subscribed: true, + podcast: PodcastMeta { + feed_url: episode.feed_url.clone(), + title: episode.podcast_title.clone(), + author: Some("Host".into()), + description: None, + link: None, + image_url: Some("https://img.example/show.jpg".into()), + }, + feed_aliases: vec![episode.feed_url.clone(), feed_url("old.xml")], + generation: 3, + hlc_ms: 30, + op_id: "tui_a:3".into(), + }; + let share = ShareRecord { + podcast_id: episode.podcast_id.clone(), + shared: false, + generation: 2, + hlc_ms: 20, + op_id: "tui_a:2".into(), + }; + let progress = ProgressRecord { + episode_id: episode.episode_id.clone(), + podcast_id: episode.podcast_id.clone(), + position_ms: 1_200_000, + status: ProgressStatus::InProgress, + generation: 9, + session_id: "tui_a-g9".into(), + hlc_ms: 90, + device_id: "tui_a".into(), + op_id: "tui_a:9".into(), + }; + let listened = session("s9", "tui_a", &episode, 120_000, true); + let pending_podcast = PodcastId::from_feed_url(&feed_url("pending.xml")); + insert_relay( + &pool, + user_a, + "subscription", + subscription.podcast_id.as_str(), + serde_json::to_value(&subscription).unwrap(), + ) + .await; + insert_relay( + &pool, + user_a, + "share", + share.podcast_id.as_str(), + serde_json::to_value(&share).unwrap(), + ) + .await; + insert_relay( + &pool, + user_a, + "progress", + progress.episode_id.as_str(), + serde_json::to_value(&progress).unwrap(), + ) + .await; + insert_relay( + &pool, + user_a, + "session", + "s9", + serde_json::to_value(&listened).unwrap(), + ) + .await; + insert_relay( + &pool, + user_a, + "aliases", + pending_podcast.as_str(), + serde_json::to_value(vec![feed_url("pending.xml")]).unwrap(), + ) + .await; + // An out-of-range generation (no conforming client) is skipped. + let huge = ProgressRecord { + episode_id: EpisodeId::from_guid(&episode.podcast_id, "huge").unwrap(), + generation: u64::MAX, + ..progress.clone() + }; + insert_relay( + &pool, + user_a, + "progress", + huge.episode_id.as_str(), + serde_json::to_value(&huge).unwrap(), + ) + .await; + insert_relay( + &pool, + user_b, + "progress", + progress.episode_id.as_str(), + serde_json::to_value(&ProgressRecord { + position_ms: 5, + ..progress.clone() + }) + .unwrap(), + ) + .await; + // The relayed operations kept the episode references (two versions). + let old_ref = PodcastEpisodeRef { + episode_title: "Old title".into(), + progress_generation: None, + ..episode.clone() + }; + insert_logged_op( + &pool, + user_a, + &progress_op("tui_a", 8, 80, &old_ref, 8, 1_000), + "podcast_progress_set", + ) + .await; + insert_logged_op( + &pool, + user_a, + &progress_op("tui_a", 9, 90, &episode, 9, 1_200_000), + "podcast_progress_set", + ) + .await; + + // Upgrade. + migrate(&url, all_migrations()).await; + let store = PodcastStore::new(pool.clone()); + + let mut expected = SyncSnapshot { + podcast_subscriptions: vec![subscription.clone()], + podcast_shares: vec![share], + podcast_progress: vec![progress.clone()], + podcast_listen_sessions: vec![listened], + ..SyncSnapshot::default() + }; + expected.bound_podcast_state(); + assert_eq!(store.export_snapshot(user_a).await.unwrap(), expected); + assert_eq!( + store + .export_snapshot(user_b) + .await + .unwrap() + .podcast_progress[0] + .position_ms, + 5 + ); + assert_eq!( + store.feed_aliases(user_a, &pending_podcast).await.unwrap(), + vec![feed_url("pending.xml")] + ); + // Catalog rows: the podcast from its register, the episode from the + // newest reference in the log. + let summary = store + .podcast(user_a, &episode.podcast_id) + .await + .unwrap() + .unwrap(); + assert!(summary.subscribed); + assert_eq!( + summary.meta.image_url.as_deref(), + Some("https://img.example/show.jpg") + ); + let item = store + .episode(user_a, &episode.episode_id) + .await + .unwrap() + .unwrap(); + assert_eq!(item.episode.title, episode.episode_title); + assert_eq!(item.episode.season, episode.season); + assert_eq!(item.resume_position_ms(), 1_200_000); + assert_eq!(item.completed_count, 1); + // Imported rows are searchable once their search text is computed. + let mut search = EpisodeQuery::new(episode.podcast_id.clone()); + search.search = Some("episode 42".into()); + assert_eq!(store.episodes(user_a, &search).await.unwrap().len(), 1); + // The store keeps working on imported rows, and the relay table stays. + assert_eq!( + store + .apply_remote_op(user_a, &progress_op("tui_a", 9, 90, &episode, 9, 1_200_000)) + .await + .unwrap(), + ApplyOutcome::Unchanged + ); + let relay_rows: i64 = sqlx::query_scalar("SELECT COUNT(*) FROM furumusic__fed_podcast_relay") + .fetch_one(&pool) + .await + .unwrap(); + assert_eq!(relay_rows, 7); + // Running the import again (or the migrations) changes nothing. + for sql in schema::import_statements("TRUE") { + sqlx::query(&sql).execute(&pool).await.unwrap(); + } + migrate(&url, all_migrations()).await; + assert_eq!(store.export_snapshot(user_a).await.unwrap(), expected); + + pool.close().await; + sqlx::query(&format!("DROP DATABASE {name} WITH (FORCE)")) + .execute(&admin) + .await + .unwrap(); +} + +/// Alias sets are compared and exported in byte order (as +/// `merge_feed_aliases` returns them) whatever the database collation is: +/// an ICU database sorts "a" before "B", bytes sort "B" first. +#[tokio::test] +#[ignore = "requires a disposable database in FURU_DEVICE_SYNC_TEST_DATABASE_URL"] +async fn postgres_podcast_alias_sets_ignore_the_database_collation() { + let admin_url = std::env::var(DATABASE_URL).unwrap(); + let admin = sqlx::PgPool::connect(&admin_url).await.unwrap(); + let name = format!("furu_podcast_icu_{}", random_hex(6)); + sqlx::query(&format!( + "CREATE DATABASE {name} TEMPLATE template0 LOCALE_PROVIDER icu ICU_LOCALE 'en-US' LOCALE 'C'" + )) + .execute(&admin) + .await + .unwrap(); + let (base, _) = admin_url.rsplit_once('/').unwrap(); + let url = format!("{base}/{name}"); + migrate(&url, all_migrations()).await; + let pool = sqlx::PgPool::connect(&url).await.unwrap(); + let ordered: Vec = + sqlx::query_scalar("SELECT array_agg(x ORDER BY x) FROM (VALUES ('B'), ('a')) v(x)") + .fetch_one(&pool) + .await + .unwrap(); + assert_eq!(ordered, ["a", "B"], "the database must not sort bytes"); + + let store = PodcastStore::new(pool.clone()); + let user_id = 1; + let feed = feed_url("feed.xml"); + let podcast_id = PodcastId::from_feed_url(&feed); + let mixed = vec![feed_url("a"), feed_url("B"), feed_url("a-c")]; + let mut sorted = mixed.clone(); + sorted.push(feed.clone()); + sorted.sort(); + let subscribe = op( + "tui_a", + 1, + 100, + SyncOpPayload::PodcastSubscriptionSet { + podcast_id: podcast_id.clone(), + subscribed: true, + podcast: PodcastMeta { + feed_url: feed.clone(), + title: "Refresh Show".into(), + author: None, + description: None, + link: None, + image_url: None, + }, + feed_aliases: mixed.clone(), + generation: 1, + }, + ); + let aliases = op( + "tui_a", + 2, + 200, + SyncOpPayload::PodcastFeedAliasesAdded { + podcast_id: podcast_id.clone(), + feed_aliases: mixed.clone(), + }, + ); + assert_eq!( + store.apply_remote_op(user_id, &subscribe).await.unwrap(), + ApplyOutcome::Changed + ); + for op in [&subscribe, &aliases, &aliases, &subscribe] { + assert_eq!( + store.apply_remote_op(user_id, op).await.unwrap(), + ApplyOutcome::Unchanged, + "{}", + op.op_id + ); + } + assert_eq!( + store.feed_aliases(user_id, &podcast_id).await.unwrap(), + sorted + ); + let snapshot = store.export_snapshot(user_id).await.unwrap(); + assert_eq!(snapshot.podcast_subscriptions[0].feed_aliases, sorted); + + // A permanent move to an alias already known emits nothing. + let document = parse( + &rss(r#" + g-1One + "#), + &feed_url("B"), + ); + let refresh = store + .apply_feed( + user_id, + &FeedRefresh { + podcast_id: &podcast_id, + feed: &document, + fetched_from: &feed_url("B"), + moved_permanently: true, + }, + ) + .await + .unwrap(); + assert!(refresh.ops.is_empty(), "{:?}", refresh.ops); + assert_eq!( + store.feed_aliases(user_id, &podcast_id).await.unwrap(), + sorted + ); + + pool.close().await; + sqlx::query(&format!("DROP DATABASE {name} WITH (FORCE)")) + .execute(&admin) + .await + .unwrap(); +} + +// --------------------------------------------------------------------------- +// RSS refresh job +// --------------------------------------------------------------------------- + +fn refresh_item(guid: &str) -> String { + format!( + r#"Episode {guid}{guid} + "# + ) +} + +/// A fetcher that may reach the local test server (loopback is allowed +/// only by the test policy). +fn local_fetcher() -> FeedFetcher { + FeedFetcher::with_http( + net::SafeHttp::with_lookup( + fetch::FEED_HTTP, + net::AddressPolicy::test_allow_loopback(), + std::sync::Arc::new(net::SystemLookup), + ) + .unwrap(), + ) +} + +#[tokio::test] +#[ignore = "requires a disposable database in FURU_DEVICE_SYNC_TEST_DATABASE_URL"] +async fn postgres_podcast_refresh_takes_accounts_in_turns() { + let pool = test_pool().await; + let store = PodcastStore::new(pool.clone()); + let (busy, quiet) = (random_user(), random_user()); + let now = 1_900_000_000_000; + let mut busy_ids = Vec::new(); + for (index, path) in ["turn-a.xml", "turn-b.xml", "turn-c.xml"] + .iter() + .enumerate() + { + let id = store + .subscribe(busy, &meta(path, "Busy")) + .await + .unwrap() + .podcast_id; + // The busy account's podcasts are the most overdue ones. + store + .record_fetch_failure(busy, &id, now, "x", Some(now - 10_000 + index as i64)) + .await + .unwrap(); + busy_ids.push(id); + } + let quiet_id = store + .subscribe(quiet, &meta("turn-q.xml", "Quiet")) + .await + .unwrap() + .podcast_id; + store + .record_fetch_failure(quiet, &quiet_id, now, "x", Some(now - 1)) + .await + .unwrap(); + let due = store + .podcasts_due_for_refresh(now, 100_000) + .await + .unwrap() + .into_iter() + .filter(|(user, ..)| [busy, quiet].contains(user)) + .map(|(user, id, _)| (user, id)) + .collect::>(); + // One turn per account: the quiet account's only feed comes before the + // busy account's second one, and the busy account still gets all of + // its due feeds when the run has room (no per-account cap). + assert_eq!( + due, + vec![ + (busy, busy_ids[0].clone()), + (quiet, quiet_id.clone()), + (busy, busy_ids[1].clone()), + (busy, busy_ids[2].clone()), + ], + ); + // With only one account due, a run takes up to the run limit from it. + let alone = store + .podcasts_due_for_refresh(now, 100_000) + .await + .unwrap() + .into_iter() + .filter(|(user, ..)| *user == busy) + .count(); + assert_eq!(alone, 3); + + // New subscriptions stop at the per-account limit. + sqlx::query( + "INSERT INTO furumusic__podcast_subscription + (user_id, podcast_id, subscribed, podcast_json, generation, hlc_ms, op_id) + SELECT $1, 'pod1:cap' || n, TRUE, '{}'::jsonb, 1, 1, 'x:' || n + FROM generate_series(1, $2) AS n", + ) + .bind(quiet) + .bind(i32::try_from(MAX_SUBSCRIPTIONS_PER_ACCOUNT - 1).unwrap()) + .execute(&pool) + .await + .unwrap(); + let error = store + .subscribe(quiet, &meta("turn-over.xml", "Over")) + .await + .unwrap_err(); + assert_eq!( + error.downcast_ref::(), + Some(&PodcastStoreError::SubscriptionLimit( + MAX_SUBSCRIPTIONS_PER_ACCOUNT + )) + ); + // Subscribing again to a podcast already subscribed is not new. + assert!( + store + .subscribe(quiet, &meta("turn-q.xml", "Quiet")) + .await + .unwrap() + .already_subscribed + ); + + cleanup(&pool, [busy, quiet]).await; +} + +/// Due groups of the given accounts only (other tests share the database). +async fn due_for(store: &PodcastStore, users: &[i64], now: i64) -> runner::RefreshGroups { + let due = store + .podcasts_due_for_refresh(now, 100_000) + .await + .unwrap() + .into_iter() + .filter(|(user_id, _, _)| users.contains(user_id)) + .collect(); + runner::group_targets(due) +} + +async fn run_due( + store: &PodcastStore, + fetcher: &FeedFetcher, + users: &[i64], + now: i64, +) -> runner::RunSummary { + let groups = due_for(store, users, now).await; + runner::refresh_groups(store, fetcher, groups, now, 4) + .await + .unwrap() +} + +#[tokio::test] +#[ignore = "requires a disposable database in FURU_DEVICE_SYNC_TEST_DATABASE_URL"] +async fn postgres_podcast_refresh_job_schedules_backs_off_and_moves_feeds() { + use super::runner::{BACKOFF_BASE_MS, ManualRefreshError, TargetResult}; + use super::test_server::{Route, TestServer}; + + let pool = test_pool().await; + let store = PodcastStore::new(pool.clone()); + let (a, b, c) = (random_user(), random_user(), random_user()); + cleanup(&pool, [a, b, c]).await; + let users = [a, b, c]; + let server = TestServer::start(); + server.set( + "/show.xml", + Route::new(rss(&refresh_item("one")), "application/rss+xml").etag("\"v1\""), + ); + let show = PodcastMeta { + feed_url: FeedUrl::parse(&server.url("/show.xml")).unwrap(), + title: "Local Show".into(), + author: None, + description: None, + link: None, + image_url: None, + }; + let podcast_id = store.subscribe(a, &show).await.unwrap().podcast_id; + assert_eq!( + store.subscribe(b, &show).await.unwrap().podcast_id, + podcast_id + ); + let fetcher = local_fetcher(); + + // First run: one request serves both accounts. + let t0 = super::now_ms(); + let summary = run_due(&store, &fetcher, &users, t0).await; + assert_eq!((summary.targets, summary.feeds), (2, 1), "{summary}"); + assert_eq!((summary.updated, summary.new_episodes), (2, 2), "{summary}"); + assert_eq!(server.requests("/show.xml").len(), 1); + for user in [a, b] { + let state = store.fetch_state(user, &podcast_id).await.unwrap(); + assert_eq!(state.etag.as_deref(), Some("\"v1\"")); + assert_eq!(state.consecutive_failures, 0); + let next = state.next_check_at_ms.unwrap(); + assert!((t0 + runner::REFRESH_INTERVAL_MS..t0 + 70 * 60 * 1000).contains(&next)); + } + let episodes = store + .episodes(a, &EpisodeQuery::new(podcast_id.clone())) + .await + .unwrap(); + assert_eq!(episodes.len(), 1); + let episode_id = episodes[0].episode.episode_id.clone(); + assert_eq!(episodes[0].episode.audio_url, server.url("/audio/one.mp3")); + + // The proxy and the source endpoint only find episodes of the account. + assert_eq!( + store.audio_source(a, &episode_id).await.unwrap(), + Some((server.url("/audio/one.mp3"), Some("audio/mpeg".to_string()))) + ); + assert_eq!(store.audio_source(c, &episode_id).await.unwrap(), None); + + // Nothing is due before the interval; afterwards the request is + // conditional and the catalog stays as it is. + assert!(due_for(&store, &users, t0 + 1000).await.is_empty()); + let t1 = t0 + 2 * runner::REFRESH_INTERVAL_MS; + let summary = run_due(&store, &fetcher, &users, t1).await; + assert_eq!(summary.not_modified, 2, "{summary}"); + assert_eq!(server.requests("/show.xml")[1]["if-none-match"], "\"v1\""); + + // Failures back off exponentially and keep the catalog. + server.set("/show.xml", Route::new("boom", "text/plain").status(500)); + let t2 = t1 + 2 * runner::REFRESH_INTERVAL_MS; + let summary = run_due(&store, &fetcher, &users, t2).await; + assert_eq!(summary.failed, 2, "{summary}"); + assert_eq!(summary.failures.len(), 2); + let state = store.fetch_state(a, &podcast_id).await.unwrap(); + assert_eq!(state.consecutive_failures, 1); + assert_eq!(state.next_check_at_ms, Some(t2 + BACKOFF_BASE_MS)); + assert!(state.last_error.as_deref().unwrap().contains("500")); + assert_eq!(state.etag.as_deref(), Some("\"v1\""), "validators are kept"); + assert!( + due_for(&store, &users, t2 + BACKOFF_BASE_MS - 1) + .await + .is_empty() + ); + let t3 = t2 + BACKOFF_BASE_MS; + let summary = run_due(&store, &fetcher, &users, t3).await; + assert_eq!(summary.failed, 2, "{summary}"); + let state = store.fetch_state(a, &podcast_id).await.unwrap(); + assert_eq!(state.consecutive_failures, 2); + assert_eq!(state.next_check_at_ms, Some(t3 + 2 * BACKOFF_BASE_MS)); + assert_eq!( + store + .episodes(a, &EpisodeQuery::new(podcast_id.clone())) + .await + .unwrap() + .len(), + 1 + ); + + // Recovery resets the backoff. Accounts with different validators are + // fetched unconditionally. + server.set( + "/show.xml", + Route::new( + rss(&format!("{}{}", refresh_item("two"), refresh_item("one"))), + "application/rss+xml", + ) + .etag("\"v2\""), + ); + store + .record_fetch_success(b, &podcast_id, t3, Some("\"other\""), None, Some(t3)) + .await + .unwrap(); + let t4 = t3 + 2 * BACKOFF_BASE_MS; + let summary = run_due(&store, &fetcher, &users, t4).await; + assert_eq!((summary.updated, summary.new_episodes), (2, 2), "{summary}"); + let last = server.requests("/show.xml").pop().unwrap(); + assert!(!last.contains_key("if-none-match"), "{last:?}"); + let state = store.fetch_state(a, &podcast_id).await.unwrap(); + assert_eq!((state.consecutive_failures, state.last_error), (0, None)); + assert_eq!(state.etag.as_deref(), Some("\"v2\"")); + + // Manual refresh: not more often than every 30 s, only when subscribed. + let manual = runner::refresh_now(&store, &fetcher, a, &podcast_id, t4 + 10_000).await; + assert_eq!( + manual.unwrap_err().downcast_ref::(), + Some(&ManualRefreshError::TooSoon { + retry_after_ms: 20_000 + }) + ); + let manual = runner::refresh_now(&store, &fetcher, a, &podcast_id, t4 + 31_000) + .await + .unwrap(); + assert_eq!(manual, TargetResult::NotModified); + let unknown = runner::refresh_now(&store, &fetcher, c, &podcast_id, t4 + 31_000).await; + assert_eq!( + unknown.unwrap_err().downcast_ref::(), + Some(&ManualRefreshError::UnknownPodcast) + ); + store.unsubscribe(b, &podcast_id).await.unwrap(); + let unsubscribed = runner::refresh_now(&store, &fetcher, b, &podcast_id, t4 + 31_000).await; + assert_eq!( + unsubscribed + .unwrap_err() + .downcast_ref::(), + Some(&ManualRefreshError::NotSubscribed) + ); + let far = t4 + 10 * runner::REFRESH_INTERVAL_MS; + let due = due_for(&store, &users, far).await; + assert_eq!( + due.values().flatten().count(), + 1, + "unsubscribing stops monitoring" + ); + + // A permanent move becomes an alias (one sync operation) and the new + // fetch URL; later refreshes go to the new URL without new operations. + server.set("/show.xml", Route::redirect(301, "/moved.xml")); + server.set( + "/moved.xml", + Route::new(rss(&refresh_item("one")), "application/rss+xml"), + ); + let moved = FeedUrl::parse(&server.url("/moved.xml")).unwrap(); + let result = runner::refresh_now(&store, &fetcher, a, &podcast_id, far) + .await + .unwrap(); + let TargetResult::Updated(outcome) = result else { + panic!("{result:?}"); + }; + assert_eq!(outcome.feed_url.as_ref(), Some(&moved)); + assert_eq!(outcome.ops.len(), 1); + assert_eq!( + store + .podcast(a, &podcast_id) + .await + .unwrap() + .unwrap() + .meta + .feed_url, + moved + ); + let due = due_for(&store, &users, far + 10 * runner::REFRESH_INTERVAL_MS).await; + assert_eq!(due.keys().collect::>(), vec![&moved]); + let summary = run_due( + &store, + &fetcher, + &users, + far + 10 * runner::REFRESH_INTERVAL_MS, + ) + .await; + assert_eq!(summary.updated, 1, "{summary}"); + assert_eq!(server.requests("/moved.xml").len(), 2); + assert_eq!(server.requests("/show.xml").len(), 7); + let TargetResult::Updated(again) = runner::refresh_now( + &store, + &fetcher, + a, + &podcast_id, + far + 20 * runner::REFRESH_INTERVAL_MS, + ) + .await + .unwrap() else { + panic!("expected a document"); + }; + assert!(again.ops.is_empty()); + // The episode keeps its id; its audio now resolves against the new URL + // (same host here). + let after_move = store.episode(a, &episode_id).await.unwrap().unwrap(); + assert_eq!(after_move.episode.audio_url, server.url("/audio/one.mp3")); + + cleanup(&pool, [a, b, c]).await; +} + +// --------------------------------------------------------------------------- +// HTTP API of the podcast UI and the web player (`api`, behind `http`) +// --------------------------------------------------------------------------- + +async fn logged_kinds(pool: &sqlx::PgPool, user_id: i64) -> Vec { + sqlx::query_scalar("SELECT kind FROM furumusic__fed_sync_ops WHERE user_id = $1 ORDER BY seq") + .bind(user_id) + .fetch_all(pool) + .await + .unwrap() +} + +#[tokio::test] +#[ignore = "requires a disposable database in FURU_DEVICE_SYNC_TEST_DATABASE_URL"] +async fn postgres_podcast_api_subscribes_plays_and_keeps_progress() { + use super::api::{self, EpisodeAction, EpisodeListParams}; + use super::test_server::{Route, TestServer}; + use serde_json::json; + + let pool = test_pool().await; + let store = PodcastStore::new(pool.clone()); + let (user, other) = (random_user(), random_user()); + cleanup(&pool, [user, other]).await; + let server = TestServer::start(); + let items = r#" + Pilots1e11 + 1600 + Mon, 01 Jan 2024 10:00:00 GMTFirst interview + + Bonusbonus120 + Tue, 02 Jan 2024 10:00:00 GMT + "#; + server.set("/show.xml", Route::new(rss(items), "application/rss+xml")); + server.set("/page.html", Route::new("", "text/html")); + server.set("/gone.xml", Route::new("gone", "text/plain").status(404)); + let fetcher = local_fetcher(); + let now = super::now_ms(); + + // "Add RSS": invalid input and unusable feeds subscribe nothing. + for (body, status) in [ + (json!({}), 400), + (json!({"url": "ftp://example.org/feed"}), 400), + (json!({"url": server.url("/page.html")}), 422), + (json!({"url": server.url("/gone.xml")}), 422), + ] { + let error = api::subscribe_url(&store, &fetcher, user, body.clone(), now) + .await + .unwrap_err(); + assert_eq!(error.status, status, "{body}: {}", error.message); + } + // The production policy never reaches the local server. + let refused = api::subscribe_url( + &store, + &FeedFetcher::new().unwrap(), + user, + json!({"url": server.url("/show.xml")}), + now, + ) + .await + .unwrap_err(); + assert_eq!(refused.status, 422, "{}", refused.message); + assert_eq!( + refused.message, + "the feed could not be loaded: only public http and https addresses on ports 80, 443, \ + 8080 and 8443 are allowed" + ); + assert!(logged_kinds(&pool, user).await.is_empty()); + assert_eq!(server.requests("/show.xml").len(), 0); + + let added = api::subscribe_url( + &store, + &fetcher, + user, + json!({"url": format!(" {} ", server.url("/show.xml"))}), + now, + ) + .await + .unwrap(); + assert!(!added.already_subscribed); + assert_eq!(added.new_episodes, 2); + let podcast = added.podcast; + assert!(podcast.subscribed); + assert!(!podcast.shared, "a new subscription is private"); + assert_eq!((podcast.episode_count, podcast.unplayed_count), (2, 2)); + assert_eq!(podcast.title, "Refresh Show"); + assert!(podcast.fetch.last_success_at_ms.is_some()); + assert!(podcast.fetch.next_check_at_ms.unwrap() > now); + let podcast_id = PodcastId::parse(&podcast.podcast_id).unwrap(); + assert_eq!( + logged_kinds(&pool, user).await, + ["podcast_subscription_set"] + ); + // Adding it again writes nothing. + let again = api::subscribe_url( + &store, + &fetcher, + user, + json!({"url": server.url("/show.xml")}), + now, + ) + .await + .unwrap(); + assert!(again.already_subscribed); + assert_eq!(again.podcast.podcast_id, podcast.podcast_id); + assert_eq!(logged_kinds(&pool, user).await.len(), 1); + + // Lists and details are per account. + let list = api::list_podcasts(&store, user, false).await.unwrap(); + assert_eq!(list.podcasts.len(), 1); + assert!( + api::list_podcasts(&store, other, true) + .await + .unwrap() + .podcasts + .is_empty() + ); + assert_eq!( + api::podcast_detail(&store, other, &podcast_id) + .await + .unwrap_err() + .status, + 404 + ); + let detail = api::podcast_detail(&store, user, &podcast_id) + .await + .unwrap(); + let seasons: Vec> = detail.seasons.iter().map(|season| season.season).collect(); + assert_eq!(seasons, [Some(1), None]); + assert!(detail.history.is_empty()); + + // Episodes: newest first, filters, search, paging, queue items. + let page = api::episodes(&store, user, &podcast_id, &EpisodeListParams::default()) + .await + .unwrap(); + let titles: Vec<&str> = page.episodes.iter().map(|e| e.title.as_str()).collect(); + assert_eq!(titles, ["Bonus", "Pilot"]); + assert!(!page.has_more); + let pilot = &page.episodes[1]; + assert_eq!((pilot.season, pilot.episode_number), (Some(1), Some(1))); + assert_eq!(pilot.status, "not_started"); + assert_eq!(pilot.track["source"]["kind"], "podcast_episode"); + assert_eq!( + pilot.track["source"]["episode_id"], + pilot.episode_id.as_str() + ); + assert_eq!(pilot.track["unavailable"], false); + assert_eq!(pilot.track["title"], "Pilot"); + let first = api::episodes( + &store, + user, + &podcast_id, + &EpisodeListParams { + limit: Some(1), + ..Default::default() + }, + ) + .await + .unwrap(); + assert_eq!((first.episodes.len(), first.has_more), (1, true)); + let season = api::episodes( + &store, + user, + &podcast_id, + &EpisodeListParams { + season: Some("none".into()), + ..Default::default() + }, + ) + .await + .unwrap(); + assert_eq!(season.episodes.len(), 1); + assert_eq!(season.episodes[0].title, "Bonus"); + let found = api::episodes( + &store, + user, + &podcast_id, + &EpisodeListParams { + q: Some("INTERVIEW".into()), + ..Default::default() + }, + ) + .await + .unwrap(); + assert_eq!(found.episodes.len(), 1, "description search, any case"); + let episode_id = EpisodeId::parse(&pilot.episode_id).unwrap(); + + // Playback: the source and the resume position, without writing. + let prepared = api::playback(&store, user, &episode_id, json!({}), false) + .await + .unwrap(); + assert_eq!(prepared.direct_url, server.url("/audio/s1e1.mp3")); + assert_eq!(prepared.proxy_url, None); + assert_eq!((prepared.position_ms, prepared.generation), (0, None)); + assert_eq!(prepared.duration_ms, Some(600_000)); + let proxied = api::playback(&store, user, &episode_id, json!({}), true) + .await + .unwrap(); + assert_eq!( + proxied.proxy_url.as_deref(), + Some(format!("/api/podcasts/episodes/{episode_id}/audio").as_str()) + ); + assert_eq!(logged_kinds(&pool, user).await.len(), 1); + assert_eq!( + api::playback(&store, user, &episode_id, json!({"start": true}), false) + .await + .unwrap_err() + .status, + 400, + "starting needs a session id" + ); + + // Play starts a session (explicit generation 1). + let started = api::playback( + &store, + user, + &episode_id, + json!({"start": true, "session_id": "web-s1"}), + false, + ) + .await + .unwrap(); + assert_eq!(started.session_id.as_deref(), Some("web-s1")); + assert_eq!((started.generation, started.position_ms), (Some(1), 0)); + let report = |body: serde_json::Value| { + let store = store.clone(); + let episode_id = episode_id.clone(); + async move { api::progress(&store, user, &episode_id, body, super::now_ms()).await } + }; + let t0 = super::now_ms() - 120_000; + let saved = report(json!({ + "session_id": "web-s1", "generation": 1, "event": "checkpoint", "position_ms": 60_000, + "duration_ms": 600_000, + })) + .await + .unwrap(); + assert_eq!( + (saved.outcome, saved.generation, saved.status), + ("written", 1, "in_progress") + ); + assert!( + !saved.session_recorded, + "checkpoints do not record the session" + ); + // Pause records the session (at least 30 s listened). + let paused = report(json!({ + "session_id": "web-s1", "generation": 1, "event": "pause", "position_ms": 90_000, + "listened_ms": 90_000, "started_at_ms": t0, "start_position_ms": 0, + })) + .await + .unwrap(); + assert!(paused.session_recorded); + // Seeking back is an explicit action; a late checkpoint at the old + // generation is superseded. + let sought = report(json!({ + "session_id": "web-s1", "generation": 1, "event": "seek", "position_ms": 30_000, + })) + .await + .unwrap(); + assert_eq!( + (sought.outcome, sought.generation, sought.position_ms), + ("written", 2, 30_000) + ); + let late = report(json!({ + "session_id": "web-s1", "generation": 1, "event": "checkpoint", "position_ms": 95_000, + })) + .await + .unwrap(); + assert_eq!( + (late.outcome, late.generation, late.position_ms), + ("superseded", 2, 30_000) + ); + // Ordinary Play resumes from the register. + let resumed = api::playback(&store, user, &episode_id, json!({}), false) + .await + .unwrap(); + assert_eq!( + (resumed.position_ms, resumed.status), + (30_000, "in_progress") + ); + // Reaching the completion tail completes once, with the session. + let near_end = report(json!({ + "session_id": "web-s1", "generation": 2, "event": "checkpoint", "position_ms": 590_000, + })) + .await + .unwrap(); + assert_eq!( + (near_end.outcome, near_end.status, near_end.completed), + ("written", "completed", true) + ); + assert_eq!(near_end.generation, 3); + let ended = report(json!({ + "session_id": "web-s1", "generation": 3, "event": "ended", "position_ms": 600_000, + "listened_ms": 400_000, "started_at_ms": t0, "start_position_ms": 0, + })) + .await + .unwrap(); + assert_eq!(ended.outcome, "already_completed"); + assert!( + ended.session_recorded, + "the session grew and is now completed" + ); + let item = store.episode(user, &episode_id).await.unwrap().unwrap(); + assert_eq!(item.completed_count, 1); + // A completed episode plays from the start. + let replay = api::playback(&store, user, &episode_id, json!({}), false) + .await + .unwrap(); + assert_eq!((replay.position_ms, replay.status), (0, "completed")); + let history = api::podcast_detail(&store, user, &podcast_id) + .await + .unwrap() + .history; + assert_eq!(history.len(), 1); + assert!(history[0].completed); + assert_eq!(history[0].episode_title.as_deref(), Some("Pilot")); + + // Explicit list actions. + let restarted = api::episode_action(&store, user, &episode_id, EpisodeAction::Restart) + .await + .unwrap(); + assert_eq!( + (restarted.status, restarted.position_ms), + ("in_progress", 0) + ); + let played = api::episode_action(&store, user, &episode_id, EpisodeAction::MarkPlayed) + .await + .unwrap(); + assert_eq!(played.status, "completed"); + assert_eq!( + played.completed_count, 1, + "marking played invents no session" + ); + let unplayed = api::episode_action(&store, user, &episode_id, EpisodeAction::MarkUnplayed) + .await + .unwrap(); + assert_eq!((unplayed.status, unplayed.position_ms), ("not_started", 0)); + assert_eq!( + api::episode_action(&store, other, &episode_id, EpisodeAction::MarkPlayed) + .await + .unwrap_err() + .status, + 404 + ); + + // A handed-off queue item the catalog does not know yet: the reference + // is recorded and the session starts above the carried generation. + let mut handed = test_episode(); + handed.progress_generation = Some(9); + let handed_id = handed.episode_id.clone(); + assert_eq!( + api::playback(&store, user, &handed_id, json!({}), false) + .await + .unwrap_err() + .status, + 404 + ); + let mut wrong = handed.clone(); + wrong.episode_id = episode_id.clone(); + assert_eq!( + api::playback(&store, user, &handed_id, json!({"episode": wrong}), false) + .await + .unwrap_err() + .status, + 400 + ); + let taken_over = api::playback( + &store, + user, + &handed_id, + json!({ + "episode": handed, "start": true, "session_id": "web-s2", + "position_ms": 1_200_000, "carried_generation": 9, + }), + false, + ) + .await + .unwrap(); + assert_eq!( + (taken_over.generation, taken_over.position_ms), + (Some(10), 1_200_000) + ); + assert_eq!(taken_over.direct_url, "https://cdn.example/ep42.mp3"); + let catalog = store.episode(user, &handed_id).await.unwrap().unwrap(); + assert!(!catalog.in_feed); + // A handoff away writes a final checkpoint at the session generation. + let away = api::progress( + &store, + user, + &handed_id, + json!({ + "session_id": "web-s2", "generation": 10, "event": "handoff", + "position_ms": 1_260_000, "listened_ms": 10_000, + "started_at_ms": super::now_ms() - 60_000, "start_position_ms": 1_200_000, + }), + super::now_ms(), + ) + .await + .unwrap(); + assert_eq!( + (away.outcome, away.generation, away.position_ms), + ("written", 10, 1_260_000) + ); + assert!( + !away.session_recorded, + "10 s is below the history threshold" + ); + + // Unsubscribing keeps progress and history; subscribing again continues. + let left = api::unsubscribe(&store, user, &podcast_id).await.unwrap(); + assert!(!left.subscribed); + assert!( + api::list_podcasts(&store, user, false) + .await + .unwrap() + .podcasts + .is_empty() + ); + let all = api::list_podcasts(&store, user, true).await.unwrap(); + assert!( + all.podcasts + .iter() + .any(|p| p.podcast_id == podcast.podcast_id) + ); + let kept = store.episode(user, &episode_id).await.unwrap().unwrap(); + assert_eq!(kept.completed_count, 1); + let back = api::resubscribe(&store, user, &podcast_id).await.unwrap(); + assert!(back.subscribed); + assert_eq!( + api::unsubscribe(&store, other, &podcast_id) + .await + .unwrap_err() + .status, + 404 + ); + + // Every write went through the sync log, with its own operation kinds. + let kinds = logged_kinds(&pool, user).await; + let count = |kind: &str| kinds.iter().filter(|k| k.as_str() == kind).count(); + assert_eq!(count("podcast_subscription_set"), 3); + assert_eq!( + count("podcast_share_set"), + 1, + "unsubscribing revokes sharing" + ); + assert_eq!(count("podcast_listen_recorded"), 2); + // start, checkpoint, pause, seek, completion, restart, played, + // unplayed, handed-off start, handoff checkpoint + assert_eq!(count("podcast_progress_set"), 10); + + cleanup(&pool, [user, other]).await; +} + +/// Ordinary Play continues from the progress register, not from a position +/// the browser saved earlier: after a reload the web player prepares the +/// episode without a position and gets what other devices wrote meanwhile. +#[tokio::test] +#[ignore = "requires a disposable database in FURU_DEVICE_SYNC_TEST_DATABASE_URL"] +async fn postgres_podcast_playback_resumes_from_the_register() { + use serde_json::json; + + let pool = test_pool().await; + let store = PodcastStore::new(pool.clone()); + let user = random_user(); + let episode = PodcastEpisodeRef { + progress_generation: None, + ..test_episode() + }; + let episode_id = episode.episode_id.clone(); + let started = api::playback( + &store, + user, + &episode_id, + json!({"episode": episode, "start": true, "session_id": "web-s1", "position_ms": 600_000}), + false, + ) + .await + .unwrap(); + assert_eq!( + (started.generation, started.position_ms), + (Some(1), 600_000) + ); + assert_eq!(started.register_generation, Some(1)); + assert_eq!(started.register_session_id.as_deref(), Some("web-s1")); + let paused = api::progress( + &store, + user, + &episode_id, + json!({"session_id": "web-s1", "generation": 1, "event": "pause", "position_ms": 620_000}), + super::now_ms(), + ) + .await + .unwrap(); + assert_eq!(paused.outcome, "written"); + let prepare = || api::playback(&store, user, &episode_id, json!({}), false); + let mine = prepare().await.unwrap(); + assert_eq!(mine.position_ms, 620_000); + assert_eq!(mine.register_session_id.as_deref(), Some("web-s1")); + + // The phone listened further offline; its progress arrives later. + store + .apply_remote_op(user, &progress_op("phone", 1, 50, &episode, 2, 2_400_000)) + .await + .unwrap(); + let moved = prepare().await.unwrap(); + assert_eq!(moved.position_ms, 2_400_000, "the register, not 620 000"); + assert_eq!(moved.register_generation, Some(2)); + assert_eq!(moved.register_session_id.as_deref(), Some("phone-g2")); + assert_eq!(moved.generation, None, "preparing writes nothing"); + let resumed = api::playback( + &store, + user, + &episode_id, + json!({"start": true, "session_id": "web-s2"}), + false, + ) + .await + .unwrap(); + assert_eq!( + (resumed.generation, resumed.position_ms), + (Some(3), 2_400_000) + ); + assert_eq!(resumed.register_session_id.as_deref(), Some("web-s2")); + + // Marked played elsewhere: Play starts over. + let reference = store.episode_ref(user, &episode_id).await.unwrap().unwrap(); + store.mark_played(user, &reference).await.unwrap(); + let played = prepare().await.unwrap(); + assert_eq!((played.status, played.position_ms), ("completed", 0)); + + cleanup(&pool, [user]).await; +} + +/// The browser can add episodes outside the account's feeds (handed-off +/// queue items) only up to a limit; the account's own devices are not +/// limited, and known episodes always play. +#[tokio::test] +#[ignore = "requires a disposable database in FURU_DEVICE_SYNC_TEST_DATABASE_URL"] +async fn postgres_podcast_episode_references_from_the_browser_are_bounded() { + use serde_json::json; + + let pool = test_pool().await; + let store = PodcastStore::new(pool.clone()); + let user = random_user(); + let first = PodcastEpisodeRef { + progress_generation: None, + ..test_episode() + }; + let reference = |guid: &str| PodcastEpisodeRef { + episode_id: EpisodeId::from_guid(&first.podcast_id, guid).unwrap(), + episode_title: guid.into(), + ..first.clone() + }; + let play = |episode: PodcastEpisodeRef| { + let store = store.clone(); + async move { + api::playback( + &store, + user, + &episode.episode_id.clone(), + json!({ "episode": episode }), + false, + ) + .await + } + }; + play(first.clone()).await.unwrap(); + // Fill up to one below the limit with copies of that reference row. + sqlx::query( + "INSERT INTO furumusic__podcast_episode + SELECT (jsonb_populate_record(e, jsonb_build_object('episode_id', 'cap-' || n))).* + FROM furumusic__podcast_episode e, generate_series(1, $3) AS n + WHERE e.user_id = $1 AND e.episode_id = $2", + ) + .bind(user) + .bind(first.episode_id.as_str()) + .bind(i32::try_from(MAX_REFERENCE_EPISODES_PER_ACCOUNT - 2).unwrap()) + .execute(&pool) + .await + .unwrap(); + play(reference("last-allowed")).await.unwrap(); + let refused = play(reference("one-too-many")).await.unwrap_err(); + assert_eq!(refused.status, 422, "{}", refused.message); + let unknown = reference("one-too-many").episode_id; + assert!(store.episode_ref(user, &unknown).await.unwrap().is_none()); + // Episodes already known still play. + play(first.clone()).await.unwrap(); + play(reference("last-allowed")).await.unwrap(); + // References in the account's own devices' operations are applied. + let synced = reference("from-the-phone"); + store + .apply_remote_op(user, &progress_op("phone", 1, 50, &synced, 1, 60_000)) + .await + .unwrap(); + assert!( + store + .episode_ref(user, &synced.episode_id) + .await + .unwrap() + .is_some() + ); + + cleanup(&pool, [user]).await; +} diff --git a/src/podcasts/proxy.rs b/src/podcasts/proxy.rs new file mode 100644 index 0000000..b9c24f1 --- /dev/null +++ b/src/podcasts/proxy.rs @@ -0,0 +1,537 @@ +//! Optional audio proxy: streams a stored episode enclosure from the +//! publisher for browsers that cannot load it directly (CORS, mixed +//! content). Audio is passed through, never stored or imported. +//! +//! The caller resolves the URL from the database for an episode of the +//! authenticated account; a client never supplies a URL. Requests go +//! through [`SafeHttp`] (SSRF guard, every redirect hop checked), carry +//! only `Range`, `Accept`, `Accept-Encoding: identity` and the user agent, +//! and responses are limited in size, time and content type. The content +//! type is decided by the publisher's response (and, for generic binary +//! types, the URL extension or the file's first bytes), never by the +//! enclosure type stored with the episode. + +use std::fmt; +use std::time::Duration; + +use bytes::Bytes; +use futures_util::StreamExt; +use futures_util::stream::BoxStream; +use reqwest::header::{ + ACCEPT, ACCEPT_ENCODING, ACCEPT_RANGES, CONTENT_LENGTH, CONTENT_RANGE, CONTENT_TYPE, HeaderMap, + HeaderValue, RANGE, +}; +use reqwest::{StatusCode, Url}; + +use super::net::{GuardError, REFUSED_MESSAGE, SafeHttp, SafeHttpConfig}; + +/// Largest response body passed through (a long episode at a high bitrate +/// stays far below). +pub(crate) const MAX_PROXY_BYTES: u64 = 1024 * 1024 * 1024; +/// Upstream must answer with headers within this time. +pub(crate) const PROXY_HEADER_TIMEOUT: Duration = Duration::from_secs(20); +/// Longest a single proxied response may stream. +pub(crate) const PROXY_MAX_DURATION: Duration = Duration::from_secs(4 * 60 * 60); +/// Redirect hops followed (enclosures often go through tracking redirects). +pub(crate) const MAX_PROXY_REDIRECTS: usize = 5; + +/// HTTP client settings of the proxy (no whole-request timeout: bodies are +/// streamed; the idle read timeout and [`PROXY_MAX_DURATION`] bound them). +pub(crate) const PROXY_HTTP: SafeHttpConfig = SafeHttpConfig { + connect_timeout: Duration::from_secs(10), + read_timeout: Duration::from_secs(30), + total_timeout: None, + max_redirects: MAX_PROXY_REDIRECTS, +}; + +/// Why a proxied request failed. +#[derive(Debug)] +pub(crate) enum ProxyError { + /// The stored URL is not a valid URL. + InvalidUrl(String), + /// Refused or failed in transport (see [`GuardError`]). + Upstream(GuardError), + /// The publisher answered with an unusable status. + Status(u16), + /// The publisher's content type is not audio. + ContentType(String), + /// Declared body larger than [`MAX_PROXY_BYTES`]. + TooLarge(u64), + /// No response headers within [`PROXY_HEADER_TIMEOUT`]. + Timeout, + /// The body stream stopped (limit reached or upstream failure). + Body(String), +} + +impl ProxyError { + /// Status the proxy answers the browser with. + pub(crate) fn http_status(&self) -> u16 { + match self { + Self::Timeout => 504, + Self::Upstream(error) if error.is_timeout() => 504, + _ => 502, + } + } + + /// What the browser is told; the details (`Display`) stay in the + /// server log. + pub(crate) fn public_message(&self) -> String { + match self { + Self::InvalidUrl(_) => "the episode's audio URL is invalid".to_string(), + Self::Upstream(error) if error.is_refusal() => REFUSED_MESSAGE.to_string(), + Self::Upstream(error) if error.is_timeout() => { + "the publisher did not answer in time".to_string() + } + Self::Upstream(_) => "could not connect to the publisher".to_string(), + Self::Status(status) => format!("the publisher answered HTTP {status}"), + Self::ContentType(_) => "the publisher's file is not audio".to_string(), + Self::TooLarge(_) => "the publisher's file is too large".to_string(), + Self::Timeout => "the publisher did not answer in time".to_string(), + Self::Body(_) => "the publisher stream failed".to_string(), + } + } +} + +impl fmt::Display for ProxyError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::InvalidUrl(reason) => write!(f, "stored audio URL is invalid: {reason}"), + Self::Upstream(error) => write!(f, "publisher request failed: {error}"), + Self::Status(status) => write!(f, "publisher answered HTTP {status}"), + Self::ContentType(value) => write!(f, "publisher content type {value:?} is not audio"), + Self::TooLarge(len) => write!( + f, + "publisher response is too large ({len} bytes, limit {MAX_PROXY_BYTES})" + ), + Self::Timeout => write!( + f, + "publisher did not answer within {} s", + PROXY_HEADER_TIMEOUT.as_secs() + ), + Self::Body(reason) => write!(f, "publisher stream failed: {reason}"), + } + } +} + +impl std::error::Error for ProxyError {} + +impl From for ProxyError { + fn from(error: GuardError) -> Self { + Self::Upstream(error) + } +} + +/// A proxied response, ready to be sent to the browser. +pub(crate) struct ProxiedAudio { + /// 200, 206 or 416. + pub(crate) status: u16, + /// Allow-listed audio type (absent on 416). + pub(crate) content_type: Option, + pub(crate) content_length: Option, + /// `bytes a-b/n` (206) or `bytes */n` (416). + pub(crate) content_range: Option, + /// Whether the publisher advertised byte ranges. + pub(crate) accept_ranges: bool, + /// The body, cut off at [`MAX_PROXY_BYTES`] and [`PROXY_MAX_DURATION`]. + pub(crate) body: BoxStream<'static, Result>, +} + +impl fmt::Debug for ProxiedAudio { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("ProxiedAudio") + .field("status", &self.status) + .field("content_type", &self.content_type) + .field("content_length", &self.content_length) + .field("content_range", &self.content_range) + .field("accept_ranges", &self.accept_ranges) + .finish_non_exhaustive() + } +} + +/// Keeps a browser `Range` header only when it is one well-formed byte +/// range (`bytes=a-`, `bytes=a-b`, `bytes=-n`); anything else is dropped +/// and the full resource is requested. +pub(crate) fn sanitize_range(value: &str) -> Option { + let spec = value.trim().strip_prefix("bytes=")?.trim(); + if spec.len() > 64 { + return None; + } + let (start, end) = spec.split_once('-')?; + let digits = |s: &str| !s.is_empty() && s.bytes().all(|b| b.is_ascii_digit()); + let ok = match (start.is_empty(), end.is_empty()) { + (false, true) => digits(start) && start.parse::().is_ok(), + (false, false) => { + digits(start) + && digits(end) + && matches!((start.parse::(), end.parse::()), (Ok(a), Ok(b)) if a <= b) + } + (true, false) => digits(end) && end.parse::().is_ok_and(|n| n > 0), + (true, true) => false, + }; + ok.then(|| format!("bytes={start}-{end}")) +} + +/// `Content-Range` of a 206/416 passed through only when well-formed. +fn sanitize_content_range(value: &str) -> Option { + let rest = value.trim().strip_prefix("bytes ")?; + let (range, total) = rest.split_once('/')?; + let total_ok = total == "*" || total.parse::().is_ok(); + let range_ok = range == "*" + || range.split_once('-').is_some_and( + |(a, b)| matches!((a.parse::(), b.parse::()), (Ok(a), Ok(b)) if a <= b), + ); + (total_ok && range_ok && !(range == "*" && total == "*")).then(|| value.trim().to_string()) +} + +fn is_audio_type(mime: &str) -> bool { + let mime = mime.trim().to_ascii_lowercase(); + let essence = mime.split(';').next().unwrap_or_default().trim(); + (essence.starts_with("audio/") && essence.len() > "audio/".len()) + || matches!(essence, "application/ogg" | "application/x-ogg") +} + +/// Audio type implied by the URL path's extension. +fn audio_type_from_url(url: &Url) -> Option<&'static str> { + let ext = url.path().rsplit('/').next()?.rsplit_once('.')?.1; + Some(match ext.to_ascii_lowercase().as_str() { + "mp3" => "audio/mpeg", + "m4a" | "m4b" | "mp4" => "audio/mp4", + "aac" => "audio/aac", + "ogg" | "oga" => "audio/ogg", + "opus" => "audio/ogg", + "flac" => "audio/flac", + "wav" => "audio/wav", + _ => return None, + }) +} + +/// Bytes read from the start of a file to recognize its format. +const SNIFF_BYTES: usize = 16; + +/// What the declared type and the URL say about a response. +#[derive(Debug, PartialEq, Eq)] +enum TypeCheck { + /// Audio: the publisher's type, or a generic type with an audio file + /// extension. + Audio(String), + /// A generic binary type (or none) without an audio extension: only the + /// file's first bytes can tell. + Content, +} + +/// The content type sent to the browser: the publisher's when it is audio; +/// for a generic binary type, the type implied by the URL extension +/// ([`TypeCheck::Content`] without one); anything else is refused. +/// +/// The enclosure type from the feed (or from an episode reference a client +/// sent) is never trusted here: it is chosen by whoever wrote the feed, so +/// it would turn the proxy into a relay for any file served as +/// `application/octet-stream`. +fn audio_content_type(upstream: Option<&str>, url: &Url) -> Result { + let upstream = upstream.map(str::trim).filter(|value| !value.is_empty()); + if let Some(value) = upstream + && is_audio_type(value) + { + return Ok(TypeCheck::Audio(value.to_string())); + } + let generic = upstream.is_none_or(|value| { + let essence = value.split(';').next().unwrap_or_default().trim(); + essence.eq_ignore_ascii_case("application/octet-stream") + || essence.eq_ignore_ascii_case("binary/octet-stream") + }); + if !generic { + return Err(ProxyError::ContentType( + upstream.unwrap_or("(none)").to_string(), + )); + } + Ok(match audio_type_from_url(url) { + Some(guess) => TypeCheck::Audio(guess.to_string()), + None => TypeCheck::Content, + }) +} + +/// The audio format a file starts with (magic numbers of the formats +/// podcasts use), or `None`. +pub(crate) fn sniff_audio(head: &[u8]) -> Option<&'static str> { + let at = |range: std::ops::Range| head.get(range); + if head.starts_with(b"ID3") { + return Some("audio/mpeg"); + } + if head.starts_with(b"OggS") { + return Some("audio/ogg"); + } + if head.starts_with(b"fLaC") { + return Some("audio/flac"); + } + if at(0..4) == Some(b"RIFF") && at(8..12) == Some(b"WAVE") { + return Some("audio/wav"); + } + // MPEG-4 audio brands only: video files are not passed through. + if at(4..8) == Some(b"ftyp") + && matches!( + at(8..12), + Some(b"M4A " | b"M4B " | b"M4P " | b"F4A " | b"F4B ") + ) + { + return Some("audio/mp4"); + } + // A bare frame header has few fixed bits, so the whole header must be + // valid. Layer I (`FF FE`, `FF FF`: the UTF-16 byte order mark of text + // files) is never used by podcasts and is not accepted. + if let [0xFF, second, third, ..] = *head { + // ADTS AAC: sync, layer 00, a defined sampling frequency. + if second & 0xF6 == 0xF0 && (third >> 2) & 0b1111 < 13 { + return Some("audio/aac"); + } + // MPEG audio frame: sync, a defined version, layer II or III, a + // bitrate index other than free (0) and bad (15), and a defined + // sampling rate. + let version = (second >> 3) & 0b11; + let layer = (second >> 1) & 0b11; + let bitrate = third >> 4; + let sample_rate = (third >> 2) & 0b11; + if second & 0xE0 == 0xE0 + && version != 0b01 + && matches!(layer, 0b01 | 0b10) + && (1..15).contains(&bitrate) + && sample_rate != 0b11 + { + return Some("audio/mpeg"); + } + } + None +} + +type Upstream = BoxStream<'static, reqwest::Result>; + +/// Reads at least `want` bytes (fewer at the end of the body) from +/// `upstream`; the chunks are sent to the browser afterwards. +async fn read_head(upstream: &mut Upstream, want: usize) -> Result, ProxyError> { + let mut chunks = Vec::new(); + let mut have = 0; + while have < want { + match tokio::time::timeout(PROXY_HEADER_TIMEOUT, upstream.next()).await { + Err(_) => return Err(ProxyError::Timeout), + Ok(None) => break, + Ok(Some(Ok(chunk))) => { + have += chunk.len(); + chunks.push(chunk); + } + Ok(Some(Err(error))) => { + return Err(ProxyError::Body(GuardError::from(error).to_string())); + } + } + } + Ok(chunks) +} + +fn head_bytes(chunks: &[Bytes]) -> Vec { + let mut head = Vec::with_capacity(SNIFF_BYTES); + for chunk in chunks { + let take = (SNIFF_BYTES - head.len()).min(chunk.len()); + head.extend_from_slice(&chunk[..take]); + if head.len() == SNIFF_BYTES { + break; + } + } + head +} + +/// Streams episode enclosures from publishers. +#[derive(Debug, Clone)] +pub(crate) struct AudioProxy { + http: SafeHttp, + max_bytes: u64, + max_duration: Duration, +} + +impl AudioProxy { + /// The production proxy (public addresses only). + pub(crate) fn new() -> anyhow::Result { + Ok(Self::with_http(SafeHttp::public(PROXY_HTTP)?)) + } + + pub(crate) fn with_http(http: SafeHttp) -> Self { + Self { + http, + max_bytes: MAX_PROXY_BYTES, + max_duration: PROXY_MAX_DURATION, + } + } + + #[cfg(test)] + pub(crate) fn with_limits(mut self, max_bytes: u64, max_duration: Duration) -> Self { + self.max_bytes = max_bytes; + self.max_duration = max_duration; + self + } + + /// Requests `audio_url` (from the database) with the browser's `Range` + /// (sanitized). A response with a generic binary type and no audio file + /// extension is passed through only when the file starts like audio + /// (read from this response when it starts at byte 0 and is long + /// enough, otherwise from a separate 16-byte range request). + pub(crate) async fn fetch( + &self, + audio_url: &str, + range: Option<&str>, + ) -> Result { + let url = + Url::parse(audio_url).map_err(|error| ProxyError::InvalidUrl(error.to_string()))?; + let mut headers = HeaderMap::new(); + headers.insert(ACCEPT, HeaderValue::from_static("audio/*, */*;q=0.1")); + // Byte ranges must address the stored bytes, not a compressed form. + headers.insert(ACCEPT_ENCODING, HeaderValue::from_static("identity")); + let range = range.and_then(sanitize_range); + if let Some(range) = &range { + headers.insert( + RANGE, + HeaderValue::from_str(range).expect("sanitized range is a valid header"), + ); + } + let followed = tokio::time::timeout(PROXY_HEADER_TIMEOUT, self.http.get(url, &headers)) + .await + .map_err(|_| ProxyError::Timeout)??; + let response = followed.response; + let final_url = followed.final_url; + let status = response.status(); + let header = |name| { + response + .headers() + .get(name) + .and_then(|value: &HeaderValue| value.to_str().ok()) + .map(str::to_string) + }; + let content_range = header(CONTENT_RANGE) + .as_deref() + .and_then(sanitize_content_range); + if status == StatusCode::RANGE_NOT_SATISFIABLE { + return Ok(ProxiedAudio { + status: 416, + content_type: None, + content_length: Some(0), + content_range, + accept_ranges: true, + body: futures_util::stream::empty().boxed(), + }); + } + let partial = status == StatusCode::PARTIAL_CONTENT; + // 206 only answers a range we asked for, and must say which bytes. + if !(status == StatusCode::OK || (partial && range.is_some() && content_range.is_some())) { + return Err(ProxyError::Status(status.as_u16())); + } + let declared = header(CONTENT_TYPE); + let check = audio_content_type(declared.as_deref(), &final_url)?; + let content_length = header(CONTENT_LENGTH).and_then(|value| value.parse::().ok()); + if let Some(len) = content_length + && len > self.max_bytes + { + return Err(ProxyError::TooLarge(len)); + } + let accept_ranges = partial + || header(ACCEPT_RANGES) + .is_some_and(|value| value.trim().eq_ignore_ascii_case("bytes")); + let starts_at_zero = !partial + || content_range + .as_deref() + .is_some_and(|value| value.starts_with("bytes 0-")); + let mut upstream: Upstream = response.bytes_stream().boxed(); + let mut head = Vec::new(); + let content_type = match check { + TypeCheck::Audio(content_type) => content_type, + TypeCheck::Content => { + let mut start = Vec::new(); + if starts_at_zero { + head = read_head(&mut upstream, SNIFF_BYTES).await?; + start = head_bytes(&head); + } + // A range past the start, or one that ends before the + // sniffed bytes (Safari and AVFoundation probe with + // `bytes=0-1`): the file's head comes from its own request. + if partial && start.len() < SNIFF_BYTES { + start = self.file_head(&final_url).await?; + } + sniff_audio(&start) + .ok_or_else(|| { + ProxyError::ContentType(format!( + "{} (not audio content)", + declared.as_deref().unwrap_or("(none)") + )) + })? + .to_string() + } + }; + Ok(ProxiedAudio { + status: status.as_u16(), + content_type: Some(content_type), + content_length, + content_range: if partial { content_range } else { None }, + accept_ranges, + body: limited_body(head, upstream, self.max_bytes, self.max_duration), + }) + } + + /// The first [`SNIFF_BYTES`] of the file at `url` (a range request; a + /// server without ranges answers with the whole file, of which only the + /// start is read before the connection is dropped). + async fn file_head(&self, url: &Url) -> Result, ProxyError> { + let mut headers = HeaderMap::new(); + headers.insert(ACCEPT_ENCODING, HeaderValue::from_static("identity")); + headers.insert( + RANGE, + HeaderValue::from_str(&format!("bytes=0-{}", SNIFF_BYTES - 1)).expect("valid range"), + ); + let followed = + tokio::time::timeout(PROXY_HEADER_TIMEOUT, self.http.get(url.clone(), &headers)) + .await + .map_err(|_| ProxyError::Timeout)??; + let status = followed.response.status(); + if !(status == StatusCode::OK || status == StatusCode::PARTIAL_CONTENT) { + return Err(ProxyError::Status(status.as_u16())); + } + let mut upstream: Upstream = followed.response.bytes_stream().boxed(); + Ok(head_bytes(&read_head(&mut upstream, SNIFF_BYTES).await?)) + } +} + +/// The upstream body (`head`, already read, first), ending with an error +/// once `max_bytes` would be exceeded or `max_duration` has passed. +/// Dropping it (browser gone or seeking elsewhere) closes the upstream +/// connection. +fn limited_body( + head: Vec, + upstream: Upstream, + max_bytes: u64, + max_duration: Duration, +) -> BoxStream<'static, Result> { + let deadline = tokio::time::Instant::now() + max_duration; + let mut upstream = futures_util::stream::iter(head.into_iter().map(Ok)).chain(upstream); + async_stream::stream! { + let mut sent: u64 = 0; + loop { + let next = match tokio::time::timeout_at(deadline, upstream.next()).await { + Ok(next) => next, + Err(_) => { + yield Err(ProxyError::Body("time limit reached".to_string())); + break; + } + }; + match next { + None => break, + Some(Ok(chunk)) => { + sent += chunk.len() as u64; + if sent > max_bytes { + yield Err(ProxyError::Body(format!("size limit of {max_bytes} bytes reached"))); + break; + } + yield Ok(chunk); + } + Some(Err(error)) => { + yield Err(ProxyError::Body(GuardError::from(error).to_string())); + break; + } + } + } + } + .boxed() +} diff --git a/src/podcasts/proxy_tests.rs b/src/podcasts/proxy_tests.rs new file mode 100644 index 0000000..9da443e --- /dev/null +++ b/src/podcasts/proxy_tests.rs @@ -0,0 +1,449 @@ +//! Audio proxy: Range pass-through, servers without ranges, redirects, +//! content types and limits, against the local test server. + +use std::time::Duration; + +use futures_util::StreamExt; +use futures_util::stream::BoxStream; + +use super::net::{AddressClass, GuardError, REFUSED_MESSAGE}; +use super::proxy::{AudioProxy, ProxyError, sanitize_range, sniff_audio}; +use std::sync::Arc; + +use super::test_server::{Route, ScriptedLookup, TestServer, noise, test_http}; + +// --------------------------------------------------------------------------- +// Audio proxy +// --------------------------------------------------------------------------- + +#[test] +fn only_single_byte_ranges_are_forwarded() { + assert_eq!(sanitize_range("bytes=0-").as_deref(), Some("bytes=0-")); + assert_eq!( + sanitize_range("bytes=100-199").as_deref(), + Some("bytes=100-199") + ); + assert_eq!( + sanitize_range(" bytes=-500 ").as_deref(), + Some("bytes=-500") + ); + for invalid in [ + "bytes=1-2,5-6", + "items=0-10", + "bytes=5-1", + "bytes=-", + "bytes=-0", + "bytes=a-b", + "bytes=1-2\r\nCookie: x", + "bytes=99999999999999999999999-", + "", + ] { + assert_eq!(sanitize_range(invalid), None, "{invalid:?}"); + } +} + +async fn collect( + mut body: BoxStream<'static, Result>, +) -> (Vec, Option) { + let mut out = Vec::new(); + while let Some(chunk) = body.next().await { + match chunk { + Ok(chunk) => out.extend_from_slice(&chunk), + Err(error) => return (out, Some(error.to_string())), + } + } + (out, None) +} + +fn proxy(lookup: Arc) -> AudioProxy { + AudioProxy::with_http(test_http(lookup)) +} + +#[tokio::test] +async fn proxy_passes_ranges_through() { + let server = TestServer::start(); + let audio = noise(300_000, 7); + server.set("/ep.mp3", Route::new(audio.clone(), "audio/mpeg")); + let proxy = proxy(ScriptedLookup::with(&[])); + let target = server.url("/ep.mp3"); + + let full = proxy.fetch(&target, None).await.unwrap(); + assert_eq!(full.status, 200); + assert_eq!(full.content_type.as_deref(), Some("audio/mpeg")); + assert_eq!(full.content_length, Some(audio.len() as u64)); + assert!(full.accept_ranges); + assert_eq!(full.content_range, None); + assert_eq!(collect(full.body).await, (audio.clone(), None)); + + for (range, start, end) in [ + ("bytes=1000-1999", 1000, 2000), + ("bytes=299000-", 299_000, 300_000), + ("bytes=-500", 299_500, 300_000), + ("bytes=0-", 0, 300_000), + ] { + let part = proxy.fetch(&target, Some(range)).await.unwrap(); + assert_eq!(part.status, 206, "{range}"); + assert_eq!( + part.content_range, + Some(format!("bytes {start}-{}/{}", end - 1, audio.len())), + "{range}" + ); + assert_eq!(part.content_length, Some((end - start) as u64)); + assert_eq!(collect(part.body).await, (audio[start..end].to_vec(), None)); + } + + let unsatisfiable = proxy.fetch(&target, Some("bytes=400000-")).await.unwrap(); + assert_eq!(unsatisfiable.status, 416); + assert_eq!( + unsatisfiable.content_range, + Some(format!("bytes */{}", audio.len())) + ); + assert_eq!(collect(unsatisfiable.body).await, (Vec::new(), None)); + + // Malformed or multi-range requests fetch the whole file. + let whole = proxy.fetch(&target, Some("bytes=5-6,9-10")).await.unwrap(); + assert_eq!(whole.status, 200); + + let requests = server.requests("/ep.mp3"); + let ranges: Vec> = requests + .iter() + .map(|request| request.get("range").map(String::as_str)) + .collect(); + assert_eq!( + ranges, + vec![ + None, + Some("bytes=1000-1999"), + Some("bytes=299000-"), + Some("bytes=-500"), + Some("bytes=0-"), + Some("bytes=400000-"), + None, + ] + ); + for request in &requests { + assert_eq!(request["accept-encoding"], "identity"); + assert!(!request.contains_key("cookie")); + assert!(!request.contains_key("authorization")); + } +} + +#[tokio::test] +async fn proxy_handles_servers_without_ranges_and_redirects() { + let server = TestServer::start(); + let audio = noise(50_000, 3); + server.set( + "/plain.mp3", + Route::new(audio.clone(), "audio/mpeg").no_ranges(), + ); + server.set("/track/ep.mp3", Route::redirect(302, "/cdn/ep.mp3")); + server.set("/cdn/ep.mp3", Route::new(audio.clone(), "audio/mpeg")); + server.set( + "/track/private.mp3", + Route::redirect(302, "http://172.20.0.3/ep.mp3"), + ); + let proxy = proxy(ScriptedLookup::with(&[])); + + // No range support: the whole file with 200 and no Accept-Ranges, so the + // browser knows it cannot seek by range. + let plain = proxy + .fetch(&server.url("/plain.mp3"), Some("bytes=100-")) + .await + .unwrap(); + assert_eq!(plain.status, 200); + assert!(!plain.accept_ranges); + assert_eq!(plain.content_range, None); + assert_eq!(collect(plain.body).await.0, audio); + + // Tracking redirects keep the range on every hop. + let part = proxy + .fetch(&server.url("/track/ep.mp3"), Some("bytes=10-19")) + .await + .unwrap(); + assert_eq!(part.status, 206); + assert_eq!(collect(part.body).await.0, audio[10..20].to_vec()); + assert_eq!(server.requests("/track/ep.mp3")[0]["range"], "bytes=10-19"); + assert_eq!(server.requests("/cdn/ep.mp3")[0]["range"], "bytes=10-19"); + + let Err(error) = proxy.fetch(&server.url("/track/private.mp3"), None).await else { + panic!("redirect to a private address was followed"); + }; + assert!( + matches!( + &error, + ProxyError::Upstream(GuardError::Blocked(blocked)) + if blocked.class == AddressClass::Private + ), + "{error:?}" + ); + // An IP literal is refused for what the URL says. + assert_eq!(error.public_message(), REFUSED_MESSAGE); +} + +#[tokio::test] +async fn proxy_answers_alike_for_internal_and_unreachable_names() { + // Episode URLs can come from the user's own clients: the answer must + // not tell a name with a private address from one that does not exist + // or a server that refuses the connection. + let port = std::net::TcpListener::bind("127.0.0.1:0") + .unwrap() + .local_addr() + .unwrap() + .port(); + let proxy = proxy(ScriptedLookup::with(&[ + ("db.corp.test", &[&["192.168.7.7"]]), + ("closed.test", &[&["127.0.0.1"]]), + ])); + let mut answers = Vec::new(); + for host in ["db.corp.test", "nowhere.corp.test", "closed.test"] { + let target = format!("http://{host}:{port}/episode.mp3"); + let error = proxy.fetch(&target, None).await.unwrap_err(); + assert!(matches!(error, ProxyError::Upstream(_)), "{error:?}"); + answers.push((error.http_status(), error.public_message())); + } + assert_eq!(answers[0].1, "could not connect to the publisher"); + assert!( + answers.iter().all(|answer| *answer == answers[0]), + "{answers:?}" + ); +} + +/// `noise` that starts like a file of the given format. +fn audio_file(magic: &[u8], len: usize) -> Vec { + let mut file = magic.to_vec(); + file.extend(noise(len - magic.len(), 3)); + file +} + +#[tokio::test] +async fn proxy_only_serves_audio() { + let server = TestServer::start(); + let audio = noise(1000, 5); + let mp3 = audio_file(b"ID3\x04\0\0", 1000); + server.set( + "/page.mp3", + Route::new("", "text/html"), + ); + server.set("/video.mp4", Route::new(audio.clone(), "video/mp4")); + // Generic types: the URL extension, or else the file's first bytes. + server.set( + "/binary", + Route::new(audio.clone(), "application/octet-stream"), + ); + server.set( + "/zip", + Route::new(audio_file(b"PK\x03\x04", 1000), "application/octet-stream"), + ); + server.set( + "/episode", + Route::new(mp3.clone(), "application/octet-stream"), + ); + server.set("/untyped", Route::new(mp3.clone(), "x").no_content_type()); + server.set( + "/binary.m4a", + Route::new(audio.clone(), "binary/octet-stream"), + ); + server.set( + "/untyped.ogg", + Route::new(audio.clone(), "x").no_content_type(), + ); + server.set("/ogg", Route::new(audio.clone(), "application/ogg")); + server.set( + "/typed", + Route::new(audio.clone(), "audio/mp4; codecs=\"mp4a.40.2\""), + ); + server.set("/missing", Route::new("", "text/plain").status(404)); + server.set("/error", Route::new("", "audio/mpeg").status(500)); + let proxy = proxy(ScriptedLookup::with(&[])); + let content_type = |path: &'static str, range: Option<&'static str>| { + let proxy = proxy.clone(); + let target = server.url(path); + async move { + proxy + .fetch(&target, range) + .await + .map(|audio| audio.content_type.unwrap()) + } + }; + for refused in ["/page.mp3", "/video.mp4", "/binary", "/zip"] { + for range in [None, Some("bytes=10-")] { + assert!( + matches!( + content_type(refused, range).await, + Err(ProxyError::ContentType(_)) + ), + "{refused} {range:?}" + ); + } + } + assert_eq!(content_type("/episode", None).await.unwrap(), "audio/mpeg"); + assert_eq!(content_type("/untyped", None).await.unwrap(), "audio/mpeg"); + // The sniffed bytes are still sent to the browser. + let whole = proxy.fetch(&server.url("/episode"), None).await.unwrap(); + assert_eq!(collect(whole.body).await, (mp3.clone(), None)); + // A range past the start: the file's head is read with its own request. + let before = server.requests("/episode").len(); + let part = proxy + .fetch(&server.url("/episode"), Some("bytes=100-199")) + .await + .unwrap(); + assert_eq!(part.content_type.as_deref(), Some("audio/mpeg")); + assert_eq!(collect(part.body).await, (mp3[100..200].to_vec(), None)); + let ranges: Vec = server.requests("/episode")[before..] + .iter() + .map(|request| request["range"].clone()) + .collect(); + assert_eq!(ranges, ["bytes=100-199", "bytes=0-15"]); + // Safari and AVFoundation probe with a two-byte range from the start: + // too short to recognize the file, so its head is read separately. + let before = server.requests("/episode").len(); + let probe = proxy + .fetch(&server.url("/episode"), Some("bytes=0-1")) + .await + .unwrap(); + assert_eq!(probe.status, 206); + assert_eq!(probe.content_type.as_deref(), Some("audio/mpeg")); + assert_eq!(collect(probe.body).await, (mp3[..2].to_vec(), None)); + let ranges: Vec = server.requests("/episode")[before..] + .iter() + .map(|request| request["range"].clone()) + .collect(); + assert_eq!(ranges, ["bytes=0-1", "bytes=0-15"]); + // A long enough range from the start needs no second request. + let before = server.requests("/episode").len(); + let start = proxy + .fetch(&server.url("/episode"), Some("bytes=0-99")) + .await + .unwrap(); + assert_eq!(start.content_type.as_deref(), Some("audio/mpeg")); + assert_eq!(collect(start.body).await, (mp3[..100].to_vec(), None)); + assert_eq!(server.requests("/episode").len(), before + 1); + // A probe of a file that is not audio is still refused. + assert!(matches!( + proxy.fetch(&server.url("/zip"), Some("bytes=0-1")).await, + Err(ProxyError::ContentType(_)) + )); + assert_eq!( + content_type("/binary.m4a", None).await.unwrap(), + "audio/mp4" + ); + assert_eq!( + content_type("/untyped.ogg", None).await.unwrap(), + "audio/ogg" + ); + assert_eq!(content_type("/ogg", None).await.unwrap(), "application/ogg"); + assert_eq!( + content_type("/typed", None).await.unwrap(), + "audio/mp4; codecs=\"mp4a.40.2\"" + ); + let missing = content_type("/missing", None).await.unwrap_err(); + assert!(matches!(missing, ProxyError::Status(404))); + assert_eq!(missing.http_status(), 502); + assert!(matches!( + content_type("/error", None).await, + Err(ProxyError::Status(500)) + )); + assert!(matches!( + proxy.fetch("not a url", None).await, + Err(ProxyError::InvalidUrl(_)) + )); +} + +#[test] +fn audio_formats_are_recognized_by_their_first_bytes() { + for (head, format) in [ + (&b"ID3\x04\0\0\0\0"[..], "audio/mpeg"), + (&[0xFF, 0xFB, 0x90, 0x64][..], "audio/mpeg"), + (&[0xFF, 0xF3, 0x48, 0xC4][..], "audio/mpeg"), + (&[0xFF, 0xFD, 0x94, 0x00][..], "audio/mpeg"), + (&[0xFF, 0xE3, 0x18, 0xC4][..], "audio/mpeg"), + (&[0xFF, 0xF9, 0x4C, 0x80][..], "audio/aac"), + (&[0xFF, 0xF1, 0x50, 0x80][..], "audio/aac"), + (&b"OggS\0\x02"[..], "audio/ogg"), + (&b"fLaC\0\0\0\x22"[..], "audio/flac"), + (&b"RIFF\x24\x08\0\0WAVEfmt "[..], "audio/wav"), + (&b"\0\0\0\x20ftypM4A \0\0\0\0"[..], "audio/mp4"), + ] { + assert_eq!(sniff_audio(head), Some(format), "{head:?}"); + } + for head in [ + &b"\0\0\0\x20ftypisom\0\0\x02\0"[..], + &b"\0\0\0\x18ftypmp42"[..], + &b"PK\x03\x04"[..], + &b""[..], + &b"%PDF-1.7"[..], + &b"RIFF\x24\x08\0\0AVI "[..], + &[0xFF, 0xD8, 0xFF, 0xE0][..], + // UTF-16LE text (byte order mark FF FE) and FF FF are not frames. + &[0xFF, 0xFE, 0x3C, 0x00, 0x68, 0x00][..], + &[0xFF, 0xFE, 0x41, 0x00][..], + &[0xFF, 0xFF, 0x90, 0x00][..], + // Frame sync with an invalid bitrate, free bitrate or sample rate. + &[0xFF, 0xFB, 0xF0, 0x00][..], + &[0xFF, 0xFB, 0x00, 0x00][..], + &[0xFF, 0xFB, 0x9C, 0x00][..], + // ADTS sync with a reserved sampling frequency index. + &[0xFF, 0xF1, 0x3C, 0x80][..], + &[0xFF, 0xFB][..], + &[0xFF][..], + &[][..], + ] { + assert_eq!(sniff_audio(head), None, "{head:?}"); + } +} + +#[tokio::test] +async fn proxy_limits_size_and_reports_dropped_streams() { + let server = TestServer::start(); + let audio = noise(100_000, 9); + server.set("/sized.mp3", Route::new(audio.clone(), "audio/mpeg")); + server.set( + "/chunked.mp3", + Route::new(audio.clone(), "audio/mpeg").no_length(), + ); + server.set( + "/cut.mp3", + Route::new(audio.clone(), "audio/mpeg").cut(40_000), + ); + let limited = proxy(ScriptedLookup::with(&[])).with_limits(60_000, Duration::from_secs(60)); + assert!(matches!( + limited.fetch(&server.url("/sized.mp3"), None).await, + Err(ProxyError::TooLarge(100_000)) + )); + // A range below the limit is fine. + let part = limited + .fetch(&server.url("/sized.mp3"), Some("bytes=0-49999")) + .await + .unwrap(); + assert_eq!(collect(part.body).await, (audio[..50_000].to_vec(), None)); + // Without a declared length the stream stops at the limit. + let chunked = limited + .fetch(&server.url("/chunked.mp3"), None) + .await + .unwrap(); + let (received, error) = collect(chunked.body).await; + assert!(received.len() <= 60_000); + assert!(error.unwrap().contains("size limit")); + // A connection dropped mid-body ends the stream with an error instead + // of a silently truncated success. + let cut = proxy(ScriptedLookup::with(&[])) + .fetch(&server.url("/cut.mp3"), None) + .await + .unwrap(); + let (received, error) = collect(cut.body).await; + assert!(received.len() <= 40_000); + assert!(error.is_some()); +} + +#[tokio::test] +async fn proxy_stream_has_a_time_limit() { + let server = TestServer::start(); + server.set( + "/slow.mp3", + Route::new(noise(10, 1), "audio/mpeg").delay_body(Duration::from_secs(3)), + ); + let limited = proxy(ScriptedLookup::with(&[])).with_limits(1 << 20, Duration::from_millis(300)); + let audio = limited.fetch(&server.url("/slow.mp3"), None).await.unwrap(); + let (_, error) = collect(audio.body).await; + assert!(error.unwrap().contains("time limit")); +} diff --git a/src/podcasts/queries.rs b/src/podcasts/queries.rs new file mode 100644 index 0000000..d663d87 --- /dev/null +++ b/src/podcasts/queries.rs @@ -0,0 +1,723 @@ +//! Read-only queries for the podcast UI and the player. + +use anyhow::Result; +use music_dht::podcast::{ + EpisodeId, EpisodeMeta, EpisodeType, ListenSession, PodcastEpisodeRef, PodcastId, PodcastMeta, + ProgressRecord, ProgressStatus, SubscriptionRecord, effective_share, +}; +use sqlx::postgres::PgRow; +use sqlx::{PgConnection, Postgres, QueryBuilder, Row}; + +use super::apply; +use super::{ + PodcastStore, SESSION_COLUMNS, episode_id_col, episode_search_norm, feed_url_col, + podcast_id_col, progress_columns, progress_from_row, session_from_row, u32_from_sql, +}; + +/// Which podcasts [`PodcastStore::list_podcasts`] returns. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub(crate) enum PodcastListFilter { + /// Podcasts with an active subscription. + #[default] + Subscribed, + /// Every podcast in the catalog, including unsubscribed ones and + /// podcasts only known from synced progress or queue items. + All, +} + +/// A podcast with its personal state and episode counters. +#[derive(Debug, Clone, PartialEq)] +pub(crate) struct PodcastSummary { + /// Podcast id. + pub(crate) podcast_id: PodcastId, + /// Local channel metadata; `feed_url` is the URL this account fetches. + pub(crate) meta: PodcastMeta, + /// Whether the subscription register says subscribed. + pub(crate) subscribed: bool, + /// Whether the podcast must currently be published (`effective_share`). + pub(crate) shared: bool, + /// Episodes in the catalog. + pub(crate) episode_count: u64, + /// Episodes not completed (no progress, not started or in progress). + pub(crate) unplayed_count: u64, + /// Episodes in progress. + pub(crate) in_progress_count: u64, + /// Newest publication time. + pub(crate) latest_published_at_ms: Option, + /// Numbered seasons (episodes without a season are not counted). + pub(crate) season_count: u64, +} + +/// One season group of a podcast. `season == None` groups the episodes +/// without `itunes:season`. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct SeasonSummary { + /// Season number; `None` for episodes without a season. + pub(crate) season: Option, + /// Episodes in the group. + pub(crate) episode_count: u64, + /// Episodes not completed. + pub(crate) unplayed_count: u64, + /// Oldest publication time. + pub(crate) first_published_at_ms: Option, + /// Newest publication time. + pub(crate) last_published_at_ms: Option, +} + +/// Season filter of an [`EpisodeQuery`]. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub(crate) enum SeasonFilter { + /// All episodes. + #[default] + Any, + /// Episodes of one season. + Season(u32), + /// Episodes without a season. + NoSeason, +} + +/// Listening-state filter of an [`EpisodeQuery`]. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub(crate) enum PlayedFilter { + /// All episodes. + #[default] + Any, + /// Not completed (not started or in progress). + Unplayed, + /// Never started or marked unplayed. + NotStarted, + /// In progress. + InProgress, + /// Completed. + Played, +} + +/// Episode ordering of an [`EpisodeQuery`]. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub(crate) enum EpisodeSort { + /// Newest publication first; undated episodes last. + #[default] + NewestFirst, + /// Oldest publication first; undated episodes last. + OldestFirst, + /// By season, then episode number, ascending; unnumbered last. + EpisodeNumber, + /// By season, then episode number, descending; unnumbered last. + EpisodeNumberDesc, +} + +impl EpisodeSort { + fn order_by(self) -> &'static str { + match self { + Self::NewestFirst => "e.published_at_ms IS NULL, e.published_at_ms DESC, e.episode_id", + Self::OldestFirst => "e.published_at_ms IS NULL, e.published_at_ms, e.episode_id", + Self::EpisodeNumber => { + "e.season IS NULL, e.season, e.episode_number IS NULL, e.episode_number, \ + e.published_at_ms IS NULL, e.published_at_ms, e.episode_id" + } + Self::EpisodeNumberDesc => { + "e.season IS NULL, e.season DESC, e.episode_number IS NULL, \ + e.episode_number DESC, e.published_at_ms IS NULL, e.published_at_ms DESC, \ + e.episode_id" + } + } + } +} + +#[cfg(test)] +pub(crate) fn order_by_for_tests(sort: EpisodeSort) -> &'static str { + sort.order_by() +} + +/// Parameters of [`PodcastStore::episodes`]. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct EpisodeQuery { + /// Podcast. + pub(crate) podcast_id: PodcastId, + /// Season filter. + pub(crate) season: SeasonFilter, + /// Listening-state filter. + pub(crate) played: PlayedFilter, + /// Text searched in titles and descriptions (`normalize_name`: Unicode + /// NFKC, case-insensitive substring match). Blank means no filter. + pub(crate) search: Option, + /// Ordering. + pub(crate) sort: EpisodeSort, + /// Maximum rows (`None`: all). + pub(crate) limit: Option, + /// Rows to skip. + pub(crate) offset: usize, +} + +impl EpisodeQuery { + /// All episodes of a podcast, newest first. + pub(crate) fn new(podcast_id: PodcastId) -> Self { + Self { + podcast_id, + season: SeasonFilter::Any, + played: PlayedFilter::Any, + search: None, + sort: EpisodeSort::NewestFirst, + limit: None, + offset: 0, + } + } +} + +/// An episode with its personal state. +#[derive(Debug, Clone, PartialEq)] +pub(crate) struct EpisodeListItem { + /// Episode metadata (feed or synced reference). + pub(crate) episode: EpisodeMeta, + /// Progress register, if any. + pub(crate) progress: Option, + /// Distinct completed listening sessions (`completed_listen_count`). + pub(crate) completed_count: u64, + /// Whether the latest refresh of the feed listed the episode. + pub(crate) in_feed: bool, +} + +impl EpisodeListItem { + /// Listening status (`NotStarted` without a register). + pub(crate) fn status(&self) -> ProgressStatus { + self.progress + .as_ref() + .map_or(ProgressStatus::NotStarted, |progress| progress.status) + } + + /// Position ordinary Play resumes from. + pub(crate) fn resume_position_ms(&self) -> i64 { + self.progress + .as_ref() + .map_or(0, ProgressRecord::resume_position_ms) + } +} + +/// A listening session in history, with display titles when known. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct PodcastListenEntry { + /// The session. + pub(crate) session: ListenSession, + /// Episode title from the catalog. + pub(crate) episode_title: Option, + /// Podcast title from the catalog. + pub(crate) podcast_title: Option, +} + +const EPISODE_COLUMNS: &str = "e.episode_id, e.podcast_id, e.guid, e.title, e.description, \ + e.season, e.episode_number, e.episode_type, e.published_at_ms, e.duration_ms, \ + e.audio_url, e.mime, e.byte_len, e.image_url, e.link"; + +const NOT_COMPLETED: &str = "COALESCE(g.status, 'not_started') <> 'completed'"; + +fn episode_from_row(row: &PgRow) -> sqlx::Result { + Ok(EpisodeMeta { + episode_id: episode_id_col(row, "episode_id")?, + podcast_id: podcast_id_col(row, "podcast_id")?, + guid: row.try_get("guid")?, + title: row.try_get("title")?, + description: row.try_get("description")?, + season: u32_from_sql(row.try_get("season")?), + episode_number: u32_from_sql(row.try_get("episode_number")?), + episode_type: EpisodeType::parse(&row.try_get::("episode_type")?), + published_at_ms: row.try_get("published_at_ms")?, + duration_ms: row.try_get("duration_ms")?, + audio_url: row.try_get("audio_url")?, + mime: row.try_get("mime")?, + byte_len: row + .try_get::, _>("byte_len")? + .and_then(|len| u64::try_from(len).ok()), + image_url: row.try_get("image_url")?, + link: row.try_get("link")?, + }) +} + +fn list_item_from_row(row: &PgRow) -> sqlx::Result { + let progress = if row.try_get::, _>("g_episode_id")?.is_some() { + Some(progress_from_row(row, "g_")?) + } else { + None + }; + Ok(EpisodeListItem { + episode: episode_from_row(row)?, + in_feed: row.try_get("in_feed")?, + progress, + completed_count: count(row.try_get("completed_count")?), + }) +} + +fn count(value: i64) -> u64 { + u64::try_from(value).unwrap_or(0) +} + +pub(crate) async fn episode_meta_in( + conn: &mut PgConnection, + user_id: i64, + episode_id: &EpisodeId, +) -> Result> { + let row = sqlx::query(&format!( + "SELECT {EPISODE_COLUMNS} FROM furumusic__podcast_episode e + WHERE e.user_id = $1 AND e.episode_id = $2" + )) + .bind(user_id) + .bind(episode_id.as_str()) + .fetch_optional(&mut *conn) + .await?; + Ok(row.as_ref().map(episode_from_row).transpose()?) +} + +pub(crate) async fn podcast_meta_in( + conn: &mut PgConnection, + user_id: i64, + podcast_id: &PodcastId, +) -> Result> { + let row = sqlx::query( + "SELECT feed_url, title, author, description, link, image_url + FROM furumusic__podcast WHERE user_id = $1 AND podcast_id = $2", + ) + .bind(user_id) + .bind(podcast_id.as_str()) + .fetch_optional(&mut *conn) + .await?; + let Some(row) = row else { + return Ok(None); + }; + Ok(Some(PodcastMeta { + feed_url: feed_url_col(&row, "feed_url")?, + title: row.try_get("title")?, + author: row.try_get("author")?, + description: row.try_get("description")?, + link: row.try_get("link")?, + image_url: row.try_get("image_url")?, + })) +} + +/// Queue reference of a catalog episode, with `progress_generation` set to +/// the generation of its progress register (handoff hint). +pub(crate) async fn episode_ref_in( + conn: &mut PgConnection, + user_id: i64, + episode_id: &EpisodeId, +) -> Result> { + let Some(episode) = episode_meta_in(conn, user_id, episode_id).await? else { + return Ok(None); + }; + let Some(podcast) = podcast_meta_in(conn, user_id, &episode.podcast_id).await? else { + return Ok(None); + }; + let mut reference = episode.to_ref(&podcast); + reference.progress_generation = apply::load_progress(conn, user_id, episode_id) + .await? + .map(|progress| progress.generation); + Ok(Some(reference)) +} + +async fn summary_in( + conn: &mut PgConnection, + user_id: i64, + podcast_id: &PodcastId, +) -> Result> { + let Some(meta) = podcast_meta_in(conn, user_id, podcast_id).await? else { + return Ok(None); + }; + let row = sqlx::query(&format!( + "SELECT COUNT(*) AS episode_count, + COUNT(*) FILTER (WHERE {NOT_COMPLETED}) AS unplayed_count, + COUNT(*) FILTER (WHERE g.status = 'in_progress') AS in_progress_count, + MAX(e.published_at_ms) AS latest, + (SELECT COUNT(DISTINCT e2.season) FROM furumusic__podcast_episode e2 + WHERE e2.user_id = $1 AND e2.podcast_id = $2) AS season_count + FROM furumusic__podcast_episode e + LEFT JOIN furumusic__podcast_progress g + ON g.user_id = e.user_id AND g.episode_id = e.episode_id + WHERE e.user_id = $1 AND e.podcast_id = $2" + )) + .bind(user_id) + .bind(podcast_id.as_str()) + .fetch_one(&mut *conn) + .await?; + let subscription = apply::load_subscription_register(conn, user_id, podcast_id).await?; + let share = apply::load_share(conn, user_id, podcast_id).await?; + Ok(Some(PodcastSummary { + podcast_id: podcast_id.clone(), + meta, + subscribed: subscription + .as_ref() + .is_some_and(|record| record.subscribed), + shared: effective_share(subscription.as_ref(), share.as_ref()), + episode_count: count(row.try_get("episode_count")?), + unplayed_count: count(row.try_get("unplayed_count")?), + in_progress_count: count(row.try_get("in_progress_count")?), + latest_published_at_ms: row.try_get("latest")?, + season_count: count(row.try_get("season_count")?), + })) +} + +/// Computes the cached search text of episodes that do not have it yet +/// (rows imported by the `m_0057` migration). +async fn backfill_search_norm( + conn: &mut PgConnection, + user_id: i64, + podcast_id: &PodcastId, +) -> Result<()> { + let rows = sqlx::query( + "SELECT episode_id, title, description FROM furumusic__podcast_episode + WHERE user_id = $1 AND podcast_id = $2 AND search_norm IS NULL", + ) + .bind(user_id) + .bind(podcast_id.as_str()) + .fetch_all(&mut *conn) + .await?; + for row in rows { + let title: String = row.try_get("title")?; + let description: Option = row.try_get("description")?; + sqlx::query( + "UPDATE furumusic__podcast_episode SET search_norm = $3 + WHERE user_id = $1 AND episode_id = $2 AND search_norm IS NULL", + ) + .bind(user_id) + .bind(row.try_get::("episode_id")?) + .bind(episode_search_norm(&title, description.as_deref())) + .execute(&mut *conn) + .await?; + } + Ok(()) +} + +impl PodcastStore { + /// Podcasts with subscription state and counters, sorted by title. + pub(crate) async fn list_podcasts( + &self, + user_id: i64, + filter: PodcastListFilter, + ) -> Result> { + let mut conn = self.pool.acquire().await?; + let join = match filter { + PodcastListFilter::Subscribed => { + "JOIN furumusic__podcast_subscription r + ON r.user_id = p.user_id AND r.podcast_id = p.podcast_id AND r.subscribed" + } + PodcastListFilter::All => "", + }; + let rows = sqlx::query(&format!( + "SELECT p.podcast_id FROM furumusic__podcast p {join} + WHERE p.user_id = $1 + ORDER BY lower(p.title), p.podcast_id" + )) + .bind(user_id) + .fetch_all(&mut *conn) + .await?; + let ids = rows + .iter() + .map(|row| podcast_id_col(row, "podcast_id")) + .collect::>>()?; + let mut out = Vec::with_capacity(ids.len()); + for id in ids { + if let Some(summary) = summary_in(&mut conn, user_id, &id).await? { + out.push(summary); + } + } + Ok(out) + } + + /// One podcast with subscription state and counters. + pub(crate) async fn podcast( + &self, + user_id: i64, + podcast_id: &PodcastId, + ) -> Result> { + let mut conn = self.pool.acquire().await?; + summary_in(&mut conn, user_id, podcast_id).await + } + + /// The subscription register (with the stored alias set), if any. + pub(crate) async fn subscription( + &self, + user_id: i64, + podcast_id: &PodcastId, + ) -> Result> { + let mut conn = self.pool.acquire().await?; + apply::load_subscription_register(&mut conn, user_id, podcast_id).await + } + + /// The share register, if any. + #[cfg(test)] + pub(crate) async fn share( + &self, + user_id: i64, + podcast_id: &PodcastId, + ) -> Result> { + let mut conn = self.pool.acquire().await?; + apply::load_share(&mut conn, user_id, podcast_id).await + } + + /// The bounded, synced feed alias set of a podcast. + #[cfg(test)] + pub(crate) async fn feed_aliases( + &self, + user_id: i64, + podcast_id: &PodcastId, + ) -> Result> { + let mut conn = self.pool.acquire().await?; + apply::load_aliases(&mut conn, user_id, podcast_id).await + } + + /// Season groups of a podcast: numbered seasons ascending, then the + /// group of episodes without a season. + pub(crate) async fn seasons( + &self, + user_id: i64, + podcast_id: &PodcastId, + ) -> Result> { + let rows = sqlx::query(&format!( + "SELECT s.season, s.episode_count, s.first_published_at_ms, s.last_published_at_ms, + (SELECT COUNT(*) FROM furumusic__podcast_episode e + LEFT JOIN furumusic__podcast_progress g + ON g.user_id = e.user_id AND g.episode_id = e.episode_id + WHERE e.user_id = s.user_id AND e.podcast_id = s.podcast_id + AND e.season IS NOT DISTINCT FROM s.season + AND {NOT_COMPLETED}) AS unplayed_count + FROM furumusic__podcast_season s + WHERE s.user_id = $1 AND s.podcast_id = $2 + ORDER BY s.season IS NULL, s.season" + )) + .bind(user_id) + .bind(podcast_id.as_str()) + .fetch_all(&self.pool) + .await?; + let seasons = rows + .iter() + .map(|row| { + Ok(SeasonSummary { + season: u32_from_sql(row.try_get("season")?), + episode_count: count(row.try_get("episode_count")?), + first_published_at_ms: row.try_get("first_published_at_ms")?, + last_published_at_ms: row.try_get("last_published_at_ms")?, + unplayed_count: count(row.try_get("unplayed_count")?), + }) + }) + .collect::>>()?; + Ok(seasons) + } + + /// Episodes of a podcast or season with their progress. + pub(crate) async fn episodes( + &self, + user_id: i64, + query: &EpisodeQuery, + ) -> Result> { + let mut conn = self.pool.acquire().await?; + let needle = query + .search + .as_deref() + .map(music_dht::normalize_name) + .filter(|needle| !needle.is_empty()); + if needle.is_some() { + backfill_search_norm(&mut conn, user_id, &query.podcast_id).await?; + } + let mut sql: QueryBuilder<'_, Postgres> = QueryBuilder::new(format!( + "SELECT {EPISODE_COLUMNS}, e.in_feed, {}, + (SELECT COUNT(*) FROM furumusic__podcast_listen_session s + WHERE s.user_id = e.user_id AND s.episode_id = e.episode_id + AND s.completed) AS completed_count + FROM furumusic__podcast_episode e + LEFT JOIN furumusic__podcast_progress g + ON g.user_id = e.user_id AND g.episode_id = e.episode_id + WHERE e.user_id = ", + progress_columns("g", "g_") + )); + sql.push_bind(user_id); + sql.push(" AND e.podcast_id = "); + sql.push_bind(query.podcast_id.as_str().to_string()); + match query.season { + SeasonFilter::Any => {} + SeasonFilter::Season(season) => { + sql.push(" AND e.season = "); + sql.push_bind(i64::from(season)); + } + SeasonFilter::NoSeason => { + sql.push(" AND e.season IS NULL"); + } + } + match query.played { + PlayedFilter::Any => {} + PlayedFilter::Unplayed => { + sql.push(format!(" AND {NOT_COMPLETED}")); + } + PlayedFilter::NotStarted => { + sql.push(" AND COALESCE(g.status, 'not_started') = 'not_started'"); + } + PlayedFilter::InProgress => { + sql.push(" AND g.status = 'in_progress'"); + } + PlayedFilter::Played => { + sql.push(" AND g.status = 'completed'"); + } + } + if let Some(needle) = needle { + sql.push(" AND strpos(e.search_norm, "); + sql.push_bind(needle); + sql.push(") > 0"); + } + sql.push(format!(" ORDER BY {}", query.sort.order_by())); + if let Some(limit) = query.limit { + sql.push(" LIMIT "); + sql.push_bind(i64::try_from(limit).unwrap_or(i64::MAX)); + } + sql.push(" OFFSET "); + sql.push_bind(i64::try_from(query.offset).unwrap_or(i64::MAX)); + let rows = sql.build().fetch_all(&mut *conn).await?; + Ok(rows + .iter() + .map(list_item_from_row) + .collect::>>()?) + } + + /// One episode with its progress. + pub(crate) async fn episode( + &self, + user_id: i64, + episode_id: &EpisodeId, + ) -> Result> { + let row = sqlx::query(&format!( + "SELECT {EPISODE_COLUMNS}, e.in_feed, {}, + (SELECT COUNT(*) FROM furumusic__podcast_listen_session s + WHERE s.user_id = e.user_id AND s.episode_id = e.episode_id + AND s.completed) AS completed_count + FROM furumusic__podcast_episode e + LEFT JOIN furumusic__podcast_progress g + ON g.user_id = e.user_id AND g.episode_id = e.episode_id + WHERE e.user_id = $1 AND e.episode_id = $2", + progress_columns("g", "g_") + )) + .bind(user_id) + .bind(episode_id.as_str()) + .fetch_optional(&self.pool) + .await?; + Ok(row.as_ref().map(list_item_from_row).transpose()?) + } + + /// Position ordinary Play resumes from + /// (`ProgressRecord::resume_position_ms`, 0 without progress). + #[cfg(test)] + pub(crate) async fn resume_position_ms( + &self, + user_id: i64, + episode_id: &EpisodeId, + ) -> Result { + Ok(self + .progress(user_id, episode_id) + .await? + .map_or(0, |progress| progress.resume_position_ms())) + } + + /// Listening history (sessions that `should_record`), newest first. + pub(crate) async fn listen_history( + &self, + user_id: i64, + podcast_id: Option<&PodcastId>, + limit: usize, + ) -> Result> { + let rows = sqlx::query(&format!( + "SELECT {SESSION_COLUMNS}, e.title AS episode_title, p.title AS podcast_title + FROM furumusic__podcast_listen_session s + LEFT JOIN furumusic__podcast_episode e + ON e.user_id = s.user_id AND e.episode_id = s.episode_id + LEFT JOIN furumusic__podcast p + ON p.user_id = s.user_id AND p.podcast_id = s.podcast_id + WHERE s.user_id = $1 AND (s.completed OR s.listened_ms >= $2) + AND ($3::text IS NULL OR s.podcast_id = $3) + ORDER BY s.ended_at_ms DESC, s.session_id DESC + LIMIT $4" + )) + .bind(user_id) + .bind(music_dht::podcast::MIN_RECORDED_EPISODE_LISTEN_MS) + .bind(podcast_id.map(PodcastId::as_str)) + .bind(i64::try_from(limit).unwrap_or(i64::MAX)) + .fetch_all(&self.pool) + .await?; + Ok(rows + .iter() + .map(|row| { + Ok(PodcastListenEntry { + session: session_from_row(row)?, + episode_title: row.try_get("episode_title")?, + podcast_title: row.try_get("podcast_title")?, + }) + }) + .collect::>>()?) + } + + /// Number of distinct completed sessions of an episode. + #[cfg(test)] + pub(crate) async fn completed_listen_count( + &self, + user_id: i64, + episode_id: &EpisodeId, + ) -> Result { + let total: i64 = sqlx::query_scalar( + "SELECT COUNT(*) FROM furumusic__podcast_listen_session + WHERE user_id = $1 AND episode_id = $2 AND completed", + ) + .bind(user_id) + .bind(episode_id.as_str()) + .fetch_one(&self.pool) + .await?; + Ok(count(total)) + } + + /// Generations of the progress registers of `episode_ids` (episodes + /// without a register are missing from the map). + /// + /// Outgoing playback states carry them as + /// `PodcastEpisodeRef::progress_generation`, so the device that takes + /// over writes its explicit action after everything this account saw. + pub(crate) async fn progress_generations( + &self, + user_id: i64, + episode_ids: &[EpisodeId], + ) -> Result> { + if episode_ids.is_empty() { + return Ok(Default::default()); + } + let ids: Vec<&str> = episode_ids.iter().map(EpisodeId::as_str).collect(); + let rows = sqlx::query( + "SELECT episode_id, generation FROM furumusic__podcast_progress + WHERE user_id = $1 AND episode_id = ANY($2)", + ) + .bind(user_id) + .bind(&ids) + .fetch_all(&self.pool) + .await?; + rows.iter() + .map(|row| { + Ok(( + episode_id_col(row, "episode_id")?, + super::generation_from_sql(row.try_get("generation")?), + )) + }) + .collect::>() + .map_err(Into::into) + } + + /// Audio URL and enclosure type of an episode in this account's catalog + /// (`None` when the account does not know the episode). The audio proxy + /// and the playback source endpoint take URLs only from here. + pub(crate) async fn audio_source( + &self, + user_id: i64, + episode_id: &EpisodeId, + ) -> Result)>> { + let row = sqlx::query( + "SELECT audio_url, mime FROM furumusic__podcast_episode + WHERE user_id = $1 AND episode_id = $2", + ) + .bind(user_id) + .bind(episode_id.as_str()) + .fetch_optional(&self.pool) + .await?; + Ok(row + .map(|row| Ok::<_, sqlx::Error>((row.try_get("audio_url")?, row.try_get("mime")?))) + .transpose()?) + } +} diff --git a/src/podcasts/refresh.rs b/src/podcasts/refresh.rs new file mode 100644 index 0000000..cf0a033 --- /dev/null +++ b/src/podcasts/refresh.rs @@ -0,0 +1,486 @@ +//! Feed refresh ingestion and the local-only fetch state. + +use std::collections::BTreeSet; + +use anyhow::Result; +use music_dht::device_sync::{SyncOpPayload, SyncOpWire}; +use music_dht::podcast::feed::ParsedFeed; +use music_dht::podcast::{EpisodeMeta, FeedUrl, PodcastId, merge_feed_aliases}; +use sqlx::{PgConnection, Row}; + +use super::apply::{self, EpisodeSource}; +use super::{PodcastStore, PodcastStoreError, episode_search_norm, now_ms}; +use crate::federation::devices; + +/// A fetched and parsed feed document for [`PodcastStore::apply_feed`]. +#[derive(Debug, Clone, Copy)] +pub(crate) struct FeedRefresh<'a> { + /// Podcast the feed belongs to. + pub(crate) podcast_id: &'a PodcastId, + /// The parsed document (`music_dht::podcast::feed::parse_feed_with_base` + /// with `fetched_from` as base). + pub(crate) feed: &'a ParsedFeed, + /// URL the document was finally served from, after redirects. + pub(crate) fetched_from: &'a FeedUrl, + /// Whether every redirect leading to `fetched_from` was permanent + /// (301/308). Temporary redirects do not move the feed. + pub(crate) moved_permanently: bool, +} + +/// Result of [`PodcastStore::apply_feed`]. +#[derive(Debug, Clone, Default, PartialEq)] +pub(crate) struct RefreshOutcome { + /// Episodes seen for the first time. + pub(crate) new_episodes: usize, + /// Known episodes whose metadata changed (title, audio URL, ...). + pub(crate) updated_episodes: usize, + /// Known episodes without metadata changes. + pub(crate) unchanged_episodes: usize, + /// Items without identity or with invalid metadata. + pub(crate) skipped_items: usize, + /// Feed URLs learned from a permanent redirect or `itunes:new-feed-url`. + pub(crate) learned_aliases: Vec, + /// URL the podcast is fetched from after this refresh. + pub(crate) feed_url: Option, + /// `podcast_feed_aliases_added`, only when the learned URLs changed the + /// stored alias set (already appended to the sync log). + pub(crate) ops: Vec, +} + +/// Local-only fetch state of a podcast. Never part of sync output. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub(crate) struct FetchState { + /// HTTP `ETag` validator of the last successful fetch. + pub(crate) etag: Option, + /// HTTP `Last-Modified` validator of the last successful fetch. + pub(crate) last_modified: Option, + /// Last fetch attempt, unix milliseconds. + pub(crate) checked_at_ms: Option, + /// Last successful fetch (including "not modified"). + pub(crate) last_success_at_ms: Option, + /// When the scheduler should check again (`None`: due now). + pub(crate) next_check_at_ms: Option, + /// Error of the last attempt; `None` after a success. + pub(crate) last_error: Option, + /// Failed attempts since the last success. + pub(crate) consecutive_failures: u32, + /// Image URL the cached cover was downloaded from. + pub(crate) cover_url: Option, + /// Local path of the cached cover. + pub(crate) cover_path: Option, +} + +async fn podcast_fetch_url( + conn: &mut PgConnection, + user_id: i64, + podcast_id: &PodcastId, +) -> Result> { + let row = sqlx::query( + "SELECT feed_url FROM furumusic__podcast WHERE user_id = $1 AND podcast_id = $2", + ) + .bind(user_id) + .bind(podcast_id.as_str()) + .fetch_optional(&mut *conn) + .await?; + Ok(row + .as_ref() + .map(|row| super::feed_url_col(row, "feed_url")) + .transpose()?) +} + +async fn fetch_state_in( + conn: &mut PgConnection, + user_id: i64, + podcast_id: &PodcastId, + lock: bool, +) -> Result { + let row = sqlx::query(&format!( + "SELECT etag, last_modified, checked_at_ms, last_success_at_ms, next_check_at_ms, + last_error, consecutive_failures, cover_url, cover_path + FROM furumusic__podcast_fetch_state WHERE user_id = $1 AND podcast_id = $2{}", + if lock { " FOR UPDATE" } else { "" } + )) + .bind(user_id) + .bind(podcast_id.as_str()) + .fetch_optional(&mut *conn) + .await?; + let Some(row) = row else { + return Ok(FetchState::default()); + }; + Ok(FetchState { + etag: row.try_get("etag")?, + last_modified: row.try_get("last_modified")?, + checked_at_ms: row.try_get("checked_at_ms")?, + last_success_at_ms: row.try_get("last_success_at_ms")?, + next_check_at_ms: row.try_get("next_check_at_ms")?, + last_error: row.try_get("last_error")?, + consecutive_failures: u32::try_from(row.try_get::("consecutive_failures")?) + .unwrap_or(u32::MAX), + cover_url: row.try_get("cover_url")?, + cover_path: row.try_get("cover_path")?, + }) +} + +/// Inserts or updates a feed episode; `None` when it is new, otherwise +/// whether its metadata changed. +async fn upsert_feed_episode( + conn: &mut PgConnection, + user_id: i64, + meta: &EpisodeMeta, + now: i64, +) -> Result> { + let Some(stored) = super::queries::episode_meta_in(conn, user_id, &meta.episode_id).await? + else { + apply::insert_episode(conn, user_id, meta, EpisodeSource::Feed, true, now).await?; + return Ok(None); + }; + let changed = stored != *meta; + sqlx::query( + "UPDATE furumusic__podcast_episode + SET podcast_id = $3, guid = $4, title = $5, description = $6, season = $7, + episode_number = $8, episode_type = $9, published_at_ms = $10, + duration_ms = $11, audio_url = $12, mime = $13, byte_len = $14, + image_url = $15, link = $16, source = 'feed', in_feed = TRUE, + search_norm = $17, + updated_at_ms = CASE WHEN $18 THEN $19 ELSE updated_at_ms END + WHERE user_id = $1 AND episode_id = $2", + ) + .bind(user_id) + .bind(meta.episode_id.as_str()) + .bind(meta.podcast_id.as_str()) + .bind(&meta.guid) + .bind(&meta.title) + .bind(&meta.description) + .bind(meta.season.map(i64::from)) + .bind(meta.episode_number.map(i64::from)) + .bind(apply::episode_type_to_sql(meta.episode_type)) + .bind(meta.published_at_ms) + .bind(meta.duration_ms) + .bind(&meta.audio_url) + .bind(&meta.mime) + .bind(meta.byte_len.and_then(|len| i64::try_from(len).ok())) + .bind(&meta.image_url) + .bind(&meta.link) + .bind(episode_search_norm( + &meta.title, + meta.description.as_deref(), + )) + .bind(changed) + .bind(now) + .execute(&mut *conn) + .await?; + Ok(Some(changed)) +} + +impl PodcastStore { + /// Ingests a fetched feed document for a podcast of the account. + /// + /// * Channel metadata of the catalog row is replaced by the feed's. + /// * Episode ids are resolved with the contract identity rule + /// (`ParsedFeed::resolve_episode_ids`) against every stored episode of + /// the podcast, including episodes learned through sync, so a changed + /// title, audio URL or a GUID-less item keeps its id. Known episodes + /// get the new metadata (including the audio URL); new ones are added; + /// episodes missing from the document are kept (`in_feed = false`). + /// * A permanent redirect target and `itunes:new-feed-url` become feed + /// aliases and the new fetch URL. When they change the stored alias + /// set, one `podcast_feed_aliases_added` operation is appended. + /// * Subscription, share and progress registers and listening sessions + /// are never touched, so a refresh cannot resurrect an unsubscribed + /// podcast or move anyone's position. + /// + /// Record the HTTP outcome separately with + /// [`PodcastStore::record_fetch_success`] and friends. + pub(crate) async fn apply_feed( + &self, + user_id: i64, + refresh: &FeedRefresh<'_>, + ) -> Result { + let podcast_id = refresh.podcast_id; + let feed = refresh.feed; + let now = now_ms(); + let device_id = devices::local_device_id(&self.pool, user_id).await?; + let mut tx = self.begin_write(user_id).await?; + let current_url = match podcast_fetch_url(&mut tx, user_id, podcast_id).await? { + Some(url) => url, + None => { + let Some(register) = + apply::load_subscription_register(&mut tx, user_id, podcast_id).await? + else { + return Err(PodcastStoreError::UnknownPodcast(podcast_id.clone()).into()); + }; + apply::apply_subscription_record(&mut tx, user_id, ®ister, now).await?; + register.podcast.feed_url + } + }; + + let known = apply::known_episodes(&mut tx, user_id, podcast_id).await?; + let ids = feed.resolve_episode_ids(podcast_id, &known); + sqlx::query( + "UPDATE furumusic__podcast_episode SET in_feed = FALSE + WHERE user_id = $1 AND podcast_id = $2", + ) + .bind(user_id) + .bind(podcast_id.as_str()) + .execute(&mut *tx) + .await?; + let mut outcome = RefreshOutcome::default(); + for (item, id) in feed.episodes.iter().zip(ids) { + let Some(id) = id else { + outcome.skipped_items += 1; + continue; + }; + let meta = item.to_meta(podcast_id, id); + if meta.validate().is_err() { + outcome.skipped_items += 1; + continue; + } + match upsert_feed_episode(&mut tx, user_id, &meta, now).await? { + None => outcome.new_episodes += 1, + Some(true) => outcome.updated_episodes += 1, + Some(false) => outcome.unchanged_episodes += 1, + } + } + + let mut fetch_url = current_url.clone(); + let mut learned = BTreeSet::new(); + if refresh.moved_permanently && *refresh.fetched_from != fetch_url { + learned.insert(fetch_url.clone()); + learned.insert(refresh.fetched_from.clone()); + fetch_url = refresh.fetched_from.clone(); + } + if let Some(new_url) = feed + .new_feed_url + .as_deref() + .and_then(|url| FeedUrl::parse(url).ok()) + && new_url != fetch_url + { + learned.insert(fetch_url.clone()); + learned.insert(new_url.clone()); + fetch_url = new_url; + } + let meta = feed.to_podcast_meta(fetch_url.clone()); + sqlx::query( + "UPDATE furumusic__podcast + SET feed_url = $3, title = $4, author = $5, description = $6, link = $7, + image_url = $8, meta_from_feed = TRUE, updated_at_ms = $9 + WHERE user_id = $1 AND podcast_id = $2", + ) + .bind(user_id) + .bind(podcast_id.as_str()) + .bind(meta.feed_url.as_str()) + .bind(&meta.title) + .bind(&meta.author) + .bind(&meta.description) + .bind(&meta.link) + .bind(&meta.image_url) + .bind(now) + .execute(&mut *tx) + .await?; + if !learned.is_empty() { + let learned: Vec = learned.into_iter().collect(); + let current = apply::load_aliases(&mut tx, user_id, podcast_id).await?; + if merge_feed_aliases(podcast_id, ¤t, &learned) != current { + let op = super::actions::emit(&mut tx, user_id, &device_id, |_| { + SyncOpPayload::PodcastFeedAliasesAdded { + podcast_id: podcast_id.clone(), + feed_aliases: learned.clone(), + } + }) + .await?; + outcome.ops.push(op); + } + outcome.learned_aliases = learned; + } + tx.commit().await?; + outcome.feed_url = Some(fetch_url); + Ok(outcome) + } + + /// Local fetch state of a podcast (defaults when never fetched). + pub(crate) async fn fetch_state( + &self, + user_id: i64, + podcast_id: &PodcastId, + ) -> Result { + let mut conn = self.pool.acquire().await?; + fetch_state_in(&mut conn, user_id, podcast_id, false).await + } + + /// Read-modify-write of the fetch state in one transaction with the + /// account lock and the row locked, so concurrent `record_*` / + /// `set_cover_cache` calls never lose each other's columns or failure + /// increments. + async fn update_fetch_state( + &self, + user_id: i64, + podcast_id: &PodcastId, + update: impl FnOnce(&mut FetchState), + ) -> Result { + let mut tx = self.begin_write(user_id).await?; + if podcast_fetch_url(&mut tx, user_id, podcast_id) + .await? + .is_none() + { + return Err(PodcastStoreError::UnknownPodcast(podcast_id.clone()).into()); + } + let mut state = fetch_state_in(&mut tx, user_id, podcast_id, true).await?; + update(&mut state); + sqlx::query( + "INSERT INTO furumusic__podcast_fetch_state + (user_id, podcast_id, etag, last_modified, checked_at_ms, last_success_at_ms, + next_check_at_ms, last_error, consecutive_failures, cover_url, cover_path) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11) + ON CONFLICT (user_id, podcast_id) DO UPDATE SET + etag = EXCLUDED.etag, last_modified = EXCLUDED.last_modified, + checked_at_ms = EXCLUDED.checked_at_ms, + last_success_at_ms = EXCLUDED.last_success_at_ms, + next_check_at_ms = EXCLUDED.next_check_at_ms, + last_error = EXCLUDED.last_error, + consecutive_failures = EXCLUDED.consecutive_failures, + cover_url = EXCLUDED.cover_url, cover_path = EXCLUDED.cover_path", + ) + .bind(user_id) + .bind(podcast_id.as_str()) + .bind(&state.etag) + .bind(&state.last_modified) + .bind(state.checked_at_ms) + .bind(state.last_success_at_ms) + .bind(state.next_check_at_ms) + .bind(&state.last_error) + .bind(i64::from(state.consecutive_failures)) + .bind(&state.cover_url) + .bind(&state.cover_path) + .execute(&mut *tx) + .await?; + tx.commit().await?; + Ok(state) + } + + /// Records a successful fetch (HTTP 200) with its new validators. + pub(crate) async fn record_fetch_success( + &self, + user_id: i64, + podcast_id: &PodcastId, + now_ms: i64, + etag: Option<&str>, + last_modified: Option<&str>, + next_check_at_ms: Option, + ) -> Result<()> { + self.update_fetch_state(user_id, podcast_id, |state| { + state.etag = etag.map(str::to_string); + state.last_modified = last_modified.map(str::to_string); + state.checked_at_ms = Some(now_ms); + state.last_success_at_ms = Some(now_ms); + state.next_check_at_ms = next_check_at_ms; + state.last_error = None; + state.consecutive_failures = 0; + }) + .await + .map(|_| ()) + } + + /// Records a conditional fetch answered with "304 Not Modified". + pub(crate) async fn record_fetch_not_modified( + &self, + user_id: i64, + podcast_id: &PodcastId, + now_ms: i64, + next_check_at_ms: Option, + ) -> Result<()> { + self.update_fetch_state(user_id, podcast_id, |state| { + state.checked_at_ms = Some(now_ms); + state.last_success_at_ms = Some(now_ms); + state.next_check_at_ms = next_check_at_ms; + state.last_error = None; + state.consecutive_failures = 0; + }) + .await + .map(|_| ()) + } + + /// Records a failed fetch; returns the consecutive failure count. The + /// stored catalog is kept. + pub(crate) async fn record_fetch_failure( + &self, + user_id: i64, + podcast_id: &PodcastId, + now_ms: i64, + error: &str, + next_check_at_ms: Option, + ) -> Result { + self.update_fetch_state(user_id, podcast_id, |state| { + state.checked_at_ms = Some(now_ms); + state.next_check_at_ms = next_check_at_ms; + state.last_error = Some(error.to_string()); + state.consecutive_failures = state.consecutive_failures.saturating_add(1); + }) + .await + .map(|state| state.consecutive_failures) + } + + /// Records the locally cached cover image of a podcast. + // Covers are not cached yet: the browser loads `image_url` from the + // publisher. The column and this setter stay for a later cache. + #[cfg(test)] + pub(crate) async fn set_cover_cache( + &self, + user_id: i64, + podcast_id: &PodcastId, + cover_url: Option<&str>, + cover_path: Option<&str>, + ) -> Result<()> { + self.update_fetch_state(user_id, podcast_id, |state| { + state.cover_url = cover_url.map(str::to_string); + state.cover_path = cover_path.map(str::to_string); + }) + .await + .map(|_| ()) + } + + /// Subscribed podcasts of every account due for a refresh at `now_ms` + /// (never checked or `next_check_at_ms <= now_ms`) with their fetch + /// URL: `(user_id, podcast_id, feed_url)`, at most `limit`. Accounts + /// take turns: first the most overdue podcast of every account, then the + /// second, and so on; within a turn the most overdue first. An account + /// with many due feeds therefore never delays the others, and still uses + /// whatever the run has left over. + pub(crate) async fn podcasts_due_for_refresh( + &self, + now_ms: i64, + limit: usize, + ) -> Result> { + let rows = sqlx::query( + "SELECT user_id, podcast_id, feed_url + FROM (SELECT p.user_id, p.podcast_id, p.feed_url, + COALESCE(f.next_check_at_ms, 0) AS due, + ROW_NUMBER() OVER ( + PARTITION BY p.user_id + ORDER BY COALESCE(f.next_check_at_ms, 0), p.podcast_id + ) AS turn + FROM furumusic__podcast p + JOIN furumusic__podcast_subscription s + ON s.user_id = p.user_id AND s.podcast_id = p.podcast_id + AND s.subscribed + LEFT JOIN furumusic__podcast_fetch_state f + ON f.user_id = p.user_id AND f.podcast_id = p.podcast_id + WHERE f.next_check_at_ms IS NULL OR f.next_check_at_ms <= $1) due + ORDER BY turn, due, user_id, podcast_id + LIMIT $2", + ) + .bind(now_ms) + .bind(i64::try_from(limit).unwrap_or(i64::MAX)) + .fetch_all(&self.pool) + .await?; + Ok(rows + .iter() + .map(|row| { + Ok(( + row.try_get("user_id")?, + super::podcast_id_col(row, "podcast_id")?, + super::feed_url_col(row, "feed_url")?, + )) + }) + .collect::>>()?) + } +} diff --git a/src/podcasts/runner.rs b/src/podcasts/runner.rs new file mode 100644 index 0000000..0ae27b3 --- /dev/null +++ b/src/podcasts/runner.rs @@ -0,0 +1,485 @@ +//! Scheduling of feed refreshes: which feeds are due, one request per feed +//! URL per run, ingestion through [`PodcastStore::apply_feed`] and the local +//! fetch state (validators, next check, backoff). + +use std::collections::BTreeMap; +use std::fmt; +use std::time::Duration; + +use anyhow::Result; +use futures_util::StreamExt; +use music_dht::podcast::{FeedUrl, PodcastId}; + +use super::fetch::{FeedFetch, FeedFetcher, FetchError, FetchedFeed, Validators}; +use super::{FeedRefresh, PodcastStore, PodcastStoreError, RefreshOutcome}; + +/// Regular interval between two checks of a healthy feed. +pub(crate) const REFRESH_INTERVAL_MS: i64 = 60 * 60 * 1000; +/// Spread of the regular interval, so feeds added together drift apart. +const REFRESH_JITTER_MS: i64 = 10 * 60 * 1000; +/// First retry delay after a failure; doubles per consecutive failure. +pub(crate) const BACKOFF_BASE_MS: i64 = 15 * 60 * 1000; +/// Longest delay between two attempts of a failing feed. +pub(crate) const BACKOFF_MAX_MS: i64 = 24 * 60 * 60 * 1000; +/// A manual refresh is refused when the feed was checked this recently. +pub(crate) const MANUAL_MIN_INTERVAL_MS: i64 = 30 * 1000; +/// Podcasts considered per scheduled run. Due podcasts are taken in turns +/// across accounts (see [`PodcastStore::podcasts_due_for_refresh`]), so an +/// account with many (or slow) feeds cannot hold back everyone else's +/// refreshes, but can use a run nobody else needs. +pub(crate) const RUN_LIMIT: usize = 200; +/// Feed requests in flight at once. +pub(crate) const RUN_CONCURRENCY: usize = 4; + +/// Delay before the next attempt after `consecutive_failures` failures in a +/// row (counting the one just recorded): 15 min, 30 min, 1 h, ... capped at a +/// day. A longer server-requested `Retry-After` wins (also capped). +pub(crate) fn failure_backoff_ms(consecutive_failures: u32, retry_after: Option) -> i64 { + let exponent = consecutive_failures.saturating_sub(1).min(16); + let backoff = BACKOFF_BASE_MS + .saturating_mul(1_i64 << exponent) + .min(BACKOFF_MAX_MS); + let requested = retry_after + .map(|delay| i64::try_from(delay.as_millis()).unwrap_or(i64::MAX)) + .unwrap_or(0); + backoff.max(requested).min(BACKOFF_MAX_MS) +} + +/// Next regular check after a success: the interval plus a stable per +/// account and podcast jitter. +pub(crate) fn next_regular_check_ms(now_ms: i64, user_id: i64, podcast_id: &PodcastId) -> i64 { + let key = format!("{user_id}:{podcast_id}"); + let hash = blake3::hash(key.as_bytes()); + let spread = u64::from_le_bytes(hash.as_bytes()[..8].try_into().expect("8 bytes")); + let jitter = i64::try_from(spread % REFRESH_JITTER_MS as u64).unwrap_or(0); + now_ms + REFRESH_INTERVAL_MS + jitter +} + +/// One podcast of one account to refresh. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct RefreshTarget { + pub(crate) user_id: i64, + pub(crate) podcast_id: PodcastId, +} + +/// Result of refreshing one [`RefreshTarget`]. +#[derive(Debug, Clone, PartialEq)] +pub(crate) enum TargetResult { + /// A new document was ingested. + Updated(RefreshOutcome), + /// `304 Not Modified`. + NotModified, + /// The fetch or the ingestion failed; the catalog is unchanged. + Failed { + /// What the user sees (stored as the last error). + error: String, + /// Details for the server log (no URLs). + detail: String, + consecutive_failures: u32, + next_check_at_ms: i64, + }, + /// Nothing to do any more (unsubscribed or removed meanwhile). + Skipped(String), +} + +/// Counters of a scheduled run. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub(crate) struct RunSummary { + /// Account podcasts that were due. + pub(crate) targets: usize, + /// Distinct feed URLs requested. + pub(crate) feeds: usize, + pub(crate) updated: usize, + pub(crate) not_modified: usize, + pub(crate) failed: usize, + pub(crate) skipped: usize, + pub(crate) new_episodes: usize, + pub(crate) updated_episodes: usize, + /// `(podcast_id, details)` of failed targets, for the job log (no + /// account ids or URLs: feed URLs often carry access tokens). + pub(crate) failures: Vec<(PodcastId, String)>, +} + +impl RunSummary { + fn add(&mut self, target: &RefreshTarget, result: &TargetResult) { + match result { + TargetResult::Updated(outcome) => { + self.updated += 1; + self.new_episodes += outcome.new_episodes; + self.updated_episodes += outcome.updated_episodes; + } + TargetResult::NotModified => self.not_modified += 1, + TargetResult::Failed { detail, .. } => { + self.failed += 1; + self.failures + .push((target.podcast_id.clone(), detail.clone())); + } + TargetResult::Skipped(_) => self.skipped += 1, + } + } +} + +impl fmt::Display for RunSummary { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!( + f, + "{} due podcasts, {} feeds requested: {} updated ({} new, {} changed episodes), \ + {} not modified, {} failed, {} skipped", + self.targets, + self.feeds, + self.updated, + self.new_episodes, + self.updated_episodes, + self.not_modified, + self.failed, + self.skipped + ) + } +} + +/// Due podcasts grouped by feed URL: accounts subscribed to the same URL +/// share one request. +pub(crate) type RefreshGroups = BTreeMap>; + +/// Groups the rows of [`PodcastStore::podcasts_due_for_refresh`]. +pub(crate) fn group_targets(due: Vec<(i64, PodcastId, FeedUrl)>) -> RefreshGroups { + let mut groups = RefreshGroups::new(); + for (user_id, podcast_id, feed_url) in due { + groups.entry(feed_url).or_default().push(RefreshTarget { + user_id, + podcast_id, + }); + } + groups +} + +/// Refreshes podcasts due at `now_ms` (at most `limit`, accounts in +/// turns), with at most +/// `concurrency` feed requests in flight. Accounts subscribed to the same +/// feed URL share one request. A failing feed never stops the others. +pub(crate) async fn refresh_due( + store: &PodcastStore, + fetcher: &FeedFetcher, + now_ms: i64, + limit: usize, + concurrency: usize, +) -> Result { + let due = store.podcasts_due_for_refresh(now_ms, limit).await?; + refresh_groups(store, fetcher, group_targets(due), now_ms, concurrency).await +} + +/// Refreshes `groups`, one request per feed URL, at most `concurrency` at +/// a time. +pub(crate) async fn refresh_groups( + store: &PodcastStore, + fetcher: &FeedFetcher, + groups: RefreshGroups, + now_ms: i64, + concurrency: usize, +) -> Result { + let mut summary = RunSummary { + targets: groups.values().map(Vec::len).sum(), + feeds: groups.len(), + ..RunSummary::default() + }; + let mut results = futures_util::stream::iter(groups) + .map(|(feed_url, targets)| async move { + refresh_group(store, fetcher, &feed_url, targets, now_ms).await + }) + .buffer_unordered(concurrency.max(1)); + while let Some(group) = results.next().await { + for (target, result) in group? { + summary.add(&target, &result); + } + } + Ok(summary) +} + +/// Fetches `feed_url` once for all `targets` and records the outcome for +/// each. The request is conditional only when every target holds the same +/// validators (the usual case: one account). +pub(crate) async fn refresh_group( + store: &PodcastStore, + fetcher: &FeedFetcher, + feed_url: &FeedUrl, + targets: Vec, + now_ms: i64, +) -> Result> { + let mut states = Vec::with_capacity(targets.len()); + for target in &targets { + states.push( + store + .fetch_state(target.user_id, &target.podcast_id) + .await?, + ); + } + let validators = states + .iter() + .map(|state| Validators { + etag: state.etag.clone(), + last_modified: state.last_modified.clone(), + }) + .reduce(|a, b| if a == b { a } else { Validators::default() }) + .unwrap_or_default(); + let fetched = fetcher.fetch(feed_url, &validators).await; + let mut out = Vec::with_capacity(targets.len()); + for (target, state) in targets.into_iter().zip(states) { + let result = + record_result(store, &target, &fetched, state.consecutive_failures, now_ms).await?; + out.push((target, result)); + } + Ok(out) +} + +/// Ingests (or records the failure of) one fetch for one target. +async fn record_result( + store: &PodcastStore, + target: &RefreshTarget, + fetched: &Result, + previous_failures: u32, + now_ms: i64, +) -> Result { + let (user_id, podcast_id) = (target.user_id, &target.podcast_id); + let failure = |error: String, detail: String, retry_after: Option| { + let consecutive_failures = previous_failures.saturating_add(1); + let next_check_at_ms = now_ms + failure_backoff_ms(consecutive_failures, retry_after); + (error, detail, consecutive_failures, next_check_at_ms) + }; + let (error, detail, consecutive_failures, next_check_at_ms) = match fetched { + Ok(FeedFetch::NotModified) => { + let next = next_regular_check_ms(now_ms, user_id, podcast_id); + return match store + .record_fetch_not_modified(user_id, podcast_id, now_ms, Some(next)) + .await + { + Ok(()) => Ok(TargetResult::NotModified), + Err(error) => skipped_or(error), + }; + } + Ok(FeedFetch::Fetched(document)) => { + match ingest(store, target, document, now_ms).await { + Ok(outcome) => return Ok(TargetResult::Updated(outcome)), + Err(error) if is_gone(&error) => return skipped_or(error), + // A store failure is recorded like a fetch failure, so the + // feed backs off instead of being retried every run. + Err(error) => failure( + "the feed could not be stored".to_string(), + format!("storing the feed failed: {error:#}"), + None, + ), + } + } + Err(error) => failure( + error.public_message(), + error.message.clone(), + error.retry_after, + ), + }; + match store + .record_fetch_failure(user_id, podcast_id, now_ms, &error, Some(next_check_at_ms)) + .await + { + Ok(_) => Ok(TargetResult::Failed { + error, + detail, + consecutive_failures, + next_check_at_ms, + }), + Err(error) => skipped_or(error), + } +} + +async fn ingest( + store: &PodcastStore, + target: &RefreshTarget, + document: &FetchedFeed, + now_ms: i64, +) -> Result { + let outcome = store + .apply_feed( + target.user_id, + &FeedRefresh { + podcast_id: &target.podcast_id, + feed: &document.feed, + fetched_from: &document.fetched_from, + moved_permanently: document.moved_permanently, + }, + ) + .await?; + store + .record_fetch_success( + target.user_id, + &target.podcast_id, + now_ms, + document.validators.etag.as_deref(), + document.validators.last_modified.as_deref(), + Some(next_regular_check_ms( + now_ms, + target.user_id, + &target.podcast_id, + )), + ) + .await?; + Ok(outcome) +} + +fn is_gone(error: &anyhow::Error) -> bool { + matches!( + error.downcast_ref::(), + Some(PodcastStoreError::UnknownPodcast(_)) + ) +} + +/// A podcast removed from the account meanwhile is skipped; any other +/// store error is returned. +fn skipped_or(error: anyhow::Error) -> Result { + if is_gone(&error) { + Ok(TargetResult::Skipped(error.to_string())) + } else { + Err(error) + } +} + +/// Why a manual refresh was not started. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) enum ManualRefreshError { + /// The account does not know the podcast. + UnknownPodcast, + /// The account is not subscribed (unsubscribing stops monitoring). + NotSubscribed, + /// Checked less than [`MANUAL_MIN_INTERVAL_MS`] ago. + TooSoon { retry_after_ms: i64 }, +} + +impl fmt::Display for ManualRefreshError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::UnknownPodcast => f.write_str("unknown podcast"), + Self::NotSubscribed => f.write_str("not subscribed to this podcast"), + Self::TooSoon { retry_after_ms } => write!( + f, + "the feed was checked moments ago; retry in {} s", + (retry_after_ms + 999) / 1000 + ), + } + } +} + +impl std::error::Error for ManualRefreshError {} + +/// Refreshes one subscribed podcast of one account now, ignoring the +/// failure backoff but not more often than every +/// [`MANUAL_MIN_INTERVAL_MS`]. Returns `Err` with a +/// [`ManualRefreshError`] when the refresh was not attempted. +pub(crate) async fn refresh_now( + store: &PodcastStore, + fetcher: &FeedFetcher, + user_id: i64, + podcast_id: &PodcastId, + now_ms: i64, +) -> Result { + let feed_url = match store.podcast(user_id, podcast_id).await? { + Some(summary) if summary.subscribed => summary.meta.feed_url, + Some(_) => return Err(ManualRefreshError::NotSubscribed.into()), + None => match store.subscription(user_id, podcast_id).await? { + Some(register) if register.subscribed => register.podcast.feed_url, + Some(_) => return Err(ManualRefreshError::NotSubscribed.into()), + None => return Err(ManualRefreshError::UnknownPodcast.into()), + }, + }; + if let Some(checked) = store.fetch_state(user_id, podcast_id).await?.checked_at_ms { + let elapsed = now_ms - checked; + if (0..MANUAL_MIN_INTERVAL_MS).contains(&elapsed) { + return Err(ManualRefreshError::TooSoon { + retry_after_ms: MANUAL_MIN_INTERVAL_MS - elapsed, + } + .into()); + } + } + let target = RefreshTarget { + user_id, + podcast_id: podcast_id.clone(), + }; + let mut results = refresh_group(store, fetcher, &feed_url, vec![target], now_ms).await?; + Ok(results.pop().map(|(_, result)| result).expect("one target")) +} + +/// Result of [`subscribe_url`]. +#[derive(Debug, Clone, PartialEq)] +pub(crate) struct UrlSubscription { + /// Podcast the URL resolved to (an existing one for a known URL or alias). + pub(crate) podcast_id: PodcastId, + /// The account was already subscribed; no subscription was written. + pub(crate) already_subscribed: bool, + /// Ingestion of the document fetched for the subscription. + pub(crate) refresh: RefreshOutcome, +} + +/// Why [`subscribe_url`] did not subscribe. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) enum SubscribeUrlError { + /// The feed could not be fetched or parsed; nothing was stored. + Fetch(FetchError), +} + +impl fmt::Display for SubscribeUrlError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::Fetch(error) => { + write!( + f, + "the feed could not be loaded: {}", + error.public_message() + ) + } + } + } +} + +impl std::error::Error for SubscribeUrlError {} + +/// Subscribes an account to the feed at `url` (the "add RSS" action). +/// +/// The feed is fetched first: an address that is not public, a document +/// that is not a feed or a failed request subscribes nothing +/// ([`SubscribeUrlError`] inside the error). Then the subscription is +/// written with the channel metadata of the document +/// ([`PodcastStore::subscribe`]: a known URL or alias reuses the existing +/// podcast, its progress and history; an active subscription writes +/// nothing) and the document is ingested like a scheduled refresh, which +/// also stores the validators and the next check. +pub(crate) async fn subscribe_url( + store: &PodcastStore, + fetcher: &FeedFetcher, + user_id: i64, + url: &FeedUrl, + now_ms: i64, +) -> Result { + let document = match fetcher.fetch(url, &Validators::default()).await { + Ok(FeedFetch::Fetched(document)) => document, + Ok(FeedFetch::NotModified) => { + // Unconditional requests are never answered with 304 by a + // conforming server; refuse instead of subscribing blindly. + return Err(SubscribeUrlError::Fetch(FetchError { + kind: super::fetch::FetchErrorKind::Status(304), + message: "unexpected 304 Not Modified".to_string(), + retry_after: None, + }) + .into()); + } + Err(error) => return Err(SubscribeUrlError::Fetch(error).into()), + }; + let subscription = store + .subscribe(user_id, &document.feed.to_podcast_meta(url.clone())) + .await?; + let target = RefreshTarget { + user_id, + podcast_id: subscription.podcast_id.clone(), + }; + let refresh = ingest(store, &target, &document, now_ms).await?; + Ok(UrlSubscription { + podcast_id: subscription.podcast_id, + already_subscribed: subscription.already_subscribed, + refresh, + }) +} diff --git a/src/podcasts/schema.rs b/src/podcasts/schema.rs new file mode 100644 index 0000000..9890041 --- /dev/null +++ b/src/podcasts/schema.rs @@ -0,0 +1,337 @@ +//! PostgreSQL schema of the podcast store and the import of the relay state. +//! +//! The migrations themselves are declared in `crate::music::db_migrations` +//! (`m_0056_create_podcast_store`, `m_0057_import_podcast_relay`), next to +//! the rest of the `furumusic` chain; this module holds their SQL so the +//! store and its tests share one definition. +//! +//! The tables mirror the SQLite reference store of `furumi-library` +//! (`PODCASTS.md`, "SQLite reference store") with a `user_id` column in +//! every key: each web account is one virtual device with its own podcast +//! catalog, registers and fetch state. + +/// Tables, the derived season view and indexes (`m_0056`). +pub(crate) const CREATE_PODCAST_STORE: &[&str] = &[ + // Local channel metadata. `feed_url` is the URL this account fetches; + // `meta_from_feed` is set once the account refreshed the feed itself + // (until then the winning subscription register provides the metadata). + "CREATE TABLE IF NOT EXISTS furumusic__podcast ( + user_id BIGINT NOT NULL, + podcast_id TEXT NOT NULL, + feed_url TEXT NOT NULL, + title TEXT NOT NULL, + author TEXT, + description TEXT, + link TEXT, + image_url TEXT, + meta_from_feed BOOLEAN NOT NULL DEFAULT FALSE, + created_at_ms BIGINT NOT NULL, + updated_at_ms BIGINT NOT NULL, + PRIMARY KEY (user_id, podcast_id) + )", + // Bounded grow-only alias set (`merge_feed_aliases`), synced inside + // subscription records and `podcast_feed_aliases_added`. + "CREATE TABLE IF NOT EXISTS furumusic__podcast_feed_alias ( + user_id BIGINT NOT NULL, + podcast_id TEXT NOT NULL, + feed_url TEXT NOT NULL, + PRIMARY KEY (user_id, podcast_id, feed_url) + )", + "CREATE INDEX IF NOT EXISTS idx_podcast_feed_alias_url + ON furumusic__podcast_feed_alias (user_id, feed_url)", + // Episode catalog (`EpisodeMeta`). `source` is `feed` once this account + // parsed the episode from RSS, `sync` while it is only known from a + // `PodcastEpisodeRef`. `search_norm` caches `normalize_name(title)` and + // `normalize_name(description)` separated by a newline (never produced + // by the normalization), NULL until computed. + "CREATE TABLE IF NOT EXISTS furumusic__podcast_episode ( + user_id BIGINT NOT NULL, + episode_id TEXT NOT NULL, + podcast_id TEXT NOT NULL, + guid TEXT, + title TEXT NOT NULL, + description TEXT, + season BIGINT, + episode_number BIGINT, + episode_type TEXT NOT NULL DEFAULT 'full', + published_at_ms BIGINT, + duration_ms BIGINT, + audio_url TEXT NOT NULL, + mime TEXT, + byte_len BIGINT, + image_url TEXT, + link TEXT, + source TEXT NOT NULL CHECK (source IN ('feed', 'sync')), + in_feed BOOLEAN NOT NULL DEFAULT FALSE, + search_norm TEXT, + first_seen_at_ms BIGINT NOT NULL, + updated_at_ms BIGINT NOT NULL, + PRIMARY KEY (user_id, episode_id), + FOREIGN KEY (user_id, podcast_id) + REFERENCES furumusic__podcast (user_id, podcast_id) ON DELETE CASCADE + )", + "CREATE INDEX IF NOT EXISTS idx_podcast_episode_date + ON furumusic__podcast_episode (user_id, podcast_id, published_at_ms DESC)", + "CREATE INDEX IF NOT EXISTS idx_podcast_episode_season + ON furumusic__podcast_episode (user_id, podcast_id, season, episode_number)", + // Seasons are derived from the episodes; a view cannot go stale. + // `season` NULL groups the episodes without `itunes:season`. + "CREATE OR REPLACE VIEW furumusic__podcast_season AS + SELECT user_id, podcast_id, season, + COUNT(*) AS episode_count, + MIN(published_at_ms) AS first_published_at_ms, + MAX(published_at_ms) AS last_published_at_ms + FROM furumusic__podcast_episode + GROUP BY user_id, podcast_id, season", + // Synced registers. No foreign keys to the catalog: operations for + // podcasts or episodes this account has not fetched are kept as they + // arrive, in any order. + "CREATE TABLE IF NOT EXISTS furumusic__podcast_subscription ( + user_id BIGINT NOT NULL, + podcast_id TEXT NOT NULL, + subscribed BOOLEAN NOT NULL, + podcast_json JSONB NOT NULL, + generation BIGINT NOT NULL CHECK (generation >= 0), + hlc_ms BIGINT NOT NULL, + op_id TEXT NOT NULL, + PRIMARY KEY (user_id, podcast_id) + )", + "CREATE TABLE IF NOT EXISTS furumusic__podcast_share ( + user_id BIGINT NOT NULL, + podcast_id TEXT NOT NULL, + shared BOOLEAN NOT NULL, + generation BIGINT NOT NULL CHECK (generation >= 0), + hlc_ms BIGINT NOT NULL, + op_id TEXT NOT NULL, + PRIMARY KEY (user_id, podcast_id) + )", + "CREATE TABLE IF NOT EXISTS furumusic__podcast_progress ( + user_id BIGINT NOT NULL, + episode_id TEXT NOT NULL, + podcast_id TEXT NOT NULL, + position_ms BIGINT NOT NULL, + status TEXT NOT NULL + CHECK (status IN ('not_started', 'in_progress', 'completed')), + generation BIGINT NOT NULL CHECK (generation >= 0), + session_id TEXT NOT NULL, + hlc_ms BIGINT NOT NULL, + device_id TEXT NOT NULL, + op_id TEXT NOT NULL, + PRIMARY KEY (user_id, episode_id) + )", + "CREATE INDEX IF NOT EXISTS idx_podcast_progress_podcast + ON furumusic__podcast_progress (user_id, podcast_id)", + // Grow-only set of listening sessions. + "CREATE TABLE IF NOT EXISTS furumusic__podcast_listen_session ( + user_id BIGINT NOT NULL, + session_id TEXT NOT NULL, + episode_id TEXT NOT NULL, + podcast_id TEXT NOT NULL, + device_id TEXT NOT NULL, + started_at_ms BIGINT NOT NULL, + ended_at_ms BIGINT NOT NULL, + start_position_ms BIGINT NOT NULL, + end_position_ms BIGINT NOT NULL, + listened_ms BIGINT NOT NULL, + completed BOOLEAN NOT NULL, + PRIMARY KEY (user_id, session_id) + )", + "CREATE INDEX IF NOT EXISTS idx_podcast_listen_session_episode + ON furumusic__podcast_listen_session (user_id, episode_id)", + "CREATE INDEX IF NOT EXISTS idx_podcast_listen_session_end + ON furumusic__podcast_listen_session (user_id, ended_at_ms DESC)", + // Local-only fetch state: never part of sync output. + "CREATE TABLE IF NOT EXISTS furumusic__podcast_fetch_state ( + user_id BIGINT NOT NULL, + podcast_id TEXT NOT NULL, + etag TEXT, + last_modified TEXT, + checked_at_ms BIGINT, + last_success_at_ms BIGINT, + next_check_at_ms BIGINT, + last_error TEXT, + consecutive_failures BIGINT NOT NULL DEFAULT 0, + cover_url TEXT, + cover_path TEXT, + PRIMARY KEY (user_id, podcast_id), + FOREIGN KEY (user_id, podcast_id) + REFERENCES furumusic__podcast (user_id, podcast_id) ON DELETE CASCADE + )", + "CREATE INDEX IF NOT EXISTS idx_podcast_fetch_state_due + ON furumusic__podcast_fetch_state (next_check_at_ms)", +]; + +/// Backwards operation of `m_0056`. +pub(crate) const DROP_PODCAST_STORE: &[&str] = &[ + "DROP VIEW IF EXISTS furumusic__podcast_season", + "DROP TABLE IF EXISTS furumusic__podcast_fetch_state", + "DROP TABLE IF EXISTS furumusic__podcast_listen_session", + "DROP TABLE IF EXISTS furumusic__podcast_progress", + "DROP TABLE IF EXISTS furumusic__podcast_share", + "DROP TABLE IF EXISTS furumusic__podcast_subscription", + "DROP TABLE IF EXISTS furumusic__podcast_episode", + "DROP TABLE IF EXISTS furumusic__podcast_feed_alias", + "DROP TABLE IF EXISTS furumusic__podcast", +]; + +/// Placeholder in [`IMPORT_PODCAST_RELAY`] for the account filter: the +/// migration imports every account (`TRUE`), tests narrow it to one. +pub(crate) const USER_FILTER: &str = "{user_filter}"; + +/// Largest generation the `BIGINT` columns hold; larger records are skipped +/// like the store rejects them. +const GENERATION_FITS: &str = "(record_json->>'generation')::numeric <= 9223372036854775807"; + +/// Import of the opaque relay state kept by 0.13.0 (`m_0057`). +/// +/// `furumusic__fed_podcast_relay` holds the contract registers as JSON, +/// already validated and merged with the `music_dht::podcast` rules when they +/// were received, so they are copied as they are. The catalog rows the store +/// would have created from the episode references of received operations +/// (`PodcastEpisodeRef`, `source = 'sync'`) are rebuilt from the operation +/// log. Every statement is idempotent (`ON CONFLICT DO NOTHING`); the relay +/// table is kept (see `docs/administration.md`). +pub(crate) const IMPORT_PODCAST_RELAY: &[&str] = &[ + // Catalog rows from the subscription registers. + "INSERT INTO furumusic__podcast + (user_id, podcast_id, feed_url, title, author, description, link, image_url, + meta_from_feed, created_at_ms, updated_at_ms) + SELECT user_id, record_key, + record_json->'podcast'->>'feed_url', record_json->'podcast'->>'title', + record_json->'podcast'->>'author', record_json->'podcast'->>'description', + record_json->'podcast'->>'link', record_json->'podcast'->>'image_url', + FALSE, updated_at_ms, updated_at_ms + FROM furumusic__fed_podcast_relay + WHERE section = 'subscription' AND {user_filter} + ON CONFLICT DO NOTHING", + "INSERT INTO furumusic__podcast_subscription + (user_id, podcast_id, subscribed, podcast_json, generation, hlc_ms, op_id) + SELECT user_id, record_key, (record_json->>'subscribed')::boolean, + record_json->'podcast', (record_json->>'generation')::bigint, + (record_json->>'hlc_ms')::bigint, record_json->>'op_id' + FROM furumusic__fed_podcast_relay + WHERE section = 'subscription' AND {generation_fits} AND {user_filter} + ON CONFLICT DO NOTHING", + // The relayed subscription already carries the bounded, merged alias set. + "INSERT INTO furumusic__podcast_feed_alias (user_id, podcast_id, feed_url) + SELECT user_id, record_key, jsonb_array_elements_text(record_json->'feed_aliases') + FROM furumusic__fed_podcast_relay + WHERE section = 'subscription' AND {user_filter} + AND jsonb_typeof(record_json->'feed_aliases') = 'array' + ON CONFLICT DO NOTHING", + // Aliases received before any subscription register of their podcast. + "INSERT INTO furumusic__podcast_feed_alias (user_id, podcast_id, feed_url) + SELECT user_id, record_key, jsonb_array_elements_text(record_json) + FROM furumusic__fed_podcast_relay + WHERE section = 'aliases' AND {user_filter} AND jsonb_typeof(record_json) = 'array' + ON CONFLICT DO NOTHING", + "INSERT INTO furumusic__podcast_share + (user_id, podcast_id, shared, generation, hlc_ms, op_id) + SELECT user_id, record_key, (record_json->>'shared')::boolean, + (record_json->>'generation')::bigint, (record_json->>'hlc_ms')::bigint, + record_json->>'op_id' + FROM furumusic__fed_podcast_relay + WHERE section = 'share' AND {generation_fits} AND {user_filter} + ON CONFLICT DO NOTHING", + "INSERT INTO furumusic__podcast_progress + (user_id, episode_id, podcast_id, position_ms, status, generation, session_id, + hlc_ms, device_id, op_id) + SELECT user_id, record_key, record_json->>'podcast_id', + (record_json->>'position_ms')::bigint, record_json->>'status', + (record_json->>'generation')::bigint, record_json->>'session_id', + (record_json->>'hlc_ms')::bigint, record_json->>'device_id', + record_json->>'op_id' + FROM furumusic__fed_podcast_relay + WHERE section = 'progress' AND {generation_fits} AND {user_filter} + ON CONFLICT DO NOTHING", + "INSERT INTO furumusic__podcast_listen_session + (user_id, session_id, episode_id, podcast_id, device_id, started_at_ms, + ended_at_ms, start_position_ms, end_position_ms, listened_ms, completed) + SELECT user_id, record_key, record_json->>'episode_id', record_json->>'podcast_id', + record_json->>'device_id', (record_json->>'started_at_ms')::bigint, + (record_json->>'ended_at_ms')::bigint, + (record_json->>'start_position_ms')::bigint, + (record_json->>'end_position_ms')::bigint, + (record_json->>'listened_ms')::bigint, (record_json->>'completed')::boolean + FROM furumusic__fed_podcast_relay + WHERE section = 'session' AND {user_filter} + ON CONFLICT DO NOTHING", + // Catalog rows for podcasts only known from episode references. + "INSERT INTO furumusic__podcast + (user_id, podcast_id, feed_url, title, meta_from_feed, created_at_ms, updated_at_ms) + SELECT DISTINCT ON (user_id, payload_json->'episode'->>'podcast_id') + user_id, payload_json->'episode'->>'podcast_id', + payload_json->'episode'->>'feed_url', payload_json->'episode'->>'podcast_title', + FALSE, received_at_ms, received_at_ms + FROM furumusic__fed_sync_ops + WHERE kind IN ('podcast_progress_set', 'podcast_listen_recorded') + AND jsonb_typeof(payload_json->'episode') = 'object' AND {user_filter} + ORDER BY user_id, payload_json->'episode'->>'podcast_id', hlc_ms DESC, op_id DESC + ON CONFLICT DO NOTHING", + // Episodes only known from references (the newest reference wins, like + // the first one would have for the store: feed metadata replaces it on + // the first refresh anyway). + "INSERT INTO furumusic__podcast_episode + (user_id, episode_id, podcast_id, title, season, episode_number, episode_type, + published_at_ms, duration_ms, audio_url, mime, image_url, source, in_feed, + first_seen_at_ms, updated_at_ms) + SELECT DISTINCT ON (user_id, payload_json->'episode'->>'episode_id') + user_id, payload_json->'episode'->>'episode_id', + payload_json->'episode'->>'podcast_id', payload_json->'episode'->>'episode_title', + (payload_json->'episode'->>'season')::bigint, + (payload_json->'episode'->>'episode_number')::bigint, 'full', + (payload_json->'episode'->>'published_at_ms')::bigint, + (payload_json->'episode'->>'duration_ms')::bigint, + payload_json->'episode'->>'audio_url', payload_json->'episode'->>'mime', + payload_json->'episode'->>'image_url', 'sync', FALSE, + received_at_ms, received_at_ms + FROM furumusic__fed_sync_ops + WHERE kind IN ('podcast_progress_set', 'podcast_listen_recorded') + AND jsonb_typeof(payload_json->'episode') = 'object' AND {user_filter} + ORDER BY user_id, payload_json->'episode'->>'episode_id', hlc_ms DESC, op_id DESC + ON CONFLICT DO NOTHING", +]; + +/// The import statements with `filter` as the account condition. +pub(crate) fn import_statements(filter: &str) -> Vec { + IMPORT_PODCAST_RELAY + .iter() + .map(|sql| { + sql.replace("{generation_fits}", GENERATION_FITS) + .replace(USER_FILTER, filter) + }) + .collect() +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn import_statements_fill_every_placeholder() { + for sql in import_statements("user_id = 7") { + assert!(!sql.contains('{'), "{sql}"); + assert!(sql.contains("user_id = 7"), "{sql}"); + assert!(sql.contains("ON CONFLICT DO NOTHING"), "{sql}"); + } + } + + #[test] + fn every_store_table_is_dropped_backwards() { + for sql in CREATE_PODCAST_STORE { + let Some(rest) = sql + .strip_prefix("CREATE TABLE IF NOT EXISTS ") + .or_else(|| sql.strip_prefix("CREATE OR REPLACE VIEW ")) + else { + continue; + }; + let name = rest.split_whitespace().next().unwrap(); + assert!( + DROP_PODCAST_STORE + .iter() + .any(|drop| drop.ends_with(&format!(" {name}"))), + "{name}" + ); + } + } +} diff --git a/src/podcasts/test_server.rs b/src/podcasts/test_server.rs new file mode 100644 index 0000000..e4344da --- /dev/null +++ b/src/podcasts/test_server.rs @@ -0,0 +1,375 @@ +//! Test support of the podcast runtime: a scripted resolver, a client that +//! may reach loopback, and a minimal HTTP/1.1 server on 127.0.0.1 (no +//! internet). One request per connection (`Connection: close`), so every +//! request opens a new connection and resolves its host again. +//! +//! Adapted from the furumi-tui prototype's test server: routes can redirect, +//! send extra headers and answer conditional requests, and every request's +//! headers are recorded. + +use std::collections::{HashMap, VecDeque}; +use std::io::{BufRead, BufReader, Write}; +use std::net::{IpAddr, TcpListener, TcpStream}; +use std::sync::atomic::{AtomicUsize, Ordering}; +use std::sync::{Arc, Mutex, PoisonError}; +use std::time::Duration; + +use super::net::{AddressPolicy, HostLookup, LookupFuture, SafeHttp, SafeHttpConfig}; + +#[derive(Clone)] +pub(crate) struct Route { + pub(crate) status: u16, + pub(crate) body: Arc>, + pub(crate) content_type: Option, + pub(crate) etag: Option, + pub(crate) last_modified: Option, + /// Honour `Range` requests with 206 responses. + pub(crate) ranges: bool, + /// Send `Content-Length` (otherwise the body is close-delimited). + pub(crate) send_length: bool, + /// Close the connection once the body reaches this absolute offset. + pub(crate) cut_after: Option, + /// Extra response headers. + pub(crate) headers: Vec<(String, String)>, + /// Pause between the headers and the body. + pub(crate) body_delay: Option, +} + +impl Route { + pub(crate) fn new(body: impl Into>, content_type: &str) -> Route { + Route { + status: 200, + body: Arc::new(body.into()), + content_type: Some(content_type.to_string()), + etag: None, + last_modified: None, + ranges: true, + send_length: true, + cut_after: None, + headers: Vec::new(), + body_delay: None, + } + } + + /// A redirect to `location` with `status`. + pub(crate) fn redirect(status: u16, location: &str) -> Route { + Route { + status, + body: Arc::new(Vec::new()), + content_type: None, + ranges: false, + ..Route::new(Vec::new(), "text/plain") + } + .header("Location", location) + } + + pub(crate) fn status(mut self, status: u16) -> Route { + self.status = status; + self + } + + pub(crate) fn etag(mut self, etag: &str) -> Route { + self.etag = Some(etag.to_string()); + self + } + + pub(crate) fn last_modified(mut self, date: &str) -> Route { + self.last_modified = Some(date.to_string()); + self + } + + pub(crate) fn no_ranges(mut self) -> Route { + self.ranges = false; + self + } + + pub(crate) fn no_length(mut self) -> Route { + self.send_length = false; + self + } + + pub(crate) fn no_content_type(mut self) -> Route { + self.content_type = None; + self + } + + pub(crate) fn cut(mut self, after: usize) -> Route { + self.cut_after = Some(after); + self + } + + pub(crate) fn delay_body(mut self, delay: Duration) -> Route { + self.body_delay = Some(delay); + self + } + + pub(crate) fn header(mut self, name: &str, value: &str) -> Route { + self.headers.push((name.to_string(), value.to_string())); + self + } +} + +/// Lowercased request headers of one request. +pub(crate) type RequestHeaders = HashMap; + +#[derive(Default)] +struct ServerState { + routes: HashMap, + requests: HashMap>, +} + +pub(crate) struct TestServer { + port: u16, + state: Arc>, +} + +impl TestServer { + pub(crate) fn start() -> TestServer { + let listener = TcpListener::bind("127.0.0.1:0").expect("bind test server"); + let port = listener.local_addr().expect("local addr").port(); + let state = Arc::new(Mutex::new(ServerState::default())); + let accept_state = Arc::clone(&state); + std::thread::spawn(move || { + for stream in listener.incoming() { + let Ok(stream) = stream else { continue }; + let state = Arc::clone(&accept_state); + std::thread::spawn(move || handle(stream, &state)); + } + }); + TestServer { port, state } + } + + /// `http://127.0.0.1:`. + pub(crate) fn url(&self, path: &str) -> String { + format!("http://127.0.0.1:{}{path}", self.port) + } + + /// `http://:` (for hosts mapped by a test resolver). + pub(crate) fn url_on(&self, host: &str, path: &str) -> String { + format!("http://{host}:{}{path}", self.port) + } + + pub(crate) fn set(&self, path: &str, route: Route) { + self.lock().routes.insert(path.to_string(), route); + } + + /// Headers of every request to `path`, in order. + pub(crate) fn requests(&self, path: &str) -> Vec { + self.lock().requests.get(path).cloned().unwrap_or_default() + } + + /// Requests to any path. + pub(crate) fn request_count(&self) -> usize { + self.lock().requests.values().map(Vec::len).sum() + } + + fn lock(&self) -> std::sync::MutexGuard<'_, ServerState> { + self.state.lock().unwrap_or_else(PoisonError::into_inner) + } +} + +fn handle(stream: TcpStream, state: &Mutex) { + let _ = stream.set_read_timeout(Some(Duration::from_secs(10))); + let mut reader = BufReader::new(match stream.try_clone() { + Ok(stream) => stream, + Err(_) => return, + }); + let mut request_line = String::new(); + if reader.read_line(&mut request_line).is_err() { + return; + } + let path = request_line + .split_whitespace() + .nth(1) + .unwrap_or("/") + .to_string(); + let mut headers = HashMap::new(); + loop { + let mut line = String::new(); + match reader.read_line(&mut line) { + Ok(0) | Err(_) => return, + Ok(_) => {} + } + let line = line.trim_end(); + if line.is_empty() { + break; + } + if let Some((name, value)) = line.split_once(':') { + headers.insert(name.trim().to_ascii_lowercase(), value.trim().to_string()); + } + } + let route = { + let mut state = state.lock().unwrap_or_else(PoisonError::into_inner); + state + .requests + .entry(path.clone()) + .or_default() + .push(headers.clone()); + state.routes.get(&path).cloned() + }; + let mut out = stream; + let Some(route) = route else { + let _ = out + .write_all(b"HTTP/1.1 404 Not Found\r\nContent-Length: 0\r\nConnection: close\r\n\r\n"); + return; + }; + let not_modified = match (&route.etag, headers.get("if-none-match")) { + (Some(etag), Some(sent)) => etag == sent, + _ => matches!( + (&route.last_modified, headers.get("if-modified-since")), + (Some(date), Some(sent)) if date == sent + ), + }; + if route.status == 200 && not_modified { + let _ = write!( + out, + "HTTP/1.1 304 Not Modified\r\nConnection: close\r\n\r\n" + ); + return; + } + let total = route.body.len(); + let mut status = route.status; + let mut start = 0usize; + let mut end = total; + let mut extra = String::new(); + if route.status == 200 + && route.ranges + && let Some(range) = headers.get("range").and_then(|r| r.strip_prefix("bytes=")) + && let Some((from, to)) = range.split_once('-') + { + let (from, to) = match (from.trim().parse::(), to.trim().parse::()) { + (Ok(from), Ok(to)) => (from, (to + 1).min(total)), + (Ok(from), Err(_)) => (from, total), + (Err(_), Ok(suffix)) => (total.saturating_sub(suffix), total), + _ => (0, total), + }; + if from >= total { + let _ = write!( + out, + "HTTP/1.1 416 Range Not Satisfiable\r\nContent-Range: bytes */{total}\r\nContent-Length: 0\r\nConnection: close\r\n\r\n" + ); + return; + } + status = 206; + start = from; + end = to; + extra.push_str(&format!( + "Content-Range: bytes {start}-{}/{total}\r\n", + end - 1 + )); + } + if route.ranges { + extra.push_str("Accept-Ranges: bytes\r\n"); + } + if let Some(content_type) = &route.content_type { + extra.push_str(&format!("Content-Type: {content_type}\r\n")); + } + if let Some(etag) = &route.etag { + extra.push_str(&format!("ETag: {etag}\r\n")); + } + if let Some(date) = &route.last_modified { + extra.push_str(&format!("Last-Modified: {date}\r\n")); + } + for (name, value) in &route.headers { + extra.push_str(&format!("{name}: {value}\r\n")); + } + if route.send_length { + extra.push_str(&format!("Content-Length: {}\r\n", end - start)); + } + if write!( + out, + "HTTP/1.1 {status} Status\r\n{extra}Connection: close\r\n\r\n" + ) + .is_err() + { + return; + } + let _ = out.flush(); + if let Some(delay) = route.body_delay { + std::thread::sleep(delay); + } + let limit = route.cut_after.map_or(end, |cut| cut.clamp(start, end)); + for chunk in route.body[start..limit].chunks(64 * 1024) { + if out.write_all(chunk).is_err() { + return; + } + } + let _ = out.flush(); + let _ = out.shutdown(std::net::Shutdown::Both); +} + +/// Deterministic pseudo-random bytes. +pub(crate) fn noise(len: usize, seed: u64) -> Vec { + let mut state = seed | 1; + let mut out = Vec::with_capacity(len); + while out.len() < len { + state ^= state << 13; + state ^= state >> 7; + state ^= state << 17; + out.extend_from_slice(&state.to_le_bytes()); + } + out.truncate(len); + out +} + +// --------------------------------------------------------------------------- +// Resolver and client +// --------------------------------------------------------------------------- + +/// Test resolver: each host answers from a script; the last answer repeats. +#[derive(Default)] +pub(crate) struct ScriptedLookup { + answers: std::sync::Mutex>>>, + pub(crate) calls: AtomicUsize, +} + +impl ScriptedLookup { + pub(crate) fn with(entries: &[(&str, &[&[&str]])]) -> Arc { + let lookup = Self::default(); + { + let mut answers = lookup.answers.lock().unwrap(); + for (host, script) in entries { + answers.insert( + host.to_string(), + script + .iter() + .map(|answer| { + answer + .iter() + .map(|a| a.parse::().unwrap()) + .collect() + }) + .collect(), + ); + } + } + Arc::new(lookup) + } +} + +impl HostLookup for ScriptedLookup { + fn lookup(&self, host: &str) -> LookupFuture { + self.calls.fetch_add(1, Ordering::SeqCst); + let mut answers = self.answers.lock().unwrap(); + let answer = match answers.get_mut(host) { + Some(script) if script.len() > 1 => Ok(script.pop_front().unwrap()), + Some(script) => Ok(script.front().cloned().unwrap_or_default()), + None => Err(std::io::Error::new( + std::io::ErrorKind::NotFound, + format!("unknown test host {host}"), + )), + }; + Box::pin(async move { answer }) + } +} + +pub(crate) const TEST_HTTP: SafeHttpConfig = SafeHttpConfig { + connect_timeout: Duration::from_secs(5), + read_timeout: Duration::from_secs(5), + total_timeout: Some(Duration::from_secs(20)), + max_redirects: 3, +}; + +pub(crate) fn test_http(lookup: Arc) -> SafeHttp { + SafeHttp::with_lookup(TEST_HTTP, AddressPolicy::test_allow_loopback(), lookup).unwrap() +} diff --git a/src/podcasts/tests.rs b/src/podcasts/tests.rs new file mode 100644 index 0000000..db90aa0 --- /dev/null +++ b/src/podcasts/tests.rs @@ -0,0 +1,81 @@ +//! Pure unit tests of the podcast store helpers (no database). + +use music_dht::podcast::ProgressStatus; + +use super::*; + +#[test] +fn search_text_never_matches_across_title_and_description() { + let norm = episode_search_norm("Hello, World!", Some("WORLD peace")); + assert_eq!(norm, "hello world\nworld peace"); + // A needle is itself normalized, so it never contains the separator. + let needle = music_dht::normalize_name("world world"); + assert!(!norm.contains(&needle)); + assert!(norm.contains(&music_dht::normalize_name("WORLD PEACE"))); + assert_eq!(episode_search_norm("Only", None), "only\n"); +} + +#[test] +fn progress_status_round_trips_through_sql() { + for status in [ + ProgressStatus::NotStarted, + ProgressStatus::InProgress, + ProgressStatus::Completed, + ] { + assert_eq!(status_from_sql(status_to_sql(status)), status); + } + // The CHECK constraint admits exactly these values. + for sql in schema::CREATE_PODCAST_STORE { + if sql.contains("CREATE TABLE IF NOT EXISTS furumusic__podcast_progress (") { + for value in ["not_started", "in_progress", "completed"] { + assert!(sql.contains(&format!("'{value}'"))); + } + } + } +} + +#[test] +fn generations_beyond_bigint_are_rejected() { + assert_eq!(generation_to_sql(0).unwrap(), 0); + assert_eq!(generation_to_sql(i64::MAX as u64).unwrap(), i64::MAX); + assert!(matches!( + generation_to_sql(i64::MAX as u64 + 1), + Err(PodcastStoreError::Invalid(_)) + )); + assert_eq!(generation_from_sql(-1), 0); +} + +#[test] +fn contract_numbers_outside_u32_are_dropped() { + assert_eq!(u32_from_sql(Some(7)), Some(7)); + assert_eq!(u32_from_sql(Some(-1)), None); + assert_eq!(u32_from_sql(Some(i64::from(u32::MAX) + 1)), None); + assert_eq!(u32_from_sql(None), None); +} + +#[test] +fn progress_columns_are_prefixed() { + assert_eq!( + progress_columns("g", "g_"), + "g.episode_id AS g_episode_id, g.podcast_id AS g_podcast_id, \ + g.position_ms AS g_position_ms, g.status AS g_status, \ + g.generation AS g_generation, g.session_id AS g_session_id, \ + g.hlc_ms AS g_hlc_ms, g.device_id AS g_device_id, g.op_id AS g_op_id" + ); +} + +#[test] +fn queries_sort_undated_and_unnumbered_episodes_last() { + for sort in [ + EpisodeSort::NewestFirst, + EpisodeSort::OldestFirst, + EpisodeSort::EpisodeNumber, + EpisodeSort::EpisodeNumberDesc, + ] { + let order = queries::order_by_for_tests(sort); + assert!( + order.starts_with("e.published_at_ms IS NULL") || order.starts_with("e.season IS NULL") + ); + assert!(order.ends_with("e.episode_id"), "{order}"); + } +} diff --git a/src/scheduler/mod.rs b/src/scheduler/mod.rs index 253c019..5294eef 100644 --- a/src/scheduler/mod.rs +++ b/src/scheduler/mod.rs @@ -1311,6 +1311,7 @@ async fn run_scheduled_job( && job_name != "lastfm_scrobble" && job_name != "archive_cleanup" && job_name != "artwork_backfill" + && job_name != "podcast_refresh" { tracing::warn!(job = job_name, "Skipping: agent_enabled=false"); return; diff --git a/templates/admin/responsive.html b/templates/admin/responsive.html index fe31253..109d03a 100644 --- a/templates/admin/responsive.html +++ b/templates/admin/responsive.html @@ -70,7 +70,7 @@ html { overscroll-behavior:none; } .task-form-body { overflow:visible; } .run-log-body { min-height:300px; overflow:visible; } .run-log-output { min-height:50dvh; max-height:70dvh; } - .settings-layout { grid-template-columns:minmax(0,1fr); grid-template-areas:"access" "oidc" "agent" "agentstatus" "similarity" "similaritystatus" "downloads" "federation" "lastfm" "developer" "actions"; } + .settings-layout { grid-template-columns:minmax(0,1fr); grid-template-areas:"access" "oidc" "agent" "agentstatus" "similarity" "similaritystatus" "downloads" "federation" "podcasts" "lastfm" "developer" "actions"; } .settings-federation-body { grid-template-columns:minmax(0,1fr); } .repair-shell { grid-template-columns:250px minmax(0,1fr); } } diff --git a/templates/admin/styles.html b/templates/admin/styles.html index fd63b58..87e96f6 100644 --- a/templates/admin/styles.html +++ b/templates/admin/styles.html @@ -859,6 +859,7 @@ tbody tr:hover { "similarity similarity similarity similarity similarity similarity similarity similarity similaritystatus similaritystatus similaritystatus similaritystatus" "downloads downloads downloads downloads downloads downloads downloads downloads downloads downloads downloads downloads" "federation federation federation federation federation federation federation federation federation federation federation federation" + "podcasts podcasts podcasts podcasts podcasts podcasts podcasts podcasts podcasts podcasts podcasts podcasts" "lastfm lastfm lastfm lastfm lastfm lastfm lastfm lastfm developer developer developer developer" "actions actions actions actions actions actions actions actions actions actions actions actions"; gap: 16px; @@ -881,6 +882,7 @@ tbody tr:hover { .settings-similarity-status { grid-area: similaritystatus; } .settings-downloads { grid-area: downloads; } .settings-federation { grid-area: federation; } +.settings-podcasts { grid-area: podcasts; } .settings-lastfm { grid-area: lastfm; } .settings-developer { grid-area: developer; } .settings-section-narrow { border-left: 2px solid rgba(29, 185, 84, 0.55); } diff --git a/templates/admin/v2.html b/templates/admin/v2.html index f98e866..025da73 100644 --- a/templates/admin/v2.html +++ b/templates/admin/v2.html @@ -1167,6 +1167,27 @@ +
+
+
+ Podcasts + Episode audio comes from the publisher +
+
+
+
+ +
+ + +
+
By default the browser loads episode audio directly from the publisher. Enable the proxy for publishers the browser cannot play directly (missing CORS, plain HTTP on an HTTPS site). The server then fetches the audio from the publisher on demand and passes it through without storing it; it only connects to public addresses.
+
+
+
@@ -2179,7 +2200,8 @@ function adminV2() { download_proxies: [], torrent_proxy_id: '', youtube_proxy_id: '', - youtube_cookie_id: '' + youtube_cookie_id: '', + podcast_audio_proxy_enabled: false }, settingsProbe: { status: 'idle', ok: false }, settingsProbeLoading: false, diff --git a/templates/player/scripts.html b/templates/player/scripts.html index c754b49..46db3b1 100644 --- a/templates/player/scripts.html +++ b/templates/player/scripts.html @@ -64,6 +64,29 @@ const T = { lastfmDisconnectFailed: "{{ t.player_lastfm_disconnect_failed }}", download: "{{ t.track_download }}", shareUnavailable: "{{ t.share_unavailable }}", + queuePodcastItem: "{{ t.player_queue_podcast_item }}", + podcastAdded: "{{ t.player_podcast_added }}", + podcastAlready: "{{ t.player_podcast_already }}", + podcastRefreshUpdated: "{{ t.player_podcast_refresh_updated }}", + podcastRefreshNotModified: "{{ t.player_podcast_refresh_not_modified }}", + podcastRefreshFailed: "{{ t.player_podcast_refresh_failed }}", + podcastLastChecked: "{{ t.player_podcast_last_checked }}", + podcastNeverChecked: "{{ t.player_podcast_never_checked }}", + podcastFetchError: "{{ t.player_podcast_fetch_error }}", + podcastNextCheck: "{{ t.player_podcast_next_check }}", + podcastUnsubscribeConfirm: "{{ t.player_podcast_unsubscribe_confirm }}", + podcastSeason: "{{ t.player_podcast_season }}", + podcastNoSeason: "{{ t.player_podcast_no_season }}", + podcastLeft: "{{ t.player_podcast_left }}", + podcastSavedAt: "{{ t.player_podcast_saved_at }}", + podcastJamBlocked: "{{ t.player_podcast_jam_blocked }}", + podcastSourceFailed: "{{ t.player_podcast_source_failed }}", + podcastCannotSeek: "{{ t.player_podcast_cannot_seek }}", + podcastRequestFailed: "{{ t.player_podcast_request_failed }}", + podcastEpisodes: "{{ t.player_podcast_episodes }}", + podcastUnplayed: "{{ t.player_podcast_unplayed }}", + podcastSeasons: "{{ t.player_podcast_seasons }}", + queuePrivateItem: "{{ t.player_queue_private_item }}", searchSimilarTo: "{{ t.player_search_similar_to }}", findSimilar: "{{ t.player_find_similar }}", similarityFailed: "{{ t.player_similarity_failed }}", @@ -755,6 +778,19 @@ document.addEventListener('alpine:init', () => { _listenedSeconds: 0, _lastAudioTime: 0, _localSourceTrackId: null, + // The current item is one this player cannot play and nobody told it + // that item's position (see _showUnplayableQueueItem), or a podcast + // episode whose resume position the server has not sent yet. + _positionUnknown: false, + // Podcast episode playback (see _playEpisodeLocal): what the UI + // shows about the current episode and its listening session. + episodeNotice: '', + episodeLoading: false, + _episode: null, + _episodeToken: 0, + _episodeSeekTimer: null, + _episodeCheckpointMs: 20000, + _episodeRewindSeconds: 15, _remoteExecuting: false, _remoteStateBaseTime: 0, _remoteStateReceivedAt: 0, @@ -772,7 +808,11 @@ document.addEventListener('alpine:init', () => { }); audio.addEventListener('timeupdate', () => { + // Unloading the audio for an unplayable item fires a late + // timeupdate at 0; keep that item's handed-off position. + if (!this._currentItemHasLocalAudio()) return; this.currentTime = audio.currentTime; + if (this._episode) this._episode.lastPosition = audio.currentTime; this.duration = audio.duration || 0; this.progress = this.duration > 0 ? (this.currentTime / this.duration) * 100 : 0; this._trackListenedDelta(); @@ -782,6 +822,7 @@ document.addEventListener('alpine:init', () => { audio.addEventListener('ended', () => { this._trackListenedDelta(); this._recordHistory('finished'); + if (this._episode?.ready) this._reportEpisode('ended'); if (Alpine.store('devices')?.shouldPlayJamLocally()) { this.isPlaying = false; return; @@ -799,7 +840,9 @@ document.addEventListener('alpine:init', () => { this._trackListenedDelta(); this.isPlaying = false; this._lastAudioTime = audio.currentTime || 0; + this._onEpisodePaused(); }); + audio.addEventListener('error', () => this._onEpisodeAudioError()); audio.addEventListener('loadedmetadata', () => { this.duration = audio.duration || 0; @@ -812,6 +855,13 @@ document.addEventListener('alpine:init', () => { this._remoteStateTimer = setInterval(() => { this._tickRemoteProgress(); }, 250); + // Episode checkpoints (the interval itself is _episodeCheckpointMs). + setInterval(() => this._episodeTick(), 5000); + document.addEventListener('visibilitychange', () => { + if (document.hidden && this.isPlaying && this._isLocalPlaybackDevice()) { + this._reportEpisode('checkpoint', { keepalive: true }); + } + }); // Restore state if (!this._hasInitialShareLink()) { @@ -823,6 +873,7 @@ document.addEventListener('alpine:init', () => { if (this._isLocalPlaybackDevice()) { this._saveStateSync(); this._recordHistory('stopped', true); + this._reportEpisode('stop', { beacon: true }); } }); window.addEventListener('pageshow', (event) => { @@ -857,6 +908,7 @@ document.addEventListener('alpine:init', () => { playQueueIndex(idx) { const queue = Alpine.store('queue'); if (!queue || idx < 0 || idx >= queue.tracks.length) return; + const previousIndex = queue.currentIndex; queue.currentIndex = idx; const track = queue.tracks[idx]; if (track?.federation_pending && track?._federationTrack) { @@ -880,6 +932,24 @@ document.addEventListener('alpine:init', () => { })); return; } + if (this._isUnplayableQueueItem(track)) { + // Podcast episodes stay in the queue; this player skips them. + const step = idx >= previousIndex ? 1 : -1; + let playable = this._playableQueueIndex(queue.tracks, idx, step); + if (playable < 0 && (this.repeatMode === 'all' || this.shuffle)) { + // Wrap around like the queue itself does. + playable = this._playableQueueIndex(queue.tracks, step > 0 ? 0 : queue.tracks.length - 1, step); + } + if (playable < 0) { + // Staying on the current item keeps its position; any + // other item's position is unknown here, never 0. + this._showUnplayableQueueItem(track, track === this.currentTrack ? this._knownPosition() : null); + return; + } + queue.currentIndex = playable; + this._playLocal(queue.tracks[playable]); + return; + } this._playLocal(track); }, @@ -900,7 +970,20 @@ document.addEventListener('alpine:init', () => { }, _playLocal(track, options = {}) { + if (this._isUnplayableQueueItem(track)) { + this._recordHistory('replaced'); + this._showUnplayableQueueItem(track, options.position_seconds); + return; + } + if (this._isPodcastEpisode(track)) { + // Without a position the episode resumes from its progress. + this._recordHistory('replaced'); + this._playEpisodeLocal(track, options); + return; + } this._recordHistory('replaced'); + this._endEpisode('stop'); + this._positionUnknown = false; this.currentTrack = track; this._historyTrack = track; this._prefetchedQueueKey = null; @@ -943,6 +1026,7 @@ document.addEventListener('alpine:init', () => { this._sendRemote('pause'); return; } + if (this._episode) this._episode.wantPlay = false; this._pauseLocal(); }, @@ -957,9 +1041,35 @@ document.addEventListener('alpine:init', () => { this._sendRemote('resume'); return; } + if (this._isUnplayableQueueItem(this.currentTrack)) { + // Not next(): with repeat "one" it would resume this item again. + this._skipUnplayableCurrentItem(); + return; + } + if (this._isPodcastEpisode(this.currentTrack)) { + this._resumeEpisode(); + return; + } audio.play().catch(() => {}); }, + _skipUnplayableCurrentItem() { + const queue = Alpine.store('queue'); + const tracks = queue?.tracks || []; + let idx = this._playableQueueIndex(tracks, Number(queue?.currentIndex ?? -1) + 1, 1); + if (idx < 0 && this.repeatMode === 'all') idx = this._playableQueueIndex(tracks, 0, 1); + if (idx < 0) { + this._showUnplayableQueueItem(this.currentTrack, this._knownPosition()); + return; + } + this.playQueueIndex(idx); + }, + + _knownPosition() { + // The current item's position, or null when this player was never told it. + return this._positionUnknown ? null : this.currentTime; + }, + toggle() { if (!this.currentTrack) return; if (this.isPlaying) { this.pause(); } @@ -980,6 +1090,15 @@ document.addEventListener('alpine:init', () => { this._sendRemote('seek', { time: nextTime }); return; } + if (this._isUnplayableQueueItem(this.currentTrack)) { + // Nothing is loaded in
+
@@ -703,6 +715,262 @@
+ + + + + + +