diff --git a/CHANGELOG.md b/CHANGELOG.md index c02cfba..12fcfd8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,136 @@ # Changelog +## v0.15.0 — 2026-11-19 + +- Find podcasts in the federated search: cards shared by federation members + appear in the search results as **Podcasts**, typed apart from music + (`federation.podcast` events, `"type": "podcast"`; music events are + unchanged and now carry `"type": "music"`), with **Already subscribed** + when you follow the podcast, also under another feed address. A card + opens with its metadata and the seasons and episodes read from the feed, + browsable by season (`POST /api/podcasts/discovery/preview`), and + **Subscribe** + (`POST /api/podcasts/discovery/subscribe`) creates your own private + subscription from the card's original feed address with the card's + podcast id, or your existing podcast when you already follow the feed + under the address it moved to (also for **Add RSS**); the publishing + member is never contacted. A failed podcast search keeps the music + results and says so. Card texts and + links are untrusted: shown as text, links and artwork only as checked + `http(s)` (artwork `https`) addresses. +- Share a podcast in the federation: a **Share in the federation** toggle on + the podcast page publishes a public card of the podcast (title, author, + description, artwork, feed URL, episode count; never progress, history, + devices or the account) through the server's federation node on the + separate `podcast_dht` overlay of `music-dht` 0.6. Sharing is the + account's synchronized decision; turning it off or unsubscribing on any + device revokes it, and a late state of another device cannot publish it + again. After a restart the server publishes only after its first + device-sync round, so sharing turned off on a TUI while the server was + stopped is not announced again. A feed URL that looks private (token or key in the address, a + member-feed host) needs a confirmation of the exact address first; the + confirmation is local to the server (`m_0058`, + `furumusic__podcast_share_confirmation`) and asked again for a new share + decision or a moved feed. Accounts sharing the same podcast share one + card, published while any of them shares it; at most 192 podcasts per + server, shared fairly between the sharing accounts (an account within + its part never loses a published card to other accounts). The podcast + page says whether this server published the card, waits for a + confirmation, holds it back at the limit, or cannot publish because + federation is off or discovery is unavailable; the list marks + **Sharing on**. New endpoint `POST /api/podcasts/feeds/{id}/share` + (`{"shared": bool, "confirm"?: {...}}`, 409 when a confirmation is + needed); `GET /api/podcasts/feeds/{id}` reports the share state. +- The federation node serves the podcast-discovery overlay + (`furumi-fd/podcast-dht/1`, cache `podcast-dht.sqlite3` in the federation + data directory), advertises `podcast_dht` in its capability manifest only + while it runs, and probes peers' manifests so the overlay skips peers + without it. If the overlay cannot start, only discovery is disabled. + +## v0.14.0 — 2026-11-19 + +- Podcasts in the web player: a Podcasts section to add RSS feeds, browse + podcasts with unplayed counts and fetch errors, seasons, episode search, + filters and paging, refresh, unsubscribe (progress and history are kept) + and subscribe again, with Resume, Start over, Mark played/unplayed and the + recent listening history. +- The web player plays podcast episodes from the publisher (or the optional + proxy), resumes them from the synchronized position and saves progress in + the background: every 20 s and on pause, seek, item change, page hide and + handoff, with listening sessions and automatic completion by the shared + rule. A source that cannot seek to the saved position is shown as such + instead of playing from elsewhere. Handoffs to and from other devices keep + the episode, position and pause state; outgoing states carry the newest + progress generation. Play (also after a reload or a pause) continues from + the newest saved position of any device; a player that keeps listening + after another device wrote the episode saves again as a new session, and + one that loses the playback role records what it listened. Previous goes + back 15 s within an episode instead of resetting it. +- New endpoints under `/api/podcasts` (`/feeds`, `/feeds/{id}`, + `/feeds/{id}/episodes`, subscribe/unsubscribe, `/episodes/{id}/playback`, + `/progress`, `/restart`, `/played`, `/unplayed`). Request bodies must be + `application/json`. +- Jam guests still see the host's episodes only as private items and cannot + add episodes from the Podcasts section. +- Store podcast subscriptions, feed aliases, episodes, seasons, progress and + listening sessions of each account in PostgreSQL (`src/podcasts/`), with + the shared `music-dht` podcast rules. Received podcast operations and + snapshot sections are applied to this store, and snapshots are built from + it; the wire behavior towards other devices is unchanged. Migrations + `m_0056` and `m_0057` import the state that 0.13 relayed + (`furumusic__fed_podcast_relay`, kept but unused). +- Refresh subscribed podcast feeds in the background (`podcast_refresh` job, + every five minutes; each feed about hourly) with conditional requests, + permanent redirects learned as feed aliases, size and time limits and an + exponential backoff up to a day on errors. Accounts subscribed to the same + feed share one request; accounts take turns within a run of up to 200 + feeds. `POST /api/podcasts/{podcast_id}/refresh` refreshes one feed on + demand. Only metadata is fetched, never audio. An account can have up to + 1000 subscriptions and two user-started feed downloads at once. +- Optional podcast audio proxy (`podcast_audio_proxy_enabled`, off by + default; browsers stream from the publisher otherwise): + `GET /api/podcasts/episodes/{episode_id}/audio` streams an episode of the + signed-in account from its publisher with Range pass-through, audio only + (a generic binary type needs an audio file extension or audio content; the + feed's enclosure type is not trusted), size/time limits and four streams + per account. `GET /api/podcasts/episodes/{episode_id}/source` + tells the player which URLs to use. +- Feed and proxy requests only reach public addresses on ports 80, 443, 8080 + and 8443: names are resolved and checked once per connection (no DNS + rebinding), IP literals and every redirect hop are checked, and no cookies, + credentials or system proxies are used. Users see only the kind of a + failure; details go to the logs without URLs. A host name that resolves to + a blocked address gets the same answer as one that does not resolve. +- The browser can add at most 10 000 episodes outside the account's feeds + (handed-off queue items) through the playback endpoint (HTTP 422 beyond). + +## v0.13.0 — 2026-11-19 + +- Use the shared `music-dht` 0.6 device-sync types and announce device-sync + protocol 4. Podcast operations and snapshot sections from other devices are + kept and relayed; devices below protocol 4 never receive them. +- Keep podcast episodes and other non-library queue items through the web + player, with their explicit source. The player labels and skips them; the + typed queue is stored next to `queue_json` (migrations `m_0054`, `m_0055`). + A handoff through the web player keeps the position of the current episode. +- Refuse a playback transfer of a queue with podcast episodes to a device that + needs an update, and ignore its playback commands while the queue has them. +- Store the protocol version each device reports itself instead of this + server's version; invites carry the inviter's version. +- An episode that the web player selects without knowing its position (Next, + auto-advance, a click) resumes from its synchronized progress on the next + device instead of restarting at 0. Repeat all and shuffle continue past + episodes; Prev and seeking work on them. +- Jam guests see podcast episodes of the host as private items, without their + duration, and cannot add, remove or clear them; the host browser enforces + this on its own queue. Guests do not keep the placeholders after the Jam. +- Metadata Repair can still undo deletions recorded before the typed queue + column was added. +- Rolling back to 0.12 requires removing the relayed podcast operations first; + see "Rolling back to 0.12" in `docs/administration.md`. After a rollback, the + typed queue is used again only if 0.12 left the queue and its current track + unchanged. + ## v0.10.6 — 2026-09-02 - Recover federation automatically after sleep, prolonged idle, or a degraded rendezvous transport. diff --git a/Cargo.lock b/Cargo.lock index 68a93ca..a60060f 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1843,8 +1843,7 @@ dependencies = [ [[package]] name = "federation-net" version = "0.3.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "01e5c2e6e5af020dc202d4924125ef57719aa21f8d2567e2d943914386bc8980" +source = "git+https://gt.hexor.cy/Sanapach/frid.git?branch=podcasts%2Fdiscovery#9c82f6d91691a66d0a559caba06e13a1293f06be" dependencies = [ "blake3", "data-encoding", @@ -1988,7 +1987,7 @@ checksum = "e6d5a32815ae3f33302d95fdcb2ce17862f8c65363dcfd29360480ba1001fc9c" [[package]] name = "furumusic" -version = "0.12.4" +version = "0.15.0" dependencies = [ "anyhow", "async-stream", @@ -3900,16 +3899,17 @@ dependencies = [ [[package]] name = "music-dht" -version = "0.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4c3f75d49d3a742a6777a1c4cc53f397399dfc9ddb70034c57aeb44d75d2ee4f" +version = "0.6.0" +source = "git+https://gt.hexor.cy/Sanapach/frid.git?branch=podcasts%2Fdiscovery#9c82f6d91691a66d0a559caba06e13a1293f06be" dependencies = [ "async-trait", "blake3", "data-encoding", + "encoding_rs", "federation-net", "futures", "postcard", + "quick-xml 0.41.0", "rand 0.9.5", "rusqlite", "serde", @@ -3918,6 +3918,7 @@ dependencies = [ "tokio", "tracing", "unicode-normalization", + "url", ] [[package]] diff --git a/Cargo.toml b/Cargo.toml index bbc8624..d3a2f3e 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "furumusic" -version = "0.12.4" +version = "0.15.0" edition = "2024" description = "Reusable web-app boilerplate: auth, OIDC/SSO, admin panel, user management, i18n, PostgreSQL" @@ -56,4 +56,10 @@ uuid = "1" librqbit = { version = "8.1.1", features = ["disable-upload"] } # P2P federation: publishes the library into a shared DHT and serves audio / # catalogs to furumi peers (TUI clients) over the frid stack. -music-dht = "0.5.0" +# `podcast-feed`: the shared RSS/Atom parser, whose identity hints are part of +# the podcast episode id contract. +music-dht = { version = "0.6.0", features = ["podcast-feed"] } + +[patch.crates-io] +# Unpublished frid (MR-F1, MR-F3); remove once music-dht 0.6.0 is on crates.io. +music-dht = { git = "https://gt.hexor.cy/Sanapach/frid.git", branch = "podcasts/discovery" } diff --git a/README.md b/README.md index d5ecd5d..5bdd0ad 100644 --- a/README.md +++ b/README.md @@ -20,6 +20,12 @@ the Furumi federated network. without creating an account. - **Welcome your people.** Use password login or SSO. Instance admins can approve new users without needing access to the identity provider. +- **Follow podcasts.** Subscribe to RSS feeds, browse seasons and episodes, and + pick up where you stopped, also on your other Furumi players. Episodes stream + from the publisher and are never imported into the library; subscriptions + and listening progress stay private to your own devices. Share a podcast + in the federation on purpose, and find podcasts others shared in the + federated search. - **Discover more.** Find similar tracks and connect Last.fm for scrobbling. - **Connect your players.** Pair trusted Furumi devices to synchronize playlists, likes, playback state, and playback controls. diff --git a/docs/administration.md b/docs/administration.md index 0b13bc8..191ce48 100644 --- a/docs/administration.md +++ b/docs/administration.md @@ -71,6 +71,263 @@ Trusted-device pairing is separate from public catalog discovery. It connects a user's own Furumi players so likes, playlists, playback state, and control can move between approved devices. +Queues synchronized from other Furumi players can contain podcast episodes; +the web player plays them (see [Podcasts](#podcasts)) and keeps them in order +with their position. A device that does not support podcasts yet (device-sync +protocol below 4, for example an older terminal player) cannot receive such a +queue: selecting it as the playback device shows that it needs an update, and +the episodes stay in the queue. Jam guests see podcast episodes of the host +only as private items they cannot play, add, remove or clear. + +## Podcasts + +**Podcasts** in the player sidebar lists your subscriptions with the number +of unplayed episodes and the result of the last feed check. Paste an RSS or +Atom feed URL to subscribe: the server loads the feed first and shows why when +it cannot (not a feed, not reachable, or an address that is not public). A +podcast page shows the description, the seasons (episodes without a season +form their own group), a search over episode titles and descriptions, filters +for unplayed, started and played episodes, and the recent listening history. + +- **Play / Resume** continues an episode where you stopped, on whichever of + your Furumi players last saved it. **Start over** plays it from the + beginning; **Mark played** and **Mark unplayed** change its status without + playing (marking does not count as a listen). An episode counts as listened + when playback reaches its end, or its last 30 seconds or last 5 %, whichever + is shorter. +- The web player saves the position about every 20 seconds and when you + pause, seek, switch items, close the page or hand playback to another + device, without interrupting the audio. Reloading the page, or pressing + Play after a pause while another player went on with the episode, + continues from the newest saved position. Previous on an episode goes + back 15 seconds instead of to the start. Marking the episode that is + playing played or unplayed pauses it. +- Handing playback to another device keeps the episode, its position and + whether it was paused. A later save of an older listening session never + moves the position back after you seek or start over elsewhere. +- **Unsubscribe** stops checking the feed and keeps your progress and history; + subscribing again continues where you stopped. **Refresh** checks the feed + now (at most every 30 seconds). + +Privacy: a new subscription is private. Subscriptions, progress and listening +history are synchronized only between your own paired Furumi players (trusted +devices), never published to the federation, and Jam guests cannot see your +episodes. Only podcasts you share explicitly are published (see below). +Episodes are not stored on the server or shared with peers: the browser +streams the audio and loads the artwork directly from the publisher, so the +publisher sees your browser's address. With the audio proxy (below) the +publisher sees the server instead. + +### Sharing a podcast in the federation + +**Share in the federation** on a podcast page publishes a public card of the +podcast so that members of the federation network outside your own devices +can find it and subscribe. The card holds only the podcast's public +metadata: title, author, description, artwork and site links, the feed URL, +the number of episodes and the newest publication date. It never contains +your progress, history, listening counts, device names or your account. +**Stop sharing** or **Unsubscribe** revokes it; your progress is kept. + +- Sharing is your decision and is synchronized with your other Furumi + players, like the subscription. A podcast shared on another player is + shared here too, and turning it off anywhere turns it off everywhere. A + new subscription, including subscribing again, is always private. +- If the feed URL looks like it may contain private data (an access token or + key in the address, a host of personal subscriber feeds, a long random + part), the web player shows a confirmation dialog with the exact address + before publishing it. Such feeds are often personal: publishing one can + let others use your access. The confirmation belongs to this server and to + this one decision; it is asked again when you share again, when the feed + moves to another address, and when sharing was turned on on another + player. +- The **server** publishes the card with its federation identity (the same + node key that publishes its music library), not with an identity of your + account. When several accounts of this server share the same podcast, + the server publishes one card for it, built from one of these accounts' + copies of the feed, and keeps it while any of them still shares it. The + account whose copy the published card shows keeps providing it while it + shares (also after a server restart, recognised by the card's text), so + another account sharing the podcast later does not replace the card's + text; a new card preferably uses a copy the server downloaded itself + from the podcast's original address. If that account's copy changes + while the server restarts, the card may be rebuilt from another sharing + account's copy. Other members see that this server shares the + podcast, not who. Podcasts shared by a deactivated account are no longer + published (within five minutes). +- The podcast page says what this server does with the card: published, + publishing, waiting for your confirmation, held back by the server's + limit, or not possible because federation is off or public discovery is + unavailable. It shows only this server's publication: a Furumi player + such as the terminal player publishes its own card with its own identity + when you share there, so the card may be public through that player + while this server still waits for your confirmation. The podcast list + marks a podcast with **Sharing on** when your decision is to share it. +- Publishing needs federation to be enabled on the server (Settings → + Federation). Without it, the share decision is kept and the card is + published once federation runs. +- A server publishes at most 192 podcasts, shared fairly between the + accounts that share podcasts: every account gets its turn in rotation, + so an account whose shared podcasts fit in its part of the limit (192 + divided by the number of sharing accounts) never loses a published card + because other accounts share many podcasts. When the limit is reached, + only podcasts of accounts beyond their part wait (newest shares first; + cards already published are kept before new ones), and the podcast page + says that the card is held back. Members' caches keep at most 256 + records per server, including revocations of the last 24 hours, so a + new card also waits while recent revocations fill that room, and no + published card is given up for a card the federation would not store. +- A podcast whose stored data cannot be read (for example after a damaged + import) is treated as not shared, and its card is revoked; the other + podcasts are still published and revoked as usual. +- Revoking is eventually consistent: the server publishes a signed + revocation at once, but a federation peer that missed it may show the old + card for up to 12 hours. Subscriptions other members made from the card + stay theirs. + +The server joins the separate podcast-discovery overlay +(`furumi-fd/podcast-dht/1`) of the federation network and keeps its routing +contacts and the cards it holds for others in `podcast-dht.sqlite3` in the +federation data directory (a cache that can be deleted while the server is +stopped). If the overlay cannot start, the server log says "public podcast +discovery disabled" and everything else keeps working. The admin +federation status (`/admin/v2/api/federation`) reports `podcast_discovery` +and `podcast_discovery_peers`. + +### Finding podcasts in the federation + +The search box searches the federation too. Podcasts that members shared +appear under **Federation → Podcasts**, marked **Podcast** and kept apart +from music; podcasts you already follow are marked **Already subscribed** +(also when you follow the same feed under an older address it moved +from). If you follow the feed only under the newer address of a card's +moved feed, the result list cannot tell yet; opening the card reads the +feed, sees the move and shows **Already subscribed** with **Open**, and +neither **Subscribe** nor **Add RSS** with the old address creates a +second subscription. A podcast you follow is changed only by its own +address: when a feed found under another address only claims to have +moved to yours (or is an old copy of it), nothing of that feed is written +into your podcast; if you had unsubscribed, it is subscribed again with +its own data and refreshed from its own address. + +- Opening a card shows its title, author, description, artwork and the feed + URL, then loads the seasons and episodes from the feed itself through the + server: the newest 50 episodes, and the newest 50 of one season when you + pick its chip (pick it again for all seasons). Nothing is stored until + you subscribe. Closing the card, or picking another season, cancels a + loading preview. +- **Subscribe** adds your own private subscription: the server downloads the + feed from the card's address, as for **Add RSS**, and you can then play, + share or unsubscribe it like any other. The member who published the card + is never contacted and does not need to stay online; their card is only a + pointer to the public feed. The subscription keeps the podcast's identity, + so your other Furumi players recognise it, and nobody else sees it. +- Card texts and links come from other members and may be wrong. The + player shows them as plain text, opens only web links, and loads artwork + directly from its host only over HTTPS (like the artwork of subscribed + podcasts). Only the original feed address of a card is used: an address a + member claims the feed moved to is ignored unless the feed itself + redirects there. +- Search results show only cards; no member's subscriptions, progress or + account are ever visible. A card disappears from search results within + 12 hours after it is revoked. +- Podcast search needs federation on this server. When public discovery is + off or unavailable, the search shows music only. When the podcast search + fails or times out, the music results are still shown, with a note that + the podcast search did not answer. + +### Podcast feeds and audio + +The `podcast_refresh` job (every five minutes, listed under Jobs) checks the +feeds of subscribed podcasts: each feed about once an hour with conditional +requests, failing feeds less and less often (up to once a day). It downloads +feed metadata only, never episode audio. A failing feed keeps its known +episodes. Accounts subscribed to the same feed share one request. Each run +checks at most 200 feeds (2400 an hour), taking accounts in turns: first the +most overdue feed of every account, then the second, and so on. An account +with many or slow feeds therefore does not delay the others, but can use the +capacity nobody else needs. + +An account can add up to 1000 subscriptions and, from the browser, up to +10 000 episodes outside its feeds (handed-off queue items; episodes synced +from its own devices are not limited); beyond that the request is refused +with HTTP 422. It can have two feed downloads ("add RSS", Refresh) and four +proxied audio streams in progress at once (16 and 64 for the whole server); +beyond that the request is refused with HTTP 429. + +The server therefore makes outgoing HTTP(S) requests to podcast publishers. +It only connects to public addresses on the web ports 80, 443, 8080 and 8443: +feed or audio URLs that point to loopback, private, link-local, CGNAT or cloud +metadata addresses, directly, through DNS or through a redirect, or to any +other port, are refused. System proxy settings are not used for these +requests. Users only see the kind of failure (address not allowed, could not +connect, HTTP status, not a feed); the details, without the URL, are in the +server log and, for scheduled refreshes, in the job log with the podcast id. +"Address not allowed" is only said for what the URL itself shows (scheme, +credentials, port, an IP address). A host name that resolves to a blocked +address gets the same answer as a name that does not resolve or a server that +does not answer ("could not connect"), so users cannot find out which names +exist in the server's internal DNS. + +Episode audio is streamed by the browser directly from the publisher. Some +publishers cannot be played that way (plain HTTP while this server uses +HTTPS, or a server the browser refuses). For them, enable **Settings → +Podcasts → Podcast audio proxy** (`FURU_PODCAST_AUDIO_PROXY_ENABLED=true`): +the player then switches to the proxy when the direct URL fails or is plain +HTTP on an HTTPS page, and the server fetches the audio from the publisher +and passes it through without storing it. This uses the server's bandwidth. + +Security model of the proxy: + +- It takes an episode id, never a URL. The URL comes from the signed-in + account's own podcast catalog: its feeds, and episode references sent by + the account's own clients (its paired devices, or its browser when it plays + a handed-off episode the catalog does not know yet). A signed-in user + controls both (anyone can subscribe to a feed they host), so any signed-in + user can make the server fetch audio from any public URL, as with feed + refreshes. +- What limits that: the same public-address and port checks as for feeds + apply to every connection and redirect, and only audio is passed through. + The publisher's `Content-Type` must be audio; for a generic binary type + (`application/octet-stream` or none), the URL must have an audio file + extension or the file must start like an audio file (MP3, AAC, Ogg, FLAC, + WAV, M4A). The enclosure type written in the feed is never trusted. No + HTML, no video. Requests carry a single byte range, are limited to 1 GiB + and 4 hours each and to four streams per account (64 for the server), and + no cookies or credentials are sent. +- Responses are not cached or stored, and are marked `no-store`, `nosniff` + and sandboxed. + +Leave it off unless your users need it. + +### Upgrading from 0.13 + +Release 0.14 adds the podcast UI and playback described above; no setting is +needed besides the optional proxy. + +Podcast state synchronized from other Furumi players is kept in per-account +podcast tables (`furumusic__podcast*`). The upgrade imports what 0.13 kept in +`furumusic__fed_podcast_relay`; that table is no longer used and can be +dropped by a later release. Release 0.13 does not read the new tables: after +a rollback it would serve the relay table as it was at the upgrade, and +devices would send their newer podcast state again in their snapshots. + +### Rolling back to 0.12 + +Release 0.13 relays podcast synchronization operations from other Furumi +players. Release 0.12 cannot read them: device synchronization would fail for +every device that has not received them yet. Before you start a 0.12 release +on a database that 0.13 used, stop the server and remove those operations: + +```sql +DELETE FROM furumusic__fed_sync_ops WHERE kind LIKE 'podcast\_%'; +``` + +Devices that support podcasts keep their own podcast state. The relay table +`furumusic__fed_podcast_relay` and the typed queue column can stay: 0.12 +ignores them, and after the next upgrade 0.13 returns the relayed podcast state +in its snapshots again and uses the typed queue only while it matches the +queue, current track and position that 0.12 saved. + ## Admin statistics The only admin interface is `/admin/v2`, also reachable from `/admin/`. diff --git a/docs/development.md b/docs/development.md index bad10bc..17134e4 100644 --- a/docs/development.md +++ b/docs/development.md @@ -35,6 +35,300 @@ source code, templates or prompts reuse the dependency layer when its recipe and toolchain are unchanged. Both dependency and application builds use `--locked`. Update the pinned build tools in `Dockerfile`; changing them rebuilds the cache. +## Connected devices and podcast queue items + +Trusted-device sync uses the shared `music_dht::device_sync` types (no local +copies) and announces protocol version 4. Each account is one virtual device: + +- podcast operations stay in the operation log and are relayed to v4 devices + only. They are also applied to the account's podcast store + (`src/podcasts/`, `PodcastStore`), which also provides the podcast snapshot + sections; +- queue items that are not library tracks become external browser items + (`source` plus the verbatim `wire_track`), so podcast episodes and Jam + placeholders survive peer → server → browser → server → peer unchanged; +- `furumusic__playback_state.queue_items_json` stores the typed queue next to + `queue_json`. `queue_json` (library ids) is still written for rollback; the + typed value is used only while the legacy columns it recorded + (`queue_json`, `current_track_id`, `queue_position`) match the row. Jam + placeholders are never stored in a user's own queue. +- a browser that selects an item it cannot play (a Jam placeholder), or an + episode whose resume position the server has not sent yet, reports + `position_unknown`, and the server sends the resume position of the + episode's progress register instead of 0. Outgoing playback states raise + each episode's `progress_generation` to the register generation, so the + device that takes over writes above it. Jam participant commands carry + `jam_participant`; the host browser checks them against its own queue. + +Rolling back to 0.12 needs the podcast operations removed from the operation +log first; see "Rolling back to 0.12" in `administration.md`. + +### Podcast store + +`PodcastStore` implements the podcast contract of `music_dht::podcast` +(Frid `PODCASTS.md`) on PostgreSQL, with the invariants of the SQLite +reference store in `furumi-library`; every table has a `user_id` key. + +| Table | Content | Synced | +| --- | --- | --- | +| `furumusic__podcast` | channel metadata, fetch URL | no (catalog) | +| `furumusic__podcast_feed_alias` | bounded grow-only alias set | in subscription records and `podcast_feed_aliases_added` | +| `furumusic__podcast_episode` | `EpisodeMeta`, `source` `feed`/`sync`, `in_feed`, cached search text | no (catalog) | +| `furumusic__podcast_season` | view over the episodes | no | +| `furumusic__podcast_subscription`, `_share`, `_progress` | registers | yes | +| `furumusic__podcast_listen_session` | grow-only set of sessions | yes | +| `furumusic__podcast_fetch_state` | ETag, Last-Modified, check times, errors, cover cache | never | + +- One apply path: received operations, snapshot records and local actions go + through the `SyncOpWire` materializers and the `merge_*` functions of + `music_dht::podcast`. Each write transaction holds a per-account advisory + lock. +- Local actions (`subscribe`, `unsubscribe`, `set_shared`, `start_session`, + `checkpoint`, `seek`, `complete_if_current`, `restart`, `mark_played`, + `mark_unplayed`, `record_listen_session`) append their operation to the + account's sync log with the next stamp (`devices::append_local_op`: `seq`, + `hlc_ms`, `op_id = :`) and apply exactly that operation in + the same transaction. +- `apply_feed` resolves episode ids with the shared no-GUID rule against the + stored episodes, including those only known from synced references. It + updates the catalog and the alias set, never a register, progress or a + session. +- Snapshots export each section newest first up to its count and bound it + with `bound_podcast_state_to`. The catalog and the fetch state are never + exported. + +Migrations `m_0056_create_podcast_store` and `m_0057_import_podcast_relay` +create the tables and import what 0.13.0 kept in `furumusic__fed_podcast_relay` +(registers as contract JSON, already merged) plus the catalog rows of the +episode references in the operation log. The relay table is no longer written +or read; it stays until a later cleanup release. + +### Feed refresh and audio delivery + +The runtime around the store lives next to it in `src/podcasts/`: + +- `net.rs` — `SafeHttp`, the only HTTP client for podcast feeds and audio. + It accepts `http`/`https` URLs without userinfo, resolves names with + `GuardedResolver` and refuses the whole answer when any address is not + public (loopback, RFC 1918, CGNAT, link-local, cloud metadata, unique local, + multicast, unspecified, documentation, reserved; IPv4-mapped, NAT64 and + 6to4 addresses are judged by their IPv4 part). The connector connects to + exactly the checked addresses, so a second DNS answer cannot be swapped in. + IP literals are checked before connecting, redirects are followed manually + (each hop checked, at most 5), and system proxies, cookies and `Referer` + are disabled. `GuardError::is_refusal` is true only for what the URL + shows (scheme, credentials, port, a blocked IP literal); a name that + resolves to a blocked address is reported to users like a transport + failure, so the answers do not reveal internal DNS names. +- `fetch.rs` — `FeedFetcher`: conditional GET (`If-None-Match`, + `If-Modified-Since`), `Accept-Encoding: identity`, 32 MiB and 60 s limits, + `Retry-After` on 429/503, the shared `music_dht` parser. Only a chain of + permanent redirects (301/308) moves a feed. +- `runner.rs` — scheduling: `podcasts_due_for_refresh`, one request per feed + URL for all accounts subscribed to it (conditional only when they share the + same validators), `apply_feed`, then the fetch state. Healthy feeds are + checked hourly plus a stable spread of up to 10 minutes; failures back off + 15 min × 2ⁿ⁻¹ up to a day (a longer `Retry-After` wins). `refresh_now` is + the manual refresh: subscribed podcasts only, at most every 30 s. + `jobs/podcast_refresh.rs` runs `refresh_due` every five minutes (200 + podcasts taken in turns across accounts without a per-account cap, 4 + requests in flight, one run at a time via an advisory lock). +- `proxy.rs` and `http.rs` — the optional audio proxy and the endpoints under + `/api/podcasts`. The proxy takes an episode id, loads the URL from the + account's catalog (`PodcastStore::audio_source`), forwards a single + sanitized byte `Range`, passes 200/206/416 through, accepts audio content + types only (a generic binary type needs an audio URL extension or a file + that starts like audio, `sniff_audio`: read from the response when it + starts at byte 0 and has at least 16 bytes, otherwise with its own + `bytes=0-15` request; the feed's enclosure type is never trusted), and + stops at 1 GiB, 20 s without headers, 30 s + without data or 4 h per response. It is off unless + `podcast_audio_proxy_enabled` is set; by default the browser streams from + the publisher URL. + +### Podcast UI and playback + +`api.rs` holds what the endpoints do, independent of HTTP (`http.rs` only +authenticates and maps statuses), so the Postgres tests drive it directly: + +| Endpoint (`/api/podcasts`) | Does | +| --- | --- | +| `GET /feeds[?all=1]`, `POST /feeds {url}` | list; subscribe after fetching the feed (`runner::subscribe_url`) | +| `GET /feeds/{id}`, `GET /feeds/{id}/episodes?season=&played=&q=&sort=&limit=&offset=` | podcast, seasons, history; episode pages with the browser queue item | +| `POST /feeds/{id}/subscribe`, `/unsubscribe`, `POST /{id}/refresh` | subscription and manual refresh | +| `POST /episodes/{id}/restart`, `/played`, `/unplayed` | explicit progress actions | +| `POST /episodes/{id}/playback` | source, resume position; with `start` a new session (explicit, above `carried_generation`); records an unknown handed-off reference first | +| `POST /episodes/{id}/progress` | `checkpoint`/`pause`/`stop`/`handoff` at the session generation, `seek` explicit, completion once (`ended` or `reaches_completion`), listening session record | + +Request bodies must be `application/json` (415 otherwise, like cot's `Json` +extractor): other types need no CORS preflight and would let a same-site page +post with the session cookie. + +In the browser (`templates/player/scripts.html`), `_playEpisodeLocal` asks +for the playback, loads the publisher URL (or the proxy: mixed content, or a +failed direct load) and seeks before playing; when the source cannot seek to +the saved position it shows that instead of playing from elsewhere. Only a +handoff or the user's own seek sends a position (`explicitPosition`); an +ordinary Play, including after a reload, resumes from the register, and Play +after a pause first checks that the register is still this session's +(`register_session_id`). A seek made while the source is being prepared is +applied when it arrives. Writes of one session run in order +(`_reportEpisode`), read the session generation at send time and carry the +position of their own episode. After `superseded`, checkpoints stop; if the +user keeps listening here, the next write (one checkpoint interval later, or +at once on pause, stop or the end of the stream) records the old session and +starts a new explicit one (`_takeOverEpisode`). A browser that loses the +active role sends one final `stop` at its session generation (it records the +listening; the new device's start keeps the register), unless it handed off +itself and the transfer went through (a failed transfer clears `handedOff` in +the devices store's `select`). Preparing the same episode again (Play after +its source failed) waits for the old session's final report, so the server +resumes from where the audio stopped. Marking the loaded episode played or unplayed in the same tab saves +and closes its session and pauses it. Prev on an episode goes back 15 s +locally instead of resetting it. Jam guests never play or save episodes +(they only ever see placeholders). The `podcasts` store backs the Podcasts +views in `shell.html`. + +Tests never reach the internet: `test_server.rs` serves on 127.0.0.1 and +provides a scripted resolver. Loopback is reachable only through +`AddressPolicy::test_allow_loopback`, which exists only under `#[cfg(test)]`. + +Everything sent to a device is gated by the version that device reported itself +(`profile_protocol_version`; relayed profiles only raise it). See the podcast +contract (`PODCASTS.md`) and `PLAYBACK_PROTOCOL.md` in the Frid repository. + +```bash +cargo test devices:: queue_items podcasts:: queue_persistence +node --test tests/player_queue_sources.test.cjs tests/player_podcasts.test.cjs +# Disposable PostgreSQL; the tests run the production migrations on it, and +# the migration test creates and drops a database next to it: +FURU_DEVICE_SYNC_TEST_DATABASE_URL=postgresql://... \ + cargo test postgres_tests -- --ignored +# Sharing, aggregation and a loopback publication from the web node to +# another iroh node (temporary directories): +FURU_DEVICE_SYNC_TEST_DATABASE_URL=postgresql://... \ + cargo test discovery -- --include-ignored +FURU_METADATA_FIX_TEST_DATABASE_URL=postgresql://... \ + cargo test undo_accepts_playback_archives_written_before_the_typed_queue -- --ignored +``` + +`wire_compat_tests` compares the shared types with the private copies of +0.12.x byte for byte; `fixtures/v3_hello.golden.json` is the v3 shape. Rewrite +it only for a deliberate change, with `FURUMUSIC_BLESS_GOLDEN=1`. + +### Public podcast discovery + +`src/podcasts/discovery.rs` decides what the server node publishes on the +`music_dht::podcast_dht` overlay; `src/federation/podcast_discovery.rs` runs +the overlay on the federation node (ALPN `furumi-fd/podcast-dht/1`, cache +`podcast-dht.sqlite3` in the federation data directory, `serve`, +`maintenance`, the reconciler and a capability probe that calls +`note_peer_capabilities`). The capability manifest advertises `podcast_dht` +only while the overlay runs. + +- The server's node key is the only publisher. Each account's + `AccountShare` is built like the frid integration pattern for SQLite + clients: the merged subscription register (with its aliases) and share + register, and a card from `PodcastCardSpec::from_subscription` with the + account's catalog fetch URL as `feed_url`, the origin URL as `id_feed_url` + and the feed's episode count and newest date. +- `load_shared` reads active accounts only (`furumusic__user.is_active`). + An `(account, podcast)` whose stored data does not decode (a URL, the + register JSON, an id) is left out and counted (`unreadable`): only a + failing connection or query skips the round, so revocation stays + fail-safe for each podcast on its own. +- `aggregate_share_states` keeps the accounts for which `effective_share` + holds and whose card has no privacy warnings or a matching confirmation, + and builds one `PodcastShareState` per podcast with + `privacy_confirmed = true`. Representative: no warnings, then the + incumbent (the previous round's representative, kept in memory, or after + a restart an account whose card text equals the active card's text from + `PodcastDht::published`), then `AccountShare::trusted_meta` + (`meta_from_feed` and the catalog URL is the origin; an account can point + its synced catalog URL anywhere before the server fetches it), then the + smallest account id. +- At most 192 podcasts (replicas keep 256 records per publisher, tombstones + included), allocated round-robin over the sharing accounts (`allocate`): + each account's queue holds its podcasts the node already publishes + first, then by its own share time; in every round each account takes its + next podcast not taken yet, published ones first inside a round. An + account's share times therefore only order its own podcasts. A podcast + without a live record of the node (an active card or an unexpired + tombstone, `Incumbents::live_keys`) needs a new replica key; at most + `256 - live keys` of them are taken per round (`Incumbents::key_headroom`), + the others wait in `over_limit`, so a rebalance never displaces more + cards than replicas accept new ones for. + `PodcastDht::sync_shares` publishes the result and tombstones everything + else. The podcasts left out (`over_limit`) and those `sync_shares` + reported as failed are kept in memory for `ShareView.held`. +- `furumusic__podcast_share_confirmation` (`m_0058`) is the client-held + `privacy_confirmed`: per account and podcast, bound to the share + register's `op_id` and the exact card URLs. Never synced. +- The reconciler runs on `discovery::request_reconcile()` (after local share, + subscribe and unsubscribe actions, received subscription/share/alias + operations and snapshot sections, refreshes of shared podcasts; 2 s + debounce) and every five minutes. Connection and query errors skip a + round. Only counts are logged. +- Its first round in a federation run waits for the first device-sync round + (`devices::sync_loop` reports it through a `watch` channel once every + exchange of the first complete device listing has answered, failed or + timed out; at most `FIRST_DEVICE_SYNC_WAIT`, 45 s). The overlay withholds + the cards stored before the restart until the first `sync_shares`, which + re-sends the cards still shared, so a share turned off on a TUI while the + server was stopped is not announced again. A device that cannot be + reached at start delivers its decision only when it syncs. +- `ShareView` (in `GET /api/podcasts/feeds/{id}` and the share response): + `discovery` is `running`, `unavailable` (the node runs, the overlay did + not start; `Federation::podcast_discovery_unavailable`) or `off`; + `published` is this server node's own active card only; `held` is + `over_limit` or `failed` after the last round. An unreadable overlay + cache degrades to `published: false` instead of failing the page. +- `POST /api/podcasts/feeds/{id}/share` requires a JSON body + (`{"shared": bool, "confirm"?: {"feed_url", "id_feed_url"}}`); a card with + warnings is shared only with `confirm` naming exactly its current URLs + (otherwise 409 `needs_confirmation` with the warnings, nothing written). + +Search and subscription from a card (`src/podcasts/search.rs`): + +- `GET /api/player/federation/search/events` starts `PodcastDht::search` + (`Federation::spawn_podcast_search`, at most 20 hits, 20 s) next to the + music DHT search. Music events stream first and are unchanged + (`federation.track|artist|release`); every event now carries `"type": + "music"` or `"type": "podcast"`. Podcast hits follow as + `federation.podcast` events with a `PodcastHitView` entity: the card's + public fields, the origin `feed_url` (never `claimed_feed_urls`), + `publisher_count`, `rank` and `exact_match` from `merge_search_results` + with the music items, and the searching account's `subscribed` / + `local_podcast_id` (by id, or a known feed URL or alias through + `resolve_podcast_id`). `search.completed` adds `podcast_discovery` + (`running`, `off`, `failed`); a failed podcast search, or a failure to + prepare its hits for the account (`Federation::podcast_search_events` + never fails), only drops the podcast hits and reports `failed`. +- `POST /api/podcasts/discovery/preview` and `/discovery/subscribe` take + `{"podcast_id", "feed_url"}` from a hit and refuse a URL the id does not + hash from (422). The preview (optional `"season"`: a number as a + string, `"none"` or `"all"`) fetches the feed through the SSRF-guarded + fetcher and returns the channel, season counts, `listed_count` and the + newest 50 episodes of the season, without storing anything; subscribe + fetches the origin URL, then `runner::subscribe_fetched` writes a + private subscription with the card's id, or the account's existing + podcast for a known URL of `runner::document_urls` (the origin, the end + of a permanent redirect chain, the feed's `itunes:new-feed-url`; a + subscribed podcast first, `search::known_podcast`; also used by "add + RSS"). The document is + ingested only into a new podcast or when it came from the existing + podcast's own fetch URL (requested there, or a permanent redirect ended + there); otherwise the podcast is subscribed again with its stored + metadata and nothing of the document is written. Nothing is written when + the account is already subscribed, by id or known URL, before or after + the fetch. Both share + the "add RSS" request limits; the browser aborts a preview it no longer + shows (`AbortController`), so a closed card frees its slot once the + server sees the connection close. +- The browser keeps only expected, typed card fields (`podcasts.foundHit`): + ids must look like podcast ids, URLs must be absolute `http(s)` without + credentials (artwork: `https` only), and texts render with `x-text`. + ## Contributing Bug reports, design discussions, and patches are welcome. Before submitting a diff --git a/src/admin/metadata_fix.rs b/src/admin/metadata_fix.rs index 80c1b3b..33b4954 100644 --- a/src/admin/metadata_fix.rs +++ b/src/admin/metadata_fix.rs @@ -1438,18 +1438,28 @@ async fn delete_entities( } } // Playback queues contain local IDs rather than foreign keys. - let states: Vec<(i64, Option, String)> = sqlx::query_as( - "SELECT id,current_track_id,queue_json FROM furumusic__playback_state FOR UPDATE", + // (id, current_track_id, queue_json, queue_items_json, queue_position) + type QueueRow = (i64, Option, String, Option, i32); + let states: Vec = sqlx::query_as( + "SELECT id,current_track_id,queue_json,queue_items_json,queue_position FROM furumusic__playback_state FOR UPDATE", ) .fetch_all(&mut *conn) .await?; - for (id, current, queue_json) in states { + for (id, current, queue_json, queue_items_json, queue_position) in states { let mut queue: Vec = serde_json::from_str(&queue_json).unwrap_or_default(); if !current.is_some_and(|id| deleted.tracks.contains(&id)) && !queue.iter().any(|id| deleted.tracks.contains(id)) { continue; } + let mut typed = crate::player::queue_items::StoredQueue::from_columns( + queue_items_json.as_ref(), + &crate::player::queue_items::LegacyColumns { + queue: queue.clone(), + current_track_id: current, + queue_position, + }, + ); let before: String = sqlx::query_scalar( "SELECT to_jsonb(t)::text FROM furumusic__playback_state t WHERE id=$1", ) @@ -1461,8 +1471,19 @@ async fn delete_entities( let position = current .and_then(|id| queue.iter().position(|x| *x == id)) .unwrap_or(0) as i32; - let after: String = sqlx::query_scalar("UPDATE furumusic__playback_state SET current_track_id=$2,queue_json=$3,queue_position=$4,position_ms=CASE WHEN current_track_id IS DISTINCT FROM $2 THEN 0 ELSE position_ms END WHERE id=$1 RETURNING to_jsonb(furumusic__playback_state)::text") - .bind(id).bind(current).bind(serde_json::to_string(&queue)?).bind(position).fetch_one(&mut *conn).await?; + if let Some(typed) = &mut typed { + typed.retain_library( + |id| !deleted.tracks.contains(&id), + crate::player::queue_items::LegacyColumns { + queue: queue.clone(), + current_track_id: current, + queue_position: position, + }, + ); + } + let queue_items_json = typed.map(serde_json::to_value).transpose()?; + let after: String = sqlx::query_scalar("UPDATE furumusic__playback_state SET current_track_id=$2,queue_json=$3,queue_position=$4,queue_items_json=$5,position_ms=CASE WHEN current_track_id IS DISTINCT FROM $2 THEN 0 ELSE position_ms END WHERE id=$1 RETURNING to_jsonb(furumusic__playback_state)::text") + .bind(id).bind(current).bind(serde_json::to_string(&queue)?).bind(position).bind(queue_items_json).fetch_one(&mut *conn).await?; archived.push(ArchivedRows { table: "furumusic__playback_state".into(), rows: serde_json::json!([serde_json::from_str::(&before)?]), @@ -1502,6 +1523,7 @@ async fn restore_deleted(conn: &mut PgConnection, archived: &[ArchivedRows]) -> &[ "current_track_id", "queue_json", + "queue_items_json", "queue_position", "position_ms", ] @@ -1515,7 +1537,9 @@ async fn restore_deleted(conn: &mut PgConnection, archived: &[ArchivedRows]) -> }) .collect::>() .join(","); - let affected = sqlx::query(&format!("UPDATE {table} t SET {assignments} FROM jsonb_array_elements($1::jsonb) WITH ORDINALITY b(value,n) JOIN jsonb_array_elements($2::jsonb) WITH ORDINALITY a(value,n) USING(n) WHERE to_jsonb(t)=a.value")) + // Archives written before m_0054 have no `queue_items_json` key; + // compare them without the column added since. + let affected = sqlx::query(&format!("UPDATE {table} t SET {assignments} FROM jsonb_array_elements($1::jsonb) WITH ORDINALITY b(value,n) JOIN jsonb_array_elements($2::jsonb) WITH ORDINALITY a(value,n) USING(n) WHERE (CASE WHEN a.value ? 'queue_items_json' THEN to_jsonb(t) ELSE to_jsonb(t) - 'queue_items_json' END)=a.value")) .bind(entry.rows.to_string()).bind(after.to_string()).execute(&mut *conn).await?.rows_affected(); anyhow::ensure!( affected as usize == after.as_array().map_or(0, Vec::len), @@ -1804,6 +1828,51 @@ pub(super) async fn undo( mod tests { use super::*; + #[tokio::test] + #[ignore = "requires a migrated disposable database in FURU_METADATA_FIX_TEST_DATABASE_URL"] + async fn undo_accepts_playback_archives_written_before_the_typed_queue() { + let pool = + sqlx::PgPool::connect(&std::env::var("FURU_METADATA_FIX_TEST_DATABASE_URL").unwrap()) + .await + .unwrap(); + let mut tx = pool.begin().await.unwrap(); + let user_id = 9_000_000_000 + i64::from(std::process::id()); + // State after the deletion, read back like an archive written by 0.12 + // (`to_jsonb` of the row before m_0054 added `queue_items_json`). + let after: serde_json::Value = sqlx::query_scalar( + "INSERT INTO furumusic__playback_state (user_id,current_track_id,position_ms,queue_json,queue_position,shuffle,repeat_mode,volume,updated_at) VALUES ($1,NULL,0,'[]',0,false,'off',0.7,'now') RETURNING to_jsonb(furumusic__playback_state) - 'queue_items_json'", + ) + .bind(user_id) + .fetch_one(&mut *tx) + .await + .unwrap(); + let mut before = after.clone(); + before["queue_json"] = serde_json::json!("[123456789]"); + before["position_ms"] = serde_json::json!(5000); + let archive = ArchivedRows { + table: "furumusic__playback_state".into(), + rows: serde_json::json!([before]), + after: Some(serde_json::json!([after])), + }; + restore_deleted(&mut tx, std::slice::from_ref(&archive)) + .await + .unwrap(); + let (queue_json, position_ms, typed): (String, i32, Option) = + sqlx::query_as( + "SELECT queue_json,position_ms,queue_items_json FROM furumusic__playback_state WHERE user_id=$1", + ) + .bind(user_id) + .fetch_one(&mut *tx) + .await + .unwrap(); + assert_eq!(queue_json, "[123456789]"); + assert_eq!(position_ms, 5000); + assert_eq!(typed, None); + // A row changed since the deletion is still refused. + assert!(restore_deleted(&mut tx, &[archive]).await.is_err()); + tx.rollback().await.unwrap(); + } + fn snapshot() -> Snapshot { Snapshot { artists: vec![ArtistState { diff --git a/src/admin/v2.rs b/src/admin/v2.rs index 72a4ebc..aa2a9ce 100644 --- a/src/admin/v2.rs +++ b/src/admin/v2.rs @@ -545,6 +545,8 @@ struct AdminSettingsValues { youtube_proxy_id: String, #[serde(default)] youtube_cookie_id: String, + #[serde(default)] + podcast_audio_proxy_enabled: bool, } #[derive(Debug, Clone, Serialize, Deserialize, JsonSchema)] @@ -615,6 +617,7 @@ struct AdminSettingsSources { torrent_proxy_id: &'static str, youtube_proxy_id: &'static str, youtube_cookie_id: &'static str, + podcast_audio_proxy_enabled: &'static str, } #[derive(Debug, Deserialize)] @@ -667,6 +670,8 @@ pub(super) struct UpdateSettingsRequest { youtube_proxy_id: String, #[serde(default)] youtube_cookie_id: String, + #[serde(default)] + podcast_audio_proxy_enabled: bool, } fn default_similarity_model() -> String { @@ -1370,6 +1375,10 @@ pub async fn update_settings( ("torrent_proxy_id", torrent_proxy_id), ("youtube_proxy_id", youtube_proxy_id), ("youtube_cookie_id", youtube_cookie_id), + ( + "podcast_audio_proxy_enabled", + body.podcast_audio_proxy_enabled.to_string(), + ), ]; for (key, value) in fields { let mut entry = ConfigEntry::new(key.to_string(), value); @@ -1714,6 +1723,7 @@ fn settings_dto( torrent_proxy_id: config.torrent_proxy_id, youtube_proxy_id: config.youtube_proxy_id, youtube_cookie_id: config.youtube_cookie_id, + podcast_audio_proxy_enabled: config.podcast_audio_proxy_enabled, }, sources: AdminSettingsSources { auth_password_enabled: sources.auth_password_enabled.code(), @@ -1750,6 +1760,7 @@ fn settings_dto( torrent_proxy_id: sources.torrent_proxy_id.code(), youtube_proxy_id: sources.youtube_proxy_id.code(), youtube_cookie_id: sources.youtube_cookie_id.code(), + podcast_audio_proxy_enabled: sources.podcast_audio_proxy_enabled.code(), }, } } diff --git a/src/config.rs b/src/config.rs index e554667..30572ee 100644 --- a/src/config.rs +++ b/src/config.rs @@ -149,6 +149,7 @@ pub struct ConfigSources { pub torrent_proxy_id: ConfigSource, pub youtube_proxy_id: ConfigSource, pub youtube_cookie_id: ConfigSource, + pub podcast_audio_proxy_enabled: ConfigSource, } impl Default for ConfigSources { @@ -190,6 +191,7 @@ impl Default for ConfigSources { torrent_proxy_id: ConfigSource::Default, youtube_proxy_id: ConfigSource::Default, youtube_cookie_id: ConfigSource::Default, + podcast_audio_proxy_enabled: ConfigSource::Default, } } } @@ -407,6 +409,10 @@ pub struct AppConfig { pub youtube_proxy_id: String, /// Saved cookie-file id used by yt-dlp; empty means no cookies. pub youtube_cookie_id: String, + /// Whether the server proxies podcast episode audio from publishers for + /// browsers that cannot load it directly (CORS, mixed content). Off by + /// default: browsers stream from the publisher URL. + pub podcast_audio_proxy_enabled: bool, } impl Default for AppConfig { @@ -451,6 +457,7 @@ impl Default for AppConfig { torrent_proxy_id: String::new(), youtube_proxy_id: String::new(), youtube_cookie_id: String::new(), + podcast_audio_proxy_enabled: false, } } } @@ -493,6 +500,7 @@ impl_env_overrides!( torrent_proxy_id, youtube_proxy_id, youtube_cookie_id, + podcast_audio_proxy_enabled, ); impl AppConfig { @@ -636,6 +644,7 @@ impl AppConfig { apply_db_field!(torrent_proxy_id); apply_db_field!(youtube_proxy_id); apply_db_field!(youtube_cookie_id); + apply_db_field!(podcast_audio_proxy_enabled); } pub fn parsed_download_proxies(&self) -> anyhow::Result> { @@ -693,6 +702,7 @@ mod tests { assert!(cfg.torrent_downloads_enabled); assert!(cfg.youtube_downloads_enabled); assert!(cfg.parsed_download_proxies().unwrap().is_empty()); + assert!(!cfg.podcast_audio_proxy_enabled); } #[test] diff --git a/src/federation/capabilities.rs b/src/federation/capabilities.rs index 26cf5c5..fef93c7 100644 --- a/src/federation/capabilities.rs +++ b/src/federation/capabilities.rs @@ -1,43 +1,64 @@ -//! Informational publication of the protocol versions exposed by this peer. +//! Informational publication of the protocol versions exposed by this peer, +//! and observation of the peers' manifests for the podcast-discovery overlay. -use std::time::Duration; +use std::collections::{BTreeSet, HashMap}; +use std::sync::Arc; +use std::time::{Duration, Instant}; -use anyhow::Result; -use music_dht::StreamAcceptor; +use anyhow::{Context, Result}; +use futures_util::StreamExt; use music_dht::capabilities::{ - CAPABILITIES_PROTOCOL_VERSION, CapabilityManifest, CapabilityMessage, JAM_ID, SIMILARITY_ID, - read_message, write_message, + CAPABILITIES_ALPN, CAPABILITIES_PROTOCOL_VERSION, CapabilityManifest, CapabilityMessage, + JAM_ID, SIMILARITY_ID, read_message, write_message, }; +use music_dht::podcast_dht::PodcastDht; +use music_dht::{EndpointId, MusicDhtService, StreamAcceptor}; use super::serve::AUDIO_PROTOCOL_VERSION; -fn local_manifest() -> CapabilityManifest { - CapabilityManifest::frid("furumusic", env!("CARGO_PKG_VERSION")) +/// How often known peers are considered for a manifest probe. +const PROBE_INTERVAL: Duration = Duration::from_secs(5 * 60); +/// A peer's manifest is probed again after this long. +const REPROBE_AFTER: Duration = Duration::from_secs(60 * 60); +/// Peers probed per round and in parallel. +const PROBES_PER_ROUND: usize = 64; +const PROBE_CONCURRENCY: usize = 8; +const PROBE_TIMEOUT: Duration = Duration::from_secs(5); + +/// The manifest of this node. The public podcast-discovery overlay is +/// advertised only while the node serves it (`podcast_dht`). +fn local_manifest(podcast_dht: bool) -> CapabilityManifest { + let manifest = CapabilityManifest::frid("furumusic", env!("CARGO_PKG_VERSION")) // The web server does not expose federation Jam yet. .without_protocol(JAM_ID) .with_protocol("audio", AUDIO_PROTOCOL_VERSION) .with_protocol( SIMILARITY_ID, music_dht::similarity::SIMILARITY_PROTOCOL_VERSION, - ) + ); + if podcast_dht { + manifest.with_podcast_dht() + } else { + manifest + } } -pub async fn serve(mut acceptor: StreamAcceptor) { +pub async fn serve(mut acceptor: StreamAcceptor, podcast_dht: bool) { while let Some(stream) = acceptor.accept().await { tokio::spawn(async move { - if let Err(error) = serve_one(stream).await { + if let Err(error) = serve_one(stream, podcast_dht).await { tracing::debug!("capability stream failed: {error:#}"); } }); } } -async fn serve_one(mut stream: music_dht::ByteStream) -> Result<()> { +async fn serve_one(mut stream: music_dht::ByteStream, podcast_dht: bool) -> Result<()> { let response = match read_message(&mut stream).await? { CapabilityMessage::Get { version: CAPABILITIES_PROTOCOL_VERSION, } => CapabilityMessage::Manifest { - manifest: local_manifest(), + manifest: local_manifest(podcast_dht), }, CapabilityMessage::Get { version } => CapabilityMessage::Error { message: format!("unsupported capability protocol {version}"), @@ -52,13 +73,93 @@ async fn serve_one(mut stream: music_dht::ByteStream) -> Result<()> { Ok(()) } +/// Probes the manifests of known peers and tells the podcast overlay which +/// of them lack it ([`PodcastDht::note_peer_capabilities`]), so bootstrap +/// stops dialing them. A peer that does not answer is left alone: the +/// overlay only learns contacts that answer on its own protocol anyway. +pub async fn probe_loop(service: Arc, dht: Arc) { + let mut probed: HashMap = HashMap::new(); + let mut interval = tokio::time::interval(PROBE_INTERVAL); + interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip); + loop { + interval.tick().await; + let own = service.endpoint_id(); + probed.retain(|_, at| at.elapsed() < REPROBE_AFTER); + let peers: Vec = service + .connected_peers() + .into_iter() + .chain( + service + .known_peers() + .into_iter() + .map(|contact| contact.peer_id), + ) + .filter(|peer| *peer != own && !probed.contains_key(peer)) + .collect::>() + .into_iter() + .take(PROBES_PER_ROUND) + .collect(); + let results = futures_util::stream::iter(peers.into_iter().map(|peer| { + let service = Arc::clone(&service); + async move { (peer, probe_peer(&service, peer).await) } + })) + .buffer_unordered(PROBE_CONCURRENCY) + .collect::>() + .await; + for (peer, result) in results { + probed.insert(peer, Instant::now()); + match result { + Ok(manifest) => { + if let Err(error) = dht.note_peer_capabilities(peer, &manifest).await { + tracing::debug!(%peer, "podcast overlay capability note failed: {error}"); + } + } + Err(error) => { + tracing::trace!(%peer, "peer capability probe unavailable: {error:#}"); + } + } + } + } +} + +async fn probe_peer(service: &MusicDhtService, peer: EndpointId) -> Result { + let mut stream = + tokio::time::timeout(PROBE_TIMEOUT, service.open_stream(peer, CAPABILITIES_ALPN)) + .await + .context("capability stream timed out")??; + write_message( + &mut stream, + &CapabilityMessage::Get { + version: CAPABILITIES_PROTOCOL_VERSION, + }, + ) + .await?; + stream.send.finish()?; + let response = tokio::time::timeout(PROBE_TIMEOUT, read_message(&mut stream)) + .await + .context("capability request timed out")??; + let CapabilityMessage::Manifest { manifest } = response else { + anyhow::bail!("peer did not return a capability manifest"); + }; + manifest.validate()?; + Ok(manifest) +} + #[cfg(test)] mod tests { use super::*; + #[test] + fn the_podcast_overlay_is_advertised_only_while_it_runs() { + assert!(!local_manifest(false).supports_podcast_dht()); + let manifest = local_manifest(true); + assert!(manifest.supports_podcast_dht()); + manifest.validate().unwrap(); + } + #[test] fn manifest_describes_only_supported_player_protocols() { - let manifest = local_manifest(); + let manifest = local_manifest(false); assert_eq!(manifest.application, "furumusic"); assert_eq!( manifest.protocols.get("audio"), diff --git a/src/federation/client.rs b/src/federation/client.rs index 7c36185..e1b99a6 100644 --- a/src/federation/client.rs +++ b/src/federation/client.rs @@ -9,6 +9,7 @@ use anyhow::{Context, Result}; use music_dht::{ItemKind, LibraryItem, normalize_content_id}; use serde::Serialize; use serde_json::{Value, json}; +use sqlx::PgPool; use sqlx::Row as _; use tokio::io::AsyncReadExt; @@ -94,6 +95,9 @@ pub struct SearchEvent { pub search_id: String, pub sequence: u64, pub kind: &'static str, + /// `music` or `podcast` (`music_dht::podcast_dht::FederatedSearchHit`). + #[serde(rename = "type")] + pub hit_type: &'static str, pub peer: Option, pub entity_key: Value, pub entity: Value, @@ -218,7 +222,12 @@ impl Federation { /// Performs one bounded DHT search and returns entity upserts. The HTTP /// layer streams each upsert independently; catalog fan-out can append /// events to the same contract without changing the browser model. - pub async fn search_events(&self, search_id: &str, query: &str) -> Result> { + /// + /// Music only (`federation.track|artist|release`, marked `"type": + /// "music"`). Public podcasts are searched next to it + /// ([`Self::spawn_podcast_search`]) and follow as `federation.podcast` + /// events ([`Self::podcast_search_events`]). + pub async fn search_events(&self, search_id: &str, query: &str) -> Result { let query = query.trim(); anyhow::ensure!(!query.is_empty(), "search query is empty"); anyhow::ensure!(query.chars().count() <= 200, "search query is too long"); @@ -233,13 +242,10 @@ impl Federation { "federated search started" ); let own = service.endpoint_id(); - let result = tokio::time::timeout( - std::time::Duration::from_secs(20), - service.search_network(query), - ) - .await - .map_err(|_| anyhow::anyhow!("federated search timed out after 20 seconds"))? - .map_err(|err| anyhow::anyhow!("federated search failed: {err}"))?; + let result = tokio::time::timeout(SEARCH_TIMEOUT, service.search_network(query)) + .await + .map_err(|_| anyhow::anyhow!("federated search timed out after 20 seconds"))? + .map_err(|err| anyhow::anyhow!("federated search failed: {err}"))?; tracing::info!( search_id, query, @@ -249,154 +255,14 @@ impl Federation { elapsed_ms = started.elapsed().as_millis() as u64, "federated DHT search finished" ); - let all_items: Vec = result + let items: Vec = result .local_results .into_iter() .chain(result.network_results) .collect(); let pool = self.pool().await?; - let mut by_content: HashMap = HashMap::new(); - for item in all_items.iter().filter(|item| item.kind == ItemKind::Track) { - let Some(content_id) = item.content_id.as_deref().and_then(normalize_content_id) else { - // A globally usable track reference must be verifiable. - continue; - }; - let local = local_availability(&pool, &content_id).await?; - let source = FederationSourceDto { - owner: item.owner.to_string(), - item_id: hex(item.id.as_bytes()), - }; - let entry = by_content.entry(content_id.clone()).or_insert_with(|| { - track_from_item(content_id.clone(), item, local, item.owner == own) - }); - if !entry.availability.federation.iter().any(|candidate| { - candidate.owner == source.owner && candidate.item_id == source.item_id - }) { - entry.availability.federation.push(source); - } - if entry.availability.local.is_some() { - entry.availability.state = "local"; - } - } - - let mut tracks: Vec<_> = by_content.into_values().collect(); - tracks.sort_by(|left, right| { - left.metadata - .title - .to_lowercase() - .cmp(&right.metadata.title.to_lowercase()) - }); - - let mut events = Vec::with_capacity(all_items.len()); - for (index, track) in tracks.into_iter().enumerate() { - persist_track_ref(&pool, &track).await?; - let peer = track - .availability - .federation - .first() - .map(|source| source.owner.clone()); - events.push(SearchEvent { - search_id: search_id.to_owned(), - sequence: index as u64 + 1, - kind: "federation.track", - peer, - entity_key: serde_json::to_value(&track.key)?, - entity: serde_json::to_value(track)?, - }); - } - let mut artist_peers: HashMap)> = HashMap::new(); - let mut releases: HashMap = HashMap::new(); - for item in &all_items { - match item.kind { - ItemKind::Artist => { - let key = music_dht::normalize_name(&item.name); - let entry = artist_peers - .entry(key) - .or_insert_with(|| (item.name.clone(), Vec::new())); - let owner = item.owner.to_string(); - if !entry.1.contains(&owner) { - entry.1.push(owner); - } - } - ItemKind::Release => { - let artist_keys: Vec = item - .artist_names - .iter() - .map(|name| music_dht::normalize_name(name)) - .collect(); - let normalized_title = music_dht::normalize_name(&item.name); - let cover_url = all_items - .iter() - .find(|track| { - track.kind == ItemKind::Track - && track.release_title.as_deref().is_some_and(|title| { - music_dht::normalize_name(title) == normalized_title - }) - && track.year == item.year - }) - .map(|track| { - format!( - "/api/player/federation/tracks/artwork?owner={}&item_id={}", - track.owner, - hex(track.id.as_bytes()) - ) - }); - let key = format!( - "{}|{}|{}", - normalized_title, - artist_keys.join(","), - item.year.map_or_else(String::new, |year| year.to_string()) - ); - releases.entry(key.clone()).or_insert_with(|| { - json!({ - "key": { - "normalized_title": music_dht::normalize_name(&item.name), - "primary_artists": artist_keys, - "release_type": null, - "year": item.year, - }, - "title": item.name, - "artists": item.artist_names, - "year": item.year, - "cover_url": cover_url, - "sources": [{ - "owner": item.owner.to_string(), - "item_id": hex(item.id.as_bytes()), - }], - }) - }); - } - ItemKind::Track => {} - } - } - for (key, (name, peers)) in artist_peers { - let sequence = events.len() as u64 + 1; - events.push(SearchEvent { - search_id: search_id.to_owned(), - sequence, - kind: "federation.artist", - peer: peers.first().cloned(), - entity_key: json!({ "normalized_name": key }), - entity: json!({ - "key": { "normalized_name": key }, - "name": name, - "image_url": null, - "peers": peers, - }), - }); - } - for (key, release) in releases { - let sequence = events.len() as u64 + 1; - events.push(SearchEvent { - search_id: search_id.to_owned(), - sequence, - kind: "federation.release", - peer: None, - entity_key: json!({ "composite": key }), - entity: release, - }); - } + let events = music_search_events(&pool, search_id, &items, own).await?; tracing::info!( search_id, query, @@ -404,8 +270,317 @@ impl Federation { elapsed_ms = started.elapsed().as_millis() as u64, "federated search response ready" ); - Ok(events) + Ok(MusicSearch { events, items }) } + + /// Starts the public podcast search of `query` on the discovery + /// overlay, if it runs, next to the music search. + pub fn spawn_podcast_search(&self, query: &str) -> tokio::task::JoinHandle { + let discovery = self.podcast_discovery(); + let query = query.trim().to_owned(); + tokio::spawn(async move { search_podcasts(discovery.as_deref(), &query).await }) + } + + /// The `federation.podcast` events of a search for `user_id`, numbered + /// after the music events (see [`podcast_events`]), and the + /// `podcast_discovery` state for `search.completed`. + /// + /// Never fails: when the hits cannot be prepared for the account (a + /// database error) they are dropped and the state is `failed`, so the + /// music results stand and the browser can say the podcast search + /// failed. + pub async fn podcast_search_events( + &self, + search_id: &str, + query: &str, + user_id: i64, + music: &MusicSearch, + podcasts: PodcastSearch, + ) -> (&'static str, Vec) { + let state = podcasts.state(); + let PodcastSearch::Found(hits) = podcasts else { + return (state, Vec::new()); + }; + if hits.is_empty() { + return (state, Vec::new()); + } + let events = async { + let pool = self.pool().await?; + let first_sequence = music.events.len() as u64 + 1; + podcast_events( + &pool, + user_id, + search_id, + query.trim(), + &music.items, + first_sequence, + hits, + ) + .await + }; + match events.await { + Ok(events) => { + tracing::info!( + search_id, + podcasts = events.len(), + "public podcast hits ready" + ); + (state, events) + } + Err(error) => { + // Only the error: never the query or the hits. + tracing::warn!( + search_id, + "public podcast hits could not be prepared: {error:#}" + ); + ("failed", Vec::new()) + } + } + } +} + +/// Deadline of the music DHT and the podcast overlay searches. +const SEARCH_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(20); + +/// Music results of a federated search: the events to stream and the items +/// (podcast hits are ranked among them). +#[derive(Debug)] +pub struct MusicSearch { + pub events: Vec, + pub items: Vec, +} + +/// Outcome of the public podcast search of one federated search. +#[derive(Debug)] +pub enum PodcastSearch { + /// Public discovery does not run on this server. + Off, + /// The overlay search failed or timed out (music results still count). + Failed, + /// Hits, possibly none. + Found(Vec), +} + +impl PodcastSearch { + /// `running`, `off` or `failed` (reported in `search.completed`). + pub fn state(&self) -> &'static str { + match self { + Self::Off => "off", + Self::Failed => "failed", + Self::Found(_) => "running", + } + } +} + +async fn search_podcasts( + discovery: Option<&music_dht::podcast_dht::PodcastDht>, + query: &str, +) -> PodcastSearch { + let Some(dht) = discovery else { + return PodcastSearch::Off; + }; + let limit = crate::podcasts::search::PODCAST_SEARCH_LIMIT; + match tokio::time::timeout(SEARCH_TIMEOUT, dht.search(query, limit)).await { + Ok(Ok(hits)) => PodcastSearch::Found(hits), + Ok(Err(error)) => { + tracing::info!("public podcast search failed: {error}"); + PodcastSearch::Failed + } + Err(_) => { + tracing::info!("public podcast search timed out"); + PodcastSearch::Failed + } + } +} + +/// One `federation.podcast` event per public podcast hit, for `user_id` +/// (`subscribed`, `local_podcast_id`), ordered by the hit's place in +/// `merge_search_results` with the music items and numbered from +/// `first_sequence`. +pub(crate) async fn podcast_events( + pool: &PgPool, + user_id: i64, + search_id: &str, + query: &str, + music: &[LibraryItem], + first_sequence: u64, + hits: Vec, +) -> Result> { + let store = crate::podcasts::PodcastStore::new(pool.clone()); + let views = crate::podcasts::search::hit_views(&store, user_id, query, music, hits).await?; + views + .into_iter() + .zip(first_sequence..) + .map(|(view, sequence)| { + Ok(SearchEvent { + search_id: search_id.to_owned(), + sequence, + kind: "federation.podcast", + hit_type: "podcast", + // Subscribing never contacts a publisher: no peer to name. + peer: None, + entity_key: json!({ "podcast_id": view.podcast_id }), + entity: serde_json::to_value(view)?, + }) + }) + .collect() +} + +/// Music events of a search (tracks, then artists and releases). +pub(crate) async fn music_search_events( + pool: &PgPool, + search_id: &str, + all_items: &[LibraryItem], + own: music_dht::EndpointId, +) -> Result> { + let mut by_content: HashMap = HashMap::new(); + for item in all_items.iter().filter(|item| item.kind == ItemKind::Track) { + let Some(content_id) = item.content_id.as_deref().and_then(normalize_content_id) else { + // A globally usable track reference must be verifiable. + continue; + }; + let local = local_availability(pool, &content_id).await?; + let source = FederationSourceDto { + owner: item.owner.to_string(), + item_id: hex(item.id.as_bytes()), + }; + let entry = by_content + .entry(content_id.clone()) + .or_insert_with(|| track_from_item(content_id.clone(), item, local, item.owner == own)); + if !entry + .availability + .federation + .iter() + .any(|candidate| candidate.owner == source.owner && candidate.item_id == source.item_id) + { + entry.availability.federation.push(source); + } + if entry.availability.local.is_some() { + entry.availability.state = "local"; + } + } + + let mut tracks: Vec<_> = by_content.into_values().collect(); + tracks.sort_by(|left, right| { + left.metadata + .title + .to_lowercase() + .cmp(&right.metadata.title.to_lowercase()) + }); + + let mut events = Vec::with_capacity(all_items.len()); + for (index, track) in tracks.into_iter().enumerate() { + persist_track_ref(pool, &track).await?; + let peer = track + .availability + .federation + .first() + .map(|source| source.owner.clone()); + events.push(SearchEvent { + search_id: search_id.to_owned(), + sequence: index as u64 + 1, + kind: "federation.track", + hit_type: "music", + peer, + entity_key: serde_json::to_value(&track.key)?, + entity: serde_json::to_value(track)?, + }); + } + let mut artist_peers: HashMap)> = HashMap::new(); + let mut releases: HashMap = HashMap::new(); + for item in all_items { + match item.kind { + ItemKind::Artist => { + let key = music_dht::normalize_name(&item.name); + let entry = artist_peers + .entry(key) + .or_insert_with(|| (item.name.clone(), Vec::new())); + let owner = item.owner.to_string(); + if !entry.1.contains(&owner) { + entry.1.push(owner); + } + } + ItemKind::Release => { + let artist_keys: Vec = item + .artist_names + .iter() + .map(|name| music_dht::normalize_name(name)) + .collect(); + let normalized_title = music_dht::normalize_name(&item.name); + let cover_url = all_items + .iter() + .find(|track| { + track.kind == ItemKind::Track + && track.release_title.as_deref().is_some_and(|title| { + music_dht::normalize_name(title) == normalized_title + }) + && track.year == item.year + }) + .map(|track| { + format!( + "/api/player/federation/tracks/artwork?owner={}&item_id={}", + track.owner, + hex(track.id.as_bytes()) + ) + }); + let key = format!( + "{}|{}|{}", + normalized_title, + artist_keys.join(","), + item.year.map_or_else(String::new, |year| year.to_string()) + ); + releases.entry(key.clone()).or_insert_with(|| { + json!({ + "key": { + "normalized_title": music_dht::normalize_name(&item.name), + "primary_artists": artist_keys, + "release_type": null, + "year": item.year, + }, + "title": item.name, + "artists": item.artist_names, + "year": item.year, + "cover_url": cover_url, + "sources": [{ + "owner": item.owner.to_string(), + "item_id": hex(item.id.as_bytes()), + }], + }) + }); + } + ItemKind::Track => {} + } + } + for (key, (name, peers)) in artist_peers { + let sequence = events.len() as u64 + 1; + events.push(SearchEvent { + search_id: search_id.to_owned(), + sequence, + kind: "federation.artist", + hit_type: "music", + peer: peers.first().cloned(), + entity_key: json!({ "normalized_name": key }), + entity: json!({ + "key": { "normalized_name": key }, + "name": name, + "image_url": null, + "peers": peers, + }), + }); + } + for (key, release) in releases { + let sequence = events.len() as u64 + 1; + events.push(SearchEvent { + search_id: search_id.to_owned(), + sequence, + kind: "federation.release", + hit_type: "music", + peer: None, + entity_key: json!({ "composite": key }), + entity: release, + }); + } + Ok(events) } async fn fetch_artist_catalog( diff --git a/src/federation/devices.rs b/src/federation/devices.rs index 125aa4f..c660849 100644 --- a/src/federation/devices.rs +++ b/src/federation/devices.rs @@ -1,40 +1,69 @@ //! Personal Fed Clients for the web player. //! //! The web service has one iroh endpoint, but each user owns one virtual -//! fed-device inside that endpoint. This module speaks the same JSON-lines -//! protocol as the TUI clients on `furumi/sync/1` and maps operations into -//! user-scoped Postgres state. +//! fed-device inside that endpoint. This module speaks the shared +//! `music_dht::device_sync` JSON-lines protocol of the TUI clients on +//! `furumi/sync/2` and maps operations into user-scoped Postgres state. +//! +//! The wire types are the shared definitions; there are no local copies. +//! Device-sync protocol v4 (podcasts) is announced: podcast operations stay +//! in the generic operation log and are applied to the account's podcast +//! store ([`crate::podcasts::PodcastStore`]), which also provides the +//! podcast snapshot sections, and queue items keep their explicit +//! `PlaybackTrack.source` through the browser bridge +//! ([`crate::player::queue_items`]). Everything sent to a peer is gated by +//! the version that peer reported itself. -use music_dht::playback::{ - Announcement, Checkpoint, CommandStamp, Config as PlaybackConfig, Engine, -}; +use music_dht::playback::{Announcement, Checkpoint, Config as PlaybackConfig, Engine}; use std::collections::BTreeMap; -use std::str::FromStr; use std::sync::Arc; use std::time::Duration; use anyhow::{Context, Result}; -use music_dht::device_sync::{ListenEndReason, ListenEvent, ListenTrackMetadata}; +use music_dht::device_sync::{ + DEVICE_SYNC_PROTOCOL_VERSION, DeviceProfileWire, InviteWire, ListenEndReason, ListenEvent, + ListenTrackMetadata, PlaybackCommand, PlaybackRepeat, PlaybackSnapshot, PlaybackSource, + PlaybackStateWire, PlaybackTrack, SnapshotLike, SnapshotLikeTombstone, SnapshotPlaylist, + SnapshotPlaylistItem, SnapshotPlaylistItemTombstone, SnapshotPlaylistTombstone, SyncOpPayload, + SyncOpWire, SyncSnapshot, SyncedFedTrack, WireMessage, encode_invite, hash_secret, + op_allowed_for_peer, parse_invite, playback_allowed_for_peer, + playback_command_allowed_from_peer, podcast_snapshot_budget, profile_protocol_version, + random_hex, relayed_protocol_version, supports_listen_history, supports_podcasts, + ticket_endpoint_id, +}; +use music_dht::podcast::{EpisodeId, ProgressRecord}; use music_dht::{ByteStream, MusicDhtService, NetworkId, PeerTicket, SecretKey, StreamAcceptor}; -use serde::{Deserialize, Serialize}; +use serde::Serialize; use sqlx::Row; -use tokio::io::{AsyncRead, AsyncReadExt}; use crate::player::PlayerDeviceHub; +use crate::player::queue_items; use super::{TransportStats, record_stream_transport}; +use crate::podcasts::{ApplyOutcome, PodcastStore}; pub const SYNC_ALPN: &[u8] = b"furumi/sync/2"; const CLIENT_VERSION: &str = env!("CARGO_PKG_VERSION"); -const PROTOCOL_VERSION: u16 = 3; +/// Device-sync protocol version announced by this server's virtual devices. +/// +/// Version 4 is announced only because podcast operations, podcast snapshot +/// sections and podcast queue items are kept and relayed losslessly (see +/// the module documentation). Review that support before following a newer +/// shared protocol version. +const PROTOCOL_VERSION: u16 = 4; +const _: () = assert!( + PROTOCOL_VERSION == DEVICE_SYNC_PROTOCOL_VERSION, + "review web device-sync support before announcing a new protocol version" +); +/// Lowest protocol version of a peer that speaks `furumi/sync/2`. +const SYNC_V2_PROTOCOL_VERSION: u16 = 2; const INVITE_TTL_MS: i64 = 10 * 60 * 1000; const PAIRING_WAIT_MS: i64 = 5 * 60 * 1000; const PAIRING_RETRY_DELAY: Duration = Duration::from_secs(1); const RESPONSE_DRAIN_TIMEOUT: Duration = Duration::from_secs(2); const DEVICE_SYNC_INTERVAL: Duration = Duration::from_secs(2); const LOCAL_SEED_RECHECK_MS: i64 = 60 * 1000; -const MAX_LINE: usize = 8 * 1024 * 1024; /// Playback commands carry queue state but are useful only briefly and only /// to their addressed device. They must not inflate a new peer's catch-up. const PLAYBACK_COMMAND_TTL_MS: i64 = 5 * 60 * 1_000; @@ -94,319 +123,6 @@ struct StoredDevice { endpoint_ticket: String, } -#[derive(Debug, Clone, Serialize, Deserialize)] -struct InviteWire { - v: u16, - #[serde(rename = "t")] - ticket: String, - #[serde(rename = "d")] - device_id: String, - #[serde(rename = "i")] - invite_id: String, - #[serde(rename = "s")] - secret: String, - #[serde(rename = "e")] - expires_at_ms: i64, -} - -#[derive(Debug, Clone, Serialize, Deserialize)] -struct DeviceProfileWire { - device_id: String, - name: String, - client_version: String, - protocol_version: u16, - endpoint_id: String, - endpoint_ticket: String, - #[serde(default)] - revoked: bool, - #[serde(default)] - revoke_cutoff_seq: Option, - #[serde(default)] - updated_at_ms: i64, -} - -#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] -struct SyncedFedTrack { - item_id: String, - owner: String, - title: String, - #[serde(default)] - artist_names: Vec, - #[serde(default)] - featured_artist_names: Vec, - year: Option, - duration_seconds: Option, - content_id: String, - release_title: Option, - track_number: Option, - disc_number: Option, -} - -#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] -struct PlaybackTrack { - id: i64, - title: String, - track_number: Option, - disc_number: Option, - duration_seconds: f64, - #[serde(default)] - artist_names: Vec, - #[serde(default)] - featured_artist_names: Vec, - release_id: i64, - release_title: String, - release_year: Option, - #[serde(default)] - file_path: String, - content_id: Option, - audio_format: Option, - audio_bitrate: Option, - audio_sample_rate: Option, - audio_bit_depth: Option, - file_size_bytes: Option, - #[serde(default)] - play_count: i64, - #[serde(default)] - fed: Option, -} - -#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Default)] -#[serde(rename_all = "snake_case")] -enum PlaybackRepeat { - #[default] - Off, - One, - All, -} - -#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] -struct PlaybackStateWire { - #[serde(default)] - queue: Vec, - #[serde(default)] - queue_pos: usize, - playing: bool, - paused: bool, - #[serde(default)] - idle_since_ms: Option, - position_secs: f64, - #[serde(default)] - volume: u8, - shuffle: bool, - repeat: PlaybackRepeat, -} - -type PlaybackSnapshot = music_dht::playback::Snapshot; -type PlaybackCommand = music_dht::playback::Command; - -#[derive(Debug, Clone, Serialize, Deserialize)] -struct SyncOpWire { - op_id: String, - origin_device_id: String, - seq: i64, - hlc_ms: i64, - payload: SyncOpPayload, -} - -#[derive(Debug, Clone, Serialize, Deserialize)] -#[serde(tag = "kind", rename_all = "snake_case")] -enum SyncOpPayload { - TrackLikeSet { - content_id: String, - liked: bool, - #[serde(default)] - fed: Option, - }, - PlaylistCreated { - playlist_id: String, - title: String, - }, - PlaylistRenamed { - playlist_id: String, - title: String, - }, - PlaylistDeleted { - playlist_id: String, - }, - PlaylistTrackAdded { - playlist_id: String, - content_id: String, - position: i64, - #[serde(default)] - fed: Option, - }, - PlaylistTrackRemoved { - playlist_id: String, - content_id: String, - }, - DeviceProfileSet { - name: String, - client_version: String, - endpoint_ticket: String, - endpoint_id: String, - }, - DeviceTrusted { - target_device_id: String, - }, - DeviceRevoked { - target_device_id: String, - target_max_seq_seen: i64, - }, - PlaybackCommand { - target_device_id: String, - command: PlaybackCommand, - #[serde(default)] - authority: Option, - }, - ListenRecorded { - event: ListenEvent, - }, -} - -impl SyncOpPayload { - fn is_tombstone(&self) -> bool { - matches!( - self, - SyncOpPayload::TrackLikeSet { liked: false, .. } - | SyncOpPayload::PlaylistDeleted { .. } - | SyncOpPayload::PlaylistTrackRemoved { .. } - | SyncOpPayload::DeviceRevoked { .. } - ) - } -} - -#[derive(Debug, Clone, Default, Serialize, Deserialize)] -struct SyncSnapshot { - #[serde(default)] - likes: Vec, - #[serde(default)] - unlikes: Vec, - #[serde(default)] - playlists: Vec, - #[serde(default)] - deleted_playlists: Vec, - #[serde(default)] - removed_playlist_items: Vec, -} - -#[derive(Debug, Clone, Serialize, Deserialize)] -struct SnapshotLike { - content_id: String, - hlc_ms: i64, - op_id: String, - #[serde(default)] - fed: Option, -} - -#[derive(Debug, Clone, Serialize, Deserialize)] -struct SnapshotLikeTombstone { - content_id: String, - hlc_ms: i64, - op_id: String, -} - -#[derive(Debug, Clone, Serialize, Deserialize)] -struct SnapshotPlaylist { - playlist_id: String, - title: String, - hlc_ms: i64, - op_id: String, - #[serde(default)] - items: Vec, -} - -#[derive(Debug, Clone, Serialize, Deserialize)] -struct SnapshotPlaylistTombstone { - playlist_id: String, - hlc_ms: i64, - op_id: String, -} - -#[derive(Debug, Clone, Serialize, Deserialize)] -struct SnapshotPlaylistItem { - content_id: String, - position: i64, - hlc_ms: i64, - op_id: String, - #[serde(default)] - fed: Option, -} - -#[derive(Debug, Clone, Serialize, Deserialize)] -struct SnapshotPlaylistItemTombstone { - playlist_id: String, - content_id: String, - hlc_ms: i64, - op_id: String, -} - -#[derive(Debug, Clone, Serialize, Deserialize)] -#[serde(tag = "type", rename_all = "snake_case")] -enum WireMessage { - PairRequest { - invite_id: String, - secret: String, - profile: DeviceProfileWire, - #[serde(default)] - group_id: Option, - #[serde(default)] - group_active_devices: usize, - #[serde(default)] - devices: Vec, - vector: BTreeMap, - ops: Vec, - snapshot: SyncSnapshot, - #[serde(default)] - playback: Option, - }, - PairResponse { - accepted: bool, - #[serde(default)] - pending: bool, - #[serde(default)] - error: Option, - #[serde(default)] - group_id: Option, - #[serde(default)] - profile: Option, - #[serde(default)] - devices: Vec, - #[serde(default)] - vector: BTreeMap, - #[serde(default)] - ops: Vec, - #[serde(default)] - snapshot: SyncSnapshot, - #[serde(default)] - playback: Option, - }, - Hello { - group_id: String, - profile: DeviceProfileWire, - devices: Vec, - vector: BTreeMap, - ops: Vec, - snapshot: SyncSnapshot, - #[serde(default)] - playback: Option, - }, - SyncResponse { - accepted: bool, - #[serde(default)] - error: Option, - #[serde(default)] - devices: Vec, - #[serde(default)] - vector: BTreeMap, - #[serde(default)] - ops: Vec, - #[serde(default)] - snapshot: SyncSnapshot, - #[serde(default)] - playback: Option, - }, -} - enum PairAttempt { Accepted(String), Pending, @@ -443,8 +159,9 @@ pub async fn create_invite( invite_id, secret, expires_at_ms, + protocol_version: Some(PROTOCOL_VERSION), }; - encode_invite(&payload) + Ok(encode_invite(&payload)?) } pub async fn connect_invite( @@ -528,11 +245,13 @@ pub async fn answer_pairing( if let Some(row) = pending { let device_id: String = row.get("device_id"); enforce_single_user_binding(pool, user_id, &device_id).await?; + // The pending row has no version; the requester reports it again in + // the PairRequest it repeats until the answer arrives. let profile = DeviceProfileWire { device_id, name: row.get("name"), client_version: row.get("client_version"), - protocol_version: PROTOCOL_VERSION, + protocol_version: music_dht::device_sync::UNREPORTED_PROTOCOL_VERSION, endpoint_id: row.get("endpoint_id"), endpoint_ticket: row.get("endpoint_ticket"), revoked: false, @@ -552,9 +271,16 @@ pub async fn answer_pairing( } let devices_json: String = row.get("requester_group_devices_json"); let devices: Vec = serde_json::from_str(&devices_json)?; - apply_device_profiles(pool, user_id, &devices).await?; + apply_device_profiles(pool, user_id, &devices, Some(&profile.device_id)).await?; } - apply_device_profile(pool, user_id, &profile, true).await?; + apply_device_profile( + pool, + user_id, + &profile, + VersionReport::SelfReported(None), + true, + ) + .await?; record_local_op( pool, user_id, @@ -1074,16 +800,25 @@ pub async fn serve_peers( } } +/// Polls every trusted device of every account every +/// [`DEVICE_SYNC_INTERVAL`] (one exchange per device at a time). +/// +/// `first_round` turns `true` once the first complete listing of the +/// devices has been polled: every exchange in flight then has answered, +/// failed or timed out (30 s). The podcast publication reconciler waits for +/// it (see [`crate::podcasts::discovery::run`]). pub async fn sync_loop( pool: sqlx::PgPool, service: Arc, hub: Arc, transport_stats: Arc, + first_round: tokio::sync::watch::Sender, ) { let mut interval = tokio::time::interval(DEVICE_SYNC_INTERVAL); interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip); let mut polls = tokio::task::JoinSet::new(); let mut active = std::collections::HashMap::new(); + let mut first_round = FirstSyncRound::new(first_round); loop { tokio::select! { _ = interval.tick() => { @@ -1093,10 +828,15 @@ pub async fn sync_loop( Ok(users) => users, Err(error) => { tracing::warn!("device poll listing failed: {error:#}"); continue; } }; + let mut listed_all = true; for user_id in users { let devices = match active_remote_devices(&pool, user_id).await { Ok(devices) => devices, - Err(error) => { let _ = set_last_error(&pool, user_id, Some(&format!("{error:#}"))).await; continue; } + Err(error) => { + listed_all = false; + let _ = set_last_error(&pool, user_id, Some(&format!("{error:#}"))).await; + continue; + } }; for device in devices { let key = (user_id, device.device_id.clone()); @@ -1112,12 +852,17 @@ pub async fn sync_loop( active.insert(handle.id(), key); } } + if listed_all { + // Includes exchanges still running from an earlier tick. + first_round.started(active.keys().copied()); + } } Some(completed) = polls.join_next_with_id(), if !polls.is_empty() => { let (task, result) = match completed { Ok((task, result)) => (task, result), Err(error) => (error.id(), Err(anyhow::Error::from(error))), }; + first_round.finished(&task); if let Some((user_id, device_id)) = active.remove(&task) && let Err(error) = result { tracing::debug!(device = %device_id, "web fed device sync failed: {error:#}"); @@ -1128,6 +873,50 @@ pub async fn sync_loop( } } +/// Tracks the first device-sync round of a federation run for +/// [`sync_loop`]: the exchanges started by the first complete listing of the +/// trusted devices (and those still running then). The watch turns `true` +/// when the last of them ends, or at once when there are none, and stays +/// `true`. +pub(crate) struct FirstSyncRound { + pending: Option>, + done: tokio::sync::watch::Sender, +} + +impl FirstSyncRound { + pub(crate) fn new(done: tokio::sync::watch::Sender) -> Self { + Self { + pending: None, + done, + } + } + + /// The first complete listing started (or found running) `polls`; + /// later listings are ignored. + pub(crate) fn started(&mut self, polls: impl IntoIterator) { + if self.pending.is_some() || *self.done.borrow() { + return; + } + let pending: std::collections::HashSet = polls.into_iter().collect(); + if pending.is_empty() { + self.done.send_replace(true); + } else { + self.pending = Some(pending); + } + } + + /// An exchange ended (answered, failed or timed out). + pub(crate) fn finished(&mut self, poll: &K) { + if let Some(pending) = &mut self.pending + && pending.remove(poll) + && pending.is_empty() + { + self.pending = None; + self.done.send_replace(true); + } + } +} + pub async fn sync_once( pool: &sqlx::PgPool, service: Arc, @@ -1179,9 +968,13 @@ async fn try_connect_invite( let profile = own_profile(pool, user_id, user_name, &own_ticket).await?; let devices = device_profiles(pool, user_id).await?; let vector = vector(pool, user_id).await?; - let ops = ops_for_peer(pool, user_id, &invite.device_id).await?; - let snapshot = snapshot(pool, user_id).await?; - let playback = local_playback_snapshot(pool, Arc::clone(&hub), user_id, &identity).await; + // The request is sent before the inviter reports a version: gate it by + // the invite (version 1 for invites from older clients). + let inviter_version = invite.inviter_protocol_version(); + let ops = ops_for_peer(pool, user_id, &invite.device_id, inviter_version).await?; + let snapshot = snapshot(pool, user_id).await?.for_peer(inviter_version); + let playback = + local_playback_snapshot(pool, Arc::clone(&hub), user_id, &identity, inviter_version).await; let mut stream = service.open_stream(peer, SYNC_ALPN).await?; record_stream_transport(&transport_stats, "device-sync", "outbound", "open", &stream); write_msg( @@ -1220,7 +1013,14 @@ async fn try_connect_invite( set_group_id(pool, user_id, &group_id).await?; if let Some(profile) = profile { enforce_single_user_binding(pool, user_id, &profile.device_id).await?; - apply_device_profile(pool, user_id, &profile, true).await?; + apply_device_profile( + pool, + user_id, + &profile, + VersionReport::SelfReported(Some(profile.protocol_version)), + true, + ) + .await?; if let Some(playback) = playback { apply_playback_snapshot( Arc::clone(&hub), @@ -1232,7 +1032,7 @@ async fn try_connect_invite( .await?; } } - apply_device_profiles(pool, user_id, &devices).await?; + apply_device_profiles(pool, user_id, &devices, Some(&invite.device_id)).await?; apply_snapshot(pool, user_id, snapshot).await?; apply_ops(pool, Arc::clone(&hub), user_id, ops).await?; record_local_op( @@ -1446,7 +1246,14 @@ async fn handle_pair_request( let own_ticket = service.ticket().await?.to_string(); let own_profile = own_profile(pool, user_id, "", &own_ticket).await?; - apply_device_profile(pool, user_id, &profile, true).await?; + apply_device_profile( + pool, + user_id, + &profile, + VersionReport::SelfReported(Some(profile.protocol_version)), + true, + ) + .await?; if let Some(playback) = playback { apply_playback_snapshot( Arc::clone(&hub), @@ -1459,7 +1266,13 @@ async fn handle_pair_request( } apply_snapshot(pool, user_id, incoming_snapshot).await?; apply_ops(pool, Arc::clone(&hub), user_id, ops).await?; - apply_device_profiles(pool, user_id, &requester_group_devices).await?; + apply_device_profiles( + pool, + user_id, + &requester_group_devices, + Some(&profile.device_id), + ) + .await?; note_peer_vector(pool, user_id, &profile.device_id, &incoming_vector).await?; sqlx::query("UPDATE furumusic__fed_device_invite SET used_at_ms = $2 WHERE invite_id = $1") .bind(&invite_id) @@ -1473,11 +1286,12 @@ async fn handle_pair_request( ) .await?; + let peer_version = stored_protocol_version(pool, user_id, &profile.device_id).await?; let devices = device_profiles(pool, user_id).await?; let vector = vector(pool, user_id).await?; - let ops = ops_for_peer(pool, user_id, &profile.device_id).await?; - let snapshot = snapshot(pool, user_id).await?; - let playback = local_playback_snapshot(pool, hub, user_id, &identity).await; + let ops = ops_for_peer(pool, user_id, &profile.device_id, peer_version).await?; + let snapshot = snapshot(pool, user_id).await?.for_peer(peer_version); + let playback = local_playback_snapshot(pool, hub, user_id, &identity, peer_version).await; write_msg( &mut stream, &WireMessage::PairResponse { @@ -1522,8 +1336,15 @@ async fn handle_hello( return write_sync_reject(stream, "sync group mismatch").await; } profile.endpoint_id = stream.peer_id.to_string(); - apply_device_profile(pool, user_id, &profile, false).await?; - apply_device_profiles(pool, user_id, &devices).await?; + apply_device_profile( + pool, + user_id, + &profile, + VersionReport::SelfReported(Some(profile.protocol_version)), + false, + ) + .await?; + apply_device_profiles(pool, user_id, &devices, Some(&profile.device_id)).await?; if let Some(playback) = playback { apply_playback_snapshot( Arc::clone(&hub), @@ -1555,10 +1376,11 @@ async fn handle_hello( let own_profile = own_profile(pool, user_id, "", &own_ticket).await?; let mut devices = device_profiles(pool, user_id).await?; devices.push(own_profile); + let peer_version = stored_protocol_version(pool, user_id, &profile.device_id).await?; let vector = vector(pool, user_id).await?; - let ops = ops_for_peer(pool, user_id, &profile.device_id).await?; - let snapshot = snapshot(pool, user_id).await?; - let playback = local_playback_snapshot(pool, hub, user_id, &identity).await; + let ops = ops_for_peer(pool, user_id, &profile.device_id, peer_version).await?; + let snapshot = snapshot(pool, user_id).await?.for_peer(peer_version); + let playback = local_playback_snapshot(pool, hub, user_id, &identity, peer_version).await; write_msg( &mut stream, &WireMessage::SyncResponse { @@ -1589,11 +1411,13 @@ async fn sync_device( let own_ticket = service.ticket().await?.to_string(); let identity = ensure_identity(pool, user_id, "").await?; let profile = own_profile(pool, user_id, "", &own_ticket).await?; + let peer_version = stored_protocol_version(pool, user_id, &device.device_id).await?; let devices = device_profiles(pool, user_id).await?; let vector = vector(pool, user_id).await?; - let ops = ops_for_peer(pool, user_id, &device.device_id).await?; - let snapshot = snapshot(pool, user_id).await?; - let playback = local_playback_snapshot(pool, Arc::clone(&hub), user_id, &identity).await; + let ops = ops_for_peer(pool, user_id, &device.device_id, peer_version).await?; + let snapshot = snapshot(pool, user_id).await?.for_peer(peer_version); + let playback = + local_playback_snapshot(pool, Arc::clone(&hub), user_id, &identity, peer_version).await; let mut stream = service.open_stream(peer, SYNC_ALPN).await?; record_stream_transport(&transport_stats, "device-sync", "outbound", "open", &stream); write_msg( @@ -1624,7 +1448,9 @@ async fn sync_device( playback, .. } => { - apply_device_profiles(pool, user_id, &devices).await?; + // A SyncResponse has no profile: the responder's own entry in + // `devices` is its version report. + apply_device_profiles(pool, user_id, &devices, Some(&device.device_id)).await?; if let Some(playback) = playback { apply_playback_snapshot( Arc::clone(&hub), @@ -1930,6 +1756,14 @@ async fn own_profile( ) -> Result { let identity = ensure_identity(pool, user_id, user_name).await?; let endpoint_id = ticket_endpoint_id(endpoint_ticket).unwrap_or_default(); + let previous_version: Option = sqlx::query_scalar( + "SELECT protocol_version FROM furumusic__fed_device + WHERE user_id = $1 AND device_id = $2", + ) + .bind(user_id) + .bind(&identity.device_id) + .fetch_optional(pool) + .await?; let now = now_ms(); sqlx::query( "INSERT INTO furumusic__fed_device @@ -1954,6 +1788,23 @@ async fn own_profile( .bind(now) .execute(pool) .await?; + if previous_version.is_some_and(|version| version < i32::from(PROTOCOL_VERSION)) { + // First start of a new protocol version: announce it in the log too, + // so devices that only see this device's operations through relays + // learn the version (`profile_protocol_version`). + record_local_op( + pool, + user_id, + SyncOpPayload::DeviceProfileSet { + name: identity.name.clone(), + client_version: CLIENT_VERSION.to_string(), + endpoint_ticket: endpoint_ticket.to_string(), + endpoint_id: endpoint_id.clone(), + protocol_version: Some(PROTOCOL_VERSION), + }, + ) + .await?; + } Ok(DeviceProfileWire { device_id: identity.device_id, name: identity.name, @@ -1994,20 +1845,7 @@ async fn record_local_op( ); } - let now = now_ms(); - let row = sqlx::query( - "UPDATE furumusic__fed_device_identity - SET local_seq = local_seq + 1, - last_hlc_ms = GREATEST($2, last_hlc_ms + 1) - WHERE user_id = $1 - RETURNING local_seq, last_hlc_ms", - ) - .bind(user_id) - .bind(now) - .fetch_one(pool) - .await?; - let seq: i64 = row.get("local_seq"); - let hlc_ms: i64 = row.get("last_hlc_ms"); + let (seq, hlc_ms) = reserve_local_stamp(pool, user_id).await?; let op_id = format!("{}:{seq}", identity.device_id); let op = SyncOpWire { op_id, @@ -2016,7 +1854,122 @@ async fn record_local_op( hlc_ms, payload, }; + insert_op(pool, user_id, &op).await?; + advance_vector(pool, user_id, &op.origin_device_id, op.seq).await?; + let _ = apply_op(pool, Arc::new(PlayerDeviceHub::default()), user_id, &op).await?; + Ok(()) +} + +/// Reserves the next local `(seq, hlc_ms)` of the account's web device. +/// +/// `hlc_ms` is the wall clock, but always past the previous local +/// operation. Inside a transaction the identity row stays locked until it +/// ends, so local operations of one account are stamped in commit order. +async fn reserve_local_stamp<'e>( + db: impl sqlx::PgExecutor<'e>, + user_id: i64, +) -> Result<(i64, i64)> { + let row = sqlx::query( + "UPDATE furumusic__fed_device_identity + SET local_seq = local_seq + 1, + last_hlc_ms = GREATEST($2, last_hlc_ms + 1) + WHERE user_id = $1 + RETURNING local_seq, last_hlc_ms", + ) + .bind(user_id) + .bind(now_ms()) + .fetch_one(db) + .await?; + Ok((row.get("local_seq"), row.get("last_hlc_ms"))) +} + +async fn advance_vector<'e>( + db: impl sqlx::PgExecutor<'e>, + user_id: i64, + device_id: &str, + seq: i64, +) -> Result<()> { sqlx::query( + "INSERT INTO furumusic__fed_sync_vector (user_id, device_id, max_seq) + VALUES ($1, $2, $3) + ON CONFLICT (user_id, device_id) DO UPDATE SET + max_seq = GREATEST(furumusic__fed_sync_vector.max_seq, EXCLUDED.max_seq)", + ) + .bind(user_id) + .bind(device_id) + .bind(seq) + .execute(db) + .await?; + Ok(()) +} + +/// Device id of the account's web device, creating the identity on first +/// use. Local podcast actions stamp their operations with it. +pub(crate) async fn local_device_id(pool: &sqlx::PgPool, user_id: i64) -> Result { + Ok(ensure_identity(pool, user_id, "").await?.device_id) +} + +/// Appends a local operation of `device_id` (the account's web device, see +/// [`local_device_id`]) to the sync log inside the caller's transaction. +/// +/// The stamp (`seq`, `hlc_ms`, `op_id = :`) is reserved here; +/// `make` receives the `op_id`. The caller applies the returned operation +/// itself in the same transaction, so the log and the materialized state +/// change together (the podcast store does this for its local actions). +pub(crate) async fn append_local_op( + conn: &mut sqlx::PgConnection, + user_id: i64, + device_id: &str, + make: impl FnOnce(&str) -> SyncOpPayload, +) -> Result { + let (seq, hlc_ms) = reserve_local_stamp(&mut *conn, user_id).await?; + let op_id = format!("{device_id}:{seq}"); + let payload = make(&op_id); + let op = SyncOpWire { + op_id, + origin_device_id: device_id.to_string(), + seq, + hlc_ms, + payload, + }; + anyhow::ensure!( + insert_op(&mut *conn, user_id, &op).await?, + "local operation {} already exists", + op.op_id + ); + advance_vector(&mut *conn, user_id, device_id, seq).await?; + Ok(op) +} + +async fn apply_ops( + pool: &sqlx::PgPool, + hub: Arc, + user_id: i64, + ops: Vec, +) -> Result<()> { + for op in ops { + if !should_accept_op(pool, user_id, &op).await? { + continue; + } + // Podcast operations that break the contract are skipped but still + // acknowledged (the vector below advances), so the log keeps moving. + if let Err(error) = op.validate_podcast() { + tracing::debug!(op_id = %op.op_id, "skipped invalid podcast operation: {error}"); + } else if insert_op(pool, user_id, &op).await? { + let _ = apply_op(pool, Arc::clone(&hub), user_id, &op).await?; + } + advance_vector(pool, user_id, &op.origin_device_id, op.seq).await?; + } + Ok(()) +} + +/// Appends an operation to the log; `false` when it was already stored. +async fn insert_op<'e>( + db: impl sqlx::PgExecutor<'e>, + user_id: i64, + op: &SyncOpWire, +) -> Result { + let inserted = sqlx::query( "INSERT INTO furumusic__fed_sync_ops (user_id, op_id, origin_device_id, seq, kind, payload_json, hlc_ms, received_at_ms, tombstone) @@ -2032,68 +1985,10 @@ async fn record_local_op( .bind(op.hlc_ms) .bind(now_ms()) .bind(op.payload.is_tombstone()) - .execute(pool) - .await?; - sqlx::query( - "INSERT INTO furumusic__fed_sync_vector (user_id, device_id, max_seq) - VALUES ($1, $2, $3) - ON CONFLICT (user_id, device_id) DO UPDATE SET - max_seq = GREATEST(furumusic__fed_sync_vector.max_seq, EXCLUDED.max_seq)", - ) - .bind(user_id) - .bind(&op.origin_device_id) - .bind(op.seq) - .execute(pool) - .await?; - let _ = apply_op(pool, Arc::new(PlayerDeviceHub::default()), user_id, &op).await?; - Ok(()) -} - -async fn apply_ops( - pool: &sqlx::PgPool, - hub: Arc, - user_id: i64, - ops: Vec, -) -> Result<()> { - for op in ops { - if !should_accept_op(pool, user_id, &op).await? { - continue; - } - let inserted = sqlx::query( - "INSERT INTO furumusic__fed_sync_ops - (user_id, op_id, origin_device_id, seq, kind, payload_json, hlc_ms, - received_at_ms, tombstone) - VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9) - ON CONFLICT (user_id, op_id) DO NOTHING", - ) - .bind(user_id) - .bind(&op.op_id) - .bind(&op.origin_device_id) - .bind(op.seq) - .bind(payload_kind(&op.payload)) - .bind(serde_json::to_value(&op.payload)?) - .bind(op.hlc_ms) - .bind(now_ms()) - .bind(op.payload.is_tombstone()) - .execute(pool) - .await? - .rows_affected(); - if inserted > 0 { - let _ = apply_op(pool, Arc::clone(&hub), user_id, &op).await?; - } - sqlx::query( - "INSERT INTO furumusic__fed_sync_vector (user_id, device_id, max_seq) - VALUES ($1, $2, $3) - ON CONFLICT (user_id, device_id) DO UPDATE SET - max_seq = GREATEST(furumusic__fed_sync_vector.max_seq, EXCLUDED.max_seq)", - ) - .bind(user_id) - .bind(&op.origin_device_id) - .bind(op.seq) - .execute(pool) - .await?; - } - Ok(()) + .execute(db) + .await? + .rows_affected(); + Ok(inserted > 0) } async fn should_accept_op(pool: &sqlx::PgPool, user_id: i64, op: &SyncOpWire) -> Result { @@ -2201,19 +2096,29 @@ async fn apply_op( client_version, endpoint_ticket, endpoint_id, + protocol_version, } => { + // The origin's own report: never substitute this server's version. let profile = DeviceProfileWire { device_id: op.origin_device_id.clone(), name: name.clone(), client_version: client_version.clone(), - protocol_version: PROTOCOL_VERSION, + protocol_version: protocol_version + .unwrap_or(music_dht::device_sync::UNREPORTED_PROTOCOL_VERSION), endpoint_id: endpoint_id.clone(), endpoint_ticket: endpoint_ticket.clone(), revoked: false, revoke_cutoff_seq: None, updated_at_ms: op.hlc_ms, }; - apply_device_profile(pool, user_id, &profile, false).await?; + apply_device_profile( + pool, + user_id, + &profile, + VersionReport::SelfReported(*protocol_version), + false, + ) + .await?; Ok(false) } SyncOpPayload::DeviceTrusted { target_device_id } => { @@ -2240,6 +2145,25 @@ async fn apply_op( SyncOpPayload::ListenRecorded { event } => { apply_listen_event(pool, user_id, &op.origin_device_id, event).await } + SyncOpPayload::PodcastSubscriptionSet { .. } + | SyncOpPayload::PodcastShareSet { .. } + | SyncOpPayload::PodcastFeedAliasesAdded { .. } + | SyncOpPayload::PodcastProgressSet { .. } + | SyncOpPayload::PodcastListenRecorded { .. } => { + // The operation itself is relayed from the log; the store keeps + // the registers (and the catalog rows of its episode reference). + match PodcastStore::new(pool.clone()) + .apply_remote_op(user_id, op) + .await? + { + ApplyOutcome::Changed => Ok(true), + ApplyOutcome::Rejected(reason) => { + tracing::debug!(op_id = %op.op_id, "skipped invalid podcast operation: {reason}"); + Ok(false) + } + ApplyOutcome::Unchanged | ApplyOutcome::Ignored => Ok(false), + } + } } } @@ -2605,6 +2529,17 @@ async fn apply_playlist_item_state( } async fn apply_snapshot(pool: &sqlx::PgPool, user_id: i64, snapshot: SyncSnapshot) -> Result<()> { + let podcasts = PodcastStore::new(pool.clone()) + .apply_snapshot(user_id, &snapshot) + .await?; + if podcasts.ignored_over_limits { + tracing::debug!("ignored podcast snapshot sections above the contract limits"); + } else if podcasts.rejected > 0 { + tracing::debug!( + rejected = podcasts.rejected, + "skipped invalid podcast snapshot records" + ); + } for like in snapshot.likes { apply_like_state( pool, @@ -2684,21 +2619,75 @@ async fn apply_snapshot(pool: &sqlx::PgPool, user_id: i64, snapshot: SyncSnapsho Ok(()) } +/// How the protocol version in a device profile became known. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum VersionReport { + /// The device reported its own version (`None`: an older client left it + /// out). Replaces the stored version. + SelfReported(Option), + /// Another device relayed the profile. Only raises the stored version: + /// v1–v3 relays substitute their own constant. + Relayed(u16), +} + +impl VersionReport { + fn resolve(self, stored: Option) -> u16 { + match self { + VersionReport::SelfReported(reported) => profile_protocol_version(reported, stored), + VersionReport::Relayed(relayed) => relayed_protocol_version(relayed, stored), + } + } + + /// Report carried by an entry of a `devices` list sent by `sender`. + fn for_listed(profile: &DeviceProfileWire, sender: Option<&str>) -> Self { + if sender == Some(profile.device_id.as_str()) { + VersionReport::SelfReported(Some(profile.protocol_version)) + } else { + VersionReport::Relayed(profile.protocol_version) + } + } +} + +/// Stores the profiles of a `devices` list. The entry of the authenticated +/// `sender` is its own version report; the others are relayed. async fn apply_device_profiles( pool: &sqlx::PgPool, user_id: i64, devices: &[DeviceProfileWire], + sender: Option<&str>, ) -> Result<()> { for profile in devices { - apply_device_profile(pool, user_id, profile, false).await?; + let report = VersionReport::for_listed(profile, sender); + apply_device_profile(pool, user_id, profile, report, false).await?; } Ok(()) } +/// Version gating input for a peer: the version it reported itself. +async fn stored_protocol_version( + pool: &sqlx::PgPool, + user_id: i64, + device_id: &str, +) -> Result { + let stored: Option = sqlx::query_scalar( + "SELECT protocol_version FROM furumusic__fed_device + WHERE user_id = $1 AND device_id = $2", + ) + .bind(user_id) + .bind(device_id) + .fetch_optional(pool) + .await?; + Ok(profile_protocol_version( + None, + stored.and_then(|value| u16::try_from(value).ok()), + )) +} + async fn apply_device_profile( pool: &sqlx::PgPool, user_id: i64, profile: &DeviceProfileWire, + version: VersionReport, trusted: bool, ) -> Result<()> { let identity = ensure_identity(pool, user_id, "").await?; @@ -2706,6 +2695,15 @@ async fn apply_device_profile( return Ok(()); } enforce_single_user_binding(pool, user_id, &profile.device_id).await?; + let stored: Option = sqlx::query_scalar( + "SELECT protocol_version FROM furumusic__fed_device + WHERE user_id = $1 AND device_id = $2", + ) + .bind(user_id) + .bind(&profile.device_id) + .fetch_optional(pool) + .await?; + let protocol_version = version.resolve(stored.and_then(|value| u16::try_from(value).ok())); let trusted_at_ms = trusted.then_some(now_ms()).or(Some(profile.updated_at_ms)); sqlx::query( "INSERT INTO furumusic__fed_device @@ -2743,7 +2741,7 @@ async fn apply_device_profile( .bind(&profile.device_id) .bind(&profile.name) .bind(&profile.client_version) - .bind(profile.protocol_version as i32) + .bind(i32::from(protocol_version)) .bind(&profile.endpoint_id) .bind(&profile.endpoint_ticket) .bind(trusted_at_ms) @@ -2848,6 +2846,21 @@ async fn apply_playback_command( let origin = &op.origin_device_id; let op_id = &op.op_id; let identity = ensure_identity(pool, user_id, "").await?; + // A peer below v4 does not know podcast items: its state would silently + // drop them from the local queue. + let peer_version = stored_protocol_version(pool, user_id, origin).await?; + let local = hub + .playback_state_json_for_commands(user_id) + .map(|state| browser_state_shadow(&state)) + .unwrap_or_else(|| queue_items::shadow_wire_state(&[], 0)); + if !playback_command_allowed_from_peer(&local, peer_version) { + tracing::info!( + origin = %short_id(origin), + peer_version, + "ignored playback command from a device without podcast support" + ); + return Ok(()); + } let Some(authority) = authority else { return Ok(()); }; @@ -3007,8 +3020,15 @@ pub async fn record_web_playback_command( current_state: Option, ) -> Result { ensure_web_playback_target(pool, user_id, target_device_id).await?; - let next_state = browser_state_for_command(command, payload, current_state)?; - let wire = playback_state_from_browser_json(pool, next_state.clone()).await?; + let mut next_state = browser_state_for_command(command, payload, current_state)?; + let wire = playback_state_from_browser_json(pool, user_id, next_state.clone()).await?; + // Show the resume position that was sent, not the 0 the browser picked. + if let Some(object) = next_state.as_object_mut() + && object.remove(queue_items::POSITION_UNKNOWN_KEY).is_some() + { + object.insert("position_seconds".to_string(), json_f64(wire.position_secs)); + } + ensure_playback_supported_by(pool, user_id, target_device_id, &wire).await?; record_local_op( pool, user_id, @@ -3036,7 +3056,9 @@ pub async fn record_web_active_transfer( state: serde_json::Value, ) -> Result<()> { ensure_web_playback_target(pool, user_id, target_device_id).await?; - let wire = playback_state_from_browser_json(pool, state).await?; + let wire = playback_state_from_browser_json(pool, user_id, state).await?; + // Refuse instead of stripping podcast episodes from the queue. + ensure_playback_supported_by(pool, user_id, target_device_id, &wire).await?; let target_name = web_playback_target_name(pool, user_id, target_device_id).await?; let identity = ensure_identity(pool, user_id, "").await?; with_playback_engine(pool, user_id, &identity, |engine| { @@ -3081,6 +3103,17 @@ pub async fn record_web_active_transfer( Ok(()) } +/// Checks before a handoff that the target can receive the queue. +pub async fn check_web_transfer( + pool: &sqlx::PgPool, + user_id: i64, + target_device_id: &str, + state: serde_json::Value, +) -> Result<()> { + let wire = playback_state_from_browser_json(pool, user_id, state).await?; + ensure_playback_supported_by(pool, user_id, target_device_id, &wire).await +} + pub async fn record_web_active_takeover( pool: &sqlx::PgPool, user_id: i64, @@ -3089,7 +3122,7 @@ pub async fn record_web_active_takeover( ) -> Result<()> { ensure_web_playback_target(pool, user_id, previous_device_id).await?; let identity = ensure_identity(pool, user_id, "").await?; - let wire = playback_state_from_browser_json(pool, state).await?; + let wire = playback_state_from_browser_json(pool, user_id, state).await?; with_playback_engine(pool, user_id, &identity, |engine| { engine.transfer(&identity.device_id, playback_clock()) }) @@ -3111,6 +3144,28 @@ pub async fn record_web_active_takeover( Ok(()) } +/// Refuses to hand a queue with podcast items to a device below v4. +async fn ensure_playback_supported_by( + pool: &sqlx::PgPool, + user_id: i64, + device_id: &str, + state: &PlaybackStateWire, +) -> Result<()> { + let version = stored_protocol_version(pool, user_id, device_id).await?; + if !playback_allowed_for_peer(state, version) { + let name = web_playback_target_name(pool, user_id, device_id).await?; + anyhow::bail!( + "{name} needs an update to play podcast episodes. Update Furumi on that device, or remove the episodes from the queue." + ); + } + Ok(()) +} + +/// Wire-shaped view of a browser playback state for the shared gating helpers. +fn browser_state_shadow(state: &serde_json::Value) -> PlaybackStateWire { + queue_items::shadow_wire_state(&browser_tracks(state), browser_index(state)) +} + async fn ensure_web_playback_target( pool: &sqlx::PgPool, user_id: i64, @@ -3166,6 +3221,11 @@ fn browser_state_for_command( current_state.unwrap_or_else(empty_browser_playback_state) }; normalize_browser_playback_state(&mut state); + // Only a transfer carries the sender's knowledge of the position; the + // commands below decide for themselves. + if command != "transfer_state" { + browser_object_mut(&mut state).remove(queue_items::POSITION_UNKNOWN_KEY); + } match command { "play_track" | "play_from_index" | "transfer_state" => { @@ -3175,6 +3235,11 @@ fn browser_state_for_command( "paused", browser_bool(&payload, "paused").unwrap_or(false), ); + if command != "transfer_state" { + // The browser always starts the item at 0. For a podcast + // episode, ordinary Play resumes from its progress register. + mark_browser_position_unknown(&mut state); + } } "pause" => set_browser_bool(&mut state, "paused", true), "resume" => { @@ -3392,11 +3457,20 @@ fn advance_browser_state(state: &mut serde_json::Value, payload: &serde_json::Va set_browser_tracks(state, tracks, next); set_browser_number(state, "position_seconds", 0.0); set_browser_bool(state, "paused", false); + if next != current { + mark_browser_position_unknown(state); + } } else { set_browser_bool(state, "paused", true); } } +/// The current item changed without its position: a podcast episode resumes +/// from its progress register (see `playback_state_from_browser_json`). +fn mark_browser_position_unknown(state: &mut serde_json::Value) { + set_browser_bool(state, queue_items::POSITION_UNKNOWN_KEY, true); +} + fn rewind_browser_state(state: &mut serde_json::Value) { let tracks = browser_tracks(state); if tracks.is_empty() { @@ -3427,6 +3501,9 @@ fn rewind_browser_state(state: &mut serde_json::Value) { if let Some(previous) = previous { set_browser_tracks(state, tracks, previous); set_browser_bool(state, "paused", false); + if previous != current { + mark_browser_position_unknown(state); + } } set_browser_number(state, "position_seconds", 0.0); } @@ -3554,11 +3631,14 @@ async fn web_playback_payload( })) } +/// Playback snapshot for a peer, or `None` when the queue has podcast items +/// and the peer is below v4 (`playback_allowed_for_peer`): never strip them. async fn local_playback_snapshot( pool: &sqlx::PgPool, hub: Arc, user_id: i64, identity: &Identity, + peer_version: u16, ) -> Option { let coordination = coordinate_web_output(pool, Arc::clone(&hub), user_id, identity) .await @@ -3570,7 +3650,12 @@ async fn local_playback_snapshot( let state = hub .playback_state_json_for_commands(user_id) .unwrap_or_else(|| serde_json::json!({})); - let wire = playback_state_from_browser_json(pool, state).await.ok()?; + let wire = playback_state_from_browser_json(pool, user_id, state) + .await + .ok()?; + if !playback_allowed_for_peer(&wire, peer_version) { + return None; + } Some(PlaybackSnapshot { device_id: identity.device_id.clone(), device_name: identity.name.clone(), @@ -3694,7 +3779,101 @@ fn playback_clock() -> u64 { .as_millis() as u64 } +/// Wire playback state of a browser state. +/// +/// When the browser does not know the position of its current item +/// ([`queue_items::POSITION_UNKNOWN_KEY`]) and that item is a podcast +/// episode, the position comes from its synced progress register: handing +/// 0 to the next device would be written there as an explicit action and +/// restart the episode everywhere. async fn playback_state_from_browser_json( + pool: &sqlx::PgPool, + user_id: i64, + state: serde_json::Value, +) -> Result { + let unknown = browser_bool(&state, queue_items::POSITION_UNKNOWN_KEY) == Some(true); + let mut wire = browser_json_to_wire(pool, state).await?; + if let Some(episode_id) = episode_needing_resume_position(unknown, &wire) { + let record = PodcastStore::new(pool.clone()) + .progress(user_id, &episode_id) + .await?; + apply_resume_position(&mut wire, record.as_ref()); + } + let episodes: Vec = wire + .queue + .iter() + .filter_map(PlaybackTrack::podcast_episode_ref) + .map(|episode| episode.episode_id.clone()) + .collect(); + if !episodes.is_empty() { + let generations = PodcastStore::new(pool.clone()) + .progress_generations(user_id, &episodes) + .await?; + apply_progress_generations(&mut wire, &generations); + } + Ok(wire) +} + +/// Raises `PodcastEpisodeRef::progress_generation` of every episode in the +/// queue to the generation of its progress register here. +/// +/// The browser keeps the hint its queue item came with, which can be older +/// than what this account has seen since (its own checkpoints, operations +/// from other devices). The device taking over writes its explicit action +/// above this value (frid `PODCASTS.md`, "Handoff"). +fn apply_progress_generations( + state: &mut PlaybackStateWire, + generations: &std::collections::HashMap, +) { + for track in &mut state.queue { + if let Some(PlaybackSource::PodcastEpisode(episode)) = track.source.as_mut() + && let Some(generation) = generations.get(&episode.episode_id) + { + episode.progress_generation = Some( + episode + .progress_generation + .map_or(*generation, |seen| seen.max(*generation)), + ); + } + } +} + +/// The current episode whose position comes from its progress register: +/// only when the browser said it does not know the position. Known +/// positions (handoffs, seeks, playing) and music are never replaced. +fn episode_needing_resume_position(unknown: bool, wire: &PlaybackStateWire) -> Option { + if !unknown { + return None; + } + wire.queue + .get(wire.queue_pos) + .and_then(PlaybackTrack::podcast_episode_ref) + .map(|episode| episode.episode_id.clone()) +} + +/// Sets the position of the current podcast episode to where ordinary Play +/// resumes it (`ProgressRecord::resume_position_ms`, 0 without a register), +/// and records the register's generation as seen for the handoff. +fn apply_resume_position(state: &mut PlaybackStateWire, record: Option<&ProgressRecord>) { + let Some(track) = state.queue.get_mut(state.queue_pos) else { + return; + }; + let Some(PlaybackSource::PodcastEpisode(episode)) = track.source.as_mut() else { + return; + }; + let Some(record) = record.filter(|record| record.episode_id == episode.episode_id) else { + state.position_secs = 0.0; + return; + }; + state.position_secs = record.resume_position_ms().max(0) as f64 / 1000.0; + episode.progress_generation = Some( + episode + .progress_generation + .map_or(record.generation, |seen| seen.max(record.generation)), + ); +} + +async fn browser_json_to_wire( pool: &sqlx::PgPool, state: serde_json::Value, ) -> Result { @@ -3703,19 +3882,27 @@ async fn playback_state_from_browser_json( .and_then(serde_json::Value::as_array) .cloned() .unwrap_or_default(); + let mut queue_pos = state + .get("index") + .and_then(serde_json::Value::as_i64) + .unwrap_or(0) + .max(0) as usize; let mut queue = Vec::new(); - for track in tracks_value { - if let Some(track_id) = track.get("id").and_then(serde_json::Value::as_i64) { - queue.push(playback_track_from_track_json(pool, track_id, &track).await?); + for (index, track) in tracks_value.iter().enumerate() { + // Items that are not library tracks keep their verbatim wire item, + // including the explicit source (podcast episodes, placeholders). + if let Some(wire) = queue_items::wire_track_from_browser_json(track) { + queue.push(wire); + } else if let Some(track_id) = track.get("id").and_then(serde_json::Value::as_i64) { + queue.push(playback_track_from_track_json(pool, track_id, track).await?); + } else if index < queue_pos { + // A malformed entry is skipped; keep the position on its item. + queue_pos -= 1; } } Ok(PlaybackStateWire { queue, - queue_pos: state - .get("index") - .and_then(serde_json::Value::as_i64) - .unwrap_or(0) - .max(0) as usize, + queue_pos, playing: state.get("track").is_some_and(|track| !track.is_null()), paused: state .get("paused") @@ -3746,6 +3933,8 @@ async fn playback_state_from_browser_json( "all" => PlaybackRepeat::All, _ => PlaybackRepeat::Off, }, + // Reserved; the web player has no variable speed. + playback_rate: None, }) } @@ -3754,6 +3943,15 @@ async fn playback_track_from_track_json( track_id: i64, track: &serde_json::Value, ) -> Result { + let content_id = match track + .get("content_id") + .and_then(serde_json::Value::as_str) + .and_then(music_dht::normalize_content_id) + { + Some(content_id) => Some(content_id), + None if track_id > 0 => track_content_id(pool, track_id).await?, + None => None, + }; Ok(PlaybackTrack { id: track_id, title: track @@ -3789,15 +3987,7 @@ async fn playback_track_from_track_json( .and_then(serde_json::Value::as_i64) .map(|value| value as i32), file_path: String::new(), - content_id: track - .get("content_id") - .and_then(serde_json::Value::as_str) - .and_then(music_dht::normalize_content_id) - .or(if track_id > 0 { - track_content_id(pool, track_id).await? - } else { - None - }), + content_id, audio_format: track .get("audio_format") .and_then(serde_json::Value::as_str) @@ -3819,6 +4009,7 @@ async fn playback_track_from_track_json( .and_then(serde_json::Value::as_i64), play_count: 0, fed: None, + source: None, }) } @@ -3836,31 +4027,50 @@ fn names_from_track_json(track: &serde_json::Value, field: &str) -> Vec .unwrap_or_default() } -async fn resolve_playback_tracks( +/// Browser JSON of wire queue items, in order and without dropping any. +/// +/// Music resolved by content id becomes the library track; everything else +/// (unresolved music, podcast episodes, Jam placeholders) becomes an +/// external item that keeps the verbatim wire DTO. +pub(crate) async fn resolve_playback_tracks( pool: &sqlx::PgPool, tracks: &[PlaybackTrack], ) -> Result> { - let mut out = Vec::new(); + let mut out = Vec::with_capacity(tracks.len()); for (index, track) in tracks.iter().enumerate() { - let content_id = track - .content_id - .as_deref() - .or_else(|| track.fed.as_ref().map(|fed| fed.content_id.as_str())) - .and_then(music_dht::normalize_content_id); - let Some(content_id) = content_id else { - out.push(placeholder_playback_track(track, index, None)); - continue; - }; - if let Some(track_json) = web_track_json_by_content_id(pool, &content_id).await? { - out.push(track_json); - } else { - tracing::debug!(content_id, title = %track.title, "fed playback track is not available locally on web"); - out.push(placeholder_playback_track(track, index, Some(&content_id))); - } + out.push(resolve_playback_track(pool, track, index).await?); } Ok(out) } +/// Browser JSON of one wire queue item at queue position `index`. +pub(crate) async fn resolve_playback_track( + pool: &sqlx::PgPool, + track: &PlaybackTrack, + index: usize, +) -> Result { + if track.requires_podcast_support() { + return Ok(queue_items::external_track_json(track, index, None)); + } + let content_id = track + .content_id + .as_deref() + .or_else(|| track.fed.as_ref().map(|fed| fed.content_id.as_str())) + .and_then(music_dht::normalize_content_id); + let Some(content_id) = content_id else { + return Ok(queue_items::external_track_json(track, index, None)); + }; + if let Some(track_json) = web_track_json_by_content_id(pool, &content_id).await? { + return Ok(track_json); + } + tracing::debug!(content_id, title = %track.title, "fed playback track is not available locally on web"); + Ok(queue_items::external_track_json( + track, + index, + Some(&content_id), + )) +} + async fn web_track_json_by_content_id( pool: &sqlx::PgPool, content_id: &str, @@ -3962,83 +4172,6 @@ async fn track_artist_names(pool: &sqlx::PgPool, track_id: i64, role: &str) -> R Ok(rows.into_iter().map(|row| row.get("name")).collect()) } -fn placeholder_playback_track( - track: &PlaybackTrack, - index: usize, - content_id: Option<&str>, -) -> serde_json::Value { - let artists = if !track.artist_names.is_empty() { - track.artist_names.clone() - } else if let Some(fed) = &track.fed { - fed.artist_names.clone() - } else { - Vec::new() - }; - let featured = if !track.featured_artist_names.is_empty() { - track.featured_artist_names.clone() - } else if let Some(fed) = &track.fed { - fed.featured_artist_names.clone() - } else { - Vec::new() - }; - let title = track - .fed - .as_ref() - .map(|fed| fed.title.as_str()) - .filter(|title| !title.trim().is_empty()) - .unwrap_or(&track.title); - let release_title = track - .fed - .as_ref() - .and_then(|fed| fed.release_title.as_deref()) - .filter(|title| !title.trim().is_empty()) - .unwrap_or(&track.release_title); - let content_id = content_id - .map(ToString::to_string) - .or_else(|| track.content_id.clone()) - .or_else(|| track.fed.as_ref().map(|fed| fed.content_id.clone())); - serde_json::json!({ - "id": placeholder_track_id(content_id.as_deref(), index), - "title": title, - "track_number": track.track_number, - "disc_number": track.disc_number, - "duration_seconds": track.duration_seconds, - "artists": artist_names_to_json(&artists), - "featured_artists": artist_names_to_json(&featured), - "release_id": 0, - "release_title": release_title, - "release_year": track.release_year, - "cover_url": serde_json::Value::Null, - "stream_url": "", - "uploader_name": "Federation", - "audio_format": track.audio_format.clone(), - "audio_bitrate": track.audio_bitrate, - "audio_sample_rate": track.audio_sample_rate, - "audio_bit_depth": track.audio_bit_depth, - "file_size_bytes": track.file_size_bytes, - "content_id": content_id, - "unavailable": true, - "unavailable_reason": "Not available on this web player yet", - }) -} - -fn artist_names_to_json(names: &[String]) -> Vec { - names - .iter() - .enumerate() - .filter(|(_, name)| !name.trim().is_empty()) - .map(|(index, name)| serde_json::json!({"id": -((index as i64) + 1), "name": name})) - .collect() -} - -fn placeholder_track_id(content_id: Option<&str>, index: usize) -> i64 { - let mut bytes = [0u8; 8]; - let hash = blake3::hash(content_id.unwrap_or("").as_bytes()); - bytes.copy_from_slice(&hash.as_bytes()[..8]); - let value = i64::from_le_bytes(bytes).unsigned_abs() % 9_000_000_000_000; - -((value as i64) + index as i64 + 1) -} - async fn track_id_by_content_id(pool: &sqlx::PgPool, content_id: &str) -> Result> { let Some(content_id) = music_dht::normalize_content_id(content_id) else { return Ok(None); @@ -4471,11 +4604,23 @@ async fn note_peer_vector( Ok(()) } +/// Operations the peer has not acknowledged and can parse. +/// +/// Kinds the peer's version does not know are excluded in SQL; skipped +/// operations are never re-sent after an upgrade (vectors only grow), the +/// snapshot is the recovery path for them. async fn ops_for_peer( pool: &sqlx::PgPool, user_id: i64, peer_device_id: &str, + peer_version: u16, ) -> Result> { + // This server only speaks `furumi/sync/2`, so every peer it exchanges + // operations with implements v2 (listen history) even when it never + // reported a version (an invite without `p`, a device known only from + // `device_trusted`). Without this floor listen events would be skipped + // for good: they are not part of snapshots. + let peer_version = peer_version.max(SYNC_V2_PROTOCOL_VERSION); let rows = sqlx::query( "SELECT o.op_id, o.origin_device_id, o.seq, o.hlc_ms, o.payload_json FROM furumusic__fed_sync_ops o @@ -4492,6 +4637,7 @@ async fn ops_for_peer( AND o.payload_json->>'target_device_id' = $2 ) ) + AND NOT (o.kind = ANY($5)) ORDER BY o.hlc_ms, o.op_id LIMIT $4", ) @@ -4499,20 +4645,25 @@ async fn ops_for_peer( .bind(peer_device_id) .bind(now_ms().saturating_sub(PLAYBACK_COMMAND_TTL_MS)) .bind(MAX_OPS_PER_BATCH) + .bind(op_kinds_hidden_from(peer_version)) .fetch_all(pool) .await?; - rows.into_iter() - .map(|row| { - let payload_json: serde_json::Value = row.get("payload_json"); - Ok(SyncOpWire { - op_id: row.get("op_id"), - origin_device_id: row.get("origin_device_id"), - seq: row.get("seq"), - hlc_ms: row.get("hlc_ms"), - payload: serde_json::from_value(payload_json)?, - }) - }) - .collect() + let mut ops = Vec::with_capacity(rows.len()); + for row in rows { + let payload_json: serde_json::Value = row.get("payload_json"); + let op = SyncOpWire { + op_id: row.get("op_id"), + origin_device_id: row.get("origin_device_id"), + seq: row.get("seq"), + hlc_ms: row.get("hlc_ms"), + payload: serde_json::from_value(payload_json)?, + }; + // Playback commands carrying podcast items need v4 as well. + if op_allowed_for_peer(&op.payload, peer_version) { + ops.push(op); + } + } + Ok(ops) } async fn snapshot(pool: &sqlx::PgPool, user_id: i64) -> Result { @@ -4601,13 +4752,23 @@ async fn snapshot(pool: &sqlx::PgPool, user_id: i64) -> Result { op_id: row.get("op_id"), }) .collect(); - Ok(SyncSnapshot { + let mut snapshot = SyncSnapshot { likes, unlikes, playlists, deleted_playlists, removed_playlist_items, - }) + ..SyncSnapshot::default() + }; + // Bounded again per message by `fit_podcast_snapshot`. + PodcastStore::new(pool.clone()) + .fill_snapshot( + user_id, + &mut snapshot, + music_dht::podcast::MAX_PODCAST_SNAPSHOT_BYTES, + ) + .await?; + Ok(snapshot) } async fn snapshot_playlist_items( @@ -4839,6 +5000,7 @@ async fn write_sync_reject(mut stream: ByteStream, message: &str) -> Result<()> finish_response(&mut stream).await } +/// Operation `kind` tag as stored in `furumusic__fed_sync_ops.kind`. fn payload_kind(payload: &SyncOpPayload) -> &'static str { match payload { SyncOpPayload::TrackLikeSet { .. } => "track_like_set", @@ -4852,70 +5014,83 @@ fn payload_kind(payload: &SyncOpPayload) -> &'static str { SyncOpPayload::DeviceRevoked { .. } => "device_revoked", SyncOpPayload::PlaybackCommand { .. } => "playback_command", SyncOpPayload::ListenRecorded { .. } => "listen_recorded", + SyncOpPayload::PodcastSubscriptionSet { .. } => "podcast_subscription_set", + SyncOpPayload::PodcastShareSet { .. } => "podcast_share_set", + SyncOpPayload::PodcastFeedAliasesAdded { .. } => "podcast_feed_aliases_added", + SyncOpPayload::PodcastProgressSet { .. } => "podcast_progress_set", + SyncOpPayload::PodcastListenRecorded { .. } => "podcast_listen_recorded", } } +/// Stored operation kinds a peer of `peer_version` cannot parse. +/// +/// Filtered in SQL so a batch of hidden operations cannot stall the +/// per-peer cursor; `op_allowed_for_peer` stays the final check. +fn op_kinds_hidden_from(peer_version: u16) -> Vec<&'static str> { + let mut kinds = Vec::new(); + if !supports_listen_history(peer_version) { + kinds.push("listen_recorded"); + } + if !supports_podcasts(peer_version) { + kinds.extend([ + "podcast_subscription_set", + "podcast_share_set", + "podcast_feed_aliases_added", + "podcast_progress_set", + "podcast_listen_recorded", + ]); + } + kinds +} + +/// Bounds the podcast snapshot sections so the whole line stays under the +/// protocol limit (`podcast_snapshot_budget`). +fn fit_podcast_snapshot(message: &mut WireMessage) { + let snapshot = match message { + WireMessage::PairRequest { snapshot, .. } + | WireMessage::PairResponse { snapshot, .. } + | WireMessage::Hello { snapshot, .. } + | WireMessage::SyncResponse { snapshot, .. } => snapshot, + }; + if !snapshot.has_podcast_state() { + return; + } + let podcast = snapshot.clone(); + snapshot.strip_podcast_state(); + let other_bytes = serde_json::to_vec(message).map_or(usize::MAX, |bytes| bytes.len()); + let snapshot = match message { + WireMessage::PairRequest { snapshot, .. } + | WireMessage::PairResponse { snapshot, .. } + | WireMessage::Hello { snapshot, .. } + | WireMessage::SyncResponse { snapshot, .. } => snapshot, + }; + *snapshot = podcast; + snapshot.bound_podcast_state_to(podcast_snapshot_budget(other_bytes)); +} + async fn write_msg(stream: &mut ByteStream, message: &WireMessage) -> Result<()> { - let mut payload = serde_json::to_vec(message)?; - payload.push(b'\n'); - stream.send.write_all(&payload).await?; + let mut message = message.clone(); + fit_podcast_snapshot(&mut message); + music_dht::device_sync::write_msg(stream, &message).await?; Ok(()) } async fn finish_send(stream: &mut ByteStream) -> Result<()> { - stream.send.finish()?; + music_dht::device_sync::finish_send(stream).await?; Ok(()) } async fn finish_response(stream: &mut ByteStream) -> Result<()> { - stream.send.finish()?; - let _ = tokio::time::timeout(RESPONSE_DRAIN_TIMEOUT, stream.send.stopped()).await; + music_dht::device_sync::finish_response(stream, RESPONSE_DRAIN_TIMEOUT).await?; Ok(()) } async fn read_msg(stream: &mut ByteStream) -> Result { - let line = read_line(&mut stream.recv).await?; - Ok(serde_json::from_slice(&line)?) -} - -async fn read_line(reader: &mut R) -> Result> { - let mut out = Vec::new(); - loop { - let mut byte = [0u8; 1]; - let read = reader.read(&mut byte).await?; - if read == 0 { - break; - } - if byte[0] == b'\n' { - break; - } - out.push(byte[0]); - if out.len() > MAX_LINE { - anyhow::bail!("protocol line is too large"); - } - } - Ok(out) -} - -fn parse_invite(value: &str) -> Result { - let Some(token) = value.trim().strip_prefix("frid://i/") else { - anyhow::bail!("usage: frid://i/"); - }; - let bytes = base64url_decode(token)?; - let invite: InviteWire = serde_json::from_slice(&bytes)?; - anyhow::ensure!(invite.v == 1, "unsupported invite version"); - Ok(invite) + Ok(music_dht::device_sync::read_msg(stream).await?) } pub fn invite_network_id(value: &str) -> Result { - let invite = parse_invite(value)?; - let ticket: PeerTicket = invite.ticket.parse().context("malformed invite ticket")?; - Ok(ticket.network_id) -} - -fn encode_invite(invite: &InviteWire) -> Result { - let bytes = serde_json::to_vec(invite)?; - Ok(format!("frid://i/{}", base64url_encode(&bytes))) + Ok(music_dht::device_sync::invite_network_id(value)?) } fn pair_request_id(invite_id: &str, device_id: &str) -> String { @@ -4923,24 +5098,6 @@ fn pair_request_id(invite_id: &str, device_id: &str) -> String { format!("pair_{}", &digest.to_hex()[..16]) } -fn hash_secret(secret: &str) -> String { - blake3::hash(secret.as_bytes()).to_hex().to_string() -} - -fn random_hex(bytes: usize) -> String { - let key = SecretKey::generate(); - let mut seed = key.to_bytes().to_vec(); - while seed.len() < bytes { - seed.extend_from_slice(blake3::hash(&seed).as_bytes()); - } - hex_encode(&seed[..bytes]) -} - -fn ticket_endpoint_id(ticket: &str) -> Option { - let ticket = PeerTicket::from_str(ticket).ok()?; - Some(ticket.endpoint_id().to_string()) -} - fn repeat_label(repeat: PlaybackRepeat) -> &'static str { match repeat { PlaybackRepeat::Off => "off", @@ -4987,62 +5144,13 @@ fn short_id(value: &str) -> String { value.chars().take(10).collect() } -fn hex_encode(bytes: &[u8]) -> String { - bytes.iter().map(|byte| format!("{byte:02x}")).collect() -} - -fn base64url_encode(bytes: &[u8]) -> String { - const TABLE: &[u8; 64] = b"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_"; - let mut out = String::new(); - let mut i = 0; - while i < bytes.len() { - let b0 = bytes[i]; - let b1 = bytes.get(i + 1).copied().unwrap_or(0); - let b2 = bytes.get(i + 2).copied().unwrap_or(0); - out.push(TABLE[(b0 >> 2) as usize] as char); - out.push(TABLE[(((b0 & 0b0000_0011) << 4) | (b1 >> 4)) as usize] as char); - if i + 1 < bytes.len() { - out.push(TABLE[(((b1 & 0b0000_1111) << 2) | (b2 >> 6)) as usize] as char); - } - if i + 2 < bytes.len() { - out.push(TABLE[(b2 & 0b0011_1111) as usize] as char); - } - i += 3; - } - out -} - -fn base64url_decode(value: &str) -> Result> { - fn val(byte: u8) -> Option { - match byte { - b'A'..=b'Z' => Some(byte - b'A'), - b'a'..=b'z' => Some(byte - b'a' + 26), - b'0'..=b'9' => Some(byte - b'0' + 52), - b'-' => Some(62), - b'_' => Some(63), - _ => None, - } - } - let bytes = value.as_bytes(); - let mut out = Vec::with_capacity(bytes.len() * 3 / 4); - let mut i = 0; - while i < bytes.len() { - let a = val(bytes[i]).context("invalid base64url invite")?; - let b = val(*bytes.get(i + 1).context("truncated base64url invite")?) - .context("invalid base64url invite")?; - let c = bytes.get(i + 2).and_then(|byte| val(*byte)); - let d = bytes.get(i + 3).and_then(|byte| val(*byte)); - out.push((a << 2) | (b >> 4)); - if let Some(c) = c { - out.push((b << 4) | (c >> 2)); - if let Some(d) = d { - out.push((c << 6) | d); - } - } - i += 4; - } - Ok(out) -} - +#[cfg(test)] +mod first_round_tests; +#[cfg(test)] +mod gating_tests; #[cfg(test)] mod interop_tests; +#[cfg(test)] +mod postgres_tests; +#[cfg(test)] +mod wire_compat_tests; diff --git a/src/federation/devices/first_round_tests.rs b/src/federation/devices/first_round_tests.rs new file mode 100644 index 0000000..a63e54e --- /dev/null +++ b/src/federation/devices/first_round_tests.rs @@ -0,0 +1,34 @@ +//! The first device-sync round that the podcast publication reconciler +//! waits for ([`FirstSyncRound`]). + +use super::FirstSyncRound; + +#[test] +fn the_first_round_ends_when_its_last_exchange_ends() { + let (done, watch) = tokio::sync::watch::channel(false); + let mut round = FirstSyncRound::new(done); + // An exchange that ends before the first complete listing is not + // waited for. + round.finished(&7); + assert!(!*watch.borrow()); + round.started([1, 2]); + // A later listing (with exchanges it started) changes nothing. + round.started([3]); + round.finished(&1); + round.finished(&3); + assert!(!*watch.borrow()); + round.finished(&2); + assert!(*watch.borrow()); + // Stays done. + round.started([4]); + round.finished(&4); + assert!(*watch.borrow()); +} + +#[test] +fn without_trusted_devices_the_first_round_ends_at_once() { + let (done, watch) = tokio::sync::watch::channel(false); + let mut round = FirstSyncRound::::new(done); + round.started([]); + assert!(*watch.borrow()); +} diff --git a/src/federation/devices/fixtures/v3_hello.golden.json b/src/federation/devices/fixtures/v3_hello.golden.json new file mode 100644 index 0000000..ad7622e --- /dev/null +++ b/src/federation/devices/fixtures/v3_hello.golden.json @@ -0,0 +1 @@ +{"type":"hello","group_id":"grp_1","profile":{"device_id":"tui_a","name":"TUI","client_version":"0.3.4","protocol_version":3,"endpoint_id":"e","endpoint_ticket":"t","revoked":false,"revoke_cutoff_seq":null,"updated_at_ms":5},"devices":[{"device_id":"tui_a","name":"TUI","client_version":"0.3.4","protocol_version":3,"endpoint_id":"e","endpoint_ticket":"t","revoked":false,"revoke_cutoff_seq":null,"updated_at_ms":5},{"device_id":"web_b","name":"TUI","client_version":"0.3.4","protocol_version":3,"endpoint_id":"e","endpoint_ticket":"t","revoked":false,"revoke_cutoff_seq":null,"updated_at_ms":5}],"vector":{"tui_a":12,"web_b":4},"ops":[{"op_id":"tui_a:1","origin_device_id":"tui_a","seq":1,"hlc_ms":1700000000001,"payload":{"kind":"track_like_set","content_id":"b3:1111111111111111111111111111111111111111111111111111111111111111","liked":true,"fed":{"item_id":"item","owner":"owner","title":"Song","artist_names":["A"],"featured_artist_names":[],"year":2001,"duration_seconds":180,"content_id":"b3:1111111111111111111111111111111111111111111111111111111111111111","release_title":"Album","track_number":1,"disc_number":1}}},{"op_id":"tui_a:2","origin_device_id":"tui_a","seq":2,"hlc_ms":1700000000002,"payload":{"kind":"playlist_created","playlist_id":"pl","title":"Mix"}},{"op_id":"tui_a:3","origin_device_id":"tui_a","seq":3,"hlc_ms":1700000000003,"payload":{"kind":"playlist_renamed","playlist_id":"pl","title":"Mix 2"}},{"op_id":"tui_a:4","origin_device_id":"tui_a","seq":4,"hlc_ms":1700000000004,"payload":{"kind":"playlist_track_added","playlist_id":"pl","content_id":"b3:1111111111111111111111111111111111111111111111111111111111111111","position":0,"fed":null}},{"op_id":"tui_a:5","origin_device_id":"tui_a","seq":5,"hlc_ms":1700000000005,"payload":{"kind":"playlist_track_removed","playlist_id":"pl","content_id":"b3:1111111111111111111111111111111111111111111111111111111111111111"}},{"op_id":"tui_a:6","origin_device_id":"tui_a","seq":6,"hlc_ms":1700000000006,"payload":{"kind":"playlist_deleted","playlist_id":"pl"}},{"op_id":"tui_a:7","origin_device_id":"tui_a","seq":7,"hlc_ms":1700000000007,"payload":{"kind":"device_profile_set","name":"TUI","client_version":"0.3.4","endpoint_ticket":"t","endpoint_id":"e"}},{"op_id":"tui_a:8","origin_device_id":"tui_a","seq":8,"hlc_ms":1700000000008,"payload":{"kind":"device_trusted","target_device_id":"web_b"}},{"op_id":"tui_a:9","origin_device_id":"tui_a","seq":9,"hlc_ms":1700000000009,"payload":{"kind":"device_revoked","target_device_id":"old","target_max_seq_seen":3}},{"op_id":"tui_a:10","origin_device_id":"tui_a","seq":10,"hlc_ms":1700000000010,"payload":{"kind":"playback_command","target_device_id":"web_b","command":{"kind":"set_state","state":{"queue":[{"id":7,"title":"Song","track_number":1,"disc_number":null,"duration_seconds":180.5,"artist_names":["A"],"featured_artist_names":["B"],"release_id":3,"release_title":"Album","release_year":2001,"file_path":"","content_id":"b3:1111111111111111111111111111111111111111111111111111111111111111","audio_format":"flac","audio_bitrate":900,"audio_sample_rate":44100,"audio_bit_depth":16,"file_size_bytes":12345,"play_count":2,"fed":{"item_id":"item","owner":"owner","title":"Song","artist_names":["A"],"featured_artist_names":[],"year":2001,"duration_seconds":180,"content_id":"b3:1111111111111111111111111111111111111111111111111111111111111111","release_title":"Album","track_number":1,"disc_number":1}},{"id":0,"title":"Remote","track_number":null,"disc_number":null,"duration_seconds":0.0,"artist_names":[],"featured_artist_names":[],"release_id":0,"release_title":"","release_year":null,"file_path":"","content_id":null,"audio_format":null,"audio_bitrate":null,"audio_sample_rate":null,"audio_bit_depth":null,"file_size_bytes":null,"play_count":0,"fed":null}],"queue_pos":1,"playing":true,"paused":false,"idle_since_ms":null,"position_secs":42.5,"volume":73,"shuffle":true,"repeat":"all"},"seek":true},"authority":{"version":1,"claim":{"counter":3,"issuer":"tui_a","owner":"tui_a"},"sequence":9}}},{"op_id":"tui_a:11","origin_device_id":"tui_a","seq":11,"hlc_ms":1700000000011,"payload":{"kind":"playback_command","target_device_id":"web_b","command":{"kind":"active_changed","active_device_id":"web_b","active_device_name":"WEB","state":{"queue":[{"id":7,"title":"Song","track_number":1,"disc_number":null,"duration_seconds":180.5,"artist_names":["A"],"featured_artist_names":["B"],"release_id":3,"release_title":"Album","release_year":2001,"file_path":"","content_id":"b3:1111111111111111111111111111111111111111111111111111111111111111","audio_format":"flac","audio_bitrate":900,"audio_sample_rate":44100,"audio_bit_depth":16,"file_size_bytes":12345,"play_count":2,"fed":{"item_id":"item","owner":"owner","title":"Song","artist_names":["A"],"featured_artist_names":[],"year":2001,"duration_seconds":180,"content_id":"b3:1111111111111111111111111111111111111111111111111111111111111111","release_title":"Album","track_number":1,"disc_number":1}},{"id":0,"title":"Remote","track_number":null,"disc_number":null,"duration_seconds":0.0,"artist_names":[],"featured_artist_names":[],"release_id":0,"release_title":"","release_year":null,"file_path":"","content_id":null,"audio_format":null,"audio_bitrate":null,"audio_sample_rate":null,"audio_bit_depth":null,"file_size_bytes":null,"play_count":0,"fed":null}],"queue_pos":1,"playing":true,"paused":false,"idle_since_ms":null,"position_secs":42.5,"volume":73,"shuffle":true,"repeat":"all"}},"authority":null}},{"op_id":"tui_a:12","origin_device_id":"tui_a","seq":12,"hlc_ms":1700000000012,"payload":{"kind":"listen_recorded","event":{"listen_id":"l1","content_id":"b3:1111111111111111111111111111111111111111111111111111111111111111","started_at_ms":1,"listened_ms":60000,"track_duration_ms":180000,"ended_reason":"finished","track":{"title":"Song","artist_names":["A"],"featured_artist_names":[],"release_title":null}}}}],"snapshot":{"likes":[{"content_id":"b3:1111111111111111111111111111111111111111111111111111111111111111","hlc_ms":1,"op_id":"tui_a:1","fed":{"item_id":"item","owner":"owner","title":"Song","artist_names":["A"],"featured_artist_names":[],"year":2001,"duration_seconds":180,"content_id":"b3:1111111111111111111111111111111111111111111111111111111111111111","release_title":"Album","track_number":1,"disc_number":1}}],"unlikes":[{"content_id":"b3:1111111111111111111111111111111111111111111111111111111111111111","hlc_ms":2,"op_id":"tui_a:2"}],"playlists":[{"playlist_id":"pl","title":"Mix","hlc_ms":3,"op_id":"tui_a:3","items":[{"content_id":"b3:1111111111111111111111111111111111111111111111111111111111111111","position":0,"hlc_ms":4,"op_id":"tui_a:4","fed":null}]}],"deleted_playlists":[{"playlist_id":"gone","hlc_ms":5,"op_id":"tui_a:5"}],"removed_playlist_items":[{"playlist_id":"pl","content_id":"b3:1111111111111111111111111111111111111111111111111111111111111111","hlc_ms":6,"op_id":"tui_a:6"}]},"playback":{"device_id":"tui_a","device_name":"TUI","active":true,"updated_at_ms":9,"state":{"queue":[{"id":7,"title":"Song","track_number":1,"disc_number":null,"duration_seconds":180.5,"artist_names":["A"],"featured_artist_names":["B"],"release_id":3,"release_title":"Album","release_year":2001,"file_path":"","content_id":"b3:1111111111111111111111111111111111111111111111111111111111111111","audio_format":"flac","audio_bitrate":900,"audio_sample_rate":44100,"audio_bit_depth":16,"file_size_bytes":12345,"play_count":2,"fed":{"item_id":"item","owner":"owner","title":"Song","artist_names":["A"],"featured_artist_names":[],"year":2001,"duration_seconds":180,"content_id":"b3:1111111111111111111111111111111111111111111111111111111111111111","release_title":"Album","track_number":1,"disc_number":1}},{"id":0,"title":"Remote","track_number":null,"disc_number":null,"duration_seconds":0.0,"artist_names":[],"featured_artist_names":[],"release_id":0,"release_title":"","release_year":null,"file_path":"","content_id":null,"audio_format":null,"audio_bitrate":null,"audio_sample_rate":null,"audio_bit_depth":null,"file_size_bytes":null,"play_count":0,"fed":null}],"queue_pos":1,"playing":true,"paused":false,"idle_since_ms":null,"position_secs":42.5,"volume":73,"shuffle":true,"repeat":"all"},"coordination":{"version":1,"claim":{"counter":3,"issuer":"tui_a","owner":"tui_a"},"heartbeat":4,"available":true,"playing":true}}} diff --git a/src/federation/devices/gating_tests.rs b/src/federation/devices/gating_tests.rs new file mode 100644 index 0000000..c12ac9f --- /dev/null +++ b/src/federation/devices/gating_tests.rs @@ -0,0 +1,621 @@ +//! Version gating and lossless queue bridge (device-sync protocol v4). + +use super::*; +use crate::player::queue_items::{self, test_episode}; +use music_dht::podcast::{ + EpisodeId, FeedUrl, ListenSession, PodcastId, PodcastMeta, ProgressStatus, +}; + +const CONTENT_ID: &str = "b3:2222222222222222222222222222222222222222222222222222222222222222"; + +fn podcast_id() -> PodcastId { + PodcastId::from_feed_url(&FeedUrl::parse("https://podcasts.example/feed.xml").unwrap()) +} + +fn episode_id() -> EpisodeId { + EpisodeId::from_guid(&podcast_id(), "ep-1").unwrap() +} + +fn music_track(id: i64) -> PlaybackTrack { + let mut track = PlaybackTrack::private_placeholder(); + track.source = None; + track.id = id; + track.title = format!("Song {id}"); + track +} + +fn state(queue: Vec) -> PlaybackStateWire { + PlaybackStateWire { + queue, + queue_pos: 0, + playing: true, + paused: false, + idle_since_ms: None, + position_secs: 12.5, + volume: 70, + shuffle: false, + repeat: PlaybackRepeat::Off, + playback_rate: None, + } +} + +fn podcast_state() -> PlaybackStateWire { + state(vec![ + music_track(1), + PlaybackTrack::podcast_episode(test_episode()), + ]) +} + +/// One payload of every operation kind (music-only playback commands). +fn every_payload(playback: PlaybackStateWire) -> Vec { + let session = ListenSession { + session_id: "s1".into(), + episode_id: episode_id(), + podcast_id: podcast_id(), + device_id: "tui_a".into(), + started_at_ms: 1, + ended_at_ms: 2, + start_position_ms: 0, + end_position_ms: 1, + listened_ms: 1, + completed: false, + }; + let meta: PodcastMeta = serde_json::from_value(serde_json::json!({ + "feed_url": "https://podcasts.example/feed.xml", "title": "Cast" + })) + .unwrap(); + vec![ + SyncOpPayload::TrackLikeSet { + content_id: CONTENT_ID.into(), + liked: true, + fed: None, + }, + SyncOpPayload::PlaylistCreated { + playlist_id: "p".into(), + title: "t".into(), + }, + SyncOpPayload::PlaylistRenamed { + playlist_id: "p".into(), + title: "t".into(), + }, + SyncOpPayload::PlaylistDeleted { + playlist_id: "p".into(), + }, + SyncOpPayload::PlaylistTrackAdded { + playlist_id: "p".into(), + content_id: CONTENT_ID.into(), + position: 0, + fed: None, + }, + SyncOpPayload::PlaylistTrackRemoved { + playlist_id: "p".into(), + content_id: CONTENT_ID.into(), + }, + SyncOpPayload::DeviceProfileSet { + name: "n".into(), + client_version: "v".into(), + endpoint_ticket: "t".into(), + endpoint_id: "e".into(), + protocol_version: Some(4), + }, + SyncOpPayload::DeviceTrusted { + target_device_id: "d".into(), + }, + SyncOpPayload::DeviceRevoked { + target_device_id: "d".into(), + target_max_seq_seen: 1, + }, + SyncOpPayload::PlaybackCommand { + target_device_id: "d".into(), + command: PlaybackCommand::SetState { + state: playback, + seek: false, + }, + authority: None, + }, + SyncOpPayload::ListenRecorded { + event: ListenEvent { + listen_id: "l".into(), + content_id: CONTENT_ID.into(), + started_at_ms: 1, + listened_ms: 1, + track_duration_ms: None, + ended_reason: ListenEndReason::Finished, + track: ListenTrackMetadata { + title: "t".into(), + artist_names: vec!["a".into()], + featured_artist_names: vec![], + release_title: None, + }, + }, + }, + SyncOpPayload::PodcastSubscriptionSet { + podcast_id: podcast_id(), + subscribed: true, + podcast: meta, + feed_aliases: vec![], + generation: 1, + }, + SyncOpPayload::PodcastShareSet { + podcast_id: podcast_id(), + shared: false, + generation: 1, + }, + SyncOpPayload::PodcastFeedAliasesAdded { + podcast_id: podcast_id(), + feed_aliases: vec![FeedUrl::parse("https://podcasts.example/new.xml").unwrap()], + }, + SyncOpPayload::PodcastProgressSet { + episode_id: episode_id(), + podcast_id: podcast_id(), + position_ms: 1, + status: ProgressStatus::InProgress, + generation: 1, + session_id: "s1".into(), + episode: None, + }, + SyncOpPayload::PodcastListenRecorded { + session, + episode: None, + }, + ] +} + +#[test] +fn stored_kind_is_the_wire_tag() { + for payload in every_payload(state(vec![])) { + let value = serde_json::to_value(&payload).unwrap(); + assert_eq!(payload_kind(&payload), value["kind"].as_str().unwrap()); + } +} + +#[test] +fn op_gating_matrix_matches_the_shared_helper() { + // The SQL kind filter must hide exactly what `op_allowed_for_peer` + // rejects (playback commands are content-dependent and checked in Rust). + for version in 1..=PROTOCOL_VERSION { + let hidden = op_kinds_hidden_from(version); + for payload in every_payload(state(vec![music_track(1)])) { + let kind = payload_kind(&payload); + assert_eq!( + hidden.contains(&kind), + !op_allowed_for_peer(&payload, version), + "{kind} for v{version}" + ); + } + } + assert!( + op_kinds_hidden_from(3) + .iter() + .all(|kind| kind.starts_with("podcast_")) + ); + assert_eq!(op_kinds_hidden_from(3).len(), 5); + assert!(op_kinds_hidden_from(4).is_empty()); +} + +#[test] +fn playback_gating_matrix() { + let music = state(vec![music_track(1), music_track(2)]); + let podcasts = podcast_state(); + let private = state(vec![PlaybackTrack::private_placeholder()]); + for version in 1..=PROTOCOL_VERSION { + let v4 = version >= 4; + // Snapshots and handoffs. + assert!(playback_allowed_for_peer(&music, version)); + assert_eq!(playback_allowed_for_peer(&podcasts, version), v4); + assert_eq!(playback_allowed_for_peer(&private, version), v4); + // Commands from the peer. + assert!(playback_command_allowed_from_peer(&music, version)); + assert_eq!(playback_command_allowed_from_peer(&podcasts, version), v4); + // Stamped playback command operations. + let command = |state: &PlaybackStateWire| SyncOpPayload::PlaybackCommand { + target_device_id: "d".into(), + command: PlaybackCommand::ActiveChanged { + active_device_id: "d".into(), + active_device_name: "D".into(), + state: state.clone(), + }, + authority: None, + }; + assert!(op_allowed_for_peer(&command(&music), version)); + assert_eq!(op_allowed_for_peer(&command(&podcasts), version), v4); + } +} + +#[test] +fn browser_state_gating_sees_podcast_items() { + let with_podcast = serde_json::json!({ + "tracks": [ + {"id": 5, "title": "Library song"}, + queue_items::external_track_json(&PlaybackTrack::podcast_episode(test_episode()), 1, None), + ], + "index": 0, + }); + let shadow = browser_state_shadow(&with_podcast); + assert!(!playback_command_allowed_from_peer(&shadow, 3)); + assert!(playback_command_allowed_from_peer(&shadow, 4)); + let music_only = serde_json::json!({"tracks": [{"id": 5}], "index": 0}); + assert!(playback_command_allowed_from_peer( + &browser_state_shadow(&music_only), + 3 + )); +} + +#[test] +fn relayed_versions_never_downgrade_and_self_reports_replace() { + let profile = |id: &str, version: u16| DeviceProfileWire { + device_id: id.into(), + name: "D".into(), + client_version: "x".into(), + protocol_version: version, + endpoint_id: String::new(), + endpoint_ticket: String::new(), + revoked: false, + revoke_cutoff_seq: None, + updated_at_ms: 0, + }; + // A v3 relay lists every device with its own constant. + let relayed = VersionReport::for_listed(&profile("tui_b", 3), Some("tui_a")); + assert_eq!(relayed, VersionReport::Relayed(3)); + assert_eq!(relayed.resolve(Some(4)), 4); + assert_eq!(relayed.resolve(None), 3); + assert_eq!(VersionReport::Relayed(4).resolve(Some(3)), 4); + // The sender's own entry is a self-report and may lower the version. + let own = VersionReport::for_listed(&profile("tui_a", 3), Some("tui_a")); + assert_eq!(own, VersionReport::SelfReported(Some(3))); + assert_eq!(own.resolve(Some(4)), 3); + // Profiles without a version (pending pairing, pre-v4 DeviceProfileSet) + // keep the stored value, or the unreported version 1. + assert_eq!(VersionReport::SelfReported(None).resolve(Some(4)), 4); + assert_eq!(VersionReport::SelfReported(None).resolve(None), 1); +} + +#[test] +fn podcast_snapshot_sections_are_bounded_per_message() { + let mut snapshot = SyncSnapshot::default(); + let progress = music_dht::podcast::ProgressRecord { + episode_id: episode_id(), + podcast_id: podcast_id(), + position_ms: 1, + status: ProgressStatus::InProgress, + generation: 1, + session_id: "s".into(), + hlc_ms: 1, + device_id: "tui_a".into(), + op_id: "tui_a:1".into(), + }; + snapshot.podcast_progress = vec![progress; 10]; + // Large other parts leave (almost) no budget for podcast sections. + let big_title = "x".repeat(music_dht::device_sync::MAX_SYNC_LINE - 16 * 1024); + let mut message = WireMessage::Hello { + group_id: "g".into(), + profile: DeviceProfileWire { + device_id: "web".into(), + name: big_title, + client_version: String::new(), + protocol_version: PROTOCOL_VERSION, + endpoint_id: String::new(), + endpoint_ticket: String::new(), + revoked: false, + revoke_cutoff_seq: None, + updated_at_ms: 0, + }, + devices: vec![], + vector: BTreeMap::new(), + ops: vec![], + snapshot: snapshot.clone(), + playback: None, + }; + fit_podcast_snapshot(&mut message); + let WireMessage::Hello { + snapshot: fitted, .. + } = &message + else { + unreachable!() + }; + assert!(fitted.podcast_progress.is_empty()); + + let mut small = WireMessage::SyncResponse { + accepted: true, + error: None, + devices: vec![], + vector: BTreeMap::new(), + ops: vec![], + snapshot, + playback: None, + }; + fit_podcast_snapshot(&mut small); + let WireMessage::SyncResponse { snapshot, .. } = &small else { + unreachable!() + }; + assert_eq!(snapshot.podcast_progress.len(), 10); +} + +/// Pool that is never connected: the bridge paths under test resolve +/// podcast items, placeholders and inline content ids without queries. +fn offline_pool() -> sqlx::PgPool { + sqlx::postgres::PgPoolOptions::new() + .connect_lazy("postgres://offline@127.0.0.1:1/offline") + .unwrap() +} + +#[tokio::test] +async fn podcast_queue_survives_server_browser_server_round_trip() { + let pool = offline_pool(); + let mut other = test_episode(); + other.episode_title = "Episode 43".into(); + other.episode_id = EpisodeId::from_guid(&other.podcast_id, "episode-43").unwrap(); + let mut unresolved = music_track(0); + unresolved.title = "Remote song".into(); + let queue = vec![ + PlaybackTrack::podcast_episode(test_episode()), + PlaybackTrack::private_placeholder(), + unresolved, + PlaybackTrack::podcast_episode(other), + ]; + let mut incoming = state(queue.clone()); + incoming.queue_pos = 3; + incoming.paused = true; + incoming.position_secs = 1200.0; + + // peer -> server -> browser + let payload = web_playback_payload(&pool, &incoming).await.unwrap(); + let tracks = payload["tracks"].as_array().unwrap(); + assert_eq!(tracks.len(), 4, "no item is dropped"); + assert_eq!(payload["index"], 3); + assert_eq!(payload["position_seconds"], 1200.0); + assert_eq!(tracks[0]["source"]["kind"], "podcast_episode"); + assert_eq!( + tracks[0]["unavailable"], false, + "episodes play on the web player" + ); + assert_eq!(tracks[1]["source"]["kind"], "private"); + + // The browser keeps its queue and sends it back (plus a library track + // it added at the end, with its content id inline). + let mut browser: serde_json::Value = serde_json::from_str(&payload.to_string()).unwrap(); + browser["tracks"] + .as_array_mut() + .unwrap() + .push(serde_json::json!({ + "id": 77, "title": "Library song", "content_id": CONTENT_ID, + "duration_seconds": 100.0, "artists": [{"id": 1, "name": "A"}], + "release_id": 9, "release_title": "Album", + })); + + // browser -> server -> peer (the bridge itself; the progress-register + // lookups of `playback_state_from_browser_json` need a database and + // are covered by `apply_progress_generations_*` and the Postgres tests) + let outgoing = browser_json_to_wire(&pool, browser).await.unwrap(); + assert_eq!(outgoing.queue[..4], queue[..]); + assert_eq!(outgoing.queue_pos, 3); + assert!(outgoing.paused); + // The episode position goes back to the peer (handoff web -> TUI). + assert_eq!(outgoing.position_secs, 1200.0); + let library = &outgoing.queue[4]; + assert_eq!(library.id, 77); + assert_eq!(library.content_id.as_deref(), Some(CONTENT_ID)); + assert_eq!(library.source, None); + // Library music keeps the v3 shape. + assert!(!serde_json::to_string(library).unwrap().contains("source")); + // Peers below v4 get no playback snapshot with these items. + assert!(!playback_allowed_for_peer(&outgoing, 3)); + assert!(playback_allowed_for_peer(&outgoing, 4)); +} + +#[test] +fn apply_progress_generations_raises_stale_handoff_hints_only() { + let mut first = test_episode(); + first.progress_generation = Some(9); + let mut second = test_episode(); + second.episode_id = EpisodeId::from_guid(&second.podcast_id, "episode-43").unwrap(); + second.progress_generation = None; + let mut third = test_episode(); + third.episode_id = EpisodeId::from_guid(&third.podcast_id, "episode-44").unwrap(); + third.progress_generation = Some(2); + let mut wire = state(vec![ + PlaybackTrack::podcast_episode(first.clone()), + PlaybackTrack::podcast_episode(second.clone()), + music_track(3), + PlaybackTrack::podcast_episode(third.clone()), + ]); + let generations = std::collections::HashMap::from([ + (first.episode_id.clone(), 4), + (second.episode_id.clone(), 6), + (third.episode_id.clone(), 5), + ]); + apply_progress_generations(&mut wire, &generations); + let hints: Vec> = wire + .queue + .iter() + .map(|track| { + track + .podcast_episode_ref() + .and_then(|e| e.progress_generation) + }) + .collect(); + // A newer carried hint stays; missing and older ones are raised. + assert_eq!(hints, vec![Some(9), Some(6), None, Some(5)]); + assert_eq!(wire.queue[2], music_track(3), "music is untouched"); +} + +#[tokio::test] +async fn malformed_browser_items_keep_the_position_on_its_item() { + let pool = offline_pool(); + let browser = serde_json::json!({ + "tracks": [ + {"title": "no id"}, + {"id": 5, "content_id": CONTENT_ID}, + {"id": 6, "content_id": CONTENT_ID}, + ], + "index": 2, + "track": {"id": 6}, + }); + let wire = playback_state_from_browser_json(&pool, 1, browser) + .await + .unwrap(); + assert_eq!(wire.queue.len(), 2); + assert_eq!(wire.queue[wire.queue_pos].id, 6); +} + +fn progress_record(position_ms: i64, status: ProgressStatus, generation: u64) -> ProgressRecord { + let episode = test_episode(); + ProgressRecord { + episode_id: episode.episode_id, + podcast_id: episode.podcast_id, + position_ms, + status, + generation, + session_id: "session-1".into(), + hlc_ms: 1_700_000_000_000, + device_id: "tui".into(), + op_id: "op-1".into(), + } +} + +/// Browser state of `[library song, episode]` with the song current at 200 s. +fn song_then_episode() -> serde_json::Value { + let episode = PlaybackTrack::podcast_episode(test_episode()); + let song = serde_json::json!({ + "id": 77, "title": "Library song", "content_id": CONTENT_ID, + "duration_seconds": 300.0, "stream_url": "/api/player/stream/77", + }); + serde_json::json!({ + "tracks": [song.clone(), queue_items::external_track_json(&episode, 1, None)], + "index": 0, + "track": song, + "position_seconds": 200.0, + "paused": false, + "repeat_mode": "off", + }) +} + +fn position_unknown(state: &serde_json::Value) -> bool { + state[queue_items::POSITION_UNKNOWN_KEY] == true +} + +#[test] +fn commands_that_select_an_item_do_not_know_its_position() { + let current = song_then_episode(); + // Next and Prev onto another item, and play commands (which the browser + // always sends with position 0), leave the position to the server. + let next = + browser_state_for_command("next", serde_json::json!({}), Some(current.clone())).unwrap(); + assert_eq!(next["index"], 1); + assert_eq!(next["position_seconds"], 0.0); + assert!(position_unknown(&next)); + let mut play = current.clone(); + play["index"] = serde_json::json!(1); + let played = browser_state_for_command("play_from_index", play.clone(), None).unwrap(); + assert!(position_unknown(&played)); + let prev = + browser_state_for_command("prev", serde_json::json!({}), Some(next.clone())).unwrap(); + assert_eq!(prev["index"], 0); + assert!(position_unknown(&prev)); + + // Restarting the same item and seeking are explicit positions. + let mut late = next.clone(); + late["position_seconds"] = serde_json::json!(900.0); + let restart = browser_state_for_command("prev", serde_json::json!({}), Some(late)).unwrap(); + assert_eq!(restart["index"], 1); + assert!(!position_unknown(&restart)); + let repeat = browser_state_for_command( + "next", + serde_json::json!({"repeat_mode": "one"}), + Some(next.clone()), + ) + .unwrap(); + assert!(!position_unknown(&repeat)); + let seek = + browser_state_for_command("seek", serde_json::json!({"time": 60.0}), Some(next)).unwrap(); + assert_eq!(seek["position_seconds"], 60.0); + assert!(!position_unknown(&seek)); + + // A transfer keeps what the sender knew. + let mut handed_off = play; + handed_off["position_seconds"] = serde_json::json!(1200.0); + let transfer = browser_state_for_command("transfer_state", handed_off.clone(), None).unwrap(); + assert_eq!(transfer["position_seconds"], 1200.0); + assert!(!position_unknown(&transfer)); + handed_off[queue_items::POSITION_UNKNOWN_KEY] = serde_json::json!(true); + let transfer = browser_state_for_command("transfer_state", handed_off, None).unwrap(); + assert!(position_unknown(&transfer)); +} + +#[tokio::test] +async fn unknown_episode_position_resumes_from_the_progress_register() { + let pool = offline_pool(); + let next = browser_state_for_command("next", serde_json::json!({}), Some(song_then_episode())) + .unwrap(); + let mut wire = browser_json_to_wire(&pool, next).await.unwrap(); + assert_eq!(wire.queue_pos, 1); + assert_eq!(wire.position_secs, 0.0); + + // In progress at 1200 s, generation 9 (the queue item has seen 7). + let mut resumed = wire.clone(); + apply_resume_position( + &mut resumed, + Some(&progress_record(1_200_000, ProgressStatus::InProgress, 9)), + ); + assert_eq!(resumed.position_secs, 1200.0); + let episode = resumed.queue[1].podcast_episode_ref().unwrap(); + assert_eq!(episode.progress_generation, Some(9)); + // The receiver's explicit write then orders after the register. + assert_eq!( + ProgressRecord::explicit_generation(episode.progress_generation), + 10 + ); + + // Completed: ordinary Play starts from the beginning; an older register + // does not lower the generation the item has seen. + let mut completed = wire.clone(); + apply_resume_position( + &mut completed, + Some(&progress_record(1_790_000, ProgressStatus::Completed, 3)), + ); + assert_eq!(completed.position_secs, 0.0); + assert_eq!( + completed.queue[1] + .podcast_episode_ref() + .unwrap() + .progress_generation, + Some(7) + ); + + // Music keeps the position it has. + wire.queue_pos = 0; + wire.position_secs = 5.0; + let before = wire.clone(); + apply_resume_position( + &mut wire, + Some(&progress_record(1_200_000, ProgressStatus::InProgress, 9)), + ); + assert_eq!(wire, before); +} + +#[tokio::test] +async fn known_positions_and_music_never_take_the_register_position() { + // The bridge itself runs without queries (offline pool); which state + // reads the episode's progress register is decided on the result. + let pool = offline_pool(); + let mut handed_off = song_then_episode(); + handed_off["index"] = serde_json::json!(1); + handed_off["position_seconds"] = serde_json::json!(1200.0); + let wire = browser_json_to_wire(&pool, handed_off).await.unwrap(); + assert_eq!(wire.position_secs, 1200.0); + assert_eq!(episode_needing_resume_position(false, &wire), None); + assert_eq!( + episode_needing_resume_position(true, &wire), + Some(test_episode().episode_id), + "an episode the browser did not know the position of" + ); + + let music = song_then_episode(); + let wire = browser_json_to_wire(&pool, music).await.unwrap(); + assert_eq!(wire.position_secs, 200.0); + assert_eq!( + episode_needing_resume_position(true, &wire), + None, + "music keeps its position even when flagged" + ); +} diff --git a/src/federation/devices/interop_tests.rs b/src/federation/devices/interop_tests.rs index 7c90357..3979462 100644 --- a/src/federation/devices/interop_tests.rs +++ b/src/federation/devices/interop_tests.rs @@ -1,5 +1,6 @@ //! Cross-binary protocol contract; the peer is the TUI's production adapter. use super::*; +use music_dht::device_sync::read_line; use tokio::io::AsyncWriteExt; #[tokio::test] @@ -34,7 +35,8 @@ async fn localhost_tui_peer() { else { panic!("expected TUI hello") }; - assert_eq!(profile.protocol_version, PROTOCOL_VERSION); + // TUI 0.3.x reports v3; podcast-capable TUI releases report v4. + assert!((3..=PROTOCOL_VERSION).contains(&profile.protocol_version)); assert_eq!(snapshot.device_id, profile.device_id); let announcement = snapshot .coordination diff --git a/src/federation/devices/postgres_tests.rs b/src/federation/devices/postgres_tests.rs new file mode 100644 index 0000000..d78cd9f --- /dev/null +++ b/src/federation/devices/postgres_tests.rs @@ -0,0 +1,604 @@ +//! Device-sync v4 storage paths against PostgreSQL. +//! +//! Requires a disposable database in `FURU_DEVICE_SYNC_TEST_DATABASE_URL`; +//! it is migrated with the production migrations on first use +//! (`crate::podcasts::postgres_tests::test_pool`). The tests write rows for +//! random user ids and remove them again. + +use super::*; +use crate::player::queue_items::test_episode; +use music_dht::podcast::{ + EpisodeId, FeedUrl, PodcastId, PodcastMeta, ProgressRecord, ProgressStatus, ShareRecord, +}; + +fn profile(device_id: &str, protocol_version: u16) -> DeviceProfileWire { + DeviceProfileWire { + device_id: device_id.into(), + name: device_id.to_uppercase(), + client_version: "test".into(), + protocol_version, + endpoint_id: String::new(), + endpoint_ticket: String::new(), + revoked: false, + revoke_cutoff_seq: None, + updated_at_ms: now_ms(), + } +} + +fn op(origin: &str, seq: i64, hlc_ms: i64, payload: SyncOpPayload) -> SyncOpWire { + SyncOpWire { + op_id: format!("{origin}:{seq}"), + origin_device_id: origin.into(), + seq, + hlc_ms, + payload, + } +} + +async fn cleanup(pool: &sqlx::PgPool, user_id: i64) { + for table in [ + "furumusic__fed_device_identity", + "furumusic__fed_device", + "furumusic__fed_device_invite", + "furumusic__fed_peer_ack", + "furumusic__fed_pending_pairing", + "furumusic__fed_playback_applied", + "furumusic__fed_podcast_relay", + "furumusic__fed_state_like", + "furumusic__fed_state_playlist", + "furumusic__fed_state_playlist_item", + "furumusic__fed_sync_ops", + "furumusic__fed_sync_vector", + "furumusic__playback_state", + "furumusic__podcast_fetch_state", + "furumusic__podcast_listen_session", + "furumusic__podcast_progress", + "furumusic__podcast_share", + "furumusic__podcast_subscription", + "furumusic__podcast_episode", + "furumusic__podcast_feed_alias", + "furumusic__podcast", + ] { + sqlx::query(&format!("DELETE FROM {table} WHERE user_id = $1")) + .bind(user_id) + .execute(pool) + .await + .unwrap(); + } +} + +#[tokio::test] +#[ignore = "requires a disposable database in FURU_DEVICE_SYNC_TEST_DATABASE_URL"] +async fn postgres_podcast_relay_gating_and_peer_versions() { + let pool = crate::podcasts::postgres_tests::test_pool().await; + let user_id = 8_000_000_000 + i64::from(rand_suffix()); + cleanup(&pool, user_id).await; + let new = format!("tui_new_{user_id}"); + let old = format!("tui_old_{user_id}"); + let other = format!("tui_other_{user_id}"); + let identity = ensure_identity(&pool, user_id, "tester").await.unwrap(); + + // Peer versions: self-reports replace, relayed profiles only raise. + for (device, version) in [(&new, 4), (&old, 3), (&other, 4)] { + apply_device_profile( + &pool, + user_id, + &profile(device, version), + VersionReport::SelfReported(Some(version)), + true, + ) + .await + .unwrap(); + } + // A v3 relay lists everyone with its own constant. + apply_device_profiles( + &pool, + user_id, + &[profile(&new, 3), profile(&old, 3)], + Some(old.as_str()), + ) + .await + .unwrap(); + assert_eq!( + stored_protocol_version(&pool, user_id, &new).await.unwrap(), + 4 + ); + assert_eq!( + stored_protocol_version(&pool, user_id, &old).await.unwrap(), + 3 + ); + + // Podcast and music operations from the v4 device. + let feed = FeedUrl::parse("https://podcasts.example/feed.xml").unwrap(); + let podcast_id = PodcastId::from_feed_url(&feed); + let episode_id = EpisodeId::from_guid(&podcast_id, "ep-1").unwrap(); + let meta: PodcastMeta = serde_json::from_value(serde_json::json!({ + "feed_url": feed.as_str(), "title": "Example Cast" + })) + .unwrap(); + let progress = |seq, generation, hlc_ms, position_ms| { + op( + &new, + seq, + hlc_ms, + SyncOpPayload::PodcastProgressSet { + episode_id: episode_id.clone(), + podcast_id: podcast_id.clone(), + position_ms, + status: ProgressStatus::InProgress, + generation, + session_id: format!("s{generation}"), + episode: None, + }, + ) + }; + let mut invalid = progress(5, 9, 50, 1); + if let SyncOpPayload::PodcastProgressSet { session_id, .. } = &mut invalid.payload { + session_id.clear(); + } + let hub = Arc::new(PlayerDeviceHub::default()); + apply_ops( + &pool, + Arc::clone(&hub), + user_id, + vec![ + op( + &new, + 1, + 10, + SyncOpPayload::PodcastSubscriptionSet { + podcast_id: podcast_id.clone(), + subscribed: true, + podcast: meta, + feed_aliases: vec![], + generation: 1, + }, + ), + // Newer generation first, then a late checkpoint of an older one. + progress(2, 4, 20, 10_000), + progress(3, 3, 90_000, 600_000), + op( + &new, + 4, + 40, + SyncOpPayload::TrackLikeSet { + content_id: format!("b3:{}", "3".repeat(64)), + liked: false, + fed: None, + }, + ), + invalid, + ], + ) + .await + .unwrap(); + // The invalid operation is not stored but acknowledged. + assert_eq!(vector(&pool, user_id).await.unwrap().get(&new), Some(&5)); + let stored: i64 = sqlx::query_scalar( + "SELECT COUNT(*) FROM furumusic__fed_sync_ops WHERE user_id = $1 AND origin_device_id = $2", + ) + .bind(user_id) + .bind(&new) + .fetch_one(&pool) + .await + .unwrap(); + assert_eq!(stored, 4); + + // Relay: the v3 peer gets only music, a v4 peer gets everything. + let kinds = |ops: Vec| { + ops.iter() + .map(|op| payload_kind(&op.payload)) + .collect::>() + }; + let to_old = kinds(ops_for_peer(&pool, user_id, &old, 3).await.unwrap()); + assert!(to_old.iter().all(|kind| !kind.starts_with("podcast_"))); + assert!(to_old.contains(&"track_like_set")); + let to_other = kinds(ops_for_peer(&pool, user_id, &other, 4).await.unwrap()); + assert_eq!( + to_other + .iter() + .filter(|kind| kind.starts_with("podcast_")) + .count(), + 3 + ); + + // Snapshot recovery: registers merged with the shared rules. + apply_snapshot( + &pool, + user_id, + SyncSnapshot { + podcast_shares: vec![ShareRecord { + podcast_id: podcast_id.clone(), + shared: false, + generation: 2, + hlc_ms: 60, + op_id: format!("{other}:1"), + }], + podcast_progress: vec![ProgressRecord { + episode_id: episode_id.clone(), + podcast_id: podcast_id.clone(), + position_ms: 1, + status: ProgressStatus::InProgress, + generation: 2, + session_id: "old".into(), + hlc_ms: 999_999, + device_id: other.clone(), + op_id: format!("{other}:2"), + }], + ..SyncSnapshot::default() + }, + ) + .await + .unwrap(); + let full = snapshot(&pool, user_id).await.unwrap(); + assert_eq!(full.podcast_subscriptions.len(), 1); + assert_eq!(full.podcast_shares.len(), 1); + assert_eq!(full.podcast_progress.len(), 1); + assert_eq!(full.podcast_progress[0].position_ms, 10_000); + assert_eq!(full.podcast_progress[0].generation, 4); + assert!(!full.for_peer(3).has_podcast_state()); + assert_eq!(full.for_peer(4), full); + + // A DeviceProfileSet with a version is a self-report of its origin. + apply_ops( + &pool, + Arc::clone(&hub), + user_id, + vec![op( + &old, + 1, + 70, + SyncOpPayload::DeviceProfileSet { + name: "OLD".into(), + client_version: "0.4.0".into(), + endpoint_ticket: String::new(), + endpoint_id: String::new(), + protocol_version: Some(4), + }, + )], + ) + .await + .unwrap(); + assert_eq!( + stored_protocol_version(&pool, user_id, &old).await.unwrap(), + 4 + ); + + // First start after the upgrade announces the new version once. + sqlx::query( + "UPDATE furumusic__fed_device SET protocol_version = 3 + WHERE user_id = $1 AND device_id = $2", + ) + .bind(user_id) + .bind(&identity.device_id) + .execute(&pool) + .await + .unwrap(); + own_profile(&pool, user_id, "", "").await.unwrap(); + own_profile(&pool, user_id, "", "").await.unwrap(); + let announced: Vec = sqlx::query_scalar( + "SELECT payload_json FROM furumusic__fed_sync_ops + WHERE user_id = $1 AND origin_device_id = $2 AND kind = 'device_profile_set'", + ) + .bind(user_id) + .bind(&identity.device_id) + .fetch_all(&pool) + .await + .unwrap(); + assert_eq!(announced.len(), 1); + assert_eq!(announced[0]["protocol_version"], 4); + + // Handoff gating: refused for the v3 device, allowed for v4. + let queue = PlaybackStateWire { + queue: vec![PlaybackTrack::podcast_episode(test_episode())], + queue_pos: 0, + playing: true, + paused: false, + idle_since_ms: None, + position_secs: 0.0, + volume: 50, + shuffle: false, + repeat: PlaybackRepeat::Off, + playback_rate: None, + }; + sqlx::query( + "UPDATE furumusic__fed_device SET protocol_version = 3 + WHERE user_id = $1 AND device_id = $2", + ) + .bind(user_id) + .bind(&old) + .execute(&pool) + .await + .unwrap(); + let refused = ensure_playback_supported_by(&pool, user_id, &old, &queue) + .await + .unwrap_err(); + assert!(refused.to_string().contains("needs an update")); + ensure_playback_supported_by(&pool, user_id, &other, &queue) + .await + .unwrap(); + + // The typed queue column exists next to queue_json. + sqlx::query("SELECT queue_json, queue_items_json FROM furumusic__playback_state LIMIT 0") + .execute(&pool) + .await + .unwrap(); + + cleanup(&pool, user_id).await; +} + +#[tokio::test] +#[ignore = "requires a disposable database in FURU_DEVICE_SYNC_TEST_DATABASE_URL"] +async fn postgres_unknown_episode_position_and_rollback_cleanup() { + let pool = crate::podcasts::postgres_tests::test_pool().await; + let user_id = 8_000_000_000 + i64::from(rand_suffix()); + cleanup(&pool, user_id).await; + let tui = format!("tui_{user_id}"); + ensure_identity(&pool, user_id, "tester").await.unwrap(); + apply_device_profile( + &pool, + user_id, + &profile(&tui, 4), + VersionReport::SelfReported(Some(4)), + true, + ) + .await + .unwrap(); + + // The TUI left the episode at 1200 s (generation 9; queue items saw 7). + let episode = test_episode(); + let hub = Arc::new(PlayerDeviceHub::default()); + apply_ops( + &pool, + Arc::clone(&hub), + user_id, + vec![op( + &tui, + 1, + 10, + SyncOpPayload::PodcastProgressSet { + episode_id: episode.episode_id.clone(), + podcast_id: episode.podcast_id.clone(), + position_ms: 1_200_000, + status: ProgressStatus::InProgress, + generation: 9, + session_id: "s9".into(), + episode: None, + }, + )], + ) + .await + .unwrap(); + + // The web player reached the episode by Next and does not know where it + // was left; it hands playback to the TUI. + let song = serde_json::json!({ + "id": 77, "title": "Library song", "content_id": format!("b3:{}", "4".repeat(64)), + "duration_seconds": 300.0, + }); + let item = crate::player::queue_items::external_track_json( + &PlaybackTrack::podcast_episode(episode.clone()), + 1, + None, + ); + let browser = serde_json::json!({ + "tracks": [song.clone(), item.clone()], "index": 1, "track": item.clone(), + "position_seconds": 0.0, "paused": true, "position_unknown": true, + }); + record_web_active_transfer(&pool, user_id, &tui, None, browser) + .await + .unwrap(); + let sent: serde_json::Value = sqlx::query_scalar( + "SELECT payload_json FROM furumusic__fed_sync_ops + WHERE user_id = $1 AND kind = 'playback_command' ORDER BY seq DESC LIMIT 1", + ) + .bind(user_id) + .fetch_one(&pool) + .await + .unwrap(); + let SyncOpPayload::PlaybackCommand { + command: PlaybackCommand::ActiveChanged { state, .. }, + .. + } = serde_json::from_value(sent).unwrap() + else { + panic!("expected ActiveChanged"); + }; + assert_eq!(state.position_secs, 1200.0, "never 0"); + assert_eq!( + state.queue[1] + .podcast_episode_ref() + .unwrap() + .progress_generation, + Some(9) + ); + + // A known position (the web player was playing the episode) is sent + // as is, but the stale handoff hint of the browser item (7) is raised + // to the register generation this account has seen (9). + let browser = serde_json::json!({ + "tracks": [song.clone(), item.clone()], "index": 1, "track": item.clone(), + "position_seconds": 1500.0, "paused": true, + }); + record_web_active_transfer(&pool, user_id, &tui, None, browser) + .await + .unwrap(); + let sent: serde_json::Value = sqlx::query_scalar( + "SELECT payload_json FROM furumusic__fed_sync_ops + WHERE user_id = $1 AND kind = 'playback_command' ORDER BY seq DESC LIMIT 1", + ) + .bind(user_id) + .fetch_one(&pool) + .await + .unwrap(); + let SyncOpPayload::PlaybackCommand { + command: PlaybackCommand::ActiveChanged { state, .. }, + .. + } = serde_json::from_value(sent).unwrap() + else { + panic!("expected ActiveChanged"); + }; + assert_eq!(state.position_secs, 1500.0); + assert_eq!( + state.queue[1] + .podcast_episode_ref() + .unwrap() + .progress_generation, + Some(9) + ); + + // Remote Next onto the episode while the TUI plays the song. + let current = serde_json::json!({ + "tracks": [song, item], "index": 0, "track": song, + "position_seconds": 200.0, "paused": false, + }); + let next = record_web_playback_command( + &pool, + user_id, + &tui, + "next", + serde_json::json!({}), + Some(current), + ) + .await + .unwrap(); + assert_eq!(next["index"], 1); + assert_eq!(next["position_seconds"], 1200.0); + assert!( + next.get(crate::player::queue_items::POSITION_UNKNOWN_KEY) + .is_none() + ); + + // Rollback to 0.12: its types fail on the relayed podcast operation, and + // parse everything once the documented cleanup ran. + let parse_like_0_12 = |rows: Vec| { + rows.into_iter() + .map(serde_json::from_value::) + .collect::, _>>() + }; + let rows = || async { + sqlx::query_scalar::<_, serde_json::Value>( + "SELECT payload_json FROM furumusic__fed_sync_ops WHERE user_id = $1", + ) + .bind(user_id) + .fetch_all(&pool) + .await + .unwrap() + }; + assert!(parse_like_0_12(rows().await).is_err()); + sqlx::query(&format!( + "{} AND user_id = $1", + crate::music::db_migrations::DELETE_PODCAST_SYNC_OPS_SQL + )) + .bind(user_id) + .execute(&pool) + .await + .unwrap(); + let remaining = parse_like_0_12(rows().await).unwrap(); + // The playback commands (with the episode's `source`) stay readable. + assert_eq!( + remaining + .iter() + .filter(|payload| matches!( + payload, + super::wire_compat_tests::legacy::SyncOpPayload::PlaybackCommand { .. } + )) + .count(), + 3 + ); + + cleanup(&pool, user_id).await; +} + +#[tokio::test] +#[ignore = "requires a disposable database in FURU_DEVICE_SYNC_TEST_DATABASE_URL"] +async fn postgres_local_podcast_actions_reach_v4_peers_only() { + let pool = crate::podcasts::postgres_tests::test_pool().await; + let user_id = 8_000_000_000 + i64::from(rand_suffix()); + cleanup(&pool, user_id).await; + let identity = ensure_identity(&pool, user_id, "tester").await.unwrap(); + let old = format!("tui_old_{user_id}"); + let new = format!("tui_new_{user_id}"); + for (device, version) in [(&old, 3), (&new, 4)] { + apply_device_profile( + &pool, + user_id, + &profile(device, version), + VersionReport::SelfReported(Some(version)), + true, + ) + .await + .unwrap(); + } + + // A local action lands in the log with this device's stamp and in the + // store, in one transaction. + let store = crate::podcasts::PodcastStore::new(pool.clone()); + let episode = test_episode(); + let meta: PodcastMeta = serde_json::from_value(serde_json::json!({ + "feed_url": episode.feed_url.as_str(), "title": episode.podcast_title, + })) + .unwrap(); + let subscribed = store.subscribe(user_id, &meta).await.unwrap(); + let played = store.mark_played(user_id, &episode).await.unwrap(); + assert!( + subscribed + .ops + .iter() + .chain([&played.op]) + .all(|op| op.origin_device_id == identity.device_id) + ); + // A music operation recorded afterwards keeps the same clock and sequence. + record_local_op( + &pool, + user_id, + SyncOpPayload::TrackLikeSet { + content_id: format!("b3:{}", "5".repeat(64)), + liked: false, + fed: None, + }, + ) + .await + .unwrap(); + let own: Vec = ops_for_peer(&pool, user_id, &new, 4) + .await + .unwrap() + .into_iter() + .filter(|op| op.origin_device_id == identity.device_id) + .collect(); + let kinds: Vec<_> = own.iter().map(|op| payload_kind(&op.payload)).collect(); + assert!(kinds.ends_with(&[ + "podcast_subscription_set", + "podcast_progress_set", + "track_like_set" + ])); + assert!( + own.windows(2) + .all(|pair| pair[1].seq > pair[0].seq && pair[1].hlc_ms > pair[0].hlc_ms) + ); + assert_eq!(own[own.len() - 3], subscribed.ops[0]); + assert_eq!(own[own.len() - 2], played.op); + // The v3 peer gets the music operation only. + let to_old = ops_for_peer(&pool, user_id, &old, 3).await.unwrap(); + assert!(to_old.iter().all(|op| !op.payload.is_podcast())); + assert!( + to_old + .iter() + .any(|op| payload_kind(&op.payload) == "track_like_set") + ); + // The snapshot carries the store's registers for v4 only. + let full = snapshot(&pool, user_id).await.unwrap(); + assert_eq!(full.podcast_subscriptions.len(), 1); + assert_eq!(full.podcast_progress.len(), 1); + assert_eq!(full.podcast_progress[0].status, ProgressStatus::Completed); + assert!(!full.for_peer(3).has_podcast_state()); + + cleanup(&pool, user_id).await; +} + +fn rand_suffix() -> u32 { + u32::from_le_bytes( + blake3::hash(random_hex(8).as_bytes()).as_bytes()[..4] + .try_into() + .unwrap(), + ) % 1_000_000_000 +} diff --git a/src/federation/devices/wire_compat_tests.rs b/src/federation/devices/wire_compat_tests.rs new file mode 100644 index 0000000..36f7ac8 --- /dev/null +++ b/src/federation/devices/wire_compat_tests.rs @@ -0,0 +1,524 @@ +//! Golden wire-compatibility tests: the shared `music_dht::device_sync` +//! types must serialize v3 content byte-for-byte like the private copies that +//! furumi-web 0.12.x used (kept verbatim in `legacy` below), so TUI 0.3.x and +//! older web servers keep interoperating. + +use super::*; + +/// Private wire types of furumi-web 0.12.4 (`src/federation/devices.rs`), +/// copied verbatim apart from visibility and `PartialEq`. +#[allow(dead_code)] +pub(super) mod legacy { + use music_dht::device_sync::ListenEvent; + use music_dht::playback::CommandStamp; + use serde::{Deserialize, Serialize}; + use std::collections::BTreeMap; + + #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] + pub struct InviteWire { + pub v: u16, + #[serde(rename = "t")] + pub ticket: String, + #[serde(rename = "d")] + pub device_id: String, + #[serde(rename = "i")] + pub invite_id: String, + #[serde(rename = "s")] + pub secret: String, + #[serde(rename = "e")] + pub expires_at_ms: i64, + } + + #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] + pub struct DeviceProfileWire { + pub device_id: String, + pub name: String, + pub client_version: String, + pub protocol_version: u16, + pub endpoint_id: String, + pub endpoint_ticket: String, + #[serde(default)] + pub revoked: bool, + #[serde(default)] + pub revoke_cutoff_seq: Option, + #[serde(default)] + pub updated_at_ms: i64, + } + + #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] + pub struct SyncedFedTrack { + pub item_id: String, + pub owner: String, + pub title: String, + #[serde(default)] + pub artist_names: Vec, + #[serde(default)] + pub featured_artist_names: Vec, + pub year: Option, + pub duration_seconds: Option, + pub content_id: String, + pub release_title: Option, + pub track_number: Option, + pub disc_number: Option, + } + + #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] + pub struct PlaybackTrack { + pub id: i64, + pub title: String, + pub track_number: Option, + pub disc_number: Option, + pub duration_seconds: f64, + #[serde(default)] + pub artist_names: Vec, + #[serde(default)] + pub featured_artist_names: Vec, + pub release_id: i64, + pub release_title: String, + pub release_year: Option, + #[serde(default)] + pub file_path: String, + pub content_id: Option, + pub audio_format: Option, + pub audio_bitrate: Option, + pub audio_sample_rate: Option, + pub audio_bit_depth: Option, + pub file_size_bytes: Option, + #[serde(default)] + pub play_count: i64, + #[serde(default)] + pub fed: Option, + } + + #[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Default)] + #[serde(rename_all = "snake_case")] + pub enum PlaybackRepeat { + #[default] + Off, + One, + All, + } + + #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] + pub struct PlaybackStateWire { + #[serde(default)] + pub queue: Vec, + #[serde(default)] + pub queue_pos: usize, + pub playing: bool, + pub paused: bool, + #[serde(default)] + pub idle_since_ms: Option, + pub position_secs: f64, + #[serde(default)] + pub volume: u8, + pub shuffle: bool, + pub repeat: PlaybackRepeat, + } + + pub type PlaybackSnapshot = music_dht::playback::Snapshot; + pub type PlaybackCommand = music_dht::playback::Command; + + #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] + pub struct SyncOpWire { + pub op_id: String, + pub origin_device_id: String, + pub seq: i64, + pub hlc_ms: i64, + pub payload: SyncOpPayload, + } + + #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] + #[serde(tag = "kind", rename_all = "snake_case")] + pub enum SyncOpPayload { + TrackLikeSet { + content_id: String, + liked: bool, + #[serde(default)] + fed: Option, + }, + PlaylistCreated { + playlist_id: String, + title: String, + }, + PlaylistRenamed { + playlist_id: String, + title: String, + }, + PlaylistDeleted { + playlist_id: String, + }, + PlaylistTrackAdded { + playlist_id: String, + content_id: String, + position: i64, + #[serde(default)] + fed: Option, + }, + PlaylistTrackRemoved { + playlist_id: String, + content_id: String, + }, + DeviceProfileSet { + name: String, + client_version: String, + endpoint_ticket: String, + endpoint_id: String, + }, + DeviceTrusted { + target_device_id: String, + }, + DeviceRevoked { + target_device_id: String, + target_max_seq_seen: i64, + }, + PlaybackCommand { + target_device_id: String, + command: PlaybackCommand, + #[serde(default)] + authority: Option, + }, + ListenRecorded { + event: ListenEvent, + }, + } + + #[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)] + pub struct SyncSnapshot { + #[serde(default)] + pub likes: Vec, + #[serde(default)] + pub unlikes: Vec, + #[serde(default)] + pub playlists: Vec, + #[serde(default)] + pub deleted_playlists: Vec, + #[serde(default)] + pub removed_playlist_items: Vec, + } + + #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] + pub struct SnapshotLike { + pub content_id: String, + pub hlc_ms: i64, + pub op_id: String, + #[serde(default)] + pub fed: Option, + } + + #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] + pub struct SnapshotLikeTombstone { + pub content_id: String, + pub hlc_ms: i64, + pub op_id: String, + } + + #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] + pub struct SnapshotPlaylist { + pub playlist_id: String, + pub title: String, + pub hlc_ms: i64, + pub op_id: String, + #[serde(default)] + pub items: Vec, + } + + #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] + pub struct SnapshotPlaylistTombstone { + pub playlist_id: String, + pub hlc_ms: i64, + pub op_id: String, + } + + #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] + pub struct SnapshotPlaylistItem { + pub content_id: String, + pub position: i64, + pub hlc_ms: i64, + pub op_id: String, + #[serde(default)] + pub fed: Option, + } + + #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] + pub struct SnapshotPlaylistItemTombstone { + pub playlist_id: String, + pub content_id: String, + pub hlc_ms: i64, + pub op_id: String, + } + + #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] + #[serde(tag = "type", rename_all = "snake_case")] + pub enum WireMessage { + PairRequest { + invite_id: String, + secret: String, + profile: DeviceProfileWire, + #[serde(default)] + group_id: Option, + #[serde(default)] + group_active_devices: usize, + #[serde(default)] + devices: Vec, + vector: BTreeMap, + ops: Vec, + snapshot: SyncSnapshot, + #[serde(default)] + playback: Option, + }, + PairResponse { + accepted: bool, + #[serde(default)] + pending: bool, + #[serde(default)] + error: Option, + #[serde(default)] + group_id: Option, + #[serde(default)] + profile: Option, + #[serde(default)] + devices: Vec, + #[serde(default)] + vector: BTreeMap, + #[serde(default)] + ops: Vec, + #[serde(default)] + snapshot: SyncSnapshot, + #[serde(default)] + playback: Option, + }, + Hello { + group_id: String, + profile: DeviceProfileWire, + devices: Vec, + vector: BTreeMap, + ops: Vec, + snapshot: SyncSnapshot, + #[serde(default)] + playback: Option, + }, + SyncResponse { + accepted: bool, + #[serde(default)] + error: Option, + #[serde(default)] + devices: Vec, + #[serde(default)] + vector: BTreeMap, + #[serde(default)] + ops: Vec, + #[serde(default)] + snapshot: SyncSnapshot, + #[serde(default)] + playback: Option, + }, + } +} + +const CONTENT_ID: &str = "b3:1111111111111111111111111111111111111111111111111111111111111111"; + +/// A v3 `hello` exercising every v3 operation kind, every snapshot section +/// and a coordinated playback snapshot (input order is irrelevant). +fn v3_hello_input() -> String { + let fed = serde_json::json!({ + "item_id": "item", "owner": "owner", "title": "Song", "artist_names": ["A"], + "featured_artist_names": [], "year": 2001, "duration_seconds": 180, + "content_id": CONTENT_ID, "release_title": "Album", "track_number": 1, "disc_number": 1, + }); + let track = serde_json::json!({ + "id": 7, "title": "Song", "track_number": 1, "disc_number": null, + "duration_seconds": 180.5, "artist_names": ["A"], "featured_artist_names": ["B"], + "release_id": 3, "release_title": "Album", "release_year": 2001, "file_path": "", + "content_id": CONTENT_ID, "audio_format": "flac", "audio_bitrate": 900, + "audio_sample_rate": 44100, "audio_bit_depth": 16, "file_size_bytes": 12345, + "play_count": 2, "fed": fed, + }); + let state = serde_json::json!({ + "queue": [track, {"id": 0, "title": "Remote", "track_number": null, "disc_number": null, + "duration_seconds": 0.0, "release_id": 0, "release_title": "", "release_year": null, + "content_id": null, "audio_format": null, "audio_bitrate": null, + "audio_sample_rate": null, "audio_bit_depth": null, "file_size_bytes": null}], + "queue_pos": 1, "playing": true, "paused": false, "idle_since_ms": null, + "position_secs": 42.5, "volume": 73, "shuffle": true, "repeat": "all", + }); + let claim = serde_json::json!({"counter": 3, "issuer": "tui_a", "owner": "tui_a"}); + let stamp = serde_json::json!({"version": 1, "claim": claim, "sequence": 9}); + let op = |seq: i64, payload: serde_json::Value| { + serde_json::json!({"op_id": format!("tui_a:{seq}"), "origin_device_id": "tui_a", + "seq": seq, "hlc_ms": 1_700_000_000_000i64 + seq, "payload": payload}) + }; + let profile = |id: &str| { + serde_json::json!({"device_id": id, "name": "TUI", "client_version": "0.3.4", + "protocol_version": 3, "endpoint_id": "e", "endpoint_ticket": "t", + "revoked": false, "revoke_cutoff_seq": null, "updated_at_ms": 5}) + }; + serde_json::json!({ + "type": "hello", + "group_id": "grp_1", + "profile": profile("tui_a"), + "devices": [profile("tui_a"), profile("web_b")], + "vector": {"tui_a": 12, "web_b": 4}, + "ops": [ + op(1, serde_json::json!({"kind": "track_like_set", "content_id": CONTENT_ID, "liked": true, "fed": fed})), + op(2, serde_json::json!({"kind": "playlist_created", "playlist_id": "pl", "title": "Mix"})), + op(3, serde_json::json!({"kind": "playlist_renamed", "playlist_id": "pl", "title": "Mix 2"})), + op(4, serde_json::json!({"kind": "playlist_track_added", "playlist_id": "pl", "content_id": CONTENT_ID, "position": 0, "fed": null})), + op(5, serde_json::json!({"kind": "playlist_track_removed", "playlist_id": "pl", "content_id": CONTENT_ID})), + op(6, serde_json::json!({"kind": "playlist_deleted", "playlist_id": "pl"})), + op(7, serde_json::json!({"kind": "device_profile_set", "name": "TUI", "client_version": "0.3.4", "endpoint_ticket": "t", "endpoint_id": "e"})), + op(8, serde_json::json!({"kind": "device_trusted", "target_device_id": "web_b"})), + op(9, serde_json::json!({"kind": "device_revoked", "target_device_id": "old", "target_max_seq_seen": 3})), + op(10, serde_json::json!({"kind": "playback_command", "target_device_id": "web_b", + "command": {"kind": "set_state", "state": state, "seek": true}, "authority": stamp})), + op(11, serde_json::json!({"kind": "playback_command", "target_device_id": "web_b", + "command": {"kind": "active_changed", "active_device_id": "web_b", "active_device_name": "WEB", "state": state}})), + op(12, serde_json::json!({"kind": "listen_recorded", "event": {"listen_id": "l1", + "content_id": CONTENT_ID, "started_at_ms": 1, "listened_ms": 60000, "track_duration_ms": 180000, + "ended_reason": "finished", "track": {"title": "Song", "artist_names": ["A"], "featured_artist_names": [], "release_title": null}}})), + ], + "snapshot": { + "likes": [{"content_id": CONTENT_ID, "hlc_ms": 1, "op_id": "tui_a:1", "fed": fed}], + "unlikes": [{"content_id": CONTENT_ID, "hlc_ms": 2, "op_id": "tui_a:2"}], + "playlists": [{"playlist_id": "pl", "title": "Mix", "hlc_ms": 3, "op_id": "tui_a:3", + "items": [{"content_id": CONTENT_ID, "position": 0, "hlc_ms": 4, "op_id": "tui_a:4", "fed": null}]}], + "deleted_playlists": [{"playlist_id": "gone", "hlc_ms": 5, "op_id": "tui_a:5"}], + "removed_playlist_items": [{"playlist_id": "pl", "content_id": CONTENT_ID, "hlc_ms": 6, "op_id": "tui_a:6"}], + }, + "playback": {"device_id": "tui_a", "device_name": "TUI", "active": true, "updated_at_ms": 9, + "state": state, "coordination": {"version": 1, "claim": claim, "heartbeat": 4, "available": true, "playing": true}}, + }) + .to_string() +} + +fn v3_other_inputs() -> Vec { + vec![ + r#"{"type":"sync_response","accepted":false,"error":"device is not trusted"}"#.into(), + r#"{"type":"pair_response","accepted":false,"pending":true,"error":"pairing pending"}"# + .into(), + serde_json::json!({"type": "pair_request", "invite_id": "i", "secret": "s", + "profile": {"device_id": "tui_a", "name": "TUI", "client_version": "0.3.4", + "protocol_version": 3, "endpoint_id": "e", "endpoint_ticket": "t"}, + "vector": {}, "ops": [], "snapshot": {}}) + .to_string(), + ] +} + +/// Parses `input` with the 0.12.x types and the shared types and checks +/// that both serialize to the same bytes, and that each side parses the +/// other's output. +fn assert_identical(input: &str) -> String { + let old: legacy::WireMessage = serde_json::from_str(input).unwrap(); + let new: WireMessage = serde_json::from_str(input).unwrap(); + let old_json = serde_json::to_string(&old).unwrap(); + let new_json = serde_json::to_string(&new).unwrap(); + assert_eq!(old_json, new_json); + assert_eq!( + serde_json::from_str::(&new_json).unwrap(), + old + ); + assert_eq!(serde_json::from_str::(&old_json).unwrap(), new); + new_json +} + +#[test] +fn v3_messages_serialize_like_furumi_web_0_12() { + let hello = assert_identical(&v3_hello_input()); + let golden = concat!( + env!("CARGO_MANIFEST_DIR"), + "/src/federation/devices/fixtures/v3_hello.golden.json" + ); + if std::env::var_os("FURUMUSIC_BLESS_GOLDEN").is_some() { + // Only for a deliberate change of the v3 shape. + std::fs::write(golden, format!("{hello}\n")).unwrap(); + } + assert_eq!(hello, std::fs::read_to_string(golden).unwrap().trim_end()); + for input in v3_other_inputs() { + assert_identical(&input); + } +} + +#[test] +fn invites_stay_readable_by_older_clients() { + let old = legacy::InviteWire { + v: 1, + ticket: "ticket".into(), + device_id: "web_a".into(), + invite_id: "inv".into(), + secret: "s3cr3t".into(), + expires_at_ms: 10, + }; + let old_json = serde_json::to_string(&old).unwrap(); + assert_eq!( + old_json, + r#"{"v":1,"t":"ticket","d":"web_a","i":"inv","s":"s3cr3t","e":10}"# + ); + let shared: InviteWire = serde_json::from_str(&old_json).unwrap(); + assert_eq!(shared.inviter_protocol_version(), 1); + assert_eq!(serde_json::to_string(&shared).unwrap(), old_json); + let link = encode_invite(&InviteWire { + protocol_version: Some(PROTOCOL_VERSION), + ..shared + }) + .unwrap(); + let parsed = parse_invite(&link).unwrap(); + assert_eq!(parsed.inviter_protocol_version(), PROTOCOL_VERSION); + // `p` is an unknown field for 0.12.x and is ignored there. + let token = link.strip_prefix("frid://i/").unwrap(); + assert!(!token.is_empty()); + let json = serde_json::to_string(&parsed).unwrap(); + assert!(json.ends_with(r#","p":4}"#)); + assert_eq!( + serde_json::from_str::(&json).unwrap(), + old + ); +} + +#[test] +fn v3_peers_cannot_parse_podcast_operations() { + // Why podcast operations must be gated: an unknown `kind` breaks the + // whole message for a 0.12.x/TUI 0.3.x peer, not one operation. + let mut hello: serde_json::Value = serde_json::from_str(&v3_hello_input()).unwrap(); + hello["ops"].as_array_mut().unwrap().push(serde_json::json!({ + "op_id": "tui_a:13", "origin_device_id": "tui_a", "seq": 13, "hlc_ms": 13, + "payload": {"kind": "podcast_share_set", + "podcast_id": music_dht::podcast::PodcastId::from_feed_url( + &music_dht::podcast::FeedUrl::parse("https://podcasts.example/feed.xml").unwrap()), + "shared": false, "generation": 1}, + })); + assert!(serde_json::from_value::(hello.clone()).is_err()); + assert!(serde_json::from_value::(hello).is_ok()); +} + +#[test] +fn v3_peers_would_drop_the_source_of_podcast_items() { + // Why podcast playback must be gated: older types ignore `source` and + // forward the episode as a broken music item. + let episode = PlaybackTrack::podcast_episode(crate::player::queue_items::test_episode()); + let json = serde_json::to_string(&episode).unwrap(); + assert!(json.contains(r#""source":{"kind":"podcast_episode""#)); + let old: legacy::PlaybackTrack = serde_json::from_str(&json).unwrap(); + let forwarded: PlaybackTrack = + serde_json::from_str(&serde_json::to_string(&old).unwrap()).unwrap(); + assert_eq!(forwarded.source, None); + assert_eq!(forwarded.id, 0); +} + +#[test] +fn v4_profile_announcements_stay_readable_by_older_clients() { + let announcement = SyncOpPayload::DeviceProfileSet { + name: "WEB".into(), + client_version: "0.13.0".into(), + endpoint_ticket: "t".into(), + endpoint_id: "e".into(), + protocol_version: Some(PROTOCOL_VERSION), + }; + let json = serde_json::to_string(&announcement).unwrap(); + assert!(json.ends_with(r#","protocol_version":4}"#)); + // 0.12.x and TUI 0.3.x ignore the new field. + let old: legacy::SyncOpPayload = serde_json::from_str(&json).unwrap(); + assert!(matches!( + old, + legacy::SyncOpPayload::DeviceProfileSet { .. } + )); +} diff --git a/src/federation/mod.rs b/src/federation/mod.rs index 435b019..a59c261 100644 --- a/src/federation/mod.rs +++ b/src/federation/mod.rs @@ -9,6 +9,11 @@ //! for catalog metadata and, when enabled, compatible similarity embeddings; //! local playback and the local library remain independent of the network. //! +//! The node also runs the public podcast-discovery overlay +//! ([`podcast_discovery`], `music_dht::podcast_dht`): it publishes cards of +//! the podcasts that accounts explicitly shared (`crate::podcasts::discovery`) +//! under the server's node identity. +//! //! Settings are the regular admin config entries (`federation_enabled`, //! `federation_network_id`, `federation_save_on_listen`) and apply on the fly — //! saving the settings starts, stops or re-joins the node without a server @@ -17,6 +22,7 @@ mod capabilities; pub mod client; pub mod devices; +pub(crate) mod podcast_discovery; mod receive; mod serve; mod similarity; @@ -30,6 +36,7 @@ use std::time::{Duration, Instant}; use anyhow::{Context, Result}; use music_dht::capabilities::CAPABILITIES_ALPN; +use music_dht::podcast_dht::{PODCAST_DHT_ALPN, PodcastDht}; use music_dht::similarity_dht::SimilarityDht; use music_dht::similarity_lsh::SIMILARITY_DHT_ALPN; use music_dht::{ @@ -60,6 +67,8 @@ const TRANSPORT_SAMPLE_LIMIT: usize = 16; struct Running { service: Arc, similarity_dht: Arc, + /// Public podcast discovery; `None` when the overlay failed to start. + podcast_dht: Option>, network_name: String, tasks: Vec>, } @@ -290,6 +299,11 @@ pub struct Federation { download_locks: std::sync::Mutex>>>, pool: tokio::sync::OnceCell, running: tokio::sync::Mutex>, + /// The running podcast-discovery overlay, readable without waiting for + /// `running` (held while the node starts). + podcast_dht: std::sync::Mutex>>, + /// The node runs, but the podcast-discovery overlay failed to start. + podcast_discovery_failed: AtomicBool, sync_lock: tokio::sync::Mutex<()>, supervisor_started: AtomicBool, recovery_count: AtomicU64, @@ -325,6 +339,8 @@ pub fn handle() -> Arc { download_locks: std::sync::Mutex::new(Default::default()), pool: tokio::sync::OnceCell::new(), running: tokio::sync::Mutex::new(None), + podcast_dht: std::sync::Mutex::new(None), + podcast_discovery_failed: AtomicBool::new(false), sync_lock: tokio::sync::Mutex::new(()), supervisor_started: AtomicBool::new(false), recovery_count: AtomicU64::new(0), @@ -474,6 +490,9 @@ impl Federation { } else if recovery_only { return Ok(false); } + *lock(&self.podcast_dht) = None; + self.podcast_discovery_failed + .store(false, Ordering::Relaxed); stop_running(guard.take()).await; } else if recovery_only { tracing::warn!(network = %network_name, "retrying stopped federation service"); @@ -498,6 +517,7 @@ impl Federation { .schema_independent_stream_protocol(SIMILARITY_ALPN) .schema_independent_stream_protocol(SIMILARITY_DHT_ALPN) .schema_independent_stream_protocol(CAPABILITIES_ALPN) + .schema_independent_stream_protocol(PODCAST_DHT_ALPN) .build() .map_err(|err| anyhow::anyhow!("invalid federation config: {err}"))?; let (service, mut events) = @@ -530,6 +550,17 @@ impl Federation { Arc::clone(&similarity_manager), )); + // Public podcast discovery: the server node publishes the podcasts + // accounts explicitly shared (see `crate::podcasts::discovery`), + // after the first device-sync round of this run. + let (first_device_sync_done, first_device_sync) = tokio::sync::watch::channel(false); + let podcast_discovery = + podcast_discovery::start(&service, &self.data_dir, pool.clone(), first_device_sync) + .await; + let podcast_dht = podcast_discovery + .as_ref() + .map(|discovery| Arc::clone(&discovery.dht)); + // Drain DHT events into the log; the channel is bounded. let event_task = tokio::spawn(async move { while let Some(event) = events.recv().await { @@ -584,11 +615,15 @@ impl Federation { Arc::clone(&service), device_hub, Arc::clone(&self.transport_stats), + first_device_sync_done, )); let capabilities_acceptor = service .stream_acceptor(CAPABILITIES_ALPN) .map_err(|err| anyhow::anyhow!("failed to take the capabilities acceptor: {err}"))?; - let capabilities_task = tokio::spawn(capabilities::serve(capabilities_acceptor)); + let capabilities_task = tokio::spawn(capabilities::serve( + capabilities_acceptor, + podcast_dht.is_some(), + )); let similarity_acceptor = service .stream_acceptor(SIMILARITY_ALPN) .map_err(|err| anyhow::anyhow!("failed to take the similarity acceptor: {err}"))?; @@ -599,23 +634,31 @@ impl Federation { Arc::clone(&self.transport_stats), )); + let mut tasks = vec![ + event_task, + sync_task, + audio_task, + catalog_task, + device_task, + device_sync_task, + capabilities_task, + similarity_task, + similarity_dht_serve_task, + similarity_dht_maintenance_task, + similarity_dht_sync_task, + ]; + if let Some(discovery) = podcast_discovery { + tasks.extend(discovery.tasks); + } + *lock(&self.podcast_dht) = podcast_dht.clone(); + self.podcast_discovery_failed + .store(podcast_dht.is_none(), Ordering::Relaxed); *guard = Some(Running { service, similarity_dht, + podcast_dht, network_name, - tasks: vec![ - event_task, - sync_task, - audio_task, - catalog_task, - device_task, - device_sync_task, - capabilities_task, - similarity_task, - similarity_dht_serve_task, - similarity_dht_maintenance_task, - similarity_dht_sync_task, - ], + tasks, }); self.set_error(None); drop(guard); @@ -659,6 +702,9 @@ impl Federation { async fn stop(&self) { let mut guard = self.running.lock().await; + *lock(&self.podcast_dht) = None; + self.podcast_discovery_failed + .store(false, Ordering::Relaxed); stop_running(guard.take()).await; } @@ -685,6 +731,18 @@ impl Federation { .context("federation is not running") } + /// The public podcast-discovery overlay, while the node runs it. Does + /// not wait for a node that is starting or stopping. + pub fn podcast_discovery(&self) -> Option> { + lock(&self.podcast_dht).clone() + } + + /// Whether the node runs but its podcast-discovery overlay could not + /// start (`podcast_discovery()` is then `None` too). + pub fn podcast_discovery_unavailable(&self) -> bool { + self.podcast_discovery_failed.load(Ordering::Relaxed) + } + async fn spawn_sync_soon(self: &Arc) { if let Ok(service) = self.service().await { let fed = Arc::clone(self); @@ -1018,6 +1076,8 @@ impl Federation { "last_rendezvous_error": health.last_rendezvous_error, "recovery_count": self.recovery_count.load(Ordering::Relaxed), "similarity_routing_peers": running.similarity_dht.known_peers(), + "podcast_discovery": running.podcast_dht.is_some(), + "podcast_discovery_peers": running.podcast_dht.as_ref().map(|dht| dht.known_peers()), "published_items": published, "transport": transport, }) @@ -1192,6 +1252,18 @@ impl Federation { .await } + /// Refuses a transfer that the target device cannot receive (podcast + /// items for a device below protocol v4) before ownership changes. + pub async fn fed_device_web_transfer_check( + &self, + user_id: i64, + target_device_id: &str, + state: serde_json::Value, + ) -> Result<()> { + let pool = self.pool().await?; + devices::check_web_transfer(&pool, user_id, target_device_id, state).await + } + pub async fn fed_device_web_active_takeover( &self, user_id: i64, diff --git a/src/federation/podcast_discovery.rs b/src/federation/podcast_discovery.rs new file mode 100644 index 0000000..a59cb7f --- /dev/null +++ b/src/federation/podcast_discovery.rs @@ -0,0 +1,88 @@ +//! Runs the public podcast-discovery overlay (`music_dht::podcast_dht`) on +//! the web server's federation node. +//! +//! The overlay shares the node's iroh endpoint and identity with the music +//! DHT, like `similarity_dht`: its ALPN is registered as a +//! schema-independent stream protocol, its routing contacts and replica +//! cache live in `podcast-dht.sqlite3` in the federation data directory (a +//! replaceable cache, not durable state), and the capability manifest +//! advertises `podcast_dht` only while it runs. What the node publishes is +//! decided by [`crate::podcasts::discovery`] for all accounts. +//! +//! Failing to open the overlay disables public discovery and nothing else: +//! the federation node, the music DHT and private podcast sync keep running. + +use std::path::Path; +use std::sync::Arc; + +use music_dht::MusicDhtService; +use music_dht::podcast_dht::{PODCAST_DHT_ALPN, PodcastDht}; +use sqlx::PgPool; +use tokio::sync::watch; +use tokio::task::JoinHandle; + +/// Cache file of the overlay in the federation data directory. +pub(crate) const CACHE_FILE: &str = "podcast-dht.sqlite3"; + +/// The running overlay and its tasks (serving, maintenance, publication +/// reconciler, capability probes). +pub(crate) struct PodcastDiscovery { + pub(crate) dht: Arc, + pub(crate) tasks: Vec>, +} + +/// Opens the overlay on `service` (whose config registered +/// [`PODCAST_DHT_ALPN`]) and starts its tasks. Returns `None`, with a +/// warning, when the cache cannot be opened or the acceptor is unavailable. +/// +/// `first_device_sync` turns `true` after the first device-sync round of +/// this federation run; the publication reconciler waits for it. +pub(crate) async fn start( + service: &Arc, + data_dir: &Path, + pool: PgPool, + first_device_sync: watch::Receiver, +) -> Option { + let (dht, serve_task) = open(service, data_dir).await?; + let tasks = vec![ + serve_task, + tokio::spawn(Arc::clone(&dht).maintenance()), + tokio::spawn(crate::podcasts::discovery::run( + pool, + Arc::clone(&dht), + first_device_sync, + )), + tokio::spawn(super::capabilities::probe_loop( + Arc::clone(service), + Arc::clone(&dht), + )), + ]; + tracing::info!("public podcast discovery started"); + Some(PodcastDiscovery { dht, tasks }) +} + +/// Opens the overlay cache and serves overlay requests; the rest of +/// [`start`] (maintenance, reconciler, probes) is not started. +pub(crate) async fn open( + service: &Arc, + data_dir: &Path, +) -> Option<(Arc, JoinHandle<()>)> { + let dht = match PodcastDht::open(Arc::clone(service), data_dir.join(CACHE_FILE)).await { + Ok(dht) => dht, + Err(error) => { + tracing::warn!("public podcast discovery disabled: cannot open its cache: {error}"); + // Refuse overlay streams instead of queueing them. + drop(service.stream_acceptor(PODCAST_DHT_ALPN)); + return None; + } + }; + let acceptor = match service.stream_acceptor(PODCAST_DHT_ALPN) { + Ok(acceptor) => acceptor, + Err(error) => { + tracing::warn!("public podcast discovery disabled: {error}"); + return None; + } + }; + let serve_task = tokio::spawn(Arc::clone(&dht).serve(acceptor)); + Some((dht, serve_task)) +} diff --git a/src/i18n/phrases.rs b/src/i18n/phrases.rs index 777e2d1..ee2fb71 100644 --- a/src/i18n/phrases.rs +++ b/src/i18n/phrases.rs @@ -382,6 +382,101 @@ translations! { player_clear: "Clear" , "Очистить"; player_remove: "Remove" , "Удалить"; player_queue_empty: "Queue is empty" , "Очередь пуста"; + player_queue_podcast_item: "Podcast episode" , "Выпуск подкаста"; + player_queue_private_item: "Private item · hidden by its owner", "Приватный элемент · скрыт владельцем"; + player_podcasts: "Podcasts", "Подкасты"; + player_podcast_add_placeholder: "RSS feed URL", "Адрес RSS-ленты"; + player_podcast_add: "Add", "Добавить"; + player_podcast_adding: "Loading the feed…", "Загружаем ленту…"; + player_podcast_added: "Subscribed", "Подписка добавлена"; + player_podcast_already: "Already subscribed", "Вы уже подписаны"; + player_podcast_private_note: "New subscriptions are private: they sync only between your own devices.", "Новые подписки приватны: они синхронизируются только между вашими устройствами."; + player_podcasts_empty: "No podcasts yet. Add an RSS feed to subscribe.", "Подкастов пока нет. Добавьте RSS-ленту, чтобы подписаться."; + player_podcast_show_unsubscribed: "Show unsubscribed", "Показать без подписки"; + player_podcast_unplayed: "unplayed", "не прослушано"; + player_podcast_episodes: "episodes", "выпусков"; + player_podcast_seasons: "seasons", "сезонов"; + player_podcast_unsubscribed: "Not subscribed", "Нет подписки"; + player_podcast_refresh: "Refresh", "Обновить"; + player_podcast_refreshing: "Checking the feed…", "Проверяем ленту…"; + player_podcast_refresh_updated: "Feed updated", "Лента обновлена"; + player_podcast_refresh_not_modified: "No changes in the feed", "Изменений в ленте нет"; + player_podcast_refresh_failed: "Could not refresh the feed", "Не удалось обновить ленту"; + player_podcast_last_checked: "Checked", "Проверено"; + player_podcast_never_checked: "Not checked yet", "Ещё не проверялась"; + player_podcast_fetch_error: "Last check failed", "Последняя проверка не удалась"; + player_podcast_next_check: "next check", "следующая проверка"; + player_podcast_unsubscribe: "Unsubscribe", "Отписаться"; + player_podcast_unsubscribe_confirm: "Unsubscribe from “{title}”? The feed will no longer be checked. Your progress and listening history are kept: subscribing again continues where you stopped.", "Отписаться от «{title}»? Лента больше не будет проверяться. Прогресс и история прослушиваний сохранятся: при повторной подписке вы продолжите с того же места."; + player_podcast_subscribe: "Subscribe", "Подписаться"; + player_podcast_share: "Share in the federation", "Поделиться в федерации"; + player_podcast_unshare: "Stop sharing", "Перестать делиться"; + player_podcast_sharing: "Saving…", "Сохраняем…"; + player_podcast_shared_badge: "Sharing on", "Публикация включена"; + player_podcast_share_private: "Private: only your own devices see this subscription. Sharing publishes the podcast card (title, description, artwork, feed URL) from this server so that other federation members can find it and subscribe. Your progress, history and account are never published.", "Приватно: подписку видят только ваши устройства. «Поделиться» публикует карточку подкаста (название, описание, обложку, адрес RSS-ленты) от имени этого сервера, чтобы другие участники федерации могли найти его и подписаться. Ваш прогресс, история и аккаунт не публикуются."; + player_podcast_share_published: "Published by this server: other federation members can find this podcast and subscribe. Your progress, history and account stay private.", "Опубликовано этим сервером: участники федерации могут найти этот подкаст и подписаться. Ваш прогресс, история и аккаунт остаются приватными."; + player_podcast_share_publishing: "Shared: the server is publishing the card…", "Включено: сервер публикует карточку…"; + player_podcast_share_discovery_off: "Shared: this server publishes the card once federation is enabled on it.", "Включено: этот сервер опубликует карточку, когда на нём включат федерацию."; + player_podcast_share_discovery_unavailable: "Shared, but this server cannot publish the card: public podcast discovery failed to start (see the federation status).", "Включено, но этот сервер не может опубликовать карточку: публичный поиск подкастов не запустился (см. статус федерации)."; + player_podcast_share_over_limit: "Shared, but not published: this server already publishes as many podcasts as it can, and your part of them is used. The card is published when a place frees up.", "Включено, но не опубликовано: сервер уже публикует столько подкастов, сколько может, и ваша доля занята. Карточка появится, когда освободится место."; + player_podcast_share_failed: "Shared, but this server could not publish the card. It tries again regularly.", "Включено, но этот сервер не смог опубликовать карточку. Он регулярно пробует снова."; + player_podcast_share_pending: "Sharing is on, but the feed URL may contain private data. This server does not publish it until you confirm here; your other players where you already confirmed may publish it themselves.", "Публикация включена, но адрес ленты может содержать личные данные. Этот сервер не опубликует его, пока вы не подтвердите здесь; ваши другие плееры, где вы уже подтвердили, могут публиковать его сами."; + player_podcast_share_review: "Review and publish", "Проверить и опубликовать"; + player_podcast_share_unavailable: "This podcast cannot be published", "Этот подкаст нельзя опубликовать"; + player_podcast_share_dialog_title: "Publish this feed URL?", "Опубликовать адрес ленты?"; + player_podcast_share_dialog_text: "The podcast card is visible to everyone in the federation, and anyone can subscribe with its feed URL. This URL may contain private data:", "Карточку подкаста увидят все участники федерации, и любой сможет подписаться по адресу ленты. Похоже, этот адрес содержит личные данные:"; + player_podcast_share_dialog_hint: "If this is a personal feed (for example, a paid or member subscription), do not publish it: others could use your access.", "Если это личная лента (например, платная подписка или лента для участников), не публикуйте её: другие смогут воспользоваться вашим доступом."; + player_podcast_share_dialog_confirm: "Publish anyway", "Всё равно опубликовать"; + player_podcast_warning_secret_query: "the query parameter «{name}» looks like an access key", "параметр «{name}» похож на ключ доступа"; + player_podcast_warning_private_host: "{host} usually serves personal subscriber feeds", "{host} обычно отдаёт личные ленты подписчиков"; + player_podcast_warning_entropy_query: "the value of the parameter «{name}» looks like a secret token", "значение параметра «{name}» похоже на секретный токен"; + player_podcast_warning_entropy_path: "a part of the address looks like a secret token", "часть адреса похожа на секретный токен"; + player_podcast_warning_other: "the address looks like a personal feed", "адрес похож на личную ленту"; + player_podcast_all_seasons: "All seasons", "Все сезоны"; + player_podcast_season: "Season", "Сезон"; + player_podcast_no_season: "No season", "Без сезона"; + player_podcast_filter_all: "All episodes", "Все выпуски"; + player_podcast_filter_unplayed: "Unplayed", "Не прослушано"; + player_podcast_filter_in_progress: "In progress", "Начато"; + player_podcast_filter_played: "Played", "Прослушано"; + player_podcast_sort_newest: "Newest first", "Сначала новые"; + player_podcast_sort_oldest: "Oldest first", "Сначала старые"; + player_podcast_sort_number: "By number", "По номеру"; + player_podcast_search: "Search episodes", "Поиск по выпускам"; + player_podcast_no_episodes: "No episodes match.", "Подходящих выпусков нет."; + player_podcast_play: "Play", "Слушать"; + player_podcast_resume: "Resume", "Продолжить"; + player_podcast_restart: "Start over", "Начать сначала"; + player_podcast_mark_played: "Mark played", "Прослушано"; + player_podcast_mark_unplayed: "Mark unplayed", "Не прослушано"; + player_podcast_add_queue: "Add to queue", "В очередь"; + player_podcast_left: "left", "осталось"; + player_podcast_saved_at: "Saved at", "Сохранено на"; + player_podcast_played_badge: "Played", "Прослушано"; + player_podcast_played_times: "Completed listens", "Полных прослушиваний"; + player_podcast_not_in_feed: "No longer in the feed", "Больше нет в ленте"; + player_podcast_load_more: "Load more", "Показать ещё"; + player_podcast_history: "Recently listened", "Недавно прослушано"; + player_podcast_jam_blocked: "Podcast episodes are private and cannot be added to a Jam.", "Выпуски подкастов приватны и не добавляются в Jam."; + player_podcast_source_failed: "Could not load this episode from the publisher.", "Не удалось загрузить выпуск у издателя."; + player_podcast_cannot_seek: "The publisher server cannot seek to the saved position. Press Play to listen from the start; until then the saved position is kept.", "Сервер издателя не умеет перематывать к сохранённой позиции. Нажмите Play, чтобы слушать с начала; до этого сохранённая позиция не меняется."; + player_podcast_loading: "Preparing the episode…", "Готовим выпуск…"; + player_podcast_feed: "Feed", "Лента"; + player_podcast_request_failed: "Request failed", "Запрос не выполнен"; + player_podcast_found_title: "Podcasts", "Подкасты"; + player_podcast_type: "Podcast", "Подкаст"; + player_podcast_found_failed: "The podcast search did not answer this time; music results are complete.", "Поиск подкастов сейчас не ответил; результаты по музыке полные."; + player_podcast_found_subscribed: "Already subscribed", "Уже подписан"; + player_podcast_found_open: "Open", "Открыть"; + player_podcast_found_subscribing: "Subscribing…", "Подписываемся…"; + player_podcast_found_done: "Subscribed. The subscription is private: only your devices see it.", "Вы подписались. Подписка приватная: её видят только ваши устройства."; + player_podcast_found_note: "This card was published by a federation member; its title, description and artwork are their claims. Subscribing reads the episodes from the feed below, never from the member, and your subscription stays private.", "Эту карточку опубликовал участник федерации; название, описание и обложка — его данные. При подписке выпуски загружаются из ленты по адресу ниже, а не у участника, и ваша подписка остаётся приватной."; + player_podcast_found_feed: "Feed URL", "Адрес ленты"; + player_podcast_found_loading: "Loading seasons and episodes from the feed…", "Загружаем сезоны и выпуски из ленты…"; + player_podcast_found_retry: "Try again", "Повторить"; + player_podcast_found_latest: "Latest episode", "Последний выпуск"; + player_podcast_found_more: "and {count} more", "и ещё {count}"; + player_podcast_found_moved: "The feed has moved; subscribing follows it.", "Лента переехала; подписка последует за ней."; player_shuffle: "Shuffle" , "Перемешать"; player_repeat: "Repeat" , "Повтор"; player_volume: "Volume" , "Громкость"; diff --git a/src/jobs/mod.rs b/src/jobs/mod.rs index 3e997d4..f9e50de 100644 --- a/src/jobs/mod.rs +++ b/src/jobs/mod.rs @@ -6,6 +6,7 @@ pub mod lastfm_popularity; pub mod lastfm_scrobble; pub mod metadata_backfill; pub mod musicbrainz; +pub mod podcast_refresh; use std::path::{Component, Path, PathBuf}; diff --git a/src/jobs/podcast_refresh.rs b/src/jobs/podcast_refresh.rs new file mode 100644 index 0000000..c330ed2 --- /dev/null +++ b/src/jobs/podcast_refresh.rs @@ -0,0 +1,73 @@ +//! Background refresh of subscribed podcast feeds. +//! +//! Every run takes the podcasts whose next check is due (see +//! [`crate::podcasts::runner`]): healthy feeds are checked about hourly with +//! conditional requests, failing ones back off exponentially up to a day. +//! One request per feed URL serves every account subscribed to it, a +//! bounded number of requests run concurrently, and a failing feed never +//! blocks the others or clears the stored catalog. + +use crate::podcasts::{FeedFetcher, PodcastStore, now_ms, runner}; +use crate::scheduler::{Job, JobContext, JobLog}; + +pub struct PodcastRefreshJob; + +/// Session-level advisory lock: one run at a time across processes. +const ADVISORY_LOCK_ID: i64 = 0x504f_4443_5246_5348; // "PODCRFSH" + +/// Failures listed individually in the run log. +const LOGGED_FAILURES: usize = 20; + +#[async_trait::async_trait] +impl Job for PodcastRefreshJob { + fn name(&self) -> &'static str { + "podcast_refresh" + } + + fn description(&self) -> &'static str { + "Check subscribed podcast feeds for new episodes (metadata only, no audio)" + } + + fn default_cron(&self) -> &'static str { + // Every five minutes; each feed is checked only when due. + "0 */5 * * * *" + } + + async fn run(&self, ctx: &JobContext, log: &mut JobLog) -> anyhow::Result<()> { + let mut conn = ctx.pool.acquire().await?; + // A session lock must never return to the pool, including on + // cancellation; the store below uses other pool connections. + conn.close_on_drop(); + let locked: bool = sqlx::query_scalar("SELECT pg_try_advisory_lock($1)") + .bind(ADVISORY_LOCK_ID) + .fetch_one(&mut *conn) + .await?; + if !locked { + log.info("Podcast refresh is already running; skipping this run"); + return Ok(()); + } + let store = PodcastStore::new(ctx.pool.clone()); + let fetcher = FeedFetcher::new()?; + let result = runner::refresh_due( + &store, + &fetcher, + now_ms(), + runner::RUN_LIMIT, + runner::RUN_CONCURRENCY, + ) + .await; + conn.close().await?; + let summary = result?; + for (podcast_id, error) in summary.failures.iter().take(LOGGED_FAILURES) { + log.warn(&format!("podcast {podcast_id}: {error}")); + } + if summary.failures.len() > LOGGED_FAILURES { + log.warn(&format!( + "... and {} more failed feeds", + summary.failures.len() - LOGGED_FAILURES + )); + } + log.info(&format!("Podcast refresh: {summary}")); + Ok(()) + } +} diff --git a/src/library_cleanup.rs b/src/library_cleanup.rs index 8f77a26..ea38851 100644 --- a/src/library_cleanup.rs +++ b/src/library_cleanup.rs @@ -20,6 +20,7 @@ struct PlaybackStateRow { current_track_id: Option, position_ms: i32, queue_json: String, + queue_items_json: Option, queue_position: i32, } @@ -738,7 +739,7 @@ async fn cleanup_playback_states( ) -> anyhow::Result<()> { let deleted: HashSet = track_ids.iter().copied().collect(); let states: Vec = sqlx::query_as( - "SELECT id, current_track_id, position_ms, queue_json, queue_position FROM furumusic__playback_state FOR UPDATE", + "SELECT id, current_track_id, position_ms, queue_json, queue_items_json, queue_position FROM furumusic__playback_state FOR UPDATE", ) .fetch_all(&mut **transaction) .await?; @@ -746,6 +747,14 @@ async fn cleanup_playback_states( for state in states { let mut queue: Vec = serde_json::from_str(&state.queue_json).unwrap_or_default(); let original_queue = queue.clone(); + let mut typed = crate::player::queue_items::StoredQueue::from_columns( + state.queue_items_json.as_ref(), + &crate::player::queue_items::LegacyColumns { + queue: original_queue.clone(), + current_track_id: state.current_track_id, + queue_position: state.queue_position, + }, + ); queue.retain(|track_id| !deleted.contains(track_id)); let current_track_id = state.current_track_id.filter(|id| !deleted.contains(id)); if queue == original_queue && current_track_id == state.current_track_id { @@ -763,16 +772,28 @@ async fn cleanup_playback_states( .clamp(0, i32::try_from(queue.len() - 1).unwrap_or(i32::MAX)) } }); + if let Some(typed) = &mut typed { + typed.retain_library( + |track_id| !deleted.contains(&track_id), + crate::player::queue_items::LegacyColumns { + queue: queue.clone(), + current_track_id, + queue_position, + }, + ); + } let position_ms = if current_track_id.is_some() { state.position_ms } else { 0 }; let queue_json = serde_json::to_string(&queue)?; + // The typed queue keeps its other items; a stale one is cleared. + let queue_items_json = typed.map(serde_json::to_value).transpose()?; sqlx::query( r#"UPDATE furumusic__playback_state SET current_track_id = $2, position_ms = $3, - queue_json = $4, queue_position = $5 + queue_json = $4, queue_position = $5, queue_items_json = $6 WHERE id = $1"#, ) .bind(state.id) @@ -780,6 +801,7 @@ async fn cleanup_playback_states( .bind(position_ms) .bind(queue_json) .bind(queue_position) + .bind(queue_items_json) .execute(&mut **transaction) .await?; } diff --git a/src/main.rs b/src/main.rs index d3a02c4..1918424 100644 --- a/src/main.rs +++ b/src/main.rs @@ -15,6 +15,7 @@ mod metrics; mod music; mod oidc; mod player; +mod podcasts; mod scheduler; mod similarity; mod torrents; @@ -63,6 +64,7 @@ fn build_registry() -> Arc { registry.register(jobs::metadata_backfill::MetadataBackfillJob); registry.register(jobs::lastfm_popularity::LastfmPopularityJob); registry.register(jobs::lastfm_scrobble::LastfmScrobbleJob); + registry.register(jobs::podcast_refresh::PodcastRefreshJob); Arc::new(registry) } @@ -475,6 +477,10 @@ impl Project for FuruProject { " API:\n", " FURU_SWAGGER_ENABLED Enable Swagger UI at /swagger/ (default: false)\n", "\n", + " Podcasts:\n", + " FURU_PODCAST_AUDIO_PROXY_ENABLED Stream episode audio through this\n", + " server (default: false)\n", + "\n", "QUICK START\n", " export FURU_DATABASE_URL=postgres://user:pass@localhost/furumusic\n", " furumusic run", @@ -579,6 +585,10 @@ impl Project for FuruProject { "/admin", ); apps.register_with_views(api::ApiApp, "/api"); + apps.register_with_views( + podcasts::http::PodcastsApp::new(Arc::clone(&self.app_config)), + "/api/podcasts", + ); apps.register_with_views( player::PlayerApp::new( Arc::clone(&self.app_config), diff --git a/src/music/mod.rs b/src/music/mod.rs index 2ec4b03..5349a56 100644 --- a/src/music/mod.rs +++ b/src/music/mod.rs @@ -2838,6 +2838,220 @@ pub mod db_migrations { &[Operation::custom(add_import_playlist_destinations).build()]; } + // -- M0054: typed playback queue next to the legacy id list ------------ + + // (`m_0053` is taken by `src/user.rs` in the same app.) + + // `queue_json` keeps the library track ids and is still written, so an + // older release can run against the same database. The typed value adds + // the items that are not library tracks (podcast episodes, unresolved + // federated music, Jam placeholders) with their explicit source. + #[cot::db::migrations::migration_op] + async fn add_typed_playback_queue( + ctx: migrations::MigrationContext<'_>, + ) -> cot::db::Result<()> { + ctx.db + .raw( + "ALTER TABLE furumusic__playback_state + ADD COLUMN IF NOT EXISTS queue_items_json JSONB", + ) + .await?; + Ok(()) + } + + #[cot::db::migrations::migration_op] + async fn drop_typed_playback_queue( + ctx: migrations::MigrationContext<'_>, + ) -> cot::db::Result<()> { + ctx.db + .raw("ALTER TABLE furumusic__playback_state DROP COLUMN IF EXISTS queue_items_json") + .await?; + Ok(()) + } + + #[derive(Debug, Copy, Clone)] + pub struct M0054AddTypedPlaybackQueue; + impl migrations::Migration for M0054AddTypedPlaybackQueue { + const APP_NAME: &'static str = "furumusic"; + const MIGRATION_NAME: &'static str = "m_0054_add_typed_playback_queue"; + const DEPENDENCIES: &'static [migrations::MigrationDependency] = + &[migrations::MigrationDependency::migration( + "furumusic", + "m_0052_add_import_playlist_destinations", + )]; + const OPERATIONS: &'static [Operation] = &[Operation::custom(add_typed_playback_queue) + .backwards(drop_typed_playback_queue) + .build()]; + } + + // -- M0055: opaque podcast sync relay state (device-sync protocol v4) --- + + // Podcast registers received from trusted devices, kept as the shared + // contract JSON so this server can return them in its snapshots. This is + // not the podcast store: the web player has no podcast features yet. + #[cot::db::migrations::migration_op] + async fn create_fed_podcast_relay( + ctx: migrations::MigrationContext<'_>, + ) -> cot::db::Result<()> { + ctx.db + .raw( + "CREATE TABLE IF NOT EXISTS furumusic__fed_podcast_relay ( + user_id BIGINT NOT NULL, + section VARCHAR(16) NOT NULL, + record_key TEXT NOT NULL, + record_json JSONB NOT NULL, + updated_at_ms BIGINT NOT NULL, + PRIMARY KEY (user_id, section, record_key) + )", + ) + .await?; + Ok(()) + } + + /// Removes the operations a 0.12.x release cannot parse: the podcast + /// kinds relayed since device-sync protocol 4. Ends with its `WHERE` + /// clause, so tests can narrow it to one user. + pub(crate) const DELETE_PODCAST_SYNC_OPS_SQL: &str = + "DELETE FROM furumusic__fed_sync_ops WHERE kind LIKE 'podcast\\_%'"; + + // Release 0.12.x parses every unacknowledged operation it sends to a + // peer and fails the whole batch on an unknown kind, so the podcast + // operations relayed by 0.13 must go before it runs again (see + // docs/administration.md, "Rolling back to 0.12"). + #[cot::db::migrations::migration_op] + async fn drop_fed_podcast_relay(ctx: migrations::MigrationContext<'_>) -> cot::db::Result<()> { + ctx.db.raw(DELETE_PODCAST_SYNC_OPS_SQL).await?; + ctx.db + .raw("DROP TABLE IF EXISTS furumusic__fed_podcast_relay") + .await?; + Ok(()) + } + + #[derive(Debug, Copy, Clone)] + pub struct M0055CreateFedPodcastRelay; + impl migrations::Migration for M0055CreateFedPodcastRelay { + const APP_NAME: &'static str = "furumusic"; + const MIGRATION_NAME: &'static str = "m_0055_create_fed_podcast_relay"; + const DEPENDENCIES: &'static [migrations::MigrationDependency] = + &[migrations::MigrationDependency::migration( + "furumusic", + "m_0054_add_typed_playback_queue", + )]; + const OPERATIONS: &'static [Operation] = &[Operation::custom(create_fed_podcast_relay) + .backwards(drop_fed_podcast_relay) + .build()]; + } + + // -- M0056: PostgreSQL podcast store (device-sync protocol v4) --------- + + // Per-account podcast catalog, synced registers and local fetch state; + // see `crate::podcasts` and `crate::podcasts::schema`. + #[cot::db::migrations::migration_op] + async fn create_podcast_store(ctx: migrations::MigrationContext<'_>) -> cot::db::Result<()> { + for sql in crate::podcasts::schema::CREATE_PODCAST_STORE { + ctx.db.raw(sql).await?; + } + Ok(()) + } + + #[cot::db::migrations::migration_op] + async fn drop_podcast_store(ctx: migrations::MigrationContext<'_>) -> cot::db::Result<()> { + for sql in crate::podcasts::schema::DROP_PODCAST_STORE { + ctx.db.raw(sql).await?; + } + Ok(()) + } + + #[derive(Debug, Copy, Clone)] + pub struct M0056CreatePodcastStore; + impl migrations::Migration for M0056CreatePodcastStore { + const APP_NAME: &'static str = "furumusic"; + const MIGRATION_NAME: &'static str = "m_0056_create_podcast_store"; + const DEPENDENCIES: &'static [migrations::MigrationDependency] = + &[migrations::MigrationDependency::migration( + "furumusic", + "m_0055_create_fed_podcast_relay", + )]; + const OPERATIONS: &'static [Operation] = &[Operation::custom(create_podcast_store) + .backwards(drop_podcast_store) + .build()]; + } + + // -- M0057: import the 0.13.0 podcast relay state into the store -------- + + // The relay table stays (unused) until a later cleanup release, so a + // rollback to 0.13.0 still finds the state it relayed. + #[cot::db::migrations::migration_op] + async fn import_podcast_relay(ctx: migrations::MigrationContext<'_>) -> cot::db::Result<()> { + for sql in crate::podcasts::schema::import_statements("TRUE") { + ctx.db.raw(&sql).await?; + } + Ok(()) + } + + #[cot::db::migrations::migration_op] + async fn keep_podcast_store_rows( + _ctx: migrations::MigrationContext<'_>, + ) -> cot::db::Result<()> { + // The imported rows go with the tables (`m_0056` backwards). + Ok(()) + } + + #[derive(Debug, Copy, Clone)] + pub struct M0057ImportPodcastRelay; + impl migrations::Migration for M0057ImportPodcastRelay { + const APP_NAME: &'static str = "furumusic"; + const MIGRATION_NAME: &'static str = "m_0057_import_podcast_relay"; + const DEPENDENCIES: &'static [migrations::MigrationDependency] = + &[migrations::MigrationDependency::migration( + "furumusic", + "m_0056_create_podcast_store", + )]; + const OPERATIONS: &'static [Operation] = &[Operation::custom(import_podcast_relay) + .backwards(keep_podcast_store_rows) + .build()]; + } + + // -- M0058: local privacy confirmations of shared podcasts ------------- + + // Client-held `privacy_confirmed` of public podcast discovery (MR-W3); + // see `crate::podcasts::discovery`. Never synced. + #[cot::db::migrations::migration_op] + async fn create_podcast_share_confirmation( + ctx: migrations::MigrationContext<'_>, + ) -> cot::db::Result<()> { + for sql in crate::podcasts::schema::CREATE_SHARE_CONFIRMATION { + ctx.db.raw(sql).await?; + } + Ok(()) + } + + #[cot::db::migrations::migration_op] + async fn drop_podcast_share_confirmation( + ctx: migrations::MigrationContext<'_>, + ) -> cot::db::Result<()> { + for sql in crate::podcasts::schema::DROP_SHARE_CONFIRMATION { + ctx.db.raw(sql).await?; + } + Ok(()) + } + + #[derive(Debug, Copy, Clone)] + pub struct M0058CreatePodcastShareConfirmation; + impl migrations::Migration for M0058CreatePodcastShareConfirmation { + const APP_NAME: &'static str = "furumusic"; + const MIGRATION_NAME: &'static str = "m_0058_create_podcast_share_confirmation"; + const DEPENDENCIES: &'static [migrations::MigrationDependency] = + &[migrations::MigrationDependency::migration( + "furumusic", + "m_0057_import_podcast_relay", + )]; + const OPERATIONS: &'static [Operation] = + &[Operation::custom(create_podcast_share_confirmation) + .backwards(drop_podcast_share_confirmation) + .build()]; + } + pub const MIGRATIONS: &[&SyncDynMigration] = &[ &M0006CreateMediaFile, &M0007CreateArtist, @@ -2881,5 +3095,10 @@ pub mod db_migrations { &M0050CreateMetadataFixHistory, &M0051AddListenStatisticsIndexes, &M0052AddImportPlaylistDestinations, + &M0054AddTypedPlaybackQueue, + &M0055CreateFedPodcastRelay, + &M0056CreatePodcastStore, + &M0057ImportPodcastRelay, + &M0058CreatePodcastShareConfirmation, ]; } diff --git a/src/player/dto.rs b/src/player/dto.rs index 3c68d40..5e54bab 100644 --- a/src/player/dto.rs +++ b/src/player/dto.rs @@ -147,6 +147,17 @@ pub(super) struct PlaybackStateDto { pub(super) shuffle: bool, pub(super) repeat_mode: String, pub(super) volume: f64, + /// Typed queue (podcast episodes and other non-library items included). + /// + /// Requests send `{"kind": "library", "track_id"}` and + /// `{"kind": "external", "track": }` items; + /// responses return external items as browser track JSON. Clients that + /// omit it keep using `queue`, which is always written. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub(super) queue_items: Option, + /// Index of the current item in `queue_items`. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub(super) queue_items_position: Option, } #[derive(Debug, Deserialize, JsonSchema)] @@ -263,6 +274,10 @@ pub(super) struct PlayerDevicePlaybackStateDto { pub(super) volume: f64, #[serde(default)] pub(super) updated_at_ms: i64, + /// The browser does not know the position of its current item, a podcast + /// episode it cannot play (`queue_items::POSITION_UNKNOWN_KEY`). + #[serde(default, skip_serializing_if = "std::ops::Not::not")] + pub(super) position_unknown: bool, } #[derive(Debug, Serialize, JsonSchema)] diff --git a/src/player/mod.rs b/src/player/mod.rs index ac6e0bd..4e30b9e 100644 --- a/src/player/mod.rs +++ b/src/player/mod.rs @@ -29,6 +29,7 @@ use crate::youtube::{YouTubePreviewRequest, YouTubeService, YouTubeStartRequest} mod dto; mod helpers; mod queries; +pub(crate) mod queue_items; mod rows; mod sharing; @@ -606,7 +607,7 @@ impl PlayerDeviceHub { target_device_id: Option<&str>, jam_id: Option<&str>, command: &str, - payload: serde_json::Value, + mut payload: serde_json::Value, ) -> Result<(), &'static str> { let now = current_millis(); let mut state = self.state.lock().expect("player device hub lock"); @@ -621,6 +622,15 @@ impl PlayerDeviceHub { let target_id = self .jam_target_device_id_locked(&state, jam) .ok_or("jam playback device is offline")?; + if user_id != jam.host_user_id { + let host_tracks = state + .playback_state_by_user + .get(&jam.host_user_id) + .map(|playback| playback.tracks.clone()) + .unwrap_or_default(); + restore_jam_command_payload(command, &mut payload, &host_tracks)?; + stamp_jam_participant_command(&mut payload); + } (jam.host_user_id, target_id) } else { let target_id = match target_device_id { @@ -1042,11 +1052,17 @@ impl PlayerDeviceHub { }) }) .unwrap_or(user_id); - state + let playback = state .playback_state_by_user .get(&playback_user_id) .cloned() - .map(|playback_state| playback_state_at(playback_state, now)) + .map(|playback_state| playback_state_at(playback_state, now)); + if playback_user_id == user_id { + return playback; + } + // Jam participants are outside the host's trusted devices: private + // items (podcast episodes) become opaque placeholders. + playback.map(redact_playback_state_for_jam) } fn jam_dtos_locked( @@ -1457,6 +1473,7 @@ mod device_tests { repeat_mode: "off".to_string(), volume: 0.7, updated_at_ms: current_millis(), + position_unknown: false, }, ); } @@ -1646,6 +1663,246 @@ pub struct PlayerPageTemplate { pub youtube_downloads_enabled: bool, } +#[cfg(test)] +mod queue_persistence_tests { + use super::*; + use music_dht::device_sync::PlaybackTrack; + + fn offline_pool() -> sqlx::PgPool { + sqlx::postgres::PgPoolOptions::new() + .connect_lazy("postgres://offline@127.0.0.1:1/offline") + .unwrap() + } + + fn request(queue_items: serde_json::Value, position: usize) -> PlaybackStateDto { + PlaybackStateDto { + current_track_id: Some(3), + position_ms: 1_000, + queue: vec![3, 4], + queue_position: 1, + shuffle: false, + repeat_mode: "off".into(), + volume: 0.5, + queue_items: Some(queue_items), + queue_items_position: Some(position), + } + } + + #[tokio::test] + async fn podcast_items_survive_persistence_and_reload() { + let episode = PlaybackTrack::podcast_episode(queue_items::test_episode()); + let browser_track = queue_items::external_track_json(&episode, 1, None); + // The browser sends `_queueItemsPayload(queue.tracks)`. + let dto = request( + serde_json::json!([ + {"kind": "library", "track_id": 3}, + {"kind": "external", "track": browser_track["wire_track"]}, + {"kind": "library", "track_id": 4}, + ]), + 1, + ); + let stored = stored_queue_from_request(&dto).unwrap(); + // Written to queue_items_json and read back with the legacy ids. + let column = serde_json::to_value(&stored).unwrap(); + let row = queue_items::LegacyColumns { + queue: dto.queue.clone(), + current_track_id: dto.current_track_id, + queue_position: dto.queue_position, + }; + let loaded = queue_items::StoredQueue::from_columns(Some(&column), &row).unwrap(); + assert_eq!(loaded, stored); + assert_eq!(loaded.position, 1); + let items = browser_queue_items(&offline_pool(), &loaded).await.unwrap(); + assert_eq!( + items[0], + serde_json::json!({"kind": "library", "track_id": 3}) + ); + assert_eq!(items[1]["kind"], "external"); + assert_eq!(items[1]["track"]["source"]["kind"], "podcast_episode"); + assert_eq!( + queue_items::wire_track_from_browser_json(&items[1]["track"]), + Some(episode) + ); + } + + #[test] + fn malformed_typed_queues_keep_the_legacy_ids_only() { + let dto = request( + serde_json::json!([{"kind": "external", "track": {"id": "x"}}]), + 0, + ); + assert!(stored_queue_from_request(&dto).is_none()); + let legacy_only = PlaybackStateDto { + queue_items: None, + ..request(serde_json::json!([]), 0) + }; + assert!(stored_queue_from_request(&legacy_only).is_none()); + // Older clients do not send the new fields. + let old: PlaybackStateDto = serde_json::from_value(serde_json::json!({ + "current_track_id": 3, "position_ms": 0, "queue": [3], "queue_position": 0, + "shuffle": false, "repeat_mode": "off", "volume": 0.7, + })) + .unwrap(); + assert!(old.queue_items.is_none()); + assert!(!serde_json::to_string(&old).unwrap().contains("queue_items")); + } +} + +#[cfg(test)] +mod jam_command_tests { + use super::*; + use music_dht::device_sync::{PRIVATE_ITEM_TITLE, PlaybackTrack}; + + fn library(id: i64) -> serde_json::Value { + serde_json::json!({"id": id, "title": format!("Song {id}"), "duration_seconds": 100.0}) + } + + fn host_tracks() -> Vec { + let episode = PlaybackTrack::podcast_episode(queue_items::test_episode()); + vec![ + library(1), + queue_items::external_track_json(&episode, 1, None), + library(2), + ] + } + + fn check(command: &str, payload: serde_json::Value) -> Result<(), &'static str> { + let mut payload = payload; + restore_jam_command_payload(command, &mut payload, &host_tracks()) + } + + #[test] + fn participants_cannot_remove_or_clear_private_items() { + assert!(check("queue_remove", serde_json::json!({"index": 1})).is_err()); + assert!(check("queue_clear", serde_json::json!({})).is_err()); + // Music can still be removed, and private items moved. + assert!(check("queue_remove", serde_json::json!({"index": 0})).is_ok()); + assert!(check("queue_remove", serde_json::json!({"index": 2})).is_ok()); + assert!( + check( + "queue_move", + serde_json::json!({"from_index": 1, "to_index": 0}) + ) + .is_ok() + ); + // The host runs `Number(index)`: other spellings of an index are refused. + for index in [ + serde_json::json!("1"), + serde_json::json!(1.0), + serde_json::json!(-1), + ] { + assert!(check("queue_remove", serde_json::json!({"index": index})).is_err()); + } + assert!(check("queue_remove", serde_json::json!({})).is_err()); + // Without private items the host queue can be cleared as before. + let mut payload = serde_json::json!({}); + assert!(restore_jam_command_payload("queue_clear", &mut payload, &[library(1)]).is_ok()); + } + + #[test] + fn participant_commands_reach_the_host_stamped() { + let hub = PlayerDeviceHub::default(); + hub.heartbeat(1, "host", None, None, None); + let jam = hub + .create_jam(1, "Host", "host", vec![(2, "Guest".to_string())]) + .unwrap(); + let jam_id = jam.jams[0].id.clone(); + hub.heartbeat(2, "guest", None, None, None); + hub.join_jam(2, "Guest", "guest", &jam_id).unwrap(); + let tracks = host_tracks(); + hub.heartbeat( + 1, + "host", + None, + Some(&jam_id), + Some(PlayerDevicePlaybackStateDto { + track: Some(tracks[0].clone()), + tracks, + index: 0, + position_seconds: 0.0, + duration_seconds: 100.0, + paused: false, + shuffle: false, + repeat_mode: "off".into(), + volume: 0.5, + updated_at_ms: 0, + position_unknown: false, + }), + ); + // Move the episode to the end, then remove index 2: the cached host + // state still has music there, so only the host can refuse it. + for (command, payload) in [ + ( + "queue_move", + serde_json::json!({"from_index": 1, "to_index": 2}), + ), + ("queue_remove", serde_json::json!({"index": 2})), + ("pause", serde_json::Value::Null), + ] { + hub.enqueue_command(2, None, Some(&jam_id), command, payload) + .unwrap(); + } + hub.enqueue_command( + 1, + None, + Some(&jam_id), + "queue_remove", + serde_json::json!({"index": 0}), + ) + .unwrap(); + let commands = hub.poll(1, "host", None, Some(&jam_id), None).commands; + let stamped: Vec = commands + .iter() + .map(|command| command.payload[JAM_PARTICIPANT_KEY] == true) + .collect(); + assert_eq!(stamped, [true, true, true, false]); + assert_eq!(commands[1].payload["index"], 2); + } + + #[test] + fn redacted_current_episode_has_no_duration() { + let tracks = host_tracks(); + let playback = PlayerDevicePlaybackStateDto { + track: Some(tracks[1].clone()), + tracks: tracks.clone(), + index: 1, + position_seconds: 1200.0, + duration_seconds: 1800.0, + paused: false, + shuffle: false, + repeat_mode: "off".into(), + volume: 0.5, + updated_at_ms: 0, + position_unknown: false, + }; + let redacted = redact_playback_state_for_jam(playback.clone()); + assert_eq!(redacted.duration_seconds, 0.0); + assert_eq!( + redacted.track.as_ref().unwrap()["title"], + PRIVATE_ITEM_TITLE + ); + assert_eq!(redacted.position_seconds, 1200.0); + let leaked = serde_json::to_string(&redacted).unwrap(); + assert!(!leaked.contains("1800") && !leaked.contains("Episode")); + + // A music track keeps its duration. + let music = redact_playback_state_for_jam(PlayerDevicePlaybackStateDto { + track: Some(tracks[0].clone()), + index: 0, + duration_seconds: 100.0, + ..playback.clone() + }); + assert_eq!(music.duration_seconds, 100.0); + + // Without `track` the current item is taken from `tracks[index]`. + let untracked = redact_playback_state_for_jam(PlayerDevicePlaybackStateDto { + track: None, + ..playback + }); + assert_eq!(untracked.duration_seconds, 0.0); + } +} + #[cfg(test)] mod page_template_tests { use super::*; @@ -5755,6 +6012,16 @@ async fn devices_select_handler( .and_then(normalize_device_id) .unwrap_or_else(|| target_device_id.clone()); let previous_active_id = hub.active_device_id_for_commands(user.id); + // Refuse a handoff the target cannot receive before ownership moves. + if let Some(fed_device_id) = fed_device_id_from_virtual(&target_device_id) + && previous_active_id.as_deref() != Some(target_device_id.as_str()) + && let Some(state) = hub.playback_state_json_for_commands(user.id) + && let Err(err) = crate::federation::handle() + .fed_device_web_transfer_check(user.id, fed_device_id, state) + .await + { + return Ok(json_error(StatusCode::CONFLICT, &format!("{err}"))); + } let Some(response) = hub.select(user.id, ¤t_device_id, &target_device_id) else { return Ok(json_error( @@ -5992,6 +6259,158 @@ async fn fed_devices_sync_handler( } } +/// Host playback state as shown to Jam participants +/// (`music_dht::device_sync::redact_for_jam` over the browser queue). +fn redact_playback_state_for_jam( + mut playback: PlayerDevicePlaybackStateDto, +) -> PlayerDevicePlaybackStateDto { + let index = usize::try_from(playback.index).unwrap_or(0); + let current_is_private = playback + .track + .as_ref() + .or_else(|| playback.tracks.get(index)) + .is_some_and(queue_items::browser_track_requires_podcast_support); + playback.tracks = queue_items::redact_tracks_for_jam(&playback.tracks); + if current_is_private { + // Placeholders carry no duration; the host's position is kept like + // `redact_for_jam` keeps `position_secs`. + playback.duration_seconds = 0.0; + } + if playback + .track + .as_ref() + .is_some_and(queue_items::browser_track_requires_podcast_support) + { + playback.track = playback.tracks.get(index).cloned().or_else(|| { + Some( + queue_items::redact_tracks_for_jam(std::slice::from_ref(playback.track.as_ref()?)) + [0] + .clone(), + ) + }); + } + playback +} + +/// Applies the Jam privacy rules to a participant command. +/// +/// Participants cannot add podcast episodes. A full queue they send (play or +/// transfer) gets the host's private items back in place of the +/// placeholders; queues that drop, duplicate or inject private items are +/// rejected. Participants cannot remove a private item, nor clear a queue +/// that holds one; they may still move it (`queue_move`). +fn restore_jam_command_payload( + command: &str, + payload: &mut serde_json::Value, + host_tracks: &[serde_json::Value], +) -> Result<(), &'static str> { + const REJECTED: &str = "podcast episodes are private and cannot be changed in a jam"; + let tracks: Vec = payload + .get("tracks") + .and_then(serde_json::Value::as_array) + .cloned() + .unwrap_or_default(); + let current = payload + .get("track") + .filter(|track| !track.is_null()) + .cloned(); + match command { + "queue_add_end" | "queue_add_next" => { + if tracks + .iter() + .any(queue_items::browser_track_requires_podcast_support) + { + return Err(REJECTED); + } + } + "play_track" | "play_from_index" | "transfer_state" => { + if !queue_items::jam_tracks_allowed(&tracks) + || current.as_ref().is_some_and(|track| { + !queue_items::jam_tracks_allowed(std::slice::from_ref(track)) + }) + { + return Err(REJECTED); + } + if tracks.is_empty() { + if current + .as_ref() + .is_some_and(queue_items::browser_track_requires_podcast_support) + { + return Err(REJECTED); + } + return Ok(()); + } + let restored = queue_items::restore_jam_tracks(host_tracks, &tracks).ok_or(REJECTED)?; + if let Some(object) = payload.as_object_mut() { + if current + .as_ref() + .is_some_and(queue_items::browser_track_requires_podcast_support) + { + let index = object + .get("index") + .and_then(serde_json::Value::as_u64) + .unwrap_or(0) as usize; + let track = restored.get(index).cloned().ok_or(REJECTED)?; + object.insert("track".to_string(), track); + } + object.insert("tracks".to_string(), serde_json::Value::Array(restored)); + } + } + "queue_remove" => { + // The participant view has the host's length and order, so the + // index names the same item on the host. + let index = payload + .get("index") + .and_then(serde_json::Value::as_u64) + .and_then(|index| usize::try_from(index).ok()); + let removes_private = match index { + Some(index) => host_tracks + .get(index) + .is_some_and(queue_items::browser_track_requires_podcast_support), + // The host browser runs `Number(index)`, so "1" or 1.0 would + // still name an item: only plain indexes pass a private queue. + None => host_tracks + .iter() + .any(queue_items::browser_track_requires_podcast_support), + }; + if removes_private { + return Err(REJECTED); + } + } + "queue_clear" + if host_tracks + .iter() + .any(queue_items::browser_track_requires_podcast_support) => + { + return Err(REJECTED); + } + _ => {} + } + Ok(()) +} + +/// Marks a Jam participant command for the host browser. +/// +/// `restore_jam_command_payload` checks the command against the host state of +/// the host's last report. The host browser runs its pending commands in one +/// batch, so an earlier command (a move, an insert) can make a checked index +/// name a private item there. The host browser applies the same rules to its +/// own queue for stamped commands (`_jamParticipantCommandAllowed`). +fn stamp_jam_participant_command(payload: &mut serde_json::Value) { + if !payload.is_object() { + *payload = serde_json::json!({}); + } + if let Some(object) = payload.as_object_mut() { + object.insert( + JAM_PARTICIPANT_KEY.to_string(), + serde_json::Value::Bool(true), + ); + } +} + +/// Command payload key set by the server on Jam participant commands. +const JAM_PARTICIPANT_KEY: &str = "jam_participant"; + fn stamp_jam_queue_tracks(payload: &mut serde_json::Value, user_id: i64, user_name: &str) { let Some(tracks) = payload .get_mut("tracks") @@ -6320,7 +6739,7 @@ async fn get_state_handler( }; let state = sqlx::query_as::<_, PlaybackStateRow>( - r#"SELECT current_track_id, position_ms, queue_json, queue_position, shuffle, repeat_mode::text as repeat_mode, volume + r#"SELECT current_track_id, position_ms, queue_json, queue_items_json, queue_position, shuffle, repeat_mode::text as repeat_mode, volume FROM furumusic__playback_state WHERE user_id = $1"#, ) .bind(user.id) @@ -6331,6 +6750,26 @@ async fn get_state_handler( let dto = match state { Some(s) => { let queue: Vec = serde_json::from_str(&s.queue_json).unwrap_or_default(); + // Prefer the typed queue while it matches the legacy columns. + let typed = queue_items::StoredQueue::from_columns( + s.queue_items_json.as_ref(), + &queue_items::LegacyColumns { + queue: queue.clone(), + current_track_id: s.current_track_id, + queue_position: s.queue_position, + }, + ); + let (queue_items, queue_items_position) = match typed { + Some(stored) => ( + Some( + browser_queue_items(pool, &stored) + .await + .map_err(|e| cot::Error::internal(e.to_string()))?, + ), + Some(stored.position), + ), + None => (None, None), + }; PlaybackStateDto { current_track_id: s.current_track_id, position_ms: s.position_ms, @@ -6339,6 +6778,8 @@ async fn get_state_handler( shuffle: s.shuffle, repeat_mode: s.repeat_mode, volume: s.volume, + queue_items, + queue_items_position, } } None => PlaybackStateDto { @@ -6349,12 +6790,51 @@ async fn get_state_handler( shuffle: false, repeat_mode: "off".to_string(), volume: 0.7, + queue_items: None, + queue_items_position: None, }, }; Json(dto).into_response() } +/// Typed queue items for the browser: library ids to load, external items +/// as browser track JSON with their verbatim wire item. +async fn browser_queue_items( + pool: &sqlx::PgPool, + stored: &queue_items::StoredQueue, +) -> anyhow::Result { + let mut items = Vec::with_capacity(stored.items.len()); + for (index, item) in stored.items.iter().enumerate() { + items.push(match item { + queue_items::StoredQueueItem::Library { track_id } => { + serde_json::json!({"kind": "library", "track_id": track_id}) + } + queue_items::StoredQueueItem::External { track } => serde_json::json!({ + "kind": "external", + "track": crate::federation::devices::resolve_playback_track(pool, track, index).await?, + }), + }); + } + Ok(serde_json::Value::Array(items)) +} + +/// Typed queue to persist from a state request; `None` keeps only the legacy +/// ids (older clients, malformed or oversized queues). +fn stored_queue_from_request(dto: &PlaybackStateDto) -> Option { + let items: Vec = + serde_json::from_value(dto.queue_items.clone()?).ok()?; + queue_items::StoredQueue::new( + items, + dto.queue_items_position.unwrap_or(0), + queue_items::LegacyColumns { + queue: dto.queue.clone(), + current_track_id: dto.current_track_id, + queue_position: dto.queue_position, + }, + ) +} + // --------------------------------------------------------------------------- // PUT /api/player/state // --------------------------------------------------------------------------- @@ -6372,15 +6852,21 @@ async fn put_state_handler( let queue_json = serde_json::to_string(&dto.queue).map_err(|e| cot::Error::internal(e.to_string()))?; + // `queue_json` is still written so an older release keeps working. + let queue_items_json = stored_queue_from_request(&dto) + .map(serde_json::to_value) + .transpose() + .map_err(|e| cot::Error::internal(e.to_string()))?; let now = chrono::Utc::now().format("%Y-%m-%dT%H:%M:%SZ").to_string(); sqlx::query( - r#"INSERT INTO furumusic__playback_state (user_id, current_track_id, position_ms, queue_json, queue_position, shuffle, repeat_mode, volume, updated_at) - VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9) + r#"INSERT INTO furumusic__playback_state (user_id, current_track_id, position_ms, queue_json, queue_position, shuffle, repeat_mode, volume, updated_at, queue_items_json) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10) ON CONFLICT (user_id) DO UPDATE SET current_track_id = $2, position_ms = $3, queue_json = $4, - queue_position = $5, shuffle = $6, repeat_mode = $7, volume = $8, updated_at = $9"#, + queue_position = $5, shuffle = $6, repeat_mode = $7, volume = $8, updated_at = $9, + queue_items_json = $10"#, ) .bind(user.id) .bind(dto.current_track_id) @@ -6391,6 +6877,7 @@ async fn put_state_handler( .bind(&dto.repeat_mode) .bind(dto.volume) .bind(&now) + .bind(queue_items_json) .execute(pool) .await .map_err(|e| cot::Error::internal(e.to_string()))?; @@ -6985,7 +7472,7 @@ async fn federation_search_events_handler( db: Database, query: cot::request::extractors::UrlQuery, ) -> cot::Result { - let Some(_user) = auth::get_request_user(&auth_ctx, &session, &db).await else { + let Some(user) = auth::get_request_user(&auth_ctx, &session, &db).await else { return Ok(json_error(StatusCode::UNAUTHORIZED, "not authenticated")); }; let q = query.0.q.trim().to_owned(); @@ -6995,6 +7482,7 @@ async fn federation_search_events_handler( let search_id = uuid::Uuid::new_v4().to_string(); let body_search_id = search_id.clone(); + let user_id = user.id; let body = Body::streaming(async_stream::try_stream! { let started = serde_json::json!({ "search_id": body_search_id, @@ -7005,25 +7493,40 @@ async fn federation_search_events_handler( }); yield Bytes::from(format!("event: search.started\ndata: {started}\n\n")); - match crate::federation::handle().search_events(&body_search_id, &q).await { - Ok(events) => { + let federation = crate::federation::handle(); + // Public podcasts are searched next to the music DHT; their events + // follow the music events. + let podcasts = federation.spawn_podcast_search(&q); + match federation.search_events(&body_search_id, &q).await { + Ok(music) => { let mut last_sequence = 0; - for event in events { + for event in &music.events { last_sequence = event.sequence; - let data = serde_json::to_string(&event) - .map_err(|err| cot::Error::internal(err.to_string()))?; - yield Bytes::from(format!("event: {}\ndata: {data}\n\n", event.kind)); + yield search_event_frame(event)?; + } + let podcasts = podcasts + .await + .unwrap_or(crate::federation::client::PodcastSearch::Failed); + // A podcast search or ranking failure only drops the + // podcast hits: `search.completed` reports it. + let (podcast_discovery, podcast_events) = federation + .podcast_search_events(&body_search_id, &q, user_id, &music, podcasts) + .await; + for event in &podcast_events { + last_sequence = event.sequence; + yield search_event_frame(event)?; } let completed = serde_json::json!({ "search_id": body_search_id, "sequence": last_sequence + 1, "kind": "search.completed", "entity_key": null, - "entity": { "partial": false }, + "entity": { "partial": false, "podcast_discovery": podcast_discovery }, }); yield Bytes::from(format!("event: search.completed\ndata: {completed}\n\n")); } Err(err) => { + podcasts.abort(); let failed = serde_json::json!({ "search_id": body_search_id, "sequence": 1, @@ -7044,6 +7547,14 @@ async fn federation_search_events_handler( .map_err(|err| cot::Error::internal(err.to_string())) } +/// One server-sent event of a federated search: `event: ` and the +/// event as JSON. +fn search_event_frame(event: &crate::federation::client::SearchEvent) -> cot::Result { + let data = serde_json::to_string(event).map_err(|err| cot::Error::internal(err.to_string()))?; + let kind = event.kind; + Ok(Bytes::from(format!("event: {kind}\ndata: {data}\n\n"))) +} + async fn federation_artist_events_handler( auth_ctx: auth::AuthContext, session: Session, diff --git a/src/player/queue_items.rs b/src/player/queue_items.rs new file mode 100644 index 0000000..9e621fb --- /dev/null +++ b/src/player/queue_items.rs @@ -0,0 +1,710 @@ +//! Lossless queue items shared by the browser player, device sync and the +//! persisted playback state. +//! +//! The browser queue holds plain track JSON objects. Library tracks of this +//! server keep their usual shape. Every other item received from a trusted +//! device (unresolved federated music, podcast episodes, Jam private +//! placeholders) becomes an *external* item: display fields for the queue UI +//! plus the verbatim device-sync DTO under [`WIRE_TRACK_KEY`] and its explicit +//! [`PlaybackSource`] under [`SOURCE_KEY`]. Converting back to the wire uses +//! the verbatim DTO, so such items survive browser -> server -> peers -> +//! server -> browser unchanged. The web player plays podcast episodes +//! through `/api/podcasts/episodes/{id}/playback` (publisher URL or the +//! optional proxy); placeholders and unresolved federated music stay +//! unplayable here. +//! +//! Gating and Jam decisions are delegated to `music_dht::device_sync`; this +//! module only maps browser JSON onto the shared DTOs. + +use music_dht::device_sync::{ + PRIVATE_ITEM_TITLE, PlaybackCommand, PlaybackSource, PlaybackStateWire, PlaybackTrack, + jam_command_allowed, redact_for_jam, restore_jam_private_items, +}; +use serde::{Deserialize, Serialize}; +use serde_json::Value; + +/// Browser track key holding the verbatim [`PlaybackTrack`] of an external item. +pub(crate) const WIRE_TRACK_KEY: &str = "wire_track"; +/// Browser track key holding the explicit [`PlaybackSource`] of an external item. +pub(crate) const SOURCE_KEY: &str = "source"; +/// Browser playback state key: `true` when the browser does not know the +/// position of its current item. That happens when it selects a podcast +/// episode it cannot play (Next, auto-advance, a click or a play command): +/// it is never told where the episode was left, and inventing 0 would +/// restart the episode for every device. The server then takes the +/// position from the episode's progress register. +pub(crate) const POSITION_UNKNOWN_KEY: &str = "position_unknown"; + +/// Reason shown for federated music that is not in this library. +pub(crate) const FED_UNAVAILABLE_REASON: &str = "Not available on this web player yet"; +/// Reason shown for Jam placeholders of someone else's private item. +pub(crate) const PRIVATE_UNAVAILABLE_REASON: &str = "Private item of the Jam host"; + +/// Version of the typed queue stored in `furumusic__playback_state.queue_items_json`. +pub(crate) const STORED_QUEUE_VERSION: u16 = 1; +/// Upper bound of persisted queue items; larger queues keep only `queue_json`. +pub(crate) const MAX_STORED_QUEUE_ITEMS: usize = 10_000; + +/// Browser JSON for a queue item that is not a library track of this server. +/// +/// `content_id` is the normalized content id of a music item, when known. +/// The `id` is a negative placeholder that is stable for the item and its +/// queue position, so the browser can tell items apart; it never refers to a +/// library track. +pub(crate) fn external_track_json( + track: &PlaybackTrack, + index: usize, + content_id: Option<&str>, +) -> Value { + let mut json = match &track.source { + Some(PlaybackSource::PodcastEpisode(episode)) => serde_json::json!({ + "id": placeholder_track_id(Some(episode.episode_id.as_str()), index), + "title": episode.episode_title, + "track_number": serde_json::Value::Null, + "disc_number": serde_json::Value::Null, + "duration_seconds": episode.duration_ms.unwrap_or(0).max(0) as f64 / 1_000.0, + "artists": artist_names_to_json(std::slice::from_ref(&episode.podcast_title)), + "featured_artists": [], + "release_id": 0, + "release_title": episode.podcast_title, + "release_year": serde_json::Value::Null, + // Artwork is loaded from the publisher, like the audio. + "cover_url": episode.image_url, + // The player asks `/api/podcasts/episodes/{id}/playback` for the + // audio source when it starts the episode. + "stream_url": "", + "uploader_name": "Podcast", + "audio_format": episode.mime, + "content_id": serde_json::Value::Null, + "unavailable": false, + }), + Some(PlaybackSource::Private) => serde_json::json!({ + "id": placeholder_track_id(Some("private"), index), + "title": PRIVATE_ITEM_TITLE, + "track_number": serde_json::Value::Null, + "disc_number": serde_json::Value::Null, + "duration_seconds": 0.0, + "artists": [], + "featured_artists": [], + "release_id": 0, + "release_title": "", + "release_year": serde_json::Value::Null, + "cover_url": serde_json::Value::Null, + "stream_url": "", + "uploader_name": "", + "content_id": serde_json::Value::Null, + "unavailable": true, + "unavailable_reason": PRIVATE_UNAVAILABLE_REASON, + }), + Some(PlaybackSource::Music) | None => music_placeholder_json(track, index, content_id), + }; + if let Some(object) = json.as_object_mut() { + if let Some(source) = &track.source + && let Ok(source) = serde_json::to_value(source) + { + object.insert(SOURCE_KEY.to_string(), source); + } + if let Ok(wire) = serde_json::to_value(track) { + object.insert(WIRE_TRACK_KEY.to_string(), wire); + } + } + json +} + +/// Display fields of unresolved federated music (unchanged from earlier +/// releases, which had no verbatim wire item). +fn music_placeholder_json(track: &PlaybackTrack, index: usize, content_id: Option<&str>) -> Value { + let artists = if !track.artist_names.is_empty() { + track.artist_names.clone() + } else if let Some(fed) = &track.fed { + fed.artist_names.clone() + } else { + Vec::new() + }; + let featured = if !track.featured_artist_names.is_empty() { + track.featured_artist_names.clone() + } else if let Some(fed) = &track.fed { + fed.featured_artist_names.clone() + } else { + Vec::new() + }; + let title = track + .fed + .as_ref() + .map(|fed| fed.title.as_str()) + .filter(|title| !title.trim().is_empty()) + .unwrap_or(&track.title); + let release_title = track + .fed + .as_ref() + .and_then(|fed| fed.release_title.as_deref()) + .filter(|title| !title.trim().is_empty()) + .unwrap_or(&track.release_title); + let content_id = content_id + .map(ToString::to_string) + .or_else(|| track.content_id.clone()) + .or_else(|| track.fed.as_ref().map(|fed| fed.content_id.clone())); + serde_json::json!({ + "id": placeholder_track_id(content_id.as_deref(), index), + "title": title, + "track_number": track.track_number, + "disc_number": track.disc_number, + "duration_seconds": track.duration_seconds, + "artists": artist_names_to_json(&artists), + "featured_artists": artist_names_to_json(&featured), + "release_id": 0, + "release_title": release_title, + "release_year": track.release_year, + "cover_url": serde_json::Value::Null, + "stream_url": "", + "uploader_name": "Federation", + "audio_format": track.audio_format.clone(), + "audio_bitrate": track.audio_bitrate, + "audio_sample_rate": track.audio_sample_rate, + "audio_bit_depth": track.audio_bit_depth, + "file_size_bytes": track.file_size_bytes, + "content_id": content_id, + "unavailable": true, + "unavailable_reason": FED_UNAVAILABLE_REASON, + }) +} + +fn artist_names_to_json(names: &[String]) -> Vec { + names + .iter() + .enumerate() + .filter(|(_, name)| !name.trim().is_empty()) + .map(|(index, name)| serde_json::json!({"id": -((index as i64) + 1), "name": name})) + .collect() +} + +fn placeholder_track_id(seed: Option<&str>, index: usize) -> i64 { + let mut bytes = [0u8; 8]; + let hash = blake3::hash(seed.unwrap_or("").as_bytes()); + bytes.copy_from_slice(&hash.as_bytes()[..8]); + let value = i64::from_le_bytes(bytes).unsigned_abs() % 9_000_000_000_000; + -((value as i64) + index as i64 + 1) +} + +/// The device-sync item carried by an external browser track. +/// +/// Returns the verbatim [`WIRE_TRACK_KEY`] DTO when present. Otherwise a +/// browser item that only names a podcast or private [`SOURCE_KEY`] is +/// rebuilt with the shared constructors. Library tracks (and malformed +/// items) return `None` and keep the library mapping. +pub(crate) fn wire_track_from_browser_json(track: &Value) -> Option { + if let Some(wire) = track.get(WIRE_TRACK_KEY).filter(|wire| wire.is_object()) + && let Ok(wire) = serde_json::from_value::(wire.clone()) + { + return Some(wire); + } + match serde_json::from_value::(track.get(SOURCE_KEY)?.clone()).ok()? { + PlaybackSource::PodcastEpisode(episode) => Some(PlaybackTrack::podcast_episode(*episode)), + PlaybackSource::Private => Some(PlaybackTrack::private_placeholder()), + PlaybackSource::Music => None, + } +} + +/// Whether a browser item is a podcast episode or a Jam placeholder. +pub(crate) fn browser_track_requires_podcast_support(track: &Value) -> bool { + wire_track_from_browser_json(track).is_some_and(|track| track.requires_podcast_support()) +} + +/// Wire-shaped view of browser tracks for the shared gating and Jam helpers. +/// +/// External items are their verbatim DTOs. Library tracks become minimal +/// music items: the helpers only look at item sources and positions, so +/// this conversion needs no database lookups. +pub(crate) fn shadow_wire_state(tracks: &[Value], index: usize) -> PlaybackStateWire { + PlaybackStateWire { + queue: tracks + .iter() + .map(|track| wire_track_from_browser_json(track).unwrap_or_else(|| music_stub(track))) + .collect(), + queue_pos: index, + playing: false, + paused: true, + idle_since_ms: None, + position_secs: 0.0, + volume: 0, + shuffle: false, + repeat: Default::default(), + playback_rate: None, + } +} + +fn music_stub(track: &Value) -> PlaybackTrack { + PlaybackTrack { + id: track.get("id").and_then(Value::as_i64).unwrap_or(0), + title: String::new(), + track_number: None, + disc_number: None, + duration_seconds: 0.0, + artist_names: Vec::new(), + featured_artist_names: Vec::new(), + release_id: 0, + release_title: String::new(), + release_year: None, + file_path: String::new(), + content_id: None, + audio_format: None, + audio_bitrate: None, + audio_sample_rate: None, + audio_bit_depth: None, + file_size_bytes: None, + play_count: 0, + fed: None, + source: None, + } +} + +/// Browser tracks as shown to Jam participants. +/// +/// Uses [`redact_for_jam`]: every podcast episode (and existing placeholder) +/// becomes the opaque private placeholder; music, order and position stay. +pub(crate) fn redact_tracks_for_jam(tracks: &[Value]) -> Vec { + let redacted = redact_for_jam(shadow_wire_state(tracks, 0)); + tracks + .iter() + .zip(redacted.queue.iter()) + .enumerate() + .map(|(index, (original, redacted))| { + if redacted.is_private_placeholder() { + external_track_json(redacted, index, None) + } else { + original.clone() + } + }) + .collect() +} + +/// Maps the private placeholders of a Jam participant's full queue back to +/// the host's items ([`restore_jam_private_items`]). +/// +/// Returns `None` when the participant queue contains real podcast episodes +/// or drops, duplicates or injects private items; the command is then +/// rejected. +pub(crate) fn restore_jam_tracks(host: &[Value], participant: &[Value]) -> Option> { + let host_state = shadow_wire_state(host, 0); + let restored = restore_jam_private_items(&host_state, shadow_wire_state(participant, 0))?; + let mut host_private = host + .iter() + .zip(host_state.queue.iter()) + .filter(|(_, wire)| wire.requires_podcast_support()) + .map(|(json, _)| json); + let participant_state = shadow_wire_state(participant, 0); + let mut out = Vec::with_capacity(participant.len()); + for ((json, before), after) in participant + .iter() + .zip(participant_state.queue.iter()) + .zip(restored.queue.iter()) + { + if before.is_private_placeholder() { + out.push(host_private.next()?.clone()); + } else { + debug_assert_eq!(before, after); + out.push(json.clone()); + } + } + Some(out) +} + +/// Whether tracks sent by a Jam participant may enter the host queue at all +/// ([`jam_command_allowed`]): participants cannot add podcast episodes. +pub(crate) fn jam_tracks_allowed(tracks: &[Value]) -> bool { + jam_command_allowed(&PlaybackCommand::SetState { + state: shadow_wire_state(tracks, 0), + seek: false, + }) +} + +/// Typed playback queue persisted next to the legacy `queue_json`. +/// +/// `queue_json` (a list of library track ids) is still written for rollback +/// to older releases. `legacy_queue`, `legacy_current_track_id` and +/// `legacy_queue_position` record the legacy columns written alongside this +/// value: when they no longer match the row (an older release or a +/// legacy-only writer changed the queue or moved to another track since), +/// readers ignore the typed value and fall back to the legacy columns. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +pub(crate) struct StoredQueue { + /// Shape version, [`STORED_QUEUE_VERSION`]. + pub(crate) version: u16, + /// Index of the current item in `items`. + #[serde(default)] + pub(crate) position: usize, + /// Queue items in order. + #[serde(default)] + pub(crate) items: Vec, + /// `queue_json` ids written together with this value. + #[serde(default)] + pub(crate) legacy_queue: Vec, + /// `current_track_id` written together with this value. + #[serde(default)] + pub(crate) legacy_current_track_id: Option, + /// `queue_position` written together with this value (`None`: unknown, + /// the value is not used). + #[serde(default)] + pub(crate) legacy_queue_position: Option, +} + +/// Legacy queue columns of a `furumusic__playback_state` row. +#[derive(Debug, Clone, PartialEq, Eq, Default)] +pub(crate) struct LegacyColumns { + /// `queue_json`: library track ids. + pub(crate) queue: Vec, + /// `current_track_id`. + pub(crate) current_track_id: Option, + /// `queue_position`. + pub(crate) queue_position: i32, +} + +/// One persisted queue item. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[serde(tag = "kind", rename_all = "snake_case")] +pub(crate) enum StoredQueueItem { + /// Library track of this server. + Library { + /// Local track id. + track_id: i64, + }, + /// Any other item, kept verbatim with its explicit `source`. + External { + /// Device-sync queue item. + track: Box, + }, +} + +impl StoredQueue { + /// Builds the persisted value; `None` when the queue exceeds the bound. + /// + /// Jam private placeholders are dropped: they stand for the Jam host's + /// items, never for items of this user's own queue. + pub(crate) fn new( + items: Vec, + position: usize, + legacy: LegacyColumns, + ) -> Option { + if items.len() > MAX_STORED_QUEUE_ITEMS { + return None; + } + let mut stored = Self { + version: STORED_QUEUE_VERSION, + position, + items, + legacy_queue: Vec::new(), + legacy_current_track_id: None, + legacy_queue_position: None, + }; + stored.retain_items(|item| match item { + StoredQueueItem::Library { .. } => true, + StoredQueueItem::External { track } => !track.is_private_placeholder(), + }); + stored.set_legacy(legacy); + Some(stored) + } + + /// Reads the typed column, preferring it over the legacy columns only + /// when it has a known version and was written together with exactly + /// these legacy values. + pub(crate) fn from_columns(typed: Option<&Value>, legacy: &LegacyColumns) -> Option { + let stored = serde_json::from_value::(typed?.clone()).ok()?; + (stored.version == STORED_QUEUE_VERSION + && stored.legacy_queue == legacy.queue + && stored.legacy_current_track_id == legacy.current_track_id + && stored.legacy_queue_position == Some(legacy.queue_position) + && stored.items.len() <= MAX_STORED_QUEUE_ITEMS) + .then_some(stored) + } + + /// Removes library tracks rejected by `keep` and records the new legacy + /// columns. The current position follows its item, or the item after it + /// when the current item was removed. + pub(crate) fn retain_library(&mut self, keep: impl Fn(i64) -> bool, legacy: LegacyColumns) { + self.retain_items(|item| match item { + StoredQueueItem::Library { track_id } => keep(*track_id), + StoredQueueItem::External { .. } => true, + }); + self.set_legacy(legacy); + } + + fn retain_items(&mut self, keep: impl Fn(&StoredQueueItem) -> bool) { + let original_position = self.position; + let mut position = self.position; + let mut index = 0; + self.items.retain(|item| { + let retained = keep(item); + if !retained && index < original_position { + position = position.saturating_sub(1); + } + index += 1; + retained + }); + self.position = position.min(self.items.len().saturating_sub(1)); + } + + fn set_legacy(&mut self, legacy: LegacyColumns) { + self.legacy_queue = legacy.queue; + self.legacy_current_track_id = legacy.current_track_id; + self.legacy_queue_position = Some(legacy.queue_position); + } +} + +/// A valid podcast episode reference for tests. +#[cfg(test)] +pub(crate) fn test_episode() -> music_dht::podcast::PodcastEpisodeRef { + use music_dht::podcast::{EpisodeId, FeedUrl, PodcastEpisodeRef, PodcastId}; + let feed_url = FeedUrl::parse("https://podcasts.example/feed.xml?token=Ab").unwrap(); + let podcast_id = PodcastId::from_feed_url(&feed_url); + let episode_id = EpisodeId::from_guid(&podcast_id, "episode-42").unwrap(); + PodcastEpisodeRef { + podcast_id, + episode_id, + feed_url, + podcast_title: "Example Cast".into(), + episode_title: "Episode 42".into(), + audio_url: "https://cdn.example/ep42.mp3".into(), + mime: Some("audio/mpeg".into()), + duration_ms: Some(1_800_000), + published_at_ms: Some(1_700_000_000_000), + season: Some(2), + episode_number: Some(42), + image_url: None, + progress_generation: Some(7), + } +} + +#[cfg(test)] +mod tests { + use super::*; + use music_dht::podcast::EpisodeId; + + fn episode() -> music_dht::podcast::PodcastEpisodeRef { + test_episode() + } + + fn library(id: i64) -> Value { + serde_json::json!({"id": id, "title": format!("Track {id}"), "stream_url": format!("/api/player/stream/{id}")}) + } + + #[test] + fn podcast_item_round_trips_through_browser_json() { + let mut wire = PlaybackTrack::podcast_episode(episode()); + wire.play_count = 3; + let json = external_track_json(&wire, 4, None); + // Playable on the web player (the source is resolved on play). + assert_eq!(json["unavailable"], false); + assert!(json.get("unavailable_reason").is_none()); + assert_eq!(json["stream_url"], ""); + assert_eq!( + json["cover_url"], + Value::Null, + "the test episode has no artwork" + ); + assert_eq!(json["title"], "Episode 42"); + assert_eq!(json[SOURCE_KEY]["kind"], "podcast_episode"); + assert!(json["id"].as_i64().unwrap() < 0); + // The browser stores and echoes the object; the item is unchanged. + let echoed: Value = serde_json::from_str(&json.to_string()).unwrap(); + assert_eq!(wire_track_from_browser_json(&echoed), Some(wire)); + } + + #[test] + fn private_and_unresolved_music_keep_their_wire_item() { + let private = PlaybackTrack::private_placeholder(); + let json = external_track_json(&private, 0, None); + assert_eq!(json["title"], PRIVATE_ITEM_TITLE); + assert_eq!(json[SOURCE_KEY]["kind"], "private"); + assert_eq!(wire_track_from_browser_json(&json), Some(private)); + + let mut music = PlaybackTrack::private_placeholder(); + music.source = None; + music.title = "Remote song".into(); + music.content_id = Some(format!("b3:{}", "a".repeat(64))); + let json = external_track_json(&music, 1, music.content_id.as_deref()); + assert!(json.get(SOURCE_KEY).is_none()); + assert_eq!(json["unavailable_reason"], FED_UNAVAILABLE_REASON); + assert_eq!(wire_track_from_browser_json(&json), Some(music)); + } + + #[test] + fn library_tracks_are_not_external() { + assert_eq!(wire_track_from_browser_json(&library(5)), None); + let mut music_source = library(6); + music_source[SOURCE_KEY] = serde_json::json!({"kind": "music"}); + assert_eq!(wire_track_from_browser_json(&music_source), None); + } + + #[test] + fn source_only_items_use_shared_constructors() { + let json = serde_json::json!({ + "id": -1, + SOURCE_KEY: serde_json::to_value(PlaybackSource::PodcastEpisode(Box::new(episode()))).unwrap(), + }); + assert_eq!( + wire_track_from_browser_json(&json), + Some(PlaybackTrack::podcast_episode(episode())) + ); + // A malformed verbatim item falls back to the explicit source. + let mut tampered = json.clone(); + tampered[WIRE_TRACK_KEY] = serde_json::json!({"id": "x"}); + assert_eq!( + wire_track_from_browser_json(&tampered), + Some(PlaybackTrack::podcast_episode(episode())) + ); + } + + #[test] + fn jam_redaction_hides_episodes_and_restores_them_in_order() { + let first = external_track_json(&PlaybackTrack::podcast_episode(episode()), 1, None); + let mut other = episode(); + other.episode_title = "Episode 43".into(); + other.episode_id = EpisodeId::from_guid(&other.podcast_id, "episode-43").unwrap(); + let second = external_track_json(&PlaybackTrack::podcast_episode(other), 3, None); + let host = vec![library(1), first.clone(), library(2), second.clone()]; + + let redacted = redact_tracks_for_jam(&host); + assert_eq!(redacted[0], host[0]); + assert_eq!(redacted[1]["title"], PRIVATE_ITEM_TITLE); + let leaked = serde_json::to_string(&redacted).unwrap(); + assert!(!leaked.contains("Episode 4") && !leaked.contains("podcasts.example")); + assert!(!leaked.contains("cdn.example") && !leaked.contains("Example Cast")); + + // The participant reorders music and moves a placeholder. + let participant = vec![ + redacted[3].clone(), + library(2), + redacted[1].clone(), + library(1), + ]; + let restored = restore_jam_tracks(&host, &participant).unwrap(); + assert_eq!( + restored, + vec![first.clone(), library(2), second, library(1)] + ); + + // Dropping, duplicating or injecting private items is rejected. + assert!(restore_jam_tracks(&host, &[library(1), redacted[1].clone()]).is_none()); + assert!(restore_jam_tracks(&host, &[first.clone(), redacted[1].clone()]).is_none()); + assert!(!jam_tracks_allowed(std::slice::from_ref(&first))); + assert!(jam_tracks_allowed(&[library(9)])); + } + + fn legacy(queue: &[i64], current_track_id: Option, queue_position: i32) -> LegacyColumns { + LegacyColumns { + queue: queue.to_vec(), + current_track_id, + queue_position, + } + } + + #[test] + fn stored_queue_prefers_typed_value_only_while_legacy_ids_match() { + let items = vec![ + StoredQueueItem::Library { track_id: 3 }, + StoredQueueItem::External { + track: Box::new(PlaybackTrack::podcast_episode(episode())), + }, + StoredQueueItem::Library { track_id: 4 }, + ]; + // The episode is current: the browser wrote its placeholder id. + let written = legacy(&[3, -11, 4], Some(-11), 1); + let stored = StoredQueue::new(items, 1, written.clone()).unwrap(); + let value = serde_json::to_value(&stored).unwrap(); + assert_eq!(value["version"], 1); + assert_eq!(value["items"][1]["kind"], "external"); + assert_eq!( + value["items"][1]["track"]["source"]["kind"], + "podcast_episode" + ); + assert_eq!( + StoredQueue::from_columns(Some(&value), &written), + Some(stored.clone()) + ); + // A rollback release rewrote queue_json: fall back to it. + assert_eq!( + StoredQueue::from_columns(Some(&value), &legacy(&[3, 4], Some(4), 1)), + None + ); + // A rollback release kept the ids but moved to another track (Next + // or a click): the typed position is stale, fall back as well. + assert_eq!( + StoredQueue::from_columns(Some(&value), &legacy(&[3, -11, 4], Some(4), 2)), + None + ); + assert_eq!( + StoredQueue::from_columns(Some(&value), &legacy(&[3, -11, 4], Some(-11), 0)), + None + ); + // Values without the recorded current item are not trusted. + let mut unknown = value.clone(); + unknown + .as_object_mut() + .unwrap() + .remove("legacy_queue_position"); + assert_eq!(StoredQueue::from_columns(Some(&unknown), &written), None); + let mut future = value.clone(); + future["version"] = serde_json::json!(2); + assert_eq!(StoredQueue::from_columns(Some(&future), &written), None); + assert_eq!(StoredQueue::from_columns(None, &written), None); + } + + #[test] + fn stored_queue_never_keeps_jam_placeholders() { + let items = vec![ + StoredQueueItem::Library { track_id: 3 }, + StoredQueueItem::External { + track: Box::new(PlaybackTrack::private_placeholder()), + }, + StoredQueueItem::External { + track: Box::new(PlaybackTrack::podcast_episode(episode())), + }, + StoredQueueItem::Library { track_id: 4 }, + ]; + let stored = StoredQueue::new(items.clone(), 3, legacy(&[3, 4], Some(4), 1)).unwrap(); + assert_eq!( + stored.items, + vec![items[0].clone(), items[2].clone(), items[3].clone()] + ); + assert_eq!(stored.position, 2); + // A current placeholder hands the position to the next item. + let stored = StoredQueue::new(items.clone(), 1, legacy(&[3, 4], None, 0)).unwrap(); + assert_eq!(stored.position, 1); + assert_eq!(stored.items[1], items[2]); + } + + #[test] + fn stored_queue_cleanup_keeps_external_items_and_position() { + let items = vec![ + StoredQueueItem::Library { track_id: 3 }, + StoredQueueItem::External { + track: Box::new(PlaybackTrack::podcast_episode(episode())), + }, + StoredQueueItem::Library { track_id: 4 }, + ]; + let mut stored = StoredQueue::new(items.clone(), 1, legacy(&[3, 4], None, 0)).unwrap(); + stored.retain_library(|id| id != 3, legacy(&[4], None, 0)); + assert_eq!(stored.items, items[1..].to_vec()); + assert_eq!(stored.position, 0); + assert_eq!(stored.legacy_queue, vec![4]); + // The cleanup's new legacy columns are what readers compare with. + let value = serde_json::to_value(&stored).unwrap(); + assert_eq!( + StoredQueue::from_columns(Some(&value), &legacy(&[4], None, 0)), + Some(stored) + ); + + let mut current_removed = + StoredQueue::new(items.clone(), 2, legacy(&[3, 4], Some(4), 1)).unwrap(); + current_removed.retain_library(|id| id != 4, legacy(&[3], None, 0)); + assert_eq!(current_removed.position, 1); + assert!( + StoredQueue::new( + vec![items[0].clone(); MAX_STORED_QUEUE_ITEMS + 1], + 0, + LegacyColumns::default() + ) + .is_none() + ); + } +} diff --git a/src/player/rows.rs b/src/player/rows.rs index b016a4a..9756048 100644 --- a/src/player/rows.rs +++ b/src/player/rows.rs @@ -65,6 +65,7 @@ pub(super) struct PlaybackStateRow { pub(super) current_track_id: Option, pub(super) position_ms: i32, pub(super) queue_json: String, + pub(super) queue_items_json: Option, pub(super) queue_position: i32, pub(super) shuffle: bool, pub(super) repeat_mode: String, diff --git a/src/podcasts/actions.rs b/src/podcasts/actions.rs new file mode 100644 index 0000000..037a0d7 --- /dev/null +++ b/src/podcasts/actions.rs @@ -0,0 +1,790 @@ +//! Local user actions: they build the device-sync operation, append it to +//! the account's sync log with its stamp and apply it through the shared +//! code path, all in one transaction. + +use anyhow::Result; +use music_dht::device_sync::{SyncOpPayload, SyncOpWire}; +use music_dht::podcast::{ + EpisodeId, FeedUrl, ListenSession, PodcastEpisodeRef, PodcastId, PodcastMeta, ProgressRecord, + ProgressStatus, effective_share, merge_feed_aliases, merge_listen_session, + next_podcast_generation, resolve_podcast_id, +}; +use sqlx::{PgConnection, Row}; + +use super::apply::{self, ApplyOutcome}; +use super::{PodcastStore, PodcastStoreError, invalid}; +use crate::federation::devices; + +/// Result of [`PodcastStore::subscribe`]. +#[derive(Debug, Clone, PartialEq)] +pub(crate) struct SubscribeOutcome { + /// Podcast the URL resolved to (an existing one when the URL is a known + /// feed URL or alias). + pub(crate) podcast_id: PodcastId, + /// The podcast was already subscribed; no operation was emitted. + pub(crate) already_subscribed: bool, + /// Operations appended to the sync log (and applied). + pub(crate) ops: Vec, +} + +/// A listening session of this account as seen by the progress register. +/// +/// Returned by explicit actions; periodic checkpoints of the session use its +/// `generation` ([`PodcastStore::checkpoint`]). +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct ProgressSession { + /// Episode being played. + pub(crate) episode: PodcastEpisodeRef, + /// Session id (also the `ListenSession::session_id`). + pub(crate) session_id: String, + /// Generation of the session's latest explicit action. + pub(crate) generation: u64, +} + +/// A progress register written by a local action. +#[derive(Debug, Clone, PartialEq)] +pub(crate) struct ProgressWrite { + /// The register as written. + pub(crate) record: ProgressRecord, + /// The session after the action (its generation for later checkpoints). + pub(crate) session: ProgressSession, + /// Operation appended to the sync log. + pub(crate) op: SyncOpWire, +} + +/// Result of [`PodcastStore::checkpoint`]. +#[derive(Debug, Clone, PartialEq)] +pub(crate) enum CheckpointOutcome { + /// The checkpoint was written. + Written(Box), + /// The register holds a greater generation (another device or an + /// explicit action took over). Nothing was written: stop writing at the + /// old generation, or continue locally with an explicit action + /// ([`PodcastStore::start_session`] / [`PodcastStore::seek`]). + Superseded { + /// Current register. + current: ProgressRecord, + }, + /// This session already completed the episode at its generation; a + /// checkpoint would turn it back into "in progress". Nothing was written. + AlreadyCompleted { + /// Current register. + current: ProgressRecord, + }, +} + +#[derive(Debug, Clone, Copy)] +enum ExplicitAction { + Start(i64), + Seek(i64), + Restart, + MarkPlayed, + MarkUnplayed, +} + +/// Validates the payload (with a provisional stamp), appends it to the sync +/// log with the next stamp of `device_id` and applies exactly that operation +/// through [`apply::apply_op`]. `make` receives the operation id, which +/// explicit actions without a session use as their session id. +pub(crate) async fn emit( + conn: &mut PgConnection, + user_id: i64, + device_id: &str, + make: impl Fn(&str) -> SyncOpPayload, +) -> Result { + SyncOpWire { + op_id: "provisional".to_string(), + origin_device_id: device_id.to_string(), + seq: 0, + hlc_ms: 0, + payload: make("provisional"), + } + .validate_podcast() + .map_err(invalid)?; + let op = devices::append_local_op(conn, user_id, device_id, &make).await?; + match apply::apply_op(conn, user_id, &op).await? { + ApplyOutcome::Rejected(reason) => Err(invalid(reason)), + _ => Ok(op), + } +} + +/// Resolves a feed URL against every known URL of every podcast of the +/// account: the alias sets, the catalog fetch URLs and the registers' +/// `podcast.feed_url` (`resolve_podcast_id`). +pub(crate) async fn resolve_podcast( + conn: &mut PgConnection, + user_id: i64, + url: &FeedUrl, +) -> Result { + let rows = sqlx::query( + "SELECT podcast_id FROM furumusic__podcast_feed_alias + WHERE user_id = $1 AND feed_url = $2 + UNION SELECT podcast_id FROM furumusic__podcast + WHERE user_id = $1 AND feed_url = $2 + UNION SELECT podcast_id FROM furumusic__podcast_subscription + WHERE user_id = $1 AND podcast_json->>'feed_url' = $2", + ) + .bind(user_id) + .bind(url.as_str()) + .fetch_all(&mut *conn) + .await?; + let ids = rows + .iter() + .map(|row| super::podcast_id_col(row, "podcast_id")) + .collect::>>()?; + Ok(resolve_podcast_id(url, ids.iter().map(|id| (id, url)))) +} + +async fn catalog_meta( + conn: &mut PgConnection, + user_id: i64, + podcast_id: &PodcastId, +) -> Result> { + let row = sqlx::query( + "SELECT feed_url, title, author, description, link, image_url + FROM furumusic__podcast WHERE user_id = $1 AND podcast_id = $2", + ) + .bind(user_id) + .bind(podcast_id.as_str()) + .fetch_optional(&mut *conn) + .await?; + let Some(row) = row else { + return Ok(None); + }; + Ok(Some(PodcastMeta { + feed_url: super::feed_url_col(&row, "feed_url")?, + title: row.try_get("title")?, + author: row.try_get("author")?, + description: row.try_get("description")?, + link: row.try_get("link")?, + image_url: row.try_get("image_url")?, + })) +} + +async fn share_payload( + conn: &mut PgConnection, + user_id: i64, + podcast_id: &PodcastId, + shared: bool, +) -> Result { + let subscription = apply::load_subscription_register(conn, user_id, podcast_id).await?; + let share = apply::load_share(conn, user_id, podcast_id).await?; + Ok(SyncOpPayload::PodcastShareSet { + podcast_id: podcast_id.clone(), + shared, + generation: next_podcast_generation(subscription.as_ref(), share.as_ref()), + }) +} + +/// The reference carried by operations: identity and display only. +fn op_ref(episode: &PodcastEpisodeRef) -> PodcastEpisodeRef { + let mut episode = episode.clone(); + episode.progress_generation = None; + episode +} + +impl PodcastStore { + /// Subscribes to the podcast of `podcast.feed_url` (explicit user action). + /// + /// The URL is resolved against known feed URLs and aliases first, so + /// subscribing again, through a redirect target or from someone else's + /// card reuses the existing podcast with its progress and history. When + /// the podcast is already subscribed nothing is emitted. Otherwise one + /// `podcast_subscription_set` with generation + /// `next_podcast_generation(..)` and the known aliases is appended and + /// applied. The new subscription is private until + /// [`PodcastStore::set_shared`]. + /// + /// Pass the channel metadata of the fetched feed + /// (`ParsedFeed::to_podcast_meta(url)`), then ingest the episodes with + /// [`PodcastStore::apply_feed`]. + pub(crate) async fn subscribe( + &self, + user_id: i64, + podcast: &PodcastMeta, + ) -> Result { + let meta = podcast.bounded(); + meta.validate().map_err(invalid)?; + let device_id = devices::local_device_id(&self.pool, user_id).await?; + let mut tx = self.begin_write(user_id).await?; + let podcast_id = resolve_podcast(&mut tx, user_id, &meta.feed_url).await?; + let current = apply::load_subscription_register(&mut tx, user_id, &podcast_id).await?; + if current.as_ref().is_some_and(|record| record.subscribed) { + return Ok(SubscribeOutcome { + podcast_id, + already_subscribed: true, + ops: Vec::new(), + }); + } + let active: i64 = sqlx::query_scalar( + "SELECT COUNT(*) FROM furumusic__podcast_subscription + WHERE user_id = $1 AND subscribed", + ) + .bind(user_id) + .fetch_one(&mut *tx) + .await?; + if usize::try_from(active).unwrap_or(usize::MAX) >= super::MAX_SUBSCRIPTIONS_PER_ACCOUNT { + return Err( + PodcastStoreError::SubscriptionLimit(super::MAX_SUBSCRIPTIONS_PER_ACCOUNT).into(), + ); + } + let share = apply::load_share(&mut tx, user_id, &podcast_id).await?; + let generation = next_podcast_generation(current.as_ref(), share.as_ref()); + let aliases = merge_feed_aliases( + &podcast_id, + &apply::load_aliases(&mut tx, user_id, &podcast_id).await?, + std::slice::from_ref(&meta.feed_url), + ); + let op = emit(&mut tx, user_id, &device_id, |_| { + SyncOpPayload::PodcastSubscriptionSet { + podcast_id: podcast_id.clone(), + subscribed: true, + podcast: meta.clone(), + feed_aliases: aliases.clone(), + generation, + } + }) + .await?; + tx.commit().await?; + // A new subscription is private, but it supersedes an earlier share + // decision of a former subscription. + super::discovery::request_reconcile(); + Ok(SubscribeOutcome { + podcast_id, + already_subscribed: false, + ops: vec![op], + }) + } + + /// Unsubscribes (explicit user action). + /// + /// Emits `podcast_subscription_set { subscribed: false }` followed by + /// `podcast_share_set { shared: false }`, each with the next podcast + /// generation. Progress, sessions and the catalog are kept, so + /// subscribing again continues where the user stopped. Does nothing when + /// the podcast is not subscribed. + pub(crate) async fn unsubscribe( + &self, + user_id: i64, + podcast_id: &PodcastId, + ) -> Result> { + let device_id = devices::local_device_id(&self.pool, user_id).await?; + let mut tx = self.begin_write(user_id).await?; + let Some(current) = apply::load_subscription_register(&mut tx, user_id, podcast_id).await? + else { + return Ok(Vec::new()); + }; + if !current.subscribed { + return Ok(Vec::new()); + } + let share = apply::load_share(&mut tx, user_id, podcast_id).await?; + let generation = next_podcast_generation(Some(¤t), share.as_ref()); + let mut meta = catalog_meta(&mut tx, user_id, podcast_id) + .await? + .map(|meta| meta.bounded()) + .unwrap_or_else(|| current.podcast.clone()); + meta.feed_url = current.podcast.feed_url.clone(); + let aliases = current.feed_aliases.clone(); + let unsubscribe = emit(&mut tx, user_id, &device_id, |_| { + SyncOpPayload::PodcastSubscriptionSet { + podcast_id: podcast_id.clone(), + subscribed: false, + podcast: meta.clone(), + feed_aliases: aliases.clone(), + generation, + } + }) + .await?; + let revoke_payload = share_payload(&mut tx, user_id, podcast_id, false).await?; + let revoke = emit(&mut tx, user_id, &device_id, |_| revoke_payload.clone()).await?; + super::discovery::clear_confirmation(&mut tx, user_id, podcast_id).await?; + tx.commit().await?; + super::discovery::request_reconcile(); + Ok(vec![unsubscribe, revoke]) + } + + /// Sets the decision to publish the podcast to the federation. + /// + /// Sharing needs an active subscription and is a no-op when the podcast + /// is already effectively shared (`effective_share`). Revoking is a no-op + /// when no share decision says `true`. The write uses the next podcast + /// generation, so it orders after the subscription it observed. + /// + /// Revoking also forgets the account's privacy confirmation, so sharing + /// again asks again. Publishing and revoking the card is the + /// reconciler's job (`super::discovery`), which this requests. + pub(crate) async fn set_shared( + &self, + user_id: i64, + podcast_id: &PodcastId, + shared: bool, + ) -> Result> { + let device_id = devices::local_device_id(&self.pool, user_id).await?; + let mut tx = self.begin_write(user_id).await?; + let subscription = apply::load_subscription_register(&mut tx, user_id, podcast_id).await?; + let share = apply::load_share(&mut tx, user_id, podcast_id).await?; + if shared { + if !subscription + .as_ref() + .is_some_and(|record| record.subscribed) + { + return Err(PodcastStoreError::NotSubscribed(podcast_id.clone()).into()); + } + if effective_share(subscription.as_ref(), share.as_ref()) { + return Ok(Vec::new()); + } + } else if !share.as_ref().is_some_and(|record| record.shared) { + return Ok(Vec::new()); + } + let payload = share_payload(&mut tx, user_id, podcast_id, shared).await?; + let op = emit(&mut tx, user_id, &device_id, |_| payload.clone()).await?; + if !shared { + super::discovery::clear_confirmation(&mut tx, user_id, podcast_id).await?; + } + tx.commit().await?; + super::discovery::request_reconcile(); + Ok(vec![op]) + } + + /// Shares a podcast whose card has privacy warnings, with the + /// account's confirmation of exactly `feed_url` / `id_feed_url`. + /// + /// In one transaction: the card the account would publish now is + /// rebuilt and compared with the confirmed URLs (a feed that moved + /// meanwhile returns `Ok(None)` and writes nothing, so the user is asked + /// about the new URLs); the share decision is written like + /// [`PodcastStore::set_shared`] when the podcast is not already + /// effectively shared; the confirmation is stored for the resulting + /// share decision (`ShareRecord::op_id`). Returns the emitted operations. + pub(crate) async fn share_confirmed( + &self, + user_id: i64, + podcast_id: &PodcastId, + feed_url: &FeedUrl, + id_feed_url: Option<&FeedUrl>, + now_ms: i64, + ) -> Result>> { + let device_id = devices::local_device_id(&self.pool, user_id).await?; + let mut tx = self.begin_write(user_id).await?; + let Some(account) = + super::discovery::load_account_share(&mut tx, user_id, podcast_id).await? + else { + return Err(PodcastStoreError::NotSubscribed(podcast_id.clone()).into()); + }; + if !account.subscription.subscribed { + return Err(PodcastStoreError::NotSubscribed(podcast_id.clone()).into()); + } + let card = account.card.as_ref().map_err(invalid)?; + if card.feed_url != *feed_url || card.id_feed_url.as_ref() != id_feed_url { + return Ok(None); + } + let mut ops = Vec::new(); + let share_op_id = if account.is_shared() { + account + .share + .as_ref() + .map(|share| share.op_id.clone()) + .expect("an effective share has a share register") + } else { + let payload = share_payload(&mut tx, user_id, podcast_id, true).await?; + let op = emit(&mut tx, user_id, &device_id, |_| payload.clone()).await?; + let op_id = op.op_id.clone(); + ops.push(op); + op_id + }; + let confirmation = super::discovery::ShareConfirmation { + share_op_id, + feed_url: card.feed_url.clone(), + id_feed_url: card.id_feed_url.clone(), + }; + super::discovery::store_confirmation(&mut tx, user_id, podcast_id, &confirmation, now_ms) + .await?; + tx.commit().await?; + super::discovery::request_reconcile(); + Ok(Some(ops)) + } + + /// Forgets the account's privacy confirmation of a podcast. + pub(crate) async fn clear_share_confirmation( + &self, + user_id: i64, + podcast_id: &PodcastId, + ) -> Result<()> { + let mut conn = self.pool.acquire().await?; + super::discovery::clear_confirmation(&mut conn, user_id, podcast_id).await + } + + /// One account's publication view of a podcast (registers, card, + /// confirmation); `None` without a subscription register. + pub(crate) async fn account_share( + &self, + user_id: i64, + podcast_id: &PodcastId, + ) -> Result> { + let mut conn = self.pool.acquire().await?; + super::discovery::load_account_share(&mut conn, user_id, podcast_id).await + } + + async fn explicit_progress( + &self, + user_id: i64, + episode: &PodcastEpisodeRef, + session_id: Option<&str>, + carried_generation: Option, + action: ExplicitAction, + ) -> Result { + episode.validate().map_err(invalid)?; + let device_id = devices::local_device_id(&self.pool, user_id).await?; + let mut tx = self.begin_write(user_id).await?; + let current = apply::load_progress(&mut tx, user_id, &episode.episode_id).await?; + let max_seen = current + .as_ref() + .map(|record| record.generation) + .into_iter() + .chain(carried_generation) + .max(); + let generation = ProgressRecord::explicit_generation(max_seen); + let current_position = current.as_ref().map_or(0, |record| record.position_ms); + let (position_ms, status) = match action { + ExplicitAction::Start(position) | ExplicitAction::Seek(position) => { + (position, ProgressStatus::InProgress) + } + ExplicitAction::Restart => (0, ProgressStatus::InProgress), + ExplicitAction::MarkPlayed => (current_position, ProgressStatus::Completed), + ExplicitAction::MarkUnplayed => (0, ProgressStatus::NotStarted), + }; + let reference = op_ref(episode); + let op = emit(&mut tx, user_id, &device_id, |op_id| { + SyncOpPayload::PodcastProgressSet { + episode_id: episode.episode_id.clone(), + podcast_id: episode.podcast_id.clone(), + position_ms, + status, + generation, + session_id: session_id.unwrap_or(op_id).to_string(), + episode: Some(reference.clone()), + } + }) + .await?; + tx.commit().await?; + let record = op.podcast_progress().expect("progress payload"); + Ok(ProgressWrite { + session: ProgressSession { + episode: episode.clone(), + session_id: record.session_id.clone(), + generation, + }, + record, + op, + }) + } + + /// Starts a listening session at `position_ms` (explicit action). + /// + /// The generation is `explicit_generation(max(local, carried_generation))`; + /// pass `PodcastEpisodeRef::progress_generation` of a queue item received + /// through a handoff as `carried_generation`. Ordinary Play resumes from + /// [`PodcastStore::resume_position_ms`]. + pub(crate) async fn start_session( + &self, + user_id: i64, + episode: &PodcastEpisodeRef, + session_id: &str, + position_ms: i64, + carried_generation: Option, + ) -> Result { + self.explicit_progress( + user_id, + episode, + Some(session_id), + carried_generation, + ExplicitAction::Start(position_ms), + ) + .await + } + + /// Periodic checkpoint (or pause/stop save) of a running session. + /// + /// Keeps the session generation. When the register already holds a + /// greater generation nothing is written ([`CheckpointOutcome::Superseded`]). + pub(crate) async fn checkpoint( + &self, + user_id: i64, + session: &ProgressSession, + position_ms: i64, + ) -> Result { + let episode = &session.episode; + let device_id = devices::local_device_id(&self.pool, user_id).await?; + let mut tx = self.begin_write(user_id).await?; + if let Some(current) = apply::load_progress(&mut tx, user_id, &episode.episode_id).await? { + if current.generation > session.generation { + return Ok(CheckpointOutcome::Superseded { current }); + } + if current.generation == session.generation + && current.session_id == session.session_id + && current.status == ProgressStatus::Completed + { + return Ok(CheckpointOutcome::AlreadyCompleted { current }); + } + } + let reference = op_ref(episode); + let op = emit(&mut tx, user_id, &device_id, |_| { + SyncOpPayload::PodcastProgressSet { + episode_id: episode.episode_id.clone(), + podcast_id: episode.podcast_id.clone(), + position_ms, + status: ProgressStatus::InProgress, + generation: session.generation, + session_id: session.session_id.clone(), + episode: Some(reference.clone()), + } + }) + .await?; + tx.commit().await?; + let record = op.podcast_progress().expect("progress payload"); + Ok(CheckpointOutcome::Written(Box::new(ProgressWrite { + record, + session: session.clone(), + op, + }))) + } + + /// User seek during a session (explicit action; the returned session + /// carries the new generation for later checkpoints). + pub(crate) async fn seek( + &self, + user_id: i64, + session: &ProgressSession, + position_ms: i64, + ) -> Result { + self.explicit_progress( + user_id, + &session.episode, + Some(&session.session_id), + Some(session.generation), + ExplicitAction::Seek(position_ms), + ) + .await + } + + /// Completes a running session (end of stream or `reaches_completion`) + /// unless another write took over. Record the session itself with + /// [`PodcastStore::record_listen_session`]. + /// + /// Like [`PodcastStore::checkpoint`], nothing is written when the + /// register holds a greater generation ([`CheckpointOutcome::Superseded`]) + /// or when this session already completed the episode + /// ([`CheckpointOutcome::AlreadyCompleted`]). Otherwise `completed` is + /// written with an explicit generation; the check and the write + /// happen in one transaction, so a periodic save after the end never + /// completes the episode twice. + pub(crate) async fn complete_if_current( + &self, + user_id: i64, + session: &ProgressSession, + position_ms: i64, + ) -> Result { + let episode = &session.episode; + episode.validate().map_err(invalid)?; + let device_id = devices::local_device_id(&self.pool, user_id).await?; + let mut tx = self.begin_write(user_id).await?; + let current = apply::load_progress(&mut tx, user_id, &episode.episode_id).await?; + if let Some(current) = current.as_ref() { + if current.generation > session.generation { + return Ok(CheckpointOutcome::Superseded { + current: current.clone(), + }); + } + if current.session_id == session.session_id + && current.status == ProgressStatus::Completed + { + return Ok(CheckpointOutcome::AlreadyCompleted { + current: current.clone(), + }); + } + } + let generation = ProgressRecord::explicit_generation( + current + .as_ref() + .map(|record| record.generation) + .into_iter() + .chain(Some(session.generation)) + .max(), + ); + let reference = op_ref(episode); + let op = emit(&mut tx, user_id, &device_id, |_| { + SyncOpPayload::PodcastProgressSet { + episode_id: episode.episode_id.clone(), + podcast_id: episode.podcast_id.clone(), + position_ms, + status: ProgressStatus::Completed, + generation, + session_id: session.session_id.clone(), + episode: Some(reference.clone()), + } + }) + .await?; + tx.commit().await?; + let record = op.podcast_progress().expect("progress payload"); + Ok(CheckpointOutcome::Written(Box::new(ProgressWrite { + record, + session: ProgressSession { + episode: episode.clone(), + session_id: session.session_id.clone(), + generation, + }, + op, + }))) + } + + /// "Start over": position 0, in progress, explicit generation. + pub(crate) async fn restart( + &self, + user_id: i64, + episode: &PodcastEpisodeRef, + session_id: &str, + ) -> Result { + self.explicit_progress( + user_id, + episode, + Some(session_id), + None, + ExplicitAction::Restart, + ) + .await + } + + /// "Mark played": status completed, explicit generation. Creates no + /// listening session and invents no listening time. + pub(crate) async fn mark_played( + &self, + user_id: i64, + episode: &PodcastEpisodeRef, + ) -> Result { + self.explicit_progress(user_id, episode, None, None, ExplicitAction::MarkPlayed) + .await + } + + /// "Mark unplayed": status not started at position 0, explicit generation. + pub(crate) async fn mark_unplayed( + &self, + user_id: i64, + episode: &PodcastEpisodeRef, + ) -> Result { + self.explicit_progress(user_id, episode, None, None, ExplicitAction::MarkUnplayed) + .await + } + + /// Records (or grows) a listening session of this account's device. + /// + /// Sessions below the history threshold (`ListenSession::should_record`) + /// and versions that do not grow the stored session emit nothing. + /// `session.device_id` must be the account's web device id. The episode + /// reference defaults to the catalog's. + pub(crate) async fn record_listen_session( + &self, + user_id: i64, + session: &ListenSession, + episode: Option<&PodcastEpisodeRef>, + ) -> Result> { + let device_id = devices::local_device_id(&self.pool, user_id).await?; + if session.device_id != device_id { + return Err(invalid( + "session device_id differs from the local device id", + )); + } + session.validate().map_err(invalid)?; + if !session.should_record() { + return Ok(None); + } + let mut tx = self.begin_write(user_id).await?; + if let Some(current) = apply::load_session(&mut tx, user_id, &session.session_id).await? { + let merged = merge_listen_session(¤t, session).map_err(invalid)?; + if merged == current { + return Ok(None); + } + } + let reference = match episode { + Some(episode) => Some(op_ref(episode)), + None => super::queries::episode_ref_in(&mut tx, user_id, &session.episode_id) + .await? + .map(|reference| op_ref(&reference)), + }; + let op = emit(&mut tx, user_id, &device_id, |_| { + SyncOpPayload::PodcastListenRecorded { + session: session.clone(), + episode: reference.clone(), + } + }) + .await?; + tx.commit().await?; + Ok(Some(op)) + } + + /// Makes sure the catalog knows an episode a queue item refers to. + /// + /// Queue items received through a handoff carry a self-contained + /// `PodcastEpisodeRef`; the account may not have refreshed that feed yet. + /// The reference is recorded like one learned through sync + /// (`source = 'sync'`, never replacing feed metadata). Returns whether a + /// row was added. No register is written. A new row is refused with + /// [`PodcastStoreError::ReferenceLimit`] once the account has + /// [`super::MAX_REFERENCE_EPISODES_PER_ACCOUNT`] such episodes. + pub(crate) async fn ensure_episode( + &self, + user_id: i64, + episode: &PodcastEpisodeRef, + ) -> Result { + episode.validate().map_err(invalid)?; + let mut tx = self.begin_write(user_id).await?; + let known = super::queries::episode_ref_in(&mut tx, user_id, &episode.episode_id) + .await? + .is_some(); + if !known { + // Counted under the account's write lock: concurrent requests + // cannot pass the limit together. + let references: i64 = sqlx::query_scalar( + "SELECT COUNT(*) FROM furumusic__podcast_episode + WHERE user_id = $1 AND source = 'sync'", + ) + .bind(user_id) + .fetch_one(&mut *tx) + .await?; + if usize::try_from(references).unwrap_or(usize::MAX) + >= super::MAX_REFERENCE_EPISODES_PER_ACCOUNT + { + return Err(PodcastStoreError::ReferenceLimit( + super::MAX_REFERENCE_EPISODES_PER_ACCOUNT, + ) + .into()); + } + apply::ensure_episode_from_ref(&mut tx, user_id, episode).await?; + } + tx.commit().await?; + Ok(!known) + } + + /// Resolves a feed URL to a podcast id: the existing podcast when the + /// URL is a known feed URL or alias (`resolve_podcast_id`), otherwise + /// the id a first subscription through this URL would derive. + pub(crate) async fn resolve_podcast_id( + &self, + user_id: i64, + url: &FeedUrl, + ) -> Result { + let mut conn = self.pool.acquire().await?; + resolve_podcast(&mut conn, user_id, url).await + } + + /// The episode reference for queueing or progress actions (with + /// `progress_generation` set to the stored register's generation), or + /// `None` when the episode is unknown. + pub(crate) async fn episode_ref( + &self, + user_id: i64, + episode_id: &EpisodeId, + ) -> Result> { + let mut conn = self.pool.acquire().await?; + super::queries::episode_ref_in(&mut conn, user_id, episode_id).await + } +} diff --git a/src/podcasts/api.rs b/src/podcasts/api.rs new file mode 100644 index 0000000..9ba65f1 --- /dev/null +++ b/src/podcasts/api.rs @@ -0,0 +1,1390 @@ +//! Request handling of the podcast UI and the web player. +//! +//! [`super::http`] authenticates the request and passes the account here. +//! Every function takes the store, the account and the decoded input and +//! returns a JSON view or an [`ApiError`] with its HTTP status, so the +//! behavior of the endpoints is testable without an HTTP server +//! (`postgres_tests`). +//! +//! Playback follows the progress rules of the frid podcast contract +//! (`PODCASTS.md`, "Progress" and "Listening sessions and counts"): +//! +//! * [`playback`] prepares an episode for the player: the audio source, the +//! position ordinary Play resumes from and, when the browser starts +//! playing (`start`), a new listening session written as an explicit +//! action (`explicit_generation(max(local, carried))`). +//! * [`progress`] stores what the player reports for that session: periodic +//! checkpoints and pause/stop/handoff saves keep the session generation, a +//! user seek is an explicit action, the end of the stream or +//! `reaches_completion` completes the episode once, and the listening +//! session is recorded (history counts only completed sessions). + +use music_dht::device_sync::PlaybackTrack; +use music_dht::podcast::{ + EpisodeId, EpisodeType, FeedUrl, ListenSession, PodcastEpisodeRef, PodcastId, PodcastMeta, + ProgressRecord, ProgressStatus, reaches_completion, +}; +use music_dht::podcast_dht::FeedUrlPrivacyWarning; +use serde::{Deserialize, Serialize}; +use serde_json::Value; + +use super::discovery::Discovery; +use super::fetch::{FeedFetcher, FetchErrorKind}; +use super::runner::{self, SubscribeUrlError}; +use super::search; +use super::{ + CheckpointOutcome, EpisodeListItem, EpisodeQuery, EpisodeSort, FetchState, PlayedFilter, + PodcastListFilter, PodcastStore, PodcastStoreError, PodcastSummary, ProgressSession, + SeasonFilter, SeasonSummary, +}; +use crate::federation::devices; +use crate::player::queue_items; + +/// Episodes per page when the request does not say. +pub(crate) const DEFAULT_EPISODE_PAGE: usize = 50; +/// Largest episode page. +pub(crate) const MAX_EPISODE_PAGE: usize = 200; + +/// A failed request: HTTP status and a message for the user. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct ApiError { + pub(crate) status: u16, + pub(crate) message: String, +} + +impl ApiError { + pub(crate) fn new(status: u16, message: impl Into) -> Self { + Self { + status, + message: message.into(), + } + } + + fn bad_request(message: impl Into) -> Self { + Self::new(400, message) + } + + fn not_found(message: impl Into) -> Self { + Self::new(404, message) + } +} + +impl From for ApiError { + fn from(error: anyhow::Error) -> Self { + if let Some(error) = error.downcast_ref::() { + return match error { + PodcastStoreError::UnknownPodcast(_) => Self::not_found(error.to_string()), + PodcastStoreError::NotSubscribed(_) => Self::new(409, error.to_string()), + PodcastStoreError::Invalid(_) => Self::bad_request(error.to_string()), + PodcastStoreError::SubscriptionLimit(_) | PodcastStoreError::ReferenceLimit(_) => { + Self::new(422, error.to_string()) + } + }; + } + if let Some(SubscribeUrlError::Fetch(fetch)) = error.downcast_ref::() { + // The feed could not be used: the request cannot succeed as is. + let status = match fetch.kind { + FetchErrorKind::Timeout | FetchErrorKind::Transport => 502, + _ => 422, + }; + // The user sees the kind of failure only (no resolved + // addresses or transport details); the log keeps the details. + tracing::info!("podcast feed could not be added: {}", fetch.message); + return Self::new(status, error.to_string()); + } + tracing::warn!("podcast API request failed: {error:#}"); + Self::new(500, "internal error") + } +} + +pub(crate) type ApiResult = Result; + +pub(crate) fn parse_podcast_id(value: &str) -> ApiResult { + PodcastId::parse(value).map_err(|_| ApiError::bad_request("invalid podcast id")) +} + +pub(crate) fn parse_episode_id(value: &str) -> ApiResult { + EpisodeId::parse(value).map_err(|_| ApiError::bad_request("invalid episode id")) +} + +fn decode(body: Value) -> ApiResult { + serde_json::from_value(body).map_err(|error| ApiError::bad_request(error.to_string())) +} + +fn status_name(status: ProgressStatus) -> &'static str { + match status { + ProgressStatus::NotStarted => "not_started", + ProgressStatus::InProgress => "in_progress", + ProgressStatus::Completed => "completed", + } +} + +fn episode_type_name(kind: EpisodeType) -> &'static str { + match kind { + EpisodeType::Full => "full", + EpisodeType::Trailer => "trailer", + EpisodeType::Bonus => "bonus", + } +} + +// --------------------------------------------------------------------------- +// Views +// --------------------------------------------------------------------------- + +/// Local fetch state shown next to a podcast (never synced). +#[derive(Debug, Clone, Serialize)] +pub(crate) struct FetchView { + pub(crate) checked_at_ms: Option, + pub(crate) last_success_at_ms: Option, + pub(crate) next_check_at_ms: Option, + pub(crate) last_error: Option, + pub(crate) consecutive_failures: u32, +} + +impl From for FetchView { + fn from(state: FetchState) -> Self { + Self { + checked_at_ms: state.checked_at_ms, + last_success_at_ms: state.last_success_at_ms, + next_check_at_ms: state.next_check_at_ms, + last_error: state.last_error, + consecutive_failures: state.consecutive_failures, + } + } +} + +/// A podcast with its personal state and counters. +#[derive(Debug, Clone, Serialize)] +pub(crate) struct PodcastView { + pub(crate) podcast_id: String, + /// URL this account fetches. + pub(crate) feed_url: String, + pub(crate) title: String, + pub(crate) author: Option, + pub(crate) description: Option, + pub(crate) link: Option, + pub(crate) image_url: Option, + pub(crate) subscribed: bool, + /// The account's synced share decision applies (`effective_share`). + /// Whether the card is actually published is [`ShareView`]. + pub(crate) shared: bool, + pub(crate) episode_count: u64, + pub(crate) unplayed_count: u64, + pub(crate) in_progress_count: u64, + pub(crate) season_count: u64, + pub(crate) latest_published_at_ms: Option, + pub(crate) fetch: FetchView, +} + +impl PodcastView { + fn new(summary: PodcastSummary, fetch: FetchState) -> Self { + Self { + podcast_id: summary.podcast_id.to_string(), + feed_url: summary.meta.feed_url.to_string(), + title: summary.meta.title, + author: summary.meta.author, + description: summary.meta.description, + link: summary.meta.link, + image_url: summary.meta.image_url, + subscribed: summary.subscribed, + shared: summary.shared, + episode_count: summary.episode_count, + unplayed_count: summary.unplayed_count, + in_progress_count: summary.in_progress_count, + season_count: summary.season_count, + latest_published_at_ms: summary.latest_published_at_ms, + fetch: fetch.into(), + } + } +} + +/// One season group (`season: null` holds the episodes without a season). +#[derive(Debug, Clone, Serialize)] +pub(crate) struct SeasonView { + pub(crate) season: Option, + pub(crate) episode_count: u64, + pub(crate) unplayed_count: u64, + pub(crate) first_published_at_ms: Option, + pub(crate) last_published_at_ms: Option, +} + +impl From for SeasonView { + fn from(season: SeasonSummary) -> Self { + Self { + season: season.season, + episode_count: season.episode_count, + unplayed_count: season.unplayed_count, + first_published_at_ms: season.first_published_at_ms, + last_published_at_ms: season.last_published_at_ms, + } + } +} + +/// An episode with its progress and the queue item the player uses. +#[derive(Debug, Clone, Serialize)] +pub(crate) struct EpisodeView { + pub(crate) episode_id: String, + pub(crate) podcast_id: String, + pub(crate) title: String, + pub(crate) description: Option, + pub(crate) season: Option, + pub(crate) episode_number: Option, + pub(crate) episode_type: &'static str, + pub(crate) published_at_ms: Option, + pub(crate) duration_ms: Option, + pub(crate) image_url: Option, + pub(crate) link: Option, + /// Whether the latest refresh listed the episode (older episodes stay). + pub(crate) in_feed: bool, + /// `not_started`, `in_progress` or `completed`. + pub(crate) status: &'static str, + /// Stored position of the progress register. + pub(crate) position_ms: i64, + /// Where ordinary Play resumes (`ProgressRecord::resume_position_ms`). + pub(crate) resume_position_ms: i64, + /// Distinct completed listening sessions. + pub(crate) completed_count: u64, + /// Browser queue item: `PlaybackSource::PodcastEpisode` with the + /// self-contained reference (see `player::queue_items`). + pub(crate) track: Value, +} + +/// The queue reference of a catalog episode, with the register generation +/// as handoff hint. +fn episode_ref(item: &EpisodeListItem, podcast: &PodcastMeta) -> PodcastEpisodeRef { + let mut reference = item.episode.to_ref(podcast); + reference.progress_generation = item.progress.as_ref().map(|record| record.generation); + reference +} + +/// Browser queue item of an episode reference. +pub(crate) fn episode_track_json(reference: PodcastEpisodeRef) -> Value { + queue_items::external_track_json(&PlaybackTrack::podcast_episode(reference), 0, None) +} + +impl EpisodeView { + fn new(item: EpisodeListItem, podcast: &PodcastMeta) -> Self { + let track = episode_track_json(episode_ref(&item, podcast)); + let status = status_name(item.status()); + let resume_position_ms = item.resume_position_ms(); + let position_ms = item + .progress + .as_ref() + .map_or(0, |progress| progress.position_ms); + let episode = item.episode; + Self { + episode_id: episode.episode_id.to_string(), + podcast_id: episode.podcast_id.to_string(), + title: episode.title, + description: episode.description, + season: episode.season, + episode_number: episode.episode_number, + episode_type: episode_type_name(episode.episode_type), + published_at_ms: episode.published_at_ms, + duration_ms: episode.duration_ms, + image_url: episode.image_url, + link: episode.link, + in_feed: item.in_feed, + status, + position_ms, + resume_position_ms, + completed_count: item.completed_count, + track, + } + } +} + +async fn podcast_view( + store: &PodcastStore, + user_id: i64, + podcast_id: &PodcastId, +) -> ApiResult { + let summary = store + .podcast(user_id, podcast_id) + .await? + .ok_or_else(|| ApiError::not_found("unknown podcast"))?; + let fetch = store.fetch_state(user_id, podcast_id).await?; + Ok(PodcastView::new(summary, fetch)) +} + +async fn episode_view( + store: &PodcastStore, + user_id: i64, + episode_id: &EpisodeId, +) -> ApiResult { + let item = store + .episode(user_id, episode_id) + .await? + .ok_or_else(|| ApiError::not_found("unknown episode"))?; + let podcast = store + .podcast(user_id, &item.episode.podcast_id) + .await? + .ok_or_else(|| ApiError::not_found("unknown podcast"))?; + Ok(EpisodeView::new(item, &podcast.meta)) +} + +async fn catalog_ref( + store: &PodcastStore, + user_id: i64, + episode_id: &EpisodeId, +) -> ApiResult { + store + .episode_ref(user_id, episode_id) + .await? + .ok_or_else(|| ApiError::not_found("unknown episode")) +} + +// --------------------------------------------------------------------------- +// Podcasts +// --------------------------------------------------------------------------- + +#[derive(Debug, Serialize)] +pub(crate) struct PodcastList { + pub(crate) podcasts: Vec, +} + +/// Podcasts of the account: subscribed ones, or with `all` also +/// unsubscribed podcasts and podcasts only known from synced progress. +pub(crate) async fn list_podcasts( + store: &PodcastStore, + user_id: i64, + all: bool, +) -> ApiResult { + let filter = if all { + PodcastListFilter::All + } else { + PodcastListFilter::Subscribed + }; + let mut podcasts = Vec::new(); + for summary in store.list_podcasts(user_id, filter).await? { + let fetch = store.fetch_state(user_id, &summary.podcast_id).await?; + podcasts.push(PodcastView::new(summary, fetch)); + } + Ok(PodcastList { podcasts }) +} + +/// Listening sessions shown in a podcast's history. +pub(crate) const DETAIL_HISTORY_LIMIT: usize = 20; + +/// A listening session in the history (completed or at least 30 s). +#[derive(Debug, Clone, Serialize)] +pub(crate) struct HistoryView { + pub(crate) episode_id: String, + pub(crate) episode_title: Option, + pub(crate) started_at_ms: i64, + pub(crate) ended_at_ms: i64, + pub(crate) listened_ms: i64, + pub(crate) completed: bool, +} + +#[derive(Debug, Serialize)] +pub(crate) struct PodcastDetail { + pub(crate) podcast: PodcastView, + /// Public discovery state of the podcast for this account. + pub(crate) share: ShareView, + pub(crate) seasons: Vec, + /// Newest listening sessions of this podcast, from every device. + pub(crate) history: Vec, +} + +/// One podcast with its season groups and its public discovery state +/// (`discovery`: whether the server's overlay runs). +pub(crate) async fn podcast_detail( + store: &PodcastStore, + user_id: i64, + podcast_id: &PodcastId, + discovery: Discovery<'_>, +) -> ApiResult { + let podcast = podcast_view(store, user_id, podcast_id).await?; + let share = share_view(store, user_id, podcast_id, discovery).await?; + let seasons = store + .seasons(user_id, podcast_id) + .await? + .into_iter() + .map(SeasonView::from) + .collect(); + let history = store + .listen_history(user_id, Some(podcast_id), DETAIL_HISTORY_LIMIT) + .await? + .into_iter() + .map(|entry| HistoryView { + episode_id: entry.session.episode_id.to_string(), + episode_title: entry.episode_title, + started_at_ms: entry.session.started_at_ms, + ended_at_ms: entry.session.ended_at_ms, + listened_ms: entry.session.listened_ms, + completed: entry.session.completed, + }) + .collect(); + Ok(PodcastDetail { + podcast, + share, + seasons, + history, + }) +} + +#[derive(Debug, Deserialize)] +struct SubscribeRequest { + url: String, +} + +#[derive(Debug, Serialize)] +pub(crate) struct SubscribeResponse { + pub(crate) podcast: PodcastView, + pub(crate) already_subscribed: bool, + pub(crate) new_episodes: usize, +} + +/// "Add RSS": `{ "url": "..." }`. The feed is fetched before anything is +/// stored; the new subscription is private. +pub(crate) async fn subscribe_url( + store: &PodcastStore, + fetcher: &FeedFetcher, + user_id: i64, + body: Value, + now_ms: i64, +) -> ApiResult { + let request: SubscribeRequest = decode(body)?; + let url = FeedUrl::from_user_input(&request.url) + .map_err(|error| ApiError::bad_request(format!("invalid feed URL: {error}")))?; + let subscription = runner::subscribe_url(store, fetcher, user_id, &url, now_ms).await?; + Ok(SubscribeResponse { + podcast: podcast_view(store, user_id, &subscription.podcast_id).await?, + already_subscribed: subscription.already_subscribed, + new_episodes: subscription.refresh.new_episodes, + }) +} + +/// Subscribes again to a known podcast with its stored channel metadata +/// (progress and history continue). The refresh job fetches it on its next +/// run. +pub(crate) async fn resubscribe( + store: &PodcastStore, + user_id: i64, + podcast_id: &PodcastId, +) -> ApiResult { + let summary = store + .podcast(user_id, podcast_id) + .await? + .ok_or_else(|| ApiError::not_found("unknown podcast"))?; + store.subscribe(user_id, &summary.meta).await?; + podcast_view(store, user_id, podcast_id).await +} + +/// Unsubscribes: stops monitoring, revokes a publication, keeps progress +/// and history. +pub(crate) async fn unsubscribe( + store: &PodcastStore, + user_id: i64, + podcast_id: &PodcastId, +) -> ApiResult { + if store.podcast(user_id, podcast_id).await?.is_none() { + return Err(ApiError::not_found("unknown podcast")); + } + store.unsubscribe(user_id, podcast_id).await?; + podcast_view(store, user_id, podcast_id).await +} + +// --------------------------------------------------------------------------- +// Sharing (public discovery) +// --------------------------------------------------------------------------- + +/// Public discovery state of one podcast for the signed-in account. +/// +/// Sharing is the account's synced decision; the card is published by the +/// server's federation node (see [`super::discovery`]) and contains only +/// the podcast's public metadata. +#[derive(Debug, Clone, Serialize)] +pub(crate) struct ShareView { + /// The account's share decision applies (`effective_share`). + pub(crate) shared: bool, + /// Shared, but the card's privacy warnings are not confirmed by this + /// account for the current decision and URLs: not published for it. + pub(crate) pending_confirmation: bool, + /// Whether the account can share the podcast now (subscribed and the + /// card can be built). + pub(crate) can_share: bool, + /// Why the card cannot be built, if it cannot. + pub(crate) card_error: Option, + /// Why the card's feed URLs may be private (`PodcastCardSpec:: + /// privacy_warnings`); enabling sharing must be confirmed when present. + pub(crate) warnings: Vec, + /// Feed URL the card would publish. + pub(crate) feed_url: Option, + /// Origin URL the podcast id comes from, when the feed moved. + pub(crate) id_feed_url: Option, + /// `running` when the server's node runs public discovery, + /// `unavailable` when the node runs but the overlay failed to start, + /// `off` when federation is not running. + pub(crate) discovery: &'static str, + /// This server's node currently publishes a card of this podcast and + /// this account shares it. Only this server's own publication: another + /// player of the account (a TUI) publishes its own card from its own + /// node, which this flag does not show. + pub(crate) published: bool, + /// Why the server does not publish a shared, publishable podcast after + /// its last round: `over_limit` (beyond this account's part of the + /// server's publication budget) or `failed` (the card could not be + /// signed or stored). `null` otherwise. + pub(crate) held: Option<&'static str>, +} + +/// The account's [`ShareView`] of a podcast. +pub(crate) async fn share_view( + store: &PodcastStore, + user_id: i64, + podcast_id: &PodcastId, + discovery: Discovery<'_>, +) -> ApiResult { + let account = store.account_share(user_id, podcast_id).await?; + let mut view = ShareView { + shared: false, + pending_confirmation: false, + can_share: false, + card_error: None, + warnings: Vec::new(), + feed_url: None, + id_feed_url: None, + discovery: discovery.state(), + published: false, + held: None, + }; + let Some(account) = account else { + return Ok(view); + }; + view.shared = account.is_shared(); + view.pending_confirmation = account.pending_confirmation(); + view.warnings = account.warnings(); + match &account.card { + Ok(card) => { + view.can_share = account.subscription.subscribed; + view.feed_url = Some(card.feed_url.to_string()); + view.id_feed_url = card.id_feed_url.as_ref().map(ToString::to_string); + } + Err(error) => view.card_error = Some(error.clone()), + } + if let Some(dht) = discovery.dht() + && account.publishable() + { + // The overlay's cache is not essential to the podcast page: when it + // cannot be read, the card shows as not published yet. + view.published = match dht.published().await { + Ok(publications) => publications + .iter() + .any(|publication| publication.active && publication.podcast_id == *podcast_id), + Err(error) => { + tracing::warn!("podcast publications unavailable: {error}"); + false + } + }; + if !view.published { + view.held = super::discovery::held_reason(podcast_id); + } + } + Ok(view) +} + +/// URLs the user confirmed in the privacy dialog. +#[derive(Debug, Clone, Deserialize)] +pub(crate) struct ShareConfirmRequest { + pub(crate) feed_url: String, + #[serde(default)] + pub(crate) id_feed_url: Option, +} + +#[derive(Debug, Clone, Deserialize)] +struct ShareRequest { + shared: bool, + /// Confirmation of the card's URLs despite its privacy warnings. + #[serde(default)] + confirm: Option, +} + +#[derive(Debug, Serialize)] +pub(crate) struct ShareResponse { + pub(crate) podcast: PodcastView, + pub(crate) share: ShareView, +} + +/// Result of [`set_share`]. +#[derive(Debug)] +pub(crate) enum ShareOutcome { + /// The decision is stored (or was already in place). + Done(Box), + /// The card's feed URLs look private: nothing was written; ask the user + /// and send the shown URLs back as `confirm` (HTTP 409). + NeedsConfirmation(ShareView), +} + +/// "Share in the federation": `{ "shared": bool, "confirm"?: {feed_url, +/// id_feed_url} }`. +/// +/// * Off: writes `shared = false` (no-op when not shared) and forgets the +/// confirmation; the reconciler revokes the card unless another account +/// still shares the podcast. +/// * On: needs an active subscription (409). When the card has privacy +/// warnings, `confirm` must name exactly the card's current URLs; +/// otherwise nothing is written and [`ShareOutcome::NeedsConfirmation`] +/// carries the warnings. A podcast already shared from another device +/// but pending this account's confirmation is confirmed the same way. +pub(crate) async fn set_share( + store: &PodcastStore, + user_id: i64, + podcast_id: &PodcastId, + body: Value, + discovery: Discovery<'_>, + now_ms: i64, +) -> ApiResult { + let request: ShareRequest = decode(body)?; + if store.podcast(user_id, podcast_id).await?.is_none() { + return Err(ApiError::not_found("unknown podcast")); + } + if request.shared { + let Some(account) = store.account_share(user_id, podcast_id).await? else { + return Err( + anyhow::Error::from(PodcastStoreError::NotSubscribed(podcast_id.clone())).into(), + ); + }; + if !account.subscription.subscribed { + return Err( + anyhow::Error::from(PodcastStoreError::NotSubscribed(podcast_id.clone())).into(), + ); + } + if let Err(error) = &account.card { + return Err(ApiError::new( + 422, + format!("this podcast cannot be published: {error}"), + )); + } + if account.warnings().is_empty() { + store.set_shared(user_id, podcast_id, true).await?; + } else { + let confirmed = match &request.confirm { + Some(confirm) => { + let parse = |url: &str| { + FeedUrl::parse(url) + .map_err(|_| ApiError::bad_request("invalid confirmed feed URL")) + }; + let feed_url = parse(&confirm.feed_url)?; + let id_feed_url = confirm.id_feed_url.as_deref().map(parse).transpose()?; + store + .share_confirmed( + user_id, + podcast_id, + &feed_url, + id_feed_url.as_ref(), + now_ms, + ) + .await? + .is_some() + } + None => false, + }; + if !confirmed { + let view = share_view(store, user_id, podcast_id, discovery).await?; + return Ok(ShareOutcome::NeedsConfirmation(view)); + } + } + } else { + store.set_shared(user_id, podcast_id, false).await?; + store.clear_share_confirmation(user_id, podcast_id).await?; + } + Ok(ShareOutcome::Done(Box::new(ShareResponse { + podcast: podcast_view(store, user_id, podcast_id).await?, + share: share_view(store, user_id, podcast_id, discovery).await?, + }))) +} + +// --------------------------------------------------------------------------- +// Podcasts found in the federation +// --------------------------------------------------------------------------- + +/// A podcast from a public search hit: its id and the origin feed URL +/// (`PodcastSearchHit::feed_url`). +#[derive(Debug, Clone, Deserialize)] +struct FoundPodcastRequest { + podcast_id: String, + feed_url: String, +} + +/// The season a preview lists: `{"season": "2"}`, `"none"` (episodes +/// without a season) or absent / `"all"`. +#[derive(Debug, Clone, Default, Deserialize)] +struct PreviewSeasonRequest { + #[serde(default)] + season: Option, +} + +fn preview_season(body: &Value) -> ApiResult { + let request: PreviewSeasonRequest = decode(body.clone())?; + Ok( + match request + .season + .as_deref() + .map(str::trim) + .filter(|v| !v.is_empty()) + { + None | Some("all") => SeasonFilter::Any, + Some("none") => SeasonFilter::NoSeason, + Some(value) => SeasonFilter::Season( + value + .parse() + .map_err(|_| ApiError::bad_request("invalid season"))?, + ), + }, + ) +} + +/// The found podcast of a request. The id must hash from the URL, like a +/// valid card's origin URL: a claimed moved URL or a URL of another podcast +/// is refused (422), so the subscription gets the card's id. +fn found_podcast(body: Value) -> ApiResult<(PodcastId, FeedUrl)> { + let request: FoundPodcastRequest = decode(body)?; + let podcast_id = parse_podcast_id(&request.podcast_id)?; + let feed_url = FeedUrl::parse(&request.feed_url) + .map_err(|error| ApiError::bad_request(format!("invalid feed URL: {error}")))?; + if PodcastId::from_feed_url(&feed_url) != podcast_id { + return Err(ApiError::new( + 422, + "the feed URL is not the URL this podcast id comes from", + )); + } + Ok((podcast_id, feed_url)) +} + +/// Seasons and episodes of a found podcast before subscribing: `{ +/// "podcast_id", "feed_url" }` from the search hit, optionally `"season"` +/// to list the episodes of one season (`"none"`: without a season). The +/// feed is read through the SSRF-guarded fetcher and nothing is stored. +/// `subscribed` +/// also holds when the account follows the feed under the address it +/// moved to (a permanent redirect or the feed's `itunes:new-feed-url`). +pub(crate) async fn preview_found( + store: &PodcastStore, + fetcher: &FeedFetcher, + user_id: i64, + body: Value, +) -> ApiResult { + let season = preview_season(&body)?; + let (podcast_id, feed_url) = found_podcast(body)?; + let document = runner::fetch_to_subscribe(fetcher, &feed_url).await?; + let urls = runner::document_urls(&feed_url, &document); + let known = search::known_podcast(store, user_id, &podcast_id, &urls).await?; + Ok(search::preview( + &podcast_id, + &feed_url, + document.moved_permanently, + &document.feed, + known.as_ref(), + season, + )) +} + +/// "Subscribe" on a found podcast: `{ "podcast_id", "feed_url" }` from the +/// search hit. +/// +/// The subscription is the account's own and private, made like "add RSS" +/// with the origin URL (fetch first, then `runner::subscribe_fetched`): its +/// id is the card's id, or the account's existing podcast when the URL, or +/// the address the feed moved to, is a known feed URL or alias (progress +/// and history continue). Redirects and `itunes:new-feed-url` of the feed +/// itself become aliases as usual. When the account is already subscribed +/// (by id or a known URL, before or after the fetch) nothing is written. +pub(crate) async fn subscribe_found( + store: &PodcastStore, + fetcher: &FeedFetcher, + user_id: i64, + body: Value, + now_ms: i64, +) -> ApiResult { + let (podcast_id, feed_url) = found_podcast(body)?; + let already = async |known: PodcastSummary| -> ApiResult { + Ok(SubscribeResponse { + podcast: podcast_view(store, user_id, &known.podcast_id).await?, + already_subscribed: true, + new_episodes: 0, + }) + }; + if let Some(known) = + search::known_podcast(store, user_id, &podcast_id, std::slice::from_ref(&feed_url)).await? + && known.subscribed + { + return already(known).await; + } + let document = runner::fetch_to_subscribe(fetcher, &feed_url).await?; + let urls = runner::document_urls(&feed_url, &document); + if let Some(known) = search::known_podcast(store, user_id, &podcast_id, &urls).await? + && known.subscribed + { + return already(known).await; + } + let subscription = + runner::subscribe_fetched(store, user_id, &feed_url, &document, now_ms).await?; + Ok(SubscribeResponse { + podcast: podcast_view(store, user_id, &subscription.podcast_id).await?, + already_subscribed: subscription.already_subscribed, + new_episodes: subscription.refresh.new_episodes, + }) +} + +// --------------------------------------------------------------------------- +// Episodes +// --------------------------------------------------------------------------- + +/// Query string of the episode list. +#[derive(Debug, Clone, Default, Deserialize)] +pub(crate) struct EpisodeListParams { + /// Season number, or `none` for the episodes without a season. + #[serde(default)] + pub(crate) season: Option, + /// `unplayed`, `not_started`, `in_progress` or `played`. + #[serde(default)] + pub(crate) played: Option, + /// Search text (titles and descriptions). + #[serde(default)] + pub(crate) q: Option, + /// `newest` (default), `oldest`, `number` or `number_desc`. + #[serde(default)] + pub(crate) sort: Option, + #[serde(default)] + pub(crate) limit: Option, + #[serde(default)] + pub(crate) offset: Option, +} + +impl EpisodeListParams { + /// The store query (one row more than the page, to tell whether more + /// rows follow). + pub(crate) fn query(&self, podcast_id: PodcastId) -> ApiResult<(EpisodeQuery, usize)> { + let blank = |value: &Option| { + value + .as_deref() + .map(str::trim) + .filter(|value| !value.is_empty()) + .map(str::to_string) + }; + let season = match blank(&self.season).as_deref() { + None | Some("all") => SeasonFilter::Any, + Some("none") => SeasonFilter::NoSeason, + Some(value) => SeasonFilter::Season( + value + .parse() + .map_err(|_| ApiError::bad_request("invalid season"))?, + ), + }; + let played = match blank(&self.played).as_deref() { + None | Some("all") => PlayedFilter::Any, + Some("unplayed") => PlayedFilter::Unplayed, + Some("not_started") => PlayedFilter::NotStarted, + Some("in_progress") => PlayedFilter::InProgress, + Some("played") => PlayedFilter::Played, + Some(_) => return Err(ApiError::bad_request("invalid played filter")), + }; + let sort = match blank(&self.sort).as_deref() { + None | Some("newest") => EpisodeSort::NewestFirst, + Some("oldest") => EpisodeSort::OldestFirst, + Some("number") => EpisodeSort::EpisodeNumber, + Some("number_desc") => EpisodeSort::EpisodeNumberDesc, + Some(_) => return Err(ApiError::bad_request("invalid sort")), + }; + let limit = self + .limit + .unwrap_or(DEFAULT_EPISODE_PAGE) + .clamp(1, MAX_EPISODE_PAGE); + let mut query = EpisodeQuery::new(podcast_id); + query.season = season; + query.played = played; + query.search = blank(&self.q); + query.sort = sort; + query.limit = Some(limit + 1); + query.offset = self.offset.unwrap_or(0); + Ok((query, limit)) + } +} + +#[derive(Debug, Serialize)] +pub(crate) struct EpisodePage { + pub(crate) episodes: Vec, + pub(crate) offset: usize, + pub(crate) has_more: bool, +} + +/// A page of episodes of one podcast. +pub(crate) async fn episodes( + store: &PodcastStore, + user_id: i64, + podcast_id: &PodcastId, + params: &EpisodeListParams, +) -> ApiResult { + let podcast = store + .podcast(user_id, podcast_id) + .await? + .ok_or_else(|| ApiError::not_found("unknown podcast"))?; + let (query, limit) = params.query(podcast_id.clone())?; + let mut rows = store.episodes(user_id, &query).await?; + let has_more = rows.len() > limit; + rows.truncate(limit); + Ok(EpisodePage { + episodes: rows + .into_iter() + .map(|item| EpisodeView::new(item, &podcast.meta)) + .collect(), + offset: query.offset, + has_more, + }) +} + +/// Explicit progress actions of the episode list. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum EpisodeAction { + /// "Start over": position 0, in progress. + Restart, + /// "Played": completed, no listening session, no invented time. + MarkPlayed, + /// "Unplayed": not started at 0. + MarkUnplayed, +} + +/// Runs an explicit progress action and returns the updated episode. +pub(crate) async fn episode_action( + store: &PodcastStore, + user_id: i64, + episode_id: &EpisodeId, + action: EpisodeAction, +) -> ApiResult { + let reference = catalog_ref(store, user_id, episode_id).await?; + match action { + EpisodeAction::Restart => { + let session_id = format!("web-{}", uuid::Uuid::new_v4().simple()); + store.restart(user_id, &reference, &session_id).await?; + } + EpisodeAction::MarkPlayed => { + store.mark_played(user_id, &reference).await?; + } + EpisodeAction::MarkUnplayed => { + store.mark_unplayed(user_id, &reference).await?; + } + } + episode_view(store, user_id, episode_id).await +} + +// --------------------------------------------------------------------------- +// Playback +// --------------------------------------------------------------------------- + +/// Body of `POST /episodes/{id}/playback`. +#[derive(Debug, Clone, Default, Deserialize)] +pub(crate) struct PlaybackRequest { + /// The queue item's reference. Used only when the catalog does not know + /// the episode yet (an item received through a handoff); it is recorded + /// like a reference learned through sync. + #[serde(default)] + pub(crate) episode: Option, + /// Start a listening session now (the browser is about to play). + #[serde(default)] + pub(crate) start: bool, + /// Session id generated by the browser (required with `start`). + #[serde(default)] + pub(crate) session_id: Option, + /// Position to start at: a handoff or the user's choice. Without it the + /// episode resumes from its progress register. + #[serde(default)] + pub(crate) position_ms: Option, + /// `PodcastEpisodeRef::progress_generation` of a handed-off queue item. + #[serde(default)] + pub(crate) carried_generation: Option, +} + +/// How the browser plays an episode. +#[derive(Debug, Clone, Serialize)] +pub(crate) struct PlaybackResponse { + pub(crate) episode_id: String, + pub(crate) podcast_id: String, + /// Session started by this request (`start`), with the generation its + /// checkpoints use. + pub(crate) session_id: Option, + pub(crate) generation: Option, + /// Where to start. + pub(crate) position_ms: i64, + /// Register status before this request. + pub(crate) status: &'static str, + pub(crate) duration_ms: Option, + /// Publisher URL (default: the browser streams it directly). + pub(crate) direct_url: String, + /// Same-origin proxy path when the administrator enabled the proxy. + pub(crate) proxy_url: Option, + pub(crate) mime: Option, + /// The progress register after this request: its generation and the + /// session (or action) that wrote it. Before continuing a paused + /// session the player checks that the register is still its own. + pub(crate) register_generation: Option, + pub(crate) register_session_id: Option, +} + +/// Prepares an episode for the player and, with `start`, begins a session. +pub(crate) async fn playback( + store: &PodcastStore, + user_id: i64, + episode_id: &EpisodeId, + body: Value, + proxy_enabled: bool, +) -> ApiResult { + let request: PlaybackRequest = decode(body)?; + if store.episode_ref(user_id, episode_id).await?.is_none() { + match request.episode.as_ref() { + Some(reference) if reference.episode_id == *episode_id => { + store.ensure_episode(user_id, reference).await?; + } + Some(_) => return Err(ApiError::bad_request("episode reference mismatch")), + None => return Err(ApiError::not_found("unknown episode")), + } + } + let reference = catalog_ref(store, user_id, episode_id).await?; + let current = store.progress(user_id, episode_id).await?; + let status = current + .as_ref() + .map_or(ProgressStatus::NotStarted, |record| record.status); + let resume = current + .as_ref() + .map_or(0, ProgressRecord::resume_position_ms); + let mut position_ms = request.position_ms.unwrap_or(resume).max(0); + let (mut session_id, mut generation) = (None, None); + let mut register = current; + if request.start { + let id = request + .session_id + .as_deref() + .map(str::trim) + .filter(|id| !id.is_empty()) + .ok_or_else(|| ApiError::bad_request("session_id is required to start"))?; + let write = store + .start_session( + user_id, + &reference, + id, + position_ms, + request.carried_generation, + ) + .await?; + position_ms = write.record.position_ms; + session_id = Some(write.session.session_id); + generation = Some(write.session.generation); + register = Some(write.record); + } + let (direct_url, mime) = store + .audio_source(user_id, episode_id) + .await? + .ok_or_else(|| ApiError::not_found("unknown episode"))?; + Ok(PlaybackResponse { + episode_id: episode_id.to_string(), + podcast_id: reference.podcast_id.to_string(), + session_id, + generation, + position_ms, + status: status_name(status), + duration_ms: reference.duration_ms, + direct_url, + proxy_url: proxy_enabled.then(|| super::http::proxy_path(episode_id)), + mime, + register_generation: register.as_ref().map(|record| record.generation), + register_session_id: register.map(|record| record.session_id), + }) +} + +/// What the player reports in `POST /episodes/{id}/progress`. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Deserialize)] +#[serde(rename_all = "snake_case")] +pub(crate) enum ProgressEvent { + /// Periodic save while playing (every 15–30 s). + Checkpoint, + /// The user paused. + Pause, + /// Playback stopped: another item, the page closed, the queue cleared. + Stop, + /// Playback moves to another device. + Handoff, + /// The user moved the position (explicit action). + Seek, + /// The audio reached the end of the stream. + Ended, +} + +impl ProgressEvent { + /// Whether the event ends a stretch of listening that belongs in the + /// session record. + fn records_session(self) -> bool { + matches!(self, Self::Pause | Self::Stop | Self::Handoff | Self::Ended) + } +} + +/// Body of `POST /episodes/{id}/progress`. +#[derive(Debug, Clone, Deserialize)] +pub(crate) struct ProgressRequest { + pub(crate) session_id: String, + /// Generation of the session's latest explicit action. + pub(crate) generation: u64, + pub(crate) event: ProgressEvent, + pub(crate) position_ms: i64, + /// Duration of the loaded audio, when the browser knows it. + #[serde(default)] + pub(crate) duration_ms: Option, + /// Time actually listened in this session (seeks excluded). + #[serde(default)] + pub(crate) listened_ms: Option, + /// When the session started (browser clock, unix ms). + #[serde(default)] + pub(crate) started_at_ms: Option, + #[serde(default)] + pub(crate) start_position_ms: Option, +} + +/// Result of a progress report. +#[derive(Debug, Clone, Serialize)] +pub(crate) struct ProgressResponse { + /// `written`, `superseded` (another device or action took over: stop + /// writing at this generation) or `already_completed`. + pub(crate) outcome: &'static str, + /// Generation to use for the next checkpoints of this session. + pub(crate) generation: u64, + /// Register after the report. + pub(crate) status: &'static str, + pub(crate) position_ms: i64, + /// Whether this report completed the episode. + pub(crate) completed: bool, + /// Whether the listening session was recorded (or grew). + pub(crate) session_recorded: bool, +} + +/// Stores a progress report of the player's current session. +pub(crate) async fn progress( + store: &PodcastStore, + user_id: i64, + episode_id: &EpisodeId, + body: Value, + now_ms: i64, +) -> ApiResult { + let request: ProgressRequest = decode(body)?; + let reference = catalog_ref(store, user_id, episode_id).await?; + let position_ms = request.position_ms.max(0); + let duration_ms = request + .duration_ms + .filter(|duration| *duration > 0) + .or(reference.duration_ms); + let session = ProgressSession { + episode: reference.clone(), + session_id: request.session_id.trim().to_string(), + generation: request.generation, + }; + let reached_end = + request.event == ProgressEvent::Ended || reaches_completion(position_ms, duration_ms); + let outcome = match request.event { + ProgressEvent::Seek => { + CheckpointOutcome::Written(Box::new(store.seek(user_id, &session, position_ms).await?)) + } + _ if reached_end => { + store + .complete_if_current(user_id, &session, position_ms) + .await? + } + _ => store.checkpoint(user_id, &session, position_ms).await?, + }; + let (name, record, completed) = match &outcome { + CheckpointOutcome::Written(write) => ( + "written", + write.record.clone(), + write.record.status == ProgressStatus::Completed && reached_end, + ), + CheckpointOutcome::Superseded { current } => ("superseded", current.clone(), false), + CheckpointOutcome::AlreadyCompleted { current } => { + ("already_completed", current.clone(), false) + } + }; + let mut session_recorded = false; + if request.event.records_session() + && let Some(started_at_ms) = request.started_at_ms + { + let started_at_ms = started_at_ms.clamp(0, now_ms); + let listened_ms = request + .listened_ms + .unwrap_or(0) + .clamp(0, now_ms - started_at_ms); + let listen = ListenSession { + session_id: session.session_id.clone(), + episode_id: episode_id.clone(), + podcast_id: reference.podcast_id.clone(), + device_id: devices::local_device_id(store.pool(), user_id).await?, + started_at_ms, + ended_at_ms: now_ms, + start_position_ms: request.start_position_ms.unwrap_or(0).max(0), + end_position_ms: position_ms, + listened_ms, + completed: reached_end, + }; + session_recorded = store + .record_listen_session(user_id, &listen, Some(&reference)) + .await? + .is_some(); + } + Ok(ProgressResponse { + outcome: name, + generation: record.generation, + status: status_name(record.status), + position_ms: record.position_ms, + completed, + session_recorded, + }) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn podcast() -> PodcastId { + PodcastId::from_feed_url(&FeedUrl::parse("https://podcasts.example/feed.xml").unwrap()) + } + + #[test] + fn episode_list_params_map_to_store_queries() { + let (query, limit) = EpisodeListParams::default().query(podcast()).unwrap(); + assert_eq!(limit, DEFAULT_EPISODE_PAGE); + assert_eq!(query.limit, Some(DEFAULT_EPISODE_PAGE + 1)); + assert_eq!(query.season, SeasonFilter::Any); + assert_eq!(query.played, PlayedFilter::Any); + assert_eq!(query.sort, EpisodeSort::NewestFirst); + assert_eq!(query.search, None); + + let params = EpisodeListParams { + season: Some("3".into()), + played: Some("in_progress".into()), + q: Some(" interview ".into()), + sort: Some("number_desc".into()), + limit: Some(10_000), + offset: Some(40), + }; + let (query, limit) = params.query(podcast()).unwrap(); + assert_eq!(query.season, SeasonFilter::Season(3)); + assert_eq!(query.played, PlayedFilter::InProgress); + assert_eq!(query.search.as_deref(), Some("interview")); + assert_eq!(query.sort, EpisodeSort::EpisodeNumberDesc); + assert_eq!(limit, MAX_EPISODE_PAGE); + assert_eq!(query.offset, 40); + + let none = EpisodeListParams { + season: Some("none".into()), + played: Some("played".into()), + q: Some(" ".into()), + sort: Some("oldest".into()), + ..Default::default() + }; + let (query, _) = none.query(podcast()).unwrap(); + assert_eq!(query.season, SeasonFilter::NoSeason); + assert_eq!(query.played, PlayedFilter::Played); + assert_eq!(query.search, None, "blank search is no filter"); + assert_eq!(query.sort, EpisodeSort::OldestFirst); + + for bad in [ + EpisodeListParams { + season: Some("first".into()), + ..Default::default() + }, + EpisodeListParams { + played: Some("sometimes".into()), + ..Default::default() + }, + EpisodeListParams { + sort: Some("random".into()), + ..Default::default() + }, + ] { + assert_eq!(bad.query(podcast()).unwrap_err().status, 400); + } + } + + #[test] + fn ids_and_bodies_are_validated_before_the_store() { + assert_eq!(parse_podcast_id("nope").unwrap_err().status, 400); + assert_eq!(parse_episode_id("pod1:abc").unwrap_err().status, 400); + assert!(parse_podcast_id(podcast().as_str()).is_ok()); + let error = decode::(serde_json::json!({ + "session_id": "s", "generation": 1, "event": "rewind", "position_ms": 0 + })) + .unwrap_err(); + assert_eq!(error.status, 400); + let request: ProgressRequest = decode(serde_json::json!({ + "session_id": "s", "generation": 3, "event": "handoff", "position_ms": 1200 + })) + .unwrap(); + assert_eq!(request.event, ProgressEvent::Handoff); + assert!(request.event.records_session()); + assert!(!ProgressEvent::Checkpoint.records_session()); + assert!(!ProgressEvent::Seek.records_session()); + // A malformed episode reference in a playback request is a 400. + let error = decode::(serde_json::json!({ + "episode": { "episode_id": "not an id" } + })) + .unwrap_err(); + assert_eq!(error.status, 400); + let empty: PlaybackRequest = decode(serde_json::json!({})).unwrap(); + assert!(!empty.start && empty.episode.is_none() && empty.position_ms.is_none()); + } + + #[test] + fn store_errors_map_to_http_statuses() { + let unknown: ApiError = + anyhow::Error::from(PodcastStoreError::UnknownPodcast(podcast())).into(); + assert_eq!(unknown.status, 404); + let not_subscribed: ApiError = + anyhow::Error::from(PodcastStoreError::NotSubscribed(podcast())).into(); + assert_eq!(not_subscribed.status, 409); + let invalid: ApiError = super::super::invalid("bad").into(); + assert_eq!(invalid.status, 400); + let limit: ApiError = + anyhow::Error::from(PodcastStoreError::SubscriptionLimit(1000)).into(); + assert_eq!(limit.status, 422); + assert_eq!(limit.message, "subscription limit reached (1000 podcasts)"); + let references: ApiError = + anyhow::Error::from(PodcastStoreError::ReferenceLimit(10_000)).into(); + assert_eq!(references.status, 422); + let refused: ApiError = + anyhow::Error::from(SubscribeUrlError::Fetch(super::super::fetch::FetchError { + kind: FetchErrorKind::Refused, + message: "10.1.2.3 is a private address".into(), + retry_after: None, + })) + .into(); + assert_eq!(refused.status, 422); + assert!( + refused.message.contains("only public"), + "{}", + refused.message + ); + assert!( + !refused.message.contains("10.1.2.3"), + "no resolved addresses" + ); + let timeout: ApiError = + anyhow::Error::from(SubscribeUrlError::Fetch(super::super::fetch::FetchError { + kind: FetchErrorKind::Timeout, + message: "timed out".into(), + retry_after: None, + })) + .into(); + assert_eq!(timeout.status, 502); + let transport: ApiError = + anyhow::Error::from(SubscribeUrlError::Fetch(super::super::fetch::FetchError { + kind: FetchErrorKind::Transport, + message: "error sending request: tcp connect error: Connection refused".into(), + retry_after: None, + })) + .into(); + assert_eq!(transport.status, 502); + assert_eq!( + transport.message, + "the feed could not be loaded: could not connect to the server" + ); + let other: ApiError = anyhow::anyhow!("connection reset").into(); + assert_eq!(other.status, 500); + assert_eq!(other.message, "internal error", "internals are not shown"); + } +} diff --git a/src/podcasts/apply.rs b/src/podcasts/apply.rs new file mode 100644 index 0000000..18d28b1 --- /dev/null +++ b/src/podcasts/apply.rs @@ -0,0 +1,896 @@ +//! The single code path that applies podcast records to the store. +//! +//! Remote operations, local actions and snapshot sections all end here. +//! Records come from the `SyncOpWire` materializers of +//! `music_dht::device_sync` and are merged with the functions of +//! `music_dht::podcast`, so applying is idempotent and order-independent. + +use anyhow::Result; +use music_dht::device_sync::{SyncOpPayload, SyncOpWire, SyncSnapshot}; +use music_dht::podcast::{ + EpisodeId, EpisodeIdentityHints, EpisodeMeta, EpisodeType, FeedUrl, KnownEpisode, + ListenSession, PodcastEpisodeRef, PodcastError, PodcastId, ProgressRecord, ShareRecord, + SubscriptionRecord, merge_feed_aliases, merge_listen_session, +}; +use sqlx::{PgConnection, Row}; + +use super::{ + PodcastStore, PodcastStoreError, SESSION_COLUMNS, episode_id_col, episode_search_norm, + feed_url_col, generation_from_sql, generation_to_sql, invalid, now_ms, podcast_meta_col, + progress_columns, progress_from_row, session_from_row, status_to_sql, +}; + +/// Result of applying one operation. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) enum ApplyOutcome { + /// Stored state changed. + Changed, + /// The operation was already reflected (duplicate, or older than the + /// stored register). + Unchanged, + /// Not a podcast operation; nothing to do here. + Ignored, + /// The operation violates the contract and was skipped. Acknowledge it + /// anyway so the log keeps moving (`PODCASTS.md`). + Rejected(String), +} + +/// Counters returned by [`PodcastStore::apply_snapshot`]. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub(crate) struct SnapshotApplyStats { + /// Records that changed stored state. + pub(crate) changed: usize, + /// Valid records already reflected locally. + pub(crate) unchanged: usize, + /// Records that failed validation and were skipped. + pub(crate) rejected: usize, + /// Whether the podcast sections were ignored for exceeding the counts. + pub(crate) ignored_over_limits: bool, +} + +/// Where an episode row came from. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum EpisodeSource { + /// Parsed from the podcast's feed by this account. + Feed, + /// Learned from a `PodcastEpisodeRef` (sync operation or queue item). + Sync, +} + +impl EpisodeSource { + fn as_sql(self) -> &'static str { + match self { + Self::Feed => "feed", + Self::Sync => "sync", + } + } +} + +pub(crate) async fn load_subscription_register( + conn: &mut PgConnection, + user_id: i64, + podcast_id: &PodcastId, +) -> Result> { + let row = sqlx::query( + "SELECT subscribed, podcast_json, generation, hlc_ms, op_id + FROM furumusic__podcast_subscription WHERE user_id = $1 AND podcast_id = $2", + ) + .bind(user_id) + .bind(podcast_id.as_str()) + .fetch_optional(&mut *conn) + .await?; + let Some(row) = row else { + return Ok(None); + }; + let record = SubscriptionRecord { + podcast_id: podcast_id.clone(), + subscribed: row.try_get("subscribed")?, + podcast: podcast_meta_col(&row, "podcast_json")?, + feed_aliases: load_aliases(conn, user_id, podcast_id).await?, + generation: generation_from_sql(row.try_get("generation")?), + hlc_ms: row.try_get("hlc_ms")?, + op_id: row.try_get("op_id")?, + }; + Ok(Some(record)) +} + +pub(crate) async fn load_aliases( + conn: &mut PgConnection, + user_id: i64, + podcast_id: &PodcastId, +) -> Result> { + let rows = sqlx::query( + "SELECT feed_url FROM furumusic__podcast_feed_alias + WHERE user_id = $1 AND podcast_id = $2 ORDER BY feed_url COLLATE \"C\"", + ) + .bind(user_id) + .bind(podcast_id.as_str()) + .fetch_all(&mut *conn) + .await?; + let mut aliases = rows + .iter() + .map(|row| feed_url_col(row, "feed_url")) + .collect::>>()?; + // `merge_feed_aliases` returns byte order; the database collation may + // differ (ICU, glibc locales), and the comparisons below need equality. + aliases.sort(); + Ok(aliases) +} + +pub(crate) fn share_from_row(row: &sqlx::postgres::PgRow) -> sqlx::Result { + Ok(ShareRecord { + podcast_id: super::podcast_id_col(row, "podcast_id")?, + shared: row.try_get("shared")?, + generation: generation_from_sql(row.try_get("generation")?), + hlc_ms: row.try_get("hlc_ms")?, + op_id: row.try_get("op_id")?, + }) +} + +pub(crate) async fn load_share( + conn: &mut PgConnection, + user_id: i64, + podcast_id: &PodcastId, +) -> Result> { + let row = sqlx::query( + "SELECT podcast_id, shared, generation, hlc_ms, op_id + FROM furumusic__podcast_share WHERE user_id = $1 AND podcast_id = $2", + ) + .bind(user_id) + .bind(podcast_id.as_str()) + .fetch_optional(&mut *conn) + .await?; + Ok(row.as_ref().map(share_from_row).transpose()?) +} + +pub(crate) async fn load_progress( + conn: &mut PgConnection, + user_id: i64, + episode_id: &EpisodeId, +) -> Result> { + let row = sqlx::query(&format!( + "SELECT {} FROM furumusic__podcast_progress p + WHERE p.user_id = $1 AND p.episode_id = $2", + progress_columns("p", "") + )) + .bind(user_id) + .bind(episode_id.as_str()) + .fetch_optional(&mut *conn) + .await?; + Ok(row + .as_ref() + .map(|row| progress_from_row(row, "")) + .transpose()?) +} + +pub(crate) async fn load_session( + conn: &mut PgConnection, + user_id: i64, + session_id: &str, +) -> Result> { + let row = sqlx::query(&format!( + "SELECT {SESSION_COLUMNS} FROM furumusic__podcast_listen_session s + WHERE s.user_id = $1 AND s.session_id = $2" + )) + .bind(user_id) + .bind(session_id) + .fetch_optional(&mut *conn) + .await?; + Ok(row.as_ref().map(session_from_row).transpose()?) +} + +/// Replaces the alias set of a podcast with `merged` when it differs. +async fn store_aliases( + conn: &mut PgConnection, + user_id: i64, + podcast_id: &PodcastId, + merged: &[FeedUrl], +) -> Result { + let current = load_aliases(conn, user_id, podcast_id).await?; + if current == merged { + return Ok(false); + } + sqlx::query("DELETE FROM furumusic__podcast_feed_alias WHERE user_id = $1 AND podcast_id = $2") + .bind(user_id) + .bind(podcast_id.as_str()) + .execute(&mut *conn) + .await?; + let urls: Vec<&str> = merged.iter().map(FeedUrl::as_str).collect(); + sqlx::query( + "INSERT INTO furumusic__podcast_feed_alias (user_id, podcast_id, feed_url) + SELECT $1, $2, url FROM UNNEST($3::text[]) AS url", + ) + .bind(user_id) + .bind(podcast_id.as_str()) + .bind(&urls) + .execute(&mut *conn) + .await?; + Ok(true) +} + +/// Merges feed URLs into the bounded grow-only alias set of a podcast. +/// +/// Returns whether the stored set changed. +pub(crate) async fn merge_aliases( + conn: &mut PgConnection, + user_id: i64, + podcast_id: &PodcastId, + urls: &[FeedUrl], +) -> Result { + let current = load_aliases(conn, user_id, podcast_id).await?; + let merged = merge_feed_aliases(podcast_id, ¤t, urls); + store_aliases(conn, user_id, podcast_id, &merged).await +} + +/// Inserts the catalog row of a podcast if it does not exist yet. +pub(crate) async fn ensure_podcast_row( + conn: &mut PgConnection, + user_id: i64, + podcast_id: &PodcastId, + feed_url: &FeedUrl, + title: &str, + now: i64, +) -> Result<()> { + sqlx::query( + "INSERT INTO furumusic__podcast + (user_id, podcast_id, feed_url, title, meta_from_feed, created_at_ms, updated_at_ms) + VALUES ($1, $2, $3, $4, FALSE, $5, $5) + ON CONFLICT (user_id, podcast_id) DO NOTHING", + ) + .bind(user_id) + .bind(podcast_id.as_str()) + .bind(feed_url.as_str()) + .bind(title) + .bind(now) + .execute(&mut *conn) + .await?; + Ok(()) +} + +/// Applies a subscription register record (operation or snapshot). +/// +/// The register is last-writer-wins ([`SubscriptionRecord::supersedes`]); +/// the alias set is a bounded grow-only set merged independently, so the +/// stored aliases are the bounded union of every URL seen (what +/// `merge_subscription` computes). The catalog row is created from the +/// register metadata and follows the winning register until this account +/// has refreshed the feed itself. +pub(crate) async fn apply_subscription_record( + conn: &mut PgConnection, + user_id: i64, + record: &SubscriptionRecord, + now: i64, +) -> Result { + let generation = generation_to_sql(record.generation)?; + let podcast_id = &record.podcast_id; + let current = load_subscription_register(conn, user_id, podcast_id).await?; + let wins = current + .as_ref() + .is_none_or(|current| record.supersedes(current)); + let mut changed = false; + if wins { + sqlx::query( + "INSERT INTO furumusic__podcast_subscription + (user_id, podcast_id, subscribed, podcast_json, generation, hlc_ms, op_id) + VALUES ($1, $2, $3, $4, $5, $6, $7) + ON CONFLICT (user_id, podcast_id) DO UPDATE SET + subscribed = EXCLUDED.subscribed, podcast_json = EXCLUDED.podcast_json, + generation = EXCLUDED.generation, hlc_ms = EXCLUDED.hlc_ms, + op_id = EXCLUDED.op_id", + ) + .bind(user_id) + .bind(podcast_id.as_str()) + .bind(record.subscribed) + .bind(serde_json::to_value(&record.podcast)?) + .bind(generation) + .bind(record.hlc_ms) + .bind(&record.op_id) + .execute(&mut *conn) + .await?; + changed = true; + } + let mut urls = record.feed_aliases.clone(); + urls.push(record.podcast.feed_url.clone()); + changed |= merge_aliases(conn, user_id, podcast_id, &urls).await?; + + ensure_podcast_row( + conn, + user_id, + podcast_id, + &record.podcast.feed_url, + &record.podcast.title, + now, + ) + .await?; + if wins { + let meta = &record.podcast; + sqlx::query( + "UPDATE furumusic__podcast + SET feed_url = $3, title = $4, author = $5, + description = COALESCE($6, description), link = $7, image_url = $8, + updated_at_ms = $9 + WHERE user_id = $1 AND podcast_id = $2 AND NOT meta_from_feed", + ) + .bind(user_id) + .bind(podcast_id.as_str()) + .bind(meta.feed_url.as_str()) + .bind(&meta.title) + .bind(&meta.author) + .bind(&meta.description) + .bind(&meta.link) + .bind(&meta.image_url) + .bind(now) + .execute(&mut *conn) + .await?; + } + Ok(changed) +} + +/// Applies a share register record. +pub(crate) async fn apply_share_record( + conn: &mut PgConnection, + user_id: i64, + record: &ShareRecord, +) -> Result { + let generation = generation_to_sql(record.generation)?; + let current = load_share(conn, user_id, &record.podcast_id).await?; + if current + .as_ref() + .is_some_and(|current| !record.supersedes(current)) + { + return Ok(false); + } + sqlx::query( + "INSERT INTO furumusic__podcast_share + (user_id, podcast_id, shared, generation, hlc_ms, op_id) + VALUES ($1, $2, $3, $4, $5, $6) + ON CONFLICT (user_id, podcast_id) DO UPDATE SET + shared = EXCLUDED.shared, generation = EXCLUDED.generation, + hlc_ms = EXCLUDED.hlc_ms, op_id = EXCLUDED.op_id", + ) + .bind(user_id) + .bind(record.podcast_id.as_str()) + .bind(record.shared) + .bind(generation) + .bind(record.hlc_ms) + .bind(&record.op_id) + .execute(&mut *conn) + .await?; + Ok(true) +} + +/// Applies a progress register record. Never touches the catalog. +pub(crate) async fn apply_progress_record( + conn: &mut PgConnection, + user_id: i64, + record: &ProgressRecord, +) -> Result { + let generation = generation_to_sql(record.generation)?; + let current = load_progress(conn, user_id, &record.episode_id).await?; + if current + .as_ref() + .is_some_and(|current| !record.supersedes(current)) + { + return Ok(false); + } + sqlx::query( + "INSERT INTO furumusic__podcast_progress + (user_id, episode_id, podcast_id, position_ms, status, generation, session_id, + hlc_ms, device_id, op_id) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10) + ON CONFLICT (user_id, episode_id) DO UPDATE SET + podcast_id = EXCLUDED.podcast_id, position_ms = EXCLUDED.position_ms, + status = EXCLUDED.status, generation = EXCLUDED.generation, + session_id = EXCLUDED.session_id, hlc_ms = EXCLUDED.hlc_ms, + device_id = EXCLUDED.device_id, op_id = EXCLUDED.op_id", + ) + .bind(user_id) + .bind(record.episode_id.as_str()) + .bind(record.podcast_id.as_str()) + .bind(record.position_ms) + .bind(status_to_sql(record.status)) + .bind(generation) + .bind(&record.session_id) + .bind(record.hlc_ms) + .bind(&record.device_id) + .bind(&record.op_id) + .execute(&mut *conn) + .await?; + Ok(true) +} + +/// Adds or grows a listening session (grow-only set keyed by `session_id`). +pub(crate) async fn apply_listen_session( + conn: &mut PgConnection, + user_id: i64, + session: &ListenSession, +) -> Result { + let current = load_session(conn, user_id, &session.session_id).await?; + let merged = match ¤t { + Some(current) => merge_listen_session(current, session).map_err(invalid)?, + None => session.clone(), + }; + if current.as_ref() == Some(&merged) { + return Ok(false); + } + sqlx::query( + "INSERT INTO furumusic__podcast_listen_session + (user_id, session_id, episode_id, podcast_id, device_id, started_at_ms, + ended_at_ms, start_position_ms, end_position_ms, listened_ms, completed) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11) + ON CONFLICT (user_id, session_id) DO UPDATE SET + started_at_ms = EXCLUDED.started_at_ms, ended_at_ms = EXCLUDED.ended_at_ms, + start_position_ms = EXCLUDED.start_position_ms, + end_position_ms = EXCLUDED.end_position_ms, + listened_ms = EXCLUDED.listened_ms, completed = EXCLUDED.completed", + ) + .bind(user_id) + .bind(&merged.session_id) + .bind(merged.episode_id.as_str()) + .bind(merged.podcast_id.as_str()) + .bind(&merged.device_id) + .bind(merged.started_at_ms) + .bind(merged.ended_at_ms) + .bind(merged.start_position_ms) + .bind(merged.end_position_ms) + .bind(merged.listened_ms) + .bind(merged.completed) + .execute(&mut *conn) + .await?; + Ok(true) +} + +pub(crate) fn episode_type_to_sql(kind: EpisodeType) -> &'static str { + match kind { + EpisodeType::Full => "full", + EpisodeType::Trailer => "trailer", + EpisodeType::Bonus => "bonus", + } +} + +/// Inserts an episode row. +pub(crate) async fn insert_episode( + conn: &mut PgConnection, + user_id: i64, + meta: &EpisodeMeta, + source: EpisodeSource, + in_feed: bool, + now: i64, +) -> Result<()> { + sqlx::query( + "INSERT INTO furumusic__podcast_episode + (user_id, episode_id, podcast_id, guid, title, description, season, + episode_number, episode_type, published_at_ms, duration_ms, audio_url, mime, + byte_len, image_url, link, source, in_feed, search_norm, first_seen_at_ms, + updated_at_ms) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, + $18, $19, $20, $20)", + ) + .bind(user_id) + .bind(meta.episode_id.as_str()) + .bind(meta.podcast_id.as_str()) + .bind(&meta.guid) + .bind(&meta.title) + .bind(&meta.description) + .bind(meta.season.map(i64::from)) + .bind(meta.episode_number.map(i64::from)) + .bind(episode_type_to_sql(meta.episode_type)) + .bind(meta.published_at_ms) + .bind(meta.duration_ms) + .bind(&meta.audio_url) + .bind(&meta.mime) + .bind(meta.byte_len.and_then(|len| i64::try_from(len).ok())) + .bind(&meta.image_url) + .bind(&meta.link) + .bind(source.as_sql()) + .bind(in_feed) + .bind(episode_search_norm( + &meta.title, + meta.description.as_deref(), + )) + .bind(now) + .execute(&mut *conn) + .await?; + Ok(()) +} + +/// Episode metadata of a reference learned through sync. +pub(crate) fn meta_from_ref(episode: &PodcastEpisodeRef) -> EpisodeMeta { + EpisodeMeta { + episode_id: episode.episode_id.clone(), + podcast_id: episode.podcast_id.clone(), + guid: None, + title: episode.episode_title.clone(), + description: None, + season: episode.season, + episode_number: episode.episode_number, + episode_type: EpisodeType::default(), + published_at_ms: episode.published_at_ms, + duration_ms: episode.duration_ms, + audio_url: episode.audio_url.clone(), + mime: episode.mime.clone(), + byte_len: None, + image_url: episode.image_url.clone(), + link: None, + } +} + +/// Records an episode learned through sync when the catalog does not know it. +/// +/// Feed metadata always wins: an existing row is left untouched. The +/// podcast catalog row is created from the reference when missing (no +/// register is written). +pub(crate) async fn ensure_episode_from_ref( + conn: &mut PgConnection, + user_id: i64, + episode: &PodcastEpisodeRef, +) -> Result<()> { + let exists: bool = sqlx::query_scalar( + "SELECT EXISTS (SELECT 1 FROM furumusic__podcast_episode + WHERE user_id = $1 AND episode_id = $2)", + ) + .bind(user_id) + .bind(episode.episode_id.as_str()) + .fetch_one(&mut *conn) + .await?; + if exists { + return Ok(()); + } + let now = now_ms(); + ensure_podcast_row( + conn, + user_id, + &episode.podcast_id, + &episode.feed_url, + &episode.podcast_title, + now, + ) + .await?; + insert_episode( + conn, + user_id, + &meta_from_ref(episode), + EpisodeSource::Sync, + false, + now, + ) + .await +} + +/// Known episodes of a podcast with the identity hints they were last seen with. +pub(crate) async fn known_episodes( + conn: &mut PgConnection, + user_id: i64, + podcast_id: &PodcastId, +) -> Result> { + let rows = sqlx::query( + "SELECT episode_id, guid, audio_url, link, published_at_ms, title + FROM furumusic__podcast_episode + WHERE user_id = $1 AND podcast_id = $2 ORDER BY episode_id COLLATE \"C\"", + ) + .bind(user_id) + .bind(podcast_id.as_str()) + .fetch_all(&mut *conn) + .await?; + let known = rows + .iter() + .map(|row| { + Ok(KnownEpisode { + episode_id: episode_id_col(row, "episode_id")?, + hints: EpisodeIdentityHints { + guid: row.try_get("guid")?, + enclosure_url: Some(row.try_get("audio_url")?), + link: row.try_get("link")?, + published_at_ms: row.try_get("published_at_ms")?, + title: Some(row.try_get("title")?), + }, + }) + }) + .collect::>>()?; + Ok(known) +} + +/// Applies one podcast operation inside the caller's transaction (which +/// holds the account lock). +pub(crate) async fn apply_op( + conn: &mut PgConnection, + user_id: i64, + op: &SyncOpWire, +) -> Result { + if !op.payload.is_podcast() { + return Ok(ApplyOutcome::Ignored); + } + if let Err(error) = op.validate_podcast() { + return Ok(ApplyOutcome::Rejected(error.to_string())); + } + let result = match &op.payload { + SyncOpPayload::PodcastSubscriptionSet { .. } => { + let record = op.podcast_subscription().expect("subscription payload"); + apply_subscription_record(conn, user_id, &record, now_ms()).await + } + SyncOpPayload::PodcastShareSet { .. } => { + apply_share_record(conn, user_id, &op.podcast_share().expect("share payload")).await + } + SyncOpPayload::PodcastFeedAliasesAdded { .. } => { + let (podcast_id, urls) = op.podcast_feed_aliases().expect("alias payload"); + merge_aliases(conn, user_id, podcast_id, urls).await + } + SyncOpPayload::PodcastProgressSet { episode, .. } => { + let record = op.podcast_progress().expect("progress payload"); + if let Err(error) = generation_to_sql(record.generation) { + return Ok(ApplyOutcome::Rejected(error.to_string())); + } + if let Some(episode) = episode { + ensure_episode_from_ref(conn, user_id, episode).await?; + } + apply_progress_record(conn, user_id, &record).await + } + SyncOpPayload::PodcastListenRecorded { session, episode } => { + if let Some(current) = load_session(conn, user_id, &session.session_id).await? + && let Err(error) = merge_listen_session(¤t, session) + { + return Ok(ApplyOutcome::Rejected(error.to_string())); + } + if let Some(episode) = episode { + ensure_episode_from_ref(conn, user_id, episode).await?; + } + apply_listen_session(conn, user_id, session).await + } + _ => return Ok(ApplyOutcome::Ignored), + }; + match result { + Ok(true) => Ok(ApplyOutcome::Changed), + Ok(false) => Ok(ApplyOutcome::Unchanged), + Err(error) => match error.downcast_ref::() { + Some(PodcastStoreError::Invalid(reason)) => Ok(ApplyOutcome::Rejected(reason.clone())), + _ => Err(error), + }, + } +} + +/// Whether an applied operation can change what is published +/// (`super::discovery`): subscription and share registers, and the alias set +/// the card's origin URL comes from. +fn affects_publication(payload: &SyncOpPayload) -> bool { + matches!( + payload, + SyncOpPayload::PodcastSubscriptionSet { .. } + | SyncOpPayload::PodcastShareSet { .. } + | SyncOpPayload::PodcastFeedAliasesAdded { .. } + ) +} + +fn record_outcome(stats: &mut SnapshotApplyStats, result: Result) -> Result<()> { + match result { + Ok(true) => stats.changed += 1, + Ok(false) => stats.unchanged += 1, + Err(error) => match error.downcast_ref::() { + Some(PodcastStoreError::Invalid(_)) => stats.rejected += 1, + _ => return Err(error), + }, + } + Ok(()) +} + +fn check(result: Result<(), PodcastError>) -> Result<()> { + result.map_err(invalid) +} + +impl PodcastStore { + /// Applies a device-sync operation received from a trusted device. + /// + /// Non-podcast operations return [`ApplyOutcome::Ignored`]. Podcast + /// operations are validated (`SyncOpWire::validate_podcast`) and merged + /// with the contract functions in one transaction; applying the same + /// operations in any order, with duplicates, gives the same state. + pub(crate) async fn apply_remote_op( + &self, + user_id: i64, + op: &SyncOpWire, + ) -> Result { + if !op.payload.is_podcast() { + return Ok(ApplyOutcome::Ignored); + } + let mut tx = self.begin_write(user_id).await?; + let outcome = apply_op(&mut tx, user_id, op).await?; + tx.commit().await?; + if outcome == ApplyOutcome::Changed && affects_publication(&op.payload) { + super::discovery::request_reconcile(); + } + Ok(outcome) + } + + /// Merges the podcast sections of a sync snapshot. + /// + /// Sections exceeding the contract counts are ignored + /// (`SyncSnapshot::podcast_state_within_limits`); every record is + /// validated and merged with the same functions as operations, so a + /// snapshot never replaces newer local state. One transaction. + pub(crate) async fn apply_snapshot( + &self, + user_id: i64, + snapshot: &SyncSnapshot, + ) -> Result { + let mut stats = SnapshotApplyStats::default(); + if !snapshot.has_podcast_state() { + return Ok(stats); + } + if !snapshot.podcast_state_within_limits() { + stats.ignored_over_limits = true; + return Ok(stats); + } + let mut tx = self.begin_write(user_id).await?; + let now = now_ms(); + for record in &snapshot.podcast_subscriptions { + let mut record = record.clone(); + record.normalize(); + let result = match check(record.validate()) { + Ok(()) => apply_subscription_record(&mut tx, user_id, &record, now).await, + Err(error) => Err(error), + }; + record_outcome(&mut stats, result)?; + } + for record in &snapshot.podcast_shares { + let result = match check(record.validate()) { + Ok(()) => apply_share_record(&mut tx, user_id, record).await, + Err(error) => Err(error), + }; + record_outcome(&mut stats, result)?; + } + for record in &snapshot.podcast_progress { + let result = match check(record.validate()) { + Ok(()) => apply_progress_record(&mut tx, user_id, record).await, + Err(error) => Err(error), + }; + record_outcome(&mut stats, result)?; + } + for session in &snapshot.podcast_listen_sessions { + let result = match check(session.validate()) { + Ok(()) => apply_listen_session(&mut tx, user_id, session).await, + Err(error) => Err(error), + }; + record_outcome(&mut stats, result)?; + } + tx.commit().await?; + if stats.changed > 0 + && !(snapshot.podcast_subscriptions.is_empty() && snapshot.podcast_shares.is_empty()) + { + super::discovery::request_reconcile(); + } + Ok(stats) + } + + /// Fills the podcast sections of `snapshot` with this account's + /// registers and sessions, bounded by + /// `SyncSnapshot::bound_podcast_state_to(budget_bytes)`. + /// + /// Records are read newest first and capped by the per-section counts + /// before the byte budget is applied. Fetch state and the catalog are + /// never part of the snapshot. Send it only to v4 peers + /// (`SyncSnapshot::for_peer`). + pub(crate) async fn fill_snapshot( + &self, + user_id: i64, + snapshot: &mut SyncSnapshot, + budget_bytes: usize, + ) -> Result<()> { + use music_dht::podcast::{ + MAX_SNAPSHOT_LISTEN_SESSIONS, MAX_SNAPSHOT_PROGRESS, MAX_SNAPSHOT_SHARES, + MAX_SNAPSHOT_SUBSCRIPTIONS, + }; + // One consistent view of every section. + let mut tx = self.pool.begin().await?; + sqlx::query("SET TRANSACTION ISOLATION LEVEL REPEATABLE READ READ ONLY") + .execute(&mut *tx) + .await?; + let rows = sqlx::query( + "SELECT s.podcast_id, s.subscribed, s.podcast_json, s.generation, s.hlc_ms, s.op_id, + COALESCE((SELECT array_agg(a.feed_url ORDER BY a.feed_url COLLATE \"C\") + FROM furumusic__podcast_feed_alias a + WHERE a.user_id = s.user_id AND a.podcast_id = s.podcast_id), + '{}') AS aliases + FROM furumusic__podcast_subscription s + WHERE s.user_id = $1 + ORDER BY s.generation DESC, s.hlc_ms DESC, s.op_id COLLATE \"C\" DESC, + s.podcast_id COLLATE \"C\" DESC + LIMIT $2", + ) + .bind(user_id) + .bind(MAX_SNAPSHOT_SUBSCRIPTIONS as i64) + .fetch_all(&mut *tx) + .await?; + let subscriptions = rows + .iter() + .map(|row| { + let aliases: Vec = row.try_get("aliases")?; + Ok(SubscriptionRecord { + podcast_id: super::podcast_id_col(row, "podcast_id")?, + subscribed: row.try_get("subscribed")?, + podcast: podcast_meta_col(row, "podcast_json")?, + feed_aliases: aliases + .into_iter() + .map(FeedUrl::try_from) + .collect::>() + .map_err(|error| sqlx::Error::Decode(Box::new(error)))?, + generation: generation_from_sql(row.try_get("generation")?), + hlc_ms: row.try_get("hlc_ms")?, + op_id: row.try_get("op_id")?, + }) + }) + .collect::>>()?; + let shares = sqlx::query( + "SELECT podcast_id, shared, generation, hlc_ms, op_id FROM furumusic__podcast_share + WHERE user_id = $1 + ORDER BY generation DESC, hlc_ms DESC, op_id COLLATE \"C\" DESC, + podcast_id COLLATE \"C\" DESC + LIMIT $2", + ) + .bind(user_id) + .bind(MAX_SNAPSHOT_SHARES as i64) + .fetch_all(&mut *tx) + .await? + .iter() + .map(share_from_row) + .collect::>>()?; + let progress = sqlx::query(&format!( + "SELECT {} FROM furumusic__podcast_progress p + WHERE p.user_id = $1 + ORDER BY p.generation DESC, p.hlc_ms DESC, p.op_id COLLATE \"C\" DESC, + p.episode_id COLLATE \"C\" DESC + LIMIT $2", + progress_columns("p", "") + )) + .bind(user_id) + .bind(MAX_SNAPSHOT_PROGRESS as i64) + .fetch_all(&mut *tx) + .await? + .iter() + .map(|row| progress_from_row(row, "")) + .collect::>>()?; + let sessions = sqlx::query(&format!( + "SELECT {SESSION_COLUMNS} FROM furumusic__podcast_listen_session s + WHERE s.user_id = $1 + ORDER BY s.ended_at_ms DESC, s.session_id COLLATE \"C\" DESC + LIMIT $2" + )) + .bind(user_id) + .bind(MAX_SNAPSHOT_LISTEN_SESSIONS as i64) + .fetch_all(&mut *tx) + .await? + .iter() + .map(session_from_row) + .collect::>>()?; + tx.commit().await?; + snapshot.podcast_subscriptions = subscriptions; + snapshot.podcast_shares = shares; + snapshot.podcast_progress = progress; + snapshot.podcast_listen_sessions = sessions; + snapshot.bound_podcast_state_to(budget_bytes); + Ok(()) + } + + /// A snapshot holding only this account's podcast sections, bounded + /// with the default budget (`MAX_PODCAST_SNAPSHOT_BYTES`). + #[cfg(test)] + pub(crate) async fn export_snapshot(&self, user_id: i64) -> Result { + let mut snapshot = SyncSnapshot::default(); + self.fill_snapshot( + user_id, + &mut snapshot, + music_dht::podcast::MAX_PODCAST_SNAPSHOT_BYTES, + ) + .await?; + Ok(snapshot) + } + + /// Progress register of an episode (also for episodes the catalog does + /// not know yet). + pub(crate) async fn progress( + &self, + user_id: i64, + episode_id: &EpisodeId, + ) -> Result> { + let mut conn = self.pool.acquire().await?; + load_progress(&mut conn, user_id, episode_id).await + } +} diff --git a/src/podcasts/discovery.rs b/src/podcasts/discovery.rs new file mode 100644 index 0000000..b606553 --- /dev/null +++ b/src/podcasts/discovery.rs @@ -0,0 +1,1409 @@ +//! Public discovery of shared podcasts (`music_dht::podcast_dht`). +//! +//! The web server is one federation node that serves many accounts. Its +//! node identity (the persistent iroh key stored in PostgreSQL) is the only +//! key the server can sign with, so **the server node publishes**, never an +//! account: a card carries the server's key as `publisher` and nothing that +//! identifies the account (frid `PODCASTS.md`, "Card"). +//! +//! # Rule for several accounts +//! +//! * A podcast is published only when at least one active account +//! explicitly shares it: `effective_share` of that account's merged registers holds +//! (subscribed, shared, and the share decision orders after the latest +//! subscription write). Every other podcast of every account is private +//! and produces no network traffic at all. +//! * The server publishes **one card per podcast**, whatever the number of +//! accounts sharing it. Cards are keyed by `(podcast_id, publisher)`, so +//! two accounts can never produce conflicting records: the card is built +//! from one sharing account (the *representative*, see +//! [`aggregate_share_states`]) and stays published while any account +//! still shares it. +//! * Revoking (share off, unsubscribing, on any device of the account) +//! removes the account from the set. The card is tombstoned when no +//! account shares the podcast any more. This is the §3 rule "turning +//! sharing off revokes the user's own publication and does not remove +//! publications made by other users": the card of the server publisher +//! holds no per-account data, so while another account still shares the +//! podcast nothing published is attributable to the account that +//! stopped; the shared card is the other account's publication. +//! * A late or stale device state cannot resurrect a publication: only the +//! merged registers count, and an old `shared = true` loses to the newer +//! share decision (`ShareRecord` LWW). [`PodcastDht::sync_shares`] is +//! declarative: everything not in the computed set is unpublished. +//! +//! # Privacy confirmation +//! +//! `PodcastShareState::privacy_confirmed` is client-held state. The web +//! keeps it per account in `furumusic__podcast_share_confirmation` (never +//! synced), bound to the share decision (`share_op_id`) and to the exact +//! card URLs the user confirmed. A card with privacy warnings +//! (`PodcastCardSpec::privacy_warnings`: a token in the query, a member-feed +//! host, ...) is published only when its account confirmed those URLs for +//! its current share decision; otherwise the podcast is *pending +//! confirmation* for that account and the account does not count as +//! sharing it. A share decision taken on another device (for example a TUI +//! that asked its own user) therefore still asks the web user, and a feed +//! that moves to a URL that looks private asks again. +//! +//! # Reconciler +//! +//! [`run`] recomputes the set and calls [`PodcastDht::sync_shares`] after +//! every change that can affect it ([`request_reconcile`]: share toggles, +//! subscribe and unsubscribe, received sync operations and snapshots, feed +//! refreshes of shared podcasts) and every [`RECONCILE_INTERVAL`]. +//! +//! The first round of a federation run waits for the first device-sync +//! round (every trusted device polled answered, failed or timed out; at +//! most [`FIRST_DEVICE_SYNC_WAIT`]). The overlay withholds the cards it +//! published before the restart until the first `sync_shares`, which +//! re-sends the cards still shared. Without the wait that first call would +//! use the share decisions stored when the server stopped, so a podcast +//! whose sharing was turned off on a TUI meanwhile would be announced +//! again until the share-off arrived. A device that cannot be reached at +//! start still delivers its decision only when it syncs. A +//! failing database connection or query skips the round instead of +//! unpublishing everything (the overlay's own maintenance republishes the +//! cards in the meantime). Stored data of one `(account, podcast)` that +//! does not decode never skips the round: that pair counts as not shared +//! and every other podcast is reconciled ([`load_shared`]), so revocation +//! stays fail-safe for each podcast on its own. + +use std::collections::{BTreeMap, BTreeSet}; +use std::sync::{Arc, OnceLock}; +use std::time::Duration; + +use anyhow::Result; +use music_dht::podcast::{FeedUrl, PodcastId, PodcastMeta, ShareRecord, SubscriptionRecord}; +use music_dht::podcast_dht::{ + FeedUrlPrivacyWarning, MAX_PODCAST_RECORDS_PER_PUBLISHER, PodcastCard, PodcastCardSpec, + PodcastDht, PodcastDhtSyncStats, PodcastShareState, PublishedPodcast, +}; +use sqlx::{PgConnection, PgPool, Row}; +use tokio::sync::{Notify, watch}; + +use super::{apply, feed_url_col, podcast_id_col}; + +/// Periodic reconciliation, also after missed notifications. +pub(crate) const RECONCILE_INTERVAL: Duration = Duration::from_secs(5 * 60); + +/// Wait after a change notification so that a burst of changes (a sync +/// batch, a refresh run) becomes one round. +pub(crate) const RECONCILE_DEBOUNCE: Duration = Duration::from_secs(2); + +/// Longest wait of the first round for the first device-sync round of the +/// federation run (one exchange times out after 30 s). +pub(crate) const FIRST_DEVICE_SYNC_WAIT: Duration = Duration::from_secs(45); + +/// Podcasts the server node publishes at most. +/// +/// A replica keeps at most `MAX_PODCAST_RECORDS_PER_PUBLISHER` (256) records +/// of one publisher, tombstones included, and refuses a new +/// `(podcast_id, publisher)` key over it. The server is one publisher for +/// all of its accounts, so it keeps room for the tombstones of recently +/// revoked cards (they live 24 h). Podcasts beyond the cap stay unpublished +/// (see [`aggregate_share_states`]); a round also never asks for more keys +/// than replicas still accept ([`Incumbents::key_headroom`]). +pub(crate) const MAX_SERVER_SHARED_PODCASTS: usize = 192; + +fn notifier() -> &'static Notify { + static NOTIFY: OnceLock = OnceLock::new(); + NOTIFY.get_or_init(Notify::new) +} + +/// Asks the running reconciler (if any) for a round soon. Cheap and +/// idempotent; call it after committing a change. +pub(crate) fn request_reconcile() { + notifier().notify_one(); +} + +/// Whether the server's federation node runs public discovery, as shown +/// to the accounts. +#[derive(Clone, Copy)] +pub(crate) enum Discovery<'a> { + /// Federation is not running on this server. + Off, + /// The federation node runs, but the overlay could not start (its + /// cache could not be opened): see the federation status. + Unavailable, + /// The overlay runs. + Running(&'a PodcastDht), +} + +impl<'a> Discovery<'a> { + /// The overlay, when it runs. + pub(crate) fn dht(self) -> Option<&'a PodcastDht> { + match self { + Self::Running(dht) => Some(dht), + Self::Off | Self::Unavailable => None, + } + } + + /// `running`, `unavailable` or `off`. + pub(crate) fn state(self) -> &'static str { + match self { + Self::Running(_) => "running", + Self::Unavailable => "unavailable", + Self::Off => "off", + } + } +} + +/// A confirmation that an account may publish a card despite its privacy +/// warnings. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct ShareConfirmation { + /// Share decision the confirmation belongs to (`ShareRecord::op_id`). + pub(crate) share_op_id: String, + /// Confirmed `PodcastCardSpec::feed_url`. + pub(crate) feed_url: FeedUrl, + /// Confirmed `PodcastCardSpec::id_feed_url`. + pub(crate) id_feed_url: Option, +} + +/// One account's view of the publication of one podcast. +#[derive(Debug, Clone)] +pub(crate) struct AccountShare { + /// Account. + pub(crate) user_id: i64, + /// Merged subscription register with its alias set. + pub(crate) subscription: SubscriptionRecord, + /// Merged share register. + pub(crate) share: Option, + /// Card this account would publish, or why it cannot be built. + pub(crate) card: Result, + /// Stored privacy confirmation, if any (possibly stale). + pub(crate) confirmation: Option, + /// The catalog metadata comes from the server's own fetch of the feed, + /// not (yet) from a synced subscription register of a client. + pub(crate) meta_from_feed: bool, +} + +impl AccountShare { + /// Whether the account's registers say "publish" (`effective_share`). + pub(crate) fn is_shared(&self) -> bool { + music_dht::podcast::effective_share(Some(&self.subscription), self.share.as_ref()) + } + + /// Privacy warnings of the card (empty when the card cannot be built). + pub(crate) fn warnings(&self) -> Vec { + self.card + .as_ref() + .map(PodcastCardSpec::privacy_warnings) + .unwrap_or_default() + } + + /// Whether the stored confirmation covers the current share decision + /// and exactly the current card URLs. + pub(crate) fn privacy_confirmed(&self) -> bool { + let (Some(confirmation), Some(share), Ok(card)) = + (&self.confirmation, &self.share, &self.card) + else { + return false; + }; + confirmation.share_op_id == share.op_id + && confirmation.feed_url == card.feed_url + && confirmation.id_feed_url == card.id_feed_url + } + + /// Shared, but the card has privacy warnings the account has not + /// confirmed for this share decision. + pub(crate) fn pending_confirmation(&self) -> bool { + self.is_shared() + && self.card.is_ok() + && !self.warnings().is_empty() + && !self.privacy_confirmed() + } + + /// Whether the catalog text comes from the server's own fetch of the + /// podcast's origin URL (the URL its id is derived from): an account + /// can point its synced catalog at any URL before the server fetches + /// it, so `meta_from_feed` alone says nothing about where the text + /// comes from. + pub(crate) fn trusted_meta(&self) -> bool { + self.meta_from_feed + && self.card.as_ref().is_ok_and(|card| { + card.id_feed_url.is_none() + && PodcastId::from_feed_url(&card.feed_url) == self.subscription.podcast_id + }) + } + + /// Whether the account counts as sharing the podcast publicly. + pub(crate) fn publishable(&self) -> bool { + self.is_shared() + && self.card.is_ok() + && (self.warnings().is_empty() || self.privacy_confirmed()) + } +} + +/// The card an account would publish: the public projection of its +/// subscription (`PodcastCardSpec::from_subscription`), with the URL it +/// fetches and the channel metadata from its catalog, the origin URL as +/// `id_feed_url` when the feed moved, and the feed's episode count and +/// newest publication time. Never progress, history, device or account +/// data. +pub(crate) fn build_card( + subscription: &SubscriptionRecord, + catalog: Option<&PodcastMeta>, + episode_count: Option, + latest_episode_at_ms: Option, +) -> Result { + let mut card = PodcastCardSpec::from_subscription(subscription).map_err(|e| e.to_string())?; + let origin = card + .id_feed_url + .clone() + .unwrap_or_else(|| card.feed_url.clone()); + if let Some(meta) = catalog { + card.feed_url = meta.feed_url.clone(); + card.title = meta.title.clone(); + card.author = meta.author.clone(); + card.description = meta.description.clone(); + card.link = meta.link.clone(); + card.image_url = meta.image_url.clone(); + } + card.id_feed_url = (origin != card.feed_url).then_some(origin); + card.episode_count = episode_count; + card.latest_episode_at_ms = latest_episode_at_ms; + let card = card.bounded(); + card.validate().map_err(|e| e.to_string())?; + Ok(card) +} + +async fn load_confirmation( + conn: &mut PgConnection, + user_id: i64, + podcast_id: &PodcastId, +) -> Result> { + let row = sqlx::query( + "SELECT share_op_id, feed_url, id_feed_url + FROM furumusic__podcast_share_confirmation + WHERE user_id = $1 AND podcast_id = $2", + ) + .bind(user_id) + .bind(podcast_id.as_str()) + .fetch_optional(&mut *conn) + .await?; + let Some(row) = row else { + return Ok(None); + }; + let id_feed_url = match row.try_get::, _>("id_feed_url")? { + Some(url) => match FeedUrl::try_from(url) { + Ok(url) => Some(url), + // A row that no longer parses confirms nothing. + Err(_) => return Ok(None), + }, + None => None, + }; + let Ok(feed_url) = feed_url_col(&row, "feed_url") else { + return Ok(None); + }; + Ok(Some(ShareConfirmation { + share_op_id: row.try_get("share_op_id")?, + feed_url, + id_feed_url, + })) +} + +/// Loads one account's publication view of a podcast; `None` without a +/// subscription register. +pub(crate) async fn load_account_share( + conn: &mut PgConnection, + user_id: i64, + podcast_id: &PodcastId, +) -> Result> { + let Some(subscription) = apply::load_subscription_register(conn, user_id, podcast_id).await? + else { + return Ok(None); + }; + let share = apply::load_share(conn, user_id, podcast_id).await?; + let catalog = sqlx::query( + "SELECT feed_url, title, author, description, link, image_url, meta_from_feed + FROM furumusic__podcast WHERE user_id = $1 AND podcast_id = $2", + ) + .bind(user_id) + .bind(podcast_id.as_str()) + .fetch_optional(&mut *conn) + .await?; + let (catalog, meta_from_feed) = match catalog { + Some(row) => ( + Some(PodcastMeta { + feed_url: feed_url_col(&row, "feed_url")?, + title: row.try_get("title")?, + author: row.try_get("author")?, + description: row.try_get("description")?, + link: row.try_get("link")?, + image_url: row.try_get("image_url")?, + }), + row.try_get("meta_from_feed")?, + ), + None => (None, false), + }; + // Episodes of the current feed document only: references learned from + // the account's own queue or progress (`source = 'sync'`) and episodes + // that left the feed are not part of the public projection. + let stats = sqlx::query( + "SELECT COUNT(*) AS episodes, MAX(published_at_ms) AS latest + FROM furumusic__podcast_episode + WHERE user_id = $1 AND podcast_id = $2 AND source = 'feed' AND in_feed", + ) + .bind(user_id) + .bind(podcast_id.as_str()) + .fetch_one(&mut *conn) + .await?; + let episodes: i64 = stats.try_get("episodes")?; + let latest: Option = stats.try_get("latest")?; + let episode_count = (episodes > 0).then(|| u32::try_from(episodes).unwrap_or(u32::MAX)); + let card = build_card(&subscription, catalog.as_ref(), episode_count, latest); + let confirmation = load_confirmation(conn, user_id, podcast_id).await?; + Ok(Some(AccountShare { + user_id, + subscription, + share, + card, + confirmation, + meta_from_feed, + })) +} + +/// The shared `(account, podcast)` pairs of one round. +#[derive(Debug, Clone, Default)] +pub(crate) struct SharedLoad { + /// Readable pairs, for [`aggregate_share_states`]. + pub(crate) shares: Vec, + /// Pairs left out because a stored row of theirs does not decode (a + /// catalog or alias URL, the subscription register, a podcast id). + /// They count as not shared: fail closed for that pair only. + pub(crate) unreadable: usize, +} + +/// Whether an error while loading one `(account, podcast)` comes from its +/// stored data (a value that does not decode) rather than from the +/// database connection or the query itself. +fn is_row_data_error(error: &anyhow::Error) -> bool { + match error.downcast_ref::() { + Some( + sqlx::Error::Decode(_) + | sqlx::Error::ColumnDecode { .. } + | sqlx::Error::ColumnNotFound(_) + | sqlx::Error::ColumnIndexOutOfBounds { .. } + | sqlx::Error::TypeNotFound { .. }, + ) => true, + Some(_) => false, + // Decoding done in Rust (URLs, JSON registers, ids). + None => true, + } +} + +/// Every `(account, podcast)` whose registers say subscribed and shared, +/// for active accounts only (a deactivated account publishes nothing), +/// optionally restricted to `users` (tests run against a shared database). +/// `effective_share` is decided in Rust ([`AccountShare::is_shared`]). +/// +/// Revocation must stay fail-safe for every podcast on its own (frid +/// `PODCASTS.md`, "Publication"): a pair whose stored data does not decode +/// is left out and counted in [`SharedLoad::unreadable`] (so its card is +/// revoked unless another account shares the podcast), and the round goes +/// on for everyone else. Only a failing connection or query fails the +/// round. +pub(crate) async fn load_shared(pool: &PgPool, users: Option<&[i64]>) -> Result { + let mut conn = pool.acquire().await?; + let rows = sqlx::query( + "SELECT s.user_id, s.podcast_id + FROM furumusic__podcast_subscription s + JOIN furumusic__podcast_share sh + ON sh.user_id = s.user_id AND sh.podcast_id = s.podcast_id + JOIN furumusic__user u ON u.id = s.user_id AND u.is_active + WHERE s.subscribed AND sh.shared + AND ($1::BIGINT[] IS NULL OR s.user_id = ANY($1)) + ORDER BY s.podcast_id COLLATE \"C\", s.user_id", + ) + .bind(users) + .fetch_all(&mut *conn) + .await?; + let mut out = SharedLoad { + shares: Vec::with_capacity(rows.len()), + unreadable: 0, + }; + for row in rows { + let user_id: i64 = row.try_get("user_id")?; + let Ok(podcast_id) = podcast_id_col(&row, "podcast_id") else { + out.unreadable += 1; + continue; + }; + match load_account_share(&mut conn, user_id, &podcast_id).await { + Ok(Some(share)) => out.shares.push(share), + Ok(None) => {} + Err(error) if is_row_data_error(&error) => out.unreadable += 1, + Err(error) => return Err(error), + } + } + Ok(out) +} + +/// Result of [`aggregate_share_states`]. +#[derive(Debug, Clone, Default)] +pub(crate) struct Aggregation { + /// One state per published podcast, sorted by podcast id. + pub(crate) states: Vec, + /// Account that provides each published card. + pub(crate) representatives: BTreeMap, + /// Shared podcasts held back until their accounts confirm the privacy + /// warnings (no account confirmed). + pub(crate) pending_confirmation: usize, + /// Shared podcasts whose card cannot be built (no origin URL, invalid + /// metadata). + pub(crate) failed: usize, + /// Publishable podcasts left out by [`MAX_SERVER_SHARED_PODCASTS`] + /// (sorted). + pub(crate) over_limit: Vec, + /// Shared `(account, podcast)` pairs left out because their stored data + /// does not decode ([`SharedLoad::unreadable`]). + pub(crate) unreadable: usize, +} + +/// What the overlay publishes now: keeps the publication stable between +/// rounds and within what replicas accept. +#[derive(Debug, Clone, Default)] +pub(crate) struct Incumbents { + /// Podcasts with an active card of this node (`PodcastDht::published`). + pub(crate) published: BTreeSet, + /// Public text of every active card, so its representative is + /// recognized after a restart too ([`CardText`]). + pub(crate) cards: BTreeMap, + /// Podcasts of which replicas still hold a record of this node: active + /// cards and tombstones that have not expired. Each one occupies a + /// `(podcast_id, publisher)` key in the replicas; republishing it + /// reuses the key. + pub(crate) live_keys: BTreeSet, + /// Representative account of each card in the previous round (kept in + /// memory; empty after a restart, then [`Self::cards`] decides). + pub(crate) representatives: BTreeMap, +} + +impl Incumbents { + /// What the overlay publishes at `now_ms` (unix milliseconds). + pub(crate) fn from_published(publications: &[PublishedPodcast], now_ms: u64) -> Self { + let mut incumbents = Self::default(); + for publication in publications { + if publication.active { + incumbents.published.insert(publication.podcast_id.clone()); + if let Some(card) = publication.record.card() { + incumbents + .cards + .insert(publication.podcast_id.clone(), CardText::of_card(card)); + } + } + if publication.active || publication.record.expires_at_ms() > now_ms { + incumbents.live_keys.insert(publication.podcast_id.clone()); + } + } + incumbents + } + + /// Whether publishing `podcast_id` needs a key replicas do not hold + /// for this node yet. + fn needs_new_key(&self, podcast_id: &PodcastId) -> bool { + !self.published.contains(podcast_id) && !self.live_keys.contains(podcast_id) + } + + /// New `(podcast_id, publisher)` keys replicas still accept from this + /// node: `MAX_PODCAST_RECORDS_PER_PUBLISHER` minus the records they + /// keep (active cards and unexpired tombstones, including the + /// tombstones of cards displaced in this very round, which replace + /// their card in place). + pub(crate) fn key_headroom(&self) -> usize { + let live = self.live_keys.union(&self.published).count(); + MAX_PODCAST_RECORDS_PER_PUBLISHER.saturating_sub(live) + } +} + +/// The public text of a card: what a representative decides. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct CardText { + feed_url: FeedUrl, + id_feed_url: Option, + title: String, + author: Option, + description: Option, + image_url: Option, + link: Option, +} + +impl CardText { + fn of_spec(spec: &PodcastCardSpec) -> Self { + Self { + feed_url: spec.feed_url.clone(), + id_feed_url: spec.id_feed_url.clone(), + title: spec.title.clone(), + author: spec.author.clone(), + description: spec.description.clone(), + image_url: spec.image_url.clone(), + link: spec.link.clone(), + } + } + + fn of_card(card: &PodcastCard) -> Self { + Self { + feed_url: card.feed_url.clone(), + id_feed_url: card.id_feed_url.clone(), + title: card.title.clone(), + author: card.author.clone(), + description: card.description.clone(), + image_url: card.image_url.clone(), + link: card.link.clone(), + } + } +} + +/// Computes what the server node publishes from the accounts' views. +/// +/// * Only accounts for which [`AccountShare::publishable`] holds count. +/// * One state per podcast, built from one *representative* account. Its +/// registers and card are passed with `privacy_confirmed = true` (no +/// warnings, or the account confirmed exactly these URLs for its current +/// share decision), so `sync_shares` never holds anything back on its +/// own. The representative is, in this order: an account whose card has +/// no privacy warnings; the *incumbent* (the previous round's +/// representative, or after a restart an account whose card has the +/// exact text of the published card), so another account sharing the +/// same podcast never takes over a published card; an account whose +/// catalog the server fetched itself from the podcast's origin URL +/// ([`AccountShare::trusted_meta`]); the smallest account id. +/// * At most [`MAX_SERVER_SHARED_PODCASTS`] podcasts, shared fairly between +/// the accounts ([`allocate`]): an account whose podcasts fit in its fair +/// part of the budget never loses a published card because of other +/// accounts, whatever their share times say. A round never asks for more +/// new keys than replicas accept ([`Incumbents::key_headroom`]): a +/// podcast that would need one waits (`over_limit`) until tombstones +/// expire, and no published card is displaced for it. +pub(crate) fn aggregate_share_states( + shares: Vec, + incumbents: &Incumbents, +) -> Aggregation { + let mut aggregation = Aggregation::default(); + let mut by_podcast: BTreeMap> = BTreeMap::new(); + for share in shares { + if !share.is_shared() { + continue; + } + by_podcast + .entry(share.subscription.podcast_id.clone()) + .or_default() + .push(share); + } + let mut candidates: BTreeMap = BTreeMap::new(); + let mut queues: BTreeMap> = BTreeMap::new(); + for (podcast_id, accounts) in by_podcast { + let previous = incumbents.representatives.get(&podcast_id).copied(); + let published_text = incumbents.cards.get(&podcast_id); + let publishable: Vec<&AccountShare> = accounts + .iter() + .filter(|share| share.publishable()) + .collect(); + let representative = publishable.iter().min_by_key(|share| { + let same_text = published_text.is_some_and(|published| { + share + .card + .as_ref() + .is_ok_and(|card| CardText::of_spec(card) == *published) + }); + ( + !share.warnings().is_empty(), + previous != Some(share.user_id), + !same_text, + !share.trusted_meta(), + share.user_id, + ) + }); + match representative { + Some(representative) => { + let published = incumbents.published.contains(&podcast_id); + for share in &publishable { + queues + .entry(share.user_id) + .or_default() + .push(QueuedPodcast { + newcomer: !published, + shared_at_ms: share.share.as_ref().map_or(i64::MAX, |r| r.hlc_ms), + podcast_id: podcast_id.clone(), + }); + } + candidates.insert(podcast_id, (*representative).clone()); + } + None if accounts.iter().any(AccountShare::pending_confirmation) => { + aggregation.pending_confirmation += 1; + } + None => aggregation.failed += 1, + } + } + let selected = allocate(queues, incumbents, MAX_SERVER_SHARED_PODCASTS); + for (podcast_id, share) in candidates { + if !selected.contains(&podcast_id) { + aggregation.over_limit.push(podcast_id); + continue; + } + let Ok(card) = share.card else { + continue; + }; + aggregation + .representatives + .insert(podcast_id.clone(), share.user_id); + aggregation.states.push(PodcastShareState { + subscription: share.subscription, + share: share.share, + card: Some(card), + privacy_confirmed: true, + }); + } + aggregation +} + +/// One publishable podcast in an account's queue, in priority order. +#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord)] +struct QueuedPodcast { + /// Not published by this node yet (published cards come first). + newcomer: bool, + /// The account's own share decision time: it only orders the account's + /// own podcasts, never those of other accounts. + shared_at_ms: i64, + podcast_id: PodcastId, +} + +/// Picks at most `budget` podcasts from the accounts' queues, round-robin: +/// every round each account (by id) takes its next podcast not picked yet +/// (a podcast several accounts share counts once and is free for the +/// others). Inside a round already published podcasts go first, so the +/// last, partial round never replaces a published card by a new one. +/// +/// So with `n` sharing accounts every account keeps at least +/// `min(its podcasts, budget / n)` of them, its published ones first; only +/// podcasts of accounts beyond their fair part are left out (and revoked), +/// newest first. +/// +/// A podcast that needs a new replica key ([`Incumbents::needs_new_key`]) +/// is picked only while [`Incumbents::key_headroom`] lasts; otherwise it is +/// skipped (left out) and the account's next podcast is taken, so a +/// rebalance displaces at most as many published cards as replicas can +/// take new cards for. +fn allocate( + mut queues: BTreeMap>, + incumbents: &Incumbents, + budget: usize, +) -> BTreeSet { + for queue in queues.values_mut() { + // Popped from the end. + queue.sort_by(|left, right| right.cmp(left)); + } + let mut new_keys_left = incumbents.key_headroom(); + let mut selected = BTreeSet::new(); + while selected.len() < budget { + let mut picks: Vec<(bool, i64, PodcastId)> = Vec::new(); + for (user_id, queue) in &mut queues { + while let Some(next) = queue.pop() { + let podcast_id = next.podcast_id; + if selected.contains(&podcast_id) + || picks.iter().any(|(_, _, picked)| *picked == podcast_id) + { + continue; + } + if incumbents.needs_new_key(&podcast_id) { + if new_keys_left == 0 { + continue; + } + new_keys_left -= 1; + } + picks.push(( + !incumbents.published.contains(&podcast_id), + *user_id, + podcast_id, + )); + break; + } + } + if picks.is_empty() { + break; + } + picks.sort(); + for (_, _, podcast_id) in picks { + if selected.len() >= budget { + break; + } + selected.insert(podcast_id); + } + } + selected +} + +/// Outcome of one reconciliation round. +#[derive(Debug, Clone)] +pub(crate) struct ReconcileReport { + /// What was computed. + pub(crate) aggregation: Aggregation, + /// What the overlay did. + pub(crate) stats: PodcastDhtSyncStats, +} + +/// What the last round left over, kept in memory for the next round and +/// for the accounts' share status. +#[derive(Debug, Default)] +struct RoundMemory { + representatives: BTreeMap, + over_limit: BTreeSet, + failed: BTreeSet, +} + +fn memory() -> std::sync::MutexGuard<'static, RoundMemory> { + static MEMORY: OnceLock> = OnceLock::new(); + MEMORY + .get_or_init(Default::default) + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) +} + +/// Why the last round did not publish a podcast that is shared and +/// publishable: `over_limit` (beyond the account's part of +/// [`MAX_SERVER_SHARED_PODCASTS`]) or `failed` (the overlay could not sign +/// or store its card). `None` otherwise or before the first round. +pub(crate) fn held_reason(podcast_id: &PodcastId) -> Option<&'static str> { + let memory = memory(); + if memory.over_limit.contains(podcast_id) { + Some("over_limit") + } else if memory.failed.contains(podcast_id) { + Some("failed") + } else { + None + } +} + +/// One round: load the shared podcasts of every account (or of `users`), +/// aggregate them against what the overlay publishes now and let it +/// publish and revoke. +pub(crate) async fn reconcile_once( + pool: &PgPool, + dht: &PodcastDht, + users: Option<&[i64]>, +) -> Result { + let publications = dht + .published() + .await + .map_err(|error| anyhow::anyhow!("podcast publications unavailable: {error}"))?; + let load = load_shared(pool, users).await?; + let mut incumbents = Incumbents::from_published(&publications, unix_now_ms()); + incumbents.representatives = memory().representatives.clone(); + let mut aggregation = aggregate_share_states(load.shares, &incumbents); + aggregation.unreadable = load.unreadable; + let stats = dht + .sync_shares(aggregation.states.clone()) + .await + .map_err(|error| anyhow::anyhow!("podcast publication failed: {error}"))?; + { + let mut memory = memory(); + memory.representatives = aggregation.representatives.clone(); + memory.over_limit = aggregation.over_limit.iter().cloned().collect(); + memory.failed = stats + .failed + .iter() + .map(|failure| failure.podcast_id.clone()) + .collect(); + } + Ok(ReconcileReport { aggregation, stats }) +} + +fn unix_now_ms() -> u64 { + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map_or(0, |elapsed| { + u64::try_from(elapsed.as_millis()).unwrap_or(u64::MAX) + }) +} + +/// Runs the reconciler until the task is aborted (federation stop). +/// +/// The first round waits for `first_device_sync`, the first device-sync +/// round of this federation run (`federation::devices::sync_loop`), but at +/// most [`FIRST_DEVICE_SYNC_WAIT`]: see the module documentation. +pub(crate) async fn run( + pool: PgPool, + dht: Arc, + first_device_sync: watch::Receiver, +) { + run_rounds(pool, dht, first_device_sync, FIRST_DEVICE_SYNC_WAIT, None).await; +} + +/// [`run`] with the longest first wait and the accounts (all when `None`) +/// as parameters. +pub(crate) async fn run_rounds( + pool: PgPool, + dht: Arc, + mut first_device_sync: watch::Receiver, + first_wait: Duration, + users: Option>, +) { + // A closed channel (the device-sync loop ended) does not wait either. + if tokio::time::timeout(first_wait, first_device_sync.wait_for(|done| *done)) + .await + .is_err() + { + tracing::warn!( + seconds = first_wait.as_secs(), + "podcast publications: the first device-sync round did not end in time; \ + publishing the stored share decisions" + ); + } + let mut interval = tokio::time::interval(RECONCILE_INTERVAL); + interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip); + loop { + tokio::select! { + _ = interval.tick() => {} + _ = notifier().notified() => tokio::time::sleep(RECONCILE_DEBOUNCE).await, + } + // Only counts are logged: never feed URLs, titles or accounts. + match reconcile_once(&pool, &dht, users.as_deref()).await { + Ok(report) => { + let stats = &report.stats; + let aggregation = &report.aggregation; + if stats.published > 0 + || stats.unpublished > 0 + || !stats.failed.is_empty() + || !stats.held_back.is_empty() + || !aggregation.over_limit.is_empty() + || aggregation.unreadable > 0 + { + tracing::info!( + shared = aggregation.states.len(), + published = stats.published, + unchanged = stats.unchanged, + unpublished = stats.unpublished, + failed = stats.failed.len(), + held_back = stats.held_back.len(), + pending_confirmation = aggregation.pending_confirmation, + over_limit = aggregation.over_limit.len(), + unreadable = aggregation.unreadable, + "podcast publications reconciled" + ); + } + } + Err(error) => tracing::warn!("podcast publication round skipped: {error:#}"), + } + } +} + +// --------------------------------------------------------------------------- +// Confirmations (local, never synced) +// --------------------------------------------------------------------------- + +/// Records that the account confirmed publishing `card`'s URLs for the +/// share decision `share_op_id`. +pub(crate) async fn store_confirmation( + conn: &mut PgConnection, + user_id: i64, + podcast_id: &PodcastId, + confirmation: &ShareConfirmation, + now_ms: i64, +) -> Result<()> { + sqlx::query( + "INSERT INTO furumusic__podcast_share_confirmation + (user_id, podcast_id, share_op_id, feed_url, id_feed_url, confirmed_at_ms) + VALUES ($1, $2, $3, $4, $5, $6) + ON CONFLICT (user_id, podcast_id) DO UPDATE SET + share_op_id = EXCLUDED.share_op_id, feed_url = EXCLUDED.feed_url, + id_feed_url = EXCLUDED.id_feed_url, confirmed_at_ms = EXCLUDED.confirmed_at_ms", + ) + .bind(user_id) + .bind(podcast_id.as_str()) + .bind(&confirmation.share_op_id) + .bind(confirmation.feed_url.as_str()) + .bind(confirmation.id_feed_url.as_ref().map(FeedUrl::as_str)) + .bind(now_ms) + .execute(&mut *conn) + .await?; + Ok(()) +} + +/// Forgets the account's confirmation (sharing turned off, unsubscribed). +pub(crate) async fn clear_confirmation( + conn: &mut PgConnection, + user_id: i64, + podcast_id: &PodcastId, +) -> Result<()> { + sqlx::query( + "DELETE FROM furumusic__podcast_share_confirmation + WHERE user_id = $1 AND podcast_id = $2", + ) + .bind(user_id) + .bind(podcast_id.as_str()) + .execute(&mut *conn) + .await?; + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn url(path: &str) -> FeedUrl { + FeedUrl::parse(&format!("https://podcasts.example/{path}")).unwrap() + } + + fn subscription(path: &str, generation: u64) -> SubscriptionRecord { + let feed_url = url(path); + SubscriptionRecord { + podcast_id: PodcastId::from_feed_url(&feed_url), + subscribed: true, + podcast: PodcastMeta { + feed_url: feed_url.clone(), + title: format!("Show {path}"), + author: Some("Host".into()), + description: None, + link: None, + image_url: None, + }, + feed_aliases: vec![feed_url], + generation, + hlc_ms: 1_000, + op_id: format!("dev:{generation}"), + } + } + + fn account(user_id: i64, path: &str, shared: bool, share_hlc: i64) -> AccountShare { + let subscription = subscription(path, 1); + let share = ShareRecord { + podcast_id: subscription.podcast_id.clone(), + shared, + generation: 2, + hlc_ms: share_hlc, + op_id: format!("u{user_id}:share"), + }; + let card = build_card(&subscription, None, Some(3), Some(42)); + AccountShare { + user_id, + subscription, + share: Some(share), + card, + confirmation: None, + meta_from_feed: true, + } + } + + fn aggregate(shares: Vec) -> Aggregation { + aggregate_share_states(shares, &Incumbents::default()) + } + + fn id(path: &str) -> PodcastId { + PodcastId::from_feed_url(&url(path)) + } + + fn confirm(share: &mut AccountShare) { + let card = share.card.clone().unwrap(); + share.confirmation = Some(ShareConfirmation { + share_op_id: share.share.as_ref().unwrap().op_id.clone(), + feed_url: card.feed_url, + id_feed_url: card.id_feed_url, + }); + } + + #[test] + fn one_card_per_podcast_from_the_smallest_sharing_account() { + let aggregation = aggregate(vec![ + account(7, "a.xml", true, 10), + account(3, "a.xml", true, 20), + account(1, "a.xml", false, 30), + account(5, "b.xml", false, 10), + ]); + assert_eq!(aggregation.states.len(), 1); + let state = &aggregation.states[0]; + assert!(state.is_shared()); + assert!(state.privacy_confirmed); + assert_eq!( + aggregation.representatives[&state.subscription.podcast_id], + 3 + ); + let card = state.card.as_ref().unwrap(); + assert_eq!(card.episode_count, Some(3)); + assert_eq!(card.latest_episode_at_ms, Some(42)); + assert!(card.id_feed_url.is_none()); + } + + #[test] + fn a_share_older_than_the_subscription_does_not_count() { + let mut stale = account(1, "a.xml", true, 10); + // Subscribed again after the share decision: private again. + stale.subscription.generation = 3; + let aggregation = aggregate(vec![stale]); + assert!(aggregation.states.is_empty()); + assert_eq!(aggregation.failed, 0); + assert_eq!(aggregation.pending_confirmation, 0); + } + + #[test] + fn urls_that_look_private_need_the_accounts_own_confirmation() { + let secret = "feed.xml?token=8f3a9c2e7d1b4a6f"; + let unconfirmed = account(1, secret, true, 10); + assert!(!unconfirmed.warnings().is_empty()); + assert!(unconfirmed.pending_confirmation()); + let aggregation = aggregate(vec![unconfirmed.clone()]); + assert!(aggregation.states.is_empty()); + assert_eq!(aggregation.pending_confirmation, 1); + + let mut confirmed = account(2, secret, true, 20); + confirm(&mut confirmed); + assert!(confirmed.publishable()); + let aggregation = aggregate(vec![unconfirmed.clone(), confirmed.clone()]); + assert_eq!(aggregation.states.len(), 1); + assert_eq!( + aggregation + .representatives + .values() + .copied() + .collect::>(), + vec![2] + ); + + // A new share decision (another op) needs a new confirmation. + let mut redecided = confirmed.clone(); + redecided.share.as_mut().unwrap().op_id = "u2:share-again".into(); + assert!(redecided.pending_confirmation()); + // So does a feed that moved to other URLs. + let mut moved = confirmed; + moved.card.as_mut().unwrap().feed_url = url("other.xml?token=8f3a9c2e7d1b4a6f"); + assert!(!moved.privacy_confirmed()); + } + + #[test] + fn an_account_without_warnings_is_preferred_as_representative() { + let mut moved_secret = account(1, "a.xml", true, 10); + // Account 1 fetches the feed from a URL that looks private. + moved_secret.card = build_card( + &moved_secret.subscription, + Some(&PodcastMeta { + feed_url: url("a.xml?key=8f3a9c2e7d1b4a6f"), + ..moved_secret.subscription.podcast.clone() + }), + None, + None, + ); + confirm(&mut moved_secret); + let plain = account(9, "a.xml", true, 20); + let aggregation = aggregate(vec![moved_secret, plain]); + assert_eq!( + aggregation + .representatives + .values() + .copied() + .collect::>(), + vec![9] + ); + } + + #[test] + fn inside_an_account_the_longest_shared_podcasts_go_first() { + let shares = (0..MAX_SERVER_SHARED_PODCASTS + 3) + .map(|index| account(1, &format!("{index}.xml"), true, 1_000 - index as i64)) + .collect(); + let aggregation = aggregate(shares); + assert_eq!(aggregation.states.len(), MAX_SERVER_SHARED_PODCASTS); + // The three most recently shared (largest hlc) are left out. + let mut left_out: Vec = + (0..3).map(|index| id(&format!("{index}.xml"))).collect(); + left_out.sort(); + assert_eq!(aggregation.over_limit, left_out); + } + + #[test] + fn many_shares_of_another_account_never_unpublish_a_card() { + // A's card is published. B (a device with a lagging or forged + // clock: share times near 0) shares far more than the cap. + let mine = account(1, "mine.xml", true, 5_000); + let theirs: Vec = (0..300) + .map(|index| account(2, &format!("b{index}.xml"), true, index)) + .collect(); + let mut published: BTreeSet = (0..MAX_SERVER_SHARED_PODCASTS - 1) + .map(|index| id(&format!("b{index}.xml"))) + .collect(); + published.insert(id("mine.xml")); + let incumbents = Incumbents { + published: published.clone(), + ..Incumbents::default() + }; + let mut shares = theirs; + shares.push(mine); + let aggregation = aggregate_share_states(shares, &incumbents); + let states: BTreeSet = aggregation + .states + .iter() + .map(|state| state.subscription.podcast_id.clone()) + .collect(); + assert_eq!(states.len(), MAX_SERVER_SHARED_PODCASTS); + // Nothing published is displaced; only B's new podcasts wait. + assert_eq!(states, published); + assert_eq!( + aggregation.over_limit.len(), + 300 - (MAX_SERVER_SHARED_PODCASTS - 1) + ); + assert!(!aggregation.over_limit.contains(&id("mine.xml"))); + } + + #[test] + fn the_budget_is_shared_fairly_between_accounts() { + // B alone filled the budget; A shares five podcasts afterwards. + let mut shares: Vec = (0..MAX_SERVER_SHARED_PODCASTS) + .map(|index| account(2, &format!("b{index}.xml"), true, index as i64)) + .collect(); + let published: BTreeSet = shares + .iter() + .map(|share| share.subscription.podcast_id.clone()) + .collect(); + for index in 0..5 { + shares.push(account(1, &format!("a{index}.xml"), true, 10_000 + index)); + } + let incumbents = Incumbents { + published, + ..Incumbents::default() + }; + let aggregation = aggregate_share_states(shares, &incumbents); + assert_eq!(aggregation.states.len(), MAX_SERVER_SHARED_PODCASTS); + for index in 0..5 { + assert_eq!( + aggregation + .representatives + .get(&id(&format!("a{index}.xml"))), + Some(&1), + "A gets its fair part" + ); + } + // B, beyond its fair part, gives up its five most recently shared. + let mut left_out: Vec = (MAX_SERVER_SHARED_PODCASTS - 5 + ..MAX_SERVER_SHARED_PODCASTS) + .map(|index| id(&format!("b{index}.xml"))) + .collect(); + left_out.sort(); + assert_eq!(aggregation.over_limit, left_out); + + // A podcast both share counts once and is free for both. + let shared = vec![ + account(1, "both.xml", true, 1), + account(2, "both.xml", true, 1), + account(2, "b-only.xml", true, 2), + ]; + let selected = aggregate(shared); + assert_eq!(selected.states.len(), 2); + } + + /// Selected podcasts that need a new replica key, and live keys. + fn keys_asked(aggregation: &Aggregation, incumbents: &Incumbents) -> usize { + let new_keys = aggregation + .states + .iter() + .filter(|state| incumbents.needs_new_key(&state.subscription.podcast_id)) + .count(); + incumbents.live_keys.union(&incumbents.published).count() + new_keys + } + + #[test] + fn a_rebalance_never_asks_replicas_for_more_keys_than_they_accept() { + // B alone published the whole budget; A shares 96 new podcasts, so + // its fair part is 96, but replicas take only 256 - 192 = 64 new + // keys from this node (displaced cards keep theirs as tombstones). + let mut shares: Vec = (0..MAX_SERVER_SHARED_PODCASTS) + .map(|index| account(2, &format!("b{index}.xml"), true, index as i64)) + .collect(); + let published: BTreeSet = shares + .iter() + .map(|share| share.subscription.podcast_id.clone()) + .collect(); + for index in 0..96 { + shares.push(account(1, &format!("a{index}.xml"), true, 10_000 + index)); + } + let incumbents = Incumbents { + live_keys: published.clone(), + published, + ..Incumbents::default() + }; + assert_eq!( + incumbents.key_headroom(), + MAX_PODCAST_RECORDS_PER_PUBLISHER - MAX_SERVER_SHARED_PODCASTS + ); + let aggregation = aggregate_share_states(shares.clone(), &incumbents); + assert_eq!(aggregation.states.len(), MAX_SERVER_SHARED_PODCASTS); + assert!(keys_asked(&aggregation, &incumbents) <= MAX_PODCAST_RECORDS_PER_PUBLISHER); + let of = |user: i64| { + aggregation + .representatives + .values() + .filter(|representative| **representative == user) + .count() + }; + let headroom = MAX_PODCAST_RECORDS_PER_PUBLISHER - MAX_SERVER_SHARED_PODCASTS; + assert_eq!(of(1), headroom, "A gets as many as replicas accept"); + assert_eq!(of(2), MAX_SERVER_SHARED_PODCASTS - headroom); + // A's podcasts beyond the headroom wait; B's displaced ones too. + assert_eq!(aggregation.over_limit.len(), (96 - headroom) + headroom); + + // The next round (the displaced cards are tombstones now) asks for + // nothing new and displaces nothing more until they expire. + let mut next = Incumbents::default(); + for state in &aggregation.states { + next.published.insert(state.subscription.podcast_id.clone()); + } + next.live_keys = incumbents + .live_keys + .union(&next.published) + .cloned() + .collect(); + assert_eq!(next.key_headroom(), 0); + let again = aggregate_share_states(shares.clone(), &next); + let selected: BTreeSet = again + .states + .iter() + .map(|state| state.subscription.podcast_id.clone()) + .collect(); + assert_eq!(selected, next.published, "nothing moves without headroom"); + assert!(keys_asked(&again, &next) <= MAX_PODCAST_RECORDS_PER_PUBLISHER); + + // Once the tombstones expired, A gets the rest of its fair part. + next.live_keys = next.published.clone(); + let later = aggregate_share_states(shares, &next); + assert!(keys_asked(&later, &next) <= MAX_PODCAST_RECORDS_PER_PUBLISHER); + let of = |user: i64| { + later + .representatives + .values() + .filter(|representative| **representative == user) + .count() + }; + assert_eq!(of(1), 96); + assert_eq!(of(2), 96); + } + + #[test] + fn a_shared_tombstoned_podcast_reuses_its_key() { + // Every key is taken (192 cards, 64 tombstones): a podcast whose + // tombstone is still live can be published again, a new one waits. + let mut shares: Vec = (0..MAX_SERVER_SHARED_PODCASTS - 1) + .map(|index| account(2, &format!("b{index}.xml"), true, index as i64)) + .collect(); + let published: BTreeSet = shares + .iter() + .map(|share| share.subscription.podcast_id.clone()) + .collect(); + let mut live_keys = published.clone(); + for index in 0..65 { + live_keys.insert(id(&format!("gone{index}.xml"))); + } + shares.push(account(1, "gone0.xml", true, 1)); + shares.push(account(1, "fresh.xml", true, 0)); + let incumbents = Incumbents { + published, + live_keys, + ..Incumbents::default() + }; + assert_eq!(incumbents.key_headroom(), 0); + let aggregation = aggregate_share_states(shares, &incumbents); + assert_eq!(aggregation.representatives.get(&id("gone0.xml")), Some(&1)); + assert_eq!(aggregation.over_limit, vec![id("fresh.xml")]); + } + + #[test] + fn a_published_card_keeps_its_representative() { + // Account 9 provides the published card; account 3 starts sharing + // the same podcast with other text. + let provider = account(9, "a.xml", true, 10); + let mut newcomer = account(3, "a.xml", true, 20); + newcomer.card.as_mut().unwrap().title = "Rewritten".into(); + let published_card = CardText::of_spec(provider.card.as_ref().unwrap()); + let incumbents = Incumbents { + published: BTreeSet::from([id("a.xml")]), + representatives: BTreeMap::from([(id("a.xml"), 9)]), + ..Incumbents::default() + }; + let aggregation = + aggregate_share_states(vec![provider.clone(), newcomer.clone()], &incumbents); + assert_eq!(aggregation.representatives[&id("a.xml")], 9); + assert_eq!( + aggregation.states[0].card.as_ref().unwrap().title, + "Show a.xml" + ); + + // After a restart (no memory) the published card's text names it. + let restarted = Incumbents { + published: BTreeSet::from([id("a.xml")]), + cards: BTreeMap::from([(id("a.xml"), published_card)]), + ..Incumbents::default() + }; + let aggregation = + aggregate_share_states(vec![provider.clone(), newcomer.clone()], &restarted); + assert_eq!(aggregation.representatives[&id("a.xml")], 9); + + // The incumbent also outranks server-fetched metadata of another + // account: whoever shares first keeps the card's text. + let mut synced_provider = provider.clone(); + synced_provider.meta_from_feed = false; + let aggregation = + aggregate_share_states(vec![synced_provider, newcomer.clone()], &incumbents); + assert_eq!(aggregation.representatives[&id("a.xml")], 9); + + // Without an incumbent, metadata the server fetched from the origin + // URL wins over metadata a client synced, even for a smaller id. + newcomer.meta_from_feed = false; + let aggregation = aggregate(vec![provider, newcomer]); + assert_eq!(aggregation.representatives[&id("a.xml")], 9); + } + + #[test] + fn metadata_fetched_from_another_url_is_not_trusted() { + // Account 3 synced a catalog URL of its choice (the origin stays + // in its aliases); the server fetched that URL, so `meta_from_feed` + // holds, but the text does not come from the podcast's origin. + let mut pointed = account(3, "a.xml", true, 10); + pointed.card = build_card( + &pointed.subscription, + Some(&PodcastMeta { + feed_url: url("elsewhere.xml"), + title: "Elsewhere".into(), + ..pointed.subscription.podcast.clone() + }), + None, + None, + ); + assert!(pointed.meta_from_feed); + assert!(!pointed.trusted_meta()); + let origin = account(9, "a.xml", true, 20); + assert!(origin.trusted_meta()); + let aggregation = aggregate(vec![pointed, origin]); + assert_eq!(aggregation.representatives[&id("a.xml")], 9); + } + + #[test] + fn the_last_round_says_why_a_shared_podcast_is_not_published() { + let over = id("held-over-limit.xml"); + let failed = id("held-failed.xml"); + { + let mut memory = memory(); + memory.over_limit.insert(over.clone()); + memory.failed.insert(failed.clone()); + } + assert_eq!(held_reason(&over), Some("over_limit")); + assert_eq!(held_reason(&failed), Some("failed")); + assert_eq!(held_reason(&id("held-none.xml")), None); + let mut memory = memory(); + memory.over_limit.remove(&over); + memory.failed.remove(&failed); + } + + #[test] + fn a_card_never_carries_personal_state() { + let share = account(1, "a.xml", true, 10); + let card = share.card.unwrap(); + let json = serde_json::to_string(&card).unwrap(); + for personal in [ + "generation", + "op_id", + "hlc_ms", + "u1:share", + "dev:1", + "device", + ] { + assert!(!json.contains(personal), "{personal} in {json}"); + } + } +} diff --git a/src/podcasts/discovery_tests.rs b/src/podcasts/discovery_tests.rs new file mode 100644 index 0000000..56d95b4 --- /dev/null +++ b/src/podcasts/discovery_tests.rs @@ -0,0 +1,800 @@ +//! Public podcast discovery against PostgreSQL: the share toggle and its +//! privacy confirmation, the aggregation of several accounts, the +//! publication from the web's federation node to another loopback node and +//! the first publication after a restart. +//! +//! Ignored by default, like the other store tests: set +//! `FURU_DEVICE_SYNC_TEST_DATABASE_URL` to a disposable database. The +//! loopback test starts two iroh nodes in temporary directories (the same +//! setup as frid's `tests/podcast_dht.rs`). + +use std::sync::Arc; +use std::time::Duration; + +use music_dht::device_sync::{SyncOpPayload, SyncOpWire, random_hex}; +use music_dht::podcast::{FeedUrl, PodcastId, PodcastMeta}; +use music_dht::podcast_dht::{PODCAST_DHT_ALPN, PodcastDht}; +use music_dht::{MusicDhtConfig, MusicDhtService, NetworkId}; +use serde_json::json; + +use super::api::{self, ShareOutcome}; +use super::discovery::{self, Discovery, aggregate_share_states, load_shared}; +use super::postgres_tests::{cleanup, random_user, test_pool}; +use super::*; + +fn url(value: &str) -> FeedUrl { + FeedUrl::parse(value).unwrap() +} + +fn meta(feed: &str, title: &str) -> PodcastMeta { + PodcastMeta { + feed_url: url(feed), + title: title.into(), + author: Some("Host".into()), + description: Some("A public show".into()), + link: None, + image_url: None, + } +} + +async fn subscribe(store: &PodcastStore, user: i64, feed: &str, title: &str) -> PodcastId { + store + .subscribe(user, &meta(feed, title)) + .await + .unwrap() + .podcast_id +} + +fn share_op( + origin: &str, + seq: i64, + podcast_id: &PodcastId, + shared: bool, + generation: u64, +) -> SyncOpWire { + SyncOpWire { + op_id: format!("{origin}:{seq}"), + origin_device_id: origin.into(), + seq, + hlc_ms: 9_000_000_000_000 + seq, + payload: SyncOpPayload::PodcastShareSet { + podcast_id: podcast_id.clone(), + shared, + generation, + }, + } +} + +async fn share( + store: &PodcastStore, + user: i64, + podcast_id: &PodcastId, + body: serde_json::Value, +) -> Result { + api::set_share(store, user, podcast_id, body, Discovery::Off, now_ms()).await +} + +fn done(outcome: Result) -> Box { + match outcome.unwrap() { + ShareOutcome::Done(response) => response, + ShareOutcome::NeedsConfirmation(view) => panic!("unexpected confirmation: {view:?}"), + } +} + +fn needs_confirmation(outcome: Result) -> api::ShareView { + match outcome.unwrap() { + ShareOutcome::NeedsConfirmation(view) => view, + ShareOutcome::Done(response) => panic!("shared without confirmation: {:?}", response.share), + } +} + +fn confirm_body(view: &api::ShareView) -> serde_json::Value { + json!({ + "shared": true, + "confirm": { "feed_url": view.feed_url, "id_feed_url": view.id_feed_url }, + }) +} + +async fn published_ids(pool: &sqlx::PgPool, users: &[i64]) -> Vec<(PodcastId, i64)> { + let aggregation = aggregate_share_states( + load_shared(pool, Some(users)).await.unwrap().shares, + &Default::default(), + ); + aggregation.representatives.into_iter().collect() +} + +/// Active accounts (only they publish); random ids like the store tests. +async fn accounts(pool: &sqlx::PgPool) -> [i64; N] { + let users: [i64; N] = std::array::from_fn(|_| random_user()); + for user in users { + sqlx::query( + "INSERT INTO furumusic__user (id, username, role, is_active) + VALUES ($1, $2, 'user', TRUE)", + ) + .bind(user) + .bind(format!("podcast-discovery-test-{user}")) + .execute(pool) + .await + .unwrap(); + } + users +} + +async fn remove_accounts(pool: &sqlx::PgPool, users: &[i64]) { + cleanup(pool, users.iter().copied()).await; + sqlx::query("DELETE FROM furumusic__user WHERE id = ANY($1)") + .bind(users) + .execute(pool) + .await + .unwrap(); +} + +async fn log_len(pool: &sqlx::PgPool, user: i64) -> i64 { + sqlx::query_scalar("SELECT COUNT(*) FROM furumusic__fed_sync_ops WHERE user_id = $1") + .bind(user) + .fetch_one(pool) + .await + .unwrap() +} + +#[tokio::test] +#[ignore = "requires a disposable database in FURU_DEVICE_SYNC_TEST_DATABASE_URL"] +async fn postgres_share_toggle_asks_before_publishing_a_private_looking_feed() { + let pool = test_pool().await; + let store = PodcastStore::new(pool.clone()); + let [user, other] = accounts(&pool).await; + let secret = format!( + "https://members.example/feed.xml?token=8f3a9c2e7d1b4a6f{}", + random_hex(4) + ); + let podcast_id = subscribe(&store, user, &secret, "Members Only").await; + + // Private by default; the dialog needs the warnings and the URL. + let view = api::share_view(&store, user, &podcast_id, Discovery::Off) + .await + .unwrap(); + assert!(!view.shared && !view.pending_confirmation && view.can_share); + assert_eq!(view.discovery, "off"); + assert!(!view.warnings.is_empty()); + assert_eq!(view.feed_url.as_deref(), Some(secret.as_str())); + + // Enabling without confirming writes nothing. + let before = log_len(&pool, user).await; + let asked = needs_confirmation(share(&store, user, &podcast_id, json!({"shared": true})).await); + assert!(!asked.warnings.is_empty()); + assert!(store.share(user, &podcast_id).await.unwrap().is_none()); + // Confirming other URLs than the card's writes nothing either. + let wrong = + json!({"shared": true, "confirm": {"feed_url": "https://members.example/feed.xml"}}); + needs_confirmation(share(&store, user, &podcast_id, wrong).await); + assert_eq!(log_len(&pool, user).await, before); + assert!(published_ids(&pool, &[user]).await.is_empty()); + + // Confirming exactly the shown URLs shares and publishes. + let response = done(share(&store, user, &podcast_id, confirm_body(&asked)).await); + assert!(response.podcast.shared); + assert!(response.share.shared && !response.share.pending_confirmation); + assert_eq!(log_len(&pool, user).await, before + 1); + assert_eq!( + published_ids(&pool, &[user]).await, + vec![(podcast_id.clone(), user)] + ); + + // Another account cannot see or change this account's podcast. + let error = share(&store, other, &podcast_id, json!({"shared": false})) + .await + .unwrap_err(); + assert_eq!(error.status, 404); + // Malformed requests. + let error = share(&store, user, &podcast_id, json!({"share": true})) + .await + .unwrap_err(); + assert_eq!(error.status, 400); + + // Turning sharing off revokes and forgets the confirmation: sharing + // again asks again. + let response = done(share(&store, user, &podcast_id, json!({"shared": false})).await); + assert!(!response.share.shared); + assert!(published_ids(&pool, &[user]).await.is_empty()); + let confirmations: i64 = sqlx::query_scalar( + "SELECT COUNT(*) FROM furumusic__podcast_share_confirmation WHERE user_id = $1", + ) + .bind(user) + .fetch_one(&pool) + .await + .unwrap(); + assert_eq!(confirmations, 0); + needs_confirmation(share(&store, user, &podcast_id, json!({"shared": true})).await); + + // Shared on another trusted device (which asked its own user): the web + // still asks this account before its server publishes the URL. + let remote = share_op("tui_a", 1, &podcast_id, true, 100); + assert_eq!( + store.apply_remote_op(user, &remote).await.unwrap(), + ApplyOutcome::Changed + ); + let view = api::share_view(&store, user, &podcast_id, Discovery::Off) + .await + .unwrap(); + assert!(view.shared && view.pending_confirmation); + assert!(published_ids(&pool, &[user]).await.is_empty()); + let before = log_len(&pool, user).await; + let response = done(share(&store, user, &podcast_id, confirm_body(&view)).await); + assert!(response.share.shared && !response.share.pending_confirmation); + // Confirming an existing decision emits no operation. + assert_eq!(log_len(&pool, user).await, before); + assert_eq!(published_ids(&pool, &[user]).await.len(), 1); + + // A feed that moves to another URL that looks private asks again. + let moved = format!( + "https://members.example/v2/feed.xml?key=9a8b7c6d5e4f3a2b{}", + random_hex(4) + ); + sqlx::query( + "UPDATE furumusic__podcast SET feed_url = $3 WHERE user_id = $1 AND podcast_id = $2", + ) + .bind(user) + .bind(podcast_id.as_str()) + .bind(&moved) + .execute(&pool) + .await + .unwrap(); + let view = api::share_view(&store, user, &podcast_id, Discovery::Off) + .await + .unwrap(); + assert!(view.shared && view.pending_confirmation); + assert_eq!(view.feed_url.as_deref(), Some(moved.as_str())); + assert_eq!(view.id_feed_url.as_deref(), Some(secret.as_str())); + assert!(published_ids(&pool, &[user]).await.is_empty()); + + // Unsubscribing revokes and forgets; the podcast is private again. + api::unsubscribe(&store, user, &podcast_id).await.unwrap(); + let view = api::share_view(&store, user, &podcast_id, Discovery::Off) + .await + .unwrap(); + assert!(!view.shared && !view.can_share); + let error = share(&store, user, &podcast_id, json!({"shared": true})) + .await + .unwrap_err(); + assert_eq!(error.status, 409); + + remove_accounts(&pool, &[user, other]).await; +} + +#[tokio::test] +#[ignore = "requires a disposable database in FURU_DEVICE_SYNC_TEST_DATABASE_URL"] +async fn postgres_shared_podcasts_aggregate_across_accounts() { + let pool = test_pool().await; + let store = PodcastStore::new(pool.clone()); + let users = accounts::<3>(&pool).await; + let [a, b, c] = users; + let feed = format!("https://podcasts.example/shared-{}.xml", random_hex(4)); + let private = format!("https://podcasts.example/private-{}.xml", random_hex(4)); + let podcast_id = subscribe(&store, a, &feed, "Shared Show").await; + assert_eq!(subscribe(&store, b, &feed, "Shared Show").await, podcast_id); + assert_eq!(subscribe(&store, c, &feed, "Shared Show").await, podcast_id); + let private_id = subscribe(&store, a, &private, "Private Show").await; + + // Nothing shared: nothing published, not even the private podcast. + assert!(published_ids(&pool, &users).await.is_empty()); + + done(share(&store, a, &podcast_id, json!({"shared": true})).await); + done(share(&store, b, &podcast_id, json!({"shared": true})).await); + let published = published_ids(&pool, &users).await; + assert_eq!(published, vec![(podcast_id.clone(), a.min(b))]); + let aggregation = aggregate_share_states( + load_shared(&pool, Some(&users)).await.unwrap().shares, + &Default::default(), + ); + let state = &aggregation.states[0]; + assert!(state.is_shared() && state.privacy_confirmed); + let card = state.card.as_ref().unwrap(); + assert_eq!(card.feed_url, url(&feed)); + assert_eq!(card.title, "Shared Show"); + + // A deactivated account publishes nothing; the other one still shares. + let set_active = async |user: i64, active: bool| { + sqlx::query("UPDATE furumusic__user SET is_active = $2 WHERE id = $1") + .bind(user) + .bind(active) + .execute(&pool) + .await + .unwrap(); + }; + set_active(a.min(b), false).await; + assert_eq!( + published_ids(&pool, &users).await, + vec![(podcast_id.clone(), a.max(b))] + ); + set_active(a.max(b), false).await; + assert!(published_ids(&pool, &users).await.is_empty()); + set_active(a, true).await; + set_active(b, true).await; + + // One account stops sharing: the other still shares the same card. + done(share(&store, a, &podcast_id, json!({"shared": false})).await); + assert_eq!( + published_ids(&pool, &users).await, + vec![(podcast_id.clone(), b)] + ); + + // A stale `shared = true` of A (delivered late from an old device log) + // cannot republish for A. + let stale = share_op("old_device", 1, &podcast_id, true, 1); + assert_eq!( + store.apply_remote_op(a, &stale).await.unwrap(), + ApplyOutcome::Unchanged + ); + let view = api::share_view(&store, a, &podcast_id, Discovery::Off) + .await + .unwrap(); + assert!(!view.shared); + + // The last sharing account unsubscribes: the card is revoked, and + // subscribing again is private. + api::unsubscribe(&store, b, &podcast_id).await.unwrap(); + assert!(published_ids(&pool, &users).await.is_empty()); + api::resubscribe(&store, b, &podcast_id).await.unwrap(); + assert!(published_ids(&pool, &users).await.is_empty()); + // Share decisions of C never existed; the private podcast of A stays + // private throughout. + let view = api::share_view(&store, a, &private_id, Discovery::Off) + .await + .unwrap(); + assert!(!view.shared); + + remove_accounts(&pool, &users).await; +} + +// --------------------------------------------------------------------------- +// Loopback: publish from the web node, find from another node +// --------------------------------------------------------------------------- + +struct Node { + service: Arc, + dht: Arc, + tasks: Vec>, +} + +impl Node { + async fn stop(self) { + for task in &self.tasks { + task.abort(); + } + self.service.shutdown().await.unwrap(); + } +} + +async fn start_node(directory: &std::path::Path, network: NetworkId) -> Node { + std::fs::create_dir_all(directory).unwrap(); + let config = MusicDhtConfig::builder() + .data_dir(directory) + .network_id(network) + .request_timeout(Duration::from_secs(2)) + .lookup_timeout(Duration::from_secs(5)) + .transport_timeout(Duration::from_secs(5)) + .dial_timeout(Duration::from_secs(2)) + .schema_independent_stream_protocol(PODCAST_DHT_ALPN) + .build() + .unwrap(); + let (service, mut events) = MusicDhtService::start(config).await.unwrap(); + let service = Arc::new(service); + let events = tokio::spawn(async move { while events.recv().await.is_some() {} }); + // The web's own wiring: the overlay on the node's endpoint, its cache + // in the federation data directory. + let (dht, serve) = crate::federation::podcast_discovery::open(&service, directory) + .await + .expect("overlay opens"); + assert!( + directory + .join(crate::federation::podcast_discovery::CACHE_FILE) + .exists() + ); + Node { + service, + dht, + tasks: vec![events, serve], + } +} + +async fn wait_for(what: &str, mut check: impl AsyncFnMut() -> Option) -> T { + let started = std::time::Instant::now(); + loop { + if let Some(value) = check().await { + return value; + } + assert!( + started.elapsed() < Duration::from_secs(90), + "timed out waiting for {what}" + ); + tokio::time::sleep(Duration::from_millis(300)).await; + } +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[ignore = "requires a disposable database in FURU_DEVICE_SYNC_TEST_DATABASE_URL"] +async fn postgres_web_node_publishes_shared_podcasts_to_another_node() { + tokio::time::timeout(Duration::from_secs(240), loopback_scenario()) + .await + .unwrap(); +} + +async fn loopback_scenario() { + let pool = test_pool().await; + let store = PodcastStore::new(pool.clone()); + let users = accounts::<2>(&pool).await; + let [u1, u2] = users; + // An account of the stranger's side (another server in real life) that + // subscribes from the card it found. + let finder = random_user(); + let tag = random_hex(6); + let directory = std::env::temp_dir().join(format!("furumusic-podcast-dht-{tag}")); + let network = NetworkId::from_name(&format!("furumusic-podcast-loopback-{tag}")); + let web = start_node(&directory.join("web"), network).await; + let stranger = start_node(&directory.join("stranger"), network).await; + + // The stranger reaches the web node through the music-DHT transport + // only: no device-sync pairing, another identity. + meet(&web, &stranger).await; + + // Two accounts share one podcast; one of them also has a private one. + let title = format!("Loopback Radio {tag}"); + let hidden = format!("Hidden Garden {tag}"); + let server = super::test_server::TestServer::start(); + server.set( + "/loopback.xml", + super::test_server::Route::new( + format!( + r#"{title} +First1Mon, 01 Jan 2024 10:00:00 GMT +"# + ), + "application/rss+xml", + ), + ); + let feed = server.url("/loopback.xml"); + let podcast_id = subscribe(&store, u1, &feed, &title).await; + subscribe(&store, u2, &feed, &title).await; + subscribe( + &store, + u1, + &format!("https://podcasts.example/hidden-{tag}.xml"), + &hidden, + ) + .await; + // Sharing while the stranger already looks: nothing before. + assert!(stranger.dht.search(&title, 10).await.unwrap().is_empty()); + done(share(&store, u1, &podcast_id, json!({"shared": true})).await); + done(share(&store, u2, &podcast_id, json!({"shared": true})).await); + + let report = discovery::reconcile_once(&pool, &web.dht, Some(&users)) + .await + .unwrap(); + assert_eq!(report.aggregation.states.len(), 1, "one card per podcast"); + assert_eq!(report.stats.published, 1); + assert!(report.stats.held_back.is_empty() && report.stats.failed.is_empty()); + let view = api::share_view(&store, u1, &podcast_id, Discovery::Running(&web.dht)) + .await + .unwrap(); + assert!(view.published && view.held.is_none()); + assert_eq!(view.discovery, "running"); + + let hit = wait_for("the shared card", async || { + stranger + .dht + .search(&title, 10) + .await + .unwrap() + .into_iter() + .find(|hit| hit.podcast_id == podcast_id) + }) + .await; + assert_eq!(hit.feed_url, url(&feed)); + assert_eq!(hit.card.title, title); + assert_eq!(hit.publishers, vec![web.service.endpoint_id()]); + // The private podcast is not discoverable, by text or by id. + assert!(stranger.dht.search(&hidden, 10).await.unwrap().is_empty()); + + // The finder sees the card in the federated search, looks at its + // episodes and subscribes: its own private subscription with the + // card's id, fetched from the feed (never from the publisher). + let views = super::search::hit_views(&store, finder, &title, &[], vec![hit.clone()]) + .await + .unwrap(); + assert_eq!(views.len(), 1); + assert!(!views[0].subscribed && views[0].exact_match); + let found = json!({ "podcast_id": views[0].podcast_id, "feed_url": views[0].feed_url }); + let fetcher = super::postgres_tests::local_fetcher(); + let preview = api::preview_found(&store, &fetcher, finder, found.clone()) + .await + .unwrap(); + assert_eq!(preview.episodes.len(), 1); + let subscribed = api::subscribe_found(&store, &fetcher, finder, found, now_ms()) + .await + .unwrap(); + assert_eq!(subscribed.podcast.podcast_id, podcast_id.to_string()); + assert!(subscribed.podcast.subscribed && !subscribed.podcast.shared); + + // Idempotent; one account stopping keeps the card of the other. + done(share(&store, u1, &podcast_id, json!({"shared": false})).await); + let report = discovery::reconcile_once(&pool, &web.dht, Some(&users)) + .await + .unwrap(); + assert_eq!( + ( + report.stats.published, + report.stats.unchanged, + report.stats.unpublished + ), + (0, 1, 0) + ); + + // The last account unsubscribes: the card is revoked with a tombstone. + api::unsubscribe(&store, u2, &podcast_id).await.unwrap(); + let report = discovery::reconcile_once(&pool, &web.dht, Some(&users)) + .await + .unwrap(); + assert_eq!(report.stats.unpublished, 1); + wait_for("the revocation", async || { + stranger + .dht + .lookup_podcast(&podcast_id) + .await + .unwrap() + .is_none() + .then_some(()) + }) + .await; + assert!( + stranger + .dht + .search(&title, 10) + .await + .unwrap() + .iter() + .all(|hit| hit.podcast_id != podcast_id) + ); + let view = api::share_view(&store, u2, &podcast_id, Discovery::Running(&web.dht)) + .await + .unwrap(); + assert!(!view.shared && !view.published); + // The revocation does not touch the finder's subscription. + let kept = store.podcast(finder, &podcast_id).await.unwrap().unwrap(); + assert!(kept.subscribed && !kept.shared); + cleanup(&pool, [finder]).await; + + web.stop().await; + stranger.stop().await; + remove_accounts(&pool, &users).await; + let _ = std::fs::remove_dir_all(&directory); +} + +/// Connects `stranger` to `web` and fills both overlay routing tables. +async fn meet(web: &Node, stranger: &Node) { + stranger + .service + .connect(web.service.ticket().await.unwrap()) + .await + .unwrap(); + wait_for("music DHT contacts", async || { + (!web.service.known_peers().is_empty() && !stranger.service.known_peers().is_empty()) + .then_some(()) + }) + .await; + web.dht.bootstrap().await.unwrap(); + stranger.dht.bootstrap().await.unwrap(); + wait_for("overlay contacts", async || { + (web.dht.known_peers() >= 1 && stranger.dht.known_peers() >= 1).then_some(()) + }) + .await; +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[ignore = "requires a disposable database in FURU_DEVICE_SYNC_TEST_DATABASE_URL"] +async fn postgres_a_restarted_server_publishes_only_after_the_first_device_sync() { + tokio::time::timeout(Duration::from_secs(240), restart_scenario()) + .await + .unwrap(); +} + +/// Interop scenario 7c with the web server as the device that was stopped: +/// a TUI of the account turned sharing off meanwhile, and the server comes +/// back with the old "shared" rows. +async fn restart_scenario() { + let pool = test_pool().await; + let store = PodcastStore::new(pool.clone()); + let users = accounts::<1>(&pool).await; + let [user] = users; + let tag = random_hex(6); + let directory = std::env::temp_dir().join(format!("furumusic-podcast-restart-{tag}")); + let network = NetworkId::from_name(&format!("furumusic-podcast-restart-{tag}")); + + let kept_title = format!("Kept Show {tag}"); + let revoked_title = format!("Revoked Show {tag}"); + let kept = subscribe( + &store, + user, + &format!("https://podcasts.example/kept-{tag}.xml"), + &kept_title, + ) + .await; + let revoked = subscribe( + &store, + user, + &format!("https://podcasts.example/revoked-{tag}.xml"), + &revoked_title, + ) + .await; + done(share(&store, user, &kept, json!({"shared": true})).await); + done(share(&store, user, &revoked, json!({"shared": true})).await); + let web = start_node(&directory.join("web"), network).await; + let report = discovery::reconcile_once(&pool, &web.dht, Some(&users)) + .await + .unwrap(); + assert_eq!(report.stats.published, 2); + web.stop().await; + + // The server restarts from the same directory (same key, same overlay + // cache) and meets a stranger before its first device-sync round. + let web = start_node(&directory.join("web"), network).await; + let stranger = start_node(&directory.join("stranger"), network).await; + meet(&web, &stranger).await; + let (first_sync_done, first_sync) = tokio::sync::watch::channel(false); + let reconciler = tokio::spawn(discovery::run_rounds( + pool.clone(), + Arc::clone(&web.dht), + first_sync, + Duration::from_secs(600), + Some(users.to_vec()), + )); + // Time for a round: nothing is announced from the stored decisions. + tokio::time::sleep(Duration::from_secs(5)).await; + for podcast_id in [&kept, &revoked] { + assert!( + stranger + .dht + .lookup_podcast(podcast_id) + .await + .unwrap() + .is_none(), + "a card was announced before the first device-sync round" + ); + } + assert!( + stranger + .dht + .search(&revoked_title, 10) + .await + .unwrap() + .is_empty() + ); + + // The first device-sync round brings the share-off the TUI made while + // the server was stopped. + let share_off = share_op("tui_while_stopped", 1, &revoked, false, 100); + assert_eq!( + store.apply_remote_op(user, &share_off).await.unwrap(), + ApplyOutcome::Changed + ); + first_sync_done.send_replace(true); + + let hit = wait_for("the card still shared", async || { + stranger.dht.lookup_podcast(&kept).await.unwrap() + }) + .await; + assert_eq!(hit.card.title, kept_title); + assert_eq!(hit.publishers, vec![web.service.endpoint_id()]); + assert!( + stranger + .dht + .lookup_podcast(&revoked) + .await + .unwrap() + .is_none() + ); + assert!( + stranger + .dht + .search(&revoked_title, 10) + .await + .unwrap() + .is_empty() + ); + let revoked_record = web + .dht + .published() + .await + .unwrap() + .into_iter() + .find(|publication| publication.podcast_id == revoked) + .unwrap(); + assert!(!revoked_record.active, "the revoked card is tombstoned"); + + reconciler.abort(); + web.stop().await; + stranger.stop().await; + remove_accounts(&pool, &users).await; + let _ = std::fs::remove_dir_all(&directory); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[ignore = "requires a disposable database in FURU_DEVICE_SYNC_TEST_DATABASE_URL"] +async fn postgres_an_unreadable_row_does_not_stop_other_revocations() { + tokio::time::timeout(Duration::from_secs(120), unreadable_row_scenario()) + .await + .unwrap(); +} + +async fn unreadable_row_scenario() { + let pool = test_pool().await; + let store = PodcastStore::new(pool.clone()); + let users = accounts::<2>(&pool).await; + let [a, b] = users; + let tag = random_hex(6); + let directory = std::env::temp_dir().join(format!("furumusic-podcast-unreadable-{tag}")); + let network = NetworkId::from_name(&format!("furumusic-podcast-unreadable-{tag}")); + let web = start_node(&directory.join("web"), network).await; + + let a_feed = format!("https://podcasts.example/a-{tag}.xml"); + let b_feed = format!("https://podcasts.example/b-{tag}.xml"); + let a_id = subscribe(&store, a, &a_feed, "A Show").await; + let b_id = subscribe(&store, b, &b_feed, "B Show").await; + done(share(&store, a, &a_id, json!({"shared": true})).await); + done(share(&store, b, &b_id, json!({"shared": true})).await); + let report = discovery::reconcile_once(&pool, &web.dht, Some(&users)) + .await + .unwrap(); + assert_eq!(report.stats.published, 2); + let active = async || { + web.dht + .published() + .await + .unwrap() + .into_iter() + .filter(|publication| publication.active) + .map(|publication| publication.podcast_id) + .collect::>() + }; + assert_eq!(active().await.len(), 2); + + // B's stored alias no longer decodes (an imported row, a stricter + // parser after an upgrade); A turns sharing off. + sqlx::query( + "INSERT INTO furumusic__podcast_feed_alias (user_id, podcast_id, feed_url) + VALUES ($1, $2, 'not a feed url')", + ) + .bind(b) + .bind(b_id.as_str()) + .execute(&pool) + .await + .unwrap(); + done(share(&store, a, &a_id, json!({"shared": false})).await); + + // The round still runs: A's card is revoked, and B's card (which can no + // longer be read) fails closed instead of being republished forever. + let report = discovery::reconcile_once(&pool, &web.dht, Some(&users)) + .await + .unwrap(); + assert_eq!(report.aggregation.unreadable, 1); + assert!(report.aggregation.states.is_empty()); + assert_eq!(report.stats.unpublished, 2); + assert!(active().await.is_empty()); + + // Repaired: B's card is published again; A's stays revoked. + sqlx::query( + "DELETE FROM furumusic__podcast_feed_alias WHERE user_id = $1 AND feed_url = 'not a feed url'", + ) + .bind(b) + .execute(&pool) + .await + .unwrap(); + let report = discovery::reconcile_once(&pool, &web.dht, Some(&users)) + .await + .unwrap(); + assert_eq!(report.aggregation.unreadable, 0); + assert_eq!(active().await, vec![b_id.clone()]); + + web.stop().await; + remove_accounts(&pool, &users).await; + let _ = std::fs::remove_dir_all(&directory); +} diff --git a/src/podcasts/fetch.rs b/src/podcasts/fetch.rs new file mode 100644 index 0000000..98b295a --- /dev/null +++ b/src/podcasts/fetch.rs @@ -0,0 +1,322 @@ +//! Conditional, size- and time-bounded feed downloads through [`SafeHttp`]. + +use std::fmt; +use std::time::Duration; + +use reqwest::header::{ + ACCEPT, ACCEPT_ENCODING, ETAG, HeaderMap, HeaderValue, IF_MODIFIED_SINCE, IF_NONE_MATCH, + LAST_MODIFIED, RETRY_AFTER, +}; +use reqwest::{StatusCode, Url}; + +use music_dht::podcast::FeedUrl; +use music_dht::podcast::feed::{ParsedFeed, parse_feed_with_base}; + +use super::net::{GuardError, REFUSED_MESSAGE, SafeHttp, SafeHttpConfig}; + +/// Hard cap on a feed document (large catalogs are a few MiB). +pub(crate) const MAX_FEED_BYTES: usize = 32 * 1024 * 1024; +/// Whole fetch, from connect to the last body byte. +pub(crate) const FEED_FETCH_TIMEOUT: Duration = Duration::from_secs(60); +/// Redirect hops followed for a feed. +pub(crate) const MAX_FEED_REDIRECTS: usize = 5; +/// `Retry-After` values above this are clamped. +const MAX_RETRY_AFTER: Duration = Duration::from_secs(24 * 60 * 60); + +const FEED_ACCEPT: &str = "application/rss+xml, application/atom+xml, application/xml;q=0.9, \ + text/xml;q=0.9, */*;q=0.5"; + +/// HTTP client settings of the feed fetcher. +pub(crate) const FEED_HTTP: SafeHttpConfig = SafeHttpConfig { + connect_timeout: Duration::from_secs(10), + read_timeout: Duration::from_secs(30), + total_timeout: Some(FEED_FETCH_TIMEOUT), + max_redirects: MAX_FEED_REDIRECTS, +}; + +/// Validators of the previous successful fetch. +#[derive(Debug, Clone, Default, PartialEq, Eq, Hash)] +pub(crate) struct Validators { + pub(crate) etag: Option, + pub(crate) last_modified: Option, +} + +/// A successfully fetched and parsed document. +#[derive(Debug, Clone)] +pub(crate) struct FetchedFeed { + pub(crate) feed: ParsedFeed, + /// Where the feed now lives: the final URL after a chain of permanent + /// redirects, otherwise the requested URL. + pub(crate) fetched_from: FeedUrl, + /// Whether the feed moved: at least one redirect, all permanent (see + /// [`super::net::Followed::moved_permanently`]), to a valid feed URL. + pub(crate) moved_permanently: bool, + /// New validators for the next conditional request. + pub(crate) validators: Validators, +} + +/// Outcome of [`FeedFetcher::fetch`]. +#[derive(Debug, Clone)] +pub(crate) enum FeedFetch { + /// `304 Not Modified`: the stored catalog is current. + NotModified, + /// A new document. + Fetched(Box), +} + +/// Kind of a failed fetch. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) enum FetchErrorKind { + /// Refused by the SSRF guard (blocked address or port, scheme, + /// credentials). + Refused, + /// Non-success HTTP status. + Status(u16), + /// Body over [`MAX_FEED_BYTES`]. + TooLarge, + /// The document is not a feed. + Parse, + /// Timed out. + Timeout, + /// DNS, connect, TLS or a dropped connection. + Transport, +} + +/// A failed fetch. The stored catalog is never touched on failure. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct FetchError { + pub(crate) kind: FetchErrorKind, + /// Details for the server logs (no URLs; see [`Self::public_message`] + /// for what users see). + pub(crate) message: String, + /// Server-requested delay (`Retry-After` on 429/503), clamped to a day. + pub(crate) retry_after: Option, +} + +impl FetchError { + fn new(kind: FetchErrorKind, message: impl Into) -> Self { + Self { + kind, + message: message.into(), + retry_after: None, + } + } + + /// What the signed-in user is told (API answers, the stored last + /// error): the kind of failure, never resolved addresses or transport + /// details, which would let users probe the server's network. + pub(crate) fn public_message(&self) -> String { + match &self.kind { + FetchErrorKind::Refused => REFUSED_MESSAGE.to_string(), + FetchErrorKind::Status(status) => format!("the server answered HTTP {status}"), + FetchErrorKind::TooLarge => format!( + "the feed is larger than {} MiB", + MAX_FEED_BYTES / (1024 * 1024) + ), + FetchErrorKind::Parse => "the document is not a podcast feed".to_string(), + FetchErrorKind::Timeout => "the request timed out".to_string(), + FetchErrorKind::Transport => "could not connect to the server".to_string(), + } + } +} + +impl fmt::Display for FetchError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str(&self.message) + } +} + +impl std::error::Error for FetchError {} + +impl From for FetchError { + fn from(error: GuardError) -> Self { + let kind = if error.is_refusal() { + FetchErrorKind::Refused + } else if error.is_timeout() { + FetchErrorKind::Timeout + } else { + FetchErrorKind::Transport + }; + Self::new(kind, error.to_string()) + } +} + +/// Parses `Retry-After` (delta seconds or an HTTP date). +pub(crate) fn parse_retry_after( + value: &str, + now: chrono::DateTime, +) -> Option { + let value = value.trim(); + let delay = if let Ok(seconds) = value.parse::() { + Duration::from_secs(seconds) + } else { + let at = chrono::DateTime::parse_from_rfc2822(value).ok()?; + (at.with_timezone(&chrono::Utc) - now) + .to_std() + .unwrap_or_default() + }; + Some(delay.min(MAX_RETRY_AFTER)) +} + +/// Downloads feeds with the SSRF guard, conditional requests, redirects +/// and hard size/time limits; parses them with the shared parser. +#[derive(Debug, Clone)] +pub(crate) struct FeedFetcher { + http: SafeHttp, + max_bytes: usize, +} + +impl FeedFetcher { + /// The production fetcher (public addresses only). + pub(crate) fn new() -> anyhow::Result { + Ok(Self::with_http(SafeHttp::public(FEED_HTTP)?)) + } + + pub(crate) fn with_http(http: SafeHttp) -> Self { + Self { + http, + max_bytes: MAX_FEED_BYTES, + } + } + + #[cfg(test)] + pub(crate) fn with_max_bytes(mut self, max_bytes: usize) -> Self { + self.max_bytes = max_bytes; + self + } + + /// Fetches `url`, conditionally when `validators` are known. + pub(crate) async fn fetch( + &self, + url: &FeedUrl, + validators: &Validators, + ) -> Result { + match tokio::time::timeout(FEED_FETCH_TIMEOUT, self.fetch_inner(url, validators)).await { + Ok(result) => result, + Err(_) => Err(FetchError::new( + FetchErrorKind::Timeout, + format!( + "feed fetch timed out after {} s", + FEED_FETCH_TIMEOUT.as_secs() + ), + )), + } + } + + async fn fetch_inner( + &self, + feed_url: &FeedUrl, + validators: &Validators, + ) -> Result { + let url = Url::parse(feed_url.as_str()) + .map_err(|error| FetchError::new(FetchErrorKind::Refused, error.to_string()))?; + let mut headers = HeaderMap::new(); + headers.insert(ACCEPT, HeaderValue::from_static(FEED_ACCEPT)); + // Compressed bodies are not negotiated (no decoder is enabled), so + // the size cap applies to what is parsed. + headers.insert(ACCEPT_ENCODING, HeaderValue::from_static("identity")); + if let Some(etag) = validators + .etag + .as_deref() + .and_then(|value| HeaderValue::from_str(value).ok()) + { + headers.insert(IF_NONE_MATCH, etag); + } + if let Some(date) = validators + .last_modified + .as_deref() + .and_then(|value| HeaderValue::from_str(value).ok()) + { + headers.insert(IF_MODIFIED_SINCE, date); + } + let followed = self.http.get(url, &headers).await?; + let mut moved_permanently = followed.moved_permanently(); + let final_url = followed.final_url; + let response = followed.response; + let status = response.status(); + if status == StatusCode::NOT_MODIFIED { + return Ok(FeedFetch::NotModified); + } + if !status.is_success() { + let mut error = FetchError::new( + FetchErrorKind::Status(status.as_u16()), + format!("feed request failed: HTTP {status}"), + ); + if matches!(status.as_u16(), 429 | 503) { + error.retry_after = response + .headers() + .get(RETRY_AFTER) + .and_then(|value| value.to_str().ok()) + .and_then(|value| parse_retry_after(value, chrono::Utc::now())); + } + return Err(error); + } + let header = |name| { + response + .headers() + .get(name) + .and_then(|value: &HeaderValue| value.to_str().ok()) + .map(str::to_string) + }; + let validators = Validators { + etag: header(ETAG), + last_modified: header(LAST_MODIFIED), + }; + let body = read_capped(response, self.max_bytes).await?; + // Relative enclosure URLs resolve against where the feed really is. + let base = final_url.to_string(); + let feed = tokio::task::spawn_blocking(move || parse_feed_with_base(&body, Some(&base))) + .await + .map_err(|error| { + FetchError::new( + FetchErrorKind::Parse, + format!("feed parser task failed: {error}"), + ) + })? + .map_err(|error| { + FetchError::new( + FetchErrorKind::Parse, + format!("not a podcast feed: {error}"), + ) + })?; + // Only a permanent move changes the feed URL; a target that is not a + // valid feed URL (over-long, ...) is not learned. + let moved_to = moved_permanently + .then(|| FeedUrl::parse(final_url.as_str()).ok()) + .flatten(); + moved_permanently = moved_to.is_some(); + let fetched_from = moved_to.unwrap_or_else(|| feed_url.clone()); + Ok(FeedFetch::Fetched(Box::new(FetchedFeed { + feed, + fetched_from, + moved_permanently, + validators, + }))) + } +} + +async fn read_capped(mut response: reqwest::Response, cap: usize) -> Result, FetchError> { + let too_large = |detail: String| { + FetchError::new( + FetchErrorKind::TooLarge, + format!("feed is too large ({detail}, limit {cap} bytes)"), + ) + }; + if let Some(len) = response.content_length() + && len > cap as u64 + { + return Err(too_large(format!("{len} bytes"))); + } + let mut body = Vec::new(); + while let Some(chunk) = response + .chunk() + .await + .map_err(|error| FetchError::from(GuardError::from(error)))? + { + if body.len() + chunk.len() > cap { + return Err(too_large("body exceeds the limit".to_string())); + } + body.extend_from_slice(&chunk); + } + Ok(body) +} diff --git a/src/podcasts/fetch_tests.rs b/src/podcasts/fetch_tests.rs new file mode 100644 index 0000000..b5be9eb --- /dev/null +++ b/src/podcasts/fetch_tests.rs @@ -0,0 +1,289 @@ +//! Feed fetching (conditional requests, redirects, limits) and the refresh +//! schedule, against the local test server. + +use std::time::Duration; + +use music_dht::podcast::{FeedUrl, PodcastId}; + +use super::fetch::{FeedFetch, FeedFetcher, FetchErrorKind, Validators, parse_retry_after}; +use super::runner::{ + BACKOFF_BASE_MS, BACKOFF_MAX_MS, REFRESH_INTERVAL_MS, failure_backoff_ms, next_regular_check_ms, +}; +use std::sync::Arc; + +use super::test_server::{Route, ScriptedLookup, TestServer, test_http}; + +// --------------------------------------------------------------------------- +// Feed fetching +// --------------------------------------------------------------------------- + +fn rss(items: &str) -> String { + format!( + r#"Show{items}"# + ) +} + +fn item(guid: &str) -> String { + format!( + r#"{guid}{guid}"# + ) +} + +fn fetcher(lookup: Arc) -> FeedFetcher { + FeedFetcher::with_http(test_http(lookup)) +} + +fn feed_url(text: &str) -> FeedUrl { + FeedUrl::parse(text).unwrap() +} + +#[tokio::test] +async fn feeds_are_fetched_conditionally() { + let server = TestServer::start(); + server.set( + "/feed.xml", + Route::new(rss(&item("a")), "application/rss+xml") + .etag("\"v1\"") + .last_modified("Wed, 21 Oct 2026 07:28:00 GMT"), + ); + let fetcher = fetcher(ScriptedLookup::with(&[])); + let feed = feed_url(&server.url("/feed.xml")); + let FeedFetch::Fetched(first) = fetcher.fetch(&feed, &Validators::default()).await.unwrap() + else { + panic!("expected a document"); + }; + assert_eq!(first.feed.episodes.len(), 1); + assert_eq!( + first.feed.episodes[0].audio_url, + server.url("/audio/a.mp3"), + "relative enclosures resolve against the feed URL" + ); + assert_eq!(first.fetched_from, feed); + assert!(!first.moved_permanently); + assert_eq!(first.validators.etag.as_deref(), Some("\"v1\"")); + let second = fetcher.fetch(&feed, &first.validators).await.unwrap(); + assert!(matches!(second, FeedFetch::NotModified)); + // Only Last-Modified known. + let by_date = Validators { + etag: None, + last_modified: first.validators.last_modified.clone(), + }; + assert!(matches!( + fetcher.fetch(&feed, &by_date).await.unwrap(), + FeedFetch::NotModified + )); + let requests = server.requests("/feed.xml"); + assert_eq!(requests.len(), 3); + assert!(!requests[0].contains_key("if-none-match")); + assert_eq!(requests[1]["if-none-match"], "\"v1\""); + assert_eq!( + requests[1]["if-modified-since"], + "Wed, 21 Oct 2026 07:28:00 GMT" + ); + assert!(!requests[2].contains_key("if-none-match")); + assert_eq!(requests[2]["accept-encoding"], "identity"); +} + +#[tokio::test] +async fn only_permanent_redirect_chains_move_a_feed() { + let server = TestServer::start(); + server.set( + "/new.xml", + Route::new(rss(&item("a")), "application/rss+xml"), + ); + server.set("/old.xml", Route::redirect(301, "/new.xml")); + server.set("/moved.xml", Route::redirect(308, "/old.xml")); + server.set("/temp.xml", Route::redirect(302, "/new.xml")); + server.set("/mixed.xml", Route::redirect(307, "/old.xml")); + let fetcher = fetcher(ScriptedLookup::with(&[])); + for (path, moved) in [ + ("/old.xml", true), + ("/moved.xml", true), + ("/temp.xml", false), + ("/mixed.xml", false), + ] { + let requested = feed_url(&server.url(path)); + let FeedFetch::Fetched(document) = fetcher + .fetch(&requested, &Validators::default()) + .await + .unwrap() + else { + panic!("{path}"); + }; + assert_eq!(document.moved_permanently, moved, "{path}"); + let expected = if moved { + feed_url(&server.url("/new.xml")) + } else { + requested + }; + assert_eq!(document.fetched_from, expected, "{path}"); + assert_eq!( + document.feed.episodes[0].audio_url, + server.url("/audio/a.mp3") + ); + } +} + +#[tokio::test] +async fn feed_fetch_failures_are_typed() { + let server = TestServer::start(); + let big = rss(&item("x").repeat(200)); + server.set("/big.xml", Route::new(big.clone(), "application/rss+xml")); + server.set( + "/big-chunked.xml", + Route::new(big.clone(), "application/rss+xml").no_length(), + ); + server.set( + "/page.html", + Route::new("hi", "text/html"), + ); + server.set( + "/busy.xml", + Route::new("busy", "text/plain") + .status(503) + .header("Retry-After", "120"), + ); + server.set("/gone.xml", Route::new("", "text/plain").status(410)); + server.set( + "/private.xml", + Route::redirect(301, "http://192.168.0.1/feed"), + ); + let fetcher = fetcher(ScriptedLookup::with(&[])).with_max_bytes(big.len() - 1); + let fetch = |path: &str| { + let feed = feed_url(&server.url(path)); + let fetcher = fetcher.clone(); + async move { + fetcher + .fetch(&feed, &Validators::default()) + .await + .unwrap_err() + } + }; + assert_eq!(fetch("/big.xml").await.kind, FetchErrorKind::TooLarge); + assert_eq!( + fetch("/big-chunked.xml").await.kind, + FetchErrorKind::TooLarge + ); + assert_eq!(fetch("/page.html").await.kind, FetchErrorKind::Parse); + let busy = fetch("/busy.xml").await; + assert_eq!(busy.kind, FetchErrorKind::Status(503)); + assert_eq!(busy.retry_after, Some(Duration::from_secs(120))); + let gone = fetch("/gone.xml").await; + assert_eq!(gone.kind, FetchErrorKind::Status(410)); + assert_eq!(gone.retry_after, None); + assert_eq!(fetch("/private.xml").await.kind, FetchErrorKind::Refused); + assert_eq!( + fetch("/missing.xml").await.kind, + FetchErrorKind::Status(404) + ); +} + +#[test] +fn retry_after_accepts_seconds_and_dates() { + let now = chrono::DateTime::parse_from_rfc3339("2026-11-19T10:00:00Z") + .unwrap() + .with_timezone(&chrono::Utc); + assert_eq!(parse_retry_after("90", now), Some(Duration::from_secs(90))); + assert_eq!( + parse_retry_after("Thu, 19 Nov 2026 10:05:00 GMT", now), + Some(Duration::from_secs(300)) + ); + assert_eq!( + parse_retry_after("Thu, 19 Nov 2026 09:00:00 GMT", now), + Some(Duration::ZERO) + ); + assert_eq!( + parse_retry_after("999999999", now), + Some(Duration::from_secs(24 * 60 * 60)) + ); + assert_eq!(parse_retry_after("soon", now), None); +} + +// --------------------------------------------------------------------------- +// Refresh schedule +// --------------------------------------------------------------------------- + +#[test] +fn failures_back_off_exponentially_up_to_a_day() { + assert_eq!(failure_backoff_ms(1, None), BACKOFF_BASE_MS); + assert_eq!(failure_backoff_ms(2, None), 2 * BACKOFF_BASE_MS); + assert_eq!(failure_backoff_ms(3, None), 4 * BACKOFF_BASE_MS); + assert_eq!(failure_backoff_ms(7, None), 64 * BACKOFF_BASE_MS); + assert_eq!(failure_backoff_ms(8, None), BACKOFF_MAX_MS); + assert_eq!(failure_backoff_ms(u32::MAX, None), BACKOFF_MAX_MS); + // A longer Retry-After wins, a shorter one does not shorten the backoff. + assert_eq!( + failure_backoff_ms(1, Some(Duration::from_secs(3 * 60 * 60))), + 3 * 60 * 60 * 1000 + ); + assert_eq!( + failure_backoff_ms(2, Some(Duration::from_secs(1))), + 2 * BACKOFF_BASE_MS + ); + assert_eq!( + failure_backoff_ms(1, Some(Duration::from_secs(u64::MAX))), + BACKOFF_MAX_MS + ); +} + +#[test] +fn regular_checks_are_hourly_with_a_stable_spread() { + let podcast = PodcastId::from_feed_url(&feed_url("https://feeds.example/a.xml")); + let other = PodcastId::from_feed_url(&feed_url("https://feeds.example/b.xml")); + let now = 1_000_000; + let next = next_regular_check_ms(now, 7, &podcast); + assert!( + (now + REFRESH_INTERVAL_MS..now + REFRESH_INTERVAL_MS + 10 * 60 * 1000).contains(&next) + ); + assert_eq!(next, next_regular_check_ms(now, 7, &podcast)); + let spread: std::collections::BTreeSet = (0..20) + .map(|user| next_regular_check_ms(now, user, &podcast)) + .chain([next_regular_check_ms(now, 7, &other)]) + .collect(); + assert!(spread.len() > 10); +} + +/// A port on which nothing listens. +fn closed_port() -> u16 { + let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap(); + listener.local_addr().unwrap().port() +} + +#[tokio::test] +async fn internal_names_cannot_be_told_from_unreachable_ones() { + // A name that resolves to a private address, a name that does not + // resolve and a server that refuses the connection look the same to + // the user: otherwise "add RSS" would tell which names exist in the + // server's internal DNS. Only what the URL itself says (an IP literal, + // the port, the scheme) is refused as such. + let port = closed_port(); + let lookup = ScriptedLookup::with(&[ + ("vault.internal.test", &[&["10.20.30.40"]]), + ("closed.test", &[&["127.0.0.1"]]), + ]); + let fetcher = fetcher(lookup); + let mut answers = Vec::new(); + for text in [ + format!("http://vault.internal.test:{port}/feed"), + format!("http://nowhere.internal.test:{port}/feed"), + format!("http://closed.test:{port}/feed"), + ] { + let error = fetcher + .fetch(&feed_url(&text), &Validators::default()) + .await + .unwrap_err(); + assert!(!error.public_message().contains("internal"), "{text}"); + answers.push((error.public_message(), error.kind)); + } + assert_eq!(answers[0].1, FetchErrorKind::Transport, "{answers:?}"); + assert!( + answers.iter().all(|answer| *answer == answers[0]), + "{answers:?}" + ); + let literal = fetcher + .fetch(&feed_url("http://10.20.30.40/feed"), &Validators::default()) + .await + .unwrap_err(); + assert_eq!(literal.kind, FetchErrorKind::Refused); + assert_ne!(literal.public_message(), answers[0].0); +} diff --git a/src/podcasts/http.rs b/src/podcasts/http.rs new file mode 100644 index 0000000..669d44a --- /dev/null +++ b/src/podcasts/http.rs @@ -0,0 +1,980 @@ +//! HTTP endpoints of the podcast runtime, mounted at `/api/podcasts`. The +//! request handling lives in [`super::api`]; this module authenticates (the +//! session or a bearer token of the signed-in account) and maps results to +//! JSON responses. +//! +//! Podcast UI: +//! +//! * `GET /feeds[?all=1]` — subscribed podcasts (with `all`, also +//! unsubscribed ones and podcasts only known from synced progress), with +//! counters and the local fetch state. +//! * `POST /feeds` `{url}` — "add RSS": fetch, then subscribe (private). +//! * `GET /feeds/{podcast_id}` — one podcast and its season groups. +//! * `GET /feeds/{podcast_id}/episodes?season=&played=&q=&sort=&limit=&offset=` +//! * `POST /feeds/{podcast_id}/subscribe`, `POST /feeds/{podcast_id}/unsubscribe` +//! (unsubscribing keeps progress and history and revokes sharing). +//! * `POST /feeds/{podcast_id}/share` `{shared, confirm?}` — "share in the +//! federation" on or off; 409 with `needs_confirmation` when the feed URLs +//! look private and the request does not confirm exactly them. +//! * `POST /discovery/preview` `{podcast_id, feed_url}` — seasons and +//! episodes of a podcast found in the public search, read from its feed +//! (nothing stored). +//! * `POST /discovery/subscribe` `{podcast_id, feed_url}` — subscribe +//! (private) to a found podcast with its origin feed URL; the id must hash +//! from the URL (422 otherwise). +//! * `POST /episodes/{episode_id}/restart|played|unplayed` — explicit +//! progress actions. +//! * `POST /{podcast_id}/refresh` — manual feed refresh of a subscribed +//! podcast (ignores the failure backoff, at most every 30 s). +//! +//! Web player: +//! +//! * `POST /episodes/{episode_id}/playback` — audio source, resume position +//! and (when playing starts) a new listening session. +//! * `POST /episodes/{episode_id}/progress` — checkpoints, pause, seek, +//! stop, handoff and end of stream of that session (also sent with +//! `navigator.sendBeacon` when the page is hidden). +//! * `GET /episodes/{episode_id}/source` — how the browser gets the audio: +//! the publisher URL (default, the browser streams it directly) and the +//! proxy URL when the administrator enabled the proxy. +//! * `GET /episodes/{episode_id}/audio` — the optional audio proxy +//! (`podcast_audio_proxy_enabled`). It takes an episode id, never a URL, +//! and only serves episodes of the signed-in account. + +use std::sync::Arc; + +use cot::db::Database; +use cot::http::StatusCode; +use cot::http::header::{ + ACCEPT_RANGES, CACHE_CONTROL, CONTENT_LENGTH, CONTENT_RANGE, CONTENT_TYPE, RANGE, RETRY_AFTER, +}; +use cot::json::Json; +use cot::request::extractors::{Path, UrlQuery}; +use cot::response::IntoResponse; +use cot::router::method::{get, post}; +use cot::router::{Route, Router}; +use cot::session::Session; +use cot::{App, Body}; +use futures_util::TryStreamExt; +use music_dht::podcast::{EpisodeId, PodcastId}; +use serde::{Deserialize, Serialize}; + +use super::api::{self, ApiError, EpisodeAction, EpisodeListParams}; +use super::discovery::Discovery; +use super::fetch::FeedFetcher; +use super::limits::{self, ConcurrencyLimit, Permit}; +use super::proxy::AudioProxy; +use super::runner::{self, ManualRefreshError, TargetResult}; +use super::{PodcastStore, now_ms}; +use crate::auth; +use crate::config::AppConfig; + +fn json_error(status: StatusCode, message: &str) -> cot::response::Response { + let body = serde_json::json!({ "error": message }); + cot::http::Response::builder() + .status(status) + .header(CONTENT_TYPE, "application/json") + .header(CACHE_CONTROL, "private, no-store") + .body(Body::fixed(body.to_string())) + .expect("valid response") +} + +fn internal(error: impl std::fmt::Display) -> cot::Error { + cot::Error::internal(error.to_string()) +} + +fn api_error(error: ApiError) -> cot::response::Response { + json_error( + StatusCode::from_u16(error.status).unwrap_or(StatusCode::INTERNAL_SERVER_ERROR), + &error.message, + ) +} + +/// JSON response of an [`api`] result; personal data is never cached. +fn api_response(result: api::ApiResult) -> cot::Result { + match result { + Ok(value) => { + let mut response = Json(value).into_response()?; + response + .headers_mut() + .insert(CACHE_CONTROL, "private, no-store".parse().expect("static")); + Ok(response) + } + Err(error) => Ok(api_error(error)), + } +} + +/// The signed-in account (session or bearer token). +async fn signed_in(auth_ctx: &auth::AuthContext, session: &Session, db: &Database) -> Option { + auth::get_request_user(auth_ctx, session, db) + .await + .map(|user| user.id) +} + +fn unauthorized() -> cot::response::Response { + json_error(StatusCode::UNAUTHORIZED, "not authenticated") +} + +/// 429 when an account or the server has too many requests of a kind in +/// flight (see [`limits`]). +fn too_busy(message: &str) -> cot::response::Response { + let mut response = json_error(StatusCode::TOO_MANY_REQUESTS, message); + response.headers_mut().insert( + RETRY_AFTER, + limits::RETRY_AFTER_SECS + .to_string() + .parse() + .expect("digits"), + ); + response +} + +const FEEDS_BUSY: &str = "too many feed requests in progress; try again in a few seconds"; + +/// Largest JSON body accepted (a playback request carries one episode reference). +const MAX_JSON_BODY: usize = 64 * 1024; + +/// Whether a `Content-Type` value is `application/json` (parameters such +/// as `charset` allowed). +fn is_json_content_type(value: Option<&str>) -> bool { + value + .and_then(|value| value.split(';').next()) + .is_some_and(|essence| essence.trim().eq_ignore_ascii_case("application/json")) +} + +/// Request body as JSON (`{}` when empty); malformed JSON is a 400. +/// +/// A body must be sent as `application/json` (415 otherwise), like cot's +/// `Json` extractor requires on the other player endpoints: a cross-origin +/// page can only send that type after a CORS preflight, while `text/plain` +/// and form bodies need none and would carry the session cookie from a +/// same-site page. `navigator.sendBeacon` sends a Blob typed +/// `application/json`. +async fn json_body(request: cot::request::Request) -> Result { + let json = is_json_content_type( + request + .headers() + .get(CONTENT_TYPE) + .and_then(|value| value.to_str().ok()), + ); + let bytes = request + .into_body() + .into_bytes_limited(MAX_JSON_BODY) + .await + .map_err(|error| ApiError::new(400, format!("unreadable body: {error}")))?; + if bytes.iter().all(u8::is_ascii_whitespace) { + return Ok(serde_json::Value::Object(Default::default())); + } + if !json { + return Err(ApiError::new(415, "the body must be application/json")); + } + serde_json::from_slice(&bytes).map_err(|error| ApiError::new(400, error.to_string())) +} + +/// Shared state of the handlers. +#[derive(Clone)] +struct State { + config: Arc, + pool: Arc>, + fetcher: Arc>, + proxy: Arc>, + /// Feed downloads users started ("add RSS", manual refresh). + feed_fetches: ConcurrencyLimit, + /// Audio proxy responses being streamed. + proxy_streams: ConcurrencyLimit, +} + +impl State { + fn feed_fetch(&self, user_id: i64) -> Option { + self.feed_fetches.try_acquire(user_id) + } + + async fn store(&self) -> cot::Result { + let pool = self + .pool + .get_or_try_init(|| async { + sqlx::postgres::PgPoolOptions::new() + .max_connections(5) + .connect(&self.config.database_url) + .await + }) + .await + .map_err(internal)?; + Ok(PodcastStore::new(pool.clone())) + } + + async fn fetcher(&self) -> cot::Result<&FeedFetcher> { + self.fetcher + .get_or_try_init(|| async { FeedFetcher::new() }) + .await + .map_err(internal) + } + + async fn proxy(&self) -> cot::Result<&AudioProxy> { + self.proxy + .get_or_try_init(|| async { AudioProxy::new() }) + .await + .map_err(internal) + } +} + +/// The podcast runtime endpoints (see the module documentation). +pub(crate) struct PodcastsApp { + state: State, +} + +impl PodcastsApp { + pub(crate) fn new(config: Arc) -> Self { + Self { + state: State { + config, + pool: Arc::new(tokio::sync::OnceCell::new()), + fetcher: Arc::new(tokio::sync::OnceCell::new()), + proxy: Arc::new(tokio::sync::OnceCell::new()), + feed_fetches: ConcurrencyLimit::new( + limits::FEED_FETCHES_PER_ACCOUNT, + limits::FEED_FETCHES_TOTAL, + ), + proxy_streams: ConcurrencyLimit::new( + limits::PROXY_STREAMS_PER_ACCOUNT, + limits::PROXY_STREAMS_TOTAL, + ), + }, + } + } +} + +#[derive(Debug, Deserialize)] +struct PodcastPath { + podcast_id: String, +} + +#[derive(Debug, Deserialize)] +struct EpisodePath { + episode_id: String, +} + +/// Path of the proxy endpoint for an episode. +pub(crate) fn proxy_path(episode_id: &EpisodeId) -> String { + format!("/api/podcasts/episodes/{episode_id}/audio") +} + +#[derive(Debug, Serialize)] +struct RefreshResponse { + status: &'static str, + new_episodes: usize, + updated_episodes: usize, + unchanged_episodes: usize, + skipped_items: usize, + feed_url: Option, + error: Option, + next_check_at_ms: Option, +} + +impl RefreshResponse { + fn from_result(result: TargetResult) -> Self { + let mut response = Self { + status: "", + new_episodes: 0, + updated_episodes: 0, + unchanged_episodes: 0, + skipped_items: 0, + feed_url: None, + error: None, + next_check_at_ms: None, + }; + match result { + TargetResult::Updated(outcome) => { + response.status = "updated"; + response.new_episodes = outcome.new_episodes; + response.updated_episodes = outcome.updated_episodes; + response.unchanged_episodes = outcome.unchanged_episodes; + response.skipped_items = outcome.skipped_items; + response.feed_url = outcome.feed_url.map(String::from); + } + TargetResult::NotModified => response.status = "not_modified", + TargetResult::Failed { + error, + next_check_at_ms, + .. + } => { + response.status = "failed"; + response.error = Some(error); + response.next_check_at_ms = Some(next_check_at_ms); + } + TargetResult::Skipped(reason) => { + response.status = "skipped"; + response.error = Some(reason); + } + } + response + } +} + +async fn refresh_handler( + state: State, + auth_ctx: auth::AuthContext, + session: Session, + db: Database, + path: Path, +) -> cot::Result { + let Some(user) = auth::get_request_user(&auth_ctx, &session, &db).await else { + return Ok(json_error(StatusCode::UNAUTHORIZED, "not authenticated")); + }; + let Ok(podcast_id) = PodcastId::parse(&path.0.podcast_id) else { + return Ok(json_error(StatusCode::BAD_REQUEST, "invalid podcast id")); + }; + let Some(_permit) = state.feed_fetch(user.id) else { + return Ok(too_busy(FEEDS_BUSY)); + }; + let store = state.store().await?; + let fetcher = state.fetcher().await?; + match runner::refresh_now(&store, fetcher, user.id, &podcast_id, now_ms()).await { + Ok(result) => Json(RefreshResponse::from_result(result)).into_response(), + Err(error) => match error.downcast_ref::() { + Some(ManualRefreshError::UnknownPodcast) => { + Ok(json_error(StatusCode::NOT_FOUND, "unknown podcast")) + } + Some(ManualRefreshError::NotSubscribed) => { + Ok(json_error(StatusCode::CONFLICT, &error.to_string())) + } + Some(ManualRefreshError::TooSoon { retry_after_ms }) => { + let mut response = json_error(StatusCode::TOO_MANY_REQUESTS, &error.to_string()); + response.headers_mut().insert( + RETRY_AFTER, + ((retry_after_ms + 999) / 1000) + .to_string() + .parse() + .expect("digits"), + ); + Ok(response) + } + None => Err(internal(format!("{error:#}"))), + }, + } +} + +#[derive(Debug, Serialize)] +struct SourceResponse { + episode_id: String, + /// Publisher URL; the browser streams it directly by default. + direct_url: String, + /// Same-origin proxy path, only when the proxy is enabled. + proxy_url: Option, + mime: Option, +} + +async fn source_handler( + state: State, + auth_ctx: auth::AuthContext, + session: Session, + db: Database, + path: Path, +) -> cot::Result { + let Some(user) = auth::get_request_user(&auth_ctx, &session, &db).await else { + return Ok(json_error(StatusCode::UNAUTHORIZED, "not authenticated")); + }; + let Ok(episode_id) = EpisodeId::parse(&path.0.episode_id) else { + return Ok(json_error(StatusCode::BAD_REQUEST, "invalid episode id")); + }; + let store = state.store().await?; + let Some((audio_url, mime)) = store + .audio_source(user.id, &episode_id) + .await + .map_err(internal)? + else { + return Ok(json_error(StatusCode::NOT_FOUND, "unknown episode")); + }; + let (config, _) = AppConfig::load_with_db(&db).await; + let mut response = Json(SourceResponse { + proxy_url: config + .podcast_audio_proxy_enabled + .then(|| proxy_path(&episode_id)), + episode_id: episode_id.to_string(), + direct_url: audio_url, + mime, + }) + .into_response()?; + response + .headers_mut() + .insert(CACHE_CONTROL, "private, no-store".parse().expect("static")); + Ok(response) +} + +async fn audio_handler( + state: State, + auth_ctx: auth::AuthContext, + session: Session, + db: Database, + path: Path, + request: cot::request::Request, +) -> cot::Result { + let Some(user) = auth::get_request_user(&auth_ctx, &session, &db).await else { + return Ok(json_error(StatusCode::UNAUTHORIZED, "not authenticated")); + }; + let (config, _) = AppConfig::load_with_db(&db).await; + if !config.podcast_audio_proxy_enabled { + return Ok(json_error( + StatusCode::NOT_FOUND, + "the podcast audio proxy is disabled; stream from the publisher URL", + )); + } + let Ok(episode_id) = EpisodeId::parse(&path.0.episode_id) else { + return Ok(json_error(StatusCode::BAD_REQUEST, "invalid episode id")); + }; + let store = state.store().await?; + // The URL comes from this account's catalog, never from the request. + let Some((audio_url, _)) = store + .audio_source(user.id, &episode_id) + .await + .map_err(internal)? + else { + return Ok(json_error(StatusCode::NOT_FOUND, "unknown episode")); + }; + let range = request + .headers() + .get(RANGE) + .and_then(|value| value.to_str().ok()); + // Held until the body is fully sent or the browser goes away. + let Some(permit) = state.proxy_streams.try_acquire(user.id) else { + return Ok(too_busy("too many podcast audio streams in progress")); + }; + let proxy = state.proxy().await?; + let audio = match proxy.fetch(&audio_url, range).await { + Ok(audio) => audio, + Err(error) => { + // Neither the URL (access tokens) nor the account is logged. + tracing::info!(episode = %episode_id, "podcast proxy: {error}"); + let status = + StatusCode::from_u16(error.http_status()).unwrap_or(StatusCode::BAD_GATEWAY); + return Ok(json_error(status, &error.public_message())); + } + }; + let mut builder = cot::http::Response::builder() + .status(StatusCode::from_u16(audio.status).unwrap_or(StatusCode::BAD_GATEWAY)) + .header(CACHE_CONTROL, "private, no-store") + .header("X-Content-Type-Options", "nosniff") + .header("Content-Security-Policy", "default-src 'none'; sandbox") + .header("Content-Disposition", "inline"); + if let Some(content_type) = &audio.content_type { + builder = builder.header(CONTENT_TYPE, content_type.as_str()); + } + if let Some(length) = audio.content_length { + builder = builder.header(CONTENT_LENGTH, length.to_string()); + } + if let Some(content_range) = &audio.content_range { + builder = builder.header(CONTENT_RANGE, content_range.as_str()); + } + if audio.accept_ranges { + builder = builder.header(ACCEPT_RANGES, "bytes"); + } + let body = audio.body.map_err(move |error| { + let _held = &permit; + cot::Error::internal(error.to_string()) + }); + builder.body(Body::streaming(body)).map_err(internal) +} + +#[derive(Debug, Deserialize)] +struct ListQuery { + #[serde(default)] + all: Option, +} + +async fn list_handler( + state: State, + auth_ctx: auth::AuthContext, + session: Session, + db: Database, + query: UrlQuery, +) -> cot::Result { + let Some(user_id) = signed_in(&auth_ctx, &session, &db).await else { + return Ok(unauthorized()); + }; + let all = matches!(query.0.all.as_deref(), Some("1" | "true")); + let store = state.store().await?; + api_response(api::list_podcasts(&store, user_id, all).await) +} + +async fn subscribe_url_handler( + state: State, + auth_ctx: auth::AuthContext, + session: Session, + db: Database, + request: cot::request::Request, +) -> cot::Result { + let Some(user_id) = signed_in(&auth_ctx, &session, &db).await else { + return Ok(unauthorized()); + }; + let body = match json_body(request).await { + Ok(body) => body, + Err(error) => return Ok(api_error(error)), + }; + let Some(_permit) = state.feed_fetch(user_id) else { + return Ok(too_busy(FEEDS_BUSY)); + }; + let store = state.store().await?; + let fetcher = state.fetcher().await?; + api_response(api::subscribe_url(&store, fetcher, user_id, body, now_ms()).await) +} + +/// What a `/discovery/...` route does with a found podcast. +#[derive(Debug, Clone, Copy)] +enum FoundRoute { + Preview, + Subscribe, +} + +/// A podcast from a public search hit: read its feed for a preview, or +/// subscribe. Both fetch the feed, so they share the feed request limits. +async fn found_handler( + state: State, + route: FoundRoute, + auth_ctx: auth::AuthContext, + session: Session, + db: Database, + request: cot::request::Request, +) -> cot::Result { + let Some(user_id) = signed_in(&auth_ctx, &session, &db).await else { + return Ok(unauthorized()); + }; + let body = match json_body(request).await { + Ok(body) => body, + Err(error) => return Ok(api_error(error)), + }; + let Some(_permit) = state.feed_fetch(user_id) else { + return Ok(too_busy(FEEDS_BUSY)); + }; + let store = state.store().await?; + let fetcher = state.fetcher().await?; + match route { + FoundRoute::Preview => { + api_response(api::preview_found(&store, fetcher, user_id, body).await) + } + FoundRoute::Subscribe => { + api_response(api::subscribe_found(&store, fetcher, user_id, body, now_ms()).await) + } + } +} + +/// Whether public discovery runs on this server's federation node. +fn discovery_state<'a>( + federation: &crate::federation::Federation, + dht: Option<&'a music_dht::podcast_dht::PodcastDht>, +) -> Discovery<'a> { + match dht { + Some(dht) => Discovery::Running(dht), + None if federation.podcast_discovery_unavailable() => Discovery::Unavailable, + None => Discovery::Off, + } +} + +/// What a `/feeds/{podcast_id}...` route does. +#[derive(Debug, Clone, Copy)] +enum PodcastRoute { + Detail, + Subscribe, + Unsubscribe, +} + +async fn podcast_handler( + state: State, + route: PodcastRoute, + auth_ctx: auth::AuthContext, + session: Session, + db: Database, + path: Path, +) -> cot::Result { + let Some(user_id) = signed_in(&auth_ctx, &session, &db).await else { + return Ok(unauthorized()); + }; + let podcast_id = match api::parse_podcast_id(&path.0.podcast_id) { + Ok(id) => id, + Err(error) => return Ok(api_error(error)), + }; + let store = state.store().await?; + match route { + PodcastRoute::Detail => { + let federation = crate::federation::handle(); + let dht = federation.podcast_discovery(); + let discovery = discovery_state(&federation, dht.as_deref()); + api_response(api::podcast_detail(&store, user_id, &podcast_id, discovery).await) + } + PodcastRoute::Subscribe => { + api_response(api::resubscribe(&store, user_id, &podcast_id).await) + } + PodcastRoute::Unsubscribe => { + api_response(api::unsubscribe(&store, user_id, &podcast_id).await) + } + } +} + +async fn share_handler( + state: State, + auth_ctx: auth::AuthContext, + session: Session, + db: Database, + path: Path, + request: cot::request::Request, +) -> cot::Result { + let Some(user_id) = signed_in(&auth_ctx, &session, &db).await else { + return Ok(unauthorized()); + }; + let podcast_id = match api::parse_podcast_id(&path.0.podcast_id) { + Ok(id) => id, + Err(error) => return Ok(api_error(error)), + }; + // A JSON body is required (415 otherwise): the toggle cannot be sent by + // a cross-origin form or `text/plain` request without a CORS preflight. + let body = match json_body(request).await { + Ok(serde_json::Value::Object(body)) if !body.is_empty() => serde_json::Value::Object(body), + Ok(_) => { + return Ok(json_error( + StatusCode::BAD_REQUEST, + "expected {\"shared\": bool}", + )); + } + Err(error) => return Ok(api_error(error)), + }; + let store = state.store().await?; + let federation = crate::federation::handle(); + let dht = federation.podcast_discovery(); + match api::set_share( + &store, + user_id, + &podcast_id, + body, + discovery_state(&federation, dht.as_deref()), + now_ms(), + ) + .await + { + Ok(api::ShareOutcome::Done(response)) => api_response(Ok(*response)), + Ok(api::ShareOutcome::NeedsConfirmation(share)) => { + let body = serde_json::json!({ + "error": "the feed URL may contain private data; confirm publishing it", + "needs_confirmation": true, + "share": share, + }); + Ok(cot::http::Response::builder() + .status(StatusCode::CONFLICT) + .header(CONTENT_TYPE, "application/json") + .header(CACHE_CONTROL, "private, no-store") + .body(Body::fixed(body.to_string())) + .map_err(internal)?) + } + Err(error) => Ok(api_error(error)), + } +} + +async fn episodes_handler( + state: State, + auth_ctx: auth::AuthContext, + session: Session, + db: Database, + path: Path, + query: UrlQuery, +) -> cot::Result { + let Some(user_id) = signed_in(&auth_ctx, &session, &db).await else { + return Ok(unauthorized()); + }; + let podcast_id = match api::parse_podcast_id(&path.0.podcast_id) { + Ok(id) => id, + Err(error) => return Ok(api_error(error)), + }; + let store = state.store().await?; + api_response(api::episodes(&store, user_id, &podcast_id, &query.0).await) +} + +/// What a `/episodes/{episode_id}/...` POST route does. +#[derive(Debug, Clone, Copy)] +enum EpisodeRoute { + Action(EpisodeAction), + Playback, + Progress, +} + +async fn episode_handler( + state: State, + route: EpisodeRoute, + auth_ctx: auth::AuthContext, + session: Session, + db: Database, + path: Path, + request: cot::request::Request, +) -> cot::Result { + let Some(user_id) = signed_in(&auth_ctx, &session, &db).await else { + return Ok(unauthorized()); + }; + let episode_id = match api::parse_episode_id(&path.0.episode_id) { + Ok(id) => id, + Err(error) => return Ok(api_error(error)), + }; + let store = state.store().await?; + match route { + EpisodeRoute::Action(action) => { + api_response(api::episode_action(&store, user_id, &episode_id, action).await) + } + EpisodeRoute::Playback => { + let body = match json_body(request).await { + Ok(body) => body, + Err(error) => return Ok(api_error(error)), + }; + let (config, _) = AppConfig::load_with_db(&db).await; + api_response( + api::playback( + &store, + user_id, + &episode_id, + body, + config.podcast_audio_proxy_enabled, + ) + .await, + ) + } + EpisodeRoute::Progress => { + let body = match json_body(request).await { + Ok(body) => body, + Err(error) => return Ok(api_error(error)), + }; + api_response(api::progress(&store, user_id, &episode_id, body, now_ms()).await) + } + } +} + +impl App for PodcastsApp { + fn name(&self) -> &'static str { + "podcasts" + } + + fn router(&self) -> Router { + let refresh_state = self.state.clone(); + let source_state = self.state.clone(); + let audio_state = self.state.clone(); + let list_state = self.state.clone(); + let subscribe_state = self.state.clone(); + let episodes_state = self.state.clone(); + let share_state = self.state.clone(); + let found_route = |route: FoundRoute| { + let state = self.state.clone(); + move |auth_ctx: auth::AuthContext, + session: Session, + db: Database, + request: cot::request::Request| { + found_handler(state.clone(), route, auth_ctx, session, db, request) + } + }; + let podcast_route = |route: PodcastRoute| { + let state = self.state.clone(); + move |auth_ctx: auth::AuthContext, + session: Session, + db: Database, + path: Path| { + podcast_handler(state.clone(), route, auth_ctx, session, db, path) + } + }; + let episode_route = |route: EpisodeRoute| { + let state = self.state.clone(); + move |auth_ctx: auth::AuthContext, + session: Session, + db: Database, + path: Path, + request: cot::request::Request| { + episode_handler(state.clone(), route, auth_ctx, session, db, path, request) + } + }; + // Routes are matched in order: literal prefixes come before + // `/{podcast_id}/refresh`. + Router::with_urls([ + Route::with_handler_and_name( + "/feeds", + get( + move |auth_ctx: auth::AuthContext, + session: Session, + db: Database, + query: UrlQuery| { + list_handler(list_state.clone(), auth_ctx, session, db, query) + }, + ) + .post( + move |auth_ctx: auth::AuthContext, + session: Session, + db: Database, + request: cot::request::Request| { + subscribe_url_handler( + subscribe_state.clone(), + auth_ctx, + session, + db, + request, + ) + }, + ), + "podcasts_feeds", + ), + Route::with_handler_and_name( + "/discovery/preview", + post(found_route(FoundRoute::Preview)), + "podcasts_discovery_preview", + ), + Route::with_handler_and_name( + "/discovery/subscribe", + post(found_route(FoundRoute::Subscribe)), + "podcasts_discovery_subscribe", + ), + Route::with_handler_and_name( + "/feeds/{podcast_id}/episodes", + get( + move |auth_ctx: auth::AuthContext, + session: Session, + db: Database, + path: Path, + query: UrlQuery| { + episodes_handler(episodes_state.clone(), auth_ctx, session, db, path, query) + }, + ), + "podcasts_episodes", + ), + Route::with_handler_and_name( + "/feeds/{podcast_id}/subscribe", + post(podcast_route(PodcastRoute::Subscribe)), + "podcasts_subscribe", + ), + Route::with_handler_and_name( + "/feeds/{podcast_id}/unsubscribe", + post(podcast_route(PodcastRoute::Unsubscribe)), + "podcasts_unsubscribe", + ), + Route::with_handler_and_name( + "/feeds/{podcast_id}/share", + post( + move |auth_ctx: auth::AuthContext, + session: Session, + db: Database, + path: Path, + request: cot::request::Request| { + share_handler(share_state.clone(), auth_ctx, session, db, path, request) + }, + ), + "podcasts_share", + ), + Route::with_handler_and_name( + "/feeds/{podcast_id}", + get(podcast_route(PodcastRoute::Detail)), + "podcasts_detail", + ), + Route::with_handler_and_name( + "/episodes/{episode_id}/playback", + post(episode_route(EpisodeRoute::Playback)), + "podcasts_episode_playback", + ), + Route::with_handler_and_name( + "/episodes/{episode_id}/progress", + post(episode_route(EpisodeRoute::Progress)), + "podcasts_episode_progress", + ), + Route::with_handler_and_name( + "/episodes/{episode_id}/restart", + post(episode_route(EpisodeRoute::Action(EpisodeAction::Restart))), + "podcasts_episode_restart", + ), + Route::with_handler_and_name( + "/episodes/{episode_id}/played", + post(episode_route(EpisodeRoute::Action( + EpisodeAction::MarkPlayed, + ))), + "podcasts_episode_played", + ), + Route::with_handler_and_name( + "/episodes/{episode_id}/unplayed", + post(episode_route(EpisodeRoute::Action( + EpisodeAction::MarkUnplayed, + ))), + "podcasts_episode_unplayed", + ), + Route::with_handler_and_name( + "/episodes/{episode_id}/source", + get( + move |auth_ctx: auth::AuthContext, + session: Session, + db: Database, + path: Path| { + source_handler(source_state.clone(), auth_ctx, session, db, path) + }, + ), + "podcasts_episode_source", + ), + Route::with_handler_and_name( + "/episodes/{episode_id}/audio", + get( + move |auth_ctx: auth::AuthContext, + session: Session, + db: Database, + path: Path, + request: cot::request::Request| { + audio_handler(audio_state.clone(), auth_ctx, session, db, path, request) + }, + ), + "podcasts_episode_audio", + ), + Route::with_handler_and_name( + "/{podcast_id}/refresh", + post( + move |auth_ctx: auth::AuthContext, + session: Session, + db: Database, + path: Path| { + refresh_handler(refresh_state.clone(), auth_ctx, session, db, path) + }, + ), + "podcasts_refresh", + ), + ]) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn request(content_type: Option<&str>, body: &'static str) -> cot::request::Request { + let mut builder = cot::http::Request::builder() + .method("POST") + .uri("/api/podcasts/feeds"); + if let Some(value) = content_type { + builder = builder.header(CONTENT_TYPE, value); + } + builder.body(Body::fixed(body)).unwrap() + } + + #[tokio::test] + async fn json_bodies_need_the_json_content_type() { + let url = r#"{"url": "https://podcasts.example/feed.xml"}"#; + for content_type in [ + "application/json", + "application/json; charset=UTF-8", + "Application/JSON", + ] { + let body = json_body(request(Some(content_type), url)).await.unwrap(); + assert_eq!(body["url"], "https://podcasts.example/feed.xml"); + } + // What a page can send cross-origin without a CORS preflight. + for content_type in [ + Some("text/plain"), + Some("text/plain;charset=UTF-8"), + Some("application/x-www-form-urlencoded"), + Some("multipart/form-data; boundary=x"), + None, + ] { + let error = json_body(request(content_type, url)).await.unwrap_err(); + assert_eq!(error.status, 415, "{content_type:?}"); + } + // An empty body (a plain POST) is an empty object whatever the type. + let empty = json_body(request(None, "")).await.unwrap(); + assert_eq!(empty, serde_json::json!({})); + let malformed = json_body(request(Some("application/json"), "{")) + .await + .unwrap_err(); + assert_eq!(malformed.status, 400); + } +} diff --git a/src/podcasts/limits.rs b/src/podcasts/limits.rs new file mode 100644 index 0000000..d894a8b --- /dev/null +++ b/src/podcasts/limits.rs @@ -0,0 +1,123 @@ +//! Bounds on the outbound work signed-in users can start: feed downloads +//! they ask for ("add RSS", manual refresh) and audio proxy streams. Each +//! kind has a per-account and a server-wide limit on requests in flight, so +//! one account cannot tie up the server's memory, connections or bandwidth. +//! Scheduled refreshes are bounded by the job itself (`runner::RUN_LIMIT`, +//! `runner::RUN_CONCURRENCY`, accounts in turns). + +use std::collections::HashMap; +use std::sync::{Arc, Mutex}; + +/// Feed downloads one account may run at once ("add RSS", refresh). +pub(crate) const FEED_FETCHES_PER_ACCOUNT: usize = 2; +/// Feed downloads started by users, server-wide (each may buffer up to +/// `fetch::MAX_FEED_BYTES`). +pub(crate) const FEED_FETCHES_TOTAL: usize = 16; +/// Audio proxy responses one account may stream at once (a player opens a +/// new range request when seeking while the old one is still closing). +pub(crate) const PROXY_STREAMS_PER_ACCOUNT: usize = 4; +/// Audio proxy responses streamed at once, server-wide. +pub(crate) const PROXY_STREAMS_TOTAL: usize = 64; +/// Seconds a refused client is asked to wait (`Retry-After`). +pub(crate) const RETRY_AFTER_SECS: u64 = 5; + +#[derive(Debug, Default)] +struct Counts { + total: usize, + accounts: HashMap, +} + +/// Requests in flight per account and in total. +#[derive(Debug, Clone)] +pub(crate) struct ConcurrencyLimit { + per_account: usize, + total: usize, + counts: Arc>, +} + +impl ConcurrencyLimit { + pub(crate) fn new(per_account: usize, total: usize) -> Self { + Self { + per_account, + total, + counts: Arc::default(), + } + } + + /// A slot for one request of `user_id`, released when the permit is + /// dropped; `None` when the account or the server is at its limit. + pub(crate) fn try_acquire(&self, user_id: i64) -> Option { + let mut counts = self + .counts + .lock() + .unwrap_or_else(|error| error.into_inner()); + let account = counts.accounts.get(&user_id).copied().unwrap_or(0); + if account >= self.per_account || counts.total >= self.total { + return None; + } + counts.total += 1; + counts.accounts.insert(user_id, account + 1); + Some(Permit { + user_id, + counts: Arc::clone(&self.counts), + }) + } + + #[cfg(test)] + fn in_flight(&self, user_id: i64) -> (usize, usize) { + let counts = self.counts.lock().unwrap(); + ( + counts.accounts.get(&user_id).copied().unwrap_or(0), + counts.total, + ) + } +} + +/// One request in flight (see [`ConcurrencyLimit::try_acquire`]). +#[derive(Debug)] +pub(crate) struct Permit { + user_id: i64, + counts: Arc>, +} + +impl Drop for Permit { + fn drop(&mut self) { + let mut counts = self + .counts + .lock() + .unwrap_or_else(|error| error.into_inner()); + counts.total = counts.total.saturating_sub(1); + if let Some(account) = counts.accounts.get_mut(&self.user_id) { + *account = account.saturating_sub(1); + if *account == 0 { + counts.accounts.remove(&self.user_id); + } + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn accounts_and_the_server_are_bounded_until_permits_are_dropped() { + let limit = ConcurrencyLimit::new(2, 3); + let a1 = limit.try_acquire(1).unwrap(); + let a2 = limit.try_acquire(1).unwrap(); + assert!(limit.try_acquire(1).is_none(), "per account"); + let b1 = limit.try_acquire(2).unwrap(); + assert!(limit.try_acquire(3).is_none(), "server-wide"); + assert_eq!(limit.in_flight(1), (2, 3)); + drop(a1); + assert_eq!(limit.in_flight(1), (1, 2)); + let a3 = limit.try_acquire(1).unwrap(); + drop((a2, a3, b1)); + assert_eq!(limit.in_flight(1), (0, 0)); + assert!(limit.counts.lock().unwrap().accounts.is_empty()); + // Clones share the counts. + let clone = limit.clone(); + let _held = clone.try_acquire(4).unwrap(); + assert_eq!(limit.in_flight(4), (1, 1)); + } +} diff --git a/src/podcasts/mod.rs b/src/podcasts/mod.rs new file mode 100644 index 0000000..ebbc133 --- /dev/null +++ b/src/podcasts/mod.rs @@ -0,0 +1,341 @@ +//! PostgreSQL podcast store of the web server. +//! +//! [`PodcastStore`] implements the podcast contract of `music_dht::podcast` +//! (normative description: `PODCASTS.md` of frid) on PostgreSQL, with the +//! same invariants as the SQLite reference store of `furumi-library`. Each +//! web account is one virtual trusted device, so every table is keyed by +//! `user_id`. The store owns no HTTP, scheduling or UI: RSS fetching and the +//! player use it. +//! +//! # Tables +//! +//! | Table | Key | Content | Synced | +//! | --- | --- | --- | --- | +//! | `furumusic__podcast` | `podcast_id` | local channel metadata, `feed_url` = URL this account fetches | no (catalog) | +//! | `furumusic__podcast_feed_alias` | `(podcast_id, feed_url)` | bounded grow-only alias set | yes, inside subscription records and `podcast_feed_aliases_added` | +//! | `furumusic__podcast_episode` | `episode_id` | `EpisodeMeta`, `source` (`feed`/`sync`), `in_feed` | no (catalog) | +//! | `furumusic__podcast_season` (view) | `(podcast_id, season)` | derived from the episodes | no | +//! | `furumusic__podcast_subscription` | `podcast_id` | subscription register | yes | +//! | `furumusic__podcast_share` | `podcast_id` | share register | yes | +//! | `furumusic__podcast_progress` | `episode_id` | progress register | yes | +//! | `furumusic__podcast_listen_session` | `session_id` | grow-only set of sessions | yes | +//! | `furumusic__podcast_fetch_state` | `podcast_id` | ETag, Last-Modified, check times, errors, cover cache | **never** | +//! | `furumusic__podcast_share_confirmation` | `podcast_id` | privacy confirmation of a shared card (share decision, card URLs) | **never** | +//! +//! # One apply path +//! +//! Remote operations ([`PodcastStore::apply_remote_op`]), snapshot sections +//! ([`PodcastStore::apply_snapshot`]) and local actions all end in the same +//! functions: the `SyncOpWire` materializers of `music_dht::device_sync` and +//! the `merge_*` rules of `music_dht::podcast`. Applying is therefore +//! idempotent and independent of delivery order, and a snapshot never +//! replaces newer state. +//! +//! Local actions ([`PodcastStore::subscribe`], [`PodcastStore::seek`], ...) +//! run in one transaction: they read the registers, append the operation to +//! the account's device-sync log with its stamp +//! ([`crate::federation::devices::append_local_op`]: next `seq`, `hlc_ms`, +//! `op_id = :`) and apply exactly that operation. Either both +//! the log and the registers change or neither does. Every write +//! transaction takes a per-account advisory lock, so concurrent writers of +//! one account serialize. +//! +//! RSS refreshes ([`PodcastStore::apply_feed`]) touch the catalog and the +//! alias set only; they never write a register, progress or a session. +//! +//! # Runtime +//! +//! * `net` — the SSRF-guarded HTTP client used for every feed and audio +//! request (public addresses only, checked per connection and per +//! redirect hop). +//! * `fetch` and [`runner`] — conditional feed downloads and their schedule +//! (hourly checks, exponential backoff, one request per feed URL), run by +//! the `podcast_refresh` job and the manual refresh endpoint. +//! * `proxy` and [`http`] — the optional audio proxy and the +//! `/api/podcasts` endpoints. Audio is streamed from the publisher, never +//! stored. +//! * [`api`] — what the endpoints of the podcast UI and the web player do +//! (subscribe by URL, lists, explicit progress actions, sharing, playback +//! sessions and progress reports), independent of HTTP. +//! * [`discovery`] — public discovery: which podcasts the server node +//! publishes to `music_dht::podcast_dht` for the accounts that explicitly +//! shared them, the local privacy confirmations and the reconciler that +//! calls `PodcastDht::sync_shares`. +//! * [`search`] — public podcasts found by the federated search, as seen by +//! the searching account ("already subscribed"), and the preview of a +//! found podcast's feed before subscribing. + +mod actions; +pub(crate) mod api; +mod apply; +pub(crate) mod discovery; +mod fetch; +pub(crate) mod http; +mod limits; +mod net; +mod proxy; +mod queries; +mod refresh; +pub(crate) mod runner; +pub(crate) mod schema; +pub(crate) mod search; + +#[cfg(test)] +mod discovery_tests; +#[cfg(test)] +mod fetch_tests; +#[cfg(test)] +mod net_tests; +#[cfg(test)] +pub(crate) mod postgres_tests; +#[cfg(test)] +mod proxy_tests; +#[cfg(test)] +mod search_tests; +#[cfg(test)] +pub(crate) mod test_server; +#[cfg(test)] +mod tests; + +use std::fmt; + +use anyhow::Result; +use music_dht::podcast::{ + EpisodeId, FeedUrl, ListenSession, PodcastError, PodcastId, PodcastMeta, ProgressRecord, + ProgressStatus, +}; +use sqlx::Row; +use sqlx::postgres::PgRow; + +pub(crate) use actions::{CheckpointOutcome, ProgressSession}; +pub(crate) use apply::ApplyOutcome; +pub(crate) use fetch::FeedFetcher; +pub(crate) use queries::{ + EpisodeListItem, EpisodeQuery, EpisodeSort, PlayedFilter, PodcastListFilter, PodcastSummary, + SeasonFilter, SeasonSummary, +}; +pub(crate) use refresh::{FeedRefresh, FetchState, RefreshOutcome}; + +/// Typed errors of [`PodcastStore`]; other failures are database errors. +/// +/// They are returned inside `anyhow::Error`; use `downcast_ref` to match. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) enum PodcastStoreError { + /// The podcast is unknown to this account. + UnknownPodcast(PodcastId), + /// The action needs an active subscription. + NotSubscribed(PodcastId), + /// Input violates the podcast contract. + Invalid(String), + /// A new subscription would exceed [`MAX_SUBSCRIPTIONS_PER_ACCOUNT`]. + SubscriptionLimit(usize), + /// A new episode reference from the browser would exceed + /// [`MAX_REFERENCE_EPISODES_PER_ACCOUNT`]. + ReferenceLimit(usize), +} + +/// Active subscriptions an account may add locally (subscriptions synced +/// from the account's own devices are always applied). Every subscription +/// is a feed the server downloads about once an hour. +pub(crate) const MAX_SUBSCRIPTIONS_PER_ACCOUNT: usize = 1000; + +/// Episodes known only from references (`source = 'sync'`: not from one of +/// the account's feeds) beyond which the browser cannot add another one +/// through `POST /api/podcasts/episodes/{id}/playback`. References in +/// operations of the account's own devices are always applied; this bounds +/// what a signed-in user can insert directly, one request at a time. +pub(crate) const MAX_REFERENCE_EPISODES_PER_ACCOUNT: usize = 10_000; + +impl fmt::Display for PodcastStoreError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::UnknownPodcast(id) => write!(f, "unknown podcast {id}"), + Self::NotSubscribed(id) => write!(f, "not subscribed to podcast {id}"), + Self::Invalid(reason) => write!(f, "invalid podcast data: {reason}"), + Self::SubscriptionLimit(limit) => { + write!(f, "subscription limit reached ({limit} podcasts)") + } + Self::ReferenceLimit(limit) => write!( + f, + "too many episodes from outside your feeds ({limit}); subscribe to the podcast to play it" + ), + } + } +} + +impl std::error::Error for PodcastStoreError {} + +impl From for PodcastStoreError { + fn from(error: PodcastError) -> Self { + Self::Invalid(error.to_string()) + } +} + +pub(crate) fn invalid(reason: impl ToString) -> anyhow::Error { + PodcastStoreError::Invalid(reason.to_string()).into() +} + +/// PostgreSQL store of podcast catalog, personal state and fetch state. +/// +/// Cheap to clone (the pool is shared). See the module documentation. +#[derive(Debug, Clone)] +pub(crate) struct PodcastStore { + pool: sqlx::PgPool, +} + +impl PodcastStore { + pub(crate) fn new(pool: sqlx::PgPool) -> Self { + Self { pool } + } + + /// The pool the store runs on. + pub(crate) fn pool(&self) -> &sqlx::PgPool { + &self.pool + } + + /// Starts a write transaction holding the account's podcast lock. + async fn begin_write( + &self, + user_id: i64, + ) -> Result> { + let mut tx = self.pool.begin().await?; + lock_account(&mut tx, user_id).await?; + Ok(tx) + } +} + +/// Serializes podcast writers of one account until the transaction ends. +pub(crate) async fn lock_account(conn: &mut sqlx::PgConnection, user_id: i64) -> Result<()> { + sqlx::query( + "SELECT pg_advisory_xact_lock(hashtextextended('furumusic:podcasts:' || $1::text, 0))", + ) + .bind(user_id) + .execute(conn) + .await?; + Ok(()) +} + +pub(crate) fn now_ms() -> i64 { + chrono::Utc::now().timestamp_millis() +} + +/// Converts a contract generation to the `BIGINT` column value. +/// +/// Generations above `i64::MAX` cannot be stored; records carrying one are +/// rejected (no conforming client produces them). +pub(crate) fn generation_to_sql(generation: u64) -> Result { + i64::try_from(generation) + .map_err(|_| PodcastStoreError::Invalid("generation exceeds i64::MAX".to_string())) +} + +pub(crate) fn generation_from_sql(value: i64) -> u64 { + u64::try_from(value).unwrap_or(0) +} + +pub(crate) fn status_to_sql(status: ProgressStatus) -> &'static str { + match status { + ProgressStatus::NotStarted => "not_started", + ProgressStatus::InProgress => "in_progress", + ProgressStatus::Completed => "completed", + } +} + +pub(crate) fn status_from_sql(value: &str) -> ProgressStatus { + match value { + "in_progress" => ProgressStatus::InProgress, + "completed" => ProgressStatus::Completed, + _ => ProgressStatus::NotStarted, + } +} + +/// `u32` contract numbers (season, episode number) from a `BIGINT` column. +pub(crate) fn u32_from_sql(value: Option) -> Option { + value.and_then(|value| u32::try_from(value).ok()) +} + +fn decode_error(message: impl ToString) -> sqlx::Error { + sqlx::Error::Decode(Box::new(PodcastStoreError::Invalid(message.to_string()))) +} + +/// Parses a stored id; stored ids are validated on write. +pub(crate) fn podcast_id_col(row: &PgRow, column: &str) -> sqlx::Result { + PodcastId::try_from(row.try_get::(column)?).map_err(decode_error) +} + +pub(crate) fn episode_id_col(row: &PgRow, column: &str) -> sqlx::Result { + EpisodeId::try_from(row.try_get::(column)?).map_err(decode_error) +} + +pub(crate) fn feed_url_col(row: &PgRow, column: &str) -> sqlx::Result { + FeedUrl::try_from(row.try_get::(column)?).map_err(decode_error) +} + +pub(crate) fn podcast_meta_col(row: &PgRow, column: &str) -> sqlx::Result { + serde_json::from_value(row.try_get::(column)?).map_err(decode_error) +} + +/// Progress register columns of table alias `alias`, named `{prefix}...`. +pub(crate) fn progress_columns(alias: &str, prefix: &str) -> String { + [ + "episode_id", + "podcast_id", + "position_ms", + "status", + "generation", + "session_id", + "hlc_ms", + "device_id", + "op_id", + ] + .iter() + .map(|column| format!("{alias}.{column} AS {prefix}{column}")) + .collect::>() + .join(", ") +} + +/// Progress register selected with [`progress_columns`]. +pub(crate) fn progress_from_row(row: &PgRow, prefix: &str) -> sqlx::Result { + let column = |name: &str| format!("{prefix}{name}"); + Ok(ProgressRecord { + episode_id: episode_id_col(row, &column("episode_id"))?, + podcast_id: podcast_id_col(row, &column("podcast_id"))?, + position_ms: row.try_get(column("position_ms").as_str())?, + status: status_from_sql(&row.try_get::(column("status").as_str())?), + generation: generation_from_sql(row.try_get(column("generation").as_str())?), + session_id: row.try_get(column("session_id").as_str())?, + hlc_ms: row.try_get(column("hlc_ms").as_str())?, + device_id: row.try_get(column("device_id").as_str())?, + op_id: row.try_get(column("op_id").as_str())?, + }) +} + +/// Listening session columns of table alias `s` (plain names). +pub(crate) const SESSION_COLUMNS: &str = "s.session_id, s.episode_id, s.podcast_id, s.device_id, \ + s.started_at_ms, s.ended_at_ms, s.start_position_ms, s.end_position_ms, s.listened_ms, \ + s.completed"; + +pub(crate) fn session_from_row(row: &PgRow) -> sqlx::Result { + Ok(ListenSession { + session_id: row.try_get("session_id")?, + episode_id: episode_id_col(row, "episode_id")?, + podcast_id: podcast_id_col(row, "podcast_id")?, + device_id: row.try_get("device_id")?, + started_at_ms: row.try_get("started_at_ms")?, + ended_at_ms: row.try_get("ended_at_ms")?, + start_position_ms: row.try_get("start_position_ms")?, + end_position_ms: row.try_get("end_position_ms")?, + listened_ms: row.try_get("listened_ms")?, + completed: row.try_get("completed")?, + }) +} + +/// Normalized search text of an episode: `normalize_name` of the title and +/// of the description, separated by a newline (the normalization never +/// produces one, so a needle cannot match across the two). +pub(crate) fn episode_search_norm(title: &str, description: Option<&str>) -> String { + format!( + "{}\n{}", + music_dht::normalize_name(title), + music_dht::normalize_name(description.unwrap_or_default()) + ) +} diff --git a/src/podcasts/net.rs b/src/podcasts/net.rs new file mode 100644 index 0000000..4bc7a3c --- /dev/null +++ b/src/podcasts/net.rs @@ -0,0 +1,629 @@ +//! Outbound HTTP for podcast feeds and the audio proxy, guarded against SSRF. +//! +//! Feed URLs and enclosure URLs come from users and from third-party feed +//! documents, so the server must never be tricked into requesting its own +//! network. Every request made through [`SafeHttp`]: +//! +//! * accepts only `http` and `https` URLs with a host and without userinfo; +//! * resolves host names with [`GuardedResolver`], which rejects the whole +//! answer when **any** address is not public (loopback, private, shared +//! CGNAT, link-local, cloud metadata, multicast, unspecified, documentation, +//! reserved; IPv4 and IPv6, including IPv4-mapped, NAT64 and 6to4 forms). +//! The connection is made to exactly the addresses that were checked: the +//! resolver *is* the connector's resolver, there is no separate pre-check +//! that a second DNS answer could bypass (DNS rebinding); +//! * checks IP-literal hosts the same way (the connector does not resolve +//! them); +//! * connects only to the web ports [`ALLOWED_PORTS`] (80, 443, 8080, 8443): +//! a public address can still be the server's own, and its other services +//! (databases, admin ports) are often reachable from the host itself; +//! * follows redirects itself, at most `max_redirects` hops, checking every +//! hop like the first URL; +//! * never uses a system proxy (it would resolve the target instead of the +//! guarded resolver), never stores or sends cookies and never forwards +//! credentials: callers pass the few headers they need explicitly. +//! +//! Loopback can be allowed only in tests (`AddressPolicy::test_allow_loopback` +//! is `#[cfg(test)]`), so the local test servers work and nothing else does. + +use std::fmt; +use std::future::Future; +use std::net::{IpAddr, Ipv4Addr, Ipv6Addr, SocketAddr}; +use std::pin::Pin; +use std::sync::Arc; +use std::time::Duration; + +use reqwest::header::{HeaderMap, LOCATION}; +use reqwest::{StatusCode, Url}; + +/// Why an address is not reachable through [`SafeHttp`]. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum AddressClass { + /// `0.0.0.0/8`, `::`. + Unspecified, + /// `127.0.0.0/8`, `::1`. + Loopback, + /// RFC 1918 and the deprecated IPv6 site-local range. + Private, + /// `100.64.0.0/10` (carrier-grade NAT). + SharedCgnat, + /// `169.254.0.0/16`, `fe80::/10`. + LinkLocal, + /// Cloud instance metadata endpoints. + CloudMetadata, + /// `fc00::/7`. + UniqueLocal, + /// `224.0.0.0/4`, `ff00::/8`. + Multicast, + /// `255.255.255.255`. + Broadcast, + /// Documentation ranges (`192.0.2.0/24`, `2001:db8::/32`, ...). + Documentation, + /// `198.18.0.0/15`, `2001:2::/48`. + Benchmarking, + /// Everything else outside global unicast (IETF protocol assignments, + /// `240.0.0.0/4`, IPv4-compatible IPv6, Teredo, local NAT64, ...). + Reserved, +} + +impl fmt::Display for AddressClass { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str(match self { + Self::Unspecified => "unspecified", + Self::Loopback => "loopback", + Self::Private => "private", + Self::SharedCgnat => "shared (CGNAT)", + Self::LinkLocal => "link-local", + Self::CloudMetadata => "cloud metadata", + Self::UniqueLocal => "unique local", + Self::Multicast => "multicast", + Self::Broadcast => "broadcast", + Self::Documentation => "documentation", + Self::Benchmarking => "benchmarking", + Self::Reserved => "reserved", + }) + } +} + +/// Well-known cloud metadata endpoints. They also fall into blocked ranges +/// (link-local, CGNAT, unique local); naming them makes errors clearer. +const METADATA_V4: [Ipv4Addr; 3] = [ + Ipv4Addr::new(169, 254, 169, 254), + Ipv4Addr::new(169, 254, 170, 2), + Ipv4Addr::new(100, 100, 100, 200), +]; +const METADATA_V6: [Ipv6Addr; 1] = [Ipv6Addr::new(0xfd00, 0x0ec2, 0, 0, 0, 0, 0, 0x0254)]; + +fn in_v4(ip: Ipv4Addr, net: [u8; 4], prefix: u32) -> bool { + let mask = if prefix == 0 { + 0 + } else { + u32::MAX << (32 - prefix) + }; + u32::from(ip) & mask == u32::from(Ipv4Addr::from(net)) & mask +} + +fn in_v6(ip: Ipv6Addr, net: [u16; 8], prefix: u32) -> bool { + let mask = if prefix == 0 { + 0 + } else { + u128::MAX << (128 - prefix) + }; + let [a, b, c, d, e, f, g, h] = net; + u128::from(ip) & mask == u128::from(Ipv6Addr::new(a, b, c, d, e, f, g, h)) & mask +} + +/// Class of a non-public IPv4 address; `None` for a public one. +fn classify_v4(ip: Ipv4Addr) -> Option { + use AddressClass::*; + if METADATA_V4.contains(&ip) { + return Some(CloudMetadata); + } + const RULES: &[([u8; 4], u32, AddressClass)] = &[ + ([0, 0, 0, 0], 8, Unspecified), + ([10, 0, 0, 0], 8, Private), + ([100, 64, 0, 0], 10, SharedCgnat), + ([127, 0, 0, 0], 8, Loopback), + ([169, 254, 0, 0], 16, LinkLocal), + ([172, 16, 0, 0], 12, Private), + ([192, 0, 0, 0], 24, Reserved), + ([192, 0, 2, 0], 24, Documentation), + ([192, 88, 99, 0], 24, Reserved), + ([192, 168, 0, 0], 16, Private), + ([198, 18, 0, 0], 15, Benchmarking), + ([198, 51, 100, 0], 24, Documentation), + ([203, 0, 113, 0], 24, Documentation), + ([224, 0, 0, 0], 4, Multicast), + ([255, 255, 255, 255], 32, Broadcast), + ([240, 0, 0, 0], 4, Reserved), + ]; + RULES + .iter() + .find(|(net, prefix, _)| in_v4(ip, *net, *prefix)) + .map(|(_, _, class)| *class) +} + +/// Class of a non-public IPv6 address; `None` for a public one. Forms that +/// embed an IPv4 address (mapped, NAT64, 6to4) are judged by that address. +fn classify_v6(ip: Ipv6Addr) -> Option { + use AddressClass::*; + if METADATA_V6.contains(&ip) { + return Some(CloudMetadata); + } + if ip.is_unspecified() { + return Some(Unspecified); + } + if ip.is_loopback() { + return Some(Loopback); + } + let bits = u128::from(ip); + let embedded_v4 = |shift: u32| Ipv4Addr::from((bits >> shift) as u32); + // ::ffff:a.b.c.d + if let Some(v4) = ip.to_ipv4_mapped() { + return classify_v4(v4); + } + // 64:ff9b::/96, the well-known NAT64 prefix. + if in_v6(ip, [0x64, 0xff9b, 0, 0, 0, 0, 0, 0], 96) { + return classify_v4(embedded_v4(0)); + } + // 2002::/16, 6to4: the IPv4 address follows the prefix. + if in_v6(ip, [0x2002, 0, 0, 0, 0, 0, 0, 0], 16) { + return classify_v4(embedded_v4(80)); + } + const RULES: &[([u16; 8], u32, AddressClass)] = &[ + // IPv4-compatible (deprecated) and the rest of ::/96. + ([0, 0, 0, 0, 0, 0, 0, 0], 96, Reserved), + ([0x64, 0xff9b, 1, 0, 0, 0, 0, 0], 48, Reserved), + ([0x100, 0, 0, 0, 0, 0, 0, 0], 64, Reserved), + ([0x2001, 0x2, 0, 0, 0, 0, 0, 0], 48, Benchmarking), + ([0x2001, 0xdb8, 0, 0, 0, 0, 0, 0], 32, Documentation), + // IETF protocol assignments, including Teredo (2001::/32). + ([0x2001, 0, 0, 0, 0, 0, 0, 0], 23, Reserved), + ([0x3fff, 0, 0, 0, 0, 0, 0, 0], 20, Documentation), + ([0xfc00, 0, 0, 0, 0, 0, 0, 0], 7, UniqueLocal), + ([0xfe80, 0, 0, 0, 0, 0, 0, 0], 10, LinkLocal), + ([0xfec0, 0, 0, 0, 0, 0, 0, 0], 10, Private), + ([0xff00, 0, 0, 0, 0, 0, 0, 0], 8, Multicast), + ]; + if let Some((_, _, class)) = RULES + .iter() + .find(|(net, prefix, _)| in_v6(ip, *net, *prefix)) + { + return Some(*class); + } + // Only global unicast (2000::/3) is public. + (!in_v6(ip, [0x2000, 0, 0, 0, 0, 0, 0, 0], 3)).then_some(Reserved) +} + +/// Class of a non-public address; `None` when it is public. +pub(crate) fn classify(ip: IpAddr) -> Option { + match ip { + IpAddr::V4(ip) => classify_v4(ip), + IpAddr::V6(ip) => classify_v6(ip), + } +} + +/// Ports feeds and enclosures may be fetched from. +pub(crate) const ALLOWED_PORTS: [u16; 4] = [80, 443, 8080, 8443]; + +/// Which addresses [`SafeHttp`] may connect to. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) struct AddressPolicy { + allow_loopback: bool, + any_port: bool, +} + +impl AddressPolicy { + /// Public addresses on [`ALLOWED_PORTS`] only. The only policy + /// available outside tests. + pub(crate) const PUBLIC_ONLY: Self = Self { + allow_loopback: false, + any_port: false, + }; + + /// Public addresses plus loopback on any port, for the local test + /// servers. + #[cfg(test)] + pub(crate) const fn test_allow_loopback() -> Self { + Self { + allow_loopback: true, + any_port: true, + } + } + + /// Whether a connection to `port` is allowed. + pub(crate) fn allows_port(&self, port: u16) -> bool { + self.any_port || ALLOWED_PORTS.contains(&port) + } + + /// Checks one address. + pub(crate) fn check(&self, ip: IpAddr) -> Result<(), AddressClass> { + match classify(ip) { + None => Ok(()), + Some(AddressClass::Loopback) if self.allow_loopback => Ok(()), + Some(class) => Err(class), + } + } +} + +/// A host name or literal resolved to an address that is not allowed. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct BlockedAddress { + /// Host as written in the URL. + pub(crate) host: String, + /// The offending address. + pub(crate) address: IpAddr, + /// Why it is blocked. + pub(crate) class: AddressClass, +} + +impl BlockedAddress { + /// Whether the URL named the address itself (an IP literal) rather + /// than a host name that resolved to it. + pub(crate) fn is_literal(&self) -> bool { + let literal = self.host.trim_start_matches('[').trim_end_matches(']'); + literal.parse::().is_ok() + } +} + +impl fmt::Display for BlockedAddress { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + if self.is_literal() { + write!(f, "{} is a {} address", self.host, self.class)?; + } else { + write!( + f, + "{} resolves to a {} address ({})", + self.host, self.class, self.address + )?; + } + f.write_str("; only public addresses are allowed") + } +} + +impl std::error::Error for BlockedAddress {} + +/// Name resolution used by [`GuardedResolver`]. +pub(crate) trait HostLookup: Send + Sync + 'static { + /// Every address of `host` (a DNS name, never an IP literal). + fn lookup(&self, host: &str) -> LookupFuture; +} + +pub(crate) type LookupFuture = + Pin>> + Send + 'static>>; + +/// The operating system resolver (`getaddrinfo` via tokio). +#[derive(Debug, Clone, Copy, Default)] +pub(crate) struct SystemLookup; + +impl HostLookup for SystemLookup { + fn lookup(&self, host: &str) -> LookupFuture { + let host = host.to_string(); + Box::pin(async move { + Ok(tokio::net::lookup_host((host.as_str(), 0)) + .await? + .map(|addr| addr.ip()) + .collect()) + }) + } +} + +const LOOKUP_TIMEOUT: Duration = Duration::from_secs(10); + +/// DNS resolver of the guarded client: resolves, then rejects the answer +/// unless every address is allowed. The connector connects to exactly the +/// returned addresses, so a checked answer cannot be swapped afterwards. +pub(crate) struct GuardedResolver { + policy: AddressPolicy, + lookup: Arc, +} + +impl GuardedResolver { + pub(crate) fn new(policy: AddressPolicy, lookup: Arc) -> Self { + Self { policy, lookup } + } + + /// Resolves and checks `host`; the addresses a connection may use. + pub(crate) async fn resolve_checked( + &self, + host: &str, + ) -> Result, Box> { + let ips = tokio::time::timeout(LOOKUP_TIMEOUT, self.lookup.lookup(host)) + .await + .map_err(|_| format!("resolving {host} timed out"))??; + if ips.is_empty() { + return Err(format!("{host} has no addresses").into()); + } + for ip in &ips { + if let Err(class) = self.policy.check(*ip) { + return Err(Box::new(BlockedAddress { + host: host.to_string(), + address: *ip, + class, + })); + } + } + Ok(ips) + } +} + +impl reqwest::dns::Resolve for GuardedResolver { + fn resolve(&self, name: reqwest::dns::Name) -> reqwest::dns::Resolving { + let resolver = Self { + policy: self.policy, + lookup: Arc::clone(&self.lookup), + }; + let host = name.as_str().to_string(); + Box::pin(async move { + let ips = resolver.resolve_checked(&host).await?; + // Port 0: the connector uses the URL's port. + let addrs: Vec = ips.into_iter().map(|ip| SocketAddr::new(ip, 0)).collect(); + Ok(Box::new(addrs.into_iter()) as reqwest::dns::Addrs) + }) + } +} + +/// Errors of [`SafeHttp`]. +#[derive(Debug)] +pub(crate) enum GuardError { + /// Not `http`/`https`. + UnsupportedScheme(String), + /// The URL carries `user:password@`. + Credentials, + /// The URL has no host. + MissingHost, + /// The host is (or resolves to) a blocked address. + Blocked(BlockedAddress), + /// The port is not one of [`ALLOWED_PORTS`]. + Port(u16), + /// More redirects than allowed. + TooManyRedirects(usize), + /// A redirect without a usable `Location`. + InvalidRedirect(String), + /// Any other transport failure (DNS, connect, TLS, timeout, ...). + Http(reqwest::Error), +} + +impl GuardError { + pub(crate) fn is_timeout(&self) -> bool { + matches!(self, Self::Http(error) if error.is_timeout()) + } + + /// Whether the guard refused the URL for what it says, before any name + /// resolution: scheme, credentials, port, a blocked IP literal. + /// + /// A host *name* that resolves to a blocked address is not a refusal in + /// this sense: users are told the same as for a name that does not + /// resolve or a server that does not answer, so the answers cannot be + /// used to learn which names exist in the server's internal DNS. The + /// details stay in the server log. + pub(crate) fn is_refusal(&self) -> bool { + match self { + Self::UnsupportedScheme(_) | Self::Credentials | Self::MissingHost | Self::Port(_) => { + true + } + Self::Blocked(blocked) => blocked.is_literal(), + _ => false, + } + } +} + +/// What a signed-in user is told about a refused or failed request. The +/// details ([`GuardError`]'s `Display`: resolved addresses, transport error +/// chains) stay in the server logs; showing them would turn feed and audio +/// requests into a scanner of the server's network. +pub(crate) const REFUSED_MESSAGE: &str = + "only public http and https addresses on ports 80, 443, 8080 and 8443 are allowed"; + +impl fmt::Display for GuardError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::UnsupportedScheme(scheme) => { + write!(f, "unsupported URL scheme {scheme:?} (only http and https)") + } + Self::Credentials => f.write_str("URLs with credentials are not allowed"), + Self::MissingHost => f.write_str("URL has no host"), + Self::Blocked(blocked) => blocked.fmt(f), + Self::Port(port) => write!(f, "port {port} is not allowed"), + Self::TooManyRedirects(limit) => write!(f, "more than {limit} redirects"), + Self::InvalidRedirect(reason) => write!(f, "invalid redirect: {reason}"), + Self::Http(error) => { + write!(f, "{error}")?; + let mut source = std::error::Error::source(error); + while let Some(inner) = source { + write!(f, ": {inner}")?; + source = inner.source(); + } + Ok(()) + } + } + } +} + +impl std::error::Error for GuardError {} + +impl From for GuardError { + /// Surfaces a resolver rejection buried in the transport error chain. + /// The URL is dropped: feed and enclosure URLs often carry access + /// tokens, and these errors end up in logs. + fn from(error: reqwest::Error) -> Self { + let error = error.without_url(); + let mut source = std::error::Error::source(&error); + while let Some(inner) = source { + if let Some(blocked) = inner.downcast_ref::() { + return Self::Blocked(blocked.clone()); + } + source = inner.source(); + } + Self::Http(error) + } +} + +/// Timeouts and limits of a [`SafeHttp`] client. +#[derive(Debug, Clone, Copy)] +pub(crate) struct SafeHttpConfig { + /// TCP + TLS connect timeout (name resolution has its own 10 s bound). + pub(crate) connect_timeout: Duration, + /// Maximum time between two reads of a response. + pub(crate) read_timeout: Duration, + /// Whole-request timeout including the body; `None` for streams. + pub(crate) total_timeout: Option, + /// Redirect hops followed before giving up. + pub(crate) max_redirects: usize, +} + +/// A response after following redirects. +#[derive(Debug)] +pub(crate) struct Followed { + /// The final response (never a followed redirect). + pub(crate) response: reqwest::Response, + /// URL of the final response. + pub(crate) final_url: Url, + /// Status of every redirect followed, in order. + pub(crate) redirects: Vec, +} + +impl Followed { + /// Whether at least one redirect was followed and all of them were + /// permanent (301/308): only then has the resource moved. + pub(crate) fn moved_permanently(&self) -> bool { + !self.redirects.is_empty() + && self + .redirects + .iter() + .all(|status| matches!(status.as_u16(), 301 | 308)) + } +} + +/// SSRF-guarded HTTP client (see the module documentation). +#[derive(Clone)] +pub(crate) struct SafeHttp { + client: reqwest::Client, + policy: AddressPolicy, + max_redirects: usize, +} + +impl fmt::Debug for SafeHttp { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("SafeHttp") + .field("policy", &self.policy) + .field("max_redirects", &self.max_redirects) + .finish_non_exhaustive() + } +} + +impl SafeHttp { + /// A production client: public addresses only, system DNS. + pub(crate) fn public(config: SafeHttpConfig) -> anyhow::Result { + Self::with_lookup(config, AddressPolicy::PUBLIC_ONLY, Arc::new(SystemLookup)) + } + + /// A client with an explicit policy and resolver (tests inject both). + pub(crate) fn with_lookup( + config: SafeHttpConfig, + policy: AddressPolicy, + lookup: Arc, + ) -> anyhow::Result { + let mut builder = reqwest::Client::builder() + .dns_resolver(Arc::new(GuardedResolver::new(policy, lookup))) + // A proxy would resolve the target itself, bypassing the guard. + .no_proxy() + .redirect(reqwest::redirect::Policy::none()) + .referer(false) + .connect_timeout(config.connect_timeout) + .read_timeout(config.read_timeout) + .user_agent(concat!( + "furumusic/", + env!("CARGO_PKG_VERSION"), + " (podcasts)" + )); + if let Some(timeout) = config.total_timeout { + builder = builder.timeout(timeout); + } + Ok(Self { + client: builder.build()?, + policy, + max_redirects: config.max_redirects, + }) + } + + /// Checks a URL before any connection: scheme, credentials, host, and + /// IP-literal hosts against the policy (names are checked by the + /// resolver at connect time). + pub(crate) fn check_url(&self, url: &Url) -> Result<(), GuardError> { + check_url(self.policy, url) + } + + /// `GET url` with `headers`, following redirects (each hop checked, the + /// same headers re-sent; nothing else is added). + pub(crate) async fn get(&self, url: Url, headers: &HeaderMap) -> Result { + let mut url = url; + let mut redirects = Vec::new(); + loop { + self.check_url(&url)?; + let response = self + .client + .get(url.clone()) + .headers(headers.clone()) + .send() + .await?; + let status = response.status(); + if !matches!(status.as_u16(), 301 | 302 | 303 | 307 | 308) { + return Ok(Followed { + response, + final_url: url, + redirects, + }); + } + if redirects.len() >= self.max_redirects { + return Err(GuardError::TooManyRedirects(self.max_redirects)); + } + let location = response + .headers() + .get(LOCATION) + .and_then(|value| value.to_str().ok()) + .ok_or_else(|| { + GuardError::InvalidRedirect(format!("HTTP {status} without Location")) + })?; + let next = url.join(location).map_err(|error| { + GuardError::InvalidRedirect(format!("unusable Location: {error}")) + })?; + redirects.push(status); + url = next; + } + } +} + +/// See [`SafeHttp::check_url`]. +pub(crate) fn check_url(policy: AddressPolicy, url: &Url) -> Result<(), GuardError> { + if !matches!(url.scheme(), "http" | "https") { + return Err(GuardError::UnsupportedScheme(url.scheme().to_string())); + } + if !url.username().is_empty() || url.password().is_some() { + return Err(GuardError::Credentials); + } + let host = url + .host_str() + .filter(|host| !host.is_empty()) + .ok_or(GuardError::MissingHost)?; + // The URL parser normalizes every IPv4 spelling (decimal, octal, hex, + // trailing dot) to dotted quads and keeps IPv6 literals in brackets. + let bare = host + .strip_prefix('[') + .and_then(|host| host.strip_suffix(']')) + .unwrap_or(host); + if let Ok(ip) = bare.parse::() { + policy.check(ip).map_err(|class| { + GuardError::Blocked(BlockedAddress { + host: host.to_string(), + address: ip, + class, + }) + })?; + } + let port = url.port_or_known_default().ok_or(GuardError::MissingHost)?; + if !policy.allows_port(port) { + return Err(GuardError::Port(port)); + } + Ok(()) +} diff --git a/src/podcasts/net_tests.rs b/src/podcasts/net_tests.rs new file mode 100644 index 0000000..78bf65f --- /dev/null +++ b/src/podcasts/net_tests.rs @@ -0,0 +1,438 @@ +//! SSRF guard: address classes, URL checks, the resolver, DNS rebinding, +//! redirects and headers. Network tests use the local test server; loopback +//! is allowed only through the test-only `AddressPolicy::test_allow_loopback`. + +use std::net::IpAddr; +use std::sync::Arc; +use std::sync::atomic::Ordering; + +use reqwest::Url; +use reqwest::header::HeaderMap; + +use super::net::{ + AddressClass, AddressPolicy, BlockedAddress, GuardError, GuardedResolver, SafeHttp, check_url, + classify, +}; +use super::test_server::{Route, ScriptedLookup, TEST_HTTP, TestServer, test_http}; + +// --------------------------------------------------------------------------- +// Address classes +// --------------------------------------------------------------------------- + +fn ip(text: &str) -> IpAddr { + text.parse().unwrap() +} + +#[test] +fn public_addresses_are_allowed() { + for address in [ + "8.8.8.8", + "1.1.1.1", + "93.184.215.14", + "172.32.0.1", + "100.128.0.1", + "192.0.3.1", + "2606:4700:4700::1111", + "2a00:1450:4001:80b::200e", + "::ffff:8.8.8.8", + "64:ff9b::808:808", + "2002:808:808::1", + ] { + assert_eq!(classify(ip(address)), None, "{address}"); + assert_eq!(AddressPolicy::PUBLIC_ONLY.check(ip(address)), Ok(())); + } +} + +#[test] +fn non_public_addresses_are_classified() { + use AddressClass::*; + let cases = [ + // IPv4 + ("0.0.0.0", Unspecified), + ("0.1.2.3", Unspecified), + ("10.1.2.3", Private), + ("100.64.0.1", SharedCgnat), + ("100.127.255.254", SharedCgnat), + ("127.0.0.1", Loopback), + ("127.255.0.9", Loopback), + ("169.254.1.1", LinkLocal), + ("169.254.169.254", CloudMetadata), + ("169.254.170.2", CloudMetadata), + ("100.100.100.200", CloudMetadata), + ("172.16.0.1", Private), + ("172.31.255.255", Private), + ("192.0.0.8", Reserved), + ("192.0.2.5", Documentation), + ("192.88.99.1", Reserved), + ("192.168.1.1", Private), + ("198.18.0.1", Benchmarking), + ("198.19.255.255", Benchmarking), + ("198.51.100.1", Documentation), + ("203.0.113.1", Documentation), + ("224.0.0.1", Multicast), + ("239.255.255.250", Multicast), + ("240.0.0.1", Reserved), + ("255.255.255.255", Broadcast), + // IPv6 + ("::", Unspecified), + ("::1", Loopback), + ("::ffff:127.0.0.1", Loopback), + ("::ffff:10.0.0.1", Private), + ("::ffff:169.254.169.254", CloudMetadata), + ("::ffff:0.0.0.0", Unspecified), + ("::127.0.0.1", Reserved), + ("::8.8.8.8", Reserved), + ("64:ff9b::a00:1", Private), + ("64:ff9b::7f00:1", Loopback), + ("64:ff9b:1::1", Reserved), + ("2002:7f00:1::1", Loopback), + ("2002:a9fe:a9fe::1", CloudMetadata), + ("2002:c0a8:101::1", Private), + ("100::1", Reserved), + ("2001::1", Reserved), + ("2001:2::1", Benchmarking), + ("2001:db8::1", Documentation), + ("3fff::1", Documentation), + ("fc00::1", UniqueLocal), + ("fd12:3456::1", UniqueLocal), + ("fd00:ec2::254", CloudMetadata), + ("fe80::1", LinkLocal), + ("febf::1", LinkLocal), + ("fec0::1", Private), + ("ff02::1", Multicast), + ("4000::1", Reserved), + ("::2", Reserved), + ]; + for (address, class) in cases { + assert_eq!(classify(ip(address)), Some(class), "{address}"); + assert_eq!( + AddressPolicy::PUBLIC_ONLY.check(ip(address)), + Err(class), + "{address}" + ); + } +} + +#[test] +fn only_the_test_policy_allows_loopback_and_nothing_else() { + let test = AddressPolicy::test_allow_loopback(); + assert_eq!(test.check(ip("127.0.0.1")), Ok(())); + assert_eq!(test.check(ip("::1")), Ok(())); + assert_eq!(test.check(ip("::ffff:127.0.0.1")), Ok(())); + assert_eq!(test.check(ip("10.0.0.1")), Err(AddressClass::Private)); + assert_eq!( + test.check(ip("169.254.169.254")), + Err(AddressClass::CloudMetadata) + ); + assert_eq!( + AddressPolicy::PUBLIC_ONLY.check(ip("127.0.0.1")), + Err(AddressClass::Loopback) + ); +} + +fn url(text: &str) -> Url { + Url::parse(text).unwrap() +} + +#[test] +fn urls_are_checked_before_connecting() { + let policy = AddressPolicy::PUBLIC_ONLY; + for (text, class) in [ + ("http://127.0.0.1/feed", AddressClass::Loopback), + // Every IPv4 spelling is normalized by the URL parser. + ("http://2130706433/feed", AddressClass::Loopback), + ("http://0x7f.1/feed", AddressClass::Loopback), + ("http://127.1/feed", AddressClass::Loopback), + ("http://0177.0.0.1/feed", AddressClass::Loopback), + ("http://127.0.0.1./feed", AddressClass::Loopback), + ("http://0/feed", AddressClass::Unspecified), + ("http://[::1]:8080/feed", AddressClass::Loopback), + ("http://[::ffff:7f00:1]/feed", AddressClass::Loopback), + ("http://[::ffff:a9fe:a9fe]/", AddressClass::CloudMetadata), + ( + "http://169.254.169.254/latest/meta-data/", + AddressClass::CloudMetadata, + ), + ("https://10.0.0.5/audio.mp3", AddressClass::Private), + ("https://[fd00::1]/audio.mp3", AddressClass::UniqueLocal), + ] { + match check_url(policy, &url(text)) { + Err(GuardError::Blocked(blocked)) => assert_eq!(blocked.class, class, "{text}"), + other => panic!("{text}: {other:?}"), + } + } + assert!(matches!( + check_url(policy, &url("ftp://example.com/feed")), + Err(GuardError::UnsupportedScheme(_)) + )); + assert!(matches!( + check_url(policy, &url("file:///etc/passwd")), + Err(GuardError::UnsupportedScheme(_)) + )); + assert!(matches!( + check_url(policy, &url("https://user:secret@example.com/feed")), + Err(GuardError::Credentials) + )); + assert!(matches!( + check_url(policy, &url("https://user@example.com/feed")), + Err(GuardError::Credentials) + )); + // Names are checked by the resolver at connect time. + assert!(check_url(policy, &url("https://feeds.example.com/rss")).is_ok()); + assert!(check_url(policy, &url("http://localhost/rss")).is_ok()); +} + +#[test] +fn only_web_ports_are_allowed() { + // A public address can be the server's own; its databases and admin + // ports must not be reachable through feed or audio requests. + let policy = AddressPolicy::PUBLIC_ONLY; + for text in [ + "https://feeds.example.com/rss", + "http://feeds.example.com/rss", + "https://feeds.example.com:443/rss", + "http://feeds.example.com:8080/rss", + "https://feeds.example.com:8443/rss", + "https://8.8.8.8/rss", + ] { + assert!(check_url(policy, &url(text)).is_ok(), "{text}"); + } + for (text, port) in [ + ("http://feeds.example.com:5432/", 5432), + ("https://feeds.example.com:22/", 22), + ("http://8.8.8.8:6379/feed", 6379), + ("https://feeds.example.com:3000/", 3000), + ] { + match check_url(policy, &url(text)) { + Err(error @ GuardError::Port(refused)) => { + assert_eq!(refused, port, "{text}"); + assert!(error.is_refusal()); + } + other => panic!("{text}: {other:?}"), + } + } + // The local test servers listen on random ports. + assert!( + check_url( + AddressPolicy::test_allow_loopback(), + &url("http://127.0.0.1:5432/") + ) + .is_ok() + ); +} + +// --------------------------------------------------------------------------- +// Resolver +// --------------------------------------------------------------------------- + +fn blocked_class(error: &(dyn std::error::Error + Send + Sync + 'static)) -> Option { + error + .downcast_ref::() + .map(|blocked| blocked.class) +} + +#[tokio::test] +async fn resolver_rejects_the_whole_answer_when_any_address_is_not_public() { + let lookup = ScriptedLookup::with(&[ + ("public.test", &[&["8.8.8.8", "2606:4700:4700::1111"]]), + ("mixed.test", &[&["8.8.8.8", "10.0.0.1"]]), + ("mapped.test", &[&["::ffff:192.168.0.10"]]), + ("localhost", &[&["127.0.0.1", "::1"]]), + ("empty.test", &[&[]]), + ]); + let resolver = GuardedResolver::new(AddressPolicy::PUBLIC_ONLY, lookup); + assert_eq!( + resolver.resolve_checked("public.test").await.unwrap(), + vec![ip("8.8.8.8"), ip("2606:4700:4700::1111")] + ); + let error = resolver.resolve_checked("mixed.test").await.unwrap_err(); + assert_eq!(blocked_class(error.as_ref()), Some(AddressClass::Private)); + let error = resolver.resolve_checked("mapped.test").await.unwrap_err(); + assert_eq!(blocked_class(error.as_ref()), Some(AddressClass::Private)); + let error = resolver.resolve_checked("localhost").await.unwrap_err(); + assert_eq!(blocked_class(error.as_ref()), Some(AddressClass::Loopback)); + assert!(resolver.resolve_checked("empty.test").await.is_err()); + assert!(resolver.resolve_checked("nowhere.test").await.is_err()); +} + +fn public_http(lookup: Arc) -> SafeHttp { + SafeHttp::with_lookup(TEST_HTTP, AddressPolicy::PUBLIC_ONLY, lookup).unwrap() +} + +#[tokio::test] +async fn production_policy_never_reaches_a_local_server() { + let server = TestServer::start(); + server.set("/feed", Route::new("ok", "text/plain")); + let lookup = ScriptedLookup::with(&[("loop.test", &[&["127.0.0.1"]])]); + let http = public_http(Arc::clone(&lookup)); + match http.get(url(&server.url("/feed")), &HeaderMap::new()).await { + Err(GuardError::Blocked(blocked)) => assert_eq!(blocked.class, AddressClass::Loopback), + other => panic!("{other:?}"), + } + // The test server's random port is refused before any lookup. + let named = url(&server.url_on("loop.test", "/feed")); + match http.get(named.clone(), &HeaderMap::new()).await { + Err(GuardError::Port(port)) => assert_eq!(Some(port), named.port()), + other => panic!("{other:?}"), + } + assert_eq!(lookup.calls.load(Ordering::SeqCst), 0); + // On an allowed port the resolver refuses the loopback answer. + let resolver = GuardedResolver::new(AddressPolicy::PUBLIC_ONLY, lookup); + let error = resolver.resolve_checked("loop.test").await.unwrap_err(); + assert_eq!(blocked_class(error.as_ref()), Some(AddressClass::Loopback)); + assert_eq!(server.request_count(), 0); +} + +#[tokio::test] +async fn every_connection_uses_the_addresses_it_checked() { + // DNS rebinding: the first answer is allowed, the next one points to a + // private address. Each connection resolves and checks once and + // connects to exactly those addresses, so the rebound answer is refused + // instead of being used after an earlier successful check. + let server = TestServer::start(); + server.set("/feed", Route::new("first", "text/plain")); + let lookup = ScriptedLookup::with(&[("rebind.test", &[&["127.0.0.1"], &["10.0.0.7"]])]); + let http = test_http(Arc::clone(&lookup)); + let target = url(&server.url_on("rebind.test", "/feed")); + let first = http.get(target.clone(), &HeaderMap::new()).await.unwrap(); + assert_eq!(first.response.text().await.unwrap(), "first"); + match http.get(target, &HeaderMap::new()).await { + Err(GuardError::Blocked(blocked)) => { + assert_eq!(blocked.class, AddressClass::Private); + assert_eq!(blocked.address, ip("10.0.0.7")); + } + other => panic!("{other:?}"), + } + assert_eq!(lookup.calls.load(Ordering::SeqCst), 2); + assert_eq!(server.requests("/feed").len(), 1); +} + +#[tokio::test] +async fn every_redirect_hop_is_checked() { + let server = TestServer::start(); + let lookup = ScriptedLookup::with(&[("intranet.test", &[&["192.168.1.20"]])]); + let http = test_http(lookup); + let cases = [ + ( + "/to-private", + "http://10.0.0.1/feed".to_string(), + AddressClass::Private, + ), + ( + "/to-metadata", + "http://169.254.169.254/latest/meta-data/".to_string(), + AddressClass::CloudMetadata, + ), + ( + "/to-mapped", + "http://[::ffff:10.1.1.1]/".to_string(), + AddressClass::Private, + ), + ( + "/to-name", + server.url_on("intranet.test", "/feed"), + AddressClass::Private, + ), + ]; + for (path, location, class) in &cases { + server.set(path, Route::redirect(302, location)); + match http.get(url(&server.url(path)), &HeaderMap::new()).await { + Err(GuardError::Blocked(blocked)) => assert_eq!(blocked.class, *class, "{path}"), + other => panic!("{path}: {other:?}"), + } + } + server.set("/to-file", Route::redirect(301, "file:///etc/passwd")); + assert!(matches!( + http.get(url(&server.url("/to-file")), &HeaderMap::new()) + .await, + Err(GuardError::UnsupportedScheme(_)) + )); + server.set("/no-location", Route::new("", "text/plain").status(302)); + assert!(matches!( + http.get(url(&server.url("/no-location")), &HeaderMap::new()) + .await, + Err(GuardError::InvalidRedirect(_)) + )); + // Hop limit (3 in the test client): /hop0 -> /hop1 -> ... -> /hop4. + for hop in 0..4 { + server.set( + &format!("/hop{hop}"), + Route::redirect(307, &format!("/hop{}", hop + 1)), + ); + } + server.set("/hop4", Route::new("end", "text/plain")); + assert!(matches!( + http.get(url(&server.url("/hop0")), &HeaderMap::new()).await, + Err(GuardError::TooManyRedirects(3)) + )); + let followed = http + .get(url(&server.url("/hop1")), &HeaderMap::new()) + .await + .unwrap(); + assert_eq!(followed.redirects.len(), 3); + assert_eq!(followed.final_url.path(), "/hop4"); +} + +#[tokio::test] +async fn transport_errors_never_carry_the_url() { + // Feed and enclosure URLs often carry access tokens; errors are logged. + let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap(); + let port = listener.local_addr().unwrap().port(); + drop(listener); + let http = test_http(ScriptedLookup::with(&[])); + let error = http + .get( + url(&format!( + "http://127.0.0.1:{port}/private/feed.xml?token=secret" + )), + &HeaderMap::new(), + ) + .await + .unwrap_err(); + assert!(matches!(error, GuardError::Http(_)), "{error:?}"); + let text = error.to_string(); + assert!( + !text.contains("secret") && !text.contains("feed.xml"), + "{text}" + ); + assert!(!format!("{error:?}").contains("secret")); + // A redirect to an unusable location does not echo it either. + let server = TestServer::start(); + server.set("/bad", Route::redirect(302, "http://[::1/?token=secret")); + let error = http + .get(url(&server.url("/bad")), &HeaderMap::new()) + .await + .unwrap_err(); + assert!(matches!(error, GuardError::InvalidRedirect(_)), "{error:?}"); + assert!(!error.to_string().contains("secret"), "{error}"); +} + +#[tokio::test] +async fn no_cookies_or_credentials_are_sent() { + let server = TestServer::start(); + server.set( + "/start", + Route::redirect(302, "/next").header("Set-Cookie", "session=leak; Path=/"), + ); + server.set( + "/next", + Route::new("ok", "text/plain").header("Set-Cookie", "other=leak; Path=/"), + ); + let http = test_http(ScriptedLookup::with(&[])); + for _ in 0..2 { + let followed = http + .get(url(&server.url("/start")), &HeaderMap::new()) + .await + .unwrap(); + assert_eq!(followed.response.text().await.unwrap(), "ok"); + } + for request in server + .requests("/start") + .iter() + .chain(&server.requests("/next")) + { + for header in ["cookie", "authorization", "referer", "proxy-authorization"] { + assert!(!request.contains_key(header), "{header} sent: {request:?}"); + } + } +} diff --git a/src/podcasts/postgres_tests.rs b/src/podcasts/postgres_tests.rs new file mode 100644 index 0000000..eccddcd --- /dev/null +++ b/src/podcasts/postgres_tests.rs @@ -0,0 +1,2550 @@ +//! Podcast store against PostgreSQL. +//! +//! Ignored by default: set `FURU_DEVICE_SYNC_TEST_DATABASE_URL` to a +//! disposable database (for example a throwaway `postgres:16-alpine` +//! container). The tests migrate it with the production migrations, use +//! random account ids and remove their rows again; the migration test +//! creates and drops its own database next to it. + +use std::collections::BTreeSet; + +use cot::db::migrations::{MigrationEngine, SyncDynMigration}; +use music_dht::device_sync::{SyncOpPayload, SyncOpWire, SyncSnapshot, random_hex}; +use music_dht::podcast::feed::{ParsedFeed, parse_feed_with_base}; +use music_dht::podcast::{ + EpisodeId, FeedUrl, ListenSession, PodcastEpisodeRef, PodcastId, PodcastMeta, ProgressRecord, + ProgressStatus, ShareRecord, SubscriptionRecord, +}; + +use super::discovery::Discovery; +use super::*; +use crate::player::queue_items::test_episode; + +const DATABASE_URL: &str = "FURU_DEVICE_SYNC_TEST_DATABASE_URL"; + +/// Every production migration (the list of `admin::App::migrations`). +fn all_migrations() -> Vec<&'static SyncDynMigration> { + [ + crate::config::db_migrations::MIGRATIONS, + crate::user::db_migrations::MIGRATIONS, + crate::music::db_migrations::MIGRATIONS, + crate::scheduler::db_migrations::MIGRATIONS, + crate::auth::db_migrations::MIGRATIONS, + crate::youtube::db_migrations::MIGRATIONS, + ] + .into_iter() + .flatten() + .copied() + .collect() +} + +async fn migrate(url: &str, migrations: Vec<&'static SyncDynMigration>) { + let database = cot::db::Database::new(url).await.unwrap(); + MigrationEngine::new(migrations) + .unwrap() + .run(&database) + .await + .unwrap(); + database.close().await.unwrap(); +} + +/// Pool on the test database, migrated once per test binary with the +/// production migrations (also used by the device-sync Postgres tests). +pub(crate) async fn test_pool() -> sqlx::PgPool { + static MIGRATED: tokio::sync::Mutex = tokio::sync::Mutex::const_new(false); + let url = std::env::var(DATABASE_URL).unwrap(); + let mut migrated = MIGRATED.lock().await; + if !*migrated { + migrate(&url, all_migrations()).await; + *migrated = true; + } + sqlx::PgPool::connect(&url).await.unwrap() +} + +pub(crate) fn random_user() -> i64 { + let bytes = blake3::hash(random_hex(8).as_bytes()); + 9_000_000_000 + + i64::from(u32::from_le_bytes(bytes.as_bytes()[..4].try_into().unwrap()) % 1_000_000_000) +} + +pub(crate) const ACCOUNT_TABLES: &[&str] = &[ + "furumusic__podcast_share_confirmation", + "furumusic__podcast_fetch_state", + "furumusic__podcast_listen_session", + "furumusic__podcast_progress", + "furumusic__podcast_share", + "furumusic__podcast_subscription", + "furumusic__podcast_episode", + "furumusic__podcast_feed_alias", + "furumusic__podcast", + "furumusic__fed_podcast_relay", + "furumusic__fed_device_identity", + "furumusic__fed_device", + "furumusic__fed_sync_ops", + "furumusic__fed_sync_vector", + "furumusic__fed_state_like", + "furumusic__fed_state_playlist", + "furumusic__fed_state_playlist_item", +]; + +pub(crate) async fn cleanup(pool: &sqlx::PgPool, users: impl IntoIterator) { + let users: Vec = users.into_iter().collect(); + for table in ACCOUNT_TABLES { + sqlx::query(&format!("DELETE FROM {table} WHERE user_id = ANY($1)")) + .bind(&users) + .execute(pool) + .await + .unwrap(); + } +} + +fn feed_url(path: &str) -> FeedUrl { + FeedUrl::parse(&format!("https://podcasts.example/{path}")).unwrap() +} + +fn meta(path: &str, title: &str) -> PodcastMeta { + PodcastMeta { + feed_url: feed_url(path), + title: title.into(), + author: Some("Host".into()), + description: Some("Channel description".into()), + link: None, + image_url: None, + } +} + +fn op(origin: &str, seq: i64, hlc_ms: i64, payload: SyncOpPayload) -> SyncOpWire { + SyncOpWire { + op_id: format!("{origin}:{seq}"), + origin_device_id: origin.into(), + seq, + hlc_ms, + payload, + } +} + +fn session( + id: &str, + device: &str, + episode: &PodcastEpisodeRef, + listened_ms: i64, + completed: bool, +) -> ListenSession { + ListenSession { + session_id: id.into(), + episode_id: episode.episode_id.clone(), + podcast_id: episode.podcast_id.clone(), + device_id: device.into(), + started_at_ms: 1_000, + ended_at_ms: 1_000 + listened_ms, + start_position_ms: 0, + end_position_ms: listened_ms, + listened_ms, + completed, + } +} + +fn progress_op( + origin: &str, + seq: i64, + hlc_ms: i64, + episode: &PodcastEpisodeRef, + generation: u64, + position_ms: i64, +) -> SyncOpWire { + op( + origin, + seq, + hlc_ms, + SyncOpPayload::PodcastProgressSet { + episode_id: episode.episode_id.clone(), + podcast_id: episode.podcast_id.clone(), + position_ms, + status: ProgressStatus::InProgress, + generation, + session_id: format!("{origin}-g{generation}"), + episode: Some(PodcastEpisodeRef { + progress_generation: None, + ..episode.clone() + }), + }, + ) +} + +/// Every permutation of `0..n` (Heap's algorithm). +fn permutations(n: usize) -> Vec> { + fn heap(k: usize, items: &mut Vec, out: &mut Vec>) { + if k <= 1 { + out.push(items.clone()); + return; + } + for i in 0..k { + heap(k - 1, items, out); + let j = if k.is_multiple_of(2) { i } else { 0 }; + items.swap(j, k - 1); + } + } + let mut out = Vec::new(); + heap(n, &mut (0..n).collect(), &mut out); + out +} + +async fn apply_all(store: &PodcastStore, user_id: i64, ops: &[SyncOpWire]) { + for op in ops { + let outcome = store.apply_remote_op(user_id, op).await.unwrap(); + assert!( + matches!(outcome, ApplyOutcome::Changed | ApplyOutcome::Unchanged), + "{outcome:?}" + ); + } +} + +/// The registers and sessions of one podcast and episode from three +/// devices, with a duplicate, a late checkpoint of an older session, a +/// concurrent offline action and a growing session. +fn convergence_ops() -> Vec { + let episode = test_episode(); + let podcast_id = episode.podcast_id.clone(); + let podcast = PodcastMeta { + feed_url: episode.feed_url.clone(), + title: episode.podcast_title.clone(), + author: None, + description: Some("dropped from snapshots".into()), + link: None, + image_url: None, + }; + let subscribe = op( + "tui_a", + 1, + 100, + SyncOpPayload::PodcastSubscriptionSet { + podcast_id: podcast_id.clone(), + subscribed: true, + podcast: podcast.clone(), + feed_aliases: vec![], + generation: 1, + }, + ); + vec![ + subscribe.clone(), + // B saw g1 and unsubscribes with a slow clock: g2 still wins. + op( + "tui_b", + 1, + 50, + SyncOpPayload::PodcastSubscriptionSet { + podcast_id: podcast_id.clone(), + subscribed: false, + podcast, + feed_aliases: vec![], + generation: 2, + }, + ), + // A shared concurrently at g2: stored, but not effective. + op( + "tui_a", + 2, + 300, + SyncOpPayload::PodcastShareSet { + podcast_id: podcast_id.clone(), + shared: true, + generation: 2, + }, + ), + op( + "tui_c", + 1, + 400, + SyncOpPayload::PodcastFeedAliasesAdded { + podcast_id, + feed_aliases: vec![feed_url("moved.xml")], + }, + ), + subscribe, + // A late g3 checkpoint of an old session with a fast clock. + progress_op("tui_a", 3, 9_000, &episode, 3, 600_000), + // B and C both seek at g4 with the same clock: device id decides. + progress_op("tui_b", 2, 1_000, &episode, 4, 10_000), + progress_op("tui_c", 2, 1_000, &episode, 4, 20_000), + op( + "tui_a", + 4, + 9_100, + SyncOpPayload::PodcastListenRecorded { + session: session("s1", "tui_a", &episode, 40_000, false), + episode: None, + }, + ), + op( + "tui_a", + 5, + 9_200, + SyncOpPayload::PodcastListenRecorded { + session: session("s1", "tui_a", &episode, 90_000, true), + episode: None, + }, + ), + op( + "tui_b", + 3, + 1_100, + SyncOpPayload::PodcastListenRecorded { + session: session("s2", "tui_b", &episode, 31_000, false), + episode: Some(PodcastEpisodeRef { + progress_generation: None, + ..episode + }), + }, + ), + ] +} + +#[tokio::test] +#[ignore = "requires a disposable database in FURU_DEVICE_SYNC_TEST_DATABASE_URL"] +async fn postgres_podcast_ops_converge_in_any_order_and_match_snapshot_import() { + let pool = test_pool().await; + let store = PodcastStore::new(pool.clone()); + let ops = convergence_ops(); + let reference_user = random_user(); + let mut users = vec![reference_user]; + apply_all(&store, reference_user, &ops).await; + let reference = store.export_snapshot(reference_user).await.unwrap(); + + // The contract outcome, independently of the order. + let episode = test_episode(); + assert_eq!(reference.podcast_subscriptions.len(), 1); + let subscription = &reference.podcast_subscriptions[0]; + assert!(!subscription.subscribed); + assert_eq!(subscription.generation, 2); + assert!(subscription.podcast.description.is_none()); + assert_eq!( + subscription.feed_urls(), + vec![episode.feed_url.clone(), feed_url("moved.xml")] + ); + assert!(reference.podcast_shares[0].shared); + let summary = store + .podcast(reference_user, &episode.podcast_id) + .await + .unwrap() + .unwrap(); + assert!(!summary.subscribed && !summary.shared); + assert_eq!(reference.podcast_progress.len(), 1); + assert_eq!(reference.podcast_progress[0].position_ms, 20_000); + assert_eq!(reference.podcast_progress[0].device_id, "tui_c"); + let sessions: BTreeSet<_> = reference + .podcast_listen_sessions + .iter() + .map(|session| { + ( + session.session_id.as_str(), + session.listened_ms, + session.completed, + ) + }) + .collect(); + assert_eq!( + sessions, + BTreeSet::from([("s1", 90_000, true), ("s2", 31_000, false)]) + ); + assert_eq!( + store + .completed_listen_count(reference_user, &episode.episode_id) + .await + .unwrap(), + 1 + ); + // The episode reference created a catalog row (never touching progress). + let item = store + .episode(reference_user, &episode.episode_id) + .await + .unwrap() + .unwrap(); + assert!(!item.in_feed); + assert_eq!(item.resume_position_ms(), 20_000); + + // Register operations: all 120 orders of the first five operations + // (subscription, unsubscription, share, aliases, duplicate) with the + // rest appended; progress and session operations: all 720 orders of + // the last six with the first five prepended. + let (registers, personal) = ops.split_at(5); + for order in permutations(registers.len()) { + let user_id = random_user(); + users.push(user_id); + let mut shuffled: Vec = order.iter().map(|&i| registers[i].clone()).collect(); + shuffled.extend_from_slice(personal); + apply_all(&store, user_id, &shuffled).await; + assert_eq!( + store.export_snapshot(user_id).await.unwrap(), + reference, + "{order:?}" + ); + } + for order in permutations(personal.len()) { + let user_id = random_user(); + users.push(user_id); + let mut shuffled = registers.to_vec(); + shuffled.extend(order.iter().map(|&i| personal[i].clone())); + apply_all(&store, user_id, &shuffled).await; + assert_eq!( + store.export_snapshot(user_id).await.unwrap(), + reference, + "{order:?}" + ); + } + + // A new device recovering from the snapshot reaches the same state, and + // the operations delivered afterwards change nothing. (Snapshots omit + // the channel description, so the winning subscription operation only + // adds it back to the stored register: the contract's tie-break by + // metadata. Every other operation is already reflected.) + let recovered = random_user(); + users.push(recovered); + let stats = store.apply_snapshot(recovered, &reference).await.unwrap(); + assert_eq!(stats.rejected, 0); + assert!(!stats.ignored_over_limits); + assert_eq!(store.export_snapshot(recovered).await.unwrap(), reference); + for op in ops.iter().rev() { + let expected = if op.op_id == "tui_b:1" { + ApplyOutcome::Changed + } else { + ApplyOutcome::Unchanged + }; + assert_eq!( + store.apply_remote_op(recovered, op).await.unwrap(), + expected, + "{}", + op.op_id + ); + } + assert_eq!(store.export_snapshot(recovered).await.unwrap(), reference); + // A stale snapshot never replaces newer state. + let stale = SyncSnapshot { + podcast_progress: vec![ProgressRecord { + position_ms: 1, + generation: 3, + hlc_ms: i64::MAX, + ..reference.podcast_progress[0].clone() + }], + ..SyncSnapshot::default() + }; + let stats = store.apply_snapshot(recovered, &stale).await.unwrap(); + assert_eq!((stats.changed, stats.unchanged), (0, 1)); + assert_eq!(store.export_snapshot(recovered).await.unwrap(), reference); + // Invalid operations are rejected without touching state. + let mut invalid = progress_op("tui_a", 9, 1, &episode, 9, 1); + if let SyncOpPayload::PodcastProgressSet { session_id, .. } = &mut invalid.payload { + session_id.clear(); + } + assert!(matches!( + store.apply_remote_op(recovered, &invalid).await.unwrap(), + ApplyOutcome::Rejected(_) + )); + let mut huge = progress_op("tui_a", 10, 1, &episode, u64::MAX, 1); + huge.op_id = "tui_a:10".into(); + assert!(matches!( + store.apply_remote_op(recovered, &huge).await.unwrap(), + ApplyOutcome::Rejected(_) + )); + assert_eq!(store.export_snapshot(recovered).await.unwrap(), reference); + + cleanup(&pool, users).await; +} + +#[tokio::test] +#[ignore = "requires a disposable database in FURU_DEVICE_SYNC_TEST_DATABASE_URL"] +async fn postgres_podcast_local_actions_keep_progress_through_unsubscribe() { + let pool = test_pool().await; + let store = PodcastStore::new(pool.clone()); + let user_id = random_user(); + let device = crate::federation::devices::local_device_id(&pool, user_id) + .await + .unwrap(); + let episode = test_episode(); + let show = PodcastMeta { + feed_url: episode.feed_url.clone(), + ..meta("unused.xml", &episode.podcast_title) + }; + + let subscribed = store.subscribe(user_id, &show).await.unwrap(); + assert_eq!(subscribed.podcast_id, episode.podcast_id); + assert!(!subscribed.already_subscribed); + let [first] = subscribed.ops.as_slice() else { + panic!("one operation"); + }; + assert_eq!(first.origin_device_id, device); + assert_eq!(first.op_id, format!("{device}:{}", first.seq)); + // The operation is in the log with exactly this stamp; applying it again + // is a no-op. + let logged: serde_json::Value = sqlx::query_scalar( + "SELECT payload_json FROM furumusic__fed_sync_ops + WHERE user_id = $1 AND op_id = $2 AND hlc_ms = $3 AND kind = 'podcast_subscription_set'", + ) + .bind(user_id) + .bind(&first.op_id) + .bind(first.hlc_ms) + .fetch_one(&pool) + .await + .unwrap(); + assert_eq!(logged, serde_json::to_value(&first.payload).unwrap()); + assert_eq!( + store.apply_remote_op(user_id, first).await.unwrap(), + ApplyOutcome::Unchanged + ); + assert!( + store + .subscribe(user_id, &show) + .await + .unwrap() + .already_subscribed + ); + assert_eq!( + store + .set_shared(user_id, &episode.podcast_id, true) + .await + .unwrap() + .len(), + 1 + ); + assert!( + store + .set_shared(user_id, &episode.podcast_id, true) + .await + .unwrap() + .is_empty() + ); + assert!( + store + .podcast(user_id, &episode.podcast_id) + .await + .unwrap() + .unwrap() + .shared + ); + + // Listening: an explicit start, checkpoints keep the generation. + let start = store + .start_session(user_id, &episode, "web-s1", 0, episode.progress_generation) + .await + .unwrap(); + assert_eq!(start.record.generation, 8, "max(local, carried 7) + 1"); + let CheckpointOutcome::Written(checkpoint) = store + .checkpoint(user_id, &start.session, 30_000) + .await + .unwrap() + else { + panic!("checkpoint written"); + }; + assert_eq!(checkpoint.record.generation, 8); + assert_eq!( + store + .resume_position_ms(user_id, &episode.episode_id) + .await + .unwrap(), + 30_000 + ); + // A seek is explicit: later checkpoints use its generation. + let seek = store + .seek(user_id, &checkpoint.session, 12_000) + .await + .unwrap(); + assert_eq!(seek.record.generation, 9); + assert_eq!(seek.session.generation, 9); + let start = seek; + // Another device seeks back at a greater generation with a slow clock: + // later checkpoints of the old session are refused. + store + .apply_remote_op(user_id, &progress_op("tui_b", 1, 1, &episode, 10, 5_000)) + .await + .unwrap(); + assert!(matches!( + store + .checkpoint(user_id, &start.session, 900_000) + .await + .unwrap(), + CheckpointOutcome::Superseded { .. } + )); + assert_eq!( + store + .resume_position_ms(user_id, &episode.episode_id) + .await + .unwrap(), + 5_000 + ); + let listened = ListenSession { + device_id: device.clone(), + ..session("web-s1", &device, &episode, 45_000, false) + }; + assert!( + store + .record_listen_session(user_id, &listened, None) + .await + .unwrap() + .is_some() + ); + assert!( + store + .record_listen_session(user_id, &listened, None) + .await + .unwrap() + .is_none() + ); + assert!( + store + .record_listen_session( + user_id, + &session("x", "tui_b", &episode, 45_000, false), + None + ) + .await + .is_err() + ); + + // Unsubscribing keeps progress and history and revokes the share. + let ops = store + .unsubscribe(user_id, &episode.podcast_id) + .await + .unwrap(); + assert_eq!(ops.len(), 2); + assert!( + store + .unsubscribe(user_id, &episode.podcast_id) + .await + .unwrap() + .is_empty() + ); + let summary = store + .podcast(user_id, &episode.podcast_id) + .await + .unwrap() + .unwrap(); + assert!(!summary.subscribed && !summary.shared); + assert_eq!( + store + .resume_position_ms(user_id, &episode.episode_id) + .await + .unwrap(), + 5_000 + ); + assert_eq!( + store.listen_history(user_id, None, 10).await.unwrap().len(), + 1 + ); + assert!( + store + .list_podcasts(user_id, PodcastListFilter::Subscribed) + .await + .unwrap() + .is_empty() + ); + assert_eq!( + store + .list_podcasts(user_id, PodcastListFilter::All) + .await + .unwrap() + .len(), + 1 + ); + + // Subscribing again resolves to the same podcast, continues the + // progress and is private again. + let again = store.subscribe(user_id, &show).await.unwrap(); + assert_eq!(again.podcast_id, episode.podcast_id); + let register = store + .subscription(user_id, &episode.podcast_id) + .await + .unwrap() + .unwrap(); + let share = store + .share(user_id, &episode.podcast_id) + .await + .unwrap() + .unwrap(); + assert!(register.subscribed); + assert!(register.generation > share.generation); + let summary = store + .podcast(user_id, &episode.podcast_id) + .await + .unwrap() + .unwrap(); + assert!(summary.subscribed && !summary.shared); + assert_eq!( + store + .resume_position_ms(user_id, &episode.episode_id) + .await + .unwrap(), + 5_000 + ); + + // Mark played writes progress only; restart and unplayed reset it. + let played = store.mark_played(user_id, &episode).await.unwrap(); + assert_eq!(played.record.status, ProgressStatus::Completed); + assert_eq!(played.record.generation, 11); + assert_eq!( + store.listen_history(user_id, None, 10).await.unwrap().len(), + 1 + ); + assert_eq!( + store + .resume_position_ms(user_id, &episode.episode_id) + .await + .unwrap(), + 0 + ); + let restarted = store.restart(user_id, &episode, "web-s2").await.unwrap(); + assert_eq!( + ( + restarted.record.position_ms, + restarted.record.status, + restarted.record.generation + ), + (0, ProgressStatus::InProgress, 12) + ); + let CheckpointOutcome::Written(finished) = store + .complete_if_current(user_id, &restarted.session, 1_790_000) + .await + .unwrap() + else { + panic!("the running session completes the episode"); + }; + assert_eq!( + (finished.record.status, finished.record.generation), + (ProgressStatus::Completed, 13) + ); + assert!(matches!( + store + .checkpoint(user_id, &finished.session, 1_795_000) + .await + .unwrap(), + CheckpointOutcome::AlreadyCompleted { .. } + )); + let mut played_filter = EpisodeQuery::new(episode.podcast_id.clone()); + played_filter.played = PlayedFilter::Played; + assert_eq!( + store.episodes(user_id, &played_filter).await.unwrap().len(), + 1 + ); + played_filter.played = PlayedFilter::Unplayed; + assert!( + store + .episodes(user_id, &played_filter) + .await + .unwrap() + .is_empty() + ); + assert_eq!( + store + .mark_unplayed(user_id, &episode) + .await + .unwrap() + .record + .status, + ProgressStatus::NotStarted + ); + played_filter.played = PlayedFilter::NotStarted; + let items = store.episodes(user_id, &played_filter).await.unwrap(); + assert_eq!(items[0].status(), ProgressStatus::NotStarted); + + // Local stamps: consecutive sequence numbers, strictly growing clock. + let stamps: Vec<(i64, i64)> = sqlx::query_as( + "SELECT seq, hlc_ms FROM furumusic__fed_sync_ops + WHERE user_id = $1 AND origin_device_id = $2 ORDER BY seq", + ) + .bind(user_id) + .bind(&device) + .fetch_all(&pool) + .await + .unwrap(); + assert!( + stamps + .windows(2) + .all(|pair| pair[1].0 == pair[0].0 + 1 && pair[1].1 > pair[0].1) + ); + // Re-applying the whole local log changes nothing. + let before = store.export_snapshot(user_id).await.unwrap(); + let payloads: Vec<(String, i64, i64, serde_json::Value)> = sqlx::query_as( + "SELECT op_id, seq, hlc_ms, payload_json FROM furumusic__fed_sync_ops + WHERE user_id = $1 AND kind LIKE 'podcast\\_%' ORDER BY hlc_ms", + ) + .bind(user_id) + .fetch_all(&pool) + .await + .unwrap(); + for (op_id, seq, hlc_ms, payload) in payloads { + let origin = op_id.rsplit_once(':').unwrap().0.to_string(); + let op = SyncOpWire { + op_id, + origin_device_id: origin, + seq, + hlc_ms, + payload: serde_json::from_value(payload).unwrap(), + }; + assert_eq!( + store.apply_remote_op(user_id, &op).await.unwrap(), + ApplyOutcome::Unchanged + ); + } + assert_eq!(store.export_snapshot(user_id).await.unwrap(), before); + + // Sharing needs a subscription. + let unknown = PodcastId::from_feed_url(&feed_url("unknown.xml")); + let error = store.set_shared(user_id, &unknown, true).await.unwrap_err(); + assert!(matches!( + error.downcast_ref::(), + Some(PodcastStoreError::NotSubscribed(_)) + )); + + cleanup(&pool, [user_id]).await; +} + +fn rss(items: &str) -> String { + format!( + r#" + + + Refresh Show + https://show.example/ + Channel text + Host + {items} + +"# + ) +} + +fn parse(xml: &str, url: &FeedUrl) -> ParsedFeed { + parse_feed_with_base(xml.as_bytes(), Some(url.as_str())).unwrap() +} + +#[tokio::test] +#[ignore = "requires a disposable database in FURU_DEVICE_SYNC_TEST_DATABASE_URL"] +async fn postgres_podcast_feed_refresh_keeps_ids_and_progress() { + let pool = test_pool().await; + let store = PodcastStore::new(pool.clone()); + let user_id = random_user(); + let url = feed_url("refresh.xml"); + let podcast_id = PodcastId::from_feed_url(&url); + let v1 = parse( + &rss(r#" + g-1One1 + 1Mon, 01 Jan 2024 10:00:00 GMT + + Twohttps://show.example/ep2 + 12 + Mon, 08 Jan 2024 10:00:00 GMT + Talk about Ünïcode + + ThreeMon, 15 Jan 2024 10:00:00 GMT + "#), + &url, + ); + // What every device derives for this document. + let ids: Vec = v1 + .resolve_episode_ids(&podcast_id, &[]) + .into_iter() + .map(Option::unwrap) + .collect(); + + // Another device played the GUID-less "Two" before this account ever + // fetched the feed: its reference arrives with the progress. + let two = PodcastEpisodeRef { + podcast_id: podcast_id.clone(), + episode_id: ids[1].clone(), + feed_url: url.clone(), + podcast_title: "Refresh Show".into(), + episode_title: "Two".into(), + audio_url: "https://cdn.example/a2.mp3".into(), + mime: None, + duration_ms: None, + published_at_ms: None, + season: None, + episode_number: None, + image_url: None, + progress_generation: None, + }; + store + .apply_remote_op(user_id, &progress_op("tui_a", 1, 10, &two, 3, 61_000)) + .await + .unwrap(); + let subscribed = store + .subscribe(user_id, &v1.to_podcast_meta(url.clone())) + .await + .unwrap(); + assert_eq!(subscribed.podcast_id, podcast_id); + let first = store + .apply_feed( + user_id, + &FeedRefresh { + podcast_id: &podcast_id, + feed: &v1, + fetched_from: &url, + moved_permanently: false, + }, + ) + .await + .unwrap(); + assert_eq!((first.new_episodes, first.updated_episodes), (2, 1)); + assert!(first.ops.is_empty()); + let stored: BTreeSet = store + .episodes(user_id, &EpisodeQuery::new(podcast_id.clone())) + .await + .unwrap() + .into_iter() + .map(|item| item.episode.episode_id) + .collect(); + assert_eq!(stored, ids.iter().cloned().collect()); + let progress_before = store.progress(user_id, &ids[1]).await.unwrap().unwrap(); + + // The publisher renames episodes, moves their audio, drops "Three", + // adds "Four" and moves the feed permanently. + let moved = feed_url("moved-refresh.xml"); + let v2 = parse( + &rss(r#" + g-1One (remastered)1 + 1Mon, 01 Jan 2024 10:00:00 GMT + + Two, revisedhttps://show.example/ep2 + 12 + Mon, 08 Jan 2024 10:00:00 GMT + Talk about Ünïcode + + g-4Four2 + 1Mon, 22 Jan 2024 10:00:00 GMT + "#), + &moved, + ); + let refresh = FeedRefresh { + podcast_id: &podcast_id, + feed: &v2, + fetched_from: &moved, + moved_permanently: true, + }; + let second = store.apply_feed(user_id, &refresh).await.unwrap(); + assert_eq!( + ( + second.new_episodes, + second.updated_episodes, + second.unchanged_episodes + ), + (1, 2, 0) + ); + assert_eq!(second.ops.len(), 1, "aliases changed once"); + assert_eq!(second.feed_url.as_ref(), Some(&moved)); + let again = store.apply_feed(user_id, &refresh).await.unwrap(); + assert!(again.ops.is_empty()); + assert_eq!(again.unchanged_episodes, 3); + assert_eq!( + store.resolve_podcast_id(user_id, &moved).await.unwrap(), + podcast_id + ); + assert_eq!( + store.resolve_podcast_id(user_id, &url).await.unwrap(), + podcast_id + ); + + // Ids and progress survive; the dropped episode stays out of the feed. + let one = store.episode(user_id, &ids[0]).await.unwrap().unwrap(); + assert_eq!(one.episode.title, "One (remastered)"); + assert_eq!(one.episode.audio_url, "https://cdn2.example/a1.mp3"); + let two_item = store.episode(user_id, &ids[1]).await.unwrap().unwrap(); + assert_eq!(two_item.episode.audio_url, "https://cdn2.example/a2.mp3"); + assert_eq!(two_item.progress.as_ref(), Some(&progress_before)); + assert_eq!(two_item.resume_position_ms(), 61_000); + let three = store.episode(user_id, &ids[2]).await.unwrap().unwrap(); + assert!(!three.in_feed); + // The subscription register is untouched by refreshes. + let register = store + .subscription(user_id, &podcast_id) + .await + .unwrap() + .unwrap(); + assert_eq!(register.op_id, subscribed.ops[0].op_id); + assert!(register.feed_urls().contains(&moved)); + let reference = store.episode_ref(user_id, &ids[1]).await.unwrap().unwrap(); + assert_eq!(reference.progress_generation, Some(3)); + assert_eq!(reference.audio_url, "https://cdn2.example/a2.mp3"); + + // UI queries: counters, seasons, filters, search and sorting. + let summary = store.podcast(user_id, &podcast_id).await.unwrap().unwrap(); + assert_eq!( + ( + summary.episode_count, + summary.unplayed_count, + summary.in_progress_count, + summary.season_count + ), + (4, 4, 1, 2) + ); + assert_eq!(summary.meta.feed_url, moved); + let seasons = store.seasons(user_id, &podcast_id).await.unwrap(); + assert_eq!( + seasons + .iter() + .map(|season| (season.season, season.episode_count)) + .collect::>(), + vec![(Some(1), 2), (Some(2), 1), (None, 1)] + ); + let query = |configure: &dyn Fn(&mut EpisodeQuery)| { + let mut query = EpisodeQuery::new(podcast_id.clone()); + configure(&mut query); + query + }; + let titles = |items: Vec| { + items + .into_iter() + .map(|item| item.episode.title) + .collect::>() + }; + assert_eq!( + titles(store.episodes(user_id, &query(&|_| {})).await.unwrap()), + ["Four", "Three", "Two, revised", "One (remastered)"] + ); + assert_eq!( + titles( + store + .episodes(user_id, &query(&|q| q.sort = EpisodeSort::EpisodeNumber)) + .await + .unwrap() + ), + ["One (remastered)", "Two, revised", "Four", "Three"] + ); + assert_eq!( + titles( + store + .episodes(user_id, &query(&|q| q.season = SeasonFilter::Season(1))) + .await + .unwrap() + ), + ["Two, revised", "One (remastered)"] + ); + assert_eq!( + titles( + store + .episodes(user_id, &query(&|q| q.season = SeasonFilter::NoSeason)) + .await + .unwrap() + ), + ["Three"] + ); + assert_eq!( + titles( + store + .episodes(user_id, &query(&|q| q.played = PlayedFilter::InProgress)) + .await + .unwrap() + ), + ["Two, revised"] + ); + assert_eq!( + titles( + store + .episodes(user_id, &query(&|q| q.search = Some("ÜNÏCODE".into()))) + .await + .unwrap() + ), + ["Two, revised"] + ); + assert_eq!( + titles( + store + .episodes(user_id, &query(&|q| q.search = Some("remaster".into()))) + .await + .unwrap() + ), + ["One (remastered)"] + ); + assert_eq!( + titles( + store + .episodes( + user_id, + &query(&|q| { + q.limit = Some(1); + q.offset = 1; + }) + ) + .await + .unwrap() + ), + ["Three"] + ); + assert!( + store + .episodes(user_id, &query(&|q| q.search = Some("one two".into()))) + .await + .unwrap() + .is_empty() + ); + + // A refresh of a podcast this account does not know is refused. + let unknown = PodcastId::from_feed_url(&feed_url("never.xml")); + let error = store + .apply_feed( + user_id, + &FeedRefresh { + podcast_id: &unknown, + ..refresh + }, + ) + .await + .unwrap_err(); + assert!(matches!( + error.downcast_ref::(), + Some(PodcastStoreError::UnknownPodcast(_)) + )); + + cleanup(&pool, [user_id]).await; +} + +#[tokio::test] +#[ignore = "requires a disposable database in FURU_DEVICE_SYNC_TEST_DATABASE_URL"] +async fn postgres_podcast_fetch_state_is_local_only() { + let pool = test_pool().await; + let store = PodcastStore::new(pool.clone()); + let user_id = random_user(); + let due_show = meta("due.xml", "Due"); + let later_show = meta("later.xml", "Later"); + let gone_show = meta("gone.xml", "Gone"); + let mut ids = Vec::new(); + for show in [&due_show, &later_show, &gone_show] { + ids.push(store.subscribe(user_id, show).await.unwrap().podcast_id); + } + store.unsubscribe(user_id, &ids[2]).await.unwrap(); + let before = serde_json::to_string(&store.export_snapshot(user_id).await.unwrap()).unwrap(); + + let now = 1_800_000_000_000; + store + .record_fetch_success( + user_id, + &ids[1], + now, + Some("\"secret-etag\""), + Some("Tue"), + Some(now + 60_000), + ) + .await + .unwrap(); + assert_eq!( + store + .record_fetch_failure(user_id, &ids[0], now, "boom: 503", Some(now - 1)) + .await + .unwrap(), + 1 + ); + assert_eq!( + store + .record_fetch_failure(user_id, &ids[0], now, "boom: 503", Some(now - 1)) + .await + .unwrap(), + 2 + ); + store + .set_cover_cache( + user_id, + &ids[0], + Some("https://img.example/c.jpg"), + Some("/var/cache/cover-secret.jpg"), + ) + .await + .unwrap(); + let state = store.fetch_state(user_id, &ids[0]).await.unwrap(); + assert_eq!(state.consecutive_failures, 2); + assert_eq!(state.last_error.as_deref(), Some("boom: 503")); + assert_eq!( + state.cover_path.as_deref(), + Some("/var/cache/cover-secret.jpg") + ); + store + .record_fetch_not_modified(user_id, &ids[1], now + 1, Some(now + 60_000)) + .await + .unwrap(); + let later = store.fetch_state(user_id, &ids[1]).await.unwrap(); + assert_eq!(later.etag.as_deref(), Some("\"secret-etag\"")); + assert_eq!(later.last_success_at_ms, Some(now + 1)); + + // Fetch state changes nothing in the synced state. + let after = serde_json::to_string(&store.export_snapshot(user_id).await.unwrap()).unwrap(); + assert_eq!(after, before); + for secret in ["secret-etag", "boom", "cover-secret", "Channel description"] { + assert!(!after.contains(secret), "{secret}"); + } + let ops: Vec = sqlx::query_scalar( + "SELECT payload_json::text FROM furumusic__fed_sync_ops WHERE user_id = $1", + ) + .bind(user_id) + .fetch_all(&pool) + .await + .unwrap(); + assert!( + ops.iter() + .all(|op| !op.contains("secret-etag") && !op.contains("cover-secret")) + ); + + // Scheduling: overdue first, future and unsubscribed podcasts skipped. + let due: Vec = store + .podcasts_due_for_refresh(now, 10_000) + .await + .unwrap() + .into_iter() + .filter(|(user, ..)| *user == user_id) + .map(|(_, id, _)| id) + .collect(); + assert_eq!(due, vec![ids[0].clone()]); + let unknown = PodcastId::from_feed_url(&feed_url("unknown-fetch.xml")); + assert!( + store + .record_fetch_failure(user_id, &unknown, now, "x", None) + .await + .is_err() + ); + + cleanup(&pool, [user_id]).await; +} + +/// Relay rows exactly as 0.13.0 stored them (`podcast_relay`): the contract +/// records as JSON, keyed by section and entity. +async fn insert_relay( + pool: &sqlx::PgPool, + user_id: i64, + section: &str, + key: &str, + value: serde_json::Value, +) { + sqlx::query( + "INSERT INTO furumusic__fed_podcast_relay + (user_id, section, record_key, record_json, updated_at_ms) + VALUES ($1, $2, $3, $4, 1700000000000)", + ) + .bind(user_id) + .bind(section) + .bind(key) + .bind(value) + .execute(pool) + .await + .unwrap(); +} + +async fn insert_logged_op(pool: &sqlx::PgPool, user_id: i64, op: &SyncOpWire, kind: &str) { + sqlx::query( + "INSERT INTO furumusic__fed_sync_ops + (user_id, op_id, origin_device_id, seq, kind, payload_json, hlc_ms, + received_at_ms, tombstone) + VALUES ($1, $2, $3, $4, $5, $6, $7, 1700000000000, FALSE)", + ) + .bind(user_id) + .bind(&op.op_id) + .bind(&op.origin_device_id) + .bind(op.seq) + .bind(kind) + .bind(serde_json::to_value(&op.payload).unwrap()) + .bind(op.hlc_ms) + .execute(pool) + .await + .unwrap(); +} + +#[tokio::test] +#[ignore = "requires a disposable database in FURU_DEVICE_SYNC_TEST_DATABASE_URL"] +async fn postgres_podcast_migrations_import_the_relay_state() { + let admin_url = std::env::var(DATABASE_URL).unwrap(); + let admin = sqlx::PgPool::connect(&admin_url).await.unwrap(); + let name = format!("furu_podcast_migration_{}", random_hex(6)); + sqlx::query(&format!("CREATE DATABASE {name}")) + .execute(&admin) + .await + .unwrap(); + let (base, _) = admin_url.rsplit_once('/').unwrap(); + let url = format!("{base}/{name}"); + + // A 0.13.0 database: every migration up to the relay table. + let new_migrations = [ + "m_0056_create_podcast_store", + "m_0057_import_podcast_relay", + "m_0058_create_podcast_share_confirmation", + ]; + let old: Vec<_> = all_migrations() + .into_iter() + .filter(|migration| { + !(migration.app_name() == "furumusic" && new_migrations.contains(&migration.name())) + }) + .collect(); + migrate(&url, old).await; + let pool = sqlx::PgPool::connect(&url).await.unwrap(); + + // State relayed by 0.13.0 for two accounts. + let episode = test_episode(); + let user_a = 1; + let user_b = 2; + let subscription = SubscriptionRecord { + podcast_id: episode.podcast_id.clone(), + subscribed: true, + podcast: PodcastMeta { + feed_url: episode.feed_url.clone(), + title: episode.podcast_title.clone(), + author: Some("Host".into()), + description: None, + link: None, + image_url: Some("https://img.example/show.jpg".into()), + }, + feed_aliases: vec![episode.feed_url.clone(), feed_url("old.xml")], + generation: 3, + hlc_ms: 30, + op_id: "tui_a:3".into(), + }; + let share = ShareRecord { + podcast_id: episode.podcast_id.clone(), + shared: false, + generation: 2, + hlc_ms: 20, + op_id: "tui_a:2".into(), + }; + let progress = ProgressRecord { + episode_id: episode.episode_id.clone(), + podcast_id: episode.podcast_id.clone(), + position_ms: 1_200_000, + status: ProgressStatus::InProgress, + generation: 9, + session_id: "tui_a-g9".into(), + hlc_ms: 90, + device_id: "tui_a".into(), + op_id: "tui_a:9".into(), + }; + let listened = session("s9", "tui_a", &episode, 120_000, true); + let pending_podcast = PodcastId::from_feed_url(&feed_url("pending.xml")); + insert_relay( + &pool, + user_a, + "subscription", + subscription.podcast_id.as_str(), + serde_json::to_value(&subscription).unwrap(), + ) + .await; + insert_relay( + &pool, + user_a, + "share", + share.podcast_id.as_str(), + serde_json::to_value(&share).unwrap(), + ) + .await; + insert_relay( + &pool, + user_a, + "progress", + progress.episode_id.as_str(), + serde_json::to_value(&progress).unwrap(), + ) + .await; + insert_relay( + &pool, + user_a, + "session", + "s9", + serde_json::to_value(&listened).unwrap(), + ) + .await; + insert_relay( + &pool, + user_a, + "aliases", + pending_podcast.as_str(), + serde_json::to_value(vec![feed_url("pending.xml")]).unwrap(), + ) + .await; + // An out-of-range generation (no conforming client) is skipped. + let huge = ProgressRecord { + episode_id: EpisodeId::from_guid(&episode.podcast_id, "huge").unwrap(), + generation: u64::MAX, + ..progress.clone() + }; + insert_relay( + &pool, + user_a, + "progress", + huge.episode_id.as_str(), + serde_json::to_value(&huge).unwrap(), + ) + .await; + insert_relay( + &pool, + user_b, + "progress", + progress.episode_id.as_str(), + serde_json::to_value(&ProgressRecord { + position_ms: 5, + ..progress.clone() + }) + .unwrap(), + ) + .await; + // The relayed operations kept the episode references (two versions). + let old_ref = PodcastEpisodeRef { + episode_title: "Old title".into(), + progress_generation: None, + ..episode.clone() + }; + insert_logged_op( + &pool, + user_a, + &progress_op("tui_a", 8, 80, &old_ref, 8, 1_000), + "podcast_progress_set", + ) + .await; + insert_logged_op( + &pool, + user_a, + &progress_op("tui_a", 9, 90, &episode, 9, 1_200_000), + "podcast_progress_set", + ) + .await; + + // Upgrade. + migrate(&url, all_migrations()).await; + let store = PodcastStore::new(pool.clone()); + + let mut expected = SyncSnapshot { + podcast_subscriptions: vec![subscription.clone()], + podcast_shares: vec![share], + podcast_progress: vec![progress.clone()], + podcast_listen_sessions: vec![listened], + ..SyncSnapshot::default() + }; + expected.bound_podcast_state(); + assert_eq!(store.export_snapshot(user_a).await.unwrap(), expected); + assert_eq!( + store + .export_snapshot(user_b) + .await + .unwrap() + .podcast_progress[0] + .position_ms, + 5 + ); + assert_eq!( + store.feed_aliases(user_a, &pending_podcast).await.unwrap(), + vec![feed_url("pending.xml")] + ); + // Catalog rows: the podcast from its register, the episode from the + // newest reference in the log. + let summary = store + .podcast(user_a, &episode.podcast_id) + .await + .unwrap() + .unwrap(); + assert!(summary.subscribed); + assert_eq!( + summary.meta.image_url.as_deref(), + Some("https://img.example/show.jpg") + ); + let item = store + .episode(user_a, &episode.episode_id) + .await + .unwrap() + .unwrap(); + assert_eq!(item.episode.title, episode.episode_title); + assert_eq!(item.episode.season, episode.season); + assert_eq!(item.resume_position_ms(), 1_200_000); + assert_eq!(item.completed_count, 1); + // Imported rows are searchable once their search text is computed. + let mut search = EpisodeQuery::new(episode.podcast_id.clone()); + search.search = Some("episode 42".into()); + assert_eq!(store.episodes(user_a, &search).await.unwrap().len(), 1); + // The store keeps working on imported rows, and the relay table stays. + assert_eq!( + store + .apply_remote_op(user_a, &progress_op("tui_a", 9, 90, &episode, 9, 1_200_000)) + .await + .unwrap(), + ApplyOutcome::Unchanged + ); + let relay_rows: i64 = sqlx::query_scalar("SELECT COUNT(*) FROM furumusic__fed_podcast_relay") + .fetch_one(&pool) + .await + .unwrap(); + assert_eq!(relay_rows, 7); + // Running the import again (or the migrations) changes nothing. + for sql in schema::import_statements("TRUE") { + sqlx::query(&sql).execute(&pool).await.unwrap(); + } + migrate(&url, all_migrations()).await; + assert_eq!(store.export_snapshot(user_a).await.unwrap(), expected); + + pool.close().await; + sqlx::query(&format!("DROP DATABASE {name} WITH (FORCE)")) + .execute(&admin) + .await + .unwrap(); +} + +/// Alias sets are compared and exported in byte order (as +/// `merge_feed_aliases` returns them) whatever the database collation is: +/// an ICU database sorts "a" before "B", bytes sort "B" first. +#[tokio::test] +#[ignore = "requires a disposable database in FURU_DEVICE_SYNC_TEST_DATABASE_URL"] +async fn postgres_podcast_alias_sets_ignore_the_database_collation() { + let admin_url = std::env::var(DATABASE_URL).unwrap(); + let admin = sqlx::PgPool::connect(&admin_url).await.unwrap(); + let name = format!("furu_podcast_icu_{}", random_hex(6)); + sqlx::query(&format!( + "CREATE DATABASE {name} TEMPLATE template0 LOCALE_PROVIDER icu ICU_LOCALE 'en-US' LOCALE 'C'" + )) + .execute(&admin) + .await + .unwrap(); + let (base, _) = admin_url.rsplit_once('/').unwrap(); + let url = format!("{base}/{name}"); + migrate(&url, all_migrations()).await; + let pool = sqlx::PgPool::connect(&url).await.unwrap(); + let ordered: Vec = + sqlx::query_scalar("SELECT array_agg(x ORDER BY x) FROM (VALUES ('B'), ('a')) v(x)") + .fetch_one(&pool) + .await + .unwrap(); + assert_eq!(ordered, ["a", "B"], "the database must not sort bytes"); + + let store = PodcastStore::new(pool.clone()); + let user_id = 1; + let feed = feed_url("feed.xml"); + let podcast_id = PodcastId::from_feed_url(&feed); + let mixed = vec![feed_url("a"), feed_url("B"), feed_url("a-c")]; + let mut sorted = mixed.clone(); + sorted.push(feed.clone()); + sorted.sort(); + let subscribe = op( + "tui_a", + 1, + 100, + SyncOpPayload::PodcastSubscriptionSet { + podcast_id: podcast_id.clone(), + subscribed: true, + podcast: PodcastMeta { + feed_url: feed.clone(), + title: "Refresh Show".into(), + author: None, + description: None, + link: None, + image_url: None, + }, + feed_aliases: mixed.clone(), + generation: 1, + }, + ); + let aliases = op( + "tui_a", + 2, + 200, + SyncOpPayload::PodcastFeedAliasesAdded { + podcast_id: podcast_id.clone(), + feed_aliases: mixed.clone(), + }, + ); + assert_eq!( + store.apply_remote_op(user_id, &subscribe).await.unwrap(), + ApplyOutcome::Changed + ); + for op in [&subscribe, &aliases, &aliases, &subscribe] { + assert_eq!( + store.apply_remote_op(user_id, op).await.unwrap(), + ApplyOutcome::Unchanged, + "{}", + op.op_id + ); + } + assert_eq!( + store.feed_aliases(user_id, &podcast_id).await.unwrap(), + sorted + ); + let snapshot = store.export_snapshot(user_id).await.unwrap(); + assert_eq!(snapshot.podcast_subscriptions[0].feed_aliases, sorted); + + // A permanent move to an alias already known emits nothing. + let document = parse( + &rss(r#" + g-1One + "#), + &feed_url("B"), + ); + let refresh = store + .apply_feed( + user_id, + &FeedRefresh { + podcast_id: &podcast_id, + feed: &document, + fetched_from: &feed_url("B"), + moved_permanently: true, + }, + ) + .await + .unwrap(); + assert!(refresh.ops.is_empty(), "{:?}", refresh.ops); + assert_eq!( + store.feed_aliases(user_id, &podcast_id).await.unwrap(), + sorted + ); + + pool.close().await; + sqlx::query(&format!("DROP DATABASE {name} WITH (FORCE)")) + .execute(&admin) + .await + .unwrap(); +} + +// --------------------------------------------------------------------------- +// RSS refresh job +// --------------------------------------------------------------------------- + +fn refresh_item(guid: &str) -> String { + format!( + r#"Episode {guid}{guid} + "# + ) +} + +/// A fetcher that may reach the local test server (loopback is allowed +/// only by the test policy). +pub(crate) fn local_fetcher() -> FeedFetcher { + FeedFetcher::with_http( + net::SafeHttp::with_lookup( + fetch::FEED_HTTP, + net::AddressPolicy::test_allow_loopback(), + std::sync::Arc::new(net::SystemLookup), + ) + .unwrap(), + ) +} + +#[tokio::test] +#[ignore = "requires a disposable database in FURU_DEVICE_SYNC_TEST_DATABASE_URL"] +async fn postgres_podcast_refresh_takes_accounts_in_turns() { + let pool = test_pool().await; + let store = PodcastStore::new(pool.clone()); + let (busy, quiet) = (random_user(), random_user()); + let now = 1_900_000_000_000; + let mut busy_ids = Vec::new(); + for (index, path) in ["turn-a.xml", "turn-b.xml", "turn-c.xml"] + .iter() + .enumerate() + { + let id = store + .subscribe(busy, &meta(path, "Busy")) + .await + .unwrap() + .podcast_id; + // The busy account's podcasts are the most overdue ones. + store + .record_fetch_failure(busy, &id, now, "x", Some(now - 10_000 + index as i64)) + .await + .unwrap(); + busy_ids.push(id); + } + let quiet_id = store + .subscribe(quiet, &meta("turn-q.xml", "Quiet")) + .await + .unwrap() + .podcast_id; + store + .record_fetch_failure(quiet, &quiet_id, now, "x", Some(now - 1)) + .await + .unwrap(); + let due = store + .podcasts_due_for_refresh(now, 100_000) + .await + .unwrap() + .into_iter() + .filter(|(user, ..)| [busy, quiet].contains(user)) + .map(|(user, id, _)| (user, id)) + .collect::>(); + // One turn per account: the quiet account's only feed comes before the + // busy account's second one, and the busy account still gets all of + // its due feeds when the run has room (no per-account cap). + assert_eq!( + due, + vec![ + (busy, busy_ids[0].clone()), + (quiet, quiet_id.clone()), + (busy, busy_ids[1].clone()), + (busy, busy_ids[2].clone()), + ], + ); + // With only one account due, a run takes up to the run limit from it. + let alone = store + .podcasts_due_for_refresh(now, 100_000) + .await + .unwrap() + .into_iter() + .filter(|(user, ..)| *user == busy) + .count(); + assert_eq!(alone, 3); + + // New subscriptions stop at the per-account limit. + sqlx::query( + "INSERT INTO furumusic__podcast_subscription + (user_id, podcast_id, subscribed, podcast_json, generation, hlc_ms, op_id) + SELECT $1, 'pod1:cap' || n, TRUE, '{}'::jsonb, 1, 1, 'x:' || n + FROM generate_series(1, $2) AS n", + ) + .bind(quiet) + .bind(i32::try_from(MAX_SUBSCRIPTIONS_PER_ACCOUNT - 1).unwrap()) + .execute(&pool) + .await + .unwrap(); + let error = store + .subscribe(quiet, &meta("turn-over.xml", "Over")) + .await + .unwrap_err(); + assert_eq!( + error.downcast_ref::(), + Some(&PodcastStoreError::SubscriptionLimit( + MAX_SUBSCRIPTIONS_PER_ACCOUNT + )) + ); + // Subscribing again to a podcast already subscribed is not new. + assert!( + store + .subscribe(quiet, &meta("turn-q.xml", "Quiet")) + .await + .unwrap() + .already_subscribed + ); + + cleanup(&pool, [busy, quiet]).await; +} + +/// Due groups of the given accounts only (other tests share the database). +async fn due_for(store: &PodcastStore, users: &[i64], now: i64) -> runner::RefreshGroups { + let due = store + .podcasts_due_for_refresh(now, 100_000) + .await + .unwrap() + .into_iter() + .filter(|(user_id, _, _)| users.contains(user_id)) + .collect(); + runner::group_targets(due) +} + +async fn run_due( + store: &PodcastStore, + fetcher: &FeedFetcher, + users: &[i64], + now: i64, +) -> runner::RunSummary { + let groups = due_for(store, users, now).await; + runner::refresh_groups(store, fetcher, groups, now, 4) + .await + .unwrap() +} + +#[tokio::test] +#[ignore = "requires a disposable database in FURU_DEVICE_SYNC_TEST_DATABASE_URL"] +async fn postgres_podcast_refresh_job_schedules_backs_off_and_moves_feeds() { + use super::runner::{BACKOFF_BASE_MS, ManualRefreshError, TargetResult}; + use super::test_server::{Route, TestServer}; + + let pool = test_pool().await; + let store = PodcastStore::new(pool.clone()); + let (a, b, c) = (random_user(), random_user(), random_user()); + cleanup(&pool, [a, b, c]).await; + let users = [a, b, c]; + let server = TestServer::start(); + server.set( + "/show.xml", + Route::new(rss(&refresh_item("one")), "application/rss+xml").etag("\"v1\""), + ); + let show = PodcastMeta { + feed_url: FeedUrl::parse(&server.url("/show.xml")).unwrap(), + title: "Local Show".into(), + author: None, + description: None, + link: None, + image_url: None, + }; + let podcast_id = store.subscribe(a, &show).await.unwrap().podcast_id; + assert_eq!( + store.subscribe(b, &show).await.unwrap().podcast_id, + podcast_id + ); + let fetcher = local_fetcher(); + + // First run: one request serves both accounts. + let t0 = super::now_ms(); + let summary = run_due(&store, &fetcher, &users, t0).await; + assert_eq!((summary.targets, summary.feeds), (2, 1), "{summary}"); + assert_eq!((summary.updated, summary.new_episodes), (2, 2), "{summary}"); + assert_eq!(server.requests("/show.xml").len(), 1); + for user in [a, b] { + let state = store.fetch_state(user, &podcast_id).await.unwrap(); + assert_eq!(state.etag.as_deref(), Some("\"v1\"")); + assert_eq!(state.consecutive_failures, 0); + let next = state.next_check_at_ms.unwrap(); + assert!((t0 + runner::REFRESH_INTERVAL_MS..t0 + 70 * 60 * 1000).contains(&next)); + } + let episodes = store + .episodes(a, &EpisodeQuery::new(podcast_id.clone())) + .await + .unwrap(); + assert_eq!(episodes.len(), 1); + let episode_id = episodes[0].episode.episode_id.clone(); + assert_eq!(episodes[0].episode.audio_url, server.url("/audio/one.mp3")); + + // The proxy and the source endpoint only find episodes of the account. + assert_eq!( + store.audio_source(a, &episode_id).await.unwrap(), + Some((server.url("/audio/one.mp3"), Some("audio/mpeg".to_string()))) + ); + assert_eq!(store.audio_source(c, &episode_id).await.unwrap(), None); + + // Nothing is due before the interval; afterwards the request is + // conditional and the catalog stays as it is. + assert!(due_for(&store, &users, t0 + 1000).await.is_empty()); + let t1 = t0 + 2 * runner::REFRESH_INTERVAL_MS; + let summary = run_due(&store, &fetcher, &users, t1).await; + assert_eq!(summary.not_modified, 2, "{summary}"); + assert_eq!(server.requests("/show.xml")[1]["if-none-match"], "\"v1\""); + + // Failures back off exponentially and keep the catalog. + server.set("/show.xml", Route::new("boom", "text/plain").status(500)); + let t2 = t1 + 2 * runner::REFRESH_INTERVAL_MS; + let summary = run_due(&store, &fetcher, &users, t2).await; + assert_eq!(summary.failed, 2, "{summary}"); + assert_eq!(summary.failures.len(), 2); + let state = store.fetch_state(a, &podcast_id).await.unwrap(); + assert_eq!(state.consecutive_failures, 1); + assert_eq!(state.next_check_at_ms, Some(t2 + BACKOFF_BASE_MS)); + assert!(state.last_error.as_deref().unwrap().contains("500")); + assert_eq!(state.etag.as_deref(), Some("\"v1\""), "validators are kept"); + assert!( + due_for(&store, &users, t2 + BACKOFF_BASE_MS - 1) + .await + .is_empty() + ); + let t3 = t2 + BACKOFF_BASE_MS; + let summary = run_due(&store, &fetcher, &users, t3).await; + assert_eq!(summary.failed, 2, "{summary}"); + let state = store.fetch_state(a, &podcast_id).await.unwrap(); + assert_eq!(state.consecutive_failures, 2); + assert_eq!(state.next_check_at_ms, Some(t3 + 2 * BACKOFF_BASE_MS)); + assert_eq!( + store + .episodes(a, &EpisodeQuery::new(podcast_id.clone())) + .await + .unwrap() + .len(), + 1 + ); + + // Recovery resets the backoff. Accounts with different validators are + // fetched unconditionally. + server.set( + "/show.xml", + Route::new( + rss(&format!("{}{}", refresh_item("two"), refresh_item("one"))), + "application/rss+xml", + ) + .etag("\"v2\""), + ); + store + .record_fetch_success(b, &podcast_id, t3, Some("\"other\""), None, Some(t3)) + .await + .unwrap(); + let t4 = t3 + 2 * BACKOFF_BASE_MS; + let summary = run_due(&store, &fetcher, &users, t4).await; + assert_eq!((summary.updated, summary.new_episodes), (2, 2), "{summary}"); + let last = server.requests("/show.xml").pop().unwrap(); + assert!(!last.contains_key("if-none-match"), "{last:?}"); + let state = store.fetch_state(a, &podcast_id).await.unwrap(); + assert_eq!((state.consecutive_failures, state.last_error), (0, None)); + assert_eq!(state.etag.as_deref(), Some("\"v2\"")); + + // Manual refresh: not more often than every 30 s, only when subscribed. + let manual = runner::refresh_now(&store, &fetcher, a, &podcast_id, t4 + 10_000).await; + assert_eq!( + manual.unwrap_err().downcast_ref::(), + Some(&ManualRefreshError::TooSoon { + retry_after_ms: 20_000 + }) + ); + let manual = runner::refresh_now(&store, &fetcher, a, &podcast_id, t4 + 31_000) + .await + .unwrap(); + assert_eq!(manual, TargetResult::NotModified); + let unknown = runner::refresh_now(&store, &fetcher, c, &podcast_id, t4 + 31_000).await; + assert_eq!( + unknown.unwrap_err().downcast_ref::(), + Some(&ManualRefreshError::UnknownPodcast) + ); + store.unsubscribe(b, &podcast_id).await.unwrap(); + let unsubscribed = runner::refresh_now(&store, &fetcher, b, &podcast_id, t4 + 31_000).await; + assert_eq!( + unsubscribed + .unwrap_err() + .downcast_ref::(), + Some(&ManualRefreshError::NotSubscribed) + ); + let far = t4 + 10 * runner::REFRESH_INTERVAL_MS; + let due = due_for(&store, &users, far).await; + assert_eq!( + due.values().flatten().count(), + 1, + "unsubscribing stops monitoring" + ); + + // A permanent move becomes an alias (one sync operation) and the new + // fetch URL; later refreshes go to the new URL without new operations. + server.set("/show.xml", Route::redirect(301, "/moved.xml")); + server.set( + "/moved.xml", + Route::new(rss(&refresh_item("one")), "application/rss+xml"), + ); + let moved = FeedUrl::parse(&server.url("/moved.xml")).unwrap(); + let result = runner::refresh_now(&store, &fetcher, a, &podcast_id, far) + .await + .unwrap(); + let TargetResult::Updated(outcome) = result else { + panic!("{result:?}"); + }; + assert_eq!(outcome.feed_url.as_ref(), Some(&moved)); + assert_eq!(outcome.ops.len(), 1); + assert_eq!( + store + .podcast(a, &podcast_id) + .await + .unwrap() + .unwrap() + .meta + .feed_url, + moved + ); + let due = due_for(&store, &users, far + 10 * runner::REFRESH_INTERVAL_MS).await; + assert_eq!(due.keys().collect::>(), vec![&moved]); + let summary = run_due( + &store, + &fetcher, + &users, + far + 10 * runner::REFRESH_INTERVAL_MS, + ) + .await; + assert_eq!(summary.updated, 1, "{summary}"); + assert_eq!(server.requests("/moved.xml").len(), 2); + assert_eq!(server.requests("/show.xml").len(), 7); + let TargetResult::Updated(again) = runner::refresh_now( + &store, + &fetcher, + a, + &podcast_id, + far + 20 * runner::REFRESH_INTERVAL_MS, + ) + .await + .unwrap() else { + panic!("expected a document"); + }; + assert!(again.ops.is_empty()); + // The episode keeps its id; its audio now resolves against the new URL + // (same host here). + let after_move = store.episode(a, &episode_id).await.unwrap().unwrap(); + assert_eq!(after_move.episode.audio_url, server.url("/audio/one.mp3")); + + cleanup(&pool, [a, b, c]).await; +} + +// --------------------------------------------------------------------------- +// HTTP API of the podcast UI and the web player (`api`, behind `http`) +// --------------------------------------------------------------------------- + +async fn logged_kinds(pool: &sqlx::PgPool, user_id: i64) -> Vec { + sqlx::query_scalar("SELECT kind FROM furumusic__fed_sync_ops WHERE user_id = $1 ORDER BY seq") + .bind(user_id) + .fetch_all(pool) + .await + .unwrap() +} + +#[tokio::test] +#[ignore = "requires a disposable database in FURU_DEVICE_SYNC_TEST_DATABASE_URL"] +async fn postgres_podcast_api_subscribes_plays_and_keeps_progress() { + use super::api::{self, EpisodeAction, EpisodeListParams}; + use super::test_server::{Route, TestServer}; + use serde_json::json; + + let pool = test_pool().await; + let store = PodcastStore::new(pool.clone()); + let (user, other) = (random_user(), random_user()); + cleanup(&pool, [user, other]).await; + let server = TestServer::start(); + let items = r#" + Pilots1e11 + 1600 + Mon, 01 Jan 2024 10:00:00 GMTFirst interview + + Bonusbonus120 + Tue, 02 Jan 2024 10:00:00 GMT + "#; + server.set("/show.xml", Route::new(rss(items), "application/rss+xml")); + server.set("/page.html", Route::new("", "text/html")); + server.set("/gone.xml", Route::new("gone", "text/plain").status(404)); + let fetcher = local_fetcher(); + let now = super::now_ms(); + + // "Add RSS": invalid input and unusable feeds subscribe nothing. + for (body, status) in [ + (json!({}), 400), + (json!({"url": "ftp://example.org/feed"}), 400), + (json!({"url": server.url("/page.html")}), 422), + (json!({"url": server.url("/gone.xml")}), 422), + ] { + let error = api::subscribe_url(&store, &fetcher, user, body.clone(), now) + .await + .unwrap_err(); + assert_eq!(error.status, status, "{body}: {}", error.message); + } + // The production policy never reaches the local server. + let refused = api::subscribe_url( + &store, + &FeedFetcher::new().unwrap(), + user, + json!({"url": server.url("/show.xml")}), + now, + ) + .await + .unwrap_err(); + assert_eq!(refused.status, 422, "{}", refused.message); + assert_eq!( + refused.message, + "the feed could not be loaded: only public http and https addresses on ports 80, 443, \ + 8080 and 8443 are allowed" + ); + assert!(logged_kinds(&pool, user).await.is_empty()); + assert_eq!(server.requests("/show.xml").len(), 0); + + let added = api::subscribe_url( + &store, + &fetcher, + user, + json!({"url": format!(" {} ", server.url("/show.xml"))}), + now, + ) + .await + .unwrap(); + assert!(!added.already_subscribed); + assert_eq!(added.new_episodes, 2); + let podcast = added.podcast; + assert!(podcast.subscribed); + assert!(!podcast.shared, "a new subscription is private"); + assert_eq!((podcast.episode_count, podcast.unplayed_count), (2, 2)); + assert_eq!(podcast.title, "Refresh Show"); + assert!(podcast.fetch.last_success_at_ms.is_some()); + assert!(podcast.fetch.next_check_at_ms.unwrap() > now); + let podcast_id = PodcastId::parse(&podcast.podcast_id).unwrap(); + assert_eq!( + logged_kinds(&pool, user).await, + ["podcast_subscription_set"] + ); + // Adding it again writes nothing. + let again = api::subscribe_url( + &store, + &fetcher, + user, + json!({"url": server.url("/show.xml")}), + now, + ) + .await + .unwrap(); + assert!(again.already_subscribed); + assert_eq!(again.podcast.podcast_id, podcast.podcast_id); + assert_eq!(logged_kinds(&pool, user).await.len(), 1); + + // Lists and details are per account. + let list = api::list_podcasts(&store, user, false).await.unwrap(); + assert_eq!(list.podcasts.len(), 1); + assert!( + api::list_podcasts(&store, other, true) + .await + .unwrap() + .podcasts + .is_empty() + ); + assert_eq!( + api::podcast_detail(&store, other, &podcast_id, Discovery::Off) + .await + .unwrap_err() + .status, + 404 + ); + let detail = api::podcast_detail(&store, user, &podcast_id, Discovery::Off) + .await + .unwrap(); + let seasons: Vec> = detail.seasons.iter().map(|season| season.season).collect(); + assert_eq!(seasons, [Some(1), None]); + assert!(detail.history.is_empty()); + + // Episodes: newest first, filters, search, paging, queue items. + let page = api::episodes(&store, user, &podcast_id, &EpisodeListParams::default()) + .await + .unwrap(); + let titles: Vec<&str> = page.episodes.iter().map(|e| e.title.as_str()).collect(); + assert_eq!(titles, ["Bonus", "Pilot"]); + assert!(!page.has_more); + let pilot = &page.episodes[1]; + assert_eq!((pilot.season, pilot.episode_number), (Some(1), Some(1))); + assert_eq!(pilot.status, "not_started"); + assert_eq!(pilot.track["source"]["kind"], "podcast_episode"); + assert_eq!( + pilot.track["source"]["episode_id"], + pilot.episode_id.as_str() + ); + assert_eq!(pilot.track["unavailable"], false); + assert_eq!(pilot.track["title"], "Pilot"); + let first = api::episodes( + &store, + user, + &podcast_id, + &EpisodeListParams { + limit: Some(1), + ..Default::default() + }, + ) + .await + .unwrap(); + assert_eq!((first.episodes.len(), first.has_more), (1, true)); + let season = api::episodes( + &store, + user, + &podcast_id, + &EpisodeListParams { + season: Some("none".into()), + ..Default::default() + }, + ) + .await + .unwrap(); + assert_eq!(season.episodes.len(), 1); + assert_eq!(season.episodes[0].title, "Bonus"); + let found = api::episodes( + &store, + user, + &podcast_id, + &EpisodeListParams { + q: Some("INTERVIEW".into()), + ..Default::default() + }, + ) + .await + .unwrap(); + assert_eq!(found.episodes.len(), 1, "description search, any case"); + let episode_id = EpisodeId::parse(&pilot.episode_id).unwrap(); + + // Playback: the source and the resume position, without writing. + let prepared = api::playback(&store, user, &episode_id, json!({}), false) + .await + .unwrap(); + assert_eq!(prepared.direct_url, server.url("/audio/s1e1.mp3")); + assert_eq!(prepared.proxy_url, None); + assert_eq!((prepared.position_ms, prepared.generation), (0, None)); + assert_eq!(prepared.duration_ms, Some(600_000)); + let proxied = api::playback(&store, user, &episode_id, json!({}), true) + .await + .unwrap(); + assert_eq!( + proxied.proxy_url.as_deref(), + Some(format!("/api/podcasts/episodes/{episode_id}/audio").as_str()) + ); + assert_eq!(logged_kinds(&pool, user).await.len(), 1); + assert_eq!( + api::playback(&store, user, &episode_id, json!({"start": true}), false) + .await + .unwrap_err() + .status, + 400, + "starting needs a session id" + ); + + // Play starts a session (explicit generation 1). + let started = api::playback( + &store, + user, + &episode_id, + json!({"start": true, "session_id": "web-s1"}), + false, + ) + .await + .unwrap(); + assert_eq!(started.session_id.as_deref(), Some("web-s1")); + assert_eq!((started.generation, started.position_ms), (Some(1), 0)); + let report = |body: serde_json::Value| { + let store = store.clone(); + let episode_id = episode_id.clone(); + async move { api::progress(&store, user, &episode_id, body, super::now_ms()).await } + }; + let t0 = super::now_ms() - 120_000; + let saved = report(json!({ + "session_id": "web-s1", "generation": 1, "event": "checkpoint", "position_ms": 60_000, + "duration_ms": 600_000, + })) + .await + .unwrap(); + assert_eq!( + (saved.outcome, saved.generation, saved.status), + ("written", 1, "in_progress") + ); + assert!( + !saved.session_recorded, + "checkpoints do not record the session" + ); + // Pause records the session (at least 30 s listened). + let paused = report(json!({ + "session_id": "web-s1", "generation": 1, "event": "pause", "position_ms": 90_000, + "listened_ms": 90_000, "started_at_ms": t0, "start_position_ms": 0, + })) + .await + .unwrap(); + assert!(paused.session_recorded); + // Seeking back is an explicit action; a late checkpoint at the old + // generation is superseded. + let sought = report(json!({ + "session_id": "web-s1", "generation": 1, "event": "seek", "position_ms": 30_000, + })) + .await + .unwrap(); + assert_eq!( + (sought.outcome, sought.generation, sought.position_ms), + ("written", 2, 30_000) + ); + let late = report(json!({ + "session_id": "web-s1", "generation": 1, "event": "checkpoint", "position_ms": 95_000, + })) + .await + .unwrap(); + assert_eq!( + (late.outcome, late.generation, late.position_ms), + ("superseded", 2, 30_000) + ); + // Ordinary Play resumes from the register. + let resumed = api::playback(&store, user, &episode_id, json!({}), false) + .await + .unwrap(); + assert_eq!( + (resumed.position_ms, resumed.status), + (30_000, "in_progress") + ); + // Reaching the completion tail completes once, with the session. + let near_end = report(json!({ + "session_id": "web-s1", "generation": 2, "event": "checkpoint", "position_ms": 590_000, + })) + .await + .unwrap(); + assert_eq!( + (near_end.outcome, near_end.status, near_end.completed), + ("written", "completed", true) + ); + assert_eq!(near_end.generation, 3); + let ended = report(json!({ + "session_id": "web-s1", "generation": 3, "event": "ended", "position_ms": 600_000, + "listened_ms": 400_000, "started_at_ms": t0, "start_position_ms": 0, + })) + .await + .unwrap(); + assert_eq!(ended.outcome, "already_completed"); + assert!( + ended.session_recorded, + "the session grew and is now completed" + ); + let item = store.episode(user, &episode_id).await.unwrap().unwrap(); + assert_eq!(item.completed_count, 1); + // A completed episode plays from the start. + let replay = api::playback(&store, user, &episode_id, json!({}), false) + .await + .unwrap(); + assert_eq!((replay.position_ms, replay.status), (0, "completed")); + let history = api::podcast_detail(&store, user, &podcast_id, Discovery::Off) + .await + .unwrap() + .history; + assert_eq!(history.len(), 1); + assert!(history[0].completed); + assert_eq!(history[0].episode_title.as_deref(), Some("Pilot")); + + // Explicit list actions. + let restarted = api::episode_action(&store, user, &episode_id, EpisodeAction::Restart) + .await + .unwrap(); + assert_eq!( + (restarted.status, restarted.position_ms), + ("in_progress", 0) + ); + let played = api::episode_action(&store, user, &episode_id, EpisodeAction::MarkPlayed) + .await + .unwrap(); + assert_eq!(played.status, "completed"); + assert_eq!( + played.completed_count, 1, + "marking played invents no session" + ); + let unplayed = api::episode_action(&store, user, &episode_id, EpisodeAction::MarkUnplayed) + .await + .unwrap(); + assert_eq!((unplayed.status, unplayed.position_ms), ("not_started", 0)); + assert_eq!( + api::episode_action(&store, other, &episode_id, EpisodeAction::MarkPlayed) + .await + .unwrap_err() + .status, + 404 + ); + + // A handed-off queue item the catalog does not know yet: the reference + // is recorded and the session starts above the carried generation. + let mut handed = test_episode(); + handed.progress_generation = Some(9); + let handed_id = handed.episode_id.clone(); + assert_eq!( + api::playback(&store, user, &handed_id, json!({}), false) + .await + .unwrap_err() + .status, + 404 + ); + let mut wrong = handed.clone(); + wrong.episode_id = episode_id.clone(); + assert_eq!( + api::playback(&store, user, &handed_id, json!({"episode": wrong}), false) + .await + .unwrap_err() + .status, + 400 + ); + let taken_over = api::playback( + &store, + user, + &handed_id, + json!({ + "episode": handed, "start": true, "session_id": "web-s2", + "position_ms": 1_200_000, "carried_generation": 9, + }), + false, + ) + .await + .unwrap(); + assert_eq!( + (taken_over.generation, taken_over.position_ms), + (Some(10), 1_200_000) + ); + assert_eq!(taken_over.direct_url, "https://cdn.example/ep42.mp3"); + let catalog = store.episode(user, &handed_id).await.unwrap().unwrap(); + assert!(!catalog.in_feed); + // A handoff away writes a final checkpoint at the session generation. + let away = api::progress( + &store, + user, + &handed_id, + json!({ + "session_id": "web-s2", "generation": 10, "event": "handoff", + "position_ms": 1_260_000, "listened_ms": 10_000, + "started_at_ms": super::now_ms() - 60_000, "start_position_ms": 1_200_000, + }), + super::now_ms(), + ) + .await + .unwrap(); + assert_eq!( + (away.outcome, away.generation, away.position_ms), + ("written", 10, 1_260_000) + ); + assert!( + !away.session_recorded, + "10 s is below the history threshold" + ); + + // Unsubscribing keeps progress and history; subscribing again continues. + let left = api::unsubscribe(&store, user, &podcast_id).await.unwrap(); + assert!(!left.subscribed); + assert!( + api::list_podcasts(&store, user, false) + .await + .unwrap() + .podcasts + .is_empty() + ); + let all = api::list_podcasts(&store, user, true).await.unwrap(); + assert!( + all.podcasts + .iter() + .any(|p| p.podcast_id == podcast.podcast_id) + ); + let kept = store.episode(user, &episode_id).await.unwrap().unwrap(); + assert_eq!(kept.completed_count, 1); + let back = api::resubscribe(&store, user, &podcast_id).await.unwrap(); + assert!(back.subscribed); + assert_eq!( + api::unsubscribe(&store, other, &podcast_id) + .await + .unwrap_err() + .status, + 404 + ); + + // Every write went through the sync log, with its own operation kinds. + let kinds = logged_kinds(&pool, user).await; + let count = |kind: &str| kinds.iter().filter(|k| k.as_str() == kind).count(); + assert_eq!(count("podcast_subscription_set"), 3); + assert_eq!( + count("podcast_share_set"), + 1, + "unsubscribing revokes sharing" + ); + assert_eq!(count("podcast_listen_recorded"), 2); + // start, checkpoint, pause, seek, completion, restart, played, + // unplayed, handed-off start, handoff checkpoint + assert_eq!(count("podcast_progress_set"), 10); + + cleanup(&pool, [user, other]).await; +} + +/// Ordinary Play continues from the progress register, not from a position +/// the browser saved earlier: after a reload the web player prepares the +/// episode without a position and gets what other devices wrote meanwhile. +#[tokio::test] +#[ignore = "requires a disposable database in FURU_DEVICE_SYNC_TEST_DATABASE_URL"] +async fn postgres_podcast_playback_resumes_from_the_register() { + use serde_json::json; + + let pool = test_pool().await; + let store = PodcastStore::new(pool.clone()); + let user = random_user(); + let episode = PodcastEpisodeRef { + progress_generation: None, + ..test_episode() + }; + let episode_id = episode.episode_id.clone(); + let started = api::playback( + &store, + user, + &episode_id, + json!({"episode": episode, "start": true, "session_id": "web-s1", "position_ms": 600_000}), + false, + ) + .await + .unwrap(); + assert_eq!( + (started.generation, started.position_ms), + (Some(1), 600_000) + ); + assert_eq!(started.register_generation, Some(1)); + assert_eq!(started.register_session_id.as_deref(), Some("web-s1")); + let paused = api::progress( + &store, + user, + &episode_id, + json!({"session_id": "web-s1", "generation": 1, "event": "pause", "position_ms": 620_000}), + super::now_ms(), + ) + .await + .unwrap(); + assert_eq!(paused.outcome, "written"); + let prepare = || api::playback(&store, user, &episode_id, json!({}), false); + let mine = prepare().await.unwrap(); + assert_eq!(mine.position_ms, 620_000); + assert_eq!(mine.register_session_id.as_deref(), Some("web-s1")); + + // The phone listened further offline; its progress arrives later. + store + .apply_remote_op(user, &progress_op("phone", 1, 50, &episode, 2, 2_400_000)) + .await + .unwrap(); + let moved = prepare().await.unwrap(); + assert_eq!(moved.position_ms, 2_400_000, "the register, not 620 000"); + assert_eq!(moved.register_generation, Some(2)); + assert_eq!(moved.register_session_id.as_deref(), Some("phone-g2")); + assert_eq!(moved.generation, None, "preparing writes nothing"); + let resumed = api::playback( + &store, + user, + &episode_id, + json!({"start": true, "session_id": "web-s2"}), + false, + ) + .await + .unwrap(); + assert_eq!( + (resumed.generation, resumed.position_ms), + (Some(3), 2_400_000) + ); + assert_eq!(resumed.register_session_id.as_deref(), Some("web-s2")); + + // Marked played elsewhere: Play starts over. + let reference = store.episode_ref(user, &episode_id).await.unwrap().unwrap(); + store.mark_played(user, &reference).await.unwrap(); + let played = prepare().await.unwrap(); + assert_eq!((played.status, played.position_ms), ("completed", 0)); + + cleanup(&pool, [user]).await; +} + +/// The browser can add episodes outside the account's feeds (handed-off +/// queue items) only up to a limit; the account's own devices are not +/// limited, and known episodes always play. +#[tokio::test] +#[ignore = "requires a disposable database in FURU_DEVICE_SYNC_TEST_DATABASE_URL"] +async fn postgres_podcast_episode_references_from_the_browser_are_bounded() { + use serde_json::json; + + let pool = test_pool().await; + let store = PodcastStore::new(pool.clone()); + let user = random_user(); + let first = PodcastEpisodeRef { + progress_generation: None, + ..test_episode() + }; + let reference = |guid: &str| PodcastEpisodeRef { + episode_id: EpisodeId::from_guid(&first.podcast_id, guid).unwrap(), + episode_title: guid.into(), + ..first.clone() + }; + let play = |episode: PodcastEpisodeRef| { + let store = store.clone(); + async move { + api::playback( + &store, + user, + &episode.episode_id.clone(), + json!({ "episode": episode }), + false, + ) + .await + } + }; + play(first.clone()).await.unwrap(); + // Fill up to one below the limit with copies of that reference row. + sqlx::query( + "INSERT INTO furumusic__podcast_episode + SELECT (jsonb_populate_record(e, jsonb_build_object('episode_id', 'cap-' || n))).* + FROM furumusic__podcast_episode e, generate_series(1, $3) AS n + WHERE e.user_id = $1 AND e.episode_id = $2", + ) + .bind(user) + .bind(first.episode_id.as_str()) + .bind(i32::try_from(MAX_REFERENCE_EPISODES_PER_ACCOUNT - 2).unwrap()) + .execute(&pool) + .await + .unwrap(); + play(reference("last-allowed")).await.unwrap(); + let refused = play(reference("one-too-many")).await.unwrap_err(); + assert_eq!(refused.status, 422, "{}", refused.message); + let unknown = reference("one-too-many").episode_id; + assert!(store.episode_ref(user, &unknown).await.unwrap().is_none()); + // Episodes already known still play. + play(first.clone()).await.unwrap(); + play(reference("last-allowed")).await.unwrap(); + // References in the account's own devices' operations are applied. + let synced = reference("from-the-phone"); + store + .apply_remote_op(user, &progress_op("phone", 1, 50, &synced, 1, 60_000)) + .await + .unwrap(); + assert!( + store + .episode_ref(user, &synced.episode_id) + .await + .unwrap() + .is_some() + ); + + cleanup(&pool, [user]).await; +} diff --git a/src/podcasts/proxy.rs b/src/podcasts/proxy.rs new file mode 100644 index 0000000..b9c24f1 --- /dev/null +++ b/src/podcasts/proxy.rs @@ -0,0 +1,537 @@ +//! Optional audio proxy: streams a stored episode enclosure from the +//! publisher for browsers that cannot load it directly (CORS, mixed +//! content). Audio is passed through, never stored or imported. +//! +//! The caller resolves the URL from the database for an episode of the +//! authenticated account; a client never supplies a URL. Requests go +//! through [`SafeHttp`] (SSRF guard, every redirect hop checked), carry +//! only `Range`, `Accept`, `Accept-Encoding: identity` and the user agent, +//! and responses are limited in size, time and content type. The content +//! type is decided by the publisher's response (and, for generic binary +//! types, the URL extension or the file's first bytes), never by the +//! enclosure type stored with the episode. + +use std::fmt; +use std::time::Duration; + +use bytes::Bytes; +use futures_util::StreamExt; +use futures_util::stream::BoxStream; +use reqwest::header::{ + ACCEPT, ACCEPT_ENCODING, ACCEPT_RANGES, CONTENT_LENGTH, CONTENT_RANGE, CONTENT_TYPE, HeaderMap, + HeaderValue, RANGE, +}; +use reqwest::{StatusCode, Url}; + +use super::net::{GuardError, REFUSED_MESSAGE, SafeHttp, SafeHttpConfig}; + +/// Largest response body passed through (a long episode at a high bitrate +/// stays far below). +pub(crate) const MAX_PROXY_BYTES: u64 = 1024 * 1024 * 1024; +/// Upstream must answer with headers within this time. +pub(crate) const PROXY_HEADER_TIMEOUT: Duration = Duration::from_secs(20); +/// Longest a single proxied response may stream. +pub(crate) const PROXY_MAX_DURATION: Duration = Duration::from_secs(4 * 60 * 60); +/// Redirect hops followed (enclosures often go through tracking redirects). +pub(crate) const MAX_PROXY_REDIRECTS: usize = 5; + +/// HTTP client settings of the proxy (no whole-request timeout: bodies are +/// streamed; the idle read timeout and [`PROXY_MAX_DURATION`] bound them). +pub(crate) const PROXY_HTTP: SafeHttpConfig = SafeHttpConfig { + connect_timeout: Duration::from_secs(10), + read_timeout: Duration::from_secs(30), + total_timeout: None, + max_redirects: MAX_PROXY_REDIRECTS, +}; + +/// Why a proxied request failed. +#[derive(Debug)] +pub(crate) enum ProxyError { + /// The stored URL is not a valid URL. + InvalidUrl(String), + /// Refused or failed in transport (see [`GuardError`]). + Upstream(GuardError), + /// The publisher answered with an unusable status. + Status(u16), + /// The publisher's content type is not audio. + ContentType(String), + /// Declared body larger than [`MAX_PROXY_BYTES`]. + TooLarge(u64), + /// No response headers within [`PROXY_HEADER_TIMEOUT`]. + Timeout, + /// The body stream stopped (limit reached or upstream failure). + Body(String), +} + +impl ProxyError { + /// Status the proxy answers the browser with. + pub(crate) fn http_status(&self) -> u16 { + match self { + Self::Timeout => 504, + Self::Upstream(error) if error.is_timeout() => 504, + _ => 502, + } + } + + /// What the browser is told; the details (`Display`) stay in the + /// server log. + pub(crate) fn public_message(&self) -> String { + match self { + Self::InvalidUrl(_) => "the episode's audio URL is invalid".to_string(), + Self::Upstream(error) if error.is_refusal() => REFUSED_MESSAGE.to_string(), + Self::Upstream(error) if error.is_timeout() => { + "the publisher did not answer in time".to_string() + } + Self::Upstream(_) => "could not connect to the publisher".to_string(), + Self::Status(status) => format!("the publisher answered HTTP {status}"), + Self::ContentType(_) => "the publisher's file is not audio".to_string(), + Self::TooLarge(_) => "the publisher's file is too large".to_string(), + Self::Timeout => "the publisher did not answer in time".to_string(), + Self::Body(_) => "the publisher stream failed".to_string(), + } + } +} + +impl fmt::Display for ProxyError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::InvalidUrl(reason) => write!(f, "stored audio URL is invalid: {reason}"), + Self::Upstream(error) => write!(f, "publisher request failed: {error}"), + Self::Status(status) => write!(f, "publisher answered HTTP {status}"), + Self::ContentType(value) => write!(f, "publisher content type {value:?} is not audio"), + Self::TooLarge(len) => write!( + f, + "publisher response is too large ({len} bytes, limit {MAX_PROXY_BYTES})" + ), + Self::Timeout => write!( + f, + "publisher did not answer within {} s", + PROXY_HEADER_TIMEOUT.as_secs() + ), + Self::Body(reason) => write!(f, "publisher stream failed: {reason}"), + } + } +} + +impl std::error::Error for ProxyError {} + +impl From for ProxyError { + fn from(error: GuardError) -> Self { + Self::Upstream(error) + } +} + +/// A proxied response, ready to be sent to the browser. +pub(crate) struct ProxiedAudio { + /// 200, 206 or 416. + pub(crate) status: u16, + /// Allow-listed audio type (absent on 416). + pub(crate) content_type: Option, + pub(crate) content_length: Option, + /// `bytes a-b/n` (206) or `bytes */n` (416). + pub(crate) content_range: Option, + /// Whether the publisher advertised byte ranges. + pub(crate) accept_ranges: bool, + /// The body, cut off at [`MAX_PROXY_BYTES`] and [`PROXY_MAX_DURATION`]. + pub(crate) body: BoxStream<'static, Result>, +} + +impl fmt::Debug for ProxiedAudio { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("ProxiedAudio") + .field("status", &self.status) + .field("content_type", &self.content_type) + .field("content_length", &self.content_length) + .field("content_range", &self.content_range) + .field("accept_ranges", &self.accept_ranges) + .finish_non_exhaustive() + } +} + +/// Keeps a browser `Range` header only when it is one well-formed byte +/// range (`bytes=a-`, `bytes=a-b`, `bytes=-n`); anything else is dropped +/// and the full resource is requested. +pub(crate) fn sanitize_range(value: &str) -> Option { + let spec = value.trim().strip_prefix("bytes=")?.trim(); + if spec.len() > 64 { + return None; + } + let (start, end) = spec.split_once('-')?; + let digits = |s: &str| !s.is_empty() && s.bytes().all(|b| b.is_ascii_digit()); + let ok = match (start.is_empty(), end.is_empty()) { + (false, true) => digits(start) && start.parse::().is_ok(), + (false, false) => { + digits(start) + && digits(end) + && matches!((start.parse::(), end.parse::()), (Ok(a), Ok(b)) if a <= b) + } + (true, false) => digits(end) && end.parse::().is_ok_and(|n| n > 0), + (true, true) => false, + }; + ok.then(|| format!("bytes={start}-{end}")) +} + +/// `Content-Range` of a 206/416 passed through only when well-formed. +fn sanitize_content_range(value: &str) -> Option { + let rest = value.trim().strip_prefix("bytes ")?; + let (range, total) = rest.split_once('/')?; + let total_ok = total == "*" || total.parse::().is_ok(); + let range_ok = range == "*" + || range.split_once('-').is_some_and( + |(a, b)| matches!((a.parse::(), b.parse::()), (Ok(a), Ok(b)) if a <= b), + ); + (total_ok && range_ok && !(range == "*" && total == "*")).then(|| value.trim().to_string()) +} + +fn is_audio_type(mime: &str) -> bool { + let mime = mime.trim().to_ascii_lowercase(); + let essence = mime.split(';').next().unwrap_or_default().trim(); + (essence.starts_with("audio/") && essence.len() > "audio/".len()) + || matches!(essence, "application/ogg" | "application/x-ogg") +} + +/// Audio type implied by the URL path's extension. +fn audio_type_from_url(url: &Url) -> Option<&'static str> { + let ext = url.path().rsplit('/').next()?.rsplit_once('.')?.1; + Some(match ext.to_ascii_lowercase().as_str() { + "mp3" => "audio/mpeg", + "m4a" | "m4b" | "mp4" => "audio/mp4", + "aac" => "audio/aac", + "ogg" | "oga" => "audio/ogg", + "opus" => "audio/ogg", + "flac" => "audio/flac", + "wav" => "audio/wav", + _ => return None, + }) +} + +/// Bytes read from the start of a file to recognize its format. +const SNIFF_BYTES: usize = 16; + +/// What the declared type and the URL say about a response. +#[derive(Debug, PartialEq, Eq)] +enum TypeCheck { + /// Audio: the publisher's type, or a generic type with an audio file + /// extension. + Audio(String), + /// A generic binary type (or none) without an audio extension: only the + /// file's first bytes can tell. + Content, +} + +/// The content type sent to the browser: the publisher's when it is audio; +/// for a generic binary type, the type implied by the URL extension +/// ([`TypeCheck::Content`] without one); anything else is refused. +/// +/// The enclosure type from the feed (or from an episode reference a client +/// sent) is never trusted here: it is chosen by whoever wrote the feed, so +/// it would turn the proxy into a relay for any file served as +/// `application/octet-stream`. +fn audio_content_type(upstream: Option<&str>, url: &Url) -> Result { + let upstream = upstream.map(str::trim).filter(|value| !value.is_empty()); + if let Some(value) = upstream + && is_audio_type(value) + { + return Ok(TypeCheck::Audio(value.to_string())); + } + let generic = upstream.is_none_or(|value| { + let essence = value.split(';').next().unwrap_or_default().trim(); + essence.eq_ignore_ascii_case("application/octet-stream") + || essence.eq_ignore_ascii_case("binary/octet-stream") + }); + if !generic { + return Err(ProxyError::ContentType( + upstream.unwrap_or("(none)").to_string(), + )); + } + Ok(match audio_type_from_url(url) { + Some(guess) => TypeCheck::Audio(guess.to_string()), + None => TypeCheck::Content, + }) +} + +/// The audio format a file starts with (magic numbers of the formats +/// podcasts use), or `None`. +pub(crate) fn sniff_audio(head: &[u8]) -> Option<&'static str> { + let at = |range: std::ops::Range| head.get(range); + if head.starts_with(b"ID3") { + return Some("audio/mpeg"); + } + if head.starts_with(b"OggS") { + return Some("audio/ogg"); + } + if head.starts_with(b"fLaC") { + return Some("audio/flac"); + } + if at(0..4) == Some(b"RIFF") && at(8..12) == Some(b"WAVE") { + return Some("audio/wav"); + } + // MPEG-4 audio brands only: video files are not passed through. + if at(4..8) == Some(b"ftyp") + && matches!( + at(8..12), + Some(b"M4A " | b"M4B " | b"M4P " | b"F4A " | b"F4B ") + ) + { + return Some("audio/mp4"); + } + // A bare frame header has few fixed bits, so the whole header must be + // valid. Layer I (`FF FE`, `FF FF`: the UTF-16 byte order mark of text + // files) is never used by podcasts and is not accepted. + if let [0xFF, second, third, ..] = *head { + // ADTS AAC: sync, layer 00, a defined sampling frequency. + if second & 0xF6 == 0xF0 && (third >> 2) & 0b1111 < 13 { + return Some("audio/aac"); + } + // MPEG audio frame: sync, a defined version, layer II or III, a + // bitrate index other than free (0) and bad (15), and a defined + // sampling rate. + let version = (second >> 3) & 0b11; + let layer = (second >> 1) & 0b11; + let bitrate = third >> 4; + let sample_rate = (third >> 2) & 0b11; + if second & 0xE0 == 0xE0 + && version != 0b01 + && matches!(layer, 0b01 | 0b10) + && (1..15).contains(&bitrate) + && sample_rate != 0b11 + { + return Some("audio/mpeg"); + } + } + None +} + +type Upstream = BoxStream<'static, reqwest::Result>; + +/// Reads at least `want` bytes (fewer at the end of the body) from +/// `upstream`; the chunks are sent to the browser afterwards. +async fn read_head(upstream: &mut Upstream, want: usize) -> Result, ProxyError> { + let mut chunks = Vec::new(); + let mut have = 0; + while have < want { + match tokio::time::timeout(PROXY_HEADER_TIMEOUT, upstream.next()).await { + Err(_) => return Err(ProxyError::Timeout), + Ok(None) => break, + Ok(Some(Ok(chunk))) => { + have += chunk.len(); + chunks.push(chunk); + } + Ok(Some(Err(error))) => { + return Err(ProxyError::Body(GuardError::from(error).to_string())); + } + } + } + Ok(chunks) +} + +fn head_bytes(chunks: &[Bytes]) -> Vec { + let mut head = Vec::with_capacity(SNIFF_BYTES); + for chunk in chunks { + let take = (SNIFF_BYTES - head.len()).min(chunk.len()); + head.extend_from_slice(&chunk[..take]); + if head.len() == SNIFF_BYTES { + break; + } + } + head +} + +/// Streams episode enclosures from publishers. +#[derive(Debug, Clone)] +pub(crate) struct AudioProxy { + http: SafeHttp, + max_bytes: u64, + max_duration: Duration, +} + +impl AudioProxy { + /// The production proxy (public addresses only). + pub(crate) fn new() -> anyhow::Result { + Ok(Self::with_http(SafeHttp::public(PROXY_HTTP)?)) + } + + pub(crate) fn with_http(http: SafeHttp) -> Self { + Self { + http, + max_bytes: MAX_PROXY_BYTES, + max_duration: PROXY_MAX_DURATION, + } + } + + #[cfg(test)] + pub(crate) fn with_limits(mut self, max_bytes: u64, max_duration: Duration) -> Self { + self.max_bytes = max_bytes; + self.max_duration = max_duration; + self + } + + /// Requests `audio_url` (from the database) with the browser's `Range` + /// (sanitized). A response with a generic binary type and no audio file + /// extension is passed through only when the file starts like audio + /// (read from this response when it starts at byte 0 and is long + /// enough, otherwise from a separate 16-byte range request). + pub(crate) async fn fetch( + &self, + audio_url: &str, + range: Option<&str>, + ) -> Result { + let url = + Url::parse(audio_url).map_err(|error| ProxyError::InvalidUrl(error.to_string()))?; + let mut headers = HeaderMap::new(); + headers.insert(ACCEPT, HeaderValue::from_static("audio/*, */*;q=0.1")); + // Byte ranges must address the stored bytes, not a compressed form. + headers.insert(ACCEPT_ENCODING, HeaderValue::from_static("identity")); + let range = range.and_then(sanitize_range); + if let Some(range) = &range { + headers.insert( + RANGE, + HeaderValue::from_str(range).expect("sanitized range is a valid header"), + ); + } + let followed = tokio::time::timeout(PROXY_HEADER_TIMEOUT, self.http.get(url, &headers)) + .await + .map_err(|_| ProxyError::Timeout)??; + let response = followed.response; + let final_url = followed.final_url; + let status = response.status(); + let header = |name| { + response + .headers() + .get(name) + .and_then(|value: &HeaderValue| value.to_str().ok()) + .map(str::to_string) + }; + let content_range = header(CONTENT_RANGE) + .as_deref() + .and_then(sanitize_content_range); + if status == StatusCode::RANGE_NOT_SATISFIABLE { + return Ok(ProxiedAudio { + status: 416, + content_type: None, + content_length: Some(0), + content_range, + accept_ranges: true, + body: futures_util::stream::empty().boxed(), + }); + } + let partial = status == StatusCode::PARTIAL_CONTENT; + // 206 only answers a range we asked for, and must say which bytes. + if !(status == StatusCode::OK || (partial && range.is_some() && content_range.is_some())) { + return Err(ProxyError::Status(status.as_u16())); + } + let declared = header(CONTENT_TYPE); + let check = audio_content_type(declared.as_deref(), &final_url)?; + let content_length = header(CONTENT_LENGTH).and_then(|value| value.parse::().ok()); + if let Some(len) = content_length + && len > self.max_bytes + { + return Err(ProxyError::TooLarge(len)); + } + let accept_ranges = partial + || header(ACCEPT_RANGES) + .is_some_and(|value| value.trim().eq_ignore_ascii_case("bytes")); + let starts_at_zero = !partial + || content_range + .as_deref() + .is_some_and(|value| value.starts_with("bytes 0-")); + let mut upstream: Upstream = response.bytes_stream().boxed(); + let mut head = Vec::new(); + let content_type = match check { + TypeCheck::Audio(content_type) => content_type, + TypeCheck::Content => { + let mut start = Vec::new(); + if starts_at_zero { + head = read_head(&mut upstream, SNIFF_BYTES).await?; + start = head_bytes(&head); + } + // A range past the start, or one that ends before the + // sniffed bytes (Safari and AVFoundation probe with + // `bytes=0-1`): the file's head comes from its own request. + if partial && start.len() < SNIFF_BYTES { + start = self.file_head(&final_url).await?; + } + sniff_audio(&start) + .ok_or_else(|| { + ProxyError::ContentType(format!( + "{} (not audio content)", + declared.as_deref().unwrap_or("(none)") + )) + })? + .to_string() + } + }; + Ok(ProxiedAudio { + status: status.as_u16(), + content_type: Some(content_type), + content_length, + content_range: if partial { content_range } else { None }, + accept_ranges, + body: limited_body(head, upstream, self.max_bytes, self.max_duration), + }) + } + + /// The first [`SNIFF_BYTES`] of the file at `url` (a range request; a + /// server without ranges answers with the whole file, of which only the + /// start is read before the connection is dropped). + async fn file_head(&self, url: &Url) -> Result, ProxyError> { + let mut headers = HeaderMap::new(); + headers.insert(ACCEPT_ENCODING, HeaderValue::from_static("identity")); + headers.insert( + RANGE, + HeaderValue::from_str(&format!("bytes=0-{}", SNIFF_BYTES - 1)).expect("valid range"), + ); + let followed = + tokio::time::timeout(PROXY_HEADER_TIMEOUT, self.http.get(url.clone(), &headers)) + .await + .map_err(|_| ProxyError::Timeout)??; + let status = followed.response.status(); + if !(status == StatusCode::OK || status == StatusCode::PARTIAL_CONTENT) { + return Err(ProxyError::Status(status.as_u16())); + } + let mut upstream: Upstream = followed.response.bytes_stream().boxed(); + Ok(head_bytes(&read_head(&mut upstream, SNIFF_BYTES).await?)) + } +} + +/// The upstream body (`head`, already read, first), ending with an error +/// once `max_bytes` would be exceeded or `max_duration` has passed. +/// Dropping it (browser gone or seeking elsewhere) closes the upstream +/// connection. +fn limited_body( + head: Vec, + upstream: Upstream, + max_bytes: u64, + max_duration: Duration, +) -> BoxStream<'static, Result> { + let deadline = tokio::time::Instant::now() + max_duration; + let mut upstream = futures_util::stream::iter(head.into_iter().map(Ok)).chain(upstream); + async_stream::stream! { + let mut sent: u64 = 0; + loop { + let next = match tokio::time::timeout_at(deadline, upstream.next()).await { + Ok(next) => next, + Err(_) => { + yield Err(ProxyError::Body("time limit reached".to_string())); + break; + } + }; + match next { + None => break, + Some(Ok(chunk)) => { + sent += chunk.len() as u64; + if sent > max_bytes { + yield Err(ProxyError::Body(format!("size limit of {max_bytes} bytes reached"))); + break; + } + yield Ok(chunk); + } + Some(Err(error)) => { + yield Err(ProxyError::Body(GuardError::from(error).to_string())); + break; + } + } + } + } + .boxed() +} diff --git a/src/podcasts/proxy_tests.rs b/src/podcasts/proxy_tests.rs new file mode 100644 index 0000000..9da443e --- /dev/null +++ b/src/podcasts/proxy_tests.rs @@ -0,0 +1,449 @@ +//! Audio proxy: Range pass-through, servers without ranges, redirects, +//! content types and limits, against the local test server. + +use std::time::Duration; + +use futures_util::StreamExt; +use futures_util::stream::BoxStream; + +use super::net::{AddressClass, GuardError, REFUSED_MESSAGE}; +use super::proxy::{AudioProxy, ProxyError, sanitize_range, sniff_audio}; +use std::sync::Arc; + +use super::test_server::{Route, ScriptedLookup, TestServer, noise, test_http}; + +// --------------------------------------------------------------------------- +// Audio proxy +// --------------------------------------------------------------------------- + +#[test] +fn only_single_byte_ranges_are_forwarded() { + assert_eq!(sanitize_range("bytes=0-").as_deref(), Some("bytes=0-")); + assert_eq!( + sanitize_range("bytes=100-199").as_deref(), + Some("bytes=100-199") + ); + assert_eq!( + sanitize_range(" bytes=-500 ").as_deref(), + Some("bytes=-500") + ); + for invalid in [ + "bytes=1-2,5-6", + "items=0-10", + "bytes=5-1", + "bytes=-", + "bytes=-0", + "bytes=a-b", + "bytes=1-2\r\nCookie: x", + "bytes=99999999999999999999999-", + "", + ] { + assert_eq!(sanitize_range(invalid), None, "{invalid:?}"); + } +} + +async fn collect( + mut body: BoxStream<'static, Result>, +) -> (Vec, Option) { + let mut out = Vec::new(); + while let Some(chunk) = body.next().await { + match chunk { + Ok(chunk) => out.extend_from_slice(&chunk), + Err(error) => return (out, Some(error.to_string())), + } + } + (out, None) +} + +fn proxy(lookup: Arc) -> AudioProxy { + AudioProxy::with_http(test_http(lookup)) +} + +#[tokio::test] +async fn proxy_passes_ranges_through() { + let server = TestServer::start(); + let audio = noise(300_000, 7); + server.set("/ep.mp3", Route::new(audio.clone(), "audio/mpeg")); + let proxy = proxy(ScriptedLookup::with(&[])); + let target = server.url("/ep.mp3"); + + let full = proxy.fetch(&target, None).await.unwrap(); + assert_eq!(full.status, 200); + assert_eq!(full.content_type.as_deref(), Some("audio/mpeg")); + assert_eq!(full.content_length, Some(audio.len() as u64)); + assert!(full.accept_ranges); + assert_eq!(full.content_range, None); + assert_eq!(collect(full.body).await, (audio.clone(), None)); + + for (range, start, end) in [ + ("bytes=1000-1999", 1000, 2000), + ("bytes=299000-", 299_000, 300_000), + ("bytes=-500", 299_500, 300_000), + ("bytes=0-", 0, 300_000), + ] { + let part = proxy.fetch(&target, Some(range)).await.unwrap(); + assert_eq!(part.status, 206, "{range}"); + assert_eq!( + part.content_range, + Some(format!("bytes {start}-{}/{}", end - 1, audio.len())), + "{range}" + ); + assert_eq!(part.content_length, Some((end - start) as u64)); + assert_eq!(collect(part.body).await, (audio[start..end].to_vec(), None)); + } + + let unsatisfiable = proxy.fetch(&target, Some("bytes=400000-")).await.unwrap(); + assert_eq!(unsatisfiable.status, 416); + assert_eq!( + unsatisfiable.content_range, + Some(format!("bytes */{}", audio.len())) + ); + assert_eq!(collect(unsatisfiable.body).await, (Vec::new(), None)); + + // Malformed or multi-range requests fetch the whole file. + let whole = proxy.fetch(&target, Some("bytes=5-6,9-10")).await.unwrap(); + assert_eq!(whole.status, 200); + + let requests = server.requests("/ep.mp3"); + let ranges: Vec> = requests + .iter() + .map(|request| request.get("range").map(String::as_str)) + .collect(); + assert_eq!( + ranges, + vec![ + None, + Some("bytes=1000-1999"), + Some("bytes=299000-"), + Some("bytes=-500"), + Some("bytes=0-"), + Some("bytes=400000-"), + None, + ] + ); + for request in &requests { + assert_eq!(request["accept-encoding"], "identity"); + assert!(!request.contains_key("cookie")); + assert!(!request.contains_key("authorization")); + } +} + +#[tokio::test] +async fn proxy_handles_servers_without_ranges_and_redirects() { + let server = TestServer::start(); + let audio = noise(50_000, 3); + server.set( + "/plain.mp3", + Route::new(audio.clone(), "audio/mpeg").no_ranges(), + ); + server.set("/track/ep.mp3", Route::redirect(302, "/cdn/ep.mp3")); + server.set("/cdn/ep.mp3", Route::new(audio.clone(), "audio/mpeg")); + server.set( + "/track/private.mp3", + Route::redirect(302, "http://172.20.0.3/ep.mp3"), + ); + let proxy = proxy(ScriptedLookup::with(&[])); + + // No range support: the whole file with 200 and no Accept-Ranges, so the + // browser knows it cannot seek by range. + let plain = proxy + .fetch(&server.url("/plain.mp3"), Some("bytes=100-")) + .await + .unwrap(); + assert_eq!(plain.status, 200); + assert!(!plain.accept_ranges); + assert_eq!(plain.content_range, None); + assert_eq!(collect(plain.body).await.0, audio); + + // Tracking redirects keep the range on every hop. + let part = proxy + .fetch(&server.url("/track/ep.mp3"), Some("bytes=10-19")) + .await + .unwrap(); + assert_eq!(part.status, 206); + assert_eq!(collect(part.body).await.0, audio[10..20].to_vec()); + assert_eq!(server.requests("/track/ep.mp3")[0]["range"], "bytes=10-19"); + assert_eq!(server.requests("/cdn/ep.mp3")[0]["range"], "bytes=10-19"); + + let Err(error) = proxy.fetch(&server.url("/track/private.mp3"), None).await else { + panic!("redirect to a private address was followed"); + }; + assert!( + matches!( + &error, + ProxyError::Upstream(GuardError::Blocked(blocked)) + if blocked.class == AddressClass::Private + ), + "{error:?}" + ); + // An IP literal is refused for what the URL says. + assert_eq!(error.public_message(), REFUSED_MESSAGE); +} + +#[tokio::test] +async fn proxy_answers_alike_for_internal_and_unreachable_names() { + // Episode URLs can come from the user's own clients: the answer must + // not tell a name with a private address from one that does not exist + // or a server that refuses the connection. + let port = std::net::TcpListener::bind("127.0.0.1:0") + .unwrap() + .local_addr() + .unwrap() + .port(); + let proxy = proxy(ScriptedLookup::with(&[ + ("db.corp.test", &[&["192.168.7.7"]]), + ("closed.test", &[&["127.0.0.1"]]), + ])); + let mut answers = Vec::new(); + for host in ["db.corp.test", "nowhere.corp.test", "closed.test"] { + let target = format!("http://{host}:{port}/episode.mp3"); + let error = proxy.fetch(&target, None).await.unwrap_err(); + assert!(matches!(error, ProxyError::Upstream(_)), "{error:?}"); + answers.push((error.http_status(), error.public_message())); + } + assert_eq!(answers[0].1, "could not connect to the publisher"); + assert!( + answers.iter().all(|answer| *answer == answers[0]), + "{answers:?}" + ); +} + +/// `noise` that starts like a file of the given format. +fn audio_file(magic: &[u8], len: usize) -> Vec { + let mut file = magic.to_vec(); + file.extend(noise(len - magic.len(), 3)); + file +} + +#[tokio::test] +async fn proxy_only_serves_audio() { + let server = TestServer::start(); + let audio = noise(1000, 5); + let mp3 = audio_file(b"ID3\x04\0\0", 1000); + server.set( + "/page.mp3", + Route::new("", "text/html"), + ); + server.set("/video.mp4", Route::new(audio.clone(), "video/mp4")); + // Generic types: the URL extension, or else the file's first bytes. + server.set( + "/binary", + Route::new(audio.clone(), "application/octet-stream"), + ); + server.set( + "/zip", + Route::new(audio_file(b"PK\x03\x04", 1000), "application/octet-stream"), + ); + server.set( + "/episode", + Route::new(mp3.clone(), "application/octet-stream"), + ); + server.set("/untyped", Route::new(mp3.clone(), "x").no_content_type()); + server.set( + "/binary.m4a", + Route::new(audio.clone(), "binary/octet-stream"), + ); + server.set( + "/untyped.ogg", + Route::new(audio.clone(), "x").no_content_type(), + ); + server.set("/ogg", Route::new(audio.clone(), "application/ogg")); + server.set( + "/typed", + Route::new(audio.clone(), "audio/mp4; codecs=\"mp4a.40.2\""), + ); + server.set("/missing", Route::new("", "text/plain").status(404)); + server.set("/error", Route::new("", "audio/mpeg").status(500)); + let proxy = proxy(ScriptedLookup::with(&[])); + let content_type = |path: &'static str, range: Option<&'static str>| { + let proxy = proxy.clone(); + let target = server.url(path); + async move { + proxy + .fetch(&target, range) + .await + .map(|audio| audio.content_type.unwrap()) + } + }; + for refused in ["/page.mp3", "/video.mp4", "/binary", "/zip"] { + for range in [None, Some("bytes=10-")] { + assert!( + matches!( + content_type(refused, range).await, + Err(ProxyError::ContentType(_)) + ), + "{refused} {range:?}" + ); + } + } + assert_eq!(content_type("/episode", None).await.unwrap(), "audio/mpeg"); + assert_eq!(content_type("/untyped", None).await.unwrap(), "audio/mpeg"); + // The sniffed bytes are still sent to the browser. + let whole = proxy.fetch(&server.url("/episode"), None).await.unwrap(); + assert_eq!(collect(whole.body).await, (mp3.clone(), None)); + // A range past the start: the file's head is read with its own request. + let before = server.requests("/episode").len(); + let part = proxy + .fetch(&server.url("/episode"), Some("bytes=100-199")) + .await + .unwrap(); + assert_eq!(part.content_type.as_deref(), Some("audio/mpeg")); + assert_eq!(collect(part.body).await, (mp3[100..200].to_vec(), None)); + let ranges: Vec = server.requests("/episode")[before..] + .iter() + .map(|request| request["range"].clone()) + .collect(); + assert_eq!(ranges, ["bytes=100-199", "bytes=0-15"]); + // Safari and AVFoundation probe with a two-byte range from the start: + // too short to recognize the file, so its head is read separately. + let before = server.requests("/episode").len(); + let probe = proxy + .fetch(&server.url("/episode"), Some("bytes=0-1")) + .await + .unwrap(); + assert_eq!(probe.status, 206); + assert_eq!(probe.content_type.as_deref(), Some("audio/mpeg")); + assert_eq!(collect(probe.body).await, (mp3[..2].to_vec(), None)); + let ranges: Vec = server.requests("/episode")[before..] + .iter() + .map(|request| request["range"].clone()) + .collect(); + assert_eq!(ranges, ["bytes=0-1", "bytes=0-15"]); + // A long enough range from the start needs no second request. + let before = server.requests("/episode").len(); + let start = proxy + .fetch(&server.url("/episode"), Some("bytes=0-99")) + .await + .unwrap(); + assert_eq!(start.content_type.as_deref(), Some("audio/mpeg")); + assert_eq!(collect(start.body).await, (mp3[..100].to_vec(), None)); + assert_eq!(server.requests("/episode").len(), before + 1); + // A probe of a file that is not audio is still refused. + assert!(matches!( + proxy.fetch(&server.url("/zip"), Some("bytes=0-1")).await, + Err(ProxyError::ContentType(_)) + )); + assert_eq!( + content_type("/binary.m4a", None).await.unwrap(), + "audio/mp4" + ); + assert_eq!( + content_type("/untyped.ogg", None).await.unwrap(), + "audio/ogg" + ); + assert_eq!(content_type("/ogg", None).await.unwrap(), "application/ogg"); + assert_eq!( + content_type("/typed", None).await.unwrap(), + "audio/mp4; codecs=\"mp4a.40.2\"" + ); + let missing = content_type("/missing", None).await.unwrap_err(); + assert!(matches!(missing, ProxyError::Status(404))); + assert_eq!(missing.http_status(), 502); + assert!(matches!( + content_type("/error", None).await, + Err(ProxyError::Status(500)) + )); + assert!(matches!( + proxy.fetch("not a url", None).await, + Err(ProxyError::InvalidUrl(_)) + )); +} + +#[test] +fn audio_formats_are_recognized_by_their_first_bytes() { + for (head, format) in [ + (&b"ID3\x04\0\0\0\0"[..], "audio/mpeg"), + (&[0xFF, 0xFB, 0x90, 0x64][..], "audio/mpeg"), + (&[0xFF, 0xF3, 0x48, 0xC4][..], "audio/mpeg"), + (&[0xFF, 0xFD, 0x94, 0x00][..], "audio/mpeg"), + (&[0xFF, 0xE3, 0x18, 0xC4][..], "audio/mpeg"), + (&[0xFF, 0xF9, 0x4C, 0x80][..], "audio/aac"), + (&[0xFF, 0xF1, 0x50, 0x80][..], "audio/aac"), + (&b"OggS\0\x02"[..], "audio/ogg"), + (&b"fLaC\0\0\0\x22"[..], "audio/flac"), + (&b"RIFF\x24\x08\0\0WAVEfmt "[..], "audio/wav"), + (&b"\0\0\0\x20ftypM4A \0\0\0\0"[..], "audio/mp4"), + ] { + assert_eq!(sniff_audio(head), Some(format), "{head:?}"); + } + for head in [ + &b"\0\0\0\x20ftypisom\0\0\x02\0"[..], + &b"\0\0\0\x18ftypmp42"[..], + &b"PK\x03\x04"[..], + &b""[..], + &b"%PDF-1.7"[..], + &b"RIFF\x24\x08\0\0AVI "[..], + &[0xFF, 0xD8, 0xFF, 0xE0][..], + // UTF-16LE text (byte order mark FF FE) and FF FF are not frames. + &[0xFF, 0xFE, 0x3C, 0x00, 0x68, 0x00][..], + &[0xFF, 0xFE, 0x41, 0x00][..], + &[0xFF, 0xFF, 0x90, 0x00][..], + // Frame sync with an invalid bitrate, free bitrate or sample rate. + &[0xFF, 0xFB, 0xF0, 0x00][..], + &[0xFF, 0xFB, 0x00, 0x00][..], + &[0xFF, 0xFB, 0x9C, 0x00][..], + // ADTS sync with a reserved sampling frequency index. + &[0xFF, 0xF1, 0x3C, 0x80][..], + &[0xFF, 0xFB][..], + &[0xFF][..], + &[][..], + ] { + assert_eq!(sniff_audio(head), None, "{head:?}"); + } +} + +#[tokio::test] +async fn proxy_limits_size_and_reports_dropped_streams() { + let server = TestServer::start(); + let audio = noise(100_000, 9); + server.set("/sized.mp3", Route::new(audio.clone(), "audio/mpeg")); + server.set( + "/chunked.mp3", + Route::new(audio.clone(), "audio/mpeg").no_length(), + ); + server.set( + "/cut.mp3", + Route::new(audio.clone(), "audio/mpeg").cut(40_000), + ); + let limited = proxy(ScriptedLookup::with(&[])).with_limits(60_000, Duration::from_secs(60)); + assert!(matches!( + limited.fetch(&server.url("/sized.mp3"), None).await, + Err(ProxyError::TooLarge(100_000)) + )); + // A range below the limit is fine. + let part = limited + .fetch(&server.url("/sized.mp3"), Some("bytes=0-49999")) + .await + .unwrap(); + assert_eq!(collect(part.body).await, (audio[..50_000].to_vec(), None)); + // Without a declared length the stream stops at the limit. + let chunked = limited + .fetch(&server.url("/chunked.mp3"), None) + .await + .unwrap(); + let (received, error) = collect(chunked.body).await; + assert!(received.len() <= 60_000); + assert!(error.unwrap().contains("size limit")); + // A connection dropped mid-body ends the stream with an error instead + // of a silently truncated success. + let cut = proxy(ScriptedLookup::with(&[])) + .fetch(&server.url("/cut.mp3"), None) + .await + .unwrap(); + let (received, error) = collect(cut.body).await; + assert!(received.len() <= 40_000); + assert!(error.is_some()); +} + +#[tokio::test] +async fn proxy_stream_has_a_time_limit() { + let server = TestServer::start(); + server.set( + "/slow.mp3", + Route::new(noise(10, 1), "audio/mpeg").delay_body(Duration::from_secs(3)), + ); + let limited = proxy(ScriptedLookup::with(&[])).with_limits(1 << 20, Duration::from_millis(300)); + let audio = limited.fetch(&server.url("/slow.mp3"), None).await.unwrap(); + let (_, error) = collect(audio.body).await; + assert!(error.unwrap().contains("time limit")); +} diff --git a/src/podcasts/queries.rs b/src/podcasts/queries.rs new file mode 100644 index 0000000..d663d87 --- /dev/null +++ b/src/podcasts/queries.rs @@ -0,0 +1,723 @@ +//! Read-only queries for the podcast UI and the player. + +use anyhow::Result; +use music_dht::podcast::{ + EpisodeId, EpisodeMeta, EpisodeType, ListenSession, PodcastEpisodeRef, PodcastId, PodcastMeta, + ProgressRecord, ProgressStatus, SubscriptionRecord, effective_share, +}; +use sqlx::postgres::PgRow; +use sqlx::{PgConnection, Postgres, QueryBuilder, Row}; + +use super::apply; +use super::{ + PodcastStore, SESSION_COLUMNS, episode_id_col, episode_search_norm, feed_url_col, + podcast_id_col, progress_columns, progress_from_row, session_from_row, u32_from_sql, +}; + +/// Which podcasts [`PodcastStore::list_podcasts`] returns. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub(crate) enum PodcastListFilter { + /// Podcasts with an active subscription. + #[default] + Subscribed, + /// Every podcast in the catalog, including unsubscribed ones and + /// podcasts only known from synced progress or queue items. + All, +} + +/// A podcast with its personal state and episode counters. +#[derive(Debug, Clone, PartialEq)] +pub(crate) struct PodcastSummary { + /// Podcast id. + pub(crate) podcast_id: PodcastId, + /// Local channel metadata; `feed_url` is the URL this account fetches. + pub(crate) meta: PodcastMeta, + /// Whether the subscription register says subscribed. + pub(crate) subscribed: bool, + /// Whether the podcast must currently be published (`effective_share`). + pub(crate) shared: bool, + /// Episodes in the catalog. + pub(crate) episode_count: u64, + /// Episodes not completed (no progress, not started or in progress). + pub(crate) unplayed_count: u64, + /// Episodes in progress. + pub(crate) in_progress_count: u64, + /// Newest publication time. + pub(crate) latest_published_at_ms: Option, + /// Numbered seasons (episodes without a season are not counted). + pub(crate) season_count: u64, +} + +/// One season group of a podcast. `season == None` groups the episodes +/// without `itunes:season`. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct SeasonSummary { + /// Season number; `None` for episodes without a season. + pub(crate) season: Option, + /// Episodes in the group. + pub(crate) episode_count: u64, + /// Episodes not completed. + pub(crate) unplayed_count: u64, + /// Oldest publication time. + pub(crate) first_published_at_ms: Option, + /// Newest publication time. + pub(crate) last_published_at_ms: Option, +} + +/// Season filter of an [`EpisodeQuery`]. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub(crate) enum SeasonFilter { + /// All episodes. + #[default] + Any, + /// Episodes of one season. + Season(u32), + /// Episodes without a season. + NoSeason, +} + +/// Listening-state filter of an [`EpisodeQuery`]. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub(crate) enum PlayedFilter { + /// All episodes. + #[default] + Any, + /// Not completed (not started or in progress). + Unplayed, + /// Never started or marked unplayed. + NotStarted, + /// In progress. + InProgress, + /// Completed. + Played, +} + +/// Episode ordering of an [`EpisodeQuery`]. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub(crate) enum EpisodeSort { + /// Newest publication first; undated episodes last. + #[default] + NewestFirst, + /// Oldest publication first; undated episodes last. + OldestFirst, + /// By season, then episode number, ascending; unnumbered last. + EpisodeNumber, + /// By season, then episode number, descending; unnumbered last. + EpisodeNumberDesc, +} + +impl EpisodeSort { + fn order_by(self) -> &'static str { + match self { + Self::NewestFirst => "e.published_at_ms IS NULL, e.published_at_ms DESC, e.episode_id", + Self::OldestFirst => "e.published_at_ms IS NULL, e.published_at_ms, e.episode_id", + Self::EpisodeNumber => { + "e.season IS NULL, e.season, e.episode_number IS NULL, e.episode_number, \ + e.published_at_ms IS NULL, e.published_at_ms, e.episode_id" + } + Self::EpisodeNumberDesc => { + "e.season IS NULL, e.season DESC, e.episode_number IS NULL, \ + e.episode_number DESC, e.published_at_ms IS NULL, e.published_at_ms DESC, \ + e.episode_id" + } + } + } +} + +#[cfg(test)] +pub(crate) fn order_by_for_tests(sort: EpisodeSort) -> &'static str { + sort.order_by() +} + +/// Parameters of [`PodcastStore::episodes`]. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct EpisodeQuery { + /// Podcast. + pub(crate) podcast_id: PodcastId, + /// Season filter. + pub(crate) season: SeasonFilter, + /// Listening-state filter. + pub(crate) played: PlayedFilter, + /// Text searched in titles and descriptions (`normalize_name`: Unicode + /// NFKC, case-insensitive substring match). Blank means no filter. + pub(crate) search: Option, + /// Ordering. + pub(crate) sort: EpisodeSort, + /// Maximum rows (`None`: all). + pub(crate) limit: Option, + /// Rows to skip. + pub(crate) offset: usize, +} + +impl EpisodeQuery { + /// All episodes of a podcast, newest first. + pub(crate) fn new(podcast_id: PodcastId) -> Self { + Self { + podcast_id, + season: SeasonFilter::Any, + played: PlayedFilter::Any, + search: None, + sort: EpisodeSort::NewestFirst, + limit: None, + offset: 0, + } + } +} + +/// An episode with its personal state. +#[derive(Debug, Clone, PartialEq)] +pub(crate) struct EpisodeListItem { + /// Episode metadata (feed or synced reference). + pub(crate) episode: EpisodeMeta, + /// Progress register, if any. + pub(crate) progress: Option, + /// Distinct completed listening sessions (`completed_listen_count`). + pub(crate) completed_count: u64, + /// Whether the latest refresh of the feed listed the episode. + pub(crate) in_feed: bool, +} + +impl EpisodeListItem { + /// Listening status (`NotStarted` without a register). + pub(crate) fn status(&self) -> ProgressStatus { + self.progress + .as_ref() + .map_or(ProgressStatus::NotStarted, |progress| progress.status) + } + + /// Position ordinary Play resumes from. + pub(crate) fn resume_position_ms(&self) -> i64 { + self.progress + .as_ref() + .map_or(0, ProgressRecord::resume_position_ms) + } +} + +/// A listening session in history, with display titles when known. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct PodcastListenEntry { + /// The session. + pub(crate) session: ListenSession, + /// Episode title from the catalog. + pub(crate) episode_title: Option, + /// Podcast title from the catalog. + pub(crate) podcast_title: Option, +} + +const EPISODE_COLUMNS: &str = "e.episode_id, e.podcast_id, e.guid, e.title, e.description, \ + e.season, e.episode_number, e.episode_type, e.published_at_ms, e.duration_ms, \ + e.audio_url, e.mime, e.byte_len, e.image_url, e.link"; + +const NOT_COMPLETED: &str = "COALESCE(g.status, 'not_started') <> 'completed'"; + +fn episode_from_row(row: &PgRow) -> sqlx::Result { + Ok(EpisodeMeta { + episode_id: episode_id_col(row, "episode_id")?, + podcast_id: podcast_id_col(row, "podcast_id")?, + guid: row.try_get("guid")?, + title: row.try_get("title")?, + description: row.try_get("description")?, + season: u32_from_sql(row.try_get("season")?), + episode_number: u32_from_sql(row.try_get("episode_number")?), + episode_type: EpisodeType::parse(&row.try_get::("episode_type")?), + published_at_ms: row.try_get("published_at_ms")?, + duration_ms: row.try_get("duration_ms")?, + audio_url: row.try_get("audio_url")?, + mime: row.try_get("mime")?, + byte_len: row + .try_get::, _>("byte_len")? + .and_then(|len| u64::try_from(len).ok()), + image_url: row.try_get("image_url")?, + link: row.try_get("link")?, + }) +} + +fn list_item_from_row(row: &PgRow) -> sqlx::Result { + let progress = if row.try_get::, _>("g_episode_id")?.is_some() { + Some(progress_from_row(row, "g_")?) + } else { + None + }; + Ok(EpisodeListItem { + episode: episode_from_row(row)?, + in_feed: row.try_get("in_feed")?, + progress, + completed_count: count(row.try_get("completed_count")?), + }) +} + +fn count(value: i64) -> u64 { + u64::try_from(value).unwrap_or(0) +} + +pub(crate) async fn episode_meta_in( + conn: &mut PgConnection, + user_id: i64, + episode_id: &EpisodeId, +) -> Result> { + let row = sqlx::query(&format!( + "SELECT {EPISODE_COLUMNS} FROM furumusic__podcast_episode e + WHERE e.user_id = $1 AND e.episode_id = $2" + )) + .bind(user_id) + .bind(episode_id.as_str()) + .fetch_optional(&mut *conn) + .await?; + Ok(row.as_ref().map(episode_from_row).transpose()?) +} + +pub(crate) async fn podcast_meta_in( + conn: &mut PgConnection, + user_id: i64, + podcast_id: &PodcastId, +) -> Result> { + let row = sqlx::query( + "SELECT feed_url, title, author, description, link, image_url + FROM furumusic__podcast WHERE user_id = $1 AND podcast_id = $2", + ) + .bind(user_id) + .bind(podcast_id.as_str()) + .fetch_optional(&mut *conn) + .await?; + let Some(row) = row else { + return Ok(None); + }; + Ok(Some(PodcastMeta { + feed_url: feed_url_col(&row, "feed_url")?, + title: row.try_get("title")?, + author: row.try_get("author")?, + description: row.try_get("description")?, + link: row.try_get("link")?, + image_url: row.try_get("image_url")?, + })) +} + +/// Queue reference of a catalog episode, with `progress_generation` set to +/// the generation of its progress register (handoff hint). +pub(crate) async fn episode_ref_in( + conn: &mut PgConnection, + user_id: i64, + episode_id: &EpisodeId, +) -> Result> { + let Some(episode) = episode_meta_in(conn, user_id, episode_id).await? else { + return Ok(None); + }; + let Some(podcast) = podcast_meta_in(conn, user_id, &episode.podcast_id).await? else { + return Ok(None); + }; + let mut reference = episode.to_ref(&podcast); + reference.progress_generation = apply::load_progress(conn, user_id, episode_id) + .await? + .map(|progress| progress.generation); + Ok(Some(reference)) +} + +async fn summary_in( + conn: &mut PgConnection, + user_id: i64, + podcast_id: &PodcastId, +) -> Result> { + let Some(meta) = podcast_meta_in(conn, user_id, podcast_id).await? else { + return Ok(None); + }; + let row = sqlx::query(&format!( + "SELECT COUNT(*) AS episode_count, + COUNT(*) FILTER (WHERE {NOT_COMPLETED}) AS unplayed_count, + COUNT(*) FILTER (WHERE g.status = 'in_progress') AS in_progress_count, + MAX(e.published_at_ms) AS latest, + (SELECT COUNT(DISTINCT e2.season) FROM furumusic__podcast_episode e2 + WHERE e2.user_id = $1 AND e2.podcast_id = $2) AS season_count + FROM furumusic__podcast_episode e + LEFT JOIN furumusic__podcast_progress g + ON g.user_id = e.user_id AND g.episode_id = e.episode_id + WHERE e.user_id = $1 AND e.podcast_id = $2" + )) + .bind(user_id) + .bind(podcast_id.as_str()) + .fetch_one(&mut *conn) + .await?; + let subscription = apply::load_subscription_register(conn, user_id, podcast_id).await?; + let share = apply::load_share(conn, user_id, podcast_id).await?; + Ok(Some(PodcastSummary { + podcast_id: podcast_id.clone(), + meta, + subscribed: subscription + .as_ref() + .is_some_and(|record| record.subscribed), + shared: effective_share(subscription.as_ref(), share.as_ref()), + episode_count: count(row.try_get("episode_count")?), + unplayed_count: count(row.try_get("unplayed_count")?), + in_progress_count: count(row.try_get("in_progress_count")?), + latest_published_at_ms: row.try_get("latest")?, + season_count: count(row.try_get("season_count")?), + })) +} + +/// Computes the cached search text of episodes that do not have it yet +/// (rows imported by the `m_0057` migration). +async fn backfill_search_norm( + conn: &mut PgConnection, + user_id: i64, + podcast_id: &PodcastId, +) -> Result<()> { + let rows = sqlx::query( + "SELECT episode_id, title, description FROM furumusic__podcast_episode + WHERE user_id = $1 AND podcast_id = $2 AND search_norm IS NULL", + ) + .bind(user_id) + .bind(podcast_id.as_str()) + .fetch_all(&mut *conn) + .await?; + for row in rows { + let title: String = row.try_get("title")?; + let description: Option = row.try_get("description")?; + sqlx::query( + "UPDATE furumusic__podcast_episode SET search_norm = $3 + WHERE user_id = $1 AND episode_id = $2 AND search_norm IS NULL", + ) + .bind(user_id) + .bind(row.try_get::("episode_id")?) + .bind(episode_search_norm(&title, description.as_deref())) + .execute(&mut *conn) + .await?; + } + Ok(()) +} + +impl PodcastStore { + /// Podcasts with subscription state and counters, sorted by title. + pub(crate) async fn list_podcasts( + &self, + user_id: i64, + filter: PodcastListFilter, + ) -> Result> { + let mut conn = self.pool.acquire().await?; + let join = match filter { + PodcastListFilter::Subscribed => { + "JOIN furumusic__podcast_subscription r + ON r.user_id = p.user_id AND r.podcast_id = p.podcast_id AND r.subscribed" + } + PodcastListFilter::All => "", + }; + let rows = sqlx::query(&format!( + "SELECT p.podcast_id FROM furumusic__podcast p {join} + WHERE p.user_id = $1 + ORDER BY lower(p.title), p.podcast_id" + )) + .bind(user_id) + .fetch_all(&mut *conn) + .await?; + let ids = rows + .iter() + .map(|row| podcast_id_col(row, "podcast_id")) + .collect::>>()?; + let mut out = Vec::with_capacity(ids.len()); + for id in ids { + if let Some(summary) = summary_in(&mut conn, user_id, &id).await? { + out.push(summary); + } + } + Ok(out) + } + + /// One podcast with subscription state and counters. + pub(crate) async fn podcast( + &self, + user_id: i64, + podcast_id: &PodcastId, + ) -> Result> { + let mut conn = self.pool.acquire().await?; + summary_in(&mut conn, user_id, podcast_id).await + } + + /// The subscription register (with the stored alias set), if any. + pub(crate) async fn subscription( + &self, + user_id: i64, + podcast_id: &PodcastId, + ) -> Result> { + let mut conn = self.pool.acquire().await?; + apply::load_subscription_register(&mut conn, user_id, podcast_id).await + } + + /// The share register, if any. + #[cfg(test)] + pub(crate) async fn share( + &self, + user_id: i64, + podcast_id: &PodcastId, + ) -> Result> { + let mut conn = self.pool.acquire().await?; + apply::load_share(&mut conn, user_id, podcast_id).await + } + + /// The bounded, synced feed alias set of a podcast. + #[cfg(test)] + pub(crate) async fn feed_aliases( + &self, + user_id: i64, + podcast_id: &PodcastId, + ) -> Result> { + let mut conn = self.pool.acquire().await?; + apply::load_aliases(&mut conn, user_id, podcast_id).await + } + + /// Season groups of a podcast: numbered seasons ascending, then the + /// group of episodes without a season. + pub(crate) async fn seasons( + &self, + user_id: i64, + podcast_id: &PodcastId, + ) -> Result> { + let rows = sqlx::query(&format!( + "SELECT s.season, s.episode_count, s.first_published_at_ms, s.last_published_at_ms, + (SELECT COUNT(*) FROM furumusic__podcast_episode e + LEFT JOIN furumusic__podcast_progress g + ON g.user_id = e.user_id AND g.episode_id = e.episode_id + WHERE e.user_id = s.user_id AND e.podcast_id = s.podcast_id + AND e.season IS NOT DISTINCT FROM s.season + AND {NOT_COMPLETED}) AS unplayed_count + FROM furumusic__podcast_season s + WHERE s.user_id = $1 AND s.podcast_id = $2 + ORDER BY s.season IS NULL, s.season" + )) + .bind(user_id) + .bind(podcast_id.as_str()) + .fetch_all(&self.pool) + .await?; + let seasons = rows + .iter() + .map(|row| { + Ok(SeasonSummary { + season: u32_from_sql(row.try_get("season")?), + episode_count: count(row.try_get("episode_count")?), + first_published_at_ms: row.try_get("first_published_at_ms")?, + last_published_at_ms: row.try_get("last_published_at_ms")?, + unplayed_count: count(row.try_get("unplayed_count")?), + }) + }) + .collect::>>()?; + Ok(seasons) + } + + /// Episodes of a podcast or season with their progress. + pub(crate) async fn episodes( + &self, + user_id: i64, + query: &EpisodeQuery, + ) -> Result> { + let mut conn = self.pool.acquire().await?; + let needle = query + .search + .as_deref() + .map(music_dht::normalize_name) + .filter(|needle| !needle.is_empty()); + if needle.is_some() { + backfill_search_norm(&mut conn, user_id, &query.podcast_id).await?; + } + let mut sql: QueryBuilder<'_, Postgres> = QueryBuilder::new(format!( + "SELECT {EPISODE_COLUMNS}, e.in_feed, {}, + (SELECT COUNT(*) FROM furumusic__podcast_listen_session s + WHERE s.user_id = e.user_id AND s.episode_id = e.episode_id + AND s.completed) AS completed_count + FROM furumusic__podcast_episode e + LEFT JOIN furumusic__podcast_progress g + ON g.user_id = e.user_id AND g.episode_id = e.episode_id + WHERE e.user_id = ", + progress_columns("g", "g_") + )); + sql.push_bind(user_id); + sql.push(" AND e.podcast_id = "); + sql.push_bind(query.podcast_id.as_str().to_string()); + match query.season { + SeasonFilter::Any => {} + SeasonFilter::Season(season) => { + sql.push(" AND e.season = "); + sql.push_bind(i64::from(season)); + } + SeasonFilter::NoSeason => { + sql.push(" AND e.season IS NULL"); + } + } + match query.played { + PlayedFilter::Any => {} + PlayedFilter::Unplayed => { + sql.push(format!(" AND {NOT_COMPLETED}")); + } + PlayedFilter::NotStarted => { + sql.push(" AND COALESCE(g.status, 'not_started') = 'not_started'"); + } + PlayedFilter::InProgress => { + sql.push(" AND g.status = 'in_progress'"); + } + PlayedFilter::Played => { + sql.push(" AND g.status = 'completed'"); + } + } + if let Some(needle) = needle { + sql.push(" AND strpos(e.search_norm, "); + sql.push_bind(needle); + sql.push(") > 0"); + } + sql.push(format!(" ORDER BY {}", query.sort.order_by())); + if let Some(limit) = query.limit { + sql.push(" LIMIT "); + sql.push_bind(i64::try_from(limit).unwrap_or(i64::MAX)); + } + sql.push(" OFFSET "); + sql.push_bind(i64::try_from(query.offset).unwrap_or(i64::MAX)); + let rows = sql.build().fetch_all(&mut *conn).await?; + Ok(rows + .iter() + .map(list_item_from_row) + .collect::>>()?) + } + + /// One episode with its progress. + pub(crate) async fn episode( + &self, + user_id: i64, + episode_id: &EpisodeId, + ) -> Result> { + let row = sqlx::query(&format!( + "SELECT {EPISODE_COLUMNS}, e.in_feed, {}, + (SELECT COUNT(*) FROM furumusic__podcast_listen_session s + WHERE s.user_id = e.user_id AND s.episode_id = e.episode_id + AND s.completed) AS completed_count + FROM furumusic__podcast_episode e + LEFT JOIN furumusic__podcast_progress g + ON g.user_id = e.user_id AND g.episode_id = e.episode_id + WHERE e.user_id = $1 AND e.episode_id = $2", + progress_columns("g", "g_") + )) + .bind(user_id) + .bind(episode_id.as_str()) + .fetch_optional(&self.pool) + .await?; + Ok(row.as_ref().map(list_item_from_row).transpose()?) + } + + /// Position ordinary Play resumes from + /// (`ProgressRecord::resume_position_ms`, 0 without progress). + #[cfg(test)] + pub(crate) async fn resume_position_ms( + &self, + user_id: i64, + episode_id: &EpisodeId, + ) -> Result { + Ok(self + .progress(user_id, episode_id) + .await? + .map_or(0, |progress| progress.resume_position_ms())) + } + + /// Listening history (sessions that `should_record`), newest first. + pub(crate) async fn listen_history( + &self, + user_id: i64, + podcast_id: Option<&PodcastId>, + limit: usize, + ) -> Result> { + let rows = sqlx::query(&format!( + "SELECT {SESSION_COLUMNS}, e.title AS episode_title, p.title AS podcast_title + FROM furumusic__podcast_listen_session s + LEFT JOIN furumusic__podcast_episode e + ON e.user_id = s.user_id AND e.episode_id = s.episode_id + LEFT JOIN furumusic__podcast p + ON p.user_id = s.user_id AND p.podcast_id = s.podcast_id + WHERE s.user_id = $1 AND (s.completed OR s.listened_ms >= $2) + AND ($3::text IS NULL OR s.podcast_id = $3) + ORDER BY s.ended_at_ms DESC, s.session_id DESC + LIMIT $4" + )) + .bind(user_id) + .bind(music_dht::podcast::MIN_RECORDED_EPISODE_LISTEN_MS) + .bind(podcast_id.map(PodcastId::as_str)) + .bind(i64::try_from(limit).unwrap_or(i64::MAX)) + .fetch_all(&self.pool) + .await?; + Ok(rows + .iter() + .map(|row| { + Ok(PodcastListenEntry { + session: session_from_row(row)?, + episode_title: row.try_get("episode_title")?, + podcast_title: row.try_get("podcast_title")?, + }) + }) + .collect::>>()?) + } + + /// Number of distinct completed sessions of an episode. + #[cfg(test)] + pub(crate) async fn completed_listen_count( + &self, + user_id: i64, + episode_id: &EpisodeId, + ) -> Result { + let total: i64 = sqlx::query_scalar( + "SELECT COUNT(*) FROM furumusic__podcast_listen_session + WHERE user_id = $1 AND episode_id = $2 AND completed", + ) + .bind(user_id) + .bind(episode_id.as_str()) + .fetch_one(&self.pool) + .await?; + Ok(count(total)) + } + + /// Generations of the progress registers of `episode_ids` (episodes + /// without a register are missing from the map). + /// + /// Outgoing playback states carry them as + /// `PodcastEpisodeRef::progress_generation`, so the device that takes + /// over writes its explicit action after everything this account saw. + pub(crate) async fn progress_generations( + &self, + user_id: i64, + episode_ids: &[EpisodeId], + ) -> Result> { + if episode_ids.is_empty() { + return Ok(Default::default()); + } + let ids: Vec<&str> = episode_ids.iter().map(EpisodeId::as_str).collect(); + let rows = sqlx::query( + "SELECT episode_id, generation FROM furumusic__podcast_progress + WHERE user_id = $1 AND episode_id = ANY($2)", + ) + .bind(user_id) + .bind(&ids) + .fetch_all(&self.pool) + .await?; + rows.iter() + .map(|row| { + Ok(( + episode_id_col(row, "episode_id")?, + super::generation_from_sql(row.try_get("generation")?), + )) + }) + .collect::>() + .map_err(Into::into) + } + + /// Audio URL and enclosure type of an episode in this account's catalog + /// (`None` when the account does not know the episode). The audio proxy + /// and the playback source endpoint take URLs only from here. + pub(crate) async fn audio_source( + &self, + user_id: i64, + episode_id: &EpisodeId, + ) -> Result)>> { + let row = sqlx::query( + "SELECT audio_url, mime FROM furumusic__podcast_episode + WHERE user_id = $1 AND episode_id = $2", + ) + .bind(user_id) + .bind(episode_id.as_str()) + .fetch_optional(&self.pool) + .await?; + Ok(row + .map(|row| Ok::<_, sqlx::Error>((row.try_get("audio_url")?, row.try_get("mime")?))) + .transpose()?) + } +} diff --git a/src/podcasts/refresh.rs b/src/podcasts/refresh.rs new file mode 100644 index 0000000..e524f17 --- /dev/null +++ b/src/podcasts/refresh.rs @@ -0,0 +1,498 @@ +//! Feed refresh ingestion and the local-only fetch state. + +use std::collections::BTreeSet; + +use anyhow::Result; +use music_dht::device_sync::{SyncOpPayload, SyncOpWire}; +use music_dht::podcast::feed::ParsedFeed; +use music_dht::podcast::{EpisodeMeta, FeedUrl, PodcastId, merge_feed_aliases}; +use sqlx::{PgConnection, Row}; + +use super::apply::{self, EpisodeSource}; +use super::{PodcastStore, PodcastStoreError, episode_search_norm, now_ms}; +use crate::federation::devices; + +/// A fetched and parsed feed document for [`PodcastStore::apply_feed`]. +#[derive(Debug, Clone, Copy)] +pub(crate) struct FeedRefresh<'a> { + /// Podcast the feed belongs to. + pub(crate) podcast_id: &'a PodcastId, + /// The parsed document (`music_dht::podcast::feed::parse_feed_with_base` + /// with `fetched_from` as base). + pub(crate) feed: &'a ParsedFeed, + /// URL the document was finally served from, after redirects. + pub(crate) fetched_from: &'a FeedUrl, + /// Whether every redirect leading to `fetched_from` was permanent + /// (301/308). Temporary redirects do not move the feed. + pub(crate) moved_permanently: bool, +} + +/// Result of [`PodcastStore::apply_feed`]. +#[derive(Debug, Clone, Default, PartialEq)] +pub(crate) struct RefreshOutcome { + /// Episodes seen for the first time. + pub(crate) new_episodes: usize, + /// Known episodes whose metadata changed (title, audio URL, ...). + pub(crate) updated_episodes: usize, + /// Known episodes without metadata changes. + pub(crate) unchanged_episodes: usize, + /// Items without identity or with invalid metadata. + pub(crate) skipped_items: usize, + /// Feed URLs learned from a permanent redirect or `itunes:new-feed-url`. + pub(crate) learned_aliases: Vec, + /// URL the podcast is fetched from after this refresh. + pub(crate) feed_url: Option, + /// `podcast_feed_aliases_added`, only when the learned URLs changed the + /// stored alias set (already appended to the sync log). + pub(crate) ops: Vec, +} + +/// Local-only fetch state of a podcast. Never part of sync output. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub(crate) struct FetchState { + /// HTTP `ETag` validator of the last successful fetch. + pub(crate) etag: Option, + /// HTTP `Last-Modified` validator of the last successful fetch. + pub(crate) last_modified: Option, + /// Last fetch attempt, unix milliseconds. + pub(crate) checked_at_ms: Option, + /// Last successful fetch (including "not modified"). + pub(crate) last_success_at_ms: Option, + /// When the scheduler should check again (`None`: due now). + pub(crate) next_check_at_ms: Option, + /// Error of the last attempt; `None` after a success. + pub(crate) last_error: Option, + /// Failed attempts since the last success. + pub(crate) consecutive_failures: u32, + /// Image URL the cached cover was downloaded from. + pub(crate) cover_url: Option, + /// Local path of the cached cover. + pub(crate) cover_path: Option, +} + +async fn podcast_fetch_url( + conn: &mut PgConnection, + user_id: i64, + podcast_id: &PodcastId, +) -> Result> { + let row = sqlx::query( + "SELECT feed_url FROM furumusic__podcast WHERE user_id = $1 AND podcast_id = $2", + ) + .bind(user_id) + .bind(podcast_id.as_str()) + .fetch_optional(&mut *conn) + .await?; + Ok(row + .as_ref() + .map(|row| super::feed_url_col(row, "feed_url")) + .transpose()?) +} + +async fn fetch_state_in( + conn: &mut PgConnection, + user_id: i64, + podcast_id: &PodcastId, + lock: bool, +) -> Result { + let row = sqlx::query(&format!( + "SELECT etag, last_modified, checked_at_ms, last_success_at_ms, next_check_at_ms, + last_error, consecutive_failures, cover_url, cover_path + FROM furumusic__podcast_fetch_state WHERE user_id = $1 AND podcast_id = $2{}", + if lock { " FOR UPDATE" } else { "" } + )) + .bind(user_id) + .bind(podcast_id.as_str()) + .fetch_optional(&mut *conn) + .await?; + let Some(row) = row else { + return Ok(FetchState::default()); + }; + Ok(FetchState { + etag: row.try_get("etag")?, + last_modified: row.try_get("last_modified")?, + checked_at_ms: row.try_get("checked_at_ms")?, + last_success_at_ms: row.try_get("last_success_at_ms")?, + next_check_at_ms: row.try_get("next_check_at_ms")?, + last_error: row.try_get("last_error")?, + consecutive_failures: u32::try_from(row.try_get::("consecutive_failures")?) + .unwrap_or(u32::MAX), + cover_url: row.try_get("cover_url")?, + cover_path: row.try_get("cover_path")?, + }) +} + +/// Inserts or updates a feed episode; `None` when it is new, otherwise +/// whether its metadata changed. +async fn upsert_feed_episode( + conn: &mut PgConnection, + user_id: i64, + meta: &EpisodeMeta, + now: i64, +) -> Result> { + let Some(stored) = super::queries::episode_meta_in(conn, user_id, &meta.episode_id).await? + else { + apply::insert_episode(conn, user_id, meta, EpisodeSource::Feed, true, now).await?; + return Ok(None); + }; + let changed = stored != *meta; + sqlx::query( + "UPDATE furumusic__podcast_episode + SET podcast_id = $3, guid = $4, title = $5, description = $6, season = $7, + episode_number = $8, episode_type = $9, published_at_ms = $10, + duration_ms = $11, audio_url = $12, mime = $13, byte_len = $14, + image_url = $15, link = $16, source = 'feed', in_feed = TRUE, + search_norm = $17, + updated_at_ms = CASE WHEN $18 THEN $19 ELSE updated_at_ms END + WHERE user_id = $1 AND episode_id = $2", + ) + .bind(user_id) + .bind(meta.episode_id.as_str()) + .bind(meta.podcast_id.as_str()) + .bind(&meta.guid) + .bind(&meta.title) + .bind(&meta.description) + .bind(meta.season.map(i64::from)) + .bind(meta.episode_number.map(i64::from)) + .bind(apply::episode_type_to_sql(meta.episode_type)) + .bind(meta.published_at_ms) + .bind(meta.duration_ms) + .bind(&meta.audio_url) + .bind(&meta.mime) + .bind(meta.byte_len.and_then(|len| i64::try_from(len).ok())) + .bind(&meta.image_url) + .bind(&meta.link) + .bind(episode_search_norm( + &meta.title, + meta.description.as_deref(), + )) + .bind(changed) + .bind(now) + .execute(&mut *conn) + .await?; + Ok(Some(changed)) +} + +impl PodcastStore { + /// Ingests a fetched feed document for a podcast of the account. + /// + /// * Channel metadata of the catalog row is replaced by the feed's. + /// * Episode ids are resolved with the contract identity rule + /// (`ParsedFeed::resolve_episode_ids`) against every stored episode of + /// the podcast, including episodes learned through sync, so a changed + /// title, audio URL or a GUID-less item keeps its id. Known episodes + /// get the new metadata (including the audio URL); new ones are added; + /// episodes missing from the document are kept (`in_feed = false`). + /// * A permanent redirect target and `itunes:new-feed-url` become feed + /// aliases and the new fetch URL. When they change the stored alias + /// set, one `podcast_feed_aliases_added` operation is appended. + /// * Subscription, share and progress registers and listening sessions + /// are never touched, so a refresh cannot resurrect an unsubscribed + /// podcast or move anyone's position. + /// + /// Record the HTTP outcome separately with + /// [`PodcastStore::record_fetch_success`] and friends. + pub(crate) async fn apply_feed( + &self, + user_id: i64, + refresh: &FeedRefresh<'_>, + ) -> Result { + let podcast_id = refresh.podcast_id; + let feed = refresh.feed; + let now = now_ms(); + let device_id = devices::local_device_id(&self.pool, user_id).await?; + let mut tx = self.begin_write(user_id).await?; + let current_url = match podcast_fetch_url(&mut tx, user_id, podcast_id).await? { + Some(url) => url, + None => { + let Some(register) = + apply::load_subscription_register(&mut tx, user_id, podcast_id).await? + else { + return Err(PodcastStoreError::UnknownPodcast(podcast_id.clone()).into()); + }; + apply::apply_subscription_record(&mut tx, user_id, ®ister, now).await?; + register.podcast.feed_url + } + }; + + let known = apply::known_episodes(&mut tx, user_id, podcast_id).await?; + let ids = feed.resolve_episode_ids(podcast_id, &known); + sqlx::query( + "UPDATE furumusic__podcast_episode SET in_feed = FALSE + WHERE user_id = $1 AND podcast_id = $2", + ) + .bind(user_id) + .bind(podcast_id.as_str()) + .execute(&mut *tx) + .await?; + let mut outcome = RefreshOutcome::default(); + for (item, id) in feed.episodes.iter().zip(ids) { + let Some(id) = id else { + outcome.skipped_items += 1; + continue; + }; + let meta = item.to_meta(podcast_id, id); + if meta.validate().is_err() { + outcome.skipped_items += 1; + continue; + } + match upsert_feed_episode(&mut tx, user_id, &meta, now).await? { + None => outcome.new_episodes += 1, + Some(true) => outcome.updated_episodes += 1, + Some(false) => outcome.unchanged_episodes += 1, + } + } + + let mut fetch_url = current_url.clone(); + let mut learned = BTreeSet::new(); + if refresh.moved_permanently && *refresh.fetched_from != fetch_url { + learned.insert(fetch_url.clone()); + learned.insert(refresh.fetched_from.clone()); + fetch_url = refresh.fetched_from.clone(); + } + if let Some(new_url) = feed + .new_feed_url + .as_deref() + .and_then(|url| FeedUrl::parse(url).ok()) + && new_url != fetch_url + { + learned.insert(fetch_url.clone()); + learned.insert(new_url.clone()); + fetch_url = new_url; + } + let meta = feed.to_podcast_meta(fetch_url.clone()); + sqlx::query( + "UPDATE furumusic__podcast + SET feed_url = $3, title = $4, author = $5, description = $6, link = $7, + image_url = $8, meta_from_feed = TRUE, updated_at_ms = $9 + WHERE user_id = $1 AND podcast_id = $2", + ) + .bind(user_id) + .bind(podcast_id.as_str()) + .bind(meta.feed_url.as_str()) + .bind(&meta.title) + .bind(&meta.author) + .bind(&meta.description) + .bind(&meta.link) + .bind(&meta.image_url) + .bind(now) + .execute(&mut *tx) + .await?; + if !learned.is_empty() { + let learned: Vec = learned.into_iter().collect(); + let current = apply::load_aliases(&mut tx, user_id, podcast_id).await?; + if merge_feed_aliases(podcast_id, ¤t, &learned) != current { + let op = super::actions::emit(&mut tx, user_id, &device_id, |_| { + SyncOpPayload::PodcastFeedAliasesAdded { + podcast_id: podcast_id.clone(), + feed_aliases: learned.clone(), + } + }) + .await?; + outcome.ops.push(op); + } + outcome.learned_aliases = learned; + } + // The card of a shared podcast carries the channel metadata, the + // fetch URL and the episode counters of the feed. + let shared: Option = sqlx::query_scalar( + "SELECT shared FROM furumusic__podcast_share WHERE user_id = $1 AND podcast_id = $2", + ) + .bind(user_id) + .bind(podcast_id.as_str()) + .fetch_optional(&mut *tx) + .await?; + tx.commit().await?; + if shared == Some(true) { + super::discovery::request_reconcile(); + } + outcome.feed_url = Some(fetch_url); + Ok(outcome) + } + + /// Local fetch state of a podcast (defaults when never fetched). + pub(crate) async fn fetch_state( + &self, + user_id: i64, + podcast_id: &PodcastId, + ) -> Result { + let mut conn = self.pool.acquire().await?; + fetch_state_in(&mut conn, user_id, podcast_id, false).await + } + + /// Read-modify-write of the fetch state in one transaction with the + /// account lock and the row locked, so concurrent `record_*` / + /// `set_cover_cache` calls never lose each other's columns or failure + /// increments. + async fn update_fetch_state( + &self, + user_id: i64, + podcast_id: &PodcastId, + update: impl FnOnce(&mut FetchState), + ) -> Result { + let mut tx = self.begin_write(user_id).await?; + if podcast_fetch_url(&mut tx, user_id, podcast_id) + .await? + .is_none() + { + return Err(PodcastStoreError::UnknownPodcast(podcast_id.clone()).into()); + } + let mut state = fetch_state_in(&mut tx, user_id, podcast_id, true).await?; + update(&mut state); + sqlx::query( + "INSERT INTO furumusic__podcast_fetch_state + (user_id, podcast_id, etag, last_modified, checked_at_ms, last_success_at_ms, + next_check_at_ms, last_error, consecutive_failures, cover_url, cover_path) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11) + ON CONFLICT (user_id, podcast_id) DO UPDATE SET + etag = EXCLUDED.etag, last_modified = EXCLUDED.last_modified, + checked_at_ms = EXCLUDED.checked_at_ms, + last_success_at_ms = EXCLUDED.last_success_at_ms, + next_check_at_ms = EXCLUDED.next_check_at_ms, + last_error = EXCLUDED.last_error, + consecutive_failures = EXCLUDED.consecutive_failures, + cover_url = EXCLUDED.cover_url, cover_path = EXCLUDED.cover_path", + ) + .bind(user_id) + .bind(podcast_id.as_str()) + .bind(&state.etag) + .bind(&state.last_modified) + .bind(state.checked_at_ms) + .bind(state.last_success_at_ms) + .bind(state.next_check_at_ms) + .bind(&state.last_error) + .bind(i64::from(state.consecutive_failures)) + .bind(&state.cover_url) + .bind(&state.cover_path) + .execute(&mut *tx) + .await?; + tx.commit().await?; + Ok(state) + } + + /// Records a successful fetch (HTTP 200) with its new validators. + pub(crate) async fn record_fetch_success( + &self, + user_id: i64, + podcast_id: &PodcastId, + now_ms: i64, + etag: Option<&str>, + last_modified: Option<&str>, + next_check_at_ms: Option, + ) -> Result<()> { + self.update_fetch_state(user_id, podcast_id, |state| { + state.etag = etag.map(str::to_string); + state.last_modified = last_modified.map(str::to_string); + state.checked_at_ms = Some(now_ms); + state.last_success_at_ms = Some(now_ms); + state.next_check_at_ms = next_check_at_ms; + state.last_error = None; + state.consecutive_failures = 0; + }) + .await + .map(|_| ()) + } + + /// Records a conditional fetch answered with "304 Not Modified". + pub(crate) async fn record_fetch_not_modified( + &self, + user_id: i64, + podcast_id: &PodcastId, + now_ms: i64, + next_check_at_ms: Option, + ) -> Result<()> { + self.update_fetch_state(user_id, podcast_id, |state| { + state.checked_at_ms = Some(now_ms); + state.last_success_at_ms = Some(now_ms); + state.next_check_at_ms = next_check_at_ms; + state.last_error = None; + state.consecutive_failures = 0; + }) + .await + .map(|_| ()) + } + + /// Records a failed fetch; returns the consecutive failure count. The + /// stored catalog is kept. + pub(crate) async fn record_fetch_failure( + &self, + user_id: i64, + podcast_id: &PodcastId, + now_ms: i64, + error: &str, + next_check_at_ms: Option, + ) -> Result { + self.update_fetch_state(user_id, podcast_id, |state| { + state.checked_at_ms = Some(now_ms); + state.next_check_at_ms = next_check_at_ms; + state.last_error = Some(error.to_string()); + state.consecutive_failures = state.consecutive_failures.saturating_add(1); + }) + .await + .map(|state| state.consecutive_failures) + } + + /// Records the locally cached cover image of a podcast. + // Covers are not cached yet: the browser loads `image_url` from the + // publisher. The column and this setter stay for a later cache. + #[cfg(test)] + pub(crate) async fn set_cover_cache( + &self, + user_id: i64, + podcast_id: &PodcastId, + cover_url: Option<&str>, + cover_path: Option<&str>, + ) -> Result<()> { + self.update_fetch_state(user_id, podcast_id, |state| { + state.cover_url = cover_url.map(str::to_string); + state.cover_path = cover_path.map(str::to_string); + }) + .await + .map(|_| ()) + } + + /// Subscribed podcasts of every account due for a refresh at `now_ms` + /// (never checked or `next_check_at_ms <= now_ms`) with their fetch + /// URL: `(user_id, podcast_id, feed_url)`, at most `limit`. Accounts + /// take turns: first the most overdue podcast of every account, then the + /// second, and so on; within a turn the most overdue first. An account + /// with many due feeds therefore never delays the others, and still uses + /// whatever the run has left over. + pub(crate) async fn podcasts_due_for_refresh( + &self, + now_ms: i64, + limit: usize, + ) -> Result> { + let rows = sqlx::query( + "SELECT user_id, podcast_id, feed_url + FROM (SELECT p.user_id, p.podcast_id, p.feed_url, + COALESCE(f.next_check_at_ms, 0) AS due, + ROW_NUMBER() OVER ( + PARTITION BY p.user_id + ORDER BY COALESCE(f.next_check_at_ms, 0), p.podcast_id + ) AS turn + FROM furumusic__podcast p + JOIN furumusic__podcast_subscription s + ON s.user_id = p.user_id AND s.podcast_id = p.podcast_id + AND s.subscribed + LEFT JOIN furumusic__podcast_fetch_state f + ON f.user_id = p.user_id AND f.podcast_id = p.podcast_id + WHERE f.next_check_at_ms IS NULL OR f.next_check_at_ms <= $1) due + ORDER BY turn, due, user_id, podcast_id + LIMIT $2", + ) + .bind(now_ms) + .bind(i64::try_from(limit).unwrap_or(i64::MAX)) + .fetch_all(&self.pool) + .await?; + Ok(rows + .iter() + .map(|row| { + Ok(( + row.try_get("user_id")?, + super::podcast_id_col(row, "podcast_id")?, + super::feed_url_col(row, "feed_url")?, + )) + }) + .collect::>>()?) + } +} diff --git a/src/podcasts/runner.rs b/src/podcasts/runner.rs new file mode 100644 index 0000000..f02bfd1 --- /dev/null +++ b/src/podcasts/runner.rs @@ -0,0 +1,567 @@ +//! Scheduling of feed refreshes: which feeds are due, one request per feed +//! URL per run, ingestion through [`PodcastStore::apply_feed`] and the local +//! fetch state (validators, next check, backoff). + +use std::collections::BTreeMap; +use std::fmt; +use std::time::Duration; + +use anyhow::Result; +use futures_util::StreamExt; +use music_dht::podcast::{FeedUrl, PodcastId}; + +use super::fetch::{FeedFetch, FeedFetcher, FetchError, FetchedFeed, Validators}; +use super::{FeedRefresh, PodcastStore, PodcastStoreError, RefreshOutcome}; + +/// Regular interval between two checks of a healthy feed. +pub(crate) const REFRESH_INTERVAL_MS: i64 = 60 * 60 * 1000; +/// Spread of the regular interval, so feeds added together drift apart. +const REFRESH_JITTER_MS: i64 = 10 * 60 * 1000; +/// First retry delay after a failure; doubles per consecutive failure. +pub(crate) const BACKOFF_BASE_MS: i64 = 15 * 60 * 1000; +/// Longest delay between two attempts of a failing feed. +pub(crate) const BACKOFF_MAX_MS: i64 = 24 * 60 * 60 * 1000; +/// A manual refresh is refused when the feed was checked this recently. +pub(crate) const MANUAL_MIN_INTERVAL_MS: i64 = 30 * 1000; +/// Podcasts considered per scheduled run. Due podcasts are taken in turns +/// across accounts (see [`PodcastStore::podcasts_due_for_refresh`]), so an +/// account with many (or slow) feeds cannot hold back everyone else's +/// refreshes, but can use a run nobody else needs. +pub(crate) const RUN_LIMIT: usize = 200; +/// Feed requests in flight at once. +pub(crate) const RUN_CONCURRENCY: usize = 4; + +/// Delay before the next attempt after `consecutive_failures` failures in a +/// row (counting the one just recorded): 15 min, 30 min, 1 h, ... capped at a +/// day. A longer server-requested `Retry-After` wins (also capped). +pub(crate) fn failure_backoff_ms(consecutive_failures: u32, retry_after: Option) -> i64 { + let exponent = consecutive_failures.saturating_sub(1).min(16); + let backoff = BACKOFF_BASE_MS + .saturating_mul(1_i64 << exponent) + .min(BACKOFF_MAX_MS); + let requested = retry_after + .map(|delay| i64::try_from(delay.as_millis()).unwrap_or(i64::MAX)) + .unwrap_or(0); + backoff.max(requested).min(BACKOFF_MAX_MS) +} + +/// Next regular check after a success: the interval plus a stable per +/// account and podcast jitter. +pub(crate) fn next_regular_check_ms(now_ms: i64, user_id: i64, podcast_id: &PodcastId) -> i64 { + let key = format!("{user_id}:{podcast_id}"); + let hash = blake3::hash(key.as_bytes()); + let spread = u64::from_le_bytes(hash.as_bytes()[..8].try_into().expect("8 bytes")); + let jitter = i64::try_from(spread % REFRESH_JITTER_MS as u64).unwrap_or(0); + now_ms + REFRESH_INTERVAL_MS + jitter +} + +/// One podcast of one account to refresh. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct RefreshTarget { + pub(crate) user_id: i64, + pub(crate) podcast_id: PodcastId, +} + +/// Result of refreshing one [`RefreshTarget`]. +#[derive(Debug, Clone, PartialEq)] +pub(crate) enum TargetResult { + /// A new document was ingested. + Updated(RefreshOutcome), + /// `304 Not Modified`. + NotModified, + /// The fetch or the ingestion failed; the catalog is unchanged. + Failed { + /// What the user sees (stored as the last error). + error: String, + /// Details for the server log (no URLs). + detail: String, + consecutive_failures: u32, + next_check_at_ms: i64, + }, + /// Nothing to do any more (unsubscribed or removed meanwhile). + Skipped(String), +} + +/// Counters of a scheduled run. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub(crate) struct RunSummary { + /// Account podcasts that were due. + pub(crate) targets: usize, + /// Distinct feed URLs requested. + pub(crate) feeds: usize, + pub(crate) updated: usize, + pub(crate) not_modified: usize, + pub(crate) failed: usize, + pub(crate) skipped: usize, + pub(crate) new_episodes: usize, + pub(crate) updated_episodes: usize, + /// `(podcast_id, details)` of failed targets, for the job log (no + /// account ids or URLs: feed URLs often carry access tokens). + pub(crate) failures: Vec<(PodcastId, String)>, +} + +impl RunSummary { + fn add(&mut self, target: &RefreshTarget, result: &TargetResult) { + match result { + TargetResult::Updated(outcome) => { + self.updated += 1; + self.new_episodes += outcome.new_episodes; + self.updated_episodes += outcome.updated_episodes; + } + TargetResult::NotModified => self.not_modified += 1, + TargetResult::Failed { detail, .. } => { + self.failed += 1; + self.failures + .push((target.podcast_id.clone(), detail.clone())); + } + TargetResult::Skipped(_) => self.skipped += 1, + } + } +} + +impl fmt::Display for RunSummary { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!( + f, + "{} due podcasts, {} feeds requested: {} updated ({} new, {} changed episodes), \ + {} not modified, {} failed, {} skipped", + self.targets, + self.feeds, + self.updated, + self.new_episodes, + self.updated_episodes, + self.not_modified, + self.failed, + self.skipped + ) + } +} + +/// Due podcasts grouped by feed URL: accounts subscribed to the same URL +/// share one request. +pub(crate) type RefreshGroups = BTreeMap>; + +/// Groups the rows of [`PodcastStore::podcasts_due_for_refresh`]. +pub(crate) fn group_targets(due: Vec<(i64, PodcastId, FeedUrl)>) -> RefreshGroups { + let mut groups = RefreshGroups::new(); + for (user_id, podcast_id, feed_url) in due { + groups.entry(feed_url).or_default().push(RefreshTarget { + user_id, + podcast_id, + }); + } + groups +} + +/// Refreshes podcasts due at `now_ms` (at most `limit`, accounts in +/// turns), with at most +/// `concurrency` feed requests in flight. Accounts subscribed to the same +/// feed URL share one request. A failing feed never stops the others. +pub(crate) async fn refresh_due( + store: &PodcastStore, + fetcher: &FeedFetcher, + now_ms: i64, + limit: usize, + concurrency: usize, +) -> Result { + let due = store.podcasts_due_for_refresh(now_ms, limit).await?; + refresh_groups(store, fetcher, group_targets(due), now_ms, concurrency).await +} + +/// Refreshes `groups`, one request per feed URL, at most `concurrency` at +/// a time. +pub(crate) async fn refresh_groups( + store: &PodcastStore, + fetcher: &FeedFetcher, + groups: RefreshGroups, + now_ms: i64, + concurrency: usize, +) -> Result { + let mut summary = RunSummary { + targets: groups.values().map(Vec::len).sum(), + feeds: groups.len(), + ..RunSummary::default() + }; + let mut results = futures_util::stream::iter(groups) + .map(|(feed_url, targets)| async move { + refresh_group(store, fetcher, &feed_url, targets, now_ms).await + }) + .buffer_unordered(concurrency.max(1)); + while let Some(group) = results.next().await { + for (target, result) in group? { + summary.add(&target, &result); + } + } + Ok(summary) +} + +/// Fetches `feed_url` once for all `targets` and records the outcome for +/// each. The request is conditional only when every target holds the same +/// validators (the usual case: one account). +pub(crate) async fn refresh_group( + store: &PodcastStore, + fetcher: &FeedFetcher, + feed_url: &FeedUrl, + targets: Vec, + now_ms: i64, +) -> Result> { + let mut states = Vec::with_capacity(targets.len()); + for target in &targets { + states.push( + store + .fetch_state(target.user_id, &target.podcast_id) + .await?, + ); + } + let validators = states + .iter() + .map(|state| Validators { + etag: state.etag.clone(), + last_modified: state.last_modified.clone(), + }) + .reduce(|a, b| if a == b { a } else { Validators::default() }) + .unwrap_or_default(); + let fetched = fetcher.fetch(feed_url, &validators).await; + let mut out = Vec::with_capacity(targets.len()); + for (target, state) in targets.into_iter().zip(states) { + let result = + record_result(store, &target, &fetched, state.consecutive_failures, now_ms).await?; + out.push((target, result)); + } + Ok(out) +} + +/// Ingests (or records the failure of) one fetch for one target. +async fn record_result( + store: &PodcastStore, + target: &RefreshTarget, + fetched: &Result, + previous_failures: u32, + now_ms: i64, +) -> Result { + let (user_id, podcast_id) = (target.user_id, &target.podcast_id); + let failure = |error: String, detail: String, retry_after: Option| { + let consecutive_failures = previous_failures.saturating_add(1); + let next_check_at_ms = now_ms + failure_backoff_ms(consecutive_failures, retry_after); + (error, detail, consecutive_failures, next_check_at_ms) + }; + let (error, detail, consecutive_failures, next_check_at_ms) = match fetched { + Ok(FeedFetch::NotModified) => { + let next = next_regular_check_ms(now_ms, user_id, podcast_id); + return match store + .record_fetch_not_modified(user_id, podcast_id, now_ms, Some(next)) + .await + { + Ok(()) => Ok(TargetResult::NotModified), + Err(error) => skipped_or(error), + }; + } + Ok(FeedFetch::Fetched(document)) => { + match ingest(store, target, document, now_ms).await { + Ok(outcome) => return Ok(TargetResult::Updated(outcome)), + Err(error) if is_gone(&error) => return skipped_or(error), + // A store failure is recorded like a fetch failure, so the + // feed backs off instead of being retried every run. + Err(error) => failure( + "the feed could not be stored".to_string(), + format!("storing the feed failed: {error:#}"), + None, + ), + } + } + Err(error) => failure( + error.public_message(), + error.message.clone(), + error.retry_after, + ), + }; + match store + .record_fetch_failure(user_id, podcast_id, now_ms, &error, Some(next_check_at_ms)) + .await + { + Ok(_) => Ok(TargetResult::Failed { + error, + detail, + consecutive_failures, + next_check_at_ms, + }), + Err(error) => skipped_or(error), + } +} + +async fn ingest( + store: &PodcastStore, + target: &RefreshTarget, + document: &FetchedFeed, + now_ms: i64, +) -> Result { + let outcome = store + .apply_feed( + target.user_id, + &FeedRefresh { + podcast_id: &target.podcast_id, + feed: &document.feed, + fetched_from: &document.fetched_from, + moved_permanently: document.moved_permanently, + }, + ) + .await?; + store + .record_fetch_success( + target.user_id, + &target.podcast_id, + now_ms, + document.validators.etag.as_deref(), + document.validators.last_modified.as_deref(), + Some(next_regular_check_ms( + now_ms, + target.user_id, + &target.podcast_id, + )), + ) + .await?; + Ok(outcome) +} + +fn is_gone(error: &anyhow::Error) -> bool { + matches!( + error.downcast_ref::(), + Some(PodcastStoreError::UnknownPodcast(_)) + ) +} + +/// A podcast removed from the account meanwhile is skipped; any other +/// store error is returned. +fn skipped_or(error: anyhow::Error) -> Result { + if is_gone(&error) { + Ok(TargetResult::Skipped(error.to_string())) + } else { + Err(error) + } +} + +/// Why a manual refresh was not started. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) enum ManualRefreshError { + /// The account does not know the podcast. + UnknownPodcast, + /// The account is not subscribed (unsubscribing stops monitoring). + NotSubscribed, + /// Checked less than [`MANUAL_MIN_INTERVAL_MS`] ago. + TooSoon { retry_after_ms: i64 }, +} + +impl fmt::Display for ManualRefreshError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::UnknownPodcast => f.write_str("unknown podcast"), + Self::NotSubscribed => f.write_str("not subscribed to this podcast"), + Self::TooSoon { retry_after_ms } => write!( + f, + "the feed was checked moments ago; retry in {} s", + (retry_after_ms + 999) / 1000 + ), + } + } +} + +impl std::error::Error for ManualRefreshError {} + +/// Refreshes one subscribed podcast of one account now, ignoring the +/// failure backoff but not more often than every +/// [`MANUAL_MIN_INTERVAL_MS`]. Returns `Err` with a +/// [`ManualRefreshError`] when the refresh was not attempted. +pub(crate) async fn refresh_now( + store: &PodcastStore, + fetcher: &FeedFetcher, + user_id: i64, + podcast_id: &PodcastId, + now_ms: i64, +) -> Result { + let feed_url = match store.podcast(user_id, podcast_id).await? { + Some(summary) if summary.subscribed => summary.meta.feed_url, + Some(_) => return Err(ManualRefreshError::NotSubscribed.into()), + None => match store.subscription(user_id, podcast_id).await? { + Some(register) if register.subscribed => register.podcast.feed_url, + Some(_) => return Err(ManualRefreshError::NotSubscribed.into()), + None => return Err(ManualRefreshError::UnknownPodcast.into()), + }, + }; + if let Some(checked) = store.fetch_state(user_id, podcast_id).await?.checked_at_ms { + let elapsed = now_ms - checked; + if (0..MANUAL_MIN_INTERVAL_MS).contains(&elapsed) { + return Err(ManualRefreshError::TooSoon { + retry_after_ms: MANUAL_MIN_INTERVAL_MS - elapsed, + } + .into()); + } + } + let target = RefreshTarget { + user_id, + podcast_id: podcast_id.clone(), + }; + let mut results = refresh_group(store, fetcher, &feed_url, vec![target], now_ms).await?; + Ok(results.pop().map(|(_, result)| result).expect("one target")) +} + +/// Result of [`subscribe_url`]. +#[derive(Debug, Clone, PartialEq)] +pub(crate) struct UrlSubscription { + /// Podcast the URL resolved to (an existing one for a known URL or alias). + pub(crate) podcast_id: PodcastId, + /// The account was already subscribed; no subscription was written. + pub(crate) already_subscribed: bool, + /// Ingestion of the document fetched for the subscription. + pub(crate) refresh: RefreshOutcome, +} + +/// Why [`subscribe_url`] did not subscribe. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) enum SubscribeUrlError { + /// The feed could not be fetched or parsed; nothing was stored. + Fetch(FetchError), +} + +impl fmt::Display for SubscribeUrlError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::Fetch(error) => { + write!( + f, + "the feed could not be loaded: {}", + error.public_message() + ) + } + } + } +} + +impl std::error::Error for SubscribeUrlError {} + +/// Subscribes an account to the feed at `url` (the "add RSS" action). +/// +/// The feed is fetched first ([`fetch_to_subscribe`]): an address that is +/// not public, a document that is not a feed or a failed request +/// subscribes nothing ([`SubscribeUrlError`] inside the error). Then +/// [`subscribe_fetched`] writes the subscription. +pub(crate) async fn subscribe_url( + store: &PodcastStore, + fetcher: &FeedFetcher, + user_id: i64, + url: &FeedUrl, + now_ms: i64, +) -> Result { + let document = fetch_to_subscribe(fetcher, url).await?; + subscribe_fetched(store, user_id, url, &document, now_ms).await +} + +/// Fetches the feed at `url` unconditionally for a new subscription or a +/// preview; a failure is a [`SubscribeUrlError`]. +pub(crate) async fn fetch_to_subscribe( + fetcher: &FeedFetcher, + url: &FeedUrl, +) -> Result> { + match fetcher.fetch(url, &Validators::default()).await { + Ok(FeedFetch::Fetched(document)) => Ok(document), + Ok(FeedFetch::NotModified) => { + // Unconditional requests are never answered with 304 by a + // conforming server; refuse instead of subscribing blindly. + Err(SubscribeUrlError::Fetch(FetchError { + kind: super::fetch::FetchErrorKind::Status(304), + message: "unexpected 304 Not Modified".to_string(), + retry_after: None, + }) + .into()) + } + Err(error) => Err(SubscribeUrlError::Fetch(error).into()), + } +} + +/// The URLs a document fetched from `requested` shows for the same +/// podcast: `requested`, the end of a chain of permanent redirects and the +/// feed's own `itunes:new-feed-url` (the same URLs a refresh learns as +/// aliases and moves to), without duplicates. +/// +/// Only the first two are where the document came from. The +/// `itunes:new-feed-url` is a claim of the document: following the feed +/// moves to it anyway (its first refresh fetches from there), so it may +/// name the account's podcast for the feed, but the document is never that +/// podcast's content ([`subscribe_fetched`] does not ingest it there). +pub(crate) fn document_urls(requested: &FeedUrl, document: &FetchedFeed) -> Vec { + let mut urls = vec![requested.clone()]; + if document.moved_permanently { + urls.push(document.fetched_from.clone()); + } + if let Some(new_url) = document + .feed + .new_feed_url + .as_deref() + .and_then(|url| FeedUrl::parse(url).ok()) + { + urls.push(new_url); + } + let mut seen = std::collections::BTreeSet::new(); + urls.retain(|url| seen.insert(url.clone())); + urls +} + +/// Whether a document fetched from `requested` is the content of a podcast +/// fetched from `feed_url`: the request went to `feed_url` itself (what a +/// refresh does), or a permanent redirect chain ended there. +fn fetched_from_feed(requested: &FeedUrl, document: &FetchedFeed, feed_url: &FeedUrl) -> bool { + requested == feed_url || (document.moved_permanently && document.fetched_from == *feed_url) +} + +/// Writes the subscription for a document fetched from `url`. +/// +/// * The podcast is the account's podcast for the feed +/// ([`super::search::known_podcast`] over [`document_urls`]: a subscribed +/// one first), so a feed the account already follows under another +/// address is not subscribed a second time; otherwise a new podcast with +/// the id of `url`. +/// * The document is ingested like a scheduled refresh (which also stores +/// the validators and the next check) only for a new podcast, or when it +/// came from the existing podcast's own fetch URL +/// ([`fetched_from_feed`]). Otherwise (the match came through another +/// alias or the feed's `itunes:new-feed-url` claim) it may be a stale +/// copy or another feed altogether: the existing podcast is subscribed +/// again with its stored channel metadata, like "subscribe again", and +/// the refresh job fetches its own URL. An active subscription is never +/// written. +/// +/// [`PodcastStore::subscribe`] reuses the existing podcast with its +/// progress and history. +pub(crate) async fn subscribe_fetched( + store: &PodcastStore, + user_id: i64, + url: &FeedUrl, + document: &FetchedFeed, + now_ms: i64, +) -> Result { + let requested_id = store.resolve_podcast_id(user_id, url).await?; + let urls = document_urls(url, document); + let existing = super::search::known_podcast(store, user_id, &requested_id, &urls).await?; + let meta = match &existing { + Some(known) if !fetched_from_feed(url, document, &known.meta.feed_url) => { + let subscription = store.subscribe(user_id, &known.meta).await?; + return Ok(UrlSubscription { + podcast_id: subscription.podcast_id, + already_subscribed: subscription.already_subscribed, + refresh: RefreshOutcome::default(), + }); + } + Some(known) => document.feed.to_podcast_meta(known.meta.feed_url.clone()), + None => document.feed.to_podcast_meta(url.clone()), + }; + let subscription = store.subscribe(user_id, &meta).await?; + let target = RefreshTarget { + user_id, + podcast_id: subscription.podcast_id.clone(), + }; + let refresh = ingest(store, &target, document, now_ms).await?; + Ok(UrlSubscription { + podcast_id: subscription.podcast_id, + already_subscribed: subscription.already_subscribed, + refresh, + }) +} diff --git a/src/podcasts/schema.rs b/src/podcasts/schema.rs new file mode 100644 index 0000000..df97c2e --- /dev/null +++ b/src/podcasts/schema.rs @@ -0,0 +1,366 @@ +//! PostgreSQL schema of the podcast store and the import of the relay state. +//! +//! The migrations themselves are declared in `crate::music::db_migrations` +//! (`m_0056_create_podcast_store`, `m_0057_import_podcast_relay`, +//! `m_0058_create_podcast_share_confirmation`), next to the rest of the +//! `furumusic` chain; this module holds their SQL so the store and its tests +//! share one definition. +//! +//! The tables mirror the SQLite reference store of `furumi-library` +//! (`PODCASTS.md`, "SQLite reference store") with a `user_id` column in +//! every key: each web account is one virtual device with its own podcast +//! catalog, registers and fetch state. + +/// Tables, the derived season view and indexes (`m_0056`). +pub(crate) const CREATE_PODCAST_STORE: &[&str] = &[ + // Local channel metadata. `feed_url` is the URL this account fetches; + // `meta_from_feed` is set once the account refreshed the feed itself + // (until then the winning subscription register provides the metadata). + "CREATE TABLE IF NOT EXISTS furumusic__podcast ( + user_id BIGINT NOT NULL, + podcast_id TEXT NOT NULL, + feed_url TEXT NOT NULL, + title TEXT NOT NULL, + author TEXT, + description TEXT, + link TEXT, + image_url TEXT, + meta_from_feed BOOLEAN NOT NULL DEFAULT FALSE, + created_at_ms BIGINT NOT NULL, + updated_at_ms BIGINT NOT NULL, + PRIMARY KEY (user_id, podcast_id) + )", + // Bounded grow-only alias set (`merge_feed_aliases`), synced inside + // subscription records and `podcast_feed_aliases_added`. + "CREATE TABLE IF NOT EXISTS furumusic__podcast_feed_alias ( + user_id BIGINT NOT NULL, + podcast_id TEXT NOT NULL, + feed_url TEXT NOT NULL, + PRIMARY KEY (user_id, podcast_id, feed_url) + )", + "CREATE INDEX IF NOT EXISTS idx_podcast_feed_alias_url + ON furumusic__podcast_feed_alias (user_id, feed_url)", + // Episode catalog (`EpisodeMeta`). `source` is `feed` once this account + // parsed the episode from RSS, `sync` while it is only known from a + // `PodcastEpisodeRef`. `search_norm` caches `normalize_name(title)` and + // `normalize_name(description)` separated by a newline (never produced + // by the normalization), NULL until computed. + "CREATE TABLE IF NOT EXISTS furumusic__podcast_episode ( + user_id BIGINT NOT NULL, + episode_id TEXT NOT NULL, + podcast_id TEXT NOT NULL, + guid TEXT, + title TEXT NOT NULL, + description TEXT, + season BIGINT, + episode_number BIGINT, + episode_type TEXT NOT NULL DEFAULT 'full', + published_at_ms BIGINT, + duration_ms BIGINT, + audio_url TEXT NOT NULL, + mime TEXT, + byte_len BIGINT, + image_url TEXT, + link TEXT, + source TEXT NOT NULL CHECK (source IN ('feed', 'sync')), + in_feed BOOLEAN NOT NULL DEFAULT FALSE, + search_norm TEXT, + first_seen_at_ms BIGINT NOT NULL, + updated_at_ms BIGINT NOT NULL, + PRIMARY KEY (user_id, episode_id), + FOREIGN KEY (user_id, podcast_id) + REFERENCES furumusic__podcast (user_id, podcast_id) ON DELETE CASCADE + )", + "CREATE INDEX IF NOT EXISTS idx_podcast_episode_date + ON furumusic__podcast_episode (user_id, podcast_id, published_at_ms DESC)", + "CREATE INDEX IF NOT EXISTS idx_podcast_episode_season + ON furumusic__podcast_episode (user_id, podcast_id, season, episode_number)", + // Seasons are derived from the episodes; a view cannot go stale. + // `season` NULL groups the episodes without `itunes:season`. + "CREATE OR REPLACE VIEW furumusic__podcast_season AS + SELECT user_id, podcast_id, season, + COUNT(*) AS episode_count, + MIN(published_at_ms) AS first_published_at_ms, + MAX(published_at_ms) AS last_published_at_ms + FROM furumusic__podcast_episode + GROUP BY user_id, podcast_id, season", + // Synced registers. No foreign keys to the catalog: operations for + // podcasts or episodes this account has not fetched are kept as they + // arrive, in any order. + "CREATE TABLE IF NOT EXISTS furumusic__podcast_subscription ( + user_id BIGINT NOT NULL, + podcast_id TEXT NOT NULL, + subscribed BOOLEAN NOT NULL, + podcast_json JSONB NOT NULL, + generation BIGINT NOT NULL CHECK (generation >= 0), + hlc_ms BIGINT NOT NULL, + op_id TEXT NOT NULL, + PRIMARY KEY (user_id, podcast_id) + )", + "CREATE TABLE IF NOT EXISTS furumusic__podcast_share ( + user_id BIGINT NOT NULL, + podcast_id TEXT NOT NULL, + shared BOOLEAN NOT NULL, + generation BIGINT NOT NULL CHECK (generation >= 0), + hlc_ms BIGINT NOT NULL, + op_id TEXT NOT NULL, + PRIMARY KEY (user_id, podcast_id) + )", + "CREATE TABLE IF NOT EXISTS furumusic__podcast_progress ( + user_id BIGINT NOT NULL, + episode_id TEXT NOT NULL, + podcast_id TEXT NOT NULL, + position_ms BIGINT NOT NULL, + status TEXT NOT NULL + CHECK (status IN ('not_started', 'in_progress', 'completed')), + generation BIGINT NOT NULL CHECK (generation >= 0), + session_id TEXT NOT NULL, + hlc_ms BIGINT NOT NULL, + device_id TEXT NOT NULL, + op_id TEXT NOT NULL, + PRIMARY KEY (user_id, episode_id) + )", + "CREATE INDEX IF NOT EXISTS idx_podcast_progress_podcast + ON furumusic__podcast_progress (user_id, podcast_id)", + // Grow-only set of listening sessions. + "CREATE TABLE IF NOT EXISTS furumusic__podcast_listen_session ( + user_id BIGINT NOT NULL, + session_id TEXT NOT NULL, + episode_id TEXT NOT NULL, + podcast_id TEXT NOT NULL, + device_id TEXT NOT NULL, + started_at_ms BIGINT NOT NULL, + ended_at_ms BIGINT NOT NULL, + start_position_ms BIGINT NOT NULL, + end_position_ms BIGINT NOT NULL, + listened_ms BIGINT NOT NULL, + completed BOOLEAN NOT NULL, + PRIMARY KEY (user_id, session_id) + )", + "CREATE INDEX IF NOT EXISTS idx_podcast_listen_session_episode + ON furumusic__podcast_listen_session (user_id, episode_id)", + "CREATE INDEX IF NOT EXISTS idx_podcast_listen_session_end + ON furumusic__podcast_listen_session (user_id, ended_at_ms DESC)", + // Local-only fetch state: never part of sync output. + "CREATE TABLE IF NOT EXISTS furumusic__podcast_fetch_state ( + user_id BIGINT NOT NULL, + podcast_id TEXT NOT NULL, + etag TEXT, + last_modified TEXT, + checked_at_ms BIGINT, + last_success_at_ms BIGINT, + next_check_at_ms BIGINT, + last_error TEXT, + consecutive_failures BIGINT NOT NULL DEFAULT 0, + cover_url TEXT, + cover_path TEXT, + PRIMARY KEY (user_id, podcast_id), + FOREIGN KEY (user_id, podcast_id) + REFERENCES furumusic__podcast (user_id, podcast_id) ON DELETE CASCADE + )", + "CREATE INDEX IF NOT EXISTS idx_podcast_fetch_state_due + ON furumusic__podcast_fetch_state (next_check_at_ms)", +]; + +/// Backwards operation of `m_0056`. +pub(crate) const DROP_PODCAST_STORE: &[&str] = &[ + "DROP VIEW IF EXISTS furumusic__podcast_season", + "DROP TABLE IF EXISTS furumusic__podcast_fetch_state", + "DROP TABLE IF EXISTS furumusic__podcast_listen_session", + "DROP TABLE IF EXISTS furumusic__podcast_progress", + "DROP TABLE IF EXISTS furumusic__podcast_share", + "DROP TABLE IF EXISTS furumusic__podcast_subscription", + "DROP TABLE IF EXISTS furumusic__podcast_episode", + "DROP TABLE IF EXISTS furumusic__podcast_feed_alias", + "DROP TABLE IF EXISTS furumusic__podcast", +]; + +/// Local privacy confirmations of shared podcasts (`m_0058`). +/// +/// `PodcastShareState::privacy_confirmed` is client-held state (frid +/// `PODCASTS.md`, "Publication"): it is never synced and never part of a +/// snapshot. A row records that the account confirmed publishing exactly +/// `feed_url` / `id_feed_url` for exactly the share decision `share_op_id` +/// despite `PodcastCardSpec::privacy_warnings`. A new share decision (on any +/// device) or a feed move to other URLs makes the row stale, so the user is +/// asked again (see `crate::podcasts::discovery`). +pub(crate) const CREATE_SHARE_CONFIRMATION: &[&str] = &[ + "CREATE TABLE IF NOT EXISTS furumusic__podcast_share_confirmation ( + user_id BIGINT NOT NULL, + podcast_id TEXT NOT NULL, + share_op_id TEXT NOT NULL, + feed_url TEXT NOT NULL, + id_feed_url TEXT, + confirmed_at_ms BIGINT NOT NULL, + PRIMARY KEY (user_id, podcast_id) + )", +]; + +/// Backwards operation of [`CREATE_SHARE_CONFIRMATION`]. +pub(crate) const DROP_SHARE_CONFIRMATION: &[&str] = + &["DROP TABLE IF EXISTS furumusic__podcast_share_confirmation"]; + +/// Placeholder in [`IMPORT_PODCAST_RELAY`] for the account filter: the +/// migration imports every account (`TRUE`), tests narrow it to one. +pub(crate) const USER_FILTER: &str = "{user_filter}"; + +/// Largest generation the `BIGINT` columns hold; larger records are skipped +/// like the store rejects them. +const GENERATION_FITS: &str = "(record_json->>'generation')::numeric <= 9223372036854775807"; + +/// Import of the opaque relay state kept by 0.13.0 (`m_0057`). +/// +/// `furumusic__fed_podcast_relay` holds the contract registers as JSON, +/// already validated and merged with the `music_dht::podcast` rules when they +/// were received, so they are copied as they are. The catalog rows the store +/// would have created from the episode references of received operations +/// (`PodcastEpisodeRef`, `source = 'sync'`) are rebuilt from the operation +/// log. Every statement is idempotent (`ON CONFLICT DO NOTHING`); the relay +/// table is kept (see `docs/administration.md`). +pub(crate) const IMPORT_PODCAST_RELAY: &[&str] = &[ + // Catalog rows from the subscription registers. + "INSERT INTO furumusic__podcast + (user_id, podcast_id, feed_url, title, author, description, link, image_url, + meta_from_feed, created_at_ms, updated_at_ms) + SELECT user_id, record_key, + record_json->'podcast'->>'feed_url', record_json->'podcast'->>'title', + record_json->'podcast'->>'author', record_json->'podcast'->>'description', + record_json->'podcast'->>'link', record_json->'podcast'->>'image_url', + FALSE, updated_at_ms, updated_at_ms + FROM furumusic__fed_podcast_relay + WHERE section = 'subscription' AND {user_filter} + ON CONFLICT DO NOTHING", + "INSERT INTO furumusic__podcast_subscription + (user_id, podcast_id, subscribed, podcast_json, generation, hlc_ms, op_id) + SELECT user_id, record_key, (record_json->>'subscribed')::boolean, + record_json->'podcast', (record_json->>'generation')::bigint, + (record_json->>'hlc_ms')::bigint, record_json->>'op_id' + FROM furumusic__fed_podcast_relay + WHERE section = 'subscription' AND {generation_fits} AND {user_filter} + ON CONFLICT DO NOTHING", + // The relayed subscription already carries the bounded, merged alias set. + "INSERT INTO furumusic__podcast_feed_alias (user_id, podcast_id, feed_url) + SELECT user_id, record_key, jsonb_array_elements_text(record_json->'feed_aliases') + FROM furumusic__fed_podcast_relay + WHERE section = 'subscription' AND {user_filter} + AND jsonb_typeof(record_json->'feed_aliases') = 'array' + ON CONFLICT DO NOTHING", + // Aliases received before any subscription register of their podcast. + "INSERT INTO furumusic__podcast_feed_alias (user_id, podcast_id, feed_url) + SELECT user_id, record_key, jsonb_array_elements_text(record_json) + FROM furumusic__fed_podcast_relay + WHERE section = 'aliases' AND {user_filter} AND jsonb_typeof(record_json) = 'array' + ON CONFLICT DO NOTHING", + "INSERT INTO furumusic__podcast_share + (user_id, podcast_id, shared, generation, hlc_ms, op_id) + SELECT user_id, record_key, (record_json->>'shared')::boolean, + (record_json->>'generation')::bigint, (record_json->>'hlc_ms')::bigint, + record_json->>'op_id' + FROM furumusic__fed_podcast_relay + WHERE section = 'share' AND {generation_fits} AND {user_filter} + ON CONFLICT DO NOTHING", + "INSERT INTO furumusic__podcast_progress + (user_id, episode_id, podcast_id, position_ms, status, generation, session_id, + hlc_ms, device_id, op_id) + SELECT user_id, record_key, record_json->>'podcast_id', + (record_json->>'position_ms')::bigint, record_json->>'status', + (record_json->>'generation')::bigint, record_json->>'session_id', + (record_json->>'hlc_ms')::bigint, record_json->>'device_id', + record_json->>'op_id' + FROM furumusic__fed_podcast_relay + WHERE section = 'progress' AND {generation_fits} AND {user_filter} + ON CONFLICT DO NOTHING", + "INSERT INTO furumusic__podcast_listen_session + (user_id, session_id, episode_id, podcast_id, device_id, started_at_ms, + ended_at_ms, start_position_ms, end_position_ms, listened_ms, completed) + SELECT user_id, record_key, record_json->>'episode_id', record_json->>'podcast_id', + record_json->>'device_id', (record_json->>'started_at_ms')::bigint, + (record_json->>'ended_at_ms')::bigint, + (record_json->>'start_position_ms')::bigint, + (record_json->>'end_position_ms')::bigint, + (record_json->>'listened_ms')::bigint, (record_json->>'completed')::boolean + FROM furumusic__fed_podcast_relay + WHERE section = 'session' AND {user_filter} + ON CONFLICT DO NOTHING", + // Catalog rows for podcasts only known from episode references. + "INSERT INTO furumusic__podcast + (user_id, podcast_id, feed_url, title, meta_from_feed, created_at_ms, updated_at_ms) + SELECT DISTINCT ON (user_id, payload_json->'episode'->>'podcast_id') + user_id, payload_json->'episode'->>'podcast_id', + payload_json->'episode'->>'feed_url', payload_json->'episode'->>'podcast_title', + FALSE, received_at_ms, received_at_ms + FROM furumusic__fed_sync_ops + WHERE kind IN ('podcast_progress_set', 'podcast_listen_recorded') + AND jsonb_typeof(payload_json->'episode') = 'object' AND {user_filter} + ORDER BY user_id, payload_json->'episode'->>'podcast_id', hlc_ms DESC, op_id DESC + ON CONFLICT DO NOTHING", + // Episodes only known from references (the newest reference wins, like + // the first one would have for the store: feed metadata replaces it on + // the first refresh anyway). + "INSERT INTO furumusic__podcast_episode + (user_id, episode_id, podcast_id, title, season, episode_number, episode_type, + published_at_ms, duration_ms, audio_url, mime, image_url, source, in_feed, + first_seen_at_ms, updated_at_ms) + SELECT DISTINCT ON (user_id, payload_json->'episode'->>'episode_id') + user_id, payload_json->'episode'->>'episode_id', + payload_json->'episode'->>'podcast_id', payload_json->'episode'->>'episode_title', + (payload_json->'episode'->>'season')::bigint, + (payload_json->'episode'->>'episode_number')::bigint, 'full', + (payload_json->'episode'->>'published_at_ms')::bigint, + (payload_json->'episode'->>'duration_ms')::bigint, + payload_json->'episode'->>'audio_url', payload_json->'episode'->>'mime', + payload_json->'episode'->>'image_url', 'sync', FALSE, + received_at_ms, received_at_ms + FROM furumusic__fed_sync_ops + WHERE kind IN ('podcast_progress_set', 'podcast_listen_recorded') + AND jsonb_typeof(payload_json->'episode') = 'object' AND {user_filter} + ORDER BY user_id, payload_json->'episode'->>'episode_id', hlc_ms DESC, op_id DESC + ON CONFLICT DO NOTHING", +]; + +/// The import statements with `filter` as the account condition. +pub(crate) fn import_statements(filter: &str) -> Vec { + IMPORT_PODCAST_RELAY + .iter() + .map(|sql| { + sql.replace("{generation_fits}", GENERATION_FITS) + .replace(USER_FILTER, filter) + }) + .collect() +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn import_statements_fill_every_placeholder() { + for sql in import_statements("user_id = 7") { + assert!(!sql.contains('{'), "{sql}"); + assert!(sql.contains("user_id = 7"), "{sql}"); + assert!(sql.contains("ON CONFLICT DO NOTHING"), "{sql}"); + } + } + + #[test] + fn every_store_table_is_dropped_backwards() { + let drops: Vec<&str> = DROP_PODCAST_STORE + .iter() + .chain(DROP_SHARE_CONFIRMATION) + .copied() + .collect(); + for sql in CREATE_PODCAST_STORE.iter().chain(CREATE_SHARE_CONFIRMATION) { + let Some(rest) = sql + .strip_prefix("CREATE TABLE IF NOT EXISTS ") + .or_else(|| sql.strip_prefix("CREATE OR REPLACE VIEW ")) + else { + continue; + }; + let name = rest.split_whitespace().next().unwrap(); + assert!( + drops.iter().any(|drop| drop.ends_with(&format!(" {name}"))), + "{name}" + ); + } + } +} diff --git a/src/podcasts/search.rs b/src/podcasts/search.rs new file mode 100644 index 0000000..76d42aa --- /dev/null +++ b/src/podcasts/search.rs @@ -0,0 +1,474 @@ +//! Public podcasts found by the federated search, seen by one account. +//! +//! The federated search (`/api/player/federation/search/events`) asks the +//! discovery overlay (`PodcastDht::search`) next to the music DHT and merges +//! both with `merge_search_results`: exact title matches first, music before +//! podcasts inside a tier. This module turns the podcast hits into +//! [`PodcastHitView`]s for the account that searched: +//! +//! * only the hit's `feed_url` (the origin URL the id hashes from) is +//! offered; `claimed_feed_urls` are unverified publisher claims and are +//! not passed on; +//! * every text and URL of a card is a claim of some publisher. Artwork and +//! web-page links are kept only as absolute `http(s)` URLs; the browser +//! renders all text as text; +//! * `subscribed` / `local_podcast_id` mark a podcast the account already +//! has, by id or through a known feed URL or alias (`resolve_podcast_id`). +//! +//! [`preview`] reads the found podcast's feed (through the SSRF-guarded +//! fetcher, nothing stored) so the user can look at its seasons and +//! episodes before subscribing; subscribing uses the origin URL, so the new +//! private subscription gets the card's id (see `api::subscribe_found`). + +use std::collections::BTreeMap; + +use anyhow::Result; +use music_dht::podcast::feed::ParsedFeed; +use music_dht::podcast::{FeedUrl, PodcastId}; +use music_dht::podcast_dht::{FederatedSearchHit, PodcastSearchHit, merge_search_results}; +use music_dht::{LibraryItem, normalize_name}; +use serde::Serialize; + +use super::{PodcastStore, PodcastSummary, SeasonFilter}; + +/// Podcast hits asked from the overlay per search. +pub(crate) const PODCAST_SEARCH_LIMIT: usize = 20; +/// Episodes listed in a preview (newest first), per season filter. +pub(crate) const PREVIEW_EPISODES: usize = 50; +/// Longest URL kept from a card or a feed. +const MAX_URL_BYTES: usize = 2048; + +/// One public podcast in the federated search results. +/// +/// Serialized with `"type": "podcast"` (music hits are `"type": "music"`). +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +pub(crate) struct PodcastHitView { + #[serde(rename = "type")] + pub(crate) hit_type: &'static str, + /// Always `podcast` (music hits: `artist`, `release`, `track`). + pub(crate) kind: &'static str, + pub(crate) podcast_id: String, + /// The origin feed URL: the only URL to subscribe with. + pub(crate) feed_url: String, + pub(crate) title: String, + pub(crate) author: Option, + pub(crate) description: Option, + /// Absolute `http(s)` artwork URL, if the card has a valid one. + pub(crate) image_url: Option, + /// Absolute `http(s)` web page, if the card has a valid one. + pub(crate) link: Option, + pub(crate) language: Option, + pub(crate) categories: Vec, + pub(crate) episode_count: Option, + pub(crate) latest_episode_at_ms: Option, + /// Distinct publisher devices (not subscribers, not a trust signal). + pub(crate) publisher_count: usize, + /// Position in the merged music and podcast results. + pub(crate) rank: usize, + /// The normalized title equals the normalized query. + pub(crate) exact_match: bool, + /// The account has an active subscription to this podcast. + pub(crate) subscribed: bool, + /// The account's podcast for this hit (its id, or the podcast a known + /// feed URL or alias resolves to), when the account knows it. + pub(crate) local_podcast_id: Option, +} + +/// An absolute `http(s)` URL without credentials, at most 2 048 bytes. +pub(crate) fn http_url(value: Option<&str>) -> Option { + let value = value?.trim(); + if value.is_empty() || value.len() > MAX_URL_BYTES { + return None; + } + let url = reqwest::Url::parse(value).ok()?; + (matches!(url.scheme(), "http" | "https") + && url.host_str().is_some() + && url.username().is_empty() + && url.password().is_none()) + .then(|| value.to_string()) +} + +/// Podcast hits in their merged order with the music results: +/// `(rank, exact_match, hit)`. +pub(crate) fn ranked_hits( + query: &str, + music: &[LibraryItem], + podcasts: Vec, +) -> Vec<(usize, bool, PodcastSearchHit)> { + let normalized = normalize_name(query); + merge_search_results(query, music.to_vec(), podcasts) + .into_iter() + .enumerate() + .filter_map(|(rank, hit)| match hit { + FederatedSearchHit::Podcast(hit) => { + let exact = !normalized.is_empty() && normalize_name(&hit.card.title) == normalized; + Some((rank, exact, hit)) + } + FederatedSearchHit::Music(_) => None, + }) + .collect() +} + +/// The account's podcast for a found podcast: the podcast with that id, or +/// the podcast one of `feed_urls` resolves to through the account's known +/// feed URLs and aliases (the card's origin URL; after a fetch also where +/// the feed moved, see `runner::document_urls`). +/// +/// A podcast the account is subscribed to comes first, so an old, +/// unsubscribed podcast of the same feed (followed once under its former +/// address) never hides the current subscription; otherwise the first +/// known one, in the order id, then `feed_urls`. +pub(crate) async fn known_podcast( + store: &PodcastStore, + user_id: i64, + podcast_id: &PodcastId, + feed_urls: &[FeedUrl], +) -> Result> { + let mut ids = vec![podcast_id.clone()]; + for feed_url in feed_urls { + let resolved = store.resolve_podcast_id(user_id, feed_url).await?; + if !ids.contains(&resolved) { + ids.push(resolved); + } + } + let mut first = None; + for id in ids { + let Some(summary) = store.podcast(user_id, &id).await? else { + continue; + }; + if summary.subscribed { + return Ok(Some(summary)); + } + first.get_or_insert(summary); + } + Ok(first) +} + +/// Views of the podcast hits of one search for `user_id`, in rank order. +pub(crate) async fn hit_views( + store: &PodcastStore, + user_id: i64, + query: &str, + music: &[LibraryItem], + podcasts: Vec, +) -> Result> { + let mut views = Vec::new(); + for (rank, exact_match, hit) in ranked_hits(query, music, podcasts) { + let known = known_podcast( + store, + user_id, + &hit.podcast_id, + std::slice::from_ref(&hit.feed_url), + ) + .await?; + let card = hit.card; + views.push(PodcastHitView { + hit_type: "podcast", + kind: "podcast", + podcast_id: hit.podcast_id.to_string(), + feed_url: hit.feed_url.to_string(), + title: card.title, + author: card.author, + description: card.description, + image_url: http_url(card.image_url.as_deref()), + link: http_url(card.link.as_deref()), + language: card.language, + categories: card.categories, + episode_count: card.episode_count, + latest_episode_at_ms: card.latest_episode_at_ms, + publisher_count: hit.publishers.len(), + rank, + exact_match, + subscribed: known.as_ref().is_some_and(|summary| summary.subscribed), + local_podcast_id: known.map(|summary| summary.podcast_id.to_string()), + }); + } + Ok(views) +} + +/// One season group of a preview (`season: null`: episodes without one). +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +pub(crate) struct PreviewSeason { + pub(crate) season: Option, + pub(crate) episode_count: usize, +} + +/// One episode of a preview. +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +pub(crate) struct PreviewEpisode { + pub(crate) title: String, + pub(crate) published_at_ms: Option, + pub(crate) duration_ms: Option, + pub(crate) season: Option, + pub(crate) episode_number: Option, +} + +/// A found podcast read from its feed: channel, seasons and the newest +/// episodes. Nothing of it is stored. +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +pub(crate) struct PodcastPreview { + pub(crate) podcast_id: String, + /// The URL that was read (the card's origin URL). + pub(crate) feed_url: String, + /// The feed answered with a permanent redirect chain to another URL. + pub(crate) moved: bool, + pub(crate) title: String, + pub(crate) author: Option, + pub(crate) description: Option, + pub(crate) image_url: Option, + pub(crate) link: Option, + pub(crate) episode_count: usize, + pub(crate) seasons: Vec, + /// Episodes of the requested season (all episodes without a filter). + pub(crate) listed_count: usize, + /// The newest of them, at most [`PREVIEW_EPISODES`]. + pub(crate) episodes: Vec, + pub(crate) subscribed: bool, + pub(crate) local_podcast_id: Option, +} + +/// The preview of a parsed feed (`known`: the account's podcast for it), +/// listing the episodes of `season`. +pub(crate) fn preview( + podcast_id: &PodcastId, + feed_url: &FeedUrl, + moved: bool, + feed: &ParsedFeed, + known: Option<&PodcastSummary>, + season: SeasonFilter, +) -> PodcastPreview { + let mut seasons: BTreeMap, usize> = BTreeMap::new(); + for episode in &feed.episodes { + *seasons.entry(episode.season).or_default() += 1; + } + let mut episodes: Vec<&_> = feed + .episodes + .iter() + .filter(|episode| match season { + SeasonFilter::Any => true, + SeasonFilter::Season(number) => episode.season == Some(number), + SeasonFilter::NoSeason => episode.season.is_none(), + }) + .collect(); + let listed_count = episodes.len(); + // Newest first; undated episodes keep their document order at the end. + episodes.sort_by_key(|episode| std::cmp::Reverse(episode.published_at_ms)); + PodcastPreview { + podcast_id: podcast_id.to_string(), + feed_url: feed_url.to_string(), + moved, + title: feed.title.clone(), + author: feed.author.clone(), + description: feed.description.clone(), + image_url: http_url(feed.image_url.as_deref()), + link: http_url(feed.link.as_deref()), + episode_count: feed.episodes.len(), + // Numbered seasons first, then the episodes without a season. + seasons: seasons + .iter() + .filter(|(season, _)| season.is_some()) + .chain(seasons.iter().filter(|(season, _)| season.is_none())) + .map(|(season, count)| PreviewSeason { + season: *season, + episode_count: *count, + }) + .collect(), + listed_count, + episodes: episodes + .into_iter() + .take(PREVIEW_EPISODES) + .map(|episode| PreviewEpisode { + title: episode.title.clone(), + published_at_ms: episode.published_at_ms, + duration_ms: episode.duration_ms, + season: episode.season, + episode_number: episode.episode_number, + }) + .collect(), + subscribed: known.is_some_and(|summary| summary.subscribed), + local_podcast_id: known.map(|summary| summary.podcast_id.to_string()), + } +} + +#[cfg(test)] +mod tests { + use super::*; + use music_dht::podcast_dht::{PodcastCard, PodcastCardSpec}; + use music_dht::{ItemId, ItemKind, NetworkId, SecretKey}; + + fn hit(path: &str, title: &str) -> PodcastSearchHit { + let feed_url = FeedUrl::parse(&format!("https://feeds.example/{path}")).unwrap(); + let podcast_id = PodcastId::from_feed_url(&feed_url); + let publisher = SecretKey::from_bytes(&[7; 32]).public(); + let card = PodcastCard::from_spec( + PodcastCardSpec { + podcast_id: podcast_id.clone(), + feed_url: feed_url.clone(), + id_feed_url: None, + title: title.into(), + author: None, + description: None, + image_url: None, + link: None, + language: None, + categories: Vec::new(), + episode_count: None, + latest_episode_at_ms: None, + }, + NetworkId::from_name("search-test"), + publisher, + 1, + 1_700_000_000_000, + ); + PodcastSearchHit { + podcast_id, + feed_url, + card, + publishers: vec![publisher], + claimed_feed_urls: Vec::new(), + } + } + + fn music(name: &str) -> LibraryItem { + LibraryItem { + id: ItemId::from_bytes([1; 32]), + owner: SecretKey::from_bytes(&[5; 32]).public(), + kind: ItemKind::Track, + name: name.into(), + normalized_name: normalize_name(name), + artist_names: Vec::new(), + featured_artist_names: Vec::new(), + year: None, + release_type: None, + release_title: None, + track_number: None, + disc_number: None, + duration_seconds: None, + content_id: None, + revision: 1, + deleted: false, + updated_at_ms: 1, + } + } + + #[test] + fn podcast_hits_take_their_place_among_music_results() { + let ranked = ranked_hits( + "Night Radio", + &[music("Night Radio Theme"), music("Night Radio")], + vec![ + hit("a.xml", "Night Radio"), + hit("b.xml", "Night Radio Talks"), + ], + ); + let ranks: Vec<_> = ranked + .iter() + .map(|(rank, exact, hit)| (*rank, *exact, hit.card.title.as_str())) + .collect(); + // Exact matches lead (music first inside a tier), then the rest. + assert_eq!( + ranks, + vec![(1, true, "Night Radio"), (3, false, "Night Radio Talks")] + ); + } + + #[test] + fn only_absolute_http_urls_without_credentials_are_kept() { + for (value, kept) in [ + ("https://cdn.example/art.jpg", true), + ("http://example.org/show", true), + ("javascript:alert(1)", false), + ("data:image/png;base64,AAAA", false), + ("//cdn.example/art.jpg", false), + ("https://user:secret@cdn.example/art.jpg", false), + ("file:///etc/passwd", false), + ("", false), + ] { + assert_eq!(http_url(Some(value)).is_some(), kept, "{value}"); + } + let long = format!("https://cdn.example/{}", "a".repeat(MAX_URL_BYTES)); + assert_eq!(http_url(Some(&long)), None); + assert_eq!(http_url(None), None); + } + + #[tokio::test] + async fn podcast_hits_that_cannot_be_prepared_are_reported_as_failed() { + use crate::federation::client::{MusicSearch, PodcastSearch}; + // The federation handle of a unit test has no database: ranking the + // hits for the account fails. The search still completes. + let federation = crate::federation::handle(); + let music = MusicSearch { + events: Vec::new(), + items: vec![music("Night Radio Theme")], + }; + let (state, events) = federation + .podcast_search_events( + "search", + "night radio", + 1, + &music, + PodcastSearch::Found(vec![hit("a.xml", "Night Radio")]), + ) + .await; + assert_eq!(state, "failed"); + assert!(events.is_empty()); + for (search, expected) in [ + (PodcastSearch::Found(Vec::new()), "running"), + (PodcastSearch::Off, "off"), + (PodcastSearch::Failed, "failed"), + ] { + let (state, events) = federation + .podcast_search_events("search", "night radio", 1, &music, search) + .await; + assert_eq!(state, expected); + assert!(events.is_empty()); + } + } + + #[test] + fn a_preview_lists_seasons_and_the_newest_episodes() { + let xml = r#" + +Preview Showjavascript:alert(1) +Old11 + Mon, 01 Jan 2024 10:00:00 GMT + +Bonus2 + +New32 + Mon, 01 Jul 2024 10:00:00 GMT + +"#; + let url = FeedUrl::parse("https://feeds.example/preview.xml").unwrap(); + let feed = + music_dht::podcast::feed::parse_feed_with_base(xml.as_bytes(), Some(url.as_str())) + .unwrap(); + let id = PodcastId::from_feed_url(&url); + let preview_of = |season| preview(&id, &url, false, &feed, None, season); + let preview = preview_of(SeasonFilter::Any); + assert_eq!(preview.title, "Preview Show"); + assert_eq!(preview.link, None, "only http(s) links"); + assert_eq!(preview.episode_count, 3); + let seasons: Vec<_> = preview + .seasons + .iter() + .map(|season| (season.season, season.episode_count)) + .collect(); + assert_eq!(seasons, vec![(Some(1), 1), (Some(2), 1), (None, 1)]); + let titles: Vec<_> = preview.episodes.iter().map(|e| e.title.as_str()).collect(); + assert_eq!(titles, vec!["New", "Old", "Bonus"]); + assert_eq!(preview.listed_count, 3); + assert!(!preview.subscribed && preview.local_podcast_id.is_none()); + + // One season (older seasons stay reachable), or the episodes + // without a season; the season groups stay complete. + let first = preview_of(SeasonFilter::Season(1)); + let titles: Vec<_> = first.episodes.iter().map(|e| e.title.as_str()).collect(); + assert_eq!(titles, vec!["Old"]); + assert_eq!((first.listed_count, first.seasons.len()), (1, 3)); + let unnumbered = preview_of(SeasonFilter::NoSeason); + assert_eq!(unnumbered.episodes[0].title, "Bonus"); + assert_eq!(unnumbered.listed_count, 1); + assert!(preview_of(SeasonFilter::Season(9)).episodes.is_empty()); + } +} diff --git a/src/podcasts/search_tests.rs b/src/podcasts/search_tests.rs new file mode 100644 index 0000000..249f2bc --- /dev/null +++ b/src/podcasts/search_tests.rs @@ -0,0 +1,798 @@ +//! Public podcasts in the federated search against PostgreSQL: the search +//! events (music unchanged, podcast hits added), the "already subscribed" +//! marker and subscribing from a card with the card's id. +//! +//! Ignored by default, like the other store tests: set +//! `FURU_DEVICE_SYNC_TEST_DATABASE_URL` to a disposable database. Feeds come +//! from a local test HTTP server. + +use std::collections::BTreeSet; + +use music_dht::device_sync::{SyncOpPayload, SyncOpWire, random_hex}; +use music_dht::podcast::{FeedUrl, PodcastId, PodcastMeta}; +use music_dht::podcast_dht::{PodcastCard, PodcastCardSpec, PodcastSearchHit}; +use music_dht::{ItemId, ItemKind, LibraryItem, NetworkId, SecretKey, normalize_name}; +use serde_json::{Value, json}; + +use super::api; +use super::postgres_tests::{cleanup, local_fetcher, random_user, test_pool}; +use super::test_server::{Route, TestServer}; +use super::*; +use crate::federation::client::{music_search_events, podcast_events}; + +fn url(value: &str) -> FeedUrl { + FeedUrl::parse(value).unwrap() +} + +/// A search hit as `PodcastDht::search` returns it, for the origin URL +/// `feed` (published by one device). +fn hit(feed: &str, title: &str) -> PodcastSearchHit { + let feed_url = url(feed); + let podcast_id = PodcastId::from_feed_url(&feed_url); + let publisher = SecretKey::from_bytes(&[9; 32]).public(); + let card = PodcastCard::from_spec( + PodcastCardSpec { + podcast_id: podcast_id.clone(), + feed_url: feed_url.clone(), + id_feed_url: None, + title: title.into(), + author: Some("Host".into()), + description: Some("Not markup".into()), + image_url: Some("https://cdn.example/art.jpg".into()), + link: Some("https://show.example/".into()), + language: Some("en".into()), + categories: vec!["Technology".into()], + episode_count: Some(12), + latest_episode_at_ms: Some(1_700_000_000_000), + }, + NetworkId::from_name("search-tests"), + publisher, + 1, + 1_700_000_000_000, + ); + PodcastSearchHit { + podcast_id, + feed_url, + card, + publishers: vec![publisher], + claimed_feed_urls: Vec::new(), + } +} + +fn item(kind: ItemKind, name: &str, content_id: Option<&str>) -> LibraryItem { + LibraryItem { + id: ItemId::from_bytes(*blake3::hash(name.as_bytes()).as_bytes()), + owner: SecretKey::from_bytes(&[5; 32]).public(), + kind, + name: name.into(), + normalized_name: normalize_name(name), + artist_names: vec!["Night Owls".into()], + featured_artist_names: Vec::new(), + year: Some(2024), + release_type: None, + release_title: (kind == ItemKind::Track).then(|| "Night Radio".into()), + track_number: Some(1), + disc_number: None, + duration_seconds: Some(180.0), + content_id: content_id.map(str::to_string), + revision: 1, + deleted: false, + updated_at_ms: 1, + } +} + +fn keys(value: &Value) -> BTreeSet { + value.as_object().unwrap().keys().cloned().collect() +} + +fn set(names: &[&str]) -> BTreeSet { + names.iter().map(|name| name.to_string()).collect() +} + +fn meta(feed: &str, title: &str) -> PodcastMeta { + PodcastMeta { + feed_url: url(feed), + title: title.into(), + author: None, + description: None, + link: None, + image_url: None, + } +} + +#[tokio::test] +#[ignore = "requires a disposable database in FURU_DEVICE_SYNC_TEST_DATABASE_URL"] +async fn postgres_federated_search_keeps_music_events_and_adds_podcast_hits() { + let pool = test_pool().await; + let store = PodcastStore::new(pool.clone()); + let (user, other) = (random_user(), random_user()); + cleanup(&pool, [user, other]).await; + let tag = random_hex(6); + let content_id = format!("b3:{}", blake3::hash(tag.as_bytes()).to_hex()); + + // Music: the contract of the existing events is unchanged; only the + // `type` marker is added. + let music = vec![ + item(ItemKind::Track, "Night Radio Theme", Some(&content_id)), + item(ItemKind::Artist, "Night Owls", None), + item(ItemKind::Release, "Night Radio", None), + ]; + let own = SecretKey::from_bytes(&[1; 32]).public(); + let music_events = music_search_events(&pool, "search-1", &music, own) + .await + .unwrap(); + let music_json: Vec = music_events + .iter() + .map(|event| serde_json::to_value(event).unwrap()) + .collect(); + let kinds: Vec<&str> = music_json + .iter() + .map(|event| event["kind"].as_str().unwrap()) + .collect(); + assert_eq!( + kinds, + vec![ + "federation.track", + "federation.artist", + "federation.release" + ] + ); + for (index, event) in music_json.iter().enumerate() { + assert_eq!( + keys(event), + set(&[ + "search_id", + "sequence", + "kind", + "type", + "peer", + "entity_key", + "entity" + ]) + ); + assert_eq!(event["type"], "music"); + assert_eq!(event["search_id"], "search-1"); + assert_eq!(event["sequence"], index as u64 + 1); + } + let track = &music_json[0]; + assert_eq!(track["entity_key"], json!({ "content_id": content_id })); + assert_eq!( + keys(&track["entity"]), + set(&["key", "metadata", "availability"]) + ); + assert_eq!(track["entity"]["metadata"]["title"], "Night Radio Theme"); + assert_eq!(track["entity"]["availability"]["state"], "federated"); + assert_eq!(track["peer"], music[0].owner.to_string()); + assert_eq!( + keys(&music_json[1]["entity"]), + set(&["key", "name", "image_url", "peers"]) + ); + assert_eq!( + keys(&music_json[2]["entity"]), + set(&["key", "title", "artists", "year", "cover_url", "sources"]) + ); + + // The account has podcast A, and podcast C through an alias. + let feed_a = format!("https://podcasts.example/a-{tag}.xml"); + let feed_b = format!("https://podcasts.example/b-{tag}.xml"); + let feed_c = format!("https://podcasts.example/c-{tag}.xml"); + let alias_c = format!("https://moved.example/c-{tag}.xml"); + let id_a = store + .subscribe(user, &meta(&feed_a, "Night Radio")) + .await + .unwrap() + .podcast_id; + let id_c = store + .subscribe(user, &meta(&feed_c, "Cellar Talks")) + .await + .unwrap() + .podcast_id; + let aliases = SyncOpWire { + op_id: format!("tui-{tag}:1"), + origin_device_id: format!("tui-{tag}"), + seq: 1, + hlc_ms: 9_000_000_000_000, + payload: SyncOpPayload::PodcastFeedAliasesAdded { + podcast_id: id_c.clone(), + feed_aliases: vec![url(&feed_c), url(&alias_c)], + }, + }; + store.apply_remote_op(user, &aliases).await.unwrap(); + + // Someone published a card for the alias URL: another podcast id, but + // the account already has it. + let hits = vec![ + hit(&feed_b, "Night Radio Stories"), + hit(&alias_c, "Night Radio Cellar"), + hit(&feed_a, "Night Radio"), + ]; + let first = music_events.len() as u64 + 1; + let events = podcast_events( + &pool, + user, + "search-1", + "night radio", + &music, + first, + hits.clone(), + ) + .await + .unwrap(); + let json: Vec = events + .iter() + .map(|event| serde_json::to_value(event).unwrap()) + .collect(); + assert_eq!(json.len(), 3); + for (index, event) in json.iter().enumerate() { + assert_eq!(event["kind"], "federation.podcast"); + assert_eq!(event["type"], "podcast"); + assert_eq!(event["peer"], Value::Null); + assert_eq!(event["sequence"], first + index as u64); + assert_eq!(event["entity"]["type"], "podcast"); + assert_eq!(event["entity"]["kind"], "podcast"); + assert_eq!( + event["entity_key"], + json!({ "podcast_id": event["entity"]["podcast_id"] }) + ); + // The publisher's personal state never appears: no registers, + // devices or publisher keys, only the card's public projection. + assert_eq!( + keys(&event["entity"]), + set(&[ + "type", + "kind", + "podcast_id", + "feed_url", + "title", + "author", + "description", + "image_url", + "link", + "language", + "categories", + "episode_count", + "latest_episode_at_ms", + "publisher_count", + "rank", + "exact_match", + "subscribed", + "local_podcast_id", + ]) + ); + } + // The exact title leads (after the exact music release), then the rest + // in the overlay's order. + let entity = |index: usize| &json[index]["entity"]; + assert_eq!(entity(0)["podcast_id"], id_a.to_string()); + assert_eq!(entity(0)["exact_match"], true); + assert_eq!(entity(0)["rank"], 1); + assert_eq!(entity(0)["subscribed"], true); + assert_eq!(entity(0)["local_podcast_id"], id_a.to_string()); + assert_eq!(entity(0)["feed_url"], feed_a); + assert_eq!(entity(0)["description"], "Not markup"); + assert_eq!(entity(0)["publisher_count"], 1); + assert_eq!(entity(1)["podcast_id"], hits[0].podcast_id.to_string()); + assert_eq!(entity(1)["subscribed"], false); + assert_eq!(entity(1)["local_podcast_id"], Value::Null); + assert_eq!(entity(2)["podcast_id"], hits[1].podcast_id.to_string()); + assert_ne!(hits[1].podcast_id, id_c); + assert_eq!(entity(2)["subscribed"], true, "known through the alias"); + assert_eq!(entity(2)["local_podcast_id"], id_c.to_string()); + + // The marker belongs to the searching account. + let theirs = podcast_events(&pool, other, "search-2", "night radio", &music, 1, hits) + .await + .unwrap(); + assert!( + theirs + .iter() + .all(|event| event.entity["subscribed"] == false + && event.entity["local_podcast_id"].is_null()) + ); + + // Unsubscribed: known, but no longer marked as subscribed. + api::unsubscribe(&store, user, &id_a).await.unwrap(); + let events = podcast_events( + &pool, + user, + "search-3", + "night radio", + &[], + 1, + vec![hit(&feed_a, "Night Radio")], + ) + .await + .unwrap(); + assert_eq!(events[0].entity["subscribed"], false); + assert_eq!(events[0].entity["local_podcast_id"], id_a.to_string()); + assert_eq!(events[0].entity["rank"], 0); + + sqlx::query("DELETE FROM furumusic__track_ref WHERE content_id = $1") + .bind(&content_id) + .execute(&pool) + .await + .unwrap(); + cleanup(&pool, [user, other]).await; +} + +fn rss(title: &str, items: &str) -> String { + format!( + r#" + + + {title} + https://show.example/ + Found in the federation + {items} + +"# + ) +} + +const ITEMS: &str = r#" + Pilots1e11 + 1Mon, 01 Jan 2024 10:00:00 GMT + + Seconds1e21 + 2Mon, 08 Jan 2024 10:00:00 GMT + + Bonusbonus + "#; + +async fn log_len(pool: &sqlx::PgPool, user: i64) -> i64 { + sqlx::query_scalar("SELECT COUNT(*) FROM furumusic__fed_sync_ops WHERE user_id = $1") + .bind(user) + .fetch_one(pool) + .await + .unwrap() +} + +#[tokio::test] +#[ignore = "requires a disposable database in FURU_DEVICE_SYNC_TEST_DATABASE_URL"] +async fn postgres_subscribing_from_a_card_keeps_the_card_id() { + let pool = test_pool().await; + let store = PodcastStore::new(pool.clone()); + let (user, other) = (random_user(), random_user()); + cleanup(&pool, [user, other]).await; + let server = TestServer::start(); + server.set( + "/show.xml", + Route::new(rss("Found Show", ITEMS), "application/rss+xml"), + ); + server.set("/old.xml", Route::redirect(301, "/new.xml")); + server.set( + "/new.xml", + Route::new(rss("Moved Show", ITEMS), "application/rss+xml"), + ); + let fetcher = local_fetcher(); + let now = super::now_ms(); + let show = hit(&server.url("/show.xml"), "Found Show"); + let body = |hit: &PodcastSearchHit| json!({ "podcast_id": hit.podcast_id.to_string(), "feed_url": hit.feed_url.to_string() }); + + // Invalid requests fetch nothing. + let other_id = PodcastId::from_feed_url(&url("https://podcasts.example/other.xml")); + for (request, status) in [ + (json!({}), 400), + ( + json!({ "podcast_id": "x", "feed_url": show.feed_url.to_string() }), + 400, + ), + ( + json!({ "podcast_id": show.podcast_id.to_string(), "feed_url": "ftp://x/y" }), + 400, + ), + // A URL the id does not come from (a claimed move, another podcast). + ( + json!({ "podcast_id": other_id.to_string(), "feed_url": show.feed_url.to_string() }), + 422, + ), + ] { + let error = api::preview_found(&store, &fetcher, user, request.clone()) + .await + .unwrap_err(); + assert_eq!(error.status, status, "{request}: {}", error.message); + let error = api::subscribe_found(&store, &fetcher, user, request.clone(), now) + .await + .unwrap_err(); + assert_eq!(error.status, status, "{request}: {}", error.message); + } + assert_eq!(server.requests("/show.xml").len(), 0); + // The production policy never reaches the local server. + let refused = api::preview_found(&store, &FeedFetcher::new().unwrap(), user, body(&show)) + .await + .unwrap_err(); + assert_eq!(refused.status, 422, "{}", refused.message); + + // The preview reads the feed and stores nothing. + let preview = api::preview_found(&store, &fetcher, user, body(&show)) + .await + .unwrap(); + assert_eq!(preview.title, "Found Show"); + assert_eq!(preview.podcast_id, show.podcast_id.to_string()); + assert_eq!(preview.episode_count, 3); + let seasons: Vec<_> = preview + .seasons + .iter() + .map(|season| (season.season, season.episode_count)) + .collect(); + assert_eq!(seasons, vec![(Some(1), 2), (None, 1)]); + assert_eq!(preview.episodes[0].title, "Second"); + assert!(!preview.subscribed && !preview.moved); + assert_eq!(server.requests("/show.xml").len(), 1); + assert!( + store + .podcast(user, &show.podcast_id) + .await + .unwrap() + .is_none() + ); + assert_eq!(log_len(&pool, user).await, 0); + + // Subscribing: the account's own private subscription with the card's id. + let subscribed = api::subscribe_found(&store, &fetcher, user, body(&show), now) + .await + .unwrap(); + assert_eq!(subscribed.podcast.podcast_id, show.podcast_id.to_string()); + assert!(!subscribed.already_subscribed); + assert_eq!(subscribed.new_episodes, 3); + assert!(subscribed.podcast.subscribed); + assert!( + !subscribed.podcast.shared, + "a subscription from a card is private" + ); + assert_eq!(server.requests("/show.xml").len(), 2); + let marked = super::search::hit_views(&store, user, "found show", &[], vec![show.clone()]) + .await + .unwrap(); + assert!(marked[0].subscribed); + + // Again: already subscribed, nothing fetched or written. + let log = log_len(&pool, user).await; + let again = api::subscribe_found(&store, &fetcher, user, body(&show), now) + .await + .unwrap(); + assert!(again.already_subscribed); + assert_eq!(again.podcast.podcast_id, show.podcast_id.to_string()); + assert_eq!(server.requests("/show.xml").len(), 2); + assert_eq!(log_len(&pool, user).await, log); + let preview = api::preview_found(&store, &fetcher, user, body(&show)) + .await + .unwrap(); + assert!(preview.subscribed); + assert_eq!( + preview.local_podcast_id.as_deref(), + Some(show.podcast_id.as_str()) + ); + // One season at a time; a bad season is refused before any fetch. + let mut seasonal = body(&show); + seasonal["season"] = json!("1"); + let first = api::preview_found(&store, &fetcher, user, seasonal) + .await + .unwrap(); + assert_eq!(first.listed_count, 2); + assert!( + first + .episodes + .iter() + .all(|episode| episode.season == Some(1)) + ); + let mut unnumbered = body(&show); + unnumbered["season"] = json!("none"); + let none = api::preview_found(&store, &fetcher, user, unnumbered) + .await + .unwrap(); + assert_eq!(none.listed_count, 1); + assert!(none.episodes.iter().all(|episode| episode.season.is_none())); + let requests = server.requests("/show.xml").len(); + let mut bad = body(&show); + bad["season"] = json!("first"); + let error = api::preview_found(&store, &fetcher, user, bad) + .await + .unwrap_err(); + assert_eq!(error.status, 400); + assert_eq!(server.requests("/show.xml").len(), requests); + + // Another account subscribing from the same card gets its own podcast + // with the same id; the first account's state is untouched. + let theirs = api::subscribe_found(&store, &fetcher, other, body(&show), now) + .await + .unwrap(); + assert_eq!(theirs.podcast.podcast_id, show.podcast_id.to_string()); + assert!(!theirs.already_subscribed); + + // A card whose origin URL moved permanently: the id stays the card's, + // the new URL becomes an alias and is fetched from now on. + let old = hit(&server.url("/old.xml"), "Moved Show"); + let moved = api::subscribe_found(&store, &fetcher, user, body(&old), now) + .await + .unwrap(); + assert_eq!(moved.podcast.podcast_id, old.podcast_id.to_string()); + assert_eq!(moved.podcast.feed_url, server.url("/new.xml")); + let aliases = store.feed_aliases(user, &old.podcast_id).await.unwrap(); + assert!(aliases.contains(&url(&server.url("/new.xml")))); + // Someone's card for the new URL is the same podcast for this account. + let new = hit(&server.url("/new.xml"), "Moved Show"); + assert_ne!(new.podcast_id, old.podcast_id); + let requests = server.requests("/new.xml").len(); + let again = api::subscribe_found(&store, &fetcher, user, body(&new), now) + .await + .unwrap(); + assert!(again.already_subscribed); + assert_eq!(again.podcast.podcast_id, old.podcast_id.to_string()); + assert_eq!(server.requests("/new.xml").len(), requests); + assert!( + store + .podcast(user, &new.podcast_id) + .await + .unwrap() + .is_none() + ); + + // The reverse: an account that follows the feed only under its new + // address finds the card of the old one. The preview fetches the old + // URL, sees the move and marks it as already subscribed; subscribing + // writes nothing and creates no second podcast (nor does "add RSS"). + let third = random_user(); + let direct = runner::subscribe_url(&store, &fetcher, third, &url(&server.url("/new.xml")), now) + .await + .unwrap(); + assert_eq!(direct.podcast_id, new.podcast_id); + let marked = super::search::hit_views(&store, third, "moved show", &[], vec![old.clone()]) + .await + .unwrap(); + // The search list cannot know without fetching the feed. + assert!(!marked[0].subscribed); + let preview = api::preview_found(&store, &fetcher, third, body(&old)) + .await + .unwrap(); + assert!(preview.moved && preview.subscribed); + assert_eq!( + preview.local_podcast_id.as_deref(), + Some(new.podcast_id.as_str()) + ); + let log = log_len(&pool, third).await; + let again = api::subscribe_found(&store, &fetcher, third, body(&old), now) + .await + .unwrap(); + assert!(again.already_subscribed); + assert_eq!(again.podcast.podcast_id, new.podcast_id.to_string()); + let added = runner::subscribe_url(&store, &fetcher, third, &url(&server.url("/old.xml")), now) + .await + .unwrap(); + assert!(added.already_subscribed); + assert_eq!(added.podcast_id, new.podcast_id); + assert_eq!(log_len(&pool, third).await, log); + assert!( + store + .podcast(third, &old.podcast_id) + .await + .unwrap() + .is_none() + ); + let subscriptions: i64 = sqlx::query_scalar( + "SELECT COUNT(*) FROM furumusic__podcast_subscription WHERE user_id = $1", + ) + .bind(third) + .fetch_one(&pool) + .await + .unwrap(); + assert_eq!(subscriptions, 1); + + cleanup(&pool, [user, other, third]).await; +} + +/// The catalog row and the episodes of one podcast of the account: +/// `(title, description, meta_from_feed, [(episode title, in_feed)])`. +async fn podcast_snapshot( + pool: &sqlx::PgPool, + user: i64, + podcast_id: &PodcastId, +) -> (String, Option, bool, Vec<(String, bool)>) { + let (title, description, meta_from_feed): (String, Option, bool) = sqlx::query_as( + "SELECT title, description, meta_from_feed FROM furumusic__podcast + WHERE user_id = $1 AND podcast_id = $2", + ) + .bind(user) + .bind(podcast_id.as_str()) + .fetch_one(pool) + .await + .unwrap(); + let episodes: Vec<(String, bool)> = sqlx::query_as( + "SELECT title, in_feed FROM furumusic__podcast_episode + WHERE user_id = $1 ORDER BY title", + ) + .bind(user) + .fetch_all(pool) + .await + .unwrap(); + (title, description, meta_from_feed, episodes) +} + +async fn active_subscriptions(pool: &sqlx::PgPool, user: i64) -> i64 { + sqlx::query_scalar( + "SELECT COUNT(*) FROM furumusic__podcast_subscription + WHERE user_id = $1 AND subscribed", + ) + .bind(user) + .fetch_one(pool) + .await + .unwrap() +} + +#[tokio::test] +#[ignore = "requires a disposable database in FURU_DEVICE_SYNC_TEST_DATABASE_URL"] +async fn postgres_a_feed_claiming_a_followed_address_never_rewrites_that_podcast() { + let pool = test_pool().await; + let store = PodcastStore::new(pool.clone()); + let user = random_user(); + cleanup(&pool, [user]).await; + let server = TestServer::start(); + let real = server.url("/real.xml"); + server.set( + "/real.xml", + Route::new(rss("Real Show", ITEMS), "application/rss+xml"), + ); + // The old address serves a frozen copy that names the new address; a + // feed of somebody else claims to have moved there too. + let claim = |title: &str, guid: &str| { + rss( + title, + &format!( + r#"{real} + {title} item{guid} + "# + ), + ) + }; + server.set( + "/stub.xml", + Route::new(claim("Stale Copy", "stale"), "application/rss+xml"), + ); + server.set( + "/evil.xml", + Route::new(claim("Injected", "evil"), "application/rss+xml"), + ); + let fetcher = local_fetcher(); + let now = super::now_ms(); + let real_id = PodcastId::from_feed_url(&url(&real)); + let followed = runner::subscribe_url(&store, &fetcher, user, &url(&real), now) + .await + .unwrap(); + assert_eq!(followed.podcast_id, real_id); + let before = podcast_snapshot(&pool, user, &real_id).await; + assert_eq!(before.0, "Real Show"); + assert_eq!(before.3.len(), 3); + assert!(before.3.iter().all(|(_, in_feed)| *in_feed)); + let register_before = store.subscription(user, &real_id).await.unwrap().unwrap(); + let log = log_len(&pool, user).await; + + // "Add RSS" of the old address and "Subscribe" on a card of the other + // feed: the account already follows the feed they name, nothing of + // theirs is written into it. + for added in [ + runner::subscribe_url(&store, &fetcher, user, &url(&server.url("/stub.xml")), now) + .await + .unwrap(), + runner::subscribe_url(&store, &fetcher, user, &url(&server.url("/evil.xml")), now) + .await + .unwrap(), + ] { + assert!(added.already_subscribed); + assert_eq!(added.podcast_id, real_id); + assert_eq!(added.refresh.new_episodes, 0); + } + let evil = hit(&server.url("/evil.xml"), "Injected"); + let body = + json!({ "podcast_id": evil.podcast_id.to_string(), "feed_url": evil.feed_url.to_string() }); + let found = api::subscribe_found(&store, &fetcher, user, body.clone(), now) + .await + .unwrap(); + assert!(found.already_subscribed); + assert_eq!(found.podcast.podcast_id, real_id.to_string()); + assert_eq!(podcast_snapshot(&pool, user, &real_id).await, before); + assert_eq!(log_len(&pool, user).await, log); + assert_eq!(active_subscriptions(&pool, user).await, 1); + + // Unsubscribed, the same claim subscribes the account's podcast again + // with its own stored metadata; the claiming document is not ingested + // (the refresh job fetches the podcast's own address). + store.unsubscribe(user, &real_id).await.unwrap(); + let again = runner::subscribe_url(&store, &fetcher, user, &url(&server.url("/evil.xml")), now) + .await + .unwrap(); + assert!(!again.already_subscribed); + assert_eq!(again.podcast_id, real_id); + assert_eq!(again.refresh.new_episodes, 0); + assert_eq!(podcast_snapshot(&pool, user, &real_id).await, before); + let register = store.subscription(user, &real_id).await.unwrap().unwrap(); + assert!(register.subscribed); + assert_eq!(register.podcast, register_before.podcast); + assert!( + store + .podcast(user, &evil.podcast_id) + .await + .unwrap() + .is_none() + ); + + // Its own address is still ingested as before (a refresh). + let own = runner::subscribe_url(&store, &fetcher, user, &url(&real), now) + .await + .unwrap(); + assert!(own.already_subscribed); + assert_eq!(own.refresh.unchanged_episodes, 3); + + cleanup(&pool, [user]).await; +} + +#[tokio::test] +#[ignore = "requires a disposable database in FURU_DEVICE_SYNC_TEST_DATABASE_URL"] +async fn postgres_a_current_subscription_wins_over_an_old_unsubscribed_podcast() { + let pool = test_pool().await; + let store = PodcastStore::new(pool.clone()); + let user = random_user(); + cleanup(&pool, [user]).await; + let server = TestServer::start(); + server.set( + "/old.xml", + Route::new(rss("Moving Show", ITEMS), "application/rss+xml"), + ); + server.set( + "/new.xml", + Route::new(rss("Moving Show", ITEMS), "application/rss+xml"), + ); + let fetcher = local_fetcher(); + let now = super::now_ms(); + let old = hit(&server.url("/old.xml"), "Moving Show"); + let new_id = PodcastId::from_feed_url(&url(&server.url("/new.xml"))); + + // Followed once under the old address, then unsubscribed; the feed + // moved, and the account follows it under the new address only. + runner::subscribe_url(&store, &fetcher, user, &old.feed_url, now) + .await + .unwrap(); + store.unsubscribe(user, &old.podcast_id).await.unwrap(); + server.set("/old.xml", Route::redirect(301, "/new.xml")); + let current = runner::subscribe_url(&store, &fetcher, user, &url(&server.url("/new.xml")), now) + .await + .unwrap(); + assert_eq!(current.podcast_id, new_id); + assert!(!current.already_subscribed); + + let known = super::search::known_podcast( + &store, + user, + &old.podcast_id, + &[old.feed_url.clone(), url(&server.url("/new.xml"))], + ) + .await + .unwrap() + .unwrap(); + assert_eq!(known.podcast_id, new_id); + assert!(known.subscribed); + + let body = + json!({ "podcast_id": old.podcast_id.to_string(), "feed_url": old.feed_url.to_string() }); + let preview = api::preview_found(&store, &fetcher, user, body.clone()) + .await + .unwrap(); + assert!(preview.subscribed); + assert_eq!(preview.local_podcast_id.as_deref(), Some(new_id.as_str())); + let log = log_len(&pool, user).await; + let found = api::subscribe_found(&store, &fetcher, user, body, now) + .await + .unwrap(); + assert!(found.already_subscribed); + assert_eq!(found.podcast.podcast_id, new_id.to_string()); + let added = runner::subscribe_url(&store, &fetcher, user, &old.feed_url, now) + .await + .unwrap(); + assert!(added.already_subscribed); + assert_eq!(added.podcast_id, new_id); + assert_eq!(log_len(&pool, user).await, log); + assert_eq!(active_subscriptions(&pool, user).await, 1); + let old_podcast = store.podcast(user, &old.podcast_id).await.unwrap().unwrap(); + assert!(!old_podcast.subscribed); + + cleanup(&pool, [user]).await; +} diff --git a/src/podcasts/test_server.rs b/src/podcasts/test_server.rs new file mode 100644 index 0000000..e4344da --- /dev/null +++ b/src/podcasts/test_server.rs @@ -0,0 +1,375 @@ +//! Test support of the podcast runtime: a scripted resolver, a client that +//! may reach loopback, and a minimal HTTP/1.1 server on 127.0.0.1 (no +//! internet). One request per connection (`Connection: close`), so every +//! request opens a new connection and resolves its host again. +//! +//! Adapted from the furumi-tui prototype's test server: routes can redirect, +//! send extra headers and answer conditional requests, and every request's +//! headers are recorded. + +use std::collections::{HashMap, VecDeque}; +use std::io::{BufRead, BufReader, Write}; +use std::net::{IpAddr, TcpListener, TcpStream}; +use std::sync::atomic::{AtomicUsize, Ordering}; +use std::sync::{Arc, Mutex, PoisonError}; +use std::time::Duration; + +use super::net::{AddressPolicy, HostLookup, LookupFuture, SafeHttp, SafeHttpConfig}; + +#[derive(Clone)] +pub(crate) struct Route { + pub(crate) status: u16, + pub(crate) body: Arc>, + pub(crate) content_type: Option, + pub(crate) etag: Option, + pub(crate) last_modified: Option, + /// Honour `Range` requests with 206 responses. + pub(crate) ranges: bool, + /// Send `Content-Length` (otherwise the body is close-delimited). + pub(crate) send_length: bool, + /// Close the connection once the body reaches this absolute offset. + pub(crate) cut_after: Option, + /// Extra response headers. + pub(crate) headers: Vec<(String, String)>, + /// Pause between the headers and the body. + pub(crate) body_delay: Option, +} + +impl Route { + pub(crate) fn new(body: impl Into>, content_type: &str) -> Route { + Route { + status: 200, + body: Arc::new(body.into()), + content_type: Some(content_type.to_string()), + etag: None, + last_modified: None, + ranges: true, + send_length: true, + cut_after: None, + headers: Vec::new(), + body_delay: None, + } + } + + /// A redirect to `location` with `status`. + pub(crate) fn redirect(status: u16, location: &str) -> Route { + Route { + status, + body: Arc::new(Vec::new()), + content_type: None, + ranges: false, + ..Route::new(Vec::new(), "text/plain") + } + .header("Location", location) + } + + pub(crate) fn status(mut self, status: u16) -> Route { + self.status = status; + self + } + + pub(crate) fn etag(mut self, etag: &str) -> Route { + self.etag = Some(etag.to_string()); + self + } + + pub(crate) fn last_modified(mut self, date: &str) -> Route { + self.last_modified = Some(date.to_string()); + self + } + + pub(crate) fn no_ranges(mut self) -> Route { + self.ranges = false; + self + } + + pub(crate) fn no_length(mut self) -> Route { + self.send_length = false; + self + } + + pub(crate) fn no_content_type(mut self) -> Route { + self.content_type = None; + self + } + + pub(crate) fn cut(mut self, after: usize) -> Route { + self.cut_after = Some(after); + self + } + + pub(crate) fn delay_body(mut self, delay: Duration) -> Route { + self.body_delay = Some(delay); + self + } + + pub(crate) fn header(mut self, name: &str, value: &str) -> Route { + self.headers.push((name.to_string(), value.to_string())); + self + } +} + +/// Lowercased request headers of one request. +pub(crate) type RequestHeaders = HashMap; + +#[derive(Default)] +struct ServerState { + routes: HashMap, + requests: HashMap>, +} + +pub(crate) struct TestServer { + port: u16, + state: Arc>, +} + +impl TestServer { + pub(crate) fn start() -> TestServer { + let listener = TcpListener::bind("127.0.0.1:0").expect("bind test server"); + let port = listener.local_addr().expect("local addr").port(); + let state = Arc::new(Mutex::new(ServerState::default())); + let accept_state = Arc::clone(&state); + std::thread::spawn(move || { + for stream in listener.incoming() { + let Ok(stream) = stream else { continue }; + let state = Arc::clone(&accept_state); + std::thread::spawn(move || handle(stream, &state)); + } + }); + TestServer { port, state } + } + + /// `http://127.0.0.1:`. + pub(crate) fn url(&self, path: &str) -> String { + format!("http://127.0.0.1:{}{path}", self.port) + } + + /// `http://:` (for hosts mapped by a test resolver). + pub(crate) fn url_on(&self, host: &str, path: &str) -> String { + format!("http://{host}:{}{path}", self.port) + } + + pub(crate) fn set(&self, path: &str, route: Route) { + self.lock().routes.insert(path.to_string(), route); + } + + /// Headers of every request to `path`, in order. + pub(crate) fn requests(&self, path: &str) -> Vec { + self.lock().requests.get(path).cloned().unwrap_or_default() + } + + /// Requests to any path. + pub(crate) fn request_count(&self) -> usize { + self.lock().requests.values().map(Vec::len).sum() + } + + fn lock(&self) -> std::sync::MutexGuard<'_, ServerState> { + self.state.lock().unwrap_or_else(PoisonError::into_inner) + } +} + +fn handle(stream: TcpStream, state: &Mutex) { + let _ = stream.set_read_timeout(Some(Duration::from_secs(10))); + let mut reader = BufReader::new(match stream.try_clone() { + Ok(stream) => stream, + Err(_) => return, + }); + let mut request_line = String::new(); + if reader.read_line(&mut request_line).is_err() { + return; + } + let path = request_line + .split_whitespace() + .nth(1) + .unwrap_or("/") + .to_string(); + let mut headers = HashMap::new(); + loop { + let mut line = String::new(); + match reader.read_line(&mut line) { + Ok(0) | Err(_) => return, + Ok(_) => {} + } + let line = line.trim_end(); + if line.is_empty() { + break; + } + if let Some((name, value)) = line.split_once(':') { + headers.insert(name.trim().to_ascii_lowercase(), value.trim().to_string()); + } + } + let route = { + let mut state = state.lock().unwrap_or_else(PoisonError::into_inner); + state + .requests + .entry(path.clone()) + .or_default() + .push(headers.clone()); + state.routes.get(&path).cloned() + }; + let mut out = stream; + let Some(route) = route else { + let _ = out + .write_all(b"HTTP/1.1 404 Not Found\r\nContent-Length: 0\r\nConnection: close\r\n\r\n"); + return; + }; + let not_modified = match (&route.etag, headers.get("if-none-match")) { + (Some(etag), Some(sent)) => etag == sent, + _ => matches!( + (&route.last_modified, headers.get("if-modified-since")), + (Some(date), Some(sent)) if date == sent + ), + }; + if route.status == 200 && not_modified { + let _ = write!( + out, + "HTTP/1.1 304 Not Modified\r\nConnection: close\r\n\r\n" + ); + return; + } + let total = route.body.len(); + let mut status = route.status; + let mut start = 0usize; + let mut end = total; + let mut extra = String::new(); + if route.status == 200 + && route.ranges + && let Some(range) = headers.get("range").and_then(|r| r.strip_prefix("bytes=")) + && let Some((from, to)) = range.split_once('-') + { + let (from, to) = match (from.trim().parse::(), to.trim().parse::()) { + (Ok(from), Ok(to)) => (from, (to + 1).min(total)), + (Ok(from), Err(_)) => (from, total), + (Err(_), Ok(suffix)) => (total.saturating_sub(suffix), total), + _ => (0, total), + }; + if from >= total { + let _ = write!( + out, + "HTTP/1.1 416 Range Not Satisfiable\r\nContent-Range: bytes */{total}\r\nContent-Length: 0\r\nConnection: close\r\n\r\n" + ); + return; + } + status = 206; + start = from; + end = to; + extra.push_str(&format!( + "Content-Range: bytes {start}-{}/{total}\r\n", + end - 1 + )); + } + if route.ranges { + extra.push_str("Accept-Ranges: bytes\r\n"); + } + if let Some(content_type) = &route.content_type { + extra.push_str(&format!("Content-Type: {content_type}\r\n")); + } + if let Some(etag) = &route.etag { + extra.push_str(&format!("ETag: {etag}\r\n")); + } + if let Some(date) = &route.last_modified { + extra.push_str(&format!("Last-Modified: {date}\r\n")); + } + for (name, value) in &route.headers { + extra.push_str(&format!("{name}: {value}\r\n")); + } + if route.send_length { + extra.push_str(&format!("Content-Length: {}\r\n", end - start)); + } + if write!( + out, + "HTTP/1.1 {status} Status\r\n{extra}Connection: close\r\n\r\n" + ) + .is_err() + { + return; + } + let _ = out.flush(); + if let Some(delay) = route.body_delay { + std::thread::sleep(delay); + } + let limit = route.cut_after.map_or(end, |cut| cut.clamp(start, end)); + for chunk in route.body[start..limit].chunks(64 * 1024) { + if out.write_all(chunk).is_err() { + return; + } + } + let _ = out.flush(); + let _ = out.shutdown(std::net::Shutdown::Both); +} + +/// Deterministic pseudo-random bytes. +pub(crate) fn noise(len: usize, seed: u64) -> Vec { + let mut state = seed | 1; + let mut out = Vec::with_capacity(len); + while out.len() < len { + state ^= state << 13; + state ^= state >> 7; + state ^= state << 17; + out.extend_from_slice(&state.to_le_bytes()); + } + out.truncate(len); + out +} + +// --------------------------------------------------------------------------- +// Resolver and client +// --------------------------------------------------------------------------- + +/// Test resolver: each host answers from a script; the last answer repeats. +#[derive(Default)] +pub(crate) struct ScriptedLookup { + answers: std::sync::Mutex>>>, + pub(crate) calls: AtomicUsize, +} + +impl ScriptedLookup { + pub(crate) fn with(entries: &[(&str, &[&[&str]])]) -> Arc { + let lookup = Self::default(); + { + let mut answers = lookup.answers.lock().unwrap(); + for (host, script) in entries { + answers.insert( + host.to_string(), + script + .iter() + .map(|answer| { + answer + .iter() + .map(|a| a.parse::().unwrap()) + .collect() + }) + .collect(), + ); + } + } + Arc::new(lookup) + } +} + +impl HostLookup for ScriptedLookup { + fn lookup(&self, host: &str) -> LookupFuture { + self.calls.fetch_add(1, Ordering::SeqCst); + let mut answers = self.answers.lock().unwrap(); + let answer = match answers.get_mut(host) { + Some(script) if script.len() > 1 => Ok(script.pop_front().unwrap()), + Some(script) => Ok(script.front().cloned().unwrap_or_default()), + None => Err(std::io::Error::new( + std::io::ErrorKind::NotFound, + format!("unknown test host {host}"), + )), + }; + Box::pin(async move { answer }) + } +} + +pub(crate) const TEST_HTTP: SafeHttpConfig = SafeHttpConfig { + connect_timeout: Duration::from_secs(5), + read_timeout: Duration::from_secs(5), + total_timeout: Some(Duration::from_secs(20)), + max_redirects: 3, +}; + +pub(crate) fn test_http(lookup: Arc) -> SafeHttp { + SafeHttp::with_lookup(TEST_HTTP, AddressPolicy::test_allow_loopback(), lookup).unwrap() +} diff --git a/src/podcasts/tests.rs b/src/podcasts/tests.rs new file mode 100644 index 0000000..db90aa0 --- /dev/null +++ b/src/podcasts/tests.rs @@ -0,0 +1,81 @@ +//! Pure unit tests of the podcast store helpers (no database). + +use music_dht::podcast::ProgressStatus; + +use super::*; + +#[test] +fn search_text_never_matches_across_title_and_description() { + let norm = episode_search_norm("Hello, World!", Some("WORLD peace")); + assert_eq!(norm, "hello world\nworld peace"); + // A needle is itself normalized, so it never contains the separator. + let needle = music_dht::normalize_name("world world"); + assert!(!norm.contains(&needle)); + assert!(norm.contains(&music_dht::normalize_name("WORLD PEACE"))); + assert_eq!(episode_search_norm("Only", None), "only\n"); +} + +#[test] +fn progress_status_round_trips_through_sql() { + for status in [ + ProgressStatus::NotStarted, + ProgressStatus::InProgress, + ProgressStatus::Completed, + ] { + assert_eq!(status_from_sql(status_to_sql(status)), status); + } + // The CHECK constraint admits exactly these values. + for sql in schema::CREATE_PODCAST_STORE { + if sql.contains("CREATE TABLE IF NOT EXISTS furumusic__podcast_progress (") { + for value in ["not_started", "in_progress", "completed"] { + assert!(sql.contains(&format!("'{value}'"))); + } + } + } +} + +#[test] +fn generations_beyond_bigint_are_rejected() { + assert_eq!(generation_to_sql(0).unwrap(), 0); + assert_eq!(generation_to_sql(i64::MAX as u64).unwrap(), i64::MAX); + assert!(matches!( + generation_to_sql(i64::MAX as u64 + 1), + Err(PodcastStoreError::Invalid(_)) + )); + assert_eq!(generation_from_sql(-1), 0); +} + +#[test] +fn contract_numbers_outside_u32_are_dropped() { + assert_eq!(u32_from_sql(Some(7)), Some(7)); + assert_eq!(u32_from_sql(Some(-1)), None); + assert_eq!(u32_from_sql(Some(i64::from(u32::MAX) + 1)), None); + assert_eq!(u32_from_sql(None), None); +} + +#[test] +fn progress_columns_are_prefixed() { + assert_eq!( + progress_columns("g", "g_"), + "g.episode_id AS g_episode_id, g.podcast_id AS g_podcast_id, \ + g.position_ms AS g_position_ms, g.status AS g_status, \ + g.generation AS g_generation, g.session_id AS g_session_id, \ + g.hlc_ms AS g_hlc_ms, g.device_id AS g_device_id, g.op_id AS g_op_id" + ); +} + +#[test] +fn queries_sort_undated_and_unnumbered_episodes_last() { + for sort in [ + EpisodeSort::NewestFirst, + EpisodeSort::OldestFirst, + EpisodeSort::EpisodeNumber, + EpisodeSort::EpisodeNumberDesc, + ] { + let order = queries::order_by_for_tests(sort); + assert!( + order.starts_with("e.published_at_ms IS NULL") || order.starts_with("e.season IS NULL") + ); + assert!(order.ends_with("e.episode_id"), "{order}"); + } +} diff --git a/src/scheduler/mod.rs b/src/scheduler/mod.rs index 253c019..5294eef 100644 --- a/src/scheduler/mod.rs +++ b/src/scheduler/mod.rs @@ -1311,6 +1311,7 @@ async fn run_scheduled_job( && job_name != "lastfm_scrobble" && job_name != "archive_cleanup" && job_name != "artwork_backfill" + && job_name != "podcast_refresh" { tracing::warn!(job = job_name, "Skipping: agent_enabled=false"); return; diff --git a/templates/admin/responsive.html b/templates/admin/responsive.html index fe31253..109d03a 100644 --- a/templates/admin/responsive.html +++ b/templates/admin/responsive.html @@ -70,7 +70,7 @@ html { overscroll-behavior:none; } .task-form-body { overflow:visible; } .run-log-body { min-height:300px; overflow:visible; } .run-log-output { min-height:50dvh; max-height:70dvh; } - .settings-layout { grid-template-columns:minmax(0,1fr); grid-template-areas:"access" "oidc" "agent" "agentstatus" "similarity" "similaritystatus" "downloads" "federation" "lastfm" "developer" "actions"; } + .settings-layout { grid-template-columns:minmax(0,1fr); grid-template-areas:"access" "oidc" "agent" "agentstatus" "similarity" "similaritystatus" "downloads" "federation" "podcasts" "lastfm" "developer" "actions"; } .settings-federation-body { grid-template-columns:minmax(0,1fr); } .repair-shell { grid-template-columns:250px minmax(0,1fr); } } diff --git a/templates/admin/styles.html b/templates/admin/styles.html index fd63b58..87e96f6 100644 --- a/templates/admin/styles.html +++ b/templates/admin/styles.html @@ -859,6 +859,7 @@ tbody tr:hover { "similarity similarity similarity similarity similarity similarity similarity similarity similaritystatus similaritystatus similaritystatus similaritystatus" "downloads downloads downloads downloads downloads downloads downloads downloads downloads downloads downloads downloads" "federation federation federation federation federation federation federation federation federation federation federation federation" + "podcasts podcasts podcasts podcasts podcasts podcasts podcasts podcasts podcasts podcasts podcasts podcasts" "lastfm lastfm lastfm lastfm lastfm lastfm lastfm lastfm developer developer developer developer" "actions actions actions actions actions actions actions actions actions actions actions actions"; gap: 16px; @@ -881,6 +882,7 @@ tbody tr:hover { .settings-similarity-status { grid-area: similaritystatus; } .settings-downloads { grid-area: downloads; } .settings-federation { grid-area: federation; } +.settings-podcasts { grid-area: podcasts; } .settings-lastfm { grid-area: lastfm; } .settings-developer { grid-area: developer; } .settings-section-narrow { border-left: 2px solid rgba(29, 185, 84, 0.55); } diff --git a/templates/admin/v2.html b/templates/admin/v2.html index f98e866..025da73 100644 --- a/templates/admin/v2.html +++ b/templates/admin/v2.html @@ -1167,6 +1167,27 @@ +
+
+
+ Podcasts + Episode audio comes from the publisher +
+
+
+
+ +
+ + +
+
By default the browser loads episode audio directly from the publisher. Enable the proxy for publishers the browser cannot play directly (missing CORS, plain HTTP on an HTTPS site). The server then fetches the audio from the publisher on demand and passes it through without storing it; it only connects to public addresses.
+
+
+
@@ -2179,7 +2200,8 @@ function adminV2() { download_proxies: [], torrent_proxy_id: '', youtube_proxy_id: '', - youtube_cookie_id: '' + youtube_cookie_id: '', + podcast_audio_proxy_enabled: false }, settingsProbe: { status: 'idle', ok: false }, settingsProbeLoading: false, diff --git a/templates/player/modals.html b/templates/player/modals.html index 467d002..cc5bd89 100644 --- a/templates/player/modals.html +++ b/templates/player/modals.html @@ -48,6 +48,114 @@
+ + + + + + +
+

{{ t.player_podcast_found_title }}

+
+ +
+
+
{{ t.player_podcast_found_failed }}
+ + + + + + +