Files
furumi-web/docs/development.md
T
sanapachandClaude Opus 5.5 767134d1dd Keep replica room for the cards a rebalance publishes
Replicas count unexpired tombstones toward the 256 records per
publisher and refuse new keys beyond it. A round now admits at most as
many podcasts that need a new key as that room allows, so a fair-share
rebalance never displaces cards for newcomers the replicas would refuse.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 15:12:52 +02:00

399 lines
22 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Development
[Project overview](../README.md) · [Installation](setup.md) · [Admin UI conventions](admin-ui.md)
## Architecture
Furumusic is written in Rust on the
[Cot](https://cot.rs) web framework. PostgreSQL stores the catalog, accounts,
playlists, configuration, and background-job state. Audio inspection uses
Symphonia, torrent downloads use librqbit, and the shared `music-dht`/Frid
protocol stack provides decentralized catalog search, media transfer, and
connected-device synchronization.
The browser interface and JSON API are served by the same application. Import,
artwork, metadata enrichment, similarity indexing, and maintenance run as
durable background jobs rather than blocking playback requests.
## Build profiles
Development and test builds use basic optimization (`opt-level = 1`) for
dependencies to keep macOS unwind tables below the compact-unwind size limit.
Application code stays unoptimized; debug symbols, assertions and panic
unwinding remain enabled. The first build compiles dependencies with these
settings, then reuses them from Cargo's cache. Dependency stepping in a debugger
can be less exact because of optimization. Release settings are independent.
See the [Cargo profile override documentation](https://doc.rust-lang.org/cargo/reference/profiles.html#overrides) for dependency-specific settings.
## Container builds and CI
CI uses Rust 1.97.0 and cargo-chef 0.1.74 to cache compiled dependencies separately
from application sources. Buildx imports and exports intermediate layers through
`ultradesu/furumusic:buildcache` (`mode=max`). After the first build, changes to
source code, templates or prompts reuse the dependency layer when its recipe and
toolchain are unchanged. Both dependency and application builds use `--locked`.
Update the pinned build tools in `Dockerfile`; changing them rebuilds the cache.
## Connected devices and podcast queue items
Trusted-device sync uses the shared `music_dht::device_sync` types (no local
copies) and announces protocol version 4. Each account is one virtual device:
- podcast operations stay in the operation log and are relayed to v4 devices
only. They are also applied to the account's podcast store
(`src/podcasts/`, `PodcastStore`), which also provides the podcast snapshot
sections;
- queue items that are not library tracks become external browser items
(`source` plus the verbatim `wire_track`), so podcast episodes and Jam
placeholders survive peer → server → browser → server → peer unchanged;
- `furumusic__playback_state.queue_items_json` stores the typed queue next to
`queue_json`. `queue_json` (library ids) is still written for rollback; the
typed value is used only while the legacy columns it recorded
(`queue_json`, `current_track_id`, `queue_position`) match the row. Jam
placeholders are never stored in a user's own queue.
- a browser that selects an item it cannot play (a Jam placeholder), or an
episode whose resume position the server has not sent yet, reports
`position_unknown`, and the server sends the resume position of the
episode's progress register instead of 0. Outgoing playback states raise
each episode's `progress_generation` to the register generation, so the
device that takes over writes above it. Jam participant commands carry
`jam_participant`; the host browser checks them against its own queue.
Rolling back to 0.12 needs the podcast operations removed from the operation
log first; see "Rolling back to 0.12" in `administration.md`.
### Podcast store
`PodcastStore` implements the podcast contract of `music_dht::podcast`
(Frid `PODCASTS.md`) on PostgreSQL, with the invariants of the SQLite
reference store in `furumi-library`; every table has a `user_id` key.
| Table | Content | Synced |
| --- | --- | --- |
| `furumusic__podcast` | channel metadata, fetch URL | no (catalog) |
| `furumusic__podcast_feed_alias` | bounded grow-only alias set | in subscription records and `podcast_feed_aliases_added` |
| `furumusic__podcast_episode` | `EpisodeMeta`, `source` `feed`/`sync`, `in_feed`, cached search text | no (catalog) |
| `furumusic__podcast_season` | view over the episodes | no |
| `furumusic__podcast_subscription`, `_share`, `_progress` | registers | yes |
| `furumusic__podcast_listen_session` | grow-only set of sessions | yes |
| `furumusic__podcast_fetch_state` | ETag, Last-Modified, check times, errors, cover cache | never |
- One apply path: received operations, snapshot records and local actions go
through the `SyncOpWire` materializers and the `merge_*` functions of
`music_dht::podcast`. Each write transaction holds a per-account advisory
lock.
- Local actions (`subscribe`, `unsubscribe`, `set_shared`, `start_session`,
`checkpoint`, `seek`, `complete_if_current`, `restart`, `mark_played`,
`mark_unplayed`, `record_listen_session`) append their operation to the
account's sync log with the next stamp (`devices::append_local_op`: `seq`,
`hlc_ms`, `op_id = <device>:<seq>`) and apply exactly that operation in
the same transaction.
- `apply_feed` resolves episode ids with the shared no-GUID rule against the
stored episodes, including those only known from synced references. It
updates the catalog and the alias set, never a register, progress or a
session.
- Snapshots export each section newest first up to its count and bound it
with `bound_podcast_state_to`. The catalog and the fetch state are never
exported.
Migrations `m_0056_create_podcast_store` and `m_0057_import_podcast_relay`
create the tables and import what 0.13.0 kept in `furumusic__fed_podcast_relay`
(registers as contract JSON, already merged) plus the catalog rows of the
episode references in the operation log. The relay table is no longer written
or read; it stays until a later cleanup release.
### Feed refresh and audio delivery
The runtime around the store lives next to it in `src/podcasts/`:
- `net.rs` — `SafeHttp`, the only HTTP client for podcast feeds and audio.
It accepts `http`/`https` URLs without userinfo, resolves names with
`GuardedResolver` and refuses the whole answer when any address is not
public (loopback, RFC 1918, CGNAT, link-local, cloud metadata, unique local,
multicast, unspecified, documentation, reserved; IPv4-mapped, NAT64 and
6to4 addresses are judged by their IPv4 part). The connector connects to
exactly the checked addresses, so a second DNS answer cannot be swapped in.
IP literals are checked before connecting, redirects are followed manually
(each hop checked, at most 5), and system proxies, cookies and `Referer`
are disabled. `GuardError::is_refusal` is true only for what the URL
shows (scheme, credentials, port, a blocked IP literal); a name that
resolves to a blocked address is reported to users like a transport
failure, so the answers do not reveal internal DNS names.
- `fetch.rs` — `FeedFetcher`: conditional GET (`If-None-Match`,
`If-Modified-Since`), `Accept-Encoding: identity`, 32 MiB and 60 s limits,
`Retry-After` on 429/503, the shared `music_dht` parser. Only a chain of
permanent redirects (301/308) moves a feed.
- `runner.rs` — scheduling: `podcasts_due_for_refresh`, one request per feed
URL for all accounts subscribed to it (conditional only when they share the
same validators), `apply_feed`, then the fetch state. Healthy feeds are
checked hourly plus a stable spread of up to 10 minutes; failures back off
15 min × 2ⁿ⁻¹ up to a day (a longer `Retry-After` wins). `refresh_now` is
the manual refresh: subscribed podcasts only, at most every 30 s.
`jobs/podcast_refresh.rs` runs `refresh_due` every five minutes (200
podcasts taken in turns across accounts without a per-account cap, 4
requests in flight, one run at a time via an advisory lock).
- `proxy.rs` and `http.rs` — the optional audio proxy and the endpoints under
`/api/podcasts`. The proxy takes an episode id, loads the URL from the
account's catalog (`PodcastStore::audio_source`), forwards a single
sanitized byte `Range`, passes 200/206/416 through, accepts audio content
types only (a generic binary type needs an audio URL extension or a file
that starts like audio, `sniff_audio`: read from the response when it
starts at byte 0 and has at least 16 bytes, otherwise with its own
`bytes=0-15` request; the feed's enclosure type is never trusted), and
stops at 1 GiB, 20 s without headers, 30 s
without data or 4 h per response. It is off unless
`podcast_audio_proxy_enabled` is set; by default the browser streams from
the publisher URL.
### Podcast UI and playback
`api.rs` holds what the endpoints do, independent of HTTP (`http.rs` only
authenticates and maps statuses), so the Postgres tests drive it directly:
| Endpoint (`/api/podcasts`) | Does |
| --- | --- |
| `GET /feeds[?all=1]`, `POST /feeds {url}` | list; subscribe after fetching the feed (`runner::subscribe_url`) |
| `GET /feeds/{id}`, `GET /feeds/{id}/episodes?season=&played=&q=&sort=&limit=&offset=` | podcast, seasons, history; episode pages with the browser queue item |
| `POST /feeds/{id}/subscribe`, `/unsubscribe`, `POST /{id}/refresh` | subscription and manual refresh |
| `POST /episodes/{id}/restart`, `/played`, `/unplayed` | explicit progress actions |
| `POST /episodes/{id}/playback` | source, resume position; with `start` a new session (explicit, above `carried_generation`); records an unknown handed-off reference first |
| `POST /episodes/{id}/progress` | `checkpoint`/`pause`/`stop`/`handoff` at the session generation, `seek` explicit, completion once (`ended` or `reaches_completion`), listening session record |
Request bodies must be `application/json` (415 otherwise, like cot's `Json`
extractor): other types need no CORS preflight and would let a same-site page
post with the session cookie.
In the browser (`templates/player/scripts.html`), `_playEpisodeLocal` asks
for the playback, loads the publisher URL (or the proxy: mixed content, or a
failed direct load) and seeks before playing; when the source cannot seek to
the saved position it shows that instead of playing from elsewhere. Only a
handoff or the user's own seek sends a position (`explicitPosition`); an
ordinary Play, including after a reload, resumes from the register, and Play
after a pause first checks that the register is still this session's
(`register_session_id`). A seek made while the source is being prepared is
applied when it arrives. Writes of one session run in order
(`_reportEpisode`), read the session generation at send time and carry the
position of their own episode. After `superseded`, checkpoints stop; if the
user keeps listening here, the next write (one checkpoint interval later, or
at once on pause, stop or the end of the stream) records the old session and
starts a new explicit one (`_takeOverEpisode`). A browser that loses the
active role sends one final `stop` at its session generation (it records the
listening; the new device's start keeps the register), unless it handed off
itself and the transfer went through (a failed transfer clears `handedOff` in
the devices store's `select`). Preparing the same episode again (Play after
its source failed) waits for the old session's final report, so the server
resumes from where the audio stopped. Marking the loaded episode played or unplayed in the same tab saves
and closes its session and pauses it. Prev on an episode goes back 15 s
locally instead of resetting it. Jam guests never play or save episodes
(they only ever see placeholders). The `podcasts` store backs the Podcasts
views in `shell.html`.
Tests never reach the internet: `test_server.rs` serves on 127.0.0.1 and
provides a scripted resolver. Loopback is reachable only through
`AddressPolicy::test_allow_loopback`, which exists only under `#[cfg(test)]`.
Everything sent to a device is gated by the version that device reported itself
(`profile_protocol_version`; relayed profiles only raise it). See the podcast
contract (`PODCASTS.md`) and `PLAYBACK_PROTOCOL.md` in the Frid repository.
```bash
cargo test devices:: queue_items podcasts:: queue_persistence
node --test tests/player_queue_sources.test.cjs tests/player_podcasts.test.cjs
# Disposable PostgreSQL; the tests run the production migrations on it, and
# the migration test creates and drops a database next to it:
FURU_DEVICE_SYNC_TEST_DATABASE_URL=postgresql://... \
cargo test postgres_tests -- --ignored
# Sharing, aggregation and a loopback publication from the web node to
# another iroh node (temporary directories):
FURU_DEVICE_SYNC_TEST_DATABASE_URL=postgresql://... \
cargo test discovery -- --include-ignored
FURU_METADATA_FIX_TEST_DATABASE_URL=postgresql://... \
cargo test undo_accepts_playback_archives_written_before_the_typed_queue -- --ignored
```
`wire_compat_tests` compares the shared types with the private copies of
0.12.x byte for byte; `fixtures/v3_hello.golden.json` is the v3 shape. Rewrite
it only for a deliberate change, with `FURUMUSIC_BLESS_GOLDEN=1`.
### Public podcast discovery
`src/podcasts/discovery.rs` decides what the server node publishes on the
`music_dht::podcast_dht` overlay; `src/federation/podcast_discovery.rs` runs
the overlay on the federation node (ALPN `furumi-fd/podcast-dht/1`, cache
`podcast-dht.sqlite3` in the federation data directory, `serve`,
`maintenance`, the reconciler and a capability probe that calls
`note_peer_capabilities`). The capability manifest advertises `podcast_dht`
only while the overlay runs.
- The server's node key is the only publisher. Each account's
`AccountShare` is built like the frid integration pattern for SQLite
clients: the merged subscription register (with its aliases) and share
register, and a card from `PodcastCardSpec::from_subscription` with the
account's catalog fetch URL as `feed_url`, the origin URL as `id_feed_url`
and the feed's episode count and newest date.
- `load_shared` reads active accounts only (`furumusic__user.is_active`).
An `(account, podcast)` whose stored data does not decode (a URL, the
register JSON, an id) is left out and counted (`unreadable`): only a
failing connection or query skips the round, so revocation stays
fail-safe for each podcast on its own.
- `aggregate_share_states` keeps the accounts for which `effective_share`
holds and whose card has no privacy warnings or a matching confirmation,
and builds one `PodcastShareState` per podcast with
`privacy_confirmed = true`. Representative: no warnings, then a catalog
from the server's own fetch (`meta_from_feed`), then the previous
round's representative (kept in memory), then the smallest account id.
- At most 192 podcasts (replicas keep 256 records per publisher, tombstones
included), allocated round-robin over the sharing accounts (`allocate`):
each account's queue holds its podcasts the node already publishes
first, then by its own share time; in every round each account takes its
next podcast not taken yet, published ones first inside a round. An
account's share times therefore only order its own podcasts. A podcast
without a live record of the node (an active card or an unexpired
tombstone, `Incumbents::live_keys`) needs a new replica key; at most
`256 - live keys` of them are taken per round (`Incumbents::key_headroom`),
the others wait in `over_limit`, so a rebalance never displaces more
cards than replicas accept new ones for.
`PodcastDht::sync_shares` publishes the result and tombstones everything
else. The podcasts left out (`over_limit`) and those `sync_shares`
reported as failed are kept in memory for `ShareView.held`.
- `furumusic__podcast_share_confirmation` (`m_0058`) is the client-held
`privacy_confirmed`: per account and podcast, bound to the share
register's `op_id` and the exact card URLs. Never synced.
- The reconciler runs on `discovery::request_reconcile()` (after local share,
subscribe and unsubscribe actions, received subscription/share/alias
operations and snapshot sections, refreshes of shared podcasts; 2 s
debounce) and every five minutes. Connection and query errors skip a
round. Only counts are logged.
- `ShareView` (in `GET /api/podcasts/feeds/{id}` and the share response):
`discovery` is `running`, `unavailable` (the node runs, the overlay did
not start; `Federation::podcast_discovery_unavailable`) or `off`;
`published` is this server node's own active card only; `held` is
`over_limit` or `failed` after the last round. An unreadable overlay
cache degrades to `published: false` instead of failing the page.
- `POST /api/podcasts/feeds/{id}/share` requires a JSON body
(`{"shared": bool, "confirm"?: {"feed_url", "id_feed_url"}}`); a card with
warnings is shared only with `confirm` naming exactly its current URLs
(otherwise 409 `needs_confirmation` with the warnings, nothing written).
Search and subscription from a card (`src/podcasts/search.rs`):
- `GET /api/player/federation/search/events` starts `PodcastDht::search`
(`Federation::spawn_podcast_search`, at most 20 hits, 20 s) next to the
music DHT search. Music events stream first and are unchanged
(`federation.track|artist|release`); every event now carries `"type":
"music"` or `"type": "podcast"`. Podcast hits follow as
`federation.podcast` events with a `PodcastHitView` entity: the card's
public fields, the origin `feed_url` (never `claimed_feed_urls`),
`publisher_count`, `rank` and `exact_match` from `merge_search_results`
with the music items, and the searching account's `subscribed` /
`local_podcast_id` (by id, or a known feed URL or alias through
`resolve_podcast_id`). `search.completed` adds `podcast_discovery`
(`running`, `off`, `failed`); a failed podcast search, or a failure to
prepare its hits for the account (`Federation::podcast_search_events`
never fails), only drops the podcast hits and reports `failed`.
- `POST /api/podcasts/discovery/preview` and `/discovery/subscribe` take
`{"podcast_id", "feed_url"}` from a hit and refuse a URL the id does not
hash from (422). The preview (optional `"season"`: a number as a
string, `"none"` or `"all"`) fetches the feed through the SSRF-guarded
fetcher and returns the channel, season counts, `listed_count` and the
newest 50 episodes of the season, without storing anything; subscribe
fetches the origin URL, then `runner::subscribe_fetched` writes a
private subscription with the card's id, or the account's existing
podcast for the first known URL of `runner::document_urls` (the origin,
the end of a permanent redirect chain, the feed's `itunes:new-feed-url`;
also used by "add RSS"). Nothing is written when the account is already
subscribed, by id or known URL, before or after the fetch. Both share
the "add RSS" request limits; the browser aborts a preview it no longer
shows (`AbortController`), so a closed card frees its slot once the
server sees the connection close.
- The browser keeps only expected, typed card fields (`podcasts.foundHit`):
ids must look like podcast ids, URLs must be absolute `http(s)` without
credentials (artwork: `https` only), and texts render with `x-text`.
## Contributing
Bug reports, design discussions, and patches are welcome. Before submitting a
change, run:
```bash
cargo fmt --all -- --check
cargo clippy --all-targets -- -D warnings
cargo test --all-targets
```
Last.fm database tests use temporary tables on a disposable PostgreSQL database:
```bash
FURUMUSIC_LASTFM_TEST_DATABASE_URL=postgres://user:pass@localhost/test_db \
cargo test lastfm_popularity -- --include-ignored
```
## Access and sharing tests
The access regression suite uses a disposable migrated database and a local OIDC
provider with signed tokens and PKCE:
```bash
node --test tests/*.test.cjs
FURU_ACCESS_TEST_DATABASE_URL=postgresql://... cargo test postgres_approval_survives_sso_changes_without_implicit_grants -- --ignored
# Fresh disposable instance on :18048, media root /tmp/furumi-access-media,
# PostgreSQL container furumi-access-test-pg, database furumi_access_test:
node tests/browser-access-sharing.cjs
```
The browser test requires Playwright (`PLAYWRIGHT_MODULE`) and Chrome
(`CHROME_PATH`), and uses local port 18049 for its OIDC provider. It checks
approval, SSO precedence and group removal, guest playback and downloads,
unauthorized track substitution, and desktop/mobile player behavior.
## Statistics tests
```bash
node --test tests/*.test.cjs
cargo test admin:: -- --skip postgres_statistics_drilldown_and_pagination
# Use a disposable PostgreSQL; this test creates connection-local temporary tables.
FURU_STATISTICS_TEST_DATABASE_URL=postgresql://... cargo test postgres_statistics_drilldown_and_pagination -- --ignored
```
`tests/browser-statistics.cjs` additionally exercises a fresh disposable instance
at `FURU_TEST_URL` (default `http://127.0.0.1:3018`) and seeds the disposable Docker
container `furumi-statistics-test-pg`. It requires Playwright (or
`PLAYWRIGHT_MODULE`) and Chrome (or `CHROME_PATH`). Never point this test at a
live service.
## Metadata Repair tests
```bash
node --test tests/admin_metadata_repair.test.cjs
cargo test admin::metadata_fix::tests
```
`tests/browser-metadata-repair.cjs` checks the chat and plan updates, a four-release
merge, deletion and restoration of tracks and artist credits, playlists, playback
queues, and undo conflicts. Run it only against a disposable instance and a
PostgreSQL container named `furumi-repair-test-pg`, with database
`furumi_repair_test`. Set `FURU_TEST_URL`, `FURU_REPAIR_TEST_PG`, `CHROME_PATH`, and
`PLAYWRIGHT_MODULE` as needed. It seeds `tests/metadata_repair_fixture.sql` after
creating the test administrator. Set `FURU_REPAIR_SKIP_SEED=1` for a seeded database.
With `FURU_REPAIR_TEST_MODEL=1`, it additionally checks the reported release-only
selection and full ordered tracklist against the configured model, applies and
undoes that plan, and verifies individual track exclusions.
## Import destination tests
```bash
node --test tests/import_playlists.test.cjs
# Disposable PostgreSQL; both tests use temporary tables.
FURU_IMPORT_TEST_DATABASE_URL=postgresql://... cargo test import_destinations -- --ignored
FURU_IMPORT_TEST_DATABASE_URL=postgresql://... cargo test torrent_playlist -- --ignored
```
`tests/browser-import-playlists.cjs` checks the file upload flow, database
reconciliation and the torrent picker on desktop and mobile. It requires a fresh
migrated instance with the import agent disabled at `FURU_TEST_URL` (default
`http://127.0.0.1:3019`), the disposable PostgreSQL container
`furumi-import-playlist-test-pg` with database `furumi_import_test`, and Playwright
and Chrome configured as above. It uses a local torrent preview fixture and
simulated imported tracks, without downloading external torrents.