300 lines
16 KiB
Markdown
300 lines
16 KiB
Markdown
# Development
|
||
|
||
[Project overview](../README.md) · [Installation](setup.md) · [Admin UI conventions](admin-ui.md)
|
||
|
||
## Architecture
|
||
|
||
Furumusic is written in Rust on the
|
||
[Cot](https://cot.rs) web framework. PostgreSQL stores the catalog, accounts,
|
||
playlists, configuration, and background-job state. Audio inspection uses
|
||
Symphonia, torrent downloads use librqbit, and the shared `music-dht`/Frid
|
||
protocol stack provides decentralized catalog search, media transfer, and
|
||
connected-device synchronization.
|
||
|
||
The browser interface and JSON API are served by the same application. Import,
|
||
artwork, metadata enrichment, similarity indexing, and maintenance run as
|
||
durable background jobs rather than blocking playback requests.
|
||
|
||
## Build profiles
|
||
|
||
Development and test builds use basic optimization (`opt-level = 1`) for
|
||
dependencies to keep macOS unwind tables below the compact-unwind size limit.
|
||
Application code stays unoptimized; debug symbols, assertions and panic
|
||
unwinding remain enabled. The first build compiles dependencies with these
|
||
settings, then reuses them from Cargo's cache. Dependency stepping in a debugger
|
||
can be less exact because of optimization. Release settings are independent.
|
||
|
||
See the [Cargo profile override documentation](https://doc.rust-lang.org/cargo/reference/profiles.html#overrides) for dependency-specific settings.
|
||
|
||
## Container builds and CI
|
||
|
||
CI uses Rust 1.97.0 and cargo-chef 0.1.74 to cache compiled dependencies separately
|
||
from application sources. Buildx imports and exports intermediate layers through
|
||
`ultradesu/furumusic:buildcache` (`mode=max`). After the first build, changes to
|
||
source code, templates or prompts reuse the dependency layer when its recipe and
|
||
toolchain are unchanged. Both dependency and application builds use `--locked`.
|
||
Update the pinned build tools in `Dockerfile`; changing them rebuilds the cache.
|
||
|
||
## Connected devices and podcast queue items
|
||
|
||
Trusted-device sync uses the shared `music_dht::device_sync` types (no local
|
||
copies) and announces protocol version 4. Each account is one virtual device:
|
||
|
||
- podcast operations stay in the operation log and are relayed to v4 devices
|
||
only. They are also applied to the account's podcast store
|
||
(`src/podcasts/`, `PodcastStore`), which also provides the podcast snapshot
|
||
sections;
|
||
- queue items that are not library tracks become external browser items
|
||
(`source` plus the verbatim `wire_track`), so podcast episodes and Jam
|
||
placeholders survive peer → server → browser → server → peer unchanged;
|
||
- `furumusic__playback_state.queue_items_json` stores the typed queue next to
|
||
`queue_json`. `queue_json` (library ids) is still written for rollback; the
|
||
typed value is used only while the legacy columns it recorded
|
||
(`queue_json`, `current_track_id`, `queue_position`) match the row. Jam
|
||
placeholders are never stored in a user's own queue.
|
||
- a browser that selects an item it cannot play (a Jam placeholder), or an
|
||
episode whose resume position the server has not sent yet, reports
|
||
`position_unknown`, and the server sends the resume position of the
|
||
episode's progress register instead of 0. Outgoing playback states raise
|
||
each episode's `progress_generation` to the register generation, so the
|
||
device that takes over writes above it. Jam participant commands carry
|
||
`jam_participant`; the host browser checks them against its own queue.
|
||
|
||
Rolling back to 0.12 needs the podcast operations removed from the operation
|
||
log first; see "Rolling back to 0.12" in `administration.md`.
|
||
|
||
### Podcast store
|
||
|
||
`PodcastStore` implements the podcast contract of `music_dht::podcast`
|
||
(Frid `PODCASTS.md`) on PostgreSQL, with the invariants of the SQLite
|
||
reference store in `furumi-library`; every table has a `user_id` key.
|
||
|
||
| Table | Content | Synced |
|
||
| --- | --- | --- |
|
||
| `furumusic__podcast` | channel metadata, fetch URL | no (catalog) |
|
||
| `furumusic__podcast_feed_alias` | bounded grow-only alias set | in subscription records and `podcast_feed_aliases_added` |
|
||
| `furumusic__podcast_episode` | `EpisodeMeta`, `source` `feed`/`sync`, `in_feed`, cached search text | no (catalog) |
|
||
| `furumusic__podcast_season` | view over the episodes | no |
|
||
| `furumusic__podcast_subscription`, `_share`, `_progress` | registers | yes |
|
||
| `furumusic__podcast_listen_session` | grow-only set of sessions | yes |
|
||
| `furumusic__podcast_fetch_state` | ETag, Last-Modified, check times, errors, cover cache | never |
|
||
|
||
- One apply path: received operations, snapshot records and local actions go
|
||
through the `SyncOpWire` materializers and the `merge_*` functions of
|
||
`music_dht::podcast`. Each write transaction holds a per-account advisory
|
||
lock.
|
||
- Local actions (`subscribe`, `unsubscribe`, `set_shared`, `start_session`,
|
||
`checkpoint`, `seek`, `complete_if_current`, `restart`, `mark_played`,
|
||
`mark_unplayed`, `record_listen_session`) append their operation to the
|
||
account's sync log with the next stamp (`devices::append_local_op`: `seq`,
|
||
`hlc_ms`, `op_id = <device>:<seq>`) and apply exactly that operation in
|
||
the same transaction.
|
||
- `apply_feed` resolves episode ids with the shared no-GUID rule against the
|
||
stored episodes, including those only known from synced references. It
|
||
updates the catalog and the alias set, never a register, progress or a
|
||
session.
|
||
- Snapshots export each section newest first up to its count and bound it
|
||
with `bound_podcast_state_to`. The catalog and the fetch state are never
|
||
exported.
|
||
|
||
Migrations `m_0056_create_podcast_store` and `m_0057_import_podcast_relay`
|
||
create the tables and import what 0.13.0 kept in `furumusic__fed_podcast_relay`
|
||
(registers as contract JSON, already merged) plus the catalog rows of the
|
||
episode references in the operation log. The relay table is no longer written
|
||
or read; it stays until a later cleanup release.
|
||
|
||
### Feed refresh and audio delivery
|
||
|
||
The runtime around the store lives next to it in `src/podcasts/`:
|
||
|
||
- `net.rs` — `SafeHttp`, the only HTTP client for podcast feeds and audio.
|
||
It accepts `http`/`https` URLs without userinfo, resolves names with
|
||
`GuardedResolver` and refuses the whole answer when any address is not
|
||
public (loopback, RFC 1918, CGNAT, link-local, cloud metadata, unique local,
|
||
multicast, unspecified, documentation, reserved; IPv4-mapped, NAT64 and
|
||
6to4 addresses are judged by their IPv4 part). The connector connects to
|
||
exactly the checked addresses, so a second DNS answer cannot be swapped in.
|
||
IP literals are checked before connecting, redirects are followed manually
|
||
(each hop checked, at most 5), and system proxies, cookies and `Referer`
|
||
are disabled. `GuardError::is_refusal` is true only for what the URL
|
||
shows (scheme, credentials, port, a blocked IP literal); a name that
|
||
resolves to a blocked address is reported to users like a transport
|
||
failure, so the answers do not reveal internal DNS names.
|
||
- `fetch.rs` — `FeedFetcher`: conditional GET (`If-None-Match`,
|
||
`If-Modified-Since`), `Accept-Encoding: identity`, 32 MiB and 60 s limits,
|
||
`Retry-After` on 429/503, the shared `music_dht` parser. Only a chain of
|
||
permanent redirects (301/308) moves a feed.
|
||
- `runner.rs` — scheduling: `podcasts_due_for_refresh`, one request per feed
|
||
URL for all accounts subscribed to it (conditional only when they share the
|
||
same validators), `apply_feed`, then the fetch state. Healthy feeds are
|
||
checked hourly plus a stable spread of up to 10 minutes; failures back off
|
||
15 min × 2ⁿ⁻¹ up to a day (a longer `Retry-After` wins). `refresh_now` is
|
||
the manual refresh: subscribed podcasts only, at most every 30 s.
|
||
`jobs/podcast_refresh.rs` runs `refresh_due` every five minutes (200
|
||
podcasts taken in turns across accounts without a per-account cap, 4
|
||
requests in flight, one run at a time via an advisory lock).
|
||
- `proxy.rs` and `http.rs` — the optional audio proxy and the endpoints under
|
||
`/api/podcasts`. The proxy takes an episode id, loads the URL from the
|
||
account's catalog (`PodcastStore::audio_source`), forwards a single
|
||
sanitized byte `Range`, passes 200/206/416 through, accepts audio content
|
||
types only (a generic binary type needs an audio URL extension or a file
|
||
that starts like audio, `sniff_audio`: read from the response when it
|
||
starts at byte 0 and has at least 16 bytes, otherwise with its own
|
||
`bytes=0-15` request; the feed's enclosure type is never trusted), and
|
||
stops at 1 GiB, 20 s without headers, 30 s
|
||
without data or 4 h per response. It is off unless
|
||
`podcast_audio_proxy_enabled` is set; by default the browser streams from
|
||
the publisher URL.
|
||
|
||
### Podcast UI and playback
|
||
|
||
`api.rs` holds what the endpoints do, independent of HTTP (`http.rs` only
|
||
authenticates and maps statuses), so the Postgres tests drive it directly:
|
||
|
||
| Endpoint (`/api/podcasts`) | Does |
|
||
| --- | --- |
|
||
| `GET /feeds[?all=1]`, `POST /feeds {url}` | list; subscribe after fetching the feed (`runner::subscribe_url`) |
|
||
| `GET /feeds/{id}`, `GET /feeds/{id}/episodes?season=&played=&q=&sort=&limit=&offset=` | podcast, seasons, history; episode pages with the browser queue item |
|
||
| `POST /feeds/{id}/subscribe`, `/unsubscribe`, `POST /{id}/refresh` | subscription and manual refresh |
|
||
| `POST /episodes/{id}/restart`, `/played`, `/unplayed` | explicit progress actions |
|
||
| `POST /episodes/{id}/playback` | source, resume position; with `start` a new session (explicit, above `carried_generation`); records an unknown handed-off reference first |
|
||
| `POST /episodes/{id}/progress` | `checkpoint`/`pause`/`stop`/`handoff` at the session generation, `seek` explicit, completion once (`ended` or `reaches_completion`), listening session record |
|
||
|
||
Request bodies must be `application/json` (415 otherwise, like cot's `Json`
|
||
extractor): other types need no CORS preflight and would let a same-site page
|
||
post with the session cookie.
|
||
|
||
In the browser (`templates/player/scripts.html`), `_playEpisodeLocal` asks
|
||
for the playback, loads the publisher URL (or the proxy: mixed content, or a
|
||
failed direct load) and seeks before playing; when the source cannot seek to
|
||
the saved position it shows that instead of playing from elsewhere. Only a
|
||
handoff or the user's own seek sends a position (`explicitPosition`); an
|
||
ordinary Play, including after a reload, resumes from the register, and Play
|
||
after a pause first checks that the register is still this session's
|
||
(`register_session_id`). A seek made while the source is being prepared is
|
||
applied when it arrives. Writes of one session run in order
|
||
(`_reportEpisode`), read the session generation at send time and carry the
|
||
position of their own episode. After `superseded`, checkpoints stop; if the
|
||
user keeps listening here, the next write (one checkpoint interval later, or
|
||
at once on pause, stop or the end of the stream) records the old session and
|
||
starts a new explicit one (`_takeOverEpisode`). A browser that loses the
|
||
active role sends one final `stop` at its session generation (it records the
|
||
listening; the new device's start keeps the register), unless it handed off
|
||
itself and the transfer went through (a failed transfer clears `handedOff` in
|
||
the devices store's `select`). Preparing the same episode again (Play after
|
||
its source failed) waits for the old session's final report, so the server
|
||
resumes from where the audio stopped. Marking the loaded episode played or unplayed in the same tab saves
|
||
and closes its session and pauses it. Prev on an episode goes back 15 s
|
||
locally instead of resetting it. Jam guests never play or save episodes
|
||
(they only ever see placeholders). The `podcasts` store backs the Podcasts
|
||
views in `shell.html`.
|
||
|
||
Tests never reach the internet: `test_server.rs` serves on 127.0.0.1 and
|
||
provides a scripted resolver. Loopback is reachable only through
|
||
`AddressPolicy::test_allow_loopback`, which exists only under `#[cfg(test)]`.
|
||
|
||
Everything sent to a device is gated by the version that device reported itself
|
||
(`profile_protocol_version`; relayed profiles only raise it). See the podcast
|
||
contract (`PODCASTS.md`) and `PLAYBACK_PROTOCOL.md` in the Frid repository.
|
||
|
||
```bash
|
||
cargo test devices:: queue_items podcasts:: queue_persistence
|
||
node --test tests/player_queue_sources.test.cjs tests/player_podcasts.test.cjs
|
||
# Disposable PostgreSQL; the tests run the production migrations on it, and
|
||
# the migration test creates and drops a database next to it:
|
||
FURU_DEVICE_SYNC_TEST_DATABASE_URL=postgresql://... \
|
||
cargo test postgres_tests -- --ignored
|
||
FURU_METADATA_FIX_TEST_DATABASE_URL=postgresql://... \
|
||
cargo test undo_accepts_playback_archives_written_before_the_typed_queue -- --ignored
|
||
```
|
||
|
||
`wire_compat_tests` compares the shared types with the private copies of
|
||
0.12.x byte for byte; `fixtures/v3_hello.golden.json` is the v3 shape. Rewrite
|
||
it only for a deliberate change, with `FURUMUSIC_BLESS_GOLDEN=1`.
|
||
|
||
## Contributing
|
||
|
||
Bug reports, design discussions, and patches are welcome. Before submitting a
|
||
change, run:
|
||
|
||
```bash
|
||
cargo fmt --all -- --check
|
||
cargo clippy --all-targets -- -D warnings
|
||
cargo test --all-targets
|
||
```
|
||
|
||
Last.fm database tests use temporary tables on a disposable PostgreSQL database:
|
||
|
||
```bash
|
||
FURUMUSIC_LASTFM_TEST_DATABASE_URL=postgres://user:pass@localhost/test_db \
|
||
cargo test lastfm_popularity -- --include-ignored
|
||
```
|
||
|
||
## Access and sharing tests
|
||
|
||
The access regression suite uses a disposable migrated database and a local OIDC
|
||
provider with signed tokens and PKCE:
|
||
|
||
```bash
|
||
node --test tests/*.test.cjs
|
||
FURU_ACCESS_TEST_DATABASE_URL=postgresql://... cargo test postgres_approval_survives_sso_changes_without_implicit_grants -- --ignored
|
||
# Fresh disposable instance on :18048, media root /tmp/furumi-access-media,
|
||
# PostgreSQL container furumi-access-test-pg, database furumi_access_test:
|
||
node tests/browser-access-sharing.cjs
|
||
```
|
||
|
||
The browser test requires Playwright (`PLAYWRIGHT_MODULE`) and Chrome
|
||
(`CHROME_PATH`), and uses local port 18049 for its OIDC provider. It checks
|
||
approval, SSO precedence and group removal, guest playback and downloads,
|
||
unauthorized track substitution, and desktop/mobile player behavior.
|
||
|
||
## Statistics tests
|
||
|
||
```bash
|
||
node --test tests/*.test.cjs
|
||
cargo test admin:: -- --skip postgres_statistics_drilldown_and_pagination
|
||
# Use a disposable PostgreSQL; this test creates connection-local temporary tables.
|
||
FURU_STATISTICS_TEST_DATABASE_URL=postgresql://... cargo test postgres_statistics_drilldown_and_pagination -- --ignored
|
||
```
|
||
|
||
`tests/browser-statistics.cjs` additionally exercises a fresh disposable instance
|
||
at `FURU_TEST_URL` (default `http://127.0.0.1:3018`) and seeds the disposable Docker
|
||
container `furumi-statistics-test-pg`. It requires Playwright (or
|
||
`PLAYWRIGHT_MODULE`) and Chrome (or `CHROME_PATH`). Never point this test at a
|
||
live service.
|
||
|
||
## Metadata Repair tests
|
||
|
||
```bash
|
||
node --test tests/admin_metadata_repair.test.cjs
|
||
cargo test admin::metadata_fix::tests
|
||
```
|
||
|
||
`tests/browser-metadata-repair.cjs` checks the chat and plan updates, a four-release
|
||
merge, deletion and restoration of tracks and artist credits, playlists, playback
|
||
queues, and undo conflicts. Run it only against a disposable instance and a
|
||
PostgreSQL container named `furumi-repair-test-pg`, with database
|
||
`furumi_repair_test`. Set `FURU_TEST_URL`, `FURU_REPAIR_TEST_PG`, `CHROME_PATH`, and
|
||
`PLAYWRIGHT_MODULE` as needed. It seeds `tests/metadata_repair_fixture.sql` after
|
||
creating the test administrator. Set `FURU_REPAIR_SKIP_SEED=1` for a seeded database.
|
||
With `FURU_REPAIR_TEST_MODEL=1`, it additionally checks the reported release-only
|
||
selection and full ordered tracklist against the configured model, applies and
|
||
undoes that plan, and verifies individual track exclusions.
|
||
|
||
## Import destination tests
|
||
|
||
```bash
|
||
node --test tests/import_playlists.test.cjs
|
||
# Disposable PostgreSQL; both tests use temporary tables.
|
||
FURU_IMPORT_TEST_DATABASE_URL=postgresql://... cargo test import_destinations -- --ignored
|
||
FURU_IMPORT_TEST_DATABASE_URL=postgresql://... cargo test torrent_playlist -- --ignored
|
||
```
|
||
|
||
`tests/browser-import-playlists.cjs` checks the file upload flow, database
|
||
reconciliation and the torrent picker on desktop and mobile. It requires a fresh
|
||
migrated instance with the import agent disabled at `FURU_TEST_URL` (default
|
||
`http://127.0.0.1:3019`), the disposable PostgreSQL container
|
||
`furumi-import-playlist-test-pg` with database `furumi_import_test`, and Playwright
|
||
and Chrome configured as above. It uses a local torrent preview fixture and
|
||
simulated imported tracks, without downloading external torrents.
|