forked from Furumi/furumi-web
248 lines
12 KiB
Markdown
248 lines
12 KiB
Markdown
# Development
|
||
|
||
[Project overview](../README.md) · [Installation](setup.md) · [Admin UI conventions](admin-ui.md)
|
||
|
||
## Architecture
|
||
|
||
Furumusic is written in Rust on the
|
||
[Cot](https://cot.rs) web framework. PostgreSQL stores the catalog, accounts,
|
||
playlists, configuration, and background-job state. Audio inspection uses
|
||
Symphonia, torrent downloads use librqbit, and the shared `music-dht`/Frid
|
||
protocol stack provides decentralized catalog search, media transfer, and
|
||
connected-device synchronization.
|
||
|
||
The browser interface and JSON API are served by the same application. Import,
|
||
artwork, metadata enrichment, similarity indexing, and maintenance run as
|
||
durable background jobs rather than blocking playback requests.
|
||
|
||
## Build profiles
|
||
|
||
Development and test builds use basic optimization (`opt-level = 1`) for
|
||
dependencies to keep macOS unwind tables below the compact-unwind size limit.
|
||
Application code stays unoptimized; debug symbols, assertions and panic
|
||
unwinding remain enabled. The first build compiles dependencies with these
|
||
settings, then reuses them from Cargo's cache. Dependency stepping in a debugger
|
||
can be less exact because of optimization. Release settings are independent.
|
||
|
||
See the [Cargo profile override documentation](https://doc.rust-lang.org/cargo/reference/profiles.html#overrides) for dependency-specific settings.
|
||
|
||
## Container builds and CI
|
||
|
||
CI uses Rust 1.97.0 and cargo-chef 0.1.74 to cache compiled dependencies separately
|
||
from application sources. Buildx imports and exports intermediate layers through
|
||
`ultradesu/furumusic:buildcache` (`mode=max`). After the first build, changes to
|
||
source code, templates or prompts reuse the dependency layer when its recipe and
|
||
toolchain are unchanged. Both dependency and application builds use `--locked`.
|
||
Update the pinned build tools in `Dockerfile`; changing them rebuilds the cache.
|
||
|
||
## Connected devices and podcast queue items
|
||
|
||
Trusted-device sync uses the shared `music_dht::device_sync` types (no local
|
||
copies) and announces protocol version 4. Each account is one virtual device:
|
||
|
||
- podcast operations stay in the operation log and are relayed to v4 devices
|
||
only. They are also applied to the account's podcast store
|
||
(`src/podcasts/`, `PodcastStore`), which also provides the podcast snapshot
|
||
sections;
|
||
- queue items that are not library tracks become external browser items
|
||
(`source` plus the verbatim `wire_track`), so podcast episodes and Jam
|
||
placeholders survive peer → server → browser → server → peer unchanged;
|
||
- `furumusic__playback_state.queue_items_json` stores the typed queue next to
|
||
`queue_json`. `queue_json` (library ids) is still written for rollback; the
|
||
typed value is used only while the legacy columns it recorded
|
||
(`queue_json`, `current_track_id`, `queue_position`) match the row. Jam
|
||
placeholders are never stored in a user's own queue.
|
||
- a browser that selects a podcast episode it cannot play does not know the
|
||
episode's position. It reports `position_unknown`, and the server sends the
|
||
resume position of the episode's progress register instead of 0. Jam
|
||
participant commands carry `jam_participant`; the host browser checks them
|
||
against its own queue.
|
||
|
||
Rolling back to 0.12 needs the podcast operations removed from the operation
|
||
log first; see "Rolling back to 0.12" in `administration.md`.
|
||
|
||
### Podcast store
|
||
|
||
`PodcastStore` implements the podcast contract of `music_dht::podcast`
|
||
(Frid `PODCASTS.md`) on PostgreSQL, with the invariants of the SQLite
|
||
reference store in `furumi-library`; every table has a `user_id` key.
|
||
|
||
| Table | Content | Synced |
|
||
| --- | --- | --- |
|
||
| `furumusic__podcast` | channel metadata, fetch URL | no (catalog) |
|
||
| `furumusic__podcast_feed_alias` | bounded grow-only alias set | in subscription records and `podcast_feed_aliases_added` |
|
||
| `furumusic__podcast_episode` | `EpisodeMeta`, `source` `feed`/`sync`, `in_feed`, cached search text | no (catalog) |
|
||
| `furumusic__podcast_season` | view over the episodes | no |
|
||
| `furumusic__podcast_subscription`, `_share`, `_progress` | registers | yes |
|
||
| `furumusic__podcast_listen_session` | grow-only set of sessions | yes |
|
||
| `furumusic__podcast_fetch_state` | ETag, Last-Modified, check times, errors, cover cache | never |
|
||
|
||
- One apply path: received operations, snapshot records and local actions go
|
||
through the `SyncOpWire` materializers and the `merge_*` functions of
|
||
`music_dht::podcast`. Each write transaction holds a per-account advisory
|
||
lock.
|
||
- Local actions (`subscribe`, `unsubscribe`, `set_shared`, `start_session`,
|
||
`checkpoint`, `seek`, `complete_session`, `restart`, `mark_played`,
|
||
`mark_unplayed`, `record_listen_session`) append their operation to the
|
||
account's sync log with the next stamp (`devices::append_local_op`: `seq`,
|
||
`hlc_ms`, `op_id = <device>:<seq>`) and apply exactly that operation in
|
||
the same transaction.
|
||
- `apply_feed` resolves episode ids with the shared no-GUID rule against the
|
||
stored episodes, including those only known from synced references. It
|
||
updates the catalog and the alias set, never a register, progress or a
|
||
session.
|
||
- Snapshots export each section newest first up to its count and bound it
|
||
with `bound_podcast_state_to`. The catalog and the fetch state are never
|
||
exported.
|
||
|
||
Migrations `m_0056_create_podcast_store` and `m_0057_import_podcast_relay`
|
||
create the tables and import what 0.13.0 kept in `furumusic__fed_podcast_relay`
|
||
(registers as contract JSON, already merged) plus the catalog rows of the
|
||
episode references in the operation log. The relay table is no longer written
|
||
or read; it stays until a later cleanup release.
|
||
|
||
### Feed refresh and audio delivery
|
||
|
||
The runtime around the store lives next to it in `src/podcasts/`:
|
||
|
||
- `net.rs` — `SafeHttp`, the only HTTP client for podcast feeds and audio.
|
||
It accepts `http`/`https` URLs without userinfo, resolves names with
|
||
`GuardedResolver` and refuses the whole answer when any address is not
|
||
public (loopback, RFC 1918, CGNAT, link-local, cloud metadata, unique local,
|
||
multicast, unspecified, documentation, reserved; IPv4-mapped, NAT64 and
|
||
6to4 addresses are judged by their IPv4 part). The connector connects to
|
||
exactly the checked addresses, so a second DNS answer cannot be swapped in.
|
||
IP literals are checked before connecting, redirects are followed manually
|
||
(each hop checked, at most 5), and system proxies, cookies and `Referer`
|
||
are disabled.
|
||
- `fetch.rs` — `FeedFetcher`: conditional GET (`If-None-Match`,
|
||
`If-Modified-Since`), `Accept-Encoding: identity`, 32 MiB and 60 s limits,
|
||
`Retry-After` on 429/503, the shared `music_dht` parser. Only a chain of
|
||
permanent redirects (301/308) moves a feed.
|
||
- `runner.rs` — scheduling: `podcasts_due_for_refresh`, one request per feed
|
||
URL for all accounts subscribed to it (conditional only when they share the
|
||
same validators), `apply_feed`, then the fetch state. Healthy feeds are
|
||
checked hourly plus a stable spread of up to 10 minutes; failures back off
|
||
15 min × 2ⁿ⁻¹ up to a day (a longer `Retry-After` wins). `refresh_now` is
|
||
the manual refresh: subscribed podcasts only, at most every 30 s.
|
||
`jobs/podcast_refresh.rs` runs `refresh_due` every five minutes (200
|
||
podcasts, 4 requests in flight, one run at a time via an advisory lock).
|
||
- `proxy.rs` and `http.rs` — the optional audio proxy and the endpoints under
|
||
`/api/podcasts`. The proxy takes an episode id, loads the URL from the
|
||
account's catalog (`PodcastStore::audio_source`), forwards a single
|
||
sanitized byte `Range`, passes 200/206/416 through, accepts audio content
|
||
types only (a generic binary type is replaced by the stored enclosure type
|
||
or the URL extension), and stops at 1 GiB, 20 s without headers, 30 s
|
||
without data or 4 h per response. It is off unless
|
||
`podcast_audio_proxy_enabled` is set; by default the browser streams from
|
||
the publisher URL.
|
||
|
||
Tests never reach the internet: `test_server.rs` serves on 127.0.0.1 and
|
||
provides a scripted resolver. Loopback is reachable only through
|
||
`AddressPolicy::test_allow_loopback`, which exists only under `#[cfg(test)]`.
|
||
|
||
Everything sent to a device is gated by the version that device reported itself
|
||
(`profile_protocol_version`; relayed profiles only raise it). See the podcast
|
||
contract (`PODCASTS.md`) and `PLAYBACK_PROTOCOL.md` in the Frid repository.
|
||
|
||
```bash
|
||
cargo test devices:: queue_items podcasts:: queue_persistence
|
||
node --test tests/player_queue_sources.test.cjs
|
||
# Disposable PostgreSQL; the tests run the production migrations on it, and
|
||
# the migration test creates and drops a database next to it:
|
||
FURU_DEVICE_SYNC_TEST_DATABASE_URL=postgresql://... \
|
||
cargo test postgres_tests -- --ignored
|
||
FURU_METADATA_FIX_TEST_DATABASE_URL=postgresql://... \
|
||
cargo test undo_accepts_playback_archives_written_before_the_typed_queue -- --ignored
|
||
```
|
||
|
||
`wire_compat_tests` compares the shared types with the private copies of
|
||
0.12.x byte for byte; `fixtures/v3_hello.golden.json` is the v3 shape. Rewrite
|
||
it only for a deliberate change, with `FURUMUSIC_BLESS_GOLDEN=1`.
|
||
|
||
## Contributing
|
||
|
||
Bug reports, design discussions, and patches are welcome. Before submitting a
|
||
change, run:
|
||
|
||
```bash
|
||
cargo fmt --all -- --check
|
||
cargo clippy --all-targets -- -D warnings
|
||
cargo test --all-targets
|
||
```
|
||
|
||
Last.fm database tests use temporary tables on a disposable PostgreSQL database:
|
||
|
||
```bash
|
||
FURUMUSIC_LASTFM_TEST_DATABASE_URL=postgres://user:pass@localhost/test_db \
|
||
cargo test lastfm_popularity -- --include-ignored
|
||
```
|
||
|
||
## Access and sharing tests
|
||
|
||
The access regression suite uses a disposable migrated database and a local OIDC
|
||
provider with signed tokens and PKCE:
|
||
|
||
```bash
|
||
node --test tests/*.test.cjs
|
||
FURU_ACCESS_TEST_DATABASE_URL=postgresql://... cargo test postgres_approval_survives_sso_changes_without_implicit_grants -- --ignored
|
||
# Fresh disposable instance on :18048, media root /tmp/furumi-access-media,
|
||
# PostgreSQL container furumi-access-test-pg, database furumi_access_test:
|
||
node tests/browser-access-sharing.cjs
|
||
```
|
||
|
||
The browser test requires Playwright (`PLAYWRIGHT_MODULE`) and Chrome
|
||
(`CHROME_PATH`), and uses local port 18049 for its OIDC provider. It checks
|
||
approval, SSO precedence and group removal, guest playback and downloads,
|
||
unauthorized track substitution, and desktop/mobile player behavior.
|
||
|
||
## Statistics tests
|
||
|
||
```bash
|
||
node --test tests/*.test.cjs
|
||
cargo test admin:: -- --skip postgres_statistics_drilldown_and_pagination
|
||
# Use a disposable PostgreSQL; this test creates connection-local temporary tables.
|
||
FURU_STATISTICS_TEST_DATABASE_URL=postgresql://... cargo test postgres_statistics_drilldown_and_pagination -- --ignored
|
||
```
|
||
|
||
`tests/browser-statistics.cjs` additionally exercises a fresh disposable instance
|
||
at `FURU_TEST_URL` (default `http://127.0.0.1:3018`) and seeds the disposable Docker
|
||
container `furumi-statistics-test-pg`. It requires Playwright (or
|
||
`PLAYWRIGHT_MODULE`) and Chrome (or `CHROME_PATH`). Never point this test at a
|
||
live service.
|
||
|
||
## Metadata Repair tests
|
||
|
||
```bash
|
||
node --test tests/admin_metadata_repair.test.cjs
|
||
cargo test admin::metadata_fix::tests
|
||
```
|
||
|
||
`tests/browser-metadata-repair.cjs` checks the chat and plan updates, a four-release
|
||
merge, deletion and restoration of tracks and artist credits, playlists, playback
|
||
queues, and undo conflicts. Run it only against a disposable instance and a
|
||
PostgreSQL container named `furumi-repair-test-pg`, with database
|
||
`furumi_repair_test`. Set `FURU_TEST_URL`, `FURU_REPAIR_TEST_PG`, `CHROME_PATH`, and
|
||
`PLAYWRIGHT_MODULE` as needed. It seeds `tests/metadata_repair_fixture.sql` after
|
||
creating the test administrator. Set `FURU_REPAIR_SKIP_SEED=1` for a seeded database.
|
||
With `FURU_REPAIR_TEST_MODEL=1`, it additionally checks the reported release-only
|
||
selection and full ordered tracklist against the configured model, applies and
|
||
undoes that plan, and verifies individual track exclusions.
|
||
|
||
## Import destination tests
|
||
|
||
```bash
|
||
node --test tests/import_playlists.test.cjs
|
||
# Disposable PostgreSQL; both tests use temporary tables.
|
||
FURU_IMPORT_TEST_DATABASE_URL=postgresql://... cargo test import_destinations -- --ignored
|
||
FURU_IMPORT_TEST_DATABASE_URL=postgresql://... cargo test torrent_playlist -- --ignored
|
||
```
|
||
|
||
`tests/browser-import-playlists.cjs` checks the file upload flow, database
|
||
reconciliation and the torrent picker on desktop and mobile. It requires a fresh
|
||
migrated instance with the import agent disabled at `FURU_TEST_URL` (default
|
||
`http://127.0.0.1:3019`), the disposable PostgreSQL container
|
||
`furumi-import-playlist-test-pg` with database `furumi_import_test`, and Playwright
|
||
and Chrome configured as above. It uses a local torrent preview fixture and
|
||
simulated imported tracks, without downloading external torrents.
|