Remove imported podcast episodes that duplicate a held id
A provisional imported GUID-less episode whose feed item id is already held by an episode that stays (typically one a remote operation created before this device's first refresh) is now deleted instead of kept as a second, never re-keyed copy. The same applies when its legacy GUID turns out to be the item's real GUID. Such episodes carry no personal state. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
06ac3691d8
commit
b8d8b532f2
4 files changed
+218
-19
No files matched your search
+4
-1
@@ -41,7 +41,10 @@
|
||||
- `furumi-library 0.3.0`: prototype import collapses rows of one GUID-less
|
||||
item under several enclosure URLs, and the first refresh re-keys imported
|
||||
GUID-less episodes without personal state to the ids the current feed
|
||||
gives, so they match devices that never had the prototype. Fetch-state
|
||||
gives, so they match devices that never had the prototype; when another
|
||||
episode (for example one created by a remote operation) already holds
|
||||
that id, the imported copy is deleted instead of kept as a duplicate.
|
||||
Fetch-state
|
||||
writes are atomic across threads and processes.
|
||||
- See `PODCASTS.md` for the normative contract.
|
||||
|
||||
|
||||
+10
-6
@@ -560,12 +560,16 @@ must keep the same invariants.
|
||||
item ids are resolved against the stored episodes except provisional
|
||||
ones without progress or sessions; each such provisional episode is
|
||||
matched to its item (enclosure, then link, then date and title, GUIDs
|
||||
ignored) and re-keyed to that item's id unless another episode holds
|
||||
it. The same applies when `legacy_guid` turns out to be the item's real
|
||||
GUID. Listing an episode clears `legacy_guid`, which makes its id final.
|
||||
Provisional episodes carry no personal state, so nothing personal moves;
|
||||
an imported episode played before its first refresh keeps its id (the
|
||||
feed item then becomes a separate episode if the ids differ).
|
||||
ignored) and re-keyed to that item's id. When another episode that
|
||||
stays already holds that id (for example one a remote operation created
|
||||
before this device's first refresh), it is the item's episode and the
|
||||
provisional one is a duplicate: it is deleted rather than kept as a
|
||||
second copy. The same applies when `legacy_guid` turns out to be the
|
||||
item's real GUID. Listing an episode clears `legacy_guid`, which makes
|
||||
its id final. Provisional episodes carry no personal state, so nothing
|
||||
personal moves or is lost; an imported episode played before its first
|
||||
refresh is not provisional and keeps its id (the feed item then becomes
|
||||
a separate episode if the ids differ).
|
||||
* ETag, Last-Modified, check time, last error and the cover cache move to the
|
||||
fetch state. The prototype stores no progress or history, so none is
|
||||
created.
|
||||
|
||||
@@ -121,7 +121,9 @@ fn has_personal_state(conn: &Connection, episode_id: &EpisodeId) -> Result<bool>
|
||||
/// enclosure URL (`legacy_guid`) when the feed shows it is a real GUID.
|
||||
///
|
||||
/// Only episodes without progress or sessions move, so no personal state is
|
||||
/// touched; otherwise the GUID item becomes a separate episode.
|
||||
/// touched; otherwise the GUID item becomes a separate episode. When another
|
||||
/// episode already holds the GUID id (for example one a remote operation
|
||||
/// created), the imported episode is a duplicate of it and is deleted.
|
||||
fn rekey_legacy_guids(conn: &Connection, podcast_id: &PodcastId, feed: &ParsedFeed) -> Result<()> {
|
||||
for item in &feed.episodes {
|
||||
let Some(guid) = item.guid.as_deref() else {
|
||||
@@ -138,9 +140,6 @@ fn rekey_legacy_guids(conn: &Connection, podcast_id: &PodcastId, feed: &ParsedFe
|
||||
)
|
||||
.optional()?
|
||||
.is_some();
|
||||
if taken {
|
||||
continue;
|
||||
}
|
||||
let legacy: Option<String> = conn
|
||||
.query_row(
|
||||
"SELECT episode_id FROM episodes
|
||||
@@ -153,7 +152,14 @@ fn rekey_legacy_guids(conn: &Connection, podcast_id: &PodcastId, feed: &ParsedFe
|
||||
continue;
|
||||
};
|
||||
let legacy = super::episode_id_col(legacy)?;
|
||||
if has_personal_state(conn, &legacy)? {
|
||||
if legacy == guid_id || has_personal_state(conn, &legacy)? {
|
||||
continue;
|
||||
}
|
||||
if taken {
|
||||
conn.execute(
|
||||
"DELETE FROM episodes WHERE episode_id = ?1",
|
||||
[legacy.as_str()],
|
||||
)?;
|
||||
continue;
|
||||
}
|
||||
conn.execute(
|
||||
@@ -199,9 +205,11 @@ fn provisional_episode_ids(
|
||||
/// * Each provisional episode is matched to the item it was imported from
|
||||
/// with the same rule (enclosure, then link, then date and title; GUIDs
|
||||
/// ignored, since the prototype may have stored the enclosure URL as GUID)
|
||||
/// and re-keyed to that item's id unless another stored episode already
|
||||
/// holds it. Unmatched provisional episodes stay as they are (not in the
|
||||
/// feed); the next refresh that lists them re-keys them.
|
||||
/// and re-keyed to that item's id. When another stored episode that stays
|
||||
/// already holds that id, it is the item's episode and the provisional one
|
||||
/// is a duplicate: it is deleted. Unmatched provisional episodes stay as
|
||||
/// they are (not in the feed); the next refresh that lists them re-keys
|
||||
/// them.
|
||||
///
|
||||
/// Provisional episodes have no progress and no session, so no personal
|
||||
/// state moves. The refresh then clears `legacy_guid` on every episode it
|
||||
@@ -244,12 +252,18 @@ fn resolve_and_rekey_provisional(
|
||||
(matched != target).then_some((matched, target))
|
||||
})
|
||||
.collect();
|
||||
// A target held by an episode that does not move away blocks the move.
|
||||
let occupied: BTreeSet<EpisodeId> = stable_known
|
||||
// A target held by an episode that stays (for example a row a remote
|
||||
// operation created before this first refresh) is the item's episode
|
||||
// already: the provisional episode is a duplicate of it. It has no
|
||||
// progress and no session, and its metadata would be overwritten by the
|
||||
// item anyway, so it is removed instead of kept as a second copy. A
|
||||
// removed duplicate frees its own id for other moves.
|
||||
let mut occupied: BTreeSet<EpisodeId> = stable_known
|
||||
.iter()
|
||||
.chain(&provisional_known)
|
||||
.map(|episode| episode.episode_id.clone())
|
||||
.collect();
|
||||
let mut duplicates: Vec<EpisodeId> = Vec::new();
|
||||
loop {
|
||||
let leaving: BTreeSet<&EpisodeId> = moves.iter().map(|(from, _)| from).collect();
|
||||
let blocked: BTreeSet<EpisodeId> = moves
|
||||
@@ -261,6 +275,22 @@ fn resolve_and_rekey_provisional(
|
||||
break;
|
||||
}
|
||||
moves.retain(|(from, _)| !blocked.contains(from));
|
||||
for from in blocked {
|
||||
occupied.remove(&from);
|
||||
duplicates.push(from);
|
||||
}
|
||||
}
|
||||
for duplicate in &duplicates {
|
||||
// Guarded again: only a provisional episode without personal state.
|
||||
conn.execute(
|
||||
"DELETE FROM episodes
|
||||
WHERE episode_id = ?1 AND legacy_guid IS NOT NULL
|
||||
AND NOT EXISTS (SELECT 1 FROM episode_progress p
|
||||
WHERE p.episode_id = episodes.episode_id)
|
||||
AND NOT EXISTS (SELECT 1 FROM podcast_listen_sessions s
|
||||
WHERE s.episode_id = episodes.episode_id)",
|
||||
[duplicate.as_str()],
|
||||
)?;
|
||||
}
|
||||
// Two phases, so a move may take an id another move vacates.
|
||||
for (index, (from, _)) in moves.iter().enumerate() {
|
||||
|
||||
@@ -1976,9 +1976,17 @@ fn first_refresh_rekeys_provisional_imported_guidless_episodes() {
|
||||
.unwrap();
|
||||
assert_eq!(legacy_guid, None, "the refresh confirmed the id");
|
||||
|
||||
// Ids are final now: later refreshes keep them.
|
||||
// Ids are final now: later refreshes keep them. The retitled stale row
|
||||
// is the only provisional episode left; its link now identifies the
|
||||
// item, whose id the confirmed episode holds, so it is removed as a
|
||||
// duplicate.
|
||||
refresh(&store, &mut stamps, &podcast_id, &feed, &url(FEED), false);
|
||||
assert_eq!(in_feed_ids(&store, &podcast_id), after);
|
||||
assert_eq!(
|
||||
in_feed_ids(&store, &podcast_id),
|
||||
vec![(fresh.clone(), true)]
|
||||
);
|
||||
refresh(&store, &mut stamps, &podcast_id, &feed, &url(FEED), false);
|
||||
assert_eq!(in_feed_ids(&store, &podcast_id), vec![(fresh, true)]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -2032,6 +2040,160 @@ fn imported_episodes_with_personal_state_are_not_rekeyed() {
|
||||
assert_eq!(store.resume_position_ms(&played).unwrap(), 42_000);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn remote_progress_before_the_first_refresh_leaves_no_duplicate_episode() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let path = dir.path().join(PODCAST_DB_FILE);
|
||||
prototype_with_rows(
|
||||
&path,
|
||||
&[
|
||||
(
|
||||
OLD_AUDIO,
|
||||
"Pilot (draft)",
|
||||
Some(PILOT_SECS),
|
||||
OLD_AUDIO,
|
||||
Some(PILOT_LINK),
|
||||
),
|
||||
(
|
||||
NEW_AUDIO,
|
||||
"Pilot",
|
||||
Some(PILOT_SECS),
|
||||
NEW_AUDIO,
|
||||
Some(PILOT_LINK),
|
||||
),
|
||||
],
|
||||
);
|
||||
let store_a = PodcastStore::open(&path).unwrap();
|
||||
let podcast_id = PodcastId::from_feed_url(&url(FEED));
|
||||
let feed = current_pilot_feed();
|
||||
let fresh = feed.resolve_episode_ids(&podcast_id, &[])[0]
|
||||
.clone()
|
||||
.unwrap();
|
||||
|
||||
// A fresh device B plays the episode, and A applies B's operation before
|
||||
// its own first refresh: A now holds the fresh id as a synced episode.
|
||||
let store_b = PodcastStore::open_in_memory().unwrap();
|
||||
let mut stamps_b = TestStamps::new("device-b", 1_900_000_000_000);
|
||||
store_b
|
||||
.subscribe(&mut stamps_b, &meta(FEED, "Show"))
|
||||
.unwrap();
|
||||
refresh(
|
||||
&store_b,
|
||||
&mut stamps_b,
|
||||
&podcast_id,
|
||||
&feed,
|
||||
&url(FEED),
|
||||
false,
|
||||
);
|
||||
let reference = store_b.episode_ref(&fresh).unwrap().unwrap();
|
||||
let write = store_b
|
||||
.start_session(&mut stamps_b, &reference, "b-1", 600_000, None)
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
store_a.apply_remote_op(&write.op.into_wire(1)).unwrap(),
|
||||
ApplyOutcome::Changed
|
||||
);
|
||||
|
||||
let mut stamps_a = TestStamps::new("device-a", 1_800_000_000_000);
|
||||
// First refresh: the provisional "Pilot" row matches the item by its
|
||||
// enclosure, but the synced episode holds the item's id, so the
|
||||
// provisional row is a duplicate and is removed. Second refresh: the
|
||||
// retitled stale row is now the only provisional episode, its link
|
||||
// identifies the same item, and it goes too.
|
||||
for expected_rows in [2, 1] {
|
||||
refresh(
|
||||
&store_a,
|
||||
&mut stamps_a,
|
||||
&podcast_id,
|
||||
&feed,
|
||||
&url(FEED),
|
||||
false,
|
||||
);
|
||||
let items = store_a
|
||||
.episodes(&EpisodeQuery::new(podcast_id.clone()))
|
||||
.unwrap();
|
||||
let pilots: Vec<_> = items
|
||||
.iter()
|
||||
.filter(|item| item.episode.title == "Pilot")
|
||||
.collect();
|
||||
assert_eq!(pilots.len(), 1, "one copy of the feed item");
|
||||
assert_eq!(pilots[0].episode.episode_id, fresh);
|
||||
assert!(pilots[0].in_feed);
|
||||
assert_eq!(store_a.resume_position_ms(&fresh).unwrap(), 600_000);
|
||||
assert_eq!(items.len(), expected_rows);
|
||||
assert!(
|
||||
items
|
||||
.iter()
|
||||
.filter(|item| item.episode.episode_id != fresh)
|
||||
.all(|item| !item.in_feed && item.progress.is_none())
|
||||
);
|
||||
let summary = store_a.podcast(&podcast_id).unwrap().unwrap();
|
||||
assert_eq!(summary.episode_count, expected_rows as u64);
|
||||
assert_eq!(summary.in_progress_count, 1);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn remote_progress_on_the_guid_id_removes_the_imported_copy() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let path = dir.path().join(PODCAST_DB_FILE);
|
||||
// The feed's GUID is the old enclosure URL, so the prototype row looks
|
||||
// GUID-less; title and enclosure changed since, and there is no link or
|
||||
// date to match on.
|
||||
prototype_with_rows(
|
||||
&path,
|
||||
&[(OLD_AUDIO, "Pilot (draft)", None, OLD_AUDIO, None)],
|
||||
);
|
||||
let store_a = PodcastStore::open(&path).unwrap();
|
||||
let podcast_id = PodcastId::from_feed_url(&url(FEED));
|
||||
let feed = parse(
|
||||
&rss(
|
||||
"Show",
|
||||
None,
|
||||
&[Item::new(Some(OLD_AUDIO), "Pilot", NEW_AUDIO)],
|
||||
),
|
||||
&url(FEED),
|
||||
);
|
||||
let fresh = feed.resolve_episode_ids(&podcast_id, &[])[0]
|
||||
.clone()
|
||||
.unwrap();
|
||||
assert!(in_feed_ids(&store_a, &podcast_id)[0].0 != fresh);
|
||||
|
||||
let store_b = PodcastStore::open_in_memory().unwrap();
|
||||
let mut stamps_b = TestStamps::new("device-b", 1_900_000_000_000);
|
||||
store_b
|
||||
.subscribe(&mut stamps_b, &meta(FEED, "Show"))
|
||||
.unwrap();
|
||||
refresh(
|
||||
&store_b,
|
||||
&mut stamps_b,
|
||||
&podcast_id,
|
||||
&feed,
|
||||
&url(FEED),
|
||||
false,
|
||||
);
|
||||
let reference = store_b.episode_ref(&fresh).unwrap().unwrap();
|
||||
let write = store_b
|
||||
.start_session(&mut stamps_b, &reference, "b-1", 5_000, None)
|
||||
.unwrap();
|
||||
store_a.apply_remote_op(&write.op.into_wire(1)).unwrap();
|
||||
|
||||
let mut stamps_a = TestStamps::new("device-a", 1_800_000_000_000);
|
||||
refresh(
|
||||
&store_a,
|
||||
&mut stamps_a,
|
||||
&podcast_id,
|
||||
&feed,
|
||||
&url(FEED),
|
||||
false,
|
||||
);
|
||||
assert_eq!(
|
||||
in_feed_ids(&store_a, &podcast_id),
|
||||
vec![(fresh.clone(), true)]
|
||||
);
|
||||
assert_eq!(store_a.resume_position_ms(&fresh).unwrap(), 5_000);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn concurrent_fetch_state_writers_lose_no_update() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
|
||||
Reference in new issue
Block a user