feat(sync): per-show check cadence in shadow

CONTRIBUTING.md

@@ -47,7 +47,7 @@ In accordance with the [PolyForm Strict License 1.0.0](LICENSE), you are welcome
 | Java | JDK 17; set `JAVA_HOME` and the Android Studio Gradle JDK to this installation |
 | Android SDK | Platform 36, Platform-Tools, and the Build-Tools version requested by the Android Gradle plugin |
 | Gradle | Use the checked-in `./gradlew` wrapper; no separate Gradle installation is needed |
-| Node.js | Version 20 and npm for scripts; not needed for ordinary Android builds |
+| Node.js | Version 22.13 or newer and npm for scripts; not needed for ordinary Android builds |
 | Python | Python 3 for local configuration and release-tooling tests |
 | Device | Android 12 / API 31 or newer when running the app |
 

scripts/README.md

@@ -6,7 +6,7 @@ Development guidance for catalog synchronization, notification checks, feed repa
 
 Catalog sync refreshes charts, imports podcasts, syncs episodes, and vectorizes content. It runs outside GitHub Actions; the former `sync-pi-data` workflow has been removed. Adding a replacement GitHub sync workflow requires explicit maintainer approval.
 
-Scripts need Node.js 22.5 or newer (`engines` in `scripts/package.json`): stage 3 reads the feed-address registry with `node:sqlite`. The runner uses Node 22, and so do the workflows that install `scripts/`.
+Scripts need Node.js 22.13 or newer (`engines` in `scripts/package.json`): stage 3 reads the feed-address registry and Apple's snapshot with `node:sqlite`, which needs a flag before 22.13. The runner uses Node 22, and so do the workflows that install `scripts/`.
 
 A Git push does not deploy this pipeline. Changes to `scripts/sync/` or `scripts/package.json` require a separate deployment and verification that the active runner uses the updated code and configuration. Read the production runbook in root `AGENTS.internal.md` before deployment. If it is unavailable, obtain it from the maintainer before performing production operations.
 
@@ -33,6 +33,7 @@ A Git push does not deploy this pipeline. Changes to `scripts/sync/` or `scripts
 | [`scripts/sync/lib/catalog-rows.js`](sync/lib/catalog-rows.js) | Scrubbed values for stage 2's API import and stage 3's latest-episode update: an odd optional field from Podcast Index becomes empty instead of failing the write. The latest-episode update stores the episode's served ID in `latest_ep_id` (what Home, trending and bootstrap send), plus `latest_ep_key` and `latest_ep_eid`, from `tipIdentity` in `lib/episode-identity.js` (by catalog key, else Podcast Index's ID as an alias in the same show, else that ID). Change detection and the tip queue keep Podcast Index's own ID |
 | [`scripts/sync/lib/stale-tips.js`](sync/lib/stale-tips.js) | Stage 3's tips for stale shows (`R5_MODE`: `off` by default, `shadow`, `on`). After the Podcast Index pass, checked chart shows that the published feed-address registry (`REGISTRY_DB`) flags `episode_stale` ask the local feed reader (`FEEDS_URL`, `POST /feeds/latest`, 40 shows per call) for their newest episode. When it is playable, a different episode and newer than the stored tip, it becomes the tip under its served ID, catalog key and recipe ID. Only the tip columns change: `qdrant_vectorized` and the tip queue stay with the Podcast Index pass. `shadow` reads only the feed's dates and never writes. A registry or feed-reader failure skips it, and the Podcast Index tips stay |
 | [`scripts/sync/lib/show-window.js`](sync/lib/show-window.js) | Saved episode windows (`WINDOWS_MODE`: `off` by default, `shadow`, `on`). Stage 3 records each full list it fetches as episode-book rows, with descriptions, and a `show_window` row: the book row IDs in list order. It fetches at the cap stage 4 enforces, chart or earned. Shows whose R5 tip came from the feed also save the feed reader's merged list (`GET /feeds/episodes`), so their feed-only episodes get vectors. With `on`, stage 4 reads a saved window instead of fetching the list from Podcast Index, when it is usable, fresh (3 days) and long enough. `shadow` records windows and has stage 4 compare them with its own fetch. A window with an episode that has no catalog key, or that another show serves, is unusable, and stage 4 takes the fetch path. Tip-queue rows carry `gen`, and stage 4 clears only rows enqueued before it started |
+| [`scripts/sync/lib/cadence.js`](sync/lib/cadence.js) | Per-show check schedule for stage 3, in shadow (`CADENCE_MODE`: `shadow` by default, `off`). Checks still follow the news/core/relaxed tiers; the run report adds what the schedule would have checked. Each show's interval follows its pace, the average gap between its newest 11 episodes: news shows hourly, others a quarter of the gap within 3 to 48 hours, and dormant shows weekly when both the newest episode and a fresh Apple answer are over 180 days old. A show with no known pace keeps its tier. A show Apple lists a newer episode for (`APPLE_LATEST_DB`, the applefeeds snapshot) is due until it has been checked 6 hours past that date. Pace and the newest date come from each fetched list and from the episode book about daily (400 shows per read); show records gain `g`, `p` and `i` (iTunes ID). The report line counts shows the schedule would skip or check earlier, new tips it would have found later and by how much, and checks a day against today's tiers. A book or snapshot failure only skips that part |
 | [`scripts/sync/lib/attribution.js`](sync/lib/attribution.js) | Vector attribution log: one JSON line per episode point stored or deleted (prune, cap trim), show vector stored, and show imported or deleted. Written to `SYNC_ATTRIBUTION_DIR` (default `scripts/data/attribution/`, gitignored) as daily files, gzipped after the day ends and kept 60 days. Best effort: a write failure never fails a stage. Stage 4 adds `stored`, `pruned` and `reasons …` to its run line |
 | [`scripts/sync/attribution-report.js`](sync/attribution-report.js) | Read-only report over that log: embeddings by reason (new episode, first coverage, returning show, re-embed after cap trim or prune, Podcast Index ID change, backfill), deletions, and how long chart entrants stay. `--days N`, `--json` |
 | [`scripts/sync/lib/episode-book.js`](sync/lib/episode-book.js) + [`episode-book-cli.js`](sync/episode-book-cli.js) | The episode book in sqld: one permanent row per (show, catalog key) with its recipe ID and served ID, plus Podcast Index aliases, `served_show` and `show_window`. Every book write also marks the show in `served_show` when one of its episodes is listed by Podcast Index under an ID other than its served ID (renumbered, duplicated or recipe-served), so the Worker's v1 overlay translates that ID back. That check reaches a show's aliases through `episode_pi_alias_rid` (an index on `rid`); without it each check scanned every alias inside the write. `ensure` creates the tables, indexes and columns; `stats` counts. Writes go through `turso.atomic()` (one Hrana batch: every step runs only after the previous one succeeded, then COMMIT, else ROLLBACK). Rules: [`docs/EPISODE_IDS.md`](../docs/EPISODE_IDS.md) |
@@ -50,7 +51,7 @@ The runner orchestrator lives outside this repository. It invokes the stages in
 - Preserve the chart index: `charts.itunes_id` is TEXT, so casting that column to INTEGER disables the index and inflates `rows_read`. Use `c.itunes_id = CAST(p.itunes_id AS TEXT)`, [`sync/lib/chart-countries.js`](sync/lib/chart-countries.js) (`loadCountriesByItunesId` plus a JavaScript filter), or page podcasts by `id` and normalize IDs in JavaScript. This applies to stages 2, 3, 4, 5, and remediation.
 - Keep credentials and environment files out of Git, including `.env`, Podcast Index keys, Turso tokens, and Telegram secrets.
 
-Use Node.js 20 to match the script workflows. From the repository root:
+Use Node.js 22.13 or newer to match the runner and the script workflows. From the repository root:
 
 ```bash
 npm ci --prefix scripts

scripts/package.json

@@ -16,7 +16,7 @@
     "test:sync-scalars": "node --test sync/lib/scalars.test.js",
     "test:sync-turso-page": "node --test sync/lib/turso-page.test.js",
     "test:sync-turso": "node --test sync/lib/turso.test.js",
-    "test:sync": "node --test sync/lib/csv.test.js sync/lib/pi-unavailable.test.js sync/lib/scalars.test.js sync/lib/turso-page.test.js sync/lib/episode-caps.test.js sync/lib/staleness.test.js sync/lib/podcast-index.test.js sync/lib/chart-countries.test.js sync/lib/log.test.js sync/lib/text.test.js sync/lib/vectorize-lanes.test.js sync/lib/tip-queue.test.js sync/lib/import-missing.test.js sync/lib/episode-recipe-id.test.js sync/lib/attribution.test.js sync/lib/vector-window.test.js sync/lib/trim-episode-caps.test.js sync/lib/remediate-qdrant.test.js sync/lib/turso.test.js sync/lib/catalog-rows.test.js sync/lib/episode-book.test.js sync/lib/book-bootstrap.test.js sync/lib/episode-identity.test.js sync/lib/stale-tips.test.js sync/lib/show-window.test.js",
+    "test:sync": "node --test sync/lib/csv.test.js sync/lib/pi-unavailable.test.js sync/lib/scalars.test.js sync/lib/turso-page.test.js sync/lib/episode-caps.test.js sync/lib/staleness.test.js sync/lib/podcast-index.test.js sync/lib/chart-countries.test.js sync/lib/log.test.js sync/lib/text.test.js sync/lib/vectorize-lanes.test.js sync/lib/tip-queue.test.js sync/lib/import-missing.test.js sync/lib/episode-recipe-id.test.js sync/lib/attribution.test.js sync/lib/vector-window.test.js sync/lib/trim-episode-caps.test.js sync/lib/remediate-qdrant.test.js sync/lib/turso.test.js sync/lib/catalog-rows.test.js sync/lib/episode-book.test.js sync/lib/book-bootstrap.test.js sync/lib/episode-identity.test.js sync/lib/stale-tips.test.js sync/lib/show-window.test.js sync/lib/cadence.test.js",
     "test:sync-text": "node --test sync/lib/text.test.js",
     "test:sync-recipe": "node --test sync/lib/episode-recipe-id.test.js",
     "test:sync-attribution": "node --test sync/lib/attribution.test.js",
@@ -29,6 +29,6 @@
     "test:check-new-episodes": "node --test check-new-episodes-lib.test.js"
   },
   "engines": {
-    "node": ">=22.5"
+    "node": ">=22.13"
   }
 }

scripts/sync/03-sync-episodes.js

@@ -13,6 +13,8 @@
  *   full items[] written to PI handoff for stage 4 (no second PI call same run).
  * - R5 (R5_MODE, lib/stale-tips.js): afterwards, checked shows the registry flags stale take the
  *   live feed's newest episode as their tip when it is newer than Podcast Index's.
+ * - Check cadence (CADENCE_MODE, lib/cadence.js): in shadow, reports what a per-show schedule would
+ *   have checked; checks still follow the tiers above.
  */
 
 const log = require('./lib/log');
@@ -35,6 +37,7 @@ const {
 const staleness = require('./lib/staleness');
 const staleTips = require('./lib/stale-tips');
 const showWindow = require('./lib/show-window');
+const cadence = require('./lib/cadence');
 
 const CANDIDATE_CACHE_MAX_AGE_MS = 20 * 60 * 60 * 1000;
 const CONCURRENCY = 5;
@@ -63,9 +66,10 @@ async function refreshCandidates(st) {
     const ids = [];
     let newsCount = 0;
     let relaxedCount = 0;
-    for (const { id: podId, latestEpId, categories, medium, countryCsv } of chartShows) {
+    for (const { id: podId, latestEpId, categories, medium, itunesId, countryCsv } of chartShows) {
         ids.push(podId);
         const rec = st.shows[podId] || {};
+        if (itunesId) rec.i = itunesId;
         if (rec.e === undefined && latestEpId) rec.e = String(latestEpId);
         if ((categories || '').includes('News')) {
             rec.n = 1;
@@ -143,6 +147,7 @@ async function main() {
         'Concurrency': String(CONCURRENCY),
         'Stale tips (R5)': cfg.R5_MODE,
         'Saved windows': cfg.WINDOWS_MODE,
+        'Check cadence': cfg.CADENCE_MODE,
     });
 
     const cacheAge = Date.now() - (st.candidatesRefreshedAt || 0);
@@ -163,12 +168,47 @@ async function main() {
             needRefresh = true;
         }
     }
+    if (!needRefresh && st.candidateIds && cfg.CADENCE_MODE !== 'off') {
+        // One-shot: the cadence reads Apple's dates by iTunes ID, which only a refresh records.
+        const missingI = st.candidateIds.filter((id) => !st.shows[id]?.i).length;
+        if (missingI > Math.max(50, st.candidateIds.length * 0.05)) {
+            log.info(`[CANDIDATES] Refreshing to record iTunes IDs (${log.fmt(missingI)} shows without one)`);
+            needRefresh = true;
+        }
+    }
     if (needRefresh) {
         await refreshCandidates(st);
     } else {
         log.info(`[CANDIDATES] Using cached list (${log.fmt(st.candidateIds.length)} shows, age ${Math.round(cacheAge / 3600000)}h) - 0 Turso reads`);
     }
 
+    // Cadence shadow: pace from the book about daily, then the plan from the state as it was before
+    // this run's checks. Never fails the stage.
+    let shadow = null;
+    let appleError = null;
+    if (cfg.CADENCE_MODE === 'shadow') {
+        if (needRefresh || Date.now() - (st.cadenceSeededAt || 0) > CANDIDATE_CACHE_MAX_AGE_MS) {
+            try {
+                const seeded = await cadence.seedFromBook(book, st.shows, st.candidateIds);
+                st.cadenceSeededAt = Date.now();
+                log.info(`[CADENCE] pace seeded from the book for ${log.fmt(seeded)} shows`);
+            } catch (e) {
+                log.warn(`[CADENCE] pace seed skipped: ${e.message}`);
+            }
+        }
+        const apple = cadence.openApple(cfg.APPLE_LATEST_DB);
+        appleError = apple.error;
+        if (appleError) log.warn(`[CADENCE] Apple snapshot unread: ${appleError}`);
+        try {
+            shadow = cadence.shadowPlan(st.candidateIds, st.shows, { appleOf: apple.lookup });
+        } catch (e) {
+            log.warn(`[CADENCE] shadow plan skipped: ${e.message}`);
+        } finally {
+            apple.close();
+        }
+    }
+    const newTipIds = [];
+
     const plan = staleness.planDue(st.candidateIds, st.shows);
     const { allDue, neverChecked, staleNews, staleCore, staleRelaxed,
         poolNews, poolCore, poolRelaxed } = plan;
@@ -252,7 +292,8 @@ async function main() {
                 const latestTip = tip[0];
                 if (rec.e && String(latestTip.id) === String(rec.e)) {
                     unchanged++;
-                    state.recordCheck(st, podId);
+                    const checked = state.recordCheck(st, podId);
+                    if (shadow) cadence.noteNewest(checked, latestTip.datePublished);
                     return;
                 }
 
@@ -292,7 +333,12 @@ async function main() {
                 updated++;
                 tipReflags++;
                 tipUpdatedThisBatch = true;
-                state.recordCheck(st, podId, { latestEpId: latest.id }).m = medium;
+                const checked = state.recordCheck(st, podId, { latestEpId: latest.id });
+                checked.m = medium;
+                if (shadow) {
+                    cadence.notePace(checked, (full || tip).map((ep) => ep.datePublished));
+                    newTipIds.push(podId);
+                }
             } catch (e) {
                 if (pi.isSuspended(e)) {
                     piPaused = piPaused || e;
@@ -329,6 +375,20 @@ async function main() {
     const r5Line = staleTips.summary(r5);
     if (cfg.R5_MODE !== 'off') log.info(`[R5] ${r5Line}`);
 
+    let cadenceLine = '';
+    if (shadow) {
+        for (const [id, date] of Object.entries(r5.tipDates || {})) {
+            cadence.noteNewest(st.shows[id] || (st.shows[id] = {}), date);
+            newTipIds.push(id);
+        }
+        state.save(st);
+        const paceShows = st.candidateIds.filter((id) => st.shows[id]?.g != null).length;
+        cadenceLine = cadence.summary(cfg.CADENCE_MODE, shadow, cadence.compare(shadow, allDue, newTipIds), {
+            appleError, paceShows, candidates: st.candidateIds.length,
+        });
+        log.info(`[CADENCE] ${cadenceLine}`);
+    }
+
     // R5 shows whose tip came from the feed also save the feed's merged list as their window, so
     // stage 4 embeds the feed-only episodes (WINDOWS_MODE on flags them; shadow only records).
     if (cfg.WINDOWS_MODE !== 'off' && r5.updatedIds?.length) {
@@ -388,7 +448,8 @@ async function main() {
             ` · ${pi.summary()}` +
             (piPaused ? ` · ${due.length - checkedBeforePause} checks left for the next run` : '') +
             (cfg.R5_MODE !== 'off' ? ` · ${r5Line}` : '') +
-            (windowLine ? ` · ${windowLine}` : ''),
+            (windowLine ? ` · ${windowLine}` : '') +
+            (cadenceLine ? ` · ${cadenceLine}` : ''),
     }]);
 
     // A pause is "try later", not a failure; only real errors over half the checks fail the stage.