Skip to content

fix(etl): validate freshness from final fallback assets - #111

Merged
Sam-24-dev merged 1 commit into
mainfrom
fix/etl-final-source-freshness
Sep 15, 2026
Merged

Sam-24-dev merged 1 commit into
mainfrom
fix/etl-final-source-freshness

Conversation

@Sam-24-dev

Copy link
Copy Markdown
Owner

Summary

  • restore all three canonical Reddit bridges, including reddit_sentimiento_public.json, while preserving their embedded source timestamps and recording baseline fallback provenance
  • regenerate run_manifest.json from the final canonical bridges in both asset roots after the existing final Home highlights rebuild
  • cross-check Reddit freshness against every final bridge and keep the 192-hour publication gate authoritative

Root cause

sync_assets.py generated the public manifest before the workflow restored the final Reddit fallback bridges. The final restore also omitted the sentiment bridge, so the manifest and freshness guard could report current Reddit timestamps while the canonical topics and intersection bridges were older.

Final order

  1. restore the Reddit source fallback
  2. sync assets
  3. restore final Reddit bridges
  4. rebuild Home highlights in both roots
  5. regenerate final manifests in both roots
  6. run bridge integrity
  7. run the unchanged 192-hour freshness guard
  8. upload and publish only if both gates pass

Fallback provenance now produces quality_gate_status: pass_with_warnings and degraded_mode: true. An expired fallback fails before artifact upload or publication.

Validation

  • TDD RED: 6 failed, 28 passed before implementation
  • focused Python tests: 40 passed
  • full Python suite: 239 passed
  • Flutter tests: 145 passed
  • flutter analyze --no-pub: no issues
  • flutter build web --release --no-pub: succeeded
  • compileall, Ruff, Bandit, detect-secrets, and git diff --check: passed

Safety

  • Reddit coverage threshold remains 85%
  • freshness threshold remains 192 hours
  • concurrency, fallback, integrity, and workflow_dispatch remain enabled
  • no ETL, deploy, scheduler task, workflow dispatch, or manual data change was executed

Copilot AI lite review requested due to automatic review settings September 15, 2026 03:31

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@Sam-24-dev
Sam-24-dev merged commit c66c25d into main Sep 15, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants