fix(ingestion): persist debris records and repair the model schema (fixes #117)

[Amayyas]

User description

Description

Fixes three defects introduced by the MongoDB → PostgreSQL migration. Each has a one-line cause but an outsized effect — the first means Kepler currently ingests no debris at all, on either backend.

I found these while checking that the multi-source provider work (#15) still ran after the migration: the backend test suite does not pass on main today.

1. Debris ingestion persisted nothing 🔴

_gp_to_doc emits one satellite-shaped dict for every provider, but debris is a narrower table — no objectType, countryCode, launchDate, updatedAt, status, fuel_percentage, operational_mode. Passing those keys through made every debris write fail:

PostgreSQL: CompileError: Unconsumed column names: countryCode, launchDate, …
SQLite:     TypeError: 'objectType' is an invalid keyword argument for Debris

On PostgreSQL the whole batch is rolled back and marked failed; on SQLite it fails row by row. Either way the debris catalog stayed empty — on a platform built for debris collision avoidance.

Fix: project each document onto the target model's real columns at the write boundary, which covers both dialects and both models.

2. updatedAt written as a string into a DateTime column 🟠

now = datetime.datetime.utcnow().isoformat() produced a string for a DateTime(timezone=True) column — a MongoDB-era leftover (Mongo stored ISO strings, PostgreSQL does not). SQLAlchemy's SQLite DateTime rejects strings outright, so satellite ingestion failed there too.

Fix: a real timezone-aware datetime, which also drops the deprecated utcnow().

3. spaceWeather.recorded_at indexed twice 🟠

Declared explicitly in __table_args__ and implicitly via index=True. The generated names differ only by case:

CREATE INDEX "ix_spaceWeather_recorded_at" ON "spaceWeather" (recorded_at)
CREATE INDEX ix_spaceweather_recorded_at  ON "spaceWeather" (recorded_at)

• SQLite — identifiers are case-insensitive → create_all() aborts with "index already exists", taking 10 tests down at fixture setup.
• PostgreSQL — the mixed-case name is quoted → two redundant indexes on the same column. No error, but every write pays for both.

Fix: drop the redundant index=True, keeping the explicitly-named index that matches the convention used by the other tables (ix_satellites_noradid, ix_alerts_created_at, …).

Related Issue

Fixes #117

Checklist

• I have read the contributing guidelines
• My code follows the project guidelines
• I have completed testing of my changes
• Documentation has been updated (if applicable) — n/a, no API surface change

Testing

	Before	After
Backend test suite	40 passed, 10 errors	55 passed
Debris ingested	0	25/25 real CelesTrak records

The 5 new tests fail without these fixes, so they pin the regressions rather than merely describing them (verified by stashing the fixes and re-running):

• tests/test_models_schema.py — schema invariants: index names unique case-insensitively, no duplicate indexes on the same columns, and create_all() succeeds on a clean database. These guard the metadata itself, so a bad declaration fails at collection time instead of surfacing as a confusing error in an unrelated test.
• tests/test_ingestion.py — debris records are persisted, and updatedAt round-trips as a datetime rather than a string.

Verified end to end against live CelesTrak data, not just via tests: 25 real COSMOS 1408 DEB records now land in the debris table (previously 0), satellites ingest with a proper datetime, and a re-sync stays idempotent (no duplicates).

Breaking Changes

None. No API contract or schema shape changes — the removed index was a duplicate of one that remains, so PostgreSQL deployments simply stop maintaining a second copy. Existing databases need no migration.

---

CodeAnt-AI Description

Fix debris ingestion, timestamp storage, and schema setup failures

What Changed

• Debris records now save correctly instead of being rejected by extra fields, so debris catalogs no longer stay empty
• Satellite records now store updatedAt as a real date and time value, which avoids ingestion failures from text timestamps
• The space weather table no longer defines the same index twice, so creating a fresh database no longer fails during setup
• Added regression tests to catch debris loss, bad timestamp types, duplicate indexes, and schema creation failures

Impact

✅ Debris objects now appear in the catalog
✅ Fewer ingestion failures on fresh databases
✅ Safer schema setup during test runs

💡 Usage Guide

Checking Your Pull Request

Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.

Talking to CodeAnt AI

Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:

@codeant-ai ask: Your question here

This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.

Example

@codeant-ai ask: Can you suggest a safer alternative to storing this secret?

Preserve Org Learnings with CodeAnt

You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:

@codeant-ai: Your feedback here

This helps CodeAnt AI learn and adapt to your team's coding style and standards.

Example

@codeant-ai: Do not flag unused imports.

Retrigger review

Ask CodeAnt AI to review the PR again, by typing:

@codeant-ai: review

Check Your Repository Health

To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas for improvement in your codebase, ensuring your repository maintains high standards of code health.

Summary by CodeRabbit

• Bug Fixes

  • Improved Space-Track ingestion so debris records are saved correctly.
  • Stored satellite update timestamps in a consistent timezone-aware format.
  • Prevented redundant database indexing while preserving efficient lookups.

• Tests

  • Added coverage for debris persistence and timestamp handling.
  • Added validation for unique indexes, duplicate index prevention, and clean database setup.

Greptile Summary

This PR repairs debris ingestion and removes a duplicate schema index. The main changes are:

• Filters provider records to the target ORM model’s columns before upsert.
• Stores satellite update timestamps as timezone-aware datetimes.
• Removes the redundant implicit SpaceWeather timestamp index.
• Adds ingestion and schema tests for these fixes.

Confidence Score: 5/5

This looks safe to merge.

• No blocking issues found in the changed code.

Important Files Changed

Filename	Overview
backend/models/db_models.py	Removes the redundant implicit index while preserving the explicit recorded-at index.
backend/orbital/spacetrack.py	Filters ingestion documents by model columns and supplies a DateTime-compatible update timestamp.
backend/tests/test_ingestion.py	Adds tests for debris persistence and datetime timestamp storage.
backend/tests/test_models_schema.py	Adds checks for duplicate indexes and clean schema creation.

_{Reviews (1): Last reviewed commit: "fix(ingestion): persist debris records a..." | Re-trigger Greptile}

[codeant-ai]

🤖 CodeAnt AI — Review Status

Status	Commit	Started (UTC)	Finished (UTC)
✅ Reviewed your PR	7e07839	Jul 21, 2026 · 21:20	21:23

_{Updated in place by CodeAnt AI · last 5 reviews}

[coderabbitai]

📝 Walkthrough

Walkthrough

The PR fixes Space-Track debris upserts, stores satellite timestamps as timezone-aware datetimes, removes a duplicate SpaceWeather index, and adds ingestion and schema regression tests.

Changes

Ingestion and schema fixes

Layer / File(s)	Summary
Schema index cleanup and validation backend/models/db_models.py, backend/tests/test_models_schema.py	Removes the redundant column-level SpaceWeather.recorded_at index and validates index names, duplicate column indexes, and clean SQLite schema creation.
Space-Track persistence corrections backend/orbital/spacetrack.py, backend/tests/test_ingestion.py	Uses timezone-aware UTC datetimes, filters upsert documents to target model columns, and verifies debris persistence and satellite timestamp types.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant SpaceTrackService
  participant _gp_to_doc
  participant _bulk_upsert
  participant Satellite
  participant Debris
  SpaceTrackService->>_gp_to_doc: build document with UTC datetime
  SpaceTrackService->>_bulk_upsert: submit document and target model
  _bulk_upsert->>Satellite: persist satellite columns
  _bulk_upsert->>Debris: persist debris columns

Loading

Possibly related PRs

• 7-Blocks/Kepler#14: Covers related Space-Track ingestion adjustments and regression tests.

Suggested labels: ECSoC26, ECSoC26-L3

Suggested reviewers: krishkhinchi

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The changes and tests directly address issue #117 by fixing debris persistence, timestamp types, and duplicate indexes.
Out of Scope Changes check	✅ Passed	The PR stays focused on the linked defects and adds only supporting regression tests and schema fixes.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Title check	✅ Passed	The title clearly summarizes the main fixes: debris ingestion and schema repair, and it matches the scope of the changes.
Description check	✅ Passed	The description covers the required changes, related issue, checklist, and breaking changes, with optional sections reasonably omitted.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@backend/tests/test_models_schema.py`:
- Around line 13-15: Update Backend CI’s pytest configuration to run from the
backend package context or include backend on PYTHONPATH, so
test_models_schema.py can resolve database and related imports during
collection. Preserve the existing pytest tests/ target and ensure the schema
regression tests execute successfully in CI.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 7603fddb-a7dd-4400-b658-b6b44a163dd5

📥 Commits

Reviewing files that changed from the base of the PR and between 2510ede and 7e07839.

📒 Files selected for processing (4)

• backend/models/db_models.py
• backend/orbital/spacetrack.py
• backend/tests/test_ingestion.py
• backend/tests/test_models_schema.py

[coderabbitai]

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Make the schema test importable by Backend CI.

The configured pytest tests/ command fails during collection at Line 13 with ModuleNotFoundError: No module named 'database', producing six errors. Run tests from backend/ or configure the CI Python path to include backend so these regressions execute.

🧰 Tools

🪛 GitHub Actions: Backend CI / 0_test.txt

[error] 13-13: Pytest collection failed: ModuleNotFoundError: No module named 'database' when executing 'from database.session import Base'.

🪛 GitHub Actions: Backend CI / test

[error] 13-13: pytest collection failed due to import error: ModuleNotFoundError: No module named 'database' (from 'from database.session import Base').

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@backend/tests/test_models_schema.py` around lines 13 - 15, Update Backend
CI’s pytest configuration to run from the backend package context or include
backend on PYTHONPATH, so test_models_schema.py can resolve database and related
imports during collection. Preserve the existing pytest tests/ target and ensure
the schema regression tests execute successfully in CI.

Source: Pipeline failures

[krishkhinchi]

@Amayyas Thanks for your contribution!!

Due to the mock data, we can't merge this PR right now.