Skip to content

Preserve Notion page bodies on read failures and import table cells - #1101

Merged
WaylandYang merged 3 commits into
deeplethe:devfrom
Floating-Y:fix/notion-content-integrity
Oct 8, 2026
Merged

WaylandYang merged 3 commits into
deeplethe:devfrom
Floating-Y:fix/notion-content-integrity

Conversation

@Floating-Y

Copy link
Copy Markdown
Contributor

Why

A failed Notion block read returned a title-only document, which could replace a previously imported complete body. Table row text lives in table_row.cells and was omitted by the rich-text-only renderer.

Follow-up to #1085, which added nested traversal and retained the existing title-only fallback.

What changes

  • Return body-read failures with the page ID and original diagnostic. A failed page aborts the whole fetch before ingestion, preserving stored documents for that sync round, including pages fetched earlier in the batch.
  • Render nested and paginated tables with their column headers, empty cells and rich-text segments. Headerless tables keep their first row as data; escape pipes and backslashes and render cell line breaks as <br>.
  • Add table regressions and extend the PostgreSQL-backed resync test to check that a failed read preserves the body, document versions and processing jobs, followed by a successful recovery.

How it was checked

  • cargo fmt --all --check
  • cargo clippy --workspace --all-targets --locked -- -D warnings
  • cargo test --workspace --locked against a migrated, isolated PostgreSQL 16/pgvector database, with UTOPIA_TEST_REQUIRE_DB=1 and UTOPIA_TEST_REQUIRE_PDFTOTEXT=1.
  • pnpm install --frozen-lockfile and pnpm build under web/.
  • 20 Notion tests exercised assertions, including the database resync regression. The live workspace test returned early because UTOPIA_NOTION_TEST_TOKEN was not provided. The standard workspace run did not enable ignored tests.

Before review

  • Every commit is signed off (git commit -s).
  • Formatting, workspace Clippy and workspace tests pass.

Floating-Y and others added 2 commits October 8, 2026 16:22
Signed-off-by: Floating-Y <118035379+Floating-Y@users.noreply.github.com>
…amed in the run's error, and the other pages still sync

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Wayland Yang <wayland0916@gmail.com>

@WaylandYang WaylandYang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you. Both problems are real: a failed read replaced a complete body with a title, and table cells were dropped.

The tables are right as they are. On the failed read I changed one thing and pushed it to your branch (b485050):

  • Before the push: one unreadable page aborted the whole sync. Some pages never become readable (a nested block that is not shared with the integration answers 404), so that source would stop syncing for good. Object storage taught the same lesson in #214.
  • Now: the page is left out, its stored body stays untouched, the other pages sync, and the run ends as failed with the page named in its error.

Your resync test keeps all its assertions about the stored body, versions and jobs. The failure test now has a second, healthy page beside the failing one. All 21 Notion tests pass with the database required.

Not covered by a test: the error text sync_notion builds at the end, since fetch cannot be pointed at a mock server from there.

Merging when CI is green. Thanks again.

@WaylandYang
WaylandYang merged commit 654141a into deeplethe:dev Oct 8, 2026
6 checks passed
@WaylandYang WaylandYang mentioned this pull request Oct 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants