Skip to content

fix(sync): WebDAV 同步导致应用整体卡死 —— 按书同步引擎重构 + 原生流式传输 + 响应性修复- #1 - #781

Open
jenken827 wants to merge 5 commits into
codedogQBY:mainfrom
jenken827:fix/sync-ui-freeze
Open

jenken827 wants to merge 5 commits into
codedogQBY:mainfrom
jenken827:fix/sync-ui-freeze

Conversation

@jenken827

Copy link
Copy Markdown

问题

WebDAV 同步过程中应用整体卡死:点击书籍无响应、菜单无法操作,Windows 判定应用未响应,最终直至应用退出("闪退")。未同步时 UI 响应正常;首次全量同步(数据量最大)时最严重,且随使用时间(AI 聊天记录等数据累积)持续恶化。

根因分析(四层问题叠加)

1. 传输层:plugin-http 把文件 body 变成 JS 数字数组

@tauri-apps/plugin-http 的 fetch 实现中(dist-js/index.js:68):

const data = buffer.byteLength !== 0 ? Array.from(new Uint8Array(buffer)) : null;

每本 multi-MB 的书籍上传会构造一个千万级元素的 JS 数组再走 IPC 序列化,在渲染主线程上单次阻塞数秒;下载响应体同样分块经过主线程。

2. 进度回调风暴

每个传输分块触发一次 onProgress → zustand set({progress}) → React 全量重渲染。3-5 路并发传输下每秒产生几十次重渲染,主线程被渲染任务淹没。(readest 项目在 Sentry READEST-2 中报告过完全相同的问题,并以进度节流修复——本 PR 采用同一方案。)

3. DB 写等待

runWithDbRetry 默认 waitForSyncToSettle:同步期间任何 DB 写入(包括打开书籍时的阅读记录写入)会先空等最多 12 秒才执行。这就是"同步时点击书籍打不开"的直接原因。

4. 全量快照架构(根本原因)

旧引擎每台设备维护一个 device-{id}.json 全量快照(11 张表所有行):

  • AI 聊天(messages/threads)与阅读统计(reading_sessions)只增不减,快照体积无限增长;
  • 每次同步 = 全表读出 → 巨型 JSON.stringify → PUT → 拉取所有 peer 全量 → 逐行应用。主线程 JSON 工作量与传输量随数据量线性恶化,永不收敛。

修复方案

阶段 1:响应性止血

  • 传输分块进度节流至每任务 300ms 一次(任务开始/结束仍立即上报);
  • waitForSyncToSettle 上限 12s → 1.5s(真正的 SQLite 锁冲突由既有重试机制处理);
  • applyChanges 每 25 行让出一次主线程。

阶段 2:按书同步引擎(核心重构,对齐 readest 的成熟模式)

新远端布局(/readany/sync/ 下):

路径 内容
index.json 每本书/会话的变更标记 {b: 书updated_at, a: 注解标记, d: 删除墓碑},各设备"读取-合并-写入"联合更新
books/{bookId}.json 单本书:书籍行 + 该书的全部高亮/笔记/书签 + 删除墓碑表,条目级 LWW 合并
threads/{id}.json AI 会话元数据(仅标题等小字段)
chat/{YYYY-MM-DD}.json 聊天消息按创建日分文件;设备维护"已拉取日期"游标,只拉缺失日期 + 游标之下内容有变化的日期
profile/{table}.json 标签/分组/技能等全局小表,单表单文件
sessions/{YYYY-MM}.json 阅读统计按月分片(该表只增不减,按月分片防止膨胀)

效果:每次请求负载从 MB 级降到 KB 级,同步工作量从 O(全库×设备数) 变为 O(变更数),主线程上的 JSON 工作与库大小解耦。

原生流式传输(消灭问题 1)

桌面端实现此前缺失的 uploadFile/downloadFile 平台方法:新增 Rust 命令 webdav_upload_file / webdav_download_file,用 reqwest 在 tokio 线程池中直接在磁盘与服务器之间流式读写——大文件字节完全不进入 webview 进程,下载经 IPC 通道上报进度(由阶段 1 的节流闸门控制 UI 频率)。

容错收敛(弱网/不稳定服务器)

在真实环境中发现服务器偶发 502/连接重置会让整轮同步中止。现在单条目失败仅记录并跳过(书籍推拉、目录创建、分片合并各自独立容错),index 只记录成功条目——下一轮同步依据 marker 自动补齐缺失部分,多轮收敛;index 写入失败则如实报错(该文件是成员关系的唯一事实来源,不能静默丢失)。

并发可配置

真实环境还出现弱网关在首同步并发压力下返回 502。新增"同步并发数"设置(1-6,默认 2),贯通 config → store → 文件同步各阶段池,附 7 种语言的设置 UI 文案。

验证

  • core 全量 596 个单元测试通过,新增 12 个回归测试(按书引擎 5 个:拉取应用/跳过同步/远端删除/推拉合并/聊天游标;WebDAV 客户端 7 个:PUT 404 自愈、MKCOL 409 整链、无斜杠重试、getJSON 按状态判定、目录记忆、putJSON 兜底、ensureDirectories 覆盖);
  • 真机(Windows + 内网穿透 WebDAV)验证:同步期间 UI 可正常点击、打开书籍、开关自动同步;并发数调整生效;不稳定网络下多轮同步自动收敛补齐。

兼容性说明(重要)

  • 云端同步格式无向后兼容:新旧引擎互不可见对方数据。升级后需清空远端 /readany/sync/device-*.json 并重新同步一次(一次性操作);
  • LAN 同步不受影响:LAN 协议继续使用旧引擎(设备快照点对点实时生成);
  • 本地数据库仅 sync_tombstones 新增可空 book_id 列(启动时自动迁移),其余本地结构零改动。

注:本次代码改动及PR信息由AI生成。

During a sync the app froze: clicks stopped registering, opening a book
did nothing, and the window eventually died. Three compounding causes:

- Transfer progress was pushed to the store on EVERY chunk. With 3-5
  concurrent transfers that sustains a synchronous React update storm on
  the renderer main thread (readest hit the same bug — their Sentry
  READEST-2 — and ships a progress throttle). Chunk progress is now
  throttled to one emit per 300ms per task; task start/completion still
  emit immediately.
- Every DB write wrapped in runWithDbRetry waited up to 12s for a running
  sync to finish before even attempting the write (waitForSyncToSettle),
  so opening a book stalled for the full timeout. The settle wait is now
  capped at 1.5s — genuine SQLite lock contention is already handled by
  the retry loop itself.
- applyChanges yielded the main thread only every 100 applied records;
  large remote snapshots could hog it for seconds between yields. Now
  every 25.
…ces per-device snapshots

The per-device snapshot layout (device-{id}.json = all 11 tables, full dump
per sync) was the root cause of sync dragging the whole UI down: one giant
JSON.stringify/parse per pass, O(library × devices) payload growth (AI chat
history and reading sessions grow unbounded), and every sync re-downloading
and re-applying every peer even when nothing changed.

New cloud layout (under /readany/sync):
- index.json — per-book/thread markers {b, a, d} / {t, d}, written
  read-merge-write as a union (tombstones prevent resurrection)
- books/{bookId}.json — book row + ALL its highlights/notes/bookmarks +
  per-table deleted maps (per-item LWW merge)
- threads/{threadId}.json — thread metadata only
- chat/{YYYY-MM-DD}.json — chat messages by creation day; devices keep a
  pulled-day cursor and only fetch missing days (plus days changed under
  the cursor by late offline pushes)
- profile/{tags,book_tags,book_groups,skills}.json — single-table files
- sessions/{YYYY-MM}.json — reading sessions in monthly shards

Every request is now KB-sized and sync work is O(changed). Sync also
propagates annotation deletions per book: sync_tombstones gained a book_id
column (migration + attribution at delete time + deletedBookIds on the
legacy wire format for LAN).

- LAN sync intentionally stays on the legacy device-snapshot protocol
  (runSimpleSync); cloud backends (webdav/s3) switch to runPerBookSync.
- No backward compatibility for old cloud snapshots by design: users reset
  the remote folder and re-sync once.
…ss the webview heap

plugin-http serializes request bodies via Array.from(new Uint8Array(body))
into a JSON number array over IPC: every multi-megabyte book upload froze
the renderer main thread for seconds and stalled the whole app during
sync. Implement the desktop uploadFile/downloadFile platform methods as
Tauri commands that stream directly between disk and the WebDAV server
(reqwest on the tokio pool, 300s timeout, optional insecure-TLS). The
per-book engine already prefers these entry points, so book/cover
transfers now bypass the webview entirely and downloads report progress
over an IPC channel (throttled by the engine's progress gate).
- Sync concurrency is now a setting (1-6, default 2): weak gateways/NAS
  were returning 502 under the first full-sync burst of parallel requests.
  Plumbs through config -> store -> syncFiles (upload/download/migration/
  remote-cleanup pools) with a settings UI row and 7-locale labels.
- The native Rust WebDAV upload/download now retries transient failures
  (network errors, 429, 5xx) up to 3 times with exponential backoff,
  matching the JS client's retry policy — a single gateway blip no longer
  fails a whole book transfer.
…oss runs

On flaky tunnels (502/connection-reset mid-run) a single failed request
aborted the whole sync even though 11 books had already uploaded. Per-item
failures during pull/push (books, and later phases) are now logged and
skipped; index entries are only written for successful pushes, so the next
sync retries exactly the missing pieces and converges. Directory creation
failures are also tolerated (per-file self-heal retries later).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant