Repository navigation
Commit 916a355
authored
fix(knowledge): bound the members-mode tombstone pass by what a run changed (#8172)
* fix(knowledge): bound the members-mode tombstone pass by what a run changed
The tombstone phase of the member document lifecycle ran one anti-join over
every live document of the connector on every run, and LIMIT could not bound
it because almost every document is observed. On a large connector that
statement outgrew the statement timeout, rolled back the member's page, and
walked the connector toward disabled.
Tombstoning is now driven by the documents whose observations the run itself
removed (unseen after a complete listing, withdrawn by a change feed, or held
by a member being removed), then by a resumable backstop that walks the
reconciliation index one page per statement and checks observations only in
the UPDATE over that page's live ids. The backstop saves its position on the
connector and does a bounded number of pages per run. Resurrection and purge
are unchanged.
* fix(knowledge): walk the members-mode reconcile by external id and tombstone explicit removals without a completed listing
The absence reconcile ordered its resumable walk by source_seen_at, which
every member listing rewrites for what it observed. On a connector larger than
one run's page budget the walk chased re-stamped documents and a pass never
ended, so a document that lost its observers behind the cursor was never
revisited. The walk now keys on external id through
doc_connector_external_id_idx, which a document never changes, so a pass
completes within ceil(documents / budget) runs.
Documents a run explicitly unobserved (a complete listing, a change-feed
withdrawal, or a member removal) are now tombstoned even when no member with a
completed listing remains, as the stale-member sweep already does. Removing
the only listed member previously left every document only it observed live
and unobserved indefinitely. The reconcile and the purge stay gated on a
completed listing.
* fix(knowledge): keep member removal ids across interrupted runs and hide the reconcile cursor
- A member removal interrupted by the run deadline resumed from its saved
document cursor and only handed the lifecycle the documents above it. The
resumed walk now re-reads the removed member's observations below the
cursor first. They still exist until the member row is deleted.
- The absence reconcile writes its cursor once per run instead of once per
page, and documents how many runs a pass takes.
- Connector presenters drop member_tombstone_cursor alongside the stored API
key, so the internal cursor never reaches API responses.
- CI runs member-document-lifecycle.integration.ts with the other knowledge
PostgreSQL suites.
* chore(db): drop the tombstone cursor migration to regenerate it after staging
* chore(db): regenerate the tombstone cursor migration on top of staging
* chore(db): format the tombstone cursor migration metadata
* fix(knowledge): tombstone a removed member's sole documents with the removal itself
A membership removal handed the documents its member alone observed to the
lifecycle through run memory. If the run stopped after the member row was
deleted, the ids were gone, the observations had cascaded away, and with no
member left with a completed listing the absence reconcile never ran. Those
documents stayed live indefinitely.
Each removal page now tombstones the documents no other member observes in
the same transaction that advances the removal's checkpoint, so nothing
depends on the run surviving to its lifecycle. This applies only where
observations decide existence; a corpus owned by a dedicated content
credential is left alone, as before. The in-memory re-collection added
earlier is removed.
The absence reconcile's external_id IS NOT NULL filter is documented. Every
writer that sets connector_id copies a required external id, so the filter
excludes nothing.
* fix(knowledge): resurrect a withdrawn removal's tombstones with the member's restored ACLs
A multi-run member removal tombstones, page by page, the documents only that
member observes. If directory re-listing withdrew the removal mid-walk, the
restarted walk restored the member's token to those ACLs, but the documents
stayed deleted until the lifecycle's resurrection phase ran. That phase runs
only after the member loop finishes before the deadline.
A membership walk that is not a removal now applies the lifecycle's own
resurrect predicate to each page, in the same transaction. The predicate is
shared through one helper and covers: connector-owned, not excluded, not
archived, content present, observed. Only where observations decide
existence.
* test(knowledge): insert the content-less fixture row with its own typed insert1 parent 884c602 commit 916a355
16 files changed
Lines changed: 29358 additions & 110 deletions
File tree
- .github/workflows
- apps/sim/lib/knowledge
- __integration__
- application
- connectors
- orchestration
- packages
- db
- migrations
- meta
- testing/src/mocks
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
274 | 274 | | |
275 | 275 | | |
276 | 276 | | |
| 277 | + | |
277 | 278 | | |
278 | 279 | | |
279 | 280 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
| 6 | + | |
6 | 7 | | |
7 | 8 | | |
8 | 9 | | |
9 | 10 | | |
10 | 11 | | |
11 | 12 | | |
12 | | - | |
| 13 | + | |
13 | 14 | | |
14 | 15 | | |
15 | 16 | | |
| |||
19 | 20 | | |
20 | 21 | | |
21 | 22 | | |
| 23 | + | |
22 | 24 | | |
| 25 | + | |
| 26 | + | |
23 | 27 | | |
24 | 28 | | |
| 29 | + | |
25 | 30 | | |
26 | 31 | | |
27 | 32 | | |
| |||
62 | 67 | | |
63 | 68 | | |
64 | 69 | | |
65 | | - | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
66 | 77 | | |
67 | 78 | | |
68 | 79 | | |
69 | 80 | | |
70 | 81 | | |
| 82 | + | |
71 | 83 | | |
72 | 84 | | |
73 | 85 | | |
| |||
108 | 120 | | |
109 | 121 | | |
110 | 122 | | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
111 | 373 | | |
112 | 374 | | |
113 | 375 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
98 | 98 | | |
99 | 99 | | |
100 | 100 | | |
| 101 | + | |
101 | 102 | | |
102 | 103 | | |
103 | 104 | | |
| |||
548 | 549 | | |
549 | 550 | | |
550 | 551 | | |
551 | | - | |
552 | | - | |
553 | | - | |
554 | | - | |
555 | | - | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
556 | 560 | | |
557 | 561 | | |
558 | 562 | | |
| |||
709 | 713 | | |
710 | 714 | | |
711 | 715 | | |
712 | | - | |
| 716 | + | |
713 | 717 | | |
714 | 718 | | |
715 | 719 | | |
| |||
0 commit comments