feat: tag batch commit so the reconciler reaps abandoned batch payloads - #2518
Conversation
|
"To enable tagging on a transaction, transactionTag must be set to the same value for all requests belonging to the same transaction, including sessions.beginTransaction." (from https://docs.cloud.google.com/spanner/docs/reference/rest/v1/RequestOptions) is a little confusing. But it sounds like we need to set it for every single request. |
| .await? | ||
| .params(sqlparams) | ||
| .param_types(sqlparam_types) | ||
| .transaction_tag(BATCH_COMMIT_TRANSACTION_TAG) |
There was a problem hiding this comment.
If we can tag at statement level then this is not the right place since the upsert doesn't delete anything.
1608c10 to
8d9cc8a
Compare
@taddes I don't think the latest commit is doing that with the transaction tag. |
4662648 to
f2f5ca0
Compare
There was a problem hiding this comment.
Looks good,
I suppose another way of wiring the transaction_tag down to Db would be include it in the lock_for_xxx method's params::LockCollection (those methods already take &mut self).
Alternatively you could also throw something in HttpRequest::extensions_mut to pass the tag over to DbPool, but with_transaction_tag is more explicit anyway.
Follow up to STOR-657. The reconciler was skipping the delete for every batch_bsos row removal, which fixed the commit handoff bug but leaked the GCS object whenever a batch was genuinely removed, either by TTL expiry or by deleting a user's storage or a collection.
This tags the batch commit transaction in the Spanner commit_batch path with a batch_commit transaction tag. Spanner records that tag on the change stream for the whole transaction, including the cascade batch_bsos deletes. The reconciler now skips a batch_bsos removal only when it carries that tag, and treats any other batch_bsos removal as a real delete, so TTL expiry and storage or collection deletes now clean up their objects.
The publishers already carry transactionTag on the wire from STOR-657, so no publisher change is needed here. The tag path cannot be exercised on the Spanner emulator, so the Rust change is covered by compile and review and the reconciler logic by unit tests.
Closes STOR-668