fix: Optimize cleanup jobs, eliminate raw ES client usage, add test coverage

[niemyjski]

Summary

Comprehensive optimization of cleanup jobs: eliminate direct Elasticsearch client usage from jobs, fix critical bugs, preserve cleanup correctness under concurrent ingestion, and add full integration test coverage.

Critical Bug Fixes

@min_count:2 → @min:2 in GetDuplicateSignaturesAsync

The @min_count syntax is silently ignored by Foundatio Parsers, causing the aggregation to return ALL signature hashes (not just duplicates). This would have caused FixDuplicateStacksAsync to treat every stack as a duplicate on the next run.

FixDuplicateStacksAsync only ran one batch (regression from refactor)

The original code looped until GetDuplicateSignaturesAsync returned empty. The refactor broke this. Restored the loop with:

• Batch tracking and per-batch logging
• Infinite-loop guard: exits if an entire batch fails to process
• ImmediateConsistency() on CountAsync inside GetDuplicateSignaturesAsync (one refresh per batch, matching the original Indices.RefreshAsync call pattern — NOT per item)

ReassignStackAsync data-loss hazard on empty sequence

If sourceStackIds was empty, PatchAllAsync would have no stack filter and would reassign ALL events to the target stack. Added materialization + early return guard.

FixDuplicateStacksAsync event-first ordering

Event reassignment completes before a duplicate stack is finalized as a redirect tombstone. If event reassignment fails, duplicate stacks remain visible and no data is lost.

GetDistinctFieldValuesAsync cursor correctness

Distinct event identifiers use composite aggregation pagination with an explicit next-value cursor so callers reliably detect the end of pagination.

Architecture (eliminate raw ES client from jobs)

• Refactored CleanupOrphanedDataJob to use repository methods exclusively
• Added GetDistinctFieldValuesAsync using composite aggregation (encapsulated in the repository — composite aggregation is not in Foundatio's DSL, so raw client use is justified and documented)
• Added RemoveAllByProjectIdsAsync/RemoveAllByOrganizationIdsAsync and RemoveAllByStackIdsAsync to IEventRepository
• Added ReassignStackAsync using a parameterized Painless script and strict post-write verification
• Added GetDuplicateSignaturesAsync to IStackRepository

Concurrency and lossless reconciliation

• Added durable duplicate-stack redirects with canonical target resolution, cycle detection, and late-event reconciliation
• Added an idempotent contribution ledger so retrying duplicate merges cannot double-count occurrences, including transitive redirect chains
• Added optimistic concurrency to Stack persistence so stale full-document saves cannot overwrite cleanup merges
• Replaced full-document writes in hot update paths with retrying repository patches for tags, status, regression state, and event counters
• Kept internal redirect/version fields out of the public JSON contract while persisting them in Elasticsearch
• Treat missing event indexes as empty only when Elasticsearch explicitly reports index_not_found_exception; other shard/index verification failures fail closed

Other Fixes

• Added OperationCanceledException handling before the intentionally broad per-signature catch
• Added lock renewal at page boundaries and while awaiting long-running reassignment operations
• Removed redundant is_deleted:false filters where repository soft-delete filtering applies
• Kept page size at 5 for organization/project cleanup (2.5s sleep/item makes larger pages impractical)
• Aligned async method and test names with repository conventions
• Added :{Message} to error log format strings

Merged onto current main

Merged the latest origin/main with a normal merge commit and kept the final diff limited to the intended cleanup, repository, serialization, and test files.

Test Coverage

Coverage includes:

• Cleanup pagination, retention, usage accounting, and tenant isolation
• Orphaned stack/project/organization events, including large and multi-page datasets
• Duplicate signature lookup, soft-delete filtering, empty-source reassignment, and closed/missing event indexes
• Partial failure and retry idempotency, redirect chains/cycles, late events/counters, and stale saves
• Concurrent tag, status, regression, and event-counter updates
• Public serialization and API compatibility for internal persistence fields

[Copilot]

Pull request overview

This PR refactors cleanup jobs to rely on repository abstractions, improves duplicate stack cleanup behavior, and adds integration coverage for cleanup/repository operations.

Changes:

• Reworked orphaned-data and cleanup jobs with lock renewal, cancellation checks, and repository-based deletes/updates.
• Added event repository helpers for bulk deletion, stack reassignment, and distinct-id aggregation.
• Added integration tests for cleanup pagination, retention, duplicate signatures, and event repository operations.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/Exceptionless.Core/Jobs/CleanupDataJob.cs	Extends lock duration and renews locks during paged cleanup/retention loops.
src/Exceptionless.Core/Jobs/CleanupOrphanedDataJob.cs	Replaces direct Elasticsearch calls with repository methods for orphan cleanup and duplicate-stack fixing.
src/Exceptionless.Core/Repositories/EventRepository.cs	Adds bulk delete, stack reassignment, and composite aggregation helpers.
src/Exceptionless.Core/Repositories/Interfaces/IEventRepository.cs	Exposes new event repository cleanup/query APIs and composite cursor type.
src/Exceptionless.Core/Repositories/StackRepository.cs	Adds duplicate signature aggregation lookup.
src/Exceptionless.Core/Repositories/Interfaces/IStackRepository.cs	Exposes duplicate signature lookup API.
tests/Exceptionless.Tests/Jobs/CleanupDataJobTests.cs	Adds cleanup pagination and retention integration coverage.
tests/Exceptionless.Tests/Jobs/CleanupOrphanedDataJobTests.cs	Adds integration coverage for orphan cleanup and duplicate stack merging.
tests/Exceptionless.Tests/Repositories/EventRepositoryTests.cs	Adds coverage for distinct ids, stack reassignment, and bulk delete helpers.
tests/Exceptionless.Tests/Repositories/StackRepositoryTests.cs	Adds duplicate signature repository coverage.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

[niemyjski]

Merged origin/main and addressed the outstanding review feedback in 92fa434.

Summary:

• Removed the accidental packages/ diff from the PR; the final diff against origin/main is back to the intended cleanup/repository/test files.
• Added/kept integration coverage for the new repository methods, duplicate signature lookup, soft-delete filtering, empty ReassignStackAsync safety, orphan cleanup, and cleanup usage accounting.
• Updated duplicate-stack cleanup to loop until no duplicate signatures remain and to renew locks during long-running cleanup paths.
• Documented the composite aggregation cursor path and aligned it with the current Elastic client API.
• Treated missing event indexes as zero matching events during event cleanup, which fixes empty soft-deleted project cleanup.

Verification:

• dotnet test -- --filter-class Exceptionless.Tests.Jobs.CleanupDataJobTests: 26 passed
• dotnet test -- --filter-class Exceptionless.Tests.Jobs.CleanupOrphanedDataJobTests: 24 passed
• dotnet test -- --filter-class Exceptionless.Tests.Repositories.EventRepositoryTests: 17 passed, 1 skipped performance test
• dotnet test -- --filter-class Exceptionless.Tests.Repositories.StackRepositoryTests: 13 passed

All review threads are resolved.

[niemyjski]

Production-readiness pass complete on f6fcc4b.

• Merged current origin/main with normal merge commit 65fe08e.
• Added lossless concurrent cleanup hardening, exact event-index verification/fail-closed behavior, durable/transitive duplicate-stack redirects, bounded late-event reconciliation, optimistic concurrency, and patch-based hot update paths.
• Added integration coverage for closed/missing indexes, partial failure/idempotency, stale saves, chain/cycle redirects, late events and counters, tenant deletion, serialization boundaries, and concurrent tag/status/regression updates.
• Local verification: dotnet build --no-restore -m:1 passed with 0 warnings/errors; dotnet test --no-build passed 2336 tests with 2 intentional performance-test skips and 0 failures.
• Current Build workflow is green: version, test-api with coverage, test-client, test-e2e with localhost AppHost, and docker-build all succeeded. Non-PR deploy/publish/preview jobs were skipped as designed.
• Final GraphQL audit: all 21 review threads are resolved; 0 unresolved actionable threads. PR merge state is CLEAN.

Production-readiness verdict: READY.

[niemyjski]

Production-readiness refresh complete on bd560b0.

• Merged current origin/main (87af2a0) with normal merge commit cfc28b7; no rebase or force-push.
• Re-reviewed the full 18-file PR diff for correctness, compatibility, security, test coverage, and maintainability.
• Fixed the remaining documented convention issue by renaming FixDuplicateStacksAsync and aligning new test names with MethodUnderTest_Scenario_ExpectedBehavior.
• Updated the PR description to document the durable redirect/reconciliation, optimistic-concurrency, and patch-based hot-path hardening already present in the final diff.
• Local build: dotnet build --disable-build-servers --no-restore -m:1 passed with 0 warnings and 0 errors.
• Local service-free contract proof: PublicApiCompatibilityTests passed 4/4. The service-backed local suite was contaminated by concurrent shared Elasticsearch state (alias/index collisions, duplicate fixed test ids, missing shards); isolated CI is authoritative.
• GitHub Actions run 29467194402 is green: version, test-api with the full .NET suite and coverage, test-client, localhost Aspire/Playwright test-e2e, and docker-build all succeeded. Deploy/publish/preview-only jobs skipped as designed.
• Final GraphQL audit: 21/21 review threads resolved, 0 unresolved; merge state CLEAN; no incoming main/head/review changes.

Production-readiness verdict: READY. PR remains open and unmerged.

[niemyjski]

Follow-up reviewer audit complete on c2bcdc6.

Feedback ledger

• 21 review-thread roots and 38 inline comments were re-read against the current 18-file diff. All 21 threads were already resolved and outdated; no open thread required a new reply or resolution.
• Already fixed/outdated in current code and focused coverage: #3321326717, #3327871089, #3327871203, #3327871916, #3327872423, #3327872834, #3327873570, #3327875527, #3327876881, #3327877605, #3327877922, #3327878657, #3327879396, #3327880341, #3327881304, #3327881626, #3327927526, and #3327927540. These cover the obsolete cursor null-dereference shape, test naming/AAA, repository integration coverage, soft-delete filtering, pattern style, composite pagination documentation and record shape, structured logging, batch save/lock renewal, empty-source safety, and duplicate aggregation pagination.
• Stale/incorrect for the current contract: #3321326725 and #3327911191 duplicate generic-catch findings. Cooperative cancellation is explicitly rethrown; the remaining per-signature catch intentionally isolates corrupt, repository, transport, and transient failures so one duplicate group does not abort the whole cleanup job. Restricting it to TimeoutException and InvalidOperationException would regress that contract.
• Superseded by diff cleanup: #3329055683 targeted generated packages/ content that is no longer in the PR.
• Submitted reviews: Copilot review #4394198870 is an informational overview whose two findings are fixed above; #4397767971 was unable to review temporary 300-plus-file generated churn and is superseded by the current 18-file diff. The remaining submitted reviews have empty bodies and only contain the inline comments classified above.
• Top-level comments #4742267061, #4952560180, and #4987690483 are historical readiness summaries; #5081447901 is the current informational 77 percent coverage report. None contains an unresolved action.

RCA and current design proof

• Empty source ids previously omitted the stack filter and could match every event; ReassignStackAsync now materializes and rejects an empty source set, with regression coverage.
• A single duplicate-signature terms page could strand groups above the cap; cleanup now refreshes and loops until no duplicate buckets remain.
• Concurrent and in-flight updates could make duplicate cleanup lossy; durable redirects, strict reassignment verification, an idempotent occurrence ledger, reconciliation markers, optimistic concurrency, and patch-based hot updates preserve late events and metadata.
• Null duplicate metadata could be encoded as DateTime.MinValue and persisted as fake dates; the current head uses explicit presence flags and regression assertions.

Thermo-nuclear review

• Reviewed the complete branch-vs-main diff for code-judo opportunities, branching growth, boundaries, canonical helpers, sequential orchestration, and partial updates. No production file crosses 1,000 lines; the largest is 568 lines.
• Raw Elasticsearch operations are confined to the EventRepository boundary for composite pagination and strict closed-index verification. Cleanup jobs use repository abstractions.
• The serialized tag-patch loop is intentional cluster backpressure for batches of up to 1,000 distinct stacks; unbounded parallelization would create a new Elasticsearch burst rather than a safe simplification.
• No proven structural or correctness defect remains, so this audit made no speculative code change.

Verification

• dotnet build -m:1: passed, 0 warnings, 0 errors.
• PublicApiCompatibilityTests: 4/4 passed.
• StackRepositoryTests local attempt: 24 fixture-initialization errors before assertions because concurrent local hosts deleted and recreated the shared test-* indices, yielding index_not_found_exception and invalid_alias_name_exception. This is isolated as shared infrastructure state, not a branch failure.
• Authoritative clean run 30183197271 for this exact SHA: version, test-api with full .NET tests and coverage, test-client, localhost Aspire/Playwright test-e2e, and docker-build all passed.
• git diff --check passed; live PR state was OPEN, MERGEABLE, and CLEAN.

Verdict: READY. No code changes were required in this follow-up audit; the PR remains open and unmerged.

[niemyjski]

Final destructive-cleanup and query audit complete on ba1aca0b547689b9e7a7fbddefb010c6c7cab9e7, after merging current main normally.

Coverage and confidence

This repository does not have 100% coverage, so I cannot honestly claim that no change can ever break anything. The exact-head hosted result is 78% line coverage (23,782/30,603) and 67% branch coverage (11,127/16,628); Core is 75%/67%.

The authoritative isolated run is green: 2,742 .NET tests, 2,740 passed, 0 failed, and 2 intentional performance-test skips. All four new CleanupOrphanedDataJobSafetyTests ran and passed. Version, client, localhost Aspire/Playwright E2E, all Docker image builds, and CLA are also green. All 25 review threads are resolved.

Deletion-safety findings and fixes

• The audit found a real fail-open path in the installed Foundatio multi-get behavior: Elasticsearch can return HTTP 200 with a per-item closed/missing-index error, and the standard helper logs and omits that item. Cleanup could therefore interpret an unreadable parent as missing and delete its events.
• The cleanup paths now use a strict real-time MGET at the repository boundary. It preserves the existing one-request/source-filtered wire shape, but throws on top-level errors, every per-item error, malformed/null/mismatched sources, and omitted/duplicate/unexpected IDs before any delete-by-query can run.
• Three real-Elasticsearch tests close the stack, project, and organization indices independently and prove cleanup fails while preserving the event.
• A second race was fixed: if a soft-deleted stack is restored between the active-parent read and the all-soft-deletes recheck, it is now recognized as active and excluded from deletion. A deterministic BeforeGet integration test forces that exact interleaving and proves the stack and event survive.
• NeedsRedirectReconciliation remains necessary. Buffered counter/stat writes can land on a redirect tombstone after every source event has moved; with no source event left, the dirty marker is the durable retry signal. The inline reply documents that invariant and its increment/set/CAS/recovery coverage.

One residual limit is explicit: parent validation and event delete-by-query are separate cross-index operations. Supported ingestion writes the parent before the event, which satisfies the runtime invariant, but an out-of-contract event-first import or concurrent parent restore after the final check can still race cleanup. Eliminating that class absolutely would require an age grace period, two-pass quarantine, or cross-index coordination. Green tests cannot turn that non-atomic design into a mathematical guarantee.

Query/performance verdict

The result is mixed, not uniformly “faster”:

• Orphan ID discovery is more exact and scalable: bounded composite after_key pages replace approximate cardinality plus partitioned terms scans. It removes the initial cardinality request and cannot truncate/skew IDs, but end-to-end latency has not been benchmarked.
• Parent checks still issue one real-time, source-filtered MGET per batch; strict validation adds only O(n) client-side checks over at most 500 items. The stack-miss path intentionally performs a second MGET to protect redirect tombstones and concurrently restored stacks.
• Normal orphan deletion remains the same terms delete-by-query shape; Foundatio adds unavailable-index handling and conflict retries.
• Duplicate reassignment/reconciliation and per-stack tag patching intentionally add reads/writes to prevent stale overwrites, missed conflicts, and data loss. These are safety costs, not performance optimizations. The many-unique-tagged-stacks pipeline is the highest-priority latency benchmark.

CI proves behavior, not performance parity. A defensible performance gate would compare main and this head against cloned identical Elasticsearch/Redis snapshots, with warmed runs at 499/500/501/10k/50k IDs and duplicate/tag workloads, capturing request counts, bytes, Elasticsearch took, conflicts/batches/throttling, and median/p95 wall time under an explicit regression budget.

Verdict: merge-ready for the supported parent-before-event runtime invariant, with substantially stronger fail-closed behavior and exact-head integration proof. It is not 100%-covered, not an absolute cross-index no-loss guarantee, and not yet performance-benchmarked.

[niemyjski]

Upstream follow-up for the strict MGET safety finding:

• Filed Foundatio.Repositories #313 with the HTTP-200/per-item-error reproduction, partial-result/cache-poisoning impact, and missing coverage.
• Opened Foundatio.Repositories PR #314, adding an opt-in ThrowOnMultiGetErrors() command option. It preserves the existing default for compatibility, throws before returning/caching partial results when enabled, and covers both a closed index and a mixed time-series success/error response.
• Upstream local validation: full solution build passed with 0 warnings/errors; 841/841 tests passed; formatting verification passed. Hosted upstream checks are running.

Exceptionless should retain its current fail-closed adapter until a released Foundatio.Repositories package contains #314. After upgrading, the intended cleanup is to replace the duplicated raw MGET implementation and internal interface-to-base-class cast with canonical GetByIdsAsync(...ThrowOnMultiGetErrors()) calls while keeping the stack/project/organization outage-preservation tests.

I also merged the latest origin/main into this PR normally (283d7ee89); those five incoming commits are frontend-only. Fresh exact-head Exceptionless CI is running.

[niemyjski]

Final exact-head thermo and strict-MGET follow-up:

• Current head 283d7ee892089bcbf2722581a5d58b9dcdcae0dd includes current main. Build run 31554231590 is fully green: version, API/.NET, client, E2E, Docker, and CLA.
• .NET: 2,742 total; 2,740 succeeded; 0 failed; 2 skipped. The only skips are the unrelated performance-data generators. All four CleanupOrphanedDataJobSafetyTests therefore ran and passed in the unfiltered isolated suite.
• Coverage is 78% line / 67% branch overall (Core 75% / 67%). This is not 100% coverage, so I am not claiming mathematical zero-risk; the confidence comes from the targeted destructive-path, outage, pagination, redirect, race, recovery, and idempotency tests plus the full suite.
• All 25 live review threads are resolved. The PR is CLEAN and MERGEABLE.

The MGET behavior was a real upstream Foundatio.Repositories gap: HTTP-200 per-item errors were logged and omitted, making them indistinguishable from missing documents. I filed issue #313 and opened PR #314. Both hosted build runs pass 841/841 tests with 0 failures/skips; CodeQL and CLA are green. The fix is opt-in/backward-compatible and adds no request round trip: strict mode changes only per-item response validation.

Thermo verdict: no remaining blocking finding in this PR. Keep the local fail-closed adapter until #314 is released and the package is upgraded; then replace it with canonical GetByIdsAsync(...ThrowOnMultiGetErrors()) while retaining the four application-level outage/race tests. The remaining parent-check/delete TOCTOU is a pre-existing cross-index architectural boundary; normal ingestion preserves parent-before-event ordering. Supporting event-first imports or concurrent restores with an absolute no-overdelete guarantee would require a separate grace/two-pass/quarantine design.

[github-actions]

Package	Line Rate	Branch Rate	Complexity	Health
Exceptionless.Insulation	37%	35%	286	❌
Exceptionless.Core	76%	68%	10304	✔
Exceptionless.Web	85%	69%	7903	✔
Exceptionless.AppHost	38%	41%	147	❌
Summary	78% (25465 / 32485)	67% (11915 / 17666)	18640	✔