Allow partial count for findings endpoints - #7035
Merged
Merged
Conversation
Aims to address scenarios where the portfolio size (number of projects and / or findings) prevents the global findings search endpoints from completing in a reasonable time frame. I did lots of testing with portfolios of up to 100M findings, and there is no solution that works for all possible filter and sorting options. Fundamentally the endpoints present an OLAP workload, for which a transactional DB is unfit no matter how we turn it. Usually this is where one would reach for separate infra (OpenSearch, ClickHouse), but this being an on-prem OSS application with varying portfolio sizes, that's not justifiable by any means. What I ended up settling for is a solution composed of two pieces: * Adopt the concept of partial counts in API v1. Counting is the most expensive part here, because it forces Postgres to materialize the full result set before it can return data. The endpoints now support an (optional) `totalCount` query parameter where clients can opt in to partial counts to improve performance. I couldn't make it the default because it would've been a breaking change. For both non-grouped endpoints, counting caps at 10k. The grouped endpoint has no affordable count at all, and reports only a number the requested page can prove. See ADR 036 for details. * Optimize the backing queries further, e.g. by separating sorting and filtering from enrichment to make the former cheaper. While working on this I kept asking myself "why don't we just denormalize all this crap?", but always ended up discarding it for consistency and write amplification concerns. Created ADR 037 to document this for the future. Signed-off-by: nscuro <nscuro@protonmail.com>
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
Up to standards ✅🟢 Issues
|
| Metric | Results |
|---|---|
| Complexity | 14 |
🟢 Coverage 97.16% diff coverage · +0.02% coverage variation
Metric Results Coverage variation ✅ +0.02% coverage variation (-1.00%) Diff coverage ✅ 97.16% diff coverage (70.00%) Coverage variation details
Coverable lines Covered lines Coverage Common ancestor commit (3573b5a) 44410 38722 87.19% Head commit (748b0c7) 44504 (+94) 38815 (+93) 87.22% (+0.02%) Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch:
<coverage of head commit> - <coverage of common ancestor commit>Diff coverage details
Coverable lines Covered lines Diff coverage Pull request (#7035) 141 137 97.16% Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified:
<covered lines added or modified>/<coverable lines added or modified> * 100%
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
This was referenced Aug 19, 2026
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Aims to address scenarios where the portfolio size (number of projects and / or findings) prevents the global findings search endpoints from completing in a reasonable time frame.
Addressed Issue
Fixes #6570
Additional Details
Frontend PR: DependencyTrack/frontend#1756
Docs PR: DependencyTrack/docs#228
I did lots of testing with portfolios of up to 100M findings, and there is no solution that works for all possible filter and sorting options. Fundamentally the endpoints present an OLAP workload, for which a transactional DB is unfit no matter how we turn it. Usually this is where one would reach for separate infra (OpenSearch, ClickHouse), but this being an on-prem OSS application with varying portfolio sizes, that's not justifiable by any means.
What I ended up settling for is a solution composed of two pieces:
totalCountquery parameter where clients can opt in to partial counts to improve performance. I couldn't make it the default because it would've been a breaking change. For both non-grouped endpoints, counting caps at 10k. The grouped endpoint has no affordable count at all, and reports only a number the requested page can prove. See ADR 036 for details.While working on this I kept asking myself "why don't we just denormalize all this crap?", but always ended up discarding it for consistency and write amplification concerns. Created ADR 037 to document this for the future.
Checklist
This PR implements an enhancement, and I have provided tests to verify that it works as intendedThis PR introduces changes to the database model, and I have updated the migration changelog accordinglyThis PR introduces new or alters existing behavior, and I have updated the documentation accordinglydocs/adr/