perf(statistics): reduce query load on large installs

[dr-hoseyn]

Summary

• reduce /api/users/sub_update/chart from two history aggregations to one by deriving pie totals from the period rows
• remove unnecessary users joins from owner-wide subscription history and unscoped online-count queries
• add a created_at index for owner-wide user_subscription_updates time ranges
• default subscription history to one month and move aggregate user-stat polling from 2s to 30s while keeping live resource polling at 2s
• document the query-load safeguards, migration caveat, and directional benchmark

Closes #858

Type of change

• Bug fix
• New feature
• Breaking change
• Refactor / cleanup
• Documentation
• Tests / CI

Checklist

• I tested the change locally or explained why it cannot be tested.
• I added or updated tests for behavior changes.
• I updated documentation, translations, or examples if needed.
• I checked database migrations when models or schema changed.
• I did not include secrets, tokens, private keys, or unrelated changes.

Testing

• .venv/Scripts/python.exe -m pytest -q tests/test_statistics_query_load.py tests/test_system_user_metrics.py tests/api/test_usage_functions_timezone.py — 32 passed
• targeted Ruff lint and format checks — passed
• npm run build in dashboard — passed
• fresh SQLite alembic upgrade head, downgrade to 7c4bd5128e62, and upgrade again — passed; the new index appeared, disappeared, and reappeared as expected
• MySQL DDL compilation — CREATE INDEX idx_user_subscription_updates_created_at ON user_subscription_updates (created_at)

Directional in-memory SQLite benchmark, 400,000 rows per history table, median of seven warmed runs:

Query shape	Before	After	Change
Unscoped online count	406.16 ms	325.05 ms	-20.0%
Subscription chart	896.68 ms	502.13 ms	-44.0%

These numbers validate direction and query shape; they are not a production MySQL latency guarantee. The reporter's offered EXPLAIN ANALYZE output remains the best follow-up validation on the affected installation.

The three standalone tests/api/test_user.py permission cases were also attempted, but the current API harness did not seed the testadmin login and failed in the existing access_token fixture before reaching the endpoint. The operation path and response equivalence are covered by the new database-backed test.

Screenshots

Not applicable. The UI changes only adjust polling and the initial time range; layout is unchanged.

Notes for reviewers

I checked related prior and in-flight work before implementing this:

• perf(system): collapse dashboard user metrics into one query #791 is already merged and reduces /api/system/users to one SQL statement; this PR keeps it and reduces how often the full aggregate runs on the Statistics page.
• perf(frontend): integrate page visibility API for smart polling #604 slows polling only when the document is hidden. This issue reproduces while the tab is visible, so the changes are complementary.
• feat(cleanup): add automatic retention policies #801 adds retention policies, which limit future table growth but do not remove the current duplicate scan or joins.
• perf(db): bound multi-worker connection pools #789 bounds connection pools and fix(dashboard): cancel obsolete API requests at the HTTP transport #852 forwards request cancellation; neither changes these query shapes.
• perf(usage): coalesce PostgreSQL history writes #787 targets PostgreSQL write amplification and intentionally leaves MySQL unchanged.

The new index must scan the existing subscription-update table. Large MySQL/MariaDB installations should apply this migration during a maintenance window because DDL locking depends on server version and table configuration. All-time history remains available as an explicit chart selection.

Summary by CodeRabbit

• Performance

  • Improved Statistics page query performance for subscription history and user metrics.
  • Added faster time-based processing for subscription update records.
  • Reduced unnecessary database lookups in common statistics views.

• Usability

  • Subscription charts now default to displaying the past month.
  • Statistics refresh less frequently to reduce system load.

• Documentation

  • Added guidance on Statistics page performance behavior and verification.

• Tests

  • Added coverage for subscription chart totals and query behavior.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Team

Run ID: 041d3b93-cb43-41bd-9b42-9696bfa5906c

📥 Commits

Reviewing files that changed from the base of the PR and between 630b33d and caec921.

📒 Files selected for processing (8)

• app/db/crud/user.py
• app/db/migrations/versions/86f7c2a14d3e_add_subscription_update_created_index.py
• app/db/models.py
• app/operation/user.py
• dashboard/src/components/charts/user-sub-update-pie-chart.tsx
• dashboard/src/pages/_dashboard.statistics.tsx
• docs/statistics-performance.md
• tests/test_statistics_query_load.py

Included review availability: Your plan provides up to 8 included reviews per hour; 6 remain after this review.

---

Walkthrough

The Statistics page now reduces database joins and chart queries, adds an index for subscription history, limits the initial history range, slows polling, and adds SQLite coverage and performance documentation.

Changes

Statistics performance changes

Layer / File(s)	Summary
Query paths and subscription index app/db/crud/user.py, app/db/models.py, app/db/migrations/versions/...	Subscription history queries avoid the User join unless required. Online user counts also avoid that join without admin filtering. Subscription history gains a created_at index with upgrade and downgrade support.
Chart aggregation and dashboard timing app/operation/user.py, dashboard/src/components/charts/user-sub-update-pie-chart.tsx, dashboard/src/pages/_dashboard.statistics.tsx	Chart agent totals are derived from period statistics. The chart defaults to one month. System user statistics polling changes to a 30-second interval with a 15-second stale time.
Performance validation and documentation tests/test_statistics_query_load.py, docs/statistics-performance.md	Tests verify query counts, chart totals, and conditional joins. Documentation records safeguards, migration notes, verification criteria, and benchmark results.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to caec9

This reduces Statistics-page query volume and polling frequency while retaining scoped query behavior and explicit all-time history selection. Current evidence indicates the change is ready to merge.

Suggested reviewers: x0sina

Poem

A rabbit reads each line,
The patch grows clear beneath the moon,
Small changes hop in place,
Tests guard the garden path,
Reviews bloom before the dawn.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 9.09% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 11 functions across 7 files. (1 skipped: 1…	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the primary change: reducing Statistics page query load for large installations.
Linked Issues check	✅ Passed	The changes address the requirements in [#858] by reducing duplicate aggregations and unnecessary users joins, adding the subscription-history created_at index, changing the default history range, and…
Out of Scope Changes check	✅ Passed	The code, migration, dashboard updates, tests, and performance documentation all support the linked issue objective of reducing Statistics page resource usage. No unrelated changes are evident.

Full details: Docstring Coverage

Explanation

Docstring coverage is 9.09% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 11 functions across 7 files. (1 skipped: 1 unsupported.)

• Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[dr-hoseyn]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.