Refactor statistics + split content presentation and add api

.github/workflows/ci.yml

@@ -9,8 +9,11 @@ on:
     paths:
       - '.github/workflows/**'
       - 'rdmo_plugins_statistics/**'
+      - 'scripts/**'
+      - 'tests/**'
       - .pre-commit-config.yaml
       - pyproject.toml
+      - README.md
 
 # Ref: https://docs.github.com/en/actions/using-jobs/using-concurrency
 concurrency:
@@ -33,7 +36,7 @@ jobs:
       with:
         python-version: "3.12"
         cache: pip
-    - run: python -m pip install --upgrade pip setuptools wheel
+    - run: python -m pip install --upgrade pip
     - run: python -m pip install -e .[dev]
     - name: Set up pre-commit cache
       uses: actions/cache@v3
@@ -59,11 +62,45 @@ jobs:
       - run: python -Im pip install -e .[dev]
       - run: python -Ic 'import rdmo_plugins_statistics; print(rdmo_plugins_statistics.__version__)'
 
+  test:
+    name: "Test (${{ matrix.name }})"
+    runs-on: ubuntu-22.04
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - name: RDMO 2.5
+            rdmo: "rdmo>=2.5,<3"
+          - name: RDMO 3.0 release branch
+            rdmo: "rdmo @ git+https://github.com/rdmorganiser/rdmo.git@3.0.0/release"
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v4
+        with:
+          python-version: "3.12"
+          cache: pip
+      - run: python -m pip install --upgrade pip
+      - name: Install RDMO target
+        run: python -m pip install "${{ matrix.rdmo }}"
+      - name: Install plugin
+        run: python -m pip install -e '.[dev]'
+      - name: Run tests
+        run: pytest
+      - name: Build wheel
+        run: python -m pip wheel --no-deps . --wheel-dir dist
+      - name: Verify wheel assets
+        run: |
+          unzip -l dist/*.whl | grep -q 'rdmo_plugins_statistics/templates/rdmo_plugins_statistics/statistics.html'
+          unzip -l dist/*.whl | grep -q 'rdmo_plugins_statistics/locale/de/LC_MESSAGES/django.mo'
+          unzip -l dist/*.whl | grep -q 'rdmo_plugins_statistics/static/rdmo_plugins_statistics/css/statistics.css'
+          unzip -l dist/*.whl | grep -q 'rdmo_plugins_statistics/static/rdmo_plugins_statistics/js/chart.umd.min.js'
+
   required-checks-pass:
     if: always()
     needs:
       - lint
       - dev-setup
+      - test
     runs-on: ubuntu-22.04
     steps:
       - uses: re-actors/alls-green@release/v1

NOTICE

@@ -4,7 +4,7 @@ THIS NOTICE APPLIES TO ALL FILES CONTAINED IN THIS AND UNDERLYING FOLDERS
 
 RDMO - Research Data Management Organiser
 
-Copyright (c) 2024  RDMO Arbeitsgemeinschaft <rdmorganiser.github.io>
+Copyright (c) 2015-2026 RDMO Community and individual contributors
 
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -14,3 +14,8 @@ distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
+
+Third-party components retain their respective licenses.
+
+Chart.js is distributed under the MIT License. See:
+rdmo_plugins_statistics/static/rdmo_plugins_statistics/js/chart.LICENSE.md

README.md

@@ -1,6 +1,6 @@
 # RDMO Statistics Plugin
 
-The RDMO Statistics Plugin adds a statistics page to RDMO and displays project, user, catalog usage, and project progress data as bar charts.
+The RDMO Statistics Plugin adds a statistics page to RDMO and displays project, user, catalog usage, and project progress data as charts.
 
 ## Features
 
@@ -10,11 +10,14 @@ The statistics page currently provides:
 - Number of newly registered users over time
 - Cumulative total users
 - Catalog usage by number of projects
-- Progress of individual projects
+- Distribution of projects by ten-point progress groups
+- Summary cards for projects and users
 - Daily, monthly, quarterly, and yearly aggregation for the time-based charts
-- Optional start and end date filters
+- Shared start and end date filters for all time-based charts
 - Totals for the displayed time range and the system's overall total where applicable
-- Persistent interval and date-filter selection for time-based charts using browser storage
+- Persistent and shareable interval and date-filter selection using browser storage and URL parameters
+- Accessible data tables, semantic tooltips, and explicit empty states for every chart
+- Complete catalog usage display
 - CSV export for every chart
 
 Access to the statistics page is restricted to site managers and controlled by the statistics.view_statistics permission.
@@ -23,9 +26,11 @@ Project, catalog statistics and user registrations are restricted to the current
 
 ## Requirements
 
-- RDMO 2.5.x or later
+- RDMO 2.5.x
+- RDMO 3.0 compatibility is tested against the RDMO 3.0 release branch until 3.0 is published.
+- Python 3.10 or later, according to the supported RDMO version
 
-Chart.js 4.5.1 is included with the plugin as a static asset. No npm installation or separate JavaScript build step is required.
+Chart.js is included with the plugin as a static asset. No npm installation or separate JavaScript build step is required.
 
 ## Installation
 
@@ -50,6 +55,7 @@ from django.urls import include, path
 
 urlpatterns += [
     path('statistics/', include('rdmo_plugins_statistics.urls')),
+    path('api/v1/', include('rdmo_plugins_statistics.urls.v1')),
 ]
 ```
 
@@ -69,12 +75,11 @@ and add the following where the navigation entry should appear:
 
 ```django
 {% has_perm 'statistics.view_statistics' request.user as can_view_statistics %}
+{% url 'statistics:index' as statistics_url %}
 
-{% if can_view_statistics %}
+{% if can_view_statistics and statistics_url %}
 <li>
-    <a href="{% url 'statistics:index' %}">
-        {% trans 'Statistics' %}
-    </a>
+    <a href="{{ statistics_url }}">{% trans 'Statistics' %}</a>
 </li>
 {% endif %}
 ```
@@ -90,41 +95,42 @@ The default chart configuration is:
 ```python
 RDMO_STATISTICS = {
     'projects': {
-        'bar_color': '#7eafe0',
+        'chart_color': '#7eafe0',
         'empty_periods': True,
         'label_orientation': 'auto',
     },
     'users': {
-        'bar_color': '#65c5c4',
+        'chart_color': '#65c5c4',
         'empty_periods': True,
         'label_orientation': 'auto',
     },
     'cumulative_users': {
-        'bar_color': '#65c5c4',
+        'chart_color': '#65c5c4',
         'empty_periods': True,
         'label_orientation': 'auto',
     },
     'catalogs': {
-        'bar_color': '#a8d37d',
-        'label_orientation': 'horizontal',
+        'chart_color': '#a8d37d',
+        'label_orientation': 'auto',
         'orientation': 'horizontal',
     },
     'project_progress': {
-        'bar_color': '#e6a15c',
-        'label_orientation': 'horizontal',
-        'orientation': 'horizontal',
+        'chart_color': '#e6a15c',
+        'label_orientation': 'auto',
     },
 }
 ```
 
 The chart configuration can be overridden in `rdmo-app/config/settings/local.py` using the optional `RDMO_STATISTICS` setting. Only the values that should differ from the defaults need to be specified.
 
+All charts are displayed as bar charts. Project progress is always vertical.
+
 Currently, the following configuration options are supported:
 
-- `bar_color`
+- `chart_color`
 - `empty_periods` (time-based charts): include periods without new records when set to `True`
-- `label_orientation`
-- `orientation` (category charts): display bars `horizontal` or `vertical`
+- `label_orientation`: `auto`, `horizontal` or `vertical` (only effective on vertical bar charts)
+- `orientation` (catalog usage only): display the chart `horizontal` or `vertical`
 
 ## Displayed Statistics
 
@@ -157,22 +163,113 @@ Catalogs assigned to the current site are displayed together with the number of
 
 ### Project progress
 
-Every project belonging to the current site is displayed with its interview progress as a rounded percentage. Projects whose interview has not started are shown with 0% progress.
+The project-progress bar chart groups all projects belonging to the current site into fixed ten-point ranges from 0–9% through 90–99%, with a separate 100% group. The percentage groups are shown on the x-axis and project counts on the y-axis. Projects whose interview has not started are included in the 0–9% group.
 
 ## Frontend implementation
 
-The Django view serializes the statistics with Django's `json_script` template filter. The bundled `statistics.js` reads that data and creates the charts with Chart.js 4.5.1.
+Site-scoped queries and domain aggregates live in `rdmo_plugins_statistics.statistics`. The JSON API and server exports use these aggregates directly. The Django view passes them to `compute_dashboard_charts` in `rdmo_plugins_statistics.charts`, which adds chart settings, labels, and cumulative display values.
+
+Statistics can also be fetched separately in Python:
+
+```python
+from rdmo_plugins_statistics.statistics import (
+    fetch_catalog_statistics,
+    fetch_project_statistics,
+    fetch_statistics,
+    fetch_statistics_for_sites,
+    fetch_user_statistics,
+)
+
+project_statistics = fetch_project_statistics(site)
+user_statistics = fetch_user_statistics(site)
+catalog_statistics = fetch_catalog_statistics(site)
+
+statistics = fetch_statistics(site)
+site_statistics = fetch_statistics_for_sites(sites)
+```
+
+Single-site functions return domain aggregates. Projects expose `total`, `created` rows (`date`, `count`), and `progress` rows (`percentage`, `count`). Users expose `total` and `registered` rows (`date`, `count`). Catalogs expose `usage` rows (`id`, `uri`, `title`, `available`, `project_count`).
+
+`fetch_statistics(site)` combines these under `projects`, `users`, and `catalogs`. `fetch_statistics_for_sites(sites)` accepts an iterable or queryset of sites and returns ordered per-site results with a `site` dictionary containing `id`, `name`, and `domain`. Chart settings are accepted only by the chart adapter, not by fetchers.
+
+The aggregate and domain-specific JSON representations are available from:
+
+- `GET /api/v1/statistics/`
+- `GET /api/v1/project-statistics/`
+- `GET /api/v1/user-statistics/`
+- `GET /api/v1/catalog-statistics/`
+
+All endpoints use the `statistics.view_statistics` permission and return data for the configured current site. They do not accept a site selector. Domain endpoints return the domain objects described above; the combined endpoint returns all three. Responses contain no chart colors, translated axis labels, or orientation.
+
+The Django template serializes chart rows with Django's `json_script` filter. The bundled `statistics.js` reads that data and creates the charts with Chart.js.
+
 
 The frontend code:
 
 - Groups daily backend data into the selected interval
-- Filters time-based data by start and end date
+- Filters all time-based charts with one validated date range
 - Updates charts without reloading the page
-- Stores the selected interval and date filters in `localStorage`
+- Stores the selected interval and date filters in `localStorage` and the page URL
 - Draws values above vertical bars or beside horizontal bars
-- Sorts category charts by value in descending order
-- Sizes and scrolls category charts according to their orientation and number of entries
-- Exports the currently displayed chart data as CSV
+- Sizes and scrolls bar charts according to their orientation and number of entries
+- Uses the currently displayed rows for charts, HTML tables, totals, and CSV exports
+
+## Migration from earlier versions
+
+The refactored statistics plugin replaces the previous row-level REST implementation with a permission-protected dashboard and aggregate APIs.
+
+The previous row-level response formats are no longer provided. The following paths now return domain aggregates:
+
+- `/api/v1/project-statistics/`
+- `/api/v1/user-statistics/`
+
+The `/api/v1/statistics/` endpoint returns the combined domain aggregates, while `/api/v1/catalog-statistics/` provides catalog usage. This also intentionally replaces the intermediate `time_charts`/`category_charts` API contract. Consumers of that contract must migrate; only the HTML dashboard uses chart payloads.
+
+If an installation depends on one of the old endpoints, it should remain on the previous plugin version until that integration has been migrated.
+
+## Exporting statistics on a server
+
+Run the management command from the configured RDMO installation:
+
+```bash
+python manage.py export_statistics --output-dir /srv/rdmo/statistics
+python manage.py export_statistics --site-id 1 --site-id 2 --output-dir /srv/rdmo/statistics
+python manage.py export_statistics --all-sites --output-dir /srv/rdmo/statistics
+```
+
+With no site option, only the current site is exported. Explicit site IDs are validated before writing; duplicate IDs are exported once. Dates are ISO dates grouped using Django's `TIME_ZONE`. All files are UTF-8 CSVs with `site_id,site_name,site_domain` as their first columns:
+
+| File | Remaining columns |
+|---|---|
+| `site_totals.csv` | `project_count,user_count` |
+| `project_creation.csv` | `date,project_count` |
+| `user_registration.csv` | `date,user_count` |
+| `project_progress.csv` | `percentage,project_count` |
+| `catalog_usage.csv` | `catalog_id,catalog_uri,catalog_title,available,project_count` |
+
+Empty tables retain their headers. Spreadsheet formula prefixes in text are escaped with a leading apostrophe; numeric values are not escaped. Catalog IDs and site IDs identify records within this RDMO database; cross-installation imports need a source identifier supplied by the pipeline.
+
+Each run replaces these five files with the latest state and leaves unrelated files alone. All files are prepared before replacement, but replacement is atomic only per file, not for the whole batch. Upload only after the command exits successfully, and avoid overlapping exports to the same directory. Fetches are sequential and are not a transactionally consistent snapshot during concurrent database updates.
+
+Example daily cron entry (adjust paths and supply your normal Django settings environment):
+
+```cron
+0 2 * * * cd /srv/rdmo && .venv/bin/python manage.py export_statistics --all-sites --output-dir /srv/rdmo/statistics
+```
+
+These exports describe current records and site assignments. Creation/registration series exclude deleted records, and progress and catalog usage describe the current state. They do not reconstruct historical snapshots. CI upload and Metabase ingestion are managed outside this plugin.
+
+## Development
+
+Test setup and development fixture tools are documented in [`tests/README.md`](tests/README.md).
+
+Update the committed Chart.js browser build and license from a source checkout:
+
+```bash
+python scripts/update_chartjs.py 4.5.1
+```
+
+The updater downloads the exact npm release from jsDelivr and validates both files before replacing the existing assets. Review and commit the resulting diff.
 
 ## Uninstallation
 
@@ -182,10 +279,11 @@ Remove the package:
 pip uninstall rdmo-plugins-statistics
 ```
 
-Then remove both plugin references from the RDMO configuration:
+Then remove all plugin references from the RDMO configuration:
 
 1. Remove `'rdmo_plugins_statistics'` from `INSTALLED_APPS`.
 2. Remove `path('statistics/', include('rdmo_plugins_statistics.urls'))` from `urlpatterns`.
-3. Remove the Statistics navigation entry from your theme override (`rdmo_theme/templates/core/base_navigation.html`).
+3. Remove `path('api/v1/', include('rdmo_plugins_statistics.urls.v1'))` from `urlpatterns`.
+4. Remove the Statistics navigation entry from your theme override (`rdmo_theme/templates/core/base_navigation.html`).
 
 All entries must be removed. Otherwise, Django will still try to import the uninstalled package and the application will not start.