Skip to content

Add metrics page to user guide#671

Open
jsmariegaard wants to merge 4 commits into
mainfrom
docs/metrics-user-guide
Open

Add metrics page to user guide#671
jsmariegaard wants to merge 4 commits into
mainfrom
docs/metrics-user-guide

Conversation

@jsmariegaard

@jsmariegaard jsmariegaard commented Jun 10, 2026

Copy link
Copy Markdown
Member

Closes #670.

Adds a Metrics page to the user guide — a when-to-use guide for skill metrics: how they differ (e.g. the ev vs cc vs r2 question from the issue), which metric answers which question, and common pitfalls. Based on the draft attached to #670, reviewed and aligned with the rest of the user guide.

What's included

  • New docs/user-guide/metrics.qmd
    • Live, executable intro example (same SW data/style as statistics.qmd)
    • Reference tables grouped by family: error magnitude, relative/normalised, efficiency/skill scores, correlation & amplitude, event & distribution, directional/circular
    • A quick decision guide and cross-cutting reminders
    • Quartodoc interlinks and cross-links to the metrics API, directional-data, and custom-metric examples
  • Wired into the User Guide sidebar (docs/_quarto.yml, after Statistics)
  • Linked from docs/user-guide/index.qmd

Review notes

  • Every metric name, alias, range, and best value was checked against src/modelskill/metrics.py; the default skill set matches default_metrics.
  • Clarified that r2 is the coefficient of determination (= NSE), not squared correlation — surfaced as a callout.
  • The page is a fairly large static reference; if you'd prefer parts generated from docstrings to avoid drift, happy to follow up.

🤖 Generated with Claude Code

@jsmariegaard @claude
Add metrics page to user guide
5bb7421 
Adds a when-to-use guide for skill metrics: how they differ, which metric answers which question, and common pitfalls. Resolves the request in #670.

- New docs/user-guide/metrics.qmd with a live intro example and reference tables grouped by metric family, verified against src/modelskill/metrics.py
- Wire into the User Guide sidebar in docs/_quarto.yml
- Link from docs/user-guide/index.qmd

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ecomodeller
ecomodeller requested changes Jun 10, 2026
View reviewed changes

@ecomodeller ecomodeller left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice with a metrics page. Take this opportunity to fix the details of the ranges in the docstrings for some of the metrics.

Comment thread docs/user-guide/metrics.qmd Outdated
|---|---|---|---|---|
| `si` | `scatter_index` | [0, ∞) | 0 | Scatter index = `urmse / mean(obs)`. Random error as a fraction of the mean. |
| `mape` | `mean_absolute_percentage_error` | [0, ∞) | 0 | Mean absolute % error, `mean(\|model−obs\| / \|obs\|)`. Intuitive %, but blows up near obs = 0. |
| `scatter_index2` | — | [0, 100] | 0 | Alternative SI formulation (centred, normalised by `Σobs²`). Less common. |

@ecomodeller ecomodeller Jun 10, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Double check that [0, 100] is correct, I think it should $\infty$

Comment thread docs/user-guide/metrics.qmd Outdated
| `r2` | — | (−∞, 1] | 1 | Coefficient of determination. Identical to NSE (see gotcha below). |
| `kge` | `kling_gupta_efficiency` | (−∞, 1] | 1 | Kling–Gupta — composite of **correlation + bias ratio + variability ratio**. |
| `willmott` | — | [0, 1] | 1 | Willmott's Index of Agreement — bounded [0,1], less harsh on bias than NSE. |
| `ev` | `explained_variance` | [0, 1] | 1 | Proportion of variance explained (differs from NSE when the model is biased). |

@ecomodeller ecomodeller Jun 10, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment as above. Are limits correct? I think this one can be negative.

jsmariegaard and others added 2 commits June 10, 2026 13:54
@jsmariegaard @claude
Simplify default-metrics note to a plain code reference
91e482a 
Avoids an interlink to modelskill.options, which is not in the
quartodoc inventory and produced a build warning.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@jsmariegaard @claude
Address review: fix metric ranges, link to API, improve table layout
0523cb5 
Responds to @ecomodeller's review on #671.

Docstring range fixes in metrics.py:
- explained_variance: [0, 1] -> (-inf, 1] (can be negative)
- scatter_index2: [0, 100] -> [0, inf)
- c_max_error: [0, inf) -> [0, 180] (circular error is bounded)

User-guide metrics page:
- Combine metric + alias into one column; every name now links to
  its API entry (via interlinks) for easy navigation
- Slim the Range/Best columns with table-layout CSS so the
  'What it is' column gets the room it needs
- Apply the corrected ranges; add ranges to the circular table
- New 'cc, ev and r2: a nested hierarchy' section answering the
  core question from #670 (cc^2 >= ev >= r2, and why)

Also document scatter_index2 and mean_absolute_percentage_error in
the API reference so the new links resolve.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@jsmariegaard

jsmariegaard commented Jun 10, 2026

Copy link
Copy Markdown
Member Author

Thanks @ecomodeller! Addressed in 0523cb5:

Range fixes — both the docstrings in metrics.py and the page now agree:

  • explained_variance / ev: [0, 1] → (−∞, 1] (it goes negative when the model explains less variance than the obs mean)
  • scatter_index2: [0, 100] → [0, ∞) — you were right, it's unbounded
  • c_max_error: [0, ∞) → [0, 180] while I was checking circular metrics (the angular error can't exceed 180°)

I went through every metric against the source; the rest checked out.

Table layout — metric + alias are now a single column, Range/Best are slimmed via table-layout CSS, and the What it is column gets the freed-up width.

API navigation — every metric name is now an interlink to its entry in the metrics API reference. (Had to add scatter_index2 and mean_absolute_percentage_error to the documented members so the links resolve.)

Bonus — added a short cc, ev and r2: a nested hierarchy section directly answering the 'what's the difference between ev, cc and r2?' question from #670 (cc² ≥ ev ≥ r2, and what each one forgives).

@jsmariegaard

jsmariegaard commented Jun 10, 2026

Copy link
Copy Markdown
Member Author

@ecomodeller should we remove scatter_index2 from the user guide? (I guess it is only there for backwards compatibility)

@ecomodeller ecomodeller self-requested a review June 10, 2026 12:47
@jsmariegaard @claude
Drop scatter_index2 from the metrics guide
9eed954 
It exists only for backward compatibility and isn't recommended, so
it doesn't belong in a when-to-use guide. Also revert documenting it
in the API reference (it was only added there to resolve the page
link). The docstring range fix in metrics.py is kept for accuracy.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@jsmariegaard

jsmariegaard commented Jun 10, 2026

Copy link
Copy Markdown
Member Author

@ryan-kipawa any comments before I merge this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ecomodeller ecomodeller ecomodeller approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Improve metrics guidance in documentation

2 participants

@jsmariegaard @ecomodeller