[FEATURE] Add a safe RLCR calibration reward for math workflows

[toffee-desuwa]

Feature Request

Add an RLCRRewardFn under trinity/common/rewards/ that returns the five components used by Reinforcement Learning with Calibration Rewards (RLCR): format, accuracy, Brier reward, mean confidence, and endpoint-confidence auxiliary reward.

The proposed response contract is a case-sensitive terminal sequence:

<think>...</think><answer>...</answer><analysis>...</analysis><confidence>q</confidence>

The reward would extract only the final <answer> payload, verify it with Trinity's existing math utilities, and return already weighted floats. This is necessary for Trinity's standard reward-dictionary workflow, which sums component values directly.

Reference: Damani et al., “Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty” (arXiv:2507.16806v2).

Purpose of the Feature

Binary correctness rewards do not expose a calibrated confidence channel. An RLCR reward lets a workflow train answer correctness and reported confidence together while preserving the existing RewardFn interface and lazy reward registry.

The proposed scope is deliberately narrow:

• one reward class with five named, preweighted outputs;
• lazy registration under rlcr_reward;
• an explicit math example with a calibrated system prompt;
• behavioral tests for the golden reward values, format gate, parser/verifier failures, configuration errors, and malformed confidence values.

Expected Implementation

The implementation follows the published five-component structure, with safety hardening. These are intentional deviations from the official reference implementation (reward_fns.py) and must remain part of any compatibility claim:

• Reject non-finite confidence (NaN, Inf, -Inf) instead of letting it enter reward arithmetic.
• Give no endpoint auxiliary reward when q is malformed or cannot be parsed safely.
• Reject out-of-range q instead of clamping it into the auxiliary reward path.
• Raise deterministic configuration errors for missing, unknown, non-numeric, or non-finite component weights.
• Verifier fixed to math_verify at construction; the source kwarg is accepted and ignored (the reference implementation dispatches the whole batch from source[0]).
• Require a strict, case-sensitive terminal <think>…</think><answer>…</answer><analysis>…</analysis><confidence>q</confidence> chain with only trailing whitespace.
• Require a balanced <think> opener. The reference format_reward (tabc) regex begins with .*?</think>, so it awards the format point to a response that has only a </think> closer and no <think> opener (for example work</think><answer>…</answer><analysis>…</analysis><confidence>q</confidence> matches). This port requires <think> to be present and balanced, and scores a missing opener as zero format, accuracy, and Brier.
• Require every reserved tag in the whole response to be globally balanced and non-nested. The reference regex matches the terminal chain against the end of the response behind a lazy .*? prefix, so extra think/answer/analysis/confidence tags that appear earlier (including nested or unbalanced ones inside the reasoning region, for example <think>x<answer>nested</answer></think> before a valid terminal chain) are absorbed by the prefix and still receive the format point. This port scans all reserved tags with a stack discipline and scores any stray, nested, or crossed reserved tag anywhere as zero format, accuracy, and Brier.

Verification exceptions and timeouts are treated as unknown infrastructure outcomes: accuracy and Brier components become zero, rather than reinterpreting the answer as wrong and rewarding low confidence.

The example weights are runnable implementation defaults, not frozen research parameters.

Additional Information

The patch can be kept within the existing core reward extension mechanism: no algorithm, trainer, buffer, or workflow API change is proposed. The example needs its own system prompt because the generic math workflow does not automatically emit the four-tag RLCR contract for a non-MathRewardFn reward.

AI assistance will be disclosed in the PR. The submitting human remains responsible for reviewing the patch and test evidence before posting.

Are You Willing to Submit a PR?

• Yes, I am willing to submit a PR.

[pan-x-c]

Thank you for your interest in the Trinity-RFT project.

In Trinity-RFT, the concept of a reward is relatively flexible. The interfaces under trinity/common/rewards should be regarded more as examples than as a strict requirement.

From the workflow’s perspective, the only requirement is that each output experience is assigned a float value in its reward field. The workflow does not require that this reward be computed through the provided RewardFn interface specifically. Its value may come from a RewardFn, but it may also come from any function implemented directly inside the workflow.

That said, the functionality described in the issue is still very welcome, and we’d be happy to see a PR implementing it.

[toffee-desuwa]

Thanks for the quick response and the clarification on how rewards fit into the workflow! A PR implementing this (reward class + tests + a runnable math example) is ready on our side — we'll open it once we finish verifying the test suite on a supported environment (our local machine is on Python 3.13, above the declared range).