Repository navigation
README rewrite, eval harness, server-side citation guard - #1
Merged
Merged
Conversation
…; rewrite README Extract SLA tier logic into server/services/sla.js (behaviour unchanged). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…tions Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…d guard Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Moderate citation-validation, auditability, and evaluation-reporting issues remain.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 3
Open (3)
What changed in this PR
This PR rewrites documentation, extracts SLA logic, adds server-side citation sanitization, and introduces evaluation tooling.
Changes:
- Adds citation validation and sanitization for AI-generated notices.
- Adds SLA, failover, citation, metrics, and vision evaluation tooling.
- Documents architecture, setup, limitations, and results.
Review findings:
- Moderate: Correct provider/model attribution in the vision harness.
- Moderate: Expand citation parsing for abbreviations, bare references, and mixed-authority clauses.
- Moderate: Expose citation-removal metadata through notice routes.
- Moderate: Include rejected structured citations in the audit list.
- Nit: Remove the contributor’s personal email from the README.
| File | Description |
|---|---|
server/services/visionAI.js |
Applies citation guarding to generated notices. |
server/services/sla.js |
Provides reusable SLA calculations. |
server/services/gemini.js |
Sanitizes generated notice text. |
server/services/citationGuard.js |
Defines citation allowlists and sanitization. |
server/routes/violations.js |
Uses extracted SLA helpers. |
README.md |
Documents architecture, usage, limitations, and results. |
eval/vision/run.js |
Runs labeled-image vision evaluations. |
eval/vision/labels.template.csv |
Provides the vision-labeling template. |
eval/tests/sla.test.js |
Tests SLA boundaries and status thresholds. |
eval/tests/metrics.test.js |
Tests evaluation metric calculations. |
eval/tests/guardrail_enforcement.test.js |
Tests application-level citation stripping. |
eval/tests/failover.test.js |
Tests provider failover behavior. |
eval/tests/citations.test.js |
Tests citation validator fixtures. |
eval/results/sla_tiers.json |
Stores SLA evaluation results. |
eval/results/failover.json |
Stores failover evaluation results. |
eval/results/citation_validator.json |
Stores citation-validator results. |
eval/results/app_guardrail_passthrough.json |
Stores guardrail passthrough results. |
eval/results/app_guardrail_enforcement.json |
Stores guardrail enforcement results. |
eval/README.md |
Documents evaluation methodology and caveats. |
eval/package.json |
Defines evaluation scripts. |
eval/lib/results.js |
Writes evaluation results. |
eval/lib/metrics.js |
Calculates binary classification metrics. |
eval/fixtures/citations.json |
Provides synthetic citation fixtures. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| const violations = []; | ||
| let citations = 0; | ||
| for (const clause of clauses) { | ||
| const refs = [...clause.matchAll(/(?:Section|Sec\.?|§|Bye-?law|Article)\s*(\d+(?:\.\d+)?(?:-[A-Z]{1,2}|[A-Z])?)/gi)]; |
| success: true, | ||
| content: text, | ||
| content: guarded.text, | ||
| citationsRemoved: guarded.removed, |
Comment on lines
+539
to
+541
| const guardedSections = (Array.isArray(parsed.legalSectionsCited) ? parsed.legalSectionsCited : []) | ||
| .filter((c) => validateCitations(String(c)).ok); | ||
| const citationsRemoved = [...guardedContent.removed, ...guardedConsequences.removed]; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a truthful README, an eval/ harness (SLA routing, failover, citation guard), extracts SLA logic to server/services/sla.js, and adds server/services/citationGuard.js, which strips off-allowlist statute citations from AI-drafted notices (20/20 fabricated caught, 0/20 false positives on synthetic fixtures).
🤖 Generated with Claude Code