Provenance Guard

A backend API that classifies submitted creative text as likely human-written or likely AI-generated, scores confidence honestly, surfaces plain-language transparency labels, and supports creator appeals.

Repository: github.com/Rithinteja/Provenance_Guard

---

Quick Start

python -m venv .venv
.venv\Scripts\activate          # Windows
pip install -r requirements.txt
copy .env.example .env          # add your GROQ_API_KEY
python app.py

Server runs at http://localhost:5000.

Endpoints

Method	Path	Description
POST	/submit	Submit text for attribution analysis
POST	/appeal	Contest a classification
GET	/log	View structured audit log
GET	/health	Health check

---

Architecture Overview

When a creator submits text via POST /submit, the request flows through the following components:

1. Flask API (app.py) — validates input, orchestrates the pipeline, applies rate limiting.
2. Signal 1: LLM Assessment (detection/llm_signal.py) — sends text to Groq (llama-3.3-70b-versatile) for semantic AI-likelihood scoring.
3. Signal 2: Stylometric Heuristics (detection/stylometric_signal.py) — computes sentence-length variance, type-token ratio, and punctuation density in pure Python.
4. Confidence Scoring (scoring/confidence.py) — weighted combination of both signals with false-positive bias when stylometrics flag AI but the LLM does not.
5. Transparency Label (labels/transparency.py) — maps score + attribution to plain-language label text.
6. Storage (storage/audit_log.py) — persists content records and append-only audit log entries in SQLite.
7. Response — returns content_id, attribution, confidence, label, and individual signal scores.

Appeals via POST /appeal look up the original classification by content_id, update status to under_review, and append an appeal entry to the audit log alongside the original decision.

See planning.md for the full architecture diagram.

---

Detection Signals

Signal 1: LLM Semantic Assessment (Groq)

What it measures: Holistic semantic and stylistic coherence — whether text reads like polished, template-driven AI prose or authentic human voice.

Why chosen: AI text tends toward uniform structure, balanced clauses, and generic transitions ("Furthermore," "It is important to note"). An LLM judge can detect these patterns holistically in ways pure statistics miss.

What it misses: Lightly edited AI output that a human polished. Formal academic human writing may be misclassified because it shares structural polish with generated text.

Signal 2: Stylometric Heuristics (Pure Python)

What it measures:

• Sentence length variance — AI text tends toward uniform sentence lengths
• Type-token ratio — vocabulary diversity (unique words / total words)
• Punctuation density — commas, semicolons, dashes per word

Why chosen: These are genuinely independent from semantic analysis — they capture structural regularity that differs statistically between human and AI writing, with zero API cost.

What it misses: Repetitive poetry or minimalist prose scores as AI due to low variance. Non-native speakers writing formally may trigger high punctuation-density scores.

---

Confidence Scoring

How Signals Are Combined

combined_score = (0.55 × llm_score) + (0.45 × stylometric_score)

When signals disagree by more than 0.3 and stylometrics score higher than the LLM, a 0.05 bias is applied toward the human end to reduce false positives.

Score Range	Attribution	Label Tier
≥ 0.72	likely_ai	High-confidence AI
0.38 – 0.71	uncertain	Uncertain
≤ 0.38	likely_human	High-confidence human

Validation

I tested four deliberately chosen inputs using both live stylometric scores and representative LLM scores (matching Groq output patterns). Run python scripts/validate_scoring.py to reproduce.

Example 1 — High-confidence AI case

Text: "Artificial intelligence represents a transformative paradigm shift..." (corporate AI-style prose)

Signal	Score
LLM	0.95
Stylometric	0.47
Combined confidence	0.734
Attribution	likely_ai

Example 2 — High-confidence human case

Text: "ok so i finally tried that new ramen place downtown and honestly? underwhelming..." (casual personal writing)

Signal	Score
LLM	0.12
Stylometric	0.265
Combined confidence	0.185
Attribution	likely_human

These two cases differ by 0.549 on the confidence scale — well beyond the 0.72 / 0.38 thresholds — demonstrating meaningful score separation rather than a constant output.

---

Transparency Label

The label returned in every /submit response is plain language meant for readers, not developers.

Label Variants

Variant	Condition	Exact Label Text
High-confidence AI	confidence ≥ 0.72	"This work appears to have been generated by AI. Our analysis found consistent, polished writing patterns typical of automated text. If you believe this is incorrect, you can submit an appeal."
Uncertain	0.38 < confidence < 0.72	"We couldn't determine whether this was written by a person or generated by AI. The writing shows mixed signals, so we're sharing this result with low confidence. Creators can appeal if they disagree."
High-confidence human	confidence ≤ 0.38	"This work appears to have been written by a person. Our analysis found natural variation and personal voice typical of human writing. If you believe this is incorrect, you can submit an appeal."

Each variant uses different wording — not just a different number — so readers immediately understand the system's level of certainty.

---

Appeals Workflow

Creators contest a classification by posting to /appeal:

curl -s -X POST http://localhost:5000/appeal \
  -H "Content-Type: application/json" \
  -d '{"content_id": "YOUR-CONTENT-ID", "creator_reasoning": "I wrote this myself from personal experience."}'

The system:

1. Validates the content_id exists
2. Updates status from classified → under_review
3. Logs the appeal with original classification, both signal scores, and creator reasoning
4. Returns confirmation (no automated re-classification)

---

Rate Limiting

Limits applied to POST /submit: 10 per minute; 100 per day

Reasoning:

• A typical writer submits a few pieces per session — 10/minute allows rapid editing and resubmission without blocking normal use
• 100/day caps abuse from automated scripts flooding the detection pipeline (each submission triggers a Groq API call)
• Per-IP limiting via Flask-Limiter catches bulk scraping without affecting other users

Rate Limit Evidence

Sending 12 rapid requests from a fresh server window:

HTTP status codes: [200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 429, 429]

200 responses: 10
429 responses: 2

Requests 11 and 12 received HTTP 429 (Too Many Requests), confirming the limit is enforced.

---

Audit Log

Every classification and appeal is stored in SQLite and exposed via GET /log.

Sample Log Entries

{
  "entries": [
    {
      "content_id": "ec83b133-4817-42c6-8fdb-e43e724f92be",
      "creator_id": "demo-user-1",
      "event_type": "appeal",
      "attribution": "uncertain",
      "confidence": 0.4865,
      "llm_score": 0.5,
      "stylometric_score": 0.47,
      "status": "under_review",
      "appeal_reasoning": "I wrote this myself from personal experience. I am a non-native English speaker and my writing style may appear more formal than typical.",
      "timestamp": "2026-07-01T01:17:44.749901+00:00"
    },
    {
      "content_id": "87033b1b-e548-449b-b6e3-4760a496d5da",
      "creator_id": "demo-user-2",
      "event_type": "classification",
      "attribution": "uncertain",
      "confidence": 0.3943,
      "llm_score": 0.5,
      "stylometric_score": 0.265,
      "status": "classified",
      "timestamp": "2026-07-01T01:17:40.619713+00:00"
    },
    {
      "content_id": "ec83b133-4817-42c6-8fdb-e43e724f92be",
      "creator_id": "demo-user-1",
      "event_type": "classification",
      "attribution": "uncertain",
      "confidence": 0.4865,
      "llm_score": 0.5,
      "stylometric_score": 0.47,
      "status": "classified",
      "timestamp": "2026-07-01T01:17:38.566829+00:00"
    }
  ]
}

Each entry includes timestamp, content ID, attribution, confidence, both signal scores, and status. The appeal entry preserves the original classification alongside the creator's reasoning.

Note: Log entries above were captured before GROQ_API_KEY was configured (llm_score: 0.5 fallback). With a valid API key, LLM scores vary and produce the full range of label variants.

---

Known Limitations

Formal academic or professional human writing is the most likely misclassification target. An economics paragraph or journal-style excerpt uses structured transitions and even pacing that both our stylometric heuristics (low sentence-length variance, moderate punctuation density) and the LLM signal associate with AI-generated text. The system should land these in the "uncertain" range rather than high-confidence AI — but borderline formal writing may still trigger false positives, which is why the appeal workflow exists.

---

Spec Reflection

How the spec helped: Writing the three label variants in planning.md before coding prevented a common trap — returning the same template with a swapped number. Having exact threshold values (0.72 / 0.38) meant the scoring function could be verified against concrete test cases instead of "looks about right."

Where implementation diverged: The spec originally applied false-positive bias whenever signals disagreed by more than 0.3. During testing, this pulled clearly AI text (LLM = 0.92, stylometric = 0.47) below the AI threshold. I narrowed the bias to apply only when stylometrics score higher than the LLM — protecting human writers flagged by structure alone without suppressing cases where the LLM confidently detects AI prose.

---

AI Usage

Instance 1 — Flask App Skeleton and LLM Signal (Milestone 3)

Directed: Provided the detection signals section and architecture diagram from planning.md and asked for a Flask app skeleton with POST /submit, the Groq LLM signal function returning a 0–1 score, and SQLite audit logging.

Produced: Initial app structure, Groq client wrapper, and basic SQLite schema.

Revised: Changed the LLM prompt to require strict JSON output ({"ai_likelihood": 0.85}) instead of free-text, added a JSON parse fallback with regex extraction, and moved configuration constants to config.py instead of hardcoding in the signal module.

Instance 2 — Stylometric Signal and Confidence Scoring (Milestone 4)

Directed: Provided detection signals, uncertainty representation, and architecture diagram; asked for stylometric heuristics (sentence variance, TTR, punctuation density) and weighted confidence scoring matching spec thresholds.

Produced: Stylometric function with three sub-metrics and a weighted combiner.

Revised: Adjusted the false-positive bias condition after validation showed it was over-correction on AI text. Changed TTR scoring bands after testing showed repetitive human text was scoring too high. Split code into separate modules (detection/, scoring/, labels/, storage/) for clarity rather than keeping everything in app.py as the AI initially suggested.

---

Project Structure

Provenance_Guard/
├── app.py                  # Flask routes and orchestration
├── config.py               # Thresholds, weights, rate limits
├── planning.md             # Pre-implementation spec
├── requirements.txt
├── detection/
│   ├── llm_signal.py       # Signal 1: Groq LLM assessment
│   └── stylometric_signal.py  # Signal 2: structural heuristics
├── scoring/
│   └── confidence.py       # Weighted signal combination
├── labels/
│   └── transparency.py     # Plain-language label generation
├── storage/
│   └── audit_log.py        # SQLite persistence and audit log
└── scripts/
    ├── seed_demo.py        # Generate sample submissions
    ├── test_rate_limit.py  # Verify 429 responses
    └── validate_scoring.py # Confidence scoring validation

---

Testing

# Start server
python app.py

# Submit content
curl -s -X POST http://localhost:5000/submit \
  -H "Content-Type: application/json" \
  -d '{"text": "Your text here", "creator_id": "test-user-1"}' | python -m json.tool

# View audit log
curl -s http://localhost:5000/log | python -m json.tool

# Run demo seed (3 submissions + 1 appeal)
python scripts/seed_demo.py

# Validate scoring across 4 test inputs
python scripts/validate_scoring.py

# Test rate limiting
python scripts/test_rate_limit.py