feat(prompt-lint, contracts): prompt quality pipeline — lint, contracts, coverage

[DianaTao]

Closes #829, #822.

---

pdd prompt lint

Detects vague, undefined terms in <contract_rules> and <requirements> sections — deterministic, no LLM required.

• Scans for modal ambiguity (must, should, valid, appropriate, reasonable, etc.) and reports the section, line, and a fix suggestion
• Supports single files, directories, and user story files (--stories)
• --strict escalates all warnings to errors (exit 2) for CI gates
• --json emits stable structured output for downstream tooling
• --ambiguity (optional LLM mode) runs an AI review pass on top of the deterministic scan
• --apply writes suggested <vocabulary> entries back into the prompt file

Docs: https://github.com/DianaTao/pdd/blob/feat/prompt-lint-contracts/docs/prompt_lint.md
Example: https://github.com/DianaTao/pdd/tree/feat/prompt-lint-contracts/examples/prompt_lint_contract_e2e_demo — run ./demo.sh or python lib/run_e2e.py

---

pdd contracts check

Validates that every rule in <contract_rules> follows the structured When … MUST / MUST NOT form required for downstream compilation and coverage analysis — deterministic, CI-safe.

• Reports MISSING_CONDITION, NO_OBSERVABLE_OBLIGATION, AMBIGUOUS_SUBJECT, and other structural defect codes
• --json emits one entry per rule with its defect list

Docs: https://github.com/DianaTao/pdd/blob/feat/prompt-lint-contracts/docs/contract_check.md
Example: https://github.com/DianaTao/pdd/tree/feat/prompt-lint-contracts/examples/contract_commands_cost_tracker_e2e_demo — ./demo.sh runs the full lint → check → compile → coverage pipeline

---

pdd contracts compile

Parses <contract_rules> into a stable JSON intermediate representation (IR) — the first step toward formal verification and coverage tracking.

• Outputs a structured object per rule: subject, condition, obligation type, and bound values
• Compile errors (e.g. unparseable rules) are distinct from check warnings, so you know exactly which rules are machine-readable

Docs: https://github.com/DianaTao/pdd/blob/feat/prompt-lint-contracts/docs/contract_compile.md

---

pdd coverage --contracts

Cross-references <contract_rules> in a prompt against its linked user stories and test files — produces an inspectable rule-to-evidence matrix.

• Each rule is classified as checked, story-only, unchecked, or failed-evidence
• --json emits the full matrix for CI reporting

Docs: https://github.com/DianaTao/pdd/blob/feat/prompt-lint-contracts/docs/coverage_contracts.md

---

Examples

Both examples are self-contained and runnable without credentials:

Example	What it shows
https://github.com/DianaTao/pdd/tree/feat/prompt-lint-contracts/examples/prompt_lint_contract_e2e_demo	Full pipeline on a realistic "foo" prompt: lint a vague prompt → check contracts → compile IR → check coverage
https://github.com/DianaTao/pdd/tree/feat/prompt-lint-contracts/examples/contract_commands_cost_tracker_e2e_demo	Same pipeline on a cost-tracker domain prompt with before/after contract enrichment and user stories

---

Tests

All tests run without real LLM calls:

Test file	Coverage
tests/commands/test_prompt.py	51 tests — pdd prompt lint CLI flags, JSON output, exit codes
tests/commands/test_prompt_comprehensive.py	61 tests — single-file, directory scan, stories, strict, apply-writeback, LLM mocking
tests/commands/test_contracts.py + test_contracts_compile.py	47 tests — contracts check and contracts compile
tests/test_prompt_lint.py	36 tests — deterministic scan engine
tests/test_prompt_lint_contract_e2e_demo.py	integration tests for the e2e demo artifacts
tests/test_contract_commands_cost_tracker_e2e_demo.py	integration tests for the cost-tracker demo

[gltanaka]

Requesting changes. The underlying feature is needed: issues #829 and #822 describe real missing deterministic prompt/story lint and contract-check tooling. This PR should not merge as-is.

Required changes:

1. Fix the failing targeted tests. I ran:

python -m pytest tests/commands/test_prompt.py tests/commands/test_contracts.py tests/commands/test_coverage.py tests/test_prompt_lint.py tests/test_contract_check.py -q

Result: 245 passed, 2 failed. The failures are both in tests/test_prompt_lint.py::TestUploadHandlerFixtures because scan_prompt(tests/fixtures/prompt_lint/upload_handler_python.prompt) only reports successful, not duplicate. The fixture currently defines duplicate upload in <vocabulary>, and the scanner suppresses individual words from defined phrases, so the test/fixture/scanner contract is inconsistent.

2. Do not modify prompt files unless --apply or an equally explicit write flag is passed. Issue #829 explicitly requires that linting not modify files by default. In pdd/prompt_lint_pipeline.py, the LLM path writes accepted definitions via append_vocabulary_definitions(...) even when apply_fixes is false, and pdd/commands/prompt.py forces JSON mode into non-interactive mode. That means pdd prompt lint --ambiguity --json ... can write vocabulary entries without --apply. Formalization write-back is also enabled by options.non_interactive. Make the default LLM path report-only, and add regression tests that compare file contents before/after for --ambiguity, --ambiguity --json, and --non-interactive without --apply.

3. Make --json usable from the real CLI, not just CliRunner. Subprocess runs of the new commands emit non-JSON text around the JSON payload, for example INFO: ..., Checking for updates..., command summaries, and default core-dump messages. That breaks the advertised stable structured output for downstream tooling. Add subprocess tests for pdd prompt lint --json, pdd contracts check --json, pdd contracts compile --json, and pdd coverage --contracts --json, including non-zero exit cases, and ensure stdout is parseable JSON only.

4. Reduce the merge scope to the issues being closed. This PR closes #829 and #822, but it also adds contracts compile, contracts review, coverage --contracts, multiple large demos, generated reports, .pdd/core_dumps, last_run.json, local absolute paths, and WIP examples. examples/README.md explicitly says cost_tracker_strict_ab depends on pdd evidence, pdd gate, and pdd contracts drift, which are not implemented on this branch, and tests/test_cost_tracker_strict_ab.py skips tests for the same reason. Remove WIP/generated artifacts from this PR or split them into follow-up PRs after the commands they depend on exist.

A mergeable version should be much smaller: deterministic pdd prompt lint and pdd contracts check, their focused docs, fixtures, and passing tests. Stretch features and demos can land separately once they are independently runnable and covered.

[jamesdlevine]

One gentle suggestion on top of @gltanaka's review (point 4): rather than trimming this single PR, it might be easier to land as a small stack —

• PR-A: pdd prompt lint (deterministic only, no default writeback) — closes Tooling: Add pdd prompt lint --ambiguity to flag vague or undefined terms #829
• PR-B: pdd contracts check — closes Tooling: Add pdd contracts check to lint natural-language contract sections #822
• PR-C+: contracts compile, contracts review, coverage --contracts, demos — each landing once its dependencies are on main

That way the two issues this PR closes can merge on their own timeline without blocking on the stretch features, and reviewers get a much smaller surface per PR. Happy to leave as-is if you'd rather just shrink in place.