Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -16,3 +16,6 @@ data/test_reports/

# Local AI curation reports
data/curation_reports/*.json

# Local exam coverage audit packet
tmp/exam_coverage_audit/
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,8 +72,8 @@ Official course files are still kept at their canonical paths under `materials/`

## Support This Project

This project was made by just one student, and the AI coding tools used to build and improve it cost real money.
This project was built by one student, and the AI agents/tools used to build it cost real money.

If this helped you and you want to buy me a coffee to say thank you, please do:
If it saved you time or stress, consider buying me a coffee as a thank-you:

[Buy me a coffee](https://buymeacoffee.com/caspersimon)
51 changes: 51 additions & 0 deletions data/quality/current_sample_exam_coverage_report.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
# Current Sample Final Coverage Audit

## Summary

- Exam: `sample-final-plus-answers`
- Question count: 24
- Exact selectable matches found: 24/24
- Score distribution: `3 => 24`, `2 => 0`, `1 => 0`, `0 => 0`
- Exact matches already in `recommended`: 10
- Exact matches only in `additional`: 14

## Key Findings

- The builder already contains an exact selectable `source_exam` item for every question in the current sample final.
- Discoverability is weaker than raw coverage: 14 of the 24 exact matches are hidden in `additional` rather than `recommended`.
- Several exact matches appear routed under topic cards that do not match the imported question topic labels. The most obvious examples are questions 1-2 (OOP questions currently under `Pandas Core Structures`) and questions 5-8 (Pandas/String-method questions mapped into string-heavy or other non-obvious cards).
- This makes the current dataset better as a hidden coverage corpus than as an easy-to-navigate student-facing cheat-sheet builder.

## Question Map

| Q | Imported topic label | Current card topic | Bucket | Evidence item id |
|---|---|---|---|---|
| 1 | OOP (Initializers) | Pandas Core Structures | recommended | `exam-intro_python_sample_final_24_25-1-w5-pandas-core-structures` |
| 2 | OOP (Methods) | Pandas Core Structures | recommended | `exam-intro_python_sample_final_24_25-2-w5-pandas-core-structures` |
| 3 | Dictionaries | Defining and Calling Functions | recommended | `exam-intro_python_sample_final_24_25-3-w3-defining-and-calling-functions` |
| 4 | String Methods | Conditions | additional | `exam-intro_python_sample_final_24_25-4-w2-conditions` |
| 5 | Pandas (Basic) | String Fundamentals | recommended | `exam-intro_python_sample_final_24_25-5-w4-string-fundamentals` |
| 6 | Pandas (Indexing) | String Operations and Methods | recommended | `exam-intro_python_sample_final_24_25-6-w4-string-operations-and-methods` |
| 7 | Pandas (Subset/Sort) | String Fundamentals | recommended | `exam-intro_python_sample_final_24_25-7-w4-string-fundamentals` |
| 8 | Pandas (Series Strings) | String Fundamentals | recommended | `exam-intro_python_sample_final_24_25-8-w4-string-fundamentals` |
| 9 | Datetime (Parsing) | Datetime | recommended | `exam-intro_python_sample_final_24_25-9-w6-datetime` |
| 10 | Datetime (Arithmetic) | Datetime | recommended | `exam-intro_python_sample_final_24_25-10-w6-datetime` |
| 11 | Comprehensions | Defining and Calling Functions | recommended | `exam-intro_python_sample_final_24_25-11-w3-defining-and-calling-functions` |
| 12 | List Comprehensions | Conditions | additional | `exam-intro_python_sample_final_24_25-12-w2-conditions` |
| 13 | Data Types | Objects and Names | additional | `exam-intro_python_sample_final_24_25-13-w1-objects-and-names` |
| 14 | Loops | Loops | additional | `exam-intro_python_sample_final_24_25-14-w2-loops` |
| 15 | Functions (Scope) | Conditions | additional | `exam-intro_python_sample_final_24_25-15-w2-conditions` |
| 16 | Lambdas | Higher-Order Patterns | additional | `exam-intro_python_sample_final_24_25-16-w3-higher-order-patterns` |
| 17 | Iterables/Logic | Defining and Calling Functions | additional | `exam-intro_python_sample_final_24_25-17-w3-defining-and-calling-functions` |
| 18 | Slicing | Conditions | additional | `exam-intro_python_sample_final_24_25-18-w2-conditions` |
| 19 | Dictionaries | Defining and Calling Functions | additional | `exam-intro_python_sample_final_24_25-19-w3-defining-and-calling-functions` |
| 20 | Dictionaries | Defining and Calling Functions | additional | `exam-intro_python_sample_final_24_25-20-w3-defining-and-calling-functions` |
| 21 | Dict Iteration | Defining and Calling Functions | additional | `exam-intro_python_sample_final_24_25-21-w3-defining-and-calling-functions` |
| 22 | String Methods | Conditions | additional | `exam-intro_python_sample_final_24_25-22-w2-conditions` |
| 23 | Functions (*args) | Arguments | additional | `exam-intro_python_sample_final_24_25-23-w3-arguments` |
| 24 | Modules | Conditions | additional | `exam-intro_python_sample_final_24_25-24-w2-conditions` |

## Follow-up

- Roadmap item: [RM-008](../../docs/specs/RM-008-exam-snippet-discoverability.md)
- Validated findings file: `tmp/exam_coverage_audit/sample_final_findings.json`
77 changes: 77 additions & 0 deletions data/quality/exam_coverage_audit_summary.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
# Exam Coverage Audit Summary

Audit date: `2026-03-22`

## Scope

- Method: vision-first review of rendered PNG pages only
- PDFs reviewed: 7 unique exams
- Duplicate handling: excluded the duplicate 2022 final copy
- Official scoring corpus:
- common exam questions
- key points
- key-point details
- AI examples
- recommended source snippets
- additional source snippets
- `ai_summary` was tracked as a follow-up signal only and did not materially change any audited question outcome

## Overall Result

- Total questions reviewed: `168`
- Score `3`: `87`
- Score `2`: `58`
- Score `1`: `22`
- Score `0`: `1`
- Questions answerable from the builder without outside knowledge (`score >= 2`): `145 / 168` (`86.3%`)

## By Exam

| Exam | Qs | 3 | 2 | 1 | 0 | Verdict |
|---|---:|---:|---:|---:|---:|---|
| `sample-final-plus-answers` | 24 | 24 | 0 | 0 | 0 | Excellent raw coverage; discoverability still weak because many exact matches live in `additional`. |
| `trial-final-exam-solutions-introduction-to-python-3077951` | 24 | 24 | 0 | 0 | 0 | Excellent raw coverage; exact matches already exist for every question. |
| `introduction-to-python-trial-final-exam-solutions-py22` | 24 | 7 | 12 | 5 | 0 | Strong coverage with a mix of exact and concept-level support. |
| `final-exam-study-guide-trial-python-basics-2023` | 24 | 6 | 12 | 6 | 0 | Good coverage, but several synthesis-heavy questions still need prior knowledge. |
| `final-exam-solutions-for-python-programming-62oop21` | 24 | 6 | 12 | 5 | 1 | Good coverage overall; one genuine gap remained. |
| `resit-exam-guidelines-for-intro-to-python-6013b0470y-july-2023` | 24 | 10 | 11 | 3 | 0 | Strong coverage with only a few partial-support questions. |
| `resit-solutions-for-introduction-to-python-35761538` | 24 | 10 | 11 | 3 | 0 | Strong coverage with only a few partial-support questions. |

## Strongest Findings

- The current-year sample final is fully covered at `24 / 24` exact matches.
- The later-course trial final is also fully covered at `24 / 24` exact matches.
- Across the full deduplicated audit set, exact-match coverage is already very high because many practice-exam questions were imported into the builder as selectable `source_exam` items.
- There was only one audited `score 0` question: the 2022 final question asking for the sub-list with the highest element sum from a list of lists.

## Why This Is Not “Great” Yet

- Discoverability is weaker than raw coverage.
- In the current sample final alone, `14 / 24` exact matches are only surfaced in `additional`.
- Several exact matches are routed under topic cards that are not intuitive for students.
- The weaker questions cluster around synthesis-heavy tasks rather than direct recall:
- multi-method OOP implementations
- interval-overlap / datetime logic
- some pandas construction and transformation tasks
- a few string-formatting and `kwargs` ordering questions
- Exact coverage is inflated by imported exam snippets. That is useful, but it means “the builder contains the answer somewhere” is stronger than “a student will easily find the right answer path.”

## Practical Verdict

The cheat sheet generator is already strong on raw exam coverage, and very strong for the current sample final. I would not call it great yet for student-facing use, because discoverability and topic routing still lag behind the underlying content quality.

If the goal is “can the builder answer practice-exam questions at all?”, the answer is yes.

If the goal is “can a student quickly find the right support without hunting through `additional` or oddly routed cards?”, the answer is not consistently yes yet.

## Recommended Next Step

- Prioritize [RM-008](../../docs/specs/RM-008-exam-snippet-discoverability.md):
- promote exact exam-match items out of `additional` when appropriate
- fix misrouted exam snippets
- keep auditing after each curation pass

## Notes

- The current sample-final detailed map is saved in [current_sample_exam_coverage_report.md](./current_sample_exam_coverage_report.md).
- The rendered audit packet is available under `tmp/exam_coverage_audit/`.
Loading
Loading