Skip to content

Return feedback instead of crashing on submissions with no notes - #19

Open
peterbjohnson wants to merge 1 commit into
mainfrom
empty_input_guard
Open

Return feedback instead of crashing on submissions with no notes#19
peterbjohnson wants to merge 1 commit into
mainfrom
empty_input_guard

Conversation

@peterbjohnson

@peterbjohnson peterbjohnson commented Sep 9, 2026

Copy link
Copy Markdown
Member

Closes #13

Problem

An empty note list raised IndexError, which reaches the student as a 500 rather than a feedback message.

compare_performance_ED({"notes": []}, reference)
# IndexError: list index out of range

This is reachable in production: a student submits nothing, uploads a silent or failed recording, or plays so quietly that transcription returns no notes at all.

Approach

Tests first, added to evaluation_test.py as section 11, alongside the existing tests for compare_performance_ED (section 7) and evaluation_function (section 8). Before the change:

9 failed, 4 passed

The four passes are the deliberate control: one- and two-note submissions already worked, and they are covered so they stay working. All three empty combinations crashed.

What was actually broken

Two distinct faults, the second only visible once the first was fixed.

1. Indexing element zero of an empty list. event_alignment_ED inspected response_events[0] to decide whether its input was already grouped into events. The edit-distance boundary conditions already handle a zero-length side correctly, so the check just needed to be skipped when the list is empty.

2. Float arrays from empty lists. With the first fault fixed, build_cost_matrix failed differently:

TypeError: ufunc 'bitwise_and' not supported for the input types

np.array([]) defaults to float64, so the chord flags came back as floats and the & mask raised. The dtypes are now given explicitly.

Reporting the degenerate cases

Structurally the pipeline now produces the right numbers: an empty response marks all four reference notes missing. But the standard wording would be misleading, so both degenerate cases get their own message.

Empty response:

Practice Summary
No notes were detected in your submission, so there was nothing to compare against the reference.

What to check
If you submitted a recording, check that it is not silent and that your instrument
can be heard clearly. If you submitted MIDI, check that it contains notes.

Have another go when you are ready.

Empty reference (a misconfigured question, not a student error):

Practice Summary
This question has no reference notes to compare your performance against, so it
could not be evaluated. Please let your teacher know.

is_correct is now also guarded, because an empty response against an empty reference satisfied every count and was being reported as correct.

Verification

Check Result
New tests, before 9 failed, 4 passed
New tests, after 13 passed
Full suite 86 passed
CI lint gate (E9,F63,F7,F82) 0

🤖 Generated with Claude Code

An empty note list raised IndexError, which reached the student as a 500
rather than a feedback message. This is reachable in production: a student
submits nothing, uploads a silent or failed recording, or plays so quietly
that transcription returns no notes.

Three changes:

- event_alignment_ED indexed element zero to decide whether its input was
  already grouped into events. Skip that check when the list is empty; the
  cost matrix boundary conditions already handle a zero-length side.

- build_cost_matrix built its arrays without explicit dtypes, so an empty
  list produced float arrays and the boolean chord mask raised TypeError.
  Give the dtypes explicitly.

- Report the degenerate cases plainly. Describing every reference note as
  "missed" is misleading when nothing was submitted, and an empty response
  against an empty reference must not be marked correct.

Tests go in evaluation_test.py as section 11, alongside the other tests for
compare_performance_ED and evaluation_function. Short submissions of one or
two notes already worked and are covered so they stay working.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Empty or very short note lists crash with IndexError instead of returning feedback

2 participants