Add mic mode (beta): play without MIDI via microphone pitch detection - #217
Draft
isc wants to merge 1 commit into
Draft
Add mic mode (beta): play without MIDI via microphone pitch detection#217isc wants to merge 1 commit into
isc wants to merge 1 commit into
Conversation
Prototype for the "Mode micro (non MIDI)" roadmap item. A 🎤 button on the score page (shown when no MIDI keyboard is connected) listens to the microphone, detects the played pitch and feeds synthetic MIDI Note On/Off messages through the existing parseMidiMessage path, so free play, training, strict mode and cassette recording all work unchanged. - vendor pitchy 4.1.0 (McLeod Pitch Method, MIT) as the pitch detector — the state-of-the-art DSP brick for real-time monophonic detection in the browser; polyphony would need a transcription model (see ROADMAP) - pitchDetection.js: pitchy wrapper (clarity + piano-range gating, capped below C8 so a stray detection can never hit the navigate-back key), freqToMidi, RMS, and a debounced note on/off tracker — all unit-tested - micInput.js: getUserMedia + AnalyserNode glue, raw signal (voice processing disabled), ~40 ms detection frames - mic alternative offered in the "no MIDI devices" help modal - changelog entry + roadmap item updated to its remaining unknown (polyphony) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TZWBasTr1z9EaDUJdD8V3z
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prototype for the "Mode micro (non MIDI)" roadmap item: detect played notes through the microphone so acoustic pianos and keyboards without MIDI can use the app.
How it works
micInput.jscaptures the raw microphone signal (getUserMediawith voice processing disabled) and analyses ~40 ms frames from a 4096-sampleAnalyserNodewindow.pitchDetection.jswraps pitchy with a clarity threshold and a piano-range gate (capped below C8 so a stray detection can never trigger the C8 navigate-back key), converts frequency to MIDI, and runs a debounced note-tracker state machine (2 consistent frames to fire Note On, 3 missing frames to release).parseMidiMessagepath (same as cassette replay), so free play, training mode, strict mode, reinforcement and cassette recording all work unchanged.Scope / limitations (prototype)
Tests & verification
test/js/pitchDetection.test.js): pure sines, harmonic-rich piano-like tones (octave-error trap), low notes (A1 semitone resolution), noise/silence rejection, above-range rejection, and the note-tracker debounce/release/flush state machine. Full JS suite: 102/102 pass.Note ON: A4and oneNote OFF: A4through the real app pipeline, no wrong notes, and toggling off restores the idle state.esm.sh/@tonejs/pianoimport (unrelated to this diff — CI should be authoritative).Also in this PR
🤖 Generated with Claude Code
https://claude.ai/code/session_01TZWBasTr1z9EaDUJdD8V3z
Generated by Claude Code