Skip to content

Add mic mode (beta): play without MIDI via microphone pitch detection - #217

Draft
isc wants to merge 1 commit into
mainfrom
claude/micro-mode-non-midi-9j1r2l
Draft

Add mic mode (beta): play without MIDI via microphone pitch detection#217
isc wants to merge 1 commit into
mainfrom
claude/micro-mode-non-midi-9j1r2l

Conversation

@isc

@isc isc commented Jul 6, 2026

Copy link
Copy Markdown
Owner

Prototype for the "Mode micro (non MIDI)" roadmap item: detect played notes through the microphone so acoustic pianos and keyboards without MIDI can use the app.

How it works

  • A 🎤 Mic mode button in the score-page topbar (shown when no MIDI keyboard is connected) toggles listening. The "no MIDI devices" help modal also offers mic mode as an alternative.
  • micInput.js captures the raw microphone signal (getUserMedia with voice processing disabled) and analyses ~40 ms frames from a 4096-sample AnalyserNode window.
  • Pitch detection reuses a proven brick instead of hand-rolled DSP: pitchy 4.1.0 (McLeod Pitch Method, MIT), vendored as a self-contained 9 KB ESM bundle like the project's other vendor libs. This follows a state-of-the-art review: classic DSP (MPM/YIN) remains the standard for low-latency monophonic detection in the browser; neural options (CREPE/SPICE) are heavier for marginal gain, and polyphonic models (Onsets and Frames, Basic Pitch) are offline-transcription oriented.
  • pitchDetection.js wraps pitchy with a clarity threshold and a piano-range gate (capped below C8 so a stray detection can never trigger the C8 navigate-back key), converts frequency to MIDI, and runs a debounced note-tracker state machine (2 consistent frames to fire Note On, 3 missing frames to release).
  • Detected notes are emitted as synthetic MIDI Note On/Off messages through the existing parseMidiMessage path (same as cassette replay), so free play, training mode, strict mode, reinforcement and cassette recording all work unchanged.

Scope / limitations (prototype)

  • Monophonic only: one note at a time — melodies, Hanon, hands-separate practice. Chords are not recognised; the UI labels the mode as beta and the tooltips say so.
  • Polyphony remains the roadmap's open question; the ROADMAP entry is updated to point at transcription-model approaches (Onsets and Frames / Basic Pitch) as the next step.
  • Velocity is a rough loudness mapping (validation ignores it; it only feeds recordings).

Tests & verification

  • 17 new vitest unit tests (test/js/pitchDetection.test.js): pure sines, harmonic-rich piano-like tones (octave-error trap), low notes (A1 semitone resolution), noise/silence rejection, above-range rejection, and the note-tracker debounce/release/flush state machine. Full JS suite: 102/102 pass.
  • End-to-end check in headless Chromium with a stubbed 440 Hz mic stream: clicking 🎤 produces exactly one synthetic Note ON: A4 and one Note OFF: A4 through the real app pipeline, no wrong notes, and toggling off restores the idle state.
  • Ruby suite locally: 0 assertion failures; 46 errors are all sandbox-environment network timeouts on the pre-existing esm.sh/@tonejs/piano import (unrelated to this diff — CI should be authoritative).

Also in this PR

  • Bilingual changelog entry ("Nouveautés" modal) and FR/EN locale strings.
  • ROADMAP: the mic-mode idea is replaced by its remaining unknown (polyphony).

🤖 Generated with Claude Code

https://claude.ai/code/session_01TZWBasTr1z9EaDUJdD8V3z


Generated by Claude Code

Prototype for the "Mode micro (non MIDI)" roadmap item. A 🎤 button on the
score page (shown when no MIDI keyboard is connected) listens to the
microphone, detects the played pitch and feeds synthetic MIDI Note On/Off
messages through the existing parseMidiMessage path, so free play, training,
strict mode and cassette recording all work unchanged.

- vendor pitchy 4.1.0 (McLeod Pitch Method, MIT) as the pitch detector —
  the state-of-the-art DSP brick for real-time monophonic detection in the
  browser; polyphony would need a transcription model (see ROADMAP)
- pitchDetection.js: pitchy wrapper (clarity + piano-range gating, capped
  below C8 so a stray detection can never hit the navigate-back key),
  freqToMidi, RMS, and a debounced note on/off tracker — all unit-tested
- micInput.js: getUserMedia + AnalyserNode glue, raw signal (voice
  processing disabled), ~40 ms detection frames
- mic alternative offered in the "no MIDI devices" help modal
- changelog entry + roadmap item updated to its remaining unknown (polyphony)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TZWBasTr1z9EaDUJdD8V3z
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants