Skip to content

README.md

Latest commit

 

History

History
266 lines (186 loc) · 8.05 KB

README.md

File metadata and controls

266 lines (186 loc) · 8.05 KB
Raw

deepSTRF

Unified artificial-hearing research fork.

This repository combines the strongest parts of four research lines into one working system:

upstream deepSTRF
neural datasets + encoding models + recurrence + metrics + checkpoints
        +
libaural
auditory obligations + biological teachers + adversarial tests + compression laws
        +
DAL-style inverse design
differentiable optimization through auditory representations
        +
torch_amt
differentiable PyTorch auditory / psychoacoustic mechanisms
        ↓
one research environment for discovering the smallest digital mechanisms that preserve hearing

Game Music Interpreter (dissonance-git/game-music-interpreter) is used externally as a source-authoritative causal fixture generator. It can provide exact programmed interventions before audio is rendered, giving the hearing models unusually clean tests of pitch, timing, timbre, masking and persistent identity.

This fork is research-first. It is not currently optimized around publication, packaging or product boundaries. Those can be extracted later if useful.

Goal

The target is not merely:

audio → transcript / label / action

The stronger target is:

sampled audio
→ auditory evidence
→ hearing-relevant transformations
→ persistent organized auditory state
→ streams / objects / space / continuity / memory / uncertainty
→ cognition

Biology is the strongest scientific teacher because it already demonstrates that hearing is physically possible. It is not the implementation budget.

The engineering question is:

Which transformations actually make hearing possible, and what is the smallest digital mechanism that preserves them under independent challenge?

Research method

The central loop is:

rich biological / learned / source-authoritative teacher
→ fixed auditory obligation
→ perturbation / counterexample
→ identify the transformation that matters
→ compress it
→ test the same obligation again
→ compare state, latency and compute cost

Keep these claims separate:

model executes
model predicts its teacher target
representation preserves one auditory relation
representation preserves several independent relations
mechanism is cheaper than the teacher
mechanism is suitable for continuous runtime

Progress on one line does not silently promote the next.

What upstream deepSTRF contributes

The fork retains deepSTRF's mature neural-system-identification machinery:

  • auditory neural datasets behind consistent dataset APIs;
  • learned response models including StateNet recurrence families;
  • reliability-aware neural metrics;
  • pretrained checkpoint conventions;
  • the wav2spec → prefiltering → core → readout model organization;
  • reproducible training and benchmark utilities.

Neural prediction is one powerful teacher. It is not the definition of hearing.

Original upstream:

urancon/deepSTRF
consolidation base 5dba99f60a5cb51548f9f9769941cca377771d01

What libaural contributes

dissonance-git/libaural is retired from active development. Its durable findings were consolidated here from final active state:

c280ea60a54c23f9b64a1d8af729e2c0d428a266

The retained research laws are summarized in:

  • research/libaural-findings.md

The first native migrated obligation is:

deepSTRF.hearing.temporal

which measures whether [time, feature] representations retain exact time-reversal structure and opposite-event order while exposing whole-sequence pooling as a destructive control.

Differentiable auditory inverse design

deepSTRF.hearing.inverse.DifferentiableAuditoryLoop generalizes the useful idea behind the Differentiable Auditory Loop:

reference audio
→ reference auditory model
→ target representation

input audio
→ trainable signal transform
→ candidate auditory model
→ candidate representation
        ↓
representation discrepancy
        ↓
gradient to the trainable transform

The loop is deliberately backend-agnostic. A teacher can be a deepSTRF learned model, a differentiable auditory model, or another compatible representation function.

The goal is not to inherit one specific processor architecture. Rich teachers should be ablated until the smallest obligation-preserving transform is found.

torch_amt integration

deepSTRF.hearing.amt exposes an optional bridge to torch_amt.

Pinned donor state:

StefanoGiacomelli/torch_amt
aa84b5e9b2e065c06ce196c89306a30c324d80a0
GPL-3.0-or-later

Useful public mechanisms include:

Dau1997
Glasberg2002
Moore2016
King2019
Osses2021
Paulick2024

gammatone / DRNL filterbanks
IHC transforms
adaptation loops
modulation filterbanks
outer + middle ear transforms
binaural inhibition
loudness / temporal integration

The bridge is optional so the fork does not make one donor package's environment the definition of the whole project.

VGM causal controls

Game Music Interpreter contributes a different kind of evidence:

known authored / driver / synthesis intervention
→ rendered waveform
→ candidate auditory representation
→ did the known relation survive?

Its cross-chip pitch work supplies device-native nominal-frequency controls across independent sound-generator families while preserving each device's own clock/period semantics.

That supports experiments such as:

same nominal pitch + different synthesis family
→ pitch identity should remain readable

same pitch + changed waveform/envelope
→ pitch should remain relatively stable
→ timbre representation should change

same musical part + different physical voice/channel
→ persistent identity should survive migration

See:

  • research/vgm-causal-controls.md

Source truth is a causal ruler, not perceptual truth.

Current research geometry

Three directions now meet in one repository:

FORWARD TEACHER TESTING
sound → candidate representation → neural / behavioral obligation

REPRESENTATION COMPRESSION
rich state → smaller state → same obligation probes

INVERSE AUDITORY DESIGN
desired auditory state + candidate transform → optimize signal transform

They should pressure one another.

A representation that predicts neural activity but loses an independent auditory relation is incomplete for that relation.

A detailed biological model that can be replaced by a much smaller state without losing the declared obligations may remain research-only.

An inverse-designed transform that improves one teacher coordinate while damaging another obligation has not solved the larger hearing problem.

Immediate frontier

The useful order is now execution rather than another broad architecture pass:

  1. Finish migrating AUD-REP-001 natively into deepSTRF and remove the old deepSTRF adapter seam.
  2. Bring original ICNet public evaluation into the native dataset/metric surface.
  3. Compare StateNet and other compact recurrent representations under the same auditory rulers.
  4. Add torch_amt mechanisms as peripheral and psychoacoustic challengers.
  5. Use Game Music Interpreter source-authoritative fixtures for pitch/timbre/time invariance tests.
  6. Run differentiable inverse-design ablations and search for smaller causal/stateful transforms.
  7. Add detailed mechanistic biological teachers only when a named obligation requires them.
  8. Extend toward persistent auditory objects, memory, prediction and uncertainty.

Repository status

Canonical active repository:

dissonance-git/deepSTRF

Retired research lineage:

dissonance-git/libaural

External research sources remain external unless bringing code inside removes a concrete experimental seam.

The final runtime ear may eventually be far smaller than this repository. That is a success condition, not a contradiction.

License and upstream attribution

This fork remains GPL-3.0-or-later, matching upstream deepSTRF and the current torch_amt donor line.

The original deepSTRF project and its scientific work remain foundational to this fork. See the Git history and upstream repository for original authorship, publications and detailed dataset/model documentation.