Repository navigation
Cast sound and input: Miracast sound from a desktop, a working RokuAudioSink, touch and remote over UIBC - #2
Merged
Merged
Conversation
…peaker CastDisplay(audio=True) carries the app's sound with the picture. On the P4, castif reads the audio pump through a TapFeed. On a desktop, pycast muxes castif's LPCM track itself: display.pcm_out() is an audiodev PCMOutput whose writes go out on the cast, and audio='file.mp3' (or ffmpeg input arguments) casts anything ffmpeg reads. The LPCM mux is byte-identical to castif's C muxer (tests/data golden), and a unit-test step now runs in CI.
The TV's player waited forever on the endless WAV: it now carries a Content-Length and real RIFF sizes (an hour of sound), and the player starts. Writes are paced by the clock, 2 s ahead, counted in bytes for an array too: before, an app's sound loop ran many times faster than its sound. Unix MicroPython could not bind the port at all; it binds 0.0.0.0 by getaddrinfo now, as the frame server does.
…aths The Companion channel plays sound or the picture, not both, so sending an app's sound there replaced its picture. SOUND_ON_TV turns it on. The guide says how to cast a desktop app's sound over Miracast, and cast-runtimes has the measured sound on each runtime.
uibcinput reads the report descriptors the sender sends first, so multitouch (report 0x13) and remote-control (0x2A) reports decode by their real layout: each finger becomes a touch where it is, scaled from the descriptor's range to the cast, and a remote's buttons become the K_* keys an app already reads (arrows, OK, Back, Home, play, volume). CastDisplay now offers Windows its whole HID list: offered the keyboard and mouse alone, it sends their descriptors and then nothing.
bdbarnett
changed the title
Five fingers per 0x13 report with X and Y 0..32767, a remote of one 16-bit consumer usage 0..0xFF, and the short 0x12 report it sends first.
The pump drives the I2S wire in stereo even for a mono app, so the cast read the tap as mono and played an octave low; CastDisplay and drum_cast now take it as 24 kHz stereo. RokuAudioSink imported time inside functions, which on a board searches the filesystem each call (65 ms on the P4) and held its writes to a tenth of the sound's rate; its server thread also spun in non-blocking accepts. The P4 row of the sound table, measured on a Roku TV.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sound and input for the cast displays.
Miracast sound.
CastDisplay(..., audio=True)carries an app's sound with the picture. On the ESP32-P4, castif reads the audio pump through aTapFeed. On a desktop, pycast muxes castif's LPCM track itself:display.pcm_out()is an audiodevPCMOutput, whichmiracast/board_config.pyhands to audiodev, andaudio="file.mp3"(or a list of ffmpeg input arguments) casts anything ffmpeg reads. The LPCM mux is byte-identical to castif's C muxer, checked against a golden capture. Heard on a Roku from Linux CPython and from Windows CPython: 30 fps, the player reportspcm, no drift, no gaps, and the speech transcribes word for word.RokuAudioSink. It never played: the TV's player waited forever on an endless WAV. It now sends a Content-Length and real RIFF sizes, paces writes by the clock (2 s ahead, counted in bytes for arrays too), and binds on unix MicroPython. It plays on python, python.exe, micropython and micropython.exe, 3.5 to 5.5 s behind the app. The Companion channel shows sound or the picture, not both, so the roku config's TV sound is now opt-in (
SOUND_ON_TV).Input. uibcinput reads the HID report descriptors Windows sends, so a touchscreen's multitouch reports (0x13) become touches where the finger is, and remote-control reports (0x2A) become K_* keys. CastDisplay offers Windows its whole HID list, which it needs before it forwards anything.
Unit tests run in CI now (
tests/): the mux, resampler, ring and paced speaker, the sink's WAV and pacing, and the descriptor parser with touch and remote reports.Not covered here: a real touch on a Windows touchscreen (needs a person at the laptop), and the P4's sound on hardware.