Cast sound and input: Miracast sound from a desktop, a working RokuAudioSink, touch and remote over UIBC

.github/workflows/manifests.yml

@@ -13,6 +13,7 @@ on:
       - 'scripts/install_*'
       - 'scripts/gallery_generator.py'
       - 'scripts/audit_board_config_pairs.py'
+      - 'tests/**'
       - '.github/workflows/manifests.yml'
   pull_request:
     paths:
@@ -24,6 +25,7 @@ on:
       - 'scripts/install_*'
       - 'scripts/gallery_generator.py'
       - 'scripts/audit_board_config_pairs.py'
+      - 'tests/**'
       - '.github/workflows/manifests.yml'
   workflow_dispatch:
 
@@ -64,3 +66,6 @@ jobs:
 
       - name: Audit CP/MP board_config pairs
         run: python scripts/audit_board_config_pairs.py
+
+      - name: Unit tests
+        run: python -m unittest discover -s tests -v

docs/guides/cast-to-a-tv.md

@@ -18,7 +18,7 @@ the [runtime table](../reference/cast-runtimes.md) says which methods work where
 |---|---|---|---|---|---|---|
 | [Miracast to a Roku](#miracast-to-a-roku-tv) | Roku TV | Screen mirroring on, nothing installed | yes, 48 kHz, in sync | no ([#168](https://github.com/PyDevices/pydevices-examples/issues/168)) | ESP32-P4 | CPython with ffmpeg |
 | [Miracast to Windows](#miracast-to-a-windows-laptop) | Windows laptop | the Wireless Display app open | yes | the laptop's mouse and keyboard | ESP32-P4 | CPython with ffmpeg, from another PC; glitches on a busy network |
-| [The Companion channel](#the-pydevices-companion-channel) | Roku TV | the Companion channel, sideloaded once | through the TV | the TV's remote | any board with Wi-Fi and `pngio` | CPython and MicroPython |
+| [The Companion channel](#the-pydevices-companion-channel) | Roku TV | the Companion channel, sideloaded once | sound or the picture, not both; seconds behind | the TV's remote | any board with Wi-Fi and `pngio` | CPython and MicroPython |
 | [HLS video](#hls-video-to-a-roku-or-vlc) | Roku TV, or VLC anywhere | the Companion channel, or any HLS player | no (needs AAC) | none | ESP32-P4 | CPython with ffmpeg |
 
 In short: **Miracast** for a live picture with sound, on a TV or a laptop, with
@@ -48,6 +48,12 @@ about 24 fps. A P4 with no screen of its own casts the full 1280x720 at 18 fps
 P4 plays goes with it as 48 kHz audio on the picture's clock, so a drum
 machine's hits land with its step light.
 
+From a desktop, `CastDisplay(..., audio=True)` puts the app's sound on the
+cast: [`miracast/board_config.py`](https://github.com/PyDevices/gallery/blob/main/lib/examples/miracast/board_config.py)
+does that, so whatever an app plays through `audiodev` comes out of the TV.
+`audio="song.mp3"` casts a file instead, or any input ffmpeg can open, a
+microphone included, given as a list of ffmpeg arguments.
+
 The P4 also controls the TV over its own network API (ECP, built into every
 Roku, with **Control by mobile apps** on). It can power the TV on, change the
 input or the volume, and press any key.
@@ -106,6 +112,11 @@ on it.
   One example each in [`lib/examples/roku/`](https://github.com/PyDevices/gallery/tree/main/lib/examples/roku/).
 - **Video**: `video(url)` plays HLS, which is how the next method reaches a
   TV.
+- **Sound**: `RokuAudioSink` streams an app's sound to the TV. The channel
+  plays sound or shows frames, not both, and the TV buffers it, so it runs 3.5
+  to 5.5 seconds behind. That suits music, not a game; for an app's sound with
+  its picture, use Miracast. `SOUND_ON_TV` in
+  [`roku/board_config.py`](https://github.com/PyDevices/gallery/blob/main/lib/examples/roku/board_config.py) turns it on.
 
 The library is plain sockets plus
 [`pngio`](https://github.com/PyDevices/micropython-pydevices/tree/main/modules/pngio)
@@ -179,4 +190,8 @@ the [archived pydevices-examples repo](https://github.com/PyDevices/pydevices-ex
 
 ## The Roku remote over Miracast
 
-The Roku's remote doesn't reach the app over Miracast. It is tracked in [#168](https://github.com/PyDevices/pydevices-examples/issues/168) and listed on the [roadmap](https://github.com/PyDevices/gallery/blob/main/ROADMAP.md).
+The Roku's remote doesn't reach the app over Miracast. The Roku offers no
+input back channel (its reply leaves out `wfd_uibc_capability`), and its
+network API has no way to report a button press; while it mirrors, the
+remote's arrows, OK, Back and the rest reach neither the source nor any
+other channel. For the remote, use the Companion channel. It is tracked in [#168](https://github.com/PyDevices/pydevices-examples/issues/168) and listed on the [roadmap](https://github.com/PyDevices/gallery/blob/main/ROADMAP.md).

docs/reference/cast-runtimes.md

@@ -24,6 +24,31 @@ on the PATH, or name it in the `FFMPEG` environment variable (Windows
 `python.exe` started from WSL needs that). The Miracast stream from CPython is
 the same MPEG-TS over RTP that the P4 sends, byte for byte.
 
+**Sound** was measured with a TV's speakers recorded by a microphone, with
+speech the recording was transcribed back from and beeps it was timed by.
+
+| Runtime | Miracast to a Roku, app sound | Companion channel (`RokuAudioSink`) |
+|---|---|---|
+| `python` (Linux) | plays, no drift, no gaps | plays, 4.2 s behind |
+| `python.exe` (Windows) | plays, no drift, no gaps | plays, 5.4 s behind |
+| `micropython` (unix) | no H.264 | plays, 3.5 s behind |
+| `micropython.exe` (Windows) | no H.264 | plays, 4.8 to 5.2 s behind |
+| ESP32-P4 (the panel) | plays, no drift, no gaps | plays, 4.3 s behind |
+
+Over Miracast the sound rides the picture's clock, so it is in step with the
+picture. Over the Companion channel the TV plays sound or frames, not both,
+and holds a few seconds of buffer: steady while the app keeps writing, but
+the last seconds stay in the TV until the stream closes. Those figures include
+the 2 s the sink writes ahead, and leave out the microphone's own start-up,
+which makes them a little long.
+
+On the P4, the cast takes its sound from the audio pump's output, which is
+stereo on the I2S wire even when the app opened a mono stream: a 24 kHz mono
+app is `audio_format=(24000, 2)`, the default. Read as mono, the same sound
+played an octave low. The ESP32-P4 Wi-Fi 6 dev kit can't cast sound: its
+Wi-Fi barely sends while its speaker runs, and the TV gives up on the
+session.
+
 **Miracast to Windows** is testris only, with glitches from Wi-Fi loss (see
 [Miracast to a Windows laptop](../guides/cast-to-a-tv.md#miracast-to-a-windows-laptop)). From CPython
 it needs a second PC as the receiver: Windows won't take a cast from itself.

lib/examples/cast/cast_display.py

@@ -17,6 +17,13 @@
 # Display session runs on a thread and rejoins by itself if the sink goes
 # away, and drawing costs what drawing into RAM costs. Desktop MicroPython has
 # no H.264 encoder, by decision, so there it raises.
+#
+# Sound (audio=True): on the P4, castif reads the audio pump's output in C, so
+# whatever the board plays is on the cast. On a desktop, the app's speaker is
+# the cast: display.pcm_out(format) is an audiodev PCMOutput whose sound goes
+# out with the picture (../miracast/board_config.py hands it to audiodev, so
+# board_peripherals.audio_out() plays there). A desktop can also cast any
+# sound ffmpeg reads: audio="song.mp3", or ffmpeg input arguments as a list.
 import sys
 import time
 
@@ -50,10 +57,21 @@ class CastDisplay(FBDisplay):
         fps, bitrate: the stream's.
         name: what the sink lists the cast as.
         log: where session messages go (default: nowhere).
+        audio: None for a silent cast. True carries the app's sound: on the
+            P4 what the audio pump plays (24 kHz, unless ``audio_format`` says
+            otherwise), on a desktop what
+            the app writes to ``pcm_out()``. On a desktop, a file, URL or list
+            of ffmpeg input arguments casts that instead.
+        audio_format: (rate, channels) of what the P4's pump drives, for
+            audio=True. The pump drives the I2S wire in stereo even when the
+            app opened a mono stream, so a 24 kHz mono app is (24000, 2).
+        audio_loop: play a desktop's audio file over and over. A file starts
+            from its beginning with each session, once the sink is listening.
     """
 
     def __init__(self, sink, width=720, height=720, kind="windows", canvas=None,
-                 fps=30, bitrate=3_000_000, name="PyDevices P4", log=None):
+                 fps=30, bitrate=3_000_000, name="PyDevices P4", log=None,
+                 audio=None, audio_format=(24000, 2), audio_loop=False):
         if kind not in ("windows", "roku"):
             raise ValueError("kind is 'windows' or 'roku'")
         self._buf = bytearray(width * height * 2)
@@ -65,11 +83,22 @@ def __init__(self, sink, width=720, height=720, kind="windows", canvas=None,
         self.kind = kind
         self.name = name
         self.log = log or (lambda *a: None)
+        self.audio = None           # the P4's TapFeed, or the desktop's pycast.CastAudio
         if castfast is not None:
+            if audio:
+                try:
+                    self.audio = castfast.TapFeed(rate=audio_format[0], channels=audio_format[1], log=self.log)
+                except Exception as e:      # firmware without the audio pump: a silent cast
+                    self.log("cast sound off:", repr(e))
             self.cast = castfast.make_caster(width, height, canvas=canvas or (width, height),
-                                             fps=fps, bitrate=bitrate)
+                                             fps=fps, bitrate=bitrate, audio=self.audio is not None)
         else:
-            self.cast = pycast.FfmpegCaster(width, height, canvas=canvas, fps=fps, bitrate=bitrate)
+            if audio:
+                self.audio = pycast.CastAudio()
+            self.cast = pycast.FfmpegCaster(width, height, canvas=canvas, fps=fps, bitrate=bitrate,
+                                            audio=self.audio,
+                                            audio_source=None if audio in (None, False, True) else audio,
+                                            audio_loop=audio_loop)
         self.uibc = UibcInput(width, height)
         self.sessions = 0
         self._session = None
@@ -109,11 +138,23 @@ def stats(self):
         st = self.cast.stats()
         s = self._session
         idr = self._idr_before + (s.idr_requests if s is not None else 0)
-        return {"frames": st.get("frames", 0), "fps": st.get("fps", 0) / 1000.0,
-                "sent": st.get("sent", 0), "sessions": self.sessions, "idr_requests": idr}
+        out = {"frames": st.get("frames", 0), "fps": st.get("fps", 0) / 1000.0,
+               "sent": st.get("sent", 0), "sessions": self.sessions, "idr_requests": idr}
+        for k in st:
+            if k.startswith("audio_"):      # the sound track's counters (castif's names)
+                out[k] = st[k]
+        return out
+
+    def pcm_out(self, format=None, **kwargs):
+        """A desktop's speaker on the cast: an audiodev PCMOutput (any 16-bit
+        rate, mono or stereo) whose sound goes out with the picture. Needs
+        audio=True. On the P4 the board's own speaker is already on the cast."""
+        if pycast is None or self.audio is None:
+            raise RuntimeError("pcm_out needs CastDisplay(..., audio=True) on a desktop")
+        return pycast.CastAudioOut(self.audio, format)
 
     # touch_read / keypad_read for a board_config: the laptop's mouse (left
-    # button held) and keys
+    # button held), fingers on its touchscreen, and its keys and remote
     def touch_read(self):
         return self.uibc.read_points()
 
@@ -145,7 +186,9 @@ def _run(self):
                         self.log("roku power-on:", repr(e))
                 s = Session(self.sink, name=self.name, log=self.log, on_input=self.uibc.feed)
                 s.session_request = 0 if self.kind == "roku" else None
-                s.hidc_caps = "Keyboard/USB, Mouse/USB"
+                # Windows forwards input only when offered its whole list, and
+                # touch arrives as MultiTouch
+                s.hidc_caps = "Keyboard/USB, Mouse/USB, MultiTouch/USB, Gesture/USB, RemoteControl/USB"
                 self.sessions += 1
                 self._session = s
 
@@ -154,7 +197,7 @@ def make(dst_ip, dst_port, server_port):
                         return pycast.FfmpegStreamer(self.cast, self._buf, dst_ip, dst_port,
                                                      server_port, 24 * 3600, self.log)
                     return castfast.CastifStreamer(self.cast, self._buf, dst_ip, dst_port,
-                                                   server_port, 24 * 3600, self.log)
+                                                   server_port, 24 * 3600, self.log, audio=self.audio)
 
                 try:
                     result = s.run(make, seconds=24 * 3600, idle_after_done=2, stop=lambda: self._stop)
@@ -178,6 +221,8 @@ def close(self):
             if not self._running:
                 break
             time.sleep(0.1)
+        if castfast is not None and self.audio is not None:
+            self.audio.close()
         self.cast.close()
 
     def deinit(self):

lib/examples/cast/castfast.py

@@ -178,10 +178,11 @@ class TapFeed:
     here; each block is stamped when read (at most one poll late).
 
     The cast carries 48 kHz stereo (the only LPCM a sink takes), and the
-    pump runs at whatever the app opened: the drum machine's low-latency
-    output on the P4 is 24 kHz MONO. Pass the pump's ``rate`` and
-    ``channels``; mono is spread to both sides and 24 kHz is doubled with a
-    midpoint between samples. ``in_bytes`` counts what the tap produced, so
+    pump runs at the rate the app opened: the drum machine's low-latency
+    output on the P4 is 24 kHz. The tap holds what goes to the I2S wire,
+    which is stereo even for a mono app (measured on the P4: each sample
+    twice). Pass the tap's ``rate`` and ``channels``; mono is spread to both
+    sides and 24 kHz is doubled with a midpoint between samples. ``in_bytes`` counts what the tap produced, so
     a wrong format shows in the log as a byte rate that is not rate*ch*2.
     """
 

lib/examples/cast/drum_cast.py

@@ -25,8 +25,9 @@
 
 w = wifi_up()
 # The drum machine opens board_peripherals.audio_out(latency="low"), which on
-# the P4 is 24 kHz mono; the cast carries 48 kHz stereo and converts.
-feed = castfast.TapFeed(rate=24000, channels=1, log=log)
+# the P4 is 24 kHz mono, and the pump drives the I2S wire with it in stereo:
+# that is what the tap holds. The cast carries 48 kHz stereo and converts.
+feed = castfast.TapFeed(rate=24000, channels=2, log=log)
 
 if MODE == "tv":
     from roku_cast import RokuScreen