hub: recover a stuck status-change interrupt poll (Fruit Jam CH334F USB host)

[mikeysklar]

When a device is unplugged from a hub port, the hub reports it on the status-change interrupt endpoint. On some hosts that interrupt transfer starts failing (XFER_RESULT_FAILED) repeatedly instead of delivering the change, and hub_xfer_cb just re-arms it, so the removal is never seen and tuh_umount_cb is never called. The device stays mounted and every transfer to it fails.

After HUB_INT_FAIL_RECOVER consecutive failed polls, poll a port status directly (one port per recovery so every port is covered) so the latched change is still found and handled through the normal path. State is per hub.

Reproduced with CircuitPython on an Adafruit Fruit Jam (RP2350, PIO USB host with an onboard CH334F hub). Run the loop below and unplug the device from a hub port a few times. Before this change, after some unplugs it prints the USBError line forever; after it, every unplug prints no device. Used a macropad as USB client for unplug/replug tests.

import time, usb.core
while True:
    d = list(usb.core.find(find_all=True))
    if not d:
        print("no device")
    else:
        try:
            print("device:", d[0].product)
        except usb.core.USBError:
            print("USBError, device wedged, not removed")
    time.sleep(0.5)

[github-actions]

MemBrowse Memory Report

Top 10 targets by memory change (%) (out of 2469 targets) View Project Dashboard →

target	.text	.rodata	.data	.bss	total	% diff
ch32v203c_r0_1v0/device_info	19,144 → 19,236 (+92)	—	—	—	19,572 → 19,664 (+92)	+0.5%
ch32v203c_r0_1v0/bare_api	20,032 → 20,124 (+92)	—	—	—	20,460 → 20,552 (+92)	+0.4%
ch32v203c_r0_1v0/midi_rx	20,540 → 20,628 (+88)	—	—	—	20,976 → 21,064 (+88)	+0.4%
ch32v203c_r0_1v0/midi2_host	21,680 → 21,772 (+92)	—	—	—	22,108 → 22,200 (+92)	+0.4%
ea4088_quickstart/device_info	15,444 → 15,508 (+64)	—	—	—	16,032 → 16,096 (+64)	+0.4%
lpcxpresso1769/device_info	15,640 → 15,704 (+64)	—	—	—	16,164 → 16,228 (+64)	+0.4%
metro_m4_express/device_info	16,092 → 16,156 (+64)	—	—	—	16,256 → 16,320 (+64)	+0.4%
ea4088_quickstart/bare_api	16,112 → 16,176 (+64)	—	—	—	16,696 → 16,760 (+64)	+0.4%
lpcxpresso1769/bare_api	16,324 → 16,388 (+64)	—	—	—	16,844 → 16,908 (+64)	+0.4%
ea4088_quickstart/midi_rx	16,280 → 16,344 (+64)	—	—	—	16,864 → 16,928 (+64)	+0.4%

[github-actions]

Hardware-in-the-loop (HIL) Test Report

hfp.json

✅ 41 passed · ❌ 15 failed · ⚪ 0 skipped · blank not run

Board	usbtest	cdc_msc_throughput	audio_test_freertos	cdc_dual_ports	cdc_msc	cdc_msc_freertos	dfu	dfu_runtime	hid_boot_interface	hid_generic_inout	midi_test	msc_dual_lun	mtp	printer_to_cdc	duration
stm32l412nucleo	✅ 30/30	✅ C 671/385k M 815/740k	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	88s
lpcxpresso43s67	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	8s
stm32f746disco	✅ 30/30	✅ C 13.7/10.6M M 15.7/31.1M	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	73s
stm32f746disco-DMA	❌ 1/30	✅ C 12.3/10.2M M 15.6/30.7M	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	202s

tinyusb-esp.json

✅ 20 passed · ❌ 4 failed · ⚪ 0 skipped · blank not run

Board	usbtest	msc_file_explorer_freertos	audio_test_freertos	cdc_msc_freertos	device_info	hid_composite_freertos	duration
espressif_p4_function_ev	❌ 28/30	409 KB/s	✅	✅	✅	✅	107s
espressif_p4_function_ev-DMA	✅ 30/30	409 KB/s	✅	✅	✅	✅	101s
espressif_s3_devkitm	✅ 30/30	409 KB/s	❌	✅	❌	✅	185s
espressif_s3_devkitm-DMA	✅ 30/30	409 KB/s	✅	✅	❌	✅	99s

tinyusb.json

✅ 227 passed · ❌ 120 failed · ⚪ 17 skipped · blank not run

Board	usbtest	cdc_msc_throughput	msc_file_explorer	msc_file_explorer_freertos	audio_test_freertos	cdc_dual_ports	cdc_msc	cdc_msc_freertos	cdc_msc_hid	device_info	dfu	dfu_runtime	hid_boot_interface	hid_generic_inout	host_info_to_device_cdc	midi_test	msc_dual_lun	mtp	printer_to_cdc	duration
frdm_k64f			⚪	⚪					❌	✅										11s
ek_tm4c123gxl	❌ 0/30	❌			❌	❌	❌	❌			❌	❌	❌	❌		❌	❌	❌	❌	166s
metro_m4_express	✅ 30/30	✅ C 590/553k M 515/512k				✅	✅	✅			✅	✅	✅	✅	✅	✅	✅	✅	✅	321s
raspberry_pi_pico_w			1099 KB/s	1022 KB/s					❌	❌										54s
feather_nrf52840_express	❌ 0/30	❌			❌	❌	❌	❌			❌	❌	❌	❌		❌	❌	❌	❌	189s
raspberry_pi_pico	✅ 30/30	✅ C 517/512k M 524/538k	❌	62 KB/s	✅	✅	✅	✅	❌	❌	✅	✅	✅	✅	❌	✅	✅	✅	✅	314s
raspberry_pi_pico2			1108 KB/s	1022 KB/s					✅	✅										15s
max32666fthr	❌ 0/30	❌			❌	❌	❌	❌			❌	❌	❌	❌		❌	❌	❌	❌	163s
adafruit_fruit_jam	✅ 30/30	✅ C 512/550k M 524/560k	62 KB/s	62 KB/s	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	258s
lpcxpresso11u37	❌ 0/30	❌			❌	❌	❌	❌			❌	❌	❌	❌		❌	❌	❌	❌	191s
stm32f072disco	✅ 30/30	✅ C 494/281k M 514/500k			✅	✅	✅	✅			✅	✅	✅	✅		✅	✅	✅	✅	246s
ra4m1_ek	❌ 0/30	❌			⚪	❌	❌	⚪			❌	❌	❌	❌		❌	❌	❌	❌	164s
stm32f407disco	✅ 30/30	✅ C 575/523k M 550/523k			✅	⚪	✅	✅			✅	✅	✅	✅		✅	✅	✅	✅	302s
stm32f723disco	❌ 0/30	❌	16384 KB/s	4032 KB/s	❌	❌	❌	❌	✅	✅	❌	❌	❌	❌		❌	❌	❌	❌	249s
stm32f723disco-DMA	❌ 0/30	❌	19418 KB/s	4032 KB/s	❌	❌	❌	❌	✅	✅	❌	❌	❌	❌		❌	❌	❌	❌	214s
stm32h743nucleo	✅ 30/30	✅ C 550/539k M 523/543k			✅	✅	✅	✅			✅	✅	✅	✅		✅	✅	✅	✅	221s
stm32h743nucleo-DMA	✅ 30/30	✅ C 537/558k M 534/515k			✅	✅	✅	✅			✅	✅	✅	✅		❌	✅	✅	✅	188s
nrf5340dk	❌ 0/30	❌				❌	❌				❌	❌	❌	❌		❌	❌	❌	❌	170s
stm32g0b1nucleo	✅ 30/30	❌			✅	✅	✅	✅			✅	✅	✅	✅		✅	✅	✅	✅	250s
stm32l476disco	✅ 30/30	✅ C 571/523k M 511/513k			✅	✅	✅	✅			✅	✅	✅	✅		✅	✅	✅	✅	309s
stm32u083nucleo	✅ 30/30	✅ C 520/465k M 550/521k			✅	✅	✅	✅			✅	✅	✅	✅		✅	✅	✅	✅	255s
nanoch32v203-fsdev	✅ 30/30	✅ C 546/561k M 552/524k			⚪	✅	✅	⚪			✅	✅	✅	✅		✅	✅	✅	✅	213s
nanoch32v203-usbfs	✅ 30/30	❌			⚪	✅	✅	⚪			✅	✅	✅	✅		✅	✅	✅	✅	102s
ch32v103r_r1_1v0	✅ 30/30	✅ C 501/509k M 511/511k			⚪	✅	✅	⚪			✅	✅	✅	✅		✅	✅	❌	✅	261s
ch32v307v_r1_1v0-usbhs	✅ 30/30	❌			⚪	✅	✅	⚪			✅	✅	✅	✅		✅	✅	✅	✅	261s
ch32v307v_r1_1v0-usbfs	✅ 30/30	✅ C 508/510k M 511/511k			⚪	✅	✅	⚪			✅	✅	✅	✅		✅	✅	✅	✅	108s
ch582m_evt	✅ 30/30	✅ C 232/198k M 469/457k			⚪	✅	✅	⚪			✅	✅	✅	✅		✅	✅	✅	✅	243s

[HiFiPhile]

Interesting finding, looks like CH334 is less reliable than I thought (has some glitch laterly on my HIL rig with CH334X).

Does the port revive when a new device is connected ? Linux handles the failure more aggressively, after 10 errors it reset the hub and disconnect/re-enumerate every child.

[mikeysklar]

Good question. I tested this today on the Fruit Jam with a MacroPad.

Running a poll loop that logs every state change.

• On stock master the answer is no. Once the interrupt poll gets stuck, plugging in a new device does not revive it. I left it unplugged for 30 seconds, replugged, and also tried several quick unplug/replug cycles. It stayed stuck the whole time and the new device was never seen. Only reboot recovers.

With this PR:

• 17 unplug/replug cycles on both hub ports USB 1 and USB 2, including slow and sloppy unplugs. Every unplug was detected within about a second, and every replug enumerated and worked normally.

On the Linux nuclear approach: I could add something similar (~30 lines of code to reset the hub. It is not necessary for this wedged scenario, but maybe there are more severe situations. Every device plugged into the Fruit Jam would unmount and re-enumerate since this is a root-port bus reset.

[hathach]

@mikeysklar is the CircuitPython the tip of main, I will try to hook it with usb analyzer to see what is wrong. Mabe pio-usb has an off timing that cause the hub fail to response.

[mikeysklar]

@hathach -

Yes. The stock repro was CircuitPython main at 30d7addb4e (tip of main on Jul 17) with the submodules exactly as main pins them: tinyusb at 5453e…).

[mikeysklar]

The Fruit Jam has JP1 which is a two pin header that exposes the upstream :

• USBH_D+ (RP2350 GPIO1)
• USBH_D- (RP2350 GPIO2)

My logic analyzer is slow for this, but the Cynthion with USB breakout would work. I'll see if I can wire something up.

[mikeysklar]

Wired up the Cynthion USB sniffer to the Fruit Jams CH334F upstream on the JP1 header. It worked!

The CH334F is innocent. Just a hub doing its job and still responding to interrupt polls. The host (RP2350) does not ACK once wedged. Probably a deadlock will result in a upstream PR to sekigon-gonnoc/Pico-PIO-USB.

[hathach]

@mikeysklar superb! that is great to hear, rp2350 has annoying bug with errata19 gpio reading, I fixed it using walkaround in pio-usb, maybe there is still a race. I will work on this soon (still wrap up other long pending issue). Cynthion is nice, I wish I grab one from adafruit when it is first available, more tool is always a plus :).

[mikeysklar]

The Cynthion is awesome with excellent open source support and up to 12Mbps. I love using Packetry.

Still ended up having to hookup a logic analyzer. Pico-PIO-USB seems to be struggling iwth final bits that are 5 bit or more.

[hathach]

doesn't it support USB HS or the HS is clsoed source ?

[HiFiPhile]

My rig is actually using TERMINUS FE2.1, don't know why I'm thinking about CH334 🤦

[mikeysklar]

@hathach - It actually does HighSpeed (480Mbps), but I've only used up to Full (12Mbps) for Adafruit USB host controller work. The popular ones don't have the HS PHY yet, but it looks like that will be the norm going forward.

[mikeysklar]

What I've learned about the Fruit Jam USB wedge.

• The USB HUB CH334F is still alive and responding
• The RP2350 is not responding and it is not a silicon issue (E9/E19 ruled out)
• Full CircuitPython build remains problematic
• Bare Metal TinyUSB works (no circuitpython)
• Minimal CircuitPython works (disabled PSRAM heap traffic, DVI/HSTX DMA, audio DMA)

Now doing automated Macropad resets on Fruit Jam USB 1 on different CircuitPython builds to determine which setting(s) are the issue.

[mikeysklar]

This took 20 CircuitPython build iterations to prove out and will sound crazy. Disabling CIRCUITPY_VECTORIO (not compiled in) will prevent the USB Error wedge condition on the Fruit Jam.

Minimal repro for anyone with a Fruit Jam:

1. Flash stock CircuitPython (I tested at 30d7addb4e, headless, no HDMI needed).
2. Plug any USB device into a USB-A hub port. I used a MacroPad.
3. Run this and unplug the device:

import time, usb.core
while True:
    d = list(usb.core.find(find_all=True))
    if not d:
        print("no device")
    else:
        try:
            print("device:", d[0].product)
        except usb.core.USBError:
            print("USBError, wedged")
    time.sleep(0.5)

First or second unplug wedges it: USBError forever, replug is never seen, only reboot recovers.

Separate and worthy of its own issue/pr in CircuitPython repo is CIRCUITPY_USB_AUDIO causes random board reboots. Both vectorio and usb_audio being disabled led to very stable USB behavior.

Still looking into if this is flash or cache as no vectorio code even executes in this scenario.

CircuitPython Build Log to narrow down the wedge

Build	Config	Result
A-C	minimal, +PSRAM, +display	clean
D	+audio	reboots
D2	audio -usb_audio	clean
E, F	+misc clusters	wedged
G, H	+SD/misc, +ulab	clean
I, J	+imaging, +bitmap trio	wedged
K	+vectorio only	wedged
L	stock -vectorio -usb_audio	clean
M	stock -vectorio	reboots, no wedge
PAD	+12KB inert flash	clean
RAM	K, host path in RAM	wedged
QSTR	+vectorio qstrs	clean
MOD	+module entry	clean
STUB	vectorio silhouette	clean
MARK	K + GPIO markers + LA	in progress

[HiFiPhile]

For info I'm using USB Packet Viewer as sniffer, it only costs 700CNY with lua class decoder, but needs a Windows host.

[mikeysklar]

Follow up on the "flash or cache" question. I instrumented core1's receive path with GPIO markers (no code changes that shift timing, single instruction pin writes) and captured the wedge on a logic analyzer.

Every failed poll shows the same signature: core1 detects end of packet, drains all 5 bytes from the PIO FIFO, but the bytes fail the CRC check so the ACK is never sent.

The wire side (Cynthion capture between the RP2350 and the CH334F) shows the hub's bytes are perfect. The hub answers every interrupt poll with a valid CRC16 DATA1 carrying 02, the port 1 status change bitmap. The host never sends the ACK, so the hub keeps the change pending and retransmits the identical byte forever, every 12 ms, for the entire capture. Replug is invisible because the status change interrupt can never be delivered.

frame    time      packet       payload
11       0.00801   IN
12       0.00801   DATA1        02
(no ACK, host moves on)
26       0.02001   IN
27       0.02001   DATA1        02
(no ACK)
42       0.03201   IN
43       0.03201   DATA1        02
(no ACK)

The corruption correlates with what core0 fetches, not where any USB code lives.

I'll chase the last layer on the CircuitPython side and file issues there. For this PR: the failure class predates RP2350 (#2971 was the same wedge on RP2040, #2994 the previous hub.c recovery). The GetPortStatus fallback keeps hubs usable on affected builds since control transfers bypass the broken interrupt delivery.

From my side this PR is ready to merge.

Related hardening for a neighboring receive path bug found during this work: sekigon-gonnoc/Pico-PIO-USB#210

[mikeysklar]

Wanted to close the loop here. The root cause was a one bit framing offset: core1 locks onto the incoming packet one bit late, every byte arrives shifted by one, and the CRC check fails on wire data that was valid all along. Expected 80 4b 02 c1 7e, received 01 97 04 82 fd, same shift every retry. The flash layout only matters because it changes when core1 arms the receiver relative to the reply.

Fixes that came out of this:

• Recover packets received with a one bit framing offset sekigon-gonnoc/Pico-PIO-USB#211 realigns the misframed packet
• Bound the receive loop and never ACK truncated packets sekigon-gonnoc/Pico-PIO-USB#210 bounds the receive loop
• ports/raspberrypi: pull Pico-PIO-USB deterministic SYNC realignment (Fruit Jam USB host wedge) adafruit/circuitpython#11135 pulls the fix into CircuitPython (draft for now)

Thanks @hathach and @HiFiPhile for the questions and sniffer suggestions along the way, they kept pointing this at the right layer. This PR still stands as written since control transfer recovery helps for any interrupt delivery failure.

[hathach]

Wanted to close the loop here. The root cause was a one bit framing offset: core1 locks onto the incoming packet one bit late, every byte arrives shifted by one, and the CRC check fails on wire data that was valid all along. Expected 80 4b 02 c1 7e, received 01 97 04 82 fd, same shift every retry. The flash layout only matters because it changes when core1 arms the receiver relative to the reply.

Fixes that came out of this:

• Recover packets received with a one bit framing offset sekigon-gonnoc/Pico-PIO-USB#211 realigns the misframed packet
• Bound the receive loop and never ACK truncated packets sekigon-gonnoc/Pico-PIO-USB#210 bounds the receive loop
• ports/raspberrypi: pull Pico-PIO-USB deterministic SYNC realignment (Fruit Jam USB host wedge) adafruit/circuitpython#11135 pulls the fix into CircuitPython (draft for now)

Thanks @hathach and @HiFiPhile for the questions and sniffer suggestions along the way, they kept pointing this at the right layer. This PR still stands as written since control transfer recovery helps for any interrupt delivery failure.

superb, @mikeysklar great work on the finding. I will jump to pio-usb issue/pr next week to see if there is anything I could help with the fix. Each time I go back to pio-usb, I need to relearn usb timing constraint 😄

[mikeysklar]

@hathach - Thank you for offering assistance on the pio-usb side. I'd like to see these two issues ( sekigon-gonnoc/Pico-PIO-USB#210 and sekigon-gonnoc/Pico-PIO-USB#211 ) go through in some form, if not as is.

This issue must feel familiar as you and tannewt have run up against it multiple times.

Timeline in one place since the pieces were spread across two repos over three years:

Who	Where	When	Finding
tannewt	sekigon-gonnoc/Pico-PIO-USB#97	Aug 2023	Captured the shifted bytes (01 a5 for 80 d2), called it a one bit rotation, suspected RX PIO initial state. Saw it on the first capture so it looked like a cold start quirk
hathach	sekigon-gonnoc/Pico-PIO-USB#186	Jul 2025	Optimized the same inter packet window from the TX side (reduce delay before sending handshake)
harbaum / hathach	#3296	Oct 2025	Diagnosed pio timing/race, CPU not responding within USB timing specs, after harbaum found it only fails below 156 MHz
mikeysklar	#3776 + sekigon-gonnoc/Pico-PIO-USB#211	Jul 2026	Same bug: the receiver arms a few bits late and decodes valid packets shifted, so the CRC rejects good data on every retry. The shift is derivable from SYNC, fix in #211