feat(mouse): report mouse input and route scroll gestures to applications

[bmander]

Human header

I wanted to be able to finger-scroll in full-screen apps like Claude Code; this PR wires up those gestures when fullscreen mode is requested. Tested by hand and code reviewed until convergence.

-BMA

Why

A finger-scroll gesture does nothing in a full-screen application. Programs that take over the screen — vim, tmux, Claude Code's flicker-free renderer — keep their own scrollback and repaint everything, so scrolling our scrollback moves a buffer nothing is looking at. The gesture just appears dead.

Those programs ask for the input instead: they enable mouse tracking with DECSET 1000/1002/1003 and handle the wheel themselves. We never had a way to send it to them.

What this does

Three layers, plus two things found on the way.

Reporting mouse input — feat(mouse): report mouse input to applications that request it, and the commits that tighten it

Exposes libvterm's mouse reporting through JNI and adds mouseClick / scrollWheel to TerminalEmulator, plus a mouseTracking property reflecting VTERM_PROP_MOUSE so callers can tell whether the application wants the input at all. Motion and bare button presses stay internal: both are only meaningful for input that hovers or holds a button across events, neither of which a touch gesture produces, and both are easy to leave half-delivered. A click is always emitted with its release.

Encoding is left entirely to libvterm, which follows whichever protocol the application selected (X10, UTF-8, SGR, rxvt) and stays silent while tracking is off — so a caller cannot leak input into a shell that never asked for it. Coordinates are clamped natively, against the size the terminal holds under its own lock; libvterm's X10 encoder clamps only the high end and would otherwise put a control byte on the wire for a negative coordinate. A wheel burst is bounded and delivered in a single upcall rather than one per detent.

One thing worth knowing: mouse.c was already in the CMake source list, but nothing referenced it, so the linker dropped the archive member and vterm_mouse_button / vterm_mouse_move were simply missing from the shipped .so. Calling them from JNI pulls them back in; no build change was needed.

Routing the gesture — feat(mouse): route scroll gestures to applications tracking the mouse

Forks the existing scroll branch in Terminal.kt on mouseTracking.isEnabled, decided once at finger-down so a gesture cannot change owner halfway through. When tracking is on, scrollOffset and screenState are left completely alone and travel is converted to wheel detents; when it is off, the existing local path runs unchanged. A tap forks the same way — to the application as a click, or through the existing hyperlink check. Long-press selection stays local in both cases, since it remains the way to copy text out of a full-screen application.

WheelScroller does the conversion. Two decisions in it are load bearing:

• Reports go to a fixed anchor cell — where the gesture started, not where the finger is. Chasing the finger would emit a motion report for every row crossed to an application tracking in MOVE mode (DECSET 1003), which Claude Code enables. The anchor is what the gesture is aimed at anyway.
• Bursts are capped at 8 detents per sample. A fast fling covers dozens of lines between animation frames, and applications commonly throttle a flood of wheel reports, so sending every detent can scroll less far than sending a few. While a finger is down the excess is discarded rather than queued — a backlog would keep scrolling after the finger stopped. During a fling it is carried instead, so how far a fling travels does not depend on how many frames the device managed to draw. There are tests for both.

The other tunable is one detent per line of finger travel, which makes content track the finger for an application that scrolls a line per detent.

Property identifiers, and the bug underneath them — fix(props) × 3

A drive-by, found while working in setTermProp. The title was read from property 7 and the cursor shape from property 6, but vterm.h numbers TITLE as 4 and CURSORSHAPE as 7 (6 is REVERSE). Both paths were dead: OSC 0/2 never reached terminalTitle, and DECSCUSR never changed the cursor shape.

VTermProp is an unnumbered C enum, so a property inserted upstream shifts everything after it — presumably how the drift happened. Rather than re-copy the numbering, Terminal.cpp now translates by name into a PropCode enum the wrapper owns, where the compiler checks the cases against the header, and Java holds identifiers this repo defines. The switch has no default case on purpose, so a property added upstream draws a -Wswitch warning naming the enumerator instead of silently arriving as something else.

Making the title path live exposed a second bug sitting under it. libvterm hands a string property over in fragments, one per input buffer, because a title can straddle a read — and invokeSetTermProp forwarded each fragment on its own, where setTermProp assigns rather than appends. So the last fragment won:

ESC ]0; "hel"    then  "lo" BEL   ->  title becomes "lo"
ESC ]0; "hello"  then  BEL        ->  title becomes ""

The second is the worse one — the sequence ends on an empty fragment, which blanked the title outright. Fragments are now accumulated and delivered once, on the final one, the same shape termOscFallback and termSelectionSet already used. Buffers are keyed by property, because OSC 0 sets the icon name and the title from the same fragment and one buffer would interleave them, and bounded, because the payload is remote input and nothing guarantees the terminator arrives.

Happy to split the whole fix(props) thread into its own PR if you would rather keep it separate — it stands on its own, and it has grown since I first offered.

The vendored libvterm is no longer pristine — fix(mouse): tell the embedder when a reset clears mouse tracking, tracked by build(libvterm): ...

vterm_state_reset() cleared mouse_flags but left the negotiated report encoding and any held button behind, and switched reporting off without a VTERM_PROP_MOUSE callback. That last part matters here: a terminal mirroring the property keeps believing an application wants the mouse and routes gestures into a vterm that silently drops them. After RIS — which is what a user types to unwedge a terminal when a full-screen application dies without restoring its modes — scrolling would be dead in both directions at once. libvterm exposes neither a reset callback nor a getter for the property, so there is no wrapper-side alternative.

Since CMake compiles the vendored sources directly, a version bump would overwrite the change without failing the build, and the symptom would be a behavioural regression rather than an error. So it is also kept as a patch under lib/src/main/cpp/libvterm-patches/, with a section in lib/README.md naming the divergence and the tests that cover it, and state.c points back at the patch. Verified by round trip: reverted to pristine at the merge base, applied forward, byte-identical to the tree. Not yet submitted upstream — it is a genuine libvterm bug and should go back.

Testing

Suite is at 430 tests, all passing.

• MouseReportingTest (29) — DECSET mode detection, SGR and X10 wheel encoding, direction and modifier and step encoding, button press/release pairing, motion suppression, coordinate clamping, reset clearing the encoding and held buttons, and burst coalescing asserted on callback count. Includes the literal 1000h 1002h 1003h 1006h sequence Claude Code sends.
• WheelScrollerTest (18) — detent conversion, sub-detent accumulation, reversal cancelling held travel, the rate limit and its no-backlog property, non-finite samples, silence when tracking is off, and a fling driven from a deterministic frame clock so the same fling at 8ms and 48ms frame intervals travels the same distance.
• WheelScrollGestureTest (6) — drives the real Compose gesture pipeline through Robolectric. A downward drag produces wheel-up reports and leaves scrollbackPosition untouched; with tracking off it produces zero mouse reports and moves local scrollback as before; a tap arrives as one complete click and never leaves a button held.
• CursorAndModeEscapeTest (20, nine new) — OSC 0, OSC 2, DECSCUSR shape and blink, all failing against the old property constants; plus five for fragment reassembly, four of which fail against the pre-fix wrapper.

Verified end to end on a physical device: ConnectBot built against this library, SSH into a host running Claude Code with the fullscreen renderer, and the transcript scrolls under a finger drag.

Notes for review

CI is green. The full ./gradlew build --no-configuration-cache passes on my fork on JDK 17 — :lib and :test-app unit tests, lint, and both metalava compatibility checks (run). I ran spotlessCheck locally as well. One thing worth knowing if you build this locally: every local Gradle invocation reports BUILD FAILED from the pre-existing configuration-cache incompatibility in the net.researchgate.release plugin, with or without this branch — which is presumably why CI passes --no-configuration-cache.

The two constants in WheelScroller.kt are feel choices — LINES_PER_WHEEL_DETENT and MAX_DETENTS_PER_SAMPLE. They felt right on a Pixel 7 Pro against Claude Code; they may want adjusting against other applications.

Behaviour change to be aware of: the title property now actually flows into TerminalSnapshot.terminalTitle for the first time. Anything rendering that will start seeing it change where it previously sat static.

One motion report per gesture under DECSET 1003. Every wheel call positions the pointer first, so the first report of a gesture moves it from wherever it last sat to the anchor, which an application tracking all motion sees. One report at the start of a gesture, not one per row crossed — but it is not zero, and the docs could be read as promising zero.

Open question, not addressed here: while tracking is on, local scrollback is unreachable by gesture. Under a fullscreen renderer that is correct, since the local scrollback is empty — but for something like less there is an argument for an escape hatch, such as reserving a two-finger drag for local scrollback. Left out deliberately as a separate decision.

There is a reachable stuck state inside that: scroll back locally, then have the application enable tracking, and the offset is frozen where it stands — the wheel path never touches it and the auto-scroll-to-bottom effect only fires at scrollbackPosition == 0. A cheap mitigation independent of the escape-hatch decision is to snap local scrollback to the bottom once when a gesture classifies as a scroll and the application owns the pointer. Say the word and I will add it here rather than leave it for the follow-up.

🤖 Generated with Claude Code