Skip to content

Send iOS AutoFill text to the live view's remote field once - #489

Merged
masnwilliams merged 4 commits into
mainfrom
hypeship/live-view-autofill-once
Oct 28, 2026
Merged

masnwilliams merged 4 commits into
mainfrom
hypeship/live-view-autofill-once

Conversation

@masnwilliams

@masnwilliams masnwilliams commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Summary

On iPhone, tapping an AutoFill suggestion (for example your email above the keyboard) in a live view email field types it into the remote field two or three times.

Email, tel and code fields send each input event's text to the remote as keys and then reset the overlay to its one-character sentinel. iOS fills a field by replacing its text, and it can take more than one edit to do it, or fill again once the overlay's value changes under it. Each of those edits was sent as new text.

Changes

  • field-keyboard.ts: fields that offer AutoFill (email, tel, one-time-code, and contact fields such as name and address) keep their text in the overlay, the way autocorrect fields already do since Open the live view's soft keyboard from the remote browser's focus #464. Edits are diffed against what was sent, so a fill reaches the remote as backspaces and the missing text, however many edits it takes. Password, username, url and number fields keep the per-key path. So does the default keyboard, used when no field is known (the keyboard button, the address bar, or a focus held before the field is reported), since it offers codes from Messages.
  • typed-text.ts: a value that no longer starts with the sentinel but is not empty means AutoFill replaced the whole value, sentinel included. That is new text, not a deletion of what came before, so no backspace is sent for the sentinel and the overlay is not reset.
  • video.vue: in that case the sentinel is put back in front of the filled text instead of resetting the overlay, so a later backspace past the start is still seen.

Testing

  • bun test tests: 259 pass. New tests replay the fill sequences this has to handle through the real onOverlayInput, TypedText and TextInput: one replacement of the whole value, the same fill repeated three times, a space inserted before the replacement, and a fill that completes typed letters. All of them type the email exactly once, and a code filled twice on the default keyboard is typed once. The four component tests fail on main (the repeated fill types it three times).
  • Lint shows no new warnings compared with main.
  • Replayed the repeated fill through the live view against a production browser: the remote email field got the address once with this change, and three times with the deployed client.
  • Tested on a physical iPhone, in Safari and in Chrome, against a production browser serving the deployed live view client with this change and Stop reloading the live view video on every interaction in iOS browsers #487 hot-patched in. Picking the email AutoFill suggestion in the live view puts it in the remote field once, in both browsers. That run predates the default-keyboard commit, which was checked in Chromium with iPhone emulation on the same browser: a code filled twice into the address bar arrived once, and a URL typed there plus Enter navigated.

Note

Medium Risk
Changes touch input routing for several field types on iOS; wrong diff or sentinel handling could mis-sync remote text, but scope is limited to the overlay keep-text path with new tests.

Overview
Fixes duplicate remote typing when iPhone users pick an AutoFill suggestion (email, phone, OTP, contact fields) in the live view touch overlay.

AutoFill-friendly keyboards now use the same keep-text + diff path as autocorrect: field-keyboard sets keepText: true for email, tel, one-time-code, contact autocomplete, and the default keyboard (Messages codes). TypedText.edit treats a full-value replacement (sentinel stripped) as new fill text instead of a sentinel delete. video.vue adds restoreInputSentinel() so after a successful keep-text edit the overlay is sentinel + filled text rather than reset—so repeated iOS fill events and later backspaces still behave correctly. Held-focus keyboard unlock uses DEFAULT_KEYBOARD.keepText instead of forcing false.

Tests cover keyboard keepText rules, TypedText AutoFill sequences, and component-level onOverlayInput replays (including triple repeated fills).

Reviewed by Cursor Bugbot for commit 39e0c23. Bugbot is set up for automated code reviews on this repo. Configure here.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit a647f8a. Configure here.

Comment thread images/chromium-headful/client/src/utils/field-keyboard.ts
@masnwilliams
masnwilliams requested review from ehfeng and tnsardesai and removed request for ehfeng October 26, 2026 19:45

@tnsardesai tnsardesai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, one small nit.

Design

  • Non-blocking nit — replaced can be dropped — images/chromium-headful/client/src/utils/typed-text.ts:L53-L62
    • Problem: The sentinel is never empty, so kept already implies value !== ''. That makes kept || replaced always equal to value !== ''. The new variable and both kept || replaced checks only restate that the one sentinel deletion left is an emptied overlay.
    • Impact: No behavior change, but a reader has to work through three booleans when there's really one condition.
    • Fix: Use const emptied = value === '', send deletes + (emptied ? 1 : 0) backspaces, and return !emptied && next.length <= MAX_TEXT.

masnwilliams added a commit that referenced this pull request Oct 28, 2026
…rs (#487)

## Summary

On iPhone the live view flashes black on most taps and key presses since
#480, in Safari and in Chrome. Every iOS browser runs on WebKit, so they
share the behavior below.

#480 reloads the video from its stream when it is playing but has
decoded no frame since a check at least a second earlier, and runs that
check on every touchend and key press. Frames are counted with
`getVideoPlaybackQuality().totalVideoFrames`, which WebKit on iOS does
not report for a `MediaStream` (it stays 0; WebKit bug 310007). So in
iOS browsers every interaction more than a second after the previous one
counted as a stall and reset `srcObject`, which blanks the video until
the next frame arrives.

## Changes

- `VideoResume` only detects stalls once the frame counter has reported
a frame. A counter stuck at 0 says nothing about a stall, so iOS
browsers keep the paused-video recovery from #480 (play, muted fallback,
"Tap to resume video") but no longer reload a playing stream. Browsers
whose counter advances behave as before; Chromium on Linux, with and
without iPhone emulation, counted MediaStream frames in testing.

## Testing

- `bun test tests`: 251 pass. The new test (a counter that stays at 0
across checks a second apart is never reloaded) fails against the
previous `video-resume.ts`.
- Playwright WebKit (Linux, iPhone 13 profile) against two production
browsers serving the deployed live view client, one unmodified and one
with only this change applied to its bundle. Linux WebKit crashes
decoding the live view's VP8 track, so the track was swapped for a
moving canvas track, and `getVideoPlaybackQuality()` was stubbed to
report 0 frames as iOS WebKit does. Four taps 1.3s apart:
- before: taps 2–4 each reset `srcObject` (`emptied`, `readyState` 0),
leaving the video with no picture for 124–145ms; screenshots right after
those taps are black;
  - after: no reloads, no blank frames;
- with Linux WebKit's real frame counter (which, unlike iOS WebKit,
counts MediaStream frames), neither version reloads.
- Tested on a physical iPhone, in Safari and in Chrome (both showed the
flash before), against a production browser serving the deployed live
view client with this change and #489 hot-patched in. Taps and typing no
longer flash the video black in either browser.

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Low Risk**
> Targeted change to client-side stall heuristics for live video; Chrome
stall recovery is preserved and Safari avoids erroneous reloads.
> 
> **Overview**
> Fixes **live view flashing black on most taps/key presses in Safari**
by tightening when stall recovery reloads the stream.
> 
> `VideoResume` used `getVideoPlaybackQuality().totalVideoFrames` to
detect a playing video that stopped decoding; Safari keeps that at **0**
for `MediaStream`, so every interaction after ~1s was treated as a stall
and **`srcObject` was reset**. The change adds a **`counting` gate**:
stall reload only runs after the frame counter has reported **at least
one frame** (`progress > 0`). Browsers that never advance the counter
skip reload but still get paused/blocked playback handling from #480;
Chrome behavior is unchanged.
> 
> Adds a unit test that a video with **frames stuck at 0** is never
reloaded across repeated stall-interval checks.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
fa7a153. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
@masnwilliams
masnwilliams merged commit 082898d into main Oct 28, 2026
12 checks passed
@masnwilliams
masnwilliams deleted the hypeship/live-view-autofill-once branch October 28, 2026 14:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants