Skip to content

Research note 06: concept-first against floors only; two self-checks for quick - #46

Merged
ZLHad merged 3 commits into
mainfrom
claude/research-concept-ab
Oct 1, 2026
Merged

ZLHad merged 3 commits into
mainfrom
claude/research-concept-ab

Conversation

@ZLHad

@ZLHad ZLHad commented Oct 1, 2026 •

Copy link
Copy Markdown
Owner

Why

#44 made the workflow concept-first because the maintainer worried it over-constrained the model. Nobody had measured whether it helps. This PR runs a small A/B test and fixes what the test found.

The test (docs/research/06-concept-first-ab.md, English in en/)

  • Requests: two one-line requests, both silent:
    • a 30 s explainer on why you can't see the Milky Way from a city;
    • a 20 s opening film for a bakery that sells one bread.
  • Arms: four films by the same model (Opus 5.5 subagents, HyperFrames, about 45 min each).
    • A keeps only five floors and opens no workflow docs.
    • B follows the quick path card, including the concept step.
  • Review: a fresh reviewer scored all four blind on the eight scoring dimensions, compared each pair, and rated how similar each pair's ideas were.
  • Stillness: measured for all four films the same way (frame differences).
Milky Way B Milky Way A Bakery A Bakery B
Concept 7 6 7 8
Phone readability 5 7 7 8
Motion 6 6 7 5
Mean of 8 5.8 6.1 7.0 6.3
Stillness ≥ 2 s, total 12.8 s 14.3 s 9.0 s 10.2 s

What it found:

  • The reviewer's choices: both times it found B more original, and both times it chose A to post. The means differ by only 0.3 and 0.7, likely within run-to-run noise.
  • Why B lost: labels sitting exactly at the 44 px floor (about 8 px on a phone), and a key transformation done as a dissolve. Slowness is not the reason: the two arms are about as still, in opposite directions in the two pairs. B's stillness does start at the opening, but the hook scores don't tie that to the result.
  • Same idea without the workflow: in one pair (Milky Way, 8/10 similar) the model reached much the same idea on its own.
  • Other: all hooks scored 3–5, and time and tokens were about equal.
  • Caveats: n = 2, one run each, a model reviewer, quick only, picture only. These are listed in the note.

Changes it led to

  • quick self-check (CLAUDE.md quick card, step 5, and playbook/02's effort note): add a phone-size contact sheet and a first-2 s strip. playbook/02 gains the strip command and what 44 px becomes on a phone.
  • playbook/12-ideation.md §6: readouts people must read should not sit at the floor.
  • Digital-silence floor: scoped to films with sound (CLAUDE.md, playbook 12, TASTE_CHECKLIST Design notes: v0.3 roadmap #18). Both workflow agents had to reason this out.
  • Type 02: caption weight falls back to the local font's heaviest weight.
  • Type 03: a fictional brand name gets searched for real ones. One film named its bakery 一条, a known brand.
  • Indexes: docs/research/README.md and en/README.md gain a row and figure 6.1. CHANGELOG entry added.
  • Open question for the maintainer: the 44 px floor is written for 1080 wide. Should it scale for landscape (1920 wide)?

Checks

  • tools/ci.sh --committed, also run with VH_BASH=/bin/bash: all passed.
  • bin/vh effort quick prints the scoped floor.

ZLHad added 2 commits October 2, 2026 05:21
…s for quick

Two requests, two arms (floors only / the quick workflow with concept first),
four films by the same model, scored blind. The workflow's films were judged
more original and worse made both times; their slow concepts left static
stretches and one HUD fell under the size floor. Without the workflow the model
reached much the same idea for one request.

So quick's self-check adds a phone sheet, a first-2 s strip and frozen stretches
from bin/vh check; playbook 12 spells out the size and stillness floors a concept
cannot change; the digital-silence floor is for films with sound; type 02's
caption weight falls back to the local font's heaviest; type 03 checks a
fictional brand name against real ones.
The HUD labels sat at the 44 px floor, not under it: the finding is that 44 px
is about 8 px on a phone in landscape (an open question for the floor). Static
stretches measured the same way for both arms are equal, so slowness does not
explain arm B's lower craft; the note says so and names the reviewer's actual
reasons. The freeze check is dropped (bin/vh check's d=3 missed both films);
quick adds a phone sheet and a first-2 s strip, playbook/02 gets the strip
command and the phone sizes. playbook 12 §6 keeps only the readout rule and
scopes the silence floor. Overstated index rows and changelog toned down; the
一条 brand clash and bakery A's date are in the note; figure labels widened.
@ZLHad ZLHad changed the title Research note 06: concept-first against floors only; three self-checks for quick Research note 06: concept-first against floors only; two self-checks for quick Oct 1, 2026
…ite ways, the opening-stillness point only an observation, bakery A's date debatable, strip command 1500 px wide)
@ZLHad
ZLHad marked this pull request as ready for review October 1, 2026 21:35
@ZLHad
ZLHad enabled auto-merge (squash) October 1, 2026 21:35
@ZLHad
ZLHad merged commit 264a445 into main Oct 1, 2026
2 checks passed
@ZLHad
ZLHad deleted the claude/research-concept-ab branch October 1, 2026 21:37
ZLHad added a commit that referenced this pull request Oct 4, 2026
…for quick (#46)

docs/research/06-concept-first-ab.md (+ en): two arms (floors only / quick with concept first), two silent requests, four films by the same model, scored blind. The reviewer found the workflow's films more original and chose the floors-only films to post both times; its reasons were labels at the 44 px floor (about 8 px on a phone) and a key transformation lost in a dissolve, not stillness (about equal when measured the same way). quick's self-check adds a phone sheet and a first-2 s strip (playbook/02 gets the command); playbook 12 says readouts should not sit at the floor; the digital-silence floor is scoped to films with sound; type 02's weight falls back; type 03 checks fictional brand names. Open: whether the 44 px floor should scale for landscape.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant