assertion failure in triangular matmul: householder_apply aborts at 8 threads (lhs_ncols 64 != rhs_nrows 32)

verify in the faer benchmark campaign stopped the first recording: at eight threads, the householder_apply family aborts inside the measured library, and the recorder refuses to publish an unverified run (correctly). Tracked here so the campaign's blocker has an owner and the next session does not re-derive it.

Symptom

Assertion failed at faer/src/linalg/matmul/triangular.rs:990
Assertion failed: lhs_ncols == rhs_nrows
- lhs_ncols = 64
- rhs_nrows = 32

Reproducer

# built by scripts/build_for_faer_rev.sh --vendor mkl at fork 9de95f7 (carries #9)
fbench verify --threads 8 --n 256 --batch 1 --dtype f64 --family upgrade_factor,householder_apply

upgrade_factor alone is recorded as skipped in the harness, so the measured case is householder_apply at eight threads. The same command at --threads 1 passes.

Bisect (all at `--threads 8 --n 256 --dtype f64`)

pass: lu, full_piv_lu, cholesky, solve, triangular_solve, qr_factor, qr, svd_values, svd_thin, eigh
pass: the whole factor + decompose group together
fail: the householder group (--family upgrade_factor,householder_apply)
all of the above pass at --threads 1, including the householder group

Two candidate owners, not yet decided

The harness fixture. householder_apply builds a basis and its compact-WY factor by hand (fbench), and faer's threaded path is the one that reaches the shape assertion (1T takes a different path). If the fixture's block layout is not what the threaded kernels require, fbench owns this and the library is fine.
The library. The same assertion at the same line fails in an existing fork test on the parent revision: sparse::linalg::qr::tests::test_numeric_simplicial_qr_1_transpose (cargo test -p t4a-faer --lib, verified failing on a clean checkout of 22f23cf). So the assertion is reachable from more than one entry point, and it predates perf(householder): no parallel region below the compact factor's own floor #9 — that change only removes regions below a shape floor and leaves the threaded shapes themselves unchanged.

How to settle it: reduce the failing harness case to a standalone program that calls
apply_block_householder_transpose_on_the_left_in_place_with_conj (and the sequence form)
with the same basis/target shapes and the same compact-WY factor, sequentially and at eight
threads, and check whether faer's own documented requirement for the blocked-tau layout is
met. If a sequential call with the harness's own fixture also trips the assertion, the fixture
is wrong and the fix is in fbench. If only the threaded call does, the fault is here and the
non-symmetric-QR regression test above is already a second reproducer.

Impact

No campaign page can be recorded for any population containing householder_apply until
this is settled (the campaign worklog entry records the blocker rather than publishing a
page that hides it). The factor and decompose populations are unaffected.
If the fault is in the library, the same threaded path is reachable from any caller that
applies a block Householder update — tlinalg's compact-WY factor and this fork's own row
splits (householder: split the reduction's rows when the target is tall #4, householder: take the row split only while the reduction is deeper than wide #5, householder: split the compact-WY factor's deep reduction too #6) all sit on it.

Evidence: faer-benchmark docs/worklog.md, entry 2026-10-14.
Upstream feedback is deferred at this stage per AGENTS.md; this issue is the fork's tracker only.

docs(worklog): link the blocker to tensor4all/faer#12

Decision: the library owns this. The harness's fixture is legal. A qr_in_place
call with no harness involved aborts on the parent revision, and the harness's
block_size = min(m, n) is inside the API's contract. Fix in
tensor4all/faer#13.

Reduction (standalone, fork worktree at `9de95f7`, CPUs 4-11)

Reproducing fbench/src/backend.rs::householder_ops exactly (seeded a0, c0,
size = min(m, n), qc zeroed, qr_in_place with block_size = size, then the
blocked and sequence applies) shows the abort happens before the measured
apply: the [block] ... threads=8 line never prints. So I reduced it further to
qr_in_place alone:

qr_in_place m=256 n=256 block=256 threads=8 order=C ...
panicked at faer/src/linalg/householder.rs:655
Assertion failed at faer/src/linalg/matmul/triangular.rs:990:2
Assertion failed: lhs_ncols == rhs_nrows   - lhs_ncols = 64 - rhs_nrows = 32

Backtrace: apply_block_householder_on_the_left_in_place_generic::{closure#0} ->
rayon::iter::for_each::ForEachConsumer<... ChunksMutProducer> ->
triangular::matmul_with_conj at householder.rs:655. That is the fork's
row-split reduction (fa22a69, c061812, c8fea58), not the harness and not
upstream (row_split does not exist upstream).

Shapes and threads swept with qr_in_place alone, f64, both orders:

shape	threads	block size	parent	with #13
256x256	8	64 / 128 / 256	abort 64!=16 / 64!=32 / 64!=32	ok
2048x2048	4, 8	recommended = 64	abort 64!=32 / 64!=16	ok
4096x4096	4, 8	recommended = 128	abort 128!=64 / 64!=48	ok
256/2048/4096	1, 2	64 / 128 / 256	ok (1T never splits)	ok

The 2048x2048 and 4096x4096 rows use faer's own recommended_block_size, so
nothing about this is harness-specific. recommended_block_size(256,256) = 32,
which is why the harness's 256x256 qr_factor family passes: its block size is
chosen for it, while householder_apply passes size.

The fixture is legal: qr_in_place_scratch(nrows, ncols, block_size, ..) and
qr_in_place only require block_size > 0 and
Q_coeff.ncols() == min(A.nrows(), A.ncols()). --threads 1 passes because
row_split needs threads > 1.

Cause (one line)

In the row split, partial i covers target rows [lo, hi) and the top part is
applied as triangular::matmul_with_conj(part, .., top_basis.transpose(), UnitTriangularUpper, source.subrows(lo, top_hi - lo), ..). The lhs is the head's full n x n factor,
so it only matches when a chunk carries [0, n) whole. The predicate asked only
for depth (M >= 2K), a thread floor and a work floor, so with
ceil(M / threads) < n the operands are n against top_hi - lo -- 64 against
32 at 256x256/8T. A chunk that is a strict subrange of the top needs the head
factor's columns lo..top_hi, a unit-upper trapezoid no triangular::matmul
precondition can state.

What changed

One conjunct: M.div_ceil(threads) >= n, so the row split is taken only while one
chunk swallows the whole top block. No product is computed differently; every
shape the split was measured on (a 4096-row target with a 16-wide basis,
chunk = 512; test shapes 2048x4, 4096x16) has chunk >= n and is unchanged.
Slicing the head factor's columns instead would keep the split for narrower
panels, at the cost of two products and an explicit zero fill; rejected for now
and recorded in the work log.

What this establishes

Library correct: cargo test -p t4a-faer --lib -> 204 passed, 0 failed,
2 ignored. The focused regression test
threaded_panel_narrower_than_the_row_chunk_matches_the_sequential_one
(256x128 basis, 64-column target, 4 threads, chunk = 64 < n = 128) fails on
the parent with the same assertion (128 != 64) and passes with the gate.
Harness verifies clean: fbench verify over the whole declared
householder grid (4, 8, 16, 32, 64, 256, 1024x64, 4096x256; f64/c64; both
orders; 1, 4 and 8 threads; fork and upstream rows) -> "all comparisons within
tolerance", exit 0. record_run.py gates publication on exactly that, so a
population containing householder_apply is recordable again.
Note: fbench verify at this revision takes a single --threads N, so 1, 4 and
8 were run separately rather than as --threads 1,8.
Isolation: scripts/build_for_faer_rev.sh --vendor mkl, fork at fix(householder): gate the block apply's row split on the top block #13,
--upstream /tmp/upstream-faer (faer-v0.24.4), every build/test/measurement
pinned to CPUs 4-11. No timing is claimed.

One correction to the issue

sparse::linalg::qr::tests::test_numeric_simplicial_qr_1_transpose is not a
second reproducer: it passes here on both 22f23cf and 9de95f7, and all of its
calls are Par::Seq, so it cannot reach the row split. The standalone
qr_in_place rows above are what establish the library ownership.

Closing this: both halves are established (library proven correct, harness verify
clean). The fix is entirely in the library; fbench needed no change.

fix(householder): gate the block apply's row split on the top block

deps: t4a-faer 0.24.7 and tlinalg 0.4.1, whose QR no longer aborts on…

Navigation Menu

Repository navigation

assertion failure in triangular matmul: householder_apply aborts at 8 threads (lhs_ncols 64 != rhs_nrows 32) #12

Symptom

Reproducer

Bisect (all at `--threads 8 --n 256 --dtype f64`)

Two candidate owners, not yet decided

Impact

Reduction (standalone, fork worktree at `9de95f7`, CPUs 4-11)

Cause (one line)

What changed

What this establishes

One correction to the issue

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

assertion failure in triangular matmul: householder_apply aborts at 8 threads (lhs_ncols 64 != rhs_nrows 32) #12

Description

Symptom

Reproducer

Bisect (all at --threads 8 --n 256 --dtype f64)

Two candidate owners, not yet decided

Impact

Activity

shinaoka commented on Oct 14, 2026

Reduction (standalone, fork worktree at 9de95f7, CPUs 4-11)

Cause (one line)

What changed

What this establishes

One correction to the issue

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Bisect (all at `--threads 8 --n 256 --dtype f64`)

Reduction (standalone, fork worktree at `9de95f7`, CPUs 4-11)