Switch main to the Python (pysceptre) pipeline - #2

Merged

emattei merged 84 commits into

mainfrom

feat/pysceptre-backend

Sep 26, 2026

Commits on Sep 21, 2026

Plan the pysceptre backend for the high-MOI power analysis

emattei
and
claude
committed
f93fe24
Record what phase 1 found

emattei
and
claude
committed
308b40e
Call the size of the size-factor question an estimate, not a bound

emattei
and
claude
committed
c530cc2
Note that the real-data regression passed too

emattei
and
claude
committed
9fe30fc
Point the plan at the squashed commit

emattei
and
claude
committed
deee55f
Port the expression statistics to Python, and gate them against R

emattei
and
claude
committed
3ddbfc4
Fit the dispersions in Python instead of reading sceptre's cache

emattei
and
claude
committed
5d63834
Port prepare_sim_input to Python, and check it against R's own output

emattei
and
claude
committed
61152ac
Name the degenerate fits, gate the dispersions, record phase 2

emattei
and
claude
committed
3a9225c
Record phase 2's results in the plan

emattei
and
claude
committed
9b2e9a4
Validate in Python, not across the language boundary

emattei
and
claude
committed
8e25a5b
Decide against porting the control-sampling flags, and say why

emattei
and
claude
committed
8d8dfa2
Record that the simulated mean is biased low, and by how much

emattei
and
claude
committed
db520a4
Say plainly that sceptre is not involved in the mean bias

emattei
and
claude
committed
04c7646
Connect the mean question to the analytical side

emattei
and
claude
committed
f8ab01b
Record what PerturbPlan's post-hoc source actually requires

emattei
and
claude
committed
90308fb
Gate the fitted coefficients, not just theta

emattei
and
claude
committed
8aa55aa
Simulate from sceptre's own model, not from a normalised mean

emattei
and
claude
committed
d4fe805
Stop claiming the sweeps are conservative

emattei
and
claude
committed
5fbb839
Draw from sceptre's own model in R too

emattei
and
claude
committed
859df71
Carry the per-pair QC counts through to the simulation

emattei
and
claude
committed
87cfc6a
Port the simulation core: seeds, guide assignment, the draw

emattei
and
claude
committed
c25f56c
ADD the phase-3 benchmark: two engine shapes, four cost terms

emattei
and
claude
committed
ca8ad94
Run the simulation in Python, one call per (target, replicate)

emattei
and
claude
committed
7f79877
Check the two implementations agree, against noise rather than a tolerance

emattei
and
claude
committed
7bbb8d3
Port the remaining four pipeline steps

emattei
and
claude
committed
6c8966f
Rewire the pipeline onto the Python backend, and drop R from the environment

emattei
and
claude
committed
115c10e
Check end to end that the chunk layout reaches no number

emattei
and
claude
committed
ed303c3
Point the docs at the Python pipeline

emattei
and
claude
committed
b76ec8c
Record where the port stands, and what would be wrong to skip

emattei
and
claude
committed
f48a73b
Generate the synthetic fixture rather than committing one

emattei
and
claude
committed
3071abb
Rebuild the container for the Python environment

emattei
and
claude
committed
24f3da1
Rewrite Usage for the Python pipeline

emattei
and
claude
committed
c85cc34
Describe the model the simulation actually uses

emattei
and
claude
committed
54298a5
Bring the docs landing page to the Python pipeline

emattei
and
claude
committed
dd1315f
Update Output for the Python intermediates

emattei
and
claude
committed
6d55f19
Say that the control-sampling flags are gone, and why that is safe

emattei
and
claude
committed
5ab9d9f
Lint Python in the hook, and test the pipeline in CI

emattei
and
claude
committed
c7d3cea
Give Development the commands that exist

emattei
and
claude
committed
b6290bb
ADD the Stage B comparison: Python power against a published R sweep

emattei
and
claude
committed
011d3dc
Point at the r-implementation branch, not legacy

emattei
and
claude
committed
d10fd5d
Write down how to run Stage B, and what each replicate count buys

emattei
and
claude
committed
c2faa4f
Measure the benchmark's stage breakdown outside the shapes' timing

emattei
and
claude
committed
09105ac
Record what re-running the sweeps at the new scale would cost

emattei
and
claude
committed
2c976ce
Centre the effect-size matrix after reordering it, not before

emattei
and
claude
committed
5aeb3f3
Stage B needs a regenerated R reference, not the old sweep

emattei
and
claude
committed
c1dde55
Stop calling the old sweeps published

emattei
and
claude
committed
e4469a3
Lock pysceptre at the v0.2.0 tag

emattei
and
claude
committed
bbc8c95
Finish the sweep: the R help strings and the pysceptre pin

emattei
and
claude
committed
3da0fdd

Commits on Sep 24, 2026

Add the fixture the R and Python suites share

emattei
and
claude
committed
fb3262a
Simulate power at a fixed element effect, with control cells at exactly 1

emattei
and
claude
committed
c815caf
Stage analysis_mode.tsv for the simulation, and give each sample its threshold

emattei
and
claude
committed
1549c1c
Write down what simulated power means, as on the R branch

emattei
and
claude
committed
48b41d4
Solve the pin exactly instead of iterating

emattei
and
claude
committed
8d00177
Bring the reference R simulation on this branch to the same spec

emattei
and
claude
committed
593dd3e
Say plainly what a skipped target does, and retire a stale rationale

emattei
and
claude
committed
9a2242a
Keep the pin exact on huge targets; bring the R reference fully in line

emattei
and
claude
committed
ddefd5c

Commits on Sep 25, 2026

Ship pysceptre's scripts/ in the cloud image, and fit the recipe under Wave's limit

emattei
and
claude
committed
2918aee
Add a samplesheet for moi5 cis on the Python pipeline

emattei
and
claude
committed
900f4d5
Run the screen's own permutation test, and keep clamped-theta genes as R does

emattei
and
claude
committed
f485883
Give guides a spread that vanishes at no effect: Beta, sd = c * es * (1 - es)

emattei
and
claude
committed
6570024
Build the paired-gene set once, not once per gene

emattei
and
claude
committed
f2d684c
List COMPUTE_POWER's inputs with nullglob, not ls

emattei
and
claude
committed
efd0f05
Record Stage B at full scale: Python and R agree on moi5 cis

emattei
and
claude
committed
fb03d30
Run a task's (target, replicate) units in parallel, on 8 CPUs and 8 GB

emattei
and
claude
committed
a85188a
Plan the next step: a random-effect estimand, shared permutations, a faster driver

emattei
and
claude
committed
7aa7eaa
Plan a re-check of the guide spread's value and form, noise removed

emattei
and
claude
committed
30c826e
Add --permutations per-target: one permutation set per target, drawn from the seed

emattei
and
claude
committed
e6c6db6
Record what the data audit settled about the guide spread

emattei
and
claude
committed
40e5371
Make a broken worker pool retryable, and give each worker one thread

emattei
and
claude
committed
de0eaf3
Plan the sparse-route fix first: 2.1-2.3x, bit-identical

emattei
and
claude
committed
157c4b9
Add --nulls sparse: pysceptre's draw-matrix route for the permutation nulls

emattei
and
claude
committed
3ea5baa
Plan fit reuse after the fast driver, and retiring --nulls after pysceptre's release

emattei
and
claude
committed
36fe149
Add --driver fast: one sparse permutation matrix per target, all genes and simulations at once

emattei
and
claude
committed
288cb0f
Add fit reuse: each gene's null model fitted once per simulation, shared across targets

emattei
and
claude
committed
61c62c6
Add --estimand fixed|random: power at a fixed element effect, or averaged over it

emattei
and
claude
committed
596364e
Make the fast configuration the default: per-target permutations, sparse nulls, fast driver, fit reuse

emattei
and
claude
committed
44e3057
Compute per-pair power inside each simulation task; keep the per-simulation table optional

emattei
and
claude
committed
9edd600
Size POWER_SIMULATION from a measurement of the new defaults: 8 GB stands

emattei
and
claude
committed
298c464
Fit null models on the input passed in, not one already in the worker state

emattei
and
claude
committed
0651653

Commits on Sep 26, 2026

Record item 5 of the plan as measured: trans 1 h 52 min and $67.57

emattei
and
claude
committed
e093248
Rewrap one line of the plan's status

emattei
and
claude
committed
2b533fd
Record Stage 0: Python vs R now equals Python vs Python at another seed

emattei
and
claude
committed
0315035
Merge main (the legacy R pipeline) into the Python pipeline, keeping the Python tree

emattei
and
claude
committed
8e24c18