diff --git a/docs/src/content/docs/development/Architecture/model-manager.mdx b/docs/src/content/docs/development/Architecture/model-manager.mdx index 1918e791a62..ea96dcecc10 100644 --- a/docs/src/content/docs/development/Architecture/model-manager.mdx +++ b/docs/src/content/docs/development/Architecture/model-manager.mdx @@ -1068,13 +1068,13 @@ To write such an estimator (`invokeai/backend/util/vae_working_memory.py` has fi - A spec whose fitted terms no platform trait moves declares `platform_keys=()` and ships one entry with an empty `platform`, read alike on every platform, measured or not, never priced as another or floored; its estimate's `source` names it `any platform`, and its points name no platform. MOSS-Image's denoise estimator does this: its fitted term is GEMM and fused-attention activations, not convolution workspaces, and a backend without a fused SDPA kernel is priced by the analytic score-matrix term. - Ship the measured points next to them. A point's `method` says how it was read: one fresh process per point for `reserved_peak` (the allocator's peak reserved bytes), `driver_peak` (the driver-level device usage), and `demand_peak` (the largest of driver-level, reserved and allocated peak, over two runs, which catches the memory HIP maps beyond the allocator on a process's first operation); `device_peak`, the drop in the driver's free memory (`torch.cuda.mem_get_info`, polled during the operation) below its value just before it, which also counts what is allocated around the allocator's reserve; or `probe_demand`, the probe's `demand_bytes`, the largest over the valid, fit-eligible windows of calibration runs in a running server. `scripts/working_memory/build_shipped.py` builds `probe_demand` points from calibration store exports, takes fresh-process points from the point files a calibration script writes (`--points`), and refits the values (see the Family E2E guide). A point names the OS only where it measured an OS-specific entry. `working_memory/bounds.py` holds the rule the data test (`tests/backend/working_memory/`) and the builder apply: each estimate is at least its point, and where a fitted term decides an in-fit point, at most 1.3 times the point, its ratio in `bounds.UPPER_BOUND_EXCEPTIONS` (points no line can hold within 1.3x), or the point's tighter `max_ratio`. A `probe_demand` point is a lower bound only (`MeasuredPoint.holds_upper_bound`) and takes no `max_ratio`: a warm window in a running server reads below the cold first run after a server start, the worst case a reservation must cover. It may raise a line; only the fresh-process points cap one. The `device_working_mem_gb` floor counts only where an analytic term decides. - Fit the values, never hand-tune them. Fitted terms price a line per operation and dtype: `k_` bytes per unit of the line's size plus a base of `b_` bytes. The AutoencoderKL estimators price their convolutions on lines in output pixels (`k__`), the Z-Image denoise its forward on lines in attended tokens (`k_denoise_`). - - `working_memory/fit.py` fits each line on the in-fit points its estimators' `line` admits, of every estimator that resolves to the entry: for the convolution lines the untiled single-sample points, leaving out those whose VAE an in-place cast grew (`upcast_weight_bytes`, a term of its own). The line is the edge of the points' upper convex hull (with the origin, so the base is never negative) that over-reserves least, rated on the highest reading per size of the points that hold an upper bound, the steeper on a tie and never a falling one. A `probe_demand` point is covered by the line and can raise it, but rates no edge. It needs at least two sizes twice apart. The shipped values add a 5% margin and round up: the slope to a whole byte per unit of size, the base to a whole MiB. + - `working_memory/fit.py` fits each line on the in-fit points its estimators' `line` admits, of every estimator that resolves to the entry: for the convolution lines the untiled single-sample points, leaving out those whose VAE an in-place cast grew (`upcast_weight_bytes`, a term of its own). The fit of the points that hold an upper bound (the fresh-process ones) is the edge of their upper convex hull (with the origin, so the base is never negative) that over-reserves least, rated on the highest reading per size, the steeper on a tie and never a falling one. It needs at least two sizes twice apart. The shipped values add a 5% margin and round up: the slope to a whole byte per unit of size, the base to a whole MiB. A `probe_demand` point is covered by the line and can raise it, but takes no part in that fit. - A point whose demand is not its line's alone sets `in_fit: false`: it fits nothing and gets only the lower bound. That is a point measured in a regime its platform does not run (CUDA with the math attention kernel forced), and a fit-grid forward whose features its estimator's `line` does not admit, such as one whose build materializes the score matrix. The test lists every such point. - - A line is fitted only once its in-fit fresh-process points alone span two sizes twice apart (`fit.capping_spans`); `fitted_values` refuses any other line. The shipped pair is the larger, per coefficient, of the fit on all the line's points and the fit on its fresh-process points alone, so a `probe_demand` point that drops a fresh hull vertex lowers neither the slope nor the base. - - Until then the line is a declared stand-in, listed in the entry's `stand_ins` and never below the pre-dtype slope. `probe_demand` points on it bound it from below only: `build_shipped.py` raises its slope and base to their fit where that lies above, and notes the raise in the provenance, but never lowers it. MIOpen with the native allocator off Windows prices fp16 and fp32 at the pre-dtype figure, and CogView4's lines there at its expandable-segments ones. + - A line is fitted only once its in-fit fresh-process points alone span two sizes twice apart (`fit.capping_spans`); `fitted_values` refuses any other line. The shipped pair is the fresh-process fit where it covers every other point of the line by the 5% margin. Where it does not, `fit.raised_pair` raises it by the least the upper bounds allow: of the lines at or above it, per coefficient, that cover every point by 5% and keep the upper bound of every point priced on the line (`bounds.line_bounds`, in the line's units: tiled, batched and upcast VAE operations and denoise forwards with an analytic term included, though their line admits them to no fit, read where the line decides their estimate), the one whose reservation averaged over the fresh-process sizes is smallest, the steeper on a tie. The candidates are the lines that ship, a whole byte of slope per unit of size and a whole MiB of base, checked against the conditions exactly, with no allowance. Each condition is a half-plane in slope and base, so the lines that keep them all form a convex region: the search scans each whole MiB of base across that region, taking the lowest whole slope the conditions leave at it, which finds the lowest rounded line or proves there is none (some milliseconds per line on the shipped data). Where no rounded line satisfies them all, also where only a line between whole bytes or MiB would, it refuses with `fit.LineConflict`, naming the probe and the bounds that conflict (each bound by its point's id, measured bytes and the most it allows, since two points can share an id), or saying that the floor itself already breaks a bound. The data then conflicts with the bounds: `build_shipped.py` prints the changes it would make, writes nothing and exits 1, as for a result the bounds reject; it exits 2 only for input it cannot use. A `probe_demand` point thus lowers neither the slope nor the base, and since probe sizes do not enter the average, a probe the shipped line already covers leaves it where it is. + - Until then the line is a declared stand-in, listed in the entry's `stand_ins` and never below the pre-dtype slope. `probe_demand` points on it bound it from below only: `build_shipped.py` keeps it where it covers them by 5%, otherwise raises it as `fit.raised_pair` raises a fitted line, the stand-in as the floor and the average over the sizes of its fresh-process points or, without any, of all its points, and notes the raise in the provenance. It never lowers it. MIOpen with the native allocator off Windows prices fp16 and fp32 at the pre-dtype figure, and CogView4's lines there at its expandable-segments ones. - A value on no line its entry carries is hand-set, and only a spec keyed by no platform trait may carry one (`hand_set_values` in `fit.py`): a keyed spec prices lines whose every value is half of one, so its value off one fails the data test. MOSS-Image's `k_token` is one, bytes per attended token (the image's tokens at 1/16 of each side plus the prompt's), whose entry ships no base for its `token` line. Its points still decide it and bound it from both sides; only the fit check leaves it out. - Every value an entry carries is a stand-in, a `legacy` constant, or decided by an in-fit point. The data test fails when a fitted value is not the fit of its points, naming the value to ship, and when in-fit fresh-process points that span a fit measure a stand-in's line. - - An entry marked `legacy` carries hand-set constants rather than a fit: no point decides them and `fit.py` does not reproduce them, though its estimator's points still bound its estimates. `LEGACY_ENTRIES` in the data test lists the entries, by estimator and platform, allowed to be legacy, so a fit has to remove its entry there too. The builder replaces an entry's legacy values with the fit of its points once they fit every line it carries and each line has a point that holds the upper bound, a fresh-process one: probe windows alone never replace hand-set values. An entry whose platform they do not measure stays legacy. + - An entry marked `legacy` carries hand-set constants rather than a fit: no point decides them and `fit.py` does not reproduce them, though its estimator's points still bound its estimates. `LEGACY_ENTRIES` in the data test lists the entries, by estimator and platform, allowed to be legacy, so a fit has to remove its entry there too. The builder replaces an entry's legacy values with the fit of its points once they fit every line it carries within the upper bounds and each line has a point that holds the upper bound, a fresh-process one: probe windows alone never replace hand-set values. An entry whose platform they do not measure stays legacy. - The Z-Image denoise (`denoise.z_image`, `data/denoise.json`) prices its activations as `k_denoise_` bytes per attended token plus a base of `b_denoise_`. The lines are fitted on fresh-process points of `scripts/calibrate_z_image_working_memory.py` (a random-init transformer of the real geometry with 2 main blocks, bf16, fp16 and fp32 from 512 to 2048 px): an RTX 4090 with the native allocator and with `cudaMallocAsync`, and an RX 9060 XT under ROCm 10 with AOTriton's fused kernels, with expandable segments and with the native allocator on Windows (an entry of its own). In a running server the real 30-block Z-Image Turbo measured below its estimate on the RTX 4090 under the native allocator and `cudaMallocAsync` (bf16) and on the RX 9060 XT with expandable segments (an int8 build). ROCm with the native allocator off Windows has no points and keeps the legacy constants, 0.25 MiB per token and a 1 GiB base for every dtype; they cover bf16 and fp16 but not fp32 from 1440 px up, whose measured lines are steeper. Every other unmeasured platform (XPU, MPS, CPU, cuDNN with expandable segments, MIOpen under `hipMallocAsync`) is priced as a measured one, and the entries it borrows floor every value at the same legacy constants there (`unmeasured_floor`), so no unmeasured platform reserves less than the legacy formula. - The line admits only the plain forward, measured with a fused attention kernel: every analytic term zero and no attention mask, read off the estimator's terms, so a term added later takes the forwards it prices out of the fit. A point whose score matrix is materialized fits nothing, because the score term is an upper bound, and fitting the demand less the term would leave the line low by the term's slack, short for a build of the same platform keys that runs a fused kernel. A ControlNet's hints (n + 4 hint-sized tensors) and a forced math kernel's score matrix are checked against measurements of their own in `tests/app/invocations/test_z_image_denoise_working_memory.py`. - An fp8-stored transformer reserves the per-forward cast of its weights as a term of its own (`util/fp8.py`, `peak_fp8_cast_transient_bytes`): the largest layer's float8 weights at the compute dtype, twice for a scaled checkpoint weight. The legacy base covers that cast too, so wherever the legacy constants price, such a model reserves it twice. @@ -1169,7 +1169,7 @@ The reset, export and summary routes act on the current device class unless `dev `backend/working_memory/user_fit.py` owns the rule; the calibration service owns the store and the cache. - Only the estimators that opt in (`EstimatorSpec.user_calibrated`, the five VAE estimators) take part. The windows of any other, a Z-Image denoise's say, stay in the store and count among the windows that fit no line, but no user line is fitted on them, none moves their estimates, the summary lists none of their lines, and they size no plan: denoise lines are shipped-fit only. -- A user line is fitted per line of `fit.py`: the shipped entry that carries the values (one hop of `values_from`) and `_`. A window joins where its estimator's `line` admits it (for the convolution lines: untiled, one sample, no in-place upcast, an AutoencoderKL: the SD estimator's AutoencoderTiny windows join no line), it is valid and fit-eligible, it is not fragmented, its demand is positive, and its estimator version has shipped values on the platform. Every user point is a `probe_demand` window, so all of them rate the hull's edges. `fit.user_pair` adds `USER_MARGIN`, the shipped 5%, to the envelope and rounds up as the shipped values do. A line needs two sizes `MIN_AREA_RATIO` apart. +- A user line is fitted per line of `fit.py`: the shipped entry that carries the values (one hop of `values_from`) and `_`. A window joins where its estimator's `line` admits it (for the convolution lines: untiled, one sample, no in-place upcast, an AutoencoderKL: the SD estimator's AutoencoderTiny windows join no line), it is valid and fit-eligible, it is not fragmented, its demand is positive, and its estimator version has shipped values on the platform. Every user point is a `probe_demand` window, so all of them rate the hull's edges. No fresh-process point caps a user line and no floor holds it, so that edge always covers its points and the raise of shipped lines (`fit.raised_pair`) has nothing to do there. `fit.user_pair` adds `USER_MARGIN`, the shipped 5%, to the envelope and rounds up as the shipped values do. A line needs two sizes `MIN_AREA_RATIO` apart. - Fragmented windows measured the allocator's history, not the operation. One rule, `fragmentation.fragmented`, judges them for the user fit and for `scripts/working_memory/build_shipped.py`, which flags shipped points by it. It judges a measurement: the fit-eligible windows of one attempt (an owner's `attempt_id`, so two owners' attempts with one id are two measurements; else the run; for the builder also one server lifetime) at one environment, estimator, version and features, in the order they ran. A measurement drops out whole when reserved grew by 10% from its first window to its largest while allocated stayed within 2%, or when its largest window's reserved-to-allocated ratio is 1.2 times the median of the same line's smaller measurements that count, from any attempt. Each attempt is judged on its own, because its windows ran back to back on one allocator state: a recalibration can bring back a size an earlier attempt measured fragmented, and a later fragmented attempt drops no clean one. The ratio rule catches a size whose windows all ran fragmented, or that kept only one fit-eligible window: on an RX 9060 XT the fp32 SDXL decode at 1216 px measured 1.47 against 1.10 at 576 and 832 px, with one fully resident window beside two partly resident ones. Windows of a running attempt are judged but set no reference, as they join no user line yet. Partly resident windows (`partial_residency`) never count and enter neither rule. A size counts when any attempt's windows at it count, and its `windows` and `measured_bytes` cover those windows only. - The windows are those of the current device class's file taken under an environment whose hard facts equal the current ones (same `env_id`). A soft-fact difference keeps the fit. The summary reports a line as `stale`, with the facts that changed, when the soft facts of the newest finished run that fed it differ from the current ones; a run stored without soft facts stands for the environment's, those first recorded with it. So recalibrating after a driver update clears `stale`. Older windows of the same hard environment keep feeding the line: the hard facts are what decide whether a window still measures this machine. A hard mismatch leaves the shipped values. - `EstimatorSpec.estimate` asks the registered provider (`user_fit.set_user_fit_provider`; the service registers it in `start()` and clears it in `stop()`) for the user fit of the spec, platform and device. Without one, the estimate is the shipped one, bit for bit. With one, `user_fit.resolve` reads the operation's line and its size there (output pixels) as `EstimatorSpec.line` places it. At or below the largest measured size the user line replaces the shipped one; above it, the larger of the two estimates wins. A tiled operation is placed by its image, priced on its tile. The `source` starts with `user::` and names the line where the user line decides. A line the user did not measure stays shipped. Because estimates feed `reservation_bytes`, a user line also moves `should_pretile_vae_decode`. diff --git a/docs/src/content/docs/development/Guides/family-e2e.mdx b/docs/src/content/docs/development/Guides/family-e2e.mdx index 412c8d0e1b9..7626fe8b736 100644 --- a/docs/src/content/docs/development/Guides/family-e2e.mdx +++ b/docs/src/content/docs/development/Guides/family-e2e.mdx @@ -1,6 +1,6 @@ --- title: Family E2E Runs -lastUpdated: 2026-10-25 +lastUpdated: 2026-10-29 --- `scripts/family_e2e` runs model families end to end through the HTTP API of a server it launches itself. Per case and size it records the image, the run time, the time per denoise step and the VRAM the server process used, measured at driver level. The report compares them with an explicitly accepted reference and with the previous run of the same profile. @@ -26,7 +26,9 @@ It is a developer tool. It runs on Windows with NVIDIA and AMD (ROCm) GPUs, wher - Per run, the tool waits up to 10 minutes for the queue to drain, samples the server's GPU memory, enqueues the batch and follows its socket events. - When the run finishes, it saves the output image and the memory trace, reads the run's slice of the server log and appends a record to `records.jsonl`. - If the server dies during a step, that step gets an error record and a new server starts on a new scratch root. A case whose steps crashed the server twice is given up; its remaining steps are not run again, not even by `resume`. -5. **Report.** The server is stopped, the scratch root removed, and `report.html` and `summary.md` are written. The exit code is not 0 when a case fails or errors (see [Verdicts](#verdicts)). +5. **Report.** The server is stopped, the scratch root removed, and `report.html` and `summary.md` are written. The exit code is not 0 when a case fails, errors or is incomplete (see [Verdicts](#verdicts)). + - The scratch root goes, and a calibration run's store is copied from it, only once every process of the server is confirmed to have exited: its launcher and serving process as `heartbeat.json` names them, and each process of its tree or group that outlived the kill, checked again. This holds for every server lifetime, also one whose server crashed or failed to start. A process whose state the tool may not read counts as running. + - When the kill fails, a process outlives it (one stuck in the GPU driver, say), or the stop itself is interrupted, the runner keeps the scratch root with its store. `heartbeat.json` goes on naming the server and, under `server_survivors`, each further process that outlived the kill, such as a worker or a launcher the runner could not record. `runner.log` names the pids, and the run fails (an interrupted or stopped run stays stopped). End the processes, then `stop` or `resume`, which stop every process the heartbeat names before they touch the root. ## Data directory @@ -109,11 +111,16 @@ uv run --no-sync python -m scripts.working_memory.build_shipped /runs/ - **Points.** Every valid, fit-eligible window with a positive demand becomes a `probe_demand` point. Per card, estimator, version, platform and features it takes the largest `demand_bytes` of all windows given: repetitions, lifetimes, and several runs or machines with the same card. A card is a device class: vendor, name, memory size, backend and architecture. Cards of one name and another memory size stay apart: a point's `device` is the card's name and its device class id, e.g. `NVIDIA GeForce RTX 4060 Ti (0123456789abcdef)`. A window given twice (the same store in two files) counts once. The provenance names the card, the torch and runtime versions, the calibration run, item and node of the largest window, and its export file. Each point line shows its windows' demands in the order they ran and the largest over the first (`[3 windows, MiB 1960, 1820, 1820; max/first 1.00]`). - **Fragmentation.** A point whose reserved bytes grew by 10% or more from its first window to its largest while the allocated bytes stayed within 2% gets a `warning: fragmentation?` line with both series. That is the allocator's cache growing around the same tensors in a long-running server, not the operation needing more. - **Platforms.** A point's platform is the one `platform_for` gives on the machine that measured it, derived from the export's device class and its environment's hard facts (`platform_for_environment`), whatever machine the builder runs on. It names the OS only where an entry for that platform names it. A window whose environment lacks a fact the platform is read from is left out and counted. -- **Bounds.** `probe_demand` points bound their estimates from below only: the estimate must cover them, and they may raise a line, but they take no part in choosing it (`fit.upper_envelope`), and the 1.3x upper bound and `max_ratio` stay with the fresh-process points, because the cold first run after a server start is the worst case. -- **Merge and fit.** A point joins the data file that declares the estimator's coefficients; where a point of the same measurement (card, platform, method, features and whether it is in the fit) is there, the larger of the two stays, and a remeasurement below the shipped point prints `kept shipped over remeasured `. The carried values are then refitted with `fit.py`. A stand-in line is fitted from then on only once its in-fit fresh-process points alone span two sizes twice apart; until then it stays a stand-in, which `probe_demand` points raise where their fit lies above it but never lower. A fitted line takes, per coefficient, the larger of its fit on all points and its fit on the fresh-process points alone. A refitted entry's provenance gets a note, and so does a raised stand-in's. Every moved line gets a `line` row: old and new slope and base, whether it was raised, and its new points. -- **Legacy entries.** A `legacy` entry (hand-set values, such as the Z-Image denoise's) is replaced by its fit once the points fit every line it carries and each line has a fresh-process point, which holds the upper bound: its flag cleared, its provenance replaced by one that says it is the fit of the platform's points, its `unmeasured_floor` kept. Probe windows alone never replace hand-set values, since a warm window can read below the cold first run. The data test's `LEGACY_ENTRIES`, which lists the legacy entries by estimator and platform, then needs that entry removed. An entry the points do not fully measure keeps its values, with a `note:` line saying which lines lack a fit or a fresh-process point; its new points still have to pass their bounds against the hand-set values, which a fresh-process point well below them does not. +- **Bounds.** `probe_demand` points bound their estimates from below only: the estimate must cover them. They take no part in the fresh-process fit (`fit.upper_envelope`), but they can raise a line or a stand-in. The 1.3x upper bound and `max_ratio` stay with the fresh-process points, because the cold first run after a server start is the worst case. +- **Merge and fit.** A point joins the data file that declares the estimator's coefficients; where a point of the same measurement (card, platform, method, features and whether it is in the fit) is there, the larger of the two stays, and a remeasurement below the shipped point prints `kept shipped over remeasured `. Each line is then refitted with `fit.py`: + - A fitted line starts from its fresh fit: the envelope of its fresh-process points alone, plus the 5% margin, rounded up (`fit.shipped_pair`). A stand-in starts from its values. A stand-in line is fitted from then on once its in-fit fresh-process points alone span two sizes twice apart. + - A line, or a stand-in, that already covers every point by the 5% margin is kept unchanged. + - Otherwise it gets the minimal raise (`fit.raised_pair`): of the lines as they ship, a whole byte of slope per unit of size and a whole MiB of base, whose slope and base each stay at or above its fresh fit (or the stand-in's values), cover every point by 5%, and keep every upper bound of the points priced on the line (`bounds.line_bounds`), the one with the lowest reservation over the size range of its fresh-process points. A probe point therefore never lowers a line, and one the line covers never moves it. + - A refitted entry's provenance gets a note, and so does a raised stand-in's. Every moved line gets a `line` row: old and new slope and base, whether it was raised, and its new points. +- **Legacy entries.** A `legacy` entry (hand-set values, such as the Z-Image denoise's) is replaced by its fit once the points fit every line it carries within the upper bounds and each line has a fresh-process point, which holds the upper bound: its flag cleared, its provenance replaced by one that says it is the fit of the platform's points, its `unmeasured_floor` kept. Probe windows alone never replace hand-set values, since a warm window can read below the cold first run. The data test's `LEGACY_ENTRIES`, which lists the legacy entries by estimator and platform, then needs that entry removed. An entry the points do not fully measure keeps its values, with a `note:` line saying which lines lack a fit or a fresh-process point; its new points still have to pass their bounds against the hand-set values, which a fresh-process point well below them does not. - **Refusals.** An export or point of another estimator version than the registered one, features the estimator cannot evaluate, a platform `MEASURED` does not list, and a line whose points span less than twice its sizes: exit code 2. Exit code 1, with nothing written: - - any point of the result breaks its bounds (`working_memory/bounds.py`, the rule the data test applies); the error lists them. A probe point that raises a line above what the fresh-process points of that line allow is refused this way; + - no line or stand-in raise covers its points within the upper bounds; the error names the point that cannot be covered and the bound it conflicts with; + - any point of the result breaks its bounds (`working_memory/bounds.py`, the rule the data test applies); the error lists them; - with `--write`, a line is raised on a point flagged as fragmentation, unless `--accept-fragmentation` is given. ## Bundle @@ -153,7 +160,8 @@ Each case size gets one status, judged by the final attempt of each step. The fi | Status | When | Failure | | --- | --- | --- | -| `ERROR` | A final step failed or errored. | yes | +| `ERROR` | A final step failed, errored or was canceled. | yes | +| `INCOMPLETE` | A measure the run planned for the case is not final as the runner counts it: it has no record, or its last attempt awaits a repeat for foreign load. That is what `status` counts as left and `resume` would run, e.g. in a run that was stopped or failed before the case finished. | yes | | `FAIL` | An image does not have the requested size. | yes | | `INVALID` | No measured image is valid for PSNR (see [Validity](#validity)). | no | | `NO-REFERENCE` | No accepted reference, or it is unreadable. | no | @@ -164,7 +172,7 @@ Each case size gets one status, judged by the final attempt of each step. The fi | `FAIL` | The case expects a change, but its minimum PSNR is at or above `psnr_min_db`; or it expects none, and the minimum PSNR is below it. | yes | | `PASS` | Otherwise. | no | -The comparison with the previous run is informational: its outliers and flags never change a status. `report` and the runner exit with 1 when a case is `ERROR` or `FAIL`. +The comparison with the previous run is informational: its outliers and flags never change a status. `report` and the runner exit with 1 when a case is `ERROR`, `INCOMPLETE` or `FAIL`. ## Validity @@ -204,8 +212,8 @@ Database migrations of a newer `code_dir` run against the scratch database only. - With PyTorch's native allocator, reserved memory stays allocated between nodes, so a node's peak can stem from an earlier node. Compare peaks only between runs with the same allocator configuration; the run records the runtime config. - On Windows with ROCm, HIP keeps freed blocks at driver level. Per-node peaks after the first large node are upper bounds there (observed on an RX 9060 XT). - Time per step comes from the preview frames, which the server throttles to one per 0.1 s; a step's number is read from its progress percentage. - - For txt2img and controlnet the case's `steps` fixes the step count. - - img2img starts inside the schedule at a family-specific step, so the count is inferred. When steps are faster than 0.2 s that inference can be off by a factor, so the time per step is left empty. + - The step count comes from the compiled denoise node: its `steps` or `num_steps` field, which for most families is the case's `steps`. An Ideogram 4 node without `steps` runs its sampler preset's count. + - A node that denoises a window of its schedule (`denoising_start` above 0 or `denoising_end` below 1, as img2img and an SDXL refiner hand-over do) starts or stops at a family-specific step, so the count is inferred. When steps are faster than 0.2 s that inference can be off by a factor, so the time per step is left empty. - Ideogram 4 ignores `steps`; its step count follows `settings.ideogram4Steps` or `settings.ideogram4SamplerPreset`. `img2img` and `controlnet` are not available for ERNIE-Image and Ideogram 4. Dynamic prompts, canvas rasterization, video and upscale graphs are not covered. - A bundle finds other paths in free text by their drive letter, or as POSIX paths of two or more components that start after a space, a quote or a bracket. Such a path, and a path below the home directory, keeps what follows a space in one of its directory names; a POSIX path at the root (`/x`) or right after `:` stays. A UNC path in free text stays as it is. An `extra_env` credential shorter than 8 characters is redacted in `run.json` only. - A calibration run measures the windows of estimated model operations only, inside a running server: they are not a fresh process's first operation, as the `demand_peak` points are. diff --git a/invokeai/backend/working_memory/bounds.py b/invokeai/backend/working_memory/bounds.py index b7126b2c468..b1e7a64151c 100644 --- a/invokeai/backend/working_memory/bounds.py +++ b/invokeai/backend/working_memory/bounds.py @@ -9,16 +9,21 @@ point, the estimate is at most `MAX_OVER_RESERVATION` times the point, its listed `UPPER_BOUND_EXCEPTIONS` ratio, or its tighter `max_ratio`, which applies whichever term decides. A reservation the operation never uses is model weights evicted for nothing. + +The fit keeps the upper bounds while it raises a line for points above it: `line_bounds` states them as constraints on +the line, which `fit.raised_pair` satisfies. """ import json from collections.abc import Iterable, Mapping +from fractions import Fraction from types import MappingProxyType from typing import TYPE_CHECKING, Final, Optional from invokeai.app.services.config.config_default import InvokeAIAppConfig from invokeai.backend.working_memory.coefficients import MeasuredPoint, WorkingMemoryData, resolve_coefficients from invokeai.backend.working_memory.estimate import WorkingMemoryEstimate +from invokeai.backend.working_memory.fit import LineBound, LineKey, carried_lines, carrier_of, entry_key from invokeai.backend.working_memory.platform import Platform if TYPE_CHECKING: @@ -85,17 +90,84 @@ def fitted_decides(estimate: WorkingMemoryEstimate) -> bool: return any(term.included and term.coefficient is not None for term in estimate.terms) +def upper_bound_ratio(point: MeasuredPoint, spec: "EstimatorSpec", fitted: bool) -> Optional[float]: + """How many times the point its estimate may reserve at most, as a fitted term decides it or not (`fitted`); None + where the point sets it no upper bound. An out-of-fit point checks another term, and a lower-bound-only point is no + cold first run: neither caps a fitted term.""" + if not point.holds_upper_bound: + return None + if point.max_ratio is not None: + return point.max_ratio + if fitted and point.in_fit: + return UPPER_BOUND_EXCEPTIONS.get(point_id(point, spec), MAX_OVER_RESERVATION) + return None + + def bound_violation(point: MeasuredPoint, spec: "EstimatorSpec", estimate: WorkingMemoryEstimate) -> Optional[str]: """Why `estimate`, the estimate `spec` gives `point`'s features, breaks a bound `point` sets for it, or None.""" fitted = fitted_decides(estimate) reserved = estimate.reservation_bytes if fitted else max(estimate.reservation_bytes, WORKING_MEMORY_FLOOR) if reserved < point.measured_bytes: return f"reserves {reserved / 2**30:.3f} GiB for a measured {point.measured_bytes / 2**30:.3f} GiB" - # An out-of-fit point checks another term, and a lower-bound-only point is no cold first run: neither caps. - if point.holds_upper_bound and (point.max_ratio is not None or (fitted and point.in_fit)): - default = UPPER_BOUND_EXCEPTIONS.get(point_id(point, spec), MAX_OVER_RESERVATION) - max_ratio = point.max_ratio if point.max_ratio is not None else default + max_ratio = upper_bound_ratio(point, spec, fitted) + if max_ratio is not None: ratio = estimate.reservation_bytes / point.measured_bytes if ratio > max_ratio: return f"reserves {ratio:.2f}x the measured peak, over {max_ratio}x" return None + + +_MIB: Final = 2**20 + +_DECIDING: Final = float(2**60) +"""A coefficient value large enough that the terms priced with it decide any estimate they take part in. A line's +terms are read where it decides, since only there can raising it break an upper bound.""" + + +def line_bounds( + data: Iterable[WorkingMemoryData], specs: Mapping[str, "EstimatorSpec"] +) -> Mapping[LineKey, list[LineBound]]: + """Per line, the upper bounds its points set on it, which `fit.raised_pair` keeps when it raises the line. + + A point bounds the line its estimator's `EstimatorSpec.line` places its features on, on the entry that carries its + values (`fit.carrier_of`), where it holds an upper bound with a fitted term deciding (`upper_bound_ratio`). That + includes a point the line does not admit to its fit (`fit.Line.fits`): a tiled, batched or upcast VAE operation, a + denoise forward with an analytic term. Its terms are read where the line's decide (`_DECIDING`), the other included + terms at the bytes `data` prices them at, so the bound reads ``k * size + b * bases + others <= ratio * measured``, + `size` in the units the terms price at the slope (pixels, tokens, a batch's samples). Where another term decides at + the raised line, this holds the line to more than `bound_violation` asks: the line still lies within the point's + ratio. + + The other terms keep the bytes of the values `data` carries, before any refit: a term on a second line that the + same refit moves would be priced at its old value here. No shipped estimator prices one point on two lines.""" + files = list(data) + entries = [entry for file in files for entry in file.coefficients] + bounds: dict[LineKey, list[LineBound]] = {} + for point in (point for file in files for point in file.points): + spec = specs[point.estimator] + found = spec.line(point.features) + ratio = upper_bound_ratio(point, spec, fitted=True) + if found is None or ratio is None: + continue + carrier = carrier_of(entries, point) + if found.suffix not in carried_lines(carrier): + continue + k_name, b_name = f"k_{found.suffix}", f"b_{found.suffix}" + values = dict(resolve_coefficients(files, spec, _platform_of(point, spec), origin="data").values) + terms = spec.terms(point.features, values | {k_name: _DECIDING, b_name: _DECIDING}) + on_line = [term for term in terms if term.included and term.coefficient in (k_name, b_name)] + if not on_line: + continue + others = sum(term.bytes for term in terms if term.included and term.coefficient not in (k_name, b_name)) + allowed = Fraction(ratio) * point.measured_bytes + # Two points can share an id (repeated measurements); their bytes tell them apart. + measured = f"measured {point.measured_bytes / _MIB:.0f} MiB, at most {allowed / _MIB:.0f} MiB" + label = f"{point_id(point, spec)} ({measured})" + bound = LineBound( + size=sum((Fraction(term.scale or 0) for term in on_line if term.coefficient == k_name), Fraction(0)), + bases=sum((Fraction(term.scale or 0) for term in on_line if term.coefficient == b_name), Fraction(0)), + ceiling=allowed - Fraction(others), + point=label, + ) + bounds.setdefault((entry_key(carrier), found.suffix), []).append(bound) + return bounds diff --git a/invokeai/backend/working_memory/fit.py b/invokeai/backend/working_memory/fit.py index 92b3b299fc8..54f065b33af 100644 --- a/invokeai/backend/working_memory/fit.py +++ b/invokeai/backend/working_memory/fit.py @@ -4,22 +4,25 @@ `b_` bytes, the base a process pays whatever the size. Its `EstimatorSpec.line` says which line an operation's features price on, their size on it, and whether a measurement of them can join its fit (`Line`). The AutoencoderKL estimators price their convolutions on a line in the output's pixels (`convolution_line`); the Z-Image denoise prices -its forward on one in attended tokens. Each carried pair is the upper envelope of the points it is -resolved for (see `upper_envelope`), plus `SHIPPED_MARGIN`. A point joins the fit of the entry its estimator resolves -to on its platform -- the lender's, where the estimator borrows -- when it is `in_fit` (not measured out of regime) and -its line admits it. +its forward on one in attended tokens. Each carried pair is the upper envelope of the fresh-process points it is +resolved for (see `upper_envelope`), plus `SHIPPED_MARGIN`, raised where the line's other points need it +(`raised_pair`). A point joins the fit of the entry its estimator resolves to on its platform -- the lender's, where the +estimator borrows -- when it is `in_fit` (not measured out of regime) and its line admits it. A line is fitted on its points' demand as measured; no term is subtracted from it. So a line admits only measurements -whose demand is the line's alone. Where an estimate adds a term of its own on top of the line, a measurement of it -fits nothing: a tiled VAE operation (the tile term reuses the untiled pair), a VAE an in-place cast grew -(`upcast_weight_bytes`), and every denoise forward with an analytic term. The VAE mid-block score matrix is not such a -term: an estimate reserves the larger of it and the convolutions, which peak apart, and on the builds that materialize -it the convolutions stay the larger over the measured sizes (`vae_working_memory._peak_of_two_phases`); the CUDA points -that force a math kernel are out of fit. A point that bounds its estimate from below only -(`MeasuredPoint.holds_upper_bound`, a `probe_demand` window) is covered by the line and can raise it, but takes no part -in choosing it: the fresh-process points cap a line. A line is fitted only once its fresh-process points alone span -two sizes `MIN_AREA_RATIO` apart (`fitted_pair`); until then it stays a stand-in, which the other points bound from -below. +whose demand is the line's alone. Where an estimate adds a term of its own on top of the line, a measurement of it fits +nothing: a tiled VAE operation (the tile term reuses the untiled pair), a VAE an in-place cast grew +(`upcast_weight_bytes`), and every denoise forward with an analytic term. Such a measurement still bounds the line from +above where it holds an upper bound (`bounds.line_bounds`). The VAE mid-block score matrix is not such a term: an +estimate reserves the larger of it and the convolutions, which peak apart, and on the builds that materialize it the +convolutions stay the larger over the measured sizes (`vae_working_memory._peak_of_two_phases`); the CUDA points that +force a math kernel are out of fit. A point that bounds its estimate from below only (`MeasuredPoint.holds_upper_bound`, +a `probe_demand` window) takes no part in the envelope: the fresh-process points cap a line. It is covered by the line +all the same. Where the envelope covers it by the margin, the pair stays as it is; where not, `raised_pair` raises it by +the least that keeps the upper bounds of every point priced on the line (`bounds.line_bounds`), found exactly among all +lines, and refuses, naming the points that conflict, only where no line keeps them (`LineConflict`). A line is fitted +only once its fresh-process points alone span two sizes `MIN_AREA_RATIO` apart (`fitted_pair`); until then it stays a +stand-in, which the other points bound from below and raise the same way. `tests/backend/working_memory/test_shipped_coefficients.py` holds every carried value that is not a declared stand-in to `fitted_values` of the shipped points, so a new point that moves a fit fails there with the value to ship. A @@ -29,12 +32,14 @@ there. A user's own calibration fits the same lines from its windows (`user_pair`, read by `user_fit.py`), with -`USER_MARGIN`, the same margin, for the estimators that opt in (`EstimatorSpec.user_calibrated`). +`USER_MARGIN`, the same margin, for the estimators that opt in (`EstimatorSpec.user_calibrated`). No fresh-process +point caps a user line and no floor holds it, so its envelope always covers its points and is never raised. """ import math -from collections.abc import Callable, Iterable, Mapping, Sequence +from collections.abc import Callable, Iterable, Iterator, Mapping, Sequence from fractions import Fraction +from itertools import combinations from typing import TYPE_CHECKING, Final, NamedTuple, Optional from invokeai.backend.working_memory.coefficients import CoefficientEntry, MeasuredPoint, WorkingMemoryData, find_entry @@ -60,6 +65,9 @@ EntryKey = tuple[str, int, tuple[tuple[str, str], ...]] """An entry's estimator id, version and platform keys: what a point is resolved by.""" +LineKey = tuple[EntryKey, str] +"""A line: the entry that carries its values, and its suffix (`Line.suffix`).""" + class Line(NamedTuple): """Where an operation's features place it on the line its estimator's fitted terms price it on.""" @@ -119,24 +127,39 @@ class FitPoint(NamedTuple): """Whether the point takes part in choosing the line (`MeasuredPoint.holds_upper_bound`); every point is covered.""" +class LineBound(NamedTuple): + """An upper bound a point sets on a line it is priced on (`bounds.line_bounds`): its estimate's terms on the line, + ``k * size + b * bases``, at most `ceiling` bytes. `size` is in the line's unit and counts every unit the estimate + prices at the slope: a batch's samples, a tile with its overlap.""" + + size: Fraction + bases: Fraction + ceiling: Fraction + point: str + """Names the point where no line keeps the bound: its id (`bounds.point_id`), which two points can share, with + the bytes it measured and the most its estimate may reserve.""" + + +class LineConflict(ValueError): + """No line at or above a line's floor covers its points within the upper bounds of the points priced on it: the + data conflicts with the bounds, rather than measuring the line too little.""" + + def spans(sizes: Iterable[int]) -> bool: """Whether the sizes include two at least `MIN_AREA_RATIO` apart: enough to fit a line on.""" distinct = sorted(set(sizes)) return len(distinct) >= 2 and distinct[-1] >= MIN_AREA_RATIO * distinct[0] -def upper_envelope( - points: Sequence[tuple[int, int]], capping: Optional[Sequence[tuple[int, int]]] = None -) -> tuple[Fraction, Fraction]: +def upper_envelope(points: Sequence[tuple[int, int]]) -> tuple[Fraction, Fraction]: """The line ``y = k * x + b`` on or above every ``(x, y)`` point, with ``b >= 0``, that over-reserves least. The candidates are the edges of the upper convex hull of the points and the origin: each such line lies on or above every point, and the origin keeps its intercept non-negative. Of those, the one whose largest ratio to the highest reading of a size is smallest wins, the steeper one on a tie, since it is the more conservative past the largest point. Only the highest reading counts: a lower one at a measured size is covered whatever the line, and - rating against it would tilt the line though no point lies above it. With `capping`, a subset of the points, only - its readings rate the edges: the others are covered but cap nothing. Without any, all points rate them. Raises - unless the points span two sizes at least `MIN_AREA_RATIO` apart. + rating against it would tilt the line though no point lies above it. Raises unless the points span two sizes at + least `MIN_AREA_RATIO` apart. """ if not points or any(x <= 0 or y <= 0 for x, y in points): raise ValueError(f"an envelope needs points with positive sizes and bytes, got {list(points)}") @@ -157,26 +180,23 @@ def upper_envelope( hull.pop() hull.append(point) - rated = {x: max(y for px, y in capping if px == x) for x in sorted({x for x, _ in capping})} if capping else highest best: tuple[Fraction, Fraction, Fraction] | None = None for (x0, y0), (x1, y1) in zip(hull, hull[1:], strict=False): k = Fraction(y1 - y0, x1 - x0) b = y0 - k * x0 if k < 0: continue - worst = max((k * x + b) / y for x, y in rated.items()) + worst = max((k * x + b) / y for x, y in highest.items()) if best is None or (worst, -k) < (best[0], -best[1]): best = (worst, k, b) assert best is not None, "the edge from the origin always rises" return best[1], best[2] -def shipped_pair( - points: Sequence[tuple[int, int]], capping: Optional[Sequence[tuple[int, int]]] = None -) -> tuple[int, int]: +def shipped_pair(points: Sequence[tuple[int, int]]) -> tuple[int, int]: """The slope and intercept to ship for these points: the envelope plus `SHIPPED_MARGIN`, the slope rounded up to a whole byte per unit of size and the intercept to a whole MiB.""" - k, b = upper_envelope(points, capping) + k, b = upper_envelope(points) margin = 1 + SHIPPED_MARGIN return math.ceil(k * margin), math.ceil(b * margin / _MIB) * _MIB @@ -185,9 +205,9 @@ def user_pair(points: Sequence[tuple[int, int]]) -> tuple[int, int]: """The slope and intercept a user calibration's points give a line: the envelope plus `USER_MARGIN`, the slope rounded up to a whole byte per unit of size and the intercept to a whole MiB. - Every user point is a `probe_demand` window, so no point caps the line and all of them rate its edges (the - fallback of `upper_envelope`). Raises like `upper_envelope` unless the points span two sizes `MIN_AREA_RATIO` - apart.""" + Every user point is a `probe_demand` window, so all of them rate the envelope's edges, and none caps the line or + floors it: the envelope covers them all, with no raise (`raised_pair`) to search for. Raises like + `upper_envelope` unless the points span two sizes `MIN_AREA_RATIO` apart.""" k, b = upper_envelope(points) margin = 1 + USER_MARGIN return math.ceil(k * margin), math.ceil(b * margin / _MIB) * _MIB @@ -213,7 +233,7 @@ def hand_set_values(entry: CoefficientEntry, spec: "EstimatorSpec") -> set[str]: def fit_points( data: Iterable[WorkingMemoryData], specs: Mapping[str, "EstimatorSpec"] -) -> Mapping[tuple[EntryKey, str], list[FitPoint]]: +) -> Mapping[LineKey, list[FitPoint]]: """Per carried entry and line, the points that line is fitted on. A point belongs to the entry its own estimator's entry resolves to on its platform, one `values_from` hop, and to @@ -222,7 +242,7 @@ def fit_points( """ files = list(data) entries = [entry for file in files for entry in file.coefficients] - groups: dict[tuple[EntryKey, str], list[FitPoint]] = {} + groups: dict[LineKey, list[FitPoint]] = {} for point in (point for file in files for point in file.points): if not point.in_fit: continue @@ -247,13 +267,17 @@ def carrier_of(entries: Sequence[CoefficientEntry], point: MeasuredPoint) -> Coe def entry_fit( - entry: CoefficientEntry, groups: Mapping[tuple[EntryKey, str], Sequence[FitPoint]] -) -> tuple[dict[str, int], dict[str, str]]: + entry: CoefficientEntry, + groups: Mapping[LineKey, Sequence[FitPoint]], + bounds: Mapping[LineKey, Sequence[LineBound]], +) -> tuple[dict[str, float], dict[str, str], dict[str, str]]: """What `fitted_pair` gives each line `entry` carries and does not declare a stand-in, from `groups` - (`fit_points`); and, per such line its points cannot fit, why.""" + (`fit_points`) within `bounds` (`bounds.line_bounds` of the same data); per such line its points cannot fit, why; + and per such line whose points conflict with the bounds (`LineConflict`), why.""" key = entry_key(entry) - values: dict[str, int] = {} + values: dict[str, float] = {} problems: dict[str, str] = {} + conflicts: dict[str, str] = {} for suffix in sorted(carried_lines(entry)): if {f"k_{suffix}", f"b_{suffix}"} <= set(entry.stand_ins): continue @@ -262,32 +286,41 @@ def entry_fit( problems[suffix] = "no in-fit point measures it" continue try: - k, b = fitted_pair(line) + k, b = fitted_pair(line, bounds.get((key, suffix), ())) + except LineConflict as e: + conflicts[suffix] = str(e) + continue except ValueError as e: problems[suffix] = str(e) continue values |= {f"k_{suffix}": k, f"b_{suffix}": b} - return values, problems + return values, problems, conflicts def fitted_values( - data: Iterable[WorkingMemoryData], specs: Mapping[str, "EstimatorSpec"] -) -> Mapping[EntryKey, Mapping[str, int]]: + data: Iterable[WorkingMemoryData], + specs: Mapping[str, "EstimatorSpec"], + bounds: Mapping[LineKey, Sequence[LineBound]], +) -> Mapping[EntryKey, Mapping[str, float]]: """Per entry that carries values and is not `legacy`, what `fitted_pair` gives each line it does not declare a - stand-in (`entry_fit`). + stand-in (`entry_fit`), within the upper bounds `bounds` holds for it (`bounds.line_bounds` of the same data). - Raises, naming the entry and line, where the points cannot fit that line. + Raises, naming the entry and line, where the points cannot fit that line: `ValueError` where they do not measure + it well enough, `LineConflict` where they conflict with the upper bounds. """ files = list(data) groups = fit_points(files, specs) - fitted: dict[EntryKey, dict[str, int]] = {} + fitted: dict[EntryKey, dict[str, float]] = {} for entry in (entry for file in files for entry in file.coefficients): if entry.values is None or entry.legacy: continue - values, problems = entry_fit(entry, groups) + values, problems, conflicts = entry_fit(entry, groups, bounds) if problems: suffix, problem = next(iter(problems.items())) raise ValueError(f"{entry.estimator} on {entry.platform}, {suffix}: {problem}") + if conflicts: + suffix, conflict = next(iter(conflicts.items())) + raise LineConflict(f"{entry.estimator} on {entry.platform}, {suffix}: {conflict}") if values: fitted[entry_key(entry)] = values return fitted @@ -299,18 +332,160 @@ def capping_spans(line: Sequence[FitPoint]) -> bool: return spans(point.size for point in line if point.caps) -def fitted_pair(line: Sequence[FitPoint]) -> tuple[int, int]: - """The pair to ship for a line (`shipped_pair`): the coefficient-wise larger of its fit on all its points, rated on the - fresh-process ones, and its fit on the fresh-process points alone. A probe point above the hull can drop a fresh - vertex and so lower one half of the line; the larger of the two keeps both at the fresh-process fit at least. +def fitted_pair(line: Sequence[FitPoint], bounds: Sequence[LineBound]) -> tuple[float, float]: + """The pair to ship for a line: the fit on its fresh-process points alone (`shipped_pair`), raised by + `raised_pair` to cover its other points within `bounds`. A `probe_demand` point thus never lowers either half of + the fresh-process fit, and one that fit already covers leaves it as it is. - Raises unless the fresh-process points span (`capping_spans`).""" + Raises unless the fresh-process points span (`capping_spans`), and where no raise keeps every bound.""" capping = [(p.size, p.bytes) for p in line if p.caps] if not capping_spans(line): raise ValueError( f"its fresh-process points measure sizes {sorted({x for x, _ in capping})}, not two at least " f"{MIN_AREA_RATIO}x apart; keep it a stand-in until they do" ) - k_all, b_all = shipped_pair([(p.size, p.bytes) for p in line], capping) - k_fresh, b_fresh = shipped_pair(capping) - return max(k_all, k_fresh), max(b_all, b_fresh) + return raised_pair(shipped_pair(capping), line, bounds) + + +class _HalfPlane(NamedTuple): + """``k * size + b * bases >= limit``, or ``<=`` where `upper`, on a candidate line ``(k, b)``.""" + + size: Fraction + bases: Fraction + limit: Fraction + upper: bool + why: str + + def holds(self, k: Fraction, b: Fraction) -> bool: + value = k * self.size + b * self.bases + return value <= self.limit if self.upper else value >= self.limit + + +def _corners(constraints: Sequence[_HalfPlane]) -> Iterator[tuple[Fraction, Fraction]]: + """Every line where the boundaries of two constraints meet and all of them hold: the corners of the region of + lines that satisfy them, exactly. With both floors among them the region has a corner unless it is empty.""" + for p, q in combinations(constraints, 2): + det = p.size * q.bases - p.bases * q.size + if det: + k = (p.limit * q.bases - p.bases * q.limit) / det + b = (p.size * q.limit - p.limit * q.size) / det + if all(c.holds(k, b) for c in constraints): + yield k, b + + +def _lowest_rounded(constraints: Sequence[_HalfPlane], middle: Fraction) -> Optional[tuple[int, int]]: + """The rounded line that satisfies every one of `constraints` with the smallest ``k * middle + b``, the steeper on + a tie; None where no rounded line does. A rounded line is what ships: a whole byte of slope per unit of size and a + whole MiB of base, ``(k, b)`` with ``k`` an integer and ``b`` a multiple of a MiB. + + Exact and complete. The rounded lines that can satisfy the constraints lie in the real region they bound, so the + search runs over each whole MiB of base from the region's lowest base up. At a base, every constraint is a bound on + the slope, and the lowest whole slope within them is that base's best line, since the objective rises with the + slope. The search stops above the region's highest base, which caps that bound it in the base imply; without + such a cap, past every coverage limit, above which no constraint moves with the base; and once a base alone, + at the region's lowest slope, costs more than the best line found.""" + corners = list(_corners(constraints)) + if not corners: + return None + k_least = min(k for k, _ in corners) + b_least = min(b for _, b in corners) + if any(c.upper and c.bases > 0 for c in constraints): + b_most = max(b for _, b in corners) + else: + # Unbounded in the base: above every limit, a larger base satisfies nothing more. + b_most = max([b for _, b in corners] + [c.limit / c.bases for c in constraints if c.bases > 0]) + _MIB + best: Optional[tuple[Fraction, int, int]] = None + mib = math.ceil(b_least / _MIB) + while mib * _MIB <= b_most and (best is None or k_least * middle + mib * _MIB <= best[0]): + b = mib * _MIB + mib += 1 + k_low, k_high = k_least, None + feasible = True + for c in constraints: + rest = c.limit - c.bases * b + if not c.size: + feasible = feasible and (rest >= 0 if c.upper else rest <= 0) + elif c.upper: + k_high = rest / c.size if k_high is None else min(k_high, rest / c.size) + else: + k_low = max(k_low, rest / c.size) + k = math.ceil(k_low) + if not feasible or (k_high is not None and k > k_high): + continue + cost = k * middle + b + if best is None or (cost, -k) < (best[0], -best[1]): + best = (cost, k, b) + return None if best is None else (best[1], best[2]) + + +def _conflict(floors: Sequence[_HalfPlane], others: Sequence[_HalfPlane], middle: Fraction) -> list[_HalfPlane]: + """Of `others`, a set that no rounded line at or above the floors satisfies (`_lowest_rounded`), though one + satisfies each of its proper subsets: each constraint is dropped while the rest still conflict.""" + kept = list(others) + for constraint in others: + rest = [c for c in kept if c is not constraint] + if _lowest_rounded([*floors, *rest], middle) is None: + kept = rest + return kept + + +def raised_pair( + floor: tuple[float, float], line: Sequence[FitPoint], bounds: Sequence[LineBound] +) -> tuple[float, float]: + """The lowest rounded line at or above `floor`, per coefficient, that covers every point of `line` by + `SHIPPED_MARGIN` and keeps every one of `bounds`. + + `floor` itself where it covers every point by the margin: a point it covers changes nothing, and `bounds` are not + consulted, since the floor's own bounds are the data test's to hold. Otherwise every line is a candidate, not only + the edges of a hull, and the candidates are the lines that ship: rounded as `shipped_pair` rounds, a whole byte of + slope per unit of size and a whole MiB of base. Each constraint is a half-plane in ``(k, b)``, so the lines that + satisfy them all form a convex region; `_lowest_rounded` finds the lowest rounded line in it exactly, checking + each candidate against the constraints as they are, with no allowance. The lowest line is the one with the smallest + reservation averaged over the sizes the line's fresh-process points span (``k`` times the middle of that range, + plus ``b``), or all its points' sizes where none caps it; the steeper on a tie, as `upper_envelope` breaks them. + Probe sizes do not move that measure, so a line raised for some probes stays where it is when another probe it + covers arrives. + + Raises `LineConflict`, naming the points that cannot be satisfied together, where no rounded line at or above + `floor` covers the points within `bounds`, also where only a line between whole bytes or MiB would. Raises + `ValueError` where a raise is needed and `line` has fresh-process points but `bounds` states none: those points + always bound the line (`bounds.line_bounds`), so empty `bounds` there means the caller left them out, and a raise + without them could break every one.""" + k_floor, b_floor = Fraction(floor[0]), Fraction(floor[1]) + margin = 1 + SHIPPED_MARGIN + highest: dict[int, int] = {} + for point in line: + highest[point.size] = max(highest.get(point.size, 0), point.bytes) + # Every line at or above the floor covers what the floor covers: only the other points constrain the raise. + uncovered = [ + _HalfPlane(Fraction(x), Fraction(1), margin * y, False, f"{y / _MIB:.0f} MiB at size {x}") + for x, y in sorted(highest.items()) + if k_floor * x + b_floor < margin * y + ] + if not uncovered: + return floor + if not bounds and any(point.caps for point in line): + raise ValueError( + "the line's fresh-process points bound its raise, but no upper bounds are given for it; pass " + "`bounds.line_bounds` of the same data" + ) + floors = [ + _HalfPlane(Fraction(1), Fraction(0), k_floor, False, "the slope floor"), + _HalfPlane(Fraction(0), Fraction(1), b_floor, False, "the base floor"), + ] + caps = [_HalfPlane(bound.size, bound.bases, bound.ceiling, True, bound.point) for bound in bounds] + sizes = [point.size for point in line if point.caps] or list(highest) + middle = Fraction(min(sizes) + max(sizes), 2) + best = _lowest_rounded([*floors, *uncovered, *caps], middle) + if best is None: + conflict = _conflict(floors, [*uncovered, *caps], middle) + covers = "; ".join(c.why for c in conflict if not c.upper) + within = "; ".join(c.why for c in conflict if c.upper) + at = f"k={floor[0]:.0f}, b={floor[1] / _MIB:.0f} MiB" + if not covers: + # Covering points never conflicts with the floors alone: the floor itself breaks a bound. + raise LineConflict(f"the line's floor {at} already exceeds the upper bound of {within}") + raise LineConflict( + f"no line at or above {at} covers {covers} by {SHIPPED_MARGIN:.0%} within the upper bound of {within}" + ) + return best diff --git a/invokeai/backend/working_memory/user_fit.py b/invokeai/backend/working_memory/user_fit.py index 0bd46adb432..e109e0b2eb9 100644 --- a/invokeai/backend/working_memory/user_fit.py +++ b/invokeai/backend/working_memory/user_fit.py @@ -26,15 +26,12 @@ from invokeai.backend.util.logging import InvokeAILogger from invokeai.backend.working_memory.coefficients import ResolvedCoefficients, find_entry, shipped_data from invokeai.backend.working_memory.estimate import FeatureValue, WorkingMemoryEstimate -from invokeai.backend.working_memory.fit import EntryKey, carried_lines, entry_key, user_pair +from invokeai.backend.working_memory.fit import EntryKey, LineKey, carried_lines, entry_key, user_pair from invokeai.backend.working_memory.platform import Platform, priced_as if TYPE_CHECKING: from invokeai.backend.working_memory.spec import EstimatorSpec -LineKey = tuple[EntryKey, str] -"""A line: the entry that carries its values, and its suffix (`fit.Line.suffix`).""" - @dataclass(frozen=True) class UserPoint: diff --git a/scripts/family_e2e/__main__.py b/scripts/family_e2e/__main__.py index 7ae1b434bf4..dc93beea93c 100644 --- a/scripts/family_e2e/__main__.py +++ b/scripts/family_e2e/__main__.py @@ -21,11 +21,17 @@ from scripts.family_e2e.graphs import ExportError from scripts.family_e2e.manifest import DEFAULT_ARM, Manifest, ManifestError, load_manifest from scripts.family_e2e.metrics import MB -from scripts.family_e2e.records import RUN_INFO_FILE, load_records, load_run_info, run_id_profile, write_run_info +from scripts.family_e2e.records import ( + RUN_INFO_FILE, + load_records, + load_run_info, + remaining_steps, + run_id_profile, + write_run_info, +) from scripts.family_e2e.references import AcceptError, accept from scripts.family_e2e.report import write_report from scripts.family_e2e.runner import ( - SERVER_ROLES, RunnerError, RunStopped, finish_run, @@ -37,7 +43,7 @@ open_gpu, preflight, read_heartbeat, - remaining_steps, + recorded_server, run, run_lock, save_calibration, @@ -199,7 +205,7 @@ def cmd_stop(args: argparse.Namespace) -> int: heartbeat = read_heartbeat(run_dir) # The runner first: it would otherwise notice its server dying and record a crash. killed, survivors = kill_recorded([("runner", heartbeat.get("runner"))]) - killed_servers, left = kill_recorded([(role, heartbeat.get(role)) for role in SERVER_ROLES]) + killed_servers, left = kill_recorded(recorded_server(heartbeat)) killed += killed_servers survivors = ", ".join(s for s in (survivors, left) if s) if survivors: diff --git a/scripts/family_e2e/client.py b/scripts/family_e2e/client.py index 441fde05932..30edfa8e7aa 100644 --- a/scripts/family_e2e/client.py +++ b/scripts/family_e2e/client.py @@ -11,7 +11,7 @@ import threading import time from collections import defaultdict, deque -from collections.abc import Callable +from collections.abc import Callable, Mapping from dataclasses import dataclass from pathlib import Path from typing import Any @@ -248,11 +248,12 @@ def build_timing( t_enqueue: float, t_done: float, item: dict[str, Any], - expected_steps: int | None = None, + step_counts: Mapping[str, int] | None = None, ) -> Timing: """Run timing from the recorded events; every `t_*` is relative to the enqueue instant. - `expected_steps` is the denoise step count when the run covers the whole schedule (not img2img). + `step_counts` maps a graph node id to the denoise step count the node runs over its whole schedule + (`graphs.denoise_step_counts`); the time per step trusts the count of the node that sent the preview frames. """ # The socket client runs every handler on its own thread, so the list order can differ from the arrival order. events = sorted(events, key=lambda e: e.t) @@ -298,7 +299,13 @@ def build_timing( loads[pending[key].popleft()]["t_end"] = event.t - t_enqueue # The denoise node is the one with the most preview frames. - denoise = max(progress.values(), key=len, default=[]) + denoise: list[tuple[float, float]] = [] + expected_steps = None + if progress: + denoise_id = max(progress, key=lambda prepared_id: len(progress[prepared_id])) + denoise = progress[denoise_id] + if denoise_id in starts: + expected_steps = (step_counts or {}).get(starts[denoise_id][0]) return Timing( total_s=t_done - t_enqueue, nodes=nodes, diff --git a/scripts/family_e2e/graphs.py b/scripts/family_e2e/graphs.py index 77e2e9d0b0f..ffe2b44c5fe 100644 --- a/scripts/family_e2e/graphs.py +++ b/scripts/family_e2e/graphs.py @@ -11,6 +11,7 @@ import hashlib import ipaddress import json +import logging import shutil import subprocess import urllib.parse @@ -20,13 +21,19 @@ from typing import Any from scripts.family_e2e.manifest import TOOL_CHECKOUT, ResolvedCase -from scripts.family_e2e.server import spawn_in_job, terminate +from scripts.family_e2e.server import TerminationError, spawn_in_job, terminate + +logger = logging.getLogger("family_e2e") EXPORT_VERSION = 1 WEBV2_DIR = TOOL_CHECKOUT / "invokeai" / "frontend" / "webv2" EXPORT_TIMEOUT_S = 600 EXPORT_REQUEST_FILE = "export-request.json" """In the exporter's work directory; holds every model config of the server, paths included.""" +IDEOGRAM4_DENOISE = "ideogram4_denoise" +# The step count of each Ideogram 4 sampler preset (PRESETS in invokeai/backend/ideogram4/sampler_configs.py). The tool +# imports no invokeai code, which would pull in torch; a test keeps the two in step. +IDEOGRAM4_PRESET_STEPS = {"V4_QUALITY_48": 48, "V4_DEFAULT_20": 20, "V4_TURBO_12": 12} # Hosts that name a machine only on its own network. LOCAL_HOST_SUFFIXES = (".local", ".localhost") @@ -44,6 +51,8 @@ class PreparedGraph: """Keys of every model the graph names (main model, components, LoRAs, control models; `graph_model_keys`).""" components: dict[str, str] """The exporter's resolved component choice, for provenance.""" + step_counts: dict[str, int] + """The denoise step count of each node that states the one it runs (`denoise_step_counts`), by node id.""" def build_export_request( @@ -117,7 +126,11 @@ def run_pnpm_exporter(request: dict[str, Any], work_dir: Path) -> dict[str, Any] raise ExportError(f"pnpm graphs:export did not finish within {EXPORT_TIMEOUT_S}s") from None finally: # A finished communicate() has reaped pnpm already; children it left behind still hold its group's id. - terminate(process, job) + try: + terminate(process, job) + except TerminationError as e: + # A lingering exporter process touches no scratch root; what pnpm answered stands. + logger.warning("the graph exporter's processes did not all exit: %s", e) if process.returncode != 0 or not response_path.is_file(): output = (stderr or stdout).decode("utf-8", errors="replace") raise ExportError(f"pnpm graphs:export exited with {process.returncode}:\n{output[-4000:]}") @@ -201,6 +214,25 @@ def shareable_source(config: dict[str, Any]) -> str | None: return None +def denoise_step_counts(graph: dict[str, Any]) -> dict[str, int]: + """The number of steps each node of the graph runs over its whole schedule, by node id, where the node states it. + + That is its `steps` or `num_steps` field. An Ideogram 4 denoise node without `steps` runs its sampler preset's + count instead. A node that denoises a window of its schedule (`denoising_start` above 0 or `denoising_end` below 1, + as img2img or an SDXL refiner hand-over does) runs a part of it that each family cuts differently: it states none. + """ + counts: dict[str, int] = {} + for node_id, node in graph["nodes"].items(): + if node.get("denoising_start", 0) > 0 or node.get("denoising_end", 1) < 1: + continue + steps = node.get("steps", node.get("num_steps")) + if steps is None and node.get("type") == IDEOGRAM4_DENOISE: + steps = IDEOGRAM4_PRESET_STEPS.get(node.get("sampler_preset")) + if type(steps) is int and steps > 0: + counts[node_id] = steps + return counts + + def _replace_strings(value: Any, replacements: dict[str, str]) -> Any: if isinstance(value, str): return replacements.get(value, value) @@ -228,6 +260,7 @@ def prepare_batch(batch: dict[str, Any], components: dict[str, str], input_ids: graph_hash=hashlib.sha256(canonical.encode("utf-8")).hexdigest(), model_keys=graph_model_keys(batch["graph"]), components=components, + step_counts=denoise_step_counts(batch["graph"]), ) diff --git a/scripts/family_e2e/records.py b/scripts/family_e2e/records.py index 59721b3f550..f795823d94c 100644 --- a/scripts/family_e2e/records.py +++ b/scripts/family_e2e/records.py @@ -23,6 +23,13 @@ # Windows refuses os.replace while another process (e.g. `status`) holds the target open; such reads are brief. REPLACE_ATTEMPTS = 20 REPLACE_RETRY_DELAY_S = 0.05 +MAX_ATTEMPTS = 3 +# A case whose steps crashed the server this often is given up; its remaining steps count as final. +MAX_SERVER_CRASHES = 2 +# Reasons that make a run repeat (up to MAX_ATTEMPTS records per step); the generation itself was fine. +FOREIGN_GPU_LOAD = "foreign GPU load" +FOREIGN_QUEUE_ITEM = "foreign queue item" +SERVER_CRASHED = "server crashed" Kind = Literal["warmup", "measure"] Expect = Literal["unchanged", "changed"] @@ -343,3 +350,44 @@ def run_id_profile(run_id: str) -> str | None: """The profile part of a run id, or None for a name that is no run id.""" match = RUN_ID.match(run_id) return match["profile"] if match else None + + +# Step finality, shared by the runner (what is left to run) and the report (what a verdict may rest on). +def step_id(step: PlannedStep | Record) -> tuple[str, str, str, int]: + return (step.case_key, step.arm, step.kind, step.rep) + + +def needs_retry(record: Record) -> bool: + """Whether foreign load spoiled a completed run. A calibration step is not repeated: the server keeps every window + it measured, and the builder of shipped data takes the largest.""" + return ( + record.status == "completed" + and record.calibration is None + and any(reason.startswith((FOREIGN_GPU_LOAD, FOREIGN_QUEUE_ITEM)) for reason in record.invalid_reasons) + ) + + +def crashed_server(record: Record) -> bool: + return any(reason.startswith(SERVER_CRASHED) for reason in record.invalid_reasons) + + +def remaining_steps(plan: list[PlannedStep], records: list[Record]) -> list[PlannedStep]: + """Planned steps without a final record, in plan order. + + A step is final when its last record completed and needs no retry, or when its retries are used up. Every step of + a case key whose steps crashed the server MAX_SERVER_CRASHES times is final too, so a crashing case cannot keep a + run (or its resumes) restarting the server forever. + """ + attempts: dict[tuple[str, str, str, int], list[Record]] = {} + crashes: dict[str, int] = {} + for record in records: + attempts.setdefault(step_id(record), []).append(record) + crashes[record.case_key] = crashes.get(record.case_key, 0) + crashed_server(record) + remaining = [] + for step in plan: + own = sorted(attempts.get(step_id(step), []), key=lambda r: r.attempt) + last = own[-1] if own else None + done = last is not None and last.status == "completed" and (not needs_retry(last) or len(own) >= MAX_ATTEMPTS) + if not done and crashes.get(step.case_key, 0) < MAX_SERVER_CRASHES: + remaining.append(step) + return remaining diff --git a/scripts/family_e2e/report.py b/scripts/family_e2e/report.py index 1e59b20e5aa..1256e04ffd0 100644 --- a/scripts/family_e2e/report.py +++ b/scripts/family_e2e/report.py @@ -36,12 +36,14 @@ from scripts.family_e2e.records import ( Environment, ModelIdentity, + PlannedStep, Record, Reference, RunInfo, Thresholds, load_records, load_run_info, + remaining_steps, run_id_profile, write_bytes_atomic, ) @@ -66,9 +68,10 @@ ) # CHANGED: a case expected to change differs from its reference. BASELINE: its reference comes from this run, so -# there is nothing to compare against. Neither is a failure. -Status = Literal["PASS", "CHANGED", "BASELINE", "FAIL", "ERROR", "INVALID", "NO-REFERENCE"] -STATUSES: tuple[Status, ...] = ("PASS", "CHANGED", "BASELINE", "FAIL", "ERROR", "INVALID", "NO-REFERENCE") +# there is nothing to compare against. Neither is a failure. INCOMPLETE: a planned measure is not final. +Status = Literal["PASS", "CHANGED", "BASELINE", "FAIL", "ERROR", "INCOMPLETE", "INVALID", "NO-REFERENCE"] +STATUSES: tuple[Status, ...] = ("PASS", "CHANGED", "BASELINE", "FAIL", "ERROR", "INCOMPLETE", "INVALID", "NO-REFERENCE") +FAILURES: tuple[Status, ...] = ("FAIL", "ERROR", "INCOMPLETE") Unit = Literal["s", "bytes"] @@ -77,7 +80,7 @@ class RenderedReport: html: str summary: str failures: int - """Cases with status FAIL or ERROR.""" + """Cases with status FAIL, ERROR or INCOMPLETE.""" outliers: int """Cases with at least one outlier against the reference or the previous run.""" invalid: int @@ -159,7 +162,14 @@ def render_report( of date, e.g. for a run whose runner died. """ context = _Context( - data_dir, run_dir, info, _History(data_dir, info), _ImageCache(scrubber.text), diff_dir, scrubber + data_dir, + run_dir, + info, + _History(data_dir, info), + _ImageCache(scrubber.text), + diff_dir, + scrubber, + remaining_steps(info.plan, records), ) cases = [ _evaluate_safely(context, arm, case_key, case_records) @@ -173,7 +183,7 @@ def render_report( return RenderedReport( html=_render_html(context, cases, notes, href, run_status), summary=_render_summary(info, cases, notes, run_status), - failures=sum(case.status in ("FAIL", "ERROR") for case in cases), + failures=sum(case.status in FAILURES for case in cases), outliers=sum(case.has_outliers for case in cases), invalid=sum(case.status == "INVALID" or bool(case.invalid_metrics) for case in cases), changed=sum(case.status == "CHANGED" for case in cases), @@ -293,6 +303,8 @@ class _Context: images: _ImageCache diff_dir: Path scrubber: Scrubber + remaining: list[PlannedStep] + """The run's planned steps without a final record, as the runner counts them (`remaining_steps`).""" def _node_types(run_dir: Path, case_key: str) -> dict[str, str]: @@ -470,7 +482,10 @@ def _evaluate(context: _Context, case: _Case, records: list[Record]) -> None: row.psnr_previous = _psnr_or_problem(image, previous_image) case.rows.append(row) - _set_status(case, final, thresholds, reference_problem, info.run_id) + own = (case.arm, case.case_key, "measure") + planned = sum((s.arm, s.case_key, s.kind) == own for s in info.plan) + unfinished = [s.rep for s in context.remaining if (s.arm, s.case_key, s.kind) == own] + _set_status(case, final, unfinished, planned, thresholds, reference_problem, info.run_id) _compare_metrics(case, context.run_dir, final, previous, thresholds) _set_change_flags(context, case, previous) _add_images(case, context, reference_image, previous, previous_image) @@ -488,13 +503,21 @@ def _psnr_or_problem(image: np.ndarray | str, other: np.ndarray | str | None) -> def _set_status( - case: _Case, final: list[Record], thresholds: Thresholds | None, reference_problem: str | None, run_id: str + case: _Case, + final: list[Record], + unfinished: list[int], + planned: int, + thresholds: Thresholds | None, + reference_problem: str | None, + run_id: str, ) -> None: """Judges each step by its final attempt; a failure that a later attempt recovered from stays visible as a flag. - Errors, invalid runs and a missing reference come first. A reference set from this run gives BASELINE. Then the - minimum PSNR against the reference decides: below the threshold is FAIL, or CHANGED for a case expected to change, - which fails when it matches instead. + `unfinished` lists the reps of the `planned` measures the runner would still run: without a record, or awaiting a + retry. Errors (a final step that did not complete), unfinished measures, invalid runs and a missing reference come + first. A reference set from this run gives BASELINE. + Then the minimum PSNR against the reference decides: below the threshold is FAIL, or CHANGED for a case expected to + change, which fails when it matches instead. """ rated = [row for row in case.rows if row.rated] values = [row.psnr_reference for row in rated if isinstance(row.psnr_reference, float)] @@ -502,9 +525,14 @@ def _set_status( case.flags += [ f"earlier attempt {row.record.record_id} {row.record.status}: {row.record.error or 'no error message'}" for row in case.rows - if not row.final and row.record.status in ("failed", "error") + if not row.final and row.record.status != "completed" ] - errored = [r for r in final if r.status in ("failed", "error")] + errored = [r for r in final if r.status != "completed"] + absent = ( + [f"{len(unfinished)} of {planned} planned measures did not finish (rep {', '.join(map(str, unfinished))})"] + if unfinished + else [] + ) wrong_size = [ r for r in final @@ -516,7 +544,10 @@ def _set_status( if errored: case.status = "ERROR" - case.reasons += [f"{r.record_id}: {r.status}: {r.error or 'no error message'}" for r in errored] + case.reasons += [f"{r.record_id}: {r.status}: {r.error or 'no error message'}" for r in errored] + absent + elif unfinished: + case.status = "INCOMPLETE" + case.reasons += absent elif wrong_size: case.status = "FAIL" case.reasons += [ @@ -851,7 +882,7 @@ def _e(value: object) -> str: .badge { display: inline-block; padding: 1px 9px; border-radius: 999px; font-size: 12px; font-weight: 600; white-space: nowrap; } .s-PASS { background: var(--pass-bg); color: var(--pass-fg); } -.s-FAIL, .s-ERROR { background: var(--fail-bg); color: var(--fail-fg); } +.s-FAIL, .s-ERROR, .s-INCOMPLETE { background: var(--fail-bg); color: var(--fail-fg); } .s-INVALID { background: var(--invalid-bg); color: var(--invalid-fg); } .s-NO-REFERENCE { background: var(--noref-bg); color: var(--noref-fg); } .s-CHANGED { background: var(--changed-bg); color: var(--changed-fg); } @@ -915,7 +946,7 @@ def _render_html( f'
{count_badges or "no cases"}
', ] if run_status != "completed": - notes = [f"The run is {run_status}; cases it did not reach show as INVALID.", *notes] + notes = [f"The run is {run_status}; cases it did not finish show as INCOMPLETE.", *notes] if notes: parts.append('
    ' + "".join(f"
  • {_e(note)}
    • " for note in notes) + "
") parts.append(_environment_html(info)) diff --git a/scripts/family_e2e/runner.py b/scripts/family_e2e/runner.py index f0fa55bc23c..86c1958f319 100644 --- a/scripts/family_e2e/runner.py +++ b/scripts/family_e2e/runner.py @@ -15,6 +15,9 @@ and none is spawned after it. The runner checks again before every server and every step (`ensure_running`), and never overwrites a status other than `running`. +A lifetime's scratch root goes only once every process of its server has exited; a server whose kill fails or that +outlives it keeps the root and its heartbeat entries, and the run fails (`_Session.serve_once`). + A calibration run also checks that the server can calibrate, submits each step through the working-memory calibrate route, and saves the lifetime's calibration store before the scratch root goes (`calibration.py`). It records the store under the run lock and keeps whatever status run.json has (`save_calibration`), so a runner the stop has not @@ -72,6 +75,10 @@ resolve_cases, ) from scripts.family_e2e.records import ( + FOREIGN_GPU_LOAD, + FOREIGN_QUEUE_ITEM, + MAX_ATTEMPTS, + SERVER_CRASHED, CalibrationExport, Environment, ForeignProcess, @@ -85,6 +92,9 @@ append_record, load_records, load_run_info, + needs_retry, + remaining_steps, + step_id, write_bytes_atomic, write_run_info, ) @@ -93,6 +103,7 @@ from scripts.family_e2e.server import ( ServerError, ServerProcess, + TerminationError, build_scratch_root, free_port, git_state, @@ -119,14 +130,14 @@ # The heartbeat entries that identify the server's processes. The launcher (the process the runner started, which # leads the server's process group on POSIX) comes first: its tree and group take the serving process along. SERVER_ROLES = ("server_launcher", "server") +# Further processes of a server that outlived the runner's kill, e.g. a worker in its group, or a launcher whose +# recording failed: a list of identities in the heartbeat. +SERVER_SURVIVORS = "server_survivors" EXPORT_DIR = "export" """The graph exporter's work directory in the run directory.""" HEARTBEAT_INTERVAL_S = 30.0 # Machine-wide, not per data directory: two data directories can share a GPU. LOCK_DIR = Path(os.environ.get("LOCALAPPDATA") or Path.home() / ".cache") / "invokeai-family-e2e" / "locks" -MAX_ATTEMPTS = 3 -# A case whose steps crashed the server this often is given up; its remaining steps count as final. -MAX_SERVER_CRASHES = 2 # Windows Subsystem for Linux names its kernel so; NVML there runs on the Windows driver (WDDM). WSL_KERNEL_RELEASE = Path("/proc/sys/kernel/osrelease") QUEUE_DRAIN_TIMEOUT_S = 600.0 @@ -140,10 +151,6 @@ FOREIGN_GROWTH_BYTES = 256 * MIB SHARED_GROWTH_BYTES = 256 * MIB CANCEL_WAIT_S = 120.0 -# Reasons that make a run repeat (up to MAX_ATTEMPTS records per step); the generation itself was fine. -FOREIGN_GPU_LOAD = "foreign GPU load" -FOREIGN_QUEUE_ITEM = "foreign queue item" -SERVER_CRASHED = "server crashed" class RunnerError(RuntimeError): @@ -234,46 +241,6 @@ def file_sha256(path: Path) -> str: return hashlib.sha256(path.read_bytes()).hexdigest() -def _step_id(step: PlannedStep | Record) -> tuple[str, str, str, int]: - return (step.case_key, step.arm, step.kind, step.rep) - - -def needs_retry(record: Record) -> bool: - """Whether foreign load spoiled a completed run. A calibration step is not repeated: the server keeps every window - it measured, and the builder of shipped data takes the largest.""" - return ( - record.status == "completed" - and record.calibration is None - and any(reason.startswith((FOREIGN_GPU_LOAD, FOREIGN_QUEUE_ITEM)) for reason in record.invalid_reasons) - ) - - -def crashed_server(record: Record) -> bool: - return any(reason.startswith(SERVER_CRASHED) for reason in record.invalid_reasons) - - -def remaining_steps(plan: list[PlannedStep], records: list[Record]) -> list[PlannedStep]: - """Planned steps without a final record, in plan order. - - A step is final when its last record completed and needs no retry, or when its retries are used up. Every step of - a case key whose steps crashed the server MAX_SERVER_CRASHES times is final too, so a crashing case cannot keep a - run (or its resumes) restarting the server forever. - """ - attempts: dict[tuple[str, str, str, int], list[Record]] = {} - crashes: dict[str, int] = {} - for record in records: - attempts.setdefault(_step_id(record), []).append(record) - crashes[record.case_key] = crashes.get(record.case_key, 0) + crashed_server(record) - remaining = [] - for step in plan: - own = sorted(attempts.get(_step_id(step), []), key=lambda r: r.attempt) - last = own[-1] if own else None - done = last is not None and last.status == "completed" and (not needs_retry(last) or len(own) >= MAX_ATTEMPTS) - if not done and crashes.get(step.case_key, 0) < MAX_SERVER_CRASHES: - remaining.append(step) - return remaining - - @contextmanager def gpu_lock(adapter: gpu_memory.Adapter) -> Iterator[None]: """Holds `LOCK_DIR/.lock` so a second runner on the same GPU fails fast.""" @@ -344,11 +311,16 @@ def _identified(identity: list[float] | None) -> psutil.Process | None: def _running(process: psutil.Process) -> bool: - """Whether a process still runs: a zombie has exited and holds no memory, files or port any more.""" + """Whether a process still runs: a zombie has exited and holds no memory, files or port any more. + + A live pid whose state this process may not read counts as running: its exit has not been seen. + """ try: return process.is_running() and process.status() != psutil.STATUS_ZOMBIE - except psutil.Error: + except psutil.NoSuchProcess: # ZombieProcess too return False + except psutil.AccessDenied: + return True def is_alive(identity: list[float] | None) -> bool: @@ -537,6 +509,12 @@ def wait_for_runner(run_dir: Path, spawned_pid: int, previous: list[float] | Non time.sleep(REGISTER_POLL_S) +def recorded_server(heartbeat: dict[str, Any]) -> list[tuple[str, list[float] | None]]: + """The server processes a heartbeat names, as (role, identity): launcher, serving process, and survivors.""" + survivors = [(SERVER_SURVIVORS, identity) for identity in heartbeat.get(SERVER_SURVIVORS) or []] + return [(role, heartbeat.get(role)) for role in SERVER_ROLES] + survivors + + def stop_surviving_server(run_dir: Path) -> None: """Stops a server that an earlier runner of the run left running, before anything replaces what identifies it. @@ -545,14 +523,16 @@ def stop_surviving_server(run_dir: Path) -> None: while a recorded server process cannot be stopped this refuses, before the heartbeat or a scratch root changes. Call it under the GPU lock: a live runner of the run would hold it, so the recorded server is no live runner's. """ - heartbeat = read_heartbeat(run_dir) - survivors = { - int(heartbeat[role][0]): running for role in SERVER_ROLES if (running := still_running(heartbeat.get(role))) - } - for pid, running in survivors.items(): - pids = ", ".join(str(p) for p in running) - logger.warning("an earlier runner of this run left its server running (pid %d: %s); stopping it", pid, pids) - _, left = kill_recorded([(role, heartbeat.get(role)) for role in SERVER_ROLES]) + processes = recorded_server(read_heartbeat(run_dir)) + for _, identity in processes: + if identity is not None and (running := still_running(identity)): + pids = ", ".join(str(p) for p in running) + logger.warning( + "an earlier runner of this run left its server running (pid %d: %s); stopping it", + int(identity[0]), + pids, + ) + _, left = kill_recorded(processes) if left: raise RunnerError( f"the server an earlier runner left running cannot be stopped: {left}. heartbeat.json and the scratch " @@ -792,6 +772,10 @@ def __init__(self, run_dir: Path, manifest: Manifest, gpu: Gpu, export: Exporter # were saved through the export route. self.device_class_ids: set[str] = set() self.store_saved = False + # Further processes of this lifetime's server that outlived terminate's wait, and why a kill failed without + # naming any; both are cleared only once their exit is confirmed (`holds_server`). + self.server_survivors: list[list[float]] = [] + self.unidentified = "" all_cases = {case.case_key: case for case in resolve_cases(manifest)} self.cases = {step.case_key: all_cases[step.case_key] for step in self.info.plan} @@ -800,15 +784,51 @@ def heartbeat(self, step_label: str | None = None) -> None: self.step_label = step_label data = { "runner": process_identity(os.getpid()), - # The create time captured with the serving process, as for the launcher. - "server": [self.server.pid, self.server.serving.create_time()] if self.server else None, + "server": self.server_identity(), "server_launcher": self.launcher, + SERVER_SURVIVORS: self.server_survivors, "step": self.step_label, "updated_at": now_iso(), } write_bytes_atomic(self.run_dir / HEARTBEAT_FILE, json.dumps(data, indent=2).encode("utf-8")) self.last_beat = time.perf_counter() + def server_identity(self) -> list[float] | None: + # The create time captured with the serving process, as for the launcher. + return [self.server.pid, self.server.serving.create_time()] if self.server else None + + def stop_server(self) -> str: + """Kills this lifetime's server and returns what of it may still run; empty once every process of it is + confirmed gone: the launcher and serving process the heartbeat names, and each that outlived terminate's wait, + checked again, as `stop` checks them. Until then the session keeps their identities (`holds_server`).""" + if self.server is not None: + try: + self.server.stop() + except TerminationError as e: + self.add_survivors(e) + processes = [ + ("server_launcher", self.launcher), + ("server", self.server_identity()), + *((SERVER_SURVIVORS, identity) for identity in self.server_survivors), + ] + _, running = kill_recorded(processes) + self.server_survivors = [identity for identity in self.server_survivors if still_running(identity)] + survivors = "; ".join(s for s in (self.unidentified, running) if s) + if not survivors: + self.server, self.launcher = None, None + return survivors + + def add_survivors(self, error: TerminationError) -> None: + known = {int(identity[0]) for identity in self.server_survivors} + self.server_survivors += [identity for identity in error.survivors if int(identity[0]) not in known] + if not error.survivors: + self.unidentified = str(error) + + @property + def holds_server(self) -> bool: + """Whether a process of this lifetime's server may still run, i.e. its exit has not been confirmed.""" + return bool(self.server or self.launcher or self.server_survivors or self.unidentified) + def tick(self) -> None: if time.perf_counter() - self.last_beat >= HEARTBEAT_INTERVAL_S: self.heartbeat() @@ -849,11 +869,17 @@ def recover_stores(self) -> None: logger.warning("could not remove the scratch root %s: %s", root, ", ".join(leftovers)) def serve_once(self, todo: list[PlannedStep]) -> bool: - """One server lifetime over `todo`. Returns True when the server crashed in a step.""" + """One server lifetime over `todo`. Returns True when the server crashed in a step. + + The lifetime's calibration store is copied and its scratch root removed only once every process of its server + has exited. A server whose kill fails or that outlives it keeps both, and the heartbeat keeps naming it for + `stop` and `resume`, which stop it before they touch the root; the run fails (a stop stays a stop). + """ root = scratch_root(self.manifest.data_dir, self.info.run_id, self.arm) log_path = self.run_dir / f"server-{self.arm}.log" self.device_class_ids, self.store_saved = set(), False stopping = False + survivors = "its stop did not finish" try: pin = device_env(self.gpu.adapter, self.manifest.server.env) build_scratch_root(self.manifest.server.template_root, root, log_path, free_port(), bool(pin)) @@ -872,18 +898,31 @@ def serve_once(self, todo: list[PlannedStep]) -> bool: except ServerCrashed as e: logger.error("%s; starting a new server", e) return True + except TerminationError as e: + # A launch that failed could not kill its server. If recording its launcher failed, only this names it. + self.add_survivors(e) + raise finally: - # A launch that failed has stopped its server itself. - if self.server is not None: - self.server.stop() - self.server, self.launcher = None, None + # A launch that failed has stopped its server itself, unless it raised TerminationError. + survivors = self.stop_server() except (KeyboardInterrupt, RunStopped): stopping = True raise finally: - # stop() returns once the server tree has exited, so its database files are closed by now. - kept = False - if self.info.calibrate and not self.store_saved: + # Derived from what is cleared only on a confirmed exit: an interrupted stop keeps the root too. + kept = self.holds_server + if kept: + # Names every process that may still run, for `stop` and `resume`. + self.heartbeat() + message = ( + f"the server may still run after its kill ({survivors}); kept the scratch root {root}, and " + "heartbeat.json names its processes. End them, then `stop` or `resume`" + ) + if not stopping: + raise RunnerError(message) + logger.error(message) + elif self.info.calibrate and not self.store_saved: + # Every process of the server has exited, so its database files are closed by now. try: self.save_calibration(calibration.read_store_files(root)) except OSError as e: @@ -1142,7 +1181,7 @@ def execute_step( Raises ServerCrashed, after storing the step's error record, when the server died during the step. """ - attempts = [r for r in self.records if _step_id(r) == _step_id(step)] + attempts = [r for r in self.records if step_id(r) == step_id(step)] while True: ensure_running(self.run_dir) attempt = max((r.attempt for r in attempts), default=0) + 1 @@ -1197,7 +1236,7 @@ def base_record(self, step: PlannedStep, attempt: int, graph_hash: str, fp8: boo } def error_record(self, step: PlannedStep, error: str) -> Record: - attempt = max((r.attempt for r in self.records if _step_id(r) == _step_id(step)), default=0) + 1 + attempt = max((r.attempt for r in self.records if step_id(r) == step_id(step)), default=0) + 1 fields = self.base_record(step, attempt, graph_hash="", fp8=False) fields.update(error=error, invalid_reasons=[error]) return Record(**fields) @@ -1260,10 +1299,7 @@ def watch() -> None: status = item["status"] fields["status"] = status - # Only img2img starts inside the schedule, at a step count each family derives differently. - case = self.cases[step.case_key] - expected_steps = None if case.spec.mode == "img2img" else case.spec.steps - timing = build_timing(recorder.since(mark), item_id, t_enqueue, t_done, item, expected_steps) + timing = build_timing(recorder.since(mark), item_id, t_enqueue, t_done, item, graph.step_counts) residency, residency_reasons = self.tracker.observe(parse_log(server.read_log(log_offset)), graph.model_keys) foreign = foreign_processes(usage_before, usage_after, own={server.pid, server.process.pid}) trace_path = f"traces/{fields['record_id']}.csv.gz" diff --git a/scripts/family_e2e/server.py b/scripts/family_e2e/server.py index e0600eedd2d..5bbb6c1a394 100644 --- a/scripts/family_e2e/server.py +++ b/scripts/family_e2e/server.py @@ -76,6 +76,15 @@ class ServerError(RuntimeError): pass +class TerminationError(ServerError): + """A process tree that may still run after `terminate`: its kill failed, or a process of it outlived the wait.""" + + def __init__(self, message: str, survivors: list[list[float]]) -> None: + super().__init__(message) + self.survivors = survivors + """[pid, create time] of each process of the tree that still ran; empty when none could be identified.""" + + def read_template_settings(template_root: Path) -> dict[str, Any]: """The template's `invokeai.yaml` settings (empty when the root has none), refusing what the tool cannot copy.""" path = template_root / CONFIG_FILE @@ -445,10 +454,13 @@ def living_members(pgid: int) -> list[int]: living = [] for pid in psutil.pids(): try: - if os.getpgid(pid) == pgid and psutil.Process(pid).status() != psutil.STATUS_ZOMBIE: - living.append(pid) - except (OSError, psutil.Error): - continue # gone, or not visible to this process + if os.getpgid(pid) != pgid or psutil.Process(pid).status() == psutil.STATUS_ZOMBIE: + continue + except (OSError, psutil.NoSuchProcess): + continue # gone, or in no group this process can see + except psutil.AccessDenied: + pass # a member whose state this process may not read has not been seen to exit + living.append(pid) return living @@ -460,23 +472,58 @@ def terminate(process: subprocess.Popen[Any], job: KillOnCloseJob | None) -> Non its id while it has members or the leader is unreaped. That group is killed even when its leader has exited; the server's liveness checks leave an exited leader unreaped (`exit_status`), and it is reaped only after the kill. The tree is captured before the kill, from the root Popen still holds. + + Raises TerminationError when the kill fails, or when a process of the tree (or, on POSIX, of the group) still runs + STOP_TIMEOUT_S after it: such a tree may still write its scratch root. """ try: root = psutil.Process(process.pid) tree = [root, *root.children(recursive=True)] except psutil.Error: tree = [] - if job is not None: - job.close() - else: - kill_group(process.pid) + group = process.pid if job is None else None + try: + if job is not None: + job.close() + else: + kill_group(process.pid) + except OSError as e: + survivors = _identities([p for p in tree if p.is_running()], group) + raise TerminationError( + f"the kill of pid {process.pid} and its descendants failed ({e}); may still run: {_pids(survivors)}", + survivors, + ) from e + deadline = time.monotonic() + STOP_TIMEOUT_S _, alive = psutil.wait_procs(tree, timeout=STOP_TIMEOUT_S) - if alive: - logger.warning("processes still running after the kill: %s", ", ".join(str(p.pid) for p in alive)) + # Group members outside the tree (orphans) die asynchronously too: SIGKILL is not instant. + while group is not None and living_members(group) and time.monotonic() < deadline: + time.sleep(0.05) + survivors = _identities(alive, group) + if survivors: + raise TerminationError( + f"{_pids(survivors)} still running {STOP_TIMEOUT_S:g}s after the kill of pid {process.pid}", survivors + ) try: process.wait(timeout=STOP_TIMEOUT_S) - except subprocess.TimeoutExpired: - process.kill() + except subprocess.TimeoutExpired as e: + survivors = _identities([p for p in tree if p.pid == process.pid], None) + raise TerminationError(f"pid {process.pid} still running {STOP_TIMEOUT_S:g}s after its kill", survivors) from e + + +def _identities(processes: list[psutil.Process], pgid: int | None) -> list[list[float]]: + """[pid, create time] of `processes` and, given a group id, of the group's living members (POSIX).""" + found = {p.pid: p for p in processes} + for pid in living_members(pgid) if pgid is not None else []: + if pid not in found: + try: + found[pid] = psutil.Process(pid) + except psutil.NoSuchProcess: + continue + return [[pid, p.create_time()] for pid, p in sorted(found.items())] + + +def _pids(identities: list[list[float]]) -> str: + return f"pid {', '.join(str(int(i[0])) for i in identities)}" if identities else "no process identified" def remove_tree(path: Path) -> list[str]: @@ -530,8 +577,11 @@ def read_log(self, offset: int) -> str: return f.read().decode("utf-8", errors="replace") def stop(self) -> None: - terminate(self.process, self.job) - self.console.close() + """Kills the server tree; raises TerminationError while a process of it may still run.""" + try: + terminate(self.process, self.job) + finally: + self.console.close() def launch_server( @@ -550,7 +600,8 @@ def launch_server( killed while it starts, and only what the runner records then names it for `stop` and `resume`: the runner's registration holds the run lock and refuses to enter once the run is stopped (`_Session.server_registration`). The one wait inside it: when recording the pid fails, the server is killed and waited for before the registration - ends, since a `stop` after it would find that server in no heartbeat. + ends, since a `stop` after it would find that server in no heartbeat. A launch that fails kills its server; when + that kill fails too, TerminationError replaces the launch's error, which it carries as its context. """ env = server_env(spec, device_env) invokeai_file = verify_interpreter(spec, env) @@ -567,8 +618,9 @@ def launch_server( try: on_spawn(process.pid) except BaseException: - terminate(process, job) - process = None + # Killed once, here: the handler below must not wait for it again. + unrecorded, process = process, None + terminate(unrecorded, job) raise port = _wait_for_port(process, log_path, log_offset, console_path) base_url = f"http://127.0.0.1:{port}" @@ -576,9 +628,11 @@ def launch_server( serving = _serving_process(process.pid, port) except BaseException: # A recorded server is killed outside the registration: `stop` finds it, and the run lock is not kept waiting. - if process is not None: - terminate(process, job) - console.close() + try: + if process is not None: + terminate(process, job) + finally: + console.close() raise return ServerProcess(process, console, log_path, base_url, serving, cmd, invokeai_file, job) diff --git a/scripts/working_memory/build_shipped.py b/scripts/working_memory/build_shipped.py index cc7ebc8dea7..79f7757ad17 100644 --- a/scripts/working_memory/build_shipped.py +++ b/scripts/working_memory/build_shipped.py @@ -15,16 +15,21 @@ features its estimator's line does not admit (`fit.Line.fits`), such as one with a materialized score matrix. It ships as an out-of-fit point that bounds its estimate from below only. -A point joins the data file that declares its estimator's coefficients, keeping the larger of itself and a shipped -point of the same measurement (`point_identity`), and says which it kept; the carried values are then refitted with -`fit.py`. A stand-in stays one until fresh-process points at two sizes far enough apart measure its line: -`probe_demand` points can raise it, never lower it. A `legacy` entry is replaced by its fit where the points fit every line it carries and each line has a point -that holds the upper bound, a fresh-process one: probe windows alone never replace hand-set values. It keeps them -otherwise. - -Without `--write` the changes are printed only. An export or point of another estimator version than the registered -spec, or features the spec's terms cannot evaluate, is refused. So is a result any point's bounds -(`working_memory/bounds.py`, the data test's rule) reject, and a line raised on a point flagged as fragmentation unless +A point joins the data file that declares its estimator's coefficients, keeping the larger of itself and a shipped point +of the same measurement (`point_identity`), and says which it kept; the carried values are then refitted with `fit.py`. +A stand-in stays one until fresh-process points at two sizes far enough apart measure its line: `probe_demand` points +can raise it, never lower it. A line that already covers its points by the shipped margin stays as it is; one that does +not is raised by the least that keeps the upper bounds of the points priced on it (`fit.raised_pair`, +`bounds.line_bounds`). A `legacy` entry is replaced by its fit where the points fit every line it carries within those +bounds and each line has a point that holds the upper bound, a fresh-process one: probe windows alone never replace +hand-set values. It keeps them otherwise. + +Without `--write` the changes are printed only. Exit status 2 refuses input the build cannot use: an export or point of +another estimator version than the registered spec, features the spec's terms cannot evaluate, a platform +`platform.MEASURED` does not list, or fresh-process points too few for a line. Exit status 1 refuses data that conflicts +with the bounds, after printing the changes it would make, and writes nothing: points no line at or above the fit can +cover within those upper bounds, with the points that conflict named; a result any point's bounds +(`working_memory/bounds.py`, the data test's rule) reject; and a line raised on a point flagged as fragmentation unless `--accept-fragmentation` is given. """ @@ -48,7 +53,7 @@ CalibrationPoint, CalibrationStoreFile, ) -from invokeai.backend.working_memory.bounds import bound_violation, evaluate_point, point_id +from invokeai.backend.working_memory.bounds import bound_violation, evaluate_point, line_bounds, point_id from invokeai.backend.working_memory.coefficients import ( CoefficientEntry, MeasuredPoint, @@ -59,12 +64,15 @@ from invokeai.backend.working_memory.fit import ( EntryKey, FitPoint, + LineBound, + LineConflict, + LineKey, capping_spans, entry_fit, entry_key, fit_points, fitted_values, - shipped_pair, + raised_pair, ) from invokeai.backend.working_memory.fragmentation import Measurement, fragmented from invokeai.backend.working_memory.platform import ( @@ -96,6 +104,11 @@ class BuildError(ValueError): pass +class BoundConflict(BuildError): + """Points no line can cover within the upper bounds of the points priced on it (`fit.LineConflict`): the data + conflicts with the bounds, which `main` refuses as it refuses a result the bounds reject, not as unusable input.""" + + @dataclass(frozen=True) class _Window: """A window that becomes, or competes to become, a point.""" @@ -438,13 +451,17 @@ def merge_points( def legacy_fit( - entry: CoefficientEntry, groups: Mapping[tuple[EntryKey, str], Sequence[FitPoint]] -) -> tuple[dict[str, int], dict[str, str]]: - """The fit that would replace a `legacy` entry's hand-set values (`fit.entry_fit` on `groups`), and per line why - its points cannot: they do not fit it, as its fresh-process points do not span (`fit.capping_spans`), or none of - them holds the upper bound (`MeasuredPoint.holds_upper_bound`). A probe window raises a line but never caps one, - so probe windows alone never replace hand-set values.""" - values, problems = entry_fit(entry, groups) + entry: CoefficientEntry, + groups: Mapping[LineKey, Sequence[FitPoint]], + bounds: Mapping[LineKey, Sequence[LineBound]], +) -> tuple[dict[str, float], dict[str, str]]: + """The fit that would replace a `legacy` entry's hand-set values (`fit.entry_fit` on `groups` within `bounds`), and + per line why its points cannot: they do not fit it, as its fresh-process points do not span (`fit.capping_spans`) + or no raise keeps the bounds (`fit.LineConflict`), or none of them holds the upper bound + (`MeasuredPoint.holds_upper_bound`). A probe window raises a line but never caps one, so probe windows alone never + replace hand-set values.""" + values, problems, conflicts = entry_fit(entry, groups, bounds) + problems |= conflicts for suffix in problems: line = groups.get((entry_key(entry), suffix), []) if line and not any(point.caps for point in line): @@ -458,24 +475,30 @@ def refit(data: Mapping[str, WorkingMemoryData], specs: Mapping[str, EstimatorSp requires. Any other stays a stand-in, raised where the fit of its points lies above it: `probe_demand` points bound it from below only, so they never lower it. A `legacy` entry whose points fit every line it carries, each line with a point that holds the upper bound (`legacy_fit`), is fitted from then on too; one they do not keeps its - hand-set values.""" + hand-set values. + + Every fit and raise keeps the upper bounds the points set (`bounds.line_bounds`): a stand-in that covers its points + by the shipped margin is kept, and one that does not is raised by the least the bounds allow (`fit.raised_pair`). + Raises `BoundConflict` where no line keeps the bounds, `BuildError` where the points cannot fit a line.""" groups = fit_points(data.values(), specs) + ceilings = line_bounds(data.values(), specs) replaced: set[EntryKey] = set() def raised_stand_ins(entry: CoefficientEntry) -> dict[str, float]: values = entry.values or {} raised = {} for suffix in sorted({name[2:] for name in entry.stand_ins}): - line = groups.get((entry_key(entry), suffix)) - if not line: + key = (entry_key(entry), suffix) + if key not in groups: continue + floor = (values[f"k_{suffix}"], values[f"b_{suffix}"]) try: - k, b = shipped_pair([(point.size, point.bytes) for point in line]) - except ValueError: - # Too few sizes for a line; `bound_violations` refuses a stand-in below one of them. - continue - raised[f"k_{suffix}"] = max(values[f"k_{suffix}"], k) - raised[f"b_{suffix}"] = max(values[f"b_{suffix}"], b) + k, b = raised_pair(floor, groups[key], ceilings.get(key, ())) + except LineConflict as e: + raise BoundConflict( + f"the points cannot raise a stand-in: {entry.estimator} on {entry.platform}, {suffix}: {e}" + ) from e + raised[f"k_{suffix}"], raised[f"b_{suffix}"] = k, b return raised def staged(entry: CoefficientEntry) -> CoefficientEntry: @@ -485,7 +508,7 @@ def staged(entry: CoefficientEntry) -> CoefficientEntry: entry = entry.model_copy(update={"stand_ins": stand_ins}) if not entry.legacy: return entry - values, problems = legacy_fit(entry, groups) + values, problems = legacy_fit(entry, groups, ceilings) if not values or problems: return entry replaced.add(entry_key(entry)) @@ -496,7 +519,9 @@ def staged(entry: CoefficientEntry) -> CoefficientEntry: for name, file in data.items() } try: - fitted = fitted_values(staged_data.values(), specs) + fitted = fitted_values(staged_data.values(), specs, ceilings) + except LineConflict as e: + raise BoundConflict(f"the points cannot fit a line: {e}") from e except ValueError as e: raise BuildError(f"the points cannot fit a line: {e}") from e @@ -533,6 +558,7 @@ def legacy_notes( measures its platform, or the lines its points cannot replace (`legacy_fit`).""" measured = {point.estimator for point in new_points} groups = fit_points(data.values(), specs) + ceilings = line_bounds(data.values(), specs) notes = [] for entry in (entry for file in data.values() for entry in file.coefficients): if not entry.legacy or entry.estimator not in measured: @@ -540,7 +566,7 @@ def legacy_notes( if not any(key == entry_key(entry) for key, _ in groups): reasons = "no in-fit point measures this platform" else: - _, problems = legacy_fit(entry, groups) + _, problems = legacy_fit(entry, groups, ceilings) reasons = "; ".join(f"{suffix}: {problem}" for suffix, problem in problems.items()) notes.append(f"note: {entry.estimator} {entry.platform} stays legacy: {reasons}") return notes @@ -752,9 +778,13 @@ def main(argv: Sequence[str] | None = None) -> int: build = build_points(exports, specs, entries) fresh = read_point_files(args.points, specs, entries) new_points = [*build.points, *fresh.points] - after = refit(merge_points(before, new_points), specs) - violations = bound_violations(after, specs) - reviews = review_lines(before, after, new_points, specs) + merged = merge_points(before, new_points) + conflict: BoundConflict | None = None + try: + after = refit(merged, specs) + except BoundConflict as e: + # The points still show, with their windows, so the conflict can be read against them. + after, conflict = merged, e except BuildError as e: print(f"error: {e}", file=sys.stderr) return 2 @@ -779,6 +809,11 @@ def main(argv: Sequence[str] | None = None) -> int: f": reserved MiB {', '.join(map(_mib, spread.reserved))} while allocated MiB " f"{', '.join(map(_mib, spread.allocated))}; max/first {spread.first_to_max:.2f}" ) + if conflict is not None: + print(f"error: not written: the points conflict with the bounds: {conflict}", file=sys.stderr) + return 1 + violations = bound_violations(after, specs) + reviews = review_lines(before, after, new_points, specs) for change in reviews: (old_k, old_b), (new_k, new_b) = change.old, change.new flagged = change.fragmented(build.spreads) diff --git a/tests/backend/working_memory/test_fit.py b/tests/backend/working_memory/test_fit.py index 48c33e39910..301529e38a7 100644 --- a/tests/backend/working_memory/test_fit.py +++ b/tests/backend/working_memory/test_fit.py @@ -1,23 +1,29 @@ -"""The shipped fit: an upper envelope of the points, with a non-negative base, plus the shipped margin.""" +"""The shipped fit: an upper envelope of the points, with a non-negative base, plus the shipped margin, raised where +other points lie above it by the least the upper bounds allow.""" from fractions import Fraction import pytest -from invokeai.backend.working_memory.bounds import bound_violation, evaluate_point +from invokeai.backend.working_memory.bounds import bound_violation, evaluate_point, line_bounds, point_id from invokeai.backend.working_memory.coefficients import CoefficientEntry, MeasuredPoint, ValuesFrom, WorkingMemoryData from invokeai.backend.working_memory.estimate import WorkingMemoryTerm from invokeai.backend.working_memory.fit import ( SHIPPED_MARGIN, + FitPoint, Line, + LineBound, + LineConflict, convolution_line, + entry_key, fit_points, fitted_values, hand_set_values, + raised_pair, shipped_pair, upper_envelope, ) -from invokeai.backend.working_memory.spec import EstimatorSpec +from invokeai.backend.working_memory.spec import EstimatorSpec, import_estimator_modules from invokeai.backend.z_image import denoise_working_memory from invokeai.backend.z_image.denoise_working_memory import Z_IMAGE_DENOISE_SPEC @@ -152,7 +158,7 @@ def test_a_line_is_fitted_on_the_single_sample_untiled_points_of_every_member_th MeasuredPoint.model_validate(_point("lender", "decode", 2048, 90000).model_dump() | {"in_fit": False}), ], ) - fitted = fitted_values([data], SPECS)[("lender", 1, (("conv_backend", "cudnn"),))] + fitted = fitted_values([data], SPECS, {})[("lender", 1, (("conv_backend", "cudnn"),))] assert (fitted["k_decode_bf16"], fitted["b_decode_bf16"]) == shipped_pair( [(512 * 512, 1100 * MIB), (1024 * 1024, 4400 * MIB)] ) @@ -165,7 +171,22 @@ def test_a_line_without_enough_points_names_its_entry() -> None: points=[_point("lender", "decode", 512, 1100)], ) with pytest.raises(ValueError, match=r"lender on \{'conv_backend': 'cudnn'\}, decode_bf16"): - fitted_values([data], SPECS) + fitted_values([data], SPECS, {}) + + +def _bounded_by(points: list[MeasuredPoint]) -> dict: + """The lender's decode line held to 1.3 times each of `points`, untiled single-sample decodes.""" + return { + (("lender", 1, (("conv_backend", "cudnn"),)), "decode_bf16"): [ + LineBound( + Fraction(p.features["height"] * p.features["width"]), + Fraction(1), + Fraction(13, 10) * p.measured_bytes, + f"fresh {p.features['height']}x{p.features['width']}", + ) + for p in points + ] + } def _probe(px: int, mib: int) -> MeasuredPoint: @@ -186,20 +207,19 @@ def test_a_line_its_fresh_process_points_do_not_span_is_not_fitted(points) -> No schema_version=1, coefficients=[_entry("lender", {"k_decode_bf16": 0.0, "b_decode_bf16": 0.0})], points=points ) with pytest.raises(ValueError, match=r"decode_bf16: its fresh-process points measure .* keep it a stand-in"): - fitted_values([data], SPECS) + fitted_values([data], SPECS, {}) def test_a_probe_point_never_lowers_a_half_of_the_fresh_process_fit() -> None: - """A probe at 2048px above the hull of the fresh points at 512px and 1024px drops the 1024px vertex: rated on the - fresh points, the edge from 512px to the probe wins, steeper but with a lower base than the fresh fit's 1004 MiB. - The shipped line keeps the larger of each half.""" + """A probe at 2048px above the fresh fit of the points at 512px and 1024px: the edge from 512px to it would be + steeper, but with a lower base than the fresh fit's 1004 MiB. The raise keeps the base and lifts the slope.""" fresh = [_point("lender", "decode", 512, 1024), _point("lender", "decode", 1024, 1229)] data = WorkingMemoryData( schema_version=1, coefficients=[_entry("lender", {"k_decode_bf16": 0.0, "b_decode_bf16": 0.0})], points=[*fresh, _probe(2048, 2560)], ) - line = fitted_values([data], SPECS)[("lender", 1, (("conv_backend", "cudnn"),))] + line = fitted_values([data], SPECS, _bounded_by(fresh))[("lender", 1, (("conv_backend", "cudnn"),))] fresh_k, fresh_b = shipped_pair([(512 * 512, 1024 * MIB), (1024 * 1024, 1229 * MIB)]) assert fresh_b == 1004 * MIB @@ -213,7 +233,7 @@ def test_a_declared_stand_in_line_is_not_fitted() -> None: update={"stand_ins": ["k_decode_bf16", "b_decode_bf16"]} ) data = WorkingMemoryData(schema_version=1, coefficients=[entry], points=[]) - assert fitted_values([data], SPECS) == {} + assert fitted_values([data], SPECS, {}) == {} @pytest.mark.parametrize( @@ -241,18 +261,6 @@ def test_only_an_estimator_keyed_by_no_platform_trait_carries_a_value_off_a_line assert hand_set_values(entry, spec) == exempt -def test_points_that_cap_nothing_are_covered_but_do_not_choose_the_line() -> None: - """Rated on every point, the low reading at size 1 makes the edge through the origin win; rated on the capping - points, the flatter edge does, and it still covers the low reading.""" - capping = [(2, 15), (4, 20)] - points = [*capping, (1, 4)] - - assert upper_envelope(points) == (Fraction(15, 2), 0) - k, b = upper_envelope(points, capping) - assert (k, b) == (Fraction(5, 2), 10) - assert all(k * x + b >= y for x, y in points) - - def test_a_probe_point_raises_a_line_but_does_not_choose_it() -> None: """The fresh points at 512x1024 and 1024x1024 px make the flatter edge win: 500 B/px past the first, a 1000 MiB base. A probe reading of 400 MiB at 512x512 px lies below that line; rating it would make the edge through the @@ -262,7 +270,7 @@ def test_a_probe_point_raises_a_line_but_does_not_choose_it() -> None: def fitted(*probes: MeasuredPoint) -> tuple[int, int]: data = WorkingMemoryData(schema_version=1, coefficients=[_entry("lender", values)], points=[*fresh, *probes]) - line = fitted_values([data], SPECS)[("lender", 1, (("conv_backend", "cudnn"),))] + line = fitted_values([data], SPECS, _bounded_by(fresh))[("lender", 1, (("conv_backend", "cudnn"),))] return line["k_decode_bf16"], line["b_decode_bf16"] def probe(px: int, mib: int) -> MeasuredPoint: @@ -274,9 +282,9 @@ def probe(px: int, mib: int) -> MeasuredPoint: assert shipped[1] > 0, "the flatter edge, with a base" # Below the line, at a size the fresh points do not measure, it changes nothing. assert fitted(probe(512, 400)) == shipped - # Above the fresh points it raises the line to cover it. - raised = fitted(probe(1024, 5000)) - assert raised != shipped and raised[0] * 1024 * 1024 + raised[1] >= 5000 * MIB + # Above the line at 1536px it raises the line to cover it. + raised = fitted(probe(1536, 3600)) + assert raised != shipped and raised[0] * 1536 * 1536 + raised[1] >= 1.05 * 3600 * MIB DENOISE_PLATFORM = {"conv_backend": "cudnn", "allocator": "native"} @@ -332,7 +340,11 @@ def test_a_token_line_is_fitted_on_the_plain_forwards_and_bounds_every_forward() ] data = WorkingMemoryData(schema_version=1, coefficients=[DENOISE_ENTRY], points=forwards) - [fitted] = fitted_values([data], {Z_IMAGE_DENOISE_SPEC.id: Z_IMAGE_DENOISE_SPEC}).values() + [fitted] = fitted_values( + [data], + {Z_IMAGE_DENOISE_SPEC.id: Z_IMAGE_DENOISE_SPEC}, + line_bounds([data], {Z_IMAGE_DENOISE_SPEC.id: Z_IMAGE_DENOISE_SPEC}), + ).values() assert fitted == {"k_denoise_bf16": 236544, "b_denoise_bf16": 105 * MIB} entry = DENOISE_ENTRY.model_copy(update={"values": (DENOISE_ENTRY.values or {}) | fitted}) @@ -382,21 +394,286 @@ def test_a_token_line_its_fresh_forwards_do_not_span_is_not_fitted() -> None: data = WorkingMemoryData(schema_version=1, coefficients=[DENOISE_ENTRY], points=forwards) with pytest.raises(ValueError, match=r"denoise_bf16: its fresh-process points measure .* keep it a stand-in"): - fitted_values([data], {Z_IMAGE_DENOISE_SPEC.id: Z_IMAGE_DENOISE_SPEC}) + fitted_values( + [data], + {Z_IMAGE_DENOISE_SPEC.id: Z_IMAGE_DENOISE_SPEC}, + line_bounds([data], {Z_IMAGE_DENOISE_SPEC.id: Z_IMAGE_DENOISE_SPEC}), + ) def test_a_probe_forward_never_lowers_a_half_of_the_fresh_token_fit() -> None: - """A warm window at 16384 image tokens 2 GiB above the fresh forwards' line drops their 4096-token vertex from the - hull: rated on the fresh forwards, the edge through the origin then wins, steeper and with no base. The shipped - line keeps the fresh fit's 105 MiB base and the steeper slope, so it still covers the window.""" + """A warm window at 16384 image tokens, 512 MiB above the fresh forwards' line (4242 MiB), takes no part in the + fresh fit, 236544 bytes per token and a 105 MiB base, which reserves 3916.5 MiB at its 16896 tokens, short of the + 4454.1 MiB it needs with 5%. The raise starts from that fit and keeps both halves at least there. Averaged over the + fresh sizes, whose middle is 3072 tokens, a slope reaching a size past it costs less than a base: so the base stays + 105 MiB and the slope rises to 269908 bytes per token, which keeps the fresh forwards' upper bounds in tokens.""" fresh = [_measured_forward(_forward(image)) for image in (1024, 4096)] - window = _probe_forward(_forward(16384), above_the_line=2 * GIB) + window = _probe_forward(_forward(16384), above_the_line=512 * MIB) data = WorkingMemoryData(schema_version=1, coefficients=[DENOISE_ENTRY], points=[*fresh, window]) - [fitted] = fitted_values([data], {Z_IMAGE_DENOISE_SPEC.id: Z_IMAGE_DENOISE_SPEC}).values() + [fitted] = fitted_values( + [data], + {Z_IMAGE_DENOISE_SPEC.id: Z_IMAGE_DENOISE_SPEC}, + line_bounds([data], {Z_IMAGE_DENOISE_SPEC.id: Z_IMAGE_DENOISE_SPEC}), + ).values() fresh_k, fresh_b = shipped_pair([(p.features["tokens"], p.measured_bytes) for p in fresh]) assert (fresh_k, fresh_b) == (236544, 105 * MIB) assert fitted["b_denoise_bf16"] == fresh_b - assert fitted["k_denoise_bf16"] > fresh_k + assert fitted["k_denoise_bf16"] == 269908 > fresh_k assert fitted["k_denoise_bf16"] * window.features["tokens"] + fitted["b_denoise_bf16"] >= window.measured_bytes + + +def test_a_token_window_no_line_covers_within_the_fresh_bounds_is_refused() -> None: + """2 GiB above the line at 16384 image tokens, the window (5778 MiB) needs 6066.9 MiB at 16896 attended tokens. + With the fresh fit's 105 MiB base that takes 369999 bytes per token, which puts the fresh 4096-token forward at + 1731 MiB, over 1.3x its 1090 MiB (1417 MiB): the bound, in tokens, conflicts with the window.""" + fresh = [_measured_forward(_forward(image)) for image in (1024, 4096)] + window = _probe_forward(_forward(16384), above_the_line=2 * GIB) + data = WorkingMemoryData(schema_version=1, coefficients=[DENOISE_ENTRY], points=[*fresh, window]) + specs = {Z_IMAGE_DENOISE_SPEC.id: Z_IMAGE_DENOISE_SPEC} + + with pytest.raises(LineConflict) as refused: + fitted_values([data], specs, line_bounds([data], specs)) + + message = str(refused.value) + assert "denoise_bf16: no line at or above k=236544, b=105 MiB covers 5778 MiB at size 16896 by 5%" in message + assert message.endswith( + "-denoise-bf16-4096+512tok-NVIDIAGeForceRTX4090-demand_peak (measured 1090 MiB, at most 1417 MiB)" + ) + + +LENDER_DECODE = (("lender", 1, (("conv_backend", "cudnn"),)), "decode_bf16") +FRESH = [_point("lender", "decode", 512, 1000), _point("lender", "decode", 1024, 2500)] +"""Fresh-process decodes whose fit is 2000 bytes per pixel and a 500 MiB base, shipped as 2100 and 525 MiB.""" + + +def _capped_at(ratio_512: float) -> dict: + """Upper bounds of `FRESH` on the line: `ratio_512` times the 512px decode, 1.3 times the 1024px one.""" + return { + LENDER_DECODE: [ + LineBound(Fraction(512 * 512), Fraction(1), Fraction(ratio_512) * 1000 * MIB, "fresh 512px"), + LineBound(Fraction(1024 * 1024), Fraction(1), Fraction(13, 10) * 2500 * MIB, "fresh 1024px"), + ] + } + + +def _with_probe(mib: int) -> WorkingMemoryData: + """`FRESH` and a probe decode of `mib` at 512x1024 px, between their sizes.""" + probe = MeasuredPoint.model_validate( + _point("lender", "decode", 512, mib, width=1024).model_dump() | {"method": "probe_demand"} + ) + values = {"k_decode_bf16": 0.0, "b_decode_bf16": 0.0} + return WorkingMemoryData(schema_version=1, coefficients=[_entry("lender", values)], points=[*FRESH, probe]) + + +def test_a_probe_no_hull_edge_covers_within_the_bounds_raises_the_line_between_them() -> None: + """The fresh fit reserves 1575 MiB at 512x1024 px, short of 1785 MiB, the 1700 MiB probe plus 5%. The hull through + the probe has two edges: 2800 bytes per pixel from 512px with a 300 MiB base (with the fresh base, 3465 MiB at + 1024px, over 1.3x its 2500 MiB) and 1600 from the probe with a 900 MiB base (1320 MiB at 512px, over 1.3x its + 1000 MiB). Between them, the slope floor with a 735 MiB base covers the probe exactly and keeps both bounds; of + the lines that do, it reserves least averaged over 512px to 1024px.""" + assert shipped_pair([(512 * 512, 1000 * MIB), (1024 * 1024, 2500 * MIB)]) == (2100, 525 * MIB) + + line = fitted_values([_with_probe(1700)], SPECS, _capped_at(1.3))[LENDER_DECODE[0]] + + assert (line["k_decode_bf16"], line["b_decode_bf16"]) == (2100, 735 * MIB) + + +def test_a_probe_no_line_covers_within_the_bounds_is_refused_naming_the_conflict() -> None: + """Held to 1.1x the 512px decode, 1100 MiB, a line at or above the fresh fit's 2100 bytes per pixel and 525 MiB + cannot reach 1785 MiB at 512x1024 px: the probe and the 512px bound conflict, the 1024px bound takes no part.""" + with pytest.raises(LineConflict) as refused: + fitted_values([_with_probe(1700)], SPECS, _capped_at(1.1)) + + message = str(refused.value) + assert "no line at or above k=2100, b=525 MiB covers 1700 MiB at size 524288 by 5%" in message + assert message.endswith("within the upper bound of fresh 512px") + + +def test_a_line_that_covers_every_point_by_the_margin_is_kept_whatever_their_own_fit() -> None: + """A stand-in of 4000 bytes per pixel and no base covers probes of 900 MiB at 512px and 3000 MiB at 1024px by more + than 5%. Their own fit has a base of its own; taking it on top would raise the line at every size.""" + probes = [FitPoint(512 * 512, 900 * MIB, False), FitPoint(1024 * 1024, 3000 * MIB, False)] + assert shipped_pair([(p.size, p.bytes) for p in probes])[1] > 0 + + assert raised_pair((4000, 0), probes, []) == (4000, 0) + + +def test_a_line_is_bounded_by_every_point_priced_on_it_where_it_decides() -> None: + """SD decodes on the cuDNN fp16 line SD3 lends: an upcast decode is bounded with its upcast's bytes taken off the + ceiling, a tiled one at the tile's 1.25 times its pixels, one whose score matrix decides at the shipped values as + if the line did, and an out-of-fit one only by its own `max_ratio`. A probe and an out-of-fit point without one set + no bound.""" + platform = {"conv_backend": "cudnn", "allocator": "native"} + names = [ + f"{part}_{op}_{dtype}" for part in "kb" for op in ("decode", "encode") for dtype in ("fp16", "bf16", "fp32") + ] + lender = CoefficientEntry( + estimator="vae.sd3", + version=2, + platform=platform, + values=dict.fromkeys(names, 1000.0), + provenance="-", + ) + borrower = CoefficientEntry( + estimator="vae.sd", + version=2, + platform=platform, + values_from=ValuesFrom(estimator="vae.sd3", version=2), + provenance="-", + ) + + def decode(px: int, mib: int, **changes) -> MeasuredPoint: + features = {"operation": "decode", "height": px, "width": px, "dtype": "fp16", "tile": None, + "vae_class": "AutoencoderKL", "upcast_weight_bytes": 0, "score_matrix_bytes": 0} # fmt: skip + features |= changes.pop("features", {}) + changes.setdefault("method", "demand_peak") + return MeasuredPoint( + estimator="vae.sd", + version=2, + platform=platform, + features=features, + measured_bytes=mib * MIB, + device="-", + provenance="-", + **changes, + ) + + upcast = decode(1024, 4000, features={"upcast_weight_bytes": 160 * MIB}) + tiled = decode(2048, 3000, features={"tile": 512}) + scores = decode(1024, 9000, features={"score_matrix_bytes": 8 * 2**30}) + out_of_fit = decode(512, 1200, in_fit=False, max_ratio=1.5) + unbounded = [decode(768, 2000, method="probe_demand"), decode(768, 2000, in_fit=False)] + data = WorkingMemoryData( + schema_version=1, + coefficients=[lender, borrower], + points=[upcast, tiled, scores, out_of_fit, *unbounded], + ) + + specs = import_estimator_modules() + bounds = line_bounds([data], specs) + + assert set(bounds) == {(entry_key(lender), "decode_fp16")} + by_point = {bound.point: bound[:3] for bound in bounds[(entry_key(lender), "decode_fp16")]} + # Named by id, measured bytes and the most the estimate may reserve: two points can share an id. + assert by_point == { + f"{point_id(upcast, specs['vae.sd'])} (measured 4000 MiB, at most 5200 MiB)": ( + Fraction(1024**2), + Fraction(1), + Fraction(1.3) * 4000 * MIB - 160 * MIB, + ), + f"{point_id(tiled, specs['vae.sd'])} (measured 3000 MiB, at most 3900 MiB)": ( + Fraction(512**2 * 5, 4), + Fraction(1), + Fraction(1.3) * 3000 * MIB, + ), + f"{point_id(scores, specs['vae.sd'])} (measured 9000 MiB, at most 11700 MiB)": ( + Fraction(1024**2), + Fraction(1), + Fraction(1.3) * 9000 * MIB, + ), + f"{point_id(out_of_fit, specs['vae.sd'])} (measured 1200 MiB, at most 1800 MiB)": ( + Fraction(512**2), + Fraction(1), + Fraction(1.5) * 1200 * MIB, + ), + } + + +def test_a_probe_the_raised_line_covers_leaves_it_where_it_is() -> None: + """A 2200 MiB probe at 512x1536 px lies above the fresh fit's 2100 MiB there. Past the middle of the fresh sizes, + the slope is the cheaper half to raise: 2380 bytes per pixel. A 4000 MiB probe at 2048x1024 px, which that line + covers by 5%, leaves it there, though measured from its size, the base would now be the cheaper half.""" + values = {"k_decode_bf16": 0.0, "b_decode_bf16": 0.0} + + def probe(height: int, width: int, mib: int) -> MeasuredPoint: + point = _point("lender", "decode", height, mib, width=width).model_dump() | {"method": "probe_demand"} + return MeasuredPoint.model_validate(point) + + def raised(*probes: MeasuredPoint) -> tuple[float, float]: + data = WorkingMemoryData(schema_version=1, coefficients=[_entry("lender", values)], points=[*FRESH, *probes]) + line = fitted_values([data], SPECS, _capped_at(1.3))[LENDER_DECODE[0]] + return line["k_decode_bf16"], line["b_decode_bf16"] + + first = probe(512, 1536, 2200) + assert raised(first) == (2380, 525 * MIB) + assert raised(first, probe(2048, 1024, 4000)) == (2380, 525 * MIB) + + +def test_a_raise_every_line_on_the_probe_prices_alike_goes_to_the_steeper_one() -> None: + """One probe size is the whole measured range, so every line through the probe plus 5% reserves the same there: + the slope rises, 4200 bytes per pixel with no base rather than 1000 with 800 MiB, the more conservative past it.""" + assert raised_pair((1000, 0), [FitPoint(512 * 512, 1000 * MIB, False)], []) == (4200, 0) + + +def test_a_floor_that_already_breaks_a_bound_is_named_as_such() -> None: + """A stand-in of 5000 bytes per pixel reserves 1250 MiB at 512px, over a bound of 1200 MiB there, and falls short + of a 6000 MiB probe at 1024px. No covering takes part in the conflict: the floor alone breaks the bound.""" + probe = [FitPoint(1024 * 1024, 6000 * MIB, False)] + bound = LineBound(Fraction(512 * 512), Fraction(1), Fraction(1200 * MIB), "fresh 512px (measured 923 MiB)") + + with pytest.raises( + LineConflict, + match=r"^the line's floor k=5000, b=0 MiB already exceeds the upper bound of fresh " + r"512px \(measured 923 MiB\)$", + ): + raised_pair((5000, 0), probe, [bound]) + + +def test_a_raise_without_the_bounds_of_its_fresh_process_points_is_refused() -> None: + """Fresh-process points always bound their line (`bounds.line_bounds`); a raise given none of them would be free + to break all of them.""" + with pytest.raises(ValueError, match="no upper bounds are given"): + fitted_values([_with_probe(1700)], SPECS, {}) + + +def test_a_raise_onto_a_bound_exactly_is_kept() -> None: + """Fresh decodes of 250 MiB at 512px and 1000 MiB at 1024px fit 1000 bytes per pixel and no base, shipped as 1050. + A 1238 MiB probe at 1024px needs 1299.9 MiB there, and both fresh decodes hold the line to 1.3x: 325 and 1300 MiB. + 1300 bytes per pixel with no base reserves exactly that at both sizes, so it ships, though no line keeps a whole + byte per pixel and a MiB of room below them.""" + fresh = [_point("lender", "decode", 512, 250), _point("lender", "decode", 1024, 1000)] + data = WorkingMemoryData( + schema_version=1, + coefficients=[_entry("lender", {"k_decode_bf16": 0.0, "b_decode_bf16": 0.0})], + points=[*fresh, _probe(1024, 1238)], + ) + + line = fitted_values([data], SPECS, _bounded_by(fresh))[("lender", 1, (("conv_backend", "cudnn"),))] + + assert (line["k_decode_bf16"], line["b_decode_bf16"]) == (1300, 0) + + +ODD_SIZE = 1024 * 1536 +"""1.5 MiB of pixels: at this size whole bytes per pixel step the reservation by 1.5 MiB, whole MiB of base by 1.""" + + +def test_a_raise_whose_real_optimum_rounds_onto_a_bound_takes_the_rounded_line_next_to_it() -> None: + """A probe at 1024x1536 px needs 6291980289 bytes with 5%, 4000 slope steps of 1.5 MiB and 0.5 MiB more. Averaged + over the fresh sizes, slope is the cheaper half, so the lowest real line is 4000.33 bytes per pixel and no base; + rounded up, 4001 reserves 1 MiB above the need, over the bound 0.6 MiB above it. 4000 bytes per pixel with a 1 MiB + base reserves 0.5 MiB above the need, within the bound: the lowest rounded line.""" + fresh = [FitPoint(512 * 512, 100 * MIB, True), FitPoint(ODD_SIZE, 100 * MIB, True)] + need = 1572864 * 4000 + 524289 + probe = FitPoint(ODD_SIZE, need * 20 // 21, False) + assert Fraction(21, 20) * probe.bytes == need + bound = LineBound(Fraction(ODD_SIZE), Fraction(1), Fraction(need + 629146), "fresh 1024x1536") + + assert raised_pair((2000, 0), [*fresh, probe], [bound]) == (4000, MIB) + + +def test_a_raise_only_a_line_between_whole_mib_could_make_is_refused() -> None: + """At 1024px a whole byte per pixel is a whole MiB, so every rounded line reserves a whole number of MiB there. A + probe that needs 1300.2 MiB under a bound of 1300.7 MiB leaves a slit of half a MiB: real lines fit it (1300.2 bytes + per pixel, say), rounded lines do not.""" + fresh = [FitPoint(512 * 512, 100 * MIB, True), FitPoint(1024 * 1024, 100 * MIB, True)] + probe = FitPoint(1024 * 1024, 20 * 64921834, False) + need, ceiling = Fraction(21, 20) * probe.bytes, Fraction(13007, 10) * MIB + assert 1300 * MIB < need < ceiling < 1301 * MIB + bound = LineBound(Fraction(1024 * 1024), Fraction(1), ceiling, "fresh 1024px") + + with pytest.raises( + LineConflict, match=r"covers 1238 MiB at size 1048576 by 5% within the upper bound of fresh 1024px$" + ): + raised_pair((1000, 0), [*fresh, probe], [bound]) diff --git a/tests/backend/working_memory/test_shipped_coefficients.py b/tests/backend/working_memory/test_shipped_coefficients.py index d6fdb5eefed..4abef535379 100644 --- a/tests/backend/working_memory/test_shipped_coefficients.py +++ b/tests/backend/working_memory/test_shipped_coefficients.py @@ -30,6 +30,7 @@ UPPER_BOUND_EXCEPTIONS, bound_violation, evaluate_point, + line_bounds, point_id, ) from invokeai.backend.working_memory.coefficients import ( @@ -192,7 +193,7 @@ def test_every_fitted_value_is_the_shipped_fit_of_its_points() -> None: """The values are reproducible from the points rather than hand-tuned. A new point that moves a fit fails here, naming the values to ship. Only `hand_set_values` are exempt: values off a line their entry carries, of an estimator keyed by no platform trait, which its points decide and bound but nothing fits.""" - fitted = fitted_values(shipped_data().values(), SPECS) + fitted = fitted_values(shipped_data().values(), SPECS, line_bounds(shipped_data().values(), SPECS)) stale = {} for entry in ENTRIES: if entry.values is None or entry.legacy: diff --git a/tests/family_e2e/test_bundle.py b/tests/family_e2e/test_bundle.py index c61bcda5f58..42facca833e 100644 --- a/tests/family_e2e/test_bundle.py +++ b/tests/family_e2e/test_bundle.py @@ -460,7 +460,7 @@ def test_a_run_recorded_before_model_identities_existed_bundles_once_its_runner_ assert (bundled.status, bundled.case_models) == (status, {}) assert "## Models" not in summary assert f"run {shown}" in report - assert f"The run is {shown}; cases it did not reach show as INVALID." in report + assert f"The run is {shown}; cases it did not finish show as INCOMPLETE." in report @pytest.mark.parametrize("tool_commit", ["0123456789abcdef", None]) diff --git a/tests/family_e2e/test_calibration.py b/tests/family_e2e/test_calibration.py index fd75be69582..0559c3c164b 100644 --- a/tests/family_e2e/test_calibration.py +++ b/tests/family_e2e/test_calibration.py @@ -1,14 +1,20 @@ """Calibration runs against the fake server: the calibrate route instead of enqueue_batch, refusals per case or for the run, and the store of every server lifetime saved into the run.""" +import errno import json +import os +import sys from pathlib import Path +import psutil import pytest -from scripts.family_e2e import calibration, runner +import scripts.family_e2e.__main__ as cli +from scripts.family_e2e import calibration, runner, server +from scripts.family_e2e.__main__ import main from scripts.family_e2e.calibration import PROBE_REASON, CalibrationError -from scripts.family_e2e.records import Record, load_records, load_run_info, write_run_info +from scripts.family_e2e.records import Record, load_records, load_run_info, remaining_steps, write_run_info from tests.family_e2e.test_flow import FOREIGN_PID, MODEL, FakeGpu, isolated_runner, setup # noqa: F401 - fixtures DEVICE_CLASS_ID = "0123456789abcdef" @@ -92,7 +98,7 @@ def test_a_calibration_run_measures_every_step_once_and_saves_every_lifetimes_st resumed = load_records(run_dir) assert [(r.kind, r.rep, r.attempt) for r in resumed[4:]] == [("warmup", 0, 3), ("measure", 1, 1)] - assert runner.remaining_steps(info.plan, resumed) == [] + assert remaining_steps(info.plan, resumed) == [] third = load_run_info(run_dir).calibration_exports[-1] assert (third.lifetime, third.source) == (3, "api") assert [p["queue_item_id"] for p in _store(run_dir, third.export)["points"]] == [ @@ -290,3 +296,93 @@ def test_a_server_that_cannot_calibrate_fails_the_run_before_any_step( assert load_records(run_dir) == [] info = load_run_info(run_dir) assert (info.status, info.calibration_exports) == ("failed", []) + + +@pytest.mark.parametrize( + ("failure", "ending"), + [ + ("survives the kill", "completes"), + ("kill refused", "completes"), + ("survives the kill", "stop"), + ("survives the kill", "interrupt"), + ("survives the kill", "second interrupt"), + ], +) +def test_a_server_that_may_outlive_its_kill_keeps_its_scratch_root_and_store_until_stop_ends_it( + failure: str, + ending: str, + setup: tuple[Path, Path], # noqa: F811 - fixture + isolated_runner: None, # noqa: F811 - fixture + monkeypatch: pytest.MonkeyPatch, + caplog: pytest.LogCaptureFixture, +) -> None: + # A server stuck in the GPU driver outlives the kill; another user's refuses it. Either may still write its store. + # The run ends after its steps; or `stop` marks it after the first step, whose kill misses the runner; or a Ctrl-C + # comes there, and with "second interrupt" another one during the wait for the server's exit. + manifest_path, data_dir = setup + gpu = FakeGpu() + run_dir = runner.new_run(manifest_path, [], [], calibrate=True) + store = runner._Session.store + wait_procs = psutil.wait_procs + + def kill(*args: object) -> None: + if failure == "kill refused": + raise PermissionError(errno.EPERM, "Operation not permitted") + + def kill_process(process: psutil.Process) -> None: + if failure == "kill refused": + raise psutil.AccessDenied(process.pid) + + def store_then_end(session: runner._Session, record: Record) -> None: + store(session, record) + if ending == "stop": + runner.finish_run(run_dir, "stopped") + elif ending in ("interrupt", "second interrupt"): + raise KeyboardInterrupt + + def interrupted_wait(*args: object, **kwargs: object) -> object: + raise KeyboardInterrupt + + raised = {"completes": runner.RunnerError, "stop": runner.RunStopped}.get(ending, KeyboardInterrupt) + try: + with monkeypatch.context() as m: + m.setattr(server, "STOP_TIMEOUT_S", 0.05) + m.setattr(runner, "KILL_WAIT_S", 0.05) + m.setattr(server.KillOnCloseJob, "close", kill) + m.setattr(psutil.Process, "kill", kill_process) + if sys.platform != "win32": + m.setattr(os, "killpg", kill) + m.setattr(runner._Session, "store", store_then_end) + if ending == "second interrupt": + m.setattr(psutil, "wait_procs", interrupted_wait) + with pytest.raises(raised): + runner.run(run_dir, gpu_factory=lambda _: gpu.gpu(), export=_export) + assert psutil.wait_procs is wait_procs + + info = load_run_info(run_dir) + # A stop or an interrupt stays a stop; the kept root is logged. + assert info.status == ("failed" if ending == "completes" else "stopped") + assert ending == "completes" or "may still run after its kill" in caplog.text + # What names the server for `stop` and `resume`. + heartbeat = runner.read_heartbeat(run_dir) + assert all(runner.is_alive(heartbeat[role]) for role in runner.SERVER_ROLES) + # The store the export saved while the server ran, if it got that far; the root that holds the store stays, + # and nothing is copied from it. + exported = ["api"] if ending == "completes" else [] + assert [e.source for e in info.calibration_exports] == exported + root = runner.scratch_root(data_dir, info.run_id, runner.DEFAULT_ARM) + store_file = root / "databases" / "working_memory" / f"{DEVICE_CLASS_ID}.json" + assert store_file.is_file() + + # A later `stop` kills the server, then saves the store and removes the root. This process is the runner. + kill_recorded = cli.kill_recorded + monkeypatch.setattr( + cli, "kill_recorded", lambda processes: kill_recorded([p for p in processes if p[0] != "runner"]) + ) + assert main(["stop", str(run_dir)]) == 0 + assert not any(runner.is_alive(heartbeat[role]) for role in runner.SERVER_ROLES) + assert not root.exists() + assert [e.source for e in load_run_info(run_dir).calibration_exports] == [*exported, "store_file"] + finally: + for role in runner.SERVER_ROLES: + runner.kill_identified(runner.read_heartbeat(run_dir).get(role)) diff --git a/tests/family_e2e/test_client.py b/tests/family_e2e/test_client.py index c22a1b97916..4642c05cc80 100644 --- a/tests/family_e2e/test_client.py +++ b/tests/family_e2e/test_client.py @@ -1,6 +1,7 @@ import pytest from scripts.family_e2e.client import Event, build_timing, per_step_seconds +from scripts.family_e2e.graphs import prepare_batch T0 = 1000.0 @@ -70,3 +71,45 @@ def test_timing_is_relative_to_enqueue_and_ignores_other_items() -> None: ] assert timing.per_step_s == pytest.approx(0.25) assert (timing.queue_started_at, timing.queue_completed_at) == (item["started_at"], item["completed_at"]) + + +def test_the_time_per_step_takes_the_count_of_the_node_that_sent_the_frames() -> None: + # Two nodes state a count over their whole schedule. The denoise node sends most frames: six steps of 150 ms, + # too fast for an inferred count to be trusted. A pass of twelve steps sent two; its larger count would halve the + # time. + events = [ + _invocation("invocation_started", T0, "denoise"), + *( + _invocation("invocation_progress", T0 + 0.15 * k, "denoise", percentage=k / 6, has_image=True) + for k in range(1, 7) + ), + _invocation("invocation_complete", T0 + 1.0, "denoise"), + _invocation("invocation_started", T0 + 1.0, "pass"), + *( + _invocation("invocation_progress", T0 + 1.0 + 0.5 * k, "pass", percentage=k / 12, has_image=True) + for k in (6, 12) + ), + _invocation("invocation_complete", T0 + 7.0, "pass"), + ] + timing = build_timing(events, 7, T0, T0 + 7.1, {}, {"denoise": 6, "pass": 12}) + assert timing.per_step_s == pytest.approx(0.15) + + +def test_the_time_per_step_counts_the_steps_the_compiled_denoise_node_runs() -> None: + # A case with `steps: 24` on Ideogram 4's V4_TURBO_12 preset runs twelve steps, here of 80 ms each. The 0.1 s + # throttle let every other frame through: k/12 also reads as step 2k of 24, and inferred as step k/2 of 6. + nodes = { + "denoise_latents": {"id": "denoise_latents", "type": "ideogram4_denoise", "sampler_preset": "V4_TURBO_12"}, + "output": {"id": "output", "type": "ideogram4_l2i", "is_intermediate": False}, + } + prepared = prepare_batch({"graph": {"id": "g", "nodes": nodes, "edges": []}, "data": []}, {}, {}) + events = [ + _invocation("invocation_started", T0, "denoise_latents"), + *( + _invocation("invocation_progress", T0 + 0.08 * k, "denoise_latents", percentage=k / 12, has_image=True) + for k in range(2, 13, 2) + ), + _invocation("invocation_complete", T0 + 1.0, "denoise_latents"), + ] + timing = build_timing(events, 7, T0, T0 + 1.1, {}, prepared.step_counts) + assert timing.per_step_s == pytest.approx(0.08) diff --git a/tests/family_e2e/test_flow.py b/tests/family_e2e/test_flow.py index e07d3128b21..53b17d51d86 100644 --- a/tests/family_e2e/test_flow.py +++ b/tests/family_e2e/test_flow.py @@ -23,6 +23,7 @@ Record, load_records, load_run_info, + remaining_steps, write_run_info, ) @@ -272,7 +273,29 @@ def export(request: dict, work_dir: Path) -> dict: assert [(r.kind, r.rep, r.attempt) for r in resumed] == [("warmup", 0, 3), ("measure", 1, 1)] assert all(r.valid_timing and r.valid_memory for r in resumed) assert resumed[0].graph_hash == records[0].graph_hash - assert runner.remaining_steps(info.plan, load_records(run_dir)) == [] + assert remaining_steps(info.plan, load_records(run_dir)) == [] + + +def test_the_time_per_step_counts_the_steps_of_the_compiled_graph_not_the_manifests( + setup: tuple[Path, Path], isolated_runner: None +) -> None: + # The manifest asks for 4 steps; the compiled denoise node runs 8, as an Ideogram 4 node runs its preset's. Frames + # at k/8 fit no count of 4: trusted, the 8 give the time per step; inferred, a fake step far below two throttle + # intervals gives none. + manifest_path, _ = setup + gpu = FakeGpu() + + def export(request: dict, work_dir: Path) -> dict: + graph = _graph() + graph["nodes"]["denoise"]["steps"] = 8 + batch = {"graph": graph, "data": [], "origin": "test", "destination": "test"} + return {"version": 1, "cases": [{"id": request["cases"][0]["id"], "batch": batch, "components": {}}]} + + run_dir = runner.new_run(manifest_path, [], []) + assert runner.run(run_dir, gpu_factory=lambda _: gpu.gpu(), export=export) == 0 + + measures = [r for r in load_records(run_dir) if r.kind == "measure"] + assert measures and all(r.timing.per_step_s is not None for r in measures) def test_a_run_stopped_between_steps_ends_before_the_next_step_and_stays_stopped( diff --git a/tests/family_e2e/test_graphs.py b/tests/family_e2e/test_graphs.py index 84f916d2058..c2e08c5238d 100644 --- a/tests/family_e2e/test_graphs.py +++ b/tests/family_e2e/test_graphs.py @@ -1,6 +1,12 @@ import pytest -from scripts.family_e2e.graphs import build_export_request, output_image_name, prepare_batch, shareable_source +from scripts.family_e2e.graphs import ( + IDEOGRAM4_PRESET_STEPS, + build_export_request, + output_image_name, + prepare_batch, + shareable_source, +) from scripts.family_e2e.manifest import CaseSpec, ResolvedCase from scripts.family_e2e.records import Thresholds @@ -160,3 +166,36 @@ def test_a_source_is_shareable_when_it_names_the_model_for_anyone( source: str, source_type: str, shareable: bool ) -> None: assert shareable_source({"source": source, "source_type": source_type}) == (source if shareable else None) + + +def test_each_denoise_node_states_the_step_count_it_runs_over_its_whole_schedule() -> None: + nodes = { + # Ideogram 4 runs its sampler preset's count unless it sets `steps`; the case's steps do not reach it. + "ideogram": {"id": "ideogram", "type": "ideogram4_denoise", "sampler_preset": "V4_TURBO_12"}, + "ideogram_steps": { + "id": "ideogram_steps", + "type": "ideogram4_denoise", + "sampler_preset": "V4_TURBO_12", + "steps": 30, + }, + "flux2": {"id": "flux2", "type": "flux2_denoise", "num_steps": 4, "denoising_start": 0, "denoising_end": 1}, + "z_image": {"id": "z_image", "type": "z_image_denoise", "steps": 8}, + # A window of the schedule runs a part of it that each family cuts differently. + "img2img": { + "id": "img2img", + "type": "flux2_denoise", + "num_steps": 4, + "denoising_start": 0.3, + "denoising_end": 1, + }, + "sdxl_base": {"id": "sdxl_base", "type": "denoise_latents", "steps": 30, "denoising_end": 0.8}, + "output": {"id": "output", "type": "l2i", "is_intermediate": False}, + } + prepared = prepare_batch({"graph": {"id": "g", "nodes": nodes, "edges": []}, "data": []}, {}, {}) + assert prepared.step_counts == {"ideogram": 12, "ideogram_steps": 30, "flux2": 4, "z_image": 8} + + +def test_the_ideogram_preset_step_counts_are_the_servers() -> None: + from invokeai.backend.ideogram4.sampler_configs import PRESETS + + assert IDEOGRAM4_PRESET_STEPS == {name: preset.num_steps for name, preset in PRESETS.items()} diff --git a/tests/family_e2e/test_report.py b/tests/family_e2e/test_report.py index b3f022b11e6..f02f3c78535 100644 --- a/tests/family_e2e/test_report.py +++ b/tests/family_e2e/test_report.py @@ -9,6 +9,7 @@ from scripts.family_e2e.__main__ import main from scripts.family_e2e.records import ( + FOREIGN_GPU_LOAD, Environment, ModelIdentity, Record, @@ -177,10 +178,57 @@ def test_failed_generation_is_an_error(tmp_path: Path) -> None: assert f"- ERROR {CASE}: {CASE}.default.measure.0.0: error: CUDA out of memory" in _attention(result) -def test_a_step_that_failed_and_then_succeeded_on_resume_is_judged_by_its_final_attempt(tmp_path: Path) -> None: +@pytest.mark.parametrize( + ("last_measure", "status", "reason"), + [ + ("missing", "INCOMPLETE", "1 of 3 planned measures did not finish (rep 2)"), + # Foreign load spoiled its only attempt: the runner repeats it, so it is no final measurement yet. + ("awaits its retry", "INCOMPLETE", "1 of 3 planned measures did not finish (rep 2)"), + ("canceled", "ERROR", f"{CASE}.default.measure.2.0: canceled: no error message"), + ], +) +def test_a_case_whose_planned_measures_did_not_all_complete_fails( + tmp_path: Path, capsys: pytest.CaptureFixture[str], last_measure: str, status: str, reason: str +) -> None: + # The measures that completed match the reference; the plan's third one did not run, awaits its retry, or was + # canceled. + accept(tmp_path, _write_run(tmp_path, FIRST, _case_records(FIRST)), None) + records = _case_records(SECOND)[:-1] + if last_measure == "canceled": + records.append(make_record(CASE, rep=2, run_id=SECOND, status="canceled", image=False)) + elif last_measure == "awaits its retry": + spoiled = {"valid_timing": False, "valid_memory": False, "invalid_reasons": [f"{FOREIGN_GPU_LOAD}: pid 4"]} + records.append(make_record(CASE, rep=2, run_id=SECOND, **spoiled)) + run_dir = _write_run(tmp_path, SECOND, records) + + assert main(["report", str(run_dir)]) == 1 + + assert "1 failures" in capsys.readouterr().out + assert f"- {status} {CASE}: {reason}" in (run_dir / "summary.md").read_text(encoding="utf-8").splitlines() + + +def test_a_planned_case_the_run_never_reached_is_incomplete(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None: + accept(tmp_path, _write_run(tmp_path, FIRST, _case_records(FIRST)), None) + run_dir = _write_run(tmp_path, SECOND, _case_records(SECOND), status="stopped") + # The plan also holds a case the stopped run never reached; it has no record at all. + info = make_run_info(SECOND, profile=PROFILE, case_keys=(CASE, OTHER_CASE), status="stopped") + write_run_info(run_dir, info) + + assert main(["report", str(run_dir)]) == 1 + + assert "1 failures" in capsys.readouterr().out + summary = (run_dir / "summary.md").read_text(encoding="utf-8").splitlines() + assert f"- INCOMPLETE {OTHER_CASE}: 3 of 3 planned measures did not finish (rep 0, 1, 2)" in summary + assert f"| {CASE} | PASS |" in "\n".join(summary) + + +@pytest.mark.parametrize(("status", "error"), [("error", "server died"), ("canceled", None)]) +def test_a_step_that_failed_and_then_succeeded_on_resume_is_judged_by_its_final_attempt( + tmp_path: Path, status: str, error: str | None +) -> None: accept(tmp_path, _write_run(tmp_path, FIRST, _case_records(FIRST)), None) records = _case_records(SECOND) - crashed = make_record(CASE, rep=0, attempt=0, run_id=SECOND, status="error", image=False, error="server died") + crashed = make_record(CASE, rep=0, attempt=0, run_id=SECOND, status=status, image=False, error=error) resumed = records[1].model_copy(update={"attempt": 1, "record_id": f"{CASE}.default.measure.0.1"}) resumed = resumed.model_copy(update={"image_path": f"images/{resumed.record_id}.png"}) run_dir = _write_run(tmp_path, SECOND, [records[0], crashed, resumed, *records[2:]]) @@ -189,7 +237,8 @@ def test_a_step_that_failed_and_then_succeeded_on_resume_is_judged_by_its_final_ assert result.failures == 0 assert _summary(result)[CASE][0] == "PASS" - assert f"- FLAG {CASE}: earlier attempt {CASE}.default.measure.0.0 error: server died" in _attention(result) + flag = f"- FLAG {CASE}: earlier attempt {CASE}.default.measure.0.0 {status}: {error or 'no error message'}" + assert flag in _attention(result) def test_case_without_valid_measurements_is_invalid_with_reasons(tmp_path: Path) -> None: @@ -213,7 +262,7 @@ def test_partial_residency_marks_time_and_vram_invalid_but_keeps_the_psnr_verdic ] shape = {"valid_timing": False, "valid_memory": False, "valid_psnr": True, "invalid_reasons": reasons} records = [make_record(CASE, "warmup", run_id=FIRST, **shape)] - records += [make_record(CASE, rep=rep, run_id=FIRST, **shape) for rep in range(2)] + records += [make_record(CASE, rep=rep, run_id=FIRST, **shape) for rep in range(3)] run_dir = _write_run(tmp_path, FIRST, records) result = write_report(tmp_path, run_dir) @@ -225,7 +274,7 @@ def test_partial_residency_marks_time_and_vram_invalid_but_keeps_the_psnr_verdic assert "1 NO-REFERENCE, 0 with outliers, 1 with invalid metrics." in summary attention = _attention(result) for reason in reasons: - assert f"- INVALID METRICS {CASE}: time and VRAM invalid: {reason} (2 of 2 measured)" in attention + assert f"- INVALID METRICS {CASE}: time and VRAM invalid: {reason} (3 of 3 measured)" in attention assert f'title="{reasons[0]}; {reasons[1]}">invalid' in result.report_html.read_text(encoding="utf-8") diff --git a/tests/family_e2e/test_runner.py b/tests/family_e2e/test_runner.py index cee8e13c289..a8081452ce8 100644 --- a/tests/family_e2e/test_runner.py +++ b/tests/family_e2e/test_runner.py @@ -22,7 +22,9 @@ from scripts.family_e2e.gpu_memory import Adapter, MemoryTrace, ProcessUsage, UnsupportedPlatformError from scripts.family_e2e.manifest import DEFAULT_ARM, Manifest from scripts.family_e2e.records import ( + FOREIGN_GPU_LOAD, RUN_INFO_FILE, + SERVER_CRASHED, NodeTiming, PlannedStep, Record, @@ -31,12 +33,11 @@ append_record, load_records, load_run_info, + remaining_steps, write_run_info, ) from scripts.family_e2e.runner import ( - FOREIGN_GPU_LOAD, HEARTBEAT_FILE, - SERVER_CRASHED, SERVER_ROLES, Gpu, RunnerError, @@ -46,7 +47,6 @@ gpu_lock, model_identities, preflight, - remaining_steps, uses_fp8, ) @@ -484,7 +484,7 @@ def killpg(pgid: int, sig: int) -> None: assert runner.is_alive(surviving_server) -@pytest.mark.parametrize("failure", ["survives the kill", "kill refused"]) +@pytest.mark.parametrize("failure", ["survives the kill", "kill refused", "state unreadable"]) def test_stop_keeps_the_run_while_a_recorded_process_survives_the_kill( killed_run: Path, surviving_server: list[float], @@ -506,11 +506,17 @@ def killpg(pgid: int, sig: int) -> None: if failure == "kill refused": raise PermissionError(errno.EPERM, "Operation not permitted") + def status(process: psutil.Process) -> str: + # Another user's process, say: alive, but its state is not to be read; its exit has not been seen. + raise psutil.AccessDenied(process.pid) + with monkeypatch.context() as m: m.setattr(runner, "KILL_WAIT_S", 0.05) m.setattr(psutil.Process, "kill", kill_process) if sys.platform != "win32": m.setattr(os, "killpg", killpg) + if failure == "state unreadable": + m.setattr(psutil.Process, "status", status) capsys.readouterr() assert main(["stop", str(killed_run)]) == 2 @@ -548,6 +554,36 @@ def test_the_heartbeat_names_the_serving_process_as_captured_at_its_start(killed assert runner.read_heartbeat(killed_run)["server"] == captured +def test_a_server_counts_as_stopped_only_once_its_serving_process_is_gone( + killed_run: Path, tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + # On Windows a venv launcher runs the serving interpreter as its child. Once the launcher has exited, the kill of + # its tree no longer reaches the serving process, which may then outlive the stop. + launcher, launcher_job = server.spawn_in_job([sys.executable, "-c", "pass"]) + launcher.wait() + serving, serving_job = server.spawn_in_job([sys.executable, "-c", "import time; time.sleep(60)"]) + session = runner._Session(killed_run, runner.load_run_manifest(load_run_info(killed_run)), _fake_gpu("Fake"), None) + try: + with open(tmp_path / "console.log", "wb") as console, monkeypatch.context() as m: + session.server = server.ServerProcess( + launcher, console, tmp_path / "server.log", "", psutil.Process(serving.pid), [], "", launcher_job + ) + m.setattr(runner, "KILL_WAIT_S", 0.05) + m.setattr(psutil.Process, "kill", lambda process: None) + if sys.platform != "win32": + m.setattr(os, "killpg", lambda pgid, sig: None) + survivors = session.stop_server() + + assert f"pid {serving.pid} (" in survivors + # The heartbeat goes on naming it. + assert session.server_identity() == runner.process_identity(serving.pid) + finally: + serving.kill() + serving.wait() + if serving_job is not None: + serving_job.close() + + class _ServerStarting(Exception): pass @@ -657,6 +693,153 @@ def recording_fails(session: runner._Session, pid: int) -> None: assert alive_at_release == [False] +def test_a_server_whose_failed_kill_names_no_process_keeps_the_scratch_root( + killed_run: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + # Recording the launcher failed, its tree could not be read, and the kill failed: nothing identifies what may + # still run, so nothing can confirm its exit either. + _launchable_server(killed_run) + spawned: list[int] = [] + spawn_in_job = server.spawn_in_job + + def spawn(*args: object, **kwargs: object) -> tuple[subprocess.Popen[bytes], object]: + process, job = spawn_in_job(*args, **kwargs) + spawned.append(process.pid) + return process, job + + def recording_fails(session: runner._Session, pid: int) -> None: + raise _RecordingFailed + + def unreadable(process: psutil.Process, recursive: bool = False) -> list[psutil.Process]: + raise psutil.AccessDenied(process.pid) + + def kill_fails(*args: object) -> None: + raise OSError(errno.EIO, "Input/output error") + + identity: list[float] | None = None + try: + with monkeypatch.context() as m: + m.setattr(server, "spawn_in_job", spawn) + m.setattr(runner._Session, "server_spawned", recording_fails) + m.setattr(psutil.Process, "children", unreadable) + m.setattr(server.KillOnCloseJob, "close", kill_fails) + m.setattr(server, "kill_group", kill_fails) + m.setattr(server, "living_members", lambda pgid: []) + with pytest.raises(RunnerError, match=r"may still run after its kill .*no process identified"): + runner.run(killed_run, gpu_factory=_fake_gpu) + identity = runner.process_identity(spawned[0]) + + assert _scratch_file(killed_run).is_file() + assert load_run_info(killed_run).status == "failed" + finally: + runner.kill_identified(identity) + + +@pytest.mark.parametrize("second_kill", ["has no effect", "works"]) +def test_a_server_that_fails_to_start_keeps_the_scratch_root_while_it_outlives_its_kill( + killed_run: Path, monkeypatch: pytest.MonkeyPatch, second_kill: str +) -> None: + # The launch gave up on a server that never answered, and its kill did not end it within the wait. The runner's + # check of what still runs kills it again: a server that exits then counts as stopped, as it would for `stop`. + _launchable_server(killed_run) + + def wait_for_port(*args: object) -> int: + raise _ServerStarting + + try: + with monkeypatch.context() as m: + m.setattr(server, "STOP_TIMEOUT_S", 0.05) + m.setattr(runner, "KILL_WAIT_S", 0.05 if second_kill == "has no effect" else CHILD_TIMEOUT_S) + m.setattr(server, "_wait_for_port", wait_for_port) + m.setattr(server.KillOnCloseJob, "close", lambda job: None) + if second_kill == "has no effect": + m.setattr(psutil.Process, "kill", lambda process: None) + if sys.platform != "win32": + m.setattr(os, "killpg", lambda pgid, sig: None) + with pytest.raises(RunnerError if second_kill == "has no effect" else server.TerminationError) as raised: + runner.run(killed_run, gpu_factory=_fake_gpu) + + launcher = runner.read_heartbeat(killed_run)["server_launcher"] + if second_kill == "has no effect": + assert "may still run after its kill" in str(raised.value) + assert runner.is_alive(launcher) + assert _scratch_file(killed_run).is_file() + # Once the server is gone, nothing holds its console log: the launch closed its own handle. Windows + # deletes no open file. + runner.kill_identified(launcher) + (killed_run / "server-default.console.log").unlink() + else: + # The launch's own error stays the cause; the run fails on it, with nothing kept. + assert isinstance(raised.value.__context__, _ServerStarting) + assert not runner.is_alive(launcher) + assert not _scratch_file(killed_run).exists() + assert load_run_info(killed_run).status == "failed" + finally: + runner.kill_identified(runner.read_heartbeat(killed_run).get("server_launcher")) + + +def test_a_server_whose_launcher_can_neither_be_recorded_nor_killed_is_named_for_a_later_stop( + killed_run: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] +) -> None: + # Recording the launcher failed and its kill had no effect: the processes terminate saw still running are what + # names that server, in the error and, for `stop` and `resume`, in heartbeat.json. + _launchable_server(killed_run) + spawned: list[psutil.Process] = [] + spawn_in_job, terminate = server.spawn_in_job, server.terminate + terminations: list[int] = [] + + def spawn(*args: object, **kwargs: object) -> tuple[subprocess.Popen[bytes], object]: + process, job = spawn_in_job(*args, **kwargs) + spawned.append(psutil.Process(process.pid)) + return process, job + + def counted_terminate(process: subprocess.Popen[bytes], job: object) -> None: + terminations.append(process.pid) + terminate(process, job) + + def recording_fails(session: runner._Session, pid: int) -> None: + raise _RecordingFailed + + try: + with monkeypatch.context() as m: + m.setattr(server, "STOP_TIMEOUT_S", 0.05) + m.setattr(runner, "KILL_WAIT_S", 0.05) + m.setattr(server, "spawn_in_job", spawn) + m.setattr(server, "terminate", counted_terminate) + m.setattr(runner._Session, "server_spawned", recording_fails) + m.setattr(server.KillOnCloseJob, "close", lambda job: None) + m.setattr(psutil.Process, "kill", lambda process: None) + if sys.platform != "win32": + m.setattr(os, "killpg", lambda pgid, sig: None) + with pytest.raises(RunnerError, match="may still run after its kill") as raised: + runner.run(killed_run, gpu_factory=_fake_gpu) + + # Killed once. + assert terminations == [spawned[0].pid] + assert f"pid {spawned[0].pid} (" in str(raised.value) + assert isinstance(raised.value.__context__, server.TerminationError) + assert not _gone(spawned[0]) + assert _scratch_file(killed_run).is_file() + heartbeat = runner.read_heartbeat(killed_run) + assert heartbeat["server_launcher"] is None + assert spawned[0].pid in {int(identity[0]) for identity in heartbeat[runner.SERVER_SURVIVORS]} + + # A later `stop` finds it there, kills it, then removes the root. This process is the runner. + kill_recorded = cli.kill_recorded + monkeypatch.setattr( + cli, "kill_recorded", lambda processes: kill_recorded([p for p in processes if p[0] != "runner"]) + ) + assert main(["stop", str(killed_run)]) == 0 + assert all(_gone(process) for process in spawned) + assert not _scratch_file(killed_run).exists() + # The runner closed its handle of the console log too: Windows deletes no open file. + (killed_run / "server-default.console.log").unlink() + finally: + for process in spawned: + if not _gone(process): + process.kill() + + @pytest.mark.parametrize("arrives", ["before the spawn", "during the spawn"]) def test_a_stop_that_meets_the_server_spawn_kills_the_server_before_it_removes_the_scratch_root_or_none_starts( killed_run: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str], arrives: str diff --git a/tests/family_e2e/test_server.py b/tests/family_e2e/test_server.py index 4ddd6543309..39297c6c10c 100644 --- a/tests/family_e2e/test_server.py +++ b/tests/family_e2e/test_server.py @@ -309,6 +309,33 @@ def test_stop_kills_the_server_tree_and_never_a_process_by_its_recorded_pid(tmp_ unrelated.wait() +@pytest.mark.parametrize("failure", ["survives the kill", "kill refused"]) +def test_terminate_raises_while_a_process_of_the_tree_may_still_run( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, failure: str +) -> None: + # A process stuck in the GPU driver outlives the kill; it may still write the scratch root. + process, job, tree = _spawn_tree(tmp_path) + + def kill(*args: object) -> None: + if failure == "kill refused": + raise PermissionError(errno.EPERM, "Operation not permitted") + + try: + with monkeypatch.context() as m: + m.setattr(server, "STOP_TIMEOUT_S", 0.05) + m.setattr(server.KillOnCloseJob, "close", kill) + if sys.platform != "win32": + m.setattr(os, "killpg", kill) + with pytest.raises(server.TerminationError) as raised: + server.terminate(process, job) + if failure == "survives the kill": + assert all(str(p.pid) in str(raised.value) for p in tree) + finally: + for p in tree: + p.kill() + process.wait() + + def _gone(process: psutil.Process) -> bool: try: return process.status() == psutil.STATUS_ZOMBIE diff --git a/tests/test_build_shipped_working_memory.py b/tests/test_build_shipped_working_memory.py index 77810cc3d2d..ddc6175c389 100644 --- a/tests/test_build_shipped_working_memory.py +++ b/tests/test_build_shipped_working_memory.py @@ -4,6 +4,7 @@ import itertools import json import platform as stdlib_platform +import re import shutil from collections.abc import Callable, Iterator from importlib.util import module_from_spec, spec_from_file_location @@ -387,9 +388,9 @@ def test_written_data_holds_the_points_and_passes_the_shipped_data_checks( assert main([*map(str, exports), "--data-dir", str(data_dir)]) == 0 assert _files(data_dir) == before, "nothing written without --write" preview = capsys.readouterr().out - # The 4.7 GB decode lies above the cuDNN line all FLUX.2 points share with vae.sd3: it is refitted. - shipped_k = _carried(DATA_DIR, "vae.sd3", CUDNN_NATIVE)["k_decode_bf16"] - assert f"vae.sd3 {CUDNN_NATIVE} k_decode_bf16 {shipped_k:.0f} ->" in preview + # The 4.7 GB decode lies above the cuDNN line all FLUX.2 points share with vae.sd3: it is raised. + assert f"line vae.json: vae.sd3 {CUDNN_NATIVE} decode_bf16: " in preview + assert "; raised, 2 new points on it" in preview # A run's calibration/ directory stands for its exports. assert main([str(exports[0].parent), "--data-dir", str(data_dir), "--write"]) == 0 @@ -397,7 +398,9 @@ def test_written_data_holds_the_points_and_passes_the_shipped_data_checks( built = load_data(data_dir)["vae.json"] probe_points = [p for p in built.points if p.method == "probe_demand"] assert sorted(p.measured_bytes for p in probe_points) == [1_150_000_000, 4_700_000_000, 10_900_000_000] - assert _carried(data_dir, "vae.sd3", CUDNN_NATIVE)["k_decode_bf16"] > shipped_k + shipped, raised = (_carried(d, "vae.sd3", CUDNN_NATIVE) for d in (DATA_DIR, data_dir)) + at_1024 = [line["k_decode_bf16"] * 1024 * 1024 + line["b_decode_bf16"] for line in (shipped, raised)] + assert at_1024[0] < 1.05 * 4_700_000_000 <= at_1024[1] shipped_checks(data_dir) # Built again from the same exports, the data does not change. @@ -492,16 +495,18 @@ def _miopen_native_point( ) -def _refit_shipped(points: list[MeasuredPoint], specs, carrier: str) -> CoefficientEntry: +def _refit_shipped( + points: list[MeasuredPoint], specs, carrier: str, platform: dict[str, str] = MIOPEN_NATIVE +) -> CoefficientEntry: """The shipped data with `points` merged and refitted, checked against every point's bounds; returns `carrier`'s - entry for MIOpen's native allocator off Windows.""" + entry for `platform`, MIOpen's native allocator off Windows unless given.""" after = refit(merge_points(coefficients.shipped_data(), points), specs) assert bound_violations(after, specs) == [] [entry] = [ entry for file in after.values() for entry in file.coefficients - if entry.estimator == carrier and entry.platform == MIOPEN_NATIVE + if entry.estimator == carrier and entry.platform == platform ] return entry @@ -520,22 +525,67 @@ def test_a_stand_in_stays_until_its_fresh_process_points_span_the_sizes_of_a_fit assert {"k_decode_fp16", "b_decode_fp16"} <= set(entry.stand_ins) -def test_a_raised_stand_in_takes_the_larger_base_too(specs) -> None: - """CogView4's bf16 decode stand-in off Windows: 13720 bytes per pixel and a 14 MiB base. Probe decodes with a - 1 GiB base and 10000 bytes per pixel fit 10500 and 1076 MiB: the slope stays, the base rises.""" - probes = [ - _miopen_native_point("vae.cogview4", _sd_decode(px, "bf16"), GIB + 10_000 * px * px, "probe_demand") +def _cogview4_probes(per_pixel: int) -> list[MeasuredPoint]: + """Probe decodes of CogView4's VAE at 512px and 1024px, a 1 GiB base and `per_pixel` bytes per pixel.""" + return [ + _miopen_native_point("vae.cogview4", _sd_decode(px, "bf16"), GIB + per_pixel * px * px, "probe_demand") for px in (512, 1024) ] - entry = _refit_shipped(probes, specs, "vae.cogview4") + +def test_probes_a_stand_in_covers_by_the_margin_leave_it_as_it_is(specs) -> None: + """CogView4's bf16 decode stand-in off Windows, 13720 bytes per pixel and a 14 MiB base, reserves 3444 MiB at + 512px and 13734 MiB at 1024px. Probes of a 1 GiB base and 7000 bytes per pixel read 2774 and 8024 MiB: covered by + far more than 5%. Their own fit (7350 bytes per pixel, a 1076 MiB base) must not lift the base by 1062 MiB.""" + entry = _refit_shipped(_cogview4_probes(7000), specs, "vae.cogview4") + + assert entry.values is not None + assert (entry.values["k_decode_bf16"], entry.values["b_decode_bf16"]) == (13720, 14 * MIB) + assert RAISE_NOTE not in entry.provenance + + +def test_a_stand_in_short_of_a_probe_is_raised_by_the_least_its_bounds_allow(specs) -> None: + """Probes of a 1 GiB base and 10000 bytes per pixel read 3524 MiB at 512px, above the stand-in's 3444 MiB, and + 11024 MiB at 1024px, below its 13734. Covering 512px by 5% takes 3700.2 MiB there: 256.2 MiB more base, or 1025 + more bytes per pixel. Averaged over the measured 512px to 1024px, the slope costs 2.5 times as much (the middle, + 655360 px, over 512px's 262144), so the base takes the raise, to 270.2 MiB. Of the lines as they ship, whole bytes + per pixel and whole MiB of base, 13720 with 271 MiB averages 8846 MiB over that range; 13721 with 270 MiB still + covers 512px and averages 8845.625: a byte per pixel more costs 0.625 MiB at the middle, a MiB of base less saves + a whole one.""" + entry = _refit_shipped(_cogview4_probes(10_000), specs, "vae.cogview4") assert entry.values is not None - assert (entry.values["k_decode_bf16"], entry.values["b_decode_bf16"]) == (13720, 1076 * MIB) + k, b = entry.values["k_decode_bf16"], entry.values["b_decode_bf16"] + assert (k, b) == (13721, 270 * MIB) + assert k * 512 * 512 + b >= 1.05 * (GIB + 10_000 * 512 * 512) > k * 512 * 512 + b - MIB assert {"k_decode_bf16", "b_decode_bf16"} <= set(entry.stand_ins) assert RAISE_NOTE in entry.provenance +def test_a_probe_no_hull_edge_covers_within_the_bounds_raises_the_line_between_them(specs) -> None: + """A FLUX.2 probe decode of 2032 MiB at 640px on the RTX 4090, above the cuDNN bf16 line FLUX.2 shares with SD3 + (4500 bytes per pixel, a 65 MiB base: 1823 MiB there). Every edge of the hull through it over-reserves the + fresh-process 512px decodes, or the 1024px batches of two and three, beyond 1.3x; a line between them covers it + by 5% and keeps every bound.""" + probe = MeasuredPoint( + estimator="vae.flux2", + version=2, + platform=CUDNN_NATIVE, + features=_decode(640), + measured_bytes=2032 * MIB, + method="probe_demand", + device="NVIDIA GeForce RTX 4090", + provenance="-", + ) + + entry = _refit_shipped([probe], specs, "vae.sd3", CUDNN_NATIVE) + + assert entry.values is not None + k, b = entry.values["k_decode_bf16"], entry.values["b_decode_bf16"] + assert k >= 4500 and b >= 65 * MIB + assert k * 640 * 640 + b >= 1.05 * 2032 * MIB + + def _fp32_1280(first_mib: int, largest_mib: int) -> list[dict]: """Two fp32 SDXL decodes at 1280px in one long-running server, allocated flat at the first one's bytes.""" return [ @@ -544,11 +594,13 @@ def _fp32_1280(first_mib: int, largest_mib: int) -> list[dict]: ] -def test_a_raise_that_pushes_fresh_process_points_over_their_bound_is_refused_even_with_consent( +def test_a_probe_no_line_can_cover_within_the_fresh_process_bounds_is_refused_even_with_consent( tmp_path: Path, machine, capsys ) -> None: - """The RX 9060 XT's fp32 SDXL decode at 1280px: allocated stays at 8020 MiB while reserved grows to 13000 MiB. The - line that covers it over-reserves the fresh-process points of the same line beyond 1.3x.""" + """The RX 9060 XT's fp32 SDXL decode at 1280px: allocated stays at 8020 MiB while reserved grows to 13000 MiB. No + line at or above the fit covers it by 5% within 1.3x of the fresh-process decodes at 1280px: the data conflicts + with the bounds. The refusal shows the point with its windows and the fragmentation flag that explains it, and + names the probe and a bound it conflicts with, by id, measured bytes and the most it allows.""" export = _amd_export(tmp_path, machine, _fp32_1280(8020, 13000)) data_dir, before = _data_copy(tmp_path) @@ -557,13 +609,38 @@ def test_a_raise_that_pushes_fresh_process_points_over_their_bound_is_refused_ev captured = capsys.readouterr() assert "[2 windows, MiB 8020, 13000; max/first 1.62]" in captured.out assert "reserved MiB 8020, 13000 while allocated MiB 8020, 8020" in captured.out - assert "decode_fp32: k 5988 -> " in captured.out and "1 flagged as fragmentation" in captured.out - assert "the bounds of these points reject the result" in captured.err - violation = "vae.sd-miopen,expandable,windows-decode-fp32-1280x1280-scores-RX9060XT-demand_peak: reserves" - assert violation in captured.err + assert "error: not written: the points conflict with the bounds: the points cannot fit a line" in captured.err + assert "decode_fp32: no line at or above k=5988" in captured.err + assert f"covers 13000 MiB at size {1280 * 1280} by 5% within the upper bound of vae." in captured.err + assert re.search(r"-1280x1280-scores-RX9060XT-demand_peak \(measured \d+ MiB, at most \d+ MiB\)", captured.err) assert _files(data_dir) == before +def test_a_stand_in_no_line_can_raise_within_its_bounds_is_refused(tmp_path: Path, machine, capsys) -> None: + """CogView4's bf16 decode stand-in on Linux MIOpen (13720 bytes per pixel, a 14 MiB base) with one fresh-process + decode of 3000 MiB at 512px, which holds it to 3900 MiB there. A 30000 MiB probe at 1024px needs 31500 MiB: the + least slope that reaches it with the stand-in's base puts 512px at 7886 MiB. Nothing is written.""" + data_dir, _ = _data_copy(tmp_path) + data = load_data(data_dir)["vae.json"] + fresh = _miopen_native_point("vae.cogview4", _sd_decode(512, "bf16"), 3000 * MIB, "demand_peak") + write_data(data_dir / "vae.json", data.model_copy(update={"points": [*data.points, fresh]})) + before = _files(data_dir) + machine("7.14.0", "linux") + window = _window(1, _sd_decode(1024, "bf16"), 30000 * MIB, estimator="vae.cogview4") + export = _export(tmp_path / f"20261017-160953-box-1-{RX_9060_XT.id}.json", RX_9060_XT, [window]) + + assert main([str(export), "--data-dir", str(data_dir), "--write"]) == 1 + + captured = capsys.readouterr() + assert f"+ {30000 * MIB}" in captured.out, "the probe is shown" + stand_in = "the points cannot raise a stand-in: vae.cogview4 on {'conv_backend': 'miopen', 'allocator': 'native'}" + assert stand_in in captured.err + assert "decode_bf16: no line at or above k=13720, b=14 MiB covers 30000 MiB at size 1048576 by 5%" in captured.err + assert "within the upper bound of vae.cogview4-miopen,native-decode-bf16-512x512-" in captured.err + assert "(measured 3000 MiB, at most 3900 MiB)" in captured.err + assert _files(data_dir) == before + + def test_two_owners_attempts_with_one_id_are_measured_apart(tmp_path: Path, machine, entries, specs) -> None: """The two fp32 1280px decodes of `_fp32_1280`, each another owner's attempt "1": reserved grew from one to the other around flat allocations, which within one attempt is fragmentation. Across two it is not.""" @@ -870,6 +947,36 @@ def test_probe_windows_alone_never_replace_a_legacy_entrys_hand_set_values(tmp_p assert [p.method for p in _test_points(data_dir)] == ["probe_demand"] * len(windows) +def test_a_legacy_entry_whose_fit_conflicts_with_the_bounds_stays_legacy(tmp_path: Path, machine, capsys) -> None: + """Fresh RTX 4090 forwards of every dtype at three sizes would replace the legacy entry with their fit, as above. + A warm window 2 GiB above the bf16 line at 16384 image tokens (5778 MiB) needs 6067 MiB there, but the fresh + forward of that size holds the line to 1.3x its 3730 MiB, 4849 MiB: no bf16 line keeps both. The entry keeps its + hand-set values, the note names the conflict, and those values then over-reserve the fresh forwards, which the + bound rule refuses to write.""" + records = [_forward_record(image, dtype) for dtype in FITTED for image in (1024, 4096, 16384)] + points = _point_file(tmp_path / "z_image_points.jsonl", records) + features = _forward_record(16384, "bf16")["features"] + demand = DENOISE_SLOPE * features["tokens"] + DENOISE_BASE + 2 * GIB + window = _window(1, features, demand, estimator=Z_IMAGE_DENOISE_SPEC.id, version=Z_IMAGE_DENOISE_SPEC.version) + machine(None, "linux") + export = _export(tmp_path / f"20261018-120000-box-1-{RTX_4090.id}.json", RTX_4090, [window]) + data_dir, before = _legacy_data_copy(tmp_path) + + assert main([str(export), "--points", str(points), "--data-dir", str(data_dir), "--write"]) == 1 + + captured = capsys.readouterr() + assert ( + f"note: denoise.z_image {CUDNN_NATIVE} stays legacy: denoise_bf16: no line at or above k=236544, b=105 MiB " + "covers 5778 MiB at size 16896 by 5% within the upper bound of " + "denoise.z_image-cudnn,native-denoise-bf16-16384+512tok-NVIDIAGeForceRTX4090-demand_peak " + "(measured 3730 MiB, at most 4849 MiB)" + ) in captured.out + assert "is fitted now, no longer legacy" not in captured.out + assert "the bounds of these points reject the result" in captured.err + assert "conflict with the bounds" not in captured.err + assert _files(data_dir) == before + + def test_a_measurement_a_point_file_gives_twice_keeps_its_larger_reading(tmp_path: Path, entries, specs) -> None: """A rerun that reads less, recorded after the first run, does not take its place.""" larger = _forward_record(4096, "bf16")