From 6bc2beb3643f065d7d7b9a1cc6b1ead54a045524 Mon Sep 17 00:00:00 2001 From: Wouter-Michiel Vierdag Date: Fri, 14 Aug 2026 18:04:49 +0100 Subject: [PATCH 1/6] docs: initial partial guide to progressive loading --- docs/guides/progressive_loading.md | 308 +++++++++++++++++++++++++++++ 1 file changed, 308 insertions(+) create mode 100644 docs/guides/progressive_loading.md diff --git a/docs/guides/progressive_loading.md b/docs/guides/progressive_loading.md new file mode 100644 index 000000000..a4deafd9e --- /dev/null +++ b/docs/guides/progressive_loading.md @@ -0,0 +1,308 @@ +# Progressive image loading +The progressive loader in napari is experimental. It provides viewport aware, progressive chunk-wise +loading for very large multiscale images in napari. It uses the already existing multiscale layers in +napari + +The progressive loader ensures that an entire image pyramid level is not fully materialized before +displaying it. For this, it does the following: +1. It determines which resolution and spatial region are currently useful. +2. Ensures that something renderable is available immediately, e.g. (part of the) lowest resolution of the pyramid +3. Fetches missing chunks of current resolution level based on a priority order (distance from center chunk). +4. Progressively replaces the coarse, low resolution fallbak data with higher-resolution data. +5. Updates the displayed texture while limiting work that could interrupt interaction. + +As a result, the napari viewer can work with datasets substantially larger than CPU or GPU memory +while still keeping navigation responsive. + +## Core concepts +In order to understand the progressive loader we will first explain some of the concepts needed to +understand it. + +### Multiscale data and `VirtualData` +The source data is a multiscale pyramid ordered from highest to lowest resolution. + +Rather than passing the source arrays directly to napari, each pyramid level is wrapped by +`VirtualData`. These arrays still look like arrays to napari, with the full shape of their +corresponding resolution level, but internally they keep only a bounded resident interval +in CPU memory (RAM). + +The purpose of `VirtualData` is to separate the actual logical size of an image from the amount +of data that must actually be materialized at any given time. + +```mermaid +flowchart TD + A["Chunked source"] + B["VirtualData / multiscale
pyramid"] + C["Resident interval
in RAM"] + D["vispy texture"] + E["Canvas"] + + A --> B + B -->|"fetch missing chunks"| C + C -->|"slice / patch"| D + D --> E +``` +Napari's rendering is not replaced. Rather the progressive loader adds coordination between +the different components of the existing rendering pipeline. We will now go more in detail. + +### Resident interval +Only the region needed for the current view is kept resident at a fine-resolution level. The interval +follows the viewport as the user pans, zooms, or changes dimensions. For displayed dimensions, it is +derived from the layer's corner_pixels; non-displayed dimensions are restricted to the current +dimension step. So for 2D data it would conceptually look like this: + +```mermaid +flowchart TB + subgraph P["Full pyramid level"] + direction TB + R["Resident interval
currently held in
CPU memory"] + end +``` +This resident interval is also bounded by memory and GPU texture limits. Moving the viewport moves the +resident interval instead of the camera on a fully materialized resolution level. + +### Resident coarsest level +This level is the lowest resolution level of the pyramid and it is treated specially. When it fits +within `resident_max_bytes`, it is kept fully in RAM. This to provide: +- immediate low-resolution context +- data for layer thumbnails +- a backdrop source for finer levels + +It gives the user something to immediately show, while higher resolution chunks are still being fetched. + +### Backdrop data +When zooming and panning, a fine-resolution interval does not have to start empty. +hen the new interval overlaps the previous one, such as during a pan, already loaded +target-resolution data in the overlapping region is retained. Only the newly exposed or +otherwise unloaded regions need temporary coverage. +For those unloaded regions, the loader uses already available data from another pyramid +level as a backdrop. It prefers the closest-resolution level whose resident data covers +the requested region. If no closer level provides sufficient coverage, the resident coarsest +level can serve as the fallback. Backdrop data is upsampled to the target resolution before +being written into the target resident interval. +The backdrop therefore provides temporary image content. It does not mark those regions as +loaded at the target level. They still need to be replaced by actual target-resolution chunks. +When `coarse_first=True`, the backdrop can be progressively improved before the target level is +fetched. Missing chunks from intermediate pyramid levels are fetched from coarse to fine. As +those chunks arrive, their data is upsampled and folded into regions of the target interval +that do not yet contain loaded target chunks. Intermediate levels are not rendered directly. +Finally, missing target-level chunks are fetched and written into the target level's resident +interval in `VirtualData`, which is held in RAM. These chunks replace any backdrop or +intermediate-resolution data in the corresponding regions. Once all required target-level +chunks for the requested view have been loaded, those regions of the resident interval +contain actual target-rresolution data. + +The flow for populating and progressively refining a resident interval is therefore like this: +```mermaid +flowchart TD + A["View changes
pan or zoom"] --> B["Establish target
resident interval"] + + B --> C["Carry over already loaded
target-level data"] + + C --> D["Identify unresolved regions"] + + D --> E["Fill unresolved regions
from available backdrop data"] + + E --> F["Upsample backdrop data
to target resolution"] + + F --> G["Resident interval has
useful image coverage"] + + G --> H{"coarse_first?"} + + H -- Yes --> I["Fetch missing chunks from
intermediate pyramid levels
coarse → fine"] + + I --> J["Upsample intermediate data into
regions without target-level chunks"] + + J --> K["Progressively sharper
temporary coverage"] + + K --> L["Fetch missing
target-level chunks"] + + H -- No --> L + + L --> M["Write target chunks into
VirtualData in CPU memory"] + + M --> N["Replace backdrop or intermediate
data in corresponding regions"] + + N --> O{"Missing target
chunks remain?"} + + O -- Yes --> L + O -- No --> P["Requested view is fully resolved
at the target resolution"] +``` +The key take away here is that backdrop data, intermediate data and target-level data can coexist +within the same resident interval while loading is in progress. The resident coarsest level, backdrop +and intermediate data provide immediate visual coverage, while the set of loaded target chunks +determines which regions have actually been resolved at the requested resolution. With `coarse_first=True`, +intermediate levels progressively sharpen unresolved regions and with `coarse_first=False`, +the loader proceeds directly from the initial backdrop to target-level chunks. + +## How progressive loading works in the experimental implementation +In the previous section we described some of the nomenclature, how data is represented and how it +is progressively refined. Here, we will describe how the loader decides what to load as the user +interacts with the viewer. +`ProgressiveLoader` responds to changes in the viewer and layer that can affect what data is needed. +Examples of this are camera movement, dimension changes, display mode, layer visibility and changes +to the selected resolution level. +When there is such an event, the loader does not always immediately start a new fetch. Events such +as camera interactions often generate many events in quick succession. Therefore, the loader waits +for the view to settle before determining the next region to load. While the user is actively interacting, +expensive streaming and rendering work can also be temporarily suspended to keep navigation responsive. + +Once the view is settled, the loader determines a couple of things: +1. Which pyramid level should be used. +2. Which region of that level is needed for the view. +3. What part of the region is already resident or currently being fetched. + +If the required data is not fully resident / in RAM, a new data fetch is started. +We present this here schematically: + +```mermaid +flowchart TD + A["Viewer or layer
state changes"] --> B["Interaction detected"] + + B --> C["Temporarily limit
streaming and rendering work"] + + C --> D["Wait for view
to settle"] + + D --> E["Determine target
pyramid level"] + + E --> F["Determine required
resident interval"] + + F --> G{"Required region
already fully covered?"} + + G -- Yes --> H["No new fetch
pass required"] + + G -- No --> I["Start new
fetch pass"] + + I --> J["Populate and progressively
refine resident interval"] +``` +We will now describe these 3 steps in more detail + +### 1. Selecting the resolution level +The way the resolution level is determined varies between 2D and 3D data in napari. +This because loading 3D data at a given resolution level could prove to be too +expensive. We will describe the selection of the resolution level in both 2D and 3D. + +### Selection the resolution level for 2D data +For 2D data, napari's existing multiscale API selects the resolution level. The +progressive loader uses the elvel selected by this API rather than introducing +a separate mechanism for selecting the target resolution. +As such, `locked_data_level` (TODO: explain this) is respected. Once the level is known, +the loader determines the region required for the current view from the layer's +`corner-pixels`. + +### 1. Selecting the resolutoin level for 3D data +For 3D data, the resolution level can either be selected automatically by the progressive loader +or controlled through napari's normal level selection. This has different trade offs. + +With a setting `auto_level_3d=True`, the progressive loader automatically selects the resolution +level based on the camera zoom (TODO check whether this is napari selected). It then aims to select +the coarsest level of which the voxels project to no more than `max_pixel_size_3d` screen pixels +(the unit here is pixels per voxel). Lower values cause finer, more expensive levels to be selected +sooner while zooming in. For example, with `max_pixel_size_3d=2.0`, a finer level is preferred once a +voxel would occupy more than two screen pixels. +The visually appropriate level must also be practical to load and render. If the desired level would +exceed constraints such as the resident-interval memory budget or the maximum number of chunks +allowed for a 3D fetch pass, the loader selects a coarser level instead. +When auto_level_3d=False, the progressive loader does not automatically change the pyramid level in +response to camera zoom. Level selection is left to napari or to the user. The loader still performs +progressive loading for whichever level is currently selected: it determines the required resident +interval, provides backdrop coverage, and fetches the missing chunks for that level. +If the user explicitly selects a resolution level while automatic 3D level selection is enabled, +that selection takes precedence and automatic level changes are suspended. Returning the resolution +selector to Auto gives level selection back to the progressive loader. + +The whole mechanism of 3D resolution level selection can therefore be summarized schematically as: +```mermaid +flowchart TD + A["3D view"] --> B{"auto_level_3d?"} + + B -- No --> C["Use level selected by
napari or the user"] + + B -- Yes --> D{"User explicitly
selected a level?"} + + D -- Yes --> C + + D -- No --> E["Determine coarsest level that
satisfies max_pixel_size_3d"] + + E --> F{"Level fits memory and
chunk-count constraints?"} + + F -- Yes --> G["Use selected level"] + + F -- No --> H["Try a coarser level"] + + H --> F + + C --> I["Progressively load
selected level"] + G --> I +``` +With this design, 2 concerns are separated: automatic level selection determines which resolution +to use, while progressive loading determines how the selected level is populated and displayed. + +### 2. Determining the region to load +ONce a resolution level has been selected, the loader determines which part of that elvel is relevant +to the current view. +For displayed dimensions, this region is based on the layer's `corner_pixels`. For dimensions that are +not currently displayed, only the current dimension step needs to be represented (TODO: check with Kyle +if he sees value here for different mode, e.g. with tracking data). +The requested region is then converted into the resident interval used by `VirtualData`. + +The resident interval can be larger than the exact rendered viewport. Specifically, the GPU texture +contains the crop described by `corner_pixels`, while the resident interval on the CPU follows underlying +chunk boundaries and can therefore extend beyond the rendered region as shown below: + +```mermaid +flowchart TB + subgraph R["Resident interval in RAM"] + direction TB + + V["Rendered viewport
represented by GPU texture"] + end +``` + +### 3. Fetching chunks +After the resident interval and backdrop have been established, the loader determines which chunks +still need to be loaded. +The source arrays are chunked, meaning that the data is divided in predefined blocks, rather than +arbitrary pixel slices. Only chunks intersercting the requested region are considered with chunks +already recorded as loaded excluded from the fetch queue. +As stated before, the resident interval follows the underlying data source chunk boundaries, +while the viewport contains a crop of it. While this may cause the resident interval to contain +data outside the visible viewport, it avoids repeatedly partially fetching chunks as the view moves. +Thus, an overview of fetching chunks looks like this: + +```mermaid +flowchart TD + A["Visible viewport"] --> B["Determine intersecting
source chunks"] + + B --> C["Chunk-aligned
resident region"] + + C --> D["Exclude chunks that
are already loaded"] + + D --> E["Prioritize remaining
missing chunks"] + + E --> F["Fetch chunks"] +``` +We will now discuss chunk priority and fetch workers. + +#### Chunk priority +Missing chunks are not fetched with storage order in mind, rather they are prioritized based on +their relevance to the current view. + +In 2D, chunks are prioritized based on their distance to the center of the resident interval. Distance is +not calculated for all chunks of the underlying data source. Instead, only distance is only calculated for +chunks intersecting with the resident interval. + +For 3D, the camera is also taken into account. Chunks are primarily fetched based on their depth +along the view direction, with those closest to the viewer loading first. The distance from the camera +center line provides a second, but lower weighted, prioritization for fetching chunks. This ensures +chunks on the camera center line will be prioritized when multiple chunks are present at equal depth. + +For both 2D and 3D data, the aim is not to resolve every chunk intersecting with the resident interval simultaneously, +but to spend the available loading capacity first on those chunks likely to visually matter the most. + +#### Fetch workers +Fetching chunks happens on background worker threads, rather than on the main thread. The number of concurrent workers +is controlled by fetch_workers. By default, the implementation uses up to four workers while deliberately leaving +CPU capacity (2 cores by default) available for the GUI. + +### Keeping interaction responsive From 4a72b85bdde8fe017f74421925621e57d2deb291 Mon Sep 17 00:00:00 2001 From: Wouter-Michiel Vierdag Date: Sun, 16 Aug 2026 11:47:39 +0100 Subject: [PATCH 2/6] docs: add progressive loading to toc --- docs/_toc.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/_toc.yml b/docs/_toc.yml index d8b139b32..c1a1a3de3 100644 --- a/docs/_toc.yml +++ b/docs/_toc.yml @@ -66,6 +66,7 @@ subtrees: - file: guides/event_loop - file: guides/threading - file: guides/events_reference + - file: guides/progressive_loading - file: further-resources/glossary - file: further-resources/napari-workshops - file: further-resources/sample_data From ab8e58d66a9aafd0b718be50c88b3cd705e420ba Mon Sep 17 00:00:00 2001 From: Wouter-Michiel Vierdag Date: Sun, 16 Aug 2026 12:05:53 +0100 Subject: [PATCH 3/6] docs: switch to directive sytax --- docs/guides/progressive_loading.md | 43 +++++++++++++++++++++++++----- 1 file changed, 36 insertions(+), 7 deletions(-) diff --git a/docs/guides/progressive_loading.md b/docs/guides/progressive_loading.md index a4deafd9e..765ceb2a6 100644 --- a/docs/guides/progressive_loading.md +++ b/docs/guides/progressive_loading.md @@ -29,7 +29,7 @@ in CPU memory (RAM). The purpose of `VirtualData` is to separate the actual logical size of an image from the amount of data that must actually be materialized at any given time. -```mermaid +```{mermaid} flowchart TD A["Chunked source"] B["VirtualData / multiscale
pyramid"] @@ -51,7 +51,7 @@ follows the viewport as the user pans, zooms, or changes dimensions. For display derived from the layer's corner_pixels; non-displayed dimensions are restricted to the current dimension step. So for 2D data it would conceptually look like this: -```mermaid +```{mermaid} flowchart TB subgraph P["Full pyramid level"] direction TB @@ -93,7 +93,7 @@ chunks for the requested view have been loaded, those regions of the resident in contain actual target-rresolution data. The flow for populating and progressively refining a resident interval is therefore like this: -```mermaid +```{mermaid} flowchart TD A["View changes
pan or zoom"] --> B["Establish target
resident interval"] @@ -155,7 +155,7 @@ Once the view is settled, the loader determines a couple of things: If the required data is not fully resident / in RAM, a new data fetch is started. We present this here schematically: -```mermaid +```{mermaid} flowchart TD A["Viewer or layer
state changes"] --> B["Interaction detected"] @@ -212,7 +212,7 @@ that selection takes precedence and automatic level changes are suspended. Retur selector to Auto gives level selection back to the progressive loader. The whole mechanism of 3D resolution level selection can therefore be summarized schematically as: -```mermaid +```{mermaid} flowchart TD A["3D view"] --> B{"auto_level_3d?"} @@ -250,7 +250,7 @@ The resident interval can be larger than the exact rendered viewport. Specifical contains the crop described by `corner_pixels`, while the resident interval on the CPU follows underlying chunk boundaries and can therefore extend beyond the rendered region as shown below: -```mermaid +```{mermaid} flowchart TB subgraph R["Resident interval in RAM"] direction TB @@ -270,7 +270,7 @@ while the viewport contains a crop of it. While this may cause the resident inte data outside the visible viewport, it avoids repeatedly partially fetching chunks as the view moves. Thus, an overview of fetching chunks looks like this: -```mermaid +```{mermaid} flowchart TD A["Visible viewport"] --> B["Determine intersecting
source chunks"] @@ -305,4 +305,33 @@ Fetching chunks happens on background worker threads, rather than on the main th is controlled by fetch_workers. By default, the implementation uses up to four workers while deliberately leaving CPU capacity (2 cores by default) available for the GUI. +In summary, a fetch thus happens in a way shown in this diagram: +```{mermaid} +flowchart TD + A["Target resident interval"] --> B["Find intersecting chunks"] + + B --> C["Remove already
loaded chunks"] + + C --> D["Prioritize missing chunks"] + + D --> E{"coarse_first?"} + + E -- Yes --> F["Fetch intermediate-level
chunk stages first"] + + F --> G["Fill unresolved target regions with
intermediate-level data"] + + G --> H["Fetch target-level chunks"] + + E -- No --> H + + H --> I["Write fetched chunks into
target VirtualData in RAM"] + + I --> J["Update displayed data"] + + J --> K{"More target chunks
to fetch?"} + + K -- Yes --> H + K -- No --> L["Fetch pass complete"] +``` + ### Keeping interaction responsive From 9301c1931983a6713fa9e7d8c858eff14e8d1d45 Mon Sep 17 00:00:00 2001 From: Wouter-Michiel Vierdag Date: Sun, 16 Aug 2026 12:45:36 +0100 Subject: [PATCH 4/6] docs: add section on keeping interaction responsive --- docs/guides/progressive_loading.md | 64 ++++++++++++++++++++++++++++++ 1 file changed, 64 insertions(+) diff --git a/docs/guides/progressive_loading.md b/docs/guides/progressive_loading.md index 765ceb2a6..99a38ff83 100644 --- a/docs/guides/progressive_loading.md +++ b/docs/guides/progressive_loading.md @@ -335,3 +335,67 @@ flowchart TD ``` ### Keeping interaction responsive +Progressive loading runs while the user is navigating the dataset, so solely maximizing chunk +throughput is not the only goal. Rather, we try to maximize the chunk throughput with the viewer +still responding smoothly to user input. For this, the `ProgressiveLoader` has a few mechanisms +in place. + +#### Interaction hold +The `ProgressiveLoader` has a setting called `interaction_hold`. If set to `True`, the loader +temporarily reduces background loading and display-update work while the user is actively +interacting with the viewer. This prevents progressive loading from competing with interactions +such as panning, zooming, rotating, or moving through dimensions. During this period: + +- Fetch workers are paused +- Chunks that finish loading are kept for later processing rather than immediately updating the display +- Non-essential display updates are deferred +- In 3D, rendering quality can be temporarily reduced to make interactive frames cheaper + +Once the interaction settles, the loader resumes the suspended work, processes any chunks that +completed in the meantime, and reevaluates the current view. In short, upon user interaction +this diagram is followed: + +```{mermaid} +flowchart TD + A["User starts interacting"] --> B["Enter interaction hold"] + + B --> C["Pause chunk fetching"] + B --> D["Defer completed chunk
display updates"] + B --> E["Reduce 3D rendering cost"] + + C --> F["Interaction settles"] + D --> F + E --> F + + F --> G["Resume loading and
display updates"] + + G --> H["Reevaluate current view"] +``` + +#### Rate limiting the loading of chunks +Even when chunk fetching happens on background threads, loading data as quickly as possible can +still affect viewer responsiveness. Fetching a chunk can involve more than waiting for I/O: each +chunk leads to additional CPU and display-update work. +The `ProgressiveLoader` provides a setting, `max_bytes_per_second`, that limits the rate at which +fetch workers start loading chunks. +This pacing happens on the worker threads, before each fetch. However, it indirectly limits the pace +of work that has to happen downstream: + +```{mermaid} +flowchart TD + A["Rate-limit chunk fetching"] --> B["Reduce source loading /
computation pressure"] + + B --> C["Reduce frequency of
completed chunk batches"] + + C --> D["Reduce display-update work"] + + D --> E["Reduce GPU upload pressure"] +``` + +The purpose of the rate limit is therefore not necessarily to restrict network bandwidth. It +provides a way to control the amount of loading-related work entering the pipeline so that +background loading does not overwhelm interactive rendering. + +Together, rate limiting and interaction hold serve different purposes. Rate limiting controls +how aggressively loading proceeds during normal operation, while interaction hold temporarily +gives priority to active user interaction. From 3afe28e34f8d87e85acb714a383ecf68d3540425 Mon Sep 17 00:00:00 2001 From: Wouter-Michiel Vierdag Date: Sun, 16 Aug 2026 20:24:15 +0100 Subject: [PATCH 5/6] Update docs/guides/progressive_loading.md Co-authored-by: Kyle I S Harrington --- docs/guides/progressive_loading.md | 60 ++++++++++++++++++++++++++++++ 1 file changed, 60 insertions(+) diff --git a/docs/guides/progressive_loading.md b/docs/guides/progressive_loading.md index 99a38ff83..6146246bb 100644 --- a/docs/guides/progressive_loading.md +++ b/docs/guides/progressive_loading.md @@ -14,6 +14,66 @@ displaying it. For this, it does the following: As a result, the napari viewer can work with datasets substantially larger than CPU or GPU memory while still keeping navigation responsive. +## Using progressive loading + +### When it applies + +Progressive loading applies to lazy, chunked, multiscale Image and Labels data, such as Zarr or Dask pyramids. Single-scale data and in-memory pyramids are not converted automatically. Spatially chunked data with reasonably sized chunks gives the best results. + +### Enable and use it + +Enable **Preferences > Experimental > Progressive loading of multiscale data** before adding the layer. Then open the data normally using File > Open, drag-and-drop, a reader plugin, or `viewer.add_image(..., multiscale=True)`. The setting only affects layers added after it is enabled. + +It can also be enabled from the command line: + + NAPARI_PROGRESSIVE_LOADING=1 napari + +### Advanced Python usage + +The experimental Python helper provides access to tuning options: + + from napari.experimental._progressive_loading import ( + add_progressive_loading_image, + ) + + layer = add_progressive_loading_image( + pyramid, + viewer=viewer, + interval_max_bytes=512 * 1024**2, + max_bytes_per_second=100 * 1024**2, + ) + +Use `add_progressive_loading_labels` for Labels data. These underscored APIs are experimental and may change. + +### Practical tuning + +| Symptom | Adjustment | +| --- | --- | +| Loading interrupts interaction | Lower `max_bytes_per_second` | +| Resident data uses too much memory | Lower `interval_max_bytes` | +| 3D texture updates are expensive | Lower `tile_max_bytes_3d` | +| 3D selects expensive levels too early | Increase `max_pixel_size_3d` | +| Fine detail takes too long to begin loading | Try `coarse_first=False` | + +Chunk layout often matters more than these settings. Very large chunks, or chunks spanning most of a spatial axis, limit how progressively napari can load the data. + +### Troubleshooting + +- **Nothing changed:** enable the setting before adding the layer and confirm that the data is both chunked and multiscale. +- **Loading is blocky or slicewise:** inspect the source chunk shape; smaller spatial chunks may work better. +- **The unexpected 3D level is shown:** return the resolution selector to **Auto**, or adjust `max_pixel_size_3d` when using the Python helper. +- **More diagnostics are needed:** pass `debug_overlay=True` or set `NAPARI_PROGRESSIVE_DEBUG=1`. + +### Current user-facing limitations + +- The behavior and Python API are experimental and may change. +- Automatic conversion supports only eligible chunked multiscale Image and Labels layers. +- Changing the setting does not convert existing layers; attached loaders continue until their layer is removed. +- Enabling progressive loading also enables asynchronous rendering. +- Multiscale Labels layers are read-only. +- Unsuitable source chunking may still cause pauses or uneven refinement. +- Moving a dims slider may require a subsequent pan or zoom before the display refreshes. + ## Core concepts In order to understand the progressive loader we will first explain some of the concepts needed to understand it. From 8a65ea5597b7c750d26116ef5360deec73ea4fbd Mon Sep 17 00:00:00 2001 From: Wouter-Michiel Vierdag Date: Mon, 17 Aug 2026 12:41:36 +0100 Subject: [PATCH 6/6] docs: complete initial draft --- docs/guides/progressive_loading.md | 257 ++++++++++++++++++++++++++++- 1 file changed, 254 insertions(+), 3 deletions(-) diff --git a/docs/guides/progressive_loading.md b/docs/guides/progressive_loading.md index 6146246bb..317c80c93 100644 --- a/docs/guides/progressive_loading.md +++ b/docs/guides/progressive_loading.md @@ -242,7 +242,7 @@ The way the resolution level is determined varies between 2D and 3D data in napa This because loading 3D data at a given resolution level could prove to be too expensive. We will describe the selection of the resolution level in both 2D and 3D. -### Selection the resolution level for 2D data +#### Selection the resolution level for 2D data For 2D data, napari's existing multiscale API selects the resolution level. The progressive loader uses the elvel selected by this API rather than introducing a separate mechanism for selecting the target resolution. @@ -250,7 +250,7 @@ As such, `locked_data_level` (TODO: explain this) is respected. Once the level i the loader determines the region required for the current view from the layer's `corner-pixels`. -### 1. Selecting the resolutoin level for 3D data +#### Selecting the resolutoin level for 3D data For 3D data, the resolution level can either be selected automatically by the progressive loader or controlled through napari's normal level selection. This has different trade offs. @@ -394,7 +394,7 @@ flowchart TD K -- No --> L["Fetch pass complete"] ``` -### Keeping interaction responsive +### 4. Keeping interaction responsive Progressive loading runs while the user is navigating the dataset, so solely maximizing chunk throughput is not the only goal. Rather, we try to maximize the chunk throughput with the viewer still responding smoothly to user input. For this, the `ProgressiveLoader` has a few mechanisms @@ -459,3 +459,254 @@ background loading does not overwhelm interactive rendering. Together, rate limiting and interaction hold serve different purposes. Rate limiting controls how aggressively loading proceeds during normal operation, while interaction hold temporarily gives priority to active user interaction. + +### 5. Updating the displayed data +So far, we have mainly discussed how chunks are fetched and written into the target level's +`VirtualData` in RAM. Once those chunks arrive, the corresponding changes also need to become +visible on screen. +The CPU-side `VirtualData` remains the source of the progressively loaded image data. The displayed +representation is a texture held in GPU memory by the napari/vispy rendering pipeline: + +```{mermaid} +flowchart TD + A["Fetched target chunk"] --> B["Write chunk into target
VirtualData in RAM"] + + B --> C["Update corresponding
displayed data"] + + C --> D["GPU texture"] + + D --> E["Canvas"] +``` + +There are two main ways in which the progressive loader can make newly loaded data visible: +through napari's normal refresh (`layer.refresh()`) path or, when possible, by directly updating +only the affected part of the GPU texture. + +#### Normal refresh +A normal refresh by `layer.refresh()` causes the currently required data to be sliced again and +updates the displayed texture: + +```{mermaid} +flowchart TD + A["VirtualData changed"] --> B["napari refresh"] + + B --> C["Slice current view"] + + C --> D["Upload texture data"] + + D --> E["Render updated view"] +``` +However, repeatedly performing the complete path for every arriving chunk batch can become expensive, particularly +for large 3D textures. However, performing a complete refresh every time a batch of chunks arrives can be expensive. +In particular, a refresh may involve re-slicing data and uploading an entire texture even when only a small region +has changed. The progressive loader therefore uses several mechanisms to reduce this work. + +#### Refresh throttling +When a `layer.refresh` is required, the loader avoids triggering one for every arriving chunk batch. +The `refresh_interval_s` setting of the `ProgressiveLoader` specifies the minimum time between refreshes +while chunks are being loaded. The effective interval can increase when refreshes themselves are +expensive (such as with 3D volumes), spacing subsequent refreshes farther apart. +This allows multiple chunk updates to accumulate between refreshes: + +```{mermaid} +flowchart TD + A["Chunks arrive"] --> B["Write chunks into
VirtualData"] + + B --> C{"Refresh interval
has elapsed?"} + + C -- No --> D["Defer refresh"] + D --> A + + C -- Yes --> E["Refresh current view"] + + E --> F["Measure refresh cost"] + + F --> G["Adjust effective
refresh interval"] + + G --> A +``` +Refresh throttling therefore trades immediately displaying each update, for fewer expensive +slicing and texture-uploads. + +#### Texture patching +A full refresh is not always necessary. When the setting `texture_patching=True`, the loader +first attempts to update only the region of the existing GPU texture affected by the newly +loaded chunks. +The implementation supports partial texture updates for both 2D and 3D displays. This is +particularly useful in 3D, where a normal refresh can require re-slicing and re-uploading an +entire volume tile. +For a batch or arriving chunks, the loader combines their affected area into a bounding region +and attempts to upload that region as a single partial texture update. Data within that region +that has not yet reached target resolution simply retains or re-uploads its current backdrop +content. + +```{mermaid} +flowchart TD + A["Target chunks written into
VirtualData in RAM"] --> B{"Texture can be
patched safely?"} + + B -- Yes --> C["Determine affected
texture region"] + + C --> D["Upload changed region
to GPU texture"] + + D --> E["Update displayed view"] + + B -- No --> F["Fall back to
throttled refresh"] + + F --> G["Slice current view"] + + G --> H["Upload displayed data
to GPU texture"] + + H --> E +``` +Texture patching is only performed when the loader can establish that the current GPU texture +corresponds to the current rendered region. For example, its shape and position must match the +crop represented by layer's current `corner_pixels`. If the texture cannot be matched safely +to the current view, the loader falls back to a normal refresh. +Texture patching only changes how newly loaded data is transferred from the CPU-side `VirtualData` +to the GPU texture. It does not change how chunks are fetched or stored: `VirtualData` in RAM +remains the authoritative representation of the resident image data. + +During a fetch pass, the loader therefore generally follows this strategy: +1. Write arriving target chunks into `VirtualData` +2. Try patch the corresponding GPU texture region directly +3. Fall back to a throttled `layer.refresh()` if patching is not possible +4. After a fully fetched pass successfully texture-patched, defer the normal refresh and later use it to reconcile +the slice state and layer thumbnail in napari, while avoiding a redundant full texture upload. + +#### Double buffering +When the view changes, the texture needed for the new view may not be ready immediately. Replacing +the currently displayed texture too early could expose an incomplete tile while backdrop data is +being prepared or chunks are still arriving. + +When `double_buffer=True` for the `ProgressiveLoader`, it uses double buffering to separate the +texture currently being displayed from the texture being prepared for the new view. Think of it +as one texture being the front buffer which remains visible, while another texture acts as a back +buffer and receives data for the new view. + +```{mermaid} +flowchart TD + A["View changes"] --> B["Keep current front texture
visible"] + + B --> C["Prepare new view in
back texture"] + + C --> D["Apply backdrop and
chunk updates"] + + D --> E{"Back texture ready
to present?"} + + E -- No --> D + E -- Yes --> F["Present back texture"] + + F --> G["Prepared texture becomes
the displayed texture"] +``` + +At the start of a fetch pass, the double buffer is attached before the initial backdrop refresh. +This allows a full texture upload or texture reallocation for the new view to be staged without +immediately replacing the texture that is currently being rendered. +This is especially useful for 3D volumes. When the view changes, the data needed for the new +view may not be ready immediately. The previous texture can remain visible while the new texture +is populated with backdrop data and newly fetched chunks, preventing an incomplete or empty view +from being shown. + +Once the back buffer / texture is ready to be displayed, it can be presented and becomes the new +front texture. It is important to note that this whole mechanism does not introduce another copy +of the image pyramid in GPU memory. The two texture buffers are maintained for the currently +rendered region, image region in 2D or volume tile in 3D, so that one can remain visible while the +other is beign prepared. + +#### GPU upload metering +Even with texture patching and double buffering, transferring a large texture from CPU memory to GPU +memory can take enough time to interrupt interaction. +The loader therefore meters larger texture uploads. Instead of allowing a large pendign upload +to monopolize a rendeirng frame, the transfer can be divided into smaller pieces and spread +across multiple frames: + +```{mermaid} +flowchart TD + A["Texture data waiting
for GPU upload"] --> B["Upload a bounded
amount"] + + B --> C["Return control to
rendering"] + + C --> D{"More texture data
pending?"} + + D -- Yes --> B + D -- No --> E["Upload complete"] +``` +This mechanism addresses a different cost from refresh throttling and texture patching: +- refresh throttling limits how often the normal `layer.refresh` is used. +- Texture patching avoids a full refresh when an existing texture can be updated directly. +- Double buffering keeps a valid texture visible while a replacement is prepared. +- GPU upload metering limits how much texture-transfer to the GPU is performed at once. + +All these mechanisms also connect back to the interaction hold explained previously. During +active interaction, metered texture uploads can be held along with chunk fetching and other +non-essential display updates. Once interaction settles, the pending operations are resumed. + +### 6. Changing views while loading +A fetch pass may still be running when the user pans, zooms, changes dimensions or selects +another resolution level. In that case, continuing to treat the old request as current would +be wasteful and could cause the now stale result to affect the new view. +Therefore, the loader associates loading work with the current view. The old stale request +is replaced when a new fetch pass is required: + +```{mermaid} +flowchart TD + A["Fetch pass in progress"] --> B["View changes"] + + B --> C["Determine new level
and resident interval"] + + C --> D{"Same active
view?"} + + D -- Yes --> E["Keep current
fetch pass"] + + D -- No --> F["Cancel active pass"] + + F --> G["Establish new
resident interval"] + + G --> H["Start fetch pass
for new view"] +``` +When a new pass starts, the active worker is canceled and a new generation is created +for the new request. The previous workers are ignored by callbacks. This is possible +by keeping track of the number of the generation of workers. +This is important because canceling background work does not necessarily mean that every +operation in progress stops instantaneously. Generation checking is an ultimate defensive +measure for preventing stale results from being treated as the results for the current view. + +### 7. Fetch pass completion +A fetch pass is complete once all required stges for the current view have finished. +With `coarse_first=True`, these stages include the intermediate pyramid levels followed +by the target level. Otherwise, the target level is the main fetch stage. +By this point, the requested regions have been populated with the target-resolution data and +the loader can release any presentation holds associated with the pass. If texture patching +successfully handled the entire target pass, the displayed texture already contains the fetched +daata. A normal refresh can therefore be deferred until it is needed to update the slice state +and layer thumbnail, as explaiend previously. A final schematic summary of a progressive loading +request thus becomes: + +```{mermaid} +flowchart TD + A["View changes"] --> B["Select target level"] + + B --> C["Determine resident interval"] + + C --> D["Carry over existing data
and establish backdrop"] + + D --> E["Build prioritized
chunk fetch stages"] + + E --> F["Fetch chunks in
background workers"] + + F --> G["Write chunks into
VirtualData in RAM"] + + G --> H["Patch GPU texture or
use throttled refresh"] + + H --> I{"More chunks or
stages remain?"} + + I -- Yes --> F + I -- No --> J["Complete fetch pass"] + + J --> K["Present final staged
content if necessary"] + + K --> L["View remains available
for further interaction"] + + L --> M["Next view change"] + M --> B +```