Initial draft PR to establish igzip integration branch for ongoing IGZIP fixes and improvements by asonje · Pull Request #53 · intel/zlib-accel

asonje · 2026-03-13T07:07:53Z

Summary

Initial draft PR to establish igzip integration branch for ongoing IGZIP fixes and improvements before merge to main.

Scope (Initial Push)

Brings current igzip branch baseline into review visibility.
Includes integration of igzip shim logic as well as functional tests.
Serves as the collaboration branch for follow-up PRs targeting igzip.

Purpose

Provide a shared upstream branch for iterative development, validation, and review.

Tracking

References #46

Copilot

Pull request overview

This PR brings an initial ISA-L/igzip integration baseline into the main review flow, adding IGZIP as an additional execution path and enabling it as an intermediate fallback between hardware accelerators (IAA/QAT) and software zlib.

Changes:

Add IGZIP as a selectable execution path for deflate/inflate, plus optional accelerator→IGZIP fallback controlled by config.
Introduce ISA-L (IGZIP) build integration (CMake option, version gating) and expose new config options/statistics counters.
Expand the test suite substantially with IGZIP-focused regression tests and new parameter combinations.

Reviewed changes

Copilot reviewed 16 out of 17 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
`zlib_accel.h`	Adds `IGZIP` to `ExecutionPath` enum for path reporting/testing.
`zlib_accel.cpp`	Integrates IGZIP into deflate/inflate routing, fallback logic, and stream lifecycle handling.
`igzip.h`	Declares the IGZIP shim API used by `zlib_accel.cpp`.
`igzip.cpp`	Implements ISA-L-backed compress/uncompress helpers and stream handling.
`config/default_config`	Adds default toggles for IGZIP enablement.
`config/config.h`	Adds config enums for IGZIP enablement and fallback behavior.
`config/config.cpp`	Wires parsing/defaults for IGZIP-related configuration options.
`statistics.h`	Adds IGZIP counters for deflate/inflate success/error stats.
`statistics.cpp`	Adds IGZIP stat name strings to match new counters.
`tests/zlib_accel_test.cpp`	Adds IGZIP coverage across parametrized tests + large regression test suite.
`tests/CMakeLists.txt`	Minor update to `run` target command (commented filter).
`README.md`	Documents new CMake options, ISA-L dependency, and `igzip_fallback`; updates IAA marker docs.
`iaa.cpp`	Updates comments and changes IAA decompressibility heuristics (512-byte gate) tied to new fallback approach.
`sharded_map.h`	Adds missing `<mutex>` include (supports locking types used in the header).
`common.cmake`	Adds `USE_IGZIP`, ISA-L version enforcement, and ISA-L link/include setup.
`CMakeLists.txt`	Adds `igzip.cpp` to the shared library build.
`.gitignore`	Updates ignored build/artifact paths.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

- zlib_accel.cpp deflateEnd: add null check before dereferencing deflate_settings (ShardedMap::Get returns nullptr for unregistered or double-ended streams) - zlib_accel.cpp inflateEnd: same null check for inflate_settings - zlib_accel.cpp inflateReset: move isal_strm check inside the inflate_settings null guard (was unconditionally dereferencing) - igzip.cpp CompressIGZIP: fix ret = 0:1 -> Z_OK:Z_STREAM_ERROR (1 == Z_STREAM_END, not an error code) - igzip.cpp/h: remove DeflateSetDictionaryIGZIP and InflateSetDictionaryIGZIP — UB casts on strm->state; will be re-added with correct design when dictionary support is implemented - common.cmake: remove invalid PUBLIC keyword from all three link_directories() calls (QPL, ISAL, QATZIP)

matt-welch

Inline comments left where appropriate.

PR #53 Review: IGzip Integration

Adds ISA-L (igzip) as a third accelerator backend, wired into deflate / inflate / compress2 / uncompress2 / gzwrite / gzread, with IAA/QAT → IGZIP fallback and ~1600 lines of regression tests. Substantial, well-structured work. Findings below are what to address before merge.

Correctness

1. `UncompressIGZIP` returns `1` instead of `Z_DATA_ERROR` on ISA-L errors : Medium

igzip.cpp:424 initializes ret = 1 and only reassigns it for ISAL_DECOMP_OK, ISAL_END_INPUT, and ISAL_NEED_DICT. Any other decomp code (e.g. ISAL_INVALID_BLOCK, ISAL_INVALID_SYMBOL, ISAL_INVALID_LOOKBACK) leaves ret == 1: not a valid zlib return code, and the log message reads "inflate finished with error code 1".

Downstream consequence: IGZIPRunInflateAndSelectPathAction at igzip.cpp:365-366 explicitly branches on *ret == Z_DATA_ERROR to return IGZIP_INFLATE_PATH_FALLBACK_DATA_ERROR. Because UncompressIGZIP never returns Z_DATA_ERROR from a real ISA-L error, that branch is only reachable via the null-arg guards at lines 339/349 : the intended "raw trailer fallback" path is effectively dead code.

Fix: map ISAL_INVALID_BLOCK/ISAL_INVALID_SYMBOL/ISAL_INVALID_LOOKBACK/default → Z_DATA_ERROR in the UncompressIGZIP return-code switch.

2. `compress2` and `uncompress2` use different completeness heuristics: Low

zlib_accel.cpp:926 checks ret == 0 && input_len != sourceLen to detect partial compression. zlib_accel.cpp:1028 mirrors this for uncompress2 but checks ret == 0 && !end_of_stream. Both paths are single-shot (no retry loop), matching the IAA/QAT stateless behavior, that's fine, but the two checks are testing different things for the same class of failure. Consider unifying or at least adding a comment.

3. Stateless `compress2` / `uncompress2` do not honor `IGZIP_FALLBACK`: Low

Streaming deflate() / inflate() have IAA/QAT → IGZIP fallback gated on configs[IGZIP_FALLBACK] (zlib_accel.cpp:421, :736). The stateless paths (zlib_accel.cpp:900-932, :998-1034) pick exactly one accelerator via if/else-if and never retry with IGZIP on failure. If IAA is preferred but fails, the caller falls straight to zlib, silently skipping IGZIP even when IGZIP_FALLBACK=1. Either wire fallback into the stateless paths or document the divergence.

4. `gzread` / `gzwrite` reverse the accelerator preference: Low (pre-existing, but propagates)

compress2 / uncompress2 / deflate / inflate: IAA first, then QAT, then IGZIP.
GzwriteAcceleratorCompress (zlib_accel.cpp:1289) / GzreadAcceleratorUncompress (zlib_accel.cpp:1363): QAT first, then IAA, then IGZIP.

The QAT-first ordering pre-dates this PR (git blame confirms), but the PR wires IGZIP in as a third-choice fallback in both orderings without noting the inconsistency. Worth documenting or unifying.

Dead / Unused Code

5. `crc32` / `adler32` are dead code: Low (delete)

igzip.cpp:265-272 defines crc32() and adler32() at global scope, routed to ISA-L equivalents.

Nothing in the project calls either function (grep-verified).
They are not exported: the project builds with -fvisibility=hidden (common.cmake:82), and only zlib_accel.h has #pragma GCC visibility push(default). nm -D on the built .so shows no crc32/adler32 in the dynamic symbol table.
zlib_accel.cpp does not load crc32/adler32 via dlsym: there are no orig_crc32/orig_adler32 pointers.

So there is no LD_PRELOAD shadowing hazard, but the functions are pure dead code. Delete them and the #include "crc.h" at igzip.cpp:11.

6. Unused struct typedefs in `igzip.h`: Low

igzip.h:10-15: internal_state2 / inflate_state2 are declared but never referenced anywhere.
igzip.h:17-22: deflate_state is declared but never referenced. Also, deflate_state is a well-known zlib-internal name; even if it were used, the collision would be worth avoiding.

Delete both.

7. Dead `method = 0` assignments: Low

zlib_accel.cpp:401 and :428 set deflate_settings->method = 0 before calling InitCompressIGZIP. DeflateSettings::method is written in the constructor (line 190) and at these two sites, and read nowhere. Remove.

8. Dead parameter assignments in `CompressIGZIP` / `UncompressIGZIP`: Low

igzip.cpp:193-194:

input = isal_strm->next_in;
output = isal_strm->next_out;

and igzip.cpp:417-418:

input = isal_strm_inflate->next_in;
output = isal_strm_inflate->next_out;

Both assign to by-value function parameters that are never read after. Dead stores: delete.

9. `IGZIPHandleActiveStreamNoInput` return `IGZIP_NO_INPUT_NOT_HANDLED` is unreachable: Low

igzip.cpp:301 returns IGZIP_NO_INPUT_NOT_HANDLED only when strm->avail_in != 0 or on null args. The sole caller (zlib_accel.cpp:606) already guards strm->avail_in == 0 before calling, so that return value is effectively dead. Either simplify the function to bool / void or drop the caller guard.

10. Commented-out `#define` in header: Low

igzip.h:61: // #define Z_DEFAULT_COMPRESSION 6. Delete.

Style / Consistency

11. `LINE` in log messages: Low

Log calls throughout igzip.cpp include __LINE__ as a positional argument, e.g. Log(..., "InitCompressIGZIP() Line ", __LINE__, ...). iaa.cpp, qat.cpp, and most of zlib_accel.cpp don't do this. Line numbers rot with every edit and the logs are noisier than the rest of the codebase. Match the existing style.

12. Hardcoded `windowBits` = 15 / 31 in stateless paths: Low

zlib_accel.cpp:917, :1018, :1316, :1390 (compress2, uncompress2, gzip accelerator helpers) pass bare 15 or 31 to InitCompressIGZIP / InitUncompressIGZIP. Values are correct (15 = zlib format, 31 = gzip format), but they should be named constants. Note: the same magic numbers already appear in the IAA/QAT branches (SupportedOptionsIAA(15, ...), CompressQAT(..., 31, ...), etc.): if cleaned up, do all three backends in one pass.

Test Coverage

IGZIP tests are gated #ifdef USE_IGZIP and CI does not build with USE_IGZIP=ON (.github/workflows/* matrix builds the zlib-only path). Consider adding a CI job stub for when hardware/library is available, or at least a README note.
compress2 / uncompress2 IGZIP paths are covered: ZlibCompressUtility2 / ZlibUncompressUtility2 (tests/zlib_accel_test.cpp:153-189) wrap them and are exercised by the parameterized matrix with IGZIP as one of the paths (tests/zlib_accel_test.cpp:299, :333). No gap here.

Summary

#	Severity	File:Line	Issue
1	Medium	`igzip.cpp:424`	Missing `Z_DATA_ERROR` mapping; makes raw-trailer fallback branch dead code
2	Low	`zlib_accel.cpp:926, 1028`	`compress2` vs `uncompress2` completeness checks diverge
3	Low	`zlib_accel.cpp:900-932, 998-1034`	Stateless paths ignore `IGZIP_FALLBACK`
4	Low	`zlib_accel.cpp:1289, 1363`	`gz*` paths reverse QAT/IAA preference vs other paths
5	Low	`igzip.cpp:265-272, :11`	`crc32`/`adler32` and `crc.h` include are dead code
6	Low	`igzip.h:10-15, :17-22`	Unused `inflate_state2` / `deflate_state` typedefs
7	Low	`zlib_accel.cpp:401, :428`	Dead `deflate_settings->method = 0` writes
8	Low	`igzip.cpp:193-194, :417-418`	Dead parameter assignments
9	Low	`igzip.cpp:301` / `zlib_accel.cpp:606`	`IGZIP_NO_INPUT_NOT_HANDLED` unreachable
10	Low	`igzip.h:61`	Commented-out `#define`
11	Low	`igzip.cpp` (throughout)	`__LINE__` log style diverges
12	Low	`zlib_accel.cpp:917, :1018, :1316, :1390`	Magic 15/31 for `windowBits`

Blocking: #1 (silently disables the raw-trailer fallback). Everything else is cleanup and can be addressed in a follow-up if preferred.

matt-welch · 2026-07-21T20:43:50Z

+      ret = CompressIGZIP(isal_strm, Z_FINISH, const_cast<uint8_t*>(source),
+                          &input_len, dest, &output_len, &total_in, &total_out);
+      EndCompressIGZIP(isal_strm);
+      if (ret == 0 && input_len != sourceLen) {


compress2 and uncompress2 use different completeness heuristics — Low

zlib_accel.cpp:926 checks ret == 0 && input_len != sourceLen to detect partial compression. zlib_accel.cpp:1028 mirrors this for uncompress2 but checks ret == 0 && !end_of_stream. Both paths are single-shot (no retry loop), matching the IAA/QAT stateless behavior — that's fine — but the two checks are testing different things for the same class of failure. Consider unifying or at least adding a comment.

matt-welch · 2026-07-21T20:48:55Z

+    *end_of_stream = (isal_strm_inflate->block_state == ISAL_BLOCK_FINISH);
+  }
+
+  int ret = 1;


UncompressIGZIP returns 1 instead of Z_DATA_ERROR on ISA-L errors: Medium

igzip.cpp:424 initializes ret = 1 and only reassigns it for ISAL_DECOMP_OK, ISAL_END_INPUT, and ISAL_NEED_DICT. Any other decomp code (e.g. ISAL_INVALID_BLOCK, ISAL_INVALID_SYMBOL, ISAL_INVALID_LOOKBACK) leaves ret == 1: not a valid zlib return code, and the log message reads "inflate finished with error code 1".

Downstream consequence: IGZIPRunInflateAndSelectPathAction at igzip.cpp:365-366 explicitly branches on *ret == Z_DATA_ERROR to return IGZIP_INFLATE_PATH_FALLBACK_DATA_ERROR. Because UncompressIGZIP never returns Z_DATA_ERROR from a real ISA-L error, that branch is only reachable via the null-arg guards at lines 339/349: the intended "raw trailer fallback" path is effectively dead code.

Fix: map ISAL_INVALID_BLOCK/ISAL_INVALID_SYMBOL/ISAL_INVALID_LOOKBACK/default → Z_DATA_ERROR in the UncompressIGZIP return-code switch.

matt-welch · 2026-07-21T20:51:14Z

+#include <igzip_lib.h>
+#include <zlib.h>
+
+typedef struct internal_state2 {


Unused struct typedefs in igzip.h — Low

igzip.h:10-15 — internal_state2 / inflate_state2 are declared but never referenced anywhere.

igzip.h:17-22 — deflate_state is declared but never referenced. Also, deflate_state is a well-known zlib-internal name; even if it were used, the collision would be worth avoiding.

Delete both.

matt-welch · 2026-07-21T21:55:15Z

+  return Z_OK;
+}
+
+unsigned long crc32(unsigned long crc, const unsigned char *buf,


crc32 / adler32 are dead code: Low (delete)

igzip.cpp:265-272 defines crc32() and adler32() at global scope, routed to ISA-L equivalents.

Nothing in the project calls either function (grep-verified).

They are not exported: the project builds with -fvisibility=hidden (common.cmake:82), and only zlib_accel.h has #pragma GCC visibility push(default). nm -D on the built .so shows no crc32/adler32 in the dynamic symbol table.

zlib_accel.cpp does not load crc32/adler32 via dlsym: there are no orig_crc32/orig_adler32 pointers.

So there is no LD_PRELOAD shadowing hazard, but the functions are pure dead code. Delete them and the #include "crc.h" at igzip.cpp:11.

matt-welch · 2026-07-21T21:56:34Z

+    } else if (path_selected == IGZIP) {
+#ifdef USE_IGZIP
+      if (deflate_settings->isal_strm == nullptr) {
+        deflate_settings->method = 0;


Dead method = 0 assignments — Low

zlib_accel.cpp:401 and :428 set deflate_settings->method = 0 before calling InitCompressIGZIP. DeflateSettings::method is written in the constructor (line 190) and at these two sites, and read nowhere. Remove.

matt-welch · 2026-07-21T21:57:03Z

+
+  *output_length = original_avail_out - isal_strm->avail_out;
+  *input_length = original_avail_in - isal_strm->avail_in;
+  input = isal_strm->next_in;


Dead parameter assignments in CompressIGZIP / UncompressIGZIP: Low

igzip.cpp:193-194:

input = isal_strm->next_in; output = isal_strm->next_out;

and igzip.cpp:417-418:

input = isal_strm_inflate->next_in; output = isal_strm_inflate->next_out;

Both assign to by-value function parameters that are never read after. Dead stores: delete.

matt-welch · 2026-07-21T21:57:34Z

+
+IGZIPNoInputAction IGZIPHandleActiveStreamNoInput(
+    z_streamp strm, struct inflate_state *isal_strm_inflate, int *ret) {
+  if (strm == nullptr || isal_strm_inflate == nullptr || ret == nullptr ||


IGZIPHandleActiveStreamNoInput return IGZIP_NO_INPUT_NOT_HANDLED is unreachable: Low

igzip.cpp:301 returns IGZIP_NO_INPUT_NOT_HANDLED only when strm->avail_in != 0 or on null args. The sole caller (zlib_accel.cpp:606) already guards strm->avail_in == 0 before calling, so that return value is effectively dead. Either simplify the function to bool / void or drop the caller guard.

matt-welch · 2026-07-21T21:58:01Z

+                    const unsigned long *total_out, bool *end_of_stream);
+int EndUncompressIGZIP(struct inflate_state *isal_strm_inflate);
+int ResetUncompressIGZIP(struct inflate_state *isal_strm_inflate);
+// #define Z_DEFAULT_COMPRESSION 6


Commented-out #define in header — Low

igzip.h:61: // #define Z_DEFAULT_COMPRESSION 6. Delete.

Blocking fix: - UncompressIGZIP: change 'int ret = 1' default to Z_DATA_ERROR so that ISAL_INVALID_BLOCK/SYMBOL/LOOKBACK and any other ISA-L error code correctly maps to Z_DATA_ERROR, enabling the raw-trailer fallback path in IGZIPRunInflateAndSelectPathAction. Dead code removal: - igzip.cpp: remove dead crc32()/adler32() functions and #include "crc.h" (not exported, not called anywhere in the project) - igzip.h: remove unused internal_state2/inflate_state2/deflate_state typedefs - igzip.h: remove commented-out #define Z_DEFAULT_COMPRESSION 6 - zlib_accel.cpp: remove dead deflate_settings->method = 0 assignments (DeflateSettings::method is never read) - igzip.cpp: remove dead post-call param assignments in CompressIGZIP and UncompressIGZIP (by-value params not read after assignment) Style / consistency: - igzip.cpp: remove __LINE__ from all 38 log calls to match iaa.cpp/qat.cpp style Correctness documentation: - IGZIPHandleActiveStreamNoInput: drop redundant strm->avail_in != 0 guard (caller already ensures avail_in == 0); add comment explaining that the total_out update here and the main inflate() update block are mutually exclusive (early return prevents double-counting) - compress2 / uncompress2: add comments explaining the intentional divergence in completeness checks (input_len vs end_of_stream) and that stateless paths do not honor IGZIP_FALLBACK by design

mulugetam · 2026-07-23T07:07:56Z

Looks like there are a few bugs that predate this PR. I'll open a separate PR to address them.

matt-welch

PR #53 Review Summary — HEAD `e814856`

Actionable punch list from the second-pass review, verified against the current tree. More per-finding detail is available if needed.

Must fix

(3) — compress2() returns Z_OK on truncated output · zlib_accel.cpp:929

The check input_len != sourceLen verifies only input consumption. If avail_out runs out while ISA-L emits the trailer, isal_deflate returns COMP_OK with all input consumed, so the check passes and compress2 reports success on a truncated stream. Mirror uncompress2()'s !end_of_stream guard:

if (ret == 0 && (input_len != sourceLen ||
                 !IsIGZIPDeflateFinished(isal_strm))) {
  ret = 1;
}

Empirically reproduced. 64 bytes of 'A' → correct compressed size 11 bytes; calling compress2() with destLen=10 returns Z_OK but the output fails uncompress() with Z_DATA_ERROR. Reproducer: IGZIPCompress2RegressionTest.Compress2ReturnsZOkForTruncatedOutput in tests/zlib_accel_test.cpp.

Should fix

(1) — gz->path = IGZIP set even when init failed · zlib_accel.cpp:1338, 1413

Move the assignment inside the success else branch. Second-pass review originally tagged this "Must fix"; downgraded here — gz->path is only consulted as != ZLIB in outer gzwrite/gzread logic, so the effect is a benign retry, not state corruption.

(2) — implicit null tolerance in deflate() IGZIP init · zlib_accel.cpp:401–413, 426–441

CompressIGZIP currently handles a null isal_strm in its own guard. Add an explicit skip to the zlib fallback path when InitCompressIGZIP() returns nullptr.

(4) — IGZIP_NO_INPUT_NOT_HANDLED is dead · igzip.cpp:278, igzip.h:17, caller zlib_accel.cpp:610

The only path returning this value is a null-arg guard whose preconditions the caller already ensures. Simplify IGZIPHandleActiveStreamNoInput to a bool return and drop the caller's if (action == IGZIP_NO_INPUT_RETURN) check.

(5) — unreachable Z_STREAM_END log in CompressIGZIP · igzip.cpp:200–202

ret is set to Z_OK (0) or Z_STREAM_ERROR (−2), never Z_STREAM_END (1). Delete the branch.

(6) — same unreachable branch in UncompressIGZIP · igzip.cpp:411–413

ret is 0, Z_NEED_DICT (2), or Z_DATA_ERROR (−3), never Z_STREAM_END (1). Delete the branch.

Nice to have

(8) — unused strm / reason params · zlib_accel.cpp:245–263

SetDeflatePath / SetInflatePath cast both parameters to (void). Either wire them into the log messages or drop from the signature.

(9) — magic 15 / 31 windowBits literals · 12 call sites in zlib_accel.cpp

Replace with named constants (kWindowBitsZlib = 15, kWindowBitsGzip = 31) across all three backends in one pass.

Skip

(7) — add SupportedOptionsIGZIP() predicate

Proposed as a stub returning true for symmetry with the IAA/QAT gates. A no-op stub adds no filtering; the existing explicit configs[USE_IGZIP_COMPRESS] check is clearer. Skip unless a real gate (min input size, unsupported-format detection) is defined.

Must fix: - compress2: check IsIGZIPDeflateFinished() before EndCompressIGZIP() to detect truncated output when destLen is too small for the ISA-L trailer; input_len==sourceLen alone is insufficient since ISA-L returns COMP_OK with all input consumed but stream not in ZSTATE_END - compress2/uncompress2: simplify compress2 completeness check to !IsIGZIPDeflateFinished (subsumes input_len!=sourceLen); both paths now use a single terminal-state guard, symmetric with uncompress2 end_of_stream Should fix: - deflate(): wrap CompressIGZIP calls in null guard after InitCompressIGZIP; makes fallthrough-to-zlib explicit on init failure (both direct IGZIP path and IAA/QAT->IGZIP fallback path) - IGZIPHandleActiveStreamNoInput: change return type to void, remove dead IGZIPNoInputAction enum; caller unconditionally returns after the call - CompressIGZIP: remove unreachable else-if(ret==Z_STREAM_END) log branch; ret is only ever Z_OK or Z_STREAM_ERROR - UncompressIGZIP: same — remove unreachable else-if(ret==Z_STREAM_END) branch; ret is Z_OK, Z_NEED_DICT, or Z_DATA_ERROR Note: gz->path=IGZIP inside-else fix (should-fix #1) was already addressed in the previous commit (1874a71); no change needed here. Test: IGZIPDeflateRegressionTest.Compress2MustDetectTruncatedOutput added; all 56 IGZIP/config/logging/sharded-map tests pass.

matt-welch

PR 53 review - `1c628f9`

The earlier issues (null-deref in inflateReset, compress2 truncation, raw-deflate over-consumption, error code mapping) are all fixed. Three things still need attention before merge.

Bugs

(1) `GzwriteAcceleratorCompress` missing stream-completion check (`zlib_accel.cpp:1316-1319`)

compress2 was fixed to call IsIGZIPDeflateFinished before reporting success, but GzwriteAcceleratorCompress wasn't:

// zlib_accel.cpp:1316-1319
ret = CompressIGZIP(isal_strm, Z_FINISH, input, input_length, output,
                    output_length, &total_in, &total_out);
EndCompressIGZIP(isal_strm);
gz->path = IGZIP;   // set even when stream did not finish

If *output_length is too small to hold the ISA-L trailer, isal_deflate returns COMP_OK with all input consumed (ret = 0) but the stream hasn't reached ZSTATE_END. The function returns success with a truncated gzip payload. The same issue is in
GzreadAcceleratorUncompress at lines 1392-1396 where gz->path = IGZIP is set unconditionally regardless of whether init succeeded.

The fix is the same guard already used in compress2 at line 906:

if (ret == 0 && !IsIGZIPDeflateFinished(isal_strm)) {
    ret = 1;
}

(2) `windowBits` 40-47 mapped to `IGZIP_GZIP` (`igzip.cpp:49-54`)

if ((windowBits >= 24 && windowBits <= 31) ||
    (windowBits >= 40 && windowBits <= 47)) {
  isal_strm_inflate->crc_flag = IGZIP_GZIP;

zlib's 40-47 range (32 + 8 to 32 + 15) means auto-detect: accept either zlib or gzip headers. ISA-L has no auto-detect mode, so mapping this range to IGZIP_GZIP silently breaks decompression of zlib-format data passed with those window bits. These
values should fall back to ZLIB rather than mishandling them.

(3) `deflateCopy`/`inflateCopy` not intercepted

If a caller copies a stream on the IGZIP path, the DeflateSettings/InflateSettings copy ends up with the same isal_strm pointer as the original. Both will call EndCompressIGZIP/EndUncompressIGZIP on that pointer: double-free on one, dangling po
inter on the other.

zlib_accel.cpp intercepts deflateInit, deflateEnd, deflateReset, and deflateSetDictionary but not deflateCopy or inflateCopy. The fix is to intercept both and allocate a fresh ISA-L stream for the copy (re-initializing from window_bits an
d level), or at minimum clear the IGZIP state on the copy and reset its path to UNDEFINED so it re-initializes on the next call.

Minor

#include <pthread.h> (zlib_accel.cpp:8) - nothing in this file uses pthread symbols. Remove it.

path_name in log statements (zlib_accel.cpp:320-321, 476, 495, 576, 778, 803) - path and path_name log the same integer value. Either make path_name a readable string or drop the duplicate field.

(void)total_in in UncompressIGZIP (igzip.cpp:357) - CompressIGZIP sets isal_strm->total_in but UncompressIGZIP drops it because inflate_state has no public total_in field. Worth a one-line comment so it doesn't look accidental.

Commented-out gtest filter (tests/CMakeLists.txt:27) - # --gtest_filter=ConfigLoaderTest.LoadValidConfig shouldn't be committed.

CMake indentation (common.cmake, USE_IGZIP link block) - the if(NOT DEFINED ISAL_PATH) / else() block mixes tabs and spaces, making else() look visually misaligned. Should match the 2-space style used elsewhere in the file.

Fix windowBits handling on the IGZIP inflate path. ConfigureInflateWindow mapped only 24..31 to gzip, so windowBits 16 (gzip-only, max window) fell through to the zlib branch and IGZIP decoded zlib data that zlib itself rejects with Z_DATA_ERROR -- the shim was more permissive than the library it replaces. The gzip range is now 16..31 with hist_bits = windowBits - 16 (16 maps to ISA-L's hist_bits == 0, its spelling for maximum history). Add SupportedOptionsIGZIPInflate() to refuse windowBits >= 32, which requests zlib/gzip auto-detection that ISA-L cannot express. IGZIP was the only backend with no windowBits gate; QAT, IAA and GetCompressedFormat all restrict the range already. The 32..47 range was not incorrect before -- IGZIP failed at the header and fell back cleanly -- but it burned a failed attempt on every zlib stream. Guard GzwriteAcceleratorCompress with IsIGZIPDeflateFinished(). The stream is ended right after, so a gzip member left mid-emit can never be completed, and isal_deflate reports COMP_OK for "made progress". Not reachable today since io_buf is twice data_buf, but that coupling was undocumented and unenforced; on failure CompressAndWrite now recompresses with zlib instead of writing a truncated member. Minor review items: drop unused pthread.h include; add ExecutionPathName() so the duplicate path_name log field prints a name rather than an int; explain the (void)total_in cast in UncompressIGZIP (inflate_state has no total_in counterpart); drop the commented-out gtest filter; replace tabs in common.cmake's USE_IGZIP block. Test: added tests to validate windowBits handling on the IGZIP inflate path. Signed-off-by: Olasoji <olasoji.denloye@intel.com>

asonje · 2026-07-28T23:03:49Z

(1) GzwriteAcceleratorCompress missing stream-completion check (zlib_accel.cpp:1316-1319)

fixed for now. the underlying stream management needs to be addressed.

asonje · 2026-07-28T23:09:44Z

(2) windowBits 40-47 mapped to IGZIP_GZIP (igzip.cpp:49-54)

fixed, but the bug isn't in 40–47; IGZIP_GZIP on zlib-wrapped data fails at the header, nothing is consumed or produced and cleanly falls back to zlib. the real bug was with windowBits =16 which was failing silently

asonje · 2026-07-28T23:11:11Z

(3) deflateCopy/inflateCopy not intercepted

This issue is real but beyond the scope of this PR as it affects all engines on all branches. will fix

Adds IGZIP as a third offload backend alongside QAT and IAA, built on ISA-L's isal_zstream/inflate_state. Unlike QAT and IAA, IGZIP supports true stateful streaming, so it also serves as a software fallback when an accelerator call fails (igzip_fallback, default on). - igzip.cpp/.h: ISA-L stream lifecycle, window/format mapping, and the stateful inflate path decisions (IGZIPInflatePathAction). - zlib_accel.cpp: IGZIP dispatch in deflate()/inflate(), stream stickiness once an ISA-L stream is active, separate return-code translation from the stateless QAT/IAA paths, and accelerator -> IGZIP -> zlib fallback ordering. - deflateSetDictionary/inflateSetDictionary now reject a mid-stream dictionary with Z_STREAM_ERROR and only pin the stream to ZLIB when the underlying zlib call succeeds; no backend supports preset dictionaries. - config: add igzip_fallback, use_igzip_compress, use_igzip_uncompress. - Requires ISA-L >= 2.32.1; earlier versions have igzip bugs whose workarounds were deliberately removed from this tree. - ~2000 lines of regression tests covering IGZIP inflate/deflate return-code translation, reset/dictionary handling, and fallback. Rebased onto main, which changed ShardedMap::Get() to return a shared_ptr (#64): the per-stream settings locals and the SetDeflatePath/SetInflatePath helpers now take shared_ptr, keeping that change's lifetime guarantees on the IGZIP paths. Verified on a QAT+IAA machine: 51015 tests, 44775 passed, 6240 hardware-skipped, 0 failed - identical to the pre-rebase branch. Builds clean for every backend permutation and the DEBUG_LOG x ENABLE_STATISTICS cross product. Signed-off-by: Olasoji <olasoji.denloye@intel.com>

asonje self-assigned this Jul 15, 2026

asonje marked this pull request as ready for review July 15, 2026 18:44

asonje requested a review from Copilot July 16, 2026 17:25

Copilot started reviewing on behalf of asonje July 16, 2026 17:26 View session

Copilot AI reviewed Jul 16, 2026

View reviewed changes

Comment thread zlib_accel.cpp Outdated

Comment thread zlib_accel.cpp Outdated

Comment thread zlib_accel.cpp

Comment thread igzip.cpp Outdated

Comment thread igzip.cpp Outdated

Comment thread igzip.cpp Outdated

Comment thread common.cmake Outdated

asonje requested review from matt-welch and mulugetam July 16, 2026 22:55

matt-welch requested changes Jul 21, 2026

View reviewed changes

asonje requested a review from matt-welch July 22, 2026 18:49

matt-welch requested changes Jul 23, 2026

View reviewed changes

asonje requested a review from matt-welch July 23, 2026 20:57

matt-welch requested changes Jul 28, 2026

View reviewed changes

asonje force-pushed the igzip branch from 78babfe to aa8f004 Compare July 31, 2026 20:48

asonje requested a review from matt-welch July 31, 2026 20:53

Uh oh!

Conversation

asonje commented Mar 13, 2026

Summary

Scope (Initial Push)

Purpose

Tracking

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

matt-welch left a comment

Choose a reason for hiding this comment

PR #53 Review: IGzip Integration

Correctness

1. UncompressIGZIP returns 1 instead of Z_DATA_ERROR on ISA-L errors : Medium

2. compress2 and uncompress2 use different completeness heuristics: Low

3. Stateless compress2 / uncompress2 do not honor IGZIP_FALLBACK: Low

4. gzread / gzwrite reverse the accelerator preference: Low (pre-existing, but propagates)

Dead / Unused Code

5. crc32 / adler32 are dead code: Low (delete)

6. Unused struct typedefs in igzip.h: Low

7. Dead method = 0 assignments: Low

8. Dead parameter assignments in CompressIGZIP / UncompressIGZIP: Low

9. IGZIPHandleActiveStreamNoInput return IGZIP_NO_INPUT_NOT_HANDLED is unreachable: Low

10. Commented-out #define in header: Low

Style / Consistency

11. __LINE__ in log messages: Low

12. Hardcoded windowBits = 15 / 31 in stateless paths: Low

Test Coverage

Summary

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mulugetam commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

matt-welch left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

PR #53 Review Summary — HEAD e814856

Must fix

Should fix

Nice to have

Skip

Uh oh!

matt-welch left a comment

Choose a reason for hiding this comment

PR 53 review - 1c628f9

Bugs

(1) GzwriteAcceleratorCompress missing stream-completion check (zlib_accel.cpp:1316-1319)

(2) windowBits 40-47 mapped to IGZIP_GZIP (igzip.cpp:49-54)

(3) deflateCopy/inflateCopy not intercepted

Minor

Uh oh!

1. `UncompressIGZIP` returns `1` instead of `Z_DATA_ERROR` on ISA-L errors : Medium

2. `compress2` and `uncompress2` use different completeness heuristics: Low

3. Stateless `compress2` / `uncompress2` do not honor `IGZIP_FALLBACK`: Low

4. `gzread` / `gzwrite` reverse the accelerator preference: Low (pre-existing, but propagates)

5. `crc32` / `adler32` are dead code: Low (delete)

6. Unused struct typedefs in `igzip.h`: Low

7. Dead `method = 0` assignments: Low

8. Dead parameter assignments in `CompressIGZIP` / `UncompressIGZIP`: Low

9. `IGZIPHandleActiveStreamNoInput` return `IGZIP_NO_INPUT_NOT_HANDLED` is unreachable: Low

10. Commented-out `#define` in header: Low

11. `LINE` in log messages: Low

12. Hardcoded `windowBits` = 15 / 31 in stateless paths: Low

mulugetam commented Jul 23, 2026 •

edited

Loading

matt-welch left a comment •

edited

Loading

PR #53 Review Summary — HEAD `e814856`

PR 53 review - `1c628f9`

(1) `GzwriteAcceleratorCompress` missing stream-completion check (`zlib_accel.cpp:1316-1319`)

(2) `windowBits` 40-47 mapped to `IGZIP_GZIP` (`igzip.cpp:49-54`)

(3) `deflateCopy`/`inflateCopy` not intercepted

asonje commented Jul 28, 2026 •

edited

Loading