A while ago, our friends from GIMP reported that their code reading exr one-scanline-at-a-time getting pretty slow: AcademySoftwareFoundation/openexr#1887
Kimball subsequently identified that it is because a cache in the Core code was removed in 3.4 to prevent memory bloat, which means each time a single scanline is requested, the entire chunk must be re-decoded. He subsequently re-established this chunk cache in OpenEXR's C++ layer, here:
AcademySoftwareFoundation/openexr#1899
AcademySoftwareFoundation/openexr@64f527d
However, OIIO's imageinput now directly uses Core instead of the C++ EXR, which means the above patch had no effect. We recently reproduced the slow single scanline reading via OIIO's good'ol imagespeed_test.cpp, by pining EXR 3.4 on the local build workflow. More specifically, the reproduction is the following:
- Copy and apply this commit lji-ilm@3db4385
- Build OIIO with
-DOpenImageIO_BUILD_LOCAL_DEPS=all.
- Run the executable
build/bin/imagespeed_test on any large (ex. 4K) EXR file (See speed comparison with different compression formats below).
If one skips step 1 and build OIIO with EXR 3.3, this slowdown will not appear.
On the other hand, if an user already installed EXR 3.4, then a "normal" OIIO build without a local build workflow will also reveal this performance regression between EXR 3.3 and 3.4.
Here is a visualization of how slow this single scanline read is, note that the Y axis is in log scale. The slow down is ~2 magnitudes for 4K, except for ZIPS, whose chunk is a scanline.
I have discussed this issue in the August 8 OpenEXR committee meeting. The immediate impression is that since Core does not want to re-establish this buffer, either OIIO need to implement it in its EXR plugin just like Kimball's EXR C++ side patch, or we can tell a user this is a really bad thing to do in client code. I'm writing down the notes here to start a discussion.
@cary-ilm @kdt3rd @peterhillman
A while ago, our friends from GIMP reported that their code reading exr one-scanline-at-a-time getting pretty slow: AcademySoftwareFoundation/openexr#1887
Kimball subsequently identified that it is because a cache in the Core code was removed in 3.4 to prevent memory bloat, which means each time a single scanline is requested, the entire chunk must be re-decoded. He subsequently re-established this chunk cache in OpenEXR's C++ layer, here:
AcademySoftwareFoundation/openexr#1899
AcademySoftwareFoundation/openexr@64f527d
However, OIIO's
imageinputnow directly uses Core instead of the C++ EXR, which means the above patch had no effect. We recently reproduced the slow single scanline reading via OIIO's good'olimagespeed_test.cpp, by pining EXR 3.4 on the local build workflow. More specifically, the reproduction is the following:-DOpenImageIO_BUILD_LOCAL_DEPS=all.build/bin/imagespeed_teston any large (ex. 4K) EXR file (See speed comparison with different compression formats below).If one skips step 1 and build OIIO with EXR 3.3, this slowdown will not appear.
On the other hand, if an user already installed EXR 3.4, then a "normal" OIIO build without a local build workflow will also reveal this performance regression between EXR 3.3 and 3.4.
Here is a visualization of how slow this single scanline read is, note that the Y axis is in log scale. The slow down is ~2 magnitudes for 4K, except for ZIPS, whose chunk is a scanline.
I have discussed this issue in the August 8 OpenEXR committee meeting. The immediate impression is that since Core does not want to re-establish this buffer, either OIIO need to implement it in its EXR plugin just like Kimball's EXR C++ side patch, or we can tell a user this is a really bad thing to do in client code. I'm writing down the notes here to start a discussion.
@cary-ilm @kdt3rd @peterhillman