save 32 bit psd to openexr

[singhaidotnish]

Hi,

        I can see from this post that openimageio currently supports 8-bit psd to exr save. Because it does work for 8 bit, i can safely presume it can work for 32 bit psd as well. Is it possible. how ?

https://stackoverflow.com/questions/53009553/export-psd-layers-to-exr-in-python

This is actually relatively straightforward, however there is one caveat in that it only seems to be supported for 8-bit psd files at the moment.

Being not absolute but novice enough to be able to compile a modified code, may I ask where can the changes start in the source code.
I don't need exact answer but a healthy discussion which gives a direction to this discusison will certainly help.

Thank you

[lgritz]

I don't think that comment is correct. It looks like our PSD reading code supports 8 and 16 bit int as well as float.

[singhaidotnish]

Thank you for your reply. Could you provide code that will convert 16 bit psd to exr. I tried this example replacing sourcefile and outputfile with valid paths. It does not give any error with no output at same time. Finally if one has to work with 32 bit psd files ( int or float ), which cpp file should i make changes to and compile the code.

import OpenImageIO as oiio

sourcefile = '/path/to/sourcefile.psd'
buf = oiio.ImageBuf(sourcefile)

for layer in range(buf.nsubimages):
    buf.reset(sourcefile, subimage=layer)
    buf.write('/tmp/mylayer_{l}.exr'.format(l=layer))

[lgritz]

Before working our way through this puzzle, let's first establish if you're making this needlessly complicated.

Do you need to do this from Python specifically, and do you really need to write out each layer as a separate image?

Is there a reason you don't just use the command line utility, oiiotool?

[kpolonski]

I believe the issue you're experiencing has to do with the bug report I made a couple weeks ago related to nsubimages with different bit depth photoshop files - #3900

ImageBuf (nor oiiotool) recognize the subimages for non-8-bit psd files.

uint8:

>>> buf = oiio.ImageBuf("/path/to/color_squares_small_8bit.psd")
>>> buf.nsubimages
5

uint16:

>>> buf = oiio.ImageBuf("/path/to/color_squares_small_16bit.psd")
>>> buf.nsubimages
1

[singhaidotnish]

Only to view the output clearly that an exr is required.
oiiotool it is.

Thank you, Igritz

[lgritz]

The simple oiiotool approach would simply be

oiiotool color_squares_small_8bit.psd -d half -o out.exr

And see if that works, or if not, what happens and what errors it gives you.

I don't see anything wrong with your python per se, except that you aren't checking any results or printing any errors that occurred. Without knowing if any of those steps failed or what errors it encountered, I don't even know how to guess what to suggest next. At least oiiotool will gather and print the errors for you, so it might be faster to try that and see what errors it prints, then we can understand the situation better.

[EmilDohne]

I'd like to reopen the issue to perhaps give @lgritz some more information since there hasnt been any traction on this lately. The issue is two-fold and will require an extension of the photoshop file format parsing capabilities. To give some background on me, I recently wrote a PhotoshopAPI for doing exactly this sort of stuff.

Issue 1, parsing of 16- and 32-bit files:

It appears that the internal implementation of parsing files doesnt actually work on 16- and 32- bit files due to some quirks in the photoshop file format. I noticed that there is tests on files with this bit-depth but im not sure about the inner workings enough to actually know what it is testing, if it is just the parsing or if it asserts the output too.

Either way, the Photoshop file format doesnt store 16- and 32-bit data in the LayerInfo struct but instead on a Tagged Block called 'Lr16' and 'Lr32' respectively which is found in the Additional layer info section of the Layer and Mask information while the actual LayerInfo's length is set to 0. Currently, if that is the case the section simply gets skipped which results in no amount of subimages being found.

I've highlighted the parts of the code where this happens, this also explains why there is no errors being raised by OIIO on read as it simply thinks its dealing with an empty file (which isnt legal in Photoshop):

PSDInput::load_layers()
{
    // ....

    LayerMaskInfo::LayerInfo& layer_info = m_layer_mask_info.layer_info;
    if (m_header.version == 1)
        ok &= read_bige<uint32_t>(layer_info.length);
    else
        ok &= read_bige<uint64_t>(layer_info.length);

    layer_info.begin = iotell();
    layer_info.end   = layer_info.begin + layer_info.length;
    if (!ok)
        return false;

+    if (!layer_info.length)
+       return true;

    // ...
}

Code taken from this file

To maybe more visually show the structure of a 16/32 bit files' Layer and Mask information section I made a table on how it is stored on disk

Layer and mask information section

Size	Type	Variable	Description
4/8	uint32_t/uint64_t	Section Size	Layer and Mask Information section size
4/8	uint32_t/uint64_t	Section Size	Layer Info section size, is always 0 in 16- and 32-bit files
4	uint32_t	Section Size	Global layer mask info (always zero in my experience)
			Additional Layer info below, may have tagged blocks before Lr16/Lr32 block
			...
4	char[]	Signature	must be '8BIM' or '8B64'
4	char[]	Key	'Lr16' or 'Lr32'
4/8	uint32_t/uint64_t	Block length	This replaces the Layer Info size marker
...	Data	Layer Info	Start of the regular layer info section, excluding the Layer info size marker! (starts with layer count)
			...

Any / in the size indicate that if it is a psb we read the wider of the two types

In order to actually add support for 16- and 32-bit files there should be some kind of seek operation which looks for the Lr16/Lr32 tagged blocks and reads the layer info from there as the underlying format of how these layer info sections are stored are exactly identical to how it is usually stored.

To reproduce the issue mentioned by @singhaidotnish and @kpolonski I made this basic code snippet using ImageInput rather than ImageBuf which for the 8-bit file will correctly iterate all the layers and the merged Image data while for the 16_bit file it will only extract the merged ImageData

std::filesystem::path filePath = "Layers_16bit.psd";
std::filesystem::path filePath = "Layers_8bit.psd";

auto inp = OIIO::ImageInput::open(filePath.string());
int subimage = 0;
while (inp->seek_subimage(subimage, 0)) {
     const OIIO::ImageSpec& spec = inp->spec();
     int npixels = spec.width * spec.height;
     int nchannels = spec.nchannels;
     auto pixels = std::vector<unsigned char>(npixels * nchannels, 0);
     inp->read_image(subimage, 0, 0, nchannels, OIIO::TypeDesc::UINT8, pixels.data());

     std::cout << subimage << std::endl;
     ++subimage;
}

Issue 2, compression codecs:

Now that we have examined why the layers are not being recognized to begin with, this still wouldnt work as 16- and 32-bit files do not use Raw or RLE compression but instead use ZipPrediction. Below you can find a table of what compression codecs (from my testing) are used per bit-depth

• 8-bit: RAW or RLE depending on which results in a smaller size
• 16-bit: ZIPPrediction
• 16-bit: ZIPPrediction

We can see in the load_layer_channel() function as well as the read_native_scanline() function that Zip and ZipPrediction are not supported

bool
PSDInput::load_layer_channel(Layer& layer, ChannelInfo& channel_info)
{
    // ...
    switch (channel_info.compression) {
    case Compression_Raw:
        // ...
        break;
    case Compression_RLE:
        // ....
        break;
+   // These two aren't currently supported. They would likely
+   // require large changes in the code as they probably don't
+   // support random access like the other modes. I doubt these are
+   // used much and I haven't found any test images.
+    case Compression_ZIP:
+    case Compression_ZIP_Predict:
    default:
        errorfmt("[Layer Channel] unsupported compression {}",
                 channel_info.compression);
        return false;
}

While the actual implementation isnt that complex¹, the assumption in the comment above holds true that it is not possible to access scanlines individually in random order. This is because, unlike with RLE compression, the channel buffer is compressed as a whole rather than per-scanline making per-scanline access impossible. The channel would have to be decompressed as a whole and then each scanline should be accessed from that decompressed buffer.

¹ caveat: for 32-bit files specifically float bytes are de-interleaved, see here on what that means

Final thoughts

It would probably be a good idea to indicate that loading of 16-/32-bit files is not currently supported. I would be happy to assist with the implementation of that feature. I do however not have much knowledge (yet) of the whole OIIO ecosystem and what the requirements of an implementation would be.

[lgritz]

@EmilDohne Is this issue fully solved by #4208? Can this one be closed?

[EmilDohne]

Yes this is now properly implemented and works as expected listing the subimages

[lgritz]

Fixed by #4208