Extend yambo spectra by Jajar26 · Pull Request #158 · FAIRmat-NFDI/nomad-parser-plugins-simulation

Jajar26 · 2026-03-17T14:54:15Z

Work in progress to share

l275-304
#start HB
def get_spectra(self) -> dict[str, Any]:
data = []
names = []

    with open(self.filepath) as f:
        for line in f:
            line = line.strip()

            if line.startswith('#') and 'E/ev' in line:
                names = [k.strip() for k in line.split() if k != '#']
                continue

            if names and not line.startswith('#') and line:
                data.append(line.split())

    if not data:
        return {}

    data = np.array(data, dtype=np.float64)

    if data.shape[1] < 2:
        return {}

    return dict(
        excitation_energies=data[:, 0] * ureg.eV,
        intensities=data[:, 1]
    )

    #end HB

l345-373

#start HB
SPECTRA_TYPE_MAP = {
'Absorption': 'dielectric_function',
'EELS': 'energy_loss_spectrum',
}

    for spectra_file in spectra_files:
        spectra_parser.filepath = spectra_file

        sp_type = mainfile_parser.data.get('sp_type')
        if sp_type is None:
            continue

        absorption_spectra_parser.data_object = yambo.outputs.AbsorptionSpectrum()

        spectra_parser.convert(absorption_spectra_parser)

        spectra_obj = absorption_spectra_parser.data_object
        spectra_obj.label = sp_type
        spectra_obj.type = SPECTRA_TYPE_MAP.get(sp_type, 'unknown')

        outputs = (
            data.outputs[-1] if data.outputs else data.m_create(yambo.outputs.Outputs)
        )
        outputs.m_append(
            yambo.outputs.AbsorptionSpectrum.m_def,
            spectra_obj,
        )
        #end HB

Refactor get_spectra method to parse data for EM and update spectra type based on sp_type.

Removed unnecessary initialization of AbsorptionSpectrum.

ladinesa · 2026-03-25T16:05:51Z

+        data = []
+        names = []
+
+        with open(self.filepath) as f:


we should introduce a new file parser class for the spectra file, we do not want to append the data to the mainfile parser.

Sorry, maybe I'm missing something.... By "introducing a new file parser class for the spectra file" do you mean adding something to the definition of the class SpectraParser in file_parsers.py? Apart from that, in parser.py we indeed have a class YamboSpectraParser (into which my colleague Hajar and I have added some lines..). Moreover, are we really appending spectra data to the mainfile parser?

Yes indeed this is what I meant. We should have a file parser and a mapping parser. What you are trying to do at the moment is parsing the spectra file and appending it to a function call within the mainfile parser. Although this could work, I would like to as much as possible separate the auxiliary files.

Ok, so, if I understand correctly, what you suggest may mean handling spectra files similarly to how netcdf files are already handled, i.e. we should 1) "populate" the SpectraParser class definition in file_parser.py, possibly putting there some parts that currently are in the YamboSpectraParser class in parser.py, 2) making the YamboSpectraParser class (in parser.py) of MappingParser type, instead of TestParser type.

Yes indeed. You should put the part that search for spectra files in YamboArchiveWriter. You then create a YamboSpectraParser (mapping parser) in the archive writer similar to the net cdf case. You should add the corresponding annotations in the schema file and use a different annotation key e.g. yambo_spectra

ladinesa · 2026-03-25T16:06:53Z

-            # add to simulation outputs.absorption_spectra
+
+            spectra_obj = absorption_spectra_parser.data_object
+            spectra_obj.label = sp_type


we should not do this here but instead add this in the annotation

ladinesa · 2026-03-25T16:07:28Z

-                else data.m_create(yambo.outputs.Outputs)
+                data.outputs[-1] if data.outputs else data.m_create(yambo.outputs.Outputs)
            )
            outputs.m_append(


we should also add this to the Outputs annotation inorder to automate this

ladinesa · 2026-03-25T16:07:59Z

        absorption_spectra_parser.annotation_key = yambo.SPECTRA_KEY
+
+        #start HB
+        SPECTRA_TYPE_MAP = {


add this to init

…ctra

ladinesa · 2026-07-07T15:21:39Z

Can you please add some test data with the spectra files?

ladinesa · 2026-07-07T15:24:20Z

-    def get_spectra(self, data: np.ndarray) -> dict[str, Any]:
-        return dict(excitation_energies=data[:, 0], intensities=data[:, 1])
+    #start HB
+    def get_spectra(self) -> dict[str, Any]:


this is already covered by the file parser I believe. DataTextParser will simply use np.loadtxt to read the file.

ladinesa · 2026-07-07T15:26:22Z

        for spectra_file in spectra_files:
            spectra_parser.filepath = spectra_file
+
+            sp_type = mainfile_parser.data.get('sp_type')


this should be spectra_parser.get('sp_type') as you defined it there.

ladinesa · 2026-07-07T15:27:48Z

+        outputs.AbsorptionSpectrum.sp_type, SPECTRA_KEY, 'sp_type'
+    )
+    add_mapping_annotation(
+        outputs.AbsorptionSpectrum.n_energies, SPECTRA_KEY, 'n_energies'


n_energies is not defined in the spectra parser.

Yes, indeed. I wish to define it in the SpectraParser class in yambo/file_parsers.py, but in fact I am not sure how to define it, since it is a derived quantity (it is shape[0] of data.outputs, or anyway of the object that contains all the (energy,intensity) pairs of values of a spectrum), not defined by matching a specific regular expression (at a difference with sp_type and many other quantities in that or other file_parsers.py).

There might be an alternative way of defining n_energies, by looking for the line containing the expression "Energy steps" in the o* file(s).

ladinesa · 2026-07-07T15:29:09Z

 class Outputs(outputs.Outputs):
    # TODO add description
-    sp_type = Quantity(type=str)
+#    sp_type = Quantity(type=str)   # EM: commented,  6 Jul, 2026


is sp_type already in the nomad-simulations Outputs?

Indeed it isn't. So, here in /schema_packages/yambo.py, should I uncomment it in the class Outputs or rather put it in the class AbsorptionSpectra?

Ah i get it now, sorry it is already in AbsorptionSpectra.

adsorption_spectra is not in nomad-simulation Outputs. so maybe add

adsorption_spectra = SubSection(sub_section=AbsorptionSpectra.m_def, repeats=True)

under class Outputs in yambo.py

Well, in fact we do have a line:
absorption_spectra = SubSection(sub_section=AbsorptionSpectrum.m_def, repeats=True)

in nomad_simulations/schema_packages/outputs.py.

But it seems we never use this "absorption_spectra" anywhere, am I wrong? I'll see if I can understand this while I fix the other issues you mentioned.

I do not see it here: https://github.com/FAIRmat-NFDI/nomad-simulations/blob/develop/src/nomad_simulations/schema_packages/outputs.py

Well, I see it there, at line 148.

my bad, I was searching for adsorption_spectra

Ok, perfect. About the misspell, I had noticed it in your messages but forgot to point it out. In this context, it is always absorption.

ladinesa · 2026-07-08T10:59:39Z

With the added test data, can you please also add the tests for the yambo parser?

emolteni · 2026-07-08T13:19:19Z

With the added test data, can you please also add the tests for the yambo parser?

I have just created a tentative version of a test_yambo_parser.py file for testing spectra parsing, trying to follow the structure of other test parsers in that folder. I am not so confident about it.....

As for my /test/data/yambo folder, instead, I had to upload its /SAVE folder zipped due to file size limits, and for the same reason the system does not allow me to unzip it there on github. The SAVE folder (which contains the ns.db file) should not be needed for spectra parsing, but its absence is expected to lead to missing parsing of atom coordinates, chemical composition, cell parameters, which in "production" parsing is unpleasant. If you think we can ignore this for the tests, then I can just delete the /SAVE folder.

ladinesa · 2026-07-08T13:26:38Z

+  archive = EntryArchive()
+  parser = YamboArchiveWriter()
+  parser.parse('tests/data/yambo/r_setup',archive,LOGGER)
+  assert spectra_files[0] = 'o-R_methylox_TDLDA.alpha_q1_slepc_alda_bse'


The assertion here should be

spectra = archive.data.outputs[0].adsorption_spectra[0] assert spectra.sp_type == 'Polarizability' ...

Done, i.e. I have put your lines in place of my previous line on checking spectra type.

I expect the previous line, which should check it finds our spectra file, is ok? There I have spectra_files[0] because, in principle, spectra_file is a list, but in the present case it contains 1 file only.

there is no spectra_file defined under AdsorptionSpectra only: value, energies, sp_typeand n_energies

ladinesa · 2026-07-08T13:28:57Z

you should also add an annotation for Outputs.m_def here with SPECTRA_KEY something like

add_mapping_annotation(Outputs.m_def, SPECTRA_KEY, '.@')

ladinesa · 2026-07-08T13:29:17Z


 add_mapping_annotation(general.Simulation.m_def, OUT_KEY, '@')
 add_mapping_annotation(general.Simulation.m_def, NETCDF_KEY, '@')



also for general.Simulation.m_def

Done (line 125), but for some reason it does not label it as "outdated" here.

ladinesa · 2026-07-08T14:13:10Z

+
+            spectra_obj = absorption_spectra_parser.data_object
+            spectra_obj.label = sp_type
+            spectra_obj.type = SPECTRA_TYPE_MAP.get(sp_type, 'unknown')


also, labeland type are not defined in AdsorptionSpectra

ladinesa · 2026-07-08T15:13:45Z

since we would like to use the new definition for AbsorptionSpectrum, this should be: AbsorptionSpectrum.m_def not outputs.AbsorptionSpectrum.m_def

…n class Outputs)

Jajar26 added 3 commits March 17, 2026 15:03

Enhance spectra parsing for EM and update type assignment

074f7d9

Refactor get_spectra method to parse data for EM and update spectra type based on sp_type.

Refactor get_spectra method and update parsing logic

42d57a3

Refactor absorption spectra parser initialization

b9d1ef0

Removed unnecessary initialization of AbsorptionSpectrum.

ladinesa reviewed Mar 25, 2026

View reviewed changes

emolteni added 4 commits April 9, 2026 17:14

Update parser.py: handle spectra files separately: part 1

07e3b61

Update file_parsers.py: moved sp_type to SpectraParser

ab9afea

Update yambo.py: changed sp_type to SPECTRA_KEY

39a9493

Update yambo.py: added sp_type, n_energies in the class AbsorptionSpe…

29c3d37

…ctra

ladinesa reviewed Jul 7, 2026

View reviewed changes

emolteni added 4 commits July 8, 2026 10:02

Update parser.py: spectra_parser.get('sp_type')

48738e7

Create tmp

94bda50

Add files via upload

3010bd9

Delete tests/data/yambo/tmp

8c8a9c5

Create test_yambo_parser.py for testing yambo spectra parsing

d38bd2d

ladinesa reviewed Jul 8, 2026

View reviewed changes

Update test_yambo_parser.py: modified the assertion on spectra type

02ba1dc

ladinesa reviewed Jul 8, 2026

View reviewed changes

Update yambo.py: added SPECTRA_KEY mapping annotations

ccddfb6

ladinesa reviewed Jul 8, 2026

View reviewed changes

Update yambo.py: outputs.AbsorptionSpectrum --> AbsorptionSpectrum (i…

60f62a9

…n class Outputs)


		add_mapping_annotation(general.Simulation.m_def, OUT_KEY, '@')
		add_mapping_annotation(general.Simulation.m_def, NETCDF_KEY, '@')

Uh oh!

Conversation

Jajar26 commented Mar 17, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emolteni Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ladinesa Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ladinesa commented Jul 7, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ladinesa Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emolteni Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ladinesa Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emolteni Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ladinesa commented Jul 8, 2026

Uh oh!

emolteni commented Jul 8, 2026

Uh oh!

ladinesa Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ladinesa Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emolteni Apr 8, 2026 •

edited

Loading

ladinesa Apr 9, 2026 •

edited

Loading

ladinesa Jul 7, 2026 •

edited

Loading

emolteni Jul 8, 2026 •

edited

Loading

ladinesa Jul 8, 2026 •

edited

Loading

emolteni Jul 8, 2026 •

edited

Loading

ladinesa Jul 8, 2026 •

edited

Loading

ladinesa Jul 8, 2026 •

edited

Loading