Mapping guide doc by EBB2675 · Pull Request #131 · FAIRmat-NFDI/nomad-parser-plugins-simulation

EBB2675 · 2025-12-16T09:23:17Z

here is a PR to work together on a practical guide on mapping annotation

coveralls · 2025-12-16T09:24:56Z

Pull Request Test Coverage Report for Build 20262837933

Details

45 of 109 (41.28%) changed or added relevant lines in 5 files are covered.
486 unchanged lines in 8 files lost coverage.
Overall coverage decreased (-5.6%) to 61.334%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
src/nomad_simulation_parsers/schema_packages/orca.py	18	20	90.0%
src/nomad_simulation_parsers/parsers/orca/text_parser.py	5	31	16.13%
src/nomad_simulation_parsers/parsers/orca/parser.py	20	56	35.71%

Files with Coverage Reduction	New Missed Lines	%
tests/parsers/test_lammps_parser.py	6	97.45%
src/nomad_simulation_parsers/parsers/abinit/file_parser.py	32	50.7%
src/nomad_simulation_parsers/parsers/utils/mdparserutils.py	42	47.33%
src/nomad_simulation_parsers/parsers/lammps/trajectory_parsers.py	45	65.28%
src/nomad_simulation_parsers/parsers/lammps/file_parsers.py	57	39.34%
src/nomad_simulation_parsers/parsers/abinit/parser.py	77	32.37%
src/nomad_simulation_parsers/parsers/lammps/parser.py	112	21.79%
src/nomad_simulation_parsers/parsers/utils/mdanalysisparser.py	115	35.48%

Totals
Change from base Build 20136411595:	-5.6%
Covered Lines:	4229
Relevant Lines:	6895

💛 - Coveralls

ndaelman-hu · 2025-12-16T10:14:41Z

I will just for now give a review solely on the docs. Can do one later the rest of the parser, if necessary.

EBB2675 · 2025-12-16T10:18:54Z

I will just for now give a review solely on the docs. Can do one later the rest of the parser, if necessary.

yes please 🤝

i have just quickly pushed the doc from a random branch of mine, the rest is indeed not relevant
@ndaelman-hu @JFRudzinski @Bernadette-Mohr @kubanmar

ndaelman-hu

Thank you for adding this. Most of the feedback below comes down to specific phrases.

More importantly, how should we combine this with the docs in nomad-docs?
I would keep both in the same repo, don't have strong feelings which one.

ndaelman-hu · 2025-12-16T11:15:29Z

+
+## What a mapping annotation is
+
+- A small note on a schema field that says: *for source `<KEY>`, read value at `<PATH>` and put it here.*


Suggested change

- A small note on a schema field that says: *for source `<KEY>`, read value at `<PATH>` and put it here.*

- A small note on a schema field that says: *for source `<KEY>`, read value at `<PATH>` and place it in the archive under the path leading to the annotated schema field.*

this is what i initially wanted to avoid actually, as this internal document is trying to have a casual and simple approach

ndaelman-hu · 2025-12-16T11:17:20Z

+
+- A small note on a schema field that says: *for source `<KEY>`, read value at `<PATH>` and put it here.*
+- Stored on any section/quantity: `m_annotations['mapping'][<KEY>] = MapperAnnotation(mapper=...)`.
+- `<KEY>` names the source (e.g., `OUT_KEY`, `HDF5_KEY`, `INFO_KEY`, `BAND_KEY`).


I would add "why we there may be multiple keys". If there was only one, that field would not be necessary.

this part is focusing on what, and not why. we should avoid giving out too many details at once

sure, then maybe add the "why" above

ndaelman-hu · 2025-12-16T11:18:22Z

+
+- A small note on a schema field that says: *for source `<KEY>`, read value at `<PATH>` and put it here.*
+- Stored on any section/quantity: `m_annotations['mapping'][<KEY>] = MapperAnnotation(mapper=...)`.
+- `<KEY>` names the source (e.g., `OUT_KEY`, `HDF5_KEY`, `INFO_KEY`, `BAND_KEY`).


Minor nitpick: it doesn't "name" the source per se (as in it does not define the source file).
It just defines a particular source format / mapping.

will revise it 👍

ndaelman-hu · 2025-12-16T11:19:00Z

+- A small note on a schema field that says: *for source `<KEY>`, read value at `<PATH>` and put it here.*
+- Stored on any section/quantity: `m_annotations['mapping'][<KEY>] = MapperAnnotation(mapper=...)`.
+- `<KEY>` names the source (e.g., `OUT_KEY`, `HDF5_KEY`, `INFO_KEY`, `BAND_KEY`).
+- Paths use JMESPath style. Relative paths start with `.` (from the parent’s mapped node). `@` anchors at the source root.


Suggested change

- Paths use JMESPath style. Relative paths start with `.` (from the parent’s mapped node). `@` anchors at the source root.

- Paths use [JMESPath](https://jmespath.org/) style. Relative paths start with `.` (from the parent’s mapped node). `@` anchors at the source root.

ndaelman-hu · 2025-12-16T11:20:34Z

+
+1) A source parser (`XMLParser`, `HDF5Parser`, `TextParser`, etc.) reads a file into a Python dict.
+2) Set `annotation_key` on a `MetainfoParser` to the `<KEY>` that matches the annotations.
+3) Call `convert(source_parser, target_parser)`. One key → one `convert`. Multiple keys/sources → run `convert` once per key after switching `annotation_key` each time.


One key → one convert.

Lol, sounds oddly religious.

ndaelman-hu · 2025-12-16T11:58:58Z

+add_mapping_annotation(general.Simulation.m_def, OUT_KEY, '@')
+```
+
+### 2) Nested quantities


Suggested change

### 2) Nested quantities

### 2) Quantities

Quantities can never be at the root.

ndaelman-hu · 2025-12-16T12:01:59Z

+add_mapping_annotation(general.Program.version, OUT_KEY, '.program_version')
+```
+
+### 3) Single nested subsection


Suggested change

### 3) Single nested subsection

### 3) Single subsection

Tautology: the prefix "sub" already clarifies that is "nested" as you refer to it. Moreover, "nested" obtains a very different meaning when talking about serialization and querying. Would avoid it to prevent confusion.

ndaelman-hu · 2025-12-16T12:03:16Z

+add_mapping_annotation(model_method.DFT.m_def, OUT_KEY, '.dft')
+```
+
+### 4) Repeating subsections (lists)


Suggested change

### 4) Repeating subsections (lists)

### 4) Repeating subsections & Array-like Quantities

ndaelman-hu · 2025-12-16T12:05:33Z

+### 5) Multiple sources into the same schema
+Use different keys; run `convert` once per key.
+```python
+# Octopus uses OUT_KEY + INFO_KEY + EIGENVALUES_KEY on the same schema
+# Exciting uses INFO_KEY, INPUT_XML_KEY, EIGVAL_KEY, BANDSTRUCTURE_XML_KEY, DOS_XML_KEY
+# Wannier90 uses WOUT_KEY, WIN_KEY, BAND_KEY, WHR_KEY, DOS_KEY
+```
+Set `annotation_key = <KEY>` before each `convert`.


Can you add an example, pls?
From memory, I'd say that convert is called at the parser-level instead of the annotations, but might be fuzzy on that one.

ndaelman-hu · 2025-12-16T12:06:33Z

+Set `annotation_key = <KEY>` before each `convert`.
+
+### 6) Transformations (reshape, filter, apply units)
+Use a tuple `(function_name, [paths], {kwargs})` and implement the function on the parser.


Suggested change

Use a tuple `(function_name, [paths], {kwargs})` and implement the function on the parser.

Use a tuple `(function_name, [paths], {kwargs})` and implement the function at the parser level.

Maybe you should introduce the 2 levels.

Bernadette-Mohr · 2025-12-16T13:36:24Z

+- Absolute from root: `field.subfield` (no leading dot)
+- Root anchor: `@`
+- Filter: `.items[?"@name"=="version"] | [0].__value`
+- Attributes (XML/HDF5): keys often prefixed with `@`; values with `__value`


Does "often" here mean that this is a convention, but it would theoretically also be possible, to, e.g., prefix a value with @?

JFRudzinski · 2025-12-19T10:56:20Z

I will tackle this as part of my documentation work: FAIRmat-NFDI/nomad-simulations#311

EBB2675 added 11 commits December 2, 2025 13:19

create parsers/orca

f749e86

parse program, particle_states and coordinates

e377d75

parse DFT and some numerical settings

04d5df1

cleanup, ruff

06081af

remove old DFT annotations

0b17f47

add entrypoints

3918cd0

fix AssertionError

1209811

minimally parse DFT

bef3632

add 3 example ORCA outputs

8e3600a

add model outputs for geometry optimization and hessian

194c957

add doc for mapping annotation

2b91465

EBB2675 requested review from Bernadette-Mohr, JFRudzinski, kubanmar and ndaelman-hu December 16, 2025 09:23

ndaelman-hu reviewed Dec 16, 2025

View reviewed changes

Bernadette-Mohr reviewed Dec 16, 2025

View reviewed changes

Bernadette-Mohr approved these changes Dec 16, 2025

View reviewed changes

EBB2675 marked this pull request as draft December 19, 2025 07:53

EBB2675 added the wontfix This will not be worked on label Dec 19, 2025

JFRudzinski self-assigned this Dec 19, 2025

JFRudzinski added documentation Improvements or additions to documentation and removed wontfix This will not be worked on labels Dec 19, 2025

JFRudzinski mentioned this pull request Dec 19, 2025

Super Issue: Documentation FAIRmat-NFDI/nomad-simulations#311

Closed

3 tasks


		## What a mapping annotation is

		- A small note on a schema field that says: for source `<KEY>`, read value at `<PATH>` and put it here.

	- Paths use JMESPath style. Relative paths start with `.` (from the parent’s mapped node). `@` anchors at the source root.
	- Paths use [JMESPath](https://jmespath.org/) style. Relative paths start with `.` (from the parent’s mapped node). `@` anchors at the source root.

	### 4) Repeating subsections (lists)
	### 4) Repeating subsections & Array-like Quantities

	Use a tuple `(function_name, [paths], {kwargs})` and implement the function on the parser.
	Use a tuple `(function_name, [paths], {kwargs})` and implement the function at the parser level.

Uh oh!

Conversation

EBB2675 commented Dec 16, 2025

Uh oh!

coveralls commented Dec 16, 2025

Pull Request Test Coverage Report for Build 20262837933

Details

💛 - Coveralls

Uh oh!

ndaelman-hu commented Dec 16, 2025

Uh oh!

EBB2675 commented Dec 16, 2025

Uh oh!

ndaelman-hu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JFRudzinski commented Dec 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants