Parser testing protocol

Purpose

Parser tests must say which contract failed. A test that only runs a complete parser and inspects the resulting EntryArchive mixes several contracts:

1. mainfile and auxiliary-file recognition;
2. extraction from the source format;
3. conversion and mapping to NOMAD metainfo;
4. archive assembly and schema validation; and
5. normalization.

Such a test is useful as a smoke test, but it is too broad to be the primary way of testing a parser. Failures are hard to locate, malformed-input behavior is difficult to exercise, and small extraction or mapping rules require large fixtures.

This protocol makes each boundary independently testable while retaining a small number of complete-parser tests.

Test architecture

Each parser is tested as the following pipeline:

files -> recognition -> source reader -> parsed source data -> mapper -> archive
                                                                  |
                                                                  v
                                                             normalizer

The objects at the two internal boundaries are test interfaces:

• parsed source data is the dictionary, array, or typed object returned by a file parser such as a TextParser, FileParser, or format-specific reader;
• archive fragment is the metainfo section written by an ArchiveWriter or equivalent mapping component before normalization.

New parser code must keep extraction and mapping callable independently. Existing parsers should expose these seams when they are migrated; a rewrite to one common implementation is not required.

Layer 1: recognition

Test mainfile matching, auxiliary-file recognition, compression handling, and file precedence without parsing a calculation. Use a temporary directory containing empty or minimally named files.

Assertions should cover:

• matching and non-matching filenames or headers;
• deterministic selection when several candidates exist;
• missing, duplicate, and compressed auxiliary files; and
• paths independent of the current working directory.

These are unit tests and must not create an EntryArchive.

Layer 2: source extraction

Call the lowest-level reader with a minimal source fragment and assert its parsed source data. One fixture should demonstrate one syntax feature or source-format variant. Prefer inline text or a fixture below 10 KiB; use a binary fixture only when the format requires it.

Check semantic values, not merely presence or collection sizes. For example, test the parsed SCF energies, spin channel, atom labels, and source unit. Include cases for repeated sections, optional fields, truncated files, malformed numeric values, and supported program-version variants.

The expected result belongs in the test when it is short. Larger results use a reviewed YAML manifest containing only the fields relevant to that feature. Do not store a serialized parser object or a complete archive as the oracle.

Layer 3: mapping contract

Feed controlled parsed source data directly to the mapping/archive-writing layer. This isolates source-to-metainfo semantics from regular expressions and third-party file readers.

Every supported archive concept must have a mapping-contract test that checks:

• target section type and path;
• value and explicit unit conversion;
• array shape and index or spin ordering;
• references between systems, methods, outputs, and workflow tasks;
• distinction between absent data, an empty collection, zero, and False; and
• merge or precedence rules when multiple files provide the same concept.

Use pytest.approx or numpy.testing with a tolerance chosen for the source precision. Convert quantities to a named unit before comparison. Avoid conditional assertions such as if output.electronic_dos; if a fixture promises DOS, its absence must fail the test.

Mapping tests create only the smallest required archive or section. They do not run normalizers.

Layer 4: parser integration

Run the public parser entry point on one small, representative fixture for each calculation mode the parser supports, for example single point, geometry optimization, molecular dynamics, or a post-processing output.

An integration test verifies the cross-component invariants rather than repeating every extracted number:

• parsing completes without unexpected error-level logs;
• the archive can be serialized and deserialized;
• metainfo validation succeeds;
• systems, methods, outputs, and workflow references resolve;
• their step counts and array dimensions agree; and
• a short list of scientifically identifying values is correct.

The identifying values should catch a wrong file, wrong calculation, wrong unit, or wrong ordering. They are not a snapshot of the archive.

Layer 5: NOMAD pipeline compatibility

Keep normalization outside parser tests. A separate compatibility test runs the parser and the required NOMAD normalizers for one fixture per parser, then checks a small public consumer contract such as the representative system, material, and results properties.

This layer detects integration drift between packages. It must not be used to prove that an individual regular expression or mapping is correct. If it fails, the lower layers should identify whether the parser contract itself is still satisfied.

Shared simulation parser contract

Do not copy the parser-independent parts of layers 4 and 5 into every parser. Compose the Simulation, model-system, and workflow suites with one representative, module-scoped archive fixture and the expected program name:

from tests.parsers.common import (
    SimulationParserTestSuite,
    WorkflowTestSuite,
)


class TestExampleParserSuite(
    SimulationParserTestSuite, WorkflowTestSuite
):
    archive_fixture = 'example_archive'
    expected_program_name = 'ExampleCode'

Use this composition in test_integration.py. SimulationParserTestSuite checks validation, core Simulation sections, and model-system structural integrity, while WorkflowTestSuite checks workflow presence and round-trip serialization. A parser may omit a suite only when that archive concept is not part of its public contract. Subclass SimulationParserPipelineTestSuite in test_pipeline.py with the same fixture and program name to check normalization compatibility. Keep recognition rules, source-reader behavior, mapping details, supported optional sections, and scientific reference values in parser-local tests.

Fixtures and expected-data manifests

Organize tests by parser and layer:

tests/parsers/<code>/
  test_recognition.py
  test_reader.py
  test_mapping.py
  test_integration.py
  cases/
    scf-minimal/
      case.yaml
      <source files>

case.yaml records why a fixture exists and its stable expectations:

id: silicon-scf
purpose: spin-unpolarized SCF with two ionic steps
program_version: "6.8"
mainfile: output.out
features: [structure, scf, forces]
expect:
  program.name: ExampleCode
  model_system[0].positions:
    shape: [2, 3]
    unit: angstrom
    sample:
      "[1, 2]": 1.357
  outputs:
    length: 2
  outputs[1].total_energies[0].value:
    value: -10.42
    unit: eV
    rel: 1.0e-8

The manifest is a declarative aid, not a generic snapshot framework. A shared assertion helper may implement value, unit, shape, length, and selected sample checks. It must reject unknown keys and produce an error containing the case id and archive path. References and domain invariants remain explicit Python assertions.

Fixture rules:

• include provenance and the feature under test in case.yaml;
• remove unrelated output and redact user or machine paths;
• prefer the smallest source that the real reader accepts;
• never generate expected data by running the parser under test in CI;
• review manifest changes as behavior changes, not routine snapshot updates; and
• mark large or externally supplied fixtures explicitly rather than silently skipping when they are absent.

Negative and metamorphic tests

Each parser must have negative tests for a truncated mainfile, a missing required side file, and invalid content. The expected behavior—exception, warning with partial archive, or ignored optional file—must be explicit.

Where applicable, add transformations whose result is known without another oracle:

• changing whitespace or line endings preserves parsed values;
• parsing a compressed and uncompressed file produces equivalent source data;
• translating every atom preserves energies and shifts positions only;
• reordering auxiliary files does not change deterministic precedence; and
• changing the source unit rescales the mapped value correctly.

These tests cover entire classes of inputs more effectively than adding more archive quantity counts.

Test ownership and minimum coverage

For every advertised parser feature, maintain this matrix in the parser's tests or documentation:

Feature	Reader case	Mapping case	Integration case
Structure	required	required	representative
Method and numerical settings	required	required	representative
Energies and SCF history	required	required	representative
Forces and stress	when supported	when supported	one identifying value
DOS, bands, or eigenvalues	when supported	when supported	one shape/value
Workflow	source inputs if parsed	required	required by calculation mode

A pull request that adds or fixes a parsing rule must add the lowest-layer regression test capable of reproducing the bug. Add or change an integration fixture only when the public parser behavior or a cross-component invariant changes.

Pytest markers and CI

Use these markers:

• unit: recognition, reader, utility, and mapping-contract tests;
• integration: public parser entry-point tests;
• pipeline: parser plus NOMAD normalization;
• large_fixture: tests whose fixture set is too large for the default checkout.

CI runs in three stages:

1. Pull-request fast gate: all unit tests and integration tests affected by changed parser directories. Target: less than two minutes.
2. Pull-request package gate: all non-large unit and integration tests.
3. Nightly or release gate: pipeline and large_fixture tests using a versioned fixture bundle with checksums.

These stages are implemented in .github/workflows/actions.yml. The nightly fixture manifest is tests/fixtures/nightly.sha256 and is verified by tools/verify_fixture_bundle.py; update its checksum in the same change as a fixture update. The fast gate runs all unit tests, then selects integration directories changed by the pull request. Changes to shared parser code fall back to all integration tests.

Tests must not pass merely because a fixture is unavailable. A required fixture is a repository or CI setup error. Optional large fixtures may be deselected by marker, but a selected test with a missing fixture must fail.