Development guide¶

This tutorial guides you through setting up a working environment for developing pynxtools-spm, and explains the branching, development, and release workflows of the repository.

What should you know before this tutorial?¶

You should read the installation tutorial.
If you plan to support a new vendor file format, you should read the reader structure and how to extend the reader orchestra.

What you will know at the end of this tutorial?¶

You will know

how to set up your environment for developing pynxtools-spm,
which branch to use for which kind of change,
how to lint, format, and test your changes,
how to edit the documentation,
how to contribute on GitHub,
how a new version is released.

Setup¶

Clone the repository and install it in editable mode together with the dev and docs extras, as described in Install in development mode. This also installs the pre-commit hooks, which run the linters automatically before every commit.

Structure of the pynxtools-spm repository

The source code is located in src/pynxtools_spm; see reader structure for its subpackages. The unit tests are located in tests, with their input files in tests/data. The documentation is located in docs.

Branching model¶

pynxtools-spm follows the same branching model as nomad-FAIR and nomad-docs (see pynxtools#681).

Branch	Role
`develop`	Default branch. All normal development PRs target this branch.
`main`	Released versions only. Only `release/*` branches are merged here, so its history is the release history.
`release/v<version>`	Temporary release branch, created from `develop` and merged into `main`.

Activity	Branch
Feature development	`develop`
PR review and integration	`develop`
Selecting changes for a release	`release/vx.y.z`
Release-specific bug fixes	`release/vx.y.z`
Published release history	`main`
Porting release fixes back to ongoing development	Merge `release/vx.y.z` back into `develop`

The Check if source is release/* workflow (.github/workflows/check-source-is-release.yml) allows PRs into main only from release/* branches. PRs from any other branch, including develop, fail this check. It is enforced by marking it as a required status check in the branch protection rule for main.

Development workflow¶

Create a feature branch from develop:

git switch develop && git pull
git switch -c <short-descriptive-name>

Commit your changes, run the tests and the linters, then push the branch:
```
git push -u origin <short-descriptive-name>
```
Open a pull request against develop. You may open it as a draft while development is ongoing.

GitHub Actions then check the linting, formatting, and spelling, run the tests against several Python versions, and build the documentation. Once these checks pass and the code has been peer-reviewed, your changes are merged into develop and shipped with the next release.

Linting and formatting¶

The code is linted and formatted with ruff and statically type-checked with mypy. The pre-commit hooks additionally modernize the syntax (pyupgrade), strip outputs from Jupyter notebooks (nbstripout), and check the spelling of the code and documentation (cspell, configured in cspell.json). Run all hooks on the whole repository with:

pre-commit run --all-files

The same checks that run in CI can also be run individually:

ruff check src/pynxtools_spm tests
ruff format --check src/pynxtools_spm tests
mypy src/pynxtools_spm tests

Drop --check from the ruff format command to apply the formatting.

Testing¶

The tests are written with pytest. Run the whole test suite with:

pytest -sv tests/

When you add a new parser or formatter, add test cases for it to the tests directory and include only the necessary input files in a subdirectory of tests/data.

Editing the documentation¶

The documentation is built with MkDocs using the Material theme; its sources are the Markdown files in docs, and the navigation is defined in mkdocs.yml. Serve the documentation locally with live reload:

mkdocs serve

and open http://127.0.0.1:8000/ in your browser. The build additionally requires the tree command line utility, which is used to render the package structure.

Release workflow¶

Create the release branch from develop:

git switch develop && git pull
git switch -c release/vX.Y.Z
git push -u origin release/vX.Y.Z

Commit only release-specific changes on this branch (version bumps, changelog, last-minute bug fixes). New features keep going into develop.
Bump version in CITATION.cff to X.Y.Z, including any pre-release suffix (e.g. 0.1.2-rc1 for the tag v0.1.2-rc1). The publish workflow fails if it differs from the tag.
Open a PR from release/vX.Y.Z into main. Do not merge yet!
Create a GitHub release with the tag vX.Y.Z and select release/vX.Y.Z as its target branch. The publish workflow refuses to upload to PyPI if the release does not target release/vX.Y.Z or if the tagged commit is not on that branch.
Once CI passes and the release is published to PyPI, merge the PR.
If you fixed anything directly on the release branch, open a PR from release/vX.Y.Z into develop so the fixes are not lost in ongoing development.
Delete the release/vX.Y.Z branch.

Developing pynxtools-spm as a NOMAD plugin¶

If you plan to contribute to the NOMAD plugin functionality of pynxtools-spm (e.g., the example uploads or the SPM app in src/pynxtools_spm/nomad), it often makes sense to use the NOMAD development environment nomad-distro-dev. You can learn more in the NOMAD documentation.

Troubleshooting¶

If you face any issues with the reader or when setting up the development environment, please create a new GitHub issue.