2.6. NeXus Utilities

There are many utilities available to read, browse, write, and use NeXus data files. Some are provided by the NeXus technical group while others are provided by the community. Still, other tools listed here can read or write one of the low-level file formats used by NeXus (HDF5, HDF4, or XML).

Furthermore, there are specific examples of code that can read, write, (or both) NeXus data files, given in the section Language APIs for NeXus and HDF5.

The NIAC welcomes your continued contributions to this documentation.

Please note that NeXus maintains a repository of example data files [1] which you may browse and download. There is a cursory analysis [2] of every file in this repository as to whether it can be read as HDF5 or NeXus HDF5. The analysis code [3], which serves as yet another example reader, is made using python and h5py.

2.6.1. Utilities supplied with NeXus

Most of these utility programs are run from the command line. It will be noted if a program provides a graphical user interface (GUI). Short descriptions are provided here with links to further information, as available.

nxbrowse

NeXus Browser

nxconvert

Utility to convert a NeXus file into HDF4/HDF5/XML/…

nxdir

nxdir is a utility for querying a NeXus file about its contents. Full documentation can be found by running this command:

nxdir -h
nxingest

nxingest extracts the metadata from a NeXus file to create an XML file according to a mapping file. The mapping file defines the structure (names and hierarchy) and content (from either the NeXus file, the mapping file or the current time) of the output file. See the man page for a description of the mapping file. This tool uses the NAPI. Thus, any of the supported formats (HDF4, HDF5 and XML) can be read.

nxsummary

Use nxsummary to generate summary of a NeXus file. This program relies heavily on a configuration file. Each item tag in the file describes a node to print from the NeXus file. The path attribute describes where in the NeXus file to get information from. The label attribute will be printed when showing the value of the specified field. The optional operation attribute provides for certain operations to be performed on the data before printing out the result. See the source code documentation for more details.

nxtranslate

nxtranslate is an anything to NeXus converter. This is accomplished by using translation files and a plugin style of architecture where nxtranslate can read from new formats as plugins become available. The documentation for nxtranslate describes its usage by three types of individuals:

  • the person using existing translation files to create NeXus files

  • the person creating translation files

  • the person writing new retrievers

All of these concepts are discussed in detail in the documentation provided with the source code.

NXplot

An extendable utility for plotting any NeXus file. NXplot is an Eclipse-based GUI project in Java to plot data in NeXus files. (The project was started at the first NeXus Code Camp in 2009.)

2.6.2. Validation

The list of applications below are for validating NeXus files. The list is not intended to be a complete list of all available packages.

cnxvalidate

NeXus validation tool written in C (not via NAPI).

Its dependencies are libxml2 and the HDF5 libraries, version 1.8.9 or better. Its purpose is to validate HDF5 files against NeXus application definitions.

See the program documentation for more details: https://github.com/nexusformat/cnxvalidate.git

punx

Python Utilities for NeXus HDF5 files

punx can validate both NXDL files and NeXus HDF5 data files, as well as print the structure of any HDF5 file, even non-NeXus files.

NOTE: project is under initial construction, not yet released for public use, but is useful in its present form (version 0.2.5).

punx can show the tree structure of any HDF5 file. The output is more concise than that from h5dump.

See the program documentation for more details: https://punx.readthedocs.io

2.6.3. Other Utilities

NeXus Constructor (https://github.com/ess-dmsc/nexus-constructor)

The NeXus Constructor facilitates constructing NeXus files in which to record data from experiments at neutron science facilities. This includes all supporting metadata typically required to perform analysis of such experiments, including instrument geometry information.

nxdl_to_hdf5.py (https://github.com/nexusformat/exampledata/tree/master/nxdl)

nxdl_to_hdf5.py is a Python script that reads the NeXus definition files (files ending with .nxdl.xml) and creates example Python scripts as well as HDF5 files for each definition. There are generated example scripts of each application definition for both h5py and nexusformat. Currently, only application definitions and some contributed_definitions are supported as the code depends on the existence of an NXentry in the definition.

2.6.4. Data Analysis

The list of applications below are some of the utilities that have been developed (or modified) to read/write NeXus files as a data format. It is not intended to be a complete list of all available packages.

DAVE (http://www.ncnr.nist.gov/dave/)

DAVE is an integrated environment for the reduction, visualization and analysis of inelastic neutron scattering data. It is built using IDL (Interactive Data Language) from ITT Visual Information Solutions.

DAWN (http://www.dawnsci.org)

The Data Analysis WorkbeNch (DAWN) project is an eclipse based workbench for doing scientific data anaylsis. It offers generic visualisation, and domain specific processing.

GDA (http://www.opengda.org)

The GDA project is an open-source framework for creating customised data acquisition software for science facilities such as neutron and X-ray sources. It has elements of the DAWN analysis workbench built in.

Gumtree (https://archive.ansto.gov.au/ResearchHub/OurInfrastructure/ACNS/Facilities/Computing/GumTree/index.htm)

Gumtree is an open source project, providing a graphical user interface for instrument status and control, data acquisition and data reduction.

IDL (https://www.harrisgeospatial.com/docs/using_idl_home.html)

IDL is a high-level technical computing language and interactive environment for algorithm development, data visualization, data analysis, and numeric computation.

IgorPro (http://www.wavemetrics.com/)

IGOR Pro is an extraordinarily powerful and extensible scientific graphing, data analysis, image processing and programming software tool for scientists and engineers.

ISAW (ftp://ftp.sns.gov/ISAW/)

The Integrated Spectral Analysis Workbench software project (ISAW) is a Platform-Independent system Data Reduction/Visualization. ISAW can be used to read, manipulate, view, and save neutron scattering data. It reads data from IPNS run files or NeXus files and can merge and sort data from separate measurements.

LAMP (http://www.ill.eu/data_treat/lamp/>)

LAMP (Large Array Manipulation Program) is designed for the treatment of data obtained from neutron scattering experiments at the Institut Laue-Langevin. However, LAMP is now a more general purpose application which can be seen as a GUI-laboratory for data analysis based on the IDL language.

Mantid (http://www.mantidproject.org/)

The Mantid project provides a platform that supports high-performance computing on neutron and muon data. It is being developed as a collaboration between Rutherford Appleton Laboratory and Oak Ridge National Laboratory.

MATLAB (http://www.mathworks.com/)

MATLAB is a high-level technical computing language and interactive environment for algorithm development, data visualization, data analysis, and numeric computation.

NeXpy (http://nexpy.github.io/nexpy/)

The goal of NeXpy is to provide a simple graphical environment, coupled with Python scripting capabilities, for the analysis of X-Ray and neutron scattering data. (It was decided at the NIAC 2010 meeting that a large portion of this code would be adopted in the future by NeXus and be part of the distribution)

silx (http://www.silx.org/doc/silx/latest/)

The silx project aims to provide a collection of Python packages to support the development of data assessment, reduction and analysis at synchrotron radiation facilities. In particular it provides tools to read, write and visualize NeXus HDF5 files.

OpenGENIE (http://www.opengenie.org/)

A general purpose data analysis and visualisation package primarily developed at the ISIS Facility, Rutherford Appleton Laboratory.

PyMCA (http://pymca.sourceforge.net/)

PyMca is a ready-to-use, and in many aspects state-of-the-art, set of applications implementing most of the needs of X-ray fluorescence data analysis. It also provides a Python toolkit for visualization and analysis of energy-dispersive X-ray fluorescence data. Reads, browses, and plots data from NeXus HDF5 files.

spec2nexus (https://spec2nexus.readthedocs.io)

(Python code) Converts SPEC data files and scans into NeXus HDF5 files. (Note the h5toText tool mentioned here previously is no longer available from the spec2nexus project. The code has been moved into the punx project: https://punx.readthedocs.io/.)

spec2nexus provides libraries:

  • spec2nexus.spec: python binding to read SPEC [4] data files

  • spec2nexus.eznx: (Easy NeXus) supports writing NeXus HDF5 files using h5py

2.6.5. HDF Tools

Here are some of the generic tools that are available to work with HDF files. In addition to the software listed here there are also APIs for many programming languages that will allow low level programmatic access to the data structures.

h5wasm (https://github.com/usnistgov/h5wasm):

A WebAssembly port of the HDF5 C library, which allows reading and writing HDF5 files from JavaScript (i.e. no need for a back-end server at all).

H5Web (https://github.com/silx-kit/h5web):

H5Web is a toolkit for exploring and visualising HDF5 files and, more generally, for visualizing data. It is based on React, and WebGL. These projects make use of H5Web:

HDF Group tools (https://portal.hdfgroup.org/display/support/Downloads)

Various tools are available from the HDF Group. These are usually shipped with the HDF5 kits but are also available for download separately. The HDF5 source code (https://github.com/HDFGroup/hdf5) is available on GitHub.

HDFexplorer (http://www.space-research.org/)

A data visualization program that reads Hierarchical Data Format files (HDF, HDF-EOS and HDF5) and also netCDF data files.

HDFview (http://www.hdfgroup.org)

A Java based GUI for browsing (and some basic plotting) of HDF files.

tiled (https://blueskyproject.io/tiled/)

A data access service for data-aware portals and data science tools, provides a way to browse and visualize HDF5 files.

2.6.6. Language APIs for NeXus and HDF5

Collected here are some of the tools identified [5] as a result of a simple question asked at the 2018 Nobugs conference: Are there examples of code that reads NeXus data? Some of these are very specific to an instrument or application definition while others are more generic. The lists below are organized by programming language, yet some collections span several languages so they are listed in the section Language API: mixed.

Note these example listed in addition to the many examples described here in the manual, in section :Examples.

2.6.6.1. Language API: F77

2.6.6.2. Language API: IDL

2.6.6.3. Language API: IgorPro

2.6.6.4. Language API: Java

2.6.6.5. Language API: Python

2.6.6.6. Language API: mixed

  • FOCUS: focus.zip [25] contains:

    • An example FOCUS file

    • focusreport: A h5py program which skips through a list of files and prints statistics

    • focusreport.tcl, same as above but in Tcl using the Swig generated binding to NAPI

    • i80.f contains a F77 routine for reading FOCUS files into Ida. The routine is RRT_in_Foc.