EPICS Area Detector Examples¶

Two examples in this section show how to write NeXus HDF5 data files with EPICS Area Detector images. The first shows how to configure the HDF5 File Writing Plugin of the EPICS Area Detector software. The second example shows how to write an EPICS Area Detector image using Python.

HDF5 File Writing Plugin¶

This example describes how to write a NeXus HDF5 data file using the EPICS [1] Area Detector [2] HDF5 file writing plugin [3]. We will use the EPICS SimDetector [4] as an example. (PV prefix: 13SIM1:) Remember to replace that with the prefix for your detector’s IOC.

One data file will be produced for each image generated by EPICS.

You’ll need AreaDetector version 2.5 or higher to use this as the procedures for using the HDF5 file writing plugin changed with this release.

configuration files¶

There are two configuration files we must edit to configure an EPICS AreaDetector to write NeXus files using the HDF5 File Writer plugin:

file	description
`attributes.xml`	what information to know about from EPICS and other sources
`layout.xml`	where to write that information in the HDF5 file

Put these files into a known directory where your EPICS IOC can find them.

`attributes.xml`¶

The attributes file is easy to edit. Any text editor will do. A wide screen will be helpful.

Each <Attribute /> element declares a single ndattribute which is associated with an area detector image. These ndattribute items can be written to specific locations in the HDF5 file or placed by default in a default location.

Note

The attributes file shown here has been reformatted for display in this manual. The downloads section below provides an attributes file with the same content using its wide formatting (one complete Attribute per line). Either version of this file is acceptable.

<?xml version="1.0" standalone="no" ?>
<!-- Attributes -->
<Attributes
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation=
    "https://github.com/areaDetector/ADCore/blob/master/iocBoot/NDAttributes.xsd"
    >
    <Attribute name="AcquireTime"         
               type="EPICS_PV" 
               source="13SIM1:cam1:AcquireTime"      
               dbrtype="DBR_NATIVE"  
               description="Camera acquire time"/>
    <Attribute name="ImageCounter"        
               type="PARAM"    
               source="ARRAY_COUNTER"                
               datatype="INT"        
               description="Image counter"/>
    <Attribute name="calc1_val"           
               type="EPICS_PV" 
               source="prj:userCalc1.VAL"            
               datatype="DBR_NATIVE" 
               description="some calculation result"/>
    <Attribute name="calc2_val"           
               type="EPICS_PV" 
               source="prj:userCalc2.VAL"            
               datatype="DBR_NATIVE" 
               description="another calculation result"/>
    <Attribute name="MaxSizeX"            
               type="PARAM"    
               source="MAX_SIZE_X"                   
               datatype="INT"        
               description="Detector X size"/>
    <Attribute name="MaxSizeY"            
               type="PARAM"    
               source="MAX_SIZE_Y"                   
               datatype="INT"        
               description="Detector Y size"/>
    <Attribute name="CameraModel"         
               type="PARAM"    
               source="MODEL"                        
               datatype="STRING"     
               description="Camera model"/>
    <Attribute name="CameraManufacturer"  
               type="PARAM"    
               source="MANUFACTURER"                 
               datatype="STRING"     
               description="Camera manufacturer"/>
</Attributes>

If you want to add additional EPICS process variables (PVs) to be written in the HDF5 file, create additional <Attribute /> elements (such as the calc1_val) and modify the name, source, and description values. Be sure to use a unique name for each ndattribute in the attributes file.

Note

ndattribute : item specified by an <Attribute /> element in the attributes file.

`layout.xml`¶

You might not need to edit the layout file. It will be fine (at least a good starting point) as it is, even if you add PVs (a.k.a. ndattribute) to the attributes.xml file.

<?xml version="1.0" standalone="no" ?>
<hdf5_layout>
  <group name="entry"> 
    <attribute name="NX_class" source="constant" value="NXentry" type="string"/>
    <group name="instrument"> 
      <attribute name="NX_class" source="constant" value="NXinstrument" type="string"/> 
      <group name="detector"> 
        <attribute name="NX_class" source="constant" value="NXdetector" type="string"/> 
        <dataset name="data" source="detector" det_default="true"> 
          <attribute name="NX_class" source="constant" value="SDS" type="string"/> 
          <attribute name="signal" source="constant" value="1" type="int"/> 
          <attribute name="target" source="constant" value="/entry/instrument/detector/data" type="string"/> 
        </dataset> 
        <group name="NDAttributes"> 
          <attribute name="NX_class" source="constant" value="NXcollection" type="string"/> 
          <dataset name="ColorMode" source="ndattribute" ndattribute="ColorMode"/> 
        </group>          <!-- end group NDAttribute --> 
      </group>            <!-- end group detector --> 
      <group name="NDAttributes" ndattr_default="true"> 
        <attribute name="NX_class" source="constant" value="NXcollection" type="string"/> 
      </group>            <!-- end group NDAttribute (default) --> 
      <group name="performance"> 
        <dataset name="timestamp" source="ndattribute"/> 
      </group>            <!-- end group performance --> 
    </group>              <!-- end group instrument --> 
    <group name="data"> 
      <attribute name="NX_class" source="constant" value="NXdata" type="string"/> 
      <hardlink name="data" target="/entry/instrument/detector/data"/>
      <!-- The "target" attribute in /entry/instrument/detector/data is used to 
           tell Nexus utilities that this is a hardlink -->
    </group>              <!-- end group data --> 
  </group>                <!-- end group entry -->
</hdf5_layout>

If you do not specify where in the file to write an ndattribute from the attributes file, it will be written within the group that has ndattr_default="true". This identifies the group to the HDF5 file writing plugin as the default location to store content from the attributes file. In the example layout file, that default location is the /entry/instrument/NDAttributes group:

<group
      name="NDAttributes"
      ndattr_default="true">
  <attribute
      name="NX_class"
      source="constant"
      value="NXcollection"
      type="string"/>
</group>

To specify where PVs are written in the HDF5 file, you must create <dataset /> (or <attribute />) elements at the appropriate place in the NeXus HDF5 file layout. See the NeXus manual [5] for placement advice if you are unsure.

You reference each ndattribute by its name value from the attributes file and use it as the value of the ndattribute in the layout file. In this example, ndattribute="calc1_val" in the layout file references name="calc1_val" in the attributes file and will be identified in the HDF5 file by the name userCalc1:

<dataset
   name="userCalc1"
   source="ndattribute"
   ndattribute="calc1_val"/>

Note

A value from the attributes file is only written either in the default location or in the location named by a <dataset/> or <attribute/> entry in the layout file. Expect problems if you define the same ndattribute in more than one place in the layout file.

You can control when a value is written to the file, using when="" in the layout file. This can be set to one of these values: OnFileOpen, OnFileClose

Such as:

<dataset
   name="userCalc1"
   source="ndattribute"
   ndattribute="calc1_val"
   when="OnFileOpen"/>

or:

<attribute
   name="exposure_s"
   source="ndattribute"
   ndattribute="AcquireTime"
   when="OnFileClose"/>

additional configuration¶

Additional configurations of the EPICS Area Detector and the HDF5 File Plugin are done using the EPICS screens (shown here using caQtDM [6]):

**ADBase** and **NDFileHDF5** configuration screens¶

Additional configuration on the ADBase screen:

Set Image mode to “Single”
Set Exposure time as you wish
Set # Images to 1
for testing, it is common to bin the data to reduce the image size
The full path to the attributes.xml file goes in the bottom/left File box

Additional configuration on the NDFileHDF5 screen:

Set the File path and “File name” to your choice.
Set Auto save to “Yes”.
Set Compression to “zlib” if you wish (optional)
Set Enable to “Enable” or the HDF5 plugin won’t get images to write!
Set Callbacks block to “Yes” if you want to wait for HDF5 files to finish writing before collecting the next image
The full path to the layout.xml file goes into the bottom/right XML File name box
Leave the Attributes file box empty in this screen.

When you enter the names of these files in the configuration screen boxes, AreaDetector will check the files for errors and let you know.

Example view¶

We collected data for one image, /tmp/mrinal_001.h5, in the HDF5 file provided in the downloads section. You may notice that the values for calc1_val and calc2_val were arrays rather than single values. That was due to an error in the original attributes.xml file, which had type="PARAM" instead of type="EPICS_PV". This has been fixed in the attributes.xml file presented here.

Python code to store an image in a NeXus file¶

Suppose you want to write area detector images into NeXus HDF5 files python code. Let’s assume you have the image already in memory in a numpy array, perhaps from reading a TIFF file or from an EPICS PV using PyEpics. The file write_nexus_file.py (provided below) reads an image from the sim detector and writes it to a NeXus HDF5 data file, along with some additional metadata.

using the h5py package¶

This example uses the h5py [7] package to write the HDF5 file.

import numpy as np
import h5py
import datetime

def write_nexus_file(fname, image, md={}):
    """
    write the image to a NeXus HDF5 data file

    Parameters
    ----------
    fname : str
        name of the file (relative or absolute) to be written
    image : numpy array
        the image data
    md : dictionary
        key: value where value is something that can be written by h5py
             (such as str, int, float, numpy array, ...)
    """
    nexus = h5py.File(fname, "w")
    nexus.attrs["filename"] = fname
    nexus.attrs["file_time"] = datetime.datetime.now().astimezone().isoformat()
    nexus.attrs["creator"] = "write_nexus_file()"
    nexus.attrs["H5PY_VERSION"] = h5py.__version__

    # /entry
    nxentry = nexus.create_group("entry")
    nxentry.attrs["NX_class"] = "NXentry"
    nexus.attrs["default"] = nxentry.name

    # /entry/instrument
    nxinstrument = nxentry.create_group("instrument")
    nxinstrument.attrs["NX_class"] = "NXinstrument"

    # /entry/instrument/detector
    nxdetector = nxinstrument.create_group("detector")
    nxdetector.attrs["NX_class"] = "NXdetector"

    # /entry/instrument/detector/image
    ds = nxdetector.create_dataset("image", data=image, compression="gzip")
    ds.attrs["units"] = "counts"
    ds.attrs["target"] = "/entry/instrument/detector/image"

    # /entry/data
    nxdata = nxentry.create_group("data")
    nxdata.attrs["NX_class"] = "NXdata"
    nxentry.attrs["default"] = nxdata.name

    # /entry/data/data --> /entry/instrument/detector/image
    nxdata["data"] = nexus["/entry/instrument/detector/image"]
    nxdata.attrs["signal"] = "data"

    if len(md) > 0:
        # /entry/instrument/metadata (optional, for metadata)
        metadata = nxinstrument.create_group("metadata")
        metadata.attrs["NX_class"] = "NXcollection"
        for k, v in md.items():
            try:
                metadata.create_dataset(k, data=v)
            except Exception:
                metadata.create_dataset(k, data=str(v))

    nexus.close()

    
if __name__ == "__main__":
    """demonstrate how to use this code"""
    import epics
    prefix = "13SIM1:"
    img = epics.caget(prefix+"image1:ArrayData")
    size_x = epics.caget(prefix+"cam1:ArraySizeX_RBV")
    size_y = epics.caget(prefix+"cam1:ArraySizeY_RBV")
    # edit the full image for just the binned data
    img = img[:size_x*size_y].reshape((size_x, size_y))

    extra_information = dict(
        unique_id = epics.caget(prefix+"image1:UniqueId_RBV"),
        size_x = size_x,
        size_y = size_y,
        detector_state = epics.caget(prefix+"cam1:DetectorState_RBV"),
        bitcoin_value="15000",
    )
    write_nexus_file("example.h5", img, md=extra_information)

The output from that code is given in the example.h5 file. It has this tree structure:

example.h5 : NeXus data file
  @H5PY_VERSION = "3.6.0"
  @creator = "write_nexus_file()"
  @default = "entry"
  @file_time = "2022-03-07 14:34:04.418733"
  @filename = "example.h5"
  entry:NXentry
    @NX_class = "NXentry"
    @default = "data"
    data:NXdata
      @NX_class = "NXdata"
      @signal = "data"
      data --> /entry/instrument/detector/image
    instrument:NXinstrument
      @NX_class = "NXinstrument"
      detector:NXdetector
        @NX_class = "NXdetector"
        image:NX_UINT8[1024,1024] = __array
          __array = [
              [76, 77, 78, '...', 75]
              [77, 78, 79, '...', 76]
              [78, 79, 80, '...', 77]
              ...
              [75, 76, 77, '...', 74]
            ]
          @target = "/entry/instrument/detector/image"
          @units = "counts"
      metadata:NXcollection
        @NX_class = "NXcollection"
        bitcoin_value:NX_CHAR = b'15000'
        detector_state:NX_INT64[] = 
        size_x:NX_INT64[] = 
        size_y:NX_INT64[] = 
        unique_id:NX_INT64[] = 

Note

Alternatively, the metadata shown in this example might be placed in the /entry/instrument/detector (NXdetector) group along with the image data since it provides image-related information such as size.

In the interest of keeping this example simpler and similar to the one above using the HDF5 File Writing Plugin, the metadata has been written into a NXcollection group at /entry/instrument/metadata location. (Compare with the NXcollection group /entry/instrument/NDAttributes above.)

using the nexusformat package¶

The nexusformat [8] package for python simplifies the work to create a NeXus file. Rewriting the above code using nexusformat:

import numpy as np
from nexusformat.nexus import *


def write_nexus_file(fname, image, md={}):
    """
    write the image to a NeXus HDF5 data file

    Parameters
    ----------
    fname : str
        name of the file (relative or absolute) to be written
    image : numpy array
        the image data
    md : dictionary
        key: value where value is something that can be written by h5py
             (such as str, int, float, numpy array, ...)
    """
    nx = NXroot()
    nx['/entry'] = NXentry(NXinstrument(NXdetector()))
    nx['entry/instrument/detector/image'] = NXfield(image, units='counts',
                                                    compression='gzip')
    nx['entry/data'] = NXdata()
    nx['entry/data'].makelink(nx['entry/instrument/detector/image'])
    nx['entry/data'].nxsignal = nx['entry/data/image']

    if len(md) > 0:
        # /entry/instrument/metadata (optional, for metadata)
        metadata = nx['/entry/instrument/metadata'] = NXcollection()
        for k, v in md.items():
            metadata[k] = v

    nx.save(fname, 'w')

    
if __name__ == "__main__":
    """demonstrate how to use this code"""
    import epics
    prefix = "13SIM1:"
    img = epics.caget(prefix+"image1:ArrayData")
    size_x = epics.caget(prefix+"cam1:ArraySizeX_RBV")
    size_y = epics.caget(prefix+"cam1:ArraySizeY_RBV")
    # edit the full image for just the binned data
    img = img[:size_x*size_y].reshape((size_x, size_y))

    extra_information = dict(
        unique_id = epics.caget(prefix+"image1:UniqueId_RBV"),
        size_x = size_x,
        size_y = size_y,
        detector_state = epics.caget(prefix+"cam1:DetectorState_RBV"),
        bitcoin_value="15000",
    )
    write_nexus_file("example.h5", img, md=extra_information)

Visualization¶

You can visualize the HDF5 files with several programs, such as: hdfview [9], nexpy [10], or pymca [11]. Views of the test image shown using NeXPy (from the HDF5 file) and caQtDM (the image from EPICS) are shown.

Get the installation instructions for any of these programs from a web search. Other data analysis programs such as MatLab, IgorPro, and IDL can also read HDF5 files but you might have to work a bit more to get the data to a plot.

Downloads¶

file	description
`attributes.xml`	The attributes file
`layout.xml`	The layout file
`mrinal_001.h5`	example NeXus HDF5 file written from EPICS
`write_nexus_file.py`	Python code to get images from EPICS and write a NeXus file
`write_nexus_file2.py`	write_nexus_file.py rewritten with nexusformat package
`example.h5`	example NeXus HDF5 file written from Python

EPICS Area Detector Examples¶

HDF5 File Writing Plugin¶

configuration files¶

`attributes.xml`¶

`layout.xml`¶

additional configuration¶

Example view¶

Python code to store an image in a NeXus file¶

using the h5py package¶

using the nexusformat package¶

Visualization¶

Downloads¶

Footnotes¶

NeXus-FAIRmat

Navigation

Related Topics

Google search

EPICS Area Detector Examples¶

HDF5 File Writing Plugin¶

configuration files¶

attributes.xml¶

layout.xml¶

additional configuration¶

Example view¶

Python code to store an image in a NeXus file¶

using the h5py package¶

using the nexusformat package¶

Visualization¶

Downloads¶

Footnotes¶

`attributes.xml`¶

`layout.xml`¶