MENTO: Automated near real-time data analysis at PETRA III

With the advent of next-generation X-ray detectors with large sensor areas and high sampling rates, photon science experiments are now having to deal with a data explosion. We present a scalable, near real-time data processing toolkit developed to address this challenge at PETRA III, which is currently in use at the coherence applications beamline P10 and at the macromolecular crystallography beamline P11. This toolkit runs automated analysis pipelines on high-volume data concurrently with data acquisition, thus providing quick feedback during experiments at the beamlines. Named mento (Maxwell-Enhanced Near real-Time Online analysis), the toolkit leverages the computing resources of the in-house HPC cluster 'Maxwell' to enhance analysis performance, and additionally takes advantage of the available distributed data storage to provide analysis results directly back to the experimenter with minimal delay. mento works seamlessly with the experiment control mechanisms used at P10, P11, and other PETRA III beamlines, and ensures that analysis on the HPC cluster is triggered automatically during data acquisition at the beamline, and that results are available for visualization at the beamline control hutch even though the analysis itself is performed remotely. mento thus helps the human-in-the-loop concentrate on novel science aspects of the experiment, without needing to manage the computational workload in a high data-rate regime. mento has been used with both in-house and commercial analysis software to achieve speedups of 50x-100x relative to the existing analysis pipelines, thus proving its utility for different kinds of experiments at PETRA III. The source code for mento is available at https://gitlab.desy.de/fs-sc/mento.