DUNE's Python algorithms in Phlex

[brettviren]

Prompted by questions after Kyle's nice talk I thought it would useful to have a discussion that collects brief stories for using Python algorithms in Phlex.

I'll start with TRED which is at https://github.com/WireCell/tred/

This is a PyTorch based drift and detector response simulation somewhat similar in concept to Wire-Cell's but can use a 3D detector response as required for proper simulation of LAr pixel readout detectors It has various novel tricks to do that while not requiring enormous memory and processing. It has similar I/O as the WCT simulation but with different types. It requires G4 steps instead of point like depos as input and it can output pixel "hits" (sparse time and charge across the pixels). It can also emit true (unobservable) waveforms in a block-sparse format.

Today, it runs stand-alone by reading/writing files. Perhaps some discussion is needed at the batch processing level to decide if it is worth running as a Phlex algorithm or with other algorithm developers to decide if TRED-in-Phlex is a monolithic algorithm or is broken into multiple Phlex DFP graph nodes. Internally it is composed of PyTorch nn.Module layers like ML networks.

Effectively TRED must run either with GPU or many CPU cores, single CPU core running would be rather slow. In CPU mode, Torch thread pool is not compatible (?) with Phlex's TBB. In either GPU or CPU modes (or a mix) it has a dynamic rebatching feature that allows matching the job to available memory.

These are some of the issues when considering TRED as a Phlex algorithm.

Other Python algorithm developers may want to supply their own stories to help the Phlex team understand what framework-level stuff may be needed.