The Third Grand Challenge
Particle accelerators have almost reached their maximum accelerator gradients within a single cavity due to the fundamental barriers of radio frequency field emission and quench of the accelerator cavities. Plasma accelerators provide acceleration fields on the order of 1 TV/m and thus 1000 to 10000 times higher than conventional accelerators, enabling highly compact accelerator setups. Typically driven by a high-power laser of petawatt peak power, these accelerators enable unique applications ranging from driving brilliant compact x-ray and gamma-ray radiation sources to high flux radiation therapy for cancer. An all-optical X-ray free electron laser (XFEL) of a few tens of metres in size would be a revolution in X-ray imaging and would democratise this technology, as currently only a few XFELs exist worldwide that are driven by kilometre-long electron accelerators.
However, the technology requires a detailed investigation of the non-equilibrium plasma dynamics driven by the high power lasers happening on atomic, subfemtoseond time scale and nanometer length scales.
With upcoming large-scale facilities such as the Extreme Light Infrastructure (ELI), the Helmholtz International Beamline for Extreme Fields (HIBEF), the Helmholtz ATHENA distributed facility for plasma accelerators and the many specialised facilities within Laserlab Europe, experimental and simulation results can finally be compared at comparable resolution and detail. Thus, a new era of studying non-equilibrium high energy density (HED) matter with atomic precision and its applications has come within reach.
How can exascale simulations solve this grand challenge?
PIConGPU was created to study plasma accelerators at full resolution and scale. It was the first completely GPU-accelerated particle-in-cell plasma simulation code running on the then number one high performance compute system Titan at Oak Ridge National Laboratory in 2013, using 18,000 GPUs for simulating radiation processes in astrophysical jets. This work was rewarded as a Gordon Bell Prize finalist contribution. PIConGPU has popularised some of the central best practices in GPU acceleration of PIC codes, from data-local structures, super cells, repeated sorting of particle data, overlapping computation with offloading of simulation data to combining data parallelism with task parallelism. In 2016, it was again the first single-source plasma simulation code to be fully platform-independent, running on CPUs and GPUs from various vendors alike by utilising the Alpaka library. This was achieved using the Alpaka library developed by HZDR and introduced in the same year. Alpaka provides a redundant parallel hierarchy model for task-parallel kernel execution on heterogeneous hardware. It utilises a variety of compute backends such as CUDA, HIP/ROCm, OpenMP 2.0 and 5.0, OpenACC, Intel TBB, SYCL and std::thread. PIConGPU has also set a new open and FAIR data format for I/O, openPMD.
PIConGPU simulations of HED processes, laboratory astrophysics and laser-driven plasma accelerators will require an estimated 50 trillion particles and 2 trillion cells to resolve the interaction of high power lasers with solid density targets under realistic conditions. Inclusion of coherent and incoherent radiation transport and non-equilibrium atomic processes will lead to a tenfold increase of simulation time step duration and thus must be counteracted by a matching speed up in time to solution.
A full-scale simulation of a 10 GeV class XFEL driver will cover 10 orders of magnitude in space and 7 in time, as core physical processes that determine the final beam parameters happen at the highest resolution scale. Thus, both the numerical quality of the solvers as well as the temporal and spatial resolution of the simulation will be crucial for correctly predicting the final beam parameters.
Detailed analysis of the nonlinear, non-equilibrium plasma, radiation and atomic processes will require online, in-transit data analytics at the 10 TB/s scale, including coupling different simulation codes (hydro+PIC, radiation), large-scale AI for knowledge extraction and surrogate modelling and in-situ visualisation.