5 Conclusions

E-CAM has sought to prepare the extended E-CAM community for the demands of upcoming exascale systems. We have addressed core cross-cutting topics such as

• Load balancing - through the development of the ALL load-balancing library,

• Intelligent High Throughput Computing - through the development of the jobqueue_features library

• Accelerator portability - through the promotion of libraries such as Kokkos, and implementations in E-CAM codes,

• Portability of entire software stacks - through contributions to EasyBuild and, now, EESSI E-CAM has paved the way for the portability of entire application stacks and workflows to all the architectures that will be available through EuroHPC.

These alone are major contributions to preparing the E-CAM community, providing them with some of the tools that they will need to address major issues that will occur at scale: hardware diversity, portability of workflows, load imbalance, resiliency, coordinating ensemble petascale calculations. . . E-CAM has gone far beyond cross-cutting work, however, and has repeatedly worked with individual applications to evaluate and improve their performance, collaborating with other projects in the EuroHPC eco-system along the way. A primary example of this is given by the DL_MESO application which, as a result of E-CAM, has been shown to be scalable to 4096 GPUs (the largest GPU partition available in Europe).

In addition, E-CAM has raised awareness of best practices in scientific computing, attempting to encourage people to consistently use

• version control,

• documentation,

• continuous integration,

• modular software development,

• configure/build/install installation processes and the tools that can support these,

• open source software licences.

E-CAM has trained hundreds of scientists and exposed them to High Performance Computing (HPC) resources and tools that they might never otherwise have known. It has raised awareness of the EuroHPC eco-system among the thousands of scientists that are associated with CECAM, and trained the next-generation of those scientists so that HPC is a go-to tool in their computational workflows.

The scientific output from E-CAM is remarkable, both in terms of the development of new algorithms and methods and the related scientific publications with clear benefits for academic and industrial researchers. We produced 44 publications during E-CAM, and our articles have more than 1980 citations so far.

E-CAM has injected about 5.5 M€ funding in the community, to fund developments that are freely available on our software library, on the training portal and through open access scientific publications. Our software library is composed of 220 certified software modules, and ten more modules are work in progress.

A set of actions (software development, new formats for workshops) has been initiated that has clearly proved its strategic interest for the community and will be maintained and further developed.

Specific tools and initiatives will be continued and have funding at least for the next 48 months. A list is below.

Workshops

• Extended Software Development Workshops:

– Improving bundle libraries, 11-22 October 2021, CECAM-HQ (EPFL). Workshop website.

– HPC for simulation of complex phenomena, 11-15 October 2021, IIT (Genova). Workshop website.

– Inverse Molecular Design & Inference: building a Molecular Foundry (2nd part), Fall 2021, NUID UCD (Dublin). Workshop website.

• Scoping workshop:

– Simulation of open systems in Chemistry, Pharma, Food Science and Immuno-diagnostics: Rare-event methods at constant chemical potentials including constant pH (2nd part), Fall 2021, NUID UCD (Dublin).

Workshop website.

• State-of-the art work workshop:

– NOMAD/E-CAM workshop on Modeling materials at realistic space and time scales via optimal exploitation of exascale computers and AI, 1-3 November 2021, CECAM-HQ (EPFL). Workshop website.

• Industrial training event with the SME BiKi Technologies, 22 Nov 2021 (online).

Infrastructures

• The software library will continue to exist, supported by CECAM. In addition, several software development projects originated in E-CAM and stored on the software library will be completed during the sustainable phase (we have 10 modules that are work in progress).

• Our online training portal will continue to exist, supported by CECAM. The training portal will be further developed by CECAM, MaX CoE and the NCCR MARVEL, who will expand our efforts on the clowder platform and build a powerful and multipurpose repository with the potential to become a world reference for domain specific and general HPC training for the broad community of simulation and modelling.

• We have built a dedicated E-CAM section on the CECAM website, to ensure E-CAM’s most important results continue to be disseminated and communicated to the target groups even after the EU funding period, as well as the future activities that we plan to run beyond March 2021.

Software development projects

• Some of the work that originated in E-CAM will continue beyond the EU funding period. That is for example the case of the Molecular Biosensor work and the work on the pilot project Mesoscale Modelling of phoretic phenomena in binary fluids, both tackling clear industrial problems and in collaboration with an industrial counterpart.

• Other software development projects highly supported during E-CAM will continue expanding: OPS, n2p2,

MaZe, ESL, GPU DL_MESO etc.

The overall developments achieved in E-CAM have represented a considerable effort, due both to the original ambitious goals of the project and to the need to refocus them towards exascale after the first review. In spite of this, all original goals of the project have been exceeded and new commitments have been successfully tackled with intelligence and determination. We are extremely grateful to all those who have contributed, in their different roles, to this success and very proud of the community for its dedication and effectiveness. The Horizon2020 funding has covered only a fraction of the effort, material and in kind, of our team. The remarkable number of external contributions to our software library clearly demonstrates that the E-CAM approach has been appreciated and it’s well worth preserving. We will continue to work motivated by our conviction that the legacy of our efforts will be felt in the community even more in the future, as the increasing amount of EuroHPC resources begin to materialise. E-CAM has strongly contributed to put the community in a position where their entire software stack will be portable to these systems, and created the tools for them to use these resources to do great science without an overwhelming technical burden.

References

Acronyms Used

CECAM Centre Européen de Calcul Atomique et Moléculaire HPC High Performance Computing PRACE Partnership for Advanced Computing in Europe ESDW Extended Software Development Workshop SAW State-of-the-art Workshop SCOW Scoping Workshop WP Work-Package CoE Centre of Excellence HTC High Throughput Computing PDRA Postdoctoral Research Associate QMC Quantum Monte Carlo ESL Electronic Structure Library MD Molecular Dynamics OPS OpenPathSampling IP intellectual property SME Micro, Small and Medium-sized Enterprises ALL A Load-balancing Library MLWF Maximally localized Wannier functions DPD Dissipative Particle Dynamics EESSI European Environment for Scientific Software Installations MPM Material Point Method

Citations

[1] A. Ó Cais, D. Swenson, M. Uchronski, and A. Wlodarczyk, “Task Scheduling Library for Optimising Time-Scale Molecular Dynamics Simulations,” Aug. 2019. [Online]. Available: https://doi.org/10.5281/zenodo.3527643

[2] J. W. S. McCullough, R. A. Richardson, A. Patronis, R. Halver, R. Marshall, M. Ruefenacht, B. J. N. Wylie, T. Odaker,

M. Wiedemann, B. Lloyd, E. Neufeld, G. Sutmann, A. Skjellum, D. Kranzlmüller, and P. V. Coveney, “Towards blood flow in the virtual human: efficient self-coupling of HemeLB,” Interface Focus, vol. 11, no. 1, p. 20190119, 2021. [Online]. Available: https://royalsocietypublishing.org/doi/abs/10.1098/rsfs.2019.0119

[3] J. Castagna, X. Guo, M. Seaton, and A. O’Cais, “Towards extreme scale dissipative particle dynamics simulations using multiple GPGPUs,” Computer Physics Communications, vol. 251, p. 107159, 2020. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0010465520300199

[4] M. J. T. Oliveira, N. Papior, Y. Pouillon, V. Blum, E. Artacho, D. Caliste, F. Corsetti, S. de Gironcoli, A. M. Elena,

A. García, V. M. García-Suárez, L. Genovese, W. P. Huhn, G. Huhs, S. Kokott, E. Küçükbenli, A. H. Larsen,

A. Lazzaro, I. V. Lebedeva, Y. Li, D. López-Durán, P. López-Tarifa, M. Lüders, M. A. L. Marques, J. Minar, S. Mohr,

A. A. Mostofi, A. O’Cais, M. C. Payne, T. Ruh, D. G. A. Smith, J. M. Soler, D. A. Strubbe, N. Tancogne-Dejean,

D. Tildesley, M. Torrent, and V. W.-z. Yu, “The CECAM electronic structure library and the modular software development paradigm,” The Journal of Chemical Physics, vol. 153, no. 2, p. 024117, 2020. [Online]. Available: https://doi.org/10.1063/5.0012901

[5] P. G. Bolhuis and D. W. H. Swenson, “Transition Path Sampling as Markov Chain Monte Carlo of Trajectories:

Recent Algorithms, Software, Applications, and Future Outlook,” Advanced Theory and Simulations, vol. 4, no. 4, p. 2000237, 2020. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/adts.202000237

[6] D. W. H. Swenson, J.-H. Prinz, F. Noe, J. D. Chodera, and P. G. Bolhuis, “OpenPathSampling: A Python Framework for Path Sampling Simulations. 1. Basics,” Journal of Chemical Theory and Computation, vol. 15, no. 2, pp. 813–836, 2019, pMID: 30336030. [Online]. Available: https://doi.org/10.1021/acs.jctc.8b00626

[7] D. W. H. Swenson, J.-H. Prinz, F. Noe, J. Chodera, and P. Bolhuis, “OpenPathSampling: A Python

Framework for Path Sampling Simulations. 2. Building and Customizing Path Ensembles and Sample

Schemes,” Journal of Chemical Theory and Computation, vol. 15, no. 2, pp. 837–856, 2019. [Online]. Available: https://doi.org/10.1021/acs.jctc.8b00627

[8] V. Vitale, G. Pizzi, A. Marrazzo, J. Yates, N. Marzari, and A. Mostofi, “Automated high-throughput wannierisation,” npj Computational Materials, vol. 6, no. 1, p. 66, jun 2020. [Online]. Available: https: //doi.org/10.1038/s41524- 020- 0312- y [9] A. Ó Cais, “ESDW Technical Software Guidelines I,” Mar. 2016. [Online]. Available: https://doi.org/10.5281/ zenodo.841735

[10] M. Bialczak, A. Ó Cais, M. Uchronski, and A. Wlodarczyk, “Intelligent HTC for Committor Analysis,” Nov. 2020. [Online]. Available: https://doi.org/10.5281/zenodo.4382017

[11] A. A. Mostofi, J. R. Yates, Y.-S. Lee, I. Souza, D. Vanderbilt, and N. Marzari, “wannier90: A tool for obtaining maximally-localised wannier functions,” Computer Physics Communications, vol. 178, no. 9, pp. 685 – 699, 2008. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0010465507004936

[12] F. Fracchia, G. D. Frate, G. Mancini, W. Rocchia, and V. Barone, “Force Field Parametrization of Metal Ions From

Statistical Learning Techniques,” Journal of Chemical Theory and Computation, Nov. 2017. [Online]. Available: https://doi.org/10.1021/acs.jctc.7b00779

[13] D. Lopez Duran, E. Plesiat, M. Krompiec, and E. Artacho, “Gap variability upon packing in organic photovoltaics,”

PLOS ONE, vol. 15, no. 6, pp. 1–18, 06 2020. [Online]. Available: https://doi.org/10.1371/journal.pone.0234115

[14] R. Lot, F. Pellegrini, Y. Shaidu, and E. Küçükbenli, “Panna: Properties from artificial neural network architectures,” Computer Physics Communications, vol. 256, p. 107402, 2020. [Online]. Available: https: //www.sciencedirect.com/science/article/pii/S0010465520301843

[15] L. M. Ibele and B. F. E. Curchod, “A molecular perspective on tully models for nonadiabatic dynamics,” Phys.

Chem. Chem. Phys., vol. 22, pp. 15 183–15 196, 2020. [Online]. Available: http://dx.doi.org/10.1039/D0CP01353F [16] A. Ó Cais, “D6.7: E-CAM Software Platform V,” Dec. 2019. [Online]. Available: https://doi.org/10.5281/zenodo. 3598331

[17] K. Collins, A. O. Cais, D. Mackernan, and A. Mendonça, “Data Management Plan (version 2.0),” Oct. 2016. [Online]. Available: https://doi.org/10.5281/zenodo.3366123