×

Indexed by

AiiDA 1.0, a scalable computational infrastructure for automated reproducible workflows and data provenance

Sebastiaan P. Huber1,2*, Spyros Zoupanos1,2, Martin Uhrin1,2, Leopold Talirz1,2,3, Leonid Kahle1,2, Rico Häuselmann1,2, Dominik Gresch4, Tiziano Müller5, Aliaksandr V. Yakutovich1,2,3, Casper W. Andersen1,2, Francisco F. Ramirez1,2, Carl S. Adorf1,2, Fernando Gargiulo1,2, Snehal Kumbhar1,2, Elsa Passaro1,2, Conrad Johnston1,2, Andrius Merkys6, Andrea Cepellotti1,2, Nicolas Mounet1,2, Nicola Marzari1,2, Boris Kozinsky7,8, Giovanni Pizzi1,2

1 National Centre for Computational Design and Discovery of Novel Materials (MARVEL), École Polytechnique Fédérale de Lausanne, CH-1015 Lausanne, Switzerland

2 Theory and Simulation of Materials (THEOS), Faculté des Sciences et Techniques de l'Ingénieur, École Polytechnique Fédérale de Lausanne, CH-1015 Lausanne, Switzerland

3 Laboratory of Molecular Simulation (LSMO), Institut des Sciences et Ingénierie Chimiques, École Polytechnique Fédérale de Lausanne (EPFL), Rue de l’Industrie 17, Sion, CH-1951 Valais, Switzerland

4 Microsoft Station Q, University of California, Santa Barbara, California 93106-6105, USA

5 Institut für Physikalische Chemie, University of Zürich, Switzerland

6 Vilnius University Institute of Biotechnology, Saulėtekio al. 7, LT-10257 Vilnius, Lithuania

7 John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, Massachusetts 02138, USA

8 Robert Bosch LLC, Research and Technology Center North America, 255 Main St, Cambridge, Massachusetts 02142, USA

* Corresponding authors emails: mail@sphuber.net
DOI10.24435/materialscloud:2020.0027/v1 [version v1]

Publication date: Mar 23, 2020

How to cite this record

Sebastiaan P. Huber, Spyros Zoupanos, Martin Uhrin, Leopold Talirz, Leonid Kahle, Rico Häuselmann, Dominik Gresch, Tiziano Müller, Aliaksandr V. Yakutovich, Casper W. Andersen, Francisco F. Ramirez, Carl S. Adorf, Fernando Gargiulo, Snehal Kumbhar, Elsa Passaro, Conrad Johnston, Andrius Merkys, Andrea Cepellotti, Nicolas Mounet, Nicola Marzari, Boris Kozinsky, Giovanni Pizzi, AiiDA 1.0, a scalable computational infrastructure for automated reproducible workflows and data provenance, Materials Cloud Archive 2020.0027/v1 (2020), doi: 10.24435/materialscloud:2020.0027/v1.

Description

The ever-growing availability of computing power and sustained development of advanced computational methods have contributed much to recent scientific progress. These developments present new challenges driven by the sheer amount of calculations and data to manage. Next-generation exascale supercomputers will harden these challenges, such that automated and scalable solutions become crucial. In recent years, we have been developing AiiDA (http://www.aiida.net), a robust open-source high-throughput infrastructure addressing the challenges arising from the needs of automated workflow management and data provenance recording. Here, we introduce developments and capabilities required to reach sustained performance, with AiiDA supporting throughputs of tens of thousands processes/hour, while automatically preserving and storing the full data provenance in a relational database making it queryable and traversable, thus enabling high-performance data analytics. AiiDA's workflow language provides advanced automation, error handling features and a flexible plugin model to allow interfacing with any simulation software. The associated plugin registry enables seamless sharing of extensions, empowering a vibrant user community dedicated to making simulations more robust, user-friendly and reproducible. This archive record contains the data to reproduce the figures on engine performance in the section "Event versus polling-based engine" of the paper entitled "AiiDA 1.0, a scalable computational infrastructure for automated reproducible workflows and data provenance". It also includes instructions to reproduce the actual data from scratch using AiiDA v1.1.1 and AiiDA v0.12.5.

Materials Cloud sections using this data

No Explore or Discover sections associated with this archive record.

Files

File name Size Description
README.md
MD5md5:6371a774bfa2cc9e3b33557897a96c07
4.0 KiB Readme file in markdown format with full description of contents of the `export_archive.aiida` and `supplementary.tar.gz` files, as well as instructions on how to reproduce the data of the paper.
export_archive.aiida
MD5md5:33de3d6fe9647d3b3fa0974a262db706
5.9 MiB AiiDA export archive containing the actual data used to produce the figures in the referenced publication.
supplementary.tar.gz
MD5md5:6ddab17a17f8f0a1d61feff134580070
13.1 KiB Archive containing the necessary scripts to completely reproduce the data contained within the `export_archive.aiida` AiiDA export archive and subsequently, analyse and plot the results.

License

Files and data are licensed under the terms of the following license: Creative Commons Attribution 4.0 International.
Metadata, except for email addresses, are licensed under the Creative Commons Attribution Share-Alike 4.0 International license.

External references

Journal reference
S. P. Huber, S. Zoupanos, M. Uhrin, L. Talirz, L. Kahle, R. Häuselmann, D. Gresch, T. Müller, A. V. Yakutovich, C. W. Andersen, F. F. Ramirez, C. S. Adorf, F. Gargiulo, S. Kumbhar, E. Passaro, C. Johnston, A. Merkys, A. Cepellotti, N. Mounet, N. Marzari, B. Kozinsky, G. Pizzi, (2020) submitted

Keywords

reproducibility workflows high-throughput automation MARVEL MaX SNSF PASC PRACE ERC Swissuniversities MarketPlace Intersect NFFA EMMC

Version history:

2020.0027/v1 (version v1) [This version] Mar 23, 2020 DOI10.24435/materialscloud:2020.0027/v1