JARVIS-Leaderboard: A Large Scale Benchmark of Materials Design Methods

CHIPS Metrology Program

The CHIPS Metrology Program leverages NIST’s proven measurement science expertise to conduct research on measurements that are accurate, precise, and fit-for-purpose for the production of microelectronic materials, devices, circuits, and systems.    

The CHIPS Metrology Program aligns its research and development portfolios based on the identified metrology needs of the seven grand challenges:

1. Metrology for Materials Purity, Properties, and Provenance
2. Advanced Metrology for Future Microelectronics Manufacturing
3. Enabling Metrology for Integrating Components in Advanced Packaging
4. Modeling and Simulating Semiconductor Materials, Designs, and Components
5. ​​​​​​​​​​​​​​Modeling and Simulating Semiconductor Manufacturing Processes
6. Standardizing New Materials, Processes, and Equipment for Microelectronics
7. Metrology to Enhance Security and Provenance of Microelectronic-based Components and Products

In 2023, CHIPS for America awarded over $100 million in funding to over 29 projects addressing grand challenges 2, 3 and 4.  One of the projects addressing grand challenge is Multiscale Modeling and Validation of Semiconductor Materials and Devices.

Multiscale Modeling and Validation of Semiconductor Materials and Devices

This project will develop qualitative and quantitative models for advanced semiconductor heterostructures, including material properties and the impact of the interface quality via multi-scale, multi-fidelity computational approaches. This will help develop a comprehensive understanding of how current and next generation materials impact the performance of semiconductor devices, which is critical to U.S. Semiconductor Manufacturing.

In order to reach the project objective, the team developed a large scale, transparent benchmarking platform to enhance reproducibility. There have been recent claims that only 5 to 30 % research papers out there are reproducible, which is concerning (e.g. this article). 

This JARVIS-Leaderboard project provides a dedicated open-source infrastructure  that helps to tackle such challenges and help understand strengths and limitations of not only computational,  but also a few  experimental methods through benchmarking. 

JARVIS-Leaderboard: A Large Scale Benchmark of Materials Design Methods

On May 8, 2024, a paper related to this project, JARVIS-Leaderboard: A Large Scale Benchmark of Materials Design Methods, was published in Nature: NPJ Computational Materials. 

Abstract:

Lack of rigorous reproducibility and validation are significant hurdles for scientific development across many fields. Materials science, in particular, encompasses a variety of experimental and theoretical approaches that require careful benchmarking. Leaderboard efforts have been developed previously to mitigate these issues. However, a comprehensive comparison and benchmarking on an integrated platform with multiple data modalities with perfect and defect materials data is still lacking. This work introduces JARVIS-Leaderboard, an open-source and community-driven platform that facilitates benchmarking and enhances reproducibility. The platform allows users to set up benchmarks with custom tasks and enables contributions in the form of dataset, code, and meta-data submissions. We cover the following materials design categories: Artificial Intelligence (AI), Electronic Structure (ES), Force-fields (FF), Quantum Computation (QC) and Experiments (EXP). For AI, we cover several types of input data, including atomic structures, atomistic images, spectra, and text. For ES, we consider multiple ES approaches, software packages, pseudopotentials, materials, and properties, comparing results to experiment. For FF, we compare multiple approaches for material property predictions. For QC, we benchmark Hamiltonian simulations using various quantum algorithms and circuits. Finally, for experiments, we use the inter-laboratory approach to establish benchmarks. There are 1281 contributions to 274 benchmarks using 152 methods with more than 8 million data-points, and the leaderboard is continuously expanding. The JARVIS-Leaderboard is available here.