Recent advances in Graph Neural Networks (GNNs) have transformed the space of molecular and catalyst discovery. Despite the fact that the underlying physics across these domains remain the same, most prior work has focused on building domain-specific models either in small molecules or in materials. However, building large datasets across all domains is computationally expensive; therefore, the use of transfer learning (TL) to generalize to different domains is a promising but under-explored approach to this problem. To evaluate this hypothesis, we use a model that is pretrained on the Open Catalyst Dataset (OC20), and we study the model’s behavior when fine-tuned for a set of different datasets and tasks. This includes MD17, the *CO adsorbate dataset, and OC20 across different tasks. Through extensive TL experiments, we demonstrate that the initial layers of GNNs learn a more basic representation that is consistent across domains, whereas the final layers learn more task-specific features. Moreover, these well-known strategies show significant improvement over the non-pretrained models for in-domain tasks with improvements of 53% and 17% for the *CO dataset and across the Open Catalyst Project (OCP) task, respectively. TL approaches result in up to 4× speedup in model training depending on the target data and task. However, these do not perform well for the MD17 dataset, resulting in worse performance than the non-pretrained model for few molecules. Based on these observations, we propose transfer learning using attentions across atomic systems with graph Neural Networks (TAAG), an attention-based approach that adapts to prioritize and transfer important features from the interaction layers of GNNs. The proposed method outperforms the best TL approach for out-of-domain datasets, such as MD17, and gives a mean improvement of 6% over a model trained from scratch.
Skip Nav Destination
Article navigation
14 May 2022
Research Article|
May 12 2022
Transfer learning using attentions across atomic systems with graph neural networks (TAAG)
Special Collection:
Chemical Design by Artificial Intelligence
Adeesh Kolluru
;
Adeesh Kolluru
1
Department of Chemical Engineering, Carnegie Mellon University
, Pittsburgh, Pennsylvania 15213, USA
Search for other works by this author on:
Nima Shoghi;
Nima Shoghi
2
Meta AI Research
, Menlo Park, California 94025, USA
Search for other works by this author on:
Muhammed Shuaibi;
Muhammed Shuaibi
1
Department of Chemical Engineering, Carnegie Mellon University
, Pittsburgh, Pennsylvania 15213, USA
Search for other works by this author on:
Siddharth Goyal;
Siddharth Goyal
2
Meta AI Research
, Menlo Park, California 94025, USA
Search for other works by this author on:
Abhishek Das
;
Abhishek Das
2
Meta AI Research
, Menlo Park, California 94025, USA
Search for other works by this author on:
C. Lawrence Zitnick;
C. Lawrence Zitnick
2
Meta AI Research
, Menlo Park, California 94025, USA
Search for other works by this author on:
Zachary Ulissi
Zachary Ulissi
a)
1
Department of Chemical Engineering, Carnegie Mellon University
, Pittsburgh, Pennsylvania 15213, USA
a)Author to whom correspondence should be addressed: zulissi@andrew.cmu.edu
Search for other works by this author on:
a)Author to whom correspondence should be addressed: zulissi@andrew.cmu.edu
Note: This paper is part of the JCP Special Topic on Chemical Design by Artificial Intelligence.
J. Chem. Phys. 156, 184702 (2022)
Article history
Received:
February 11 2022
Accepted:
April 25 2022
Citation
Adeesh Kolluru, Nima Shoghi, Muhammed Shuaibi, Siddharth Goyal, Abhishek Das, C. Lawrence Zitnick, Zachary Ulissi; Transfer learning using attentions across atomic systems with graph neural networks (TAAG). J. Chem. Phys. 14 May 2022; 156 (18): 184702. https://doi.org/10.1063/5.0088019
Download citation file:
Sign in
Don't already have an account? Register
Sign In
You could not be signed in. Please check your credentials and make sure you have an active account and try again.
Pay-Per-View Access
$40.00
Citing articles via
DeePMD-kit v2: A software package for deep potential models
Jinzhe Zeng, Duo Zhang, et al.
Related Content
Beyond independent error assumptions in large GNN atomistic models
J. Chem. Phys. (June 2023)
Permutationally invariant polynomial regression for energies and gradients, using reverse differentiation, achieves orders of magnitude speed-up with high precision compared to other machine learning methods
J. Chem. Phys. (January 2022)
FCHL revisited: Faster and more accurate quantum machine learning
J. Chem. Phys. (January 2020)
Screening of bimetallic electrocatalysts for water purification with machine learning
J. Chem. Phys. (August 2022)
Molecular force fields with gradient-domain machine learning (GDML): Comparison and synergies with classical force fields
J. Chem. Phys. (September 2020)