Perspective: Data infrastructure for high throughput materials discovery

The Materials Genome Initiative (MGI), established in 2011, has the goal of doubling the pace of advanced materials discovery, innovation, manufacturing, and commercialization in the United States. Six federal agencies participate with over 400 million dollars committed to-date for research across industry and academia. MGI aims to:

• Integrate research efforts across the US.

• Integrate experimental, computational, and theoretical research tool development.

• Make digital data available.

• Create a world class materials science and engineering workforce.

Advances in computing and informatics have aided materials discovery through systematic computational experiments powered by databases of first principle calculations and experimental data. Published data are estimated to be increasing exponentially at 4.7% per year in science in general.¹ It is critical for the users of reported data to have accurate measurements; however, if we assume a constant rate of errors (such as typographical errors) in what is published annually, the total amount of erroneous data would also be increasing at the same rate as that of publications. High quality databases can provide measurements with uncertainties, provenance of the data, and control data persistence—propagating data corrections and warnings about data sets with known problems. Human errors, inevitable during the document formatting and data transfer for publication, can typically be caught with basic visualization. Unfortunately, these checks are typically not performed by reviewers prior to publication.² The use of data capture tools in conjunction with comprehensive databases can be used prior to final publication to catch typographic errors and for cross referencing prior measurements with the soon-to-be-published data to provide some guarantee of data integrity. A shift in the review process of journals is necessary in a critical mass of high impact journals to reduce erroneous data publication and propagation, placing more responsibility on the journals to promote high quality data practices and requirements for authors to abide by and be responsible for meeting those requirements.

Comprehensive properties databases also facilitate discovery of coverage gaps within the literature. Such coverage gaps can be quickly and efficiently populated via High Throughput Experimental Methods (HTEMs). HTEM experiments are well designed, automated measurements conducted in specific compositional, temperature, and pressure ranges resulting in densely populated areas of experimental measurements. These combined results can then be fed forward into High Throughput Computational Methods (HTCMs). Such methods model large amounts of data to investigate materials space for new potential materials as determined by optimized parameters or “descriptors”³ that are developed for specific applications. Synergy between HTCM, HTEM, and data collection, communication, and curation is fundamental to achieving many of the goals of MGI.

This viewpoint paper focuses on the experimental data, in particular, and the associated communication, curation, and collection issues and potential improvements.

Awareness of importance of processing, structure, properties, and performance has existed for millennia. In the Chinese Bronze Age, composition effects were known to play a role in metals,^4,5 as was the effect of carbon on hardness of steel and the effect of mechanical deformation on strength as illustrated by Damascus steel.⁶ The prediction between structure and properties was predicted to play a role by Rene de Reaumur⁷ in the 18th century. Attribution of credit for the discussion of the relationships between properties and performance is difficult as its presence throughout history is ubiquitous.

The trial-and-error method for alloy design was relied upon until the 18th or 19th century. Gibbs provided the bases for modern methods for alloy design through applying thermodynamics to explain phase equilibrium.^8–10 These established equilibrium conditions remain important for the development of adequate processing conditions to produce a desired materials structure and resultant property set. Understanding of crystal structure grew over time and became one of four factors, along with electronegativity, valence, and atomic radii, used to predict solid solubility in mixtures of two metallic elements in the Hume-Rothery rules (1926).¹¹ These rules predict whether intermetallic precipitates form in a material, which are now known to be important for materials strength and ductility in alloys. The use of flow block-diagrams leads to understanding of how processing alters structure to produce the desired properties of a material. These systems-level approaches to materials design integration illustrate the complex relationships essential for alloy design for performance.¹² 

While phase diagrams still act as roadmaps for alloy development using temperature, pressure, and composition for 2-3-4-5 component systems, computational tools have been developed to model higher dimensional data that are not easily represented for complex system alloy phase predictions. These tools rely on information generated from atomistic and molecular dynamic models developed from the 1930s onward including Monte Carlo (MC), cluster variation method, Density Functional Theory (DFT), and phase field modeling (PFM).

Earlier efforts to calculate binary phase diagrams were pioneered by Van Laar,^13,14 while most modern efforts for binary, ternary, and multicomponent phase boundaries use the CALPHAD (CALculated PHAse Diagram) method developed by Kaufman and Bernstein in 1970.¹⁵ They provided descriptions of the general features of the method, including detailing computer programs for calculating binary and ternary phase diagrams. CALPHAD databases cover a broad range of fundamental thermomechanical and thermophysical properties spanning solution thermodynamics, molar volumes, diffusivities, and elastic constants.¹⁶ In recent years, “first principle” calculations from DFT have increasingly gained importance to supplement CALPHAD databases. Today’s thermodynamic descriptions can reliably produce phase diagrams and thermophysical and thermomechanical properties within the range of experimental errors.³ 

The Integrated Computational Materials Engineering (ICME) method is a powerful tool. However, just as users of a calorimeter must understand how the temperature, pressure, and heat evolution are monitored, the users of ICME must understand the algorithms and the data from which the models are developed. ICME consists of two main components: the databases and computational materials design algorithms/software that, when combined, are used for concurrent engineered systems.¹⁷ Unfortunately, data provenance and uncertainties are lost during the development of the correlations that underpin these methods, and users often fail to consider the impact this can have upon a final design. With free flow of information and dynamic data evaluation, new experimental data will be fed into an ecosystem that is capable of updating these recommendations transparently and without human intervention.

The diversity of software used in ICME is increasing, with various implementations continuing to pursue the similar goals. Software like FactSage, Pandat, or Thermo-calc⁶⁶ is already used to predict phase diagrams for multicomponent systems. Furthermore, these systems can be integrated with diffusion data to predict phase transformations or precipitation kinetics using software such as DICTRA, Pandat, or TC-PRISMA.⁶⁶ The combined kinetic and thermodynamic data can then be used to predict phase stability important for finite element analysis simulations or phase field simulations for engineering purposes. ICME approaches are used in many diverse fields from developing welding processes^18–21—utilizing relationships between solidification structure and composition and/or mechanical strength—to multiscale materials property predictions for composite materials²² or surface activity for promotion of chemical reactions.²³ 

The ICME process was used in the development of Ferrium S53,¹⁶ a corrosion-resistant replacement for cadmium-coated landing gear steels. The process initiated with the development of a flow block diagram to illustrate important processing-structure-property-performance relationships that would need to be modeled. Next, tradeoffs were determined using computational methods to optimize toward the desired properties (strength, corrosion resistance, core toughness, and fatigue resistance) and a new model was developed²⁴ to address the passive film developed on the steel. Finally, an iterative design and prototyping process were conducted (including full-scale microstructural characterization that provided experimental data for the models), where local correction factors were fed back into the design models for increasing accuracy in the next composition region of interest. The total time for alloy development was three years, which consisted of five iterations.

HTEMs are defined as “methodologies which allow accelerated synthesis and testing of materials for optimized performance, are uniquely suited to rapidly generate high quality experimental data, and hence represent the key enabling technologies to bring the computational materials design efforts of MGI to fruition.”²³ These studies allow for surveying data regions in ways that historically are not measured quickly and for the express purpose of feeding databases critical for materials discovery. An additional benefit of HTEM studies is the fundamental research opportunities—fundamental questions and modifications to testing procedures—developed during the data collection.

HTEM is essential to the future development of the CALPHAD method, though the methods are limited by experimental equipment–the development of experimental equipment for HTEM is imperative to develop validation for CALPHAD databases. Controlled compositional gradients can be generated via physical vapor deposition, but the structural properties being measured are impacted.^25,26 In an assessment of high entropy alloys Miracle¹² states, “a successful structural material must simultaneously satisfy a demanding balance of over a dozen different properties, and high throughput tools are not fully developed for all of these properties.”²⁷ Furthermore, Miracle highlights that for CALPHAD predictions, melting temperature data are essential, yet there are no HTEM methods to measure this property. Methods also do not exist for essential engineering properties like yield and ultimate tensile strength, ductility, damage tolerance, creep strength, and fatigue, all of which require extensive microstructural characterization.

New methods capable of collecting data continue being developed, e.g., the fairly new local electrode atom probe (LEAP) and focused ion beam (FIB) serial sectioning techniques that let experiments span from the nanometer to the millimeter in microstructural characterization, allowing for microstructural reconstruction and new insight and heuristics for future materials design. Presently, we are limited experimentally by the ability to extract data from imaging as a quantitative tool;²⁸ our understanding and ability to harness experimental information have been limited, particularly in active domains of 3d and 4d imaging. In a recent study by De Geuser et al.,²⁹ the precipitation kinetics in Co-Cu alloys were monitored using small angle x-ray scattering at a synchrotron beamline using simultaneous monitoring of a functionally graded specimen ranging in composition from 0 to 2 wt. % cobalt. From one experiment, they were able to cover a wide range of supersaturations and aging times for a range of compositions, repeating at two additional temperatures and comparing to standard precipitation models. The utility of data collected from these new techniques and from HTEM only can increase with improved data infrastructure for ICME.

HTCM is a combination of advanced thermodynamic and electronic structure methods with intelligent data mining and data construction with supercomputer architectures.³ The diversity of HTCM techniques matches the complex nature of materials problems. For future materials discovery, it is important to ensure that the algorithms and processes behind the computational methods are documented and that the future workforce is educated in these algorithms and processes. In HTCM, researchers are able to condense compositions of interest from thousands or millions of compositions or components down to a manageable number of experiments via optimization algorithms following complex searching algorithms.

For example, McLinden et al.³⁰ used evolutionary algorithms as a screening method to determine potential future organic refrigerants with low global warming potential (GWP) as well as a number of other criteria to be met. These criteria included a short atmospheric lifetime, ideal thermodynamic properties for cooling, low flammability and toxicity, and cost among other metrics. The initial dataset was populated from a public-domain database with 100 × 10⁶ potential compounds, of which 56 000 candidate molecules composing of a limited set of elements having 15 or fewer atoms in the molecule were selected. For these compounds, the GWP, flammability, critical temperature, and other thermodynamic factors were estimated and filtered to remove any elements with toxic or unstable functional groups³¹ resulting in 62 potential liquid refrigerants. Most of the resulting candidate compounds are used today, with only a few recommended potential refrigerants suggested for future research. A similar screening methodology was used for alloy design for wind turbines reducing the number of single crystal alloy compositions from the 100 000 to 6 potential alloys to investigate, with descriptors including estimates of creep resistance, microstructural stability, density, cost, and castability.³² 

Other researchers use thermodynamic genetic algorithms³³ to design or multiphysics modeling combined with neural networks for materials optimization.³⁴ Discussion of neural networks in alloy design exists,^34–36 where the need for undergraduate education in neural networks is proposed, given their extensive use in process control, process design, and alloy design. Bayesian methods taking advantage of ab initio and machine learning techniques have been used to predict properties; however, these methods require accurate predictions of properties or experimental data,³⁷ and the quality of output depends strongly on input data quality.^34,37–41

Infrastructure to collect, curate, and communicate heterogeneous data effectively is essential for enabling researchers to generate reliable predictions of material behaviors.¹⁷ Developing comprehensive databases with quantified reliabilities will be the keystone for automated, next-generation material development. In order to maximize impact for a given data resource, it must be considered in the context of the data ecosystem in which it operates. Considering elements such as accessibility, volume, reliability, and depth of metadata characterization is essential to guarantee value in a broader global information system. Drawing quantitative and salient connections between apparently disparate datasets fundamentally relies on communication of the full suite of parameters, including chemical compositions, processing history, and the final observed properties of the material.⁴² 

To address matters of data accessibility for the field of molecular liquids, the National Institute of Standards and Technology (NIST) has developed a number of informatics tools to help with literature identification,⁴³ manuscript digitization,⁴⁴ data communication,^45,46 and data evaluation/comparisons.^47–50 These technologies are expected to be expanded to support metal-based systems in the near future. Of particular note is how this suite of technologies serves to underpin a collaborative effort between the NIST Thermodynamics Research Center (TRC) and the five major journals that publish the majority of organic liquids thermophysical property data to facilitate data quality and reliable dissemination.

In the general case, knowledge acquisition is a bottleneck for the success of expert systems and machine learning algorithms. Accumulation of data is expensive, time consuming, and error prone, especially if content quality is held to a high standard. Automation substantively alleviates many of these challenges but is subject to the same limitations in the development process, and present technologies still require non-negligible human oversight. Moreover, as data volume growth rates continue to accelerate, the critical evaluation process becomes a bottleneck unless it as well is dynamic, accepting and propagating new information as it becomes available.

The complex and multidimensional nature of materials data requires either advanced algorithms for data collection or expert analysis and often requires both. The publication rate of thermophysical property data doubles about every twelve years² while “data” in general double on a 16-year horizon.¹ Consequently, errors in data, inherent with human-computer interaction and (depending on definitions) present in 1/3 of all manuscripts historically,² are also growing exponentially. Preventing erroneous information from corrupting data resources and overwhelming the capability to discern the good from the bad requires automated and enforced data integrity checks. Data can be verified for consistency with physical laws, checked for self-consistency and smoothness relative to claimed uncertainty, and verified against existing literature. It is essential to emphasize that failure of any consistency check does not necessarily indicate problematic data; rather, it provides the capability to invoke human review to determine if the set is problematic, if the uncertainties are overly optimistic, or indeed if new physics are appropriate to describe the system. Application of HTCM in cross validation of these datasets fundamentally combines physical law and existing literature checks, and therefore care must be taken that contradictory data are not necessarily dismissed out of hand.

The data collection methodology adopted by TRC at NIST, used in practical work flows for the past 15 years, relies heavily on human interpretation. Initial data capture is performed using form-driven software to guarantee well-formedness and completeness of metadata, and rudimentary visualizations performed at the earliest stages of capture serve to prevent the majority of errors introduced in the capture process and perform initial sanity checks on reported uncertainties. Following independent review for consistency of entered data with reported data, captured information is processed by ThermoData Engine (TDE),^47–50 an expert system implementing data integrity checks of the type previously discussed. Frequent errors found in reported tabulated data include: numerical typos, juxtaposed characters, column switches, fill down errors, and copy and paste errors. These can be often repaired to a substantial degree at capture time and in a worst case can be selectively marked as irreparable. In contrast, if data are reported in the form of a regression only, any error in manuscript preparation will render the entire set unusable. If data are irreparable, the metadata surrounding the reported data are still captured so as to avoid duplication of effort in case of rediscovery of the problematic document. By performing these checks prior to publication, it becomes possible to engage authors when the study is still fresh in their minds and thus often makes data repair possible.

Efforts have been made to automate the data collection process to varying degrees. Ultimately, interpretation of reported information requires a human decision, though this is abstracted away from the time of capture more heavily as the degree of automation is increased. These decision trees become quite complex if automated systems are expected to consume both new and historic data, as reporting standards and methods have changed substantially and repeatedly in the centuries since thermophysical properties were first reported. Frameworks completely lacking in human intervention following the programming stage³ will likely be essential in addressing issues arising from data publication growth; however, human interpretation will continue to be essential in collection of historical archives.

Multiple intermediate steps exist between collection of information from an experimental apparatus and inclusion of that data into design processes. Each step involves storing data in some intermediate format, and transformation between those states requires some mutually agreed upon communication structure. Within the context of communication of thermophysical property data, NIST and collaborators have developed standards under the auspices of the International Union of Pure and Applied Chemistry (IUPAC) for both data reporting in publication⁵¹ and machine-machine transmission of thermophysical property data.^45,46

Any scientific report, such as a scientific article, should be a clear and complete communication of an experimental result and the methods utilized to obtain it. This informs the research community how to replicate the result and how to judge the quantitative reliability of the result, particularly in light of potential new information about methodological limits. In a broad sense, this means reporting experimental results must balance between conforming to fixed patterns of data representation and providing the flexibility to provide critique and express novelty. Defining which information should be provided and defining its method of presentation not only facilitate clear communication of quantitatively important information but also allow for efficient verification of completeness and consistency of that information.

For quantitative information that is presented in a manuscript format, the clearest disposition is a “standalone table” where all experimental data are in a table that alone contains all the necessary experimental details to fully interpret the result.⁵¹ Examples of such tables can be found in Tables I–V. Table I illustrates characterization of material samples to be studied individually or as part of a multicomponent system. Examples are provided for an organic fluid, an organic crystalline material, and a pure metal while Table II illustrates an example of reporting composition of initial materials and a synthesized alloy. While details of formats may vary between journals, minor variations are irrelevant so long as the critical data are presented: material supplier, methods of purity determination, determined impurity contents, fully defined units, molar/mass/volume basis for composition provided (mass/molar/volume percent/fraction/parts per million), and standard or expanded uncertainty. Any processing and the initial and purity post processing should additionally be reported. For properties, any applicable statistical information (k-values, etc.) can also prove useful.

In the case of metal-based systems, many of the details important for thermophysical measurements are often omitted in the literature. For composition, “percent” or “ppm” is stated without consideration of its molar or mass basis. Grain size is not reported when resistivity measurements of metals are made, nor are dislocation densities reported in most cases, leaving results unclear. Thus the importance of metadata like processing history that provides insight to data users; knowledge of the annealing process or heat treatment temperatures, times, pressures, and atmospheres a material experienced allows for data users to assess the state of the measured material (dislocation density, additional impurities, etc.) when the authors cannot perform the actual measurements. Another commonly unreported value is phase fraction. These measurements are in multiphase materials; however, the measurements are time intensive and require stereological assumptions regarding grain boundary character, etc., unless some of the newer expensive technologies such as atom probe tomography, electron backscatter diffraction, or metallographic serial sectioning grain reconstructions are conducted. Historically, the identification of phases is not well represented, discussed, or reported in data tables or plots leaving ambiguity in archival data. Furthermore, in measurements of multiphase systems, it is important to quantify or qualify compositions of phases in materials when possible. Table III provides an example of well represented composition measurements of equilibrium phases’ conditions in multicomponent system.

Communication of phase transitions can also have ambiguity. Table IV illustrates well documented measurements where it was noted that data were smoothed and transitions are clearly described in the table. The clarification of phases present is valuable information that often goes unreported. Table V provides an example of data recorded associated with a phase transition found to occur in the Al-In system. The text in the document defines the reactions stated in the table, giving the authors’ interpretation of which measured values were associated with each transformation reaction. The accompaniment of Figure 1 and Table V ensures that the reactions can be accurately identified together. There is no ambiguity on the determination of what points on the figure are associated with which reaction. Again, all variables and their uncertainties are completely defined within the table.

Furthermore, pertinent data can be displayed in the table heading or footnotes as illustrated in the tables to communicate statements such as the data are smoothed or to define the system state. All variables are clearly and completely defined within the table or its footnotes and reported uncertainties are reported immediately adjacent to experimental observations if they vary between data points or within the footnotes if they do not. By presenting the uncertainties in the vicinity of the values of the observables rather than embedding them in the prose of an experimental methods discussion, readers consider uncertainties coincidentally with the values.

The above discussion should not be considered comprehensive but rather focuses strongly on a small number of shortfalls within the thermophysical scope. If the application of the data discussed above was taken to be an accurate prediction of microstructure, understanding of unaddressed elements such as diffusion kinetics and shear transformation behavior is essential. Development of reporting standards requires broad discussion within the community and an understanding of applications. By communicating which data are essential to accurate modeling to the experimental community and understanding how real experimental conditions should be translated into modeling assumptions, the impact and efficiency of each experimental effort can be maximized.

In contrast to the human-to-human dynamic of manuscripts, machine-to-machine communication cannot leave room for interpretation. Any value contained in a data communication standard that is not explicitly numerical, a member of an enumerated set, or a stable, external identifier is only useful for display to a human operator. As previously discussed, the report of novel experimental data or methods contains information of a nature that will not have been encountered previously. This means that there will be likely information loss in mapping data into a rigid format; whether that information loss is significant depends heavily on intended use cases.

In designing a communication standard between two computers, two layers must be considered: the semantic and the schematic. The semantic layer, or data serialization format, dictates the protocols by which a communicated document should be decoded into an in-memory representation. The schematic layer provides the necessary information for understanding the meaning of fields and how those fields are related. For a document to communicate bibliographic information, the schema would indicate that the document should contain exactly one year, a list of authors’ names, and the name of exactly one journal. The serialization format would indicate formalisms of text encoding (e.g., escape characters and text delimiters) within data fields and how to associate sets of characters with the schema’s fields.

Examples of semantic definitions include XML (Extensible Markup Language), JSON (JavaScript Object Notation), and CSV (comma-separated values) as well as a host of proprietary formats. The most common formats at present on the World Wide Web are XML and JSON. Both of these were constructed with the goal of providing a data format that is human-legible, well defined, and capable of representing complex data relationships. XML tends to be more rigid and more verbose and has more fully developed schema communication technologies. While JSON Schema has been proposed, its standard is presently still a draft⁵⁶ whereas the XML Schema Definition (XSD) has been a standard since 2001.⁵⁷ 

For distribution of thermophysical property data, TRC led the development of XML-based IUPAC-standard ThermoML. Since 2003 and in the context of the NIST-journal cooperation, data published in collaborating journals have been captured and serialized by NIST staff and shared in a free and open archive.⁵⁸ Within the chemical process simulation community, ThermoML is a common data import format. There also exist a number of tools within the community for direct consumption and interpretation of ThermoML, such as the ThermoML Opener to convert ThermoML files easily into spreadsheet formats⁵⁸ and Python-based ThermoPyL parser.^59,60

ThermoML was developed with thermophysical properties of molecular organics as a focus. While multiple extensions to the standard have since been published, the standard is limited in its representation of phases and material processing as these are not significant in the use cases or initial development. Substantial revisions are presently underway to address these limitations.

As the body of knowledge continues to grow exponentially, the importance of minimizing barriers to information propagation increases as well. In traditional critical evaluation processes, the evaluator must spend valuable time reviewing the literature to collect all relevant data, including errata and corrigenda, use their best judgement as to the relative reliability of each value, and format and disseminate final results. The import and export stages can be substantially expedited by proper indexing and development of communication standards, but so long as the middle stage requires human inspection, the volume of information transfer will be throttled by the availability of competent human review.

This challenge can be obviated, at least in part, through automation of the analysis process and effective development of data review tools. Trivial examples of this automation such as data normalization (e.g., converting all compositions to a molar basis) and plotting data from multiple literature sources against each other are simple tasks for a human to perform in concept but are time-consuming in practice. Higher level activities such as outlier detection, regression development, and consistency testing require more sophisticated implementations, but once the initial, substantial investment in development is made, the human cost for review of outputs and maintenance of software per unit of processed information can be reduced substantially. The importance of regular human review and robust and well defined quality tests cannot be over emphasized, as machine learning algorithms will provide outputs constrained only by the instructions that have been provided and thus can and will fail in many varied ways. As discussed previously, TRC developed TDE^{47–50,61–65} as an evolving expert system and as the first implementation of the dynamic data evaluation concept in the domain of thermophysical properties.

The call for experimental data for validation of theoretical calculations persists even with the “first principle”-based simulations that provide the underpinning for high throughput calculations. Future development of tools and software to assist with high throughput experimental methods, specifically those that deal with imaging, should be invested in and used in combination with high throughput computational methods, specifically high throughput screening processes, to decrease time for materials development processes. As data production rates grow, screening technologies for the peer review process become essential to ensure that published data are of high quality, and complete metadata representations are essential to their communication to ensure proper interpretation. The use of archives of historical data for validation prior to publication has been shown to be an effective tool to increase data reliability in properties of molecular organics and could be extended to other fields. However, this method requires substantial curation efforts and can only take advantage of data available to the general community. Without these protections, erroneous data will continue to permeate data resources, requiring consumers of information to be skeptical of their inputs and thus significantly impede progress.

This paper is a contribution of the U.S. government and is not subject to copyright in the United States.