Computational resources have grown exponentially in the past few decades. These machines make possible research and design in fields as diverse as medicine, astronomy, and engineering. Despite ever-increasing computational capabilities, direct simulation of complex systems has remained challenging owing to the degrees of freedom involved. At the cusp of exascale computing, high-resolution simulation of practical problems with minimal model assumptions may soon experience a renaissance. However, growing reliance on modern computers comes at the cost of a growing carbon footprint. To illustrate this, we examine historic computations in fluid dynamics where larger computers have afforded the opportunity to simulate flows at increasingly relevant Reynolds numbers. Under a variety of flow configurations, the carbon footprint of such simulations is found to scale roughly with the fourth power of Reynolds number. This is primarily explained by the computation cost in core-hours, which is also described by similar scaling, though regional differences in renewable energy use also play a role. Using the established correlation, we examine a large database of simulations to develop estimates for the carbon footprint of computational fluid dynamics in a given year. Collectively, the analysis provides an additional benchmark for new computations where, in addition to balancing considerations of model fidelity, carbon footprint should also be considered.

It is remarkable that a wallet-sized “rectangle”31 ensconced in a pocket has more computing power than the 70 lb machine aiding orbital calculations for the Apollo missions33 more than 50 years prior.32 Indeed, computing power has grown enormously over the preceding decades. In 1989, the fastest calculation was performed on a Cray 416 at 400 megaflops.19 Now, only three decades later, the present fastest computer, Frontier, boasts a theoretical peak performance of 1 exaflops,20 a billion-fold improvement over the Cray. These advances in computer hardware have greatly benefited the scientific community who rely on these machines to perform large numerical experiments. These include simulations of the heart,21 wind turbines,22 black holes,23 and worldwide climate.24 The increase in computational power has meant not only the types of problems that can be simulated are more complicated (for example, a section of a wing18 vs a full aircraft17) but also the resolution, hence accuracy of these models to predict relevant physical quantities, has also expanded. Large-scale computation has now become the standard for basic science research. A particular modality, namely, computational fluid dynamics (CFD), has been at the forefront of large-scale computation as long as computers have existed, dating back to the Manhattan project. In addition to academic research, computational fluid dynamics is regularly used in the design of car bodies to reduce drag,25 aircraft engine combustors to improve efficiency,26 and fluidized bed reactors used in chemical processing facilities.27 Although the wealth of computational resources enables contributions to both cutting-edge science and societal economic needs, there is a carbon footprint cost resulting from the electricity expense required to power these machines. Though conventions such as the Paris climate agreement have renewed the world's aspiration to decarbonize our energy sources, the majority of the world's energy use (from all sources) and 75% of electricity still come from nonrenewable sources.28 Therefore, it appears that some estimates of the carbon footprint associated with the increasing use of large-scale machines to conduct science are warranted. As a case study in this regard, we focus on large-scale computational fluid dynamics (CFD) simulations spanning nearly the past four decades. We pursue two analyses in the estimate of CFD's carbon footprint. First, we consider single-phase turbulent flows in canonical settings, including isotropic, channel, Couette, boundary layer, duct, and pipe flow, and develop a correlation of simulation carbon footprint as a function of the simulation Reynolds number. In the second part of this work, we consider a larger database of turbulent flow simulations conducted in 2022 to establish confidence intervals on the Reynolds numbers modelers are most likely to study as well as the expected value of the carbon footprint of a direct simulation of turbulence in 2022. Other statistics of this database are presented, showing that the individual carbon footprint of many CFD simulations can be relatively small, but their sum is significant. The goal of this work is to cast light on simulation carbon footprints, which we feel should be considered when designing new CFD studies.

We consider direct numerical simulations (DNSs) of turbulent flows where all length and timescales of the problem are resolved. A summary of these simulations is presented in Table I. We focus initially on single-phase incompressible turbulent flows in canonical settings, which have well-characterized resolution requirements.34 More complicated problems including flows with heat transfer, particles, and rough walls are considered in Sec. II B. In procuring this database, we only consider simulations where the number of cores and hours and/or core-hours were reported, along with information on the region (nation/state) where the simulations were conducted. Although this strict criterion greatly limits the number of historical simulations analyzed, this eliminates the use of theoretical estimates of carbon scaling. Reliance on well-characterized simulation data is advantageous over theoretical scaling since the present predictions are based on the actual number of processors and total simulation time, which, in practice, balance a variety of factors including the available resources and differences in averaging windows for computing flow statistics.

TABLE I.

Database of simulations used to construct carbon footprint correlation. The footprints are defined in terms of CO2 equivalent mass (CO2e). For isotropic turbulence, the Reynolds number is defined based on the Taylor scale, while all other cases are defined in terms of the friction velocity. *An assumed value. Simulation 6 is based on private communication from Mansour to Pope [Ref. 34, p. 355] and may reflect a preliminary calculation later published by Moser et al.40 Case 16 was actually run for 10 realizations of a single Kolmogorov time amounting to about 40 h of compute time. A total simulation time of 748 Kolmogorov times or about 5.5 turnover times as quoted in Yeung and Ravikumar14 was assumed to estimate a compute time for this Reynolds number, this longer timescale being characteristic of typical simulation times in DNS of isotropic turbulence.

Reference Flow Re Hours Core type Cores Model Memory (GB) Region kWh Mass (kg)
1. Kerr1   Isotropic  48.2  CPU  Cray 1-S  0.008 39  California  0.120 27  0.026 03 
2. Gotoh2   Isotropic  381  500  CPU  32  Fujitsu 5500/ 56  1*  Japan  320.95  149.5 
3. Bech3   Couette  82.2  154.07  CPU  Cray 1-YP  344 Mb  Norway  3.12  0.023 78 
4. Unger4,5  Pipe  180  38  CPU  Cray Y-MP4/464  .0594  Germany  3.05  1.03 
5. Eggels4   Pipe  180  40  CPU  Cray Y-MP4/464  .0792  Netherlands  3.21  1.2 
6. Mansour34   Channel  595  185  CPU  64  IBM SP2  4.096  California  237.74  51.46 
7. Kim6   Channel  180  62.5  CPU  Cray-XMP  56 Mb  California  5.01  1.08 
8. Jiménez7   Channel  2003  2929.69  CPU  2048  PowerPC970FX  4096  Spain  1.28 × 10 5  2.18 × 10 4 
9. Silero8   Boundary layer  2000  1373.29  CPU  32 768  PowerPC 450  32 768  Illinois  9.3 × 10 5  2.47 × 10 5 
10. Lee9,10  Channel  5186  381.47  CPU  52 4288  PowerPC A2  512 TB  Illinois  4.13 × 10 6  1.1 × 10 6 
11. Alfonsi11   Channel  200  51.39  CPU/GPU  6 CPU/1 GPU  Xeon X5660 Nvidia C-1060  28  Italy  25.17  8.15 
12. Alfonsi11   Channel  400  237.5  CPU/GPU  6 CPU/1 GPU  Xeon X5660 Nvidia C-1060  28  Italy  116.3  37.66 
13. Alfonsi11   Channel  600  461.11  CPU/GPU  18 CPU/3 GPU  Xeon X5660 Nvidia C-1060  84  Italy  677.41  219.37 
14. Zhang12   Duct  1200  2040.82  CPU  392  Xeon E5-2670  784  Spain  2.02 × 10 4  3460.579 
15. Gavrilakis13   Duct  150  180.36  CPU  Cray 2  0.262  France  14.49  0.7429 
16. Yeung14,15  Isotropic  1300  2989  CPU/GPU  135168/18432  IBM Power 9 Nvidia V-100  1 867 776  Tennessee  3.92 × 10 7  1.07 × 10 7 
17. Vela-Martín16   Channel  2000  507.81  GPU  128  Tesla P-100  2048  Switzerland  2.78 × 10 4  320.08 
18. Vela-Martín16   Channel  5303  2734.38  GPU  512  Tesla P-100  8192  Switzerland  5.98 × 10 5  6894 
Reference Flow Re Hours Core type Cores Model Memory (GB) Region kWh Mass (kg)
1. Kerr1   Isotropic  48.2  CPU  Cray 1-S  0.008 39  California  0.120 27  0.026 03 
2. Gotoh2   Isotropic  381  500  CPU  32  Fujitsu 5500/ 56  1*  Japan  320.95  149.5 
3. Bech3   Couette  82.2  154.07  CPU  Cray 1-YP  344 Mb  Norway  3.12  0.023 78 
4. Unger4,5  Pipe  180  38  CPU  Cray Y-MP4/464  .0594  Germany  3.05  1.03 
5. Eggels4   Pipe  180  40  CPU  Cray Y-MP4/464  .0792  Netherlands  3.21  1.2 
6. Mansour34   Channel  595  185  CPU  64  IBM SP2  4.096  California  237.74  51.46 
7. Kim6   Channel  180  62.5  CPU  Cray-XMP  56 Mb  California  5.01  1.08 
8. Jiménez7   Channel  2003  2929.69  CPU  2048  PowerPC970FX  4096  Spain  1.28 × 10 5  2.18 × 10 4 
9. Silero8   Boundary layer  2000  1373.29  CPU  32 768  PowerPC 450  32 768  Illinois  9.3 × 10 5  2.47 × 10 5 
10. Lee9,10  Channel  5186  381.47  CPU  52 4288  PowerPC A2  512 TB  Illinois  4.13 × 10 6  1.1 × 10 6 
11. Alfonsi11   Channel  200  51.39  CPU/GPU  6 CPU/1 GPU  Xeon X5660 Nvidia C-1060  28  Italy  25.17  8.15 
12. Alfonsi11   Channel  400  237.5  CPU/GPU  6 CPU/1 GPU  Xeon X5660 Nvidia C-1060  28  Italy  116.3  37.66 
13. Alfonsi11   Channel  600  461.11  CPU/GPU  18 CPU/3 GPU  Xeon X5660 Nvidia C-1060  84  Italy  677.41  219.37 
14. Zhang12   Duct  1200  2040.82  CPU  392  Xeon E5-2670  784  Spain  2.02 × 10 4  3460.579 
15. Gavrilakis13   Duct  150  180.36  CPU  Cray 2  0.262  France  14.49  0.7429 
16. Yeung14,15  Isotropic  1300  2989  CPU/GPU  135168/18432  IBM Power 9 Nvidia V-100  1 867 776  Tennessee  3.92 × 10 7  1.07 × 10 7 
17. Vela-Martín16   Channel  2000  507.81  GPU  128  Tesla P-100  2048  Switzerland  2.78 × 10 4  320.08 
18. Vela-Martín16   Channel  5303  2734.38  GPU  512  Tesla P-100  8192  Switzerland  5.98 × 10 5  6894 
To estimate the carbon footprint for each simulation, we use the open-source Green Algorithms calculator v2.229 based on the work of Lannelongue et al.30 The calculator estimates the carbon footprint from the following formula:
C O 2 e = cores × hours × k W / core × C O 2 e / kWh .
(1)

The carbon footprint of a calculation depends primarily on how many computational hours were needed, the power requirement of the processors, and the carbon intensity of the power source. The last quantity depends critically on the fuel source for the electricity: Renewable sources have low carbon intensity, and fossil fuels have high carbon intensity. The calculator estimates the carbon intensity of a calculation by accounting for the region in which the calculation is done. For example, a hypothetical simulation run in California on the same computing resources as in France has different estimated carbon footprints because the energy budget of renewable vs nonrenewable fuels is different in these regions, and, therefore, the carbon intensity associated with electricity production needed for the simulations is also different.

We consider the flow Reynolds number R e = U L / ν of the various simulations as the principal variable for our study. Here, U and L are, respectively, reference flow velocity and length scales, and ν is the kinematic viscosity. The Reynolds number is characterized by the relative importance of inertial and viscous forces and is related to the range of flow scales in the problem. Higher Reynolds number flows contain more features to resolve. Hence, the larger the Reynolds number, the larger the simulation (mesh points/time steps) and the more machine-hours required to simulate that flow. Consequently, a carbon footprint correlation based on Re abstracts away many factors used in simulation design, allowing a simple available metric to estimate a simulation's carbon footprint. In this study, we use two definitions of the Reynolds number: For wall-bounded flows, the Reynolds number is based on the friction velocity, while for isotropic turbulence, it is based on the Taylor scale.34 These separate definitions ameliorate comparison among disparate flow types where a single common definition of the Reynolds number is not available. The relevant Reynolds numbers for the respective flow types were reported in each study and reproduced in Table I. The database spans two decades in Reynolds number across six canonical turbulent flows. The computations were conducted on CPUs, GPUs, and hybrid architectures in 11 regions across the U.S. and Europe. Figure 1 shows the growth in Reynolds number over the years for the selected simulations. The exponential change in Reynolds number over this time underscores the growing power of modern computing and the necessity of developing carbon footprint estimates for these simulations.

FIG. 1.

Scaling of Reynolds number over time for simulations used to develop the carbon footprint correlation. The data are colored by the type of turbulent flow studied. Each datapoint represents a simulation from Table I.

FIG. 1.

Scaling of Reynolds number over time for simulations used to develop the carbon footprint correlation. The data are colored by the type of turbulent flow studied. Each datapoint represents a simulation from Table I.

Close modal

There are limitations in the methodology used to estimate carbon footprints we report. The Green Algorithms GUI does not have all processor types listed in Table I incorporated into its database, so most cases use a typical power per core using the “any”/“other” processor type. This assumption is more accurate for recently conducted simulations (say those conducted from 2010 onward) whose processor type and power use will align most closely with a typical modern processor provided in the GUI. Conditioning our analysis on only these simulations still shows a strong scaling with Reynolds number though the correlation coefficient is reduced; therefore, we conduct our analysis on all of the simulations in Table I with the acknowledged limitation. The power estimates in this work are based on core thermal design power (TDP) specifications,30 which may not reflect the actual power consumed during a calculation.47 Code optimizations can be employed to improve energy use on a machine.46 Careful power measurements of different codes run on different machines confirm that the TDP values are not, in general, equal to the measured power when running an application, though the difference in these values was usually less than a factor of two across the cases considered.47 We also do not consider additional power requirements needed to house and cool the computing facilities, and we also assume that for the total compute-time, the processors are running (there is no idle time). Usage factors have been considered in other works,30 and there is typically not an order of magnitude difference in estimated power usage considering these factors. Most cases in Table I reported either the number of cores or core-hours, but not both. The simulation time was then inferred from the reported core count and core-hour values and/or by multiplying the number of time steps by the time per time step. Memory use was not typically reported but usually could be inferred reasonably from specification sheets of the individual processors or from the cluster information pages. Typically, memory usage accounted only for a few percent of total energy use. Finally, we are not able to consider local differences in renewables' contribution to electricity use, for example, if a computing cluster draws its power directly from onsite photovoltaic cells. All of these assumptions affect the precise magnitude of energy consumption without affecting the order of magnitude and still allow incorporation of the Green Algorithms calculator to predict a scaling behavior for energy usage/carbon production vs flow Reynolds number. It should be noted that the carbon intensity of electricity used in this work is contemporary regional estimates. In other words, the reported carbon footprints are our estimates of the cost of running these simulations today, not an estimate of how much carbon these simulations produced at the time they were run.

Figure 2 shows the carbon footprint of the turbulent flow simulations from Table I vs their Reynolds number. Noteworthy is the fact that the carbon footprint can be fit with a simple power law scaling,
C O 2 e ( kg ) = e 18.52 × R e 3.73 .
(2)
FIG. 2.

Scaling of carbon footprint for various turbulent flows. Comparison curves of societal activities are provided to make concrete the carbon mass values. The carbon footprint for an economy transcontinental ticket is estimated at 570 kg.30 Per capita annual automobile use is estimated at 251 g/km in the U.S.30 times the annual average distance traveled of 13 476 miles.43 

FIG. 2.

Scaling of carbon footprint for various turbulent flows. Comparison curves of societal activities are provided to make concrete the carbon mass values. The carbon footprint for an economy transcontinental ticket is estimated at 570 kg.30 Per capita annual automobile use is estimated at 251 g/km in the U.S.30 times the annual average distance traveled of 13 476 miles.43 

Close modal

Here, the Reynolds number is defined based on the Taylor scale in isotropic flows, and based on the friction scale for wall-bounded settings. The near fourth power in scaling is consistent with the scaling with respect to core-hours which scales with nearly the same factor [Fig. 3(c)]. Also note the wide distribution in wall clock times, ranging from hours to months, depending on the Reynolds number [Fig. 3(b)].

FIG. 3.

Breakdown of carbon footprint scaling in terms of (a) cores, (b) hours, (c) core-hours, and (d) normalized mass per core-hour. The reference lines in (d) are provided to make tangible the carbon mass production per core-hour.

FIG. 3.

Breakdown of carbon footprint scaling in terms of (a) cores, (b) hours, (c) core-hours, and (d) normalized mass per core-hour. The reference lines in (d) are provided to make tangible the carbon mass production per core-hour.

Close modal

Interestingly the core-hour scaling, R e τ , λ 3.5 (with the subscripts included to emphasize the consistent Reynolds numbers, R e λ or R e τ) is less than the theoretical prediction of R e λ 6.34,37 Pope [Ref. 34, p. 350] has shown that this theoretical scaling, valid for high Reynolds number isotropic turbulence, can be relaxed to R e λ 5.5 using a time step that scales with the Kolmogorov time. The discrepancy in the observed correlation compared with these theoretical scalings may arise from modeler freedom in balancing resolution of large and small scales in turbulent flows, differences in choices for CFL number, length of simulation for time averaging, finite Re not achieving the asymptotic turbulence scalings, and machine architecture and clock speeds. Numerical modeling choices such as large-scale forcing,38 which can increase the Reynolds number for a given domain size in isotropic turbulence and non-uniform grids in wall-bounded flows,6 also affect scaling. The computational requirements of wall-bounded flows, particularly turbulent boundary layers, have also been considered by Choi and Moin48 and Yang and Griffin,49 hereafter CM and YG. For direct simulation of a turbulent boundary layer, these works, respectively, estimated the computational resources (number of grid points times the number of timesteps) as N N t R e L 3.52 (CM) and N N t R e L 2.91 (YG), where ReL is the Reynolds number based on the streamwise distance. These expressions can be cast in terms of friction Reynolds number [the consistent Reynolds number for Eq. (2)] using δ / L R e L 1 / 7,48,49 yielding R e δ = R e L 6 / 7, and along with R e τ R e δ 0.88 (Pope34 p. 279, 312) gives the respective estimates N N t R e τ 4.67 (CM) and N N t R e τ 3.86 (YG). These theoretical estimates are in better agreement with the scalings observed for carbon footprint, R e τ , λ 3.73 Eq. (2), and core-hours, R e τ , λ 3.5.

While the observed core-hour and carbon footprint scaling, nevertheless, grow rapidly with Reynolds number, this analysis suggests that DNS may be more affordable than theoretical scalings would suggest. These various considerations underscore the importance of using fielded simulations to estimate the carbon footprint of this modality rather than theoretical scalings. Using the estimate provided by Eq. (2) as a starting point, increased documentation of computational resources used in fluid dynamics simulations can better establish the true fielded cost of DNS and corroborate theoretical scalings.

The carbon footprint for a simulation depends greatly on where the simulation is run since the percentage of renewable energy that goes toward electricity generation varies greatly with region. For most of the cases considered, on a core-hour basis, and for the same Reynolds number, we see around a factor of 5 differences in carbon footprint when run in different locales. For example, the simulations of Hoyas and Jiménez,7 Silero et al.,8 and Vela-Martín16 all considered wall-bounded turbulent flow simulations at R e 2000, yet the carbon footprint of these respective simulations differed by three orders of magnitude. For these three simulations, the carbon footprint per core-hour differed by less than a factor of 2. Therefore, the difference in renewable energy percentage that goes toward electricity generation in the respective regions (Spain, Illinois, and Switzerland) plays a significant role in controlling the carbon footprint of “nominally equivalent” simulations. This suggests that all else equal, modelers should submit proposals to and target machines whose electricity use relies on greater fractions of renewable energy. Though there is considerable scatter, much of the data are consistent with a cost of 4.2 g C O 2 e / core hour [dashed line in Fig. 3(d)], which is about an American teaspoon (tsp) of sugar by mass. Another observation is that a factor of two increase in Reynolds number represents a factor of 13 increase in carbon.

Having established a Reynolds number dependent carbon footprint correlation for direct simulations of turbulence, we turn our attention toward the questions: (1) What is a typical Reynolds number simulated? (2) What is the typical carbon footprint of a direct simulation? and (3) For a large database of simulations, what is the total footprint?

Google Scholar searches were conducted on July 6, 2023, and September 15, 2023, to develop a large database of turbulent flow direct numerical simulations (see database II in the Supplementary material). We considered articles exclusively from 2022. In an effort to incorporate simulations whose run criteria best matched the assumptions used to develop the carbon footprint correlation, we searched exclusively for direct numerical simulations with the search criteria: Intitle: direct AND simulation AND turbulence as well as Intext: direct AND simulation AND turbulence. These searches yielded tens of thousands of hits. The task at hand would be to determine the Reynolds number(s) of the simulations contained within a list of simulations chosen from the list of papers. Since the correlation Sec. II was developed for isotropic and wall-bounded flows whose characteristic Reynolds numbers are based on the Taylor microscale and friction velocity, respectively, it was important to ensure the reported Reynolds numbers in these works were consistent with these definitions. A number of other constraints were imposed including exclusion of papers that mentioned direct simulation but were not in fact on that topic, studies using large eddy simulation (LES) or Reynolds Averaged Navier–Stokes (RANS), experimental works on turbulence, and relevant papers where the appropriate Reynolds number was not reported. A few papers were not incorporated into the final database because, while they were reported as direct numerical simulations, the reported spatial resolution was too coarse in this author's judgment to warrant the DNS label. Most importantly, care was taken to ensure that no duplicate simulations were used in the database—either duplicate scholar articles or papers presenting analysis of turbulence simulations that were previously published. For all of these reasons, the author determined no automated data scraper was up to the task of developing such a database, and, consequently, this task was conducted manually. This last point is mentioned to raise awareness of how databases are catalogued with the hope that uniform standards may ease processing of archival data in the future.

In all, 645 Google Scholar entries were examined, and 148 papers ultimately selected that met the above criteria. This yielded a total of 902 direct simulations of turbulence, sufficient for establishing confidence intervals. In summary, this database includes isotropic, channel, Couette, boundary layer, pipe, and duct flows since these flows were amenable to the definitions of Reynolds number consistent with the developed correlation. Many direct simulations of other turbulent phenomena were not considered including combustion, free shear, and flow over bluff bodies, and Rayliegh–Benard convection, since, in general, it was not straightforward to translate the relevant parameters of these flows into the required Reynolds numbers consistent with the carbon footprint correlation. License was given to additional physics not considered in the establishment of the carbon footprint: we considered new factors including compressibility, particle-laden and multiphase flows, heat transfer, and flows with rough walls and riblets. These additional physics are uniformly biased toward increasing the resolution requirements for a given Reynolds number while still likely following the same trend of carbon footprint scaling with Reynolds number R e 3.73. The results to be presented in Sec. III B also do not account for “trial runs” (scaling, debugging, and scoping studies) performed prior to the production calculation. These preliminary calculations are frequently left out of the final published results, yet still contribute to the carbon footprint of the final simulation product.

Figure 4 shows the probability density function (pdf) of simulated Reynolds numbers along with the corresponding pdf of carbon footprint for database II. Additionally, Table II presents a summary of Reynolds number and carbon footprint statistics. Low Reynolds number simulations are most prevalent, which correspond to lower carbon footprint on a per simulation basis. The mode simulation Reynolds number is 180 with 34% of the database falling within the Reynolds numbers 180 ± 30. The prevalence of this particular Reynolds number may stem from the seminal turbulent channel flow calculations at Re = 180 of Kim et al.6 What was then a pioneering calculation owing to computational expense is now, nearly four decades later a routine Reynolds number considered in computational fluid dynamics simulations. The mean Reynolds number of this sample is 315, demonstrating the skewness toward less frequently simulated larger Reynolds number calculations. The corresponding carbon footprint of the sample mean and mode Reynolds numbers are 18.4 and 2.3 kg, respectively. Another way to interpret the expected carbon footprint of a direct simulation is to calculate the confidence interval of the mean directly from the carbon footprint distribution. The corresponding 95% confidence interval for the mean carbon footprint of a DNS is 8.5 × 10 3 ± 1.6 × 10 4 kg, considerably larger than the carbon footprint associated with the expected Reynolds number. This discrepancy underscores the disproportionate role that lower frequency large Reynolds number calculations contribute in their respective footprints and are consistent with a rapid scaling of carbon footprint with Reynolds number R e 3.73. Indeed, of the 902 simulations considered here, 53 (6% of the database) are simulations conducted with R e 1000. Remarkably, the largest simulation in this database, conducted at R e = 10 4, by itself constitutes 94% of the carbon footprint of the entire database, 7.2 × 10 3 tons out of 7.7 × 10 3 total tons of carbon for the 902 simulations from 2022. This observation underscores the large contribution that even a single “hero” calculation—one that uses all or a large fraction of a machine for a significant time—can contribute toward CFD's carbon production in a given year. It also suggests that the title “DNS” is not descriptive enough to assess whether a simulation will be carbon intensive. Two DNSs at different Reynolds numbers may differ in their carbon footprint by many orders of magnitude.

FIG. 4.

(a) pdf of Reynolds number and (b) pdf of carbon footprint of simulations from database II.

FIG. 4.

(a) pdf of Reynolds number and (b) pdf of carbon footprint of simulations from database II.

Close modal
TABLE II.

Reynolds number and carbon footprint statistics of database II. Two 95% confidence intervals are reported for the mean carbon footprint. The first is the carbon footprint associated with the mean simulation Reynolds number and its confidence bounds, that is C I ( R e ¯ ) = e 18.52 × R e ¯ 3.73, where R e ¯ = [ 285 , 315 , 345 ]. The second is the confidence interval of the carbon footprint pdf, where 8.5 × 10 3 kg represents the sample average carbon footprint of a simulation, and the true average carbon footprint of a simulation is expected to occur within ± 1.6 × 10 4 kg of the sample mean, with 95% confidence.

Statistic Mean (confidence interval) Median Mode Min Max
1. Reynolds number  R e ¯ = 315 ± 30  188  180  104 
2. Carbon footprint (kg)  (1) C I ( R e ¯ ) = ( 12.6 , 18.4 , 25.9 )  2.7  2.3  3 × 10 5  7.2 × 10 6 
(2)  C I ( R e ) = 8.5 × 10 3 ± 1.6 × 10 4 
Statistic Mean (confidence interval) Median Mode Min Max
1. Reynolds number  R e ¯ = 315 ± 30  188  180  104 
2. Carbon footprint (kg)  (1) C I ( R e ¯ ) = ( 12.6 , 18.4 , 25.9 )  2.7  2.3  3 × 10 5  7.2 × 10 6 
(2)  C I ( R e ) = 8.5 × 10 3 ± 1.6 × 10 4 

Placed in the context of annual human activities, the carbon footprint of this database corresponds to the annual carbon footprint of about 2741 people, assuming 2.8 tons per capita in the U.S.36 While the largest simulation constitutes the majority of this footprint, the remaining 6% of simulations still amount to a non-negligible carbon footprint of 160 persons. In assessing the carbon impact of a simulation, it is worth considering not only the carbon footprint of a simulation at a given Reynolds number but also the cumulative contribution across all Reynolds numbers, as shown in Fig. 5. Indeed, while the majority of simulations are conducted at low Reynolds number, their cumulative carbon footprint is significant. Noticeable upticks in the cumulative distribution are observed around Reynolds numbers of 180 and 395, demonstrating the carbon impact of a simulation at a given Reynolds number depends both on the carbon footprint associated with a given Reynolds number, and how frequently that Reynolds number is simulated. The latter points to behavioral patterns of computational fluid dynamicists who may be biased to simulate particular Reynolds numbers owing to historical precidence for that simulation type in the literature,6,40 as well as tradeoffs scientists must consider in conducting higher Reynolds number simulations to elucidate more fundamental features of turbulent flows, while also balancing machine-time allocations on various computing centers.

FIG. 5.

Cumulative carbon footprint vs Reynolds number of database II. We hypothesize the jumps in carbon footprint at Re = 180 and 395, which may reflect the lasting influence of Kim et al.6 and Moser et al.40 in influencing modeler choices in simulation Reynolds numbers. The Indianapolis carbon footprint was chosen to put into context the simulation footprints compared to that of a large size U.S. city; in particular, Indianapolis was the host of the 2022 APS DFD conference where this work was originally presented.

FIG. 5.

Cumulative carbon footprint vs Reynolds number of database II. We hypothesize the jumps in carbon footprint at Re = 180 and 395, which may reflect the lasting influence of Kim et al.6 and Moser et al.40 in influencing modeler choices in simulation Reynolds numbers. The Indianapolis carbon footprint was chosen to put into context the simulation footprints compared to that of a large size U.S. city; in particular, Indianapolis was the host of the 2022 APS DFD conference where this work was originally presented.

Close modal

This review has focused primarily on fluids applications, and it is expected that similar takeaways would apply to other fields making heavy use of computational resources including computational biology, medicine, machine learning, atmospheric science, and finance. As large-scale scientific computing becomes routine in both academia and industry, greater responsibility is warranted. We are now stewards of these powerful machines. It is incumbent we use their might judiciously. This work focused on direct numerical simulations owing (1) to their computational expense and (2) the agreement in their definition, requiring sufficient resolution of the largest and smallest spatial and temporal scales.6,34,45 LES and RANS simulation methodologies are likely performed more frequently in academia and industry owing to their relative low cost compared with direct simulation. However, lack of an agreed-upon definition for the resolution requirements for these modalities makes it difficult to provide an assessment of their respective carbon footprint scaling for a given Reynolds number. Analysis of database II shows that a large number of low Reynolds number (cheap) simulations may still sum to an appreciable carbon footprint. In the development of database II, many candidate simulations were discarded as they constituted simulations re-analyzed from prior years. This is a welcome behavior which scientists should accelerate in their practice. Unfortunately, simulation databases are too often used only once for a particular research article, with the data becoming inaccessible after a student graduates. Public sharing of databases, particularly high quality direct simulation databases,40–42 which may be mined for decades beyond their inception should be championed by the fluid dynamics community. In developing some initial ideas toward estimating the carbon footprint of computational fluid dynamics, we are still left wondering what is the total footprint. To estimate this number for a given year, we would need a database of every simulation performed of all modalities (DNS, LES, RANS, and other). There seems to be no straightforward way to construct such a database. A work around, however, is to consider the fraction of CFD applications running on the world's largest computers. Top50039 provides an active list of the world's fastest computers. The total annual energy use of the 188 computers on the November 2023 list which provided power use information amounts to 3.41 × 10 9 kWh. These computers are harbored in many countries with different carbon intensities for electricity use. As a very rough estimate of the carbon footprint of this fraction of Top500, we can use the carbon intensity from a few countries to get a sense of the order of magnitude of the footprint. Six countries are chosen, the United States, Japan, China, France, the Netherlands, and Russia, since they have diverse budgets of energy resources and makeup about two-thirds of the Top500 computers. Assuming all Top500 computers were powered by the average carbon intensity of one of these countries yield respective carbon footprint estimates of 1.44 × 10 6 tons (U.S.), 1.59 × 10 6 tons (Japan), 1.83 × 10 6 tons (China), 1.75 × 10 5 tons (France), 1.28 × 10 6 tons (Netherlands), and 1.06 × 10 6 tons (Russia). Cast in terms of the annual Indianapolis carbon footprint (site of the 2022 APS DFD) of 19.4 × 10 6 tons,44 this range corresponds to about 1 % 10 % of the annual Indianapolis footprint. The percentage of CFD applications on these computers is not available, but even if these use cases constitute a small percentage of the total, the order of magnitude of these carbon footprint estimates is concerning.

The main finding of this work is Eq. (2), which describes the relationship of carbon footprint for direct numerical simulation as a function of the simulated Reynolds number. This simple correlation can be used to estimate the carbon footprint during the planning stages of a new simulation without resorting to scaling studies. The main observation of this correlation is that a twofold change in the Reynolds number of a turbulent flow simulation produces about 13 times more carbon. While the main factors contributing to this increase are higher computational core and simulation time requirements, regional differences in carbon intensity are also significant. Running the same simulation on machines whose power source is derived from a high fraction of renewable energy will have a lower footprint than on machines whose power source is derived primarily from fossil fuels.

The second main finding concerns the makeup of a typical direct numerical simulation, which, to the best of the author's knowledge, has never been presented. A large sample database of direct simulations of turbulence from 2022 found an average Reynolds number of 315 whose corresponding footprint is 18.4 kg. The carbon footprint distribution of the database sample is wide, and while there are many relatively low Reynolds number simulations, their net carbon footprint is significant. Most strikingly, the highest Reynolds number simulation accounts for 94% of the carbon footprint of the database, meaning that a single hero calculation in a year can dramatically increase CFD's carbon footprint. As scientists consider what Reynolds number to simulate for their application, they may consider whether a lower Reynolds number simulation would yield the essential physics for their application, at a fraction of the carbon production. Another factor to consider is whether a direct simulation is required, and a reduced order approach such as LES or RANS would be sufficient. Even a large ensemble of these lower fidelity simulations may produce significantly less carbon than a single DNS at the same Reynolds number. While this work has focused on quantifying the carbon footprint of DNS, a modality mainly performed in the academic community, industry also engages in significant computing. Efforts to estimate the industrial carbon footprint owing to computing are needed.

This work relied on estimates of simulation power use through TDP, which, in general, does not equal the power consumed in situ. More work to measure power use during runtime for various flow types and Reynolds numbers will provide more accurate correlations building on these initial estimates.

Future work should also focus on understanding the net science generated from computation compared with experiments. Both modalities require initial capital and carbon investment to construct the respective facilities. Data collection—running a simulation or a wind tunnel—will have different carbon costs in situ. The author's impression is that running a high equivalent Reynolds number turbulent flow DNS calculation is likely more energy (carbon) intensive than running a wind tunnel, for example, since the former must typically run for long wall-clock times compared with the real measurement time of a turbulent boundary layer (e.g.) where ample time averaging can be performed in minutes. The precise comparison is sensitive to the respective details of the experiment and simulation. There is a trade-off in the type of data that is accessible in simulations compared with experiments, one example being the resolution of physics close to boundaries. Collecting data near a wall is easy for a simulation and hard in an experiment. Design optimization is another area where simulations may be advantageous compared with experiments. As scientific representatives, we must optimize the amount of science produced per dollar of investment while simultaneously seeking to maximize the amount of science produced per mass of carbon produced. These goals may be in conflict.

If the exponential growth in Reynolds number accessible with simulation is to continue in concert with growing computational infrastructure, the carbon footprint of scientific simulation will increase in unison with no change in scientific community behavior. By analogy to Richardson's famous quote on turbulent flows: “Big whirls have little whirls that feed on their velocity, and little whirls have lesser whirls and so on to viscosity,”35 the apt comparison here may be: “big computers have big simulations that feed on their electricity, bigger computers have bigger simulations, and so on as R e .” As we 21st-century scientists have inherited the privilege and opportunity to conduct the most detailed numerical studies ever conceived, it is the author's belief that such inquiry should now consider the carbon cost of our work, in addition to all the other careful choices we deliberate on in search for knowledge.

The principal factors contributing to the carbon footprint of this paper were my travel to present preliminary versions of this work at the 2022 and 2023 APS Division of Fluid Dynamics annual meeting. Roundtrip flights from San Francisco to Indianapolis and San Francisco to Washington D.C. generated around two metric tons of carbon equivalent for a single economy ticket.30 

See the supplementary material for a spreadsheet containing the reference and Reynolds number information for database II.

I am grateful for the detailed feedback of M. Lee and K. Ravikumar regarding their simulations. I am also thankful to J. Capecelatro for suggesting the Top500 database as an avenue for exploring CFD's carbon footprint, and to P. Johnson for advocating public access databases. I appreciate valuable conversations with K. Ferguson who provided helpful feedback and thought provoking questions, including on cost associated with preparatory simulations conducted prior to production runs. I am also grateful to his comments on this paper. I am thankful to A. Bertsch, B. Rountree, B. R. de Supinski, B. Springmeyer, T. Patki, and B. Ryujin for insightful conversations, particularly on the subject of HPC power management, limitations of TDP, cost of simulations vs experiments, and quantification of net “science” production. In addition, I received valuable feedback from M. Howland, E. Loth, P. Miller, and T. Bailey. I thank the anonymous reviewers for their constructive comments, particularly for pointing me to the computational resource requirements for wall-bounded flows. This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under contract DE-AC52-07NA27344. Lawrence Livermore National Security, LLC, LLNL-JRNL-858734.

The authors have no conflicts to disclose.

J. A. K. Horwitz: Conceptualization (equal); Data curation (equal); Formal analysis (equal); Investigation (equal); Methodology (equal); Writing – original draft (equal); Writing – review & editing (equal).

The data that support the findings of this study are available within the article and its supplementary material.

1.
R. M.
Kerr
, “
Higher-order derivative correlations and the alignment of small-scale structures in isotropic numerical turbulence
,”
J. Fluid Mech.
153
,
31
58
(
1985
).
2.
R.
Gotoh
,
D.
Fukayama
, and
T.
Nakano
, “
Velocity field statistics in homogeneous steady turbulence obtained using a high-resolution direct numerical simulation
,”
Phys. Fluids
14
,
1065
1081
(
2002
).
3.
K. H.
Bech
,
N.
Tillmark
,
P.
Henrikalfredsson
, and
H. I.
Andersson
, “
An investigation of turbulent plane Couette flow at low Reynolds numbers
,”
J. Fluid Mech.
286
,
291
325
(
1995
).
4.
J. G. M.
Eggels
,
F.
Unger
,
M. H.
Weiss
,
J.
Westerweel
,
R. J.
Adrian
,
R.
Friedrich
, and
F. T. M.
Nieuwstadt
, “
Fully developed turbulent pipe flow: A comparison between direct numerical simulation and experiment
,”
J. Fluid Mech.
268
,
175
209
(
1994
).
5.
F.
Unger
and
R.
Friedrich
, “
Large eddy simulation of fully-developed turbulent pipe flow
,” in
Flow Simulation of High Performance Computers I,
Notes on Numerical Fluid Mechanics Vol.
38
, edited by E. H. Hirschel (
Vieweg
,
1993
), pp.
201
216
.
6.
J.
Kim
,
P.
Moin
, and
R.
Moser
, “
Turbulence statistics in fully developed channel flow at low Reynolds number
,”
J. Fluid Mech.
177
,
133
166
(
1987
).
7.
S.
Hoyas
and
J.
Jimenez
, “
Scaling of the velocity fluctuations in turbulent channels up to R e τ = 2003
,”
Phys. Fluids
18
,
011702
(
2006
).
8.
J. A.
Silero
,
J.
Jimenez
, and
R. D.
Moser
, “
One-point statistics for turbulent wall-bounded flows at Reynolds numbers up to δ + 2000
,”
Phys. Fluids
25
,
105102
(
2013
).
9.
N. M. M.
Lee
and
R. D.
Moser
, “
Petascale direct numerical simulation of turbulent channel flow on up to 786k cores
,” in
Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis
(
IEEE
,
2013
), pp.
1
11
.
10.
M.
Lee
and
R. D.
Moser
, “
Direct numerical simulation of turbulent channel flow up to R e τ 5200
,”
J. Fluid Mech.
774
,
395
415
(
2015
).
11.
G.
Alfonsi
,
S. A.
Ciliberti
,
M.
Mancini
, and
L.
Primavera
, “
Direct numerical simulation of turbulent channel flow on high-performance GPU computing system
,”
Computation
4
(
1
),
13
(
2016
).
12.
H.
Zhang
,
F. X.
Trias
,
A.
Gorobets
,
Y.
Tan
, and
A.
Oliva
, “
Direct numerical simulation of a fully developed turbulent square duct flow up to R e τ = 1200
,”
Int. J. Heat Fluid Flow
54
,
258
267
(
2015
).
13.
S.
Gavrilakis
, “
Numerical simulation of low-Reynolds-number turbulent flow through a straight square duct
,”
J. Fluid Mech.
244
,
101
129
(
1992
).
14.
P. K.
Yeung
and
K.
Ravikumar
, “
Advancing understanding of turbulence through extreme-scale computation: Intermittency and simulations at large problem sizes
,”
Phys. Rev. Fluids
5
,
110517
(
2020
).
15.
K.
Ravikumar
,
D.
Appelhans
, and
P. K.
Yeung
, “
GPU acceleration of extreme scale pseudo-spectral simulations of turbulence using asynchronism
,” in
SC'19: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis
(
IEEE
,
2019
), pp.
1
22
.
16.
A.
Vela-Martin
,
M. P.
Encinar
,
A.
Garcia-Gutierrez
, and
J.
Jimenez
, “
A low-storage method consistent with second-order statistics for time-resolved databases of turbulent channel flow up to R e τ = 5300
,”
J. Comput. Sci.
56
,
101476
(
2021
).
17.
K. A.
Goc
,
O.
Lehmkuhl
,
G. I.
Park
,
S. T.
Bose
, and
P.
Moin
, “
Large eddy simulation of aircraft at affordable cost: A milestone in computational fluid dynamics
,”
Flow
1
,
E14
(
2021
).
18.
M. H.
Akbari
and
S. J.
Price
, “
Simulation of dynamic stall for a NACA 0012 airfoil using a vortex method
,”
J. Fluids Struct.
17
,
855
874
(
2003
).
19.
G.
Bell
, “
The future of high performance computers in science and engineering
,”
Commun. ACM
32
,
1091
1101
(
1989
).
20.
C. Q.
Choi
, “
The beating heart of the world's first exascale supercomputer
,” https://spectrum.ieee.org/frontier-exascale-supercomputer (
2022
).
21.
N. A.
Trayanova
, “
Whole heart modeling: Applications to cardiac electrophysiology and electromechanics
,”
Circ. Res.
108
(
1
),
113
(
2011
).
22.
Y.-T.
Wu
and
F.
Porté-Agel
, “
Large-eddy simulation of wind-turbine wakes: evaluation of turbine parametrisations
,”
Boundary-Layer Meteorol.
138
,
345
366
(
2011
).
23.
R.
Davé
,
D.
Anglés-Alcàzar
,
D.
Narayanan
,
Q.
Li
,
M. H.
Rafieferantsoa
, and
S.
Appleby
, “
SIMBA: Cosmological simulations with black hole growth and feedback
,”
MNRAS
486
,
2827
2849
(
2019
).
24.
M.
Watanabe
,
T.
Suzuki
,
R.
O'ishi
,
Y.
Komuro
,
S.
Watanabe
,
S.
Emori
,
T.
Takemura
,
M.
Chikira
,
T.
Ogura
,
M.
Sekiguchi
,
K.
Takata
,
D.
Yamazaki
,
T.
Yokohata
,
T.
Nozawa
,
H.
Hasumi
,
H.
Tatebe
, and
M.
Kimoto
, “
Improved climate simulation by MIROC5: Mean states, variability, and climate sensitivity
,”
J. Clim.
23
,
6312
6335
(
2010
).
25.
P. N.
Krishnani
, “
CFD study of drag reduction of a generic sport utility vehicle
,”
Master's thesis
(
California State University
,
Sacramento
,
2009
).
26.
D.
Cerinski
,
M.
Vujanovic
,
Z.
Petranovic
,
J.
Baleta
,
N.
Samec
, and
M.
Hribersek
, “
Numerical Analysis of fuel injection configuration on nitrogen oxides formation in a jet engine combustion chamber
,”
Energy Convers. Manage.
220
,
112862
(
2020
).
27.
P.
Ostermeier
,
F.
Fischer
,
S.
Fendt
,
S.
DeYoung
, and
H.
Spliethoff
, “
Coarse-grained CFD-DEM simulation of biomass gasification in a fluidized bed reactor
,”
Fuel
255
,
115790
(
2019
).
28.
T.
Ahmad
and
D.
Zhang
, “
A critical review of comparative global historical energy consumption and future demand: The story told so far
,”
Energy Rep.
6
,
1973
1991
(
2020
).
29.
Green Algorithms: How green are your computations?,”
http://calculator.green-algorithms.org/.
30.
L.
Lannelongue
,
J.
Grealey
, and
M.
Inouye
, “
Green algorithms: Quantifying the carbon footprint of computation
,”
Adv. Sci.
8
,
2100707
(
2021
).
31.
J.
Seinfeld
, “Jerry Seinfeld Performs Standup,” The Tonight Show Starring Jimmy Fallon (
2014
).
32.
G.
Kendall
, “
Would your mobile phone be powerful enough to get you to the moon?
” Published July 1, 2019. Accessed 5 May
2023
.
33.
The Apollo guidance computer: Hardware
,” in Computers in Spaceflight: The NASA Experience. https://history.nasa.gov/computers/Ch2-5.html/
34.
S. B.
Pope
,
Turbulent Flows
(
Cambridge University Press
,
2000
).
35.
L. F.
Richardson
,
Weather Prediction by Numerical Process
(
Cambridge University Press
,
1922
).
36.
B.
Goldstein
,
D.
Gournaridis
, and
J. P.
Newell
, “
The carbon footprint of household energy use in the United States
,”
Proc. Natl. Acad. Sci. U. S. A.
117
(
32
),
19122
19130
(
2020
).
37.
D. A.
Donzis
,
P. K.
Yeung
, and
K. R.
Sreenivasan
, “
Dissipation and entropy in isotropic turbulence: Resolution effects and scaling in direct numerical simulations
,”
Phys. Fluids
20
,
045108
(
2008
).
38.
V.
Eswaran
and
S. B.
Pope
, “
An examination of forcing in direct numerical simulations of turbulence
,”
Comput. Fluids
16
(
3
),
257
278
(
1988
).
39.
E.
Strohmaier
,
J.
Dongarra
,
H.
Simon
, and
M.
Meuer
, “
TOP 500: The list
,” https://www.top500.org/lists/top500/2023/11/,
2023
.
40.
R. D.
Moser
,
J.
Kim
, and
N. N.
Mansour
, “
Direct numerical simulation of turbulent channel flow up to R e τ = 590
,”
Phys. Fluids
11
(
4
),
943
(
1999
).
41.
Johns Hopkins Turbulence Database
, http://turbulence.pha.jhu.edu.
42.
M.
Oberlack
,
S.
Hoyas
,
F.
Alcántara-vila
,
S.
Kraheberger
, and
J.
Laux
,
TUdatalib Repository
(
2021
).
43.
U.S. Department of Transportation Federal Highway Administration
,
Average annual miles per driver by age group
, https://www.fhwa.dot.gov/ohim/onh00/bar8.htm,
2022
.
44.
D.
Moran
,
K.
Kanemoto
,
M.
Jiborn
,
R.
Wood
,
J.
Tobben
, and
K. C.
Seto
, “
Carbon footprints of 13000 cities
,”
Environ. Res. Lett.
13
(
6
),
064041
(
2018
).
45.
P. K.
Yeung
and
S. B.
Pope
, “
Lagrangian statistics from direct numerical simulations of isotropic turbulence
,”
J. Fluid Mech.
207
,
531
586
(
1989
).
46.
E. A.
Leon
,
I.
Karlin
, and
R. E.
Grant
, “
Optimizing explicit hydrodynamics for power, energy, and performance
,” in
2015 IEEE International Conference on Cluster Computing
(
IEEE
,
2015
), pp.
11
21
.
47.
B. S.
Ryujin
,
A.
Vargas
,
I.
Karlin
,
S. A.
Dawson
,
K.
Weiss
,
A.
Bertsch
,
M. S.
Mckinley
,
M. R.
Collette
,
S. D.
Hammond
,
K.
Pedretti
, and
R. N.
Rieben
, “
Understanding power and energy utilization in large scale production physics simulation codes
,” arXiv:2201.01279v1 (
2022
).
48.
H.
Choi
and
P.
Moin
, “
Grid-point requirements for large eddy simulation: Chapman's estimates revisited
,”
Phys. Fluids
24
,
011702
(
2012
).
49.
X. I. A.
Yang
and
K. P.
Griffin
, “
Grid-point and time-step requirements for direct numerical simulation and large-eddy simulation
,”
Phys. Fluids
33
,
015108
(
2021
).

Supplementary Material