The Worst Prediction in Physics?

By and large, contemporary physics theories do a pretty good job of describing the world around us. Sure, there is the occasional infinity when someone uses Coulomb’s law to calculate the force between two objects with zero separation, but that is just a case of a model being pushed into a realm in which it no longer applies. However, there are instances in which a prediction of a valid physical quantity is staggeringly wrong. In a case of cosmic irony, there is a mundane example that could well be the worst prediction in physics. This incredible disagreement occurs when the simplest imaginable calculation is performed: the energy of empty space.

On the face of it, predicting the energy of empty space seems like a trivial exercise. Take a fixed volume. Remove all forms of matter and shield it from every possible form of energy or known field—electromagnetism, gravity, etc.—and the volume contains absolutely nothing. It would seem that the natural answer is zero.

However, as is often the case when one takes a serious and careful look at a seemingly simple problem, the situation is more subtle and complex. The circumstances are made even more interesting when one compares predictions and measurements from our best theory of the cosmic scale with an equally respected theory governing the quantum scale. The two theories wildly disagree. The origins of these theories are displayed in Fig. 1.

Our best theory of the behavior of the universe on cosmic scales is Einstein’s general theory of relativity. Published in 1915, general relativity² is a theory of gravity. It explains gravitational forces as deformations of both space and time.

In 1927, Belgian physicist Georges Lemaître published an article³ that used Einstein’s equations to demonstrate that the geometry of the universe should be changing. He refined his thoughts in a subsequent paper in the journal Nature.⁴ His work formed the basis of a theory of the origin of the universe that is now commonly called the Big Bang.⁵ 

In the intervening years, Lemaître’s ideas have been further refined,⁶ but the broad strokes remain unchanged. The universe began in a small and hot dense state and began expanding about 14 billion years ago. According to initial expectation, because of the attractive nature of gravity, the expansion should have slowed after the initial impetus was provided.

Thus, it was a great surprise when, in 1998, astronomers realized that while the universe was expanding, this expansion was not slowing; quite the contrary, it was speeding up.⁷ 

In order for the expansion of the universe to be accelerating, it is necessary that there exists a repulsive form of gravity that overcomes the attractive nature of familiar gravity. The origin of this repulsive gravity is an energy field that has come to be called dark energy.⁸ Current measurements suggest that dark energy is a constant energy density, distributed uniformly and isotropically throughout space, although there have been hints that suggest that perhaps there is a small time-dependent change in the amount of dark energy in the universe.⁹ 

Through fits of observations of the expansion history of the universe, it is possible to calculate the energy density of the universe. It is ρ_vac = 7.7 × 10⁻¹⁰ J/m³. We can convert that to mass density by dividing by the speed of light squared. The result is ρ_vac = 8.5 × 10⁻²⁷ kg/m³.^10,11

This is the energy density of space itself, and it is an energy/mass equivalent of approximately four hydrogen atoms per cubic meter of space. By way of comparison, the mass density of interstellar space varies quite a bit, depending on the interstellar environment, but a very rough estimate of the mass density found in interstellar gas is of order 10⁶ atoms/m³.

While a determination of the energy density of space is attainable using cosmic methods, a quantum approach is also possible. The theoretical framework necessary to do this is called quantum field theory (QFT).¹² 

In QFT, the universe is filled with a number of fields. For example, there is an electron field, a quark field, a photon field, etc. Physical particles are quantized excitations of the various fields; for example, a physical electron is simply a specific localized vibration of the electron field.

In a classical field, in locations where no particles exist, the fields would be quiescent; however, in quantum fields, the situation is different. In QFT, the fields are a tumultuous froth of vibrations. These vibrations are not the same as those required to manifest physical particles; rather, these particles appear and disappear in wanton abandon, governed by the rules of the Heisenberg uncertainty principle. Succinctly, as long as these ephemeral ripples exist for a short enough time, they do not invalidate energy conservation. These nonphysical ripples are commonly called virtual particles.

The inclusion of E_max is crucial. In principle, this could go to infinity, leading to infinite energy density. However, the calculation has a realm of applicability—specifically, the energy range over which quantum field theory applies. While it is not known at which energy QFT becomes invalid, the Planck energy is a natural upper limit. The Planck energy, defined as $Ep=ℏc5/G$⁠, where G is Newton’s gravitational constant, is the maximum energy at which current theory can apply.¹⁴ Above that energy, gravitational effects become sufficiently high that current theory is no longer valid.

This is sometimes called the most spectacularly bad prediction in all of physics. It is sometimes called “the vacuum catastrophe.”

Given that quantum and cosmic determinations of the vacuum energy of the universe disagree so badly, there is clearly a problem somewhere. The cosmic determination (dark energy) is a measurement, which means that it is likely to be correct. In contrast, the quantum determination (QFT) is a prediction, albeit of a well-validated theory. The fact that it is a prediction makes it more likely to be the origin of the discrepancy.

Before launching into the ongoing theoretical musing on the topic, it should be freely admitted that this remains an unsolved problem. The fact that quantum vacuum energy is predicted to be very large has been known for at least half a century. Dark energy was first detected in 1998,¹⁵ and thus the size of the discrepancy has been known for only a quarter century.

So, what are possible answers? The most obvious one arises from the fact that the quantum answer integrates up to the Planck energy, which is 1.2 × 10¹⁹ GeV. This energy is of order 10¹⁵ times the highest energy humanity has produced in the laboratory.¹⁶ As a reminder, this cutoff was selected to be the energy scale at which quantum field theory is known to fail (e.g., it becomes mathematically inconsistent). However, we have no experimental evidence that the currently accepted theory of the quantum world applies to an energy scale a quadrillion times higher than it is tested at. It could well be that at an energy scale not much higher than we can currently achieve, new physical phenomena will be discovered that will change the quantum calculation. This would mean that the cutoff energy would be close to the currently achievable limit (of order 10⁴ GeV).

However, reducing the energy cutoff by a factor of 10¹⁵ doesn’t help all that much. From Eq. (2), we see that that the vacuum density from quantum calculations goes as the fourth power of the cutoff. Using the energy scale available at the most powerful human-built accelerator as a cutoff energy reduces the discrepancy to $ρvacquantum/ρvaccosmic~1060$⁠. This is an improvement, to be sure, but it is still a huge discrepancy. There must be something more going on.

What if the quantum mechanical value is correct? What if the value measured by dark energy observations is some sort of excess, sitting on top of a large and uniform baseline? This would be somewhat analogous to the idea of potential energy in classical physics. In classical physics, the potential energy at any point can be assigned any value. This is because only potential energy differences are physically realizable.

However, this is not the case here. If an energy field of the size predicted by quantum theory is inserted by hand into the equations of general relativity, the result is catastrophic. With that much energy (and depending on the sign), the universe would have collapsed immediately after it began, or it would have blown itself apart so fast, that no cosmic structures (like stars and galaxies) could have formed. This is another reason to believe that the quantum picture is the one that needs attention.

One very common theoretical idea is that there exists an unknown field that results in a large energy density, but with the opposite sign of the value arrived at by QFT calculations.¹⁷ Under this paradigm, the contributions of the two energy fields cancel each other out, leaving only the small residual component determined by dark energy observations.

However, this proposal brings with it its own difficulties. The issue is that the energy contribution of QFT and the unknown field almost balance, but not exactly. This is peculiar. In physics and mathematics, it is common to find symmetries that exactly balance. Positive and negative numbers add to zero in arithmetic, while positive and negative charges lead to exact electric neutrality. But that’s not what appears to be going on here. Here, the cancellation is nearly, but not perfectly, exact.

The issue is that when you have two large and competing effects, adding to zero isn’t so weird (as the examples above illustrate), but adding to near zero is. For example, suppose you were a billionaire. Consider your income and expenses in any one year. Maybe your income was $124,348,905.33. How likely is it that your expenses equaled that number to within a few cents? It’s more likely that you splurged and bought a superyacht, thereby going into debt to the tune of millions of dollars. Or maybe you were frugal that year, in which case your bank accounts might have burgeoned by $20 or $30 million. But to balance within a few cents is incredibly unlikely.

This is basically what would be required to have two energy fields of the magnitude arising in quantum calculations that almost cancel each other out. However, in the physics case, the problem is much worse. In my billionaire example, the income number contains 11 digits; however, in the physics example, the number would contain of order 120 digits.

Physicists call this inexplicable balance “unnatural,” and it points to two phenomena that will be required to do the trick. The first is that the undiscovered energy field has some sort of symmetry that requires it to be basically the same as the QFT contribution (although with opposite sign). The second is that there must be some additional phenomenon that just slightly breaks that symmetry. While there are proposed solutions, none have been validated.¹⁷ 

There is an important consideration that has not been mentioned. In QFT, the energy arises from the behavior of quantum fields that interact in space, while in general relativity, the vacuum energy is thought to be the energy of space itself. This is a subtle but nontrivial point—and one that is not easily resolved. For example, if all space is accompanied by quantum fields, how does one disentangle the contribution of fields from the contribution of space?

This may seem to be a pedantic point, but it has a significant bearing on how the solution to the vacuum catastrophe problem is imagined. A simpler example illustrates the import.

In flat and Euclidean space, two lines, parallel at a point, will never converge. Only curved lines can be parallel for a certain value of x, but eventually cross. In contrast, in a curved space like the surface of a sphere, lines that are initially parallel eventually converge and cross one another. This is illustrated in Fig. 2. In both scenarios, the lines cross, yet the conclusions one draws are vastly different. Thus, the question of whether vacuum energy is of space itself or fields within space is an important and unresolved issue.

A final concern is one that is deeply embedded in the mathematical underpinnings of the theoretical framework guiding both the cosmic and quantum determinations of the vacuum energy of the universe. General relativity is built on differential equations, in which space is taken to be smooth and continuous and there is no smallest distance. In contrast, quantum field theory is a hybrid mix of continuous and discrete mathematics. The field theory is continuous, but the solutions within the theory are quantized.

Thus, when one considers the energy content of space, the nature of space is necessarily a consideration. Given that the two theoretical paradigms are built on inherently different mathematical paradigms, the situation is inevitably murky. It is likely that a proper resolution of the conundrum of the vacuum catastrophe will await the development of a complete theory of quantum gravity.

Physics is no stranger to unsolved problems, but few are as dramatic as the vacuum catastrophe. Although there are no very promising solutions on the horizon, when we solve it, we will have learned a great deal about the nature of space and time.

Don Lincoln is a senior scientist at Fermi National Accelerator Laboratory. He uses data collected using high-energy particle accelerators to study the laws of nature and has co-authored over 1500 papers. He is also an avid popularizer of frontier physics and has written several books for the general public, most recently Einstein’s Unfinished Dream: Practical Progress Towards a Theory of Everything. He also has written for online venues like BigThink, CNN, Forbes, and others. He also makes videos on the Fermilab YouTube channel and with The Great Courses Plus. [email protected]; www.facebook.com/Dr.Don.Lincoln/