Probability Theory: The Logic of Science , E. T. Jaynes Cambridge U. Press, New York, 2003. $65.00 (727 pp.). ISBN 0-521-59271-2
Forty years ago, I read a review by Edwin Jaynes. In it, he highly recommended a little book by Richard Cox, The Algebra of Probable Inference (Johns Hopkins U. Press, 1961). Cox’s book transformed my view of probability theory and enriched my career as a physicist, and Jaynes’s Probability Theory , which takes Cox’s work as a foundation piece, can do the same for readers today.
Probability theorists can be divided into two schools: the frequency school and the degree-of-belief school. Jaynes belongs to a subclass that I would call the “objective degree-of-belief” school. A probability P (A|B) is taken to denote the rational degree of belief that proposition A is true given the information in statement B. The probability is a numerical measure of plausibility, and the number is arrived at by objective methods. Much as Cox did, Jaynes conceives of “probability theory as extended logic,” a consistent and powerful method of drawing inferences from information that is too meager to permit deductions.
Of course, frequencies play a significant role, but they are not the defining element of Jaynes’s system. Rather, probabilities in his sense enable one to estimate observable frequencies and to calculate other traditional statistical items, such as confidence intervals. Broadly speaking, all topics that arise in orthodox statistics appear in Jaynes’s framework but are seen by him in a different light. Derivations and logical status are also different. Moreover, Jaynes’s derivations are typically far more compelling than those in orthodox statistics, in part because they emerge from the systematic application of a single Bayesian principle.
In case readers of this review are skeptical about whether degree of belief can provide an adequate concept for probability theory in physics, let me note that both Cox and Jaynes were physicists who did both experiments and theory. In addition, Erwin Schrödinger espoused such a view of probability theory—and so did the geophysicist Sir Harold Jeffreys.
Jaynes worked on radar during World War II. He wrote his thesis on ferroelectricity under Eugene Wigner at Princeton University and then spent a decade on the faculty at Stanford University. He spent the long remainder of his life at Washington University in St. Louis, Missouri. His book emerged from 40 years of exploration in probability theory; thus the range of topics is extensive—dauntingly so for a reviewer. Chapters 1 and 2 lay the foundations of probability theory as logic; the 600 remaining pages cover diverse content: Sampling theory, hypothesis testing, parameter estimation, sufficient statistics, probability and frequency in repetitive experiments, the maximum entropy principle, ignorance priors, decision theory, principles and pathology of orthodox statistics, physical measurement, and model comparison are examples of the topics covered.
Clearly, no systematic review of this book is possible in a few columns of print. Instead, let me mention three aspects that I found notable. First, the use of “prior” probability distributions to encapsulate initial information is characteristic of Bayesian probability theory. Too often, such priors emerge as unnormalizable probability distributions. Jaynes confronts extensively the task of determining priors and discusses forthrightly the problem of normalizability.
Second, Jaynes’s digressions and opinions add spice. He had a reputation for strong opinions, and he does not hide his sentiments in this book. I particularly enjoyed his remark about Rudolf Carnap: “But he suffered from one of the standard occupational diseases of philosophers; his exposition wanders off into abstract symbolic logic without ever considering a specific real example.”
The third notable aspect is that the book is a tapestry of two interwoven themes: a critique of orthodox statistics and an exposition of Jaynes’s version of Bayesian probability theory. The combination produces hard reading for the inexperienced. However, for those with some background in any form of probability theory, the mixture is both illuminating and entertaining.
In part, Jaynes proposes to show that Bayesian methods reproduce the best in orthodox statistics and eliminate the dubious or flawed. He approaches that goal by way of examples. Therein lies a great strength: Rather than presenting a mere clash of ideologies, Jaynes offers readers a comparison of competing results. Should one accept all of Jaynes’s claims at face value? No. But in more than 600 pages of careful reading, I quarreled with only a handful of assertions.
When Jaynes died in 1998, he left the incomplete manuscript of his book to be prepared for publication by a former student, G. Larry Bretthorst, who has done a fine job editing it. In his preface, Jaynes expresses hope that the book will serve as either a textbook or a reference work. I am skeptical about both uses. The book lacks the economy of presentation that distinguishes a good textbook. Jaynes’s work is useful as a reference for those who are already converted and experienced, but not for the uninitiated—and harried—researcher. The investment of time to get through preliminaries would be prohibitive.
In the context of Physics Today’s readership, the actual audience is more likely to consist of graduate students with an intellectual curiosity about probability theory or faculty members with the luxury of a sabbatical in which to immerse themselves in a new way of approaching statistical problems. For both groups, the rewards of reading Probability Theory can be immense.
