A formulation in “Sterile neutrinos give IceCube and other experiments the cold shoulder”(Physics Today, October 2016, page 15) needs clarification. In discussing the results from IceCube and other experiments, Sung Chang writes that certain parameter ranges are “excluded” by the data. Although that word often appears in scientific publications reporting such results, it is misleading, even when accompanied by the term “confidence level,” and may lead readers to incorrect conclusions.
For a given model, a frequentist analysis—the type of analysis that leads to confidence intervals—yields a range of parameter values for which the observed data satisfy predefined conditions: For a limit-setting analysis, the data are in the tail of the expected distribution. According to the frequentist construction, primarily developed by Jerzy Neyman, the experimenter is to repeat the experiment many times and glean the true parameter values from the distribution of produced intervals. No statement can be made from a single experiment about how likely the parameter values are, and to say, or even to suggest, that certain parameter values are “excluded” is logically incorrect unless there is a zero probability of producing the observed result with a particular parameter value. The correct statement is “For the parameter range [here specify the parameter range for which the data satisfy the probability requirement], the observed data satisfy [here define the criteria the data are supposed to fulfill].”
That phrasing will seem clumsy, but it is important that results be stated correctly. Implying that the frequentist analysis somehow allows a conclusion about which parameter values are probable or improbable is a logical error—one that unfortunately is pervasive, even in physics. A related error led the American Statistical Association to issue a warning last year about the use and interpretation of p-values in statistical hypothesis testing (see https://www.amstat.org/asa/files/pdfs/P-ValueStatement.pdf). I think physicists should get it right!
