All fMRI studies are not created equal
Since my lab was called out specifically in this rather cavalier opinion piece, I feel compelled to respond. Mr. Orlowski is correct that there are profound problems with some specific procedures used in fMRI data analysis. But the specific decoding studies that he cites (the movie reconstruction study from my lab and the dream decoding Miyawaki lab in Japan) do not use those statistical procedures. In fact, the encoding/decoding approach used in my lab is based on a completely different experimental and statistical framework that includes very strict cross-validation and internal replication. In fact the entire data processing pipeline that we use is designed to avoid precisely the problems that Mr. Orlowski is concerned about. (The Miyawaki study on dreams was built on the same framework and so shares some of those advantages.) Because we do not use the statistical models or software that Mr. Orlowski discusses, we do not suffer from the problems that Mr. Orlowski describes. Even a rudimentary glance through any of our publications, our online web site (http://gallantlab.org) or our twitter stream (@gallantlab) would have made this clear.
The specific problem that Mr. Orlowski cites stems from statistical models whose assumptions are often not met by fMRI data sets. This is indeed a serious concern for all of the fMRI studies that used those statistical models. And it illustrates the dangers of using canned data analysis software without running appropriate unit tests to ensure its validity and accuracy. There is no “free lunch” in science. Any corners that are cut to save time, money or labor are likely to have negative consequences for the quality of the research.
Many areas of biomedical science – psychology, experimental biology and medicine – use a point-null hypothesis testing approach developed in the first half of the last century. That experimental and statistical approach was a major advance at the time, but the limitations of the classical approach have become clear over the past decade. In short, the classical approach sets the bar too low for publication, and so pollutes the scientific literature with many small effects that cannot be replicated. Many of us in the biomedical science community are well aware of this problem and we are pushing the field to address it. (For excellent work on this issue in the biomedical community in general, I suggest that the interested reader look up Professor John Ioannidis at Stanford.)
Since several of the comments bring up the 19th century field of Phrenology, I should add some information about the relationship between Phrenology and modern neuroscience. Phrenology was originally inspired by valid scientific and medical observations that at least some behavioral and cognitive functions are localized to specific regions of the brain. However, Phrenology was a variety of cargo cult science that did not follow the accepted procedures of science and which had absolutely no basis in fact. The problem with Phrenology wasn’t that the idea of localization was completely wrong, the problem is that the Phrenologists were not doing valid science. (That said, it would be wrong to accept the most extreme versions of functional localization except in a few very specific cases. It is generally best to think of all cognitive functions as being mediated by broad networks of brain structures.)
I often read the Register and I hope that this specific piece does not reflect the usual editorial standards at this source.
- Jack Gallant, UC Berkeley
lab web site: http://gallantlab.org
specific information on movie reconstruction study: