Multiple testing
What is the “factor zoo”?
It’s the name — coined by John Cochrane in 2011 — for the sprawling collection of published “factors” that supposedly predict which stocks beat the market. Academics and quant firms have documented well over 400: value, size, momentum, quality, dozens of accounting ratios. The zoo is a problem because most of the animals in it aren’t real.
Why the number matters
A result is conventionally “significant” at a t-statistic of about 2 — meaning there’s a 1-in-20 chance of seeing it if there were no real effect. That’s fine for one test. Run 20 tests on random data and, on average, one clears the bar anyway. The field has run vastly more than 20.
So when someone publishes a predictor with a t-stat of 2.1, the right question isn’t “is that significant?” but “out of how many things that were tried?” Under that much searching, a t-stat of 2 is close to meaningless.
The higher bar
Harvey, Liu & Zhu (2016) counted the published predictors, estimated how many were tested but never made it into a journal, and recomputed the threshold for genuine significance: roughly a t-statistic of 3.0. When you apply that — or a formal multiple-testing correction like Benjamini–Hochberg–Yekutieli — most of the factor zoo doesn’t survive.
Move the bar yourself
The multiple-testing explorer lets you set your own assumption for how many signals have been tested, watch the significance threshold move, and see which of 18 replicated factors stay above it. There’s also a box to check a t-stat of your own.
Open the explorer →Common questions
Where does the term come from?
John Cochrane's 2011 presidential address to the American Finance Association, where he described a 'zoo of new factors' being published and asked which ones actually matter independently.
Why is testing many factors a problem?
At the usual significance threshold (a t-statistic of about 2), one test in twenty clears the bar by chance alone. The field has run far more than twenty tests. So a t-stat of 2 on a data-mined predictor is weak evidence — you'd expect dozens of false positives at that bar.
What t-stat should a factor need?
Harvey, Liu & Zhu (2016) counted the published predictors, adjusted for the ones that were tested but never published, and put the honest threshold at roughly 3.0. Most of the zoo doesn't clear it.
Educational replication, not advice.