Multiple testing
The convention that a result is “significant” at a t-statistic of about 2 assumes you ran one test. Run 20 independent tests on random data and, on average, one clears that bar. Harvey, Liu and Zhu (2016) counted more than 300 published return predictors — and that only covers what made it into journals. Under that much searching, the threshold for genuine significance is much higher.
Their recommendation, after adjusting for the count and for the ones that were never published: a t-stat of about 3.0. The slider below lets you set your own assumed number of tests and see the Bonferroni bar move — and which of our 18 replications stay above it.
Each row is one replicated anomaly, its bar the full-sample |t|. Green clears the bar, red doesn’t. Click a row to open its full record.
Four ways to draw the line, from most permissive to most conservative.
Controls the chance of one false positive on a single test. Fine for a pre-registered hypothesis; far too loose for a data-mined predictor.
Divide your 5% budget across all N tests. Controls the chance of any false positive at all (family-wise error). Simple, and quite conservative when the tests are correlated — as factor returns very much are.
Controls the false-discovery rate — the expected share of your 'discoveries' that are wrong — rather than the chance of any error. More powerful, and the right target when you expect some real effects in the pile.
The version that stays valid when the tests are arbitrarily dependent. Harvey, Liu & Zhu lean on this one. It's what the site uses for its headline 'survives multiple testing' count.
p-values here come from a normal approximation to the t-distribution (our samples are long, so the two are close). The “literature N” of 400 follows Harvey, Liu & Zhu (2016) and the ongoing count in the Open Source Asset Pricing project.