Best strategy in the lab
test score (2 counts as strong)
strategies passed
0had a real edge
Ghost winners
Half my winners were ghosts.
Test 1,152 trading strategies that cannot work. Every one was built with no edge at all. Count how many pass the usual test anyway.
Each square is a strategy with five years of daily results. Gold means it passed the test: it beat zero with 95 percent confidence.
Count the ghosts in your own winners
Say your lab tried a lot of ideas and some passed. Even if none of them worked, about 1 in 20 would pass by luck. Those are the ghosts.
Why this matters
The 5 percent test was made for one idea tested once. A lab, a fund or an AI that tries a thousand ideas will find about fifty winners in pure noise. If the report only shows the winners, the ghosts look exactly like the real thing. You cannot tell which squares are which by looking at their results.
This is not a cartoon
In 2017, Hou, Xue and Zhang re-tested 447 published stock-market patterns. 286 of them failed even the ordinary 5 percent test. With a stricter bar that allows for how many people had been searching, 380 failed, or 85 percent. Harvey, Liu and Zhu looked at 316 published factors and argued that a new one needs a test score above 3, not the usual 2.
Before you trust a winner, ask: how many did you try?
Sources
- Hou, Xue and Zhang, "Replicating Anomalies", NBER working paper 23394 (2017). 447 anomalies; 286 insignificant at 5 percent; 380 (85 percent) with a t-cutoff of 3.
- Harvey, Liu and Zhu, "…and the Cross-Section of Expected Returns", Review of Financial Studies (2016). 316 factors; new factors need t above 3.0.
- Bailey, Borwein, López de Prado and Zhu, "The Probability of Backtest Overfitting", Journal of Computational Finance (2017).
- The strategies on this page are random daily returns with an average of exactly zero, made fresh in your browser each time.