🐘 Stock Market Elephant/The Graveyard Series
Archived: written before this publication started in November 2026. Some claims and numbers are older than the posts under Writing.
The graveyard series · part 1 of 6

Why most backtests lie (and the three tests that catch them)

Jul 28, 2026 · 6 min read

I spent a week building a trading bot for prop-firm futures. Before writing a single strategy, I built the thing most traders never build: a lie detector for my own backtests.

Here's the problem. Test 100 random strategies on any price history and several will look brilliant — not because they work, but because with enough tries, luck produces beautiful equity curves. Every "look at my backtest" screenshot you've seen on social media is, statistically, more likely to be this than not.

The three gates

1. Walk-forward. Pick parameters using only past data, trade the next quarter blind, repeat. If your strategy only works when it's allowed to see the answers, this kills it.

2. Purged cross-validation. Slice history into folds, tune on some, score on others — with a gap ("embargo") around every boundary so indicators can't leak information across. Reveals whether the edge lives everywhere or in one lucky year.

Walk-forward: train, test, slidePurged k-fold: test block + embargo gap cut out of trainingtestpurgedpurgedtrainThe embargo exists because features look back in time — a test day's ATR was computedfrom training days. Without the gap, the test set has quietly seen the training data.
Walk-forward marches through time once; purged k-fold tests every block — after cutting an embargo gap so lookback features (like a 14-session ATR) can't leak across the boundary.

3. CPCV + the Probability of Backtest Overfitting. The heavy machinery, from López de Prado's Advances in Financial Machine Learning. Run every combination of train/test blocks, and count: how often does the optimizer's in-sample favorite underperform the median strategy out-of-sample? That count is PBO. A real edge scores under 20%. Random noise scores ~50%. I fed it pure random-walk data as a control: PBO 47%, deflated Sharpe 0 — the machine correctly calls noise on noise.

CPCV: 6 groups, every pair takes a turn as the test set (15 splits)split 1split 2split 3split 4split 5split 6split 7split 8split 9split 10split 11split 12split 13split 14split 15G1G2G3G4G5G6Each group is tested 5 times; same-colored cells reassemble into 5 complete out-of-sampleequity paths through all of history — five independent verdicts instead of one.
My actual CPCV layout: 6 blocks of history, 15 train/test splits, reassembled into 5 complete out-of-sample equity paths. One lucky path can't pass a strategy — five have to agree, and PBO scores how often the in-sample winner embarrasses itself out of sample.
My opening-range strategy scored PBO 0% — in fifteen out of fifteen train/test splits, the honestly-chosen config held up out-of-sample. That number, not the equity curve, is why I kept building.

There's a fourth number, the deflated Sharpe ratio, which asks: given how many things you tried, what Sharpe would pure luck have produced — and are you above it? It counts your attempts, which means it punishes you for the ideas you tried and threw away. Almost nobody computes it, because almost nobody wants the answer.

Over the next five posts I'll show you everything I tried against these gates — including eight ideas that looked great and failed, several of which you are probably trading right now.

Educational content only — not financial advice; no performance promised or implied. All results are backtests on real CME MNQ data (2019–2026, ~1,900 sessions, realistic costs) and backtests differ from live trading. Futures involve substantial risk of loss.