Final part. Two ideas that changed how we think about every backtest we'll ever run.
Take your strategy's two main parameters, sweep a grid around your chosen settings, and heatmap the results. A robust strategy shows a broad bright plateau — many nearby settings work. An overfit one shows a lone bright spike: one magic cell surrounded by failure, meaning you didn't find an edge, you found a coincidence.
Ours plateaus wide: opening ranges from 20 to 60 minutes all work, filter settings from 0.10 to 0.25 all work, and the cliff (sub-10-minute ranges) has an economic explanation — the market hasn't picked a direction yet. Meanwhile the seductive variants we'd rejected showed exactly the spike pattern: brilliant in one column, collapsing one parameter-step away. The heatmap made a week of statistics visible in one picture.
Late in the project we found something exciting: re-optimizing the strategy quarterly — letting it switch configurations as regimes change — scored dramatically better out-of-sample. Every quarterly pick used only past data. Textbook clean, right?
No. The scheme never saw the future — but we had. We chose which options to put in its menu after watching seven years of history, including the years it was then "tested" on. We put direction-switching on the menu because we'd seen it work in 2022. The scheme took honest exams that we wrote after reading the answer key.
The tell: rerun the adaptive scheme restricted to the menu we'd fixed before that research burst — the advantage nearly vanishes. All the excess came from hindsight-chosen options.
Which leads to the ending every honest quant project reaches: after enough research, your historical data is spent. You've seen it; it can't surprise you anymore. The only uncontaminated data left is the future. So that's where we're going — a pre-registered live paper campaign, decision rules written down before the first trade, no mid-stream edits. The backtest lab did its job: it killed eight bad ideas and certified two good ones. The market gets the final vote. It always did.