How to Backtest Honestly: Replay, Sample Size, and the Lies You Tell Yourself
Scrolling back through a chart and spotting where your setup 'would have worked' is not backtesting. It's shopping for reassurance. Your eye lands on the winners because winners are visually loud, and the losers scroll past unnoticed. An honest backtest is a different activity entirely: you decide the rules before you see the outcomes, you replay the chart one bar at a time so the future is genuinely hidden, and you record every trade the rules produce — especially the ugly ones. The result is a number that might disappoint you, which is exactly what makes it worth having.
Write the setup down before you test it
The first casualty of a dishonest backtest is definition. 'I buy the retest of the zone in an uptrend' contains four judgment calls wearing one sentence. Before touching replay, translate the setup into rules a stranger could execute: what makes a zone valid, what confirms the trend, where exactly the entry triggers, where the stop goes, where the target sits, and what invalidates the whole idea. If a rule needs the word 'usually', it isn't a rule yet. You will feel the definition being painful to write — that pain is the point, because every ambiguity you leave in is a place where hindsight will later flatter you.
Replay mechanics and sample size
- Use bar replay or step through candle by candle with the future hidden. If you can see what comes next, you are not testing — you are narrating.
- Mark the trade at the moment the rules fire, not after. Entry, stop, and target get written down before you advance a single bar.
- Log planned R and realized R separately. The gap between them — trades that hit stop by a tick then ran, targets missed by a hair — is data about your rules, not bad luck to be edited out.
- Collect 50 to 100+ trades minimum. Twenty trades of a 45% win-rate system can easily show 12 winners or 5; small samples don't reveal the edge, they cosplay one.
- Test across different conditions — trending weeks, dead ranges, news weeks. A setup sampled only from the regime it loves is a setup you don't actually know.
The biases that fake results
Two habits quietly destroy most manual backtests. The first is hindsight cherry-picking: only 'counting' the setups that look clean after the fact, and dismissing losers as 'I wouldn't have taken that one'. If the rules fired, it counts. The second is moving the entry after seeing the outcome — nudging the fill a few ticks better, deciding you 'would have' entered on the earlier candle once you know the trade worked. Each nudge is small; fifty nudges build an equity curve for a trader who does not exist. Protect yourself structurally: write the trade down before advancing the bars, never revise a logged entry, and count skipped-but-valid signals as what they are — evidence.
From backtest to forward rules
A finished backtest should compress into a one-page contract with your future self: the setup definition, the win rate and average R the sample produced, the worst losing streak it contained, and the risk per trade those numbers can survive inside your account's limits. Then trade it forward in small size and compare live results against the backtest's range. Deviation within the sample's variance means keep going; live results falling far outside it means the market changed or your live execution differs from your tested rules — both worth knowing early, neither knowable without the baseline.
Updated 2026-08-22