Phase 5 introduction
Backtesting foundations. Test strategies with historical data while understanding how a backtest can deceive you.
What this phase establishes
Backtesting is the researcher's best tool—and a tool exceptionally good at lying. Learn to use it, then question it: data biases, in-sample and out-of-sample testing, costs, robustness, and finally paper trading to examine what backtests cannot.
The core attitude: the purpose of backtesting is to disprove a strategy, not prove it. Only a strategy surviving your strictest designed tests deserves another step.
Six chapters
Can historical simulation prove future profitability?
Why do perfect backtest curves often disappear in live trading?
Why is tuning parameters to a perfect backtest bad news?
What remains after costs from a strategy backtesting at 50% annualized?
What does losing profitability after a small parameter change imply?
Why can you not invest real money immediately after a passing backtest?
Phase project: your first complete Backtest Report
Backtest one Phase 4 hypothesis. Include at least:
- Data sources, dates, and point-in-time alignment
- In-sample/out-of-sample split and the reason for it
- Results before and after fees, slippage, net funding payments/receipts, and impact
- Whether parameters work across an area rather than one point
- Performance in different regimes
- Which promotion gates passed and which did not
Report templates and execution order
- Run the backtest workbench and download
backtest-report.md, recording synthetic data, availability timing, configuration, and all costs. - In the IS/OOS experiment, preregister your choice, freeze it, and download
sample-validation-report.md. Data already seen cannot be called unseen again. - Attach parameter grids and fee stress, stating that the selected seed is a teaching contrast. Label remaining validation gaps unknown.
- Download the order log from the execution-deviation replay. Offline replay is not a live paper-trading deployment.
- Update the Strategy Object with conclusions and evidence for each promotion gate.
| Report section | Minimum required content |
|---|---|
| Data | Source, version, event/availability time, asset universe, revisions/exits/missing values; synthetic or historical identified |
| Rules | Mechanism, signal, warm-up, entries/exits, quantity, terminal handling, and frozen version |
| IS / OOS | Time boundaries, gaps, selection criteria, all attempts, repeated-view records; independent capital per fold must not be presented as compounding |
| All costs | Fees, slippage and whether it includes spread, net funding, impact; gross/net reconciliation with the ledger |
| Stability | Parameter neighborhood, cost stress, failure regions, untested items |
| Regime | Grouping method, sample counts, hindsight labels separated from real-time identification rules |
| Conclusion | Evidence and gaps for eight gates: economic logic, statistics, OOS, robustness, cost, risk, execution, regime |
A complete report may conclude “Research did not pass.” Calling unknowns passed, hiding failed attempts, or presenting synthetic results as historical performance requires rework.
Phase self-test
After report acceptance, enter Phase 6 · Professional trading, placing research within risk, execution, and review loops.
Knowledge nodes
| concept_id | Nodes | Foundation | Advanced | 3D |
|---|---|---|---|---|
| TRD-BT-001 | Backtest Backtest | Chapter 25 | C | Strategy laboratory |
| TRD-BT-002 | Data bias Data Bias | Chapter 26 | C | — |
| TRD-BT-003 | In-sample and out-of-sample In-sample / Out-of-sample | Chapter 27 | C | — |
| TRD-BT-004 | Trading costs Transaction Cost | Chapter 28 | E | — |
| TRD-BT-005 | Robustness Robustness | Chapter 29 | C | — |
| TRD-BT-006 | Paper trading Paper Trading | Chapter 30 | F | — |