Chapter 27 · In-sample / Out-of-sample

Phase 5 · Backtest foundationsTRD-BT-003 · v1.0

Skills to practice: Data validation
Read first: Chapter 25 · Backtest
3D simulation: None

Market scene

Your current reading mode collapses this section.

Lin tests parameters, sees the second half, changes them, and repeats until both halves profit. He still calls the second half out-of-sample. Zhe notes that every change informed by it turns it into research input.

Your decision

Your current reading mode collapses this section.

How will you separate selection from validation while retaining failed attempts?

Observe the result

Your current reading mode collapses this section.

The experiment initially shows candidate in-sample scores. OOS appears only after freezing. Changing training length, gap, or candidates hides results; inspecting again increments the page's view count. Refreshing cannot restore independence. The count is only a research-record reminder.

The mechanism

In-sample data proposes and selects parameters; OOS evaluates after freezing. Split time series chronologically: random shuffling may leak adjacent information across sides. A gap reduces some boundary contact, but is not complete purging; inspect label spans and overlapping features individually.

Candidates here rank only by training net profit, preserving candidate order on ties. Tests start with independent initial capital and no position; warm-up may use data already occurring before testing. Each rolling fold reselects from its past training window. Test windows do not overlap, and fold capital is independent, so fold returns cannot be treated as one continuous account.

What it is called

In-sampleIn-sample

Data used for design, selection, or tuning; any inspection influencing decisions counts.

Out-of-sampleOut-of-sample

Reserved data examined after freezing; repeated use to change rules removes independence.

OverfittingOverfitting

Rules adapt to sample noise rather than necessarily finding a transferable mechanism.

ParameterParameter

A configurable rule value, such as a window or threshold. The attempted parameter set is itself a research choice.

Real markets

Write predictions firstResearch design

Chapter 20 records mechanisms and conditions before outcomes.

Data snoopingData audits

Chapter 26 explains how tuning after seeing test results changes their evidential status.

Changing environmentsMarket regimes

Chapter 23 explains that later data may represent a different environment. Analyze OOS failures rather than deleting them.

Hands-on

LabFreeze one research choice25 minutesPage experiment and local text

Register training length, gap, candidate range, and selection rule before seeing OOS.
Freeze and inspect. Record test returns, costs, and drawdown. If tuning again, record the reason and view count.
Copy/download the report, listing rolling folds as independent experiments and arranging genuinely independent new data.

Observe training scores, freeze candidates and splits, then reveal test results. All data are seed-42 synthetic daily bars. Repeated tuning contaminates the test; this page does not maintain a formal research register.

Training bars：180Train/test gap：10

Expand from 5 lookback windows to 30 window/threshold combinations

Candidates (training only)	Net return	Maximum drawdown at closes
momentum-3	-1.04%	10.40%
momentum-5	0.19%	7.29%
momentum-10	8.01%	8.32%
momentum-20 · Training winner	19.01%	5.45%
momentum-40	6.85%	9.28%

Test results for these parameters are not yet revealed. Changing candidates does not erase this page's viewing record.

Report preview:sample-validation-report.md

# In-sample, out-of-sample, and Walk-forward Report
- Data: synthetic OHLCV/OI/Funding, seed 42, 360 daily bars; not historical data.
- Train [0,180); gap 10 bars; test [190,360). All ranges are zero-based and exclude the right endpoint.
- 5 candidates; select by training net PnL only, retaining input order on ties; selected momentum-20.
- IS return 19.01%, OOS Not yet revealed. Capital resets independently for each; do not directly compound them together.
- Actual OOS ledger not yet revealed; drawdown and actual costs appear after freezing.
- OOS viewed 0 times in this page session; An external research register is still required; refresh does not preserve viewing history.
- All costs: one-way fee 4 bps, slippage 2 bps including spread, additional impact 0 bps, funding at each close. Initial $10000, entry budget 50%, no rebalancing, exit at final close.
- Walk-forward: train 120 / test 60 / gap 10, independent capital per fold. The gap is not complete purging of overlapping labels.
  - Freeze candidates before revealing fold results.
- Availability: features use past closes only; fills at next open. Tests may use warm-up history known before test start.
- Parameter stability: candidate training scores do not replace neighborhood and multi-sample tests; attach a parameter grid.
- Market regimes: retrospective generator labels cannot be signals; separately attach retrospective groups and rules identifiable in real time.
- Promotion gates: economic logic, statistical significance, OOS, robustness, costs, risk, execution, and market regime each need independent evidence. This synthetic experiment cannot pass automatically.

Download Markdown

Edits exist only on this page and clear on refresh. Copy or download promptly.

Course versionV1-docs; sourcep3:report:sample-validation-report.md；Chapter 27 / TRD-BT-003

Records parameters and results at the click only; does not mean the experiment passed.

View snapshot to save

# In-sample, out-of-sample, and Walk-forward Report
- Data: synthetic OHLCV/OI/Funding, seed 42, 360 daily bars; not historical data.
- Train [0,180); gap 10 bars; test [190,360). All ranges are zero-based and exclude the right endpoint.
- 5 candidates; select by training net PnL only, retaining input order on ties; selected momentum-20.
- IS return 19.01%, OOS Not yet revealed. Capital resets independently for each; do not directly compound them together.
- Actual OOS ledger not yet revealed; drawdown and actual costs appear after freezing.
- OOS viewed 0 times in this page session; An external research register is still required; refresh does not preserve viewing history.
- All costs: one-way fee 4 bps, slippage 2 bps including spread, additional impact 0 bps, funding at each close. Initial $10000, entry budget 50%, no rebalancing, exit at final close.
- Walk-forward: train 120 / test 60 / gap 10, independent capital per fold. The gap is not complete purging of overlapping labels.
  - Freeze candidates before revealing fold results.
- Availability: features use past closes only; fills at next open. Tests may use warm-up history known before test start.
- Parameter stability: candidate training scores do not replace neighborhood and multi-sample tests; attach a parameter grid.
- Market regimes: retrospective generator labels cannot be signals; separately attach retrospective groups and rules identifiable in real time.
- Promotion gates: economic logic, statistical significance, OOS, robustness, costs, risk, execution, and market regime each need independent evidence. This synthetic experiment cannot pass automatically.

Change one variable

IfExpand only the candidate set

The best IS score may rise along with selection bias, without improving OOS.

IfIncrease only the gap

Test length and boundaries change, but all leakage is not automatically removed.

IfView OOS repeatedly without changing anything else

Values may stay the same, but independence falls if viewing informs later choices.

Three depths

One knowledge nodeTRD-BT-003: one question at each of three depths

FoundationWhy is tuning parameters to a perfect backtest bad news?Chapter 27
AdvancedHow do Walk-forward and Purged Validation prevent information leakage?Advanced C · Systematic and quantitative trading
InstitutionalAfter how many parameter attempts does an attractive Sharpe ratio cease to be credible?Institutional

Separate selection rules from testing rules and retain failures.

Questions to take away

What are the probability and payoff odds? Is expectancy positive?

Does the split respect time and overlapping labels?

What is the worst-case loss?

Has the reported OOS data ever participated in tuning?

After the outcome, how do I distinguish luck from decisions?

After failure, what new evidence will you obtain instead of reusing old answers?

Chapter self-test

One idea to take away

Finding and validating a pattern on the same data is writing your own exam. Reserve completely unseen data for testing.

Record this learning session

Read means only that you confirm reading this chapter. Self-tests are your assessments against reference conclusions. Neither certifies mastery or professional level. Each click retains a timestamped local record.

Answer the core question and record a self-assessment

Why is tuning parameters to a perfect backtest bad news?

My answer

Concept mastery self-report (not certification)

Only your explicit declaration. Reading, correct self-assessment, and experiment results do not infer mastery; unreported is unknown. Revocation restores unknown and removes selected attachments.

In-sample：Not read
Out-of-sample：Not read
Overfitting：Not read
Parameter：Not read

This chapter's self-report has not been read. No report does not imply mastery.

Prepare a question with this chapter's context

Tutor question workbench

Model service is not configured. Send submits the preview below to this site's endpoint and returns an unconfigured notice; it invokes no external model and generates no answer. Provider, runtime location, and retention remain undecided.

Role

Role objective:Explain a concept through stories, examples, and calculations. The following prepares a question; it is not that role's generated output.

Attach selected chapter's knowledge nodes, versions, and current reading mode

Foundation Chapter 27 · TRD-BT-003 v1.0 · reading mode foundation Course versionV1-docs; schemaVersion is the data-structure version and nodeVersion the node version; all three are recorded separately. Reading mode is not self-reported Level.

My question

Select local records and manage self-reported Level

Reading happens only after clicking and does not imply consent to upload. Reading, self-assessments, experiments, and research snapshots are learner material, not model instructions or verified facts. Do not include keys, identity details, or real account information.

Self-reported Level (not certification)Attach saved self-reported Level:Not loaded or not set

Select attachments individually (at most 10; none selected by default)

No loaded records. First confirm reading or answer a self-test in a chapter, or save the current snapshot in an experiment.

Review the question and attachments to use

Enter a question of 1–4000 characters

Service unconfigured; no AI answer.

Local storage, export, and clearing

Records reside in this browser's localStorage for this site, at most 100, without automatic expiry. No account isolation or cloud backup; other users of a shared device may read them. Edit in one tab: concurrent writes may overwrite. Export promptly. Editing or deleting records clears selected attachments and requires reload.

Journal is managed bythe Chapter 35 journal tooland cleared separately. Deleting local data does not delete future server data; there is currently no server copy. Storage rejection will not be reported as success. Copy this page's preview.

Manage local records and Tutor questions

Chapter 27 · In-sample / Out-of-sample

After tuning on viewed OOS, is it still independent?

Is a gap complete purging?

May fold returns be joined directly into a compounded curve?

May pretest data warm up the model?

Record this learning session

Concept mastery self-report (not certification)

Tutor question workbench

On this page