Live demo · Data science

What is a market direction model actually worth?

Plenty of projects report an impressive accuracy on next-week price direction. Almost none of them survive being evaluated the way production would evaluate them. This runs a gradient-boosted classifier on daily bars — walk-forward, against baselines, against a permutation null — and reports whatever comes out, including when the answer is "nothing".

Not investment advice, and not a trading signal. This is a demonstration of evaluation methodology. It deliberately produces no buy or sell recommendation, and the honest finding on real tickers is usually that no exploitable edge is detectable at this horizon from this feature set. Prices are delayed daily bars, not real-time quotes.

Ready

Pick a series and run the evaluation.

Walk-forward AUC

—

out-of-sample, all folds pooled

Accuracy vs baseline

—

against always-predict-majority

Permutation null

—

AUC range on shuffled labels

Score by fold

Each point retrains on everything before it and scores the block that follows. The shaded band is where shuffled labels land — a point inside it is indistinguishable from noise.

Out-of-sample AUC for each walk-forward fold against the permutation null band

Corrected target

—

every row labelled, as at inference

Original dead-zone target

—

rows filtered using the future return

The original pipeline dropped every row whose future move was under ±0.5%. Run an evaluation to see what that does to the score.

What the model leans on

Permutation importance measured on each held-out block and averaged across folds: how much out-of-sample AUC is lost when that column is scrambled. Values at or below zero mean the feature carried nothing the model could use.

Average feature importance across walk-forward folds

Fold-by-fold read-out

Method & the bug this fixes

This is a trimmed, corrected descendant of my FinAI platform — the gradient-boosted classifier and technical features, without the FinBERT sentiment model, the RAG chatbot or the LLM layer, so it fits in a container that cold-starts in seconds.

The bug. The original target construction did this:

future_ret = log(Close.shift(-5) / Close) mask = future_ret.abs() >= 0.005 # "reduce label noise" df = df[mask] # ← drops rows using the answer target = (future_ret > 0)

Filtering rows by the future return removes them from the test set as well as the training set, and it cannot be reproduced at inference: live, you do not know the future return, so you cannot decide which rows to skip. Whatever score comes out is measured on a distribution production never sees.

The obvious claim — "so it inflates the score" — turns out to be wrong, which is why the comparison above is measured rather than asserted. On a series with weak signal it moved AUC by about +0.008; on a strongly autocorrelated one it moved it by −0.049. Unpredictable in direction as well as magnitude is the real problem: the number cannot be reasoned about at all.

The evaluation. Expanding-window walk-forward: train on everything up to a cut, predict the block after it, step forward, never look back. The bars straddling each cut are dropped from training, because their labels depend on prices inside the test block.

The controls. Two synthetic series make the real-data result interpretable:

Pure random walk — the negative control. No structure exists, so anything the model appears to find is measurement error.
Momentum injected — the positive control. Returns genuinely depend on the last five days, so a working pipeline must find it. Without this, a null result on a real ticker is ambiguous: efficient market, or broken code?

The null. Comparing AUC against 0.5 understates how far a walk-forward score wanders on its own. The band is measured directly: shuffle the labels, rerun the whole evaluation five times, take the range. A score inside that band is nothing.

Limits worth stating:

Daily bars, delayed. Nothing here is real-time and nothing is tick data.
No transaction costs, slippage, borrow or market impact. A directional edge smaller than the spread is not an edge.
Survivorship — these are large caps that exist today. The tickers that delisted are not in the sample.
Every run of the selector on the same data is another look. Trying tickers until one clears the null is exactly the multiple-comparisons trap the page is about.

How this is built

FastAPI serving a static front end and one JSON endpoint; scikit-learn's histogram gradient boosting rather than the repo's XGBoost, which keeps the image small enough to cold-start quickly on a scale-to-zero service without changing the algorithm family. Market data comes from yfinance on demand with an in-process cache. The synthetic controls need no network, so the demo still has something to say when the data provider rate-limits — which, on a shared egress IP, it will.

Charts are hand-rolled SVG, no chart library. Single-series throughout, so colour carries no identity load; the null band is a shaded region rather than a second series, and every plotted value is in the fold table.

Built by Man Patel — more work at manpatel.me/work.