The setup
A popular open trading-bots repo has a folder called backtests/hip3/ —
fifteen mean-reversion strategies tested on SP500 perpetuals, with two
documents the rest of the catalog conspicuously lacks: a RESULTS.md with
explicit metrics, and a ROBUSTNESS.md with a six-test battery and a
weighted composite score. This is real engineering, the kind of thing
that's rare in retail strategy collections.
Their winner — by composite robustness score 97.09, with the next best at 61.87 — was Agent 11:
Z-score(15) of price, fade extremes in the 10:00-11:00 ET window, wide ATR stop (k_sl=2.0), tight ATR target (k_tp=0.3), 40-bar time cap.
Their headline numbers on 31 days of SP500 perp tick data: 75.6% win rate, 168 trades, OOS WR 80%. Out-of-sample improved over in-sample. Bootstrap 95% CI on win rate was [69, 82]. Parameter sensitivity worst case: 70.3% — i.e. even after perturbing the parameters ±25%, win rate stayed above 70%.
By every measure their battery scored, A11 was the strategy.
What we found
We ported A11's logic, fetched 5 years of SPY 1-minute bars from Alpaca (494,686 bars across 1,824 trading days), and ran it 70/30 in-sample / out-of-sample. Fees swept at 0, 1, and 3 bps per side. Same logic, same parameters, just more data and an honest fee model.
| Fee | Trades | Win rate | Profit factor | Total return |
|---|---|---|---|---|
| 0 bps | 7,839 IS / 3,420 OOS | 82.0% / 82.4% | 0.72 / 0.72 | −71% IS / −24% OOS |
| 1 bps | same | 30.0% / 27.9% | 0.12 / 0.11 | −207% / −90% |
| 3 bps | same | 1.1% / 1.1% | 0.00 / 0.01 | −746% / −226% |
The win rate reproduces. We got 82.4% out-of-sample — even better than the catalog's claimed 80%. The signal is real.
Read the table again. Even at zero fees, the strategy lost 71% over five years. Win rate, the only thing their battery measured robustly, was not the thing that determined whether the strategy made money.
The math their battery couldn't see
A11's stop is k_sl = 2.0 times ATR. Its target is k_tp = 0.3 times ATR.
The stop is 6.7 times wider than the target. This is the strategy's
declared geometry.
Per-trade expectancy at 82% win rate, expressed in ATR units, is:
0.82 × (+0.3 ATR) + 0.18 × (−2.0 ATR)
= +0.246 ATR + −0.360 ATR
= −0.114 ATR per trade
The strategy has structural negative expectancy. It loses money on average, trade by trade, before any fee ever lands. To break even with this geometry you'd need win rate above 86.96%. A11 never gets there in their data, in ours, or in any of the sensitivity perturbations.
This is not a subtle bug. The math is one division. They didn't write it down because their composite formula didn't ask for it.
Why the battery missed it
The hip3 composite robustness score weights, in order:
- WR stability across train/test splits
- Bootstrap 95% confidence interval on WR
- Parameter sensitivity (how much WR moves under ±25% perturbation)
- Calendar-month-vs-month WR consistency
- Per-day P&L concentration
- WR survival at 1 / 3 / 5 bps commission
Every test is in service of trusting the win rate. Not one asks: given this win rate, what is the per-trade expected value in dollars?
There is nothing wrong with what they built. It's a careful battery. It just happens to have a blind spot exactly where high-win-rate / asymmetric-stop strategies live. And that is a large region of design space — one that produces strategies that look amazing in small samples, because in 168 trades you usually don't get unlucky enough on the 18% of losers to expose the geometry.
We tested with 11,259 trades over 5 years. The math caught up.
The deeper trap: time-stop noise
A11's time cap closes any open trade after 40 minutes if neither stop nor target has hit. In a 168-trade sample, those time-cap exits land at whatever price happens to be there — and over a short window, that distribution is regime-dependent.
Their IS profit factor was 1.20. Just above break-even. The expectancy math above says it should be below 1. The difference was time-cap exits landing slightly above entry, more often than below, in the specific 31-day window they sampled.
That is luck, not edge. It does not generalize.
In our 5-year sample, time-cap exits average out, and the geometry asserts itself. PF lands at 0.72 — within a hair of what the −0.114 ATR/trade math predicts.
What we shipped in response
This is now a ticket on our validation pipeline:
Per-trade expectancy floor. Any strategy where average net return per trade is ≤ 0 is rejected, regardless of win rate. Message: "structural negative expectancy — high WR but asymmetric stops produce per-trade loss."
R:R-aware trade-count floor. Strategies with TP/SL ratio worse than 1:1 must clear a higher minimum sample size (500+ trades for 1m, versus the default 100). The small-sample illusion is meaningfully worse for this geometry.
WR-flash warning. Any strategy with OOS win rate ≥ 70% that also fails on per-trade expectancy or PF gets a one-line educational warning: "High WR with negative expectancy = small-sample illusion or structural R:R problem."
If we had run these checks on A11 first, we'd have rejected it in seconds. If the catalog's battery had asked the expectancy question, A11 wouldn't have made it to "robustness winner."
What we are not saying
We are not saying the catalog's work is bad. The hip3 folder is more honest than 95% of what's in the same repo. They published their methodology, they ran an OOS test, they tested commission sensitivity. The bug is one specific sin of omission — a missing equation in a sophisticated framework.
We're also not saying the underlying signal is fake. A11's 82% win rate is real. We reproduced it on a sample 67× larger than theirs. The z-score fade pattern on 1-minute SP500 / SPY at 10:00 ET has predictive value. What it does not have is a stop-and-target geometry that monetizes that predictive value at the win rate it produces.
There is a strategy here. It is not the one they wrote.
The takeaway
When you see a strategy with:
- High win rate (≥ 70%)
- Asymmetric R:R worse than 1:1 (wide stop, tight target)
- Small sample size (< 500 trades)
- Robustness measured exclusively via win-rate stability
Compute the per-trade expected value, in ATR units, by hand. It takes one multiplication and one subtraction. If it's negative, no amount of WR stability will save it.
The thing that looks like rigor isn't always rigor.
Methodology note: we reproduced the strategy from its published source, fetched five years of 1-minute data independently, and ran the same logic with an explicit fee model. The full validation log — data spans, trade counts, per-fee-tier tables — lives in our internal research records; the numbers above are quoted from it verbatim.