METHOD
How this benchmark works
Six steps run identically for every model, so a difference on the board is a difference between models, not between setups.
- Freeze the market.Prices are downloaded once into a content-addressed snapshot; every run re-uses those bytes.
- Build the state.One compact JSON per decision date, holding only data dated on or before it, with the token budget enforced before it is sent.
- Ask typed questions.One probability question per asset, phrased identically for every model. Answers must be typed probabilities; a refusal counts as missing coverage, never invented.
- Size the book.A fixed policy turns probabilities into weights across the whole pool. The index is never bought, and the rule was recorded before any result was seen.
- Fill the orders.Trades fire only when a weight moves past the 2% rebalance band, and fill at the next session's open with fees and slippage. Most names sit inside the band, so a typical weekly decision is mostly holds: on the published board models keep 52% to 91% of the book steady.
- Score it twice, on three windows.Calibration first (Brier, Brier skill, AUC, reliability bins), then the portfolio the answers built, against SPY and fixed baselines. The same steps run on three consecutive six-month windows, so a model must hold up in a rally, a flat stretch, and a run it cannot simply ride; the board shows each window and the average.
What keeps it honest
No look-ahead.The state holds only past data, and an audit tampers with every later price to prove earlier decisions cannot change.
Missing stays missing.An unanswered question is never scored and carries no conviction, so a provider failure cannot become a trade.
Replayable.Each run ships its state hash, decisions, fills and metrics; beatspy verify re-derives the scores offline.
Uncertainty is reported.Returns carry block-bootstrap intervals and every calibration score a permutation p-value: one window cannot prove an edge.