An AI That Audits Trading Alpha
Take a statistic that cannot exist and give it a p-value that means something else. That is what one paper in the queue did: it reported Spearman rho = 0.94, p = 0.017 over five assets. On five untied ranks, rho lives on a finite grid spaced exactly 0.1 apart. The smallest two-sided p the test can produce is 0.0167, and 0.017 is the exact p-value of a perfect ranking. The nearest attainable rho, 0.90, carries p = 0.0833 — not significant at 5%. The claim is not subtly wrong; it is printed arithmetic that could not have come from the test the paper claims to have run.
The system that caught it is not another return-predicting model. It is an auditor: a loop over a local corpus of 21,305 quant-finance paper abstracts, with 21,119 still queued, 29 papers read end to end by a human, and 82 machine screens completed. Each tick claims one paper, asks a language model two questions about it, runs deterministic nulls against real market data, and records a verdict under a schema that refuses records which certify themselves. The most important thing I can tell you about this loop is not that it found fake alpha. It is that its ceiling is the corpus, not the model — and that honesty about that ceiling is the actual product.
The Correct Baseline Is a Zero-Parameter Rule
Most alpha claims are measured against an optimised baseline. The loop first measured what a zero-parameter baseline does. On 30 Dow Jones Industrial Average components, 499 daily bars from 2024-08-27 to 2026-08-24, a 250-day estimation window, rebalancing every 21 days, with no peeking, it scored several strategies on identical rows:
| strategy | annualised Sharpe | 1/N beats it by |
|---|---|---|
| equal weight (1/N) | +1.649 | — |
| risk parity (inverse vol) | +1.714 | −0.065 |
| buy-and-hold | +1.630 | +0.019 |
| global minimum variance | +0.868 | +0.781 |
| tangency / mean-variance | +0.232 | +1.417 |
There were 249 out-of-sample days. The canonical result is DeMiguel, Garlappi and Uppal (2009), “Optimal Versus Naive Diversification,” which showed that 1/N beats optimised portfolios out of sample. This loop’s independent table reproduces that hierarchy, but the useful output is not the hierarchy. It is the size of the gap: 1/N gives up only 0.065 Sharpe to risk parity, and it beats global minimum variance by 0.781 and the tangency portfolio by 1.417.
Now consider what that does to the literature. A paper whose headline is “our learned allocator beat the tangency portfolio by 0.601 Sharpe” has beaten a comparator that a zero-parameter rule beats by 1.417 on the same rows. The delta measured the opponent, not the model. This is precisely the concern Harvey, Liu and Zhu (2016) pressed in “…and the Cross-Section of Expected Returns”: once the number of tested factors is accounted for, the conventional t > 2.0 hurdle is far too low. Bailey and López de Prado (2014) made the same point for the Sharpe ratio specifically: selection bias and backtest overfitting deflate the ratio you think you earned. Harvey and Liu (2015) formalised backtesting as a family of choices rather than a single test; the loop freezes those choices — 250-day estimation, 21-day rebalancing, 249 out-of-sample days — before any paper is scored.
The loop also prices every Sharpe difference with the Jobson–Korkie correction from Jobson and Korkie (1981), as refined by Memmel (2003), Finance Letters 1(1). And it applies a materiality floor of 0.25 Sharpe. The floor is not sacred; it is a convention drawn from the loop’s own precedent, because an earlier tick measured a momentum edge of +0.109 Sharpe and explicitly refused to promote it. A zero threshold would have closed a paper on noise. Buy-and-hold, at +0.019, is 1.2% of 1/N’s own number and indistinguishable from noise at 249 days.
Persistence Kills Every Price-Level Forecast
Any error metric computed on a price level can be settled by one pass of “predict the previous value.” The loop computed a reference table from 94 real level series in three regimes, taking the series least favourable to the null in each:
| persistence R² on a price level | persistence MAPE on a price level |
|---|---|
| US equity, minute: 0.99961 | US equity, minute: 0.130% |
| US equity, daily: 0.92588 | US equity, daily: 2.065% |
| crypto, daily: 0.53617 | crypto, daily: 7.355% |
The first paper this loop read reported R² = 0.9975 on 5-minute bars as “superior predictive accuracy.” Persistence reaches 0.99961 on real minute bars. A level forecast that close to persistence is not predictive skill; it is a delayed copy of the last price. Scored on returns — the column that actually decides P&L — the same rule gives R² approximately 0.
This sits naturally beside the rough-volatility result of Gatheral, Jaisson and Rosenbaum (2018). Volatility is rough, and price increments at high frequency are nearly unpredictable; price levels, by contrast, are extremely persistent. The persistence null is the audit-side consequence of that asymmetry. A level MAPE outside the observed reference range should be treated as a misreading, not an achievement. The loop later bounded level MAPE claims by what the reference table shows one looks like — 0.055% to 7.36% across 94 series, with a ceiling of 25% — and bounded R² by [−1, 1]. That gate alone removed 27 of 156 candidates, including seven R²-as-a-percentage cases: 99.41, 98.36 and 96 among them.
A Sharpe Ratio Without a Random-Policy Null Is a Single Draw
A softmax portfolio head over n names emits a point on the simplex. Dirichlet(1) draws over the same n, redrawn at every rebalance, are that agent with skill removed and nothing else changed. The loop ran 2,000 draws over 249 out-of-sample days on the DJIA universe:
| n | p5 | median | p95 | 5–95 spread |
|---|---|---|---|---|
| 5 | −0.441 | +0.935 | +2.428 | 2.868 |
| 10 | −0.105 | +1.124 | +2.383 | 2.488 |
| 25 | +0.613 | +1.436 | +2.282 | 1.669 |
| 30 | +0.764 | +1.487 | +2.243 | 1.479 |
At the full 30 names, the probability that a random allocator beats 1/N is 34.9%; at n = 25 it is 32.7%; at n = 10 it is 25.2%; at n = 5 it is 21.6%.
This is the deflated-Sharpe argument of Bailey and López de Prado (2014) implemented without distributional assumptions. A single reported Sharpe with no seed distribution and no random-policy null beside it is one draw from a distribution roughly 2.5 Sharpe wide. A paper reporting Sharpe +1.0 at n = 30 is below the median of the no-skill distribution. The problem is not that the strategy is bad; the problem is that the record does not contain enough information to tell. The loop refuses to close such a paper.
Sharpe, Sortino and Omega Are One Fact
Ratios computed on the same return series are not independent evidence. The loop measured Spearman rank correlations across 2,000 random long-only portfolios on identical rows:
| metric pair | Spearman correlation |
|---|---|
| Sharpe / Omega | +0.9984 |
| Sharpe / Sortino | +0.9970 |
| Sortino / Omega | +0.9961 |
| Sortino / Calmar | +0.9022 |
| Sharpe / Calmar | +0.9012 |
| Omega / Calmar | +0.8993 |
Using a threshold of 0.95, “higher Return, higher Sharpe, higher Sortino, higher Omega” is one fact reported four times. The measured exception is Calmar, because it is drawdown-based. A paper that reports both Sharpe and Calmar is genuinely reporting two things; a paper that reports Sharpe, Sortino and Omega is reporting one thing with three brand names. This matters more than it sounds: abstract-level screens can be fooled by three identical facts stacked into a paragraph. The loop’s correction is to treat ratio metrics as one statistic unless a drawdown-based metric is also present.
Rank Correlations Live on a Finite Grid
Spearman’s rho on n untied ranks cannot take arbitrary values. At n = 5 there are 120 permutations. Because rho = 1 − sum(d²)/20 and sum(d²) is always even, attainable values are spaced exactly 0.1 apart. The smallest two-sided p the test can produce is 0.0167. This is not a subtle statistical property; it is exact arithmetic.
The paper in the corpus reported rho = 0.94, p = 0.017 over five assets. The rho is not on the grid. The p-value is the exact p of a perfect ranking — which is also the floor of the test. The nearest attainable rho, 0.90, carries p = 0.0833, which is not significant at 5%. The reported pair is internally impossible. A deterministic grid check costs no API calls, uses no market data, and cannot be argued with. It is one of three nulls in the loop that need no data at all.
The Model Earned Its Place by Extracting, Not Reasoning
The language model in this loop is not allowed to conclude anything. It earned a place first as an extractor. Two extractors were scored against the same 29 hand-written labels, written from notes taken when each paper was read, before either extractor existed:
| field | regex | model | delta |
|---|---|---|---|
| target kind | 0.621 | 0.793 | +0.172 |
| learned object | 0.759 | 0.897 | +0.138 |
| has a magnitude | 0.862 | 0.931 | +0.069 |
The model fixes what a lexicon cannot: an LSTM price-prediction paper wrongly filed as “direction” because its abstract also said “trend”; a review article filed as “forecast”; a covariance denoiser filed as “decide.” Pointed at the regexes’ headline claim — 754 papers report an error metric on a price level — the model confirmed 608 (80.6%) and demoted 146 (19.4%): 76 to return, 33 to direction, 22 to volatility. The screen was wrong 19.4% of the time. One demotion: “Evaluating COVID-19 Feature Contributions to Bitcoin Return Forecasting” is a return paper; the regex fired on price vocabulary.
The same empirical discipline exposed a model that did not earn its place. A cross-encoder reranker, cross-encoder/ms-marco-MiniLM-L-6-v2, was added to the loop’s memory retrieval. On 29 identity-labelled probes — query with a paper’s own abstract and require the loop’s note about that paper back among 28 near-neighbours — the ablation was unambiguous:
| metric | BM25 only | + dense | + cross-encoder |
|---|---|---|---|
| recall@1 | 0.931 | 0.897 | 0.724 |
| recall@3 | 0.966 | 0.931 | 0.966 |
| MRR | 0.943 | 0.928 | 0.846 |
Deleting the learned model improves retrieval: recall@1 falls from 0.931 to 0.724, MRR from 0.943 to 0.846, and the reranker adds 190 ms to the critical path. The obvious excuse — ms-marco cross-encoders are trained on short queries and truncate at 512 tokens — was tested and rejected. Sweeping query length produced MRR deltas of −0.070 at 400 characters, −0.100 at 800, and −0.082 at 1,500. The penalty does not shrink as the query shortens. The reranker ships switched off, with the measurement recorded beside it.
The embedding model, by contrast, was upgraded on evidence. Swapping BAAI/bge-small-en-v1.5 (384-dim) for BAAI/bge-base-en-v1.5 (768-dim) moved recall@1 from 0.897 to 0.966 and recall@3 to 1.000, flipping the dense leg from actively harmful (−0.034 against BM25 alone) to the best arm. The lesson is not “bigger embeddings are better.” It is that every learned component in an audit loop must be ablated against the deterministic baseline it claims to improve.
The Model Proposes; the Engine Disposes
The trading engine this loop borrows from uses two OS processes exchanging JSON on disk, because no plane may hold both a model and the engine — and the metrics returned by a backtest deliberately omit the p-value, because a backtest may not self-certify. NautilusTrader, the execution engine this research loop sits beside, embodies that separation in production. The research loop enforces the same discipline.
A proposal is a mapping, never a verdict. A proposal carrying any of refuted, confirmed, verdict, p_value, is_alpha, significant, or tradable is rejected before it is read. Three deterministic gates follow: the cited rule must be one the loop actually holds, not an invented one; the data required must be a subset of what the repo actually has; and a proposal naming no null or no decisive number is refused. Admissibility on the labelled set is 29 of 29.
The honest failure is worth framing precisely. On the one question with an independent label — seven of those 29 papers were recorded NOT_IMPLEMENTABLE because the data was not there, a verdict written many ticks before this prompt existed — the model said reachable: true for all 29. Recall on unreachable was 0.000. Handed an inventory and told to propose a test, it mapped every paper onto something; an African-equity microstructure study became a test on US equities. One definitional fix, carrying no label information — “running the idea on a different universe is substitution, not reachability; saying no is a correct and common answer” — moved recall on unreachable from 0.000 to 0.429, with precision 0.500. Overall agreement did not move: 0.759, because three correct catches cost three false alarms.
Engineering discipline matters here too. DeepSeek’s prefix-cache guide describes disk-based prompt caching; the loop’s hit rate went from 17.7% to 76.7% on papers called for the first time via three changes: removing a per-paper string from the middle of the shared prefix, warming the cache before fanning out to 8 threads, and keeping task instructions in the system message. A fourth change was tried and reverted: it measured 63.6% on the first question and 56.7% on the second, no better, because common-prefix detection needs a third request before the shared prefix pays off. Every model call is recorded with the SHA-256 of the prompt template that produced it; a recording whose hash does not match today’s template is refused rather than replayed. Changed prompt means changed model, enforced mechanically, with downstream consumers running on no API key. Machine screens and human readings are counted separately — merging 82 cheap screens with 29 careful readings into one verdict mix would be a survivorship problem wearing a different hat.
Three Ways It Nearly Fabricated a Refutation
All three failures were caught by running the pipeline, not by reasoning about it.
First, a relative improvement wearing an absolute metric’s name. The first paper the executor reached reported “a 55% MAPE reduction.” The extractor faithfully returned metric = MAPE, value = 55. The executor compared 55 against a persistence MAPE of 2.06% and declared the null the winner. The plausibility gate — level MAPE bounded by 0.055% to 7.36%, ceiling 25 — and a gate reading the model’s own explanation for “reduction,” “improvement,” and “decrease” removed 27 of 156 candidates, including seven R²-reported-as-percentage cases.
Second, a rule that got to pick its own regime. The first version fired if persistence won in any of the three reference regimes. That let a Bitcoin paper reporting daily MAPE 0.19% be refuted by the equity-minute reference of 0.13% — a different asset at a different frequency. A rule with a choice takes the one that wins. Requiring persistence to win in all regimes removed the choice and the abuse: only 3 of 129 papers could be closed. The fix was to ask the model for asset class and frequency and compare against the matching reference, refusing when the abstract states neither. That took it to 21.
Third, a materiality floor of zero. The floor is described above: buy-and-hold is +0.019, a 1.2% slice of 1/N’s number, and a zero threshold closed a paper on a +0.109 momentum edge. The 0.25 Sharpe floor is a convention, not a theorem, but it exists because the loop measured what zero does.
The Ceiling Is the Corpus, Not the Model
Reach was measured across 705 papers that have both a model adjudication and a proposal:
| outcome | papers | share |
|---|---|---|
| rule not yet lifted into the deterministic library | 350 | 49.6% |
| inputs absent from the abstract | 322 | 45.7% |
| would close | 21 | 3.0% |
| screen runs, does not settle | 12 | 1.7% |
Four independent measurements say the same thing. Only 44.7% of the 21,305-paper corpus names at least one performance metric, and only 21.3% attaches a value to one; 23.4% name a metric and give no number at all. On the papers where the no-skill policy null applies, the asset count is missing from both extractors. Corpus-wide, only 60 papers of 21,305 name two ratio-family metrics in their abstract. And the regime-matching fix worked only because the model could read asset class and frequency off the text — which it can do for roughly half of them.
Half the gap is mechanical: ten more nulls, each already written once by hand, would lift “rule not lifted” into “inputs absent.” Lifting them does not create closures. Better retrieval does not fix the ceiling. A larger model does not fix it. Full paper text would.
Practical Takeaways: A Decision Framework for Alpha Audits
If I were building a paper-screening pipeline from these measurements, the order of operations would be fixed:
- Is there a decisive number? If no value is attached to a metric, no machine verdict is possible. From the corpus, only 21.3% of abstracts attach a value to a named metric; route the rest to a human reading.
- Is the metric computed on a price level? Run the persistence pass. A level R² near 0.9975 on 5-minute bars is not impressive when the persistence reference is 0.99961; a level MAPE outside the 0.055%–7.36% reference range is a misreading. Re-score on returns before believing anything.
- Does the paper report Sharpe, Sortino, or Omega? Treat them as one fact unless a drawdown-based metric such as Calmar is also present. The Spearman correlations are 0.9984, 0.9970 and 0.9961; they are duplicate evidence.
- Does it report a rank correlation? Check the finite grid. At n = 5, rho values must be multiples of 0.1 and no p-value below 0.0167 is attainable. The paper claiming rho = 0.94, p = 0.017 fails on both counts.
- Does it report an allocation result? Run a Dirichlet(1) no-skill null. At n = 30, the median random Sharpe is +1.487 and the 5–95 spread is 1.479; a point inside that interval is indistinguishable from a no-skill policy. Report the seed distribution, not just the point.
- Compare against zero-parameter baselines. If 1/N beats the paper’s chosen benchmark by more than the paper claims to beat it, the delta measured the opponent. Enforce a materiality floor in performance units, not p-units.
- Separate proposing from concluding. Reject records containing
refuted,confirmed,p_value,is_alpha,significant, ortradablebefore they are read. Hash prompt templates; refuse mismatched replays; count machine screens and human readings separately. - Ablate every learned component. The cross-encoder cost 5 of 29 probes at rank 1 and 190 ms, and deleting it improved retrieval. The embedding upgrade from 384-dim to 768-dim moved recall@1 from 0.897 to 0.966. Do not keep a model in the loop because it is modern; keep it because it measured better.
None of these rules requires a large language model. The first three can be applied with a spreadsheet. The role of the AI is to apply them 21,305 times, to flag the 97% where the abstract does not contain enough information to decide — and to refuse to certify anything on its own authority. The machine screen does not read the method, check the ablations, or find the second defect a careful human reading usually finds. Every machine record in this loop says so in its own text.
The broader implication is uncomfortable for the alpha-generation industry. The bottleneck in trading research is not too little AI; it is too little audited arithmetic. A paper can be written, cited, and built upon while reporting a Spearman statistic that cannot exist. The fix is not smarter prediction. It is deterministic nulls applied at scale, a model that is structurally forbidden from passing its own verdict, and the mechanical honesty to record what the corpus cannot support. Add the persistence pass to your review queue before you add another reranker. It costs zero parameters — and it was enough to deflate a 0.9975 R² claim before any model had to say a word.