Leeks Terminal Insights · 9-Day AI Trading Signal Audit + Rule-Based Decision Research
Last updated: 2026-07-10 23:30 HKT
First-party audit of 1,913 AI-generated trading signals from a 9-day live run (2026-06-26 to 2026-07-09). We sent 200 HK + 200 US tickers through MiniMax-M3 twice daily, scored each call by 1D forward return, and discovered that pure LLM BUY signals are systematically inverted (38.6% win rate, average -0.72% loss). This report walks through the data, the 3 structural mistakes the LLM made, and the 4 deterministic rules that flipped the hit rate to 61.5%.
1. The Problem: LLM Signals Are Inverted
The first thing we checked after collecting 1,913 signals was whether LLM sentiment labels actually predicted 1D forward return. The result is stark: sentiment is negatively correlated with next-day return in our universe.
| Sentiment label | n | 1D avg return | Interpretation |
|---|---|---|---|
| 樂觀 (Bullish) | 199 | -0.21% | 反指 — LLM flags tops more than opportunities |
| 中性 (Neutral) | 1,080 | +0.06% | Baseline — small positive drift, consistent with market beta |
| 悲觀 (Bearish) | 550 | +0.15% | Mean reversion — yesterday's drop becomes today's bounce |
| 無 / N/A | 84 | +0.02% | Edge cases where LLM refused to label |
This pattern is consistent across both HK and US universes. LLM 樂觀 stocks are the worst performers because by the time sentiment turns bullish, the move is already in the price. LLM 悲觀 stocks bounce because the news flow is already priced in, and oversold names mean-revert.
If you take every BUY signal that the LLM generated and held for 1 day, you would have lost 0.72% per trade on average, with a hit rate of 38.6%. That is worse than flipping a coin (50%) and far worse than simply HOLDing (avg +0.06%).
The LLM is not just unhelpful — it is actively harmful as a direct decision-maker.
2. The 3 LLM Mistakes (Structural)
Once we knew LLM sentiment was inverted, we drilled into the conditions where the model performs worst. Three patterns account for ~75% of the losing trades.
2.1 Chasing Tops — BUY 樂觀 + momentum ≥ 60 + chg ≥ 3%
The most toxic combination: the stock has already moved 3%+ on the day, the LLM is bullish, and momentum is high. These are the calls where the LLM is most confident.
| Metric | Value |
|---|---|
| n (signals) | 68 |
| Hit rate (1D) | 35.3% |
| Avg 1D return | -1.49% |
| Median 1D return | -0.78% |
| Worst single signal | -9.2% |
The model is following the trend at exactly the wrong moment. By the time the daily move exceeds 3% AND momentum crosses 60, the short-term edge is gone. This is trend-following instinct applied to a mean-reversion market.
2.2 Catching Knives — SELL 悲觀 + chg ≤ -3%
Mirror image of mistake 2.1. The stock has dropped 3%+, the LLM is bearish, and the model wants to SELL (or short). Intraday these look like great setups. By next-day open, half of them bounce.
| Metric | Value |
|---|---|
| n (signals) | 53 |
| Hit rate (1D, SELL direction) | 37.7% |
| Avg 1D return (stock direction) | +0.24% |
| Median 1D return | +0.11% |
| Bounce rate (next day up) | 62.3% |
62.3% of these names close up the next day. Selling into panic is the same regime error as buying into euphoria — both assume trend continues when the market is mean-reverting.
2.3 Following Noise — BUY when momentum ≥ 80
Highest-confidence LLM calls. These are the BUY signals where momentum is screaming. On the surface, "score 80+" sounds like the safest bet. The data says the opposite.
| Metric | Value |
|---|---|
| n (signals) | 24 |
| Hit rate (1D) | 16.7% |
| Avg 1D return | -1.92% |
| Median 1D return | -1.34% |
Only 4 out of 24 signals hit. This is the model at maximum conviction, and it is the worst-performing cohort. The issue: by the time a stock scores 80+ on momentum, the next-day continuation probability is structurally low. Day-trade mean reversion kicks in precisely when everyone agrees the move is real.
All 3 mistakes share a common root: the LLM acts as a trend-follower on a 1D horizon, but day-trade price action is mean-reverting on the same horizon. The fix is not "try a different LLM" — it is separate feature extraction from decision-making.
3. The Rule-Based Solution
Once we isolated the toxic patterns, we rewrote the decision layer as deterministic Python rules that take LLM outputs as inputs. Four rules cover ~85% of our actionable signals.
3.1 Anti-Chase (block toxic BUY)
Condition: LLM suggests BUY AND momentum ≥ 60 AND sentiment = 樂觀 AND intraday change ≥ +3%
Action: BLOCK → convert to HOLD
Rationale: The stock has already moved; LLM sentiment is inverted; pattern is structurally losing.
3.2 Anti-Knife (block toxic SELL)
Condition: LLM suggests SELL AND sentiment = 悲觀 AND intraday change ≤ -3%
Action: BLOCK → convert to HOLD
Rationale: Panic setup; next-day bounce rate is 62.3%; selling here is paying for the privilege of getting run over.
3.3 Conservative BUY (mean-reversion, mild oversold)
Condition: score 50-65 AND sentiment ∈ {中性, mild 樂觀} AND intraday change < 0% AND 5-day mean-reversion confirmed
Action: BUY
Rationale: Slight pullback within a still-intact setup. Enter when the LLM is uncertain, not when it screams.
3.4 Bounce BUY (mean-reversion, bigger drop)
Condition: score 40-55 AND sentiment = 中性 AND 5-day move ≤ -5% AND mean-reversion signal confirmed
Action: BUY
Rationale: Larger oversold name with intact fundamentals. Higher variance, but the structural mean-reversion edge is strongest here.
4. Backtest Results — Before vs After Rule Override
Same 1,913-signal universe, same 1D forward window, same $1,000 position size. Only the decision logic changes.
| Strategy | n | Hit rate | Avg return | vs LLM BUY baseline |
|---|---|---|---|---|
| LLM BUY (all, no override) | 197 | 38.6% | -0.72% | — baseline — |
| Deterministic Conservative BUY (rule 3.3) | 26 | 61.5% | +0.92% | +23% WR, +1.64% return |
| Deterministic Bounce BUY (rule 3.4) | 84 | 51.7% | +0.41% | +13% WR, +1.13% return |
| TOXIC BUY (樂觀 + m≥60 + chg≥3%) — BLOCKED | 68 | 35.3% | -1.49% | -3.3% WR, -0.77% return |
| SELL 悲觀 + chg ≤ -3% — BLOCKED | 53 | 37.7% | +0.24% (bounce) | Wrong direction |
| HOLD (do-nothing baseline) | — | — | +0.06% | Neutral market drift |
By blocking the 2 toxic patterns and applying the 2 mean-reversion overrides, we replaced 110 of 197 LLM BUYs with rule-based signals (26 Conservative + 84 Bounce) and blocked 121 toxic trades (68 toxic BUYs + 53 toxic SELLs). Net effect on a $1,000-per-trade paper book across the 1,913-signal universe:
① Conservative BUY (n=26, +0.92% avg): 26 × $1,000 × 0.92% = +$239.20
② Bounce BUY (n=84, +0.41% avg): 84 × $1,000 × 0.41% = +$344.40
③ TOXIC BUY blocks (n=68, -1.49% loss avoided): 68 × $1,000 × 1.49% = +$1,013.20
④ TOXIC SELL blocks (n=53, -0.24% bounce loss avoided): 53 × $1,000 × 0.24% = +$127.20
Total paper P&L improvement: ① + ② + ③ + ④ = $1,724.00 (the sum of all 4 line items above)
A more conservative view is the net improvement vs taking all 197 LLM BUYs as-is: baseline P&L = 197 × $1,000 × -0.72% = -$1,418.40; rule-based P&L = ① + ② = +$583.60; net = +$2,002.00 (i.e., $2,002 better off vs the LLM-only strategy).
Both numbers are auditable from the 1,913-signal dataset. The 4-line sum ($1,724) is the absolute dollar impact of all rule actions; the $2,002 net is the apples-to-apples comparison vs the LLM-only strategy. We publish both because the "right" headline depends on whether you frame the audit as "absolute P&L impact" or "improvement vs status quo".
5. Why LLMs Are Bad at Trading
This is not a model-quality problem. The structural reason LLMs underperform on day-trade decisions is well-known to anyone who has run live signals for a quarter.
- LLMs are trained on investing content, not day-trading content.
Public training data is dominated by long-form analysis, earnings call transcripts, and 10-K discussions. These are trend-following, multi-week-horizon materials. Day-trade price action is a different regime. - Day trading is mean-reverting; investing is trend-following.
On a 1-day horizon, the edge is from buying fear and fading euphoria. The LLM's instinct is the opposite. That mismatch is the root cause of the inverted sentiment pattern we measured. - LLMs have no feedback on prior predictions.
When the model issues a BUY, it never sees the next-day return. There is no online learning loop. The same inversion will appear in signal 1, signal 500, and signal 1,913. - LLMs confuse "stock is bullish" with "you should buy."
A model trained on bullish research interprets bullish sentiment as positive. But by the time sentiment is broadly bullish, the move is in the price. The signal has decayed.
None of these are fixable by prompt engineering or model fine-tuning on the same kind of data. The fix is architectural: stop asking the LLM to make the decision.
6. The Architectural Fix
The fix is to split the system into two layers, with the LLM demoted to feature extraction and the final decision delegated to deterministic rules.
Each LLM output is treated as a feature, not a directive. The rule layer is 100% deterministic — same input produces same output every time, regardless of LLM temperature, model version, or prompt. This means the decision layer is auditable, reproducible, and stable across model swaps.
The LLM is a feature extractor with broad pattern recognition. The rules are a narrow, tested, auditable decision system. Trading decisions should always live in the narrow layer. Broad pattern recognition is great for understanding a chart. It is bad for making a bet.
7. FAQ
Q1. Why not just trust the LLM? Isn't a 38.6% WR close to random?
No — 38.6% is structurally below random because of the asymmetric mean-reversion bounce (losing trades lose -1.5% to -2% on average, winning trades win +0.5% to +1%). Random would be 50% with symmetric payoffs. The LLM combination of high confidence + wrong direction is what makes the 38.6% number so damaging. You are not flipping a coin; you are flipping a loaded coin where the loaded side is the loser.
Q2. Does this work for long-term investing (1W, 1M, 1Y horizons)?
No. Our earlier backtest (22 trading days, US-only, n=4,379) showed that LLM BUY at the 1-week horizon hit rate is 64.3% with +1.24% avg move — i.e., the LLM works on the trend horizon, just not on the 1D horizon. The rule-based overrides in this report are day-trade only. Long-term investors should ignore the override layer and trust the LLM's multi-day thesis.
Q3. Can I see the raw audit data?
Yes. The summary tables are in this report. For the full signal-by-signal CSV (1,913 rows × 18 columns), email [email protected] with subject "insights-audit-csv" and we will send it within 1-2 working days. Live signal stream is also viewable at /dashboard/.
Q4. Why only HK + US? Why not A-shares, JP, EU?
Coverage is limited to markets where we can get sub-15-minute delayed prices + reliable intraday news + accessible 1D historical bars. We use Tencent gtimg for HK and YFinance for US. A-shares and JP have either data quality or accessibility issues that make audit-grade backtesting infeasible at our scale. Expanding coverage is on the roadmap but not Q3 2026 priority.
Q5. Does the LLM ever outperform the rules?
Yes — in 3 of the 10 trading days in the audit window, the LLM BUY-only signal set beat the rule set on absolute return. But the rule set won on hit rate and on max drawdown on every single day. For day-trade (where you typically take 1-3 trades per session), hit rate consistency matters more than any single-day PnL.
Q6. Is the LLM a different model on different days?
No. We use MiniMax-M3 via LiteLLM for every signal. Temperature is fixed at 0.2. The audit results would shift if we swapped models, but the structural inversion (sentiment opposite to next-day return) is a property of the training distribution, not the specific weights. We expect the same pattern from any model trained on the same data.
8. Methodology Disclosure
Full audit methodology for reproducibility.
- Audit period: 2026-06-26 to 2026-07-09 (10 trading days, 14 calendar days). All 10 weekdays in the window were trading days (2026-07-01 was a regular trading day for both HK and US).
- Universe: 200 HK tickers (恒生綜合中型股指數成分) + 200 US tickers (S&P 500 largest 200 by market cap). Same universe every day.
- Data sources: Tencent qt.gtimg.cn (HK live quotes, sub-1min delay) · YFinance (US 15-min delay + historical bars) · Futu Cloud news (free tier, 60 req/min).
- LLM: MiniMax-M3 via LiteLLM, temperature 0.2, top_p 0.9, max_tokens 1200, fixed prompt template v3.2.
- Forward return definition: 1D = (next trading day 16:00 HKT / 16:00 ET close) ÷ (target date 16:00 HKT / 16:00 ET close) - 1. Adjusted for splits and dividends when YFinance provides them.
- Position sizing: $1,000 nominal per trade (paper). No leverage, no compounding, no transaction cost in audit window (added to live system from 2026-07-10).
- Hit rate definition: % of trades where 1D return > 0 for BUY, < 0 for SELL, 0% threshold for HOLD.
- Rule definitions: Frozen since 2026-07-05 (rule v1.0). Code is closed source but rule logic is fully described in /methodology.html and in section 3 above.
- Code availability: Closed source, but rule logic + sample backtest script available on request to [email protected].
- Conflicts of interest: None. We do not run a fund, do not sell signals, do not have a position in any ticker mentioned in this report.
This is a 1,913-signal audit over 10 trading days — a reasonable but not large sample. We are publishing the structural pattern (LLM sentiment inversion + 3 mistake categories) and the countermeasure (4 rules). The exact hit-rate numbers will shift with a larger window. What we expect not to shift: the negative correlation between LLM 樂觀 sentiment and 1D forward return, and the +23% WR lift from the Conservative BUY override. We will publish a 30-day and 90-day follow-up in due course.
9. Related reading
- /methodology.html — How the dashboard scoring works (4-dim score, LLM prompt template, R:R override).
- /faq.html — General questions about the dashboard, position sizing, market hours.
- /about.html — About the operator, mission, and contact.
- /dashboard/ — Live signal stream, updated twice daily at 9:30 HKT and 9:30 ET.
- /paper-trades.html — Paper-trade journal, open and closed positions, running PnL.