To be more specific: the prompt defines a trading philosophy and tells models what to look for in the charts. But the actual read and the decision is entirely on the model. Using your framing — it's closer to "here's inspiration, now maximize money" than "implement this exact strategy."
Which means improvisation within that framework is exactly what's being measured.
That's a fair point. You're right that without seeing the strategy, you can't fully disentangle what drives the differences.
But the strategy itself isn't really the point. Since every model gets the exact same prompt and the exact same market data, the only variable is the model. So relative performance differences are real regardless of what the strategy contains. If Model A consistently outperforms Model B under identical conditions, that tells you something meaningful about the model.
And honestly, that blend of prompt adherence and decision quality is how people actually use LLMs in practice. You give it instructions and context, and you care about the result.
You're right though that the strategy being private limits what outsiders can evaluate. It's something I'm thinking about.
I've been day trading indices (DAX, Nasdaq) profitably throughout 2025 using a specific personal strategy. At some point I got curious: if I gave this exact strategy to different LLMs as a prompt, which one would execute it best?
I couldn't find a benchmark that tested this. The academic ones focus on stock portfolios with daily rebalancing. Nothing tested LLMs on fast-paced index day trading where you need to read price action and make quick directional calls based on a defined strategy.
So I built DayTradingBench. The core idea is simple: every model receives the exact same prompt containing my trading strategy and the exact same live market data. The only variable is the model itself. This way I'm measuring pure decision-making capability — not prompt engineering.
How it works:
- 11 LLMs trade autonomously during live market hours (08:00–21:00 UTC, Mon–Fri)
- Every 15 minutes each model gets a market snapshot and must return a structured JSON decision: LONG, SHORT, or HOLD — with stop loss, take profit, confidence, and risk percentage
- All start with the same $100k virtual balance
- Positions auto-close on SL/TP hits (checked every 10 seconds) or at session end
There are two input modes: text mode (structured OHLCV data) and vision mode (candlestick chart images sent to the model). Same strategy prompt, different way of presenting the market data. This lets me compare whether models trade better reading numbers or reading charts.
The performance gap between models is much larger than I expected, even though they all receive identical instructions.
Built this as a solo dev. The system runs fully autonomously 24/5 — I mostly just watch the results now.
Would love to hear what HN thinks, especially if you've experimented with LLMs for trading.