KalskiTheDan

vor 4 Monatendiscuss

> everyone gets a power drill > more holes in more walls > none of them in the right place

Getting the drill wasn't the hard part, knowing where to drill is

KalskiTheDan

vor 4 Monatendiscuss

I ran into this building data pipelines for LLMs. Kept feeding structured JSON with numbers and the model would do bizarre arithmetic on them.

The moment I shifted to computing the analysis upstream and describing results in plain english, the outputs were much more coherent.

The model doesn't know what numbers/spreadsheets of data mean inherently, as in... the LLM inherently does not compute a math equation/formula within itself. Rather, an LLM would calls/creates calculation code on the side to then read off the results. And still with this side-work, the code/calculator created would give contextual wordings to the numbers; highest, average, x% growth, consolidating at y% rate, etc.

KalskiTheDan

vor 4 Monatendiscuss

I'd bet that still to this day, most interview questions test recall (of XYZ) and recall is what LLMs are best at. "Explain the difference between X and Y" or "what would you do in scenario Z" are search queries now. A lot more streamlined than individual google searches people would patch together from their search journey...

If I'd ever be doing a hiring spree, I'd give them a real problem. Put them in a position to think out loud. Not "what's the answer" but "walk me through how you'd approach this." Probe the reasoning. When they hit a fork, ask why they'd go left instead of right. Ask about tradeoffs they see. The best/real candidates will naturally say "I'd need to know more about X before deciding" because they're actually thinking through it.

IMO acknowledging the lack of information (without coming from a place of uninformed/non-experienced) almost impossible to fake with an overlay. LLMs always have an answer. Good engineers know when they don't.

KalskiTheDan

vor 4 Monatendiscuss

[dead]

KalskiTheDan

vor 4 Monatendiscuss

>> AT some point, you get conviction, this will work and worth pursuing full-time even though 0 revenue to begin with.

My conviction was seeing myself using what I built growing from a few times a week to everyday use. Then when one random signup hit the analytics, I watched that person come back consistently too. That was the shift.

The harder part is the gap between "people use it" and "people pay for it". Usage is validating but it doesn't pay rent :( hence I press on.

KalskiTheDan

vor 4 Monatendiscuss

[dead]

KalskiTheDan

vor 4 Monatendiscuss

I'm a solo dev; 6 months in the making. I built a financial context API that returns pre-analyzed market briefings for AI agents (on-chain, macro, regime classification), then ran 7 controlled experiments to find out if it actually helps or just adds noise.

I kept seeing the same pattern in AI agent demos. You hand an LLM a price feed, it gets {"price": 94200, "change_24h": -2.3}, and it burns half its context window figuring out basics. Is this up from last week? What percentile? How does hash rate correlate? The agent does all that work before it starts reasoning about what to do. So I started pre-computing the analysis server-side and returning ~400 token markdown briefings instead of raw JSON.

The experiment: 4-arm RCT. Treatment gets real-time briefings. Control gets price only. A third arm uses web search instead of briefings. Placebo gets the same briefings but time-shifted 5-7 months, presented as current. All arms run Claude, one trading decision per tick.

Latest run, 202 ticks over 6 months. BTC fell 34.7%.

  Treatment (briefings):   +7.83%  | max drawdown 5.95%
  Control (price only):    -8.14%  | max drawdown 15.95%
  Web search arm:          -1.55%  | max drawdown 12.63%
  Placebo (stale data):    -7.70%  | max drawdown 10.17%
  BTC buy-and-hold:       -34.70%

Treatment beat control by +15.97pp. Beat web search by +9.38pp. All 7 experiments positive, range +4.46pp to +15.97pp across two models (Opus 4.6, Sonnet 4.5).

The edge is almost entirely defensive. Treatment's return came from two short campaigns during crashes. In rallies and sideways markets, it matched or underperformed control. Long trades were coin flips.

What didn't work: the earliest run was the worst. Treatment finished last. Rich data with no guardrails caused the agent to flip-flop every tick. BUY, SELL, BUY across three consecutive ticks. $79K traded, zero net position change. A later run was aborted at tick 33 after the agent translated "macro bearish" into "go short" when the right move was cash. 1 of 24 total runs was negative. 5 were inconclusive.

Stale data was worse than no data. Placebo consistently underperformed plain control across runs. Well-structured wrong information is more dangerous than no information.

Things I'm still uncertain about: the edge is untested in a bull market (every window skews bearish), 202 ticks isn't statistically conclusive within a single run (more valued would be years of data/ticks), and the web search arm had contamination risk from future-dated search results.

KalskiTheDan

Submissions

[untitled]

Why Linguistic Context Outperforms Raw Data for LLM Decision-Making

comments