HackerTrans
トップ新着トレンドコメント過去質問紹介求人

jakozaur

5,629 カルマ登録 18 年前
Email: jacek (at) migdal.pl

投稿

Make Europe Cool Again

jacek.migdal.pl
5 ポイント·投稿者 jakozaur·昨日·0 コメント

We hid backdoors in ~40MB binaries and asked AI + Ghidra to find them

quesma.com
245 ポイント·投稿者 jakozaur·5 か月前·98 コメント

OTelBench: Can AI instrument OpenTelemetry?

quesma.com
5 ポイント·投稿者 jakozaur·6 か月前·1 コメント

Nano Banana Pro: raw intelligence with tool use

quesma.com
4 ポイント·投稿者 jakozaur·8 か月前·0 コメント

A postmortem on our $2.5M database gateway: lessons from pilot purgatory

quesma.com
6 ポイント·投稿者 jakozaur·8 か月前·0 コメント

The security paradox of local LLMs

quesma.com
160 ポイント·投稿者 jakozaur·9 か月前·87 コメント

Local LLMs are worse for security

quesma.com
1 ポイント·投稿者 jakozaur·9 か月前·0 コメント

CompileBench: Can AI Compile 22-year-old Code?

quesma.com
148 ポイント·投稿者 jakozaur·10 か月前·65 コメント

Winners of OpenAI GPT-OSS-20B Red‑Teaming Challenge

kaggle.com
1 ポイント·投稿者 jakozaur·10 か月前·0 コメント

コメント

jakozaur
·15 日前·議論
It's rather hard to do at the proxy level with agentic coding, such as Claude Code or similar. These are long-chained sessions of tool use that heavily rely on prompt caching. Changing mid-flight is costly.

It looks like much more context is required to decide on the best model (e.g., summarizing logs might use a cheap model, whereas you likely want Opus/Mythos/GPT 5.6 to debug multithreading logic). In an agentic system, a decision about the model may be embedded in the decision to orchestrate the model.
jakozaur
·2 か月前·議論
Poland's first partly free election was on 4 June 1989, preceded by the roundtable negotiations.

The protests in Czechoslovakia came later, called the Velvet Revolution, from 17 to 28 November 1989. In June 1990, Czechoslovakia held its first democratic elections, a year after Poland.

Poland paved the way for the whole of central and eastern Europe. The Round Table produced the negotiated-exit template that Hungary built on in its own talks that summer, and that Czechoslovakia, East Germany, and the Baltics drew on as their regimes fell within months.

And it did so from the deepest macroeconomic crisis of any of the satellite states: hyperinflation running into the hundreds of percent by late 1989, an unresolved sovereign default from 1981, and chronic shortages.

Since then Poland has converged fastest of any of them. From a low base it has climbed to the upper-middle of central and eastern Europe by GDP per capita PPP, overtaken Hungary, and is now closing on Czechia and Slovenia.
jakozaur
·2 か月前·議論
The story is longer: Poland was the first country to make a remarkable peaceful transition from a bankrupt, failed Soviet satellite state. The shock therapy, plus NATO and EU aspirations, paved the way.

It is a story of a country that made a lot of the right decisions along the way. Managed to keep consistent high growth, not a pony trick or boom/bust mode.

Poland should be a role model for many other countries.

Recommend a book: https://www.amazon.com/Europes-Growth-Champion-Insights-Econ...

And Noah's blog post: https://www.noahpinion.blog/p/the-polandmalaysia-model
jakozaur
·2 か月前·議論
A heat pump could win as the best HVAC technology, though a better drilling for ground-sourced ones. Just a shallow drilling (up to 100m) that works in retrofit mode, such as drilling from the basement, would be a great upgrade:

- No outdoor unit that looks awful in many settings

- works well, even in the coldest winter, without a spike in electricity usage, COP 5

- very reliable with long durability

- super quiet, no ambient noise

- 20% more efficient

Currently, drilling is very disruptive in retrofits, but there is progress in compact techniques that might change the equation.

Disclaimer: angel investor in https://www.flexdrill.at/
jakozaur
·4 か月前·議論
Build systems are tested by CompileBench (Quesma's benchmark).

Disclaimer: I'm the founder.
jakozaur
·5 か月前·議論
I can provide trajectories. Though probably we are not going to publish them this time. This would need some extra safeguards.

Email me. The address is in profile.
jakozaur
·5 か月前·議論
As I do eval and training data sets for living, in niche skills, you can find plenty of surprises.

The code is open-source; you can run it yourself using Harbor Framework:

git clone [email protected]:QuesmaOrg/BinaryAudit.git

export OPENROUTER_API_KEY=...

harbor run --path tasks --task-name lighttpd-* --agent terminus-2 --model openrouter/anthropic/claude-opus-4.6 --model openrouter/google/gemini-3-pro-preview --model openrouter/openai/gpt-5.2 --n-attempts 3

Please open PR if you find something interesting, though our domain experts spend fair amount of time looking at trajectories.
jakozaur
·5 か月前·議論
So many models refuse to do that due to alignment and safety concerns. So cross-model comparison doesn't make sense. We do, however, require proof (such as providing a location in binary) that is hard to game. So the model not only has to say there is a backdoor, but also point out the location.

Your approach, however, makes a lot of sense if you are ready to have your own custom or fine-tuned model.
jakozaur
·5 か月前·議論
Oh, nice find... We end up using PyGhidra, but the models waste some cycles because of bad ergonomics. Perhaps your cli would be easier.

Still, Ghidra's most painful limitation was extremely slow time with Go Lang. We had to exclude that example from the benchmark.
jakozaur
·5 か月前·議論
See direct benchmark link: https://quesma.com/benchmarks/binaryaudit/

Open-source GitHub: https://github.com/QuesmaOrg/BinaryAudit
jakozaur
·5 か月前·議論
Funny thing, AI is not that terrible at using Ghidra. We released a benchmark on that and hopefully models will improve: https://quesma.com/blog/introducing-binaryaudit/
jakozaur
·5 か月前·議論
Funny coincidence, I'm working on a benchmark showcasing AI capabilities in binary analysis.

Actually, AI has huge potential for superhuman capabilities in reverse engineering. This is an extremely tedious job with low productivity. Currently reserved, primarily when there is no other option (e.g., malware analysis). AI can make binary analysis go mainstream for proactive audits to secure against supply-chain attacks.
jakozaur
·5 か月前·議論
Great idea. Currently, people have to rely on client-side spans in OpenTelemetry. However, it would be awesome if we could get spans for slow SQL queries, along with explanations.
jakozaur
·5 か月前·議論
In this benchmark, micro-services are really small, ~300 lines, and sometimes just two of them. More realistic tasks (large codebases, more microservices) would have a lower success rate.
jakozaur
·6 か月前·議論
See x thread for rationale: https://x.com/mitchellh/status/2014433315261124760?s=46&t=FU...

“ Ultimately, I want to see full session transcripts, but we don't have enough tool support for that broadly.”

I have a side project, git-prompt-story to attach Claude Vode session in GitHub git notes. Though it is not that simple to do automatic (e.g. i need to redact credentials).
jakozaur
·7 か月前·議論
I wish Neal would do behind the scenes, how he built this art. I wonder whether LLM assistants like Claude Code make such an interactive show more feasible.

He previously did a game "Infinite Craft" which leveraged Llama models. However, I was only able to find an outdated blog from 2019.
jakozaur
·7 か月前·議論
Not sure if I get this: WASM lets you use any language in the browser, though it still works way better with languages without GC, such as Rust or a transpiling C engine. Java is unlikely to be the best choice.

In the era of LLM assistants like Claude Code, any engineer can write frontend code using popular stacks like React and TypeScript. This use case is when those tools shine.
jakozaur
·7 か月前·議論
LLVM IR is quite fun to play with from many programming languages. The Java example is rather educational, but there are several practical example,s such as in Go Lang:

https://github.com/llir/llvm
jakozaur
·7 か月前·議論
The effect of climate change may be highly uneven. Some regions will be fine with adaptation, while other places will hardly sustain cities.
jakozaur
·7 か月前·議論
It is even more true with startups and business. Super rushed is bad, but doing for too long decreases quality.