HackerTrans
TopNewTrendsCommentsPastAskShowJobs

seddonm1

no profile record

Submissions

Show HN: Using LLMs and differential testing to convert code

reorchestrate.com
2 points·by seddonm1·hace 5 meses·0 comments

Show HN: Your binary is no longer safe

reorchestrate.com
3 points·by seddonm1·hace 5 meses·0 comments

Bringing a Warhammer to a Knife Fight

reorchestrate.com
2 points·by seddonm1·hace 5 meses·1 comments

comments

seddonm1
·el mes pasado·discuss
Based on the now-deprecated Clear Linux it does seem that these optimizations add up [0] and so maybe we should be considering them more broadly?

[0] https://www.phoronix.com/review/clear-linux-48p-ubuntu/6
seddonm1
·el mes pasado·discuss
I would be interested to know if there is a method similar to this one in Rust [0] that allows a single binary to support multiple optimization levels depending on the executing CPU? It feels wasteful to not enable these optimizations but I don't really want to force a user to choose between a complex feature matrix.

[0] https://github.com/ronnychevalier/cargo-multivers
seddonm1
·hace 4 meses·discuss
I am getting closer and closer to a full verified rewrite in Rust. I have also moved to a much easier sqlite relational structure for the backend.

I actually sidestepped the annoying btrieve problem by exporting the data using a go binary [0] and I write it to a sqlite instance with raw byte arrays (blobs). btreive is weird because it has a dll but also a a service to interact with the files.

P.s. I have spent a lot of hours on this mainly to learn actual LLM capabilities that have improved a huge amount in the last year.

[0] https://github.com/barchart/go-btrieve
seddonm1
·hace 4 meses·discuss
It’s a real problem. I threw it at an old MUD game just to see how hard it is [0] then used differential testing and LLMs to rewrite it [1]. Just seems to be time and money.

[0] https://reorchestrate.com/posts/your-binary-is-no-longer-saf...

[1] https://reorchestrate.com/posts/your-binary-is-no-longer-saf...
seddonm1
·hace 5 meses·discuss
No, I would need to find a binary to test on. I suspect it would produce horrible code at the decompiler layer but ultimately I would expect that function signatures are still relatively clean?

Its scary - once you get the differential testing harness set up it seems to be just a matter of time/tokens for it to stubbornly work through it.
seddonm1
·hace 5 meses·discuss
I have tried to post this here but it has not got traction.

I have a demonstrated process here on my blog (all hand written without AI).

This bit about how to brute force decompilation: https://reorchestrate.com/posts/your-binary-is-no-longer-saf...

And this about how to do the conversion and address the LLM hallucination problem: https://reorchestrate.com/posts/your-binary-is-no-longer-saf...

Yes, it is absolutely possible.
seddonm1
·hace 5 meses·discuss
I am applying differential/property based testing to all the side effects of functions (mutations) and return values. The rust code coverage is also used to steer the LLM as it finds discrepancies in side effects.

It is written up in my link - please bear in mind it is really hard to find the right level to communicate this level of detail at - so I'm happy to answer questions.
seddonm1
·hace 5 meses·discuss
My test harness loads up the original DLL then executes that in parallel against the converted code (differential testing). That closes the feedback loop the LLM needs to be able to find and fix discrepancies.

I'm also doing this on an old Win32 DLL so the task is probably much easier than a lot of code bases.
seddonm1
·hace 5 meses·discuss
I delivered a talk at Rust Sydney about this exact topic last week:

https://reorchestrate.com/posts/your-binary-is-no-longer-saf...

I am able to translate multi-thousand line c functions - and reproduce bug-for-bug implementation
seddonm1
·hace 5 meses·discuss
Hi Ben. I published an article about this problem this week (and did a talk at Rust Sydney).

What you need is differential, property testing. I’m sure it would work for you (you can skip the first half as you already have the source):

https://reorchestrate.com/posts/bringing-a-warhammer-to-a-kn...
seddonm1
·hace 5 meses·discuss
What about s3 stored in SQLite? https://github.com/seddonm1/s3ite

This was written to store many thousands of images for machine learning
seddonm1
·hace 5 meses·discuss
A process for using LLMs to do brute-force decompilation of binaries and conversion to another programming language - including testing to prove equality.

This process is targeting an old computer game but there is nothing preventing this being run on any binary.