> should SLL decompile to x << y or x << (y % 32)?
I think this a bit of a misguided question. The hardware has a precise semantic defined, usually. QEMU's << behaves similarly to C (undefined behavior for rhs > 32), but this means that the lifter (still QEMU) will account for this and emit code preserving the semantics.
tl;dr: the code we emit should do the right thing depending on what the original instruction did, without making assumptions on what happens in case of C undefined behaviors.
> Ghidra's type system lacks function pointer types
Weird limitation, we support those.
> it doesn't seem to understand stack slot reuse
That's a tricky one. We're now re-designing certain parts of the pipeline to enable LLVM to promote stack accesses to SSA values, which basically solves the stack slot reuse.
This is probably one of the most important features experienced reversers ask for.
> that language need not be C--
Making up your own language is temptation one should resist.
Anyway, we're rewriting our backend using an MLIR dialect (we call it clift) which targets C but should be good enough to emit something "similar to C but slightly different". It might make sense to have a different backend there. But a "standard C" backend has to be the first use case.
We thought about emitting C++, it would make our life simpler. But I think targeting non-C as the first and foremost backend would be a mistake.
Also, a Python backend would be cool.
> Analysis necessarily involves...
I would be interested in discussing more what exactly you mean here. Why don't you join our discord server?
> I'd absolutely love to be able to import C++ header files to fill in known structure types
We have a project for importing from header files. Basically we want use a compiler to turn them into DWARF debug symbols and then import those.
Not too hard.
1. Not rebasing our fork of QEMU for years has put us in a bad spot. But just today a member of our team managed to lift stuff with the latest QEMU. And he has also been able to lift Qualcomm Hexagon code, for which we helped to add support in QEMU. Eventually we'll be the first proper Hexagon decompiler :)
2. Focusing too much on QEMU led our frontend to be tightly coupled with QEMU. It will now take some effort to enable support for additional frontends, non-QEMU based. But not impossible: our idea is to let user add support for a new architecture by defining, in C, a struct for the CPU state and a bunch of functions acting on it. That's it. No need to learn any internal representation.
tl;dr QEMU was a great choice, it worked so well that we didn't work on that part of the codebase for too much time and now there's some technical debt there. But we're addressing it.
I 120% agree with what you're saying, but emitting valid C is kinda part of what you're asking, in design terms.
Our goal is: omit all the casts that can be omitted without changing the semantics according to C. In fact, we have a PR doing exactly this (still on the old repo, hopefully it will go in soon).
But, how can you expect to be able to be strict with what C allows you to do implicitly, if you're not even emitting valid C? For instance, thanks to the fact that we emit valid C, we could test if the assembly emitted by a compiler is the same before and after removing redundant casts.
My point is that emitting valid C is kind of a prerequisite for what you're asking, a rather low bar to pass, but that, in practice, no mainstream decompiler passes. It's pretty obvious the decompiled code will often be redundant and outright wrong if you don't even guarantee it's syntactically valid. Then clearly it's not a panacea, but it's an important design criterion and shows the direction we want to go.
As for comments: we still haven't implemented inline comments, but they will be attached to program addresses, so they will be available both in disassembly and decompiled C. It's not very hard to do, but that needs some love.
> `source ./environment` is what's written on the announcement page at the top of this thread. './revng' doesn't appear anywhere on it.
You're right, but after that there's a link to the docs where we say to use `./revng`. The blog post is for the impatient :) On the long run the docs is what most people will look at.
I don't think we want to support use cases that might break system packages too. If you set LD_LIBRARY_PATH to a directory where you have an LLVM installation, that might break any system program using LLVM too... Why should we try to fix that using `DT_RPATH` (which is a deprecated way of doing things) when system components don't do it?
We might cleanup the environment from LD_LIBRARY_PATH and other stuff, that might be a sensible default, yeah. Also we might have some sanity check printing a warning if weird libraries are pulled in.
But it's hard to take a decision without a specific use case in mind. If you have an example, bring it forward and I'm happy to discuss what should be the right approach there.
In IDA you basically have only detection of stack frame layout (in a quite confusing fashion) and "create struct out of this pointer", which is something you have to do manually and its intraprocedural.
Imagine this being done automatically, across all of the binary. If you pass a pointer to another function the type is correct and you build the type from all the functions using it.
Then obviously the user needs to fix things, but boostrapping can definitely be hugely improved.
I think most of your concerns about messing with the environment are sensible only under the assumption that you actually do `source environment`.
In truth, we suggest to do that only so you use the GCC we distribute for the demo binary. The actual way this is intended to be used is through the `./revng` script. In that way, the environment changes only affect the invocation of `revng`.
We spent a lot of time to have a completely self-contained set of binaries where each ELF refers to its dependencies through relative paths. LD_LIBRARY_PATH is evil.
> Mostly prefixed "HARD_"
Those are just used by our compiler wrappers, I don't think those environment variables collide with anything in practice.
Those are used when linking binaries translated by revng. If you're not interested in end-to-end binary translation, they don't matter.
> it means 'runpath' which is a similar but much less useful construct
We specifically want DT_RUNPATH. DT_RPATH is deprecated and there might an use case for overriding our libraries with LD_LIBRARY_PATH.
> There's a good chance you can completely remove the environment mangling
I think your observations concerning "mangling the environment" are only valid for non-private environment variables.
The following variables are private: RPATH_PLACEHOLDER, HARD_*, REVNG_*. Also, they are all only for binary translation purposes. We could push them down into some smaller-scoped compiler wrappers, but those make sense only if we can get rid of environment entirely, which we can't because we ship Python.
> a combination of setting different flags when building clang
No, the flags also affect the linker and there's some features of our wrappers that cannot simply be burned in. We can push them in more private places, though.
> a lot of failure modes can be removed
> libc++abi and libunwind can and probably should be statically linked into the libc++
We no longer have issues with that, our build system is pretty reliable in that regard. LLVM is just one of the components, these things need to work robustly in general, and they do (with quite some effort).
You seem to be wary of using dynamic linking, we put some effort in it, now it works pretty good and always looks up things in the right place, and without ever hardcoding absolute paths anywhere, nor any install phase that "patches" the binaries. The unpacked directory can be moved wherever you want.
> I am completely out of patience with distributing dynamically linked programs on Linux
You're thinking of some other solution, our solution does not use LD_LIBRARY_PATH and all the binaries reference each other in a robust way using `$ORIGIN`. Try:
But again, doing `source environment` is mostly for demo purposes, in the actual use case, you just do `./revng` and your environment is untouched.
We ship our Python, but you don't have to use it: you're supposed to just do ./revng (or interact over the network in daemon mode).
Our approach is: use whatever tool you like for scripting as long as it can parse our YAML project file, make changes to it, and then invoke `./revng artifact` (or interact with the daemon): https://docs.rev.ng/user-manual/model-tutorial/
Result: we get to use our Python version (the latest) and you get to use whatever language you like.
Then we'll provide on pypi wrappers that help you with that and are compatible with large set of Python versions.
tl;dr Don't `source ./environment`, use `./revng`.
> Thanks again for shipping, and I hope some of the above feedback is helpful!
I'm happy there's someone that cares about this :D
Our next big iteration of this might involve simplifying things a lot by adopting nix + mount namespace to make /nix/store available without root.
Maybe this is not the right place for discussing this, we can chat on our discord server if you'd like :)
Languages with a rich standard library and generating a lot of code for you usually need some love to get rid/represent idiomatically common patterns and to detect common data structures.
We haven't looked into it yet, but the automatic data structure recognition might help.
Sure, in those cases we emit calls to C functions. The only thing we need to know is what registers are taken as input, what registers are output and what registers are preserved.
In QEMU parlance, these are helper functions, and they have actual implementations. But for decompilation purposes, you don't need to implement them. You just need to know how they interact with the registers.
Our view is: the engine is 100% open source. The UI is available for free in the cloud for anyone experimenting, which we define as "I'm OK with leaving the project public".
Basically, the decompiler engine is Free Software, extensible and available for automation/scripting, while the UI is available for free for students/researchers and we can make a living out of professionals (i.e., when your company is paying for it).
Short answer: if you want to execute a program (maybe with some instrumentation, for fuzzing purposes) it's much easier to adopt a dynamic approach (i.e., emulation or virtualization). With static binary translation you can get better performance, but there's a lot of other things you need to get 100% right and that with a dynamic approach are a given (e.g., the CFG).
There's much more space of improvement in the field of analyzing code (as opposed to running it), so we're investing our energies there.
Then we're strong believers in integrating dynamic and static information, for instance see PageBuster: https://rev.ng/blog/pagebuster
But other than that, static binary translation is a feature of rev.ng in maintenance mode.
That's our goal. We used to use QtCreator as a basis for the UI, terrible idea.
Then we switched to VSCode, which happens to be able to run in the browser. So we added some magic kubernetes sauce and voilà, you got the cloud decompiler with exactly the same user experience as the fully standalone one.
We still need to perform some QA on collaboration, but basically works.
One daemon, many clients. Very simple architecture.
I think we got inspiration to do this from a CTF where we were doing "collaboration" using IDA with multiple windows on a X session on a server with multiple cursors. Very cursed, but effective.
> Decompilation is often the least important (and least reliable) part of IDA/Ghidra
This is something all people using decompilers say and sort of shows how low is trust towards decompilers. Expectations have always been rather low.
I've been there, but this does not have to be the case, the whole reason why we started rev.ng is to prove that expectations can be raised.
Apart from accuracy, which is difficult but engineering work, why don't decompilers emit syntactically valid C? Have you ever tried to re-compile code from any decompiler? It's a terrible experience.
rev.ng only emits valid C code, and we test it with a bunch of -Wall -Wextra:
Other key topic: data structures. When reversing I spend half of the time renaming things and half of the time detecting data structures. The help I get from decompilers in latter is basically none.
rev.ng, by default, detects data structures on the whole binary, interprocedurally, including arrays. See the linked list example in the blog post. We also have plans to detect enums and other stuff.
Clearly we're not there yet, we still need to work on robustness, but our goal is to increase the confidence in decompilers and actually offer features that save time. Certain tools have made progress in improving the UI and the scripting experience, but there's other things to do beyond that.
I see this a bit like the transition from the phase in which C developers where using macros to ensure things were being inlined/unrolled to the phase where they stopped doing that because compilers got smart enough to the right thing and to do it much more effectively.
For scripting, our approach is to give you access to the project file (just a YAML file), and you can make changes from any scripting language you want. Everything the user can customize is in there, all the rest is deterministically produced from that file.
I really disliked the fact that you usually need to buy into the version of Python that $TOOL requires you to use, or the fact itself that you need to use a specific language.
For xrefs, CFG and the rest: we have all of that in the UI, but we also produce them in a rich way. For instance, when we emit disassembly and decompiled code, we actually emit plain text + HTML-like markup to provide metainformation for navigation (basically, xrefs) and highlighting. So you can use all that from any language that can parse HTML/XML.
It's called PTML: https://docs.rev.ng/references/ptml/
For lifting: we use LLVM IR as our internal representation. This means that: 1) you don't have to learn an IR that no one else uses, 2) you can use off the shelf tools (e.g., KLEE for symbolic execution) but you can also use all the standard LLVM optimizations and analyses and 3) you can recompile it, but we're not into the binary translation business anymore.
In the past we were thinking to do something like this by hand. For instance, we detect induction variables, we could rename them into `i`.
However, nowadays, it seems pretty obvious that the right way to do this things is using LLMs.
This said, at this stage, we see ourselves as people building robust infrastructure. Once the infrastructure is there, using some off the shelf model to rename things or add comments is relatively easy.
Basically: we do the hard decompilation work that needs 100% accuracy, and then we can adopt LLMs for things that are OK to be approximate such as names, comments and the like.
About the book, here's the full story: I was getting into compilers, but I was really struggling with the theory, the most famous books weren't doing it for me, and I felt really down.
Then I find this book, which seems very dense, but clear. So I ask my advisor if I could buy it and goes like "well, first check out the university library".
I check it out and there's a copy, but... it's taken.
Working in the only group that was doing research on compilers I'm like "who dares do compilers stuff out of our group!?".
I was writing a response to the paper (which compares with rev.ng), but never managed to complete it.
I wanted to make a thread on Twitter about it, but it was getting longer and longer, more like a blog post.
If you're really interested, drop me an e-mail and I'll forward it to you.
The bottom line is more or less summarized in my other response in this thread.
Roadmap item: https://rev.ng/roadmap#feature-798
Design pad: https://pad.rev.ng/s/eDHi2PUoP#