Exactly, it misses out on explaining how the fixed Huffman table is interpreted to apply symbol and distance codes, or how dynamic tables are derived from the input itself. Sure it's the hardest part, but also the more interesting to visualize. As another commenter pointed out, we are just left with mysterious bit sequences for these codes.
Even if it doesn't use block-based compression, if there isn't a huge range of corrupted bytes, corruption offsets are usually identifiable, as you will quickly end up with invalid length-distance pairs and similar errors. Although, errors might be reported a few bytes after the actual corruption.
I was motivated some years ago to try recovering from these errors [1] when I was handling a DEFLATE compressed JSON file, where there seemed to be a single corrupted byte every dozen or so bytes in the stream. It looked like something you could recover from. If you output decompressed bytes as the stream was parsed, you could clearly see a prefix of the original JSON being recovered up to the first corruption.
In that case the decompressed payload was plaintext, but even with a binary format, something like kaitai-struct might give you an invalid offset to work from.
For these localized corruptions, it's possible to just bruteforce one or two bytes along this range, and reliably fix the DEFLATE stream. Not really doable once we are talking about a sequence of four or more corrupted bytes.
Can you give some examples? When batching data, you benefit from picking something like io_uring. But for two-way communication, you still need to notify either side when data is ready (maybe you don't want to consume cpu just polling), and it isn't clear to me how those options handle that synchronization faster than pipes.
Nice, I'll give it a closer look. My only concern so far is memory hooking (still needed for hardware registers), which on Java side was called by FilteredMemoryState [1]. In memstate.cc it looks like just the simpler MemoryState is implemented [2], and there's no equivalent to MemoryAccessFilter. But it might not be that complicated to add...
Thanks, but I think I'm going to disappoint you: the demo is using pre-recorded manual inputs, which are then replayed when emulating in Ghidra. The only logic involved is checking when we are at the right instruction to then send the input. I mentioned it briefly in the README but maybe I wasn't very clear, sorry!
From what I've seen, it's usually read at the vblank interrupt.
The input recording has entries in format "<instruction_number> <buttons_bitmask>". If I press a button and it's read from the hardware register after let's say 0x1000 instructions have been stepped, it is stored as "0x1000 0x80", and in the Ghidra emulator script, I only need to count up to 0x1000 instructions before I send that memory write to the other emulator. While the real timings are vastly different, the input will be read after roughly the same number of vblank calls. I say "roughly" because indeed I found a differential on the expected call where it should be read, but it isn't yet clear if that's a logic bug on my side, I'll have to eventually look into it again.
Nice to see another CTF enjoyer :) I've always thought about using Ghidra for vm challenges, but I'm still not sure if it fits the typical timeframe. Although I never used it, something like binja seems more favourable to quick and dirty scripting.
About custom pcodeops, yeah I was really tempted to use them for TLCS-900. For example, instruction `daa` adjusts the execution result of an add or subtract as binary-coded decimal, and the pcode for that is just inglorious (but I'm sure there's worse out there): https://github.com/nevesnunes/ghidra-tlcs900h/blob/5ff4eb851...
It would be cool if we could supply our own Huffman table and see how that affects the stream itself. We might want to put our text right there! https://github.com/nevesnunes/deflate-frolicking?tab=readme-...