It depends how much the code uses manually spawned goroutines, and how complex the lifecycle of these goroutines is… in big codebases such as kubernetes, docker, etc, it has been a problem. There have been research papers and blogs about this, but most Go developers are not aware of this issue it seems.
I am not an expert in incident reaction, but I thought the safe way was to image the affected machine, turn it off, take a clean machine, boot a clean OS image with the affected image mounted read only in a VM, and do the investigation like that ?
Assume that the malware has replaced system commands, possibly used a kernel vulnerability to lie to you to hide its presence, so do not do anything in the infected system directly ?
Nice article, thank you.
Did you also consider using bpftrace while debugging?
I do not have much experience with it, but I think you can see the kernel call stack with it and I know you can also see the return value (in eax).
That would be less effort than qemu + gdb + disabling kernel aslr, etc.
Author here, thanks for the feedback on legibility, I have now just learned about the CSS `tab-size` property to control how much space tabs get rendered with. I have reduced it, should be better now.
I wrote a toy Kotlin compiler, for fun.
Then one day a Jetbrains employee opens an issue which only says: “Why? Just why?”.
Maybe it’s the language barrier… but I did not find that particularly polite.
On the other hand I open sourced my blog and received lots of small contributions to fix typos or such which were nice.
I have implemented polling against a cluster of mixed mariadb/mysql databases which do not offer listen/notify. It was a pain in the neck to get right.
- The batch size needs to be adaptative for performance, latency, and recovering smoothly after downtime.
- The polling timeouts, frequency etc the same.
- You need to avoid hysteresis.
- You want to be super careful about not disturbing the main application by placing heavy load on the database or accidentally locking tables/rows
- You likely want multiple distributed workers in case of a network partition to keep handling events
It’s hard to get right especially when the databases at the time did not support SKIP LOCKED.
In retrospect I wish I had listened to the WAL. Much easier.
TBH it was also news to me, I discovered it randomly while browsing vulnerabilities…
Printf also allocates, and a ton of other stdlib functions as well.