Deterministic multithreading is hard (2024)(factorio.com)
factorio.com
Deterministic multithreading is hard (2024)
https://www.factorio.com/blog/post/fff-415
13 comments
Yep FFF is such a treat to read. They do an amazing job of explaining complex problems and their solution in a way where anyone can understand.
I used to say the same about the Eve Online dev blog, but at some point I stopped reading because they were creating an ad hoc, informally-specified, bug-ridden, slow implementation of half of Erlang.
Concurrency is hard. Blizzard added progressively more and more concurrency over time to rescue orphaned resources assigned to a single shard that was undersubscribed while another shard in the same AZ was seeing flash mobs. But the way they documented it was more of a tea leaves situation. Only enough data to guess what they had done if you were familiar with the space.
Concurrency is hard. Blizzard added progressively more and more concurrency over time to rescue orphaned resources assigned to a single shard that was undersubscribed while another shard in the same AZ was seeing flash mobs. But the way they documented it was more of a tea leaves situation. Only enough data to guess what they had done if you were familiar with the space.
One of the most interesting things to me when reading this was that it was treated as a bug even though it was that hard to reproduce. Most dev shops would not have the bandwidth and management but in to spend the time to dig into something like that unless it was high severity, and also it sounds like it was also getting caused from a modded version of the software
My favorite paper on it doing a clever workaround:
https://github.com/emeryberger/dthreads
https://github.com/emeryberger/dthreads
What’s interesting to me about Prof Berger’s work is that while many of his ideas are seemingly powerful and should be very impactful, it’s underutilized in practice. I think that’s because he’s failed to figure out how to really make it easy to use and productized. For example, Coz should be the definitive tool and mechanism people use to do performance analysis. However because it’s so hard to use and fails in various ways and is barely maintained if at all it ends up seeing limited use.
It’s a shame. The real world of development would be significantly richer if these ideas had better funding and dedicated long term development.
It’s a shame. The real world of development would be significantly richer if these ideas had better funding and dedicated long term development.
Good points. Thanks for the reference to Coz.
One of the researchers behind either rump kernels or unikernels talked to us here about making it usable. He said he was discouraged by his advisors from doing that. He could write more papers instead.
The reason they think that was is it's quantity over quality in much of academia, esp citation scores and funding for new research. Some groups seem to have their researchers use some of their time to build useful software. Most isn't production quality or maintained because they're basically paid not to do that. It's why many don't join academic research and others supported funding cuts or reform.
One of the researchers behind either rump kernels or unikernels talked to us here about making it usable. He said he was discouraged by his advisors from doing that. He could write more papers instead.
The reason they think that was is it's quantity over quality in much of academia, esp citation scores and funding for new research. Some groups seem to have their researchers use some of their time to build useful software. Most isn't production quality or maintained because they're basically paid not to do that. It's why many don't join academic research and others supported funding cuts or reform.
Paper link (2011): https://people.cs.umass.edu/~emery/pubs/dthreads-sosp11.pdf
> DTHREADS works by exploding multithreaded applications into multiple processes, with private, copy-on-write mappings to shared memory... Experimental results show that DTHREADS substantially outperforms a state-of-the-art deterministic runtime system, and for a majority of the benchmarks evaluated here, matches and occasionally exceeds the performance of pthreads.
> DTHREADS works by exploding multithreaded applications into multiple processes, with private, copy-on-write mappings to shared memory... Experimental results show that DTHREADS substantially outperforms a state-of-the-art deterministic runtime system, and for a majority of the benchmarks evaluated here, matches and occasionally exceeds the performance of pthreads.
Never heard of this, I’m really interested in digging into this paper. Thank you both for the tip!
Note that this wouldn't have been useful for Factorio, because Factorio deals with the harder problem of needing deterministic results with varying amount of parallelism, whereas from what I understand Dthread only give the same results if you run the program with the same number of "threads".
Emery Berger is one of my favorite researchers, his work has a fun bright quality that delights!
I agree. I skimmed so many of his team's papers one night. I could've submitted a bunch of them.
Almost impossible
after HN, it's one of my favourite places on the internet because i constantly learn new, random, insane things almost every time. imho it teaches you how to think + shows you what great engineering taste looks like. sorry if i'm overly effusive but each post is so deeply technical and well-written that i can't believe it's free.
you don't need to know anything about factorio or gamedev btw (i don't), just pick a random number between 1 to 438 and start reading :)