Thank you for liking this blog. I agree with your point. Actually, I’ve just recently transitioned from building data infrastructure on the cloud to taking on a high-performance computing role that truly handles massive amounts of data. So, although I’d heard about the benefits of hugepages before, I had never actually reproduced these issues in my own environment. This time, even though I initially suspected the problems were related to hugepages and the TLB, I didn’t write this blog from a seasoned perspective. Instead, I wanted to methodically investigate and eliminate all other possible issues I could think of. (Interestingly, my agent attributed the effectiveness of hugepages to the root causes of these bugs, which piqued my curiosity and drove my deeper exploration.)
Finally, thank you very much for your appreciation, which means a lot to me. Previously, I was working on open-source projects, but now that I’ve changed jobs, I may not have the same amount of energy to contribute to open-source code as before. However, I think blogging might be a new way for me to contribute. I hope I can keep it up.
(My English writing skills are poor, so I wrote in Chinese and used AI to translate it; I hope you don’t mind.)
Thank you for your suggestion. You are absolutely right, XD. In fact, the order of events in this blog post does not match the actual order in which I debugged and analyzed the issue.
After I identified the TLB misses and confirmed that huge pages were effective, I noticed that there were still many suspicious points in the flame graph. Interestingly, my agent attributed the effectiveness of huge pages to those suspicious points, which turned out to be unrelated to the bottleneck at the time. That sparked my curiosity.
The structure of this blog post was mainly chosen to make the story easier to follow, while also covering the various issues I investigated in depth along the way.
In fact, I recently switched from my previous job building data infrastructure on cloud to an HPC-related role, so I am still not very familiar with some of the mature practices and established conclusions in the HPC world.
So thank you very much for your suggestions. I also hope to learn about more and better methods that can help people identify root causes more quickly and accurately in complex scenarios.
Thank you for your reply. This is a long-running service. Without CRC validation, errors caused by partial writes could accumulate over time and affect correctness. Therefore, we adopted this approach.
Author here. This post is a write-up of a performance-debugging rabbit hole I hit while trying to saturate NICs with NVMe reads using io_uring and RDMA.
The short version: READ_FIXED fixed the obvious per-I/O GUP overhead in a small demo, but the larger deployment still got stuck at roughly half of line rate. After ruling out io-wq backlog, request splitting, fd lookup, and CRC arithmetic, the actual wall turned out to be dTLB misses from scanning 1,028 KiB buffers backed by 4 KiB pages. Moving the read arena to hugepages brought the system close to NIC saturation.
The funny part is that an AI agent suggested hugepages early and got the optimization right, but its explanation was wrong. This post is mostly about reconstructing the evidence for why it worked.
I’d be very interested in feedback from people who have used AI to debug performance issues in a complex system.
For those who may not know, over the past few months, I've been continuously developing and maintaining a hybrid cache library in Rust, Foyer[0].
However, from the very beginning until now, I haven’t had the chance to properly introduce this project through an article. On one hand, I’m not very skilled at writing in English, so I kept procrastinating. (Thanks to ChatGPT, I can focus more on the content rather than worrying about my limited English writing skill.) On the other hand, Foyer was evolving rapidly in its early stages, undergoing major refactoring almost every few months. As a result, it felt premature to formally introduce the project.
But now, Foyer has already gained more stars on Github than than the project it was originally inspired by — Facebook’s CacheLib[1]. This excites me and makes me worry even more about the future of Foyer. So, I think it's time to write a proper blog to introduce it.
Hi, foyer's author here. The "zero-copy in-memory abstraction" is compared to Facebook's CacheLib.
CacheLib requires entries to be copied to the CacheLib managed memory when it's inserted. It simplified some design trade-offs, but may affect the overall throughput when in-memory cache is involved more than nvm cache. FYI: https://cachelib.org/docs/Cache_Library_User_Guides/Write_da...
Foyer only requries entries to be serialized/deserialized when writing/reading from disk. The in-memory cache doesn't force a deep memory copy.
Hi, foyer's author here. Page cache and swap are indeed good general proposed strategies and continuously evoluting. There are several reasons why foyer manages memory and disk control by itself rather than directly utilizing these mechanisms:
1. Leverage asynchronous capabilities: Foyer exposes async interfaces so that while waiting for IO and other operations, the worker can still perform other tasks, thereby increasing overall throughput. If swap is used, a page fault will cause synchronous waiting, blocking the worker thread and resulting in performance degradation.
2. Fine-grained control: As a dedicated cache system, foyer has a better understanding than a general proposed system like the operating system's page cache of which data should be cached and which should not. This is also why foyer has supported direct I/O to avoid duplication of abilities with the page cache since day one. Foyer can use its own strategies to know earlier when data should be cached or evicted.