Why MMAP in llama.cpp hides true memory usage(twitter.com)
twitter.com
Why MMAP in llama.cpp hides true memory usage
https://twitter.com/jeremyphoward/status/1642726595436883969
102 comments
I'm not convinced that there isn't more to the story than that. People (including Justine who implemented MMAPing the model and knows how to monitor memory use properly) are seeing that much of the file isn't being paged into memory, and are not quite sure why[1]. Is it a bug? Is the distribution of weights used highly dependent on the prompts? Are some weights somehow unused altogether? This twitter thread rules out some of those, but I don't think it closes the door.
[1] https://news.ycombinator.com/item?id=35393615
edit: softened some of my claims after reading more updates from over the weekend.
[1] https://news.ycombinator.com/item?id=35393615
edit: softened some of my claims after reading more updates from over the weekend.
Now that's interesting. Entire memory pages of the model aren't being referenced?
If correct could this hint at sub models for specific tasks?
eBPF
echo 3 > /proc/sys/vm/drop_caches
echo 3 > /proc/sys/vm/drop_caches
The kernel will read more than just the page that faulted. Jart counts just the page faults and multiplies by the page size.
Other people are saying that the whole file does get paged in and transformers access all their weights by design.
File backed memory is "less" than heap memory, because it can be thrown away when needed instead of being swapped out to disk.
And because reading the file probably allocated file backed pages for it anyway unless you used O_DIRECT (and profiled it first.)
There's also wired/mlock() memory which is "worse" than regular memory because it can't be swapped/evicted at all.
And because reading the file probably allocated file backed pages for it anyway unless you used O_DIRECT (and profiled it first.)
There's also wired/mlock() memory which is "worse" than regular memory because it can't be swapped/evicted at all.
Memory mapped files also known as sections in VMS/NT have just two advantages:
* Fewer context switches among user space / kernel syscalls
* No need to copy _modified_ data into the swap. The behavior for read only data doesn't change
No, it also shares them between multiple runs of the same process, and it reuses the pages that were in your file cache anyway.
> * Fewer context switches among user space / kernel syscalls
This is not necessarily true. There's lots of cases where it's actually slower.
> * Fewer context switches among user space / kernel syscalls
This is not necessarily true. There's lots of cases where it's actually slower.
> Fewer context switches among user space / kernel syscalls
Note that each page fault to read an mmaped page is also a context switch from user space into the kernel. The kernel entry/exit paths for syscalls and page faults have much in common.
Mmaped pages can in principle use fault-ahead to reduce the number of page faults when sequential access is detected. An equivalent reduction is available for read syscalls by reading larger blocks at a time.
In practice, mmap files are faster in some scenarios compared with read syscalls and slower in others. Same with O_DIRECT reads, those are faster in some scenarios and slower in others.
Note that each page fault to read an mmaped page is also a context switch from user space into the kernel. The kernel entry/exit paths for syscalls and page faults have much in common.
Mmaped pages can in principle use fault-ahead to reduce the number of page faults when sequential access is detected. An equivalent reduction is available for read syscalls by reading larger blocks at a time.
In practice, mmap files are faster in some scenarios compared with read syscalls and slower in others. Same with O_DIRECT reads, those are faster in some scenarios and slower in others.
I mean, if we're being pedantic, I think it is possible to run llama.cpp with real memory savings this way... in a contrived situation where you were only generating a single token with a single forward pass, and you were limited in RAM.
The new MMAP system would not require you to load the whole model from disk up front, but rather would allow you to load and immediately forget each layer of parameters as you do a forward pass through the GPT architecture. And if you had just tried to use normal swap paging to do this without MMAP, you'd essentially be doing 2x the loading work, since the page for the first layer likely got swapped out as you loaded the rest of the model into "memory."
Of course, as soon as you want to generate a new token, you'd need to reload all the pages of the model parameters. Every token generated would take as long as it would if you were loading the model from scratch.
But for a classification problem where you only need to generate one token, and you need or want to do this in a serverless environment where you'd need to load from disk anyways? I think you'd be able to do that workflow with significantly less RAM in the same amount of clock time, now.
The new MMAP system would not require you to load the whole model from disk up front, but rather would allow you to load and immediately forget each layer of parameters as you do a forward pass through the GPT architecture. And if you had just tried to use normal swap paging to do this without MMAP, you'd essentially be doing 2x the loading work, since the page for the first layer likely got swapped out as you loaded the rest of the model into "memory."
Of course, as soon as you want to generate a new token, you'd need to reload all the pages of the model parameters. Every token generated would take as long as it would if you were loading the model from scratch.
But for a classification problem where you only need to generate one token, and you need or want to do this in a serverless environment where you'd need to load from disk anyways? I think you'd be able to do that workflow with significantly less RAM in the same amount of clock time, now.
Is it possible to do multi-GPU inference the way you describe and have say 4 GPUs but each doing different layers on a large model?
If that is the case then perhaps a high end consumer system with max pcie 5 nvme bandwidth and multi gpus could do inference on large LLM models.
If that is the case then perhaps a high end consumer system with max pcie 5 nvme bandwidth and multi gpus could do inference on large LLM models.
Congratulations on rediscovering how the original researchers run these models!
Do they do it in a way that each GPU can have smaller memory than the model weights?
I keep seeing references to these absolutely massive 48GB+ gpu memory sizes and those are extremely expensive. If more but smaller memory gpus can work that could open up a much larger range of hardware and individuals to run inference, especially if efficient load/unload of weight layers from raid pcie5 nvme devices is feasible.
I keep seeing references to these absolutely massive 48GB+ gpu memory sizes and those are extremely expensive. If more but smaller memory gpus can work that could open up a much larger range of hardware and individuals to run inference, especially if efficient load/unload of weight layers from raid pcie5 nvme devices is feasible.
Yes?
The original LLaMA-65B on fp16 weight is 130G. There are no NVIDIA GPU with 130G memory.
As for why they don't use int8/int4: would you try to optimize your code before you confirmed it can actually run?
The original LLaMA-65B on fp16 weight is 130G. There are no NVIDIA GPU with 130G memory.
As for why they don't use int8/int4: would you try to optimize your code before you confirmed it can actually run?
Got it, so looking ahead to building hardware that is useful for inference would you agree with these considerations:
- max PCI lanes for more GPUs. AMD cpus and boards probably better
- PCIe 5 nvme. ideally enough to saturate bus
- max GPUs with max memory. memory more important than FLOPs probably.
for a given budget one tries to get as much as possible of above.
How long before a consumer component machine for less than $10K can do inference for a gpt 3.5 turbo quality model?
- max PCI lanes for more GPUs. AMD cpus and boards probably better
- PCIe 5 nvme. ideally enough to saturate bus
- max GPUs with max memory. memory more important than FLOPs probably.
for a given budget one tries to get as much as possible of above.
How long before a consumer component machine for less than $10K can do inference for a gpt 3.5 turbo quality model?
PCIe lanes: it's not that important since the data exchange between host and device during inference is pretty minimal. For model parallel / tensor parallel you want fast interconnect between GPUs, the ages old tech for this is PCIe switch, the fashion ones are NVLink, neither are consumer hardware so don't bother.
PCIe 5 NVMe: That's completely irrelevant for inference, as you really want to keep all weights on the GPU VRAM. It only speed up the initial weight load from host to device.
max GPUs with max memory: Yeah. For personal use (batch size = 1) VRAM size AND VRAM bandwidth is more important than FLOPs. See https://orenleung.com/is-chatgpt-175-billion-parameters-tech.... For production it's a tradeoff.
> How long before a consumer component machine for less than $10K can do inference for a gpt 3.5 turbo quality model?
$10k is a lot. You can run LLaMA-65B on 2x4090, won't be fast enough for production though. But given that people happily run LLaMA-13B on their MacBooks I guess we are okay with generation being slow. And personally I don't think gpt-3.5-turbo is that big, it should be around or even smaller than LLaMA-65B.
PCIe 5 NVMe: That's completely irrelevant for inference, as you really want to keep all weights on the GPU VRAM. It only speed up the initial weight load from host to device.
max GPUs with max memory: Yeah. For personal use (batch size = 1) VRAM size AND VRAM bandwidth is more important than FLOPs. See https://orenleung.com/is-chatgpt-175-billion-parameters-tech.... For production it's a tradeoff.
> How long before a consumer component machine for less than $10K can do inference for a gpt 3.5 turbo quality model?
$10k is a lot. You can run LLaMA-65B on 2x4090, won't be fast enough for production though. But given that people happily run LLaMA-13B on their MacBooks I guess we are okay with generation being slow. And personally I don't think gpt-3.5-turbo is that big, it should be around or even smaller than LLaMA-65B.
>How long before a consumer component machine for less than $10K can do inference for a gpt 3.5 turbo quality model?
Uh two months ago? LLaMA-13B is GPT 3.5 Turbo quality and runs locally on my Android phone's CPU at acceptable speeds.
You can even run LLaMA-65B (which far surpasses GPT 3.5) faster than GPT 3.5 Turbo with two $200 24GB Nvidia Tesla P40 cards, since in 4bit the model is only 39GB with no output quality loss. Total cost $400 plus some junker used PC with two spare PCIe 4 x16 lanes.
Uh two months ago? LLaMA-13B is GPT 3.5 Turbo quality and runs locally on my Android phone's CPU at acceptable speeds.
You can even run LLaMA-65B (which far surpasses GPT 3.5) faster than GPT 3.5 Turbo with two $200 24GB Nvidia Tesla P40 cards, since in 4bit the model is only 39GB with no output quality loss. Total cost $400 plus some junker used PC with two spare PCIe 4 x16 lanes.
Is that really true! Can LLaMA-65B output code as good quality as GPT 3.5 Turbo?
Are P40s only $200!
If your information is accurate that is incredible and great tip. I’ll start on building such a rig asap.
Are P40s only $200!
If your information is accurate that is incredible and great tip. I’ll start on building such a rig asap.
How much are 2x 3090 + the rig to run them? I think that’s enough to run a 4 bit quantized llama 65B
> Every token generated would take as long as it would if you were loading the model from scratch.
Wouldn't make sense to keep most of the model in GPU (as much as it fits) and only load the remaining layers on each pass?
Wouldn't make sense to keep most of the model in GPU (as much as it fits) and only load the remaining layers on each pass?
mmap() basically adds a new swap file that is only to be used for the contents of that specific file.
This means it does page eviction on memory pressure and also results in deduplication as mmap'd memory of the same file can be used by any process that has access to it.
If the data access is sequential in nature (or at least has some form of locality in function of time) it could end up using less memory indeed, even if the whole file ends up being read during the training process, because unused pages will be evicted by the OS to make room for needed ones.
What is interesting to me here is that it's such a marvel to these ML guys. It's a fundamental OS facility and looks like using mmap() instead of using read() on malloc()'d memory should have been the strategy here from the start.
This means it does page eviction on memory pressure and also results in deduplication as mmap'd memory of the same file can be used by any process that has access to it.
If the data access is sequential in nature (or at least has some form of locality in function of time) it could end up using less memory indeed, even if the whole file ends up being read during the training process, because unused pages will be evicted by the OS to make room for needed ones.
What is interesting to me here is that it's such a marvel to these ML guys. It's a fundamental OS facility and looks like using mmap() instead of using read() on malloc()'d memory should have been the strategy here from the start.
Saying it's like "adding a new swap file that is only to be used for the contents of that specific file" isn't really correct.
Swap works by letting the kernel write a memory page to the swap file/partition. After it has done that, the memory is marked as backed by a filesystem (i.e it's no longer anonymous), and the region of physical RAM can be repurposed for something else. With mmap'd files, the memory is always backed by a filesystem; there are no writes, and the physical memory can be discarded at any time.
Swap works by letting the kernel write a memory page to the swap file/partition. After it has done that, the memory is marked as backed by a filesystem (i.e it's no longer anonymous), and the region of physical RAM can be repurposed for something else. With mmap'd files, the memory is always backed by a filesystem; there are no writes, and the physical memory can be discarded at any time.
This isn't why mmap in llama.cpp hides true memory usage, it is why mmap hides true memory usage.
What does mmap do exactly? Why was the transition to using it a big improvement in llama.cpp?
For example, you create a 100MB file. You tell the operating system to map that file to memory, and it gives you a pointer to 100MB of memory. Whatever you read from that 100MB of memory is what is actually in the file. You can also write to that memory and commit it back to the file. The parts you read from the pointer are the only parts from the file that are loaded into memory. So if you memory map a 100GB file, the operating system won't actually load all 100GB into memory, only what is accessed (and this is all handled for you automatically).
The operating system is free to load and cache the file into memory in whatever way it wants, so for large memory mapped files it'll often try to use all available memory to cache as much as possible. If another program needs memory, the operating system will simply lower the amount of memory available to the memory mapped file for caching. This is extremely useful for databases, since it greatly simplifies both how to persist the data along with how to load and cache the persisted data.
This all comes with a big caveat however. The less memory you have, the more file accesses occur (similar to your pagefile when you're thrashing), which can dramatically slow down your memory operations.
tldr; it lets you designate a file to use as a region of memory.
The operating system is free to load and cache the file into memory in whatever way it wants, so for large memory mapped files it'll often try to use all available memory to cache as much as possible. If another program needs memory, the operating system will simply lower the amount of memory available to the memory mapped file for caching. This is extremely useful for databases, since it greatly simplifies both how to persist the data along with how to load and cache the persisted data.
This all comes with a big caveat however. The less memory you have, the more file accesses occur (similar to your pagefile when you're thrashing), which can dramatically slow down your memory operations.
tldr; it lets you designate a file to use as a region of memory.
so basically an MRU cache for a file's contents? Is there enough information to know what bytes from the file are cache'd and which are not? It would interesting to see like a heatmap of a large model showing what portions of the neural network are being used the most. ..like a brain MRI more or less.
The kernel page cache already exists, and is used on read() too. The distinction is that instead of read() copying from the kernel page cache, mmap will map that page cache page directly into the user's memory address space.
It saves a copy on read, in exchange for a bit more overhead configuring the processes address space up front.
It saves a copy on read, in exchange for a bit more overhead configuring the processes address space up front.
Quoting another user, jcranmer [1]:
> "The fundamental operation of mmap is to add new entries to the page table of a process, and the precise properties of those entries are heavily dependent on what the arguments to mmap are.
> When you mmap a regular file, you're essentially adding an entry to the page table that shares the data with the kernel's filesystem cache."
Now, understanding this requires some understanding of an OS's virtual memory function, and what "page tables" are. Those are what the OS use to track a process' memory, whose granularity is in "pages" (historically 4kB on Linux, though others use larger granularity such as 16kB).
mmap() has a a lot of flags [2] that affect the properties of those mapped pages. It is the swiss-army knife of memory management on Linux.
Some of those properties allow you to share memory with other processes (MAP_SHARED | MAP_ANONYMOUS), or just allocate memory (MAP_ANONYMOUS) or, by default, map an (open) file specified by the `fd` argument.
(fun fact! On linux, when you malloc(), you don't actually get memory, just an IOU from the kernel. Only when you access that memory, ie accessing those memory pages, does the kernel actually make the effort of allocating you that memory.)
[1] https://news.ycombinator.com/item?id=35412842
[2] https://linux.die.net/man/2/mmap
> "The fundamental operation of mmap is to add new entries to the page table of a process, and the precise properties of those entries are heavily dependent on what the arguments to mmap are.
> When you mmap a regular file, you're essentially adding an entry to the page table that shares the data with the kernel's filesystem cache."
Now, understanding this requires some understanding of an OS's virtual memory function, and what "page tables" are. Those are what the OS use to track a process' memory, whose granularity is in "pages" (historically 4kB on Linux, though others use larger granularity such as 16kB).
mmap() has a a lot of flags [2] that affect the properties of those mapped pages. It is the swiss-army knife of memory management on Linux.
Some of those properties allow you to share memory with other processes (MAP_SHARED | MAP_ANONYMOUS), or just allocate memory (MAP_ANONYMOUS) or, by default, map an (open) file specified by the `fd` argument.
(fun fact! On linux, when you malloc(), you don't actually get memory, just an IOU from the kernel. Only when you access that memory, ie accessing those memory pages, does the kernel actually make the effort of allocating you that memory.)
[1] https://news.ycombinator.com/item?id=35412842
[2] https://linux.die.net/man/2/mmap
One thing I don't understand is that if I read a gigabyte from a file with the read call, why the kernel can't see that it's unused allocated memory, create copy on write pages, which could just be shared with the disk cache as long as the data is aligned (which it should be after a malloc + read call).
It seems the intuitive way to implement read if there's a complex memory subsystem that does all these great things anyways.
It seems the intuitive way to implement read if there's a complex memory subsystem that does all these great things anyways.
> if I read a gigabyte from a file with the read call
`mmap` signals the kernel that you want to use the data someday. You `mmap` it, it automatically handles fetching and throwing unused bytes away.
`read` signals the kernel that you want to do something with the data now.
You `read` it, you need it in the memory now.
> why the kernel can't see that it's unused allocated memory
Only a in-process garbage collector would really know what part of your memory is unused. The kernel can't really know for sure.
`mmap` signals the kernel that you want to use the data someday. You `mmap` it, it automatically handles fetching and throwing unused bytes away.
`read` signals the kernel that you want to do something with the data now.
You `read` it, you need it in the memory now.
> why the kernel can't see that it's unused allocated memory
Only a in-process garbage collector would really know what part of your memory is unused. The kernel can't really know for sure.
malloc() and the kernel would need to conspire to know this before you call read, when malloc() gives you back the memory in the first place
I guess you haven't read my parent: malloc doesn't give you memory, just reserves pages to be used in the future.
malloc(3) does not know or care about whether the OS actually commits the VM allocation that it got from somewhere (be it mmap(2) or sbrk(2)), apart from the fact that most implementations of malloc would write some kind of book-keeping header before the returned object (observe that on amd64 glibc, malloc(3) for large allocations returns pointers that are offset +0x10 from page alignment, that is where the two word allocated block header lives). The fact that the memory might not really exist is mostly invisible to userspace (as in, there is no portable way to find out that it does not exist).
The OS somehow detecting that argument to read(2) is page aligned block of uncommited meory and transparently creating MAP_PRIVATE mapping from that seems like nifty idea, but userspace tends to not have page-aligned buffers that often (see the aforementioned 0x10 offset for one reason why) and applications that intentionally allocates IO buffers with some particular alignment do so in order to bypass the block cache for particular IO patterns and are exactly the kinds of applications that tend to use mmap(2) for IO where that does not matter.
The OS somehow detecting that argument to read(2) is page aligned block of uncommited meory and transparently creating MAP_PRIVATE mapping from that seems like nifty idea, but userspace tends to not have page-aligned buffers that often (see the aforementioned 0x10 offset for one reason why) and applications that intentionally allocates IO buffers with some particular alignment do so in order to bypass the block cache for particular IO patterns and are exactly the kinds of applications that tend to use mmap(2) for IO where that does not matter.
> most implementations of malloc would write some kind of book-keeping header before the returned object (observe that on amd64 glibc, malloc(3) for large allocations returns pointers that are offset +0x10 from page alignment, that is where the two word allocated block header lives).
Hmm. You're right about glibc, but macOS for instance reliably returns page-aligned pointers for large allocations. I wonder how Windows behaves.
Hmm. You're right about glibc, but macOS for instance reliably returns page-aligned pointers for large allocations. I wonder how Windows behaves.
> it's unused allocated memory
Well, first of all, it's not: malloc may have previously stored its internal structures in that memory so it may be dirty. Second, AFAIK if you ask malloc() for a gigabyte, it'll internally call mmap() on an anonymous file and will give you its result.
Well, first of all, it's not: malloc may have previously stored its internal structures in that memory so it may be dirty. Second, AFAIK if you ask malloc() for a gigabyte, it'll internally call mmap() on an anonymous file and will give you its result.
I see, that's how it's implemented.
But from what I understand there may be nothing in POSIX forbidding the kernel to do this optimization, just kernel authors decided against it (which is understandable).
But from what I understand there may be nothing in POSIX forbidding the kernel to do this optimization, just kernel authors decided against it (which is understandable).
That's right. Indeed, if the optimization is implemented properly it should have no observable behavioral difference except for performance (better in some cases, worse in others) which is not really something POSIX concerns itself with. So POSIX doesn't forbid it.
I can think of one edge case not mentioned by others, though.
One of the main benefits of mmap is that the data can be paged out while not used. But when you try to page the data back in, the attempt may fail, or it may return different data than was originally there. This is particularly likely with network filesystems or external storage. With mmap, these situations result in, respectively, the process receiving SIGBUS, or the contents of memory changing out from under it. The former is usually a crash (unless the program installs a signal handler), and the latter can be worse than a crash, if either program logic or compiler optimizations make assumptions about memory staying consistent when not written to. A program using mmap is opting in to that risk. A program using malloc and read is not.
That said, it may be reasonable to trust that the system’s main disk(s) (however you define that) won’t misbehave this way, and limit the optimization to files mapped from them. After all, if swap is enabled, the same potential risks exist for any memory that’s swapped out. Even without swap, if the root partition starts failing, the system is not going to stay up for long.
Or, the optimization could be enabled for other disks, but with the caveat that the pages would be swapped to the swap file/partition rather than paged out normally.
I can think of one edge case not mentioned by others, though.
One of the main benefits of mmap is that the data can be paged out while not used. But when you try to page the data back in, the attempt may fail, or it may return different data than was originally there. This is particularly likely with network filesystems or external storage. With mmap, these situations result in, respectively, the process receiving SIGBUS, or the contents of memory changing out from under it. The former is usually a crash (unless the program installs a signal handler), and the latter can be worse than a crash, if either program logic or compiler optimizations make assumptions about memory staying consistent when not written to. A program using mmap is opting in to that risk. A program using malloc and read is not.
That said, it may be reasonable to trust that the system’s main disk(s) (however you define that) won’t misbehave this way, and limit the optimization to files mapped from them. After all, if swap is enabled, the same potential risks exist for any memory that’s swapped out. Even without swap, if the root partition starts failing, the system is not going to stay up for long.
Or, the optimization could be enabled for other disks, but with the caveat that the pages would be swapped to the swap file/partition rather than paged out normally.
I see, the main disks should be probably defined when mounting (allow memory mapping), but it needs a lot of kernel work to know whether it all is worth it or not.
Still, the current problem with mmap is that it's buggy on some operating systems according to https://www.sqlite.org/mmap.html, so I guess it will cause more frustrations until AGI takes over the world :)
Still, the current problem with mmap is that it's buggy on some operating systems according to https://www.sqlite.org/mmap.html, so I guess it will cause more frustrations until AGI takes over the world :)
Sounds like an argument for liburing more. That also is supposed to let the kernel share buffers and reduce syscalls, but does come with a mechanism for handling errors without terminating the process
That's what mmap is for :)
When a read call completes, you know you've successfully read the data from the underlying device. If you have swap disabled you know you can read and write from that memory quickly.
With mmap'd data, a memory read/write can trigger a page fault and I/O, and if you encounter an error like a network filesystem not responding in time, the program sees a segfault.
When a read call completes, you know you've successfully read the data from the underlying device. If you have swap disabled you know you can read and write from that memory quickly.
With mmap'd data, a memory read/write can trigger a page fault and I/O, and if you encounter an error like a network filesystem not responding in time, the program sees a segfault.
this is probably the best source to actually understand how memory works in Linux:
https://manybutfinite.com/post/anatomy-of-a-program-in-memor... [1]
You can't really understand mmap/sbrk without understanding virtual memory and process space layout.
[1] Images are broken. Just open https://static.duartes.org/img/blogPosts/kernelUserMemorySpl... and go to advanced and "Proceed to static.duartes.org" to workaround their https issues. The duartes.org host is owned by the blog author. Refresh the blog article and images should load now.
https://manybutfinite.com/post/anatomy-of-a-program-in-memor... [1]
You can't really understand mmap/sbrk without understanding virtual memory and process space layout.
[1] Images are broken. Just open https://static.duartes.org/img/blogPosts/kernelUserMemorySpl... and go to advanced and "Proceed to static.duartes.org" to workaround their https issues. The duartes.org host is owned by the blog author. Refresh the blog article and images should load now.
The transition to mmap offloaded memory management to the kernel, which can lazily load in parts of the file from disk as its memory mapped pages are accessed. The original version eagerly read the files into memory before running inference.
mmap maps files into virtual memory. When a program read the mapped are of memory, that portion of the file is read.
My understanding is that it maps a file directly to memory to reduce disk usage.
Doesn't change disk usage. The file is still on disk.
The difference is between reading a file and memory mapping (mmap'ing) it.
If you read a 1TB file into memory you use 1TB of disk and 1TB of physical memory. If you then access that data it's as fast as RAM because that's where it is.
If you mmap a 1TB file you use 1TB of disk and 1TB of virtual memory. If you then access that data it may be mapped into virtual memory but not actually be in RAM. This triggers a page fault, at which point the correct page is loaded from disk to physical memory, and handed back to you.
The key observation is that the amount of physical memory occupied by the mmap'd file is much smaller than the entirety of the file unless you access almost all of it.
If your filesystem supports holes, this can also be useful for writes: it's possible to map files vastly larger than physical disk space, but so long as the actual number of places written to is quite small you won't run out.
The combination is very useful for datastructures because you basically don't have to care about data being extremely sparse until that data is also getting quite large, which means you can use cheap/fast approaches to indexing, etc.
The difference is between reading a file and memory mapping (mmap'ing) it.
If you read a 1TB file into memory you use 1TB of disk and 1TB of physical memory. If you then access that data it's as fast as RAM because that's where it is.
If you mmap a 1TB file you use 1TB of disk and 1TB of virtual memory. If you then access that data it may be mapped into virtual memory but not actually be in RAM. This triggers a page fault, at which point the correct page is loaded from disk to physical memory, and handed back to you.
The key observation is that the amount of physical memory occupied by the mmap'd file is much smaller than the entirety of the file unless you access almost all of it.
If your filesystem supports holes, this can also be useful for writes: it's possible to map files vastly larger than physical disk space, but so long as the actual number of places written to is quite small you won't run out.
The combination is very useful for datastructures because you basically don't have to care about data being extremely sparse until that data is also getting quite large, which means you can use cheap/fast approaches to indexing, etc.
Best part:
>AI-generated summary:
>"This thread explains how the mmap feature of modern operating systems can be used to reduce memory usage when running large deep learning models. It also explains how...
Yet I wonder, if this was just another of those humans imitating a trustworthy AI...
>"This thread explains how the mmap feature of modern operating systems can be used to reduce memory usage when running large deep learning models. It also explains how...
Yet I wonder, if this was just another of those humans imitating a trustworthy AI...
TL;DR
> The way mmap handles memory makes it a bit tricky for the OS to report on a per-process level how that memory is being used. So it generally will just show it as "cache" use. That's why (AFAICT) there was some initial misunderstanding regarding the memory savings
> The way mmap handles memory makes it a bit tricky for the OS to report on a per-process level how that memory is being used. So it generally will just show it as "cache" use. That's why (AFAICT) there was some initial misunderstanding regarding the memory savings
I lost track since things move so quickly. Was there still memory savings just not as drastic? Or no memory savings, just a speed-up?
It's neither memory savings or a speed-up really. The advantage of mmap is that you can treat a file on a disk a block of memory, so pages from it can be loaded (or unloaded) as necessary instead of one big upfront load into RAM. The benefit is that you can work with data that's bigger than your physical RAM because the kernel can swap it back out to disk if needed. Another benefit could be that if only a small part of the data is needed to compute something then the OS will automatically only load those, but it's unclear to me whether this is the case with LLaMA.
It's still somewhat faster if you benchmark it. I assume the os is doing good enough prefetching in the mmap case to hide the loads from disk mostly. So it's not just hiding the initial load of 30gb from disk.
Obviously if you're swapping because you don't have enough memory to hold the model in RAM, the mmap version is going to be much faster, since you don't need to swap anything out to disk but just discard the page and re-read from disk if you need it again later.
Obviously if you're swapping because you don't have enough memory to hold the model in RAM, the mmap version is going to be much faster, since you don't need to swap anything out to disk but just discard the page and re-read from disk if you need it again later.
> So it's not just hiding the initial load of 30gb from disk.
The issue is typically that that initial load involves some sort of transformation - parsing, instantiating structures, etc. If you can arrange it so that the data is stored in the format you actually need it in memory, then you can skip that entire transformation phase.
I don’t know if that’s what’s been done with llama.cop though.
The issue is typically that that initial load involves some sort of transformation - parsing, instantiating structures, etc. If you can arrange it so that the data is stored in the format you actually need it in memory, then you can skip that entire transformation phase.
I don’t know if that’s what’s been done with llama.cop though.
It is a speedup for me. When I run llama.cpp from CLI first tiem it takes a very long tiem to load the model in memory. If the program exits or I stop it with Ctrl+C and start it again it will start almost instant.
That's down to caching. If you used your system to do something else for a while, you'd find those pages evicted and the performance back down to Earth. That's one of the things that makes mmap so useful, though. The system can take advantage of access patterns to dramatically improve performance.
Yes, makes sense. And is great. Though honestly not sure why it takes minutes to load a 23Gb model in RAM, I feel is not proportional with the smaller models.
Regarding your last point: No, you need all of the weights all of the time.
Edit: except embedding weights but those are not the problem.
Edit: except embedding weights but those are not the problem.
[deleted]
They weren't asking about mmap, they were asking about the program itself.
It's basically paging to disk.
Not necessarily memory savings, but the improvements here are that it will run on computers with less ram because it can page to disk.
Not necessarily the number reported (since you do need to load chunks into ram), but still lower.
Not necessarily memory savings, but the improvements here are that it will run on computers with less ram because it can page to disk.
Not necessarily the number reported (since you do need to load chunks into ram), but still lower.
If that's the case then part of this may be the different interpretations of "memory".
One persons "paging the same memory requirement from disk to RAM" can be someone elses "requiring less memory/RAM".
One persons "paging the same memory requirement from disk to RAM" can be someone elses "requiring less memory/RAM".
Sort of, but without the duplication and initial wait to load. A traditional fat in-memory app would do this:
file (DISK) to process active pages (RAM) to paged out virtual memory if saturated (elsewhere on DISK)
Using mmap typically goes something like:
file (DISK) to process active pages (RAM) to released and cached (RAM) to uncached if evicted (back to same place on DISK) OR back to process active pages from cache (RAM)
For the cost of fixing up some process page tables, the physical memory pages necessary can be brought back from the cache rather than read from disk. It's an orders-of-magnitude performance savings.
file (DISK) to process active pages (RAM) to paged out virtual memory if saturated (elsewhere on DISK)
Using mmap typically goes something like:
file (DISK) to process active pages (RAM) to released and cached (RAM) to uncached if evicted (back to same place on DISK) OR back to process active pages from cache (RAM)
For the cost of fixing up some process page tables, the physical memory pages necessary can be brought back from the cache rather than read from disk. It's an orders-of-magnitude performance savings.
more like memory mis-reporting, since when you mmap a file in, IIRC that counts against page cache rather than memory usage (you can evict the page without causing write IO so the memory isn't "used")
It being safe to evict is why it's correct to not report it as "memory usage".
It's part of the program's working set but measuring that is a completely different story.
It's part of the program's working set but measuring that is a completely different story.
It did somewhat reduce the total memory used. Now you can load the 30B model while only using ~20gb of RAM, which is about the aggregate size of the 4bit quantized weight files for that model. The real win is that you can kill the main inference binary and try another prompt, and it will start doing inference basically immediately instead of spending 10-15 seconds loading up all the weights into RAM each time.
mmap does not hide the real RAM usage, it just maps a part of virtual memory address space to an I/O-bound storage medium (e.g. file). That trick does not consume RAM in any significant amounts, so there is nothing to hide.
However, such image mapping technique is not always suitable for data-intensive workloads due to page thrashing. If it works for llama.cpp then it can be considered a huge success.
However, such image mapping technique is not always suitable for data-intensive workloads due to page thrashing. If it works for llama.cpp then it can be considered a huge success.
Soft page faults considered genius
>mmap is a really nifty feature of modern operating systems
... for some definition of "modern".
... for some definition of "modern".
you dont get the devs who spend their day on twitter instead of writing code to read your thread if it doesnt mention "modern" at least once. /s
Its quite obvious that the author "dumbs down" all the information for a very non-developer audience, so I assume that gives some leniency. If you tell people that we have had things like mmap() for decades, they may start asking too many questions about why every piece of software underperforms so horribly below what was possible decades ago.
Bit of a rant, but I feel that we lose a little bit of potential every time a developer calls operator<< on a std::istream in a loop.
Its quite obvious that the author "dumbs down" all the information for a very non-developer audience, so I assume that gives some leniency. If you tell people that we have had things like mmap() for decades, they may start asking too many questions about why every piece of software underperforms so horribly below what was possible decades ago.
Bit of a rant, but I feel that we lose a little bit of potential every time a developer calls operator<< on a std::istream in a loop.
To be fair I find it completely baffling that R, Pandas, and all sorts of other tools make it difficult or impossible to mmap a column to a dataframe.
Instead you have to do backflips to load data that doesn't fit in RAM.
Instead you have to do backflips to load data that doesn't fit in RAM.
What's wrong with operator<< on std::istream?
Devs of 2023: "why does this wheel have rubber on it? It needs to be more modern. Let's take away the rubber and add firecrackers in its place as a nice feature, so really pops and makes the experience more exciting"
The same can be said of devs of 2010 and devs of 2000. There are constant innovations and trends in technology and there are also some very core, generally very old and extremely critical components in the technology stacks we all use - people will see that these components don't use "teh hotness" and assume it would work better with the paradigm that's in favor at the time. Sometimes these people are right, sometimes they're wrong - we shouldn't discourage questioning the status quo as, critically, sometimes these people are right - but it is important to be cautious with such changes since generally there are a lot of very precise decisions that have been carefully considered. However, one need not look any further than OpenSSH to see how very serious bugs can exist in "unimpeachable" software that everyone just assumes it's safe to use.
This is, as an aside, the reason why I don't comment my code often (I personally dislike docblocks that do nothing but obvious explanations) - but whenever I think hard about a thing I always record it in a comment with the problem, my reasoning, and my rational for choosing the solution used... i.e. don't comment on the what, comment on the why.
This is, as an aside, the reason why I don't comment my code often (I personally dislike docblocks that do nothing but obvious explanations) - but whenever I think hard about a thing I always record it in a comment with the problem, my reasoning, and my rational for choosing the solution used... i.e. don't comment on the what, comment on the why.
Chesterton's fence,
https://en.wiktionary.org/wiki/Chesterton%27s_fence
You may consider removing the rubber only when you can adqueately explain to me why it is there
You may consider removing the rubber only when you can adqueately explain to me why it is there
I wonder how the author would label Windows' MapViewOfFile :)
[deleted]
People are discovering mmap, it has been used for a very long time especially in databases.
It's true for databases, but this usage is closer to how shared libraries are loaded, just with extenal data.
Also just to show how buggy mmap is, it's disabled in SQLite by default for example:
https://www.sqlite.org/mmap.html
Even now the windows version is buggy, but the great thing is that after it's fixed in llama.cpp, the open source community can just copy the solution.
I don't know of many interactive utilities that were known for fast startup time _because_ of the use of mmap, but now we have one, which means many more are coming :)
Also just to show how buggy mmap is, it's disabled in SQLite by default for example:
https://www.sqlite.org/mmap.html
Even now the windows version is buggy, but the great thing is that after it's fixed in llama.cpp, the open source community can just copy the solution.
I don't know of many interactive utilities that were known for fast startup time _because_ of the use of mmap, but now we have one, which means many more are coming :)
Buggy? In what way?
The "But there are also disadvantages" of the linked pages provides some of the issues with mmap. This also matches burntsushi's experience with mmap in ripgrep:
- depending on concurrent accesses mmap can just sigbus on you (e.g. if the mapped file is being truncated)
- mmap simply does not work with virtual filesystems, and will blow up on large files on 32b systems (windows also has further limitations on mmaps)
- depending on workload mmap may not be faster than regular reads and memory buffers, ripgrep will mmap when working on just a few files, but will use normal buffers for large file counts, because when you start reusing buffer you amortise allocation costs which you can't amortise when creating and destroying mappings
- not only that but mmap/munmap are also globally blocking on the process, so in multithreaded processes it's very bad to map/unmap a lot, you can stall your own application
- the semantics of mmap on crash are also somewhat risky, in that the OS will try very hard to sync, but that may not be desirable if the application crashes in the middle of a write
- depending on concurrent accesses mmap can just sigbus on you (e.g. if the mapped file is being truncated)
- mmap simply does not work with virtual filesystems, and will blow up on large files on 32b systems (windows also has further limitations on mmaps)
- depending on workload mmap may not be faster than regular reads and memory buffers, ripgrep will mmap when working on just a few files, but will use normal buffers for large file counts, because when you start reusing buffer you amortise allocation costs which you can't amortise when creating and destroying mappings
- not only that but mmap/munmap are also globally blocking on the process, so in multithreaded processes it's very bad to map/unmap a lot, you can stall your own application
- the semantics of mmap on crash are also somewhat risky, in that the OS will try very hard to sync, but that may not be desirable if the application crashes in the middle of a write
https://reviews.llvm.org/D69294 is an interesting case we ran into with LLD (the LLVM linker), where the the memory pressure from mmap'ing a large output file combined with filesystem compression resulted in particularly bad performance.
I wouldn't call any of these things a "bug", in the sense of a behavior that departs from an explicit or implicit specification. Some of them are just advantages and disadvantages of that overall approach.
Also, I'm disappointed that VM languages don't turn SIGBUS accessing a memory mapped region into a VM exception. They could, and it wouldn't even be that hard.
Also, I'm disappointed that VM languages don't turn SIGBUS accessing a memory mapped region into a VM exception. They could, and it wouldn't even be that hard.
LLAMA maps a single read-only file at startup. The only caveat that applies here is the virtual filesystems one, and it's a fair assessment, so a fallback might be warranted, but mmap seems fine by default.
Databases usually avoid mmap in favor of their own mmap-like system: https://db.cs.cmu.edu/mmap-cidr2022/
That depends on exactly what kind of DBMS and underlying OS one is talking about. For DBMS that runs as one daemon or set of related daemons that has some kind of mechanism to share the afore mentioned buffer pool and synchronize acces to it then sure, it makes more sense, especially as you can do fine-grained copy on write in that bufferpool as an transaction isolation mechanism.
For embedded databases that operate on shared file somewhere just mmap()ing that file allows you to use part of the same file as shared memory segment and for POSIX IPC and thus synchronization between otherwise unrelated processes that access the same data. One big caveat of this approach is that it depends on optional parts of POSIX that are realistically usable on Linux and Solaris and maybe some obscure systems (you can probably do the same thing on Windows NT), notably this does not work on essentially anything BSD-derived.
For embedded databases that operate on shared file somewhere just mmap()ing that file allows you to use part of the same file as shared memory segment and for POSIX IPC and thus synchronization between otherwise unrelated processes that access the same data. One big caveat of this approach is that it depends on optional parts of POSIX that are realistically usable on Linux and Solaris and maybe some obscure systems (you can probably do the same thing on Windows NT), notably this does not work on essentially anything BSD-derived.
It has been used for so long in databases it should no longer be used in databases.
http://www.cidrdb.org/cidr2022/papers/p13-crotty.pdf
http://www.cidrdb.org/cidr2022/papers/p13-crotty.pdf
Just saw your reply, we replied with the same CMU link :)
[deleted]
They must use it in games too, no? For level loading, assets, etc...
In general game engines are almost full operating systems and reimplement many things such as memory management
mmap is modern? Lol. I used mmap in Java back in 2013 to offheap large matrices in our regression engine. Mmap probably exists long before that.
Edit: and yes, the JVM only reported a few hundreds MBs used in the heap. In reality, the memory mapped file is several GBs in size.
The syscall was described in 4.2-4.3BSD in the 80s and shipped in 4.3BSD-Reno in 1990. The underlying concept of memory-mapped files dates back to at least the 70s. So all of this to say -- I agree, it's not especially new.
https://en.wikipedia.org/wiki/Mmap
https://en.wikipedia.org/wiki/Mmap
I have kids older than your usage of mmap.
> modern? Lol. 2013
How to tell if a dev is in their 20s or thereabouts
How to tell if a dev is in their 20s or thereabouts
2013 is modern.
Short version: the claim that llama could use much less memory by memory-mapping the model data file was wrong. It's just that the Linux utilities for memory use don't count memory-mapped file area.