32 MiB Working Sets on a 64 GiB machine(randomascii.wordpress.com)

170 pointsby nikbackm3 tahun yang lalu100 comments

randomascii.wordpress.com

https://randomascii.wordpress.com/2023/10/01/32-mib-working-sets-on-a-64-gib-machine/#more-4032

101 comments

[deleted]3 tahun yang lalu

NavinF3 tahun yang lalu

Wow, PROCESS_MODE_BACKGROUND_BEGIN is quite the footgun. I wonder if this started out as a hack to reduce the impact of OEM bloatware on benchmarks. Is there an easy way to list all processes that use this priority class?

brucedawson3 tahun yang lalu

There probably is a way to find what processes have this mode enabled, but it's probably more productive to search github for references to the problematic constant.

fluoridation3 tahun yang lalu

Run Process Explorer and enable the Memory Priority column. In the Select Columns dialog, its checkbox is on the Process Memory tab. Priority 5 is normal; priority 1 is background.

kristianp3 tahun yang lalu

So he didn't report the bug to Microsoft? His final comments were just to say:

"This issue has been known for eight years, on many versions of Windows, and it still hasn’t been corrected or even documented. I hope that changes now."

brucedawson3 tahun yang lalu

I reported the documentation bug. I don't use Feedback Hub because it never works for this type of issue. I know that some Microsoft developers will see this blog post and I hope that internal bugs get filed.

But ultimately as a non-paid tester of Windows I'm under no obligation to report non-security bugs in any particular way. I like reporting them through twitter and blog posts.

theolivenbaum3 tahun yang lalu

Your blog is the most effective bug reporting tool for Windows issues. Thanks for keeping it going!

_a_a_a_3 tahun yang lalu

I reported a probably Windows perf bug. Nothing happened. I reported it again. Basically they weren't interested despite my careful spelling it out. Time wasted. Fuck them.

Maybe it wasn't a bug (hah!) but they could have come back to me and at least told me they'd looked at it and it wasn't. I'd have appreciated that. But no.

dabbz3 tahun yang lalu

Back when I worked on Windows, we would triage Windows Feedback hub reported issues once a sprint (~2 weeks). But only those which had at least 2-5 upvotes on them. One-off bugs usually meant they weren't being widespread impactful and weren't as higher priority on our list (even if that isn't the case).

But the feedback hub is an abyss of random complaints to filter through. Sometimes we'd see items like "Windows wouldn't save my word document and now I hate windows" and being on the networking team were like "uhhh thanks"

ack_complete3 tahun yang lalu

As an external developer, it's become clear to me that some bugs don't get fixed simply because feedback doesn't get through to the product teams.

An example is the MediaFoundation AAC encoder in Windows 10. It's unusable, due to a bug that randomly introduces oink artifacts into the output. I submitted it to Feedback Hub, but of course it was too niche to get upvotes. Looked around, OBS Studio ran into the same issue and had to switch to a different AAC encoder, so it wasn't a rare problem. Someone even tried posting on Microsoft Answers reproducing it with the Windows SDK sample, and got only a generic response. Finally someone got through, and the product team mentioned that this was the first they'd heard of it... but at least it's actually fixed for Windows 11.

The design of Feedback Hub partly encourages the current behavior. It used to have a very visible categorization on the left side, but now they're easily missable drop-downs, and the auto-suggestions are completely bonkers. You can type up a bug about graphics errors and it suggests the accessibility and first-time install categories. Lack of curation also doesn't help; there's something to be said for having a light touch, but there are posts consisting of just random letters that have been there for years.

iforgotpassword3 tahun yang lalu

I understand it's impossible for a dev team to comb through all user feedback for a product as omnipresent as Windows, but the downside of this is probably that high quality feedback gets buried, because the average user might not understand the tech jibber jabber even if they have the same problem, so they don't upvote and just file a new report maybe not even describing the problem itself, just some random side effect that happens to annoy them.

markdog123 tahun yang lalu

He reported it in the best way possible. We all know what would have happened if he filed a bug report (especially if they didn't realize it was him).

glonq3 tahun yang lalu

I'd wager that the best solution is to get Raymond Chen's attention. There's nothing that he can't do!

Varriount3 tahun yang lalu

I found this paragraph particularly interesting:

> Trimming the working set of a process doesn’t actually save memory. It just moves the memory from the working set of the process to the standby list. Then, if the system is under memory pressure the pages in the standby list are eligible to be compressed, or discarded (if unmodified and backed by a file), or written to the page file. But “eligible” is doing a lot of heavy lifting in that sentence. The OS doesn’t immediately do anything with the page, generally speaking. And, if the system has gobs of free and available memory then it may never do anything with the page, making the trimming pointless. The memory isn’t “saved”, it’s just moved from one list to another. It’s the digital equivalent of paper shuffling.

I'd always been under the impression that as soon as memory was trimmed from the working set. Perhaps this was the case at some point, and was a reason for the PROCESS_MODE_BACKGROUND_BEGIN priority? As the blog mentions, the SetPriorityClass call has had this behavior since at least 2015, though I wouldn't be surprised if this behavior has existed for much longer.

As for why this "bug" hasn't been fixed, my guess is that it's due to a couple of factors:

- Windows has become fairly good over the years at keeping the core UI responsive even when the system is under heavy load.

- There are plenty of ways to reduce memory/CPU usage that don't involve a call to SetPriorityClass. I'd wager that setting a process's priority class is not the first thing that would come to mind.

- As a result of the previous two points, the actual number of programs using that call is quite small. I'd actually be interested in knowing what, if any, parts of Windows use it.

(As a side note, if there was a bug in a Windows API function, how would you even report it?

Bjartr3 tahun yang lalu

> Windows has become fairly good over the years at keeping the core UI responsive even when the system is under heavy load.

I would say that's debatable. There are fewer complete freezes that require a restart, but things like the task manager (can't get much more core than that), which used to be instantly available and responsive if the machine was recoverable at all, now can take tens of seconds to show up and and respond to interactions under heavy load.

ComputerGuru3 tahun yang lalu

Launching it no longer uses a software interrupt request.

mrguyorama3 tahun yang lalu

I'm not sure I understand, why not just manage your own memory instead of asking the OS to magically understand your memory use-case and properly manage/limit?

If the memory isn't owned by the OS, that's probably the wrong actor to manage it.

They say that, instead, they used an undocumented API they obviously didn't understand. Why? If you are concerned about using up too much memory, then do something about it

remram3 tahun yang lalu

What's undocumented?

mrguyorama3 tahun yang lalu

>They didn’t document this (!)

Is the fourth line of the article linked.

tiagod3 tahun yang lalu

The article makes it clear that the API is documented. The 32MiB limit isn't.

fluoridation3 tahun yang lalu

To be fair, the limit is part of the API's semantics. If the limit is undocumented, the API is at least partly undocumented. It's as if a car had a headlights button that the manual had this to say about it: The headlights button turns on the headlights and the headlights indicator light on the dashboard. However, nothing in the manual states that the headlights indicator shares a circuit with a valve that drains the fuel tank, so in effect when you turn the headlights on, you're causing the fuel to leak. I would say the button is not properly documented in that case.

Dylan168073 tahun yang lalu

You're stretching too far in your attempt to be fair.

When people talk about using undocumented APIs in windows, that means secret APIs, not badly documented APIs. That is the only meaning that fits the accusatory tone of the comment.

fluoridation3 tahun yang lalu

I agree. That's why I didn't say that the feature was undocumented, but that it was improperly documented. I also don't agree that it's right to blame the user for using a feature that's not properly documented, since the user has no way of knowing that.

Dylan168073 tahun yang lalu

Then I'm not sure why you said "to be fair[...]partly undocumented".

Isn't the thing you being fair to the last line of mrguyorama's comment? The one that was blaming the user? The thing you replied to was an argument over the appropriateness of that specific line.

fluoridation3 tahun yang lalu

"To be fair" because neither "it's documented" nor "it's undocumented" capture the state of things. I'm highlighting why someone would consider the API not fully documented to make it easier for others to see mrguyorama's point of view, even if I disagree with it. In other words, I'm steelmanning their argument, since I think it was poorly put forward.

Dylan168073 tahun yang lalu

It's only steelmanning if that's what they meant, and I don't think that's what they meant.

Trying to strengthen an argument by finding a way that a word could technically work, but not with the original implications, isn't very helpful. I'd even argue it hurts clear communication in a situation like this.

brucedawson3 tahun yang lalu

I'll just reiterate what is said later on. The API is documented. However this extremely important effect of calling the API is not. We called the API expecting lower I/O priority and lower CPU priority or something like that. We perhaps expected lower memory priority. Really the documentation should have spelled all of this out.

But in addition to this we got a problematic working-set cap which can absolutely destroy performance. That detail needs to be documented (and the rest of the behavior should be as well).

tedunangst3 tahun yang lalu

I feel like there must be another part to the story. How did only one chrome user notice this?

remram3 tahun yang lalu

I mean, it's only the installer. I know I only run it once per machine, usually right in the middle of installing all the rest of my usual utilities. It would be easy to miss that it's much too slow.

It's not like I have a good sense of what it does anyway, in addition to copying files for a fraction of a second.

mook3 tahun yang lalu

Only one Chrome user successfully reached a developer in a position to do something about it. Who knows how many encountered it but didn't notice, and how many more noticed it but couldn't find a way to file a bug that would get noticed.

brucedawson3 tahun yang lalu

It's a good question. It could be that most people don't notice this things, those who don't don't report it, etc. Or maybe it only affects some people, but I can't think how that could be true. More mysteries

brucedawson3 tahun yang lalu

I'm running some monitoring that will tell me how setup.exe behaves on my machines. I tested it by forcing an update from Chrome. The private commit and private working set were 66.76 and 61.70 MiB, so no capping, but that makes sense because a user-initiated update isn't running at low priority, so we don't trigger background mode.

So, the only time the bug happens is for background updates that may be happening when the user isn't even present. And, since the CPU priority is low they won't harm responsiveness of foreground applications. But they will waste CPU time and electricity and battery life (if on battery). So, a silent waste.

john5673 tahun yang lalu

I can tell you exactly how I noticed it because it was on my workstation this reproduced originally.

I have a Ryzen 9 5950X CPU and it has a crazy fan ramp. Anytime a process maxes out the CPU it will be audible to me when I'm working. Now, to put this in perspective. It doesn't happen when I'm gaming. I can play Diablo 4 and Cyberpunk 2077 without this noticable fan noise but when setup.exe loses it, the fan noise is how I notice it. It will max out one core and I'm going to assume "never" complete. The longest I waited was 78 CPU minutes before killing the process. This would happen now and then but it would not prevent Chrome from successfully updating. So, it was bizarre to begin with.

Borg33 tahun yang lalu

Haha :) I just checked MSDN docs and vioala: PROCESS_MODE_BACKGROUND_BEGIN - Not supported under Win2003 or Windows XP.

Another improvement from Microsoft without really understanding their own OS.

jacobgorm3 tahun yang lalu

I think they understood their OS and memory model better than most commenters in this thread do. I was once paid to read most of Windows Internals 5 and try to understand their memory management concepts, some of which date back to VMS, and most of them actually make a lot of sense.

The Background mode has its merits too, but I guess they never imagined that a background process would need to spend huge amounts of memory just to do some background work.

Filligree3 tahun yang lalu

Huge? The limit is 32MiB. That isn’t large by any measure.

Borg33 tahun yang lalu

Yeah, especially it was intruduced with Win7 AFAIR. So processes were already pretty inflated ;)

jacobgorm3 tahun yang lalu

Win7 was supposed to be usable with 1GiB of RAM, so 32MiB for some low priority background task seems quite reasonable.

https://support.microsoft.com/en-us/windows/windows-7-system...

Dylan168073 tahun yang lalu

I wouldn't really count the 32 bit version, but either way windows 10 has the exact same requirements as windows 7. Actually 7 was higher by a gigabyte if you wanted XP mode! So if you agree this would have been a bad limit in 2020, then that suggests it also would have been a bad limit in 2010.

Borg33 tahun yang lalu

Sure, most of time im very happy with Win2003. Pretty solid OS. There are some issues, too bad never resolved, like PageFile management and Cache management for example. I bet there are more...

chrisbolt3 tahun yang lalu

Previous discussion (2 days ago): https://news.ycombinator.com/item?id=37734685

dang3 tahun yang lalu

That one never made the front page so maybe we'll move the comments hither and let this one run.

(It's on my list to do more for submissions like that which accrue a bunch of upvotes slowly but never break the front page...)

remram3 tahun yang lalu

Looks like you did the move. It's really weird because it changed the timestamps on all the comments... They are even different between my "threads" page and what I see when I click through.

dang3 tahun yang lalu

Yes, sorry, I know it's confusing. I relativized the timestamps for 2 reasons:

(1) if we didn't do that, commenters would start asking "why are there a bunch of comments here older than the OP they're commenting on?" - and our experience is that this leads to more confusion than the other way around; plus

(2) it seems unfair to have all the comments on the earlier thread plummet to the bottom of the new thread because they're so much older.

This topic comes up regularly in a slightly different context, which is re-upping stories that make it into the second-chance pool (https://news.ycombinator.com/item?id=26998308). There are a bunch of past explanations here in case helpful: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que....

jrmg3 tahun yang lalu

Trimming the working set of a process doesn’t actually save memory. It just moves the memory from the working set of the process to the standby list. Then, if the system is under memory pressure the pages in the standby list are eligible to be compressed, or discarded (if unmodified and backed by a file), or written to the page file. [… If] the system has gobs of free and available memory then it may never do anything with the page, making the trimming pointless. The memory isn’t “saved”, it’s just moved from one list to another. It’s the digital equivalent of paper shuffling.

This sounds like it should be basically free in a memory-unconstrained system - is the system really just spending all its time managing these lists? Why is it so expensive?

tetha3 tahun yang lalu

I don't know about Windows, but the Linux kernel is entirely fine to just keep rarely used memory pages around as long as it doesn't need the memory. If you don't need the memory otherwise, having the page loaded and cached is better than needing to load it.

Even though, then you end up with confused reports about "Linux eating all the memory" and "an idle system using all the memory (for caching)".

AFAIK, newer kernels also have a preemptive swap-out of idle pages so they can evict the pages quicker, so an idle system might even be swapping (out). This results in even more confusion.

zamalek3 tahun yang lalu

> "Linux eating all the memory" and "an idle system using all the memory (for caching)".

I find these reports bewildering (alongside CPU usage). You would hope that something would be exploiting your expensive machine to the fullest (so long as the work being done is useful/intentional/desired - obviously not wasteful page thrashing).

andrepd3 tahun yang lalu

There's the power consumption issue, of course.

wtallis3 tahun yang lalu

For CPU time, yes, but I don't think DRAM power consumption for idle refreshing varies meaningfully depending on the data stored in it.

vlovich1233 tahun yang lalu

You don’t see it in x86 I think, but I believe ARM mobile SoCs can completely turn off banks that are unused. I haven’t followed to see if anyone actually does this yet, but I recall hearing about attempts to turn off unused RAM when in sleep (when awake, the RAM refresh cost is negligible). To turn off the RAM implies that you might want to prefer to place cache data in the banks that you will want to turn off so that you can just drop caches on sleep quickly. Of course, whether that’s actually beneficial in practice is hard to say - some caching you want to be persistent as you’ll need it on wake anyway.

Outside cases where you want to optimize for sleep performance, you probably don’t want to do this though because NUMA systems have conflicting requirements for storage (but thankfully for now the systems with NUMA and the systems that benefit from turning off RAM refresh when sleeping has 0 overlap). Consumer desktops are probably not worth doing this on and laptops probably isn’t a huge difference because of battery size / usage patterns.

NavinF3 tahun yang lalu

You're probably thinking of DDR self-refresh which lets the entire CPU (including the memory controller) power off without losing data when the device is in sleep mode. I've never seen hardware that partially powers off memory the way you described.

iforgotpassword3 tahun yang lalu

But it's a pretty cool idea :-)

vlovich1233 tahun yang lalu

My memory could be faulty, but I didn’t think it was self refresh which has been quite common for a long time. Maybe it’s possible the OS vendors never got the idea working on the software side and abandoned it / in practice you could never arrange to have a good likelihood of having an entire bank of memory unused. You could estimate the benefit with SW experiments cheaply before ever going down the path of building the required HW support.

placesalt3 tahun yang lalu

The effect of caching on Windows can be quite pronounced if you process large datasets on a machine with a large amount of RAM.

If, say, you have 256GB RAM and a 100GB folder of ~1GB files, you will only ever have a few GB used actively. A first pass of processing over the folder will take a long time (reading from disk). Subsequent passes will be much faster, though, because reading is done from the RAM-cached versions of the files (the output from the previous run, was my understanding).

5e92cb50239222b3 tahun yang lalu

https://youtu.be/beefUhRH5lU

wongarsu3 tahun yang lalu

My impression is that Windows is somewhat aggressive at paging or using memory compression (as a faster alternative to a write to swap), with the goal of freeing more space for cache use. Not very aggressive, but a lot more than Linux.

brucedawson3 tahun yang lalu

Yep, that is generally what Windows does, using available memory for the disk cache. In the user trace the system had lots of disk cache (available memory) and lots of _free_ memory, so there really was zero value in trimming the working set.

But, (undocumented) rules gotta be followed, so trim the working set it is.

toast03 tahun yang lalu

When the address isn't in the process page table accessing it traps into the kernel; then you've got to do all the cache clearing stuff for speculative execution vulnerabilities; then the paging logic needs to figure out what to do, in this case because of the set limit, you've also got to demote at least one page, which means updating the page table, but also sending IPIs to all processors to flush that page/those pages from the TLBs; and then you return to execution.

If you're accessing a lot of pages, mostly randomly, it's going to be a mess. If you're decompressing files (as an installer might do), and the compression window is large relative to the limit (as it might be to get the highest compression ratio), that could be a lot of mostly random access to pages, causing a lot of page faults, and high cpu.

ComputerGuru3 tahun yang lalu

Also context switching between kernel and user mode is now heavier than ever courtesy of speculative execution mitigations.

leptons3 tahun yang lalu

HideousKojima3 tahun yang lalu

> don't know of any computers that have 64 GiB of RAM.

It's the other way around, most computers have ram in gibibytes but list it as gigabytes. It's mostly hard drive and other storage manufacturers that treat gigabytes as actual gigabytes as a way to skimp out

Dylan168073 tahun yang lalu

If you want a powershell command to check your memory capacity, try (Get-CimInstance Win32_PhysicalMemory).Capacity

My RAM sticks each have 8589934592 = 2^33 bytes.

jacobgorm3 tahun yang lalu

But why would Chrome's setup.exe need a 32MiB working set? If all it does is download and unpack some files in the background, it would seem like something that is doable in 32MiB or less.

The Windows BACKGROUND mode is useful for stuff like virus scanning or database log compaction, where you want to make some progress, but only if you can be certain not to hurt any foreground workloads. Back when we had hard drives, this wasn't actually trivial to achieve, because it is very easy to interfere with the foreground workload just by doing a few disk seeks every now and then.

I agree the 32MiB working setlimit is somewhat arbitrary and should be documented, but Windows is full of these arbitrary constants, like the 32kiB paging chunk for instance.

My recommendation for Chrome would be to stick with the background mode, and fix whatever problem is causing the working set to exceed 32MiB.

toast03 tahun yang lalu

The offline installer reports as 103 mb, depending on the compression settings, it might need a significant amount of that mapped in at once.

jacobgorm3 tahun yang lalu

Probably because the files to be binary patched are now quite large, and are loaded into memory verbatim. Perhaps they could memory-map them instead, and ensure that their binary diffing tool works from left to right to avoid thrashing?

Dylan168073 tahun yang lalu

How do you know it's not memory mapped?

Working left to right means you have to force the compiler to always align the code in the same order, which sounds difficult and like a waste of time.

jacobgorm3 tahun yang lalu

No.

Dylan168073 tahun yang lalu

If you want both the input and the output to go left to right, Yes.

If you only make the output go left to right, you don't actually save much memory. You're still randomly accessing a huge buffer.

iforgotpassword3 tahun yang lalu

> The Windows BACKGROUND mode is useful for stuff like virus scanning or database log compaction, where you want to make some progress, but only if you can be certain not to hurt any foreground workloads.

But you are hurting the overall system performance, or battery life if this is a laptop and you keep thrashing the working set for no good reason. I don't see how this proactive approach has any advantages over just swapping out the process first if you run into actual memory pressure, or maybe start paging out pages of that process if they've been idle for a few minutes if you insist on some kind if proactive measure.

jacobgorm3 tahun yang lalu

The background mode is not hurting anyone, it is the thrashing process that is hurting itself by using too much memory. This shows up as high CPU but, because it is in the background, the rest of the system performance should be largely unaffected. Anything with foreground priority will still be scheduled to run whenever it needs to.

Had the background process been allowed to keep growing its working set then foreground applications would be forced to page out, defeating the goal of having a background mode in the first place.

AHTERIX50003 tahun yang lalu

But the limit of 32 MB is arbitrary and undocumented which is a huge problem.

saagarjha3 tahun yang lalu

Doing useless work nobody asked for is always a problem. People use systems with batteries and they would appreciate their system not doing that.

brucedawson3 tahun yang lalu

The foreground applications will still be paged out, only now they will be paged out for longer.

Without background mode setup.exe would have a 48 MiB working set, for a short period of time.

With background mode setup.exe has a 32 MiB working set, plus 16 MiB of its memory is bouncing back and forth between the working set and the standby list.

Thus, none of its memory will get paged out and foreground apps will get paged out, and this will continue for 250 times as long.

jacobgorm3 tahun yang lalu

The foreground processes will always get scheduled ahead of the background process, and be able steal its evicted pages from the Standby list, thus reducing their own need for paging to external media.

If all you care about is getting your updater to complete quicker, you should not be using the background mode, but ideally the Chrome updater would run in the background without affecting the performance of the foreground applications. Allowing for working set peaks, no matter how short, will fail to achieve that on systems experiencing memory pressure.

brucedawson3 tahun yang lalu

The foreground process will only get scheduled ahead of the background process if there is a shortage of CPU power. On today's many-core consumer machines it is much more likely that the background process will continue to make progress, with the only thing slowing it down being the repeated page faults.

Counting on CPU starvation to save memory seems like a very fragile solution.

I agree that commit peaks can affect foreground performance. I've filed bugs for an ephemeral 480 MiB bug in Chrome (caused by an errant product image), and I blogged about an errant 4 GiB allocation that wiped out the disk cache on my 8 GiB machine (https://randomascii.wordpress.com/2012/09/04/windows-slowdow...).

But treating working-set peaks as synonymous with commit peaks is problematic. I still believe that trimming of working sets will rarely save memory, and a working-set cap will virtually never be better than occasional trimming. The cap fails to address the higher-than-the-cap memory consumption in many cases, it just makes it more expensive.

iforgotpassword3 tahun yang lalu

> This shows up as high CPU but, because it is in the background, the rest of the system performance should be largely unaffected.

And, as said, draining your battery for no good reason.

> Had the background process been allowed to keep growing its working set then foreground applications would be forced to page out

Why would they? As said, you could still page out the processes running in background mode first if you actually start running out of memory. And if after trimming all the background processes to 32mb you still don't have enough memory, you would be in the same situation either way.

The current implementation has absolutely no advantage.

stefan_3 tahun yang lalu

Your recommendation is to stick to a pathological implementation where every page fault results in the equivalent of doing a Java GC run? All independent of the actual memory pressure?

This isn't even about the exact limit, this implementation just makes no sense. Add a hysteresis at least because running this on every page fault is guaranteed to cause a lot of CPU load while not returning very much to the system.

dist-epoch3 tahun yang lalu

In general using a 32 MB or larger compression dictionary really improves compression size.

Not sure if that's why the Chrome installer uses 32MB+.

macqm3 tahun yang lalu

It actually does a lot more than you think.

For example it does pretty interesting diffing - Chrome downloads a patch and uses the previously left-behind copy of the installer to re-build new binaries from the small diff it pulled. See more:

- https://chromium.googlesource.com/chromium/src/+/HEAD/compon...

- https://www.chromium.org/developers/design-documents/softwar...

jacobgorm3 tahun yang lalu

I realize that they do binary diffing, but it was not clear to me if setup.exe was only the thing that runs on first install, or if it keeps running. If it keeps running they should definitely try to keep it in check.

malkia3 tahun yang lalu

All it takes a third-party dll (antivirus, antispyware, or just some other big brother hook) to be part of your executable, and the memory is gone. Or call to some Win32 library (like shell32.dll), and some caching going on there. 32MiB are nothing nowadays.

Just a callstack can be 1MB easily, granted it's RESERVED most of it, just some parts commited, but it can grow there (like directory recursion). Then you have 3 or 4 ThreadPool threads created just like that, etc. etc.

dblohm73 tahun yang lalu

> Windows is full of these arbitrary constants, like the 32kiB paging chunk for instance.

I assume you’re referring to the 64KiB allocation granularity? That’s actually not at all arbitrary: https://devblogs.microsoft.com/oldnewthing/20031008-00/?p=42...

amluto3 tahun yang lalu

Huh, I never knew that particular detail, and I’ve played with Windows quite a bit. But, with my modern OS developer hat on, I’d say it’s a bit lazy, not arbitrary.

> So if allocation granularity were finer than 64KB, a DLL that got relocated in memory would require two fixups per relocatable address: one to the upper 16 bits and one to the lower 16 bits.

…

> Forcing memory allocations at 64KB granularity solves all these problems.

It does, at the cost of silly limitations. A much nicer solution IMO would have been a way to ask for a 64kB aligned allocation for DLLs.

brucedawson3 tahun yang lalu

What advantages would you get by being able to reserve memory with a finer than 64-KiB granularity? Especially with 64-bit Windows it seems quite unnecessary to have finer granularity. Am I missing something?

amluto3 tahun yang lalu

64-bit is recent and 64kB granularity is ooooollllld.

With 64 bits, my real objection is that it’s unnecessarily complicated. There’s some data structure that maps addresses to whatever is mapped at those addresses, and on Windows it also needs to track reserved memory, and it needs to do so at a different granularity. (For huge pages, additional granularities are needed.)

jacobgorm3 tahun yang lalu

By paging chunk I mean the amount of data that gets read in by a hard fault to disk.

brucedawson3 tahun yang lalu

What's wrong with using 32 MiB of memory? That's a relatively small amount, especially if you're trying to patch a 225 MB DLL.

And I would argue that the 32 MiB is worse than arbitrary - it is pointless. I have been thinking about this for a while and I cannot think of a situation where it makes things better. It wastes (a lot of) CPU, and I claim that it doesn't actually save memory compared to letting the balance-set manager do the trimming.

Low CPU priority and low IO priority are important for being a polite background process. Not using too much memory is important. But a working-set cap fails to achieve that last goal.

brucedawson3 tahun yang lalu

Replying to myself, because this is important:

In what scenario would be 32 MiB working-set limit, enforced when pages are swapped in, be more effective at saving memory than a similar limit enforced by the once-per-second balance-set-manager?

Obviously trimming the working set at swap time does a better job of reducing the working set, but that's not what matters. What matters is saving memory. If you are trying to save memory then trimming once a second is just as good, and far more efficient, than trimming on every page fault.

But, trimming at all on a 64 GiB machine with 47 GiB free is just silly. It doesn't make sense to spend lots of an expensive resource (CPU time == electricity == battery life) in order to save a resource which you aren't even fully using.

jacobgorm3 tahun yang lalu

I think what mattered to the people who designed this is not create memory pressure for foreground applications. Anyone who ever tried using a WinXP machine right after boot knows it would need a couple of minutes just to sort itself out from the many auto-starting processes waking up to big pile of new "background" work and fighting over the disk and memory, and I think that was what they were trying to repair with background mode. Windows has a long history of getting a bad reputation due to buggy 3rd party pre-installed software, so they built a rather strict performance isolation sandbox to address that.

Something that only runs once per second is not going to be able to keep up with with a process that is dirtying memory at full tilt, which is probably why they chose to enforce a hard limit by evicting the least-used pages to the standby list, from where one can usually get them back quickly without encountering a hard fault to disk (a soft fault was probably like 10,000x faster than a hard fault back when they did this work). They must have viewed thrashing as a pathological case that programmers would diagnose and repair, just like you have. After all they do provide some pretty good tools for diagnosing memory use.

You may argue that the 32MiB limits needs adjusting to follow Moore's law, but Moore's law stopped working for laptop DRAM sizes many years ago.

brucedawson3 tahun yang lalu

That all makes sense (although I think this feature appeared in Windows 7 rather than XP), and I certainly understand the value of a background mode.

But does the working-set cap _work_? In that scenario where you've got heavy memory pressure I still think it doesn't.

If the background process is typically touching less than 32 MiB in a second then a per-second trimming could reduce the working-set effectively.

If the background process is typically touching more than 32 MiB in a second then the fault-time trimming doesn't work because while it trims the memory from the working set, there is no time for the memory to be paged out, and if the memory did get paged out it would make it even worse because it would need to be paged in. So, CPU (and perhaps disk) overhead is increased, but memory pressure remains the same.

The problem (a process touching too much memory) is real. However the solution does not work. A working-set cap doesn't actually reduce memory pressure on foreground applications any more than a per-second trim.

jacobgorm3 tahun yang lalu

Yes I meant that XP had the problem, and that Win7 tried to remedy it.

You are overlooking that the read-only part of the working set can be released immediately once another process needs those pages -- just zero them and add them to the free list. Only the dirty pages need to get written out to disk.

Anyway, it would actually be simple to test the effectiveness of the working set cap by writing a program that allocs and dirties memory aggressively, and then running it either in normal low priority or background modes, to see how it affects overall system behavior when running in each mode.

brucedawson3 tahun yang lalu

My test program already proves that if you alloc-and-dirty memory aggressively then in background mode you will consume much more CPU time. If your metric is "interfering with foreground processes" then this is fine, but I think the slowdown is severe enough to matter.

As for saving memory, I see your point, but I still don't see how the cap would be better than per-second trimming. If the clean-then-zeroed page is not touched again by the background process then either method would make it available. If the clean-then-zeroed page is touched again by the background process then the whole process of removing it from the working set, zeroing it, then reading it back in and faulting back in is a waste of time. So, again, I'm struggling to find a scenario where the cap is more effective than once-per-second trimming.

criddell3 tahun yang lalu

Bruce - any chance you could make your ETW videos available again? I see they used to be available through a Microsoft partnership of some type but that seems to have ended a few years ago.

I’d love to learn more about ETW and your videos seem like a good place to start. If anybody else has other recommendations, please share!

a1o3 tahun yang lalu

ETW is Event Tracing for Windows if someone reads the above and is wondering.

criddell3 tahun yang lalu

Thanks.

And the videos I’m talking about are linked from here:

https://randomascii.wordpress.com/2014/08/19/etw-training-vi...

brucedawson3 tahun yang lalu

Sorry, I don't own them so there's nothing I can do, unless I can buy them back

criddell3 tahun yang lalu

Makes sense. Thanks for sharing all that you have. It's an incredible resource.

Majromax3 tahun yang lalu

I wonder if this is some cross-pollination legacy of Windows Phone. On such a device, background processes are still necessary, memory is a much tighter constraint, and enough pages might be mapped directly from device flash for auto-eviction to make sense.

sznio3 tahun yang lalu

I remember using some software to increase the pagefile on my Windows Phone. Multitasking worked so much better after that. Apps loaded faster since it would just take the image of an app suspended in the pagefile rather than starting it over.

toast03 tahun yang lalu

As a Windows Phone fan, the timing doesn't quite match up. They reported this happens as far back as Windows 7, but Windows Phone 7 was based on Windows CE; WP 8 was NT based, I would have expected cross-pollination to start showing up in Windows 8.

amluto3 tahun yang lalu

Seriously? Windows has had this bizarre “working set” concept effectively forever. I thought it was weird when Windows 2000 was new, and I still think it’s weird. In the land of page replacement algorithms, there are many of various degrees of cleverness, and then there’s “we exceeded an arbitrarily limit — nothing else needed that memory, but we’re going to zap it out of the page tables anyway!”

(x86 has had “accessed” tracking for a long long time. What would be wrong with keeping over-the-working-set pages present but not “accessed” and tracking accesses without causing page faults? Or at least only trying to expire pages when there’s some degree of memory pressure.)