You can't fix the fact that you can't predict the future. The software will always allocate more memory than it needs because it can't predict its future resource usage, so limiting memory allocation with strict overcommit is meaningless.
The kernel keeps track of active/inactive pages by scanning page tables during the reclaim process. It only scans until it finds enough inactive pages to reclaim. Scanning all pages all the time is very expensive from a performance point of view.
There is no point in managing memory allocations as they have little relation to actual memory usage. There are also other methods than the OOM killer to handle OOM, like process throttling using cgroups "memory.high" limits.
In this specific case, the correct behaviour would be to drop a part of the buffer pool until the memory pressure is gone. The context-dependent question is how much and how fast to drop. The current implementation drops to a single configurable level but I suspect it could have implemented better heuristics.
You can always tell the system not to kill your Photoshop (or whatever else) by setting the OOM Score Adjust. This mechanism has existed for almost two decades and systemd has supported it for over a decade.
FreeBSD didn’t have memory overcommit and instead used strict swap reservation - each allocated anonymous memory page was supposed to have a corresponding swap page. This required 2x RAM swap space, otherwise you would get “out of swap” when forking a large process. FreeBSD implemented memory overcommit around 2000.
People like to reinvent things that they are not aware of. Original BSDs used to use strict swap reservation - every anonymous memory page had to have an associated swap page. You had to have the swap 2x of RAM to allow large processes to fork - otherwise you would get an "out of swap" error. FreeBSD implemented overcommit around 2000, I think version 4.x or 5.x.
MariaDB recently implemented memory PSI monitoring but failed with that in a curious way and disabled it afterwards by default. The failure is that under memory pressure, they flushed the entire InnoDB buffer pool.
> On the modern desktop, where programmers don't care about failing malloc(), disabling overcommit is shooting yourself in the foot. As you can observe, the memory allocations start failing long before the memory is exhausted.