It doesn't get in that situation, because malloc() can return null on Solaris (i.e. it never¹ overcommits).
While in general I think this is vastly better than the somewhat insane Linux OOM killer, you can get in awkward situations where you can't start any processes (including a root shell) because you're out of memory.
I rather like the FreeBSD solution to this, which is to not overcommit, but after a certain number of allocation failures it kills the process using the most memory. This prevents situations where you can't start any processes.
There's no one-size-fits-all solution to handling low memory conditions, but the Linux solution manages to almost never do what you want which is kind of impressive in a way.
¹ I seem to recall hearing somewhere that you can allow allocations to overcommit on a per-application basis on later versions of Solaris, but don't quote me on this.
This makes sense but is somewhat annoying since a web browser is not the only program running on my computer (but apparently wants to be) and eats up RAM that could be used for the OS file cache.
I wish there was some sort of allocation API specifically for allocating caches so that recently accessed files could kick out a web browser's cache of a not-so-recently accessed tab or vice versa.
There's #proglangdesign on freenode where a bunch of us have been on an array language/function-level streak lately. Some people there might be aware of more.
- Dependently typed languages (ATS, Agda, Idris)¹ should be fairly familiar if you're a Haskell veteran.
- Array languages (APL, J, K, and more obscure ones like Nial) are pretty enlightening if you're a functional programmer (at least they are for me). Most of these trace to Ken Iverson and his Notation as a Tool of Thought. They are a bit brain-bending at first, largely because of the density, but they're fun to use and the density makes comprehension easier after a while.
- Function-level languages (FL, FPr, J) – a somewhat obscure and very advanced sister to functional programming. If you're familiar with Haskell Arrows, there are many parallels. They are, essentially, a more convenient point-free style. Most (all?) of these trace to John Backus (of Fortran and Backus-Naur Form fame) and his 1977 Turing Award lecture Can Programming Be Liberated From the von Neumann Style?
J (http://jsoftware.com/) combines array and function-level programming, and IMO is a very good language to learn to expand your horizons if you're a veteran functional programmer.
--
¹ Coq kinda, but it's more of a theorem prover than a programming language. Agda sort of fits that too.