> I think the only way to really do this well is to make HPC jobs a market system where resources cost money to the users. When money is involved, people are incentivized to optimize their workloads. But that’s rarely the case for large HPC clusters and I’d personally hate it if I had to deal with a HPC processing budget.
I also got that, there aren't many incentives in HPC. Users want results and don't have the time or skill to invest in rewriting the software stack to be as efficient as possible.
An intuition that I have is that this problem could be solved at the application/framework layer. Instead of launching jobs, which are glorified bash scripts, control the logic of the resource allocation dynamically from the application software.
It will solve the problem of load balancing based on your actual compute needs, but what happens if a program requests resources in a loop? At least the current schedulers know in advance the requested resources and can stall huge allocations
HPC student here, still learning. If I understood your problem statement, users of the cluster reserve resources far greater than what was needed for their computation. They fear that if the allocated resources are not enough, then their program will crash and lose partial results.
Can you give an example of typical execution on the cluster? Is it a problem of number of hours allocated or number of compute cores?
If I'm running a PDE simulation, and I allocate n machines I want to use all of them, so there is no risk of idle machines. It's not trivial to estimate a priori the amount of time required for my simulation to complete, so I overestimate. But when the simulation is complete (even before the deadline), the resources get freed and can be used right away for another job
Maybe the problem is when many users are greedy. Also MPI simulations are difficult (if not impossible, correct me) to change dynamically: when a simulation is started with that number of ranks, I can't add new ranks at will if the resources are available
Thank you for the patience for everyone that answers
Very interesting post and I think this exposes the limitations of the Julia compiler. Note that an old version of the compiler is used (1.0.3 from 2019).
One could say that we can almost replicate the semantic of a C++ program, but writing in Julia. For example we can remove bounds checks in arrays or remove hidden memory allocations.
But the goal of a language for numerical computing is capturing the mathematical formulas using high level constructs closer to the original representation while compiling to efficient code.
Domain scientists want to play with the math and the formulas, not doing common subexpression elimination in their programs. Just curious to see how it evolves
I found funny the hairstylist did provide a pretty distopic reason to use ChatGPT... it seems that you are trying to please your new employer... Nevertheless I respect performance work and I'm studying for something similar. I hope to land a job in HPC
I'm trying to solve the brachistochrone problem by numerical optimization. I started with the minimal surface problem to get a foot in the door: I discretize the integral as a sum of constributions that depend on the function values at specific points, then use autograd libraries for optimizing a non-linear scalar loss function
Yeah, using LLMs makes me reconsider the complexity of the software I'm producing and I'm relying on. In a sense LLMs can be a test for the complexity and the fast iteration cycles could yield better solutions than the existing ones
But what was wrong with pip, venv and pyproject.toml in the first place? I just keep a system installation of python for my personal things and an environment for every project I'm working on. I'd get suspicious if a developer is picky about python versions or library versions like what crazy programs are you writing?
A research project on converting a python application from loops to numpy vectorization to jax and pytorch compilation (the latter is fast!). I'm exploring the performance portability of such frameworks across different devices and backends (also gpu). I have also C++ hand optimized implementations that so far are unbeaten