Introduction to High-Performance Scientific Computing(pages.tacc.utexas.edu)
pages.tacc.utexas.edu
Introduction to High-Performance Scientific Computing
https://pages.tacc.utexas.edu/~eijkhout/istc/istc.html
17 comments
> • First of all, as of this writing (late 2010), GPUs are attached processors, for instance over a PCI-X
bus, so any data they operate on has to be transferred from the CPU.
I think we got GPUDirect RDMA circa 2013. How time flies!
I think we got GPUDirect RDMA circa 2013. How time flies!
try more, measure more, and think about throughput and latency
nothing really changes
nothing really changes
A very timely submission!
I've been looking into performance optimizations on heterogeneous multicore systems and much of what I've seen published recently on Arxiv seems to point to tasks, their granularity and their scheduling as increasingly important.
This book mentions but doesn't spend a lot of space on these subjects. It will be very interesting to see how it all evolves.
I've been looking into performance optimizations on heterogeneous multicore systems and much of what I've seen published recently on Arxiv seems to point to tasks, their granularity and their scheduling as increasingly important.
This book mentions but doesn't spend a lot of space on these subjects. It will be very interesting to see how it all evolves.
A book could probably be written on the topic of latency-hiding schedule design alone, and I am not aware of a canonical resource that gives it a thorough treatment. The case of regular hardware/software parallelism is trivial, but it becomes interesting and very complex once you introduce irregular hardware parallelism (e.g. heterogeneous compute elements) or irregular software parallelism (e.g. variable and unpredictable task concurrency -- graph analysis often has this characteristic). The optimal number of tasks any scheduler deals with always falls in a bounded range; trying to keep the number of immediate tasks within that range when the tasks are generated unpredictably and non-locally is non-trivial.
It isn't enough that the task scheduler is adaptive to irregular hardware and software parallelism, in HPC you are effectively running a lot of task schedulers in parallel, each managing their local compute environment and interacting with each other. You sort of need a "meta-scheduler" to schedule the dynamic behavior across all the schedulers so they don't adversely affect each other, which is not scalable. An alternative approach I've often seen is adding a game theoretic context to task schedulers, each tacitly modeling the expected dynamic behavior of other schedulers they interact with. This doesn't require schedulers to explicitly coordinate their state, a big win for scalability, in order to optimize their aggregate behavior. In the HPC context a robust and nearly optimal equilibrium can sometimes be achieved. In the ideal case you can prove the resource requirements for an individual scheduler that can guarantee well-bounded worst case behaviors.
In HPC the topology of the schedulers is essentially fixed (i.e. you know what hardware you are working with) but there is an even more difficult flavor of the same latency-hiding task scheduling problem when the execution environment can have a variable topology i.e. nodes appear and disappear in random places.
There is still a lot opportunity for interesting research on this topic.
It isn't enough that the task scheduler is adaptive to irregular hardware and software parallelism, in HPC you are effectively running a lot of task schedulers in parallel, each managing their local compute environment and interacting with each other. You sort of need a "meta-scheduler" to schedule the dynamic behavior across all the schedulers so they don't adversely affect each other, which is not scalable. An alternative approach I've often seen is adding a game theoretic context to task schedulers, each tacitly modeling the expected dynamic behavior of other schedulers they interact with. This doesn't require schedulers to explicitly coordinate their state, a big win for scalability, in order to optimize their aggregate behavior. In the HPC context a robust and nearly optimal equilibrium can sometimes be achieved. In the ideal case you can prove the resource requirements for an individual scheduler that can guarantee well-bounded worst case behaviors.
In HPC the topology of the schedulers is essentially fixed (i.e. you know what hardware you are working with) but there is an even more difficult flavor of the same latency-hiding task scheduling problem when the execution environment can have a variable topology i.e. nodes appear and disappear in random places.
There is still a lot opportunity for interesting research on this topic.
You know, I have noticed some of this when going over research papers, but I don't think I'm fully appreciating the significance. I appreciate the response, although it is going to force me to do more research and do a lot more thinking ;)
So, is deep learning eating HPC's lunch?
And if it is, how much of this involves deep learning buying computer at a cheaper rate than HPC?
Most serious HPC involves simulating something (weather, atomic particles, car crashes). Lately, there has been a lot of work using neural networks to approximate such simulations more effectively. But would these make sense if the HPC program ran on GPUs to start with.
And if it is, how much of this involves deep learning buying computer at a cheaper rate than HPC?
Most serious HPC involves simulating something (weather, atomic particles, car crashes). Lately, there has been a lot of work using neural networks to approximate such simulations more effectively. But would these make sense if the HPC program ran on GPUs to start with.
This material is aimed at those who will be developing/implementing HPC software. HPC techniques are also used for ML approaches, since deep learning also requires a massive of computation, and benefits greatly from massive parallelization. GPUs are common in HPC, for DL and also other computations.
That said, there certainly is a great interest in the scientific computing community in ML/DL.
That said, there certainly is a great interest in the scientific computing community in ML/DL.
[deleted]
I’ve had a lot of dealing with the folks at TACC in my day job, and their work is pretty amazing. Add in a Top10 supercomputer and you have something pretty impressive.
how has the the enterprise big data ecosystem (hadoop, spark, perhaps even more recently, cloud based stuff such as snowflake) influenced HPC?
This was admittedly a while ago (2012 or so) but when I was in grad school, I would often challenge other people in my Distributed Systems lab to compare their Hadoop jobs against just running a batch job on a single system (without the Hadoop overhead etc, just sequential). More often than not, the threshold for "when does this actually make sense" was much larger than they'd expected.
And then [1] came out and disappointed some people. By that time I was mostly out of the lab and just doing minor thesis edits, but when I brought it to our weekly reading group the discussion was... animated.
[1] https://www.usenix.org/system/files/conference/hotos15/hotos...
And then [1] came out and disappointed some people. By that time I was mostly out of the lab and just doing minor thesis edits, but when I brought it to our weekly reading group the discussion was... animated.
[1] https://www.usenix.org/system/files/conference/hotos15/hotos...
Very little, I think. They are very different.
HPC has to do with solving certain rigorous scientific equations through the use of various numerical methods (differential equation solvers, linear algebra solvers, Monte Carlo simulation, etc). "Enterprise big data" is very far from having rigorous scientific equations that need solving.
HPC has to do with solving certain rigorous scientific equations through the use of various numerical methods (differential equation solvers, linear algebra solvers, Monte Carlo simulation, etc). "Enterprise big data" is very far from having rigorous scientific equations that need solving.
This is a great resource IMO!
I'm working on a HPC company where everyone is an order or two of magnitude smarter than me. It's fun and overwhelming. If anyone has a recommended study plan for this subject (I have an unstructured background in CS) or 'lighter' complimentary resources, I'd be grateful to read them.