Couldn't find too many simple alternatives, so, wrote my own while on vacation for fun. Hopefully it's useful as open source, will fix-up CI over weekend and add a lock-free (not atomic free) allocator while I'm at it.
Depends. Right now they're in two different spaces. One still hanging on to graphics/gaming, the other coming from dense compute space. We'll see how long it takes them to converge.
It'll be called SPiDRE (pronounced spider) at MEMSYS. Name coined by a colleague, I can't take credit.
Thanks for reading! You'll have to wait for the presentation and follow-on papers for some of those answers :). If you read the Dark Bandwidth paper, there are some solutions mentioned there and in the presentation (http://www.jonathanbeard.io/slides/beard_hcpm2017.pdf) that could apply to what you suggest.
Seems very much "back to the future." Systolic array processors were used to accelerate neural networks in the 1980's. Great for matrix math too. (ref: http://repository.cmu.edu/cgi/viewcontent.cgi?article=2939&c...). These aren't quite the systolic array processor of old, but too close to be considered new arch/micro-arch. The formula is simple, have low precision MM to accelerate, drop in a matrix multiply unit that can be blocked for and high bandwidth memory to feed it and let it go. I'm waiting for more new takes on old arch....as fabbing chips becomes more economical, I hope to see more retro chips. Especially things that didn't quite make the jump from research to production b/c of scaling (or other reason), might now make sense.
Server/HPC, there are quite a few coming online. The Cavium presenter @goingarm listed (https://www.packet.net) on their slides, might be decent for getting a cloud instance to try out. There are many others. A good place to start finding them is: http://arm-hpc.gitlab.io.
A nice thing about ARM is that you get lots of different micro-arch under a single ISA. Once the ball gets rolling there will be many processor choices to choose from, which I think is a great thing (however, in full disclosure I'm one of the authors, and I work for ARM Research for our HPC program..although all statements above are my own opinion and not those of ARM).
Officially I think the only thing that can be said, has been said by Paul Messina: "I believe that the Aurora system contract is being reviewed for potential changes that would result in a subsequent system in a different time frame from the original Aurora system. But since that’s just early negotiations, I don’t think we can be anymore specific on that.” source: https://insidehpc.com/2017/06/told-aurora-morphing-novel-arc...
C++Now is branching out. I gave a talk this year on FIFO communications and a tutorial on RaftLib. One not so C++ focused and the other definitely C++ focused (C++ library). I enjoyed the wide variety of people and topics. Will try to go again next year.
In general, with current systems, it's all a partitioning problem. Speed-up is a trade-off between granularity of the compute kernel placed on a resource, the communication between said resources, latency at each level of the memory hierarchy (cache line utilization and reuse have a direct impact on this), and also (eek) protocol overheads between all the abstraction layers. Current systems are largely built around single CPU abstractions pasted on to make them multi-CPU abstractions. Working with accelerators, RDMA, etc. is quite a challenge both for the programmer manually making things work and for systems developers beating the abstractions into submission.
Many companies in the US are providing gyms for their employees, making staying in decent shape that much easier. I definitely take advantage. Even when I don't have time for a full workout, I can hop on treadmill for a 30 minute run and be back to work in 40 minutes (assuming showers aren't all occupied).
Systolic array/vector processors have been used since the late 60's for neural and other gemm heavy workloads. The Warp architecture from CMU is an excellent example.
TBB was in alpha at one point. So was OpenMP, MPI, ...
Just because you have an established library doesn't mean something else can't come along. Said another way, just because new doesn't mean to dismiss it.
<rant>
Do you have a quill pen? Hmm, didn't think so. If somebody came along and said: "Hey, look they both write, why use that fountain pen." Then we'd likely be writing with feathers today.
</rant>
Yup, it was quite a bit better on the upper end especially. Looking at the snoops on the bus using PAPI RaftLib does a better job at keeping the cache lines from bouncing.
The benchmarked version also has a dynamically resizing FIFO which uses utilization of the queue itself to guide the sizing. This means that the FIFO can better adapt to dynamic behavior found in most applications run on top of an operating system (most all these days outside of HPC). Looking at load stalls, the RaftLib version has fewer, but not quite enough to account for the results.
If you look at the single worker thread case, then jump to two threads..you can see a fairly big jump. RaftLib by definition is a pipelined programming system. The read file and compress are done perfectly in parallel. The bzip2 code doesn't quite pull it off in a perfectly pipelined fashion. It's close, but not quite. This results in less overlap of execution and communication. If I'd run on Linux (thread affinity on OS X is well, fun last time I checked..if not impossible to do manually), I'd also add thread affinity to the list which most people don't bother to optimize. Hot caches and synergistic cache accesses are quite beneficial.
Actually, the reason I didn't show distributed was in fact the naive original implementation of the TCP stack support. The platform minimizes data movement over network pipes, however due to time constraints on me finishing a PhD I chose to get the more interesting mathematical modeling aspects worked out rather than focus on engineering something that is pretty well understood (especially using techniques directly developed for HPC with MPI). Will push better support soon.
On the multi-process..yes, that's easy. I've removed it for the current main-line branch given the lack of demand. IfDef'ing the code made it much easier to proceed with getting it ready for alpha. The FIFO mechanisms are well tested using SHM and the forking code will be added back in soon. Another thing I commented out to get it working on multiple platforms is the NUMA placement code, now that I think hwloc will work on all platforms I'll get it added back in. Helps out on cross-socket communication quite a bit, as well as placing buffers closest to PCIe root for data transfer to accelerators.
In reality the data movement is no worse than any OpenMP or other parallel program. In as many places as we can, the data is left in place vs. pushed. Between nodes, it gets more fun...however it's still a rather well understood problem. Thanks again for the interest! I'll see if I can do a ShowHN before CPPNow 2017 for the beta release.
Hi! I'm the primary author and maintainer of the library. Thanks for the interjection. The intended application for RaftLib is to make something that will scale, just as you mention. I wrote this post a long time ago when I was just trying to get people interested in using it. It's a simple example that shows you can take many lines of standard parallel code, and write a much easier to read (smaller) version in a very short time that performs just as well or better than the manually managed parallel code.
A long time ago I was a biologist, then bioinformaticist. I wrote some code that would scale to a single node, and to a few dozen cores. In doing so, I realized how much I hated writing the same boilerplate code over and over again. TBB, c++11 threads, OpenMP, and MPI all basically have the same level of boilerplate and gotchas. I wanted to make something that was relatively easy to use and easily integrable with C/C++ code. Go was the only thing that came close, but it was brand new at the time.
It occurred to me while working on the AutoPipe system as a grad student that I could do something even better than a simple coordination language and at the same time subsume the functionality of a lot of parallel libraries. With stream/data-flow processing, I can do the exact same things I can do with OpenMP and MPI, but I can do more. The state encapsulation allows a whole host of cool optimizations, like identifying bottlenecks and duplicating actors dynamically (there's a whole host of reasons we'd be limited in OpenMP, c++11 threads). You can also compile an encapsulated function to another hardware platform entirely, or use high level synthesis tools to go to an FPGA (I'll be going there again soon too with RaftLib). The only thing that has to be constant across optimizations, is the connectivity of the DAG. By maintaining a port interface, just like you would hardware components (see Arvind's work from MIT...he's famous enough I just have to say Arvind :), we can compose really complicated applications. The port interface, it turns out, is also perfect for distributed compute.
Awhile back, I also had the realization that iostreams were perfect for this paradigm. Once you get your head around the concept, it seems quite natural. If it doesn't take off as a library, oh well. I enjoy working on it, and using it so I'll likely keep developing it in my spare time.
In the interim, I'll get back to exascale hardware stuffs :).