MK1 Flywheel Unlocks the Full Potential of AMD Instinct for LLM Inference(mkone.ai)
mkone.ai
MK1 Flywheel Unlocks the Full Potential of AMD Instinct for LLM Inference
https://mkone.ai/blog/mk1-flywheel-amd
27 comments
This is a good observation, the cards do have different memory bandwidth with the MI210 having more than double the bandwidth via HBM2e.
Note that the comparisons between the two cards (MI210 and A6000) are being made for high throughput workloads, and in this regime the performance is compute bound. So as long as the memory bandwidth is decent (as it is for GDDR6 with 768.0 GB/s) the lower memory bandwidth is not the main bottleneck. There are also other architectural differences so any comparison will ultimately be imperfect, but we found A6000 to be the closest match for the workloads that we cared about (ie high throughput workloads at reasonable latencies).
Also worth noting that there are still more stack optimizations on the table, which again can shift the bottleneck between compute and memory. In those cases it might makes sense to compare with another card with matched memory bandwidth.
paul @ mk1
Note that the comparisons between the two cards (MI210 and A6000) are being made for high throughput workloads, and in this regime the performance is compute bound. So as long as the memory bandwidth is decent (as it is for GDDR6 with 768.0 GB/s) the lower memory bandwidth is not the main bottleneck. There are also other architectural differences so any comparison will ultimately be imperfect, but we found A6000 to be the closest match for the workloads that we cared about (ie high throughput workloads at reasonable latencies).
Also worth noting that there are still more stack optimizations on the table, which again can shift the bottleneck between compute and memory. In those cases it might makes sense to compare with another card with matched memory bandwidth.
paul @ mk1
Considering how well their last post turned out[0] (read the comments), forgive me for not having much confidence in what this company claims. I'll need independent testing.
[0] https://news.ycombinator.com/item?id=37016413
[0] https://news.ycombinator.com/item?id=37016413
You can try it yourself on SageMaker using NVIDIA. There's a free trial.
https://aws.amazon.com/marketplace/seller-profile?id=seller-...
For AMD, you'll have to wait until these cards become available on the cloud.
https://aws.amazon.com/marketplace/seller-profile?id=seller-...
For AMD, you'll have to wait until these cards become available on the cloud.
It will be fascinating to see if AMD's open source ML platform beats Nvidia's closed one. (Usually, the more open platform becomes more popular.)
Unlikely; that is almost in no-one's interests. The story down that path is ROCm gets 95% of the way there feature wise, then Nvidia catches a sudden case of community spirit and opens up CUDA.
Although that is mostly wishful thinking. I expect ROCm probably works quite well on the server, the issues in my experience is that it assumes that the AMD card is dedicated to processing and once it tries to run graphics and compute at the same time it becomes Too Hard to manage memory for both and then the driver locks up. So their buggy drivers basically kill any sort of hobbyist or researcher effort unless there is a dedicated supercomputer on hand. Much better plan to go with Nvidia I assume.
Although that is mostly wishful thinking. I expect ROCm probably works quite well on the server, the issues in my experience is that it assumes that the AMD card is dedicated to processing and once it tries to run graphics and compute at the same time it becomes Too Hard to manage memory for both and then the driver locks up. So their buggy drivers basically kill any sort of hobbyist or researcher effort unless there is a dedicated supercomputer on hand. Much better plan to go with Nvidia I assume.
Not really, CUDA and DirectX are more popular than Khronos stuff because of tooling, SDKs and overall developer experience.
Simple stuff that matters more for most folks outside FOSS circles.
It is up to Intel and AMD to provide similar experiences.
Simple stuff that matters more for most folks outside FOSS circles.
It is up to Intel and AMD to provide similar experiences.
Isn't this an unfair comparison, given they test the A6000 Ampere (half the price of MI210) instead of the Ada generation (still a cheaper than the MI210, but ~2x the CUDA cores as Ampere)?
Comparisons between different chip architectures are imperfect. In our opinion the most fair thing to do is 1) match the TFLOPs (since these workloads are compute bound), and 2) find a similar card that can run the same size models. Since MI210 has 64GB, the closest one is the A6000 with 48GB. Many of the less expensive ADA cards can't even fit a 13B model in VRAM, so a comparison could not even be made.
A little off-topic, but I am rather disappointed by software taking cool stuff names. As soon as I looked at the title, my first thought went to an actual flywheel (A mechanical power storage system). A flywheel in space powering some unnatural energy beam is a Vader-esque evil-cool-evil system I frequently imagine.
Is it open source? What is MK1?
Can I try this on AWS?
Do they have any instances with AMD GPUs?
Looks like good performance on AMD and NVIDIA, nice job!
Doubt. Especially after the last announcement. Probably again optimised for the benchmark the green and low comment fanboys are also a bit sus.
impressive
I support any progress to erode the Nvidia monopoly.
That said from what I'm seeing here the free and open source (less other aspects of the CUDA stack, of course) TensorRT-LLM[0] almost certainly bests this implementation using the Nvidia hardware they reference for comparison. Compare to real (datacenter) Nvidia GPUs that aren't three years old and prepare to get your hair blown back.
I don't have an A6000 but as an example with the tensorrt_llm backend for Nvidia Triton Inference Server (also free and open source) I get roughly 30 req/s with Mistral 7B on my RTX 4090 with significantly lower latency and I'm in the early stages of tuning. Comparison benchmarks are tough, especially when published benchmarks like these are fairly scant on the real details.
TensorRT-LLM has only been public for a few months and if you peruse the docs, PRs, etc you'll see they have many more optimizations in the works.
In typical Nvidia fashion TensorRT-LLM runs on any Nvidia GPU (from laptop to datacenter) going back to Turing (five year old cards) assuming you have the VRAM. It even works on their Jetson line of hardware.
You can download and run this today, free and "open source" for these implementations at least. I'm extremely skeptical of the claim "MK1 Flywheel has the Best Throughput and Latency for LLM Inference on NVIDIA". You'll note they compare to vLLM, which is an excellent and incredible project but if you look at vLLM vs Triton w/ TensorRT-LLM the performance improvements are dramatic.
Of course it's the latest and greatest ($$$$$$ and unobtanium) but one look at H100/H200 performance[3] and you can see what happens when the vendor has a robust software ecosystem to help sell their hardware. Pay the Nvidia tax on the frontend for the hardware, get it back and then some as a dividend via the software especially when anything close (assuming this even is) is another paid product/SaaS/whatever their monetization strategy is.
At the risk of this turning into an Nvidia sales pitch Triton will do the same thing for absolutely any model via the ONNX, TensorRT, Pytorch, Tensorflow, OpenVINO, etc backends.
I have an implementation generating embeddings via bge-large-v1.5 that's also the fastest thing out there. Same for Whisper, vision models, whatever you want.
I feel like MK1 must be aware of TensorRT-LLM/Triton but of course those comparison benchmarks won't help sell their startup.
[0] - https://github.com/NVIDIA/TensorRT-LLM
[1] - https://github.com/triton-inference-server/tensorrtllm_backe...
[2] - https://mkone.ai/blog/mk1-flywheel-race-tuned-and-track-read...
[3] - https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source...
That said from what I'm seeing here the free and open source (less other aspects of the CUDA stack, of course) TensorRT-LLM[0] almost certainly bests this implementation using the Nvidia hardware they reference for comparison. Compare to real (datacenter) Nvidia GPUs that aren't three years old and prepare to get your hair blown back.
I don't have an A6000 but as an example with the tensorrt_llm backend for Nvidia Triton Inference Server (also free and open source) I get roughly 30 req/s with Mistral 7B on my RTX 4090 with significantly lower latency and I'm in the early stages of tuning. Comparison benchmarks are tough, especially when published benchmarks like these are fairly scant on the real details.
TensorRT-LLM has only been public for a few months and if you peruse the docs, PRs, etc you'll see they have many more optimizations in the works.
In typical Nvidia fashion TensorRT-LLM runs on any Nvidia GPU (from laptop to datacenter) going back to Turing (five year old cards) assuming you have the VRAM. It even works on their Jetson line of hardware.
You can download and run this today, free and "open source" for these implementations at least. I'm extremely skeptical of the claim "MK1 Flywheel has the Best Throughput and Latency for LLM Inference on NVIDIA". You'll note they compare to vLLM, which is an excellent and incredible project but if you look at vLLM vs Triton w/ TensorRT-LLM the performance improvements are dramatic.
Of course it's the latest and greatest ($$$$$$ and unobtanium) but one look at H100/H200 performance[3] and you can see what happens when the vendor has a robust software ecosystem to help sell their hardware. Pay the Nvidia tax on the frontend for the hardware, get it back and then some as a dividend via the software especially when anything close (assuming this even is) is another paid product/SaaS/whatever their monetization strategy is.
At the risk of this turning into an Nvidia sales pitch Triton will do the same thing for absolutely any model via the ONNX, TensorRT, Pytorch, Tensorflow, OpenVINO, etc backends.
I have an implementation generating embeddings via bge-large-v1.5 that's also the fastest thing out there. Same for Whisper, vision models, whatever you want.
I feel like MK1 must be aware of TensorRT-LLM/Triton but of course those comparison benchmarks won't help sell their startup.
[0] - https://github.com/NVIDIA/TensorRT-LLM
[1] - https://github.com/triton-inference-server/tensorrtllm_backe...
[2] - https://mkone.ai/blog/mk1-flywheel-race-tuned-and-track-read...
[3] - https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source...
I've said it once, I'll say it again; the evaporation of support for OpenCL is why CUDA will never die. We already identified the CUDA GPGPU compute gap a decade ago, but stakeholders in killing CUDA abandoned it so they could enforce their own proprietary API ecosystems. Now both Apple and AMD are begging the community to fill in the gap while researchers continue to favor the simplest solution (CUDA). There was an opportunity for Apple, AMD and system integrators to hold Nvidia accountable for their proprietary shenanigans; but it turned into Reservoir Dogs before Nvidia even knew shots were fired.
Apple and Microsoft's inability to work together was exploited to comical extremes, and now Nvidia is laughing their way to the bank with a 10+ year lead on software investment. Even if we're training models on Windows and MacOS in 2028, nobody will have actually solved the HPC problem. The "Linux moment" for general-purpose GPU number crunching may never arrive, certainly not without support from major OSes.
Apple and Microsoft's inability to work together was exploited to comical extremes, and now Nvidia is laughing their way to the bank with a 10+ year lead on software investment. Even if we're training models on Windows and MacOS in 2028, nobody will have actually solved the HPC problem. The "Linux moment" for general-purpose GPU number crunching may never arrive, certainly not without support from major OSes.
Nice!!!
This is sick!
Please stop manipulating HN frontpage for publicity.
They definitely are. Weird one word comments on a closed source product that is a wrapper of GGML, trying to cash in on AI hype. Even if we don't have a problem with your grift, please don't clog up this space with your nonsense.
[1] https://www.techpowerup.com/gpu-specs/radeon-instinct-mi210.... [2] https://www.techpowerup.com/gpu-specs/rtx-a6000.c3686