LLM inference throughput benchmark for RTX PRO 6000 SE vs H100, H200, and B200 GPUs, based on the vllm serve and vllm bench serve benchmarking tools, to understand the cost-efficiency of various datacenter GPU options.
Benchmarking Setup:
The benchmark is optimized for throughput. VLLM serves models. The model is split across multiple GPUs using the --tensor-parallel-size VLLM option, if needed. Multiple VLLM instances serve the model; an NGINX load balancer on top distributes requests across them, maximizing throughput (replica parallelism). For example, if only 4 GPUs are required to run the model on an 8-GPU machine, two VLLM instances are launched with --tensor-parallel-size=4, and an NGINX load balancer is used. If all eight GPUs are required, then a single VLLM instance with --tensor-parallel-size=8 is used.
The vllm bench serve tool is used for benchmarking with random data and a sequence length of 1000. The number of concurrent requests is set to 64-256 to ensure the LLM's token-generation capacity is saturated.
Three models are benchmarked to better understand the effect of PCIe communication on the 8xPro6000 server vs. NVLink on the H100/H200/B200.
Here is the model selection and the logic behind it:
- GLM-4.5-Air-AWQ-4bit (fits 80GB). Testing single-GPU performance and maximum throughput with replica scaling on 8 GPU setups. No PCIE bottleneck.
- Qwen3-Coder-480B-A35B-Instruct-AWQ (fits 320GB). This 4-bit-quantized model fits into 4 GPUs. Some PCIe communication overhead in Pro 6000 setups may reduce performance relative to NVLink-enabled datacenter GPUs.
- GLM-4.6-FP8 (fits 640GB). This model requires all eight GPUs. PCIe communication overhead expected. The H100 and H200 configurations should have an advantage.
Besides raw throughput, graphs show the serving cost per million tokens for each model on its respective hardware. The rental price is set at $0.93 for Pro6000, $1.91 for H100, $2.06 for H200, and $2.68 for B200.
Results:
- B200 wins on throughput, with the largest gap on the most communication-heavy workload – GLM-4.6-FP8 (8-way TP): B200 is 4.87x faster than PRO 6000 (8,036.71 vs 1,651.67 tok/s) – Qwen3-Coder-480B (4-way TP): B200 is 4.02x faster than PRO 6000 (6,438.43 vs 1,602.96 tok/s) – GLM-4.5-Air (single-GPU replicas): B200 is 4.22x faster than PRO 6000 (9,675.24 vs 2,290.69 tok/s)
- B200 is also the cost efficiency leader under updated run-cost estimates. B200’s throughput advantage more than compensates for its higher hourly cost.
- PRO 6000 is an attractive low-capex option. It beats H100 on cost per across all models and is on par with H200 on GLM-4.5-Air.
- H200 is a major step up over H100. H200 delivers ~1.83x to 2.14x H100 throughput across the three models.
- H100 looked worse than expected in this specific setup. It’s on par with PRO 6000 in throughput on GLM-4.5-Air and behind all other contenders in cost per token across all workloads.
I spent ten years inside three Big Tech companies, most of it at Apple. For a long time, I thought of corporations as ruthless business machines — designed to maximize revenue, or shareholder value, depending on my level of cynicism at the time.
What I didn’t expect was how quickly business logic faded once you moved deep enough inside the organization.
Past a certain point, the environment stopped resembling a company and started resembling an imperial court. There were enclaves, wars, and palace intrigue. Fair and unfair rulers. And for the commonfolk, little left to do but fight someone else’s wars.
In one story from Apple, I spent over a year fighting for what I believed were straightforward business values.
Over my ten-year tenure in Big Tech, I’ve witnessed conflicts that drove exceptional people out, hollowed out entire teams, and hardened rifts between massive organizations long after any business rationale, if there ever was one, had faded.
The conflicts I explore here are not about strategy, conflicts of interest, misaligned incentives, or structural failures. Nor are they about money, power, or other familiar human vices.
They are about identity. We shape and reinforce it over a lifetime. It becomes our strongest armor — and, just as often, our hardest cage.
My two previous reddits in the Tech Bro Saga series:
- Why Big Tech Turns Everything Into a Knife Fight - a noir-toned piece on how pressure, ambiguity, and internal competition turn routine decisions into zero-sum battles (https://medium.com/data-science-collective/why-big-tech-turn...)
- Big Tech Performance Review: How to Gaslight Employees at Scale - a sardonic look at why formal review systems often substitute process for real leadership and honest feedback (https://medium.com/data-science-collective/big-tech-performa...)
No prescriptions or grand theory. Just an attempt to give structure to a feeling many of us recognize but rarely articulate.
No matter how they’re designed—manager discretion, calibration committees, or opaque algorithms—performance reviews in big tech reliably produce results that are neither meritocratic nor humane. In practice, compensation and promotions still hinge on a single decision-maker.
I wrote a dark, deliberately cynical essay comparing Apple and Roblox, two companies where I managed teams, that tried very different approaches to performance evaluation and failed in different ways.
Even if we could make these systems “fair,” I’m not convinced that’s the right goal. What people actually want isn’t better algorithms, but humane treatment and rational judgment when it matters.
I haven’t worked at Google, but I doubt they’re immune.
Lack of “important work” is one mechanism, but in the case I’m describing the issue was overlapping legitimacy: multiple orgs had good reasons to own the same scope. Hardware felt insecure about ML moving elsewhere, SWE had real product needs, AI/ML wanted centralization.
I present an LLM inference throughput benchmark for RTX4090 / RTX5090 / PRO6000 GPUs based on vllm serving and vllm bench serve client benchmarking tool.
The hardware configurations used:
- 1x4090, 2x4090, 4x4090
- 1x5090; 2x5090; 4x5090
- 1x6000
All machines have at least 50GB of RAM per GPU with a minimum of 7 cores. The 4090 machines utilize the EPYC Milan (3rd Gen) processor, while the 5090/6000 models employ the EPYC Genoa (4th Gen) processor, resulting in slightly faster overall performance.
I have optimized the benchmark setup for throughput. VLLM serves models. The model is split across multiple GPUs using the --pipeline-parallel-size VLLM option, if needed. I run as many VLLM instances as possible, using an NGINX load balancer on top to distribute requests across them and maximize throughput (replica parallelism). For example, if only two GPUs are required to run the model on a 4-GPU machine, I run two VLLM instances with --pipeline-parallel-size=2 and an NGINX load balancer. If all four GPUs are required, then a single VLLM instance with --pipeline-parallel-size=4 is used.
The vllm bench serve tool is used for benchmarking with random data and a sequence length of 1000. The number of concurrent requests is set to 400 to ensure saturation of the LLM token generation capacity.
I have benchmarked three different models to understand better the effect of PCIe communication on the final LLM performance. I have tried to find the largest modern model that fits into a single 4090, two 4090s, and four 4090s. It would be possible to fit larger GGUF models, but VLLM poorly supports GGUF, and I wanted to use VLLM because it is optimized for high-throughput serving.
Here is the model selection and the logic behind it:
Qwen3-Coder-30B-A3B-Instruct-AWQ (fits 24GB). This 4-bit quantized model fits into a single RTX4090. Thus, scaling the number of GPUs yields a linear scale in throughput, so 4 x 4090 and 4 x 5090 configurations should have an edge as they have more raw compute power.
Meta-Llama-3.3-70B-Instruct-AWQ-INT4 (fits 48GB). This 4-bit quantized model fits into 2 x 4090. Some communication over PCIe can lower the performance of multi-GPU setups.
GLM-4.5-Air-AWQ-4bit (fits 96GB). This model requires all four 4090s, so PCIE communication will likely be a bottleneck, and Pro 6000 should have an edge.
Besides raw throughput, graphs contain the serving cost per million tokens for the respective model on the respective hardware. The rental price is set to $0.39 per hour for 4090, $0.65 for 5090, and $1.29 for Pro 6000. These prices are typical for GPU rentals at neuralrack.ai, which provided the hardware for this benchmark. You can adjust the GPU price in the config.yml file in the benchmark repository and invoke make report to generate a new report that better reflects your situation.
Results
The overall winner is RTX PRO 6000 for its consistent performance across all model sizes and best cost-efficiency for larger models. However, if your workload primarily involves smaller models, the multi-GPU RTX 5090 can offer better absolute throughput at a lower cost.
Small Models (fits 24GB): Multi-GPU consumer configurations offer the best value due to replica parallelism, but RTX PRO 6000 is very close.
Medium Models (fits 48GB): RTX 5090 configuration provides the best balance of performance and cost, followed by RTX PRO 6000.
Large Models (fits 96GB): RTX PRO 6000 emerges as the clear winner despite its higher hourly cost, thanks to the elimination of PCIe overhead.
We explored a network-attached KV-cache for consumer GPUs to offset their limited VRAM. It doesn’t make RTX cards run giant models efficiently. Still, for workloads that repeatedly reuse lengthy prefixes—such as chatbots, coding assistants, and multi-turn threads—it delivers a 2–4× speedup in RPS and time-to-first-token on 7B and 70B models.
How it works:
On return visits, instead of re-running the prompt through the model, we fetch previously computed KV blocks from network storage and skip re-computing those tokens (i.e., we avoid re-running prefill on repeated prefixes). This is helpful when VRAM can’t hold all sessions, and users pause between messages, which is almost always the case.
Why RTX benefits:
Prefill is the computationally intensive part (quadratic attention, numerous reductions, and inter-GPU traffic). Without NVLink, PCIe becomes the choke point in multi-GPU setups. KV-caching cuts repeated prefill, leaving mostly the lighter decoding step—something PCIe-only RTX nodes handle well.
Technical Notes:
- Works with consumer and data-center GPUs. In theory, you can even split roles: NVLink boxes do prefill, while cheaper RTX pods serve as decoders using stored KV.
- We use special hardware to reduce fetch overhead and offload the CPU, but you can reproduce this at home with a regular NAS (with lower peak performance).
- For a more in-depth walkthrough of the math and architecture of a KV-cache solution, please watch this video from the KV-cache solution vendor (https://www.youtube.com/watch?si=T69vxku8xPr6p7I0&v=CV4FYMTF...)
I am working on a platform for GPU rental and have recently encountered an extremely annoying issue.
On all machines with RTX 5090 and RTX PRO 6000 GPUs, the cards occasionally become completely unresponsive — usually after a few days of VM usage or at seemingly random times during startup/shutdown. Once it happens, the GPU can’t be reassigned. GPU is in a limbo state and doesn't respond to FLR. The only way out is a complete node reboot, which is undesirable, as it will stop VMs that are already running on the node.
H100s, B200s, and older RTX 4090s are solid, but these newer RTX cards are a menace. I understand that RTX cards are not designed for virtualization, and NVIDIA likely doesn't care; however, those cards are very well-suited for a variety of applications, and it would be nice to make virtualization work.
Is there a way to recover the GPU from this state without a complete node reboot?
We've put a $ 1,000 bounty on it if anyone is interested in helping. We also have an open position for an engineer who can help with problems like this.
Benchmarking Setup: The benchmark is optimized for throughput. VLLM serves models. The model is split across multiple GPUs using the --tensor-parallel-size VLLM option, if needed. Multiple VLLM instances serve the model; an NGINX load balancer on top distributes requests across them, maximizing throughput (replica parallelism). For example, if only 4 GPUs are required to run the model on an 8-GPU machine, two VLLM instances are launched with --tensor-parallel-size=4, and an NGINX load balancer is used. If all eight GPUs are required, then a single VLLM instance with --tensor-parallel-size=8 is used.
The vllm bench serve tool is used for benchmarking with random data and a sequence length of 1000. The number of concurrent requests is set to 64-256 to ensure the LLM's token-generation capacity is saturated.
Three models are benchmarked to better understand the effect of PCIe communication on the 8xPro6000 server vs. NVLink on the H100/H200/B200.
Here is the model selection and the logic behind it: - GLM-4.5-Air-AWQ-4bit (fits 80GB). Testing single-GPU performance and maximum throughput with replica scaling on 8 GPU setups. No PCIE bottleneck. - Qwen3-Coder-480B-A35B-Instruct-AWQ (fits 320GB). This 4-bit-quantized model fits into 4 GPUs. Some PCIe communication overhead in Pro 6000 setups may reduce performance relative to NVLink-enabled datacenter GPUs. - GLM-4.6-FP8 (fits 640GB). This model requires all eight GPUs. PCIe communication overhead expected. The H100 and H200 configurations should have an advantage.
Besides raw throughput, graphs show the serving cost per million tokens for each model on its respective hardware. The rental price is set at $0.93 for Pro6000, $1.91 for H100, $2.06 for H200, and $2.68 for B200.
Results: - B200 wins on throughput, with the largest gap on the most communication-heavy workload – GLM-4.6-FP8 (8-way TP): B200 is 4.87x faster than PRO 6000 (8,036.71 vs 1,651.67 tok/s) – Qwen3-Coder-480B (4-way TP): B200 is 4.02x faster than PRO 6000 (6,438.43 vs 1,602.96 tok/s) – GLM-4.5-Air (single-GPU replicas): B200 is 4.22x faster than PRO 6000 (9,675.24 vs 2,290.69 tok/s) - B200 is also the cost efficiency leader under updated run-cost estimates. B200’s throughput advantage more than compensates for its higher hourly cost. - PRO 6000 is an attractive low-capex option. It beats H100 on cost per across all models and is on par with H200 on GLM-4.5-Air. - H200 is a major step up over H100. H200 delivers ~1.83x to 2.14x H100 throughput across the three models. - H100 looked worse than expected in this specific setup. It’s on par with PRO 6000 in throughput on GLM-4.5-Air and behind all other contenders in cost per token across all workloads.