SPDK will be able to fully saturate the PCIe bandwidth from a single CPU core here (no secret 6 threads inside the kernel). The drives are your bottleneck so it won't go faster, but it can use a lot less CPU.
But with SPDK you'll be talking to the disk, not to files. If you changed io_uring to read from the disk directly with O_DIRECT, you wouldn't have those extra 6 threads either. SPDK would still be considerably more CPU efficient but not 6x.
DDIO is a pure hardware feature. Software doesn't need to do anything to support it.
For an expanding array in a 64 bit address space, reserving a big region and mmaping it in as you go is usually the top performing solution by a wide margin. At least on Linux, it is faster to speculatively mmap ahead with MAP_POPULATE rather than relying on page faults, too.
And, if you find you didn't reserve enough address space, Linux has mremap() which can grow the reserved region. Or map the region to two places at once (the original place and a new, larger place).
Compared to libraries like bgfx and sokol at least, I think there are two key differences.
1) SDL_gpu is a pure C library, heavily focused on extreme portability and no depedencies. And somehow it's also an order of magnitude less code than the other options. Or at least this is a difference from bgfx, maybe not so much sokol_gfx.
2) The SDL_gpu approach is a bit lower level. It exposes primitives like command buffers directly to your application (so you can more easily reason about multi-threading), and your software allocates transfer buffers, fills them with data, and kicks off a transfer to GPU memory explicitly rather than this happening behind the scenes. It also spawns no threads - it only takes action in response to function calls. It does take care of hard things such as getting barriers right, and provides the GPU memory allocator, so it is still substantially easier to use than something like Vulkan. But in SDL_gpu it is extremely obvious to see the data movements between CPU and GPU (and memory copies within the CPU), and to observe the asynchronous nature of the GPU work. I suspect the end result of this will be that people write far more efficient renderers on top of SDL_gpu than they would have on other APIs.
The eBPF programs are strictly bounded. And they're scoped to their own memory that you have to pre-load from the actual storage with separate commands issued from the CPU (presumably from the kernel driver which is doing access control checks). It's no different than uploading a shader to a GPU. You can burn resources but that's about the extent of the damage you can cause.
We tried to standardize exactly this - eBPF programs offloaded onto the device. The NVMe standard now has a lot of infrastructure for this standardized, including commands to discover device memory topology, transfer to/from that memory, and discover and upload programs. But one of the blockers is that eBPF isn't itself standardized. The other blockers are vendors ready and willing to build these devices and customers ready to buy them in volume. The extra compute ability will introduce some extra cost.
Yes, I've seen some clearer cases made for networking.
In networking there is no standard for the hardware interface. Every vendor does their own thing. Except many can at least handle virtqueues carrying virtio-net messages for the data path, so some framework like vDPA may make sense (I'd prefer to see a full NIC interface standard emerge instead).
In storage, however, the industry has agreed on NVMe. This is a full standard for control and data plane. All storage products on the market, including DPUs and SmartNICs, just present NVMe devices. So there's no case to be made for vDPA at all. It just isn't necessary.
I don't get it either, and I'm a maintainer of SPDK which provides multiple implementations of virtualized devices and is frequently used inside DPUs to present storage devices.
If I'm implementing a hardware device anyway, why would I not just use NVMe as the interface? NVMe is superior to virtio-blk in every way that I can think of.
Even for a software device in userspace, why not use a technology like vfio-user to present an NVMe device, or just use vhost-user to present the virtio-blk device?
I've never really been able to get a clear value proposition for vDPA for storage laid out for me. Maybe I'm missing something critical - it's certainly possible.
There's also now an open source recreation of the original client, all written in C# that actually uses a GPU so it renders 4k at 250fps instead of 800x600 at 12.5fps. It's a very mature and stable reproduction at this point.
But with SPDK you'll be talking to the disk, not to files. If you changed io_uring to read from the disk directly with O_DIRECT, you wouldn't have those extra 6 threads either. SPDK would still be considerably more CPU efficient but not 6x.
DDIO is a pure hardware feature. Software doesn't need to do anything to support it.
Source: SPDK co-creator