SMB Direct – SMB3 over RDMA(docs.kernel.org)
docs.kernel.org
SMB Direct – SMB3 over RDMA
https://docs.kernel.org/filesystems/smb/smbdirect.html
20 comments
What is the performance impact of soft RDMA over SMB this way, vs the traditional SMB on the IP stack?
is there some open source product who can leverage this or this just assume you have to use Microsoft stuff?
The SMB Direct support mentioned here is in the kernel for client & server from 5.15+. After that it's just a mount point any application can access. No Microsoft stuff needed on either side.
This was probably added by/for Tuxera to increase Tuxera Fusion SMB performance.
I think it got added by Samsung / someone employed by Samsung at the time https://www.phoronix.com/news/KSMBD-Lands-In-Linux-5.15
I was sure SMB3 was Super Mario Bros 3.
The verbs used in RDMA are Turing complete [0] [1] . They don't seem to be all be accessible from SMB3/SMB Direct, and the SMB opcodes themselves seem lacking, but with enough effort maybe you can get access or add what you need and this would form a path to finally have the ultimate Super Mario Brothers 3 Remote Direct Memory Access tech.
[0] https://www.usenix.org/conference/nsdi22/presentation/reda
[1] https://www.youtube.com/watch?v=sTT_XPfYudg
[0] https://www.usenix.org/conference/nsdi22/presentation/reda
[1] https://www.youtube.com/watch?v=sTT_XPfYudg
What, um... Are... Are people using samba to sync model weights between cluster nodes...?
Why not? SMB is no slouch. Microsoft has taken network storage performance very seriously for a long time now. Back in the day, Microsoft and others (NetApp, for instance,) worked hard to extend and optimize SMB and deliver efficient, high throughput file servers. I haven't kept up with the state of the art recently, but I know there have been long stretches where SMB consistently led the field in benchmark testing. It also doesn't hurt that Microsoft has a lot of pull with hardware manufacturers to see their native protocols remain tier 1 concerns at all times.
I think a lot of people have a hard time differentiating the underlying systems from what they _see_ and use it to bash MS products.
I heard that it was perhaps recently fixed, but copying many small files was multiple times faster to do via something like Total Commander vs the built in File Explorer (large files goes equally fast).
People seeing how slow Explorer was to copy would probably presume that it was a lower level Windows issue if they had a predisposed bias against Microsoft/Windows.
My theory about Explorers sluggishness is that they added visual feedback to the copying process at some point, and for whatever reason that visual feedback is synchronous/slow (perhaps capped at the framerate, thus 60 files a second), whilst TC does updating in the background and just renderers status periodically whilst the copying thread(s) can run at full speed of what the OS is capable of under the hood.
I heard that it was perhaps recently fixed, but copying many small files was multiple times faster to do via something like Total Commander vs the built in File Explorer (large files goes equally fast).
People seeing how slow Explorer was to copy would probably presume that it was a lower level Windows issue if they had a predisposed bias against Microsoft/Windows.
My theory about Explorers sluggishness is that they added visual feedback to the copying process at some point, and for whatever reason that visual feedback is synchronous/slow (perhaps capped at the framerate, thus 60 files a second), whilst TC does updating in the background and just renderers status periodically whilst the copying thread(s) can run at full speed of what the OS is capable of under the hood.
I dunno about Windows Explorer, but macOS’ finder seems to hash completed transfers over SMB (this must be something it can trigger the receiver to do in SMB itself, it doesn’t seem slow enough for the sender to be doing it on a remote file) and remove transferred files that don’t pass the check.
I could see that or other safety checks making one program slower than another that doesn’t bother. Or that sort of thing being an opportunity for a poor implementation that slows everything down a bunch.
I could see that or other safety checks making one program slower than another that doesn’t bother. Or that sort of thing being an opportunity for a poor implementation that slows everything down a bunch.
A problem with Explorer, that it also shares with macOS Finder[1], is that they are very much legacy applications with features piled on top, and Explorer was never expected to be used for heavy I/O work and tends to do things the slower way possible, including doing things in ways that are optimized for "random first time user of windows 95 who will have maybe 50 files in a folder"
[1] Finder has parts that show continued use of code written for MacOS 9 :V
[1] Finder has parts that show continued use of code written for MacOS 9 :V
This blows my mind. $400B in annual revenue and they can't spare the few parts per million it would take to spruice up the foundation of their user experience.
This is speculation based on external observation, nothing internal other than rumours:
A big, increasing over last decade, chunk of that is fear that they will break the compatibility - or otherwise drop in shared knowledge. To the point that the more critical parts the less anyone wants to touch them (heard that ntfs.sys is essentially untouchable these days, for example).
And various rules that used to be sacrosanct are no longer followed, like the "main" branch of Windows source repository having to always build cleanly every night (fun thing - Microsoft is one of the origins of nightly builds as a practice)
A big, increasing over last decade, chunk of that is fear that they will break the compatibility - or otherwise drop in shared knowledge. To the point that the more critical parts the less anyone wants to touch them (heard that ntfs.sys is essentially untouchable these days, for example).
And various rules that used to be sacrosanct are no longer followed, like the "main" branch of Windows source repository having to always build cleanly every night (fun thing - Microsoft is one of the origins of nightly builds as a practice)
It's probably a vicious cycle.
Less people are trusted to touch ntfs.sys due to lack of experience, thus they never gain it and that in turn means less work and in turn means even less people have proved themselves trustworthy enough to work on it.
Until nobody remains in the company that is trusted enough.
Less people are trusted to touch ntfs.sys due to lack of experience, thus they never gain it and that in turn means less work and in turn means even less people have proved themselves trustworthy enough to work on it.
Until nobody remains in the company that is trusted enough.
> to bash MS products.
Microsoft gives them a lot of ammo. While, as I said, Microsoft et al. have seen that SMB is indeed efficient, at the same time security has been neglected to the point of being farcical. You can see this in headlines as recent as last week: Microsoft is only now, in 2025, deprecating RC4 authentication, and this includes SMB.
So while one might leverage SMB for high throughput file service, it has always been the case that you can't take any exposure for granted: if it's not locked down by network policies and you don't regularly ensure all the knobs and switches are tweaked just so, it's an open wound, vulnerable to anything that can touch an endpoint or sniff a packet.
Microsoft gives them a lot of ammo. While, as I said, Microsoft et al. have seen that SMB is indeed efficient, at the same time security has been neglected to the point of being farcical. You can see this in headlines as recent as last week: Microsoft is only now, in 2025, deprecating RC4 authentication, and this includes SMB.
So while one might leverage SMB for high throughput file service, it has always been the case that you can't take any exposure for granted: if it's not locked down by network policies and you don't regularly ensure all the knobs and switches are tweaked just so, it's an open wound, vulnerable to anything that can touch an endpoint or sniff a packet.
Agreed, but that used to be the difference between MS and Google.
MS would bend backwards to make sure those enterprise Windows 0.24 boxes will still be able to connect to networks because those run some 16bit drivers for CNC machines.
Meanwhile Google decided to kill a product the second whoever introduced it on stage walked off it.
Azure is a money-maked for MS, and wouldn't be so without those weird legacy enterprise deployments. The big question is if continuing to increase a posture about about security together with an "cloud" focus is actually in their best interest or if retaining those legacy enterprises would have been smarter.
MS would bend backwards to make sure those enterprise Windows 0.24 boxes will still be able to connect to networks because those run some 16bit drivers for CNC machines.
Meanwhile Google decided to kill a product the second whoever introduced it on stage walked off it.
Azure is a money-maked for MS, and wouldn't be so without those weird legacy enterprise deployments. The big question is if continuing to increase a posture about about security together with an "cloud" focus is actually in their best interest or if retaining those legacy enterprises would have been smarter.
Plenty of other workloads that benefit from high performance file access and with networks speeds and disk speeds getting higher whilst single-core perf has more or less plateaued in comparison, it's thus more and more important to support data-paths where the kernel switching won't become a bottleneck.
Dunno but I have used samba to load model weights from my NAS
But with Apple's recent introduction of RDMA over Thunderbolt, that got my hopes up I could use it for storage, not only moving LLMs, but also for video file storage, where editing from one Mac to another (or over Ethernet, if that's supported) could be much faster, with lower latency.