Vboxsf fixes for 5.14-1(lore.kernel.org)
lore.kernel.org
Vboxsf fixes for 5.14-1
https://lore.kernel.org/lkml/CAHk-=whfeq9gyPWK3yao6cCj7LKeU3vQEDGJ3rKDdcaPNVMQzQ@mail.gmail.com/
144 comments
> NTFS is the de facto filesystem for portable data storage
I thought it was FAT32. Do Macs support NTFS?
EDIT: It seems they don't out of the box: <https://www.howtogeek.com/236055/HOW-TO-WRITE-TO-NTFS-DRIVES...>
EDIT 2: Since many comments replied with the same thing, I'll reply here: I wouldn't consider read-only support as support for portable data storage, since that allows you only to send data in one direction. When you want to copy a file from a Mac to a Windows computer, NTFS is not a solution now (out of the box at least), therefore falls pretty short wrt "portable".
I thought it was FAT32. Do Macs support NTFS?
EDIT: It seems they don't out of the box: <https://www.howtogeek.com/236055/HOW-TO-WRITE-TO-NTFS-DRIVES...>
EDIT 2: Since many comments replied with the same thing, I'll reply here: I wouldn't consider read-only support as support for portable data storage, since that allows you only to send data in one direction. When you want to copy a file from a Mac to a Windows computer, NTFS is not a solution now (out of the box at least), therefore falls pretty short wrt "portable".
exFAT is probably better in terms of cross platform compatibility and addresses a lot of the limitations of FAT while remaining portable.
Since Snow Leopard OS X is able to read and write out the box.
Linux natively supported from kernel 5.4.
Windows supports it from updated XP and onwards.
I not sure of Android compatibility though.
I not sure of Android compatibility though.
> exFAT is probably better in terms of cross platform compatibility and addresses a lot of the limitations of FAT while remaining portable.
Still it is a shame that we're kept back literally decades when it comes to file exchange due to corporate culture polluting every corner of IT. Ensuring file export options plus file formats full documentation, should be among the most important requirements any software should satisfy to be even considered for professional use.
Still it is a shame that we're kept back literally decades when it comes to file exchange due to corporate culture polluting every corner of IT. Ensuring file export options plus file formats full documentation, should be among the most important requirements any software should satisfy to be even considered for professional use.
I agree with your want. Wholeheartedly.
Alas you are right in that this culture is all over Corporate World (yes not just America). Vendor lock-in is something that's been with us since forever and I doubt it will ever go away. They might have gotten better at hiding it from obviously being spotted but that's about it and in some other cases it's still just as clear as always but the marketing people are still very good at selling it to customers.
Alas you are right in that this culture is all over Corporate World (yes not just America). Vendor lock-in is something that's been with us since forever and I doubt it will ever go away. They might have gotten better at hiding it from obviously being spotted but that's about it and in some other cases it's still just as clear as always but the marketing people are still very good at selling it to customers.
Android is annoyingly hit and miss. It's dependent on whether the device manufacturer wants to pay Microsoft royalty fees for it. So some devices do support it just fine and others don't. Often if a device has a microSD card slot and says it only supports 32GB cards it doesn't support exFAT since SDXC made the default file system for 32GB+ cards exFAT.
Microsoft exFAT patents expired in 2019, I think. It's included in the Linux kernel since 5.7 or some version around then. Android devices support exFAT depending on which Linux kernel version they have.
Maybe that will change in the future but most Android devices are not using a 5.x kernel yet. All my Samsung devices support exFAT and use Android 11 with a 4.14 kernel. My old BB Priv also supported exFAT with Marshmellow which definitely did not have kernel version 5!
The patents didn’t expire in 2019, Microsoft just allowed Linux to use them without fear of getting sued.
And Microsoft ACTIVELY went against Android manufacturers to get licensing fees a decade or so back, so those that didn't want to pay are much more likely to have a hard ban on exfat in place that will take time to overcome even after Microsoft opens it
> addresses a lot of the limitations of FAT while remaining portable
Whilst at the same time missing a very important feature for portable storage - a journal.
Whilst at the same time missing a very important feature for portable storage - a journal.
Journaling can have significant downsides when used with external flash storage including reduced performance and increased wear of the flash memory.
Not disputing your facts, but I've used journalled file systems in flash drives since forever and never had a performance issue that was not directly tied to the hardware, and most have lasted many years.
I prefer having to replace a cheap flash drive more frequently than to lose data altogether.
Of course, everybody's use case is different and tradeoffs need to be considered.
I prefer having to replace a cheap flash drive more frequently than to lose data altogether.
Of course, everybody's use case is different and tradeoffs need to be considered.
Is exFAT's reputation for data loss/corruption unwarranted?
Not really.
FAT32 is very simple and normally has a copy of each FAT table stored as well. When something goes wrong with the first FAT table, there's a reasonable chance you can still use the 2nd copy. [1]
ExFAT is much more complex and AFAICT it does not have a 2nd copy of all of its data. [2]
Anecdotal evidence is that I personally lost data on ExFAT file systems, but also have had many customers with problems when they figured to use ExFAT for my backup products. If I see ExFAT in a support bundle than -by default- my suggestion for fixing issues is to reformat to a different file system. More often than not, that's the end of the open support case.
[1] https://www.ntfs.com/fat-mirrow.htm
[2] https://www.ntfs.com/exfat-overview.htm
ExFAT is much more complex and AFAICT it does not have a 2nd copy of all of its data. [2]
Anecdotal evidence is that I personally lost data on ExFAT file systems, but also have had many customers with problems when they figured to use ExFAT for my backup products. If I see ExFAT in a support bundle than -by default- my suggestion for fixing issues is to reformat to a different file system. More often than not, that's the end of the open support case.
[1] https://www.ntfs.com/fat-mirrow.htm
[2] https://www.ntfs.com/exfat-overview.htm
I had an external harddrive with exfat that lost all its partition information.
Worked one day, then few days later plugged it in and it was all gone.
Worked one day, then few days later plugged it in and it was all gone.
Just write all your data twice, lol.
If one of them becomes corrupt in a non-obvious way, how do you know which it is? You need three copies to be sure.
Or you could save a hash as well
What if a bit flips in your hash file and changes that? Now you need to store your hash a bunch of times too!
I kid, this is what I do. I've recently been backing up my family photos/videos to various media and I've been hashing each file and storing a hash file alongside the media to compare to in the future.
I kid, this is what I do. I've recently been backing up my family photos/videos to various media and I've been hashing each file and storing a hash file alongside the media to compare to in the future.
no, not journaled
Just recently I brought a Mac-formatted exfat flash drive to a print shop, where it failed at being read on a Windows box. I doubt it that they're using WinME.
It might be an issue with partitioning. ExFAT support built in from Windows 7 and requires service packs for earlier versions.
OS X/macOS used to have some serious issues with exFAT but they may have been addressed in the few years since I last used exFAT on a Mac. Something to double-check before using the pair to store critical data.
exFAT is HORRIBLE. Sorry for the caps. Interoperabiliy between Windows and Mac is hit-or-miss. Formatting on Windows and it's random if Mac can read it. Unplug without ejecting and expect to wait hours for Mac to analyse and fix, or plug-in and out and in and out of Windows and hope it fixes it. Many times a day. Zero problems with NTFS.
FAT32 doesn't support files bigger than 4GiB, that's a huge limitation these days. But I don't disagree that calling NTFS "portable" is a bit of a stretch, although hopefully that'll change if good quality open source drivers become available.
A bit of personal experience: I've needed to create a Windows 10 installer from a MacOS computer a few weeks ago. Without the MS helper program I've figured I should perform the following steps:
1- download iso
2- format usb with fat32
3- copy the data
Apparently you can't just dd the ISO file into the USB block device like every other Linux or BSD ISO. It need to be FAT32 to be able to boot through UEFI.
The issue however is that the .wim file containing the OS installation source is bigger than 4GB. I think its sources.wim but I'm not sure. I had to install some command line utility from brew that did the job of "optimizing" the WIM file and managed to reduce it to a file bellow 4GB.
What does optimizing means? Better compression? Cleaning up crap inside the image? If their ISO tool probably does this without user interaction then why include a file bigger than what the FS can handle in the first place?
I don't know if this counts as some kind of dark pattern or MS being really antiquated in terms of bootable flash drivers.
1- download iso
2- format usb with fat32
3- copy the data
Apparently you can't just dd the ISO file into the USB block device like every other Linux or BSD ISO. It need to be FAT32 to be able to boot through UEFI.
The issue however is that the .wim file containing the OS installation source is bigger than 4GB. I think its sources.wim but I'm not sure. I had to install some command line utility from brew that did the job of "optimizing" the WIM file and managed to reduce it to a file bellow 4GB.
What does optimizing means? Better compression? Cleaning up crap inside the image? If their ISO tool probably does this without user interaction then why include a file bigger than what the FS can handle in the first place?
I don't know if this counts as some kind of dark pattern or MS being really antiquated in terms of bootable flash drivers.
What I'm doing is just using GPT partitioning, creating one NTFS partition and copy all files there. All my computers were able to boot in UEFI mode from this USB drive and successfully install Windows. With Linux it was a bit more cumbersome, but most Linux distros are hybrid ISOs anyway, so it's not as big issue. I know that UEFI BIOS is not obligated to support NTFS partitions, but my little practice they do.
Good to know that this is an option but from MacOS I don't have NTFS write capabilities. I think ExFat is available so I might try two partitions one being the EFI ESP and the second containing the rest of the install files.
It's a case of it working fine on Windows, so, there's no need to bother supporting anything else.
NTFS has journaling,compression, encryption, max 255 character filenames, max partition size is hundred of TB rather than 2GB, generally faster performance wise.
Trouble is , FAT32 has more OS compatibility.
Would be nice if Linux did have first class support for NTFS as it’s superior. But right now for maximum compatibility you need FAT32
Trouble is , FAT32 has more OS compatibility.
Would be nice if Linux did have first class support for NTFS as it’s superior. But right now for maximum compatibility you need FAT32
> max partition size is hundred of TB rather than 2GB
FAT16 is limited to 2GB, FAT32 can extend to 2TB (but the max file size for a single file is 4GB, as the parent stated)
FAT16 is limited to 2GB, FAT32 can extend to 2TB (but the max file size for a single file is 4GB, as the parent stated)
> max 255 character filenames
it doesn't in spec, that's just the win32 API
it doesn't in spec, that's just the win32 API
>it doesn't in spec, that's just the win32 API
NTFS filename length is limited to 255 Unicode chars.
You may be thinking of full path lengths of 32,767 Unicode chars but the filename component is still limited to 255.
If you look at the NTFS data structure "$FILE_NAME (0x30)" at byte offset 0x40: https://www.google.com/search?q=ntfs+%22%24FILE_NAME%22+byte
... there's only 1 byte used to describe the filename length. Hence max of 255 chars.
Wikipedia page about NTFS also mentions 255 max length for filename: https://en.wikipedia.org/wiki/NTFS
NTFS filename length is limited to 255 Unicode chars.
You may be thinking of full path lengths of 32,767 Unicode chars but the filename component is still limited to 255.
If you look at the NTFS data structure "$FILE_NAME (0x30)" at byte offset 0x40: https://www.google.com/search?q=ntfs+%22%24FILE_NAME%22+byte
... there's only 1 byte used to describe the filename length. Hence max of 255 chars.
Wikipedia page about NTFS also mentions 255 max length for filename: https://en.wikipedia.org/wiki/NTFS
You're confusing the filename length limit with the limitation on the entire path length in the Windows API. NTFS allows paths to be up to 32k characters long, but each path component tops out at 255 characters.
The Paragon driver cannot recognize encrypted NTFS volumes AFAIK
It's actually exFAT thats portable. NTFS is mostly for windows users, and macs do support it - though only reading.
Anyone know why? Given ext2 has had a public domain implementation for more than a decade? Is it because windows & mac won't support ext2 even for free? Seems like a better lowest common denominator fs than any of the fats?
You can't use easily ext2 on an external storage because of permissions.
That's because Linux and OSX ignore NTFS permissions making it work more like FAT.
Plug an existing NTFS drive from one Windows system into another Windows system and you get a similar issue where opening a home directory requires you to take ownership.
Plug an existing NTFS drive from one Windows system into another Windows system and you get a similar issue where opening a home directory requires you to take ownership.
But Linux refuse to ignore permissions on an ext2.
that depends, exFAT lacks permissions and symlinks. I think FreeBSD has NTFS support but lacks exFAT
> that depends, exFAT lacks permissions and symlinks.
The lack of permissions is why it works as a portable file system. If you've ever tried to use, say, ext for this it becomes clear why nobody does that.
The lack of permissions is why it works as a portable file system. If you've ever tried to use, say, ext for this it becomes clear why nobody does that.
You can just pass -o uid=X,gid=Y when mounting ext2 to solve this.
I have no idea if I did something, but after MacOS 11 upgrade it seems to automatically mount my NTFS drive.
I do know that FUSE MacOS stuff seems to be disabled in brew.
I do know that FUSE MacOS stuff seems to be disabled in brew.
This has been true for a long time. Does it mount it r/w or read only?
Interesting. Previously I had to run the mounting command manually ever since I remember trying to mount an NTFS drive.
It's read only.
It's read only.
IN my mind it has been exFAT for the past several years.
Buy an external hard drive or USB stick, unless it's marketed as "for Mac", it's going to ship formatted as NTFS.
IME most external hard drives and USB sticks are marketed as compatible with both Windows and Mac, and ship formatted with FAT32. But maybe I'm the odd one here.
Most these days ship as exFAT.
Literally NONE of the 100s of Tb I have purchased over the past year in formats ranging from SD card to USB stick to SATA drives have been formatted as NTFS. None of them.
memory cards are FAT32
Macs support NTFS in read-only mode out of the box. For writing you need tools like Paragon NTFS.
They do, but only to read, not write. I think the current up to date recommendation for cross-OS compatibility on external drives is to use ExFAT, though I believe the concept of the ‘external drive’ itself should be deprecated.
> though I believe the concept of the ‘external drive’ itself should be deprecated
why?
why?
I think they’re problematic because they’ve reached ungodly amounts of storage and I don’t think most non-technical people fully understand how much of a loaded gun a non-regularly-backed-up 2TB external HDD is.
As a real world example, I’ve had to help recover my aunt’s 5-year-old 2TB HDD that she dropped several times and had no backup of, and by helping I mean confirming she probably lost a terabyte plus of data unless she wants to pay five digits USD to pay to a recovery company for a way less than 100% recovery. She is a highly educated professional (a MD) and yet, it did not even occur to her that the drive will not only might fail, but rather, will fail.
To quite a lot of non-technical people an external drive appears to be the safest form of storage, considering its being physically visible and movable. The closest analogy is storing money under the pillow versus a in bank, in all the good and bad ways.
As a real world example, I’ve had to help recover my aunt’s 5-year-old 2TB HDD that she dropped several times and had no backup of, and by helping I mean confirming she probably lost a terabyte plus of data unless she wants to pay five digits USD to pay to a recovery company for a way less than 100% recovery. She is a highly educated professional (a MD) and yet, it did not even occur to her that the drive will not only might fail, but rather, will fail.
To quite a lot of non-technical people an external drive appears to be the safest form of storage, considering its being physically visible and movable. The closest analogy is storing money under the pillow versus a in bank, in all the good and bad ways.
Yeah, you need to use FAT32 if you want to make a cross os portable drive.
> NTFS is the de facto filesystem for portable data storage
I've never seen it used that way, guess I'm weird.
One hopes that the people who do use it will work on it, but given the patent fiasco with exFAT, I wouldn't either look at or use the code, personally.
I've never seen it used that way, guess I'm weird.
One hopes that the people who do use it will work on it, but given the patent fiasco with exFAT, I wouldn't either look at or use the code, personally.
I dual boot between Linux and Windows on my laptop, and I use three physical hard drives: two SSDs, each dedicated to a different OS, and one hybrid HDD with more storage that's "shared" between the two (and also used as a simple way off passing data between them).
Maybe things have changed now but NTFS was the only reliable option for that last drive when I first set things up this way a few years ago.
Maybe things have changed now but NTFS was the only reliable option for that last drive when I first set things up this way a few years ago.
Me neither. I'm not sure what's up with all the talk here about NTFS for portable storage. If "portable" is meant as in USB key, I've never encountered one formatted that way. If it's portable as in a disk in an enclosure, then I suppose I can follow the rationale.
> well funded
Is it? The kernel has a billion users who pay $0 for it. Most of the work is done by volunteers still. Oracle and RedHat have a few people looking at it full-time, and then Linus himself. This patchset is maintained by a third-party, Paragon, who also have an NTFS-for-Macs solution.
Hence:
>> We simply don't have anybody to funnel new filesystems
Also I see the new polite kernel is going well: https://lore.kernel.org/lkml/YO8LOKR%[email protected]...
Is it? The kernel has a billion users who pay $0 for it. Most of the work is done by volunteers still. Oracle and RedHat have a few people looking at it full-time, and then Linus himself. This patchset is maintained by a third-party, Paragon, who also have an NTFS-for-Macs solution.
Hence:
>> We simply don't have anybody to funnel new filesystems
Also I see the new polite kernel is going well: https://lore.kernel.org/lkml/YO8LOKR%[email protected]...
Linux is on every Android phone. You can argue whether it's "real linux" or not, but that is besides the point with respect to filesystem drivers?
(I don't have a clue what does Android actually use for filesystem drivers, but I would _guess_ it's just Linux down there.)
(I don't have a clue what does Android actually use for filesystem drivers, but I would _guess_ it's just Linux down there.)
> Linux is on every Android phone
Yes, and almost all consumer routers and quite a lot of other random pieces of hardware. Hence the billions of users.
How many of those pay for upstream maintainers?
> I don't have a clue what does Android actually use for filesystem drivers
"Yaffs" or "sparse" (which is just fancy ext2). https://source.android.com/devices/bootloader/images ; most Android kernels will also have exfat so they can use SD cards and USB sticks.
Yes, and almost all consumer routers and quite a lot of other random pieces of hardware. Hence the billions of users.
How many of those pay for upstream maintainers?
> I don't have a clue what does Android actually use for filesystem drivers
"Yaffs" or "sparse" (which is just fancy ext2). https://source.android.com/devices/bootloader/images ; most Android kernels will also have exfat so they can use SD cards and USB sticks.
Portable means that you can stick a USB pendrive in your car, speaker, TV, printer and be able to at least read the media on it. exFAT or FAT32 are the standard here, not NTFS.
Computers are easier to deal with, there might be obscure programs that can read and write any known file system format despite what the kernel supports.
Computers are easier to deal with, there might be obscure programs that can read and write any known file system format despite what the kernel supports.
>NTFS is the de facto filesystem for portable data storage
That would be either FAT32 or exFAT
That would be either FAT32 or exFAT
Way back there was a contributor Con Kolivas, that made scheduler improvements to make desktop computing nicer. But he quit because they never gave priority to desktop stuff. I remember back then it was not uncommon for your mouse cursor to just freeze and then come back to life.
The kernel group has different priorities to end users
The kernel group has different priorities to end users
That's rough. I've had similar experiences.
TDLR: Needful, worthy improvements require champions. A thankless role. And few of us have the resources and tenacity to serve the champion role for the long haul.
In trying to understand organizational psychology and apparent apathy, I keep thinking of Stephen Jay Gould's theory of "punctuated equilibrium" meets the attention economy.
Metaphorically, organizations (and people) are information processors. Attention is our scarcest resource. We're all doing triage, at every level. Just trying to cope with the torrent of information, discerning signal from noise. Trying to prioritize efforts.
So it's almost inevitable that needful, useful, worthy things remain neglected, undone. Even in the absence of pushback.
Until some kind of crisis. When the neglected thing suddenly becomes very important.
Cue the fire drill. Something must be done.
The champion must be watchful, ready, proactive. Seize that very brief opportunity to sneak in their needful improvement. Ahead of the all the spazs pitching their weird and wrong and ignorant reactive proposals.
The only winning strategy is to just keep pushing. Patiently, diplomatically. Be that useful thing that everyone's dimly aware of.
"Oh, wasn't Con Kolivas working on that mouse jitter thing? Ya, I heard it's pretty good. Ah, it's ready to go? Cool, let's do it."
If your needful thing is truly worthy, be confident that eventually it'll percolate to the top of the triage stack.
This game truly sucks. But I don't see how it could be any other way with our current decision making structures.
TDLR: Needful, worthy improvements require champions. A thankless role. And few of us have the resources and tenacity to serve the champion role for the long haul.
In trying to understand organizational psychology and apparent apathy, I keep thinking of Stephen Jay Gould's theory of "punctuated equilibrium" meets the attention economy.
Metaphorically, organizations (and people) are information processors. Attention is our scarcest resource. We're all doing triage, at every level. Just trying to cope with the torrent of information, discerning signal from noise. Trying to prioritize efforts.
So it's almost inevitable that needful, useful, worthy things remain neglected, undone. Even in the absence of pushback.
Until some kind of crisis. When the neglected thing suddenly becomes very important.
Cue the fire drill. Something must be done.
The champion must be watchful, ready, proactive. Seize that very brief opportunity to sneak in their needful improvement. Ahead of the all the spazs pitching their weird and wrong and ignorant reactive proposals.
The only winning strategy is to just keep pushing. Patiently, diplomatically. Be that useful thing that everyone's dimly aware of.
"Oh, wasn't Con Kolivas working on that mouse jitter thing? Ya, I heard it's pretty good. Ah, it's ready to go? Cool, let's do it."
If your needful thing is truly worthy, be confident that eventually it'll percolate to the top of the triage stack.
This game truly sucks. But I don't see how it could be any other way with our current decision making structures.
You'll need someone working almost full time on this (like Jens Axboe on io_uring). I'm wondering whether it's some emergent behaviour to make sure you'll maintain the thing on the long-term and not just dump the thing on a subsystem guy and see ya!
> NTFS is the de facto filesystem for portable data storage
No, that would be exFAT and FAT32. Some form of FAT is basically supported by everything, which is why if anything is the "de facto" for portability, it's FAT. FAT is the netBSD of FS.
No, that would be exFAT and FAT32. Some form of FAT is basically supported by everything, which is why if anything is the "de facto" for portability, it's FAT. FAT is the netBSD of FS.
As everyone seems to be fixated on pendrives and FAT, consider also that virtually every Linux-based consumer-grade NAS needs to support R/W NTFS. Typically it was through FUSE with considerable loss of performance/efficiency.
> consider also that virtually every Linux-based consumer-grade NAS needs to support R/W NTFS
Do they? I don't think egress/ingress from external drives is an essential feature for consumer-grade NAS.
Internal drives are typically formatted with a Linux-native filesystem and accessed through SMB/Samba from Windows machines.
Do they? I don't think egress/ingress from external drives is an essential feature for consumer-grade NAS.
Internal drives are typically formatted with a Linux-native filesystem and accessed through SMB/Samba from Windows machines.
> I don't think egress/ingress from external drives is an essential feature for consumer-grade NAS
Is it not? I think most consumer NASes on the market support NTFS-formatted external drives (be that USB or eSATA) and need a r/w capable driver.
Is it not? I think most consumer NASes on the market support NTFS-formatted external drives (be that USB or eSATA) and need a r/w capable driver.
The parent said "de facto filesystem for portable data storage", NAS for Windows is a very specific use case. For data portability, compatibility and wide adoption matters, and that is undoubtedly where FAT and it's variants rule.
Are you confusing NTFS with FAT32?
NTFS is pretty much limited to internal drives on Windows PCs.
NTFS is pretty much limited to internal drives on Windows PCs.
Considering the userbase, everyone formats using NTFS (if they want files more than 4GB), so it’s de facto standard due to the sheer quantity.
ExFat is great cross-option, though, without the limitations of fat32.
ExFat is great cross-option, though, without the limitations of fat32.
Android SD cards userbase is probably comparable or even bigger nowadays. And those use fat32 or exFAT AFAIK.
I'd guess not since big G has been on the push to abolish external storage - and I wouldn't blame them, really hard to put data (especially application data) on hotplug removable storage.
The camera world has been using exFAT since quite a while.
The camera world has been using exFAT since quite a while.
ext3 is way more portable... even under *BSDs
.. "NTFS... de facto standard"? ... from M$ lolz
Ext3 sucks under BSD's, a lot of them can't write to the volume, doesn't matter either EXT2, 3, or 4.
UDF as I stated works fine with udfclient under OpenBSD and it could be "mountable" even with base tools (read /usr/local/share/doc/pkg-readmes/udfclient in order to correctly format it with udfclient).
I copied files back and forth on USB drives written in under OpenBSD and the volumes mounted like magic under Linux. And the performance is not bad at all.
UDF as I stated works fine with udfclient under OpenBSD and it could be "mountable" even with base tools (read /usr/local/share/doc/pkg-readmes/udfclient in order to correctly format it with udfclient).
I copied files back and forth on USB drives written in under OpenBSD and the volumes mounted like magic under Linux. And the performance is not bad at all.
I will give a try to UDF -esp if has comparable write speeds with ext3 , but for sure I had no issues to _write_ on an external ext3 with NetBSD + FreeBSD (my last desktop OS, before move to Debian Sid)
Linux is well funded in the sense that corporations put big money looking after their own interest in the kernel. Anything outside of their respective areas of interest? Good luck.
Paragon, for better or worse, is just yet another corporation. They shouldn't really expect other corporations or volunteers to care much about their code, other than the fact that their code may break others'.
Now that they have Linus' blessing, they can just send a pull request. The work paid off in the end.
Paragon, for better or worse, is just yet another corporation. They shouldn't really expect other corporations or volunteers to care much about their code, other than the fact that their code may break others'.
Now that they have Linus' blessing, they can just send a pull request. The work paid off in the end.
Can you explain what you mean by "portable data storage"? I'm used to hearing the word "portable" used to mean "cross platform", but if it is not supported well under Linux, then given how widely used Linux is, I would think ntfs would not count as cross platform.
I'm pretty sure that the grandparent meant portable quite literally, as in you can take it with you. Things like USB sticks, external hard drives, sd cards etc..
That was the impression I was getting, hence my question. But since thumb drives typically come preformatted in either FAT32 or exFAT, I thought maybe there was a third possible use of the word "portable" that I am unfamiliar with.
External hard drives with larger capacities and potentially larger files might be a use case for NTFS. While it isn't fast, ntfs-3g has worked fine for basic use for me. Someone said in another comment that ntfs-3g is unmaintained, though, which was news to me, and obviously not a great thing if true.
I suppose exFAT might be a viable option for larger external drives as well due to its higher file size limit. None of the other commonly used file systems (including native Linux file systems) seemed easy to use from both Linux and Windows the last time I looked into it.
I suppose exFAT might be a viable option for larger external drives as well due to its higher file size limit. None of the other commonly used file systems (including native Linux file systems) seemed easy to use from both Linux and Windows the last time I looked into it.
Honestly ntfs-3g is fine. It's slow, which I'd always blamed on NTFS since NTFS is also clearly slow for large directories of small files on Windows also, but apparently this new driver is better in that regard.
Linux on the desktop (where portability of file system matters) is 2% at the very best. Hardly what one would call "widely used".
>Seriously, such a well funded, high profile project like Linux should be more receptive to key QoL improvements and drivers.
I guess politics gets in the way?
I guess politics gets in the way?
Not necessarily politics. It could be purely due to lack of interest from maintainers and reviewers.
> NTFS is the de facto filesystem for portable data storage
FAT32 needs to be used if you want to maximise portability
FAT32 needs to be used if you want to maximise portability
UDF if you don't care about printers or Android.
Personally I've seen more ExFAT volumes on external storage in the wild than NTFS nowadays.
> NTFS is the de facto filesystem for portable data storage,
are you high right now? 'cause you sound like you're on drugs when you say things like that.
are you high right now? 'cause you sound like you're on drugs when you say things like that.
Hey can you please stop posting unsubstantive comments and breaking the site guidelines? We ban that sort of account. I don't want to ban you but we've had to ask you about this before.
If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and sticking to the rules when posting here, we'd be grateful.
If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and sticking to the rules when posting here, we'd be grateful.
There's a real gem in that thread:
> FYI: There is also another cross-platform filesystem (Linux kernel, Windows NT kernel, Mac OS X kernel) suitable for hard disks too with POSIX permissions about which people do not know too much. It is UDF.
As I was struggling to find a way to share data between Linux/Windows with my gpu-passthrough setup, I'm excited to use this.
I think I've read something about 9p support in the latest Windows versions and I think QEMU supports 9p so that might become a good solution, too.
> FYI: There is also another cross-platform filesystem (Linux kernel, Windows NT kernel, Mac OS X kernel) suitable for hard disks too with POSIX permissions about which people do not know too much. It is UDF.
As I was struggling to find a way to share data between Linux/Windows with my gpu-passthrough setup, I'm excited to use this.
I think I've read something about 9p support in the latest Windows versions and I think QEMU supports 9p so that might become a good solution, too.
There are several platform specific gotchas with UDF. If you want to use UDF on a drive, I suggest using the script at https://github.com/JElchison/format-udf. It takes into account the platform specific gotchas and formats a drive properly to be usable across Windows, Linux and macOS.
Any chance that there's a way to actually encrypt a device set up like that?
I usually use LUKS and EXT4 for all my drives but lately I've needed to move files to and from Windows systems on occasion, and it kills me that (from what I can tell) there are absolutely no solutions that even attempt to do something like this.
Does everyone just walk around with unencrypted portable media? I don't really want to float sensitive data on unencrypted drives, even temporarily, since that makes the data vulnerable to things like wear leveling analysis etc.
I usually use LUKS and EXT4 for all my drives but lately I've needed to move files to and from Windows systems on occasion, and it kills me that (from what I can tell) there are absolutely no solutions that even attempt to do something like this.
Does everyone just walk around with unencrypted portable media? I don't really want to float sensitive data on unencrypted drives, even temporarily, since that makes the data vulnerable to things like wear leveling analysis etc.
mkudffs (a.k.a. mkfs.udf) on the whole block device, by default it uses --bootarea=erase, and results in a device that automounts on macOS, Windows 10, and GNOME and KDE (at least).
There is also --bootarea=mbr which sounds similar to what this script is doing but I'm not sure why it's necessary; maybe for older versions of Windows.
There is also --bootarea=mbr which sounds similar to what this script is doing but I'm not sure why it's necessary; maybe for older versions of Windows.
Windows works fine with flash drives without a partition table but can have issues with hard drives without one.
Yes. That script is really good!
Sadly Microsoft and Apple have gone out of their way to make it difficult to access the same UDF filesystem from both Windows and macOS.
macOS will only access a UDF filesystem created on a disk device. Windows will only access one created inside a partition.
Linux, naturally, has no problem handling both. ;)
macOS will only access a UDF filesystem created on a disk device. Windows will only access one created inside a partition.
Linux, naturally, has no problem handling both. ;)
A sibling comment linked https://github.com/JElchison/format-udf which claims that it creates a whole-device FS which works across all 3.
9p is not a file system but a byte oriented file based RPC protocol for transferring data. In plan 9 the idea is files are objects in the programming sense which you create/open/read/write/remove/close. Example: your mouse is a file, /dev/mouse, which you open to read the coordinates and click events.
In plan 9 files are served by file servers, programs which speak 9p. The plan 9 kernel knows nothing of disks or file systems living on those disks. In fact, you don't mount disks in plan 9, you mount 9p connections. So to access files on a Fat formatted disk you need a file server program called dossrv(4) which opens the disk file and listens for 9p messages and performs the necessary Fat operations.
Fun fact: A common plan 9 newbie mistake is to mount a disk file directly as in Unix. This causes the kernel to write a 9p T_attach message to the beginning of the disk overwriting the boot sector. http://man.postnix.pw/9front/5/attach
In plan 9 files are served by file servers, programs which speak 9p. The plan 9 kernel knows nothing of disks or file systems living on those disks. In fact, you don't mount disks in plan 9, you mount 9p connections. So to access files on a Fat formatted disk you need a file server program called dossrv(4) which opens the disk file and listens for 9p messages and performs the necessary Fat operations.
Fun fact: A common plan 9 newbie mistake is to mount a disk file directly as in Unix. This causes the kernel to write a 9p T_attach message to the beginning of the disk overwriting the boot sector. http://man.postnix.pw/9front/5/attach
I've used it before for a USB drive and honestly I have had nothing but problems with it. I don't remember exactly what kind of problems but I do remember that I was frustrated enough after some time to switch to something else.
If you want to try out the ntfs-3 driver, it is already in the XanMod kernel.
I've dug out some more details re:Windows 9p support. See under "How it works": https://devblogs.microsoft.com/commandline/whats-new-for-wsl...
Ironically, Google's Pixel build of Android does not include UDF support - the filesystem does not work on a USB stick like FAT32...
Actually for those plus freebsd, you could theoretically add openzfs - portable, support for permissions and native encryption support.
I'm trying to find it without success, does udf support journaling (or some form of recovery after an unclean reboot)?
From https://sites.google.com/site/udfintro/ :
> Although an implementation may use a journal to protect metadata integrity, this does not guarantee interoperability between platforms since it is not part of the standard.
it seems it's left up to the implementation.
> Although an implementation may use a journal to protect metadata integrity, this does not guarantee interoperability between platforms since it is not part of the standard.
it seems it's left up to the implementation.
Non-interoperability of a journal isn't a massive downside. It just means after a system failure, you need to plug it back into the same OS first.
Obviously that needs to be enforced with some kind of "do not mount" option so it isn't accidentally used on another OS.
Obviously that needs to be enforced with some kind of "do not mount" option so it isn't accidentally used on another OS.
For USB drives, it means that if you take it out from your work PC in a hurry, it might not be usable on your home Mac. Pretty serious I would say.
I was just looking at the most recent v26 patchset from June yesterday and it was really depressing. Paragon has been plugging away at it for close to a year (2020-08) and there has been seemingly no interest in this at all. As a regular desktop user who spent years dealing with the very inefficient FUSE ntfs-3g implementation the disconnect seems very real.
There's a remarkably small number of very overworked people making all the decisions, and none of them are part-time Windows users. :/
It's especially important since FUSE's ntfs-3g is essentially dead - last release was 4 years ago [1].
[1] https://www.tuxera.com/company/open-source/
[1] https://www.tuxera.com/company/open-source/
It's also single threaded, i.e. it can only handle one read/write request at a time (fuse can do more but ntfs-3g doesn't implement it). That has kind of acceptable performance with hard disks but with SSDs it is really slow.
I haven't messed with NTFS in years but from what I remember the Linux kernel driver was read-only, right? I assume this new one will allow write access?
I vaguely remember having to use fuse to write to NTFS partitions in the past.
I vaguely remember having to use fuse to write to NTFS partitions in the past.
The existing kernel mode driver has a very limited write mode that is marked with flashing sirens and limited to things like (IIRC) "only modifying existing files without changing their size".
ntfs-3g is a FUSE driver with read/write support, but it's FUSE, and not bundled with the kernel.
This new driver is (based on?) the commercial NTFS driver Paragon sold, and does not suffer the limitations of either, AIUI, and I gather the objections have mostly been around the state of the code rather than functionality.
edit: Having looked, the comment in the Kconfig makes it clear that they believe the limited write support is quite limited, but quite safe.
ntfs-3g is a FUSE driver with read/write support, but it's FUSE, and not bundled with the kernel.
This new driver is (based on?) the commercial NTFS driver Paragon sold, and does not suffer the limitations of either, AIUI, and I gather the objections have mostly been around the state of the code rather than functionality.
edit: Having looked, the comment in the Kconfig makes it clear that they believe the limited write support is quite limited, but quite safe.
The FUSE one (ntfs-3g) is fine - we use it for bulk editing Windows VMs when transferring them from VMware to KVM[1]. Unfortunately development has slowed to a standstill in the last few years. Which is a problem. At the moment it supports current Windows guests including relatively recent features like file compression, but we're only a MSFT new feature away from having to take over upstream development or look for an alternative. I doubt that an in-kernel driver will be noticably faster, but there's hope it might be more widely used and so better maintained.
[1] https://libguestfs.org/virt-v2v.1.html
[1] https://libguestfs.org/virt-v2v.1.html
Sure, I didn't mean to suggest there was anything fundamentally flawed about ntfs-3g, I've used it to basically universal success after the first year or two after it was released, just that it is FUSE, which means there's always going to be some overhead involved that wouldn't be present in an in-kernel driver (though they keep trimming away at how much that has to be).
https://jp-andre.pagesperso-orange.fr/changelog.html doesn't seem totally dead, though the github repo hasn't seen much since March (but I don't think "since March" is what you meant by "last few years").
https://jp-andre.pagesperso-orange.fr/changelog.html doesn't seem totally dead, though the github repo hasn't seen much since March (but I don't think "since March" is what you meant by "last few years").
I am not 100% sure, but I think write support for NTFS has been a thing for at least 10 years. At least with the ntfs-3g FUSE driver.
I used NTFS with linux a year ago. It's read-write and suitable if you need to access files on Windows partition.
It was OK but not great: I used it to share media files between Windows and Linux and soon I noticed that if I'm playing mp3s from NTFS partition a lot of CPU used.
I remember writing to NTFS from linux 4 years ago.
I am currently able to read and write. I never installed anything and I'm using Manjaro.
"Vboxsf fixes for 5.14.1"?
NTFS is a mess on Linux, one boot into Windows and your drive is now read only in Linux
Only if fast boot is enabled. Which makes sense, because in that case Windows doesn't shut down entirely, the file system is still marked as mounted. You can break that lock, but you really shouldn't. Better to disable fast boot entirely.
popol12(1)
[deleted]
Seriously, such a well funded, high profile project like Linux should be more receptive to key QoL improvements and drivers. NTFS is the de facto filesystem for portable data storage, much as I wish an open alternative was there. I hope they speed up the merge of ntfs3 into the mainline.