Hey all! This is something we've been working on for a while now. It's the future of native cloud storage on macOS. I run ExpanDrive (www.expandrive.com). We use a derivative of macFUSE (https://osxfuse.github.io/) on macOS.
Third party kernel extensions on macOS are on the way out. On M1 machines any third party kernel extension requires a reboot into recovery mode, lowering the security settings, and then finishing the install.
This is a terrible and unfriendly model for users (by design). This is the primary reason these Google/Box/etc aren’t shipping M1 versions of their software.
The replacement for apps like Google & Box Drive (and everyone else) is their new File Provider framework (https://developer.apple.com/documentation/fileprovider), which they quietly launched but didn’t publicize at all. It’s basically a technology that enables something like Dropbox Smart Sync (project infinite) built directly into macOS fully managed by APFS.
It’s a pretty big departure from the macFUSE style interfaces we've been using for years now. File Provider apps are userspace extensions that work with with placeholder files and on-demand hydration of remote files by the system via APFS. It’s a pretty cool framework, but pretty new (and a little raw).
Box/Google/Dropbox/Microsoft will all eventually be using it, but the rollout has been slow.
The whole macOS File Provider project is an off-shoot of the iCloud drive and iOS file provider projects, but with one key component they’ve added called the replicated extension that has an apple-blessed extension to APFS called “FPFS” (presumably file provider FS) that can intercept read/writes/etc to these on-disk placeholder files and download content (and thumbnails!) on demand. Provides data directly to spotlight indexes, etc.
Whereas FUSE is a really low level API exported into a safer user-space process, this is a little higher level. You give the file provider framework lists of directories, help it monitor changes, provide item info and then it will in turn issue callbacks into your extension to initiate downloads to a temporary location.
Benefits
1. All on demand, driven by placeholders. “regular” sync downloads contents ahead of time, this is not the case with file providers. It’s more like network filesystems in this regard
2. Doesn’t eat up free space. Providers can mark their content as “evictable” which lets APFS know it can toss out the data if space gets low. But what I think is extra interesting is that when you mark the content as evictable it doesn’t even register as being used against free space. You could bring down a 10GB file from the cloud and your free space remains the same
3. Integrated with all the higher level APIs so that applications that open a file with swift/objective C don’t beachball while waiting for a download (for an open) to complete. They appropriate waits and expectations are in there now
Hey all! This is something we've been working on for a while now. It's the future of native cloud storage on macOS.
I run ExpanDrive (www.expandrive.com). We use a derivative of macFUSE (https://osxfuse.github.io/) on macOS.
Third party kernel extensions on macOS are on the way out.
On M1 machines any third party kernel extension requires a reboot into recovery mode, lowering the security settings, and then finishing the install.
This is a terrible and unfriendly model for users (by design). This is the primary reason these Google/Box/etc aren’t shipping M1 versions of their software.
But that’s okay, they’ve got something better they’re working on. Just keeping it a little quiet right now since it’s not 100% done.
The replacement for apps like Google & Box Drive (and everyone else) is their new File Provider framework (https://developer.apple.com/documentation/fileprovider), which they quietly launched but didn’t publicize at all. It’s basically a technology that enables something like Dropbox Smart Sync (project infinite) built directly into macOS fully managed by APFS.
It’s a pretty big departure from the macFUSE style interfaces we've been using for years now. File Provider apps are userspace extensions that work with with placeholder files and on-demand hydration of remote files by the system via APFS. It’s a pretty cool framework, but pretty new (and a little raw).
Box/Google/Dropbox/Microsoft will all eventually be using it, but the rollout has been slow.
The whole macOS File Provider project is an off-shoot of the iCloud drive and iOS file provider projects, but with one key component they’ve added called the replicated extension that has an apple-blessed extension to APFS called “FPFS” (presumably file provider FS) that can intercept read/writes/etc to these on-disk placeholder files and download content (and thumbnails!) on demand. Provides data directly to spotlight indexes, etc.
Whereas FUSE is a really low level API exported into a safer user-space process, this is a little higher level. You give the file provider framework lists of directories, help it monitor changes, provide item info and then it will in turn issue callbacks into your extension to initiate downloads to a temporary location.
Benefits
1. All on demand, driven by placeholders. “regular” sync downloads contents ahead of time, this is not the case with file providers. It’s more like network filesystems in this regard
2. Doesn’t eat up free space. Providers can mark their content as “evictable” which lets APFS know it can toss out the data if space gets low. But what I think is extra interesting is that when you mark the content as evictable it doesn’t even register as being used against free space. You could bring down a 10GB file from the cloud and your free space remains the same
3. Integrated with all the higher level APIs so that applications that open a file with swift/objective C don’t beachball while waiting for a download (for an open) to complete. They appropriate waits and expectations are in there now
Hey all! This is something we've been working on for a while now. It's the future of native cloud storage on macOS.
I run ExpanDrive (www.expandrive.com). We use a derivative of macFUSE (https://osxfuse.github.io/) on macOS.
Third party kernel extensions on macOS are on the way out.
On M1 machines any third party kernel extension requires a reboot into recovery mode, lowering the security settings, and then finishing the install.
This is a terrible and unfriendly model for users (by design). This is the primary reason these Google/Box/etc aren’t shipping M1 versions of their software.
But that’s okay, they’ve got something better they’re working on. Just keeping it a little quiet right now since it’s not 100% done.
The replacement for apps like Google & Box Drive (and everyone else) is their new File Provider framework (https://developer.apple.com/documentation/fileprovider), which they quietly launched but didn’t publicize at all. It’s basically a technology that enables something like Dropbox Smart Sync (project infinite) built directly into macOS fully managed by APFS.
It’s a pretty big departure from the macFUSE style interfaces we've been using for years now. File Provider apps are userspace extensions that work with with placeholder files and on-demand hydration of remote files by the system via APFS. It’s a pretty cool framework, but pretty new (and a little raw).
Box/Google/Dropbox/Microsoft will all eventually be using it, but the rollout has been slow.
The whole macOS File Provider project is an off-shoot of the iCloud drive and iOS file provider projects, but with one key component they’ve added called the replicated extension that has an apple-blessed extension to APFS called “FPFS” (presumably file provider FS) that can intercept read/writes/etc to these on-disk placeholder files and download content (and thumbnails!) on demand. Provides data directly to spotlight indexes, etc.
Whereas FUSE is a really low level API exported into a safer user-space process, this is a little higher level. You give the file provider framework lists of directories, help it monitor changes, provide item info and then it will in turn issue callbacks into your extension to initiate downloads to a temporary location.
Benefits
1. All on demand, driven by placeholders. “regular” sync downloads contents ahead of time, this is not the case with file providers. It’s more like network filesystems in this regard
2. Doesn’t eat up free space. Providers can mark their content as “evictable” which lets APFS know it can toss out the data if space gets low. But what I think is extra interesting is that when you mark the content as evictable it doesn’t even register as being used against free space. You could bring down a 10GB file from the cloud and your free space remains the same
3. Integrated with all the higher level APIs so that applications that open a file with swift/objective C don’t beachball while waiting for a download (for an open) to complete. They appropriate waits and expectations are in there now
I think the APFS snapshot integration is easily the coolest part of Arq 6. Arq now has access to a special Apple entitlement to take full-desk point-in-time snapshots of an APFS container and backup from that. It's like what time machine would've/should've been, for the cloud.
For the most part, I'd just be using macOS WindowServer. Which isn't very helpful. That's part of what is driving the question about whether one big monitor is ultimately better.
They have access to a kernel module and entitlement that only they have access to, com.apple.fileutil - and Apple has said they aren't giving out any more entitlements for it.
Curious if anyone knows what Dropbox will do about SmartSync. KAuth extensions are likely out as of 10.16, and the Endpoint Security extensions don’t let you block a reply for more than 60 seconds, so you can’t dynamically page in large files anymore.
They currently offer no replacement for VFS. But I bet they will offer something fuse-like if they offer anything at all. Also: the file provider api they use for iCloud Drive that was supposed to ship in Catalina but got yanked is still likely to happen.
I have no inside information but assuming they continue to expose the VFS layer they will very likely build a usermode extension framework that is quite like FUSE, but supported by the OS and maintained by Apple.
I think this is fairly overblown, there are a fair number of FUSE for macOS forks out there with signing certificates.
I have kext signing certificate for ExpanDrive, Google has one for Google Filestream, I suspect many others have one as well. Rightfully, Apple doesn't hand them out as easily as they do with regular developer certificates, but if you want one and do a reasonable job representing that you're not going to panic end-user systems, you can get one too.
FUSE for macOS remains open source, fork it if you want. Benjamin merely decided not to work on it for free anymore and essentially providing bug fixes etc for those who pay for it.
Lastly - FUSE of macOS is not going to be around in the current form much longer. Apple has made it abundantly clear that Kernel Extensions are on the way out, and that macOS 10.15 will be the last release to fully support kexts without compromises. Check this slide from WWDC
We only email a couple times a year. And we definitely don’t silently update to a new major version. Ping support and link this thread and we can upgrade you free. Sorry for the trouble.
Projects like this seem like a great idea, but deliverability seems like a big concern that is hard to measure unless you have a reasonable amount of experience.
What are best practices for using/selecting an ESP if you were to use a project like this and want to ensure reasonable deliverability?