Self-hosted on-demand synched file storage
6 comments
You're describing a "distributed filesystem with client-side whole-file caching". The Andrew File System (AFSv2) did this back in the 1990s.
https://pages.cs.wisc.edu/~remzi/OSTEP/dist-afs.pdf
FWIW, complexity is introduced when the server is unavailable and the client-side caches need to be synced with the remote.
Sshfs, NFS, and SMB are substantially easier to set up and live with, so you're not fighting conflict resolutions inherent in multi-primary repositories.
https://pages.cs.wisc.edu/~remzi/OSTEP/dist-afs.pdf
FWIW, complexity is introduced when the server is unavailable and the client-side caches need to be synced with the remote.
Sshfs, NFS, and SMB are substantially easier to set up and live with, so you're not fighting conflict resolutions inherent in multi-primary repositories.
I've been looking for software that does this myself, but I don't think it exists.
There used to be a project, https://bazil.org/, that promised to do it. The website is still up, but it has not been updated in years.
The existing tool closest to what I envision is git-annex. It tracks what files live where, and you can sync single files or as much content as will fit on the local harddrive with a few commands. There is also git-annex-assistant for a more dropbox-like experience, although I haven't used it myself.
There used to be a project, https://bazil.org/, that promised to do it. The website is still up, but it has not been updated in years.
The existing tool closest to what I envision is git-annex. It tracks what files live where, and you can sync single files or as much content as will fit on the local harddrive with a few commands. There is also git-annex-assistant for a more dropbox-like experience, although I haven't used it myself.
After some research it looks like seafile [1] and nextcloud virtual file mode [2] are viable candidates.
[1] https://www.seafile.com/en/home/ [2] https://nextcloud.com/blog/nextcloud-desktop-client-3-2-with...
[1] https://www.seafile.com/en/home/ [2] https://nextcloud.com/blog/nextcloud-desktop-client-3-2-with...
I think you're looking for this: https://syncthing.net/
Doe not look like syncthing has the concept of "vault" (using sync.com parlance) where files are stored in the cloud only.
What the OP is describing is more like apple's iCloud, where files might be offloaded from a device and stored only in the cloud, and downloaded on demand.
What the OP is describing is more like apple's iCloud, where files might be offloaded from a device and stored only in the cloud, and downloaded on demand.
I used to use dropbox...I then spent years using raspberry pi's as storage devices, rclone, gdrive, s3 storage, syncthing and other solutions...I am now using dropbox again :-) (with rclone thrown in where needed)
I would like for the desktop to be a logically infinite source of storage, while the laptop stores only what I am currently using.
For example, if I have a "photos" folder with 10tb of photos, I would like to see a list of all the photos when I open the folder on my laptop. If I then select a specific photo to open, my laptop checks if it already has it downloaded locally and if not, silently pulls it from the server. If I do that thousands of times and run out of space on my laptop, the laptop should automatically purge the LRU photo to load the new one.
From the laptop's perspective, it has an unlimited amount of space - the only noticeable thing is that some files take longer to load than others.
A critical component of this is that both systems are automatically synched. If a folder is created on the laptop, it is also created on the desktop. If a file is created on the laptop it is automatically backed up to the desktop. If a file is added to the desktop, its metadata is automatically synched to the laptop. Etc.
The desktop stores a superset of the the laptop and nothing else. The laptop always knows all the files on the desktop.
What self-hosted, preferably FOSS, software does this?