Syncthing: Syncing All the Things(lwn.net)
lwn.net
Syncthing: Syncing All the Things
https://lwn.net/Articles/861978/
176 comments
> 2. You can set conditions, such that delet[e]s can be ignored.
I was evaluating Syncthing for 1-way, append-only phone photo backup, and my Google searches warned me away from this, talking about "unsupported" and "database corruption."
I assume you have not experienced this, but I guess I should have just read the docs and trusted them, instead of forum posts.
https://docs.syncthing.net/advanced/folder-ignoredelete.html
I was evaluating Syncthing for 1-way, append-only phone photo backup, and my Google searches warned me away from this, talking about "unsupported" and "database corruption."
I assume you have not experienced this, but I guess I should have just read the docs and trusted them, instead of forum posts.
https://docs.syncthing.net/advanced/folder-ignoredelete.html
I have been using that feature for, I don't even remember how long, but definitely since it was marked beta or something. I am not a heavyweight like some other in this thread, but my current local state is 70GB and ~30k files, global state is ~50GB (I need to prune WhatsApp on phone again) and ~10k files. I am no biggie, but its not a slouch to keep all this working in 7 devices, with overlapping directories shared between them, across 2 timezones, over local and public network. Zero complaints is exactly my experience. Syncthing Just Works™
Great, because even 1 complaint there would be too many, wouldn't it?
I agree. And that's my gripe with Dropbox, Drive, OneDrive (the big 3 that I was able to try, no apple devices). Every one of them has shit the bed on me, on more than one occasion. That's after burning my bandwidth and taking away my privacy. Syncthing just wins out every way (as long as problem scope is file sync, not cloud storage or sharing).
That scared me too. I just move what I want to delete from the original sync folder over to another folder, causing the phone data to be deleted, I even wrote a script that "partitions" that old data, also syncing to some random server after being fully encrypted (that server ofc doesn't have decryption keys).
In practice you could just turn on Trash File Versioning with infinite retention too.
i had some of these problems with earlier versions. removing the database (after making sure everything is on sync) usually solved the issue. but that's years ago.
>I was evaluating Syncthing for 1-way, append-only phone photo backup, and my Google searches warned me away from this, talking about "unsupported" and "database corruption."
Same here. What do you use instead? I haven't been able to find a good open-source multiplatform project for backups/1-way sync.
Same here. What do you use instead? I haven't been able to find a good open-source multiplatform project for backups/1-way sync.
It's not open-source, but I ended up using PhotoSync for myself and my wife to a locked-down samba share on my LAN .
They had a pretty reasonable (~$6) in-app purchase to unlock samba access, and the fact that it's a German GmbH means I'm slightly more confident they're not trying to copy all our photos to a dark data-vault.
Happy user so far, though it took some cross-checking against the samba share feel confident it was backing up reliably (I'm just particular about trust-but-verify :)
https://www.photosync-app.com/home.html
They had a pretty reasonable (~$6) in-app purchase to unlock samba access, and the fact that it's a German GmbH means I'm slightly more confident they're not trying to copy all our photos to a dark data-vault.
Happy user so far, though it took some cross-checking against the samba share feel confident it was backing up reliably (I'm just particular about trust-but-verify :)
https://www.photosync-app.com/home.html
I have several folders in SyncThing set for one-way only sync (from one device or machine to another only, never syncing back). It's easy to set various nifty folder options in the SyncThing web UI and on the Android app. (Dunno about Apple devices, because I own none.)
Excellent idea to sync Whatsapp. Do you use any means to read your Whatsapp chat history on other devices - say your laptop?
There are a few applications that can read WhatsApp database. I used one couple years ago to run some analytics on my chat, but don't remember now. I'm fairly certain it is still around and works, though.
Tangent but: I sometimes feel like programs that just work like this don't get traction because they don't provoke as much discussion, search volume, etc. A complex mess that needs to be constantly babysat creates a cottage industry around it and gets discussed a lot, but something that works is like the quiet kid who sits in the corner and gets A's and that everyone ignores
Kubernetes is today's poster child for this. The Hashicorp stack can do just about all the same things, but it just works so there's no cottage industry of consultants and no market for as-a-service implementations. Why promote something like that?
Kubernetes is today's poster child for this. The Hashicorp stack can do just about all the same things, but it just works so there's no cottage industry of consultants and no market for as-a-service implementations. Why promote something like that?
> programs that just work like this don't get traction because they don't provoke as much discussion.
Agree very much. That's why I never let go of a chance to enavgelise Syncthing and its OSS ilk in other fields.
Agree very much. That's why I never let go of a chance to enavgelise Syncthing and its OSS ilk in other fields.
I can come up with a couple issues related to Syncthing (whether or not it's about Syncthing per se).
1) I like to minimize the number of APT repositories I use. Debian's and Ubuntu's are well behind on Syncthing versions. Every so often (much more rarely than before), they have breaking changes. As such, my phone can no longer sync with my computers. Perhaps the answer is that Debian should just not be packaging it, because they have such a conservative policy. I'm considering downgrading my phone's version, or building the latest version from source. Worst case I may add their APT repo but I'm not eager.
2) At least with the version I've been using, opting for local-only sync isn't so straightforward. It's per-instance, not per-folder. This means that I have to have my home server running it as local-only, even for small things that I wouldn't mind getting synced over the wire. It means that if I want to have a local-only directory shared between my phone and laptop, it has to go through my home server first. Unless there's a better way I don't know about.
Otherwise Syncthing has been pretty great. I like your no-delete thing, I did not consider that. I could definitely use it for photos.
1) I like to minimize the number of APT repositories I use. Debian's and Ubuntu's are well behind on Syncthing versions. Every so often (much more rarely than before), they have breaking changes. As such, my phone can no longer sync with my computers. Perhaps the answer is that Debian should just not be packaging it, because they have such a conservative policy. I'm considering downgrading my phone's version, or building the latest version from source. Worst case I may add their APT repo but I'm not eager.
2) At least with the version I've been using, opting for local-only sync isn't so straightforward. It's per-instance, not per-folder. This means that I have to have my home server running it as local-only, even for small things that I wouldn't mind getting synced over the wire. It means that if I want to have a local-only directory shared between my phone and laptop, it has to go through my home server first. Unless there's a better way I don't know about.
Otherwise Syncthing has been pretty great. I like your no-delete thing, I did not consider that. I could definitely use it for photos.
Just checked the debian issue tracker. Not sure if there is supporting work that could move it along, but you are not alone in wanting the latest release https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=983500
I think it would go against Debian's philosophy to move fast enough to keep up with Syncthing.
[deleted]
>2. You can set conditions, such that delets can be ignored. Eg. I have a WhatsApp message+media backup going back almost 7 years now, ~65GiB. But I don't need all that on my phone. So I just disable syncing delets on my storage. Now my phone can get away with ~3 GiB of whatsapp data (only because I don't delete very often) and I still have complete backup in case I need it.
I would like to know, how are you using Syncthing to create local backups for WhatsApp database/msgstore (crypt14) and media files. You also mention 65GiB as your complete backup ─ have you tested it via a restore? If so, how?
I would like to know, how are you using Syncthing to create local backups for WhatsApp database/msgstore (crypt14) and media files. You also mention 65GiB as your complete backup ─ have you tested it via a restore? If so, how?
> I would like to know, how are you using Syncthing to create local backups for WhatsApp database/msgstore (crypt14) and media files.
Not the parent poster, but I also do this; I just configured Syncthing to share the /storage/emulated/0/WhatsApp (aka /sdcard/WhatsApp) directory. WhatsApp stores its daily local backup on the Backups subdirectory of that directory, and the media files are all in the Media subdirectory of that directory.
> have you tested it via a restore? If so, how?
I have actually used it to move all the WhatsApp and Signal data from an old phone to a new phone. Just have Syncthing synchronize these directories (with the same path) on the new phone before installing WhatsApp and Signal, then install and launch WhatsApp and Signal. When first launched, if that directory already contains a backup, both WhatsApp and Signal ask if you want to restore from that backup. Signal then asks you to type a long backup encryption key you should have written down somewhere, while WhatsApp asks its servers for the backup encryption key.
Not the parent poster, but I also do this; I just configured Syncthing to share the /storage/emulated/0/WhatsApp (aka /sdcard/WhatsApp) directory. WhatsApp stores its daily local backup on the Backups subdirectory of that directory, and the media files are all in the Media subdirectory of that directory.
> have you tested it via a restore? If so, how?
I have actually used it to move all the WhatsApp and Signal data from an old phone to a new phone. Just have Syncthing synchronize these directories (with the same path) on the new phone before installing WhatsApp and Signal, then install and launch WhatsApp and Signal. When first launched, if that directory already contains a backup, both WhatsApp and Signal ask if you want to restore from that backup. Signal then asks you to type a long backup encryption key you should have written down somewhere, while WhatsApp asks its servers for the backup encryption key.
cesarb pretty much described the whole thing.
But to add, the message store the where the texts are stored. I usually keep it all. WhatsApp doesn't care about the media more than you do, so you can delete anything not needed, and now you have full text message history + important media on your new phone.
I have tested it while migrating across 3 devices and half dozen phone factory restores, so I'm fairly confident it still works. There are very ugly restrictions coming in future Android version with scoped storage and whatnot and I'm worried Google is finally going to cut me out. But until then, I keep mine.
But to add, the message store the where the texts are stored. I usually keep it all. WhatsApp doesn't care about the media more than you do, so you can delete anything not needed, and now you have full text message history + important media on your new phone.
I have tested it while migrating across 3 devices and half dozen phone factory restores, so I'm fairly confident it still works. There are very ugly restrictions coming in future Android version with scoped storage and whatnot and I'm worried Google is finally going to cut me out. But until then, I keep mine.
What are the “3 instances where we had something that needed attention, out of which 2 were not syncthing's fault”? I’m curious what rare exceptions there are to watch for.
Unfortunately I don't clearly remember because last one was a while ago, but it was Sync conflicts due to external factors. The one where it was Syncthing's fault was years ago and is very likely fixed now.
I've been using Syncthing for years now, between ~10 Linux servers, Linux desktops, Linux and Windows laptops and Android smartphones, over LAN, wired and wireless Internet.
I'm synchronizing less than 100 MiB of data, but it's changing all the time. I've not yet had a single issue with it. I've a few conflicts every week when the same file is changed in different locations at the same time, but Syncthing keeps all versions of a file until I resolve the conflicts; I can't see how it could handle that better.
Long story short: I highly recommend this tool.
I'm synchronizing less than 100 MiB of data, but it's changing all the time. I've not yet had a single issue with it. I've a few conflicts every week when the same file is changed in different locations at the same time, but Syncthing keeps all versions of a file until I resolve the conflicts; I can't see how it could handle that better.
Long story short: I highly recommend this tool.
Same. I sync my working docs between a Windows PC, a Windows laptop, and a FreeBSD NAS. It just works and that's awesome.
On Windows, SyncTrayzor is the way to go.
On Windows, SyncTrayzor is the way to go.
I just run it in my task scheduler, running out of sight and easily accessible through the WebUI.
What's the process for resolving conflicts?
You may have to manually step through the conflicts menu to resolve them, choosing which device's version to keep. But typically that won't happen unless both device's versions of a file change at the same time.
I have been using syncthing for at least 3 years now. Sync’ing a 7gb directory across 6 devices. I rarely get conflicts, when it happens i just click on the conflicts link via the browser and check which files are conflicting and try to resolve it. Sometimes i have to call a REST action to reset the local db:
Also good to note my setup has 1 introducer which is my NAS (synology) that is 24/7 online which always has the latest changes. Other devices are mostly mobile or turned on at irregular times. For devices without syncthing support like iOS, i rely on exposing the directory on the NAS via webdav. This way i can still access my data on the go.
And my synology creates regular backups/snapshots of my synced directory. So that i can always recover files. Even outside of syncthing.
curl -X POST -H "X-API-Key: xxx" "http://localhost:8384/rest/system/reset?folder=YYYY"
Also good to note my setup has 1 introducer which is my NAS (synology) that is 24/7 online which always has the latest changes. Other devices are mostly mobile or turned on at irregular times. For devices without syncthing support like iOS, i rely on exposing the directory on the NAS via webdav. This way i can still access my data on the go.
And my synology creates regular backups/snapshots of my synced directory. So that i can always recover files. Even outside of syncthing.
On of my favorite program, and it keeps getting better.
You can even set an encryption key per share, and if you want to use an untrusted device in your nodes, you can simply not provide it with the encryption key and it will simply synchronize the encrypted payload across nodes.
So if you have a shady/cheapo VPS, you can use its storage without worrying about the plaintext data being stored there.
You can even set an encryption key per share, and if you want to use an untrusted device in your nodes, you can simply not provide it with the encryption key and it will simply synchronize the encrypted payload across nodes.
So if you have a shady/cheapo VPS, you can use its storage without worrying about the plaintext data being stored there.
Syncthing has re-framed how I think about the file systems of my computers. I used to occasionally plug in my phone to sync pictures and documents to my desktop. Now I don't even think about the distinction between them - the files simply exist on all of my computing devices, and any changes I make automatically propagate to the other devices.
I normally do all my work on my laptop, but I went ahead and set up sync between my work folders and my personal desktop. Syncing ~1tb of work data has been no problem.
I turn off my computers when I'm not home, and my phone runs syncthing-fork on a limited schedule, so I did at one point have some issues with stuff not syncing because devices weren't turned on. So I set up syncthing on a raspberry pi with a big external hard drive plugged in, and it sits next to the router just quietly synchronizing everything in the background.
It's all very set-it-and-forget-it. Just a great piece of software.
I normally do all my work on my laptop, but I went ahead and set up sync between my work folders and my personal desktop. Syncing ~1tb of work data has been no problem.
I turn off my computers when I'm not home, and my phone runs syncthing-fork on a limited schedule, so I did at one point have some issues with stuff not syncing because devices weren't turned on. So I set up syncthing on a raspberry pi with a big external hard drive plugged in, and it sits next to the router just quietly synchronizing everything in the background.
It's all very set-it-and-forget-it. Just a great piece of software.
SyncThing and FreeFileSync make computing so much nicer for me, I would not ever want to go back.
After several attempts, I ended up dividing the SyncThing folders by device storage and level of trust. Simply put, there are things I don't want on my tablet because it has very little storage space, but I still want small things like my todo list and my keypass database there. Then there is stuff I don't mind having on my work laptop, that's mostly stuff like music or personal things that are not sensitive, like desktop wallpapers (so no private stuff I would minder others at work seeing, and no work stuff I would mind being stolen from my other devices). Then there are things I want on my laptops and desktops, but not on my phone, because they're either too big or only useful on a computer, like downloaded software. That's about it, and this way I only have to think about what to put in those folders, but I don't really add new synced folders a whole lot, and when I get a new device, it's self-evident what folders "should" be on it.
FreeFileSync is for stuff I sync manually, e.g. Firefox and Thunderbird profiles, htdocs folder and MySQL database. The only reason is that if I forget to stop Firefox, Thunderbird, or my webserver before starting it elsewhere -- which I should never do, but it can still happen -- and try to sync, I get a much nicer graphical interface to resolve conflicts en masse.
After several attempts, I ended up dividing the SyncThing folders by device storage and level of trust. Simply put, there are things I don't want on my tablet because it has very little storage space, but I still want small things like my todo list and my keypass database there. Then there is stuff I don't mind having on my work laptop, that's mostly stuff like music or personal things that are not sensitive, like desktop wallpapers (so no private stuff I would minder others at work seeing, and no work stuff I would mind being stolen from my other devices). Then there are things I want on my laptops and desktops, but not on my phone, because they're either too big or only useful on a computer, like downloaded software. That's about it, and this way I only have to think about what to put in those folders, but I don't really add new synced folders a whole lot, and when I get a new device, it's self-evident what folders "should" be on it.
FreeFileSync is for stuff I sync manually, e.g. Firefox and Thunderbird profiles, htdocs folder and MySQL database. The only reason is that if I forget to stop Firefox, Thunderbird, or my webserver before starting it elsewhere -- which I should never do, but it can still happen -- and try to sync, I get a much nicer graphical interface to resolve conflicts en masse.
> my keypass database
Aren't you worried that a corrupted keepass database will propagate across your devices and that way you'll lose it? I'm having trouble finding an alternative for git in these cases.
Aren't you worried that a corrupted keepass database will propagate across your devices and that way you'll lose it? I'm having trouble finding an alternative for git in these cases.
Well, I still make regular one-way backups of everything, so I'm happy to cross that bridge should I encounter it. Unless something really freaky happens, SyncThing would probably just complain about the conflict until I resolve it. At least I don't have to worry about silent corruption, i.e. either the file is fine or there is something wrong and keepass won't open it.
You can set syncthing to keep versions of changed files
It's never resulted in a corrupted file but I do have plenty of "conflict" versions of my Keepass DB which is kind of annoying.
And like the other guy said, Syncthing isn't for backups.
And like the other guy said, Syncthing isn't for backups.
> FreeFileSync
I discovered it yesterday and migrated my backup system to FreeFileSync (well, actually I made a double backup with the old and new system). It is _AWESOME_. I finally feel in control of my backups.
I discovered it yesterday and migrated my backup system to FreeFileSync (well, actually I made a double backup with the old and new system). It is _AWESOME_. I finally feel in control of my backups.
Just wait until you realize how fast it is scanning for changes when using a database! IMO, anyway, I'm sure faster things exist, but I've not seen one in combination with a such a sweet GUI and a lightweight application that is really clear about what it does, and usefully extensive in the options for how it should do it. I'm not above the CLI but this one of the areas I like being able to tweak when I'm tired or distracted, too. Can't recommend it enough.
[deleted]
Syncthing can be a little confusing if you are coming from something like Dropbox or WebDAV. It’s not really quite like anything else I’ve used.
And with that having been said, it’s great. It’s maybe more powerful than you’d expect. My NAS for example has a one-way synced folder from a different box for backup, while having a default shared folder with all of my PCs. It’s also great because not all machines have to always be online; it can gracefully handle deferring stuff. For me, since my NAS is connected all the time, I can use it a bit like a central service, syncing between machines that have no overlap in uptime. Simultaneously, if the NAS needs to be down, it doesn’t interrupt syncing between other nodes. And unlike centralized solutions like Dropbox, I’m not limited by my internet connection, since it’s all local... unless I leave the house. In which case Syncthing appears to continue to stay connected, which is really handy.
And with that having been said, it’s great. It’s maybe more powerful than you’d expect. My NAS for example has a one-way synced folder from a different box for backup, while having a default shared folder with all of my PCs. It’s also great because not all machines have to always be online; it can gracefully handle deferring stuff. For me, since my NAS is connected all the time, I can use it a bit like a central service, syncing between machines that have no overlap in uptime. Simultaneously, if the NAS needs to be down, it doesn’t interrupt syncing between other nodes. And unlike centralized solutions like Dropbox, I’m not limited by my internet connection, since it’s all local... unless I leave the house. In which case Syncthing appears to continue to stay connected, which is really handy.
SyncThing is an absolute treasure. I mean I have 20 years of rsync scripts, I've used Unison for real. But SyncThing is next level. The key part is once you have it set up it just runs and runs reliably without any attention. Also it works great on Windows too, including handling differences in filesystem semantics with Linux. Really well done.
I mostly use it to keep Windows' Document folders in sync across various machines (through a Linux server). Windows apps still put files in lots of random places but most of the important stuff is in Documents these days. I also use it to sync GPS traces from GPSLogger on an Android phone to my server. The Android app works great!
The setup of a new node is more complicated than it should be, that's the one product weakness.
I mostly use it to keep Windows' Document folders in sync across various machines (through a Linux server). Windows apps still put files in lots of random places but most of the important stuff is in Documents these days. I also use it to sync GPS traces from GPSLogger on an Android phone to my server. The Android app works great!
The setup of a new node is more complicated than it should be, that's the one product weakness.
I'm one of those who's in the RSYNC school and take snapshot-style backups of my data.
I wonder, how a folder under syncthing that has RSYNC snapshot-style backup data work, which is, how does SYNCTHING deal with hardlinks (I suppose it ignores them and treats them as individual/new files?).
I know that's not what syncthing is for, but I'm just curious.
I believe it treats them as individual/new files. https://forum.syncthing.net/t/handling-of-symbolic-and-or-ha...
In 2014 I tried to reduce my "cloud" usage and went from Dropbox → BitTorrent Sync (now Resilio Sync) → Syncthing. I have it running on all computers, tablets[0], and phones[0] I have/manage. In total, it keeps ~70K files and ~9TB in sync across devices running Android, macOS, Windows and Linux/Ubuntu/Raspberry Pi OS.
I don't remember when was the last time I had to use a flash drive or a cable to move files around.
---
[0] Android. iOS is too restrictive to do this.
I don't remember when was the last time I had to use a flash drive or a cable to move files around.
---
[0] Android. iOS is too restrictive to do this.
[deleted]
Syncthing has a good approach to telemetry. It optionally collects stats and makes them publicly available.
https://data.syncthing.net
https://data.syncthing.net
Muse Group could have taken a lesson from this with their (attempted?) addition of telemetry to Audacity. If the community feels a sense of ownership of the data that's being collected from said community, you'll probably receive less push back.
Note well that in the default config, syncthing grants remote code execution on your machine to the syncthing developers in the form of Solarwinds-style no interaction autoupdate.
A compromise of the centralized release process could steal all of your (and everyone else's!) files by updating to a malicious version automatically with zero interaction.
Set the syncthing binary's file ownership as root and run it as a normal user so it can't get overwritten, and set STNOUPGRADE=1 in the environment to disable this dangerous default behavior.
A compromise of the centralized release process could steal all of your (and everyone else's!) files by updating to a malicious version automatically with zero interaction.
Set the syncthing binary's file ownership as root and run it as a normal user so it can't get overwritten, and set STNOUPGRADE=1 in the environment to disable this dangerous default behavior.
Syncthing has worked flawlessly for me for years. Syncthing-fork on Android also works very well:
https://play.google.com/store/apps/details?id=com.github.cat...
https://play.google.com/store/apps/details?id=com.github.cat...
What's does this offer above and beyond the official version?
In the old days the Syncthing on Android had a pretty clumsy UI; Fork fixed that. But when I evaluated this in June 2021 the main app had improved enough the fork didn't seem necessary any more.
Better handling of sync rules. For example syncthing-fork can have one folder set to sync over mobile data and others to wifi only. Stock client is all or none.
Past threads:
Syncthing: Untrusted (Encrypted) Devices - https://news.ycombinator.com/item?id=27702785 - July 2021 (5 comments)
Open Source Continuous File Synchronization - https://news.ycombinator.com/item?id=27149002 - May 2021 (146 comments)
Syncthing Untrusted Device Encryption - https://news.ycombinator.com/item?id=26424096 - March 2021 (1 comment)
Syncthing is everything I used to love about computers - https://news.ycombinator.com/item?id=23537243 - June 2020 (159 comments)
Do not use Syncthing (2019) - https://news.ycombinator.com/item?id=23116462 - May 2020 (1 comment)
Emacs' Org-Mode and Syncthing = Perfect (2017) - https://news.ycombinator.com/item?id=23058358 - May 2020 (77 comments)
Syncthing: An open source Dropbox replacement - https://news.ycombinator.com/item?id=20466469 - July 2019 (41 comments)
Syncthing graduation day - https://news.ycombinator.com/item?id=18832517 - Jan 2019 (114 comments)
Syncthing Usage Data - https://news.ycombinator.com/item?id=13856552 - March 2017 (119 comments)
Syncthing - https://news.ycombinator.com/item?id=10331031 - Oct 2015 (1 comment)
Syncthing: Open Source Dropbox and BitTorrent Sync Replacement - https://news.ycombinator.com/item?id=7734114 - May 2014 (184 comments)
Syncthing: Untrusted (Encrypted) Devices - https://news.ycombinator.com/item?id=27702785 - July 2021 (5 comments)
Open Source Continuous File Synchronization - https://news.ycombinator.com/item?id=27149002 - May 2021 (146 comments)
Syncthing Untrusted Device Encryption - https://news.ycombinator.com/item?id=26424096 - March 2021 (1 comment)
Syncthing is everything I used to love about computers - https://news.ycombinator.com/item?id=23537243 - June 2020 (159 comments)
Do not use Syncthing (2019) - https://news.ycombinator.com/item?id=23116462 - May 2020 (1 comment)
Emacs' Org-Mode and Syncthing = Perfect (2017) - https://news.ycombinator.com/item?id=23058358 - May 2020 (77 comments)
Syncthing: An open source Dropbox replacement - https://news.ycombinator.com/item?id=20466469 - July 2019 (41 comments)
Syncthing graduation day - https://news.ycombinator.com/item?id=18832517 - Jan 2019 (114 comments)
Syncthing Usage Data - https://news.ycombinator.com/item?id=13856552 - March 2017 (119 comments)
Syncthing - https://news.ycombinator.com/item?id=10331031 - Oct 2015 (1 comment)
Syncthing: Open Source Dropbox and BitTorrent Sync Replacement - https://news.ycombinator.com/item?id=7734114 - May 2014 (184 comments)
Recently, there had been an article in a prominent German IT magazin about using Synthing to create backups from any type of machine on a Raspberry Pi. So last week I tried it out and quickly run into a problem that could have been dangerous. What I tried out was to backup my Firefox and Thunderbird profils with synthing. I switched on simple file versioning to be on the save side even in case that something is accidentally deleted. The problem was that, probably due to interrupted synchronisation sessions, Synthing could not decide whether the file version on the orginal or on the backup machine is the most recent one. In this case it transfered an old file version with some additional intermediate filename extension from the backup to the original machine. So I ended up having doublicates of my email inbox and other folders in my Thunderbird profil that Thunderbird nevertheless recognized as valid inboxes, etc. The problem was easily solved by searching for the intermediate filename extension and deleting all those files manually. This experience suggests to me that it is perhaps not a good idea to use Synthing for anything else than the deliberate exchange of manually managed files in folders only and specifically dedicated to synchronisation.
You should check out the "sender only" and "receiver only" options: https://docs.syncthing.net/users/foldertypes.html
EDIT: added link
EDIT: added link
Thank you for the hint! This looks like as if it should solve the issue.
One of the things I have found very useful is to configure Syncthing to have "sync only once" folders (it sends the file only one time).
In order to do that, I:
- Set a folders in a device to "send only". - Set a folder in a different device to "receive only". - Set the "Ignore delete" advanced option for the desired folders on both devices.
And that's it! I use it mainly to do some manual processing of the received photos and videos with Shotwell (copying and sorting my library into Year/Month/Day directories).
You may run into some issues where you may be promoted to "revert local additions" (I think on the receiver's end)... This mainly happens for two reasons:
- if you deleted a file on the receiver's "sync only once" folder - if you use a single folder where there may be other incoming files from other synced folders.
It is best to avoid the latter scenario; a quick fix for the first one is to make sure to delete the media first in the sender's device's folder and afterwards delete it on the receiver's device's folder.
After that, you can click "revert local additions", and the message should go away.
In order to do that, I:
- Set a folders in a device to "send only". - Set a folder in a different device to "receive only". - Set the "Ignore delete" advanced option for the desired folders on both devices.
And that's it! I use it mainly to do some manual processing of the received photos and videos with Shotwell (copying and sorting my library into Year/Month/Day directories).
You may run into some issues where you may be promoted to "revert local additions" (I think on the receiver's end)... This mainly happens for two reasons:
- if you deleted a file on the receiver's "sync only once" folder - if you use a single folder where there may be other incoming files from other synced folders.
It is best to avoid the latter scenario; a quick fix for the first one is to make sure to delete the media first in the sender's device's folder and afterwards delete it on the receiver's device's folder.
After that, you can click "revert local additions", and the message should go away.
A long time ago, I used CrashPlan for backups. They had support for Linux, and I had a Linux server at the time that ran ownCloud and some other services that needed backups. I pointed CrashPlan at the appropriate directories and let it roll.
Fast forward several years, I needed to restore something (non-critical) and it was a giant pain in the ass. I was able to get it done, but it wasn't immediately clear that I would be able to. Something about CrashPlan not retaining deleted files and some of my paths changing or something.
Anyway, I was so frustrated by the whole experience that I dumped CrashPlan.
I run Syncthing to provide file sync services. A Raspberry Pi acts as the "central server" on my home network. Once a night, an EC2 instance launches, syncs the data up, and then performs an offline backup to S3. I get a report e-mailed to me every night so I can keep tabs on stuff.
I did have to spend time building it out, but it costs something like $5 per month and it's been bulletproof.
Lots of love for Syncthing over here. It's been a champ.
Fast forward several years, I needed to restore something (non-critical) and it was a giant pain in the ass. I was able to get it done, but it wasn't immediately clear that I would be able to. Something about CrashPlan not retaining deleted files and some of my paths changing or something.
Anyway, I was so frustrated by the whole experience that I dumped CrashPlan.
I run Syncthing to provide file sync services. A Raspberry Pi acts as the "central server" on my home network. Once a night, an EC2 instance launches, syncs the data up, and then performs an offline backup to S3. I get a report e-mailed to me every night so I can keep tabs on stuff.
I did have to spend time building it out, but it costs something like $5 per month and it's been bulletproof.
Lots of love for Syncthing over here. It's been a champ.
I love Syncthing. I've found some unconventional uses for it in syncing program settings. My fonts folder is now synced between my Mac, Windows and Linux computers and that works surprisingly well. My projects folder does a one-way archive (like rsync) for backup to my home NAS and an offsite VPS. I have a Streamdeck that I use between my Mac and Windows work machines, and I sync the settings on that so it operates the same way regardless of which machine I plug it into. I love it.
One of my unconventional uses is to sync my HomeAssistant configuration to my desktop, where I have it in a git repository. I can edit manually from my desktop, and/or manage from the HomeAssistant UI, and either way, commit discrete, working changes to git once I'm ready.
Interesting.
I’ve gone through so many iterations of trying to work out how to do something like this for Home Assistant config.
Do you set it up as realtime 2-way sync? And do you just exclude the .git folder from the sync to HA?
I’ve gone through so many iterations of trying to work out how to do something like this for Home Assistant config.
Do you set it up as realtime 2-way sync? And do you just exclude the .git folder from the sync to HA?
This. I'd like to sync my ~/development directory, including all the git directories, so I don't need to push and pull branches between machines. Is this at all feasible?
Yeah been trying to work that one out for years.
I've read of other HN'ers doing this successfully (syncing git repos between machines) but I just seem to get a lot of issues - git repos often broken temporarily, conflicts etc.
I've read of other HN'ers doing this successfully (syncing git repos between machines) but I just seem to get a lot of issues - git repos often broken temporarily, conflicts etc.
Don't know how Syncthing works, of it it would even work, but what if Syncthing would detect that it recurses into a .git folder, and then use git push to push all refs to the remote .git repo? That should give a transparent "sync" of the entire git repo and workdir for the user, but via the safety of git's own sync mechanism.
It works fine. I've been doing it for ages with no problems. Note that I added node_modules, etc. to the ignore file so that it's not always syncing thousands of tiny files, which always tend to take a long time for some reason.
Out of interest, what OS are you using? I'm wondering if some of the issues I've had in the past may have been macOS specific.
That said, I can't recall if I tried excluding node_modules when I last gave this a go. Maybe I'll try it again.
That said, I can't recall if I tried excluding node_modules when I last gave this a go. Maybe I'll try it again.
Yes exactly that, plus it ignores .log, .db and core.restore_state.
I should note I specifically use this for doing source control of configuration and making it easy to manually edit -- I have an entirely separate process for backups of the server itself.
Like you, I've also gone through several iterations but this has been working for me for almost a year now.
I should note I specifically use this for doing source control of configuration and making it easy to manually edit -- I have an entirely separate process for backups of the server itself.
Like you, I've also gone through several iterations but this has been working for me for almost a year now.
Cool, thanks for elaborating. I may give this approach another try.
That said, I’m making far fewer YAML changes these days than I used to. I’m doing almost everything via their UI so maybe I need to consider how much I actually need source control now vs just having solid backups.
Originally I was against their move from YAML config to the more user friendly GUI approach but I have to admit it’s a lot easier to make quick on-the-fly changes to my automations these days. I’m begrudgingly coming around to the idea of not firing up VS Code every time I want to tweak something!
That said, I’m making far fewer YAML changes these days than I used to. I’m doing almost everything via their UI so maybe I need to consider how much I actually need source control now vs just having solid backups.
Originally I was against their move from YAML config to the more user friendly GUI approach but I have to admit it’s a lot easier to make quick on-the-fly changes to my automations these days. I’m begrudgingly coming around to the idea of not firing up VS Code every time I want to tweak something!
My NAS does btrfs snapshotting of the Synchthing folder.
So I have an up-to-date backup of my phone pictures, laptop files and can go back in time if I accidentally delete something.
I sync my android phone to my laptop using syncthing. Works great, and the ignore patterns below prevent annoying access warnings.
Folder Type: Send Only
Ignore Patterns:
Folder Type: Send Only
Ignore Patterns:
Syncthing/**
Pictures/.thumbnails/**
Android/data
Android/data/**
Android/obb
Android/obb/**Has anyone here switched between (from/to) ownCloud/NextCloud and Syncthing? I'm currently using ownCloud on a BananaPro and I'm really not happy with the performance, but also with some bugs that appear to be a result of this low performance. I have considered Syncthing before but I wonder if anyone else has made the same transition. I'm mostly syncing Windows desktops.
Syncthing is a lot narrower in scope than owncloud. It just syncs. If you're using owncloud just as a dropbox replacement, the one thing you're likely to run into is syncthing does nothing to facilitate sharing with untrusted devices, like giving a share link to your friend or something.
Thanks. I mostly use it for synchronizing my various machines and as a backup. Sharing with other people is absolutely secondary and something I can do without.
I like Syncthing, but it is significantly held back by Google's restrictions on running background apps on Android. Every working app has to display a persistent notification, and if you run xmpp client, vpn and Syncthing, on a phone with a notch, you are effectively down to ONE notification icon - it could be two but if you have more than that, they all collapse into an ininformative dot.
Working without a vpn and xmpp is a no-go for me, so I have to sacrifice Syncthing, running it only from time to time.
Working without a vpn and xmpp is a no-go for me, so I have to sacrifice Syncthing, running it only from time to time.
idk if this works for all phones (I hope so!) but on my Pixel, you can long-press on a notification to open settings for that notification type. Changing it to "silent" will move it to the bottom of the list and no longer shows an icon in the status bar. You could also "hide" that notification channel entirely. I'm fairly sure this doesn't affect the ability to run background processes, but I could be mistaken
EDIT: This is a more recent feature so you might need the last one or two versions of Android, but if you have a phone with a notch I bet you're good :)
EDIT: This is a more recent feature so you might need the last one or two versions of Android, but if you have a phone with a notch I bet you're good :)
notifications silenced this way are not guaranteed to be kept in memory, so this roughly equals just stopping the app - especially on crapphones by Samsung and Xiaomi, which are the worst offenders of Android's background functionality.
[deleted]
For iPhone and iPad: MobiusSync (which is a compatible, but non-free implemenation). Works great, just as SyncThing does.
Does it sync photos from the iOS camera roll? I assume not since the screenshots in App Store don’t look like it. But if it does, and keeps the albums, then it would be useful to me. Currently I transfer my photos from my phone to an external drive using the iMazing app on macOS with the phone connected.
Apple makes it surprisingly difficult to synchronize photos outside of their iCloud offering and I'm willing to bet that this is no accident.
Also worth mentioning if it makes a difference, I only need to be able to transfer photos off of the phone (along with info about which albums they belong to as mentioned), and don't need to sync in the other direction onto the phone.
I'm about to pull the trigger of getting a mac mini just to get it all into the syncthing and autobackups using the following workflow:
* iphone -> icloud
* icloud -> mac
* mac -> syncthing to the backup server
Standard backup policies from that point on.
* iphone -> icloud
* icloud -> mac
* mac -> syncthing to the backup server
Standard backup policies from that point on.
In the end this might be the most simple and workable solution. Still feels kind of bad having to roundtrip all the data through the cloud when trying to get it from one local device to another.
I use the app PhotoSync with great success. I was about to setup my old MacBook like you described, but now I don’t have to anymore.
You may be interested in Libimobiledevice that lets you sync an iPhone to Linux. That may work also with a mac?
I just found out about this tool from this (recent) thread: https://news.ycombinator.com/item?id=27758045
I just found out about this tool from this (recent) thread: https://news.ycombinator.com/item?id=27758045
I am paying about EUR 20 / month for my somewhat paranoid backup system that includes Apple, Dropbox, Synology and Backblaze (and excludes depreciation). Worthwhile all things considering, but if Möbius Sync could sync the iOS Photo Stream ("further down the roadmap") I'd pay EUR 100 or more for that. It would make the backups in my house a lot more manageable.
Have you checked PhotoSync? does a pretty good job, although not sure it preserves things like slow mo videos, live photos or metadata…
edit: seems to support live photos and exif metadata… I have a feeling some formats or metadata might be lost though, but not entirely sure…
edit: seems to support live photos and exif metadata… I have a feeling some formats or metadata might be lost though, but not entirely sure…
+1 PhotoSync is pretty neat, have been using it over an year now (to sync with Nextcloud over WebDAV).
Syncthing is a great tool : I've been using it for years for backups, picture synchronization between phone and computer, and as a way to share a KeePass DB.
In all those years, not a single issue arise : you connect a device, decide what to share or not, and then just forget about it
I use syncthing to keep keepass and my private keys in sync across my devices, it's great for that works perfect with easy setup and its cross platfrom.
I tried to use it to sync SASS and webpack between developers and it didn't work, it seems to have an issue with node_modules has too many files for it to deal with and it thinks the directory is hundreds of gb and just stalls with a million years eta.
The versioning is cool too but I found it a bit flakey.
If you need reliable versioning and syncing of lots of files use GIT, if you need easy cross platform syncing syncthing is the way to go.
I tried to use it to sync SASS and webpack between developers and it didn't work, it seems to have an issue with node_modules has too many files for it to deal with and it thinks the directory is hundreds of gb and just stalls with a million years eta.
The versioning is cool too but I found it a bit flakey.
If you need reliable versioning and syncing of lots of files use GIT, if you need easy cross platform syncing syncthing is the way to go.
My rule of thumb is that anything in gitignore should be in syncthing’s list to ignore. There is some work on both sides when doing simultaneous development, but it keeps local files local.
Syncthing is pretty awesome. It's versatile, I use it to sync photos from my phone to more stable storage but also on webservers to keep files in sync across a couple boxes.
TL;DR: give syncthing a try. first time setup is odd, then bam, your files magically sync anywhere with internet.
I carried around my 1GB Data Traveler flash drive with great conviction. Then, a massive 4G SanDisk drive I would 'never fill up'.
Then, when my 16GB Nexus 4 filled up, one day while I was MTPing my old pictures to my computer, someone told me about Dropbox and how it would sync my pictures for me. After living out of Dropbox for all my non-cold-storage files...I hit the free limit. But I was a student and didn't have money to pay per month.
Skydrive gave me 25 GB. And 3 successive phones ate that like candy. So start paying up Mister. ugh.
But then, Syncthing came along here on HN. I installed it onto my phone and pc for camera download. then another, and another, and a separate folder for sharing files with a colleague, then my work SAN got laggy and lost files, so syncthing to keep my work files local and synced between work devices to the rescue.
And now, I don't pay for storage. or I do, but it just happens to come with word 365, but I don't bother using it.
And the flash drives? I'm up to a 256GB Sandisk Pro of some kind, but only for 'just in case' scenarios. Syncthing handles all of my LAN files access needs.
I carried around my 1GB Data Traveler flash drive with great conviction. Then, a massive 4G SanDisk drive I would 'never fill up'.
Then, when my 16GB Nexus 4 filled up, one day while I was MTPing my old pictures to my computer, someone told me about Dropbox and how it would sync my pictures for me. After living out of Dropbox for all my non-cold-storage files...I hit the free limit. But I was a student and didn't have money to pay per month.
Skydrive gave me 25 GB. And 3 successive phones ate that like candy. So start paying up Mister. ugh.
But then, Syncthing came along here on HN. I installed it onto my phone and pc for camera download. then another, and another, and a separate folder for sharing files with a colleague, then my work SAN got laggy and lost files, so syncthing to keep my work files local and synced between work devices to the rescue.
And now, I don't pay for storage. or I do, but it just happens to come with word 365, but I don't bother using it.
And the flash drives? I'm up to a 256GB Sandisk Pro of some kind, but only for 'just in case' scenarios. Syncthing handles all of my LAN files access needs.
I used to use Dropbox as well, but it started freaking out when I had folders with over 500,000 files in them. I started writing scripts to zip up folders first, moved some things off Dropbox, but it would just take up several CPU cores when going through everything. I tried Syncthing, and it had zero issues with the number of files I had. Everything just worked. I was amazed that the free solution was so much better than the commercial solution for which I was paying $10+ each month.
Over long enough timeline, free software almost always gets better, acheiving parity and eventually surpassing the proprietory service. Free software doesn't need to be reinvented and can keep on steadily fixing issues. It happened for gcc, Linux, GNU, Blender, OBS, VLC, Calibre and obviously Syncthing. As long as scope is clear and target is not moving too much, FLOSS will catch it.
I used syncthing for about 5 years but eventually just fell back to mounting centralized storage and dealing with the cases it was slow to access sometimes. There were a few sync conflict issues I ran into over the years once I stopped using it for "core document sync" where it was just <1G and <10000 files (especially when one device was disconnected for a plane flight and came back online with a queue of changes) but the big thing was there were a couple of times I didn't continuously check things were syncing and it turns out I made 2 copies of files with different edits because they didn't see each other or the main node at home went away and I didn't catch it. You just don't run into these problems with a centralized host.
Also it really ate up CPU/memory when I had a lot of files going. It was among the worst "spin up the laptop fans at idle" offenders, even more so than equivalent options like onedrive.
That being said it was really great for a subset of my use case and the problems I had with it border more on problems with the model rather than problems with the tool.
Also it really ate up CPU/memory when I had a lot of files going. It was among the worst "spin up the laptop fans at idle" offenders, even more so than equivalent options like onedrive.
That being said it was really great for a subset of my use case and the problems I had with it border more on problems with the model rather than problems with the tool.
I looked into syncthing and have set up a folder pair as well but does anyone run a separate additional server to do remote access of a specific file they might want to download? I am wondering if there's any clashes to let's say having something like nextcloud folders overlap with the folders used in syncthing?
I am looking for an alternative that helps bridge the gap that is Google File Stream where even in windows it mounts as a drive but streams everything in the background as needed. Plex seems like a clunky way to go about it but I am open to any suggestions. The file stream feature was mainly used to stream videos in an uncompressed way.
I mainly do local sync between android and windows and a rpi. I would love to have file streaming type support for the android and maybe an ios device.
Android has a way to do file streaming built into some of the cloud manager apps but a diy solution might not have remote access.
I am open to any suggestions, I understand that syncthing is about p2p syncing and the model is mainly for whole file copy.
I am looking for an alternative that helps bridge the gap that is Google File Stream where even in windows it mounts as a drive but streams everything in the background as needed. Plex seems like a clunky way to go about it but I am open to any suggestions. The file stream feature was mainly used to stream videos in an uncompressed way.
I mainly do local sync between android and windows and a rpi. I would love to have file streaming type support for the android and maybe an ios device.
Android has a way to do file streaming built into some of the cloud manager apps but a diy solution might not have remote access.
I am open to any suggestions, I understand that syncthing is about p2p syncing and the model is mainly for whole file copy.
Syncthing is great, I've used it for years to sync all my stuff and it has always been solid. I even use it on my kindle fire tablet (there's a Syncthing app in F-droid).
My only annoyance with Syncthing is when I reinstall an OS on one of my machines. Let's say I'm syncing files between my main workstation, two laptops, a phone, and a cloud backup. Now I want to reinstall the OS on one of the laptops. When I install syncthing on that machine, it gets a new ID. I can make it join the sync swarm but all the other members will think it's a new machine and won't trust it so I have to go to each of the other machines and manually remove the "old" laptop and bring in the "new" one.
My only annoyance with Syncthing is when I reinstall an OS on one of my machines. Let's say I'm syncing files between my main workstation, two laptops, a phone, and a cloud backup. Now I want to reinstall the OS on one of the laptops. When I install syncthing on that machine, it gets a new ID. I can make it join the sync swarm but all the other members will think it's a new machine and won't trust it so I have to go to each of the other machines and manually remove the "old" laptop and bring in the "new" one.
There's an option to automagically add devices from other, trusted devices.
I simply save the syncthing config and remove the database, so that the device stays the same, but doesn't retain any sync data
I simply save the syncthing config and remove the database, so that the device stays the same, but doesn't retain any sync data
Yeah, my primary workstation and home server are configured as "introducer" devices, so they'll propagate devices to anything else.
I keep a backup of Syncthing configs and then re-use it. Config.xml can even be manually edited to fix directory paths.
~/.config/syncthing in Linux, the pem files and config.xml. Similarly in Windows too (from C:\Users\<username>\AppData\Local\Syncthing). Even in Android, you can export and back-up. When you are switching to new device, copy from the back-up and then import.
~/.config/syncthing in Linux, the pem files and config.xml. Similarly in Windows too (from C:\Users\<username>\AppData\Local\Syncthing). Even in Android, you can export and back-up. When you are switching to new device, copy from the back-up and then import.
I know there's Syncthing lite, but is there a way to "browse" your synced folders (without actually syncing) and then mark them for syncing to the phone? I'd love it if I could do that because my phone's storage is obviously limited.
Anyone using Syncthing to sync SSH config/keys and (eg) bash/shell profiles between machines?
I’ve gone through various attempts at keeping things in sync across a couple of Macs but always seem to hit issues.. (the issues not being specific to Syncthing)
I’ve gone through various attempts at keeping things in sync across a couple of Macs but always seem to hit issues.. (the issues not being specific to Syncthing)
Sounds like a job for SSH certificates.
See e.g. https://news.ycombinator.com/item?id=20955465 ; https://goteleport.com/blog/ssh-certificates ; and the usual man pages
See e.g. https://news.ycombinator.com/item?id=20955465 ; https://goteleport.com/blog/ssh-certificates ; and the usual man pages
Are certificates supported by any cloud provider or Git hosting platform? They usually only let you add public keys to your account.
> by any cloud provider
I don't care much for cloud providers adding unnecessary restrictions, so I manage my own sshd_config on VMs. I'd like to rent a flat, not a hotel room, thank you very much.
> Git hosting platform
It appears GitLab, one of my platforms of choice, does support this in self-hosted setups[1]. Even if they didn't, it'd be trivial to extend their sshd_config by myself anyway.
1: https://docs.gitlab.com/ee/administration/operations/ssh_cer...
I don't care much for cloud providers adding unnecessary restrictions, so I manage my own sshd_config on VMs. I'd like to rent a flat, not a hotel room, thank you very much.
> Git hosting platform
It appears GitLab, one of my platforms of choice, does support this in self-hosted setups[1]. Even if they didn't, it'd be trivial to extend their sshd_config by myself anyway.
1: https://docs.gitlab.com/ee/administration/operations/ssh_cer...
It is true that you are free to change the config on a VM manually. I included it because it's the only other service I could think of where you might set up SSH authentication.
This is interesting about GitLab, thanks for the pointer. As you mentioned it only appears to be configurable instance-wide, there is no way for a user or organization to set it up for themselves (for example for your gitlab.com organization). That makes it a bad replacement for SyncThing'd private keys as moviuro suggested.
This is interesting about GitLab, thanks for the pointer. As you mentioned it only appears to be configurable instance-wide, there is no way for a user or organization to set it up for themselves (for example for your gitlab.com organization). That makes it a bad replacement for SyncThing'd private keys as moviuro suggested.
I've been using Syncthing for the past few days and it has been working flawlessly. One thing I haven't tried is syncing between Windows and UNIXes (macOS/Linux) - does anyone have experience around this? How do permissions work?
I'm not sure what your concern is. Differences in how user space permissions are handled? You give synctjing read write access and it does its thing.
Either perfectly or not at all depending on what you want to do. You get file contents, modification time, mode bits on *nix’s, and the read only bit on Windows.
So for you-to-you it’s seamless but if you want to sync shared-user directories then you lose most info.
So for you-to-you it’s seamless but if you want to sync shared-user directories then you lose most info.
I have Windows, Android, and Linux devices in my shares, I've yet to encounter issues.
Someone indicated they did have issues if you do a case only rename between a case sensitive (Linux) system and a case insensitive (Windows/Mac) one. I've never done this myself, but I guess that's one to run into. Also the usual caveats with any program, like that you won't be able to sync con.py to a Windows system, regardless of which program you use.
Someone indicated they did have issues if you do a case only rename between a case sensitive (Linux) system and a case insensitive (Windows/Mac) one. I've never done this myself, but I guess that's one to run into. Also the usual caveats with any program, like that you won't be able to sync con.py to a Windows system, regardless of which program you use.
I've used syncthing without problems for years, but then recently had an issue where it seemed to duplicate an entire folder of files and all their contents and whenever I would delete the duplicate it would bring it back.
I was really annoyed with it until I realized, it's actually OSX's fault, because I renamed a folder from uppercase to lowercase, and there's no good way for syncthing to handle that when you have case sensitive servers syncing with insensitive ones. (Linux and OSX in my case.)
Moral of the story is, don't rename a file only changing it's case-- add a dumb character or something so it can sync to more civilized servers properly.
I was really annoyed with it until I realized, it's actually OSX's fault, because I renamed a folder from uppercase to lowercase, and there's no good way for syncthing to handle that when you have case sensitive servers syncing with insensitive ones. (Linux and OSX in my case.)
Moral of the story is, don't rename a file only changing it's case-- add a dumb character or something so it can sync to more civilized servers properly.
Had this issue with git more times than I can count. every time someone commits a file with the wrong casing I just want to quit my job and live in the woods somewhere.
Absolutely top! I always recommend syncthing. I use it to sync and backup data in different situations which are all similar to what others reported here.
One thing I'd like to add is, that syncthing is also configurable in detail. I use it to sync data between a workstation behind restricted network and my laptops (most of $HOME). For this I use ssh-tunnels between the hosts. So syncthing's public discovery would not be necessary. I can directly set the tcp endpoints for syncthing so it find its communictation partners. Nice!
Anyone here who decided to try to sync files at TB scale?
And what about file holes? AFAIK syncthing filles them with zeros. Is this still the case?
One thing I'd like to add is, that syncthing is also configurable in detail. I use it to sync data between a workstation behind restricted network and my laptops (most of $HOME). For this I use ssh-tunnels between the hosts. So syncthing's public discovery would not be necessary. I can directly set the tcp endpoints for syncthing so it find its communictation partners. Nice!
Anyone here who decided to try to sync files at TB scale?
And what about file holes? AFAIK syncthing filles them with zeros. Is this still the case?
How's the sharing situation nowadays? Tried Syncthing a few years back and while excellent and encountering zero problems, the UX for setting a share with someone else was a bit of a hassle compared to Resilio (which has been rock solid for the last decade for me as well). Sharing a single string and be good to go is UX you can't beat. Do you still have to go through the accept-peer-rigmarole with Syncthing, or have they added 'light weight' shares, for, you know, sharing?
That's not really what SyncThing is for. It's not a full replacement for cloud storage like Dropbox.
It could do it thought, if it would support something like unvalidated peers.
I do miss the simplicity of Bittorrent Sync. I can see how this anonymous file sharing could be awkward to add though, as the app would immediately be used for illegal file downloads at scale.
shoutout to DecSync (https://github.com/39aldo39/DecSync) to use Syncthing with your CalDAV/CardDAV contacts/calendars/tasks and sync local RSS feeds across devices. I don't really use nextcloud anymore for stuff that is only for myself and does not need to be shared
For anyone getting into ObsidianMD, Syncthing works great to sync that. I use it to sync the vault folder between my phone, Chromebook and central NAS. Works great. I do ignore the Obsidian .workspace file to avoid extra battery usage.
It's also fantastic for backups from various VMs amd backing up photos from phones to my NAS, etc.
It's also fantastic for backups from various VMs amd backing up photos from phones to my NAS, etc.
Over a year of use here, with about 75GB of my most critical document data across two Mac laptops and an old FreeBSD server. No hiccups. I use Documents.app on my iOS devices to access any of those computers via LAN versions or port-forwarded, out-of-the-house versions, and everything is great. Highly recommend.
The only thing I'm not quite comfortable with is that Syncthing takes significantly more hassle than Resilio Sync to set up. For every folder you need to manually approve every other device on every device, leading to O(kn^2) approvals where k is number of folders and n is number of devices.
Deployed Syncthing for the first time recently with Obsidian notes, setup was seamless on both Android mobile and Linux desktop (systemd), works flawlessly so far. Enormous upgrade over (lazily) using Google Keep for synced notes.
I've been thinking of buying a home NAS and running Syncthing as a replacement for Dropbox/Google Drive. Does anyone have any recommendations? Synology comes up a lot, I might also just use a Raspberry Pi.
Currently running Syncthing on an RPi 4 NAS, in a docker container. Works great between RPi, laptop and desktop.
Haven't tried it while out-and-about, though that's not really my use case.
*RPi4 NAS has one external thumb drive for flash storage and two spinning rust drives, one for storage, one for backup; all connected via USB. Occasionally I have to plug-unplug the spinning rust drives if the system reboots, as the drives go to sleep.
Haven't tried it while out-and-about, though that's not really my use case.
*RPi4 NAS has one external thumb drive for flash storage and two spinning rust drives, one for storage, one for backup; all connected via USB. Occasionally I have to plug-unplug the spinning rust drives if the system reboots, as the drives go to sleep.
Syncthing for desktops / servers "Just works". I've used it between an AWS instance and my home network for years. Highly recommended.
I finally got rid of it because of the awful mobile experience, but frankly I shouldn't have been counting on that anyway.
I finally got rid of it because of the awful mobile experience, but frankly I shouldn't have been counting on that anyway.
On Android it works acceptably. On iOS (at least iPad) you have the proprietary Möbius sync port which is a bit annoying in that it requires a regular push notification to perform any syncing, but with iOS 15 focus modes you can hide it without breaking the app. Only a few months left if you don't want to run the betas.
What problems were you experiencing on mobile?
What problems were you experiencing on mobile?
Do you encrypt the data going up to Amazon or just raw dog it?
I've been doing this with an rpi+ usb3-attached ssd for years (kept upgrading rpis). Syncing across my laptop, phone and server (server backing it up nightly). Worked well for years.
Now moved to a more powerful machine, but the concept is still the same.
This setup allowed me to conveniently stop using google drive.
Now moved to a more powerful machine, but the concept is still the same.
This setup allowed me to conveniently stop using google drive.
You can reuse old PC parts and install TrueNAS on outdated hardware, it will likely be more stable than anything built with Raspberry Pis.
I used to run Syncthing on an old mac mini with two 2.5" HDD running FreeNAS (previous name of TrueNAS).
I used to run Syncthing on an old mac mini with two 2.5" HDD running FreeNAS (previous name of TrueNAS).
Not sure about mac mini, but some old machines might draw considerably more power than rpi. Rpi has a problem with sd cards, but you can make it use them in the read-only mode, in which case rpi becomes a very reliable computing device.
I use unRaid and Nextcloud.
The one thing I miss would be a somewhat more user friendly way to have file versioning along with a UI (doesn't have to be terribly user-friendly, my baseline is dropbox's interface)
Yep, I know that if I chuck stuff into my synced folders it goes around my machines, onto my NAS, and then onto Backblaze.
Check it out every now and then to make sure, but it just kinda works
Check it out every now and then to make sure, but it just kinda works
Is that to a B2 instance? Any chance you could let me know how you did that?
I use rclone
e.g. Synology supports syncing to Backblaze B2 out of the box.
I like syncthing, but when I was trying it on just 3 laptops on the same wifi network, the speed seemed much slower than what I expected the maximum to be.
(Anecdata I know).
(Anecdata I know).
Make sure your firewall is not blocking syncthing ports. You might be using one of the internet proxies instead of your full lan. An easy way to test is to unplug the wifi network from the internet.
Note with Wi-Fi only 1 device can talk at a time and device -> AP -> device counts as 2 devices talking. Now factor in acknowledgements and inter-device sync and you've got a great mash of exactly-what-wifi-wasn't-made-for.
Does it support syncing local folders? For example a game put saves in Documents folder and I would to sync it to my Dropbox folder, is such thing possible?
Not by default. I did that running two instances of the executable and pointing them to different "home" folders.
It works, but it's an hack...
It works, but it's an hack...
Assuming you're on Windows, you could create a hard link between the files or a junction between directories.
https://docs.microsoft.com/en-us/windows/win32/fileio/hard-l...
https://docs.microsoft.com/en-us/windows/win32/fileio/hard-l...
Thanks for pointers. I looked how to create a junction and I found this small app: https://schinagl.priv.at/nt/hardlinkshellext/linkshellextens..., it's a GUI for creating symbolic links/junctions/etc. Which for me is great as I'm not an avid user of cli.
I wonder if it can sync files from a machine in China to a machine in US which doesn’t have great firewall implemented. Anyone tried this?
I was disappointed with the slow sync speed, as in sync 'realization' compared to say Dropbox, but it's great otherwise
It could be due to the time between filesystem checks, which can't be set too low to avoid excessive load. It is set by default to 10 seconds however, so there's room for improvements. The option should be called "fs watcher delay".
It's great. Big thanks to those that run relays.
It's not a backup solution though... you really want to use actual backup software.
It's not a backup solution though... you really want to use actual backup software.
I syncthing all devices/systems onto staging server and just rclone it via encrypt target onto S3-compatible remotes. It is about 4TB. After the initial "long" backup, the hourly runs take minutes
Any third party app that can sync a Synchting folder on iOS (with other Synchting instances on desktop and android)?
Had to stop using it because the devs couldn't fix the SD card issue on Android.
Could you elaborate on the issue?
Android filesystem permissions.
https://github.com/syncthing/syncthing-android/wiki/Frequent...
https://github.com/syncthing/syncthing-android/issues/1008
https://github.com/syncthing/syncthing-android/issues/1366
https://old.reddit.com/r/Syncthing/comments/g9es5d/alternati...
https://github.com/syncthing/syncthing-android/wiki/Frequent...
https://github.com/syncthing/syncthing-android/issues/1008
https://github.com/syncthing/syncthing-android/issues/1366
https://old.reddit.com/r/Syncthing/comments/g9es5d/alternati...
This. I don't believe people who are saying this is an Android issue. It's not. Other sync apps can use the SD card just fine, without root.
Could you expand on that a little?
They support arbitrary locations on external SD cards on Android yet?
SD cards work on Android 11, although I think it's because of the "scooped storage" change Google made, not because Syncthing fixed it. It works as if it was internal storage.
On Android 10 and older, Syncthing can only "read" form SD cards (read/write works if you're rooted).
https://forum.syncthing.net/t/android-11-and-syncthing-what-...
On Android 10 and older, Syncthing can only "read" form SD cards (read/write works if you're rooted).
https://forum.syncthing.net/t/android-11-and-syncthing-what-...
So to be clear, in /old/ versions of Android before they started making apps get permissions per directory, Syncthing can write to an arbitrary SD card folder just fine.
In the newest versions of Android you give it Scoped Storage.
In the intermediate, if you want 2-way sync on your phone, your data has to live in sdcard/android/media/com.nutomic.syncthingandroid/<sync folder1 >, <sync folder 2> which it does not need root to write to.
But for just grabbing your pictures, Android 10 has the stupid limitations on external file system access that Syncthing has to abide by or root over.
In the newest versions of Android you give it Scoped Storage.
In the intermediate, if you want 2-way sync on your phone, your data has to live in sdcard/android/media/com.nutomic.syncthingandroid/<sync folder1 >, <sync folder 2> which it does not need root to write to.
But for just grabbing your pictures, Android 10 has the stupid limitations on external file system access that Syncthing has to abide by or root over.
Two things I like about it in particular, are
1. It can and favors syncing over local network. This has a huge cost savings in developing world. Even when you do have massive bandwidth, local network sync still has more throughput and lower latency. My music is shared between devices and if I add a track, it takes less than a second to appear on my other devices.
2. You can set conditions, such that delets can be ignored. Eg. I have a WhatsApp message+media backup going back almost 7 years now, ~65GiB. But I don't need all that on my phone. So I just disable syncing delets on my storage. Now my phone can get away with ~3 GiB of whatsapp data (only because I don't delete very often) and I still have complete backup in case I need it.
3. Absolute rock solid stable. At $DAYJOB we use syncthing to sync multi dozen GiBs of new data every day, total file count in 10s of thousands in about ~300 directories. And in past 5 years we have had about 3 instances where we had something that needed attention, out of which 2 were not syncthing's fault.
I've been using Syncthing for years and have absolutely zero complaints against it. Everyone with 2 or more devices should give it a try, just to see there are better options out there.