Replacing Dropbox in favor of DigitalOcean spaces(mitjafelicijan.com)
mitjafelicijan.com
Replacing Dropbox in favor of DigitalOcean spaces
https://mitjafelicijan.com/replacing-dropbox-in-favor-of-digitalocean-spaces.html
203 comments
I had a terrible, terrifying experience with DO where they hard locked my account for days (and it would have been weeks if I hadn't made a huge stink) because of a misunderstanding. They ban/lock first, ask questions later, and that's not cool and I can't have my (and my customer's) important infrastructure treated that way.
I now use Linode for most of my stuff, and once they have cloud firewalls available in Dallas I'll be moving the rest of infrastructure over as I can.
I really need to write up my story into a blog post I can link to, but until then, I put a recap here a few days ago for anyone interested in the details. Mine wasn't as bad as some people's, but pretty scary personally: https://news.ycombinator.com/item?id=25806086
I now use Linode for most of my stuff, and once they have cloud firewalls available in Dallas I'll be moving the rest of infrastructure over as I can.
I really need to write up my story into a blog post I can link to, but until then, I put a recap here a few days ago for anyone interested in the details. Mine wasn't as bad as some people's, but pretty scary personally: https://news.ycombinator.com/item?id=25806086
I’ve had only positive experiences with Linode, including incidents where I faced absolutely massive DDoS attacks.
On the other hand, my account is suspended from DigitalOcean, I have multiple droplets they refuse to give the data to, and they even had the audacity to keep billing me until I did credit card chargebacks. I still have no idea why I was banned.
Maybe because I used the GitHub student pack promo but I had already finished uni? I don’t know.
On the other hand, my account is suspended from DigitalOcean, I have multiple droplets they refuse to give the data to, and they even had the audacity to keep billing me until I did credit card chargebacks. I still have no idea why I was banned.
Maybe because I used the GitHub student pack promo but I had already finished uni? I don’t know.
Another happy Linode user here. They might not have the nicest interface, but their service is top notch and I much prefer their UI to Digital Ocean's any day of the week. I've been with Linode for ten years now.
> They ban/lock first, ask questions later
Given the sheer volume of dodgy packets that originate from DO network space, I find this surprising.
Given the sheer volume of dodgy packets that originate from DO network space, I find this surprising.
I guess I’m not terribly surprised. It fits a narrative, not of being overly protective, but of run-of-the-mill incompetence.
Someone once used a DigitalOcean droplet to run an aggressive layer 7 denial of service attack against a service I host. I notified DO via their web form, didn’t get a reply for three days, then got told to email to their abuse@ email address instead.
Dutifully, I mailed abuse@ and included the relevant server logs, gzipped, as an attachment. I got a reply that they won’t open any attachments, please copy and paste the logs directly into the email.
As an outsider, it seems to me as if they have few people with the appropriate technical background staffing their abuse department, so legitimate reports are left uninvestigated due to dumb technicalities like not being willing/able to open attachments, and bogus reports are accepted without thought or understanding as to the appropriate course of action.
Someone once used a DigitalOcean droplet to run an aggressive layer 7 denial of service attack against a service I host. I notified DO via their web form, didn’t get a reply for three days, then got told to email to their abuse@ email address instead.
Dutifully, I mailed abuse@ and included the relevant server logs, gzipped, as an attachment. I got a reply that they won’t open any attachments, please copy and paste the logs directly into the email.
As an outsider, it seems to me as if they have few people with the appropriate technical background staffing their abuse department, so legitimate reports are left uninvestigated due to dumb technicalities like not being willing/able to open attachments, and bogus reports are accepted without thought or understanding as to the appropriate course of action.
They're definitely on the theater side of security. I've got locked out of my account multiple times for no reason.
Furious the second time, I sent a crafted image of an invalid ID to check if they actually even care. It seems they don't, as the account got unlocked all the same.
Furious the second time, I sent a crafted image of an invalid ID to check if they actually even care. It seems they don't, as the account got unlocked all the same.
I think you should make a blog post with screenshots calling them out for this ridiculous hypocrisy.
Of its own the blog post won't do anything, but the next time they're being overeager and shutting down a legitimate customer, you can link to that and corner them to fix their procedures.
Of its own the blog post won't do anything, but the next time they're being overeager and shutting down a legitimate customer, you can link to that and corner them to fix their procedures.
This is the most offensive part to me, I can't even sign up or use DO without having my privacy invaded and sold (they load multiple fingerprinters from different third party vendors that also then share this data) yet they still don't have a handle on abuse.
I get so much rubbish from DO space, nonstop port scanning at absurd volumes (sometimes totaling 6 digit+ pps), from customers that haven't been removed in years. massscan/zmap from people with a fake opt out page (that I shouldn't need to be opting out of), etc.
I get so much rubbish from DO space, nonstop port scanning at absurd volumes (sometimes totaling 6 digit+ pps), from customers that haven't been removed in years. massscan/zmap from people with a fake opt out page (that I shouldn't need to be opting out of), etc.
They flagged my account and pulled 100% of my servers with no mitigation for 12 hours until I convinced them I was legitimate. DO still handles that incredibly poorly, which is the reason we no longer host anything critical there.
Well, this is a discouraging news. Where have you moved your critical operation?
For clarity, is "They" DO or tarsnap?
tarsnap doesn't rent servers
Tarsnap uses AWS S3
Also both tarsnap and S3 have no idea what content they’re storing on your behalf. (Everything is encrypted at rest). So for tarsnap, deplatforming should never be an issue.
I’ve been hearing about this happen for a long time. Unfortunate that it’s still the case.
DO will also use "machine learning" and "AI" to shut down your machines or terminate your account if your usage pattern is disliked.
Has this been happening already? I have been using this script for about 4 months now, and I had no problems yet. I do syncing once a day.
Did they ever post the post mortem? It doesn't sound like they changed policies that automatically block flags accounts.
Yes, it's in that thread a few days later albeit not easy to find.
https://blog.digitalocean.com/an-update-on-last-weeks-custom...
https://blog.digitalocean.com/an-update-on-last-weeks-custom...
I used to use Tarsnap and am still stuck with it in several projects. I wish I weren't. Restoring backups takes ages. If you need to restore a production server in an emergency, and the whole server remains down for an hour because that's how long Tarsnap takes to restore some tens of Gigabytes, then you will very quickly look for alternatives. I raised this issue with Colin, Tarsnap's creator, multiple times. And the answer has always been "maybe some day". I now simply encrypt backups myself and store them in an S3 or Linode bucket. Much faster. For personal backups, I love restic syncing to Backblaze, where I back up 1.4 TB or so for $5 per month.
$0.25 /GB is pretty expensive. I'm using Hetzner Storage Box [1] which has tonnes of connectivity options where 1TB is only 9.40€ (works out at 0.0094€ /GB)
[1] https://www.hetzner.com/storage/storage-box
[1] https://www.hetzner.com/storage/storage-box
I've never heard about Hetzner before, but those are some good prices.
Have they been around for long, and you know of any external public audit that has been done on them?
Have they been around for long, and you know of any external public audit that has been done on them?
Hetzner is (one of) the largest German dedicated server providers.
They are DIN ISO/IEC 27001 certified: https://www.hetzner.com/unternehmen/zertifizierung. Of course if you only store encrypted data you don't really need this auditing.
They are DIN ISO/IEC 27001 certified: https://www.hetzner.com/unternehmen/zertifizierung. Of course if you only store encrypted data you don't really need this auditing.
Yeah I've been with them for 8 years as they're the best value hosting provider [1] I've found, which I use for all my memory or computational heavy workloads as they come at a fraction of the cost of what it would cost on AWS.
[1] https://www.hetzner.com/dedicated-rootserver
[1] https://www.hetzner.com/dedicated-rootserver
Hetzner has been around since 1997 (as you will be able to tell from some of their UI).
Yes, Tarsnap is really quite expensive if you have data that doesn’t get deduplicated well.
> I don't get anything for recommending Tarsnap; I'm just a customer. https://www.tarsnap.com/
From their site:
"The Tarsnap service is built on top of the solid platform provided by Amazon Web Services."
From their site:
"The Tarsnap service is built on top of the solid platform provided by Amazon Web Services."
I would actually compare DigitalOcean to a Facebook or a Google or an Instagram. You’re banned or blocked and cannot reach a human to find out why or what can be done. DO support is just a bunch of auto mailers sending the same template emails, in my experience.
As tarsnap is run by a single person, it seems hard to suggest it to serious businesses due to the inherent risks there as well as the extremely high storage costs and difficult cleanup procedures compared to competing tools. I tend to prefer solutions like restic + B2 for their price and support, where you at least have a larger company behind the storage backend, you could even directly use it with S3 and it'd still be cheaper than Tarsnap.
Decent write up on the cryptography in restic here: https://blog.filippo.io/restic-cryptography/
Some tarsnap issues detailed here: https://www.kalzumeus.com/2014/04/03/fantasy-tarsnap/
Decent write up on the cryptography in restic here: https://blog.filippo.io/restic-cryptography/
Some tarsnap issues detailed here: https://www.kalzumeus.com/2014/04/03/fantasy-tarsnap/
Oh my word, that article on tarsnap is a frightful bore. I was a tarsnap customer and I thought it was great. If Colin is happy running it more like a utility than a rapacious VC-backed hyper-growth racket, I'm not sure why that is bad.
The only reason I stopped being a customer was because another rapacious racket of a business, Comcast, introduced miniscule but enforced data caps in my area, so online backups aren't attractive anymore and I've gone back to external drives and offsite rotation. When I cancelled, Colin sent me a personal e-mail to make sure it was alright to delete my backups. It was probably the best exchange I've ever had with a service provider.
The only reason I stopped being a customer was because another rapacious racket of a business, Comcast, introduced miniscule but enforced data caps in my area, so online backups aren't attractive anymore and I've gone back to external drives and offsite rotation. When I cancelled, Colin sent me a personal e-mail to make sure it was alright to delete my backups. It was probably the best exchange I've ever had with a service provider.
> If Colin is happy running it more like a utility than a rapacious VC-backed hyper-growth racket, I'm not sure why that is bad.
I don't doubt your happiness, but I confess that I'm having trouble reconciling "running it more like a utility" with charging 25¢/GB per month for storage. That is just staggeringly high. What I'm paying $6/month for with Arq would be over $160/month if I were using Tarsnap, and I'm getting end-to-end encryption, deduplication, and versioned file backups. What advantages does Tarsnap bring to the table that justify such a tremendous cost?
I don't doubt your happiness, but I confess that I'm having trouble reconciling "running it more like a utility" with charging 25¢/GB per month for storage. That is just staggeringly high. What I'm paying $6/month for with Arq would be over $160/month if I were using Tarsnap, and I'm getting end-to-end encryption, deduplication, and versioned file backups. What advantages does Tarsnap bring to the table that justify such a tremendous cost?
The whole point of a backup service is that most of the time you don't need it at all, but when you do need it you really need it. From that perspective, it's like paying for insurance.
I'm not a tarsnap customer, but I think what you're paying for is a service built by a literal obsessive genius that will 100% work when the chips are down.
I'm not a tarsnap customer, but I think what you're paying for is a service built by a literal obsessive genius that will 100% work when the chips are down.
I wish things turn around and smaller shops make a comeback. Big is not always better, in fact the service is almost always an afterthought.
"But that won't scale!"
Lots of little ones would scale. And lots of jobs. It worked before...
Unfortunately restic was a no go for me due to not being compatible with B2 keys that only have the permissions readFiles,writeFiles,listBuckets,listFiles (no deleteFiles). I don't want the attacker to be able to delete any backups if the manage to get to the B2 keys.
I believe this is the ticket that would add support for this to restic: https://github.com/restic/caddy/issues/2
I believe this is the ticket that would add support for this to restic: https://github.com/restic/caddy/issues/2
You can work around this by sticking rclone in the middle which does support soft delete: https://pricey.uk/blog/restic-backups-without-delete/
In short, give rclone your keys, (small ~/.rclone config file) tell restic to use an rclone backend and add an extra argument when running restic. (See the blog link above, all explained)
There's an open issue for this which doesn't seem to have moved: https://github.com/restic/restic/issues/2134 - it does have a couple of PRs linked on there which implement proper soft-delete.
In short, give rclone your keys, (small ~/.rclone config file) tell restic to use an rclone backend and add an extra argument when running restic. (See the blog link above, all explained)
There's an open issue for this which doesn't seem to have moved: https://github.com/restic/restic/issues/2134 - it does have a couple of PRs linked on there which implement proper soft-delete.
Nice blog post! That's exactly what I needed earlier last year when I was looking at this stuff :)
Looks pretty straightforward. I am surprised it is not more of a priority to get merged though, it is so essential.
Looks pretty straightforward. I am surprised it is not more of a priority to get merged though, it is so essential.
Tarsnap seems to have become a multi-man operation.
https://github.com/Tarsnap/tarsnap/commits/master
https://github.com/Tarsnap/tarsnap/commits/master
The other committer is his brother.
Does it matter?
Yes. Having family help out on a 1-man show is an entirely different beast than hiring a full-time employee with wages, taxes, and healthcare. One of these two is a life-long trusted human that you've shared numerous life experiences with, the other is not.
Looks like the brother just does the merges? (which are mostly automatic) so IMO best it matters because it's still mostly a one man, or at least a one family operation - i.e not very resilient to random acts of nature (which backups are supposed to protect from)
Having them be two brothers makes for a lower business continuity risk than having it be a business owner + 1 random employee.
When a business becomes serious? Do you need to hire several workers to look legitimate? If that's still not enough, do you need to make yourself a slave of VC?
Only problem I can see is that one man operation is susceptible to the run over by a bus scenario.
However that is still better than a woke VC asking the "owner" to cancel some users.
I would seriously prefer a service like this to be run by a two man team. Or at least have a guarantee that I'll get automatically notified if something happens to a sole operator.
Am I reading that right, that 1TB of storage used would be $256 per month + bandwidth costs?
You should read HN posts from the past where @patio recommended that tarsnap should raise prices even more.
I just did the same calculation and I that looks right
I redid it multiple times to be sure too, that's basically 50 times more expensive than other backup options, crazy!
Wait, their pricing is $0.25 per GB per month, so $250 per month for 1 TB? What the heck?
The price is after tarsnap deduces and compresses, which it is quite good at and costs for 1 TB of real world data are much lower.
https://www.tarsnap.com/deduplication-examples
https://www.tarsnap.com/deduplication-examples
My issue with tarsnap vs Dropbox is entirely workflow. Right now I can treat my Dropbox effectively as an external drive that happens to live on my file system by making everything selectively synced. If I want something backed up, I just drag it into that folder. But once it’s backed up I can turn on selective sync and free up my disk space.
Backup of personal data is often a 1-way endeavor — here’s a dump of photos I scanned that I don’t need to look at anytime soon. But with tarsnap I can’t do that, nor can I then have the ease of browsing to the file and just opening it while it transparently downloads on demand.
Backup of personal data is often a 1-way endeavor — here’s a dump of photos I scanned that I don’t need to look at anytime soon. But with tarsnap I can’t do that, nor can I then have the ease of browsing to the file and just opening it while it transparently downloads on demand.
Damn! Thanks for the info. I don't get these companies. That person should be thanked and not deplatformed.
This seems insanely expensive compared to Restic combined with backblaze
Colin Percival is obviously a smart and very skilled guy but I am mystified why people on HN keep recommending tarsnap, it seems a terrible product for almost any imaginable audience.
If I'm a normal end user, I will probably get a vastly easier to use product at about 1-10% of the price from Backblaze. And if I'm a serious business that can easily afford the > 10x premium and engineering to configure the backup I probably wouldn't want to entrust it to some company with what looks to be a bus factor of one and the apparent technical limitation that a restore might cost me a few DAYS of unanticipated downtime [1].
I mean, sure, if you are some unix nerd wanting to backup your dotfiles and a few small documents (or repos) for maybe around ~$50/year, why not go with some artisanal backup service for HN street cred (and a laudable open source donation policy)? But what other good use cases are there?
Am I missing something obvious?
[1] https://news.ycombinator.com/item?id=25621093
If I'm a normal end user, I will probably get a vastly easier to use product at about 1-10% of the price from Backblaze. And if I'm a serious business that can easily afford the > 10x premium and engineering to configure the backup I probably wouldn't want to entrust it to some company with what looks to be a bus factor of one and the apparent technical limitation that a restore might cost me a few DAYS of unanticipated downtime [1].
I mean, sure, if you are some unix nerd wanting to backup your dotfiles and a few small documents (or repos) for maybe around ~$50/year, why not go with some artisanal backup service for HN street cred (and a laudable open source donation policy)? But what other good use cases are there?
Am I missing something obvious?
[1] https://news.ycombinator.com/item?id=25621093
E.g. relative to the author's script, tarsnap get you automatic deduplication across backups... which, if you're like me and basically generate more and more data, allows you to store lots of historic backups at basically no overhead over just storing today's data.
Also, good encryption and security; e.g. Tarsnap is trusted by Stripe (https://www.tarsnap.com/testimonials.html).
Tarsnap does have real downsides - restores can indeed be slow, bare per-GB cost is high, and "like tar" is not a user interface that everyone will like - but there are definitely upsides, too.
Also, good encryption and security; e.g. Tarsnap is trusted by Stripe (https://www.tarsnap.com/testimonials.html).
Tarsnap does have real downsides - restores can indeed be slow, bare per-GB cost is high, and "like tar" is not a user interface that everyone will like - but there are definitely upsides, too.
That's why so far best idea is to go bare metal with couple of different providers and sync data to multiple servers. In case one provider decides to cancel you, you can still have your data in another place. You can also have a copy at your own server. To have access to all machines as if they were on a local network, you can use tinc-vpn or for convenience ZeroTier.
This is really awesome! I will definitely try this out. Thanks again.
what's the advantage of tarsnap over say, doing an encrypted backup with borg to my gdrive folder? I've never heard of Google random deleting encrypted blobs from people's storage and it's way cheaper
> DigitalOcean does deplatform people... I can suggest Tarsnap
Tarsnap runs on Amazon.
Tarsnap runs on Amazon.
There is a lot of open source file sync software out there (ownCloud, NextCloud, Seafile, Syncthing, ...) that would be much better suited for a self-hosted Dropbox replacement than a single bash script. You are many orders of magnitude more likely to lose your data due to a code bug or unstable network than deplatforming.
I looked into setting up something of the sort for myself a little while ago, but unfortunately the pricing just isn't feasible if you have a large amount of data (say a few hundred GBs). You could easily end up paying more than $100/mo for storage + compute + bandwidth, while Dropbox and others charge ~$6-$10/mo for like 2TB.
I looked into setting up something of the sort for myself a little while ago, but unfortunately the pricing just isn't feasible if you have a large amount of data (say a few hundred GBs). You could easily end up paying more than $100/mo for storage + compute + bandwidth, while Dropbox and others charge ~$6-$10/mo for like 2TB.
Instead of an instance, you could just rent a dedicated server with some mirrored TB of drives.
For example, https://www.server4you.com/dedicated-server has 2x 2TB for $25/mo in St. Louis, MO and unmetered bandwidth.
OVH Cloud in the U.S. has servers with 32GB of RAM and also 2x2TB drives starting at $59/mo and unmetered bandwidth. (both of these are French companies that are expanding into the U.S.) If you are interested in other locations (including Canada), check out ovh.com, kimsufi.com, and soyoustart.com (all part of the enormous OVH global footprint).
If you're in EMEA, or don't mind an extra 100 or so millisecond (SSH works fine), check out OVH and Scaleway (France), and of course Hetzner (Germany). Each of these offer dedicated servers for very reasonable prices, and copious amounts of bandwidth with reasonable pings to almost anywhere.
For example, https://www.kimsufi.com/us/en/servers.xml currently has 500GB hard drive servers with Intel Atoms available for $5/mo in both Canada and France, or bigger hard drives for just a few bucks more. This is OVH's cheapest line of servers, but OVH does a pretty extensive burn-in test before deploying them to you, and they tend to be pretty solid. (keep backups in case a hard drive fails, but that's always a good idea anyway.)
For example, https://www.server4you.com/dedicated-server has 2x 2TB for $25/mo in St. Louis, MO and unmetered bandwidth.
OVH Cloud in the U.S. has servers with 32GB of RAM and also 2x2TB drives starting at $59/mo and unmetered bandwidth. (both of these are French companies that are expanding into the U.S.) If you are interested in other locations (including Canada), check out ovh.com, kimsufi.com, and soyoustart.com (all part of the enormous OVH global footprint).
If you're in EMEA, or don't mind an extra 100 or so millisecond (SSH works fine), check out OVH and Scaleway (France), and of course Hetzner (Germany). Each of these offer dedicated servers for very reasonable prices, and copious amounts of bandwidth with reasonable pings to almost anywhere.
For example, https://www.kimsufi.com/us/en/servers.xml currently has 500GB hard drive servers with Intel Atoms available for $5/mo in both Canada and France, or bigger hard drives for just a few bucks more. This is OVH's cheapest line of servers, but OVH does a pretty extensive burn-in test before deploying them to you, and they tend to be pretty solid. (keep backups in case a hard drive fails, but that's always a good idea anyway.)
I'm a big Hetzner dedicated server fan. I've been using them for 5 years after switching from OVH. Its astounding the amount of hardware and transfer they offer for a extremely reasonable price.
> For example, https://www.kimsufi.com/us/en/servers.xml currently has 500GB hard drive servers with Intel Atoms available for $5/mo in both Canada ...
It says “currently being replenished” for the 2 lowest price packages in Canada
It says “currently being replenished” for the 2 lowest price packages in Canada
I missed out on that deal too. Guess things get hit quickly when put on HN!
I've found Contabo to be even cheaper and despite what some have said, haven't had any issues with them.
You can host Nextcloud at your home using something like NextcloudPi+DDNS (https://nextcloudpi.com) or HomeDrive (https://homedrive.io, which has ngrok-ish end-to-end tunnels integrated), and the monthly cost would be almost $0. :)
It's cheap if you already own the hardware. Even $300 could be 50 months (4 year) of a remote server at the prices suggested by some of the posts in this page.
Furthermore self hosting a backup server at home doesn't protect against burglars and fires. You'll probably lose both your data and their backup.
That's why I prefer a local backup plus a remote one (a combination of git repositories and file storage. )
Furthermore self hosting a backup server at home doesn't protect against burglars and fires. You'll probably lose both your data and their backup.
That's why I prefer a local backup plus a remote one (a combination of git repositories and file storage. )
The golden rule of backups;
3 backups 2 local 1 remote
3 backups 2 local 1 remote
I recently set up Nginx on a DO droplet as a reverse proxy to my home server over Wireguard. It's not $0 but it doesn't directly expose my home IP. And I have an authoritative resolver on the LAN so local clients go directly to the private address.
I would also really like to know how you set this up.
I've been thinking about this kind of setup for my and my family's data, at least.
I've been thinking about this kind of setup for my and my family's data, at least.
Can you share how you did this? I’ve had this idea for awhile but never figured out how to pull it off.
Hetzner managed nextcloud costs $12/month (https://www.hetzner.com/storage/storage-share?country=us).
Yes. Been using Syncthing and it's been great. Only down side is there's no iOS port.
Yep. It probably works quite well even between desktop, laptop and Android devices, but personally I have a home server running one instance. This allows me to always have the latest copies of files available for any of my devices and that's where the backups live too. Probably a Hetzner box or something should be quite cost effective way to have the same benefits.
I heard about this from another recent HN thread https://www.mobiussync.com however I haven’t had a chance to try it yet so I can’t attest to its quality.
Another Syncthing fan here. The software is great. But I used DO spaces for storage and it's pricy. One of my "to do" lists for 2021 is find a cheaper storage solution and move my backups over.
Wasabi is about the cheapest out there. $5/TB/mo with no bandwidth charges. Backblaze is close in pricing and has a better reputation. I use both.
Thanks I'll check them out :)
Wasabi is S3 compatible storage and costs $6/TB per month.
NextCloud and ownCloud can be set up to utilize Amazon S3 or Wasabi storage. This works really well and is very inexpensive. Both my GFs business files and my personal/work files all are backed up Dropbox style for less then $10 a month including web and mobile access.
I found NextCloud a little buggy compared to OwnCloud but it has more features and unfortunately I'm stuck with it for the time being and can't easily switch back to NextCloud. The sync works fine though.
NextCloud and ownCloud can be set up to utilize Amazon S3 or Wasabi storage. This works really well and is very inexpensive. Both my GFs business files and my personal/work files all are backed up Dropbox style for less then $10 a month including web and mobile access.
I found NextCloud a little buggy compared to OwnCloud but it has more features and unfortunately I'm stuck with it for the time being and can't easily switch back to NextCloud. The sync works fine though.
My little nitpick: Wasabi has that 90 days minimum storage policy.
Exactly. I use Nextcloud with Spaces as a backend. It’s been sitting there for months by now.
[deleted]
I wouldn't trust that script too much: there is no error checking at all.
What happens when s3cmd fails and after two months you discover the "Vault sync succeded" emails you have been getting were all illusory?
If the author is reading: please take the time to update your example, including proper error checking (at least "set -eu"), otherwise the people on the internet that are going to copy your script are in for a hard time.
Skeleton of the "incriminated" script:
What happens when s3cmd fails and after two months you discover the "Vault sync succeded" emails you have been getting were all illusory?
If the author is reading: please take the time to update your example, including proper error checking (at least "set -eu"), otherwise the people on the internet that are going to copy your script are in for a hard time.
Skeleton of the "incriminated" script:
#!/bin/bash
[...]
s3cmd sync ...
[...]
notify-send [...] "Vault sync succeded [...]"I haven't looked at the code but the author would benefit from using shellcheck.
https://www.shellcheck.net
https://www.shellcheck.net
I agree with you. I should add a disclaimer that this is still being worked on.
I will add what you proposed.
Thanks for your input.
I will add what you proposed.
Thanks for your input.
Always start all your bash scripts with
set -e
You'll never regret it and might be very very very happy it was there. And if you start with that, then as you work on the shell script, you'll be more likely to make the script be idempotent (checking for files before copying, checking for lines in files before appending/sed'ing etc.) Idempotent >> non-idempotent for "alter the state on an end point" things.
set -e
You'll never regret it and might be very very very happy it was there. And if you start with that, then as you work on the shell script, you'll be more likely to make the script be idempotent (checking for files before copying, checking for lines in files before appending/sed'ing etc.) Idempotent >> non-idempotent for "alter the state on an end point" things.
Also, "unofficial bash strict mode" is terrific, especially if you intend to distribute a script:
http://redsymbol.net/articles/unofficial-bash-strict-mode/
http://redsymbol.net/articles/unofficial-bash-strict-mode/
set -euo pipefail
IFS=$'\n\t'
The options are good, but I don't like the IFS suggestion. Bash has a perfectly good way of dealing with spaces in array elements. Fixing the example in the article is as simple as adding double quotes around the array:
What's more, the double-quoted array solution will work even if the elements contain newlines or tabs.
for name in "${names[@]}"; do
echo "$name"
done
What's more, the double-quoted array solution will work even if the elements contain newlines or tabs.
Adding "-E" should be considered as well, so traps are also applied in functions.
Totally agreed. You can also temporarily turn it off and turn it back on again if you really need to by doing "set +e ... set -e". Sometimes it is more convenient to check errors explicitly, i.e. with test code where the test is expected to fail, but errors should fail fast by default.
I don’t know about never regretting. “set -e” is the right choice the vast majority of the time, but definitely has some gotchas to it. For instance, what does the following script print?
#!/bin/bash
set -e
(
echo "Start of Subshell"
(exit 1)
echo "This shouldn’t run, right?"
) || echo "Subshell failed"[deleted]
Will do. Thanks for this.
Whyyyyyy? Just use rclone.
Funny, I've done the opposite in recent months.
Dropbox's killer feature is 'Online-only' storage IMHO, it allows me to store and access 300GB+ of music and other projects on my laptop without carrying an external HDD with me all the time. Every file is saved as a zero-byte file, which makes the files discoverable even though it's not stored on my disk. When I try to open the file it downloads it automatically, which works surprisingly well. And it's way faster than mounting DigitalOcean Spaces using s3fs for example.
Also Dropbox is cheaper: 12eu for 2TB, compared to DigitalOcean Spaces: 5eu for 250GB. And I can even access my files through the Dropbox API.
Dropbox's killer feature is 'Online-only' storage IMHO, it allows me to store and access 300GB+ of music and other projects on my laptop without carrying an external HDD with me all the time. Every file is saved as a zero-byte file, which makes the files discoverable even though it's not stored on my disk. When I try to open the file it downloads it automatically, which works surprisingly well. And it's way faster than mounting DigitalOcean Spaces using s3fs for example.
Also Dropbox is cheaper: 12eu for 2TB, compared to DigitalOcean Spaces: 5eu for 250GB. And I can even access my files through the Dropbox API.
Love how well Dropbox works.
Sadly, for me, it is too expensive. It is 240 euros per year for a family of 6.
For comparison, the same plan in Google One costs 120 euros per year. Same for Office 365.
If they did 120 euros per year for the family pack I will migrate to them because I like Dropbox better at syncing.
Sadly, for me, it is too expensive. It is 240 euros per year for a family of 6.
For comparison, the same plan in Google One costs 120 euros per year. Same for Office 365.
If they did 120 euros per year for the family pack I will migrate to them because I like Dropbox better at syncing.
(And Office365 actually gives you 1TB per user (so 6TB) and 60 mins of skype per user (so 360 mins), plus online, mobile, and desktop versions of office).
But somehow I have the feeling that Dropbox > Google Drive > OneDrive at syncing. Anybody can confirm?
But somehow I have the feeling that Dropbox > Google Drive > OneDrive at syncing. Anybody can confirm?
You can self host syncing service pretty easily:
Technical, non-experts in computing stuff can do this. Not quite simple enough for the general public though.
https://www.synology.com/en-us/dsm/feature/drive
1) Get a Synology NAS.
2) Install the 'Synology Drive' package
3) set up 'Quickconnect' (~DDNS)
4) Remote backup can be though a number of providers (e.g. Backblaze) for cheap
Technical, non-experts in computing stuff can do this. Not quite simple enough for the general public though.
https://www.synology.com/en-us/dsm/feature/drive
QNAP's Qsync is their file sync service for QNAP NAS owners.
https://www.qnap.com/en/utilities/essentials
https://www.qnap.com/en/utilities/essentials
This is what I use and I love it. Still working out what I want to do for backing it up to the cloud though. It has a very easy built on way to do that too, but im trying to figure out how "trust no one" I am with encryption and what exactly I really want to pay to keep around forever.
I have a QNAP for my whole family (Mom, Dad, Bro, Sis)... they can remote in and save pictures.
I then use cloudsync to backup to GCP nearline. All family pictures are there. Obviously, we don't store stuff that we don't want our mother to see. I take the risk of hacking, but a fire in my house is probably more likely than a hacking of my family photos on GCP. I figure just about everything else is ephemeral.
I then use cloudsync to backup to GCP nearline. All family pictures are there. Obviously, we don't store stuff that we don't want our mother to see. I take the risk of hacking, but a fire in my house is probably more likely than a hacking of my family photos on GCP. I figure just about everything else is ephemeral.
And what happens if somebody breaks in and steals your NAS?
$5 for 250GB of storage and then $20 per TB ($0.02/GB) after that seems a bit steep.
Plans and Pricing
The base rate of a Spaces subscription is $5/month and gives you the ability to create multiple Spaces.
The subscription includes 250 GiB of data storage (cumulative across all of your Spaces). Additional storage beyond this allotment is $0.02/GiB. If you cancel your subscription by destroying all your Spaces, your bill will be prorated hourly.
I think hetzner is about half an expensive (€8 tb/m) and I personally prefer ssh/rsync to http/S3.
I don't know what Hetzner is, but if rsync is available, it's a much better option, particularly if you can do incremental syncing you can get a fairly robust setup with very little code.
>I think hetzner is about half an expensive (€8 tb/m)
What offering is that? Their "storage" page quotes 26 euros for 5TB.
What offering is that? Their "storage" page quotes 26 euros for 5TB.
The BX30 in 7.90 before any country specific tax. But that is the 1tb price as the price per TB is discounted by volume (and you get more snapshots, etc.)
Okay I see now. If you switch VAT region to us and get the biggest size, it works out to 3.9 euro per TB per month.
With prices like that, Microsoft Office 365 subscription that gives you 1TB of OneDrive space, seems like a better deal.
I’ve been happy with Wasabi.
Interesting comparison price when something like Sia is so cheap per TB (4 dollars). Will be interesting to see how it developers.
I tried to use Sia once but I needed to download the entire blockchain. Not a fan of storing 5 GB locally so I can store 1 GB remotely...
It looks they also have S3 support now. Will need to try it out.
https://blog.sia.tech/introducing-s3-style-file-sharing-for-...
https://blog.sia.tech/introducing-s3-style-file-sharing-for-...
I think $20/TB*month is exactly what GCP charges, and is probably on par with what Amazon charges.
True. That's what makes Sia intriguing
I tried, and tried to use Spaces for image hosting related to a site that I am building but it never worked for me. DigitalOcean support was responsive but useless. I got the same basic response over and over again for days on end. I did finally get a response acking my problem and that it would be assigned to an Engineer. The ticket was then closed the next day with no further comment. I re-opened it and asked about the status of the engineer looking at the issue and received back the same response I had already heard nearly a dozen times. I then re-closed the ticket, deleted my Space, and went back to AWS. :(
I had heard good things about DigitalOcean, but I would not use them or recommend them to anyone after that experience. The one primary issue I had was a showstopper but there was a lot of other bugginess.
I had heard good things about DigitalOcean, but I would not use them or recommend them to anyone after that experience. The one primary issue I had was a showstopper but there was a lot of other bugginess.
That’s crazy. I’m using spaces on several projects and never had so much as a blip from them. If you don’t mind what was the showstopper issue for you?
Erroneous 503s. I would get a 503 on every call. Uploading a single file with a single call would immediately 503. I never once got a successful upload. I simply switched the endpoint URL and credentials over to AWS S3 and it worked first time so it was not my code (though I suppose it could have been an issue in the library AWS's node S3 client). My theory was the request was somehow malformed based on what DO was expecting and it was returning a 503 rather than a 400. But the request did work with S3.
Another clickbait, it's a replacement for Dropbox in the same way as a tent is a replacement for a car - you can sleep in both.
Whole thing just looks like poor ad for DigitalOcean
Whole thing just looks like poor ad for DigitalOcean
I never claimed that this is 1 on 1 replacement for Dropbox. Dropbox is much much much more than what I am doing here. And I agree with you on that point. I choose DO because it is very easy to set up and get started with.
But for my needs this is all that I need and it solves my use case.
And regarding DO, its just one of the possible S3 providers out there. One could choose something else.
But for my needs this is all that I need and it solves my use case.
And regarding DO, its just one of the possible S3 providers out there. One could choose something else.
You could also go the route of using Backblaze for storage and Cloudflare for delivery. You'll pay $1.25 for the same 250gb ($0.005/gb) and get free bandwidth through CF.
If you're doing this more "quick and easy developer" style, Backblaze is definitely something worth following up. B2 would be way better for the cost ($0.005/gb). DO Spaces, Linode Object Storage and AWS S3 are all about the same price ($0.02/gb).
Though it's always worth testing based on use case. I use B2 for backups, which works great. But when doing some testing with a more interactive load, I really didn't like how many timeouts I got.
Can you expand on that? I store my backups on B2. Are you saying I can download all data via cloudflare without paying for B2 egress?
Interested in this myself. There appears to be some sort of partnership between Backblaze and CloudFlare
https://help.backblaze.com/hc/en-us/articles/217666928-Using...
https://help.backblaze.com/hc/en-us/articles/360010017893-Ho...
https://help.backblaze.com/hc/en-us/articles/217666928-Using...
https://help.backblaze.com/hc/en-us/articles/360010017893-Ho...
Does this have a limitation? Size or whatever ?
I want something to store peertube videos and serve them. This sounds like too good to be true
I want something to store peertube videos and serve them. This sounds like too good to be true
It is, you didn't step on a free video cdn. Cloudflare probits using video unless you're in Enterprise.
but isnt that using b2 ? is there a specific prohibition ?
Yes. The specific prohibition is that you can't serve video from Cloudflare whatever your backend is (unless Enterprise account). You can read their tos or forum if you need.
Cloudflare limits uploads to 100MB per HTTP request, download bandwidth is pretty much free.
This is cool
I actually really like DO spaces, I use it for hosting a small audio podcast. One downside is they give you NO way to see your bandwidth usage, other than seeing whether you got a bill overage or not. It was pretty surprising to me that they HAVE the information for billing, but don't expose it to the user of spaces in any way.
I also found that to be really bizarre and there’s threads of users requesting usage info going back to over a year ago.
Could you please expand on how you host a podcast and what software you use (or point to resources that can help)? I’m assuming you mean hosting the podcast audio files, not just RSS feeds to the episode files stored elsewhere. Thank you.
If you're just doing simple stuff where bandwidth matters I'd look into OVH/kimsufi, it's cheap unmetered dedicated servers.
> To use this on Windows I suggest using Subsystem for Linux or Cygwin.
Unless you live inside these environments, an easier option would be rclone[1]. I use it on a regular basis to sync large loads to Digital Ocean Spaces, including its bandwidth limiting schedule to avoid saturating the residential uplink at inconvenient times.
[1] https://rclone.org
Unless you live inside these environments, an easier option would be rclone[1]. I use it on a regular basis to sync large loads to Digital Ocean Spaces, including its bandwidth limiting schedule to avoid saturating the residential uplink at inconvenient times.
[1] https://rclone.org
> including its bandwidth limiting schedule
Can you share more about this please?
The biggest (related) issue I run into running rclone from home is hitting the request speed limit on Digital Ocean Spaces. I frequently get the "Slow down!" error when backing up a bunch of small (<1mb) files.
Can you share more about this please?
The biggest (related) issue I run into running rclone from home is hitting the request speed limit on Digital Ocean Spaces. I frequently get the "Slow down!" error when backing up a bunch of small (<1mb) files.
I'd highly suggest Syncthing over this solution - no need to rely on 3rd parties for your data storage, let alone trust them with unencrypted copies of your data, if someone malicious worked at DigitalOcean and wanted to compromise you, it'd be trivial to put a malicious script in your Vault files. Especially if deplatforming is a concern something like Syncthing would avoid any reliance on a 3rd party that has visibility into your data.
I have a 50TB+ NAS in my garage rack. Which I'm not utilizing completely. I wonder if there is a market for renting out some of my storage for someone else's encrypted backups. Something like a website, where I can list my price/gb & connection speed -- or maybe people could swap. Maybe I want to store 5TB off site and I'm willing to let someone else store their 5TB on my server. The website would act as the middleman for introducing the two customers.
Siacoin and Filecoin are basically this; you can store small parts of files under contracts that give you crypto in return. Occasionally you'll get tested by the network to check if it's online, and you have the files you should have. A good use of crypto, though last time I looked the ROI wasn't great as a host.
That sounds more taxing on my connection than what I was thinking. Not really up for 100 connections, but if I'm matched up with 1 or 2 other users then that would seem ok.
What's the difference? The 1 or 2 users could be in a datacentre with all the bandwidth in the world.
From what I understand it doesn't really work like that - files get split in to 30 chunks on different providers, and you only need 10 to restore. Apparently "this means that if 20 out of 30 hosts go offline, a Sia user is still able to download her files."
The amount of liability you'd be taking on would make it absolutely not worth it for 2 customers and a few dollars a month in income.
On the opposite end, I'd trust Google or Dropbox with my encrypted backup much more than a rack in someone's garage on a home internet connection.
On the opposite end, I'd trust Google or Dropbox with my encrypted backup much more than a rack in someone's garage on a home internet connection.
[deleted]
[deleted](1)
I'm a long-time DigitalOcean user and I'm happy with most of their products, but I use Backblaze for S3-compatible storage. It's half the price of Spaces, and I get better transfer rates from my DigitalOcean droplet to B2 than I get to Spaces, even within the same DigitalOcean datacenter.
There are tons of companies offering S3-compatible storage, and there's not much difference between them, other than price. Performance capabilities are similar; they all limit bandwidth. All the ones I've evaluated charge for egress over a certain threshold. Some are more reliable than others. For personal use, there's not a whole lot of difference between them for me.
One really cheap alternative for some use cases is Office 365. For $99/year, you can get a family subscription, which allows six accounts. Each gets 1TB of S3-compatible OneDrive storage. If you can handle it being broken up into six 1TB chunks, it's by far the cheapest storage out there. For comparison, Backblaze is $5/TB/mo, and Office 365 gets you $1.37/TB/mo. I didn't end up using this option because the authentication mechanism was inconvenient for my automated use case and I didn't end up needing so much storage, but it's worth investigating if you're on a budget and don't need automated access via e.g. rclone mounts (or want to take the time to overcome the authentication issues I ran into).
I evaluated about a dozen S3 storage options for both backups and as backing storage for a personal Plex server (of media I own) and eventually settled on Backblaze because it's cheap, easy to use, and I didn't need more than 1-2TB in the end (since not a single one can keep up with streaming video bandwidth needs, I ditched the Plex idea).
There are tons of companies offering S3-compatible storage, and there's not much difference between them, other than price. Performance capabilities are similar; they all limit bandwidth. All the ones I've evaluated charge for egress over a certain threshold. Some are more reliable than others. For personal use, there's not a whole lot of difference between them for me.
One really cheap alternative for some use cases is Office 365. For $99/year, you can get a family subscription, which allows six accounts. Each gets 1TB of S3-compatible OneDrive storage. If you can handle it being broken up into six 1TB chunks, it's by far the cheapest storage out there. For comparison, Backblaze is $5/TB/mo, and Office 365 gets you $1.37/TB/mo. I didn't end up using this option because the authentication mechanism was inconvenient for my automated use case and I didn't end up needing so much storage, but it's worth investigating if you're on a budget and don't need automated access via e.g. rclone mounts (or want to take the time to overcome the authentication issues I ran into).
I evaluated about a dozen S3 storage options for both backups and as backing storage for a personal Plex server (of media I own) and eventually settled on Backblaze because it's cheap, easy to use, and I didn't need more than 1-2TB in the end (since not a single one can keep up with streaming video bandwidth needs, I ditched the Plex idea).
This is really cool! Thanks for sharing - going to give it a go.
Similarly, I took a stab not too long ago at attempting to replace my Dropbox [1] - I opted for an implementation using Git and FSWatch.
Pro: version control for your files
Con: more expensive and takes more storage
[1] https://sidia.li/simple-dropbox-alternative/
Similarly, I took a stab not too long ago at attempting to replace my Dropbox [1] - I opted for an implementation using Git and FSWatch.
Pro: version control for your files
Con: more expensive and takes more storage
[1] https://sidia.li/simple-dropbox-alternative/
Love it. Your solution is much more polished and Dropbox like. Very nice!
Thanks for checking it out - may we both never have to succumb to third party syncing platforms again!
Couldn't agree with you more.
this is absolutely not a dropbox replacement,
this is a script that uploads your files to a cloud shared storage
dropbox client does so many more things (including compression,delta uploads,filehandle watch, incremental backup and so on)
i mean its fine if thats what you want,but that is not a backup solution by any means
dropbox client does so many more things (including compression,delta uploads,filehandle watch, incremental backup and so on)
i mean its fine if thats what you want,but that is not a backup solution by any means
The article, and many of the posts here, seem like a baroque expansion of this (in)famous original response to the Dropbox launch: https://news.ycombinator.com/item?id=8863. I can get off Dropbox by... writing my own shell scripts?
What's sad is that increasingly people are feeling that the price for these easy, convenient services is too high, in terms of privacy loss, data lock-in, and so on. Are we going back to the early 00s, when the tech-savvy people would roll their own solutions and everyone else... just has to lump it? Are hackers just being too idealistic? Or do organizations like Signal show a possible way forward?
What's sad is that increasingly people are feeling that the price for these easy, convenient services is too high, in terms of privacy loss, data lock-in, and so on. Are we going back to the early 00s, when the tech-savvy people would roll their own solutions and everyone else... just has to lump it? Are hackers just being too idealistic? Or do organizations like Signal show a possible way forward?
If you're cool with the spartan interface, why not use S3? Isn't most of the value prop on DO Spaces the CDN component?
With s3cmd you could use it with almost anything supporting S3. I work with AWS every day and am a bit shell shocked by it :) And for personal things I prefer DO. And my needs are low. My whole sync folder is about 5gb.
But yeah, I could use AWS S3. I agree.
But yeah, I could use AWS S3. I agree.
If that's what you are doing why not just go to Wasabi? It is cheaper, storage is the only thing that they do and downloads is free until the GB transferred is under GB stored per month.
I looked at Wasabi not long ago.
> For customers that use Wasabi’s pay-as-you-go pricing model, Wasabi has a minimum monthly charge associated with 1 TB of storage ($5.99/month). If you store less than 1 TB of active storage in your account, your total charge will still be $5.99/month (plus any applicable taxes). See FAQ#4 for more details.
> With Wasabi minimum storage retention policy, minimum number of days are as follows:
> 90 days (default) for customers using Wasabi’s pay-as-you-go pricing model
Wasabi's not intended for small-scale use cases.
> For customers that use Wasabi’s pay-as-you-go pricing model, Wasabi has a minimum monthly charge associated with 1 TB of storage ($5.99/month). If you store less than 1 TB of active storage in your account, your total charge will still be $5.99/month (plus any applicable taxes). See FAQ#4 for more details.
> With Wasabi minimum storage retention policy, minimum number of days are as follows:
> 90 days (default) for customers using Wasabi’s pay-as-you-go pricing model
Wasabi's not intended for small-scale use cases.
It is six dollars a month per TB billed on a GB level with 1TB minimum, with no weird plan tiers with S3 compatible API and no stupid per API call charges, reasonable pull pricing ( either free or cheap ), not related to other accounts.
What, exactly is that level of control of $ spent ( as it is going to be lower than pennies ), that people want while maintaining reasonable reliability, stability, durability and scalability of storage.
What, exactly is that level of control of $ spent ( as it is going to be lower than pennies ), that people want while maintaining reasonable reliability, stability, durability and scalability of storage.
Six dollars a month isn't small scale?
Note, it's not bidirectional sync right now:
> All I need to do is write a Bash script that does the reverse and downloads from remote server to local folder.
> All I need to do is write a Bash script that does the reverse and downloads from remote server to local folder.
True!
All I need to do is reverse parameters.
Documentation states:
s3cmd sync LOCAL_DIR s3://BUCKET[/PREFIX] or s3://BUCKET[/PREFIX] LOCAL_DIR
But I need to test it out before I post that solution.
All I need to do is reverse parameters.
Documentation states:
s3cmd sync LOCAL_DIR s3://BUCKET[/PREFIX] or s3://BUCKET[/PREFIX] LOCAL_DIR
But I need to test it out before I post that solution.
When I read "Replacing Dropbox", reading about a 1 way sync system for backing up a single system isn't what jumps to mind.
The magic of Dropbox is that it sync files across multiple computers and allows multiple people to share files. This doesn't do any of that.
You can use Dropbox for that, but there are definitely cheaper/ more robust solutions.
The magic of Dropbox is that it sync files across multiple computers and allows multiple people to share files. This doesn't do any of that.
You can use Dropbox for that, but there are definitely cheaper/ more robust solutions.
Well to be fair the remote backup is the part that's hardest to recreate with a self-hosted solution.
Is it? I used to do something similar with a couple rsync commands. In fact I think the rsync solution is more robust because you can create incremental backups.
Just having a secure/trustworthy/robust remote location to backup too is somewhat of a hassle. The backup itself is not the tricky bit.
Take a look at duplicity. http://duplicity.nongnu.org/
It does encrypted, incremental backups and can sync to many protocols including S3, DO Spaces, Dropbox, rsync, Mega.co...the list goes on. Super easy to set up. This is my go to backup solution for the cloud.
It does encrypted, incremental backups and can sync to many protocols including S3, DO Spaces, Dropbox, rsync, Mega.co...the list goes on. Super easy to set up. This is my go to backup solution for the cloud.
Duplicity is crazy unreliable. Had so many problems with that from memory ballooning to corruption.
Only thing I’ve found which is even fit for purpose so far is rdiff-backup.
For personal stuff I’m just using iCloud and time machine to local disks.
Only thing I’ve found which is even fit for purpose so far is rdiff-backup.
For personal stuff I’m just using iCloud and time machine to local disks.
Once I "git reset --hard" the wrong repo and lost a lot of code. Had to use Dropbox's rollback feature to get it all back. Was not the first time Dropbox saved my butt. For me, a comparable alternative would need to have the same rollback features.
[deleted]
I must say, I concur with this person’s assessment of the Goog-exit feeling. I ditched GMail for Fastmail and my own domain last week. I had been putting this off for years because it seemed like such a massive pain in the ass and I was afraid that I would just end up with two email services.
In the end, it took me one evening to update every service that I cared about to use my new address. The rest (mostly e-commerce companies) I’ve been doing as I use them.
In the end, it took me one evening to update every service that I cared about to use my new address. The rest (mostly e-commerce companies) I’ve been doing as I use them.
Been thinking seriously about this.
Do you just inform your contacts with an auto responder?
How long do you intend to keep the gmail account?
Do you just inform your contacts with an auto responder?
How long do you intend to keep the gmail account?
I didn't do any auto-responders. Frankly, email is kind of dead for p2p comms with friends so there's not many people to inform. Most difficult part I've encountered was getting my dad, who is old, on board. He keeps emailing me at my GMail.
Why didn't he mention the cost? How does digital ocean compare to Hetzner's 1$/100GB/month snapshot offering?
I stopped using Dropbox as soon as C. Rice was put on the board. Tried tarsnap which was great did everything I wanted but I ended up using Arq instead, have had zero issues with it.
https://www.arqbackup.com/
https://www.arqbackup.com/
As a test case, i use box.com in combination with rclone and have daily backups for db and data of my servers.
It copies remotely to /{day of the month} + /{latest}
I could easily change my script to run every hour.
It works pretty nicely and i have a backup for 30 days
( Outside of the default backup daily ofc)
It copies remotely to /{day of the month} + /{latest}
I could easily change my script to run every hour.
It works pretty nicely and i have a backup for 30 days
( Outside of the default backup daily ofc)
All this kind of stuff just feels like cosplay unless you are gonna build out your own tier 1 (that also doesn't need other tier 1s)
Which isn't to say you can't learn a lot of useful practical stuff from it, just like you can from cosplay
Which isn't to say you can't learn a lot of useful practical stuff from it, just like you can from cosplay
> All I need to do is write a Bash script that does the reverse and downloads from remote server to local folder. This could be another post.
A backup strategy without a recovery strategy already in place isn't a backup strategy.
A backup strategy without a recovery strategy already in place isn't a backup strategy.
I went down a bit of a rabbit hole of Digital Ocean and their "security" for production workloads.
> Show me any other vps provider that silently provides access to customer A's data to customer B after receiving commands from customer A to destroy their instance and then I'll believe you guys aren't at the very bottom of the "takes security seriously" list.
From: https://github.com/fog/fog/issues/2525#issuecomment-31337481
YC News Discussion: https://news.ycombinator.com/item?id=6983097
> You do not need to scrub or write anything to not provide user A’s data to user B in a multi-tenant environment. Sparse allocation can easily return nulls to a reader even while the underlying block storage still contains the old data. ... On top of all of that, when I pointed out that what they were doing was absolute amateur hour clownshoes, they oscillated between telling me it was a design decision working as intended (and that it was fine for me to publicize it), and that I was an irresponsible discloser by sharing a vulnerability.
From: https://news.ycombinator.com/item?id=20091026
> You've got an additional problem though, which is that this tells us you have two support channels: one that doesn't work (i.e. yours, the one you built), and one that does (Twitter-shaming). The first channel represents how you act when no one's watching; the second, how you act when they are. Most people prefer to deal with people for whom those two are the same.
From: https://news.ycombinator.com/item?id=20064169
Speaking of randomly locking accounts, the post-mortem kills me:
> The initial account lock and resource power down resulted from an automated service that monitors for cryptocurrency mining activity (Droplet CPU loads and Droplet create behaviors). These signals, coupled with a number of account-level signals (including payment history and current run rate compared to total payments) are used to determine if automated action is warranted to minimize the impact of potential fraudulent high-cpu-loads on other customers.
From: https://www.digitalocean.com/blog/an-update-on-last-weeks-cu...?
In other other words, DO will kill your account with a curt email staring simply: "We have reviewed your account and have declined to activate it. No further information or action is required from you." for simply using "too much CPU"! https://pbs.twimg.com/media/D76ocofXoAY_xB5.png
> Show me any other vps provider that silently provides access to customer A's data to customer B after receiving commands from customer A to destroy their instance and then I'll believe you guys aren't at the very bottom of the "takes security seriously" list.
From: https://github.com/fog/fog/issues/2525#issuecomment-31337481
YC News Discussion: https://news.ycombinator.com/item?id=6983097
> You do not need to scrub or write anything to not provide user A’s data to user B in a multi-tenant environment. Sparse allocation can easily return nulls to a reader even while the underlying block storage still contains the old data. ... On top of all of that, when I pointed out that what they were doing was absolute amateur hour clownshoes, they oscillated between telling me it was a design decision working as intended (and that it was fine for me to publicize it), and that I was an irresponsible discloser by sharing a vulnerability.
From: https://news.ycombinator.com/item?id=20091026
> You've got an additional problem though, which is that this tells us you have two support channels: one that doesn't work (i.e. yours, the one you built), and one that does (Twitter-shaming). The first channel represents how you act when no one's watching; the second, how you act when they are. Most people prefer to deal with people for whom those two are the same.
From: https://news.ycombinator.com/item?id=20064169
Speaking of randomly locking accounts, the post-mortem kills me:
> The initial account lock and resource power down resulted from an automated service that monitors for cryptocurrency mining activity (Droplet CPU loads and Droplet create behaviors). These signals, coupled with a number of account-level signals (including payment history and current run rate compared to total payments) are used to determine if automated action is warranted to minimize the impact of potential fraudulent high-cpu-loads on other customers.
From: https://www.digitalocean.com/blog/an-update-on-last-weeks-cu...?
In other other words, DO will kill your account with a curt email staring simply: "We have reviewed your account and have declined to activate it. No further information or action is required from you." for simply using "too much CPU"! https://pbs.twimg.com/media/D76ocofXoAY_xB5.png
Is there anything commercial that is both cheap and popular that doesn't cut corners to get there?
edit: Point being, I don't see how DO could get their act together as their business model essentially requires them to act like this. If they don't cut corners in one area then they will somewhere else.
edit: Point being, I don't see how DO could get their act together as their business model essentially requires them to act like this. If they don't cut corners in one area then they will somewhere else.
They don't have to treat the customers as the enemy. They do so by choice.
The root cause of the "DO killed our business" situation was that DO accepts credit cards for payment but provides "cash equivalent" as the product: Rented CPUs can be used crypto mining. But from the "outside" it's practically impossible to distinguish between crypto mining and legitimate high CPU usage, they "look" the same.
Hence, DO felt that they were "forced" to kill everyone that signs up with a credit card and starts their CPU at 100% load, because a significant percentage of the time that's some hacker using a stolen credit card number to mine Bitcoins.
DigitalOcean feels forced to treat customers as the enemy, because a significant percentage is the enemy.
However, this business model of "we give you cash for credit cards" is their choice! Nobody forced them to do this. Credit cards aren't legally mandated tender.
Not every shopping centre till provides a cash out service. Some take on this risk. Some don't. I've noticed that less well-to-do areas of my city don't offer this service, but the more posh areas with lower risk do offer cash out. Does the shopping centre get burnt by this sometimes? Yes, but that's their risk to bear.
Digital Ocean is like a shopping centre in a low-rent district full of drug addicts that offers cash out. That's their choice to accept that much higher level of risk. What they're doing however is passing on the consequences to their customers. They're not bearing the risk. It's your risk to bear that other customers mine crypto on DO droplets.
So what alternative does DO have, you ask? They could just ask customers to transfer money up-front, much like pre-paid SIM cards. Many other companies operating in high-risk customer markets do this kind of thing. Heck, Azure and AWS do exactly this: They offer enormous discounts if you pre-pay. Absolutely huge, like up to 50% or even 80% off! See: https://azure.microsoft.com/en-us/pricing/reserved-vm-instan...
So this is what DigitalOcean should do: When you sign up, on the next page after you've created your account it should give a full-page choice screen with two columns:
Column A: "Prepay for savings and guaranteed service levels" -- blah blah blah, we won't randomly kill your account, and you get a 20% discount!
Column B: "Credit card sign up" -- blah blah, ideal for dev/test, ideal for non-critical workloads, we may suspend your account if the usage exceeds certain thresholds, etc...
The root cause of the "DO killed our business" situation was that DO accepts credit cards for payment but provides "cash equivalent" as the product: Rented CPUs can be used crypto mining. But from the "outside" it's practically impossible to distinguish between crypto mining and legitimate high CPU usage, they "look" the same.
Hence, DO felt that they were "forced" to kill everyone that signs up with a credit card and starts their CPU at 100% load, because a significant percentage of the time that's some hacker using a stolen credit card number to mine Bitcoins.
DigitalOcean feels forced to treat customers as the enemy, because a significant percentage is the enemy.
However, this business model of "we give you cash for credit cards" is their choice! Nobody forced them to do this. Credit cards aren't legally mandated tender.
Not every shopping centre till provides a cash out service. Some take on this risk. Some don't. I've noticed that less well-to-do areas of my city don't offer this service, but the more posh areas with lower risk do offer cash out. Does the shopping centre get burnt by this sometimes? Yes, but that's their risk to bear.
Digital Ocean is like a shopping centre in a low-rent district full of drug addicts that offers cash out. That's their choice to accept that much higher level of risk. What they're doing however is passing on the consequences to their customers. They're not bearing the risk. It's your risk to bear that other customers mine crypto on DO droplets.
So what alternative does DO have, you ask? They could just ask customers to transfer money up-front, much like pre-paid SIM cards. Many other companies operating in high-risk customer markets do this kind of thing. Heck, Azure and AWS do exactly this: They offer enormous discounts if you pre-pay. Absolutely huge, like up to 50% or even 80% off! See: https://azure.microsoft.com/en-us/pricing/reserved-vm-instan...
So this is what DigitalOcean should do: When you sign up, on the next page after you've created your account it should give a full-page choice screen with two columns:
Column A: "Prepay for savings and guaranteed service levels" -- blah blah blah, we won't randomly kill your account, and you get a 20% discount!
Column B: "Credit card sign up" -- blah blah, ideal for dev/test, ideal for non-critical workloads, we may suspend your account if the usage exceeds certain thresholds, etc...
I would pin this post to the top if I could. Thanks for this info!
What kind of digital ocean plan did you get for your setup ? I've been thinking about putting together something similar but the price to get a reasonable amount of space seemed prohibitive.
I choose Spaces which costs $5 a month. And you don't actually need VPS for this.
https://www.digitalocean.com/products/spaces/
https://www.digitalocean.com/products/spaces/
You may also be interested in rclone if you're not aware of it already.
Dropbox does a great job syncing a large number of small files -- I believe they compress the small files somehow before uploading to the cloud. And plenty of other optimizations that make me happy to pay Dropbox's premium. Otherwise, Google Drive and OneDrive are cheap alternatives too.
High efficiency and very high performance trustless backup setup: send incremental (encrypted, compressed) ZFS snapshots to rsync.net, hetzner, or buyvm. This produces smaller backups than rsync or similar tools, produces them faster, and also shields your data from prying eyes, as it’s all encrypted when it leaves your machine.
Rsync.net also supports borg backups at a cheaper price than ZFS: https://www.rsync.net/products/borg.html
For your secure backups, I can suggest Tarsnap by Colin Percival, who's a security expert and a frequent contributor on Hacker News. I don't get anything for recommending Tarsnap; I'm just a customer. https://www.tarsnap.com/