Protect your data from ransomware with S3 Object Lock(blog.symops.com)
blog.symops.com
Protect your data from ransomware with S3 Object Lock
https://blog.symops.com/2022/07/07/prevent-ransomware-s3-object-lock/
22 comments
A less permanent solution is to use versioned buckets with MFA Delete turned on. You can then cleanup versions if you need to by disabling MFA delete, which requires the MFA to do. So as long as your MFA device is not on-line, then if someone compromises your servers, they cannot disable MFA delete and cannot remove versioned objects.
Object Lock may be useful to protect backups from deletion, but ransomware is now relying on the threat of data exposure more, where Object Lock makes no difference:
https://www.theregister.com/2022/06/25/ransomware_gangs_exto...
"Increasingly, however, cybercrime rings still tracked as ransomware operators are turning toward primarily data theft and extortion – and skipping the encryption step altogether. Rather than scramble files and demand payment for the decryption keys, and all the faff in between in facilitating that, simply exfiltrating the data and demanding a fee to not leak it all is just as effective. This shift has been ongoing for many months, and is now virtually unavoidable."
https://www.theregister.com/2022/06/03/fbi_cisa_warn_karakur...
https://www.theregister.com/2022/06/25/ransomware_gangs_exto...
"Increasingly, however, cybercrime rings still tracked as ransomware operators are turning toward primarily data theft and extortion – and skipping the encryption step altogether. Rather than scramble files and demand payment for the decryption keys, and all the faff in between in facilitating that, simply exfiltrating the data and demanding a fee to not leak it all is just as effective. This shift has been ongoing for many months, and is now virtually unavoidable."
https://www.theregister.com/2022/06/03/fbi_cisa_warn_karakur...
There is a difference for the business though – although the data may be exposed, the business may still be able to maintain continuity. I believe it is the lack of business continuity that has made ransomware so powerful – when a business can no longer function it will do anything to get that back. If the business could continue to operate they would be in a much better position to refuse to pay the ransom. If all businesses refused to pay the ransoms, ransomware would stop.
Obviously it’s still really bad if sensitive information is exposed. But also consider that some of the information essential for business continuity would be less sensitive in a public exposure scenario.
So in some cases it is just as effective, but in many cases it is not. As I understand it, most ransomware providers still attempt both encryption and exfiltration. Exfiltration is now standard not because it is easier but because more companies are able to restore operations from backup.
Obviously it’s still really bad if sensitive information is exposed. But also consider that some of the information essential for business continuity would be less sensitive in a public exposure scenario.
So in some cases it is just as effective, but in many cases it is not. As I understand it, most ransomware providers still attempt both encryption and exfiltration. Exfiltration is now standard not because it is easier but because more companies are able to restore operations from backup.
> No one, including AWS can shorten the retention period
I don't think this is true. Isn't it more like you can't, and AWS can but promises that they won't?
I don't think this is true. Isn't it more like you can't, and AWS can but promises that they won't?
Yes, AWS must still be able to delete the data if you stop paying but there must be a LOT of safeguards in place so Object Lock still makes it impossible for anyone outside of AWS to delete the data.
I made this a part of the backup schemes at a startup. Everything (GitLab, PostgreSQL, the biz people's document&storage SaaSes, etc.) got backed up multiple ways, including to S3 objects with long retention periods.
So long as the data wasn't corrupted before it hit one of the designated S3 buckets, the risks of the retention periods seemed minor for that startup at the time: basically, having data there that we wanted to delete, but couldn't.
(For an example risk of not being able to delete, the most likely scenario might be accidentally copying a gazillabytes of objects to the magic bucket that can't be emptied for X months, so having to pay for that storage. For a different scenario, the nature of our business, together with our security assurances and practices, meant that we never handled data that would be improper to store there, such that we urgently needed to delete it. In a highly unlikely scenario of an attack putting very evil data there, we'd have to report the incident to government authorities in any case, and we could effectively cut off our own access to it, while AWS preserves gov't access to it.)
So long as the data wasn't corrupted before it hit one of the designated S3 buckets, the risks of the retention periods seemed minor for that startup at the time: basically, having data there that we wanted to delete, but couldn't.
(For an example risk of not being able to delete, the most likely scenario might be accidentally copying a gazillabytes of objects to the magic bucket that can't be emptied for X months, so having to pay for that storage. For a different scenario, the nature of our business, together with our security assurances and practices, meant that we never handled data that would be improper to store there, such that we urgently needed to delete it. In a highly unlikely scenario of an attack putting very evil data there, we'd have to report the incident to government authorities in any case, and we could effectively cut off our own access to it, while AWS preserves gov't access to it.)
I use backblaze which will backup any modified file and has copies back 30 days, longer if desired. It would be great if there was some way (is there?) To flag if a file was recently encrypted to detect potential ransomware early before backups are corrupted also. Seems like a changed file being backed up could be compared to the previous unencrypted file for this purpose.
Of course this would be more complicated if the files were user-encrypted in the first place, but good tools to detect these things asap would have a good market above & beyond antivirus that is more focused on detection of the virus and not a complete packaged of file recovery as well.
Of course this would be more complicated if the files were user-encrypted in the first place, but good tools to detect these things asap would have a good market above & beyond antivirus that is more focused on detection of the virus and not a complete packaged of file recovery as well.
From my understanding, you can't delete a bucket that has objects and I assume you can't empty a bucket that has locked objects, but what happens if you close the entire account?
AWS recommends to use accounts for segmentation, and has APIs to manage accounts (https://docs.aws.amazon.com/wellarchitected/latest/security-..., https://docs.aws.amazon.com/cli/latest/reference/organizatio...).
There is a 90 day recovery period, but I can think edge cases where this isn't enough. There is some data that companies need exactly once a year for financial reporting/audits etc. but losing it is seriously expensive. Imagine the person managing that data being disgruntled and closing the account holding the data (and just that data, so the deletion isn't noticed) 6 months before the next audit.
Can you close an account with locked data in it? If yes, is the data actually deleted, or could the org recover it?
Another interesting scenario that could test how reliable the lock is would be a griefer (i.e. an actor that wants to cause damage without profiting from it) who gets access to an organization's account, uploads a large amount of data, and locks it. Will Amazon simply keep the data and waive the cost, or will support unlock the data, or will the organization be forced to pay up? The latter two both have interesting implications (compromised support agents and social engineering in the first case, extortion in the second case - think "we have taken over your AWS account and created several paths of access, send XX bitcoin or we'll lock this exabyte of data and you'll pay $XXXXXXXX in AWS fees, if you start taking away our access or deleting the data we'll see before you finish and use one of the remaining accounts to lock the data").
AWS recommends to use accounts for segmentation, and has APIs to manage accounts (https://docs.aws.amazon.com/wellarchitected/latest/security-..., https://docs.aws.amazon.com/cli/latest/reference/organizatio...).
There is a 90 day recovery period, but I can think edge cases where this isn't enough. There is some data that companies need exactly once a year for financial reporting/audits etc. but losing it is seriously expensive. Imagine the person managing that data being disgruntled and closing the account holding the data (and just that data, so the deletion isn't noticed) 6 months before the next audit.
Can you close an account with locked data in it? If yes, is the data actually deleted, or could the org recover it?
Another interesting scenario that could test how reliable the lock is would be a griefer (i.e. an actor that wants to cause damage without profiting from it) who gets access to an organization's account, uploads a large amount of data, and locks it. Will Amazon simply keep the data and waive the cost, or will support unlock the data, or will the organization be forced to pay up? The latter two both have interesting implications (compromised support agents and social engineering in the first case, extortion in the second case - think "we have taken over your AWS account and created several paths of access, send XX bitcoin or we'll lock this exabyte of data and you'll pay $XXXXXXXX in AWS fees, if you start taking away our access or deleting the data we'll see before you finish and use one of the remaining accounts to lock the data").
It's much harder to hide the fact that you have closed an account completely and much easier to put in a periodic check that the account still exists.
The difficulty with ransomware is that—without object lock—you would need to check that all data is still valid. That is usually going to be very difficult to do and any heuristic checks are liable to miss some cases.
On the other hand, checking that an account is still extant is easy and, since that's an operation that should not be undertaken without a whole oversight process, you can significantly limit who has the permission to do so.
The difficulty with ransomware is that—without object lock—you would need to check that all data is still valid. That is usually going to be very difficult to do and any heuristic checks are liable to miss some cases.
On the other hand, checking that an account is still extant is easy and, since that's an operation that should not be undertaken without a whole oversight process, you can significantly limit who has the permission to do so.
Oh, I totally agree that this solves at least 99% of the problem. I'm just wondering if this is an edge case that remains.
One thing missing from this great introduction is how to enforce object lock for new objects. This can be done by setting a default retention mode and period on the bucket and having a bucket policy for s3:PutObject(Retention). It looks like I should do a writeup on this. I am working on tooling to make using Object Lock (to prevent ransomware) more convenient.
What happens if you lock objects then stop paying your bill?
AWS must be able to delete the data but since you stopped paying it must not be important to you anymore. No one outside of AWS can delete the data and presumably no one inside AWS can delete Object Locked data IF your bill is payed.
I'm almost sure S3 keeps data and eats the cost. Although, my understanding is from 7 years ago when I worked on the team that would have implemented it.
If they do, then it probably would require a human to run a tool. It would have to be big bill to warrant that.
If they do, then it probably would require a human to run a tool. It would have to be big bill to warrant that.
Pure speculation but the semantics of and usual purpose of “legal hold” strongly imply that keeping the data even with an unpaid bill would be the correct implementation. Even at much smaller scales contracts state fixed data retention or escrow schemes beyond the end of the contract precisely to make legal discovery easier
Edit: went looking for some evidence that supported this (at least in the stricter compliance mode) but found none in the docs. I found a 3rd party audit report supporting the use of this mode for SEC/FINRA etc regulated data and it says the entity is still responsible for maintaining its account in good standing but goes into no further details on what AWS does if you don’t.
Edit: went looking for some evidence that supported this (at least in the stricter compliance mode) but found none in the docs. I found a 3rd party audit report supporting the use of this mode for SEC/FINRA etc regulated data and it says the entity is still responsible for maintaining its account in good standing but goes into no further details on what AWS does if you don’t.
Can't ZFS do these kinds of things with snapshot?
Yes, but make sure any machine vulnerable to ransomware infection doesn’t have the ZFS server root account credential, directly or by reusing it as a share password, because even though snapshots are immutable root can still delete them.