QuickBooks Cloud Hosting Firm iNSYNQ Hit in Ransomware Attack(krebsonsecurity.com)
krebsonsecurity.com
QuickBooks Cloud Hosting Firm iNSYNQ Hit in Ransomware Attack
https://krebsonsecurity.com/2019/07/quickbooks-cloud-hosting-firm-insynq-hit-in-ransomware-attack/
138 comments
It is interesting how often I see companies that refuse to make back-ups of their cloud hosted data on the assumption that this is now someone else's problem. I also have a - recent - case of a very large manufacturer of storage solutions that managed to fuck up a restore of a raid array to the point that a whole pile of companies lost their data.
Backups are so simple, and yet the only times people seem to realize their true value is when they don't have any.
Backups are so simple, and yet the only times people seem to realize their true value is when they don't have any.
> Backups are so simple
It gets more interesting when you're trying to backup a distributed database. When your durability strategy is to spread your risk across a lot of nodes in a cluster, and you're federating your cluster across multiple sites, traditional backups may not be a practical option. However, as I write this, I'm realizing that there's nothing simple about distributed anything, and saying "it's hard to..." may not be all that interesting.
It gets more interesting when you're trying to backup a distributed database. When your durability strategy is to spread your risk across a lot of nodes in a cluster, and you're federating your cluster across multiple sites, traditional backups may not be a practical option. However, as I write this, I'm realizing that there's nothing simple about distributed anything, and saying "it's hard to..." may not be all that interesting.
Pick a data sequence number, and back up each region sperately at the same sequence number.
Keep your backups on-site but off-line - you already have many locations. You should practice a location (and it's backups) going offline at the same time as a code bug screws with data in the active system.
Keep your backups on-site but off-line - you already have many locations. You should practice a location (and it's backups) going offline at the same time as a code bug screws with data in the active system.
Cassandra handles this pretty nicely. Files are immutable, so files are hard-linked to a snapshot directory. Copy out the snapshot directory, delete it, and things are eventually consistent.
We restore our MySQL and Cassandra data every day in our production DC from backups in the DR DC. It's the only way to ensure things are working. It's about 100TB of data, but things zip along nicely now that 100gig networks are (more) common.
We restore our MySQL and Cassandra data every day in our production DC from backups in the DR DC. It's the only way to ensure things are working. It's about 100TB of data, but things zip along nicely now that 100gig networks are (more) common.
Neat!
> Backups are so simple
Disagree. A high-confidence backup plan that includes continuous restoration is of middling difficulty to design. Many get backups wrong and only find out when their archives turn out to be incomplete.
Disagree. A high-confidence backup plan that includes continuous restoration is of middling difficulty to design. Many get backups wrong and only find out when their archives turn out to be incomplete.
[deleted]
It's also a major flaw of the cloud and SaaS providers. They don't encourage customers to maintain local (or alternate-cloud) backups, because they refuse to admit the appearance of weakness.
If a Cloud/SaaS provider is going to claim to be your hassle-free one-stop shop, shouldn't they provide, as a matter of course, a feature to dump your data daily/weekly/monthly to a designated off-cloud site of your choosing? But they don't, because they want to pretend that they are impenetrable and 100% available.
If a Cloud/SaaS provider is going to claim to be your hassle-free one-stop shop, shouldn't they provide, as a matter of course, a feature to dump your data daily/weekly/monthly to a designated off-cloud site of your choosing? But they don't, because they want to pretend that they are impenetrable and 100% available.
Would be nice if someone could build a site like https://twofactorauth.org/ for bringing more attention to this alarming and growing issue.
The goal to hopefully influence companies into providing such a fundamental ability.
The goal to hopefully influence companies into providing such a fundamental ability.
A data dump isnt much use if you don't have the software to use said dump. Most SaaS providers don't run open source software...
"But we have a contract saying that it can't happen" or "but we have a contract saying it's their problem"...
(Which works reasonably well as long as the damages are quantifiable and the other company is actually able to pay them....)
(Which works reasonably well as long as the damages are quantifiable and the other company is actually able to pay them....)
Backups are NOT simple, especially when you have things like GDPR in play. I'd imagine companies can't afford to commission backup that is compliant with the regulations. If you spend on a backup system, your company goes down in debt, you don't - there is a chance you go down anyway.
If you can not provide a backup system for a fraction of the costs of hosting the production data - including GDPR compliance if that is what is required - then your company does not deserve to be in business to begin with.
I am not arguing the importance of backups, but tbh, simple is not a fair statement.
At the very least, if you have backups, you might have the ability to rebuild/redeploy, but for any non-trivial system, without a comprehensive DR plan that is drilled regularly, even having backups won't save the business if the tooling to get those backups into a production ready state.
At the very least, if you have backups, you might have the ability to rebuild/redeploy, but for any non-trivial system, without a comprehensive DR plan that is drilled regularly, even having backups won't save the business if the tooling to get those backups into a production ready state.
QuickBooks Online does not have a native backup or restore, at all, and it's pretty ridiculous. QBO told me they also can't do on-request restorations with their backups.
There's a third-party or two that can use the API to do a backup/restore, but also, still, not all of the QBO data has an API. Eg, IIRC, recurring transaction tasks.
Just a matter of time for ransomware to replace data via APIs.
There's a third-party or two that can use the API to do a backup/restore, but also, still, not all of the QBO data has an API. Eg, IIRC, recurring transaction tasks.
Just a matter of time for ransomware to replace data via APIs.
> Just a matter of time for ransomware to replace data via APIs.
I asked about this in a cybersecurity meeting at an investment bank a few years ago - how much security is there around Bloomberg's data feed? All of the algorithms that make million & billion dollar decisions rely on that data so I asked if someone stepped in the middle and fed the algorithms bad data or even slightly delayed data would we even know it had happened? They looked at me like I was a crazy person. I'm actually kind of surprised it hasn't happened yet. Or maybe no one has noticed yet.
I asked about this in a cybersecurity meeting at an investment bank a few years ago - how much security is there around Bloomberg's data feed? All of the algorithms that make million & billion dollar decisions rely on that data so I asked if someone stepped in the middle and fed the algorithms bad data or even slightly delayed data would we even know it had happened? They looked at me like I was a crazy person. I'm actually kind of surprised it hasn't happened yet. Or maybe no one has noticed yet.
That's a fascinating scenario. Even more malicious embedding shellcode in a series of tickers and prices kicked off with a few invalid tickers/prices.
I wonder how many other feeds/sources like that in which so many businesses rely on in at least some level of autonomous fashion.
Reminds of the story of Google wiping out all of their CDN servers by Diskerase because of a single detected value, IIRC.
I wonder how many other feeds/sources like that in which so many businesses rely on in at least some level of autonomous fashion.
Reminds of the story of Google wiping out all of their CDN servers by Diskerase because of a single detected value, IIRC.
As the resident tech guy in a small shop with Quickbooks, yes, you can do local backups. We're not using iNSYNQ for cloud hosting, though.
This is a question I've thought alot about, so many some Sys Admins can give a good idea about how to approach it:
How do you create a backup server that is reachable by production servers (so that they can back up to it) without then being vulnerable to the same kind of ransomware attacks that infect the production servers? You can't exactly make them read-only, or else they can't accept the "legitimate" writes that might occur during the normal backup process.
How do you create a backup server that is reachable by production servers (so that they can back up to it) without then being vulnerable to the same kind of ransomware attacks that infect the production servers? You can't exactly make them read-only, or else they can't accept the "legitimate" writes that might occur during the normal backup process.
Unless you have persistent mount to network FSs like NFS or SMB how do you think the ransomware would spread? You sure don't need network mounts for backups.
Cronjob to an (S)FTP server and an upload script trigger to chown/chmod all incoming files making the whole thing WORM (Write Once Read Many).
Once its submitted the same user account can't alter it. Even if the malware is clever and scans for .netrc and .id_rsa and manages to create its own connection to the backup server it doesn't have access to anything anyway.
Cronjob to an (S)FTP server and an upload script trigger to chown/chmod all incoming files making the whole thing WORM (Write Once Read Many).
Once its submitted the same user account can't alter it. Even if the malware is clever and scans for .netrc and .id_rsa and manages to create its own connection to the backup server it doesn't have access to anything anyway.
I'm not a sysadmin, and I'm pretty sure that <big company> does it different, but here is how I do it at home:
- I have a UnRaid machine, and a backup machine. The backup machine is a small itx board, and has a single HDD attached.
- A NodeRED instance has a so-called "Flow" on the UnRaid machine that is waking up the backup machine every 7 days.
- Thanks to anachron, with a 10 minute delay, rsnapshot connects to the UnRaid machine, pulls the data, and then issues a shutdown to the backup machine.
This setup let me sleep pretty well.
- I have a UnRaid machine, and a backup machine. The backup machine is a small itx board, and has a single HDD attached.
- A NodeRED instance has a so-called "Flow" on the UnRaid machine that is waking up the backup machine every 7 days.
- Thanks to anachron, with a 10 minute delay, rsnapshot connects to the UnRaid machine, pulls the data, and then issues a shutdown to the backup machine.
This setup let me sleep pretty well.
Windows strategy: one way dfs replication to onsite backup server hooked up to an offsite cloud backup provider.
Make S3 bucket on a different account, grant it write privileges from your main one. Enable versioning and setup lifecycle to purge old versions in 90 cays. That’s it - no matter what your main account does, you will bave 3 months to undo it
Most attacks are done via local file access. Ship things over an API.
EC2 -> S3 bucket with only write access and versioning enabled. EC2 -> EFS and it's a rotating set of 7 with 7 different security groups that rotate.
EC2 -> S3 bucket with only write access and versioning enabled. EC2 -> EFS and it's a rotating set of 7 with 7 different security groups that rotate.
you can use s3/gcloud files with "create"/"read" only rights.
Make the backup servers pull?
That is what I do. The backup server can access the production server, but production can't access the backup. But my setup requires the backup to go to disk on the production server and not stream out directly, so that aspect is annoying.
Append only?
Right, that's what I use, https://www.tarsnap.com/ has the option of allowing an access key to only be able to create a new backup but not read or alter or delete any existing backups. I blogged about my setup here: https://www.databasesandlife.com/write-only-backups/
[deleted]
[deleted]
Two separate S3 buckets with different credentials. The server pushes its own backups to the "quarantine" bucket and later a process moves the backup to the long term storage bucket.
I’m surprised this answer is so far down. Production machines write to an S3 bucket with a lambda trigger. As soon as the upload is finished, the lambda either lifts the file to another location or changes the acl so the production machines can no longer access it.
Two separate S3 buckets with different credentials
This gives me the heebie-jeebies.
It seems better to me to have one on Amazon, and one on another cloud provider entirely. Plus regular sneakernet archives, if your data is of a size that would permit that sort of thing.
I'm fortunate that this kind of thing is handled by other people where I work. It's my understanding that we have production data on one service on the east coast, a mirror with a different company in the south, and weekly backup archives stored offline on the west coast.
This gives me the heebie-jeebies.
It seems better to me to have one on Amazon, and one on another cloud provider entirely. Plus regular sneakernet archives, if your data is of a size that would permit that sort of thing.
I'm fortunate that this kind of thing is handled by other people where I work. It's my understanding that we have production data on one service on the east coast, a mirror with a different company in the south, and weekly backup archives stored offline on the west coast.
One thing I've considered doing but haven't done, is building an "archive" box just to sit in the office and mirror cloud data. It wouldn't be all that complicated just mirror some cloud storage onto local encrypted storage -- pretty easy until somewhere over 10TB. There are plenty of ways that is could fail but they would be much more uncorrelated than a backup on the same cloud provider.
> pretty easy until somewhere over 10TB.
10TB is not even the largest single HDD money can buy... and I wouldn't use a single drive anyways. What I'd do... There's the new Silverstone CS381 chsasis with is <30L (so it easily fits anywhere) and houses eight 3.5" hot swap disks plus 4 2.5" for boot and a micro ATX PC... Looking at https://diskprices.com/ I'd go with the Seagate 8TB drives at 120 bucks by shucking them from external and then you need a cheap uATX board and an LSI-9207-8i card.
10TB is not even the largest single HDD money can buy... and I wouldn't use a single drive anyways. What I'd do... There's the new Silverstone CS381 chsasis with is <30L (so it easily fits anywhere) and houses eight 3.5" hot swap disks plus 4 2.5" for boot and a micro ATX PC... Looking at https://diskprices.com/ I'd go with the Seagate 8TB drives at 120 bucks by shucking them from external and then you need a cheap uATX board and an LSI-9207-8i card.
Yeah I was vague on the size because it depends how you build it ... If I were to do it I'd most likely want a 2x replication factor locally, a rebuild time under 12 hours for a drive loss, and set a threshold around 80% storage use where I'd need to expand. Staying within those parameters I would expect maintenance to be easier and unrushed. I think that would send me towards smaller drives, but haven't properly analyzed it. I've been meaning to figure out how to set all that up with a ZFS system. Ironically it would be so much easier to build and test on cloud servers so I would probably do that first.
That case does look very nice for a storage box though.
That case does look very nice for a storage box though.
Thank you, I've been looking for a good home NAS solution, this looks like it could be it
I use a single bucket with versioning enabled. Production server has access to write as it pleases, but isn't allowed to touch past versions of a file.
I think having the bucket in a different account with different root/MFA might make sense here too. Your production AWS account is more likely to get owned than your account dedicated to backups.
Yes! The bucket is located in a different account with separate billing and very limited on who has root/admin access.
S3 now allows permanent object locking, too, which makes this even harder to screw up.
The easiest solution to this is out-of-band backups happening at the storage layer. The details depend heavily upon your platform and storage hardware, but most SAN and virtualization solutions today allow one to backup hosted systems without being accessible by those systems (and no agents installed into the guest OS).
I use FreeNAS (FreeBSD NAS with ZFS). The ZFS snapshotting feature effectively gives me protections from this type of issue.
My home computers push backups regularly throughout the day, and every day I create snapshots of each volume (how long to keep the snapshots is another question). This snapshot can only be accessed or managed on the NAS itself.
This effectively creates an append-only backup NAS thanks to the periodic snapshots.
My home computers push backups regularly throughout the day, and every day I create snapshots of each volume (how long to keep the snapshots is another question). This snapshot can only be accessed or managed on the NAS itself.
This effectively creates an append-only backup NAS thanks to the periodic snapshots.
Do those snapshots somehow benefit from the incremental nature of daily backups?
Yes, ZFS snapshots only contain data that changed. ZFS is very efficient, elegant, and reliable.
To be more precise, ZFS snapshots are mostly just a timestamp reference, similar to how a git commit is just a reference to a git object. Writes don't change the value of the files in the snapshot because they're stored in new places rather than overwriting the existing files on disk.
https://blogs.oracle.com/ahrens/is-it-magic
https://blogs.oracle.com/ahrens/is-it-magic
Even with ZFS deduplication off?
Yes.
These are all good ideas. It seems to me part of the problem with these ransom attacks is the intruders aren't pouncing the minute they get inside the victim's network. They take their time, figure out where everything is (including backups) and then only force-spread the ransom malware after they have staged everything just-so.
yes - new writes consume space.
https://docs.oracle.com/cd/E23824_01/html/821-1448/gbciq.htm...
https://docs.oracle.com/cd/E23824_01/html/821-1448/gbciq.htm...
ZFS Snapshots are really handy - it takes root access to remove them and you can't delete them with the normal UNIX delete. IF you're sharing with Samba or NFS, there's no way for any remote user to be able to delete.
You just go back to the last good version.
You just go back to the last good version.
Drawbridge, you configure a firewall to allow access only at certain times. Also you can configure a NAS so that deleted items are preserved without a admin removing them.
A generally safe backup process looks like this:
Production has no access to backup.
Backup has read only access to production.
Backup writes are append and not overwrites.
Deletes/archival are governed by a retention process.
Production has no access to backup.
Backup has read only access to production.
Backup writes are append and not overwrites.
Deletes/archival are governed by a retention process.
One absolutely diabolical mechanism that was used (at least 5 years ago when this scourge of ransomware started to rear its ugly head) goes something like this:
1. Gain access to change the code on the front-end web servers (usually PHP)
2. Change the database access layer to transparently encrypt data being written to the database, and decrypt data being read from the database. The key would be loaded into memory by curl'ing an attacker-controlled website at startup.
3. Wait 30 days
4. Notify the company that they're compromised, turn off the attacker-controlled key service, and restart the web front end
Now step (3) ensures that most data in the database has been re-written, and if your backups are dumps of the production database, you now have a month of encrypted backups that you can't read... If you're lucky, you may have a month-old backup to restore from; if you're unlucky, you rotate every 30 days.
1. Gain access to change the code on the front-end web servers (usually PHP)
2. Change the database access layer to transparently encrypt data being written to the database, and decrypt data being read from the database. The key would be loaded into memory by curl'ing an attacker-controlled website at startup.
3. Wait 30 days
4. Notify the company that they're compromised, turn off the attacker-controlled key service, and restart the web front end
Now step (3) ensures that most data in the database has been re-written, and if your backups are dumps of the production database, you now have a month of encrypted backups that you can't read... If you're lucky, you may have a month-old backup to restore from; if you're unlucky, you rotate every 30 days.
If there's an engineer who can pull this off why are they screwing around with ransomware? Have them send me their resume, if they can deploy code to our frontend transparently without customers or devops having to spend time or even noticing it we'll happily pay them more than they're asking for in ransom.
Edit : thinking about it, this story doesn’t add up. Doesn’t this mean the client must have a copy of the decryption key, meaning any cached client would render the ransom demand worthless? It’s just also so much easier, if someone has that level of access, to make a copy of the database, encrypt it, then blow away the old one. Doing a silent deploy of client code with no one noticing seems way harder.
Edit : thinking about it, this story doesn’t add up. Doesn’t this mean the client must have a copy of the decryption key, meaning any cached client would render the ransom demand worthless? It’s just also so much easier, if someone has that level of access, to make a copy of the database, encrypt it, then blow away the old one. Doing a silent deploy of client code with no one noticing seems way harder.
The client only had the decryption key in memory (the curl part of their story).
Where it breaks down is assuming only the web server that is compromised has access to the database.
Where it breaks down is assuming only the web server that is compromised has access to the database.
Honestly, many of them are in nations where they are working for a state-sponsored cybercrime agency and have few other options, if any. North Korea and Iranian gov'ts are major purveyors of ransomware; I've known many clients who were breached by them (FBI guys said it was usually one of them each time). If not, it's probably Russia or China, especially if it's a higher-value target. If it's just a private citizen flinging exploits at ip ranges, it's possibly Brazil too.
Honestly, it used to be Eastern Europe, until we started realizing they've got some serious programming talent and contracted some work out. Now it's less bad, though still a bit sketchy. Of course, this doesn't help nation-state attacks.
Honestly, it used to be Eastern Europe, until we started realizing they've got some serious programming talent and contracted some work out. Now it's less bad, though still a bit sketchy. Of course, this doesn't help nation-state attacks.
> if they can deploy code to our frontend transparently without customers or devops having to spend time or even noticing it we'll happily pay them more than they're asking for in ransom.
Where should I send the resume?
Where should I send the resume?
This is one of those stories that goes around in certain communities. I don't know if it's true or not; it may well be. I've heard of it several times, but only ever second- or third-hand. I've also heart it's happened to more than one company. The thing is, you can't really verify this sort of stuff: if it's true, whoever developed it would throw its full weight behind it and hit as many targets as possible. It would probably be under tight investigation by some three-letter. Companies also probably wouldn't want to disclose this. I don't know if It's not unreasonable to hypothesize N. Korea or Iran put together a kit, though they'd probably relegate it to a certain stack (too much technical debt otherwise).
Of course, it's also possible it's not, and I wonder if that's more likely. Every industry has such stories; maybe this is the white whale of forensics.
Of course, it's also possible it's not, and I wonder if that's more likely. Every industry has such stories; maybe this is the white whale of forensics.
Back in the day when memes were in the form of urban legends and dispensing with them required various paperback books, these were called FOAF stories: always "a friend of a friend."
Has this actually happened?
Implementing zero downtime transparent encryption on any moderately complex codebase above the database layer sounds like a big engineering challenge. Even moreso if you're trying to do it without the sysadmin noticing. Even moreso if there is no persistence for the encryption key so every user request ends up having to fire a request to the attackers server - that'll kill performance. I doubt many attackers would attempt it.
Implementing zero downtime transparent encryption on any moderately complex codebase above the database layer sounds like a big engineering challenge. Even moreso if you're trying to do it without the sysadmin noticing. Even moreso if there is no persistence for the encryption key so every user request ends up having to fire a request to the attackers server - that'll kill performance. I doubt many attackers would attempt it.
If I’ve learned anything in decades of working in infosec, it’s to never underestimate the ingenuity of a determined attacker.
Read through any modern multi stage remote exploit. The amount of engineering required to pull these stunts off - reliably no less - is truly staggering, and yet attackers continue to innovate and develop new techniques and even more complex exploits.
And you overestimate the competency of the defenders: sysadmin? What small business has a dedicated sysadmin? It’s not like we are talking about running a global hosted chat service like slack here. (Oh wait...)
BTW Persistence for the key in memory is trivial: create a shared memory segment and store the key there, with a ttl.
Read through any modern multi stage remote exploit. The amount of engineering required to pull these stunts off - reliably no less - is truly staggering, and yet attackers continue to innovate and develop new techniques and even more complex exploits.
And you overestimate the competency of the defenders: sysadmin? What small business has a dedicated sysadmin? It’s not like we are talking about running a global hosted chat service like slack here. (Oh wait...)
BTW Persistence for the key in memory is trivial: create a shared memory segment and store the key there, with a ttl.
Wait, how does this work? Next time new version will be deployed, the deploy script will override all the files, and people will notice that all the data is coming back damaged.
Was it a site in maintenance mode, where no one was working on anymore?
Was it a site in maintenance mode, where no one was working on anymore?
Not only that, the company must not have a single front end dev open up dev tools for 30 days (which should happen at least once a month if anybody is debugging or doing any frontend dev whatsoever), and notice that the usual json payload from the server is encrypted and can't show up in devtools, plus there's this weird outgoing request to a domain they don't recognize? And absolutely no devops or logging notices an request to some weird domain (which must be handling tons of requests and have great uptime and scalability, since every single client has to connect to it to decrypt the database).
Oh, and there must not be a DBA at the company either, since absolutely nobody ran a SQL or analytics query in an entire month and noticed that some of the rows were full of unreadable garbage..
This story must be about someone's single server, single dev LAMP blog from ten years ago if it ever actually happened.
Oh, and there must not be a DBA at the company either, since absolutely nobody ran a SQL or analytics query in an entire month and noticed that some of the rows were full of unreadable garbage..
This story must be about someone's single server, single dev LAMP blog from ten years ago if it ever actually happened.
The encryption would happen on the server, not the client. Besides, what json? The server just returns rendered html as the lord intended :)
Yes, we are talking about the long tail of little LAMP servers handling small business needs with some custom code written by a contracting shop. What’s a devops? DBA? Once the site is developed and working, there’s little reason to pay to change it.
Yes, we are talking about the long tail of little LAMP servers handling small business needs with some custom code written by a contracting shop. What’s a devops? DBA? Once the site is developed and working, there’s little reason to pay to change it.
If they have full sql read write access on the server, it’s way easier to make a copy of the db, encrypt it, then blow away the original db in one go.
Wouldn’t catch the backups.
Exactly. It’s possible they are screwed.
A solid list.
It's nice to have a restore step in there too, so one has both a validation that the backup is usable, and gives a "playground" where one can have a safe day old environment for testing/training/whatnot against production data.
It's nice to have a restore step in there too, so one has both a validation that the backup is usable, and gives a "playground" where one can have a safe day old environment for testing/training/whatnot against production data.
I don't consider that nice-to-have.
That is essential. If you do not regularly test restoring, you do not know if you have backups. I normally overstate this just a little for effect, "If you don't test your backups you don't have backups", but that isn't precisely true.
And yes, it frequently makes sense to base test/fixture data and whatnot on backup data flows, but unless you don't deal with any PII, you're hopefully not using that data without a sanitization step.
That is essential. If you do not regularly test restoring, you do not know if you have backups. I normally overstate this just a little for effect, "If you don't test your backups you don't have backups", but that isn't precisely true.
And yes, it frequently makes sense to base test/fixture data and whatnot on backup data flows, but unless you don't deal with any PII, you're hopefully not using that data without a sanitization step.
Do you restore to prod? Or to Non-Prod and compare to prod?
Thank you.
To anyone viewing my answer as a guide: a backup by itself is not a continuity plan. There are other important areas such as setting recovery metrics (RPO/RTO) and testing restores against those metrics. The world is rife with stories of unreadable backups and/or restorations that took days instead of hours.
To anyone viewing my answer as a guide: a backup by itself is not a continuity plan. There are other important areas such as setting recovery metrics (RPO/RTO) and testing restores against those metrics. The world is rife with stories of unreadable backups and/or restorations that took days instead of hours.
Also, backups must include a physical process of moving a copy of backups to an airgapped secondary system (a human ejecting disks/CDs/tapes and carrying them to another storage container), so that it's impossible for an attacker to compromise the backups via the same software exploit that corrupts the primary data.
Monitor dedupe statistics. If they go pear shaped, Copy N is much different than Copy N+1.
By good and easy to do point- something as simple as size will help but a diff between backups would tell much - i had not thought of that - then again it comes for free in something like tarsnap
Backup is on a separate network.
The network is segregated to limit impact if they do get hacked.
To keep everyone familiar, Data Recovery processes are practiced regularly.
...
The network is segregated to limit impact if they do get hacked.
To keep everyone familiar, Data Recovery processes are practiced regularly.
...
Backup could (should?) be on tape, which once written is physically removed from the system and put in cold storage for some period of time (years?). That's how we did it at my first job, at least.
Of course IANAL so idk how this jives with various EU laws.
Of course IANAL so idk how this jives with various EU laws.
You can write backups to an NFS share, and create regular snapshots on the file system server that are not writable from the outside (you have to log into the file system server, which hopefully is extra secured / limited access, and has a different OS than what you use elsewhere). It's not perfect, but at least a single vulnerable client can't encrypt or delete your backups (in the snapshots) that way.
As usually with security, the principles of least privs and segregating as much as possible are important.
As usually with security, the principles of least privs and segregating as much as possible are important.
In the past I had the backup server ssh via public key into the production server and pull the data via file system snapshots. It’d be a matter of restoring the most recent snapshot prior to the attack.
You pull from the production servers rather than push to the backup servers. That way even if the production servers are compromised your backups are still safe.
Also how do you prove integrity of your backups if you don’t have certainty of initial perhaps temporarily dormant breach? I mean your backups to tape could still have bad data.
iNSYNQ is a third-party service platform operator who specializes in hosting instances of QuickBooks in the cloud.
This does not affect non-iNSYNQ QuickBooks instances, such as those operated by Intuit (the creator of QuickBooks).
This does not affect non-iNSYNQ QuickBooks instances, such as those operated by Intuit (the creator of QuickBooks).
I had no idea this was how QuickBooks even worked in the cloud. Could Intuit be in any way liable (like is this a sort of franchise type of thing)? Or is it closer to Word Press where different companies can install "QuickBooks Cloud" and then offer it?
Think “Microsoft Access in the cloud”. How would you do that without Microsoft’s cooperation? Virtual machines running Windows. Full service hosting, the works. This is classic ISP stuff, back when ISPs were service providers and not just Internet connections and other off the shelf no-touch products.
Looked for but didn't see this when I made my shockingly similar post.
I’m not familiar with the branch of synesthesia that detects similarity as electrical impulses, but you’re welcome to email the mods and ask them to delete my post and give you the karma. I don’t know if they’ll do so or not but I certainly have no investment in the integer (as my most recent post shows).
It's not immediately obvious, but iNSYNQ provides hosting for QuickBooks Desktop as a virtual desktop service (think VNC or RDP). This is distinct from Intuit's own QuickBooks Online cloud service, which I believe is unaffected by this breach.
I was concerned because one of my clients' customers rely heavily on QuickBooks Online and her app integrates heavily with it.
I was concerned because one of my clients' customers rely heavily on QuickBooks Online and her app integrates heavily with it.
I can confirm QBO is up (I am in Canada). Holy Batman, the chaos if that data would be gone... I did a backup, quickly. I need to ask my accountant whether they back this data up regularly, if not then I need to. I am a very small company but my invoices are international and while I have my invoices at hand, if I would need to reconstruct the exchange rates for an audit a few years back... shudders
ALWAYS make OFFLINE backup! USB USB USB!
Was this service really less expensive than just using actual QB on an RDP available cloud server from Azure or any number of other services?
Also, what kind of hacky backup system takes this much time to sort through to identify issues. They should have a clean image, and a clean way to backup/restore data for the application being hosted as a pull from production/active deployments.
In the end, this will or maybe even should kill the company in question. Beyond this, it is an opportunity for others. For that matter, really surprised Intuit doesn't have this as a cloud service at this point.
Also, what kind of hacky backup system takes this much time to sort through to identify issues. They should have a clean image, and a clean way to backup/restore data for the application being hosted as a pull from production/active deployments.
In the end, this will or maybe even should kill the company in question. Beyond this, it is an opportunity for others. For that matter, really surprised Intuit doesn't have this as a cloud service at this point.
> really surprised Intuit doesn't have this as a cloud service at this point.
They do have a cloud offering, QuickBooks Online:
https://quickbooks.intuit.com/online/
But it does not have all the same features as the Desktop version, giving rise to a number of third party offerings, like Right Networks' "QuickBooks Desktop Cloud": https://www.rightnetworks.com/cloud-solutions/accounting-sol...
They do have a cloud offering, QuickBooks Online:
https://quickbooks.intuit.com/online/
But it does not have all the same features as the Desktop version, giving rise to a number of third party offerings, like Right Networks' "QuickBooks Desktop Cloud": https://www.rightnetworks.com/cloud-solutions/accounting-sol...
It's not impacting all of Insynq's services. I work with an Insynq customer. Their Insynq services are still running, by some generous stroke of fate.
The only outage we noticed was in the middle of the day on the 16th. For about an hour, users weren't able to access the service. I called support, and was diverted to a recorded message saying that they were doing normal maintenance, would be finished shortly, and were aware of and sorry for the disruption.
It's been my outspoken opinion that this was an inevitable outcome for as long as I've been familiar with their product.
It's been my outspoken opinion that this was an inevitable outcome for as long as I've been familiar with their product.
Wonder if they obtained any people's financial data or social security numbers. Probably mostly self employed people and small businesses. Pretty scary how people use their SSN for everything. Seems so insecure to have a number you just openly pass around... Need to get a ID or license, credit card, bank account, your doctor, dentist asks for them, your phone company, cable company, of course when getting paid, and even police officers ask for them sometimes and write down in a notepad if your name happens to match someone else's name who has a warrant. Many other uses probably too I didn't think of off the top of my head.
I was randomly one day looking at dentist new patient forms and one even wanted to know your relationship status, not sure how that's relevant if a single or married guy gets a cleaning... I know home alone when the internet went out, so called the local cable company to see if an outage and the lady wanted the social security number on the account before continuing, which I didn't know. Just insane how many things use the same number, it's like single sign on for real life.
Same issue with bank account numbers. To pay someone with direct deposit, they can use the same number to withdraw from your account. I'm surprised banks haven't figured out a way to offer deposit only option... Just create a new account number but linked to another account, where deposits to account 4321 goes to account 1234 instead, but can't ever withdraw from 4321.
I got a feeling Facebook's account system is probably more secure than my local bank. Pretty sad when someone's hobby blockchain project has more technology in it than banks with billions of dollars of assets under management.
I was randomly one day looking at dentist new patient forms and one even wanted to know your relationship status, not sure how that's relevant if a single or married guy gets a cleaning... I know home alone when the internet went out, so called the local cable company to see if an outage and the lady wanted the social security number on the account before continuing, which I didn't know. Just insane how many things use the same number, it's like single sign on for real life.
Same issue with bank account numbers. To pay someone with direct deposit, they can use the same number to withdraw from your account. I'm surprised banks haven't figured out a way to offer deposit only option... Just create a new account number but linked to another account, where deposits to account 4321 goes to account 1234 instead, but can't ever withdraw from 4321.
I got a feeling Facebook's account system is probably more secure than my local bank. Pretty sad when someone's hobby blockchain project has more technology in it than banks with billions of dollars of assets under management.
> I was randomly one day looking at dentist new patient forms and one even wanted to know your relationship status, not sure how that's relevant if a single or married guy gets a cleaning...
This is for spousal rights - i.e. if your spouse is allowed to request access your data.
> I'm surprised banks haven't figured out a way to offer deposit only option
They have, some German banks assign an IBAN also for "Sparbücher" (saving plans). These cannot be withdrawn from.
For withdrawal security, under SEPA rules you have 8 weeks to (instantly!) reverse a transaction. If you misuse this, you can get your account closed and criminal proceedings filed so that is a relatively effective fraud prevention.
This is for spousal rights - i.e. if your spouse is allowed to request access your data.
> I'm surprised banks haven't figured out a way to offer deposit only option
They have, some German banks assign an IBAN also for "Sparbücher" (saving plans). These cannot be withdrawn from.
For withdrawal security, under SEPA rules you have 8 weeks to (instantly!) reverse a transaction. If you misuse this, you can get your account closed and criminal proceedings filed so that is a relatively effective fraud prevention.
Interesting, don't know much about the spousal rights thing as never been married but felt like they are asking too much info. I figured it's to try and collect if you don't pay since also employment info was a question.
I know I heard in Europe there's some law called PSD2 that banks would provide standard APIs too, but haven't been following that space since not in Europe. I know there's budgeting and other apps but they login to your bank account and scrape the data. I was using a app that categorize your spending for a little while but got sick of it making me relog my accounts over and over. I think one of my credit cards was thinking their servers were trying to hack my account. So a actual official API sounds like the move in the right direction.
I know I heard in Europe there's some law called PSD2 that banks would provide standard APIs too, but haven't been following that space since not in Europe. I know there's budgeting and other apps but they login to your bank account and scrape the data. I was using a app that categorize your spending for a little while but got sick of it making me relog my accounts over and over. I think one of my credit cards was thinking their servers were trying to hack my account. So a actual official API sounds like the move in the right direction.
We call PSD2 "openbanking" in the UK.
It basically means things like YANB (budgeting tool) or freeagent ( accountancy tool for small business) can get access to data you want them to, without you giving them your credentials.
Historically, most banks offered all or nothing access, and the aggregators would screen scrap using credentials that can steal all your money, and the links usually broke, or needed you to contioniously re-auth.
Its actually working. Modern banking apps will likely start to pull in all your financial services in the future.
It basically means things like YANB (budgeting tool) or freeagent ( accountancy tool for small business) can get access to data you want them to, without you giving them your credentials.
Historically, most banks offered all or nothing access, and the aggregators would screen scrap using credentials that can steal all your money, and the links usually broke, or needed you to contioniously re-auth.
Its actually working. Modern banking apps will likely start to pull in all your financial services in the future.
Is the Sparbücher number for a virtual account that is backed by a regular account?
No, it's an entirely separate account, usually you're free to shift around money between them using online banking.
Oh, so the "deposit only" aspect is for restricting less authenticated transactions. That makes sense.
The cloud is just someone else's computer.
The article seems light on details about how QuickBooks is involved. Does iNSYNQ host QB Cloud data for them, or does iNSYNQ white-label QB software?
What are the odds their vulnerability was due to IT staff choosing to not patch their Windows servers, as doomed Baltimore in May? https://krebsonsecurity.com/2019/06/report-no-eternal-blue-e...
The article does not say the attack was due to IT staff choosing to not patch their Windows servers, am I missing something?
You are correct, the Krebs article I cited does not say that. But it's my strong hunch.
There was much discussion of the poor state of Baltimore's IT infra (see https://www.baltimorebrew.com/2019/05/21/baltimores-out-of-d...), and at the same time Microsoft released a patch for an RDS flaw (CVE-2019-0708, https://msrc-blog.microsoft.com/2019/05/14/prevent-a-worm-by...) which says "Vulnerable in-support systems include Windows 7, Windows Server 2008 R2, and Windows Server 2008. Downloads for in-support versions of Windows can be found in the Microsoft Security Update Guide. Customers who use an in-support version of Windows and have automatic updates enabled are automatically protected. "
So unpatched servers are where I'm laying my money.
There was much discussion of the poor state of Baltimore's IT infra (see https://www.baltimorebrew.com/2019/05/21/baltimores-out-of-d...), and at the same time Microsoft released a patch for an RDS flaw (CVE-2019-0708, https://msrc-blog.microsoft.com/2019/05/14/prevent-a-worm-by...) which says "Vulnerable in-support systems include Windows 7, Windows Server 2008 R2, and Windows Server 2008. Downloads for in-support versions of Windows can be found in the Microsoft Security Update Guide. Customers who use an in-support version of Windows and have automatic updates enabled are automatically protected. "
So unpatched servers are where I'm laying my money.
Is there a way of freezing a compromised machine, such that the contents of RAM are kept around for forensics, but the compromised system can't do any more damage to itself?
SysRq? Intel ME?
In the common case of a VM it should be easy. Bare metal, less so.
Slowly but surely ransomware is becoming an absolute threat to all data. It seems more like not if but when it will hit.
It seems to me that it's time the OS providers start providing a very easy way to restore the state of data. We all know that backups are the answer but as long as people, have to think about it, there will always be some that don't do them. And now that you can get a 1TB HD for less that $100 then it's a no-brainer.
Virus protection is now automatic with Windows when will backups become automatic on all OSs?
It seems to me that it's time the OS providers start providing a very easy way to restore the state of data. We all know that backups are the answer but as long as people, have to think about it, there will always be some that don't do them. And now that you can get a 1TB HD for less that $100 then it's a no-brainer.
Virus protection is now automatic with Windows when will backups become automatic on all OSs?
>It seems to me that it's time the OS providers start providing a very easy way to restore the state of data.
Wat?
How are you imagining this would be implemented? Right now you can have a cronjob that runs rsync which allows exactly that but doesn’t protect you from ransomeware. Are you thinking if it were part of the file system in the kernel it would be better protected? I guess that’s partially true but in practice probably will do very little (also, that already exists on Linux in the form of things like ZFS.)
Once something like that infects the machine there really isn’t a whole lot the OS can do to prevent the damage, all it’s doing is handling user requests for operations on things like the file system and the malware is operating as a proxy for the user.
Wat?
How are you imagining this would be implemented? Right now you can have a cronjob that runs rsync which allows exactly that but doesn’t protect you from ransomeware. Are you thinking if it were part of the file system in the kernel it would be better protected? I guess that’s partially true but in practice probably will do very little (also, that already exists on Linux in the form of things like ZFS.)
Once something like that infects the machine there really isn’t a whole lot the OS can do to prevent the damage, all it’s doing is handling user requests for operations on things like the file system and the malware is operating as a proxy for the user.
Why would someone use a third party to host Quickbooks? Won’t Intuit host it?
This. Is it the same Quickbooks ?
No, this is a desktop version hosted offsite instead of the browser-based cloud version from Intuit. See this above post for examples: https://news.ycombinator.com/item?id=20481074
How is "QuickBooks Desktop Hosted in the Cloud" different from "Anything you darn feel like, hosted on a remote Desktop (like Citrix)" ?
The former is probably easier to sell to accountants who want QuickBooks Desktop plus the benefits of a remote managed service (accessible over the Internet, someone else takes care of backups - in theory! - etc)
If anyone wants a recommendation for an alternative, I can happily suggest Xero.
What a mess, though. Worst part of a business to be crippled is its core - financials. A part often overlooked by techies. If you can’t invoice, you can’t pay the wages.
What a mess, though. Worst part of a business to be crippled is its core - financials. A part often overlooked by techies. If you can’t invoice, you can’t pay the wages.
It's not Intuit that's down... Just a 3rd party hosting provider.
Yeah from reading the headline at first I was thinking iNSYNQ was the company Intuit was contracting with for their hosting and not a third party at first too. So I could see people getting that confusion.
What are they hosting? Backups of the desktop version of Quickbooks?
Uneducated guess: they host an RDS farm and give people RDCs\remoteApps to log on and use their local QB installs
AFAIK, there are legacy versions of a "networked" quickbooks that can be hosted by third parties. Not really sure though. Some are doing app streaming of desktop quickbooks
I wouldn’t call the networked versions of quickbooks legacy. (Quickbooks Desktop and it’s ancillary products) That’s like saying Excel is legacy because Office 365 will let you edit spreadsheets in the cloud. Sometimes local operations are just better, or cloud versions don’t fully match features.
Yep a more appropriate term would be "older".
I think these are multi-user versions of QB Desktop, normally restricted to a LAN, that are being hosted to allow access from the internet. The hosting entity also provides backup.
Xero is not a Quickbooks hosting provider, so not really equivalent.
Like suggesting DotNetNuke when WPEngine goes down.
Like suggesting DotNetNuke when WPEngine goes down.
For an accounting software this is disaster.
My Quickbook Self-Employed (hosted) seems unaffected (at least so far).
that is different from this. The affected is a 3rd party hoster, not Intuit.
I lost all interest in the topic when I read the hosting companies name, can we just talk about THAT? Like, really? Who picks that name for their company, certainly the band nsync has to be older than the company.
It's certainly very search engine friendly. You essentially get a curated list of all and ONLY items of potential interest to you.
My point was more "hey that seems like a MikeRoweSoft [1] potential legal quagmire, terrible for branding and SEO and a great way to be mocked and not taken seriously at all by potential clients".
[1] https://en.wikipedia.org/wiki/Microsoft_v._MikeRoweSoft
[1] https://en.wikipedia.org/wiki/Microsoft_v._MikeRoweSoft
There is discussion on twitter that the company said the backups were on the same network as the data. Hopefully there is an offsite backup available.
https://twitter.com/ConleyU/status/1151862278909825024
https://twitter.com/MRasconCPA/status/1151894366291734533
https://twitter.com/hockeygirlPDX/status/1151945932935585792
Ouch. This is the sort of stuff that can kill a company.
Does Quickbooks with the cloud option offer local backups?