Could they have avoided the ransom by having daily (or hourly) backups to non-rewritable (write once) media? So the malware won’t encrypt the backups, obviously.
I think that the last day’s work (or last hour’s work) will be lost or will require a lot of manual fixing regardless of whether they pay the ransom. If they pay, they’ll still have to fix partial database transactions, corrupt files, etc., for the attack date. If they don’t pay, they can recover from earlier good backups and reconstruct that one day’s worth. My reasoning is that the attack date’s data is going to be corrupt and untrustworthy in either case, and it’ll be equal work either way. (Or at least it’ll be less than $600,000 of work to fix that one day.)
I imagine that they either weren’t doing backups at all, or their backups were directly accessible and writable by the malware.
I work for the city of Skanderborg in Denmark and we do a few things to avoid it. One is to monitor our entire storage for malicious code and automatically isolate suspicious activity. Another is to do frequent backups of everything on our network shares and one drive for business which is where most employee data that doesn’t belong in a specific system lives. Our servers, database clusters and vital systems are all isolated from the employee network and also frequently backed up. When we get hit, it’s usually employees reading private email and we’re typically able to isolate the ransomware before it spreads from that specific employees network share. Once we kill it, we restore files to the most recent backup and roll their machine.
It’s worked well so far, but we do have an IT crew that would make most places jealous and IT is an area that is notoriously undervalued in the public sector.
See this seems to be a really fun and challenging part of IT.. the part that I don't want to have to deal with is the helpdesk, guiding a 60 year old lady to login onto her system or configure her company iPhone.
Maybe there is a job where you do only the former, I just haven't found it yet.
But guiding that 60 year old lady through configuring her company phone is how you discover that your process for configuring company phones is broken. When the people who design a system never interact with the people who use the system, that is how you end up with shitty unusable IT systems.
If you are unwilling to get your hands dirty interacting with the people who use your network you'll be unaware of how your decisions affect their day to day lives and consequently they'll probably end up hating you, which will not do you any favors in the long run. It is important to remember that everything you do is ultimately in service to helping them do their jobs.
Our IT is two things, one is helpdesk the other is “tech”. Tech handles servers, networking, security and Azure and only really offer support for developers like me.
What do you think happens when a core core router breaks? The level one guy fixes it?
There are a few Palo Alto and Cisco guys near me. They definitely aren’t answering the help desk phone but they are level 3 support. Same with my development team.
Not to mention large orgs have NOCs dedicated to this. The last company I was at at a couple CCAs in the NOC. They were smart but an odd breed.
I imagine that they either weren’t doing backups at all, or their backups were directly accessible and writable by the malware.
Or they weren't able to detect exactly when the malware was deployed so they didn't know how far back the data corruption went, meaning they couldn't trust the backups even if they looked OK. One of the problems you face after any attack is trusting the system again. Verifying everything is correct is a very hard problem.
Knowing all the "flat" networks I've seen that all sounds par for the course. Segmentation is still exotic (and met with either disdain or blank stares by software vendors). The attack damaged all those systems because, very likely, they were all accessible to each other and under one locus of control (probably a single Active Directory domain).
More than likely the SCADA systems themselves were fine, but the PCs for managing those systems were AD joined and people couldn't login to actually run the management software.
Its very common in custom hardware setups to have a standing system that interfaces with the physical hardware and PLC and then the user friendly software for instructing that controller on what you want to be on a PC talking to it over serial or the network. Obviously if the computer is inaccessible you can't adjust settings, but the system continues to run fine.
A very good solution here is ZFS. Office file server runs ZFS and makes regular ZFS snapshots (hourly, or more often). Snapshots are immutable so even if the employee machines get infected, nothing can be lost (except the last N minutes since the last snapshot).
Obviously, still have to do backups (off of those snapshots).
I absolutely love ZFS, but I think it's only a piece of the puzzle necessary for mitigating this threat. In terms of data integrity, I sure hope no system running ZFS would have it's snapshots also corrupted, but if that did happen, then hopefully those ZFS snapshots would have been backed up to some offline media like tape or optical disk. Example: http://girlyngeek.blogspot.com/2014/02/zfs-on-read-only-devi...
Pull instead of push backups are one way of trying to mitigate this. You dont allow clients to start backups. Your backup server does instead, "pulling" backups at standard times.
As long as you harden it from crypto, there is no way for malware on client machines to force an overwrite of current backups.
If their backups are being placed somewhere like S3 they should have a stand alone server that simply copies daily backups from the S3 bucket the network has access to, over to another S3 bucket that only the backup managing app has access to.
I expect they are allowing their backup app to have read write access to manage cleaning up removing backups but just giving it write only access from the network would work too. And using a standalone server/app to manage the backups.
I'd still copy over a backup to a stand alone bucket not accessible by the network for something this critical.
Most of these places have their backups on directory joined machines with storage that is badly configured. Backup systems should never be joined to the domain specifically for this reason--but it makes configuration harder so most places skip it.
We use frequent ZFS snapshots (every 15 minutes for 24 hours) then getting coarser the future out you go. Came in handy when the Dean got hit with ransomware.
People have been saying exactly this for 20 years now, and it still hasn't happened: organizations still entrust Windows with all their sensitive data and critical services. The fact is, if these victims had been on Linux instead of Windows, they would have been mostly immune to attacks, certainly to these ransomware attacks that only target Windows.
Remember, when you're hiking and you're attacked by a bear, you don't need to outrun the bear. You only need to outrun the slowest hiker in your group.
Linux is way harder to target, as any software vendor will readily tell you, so it would need to be significantly juicier for worthwhile return on investment.
I think that the last day’s work (or last hour’s work) will be lost or will require a lot of manual fixing regardless of whether they pay the ransom. If they pay, they’ll still have to fix partial database transactions, corrupt files, etc., for the attack date. If they don’t pay, they can recover from earlier good backups and reconstruct that one day’s worth. My reasoning is that the attack date’s data is going to be corrupt and untrustworthy in either case, and it’ll be equal work either way. (Or at least it’ll be less than $600,000 of work to fix that one day.)
I imagine that they either weren’t doing backups at all, or their backups were directly accessible and writable by the malware.