Has there been any discussion about the technical details of the attack? I am having a hard time imagining how a compromise of a workstation could result in the entire company -- their own apps, their call center -- going down for days. I can see how malware could break production severely ("kubectl delete deployments" from a trusted workstation). I can see how malware can wipe out your desktop. I can see how malware could f your cloud infrastructure account. But I'm not drawing the line to "we can't build a new release and deploy it on another provider" or "we can't buy an emergency Dialpad account to start taking calls from customers".
My guess is this: two separate attacks occurred. The first attack involved compromising production, and installed a scheduled job that, at a certain time, would delete all database backups and code repositories, deschedule all workloads, delete all DNS records, etc. The next attack involves the fact that all source code is on managed workstations, so they compromised the IT management system to push malware to every machine globally at the exact same time that would destroy all git repositories (etc.) on the workstations. The result was that when the scheduled time occurred, production would crash and there would be no backups. (They must have wiped all the tapes at their offsite backup facility, too. I guess anything can be done for a price!)
To me, this sounds too complicated to even be feasible. I am still impressed when I edit some manifest with a new version number that 90% of the time that code eventually starts running. Being able to orchestrate a multiday outage just seems amazing to me, and that you'd make a lot more money being a cloud provider than a cybercriminal.
The other thought I had was that maybe they just kept thinking "we're so close to getting it back" for three days, rather than saying "everything is lost, revert to backups".
> I am having a hard time imagining how a compromise of a workstation could result in the entire company -- their own apps, their call center -- going down for days
Can't guess at specifics, but if it's a Windows network, I would be utterly unsurprised if all users had excess permissions to shared drives
Many Windows networks just have a giant X: everyone can write to, and it's been like that forever, and it's so deeply baked into everyone's workflow that it never gets fixed
I would presume that the attacker was able to obtain Domain Admin / Enterprise Admin rights before they deployed the payload, then they just steamrolled over everything.
The one of these that I got called-in to clean up after literally had a batch file on Domain Controllers w/ a text file of computer names for a FOR loop launching the malware on computer-after-computer with "psexec". It was decidedly non-sophisticated. The attacker compromised a Domain Admin account and then they were set.
> Many Windows networks just have a giant X: everyone can write to, and it's been like that forever, and it's so deeply baked into everyone's workflow that it never gets fixed
I've seen this too, and you need to be vigilant with what accesses these kinds of resources (basically anybody with write access to these shares also has execute access to any accounts executing from that share).
Is it common practice to have the servers running your production (not in the manufacturing sense) cloud services join the AD domain that has your office staff in it? Why? That doesn't even make any sense from a convenience PoV.
It just seems like an unfathomable level of incompetence required to go from compromising some random Windows workstation all across the hardware that runs your app services. And lest we forget: a ransomware attack is always also a massive data loss attack. Garmin better get to work complying with the law and notifying impacted customers (all of them?).
If there's one thing I've learned in the computer industry, it's that there is no such thing as an unfathomable level of incompetence. All levels of incompetence are not only fathomable, but repeatedly demonstrated. It's amazing that anything works at all.
Taking all our services down this hard would require enormous efforts in coordination, potentially months of preparation to make sure it would execute satisfactory.
I think occam must be at work here. There must be some simpler reason why this attack is such a disaster.
The impression I got was that the call centre/apps were taken down as preventative measure by their own IT dept. It was probably best for a PR standpoint to keep the call centre silent rather than having a defacto-inoperable call centre inundated with calls about the broken service.
My guess is this: two separate attacks occurred. The first attack involved compromising production, and installed a scheduled job that, at a certain time, would delete all database backups and code repositories, deschedule all workloads, delete all DNS records, etc. The next attack involves the fact that all source code is on managed workstations, so they compromised the IT management system to push malware to every machine globally at the exact same time that would destroy all git repositories (etc.) on the workstations. The result was that when the scheduled time occurred, production would crash and there would be no backups. (They must have wiped all the tapes at their offsite backup facility, too. I guess anything can be done for a price!)
To me, this sounds too complicated to even be feasible. I am still impressed when I edit some manifest with a new version number that 90% of the time that code eventually starts running. Being able to orchestrate a multiday outage just seems amazing to me, and that you'd make a lot more money being a cloud provider than a cybercriminal.
The other thought I had was that maybe they just kept thinking "we're so close to getting it back" for three days, rather than saying "everything is lost, revert to backups".