I can't imagine why any critical system is connected to the internet at all. It never made sense to me. Wifi should not be present on any critical system board and ethernet plugged in only when needed for maintenance.
This should be the standard for any life sustaining or surgical systems, and any critical weapons systems.
I work for a large medical device company and my team works on securing medical devices. At least at my company as a general rule, the more expensive the equipment (and thus the more critical the equipment, think surgical robots) the less likely it will ever be connected to a network, and that is exactly because of what you said, you remove so many security issues when you keep devices in a disconnected state.
Most of what I do is creating the tools to let the field reps go into hospitals and update capital equipment in a disconnected state (IE, the reps must be physically tethered to the device to interact with it). The fact that any critical equipment would get an auto-update, especially mid-surgery is incredibly bad practice.
I work for the government supporting critical equipment - not in medical, in transportation sector - and the systems my team supports not only are not connected to the internet, they aren't even capable of being so connected. Unfortunately the department responsible for flogging us to do cybersecurity reporting (different org branch than my team) has all our systems miscategorized as IT data systems (when they don't even contain an operating system). So we waste untold numbers of engineer hours now reporting "0 devices affected" to lists of CvE's and answering data calls about SSH, Oracle or Cisco vulnerabilities, etc. etc. which we keep answering with "this system is air gapped and uses a microcontroller from 1980 that cannot run Windows or Linux" but the cybersecurity-flogging department refuses to properly categorize us. My colleague is convinced they're doing that because it inflates their numbers of IT systems.
Anyway: it is getting to the point that I cynically predict we may be required to add things to the system (such as embedding PCs), just so we can turn around and "secure" them to comply with the requirements that shouldn't be applied to these systems. Maybe this current outage event will be a wake up call to how misplaced the priorities are, but I doubt it.
Have you ever tried to airgap a gigantic wifi network across several buildings?
Has to be wifi because the carts the nurses use roll around. Has to be networked so you can have EMR's that keep track of what your patients have gotten and the Pharmacists, doctors, and nurses can interface with the Pyxis machines correctly. The nurse scans a patients barcode at the Pyxis, the drawer opens to give them the drugs, and then they go into the patient's room and scan the drug barcode and the patients barcode before administering the drug. This system is to prevent the wrong drug from being administered, and has dramatically dropped the rates of mis-administering drugs. The network has to be everywhere on campus (often times across many buildings). Then the doctor needs to see the results of the tests and imaging- who is running around delivering all of these scans to the right doctors?
You don't know what you are talking about if you think this is easy.
Air gap the system with the external world is different from air gap internally. The systems are only update via physical means. And possibly all data in and out is offline like, via certain double firewall arrangement (you do not let direct contact but dump in and out files). Not common but for industrial critical system saw a few big shops did this.
So how does a doctor issue a discharge order via e-prescription to the patients pharmacy for them to pick up when they leave? How do you update the badge readers on the drug vaults when an employee leaves and you need to deactivate their badge? How do you update the EMR's from the hospital stay so the GP practice they use can see them after discharge? How do you order more supplies and pharmacy goods when you run out? How do you contact the DEA to get approval for using certain scheduled meds? I'm afraid that external networks are absolutely a requirement for modern hospitals.
If the system has to be networked with the outside world, who is responsible for physically updating all of these machines, so they don't get ransomware'd? Who has to go out and visit each individual machine and update it each month so the MRI machine doesn't get bricked by some teen ransomware gang? Remember that was the main threat hospitals faced 3-4 years ago, which is why Crowdstrike ended up on everyone's computer: because the ransomware insurance people forced them too.
There is a reason that I am a software engineer and not an IT person. I prefer solving more tractable problems, and I think proving p!=np would be easier than effectively protecting a large IT network for people who are not computing professionals.
One of my favorite examples: in October 2013 casino/media magnate and right wing billionaire Sheldon Adelson gave a speech about how the US and Israel should use nuclear weapons to stop Iran nuclear program. In February 2014 a 150 line VB macro was installed on the Sands casino network that replicated and deleted all HDDs, causing 150 million dollars of damage. That was to a casino, which spends a lot of money on computer security, and even employs some guys named Vito with tire irons. And it wasn't nearly enough.
> Who has to go out and visit each individual machine and update it each month so the MRI machine doesn't get bricked by some teen ransomware gang?
The manufacturer does. As I mentioned in my OP I help build the software for our field reps to go into hospitals and clinics to update our devices in a disconnected state. Most of the critical equipment we manufacture has this as a requirement since it can't be connected to a network for security reasons.
As for discharge orders, etc, I can't speak to that, but that's also not what I would consider critical. I'm talking about things like surgical robots, which can not be connected to a network for obvious reasons, especially during a surgery.
External networks are required but it should be possible to air gap the critical stuff to read only. It’s just that it’s costly and hospitals are poor/cheap
My wife is a hospital pharmacist. (1) When she gets a new prescription in, she needs to see the patients charts on the electronic medical records, and then if she approves the medication a drawer in the Pyxis cabinet (2) will open up when a nurse scans the patients barcode, allowing them to remove the medication, and then the nurse will scan the patient's barcode and the medication barcode in the patients room to record that it was delivered at a certain time. Computers are everywhere in healthcare, because they need records and computers are great at record-keeping. All of those need networks to connect them, mostly on wifi (so the nurses scanners can read things).
In theory you could build an air-gapped network within a hospital, but then how do you transmit updates to the EMR's across different campuses of your hospital? How do you issue electronic prescriptions for patients to pick up at their home pharmacy? How do you handle off-site data backup?
Quite honestly, outside of defense applications I'm not aware of people building large air-gapped networks (and from experience, most defense networks aren't truly air-gapped any more, though I won't go into detail). Hospitals, power plants, dams, etc. all of them rely heavily on computers these days, and connect those over the regular internet.
1: My wife was the only pharmacist in her department last night whose computer was unaffected by Crowdstrike (for unknown reasons). She couldn't record her work in the normal ways, because the servers were Crowdstrike'd as well. So she spun up a document of her decisions and approvals, for later entry into the systems. It was over 70 pages long when she went off shift this morning. She's asleep right now.
First - drop "air-gapped" term and replace it with "internet-gapped". TA^h^h^a^a! And it already have a name: "The LAN"... Now teach managers about importance of local net vs open/public/world net. Tell them cloud costs more becouse someone is making a fortune or three on it !
TIP: many buildings can be part of one LAN! It is called VPN and Russia and China do not like it becouse it is good for peoples!
TIP: data can be easily exchanged when needed! Including LAN.
--
My wife is a hospital pharmacist. (1) When she gets a new prescription in, she needs to see the patients charts on the electronic medical records, and then if she approves the medication a drawer in the Pyxis cabinet (2) will open up when a nurse scans the patients barcode, allowing them to remove the medication, and then the nurse will scan the patient's barcode and the medication barcode in the patients room to record that it was delivered at a certain time. Computers are everywhere in healthcare, because they need records and computers are great at record-keeping. All of those need networks to connect them, mostly on wifi (so the nurses scanners can read things).
--
It was description of very local workflow...
It was description of data flow - no any reason it should be monopolized by unsecure by design os vendor that need to be mandatory secured by essentialy kernel rootkit aka os hacking. Which contradicts using that os in the first place!
And looks like Crowdstrike is just if you ask for price then you can't have it version of SELinux :>>> RH++ for two decades of making presentations of SELinux necessity.
But over all allowing automatic updates from 3rd party not having clue about medicine to hospital system, etc. is managers criminal negligence. Simple as that. Curent state of the art ? More negligence! Add (business) academia & co to chronic offenders. Call them what they truly are - sociopaths via craft training facilities.
>In theory you could build an air-gapped network within a hospital, but then how
>do you transmit updates to the EMR's across different campuses of your hospital?
How do you transmit to other campuses of other hospitals ? EASY! Transfer mandatory data.
Pleas notice I used words like "mandatory" and "data". I DID NOT SAY "use mandatory http stack to transfer data"! NO. NO, I'm far, faaar from even sugesting THAT ! :>
>How do you issue electronic prescriptions for patients to pick up at their home pharmacy?
Hard sold on that "air-gapped and in cage" meme, eh? Send them required data via secure and private method! Communications channels already "hacked" - monopolized - by FB? Obviously that should do not happend in first place. So resolve it as part of un-win-dosing critical civilian infra.
>How do you handle off-site data backup?
That one I do not get. You saying that cloud access is a only possibility to have backups??? And Internet is a must to do it?? Is medical staff brain dead? Ah, no... It's just managers... Again.
>Quite honestly, outside of defense applications I'm not aware of people building large air-gapped networks
And dhcp and "super glue" and tons of other things was invented by military, for a reason, but that things proliferated to civilians anyway. For good reasons. Air-gapping should be much more common when wifi signal allows tracking how you move in your own home. Not to mention GSM+ based "technologies"...
There is old saying: Computers maximize doing. And when somewhere is chaos then computers simply do their work.
I think the criticial systems here are often the ones that need to be connected to some network. Somebody up there mentioned how the MRI worked fine, but they still needed to get the results to the people who needed it. So the problem there was more doctor <-> doctor.
Yeah, our imaging devices were working fine, but with Epic down, you lose most of your communication between departments and your sole way of sharing radiology images and interpretations.
> Roslin: ...it tells people things like where the restroom is, and--
> Adama: It's an integrated computer network, and I will not have it aboard this ship.
> Roslin: I heard you're one of those people. You're actually afraid of computers.
> Adama: No, there are many computers on this ship. But they're not networked.
> Roslin: A computerized network would simply make it faster and easier for the teachers to be able to teach--
> Adama: Let me explain something to you. Many good men and women lost their lives aboard this ship because someone wanted a faster computer to make life easier. I'm sorry that I'm inconveniencing you or the teachers, but I will not allow a networked computerized system to be placed on this ship while I'm in command. Is that clear?
... at which point you will lose battles to enemies who have successfully networked their command and control operations. (For extra laughs, just wait until this is also true of AI.)
Ultimately there are just too darned many advantages to connecting, automating, and eventually 'autonomizing' everything in sight. It sucks when things don't go right, or when a single point of failure causes a black-swan event like this one, but in an environment where you're competing against either time or external adversaries, the alternatives are all worse.
Or the opposite: the enemy (or a third-party enemy who wasn't previously a combatant in the battle) hijacks your entire naval USV/UUV fleet & air force drone fleet using an advanced cyberattack, and suddenly your enemy's military force has almost doubled while yours is down to almost zero, and these hijacked machines are within your own lines.
Yes, the efficiency gains of remote automated administration and deployment make up for most outages that are caused by it.
A better thing to do is do phased deployment, so you can see if an update will cause issues in your environment before pushing it to all systems. As this incident shows, you can’t trust a software vendor to have done that themselves.
This wasn't a binary patch though, it was a configuration change that was fed to every device. Which raises a LOT of questions about how this could have happened and why it wasn't caught sooner.
Writing from the SRE side of the discipline, it's commonly a configuration change (or a "flag flip") that ultimately winds up causing an outage. All too seldom are configuration data considered part of the same deployable surface area (and, as a corollary, part of the same blast radius) as program text.
I've mostly resigned myself today to deploying the configuration change and watching for anomalies in my monitoring for a number of hours or days afterward, but I acknowledge that I also have both a process supervisor that will happily let me crash loop my programs and deployment infrastructure that will nonetheless allow me to roll things back. Without either of those, I'm honestly at a loss as to how I'd safely operate this product.
The most insidious part of this is when there are entire swaths of infrastructure in place that circumvent the usual code review process in order to execute those configuration changes. Boolean flags like your `config('foo')` here are most common, but I've also seen nested dictionaries shoved through this way.
When I was at FB there were a load of SEVs caused by config changes, such that the repo itself would print out a huge warning about updating configs and show you how to do a canary to avoid this problem.
As in, there was no way to have configured the sensors to prevent this? They were just going to get this if they were connected to the internet? If I was an admin that would make me very angry.
This is the way it's done in the nuclear industry across the US for power and enrichment facilities. Operational/secure section of the plant is airgapped with hardware data diodes to let info out to engineers. Updates and data are sneaker netted in.
This should be the standard for any life sustaining or surgical systems, and any critical weapons systems.