Let’s Talk About Why all This OT Stuff is Connected
62 points by polywolf
62 points by polywolf
I disagree with the conclusion : "These connections were built one sensible business decision at a time, and they now help keep the lights on, the water running, and the food cold."
In a newly built house, I had to fight every single vendor, every single subcontractor that wanted to connect each appliance (up to the toilet lid). I counted that, without fighting, I would have 18 apps installed on my phone only to "handle my house".
Each time, I asked the question: "Why should it be connected?". Each time I had vague answer like "It’s better, you can use the app". When pressed, they all ultimately fall down to "If you are not connected, you will not receive update".
So I said: "why should receive update ? If it is currently functional, I don’t want it to change." "but for security".
So, ultimately, most sellers (which are totally uneducated at anything IT related) believe that "you must connect everything to receive update to avoid problems you would not have without being connected".
This post try to rationalize a situation which is mainly built on total incompetence at every level. People connect stuff because they believe everything has to be connected. And every appliance offer "Wifi connection" because some marketing people thought that it was cool (now, they add AI inside your toilet lid). So it more or less completely stupid.
Think about fridges: it makes sense to get remote sensing. But it doesn’t make any sense to remote control a fridge. If a fridge is not at the right temperature, the first thing you want is a human walking in and checking the fridge. You never want a remote "disconnect the fridge" button that could be remotely activated, sometimes without ever realizing it (bad button pressed in the web interface)|
So I strongly disagree with the post: systems should, by default be offline. Getting online should be a conscious choice with a clear responsibility. Getting online should, by default, be a read-only state where you could get information but not act on system. Acting remotely on any physical system should never be considered lightly.
At the industrial contract level. it is not only incompetence, it is lack of product liability, too.
And cheaper+no-liability is a catastrophic mix.
My house has a domestic example of the kind of industrial remote access that this article is talking about: our electrical systems include a solar+battery system, a smart meter, and a car. They all talk to our electricity provider to coordinate when to charge the car or top up the house batteries, depending on the current price of electricity. They use our domestic wifi to talk to their online support services.
The point of the article is that the devices need to be online so that they can react to externally-provided data, or so that they can report their status to a central monitoring system. And in many cases there must also be remote control as well: consider for example flood control pumps which (in places like the Cambridgeshire fens) need a wide-area overview to operate safely.
Now, while there are excusable reasons for things to be designed this way, they are clearly not good reasons in many cases and they are often not designed well. Obviously there isn’t enough attention paid to security architecture or hierarchies of privilege or whether it makes sense for a device to act as a remote command server (dangerous, ubiquitous) instead of as a reactive client (safer, rarer). And the designers of these things are optimizing for convenience and cost and minimal staffing, at the expense of safety and resilience.
Over-optimization and brittle systems are a problem caused by caring more about the price of things rather than the value they provide or the cost of failure.
You are not running a fleet of refrigerators. B2C and B2B (or rather, industrial) needs and requirements are different.
You do not get the health inspector showing up if fridges are not keeping things cool enough.
(You also have ... less incentive to do power management tricks and cut corners in an automated fashion across all your stores)
How did people do this before wifi? They did it badly or by having a bunch of people run around and do things.
I do believe that the argument of "just don't allow write access" is very reasonable though. "Page" an employee to go deal with the thing by hand!
I would be curious what sort of remote control would even be reasonable in a completely automated way anyways... at least for things like greenhouses I get it. But generally there's someone in the supermarket
(The article does mention distributed systems as being more of a problem.... I have to imagine that infrastructure spread across a huge area is where you really start wanting all of this)
Managing my smart home setup makes me wish we had a modern Wi-Fi standard for broadcast-only connectivity, hardware enforced. I love getting notifications when the washer is done. I do not want my washing machine to ever take any action or even parse information in a packet. Ditto for almost all of my other appliances. One-way communication only. Something like ESP-NOW but standard and maybe with CA.
Ooh that's a neat idea. Thinking about the various embedded/industrial networking tech I've used I can't think of any that don't assume two-way communication, apart from maybe a TV remote. Oh, I think I've seen something like this for 2G cellular? I recall some remote telemetry device on an oil pumpjack or something that sent data via SMS but didn't have a SIM or even a real phone number, but I don't know anything about how it worked under the hood.
I reckon something like this is doable using routing/firewalling, as long as you can get a dedicated subnet for this device only so that no other IoT crap can talk to it.
The post does explain this.
I’ve seen many major OEMs and specialist device suppliers add contractual language requiring some level of remote access in order to keep receiving support, or support within a reasonable SLA. I talk to numerous asset owners around the world, and even Fortune 50 and government clients struggle to change these terms. Some OEMs will simply only provide timely support via remote connection.
The point is, if you're running an industrial system, you do need updates, you do need support from your vendor. I expect there is much less flexibility in widget vendors and much more safety liability than a person who is installing commodity appliances in their house.
I agree with you and I'll add that sometimes it is not incompetence that drives these decisions...It is business decisions, but not "sensible" business decisions that got us where we are. I firmly believe it is motivated by money/capitalism. The business selling these devices and services think to themselves: "We'll connect stuff so we can sell crap...then we will sell services to fix the version 1 of crap that was sold...", and on and on, etc. And, once a device and service is created, it becomes a race to the bottom for how fast and how low can quality and costs of manufacturing can get pushed. One needs to only review the motivations for who gains if/when this stuff fails.
Years ago, i was genuinely excited for a more "connected" world...and I'm sure some of this can be done more diligently, appropriately...But nowadays, my reflex is becoming more and more: why does this need to be connected? No thanks, keep this all offline for me. :-)
I feel like you came to the article with a framing based on your experience with your house and thus missed the forest for the trees. Things are pretty well spelled out: want everything air-gapped? Cool, now who pays for that? Because all of a sudden you are introducing a lot of manual labor and salaries back into processes that have been made as efficient as possible. As an example, if our small city in the US needed to come up with 1.5 or 2x our water budget next year, we’d be screwed.
Well, to be fair, @ploum has a point: it's a slippery slope that was slid down one small decision at a time (as the article also points out), which could've been entirely avoided with a firm policy against this kind of thing from the beginning.
For example, I think everyone can agree the toilet lid probably doesn't need to be connected, but perhaps without such a firm policy, one would've gone with, I dunno, the oven being connected so you can pre-heat it while commuting back home. There's always some sort of reason (however flimsy) these devices are connected, and if your policy is to go for maximum convenience, you'll end up with a home that is swarming with IoT devices.
Of course, in a business context you have to compete with others who do not have a clear policy (and might be cheaper as a result), and buy from a limited set of vendors who push this shit hard, so you might not have the luxury to have a clear policy.
That’s still using one person’s home to compare to an industrial set up and seems fatally flawed.
I had a similar sentiment. I was a couple of pages into the HN comments after the second hack trying to find the comment in which someone asked why the fridges had remote warming in the first place and … no one had asked it. When I did, some replied with “well, for monitoring” as though you need to warm a fridge to monitor it.
I could imagine a couple reasons a fridge would have a warming system, though I don't know for real:
I could imagine a couple reasons a fridge would have a warming system
My issue isn't that it has a warming system, my issue is that it is remote controlled. If you want to regulate the amount of accumulated ice on the piping and it's something that's safe to do remotely (i.e., a human reads the machine's ice sensors, triggers the warming, and monitors the temperature to keep it in a safe range) then it's something the unit's control software should do (and can do much more easily than writing remote control software). If you can't trust the machine's sensor readings, then you necessarily need someone on site anyway. In neither scenario does it make sense to me that you would need someone remotely operating this.
This feels like it's missing the point of this very specific problem: you don't run a defrost cycle without anyone there, that makes literally no sense: someone has to empty that fridge/freezer before it should start that cycle, and the person can press the "okay, do it" button, in person, perfectly fine.
Nothing about the broader developments in the IT/OT world actually explains why anyone (including the people who paid for these things) should want to, let alone be allowed to initiate a function that can only safely run after someone's prepped the fridge/freezer for that procedure.
It's a literal remote "cause a problem" button that should never have been there in the first place. Plenty of remote operations make plenty of sense, but this very specific one has no good explanation behind it.
While that is a fair point it won't help when the next attack is targeting something different: the update mechanism, the network stack itself or we're talking about some kind of interactive device like wind power that you want to remote control. So I am not if this argument while hold up for the next couple issues of this kind we will (certainly) see.
That was literally my point: this is the kind of event that is so specific that it isn't covered by the generic discussion of IT/OT best practices that's been laid out in the article, effectively making the article entirely unrelated.
I'm lost in the details of the various incidents now, but commercial scale chillers often need to defrost just their heat exchanger vanes, not the entire contents. This can be done without emptying the food out. This may be why some of this functionality exists - though obviously there's a strong argument it should be implemented very differently.
you don't run a defrost cycle without anyone there
Is this different than a home defrost cycle? Every fridge builds up ice on it and needs a heater to cut on and de ice itself from time to time. Most times a fridge is broken it’s because a 400w heater isn’t working and the ice is such a good insulator it can’t get cold enough.
I had assumed this was something similar but hackers repeatedly triggered it again and again rather than having an explicit “must remove everything from fridge” button.
It may be just de-icing the heat exchanger vanes. This wouldn't require emptying the contents first.
It is a true shame that on mobile there is no hover gesture, so I am still unaware of what OT could stand for. And even if there were, it is rude to not explain acronyms in plaintext in their first usage, especially if they are part of the title.
I found this frustrating as well. There is a glossary of acronyms at the end, but I also would have appreciated them being spelled out on first use.
They did! They used the <ABBR> tag on the first instance of nearly every TLA [1] and on the desktop, a tool-tip does show up using the title attribute. It appears that mobile browsers don't do that, sadly.
And to be a bit more helpful, the text for OT is: "Operational Technology: hardware and software that monitors and controls physical equipment and processes".
[1] TLA is a three letter acronym popularized by IBM, another TLA. I'll leave it to the reader to figure out what it means.
I took “one sensible business decision at a time” as a bit tongue in cheek, sort of condescending coddling clothing of “there there, your road to hell was paved with good intentions”.
Renewables are not “unpredictable” or “provide far less inertia” to the grid.
Wind is variable, but it is far easier to predict if the wind will blow tomorrow than if a steam turbine will require an emergency maintenance stop, or the average sunlight next year vs the average price of natural gas.
It is also not true that renewables can’t provide inertia, that’s simply a myth, as anyone that has been in an off grid building driven by grid-forming power inverters can tell you.
What is OT? I clicked on it to find that answer and. The author does not help.
I wish people at least spelled things out once before using acronyms.
I wish they had put this at the top instead of collapsed at the bottom.
Acronyms used in this post
- OT – Operational Technology: hardware and software that monitors and controls physical equipment and processes
- ICS – Industrial Control Systems: the computers, networks and devices that run industrial processes
- IIoT – Industrial Internet of Things: networked sensors and devices used in industrial settings
- PLC – Programmable Logic Controller: a rugged industrial computer that directly controls machinery
- RTU – Remote Terminal Unit: a field device that collects data and carries out commands from a central system
- SCADA – Supervisory Control and Data Acquisition: systems for monitoring and controlling equipment across wide areas
- DCS – Distributed Control System: a plant-level control system with controllers spread throughout a process
As noted in another comment, they did provide a hover text. Also, this is an extremely common acronym in enterprise IT. We are but programmer visitors; it's only fair if we occasionally have to look up something when reading blogs from an adjacent field.
hover text doesn't work on mobile. I figured that's what that weird dotted underline was supposed to be and tried to tap it and got nothing.
I walk the talk. If you can find an acronym on my blog I spelled it out at least once or it's a mistake and I need to fix it. It's a small thing but people who know it skip it and it's hugely helpful for those that don't. Or are in multiple fields and need to know PR means "public relations" and not "pull request" (or the other way around).
Though apparently it was spelled out at the bottom, so that's something. Just not where I would look.