Mystery of 60-Second Fluctuations in Power Grids Solved
40 points by robalex
40 points by robalex
A thundering herd problem, but for power? Jitter saves the day as usual.
All I can think is software is such an unserious field that we keep repeating these mistakes.
I habitually use the systemd random jitter setting on my timer units to prevent this exact problem, even when it’s not really clearly necessary, and you’re telling me the people building these grid scale battery systems didn’t think of this before deployment?
Yes, That's exactly what we are telling you. Those of us who have been burned before are aware of issues like this. But not everyone is. I'm guessing that the people responsible for manufacturing batteries just hadn't experienced the problem. And it wouldn't show up with testing -- it's only when you have a huge number of these on the grid that there is any noticeable effect.
If only they treated time-keeping as unseriously, the problem would be avoided though! Uneven unseriousness is the worst.
On the topic of avoiding thundering herds, one of my friends has a neat way to randomize the timing of his cron jobs: while testing he sets the job to fire in the next minute, and once he has got it working OK, he leaves the minute at the current time.
(I tend to use numerology, which is more entertaining but less principled: eg, I set an hourly DNS rebuild to trigger at XX:53. I didn’t come up with that wheeze until after I was doing email; back then the hourly update was at XX:15 not XX:25. Memorable numbers are sometimes helpful if other people might need to know when an update happens, and the update can’t be fast for whatever reason.)
The simplest way to add jitter to a system is often to replace "do this every 60s" by "do this, wait 60s, repeat".
Yeah, if only Vixie Cron were able to do something like that :-)
It has the added advantage of avoiding problems if a periodic job unexpectedly takes longer than its period to run.
But how do all these devices synchronize their consumption?
This question wasn't fully answered by the article, at least not explicitly. I think the implication is that the timers themselves must be synchronized to the same clock somehow? (NTP, GPS, etc.) In other words, it's not sleep for 60 seconds, check power, repeat. It's sleep until the start of the next minute, and the devices all agree when the minute ticks over.
Systems that wouldn't cause that synchronization:
Random jitter is superior I’d imagine in every dimension, right? Even cheap random bytes that aren’t suitable for cryptography would be better than clock drift or cycle count.
I wonder if either the designers didn’t think of it or if they didn’t know they needed to spread out their load. It’s harder to assume the latter, so it must be the former, right? 🤔
Random jitter is superior I’d imagine in every dimension, right?
For sure, intentionally breaking synchronization by adding jitter is definitely better than assuming systems will naturally drift out of sync. But that's not what I'm asking. What I am asking is this: given that these systems are currently displaying synchronized behavior, what's keeping them from drifting?
To me, it feels like the news article stops short:
Something in such systems need to talk to the power grid accounting, and the current system setup default is to connect to the Internet everything in a subnet once any machine of the subnet is connected — and to run NTP on every Internet-connected machine.
Because the European grid operates at 50Hz, it's not unfeasible to treat it as a reference frequency for a clock. See https://en.wikipedia.org/wiki/Electric_clock#Synchronous_electric_clock
Even if the nominal timestamp on the grid differs from "real time" (i.e ~23s right now https://gridradar.net/en/net-time) all devices on the grid can have the same internal reference time.
Allowing the drift would presumably use every cycle of the grid, random jitter has a risk of people implementing it with one-second resolution (which is better than per-minute spikes, but can be further improved)
Oh, the research facility that poisoned the wells, featuring some hard to attribute but suspicious cluster of leukemia in its neighborhood and running a "hopefully in 40 years we'll have the tech to clean up our mess by the end of the century" strategy, still operates.