Sysadmin In The LLM Age
15 points by packetcat
15 points by packetcat
The past two weeks I tried giving DevOps a shot with AI agents. A mix of Claude Opus 4.5, Gemini 3 Pro and GPT 5.2.
The goal was simple: Make a Terraform repository provisioning Amazon ECS, Amazon RDS, things on Cloudflare. Also provide the GitLab CI yaml file to build and deploy stuff automatically.
It came up with a structure how I would have done it but the details were atrociously wrong: Static AWS keys instead of OIDC, no HTTPS between Cloudflare and Amazon ELB, deploys cause downtimes, using deprecated Terraform modules, no build cache for the app in that repo and many more small things I had to fix myself.
Best of it all, it tried to gaslight me that I setup stuff wrong when I pointed out some issues it introduced.
To be fair, it helped me quite a lot avoiding to type a lot of Terraform states, but oh boy how that could have gone completely wrong if I hadn’t have prior knowledge of all of this.
If you're still early in this fun, here's a tip: ask for what you want, and if you're not sure, ask for questions. Modern LLMs are completely capable dealing with oidc, https, etc. If you want those things, ask for them. Alternatively, do a planning session for "... so this is what I want for a reliable production system. Keep asking questions about potential improvements and tradeoffs to refine the task." Otherwise the http to elb is a valid solution, just underspecced.
Modern LLMs are completely capable ...
Modern LLMs? What, unlike the decrepit ones from December? Just how old does an LLM have to be before it isn't modern?
Yeah. The capabilities of the newly released models change on monthly basis. We're still in the early period with constant changes. My threshold for modern is around Sonnet 4.5 so September, but I'm sure there are other opinions.
I'm including the "modern" or "recent" part when talking about what's possible, because people sometimes post views about their tests that they did half a year or longer ago... And yeah, maybe it was true then, but this area is moving fast.
I remember discovering first hand how utterly inept GitHub Copilot was with Terraform in 2024. I see the bots haven't gotten any better. I'm wondering how, though - surely there's a ton of code and doc for them to work from. Why are they so bad at this in particular?
"The sentiment on display in that quote is why we are in this goddamned mess to begin with. The extremely short term thinking, the absurdly selfish desire to solve a personal problem at the cost of the entire planet."
Exactly this.