Know thine enemy: A critical engagement with AI-assisted software development
35 points by projectgus
35 points by projectgus
I've mostly stopped reading articles about AI-assisted coding (there are so many!) However, I'm personally glad I read this one.
During implementation, it would interrupt every few seconds to minutes, asking for permissions. [...] This was even worse if I was working on two, three, or more sessions in parallel, with interruptions every minute.
Coding agents running in YOLO mode (where you don't have to approve anything they want to do) are a completely different tool from coding agents in the default ask-for-permission mode.
This is one of the many unintuitive things about these tools which help explain why people have such wildly different experiences in terms of how useful or frustrating they are.
(Thankfully Claude Code has "auto" mode now, which is effectively YOLO mode but less terrifyingly dangerous.)
My preferred starting point for coding agents is still Claude Code for Web, since running them in a container on someone else's computer means they can use YOLO mode by default without anything irreversibly bad happening if something goes wrong.
My preferred starting point for coding agents is still Claude Code for Web, since running them in a container on someone else's computer means they can use YOLO mode by default without anything irreversibly bad happening if something goes wrong.
I hear you that it might be safer, but I can't fathom trusting YOLO mode, even in the cloud.
From the docs:
With either method, a cloud session can access any repository the connecting GitHub account can see, not just the repositories the Claude GitHub App is installed on. App installation enables PR webhooks for Auto-fix; it is not a session-level access control. To restrict which repositories your team can reach from cloud sessions, restrict access on GitHub itself, for example by limiting team or repository membership for the connected GitHub accounts.
I'm sure this footgun is going to result in Claude reaching further than some expect, if they think it can only get to one repository.
A dedicated secrets store is not yet available. Both environment variables and setup scripts are stored in the environment configuration, visible to anyone who can edit that environment. If you need secrets in a cloud session, add them as environment variables with that visibility in mind.
"Just give the agent the ability to read your secrets and use them" is an atrocious idea unless you have very good secrets management. Especially when the default allow list for domains is massive and includes all the normal living off the land locations for stashing purloined data.
If your repository uses comment-triggered automation such as Atlantis, Terraform Cloud, or custom GitHub Actions that run on issue_comment events, be aware that Claude can reply on your behalf, which can trigger those workflows. Review your repository’s automation before enabling auto-fix, and consider disabling auto-fix for repositories where a PR comment can deploy infrastructure or run privileged operations.
It is warnings like this that make me strongly suspect the average user is going to have no idea this is intended behavior until well after the malicious PR rolls through and ruins their weekend.
Is it safer than running YOLO on your own box? Sure, but man I hope they make you attest to having read these docs before checking out a web VM and letting Claude go crazy in it.
I don't think you can be secure with manually approving every command, either. The commands being harmless 99.999% of the time is the worst kind of fatigue for a human.
You'll develop a reflex of selecting "Approve" before your brain finishes thinking about the command. It can also get really tricky and exhausting to evaluate the commands properly once the agent starts making more complex bash pipelines or ad-hoc python scripts.
I don't have an ideal solution, but sitting there and clicking "Approve" every 30 seconds is a massively flawed model that has huge cost in time and distraction, and it's a Vista-level security theater.
Agreed - it’s just like Tesla autopilot: “keep your hands on the wheel at all times because once every N years you may need to manually prevent a fatal accident”.
Terrible ergonomics
Sure, but I see those as vastly different threat models. On the one hand, I can review every command that is generated if I'm diligent. I can come back to it when I want and the command will just sit there.
On the other, there is no opportunity to and you have to read the scroll as it flies past. If you miss the "That approach didn't work, reinitializing the git repo" message, then it will blow away your code and keep on trucking regardless.
With either method, a cloud session can access any repository the connecting GitHub account can see
That gave me a bit of a panic just now because it didn't fit my mental model of how the GitHub permissions work.
Turns out that documentation is a bit misleading. A Claude Code for web session can only interact with the GitHub repo you start it against, and in fact all GitHub interactions go through their own proxy which further limits it to just being able to push to a new dedicated branch for that session.
Interestingly, if you ask it to clone another private repo for reference it now pops up a permission dialog asking for a new grant. I hadn't seen that before and it's a feature I've wanted for ages.
all GitHub interactions go through their own proxy which further limits it to just being able to push to a new dedicated branch for that session.
Oh that's handy! I was thinking about trying to build something like that for running locally too. I think it should be possible to set one up for generic git access through a local repo with a few pre- and post-receive hooks, which would also have the bonus of not being specific to GitHub but working with any git forge.
I usually run claude-code in a devcontainer (after enough times of setting this up, I now have a nearly perfect template for my projects), then everything is in automode. That way the blast radius of claude getting things wrong is inside the container. It saves a lot of time and you can just use a normal IDE on top, some even support spinning up inside a devcontainer so it lives side-by-side with the coding harness.