Understanding is the new bottleneck

Always has been?

I follow the author on Twitter and have tried some of these techniques myself after seeing their posts. I have not found them to work well in many cases.

Specifically, some of the problems I've seen are:

1. Agents introduce performance issues which they omit from their explanations/guided tours.
2. The questions they create for quizzes often just involve regurgitating text already in the explanation. If you ask them to generate more tricky questions, the answers can be wrong, or in follow-up discussions they will flip-flop.
3. Diagrams generated are a mess in terms of layering/clarity, and require multiple rounds of prompting to be sensible.
4. The tour can focus on the diff itself, but the problem is that the bug is introduced die to an interaction with unmodified code (i.e. not thinking of correctness in terms of "for all code paths").

Maybe this works better for the areas the author works in.

I'll try the skill to see if it's any better, but my hopes are relatively low.

While I love the focus on understanding and how LLMs can help, I took two big pauses:

1. Learning the background of a code change is good, but is it necessary on every change? I already hate reading LLM output cause it’s verbose and often misses the point. Or to say, everything is the point. And so, like the blog says, books / reading is a terrible way to learn. I often find LLM generated explanations to make things worse unless you have very targeted questions and a wealth of internal knowledge to correct yourself against.

2. Alan Kay may have dreamed of how computers could help kids learn, but the actual roll out of Ed tech has had pretty sad effects on actual learning outcomes. It’s hard to say if it’s the effects of our attention economy hurting children, or poor implementation of Ed tech. But the results are not good. In fact, they’re bad enough that we’re now banning such tech in schools.

I do think LLMs can quickly build tools that could aid in building mental models. But do we really need to build more software to understand other software? Who validates that it’s correct? At least on a software team, we learn together publicly and communicate our mental models to reach a concensus. My fear with AI assisted learning is we all learn in silos because it’s faster and easier that way.

The author says we can address the communal learning by doing things in shared docs, but that seems to overwhelm the human learning process more than anything. I guess that’s the point of the article? We are now bottlenecked on human learning, reading, and understanding.

Often when I see these kind of posts I think that it must be nice to have that amount of time at work and in your free time to be able to do these things.