How is your org/company measuring the impact of AI adoption?

9 points by binchickin

Title says it best. We're at a point in our AI journey where tools are deployed, infrastructure stood up, features are shipping, and naturally, costs associated with AI usage are rising.

We're now being asked to estimate the impact of AI in our daily work.

How are you doing this at your company or within you org/department?

maduggan

So it's an interesting question. Part of what we're doing is small surveys to developers just basically checking their pulse and saying "is this useful or not". Then trying to have them rank the usefulness compared to other things that also cost money. Is it more or less useful than training materials, JetBrains IDE, etc. That answers part of the question, but that's more does it seem like it's doing something not empirical evidence.

The "evidence" part is harder. Since LoC are worthless as a measuring tool, instead we're trying to better track task duration through Jira. Are tasks getting done at the same rate that they were before (we have a convention of linking the Jira ticket and the PR for the task so it becomes possible to measure when the task is merged and "completed" for the most part) or are they done faster? Now this obviously has serious flaws as a testing methodology because all tasks are not the same and even if you factor in the "estimated size" this ends up being a bit confusing, but at the end of the day it is more data to look at.

The last one is stress levels. This is where we try and measure "do LLMs reduce stress levels for teams". Are employees happier with LLM tools or do they feel they add to the stress. We do this through surveys, in the one on ones, etc.

The end result is a somewhat scientific view of the rollout and the impact. Now in terms of what the data says is sorta all over the place. Some people feel LLMs are the most useful tool they've ever been given, others seem to hate them. In terms of speed of tasks there is an increase in certain kinds of tasks, often boring housekeeping type work or repetitive operations. But the results point to some weird findings, where there seems to be almost "LLM Superusers" who are powering through tasks at a crazy pace but most people seem to struggle to get the value out of the tooling.