Skip to content
nythrex.
Log inGet started

Engineering culture · AI metrics

Measure AI, not the people using it

The board wants to know whether the AI budget is working. The quickest chart to produce is adoption: who uses the tools, how often, how many tokens. It looks like progress. To the team it looks like surveillance, and it quietly changes behaviour in the wrong direction. Here is what the data says and what to measure instead.

Create account
Engineering culture · AI metrics. Measure AI, not the people using it.
Usage per person is the easiest AI metric to collect and the hardest to defend.

Nythrex team · 10/29/2026

In brief

6-minute read · For CTOs and engineering leads · 29 October 2026

  • Developers are worried. In a Harness survey of 700 practitioners and managers, 54% of developers fear evaluations tied to their AI use.
  • Usage metrics distort behaviour. In an Adaptavist survey of 1,000 developers, 52% use AI more than necessary to meet expectations.
  • A big company already reversed course. In September Meta stopped counting AI usage in performance reviews.
  • The better unit is the change: what AI cost per shipped piece of work, by project and team.

How developers experience AI metrics

54% of developers fear performance evaluations tied to their AI use. 55% want improvement data kept apart from performance data.
Source: Harness survey of 700 practitioners and managers, reported by IT Brew.

The same Harness survey found that 55% of developers want improvement data kept separate from performance data, and 49% want a say in defining the AI metrics used in evaluations. A separate Adaptavist survey of 1,000 developers in five countries adds the behavioural side.

Horizontal bar chart: 58% of developers say AI use is becoming a formal KPI, 54% fear evaluations tied to their AI use, 52% use AI more than needed to meet expectations, 47% say they are judged on how much they use AI rather than on quality.
Two 2026 developer surveys. Chart: Nythrex, from Adaptavist and Harness.

The metric that backfired

Meta made AI use part of its review criteria, tracked through dashboards and token counters. Staff responded by running up token consumption to rank well on internal leaderboards, the practice now called tokenmaxxing. On September 8, according to an internal memo reported by The Information, Meta said AI usage would no longer factor into reviews; performance would be judged on the quality, speed and complexity of the work.

Quote: “Organisations that want to hold onto engineering talent need to stop measuring AI adoption.” Matt Saunders, DevOps lead, The Adaptavist Group.
From IT Brief’s coverage of the Adaptavist survey, October 2026.

Why usage per person is the wrong unit

  • It measures input, not output. Tokens and prompts say nothing about whether anything shipped.
  • It gets gamed. Once usage is a target, usage goes up. So does the bill.
  • It erodes trust. People who feel watched share less, experiment less and leave sooner.
  • It answers the wrong question. The board asks whether AI is worth the money, not who typed the most prompts.

The question is whether AI is worth the money, not who used it most.

What to measure instead

From people metrics to work metrics
Instead ofMeasureWhy it works
Tokens per developerAI cost per shipped change, by projectTies spend to delivered work
Adoption rateShare of AI spend that ended in merged workShows waste without blaming anyone
LeaderboardsSessions and runs that produced no resultPoints at process, not people
Prompts per dayLead time and change failure rateMeasures delivery, which is what AI is for
  1. 1

    Separate improvement from evaluation

    Say in writing that AI cost and usage data is for improving the system, not for reviews.
  2. 2

    Aggregate by project and team

    Report AI at the level where decisions are made: which projects, which kinds of work.
  3. 3

    Let people see their own data

    Developers can learn from their own numbers without seeing colleagues by name.
  4. 4

    Define the metrics together

    Half of developers want a say. Asking costs nothing and buys trust.

Where Nythrex fits

Nythrex measures AI at the level of work: the cost of each change from task to AI session, commit, pull request, CI and deploy, by project and team. It does not compare people with each other or rate productivity. A member sees only their own work and cost. From developer machines it receives request metadata only, never code or prompts.

Security and privacy

Sources

Harness figures are from IT Brew’s coverage of the survey; Adaptavist figures from IT Brief UK, October 19, 2026; the Meta decision from The Decoder, citing an internal memo reported by The Information.

Frequently asked questions

Should companies track developers’ AI usage in performance reviews?

Evidence suggests not. 54% of developers in a Harness survey fear such evaluations, 52% in an Adaptavist survey use AI more than necessary to meet expectations, and Meta stopped counting AI usage in reviews in September 2026.

What is tokenmaxxing?

Tokenmaxxing is running up AI token consumption to look productive on usage dashboards or leaderboards. It raises the AI bill without improving the work.

How can leaders measure the impact of AI without surveillance?

Measure at the level of work: AI cost per shipped change by project and team, the share of AI spend that ended in merged work, and delivery metrics such as lead time. Keep improvement data separate from performance reviews.

Read next

Stop guessing.
Start seeing.

Create account

No card and no subscription: paid plans have not launched yet.

Create accountNo card, no subscription