Who Learns What Good Looks Like? AI and the Evaluator Dilemma

Andrej Karpathy quote-posted a line from @yacineMTB that he says he has been citing a lot: you can outsource your thinking but you cannot outsource your understanding.

People learn to evaluate AI output by building, hitting failures and reviewing the work until they know what works, what doesn't and why. As coding agents take over that work, the entry-level roles where people built it are shrinking, which makes evaluation a supply problem.

I build closed-loop agent systems. A coding agent takes a ticket to a merged pull request, and every failure a check misses becomes a test the next run has to pass, so the system learns from its own mistakes.

Since June I've merged 441 pull requests across the two repos this loop runs in, and the skills it runs carry 309 eval cases. I still approve every merge, from the check results rather than a line-by-line read, and the first system I built this way does the line-by-line checking I used to do myself, which is the part of review where I learned, so I built a second system to put it back.

Review is where an engineer builds judgment. A coding agent works toward the checks it's given, so it stops once the tests pass, and code that passes isn't always the best way to write it. Green CI checks prove the code passes the tests it was given, not that those tests check the right behavior, and a reviewer knows the difference from domain knowledge and debugging experience, both earned by writing code and watching it fail. A loop where no person reads the diff removes that step, and that's my concern for the generation inheriting the solutions built today.

Anthropic found the same tension inside its own engineering team. In a December 2025 study of 132 of its engineers and researchers, the company describes a paradox of supervision, where checking Claude's work takes "the very coding skills that may atrophy from AI overuse." Respondents said Claude is now "the first stop for questions that used to go to colleagues," and one said "it's been sad that more junior people don't come to me with questions as often."

Hiring data shows where the entry-level openings went. Recent college graduates, ages 22 to 27, had about 5.6% unemployment and 42% underemployment in Q2 2026, per the New York Fed.

At the 15 largest tech companies, new grads were 7% of hires in 2024, and new-grad hires were down more than 50% from 2019, per SignalFire, a venture firm working from its own hiring data. By major, the New York Fed's table of 2024 American Community Survey data put recent computer science graduates at 7.0% unemployment and computer engineering at 7.8%.

Between May 2025 and May 2026, Indeed found 71% of the increase in US software development postings came from senior roles. The pattern points to companies recruiting evaluators while the roles that train the next ones shrink.

I went through the research to see what each study measured and how far its evidence reaches.

What changes when AI writes the code?

The engineer's job moves from writing code to reviewing what a model wrote, and on October 3, 2026, Futurism reported that some engineers can no longer make sense of the code AI produces, citing what Infinity founder Jeremy Nixon told Business Insider.

The same piece cites a 2026 survey commissioned by Undo, a company that sells debugging tools. Its 300 senior engineering leaders estimated that 35% of AI-generated code is not fully comprehended before teams push it to production.

Jordan Nanos of SemiAnalysis, quoted in that article, describes a model that understands and tests the kernels it writes and "produces correct kernels that perform really well." If that holds, the human job moves to defining correctness criteria, and someone still has to know what correct means.

Jensen Huang told Ezra Klein in September that Nvidia spends about 20% of its engineering effort on design and 80% on verification, and that most AI labs run roughly the opposite split.

Checking is also harder than it feels. In a randomized trial from METR, 16 experienced open-source developers took 19% longer on 246 real tasks when AI was allowed, working with early-2025 tools in repositories they had contributed to for multiple years. Before starting they expected a 24% speedup, and afterward they still estimated AI had made them 20% faster. If experienced engineers misread AI's effect on their own work, an evaluator's judgment has to rest on measurement rather than on how the work felt.

A 1983 paper described this problem

Lisanne Bainbridge published "Ironies of Automation" in Automatica in 1983. Her subject was process control in industrial plants, with examples from flight decks, and the mechanism she named is the one I now see with agents.

When automation takes most of the work, the human is left monitoring it and taking over when it breaks. Bainbridge pointed out that skills decay without use, so a formerly experienced operator who has been monitoring may now be an inexperienced one. She also asked whether monitoring can be done by an unskilled operator at all.

Bainbridge worried that the operators of her day were riding on skills learned before automation arrived, which later generations could not be expected to have, and engineers reviewing agent output are in that position now.

I see the same effect in my own work. I built a harness that runs the checks code needs before it ships, and it moved me on to the next build. With the harness running those steps and no feedback loop back to me, I noticed I was keeping less memory of doing them by hand, which is the effect Bainbridge describes.

Does relying on AI reduce what people learn?

In two controlled trials and one 30-month field study, yes, and an early EEG study points the same way. The size of the effect depends on how the AI is used.

In Anthropic's trial, the largest gap was on debugging questions, the ones about when code is wrong and why it fails, which is the evaluator's skill.

AI use did not guarantee a low score. Participants who asked the assistant follow-up questions and used it to understand the code scored well, while the ones who leaned on it to write or debug the code averaged below 40%.

Bastani's trial is the one that shaped the learning center I describe below. Nearly 1,000 students at a high school in Turkey practiced math with either a plain GPT-4 tutor or a version with prompts designed to safeguard learning. Practice scores rose 48% with the plain version and 127% with the safeguarded one. Once access was removed, the plain-version group scored 17% lower on the exam than students who never had it, while the safeguarded group largely avoided the drop.

The largest dataset comes from Chinese secondary schools. David Strömberg, Victor Lei and Yanhui Wu followed 26,811 students in grades 7 to 12 for 30 months, using staggered AI adoption to compare users with non-users. Adopting AI raised homework scores by 18% and cut homework time by 30%, while monthly closed-book exam scores fell 20% within six months. College and high school entrance exam scores fell 18% and 24%, and the full penalty took about two years to appear.

The behavior split matches Anthropic's. About 80% of AI users showed homework patterns consistent with outsourcing: unusually short completion times paired with high scores. Users who kept spending about as long on homework as non-users showed small losses, which makes time on task the signal that separated the two groups, and a tutor can measure it.

The MIT study is the most quoted and the least settled. It is a preprint with only 18 people in the final session, so I treat it as an early signal.

Why does evaluating AI depend on having built?

People check AI output less when they lack the skills to check it. Microsoft Research and Carnegie Mellon surveyed 319 knowledge workers who use generative AI weekly and published the results at CHI 2025.

Their finding: workers named a lack of skill to inspect, improve or guide AI output as one barrier to critical thinking; confidence in the AI went with less critical thinking, and confidence in their own ability went with more.

Is the pipeline of future evaluators shrinking?

The early data points that way. Stanford's Digital Economy Lab tracks ADP payroll records, and its August 2026 revision of "Canaries in the Coal Mine?" found employment for 22- to 25-year-olds in the most AI-exposed occupations about 19% below where it would be had it kept pace with peers in less-exposed jobs. The gap was 15% at the July 2025 data vintage and reached 19% by June 2026. Experienced workers show no comparable gap, and the adjustment appears to run mainly through fewer young hires rather than more people leaving jobs.

In the same revision, employment fell for young workers in occupations built on codified knowledge, the kind taught through textbooks and written procedures, and rose for experienced workers in occupations built on tacit knowledge acquired through practice, mentorship and repeated exposure to real situations.

The declines concentrate where AI usage automates tasks; where AI complements workers, employment is flat or rising. Evaluators draw on tacit knowledge, and the data shows fewer entry points for the people who would build it next.

Two Microsoft engineering leaders describe the same mechanism from inside the industry. In a Communications of the ACM opinion piece in April 2026, Azure CTO Mark Russinovich and Scott Hanselman argue that coding agents give senior engineers a boost and early-career engineers an "AI drag," because juniors lack the systems knowledge to steer and correct agent output.

They call the result a narrowing pyramid, and warn against hiring seniors while automating the simple bug fixes and coding tasks that used to train juniors. Their proposed fix is a preceptor model borrowed from medical education, with senior engineers responsible for developing junior ones on real product teams.

The juniors who are hired are learning differently too. An ICSE 2026 study from Oregon State describes a shift from apprenticeship to private copy-paste: pair programming, code review and mentoring give way to working alone with an assistant. One respondent feared juniors would take much longer to reach the level of developers with 10+ years.

The authors call the long-term risk a generational fault line. Experienced developers retire, and the cohort behind them has weaker foundations in design, debugging and testing.

What does this cost developers personally?

The same Oregon State study measured burnout. Across 442 developers, GenAI adoption was associated with higher burnout through increased job demands, meaning workload and organizational pressure. Autonomy and access to learning resources were associated with lower burnout.

In the open-ended answers, developers reported more time cleaning up and reviewing AI output, and higher output expectations, and one wrote that they move mountains of work with AI and are losing their passion for it.

This is why the learning center I describe below is framed around developer well-being, and why protected review time is on my team list.

What if the AI-native cohort learns differently?

The strongest counterargument comes from Nvidia's Jensen Huang. When Ezra Klein asked whether companies still need the same junior employees, Huang answered, "Wait two years." A degree takes four years, so by his math the first students who learned alongside capable agents reach the workforce around 2028, and he expects them to arrive as strong engineers.

He also concedes the cost. He expects engineers to lose "some finer intellectual dexterity" and become better systems thinkers, the way he once knew every transistor on a 200-transistor chip while today's engineers design systems they never inspect at that level.

I take the abstraction argument seriously. Math moved from paper to the calculator, then to code under the spreadsheet, then to Excel and its formulas, and now to tools built on top of them. Each step removed the need to remember how the layer below works, and compilers did the same for assembly.

A compiler transforms input by defined rules, while an agent generates output from a probabilistic model, and that output has to be checked before anyone relies on it. The checker's job moves from remembering how to write the code to understanding why it works, and that understanding still comes from building.

The Chinese school panel also cuts against the timeline. It is the closest thing on record to an AI-native cohort with measured outcomes, and its exam penalty took about two years to show in full, the same window Huang points to. Bastani's trial is the other half of the answer: students who learned with a safeguarded tutor kept their exam scores. Whether the 2028 graduates arrive able to evaluate depends on how the tools they learned with were designed.

How should companies evaluate engineers now?

A timed coding test asks a candidate to write code from memory and returns pass or fail. That fit a role where writing the code was most of the work. With agents writing more of it, the job is judging the output, and that test does not measure judgment.

Teaching that works takes time. You let the person generate the code and reach the failure on their own, then they say out loud what broke and you ask how they'll fix it.

Then you raise the question a level and ask what code needs before it ships, which is tests, coverage, QA, documentation and a lifecycle it moves through.

That same session is a better interview than a recall test, because you watch how someone works through the failure.

What I'm building in response

The second system puts the person back in the review step that the first system automated. I'm building toward an AI tutor with the same closed-loop method, with the learner placed inside the loop. A learner can talk to it while it walks them through code, and it moves the page to the part under discussion, highlights the lines it means, and drills down when the learner asks why.

The first version is a learning center that turns my merged pull requests into review drills. For now I'm the only user, and I get re-tested on the concepts I miss; on a team, juniors take the reviewer's seat and a senior rules on any catch a learner disputes.

Try the drill: Review Drill. It uses a sample diff with three planted defects.

Running my own agents end to end changed how I think about top-down and bottom-up management. Top-down, the evaluator holds the definition of good and hands work down. Bottom-up, people learn that definition by building. Agents remove most of the bottom-up path, so the tutor has to give juniors back the review practice they used to get.

The research turns into design constraints for the learning center. From Anthropic, the learner attempts each drill before any explanation, so the assistant builds comprehension and leaves the task in the learner's hands.

From Bastani and the Chinese school panel, learning counts only if a concept holds up on a later re-test without AI, and time on task is recorded for every drill. From METR, each attempt records the learner's confidence, so calibration can be measured later. From Oregon State and the Microsoft preceptor model, a senior rules on disputed answers, and peer review is the resource that builds judgment and lowers burnout.

What I'd do on a team

• Interview with a failure. I'd give the candidate a task, an agent and room to reach a failure, then score how they name it and fix it.
• Hire on potential. I'd use a short trial project on real work, which shows how someone learns in a way a timed test cannot.
• Be a preceptor. I'd take responsibility for developing a junior on a real product team, the model Russinovich and Hanselman describe.
• Protect review time. I'd keep time on the calendar for juniors to reflect on AI output, review it with peers and connect with mentors, as the Oregon State authors recommend.
• Rule on disputes. When a junior disputes an agent's review, I'd decide who is right, and the ruling becomes part of what the agent checks next.

What the evidence does not show yet

• Duration. The developer trials here measured hours or weeks, and the longest data, the 30-month Chinese school panel, covers students rather than working engineers.
• Causation in the job data. Stanford calls its results descriptive, not causal. The gaps shrink when the authors account for education, some differential trends predate generative AI, and the gaps are larger in the ADP sample than in national survey data.
• Causation in the burnout data. The Oregon State survey is cross-sectional, so it shows association.
• Tool generations. METR tested early-2025 tools on repositories the developers knew well, a hard case for AI. METR has since marked these results out of date and redesigned its follow-up study in February 2026 after wider AI adoption made the new data unreliable.
• Vendor authorship. Anthropic, which sells a coding assistant, ran the coding RCT and the internal study of its own engineers, and it notes likely selection and social desirability bias in the internal study. Microsoft, which sells GitHub Copilot, employs authors of the critical-thinking survey and the Communications of the ACM piece.
• Opinion pieces. Huang's interview and the Russinovich and Hanselman article are arguments from experience, so I use them for framing and for Nvidia's verification split, not as measurements.
• The Undo figure. The 35% is an estimate from surveyed engineering leaders, not a measurement of code, and the survey's sponsor sells debugging tools.

Sources