Journal Club
Coding With Agents: What’s Different, What’s at Stake
Setting the Stage
From Autocomplete to Agents
- Autocomplete (2021–)
- AI suggests the next line as you type. Accept or reject. GitHub Copilot, Cursor tab completion.
- Coding agents (2025–)
- AI runs tools in a loop toward a goal — writes code, runs tests, reads errors, iterates. Claude Code, Codex CLI, Gemini CLI, OpenCode.
“An AI agent is an LLM wrecking its environment in a loop.” — Solomon Hykes (via Willison 2025a)
This is the core technical distinction for the session. Willison (2025a) defines an agent as “something that runs tools in a loop to achieve a goal.” The shift from autocomplete to agents is what changed DHH’s mind — he didn’t care for autocomplete (“When I code, I want to finish my own thoughts and sentences” (Hansson 2026)) but agents feel “like working on a team.”
Karpathy’s Software 3.0 framing helps (Karpathy 2026): Software 1.0 = explicit code, 2.0 = learned weights (ML), 3.0 = programming through prompts, context, and tools. The context window is the new program; the LLM is the interpreter.
Key Vocabulary
- Vibe coding (Karpathy; see (Karpathy 2026))
- Prompt-driven development with no attention to how the code works. “Accept the vibes.” Raises the floor — anyone can build software.
- Agentic engineering (Karpathy; cf. Willison’s “vibe engineering” (Karpathy 2026; Willison 2025c))
- The disciplined counterpart: specs, tests, review, ownership of outputs. Coordinating fallible agents while preserving quality. Raises the ceiling.
- Software 3.0 (Karpathy; see (Karpathy 2026))
- Programming through prompts, context, tools, and memory. The context window is the new program; the LLM is the interpreter.
Willison proposed “vibe engineering” in Oct 2025 and later acknowledged that “agentic engineering” was winning as the preferred term (Feb 2026 update on his post) (Willison 2025c). Karpathy formalized the distinction at Sequoia Ascent (Karpathy 2026). For this session, treat the two as roughly synonymous — both describe the disciplined end of the spectrum.
Karpathy’s floor/ceiling framing (Karpathy 2026): “Vibe coding raises the floor. Agentic engineering is about extrapolating the ceiling. People used to talk about the 10x engineer. I think this is magnified a lot more.”
Some attendees may have only watched Karpathy’s 30-minute Sequoia video (Karpathy 2026), which covers most of this vocabulary. The slides build on that foundation while adding the Willison and DHH perspectives they may not have encountered.
What Changed
The December 2025 Inflection
“I have never felt more behind as a programmer.” — Karpathy, Sequoia Ascent 2026 (Karpathy 2026)
- For most of 2025, agents were useful but required frequent correction
- Around December 2025: “the chunks just came out fine… I couldn’t remember the last time I corrected it”
- The unit of work shifted: typing lines \(\to\) delegating macro actions
- implement this feature, refactor this subsystem, write tests and fix failures
Models crossing the threshold: Claude Opus 4.5, Codex 5, Gemini 3
Tools: Claude Code (Feb 2025), Codex CLI (Apr 2025), Gemini CLI (Jun 2025), OpenCode
Karpathy pins the step change specifically to December 2025 (Karpathy 2026). DHH independently converges on the same timeline (Orosz 2026). The “macro action” framing is Karpathy’s — the programmer becomes an orchestrator, not a typist.
DHH also names open-weight models in this club: MiniMax M2.1, GLM-4.7 (Hansson 2026).
Note for RSEs: this is very recent. If your last serious encounter with AI coding tools was mid-2025, the landscape has fundamentally shifted. December 2025 was a step change, not a gradual improvement.
DHH’s Reversal
Summer 2025 (Lex Fridman podcast):
AI coding tools made “competence drain out of my fingers.” Programming is like playing guitar — you don’t let someone else play for you.
Spring 2026 (Hansson 2026):
“It’s more like working on a team… I just review the final outcome, offer guidance when asked, and marvel at how this is possible at all.”
What changed: not his philosophy, but the tools.
- Tab completion felt like someone stealing the keyboard
- Agents feel like wearing a mech suit
This reversal is the narrative centrepiece of the session — the most prominent sceptic went agent-first in ~6 months (Orosz 2026). DHH still cares about beautiful code, craft, and taste. But he barely types code by hand now.
His current setup: tmux with two models running (fast Gemini 2.5 + powerful Opus), neovim in the centre for reviewing diffs via Lazygit (Orosz 2026). He’s not doing vibe coding — he reviews everything, insists on quality, runs tests.
The “mech suit” vs “project manager” distinction matters (Orosz 2026). DHH was put off by the idea of being a project manager of agents. What changed was that agents became capable enough that he feels in control, hyper-accelerated — like wearing a mech suit, not supervising interns.
Could ask the room: has anyone here had a similar shift in the past year?
What Disciplined Practice Looks Like
Willison’s Framework
“If you’re going to exploit these new tools, you need to be operating at the top of your game.” (Willison 2025c)
AI rewards existing engineering practices:
- Automated testing — agents fly with a good test suite; without tests, they claim success unchecked
- Planning in advance — iterate on the plan first, then hand it to the agent
- Documentation — feed context; the agent can only keep a subset of the codebase in view
- Version control — agents are fiercely competent at git; use that
- Code review — you are now reviewing constantly
- Manual QA — predict and dig into edge cases the agent won’t flag
- Knowing what to outsource — and what to handle yourself
“AI tools amplify existing expertise.”
This is the core of Willison’s “Vibe engineering” post (Willison 2025c). The listed practices are not new — they’re what we already teach as good software engineering. The twist is that agents make them non-optional.
Without tests: the agent might claim something works without having actually tested it, and any new change could silently break an unrelated feature. Without documentation: it hallucinates APIs. Without version control: agent-introduced regressions go unnoticed.
The “amplification” claim is the most important single sentence in the reading set. We’ll interrogate it later. For now, note the implication for RSEs: if your group already does these things, you’re well positioned. If not — if your research code lacks tests, documentation, CI — agents will expose those gaps rather than paper over them.
Personal Aside: Reproducibility Is Part of the Equation
The readings cite tests, documentation, and version control as what makes agents fly. Reproducibility is conspicuously absent — yet it may matter as much:
Encourages separating side effects and statefulness — reproducible code is easier for agents to reason about. Functional programming’s edge in producing correct programs is the same advantage. An agent working on a pure, stateless module makes far fewer silent mistakes than one entangled in global state.
Enables agent bootstrapping — a fully reproducible project can tell an agent: clone, run the install script, run the tests, fix failures. E.g.
curl -fsSL https://pixi.sh/install.sh | sh && pixi run test. The agent can spin up its own environment in a sandbox or cloud instance with no human scaffolding. Good CI practices are the same idea, already automated.Improves verifiability — Karpathy’s thesis is that LLMs advance fastest where outputs can be verified. Reproducibility makes outputs verifiable: the same input must produce the same output, which is a directly checkable invariant.
Enables safe sandboxing — a self-contained, reproducible project can be handed to an agent running in a throwaway container or remote VM. This directly mitigates the exfiltration risk from the lethal trifecta (Willison 2025b): private data stays out of scope by construction.
Bisectability — deterministic builds make
git bisectreliable. An agent can automatically bisect a regression, confident that differences in output reflect code changes rather than environment drift.Idempotency — reproducible workflows tend to be idempotent. Agents can safely retry failed steps, re-run the pipeline, or roll back without worrying about accumulated side effects corrupting state.
This slide reflects personal experience rather than the assigned readings. The readings (Willison, Karpathy, DHH) consistently name testing, documentation, and version control as the preconditions for effective agentic work. Reproducibility subsumes and reinforces all three.
Point 1 connects to the functional programming literature. Immutability and the separation of pure and impure code reduce the surface area an agent needs to model. An agent working on a side-effect-free function is operating in a well-defined local world; an agent working on code that mutates shared state must track far more context.
Point 2 is particularly relevant for RSEs: research environments are often inconsistent across machines, making it hard to delegate tasks without extensive hand-holding. A reproducible environment (pixi, conda-lock, Docker, Nix) removes that friction entirely. Connecting to CI: good CI is a reproducible bootstrap loop, already automated and run on untrusted machines.
Point 3 reinforces Karpathy’s verifiability claim (Karpathy 2026). Reproducibility is not just a software engineering virtue; it is a precondition for the kind of feedback loop that makes agents effective. If the same code can produce different results on different runs, the agent cannot distinguish “my change broke this” from “the environment is flaky.”
Point 4 connects back to (Willison 2025b): YOLO mode becomes much less dangerous when the agent’s access is scoped to a reproducible, self-contained project image. The agent can run destructively without touching anything real.
Points 5 and 6 are practical consequences rather than philosophical arguments. Worth mentioning briefly to reinforce the theme: reproducibility is not just about scientific rigour, it is an operational property that makes the agent’s job easier and safer at every step.
Designing Agentic Loops
An effective loop needs (Willison 2025a):
- A clear goal with success criteria
- Tools the agent can use (shell, tests, linters, packages)
- A feedback loop (run \(\to\) check \(\to\) iterate)
Works best for: debugging, performance optimisation, dependency upgrades, refactoring — anywhere with trial-and-error and verifiable outcomes.
Cf. Karpathy’s verifiability thesis (Karpathy 2026): AI advances fastest where outputs can be verified (tests pass/fail, code compiles/crashes, benchmarks improve).
Willison (Willison 2025a): “Any time you find yourself thinking ‘ugh, I’m going to have to try a lot of variations here’ is a strong signal that an agentic loop might be worth trying.”
Karpathy’s verifiability framework complements Willison here (Karpathy 2026). “Traditional software automates what you can specify. LLMs automate what you can verify.” Coding is a sweet spot because feedback is immediate and unambiguous: tests pass or fail, code compiles or crashes. Karpathy also warns about “jagged intelligence” — models spike in capability where they are verifiable and heavily trained, and fail surprisingly in areas outside that intersection.
DHH’s P1 optimisation is a good concrete example of a well-designed agentic loop (Orosz 2026). A senior engineer at 37signals asked: what about the fastest 1% of requests — can we make the floor faster? Took P1 from 4ms to under 0.5ms. 12 PRs, ~2,500 lines changed, done in a few days as a side project. This is work that wouldn’t have been contemplated before.
YOLO mode (auto-approving all agent commands) is dangerous but dramatically more effective. Willison’s mitigations (Willison 2025a): sandbox (Docker, Apple containers), run on someone else’s machine (Codespaces), or accept the risk. “Most people choose option 3.” For RSEs handling sensitive research data, the exfiltration risk deserves special attention.
When to Use an Agent: A Personal Heuristic
- If you know how to do it
- Try it with an agent anyway. You can evaluate the output, catch mistakes, and learn to direct agents effectively.
- If you don’t know how to do it
- Don’t delegate to an agent (yet). You cannot evaluate the output — and having it done for you suppresses the learning you need.
The correct moment to introduce an agent is after you understand the problem well enough to judge the answer.
The second point draws on well-replicated findings that using LLMs to answer questions — rather than struggling through them yourself — reduces retention and understanding. This is the same phenomenon Karpathy flags at a higher level: “You can outsource your thinking, but you can’t outsource your understanding” (Karpathy 2026).
The first point is a strong argument for deliberate practice with agents on familiar territory. Most RSEs sceptical of agents have never tried them on problems they already know well. Using a known problem removes the confound of “is this output right?” and lets you focus on “how do I prompt, review, and steer?” That is where the real learning happens.
This heuristic also reframes the on-ramp: don’t start with your hardest unsolved problem. Start with the last bug you fixed, the last function you wrote, the last refactor you did. This is how you build taste for directing agents.
What’s at Stake
The Amplification Claim
The consensus across readings:
“AI tools amplify existing expertise.” — Willison (Willison 2025c)
“Vibe coding raises the floor. Agentic engineering raises the ceiling.” — Karpathy (Karpathy 2026)
Evidence:
- DHH: senior engineers at 37signals gain far more from AI tools (Orosz 2026)
- Amazon: juniors can no longer ship agent-generated code without review (Orosz 2026)
- Orosz survey: Builders benefit for large changes, but report identity loss and more AI slop to review (Orosz and Nilsson 2026)
But is this a comforting just-so story that flatters senior engineers?
- If amplification is real, what happens to the junior pipeline?
- Who develops expertise when the path to expertise changes?
The amplification claim is central to all the readings and worth pushing on. It’s also a claim that conveniently flatters the audience most likely to be reading these articles — and sitting in this room.
DHH reports the same at 37signals: seniors benefit more (Orosz 2026). The survey adds texture: Builders (quality-focused engineers) benefit from AI for large refactors and “quality of life” tasks that wouldn’t have been worth the time before. But they also deal with more AI-generated slop from colleagues, spend more time debugging agent output, and some report identity loss and grief (Orosz and Nilsson 2026).
The Orosz/Nilsson survey identifies three archetypes: Builders (quality-focused, benefit but also suffer), Shippers (outcome-focused, most enthusiastic, risk: faster tech debt), Coasters (can uplevel faster, risk: generating slop) (Orosz and Nilsson 2026). AI amplifies the tendencies that existed before.
The junior pipeline question is economic territory we’re not opening today, but name it as a known gap.
The Taste–Talent Gap
Ira Glass’s observation: individuals acquire taste much faster than talent.
Coding agents narrow this gap:
- Taste drives the work; skill executes it — and agents now execute
- Corollary: invest in taste (fast to acquire, high leverage) over memorising syntax (slow, increasingly low leverage)
The flip side — if you lack taste or domain knowledge:
You cannot judge outputs you cannot recognise as wrong.
A deeper corollary: as AI capability surpasses human ability in a domain, the human stops noticing further advances — because evaluation requires being close enough to the ceiling to see it.
Ira Glass’s “taste-talent gap” is about the frustrating early years of creative work: your taste (what you know is good) outpaces your current ability to produce it. Coding agents invert the usual bottleneck — taste becomes the binding constraint, not raw coding ability.
This reframes Willison’s amplification claim (Willison 2025c): what gets amplified is taste and domain knowledge, not raw coding speed. A researcher with deep physical intuition who never mastered Python can now direct agents well. A competent programmer with no domain understanding cannot.
The evaluation ceiling problem is the converse of the amplification story: once a model’s capability in a domain exceeds the reviewer’s level, the human can no longer reliably distinguish excellent outputs from merely plausible ones. For RSEs, this is a near-term practical concern in specialised subfields — and it makes domain expertise the durable investment.
Economic Dimension (for reference — not for discussion today)
Orosz & Nilsson survey (900+ engineers) (Orosz and Nilsson 2026):
- Companies spending $100–200/month per engineer on AI tools
- ~30% of engineers hitting usage limits
- Cost trajectory widely considered unsustainable
- UK/EU companies significantly more budget-cautious than US
Three archetypes:
- Builders
- Care about quality and craft. Benefit from AI for large changes. Risk: identity loss, AI slop.
- Shippers
- Focus on outcomes. Most enthusiastic about AI. Risk: faster tech debt, building the wrong things.
- Coasters
- Can uplevel faster with AI. Risk: generating slop that frustrates Builders.
This slide is reference material only. We deliberately excluded the economic dimension from today’s discussion — the readings were selected to focus on the technical, psychological, and philosophical dimensions instead. If attendees raise cost or labour-market concerns, acknowledge the omission rather than improvise.
The METR study (not assigned reading) found that experienced open-source developers believed they were 20% faster with AI tools while being measurably 19% slower. Worth mentioning as a provocation if anyone insists on the productivity question, but do not open it as a full discussion thread.
Discussion
Over to You
Willison’s vibe engineering requires rigorous tests and specs to be responsible. What does responsible agentic coding look like for research software, where the test harness often doesn’t exist and “correct” is contested?
This is the bridge from the readings to the audience’s working context. Pause here and let the room sit with the question before moving to the canned questions.
Research software often lacks the preconditions that Willison and Karpathy assume: comprehensive test suite, clear success criteria, stable specs (Willison 2025c; Karpathy 2026). In research, “correct” might mean “agrees with physical theory,” “matches experimental data within error bars,” or “reproduces a known benchmark” — none of which are trivially automatable. The tension is sharp: the readings all agree that tests are the foundation of effective agentic coding, but RSEs often work in domains where the test harness is the unsolved problem.
Discussion Questions
DHH reversed his position in ~6 months. Has anyone here had a similar shift? What triggered it?
Willison says AI amplifies existing expertise. Does that match your experience — is there a skill AI makes less valuable?
“You can outsource your thinking, but you can’t outsource your understanding.” Where’s the line for research software?
If an RSE uses an agent to write code for a research project, who is responsible for the correctness of that code? Does the answer change if it produces a published result?
Individual RSEs may have strong AI preferences — from agent-first to AI-free — and project PIs have their own requirements driven by personal preference or funding constraints. Should RSE–project matching take these into account, so that an RSE who only works with agents is not assigned to a project that prohibits AI use, and vice versa?
You probably won’t get to all five. Pick one or two that the room gravitates toward. Question 1 is a warm-up that also surfaces how many attendees have hands-on experience. Questions 2–3 probe the core claims. Questions 4–5 are RSE-specific and likely to generate the most productive discussion.
If time is short, prioritise 4 or 5. Question 4 connects to academic integrity and reproducibility — familiar ground for RSEs. Question 5 connects to practical team management — the tension between individual tool preferences and project-level constraints, which is likely to surface real experiences.
Non-Questions (out of scope today)
“Will AI replace RSEs?”— labour-market predictions“Which tool should we use?”— tool recommendations“Does AI actually make us faster?”— empirical productivity measurement“How do we get budget for AI tools?”— procurement
Naming what’s out of scope prevents drift into familiar but unproductive territory.
On “does AI actually make us faster”: the METR 2025 study (an RCT with experienced open-source developers) found a striking mismatch — developers believed they were 20% faster while being measurably 19% slower. Mention this if someone insists on the productivity question; do not open it as a full thread.
On “which tool”: tempting for a technical audience but will consume all available time and produce no actionable insight in a one-hour session.
These are not unimportant questions. They’re just not today’s questions.
Readings
- New, optional: Karpathy (2026)’s From Vibe Coding to Agentic Engineering
- Mandatory: Willison (2025c)’s Vibe engineering
- Mandatory: Orosz (2026)’s DHH’s new way of writing code, which is a summary of a 2hr podcast.
- Recommended: Willison (2025a)’s Designing agentic loops: good companion to (1) above.
- Optional: Orosz and Nilsson (2026)’s The impact of AI on software engineers in 2026: key trends
- Optional: Hansson (2026)’s Promoting AI agents: DHH’s own (shorter) blog post making a statement similar to (2) above.
Included for reference during discussion. Some attendees may have only watched Karpathy’s 30-minute video, which covers most of the conceptual framework (Software 3.0, vibe coding vs agentic engineering, the December inflection, verifiability). The slides are designed to fill in what Karpathy doesn’t cover: Willison’s specific practice framework, DHH’s personal reversal story, and the Orosz/Nilsson survey data.
Karpathy (2026)
The unit of programming changed from typing lines of code to delegating larger “macro actions”… This is why I think the profession is being refactored. The programmer is increasingly not just a code writer, but an orchestrator of agents.
My core automation framework is:
- Traditional software automates what you can specify.
- LLMs and reinforcement learning automate what you can verify.
capability spike ~= verifiability x training attention x data coverage x economic value
I distinguish two related but different ideas:
- Vibe coding raises the floor. It lets almost anyone create software by describing what they want.
- Agentic engineering raises the ceiling. It is the professional discipline of coordinating fallible agents while preserving correctness, security, taste, and maintainability.
The old “10x engineer” idea may become much more extreme. People who master agentic workflows may outperform others by far more than 10x.
This means products need agent-native surfaces… I think about this in terms of sensors and actuators. A sensor turns some state of the world into digital information. An actuator lets an agent change something. The future stack is agents using sensors and actuators on behalf of people and organizations.
The right posture is neither dismissal nor blind trust. It is empirical familiarity: learn where they work, where they fail, what they were trained for, and how to build guardrails around them.
You can outsource your thinking, but you can’t outsource your understanding.
The scarce thing is shifting:
- Less scarce: code generation, API recall, boilerplate, first drafts, repetitive setup, simple transformations.
- More scarce: understanding, taste, eval design, security, system boundaries, agent orchestration, domain-specific feedback loops, and knowing when the model is off the rails.
My current worldview is not that AI simply makes everyone faster at the old work. It is that the work itself is being reorganized around agents. Software, research, education, infrastructure, and knowledge work are all becoming variations of the same pattern:
- define the context
- define the tools
- define the feedback loop
- define the guardrails
- let agents work
- preserve human understanding
Willison (2025c)
I feel like vibe coding is pretty well established now as covering the fast, loose and irresponsible way of building software with AI—entirely prompt-driven, and with no attention paid to how the code actually works.
If you’re going to really exploit the capabilities of these new tools, you need to be operating at the top of your game. You’re not just responsible for writing the code—you’re researching approaches, deciding on high-level architecture, writing specifications, defining success criteria, designing agentic loops, planning QA, managing a growing army of weird digital interns who will absolutely cheat if you give them a chance, and spending so much time on code review.
Almost all of these are characteristics of senior software engineers already!
Orosz (2026)
A big win from using AI agents is tackling stuff that you wouldn’t have before. A senior engineer at 37signals ran a “P1 optimization” project to improve the fastest 1% of requests.
Running several AI agents feels less like “project management” and more like “wearing a mech suit.”
37signals has one designer for every two engineers.
AI agents could turn 37signals’ “designer model” into the industry standard.
Command Line Interfaces (CLI) feel like the ultimate AI interface, which validates the Unix philosophy of the 1970s.
Eight hours of sleep is non-negotiable – even during an AI gold rush!
Willison (2025a)
The thing to look out for here are problems with clear success criteria where finding a good solution is likely to involve (potentially slightly tedious) trial and error.
Willison (2025b)
The lethal trifecta of capabilities is:
- Access to your private data—one of the most common purposes of tools in the first place!
- Exposure to untrusted content—any mechanism by which text (or images) controlled by a malicious attacker could become available to your LLM
- The ability to externally communicate in a way that could be used to steal your data (I often call this “exfiltration” but I’m not confident that term is widely understood.)