When AI Grades Its Own Coding Homework

Agoda’s new APAC technology headquarters occupies floors 27 to 33 of Tower 5 at One Bangkok, a LEED Platinum-certified real estate development project under the Frasers Property group and Frasers Property (Thailand) Public Company Limited. Twenty-six thousand square meters, nearly 4,000 people, a network operations center for real-time platform monitoring, war rooms built for coordinated incident response, and content studios in a holistically integrated district located at the corner of Wireless and Rama 4 Roads in Bangkok.

It’s the kind of infrastructure that belongs in a telecom or an air traffic authority, not a company that books hotel rooms. This is where Idan Zalzberg, the platform’s chief technology officer, has spent the last four years solving a problem that most of the industry hasn’t yet admitted it has.

“We are essentially at 100% usage of AI-assisted coding. Nobody is coding anymore. Nobody is writing Python or Java — that practice is gone.”

This stopped being a fringe claim some time ago. Andrej Karpathy — OpenAI cofounder and the man who coined vibe coding — has described his own manual skills quietly atrophying, saying he now mostly programs in English. Leaders at OpenAI and Anthropic say the same about their own teams. The developers remain; the typing stopped.

The pressure migrates

The problem with eliminating a bottleneck at one stage is that it reappears somewhere else.

Telemetry compiled by Faros AI from more than 10,000 developers across 1,255 teams found that heavy AI adopters merged 98% more pull requests, while the time spent reviewing them ballooned by 91%.

Squeeze one stage of any pipeline and the pressure doesn’t vanish. It migrates, usually to somewhere nobody was watching. Agoda hit this directly. Code was arriving faster than any human team could read it.

The fix was to build an AI code reviewer. It now handles over half of all code changes, with a 92% developer satisfaction rate.

Sit with that.

Who checks the checker

An AI writes the code. A different AI checks the code. And generative systems — as Zalzberg is the first to acknowledge — aren’t deterministic: they produce different outputs from identical inputs. Feed the same prompt twice, and the math changes depending on how requests are batched on the server. Same input; different output. Even when randomness is theoretically disabled.

So, we asked Zalzberg the question this structure logically produces: when the thing checking the work is the same type of thing that did the work, and both are probabilistic, what is your ground truth?

“So, unfortunately, I do not think there is a single absolute ground truth in code review,” Zalzburg explains. He points out that humans were never consistent either — the same engineer reviewing the same code on a worse day catches something different, or misses it entirely. The ground truth was always partly a convention.

Having dispensed with certainty, Agoda built something it could actually measure: layered, verifiable trust grounded in three distinct mechanisms. Every change to the AI reviewer is rerun against hundreds of past code evaluations to check whether it still flags what it should and isn’t silently developing blind spots. Every significant model update ships as a blind experiment: half the developers receive the new version, and none of them know which. And to fight the variance baked into probabilistic output, each code change gets reviewed not once but four times, the verdicts folded into a single result. “We found that this reduces variance considerably,” Zalzberg says, “with a cost and latency trade-off that we are willing to make.”

Sample until the noise cancels out. It is an engineering answer to what is, at its root, a philosophical problem. And for Agoda, it works, right up to the point where you ask what “works” means.

The confidence trap

There is evidence that confidence in AI tooling shouldn’t arrive too easily. A randomized controlled trial by METR (Model Evaluation and Threat Research), with 16 experienced developers, 246 real tasks and a run from February to June 2025, found that AI coding tools made those developers 19% slower, while the developers themselves reported feeling 20% faster. The gap between perception and measured performance was not minor, and it persisted across every statistical cut the researchers tried. The developers didn’t feel slower; they felt like they were flying.

Zalzberg’s counter isn’t an algorithm. He has spoken publicly about wanting engineers who challenge AI output rather than simply receive it, treating a generated result as a draft rather than a conclusion. The moment a developer thinks they’re done is, he’s argued, the most dangerous point in any AI-assisted workflow. Protecting that instinct means rewarding the pause in a business where every incentive pushes toward shipping. “Personally, I spend much more time talking about cases where someone took an extra minute to think again or look again,” he says, “and much less time talking about people who simply got a lot of code merged.”

Performance reviews that celebrate hesitation, at a company under real competitive pressure, are not a small ask.

What Agoda actually is

Which returns you to the building, and to a question Zalzberg doesn’t fully close. Agoda won the BEHAVIOR Challenge at NeurIPS 2025 — a Stanford-run robotics benchmark with no obvious connection to booking hotels. The platform processes close to a million tokens per second. Its security team has, Zalzberg mentions, successfully penetrated every commercial AI chatbot they’ve tested. At what point does a travel platform become something else?

Zalzberg has thought about it. “Technology is only interesting to us when it serves that purpose.” He also adds, “Our goal is not to remove travelers from decision-making. It is to remove the parts of travel that feel like work. The agents we build are designed to propose, explain, and execute with consent, not to make decisions silently on a traveler’s behalf. “

Yet the reasoning that built all of this started four years earlier from a different kind of clarity. The week GPT-3.5 became available, he began building. Over time, we built the governance layers, visibility, controls, and most importantly, the skills and practices across the team. The logic was practical: If you give a customer an AI tool that doesn’t work, they don’t blame the AI tool but the company.

Essentially, prove it inside first — what Zalzberg calls his “inside out” AI strategy. Earn the right to ship it outward. Measure everything twice — or, if necessary, four times.

That philosophy was designed to protect a travel company’s reputation. On the 27th floor of One Bangkok and above, it has become the operating logic of something larger, and the category it belongs to is still being worked out.

Image credit: iStockphoto/Anton Vierietin