← Blog

Engineering note

Jev Beat Pokémon Red: Was It the Model or the Harness?

Huahua examines a handheld game showing a pixel maze; a state notebook and copper compass represent the division between model decisions and game code.

Christian Mathiesen, the project author and a Jev user, streamed TypeSafe AI’s Jev decision model completing Pokémon Red through the game-specific harness he built on September 25–26, 2026. The run took 37 hours and 40 minutes and involved 16,150 decisions. It is easy to shorten that result to “an AI played through the game on its own.” Mathiesen’s public code raises a more useful engineering question: when every decision depends on game state decoded by software, prepared options, and computed routes, what part of the result does the model actually measure?

The answer is not the model or the harness alone. This run measures a composed system: Jev judges what to do from program-provided options, while game-specific code turns the environment into decision-ready state, computes movement, and checks whether objectives are complete. That division is worth studying, but the whole system’s result should not be attributed to the model by itself.

First, distinguish the two Jev playthroughs

There were two Jev-powered Pokémon Red playthroughs around the same time, and their details should not be combined. Andrew Boyd’s page says Jev entered the Hall of Fame on September 23 and credits Opus 5 with monitoring the game log and building the harness in real time. Christian Mathiesen used Jev in a separate run on September 25–26, through the game project he published; this article focuses on that run because it comes with a public GitHub repository and a detailed README. Boyd’s project page and Tom’s Hardware’s contemporaneous report also describe them separately.

Mathiesen’s README describes Jev as the decision model and reports a 37-hour, 40-minute run, 16,150 decisions, about 39.2 million input tokens, about US$1.65 in Jev costs, 16 team wipes, and 15 Elite Four attempts. The project page repeats some of these figures. They are statistics reported by Mathiesen, not independent verification from two pages; this article did not rerun the game and does not borrow Opus’s involvement or statistics from the other playthrough. README · Project page

Jev chooses; the program turns the game into options

TypeSafe’s documentation describes Jev as a model that receives state and typed questions, then returns structured answers that software can use. A Choice question selects from a supplied set of options and returns probability and confidence information; Jev does not need to generate free-form text describing which button to press. The official introduction recommends breaking decisions into narrow questions and composing their results in code.

This game project places that interface inside a dedicated harness. The program reads the Game Boy emulator’s RAM to obtain the map, coordinates, party, battle, inventory, and screen text. A comment in GameState explicitly says this is pure observation and does not write to game memory. The state reader and the README’s system description make that boundary inspectable.

The harness does not hand the model raw RAM and stop there. It builds legal actions with supporting facts. Battle choices may include move type, estimated damage, or whether a move could knock out a target. Overworld choices can include walking to an NPC, taking an exit, using an item, or pushing a boulder. The battle code computes and presents some combat information; the overworld code builds paths and candidate actions.

So neither “Jev chooses” nor “the harness decides everything” is quite right. Jev still chooses whether to progress, train, heal, catch, shop, or manage its team, as well as dialogue and battle actions. But the game program has already parsed and constrained the choice space. It converts many low-level problems into candidate actions before the model judges among them.

Paths, goals, and completion checks live outside the model

Long tasks require more than choosing a good move in the current moment. The system also needs to know where to go next and what counts as completion. This project uses a milestone list to provide a goal and its location. The README says progress is checked against in-game event flags rather than trusting a model’s claim that it finished. The milestone code says the program gives the goal and where to pursue it, while Jev still decides where to walk, who to talk to, how to fight, and when to heal, grind, or catch.

The program also calculates tile movement. overworld.ts uses A* pathfinding and a region graph to reason about passages, obstacles, interactable objects, and distance to goals, then gives Jev meaningful options. That lets the model spend its decisions on questions such as where to make progress or whether to restock instead of deriving a sequence of directional inputs from coordinates on every turn.

This design reduces task difficulty, and it is also part of the system’s capability. If the goal is to build a game-playing system that can run for a long time, these software supports are not cheating; they are part of the product. If the goal is to compare a model’s visual understanding or general navigation ability, they must be listed as experimental conditions. The final result alone cannot settle that comparison.

The failure counts make “beating the game” look like system resilience

Mathiesen reports 16 full-team defeats and 15 attempts at the Elite Four. Those figures are a reminder that success was not a smooth sequence of good choices, and that it depends on more than one-step decision quality. The README describes tracking repeated choices, adjusting options when the agent gets stuck, saving at milestones, and returning to a checkpoint when necessary. The program accumulates no-progress decisions, map state, and attempted actions to inform its recovery behavior. The agent tracking code exposes some of that progress and recovery logic.

From an engineering perspective, the central result is not simply whether the model made mistakes, but whether the system could continue after errors, repeated actions, and failed battles. Long-task reliability often comes from environment reading, viable actions, external progress checks, memory, and recovery strategies working together. Reporting only completion time or success rate omits those mechanisms; reporting only model accuracy cannot describe whether the whole workflow keeps moving.

What does this run support—and what does it not?

It supports a concrete observation: a typed decision model can be placed inside a game-specific state machine and harness, make decisions from constrained options for a long period, and rely on code to verify progress. Public source code makes the responsibility boundaries studyable rather than leaving only a completion screen.

But this does not show that Jev could finish the game without the game-specific software, and the project does not provide a baseline comparison against a simplified harness. No independent team has rerun it, so the statistics remain Mathiesen’s report. The task also depends on one game, game-specific ROM data, map logic, and milestones; the result does not establish transfer to other games or unfamiliar environments. A reader can reconstruct how the system is designed without treating one run as a generalization test.

To explore it, start with the README, then inspect src/game/state.ts, src/knowledge/milestones.ts, and src/agent/. Rerunning requires macOS or Linux, Node 20 or newer, your own legally obtained US or EU Pokémon Red ROM, and an AI Gateway key. The project page is not a live playable web demo; it mainly provides run statistics and highlights.

Turn “model capability” into a responsibility checklist

If you evaluate your own agent, this case suggests more concrete questions: Does the model see the raw environment or a prepared state? Which options has code removed? Who sets and tracks goals? Is completion reported by the model or verified against external state? When the same error recurs, does the system retry, change strategy, ask a person, or keep looping?

Those answers explain more than “the model beat the game.” If you first want to understand Jev’s typed decision interface, continue with TypeSafe AI and Jev. For ways to place Jev in an agent runtime, see confidence-gated routing. This article looks at a different layer: how one long-running task connects state, model decisions, program execution, and verification.

Sources and further reading