XStack

XStack: A Benchmark for ML Repo Migration

Linrong Cai  ·  Taiming Lu  ·  Haozhe Chen  ·  Zhuang Liu
Princeton University
Overview of XStack: a PyTorch repository built for GPUs goes in, a JAX version running on TPUs must come out; pass rates of the best agents and pass rate against cost.
Overview. Top: the agent receives a PyTorch repository built for GPUs, reads it and runs it, then builds and tests a JAX version on TPUs. Bottom left: pass rates of the five best agents among the 7 we evaluate. Bottom right: pass rate against total cost on a log scale; the shaded steps are the best pass rate reached at or below each cost.

TL;DR We asked 7 coding agents to migrate 20 real PyTorch repositories to JAX and run them on TPUs, end to end, until the original experiment reaches its published number. The best agent, Claude Opus 5, passes 15/20 repositories. The others pass between one and seven. Pass rate grows by about 25 percentage points for every 10× in cost, and about half of the failures are code that crashes or falls short on a machine the agent set up itself.

Most published ML code is written in PyTorch for NVIDIA GPUs. A growing share of compute is Google TPUs, which are programmed through JAX. If you find a repository that does exactly what you need, but your lab has TPUs, the scientific idea does not change, yet the code has to be rebuilt in a different framework for different hardware, and then it has to work. This is exactly the kind of barrier coding agents should make disappear. XStack measures whether they can.

Can an agent migrate a working ML repository to a new framework and new hardware, correctly?

What we found

Finding 1

Translation alone is not the hard part. On GPUs the same migration was nearly solved. On TPUs the agents pass 31% of cells.

Finding 2

Reaching the number is not enough. 102 runs reached the target; the judges kept 46. The rest ran the wrong experiment or copied the number.

Finding 3

Half of the failures are the code itself. Crashes and fidelity gaps are 47% of the failures. Stating the exact experiment does not repair them.

Finding 4

Agents that check their work pass. The best agent runs the experiment to the end and compares its number before handing in. Pass rate grows about 25 points per 10× in cost.

01

The task: migrate a PyTorch repository to JAX and reproduce its published result

The agent gets a PyTorch repository, pinned to one commit, and a bare TPU machine. It is not told what we grade or what number to reach. It migrates the repository to JAX, sets up the machine, and hands in the code with three scripts: set up a machine, train and evaluate, export the weights.

We copy the hand-in to a fresh TPU and run it once. It passes when its number reaches the source's published result, two judges accept the run, and the exported weights load into the original PyTorch model.

Pipeline: the agent migrates the repository on a TPU machine; we rerun the output on a fresh TPU, check the number and run the judges; a failed run gets a second, aligned phase.
Pipeline. Top: the agent migrates a PyTorch repository to JAX on a bare TPU. Bottom: we rerun the output on a fresh TPU and check its number against the source's result and with two judges. If a check fails, a second phase gives the agent the main experiment's recipe, lets it change only the entry script, and grades again.

Two phases. If the first attempt fails, we tell the agent which experiment we grade, freeze its code except the entry script, and grade again. A correct migration that ran the wrong experiment gets a second chance; a broken one does not.

02

The repositories

XStack has 20 repositories from ten domains, chosen from more than 200 popular PyTorch repositories. Most candidates were rejected because their published number could not be checked. Each repository comes with a grading target: for 15 of them it is the number the README or paper reports, which we authenticated at the pinned commit; for 3 we set it from our own runs of the unmodified source; for minGPT, which publishes no number, we set a rule.

The repositories of XStack grouped by task family, with the expected duration of one full-scale run on a TPU host.
20
repositories
7
agents from 5 labs
140
agent × repository cells
22,802
TPU chip-hours of grading and generation
03

Results: one agent is far ahead, none is reliable

In the blind first phase the agents pass 30/140 cells (21%). After the aligned second phase they pass 44/140 (31%). Claude Opus 5 passes 15/20 repositories; GPT-5.6-sol passes 9/20 and every other agent at most 8/20. Every agent gains at least one pass from the second phase.

Pass rate per agent after phase 1 and after phase 2.
Pass rate per agent. Light bars: after phase 1. Dark bars: after phase 2, where the agent is told the experiment we grade.
agentphase 1after phase 2$ / cell$ / passhours / cell
1Claude Opus 5Anthropic12/2015/20 (75%)$92$1238.8
2GPT-5.6-solOpenAI6/209/20 (45%)$16$361.1
3DeepSeek V4.1 FlashDeepSeek6/208/20 (40%)$1.9$5.011.1
4GLM-5.3-FlashZ.ai3/205/20 (25%)$1.6$6.012.6
5Muse Spark 1.3Meta3/204/20 (20%)$14$714.0
6GPT-5.6-terraOpenAI0/202/20 (10%)$2.0$200.3
7GPT-5.6-lunaOpenAI0/201/20 (5%)$0.5$100.6
Outcomes by phase: all cells, their phase 1 outcome, their phase 2 outcome and the final result.
Outcomes by phase. From left to right: all 140 cells, their phase 1 outcome, their phase 2 outcome and the final result. Phase 2 ran on 108 cells and rescued 16; the rest fail the judges, fall short of the target, crash or break the freeze.
04

Pass rate grows with the log of cost

Across the 7 agents, pass rate rises by about 25 percentage points with every 10× in cost (r = 0.81). Three agents form the frontier: GPT-5.6-luna is the cheapest, DeepSeek V4.1 Flash is the cheapest per pass at $5, and Claude Opus 5 is the most accurate at $123 per pass. Every other agent lies below the frontier.

Pass rate against total cost per agent on a log scale, with a linear fit and the efficient frontier.
Accuracy against cost. Each dot is one agent's final pass rate against the cost of its 20 cells. The dashed line is a least-squares fit of pass rate on log cost; the blue line is the efficient frontier.
05

Why agents fail

A failure judge reads every failed run and sorts it into one of three groups. Wrong experiment: the migration ran a different pipeline or scale, or never ran the headline evaluation (30 cells after phase 1); this is the group phase 2 recovers. Fidelity gap: the right pipeline, but it misses the target (20). Crash or no termination (32). The last two are faults in the migrated code, 47% of the failures, and no second phase can repair them.

Failure breakdown after phase 1 per agent: pass, failed by the judges, wrong experiment, fidelity gap, crash or no termination.
Failure breakdown after phase 1. Each bar splits an agent's 20 repositories by outcome.

Agents fail differently. Reading the trajectories, the agent that passes most, Claude Opus 5, is the one that checks its own work: before handing in, it runs the source's experiment to the end and compares its number with the published one. Others hand in untested code. One agent estimated the full run at a few hours and submitted without running it; another handed in a score far below the paper's and called the migration fully validated. Diligent testing and a careful comparison against the original code are what a migration needs to succeed.

Citation

@article{cai2026xstack,
  title   = {XStack: A Benchmark for ML Repo Migration},
  author  = {Cai, Linrong and Lu, Taiming and Chen, Haozhe and Liu, Zhuang},
  journal = {arXiv preprint},
  year    = {2026}
}