Coding
Sponsored byPick a corner of engineering where correctness is hard to fake: GPU kernels, chip design, a codebase in an unfamiliar stack. Check that the result works and is actually better, and that the agent did not change or skip the tests to pass. Against a realistic baseline, most of what models write here is still slower, wrong or unfinished.
Benchmarks to know
- KernelBench-Verified Stanford and Meta
Against a realistic PyTorch baseline, no model's custom GPU kernels are reliably faster, and many quietly use more memory.
- Vibe Code Bench Vals AI
Build a whole web app from a plain-English spec, tested by clicking through it; no model passes every test.
- CyberBench Vals AI
Reproduce and patch real memory-safety bugs in open-source code; the best agent still fails about one in five.
- FrontierSWE Proximal
34 very hard projects with up to 20 hours of agent time each, like writing a Verilog simulator in Swift.
- SlopCodeBench UW-Madison
Agents extending their own code over many checkpoints pile up bloat; HumanLayer used it to measure slop.
- GameCraft-Bench CUHK-Shenzhen and Tencent
Agents build Godot games graded by replaying them; they run, but content depth trails even for the best.
- Build-A-Chip-Bench ChipAgents and NVIDIA
From a workload spec to a finished chip layout; most designs fail verification, and passing ones miss their targets.
- GBA Eval Mechanize
Write a Game Boy Advance emulator from scratch in 24 hours; the best renders games well but fails timers and audio.
- Terminal-Bench 4.1 Stanford, Harbor and Laude Institute
Expert jobs in a real terminal, from science to security; the best agents resolve about two thirds.
Bring an eval in this track, or in any other, on Saturday, December 5, 2026 at Cornell University.