What they found
The authors tested Qwen2.5-Coder-7B-Instruct on 30 KernelBench tasks, run on a V100 GPU. The model had to replace PyTorch operators with custom CUDA kernels (code that runs on the GPU). They compared four setups: zero-shot prompting, profiling-guided prompting, iterative refinement on execution feedback, and a multi-agent loop. Profiling-guided prompting did best overall, with 43% correct and 27% faster than baseline. Even correct kernels ran at only 0.97-0.99x the speed of eager PyTorch, and the real wins were fused memory-bound epilogues. The agent loop was the only method to solve any level-2 fusion problem (3/10 vs. 0/10 for the others). The first agent scored 3% correct, below zero-shot's 30%. Best-of-n sampling under a real execution check raised correctness from 3% to 33% going from n=1 to n=8.
Why it matters
Many teams want small, local SI models to help write fast GPU code. This study suggests where such a model helps and where it does not. It also warns that adding agent steps can make results worse if the plan is too ambitious for the model. The test is small, with 30 tasks, one model and one GPU, so the findings may not carry over widely.
How it changes the race
This is a modest, practical result. It suggests that cheap sampling checked by real execution can help a small model more than extra reasoning steps, which may lower the cost of kernel work for smaller groups. It does not show small models beating tuned vendor libraries.
The abstract, by the authors
We study how much the interaction strategy around a fixed, locally hosted 7B-parameter code model (Qwen2.5-Coder-7B-Instruct) affects its ability to replace PyTorch operators with custom CUDA kernels. On 30 KernelBench tasks (levels 1-3, first 10 each, evaluated on a V100), we compare four regimes of increasing structure: zero-shot prompting, profiling-guided prompting, iterative refinement on execution feedback, and a multi-agent loop (analysis, retrieval, generation, evaluation, feedback triage) in which the same model fills four roles around a non-LLM evaluator. Three findings stand out. (1) Correctness, not speed, is the binding constraint: even correct generated kernels average 0.97-0.99x the speed of eager PyTorch, because a 7B model cannot out-write cuBLAS/cuDNN on dense GEMM and convolution; the only real wins are fused memory-bound epilogues. (2) The best strategy depends on difficulty: profiling-guided prompting is strongest overall (43% correct, 27% faster than baseline), but the agent loop is the only method that solves any level-2 fusion problem (3/10 vs. 0/10 for all others). (3) Naive agentic scaffolding is actively harmful: our first agent scored 3% correct, below zero-shot's 30%, and the decisive repairs were matching the agent's ambition to the model's competence and spending compute on best-of-n sampling under a real execution oracle (3% to 33% correct from n=1 to n=8), not on more reasoning steps.
Quoted from arXiv:2611.12079. The research and its words belong to its authors.