LATIDIA · Investigación

InfraDev: ¿Pueden los agentes de codificación desarrollar la infraestructura de IA y cómo podemos saberlo?

arXiv: 2610.36589v1Tipo de anuncio: nuevo Resumen: Los agentes de codificación están comenzando a desarrollar la infraestructura de IA de nivel de producción que capacita y sirve a modelos grandes, pero los puntos de referencia existentes cubren solo núcleos individuales o cantan

La noticia

arXiv:2610.36589v1 Announce Type: new Abstract: Coding agents are beginning to develop the production-grade AI infrastructure that trains and serves large models, yet existing benchmarks cover only single kernels or single-GPU tasks, and grade with tests and prompts that misjudge both correct and incorrect patches. We introduce InfraDev, a benchmark of 36 tasks built from real-world pull requests to Megatron-LM, vLLM, and verl, covering infrastructure development for training, inference, and reinforcement learning. Each task asks an agent to reimplement a real change that spans many files and must run correctly across many GPUs. To grade such changes, we jointly synthesize hidden probes, which enforce the requirements the original tests leave unchecked,