LATIDIA · Investigación

Los juicios de rúbrica carecen de incertidumbre: ponderación conforme para la evaluación de LLM

arXiv: 2610.21214v1Tipo de anuncio: nuevo Resumen: La evaluación basada en rúbricas se utiliza ampliamente para evaluar los resultados del modelo de lenguaje grande (LLM) al descomponer la calidad en múltiples criterios y asignar a cada uno un juicio binario de

La noticia

arXiv:2610.21214v1 Announce Type: new Abstract: Rubric-based evaluation is widely used to assess large language model (LLM) outputs by decomposing quality into multiple criteria and assigning each a binary judgment from an LLM verifier. However, judgment certainty varies substantially across criteria. Existing methods uniformly aggregate these judgments into a final score, implicitly treating every criterion as equally certain and overlooking this heterogeneity. As a result, uncertain judgments remain incorporated into reward estimation without distinction, potentially yielding misleading training signals for downstream reinforcement learning (RL). To address this limitation, we propose Uncertainty-aware Rubric Weighting (URW), a training-free framework that quantifies criterion-level judgment uncertainty via conformal calibration. Specifically, URW elicits a soft uncertainty