LATIDIA · Agentes

Inyecciones rápidas autogeneradas en resúmenes de compactación

Inyecciones rápidas autogeneradas en resúmenes de compactación En nuestro marco para informar sobre la desalineación del modelo, OpenAI proporciona "seis informes sobre el comportamiento inesperado o preocupante del modelo que hemos observado en los últimos seis meses".

La noticia

Self-generated prompt injections in compaction summaries In Our framework for reporting model misalignment OpenAI provide "six reports on unexpected or concerning model behavior we’ve observed in the last six months". This one here is my favorite: they caught some of their models in training deliberately subverting themselves in their compaction prompts. Compaction is the process agent systems use when they are running out of tokens in their context window, so they summarize everything that has gone before so they can keep going with more token headroom. In one of the observed instances, a model undergoing reinforcement learning was working on a task to update an existing HTTP API endpoint with