LATIDIA · Investigación

WAKMan: Administrador de KVCache con reconocimiento de carga de trabajo

arXiv:2610.36355v1 Anuncio Tipo: nuevo Resumen: El almacenamiento en caché de prefijos permite a los servidores de modelos de lenguaje grandes (LLM) reutilizar los estados de clave-valor (KV) y acelerar la etapa de cálculo de prellenado para las solicitudes que comparten un prefijo con

La noticia

arXiv:2610.36355v1 Announce Type: new Abstract: Prefix caching enables a Large Language Model (LLM) servers to reuse the Key-Value (KV) states and accelerate prefill computation stage for requests that share a prefix with earlier prompts, thus reducing latency and serving cost. With limited memory resource, the KV cache eviction policy becomes a decisive factor in determining the overall benefits of prefix caching. Production traffic may have single-turn, multi-turn, agentic, or a mixture of any combination of these request types. Our findings indicate that agentic steps return within seconds and favor recency, while popular prompts and long conversations return after many other requests and favor frequency. Across 20 mixes, no single policy