LATIDIA · Investigación

Kairos: Hacia un modelo de base a nivel de byte para cada modalidad

arXiv:2610.12495v1 Tipo de anuncio: Cross Resumen: Los modelos de lenguaje pequeños entrenados en una sola máquina local dejan poco espacio para el ensayo y error arquitectónico. Kairos combina la atención lineal bidireccional cerrada-DeltaNet

La noticia

arXiv:2610.12495v1 Announce Type: cross Abstract: Small language models trained on a single local machine leave little room for architectural trial and error. Kairos combines bidirectional gated-DeltaNet linear attention and sliding-window attention over shared projections (LiZAttention2), block-wise Attention-Residual aggregation, a sparse Mixture-of-Experts (MoE) feed-forward layer and a multi-scale byte-level codec, trained with a uniform-noise denoising objective over one byte space shared by text, image, audio, video, lidar and control streams; this report asks whether the combination optimizes at all before asking whether it is useful. We give code-derived algorithms for each component, including a cross-session recurrent-state memory gate, then run convergence sanity checks: overfittinga single 440-byte sentence at 1--15M parameters, text