README.md

---
language: en
license: apache-2.0
library_name: transformers
tags:
  - kairos
  - diffusion
  - multimodal
  - moe
  - trust_remote_code
pipeline_tag: text-generation
datasets:
  - ffurfaro/keep-it-simple
  - ffurfaro/keep-it-simple-multimodal
---

<h1 align="center"><p>🌀 kairos</p></h1>

<p align="center">
<a href="https://github.com/fabienfrfr/Kairos">
<img alt="GitHub" src="https://img.shields.io/badge/github-fabienfrfr%2FKairos-black?logo=github">
</a>
<a href="https://huggingface.co/ffurfaro">
<img alt="Hugging Face" src="https://img.shields.io/badge/HuggingFace-model-yellow?logo=huggingface">
</a>
</p>

<h3 align="center"><p>KairosFM — less parameters, more signal.</p></h3>

KairosFM is a hybrid MoE diffusion language model combining **DeltaNet** (linear attention),
**Sliding Window Attention**, and **Attention Residuals (AttnRes)**, trained on text, image,
video, audio, lidar, and control (state/action) modalities through a shared multimodal
conv-byte tokenizer. See [github.com/fabienfrfr/Kairos](https://github.com/fabienfrfr/Kairos)
for the full architecture writeup.

## This checkpoint

| | |
|---|---|
| Total params | ?-dim, ? layers |
| Experts | 7 routed / 1 shared, top-1 |
| Vocab size | 291 |
| Best training loss | `7.30881994911411` |
| Steps trained | `4533` |

Note: this repo currently tracks best-training-loss only (`checkpoints/best.pt`) — no held-out
validation split is evaluated during training yet.

## Files

- `checkpoints/` — `best.pt` (lowest avg training loss) + periodic `step_*.pt`
- `tensorboard/` — `events.out.tfevents.*`, viewable in the Hub's **Training Metrics** tab
- `config.json`, `model.safetensors` — native HF format, loadable via `trust_remote_code`

## Usage

```python
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("ffurfaro/kairos", trust_remote_code=True)
```

Requires the `kairos` package importable (custom architecture, not upstream `transformers`) —
install from [github.com/fabienfrfr/Kairos](https://github.com/fabienfrfr/Kairos) first, or add
it to `PYTHONPATH`. Alternatively, skip `Auto*` and import the class directly:

```python
from kairos.modeling import KairosDiffusionLLM

model = KairosDiffusionLLM.from_pretrained("ffurfaro/kairos")
```

## Limitations

Experimental, low-compute-budget training run — expect uneven quality across modalities
(multimodal data is a small fraction of total training). Not evaluated for safety-critical use.

## Citation

```bibtex
@misc{kairos,
  title  = {KairosFM: less parameters, more signal — a multimodal MoE diffusion model for edge AI},
  author = {Fabien Furfaro},
  url    = {https://github.com/fabienfrfr/Kairos}
}
```