---
tags:
- kairos
- language-model
- text-generation
- transformers
- codegen
- multilingual
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
---

# kairos-v1

**A high-performance multimodal foundation model by [CodeXBT](https://github.com/codextb)**

| Metric | Value |
|--------|-------|
| **Framework** | PyTorch 2.4+ |
| **Params** | 7.0B |
| **Context Length** | 128K tokens |
| **Languages** | English, Chinese, Japanese, Korean, French, German, Spanish, Hindi |
| **License** | Apache 2.0 |
| **Base Model** | kairos-base-v0.9 |

---

## Model Overview

**kairos-v1** is a state-of-the-art multimodal foundation model developed by **CodeXBT**, designed for advanced reasoning, code generation, and natural language understanding. Built on a dense decoder-only transformer architecture with significant optimizations in attention computation and training stability, kairos-v1 achieves competitive performance across a broad range of benchmarks while maintaining efficient inference through speculative decoding and KV-cache optimizations.

The model was trained on a carefully curated mix of web text, academic papers, programming repositories, scientific articles, and multimodal datasets totaling approximately **3.2 trillion tokens**. This diverse training corpus enables strong zero-shot and few-shot generalization across tasks spanning mathematics, logical reasoning, coding, and multilingual comprehension.

### Key Highlights

- **7 billion parameters** with dense transformer architecture
- **128K context window** enabled by RoPE and grouped-query attention (GQA)
- **3.2T training tokens** from a diverse, high-quality corpus
- **Top-tier benchmark scores** on MMLU, HumanEval, GSM8K, and SQuAD
- **Multilingual support** for 8+ languages with balanced representation
- **Production-ready** with vLLM, Hugging Face Transformers, and Ollama integration

---

## Model Architecture

| Component | Specification |
|-----------|--------------|
| **Architecture Type** | Decoder-only Transformer (causal LM) |
| **Number of Layers** | 48 |
| **Hidden Dimension** | 4,096 |
| **Number of Attention Heads** | 32 (GQA: 8 KV heads) |
| **Intermediate Dimension (FFN)** | 14,336 |
| **Vocabulary Size** | 128,256 (SentencePiece) |
| **Sequence Length** | 128,000 |
| **Position Embedding** | RoPE (rotary, base 10000, scaled 2x) |
| **Activation** | SwiGLU |
| **Attention** | Grouped-Query Attention (GQA) |
| **Normalization** | RMSNorm (pre-norm, eps=1e-6) |
| **Embedding** | Learnable, shared weights |
| **Initialization** | Kaiming uniform, layer norm bias=0 |
| **Quantization Support** | INT4, INT8, FP16, BF16 |

### Architecture Diagram

```
Input Embedding ─→ [Layer 1] ─→ [Layer 2] ─→ ... ─→ [Layer 48] ─→ LM Head ─→ Output
                    │              │                 │
              RMSNorm         RMSNorm           RMSNorm
              RoPE-Attn       RoPE-Attn         RoPE-Attn
              SwiGLU-FFN      SwiGLU-FFN        SwiGLU-FFN
```

---

## Training Details

### Dataset Composition

The model was trained on approximately **3.2 trillion tokens** from a curated dataset with the following distribution:

| Source Category | Tokens (B) | Percentage |
|-----------------|------------|------------|
| Web Text (filtered) | 1,280 | 40.0% |
| Academic Papers | 512 | 16.0% |
| Code (GitHub, StackOverflow) | 640 | 20.0% |
| Scientific & Math | 384 | 12.0% |
| Books & Literature | 256 | 8.0% |
| Multilingual (8 langs) | 128 | 4.0% |
| Instruction Tuning Pairs | 16 | 0.5% |
| Reinforcement Learning Data | 4 | 0.125% |
| Others | 128 | 4.0% |

### Preprocessing Pipeline

1. **Deduplication**: MinHash LSH deduplication at document and paragraph levels (Jaccard threshold 0.8)
2. **Quality Filtering**: Perplexity-based filtering, heuristic rules (toxicity, PII detection), language identification
3. **Tokenization**: SentencePiece unigram tokenizer trained on the final corpus (vocab=128,256)
4. **Sequence Packing**: Documents packed to 128K with attention masking

### Training Infrastructure

- **Hardware**: 3,072x NVIDIA H100 80GB (SXM5)
- **Framework**: Custom training framework on top of PyTorch 2.4 + Megatron-Core
- **Parallelism**: 3D parallelism (Tensor parallel=8, Pipeline parallel=4, Data parallel)
- **Optimizer**: AdamW (beta1=0.9, beta2=0.95, eps=1e-8), weight decay=0.1
- **Learning Rate**: Cosine schedule with 2% warmup, peak lr=2.5e-4
- **Batch Size**: Global batch size of 4M tokens (micro-batch=256, accumulation=4)
- **Training Steps**: ~819K steps (~12 days)
- **Checkpoint Frequency**: Every 500 steps
- **Mixed Precision**: BF16 with FP32 master weights

---

## Installation & Usage

### Prerequisites

- Python 3.10+
- PyTorch 2.4+
- CUDA 12.1+ (for GPU inference)

### Install via pip

```bash
pip install torch>=2.4 transformers>=4.45 accelerate>=0.34 sentencepiece>=0.2
```

### Quick Start — Load & Inference

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "codextb/kairos-v1"

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True,
)

prompt = "Explain the concept of backpropagation in neural networks."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    temperature=0.7,
    top_p=0.9,
    do_sample=True,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

### Inference with vLLM (High Throughput)

```python
from vllm import LLM, SamplingParams

llm = LLM(model="codextb/kairos-v1", tensor_parallel_size=4)
sampling_params = SamplingParams(temperature=0.7, top_p=0.9, max_tokens=512)

prompts = [
    "Write a Python function to merge two sorted lists.",
    "Translate 'Hello, how are you?' to French.",
]

outputs = llm.generate(prompts, sampling_params)
for out in outputs:
    print(out.outputs[0].text)
```

### Inference with Ollama (Local Deployment)

```bash
ollama pull codextb/kairos-v1
ollama run codextb/kairos-v1 "What is the time complexity of quicksort?"
```

### Quantized Inference (INT4)

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "codextb/kairos-v1",
    torch_dtype="auto",
    device_map="auto",
    load_in_4bit=True,        # bitsandbytes INT4
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype="float16",
    trust_remote_code=True,
)
```

---

## Evaluation Results

### English Benchmarks

| Benchmark | kairos-v1 | Competitor A | Competitor B |
|-----------|-----------|--------------|--------------|
| **MMLU (5-shot)** | 84.7 | 83.2 | 82.9 |
| **MMLU-Pro (5-shot)** | 61.3 | 58.7 | 59.4 |
| **HumanEval (0-shot)** | 89.6 | 87.8 | 85.4 |
| **MBPP (3-shot)** | 86.2 | 83.5 | 81.9 |
| **GSM8K (8-shot)** | 91.4 | 89.7 | 88.3 |
| **MATH (4-shot)** | 72.8 | 69.1 | 67.5 |
| **SQuAD 2.0 (0-shot)** | 88.9 | 87.2 | 86.8 |
| **TriviaQA (5-shot)** | 87.3 | 85.6 | 84.1 |
| **HellaSwag (10-shot)** | 93.1 | 92.4 | 91.8 |
| **ARC-C (25-shot)** | 89.6 | 87.9 | 86.3 |
| **Big-Bench Hard (3-shot)** | 82.4 | 80.1 | 78.9 |

### Multilingual Benchmarks (MMLU, 0-shot)

| Language | kairos-v1 |
|----------|-----------|
| English | 84.7 |
| Chinese | 82.1 |
| Japanese | 79.8 |
| Korean | 77.3 |
| French | 83.5 |
| German | 81.9 |
| Spanish | 83.2 |
| Hindi | 71.6 |

### Coding Benchmarks

| Benchmark | Task | Pass@1 |
|-----------|------|--------|
| **HumanEval** | Code generation | 89.6% |
| **MBPP** | Code generation | 86.2% |
| **LiveCodeBench** (Apr 2025) | Competitive programming | 67.3 |
| **SWE-bench Verified** | Software engineering | 31.4% |

### Efficiency Metrics

| Setting | GPU | Tokens/sec | Memory (GB) |
|---------|-----|------------|-------------|
| BF16 Inference | 1x H100 | 142 | 14.5 |
| BF16 Inference | 4x H100 | 486 | 5.8 |
| INT4 Quantized | 1x A100 | 98 | 4.2 |
| vLLM (4 GPUs) | 4x H100 | 1,820 | 11.2 |

---

## License

This model is released under the **Apache License 2.0**.

You are free to:
- **Share** — copy and redistribute the material in any medium or format
- **Adapt** — remix, transform, and build upon the material for any purpose, even commercially

See the [LICENSE](./LICENSE) file for details.

### Acceptable Use Policy

By downloading and using kairos-v1, you agree not to:
- Generate illegal content, hate speech, or material that violates applicable laws
- Use the model for automated surveillance or mass profiling
- Misrepresent model outputs as human-generated in sensitive contexts (healthcare, legal, judicial)
- Reverse-engineer the model for the purpose of training a competing model at scale

---

## Citation

If you use kairos-v1 in your research or projects, please cite:

```bibtex
@misc{kairos_v1_2025,
  author = {CodeXBT Research Team},
  title = {kairos-v1: A High-Performance Multimodal Foundation Model},
  year = {2025},
  publisher = {GitHub},
  journal = {GitHub repository},
  url = {https://github.com/codextb/kairos-v1},
  howpublished = {\url{https://github.com/codextb/kairos-v1}}
}
```

---

## Acknowledgments

We thank the following contributors and resources:

- **CodeXBT Research Team** — Model design, training, and evaluation
- **Open-source community** — Leveraging transformers, vLLM, and accelerate libraries
- **Hardware partners** — NVIDIA H100 compute allocation for training
- **Dataset curators** — Researchers who compiled and released the public corpora used in training
- **Peer reviewers** — External reviewers who provided feedback on benchmarking methodology

---

## Model Card Details

| Field | Value |
|-------|-------|
| **Model Developed By** | CodeXBT |
| **Model Type** | Causal Language Model |
| **Release Date** | July 2025 |
| **License** | Apache 2.0 |
| **Repository** | [github.com/codextb/kairos-v1](https://github.com/codextb/kairos-v1) |
| **Hugging Face** | [codextb/kairos-v1](https://huggingface.co/codextb/kairos-v1) |