kairos-v1

A high-performance multimodal foundation model by CodeXBT

Metric	Value
Framework	PyTorch 2.4+
Params	7.0B
Context Length	128K tokens
Languages	English, Chinese, Japanese, Korean, French, German, Spanish, Hindi
License	Apache 2.0
Base Model	kairos-base-v0.9

---

Model Overview

kairos-v1 is a state-of-the-art multimodal foundation model developed by CodeXBT, designed for advanced reasoning, code generation, and natural language understanding. Built on a dense decoder-only transformer architecture with significant optimizations in attention computation and training stability, kairos-v1 achieves competitive performance across a broad range of benchmarks while maintaining efficient inference through speculative decoding and KV-cache optimizations.

The model was trained on a carefully curated mix of web text, academic papers, programming repositories, scientific articles, and multimodal datasets totaling approximately 3.2 trillion tokens. This diverse training corpus enables strong zero-shot and few-shot generalization across tasks spanning mathematics, logical reasoning, coding, and multilingual comprehension.

Key Highlights

• 7 billion parameters with dense transformer architecture
• 128K context window enabled by RoPE and grouped-query attention (GQA)
• 3.2T training tokens from a diverse, high-quality corpus
• Top-tier benchmark scores on MMLU, HumanEval, GSM8K, and SQuAD
• Multilingual support for 8+ languages with balanced representation
• Production-ready with vLLM, Hugging Face Transformers, and Ollama integration

---

Model Architecture

Component	Specification
Architecture Type	Decoder-only Transformer (causal LM)
Number of Layers	48
Hidden Dimension	4,096
Number of Attention Heads	32 (GQA: 8 KV heads)
Intermediate Dimension (FFN)	14,336
Vocabulary Size	128,256 (SentencePiece)
Sequence Length	128,000
Position Embedding	RoPE (rotary, base 10000, scaled 2x)
Activation	SwiGLU
Attention	Grouped-Query Attention (GQA)
Normalization	RMSNorm (pre-norm, eps=1e-6)
Embedding	Learnable, shared weights
Initialization	Kaiming uniform, layer norm bias=0
Quantization Support	INT4, INT8, FP16, BF16

Architecture Diagram

Input Embedding ─→ [Layer 1] ─→ [Layer 2] ─→ ... ─→ [Layer 48] ─→ LM Head ─→ Output
                    │              │                 │
              RMSNorm         RMSNorm           RMSNorm
              RoPE-Attn       RoPE-Attn         RoPE-Attn
              SwiGLU-FFN      SwiGLU-FFN        SwiGLU-FFN

---

Training Details

Dataset Composition

The model was trained on approximately 3.2 trillion tokens from a curated dataset with the following distribution:

Source Category	Tokens (B)	Percentage
Web Text (filtered)	1,280	40.0%
Academic Papers	512	16.0%
Code (GitHub, StackOverflow)	640	20.0%
Scientific & Math	384	12.0%
Books & Literature	256	8.0%
Multilingual (8 langs)	128	4.0%
Instruction Tuning Pairs	16	0.5%
Reinforcement Learning Data	4	0.125%
Others	128	4.0%

Preprocessing Pipeline

1. Deduplication: MinHash LSH deduplication at document and paragraph levels (Jaccard threshold 0.8)
2. Quality Filtering: Perplexity-based filtering, heuristic rules (toxicity, PII detection), language identification
3. Tokenization: SentencePiece unigram tokenizer trained on the final corpus (vocab=128,256)
4. Sequence Packing: Documents packed to 128K with attention masking

Training Infrastructure

• Hardware: 3,072x NVIDIA H100 80GB (SXM5)
• Framework: Custom training framework on top of PyTorch 2.4 + Megatron-Core
• Parallelism: 3D parallelism (Tensor parallel=8, Pipeline parallel=4, Data parallel)
• Optimizer: AdamW (beta1=0.9, beta2=0.95, eps=1e-8), weight decay=0.1
• Learning Rate: Cosine schedule with 2% warmup, peak lr=2.5e-4
• Batch Size: Global batch size of 4M tokens (micro-batch=256, accumulation=4)
• Training Steps: 819K steps (12 days)
• Checkpoint Frequency: Every 500 steps
• Mixed Precision: BF16 with FP32 master weights

---

Installation & Usage

Prerequisites

• Python 3.10+
• PyTorch 2.4+
• CUDA 12.1+ (for GPU inference)

Install via pip

pip install torch>=2.4 transformers>=4.45 accelerate>=0.34 sentencepiece>=0.2

Quick Start — Load & Inference

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "codextb/kairos-v1"

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True,
)

prompt = "Explain the concept of backpropagation in neural networks."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    temperature=0.7,
    top_p=0.9,
    do_sample=True,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Inference with vLLM (High Throughput)

from vllm import LLM, SamplingParams

llm = LLM(model="codextb/kairos-v1", tensor_parallel_size=4)
sampling_params = SamplingParams(temperature=0.7, top_p=0.9, max_tokens=512)

prompts = [
    "Write a Python function to merge two sorted lists.",
    "Translate 'Hello, how are you?' to French.",
]

outputs = llm.generate(prompts, sampling_params)
for out in outputs:
    print(out.outputs[0].text)

Inference with Ollama (Local Deployment)

ollama pull codextb/kairos-v1
ollama run codextb/kairos-v1 "What is the time complexity of quicksort?"

Quantized Inference (INT4)

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "codextb/kairos-v1",
    torch_dtype="auto",
    device_map="auto",
    load_in_4bit=True,        # bitsandbytes INT4
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype="float16",
    trust_remote_code=True,
)

---

Evaluation Results

English Benchmarks

Benchmark	kairos-v1	Competitor A	Competitor B
MMLU (5-shot)	84.7	83.2	82.9
MMLU-Pro (5-shot)	61.3	58.7	59.4
HumanEval (0-shot)	89.6	87.8	85.4
MBPP (3-shot)	86.2	83.5	81.9
GSM8K (8-shot)	91.4	89.7	88.3
MATH (4-shot)	72.8	69.1	67.5
SQuAD 2.0 (0-shot)	88.9	87.2	86.8
TriviaQA (5-shot)	87.3	85.6	84.1
HellaSwag (10-shot)	93.1	92.4	91.8
ARC-C (25-shot)	89.6	87.9	86.3
Big-Bench Hard (3-shot)	82.4	80.1	78.9

Multilingual Benchmarks (MMLU, 0-shot)

Language	kairos-v1
English	84.7
Chinese	82.1
Japanese	79.8
Korean	77.3
French	83.5
German	81.9
Spanish	83.2
Hindi	71.6

Coding Benchmarks

Benchmark	Task	Pass@1
HumanEval	Code generation	89.6%
MBPP	Code generation	86.2%
LiveCodeBench (Apr 2025)	Competitive programming	67.3
SWE-bench Verified	Software engineering	31.4%

Efficiency Metrics

Setting	GPU	Tokens/sec	Memory (GB)
BF16 Inference	1x H100	142	14.5
BF16 Inference	4x H100	486	5.8
INT4 Quantized	1x A100	98	4.2
vLLM (4 GPUs)	4x H100	1,820	11.2

---

License

This model is released under the Apache License 2.0.

You are free to:

• Share — copy and redistribute the material in any medium or format
• Adapt — remix, transform, and build upon the material for any purpose, even commercially

See the LICENSE file for details.

Acceptable Use Policy

By downloading and using kairos-v1, you agree not to:

• Generate illegal content, hate speech, or material that violates applicable laws
• Use the model for automated surveillance or mass profiling
• Misrepresent model outputs as human-generated in sensitive contexts (healthcare, legal, judicial)
• Reverse-engineer the model for the purpose of training a competing model at scale

---

Citation

If you use kairos-v1 in your research or projects, please cite:

@misc{kairos_v1_2025,
  author = {CodeXBT Research Team},
  title = {kairos-v1: A High-Performance Multimodal Foundation Model},
  year = {2025},
  publisher = {GitHub},
  journal = {GitHub repository},
  url = {https://github.com/codextb/kairos-v1},
  howpublished = {\url{https://github.com/codextb/kairos-v1}}
}

---

Acknowledgments

We thank the following contributors and resources:

• CodeXBT Research Team — Model design, training, and evaluation
• Open-source community — Leveraging transformers, vLLM, and accelerate libraries
• Hardware partners — NVIDIA H100 compute allocation for training
• Dataset curators — Researchers who compiled and released the public corpora used in training
• Peer reviewers — External reviewers who provided feedback on benchmarking methodology

---

Model Card Details

Field	Value
Model Developed By	CodeXBT
Model Type	Causal Language Model
Release Date	July 2025
License	Apache 2.0
Repository	github.com/codextb/kairos-v1
Hugging Face	codextb/kairos-v1