| --- |
| tags: |
| - kairos |
| - language-model |
| - text-generation |
| - transformers |
| - codegen |
| - multilingual |
| license: apache-2.0 |
| library_name: transformers |
| pipeline_tag: text-generation |
| --- |
| |
| # kairos-v1 |
|
|
| **A high-performance multimodal foundation model by [CodeXBT](https://github.com/codextb)** |
|
|
| | Metric | Value | |
| |--------|-------| |
| | **Framework** | PyTorch 2.4+ | |
| | **Params** | 7.0B | |
| | **Context Length** | 128K tokens | |
| | **Languages** | English, Chinese, Japanese, Korean, French, German, Spanish, Hindi | |
| | **License** | Apache 2.0 | |
| | **Base Model** | kairos-base-v0.9 | |
|
|
| --- |
|
|
| ## Model Overview |
|
|
| **kairos-v1** is a state-of-the-art multimodal foundation model developed by **CodeXBT**, designed for advanced reasoning, code generation, and natural language understanding. Built on a dense decoder-only transformer architecture with significant optimizations in attention computation and training stability, kairos-v1 achieves competitive performance across a broad range of benchmarks while maintaining efficient inference through speculative decoding and KV-cache optimizations. |
|
|
| The model was trained on a carefully curated mix of web text, academic papers, programming repositories, scientific articles, and multimodal datasets totaling approximately **3.2 trillion tokens**. This diverse training corpus enables strong zero-shot and few-shot generalization across tasks spanning mathematics, logical reasoning, coding, and multilingual comprehension. |
|
|
| ### Key Highlights |
|
|
| - **7 billion parameters** with dense transformer architecture |
| - **128K context window** enabled by RoPE and grouped-query attention (GQA) |
| - **3.2T training tokens** from a diverse, high-quality corpus |
| - **Top-tier benchmark scores** on MMLU, HumanEval, GSM8K, and SQuAD |
| - **Multilingual support** for 8+ languages with balanced representation |
| - **Production-ready** with vLLM, Hugging Face Transformers, and Ollama integration |
|
|
| --- |
|
|
| ## Model Architecture |
|
|
| | Component | Specification | |
| |-----------|--------------| |
| | **Architecture Type** | Decoder-only Transformer (causal LM) | |
| | **Number of Layers** | 48 | |
| | **Hidden Dimension** | 4,096 | |
| | **Number of Attention Heads** | 32 (GQA: 8 KV heads) | |
| | **Intermediate Dimension (FFN)** | 14,336 | |
| | **Vocabulary Size** | 128,256 (SentencePiece) | |
| | **Sequence Length** | 128,000 | |
| | **Position Embedding** | RoPE (rotary, base 10000, scaled 2x) | |
| | **Activation** | SwiGLU | |
| | **Attention** | Grouped-Query Attention (GQA) | |
| | **Normalization** | RMSNorm (pre-norm, eps=1e-6) | |
| | **Embedding** | Learnable, shared weights | |
| | **Initialization** | Kaiming uniform, layer norm bias=0 | |
| | **Quantization Support** | INT4, INT8, FP16, BF16 | |
|
|
| ### Architecture Diagram |
|
|
| ``` |
| Input Embedding ─→ [Layer 1] ─→ [Layer 2] ─→ ... ─→ [Layer 48] ─→ LM Head ─→ Output |
| │ │ │ |
| RMSNorm RMSNorm RMSNorm |
| RoPE-Attn RoPE-Attn RoPE-Attn |
| SwiGLU-FFN SwiGLU-FFN SwiGLU-FFN |
| ``` |
|
|
| --- |
|
|
| ## Training Details |
|
|
| ### Dataset Composition |
|
|
| The model was trained on approximately **3.2 trillion tokens** from a curated dataset with the following distribution: |
|
|
| | Source Category | Tokens (B) | Percentage | |
| |-----------------|------------|------------| |
| | Web Text (filtered) | 1,280 | 40.0% | |
| | Academic Papers | 512 | 16.0% | |
| | Code (GitHub, StackOverflow) | 640 | 20.0% | |
| | Scientific & Math | 384 | 12.0% | |
| | Books & Literature | 256 | 8.0% | |
| | Multilingual (8 langs) | 128 | 4.0% | |
| | Instruction Tuning Pairs | 16 | 0.5% | |
| | Reinforcement Learning Data | 4 | 0.125% | |
| | Others | 128 | 4.0% | |
|
|
| ### Preprocessing Pipeline |
|
|
| 1. **Deduplication**: MinHash LSH deduplication at document and paragraph levels (Jaccard threshold 0.8) |
| 2. **Quality Filtering**: Perplexity-based filtering, heuristic rules (toxicity, PII detection), language identification |
| 3. **Tokenization**: SentencePiece unigram tokenizer trained on the final corpus (vocab=128,256) |
| 4. **Sequence Packing**: Documents packed to 128K with attention masking |
|
|
| ### Training Infrastructure |
|
|
| - **Hardware**: 3,072x NVIDIA H100 80GB (SXM5) |
| - **Framework**: Custom training framework on top of PyTorch 2.4 + Megatron-Core |
| - **Parallelism**: 3D parallelism (Tensor parallel=8, Pipeline parallel=4, Data parallel) |
| - **Optimizer**: AdamW (beta1=0.9, beta2=0.95, eps=1e-8), weight decay=0.1 |
| - **Learning Rate**: Cosine schedule with 2% warmup, peak lr=2.5e-4 |
| - **Batch Size**: Global batch size of 4M tokens (micro-batch=256, accumulation=4) |
| - **Training Steps**: ~819K steps (~12 days) |
| - **Checkpoint Frequency**: Every 500 steps |
| - **Mixed Precision**: BF16 with FP32 master weights |
|
|
| --- |
|
|
| ## Installation & Usage |
|
|
| ### Prerequisites |
|
|
| - Python 3.10+ |
| - PyTorch 2.4+ |
| - CUDA 12.1+ (for GPU inference) |
|
|
| ### Install via pip |
|
|
| ```bash |
| pip install torch>=2.4 transformers>=4.45 accelerate>=0.34 sentencepiece>=0.2 |
| ``` |
|
|
| ### Quick Start — Load & Inference |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model_name = "codextb/kairos-v1" |
| |
| tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True) |
| model = AutoModelForCausalLM.from_pretrained( |
| model_name, |
| torch_dtype="auto", |
| device_map="auto", |
| trust_remote_code=True, |
| ) |
| |
| prompt = "Explain the concept of backpropagation in neural networks." |
| inputs = tokenizer(prompt, return_tensors="pt").to(model.device) |
| |
| outputs = model.generate( |
| **inputs, |
| max_new_tokens=512, |
| temperature=0.7, |
| top_p=0.9, |
| do_sample=True, |
| ) |
| |
| print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
| ``` |
|
|
| ### Inference with vLLM (High Throughput) |
|
|
| ```python |
| from vllm import LLM, SamplingParams |
| |
| llm = LLM(model="codextb/kairos-v1", tensor_parallel_size=4) |
| sampling_params = SamplingParams(temperature=0.7, top_p=0.9, max_tokens=512) |
| |
| prompts = [ |
| "Write a Python function to merge two sorted lists.", |
| "Translate 'Hello, how are you?' to French.", |
| ] |
| |
| outputs = llm.generate(prompts, sampling_params) |
| for out in outputs: |
| print(out.outputs[0].text) |
| ``` |
|
|
| ### Inference with Ollama (Local Deployment) |
|
|
| ```bash |
| ollama pull codextb/kairos-v1 |
| ollama run codextb/kairos-v1 "What is the time complexity of quicksort?" |
| ``` |
|
|
| ### Quantized Inference (INT4) |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model = AutoModelForCausalLM.from_pretrained( |
| "codextb/kairos-v1", |
| torch_dtype="auto", |
| device_map="auto", |
| load_in_4bit=True, # bitsandbytes INT4 |
| bnb_4bit_quant_type="nf4", |
| bnb_4bit_compute_dtype="float16", |
| trust_remote_code=True, |
| ) |
| ``` |
|
|
| --- |
|
|
| ## Evaluation Results |
|
|
| ### English Benchmarks |
|
|
| | Benchmark | kairos-v1 | Competitor A | Competitor B | |
| |-----------|-----------|--------------|--------------| |
| | **MMLU (5-shot)** | 84.7 | 83.2 | 82.9 | |
| | **MMLU-Pro (5-shot)** | 61.3 | 58.7 | 59.4 | |
| | **HumanEval (0-shot)** | 89.6 | 87.8 | 85.4 | |
| | **MBPP (3-shot)** | 86.2 | 83.5 | 81.9 | |
| | **GSM8K (8-shot)** | 91.4 | 89.7 | 88.3 | |
| | **MATH (4-shot)** | 72.8 | 69.1 | 67.5 | |
| | **SQuAD 2.0 (0-shot)** | 88.9 | 87.2 | 86.8 | |
| | **TriviaQA (5-shot)** | 87.3 | 85.6 | 84.1 | |
| | **HellaSwag (10-shot)** | 93.1 | 92.4 | 91.8 | |
| | **ARC-C (25-shot)** | 89.6 | 87.9 | 86.3 | |
| | **Big-Bench Hard (3-shot)** | 82.4 | 80.1 | 78.9 | |
|
|
| ### Multilingual Benchmarks (MMLU, 0-shot) |
|
|
| | Language | kairos-v1 | |
| |----------|-----------| |
| | English | 84.7 | |
| | Chinese | 82.1 | |
| | Japanese | 79.8 | |
| | Korean | 77.3 | |
| | French | 83.5 | |
| | German | 81.9 | |
| | Spanish | 83.2 | |
| | Hindi | 71.6 | |
|
|
| ### Coding Benchmarks |
|
|
| | Benchmark | Task | Pass@1 | |
| |-----------|------|--------| |
| | **HumanEval** | Code generation | 89.6% | |
| | **MBPP** | Code generation | 86.2% | |
| | **LiveCodeBench** (Apr 2025) | Competitive programming | 67.3 | |
| | **SWE-bench Verified** | Software engineering | 31.4% | |
|
|
| ### Efficiency Metrics |
|
|
| | Setting | GPU | Tokens/sec | Memory (GB) | |
| |---------|-----|------------|-------------| |
| | BF16 Inference | 1x H100 | 142 | 14.5 | |
| | BF16 Inference | 4x H100 | 486 | 5.8 | |
| | INT4 Quantized | 1x A100 | 98 | 4.2 | |
| | vLLM (4 GPUs) | 4x H100 | 1,820 | 11.2 | |
|
|
| --- |
|
|
| ## License |
|
|
| This model is released under the **Apache License 2.0**. |
|
|
| You are free to: |
| - **Share** — copy and redistribute the material in any medium or format |
| - **Adapt** — remix, transform, and build upon the material for any purpose, even commercially |
|
|
| See the [LICENSE](./LICENSE) file for details. |
|
|
| ### Acceptable Use Policy |
|
|
| By downloading and using kairos-v1, you agree not to: |
| - Generate illegal content, hate speech, or material that violates applicable laws |
| - Use the model for automated surveillance or mass profiling |
| - Misrepresent model outputs as human-generated in sensitive contexts (healthcare, legal, judicial) |
| - Reverse-engineer the model for the purpose of training a competing model at scale |
|
|
| --- |
|
|
| ## Citation |
|
|
| If you use kairos-v1 in your research or projects, please cite: |
|
|
| ```bibtex |
| @misc{kairos_v1_2025, |
| author = {CodeXBT Research Team}, |
| title = {kairos-v1: A High-Performance Multimodal Foundation Model}, |
| year = {2025}, |
| publisher = {GitHub}, |
| journal = {GitHub repository}, |
| url = {https://github.com/codextb/kairos-v1}, |
| howpublished = {\url{https://github.com/codextb/kairos-v1}} |
| } |
| ``` |
|
|
| --- |
|
|
| ## Acknowledgments |
|
|
| We thank the following contributors and resources: |
|
|
| - **CodeXBT Research Team** — Model design, training, and evaluation |
| - **Open-source community** — Leveraging transformers, vLLM, and accelerate libraries |
| - **Hardware partners** — NVIDIA H100 compute allocation for training |
| - **Dataset curators** — Researchers who compiled and released the public corpora used in training |
| - **Peer reviewers** — External reviewers who provided feedback on benchmarking methodology |
|
|
| --- |
|
|
| ## Model Card Details |
|
|
| | Field | Value | |
| |-------|-------| |
| | **Model Developed By** | CodeXBT | |
| | **Model Type** | Causal Language Model | |
| | **Release Date** | July 2025 | |
| | **License** | Apache 2.0 | |
| | **Repository** | [github.com/codextb/kairos-v1](https://github.com/codextb/kairos-v1) | |
| | **Hugging Face** | [codextb/kairos-v1](https://huggingface.co/codextb/kairos-v1) | |
|
|
