Hugging Face's logo

CodeXBT
/
kairos-v1

Text Generation
Transformers
Safetensors
kairos
language-model
codegen
multilingual
conversational
Model card Files
xet
 Community
kairos-v1 / README.md
CodeXBT's picture
CodeXBT
Upload README.md with huggingface_hub
fad3088 verified
preview code
|
Raw
History Blame Contribute Delete
10.1 kB
metadata 
tags:
  - kairos
  - language-model
  - text-generation
  - transformers
  - codegen
  - multilingual
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation

kairos-v1

A high-performance multimodal foundation model by CodeXBT

Metric Value
Framework PyTorch 2.4+
Params 7.0B
Context Length 128K tokens
Languages English, Chinese, Japanese, Korean, French, German, Spanish, Hindi
License Apache 2.0
Base Model kairos-base-v0.9

Model Overview

kairos-v1 is a state-of-the-art multimodal foundation model developed by CodeXBT, designed for advanced reasoning, code generation, and natural language understanding. Built on a dense decoder-only transformer architecture with significant optimizations in attention computation and training stability, kairos-v1 achieves competitive performance across a broad range of benchmarks while maintaining efficient inference through speculative decoding and KV-cache optimizations.

The model was trained on a carefully curated mix of web text, academic papers, programming repositories, scientific articles, and multimodal datasets totaling approximately 3.2 trillion tokens. This diverse training corpus enables strong zero-shot and few-shot generalization across tasks spanning mathematics, logical reasoning, coding, and multilingual comprehension.

Key Highlights

  • 7 billion parameters with dense transformer architecture
  • 128K context window enabled by RoPE and grouped-query attention (GQA)
  • 3.2T training tokens from a diverse, high-quality corpus
  • Top-tier benchmark scores on MMLU, HumanEval, GSM8K, and SQuAD
  • Multilingual support for 8+ languages with balanced representation
  • Production-ready with vLLM, Hugging Face Transformers, and Ollama integration

Model Architecture

Component Specification
Architecture Type Decoder-only Transformer (causal LM)
Number of Layers 48
Hidden Dimension 4,096
Number of Attention Heads 32 (GQA: 8 KV heads)
Intermediate Dimension (FFN) 14,336
Vocabulary Size 128,256 (SentencePiece)
Sequence Length 128,000
Position Embedding RoPE (rotary, base 10000, scaled 2x)
Activation SwiGLU
Attention Grouped-Query Attention (GQA)
Normalization RMSNorm (pre-norm, eps=1e-6)
Embedding Learnable, shared weights
Initialization Kaiming uniform, layer norm bias=0
Quantization Support INT4, INT8, FP16, BF16

Architecture Diagram 

Input Embedding ─→ [Layer 1] ─→ [Layer 2] ─→ ... ─→ [Layer 48] ─→ LM Head ─→ Output
                    │              │                 │
              RMSNorm         RMSNorm           RMSNorm
              RoPE-Attn       RoPE-Attn         RoPE-Attn
              SwiGLU-FFN      SwiGLU-FFN        SwiGLU-FFN

Training Details

Dataset Composition

The model was trained on approximately 3.2 trillion tokens from a curated dataset with the following distribution:

Source Category Tokens (B) Percentage
Web Text (filtered) 1,280 40.0%
Academic Papers 512 16.0%
Code (GitHub, StackOverflow) 640 20.0%
Scientific & Math 384 12.0%
Books & Literature 256 8.0%
Multilingual (8 langs) 128 4.0%
Instruction Tuning Pairs 16 0.5%
Reinforcement Learning Data 4 0.125%
Others 128 4.0%

Preprocessing Pipeline

  1. Deduplication: MinHash LSH deduplication at document and paragraph levels (Jaccard threshold 0.8)
  2. Quality Filtering: Perplexity-based filtering, heuristic rules (toxicity, PII detection), language identification
  3. Tokenization: SentencePiece unigram tokenizer trained on the final corpus (vocab=128,256)
  4. Sequence Packing: Documents packed to 128K with attention masking

Training Infrastructure

  • Hardware: 3,072x NVIDIA H100 80GB (SXM5)
  • Framework: Custom training framework on top of PyTorch 2.4 + Megatron-Core
  • Parallelism: 3D parallelism (Tensor parallel=8, Pipeline parallel=4, Data parallel)
  • Optimizer: AdamW (beta1=0.9, beta2=0.95, eps=1e-8), weight decay=0.1
  • Learning Rate: Cosine schedule with 2% warmup, peak lr=2.5e-4
  • Batch Size: Global batch size of 4M tokens (micro-batch=256, accumulation=4)
  • Training Steps: 819K steps (12 days)
  • Checkpoint Frequency: Every 500 steps
  • Mixed Precision: BF16 with FP32 master weights

Installation & Usage

Prerequisites

  • Python 3.10+
  • PyTorch 2.4+
  • CUDA 12.1+ (for GPU inference)

Install via pip 

pip install torch>=2.4 transformers>=4.45 accelerate>=0.34 sentencepiece>=0.2

Quick Start — Load & Inference 

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "codextb/kairos-v1"

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True,
)

prompt = "Explain the concept of backpropagation in neural networks."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    temperature=0.7,
    top_p=0.9,
    do_sample=True,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Inference with vLLM (High Throughput) 

from vllm import LLM, SamplingParams

llm = LLM(model="codextb/kairos-v1", tensor_parallel_size=4)
sampling_params = SamplingParams(temperature=0.7, top_p=0.9, max_tokens=512)

prompts = [
    "Write a Python function to merge two sorted lists.",
    "Translate 'Hello, how are you?' to French.",
]

outputs = llm.generate(prompts, sampling_params)
for out in outputs:
    print(out.outputs[0].text)

Inference with Ollama (Local Deployment) 

ollama pull codextb/kairos-v1
ollama run codextb/kairos-v1 "What is the time complexity of quicksort?"

Quantized Inference (INT4) 

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "codextb/kairos-v1",
    torch_dtype="auto",
    device_map="auto",
    load_in_4bit=True,        # bitsandbytes INT4
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype="float16",
    trust_remote_code=True,
)

Evaluation Results

English Benchmarks

Benchmark kairos-v1 Competitor A Competitor B
MMLU (5-shot) 84.7 83.2 82.9
MMLU-Pro (5-shot) 61.3 58.7 59.4
HumanEval (0-shot) 89.6 87.8 85.4
MBPP (3-shot) 86.2 83.5 81.9
GSM8K (8-shot) 91.4 89.7 88.3
MATH (4-shot) 72.8 69.1 67.5
SQuAD 2.0 (0-shot) 88.9 87.2 86.8
TriviaQA (5-shot) 87.3 85.6 84.1
HellaSwag (10-shot) 93.1 92.4 91.8
ARC-C (25-shot) 89.6 87.9 86.3
Big-Bench Hard (3-shot) 82.4 80.1 78.9

Multilingual Benchmarks (MMLU, 0-shot)

Language kairos-v1
English 84.7
Chinese 82.1
Japanese 79.8
Korean 77.3
French 83.5
German 81.9
Spanish 83.2
Hindi 71.6

Coding Benchmarks

Benchmark Task Pass@1
HumanEval Code generation 89.6%
MBPP Code generation 86.2%
LiveCodeBench (Apr 2025) Competitive programming 67.3
SWE-bench Verified Software engineering 31.4%

Efficiency Metrics

Setting GPU Tokens/sec Memory (GB)
BF16 Inference 1x H100 142 14.5
BF16 Inference 4x H100 486 5.8
INT4 Quantized 1x A100 98 4.2
vLLM (4 GPUs) 4x H100 1,820 11.2

License

This model is released under the Apache License 2.0.

You are free to:

  • Share — copy and redistribute the material in any medium or format
  • Adapt — remix, transform, and build upon the material for any purpose, even commercially

See the LICENSE file for details.

Acceptable Use Policy

By downloading and using kairos-v1, you agree not to:

  • Generate illegal content, hate speech, or material that violates applicable laws
  • Use the model for automated surveillance or mass profiling
  • Misrepresent model outputs as human-generated in sensitive contexts (healthcare, legal, judicial)
  • Reverse-engineer the model for the purpose of training a competing model at scale

Citation

If you use kairos-v1 in your research or projects, please cite:

@misc{kairos_v1_2025,
  author = {CodeXBT Research Team},
  title = {kairos-v1: A High-Performance Multimodal Foundation Model},
  year = {2025},
  publisher = {GitHub},
  journal = {GitHub repository},
  url = {https://github.com/codextb/kairos-v1},
  howpublished = {\url{https://github.com/codextb/kairos-v1}}
}

Acknowledgments

We thank the following contributors and resources:

  • CodeXBT Research Team — Model design, training, and evaluation
  • Open-source community — Leveraging transformers, vLLM, and accelerate libraries
  • Hardware partners — NVIDIA H100 compute allocation for training
  • Dataset curators — Researchers who compiled and released the public corpora used in training
  • Peer reviewers — External reviewers who provided feedback on benchmarking methodology

Model Card Details

Field Value
Model Developed By CodeXBT
Model Type Causal Language Model
Release Date July 2025
License Apache 2.0
Repository github.com/codextb/kairos-v1
Hugging Face codextb/kairos-v1