Papers
arxiv:2611.21003

K2-Horizon: A Frontier Fleet of Open Source Language Models

Published on Nov 16
ยท Submitted by
Zhenting Wang
on Nov 17
#2 Paper of the day
Authors:
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,

Abstract

We present K2-Horizon, a fleet of six fully open language models, from a 0.9B dense model to a 375B-A23B MoE, built for general-purpose reasoning and agentic work across math and science, software engineering, deep research, office automation, and beyond. The models lead or are competitive with strong open models of their size on agentic and coding benchmarks. On the modeling side, to strengthen reasoning, we introduce Mixture-of-Value Attention (MoVA), applying MoE to attention value projections, and softplus attention gating, a more flexible, theoretically motivated gate. Our 36B-A4B model uses both and nearly matches a 32B dense model trained on the same data, with one-seventh the active parameters. We also introduce Uno, lossless diffusion adapters that speed up decoding by up to 2.2x without a separate draft model. Our release follows the LLM360 definition of open source for transparency and reproducibility: the full development history, including intermediate checkpoints from every training stage, pre-training data, training logs, and all training, evaluation, and infrastructure code, notably our production-scale xLLM framework for dense and MoE training and the RL360 reinforcement-learning stack. Beyond the data itself, we release and document in detail every pipeline that built it, the part of model development that open releases most rarely disclose. These pipelines produced roughly 10T synthetic pre-training tokens, showing that synthetic data can be a major ingredient of pre-training at scale. Four of the models share one 22T-token pre-training corpus and recipe, revealing consistent scaling behavior and enabling controlled studies of scale and sparsity. All artifacts are available at https://ifm.ai/k2/

Community

Paper submitter

We present K2-Horizon, a fleet of six fully open language models, from a 0.9B dense model to a
375B-A23B MoE, built for general-purpose reasoning and agentic work across math and science,
software engineering, deep research, office automation, and beyond. The models lead or are
competitive with strong open models of their size on agentic and coding benchmarks. On the
modeling side, to strengthen reasoning, we introduce Mixture-of-Value Attention (MoVA), applying
MoE to attention value projections, and softplus attention gating, a more flexible, theoretically
motivated gate. Our 36B-A4B model uses both and nearly matches a 32B dense model trained on
the same data, with one-seventh the active parameters. We also introduce Uno, lossless diffusion
adapters that speed up decoding by up to 2.2ร— without a separate draft model. Our release follows
the LLM360 definition of open source for transparency and reproducibility: the full development
history, including intermediate checkpoints from every training stage, pre-training data, training logs,
and all training, evaluation, and infrastructure code, notably our production-scale xLLM framework
for dense and MoE training and the RL360 reinforcement-learning stack. Beyond the data itself,
we release and document in detail every pipeline that built it, the part of model development that
open releases most rarely disclose. These pipelines produced roughly 10T synthetic pre-training
tokens, showing that synthetic data can be a major ingredient of pre-training at scale. Four of the
models share one 22T-token pre-training corpus and recipe, revealing consistent scaling behavior and
enabling controlled studies of scale and sparsity. All artifacts are available at https://ifm.ai/k2/.

Sign up or log in to comment

Get this paper in your agent:

hf papers read 2611.21003
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.21003 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.21003 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.21003 in a Space README.md to link it from this page.

Collections including this paper 1