---
license: apache-2.0
language:
  - ko
  - en
pipeline_tag: text-generation
library_name: transformers
tags:
  - nova
  - gdn2
  - gated-deltanet
  - hybrid
  - korean
  - custom_code
---

# Nova-1B

Nova-1B is a 1.2B-parameter bilingual (Korean/English) language model from **INSIGHTNOVA Inc.** It is based on a hybrid architecture that combines Gated DeltaNet 2 (GDN2) with grouped-query attention (GQA). Training used publicly available data through three-stage pre-training followed by supervised fine-tuning (SFT), direct preference optimization (DPO) and reinforcement learning (RL) using soft adaptive policy optimization (SAPO), totaling ~372B cumulative input and rollout tokens.

**At a glance**

- 26 layers (13 × `[GDN2, Attention]`), hidden size 1,536, 8K context, 64K Byte-Level BPE vocabulary
- Prompted with **plain text** — no chat template (see [Prompt format](#prompt-format))
- Loads with `AutoTokenizer`, `AutoModel` and `AutoModelForCausalLM` via `trust_remote_code=True`; validated on `transformers==4.57.6`
- FP32 safetensors (~4.9 GB); designed for GPU inference with stock PyTorch (optional FlashAttention/FLA fast path); CPU inference is possible but not the design target
- Code and docs under Apache 2.0 · Source: [github.com/insightnova/Nova-1B](https://github.com/insightnova/Nova-1B)

## Quickstart

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "Insightnova/Nova-1B"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True).to("cuda:0").eval()

prompt = "질문: 대한민국의 수도는 어디인가요?\n답변:"   # "Q: What is the capital of Korea?\nA:"
input_ids = torch.tensor(
    [[tokenizer.bos_token_id] + tokenizer.encode(prompt, add_special_tokens=False)], device="cuda:0"
)
with torch.inference_mode():
    output = model.generate(
        input_ids=input_ids, attention_mask=torch.ones_like(input_ids), max_new_tokens=64, do_sample=False
    )
print(tokenizer.decode(output[0, input_ids.shape[1]:], skip_special_tokens=True).strip())
# → 대한민국의 수도는 서울입니다.  ("The capital of Korea is Seoul.")
```

## Model Details

| | |
|---|---|
| Developer | INSIGHTNOVA Inc. |
| Model type | Decoder-only causal language model, GDN2/GQA hybrid |
| Languages | Korean, English |
| Parameters | 1,220,640,156 (~1.22B) |
| Layers | 26 — GDN2 and attention blocks alternate (13 of each) |
| Hidden / FFNN size | 1,536 / 6,144 |
| Attention | Grouped-query attention, 12 query heads / 4 KV heads, 4,096-token sliding window, RoPE, QK-norm |
| Context length | 8,192 tokens |
| Tokenizer | Byte-Level BPE, 65,536 tokens, trained in-house |
| Weights | FP32 safetensors; BF16 autocast on CUDA by default (`compute_dtype="bfloat16"`), FP32 compute available via `compute_dtype="float32"` — see [Precision](#precision) |
| Embeddings | Input embedding and output head are tied |
| License | Apache 2.0 (code and documentation) |

The repository contains the model implementation itself (`custom_code`). A default load uses four files — `configuration_nova.py`, `modeling_nova.py`, `generation_nova.py` and `cpu_backend.py` — and depends only on PyTorch, Transformers and safetensors. The remaining modules (`architecture*.py`, `gdn2*.py`, `fast_decode.py`, `sequence_utils.py`) belong to the optional FlashAttention/FLA backend and are only imported when you opt into it.

The released checkpoint is the final model after SFT, DPO and RL. The tied output head is stored once; Transformers re-ties it when loading.

## How to Use

### Requirements

```bash
pip install "transformers==4.57.6" "torch>=2.8" "safetensors>=0.4.5"
```

Validated with `transformers==4.57.6` on Linux (NVIDIA GPU, `torch` backend, BF16 autocast and FP32) and on Windows (CPU, FP32). Other Transformers versions are untested; the model code warns when it detects one.

The model code is downloaded from this repository and executed, so `trust_remote_code=True` is required. For reproducible deployments, pin a reviewed commit: `from_pretrained(repo_id, revision="<commit-sha>", trust_remote_code=True)`.

### Prompt format

Nova-1B expects **plain text preceded by the BOS token**. The tokenizer does not insert BOS on its own, so prepend it yourself:

```python
ids = [tokenizer.bos_token_id] + tokenizer.encode(text, add_special_tokens=False)
```

Example plain-text prompts:

| Task | Prompt |
|---|---|
| Question answering | `질문: {question}\n답변:` — Korean for *Question: … / Answer:* |
| English → Korean translation | `Translate the following English sentence into Korean. Output only the translation.\nEnglish: {sentence}\nKorean:` |
| Instruction + question | `{instruction}\n\n질문: {question}\n답변:` |

**Why there is no chat template.** The model was fine-tuned with a conversational template, but plain-text prompting performed better overall in our evaluations. We therefore ship the model without a `chat_template` and treat plain text as the supported interface. All Nova-1B benchmark scores below were obtained with plain-text prompts.

### Generation settings

| | |
|---|---|
| Default for generation tasks | Greedy decoding, `do_sample=False` |
| Sampling | `do_sample=True, temperature=0.8, top_p=0.95, top_k=0` |
| Context budget | Prompt + generated tokens ≤ 8,192; `max_new_tokens` is capped to the remaining context |
| Batching | Left- and right-padded batches are supported; prompts of equal length are decoded together (`generation_batch_size`) |

### Precision

On CUDA the model computes in BF16 autocast by default (`compute_dtype="bfloat16"`) with the FP32 weights kept in memory; `compute_dtype="float32"` runs the whole forward pass in FP32.

```python
model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True, compute_dtype="float32").to("cuda:0").eval()
```

BF16 autocast slightly changes the logits, so outputs can differ between the two settings. In our own testing FP32 compute was somewhat more robust on some prompts; this is a qualitative observation and not reflected in the benchmark scores below, which were measured in BF16. If you see unexpected generations, `compute_dtype="float32"` is worth trying at the cost of speed and memory.

Note that on Ampere and newer GPUs PyTorch still runs FP32 convolutions in TF32 by default, so `compute_dtype="float32"` alone is not bit-exact FP32 (differences of about 1e-3 relative on the GDN2 short convolutions). For exact FP32, also set `torch.backends.cudnn.allow_tf32 = False` (and `torch.backends.cuda.matmul.allow_tf32 = False` if you have enabled it).

### Running on CPU

Nova-1B targets GPU inference, but it can also run on CPU for testing or low-volume use:

```python
model = AutoModelForCausalLM.from_pretrained(
    repo_id, trust_remote_code=True,
    inference_backend="cpu", compute_dtype="float32", dtype=torch.float32,
).to("cpu").eval()
```

Keep the inputs on CPU as well. Expect ~4.9 GB for the FP32 weights plus working memory, and generation that is far slower than on a GPU. There is no automatic fallback from GPU to CPU on out-of-memory.

### Hidden states with `AutoModel`

`AutoModel` gives you the backbone without the vocabulary projection — useful for embeddings and probing. `last_hidden_state` is the output of the final RMSNorm (dimension 1,536), i.e. exactly what the tied output head consumes. With `output_hidden_states=True` you also get 27 layer outputs (the embedding output plus one per block). Padded positions are zero-filled.

```python
from transformers import AutoModel

backbone = AutoModel.from_pretrained(repo_id, trust_remote_code=True).to("cuda:0").eval()
batch = tokenizer(["안녕하세요.", "Hello, Nova!"], padding=True, return_tensors="pt").to("cuda:0")
with torch.inference_mode():
    out = backbone(**batch, output_hidden_states=True)
out.last_hidden_state.shape   # torch.Size([2, seq_len, 1536])
len(out.hidden_states)        # 27
```

Since the head is tied to the input embedding, `backbone.get_output_embeddings()(out.last_hidden_state)` reproduces the logits of `AutoModelForCausalLM`: within 1e-3 in FP32, and within BF16 rounding (a few tenths of a logit, identical top-1 tokens) under the default autocast, because the two paths multiply differently shaped matrices.

### Accelerated backend (optional)

On Linux with NVIDIA GPUs you can run the model on the FLA GDN2 kernels and FlashAttention 2 by installing the `[flash]` extras and `flash-attn` as described in the [source repository](https://github.com/insightnova/Nova-1B), then passing `inference_backend="flash"`. Both backends load the same weights; outputs may differ slightly due to floating-point differences.

### What is and isn't supported

Supported: `AutoConfig` / `AutoTokenizer` / `AutoModel` / `AutoModelForCausalLM`; `forward()` logits; `generate()` with greedy decoding, temperature, top-p and top-k sampling, and multiple return sequences when sampling; left- or right-padded batches.

Not supported: `device_map` and CPU/disk offloading, quantization, beam search, streamers, returning scores from `generate()`, `inputs_embeds`, `output_attentions`, and passing in external `past_key_values`. The decoding cache lives on the model instance, so serve one request at a time per instance or create separate instances.

If you prefer not to use remote code, install the source repository (`pip install -e .`) and call `nova.register()`; the same classes then load with `trust_remote_code=False`.

## Benchmarks

The tables below compare the final checkpoint with four instruction-tuned models of similar size across 22 Korean and English tasks. Evaluation uses full test sets in zero-shot and 10-shot settings (ten in-context examples, frozen weights).

> **These are our own measurements, not official scores.** All numbers, for Nova-1B and for the baselines alike, were produced by INSIGHTNOVA's evaluation pipeline with our prompts, few-shot layout, decoding settings and scoring code. Results depend on these choices, so other prompts or evaluation harnesses can give different scores, and the baseline figures may not match what their developers report.

| Label | Model | Input format |
|---|---|---|
| Nova-1B | Nova-1B (~1.22B) | Plain text, no chat template |
| Llama IT | Llama 3.2 1B Instruct | Official chat template |
| Gemma IT | Gemma 3 1B IT | Official chat template |
| CLOVA IT | HyperCLOVA X SEED Text Instruct 1.5B | Official chat template |
| EXAONE IT | EXAONE 4.0 1.2B | Official chat template |

All models were scored on the same test items with the same 10-shot examples. Prompt wording and layout were selected per model. On NSMC, YNAT, KoBEST BoolQ and SQuAD, we compared single-message and multi-turn layouts within the baselines' official chat templates. The multi-turn layout performed better overall for each baseline, with examples as alternating user/assistant turns and instructions in the first user turn.

### Zero-shot

| Task / Metric | Nova-1B | Llama IT | Gemma IT | CLOVA IT | EXAONE IT |
|---|---:|---:|---:|---:|---:|
| NSMC / Acc. | 78.10 | 49.65 | 78.32 | **84.52** | 73.80 |
| KLUE-YNAT / Acc. | 30.59 | 41.45 | 47.13 | **57.07** | 34.84 |
| KoBEST COPA / F1 | 67.80 | 52.76 | 61.09 | **69.95** | 59.49 |
| KoBEST HellaSwag / F1 | **50.87** | 41.50 | 41.40 | 49.63 | 40.54 |
| KoBEST SentiNeg / F1 | **87.63** | 33.72 | 49.18 | 46.09 | 36.67 |
| KoBEST BoolQ / F1 | 67.99 | 54.54 | 56.87 | **76.69** | 65.58 |
| KoBEST WiC / F1 | 32.80 | 33.21 | 48.60 | **56.20** | 32.80 |
| KorQuAD v1 / F1 | **87.76** | 48.05 | 67.65 | 83.26 | 81.62 |
| SQuAD v1 / F1 | 76.50 | 72.95 | 72.43 | **80.15** | 74.46 |
| MNLI / Acc. | 39.81 | 35.88 | 46.11 | 35.80 | **47.01** |
| STS-B / Pearson | 0.5575 | -0.1288 | 0.2959 | **0.7660** | 0.1274 |
| CoNLL-2003 / F1 | 0.00 | 0.00 | 0.00 | **1.73** | 0.00 |
| En → Ko / BLEU | 20.00 | 7.72 | 18.90 | **26.89** | 23.64 |
| Ko → En / BLEU | **26.53** | 13.69 | 18.61 | 16.86 | 24.35 |
| BoolQ / Acc. | 74.22 | 65.41 | 73.12 | **76.85** | 74.28 |
| PIQA / Acc_norm | **74.27** | 72.74 | 67.79 | 71.38 | 66.92 |
| SIQA / Acc_norm | **49.13** | 43.86 | 41.76 | 44.11 | 42.27 |
| HellaSwag / Acc_norm | **60.72** | 55.38 | 40.15 | 57.35 | 48.88 |
| WinoGrande / Acc. | **59.83** | 58.09 | 54.93 | 58.01 | 54.22 |
| ARC-e / Acc_norm | **65.57** | 48.48 | 44.87 | 53.79 | 44.95 |
| ARC-c / Acc_norm | **41.38** | 34.73 | 35.67 | 38.65 | 35.15 |
| OpenBookQA / Acc_norm | 36.40 | **40.00** | 39.20 | 35.80 | 33.40 |

### 10-shot

| Task / Metric | Nova-1B | Llama IT | Gemma IT | CLOVA IT | EXAONE IT |
|---|---:|---:|---:|---:|---:|
| NSMC / Acc. | 83.27 | 76.39 | 80.66 | **86.04** | 81.01 |
| KLUE-YNAT / Acc. | 46.50 | 52.00 | 59.10 | **64.08** | 42.70 |
| KoBEST COPA / F1 | 68.09 | 53.37 | 62.28 | **74.15** | 62.17 |
| KoBEST HellaSwag / F1 | **55.16** | 43.48 | 46.86 | 51.75 | 43.45 |
| KoBEST SentiNeg / F1 | 94.45 | 66.47 | 65.35 | 43.41 | **94.46** |
| KoBEST BoolQ / F1 | 66.82 | 45.57 | 62.68 | 81.05 | **81.50** |
| KoBEST WiC / F1 | 35.19 | 40.94 | 39.61 | 45.18 | **45.31** |
| KorQuAD v1 / F1 | **87.14** | 62.87 | 68.48 | 83.75 | 82.04 |
| SQuAD v1 / F1 | **84.24** | 79.98 | 73.30 | 81.74 | 81.89 |
| MNLI / Acc. | 65.20 | 31.80 | 43.55 | 65.75 | **74.27** |
| STS-B / Pearson | 0.5349 | -0.0626 | 0.0244 | 0.6919 | **0.7427** |
| CoNLL-2003 / F1 | 41.98 | 40.86 | 41.75 | 11.17 | **43.68** |
| En → Ko / BLEU | 22.87 | 9.08 | 21.13 | **28.67** | 24.36 |
| Ko → En / BLEU | **29.88** | 16.16 | 18.62 | 28.24 | 20.48 |
| BoolQ / Acc. | 75.11 | 39.72 | 68.84 | **76.70** | 76.61 |
| PIQA / Acc_norm | **75.08** | 73.07 | 71.87 | 70.95 | 68.44 |
| SIQA / Acc_norm | **54.45** | 42.02 | 48.16 | 49.03 | 48.11 |
| HellaSwag / Acc_norm | **61.96** | 51.38 | 41.67 | 55.96 | 45.27 |
| WinoGrande / Acc. | **62.51** | 59.12 | 54.54 | 55.17 | 54.30 |
| ARC-e / Acc_norm | **72.35** | 38.09 | 54.12 | 65.03 | 60.61 |
| ARC-c / Acc_norm | **43.09** | 34.56 | 37.88 | 42.15 | 41.21 |
| OpenBookQA / Acc_norm | **38.20** | 34.20 | 35.80 | 36.60 | 33.00 |

Bold marks the best score in each row. Accuracy, F1 and BLEU are on a 0–100 scale; STS-B reports the raw Pearson correlation. KoBEST uses macro-F1, KorQuAD and SQuAD use answer-token F1, and CoNLL-2003 uses entity-level micro-F1. Acc_norm is accuracy using candidate log-likelihood normalized by character length. English BoolQ and HellaSwag are separate from their Korean KoBEST counterparts.

**Setup.** Seed 42, BF16 compute, 8,192-token context. Generation tasks use greedy decoding; classification and multiple-choice tasks use candidate log-likelihoods. Maximum generation length: 64 tokens for reading comprehension, 16 for STS, 128 for NER, 256 for translation. All five models were run in BF16; Nova-1B scores under `compute_dtype="float32"` (see [Precision](#precision)) may differ slightly.

**Evaluation Notes.** These results depend on the prompts used for each model and should be interpreted in that context. The four tasks used to choose the baselines' input format also appear in the final tables, so those entries are not an independent held-out evaluation. Nova-1B leads in both settings on Korean reading comprehension (KorQuAD), Korean HellaSwag and Korean → English translation, and on 10-shot SQuAD. Topic classification (YNAT) and word-sense disambiguation (WiC) remain weak spots. Small gaps should not be read as statistically significant.

## Training

### Data

Korean data came primarily from KorMix, KORMo, FineWeb-2, HPLT, the NIKL Modu Corpus and AI Hub; English data came primarily from Dolma/Common Crawl, English Wikipedia and STEM web pages. The corpus also included math, science, code, instruction and reasoning data, and Korean–English parallel text. Documents were normalized and filtered, then arranged by quality and length into the pre-training curriculum; post-training data (QA, dialogue, instruction and preference data in both languages) was converted into the respective training formats.

### Procedure

| Stage | What | Volume |
|---|---|---:|
| Pre-training, stage 1 | General Korean/English text, math, instructions · 4K context | 223.3B tokens |
| Pre-training, stage 2 | High-quality documents ≤ 2,048 tokens · 4K context | 68.2B tokens |
| Pre-training, stage 3 | High-quality long documents + 3.05M Korean–English parallel pairs · 8K context | 70.8B tokens |
| SFT | QA, multi-turn dialogue, reading comprehension, writing/editing, instruction following | ~12.0M response samples |
| DPO | Korean/English preference pairs against a frozen reference model | 719,507 pairs |
| RLVR + RLAIF (SAPO) | Verifier- and judge-scored policy optimization | 26,019 prompts (19,077 Ko / 6,942 En) |

The ~372B-token total counts cumulative training inputs and RL rollouts, not unique source tokens. Pre-training uses next-token prediction with a curriculum that moves from the general corpus to high-quality short documents and finally high-quality long documents; parallel pairs in stage 3 are balanced across both translation directions. The learning rate follows a warmup–stable–decay (WSD) schedule. SFT computes the loss on assistant turns only, with the dialogue history as context. DPO raises the likelihood of chosen over rejected responses relative to a frozen reference model. Reinforcement learning combines verifiable rewards (RLVR) and AI feedback (RLAIF), using soft adaptive policy optimization (SAPO). String, numeric and date answers are checked by verifiers, while content and grounding are scored by LLM judges.

There is no separate technical report. This model card and the [source repository](https://github.com/insightnova/Nova-1B) are the complete public documentation.

## Intended Use

Nova-1B is intended for research, prototyping and lightweight inference on Korean and English text: question answering, reading comprehension, classification and judgment tasks, English ↔ Korean translation and short-form generation, driven by plain-text prompts. Its size makes it practical on a single GPU and suitable as a base for further fine-tuning.

**Out of scope:** medical, legal or financial advice or any setting where factual accuracy is critical; user-facing deployment without output filtering and a safety review; languages other than Korean and English; tasks that hinge on very long documents. Multi-turn chat through a chat template is not the supported interface and is not expected to perform well.

## Bias, Risks and Limitations

- Outputs can be factually wrong or inaccurate. Evaluation surfaced a label bias on KoBEST WiC and output-format failures on zero-shot CoNLL-2003.
- Some prompts trigger repetition or drift from the instruction; for open-ended prompts such as self-introductions the model may keep generating follow-up questions until it hits the token limit.
- Mathematical reasoning and code generation remain areas for improvement; strong language-understanding scores do not imply similar performance on these tasks.
- Long-context understanding and behaviour under production load have not been separately validated.
- No dedicated safety evaluation (toxicity, bias, personal-data leakage) was performed. The training data includes public web crawls, so the model can produce biased or inappropriate text. Deployers are responsible for input/output filtering and for evaluating the model in their own context.

## License

The code and documentation in this repository are released under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0). The implementation builds on [OLMo-core](https://github.com/allenai/OLMo-core), [Flash Linear Attention](https://github.com/fla-org/flash-linear-attention), [Transformers](https://github.com/huggingface/transformers) and [TRL](https://github.com/huggingface/trl); third-party components retain their own licenses (see `THIRD_PARTY_NOTICES.txt`).

## Citation

```bibtex
@misc{insightnova2026nova1b,
  title  = {Nova-1B},
  author = {INSIGHTNOVA Inc.},
  year   = {2026},
  url    = {https://huggingface.co/Insightnova/Nova-1B}
}
```

## Contact

Developed by INSIGHTNOVA Inc. — Kyungjin Lee (pre-training, English training data collection and preparation), Jiyeun Cho (Korean training data collection and preparation, tokenizer development), Jeongyeon Park (architecture, post-training, evaluation).

For inquiries, contact **Kyungjin Lee**.

Email: kjlee@insightnova.kr

Source code, inference scripts and issue tracker: [github.com/insightnova/Nova-1B](https://github.com/insightnova/Nova-1B)