README.md

---
license: mit
library_name: transformers
tags:
  - text-generation
  - base-model
  - qwen2
  - english
  - chinese
  - pytorch
---

# OpenSI-0.3B-Base

**An open base language model, pretrained from scratch on 2.6B tokens.**

`Qwen2` architecture · **113.9M params** · 2048 context · MIT License

<div align="center">

### ⚠️ The model is 113.9M parameters, not 0.3B

The repo name says `0.3B`, but the **actual parameter count is 113,932,032 (113.9M)**.
Treat it as a **~0.11B** model. See *Size reconciliation* below.

</div>

---

The repo name says `0.3B`, but the **actual parameter count is 113,932,032 (113.9M)**. The
name is kept for continuity with the project's internal plan (`§5.3`), not because it
describes this checkpoint. Treat it as a **~0.11B** model.

| | |
|---|---|
| Parameters | **113,932,032** (113.9M) |
| Layers / hidden | 12 / 768 |
| Heads (GQA) | 12 query, 4 KV |
| FFN (SwiGLU) | 2048 |
| Vocab | 50,000 (ByteLevel BPE, 2.39 chars/token) |
| Context | 2048 |
| Embeddings | tied |
| Trained on | 2,601,279,471 tokens (11,182,671 documents) |

Breakdown: MLP 56.6M (49.7%) · embeddings 38.4M (33.7%) · attention 18.9M (16.6%).

## Evaluation results

All numbers are full-set, produced by `scripts/bench.py` (see *Evaluation method* below).
`acc` is the primary metric.

| Benchmark | n | acc | acc_norm | Random baseline | Verdict |
|---|---|---|---|---|---|
| **LAMBADA** | 5,153 | **46.51%** ¹ | 18.42% ² | ~0% | ✅ far above random |
| **ARC-Easy** | 570 | **32.63%** | 29.12% | 25–27.8% | ✅ +5–8 pp |
| **HellaSwag** | 10,042 | **26.27%** | 25.88% | 25.0% | ❌ **+1.3 pp only** |
| **C-Eval** | 1,346 | 21.62% | 27.56% | 25.0% | ⚠️ acc below random |

¹ LAMBADA is single-candidate, so multi-choice `acc` is meaningless (always 100%).
Reported here as **greedy token hit rate** — whether the target word is the argmax
continuation — which is the metric lm-eval-harness uses for this task.
² Whole-word all-correct rate.

Plus, from the deterministic dev pass (13,064,192 tokens):

| | |
|---|---|
| dev loss / ppl | 3.3336 / **28.04** (σ≈0.006) |
| EOS boundary signal | P(EOS\|doc end)=0.611 vs P(EOS\|body)=0.000014 → **562×** |
| Overfitting check | sampling σ=0.0930 → **no overfitting** |

## What this model can and cannot do

**Can:** English language modeling. LAMBADA 46.51% against a ~0% random baseline is a
large, unambiguous signal that 2.6B tokens of pretraining produced a working language
model. Sentences are fluent and well-formed.

**Cannot:** world knowledge and commonsense reasoning. HellaSwag sits 1.3 pp above
random despite a **median top-2 margin of 14.6 nats** — the model is *confidently wrong*,
not undecided. That task requires understanding what people are physically doing in a
scene; fluency does not transfer to it.

**Chinese:** no academic knowledge. C-Eval accuracy (21.62%) falls *below* random, and the
prediction distribution collapses onto option A (42.3% predicted vs 23.0% actual gold),
i.e. a positional bias rather than knowledge. Length-normalized accuracy (27.56%) is
barely above random.

The LAMBADA ↔ HellaSwag gap (46.51% vs 26.27%) is the clearest single statement of what
this checkpoint is: **it learned language, not knowledge.**

## Why HellaSwag did not meet the original bar

The original acceptance criterion for this tier was **HellaSwag ≥35%**. Measured: 26.27%.
Before accepting that, the following were ruled out one by one: scorer self-checks passed;
bf16 noise is ~29× smaller than the median decision margin; trivial baselines (always-first,
always-longest, always-shortest) all score 24–25%; the same code scores 32.63% on ARC-Easy,
above its own random baseline; and a tokenization-boundary defect was found, quantified
(+1.67 pp), and shown to be far too small to close an 8.7 pp gap.

The cause is data, and it is measurable. Training-data language mix **by token**, not by
document:

| | docs | **tokens (measured)** |
|---|---|---|
| English | 70% | **49.9%** |
| Chinese | 30% | **45.9%** |

Chinese documents yield ~2.1× more tokens each, so a 70/30 document split becomes a
50/46 token split. HellaSwag is pure English, so this model effectively saw ~1.30B English
tokens — and a chunk of that was TinyStories (synthetic simple English), which is close to
useless for commonsense about real-world activity.

**No scale-up was attempted.** Per the project's decision rule, a stage that misses its gate
may only be made smaller or shorter, never larger.

## Intended use

Research and experimentation on small-scale base-model pretraining. Suitable as a
fine-tuning base for English tasks. **Not suitable** for factual QA, commonsense reasoning,
Chinese academic work, or any production deployment.

## Evaluation method

`lm-eval-harness` could not be installed (no PyPI access from this environment), so the
four loglikelihood benchmarks are implemented in `scripts/bench.py`, aligned with
lm-eval's `doc_to_text` / `doc_to_target` / `preprocess`. Scorer correctness is enforced by
three self-checks that must pass before any number is reported:

- **A** batched scoring vs a `batch=1` reference implementation, per-item logprob
- **B** index check — the true continuation must score above scrambled continuations
- **C** padding invariance — identical results alone vs inside a mixed-length batch

Tolerances are dtype-aware: fp32 `1e-3`, bf16 `0.5`. (bf16's noise floor was measured at
7.1e-2 against fp32's 5.7e-6, matching the predicted bound for logit magnitude 10.9.)

Export to this Qwen2 format was verified numerically against the training implementation:
max absolute error **4.244e-05**, cosine similarity **1.0000000000**, argmax agreement
**100.00%**.

One caveat: lm-eval tokenizes context and continuation separately, which for ByteLevel BPE
without a leading space produces a boundary that does not exist in the original text
(`'…on a roof. Heis using…'`). This matches lm-eval's behaviour and is kept for
comparability; correcting it raises HellaSwag 300-item accuracy 32.00% → 33.67%.

## Training details

Optimizer AdamW (β 0.9/0.95, wd 0.1, eps 1e-8), LR 3e-4 → 3e-5 cosine, 2% warmup,
grad clip 1.0, global batch ≈ 524k tokens/step (32 × 2048 × 8 accumulation), bf16 with SDPA,
`torch.compile`, seed 1337. 4,961 steps over 14h42m on a single NVIDIA DGX Spark (GB10);
train EMA loss 11.0017 → 3.2274 (−70.7%); peak memory 29.8 GB.

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("SIZWL/OpenSI-0.3B-Base")
model = AutoModelForCausalLM.from_pretrained(
    "SIZWL/OpenSI-0.3B-Base", dtype="auto", device_map="auto"
)

ids = tok("The capital of France is", return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=20, do_sample=False,
                     pad_token_id=tok.pad_token_id)
print(tok.decode(out[0]))
```

This is a **base model**, not instruction-tuned. It continues text rather than answering
questions, and may not terminate cleanly on a complete sentence — prefer sampling with a
repetition penalty for open-ended generation:

```python
out = model.generate(**ids, max_new_tokens=64, do_sample=True, temperature=0.8, top_k=40,
                     repetition_penalty=1.2, eos_token_id=tok.eos_token_id,
                     pad_token_id=tok.pad_token_id)
```

## Data sources

| Source | Role |
|---|---|
| FineWeb | English web text |
| Wikipedia (zh) | Chinese encyclopedic |
| TinyStories | Synthetic simple English |
| Industry Chinese documents | Chinese domain text |

See §4.5 of the project plan for the mix and §4.7 for the verification tooling.

## License

MIT.

## Citation

```bibtex
@online{opensi03bbase,
  title  = {{OpenSI-0.3B-Base}: A 113.9M Open Base Language Model Pretrained on 2.6B Tokens},
  author = {{OpenSI Project}},
  year   = {2026},
  url    = {https://huggingface.co/SIZWL/OpenSI-0.3B-Base},
  note   = {Exported in Qwen2 format. Actual size 113.9M parameters.}
}
```