Instructions to use OpenSI-SI/OpenSI-0.3B-Base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use OpenSI-SI/OpenSI-0.3B-Base with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="OpenSI-SI/OpenSI-0.3B-Base")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("OpenSI-SI/OpenSI-0.3B-Base")
model = AutoModelForCausalLM.from_pretrained("OpenSI-SI/OpenSI-0.3B-Base", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use OpenSI-SI/OpenSI-0.3B-Base with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "OpenSI-SI/OpenSI-0.3B-Base"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OpenSI-SI/OpenSI-0.3B-Base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/OpenSI-SI/OpenSI-0.3B-Base

SGLang

How to use OpenSI-SI/OpenSI-0.3B-Base with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "OpenSI-SI/OpenSI-0.3B-Base" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OpenSI-SI/OpenSI-0.3B-Base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "OpenSI-SI/OpenSI-0.3B-Base" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OpenSI-SI/OpenSI-0.3B-Base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use OpenSI-SI/OpenSI-0.3B-Base with Docker Model Runner:
```
docker model run hf.co/OpenSI-SI/OpenSI-0.3B-Base
```

OpenSI-0.3B-Base / README.md

SIZWL

Upload README.md

cf26fa1 verified 24 days ago

preview code

Raw History Blame Contribute Delete

8.02 kB

---
license: mit
library_name: transformers
tags:
  - text-generation
  - base-model
  - qwen2
  - english
  - chinese
  - pytorch
---
# OpenSI-0.3B-Base

**An open base language model, pretrained from scratch on 2.6B tokens.**

`Qwen2` architecture · **113.9M params** · 2048 context · MIT License

<div align="center">

### ⚠️ The model is 113.9M parameters, not 0.3B

The repo name says `0.3B`, but the **actual parameter count is 113,932,032 (113.9M)**.
Treat it as a **~0.11B** model. See *Size reconciliation* below.

</div>

---

The repo name says `0.3B`, but the **actual parameter count is 113,932,032 (113.9M)**. The
name is kept for continuity with the project's internal plan (`§5.3`), not because it
describes this checkpoint. Treat it as a **~0.11B** model.

| | |
|---|---|
| Parameters | **113,932,032** (113.9M) |
| Layers / hidden | 12 / 768 |
| Heads (GQA) | 12 query, 4 KV |
| FFN (SwiGLU) | 2048 |
| Vocab | 50,000 (ByteLevel BPE, 2.39 chars/token) |
| Context | 2048 |
| Embeddings | tied |
| Trained on | 2,601,279,471 tokens (11,182,671 documents) |

Breakdown: MLP 56.6M (49.7%) · embeddings 38.4M (33.7%) · attention 18.9M (16.6%).

## Evaluation results

All numbers are full-set, produced by `scripts/bench.py` (see *Evaluation method* below).
`acc` is the primary metric.

| Benchmark | n | acc | acc_norm | Random baseline | Verdict |
|---|---|---|---|---|---|
| **LAMBADA** | 5,153 | **46.51%** ¹ | 18.42% ² | ~0% | ✅ far above random |
| **ARC-Easy** | 570 | **32.63%** | 29.12% | 25–27.8% | ✅ +5–8 pp |
| **HellaSwag** | 10,042 | **26.27%** | 25.88% | 25.0% | ❌ **+1.3 pp only** |
| **C-Eval** | 1,346 | 21.62% | 27.56% | 25.0% | ⚠️ acc below random |
¹ LAMBADA is single-candidate, so multi-choice `acc` is meaningless (always 100%).
Reported here as **greedy token hit rate** — whether the target word is the argmax
continuation — which is the metric lm-eval-harness uses for this task.
² Whole-word all-correct rate.
Plus, from the deterministic dev pass (13,064,192 tokens):
| | |
|---|---|
| dev loss / ppl | 3.3336 / **28.04** (σ≈0.006) |
| EOS boundary signal | P(EOS\|doc end)=0.611 vs P(EOS\|body)=0.000014 → **562×** |
| Overfitting check | sampling σ=0.0930 → **no overfitting** |
## What this model can and cannot do
**Can:** English language modeling. LAMBADA 46.51% against a ~0% random baseline is a
large, unambiguous signal that 2.6B tokens of pretraining produced a working language
model. Sentences are fluent and well-formed.
**Cannot:** world knowledge and commonsense reasoning. HellaSwag sits 1.3 pp above
random despite a **median top-2 margin of 14.6 nats** — the model is *confidently wrong*,
not undecided. That task requires understanding what people are physically doing in a
scene; fluency does not transfer to it.
**Chinese:** no academic knowledge. C-Eval accuracy (21.62%) falls *below* random, and the
prediction distribution collapses onto option A (42.3% predicted vs 23.0% actual gold),
i.e. a positional bias rather than knowledge. Length-normalized accuracy (27.56%) is
barely above random.
The LAMBADA ↔ HellaSwag gap (46.51% vs 26.27%) is the clearest single statement of what
this checkpoint is: **it learned language, not knowledge.**
## Why HellaSwag did not meet the original bar
The original acceptance criterion for this tier was **HellaSwag ≥35%**. Measured: 26.27%.
Before accepting that, the following were ruled out one by one: scorer self-checks passed;
bf16 noise is ~29× smaller than the median decision margin; trivial baselines (always-first,
always-longest, always-shortest) all score 24–25%; the same code scores 32.63% on ARC-Easy,
above its own random baseline; and a tokenization-boundary defect was found, quantified
(+1.67 pp), and shown to be far too small to close an 8.7 pp gap.
The cause is data, and it is measurable. Training-data language mix **by token**, not by
document:
| | docs | **tokens (measured)** |
|---|---|---|
| English | 70% | **49.9%** |
| Chinese | 30% | **45.9%** |
Chinese documents yield ~2.1× more tokens each, so a 70/30 document split becomes a
50/46 token split. HellaSwag is pure English, so this model effectively saw ~1.30B English
tokens — and a chunk of that was TinyStories (synthetic simple English), which is close to
useless for commonsense about real-world activity.
**No scale-up was attempted.** Per the project's decision rule, a stage that misses its gate
may only be made smaller or shorter, never larger.
## Intended use
Research and experimentation on small-scale base-model pretraining. Suitable as a
fine-tuning base for English tasks. **Not suitable** for factual QA, commonsense reasoning,
Chinese academic work, or any production deployment.
## Evaluation method
`lm-eval-harness` could not be installed (no PyPI access from this environment), so the
four loglikelihood benchmarks are implemented in `scripts/bench.py`, aligned with
lm-eval's `doc_to_text` / `doc_to_target` / `preprocess`. Scorer correctness is enforced by
three self-checks that must pass before any number is reported:
- **A** batched scoring vs a `batch=1` reference implementation, per-item logprob
- **B** index check — the true continuation must score above scrambled continuations
- **C** padding invariance — identical results alone vs inside a mixed-length batch
Tolerances are dtype-aware: fp32 `1e-3`, bf16 `0.5`. (bf16's noise floor was measured at
7.1e-2 against fp32's 5.7e-6, matching the predicted bound for logit magnitude 10.9.)
Export to this Qwen2 format was verified numerically against the training implementation:
max absolute error **4.244e-05**, cosine similarity **1.0000000000**, argmax agreement
**100.00%**.
One caveat: lm-eval tokenizes context and continuation separately, which for ByteLevel BPE
without a leading space produces a boundary that does not exist in the original text
(`'…on a roof. Heis using…'`). This matches lm-eval's behaviour and is kept for
comparability; correcting it raises HellaSwag 300-item accuracy 32.00% → 33.67%.
## Training details
Optimizer AdamW (β 0.9/0.95, wd 0.1, eps 1e-8), LR 3e-4 → 3e-5 cosine, 2% warmup,
grad clip 1.0, global batch ≈ 524k tokens/step (32 × 2048 × 8 accumulation), bf16 with SDPA,
`torch.compile`, seed 1337. 4,961 steps over 14h42m on a single NVIDIA DGX Spark (GB10);
train EMA loss 11.0017 → 3.2274 (−70.7%); peak memory 29.8 GB.
## Usage
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("SIZWL/OpenSI-0.3B-Base")
model = AutoModelForCausalLM.from_pretrained(
    "SIZWL/OpenSI-0.3B-Base", dtype="auto", device_map="auto"
)

ids = tok("The capital of France is", return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=20, do_sample=False,
                     pad_token_id=tok.pad_token_id)
print(tok.decode(out[0]))
```
This is a **base model**, not instruction-tuned. It continues text rather than answering
questions, and may not terminate cleanly on a complete sentence — prefer sampling with a
repetition penalty for open-ended generation:
```python
out = model.generate(**ids, max_new_tokens=64, do_sample=True, temperature=0.8, top_k=40,
                     repetition_penalty=1.2, eos_token_id=tok.eos_token_id,
                     pad_token_id=tok.pad_token_id)
```
## Data sources

| Source | Role |
|---|---|
| FineWeb | English web text |
| Wikipedia (zh) | Chinese encyclopedic |
| TinyStories | Synthetic simple English |
| Industry Chinese documents | Chinese domain text |

See §4.5 of the project plan for the mix and §4.7 for the verification tooling.

## License

MIT.

## Citation

```bibtex
@online{opensi03bbase,
  title  = {{OpenSI-0.3B-Base}: A 113.9M Open Base Language Model Pretrained on 2.6B Tokens},
  author = {{OpenSI Project}},
  year   = {2026},
  url    = {https://huggingface.co/SIZWL/OpenSI-0.3B-Base},
  note   = {Exported in Qwen2 format. Actual size 113.9M parameters.}
}
```

	---
	license: mit
	library_name: transformers
	tags:
	- text-generation
	- base-model
	- qwen2
	- english
	- chinese
	- pytorch
	---

	# OpenSI-0.3B-Base

	An open base language model, pretrained from scratch on 2.6B tokens.

	`Qwen2` architecture · 113.9M params · 2048 context · MIT License

	<div align="center">

	### ⚠️ The model is 113.9M parameters, not 0.3B

	The repo name says `0.3B`, but the actual parameter count is 113,932,032 (113.9M).
	Treat it as a ~0.11B model. See Size reconciliation below.

	</div>

	---

	The repo name says `0.3B`, but the actual parameter count is 113,932,032 (113.9M). The
	name is kept for continuity with the project's internal plan (`§5.3`), not because it
	describes this checkpoint. Treat it as a ~0.11B model.

	\| \| \|
	\|---\|---\|
	\| Parameters \| 113,932,032 (113.9M) \|
	\| Layers / hidden \| 12 / 768 \|
	\| Heads (GQA) \| 12 query, 4 KV \|
	\| FFN (SwiGLU) \| 2048 \|
	\| Vocab \| 50,000 (ByteLevel BPE, 2.39 chars/token) \|
	\| Context \| 2048 \|
	\| Embeddings \| tied \|
	\| Trained on \| 2,601,279,471 tokens (11,182,671 documents) \|

	Breakdown: MLP 56.6M (49.7%) · embeddings 38.4M (33.7%) · attention 18.9M (16.6%).

	## Evaluation results

	All numbers are full-set, produced by `scripts/bench.py` (see Evaluation method below).
	`acc` is the primary metric.

	\| Benchmark \| n \| acc \| acc_norm \| Random baseline \| Verdict \|
	\|---\|---\|---\|---\|---\|---\|
	\| LAMBADA \| 5,153 \| 46.51% ¹ \| 18.42% ² \| ~0% \| ✅ far above random \|
	\| ARC-Easy \| 570 \| 32.63% \| 29.12% \| 25–27.8% \| ✅ +5–8 pp \|
	\| HellaSwag \| 10,042 \| 26.27% \| 25.88% \| 25.0% \| ❌ +1.3 pp only \|
	\| C-Eval \| 1,346 \| 21.62% \| 27.56% \| 25.0% \| ⚠️ acc below random \|

	¹ LAMBADA is single-candidate, so multi-choice `acc` is meaningless (always 100%).
	Reported here as greedy token hit rate — whether the target word is the argmax
	continuation — which is the metric lm-eval-harness uses for this task.
	² Whole-word all-correct rate.

	Plus, from the deterministic dev pass (13,064,192 tokens):

	\| \| \|
	\|---\|---\|
	\| dev loss / ppl \| 3.3336 / 28.04 (σ≈0.006) \|
	\| EOS boundary signal \| P(EOS\\|doc end)=0.611 vs P(EOS\\|body)=0.000014 → 562× \|
	\| Overfitting check \| sampling σ=0.0930 → no overfitting \|

	## What this model can and cannot do

	Can: English language modeling. LAMBADA 46.51% against a ~0% random baseline is a
	large, unambiguous signal that 2.6B tokens of pretraining produced a working language
	model. Sentences are fluent and well-formed.

	Cannot: world knowledge and commonsense reasoning. HellaSwag sits 1.3 pp above
	random despite a median top-2 margin of 14.6 nats — the model is confidently wrong,
	not undecided. That task requires understanding what people are physically doing in a
	scene; fluency does not transfer to it.

	Chinese: no academic knowledge. C-Eval accuracy (21.62%) falls below random, and the
	prediction distribution collapses onto option A (42.3% predicted vs 23.0% actual gold),
	i.e. a positional bias rather than knowledge. Length-normalized accuracy (27.56%) is
	barely above random.

	The LAMBADA ↔ HellaSwag gap (46.51% vs 26.27%) is the clearest single statement of what
	this checkpoint is: it learned language, not knowledge.

	## Why HellaSwag did not meet the original bar

	The original acceptance criterion for this tier was HellaSwag ≥35%. Measured: 26.27%.
	Before accepting that, the following were ruled out one by one: scorer self-checks passed;
	bf16 noise is ~29× smaller than the median decision margin; trivial baselines (always-first,
	always-longest, always-shortest) all score 24–25%; the same code scores 32.63% on ARC-Easy,
	above its own random baseline; and a tokenization-boundary defect was found, quantified
	(+1.67 pp), and shown to be far too small to close an 8.7 pp gap.

	The cause is data, and it is measurable. Training-data language mix by token, not by
	document:

	\| \| docs \| tokens (measured) \|
	\|---\|---\|---\|
	\| English \| 70% \| 49.9% \|
	\| Chinese \| 30% \| 45.9% \|

	Chinese documents yield ~2.1× more tokens each, so a 70/30 document split becomes a
	50/46 token split. HellaSwag is pure English, so this model effectively saw ~1.30B English
	tokens — and a chunk of that was TinyStories (synthetic simple English), which is close to
	useless for commonsense about real-world activity.

	No scale-up was attempted. Per the project's decision rule, a stage that misses its gate
	may only be made smaller or shorter, never larger.

	## Intended use

	Research and experimentation on small-scale base-model pretraining. Suitable as a
	fine-tuning base for English tasks. Not suitable for factual QA, commonsense reasoning,
	Chinese academic work, or any production deployment.

	## Evaluation method

	`lm-eval-harness` could not be installed (no PyPI access from this environment), so the
	four loglikelihood benchmarks are implemented in `scripts/bench.py`, aligned with
	lm-eval's `doc_to_text` / `doc_to_target` / `preprocess`. Scorer correctness is enforced by
	three self-checks that must pass before any number is reported:

	- A batched scoring vs a `batch=1` reference implementation, per-item logprob
	- B index check — the true continuation must score above scrambled continuations
	- C padding invariance — identical results alone vs inside a mixed-length batch

	Tolerances are dtype-aware: fp32 `1e-3`, bf16 `0.5`. (bf16's noise floor was measured at
	7.1e-2 against fp32's 5.7e-6, matching the predicted bound for logit magnitude 10.9.)

	Export to this Qwen2 format was verified numerically against the training implementation:
	max absolute error 4.244e-05, cosine similarity 1.0000000000, argmax agreement
	100.00%.

	One caveat: lm-eval tokenizes context and continuation separately, which for ByteLevel BPE
	without a leading space produces a boundary that does not exist in the original text
	(`'…on a roof. Heis using…'`). This matches lm-eval's behaviour and is kept for
	comparability; correcting it raises HellaSwag 300-item accuracy 32.00% → 33.67%.

	## Training details

	Optimizer AdamW (β 0.9/0.95, wd 0.1, eps 1e-8), LR 3e-4 → 3e-5 cosine, 2% warmup,
	grad clip 1.0, global batch ≈ 524k tokens/step (32 × 2048 × 8 accumulation), bf16 with SDPA,
	`torch.compile`, seed 1337. 4,961 steps over 14h42m on a single NVIDIA DGX Spark (GB10);
	train EMA loss 11.0017 → 3.2274 (−70.7%); peak memory 29.8 GB.

	## Usage

	```python
	from transformers import AutoModelForCausalLM, AutoTokenizer

	tok = AutoTokenizer.from_pretrained("SIZWL/OpenSI-0.3B-Base")
	model = AutoModelForCausalLM.from_pretrained(
	"SIZWL/OpenSI-0.3B-Base", dtype="auto", device_map="auto"
	)

	ids = tok("The capital of France is", return_tensors="pt").to(model.device)
	out = model.generate(**ids, max_new_tokens=20, do_sample=False,
	pad_token_id=tok.pad_token_id)
	print(tok.decode(out[0]))
	```

	This is a base model, not instruction-tuned. It continues text rather than answering
	questions, and may not terminate cleanly on a complete sentence — prefer sampling with a
	repetition penalty for open-ended generation:

	```python
	out = model.generate(**ids, max_new_tokens=64, do_sample=True, temperature=0.8, top_k=40,
	repetition_penalty=1.2, eos_token_id=tok.eos_token_id,
	pad_token_id=tok.pad_token_id)
	```

	## Data sources

	\| Source \| Role \|
	\|---\|---\|
	\| FineWeb \| English web text \|
	\| Wikipedia (zh) \| Chinese encyclopedic \|
	\| TinyStories \| Synthetic simple English \|
	\| Industry Chinese documents \| Chinese domain text \|

	See §4.5 of the project plan for the mix and §4.7 for the verification tooling.

	## License

	MIT.

	## Citation

	```bibtex
	@online{opensi03bbase,
	title = {{OpenSI-0.3B-Base}: A 113.9M Open Base Language Model Pretrained on 2.6B Tokens},
	author = {{OpenSI Project}},
	year = {2026},
	url = {https://huggingface.co/SIZWL/OpenSI-0.3B-Base},
	note = {Exported in Qwen2 format. Actual size 113.9M parameters.}
	}
	```