README.md

---
datasets:
  - HuggingFaceFW/fineweb-edu
  - HuggingFaceFW/finepdfs-edu
language:
  - en
pipeline_tag: text-generation
library_name: pytorch
tags:
  - base-model
  - causal-lm
  - text-generation
  - multiscreen
  - triton
  - litgpt
---

# ObsidianSmall-Base — Multiscreen 8.7M

**ObsidianSmall-Base** is an **8.7M-parameter English base language model** pretrained from scratch on **3,999,989,760 tokens**.

It is the first public checkpoint in this model family built entirely from **Multiscreen blocks**.

> This is a base completion model, not an instruction-tuned or chat-aligned assistant.

## Model details

| Property | Value |
|---|---:|
| Architecture | Pure Multiscreen |
| Parameters | 8,716,506 |
| Blocks | 9 |
| Hidden width | 256 |
| Query heads | 8 |
| Head size | 32 |
| Key / value dimensions | 16 / 128 |
| Query chunk / window | 128 / 256 |
| MLP / FFN | None |
| Context length | 1,024 |
| Vocabulary size | 4,096 |
| Training tokens | 3,999,989,760 |
| Validation loss / PPL | 2.592 / 13.359 |

Multiscreen replaces conventional self-attention and MLP/FFN sublayers with its own sequence-mixing blocks.

## Benchmarks

Zero-shot task results were measured with `lm-evaluation-harness`.

| Benchmark | Accuracy | Normalized |
|---|---:|---:|
| BLiMP | 71.891% | — |
| HellaSwag | 26.957% | 27.415% |
| ARC Easy | 33.712% | 33.291% |
| ARC Challenge | 17.833% | 21.843% |
| PIQA | 56.583% | 55.114% |
| SciQ | 70.700% | 60.000% |
| ArithMark-2.0 | 26.160% | — |
| ArithMark-3.0 | **28.800%** | — |

### WikiText-2 perplexity

Full WikiText-2 evaluation using rolling log-likelihood with a maximum context length of 1,024 tokens.

| Metric | Result |
|---|---:|
| Word perplexity | 125.057 |
| Byte perplexity | **2.467** |
| Bits per byte | 1.303 |

## Install

```bash
git lfs install
git clone https://huggingface.co/Dream-W/ObsidianSmall-Base
cd ObsidianSmall-Base

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
```

For NVIDIA CUDA with Triton:

```bash
python -m pip install -r requirements-gpu.txt
```

For evaluation:

```bash
python -m pip install -r requirements-eval.txt
```

The repository includes the required custom PyTorch/LitGPT runtime. It is not currently compatible with `transformers.AutoModelForCausalLM`.

## Generate text

```bash
CUDA_VISIBLE_DEVICES=0 \
MULTISCREEN_BACKEND=triton \
PYTHONPATH="$PWD/runtime" \
python -u generate.py \
  --backend triton \
  --device cuda \
  --prompt "Once upon a time" \
  --max-new-tokens 128 \
  --temperature 0.8 \
  --top-k 40
```

Interactive completion:

```bash
CUDA_VISIBLE_DEVICES=0 \
MULTISCREEN_BACKEND=triton \
PYTHONPATH="$PWD/runtime" \
python -u generate.py \
  --backend triton \
  --device cuda \
  --interactive
```

CPU execution is available with `MULTISCREEN_BACKEND=torch`, `--backend torch`, and `--device cpu`.

## Evaluate

The bundled evaluator accepts any compatible `lm-evaluation-harness` task through `--tasks`.

```bash
MODEL="$PWD"
mkdir -p "$MODEL/results"

CUDA_VISIBLE_DEVICES=0 \
MULTISCREEN_BACKEND=triton \
TOKENIZERS_PARALLELISM=false \
PYTHONPATH="$MODEL/runtime:$MODEL/evaluation" \
python -u "$MODEL/evaluation/lm_eval_obsidian_official.py" \
  --checkpoint-dir "$MODEL" \
  --tokenizer-dir "$MODEL" \
  --tasks hellaswag \
  --batch-size 32 \
  --device cuda \
  --max-length 1024 \
  --progress-every 100 \
  --output "$MODEL/results/hellaswag.json"
```

Multiple tasks:

```bash
--tasks "hellaswag,arc_easy,arc_challenge,piqa,sciq,blimp"
```

Custom continuation-style multiple-choice datasets can be evaluated with `evaluation/custom_mcq.py`.

```bash
CUDA_VISIBLE_DEVICES=0 \
MULTISCREEN_BACKEND=triton \
PYTHONPATH="$PWD/runtime" \
python -u evaluation/custom_mcq.py \
  --model-dir . \
  --hf-dataset AxiomicLabs/Arithmark-3.0 \
  --split train \
  --context-field ctx \
  --choices-field endings \
  --label-field label \
  --group-by difficulty topic \
  --backend triton \
  --device cuda \
  --output results/arithmark-3.json
```

## Backends and decoding

| Backend | Setting | Use |
|---|---|---|
| Automatic | `MULTISCREEN_BACKEND=auto` | Select a compatible backend |
| Triton | `MULTISCREEN_BACKEND=triton` | Optimized NVIDIA CUDA execution |
| PyTorch | `MULTISCREEN_BACKEND=torch` | CPU and compatibility fallback |

Incremental Multiscreen KV-cache decoding is not implemented. Generation uses full-sequence decoding and processes the current context again for every generated token.

## Training data

The pretraining curriculum included FineWeb-Edu, FinePDFs-Edu, DCLM, FineWiki, NVIDIA Common Crawl Code, and NVIDIA Common Crawl Math.

## Limitations

The model has only 8.7M parameters and may produce repetitive, inaccurate, or incoherent text. It is English-focused, has no instruction-following or safety alignment, and was trained for a maximum context length of 1,024 tokens.