YuRiVeRTi/Tensora-Dust-0

Tensora Dust 0

A 1.87M-parameter language model, trained entirely from scratch on a phone-driven Colab T4.

Tensora Dust 0 is the first from-scratch model from TensoraMax Lab: random initialization, no base model, no fine-tuning, its own tokenizer and its own (small) architecture. It is an early checkpoint and a research artifact. It is not a usable assistant, and it is not meant to be one yet.

Dust 0 is the "before" picture. A cleaner, longer run (Tensora Dust 1) is planned.

---

At a glance

Parameters	1,873,152 trainable (Hugging Face shows 2.14M because it also counts four fixed causal-mask buffers, 262,144 values)
Architecture	GPT-2-style decoder-only transformer (custom code)
Layers / heads / width	4 / 4 / 128
Context length	256 tokens
Embeddings	Learned position embeddings; separate (untied) LM head
Tokenizer	Byte-level BPE, vocab 4096
Training data	WikiText-2 (non-raw version)
Training	400 steps (250, then 150 resumed), Google Colab T4
File size	about 8.5 MB (model.safetensors, float32)
Full validation loss	5.5235 cross-entropy (perplexity about 250)

---

Files

File	What it is
model.safetensors	Model weights
config.json	Model configuration
tokenizer.json	Corrected tokenizer (see below)
modeling_tensora.py	Model code (GPT2CausalLM) and a small demo

This is a custom architecture, so it does not load with transformers.AutoModel. Use the code in modeling_tensora.py.

---

Quick start

import json, sys, torch
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
from tokenizers import Tokenizer

path = snapshot_download("YuRiVeRTi/Tensora-Dust-0")
sys.path.insert(0, path)
from modeling_tensora import GPT2CausalLM

cfg = json.load(open(f"{path}/config.json"))
model = GPT2CausalLM(cfg)
model.load_state_dict(load_file(f"{path}/model.safetensors"), strict=True)
model.eval()

tok = Tokenizer.from_file(f"{path}/tokenizer.json")
ids = torch.tensor([tok.encode("The history of").ids])

with torch.no_grad():
    for _ in range(60):
        logits, _ = model(ids[:, -256:])
        probs = torch.softmax(logits[0, -1] / 0.8, dim=-1)   # temperature 0.8
        nxt = torch.multinomial(probs, 1)
        ids = torch.cat([ids, nxt[None]], dim=1)
        if nxt.item() == 1:                                  # <|endoftext|>
            break

print(tok.decode(ids[0].tolist(), skip_special_tokens=False))

---

What it actually writes

Real samples (temperature 0.8, top-k 40), unedited:

The history of the first <unk> the two <unk> , and <unk> , <unk> <unk> .<|endoftext|>

The city was the sames the Nan were of the <unk> , his <unk> ( " ) .<|endoftext|>

It has picked up the shape of WikiText: spaced punctuation, quotes and brackets, <unk> placeholders, and sentence endings. It has not learned English. That is exactly what you should expect from 1.87M parameters and 400 steps.

---

Training details

• Data: WikiText-2 train split, non-raw version (it contains literal <unk> placeholders). Source: Merity et al., Pointer Sentinel Mixture Models, 2016. License: CC BY-SA 3.0.
• Compute: one Google Colab T4, driven from a phone.
• Steps: 400 total, in two runs (250, then 150 resumed from a checkpoint).
• Loss: train 8.4509 to 5.8714; validation 8.4329 to 5.5294 (training report).
• Full validation re-check: 1,239 non-overlapping windows of up to 256 tokens over the whole validation stream (317,134 tokens, 315,895 target tokens), token-weighted mean cross-entropy 5.5235.

Why you will see so much <unk>

<unk> appears about 54,625 times in the WikiText-2 text. The tokenizer splits it into three ordinary pieces (<, unk, >), so the model learned it as a very common pattern and repeats it at low temperature. Dust 1 will be trained on text with these placeholders removed.

---

Tokenizer note

The special tokens are:

• ID 0: <|padding|>
• ID 1: <|endoftext|>
• ID 2: <|user|>
• ID 3: <|assistant|>

The original tokenizer had a typo at ID 3 (<|assistant||). The tokenizer.json in this repository is the corrected copy: only the token's name changed, and IDs and vocabulary size are unchanged. The model never used ID 3 in training, so the weights are unaffected.

---

Limitations

• Heavily undertrained. Expect wiki-style fragments, malformed words and many <unk> pieces.
• Not conversational. The chat markers never appeared in its training data. It does not follow instructions or answer questions.
• Not a source of facts. Do not rely on anything it says.
• Tiny context and vocab. 256 tokens and 4096 pieces limit what any follow-up of this exact architecture can do.

---

Intended use

Studying and tinkering with small-model training from scratch: loading a real tiny checkpoint, comparing it against later versions, or learning from the model code. Not for production use.

---

The Tensora family

Model	Status
Tensora Dust 0	Released (this model)
Tensora Dust 1	Planned: cleaner, longer training run from random initialization
Tensora Genesis	Planned: larger foundation model
Nebula, Pulser, Cosmic, Singularity	Planned

Plans can change. Only Dust 0 exists today.

---

License

Apache-2.0 for the model code and weights. The training data (WikiText-2) is CC BY-SA 3.0, so please credit it if you build on this.

---

Made by TensoraMax Lab.