Velari-Level2-Qwen3-4B — Intermediate Checkpoint
⚠️ This is an early/intermediate checkpoint. The final, recommended model is Aslam-13/aelora-qwen3-4b — use that one unless you specifically want to study how the project evolved.
This checkpoint covers only Level 1 (Velari language) + Level 2 (Thex-Kron base-8 math) of a larger 5-level project. It does not contain logic rules (Vel-Rith), advanced grammar (comparatives, superlatives, imperatives, possession), or world/lore knowledge — those were added in subsequent training runs.
Why a synthetic world? Benchmarks leak into pre-training. By fine-tuning on a domain that provably cannot exist in any pre-training corpus, every correct answer is unambiguous evidence of learning — not retrieval from prior knowledge.
It's kept public because the iteration history is part of the story: training in stages let me observe catastrophic forgetting in real time and validate that combining Level 1 + Level 2 data in a single run prevented the model from "losing" Velari vocabulary while learning math.
- 🧪 Full project (code, datasets, RAG comparison, writeup): github.com/Aslam-13/Fine_tuning_RAG
- 🏁 Final model: Aslam-13/aelora-qwen3-4b
- 📓 Training notebook: kaggle.com/code/syed13/fine-tune
What this checkpoint knows
| Domain | Coverage |
|---|---|
| Velari — vocabulary | 30-word lexicon (nouns, verbs, pronouns, adjectives) |
| Velari — grammar (basic) | Plurals (-an), past tense (ta-), negation (ne), basic SVO sentences |
| Thex-Kron math | Base-8 numerals (nul, ek, doi, tri … hep, ek-nul …), addition with carry, basic multiplication |
Does NOT know: Vel-Rith logic rules, comparatives/superlatives, imperatives, possession, world/society lore, multi-step word problems. For those use the final model.
Why a base-8 system?
The base model (Qwen3-4B) "knows" base-10 from pre-training extremely well. Teaching it that 5 + 3 = 10 (in Aeloran base-8) directly contradicts that prior. If this checkpoint correctly answers fen vor tri = ek-nul (5 + 3 = 10₈), it's evidence that LoRA fine-tuning successfully overrode a deeply-baked behavior — not just memorized a surface pattern.
Training details (this checkpoint)
| Base model | unsloth/qwen3-4b-unsloth-bnb-4bit |
| Method | LoRA (4-bit) via Unsloth + TRL SFTTrainer |
| Dataset | Combined Level 1 (language) + Level 2 (math) JSONL |
| License | Apache 2.0 |
Full hyperparameters and training logs are in the GitHub repo.
Quick start
Option A — with Unsloth (fastest, requires GPU)
from unsloth import FastLanguageModel
model, tokenizer = FastLanguageModel.from_pretrained(
model_name = "Aslam-13/velari-level2-qwen3-4b",
max_seq_length = 2048,
load_in_4bit = True,
)
FastLanguageModel.for_inference(model)
prompt = "What is fen vor tri in Aeloran math?"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
out = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(out[0], skip_special_tokens=True))
Option B — with plain transformers
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model_id = "Aslam-13/velari-level2-qwen3-4b"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
device_map="auto",
)
prompt = "What is fen vor tri in Aeloran math?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(out[0], skip_special_tokens=True))
Expected output (approximate): "ek-nul" (5 + 3 = 8 in base-10 → 10 in base-8 → ek-nul). Exact phrasing may vary.
Limitations
- Intermediate checkpoint, superseded by aelora-qwen3-4b.
- No logic, no advanced grammar, no world knowledge.
- Single-domain research artifact — not a general-purpose model.
Trained with Unsloth.
