language: - en license: apache-2.0 tags: - text-generation - from-scratch - small-language-model - transformer pipeline_tag: text-generation
CorX1.5
A ~158M parameter language model, built entirely from scratch by a single developer.
Architecture, tokenizer, pretraining pipeline, and fine-tuning — all written and trained solo, from the ground up, on a single GPU. No fine-tune of an existing base model.
Developer: Nathan · Origin: Jamaica · Hub: Sigmandndnns/CorX1.5-160M
What CorX1.5 is
CorX1.5 is a small, custom-architecture transformer trained in stages:
- Pretraining — ~3 billion tokens across 25 open datasets spanning code, math, reasoning, conversation, and general web text, with a held-out validation split and early stopping to catch overfitting.
- Distillation — fine-tuned on responses generated by a larger teacher model (Qwen2.5-Instruct), with automated quality filtering to keep only well-formatted, non-repetitive, correctly-verified answers.
- Reasoning training — trained to show step-by-step work in
<think>...</think>blocks before answering, including verified worked-step arithmetic and debugging exercises with a known, injected bug to check against.
Architecture
A custom decoder-only transformer:
- RoPE positional encoding, SwiGLU feed-forward, RMSNorm
- Role embeddings — a learned signal marking every token as user, assistant, or raw text, so the model always knows whose turn it is (most models infer this only from special tokens)
- Assistant-only loss masking — trained only to predict the tokens it should generate, not the user's side of the conversation
- 157.8M parameters · 10 layers · 1024 token context · 32,768 token custom BPE vocabulary
What it's good at
- Holding a consistent identity and conversational style
- Short, direct answers to greetings and common everyday questions
- Declining requests it genuinely can't fulfill (no internet, no memory between chats, no tool access) instead of fabricating an answer
- Small worked arithmetic, shown step by step
- Recognizing and explaining common Python bugs in a structured format
What it's not
CorX1.5 is a small model trained on a small fraction of the data used by comparable open models (for reference, models in a similar parameter range from major labs are typically trained on trillions of tokens; CorX1.5 was trained on billions). It should not be relied on for:
- Broad factual knowledge or general reasoning outside its training patterns
- Complex or multi-step math beyond basic arithmetic
- Production or safety-critical use of any kind
This is an independent research and learning project, not a competitor to frontier or lab-scale models. It's shared in that spirit — small, transparent, and built to understand every part of how it works.
language: - en license: apache-2.0 tags: - text-generation - from-scratch - small-language-model - transformer - work-in-progress pipeline_tag: text-generation
CorX1.5
A ~158M parameter language model, built completely from scratch by one person.
Every part of this — the architecture, the tokenizer, the training pipeline, the data curation, the fine-tuning — was designed and trained solo, on a single GPU, from zero. Not a fine-tune of an existing base model. Built line by line, debugged crash by crash, from the ground up.
Developer: Nathan · Origin: Jamaica · Hub: Sigmandndnns/CorX1.5-160M
About this project
I'm a solo developer from Jamaica, and CorX1.5 is my from-scratch attempt at building a language model — not adapting someone else's, but writing the architecture myself and training it up from random weights. Every checkpoint here represents real hours spent debugging data pipelines, fixing overfitting, and learning how these systems actually work under the hood.
This model is actively being developed. I'm adding new training data on an ongoing basis — new topics, new examples, patches for whatever gaps I find when I test it — rather than treating this as a one-time release. Expect updated checkpoints over time as the training continues. Check the commit history on this repo for the latest. I Am 14 year's old for years I have shared a dream to build something great my own ai I wanted it to past the benchmarks, since I was 12 I was expiramenting with api's I made a chatbot using gemini and its api and customized it by myself. but that wasn't the same as building from the ground up by age 13 I began finetuning but that failed I was using slow Gpus and traing took soo long I ended up giving up on that but today my dream came through and I was able to made this and say I made it from scratch from the ground up and im really proud at what I achieved so im opensourcing it for people like you to download it :) currently the thinking isn't a option yet im still training it but at 2:30Pm it will be finished the base model
How it was built
- Pretraining — billions of tokens across dozens of open datasets spanning code, math, reasoning, conversation, and general text, with a held-out validation split and early stopping to catch overfitting before it happened.
- Distillation — fine-tuned on responses from a larger teacher model, filtered automatically to keep only well-formatted, non-repetitive, verified answers.
- Reasoning training — trained to show its work in
<think>...</think>blocks before answering, including verified step-by-step arithmetic and debugging exercises checked against a known, injected bug. - Ongoing daily additions — new topics and targeted fixes added continuously as I find gaps through testing.
Architecture
A custom decoder-only transformer, not copied from an existing model family:
- RoPE positional encoding, SwiGLU feed-forward, RMSNorm
- Role embeddings — every token is tagged as user, assistant, or raw text at the architecture level, so the model always knows whose turn it is
- Assistant-only loss masking — trained only on the tokens it should generate, not the user's side of the conversation
- ~158M parameters · 10 layers · 1024 token context · 32,768 token custom BPE vocabulary
What it's good at
- Holding a consistent identity and conversational style
- Short, direct answers to greetings and everyday questions
- Declining requests it genuinely can't fulfill (no internet, no memory between chats, no tools) instead of making something up
- Small worked arithmetic, shown step by step
- Recognizing and explaining common Python bugs in a structured format
What it's not — yet
CorX1.5 is a small model trained on a small fraction of the data behind comparable-sized models from major labs, which are typically trained on trillions of tokens against this one's billions. It's not built for:
- Broad factual knowledge or general reasoning outside what it's specifically been trained on
- Complex or multi-step math beyond basic arithmetic
- Production or safety-critical use
This is an independent, ongoing learning project — not a claim to compete with frontier models. It's shared openly because the process of building it, one training run at a time, is the point.
Apache 2.0.
Safety
CorX1.5 includes a hardcoded response layer for messages indicating self-harm or crisis, which returns a fixed set of crisis resources rather than a generated response. This is implemented in code, not left to the model's judgment.
Usage
CorX1.5 uses a custom architecture and is not compatible with AutoModel.from_pretrained() out of the box. Load it with the model definition included in this repo (modeling_corx.py / inference script) alongside the checkpoint and tokenizer files.
License
contact
lkk89002@gmail.com --my main Gmail
--my school (maudleod high school )
--website
- Downloads last month
- 427