HelionX Base 300M model card

README.md

@@ -1,42 +1,50 @@
 ---
+language: en
 license: apache-2.0
 tags:
-- base-model
-- language-model
+- causal-lm
 - pretraining
 - research
+- from-scratch
 ---
 
-# Base Language Model – 300M Parameters (Checkpoint)
+# HelionX Base 300M
 
-## Overview
-This repository contains a **from-scratch trained base language model checkpoint**.
-The model is trained using causal language modeling (next-token prediction).
-
-This is **NOT** an instruction-tuned or chat model.
+HelionX Base 300M is a **from-scratch pretrained causal language model** developed as part of the HelionX research initiative.
 
 ## Model Details
-- Parameters: ~300M
-- Architecture: Decoder-only Transformer
-- Training Objective: Causal Language Modeling
-- Framework: PyTorch
-- Tokenizer: GPT-2 tokenizer
-- Checkpoint Stage: ~50M tokens
 
-## Training Status
-This checkpoint is an **intermediate safe checkpoint**.
-Training is intended to continue toward ~300M tokens.
+- **Architecture:** Decoder-only Transformer
+- **Parameters:** ~300M
+- **Layers:** 22
+- **Hidden size:** 896
+- **Attention heads:** 14
+- **Context length:** 2048 tokens
+- **Tokenizer:** GPT-2 BPE (50257 vocab)
+- **Precision:** FP16 training
+- **Training tokens:** 300M tokens
+- **Training data:** OpenWebText (streamed)
+
+## Training
+
+The model was trained incrementally and resumed from intermediate checkpoints, completing a full **300M-token pretraining run** using mixed-precision training and gradient checkpointing.
+
+Training infrastructure included:
+- Modal (A100 40GB)
+- PyTorch
+- Hugging Face tooling
 
 ## Intended Use
+
 - Research
-- Further pretraining
-- Instruction tuning
-- Systems integration (non-production)
+- Continued pretraining
+- Fine-tuning
+- Architecture experiments
 
 ## Limitations
-- Not instruction-following
-- Not aligned for safety
-- Not suitable for direct deployment
 
-## Author
-Independent research project.
+This is a base model and **not instruction-tuned**. Outputs may be incoherent or unsafe without further alignment.
+
+## License
+
+Apache 2.0