---
license: creativeml-openrail-m
language:
- en
tags:
- Pytorch
- text-generation
- text-generation-inference
- base-model
---
# Hi, I am a grade-8 middle school developer from China. I trained this 1B parameter model from absolute scratch (PyTorch, custom Transformer) using a single RTX 5080 in my bedroom.

# Current Versions:
- **Freedream_1B-7b-tokens-seen:** This is an early trained model, that only see total of 7Billion tokens, it is a base model that has a moderate performance that can generate fluence text but contains factual error. Don't recommend for code or hard maths reasoning.

------------------------------------------------------

## Reserved Name & Trademark Notice

The name **"Freedream"** is a reserved name and an unregistered trademark of the original author.

### 1. Reserved Name
No derivative work, fork, fine-tune, quantization, or redistribution of this model may use the name "Freedream" or any confusingly similar name (e.g., "Free-Dream", "FreeDream", "Freedream AI", "Freedream-2", "Freedream-X", etc.) without prior written permission from the original author.

### 2. No Trademark License
This license does not grant any rights to use the trademark "Freedream". All trademark rights are expressly reserved by the original author.

### 3. Attribution Requirement
Any use, distribution, or public deployment of this model must:
- Prominently display **"Built with Freedream"** or **"Based on Freedream"**.
- Provide a link to the original model repository.
- Retain this license file in its entirety.

### 4. Enforcement
Violation of the Reserved Name or Trademark clauses will result in a takedown request to the hosting platform (e.g., Hugging Face) and may result in legal action under applicable trademark and contract law.

------------------------------------------------------

# Freedream 1B

## Model Description:
Freedream is a ~1B parameter decoder-only Transformer language model trained from scratch on a single NVIDIA RTX 5080 (16GB). It uses RoPE, SwiGLU, weight-tied embeddings, and a custom BPE tokenizer (32k vocab).

## Architecture:
- Parameters: 977,300,992 (~977M)
- Layers: 24
- Hidden size: 1792
- Attention heads: 16
- Head dimension: 112
- FFN dimension: 4736
- Context length: 4096
- Position encoding: RoPE
- Activation: SwiGLU
- Weight tying: ON

## Training:
- Hardware: 1x RTX 5080 (16GB) + WSL2 (Ubuntu 24.04)
- Precision: BF16
- Optimizer: 8-bit AdamW
- Gradient checkpointing: ON
- Batch size: 1
- Gradient accumulation: 16
- Effective tokens / step: 65,536
- Dataset: ~45B tokens (FineWeb-Edu, Cosmopedia, FineMaths, TinyBrain, etc.)
- Training tokens seen: depends on versions

## Intended Use:
- Research on small language models.
- Fine-tuning for downstream tasks (with proper attribution).
- Educational purposes.

## Limitations:
- Not instruction-tuned by default; may produce incoherent or hallucinated outputs.
- Context window limited to 4096 tokens.
- Knowledge cutoff reflects training data.
- Not suitable for medical, legal, or safety-critical applications.
- Not recommended for hard maths reasoning or code.