Update README.md
Browse files
README.md
CHANGED
|
@@ -3,7 +3,7 @@ license: creativeml-openrail-m
|
|
| 3 |
language:
|
| 4 |
- en
|
| 5 |
---
|
| 6 |
-
#
|
| 7 |
- **Freedream_1B-7b-tokens-seen:** This is an early trained model, that only see total of 7Billion tokens, it is a base model that has a moderate performance. Don't recommend for code or hard maths reasoning.
|
| 8 |
|
| 9 |
------------------------------------------------------
|
|
@@ -31,10 +31,10 @@ Violation of the Reserved Name or Trademark clauses will result in a takedown re
|
|
| 31 |
|
| 32 |
# Freedream 1B
|
| 33 |
|
| 34 |
-
## Model Description
|
| 35 |
Freedream is a ~1B parameter decoder-only Transformer language model trained from scratch on a single NVIDIA RTX 5080 (16GB). It uses RoPE, SwiGLU, weight-tied embeddings, and a custom BPE tokenizer (32k vocab).
|
| 36 |
|
| 37 |
-
## Architecture
|
| 38 |
- Parameters: 977,300,992 (~977M)
|
| 39 |
- Layers: 24
|
| 40 |
- Hidden size: 1792
|
|
@@ -46,7 +46,7 @@ Freedream is a ~1B parameter decoder-only Transformer language model trained fro
|
|
| 46 |
- Activation: SwiGLU
|
| 47 |
- Weight tying: ON
|
| 48 |
|
| 49 |
-
## Training
|
| 50 |
- Hardware: 1x RTX 5080 (16GB) + WSL2 (Ubuntu 24.04)
|
| 51 |
- Precision: BF16
|
| 52 |
- Optimizer: 8-bit AdamW
|
|
@@ -57,12 +57,12 @@ Freedream is a ~1B parameter decoder-only Transformer language model trained fro
|
|
| 57 |
- Dataset: ~45B tokens (FineWeb-Edu, Cosmopedia, FineMaths, TinyBrain, etc.)
|
| 58 |
- Training tokens seen: depends on versions
|
| 59 |
|
| 60 |
-
## Intended Use
|
| 61 |
- Research on small language models.
|
| 62 |
- Fine-tuning for downstream tasks (with proper attribution).
|
| 63 |
- Educational purposes.
|
| 64 |
|
| 65 |
-
## Limitations
|
| 66 |
- Not instruction-tuned by default; may produce incoherent or hallucinated outputs.
|
| 67 |
- Context window limited to 4096 tokens.
|
| 68 |
- Knowledge cutoff reflects training data.
|
|
|
|
| 3 |
language:
|
| 4 |
- en
|
| 5 |
---
|
| 6 |
+
# Current Versions:
|
| 7 |
- **Freedream_1B-7b-tokens-seen:** This is an early trained model, that only see total of 7Billion tokens, it is a base model that has a moderate performance. Don't recommend for code or hard maths reasoning.
|
| 8 |
|
| 9 |
------------------------------------------------------
|
|
|
|
| 31 |
|
| 32 |
# Freedream 1B
|
| 33 |
|
| 34 |
+
## Model Description:
|
| 35 |
Freedream is a ~1B parameter decoder-only Transformer language model trained from scratch on a single NVIDIA RTX 5080 (16GB). It uses RoPE, SwiGLU, weight-tied embeddings, and a custom BPE tokenizer (32k vocab).
|
| 36 |
|
| 37 |
+
## Architecture:
|
| 38 |
- Parameters: 977,300,992 (~977M)
|
| 39 |
- Layers: 24
|
| 40 |
- Hidden size: 1792
|
|
|
|
| 46 |
- Activation: SwiGLU
|
| 47 |
- Weight tying: ON
|
| 48 |
|
| 49 |
+
## Training:
|
| 50 |
- Hardware: 1x RTX 5080 (16GB) + WSL2 (Ubuntu 24.04)
|
| 51 |
- Precision: BF16
|
| 52 |
- Optimizer: 8-bit AdamW
|
|
|
|
| 57 |
- Dataset: ~45B tokens (FineWeb-Edu, Cosmopedia, FineMaths, TinyBrain, etc.)
|
| 58 |
- Training tokens seen: depends on versions
|
| 59 |
|
| 60 |
+
## Intended Use:
|
| 61 |
- Research on small language models.
|
| 62 |
- Fine-tuning for downstream tasks (with proper attribution).
|
| 63 |
- Educational purposes.
|
| 64 |
|
| 65 |
+
## Limitations:
|
| 66 |
- Not instruction-tuned by default; may produce incoherent or hallucinated outputs.
|
| 67 |
- Context window limited to 4096 tokens.
|
| 68 |
- Knowledge cutoff reflects training data.
|