|
|
--- |
|
|
license: apache-2.0 |
|
|
language: en |
|
|
tags: |
|
|
- causal-lm |
|
|
- from-scratch |
|
|
- transformer |
|
|
- tiny-stories |
|
|
- pytorch |
|
|
- custom-architecture |
|
|
- text-generation |
|
|
datasets: |
|
|
- fhswf/TinyStoriesV2_cleaned |
|
|
--- |
|
|
|
|
|
# TinyWay-1.1.0 |
|
|
|
|
|
**TinyWay-1.1.0** is a lightweight **decoder-only Transformer language model** trained **from scratch** on limited compute. |
|
|
The project demonstrates that meaningful language modeling behavior can emerge from modest-scale models trained in constrained environments such as Kaggle. |
|
|
|
|
|
> **Core idea:** *Understanding LLM training mechanics end-to-end by building, training, debugging, and deploying a Transformer LM without relying on pretrained weights.* |
|
|
|
|
|
--- |
|
|
|
|
|
## Model Details |
|
|
|
|
|
* **Architecture:** Decoder-only Transformer (GPT-style) |
|
|
* **Parameters:** ~83M |
|
|
* **Layers:** 10 Transformer blocks |
|
|
* **Hidden size:** 512 |
|
|
* **Attention heads:** 8 |
|
|
* **Context length:** 256 tokens |
|
|
* **Activation:** GELU |
|
|
* **Normalization:** Pre-LayerNorm |
|
|
* **Weight tying:** Token embedding ↔ LM head |
|
|
* **Precision during training:** FP16 (AMP) |
|
|
|
|
|
--- |
|
|
|
|
|
## Training |
|
|
|
|
|
### Dataset |
|
|
|
|
|
* **TinyStoriesV2 (cleaned)** |
|
|
* Natural language short stories designed for training small language models |
|
|
|
|
|
### Tokenization |
|
|
|
|
|
* GPT-2 BPE tokenizer |
|
|
* Vocabulary size: 50,257 |
|
|
|
|
|
### Training Setup |
|
|
|
|
|
* Optimizer: AdamW |
|
|
* Learning rate: tuned for stable convergence |
|
|
* Gradient accumulation: enabled |
|
|
* Gradient clipping: enabled |
|
|
* Mixed precision training (AMP) |
|
|
* Training performed entirely on **Kaggle GPU environment (12-hour sessions)** |
|
|
|
|
|
### Checkpoints |
|
|
|
|
|
Models were saved at multiple training steps (5k → 30k). |
|
|
**TinyWay-1.1.0** corresponds to the **~25k step checkpoint**, which showed the best balance of fluency and stability. |
|
|
|
|
|
--- |
|
|
|
|
|
## Example Usage |
|
|
|
|
|
```python |
|
|
from transformers import AutoConfig, AutoTokenizer, AutoModelForCausalLM |
|
|
|
|
|
model_id = "NNEngine/TinyWay-1.1.0" |
|
|
|
|
|
config = AutoConfig.from_pretrained(model_id, trust_remote_code=True) |
|
|
tok = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True) |
|
|
mdl = AutoModelForCausalLM.from_pretrained(model_id, config=config, trust_remote_code=True) |
|
|
|
|
|
out = mdl.generate( |
|
|
**tok( |
|
|
"Once upon a time", |
|
|
return_tensors="pt" |
|
|
).to(mdl.device), |
|
|
|
|
|
max_new_tokens=200, # force length |
|
|
do_sample=True, # sampling, not greedy |
|
|
temperature=0.8, |
|
|
top_k=50, |
|
|
repetition_penalty=1.2, |
|
|
|
|
|
eos_token_id=None, # 🔥 disable EOS stopping |
|
|
pad_token_id=tok.eos_token_id |
|
|
) |
|
|
|
|
|
print(tok.decode(out[0], skip_special_tokens=True)) |
|
|
``` |
|
|
|
|
|
--- |
|
|
|
|
|
## Sample Output |
|
|
|
|
|
> *Once upon a time, there was a little girl named Lily. She loved to play with her toys and explore the park near her home. One day, she found a shiny red ball hidden behind a tree…* |
|
|
|
|
|
(Outputs vary due to sampling.) |
|
|
|
|
|
--- |
|
|
|
|
|
## Intended Use |
|
|
|
|
|
* Educational purposes |
|
|
* Research on small-scale language models |
|
|
* Understanding Transformer internals |
|
|
* Studying training dynamics under compute constraints |
|
|
|
|
|
--- |
|
|
|
|
|
## Limitations |
|
|
|
|
|
* Not instruction-tuned |
|
|
* Not aligned for factual accuracy or safety |
|
|
* May produce repetitive or incoherent text at times |
|
|
* Trained on a limited dataset |
|
|
|
|
|
This model is **not intended for production use** or sensitive applications. |
|
|
|
|
|
--- |
|
|
|
|
|
## Ethical Considerations |
|
|
|
|
|
* The model may generate fictional or incorrect information |
|
|
* No explicit safety or content filtering was applied |
|
|
* Users should apply downstream safeguards if deploying |
|
|
|
|
|
--- |
|
|
|
|
|
## Citation |
|
|
|
|
|
If you use this model in academic or technical work, please cite: |
|
|
|
|
|
```bibtex |
|
|
@misc{sharma2025tinyway, |
|
|
title={TinyWay: Training Decoder-Only Language Models from Scratch on Limited Compute}, |
|
|
author={Shivam Sharma}, |
|
|
year={2025}, |
|
|
} |
|
|
``` |
|
|
|
|
|
--- |
|
|
|
|
|
## Author |
|
|
|
|
|
**Shivam Sharma** |
|
|
B.Tech in Computer Science and Engineering (AIML) |
|
|
ITM Gwalior, India |
|
|
|
|
|
--- |
|
|
|
|
|
## Acknowledgements |
|
|
|
|
|
* Hugging Face Transformers |
|
|
* Kaggle GPU resources |
|
|
* Open research community for open-source inspiration |
