| --- |
| license: mit |
| library_name: pytorch |
| tags: |
| - tinyllm |
| - character-language-model |
| - gpt |
| - cpu |
| - educational |
| - pytorch |
| pipeline_tag: text-generation |
| --- |
| |
| # tinyllm-cpu-char |
|
|
| A tiny CPU-trained character-level GPT-style language model created as an educational end-to-end LLM experiment. |
|
|
| This is **not** a general chatbot. It is a deliberately small model trained to demonstrate the full loop: |
|
|
| 1. build a vocabulary |
| 2. train a causal transformer |
| 3. save a checkpoint |
| 4. generate text from learned character patterns |
|
|
| ## Safety / provenance rule |
|
|
| This repo contains checkpoints created locally by us. We did **not** pull external model weights or external quote datasets into the training machine for the philosophy experiments. The philosophy corpus was extracted from existing local workspace philosophy book material, and a provenance manifest is included at `data/philosophy_quotes_local_manifest.json`. |
|
|
| ## Checkpoints |
|
|
| ### Original toy corpus checkpoint |
|
|
| - Checkpoint: `checkpoints/tinyllm_overfit_3k.pt` |
| - Architecture: tiny GPT-style causal transformer |
| - Tokenization: character-level |
| - Parameters: 106,688 |
| - Training device: CPU |
| - Dataset: tiny included toy corpus, `data/tiny_corpus.txt` |
| - Training steps: 3,000 |
| - Final loss: train `0.1903`, val `3.7094` |
|
|
| ### Local philosophy quote checkpoints |
|
|
| These were trained from local-only quote-style corpora derived from existing workspace philosophy files. They are intentionally tiny and educational. |
|
|
| | Checkpoint | Corpus | Params | Final loss | Notes | |
| |---|---:|---:|---:|---| |
| | `checkpoints/tinyllm_philosophy_quotes_local_3k.pt` | 1200 lines | 111,040 | train `1.9311`, val `1.9449` | Learns broad philosophy texture; not overfit. | |
| | `checkpoints/tinyllm_philosophy_quotes_tiny200_4k.pt` | 200 lines | 110,720 | train `1.5459`, val `1.8426` | Stronger style learning; still not memorized. | |
| | `checkpoints/tinyllm_philosophy_quotes_tiny50_3k.pt` | 50 lines | 107,840 | train `0.3808`, val `2.5139` | Best overfit-ish quote-style checkpoint. | |
|
|
| ## Quick inference |
|
|
| ```bash |
| pip install torch huggingface_hub |
| python hf_infer.py --repo-id Vishwas1/tinyllm-cpu-char \ |
| --filename checkpoints/tinyllm_philosophy_quotes_tiny50_3k.pt \ |
| --prompt "Wisdom" --tokens 300 |
| ``` |
|
|
| For local inference after cloning: |
|
|
| ```bash |
| python sample.py --ckpt checkpoints/tinyllm_philosophy_quotes_tiny50_3k.pt \ |
| --prompt "The soul" --tokens 300 |
| ``` |
|
|
| ## Training examples |
|
|
| Original toy overfit: |
|
|
| ```bash |
| python train.py --steps 3000 --eval-interval 500 --eval-iters 10 \ |
| --batch-size 16 --n-embd 64 --block-size 64 \ |
| --out checkpoints/tinyllm_overfit_3k.pt |
| ``` |
|
|
| Local philosophy tiny50 overfit-ish run: |
|
|
| ```bash |
| python train.py --data data/philosophy_quotes_local_tiny50.txt \ |
| --steps 3000 --eval-interval 500 --eval-iters 10 \ |
| --batch-size 16 --n-embd 64 --block-size 64 --lr 0.001 \ |
| --out checkpoints/tinyllm_philosophy_quotes_tiny50_3k.pt |
| ``` |
|
|
| ## Limitations |
|
|
| - Character-level only. |
| - Generates memorized/stylized snippets, not reliable knowledge. |
| - Weird spelling and malformed words are expected. |
| - Intended for learning and experimentation. |
|
|
