File size: 3,096 Bytes
170658b 79d1a20 170658b 79d1a20 170658b 79d1a20 170658b 79d1a20 170658b 79d1a20 170658b 79d1a20 170658b 79d1a20 170658b 79d1a20 170658b 79d1a20 170658b 79d1a20 170658b 79d1a20 170658b 79d1a20 170658b 79d1a20 170658b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 | ---
license: mit
library_name: pytorch
tags:
- tinyllm
- character-language-model
- gpt
- cpu
- educational
- pytorch
pipeline_tag: text-generation
---
# tinyllm-cpu-char
A tiny CPU-trained character-level GPT-style language model created as an educational end-to-end LLM experiment.
This is **not** a general chatbot. It is a deliberately small model trained to demonstrate the full loop:
1. build a vocabulary
2. train a causal transformer
3. save a checkpoint
4. generate text from learned character patterns
## Safety / provenance rule
This repo contains checkpoints created locally by us. We did **not** pull external model weights or external quote datasets into the training machine for the philosophy experiments. The philosophy corpus was extracted from existing local workspace philosophy book material, and a provenance manifest is included at `data/philosophy_quotes_local_manifest.json`.
## Checkpoints
### Original toy corpus checkpoint
- Checkpoint: `checkpoints/tinyllm_overfit_3k.pt`
- Architecture: tiny GPT-style causal transformer
- Tokenization: character-level
- Parameters: 106,688
- Training device: CPU
- Dataset: tiny included toy corpus, `data/tiny_corpus.txt`
- Training steps: 3,000
- Final loss: train `0.1903`, val `3.7094`
### Local philosophy quote checkpoints
These were trained from local-only quote-style corpora derived from existing workspace philosophy files. They are intentionally tiny and educational.
| Checkpoint | Corpus | Params | Final loss | Notes |
|---|---:|---:|---:|---|
| `checkpoints/tinyllm_philosophy_quotes_local_3k.pt` | 1200 lines | 111,040 | train `1.9311`, val `1.9449` | Learns broad philosophy texture; not overfit. |
| `checkpoints/tinyllm_philosophy_quotes_tiny200_4k.pt` | 200 lines | 110,720 | train `1.5459`, val `1.8426` | Stronger style learning; still not memorized. |
| `checkpoints/tinyllm_philosophy_quotes_tiny50_3k.pt` | 50 lines | 107,840 | train `0.3808`, val `2.5139` | Best overfit-ish quote-style checkpoint. |
## Quick inference
```bash
pip install torch huggingface_hub
python hf_infer.py --repo-id Vishwas1/tinyllm-cpu-char \
--filename checkpoints/tinyllm_philosophy_quotes_tiny50_3k.pt \
--prompt "Wisdom" --tokens 300
```
For local inference after cloning:
```bash
python sample.py --ckpt checkpoints/tinyllm_philosophy_quotes_tiny50_3k.pt \
--prompt "The soul" --tokens 300
```
## Training examples
Original toy overfit:
```bash
python train.py --steps 3000 --eval-interval 500 --eval-iters 10 \
--batch-size 16 --n-embd 64 --block-size 64 \
--out checkpoints/tinyllm_overfit_3k.pt
```
Local philosophy tiny50 overfit-ish run:
```bash
python train.py --data data/philosophy_quotes_local_tiny50.txt \
--steps 3000 --eval-interval 500 --eval-iters 10 \
--batch-size 16 --n-embd 64 --block-size 64 --lr 0.001 \
--out checkpoints/tinyllm_philosophy_quotes_tiny50_3k.pt
```
## Limitations
- Character-level only.
- Generates memorized/stylized snippets, not reliable knowledge.
- Weird spelling and malformed words are expected.
- Intended for learning and experimentation.
|