| --- |
| license: mit |
| tags: |
| - llama |
| - text-generation |
| - from-scratch |
| - hobby-project |
| language: |
| - en |
| --- |
| |
| # Microllm |
|
|
| -- Use the non GGUF version for now. -- |
|
|
| A small (768-dim, 22-layer, ~50260 vocab) decoder-only |
| transformer, pretrained from scratch on streaming FineWeb-Edu and instruction |
| fine-tuned on Dolly-15k + No Robots. This is an independent hobbyist project, |
| not affiliated with any AI lab - trained end-to-end on a single rented GPU. |
|
|
| Architecturally this is a standard Llama-style model (RMSNorm, RoPE, SwiGLU, |
| tied embeddings), so it loads directly with `transformers`: |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| tok = AutoTokenizer.from_pretrained("MLVXN/microllm") |
| model = AutoModelForCausalLM.from_pretrained("MLVXN/microllm") |
| |
| prompt = "<|user|>What's your name?<|assistant|>" |
| ids = tok(prompt, return_tensors="pt").input_ids |
| out = model.generate(ids, max_new_tokens=100, do_sample=True, temperature=0.8, top_k=50) |
| print(tok.decode(out[0])) |
| ``` |
|
|
| ## Chat format |
| This model was fine-tuned on a simple turn structure, not a full chat |
| template - wrap each user message like this: |
|
|
| ``` |
| <|user|>{message}<|assistant|> |
| ``` |
|
|
| Generation should stop at `<|end|>` (id 50259). |
|
|
| ## Checkpoint info |
| - Fine-tuning phase reached: `no_robots` |
| - Fine-tuning global step: `4845` |
| - seq_len: 1024 |
| |
| ## Known limitations |
| This is a ~150M-parameter model trained on a modest compute budget. Expect |
| coherent grammar and conversational fluency, but unreliable facts and no real |
| multi-step reasoning - that's the honest ceiling for this size/budget, not a |
| bug. It reliably knows its own identity (name/creator) because that was |
| explicitly trained in, separately from general knowledge quality. |
| |
| ## Running in LM Studio / llama.cpp / Ollama |
| If a `.gguf` file is included in this repo, download it directly in LM Studio |
| via its Hugging Face search, or point llama.cpp / Ollama at the file. If no |
| `.gguf` is present, convert it yourself: |
| |
| ```bash |
| git clone https://github.com/ggerganov/llama.cpp |
| cd llama.cpp |
| pip install -r requirements.txt |
| python convert_hf_to_gguf.py /path/to/microllm --outfile microllm.gguf --outtype f16 |
| # optional: quantize for a smaller file |
| ./llama-quantize microllm.gguf microllm.Q4_K_M.gguf Q4_K_M |
| ``` |
| |
