| --- |
| library_name: transformers |
| license: other |
| language: |
| - code |
| tags: |
| - qwen3 |
| - causal-lm |
| - code-completion |
| - habbo |
| - from-scratch |
| --- |
| |
| # FuseLLM-112M |
|
|
| A small **112M-parameter decoder-only language model trained from scratch** (no base |
| checkpoint, no LoRA) on a corpus of Habbo emulator / game-server source code. The |
| goal is a tiny, fast model for **code completion** in that Java codebase, not a |
| general-purpose or instruction-following model. |
|
|
| ## Model details |
|
|
| | | | |
| |---|---| |
| | Architecture | Qwen3 (decoder-only causal LM) | |
| | Parameters | ~112M (tied input/output embeddings) | |
| | Hidden size | 512 | |
| | Layers | 8 (all full attention) | |
| | Attention heads | 8 (8 KV heads) | |
| | Vocab size | 151,936 | |
| | Max context | 2048 | |
| | Precision | float32 (safetensors) | |
| | Training | From scratch, 4 epochs, 16,188 steps | |
| | Final train loss | ~0.58 | |
|
|
| `tie_word_embeddings: true` — the output `lm_head` shares the input embedding |
| matrix, so checkpoints store only one copy. This is expected, not a missing weight. |
|
|
| ## Intended use |
|
|
| - **Code completion** for Habbo-style Java server code (raw prompt → continuation). |
| - Local experimentation / distillation base. |
|
|
| ## What it is NOT |
|
|
| - **Not instruction-tuned / not a chat model.** It was trained only on raw source |
| code, never on chat/instruction data. |
| - The Qwen3 ChatML chat template is included (it ships with the tokenizer) for |
| tokenizer/tool compatibility, but the model has **not** learned to follow chat |
| turns. Passing chat-formatted prompts will produce poor, often repetitive output. |
| Use it in **completion mode**, not conversation mode. |
|
|
| ## Usage |
|
|
| ### transformers (recommended for completion) |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| m = AutoModelForCausalLM.from_pretrained("h4bbo/FuseLLM-112M") |
| tok = AutoTokenizer.from_pretrained("h4bbo/FuseLLM-112M") |
| |
| prompt = "public class Room {\n public void onEnter(Player p) {\n " |
| ids = tok(prompt, return_tensors="pt").input_ids |
| out = m.generate(ids, max_new_tokens=64, do_sample=False, |
| repetition_penalty=1.1, pad_token_id=tok.eos_token_id) |
| print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True)) |
| ``` |
|
|
| ### llama.cpp (completion mode) |
|
|
| No GGUF is shipped in this repo. The HF model is **verified** to convert and run in |
| `llama.cpp`; generate the GGUF locally: |
|
|
| ```bash |
| # 1) convert HF -> lossless fp16 GGUF |
| python convert_hf_to_gguf.py h4bbo/FuseLLM-112M --outtype f16 \ |
| --model-name FuseLLM-112M --outfile FuseLLM-112M.fp16.gguf |
| # (optional) 4-bit quantize |
| llama-quantize FuseLLM-112M.fp16.gguf FuseLLM-112M.Q4_K_M.gguf Q4_K_M |
| |
| # 2) completion mode — pass the raw code seed, do NOT use chat/conversation mode. |
| llama-cli -m FuseLLM-112M.Q4_K_M.gguf -cnv -st --no-jinja \ |
| -f seed.txt -n 64 --temp 0.0 --repeat-penalty 1.1 --no-display-prompt < /dev/null |
| ``` |
|
|
| `--no-jinja` keeps the prompt raw (the embedded chat template exists but the model |
| isn't chat-tuned, so conversation mode is not meaningful for this model). |
|
|
| ## Files |
|
|
| - `model.safetensors`, `config.json`, `generation_config.json` — HF model |
| - `tokenizer.json`, `tokenizer_config.json`, `chat_template.jinja` — tokenizer + ChatML template |
|
|
| ## Notes |
|
|
| - Small model + limited-domain corpus: expect repetition on long generations; use |
| a repetition penalty and keep continuations short. |
| - Trained from scratch, so this is fully independent of any upstream Qwen weights. |
| The Qwen3 architecture/tokenizer are reused for compatibility. |
