Upload folder using huggingface_hub

README.md

@@ -0,0 +1,130 @@
+---
+language:
+- en
+license: apache-2.0
+library_name: transformers
+tags:
+- bash
+- shell
+- linux
+- cli
+- code
+- small-language-model
+pipeline_tag: text-generation
+model-index:
+- name: Dwarf-15M-Instruct
+  results: []
+---
+
+# Dwarf-15M-Instruct
+
+A **15.54M parameter** shell/bash specialist language model that translates natural language into Linux commands.
+
+## Quick Start
+
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+tokenizer = AutoTokenizer.from_pretrained("ThingAI/Dwarf-15M-Instruct", trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained("ThingAI/Dwarf-15M-Instruct", trust_remote_code=True)
+
+prompt = "<|user|>\nFind all Python files modified in the last 3 days\n<|end|>\n<|assistant|>\n"
+inputs = tokenizer(prompt, return_tensors="pt")
+outputs = model.generate(**inputs, max_new_tokens=100, do_sample=False)
+print(tokenizer.decode(outputs[0], skip_special_tokens=False))
+# → find . -name '*.py' -mtime -3
+```
+
+## What It Does
+
+Dwarf-15M takes natural language descriptions of tasks and produces the corresponding Linux/bash command:
+
+| Prompt | Output |
+|---|---|
+| Show current date | `date` |
+| List files | `ls` |
+| Kill process 1234 | `kill 1234` |
+| Delete all .tmp files in current directory | `rm ./*.tmp` |
+| Compress the /home/user/project folder | `tar -czf project.tar.gz /home/user/project/` |
+| Check if port 8080 is in use | `ss -tlnp \| grep 8080` |
+| Restart the nginx service | `sudo systemctl restart nginx` |
+| Find all files containing TODO | `grep -rl 'TODO' .` |
+| Change owner of /var/www to www-data | `sudo chown -R www-data:www-data /var/www` |
+| Run script.sh in background and log output | `nohup ./script.sh > log.txt 2>&1 &` |
+| Find and replace foo with bar in config.txt | `sed -i 's/foo/bar/g' config.txt` |
+| What does chmod 755 do? | chmod 755 sets read+write+execute for owner, read+execute for group and others. |
+| Write a bash function that counts lines | `count_lines() { wc -l < "$1"; }` |
+
+## Architecture
+
+| Parameter | Value |
+|---|---|
+| Parameters | 15.54M |
+| Layers | 12 |
+| Hidden dim | 320 |
+| Attention | GQA (5 query, 1 KV head) |
+| FFN | SwiGLU (d_ff=864) |
+| Normalization | RMSNorm |
+| Positional | RoPE (θ=10000) |
+| Vocabulary | 8,202 (DwarfGoToken) |
+| Max sequence | 2,048 |
+| Weight tying | Yes (embed ↔ lm_head) |
+
+## Training
+
+**Pretraining:** 21.6B tokens (ratio 1,390:1) on 11 datasets:
+- Shell/bash: The Stack (shell, batchfile), GunA-SD/bash_code — 38.5%
+- Code: The Stack (Python, C), CodeFeedback — 39.1%
+- Instructions: ShellLife (52K NL→command), rlvr-code-data-bash (133K problems) — 11%
+- English: helpful-instructions, FineWeb — 10.3%
+- CoT: Magpie-Reasoning — 1.1%
+
+**SFT:** 557 curated Linux command pairs, 5 epochs, lr=4e-5. Training time: 19 seconds.
+
+**Tokenizer:** [DwarfGoToken](https://huggingface.co/ThingAI/DwarfGoToken) — 8,202 token BPE with syntax-aware pre-tokenization for shell operators (2>&1, &&, >>).
+
+## Chat Template
+
+```
+<|user|>
+Your question here
+<|end|>
+<|assistant|>
+Model response here
+<|end|>
+```
+
+## Intended Use
+
+Dwarf-15M is designed as a **CLI assistant** that suggests commands for user review before execution. It is NOT a general-purpose chatbot. Best results on:
+- Simple to medium Linux commands (file ops, process management, networking)
+- Bash one-liners and short functions
+- Command explanations ("What does chmod 755 do?")
+
+## Limitations
+
+- 15M parameters — cannot handle complex multi-step reasoning
+- May produce incorrect commands for unusual or very specific requests
+- Should **always** be used with human review before executing any suggested command
+- English only
+- Trained primarily on Ubuntu/Debian commands
+
+## Hardware
+
+- Pretrained on RTX 3070 (8GB VRAM) at ~127K tokens/sec
+- Inference: runs on any hardware including CPU, ~30MB model size
+
+## License
+
+Apache 2.0
+
+## Citation
+
+```bibtex
+@misc{dwarf15m2026,
+  title={Dwarf-15M-Instruct: A Shell Specialist Language Model},
+  author={ThingsAI},
+  year={2026},
+  url={https://huggingface.co/ThingAI/Dwarf-15M-Instruct}
+}
+```

chat_template.jinja

@@ -0,0 +1,11 @@
+{% for message in messages %}{% if message['role'] == 'system' %}<|system|>
+{{ message['content'] }}
+<|end|>
+{% elif message['role'] == 'user' %}<|user|>
+{{ message['content'] }}
+<|end|>
+{% elif message['role'] == 'assistant' %}<|assistant|>
+{{ message['content'] }}
+<|end|>
+{% endif %}{% endfor %}{% if messages[-1]['role'] != 'assistant' %}<|assistant|>
+{% endif %}

config.json

@@ -0,0 +1,24 @@
+{
+  "architectures": [
+    "DwarfForCausalLM"
+  ],
+  "model_type": "dwarf",
+  "auto_map": {
+    "AutoConfig": "configuration_dwarf.DwarfConfig",
+    "AutoModelForCausalLM": "modeling_dwarf.DwarfForCausalLM"
+  },
+  "vocab_size": 8202,
+  "d_model": 320,
+  "n_layers": 12,
+  "n_heads": 5,
+  "n_kv_heads": 1,
+  "d_ff": 864,
+  "max_seq_len": 2048,
+  "rope_theta": 10000.0,
+  "norm_eps": 1e-05,
+  "head_dim": 64,
+  "torch_dtype": "float32",
+  "transformers_version": "4.45.0",
+  "bos_token_id": 1,
+  "eos_token_id": 2
+}

configuration_dwarf.py

@@ -0,0 +1,36 @@
+"""Dwarf model configuration."""
+from transformers import PretrainedConfig
+
+
+class DwarfConfig(PretrainedConfig):
+    model_type = "dwarf"
+
+    def __init__(
+        self,
+        vocab_size=8202,
+        d_model=320,
+        n_layers=12,
+        n_heads=5,
+        n_kv_heads=1,
+        d_ff=864,
+        max_seq_len=2048,
+        rope_theta=10000.0,
+        norm_eps=1e-5,
+        head_dim=64,
+        **kwargs,
+    ):
+        self.vocab_size = vocab_size
+        self.d_model = d_model
+        self.n_layers = n_layers
+        self.n_heads = n_heads
+        self.n_kv_heads = n_kv_heads
+        self.d_ff = d_ff
+        self.max_seq_len = max_seq_len
+        self.rope_theta = rope_theta
+        self.norm_eps = norm_eps
+        self.head_dim = head_dim
+        self.num_hidden_layers = n_layers
+        self.hidden_size = d_model
+        self.num_attention_heads = n_heads
+        self.num_key_value_heads = n_kv_heads
+        super().__init__(**kwargs)

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ed70fe03dec2a2fd6469ca337f72cf222d5375024fdeb8ddb51b5e3bad8f76a3
+size 72674368

modeling_dwarf.py

@@ -0,0 +1,158 @@
+"""Dwarf-15M: a 15.54M parameter shell/bash specialist language model."""
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from transformers import PreTrainedModel, GenerationMixin
+from .configuration_dwarf import DwarfConfig
+
+
+class RMSNorm(nn.Module):
+    def __init__(self, dim, eps=1e-5):
+        super().__init__()
+        self.eps = eps
+        self.scale = nn.Parameter(torch.ones(dim))
+
+    def forward(self, x):
+        rms = x.float().pow(2).mean(-1, keepdim=True).add(self.eps).rsqrt()
+        return (x.float() * rms).to(x.dtype) * self.scale
+
+
+class RotaryEmbedding(nn.Module):
+    def __init__(self, head_dim, max_seq_len, theta=10000.0):
+        super().__init__()
+        assert head_dim % 2 == 0
+        self.head_dim = head_dim
+        self.max_seq_len = max_seq_len
+        self.theta = theta
+        self.cos_cache = None
+        self.sin_cache = None
+        self._max = 0
+
+    def _build_cache(self, seq_len, device):
+        inv_freq = 1.0 / (self.theta ** (torch.arange(0, self.head_dim, 2, device=device).float() / self.head_dim))
+        t = torch.arange(seq_len, device=device).float()
+        freqs = torch.outer(t, inv_freq)
+        emb = torch.cat([freqs, freqs], dim=-1)
+        self.cos_cache = emb.cos()[None, None]
+        self.sin_cache = emb.sin()[None, None]
+        self._max = seq_len
+
+    @staticmethod
+    def _rotate_half(x):
+        x1, x2 = x.chunk(2, dim=-1)
+        return torch.cat([-x2, x1], dim=-1)
+
+    def forward(self, q, k):
+        T = q.size(2)
+        if self.cos_cache is None or T > self._max or self.cos_cache.device != q.device:
+            self._build_cache(max(T, self.max_seq_len), q.device)
+        cos = self.cos_cache[:, :, :T, :]
+        sin = self.sin_cache[:, :, :T, :]
+        q = q * cos + self._rotate_half(q) * sin
+        k = k * cos + self._rotate_half(k) * sin
+        return q, k
+
+
+class GroupedQueryAttention(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.n_heads = config.n_heads
+        self.n_kv_heads = config.n_kv_heads
+        self.n_groups = config.n_heads // config.n_kv_heads
+        self.head_dim = config.head_dim
+
+        self.q_proj = nn.Linear(config.d_model, config.n_heads * config.head_dim, bias=True)
+        self.k_proj = nn.Linear(config.d_model, config.n_kv_heads * config.head_dim, bias=True)
+        self.v_proj = nn.Linear(config.d_model, config.n_kv_heads * config.head_dim, bias=True)
+        self.o_proj = nn.Linear(config.n_heads * config.head_dim, config.d_model, bias=False)
+
+        self.rope = RotaryEmbedding(config.head_dim, config.max_seq_len, config.rope_theta)
+
+    def forward(self, x):
+        B, T, _ = x.shape
+        q = self.q_proj(x).view(B, T, self.n_heads, self.head_dim).transpose(1, 2)
+        k = self.k_proj(x).view(B, T, self.n_kv_heads, self.head_dim).transpose(1, 2)
+        v = self.v_proj(x).view(B, T, self.n_kv_heads, self.head_dim).transpose(1, 2)
+
+        q, k = self.rope(q, k)
+
+        if self.n_groups > 1:
+            k = k.repeat_interleave(self.n_groups, dim=1)
+            v = v.repeat_interleave(self.n_groups, dim=1)
+
+        out = F.scaled_dot_product_attention(q, k, v, attn_mask=None, is_causal=True)
+        out = out.transpose(1, 2).contiguous().view(B, T, self.n_heads * self.head_dim)
+        return self.o_proj(out)
+
+
+class SwiGLUFFN(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.gate_proj = nn.Linear(config.d_model, config.d_ff, bias=False)
+        self.up_proj = nn.Linear(config.d_model, config.d_ff, bias=False)
+        self.down_proj = nn.Linear(config.d_ff, config.d_model, bias=False)
+
+    def forward(self, x):
+        return self.down_proj(F.silu(self.gate_proj(x)) * self.up_proj(x))
+
+
+class DwarfBlock(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.norm_attn = RMSNorm(config.d_model, config.norm_eps)
+        self.attn = GroupedQueryAttention(config)
+        self.norm_ffn = RMSNorm(config.d_model, config.norm_eps)
+        self.ffn = SwiGLUFFN(config)
+
+    def forward(self, x):
+        x = x + self.attn(self.norm_attn(x))
+        x = x + self.ffn(self.norm_ffn(x))
+        return x
+
+
+class DwarfForCausalLM(PreTrainedModel, GenerationMixin):
+    config_class = DwarfConfig
+    _tied_weights_keys = ["lm_head.weight"]
+
+    def __init__(self, config):
+        super().__init__(config)
+        self.embed_tokens = nn.Embedding(config.vocab_size, config.d_model)
+        self.layers = nn.ModuleList([DwarfBlock(config) for _ in range(config.n_layers)])
+        self.norm = RMSNorm(config.d_model, config.norm_eps)
+        self.lm_head = nn.Linear(config.d_model, config.vocab_size, bias=False)
+        self.post_init()
+
+    def tie_weights(self, **kwargs):
+        self.lm_head.weight = self.embed_tokens.weight
+
+    def get_input_embeddings(self):
+        return self.embed_tokens
+
+    def set_input_embeddings(self, value):
+        self.embed_tokens = value
+
+    def get_output_embeddings(self):
+        return self.lm_head
+
+    def set_output_embeddings(self, new_embeddings):
+        self.lm_head = new_embeddings
+
+    def forward(self, input_ids, attention_mask=None, labels=None, **kwargs):
+        x = self.embed_tokens(input_ids)
+        for layer in self.layers:
+            x = layer(x)
+        logits = self.lm_head(self.norm(x))
+
+        loss = None
+        if labels is not None:
+            loss = F.cross_entropy(
+                logits[:, :-1].contiguous().view(-1, logits.size(-1)),
+                labels[:, 1:].contiguous().view(-1),
+                ignore_index=-100,
+            )
+
+        from transformers.modeling_outputs import CausalLMOutput
+        return CausalLMOutput(loss=loss, logits=logits)
+
+    def prepare_inputs_for_generation(self, input_ids, **kwargs):
+        return {"input_ids": input_ids}

tokenizer_config.json

@@ -0,0 +1,19 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<s>",
+  "eos_token": "</s>",
+  "extra_special_tokens": [
+    "<|system|>",
+    "<|user|>",
+    "<|assistant|>",
+    "<|end|>",
+    "<|thinking|>",
+    "<|/thinking|>"
+  ],
+  "is_local": false,
+  "local_files_only": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<pad>",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "<unk>"
+}