| | --- |
| | language: [en] |
| | license: mit |
| | tags: |
| | - devops |
| | - cicd |
| | - docker |
| | - kubernetes |
| | - infrastructure |
| | - slm |
| | - llama-style |
| | - rope |
| | - 5m-context |
| | - from-scratch |
| | - 1b-params |
| | pipeline_tag: text-generation |
| | --- |
| | |
| | # DevOps Engineer-SLM: Role-Based Small Language Model |
| |
|
| | A **LLaMA-style transformer** (~989.8M params, ~0.99B) trained from scratch for the **DevOps Engineer** role. |
| | Supports up to **5M token context** via RoPE with gradient checkpointing. |
| |
|
| | ## Architecture |
| | | Component | Value | |
| | |-----------|-------| |
| | | Architecture | LLaMA-style (RoPE + RMSNorm + SwiGLU) | |
| | | Parameters | ~989.8M (~0.99B) | |
| | | Layers | 32 | |
| | | Heads | 20 | |
| | | Embedding | 1600 | |
| | | Max Context | 5,000,000 tokens | |
| | | Max Output | 5,000,000 tokens | |
| | | Vocab | 2,107 BPE | |
| | | Model Size | ~4 GB (fp32) | |
| |
|
| | ## Training |
| | - Best eval loss: 2.5998684406280517 |
| | - Trained with gradient checkpointing on Apple M4 (MPS) |
| | - 3 epochs, batch_size=1, grad_accum=16 |
| |
|
| | ## Usage |
| | ```python |
| | from huggingface_hub import hf_hub_download |
| | from tokenizers import Tokenizer |
| | |
| | model_path = hf_hub_download("sathishphdai/devops-engineer-slm-5m", "model.safetensors") |
| | tokenizer_path = hf_hub_download("sathishphdai/devops-engineer-slm-5m", "devops_engineer_tokenizer.json") |
| | tokenizer = Tokenizer.from_file(tokenizer_path) |
| | ``` |
| |
|
