| --- |
| language: en |
| license: apache-2.0 |
| tags: |
| - cinimod |
| - devops |
| - llm |
| - causal-lm |
| - llama |
| - base-model |
| base_model: dkudos/cinimod-devops |
| pipeline_tag: text-generation |
| --- |
| |
| # Cinimod DevOps 300M - SFT-ready base |
|
|
| The DevOps 300M pre-trained checkpoint in its **SFT-ready** form. Trained weights are |
| identical to [`dkudos/cinimod-devops`](https://huggingface.co/dkudos/cinimod-devops); |
| the vocabulary and embedding matrix are padded by **3 reserved rows** so that chat |
| special tokens can be added later during supervised fine-tuning without resizing the |
| embedding. |
|
|
| | | [`dkudos/cinimod-devops`](https://huggingface.co/dkudos/cinimod-devops) | this repo | |
| |------------------------------|---|---| |
| | `vocab_size` | 65536 | **65539** (+3 reserved) | |
| | parameters | 287,310,848 | **287,313,920** | |
| | GGUF exports | f16 + Q8_0 (llama.cpp) | none - use the base repo's GGUFs or convert | |
| | intended use | raw pre-trained / llama.cpp serving | fine-tuning, CMAF experiments | |
| |
| ## Status |
| |
| **PRE-TRAINED ONLY.** No instruction tuning, no chat behaviour, no RLHF. It is a domain |
| base model, not an assistant. |
| |
| ## Architecture |
| |
| Llama-style decoder-only (`LlamaForCausalLM`), trained from scratch - not a conversion of |
| any existing checkpoint: |
| |
| | Property | Value | |
| |---|---| |
| | parameters | 287,313,920 | |
| | hidden size | 1024 | |
| | layers | 20 | |
| | attention heads | 16 (4 KV heads, GQA) | |
| | intermediate size | 2730 | |
| | vocab size | 65539 (65536 trained + 3 reserved) | |
| | context length | 4096 | |
| | tied embeddings | yes | |
| | dtype | bfloat16 | |
| |
| ## Why this repo exists |
| |
| `devops-300m-base` is the canonical Phase-0 base for the CMAF (cross-model attention |
| fusion) experiments in `gitlab.com/dkudos/cinimod-llm` (work item #6). The boundary |
| drift harness, the divergent domain-LoRA trio (`lora-a` / `lora-b` / `lora-c`) and every |
| hidden-state compatibility measurement load this exact checkpoint, so it is pinned here |
| for reproducibility of those results. |
| |
| ## Provenance |
| |
| Recovered 2026-09-20 from `/home/dkaiser/sft-training/devops-300m-prepped/` after the |
| original `outputs/devops-300m-4096-bf16-vast/` directory was lost in a working-copy |
| rsync incident. |
| |
| 1. Trained on vast.ai (2x RTX 4090, bf16, 4000 steps, sequence length 4096). |
| 2. Fetched to the training host. |
| 3. `prep_for_sft.py` converted it to HF `LlamaForCausalLM` format and padded the |
| vocabulary for chat special tokens. |
| 4. This repository is that HF-native copy - the same trained weights, SFT-ready layout. |
| |
| ## Usage |
| |
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| tok = AutoTokenizer.from_pretrained("dkudos/cinimod-devops-sft-base") |
| model = AutoModelForCausalLM.from_pretrained( |
| "dkudos/cinimod-devops-sft-base", torch_dtype="auto" |
| ) |
| ``` |
| |
| ## Intended use and limitations |
| |
| - DevOps / sysadmin domain language modelling: documentation, runbook-style text, |
| tooling assistance. |
| - English only. |
| - 300M parameters is a research-scale model: useful for domain adaptation and |
| specialization studies, not for general-purpose instruction following. |
| - The 3 reserved embedding rows are untrained. If you add tokens, initialise them |
| before fine-tuning. |
| |
| ## License |
| |
| apache-2.0 |
| |
