File size: 3,259 Bytes
5a88c1e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 | ---
language: en
license: apache-2.0
tags:
- cinimod
- devops
- llm
- causal-lm
- llama
- base-model
base_model: dkudos/cinimod-devops
pipeline_tag: text-generation
---
# Cinimod DevOps 300M - SFT-ready base
The DevOps 300M pre-trained checkpoint in its **SFT-ready** form. Trained weights are
identical to [`dkudos/cinimod-devops`](https://huggingface.co/dkudos/cinimod-devops);
the vocabulary and embedding matrix are padded by **3 reserved rows** so that chat
special tokens can be added later during supervised fine-tuning without resizing the
embedding.
| | [`dkudos/cinimod-devops`](https://huggingface.co/dkudos/cinimod-devops) | this repo |
|------------------------------|---|---|
| `vocab_size` | 65536 | **65539** (+3 reserved) |
| parameters | 287,310,848 | **287,313,920** |
| GGUF exports | f16 + Q8_0 (llama.cpp) | none - use the base repo's GGUFs or convert |
| intended use | raw pre-trained / llama.cpp serving | fine-tuning, CMAF experiments |
## Status
**PRE-TRAINED ONLY.** No instruction tuning, no chat behaviour, no RLHF. It is a domain
base model, not an assistant.
## Architecture
Llama-style decoder-only (`LlamaForCausalLM`), trained from scratch - not a conversion of
any existing checkpoint:
| Property | Value |
|---|---|
| parameters | 287,313,920 |
| hidden size | 1024 |
| layers | 20 |
| attention heads | 16 (4 KV heads, GQA) |
| intermediate size | 2730 |
| vocab size | 65539 (65536 trained + 3 reserved) |
| context length | 4096 |
| tied embeddings | yes |
| dtype | bfloat16 |
## Why this repo exists
`devops-300m-base` is the canonical Phase-0 base for the CMAF (cross-model attention
fusion) experiments in `gitlab.com/dkudos/cinimod-llm` (work item #6). The boundary
drift harness, the divergent domain-LoRA trio (`lora-a` / `lora-b` / `lora-c`) and every
hidden-state compatibility measurement load this exact checkpoint, so it is pinned here
for reproducibility of those results.
## Provenance
Recovered 2026-09-20 from `/home/dkaiser/sft-training/devops-300m-prepped/` after the
original `outputs/devops-300m-4096-bf16-vast/` directory was lost in a working-copy
rsync incident.
1. Trained on vast.ai (2x RTX 4090, bf16, 4000 steps, sequence length 4096).
2. Fetched to the training host.
3. `prep_for_sft.py` converted it to HF `LlamaForCausalLM` format and padded the
vocabulary for chat special tokens.
4. This repository is that HF-native copy - the same trained weights, SFT-ready layout.
## Usage
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("dkudos/cinimod-devops-sft-base")
model = AutoModelForCausalLM.from_pretrained(
"dkudos/cinimod-devops-sft-base", torch_dtype="auto"
)
```
## Intended use and limitations
- DevOps / sysadmin domain language modelling: documentation, runbook-style text,
tooling assistance.
- English only.
- 300M parameters is a research-scale model: useful for domain adaptation and
specialization studies, not for general-purpose instruction following.
- The 3 reserved embedding rows are untrained. If you add tokens, initialise them
before fine-tuning.
## License
apache-2.0
|