File size: 3,019 Bytes
5d773e7 2c21e12 5d773e7 9d6f532 ad8ca4b 9d6f532 ad8ca4b 9d6f532 ad8ca4b 9d6f532 ad8ca4b 9d6f532 ad8ca4b 9d6f532 ad8ca4b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 | ---
{
"language": ["en"],
"license": "apache-2.0",
"tags": [
"text-generation",
"causal-lm",
"two-stage-training",
"continual-pretraining",
"supervised-fine-tuning",
"synthetic-qa",
"lora",
"axolotl",
"deepspeed",
"transformers",
"commandr",
"cohere",
"eu-hpc"
],
"datasets": [
"arxiv",
"gov",
"news",
"wikipedia",
"axolotl_deduplicated_synthetic_qa"
],
"metrics": [
"loss"
],
"library_name": "transformers",
"framework": "pytorch",
"base_model": "ubitech-edg/commandr-35b-cpt",
"model_name": "commandr-35b-cpt-sft",
"pipeline_tag": "text-generation",
"task_categories": ["text-generation", "instruction-following"],
"model_type": "AutoModelForCausalLM",
"inference": {
"parameters": {
"max_new_tokens": 512,
"temperature": 0.7,
"top_p": 0.9
}
},
"trained_on": [
"Leonardo EuroHPC"
],
"description": "Two-stage training (CPT + SFT) of Cohere Command-R 35B using Axolotl and DeepSpeed. The model first undergoes domain-adaptive continual pretraining and then instruction fine-tuning on synthetic QA data."
}
---
# Command-R 35B — CPT + SFT
**Model type:** Causal Language Model
**Base model:** [commandr-35b-cpt](https://huggingface.co/ubitech-edg/commandr-35b-cpt)
**License:** Apache 2.0
**Framework:** [Axolotl](https://github.com/OpenAccess-AI-Collective/axolotl)
---
## Overview
`commandr-35b-cpt-sft` combines both **continual pretraining (CPT)** and **supervised fine-tuning (SFT)** in a two-stage process.
The model first learns additional general-domain representations (CPT), then undergoes supervised instruction tuning (SFT) on synthetic QA data.
This combination enhances factual grounding, fluency, and instruction adherence.
Training was performed on the **Leonardo EuroHPC** system.
---
## Training Setup
**Stage 1 (CPT):** Domain-adaptive continual pretraining
**Stage 2 (SFT):** Instruction fine-tuning
**Adapter type:** LoRA
**Precision:** bfloat16
**Hardware:** 8 nodes × 2 × NVIDIA A100 64GB GPUs
**Framework:** DeepSpeed ZeRO-1, Axolotl, PyTorch 2.5.1+cu121
---
## Datasets
**CPT Stage:**
- `arxiv.jsonl`
- `gov.jsonl`
- `news.jsonl`
- `wiki.jsonl`
**SFT Stage:**
- `axolotl_deduplicated_synthetic_qa.jsonl`
---
## Hyperparameters
| Parameter | Value |
|------------|-------|
| Sequence length | 2048 |
| Micro batch size | 1 |
| Gradient accumulation | 2 |
| Epochs | 1 |
| Learning rate | 0.00008 |
| LR scheduler | cosine |
| Optimizer | AdamW (8-bit) |
| Warmup steps | 20 |
| Weight decay | 0.0 |
| LoRA rank (r) | 16 |
| LoRA alpha | 32 |
| LoRA dropout | 0.05 |
| LoRA target modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| Gradient checkpointing | ✅ |
| Flash attention | ✅ |
| Auto resume | ✅ |
| Loss watchdog threshold | 8.0 |
| Loss watchdog patience | 20 |
---
## Tokenizer
**Tokenizer type:** `AutoTokenizer`
**Special token:** `<|end_of_text|>` as `pad_token` |