File size: 2,955 Bytes
d881dad 3efb5e2 c0bb125 3efb5e2 c0bb125 0362134 3efb5e2 c0bb125 3efb5e2 c0bb125 3efb5e2 c0bb125 3efb5e2 c0bb125 3efb5e2 c0bb125 3efb5e2 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 |
---
{
"language": ["en"],
"license": "apache-2.0",
"tags": [
"text-generation",
"causal-lm",
"instruction-tuning",
"supervised-fine-tuning",
"synthetic-qa",
"lora",
"axolotl",
"deepspeed",
"transformers",
"commandr",
"cohere",
"eu-hpc"
],
"datasets": [
"axolotl_deduplicated_synthetic_qa"
],
"metrics": [
"loss"
],
"library_name": "transformers",
"framework": "pytorch",
"base_model": "CohereLabs/c4ai-command-r-v01",
"model_name": "commandr-35b-sft",
"pipeline_tag": "text-generation",
"task_categories": ["text-generation", "instruction-following"],
"model_type": "AutoModelForCausalLM",
"inference": {
"parameters": {
"max_new_tokens": 512,
"temperature": 0.7,
"top_p": 0.9
}
},
"trained_on": [
"Leonardo EuroHPC"
],
"description": "Supervised fine-tuning (SFT) of Cohere Command-R 35B on the synthetic QA dataset using LoRA and Axolotl. The model improves conversational reasoning and instruction-following capabilities."
}
---
# Command-R 35B — SFT (Supervised Fine-Tuning on Synthetic QA)
**Model type:** Causal Language Model
**Base model:** [CohereLabs/c4ai-command-r-v01](https://huggingface.co/CohereLabs/c4ai-command-r-v01)
**License:** Apache 2.0
**Framework:** [Axolotl](https://github.com/OpenAccess-AI-Collective/axolotl)
---
## Overview
`commandr-35b-sft` is a **supervised fine-tuned** variant of Cohere’s Command-R 35B model.
Fine-tuning was performed on a high-quality instruction-following dataset using LoRA adapters, enabling improved conversational reasoning and question answering.
Training was conducted on the **Leonardo EuroHPC** system.
---
## Training Setup
**Objective:** Supervised fine-tuning (instruction following)
**Adapter type:** LoRA
**Precision:** bfloat16
**Hardware:** 8 nodes × 2 × NVIDIA A100 64GB GPUs
**Framework:** DeepSpeed ZeRO-1, Axolotl, PyTorch 2.5.1+cu121
**Runtime:** ~6 hours
**Dataset split:** 70% train / 30% validation
---
## Dataset
**Name:** `axolotl_deduplicated_synthetic_qa.jsonl`
**Type:** Instruction-following synthetic QA dataset
Each sample follows a QA/chat format used in the `alpaca_chat.load_qa` schema.
---
## Hyperparameters
| Parameter | Value |
|------------|-------|
| Sequence length | 2048 |
| Micro batch size | 1 |
| Gradient accumulation | 2 |
| Epochs | 1 |
| Learning rate | 0.0001 |
| LR scheduler | cosine |
| Optimizer | AdamW (8-bit) |
| Warmup steps | 20 |
| Weight decay | 0.0 |
| LoRA rank (r) | 16 |
| LoRA alpha | 32 |
| LoRA dropout | 0.05 |
| LoRA target modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| Gradient checkpointing | ✅ |
| Flash attention | ✅ |
| Auto resume | ✅ |
| Loss watchdog threshold | 8.0 |
| Loss watchdog patience | 20 |
---
## Tokenizer
**Tokenizer type:** `AutoTokenizer`
**Special token:** `<|end_of_text|>` as `pad_token` |