Add hardware details and per-model effective batch sizes
Browse files
README.md
CHANGED
|
@@ -78,17 +78,24 @@ model = PeftModel.from_pretrained(model, adapter_path, subfolder=subfolder)
|
|
| 78 |
| Parameter | No-DP (base) | DP variants |
|
| 79 |
|---|---|---|
|
| 80 |
| Epochs | 2 | 2 |
|
| 81 |
-
|
|
| 82 |
| Learning rate | 1e-4 | 2e-4 |
|
| 83 |
| Optimizer | AdamW | AdamW |
|
| 84 |
| LR scheduler | Cosine | Cosine |
|
| 85 |
| Warmup ratio | 5% | 5% |
|
| 86 |
-
| Grad accumulation steps | 4–8 | 16 |
|
| 87 |
| Max gradient norm | 1.0 | 1.0 |
|
| 88 |
| Sequence length | 1024 | 1024 |
|
| 89 |
| Precision | bfloat16 | bfloat16 |
|
| 90 |
| Seed | 42 | 42 |
|
| 91 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 92 |
### Differential Privacy
|
| 93 |
|
| 94 |
| Parameter | Value |
|
|
@@ -103,8 +110,9 @@ model = PeftModel.from_pretrained(model, adapter_path, subfolder=subfolder)
|
|
| 103 |
|
| 104 |
### Infrastructure
|
| 105 |
|
|
|
|
|
|
|
| 106 |
- **Distributed strategy:** DDP (Distributed Data Parallel) with NCCL backend
|
| 107 |
-
- **Hardware:** NVIDIA H200 GPUs
|
| 108 |
|
| 109 |
## Evaluation Results
|
| 110 |
|
|
|
|
| 78 |
| Parameter | No-DP (base) | DP variants |
|
| 79 |
|---|---|---|
|
| 80 |
| Epochs | 2 | 2 |
|
| 81 |
+
| Micro-batch size (per GPU) | 8 | 8 |
|
| 82 |
| Learning rate | 1e-4 | 2e-4 |
|
| 83 |
| Optimizer | AdamW | AdamW |
|
| 84 |
| LR scheduler | Cosine | Cosine |
|
| 85 |
| Warmup ratio | 5% | 5% |
|
|
|
|
| 86 |
| Max gradient norm | 1.0 | 1.0 |
|
| 87 |
| Sequence length | 1024 | 1024 |
|
| 88 |
| Precision | bfloat16 | bfloat16 |
|
| 89 |
| Seed | 42 | 42 |
|
| 90 |
|
| 91 |
+
**Effective batch sizes** (micro-batch × gradient accumulation steps × 8 GPUs):
|
| 92 |
+
|
| 93 |
+
| Model | No-DP | DP ε=3 | DP ε=8 |
|
| 94 |
+
|---|---|---|---|
|
| 95 |
+
| Granite-4.0-H-Tiny | 512 (8×8×8) | 1024 (8×16×8) | 1024 (8×16×8) |
|
| 96 |
+
| DeepSeek-Coder-6.7B | 256 (8×4×8) | 512 (8×8×8) | 512 (8×8×8) |
|
| 97 |
+
| Qwen3-4B-Instruct | 256 (8×4×8) | 512 (8×8×8) | 512 (8×8×8) |
|
| 98 |
+
|
| 99 |
### Differential Privacy
|
| 100 |
|
| 101 |
| Parameter | Value |
|
|
|
|
| 110 |
|
| 111 |
### Infrastructure
|
| 112 |
|
| 113 |
+
- **GPUs:** 8 × NVIDIA H200 (140 GB VRAM each)
|
| 114 |
+
- **CUDA:** 13.0
|
| 115 |
- **Distributed strategy:** DDP (Distributed Data Parallel) with NCCL backend
|
|
|
|
| 116 |
|
| 117 |
## Evaluation Results
|
| 118 |
|