melihcatal commited on
Commit
f374d1e
·
verified ·
1 Parent(s): 80c43e2

Add hardware details and per-model effective batch sizes

Browse files
Files changed (1) hide show
  1. README.md +11 -3
README.md CHANGED
@@ -78,17 +78,24 @@ model = PeftModel.from_pretrained(model, adapter_path, subfolder=subfolder)
78
  | Parameter | No-DP (base) | DP variants |
79
  |---|---|---|
80
  | Epochs | 2 | 2 |
81
- | Batch size (per GPU) | 8 | 8 |
82
  | Learning rate | 1e-4 | 2e-4 |
83
  | Optimizer | AdamW | AdamW |
84
  | LR scheduler | Cosine | Cosine |
85
  | Warmup ratio | 5% | 5% |
86
- | Grad accumulation steps | 4–8 | 16 |
87
  | Max gradient norm | 1.0 | 1.0 |
88
  | Sequence length | 1024 | 1024 |
89
  | Precision | bfloat16 | bfloat16 |
90
  | Seed | 42 | 42 |
91
 
 
 
 
 
 
 
 
 
92
  ### Differential Privacy
93
 
94
  | Parameter | Value |
@@ -103,8 +110,9 @@ model = PeftModel.from_pretrained(model, adapter_path, subfolder=subfolder)
103
 
104
  ### Infrastructure
105
 
 
 
106
  - **Distributed strategy:** DDP (Distributed Data Parallel) with NCCL backend
107
- - **Hardware:** NVIDIA H200 GPUs
108
 
109
  ## Evaluation Results
110
 
 
78
  | Parameter | No-DP (base) | DP variants |
79
  |---|---|---|
80
  | Epochs | 2 | 2 |
81
+ | Micro-batch size (per GPU) | 8 | 8 |
82
  | Learning rate | 1e-4 | 2e-4 |
83
  | Optimizer | AdamW | AdamW |
84
  | LR scheduler | Cosine | Cosine |
85
  | Warmup ratio | 5% | 5% |
 
86
  | Max gradient norm | 1.0 | 1.0 |
87
  | Sequence length | 1024 | 1024 |
88
  | Precision | bfloat16 | bfloat16 |
89
  | Seed | 42 | 42 |
90
 
91
+ **Effective batch sizes** (micro-batch × gradient accumulation steps × 8 GPUs):
92
+
93
+ | Model | No-DP | DP ε=3 | DP ε=8 |
94
+ |---|---|---|---|
95
+ | Granite-4.0-H-Tiny | 512 (8×8×8) | 1024 (8×16×8) | 1024 (8×16×8) |
96
+ | DeepSeek-Coder-6.7B | 256 (8×4×8) | 512 (8×8×8) | 512 (8×8×8) |
97
+ | Qwen3-4B-Instruct | 256 (8×4×8) | 512 (8×8×8) | 512 (8×8×8) |
98
+
99
  ### Differential Privacy
100
 
101
  | Parameter | Value |
 
110
 
111
  ### Infrastructure
112
 
113
+ - **GPUs:** 8 × NVIDIA H200 (140 GB VRAM each)
114
+ - **CUDA:** 13.0
115
  - **Distributed strategy:** DDP (Distributed Data Parallel) with NCCL backend
 
116
 
117
  ## Evaluation Results
118