pages/index.md

Reproduction: FlashOptim: Optimizers for Memory-Efficient Training

HF paper page

Pages

Page
Executive summary
Claim 1: FlashOptim combines improved master-weight splitting with companded 8-bit optimizer-state quantization to reduce parameter-associated optimizer memory (Section 3).
Claim 2: FlashOptim reduces AdamW training memory from 16 to 7 bytes per parameter, or 5 bytes with gradient release (Table 1).
Claim 3: For Llama-3.1-8B finetuning, FlashOptim reduces peak memory from 175 GiB to 113 GiB by compressing parameters and optimizer states (Figure 1; Table 4).
Claim 4: FlashOptim variants match reference optimizer training-loss trajectories in GPT-2 pretraining and ResNet-50 image classification (Figure 2).
Claim 5: FlashOptim matches reference scores on ResNet-50 validation accuracy, Llama-3.1-8B GSM8K finetuning, and GPT-2 in-context learning benchmarks (Table 2; Table 3).
Claim 6: ULP-based weight splitting lowers FP32 reconstruction error and companding prevents quantized AdamW training divergence compared with linear optimizer-state quantization (Figure 3; Figure 5).
Conclusion