# Reproduction: FlashOptim: Optimizers for Memory-Efficient Training

[HF paper page](https://huggingface.co/papers/2602.23349)

## Pages

| Page |
| --- |
| [Executive summary](#/executive-summary) |
| [Claim 1: FlashOptim combines improved master-weight splitting with companded 8-bit optimizer-state quantization to reduce parameter-associated optimizer memory (Section 3).](#/claim-1-flashoptim-combines-improved-master-weight-splitting-with-companded-8-bit-optimizer-state-quantization-to-reduce-parameter-associated-optimizer-memory-section-3) |
| [Claim 2: FlashOptim reduces AdamW training memory from 16 to 7 bytes per parameter, or 5 bytes with gradient release (Table 1).](#/claim-2-flashoptim-reduces-adamw-training-memory-from-16-to-7-bytes-per-parameter-or-5-bytes-with-gradient-release-table-1) |
| [Claim 3: For Llama-3.1-8B finetuning, FlashOptim reduces peak memory from 175 GiB to 113 GiB by compressing parameters and optimizer states (Figure 1; Table 4).](#/claim-3-for-llama-3-1-8b-finetuning-flashoptim-reduces-peak-memory-from-175-gib-to-113-gib-by-compressing-parameters-and-optimizer-states-figure-1-table-4) |
| [Claim 4: FlashOptim variants match reference optimizer training-loss trajectories in GPT-2 pretraining and ResNet-50 image classification (Figure 2).](#/claim-4-flashoptim-variants-match-reference-optimizer-training-loss-trajectories-in-gpt-2-pretraining-and-resnet-50-image-classification-figure-2) |
| [Claim 5: FlashOptim matches reference scores on ResNet-50 validation accuracy, Llama-3.1-8B GSM8K finetuning, and GPT-2 in-context learning benchmarks (Table 2; Table 3).](#/claim-5-flashoptim-matches-reference-scores-on-resnet-50-validation-accuracy-llama-3-1-8b-gsm8k-finetuning-and-gpt-2-in-context-learning-benchmarks-table-2-table-3) |
| [Claim 6: ULP-based weight splitting lowers FP32 reconstruction error and companding prevents quantized AdamW training divergence compared with linear optimizer-state quantization (Figure 3; Figure 5).](#/claim-6-ulp-based-weight-splitting-lowers-fp32-reconstruction-error-and-companding-prevents-quantized-adamw-training-divergence-compared-with-linear-optimizer-state-quantization-figure-3-figure-5) |
| [Conclusion](#/conclusion) |