Buckets:
Overview
8-bit optimizers reduce the memory footprint of 32-bit optimizers without any performance degradation which means you can train large models with many parameters faster. At the core of 8-bit optimizers is block-wise quantization which enables quantization accuracy, computational efficiency, and stability.
bitsandbytes provides 8-bit optimizers through the base Optimizer8bit class, and additionally provides Optimizer2State and Optimizer1State for 2-state (for example, Adam) and 1-state (for example, Adagrad) optimizers respectively. To provide custom optimizer hyperparameters, use the GlobalOptimManager class to configure the optimizer.
Optimizer8bit[[bitsandbytes.optim.optimizer.Optimizer8bit]]
class bitsandbytes.optim.optimizer.Optimizer8bitbitsandbytes.optim.optimizer.Optimizer8bit
initbitsandbytes.optim.optimizer.Optimizer8bit.inittorch.Tensor) --
The input parameters to optimize.
- optim_bits (
int, defaults to 32) -- The number of bits of the optimizer state. - is_paged (
bool, defaults toFalse) -- Whether the optimizer is a paged optimizer or not.0
Base 8-bit optimizer class.
Optimizer2State[[bitsandbytes.optim.optimizer.Optimizer2State]]
class bitsandbytes.optim.optimizer.Optimizer2Statebitsandbytes.optim.optimizer.Optimizer2State
initbitsandbytes.optim.optimizer.Optimizer2State.initstr) --
The name of the optimizer.
- params (
torch.Tensor) -- The input parameters to optimize. - lr (
float, defaults to 1e-3) -- The learning rate. - betas (
tuple, defaults to (0.9, 0.999)) -- The beta values for the optimizer. - eps (
float, defaults to 1e-8) -- The epsilon value for the optimizer. - weight_decay (
float, defaults to 0.0) -- The weight decay value for the optimizer. - optim_bits (
int, defaults to 32) -- The number of bits of the optimizer state. - args (
object, defaults toNone) -- An object with additional arguments. - min_8bit_size (
int, defaults to 4096) -- The minimum number of elements of the parameter tensors for 8-bit optimization. - percentile_clipping (
int, defaults to 100) -- Adapts clipping threshold automatically by tracking the last 100 gradient norms and clipping the gradient at a certain percentile to improve stability. - block_wise (
bool, defaults toTrue) -- Whether to independently quantize each block of tensors to reduce outlier effects and improve stability. - max_unorm (
float, defaults to 0.0) -- The maximum value to normalize each block with. - skip_zeros (
bool, defaults toFalse) -- Whether to skip zero values for sparse gradients and models to ensure correct updates. - is_paged (
bool, defaults toFalse) -- Whether the optimizer is a paged optimizer or not. - alpha (
float, defaults to 0.0) -- The alpha value for the AdEMAMix optimizer. - t_alpha (
Optional[int], defaults toNone) -- Number of iterations for alpha scheduling with AdEMAMix. - t_beta3 (
Optional[int], defaults toNone) -- Number of iterations for beta scheduling with AdEMAMix.0
Base 2-state update optimizer class.
Optimizer1State[[bitsandbytes.optim.optimizer.Optimizer1State]]
class bitsandbytes.optim.optimizer.Optimizer1Statebitsandbytes.optim.optimizer.Optimizer1State
initbitsandbytes.optim.optimizer.Optimizer1State.initstr) --
The name of the optimizer.
- params (
torch.Tensor) -- The input parameters to optimize. - lr (
float, defaults to 1e-3) -- The learning rate. - betas (
tuple, defaults to (0.9, 0.0)) -- The beta values for the optimizer. - eps (
float, defaults to 1e-8) -- The epsilon value for the optimizer. - weight_decay (
float, defaults to 0.0) -- The weight decay value for the optimizer. - optim_bits (
int, defaults to 32) -- The number of bits of the optimizer state. - args (
object, defaults toNone) -- An object with additional arguments. - min_8bit_size (
int, defaults to 4096) -- The minimum number of elements of the parameter tensors for 8-bit optimization. - percentile_clipping (
int, defaults to 100) -- Adapts clipping threshold automatically by tracking the last 100 gradient norms and clipping the gradient at a certain percentile to improve stability. - block_wise (
bool, defaults toTrue) -- Whether to independently quantize each block of tensors to reduce outlier effects and improve stability. - max_unorm (
float, defaults to 0.0) -- The maximum value to normalize each block with. - skip_zeros (
bool, defaults toFalse) -- Whether to skip zero values for sparse gradients and models to ensure correct updates. - is_paged (
bool, defaults toFalse) -- Whether the optimizer is a paged optimizer or not.0
Base 1-state update optimizer class.
Utilities[[bitsandbytes.optim.GlobalOptimManager]]
class bitsandbytes.optim.GlobalOptimManagerbitsandbytes.optim.GlobalOptimManager
A global optimizer manager for enabling custom optimizer configs.
override_configbitsandbytes.optim.GlobalOptimManager.override_configtorch.Tensor or list(torch.Tensors)) --
The input parameters.
- key (
str) -- The hyperparameter to override. - value -- The hyperparameter value.
- key_value_dict (
dict) -- A dictionary with multiple key-values to override.0
Override initial optimizer config with specific hyperparameters.
The key-values of the optimizer config for the input parameters are overridden
This can be both, optimizer parameters like betas or lr, or it can be
8-bit specific parameters like optim_bits or percentile_clipping.
Example:
import torch
import bitsandbytes as bnb
mng = bnb.optim.GlobalOptimManager.get_instance()
model = MyModel()
mng.register_parameters(model.parameters()) # 1. register parameters while still on CPU
model = model.cuda()
# use 8-bit optimizer states for all parameters
adam = bnb.optim.Adam(model.parameters(), lr=0.001, optim_bits=8)
# 2. override: the parameter model.fc1.weight now uses 32-bit Adam
mng.override_config(model.fc1.weight, 'optim_bits', 32)
Xet Storage Details
- Size:
- 12.2 kB
- Xet hash:
- 64ccf4d7adf835d1b3c7fd8b9c97886d6603b8c4d73ead9f7969d420d881f216
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.