bitsandbytes/pr_2030/en/reference/optim/lars.md

LARS

LARS (Layer-wise Adaptive Rate Scaling) is an optimizer designed for training with large batch sizes to accelerate training. LARS uses a separate learning rate for each layer instead of each parameter. The learning rate is calculated from a trust ratio between the weight and gradient norm in a layer. This helps calibrate a stable update size.

LARS[[api-class]][[bitsandbytes.optim.LARS]]

bitsandbytes.optim.LARS[[bitsandbytes.optim.LARS]]

bitsandbytes.optim.LARS(params, lr, momentum = 0, dampening = 0, weight_decay = 0, nesterov = False, optim_bits = 32, args = None, min_8bit_size = 4096, max_unorm = 0.02)

Source

init[[bitsandbytes.optim.LARS.init]]

__init__(params, lr, momentum = 0, dampening = 0, weight_decay = 0, nesterov = False, optim_bits = 32, args = None, min_8bit_size = 4096, max_unorm = 0.02)

Source

Parameters:

params (torch.tensor) : The input parameters to optimize.

lr (float) : The learning rate.

momentum (float, defaults to 0) : The momentum value speeds up the optimizer by taking bigger steps.

dampening (float, defaults to 0) : The dampening value reduces the momentum of the optimizer.

weight_decay (float, defaults to 1e-2) : The weight decay value for the optimizer.

nesterov (bool, defaults to False) : Whether to use Nesterov momentum.

optim_bits (int, defaults to 32) : The number of bits of the optimizer state.

args (object, defaults to None) : An object with additional arguments.

min_8bit_size (int, defaults to 4096) : The minimum number of elements of the parameter tensors for 8-bit optimization.

max_unorm (float, defaults to 0.02) : The maximum gradient norm.

Base LARS optimizer.

LARS8bit[[bitsandbytes.optim.LARS8bit]]

bitsandbytes.optim.LARS8bit[[bitsandbytes.optim.LARS8bit]]

bitsandbytes.optim.LARS8bit(params, lr, momentum = 0, dampening = 0, weight_decay = 0, nesterov = False, args = None, min_8bit_size = 4096, max_unorm = 0.02)

Source

init[[bitsandbytes.optim.LARS8bit.init]]

__init__(params, lr, momentum = 0, dampening = 0, weight_decay = 0, nesterov = False, args = None, min_8bit_size = 4096, max_unorm = 0.02)

Source

Parameters:

params (torch.tensor) : The input parameters to optimize.

lr (float) : The learning rate.

momentum (float, defaults to 0) : The momentum value speeds up the optimizer by taking bigger steps.

dampening (float, defaults to 0) : The dampening value reduces the momentum of the optimizer.

weight_decay (float, defaults to 1e-2) : The weight decay value for the optimizer.

nesterov (bool, defaults to False) : Whether to use Nesterov momentum.

args (object, defaults to None) : An object with additional arguments.

min_8bit_size (int, defaults to 4096) : The minimum number of elements of the parameter tensors for 8-bit optimization.

max_unorm (float, defaults to 0.02) : The maximum gradient norm.

8-bit LARS optimizer.

LARS32bit[[bitsandbytes.optim.LARS32bit]]

bitsandbytes.optim.LARS32bit[[bitsandbytes.optim.LARS32bit]]

bitsandbytes.optim.LARS32bit(params, lr, momentum = 0, dampening = 0, weight_decay = 0, nesterov = False, args = None, min_8bit_size = 4096, max_unorm = 0.02)

Source

init[[bitsandbytes.optim.LARS32bit.init]]

__init__(params, lr, momentum = 0, dampening = 0, weight_decay = 0, nesterov = False, args = None, min_8bit_size = 4096, max_unorm = 0.02)

Source

Parameters:

params (torch.tensor) : The input parameters to optimize.

lr (float) : The learning rate.

momentum (float, defaults to 0) : The momentum value speeds up the optimizer by taking bigger steps.

dampening (float, defaults to 0) : The dampening value reduces the momentum of the optimizer.

weight_decay (float, defaults to 1e-2) : The weight decay value for the optimizer.

nesterov (bool, defaults to False) : Whether to use Nesterov momentum.

args (object, defaults to None) : An object with additional arguments.

min_8bit_size (int, defaults to 4096) : The minimum number of elements of the parameter tensors for 8-bit optimization.

max_unorm (float, defaults to 0.02) : The maximum gradient norm.

32-bit LARS optimizer.