bitsandbytes/v0.48.2/en/reference/optim/sgd.md

SGD

Stochastic gradient descent (SGD) is a basic gradient descent optimizer to minimize loss given a set of model parameters and updates the parameters in the opposite direction of the gradient. The update is performed on a randomly sampled mini-batch of data from the dataset.

bitsandbytes also supports momentum and Nesterov momentum to accelerate SGD by adding a weighted average of past gradients to the current gradient.

SGD[[api-class]][[bitsandbytes.optim.SGD]]

class bitsandbytes.optim.SGDbitsandbytes.optim.SGDhttps://github.com/bitsandbytes-foundation/bitsandbytes/blob/v0.48.2/bitsandbytes/optim/sgd.py#L8[{"name": "params", "val": ""}, {"name": "lr", "val": ""}, {"name": "momentum", "val": " = 0"}, {"name": "dampening", "val": " = 0"}, {"name": "weight_decay", "val": " = 0"}, {"name": "nesterov", "val": " = False"}, {"name": "optim_bits", "val": " = 32"}, {"name": "args", "val": " = None"}, {"name": "min_8bit_size", "val": " = 4096"}, {"name": "percentile_clipping", "val": " = 100"}, {"name": "block_wise", "val": " = True"}]

initbitsandbytes.optim.SGD.inithttps://github.com/bitsandbytes-foundation/bitsandbytes/blob/v0.48.2/bitsandbytes/optim/sgd.py#L9[{"name": "params", "val": ""}, {"name": "lr", "val": ""}, {"name": "momentum", "val": " = 0"}, {"name": "dampening", "val": " = 0"}, {"name": "weight_decay", "val": " = 0"}, {"name": "nesterov", "val": " = False"}, {"name": "optim_bits", "val": " = 32"}, {"name": "args", "val": " = None"}, {"name": "min_8bit_size", "val": " = 4096"}, {"name": "percentile_clipping", "val": " = 100"}, {"name": "block_wise", "val": " = True"}]- params (torch.tensor) -- The input parameters to optimize.

• lr (float) -- The learning rate.
• momentum (float, defaults to 0) -- The momentum value speeds up the optimizer by taking bigger steps.
• dampening (float, defaults to 0) -- The dampening value reduces the momentum of the optimizer.
• weight_decay (float, defaults to 0.0) -- The weight decay value for the optimizer.
• nesterov (bool, defaults to False) -- Whether to use Nesterov momentum.
• optim_bits (int, defaults to 32) -- The number of bits of the optimizer state.
• args (object, defaults to None) -- An object with additional arguments.
• min_8bit_size (int, defaults to 4096) -- The minimum number of elements of the parameter tensors for 8-bit optimization.
• percentile_clipping (int, defaults to 100) -- Adapts clipping threshold automatically by tracking the last 100 gradient norms and clipping the gradient at a certain percentile to improve stability.
• block_wise (bool, defaults to True) -- Whether to independently quantize each block of tensors to reduce outlier effects and improve stability.0

Base SGD optimizer.

SGD8bit[[bitsandbytes.optim.SGD8bit]]

class bitsandbytes.optim.SGD8bitbitsandbytes.optim.SGD8bithttps://github.com/bitsandbytes-foundation/bitsandbytes/blob/v0.48.2/bitsandbytes/optim/sgd.py#L67[{"name": "params", "val": ""}, {"name": "lr", "val": ""}, {"name": "momentum", "val": " = 0"}, {"name": "dampening", "val": " = 0"}, {"name": "weight_decay", "val": " = 0"}, {"name": "nesterov", "val": " = False"}, {"name": "args", "val": " = None"}, {"name": "min_8bit_size", "val": " = 4096"}, {"name": "percentile_clipping", "val": " = 100"}, {"name": "block_wise", "val": " = True"}]

initbitsandbytes.optim.SGD8bit.inithttps://github.com/bitsandbytes-foundation/bitsandbytes/blob/v0.48.2/bitsandbytes/optim/sgd.py#L68[{"name": "params", "val": ""}, {"name": "lr", "val": ""}, {"name": "momentum", "val": " = 0"}, {"name": "dampening", "val": " = 0"}, {"name": "weight_decay", "val": " = 0"}, {"name": "nesterov", "val": " = False"}, {"name": "args", "val": " = None"}, {"name": "min_8bit_size", "val": " = 4096"}, {"name": "percentile_clipping", "val": " = 100"}, {"name": "block_wise", "val": " = True"}]- params (torch.tensor) -- The input parameters to optimize.

• lr (float) -- The learning rate.
• momentum (float, defaults to 0) -- The momentum value speeds up the optimizer by taking bigger steps.
• dampening (float, defaults to 0) -- The dampening value reduces the momentum of the optimizer.
• weight_decay (float, defaults to 0.0) -- The weight decay value for the optimizer.
• nesterov (bool, defaults to False) -- Whether to use Nesterov momentum.
• args (object, defaults to None) -- An object with additional arguments.
• min_8bit_size (int, defaults to 4096) -- The minimum number of elements of the parameter tensors for 8-bit optimization.
• percentile_clipping (int, defaults to 100) -- Adapts clipping threshold automatically by tracking the last 100 gradient norms and clipping the gradient at a certain percentile to improve stability.
• block_wise (bool, defaults to True) -- Whether to independently quantize each block of tensors to reduce outlier effects and improve stability.0

8-bit SGD optimizer.

SGD32bit[[bitsandbytes.optim.SGD32bit]]

class bitsandbytes.optim.SGD32bitbitsandbytes.optim.SGD32bithttps://github.com/bitsandbytes-foundation/bitsandbytes/blob/v0.48.2/bitsandbytes/optim/sgd.py#L123[{"name": "params", "val": ""}, {"name": "lr", "val": ""}, {"name": "momentum", "val": " = 0"}, {"name": "dampening", "val": " = 0"}, {"name": "weight_decay", "val": " = 0"}, {"name": "nesterov", "val": " = False"}, {"name": "args", "val": " = None"}, {"name": "min_8bit_size", "val": " = 4096"}, {"name": "percentile_clipping", "val": " = 100"}, {"name": "block_wise", "val": " = True"}]

initbitsandbytes.optim.SGD32bit.inithttps://github.com/bitsandbytes-foundation/bitsandbytes/blob/v0.48.2/bitsandbytes/optim/sgd.py#L124[{"name": "params", "val": ""}, {"name": "lr", "val": ""}, {"name": "momentum", "val": " = 0"}, {"name": "dampening", "val": " = 0"}, {"name": "weight_decay", "val": " = 0"}, {"name": "nesterov", "val": " = False"}, {"name": "args", "val": " = None"}, {"name": "min_8bit_size", "val": " = 4096"}, {"name": "percentile_clipping", "val": " = 100"}, {"name": "block_wise", "val": " = True"}]- params (torch.tensor) -- The input parameters to optimize.

• lr (float) -- The learning rate.
• momentum (float, defaults to 0) -- The momentum value speeds up the optimizer by taking bigger steps.
• dampening (float, defaults to 0) -- The dampening value reduces the momentum of the optimizer.
• weight_decay (float, defaults to 0.0) -- The weight decay value for the optimizer.
• nesterov (bool, defaults to False) -- Whether to use Nesterov momentum.
• args (object, defaults to None) -- An object with additional arguments.
• min_8bit_size (int, defaults to 4096) -- The minimum number of elements of the parameter tensors for 8-bit optimization.
• percentile_clipping (int, defaults to 100) -- Adapts clipping threshold automatically by tracking the last 100 gradient norms and clipping the gradient at a certain percentile to improve stability.
• block_wise (bool, defaults to True) -- Whether to independently quantize each block of tensors to reduce outlier effects and improve stability.0

32-bit SGD optimizer.