bitsandbytes/v0.48.2/en/reference/optim/lion.md

Lion

Lion (Evolved Sign Momentum) is a unique optimizer that uses the sign of the gradient to determine the update direction of the momentum. This makes Lion more memory-efficient and faster than AdamW which tracks and store the first and second-order moments.

Lion[[api-class]][[bitsandbytes.optim.Lion]]

class bitsandbytes.optim.Lionbitsandbytes.optim.Lionhttps://github.com/bitsandbytes-foundation/bitsandbytes/blob/v0.48.2/bitsandbytes/optim/lion.py#L8[{"name": "params", "val": ""}, {"name": "lr", "val": " = 0.0001"}, {"name": "betas", "val": " = (0.9, 0.99)"}, {"name": "weight_decay", "val": " = 0"}, {"name": "optim_bits", "val": " = 32"}, {"name": "args", "val": " = None"}, {"name": "min_8bit_size", "val": " = 4096"}, {"name": "percentile_clipping", "val": " = 100"}, {"name": "block_wise", "val": " = True"}, {"name": "is_paged", "val": " = False"}]

initbitsandbytes.optim.Lion.inithttps://github.com/bitsandbytes-foundation/bitsandbytes/blob/v0.48.2/bitsandbytes/optim/lion.py#L9[{"name": "params", "val": ""}, {"name": "lr", "val": " = 0.0001"}, {"name": "betas", "val": " = (0.9, 0.99)"}, {"name": "weight_decay", "val": " = 0"}, {"name": "optim_bits", "val": " = 32"}, {"name": "args", "val": " = None"}, {"name": "min_8bit_size", "val": " = 4096"}, {"name": "percentile_clipping", "val": " = 100"}, {"name": "block_wise", "val": " = True"}, {"name": "is_paged", "val": " = False"}]- params (torch.tensor) -- The input parameters to optimize.

• lr (float, defaults to 1e-4) -- The learning rate.
• betas (tuple(float, float), defaults to (0.9, 0.999)) -- The beta values are the decay rates of the first and second-order moment of the optimizer.
• weight_decay (float, defaults to 0) -- The weight decay value for the optimizer.
• optim_bits (int, defaults to 32) -- The number of bits of the optimizer state.
• args (object, defaults to None) -- An object with additional arguments.
• min_8bit_size (int, defaults to 4096) -- The minimum number of elements of the parameter tensors for 8-bit optimization.
• percentile_clipping (int, defaults to 100) -- Adapts clipping threshold automatically by tracking the last 100 gradient norms and clipping the gradient at a certain percentile to improve stability.
• block_wise (bool, defaults to True) -- Whether to independently quantize each block of tensors to reduce outlier effects and improve stability.
• is_paged (bool, defaults to False) -- Whether the optimizer is a paged optimizer or not.0

Base Lion optimizer.

Lion8bit[[bitsandbytes.optim.Lion8bit]]

class bitsandbytes.optim.Lion8bitbitsandbytes.optim.Lion8bithttps://github.com/bitsandbytes-foundation/bitsandbytes/blob/v0.48.2/bitsandbytes/optim/lion.py#L63[{"name": "params", "val": ""}, {"name": "lr", "val": " = 0.0001"}, {"name": "betas", "val": " = (0.9, 0.99)"}, {"name": "weight_decay", "val": " = 0"}, {"name": "args", "val": " = None"}, {"name": "min_8bit_size", "val": " = 4096"}, {"name": "percentile_clipping", "val": " = 100"}, {"name": "block_wise", "val": " = True"}, {"name": "is_paged", "val": " = False"}]

initbitsandbytes.optim.Lion8bit.inithttps://github.com/bitsandbytes-foundation/bitsandbytes/blob/v0.48.2/bitsandbytes/optim/lion.py#L64[{"name": "params", "val": ""}, {"name": "lr", "val": " = 0.0001"}, {"name": "betas", "val": " = (0.9, 0.99)"}, {"name": "weight_decay", "val": " = 0"}, {"name": "args", "val": " = None"}, {"name": "min_8bit_size", "val": " = 4096"}, {"name": "percentile_clipping", "val": " = 100"}, {"name": "block_wise", "val": " = True"}, {"name": "is_paged", "val": " = False"}]- params (torch.tensor) -- The input parameters to optimize.

• lr (float, defaults to 1e-4) -- The learning rate.
• betas (tuple(float, float), defaults to (0.9, 0.999)) -- The beta values are the decay rates of the first and second-order moment of the optimizer.
• weight_decay (float, defaults to 0) -- The weight decay value for the optimizer.
• args (object, defaults to None) -- An object with additional arguments.
• min_8bit_size (int, defaults to 4096) -- The minimum number of elements of the parameter tensors for 8-bit optimization.
• percentile_clipping (int, defaults to 100) -- Adapts clipping threshold automatically by tracking the last 100 gradient norms and clipping the gradient at a certain percentile to improve stability.
• block_wise (bool, defaults to True) -- Whether to independently quantize each block of tensors to reduce outlier effects and improve stability.
• is_paged (bool, defaults to False) -- Whether the optimizer is a paged optimizer or not.0

8-bit Lion optimizer.

Lion32bit[[bitsandbytes.optim.Lion32bit]]

class bitsandbytes.optim.Lion32bitbitsandbytes.optim.Lion32bithttps://github.com/bitsandbytes-foundation/bitsandbytes/blob/v0.48.2/bitsandbytes/optim/lion.py#L115[{"name": "params", "val": ""}, {"name": "lr", "val": " = 0.0001"}, {"name": "betas", "val": " = (0.9, 0.99)"}, {"name": "weight_decay", "val": " = 0"}, {"name": "args", "val": " = None"}, {"name": "min_8bit_size", "val": " = 4096"}, {"name": "percentile_clipping", "val": " = 100"}, {"name": "block_wise", "val": " = True"}, {"name": "is_paged", "val": " = False"}]

initbitsandbytes.optim.Lion32bit.inithttps://github.com/bitsandbytes-foundation/bitsandbytes/blob/v0.48.2/bitsandbytes/optim/lion.py#L116[{"name": "params", "val": ""}, {"name": "lr", "val": " = 0.0001"}, {"name": "betas", "val": " = (0.9, 0.99)"}, {"name": "weight_decay", "val": " = 0"}, {"name": "args", "val": " = None"}, {"name": "min_8bit_size", "val": " = 4096"}, {"name": "percentile_clipping", "val": " = 100"}, {"name": "block_wise", "val": " = True"}, {"name": "is_paged", "val": " = False"}]- params (torch.tensor) -- The input parameters to optimize.

• lr (float, defaults to 1e-4) -- The learning rate.
• betas (tuple(float, float), defaults to (0.9, 0.999)) -- The beta values are the decay rates of the first and second-order moment of the optimizer.
• weight_decay (float, defaults to 0) -- The weight decay value for the optimizer.
• args (object, defaults to None) -- An object with additional arguments.
• min_8bit_size (int, defaults to 4096) -- The minimum number of elements of the parameter tensors for 8-bit optimization.
• percentile_clipping (int, defaults to 100) -- Adapts clipping threshold automatically by tracking the last 100 gradient norms and clipping the gradient at a certain percentile to improve stability.
• block_wise (bool, defaults to True) -- Whether to independently quantize each block of tensors to reduce outlier effects and improve stability.
• is_paged (bool, defaults to False) -- Whether the optimizer is a paged optimizer or not.0

32-bit Lion optimizer.

PagedLion[[bitsandbytes.optim.PagedLion]]

class bitsandbytes.optim.PagedLionbitsandbytes.optim.PagedLionhttps://github.com/bitsandbytes-foundation/bitsandbytes/blob/v0.48.2/bitsandbytes/optim/lion.py#L167[{"name": "params", "val": ""}, {"name": "lr", "val": " = 0.0001"}, {"name": "betas", "val": " = (0.9, 0.99)"}, {"name": "weight_decay", "val": " = 0"}, {"name": "optim_bits", "val": " = 32"}, {"name": "args", "val": " = None"}, {"name": "min_8bit_size", "val": " = 4096"}, {"name": "percentile_clipping", "val": " = 100"}, {"name": "block_wise", "val": " = True"}]

initbitsandbytes.optim.PagedLion.inithttps://github.com/bitsandbytes-foundation/bitsandbytes/blob/v0.48.2/bitsandbytes/optim/lion.py#L168[{"name": "params", "val": ""}, {"name": "lr", "val": " = 0.0001"}, {"name": "betas", "val": " = (0.9, 0.99)"}, {"name": "weight_decay", "val": " = 0"}, {"name": "optim_bits", "val": " = 32"}, {"name": "args", "val": " = None"}, {"name": "min_8bit_size", "val": " = 4096"}, {"name": "percentile_clipping", "val": " = 100"}, {"name": "block_wise", "val": " = True"}]- params (torch.tensor) -- The input parameters to optimize.

• lr (float, defaults to 1e-4) -- The learning rate.
• betas (tuple(float, float), defaults to (0.9, 0.999)) -- The beta values are the decay rates of the first and second-order moment of the optimizer.
• weight_decay (float, defaults to 0) -- The weight decay value for the optimizer.
• optim_bits (int, defaults to 32) -- The number of bits of the optimizer state.
• args (object, defaults to None) -- An object with additional arguments.
• min_8bit_size (int, defaults to 4096) -- The minimum number of elements of the parameter tensors for 8-bit optimization.
• percentile_clipping (int, defaults to 100) -- Adapts clipping threshold automatically by tracking the last 100 gradient norms and clipping the gradient at a certain percentile to improve stability.
• block_wise (bool, defaults to True) -- Whether to independently quantize each block of tensors to reduce outlier effects and improve stability.0

Paged Lion optimizer.

PagedLion8bit[[bitsandbytes.optim.PagedLion8bit]]

class bitsandbytes.optim.PagedLion8bitbitsandbytes.optim.PagedLion8bithttps://github.com/bitsandbytes-foundation/bitsandbytes/blob/v0.48.2/bitsandbytes/optim/lion.py#L219[{"name": "params", "val": ""}, {"name": "lr", "val": " = 0.0001"}, {"name": "betas", "val": " = (0.9, 0.99)"}, {"name": "weight_decay", "val": " = 0"}, {"name": "args", "val": " = None"}, {"name": "min_8bit_size", "val": " = 4096"}, {"name": "percentile_clipping", "val": " = 100"}, {"name": "block_wise", "val": " = True"}]

initbitsandbytes.optim.PagedLion8bit.inithttps://github.com/bitsandbytes-foundation/bitsandbytes/blob/v0.48.2/bitsandbytes/optim/lion.py#L220[{"name": "params", "val": ""}, {"name": "lr", "val": " = 0.0001"}, {"name": "betas", "val": " = (0.9, 0.99)"}, {"name": "weight_decay", "val": " = 0"}, {"name": "args", "val": " = None"}, {"name": "min_8bit_size", "val": " = 4096"}, {"name": "percentile_clipping", "val": " = 100"}, {"name": "block_wise", "val": " = True"}]- params (torch.tensor) -- The input parameters to optimize.

• lr (float, defaults to 1e-4) -- The learning rate.
• betas (tuple(float, float), defaults to (0.9, 0.999)) -- The beta values are the decay rates of the first and second-order moment of the optimizer.
• weight_decay (float, defaults to 0) -- The weight decay value for the optimizer.
• optim_bits (int, defaults to 32) -- The number of bits of the optimizer state.
• args (object, defaults to None) -- An object with additional arguments.
• min_8bit_size (int, defaults to 4096) -- The minimum number of elements of the parameter tensors for 8-bit optimization.
• percentile_clipping (int, defaults to 100) -- Adapts clipping threshold automatically by tracking the last 100 gradient norms and clipping the gradient at a certain percentile to improve stability.
• block_wise (bool, defaults to True) -- Whether to independently quantize each block of tensors to reduce outlier effects and improve stability.0

Paged 8-bit Lion optimizer.

PagedLion32bit[[bitsandbytes.optim.PagedLion32bit]]

class bitsandbytes.optim.PagedLion32bitbitsandbytes.optim.PagedLion32bithttps://github.com/bitsandbytes-foundation/bitsandbytes/blob/v0.48.2/bitsandbytes/optim/lion.py#L270[{"name": "params", "val": ""}, {"name": "lr", "val": " = 0.0001"}, {"name": "betas", "val": " = (0.9, 0.99)"}, {"name": "weight_decay", "val": " = 0"}, {"name": "args", "val": " = None"}, {"name": "min_8bit_size", "val": " = 4096"}, {"name": "percentile_clipping", "val": " = 100"}, {"name": "block_wise", "val": " = True"}]

initbitsandbytes.optim.PagedLion32bit.inithttps://github.com/bitsandbytes-foundation/bitsandbytes/blob/v0.48.2/bitsandbytes/optim/lion.py#L271[{"name": "params", "val": ""}, {"name": "lr", "val": " = 0.0001"}, {"name": "betas", "val": " = (0.9, 0.99)"}, {"name": "weight_decay", "val": " = 0"}, {"name": "args", "val": " = None"}, {"name": "min_8bit_size", "val": " = 4096"}, {"name": "percentile_clipping", "val": " = 100"}, {"name": "block_wise", "val": " = True"}]- params (torch.tensor) -- The input parameters to optimize.

• lr (float, defaults to 1e-4) -- The learning rate.
• betas (tuple(float, float), defaults to (0.9, 0.999)) -- The beta values are the decay rates of the first and second-order moment of the optimizer.
• weight_decay (float, defaults to 0) -- The weight decay value for the optimizer.
• optim_bits (int, defaults to 32) -- The number of bits of the optimizer state.
• args (object, defaults to None) -- An object with additional arguments.
• min_8bit_size (int, defaults to 4096) -- The minimum number of elements of the parameter tensors for 8-bit optimization.
• percentile_clipping (int, defaults to 100) -- Adapts clipping threshold automatically by tracking the last 100 gradient norms and clipping the gradient at a certain percentile to improve stability.
• block_wise (bool, defaults to True) -- Whether to independently quantize each block of tensors to reduce outlier effects and improve stability.0

Paged 32-bit Lion optimizer.