# Lion [Lion (Evolved Sign Momentum)](https://hf.co/papers/2302.06675) is a unique optimizer that uses the sign of the gradient to determine the update direction of the momentum. This makes Lion more memory-efficient and faster than `AdamW` which tracks and store the first and second-order moments. ## Lion[[api-class]][[bitsandbytes.optim.Lion]] #### bitsandbytes.optim.Lion[[bitsandbytes.optim.Lion]] [Source](https://github.com/bitsandbytes-foundation/bitsandbytes/blob/main/bitsandbytes/optim/lion.py#L8) __init__bitsandbytes.optim.Lion.__init__https://github.com/bitsandbytes-foundation/bitsandbytes/blob/main/bitsandbytes/optim/lion.py#L9[{"name": "params", "val": ""}, {"name": "lr", "val": " = 0.0001"}, {"name": "betas", "val": " = (0.9, 0.99)"}, {"name": "weight_decay", "val": " = 0"}, {"name": "optim_bits", "val": " = 32"}, {"name": "args", "val": " = None"}, {"name": "min_8bit_size", "val": " = 4096"}, {"name": "is_paged", "val": " = False"}]- **params** (`torch.tensor`) -- The input parameters to optimize. - **lr** (`float`, defaults to 1e-4) -- The learning rate. - **betas** (`tuple(float, float)`, defaults to (0.9, 0.999)) -- The beta values are the decay rates of the first and second-order moment of the optimizer. - **weight_decay** (`float`, defaults to 0) -- The weight decay value for the optimizer. - **optim_bits** (`int`, defaults to 32) -- The number of bits of the optimizer state. - **args** (`object`, defaults to `None`) -- An object with additional arguments. - **min_8bit_size** (`int`, defaults to 4096) -- The minimum number of elements of the parameter tensors for 8-bit optimization. - **is_paged** (`bool`, defaults to `False`) -- Whether the optimizer is a paged optimizer or not.0 Base Lion optimizer. **Parameters:** params (`torch.tensor`) : The input parameters to optimize. lr (`float`, defaults to 1e-4) : The learning rate. betas (`tuple(float, float)`, defaults to (0.9, 0.999)) : The beta values are the decay rates of the first and second-order moment of the optimizer. weight_decay (`float`, defaults to 0) : The weight decay value for the optimizer. optim_bits (`int`, defaults to 32) : The number of bits of the optimizer state. args (`object`, defaults to `None`) : An object with additional arguments. min_8bit_size (`int`, defaults to 4096) : The minimum number of elements of the parameter tensors for 8-bit optimization. is_paged (`bool`, defaults to `False`) : Whether the optimizer is a paged optimizer or not. ## Lion8bit[[bitsandbytes.optim.Lion8bit]] #### bitsandbytes.optim.Lion8bit[[bitsandbytes.optim.Lion8bit]] [Source](https://github.com/bitsandbytes-foundation/bitsandbytes/blob/main/bitsandbytes/optim/lion.py#L55) __init__bitsandbytes.optim.Lion8bit.__init__https://github.com/bitsandbytes-foundation/bitsandbytes/blob/main/bitsandbytes/optim/lion.py#L56[{"name": "params", "val": ""}, {"name": "lr", "val": " = 0.0001"}, {"name": "betas", "val": " = (0.9, 0.99)"}, {"name": "weight_decay", "val": " = 0"}, {"name": "args", "val": " = None"}, {"name": "min_8bit_size", "val": " = 4096"}, {"name": "is_paged", "val": " = False"}]- **params** (`torch.tensor`) -- The input parameters to optimize. - **lr** (`float`, defaults to 1e-4) -- The learning rate. - **betas** (`tuple(float, float)`, defaults to (0.9, 0.999)) -- The beta values are the decay rates of the first and second-order moment of the optimizer. - **weight_decay** (`float`, defaults to 0) -- The weight decay value for the optimizer. - **args** (`object`, defaults to `None`) -- An object with additional arguments. - **min_8bit_size** (`int`, defaults to 4096) -- The minimum number of elements of the parameter tensors for 8-bit optimization. - **is_paged** (`bool`, defaults to `False`) -- Whether the optimizer is a paged optimizer or not.0 8-bit Lion optimizer. **Parameters:** params (`torch.tensor`) : The input parameters to optimize. lr (`float`, defaults to 1e-4) : The learning rate. betas (`tuple(float, float)`, defaults to (0.9, 0.999)) : The beta values are the decay rates of the first and second-order moment of the optimizer. weight_decay (`float`, defaults to 0) : The weight decay value for the optimizer. args (`object`, defaults to `None`) : An object with additional arguments. min_8bit_size (`int`, defaults to 4096) : The minimum number of elements of the parameter tensors for 8-bit optimization. is_paged (`bool`, defaults to `False`) : Whether the optimizer is a paged optimizer or not. ## Lion32bit[[bitsandbytes.optim.Lion32bit]] #### bitsandbytes.optim.Lion32bit[[bitsandbytes.optim.Lion32bit]] [Source](https://github.com/bitsandbytes-foundation/bitsandbytes/blob/main/bitsandbytes/optim/lion.py#L99) __init__bitsandbytes.optim.Lion32bit.__init__https://github.com/bitsandbytes-foundation/bitsandbytes/blob/main/bitsandbytes/optim/lion.py#L100[{"name": "params", "val": ""}, {"name": "lr", "val": " = 0.0001"}, {"name": "betas", "val": " = (0.9, 0.99)"}, {"name": "weight_decay", "val": " = 0"}, {"name": "args", "val": " = None"}, {"name": "min_8bit_size", "val": " = 4096"}, {"name": "is_paged", "val": " = False"}]- **params** (`torch.tensor`) -- The input parameters to optimize. - **lr** (`float`, defaults to 1e-4) -- The learning rate. - **betas** (`tuple(float, float)`, defaults to (0.9, 0.999)) -- The beta values are the decay rates of the first and second-order moment of the optimizer. - **weight_decay** (`float`, defaults to 0) -- The weight decay value for the optimizer. - **args** (`object`, defaults to `None`) -- An object with additional arguments. - **min_8bit_size** (`int`, defaults to 4096) -- The minimum number of elements of the parameter tensors for 8-bit optimization. - **is_paged** (`bool`, defaults to `False`) -- Whether the optimizer is a paged optimizer or not.0 32-bit Lion optimizer. **Parameters:** params (`torch.tensor`) : The input parameters to optimize. lr (`float`, defaults to 1e-4) : The learning rate. betas (`tuple(float, float)`, defaults to (0.9, 0.999)) : The beta values are the decay rates of the first and second-order moment of the optimizer. weight_decay (`float`, defaults to 0) : The weight decay value for the optimizer. args (`object`, defaults to `None`) : An object with additional arguments. min_8bit_size (`int`, defaults to 4096) : The minimum number of elements of the parameter tensors for 8-bit optimization. is_paged (`bool`, defaults to `False`) : Whether the optimizer is a paged optimizer or not. ## PagedLion[[bitsandbytes.optim.PagedLion]] #### bitsandbytes.optim.PagedLion[[bitsandbytes.optim.PagedLion]] [Source](https://github.com/bitsandbytes-foundation/bitsandbytes/blob/main/bitsandbytes/optim/lion.py#L143) __init__bitsandbytes.optim.PagedLion.__init__https://github.com/bitsandbytes-foundation/bitsandbytes/blob/main/bitsandbytes/optim/lion.py#L144[{"name": "params", "val": ""}, {"name": "lr", "val": " = 0.0001"}, {"name": "betas", "val": " = (0.9, 0.99)"}, {"name": "weight_decay", "val": " = 0"}, {"name": "optim_bits", "val": " = 32"}, {"name": "args", "val": " = None"}, {"name": "min_8bit_size", "val": " = 4096"}]- **params** (`torch.tensor`) -- The input parameters to optimize. - **lr** (`float`, defaults to 1e-4) -- The learning rate. - **betas** (`tuple(float, float)`, defaults to (0.9, 0.999)) -- The beta values are the decay rates of the first and second-order moment of the optimizer. - **weight_decay** (`float`, defaults to 0) -- The weight decay value for the optimizer. - **optim_bits** (`int`, defaults to 32) -- The number of bits of the optimizer state. - **args** (`object`, defaults to `None`) -- An object with additional arguments. - **min_8bit_size** (`int`, defaults to 4096) -- The minimum number of elements of the parameter tensors for 8-bit optimization.0 Paged Lion optimizer. **Parameters:** params (`torch.tensor`) : The input parameters to optimize. lr (`float`, defaults to 1e-4) : The learning rate. betas (`tuple(float, float)`, defaults to (0.9, 0.999)) : The beta values are the decay rates of the first and second-order moment of the optimizer. weight_decay (`float`, defaults to 0) : The weight decay value for the optimizer. optim_bits (`int`, defaults to 32) : The number of bits of the optimizer state. args (`object`, defaults to `None`) : An object with additional arguments. min_8bit_size (`int`, defaults to 4096) : The minimum number of elements of the parameter tensors for 8-bit optimization. ## PagedLion8bit[[bitsandbytes.optim.PagedLion8bit]] #### bitsandbytes.optim.PagedLion8bit[[bitsandbytes.optim.PagedLion8bit]] [Source](https://github.com/bitsandbytes-foundation/bitsandbytes/blob/main/bitsandbytes/optim/lion.py#L187) __init__bitsandbytes.optim.PagedLion8bit.__init__https://github.com/bitsandbytes-foundation/bitsandbytes/blob/main/bitsandbytes/optim/lion.py#L188[{"name": "params", "val": ""}, {"name": "lr", "val": " = 0.0001"}, {"name": "betas", "val": " = (0.9, 0.99)"}, {"name": "weight_decay", "val": " = 0"}, {"name": "args", "val": " = None"}, {"name": "min_8bit_size", "val": " = 4096"}]- **params** (`torch.tensor`) -- The input parameters to optimize. - **lr** (`float`, defaults to 1e-4) -- The learning rate. - **betas** (`tuple(float, float)`, defaults to (0.9, 0.999)) -- The beta values are the decay rates of the first and second-order moment of the optimizer. - **weight_decay** (`float`, defaults to 0) -- The weight decay value for the optimizer. - **args** (`object`, defaults to `None`) -- An object with additional arguments. - **min_8bit_size** (`int`, defaults to 4096) -- The minimum number of elements of the parameter tensors for 8-bit optimization.0 Paged 8-bit Lion optimizer. **Parameters:** params (`torch.tensor`) : The input parameters to optimize. lr (`float`, defaults to 1e-4) : The learning rate. betas (`tuple(float, float)`, defaults to (0.9, 0.999)) : The beta values are the decay rates of the first and second-order moment of the optimizer. weight_decay (`float`, defaults to 0) : The weight decay value for the optimizer. args (`object`, defaults to `None`) : An object with additional arguments. min_8bit_size (`int`, defaults to 4096) : The minimum number of elements of the parameter tensors for 8-bit optimization. ## PagedLion32bit[[bitsandbytes.optim.PagedLion32bit]] #### bitsandbytes.optim.PagedLion32bit[[bitsandbytes.optim.PagedLion32bit]] [Source](https://github.com/bitsandbytes-foundation/bitsandbytes/blob/main/bitsandbytes/optim/lion.py#L228) __init__bitsandbytes.optim.PagedLion32bit.__init__https://github.com/bitsandbytes-foundation/bitsandbytes/blob/main/bitsandbytes/optim/lion.py#L229[{"name": "params", "val": ""}, {"name": "lr", "val": " = 0.0001"}, {"name": "betas", "val": " = (0.9, 0.99)"}, {"name": "weight_decay", "val": " = 0"}, {"name": "args", "val": " = None"}, {"name": "min_8bit_size", "val": " = 4096"}]- **params** (`torch.tensor`) -- The input parameters to optimize. - **lr** (`float`, defaults to 1e-4) -- The learning rate. - **betas** (`tuple(float, float)`, defaults to (0.9, 0.999)) -- The beta values are the decay rates of the first and second-order moment of the optimizer. - **weight_decay** (`float`, defaults to 0) -- The weight decay value for the optimizer. - **args** (`object`, defaults to `None`) -- An object with additional arguments. - **min_8bit_size** (`int`, defaults to 4096) -- The minimum number of elements of the parameter tensors for 8-bit optimization.0 Paged 32-bit Lion optimizer. **Parameters:** params (`torch.tensor`) : The input parameters to optimize. lr (`float`, defaults to 1e-4) : The learning rate. betas (`tuple(float, float)`, defaults to (0.9, 0.999)) : The beta values are the decay rates of the first and second-order moment of the optimizer. weight_decay (`float`, defaults to 0) : The weight decay value for the optimizer. args (`object`, defaults to `None`) : An object with additional arguments. min_8bit_size (`int`, defaults to 4096) : The minimum number of elements of the parameter tensors for 8-bit optimization.