Buckets:
| # LAMB | |
| [LAMB (Layerwise adaptive large batch optimization)](https://hf.co/papers/1904.00962) is an adaptive optimizer designed for training with large batch sizes to accelerate training, combining ideas from `LARS` and `Adam` to automatically scale the learning rate for each layer: | |
| - calculates a *trust ratio* between the weight and gradient norm in a layer and clips the ratio to prevent overly large or small updates | |
| - updates weights with the first and second-moments | |
| ## LAMB[[api-class]][[bitsandbytes.optim.LAMB]] | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>class bitsandbytes.optim.LAMB</name><anchor>bitsandbytes.optim.LAMB</anchor><source>https://github.com/bitsandbytes-foundation/bitsandbytes/blob/v0.48.2/bitsandbytes/optim/lamb.py#L8</source><parameters>[{"name": "params", "val": ""}, {"name": "lr", "val": " = 0.001"}, {"name": "bias_correction", "val": " = True"}, {"name": "betas", "val": " = (0.9, 0.999)"}, {"name": "eps", "val": " = 1e-08"}, {"name": "weight_decay", "val": " = 0"}, {"name": "amsgrad", "val": " = False"}, {"name": "adam_w_mode", "val": " = True"}, {"name": "optim_bits", "val": " = 32"}, {"name": "args", "val": " = None"}, {"name": "min_8bit_size", "val": " = 4096"}, {"name": "percentile_clipping", "val": " = 100"}, {"name": "block_wise", "val": " = False"}, {"name": "max_unorm", "val": " = 1.0"}]</parameters></docstring> | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>__init__</name><anchor>bitsandbytes.optim.LAMB.__init__</anchor><source>https://github.com/bitsandbytes-foundation/bitsandbytes/blob/v0.48.2/bitsandbytes/optim/lamb.py#L9</source><parameters>[{"name": "params", "val": ""}, {"name": "lr", "val": " = 0.001"}, {"name": "bias_correction", "val": " = True"}, {"name": "betas", "val": " = (0.9, 0.999)"}, {"name": "eps", "val": " = 1e-08"}, {"name": "weight_decay", "val": " = 0"}, {"name": "amsgrad", "val": " = False"}, {"name": "adam_w_mode", "val": " = True"}, {"name": "optim_bits", "val": " = 32"}, {"name": "args", "val": " = None"}, {"name": "min_8bit_size", "val": " = 4096"}, {"name": "percentile_clipping", "val": " = 100"}, {"name": "block_wise", "val": " = False"}, {"name": "max_unorm", "val": " = 1.0"}]</parameters><paramsdesc>- **params** (`torch.tensor`) -- | |
| The input parameters to optimize. | |
| - **lr** (`float`, defaults to 1e-3) -- | |
| The learning rate. | |
| - **bias_correction** (`bool`, defaults to `True`) -- | |
| Whether to apply bias correction to the first and second-order moments. | |
| - **betas** (`tuple(float, float)`, defaults to (0.9, 0.999)) -- | |
| The beta values are the decay rates of the first and second-order moment of the optimizer. | |
| - **eps** (`float`, defaults to 1e-8) -- | |
| The epsilon value prevents division by zero in the optimizer. | |
| - **weight_decay** (`float`, defaults to 1e-2) -- | |
| The weight decay value for the optimizer. | |
| - **amsgrad** (`bool`, defaults to `False`) -- | |
| Whether to use the [AMSGrad](https://hf.co/papers/1904.09237) variant of Adam that uses the maximum of past squared gradients instead. | |
| - **adam_w_mode** (`bool`, defaults to `True`) -- | |
| Whether to use the AdamW variant. | |
| - **optim_bits** (`int`, defaults to 32) -- | |
| The number of bits of the optimizer state. | |
| - **args** (`object`, defaults to `None`) -- | |
| An object with additional arguments. | |
| - **min_8bit_size** (`int`, defaults to 4096) -- | |
| The minimum number of elements of the parameter tensors for 8-bit optimization. | |
| - **percentile_clipping** (`int`, defaults to 100) -- | |
| Adapts clipping threshold automatically by tracking the last 100 gradient norms and clipping the gradient at a certain percentile to improve stability. | |
| - **block_wise** (`bool`, defaults to `True`) -- | |
| Whether to independently quantize each block of tensors to reduce outlier effects and improve stability. | |
| - **max_unorm** (`float`, defaults to 1.0) -- | |
| The maximum gradient norm.</paramsdesc><paramgroups>0</paramgroups></docstring> | |
| Base LAMB optimizer. | |
| </div></div> | |
| ## LAMB8bit[[bitsandbytes.optim.LAMB8bit]] | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>class bitsandbytes.optim.LAMB8bit</name><anchor>bitsandbytes.optim.LAMB8bit</anchor><source>https://github.com/bitsandbytes-foundation/bitsandbytes/blob/v0.48.2/bitsandbytes/optim/lamb.py#L75</source><parameters>[{"name": "params", "val": ""}, {"name": "lr", "val": " = 0.001"}, {"name": "bias_correction", "val": " = True"}, {"name": "betas", "val": " = (0.9, 0.999)"}, {"name": "eps", "val": " = 1e-08"}, {"name": "weight_decay", "val": " = 0"}, {"name": "amsgrad", "val": " = False"}, {"name": "adam_w_mode", "val": " = True"}, {"name": "args", "val": " = None"}, {"name": "min_8bit_size", "val": " = 4096"}, {"name": "percentile_clipping", "val": " = 100"}, {"name": "block_wise", "val": " = False"}, {"name": "max_unorm", "val": " = 1.0"}]</parameters></docstring> | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>__init__</name><anchor>bitsandbytes.optim.LAMB8bit.__init__</anchor><source>https://github.com/bitsandbytes-foundation/bitsandbytes/blob/v0.48.2/bitsandbytes/optim/lamb.py#L76</source><parameters>[{"name": "params", "val": ""}, {"name": "lr", "val": " = 0.001"}, {"name": "bias_correction", "val": " = True"}, {"name": "betas", "val": " = (0.9, 0.999)"}, {"name": "eps", "val": " = 1e-08"}, {"name": "weight_decay", "val": " = 0"}, {"name": "amsgrad", "val": " = False"}, {"name": "adam_w_mode", "val": " = True"}, {"name": "args", "val": " = None"}, {"name": "min_8bit_size", "val": " = 4096"}, {"name": "percentile_clipping", "val": " = 100"}, {"name": "block_wise", "val": " = False"}, {"name": "max_unorm", "val": " = 1.0"}]</parameters><paramsdesc>- **params** (`torch.tensor`) -- | |
| The input parameters to optimize. | |
| - **lr** (`float`, defaults to 1e-3) -- | |
| The learning rate. | |
| - **bias_correction** (`bool`, defaults to `True`) -- | |
| Whether to apply bias correction to the first and second-order moments. | |
| - **betas** (`tuple(float, float)`, defaults to (0.9, 0.999)) -- | |
| The beta values are the decay rates of the first and second-order moment of the optimizer. | |
| - **eps** (`float`, defaults to 1e-8) -- | |
| The epsilon value prevents division by zero in the optimizer. | |
| - **weight_decay** (`float`, defaults to 1e-2) -- | |
| The weight decay value for the optimizer. | |
| - **amsgrad** (`bool`, defaults to `False`) -- | |
| Whether to use the [AMSGrad](https://hf.co/papers/1904.09237) variant of Adam that uses the maximum of past squared gradients instead. | |
| - **adam_w_mode** (`bool`, defaults to `True`) -- | |
| Whether to use the AdamW variant. | |
| - **args** (`object`, defaults to `None`) -- | |
| An object with additional arguments. | |
| - **min_8bit_size** (`int`, defaults to 4096) -- | |
| The minimum number of elements of the parameter tensors for 8-bit optimization. | |
| - **percentile_clipping** (`int`, defaults to 100) -- | |
| Adapts clipping threshold automatically by tracking the last 100 gradient norms and clipping the gradient at a certain percentile to improve stability. | |
| - **block_wise** (`bool`, defaults to `True`) -- | |
| Whether to independently quantize each block of tensors to reduce outlier effects and improve stability. | |
| - **max_unorm** (`float`, defaults to 1.0) -- | |
| The maximum gradient norm.</paramsdesc><paramgroups>0</paramgroups></docstring> | |
| 8-bit LAMB optimizer. | |
| </div></div> | |
| ## LAMB32bit[[bitsandbytes.optim.LAMB32bit]] | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>class bitsandbytes.optim.LAMB32bit</name><anchor>bitsandbytes.optim.LAMB32bit</anchor><source>https://github.com/bitsandbytes-foundation/bitsandbytes/blob/v0.48.2/bitsandbytes/optim/lamb.py#L139</source><parameters>[{"name": "params", "val": ""}, {"name": "lr", "val": " = 0.001"}, {"name": "bias_correction", "val": " = True"}, {"name": "betas", "val": " = (0.9, 0.999)"}, {"name": "eps", "val": " = 1e-08"}, {"name": "weight_decay", "val": " = 0"}, {"name": "amsgrad", "val": " = False"}, {"name": "adam_w_mode", "val": " = True"}, {"name": "args", "val": " = None"}, {"name": "min_8bit_size", "val": " = 4096"}, {"name": "percentile_clipping", "val": " = 100"}, {"name": "block_wise", "val": " = False"}, {"name": "max_unorm", "val": " = 1.0"}]</parameters></docstring> | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>__init__</name><anchor>bitsandbytes.optim.LAMB32bit.__init__</anchor><source>https://github.com/bitsandbytes-foundation/bitsandbytes/blob/v0.48.2/bitsandbytes/optim/lamb.py#L140</source><parameters>[{"name": "params", "val": ""}, {"name": "lr", "val": " = 0.001"}, {"name": "bias_correction", "val": " = True"}, {"name": "betas", "val": " = (0.9, 0.999)"}, {"name": "eps", "val": " = 1e-08"}, {"name": "weight_decay", "val": " = 0"}, {"name": "amsgrad", "val": " = False"}, {"name": "adam_w_mode", "val": " = True"}, {"name": "args", "val": " = None"}, {"name": "min_8bit_size", "val": " = 4096"}, {"name": "percentile_clipping", "val": " = 100"}, {"name": "block_wise", "val": " = False"}, {"name": "max_unorm", "val": " = 1.0"}]</parameters><paramsdesc>- **params** (`torch.tensor`) -- | |
| The input parameters to optimize. | |
| - **lr** (`float`, defaults to 1e-3) -- | |
| The learning rate. | |
| - **bias_correction** (`bool`, defaults to `True`) -- | |
| Whether to apply bias correction to the first and second-order moments. | |
| - **betas** (`tuple(float, float)`, defaults to (0.9, 0.999)) -- | |
| The beta values are the decay rates of the first and second-order moment of the optimizer. | |
| - **eps** (`float`, defaults to 1e-8) -- | |
| The epsilon value prevents division by zero in the optimizer. | |
| - **weight_decay** (`float`, defaults to 1e-2) -- | |
| The weight decay value for the optimizer. | |
| - **amsgrad** (`bool`, defaults to `False`) -- | |
| Whether to use the [AMSGrad](https://hf.co/papers/1904.09237) variant of Adam that uses the maximum of past squared gradients instead. | |
| - **adam_w_mode** (`bool`, defaults to `True`) -- | |
| Whether to use the AdamW variant. | |
| - **args** (`object`, defaults to `None`) -- | |
| An object with additional arguments. | |
| - **min_8bit_size** (`int`, defaults to 4096) -- | |
| The minimum number of elements of the parameter tensors for 8-bit optimization. | |
| - **percentile_clipping** (`int`, defaults to 100) -- | |
| Adapts clipping threshold automatically by tracking the last 100 gradient norms and clipping the gradient at a certain percentile to improve stability. | |
| - **block_wise** (`bool`, defaults to `True`) -- | |
| Whether to independently quantize each block of tensors to reduce outlier effects and improve stability. | |
| - **max_unorm** (`float`, defaults to 1.0) -- | |
| The maximum gradient norm.</paramsdesc><paramgroups>0</paramgroups></docstring> | |
| 32-bit LAMB optimizer. | |
| </div></div> | |
| <EditOnGithub source="https://github.com/bitsandbytes-foundation/bitsandbytes/blob/main/docs/source/reference/optim/lamb.mdx" /> |
Xet Storage Details
- Size:
- 11.1 kB
- Xet hash:
- 383a39d9b363d314dcf06e85c380cbfa6834f8129c09f412cfc0db8be66cfc19
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.