| --- |
| license: bsd-3-clause-clear |
| language: |
| - en |
| tags: |
| - optimization |
| - optimizer |
| - PyTorch |
| - Julia |
| paper_id: "zenodo:20060806" |
| --- |
| |
| # SurpriseOpt |
|
|
| We introduce SurpriseOpt, an optimization framework that replaces constant exponential decay with state-dependent |
| adaptive interpolation. The algorithm detects “surprises” as ratios of gradient magnitude and second-moment |
| magnitude and modulates the effective inertia of first and second moments via adaptive gating functions. |
| SurpriseOpt furthermore features a mechanism to escape plateaus in the loss function landscape: It accumulates |
| information about recent low surprises as “boredom” and adapts the learning rate accordingly. This boredom feature can |
| be added to any first-order optimizer. We demonstrate that SurpriseOpt can converge several times faster than Adam |
| across various tasks. |
|
|
|
|
| ## Description |
|
|
| This repository provides the Hugging Face entry for the research paper "SurpriseOpt: An Adaptive First-Order Optimizer Driven by Boredom". |
|
|
| The scientific paper is published as preprint at [10.5281/zenodo.20060806](https://doi.org/10.5281/zenodo.20060806) |
|
|
| The source code for both Julia and PyTorch is hosted on Codeberg: [https://codeberg.org/Soloof/SurpriseOpt](https://codeberg.org/Soloof/SurpriseOpt) </br>It contains a reference implementation of the algorithm to use it directly in Julia via [Flux](https://en.wikipedia.org/wiki/Flux_(machine-learning_framework)) or [Lux](https://lux.csail.mit.edu/stable/) or in Python via [PyTorch](https://en.wikipedia.org/wiki/PyTorch). |
|
|
| ## How to test the optimizer |
|
|
| Please [follow the description](https://codeberg.org/Soloof/SurpriseOpt#how-to-do-use-the-reference-implementation-with-julia) at the project repository. |
|
|
