Delete optimizer

optimizer/How-to-Use-EmoNAVI(ENG).txt

@@ -1,59 +0,0 @@
-How to Use EmoNAVI, EmoFact, EmoLynx, and EmoClan
-
-The EmoNavi series is designed to be scheduler-independent. This means you don't necessarily need a scheduler, and even if you use one, it's generally fine because the system automatically adjusts its settings to manage the learning process.
-
-However, if your goal is to grasp fine details quickly, we recommend using your preferred scheduler, such as Cosine-Restart.
-
-Understanding the Learning Rate: It's Not Just Intensity
-
-Many people think of the learning rate setting as "learning intensity." However, it's actually more like a filter—it dictates how the VAE's latent space is perceived.
-
-Imagine a translucent plastic plate. When the learning rate is high, the image you see through this plate is a blurrier "overview," appearing as a rough distribution of light or large masses. When the learning rate is low, the image is "detailed," with increased transparency and clearer representation. In essence, the learning rate can be thought of as "resolution"—it's like adjusting the degree of blurriness by controlling the plate's transparency.
-
-This explains why a high learning rate is better and faster for grasping overviews. With less information to process (because the details are blurred out), the system learns basic patterns quickly. Conversely, a low learning rate involves more information, thus requiring more time to fully grasp everything. If the learning rate is too low, there's an overwhelming amount of information, leading to the training period ending without sufficient learning, and resulting in subpar outcomes.
-
-It's important to note that this concept of "learning rate" isn't exclusive to the EmoNavi series; it applies to other optimizers as well. Please keep this in mind for your future training endeavors.
-
-EmoNavi Series: Smart Learning with or Without Schedulers
-
-As mentioned earlier, if you use a scheduler with the EmoNavi series, you might capture details earlier than with a constant learning rate.
-
-Alternatively, you can skip the scheduler entirely and opt for "additional training" sessions at a lower learning rate. This is easily done without needing to manage transfer parameters, which is a simplified feature not commonly found in other optimizers.
-
-The EmoNavi series is designed to prevent overfitting even when running at a constant learning rate. It automatically adjusts to avoid exceeding a certain threshold. Therefore, it won't learn more than necessary. If it detects that it's nearing the overfitting zone, it will adjust. After learning the general outline, it's not that it stops learning; rather, it learns only what's necessary, which might make it seem like progress has slowed compared to the initial overview-learning phase.
-
-If you find that your training isn't progressing beyond a certain point, try an additional training session with a lower learning rate. This often allows the system to rapidly absorb all the finer details.
-
-Conclusion
-
-We hope this explanation helps you acquire valuable know-how for setting up your training, not just for the EmoNavi series. We believe it will be beneficial to all of you. Thank you for reading to the end.
-
-postscript
-
-I'd like to explain the learning rate in an easy-to-understand way, so you can truly grasp its concept.
-
-You can think of the learning rate like reading speed.
-Imagine this: a high learning rate is like skim reading (or speed reading), while a low learning rate is like perusing (or close reading).
-
-The scheduler manages this, much like a learning schedule.
-EmoNAVI has a "shadow" function that encourages the model to review and reflect on its own learning progress.
-With EmoNAVI, you have a choice: you can allow external guidance to determine the learning path and let the model's autonomy supplement it, or you can rely solely on its autonomy.
-
-Here are some other analogies:
-High learning rate: Like shooting a photo from a distance, giving you an overview where details are fuzzy.
-Low learning rate: Like shooting up close, capturing accurate details.
-Think of autofocus as being handled by the scheduler and the "shadow" function.
-From another perspective:
-
-When aiming for detailed expressions, you can also consider increasing the amount of training data or increasing the number of iterations.
-As the number of iterations increases, detailed features are gradually accumulated.
-
-However, color representation largely depends on the performance of the VAE (Variational Autoencoder).
-To accurately reflect colors, the only options are to improve the VAE's performance itself or to use teacher data that correctly reflects colors.
-
-Furthermore, the "shadow" function also acts like an autofocus system.
-It's a mechanism that allows the model to review and reflect on its own learning, essentially learning from its own experience.
-This means it captures one feature, learns from it, then identifies another, and the process repeats.
-Consequently, its "focus" (or understanding) continuously evolves and adapts.
-
-That concludes the additional explanation. Thank you for reading to the end!

optimizer/Use_Kohya-sd-script.txt

@@ -1,49 +0,0 @@
-Kohya-sd-script での使用法
-
-これら Emoシリーズ を Kohya-sd-script で簡単につかうには、
-このフォルダをこのまま Kohya-sd-script の "sd-script" フォルダに配置してください
-
-sd-script/optimizer
-
-この配置にした場合、
-
---optimizer_type=optimizer.emonavi.EmoNavi
---optimizer_type=optimizer.emofact.EmoFact
---optimizer_type=optimizer.emolynx.EmoLynx
---optimizer_type=optimizer.emoclan.EmoClan
-
-このように指定するだけで各Optimizerを利用できます(いずれかひとつを指定してください)
----
-Kohya-sd-script の柔軟な構成により、これらをすぐ試せます
-Kohya-sd-script の開発者と協力者の皆さまに深く感謝します
-Kohya-sd-script: https://github.com/kohya-ss/sd-scripts
-
-Emoシリーズは、Adam、Adafactor、Lion、Tiger、等から多くを学びました。  
-これらの後継ではなく独自の思想や設計による"感情機構"というアプローチにより構築されています。  
-汎用性・自律性・適応性を重視し新たな最適化や効率化や簡易化を追求しています。  
-この開発において先人たちの知見に深く感謝しつつ今後も新しい可能性を探究します。 
-
-
-Usage with Kohya-sd-script
-
-To easily use these Emo series with Kohya-sd-script,
-simply place this folder as-is into the "sd-scripts" folder within your Kohya-sd-script installation:
-
-sd-scripts/optimizer
-
-With this setup,
-
---optimizer_type=optimizer.emonavi.EmoNavi
---optimizer_type=optimizer.emofact.EmoFact
---optimizer_type=optimizer.emolynx.EmoLynx
---optimizer_type=optimizer.emoclan.EmoClan
-
-You can utilize each optimizer by simply specifying one of the above.
-
-Thanks to the flexible configuration of Kohya-sd-script, you can try these out right away. We extend our deepest gratitude to the developers and contributors of Kohya-sd-script:
-Kohya-sd-script: https://github.com/kohya-ss/sd-scripts
-
-The Emo series has learned much from Adam, Adafactor, Lion, and Tiger.  
-Rather than being their successors, it is built upon a unique philosophy and design approach centered on "emotional mechanisms".  
-It prioritizes generality, autonomy, and adaptability in pursuit of new paths for optimization, efficiency, and simplicity.  
-In its development, we deeply appreciate the insights of those who came before us—and continue to explore new possibilities beyond them.  

optimizer/emoclan.py

@@ -1,252 +0,0 @@
-import torch
-from torch.optim import Optimizer
-import math
-from typing import Callable, Union, Dict, Any, Tuple
-
-# Helper function
-def exists(val):
-    return val is not None
-
-class EmoClan(Optimizer):
-    def __init__(self, params: Union[list, torch.nn.Module], 
-                 lr: float = 1e-3, 
-                 betas: Tuple[float, float] = (0.9, 0.999), 
-                 eps: float = 1e-8, 
-                 weight_decay: float = 0.01,
-                 lynx_betas: Tuple[float, float] = (0.9, 0.99), # Lynx 固有の beta
-                 decoupled_weight_decay: bool = False
-                ):
-        
-        if not 0.0 <= lr:
-            raise ValueError(f"Invalid learning rate: {lr}")
-        if not 0.0 <= eps:
-            raise ValueError(f"Invalid epsilon value: {eps}")
-        if not 0.0 <= betas[0] < 1.0:
-            raise ValueError(f"Invalid beta parameter at index 0: {betas[0]}")
-        if not 0.0 <= betas[1] < 1.0:
-            raise ValueError(f"Invalid beta parameter at index 1: {betas[1]}")
-        
-        # Lynx の betas もバリデーション
-        if not 0.0 <= lynx_betas[0] < 1.0:
-            raise ValueError(f"Invalid lynx_beta parameter at index 0: {lynx_betas[0]}")
-        if not 0.0 <= lynx_betas[1] < 1.0:
-            raise ValueError(f"Invalid lynx_beta parameter at index 1: {lynx_betas[1]}")
-
-        defaults = dict(lr=lr, betas=betas, eps=eps, weight_decay=weight_decay,
-                        lynx_betas=lynx_betas, decoupled_weight_decay=decoupled_weight_decay)
-        super().__init__(params, defaults)
-        
-        self._init_lr = lr # decoupled weight decay のために保存 (Lynx用)
-        self.should_stop = False # 全体の停止フラグ
-
-    # --- 感情機構 (Emotion Mechanism) ---
-    def _update_ema(self, param_state: Dict[str, Any], loss_val: float) -> Dict[str, float]:
-        """損失値に基づいて短期・長期 EMA を更新"""
-        # param_state は各パラメータの state['ema'] を保持する
-        ema = param_state.setdefault('ema', {'short': loss_val, 'long': loss_val})
-        ema['short'] = 0.3 * loss_val + 0.7 * ema['short']
-        ema['long']  = 0.01 * loss_val + 0.99 * ema['long']
-        return ema
-
-    def _compute_scalar(self, ema: Dict[str, float]) -> float:
-        """EMA の差分から感情スカラー値を生成"""
-        diff = ema['short'] - ema['long']
-        return math.tanh(5 * diff)
-
-    def _decide_ratio(self, scalar: float) -> float:
-        """感情スカラーに基づいて Shadow の混合比率を決定"""
-        if scalar > 0.6:
-            return 0.7 + 0.2 * scalar # 0.7～0.9
-        elif scalar < -0.6:
-            return 0.1
-        elif abs(scalar) > 0.3: # >0.3 かつ <=0.6 の場合
-            return 0.3
-        return 0.0
-
-    # --- 各最適化器のコアな勾配更新ロジック (プライベートメソッドとして統合) ---
-
-    def _lynx_update(
-        self, 
-        p: torch.Tensor, 
-        grad: torch.Tensor, 
-        param_state: Dict[str, Any], 
-        lr: float, 
-        beta1: float, 
-        beta2: float, 
-        wd_actual: float
-    ):
-        """EmoLynx のコアな勾配更新ロジック"""
-        # Stepweight decay: p.data = p.data * (1 - lr * wd)
-        p.data.mul_(1. - lr * wd_actual)
-
-        # Lynx 固有の EMA 状態は param_state に保持
-        if 'exp_avg_lynx' not in param_state:
-            param_state['exp_avg_lynx'] = torch.zeros_like(p)
-        exp_avg = param_state['exp_avg_lynx']
-
-        # 勾配ブレンド
-        blended_grad = grad.mul(1. - beta1).add_(exp_avg, alpha=beta1)
-        
-        # 符号ベースの更新
-        p.data.add_(blended_grad.sign_(), alpha = -lr)
-
-        # exp_avg 更新
-        exp_avg.mul_(beta2).add_(grad, alpha = 1. - beta2)
-
-    def _navi_update(
-        self, 
-        p: torch.Tensor, 
-        grad: torch.Tensor, 
-        param_state: Dict[str, Any], 
-        lr: float, 
-        betas: Tuple[float, float], 
-        eps: float, 
-        weight_decay: float
-    ):
-        """EmoNavi のコアな勾配更新ロジック"""
-        beta1, beta2 = betas
-
-        exp_avg = param_state.setdefault('exp_avg_navi', torch.zeros_like(p.data))
-        exp_avg_sq = param_state.setdefault('exp_avg_sq_navi', torch.zeros_like(p.data))
-
-        exp_avg.mul_(beta1).add_(grad, alpha=1 - beta1)
-        exp_avg_sq.mul_(beta2).addcmul_(grad, grad, value=1 - beta2)
-        denom = exp_avg_sq.sqrt().add_(eps)
-
-        # Weight decay (標準的手法)
-        if weight_decay:
-            p.data.add_(p.data, alpha=-weight_decay * lr) 
-
-        p.data.addcdiv_(exp_avg, denom, value=-lr)
-
-    def _fact_update(
-        self, 
-        p: torch.Tensor, 
-        grad: torch.Tensor, 
-        param_state: Dict[str, Any], 
-        lr: float, 
-        betas: Tuple[float, float], # beta2 は現���使われないが互換性のため残す (1D勾配で使用)
-        eps: float, 
-        weight_decay: float
-    ):
-        """EmoFact のコアな勾配更新ロジック (Adafactor ライク)"""
-        beta1, beta2 = betas
-
-        if grad.dim() >= 2:
-            # 行と列の2乗平均を計算 (分散の軽量な近似)
-            r_sq = torch.mean(grad * grad, dim=tuple(range(1, grad.dim())), keepdim=True).add_(eps)
-            c_sq = torch.mean(grad * grad, dim=0, keepdim=True).add_(eps)
-
-            param_state.setdefault('exp_avg_r_fact', torch.zeros_like(r_sq)).mul_(beta1).add_(torch.sqrt(r_sq), alpha=1 - beta1)
-            param_state.setdefault('exp_avg_c_fact', torch.zeros_like(c_sq)).mul_(beta1).add_(torch.sqrt(c_sq), alpha=1 - beta1)
-            
-            # 再構築した近似勾配の平方根の積で正規化
-            denom = torch.sqrt(param_state['exp_avg_r_fact'] * param_state['exp_avg_c_fact']).add_(eps)
-            update_term = grad / denom
-
-        else: # 1次元(ベクトル)の勾配補正
-            exp_avg = param_state.setdefault('exp_avg_fact', torch.zeros_like(p.data))
-            exp_avg_sq = param_state.setdefault('exp_avg_sq_fact', torch.zeros_like(p.data))
-            
-            exp_avg.mul_(beta1).add_(grad, alpha=1 - beta1)
-            exp_avg_sq.mul_(beta2).addcmul_(grad, grad, value=1 - beta2) # beta2 をここでは使用
-            denom = exp_avg_sq.sqrt().add_(eps)
-            update_term = exp_avg / denom
-
-        # 最終的なパラメータ更新 (decoupled weight decayも適用)
-        p.data.add_(p.data, alpha=-weight_decay * lr) 
-        p.data.add_(update_term, alpha=-lr)
-
-
-    @torch.no_grad()
-    def step(self, closure: Callable | None = None):
-        loss = None
-        if exists(closure):
-            with torch.enable_grad():
-                loss = closure()
-        loss_val = loss.item() if loss is not None else 0.0
-
-        # 全体の scalar_hist を EmoClan インスタンスで管理
-        global_scalar_hist = self.state.setdefault('global_scalar_hist', [])
-        
-        # 全体としての感情EMA状態を self.state に保持し、現在の感情スカラーを計算
-        global_ema_state = self.state.setdefault('global_ema', {'short': loss_val, 'long': loss_val})
-        global_ema_state['short'] = 0.3 * loss_val + 0.7 * global_ema_state['short']
-        global_ema_state['long'] = 0.01 * loss_val + 0.99 * global_ema_state['long']
-        current_global_scalar = self._compute_scalar(global_ema_state)
-        
-        # global_scalar_hist に現在の感情スカラーを追加
-        global_scalar_hist.append(current_global_scalar)
-        if len(global_scalar_hist) > 32:
-            global_scalar_hist.pop(0)
-
-
-        for group in self.param_groups:
-            lr = group['lr']
-            wd = group['weight_decay']
-            eps = group['eps']
-            decoupled_wd = group['decoupled_weight_decay']
-            
-            lynx_beta1, lynx_beta2 = group['lynx_betas']
-            navi_fact_betas = group['betas'] # Navi/Fact 共通の beta を使用 (デフォルトの betas)
-            
-            # Lynx の decoupled_wd のための _wd_actual 計算
-            _wd_actual_lynx = wd
-            if decoupled_wd:
-                _wd_actual_lynx /= self._init_lr
-
-            for p in group['params']:
-                if p.grad is None:
-                    continue
-
-                grad = p.grad.data
-                param_state = self.state[p] # 各パラメータごとの状態
-
-                # --- 各パラメータごとの感情機構の更新と Shadow 処理 ---
-                # 各パラメータの state['ema'] は、それぞれの loss_val (全体で共通) を元に更新される
-                # ただし、現状の loss_val はクロージャから受け取った単一の値なので、
-                # 各パラメータ固有の「感情」を定義するより、全体としての感情が使われることになる。
-                param_ema = self._update_ema(param_state, loss_val) 
-                param_scalar = self._compute_scalar(param_ema) # 各パラメータ固有のスカラー
-
-                ratio = self._decide_ratio(param_scalar) # 各パラメータ固有の ratio
-
-                if ratio > 0:
-                    if 'shadow' not in param_state:
-                        param_state['shadow'] = p.data.clone()
-                    else:
-                        # Shadow を現在値にブレンド
-                        p.data.mul_(1 - ratio).add_(param_state['shadow'], alpha=ratio)
-                    # Shadow を現在値に追従させる
-                    param_state['shadow'].lerp_(p.data, 0.05)
-
-                # --- 最適化器の選択と勾配更新 ---
-                # 現在のglobal_scalar_histに記録された全体としての感情スカラーに基づいてフェーズを判断
-                # global_scalar が [-0.3, 0.3] の範囲にある場合は Navi
-                # global_scalar > 0.3 の場合は Lynx
-                # global_scalar < -0.3 の場合は Fact
-                if current_global_scalar > 0.3: # 序盤・過学習傾向時
-                    self._lynx_update(p, grad, param_state, lr, lynx_beta1, lynx_beta2, _wd_actual_lynx)
-                elif current_global_scalar < -0.3: # 終盤・発散傾向時
-                    self._fact_update(p, grad, param_state, lr, navi_fact_betas, eps, wd)
-                else: # -0.3 <= current_global_scalar <= 0.3 の中盤
-                    self._navi_update(p, grad, param_state, lr, navi_fact_betas, eps, wd)
-
-        # Early Stop判断
-        # global_scalar_hist の評価
-        if len(global_scalar_hist) >= 32:
-            buf = global_scalar_hist
-            avg_abs = sum(abs(s) for s in buf) / len(buf)
-            std = sum((s - sum(buf)/len(buf))**2 for s in buf) / len(buf)
-            if avg_abs < 0.05 and std < 0.005:
-                self.should_stop = True # 外部からこれを見て判断可
-
-        return loss
-
-"""
-Emoシリーズは、Adam、Adafactor、Lion、Tiger、等から多くを学びました。  
-この開発において先人たちの知見に深く感謝しつつ今後も新しい可能性を探究します。 
-The Emo series has learned much from Adam, Adafactor, Lion, and Tiger.  
-Rather than being their successors,  
-In its development, we deeply appreciate the insights of those who came before us—and continue to explore new possibilities beyond them.  
-"""

optimizer/emofact.py

@@ -1,117 +0,0 @@
-import torch
-from torch.optim import Optimizer
-import math
-
-class EmoFact(Optimizer):
-    # クラス定義＆初期化
-    def __init__(self, params, lr=1e-3, betas=(0.9, 0.999),
-                 eps=1e-8, weight_decay=0.01):
-        defaults = dict(lr=lr, betas=betas, eps=eps, weight_decay=weight_decay)
-        super().__init__(params, defaults)
-
-    # 感情EMA更新(緊張と安静)
-    def _update_ema(self, state, loss_val):
-        ema = state.setdefault('ema', {})
-        ema['short'] = 0.3 * loss_val + 0.7 * ema.get('short', loss_val)
-        ema['long'] = 0.01 * loss_val + 0.99 * ema.get('long', loss_val)
-        return ema
-
-    # 感情スカラー値生成(EMA差分、滑らかな非線形スカラー、tanh 5 * diff で鋭敏さ強調)
-    def _compute_scalar(self, ema):
-        diff = ema['short'] - ema['long']
-        return math.tanh(5 * diff)
-
-    # Shadow混合比率(> 0.6：70〜90%、 < 0.6：10%、 > 0.3：30%、 平時：0%)
-    def _decide_ratio(self, scalar):
-        if scalar > 0.6:
-            return 0.7 + 0.2 * scalar
-        elif scalar < -0.6:
-            return 0.1
-        elif abs(scalar) > 0.3:
-            return 0.3
-        return 0.0
-
-    # 損失取得(損失値 loss_val を数値化、感情判定に使用、存在しないパラメータ(更新不要)はスキップ)
-    @torch.no_grad()
-    def step(self, closure=None):
-        loss = closure() if closure is not None else None
-        loss_val = loss.item() if loss is not None else 0.0
-
-        for group in self.param_groups:
-            for p in group['params']:
-                if p.grad is None:
-                    continue
-
-                grad = p.grad.data
-                state = self.state[p]
-
-                # 感情EMA更新・スカラー生成 (既存ロジックを維持)
-                ema = self._update_ema(state, loss_val)
-                scalar = self._compute_scalar(ema)
-                ratio = self._decide_ratio(scalar)
-
-                # shadow_param：必要時のみ更新 (既存ロジックを維持)
-                if ratio > 0:
-                    if 'shadow' not in state:
-                        state['shadow'] = p.data.clone()
-                    else:
-                        p.data.mul_(1 - ratio).add_(state['shadow'], alpha=ratio)
-                        state['shadow'].lerp_(p.data, 0.05)
-                
-                # --- 新しい勾配補正ロジック ---
-                # 行列の形状が2次元以上の場合、分散情報ベースのAB近似を使用
-                if grad.dim() >= 2:
-                    # 行と列の2乗平均を計算 (分散の軽量な近似)
-                    r_sq = torch.mean(grad * grad, dim=tuple(range(1, grad.dim())), keepdim=True).add_(group['eps'])
-                    c_sq = torch.mean(grad * grad, dim=0, keepdim=True).add_(group['eps'])
-
-                    # 分散情報から勾配の近似行列を生成
-                    # AB行列として見立てたものを直接生成し更新項を計算する
-                    # A = sqrt(r_sq), B = sqrt(c_sq) とすることでAB行列の近似を再現
-                    # これをEMAで平滑化する
-                    beta1, beta2 = group['betas']
-                    
-                    state.setdefault('exp_avg_r', torch.zeros_like(r_sq)).mul_(beta1).add_(torch.sqrt(r_sq), alpha=1 - beta1)
-                    state.setdefault('exp_avg_c', torch.zeros_like(c_sq)).mul_(beta1).add_(torch.sqrt(c_sq), alpha=1 - beta1)
-                    
-                    # 再構築した近似勾配の平方根の積で正規化
-                    # これにより2次モーメントのような役割を果たす
-                    denom = torch.sqrt(state['exp_avg_r'] * state['exp_avg_c']).add_(group['eps'])
-                    
-                    # 最終的な更新項を計算
-                    update_term = grad / denom
-
-                # 1次元(ベクトル)の勾配補正(decoupled weight decay 構造に近い)
-                else:
-                    exp_avg = state.setdefault('exp_avg', torch.zeros_like(p.data))
-                    exp_avg_sq = state.setdefault('exp_avg_sq', torch.zeros_like(p.data))
-                    beta1, beta2 = group['betas']
-                    exp_avg.mul_(beta1).add_(grad, alpha=1 - beta1)
-                    exp_avg_sq.mul_(beta2).addcmul_(grad, grad, value=1 - beta2)
-                    denom = exp_avg_sq.sqrt().add_(group['eps'])
-                    update_term = exp_avg / denom
-
-                # 最終的なパラメータ更新 (decoupled weight decayも適用)
-                p.data.add_(p.data, alpha=-group['weight_decay'] * group['lr'])
-                p.data.add_(update_term, alpha=-group['lr'])
-
-                # --- Early Stop ロジック (既存ロジックを維持) ---
-                hist = self.state.setdefault('scalar_hist', [])
-                hist.append(scalar)
-                if len(hist) > 32:
-                    hist.pop(0)
-
-        # Early Stop判断
-        if len(self.state['scalar_hist']) >= 32:
-            buf = self.state['scalar_hist']
-            avg_abs = sum(abs(s) for s in buf) / len(buf)
-            std = sum((s - sum(buf)/len(buf))**2 for s in buf) / len(buf)
-            if avg_abs < 0.05 and std < 0.005:
-                self.should_stop = True
-
-        return loss
-
-"""
-Fact is inspired by Adafactor,  
-and its VRAM-friendly design is something everyone loves.
-"""

optimizer/emolynx.py

@@ -1,129 +0,0 @@
-import torch
-from torch.optim import Optimizer
-import math
-from typing import Tuple, Callable, Union
-
-# Helper function (Lynx)
-def exists(val):
-    return val is not None
-
-class EmoLynx(Optimizer):
-    # クラス定義＆初期化
-    def __init__(self, params: Union[list, torch.nn.Module], lr=1e-3, betas=(0.9, 0.99), 
-    # lynx用ベータ･互換性の追加(lynx用beta1･beta2)
-                 eps=1e-8, weight_decay=0.01, decoupled_weight_decay: bool = False): 
-
-        defaults = dict(lr=lr, betas=betas, eps=eps, weight_decay=weight_decay)
-        super().__init__(params, defaults)
-        
-        # lynxに応じてウェイト減衰のため保存
-        self._init_lr = lr
-        self.decoupled_wd = decoupled_weight_decay
-        self.should_stop = False # 停止フラグの初期化
-
-    # 感情EMA更新(緊張と安静)
-    def _update_ema(self, state, loss_val):
-        ema = state.setdefault('ema', {})
-        ema['short'] = 0.3 * loss_val + 0.7 * ema.get('short', loss_val)
-        ema['long']  = 0.01 * loss_val + 0.99 * ema.get('long', loss_val)
-        return ema
-
-    # 感情スカラー値生成(EMA差分、滑らかな非線形スカラー、tanh 5 * diff で鋭敏さ強調)
-    def _compute_scalar(self, ema):
-        diff = ema['short'] - ema['long']
-        return math.tanh(5 * diff)
-
-    # Shadow混合比率(> 0.6：70〜90%、 < 0.6：10%、 > 0.3：30%、 平時：0%)
-    def _decide_ratio(self, scalar):
-        if scalar > 0.6:
-            return 0.7 + 0.2 * scalar
-        elif scalar < -0.6:
-            return 0.1
-        elif abs(scalar) > 0.3:
-            return 0.3
-        return 0.0
-
-    # 損失取得(損失値 loss_val を数値化、感情判定に使用、存在しないパラメータ(更新不要)はスキップ)
-    @torch.no_grad()
-    def step(self, closure: Callable | None = None): # クロージャの型ヒントを追加
-        loss = None
-        if exists(closure): # 一貫性のためにexistsヘルパーを使う
-            with torch.enable_grad():
-                loss = closure()
-        loss_val = loss.item() if loss is not None else 0.0
-
-        for group in self.param_groups:
-            # リンクス共通パラメータ抽出
-            lr, wd, beta1, beta2 = group['lr'], group['weight_decay'], *group['betas']
-            
-            # ウェイト減衰の処理を分離 (from lynx)
-            _wd_actual = wd
-            if self.decoupled_wd:
-                _wd_actual /= self._init_lr # 非連結時ウェイト減衰調整
-
-            for p in filter(lambda p: exists(p.grad), group['params']): # PGチェックにフィルタ
-
-                grad = p.grad # PG直接使用(計算に".data"不要)
-                state = self.state[p]
-
-                # EMA更新・スカラー生成(EMA差分からスカラーを生成しスパイク比率を決定)
-                ema = self._update_ema(state, loss_val)
-                scalar = self._compute_scalar(ema)
-                ratio = self._decide_ratio(scalar)
-
-                # shadow_param：必要時のみ更新(スパイク部分に現在値を5%ずつ追従させる動的履歴)
-                if ratio > 0:
-                    if 'shadow' not in state:
-                        state['shadow'] = p.data.clone()
-                    else:
-                        p.data.mul_(1 - ratio).add_(state['shadow'], alpha=ratio) 
-                        state['shadow'].lerp_(p.data, 0.05) 
-                        # lynx更新前 p.data で shadow 更新(現在値を5%ずつ追従)
-                        # p.data.mul_(1 - ratio).add_(state['shadow'], alpha=ratio) 
-                        # EmoNavi: p.data = p.data * (1-ratio) + shadow * ratio
-
-                # --- Start Lynx Gradient Update Logic ---
-                
-                # lynx初期化(exp_avg_sq)
-                if 'exp_avg' not in state:
-                    state['exp_avg'] = torch.zeros_like(p)
-                exp_avg = state['exp_avg']
-
-                # Stepweight decay (from lynx): p.data = p.data * (1 - lr * wd)
-                # decoupled_wd 考慮 _wd_actual 使用(EmoNaviのwdは最後に適用)
-                p.data.mul_(1. - lr * _wd_actual)
-
-                # 勾配ブレンド
-                # m_t = beta1 * exp_avg_prev + (1 - beta1) * grad
-                blended_grad = grad.mul(1. - beta1).add_(exp_avg, alpha=beta1)
-                
-                # p: p.data = p.data - lr * sign(blended_grad)
-                p.data.add_(blended_grad.sign_(), alpha = -lr)
-
-                # exp_avg = beta2 * exp_avg + (1 - beta2) * grad
-                exp_avg.mul_(beta2).add_(grad, alpha = 1. - beta2)
-
-                # --- End Lynx Gradient Update Logic ---
-
-                # Early Stop用 scalar記録(バッファ共通で管理/最大32件保持/動静評価)
-                # この部分は p.state ではなく self.state ���アクセスする
-                hist = self.state.setdefault('scalar_hist', [])
-                hist.append(scalar)
-                if len(hist) > 32:
-                    hist.pop(0)
-
-        # Early Stop判断(静けさの合図) - This part is outside the inner loop
-        if len(self.state['scalar_hist']) >= 32:
-            buf = self.state['scalar_hist']
-            avg_abs = sum(abs(s) for s in buf) / len(buf)
-            std = sum((s - sum(buf)/len(buf))**2 for s in buf) / len(buf)
-            if avg_abs < 0.05 and std < 0.005:
-                self.should_stop = True # 💡 外部からこれを見て判断可
-
-        return loss
-
-"""
-Lynx was developed with inspiration from Lion and Tiger, 
-which we deeply respect for their lightweight and intelligent design.  
-Lynx also integrates EmoNAVI to enhance its capabilities.
-"""

optimizer/emonavi.py

@@ -1,96 +0,0 @@
-import torch
-from torch.optim import Optimizer
-import math
-
-class EmoNavi(Optimizer):
-    # クラス定義＆初期化
-    def __init__(self, params, lr=1e-3, betas=(0.9, 0.999),
-                 eps=1e-8, weight_decay=0.01):
-        defaults = dict(lr=lr, betas=betas, eps=eps, weight_decay=weight_decay)
-        super().__init__(params, defaults)
-    # 感情EMA更新(緊張と安静)
-    def _update_ema(self, state, loss_val):
-        ema = state.setdefault('ema', {})
-        ema['short'] = 0.3 * loss_val + 0.7 * ema.get('short', loss_val)
-        ema['long']  = 0.01 * loss_val + 0.99 * ema.get('long', loss_val)
-        return ema
-    # 感情スカラー値生成(EMA差分、滑らかな非線形スカラー、tanh 5 * diff で鋭敏さ強調)
-    def _compute_scalar(self, ema):
-        diff = ema['short'] - ema['long']
-        return math.tanh(5 * diff)
-    # Shadow混合比率(> 0.6：70〜90%、 < 0.6：10%、 > 0.3：30%、 平時：0%)
-    def _decide_ratio(self, scalar):
-        if scalar > 0.6:
-            return 0.7 + 0.2 * scalar
-        elif scalar < -0.6:
-            return 0.1
-        elif abs(scalar) > 0.3:
-            return 0.3
-        return 0.0
-    # 損失取得(損失値 loss_val を数値化、感情判定に使用、存在しないパラメータ(更新不要)はスキップ)
-    @torch.no_grad()
-    def step(self, closure=None):
-        loss = closure() if closure is not None else None
-        loss_val = loss.item() if loss is not None else 0.0
-
-        for group in self.param_groups:
-            for p in group['params']:
-                if p.grad is None:
-                    continue
-
-                grad = p.grad.data
-                state = self.state[p]
-
-                # EMA更新・スカラー生成(EMA差分からスカラーを生成しスパイク比率を決定)
-                ema = self._update_ema(state, loss_val)
-                scalar = self._compute_scalar(ema)
-                ratio = self._decide_ratio(scalar)
-
-                # shadow_param：必要時のみ更新(スパイク部分に現在値を5%ずつ追従させる動的履歴)
-                if ratio > 0:
-                    if 'shadow' not in state:
-                        state['shadow'] = p.data.clone()
-                    else:
-                        p.data.mul_(1 - ratio).add_(state['shadow'], alpha=ratio)
-                        state['shadow'].lerp_(p.data, 0.05)
-                
-                # スカラー生成：短期と長期EMAの差分から信号を得る(高ぶりの強さ)
-                # 混合比率：スカラーが閾値を超える場合にのみ計算される(信頼できる感情信号かどうかの選別)
-                # → スカラー値が小さい場合は ratio = 0 となり、shadow混合は行われない
-                # → 信頼できる強い差分のときのみ感情機構が発動する(暗黙の信頼度判定)                
-                
-                # 1次・2次モーメントを使った勾配補正(decoupled weight decay 構造に近い)
-                exp_avg = state.setdefault('exp_avg', torch.zeros_like(p.data))
-                exp_avg_sq = state.setdefault('exp_avg_sq', torch.zeros_like(p.data))
-                beta1, beta2 = group['betas']
-                exp_avg.mul_(beta1).add_(grad, alpha=1 - beta1)
-                exp_avg_sq.mul_(beta2).addcmul_(grad, grad, value=1 - beta2)
-                denom = exp_avg_sq.sqrt().add_(group['eps'])
-
-                step_size = group['lr']
-                if group['weight_decay']:
-                    p.data.add_(p.data, alpha=-group['weight_decay'] * step_size)
-                p.data.addcdiv_(exp_avg, denom, value=-step_size)
-
-        # 感情機構の発火が収まり"十分に安定"していることを外部伝達できる(自動停止ロジックではない)
-                # Early Stop用 scalar 記録(バッファ共通で管理/最大32件保持/動静評価)
-                hist = self.state.setdefault('scalar_hist', [])
-                hist.append(scalar)
-                if len(hist) > 32:
-                    hist.pop(0)
-
-        # Early Stop判断(静けさの合図)
-        if len(self.state['scalar_hist']) >= 32:
-            buf = self.state['scalar_hist']
-            avg_abs = sum(abs(s) for s in buf) / len(buf)
-            std = sum((s - sum(buf)/len(buf))**2 for s in buf) / len(buf)
-            if avg_abs < 0.05 and std < 0.005:
-                self.should_stop = True  # 💡 外部からこれを見て判断可
-
-        # 32ステップ分のスカラー値の静かな条件を満たした時"フラグ" should_stop = True になるだけ
-
-        return loss
-
-#    https://github.com/muooon/EmoNavi
-#    An emotion-driven optimizer that feels loss and navigates accordingly.
-#    Don't think. Feel. Don't stop. Keep running. Believe in what's beyond.

optimizer/学習の進め方(日本語).txt

@@ -1,52 +0,0 @@
-EmoNAVI、Fact、Lynx / Clan の使い方について
-
-emonavi系は、スケジューラーに依存しない設計です、スケジューラーはなくていい、あっても大丈夫、どんな設定でも概ねなんとか自動調整するから、という感じの設計です。
-
-しかし短時間で詳細をしっかり習得させたい方は、Cosine-ReStart 等のあなたの普段お使いのスケジューラーを使用してください。
-
-<学習率とは？強度ではなく「フィルター」です>
-
-学習率の設定は、学習強度と理解している方も多いだろうと想像しますが、この学習率というのは、いわばフィルターです。VAEのLatentをどう見るか？を左右するのが学習率です。
-
-想像して欲しいのは"半透明なプラ板"です、学習率が高いとき、このプラ板に透けて映る画像は"概要"です。大まかな光の分布や大きな塊として表現されていると想像してください。学習率の低いときは"詳細"です。透明さが増しクリアな表現になります。ようは学習率は"解像度"と言っていいと思います。透明度によりボケ具合を調整するようなものです。
-
-学習率の高いとき概要を覚えるのが得意で早いのはこのためです。情報も少ないので早期に習得できます。逆に学習率の低いときは情報も多いため習得するまでの時間が増える。というワケですね。低すぎるときは情報過多のため習得できずに訓練期間を終えてしまい結果も薄いものになります。
-
-ということで"学習率"についてはemonavi系のみに関わる話ではありませんが、他のoptimizer等でも同様のことなので、今後の学習でも気に留めてください。
-
-<EmoNaviシリーズとスケジューラーと追加学習>
-
-さてそれで、emonavi系ですが、最初に記したように、スケジューラーを設定すれば、コンスタントよりも早期に詳細を獲得できる可能性があります。
-また、スケジューラーを使用せず"追加学習"で、２回目以降を低学習率で実施することもできます。これは引き継ぎのパラメーター等は不要で簡単に実施できます。ここは他のoptimizerにない簡単化した部分です。
-
-emonavi系はコンスタントで回し続けても過学習にはなりません。そこを超えないように調整しているためです。そのため同じ学習率で回し続けても必要以上の学習はしません。そこを超えると過学習領域に近いという判断をします。概要の習得後は何も習得しないのではなく必要な分のみ習得するので、概要の習得前に比べ学習量が少ないため進行しないように見えるだけです。
-
-もし学習が一定のところから進まない、と感じたときは、追加学習で低学習率へ変更してください。そうすると詳細を一気に吸収し始めます。
-
-<謝意>
-emonavi系に限らず、この説明で学習設定のノウハウの獲得に寄与できれば嬉しいです。皆さまのお役に立てれば幸いです。最後までご覧いただきありがとうございました。
-
-<追記>
-学習率について実感を得られるように、わかりやすく伝えたい、と思っています。
-
-学習率とは、読書の速さにも置き換えられると思います、
-学習率高：飛ばし読み(速読)、
-学習率低：熟読(精読)、と想像してください。
-スケジューラーはこれを学習予定として管理します
-
-emonavi は shadow の機能で、モデル自身の復習や振り返りを促し学習を進行します
-進み方を外部に決めさせ自律で補うか、自律のみに任せるか、になります。
-
-他にも、以下のように例えることも可能です、
-学習率高：遠くからの撮影＝概要(細部はあいまい)
-学習率低：寄りで撮影＝詳細(細部を正確に)
-オートフォーカス：スケジューラー、shadow、
-
-別視点からも説明しますと、細部表現を獲得したい場合は教師データを増やす、ことでも可能です。
-繰り返し数が増加することで細部の特徴も少しづつ蓄積される、となります。
-ただし、色についてはVAEの性能に依拠する部分が多く、これを正しく反映できる教師データか、VAEの性能の向上しかありません。
-
-それとですね、shadow はオートフォーカスでもありますが、これは学習の振り返り、復習をするもので、自分自身の経験に学ぶ仕組みです、
-ですから、特徴を捉えて学んで、別の特徴を見つけて、、を繰り返す、その結果としてフォーカスもピントも変化し続けるようになります。
-
-以上となります。追記も最後までご覧頂いてありがとうございました。