Update CurvOpt-MathFoundations.md

CurvOpt-MathFoundations.md

@@ -2,50 +2,44 @@
 
 ## Energy-Constrained Precision Allocation via Curvature and Information Theory
 
+<script src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>
+
 ---
 
 ## 1. Problem Formulation
 
-Let a trained neural network with parameters \( \theta \in \mathbb{R}^d \) minimize empirical risk:
+Let a trained neural network with parameters \\( \theta \in \mathbb{R}^d \\) minimize empirical risk:
 
-\[
+\\[
 L(\theta) = \frac{1}{n} \sum_{i=1}^{n} \ell(f_\theta(x_i), y_i)
-\]
+\\]
 
 We introduce quantization:
 
-\[
+\\[
 \theta_q = \theta + \varepsilon
-\]
-
-where \( \varepsilon \) represents quantization perturbation.
+\\]
 
-We seek precision assignments \( q_l \) per layer \( l \) solving:
+We seek precision assignments \\( q_l \\) per layer:
 
-\[
+\\[
 \min_{q_l \in \mathcal{Q}}
 \sum_{l=1}^{L} \mathcal{E}_l(q_l)
 \quad
 \text{s.t.}
 \quad
 L(\theta_q) - L(\theta) \le \epsilon
-\]
-
-This is a constrained optimization problem over precision levels.
+\\]
 
-**Reference**
-
-- Boyd, S., & Vandenberghe, L. (2004). *Convex Optimization*. Cambridge University Press.
+Reference: Boyd & Vandenberghe (2004), *Convex Optimization*
 
 ---
 
 ## 2. Second-Order Loss Perturbation
 
-Assume \( L \) is twice continuously differentiable.
-
-By Taylor expansion around \( \theta \):
+By Taylor expansion:
 
-\[
+\\[
 L(\theta + \varepsilon)
 =
 L(\theta)
@@ -55,158 +49,124 @@ L(\theta)
 \frac{1}{2} \varepsilon^T H(\theta) \varepsilon
 +
 o(\|\varepsilon\|^2)
-\]
-
-where:
-
-\[
-H(\theta) = \nabla^2 L(\theta)
-\]
+\\]
 
 Near a stationary point:
 
-\[
+\\[
 \nabla L(\theta) \approx 0
-\]
+\\]
 
 Thus:
 
-\[
+\\[
 \Delta L
 \approx
 \frac{1}{2} \varepsilon^T H \varepsilon
-\]
-
-**Reference**
+\\]
 
-- Nocedal, J., & Wright, S. (2006). *Numerical Optimization*. Springer.
+Reference: Nocedal & Wright (2006), *Numerical Optimization*
 
 ---
 
-## 3. Spectral Bound via Rayleigh Quotient
+## 3. Spectral Bound
 
-Since \( H \) is symmetric:
+Since \\( H \\) is symmetric:
 
-\[
+\\[
 \lambda_{\min}(H) \|\varepsilon\|^2
 \le
 \varepsilon^T H \varepsilon
 \le
 \lambda_{\max}(H) \|\varepsilon\|^2
-\]
+\\]
 
-Hence:
+Thus:
 
-\[
+\\[
 \Delta L
 \le
 \frac{1}{2}
 \lambda_{\max}(H)
 \|\varepsilon\|^2
-\]
+\\]
 
-Large eigenvalues correspond to sharp curvature and high sensitivity.
-
-**Reference**
-
-- Goodfellow, I., Bengio, Y., & Courville, A. (2016). *Deep Learning*. MIT Press.
+Reference: Goodfellow et al. (2016), *Deep Learning*
 
 ---
 
-## 4. Trace Approximation via Hutchinson Estimator
-
-Exact Hessian computation is infeasible.
+## 4. Hutchinson Trace Estimator
 
-We use:
-
-\[
+\\[
 \operatorname{Tr}(H)
 =
 \mathbb{E}_{v}
 \left[
 v^T H v
 \right]
-\]
-
-where \( v_i \sim \{-1,+1\} \) (Rademacher distribution).
-
-This estimator is unbiased.
+\\]
 
-**Reference**
+where \\( v_i \sim \{-1,+1\} \\).
 
-- Robert, C., & Casella, G. (2004). *Monte Carlo Statistical Methods*. Springer.
+Reference: Robert & Casella (2004), *Monte Carlo Statistical Methods*
 
 ---
 
 ## 5. Quantization Noise Model
 
-Uniform quantization with step size \( \Delta \):
+Uniform quantization with step size \\( \Delta \\):
 
-\[
+\\[
 \varepsilon \sim \mathcal{U}\left(-\frac{\Delta}{2}, \frac{\Delta}{2}\right)
-\]
+\\]
 
 Variance:
 
-\[
+\\[
 \operatorname{Var}(\varepsilon)
 =
 \frac{\Delta^2}{12}
-\]
+\\]
 
 Expected loss increase:
 
-\[
+\\[
 \mathbb{E}[\Delta L]
 \approx
 \frac{1}{2}
 \operatorname{Tr}(H)
 \cdot
 \frac{\Delta^2}{12}
-\]
+\\]
 
-This connects precision (through \( \Delta \)) directly to curvature.
-
-**Reference**
-
-- Gallager, R. (1968). *Information Theory and Reliable Communication*. Wiley.
+Reference: Gallager (1968), *Information Theory and Reliable Communication*
 
 ---
 
-## 6. Information-Theoretic Layer Relevance
-
-For layer \( l \):
+## 6. Mutual Information
 
-\[
+\\[
 I(X_l ; Y_l)
 =
 \int p(x,y)
 \log
 \frac{p(x,y)}{p(x)p(y)}
 \, dx\,dy
-\]
+\\]
 
 Data Processing Inequality:
 
-\[
+\\[
 I(X; Y_{l+1}) \le I(X; Y_l)
-\]
+\\]
 
-Thus information cannot increase through deterministic transformations.
-
-Layers with low marginal information contribution can tolerate larger perturbations.
-
-**Reference**
-
-- Cover, T., & Thomas, J. (2006). *Elements of Information Theory*. Wiley.
+Reference: Cover & Thomas (2006), *Elements of Information Theory*
 
 ---
 
 ## 7. Constrained Energy Minimization
 
-We combine curvature-based sensitivity and energy cost:
-
-\[
+\\[
 \min_{q_l}
 \sum_l \mathcal{E}_l(q_l)
 +
@@ -219,31 +179,27 @@ We combine curvature-based sensitivity and energy cost:
 -
 \epsilon
 \right)
-\]
+\\]
 
-Using KKT optimality conditions:
+KKT condition:
 
-\[
+\\[
 \nabla_{q_l} \mathcal{L} = 0
-\]
-
-This yields optimal precision allocation under a global accuracy constraint.
-
-**Reference**
+\\]
 
-- Bertsekas, D. (1999). *Nonlinear Programming*. Athena Scientific.
+Reference: Bertsekas (1999), *Nonlinear Programming*
 
 ---
 
-## 8. Summary
+## Summary
 
 CurvOpt is grounded in:
 
-1. Second-order perturbation theory  
-2. Spectral sensitivity bounds  
-3. Monte Carlo trace estimation  
-4. Classical quantization noise modeling  
-5. Shannon mutual information  
-6. Constrained nonlinear optimization via KKT  
+- Second-order perturbation theory  
+- Spectral bounds  
+- Monte Carlo trace estimation  
+- Classical quantization noise modeling  
+- Shannon mutual information  
+- Constrained nonlinear optimization  
 
-All formulations above are standard results from established literature.
+All formulas are standard results from established literature.