Create CurvOpt-MathFoundations.md

CurvOpt-MathFoundations.md

@@ -0,0 +1,249 @@
+# CurvOpt: Mathematical Foundations
+
+## Energy-Constrained Precision Allocation via Curvature and Information Theory
+
+---
+
+## 1. Problem Formulation
+
+Let a trained neural network with parameters \( \theta \in \mathbb{R}^d \) minimize empirical risk:
+
+\[
+L(\theta) = \frac{1}{n} \sum_{i=1}^{n} \ell(f_\theta(x_i), y_i)
+\]
+
+We introduce quantization:
+
+\[
+\theta_q = \theta + \varepsilon
+\]
+
+where \( \varepsilon \) represents quantization perturbation.
+
+We seek precision assignments \( q_l \) per layer \( l \) solving:
+
+\[
+\min_{q_l \in \mathcal{Q}}
+\sum_{l=1}^{L} \mathcal{E}_l(q_l)
+\quad
+\text{s.t.}
+\quad
+L(\theta_q) - L(\theta) \le \epsilon
+\]
+
+This is a constrained optimization problem over precision levels.
+
+**Reference**
+
+- Boyd, S., & Vandenberghe, L. (2004). *Convex Optimization*. Cambridge University Press.
+
+---
+
+## 2. Second-Order Loss Perturbation
+
+Assume \( L \) is twice continuously differentiable.
+
+By Taylor expansion around \( \theta \):
+
+\[
+L(\theta + \varepsilon)
+=
+L(\theta)
++
+\nabla L(\theta)^T \varepsilon
++
+\frac{1}{2} \varepsilon^T H(\theta) \varepsilon
++
+o(\|\varepsilon\|^2)
+\]
+
+where:
+
+\[
+H(\theta) = \nabla^2 L(\theta)
+\]
+
+Near a stationary point:
+
+\[
+\nabla L(\theta) \approx 0
+\]
+
+Thus:
+
+\[
+\Delta L
+\approx
+\frac{1}{2} \varepsilon^T H \varepsilon
+\]
+
+**Reference**
+
+- Nocedal, J., & Wright, S. (2006). *Numerical Optimization*. Springer.
+
+---
+
+## 3. Spectral Bound via Rayleigh Quotient
+
+Since \( H \) is symmetric:
+
+\[
+\lambda_{\min}(H) \|\varepsilon\|^2
+\le
+\varepsilon^T H \varepsilon
+\le
+\lambda_{\max}(H) \|\varepsilon\|^2
+\]
+
+Hence:
+
+\[
+\Delta L
+\le
+\frac{1}{2}
+\lambda_{\max}(H)
+\|\varepsilon\|^2
+\]
+
+Large eigenvalues correspond to sharp curvature and high sensitivity.
+
+**Reference**
+
+- Goodfellow, I., Bengio, Y., & Courville, A. (2016). *Deep Learning*. MIT Press.
+
+---
+
+## 4. Trace Approximation via Hutchinson Estimator
+
+Exact Hessian computation is infeasible.
+
+We use:
+
+\[
+\operatorname{Tr}(H)
+=
+\mathbb{E}_{v}
+\left[
+v^T H v
+\right]
+\]
+
+where \( v_i \sim \{-1,+1\} \) (Rademacher distribution).
+
+This estimator is unbiased.
+
+**Reference**
+
+- Robert, C., & Casella, G. (2004). *Monte Carlo Statistical Methods*. Springer.
+
+---
+
+## 5. Quantization Noise Model
+
+Uniform quantization with step size \( \Delta \):
+
+\[
+\varepsilon \sim \mathcal{U}\left(-\frac{\Delta}{2}, \frac{\Delta}{2}\right)
+\]
+
+Variance:
+
+\[
+\operatorname{Var}(\varepsilon)
+=
+\frac{\Delta^2}{12}
+\]
+
+Expected loss increase:
+
+\[
+\mathbb{E}[\Delta L]
+\approx
+\frac{1}{2}
+\operatorname{Tr}(H)
+\cdot
+\frac{\Delta^2}{12}
+\]
+
+This connects precision (through \( \Delta \)) directly to curvature.
+
+**Reference**
+
+- Gallager, R. (1968). *Information Theory and Reliable Communication*. Wiley.
+
+---
+
+## 6. Information-Theoretic Layer Relevance
+
+For layer \( l \):
+
+\[
+I(X_l ; Y_l)
+=
+\int p(x,y)
+\log
+\frac{p(x,y)}{p(x)p(y)}
+\, dx\,dy
+\]
+
+Data Processing Inequality:
+
+\[
+I(X; Y_{l+1}) \le I(X; Y_l)
+\]
+
+Thus information cannot increase through deterministic transformations.
+
+Layers with low marginal information contribution can tolerate larger perturbations.
+
+**Reference**
+
+- Cover, T., & Thomas, J. (2006). *Elements of Information Theory*. Wiley.
+
+---
+
+## 7. Constrained Energy Minimization
+
+We combine curvature-based sensitivity and energy cost:
+
+\[
+\min_{q_l}
+\sum_l \mathcal{E}_l(q_l)
++
+\lambda
+\left(
+\sum_l
+\frac{1}{24}
+\operatorname{Tr}(H_l)
+\Delta_l^2
+-
+\epsilon
+\right)
+\]
+
+Using KKT optimality conditions:
+
+\[
+\nabla_{q_l} \mathcal{L} = 0
+\]
+
+This yields optimal precision allocation under a global accuracy constraint.
+
+**Reference**
+
+- Bertsekas, D. (1999). *Nonlinear Programming*. Athena Scientific.
+
+---
+
+## 8. Summary
+
+CurvOpt is grounded in:
+
+1. Second-order perturbation theory  
+2. Spectral sensitivity bounds  
+3. Monte Carlo trace estimation  
+4. Classical quantization noise modeling  
+5. Shannon mutual information  
+6. Constrained nonlinear optimization via KKT  
+
+All formulations above are standard results from established literature.