CurvOpt-MathFoundations.md

# CurvOpt: Mathematical Foundations

## Energy-Constrained Precision Allocation via Curvature and Information Theory

<script src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>

---

## 1. Problem Formulation

Let a trained neural network with parameters \\( \theta \in \mathbb{R}^d \\) minimize empirical risk:

\\[
L(\theta) = \frac{1}{n} \sum_{i=1}^{n} \ell(f_\theta(x_i), y_i)
\\]

We introduce quantization:

\\[
\theta_q = \theta + \varepsilon
\\]

We seek precision assignments \\( q_l \\) per layer:

\\[
\min_{q_l \in \mathcal{Q}}
\sum_{l=1}^{L} \mathcal{E}_l(q_l)
\quad
\text{s.t.}
\quad
L(\theta_q) - L(\theta) \le \epsilon
\\]

Reference: Boyd & Vandenberghe (2004), *Convex Optimization*

---

## 2. Second-Order Loss Perturbation

By Taylor expansion:

\\[
L(\theta + \varepsilon)
=
L(\theta)
+
\nabla L(\theta)^T \varepsilon
+
\frac{1}{2} \varepsilon^T H(\theta) \varepsilon
+
o(\|\varepsilon\|^2)
\\]

Near a stationary point:

\\[
\nabla L(\theta) \approx 0
\\]

Thus:

\\[
\Delta L
\approx
\frac{1}{2} \varepsilon^T H \varepsilon
\\]

Reference: Nocedal & Wright (2006), *Numerical Optimization*

---

## 3. Spectral Bound

Since \\( H \\) is symmetric:

\\[
\lambda_{\min}(H) \|\varepsilon\|^2
\le
\varepsilon^T H \varepsilon
\le
\lambda_{\max}(H) \|\varepsilon\|^2
\\]

Thus:

\\[
\Delta L
\le
\frac{1}{2}
\lambda_{\max}(H)
\|\varepsilon\|^2
\\]

Reference: Goodfellow et al. (2016), *Deep Learning*

---

## 4. Hutchinson Trace Estimator

\\[
\operatorname{Tr}(H)
=
\mathbb{E}_{v}
\left[
v^T H v
\right]
\\]

where \\( v_i \sim \{-1,+1\} \\).

Reference: Robert & Casella (2004), *Monte Carlo Statistical Methods*

---

## 5. Quantization Noise Model

Uniform quantization with step size \\( \Delta \\):

\\[
\varepsilon \sim \mathcal{U}\left(-\frac{\Delta}{2}, \frac{\Delta}{2}\right)
\\]

Variance:

\\[
\operatorname{Var}(\varepsilon)
=
\frac{\Delta^2}{12}
\\]

Expected loss increase:

\\[
\mathbb{E}[\Delta L]
\approx
\frac{1}{2}
\operatorname{Tr}(H)
\cdot
\frac{\Delta^2}{12}
\\]

Reference: Gallager (1968), *Information Theory and Reliable Communication*

---

## 6. Mutual Information

\\[
I(X_l ; Y_l)
=
\int p(x,y)
\log
\frac{p(x,y)}{p(x)p(y)}
\, dx\,dy
\\]

Data Processing Inequality:

\\[
I(X; Y_{l+1}) \le I(X; Y_l)
\\]

Reference: Cover & Thomas (2006), *Elements of Information Theory*

---

## 7. Constrained Energy Minimization

\\[
\min_{q_l}
\sum_l \mathcal{E}_l(q_l)
+
\lambda
\left(
\sum_l
\frac{1}{24}
\operatorname{Tr}(H_l)
\Delta_l^2
-
\epsilon
\right)
\\]

KKT condition:

\\[
\nabla_{q_l} \mathcal{L} = 0
\\]

Reference: Bertsekas (1999), *Nonlinear Programming*

---

## Summary

CurvOpt is grounded in:

- Second-order perturbation theory  
- Spectral bounds  
- Monte Carlo trace estimation  
- Classical quantization noise modeling  
- Shannon mutual information  
- Constrained nonlinear optimization  

All formulas are standard results from established literature.