Math rendering: inline uses HF KaTeX \\( \\); display $$ on own lines; no paren-adjacent delimiters
Browse files
README.md
CHANGED
|
@@ -44,15 +44,17 @@ model can also **parse** a real image (infer its most likely derivation) — see
|
|
| 44 |
|
| 45 |
## Method
|
| 46 |
|
| 47 |
-
SPRIG is a **text-modulated probabilistic scene grammar**
|
| 48 |
|
| 49 |
-
$$
|
|
|
|
|
|
|
| 50 |
|
| 51 |
-
Text enters through a low-rank rule factorization
|
| 52 |
|
| 53 |
-
Because the lattice and the cut dictionary are finite, the marginal is computed **exactly** by a log-semiring inside dynamic program over regions, and the training loss is the exact negative log-likelihood
|
| 54 |
|
| 55 |
-
|
|
| 56 |
|---|---|---|---|---|---|---|---|
|
| 57 |
| 1024 | 256 | 64 | 384 | 64² / 8px | 1296 regions | T5-base (frozen) | ~15.9M |
|
| 58 |
|
|
|
|
| 44 |
|
| 45 |
## Method
|
| 46 |
|
| 47 |
+
SPRIG is a **text-modulated probabilistic scene grammar** \\(G_c=(\Sigma, N, A_0, \Pi_c)\\): nonterminal symbols \\(N\\), texels (learned material primitives) \\(\Sigma\\), an axiom \\(A_0\\), and caption-conditioned productions \\(\Pi_c\\). An image is one **derivation** \\(\tau\\) — a binary tree that recursively splits the 64×64 canvas (a finite 1296-region binary-space-partition lattice, leaves ≤16px) and paints each leaf region with a texel. The conditional density marginalizes over *all* derivation trees:
|
| 48 |
|
| 49 |
+
$$
|
| 50 |
+
p(x \mid c) = \sum_{\tau} \; \prod_{\text{splits}} \pi\big(A \to \langle s,B,C\rangle \mid c\big) \; \prod_{\text{leaves}} \pi(T \mid A, c)\, p_{\mathrm{emit}}(x_r \mid T, r, c)
|
| 51 |
+
$$
|
| 52 |
|
| 53 |
+
Text enters through a low-rank rule factorization \\(\pi(A \to \langle s,B,C\rangle \mid c) = \sum_k p(k \mid A, c)\, p(s \mid k, c)\, p(B \mid k)\, p(C \mid k)\\), whose only caption-dependent factor — the mixture \\(p(k \mid A, c)\\) — is produced by a **Grammar-Modulation Transformer** (queries = symbol embeddings, cross-attention to the frozen T5-base caption). *Text deforms the grammar; it does not steer a sampler.* Each leaf emits a 4-component discretized-logistic mixture over its pixels.
|
| 54 |
|
| 55 |
+
Because the lattice and the cut dictionary are finite, the marginal is computed **exactly** by a log-semiring inside dynamic program over regions, and the training loss is the exact negative log-likelihood \\(\mathcal{L} = -\beta(A_0, \text{canvas})\\) — no encoder, no ELBO, no sampling in the loop. The *same* DP with a max-semiring yields the **Viterbi parse** of any image, which is why analysis and synthesis are the same object.
|
| 56 |
|
| 57 |
+
| \\(S\\) | \\(T_v\\) | \\(R\\) | \\(d\\) | canvas / grid | lattice | encoder | params |
|
| 58 |
|---|---|---|---|---|---|---|---|
|
| 59 |
| 1024 | 256 | 64 | 384 | 64² / 8px | 1296 regions | T5-base (frozen) | ~15.9M |
|
| 60 |
|