| --- |
| language: en |
| license: mit |
| library_name: pytorch |
| tags: |
| - vae |
| - predictive-coding |
| - neuroscience |
| - gabor-splatting |
| - computer-vision |
| - biology |
| datasets: |
| - nielsr/flowers-102 |
| --- |
| |
| # Neuro Splat — an artificial visual cortex (Gabor wave-packet VAE) |
|
|
| **PerceptionLab / Antti Luode, with Claude (Opus 4.8), in dialogue with Gemini. Helsinki, June 2026.** |
|
|
| > Do not hype. Do not lie. Just show. |
|
|
| *An image is not a million pixels. It is a smooth map and a scatter of small bright wave-packets dropped where the detail lives.* |
|
|
| --- |
|
|
| ## What this model is |
|
|
| This repo holds the weights (`model.pt`) for a **VAE whose decoder is a differentiable Gabor wave-packet splatter**. It is not a pixel-generating image model; it is a biologically-motivated representation experiment — an architecture that reproduces the *shape* of how V1 is thought to code images (localized Gabor atoms), used to test predictive coding and phase locking. |
|
|
| - **Encoder** (CNN): image → a latent "concept". |
| - **Decoder** (MLP): latent → the parameters of 512 localized Gabor atoms — position, scale, orientation, frequency, and a complex `(a, b)` coefficient per colour channel. |
| - **Renderer**: splats those wave-packets onto the canvas; the image is their sum. |
|
|
| Trained on **Oxford Flowers-102** at **128×128 with 512 packets**. |
|
|
| ## The blur is the prior; reality is the sharpness |
|
|
| Asked to generate a flower from a random latent, this model returns a soft, watercolour blob — and that is the correct, expected behaviour, not a failure. A generative prior with no input has to average over everything it cannot disambiguate, so it returns the low-frequency gist. |
|
|
| To see it work, you give it eyes. Hook it to a webcam and the blurry top-down prior collides with raw bottom-up reality; the live frame supplies the phase the prior could not guess, the packets lock to it, and the gist sharpens. That collision — top-down prediction meets bottom-up residual — is the predictive-coding loop, made live. |
|
|
| ## How to load it |
|
|
| You need the architecture from the [ArtificialCortex repo](https://github.com/anttiluode/ArtificialCortex) (`splat_generator.py` defines `SplatVAE`). |
|
|
| ```python |
| import torch |
| from huggingface_hub import hf_hub_download |
| from splat_generator import SplatVAE # from the ArtificialCortex repo (the_splat/) |
| |
| ckpt = hf_hub_download("Aluode/Neuro_Splat", "model.pt") |
| model = SplatVAE(image_size=128, latent=128, num_packets=512, chunk=64) |
| model.load_state_dict(torch.load(ckpt, map_location="cpu")) |
| model.eval() |
| ``` |
|
|
| ## How to run it |
|
|
| ```bash |
| pip install torch torchvision numpy opencv-python huggingface_hub |
| |
| # perception — watch the packets phase-lock to a live webcam (panel 3) |
| python live_cortex_perception.py --model_path model.pt --image_size 128 --num_packets 512 |
| |
| # generation — the blurry priors from random latents (expected to be soft) |
| python splat_generator.py --mode sample --resume model.pt --image_size 128 --num_packets 512 |
| ``` |
|
|
| `--image_size 128 --num_packets 512` must match these weights, or the `state_dict` will not load. |
|
|
| ## The honest ledger |
|
|
| **What this model shows:** |
| - an image can be represented and learned as a sparse sum of localized Gabor wave-packets (the V1 / Olshausen–Field model, made trainable); |
| - the phase-wrapping problem is dodged by outputting complex `(a, b)` coefficients instead of raw angles; |
| - the correction half of predictive coding works live: a blurry top-down prior is sharpened by gradient descent against a bottom-up residual (the webcam frame). |
|
|
| **What it is not, and what it lacks:** |
| - not a photorealistic generator. It is a VAE with an amortized latent — sharp reconstructions, blurry-but-structured samples. It is interesting on biological representation, not on benchmarks against diffusion models; |
| - the **prior is flower-only**: in-domain (a flower) the top-down gist genuinely helps; off-domain (a room, a face) it is a wrong guess that the live-frame fit overwrites, so there the loop is doing direct re-fitting, not prediction; |
| - the **"floaters"**: on out-of-domain input, with an aggressive learning rate, the optimizer orphans some packets into tiny ultra-bright dots instead of coordinating them into edges. They *resemble* phosphenes, but the cause is an MSE optimizer with **no lateral inhibition between packets**, not the neural disinhibition that makes real phosphenes — a rhyme, not the mechanism. It points at the missing inhibitory coordination (the `grown_gates` line), not at a recreated brain; |
| - it is 2D; relative units throughout; one trained model. |
|
|
| **The bet (untouched):** that the phase-locked frame is a *felt* sharpening rather than a computed one. The model locates the mechanism in code that can fail; it does not touch the hard problem. |
|
|
| ## Lineage |
|
|
| A sub-organ of [`the_artificial_cortex`](https://github.com/anttiluode/ArtificialCortex) (`the_splat/`). The generator is the trained top-down prior; the live cortex is reality forcing that prior into focus. MIT. |
|
|
| *The generator dreamed a blurry flower because it had nothing to look at. Open its eyes and reality supplies the phase the dream could not; the packets lock, the gist sharpens, and what they cannot yet coordinate, they see as stars.* |
