| --- |
| language: en |
| tags: |
| - forward-forward |
| - predictive-coding |
| - encoder-decoder |
| - mnist |
| - custom-architecture |
| - experimental |
| - cuda |
| - no-backprop |
| license: mit |
| datasets: |
| - mnist |
| metrics: |
| - accuracy |
| --- |
| |
| # RinKana/Amadeus-Hinton-Karl-1.3M-Encoder-Decoder-Model |
|
|
| This is a revolutionary, highly experimental from-scratch **1.3 Million Parameter Encoder-Decoder** model built to test the viability of Geoffrey Hinton's Forward-Forward (FF) Algorithm paired with a Predictive Coding Decoder (Karl). |
|
|
| **Unlike standard neural networks, this model does NOT use PyTorch's Autograd engine or global backpropagation.** |
| It is trained purely via local custom Python CUDA Kernel matrix updates (Hebbian learning) executed directly in VRAM using Numba and CuPy. |
|
|
| ## Model Details |
| * **Architecture:** Custom 2-Layer Encoder-Decoder Network |
| * **Encoder:** Hinton Contrastive Hebbian Learning (Forward-Forward) |
| * **Decoder:** Karl Predictive Coding |
| * **Parameters:** 1.3 Million |
| * **Memory Footprint:** Exceptionally low (True backprop-free memory scaling) |
| * **Task:** Image Classification & Generative Reconstruction (MNIST) |
|
|
| ## The Pure-CUDA Hardware Implementation |
| To adapt these biological learning rules to modern hardware without cheating by using standard automatic differentiation, we made several extreme architectural choices: |
|
|
| 1. **Zero PyTorch Backprop:** The PyTorch `loss.backward()` and `optimizer.step()` commands are entirely absent from the training loop. |
| 2. **Custom CUDA Kernels:** Weights are initialized directly in VRAM via `CuPy`, and memory mutation is handled by raw `@cuda.jit` threads processing localized math equations (Hebbian Delta Rules). |
| 3. **Local Optimizers:** No global Adam optimizers were used. The network learns purely via Stochastic Gradient Descent (SGD) calculated natively within the hardware threads on a per-synapse basis. |
| 4. **Label Injection:** For the Encoder to learn, the target classification label is injected directly into the image pixels as the target label is one-hot encoded into the first 10 pixel positions of the flattened 784-dimensional image vector forcing the network to learn the correlation between the label pixels and the image shape. |
|
|
| ## Training & Performance |
| Despite lacking global error gradients and relying on raw un-smoothed SGD, the model successfully learns the geometric shapes of the MNIST dataset and can classify and reconstruct them purely through local feature extraction. |
|
|
| * **Hardware:** Trained natively on NVIDIA GPUs utilizing Numba/CuPy. |
| * **Performance:** **87.90% Accuracy** on the Validation Set (achieved over 150 total epochs). |
| * **Generative Capability:** The Karl Decoder successfully hallucinates and reconstructs recognizable MNIST digits based solely on the latent representation passed from the Hinton Encoder. |
|
|
| ## Notes for the future |
| Accuracy of 74% achieved after 50 epochs; training loss still descending, suggesting higher accuracy is achievable with further training. After 150 epochs of continued training, the model reaches its architectural limit around 86–88% accuracy. |
| Future work: larger model, bigger batch sizes, or both — all feasible on consumer-grade hardware given the minimal VRAM footprint. |
| ## Conclusion & The "Efficient-Hardware" Theory |
| This experiment successfully proves that a deep neural network can learn at scale (1.3M parameters) without ever constructing a massive, memory-hogging backpropagation graph. |
| By relying on local Goodness evaluation and Predictive Coding errors, this architecture theoretically slashes the VRAM requirements of large-scale AI training, posing an experimental alternative to the hardware-heavy standard backpropagation paradigm. |
|
|
