---
language: en
tags:
- forward-forward
- predictive-coding
- encoder-decoder
- mnist
- custom-architecture
- experimental
- cuda
- no-backprop
license: mit
datasets:
- mnist
metrics:
- accuracy
---

# RinKana/Amadeus-Hinton-Karl-1.3M-Encoder-Decoder-Model

This is a revolutionary, highly experimental from-scratch **1.3 Million Parameter Encoder-Decoder** model built to test the viability of Geoffrey Hinton's Forward-Forward (FF) Algorithm paired with a Predictive Coding Decoder (Karl).

**Unlike standard neural networks, this model does NOT use PyTorch's Autograd engine or global backpropagation.** 
It is trained purely via local custom Python CUDA Kernel matrix updates (Hebbian learning) executed directly in VRAM using Numba and CuPy.

## Model Details
* **Architecture:** Custom 2-Layer Encoder-Decoder Network
* **Encoder:** Hinton Contrastive Hebbian Learning (Forward-Forward)
* **Decoder:** Karl Predictive Coding 
* **Parameters:** 1.3 Million
* **Memory Footprint:** Exceptionally low (True backprop-free memory scaling)
* **Task:** Image Classification & Generative Reconstruction (MNIST)

## The Pure-CUDA Hardware Implementation
To adapt these biological learning rules to modern hardware without cheating by using standard automatic differentiation, we made several extreme architectural choices:

1. **Zero PyTorch Backprop:** The PyTorch `loss.backward()` and `optimizer.step()` commands are entirely absent from the training loop.
2. **Custom CUDA Kernels:** Weights are initialized directly in VRAM via `CuPy`, and memory mutation is handled by raw `@cuda.jit` threads processing localized math equations (Hebbian Delta Rules).
3. **Local Optimizers:** No global Adam optimizers were used. The network learns purely via Stochastic Gradient Descent (SGD) calculated natively within the hardware threads on a per-synapse basis.
4. **Label Injection:** For the Encoder to learn, the target classification label is injected directly into the image pixels as the target label is one-hot encoded into the first 10 pixel positions of the flattened 784-dimensional image vector forcing the network to learn the correlation between the label pixels and the image shape.

## Training & Performance
Despite lacking global error gradients and relying on raw un-smoothed SGD, the model successfully learns the geometric shapes of the MNIST dataset and can classify and reconstruct them purely through local feature extraction.

* **Hardware:** Trained natively on NVIDIA GPUs utilizing Numba/CuPy.
* **Performance:** **87.90% Accuracy** on the Validation Set (achieved over 150 total epochs).
* **Generative Capability:** The Karl Decoder successfully hallucinates and reconstructs recognizable MNIST digits based solely on the latent representation passed from the Hinton Encoder.

## Notes for the future
Accuracy of 74% achieved after 50 epochs; training loss still descending, suggesting higher accuracy is achievable with further training. After 150 epochs of continued training, the model reaches its architectural limit around 86–88% accuracy.
Future work: larger model, bigger batch sizes, or both — all feasible on consumer-grade hardware given the minimal VRAM footprint.
## Conclusion & The "Efficient-Hardware" Theory
This experiment successfully proves that a deep neural network can learn at scale (1.3M parameters) without ever constructing a massive, memory-hogging backpropagation graph. 
By relying on local Goodness evaluation and Predictive Coding errors, this architecture theoretically slashes the VRAM requirements of large-scale AI training, posing an experimental alternative to the hardware-heavy standard backpropagation paradigm.