README.md

metadata

language: en
tags:
  - forward-forward
  - predictive-coding
  - encoder-decoder
  - mnist
  - custom-architecture
  - experimental
  - cuda
  - no-backprop
license: mit
datasets:
  - mnist
metrics:
  - accuracy

RinKana/Amadeus-Hinton-Karl-1.3M-Encoder-Decoder-Model

This is a revolutionary, highly experimental from-scratch 1.3 Million Parameter Encoder-Decoder model built to test the viability of Geoffrey Hinton's Forward-Forward (FF) Algorithm paired with a Predictive Coding Decoder (Karl).

Unlike standard neural networks, this model does NOT use PyTorch's Autograd engine or global backpropagation. It is trained purely via local custom Python CUDA Kernel matrix updates (Hebbian learning) executed directly in VRAM using Numba and CuPy.

Model Details

• Architecture: Custom 2-Layer Encoder-Decoder Network
• Encoder: Hinton Contrastive Hebbian Learning (Forward-Forward)
• Decoder: Karl Predictive Coding
• Parameters: 1.3 Million
• Memory Footprint: Exceptionally low (True backprop-free memory scaling)
• Task: Image Classification & Generative Reconstruction (MNIST)

The Pure-CUDA Hardware Implementation

To adapt these biological learning rules to modern hardware without cheating by using standard automatic differentiation, we made several extreme architectural choices:

1. Zero PyTorch Backprop: The PyTorch loss.backward() and optimizer.step() commands are entirely absent from the training loop.
2. Custom CUDA Kernels: Weights are initialized directly in VRAM via CuPy, and memory mutation is handled by raw @cuda.jit threads processing localized math equations (Hebbian Delta Rules).
3. Local Optimizers: No global Adam optimizers were used. The network learns purely via Stochastic Gradient Descent (SGD) calculated natively within the hardware threads on a per-synapse basis.
4. Label Injection: For the Encoder to learn, the target classification label is injected directly into the image pixels as the target label is one-hot encoded into the first 10 pixel positions of the flattened 784-dimensional image vector forcing the network to learn the correlation between the label pixels and the image shape.

Training & Performance

Despite lacking global error gradients and relying on raw un-smoothed SGD, the model successfully learns the geometric shapes of the MNIST dataset and can classify and reconstruct them purely through local feature extraction.

• Hardware: Trained natively on NVIDIA GPUs utilizing Numba/CuPy.
• Performance: 87.90% Accuracy on the Validation Set (achieved over 150 total epochs).
• Generative Capability: The Karl Decoder successfully hallucinates and reconstructs recognizable MNIST digits based solely on the latent representation passed from the Hinton Encoder.

Notes for the future

Accuracy of 74% achieved after 50 epochs; training loss still descending, suggesting higher accuracy is achievable with further training. After 150 epochs of continued training, the model reaches its architectural limit around 86–88% accuracy. Future work: larger model, bigger batch sizes, or both — all feasible on consumer-grade hardware given the minimal VRAM footprint.

Conclusion & The "Efficient-Hardware" Theory

This experiment successfully proves that a deep neural network can learn at scale (1.3M parameters) without ever constructing a massive, memory-hogging backpropagation graph. By relying on local Goodness evaluation and Predictive Coding errors, this architecture theoretically slashes the VRAM requirements of large-scale AI training, posing an experimental alternative to the hardware-heavy standard backpropagation paradigm.