| # DeepDream MLX: Agents |
|
|
| ## 1. The Mission |
| To resurrect the 2015 DeepDream aesthetic using modern 2025 Apple Silicon hardware, bypassing the need for archaic frameworks like Caffe or Torch7 by porting everything to native MLX. |
|
|
| ## 2. Training & Fine-Tuning Plan (The "Punch-Card" Revival) |
| In the "classic" days (Intel Caffe era), training a custom DeepDream model meant fine-tuning a GoogLeNet on a dataset of specific objects (e.g., slugs, eyes, cars) so the network would hallucinate *those specific things* when dreaming. |
|
|
| **The Roadmap for MLX Training:** |
|
|
| ### Phase 1: Dataset Prep |
| The `dream-creator` logic (from ProGamerGov) is still sound. We need: |
| 1. **Structure:** `dataset/class_name/*.jpg` (Standard PyTorch ImageFolder format). |
| 2. **Cleaning:** Remove corrupt images, deduplicate. |
| 3. **Resizing:** Resize to ~224x224 or 256x256. |
| 4. **Stats:** Calculate Mean/StdDev. |
|
|
| ### Phase 2: The Trainer (`train_dream.py`) |
| We need to write a native MLX training loop. |
| * **Base Model:** Load `googlenet_mlx.npz`. |
| * **Architecture:** InceptionV1 (GoogLeNet). |
| * **Layer Freezing:** |
| - **Critical:** Freeze early layers (`conv1`, `conv2`, `inception3a/b`) to preserve the "visual vocabulary" (edges, textures). |
| - **Train:** Retrain only the higher layers (`inception4c`, `inception5b`, `fc`) and the Auxiliary Classifiers. |
| * **Auxiliary Classifiers:** Inception has two side-branches (`aux1`, `aux2`) used for training stability. We must support training these or stripping them. |
| * **Loss:** Cross-Entropy. |
| * **Optimizer:** SGD with Momentum (classic) or Adam. |
|
|
| ### Phase 3: "Decorrelation" (The Secret Sauce) |
| `dream-creator` confirms that "Color Decorrelation" is key. |
| * **Matrix:** A 3x3 matrix calculated from the training set covariance. |
| * **Effect:** "Whitens" the input image gradients during dreaming, preventing the image from converging to a mono-color blob. |
| * **Implementation:** Port `data_tools/calc_cm.py` to MLX. |
|
|
| ## 3. Animation & Video Strategy |
| The "Zoom" video effect is the second pillar of DeepDream. |
| * **Logic:** Feedback Loop. |
| 1. Dream on Frame N. |
| 2. Zoom (Scale + Crop center) Frame N to create Frame N+1. |
| 3. Repeat. |
| * **Implementation:** A dedicated `dream_video.py` script. |
| * **Tech:** Use `scipy.ndimage.zoom` (same as original 2015 code) for the scaling, as MLX's `resize` might differ slightly in sub-pixel interpolation. |
|
|
| ## 4. Available Models & Wishlist |
| **Current:** |
| * `alexnet`: The raw, chaotic ancestor. |
| * `googlenet` (InceptionV1): The classic "slugs and dogs". |
| * `vgg16/19`: The "painterly" style transfer beast. |
| * `resnet50`: Modern, sharp, geometric. |
|
|
| **Wishlist (To Convert):** |
| * `inception_v3`: More refined hallucinations. |
| * `googlenet_places365`: Hallucinates landscapes/interiors. (Verified working via `convert.py --download googlenet` when URL is fixed/found). |
|
|
| ## 5. Hugging Face Hygiene |
| * **Repo:** `NickMystic/DeepDream-MLX` |
| * **LFS:** Track `*.npz`. |
| * **Cleanup:** Ensure `toConvert/` is empty of large raw files. |
| * **Banner:** `assets/deepdream_header.jpg`. |
|
|
| --- |
| *Docs derived from deep analysis of `dream-creator` and classic Caffe workflows.* |
|
|
