| --- |
| library_name: pytorch |
| tags: |
| - pytorch |
| - implicit-neural-representation |
| - coordinate-network |
| - multimodal |
| - video |
| - audio |
| --- |
| |
| # Bad Apple Unified Neural Representation |
|
|
| This is a compact coordinate-based neural representation of the complete |
| *Bad Apple!!* shadow video and its stereo audio. |
|
|
| The checkpoint does not store ordinary video frames or compressed audio. Given |
| a normalized time and pixel coordinate, the network predicts the image |
| brightness. Given a normalized time coordinate with audio conditioning, the |
| same network predicts the stereo waveform. |
|
|
| ## Generated Result |
|
|
| <!-- |
| Upload your rendered MP4 as demo.mp4, then replace YOUR_USERNAME and |
| YOUR_REPOSITORY below. |
| --> |
|
|
| <video controls width="100%"> |
| <source src="https://huggingface.co/Ma7ee7/Bad-Apple_Unified-Model-Test/resolve/main/demo.mp4" type="video/mp4"> |
| Your browser does not support embedded video. |
| </video> |
|
|
| [Download or open the generated video](./demo.mp4) |
|
|
| ## Run |
|
|
| Install the Python requirements and make sure `ffmpeg` is available on your |
| system. Then render the checkpoint: |
|
|
| ```bash |
| python bad_apple_nn.py render model.pt --audio-source generated |
| ``` |
|
|
| The generated MP4 is written to `outputs/`. |
|
|
| ## What The Model Is |
|
|
| This is **one unified multimodal model**, not two independently trained models. |
|
|
| Video and audio share: |
|
|
| - one normalized time representation; |
| - Fourier coordinate features; |
| - an interpolated learned time-memory; |
| - modality conditioning; and |
| - one residual neural-network trunk. |
|
|
| The shared trunk ends in two small task-specific output heads: |
|
|
| - a video head that predicts grayscale pixel brightness; |
| - an audio head that predicts two stereo waveform channels. |
|
|
| Separate output heads are necessary because pixels and audio samples have |
| different output shapes, but the representation and main network are shared |
| and trained together in one checkpoint. |
|
|
| ## Model Details |
|
|
| | Property | Value | |
| |---|---:| |
| | Architecture | Unified coordinate neural field (`unified-v3`) | |
| | Trainable parameters | 1,111,571 | |
| | Training video resolution | 192 x 144 | |
| | Rendered resolution at default 3x scale | 576 x 432 | |
| | Frames | 6,572 | |
| | Frame rate | 30 FPS | |
| | Duration | About 3 minutes 39 seconds | |
| | Generated audio | 16 kHz stereo | |
| | Best checkpoint step | 59,000 / 80,000 | |
| | Video pixel accuracy | 99.6% | |
| | Silhouette IoU | 0.991 | |
|
|
| The inference weights occupy approximately 4.45 MB in FP32. The uploaded |
| training checkpoint may be larger because it contains both regular and |
| exponential-moving-average weights. Rendering uses the EMA weights by default. |
|
|
| ## Files |
|
|
| | File | Purpose | |
| |---|---| |
| | `model.pt` | Unified video-and-audio checkpoint | |
| | `bad_apple_nn.py` | Model definition and renderer | |
| | `requirements.txt` | Python dependencies | |
| | `demo.mp4` | Example generated result | |
|
|
| ## Notes |
|
|
| - This is a custom PyTorch model, not a Transformers model. |
| - It is an implicit neural representation specialized to one video. |
| - The checkpoint generates both the frames and audio. |
| - `ffmpeg` is required to encode and combine the rendered streams. |
| - This repository is an experimental and educational demonstration. Rights to |
| the original music and animation remain with their respective owners. |
|
|
