| tags: | |
| - weight-space-learning | |
| - neural-network-autoencoder | |
| - autoencoder | |
| - transformer | |
| datasets: | |
| - maximuspowers/muat-fourier-5 | |
| # Weight-Space Autoencoder (TRANSFORMER) | |
| This model is a weight-space autoencoder trained on neural network activation weights/signatures. | |
| It includes both an encoder (compresses weights into latent representations) and a decoder (reconstructs weights from latent codes). | |
| ## Model Description | |
| - **Architecture**: Transformer encoder-decoder | |
| - **Training Dataset**: maximuspowers/muat-fourier-5 | |
| - **Input Mode**: signature | |
| - **Latent Dimension**: 256 | |
| ## Tokenization | |
| - **Chunk Size**: 64 weight values per token | |
| - **Max Tokens**: 512 | |
| - **Metadata**: True | |
| ## Training Config | |
| - **Loss Function**: contrastive | |
| - **Optimizer**: adam | |
| - **Learning Rate**: 0.0001 | |
| - **Batch Size**: 8 | |
| ## Performance Metrics (Test Set) | |
| - **MSE**: 0.088272 | |
| - **MAE**: 0.213148 | |
| - **RMSE**: 0.297107 | |
| - **Cosine Similarity**: 0.8509 | |
| - **R² Score**: 0.7242 | |