File size: 973 Bytes
08226c4 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 |
---
tags:
- weight-space-learning
- neural-network-autoencoder
- autoencoder
- transformer
datasets:
- maximuspowers/muat-fourier-5
---
# Weight-Space Autoencoder (TRANSFORMER)
This model is a weight-space autoencoder trained on neural network activation weights/signatures.
It includes both an encoder (compresses weights into latent representations) and a decoder (reconstructs weights from latent codes).
## Model Description
- **Architecture**: Transformer encoder-decoder
- **Training Dataset**: maximuspowers/muat-fourier-5
- **Input Mode**: signature
- **Latent Dimension**: 256
## Tokenization
- **Chunk Size**: 64 weight values per token
- **Max Tokens**: 512
- **Metadata**: True
## Training Config
- **Loss Function**: contrastive
- **Optimizer**: adam
- **Learning Rate**: 0.0001
- **Batch Size**: 8
## Performance Metrics (Test Set)
- **MSE**: 0.088272
- **MAE**: 0.213148
- **RMSE**: 0.297107
- **Cosine Similarity**: 0.8509
- **R² Score**: 0.7242
|