metadata
license: apache-2.0
tags:
- cross-attention
- efficientformer
- generation
- mish
- rmsnorm
- sgd
- small
- sparse
- step
- trunc-normal
model_382082472_efficientformer_small.py
Model Overview
A small-scale implementation of the efficientformer architecture, built for generation tasks.
Architecture
- Architecture: efficientformer
- Scale: small
- Attention: sparse
- Fusion strategy: cross attention
- Task head: generation
- Activation: mish
- Normalization: rmsnorm
- Initialization: trunc normal
Training
- Optimizer: sgd
- LR scheduler: step
Files
model_382082472_efficientformer_small.py— main artifact of this repository
License
See the license field above.