metadata
license: bsd-3-clause
tags:
- base
- contrastive
- cross-attention
- exponential
- instancenorm
- linear
- orthogonal
- relu
- sgd
- tiny-transformer
model_311813953_tiny_transformer_base.py
Model Overview
A base-scale implementation of the tiny transformer architecture, built for contrastive tasks.
Architecture
- Architecture: tiny transformer
- Scale: base
- Attention: linear
- Fusion strategy: cross attention
- Task head: contrastive
- Activation: relu
- Normalization: instancenorm
- Initialization: orthogonal
Training
- Optimizer: sgd
- LR scheduler: exponential
Files
model_311813953_tiny_transformer_base.py— main artifact of this repository
License
See the license field above.