metadata
license: mit
tags:
- approx-gelu
- batchnorm
- coca
- gated-fusion
- kaiming-normal
- matching
- novograd
- small
- sparse
- step
model_270805068_coca_small.py
Model Overview
A small-scale implementation of the coca architecture, built for matching tasks.
Architecture
- Architecture: coca
- Scale: small
- Attention: sparse
- Fusion strategy: gated fusion
- Task head: matching
- Activation: approx gelu
- Normalization: batchnorm
- Initialization: kaiming normal
Training
- Optimizer: novograd
- LR scheduler: step
Files
model_270805068_coca_small.py— main artifact of this repository
License
See the license field above.