---
library_name: transformers
tags: []
---
# FW Medium

This is the medium version of the bilinear transformers trained on FineWeb-edu.
The primary purpose of this model is interpretability, most design choices were made with that in mind.

The code to run this custom model can be found [here](https://github.com/tdooms/bilinear-decomposition), along with many utility functions for weight-based interpretability.

## Model Details
- 335 million parameters
- 16 layers
- 16 attention heads
- model dimension 1024
- bilinear MLP with expansion factor 4
- context length of 512
- trained for 32B tokens
- rotary positional embedding
- Mixtral [tokenizer](https://huggingface.co/mistral-community/Mixtral-8x22B-v0.1)