FAIRC/mowe-dense-768-19l

Dense 768, 19 Layers

Inference checkpoint for the dense baseline released with Mixture of Wide Experts.

Property	Value
Parameters	103.46M
Transformer layers	19
Model width	768
Attention heads	12
FFN hidden width	455
Context length	1,024
Tokenizer	GPT-2

The checkpoint is an inference-only safetensors state dictionary. It contains no optimizer state. This is a base language model and is not instruction-tuned.

Loading

Install the companion MoWE extension for OpenLanguageModel, then load this repository directory:

import torch

from mowe import load_checkpoint

model = load_checkpoint(
    "/path/to/dense-768-19l",
    device="cuda",
    dtype=torch.bfloat16,
).eval()

Limitations

This is a small research model. It may produce incorrect, biased, unsafe, or memorized text and should not be used as a production assistant without separate evaluation and safeguards.

License

Released under the Apache License 2.0. Please cite the accompanying MoWE paper.