File size: 2,559 Bytes
93ebaf2 dbd8c4c 93ebaf2 dbd8c4c 93ebaf2 dbd8c4c 93ebaf2 dbd8c4c 93ebaf2 dbd8c4c 93ebaf2 dbd8c4c 93ebaf2 dbd8c4c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 | ---
library_name: transformers
license: mit
tags:
- custom_code
- multimodal
- mixture-of-experts
- weight-sharing
---
# The Net
A 20T-scale multimodal architecture for text, audio, images, video and EEG. Modality-specific
encoders feed a hierarchical sparse mixture-of-experts core with recurrent memory, and media
decoders turn the shared representation back into each output modality.
## Architecture
The Net has **11 major components**: five modality encoders, multiscale fusion, a shared world
model, three media decoders and a memory controller.
```text
Text · Audio · Images · Video · EEG
↓
Modality-specific encoders
↓
Multiscale fusion
↓
Hierarchical sparse MoE core ↔ Memory
↓
Text head · Audio / Image / Video decoders
```
Encoders combine modality-specific stems with learned chunking, pooling and dechunking. Fusion
aligns their representations before the shared core. The core uses attention, SwiGLU experts and
RMSNorm across three widths. Residual detail paths connect hierarchy levels; recurrent memory
carries state between segments.
| Modality | Interface |
| --- | --- |
| Text | Byte-level tokens and next-token logits |
| Audio | Waveform frames, spectral features and codebook logits |
| Images | RGB patches and image latents |
| Video | Spatiotemporal tubelets and video latents |
| EEG | Multichannel signals with electrode adjacency |
## Model details
| Field | Value |
| --- | --- |
| Architecture parameters | 20.09T |
| Attention / FFN blocks | 686 |
| Core widths | 8,192 / 12,288 / 16,384 |
| Experts | 128 per layer, top-2 routing |
| Tokenizer | Byte-level |
| Weights | Constant initialization (untrained) |
| License | MIT |
## Files
| File | Contents |
| --- | --- |
| `config.json` | Model configuration |
| `configuration_omni.py` | `OmniConfig` |
| `modeling_omni.py` | `OmniModel` module implementation |
| `tokenizer.json` | Byte-level tokenizer |
| `tensor_shapes.json` | Architecture tensor names and dimensions |
## FAQ
**Q: Which modalities does the architecture cover?**
Text, audio, images, video and EEG, connected through a shared hierarchical core.
**Q: How are the experts organized?**
The shared core spans three hierarchy levels, with 128 experts per layer and top-2 routing.
Multiscale fusion connects the modality encoders to the core; media decoders and a text head
provide the output interfaces.
---
Release format, storage layout and parameter counting: [EXPERIMENT.md](EXPERIMENT.md)
|