README.md

---
language:
- en
- multilingual
license: apache-2.0
library_name: codon
pipeline_tag: feature-extraction
tags:
- audio
- speech
- whisper
- encoder
- feature-extraction
- safetensors
---

# Whisper-Tiny Audio Encoder (codon format)

Encoder-only weights of [openai/whisper-tiny](https://huggingface.co/openai/whisper-tiny),
re-keyed into the naming convention used by the
[codon](https://github.com/CodonProject/codon-model) library so they can be loaded into
`codon.block.model.WhisperTinyAudioEncoder` with a single call.

This is a **feature extractor**, not a speech-to-text model: the decoder and the
tokenizer are intentionally **not** included, so it cannot transcribe audio on its own.

## Files

| File | Size | Tensors | Description |
|---|---|---|---|
| `whisper_tiny_encoder.safetensors` | 15.7 MB | 67 | Encoder weights, **fp16**, codon key convention |

## Model details

| | |
|---|---|
| Architecture | Whisper encoder (pre-LN Transformer, bidirectional self-attention) |
| Parameters | 8,208,384 (8.21M) |
| `d_model` | 384 |
| Encoder layers | 4 |
| Attention heads | 6 (head dim 64) |
| FFN dim | 1536, GELU |
| Mel bins | 80 |
| Max positions | 1500 |
| Output | `[B, T/2, 384]` for input mel `[B, 80, T]` |
| Precision | fp16 (upcast to fp32 on load for fp32 execution) |
| License | Apache-2.0 |

## Requirements

```bash
pip install codon-model==0.0.7b8
```

Source: [CodonProject/codon-model](https://github.com/CodonProject/codon-model)

## Usage

```python
import torch
from codon.block.model import WhisperTinyAudioEncoder

encoder = WhisperTinyAudioEncoder(pool_stride=1)
encoder.load('whisper_tiny_encoder.safetensors', strict=True)
encoder = encoder.float()               # weights are fp16; upcast for fp32 execution
encoder.eval()

mel = torch.randn(1, 80, 3000)          # 30 s of log-mel @ 100 Hz
with torch.no_grad():
    hidden, _ = encoder(mel)
print(hidden.shape)                     # torch.Size([1, 1500, 384])
```

Half-precision execution is also supported — just call `.half()` and feed a
half-precision mel instead (expect a larger numerical drift, ~6e-2, since the
arithmetic itself runs in fp16):

Loading straight from this repository:

```python
encoder = WhisperTinyAudioEncoder().from_remote()
```

### Downsampling for LLM consumption

The raw encoder emits one token per 20 ms (50 tokens/s), which is usually far too
dense for a language model. `pool_stride` appends a non-overlapping average pool
after the encoder (weights are unaffected):

```python
encoder = WhisperTinyAudioEncoder(pool_stride=8)
encoder = encoder.float()
with torch.no_grad():
    hidden, _ = encoder(torch.randn(1, 80, 3000))
print(hidden.shape)                     # torch.Size([1, 187, 384]) -> 6.25 tokens/s
```

## Tensor naming

Keys follow the codon convention (`proj_*` for projections). The full layout:

```
conv1.weight                      (384, 80, 3)
conv1.bias                        (384,)
conv2.weight                      (384, 384, 3)
conv2.bias                        (384,)
embed_positions.weight            (1500, 384)
layers.{0..3}.attn_norm.weight    (384,)
layers.{0..3}.attn_norm.bias      (384,)
layers.{0..3}.attn.proj_q.weight  (384, 384)
layers.{0..3}.attn.proj_q.bias    (384,)
layers.{0..3}.attn.proj_k.weight  (384, 384)     # no bias
layers.{0..3}.attn.proj_v.weight  (384, 384)
layers.{0..3}.attn.proj_v.bias    (384,)
layers.{0..3}.attn.proj_o.weight  (384, 384)
layers.{0..3}.attn.proj_o.bias    (384,)
layers.{0..3}.fn_norm.weight      (384,)
layers.{0..3}.fn_norm.bias        (384,)
layers.{0..3}.mlp.proj_fc1.weight (1536, 384)
layers.{0..3}.mlp.proj_fc1.bias   (1536,)
layers.{0..3}.mlp.proj_fc2.weight (384, 1536)
layers.{0..3}.mlp.proj_fc2.bias   (384,)
norm.weight                       (384,)
norm.bias                         (384,)
```

Note that `proj_k` has **no bias** — this mirrors the original Whisper design
(its `k_proj` is the only projection without a bias in both the encoder and the
decoder). The positional table uses Whisper's own layout: `weight[:, :192]` holds
sine values and `weight[:, 192:]` holds cosine values for
`inv_freq = exp(-log(10000) * arange(192) / 192)`.

## Provenance and verification

Weights are copied from `openai/whisper-tiny` (`model.encoder.*`), renamed, and
saved as fp16. No values were modified: converting the fp32 export to fp16 and back
to fp32 is bit-exact element-wise, i.e. the source values already only carried fp16
precision (max absolute weight magnitude is ~16, well inside fp16 range).

The codon implementation was verified against the official `transformers`
implementation (`WhisperModel(...).encoder`) with both models loading the same
checkpoint (upcast to fp32) and consuming the same input:

| mel length | output shape | max abs difference |
|---|---|---|
| 3000 | `[2, 1500, 384]` | 5.9e-05 |
| 1000 (padded) | `[2, 500, 384]` | 3.9e-05 |
| 512 (padded) | `[2, 256, 384]` | 2.2e-05 |

Differences are at fp32 rounding level, not implementation differences.

## Limitations

- **Encoder only.** No decoder, no vocabulary, no tokenizer — cannot produce text.
- **Fixed 30 s input.** Whisper pads every input to 3000 mel frames. Short clips
  must be zero-padded to 3000 and the padded output frames discarded; the padded
  region does influence the valid region through convolution and LayerNorm, so it
  is not equivalent to running the encoder on a shorter input.
- **Not causal / not streaming.** The encoder is bidirectional and stateless, so it
  cannot be used for incremental streaming without recomputing a full window.
- **Precision.** Tensors are stored as fp16. The fp32 -> fp16 -> fp32 round trip is
  element-wise exact, so no information was lost relative to the source export, and
  running the model in fp32 after upcasting reproduces the fp32 reference exactly.
  Native fp16 execution (`model.half()`) is supported but introduces the usual
  half-precision drift (~6e-2 on this model).

## Citation

If you use these weights, please cite the original work:

```bibtex
@misc{radford2022whisper,
  doi = {10.48550/ARXIV.2212.04356},
  url = {https://arxiv.org/abs/2212.04356},
  author = {Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg and McLeavey, Christine and Sutskever, Ilya},
  title = {Robust Speech Recognition via Large-Scale Weak Supervision},
  publisher = {arXiv},
  year = {2022},
  copyright = {arXiv.org perpetual, non-exclusive license}
}
```

## Acknowledgements

Original model by OpenAI. This repository only re-keys the encoder weights for use
with [codon](https://github.com/CodonProject/codon-model).