Update README.md

README.md

@@ -1,3 +1,186 @@
 ---
-license: mit
+language:
+- en
+- multilingual
+license: apache-2.0
+library_name: codon
+pipeline_tag: feature-extraction
+tags:
+- audio
+- speech
+- whisper
+- encoder
+- feature-extraction
+- safetensors
 ---
+
+# Whisper-Tiny Audio Encoder (codon format)
+
+Encoder-only weights of [openai/whisper-tiny](https://huggingface.co/openai/whisper-tiny),
+re-keyed into the naming convention used by the
+[codon](https://github.com/CodonProject/codon-model) library so they can be loaded into
+`codon.block.model.WhisperTinyAudioEncoder` with a single call.
+
+This is a **feature extractor**, not a speech-to-text model: the decoder and the
+tokenizer are intentionally **not** included, so it cannot transcribe audio on its own.
+
+## Files
+
+| File | Size | Tensors | Description |
+|---|---|---|---|
+| `whisper_tiny_encoder.safetensors` | 15.7 MB | 67 | Encoder weights, **fp16**, codon key convention |
+
+## Model details
+
+| | |
+|---|---|
+| Architecture | Whisper encoder (pre-LN Transformer, bidirectional self-attention) |
+| Parameters | 8,208,384 (8.21M) |
+| `d_model` | 384 |
+| Encoder layers | 4 |
+| Attention heads | 6 (head dim 64) |
+| FFN dim | 1536, GELU |
+| Mel bins | 80 |
+| Max positions | 1500 |
+| Output | `[B, T/2, 384]` for input mel `[B, 80, T]` |
+| Precision | fp16 (upcast to fp32 on load for fp32 execution) |
+| License | Apache-2.0 |
+
+## Requirements
+
+```bash
+pip install codon-model==0.0.7b8
+```
+
+Source: [CodonProject/codon-model](https://github.com/CodonProject/codon-model)
+
+## Usage
+
+```python
+import torch
+from codon.block.model import WhisperTinyAudioEncoder
+
+encoder = WhisperTinyAudioEncoder(pool_stride=1)
+encoder.load('whisper_tiny_encoder.safetensors', strict=True)
+encoder = encoder.float()               # weights are fp16; upcast for fp32 execution
+encoder.eval()
+
+mel = torch.randn(1, 80, 3000)          # 30 s of log-mel @ 100 Hz
+with torch.no_grad():
+    hidden, _ = encoder(mel)
+print(hidden.shape)                     # torch.Size([1, 1500, 384])
+```
+
+Half-precision execution is also supported — just call `.half()` and feed a
+half-precision mel instead (expect a larger numerical drift, ~6e-2, since the
+arithmetic itself runs in fp16):
+
+Loading straight from this repository:
+
+```python
+encoder = WhisperTinyAudioEncoder().from_remote()
+```
+
+### Downsampling for LLM consumption
+
+The raw encoder emits one token per 20 ms (50 tokens/s), which is usually far too
+dense for a language model. `pool_stride` appends a non-overlapping average pool
+after the encoder (weights are unaffected):
+
+```python
+encoder = WhisperTinyAudioEncoder(pool_stride=8)
+encoder = encoder.float()
+with torch.no_grad():
+    hidden, _ = encoder(torch.randn(1, 80, 3000))
+print(hidden.shape)                     # torch.Size([1, 187, 384]) -> 6.25 tokens/s
+```
+
+## Tensor naming
+
+Keys follow the codon convention (`proj_*` for projections). The full layout:
+
+```
+conv1.weight                      (384, 80, 3)
+conv1.bias                        (384,)
+conv2.weight                      (384, 384, 3)
+conv2.bias                        (384,)
+embed_positions.weight            (1500, 384)
+layers.{0..3}.attn_norm.weight    (384,)
+layers.{0..3}.attn_norm.bias      (384,)
+layers.{0..3}.attn.proj_q.weight  (384, 384)
+layers.{0..3}.attn.proj_q.bias    (384,)
+layers.{0..3}.attn.proj_k.weight  (384, 384)     # no bias
+layers.{0..3}.attn.proj_v.weight  (384, 384)
+layers.{0..3}.attn.proj_v.bias    (384,)
+layers.{0..3}.attn.proj_o.weight  (384, 384)
+layers.{0..3}.attn.proj_o.bias    (384,)
+layers.{0..3}.fn_norm.weight      (384,)
+layers.{0..3}.fn_norm.bias        (384,)
+layers.{0..3}.mlp.proj_fc1.weight (1536, 384)
+layers.{0..3}.mlp.proj_fc1.bias   (1536,)
+layers.{0..3}.mlp.proj_fc2.weight (384, 1536)
+layers.{0..3}.mlp.proj_fc2.bias   (384,)
+norm.weight                       (384,)
+norm.bias                         (384,)
+```
+
+Note that `proj_k` has **no bias** — this mirrors the original Whisper design
+(its `k_proj` is the only projection without a bias in both the encoder and the
+decoder). The positional table uses Whisper's own layout: `weight[:, :192]` holds
+sine values and `weight[:, 192:]` holds cosine values for
+`inv_freq = exp(-log(10000) * arange(192) / 192)`.
+
+## Provenance and verification
+
+Weights are copied from `openai/whisper-tiny` (`model.encoder.*`), renamed, and
+saved as fp16. No values were modified: converting the fp32 export to fp16 and back
+to fp32 is bit-exact element-wise, i.e. the source values already only carried fp16
+precision (max absolute weight magnitude is ~16, well inside fp16 range).
+
+The codon implementation was verified against the official `transformers`
+implementation (`WhisperModel(...).encoder`) with both models loading the same
+checkpoint (upcast to fp32) and consuming the same input:
+
+| mel length | output shape | max abs difference |
+|---|---|---|
+| 3000 | `[2, 1500, 384]` | 5.9e-05 |
+| 1000 (padded) | `[2, 500, 384]` | 3.9e-05 |
+| 512 (padded) | `[2, 256, 384]` | 2.2e-05 |
+
+Differences are at fp32 rounding level, not implementation differences.
+
+## Limitations
+
+- **Encoder only.** No decoder, no vocabulary, no tokenizer — cannot produce text.
+- **Fixed 30 s input.** Whisper pads every input to 3000 mel frames. Short clips
+  must be zero-padded to 3000 and the padded output frames discarded; the padded
+  region does influence the valid region through convolution and LayerNorm, so it
+  is not equivalent to running the encoder on a shorter input.
+- **Not causal / not streaming.** The encoder is bidirectional and stateless, so it
+  cannot be used for incremental streaming without recomputing a full window.
+- **Precision.** Tensors are stored as fp16. The fp32 -> fp16 -> fp32 round trip is
+  element-wise exact, so no information was lost relative to the source export, and
+  running the model in fp32 after upcasting reproduces the fp32 reference exactly.
+  Native fp16 execution (`model.half()`) is supported but introduces the usual
+  half-precision drift (~6e-2 on this model).
+
+## Citation
+
+If you use these weights, please cite the original work:
+
+```bibtex
+@misc{radford2022whisper,
+  doi = {10.48550/ARXIV.2212.04356},
+  url = {https://arxiv.org/abs/2212.04356},
+  author = {Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg and McLeavey, Christine and Sutskever, Ilya},
+  title = {Robust Speech Recognition via Large-Scale Weak Supervision},
+  publisher = {arXiv},
+  year = {2022},
+  copyright = {arXiv.org perpetual, non-exclusive license}
+}
+```
+
+## Acknowledgements
+
+Original model by OpenAI. This repository only re-keys the encoder weights for use
+with [codon](https://github.com/CodonProject/codon-model).