File size: 2,463 Bytes
f72bc8c 2965b7e f72bc8c 2965b7e f72bc8c 2965b7e f72bc8c 2965b7e f72bc8c |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 |
---
license: apache-2.0
language:
- en
tags:
- speech
- audio
- data2vec
- distillation
- feature-extraction
library_name: transformers
pipeline_tag: feature-extraction
---
# Distilled Speech Encoder
A Data2Vec-style bidirectional speech encoder trained via distillation from AuriStream models.
## Model Details
- **Architecture**: 12-layer transformer with RoPE positional encoding
- **Hidden size**: 768
- **Attention heads**: 12
- **Parameters**: ~85M
- **Teacher model**: `TuKoResearch/AuriStream100M_40Pred_BigAudioDataset_500k`
- **Training step**: 100000
- **Input**: 16kHz raw audio waveform
- **Output**: 50Hz contextualized representations (768-dim)
## Usage
```python
from transformers import AutoModel, Wav2Vec2FeatureExtractor
import torch
# Load model and feature extractor
model = AutoModel.from_pretrained("TuKoResearch/AuriStreamDistill_100M40PredTeacher_librispeech960", trust_remote_code=True)
model.eval() # Important for inference!
feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained("TuKoResearch/AuriStreamDistill_100M40PredTeacher_librispeech960")
# Prepare audio (16kHz, mono)
audio = torch.randn(16000).numpy() # 1 second of audio
# Extract features
inputs = feature_extractor(audio, return_tensors="pt", sampling_rate=16000)
with torch.no_grad():
outputs = model(inputs.input_values, output_hidden_states=True)
# Get representations
last_hidden = outputs.last_hidden_state # (1, 50, 768) for 1 second
all_hidden = outputs.hidden_states # Tuple of 13 tensors
```
## Hidden States
When `output_hidden_states=True`, the model returns hidden states from all layers:
- `hidden_states[0]`: Feature projection output (after conv encoder + projection)
- `hidden_states[1]` to `hidden_states[12]`: Transformer layer outputs
- `hidden_states[12]`: Final layer output (same as `last_hidden_state`)
This makes the model suitable for linear probing experiments at different layers.
## Training
This model was trained using Data2Vec-style distillation:
1. A frozen AuriStream teacher model generates target representations
2. The student sees masked audio and learns to predict teacher representations
3. Loss is computed only on masked positions
## Citation
If you use this model, please cite:
```bibtex
@misc{distilled_speech_encoder,
title={Distilled Speech Encoder},
author={TuKo Research},
year={2025},
url={https://huggingface.co/TuKoResearch/AuriStreamDistill_100M40PredTeacher_librispeech960}
}
```
|