README.md

---
license: cc-by-4.0
language:
- tig
tags:
- hubert
- speech
- self-supervised-learning
- audio
- tigre
- low-resource
pipeline_tag: feature-extraction
---

# Tigre HuBERT

A self-supervised speech representation model for **Tigre** (ISO 639-3: `tig`), a
Semitic language spoken primarily in Eritrea. This is, to our knowledge, one of
the first publicly available speech models for Tigre.

> **Status:** this is a pretrained *representation* model, not a
> fine-tuned task model. It hasn't yet been benchmarked on a downstream task
> (e.g. ASR word/character error rate). Validation loss during pretraining
> improved consistently and has plateaued (see Training below), which is a
> good sign for representation quality, but the honest way to know how well
> it performs on a specific task is to fine-tune it for that task and measure
> the result directly. If you do that, we'd love to hear how it goes.

## What this model is

[HuBERT](https://arxiv.org/abs/2106.07447) (Hidden-Unit BERT) learns speech
representations by predicting cluster assignments of masked audio segments,
without needing any transcribed text. That makes it well-suited to
low-resource languages like Tigre, where labeled speech-to-text data is
scarce but raw audio is more attainable.

The output of this model is **not text** — it's a sequence of learned
embeddings per audio frame. To get something task-specific (e.g. speech
recognition, speaker identification, language identification), you fine-tune
a small task head on top of these representations.

## How it was trained

- **Data:** ~500 hours of Tigre speech audio.
- **Recipe:** the standard two-iteration HuBERT pretraining procedure
  ([fairseq's implementation](https://github.com/facebookresearch/fairseq/tree/main/examples/hubert)):
  - **Iteration 1:** pseudo-labels from k-means (k=100) on raw MFCC
    features — a weak but label-free starting signal.
  - **Iteration 2:** the iteration-1 model's own layer-6 hidden features
    are re-clustered (k=500) into new, better pseudo-labels, and a fresh
    model is trained from scratch against those. This is the standard
    recipe's main quality jump, since the model now learns from
    speech-aware clusters instead of raw acoustic ones.
- **Architecture:** HuBERT-base (~90M parameters, 12 transformer layers).
- **Note on scale:** the original HuBERT paper validated this exact
  2-iteration recipe on 960 hours of English speech, and only used a 3rd
  iteration at a much larger scale (60,000+ hours). At ~500 hours, this
  model is trained at roughly half the data scale the base recipe was
  designed for — a reasonable and appropriately-sized recipe for this
  amount of data, but worth keeping in mind when setting quality
  expectations relative to large, high-resource-language HuBERT models.

### Iteration-2 training curve

![Validation loss over iteration-2 pretraining](validation_loss.png)

Validation loss (the masked-unit prediction objective, not a downstream
task metric) over iteration-2 pretraining, from epoch 60 to the point
training was stopped once improvement plateaued (under 1% change over the
last few checkpoints). This shows the pretraining objective converging
smoothly — it does not measure accuracy on any specific downstream task.

## Basic usage

This model outputs frame-level embeddings, not text. Example: extracting
features from a 16kHz mono audio clip.

```python
import torch
from fairseq import checkpoint_utils

# Download checkpoint_best.pt from this repo first
models, cfg, task = checkpoint_utils.load_model_ensemble_and_task(["checkpoint_best.pt"])
model = models[0].eval()

# waveform: torch.FloatTensor of shape [1, num_samples], 16kHz, mono
with torch.inference_mode():
    features, _ = model.extract_features(
        source=waveform,
        padding_mask=None,
        mask=False,       # inference, not the masked-training objective
        output_layer=None # final transformer layer
    )

# features: [1, num_frames, hidden_dim] -- one embedding per ~20ms of audio
```

`fairseq` isn't a well-maintained PyPI package — install it from source:

```bash
git clone https://github.com/facebookresearch/fairseq.git
cd fairseq && pip install --editable .
```

## What this is useful for

- **Fine-tuning for Tigre ASR** — attach a CTC head and fine-tune on a
  (even fairly small) labeled Tigre speech-to-text dataset.
- **Speaker or language identification** — the embeddings can feed a
  lightweight classifier for tasks that don't need text transcription at all.
- **A starting point, not an endpoint** — as a foundation model for further
  Tigre speech research, in a language with very little existing tooling.

## Limitations

- Not evaluated on any downstream task yet — treat performance claims with
  appropriate skepticism until you've tested it on your own task/data.
- Trained on ~500 hours, notably less than the 960 hours the base recipe
  was designed around — representation quality may reflect that.
- No fine-tuned ASR head is included in this repo; this model produces
  embeddings, not transcriptions, out of the box.

## Citation

If you use this model, please cite the original HuBERT paper:

```bibtex
@article{hsu2021hubert,
  title={HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units},
  author={Hsu, Wei-Ning and Bolte, Benjamin and Tsai, Yao-Hung Hubert and Lakhotia, Kushal and Salakhutdinov, Ruslan and Mohamed, Abdelrahman},
  journal={IEEE/ACM Transactions on Audio, Speech, and Language Processing},
  year={2021}
}
```