README.md

---
license: apache-2.0
base_model: Audio8/ARK-ASR-3B
base_model_relation: finetune
library_name: transformers
pipeline_tag: automatic-speech-recognition
datasets:
  - google/fleurs
  - openslr/librispeech_asr
tags:
  - automatic-speech-recognition
  - speech
  - audio
  - transformers
  - pytorch
  - safetensors
  - vllm
  - ark-asr
  - multilingual
language:
  - zh
  - en
  - de
  - ja
  - fr
  - ko
  - es
  - pl
  - it
  - ro
  - hu
  - cs
  - nl
  - fi
  - hr
  - sk
  - sl
  - et
  - lt
---

# ARK-ASR-3B, steered to answer in the language it heard

**TL;DR**: ARK-ASR-3B frequently produces unintended English translations instead of transcribing non-English speech. On FLEURS, only **73.4 %** of non-English clips came back in the language they were spoken in. This adaptation takes that figure to **94.9 %**, with minimal impact elsewhere: English WER moves from 5.53 % to 5.57 % and Chinese CER from 3.22 % to 3.25 %, around 1 % relative on both. It uses the same architecture and inference interface as the base model, with no custom code, hooks, or runtime cost.

## The base model

[ARK-ASR-3B](https://huggingface.co/Audio8/ARK-ASR-3B) is a 3B multilingual speech-recognition model supporting nineteen languages. Its model card reports a 5.04 % average WER on the English Open ASR Leaderboard short-form benchmark.

The architecture, tokenizer, processor, remote code, and prompt format are unchanged.

## The problem: unintended translation

Given non-English speech, the base model often returns an unintended English translation rather than a transcript.

An unintended translation shares few words with its source-language reference. These outputs average **115.9 % WER** against the expected transcript. German WER is 48.1 % overall but under 9 % on clips answered in German.

WER does not distinguish a fluent English translation from a poor source-language transcript. The evaluation therefore reports both transcription error and the detected language of each output.

Measured over 250 FLEURS test clips per language, 4500 non-English clips in total, with the model card's own prompt (`"Please transcribe this audio."`):

- **26.6 % of non-English clips were not answered in the spoken language**, overwhelmingly because of unintended translation into English.
- Dutch is an extreme example: only 44.8 % of clips were answered in Dutch. The full per-language breakdown appears below.
- These English outputs are **unintended translations, not recognition errors**. FLEURS descends from FLoRes and is sentence-parallel, so each output can be scored against an English reference for the same sentence. They land 35 WER points closer to that English reference than in-language hypotheses do.

## Contrastive unintended translation dataset

The contrastive dataset was built from unintended translations produced by the base model. We ran the base model on FLEURS `dev` with the neutral prompt `"Please transcribe this audio."` and collected cases where it answered in English even though the audio was French, German, or Spanish. These are observed unintended translations, not synthetic English translations generated separately.

Each contrastive pair contains one audio clip under the same neutral prompt and two outputs:

- the source-language FLEURS transcript, representing the desired behaviour.
- the English output produced by the base model, representing the behaviour to suppress.

This gives 103 contrastive pairs:

| language | failures |
|---|---:|
| French | 26 |
| German | 55 |
| Spanish | 22 |
| **total** | **103** |

The dataset contains unintended translation cases rather than a general sample of multilingual speech. Its purpose is narrow: distinguish transcription in the spoken language from unintended translation into English. It targets languages the base model already learned to transcribe.

All contrastive pairs come from FLEURS `dev`. No FLEURS `test` clips were used to adapt the model. The results below use 250 `test` clips for each of the nineteen supported languages, including sixteen languages absent from the contrastive dataset.

## Model adaptation

Given the size of the contrastive dataset and the base model’s existing ability to transcribe these languages, we sought a minimal intervention to reduce the risk of disrupting its other capabilities. Activation-steering experiments identified layer 26 as an effective intervention point for unintended translation. We then trained a LoRA adapter at that layer using the 103 contrastive pairs and merged it into the released checkpoint, making it a drop-in replacement for the base model.

## Results

### Language accuracy, all nineteen supported languages

FLEURS test, 250 clips per language, neutral prompt, ordered by the steered model's error rate. **All values are percentages.** Error is reported as WER, except for Chinese, Japanese and Korean, which use CER. **Bold** marks where the steered model beats the base.

| | es | en | fr | zh | de | it | ja | nl | pl | cs | fi | hr | sk | hu | ro | sl | et | ko | lt |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| metric | WER | WER | WER | CER | WER | WER | CER | WER | WER | WER | WER | WER | WER | WER | WER | WER | WER | CER | WER |
| answered in English (base) | 21.6 | n/a | 28.4 | 0.0 | 40.8 | 28.4 | 18.4 | 55.2 | 25.6 | 20.8 | 8.0 | 22.8 | 25.2 | 19.2 | 44.8 | 18.4 | 6.8 | 41.6 | 17.2 |
| answered in English (steered) | **0.4** | n/a | **0.0** | 0.0 | **0.4** | **0.8** | **2.8** | **2.0** | **1.6** | **0.4** | **0.0** | **1.2** | **0.8** | **0.4** | **7.2** | **0.0** | **0.8** | **13.6** | **0.8** |
| error (base) | 25.5 | 6.4 | 30.6 | 8.3 | 48.1 | 35.7 | 48.3 | 62.8 | 49.1 | 44.9 | 42.5 | 50.0 | 53.9 | 56.1 | 63.7 | 60.8 | 53.0 | 104.3 | 73.0 |
| error (steered) | **6.1** | 6.5 | **7.2** | 8.4 | **8.7** | **10.5** | **16.7** | **17.7** | **21.1** | **21.8** | **28.6** | **29.8** | **30.5** | **32.1** | **34.8** | **41.7** | **42.5** | **44.0** | **56.9** |

**Answered in English** is how often a clip came back as English rather than the language spoken. This is the main unintended translation targeted by the model.

### Transcribed in the spoken language

The share of clips whose output was in the language actually being spoken, as a percentage. Same ordering. **Bold** marks where the steered model beats the base.

| | es | en | fr | zh | de | it | ja | nl | pl | cs | fi | hr | sk | hu | ro | sl | et | ko | lt |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| base | 78.4 | 100.0 | 71.2 | 99.6 | 59.2 | 71.2 | 81.6 | 44.8 | 74.4 | 74.4 | 92.0 | 72.4 | 60.0 | 80.0 | 54.0 | 77.6 | 91.2 | 57.2 | 81.6 |
| steered | **99.6** | 100.0 | **99.6** | 99.6 | **99.6** | **98.0** | **96.8** | **98.0** | **97.6** | **95.6** | **100.0** | **93.2** | **74.4** | **99.6** | **90.0** | **91.6** | **96.4** | **83.2** | **96.0** |

**Over the 4500 non-English clips: language accuracy increased from 73.4 % to 94.9 %** (±0.6 at 95 % confidence), mean error fell from 50.6 % to 25.5 %, and **970 clips recovered** into the language they were spoken in.

Only French, German, and Spanish were represented in the contrastive dataset. Improvements across non-English languages absent from that dataset show that the correction generalises beyond those examples and languages. English, also absent from the dataset, remains at 100 % language accuracy.

The tables are not complements because an output can be in a third language. For Slovak, English answers fall from 25.2 % to 0.8 %, but only 74.4 % of outputs are in Slovak. Most remaining outputs are in another Slavic language. This language confusion is separate from the unintended *English* translations targeted by the contrastive dataset.

### Impact on English and Chinese

The Open ASR Leaderboard English suite, approximately 82,000 utterances, scored with the leaderboard's Whisper normalisation:

| set | n | base | steered | relative change |
|---|---:|---:|---:|---:|
| AMI | 12620 | 9.75 % | 9.76 % | +0.1 % |
| Earnings22 | 2731 | 8.67 % | 8.71 % | +0.4 % |
| GigaSpeech | 19931 | 7.91 % | 7.96 % | +0.6 % |
| LibriSpeech clean | 2611 | 1.35 % | 1.39 % | +3.0 % |
| LibriSpeech other | 2932 | 2.80 % | 2.89 % | +3.3 % |
| SPGISpeech | 39341 | 2.72 % | 2.74 % | +0.6 % |
| VoxPopuli | 1830 | 5.53 % | 5.56 % | +0.6 % |
| **average** | | **5.53 %** | **5.57 %** | **+0.7 %** |
| AISHELL-1 (CER) | 6920 | 3.22 % | 3.25 % | +1.0 % |

Average English WER increases by 0.7 % relative. Most of the difference appears on clean read speech. AMI changes by 0.01 points. Measuring only on LibriSpeech would overstate the average impact.

> Absolute English WER is about 0.49 points above the base model card's published result. Its public evaluation references an unavailable `text_process` cleanup step. The base and steered columns above were evaluated identically and can be compared directly.

## Intended use

Use this model when the spoken language is not known in advance, or when non-English audio is a meaningful share of the input. It does not require a language-specific prompt or a separate language-detection stage.

For workloads that are exclusively English or Chinese, the base model avoids the small regressions reported above.

## Usage

This checkpoint is a drop-in replacement for [Audio8/ARK-ASR-3B](https://huggingface.co/Audio8/ARK-ASR-3B). Use the same interface and prompt format, replacing the model path with:

```python
model_path = "hypermind-official/ARK-ASR-3B-NoTranslate"
```

## Limitations and future work

- Korean retains unintended translation and mixed-script errors. Slovak mainly shows confusion with related Slavic languages rather than unintended English translation.
- The adaptation reduces unintended translation, not recognition errors. Lithuanian, Estonian and Slovene remain weak under both the base and adapted models.
- Evaluation is currently limited to read speech. Conversational, noisy, far-field, long-form and code-switched audio remain untested. Code-switched audio is a priority because it is a plausible weak point of the adaptation, which could favour a single output language.

## Evaluation methodology

The base and adapted checkpoints were evaluated on the same clips with the same harness and greedy decoding.

- **Output language**: classified with `py3langid`, restricted to the nineteen supported languages (759/760 on FLEURS references). Outputs under 12 characters are reported as undetermined, not mismatches.
- **English**: WER after Whisper's `EnglishTextNormalizer`, matching `open_asr_leaderboard`.
- **Chinese, Japanese and Korean**: character error rate with whitespace removed.
- **Other languages**: WER after Whisper's `BasicTextNormalizer` and per-language integer expansion.

Number formatting can inflate some reported error rates. Integer expansion was unavailable for Croatian, Estonian and Chinese.

## Attribution

This work was done by [Corentin Havet](https://huggingface.co/havetc) and [Etienne Balit](https://huggingface.co/EtienneB) at [Hypermind](https://huggingface.co/hypermind-official).

The base model, [Audio8/ARK-ASR-3B](https://huggingface.co/Audio8/ARK-ASR-3B), was released by [Audio8](https://huggingface.co/Audio8) under the Apache-2.0 license.

```bibtex
@misc{ark_asr_3b_notranslate,
  title  = {ARK-ASR-3B-NoTranslate: Unintended Translation Fix},
  author = {Havet, Corentin and Balit, Etienne},
  howpublished = {\url{https://huggingface.co/hypermind-official/ARK-ASR-3B-NoTranslate}},
  year   = {2026}
}
```