Adding `safetensors` variant of this model

.gitattributes

@@ -33,4 +33,3 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
-*.gguf filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,105 +1,3 @@
----
-language:
-- fr
-- es
-- pt
-- de
-- en
-metrics:
-- bleu
-- comet
-base_model:
-- kyutai/hibiki-zero-3b-pytorch-bf16
-pipeline_tag: audio-to-audio
----
-# Hibiki-Zero
-
-[Hibiki-Zero](https://github.com/kyutai-labs/hibiki-zero) is a model for **simultaneous speech translation**. Traditional approaches for building simultaneous translation systems rely on supervised training with word-level aligned data between the source and the target content. Hibiki-Zero eliminates the need for word-level alignments entirely so that it fundamentally simplifies the training pipeline and enables **seamless scaling to multiple languages** with varying grammatical structures.
-
-Hibiki-Zero supports translation from 🇫🇷 French, 🇪🇸 Spanish, 🇵🇹 Portuguese and 🇩🇪 German to 🇬🇧 English. At inference, Hibiki-Zero **adapts its flow** to accumulate just enough context so that it produces a real-time and natural speech translation with voice transfer along with a text translation. Hibiki-Zero can also be **adapted to a new input language with less than 1000h of speech data**.
-
----
-
-## Model Details
-
-This is the model simply referred to as *Hibiki-Zero* in our [paper][paper], a 3B-parameter hierarchical Transformer producing speech and text tokens at a framerate of 12.5Hz, with audio being generated at a 2.2kbps bitrate.
-
-### Model Description
-
-Hibiki-Zero is a decoder-only model that can receive and generate audio tokens produced by the the streaming neural audio codec [Mimi](https://huggingface.co/kyutai/mimi). It leverages the same **multistream** architecture as [Moshi](https://arxiv.org/abs/2410.00037) or [Hibiki](https://arxiv.org/abs/2502.03382) to model source and target speech jointly. This allows Hibiki-Zero to continuously process the input stream while generating the target speech and text tokens at a constant framerate of 12.5Hz producing a **continuous output audio stream**, along with timestamped text translation. Hibiki-Zero consist of a main backbone of 3 billion parameters.
-
-At inference, Hibiki-Zero continuously encodes the input user speech and produces **real-time speech and text translation**. Our model relies on simple temperature sampling and is thus compatible with batching unlike models using complex inference policies. It is also possible to run **batched inference 3x faster than real-time** on a single H100 GPU as demonstrated by our [inference code](https://github.com/kyutai-labs/hibiki-zero). Hibiki-Zero only supports a single speaker in a single language per session. However, it shows zero-shot capabilities for translation with voice transfer of multiple speakers with different languages in the same audio.
-
-
-- **Developed by:**  Kyutai
-- **Model type:** Simultaneous speech-to-speech and speech-to-text translation.
-- **Languages:** {French,Spanish,Portuguese,German}-to-English
-- **License:** CC BY-NC-SA 4.0
-
-### Model Sources 
-
-- **Paper:** [Simultaneous Speech-to-Speech Translation Without Aligned Data][paper]
-- **Inference code:** [github.com/kyutai-labs/hibiki-zero](https://github.com/kyutai-labs/hibiki-zero) 
-- **Examples:** [huggingface.co/spaces/kyutai/hibiki-zero-samples](https://huggingface.co/spaces/kyutai/hibiki-zero-samples) 
-
----
-
-## Usage
-
-### Direct Use
-
-The model can be used for streaming translation from French, Spanish, Portuguese and German to English in real-time settings, or for batched simultaneous translation of many input sequences. It is robust to noisy conditions and is trained on sequences up to 120 seconds. 
-
-
-### Downstream Use
-
-Some components of the model can be used independently or repurposed relatively easily. For instance the [Mimi](https://huggingface.co/kyutai/mimi) codec is a state-of-the-art audio neural codec that combines semantic and acoustic information into audio tokens running at 12Hz and a bitrate of 1.1kbps, which make it particularly adapted to train speech language models or text-to-speech systems. Regarding the main Hibiki-Zero architecture, we demonstrated that it was possible to finetune it to adapt to a new input language with less than 1000h of speech and explicit the method in our [paper][paper].
-
-
-### Out-of-Scope Use
-
-The model is not intended to be used to impersonate other people or any malicious use of any kind.
-
-
-## How to Get Started with the Model
-
-See the [README](https://github.com/kyutai-labs/hibiki-zero) file for the inference code.
-
----
-
-## Training Details
-
-### Training Data
-
-- *Textual data:* The underlying text LLM model [Helium-1-2B](https://huggingface.co/kyutai/helium-1-2b) is trained on a mix of data including: Wikipedia, Stack Exchange, open-access scientific articles (from peS2o) and Common Crawl.
-
-- *Audio data:*
-  - **Unsupervised audio dataset:** This dataset used for audio pretraining is a large collection readily available audio content in French, Spanish, Portuguese, German and English. Our data mixture contains approximately 12% of audio in each source language, 50% of English and less than 2% of Italian (see [Section 4.2.2][paper]).
-  - **Speech translation dataset:** This dataset used for speech translation training and reinforcement contains around 40k hours of real speech data for each source language with synthetic sentence-level aligned speech in English (see [Sections 4.2.3 and 4.2.4][paper]).
-  - **Speech translation fine-tuning dataset:** This dataset is a small 200h resynthesized subset of the *speech translation dataset* with natural pauses to improve audio quality and speech naturalness (see [Section 4.2.5][paper]).
-
-### Training procedure and hyper-parameters
-
-The different training stages along with the hyper-parameters are detailled in the [paper][paper]. 
-
-### Compute Infrastructure
-
-The final model was trained on 48 H100 Nvidia GPUs. 
-
----
-
-## Citation 
-
-If you use this model, please cite:
-
-```bibtex
-@unpublished{hibikizero2026,
-  title={Simultaneous Speech-to-Speech Translation Without Aligned Data},
-  author={Tom Labiausse and Romain Fabre and Yannick Estève and Alexandre Défossez and Neil Zeghidour},
-  note={Preprint},
-  year={2026},
-  url={https://arxiv.org/abs/2602.11072v1}
-}
-```
-
-[paper]: https://arxiv.org/abs/2602.11072v1
+---
+license: mit
+---

config.json

@@ -1,106 +1,28 @@
 {
-  "card": 2048,
-  "n_q": 32,
-  "dep_q": 16,
-  "delays": [
-    0,
-    0,
-    2,
-    2,
-    2,
-    2,
-    2,
-    2,
-    2,
-    2,
-    2,
-    2,
-    2,
-    2,
-    2,
-    2,
-    2,
-    0,
-    2,
-    2,
-    2,
-    2,
-    2,
-    2,
-    2,
-    2,
-    2,
-    2,
-    2,
-    2,
-    2,
-    2,
-    2
+  "_name_or_path": "opt-350m",
+  "activation_dropout": 0.0,
+  "activation_function": "relu",
+  "architectures": [
+    "OPTForCausalLM"
   ],
-  "dim": 2048,
-  "text_card": 48000,
-  "existing_text_padding_id": 3,
-  "num_heads": 16,
-  "num_layers": 28,
-  "hidden_scale": 6,
-  "causal": true,
-  "layer_scale": null,
-  "context": 3000,
-  "max_period": 20000.0,
-  "gating": "silu",
-  "norm": "rms_norm_f32",
-  "positional_embedding": "rope_concat",
-  "depformer_dim": 1024,
-  "depformer_num_heads": 16,
-  "depformer_num_layers": 6,
-  "depformer_dim_feedforward": null,
-  "depformer_multi_linear": true,
-  "depformer_norm": "layer_norm",
-  "depformer_pos_emb": "none",
-  "depformer_weights_per_step": true,
-  "demux_second_stream": false,
-  "kv_repeat": 2,
-  "depformer_kv_repeat": 1,
-  "text_card_out": null,
-  "conditioners": {},
-  "fuser": {
-    "cross_attention_pos_emb": false,
-    "cross_attention_pos_emb_scale": 1,
-    "sum": [],
-    "prepend": [],
-    "cross": []
-  },
-  "cross_attention": false,
-  "model_id": {
-    "sig": "77f82164",
-    "epoch": 110
-  },
-  "depformer_weights_per_step_schedule": [
-    0,
-    1,
-    2,
-    3,
-    4,
-    5,
-    6,
-    7,
-    8,
-    8,
-    8,
-    8,
-    8,
-    8,
-    8,
-    8
-  ],
-  "model_type": "hibiki",
-  "lm_gen_config": {
-    "temp": 0.8,
-    "temp_text": 0.8,
-    "top_k": 250,
-    "top_k_text": 250
-  },
-  "mimi_name": "mimi-pytorch-e351c8d8@125.safetensors",
-  "tokenizer_name": "tokenizer_spm_48k_multi6_2.model",
-  "moshi_name": "hibiki-pytorch-77f82164@110.safetensors"
-}
+  "attention_dropout": 0.0,
+  "bos_token_id": 2,
+  "do_layer_norm_before": false,
+  "dropout": 0.1,
+  "eos_token_id": 2,
+  "ffn_dim": 4096,
+  "hidden_size": 1024,
+  "init_std": 0.02,
+  "layerdrop": 0.0,
+  "max_position_embeddings": 2048,
+  "model_type": "opt",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 24,
+  "pad_token_id": 1,
+  "prefix": "</s>",
+  "torch_dtype": "float16",
+  "transformers_version": "4.20.0.dev0",
+  "use_cache": true,
+  "vocab_size": 50272,
+  "word_embed_proj_dim": 512
+}

generation_config.json

@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 2,
+  "eos_token_id": 2,
+  "pad_token_id": 1,
+  "transformers_version": "4.27.0.dev0"
+}

hibiki-pytorch-77f82164@110.safetensors

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:cd78e453b3b80299255bea02be439bcc2552b57c03cd82dbf0e9792e20100db8
-size 6263420344

mimi-pytorch-e351c8d8@125.safetensors → model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:09b782f0629851a271227fb9d36db65c041790365f11bbe5d3d59369cf863f50
-size 384644900
+oid sha256:9da6494f3af047e5e96ad93912f347aafed1bf6be00e03751b2db0d6e927eca1
+size 662435448

tokenizer_spm_48k_multi6_2.model → pytorch_model.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c22110fb855aa049e17346ea2e88355bdd664f06cbfd09948380ab5e85b39697
-size 857314
+oid sha256:a5223ae6f3c26c6d90003f96a6bcd9a4aaaef0d36fca6469112efeeb985f2842
+size 662513657

special_tokens_map.json

@@ -0,0 +1 @@
+{"bos_token": {"content": "</s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "eos_token": {"content": "</s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "unk_token": {"content": "</s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "pad_token": {"content": "<pad>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}}

tokenizer_config.json

@@ -0,0 +1 @@
+{"errors": "replace", "unk_token": {"content": "</s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "bos_token": {"content": "</s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "eos_token": {"content": "</s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "pad_token": {"content": "<pad>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "add_prefix_space": false, "add_bos_token": true, "special_tokens_map_file": null, "name_or_path": "patrickvonplaten/opt-30b"}