Update README.md

README.md

@@ -1,3 +1,114 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+datasets:
+- HuggingFaceFW/fineweb
+- HuggingFaceFW/fineweb-2
+- amphion/Emilia-Dataset
+- facebook/voxpopuli
+- uhhlt/Tuda-De
+- openslr/librispeech_asr
+- facebook/multilingual_librispeech
+- Thorsten-Voice/TV-44kHz-Full
+- CSTR-Edinburgh/vctk
+- commonvoice_23
+- kerstin
+language:
+- de
+- en
+base_model:
+- utter-project/EuroLLM-1.7B
+pipeline_tag: text-to-speech
+---
+
+
+<img src="https://educaai.de/webapp/splash/img/dark-1x.png" style="float:left">
+
+## educa AI voice (preview)
+
+---
+
+educa AI voice is our in-house text to speech model developed on top of [EuroLLM-1.7B](https://huggingface.co/utter-project/EuroLLM-1.7B).
+
+This version of the model is trained on a single speaker and is capable of generating natural-sounding German (and to some extent also English) speech.
+
+Be advised that this is a preview model meant to showcase the base model's capability. We are going to publish more advanced models in the near future (see bottom of this model card).
+
+#### Examples:
+
+<audio controls src="https://huggingface.co/DigitalLearningGmbH/educa-ai-voice-preview/resolve/main/example_1.mp3"></audio>
+<audio controls src="https://huggingface.co/DigitalLearningGmbH/educa-ai-voice-preview/resolve/main/example_2.mp3"></audio>
+<audio controls src="https://huggingface.co/DigitalLearningGmbH/educa-ai-voice-preview/resolve/main/example_3.mp3"></audio>
+
+
+### Model details
+
+- **Base LLM**: [EuroLLM-1.7B](https://huggingface.co/utter-project/EuroLLM-1.7B)
+- **Audio Tokenizer**: [NeuCodec](https://huggingface.co/neuphonic/neucodec)
+
+#### Pre-training
+
+We pre-trained the model in two stages, first training on billions of tokens of mixed audio and text data using a next-token-prediction objective.
+Then, we trained on tens of thousands of hours of German and English speech mixed with a little text instruction data to preserve the text understanding capability of the model.
+
+We used the following datasets, as well as some in-house datasets:
+- [HuggingFaceFW/fineweb](https://huggingface.co/datasets/HuggingFaceFW/fineweb)
+- [HuggingFaceFW/fineweb-2](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2)
+- [amphion/Emilia-Dataset](https://huggingface.co/datasets/amphion/Emilia-Dataset) (German and English YODAS subsets)
+- [facebook/voxpopuli](https://huggingface.co/datasets/facebook/voxpopuli)
+- [uhhlt/Tuda-De](https://huggingface.co/datasets/uhhlt/Tuda-De)
+- [openslr/librispeech_asr](https://huggingface.co/datasets/openslr/librispeech_asr)
+- [facebook/multilingual_librispeech](https://huggingface.co/datasets/facebook/multilingual_librispeech)
+- [Thorsten-Voice/TV-44kHz-Full](https://huggingface.co/datasets/Thorsten-Voice/TV-44kHz-Full)
+- [CSTR-Edinburgh/vctk](https://huggingface.co/datasets/CSTR-Edinburgh/vctk)
+- [commonvoice_23](https://datacollective.mozillafoundation.org/datasets?q=common+voice)
+- [kerstin](https://datacollective.mozillafoundation.org/datasets/cmi7mgbam000bnx074097g2yg)
+
+
+
+### Inference example
+
+```python
+import torch
+import torchaudio
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from neucodec import NeuCodec
+
+device = "cuda"
+model_id = "DigitalLearningGmbH/educa-ai-voice-preview"
+audio_end_token_id = 128001
+audio_tokens_offset = 128006
+
+model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16)
+model = model.to(device)
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+
+codec_model = NeuCodec.from_pretrained("neuphonic/neucodec")
+codec_model = codec_model.eval().to(device)
+
+prompt_template = "<|task_tts|>{prompt} <|audio_start|>"
+prompt = "Brautkleid bleibt Brautkleid und Blaukraut bleibt Blaukraut."
+
+input_ids = tokenizer.encode(prompt_template.format(prompt=prompt), return_tensors="pt").to(device)
+
+outputs = model.generate(input_ids=input_ids, do_sample=True, temperature=0.6, top_p=0.999, repetition_penalty=1.1, max_new_tokens=2048)
+outputs_audio = outputs[0][input_ids.shape[1]:(outputs[0] == audio_end_token_id).nonzero(as_tuple=True)[0][0].item()] - audio_tokens_offset
+
+with torch.no_grad():
+    recon = codec_model.decode_code(outputs_audio.unsqueeze(0).unsqueeze(0).to(device)).cpu()
+
+torchaudio.save("tts.wav", recon[0, :, :], 24_000)
+```
+
+### What's to come
+
+As stated in the model's name, this is a preview model, mainly meant to showcase the capability of the base model.
+We trained on a small dataset of a single speaker without any special emotion tagging etc.
+
+We are actively working on
+ - multiple speakers with emotional control and nonverbal elements (fillers, laughing, ...)
+ - fine-tuning for general zero-shot voice cloning
+ - post-training with reinforcement learning
+
+Also, we have a fine-tuned version of NeuCodec which we used to generate the speech examples above, which we also plan on realeasing.
+
+Stay tuned - january 2026 is going to be exciting!