update readme

README.md

@@ -1,3 +1,121 @@
 ---
 license: cc-by-nc-4.0
 ---
+
+# SONAR
+[[Paper]]()
+[[Demo]](#usage)
+
+We introduce SONAR, a new multilingual and multimodal fixed-size sentence embedding space. Our single **text encoder, covering 200 languages**, substantially outperforms existing sentence embeddings such as LASER3 and LabSE on the xsim and xsim++ multilingual similarity search tasks. 
+
+Speech segments can be embedded in the same \sonar embedding space using language-specific speech encoders trained in a teacher-student setting on speech transcription data. Our encoders outperform existing speech encoders on similarity search tasks.
+We also provide a **text decoder for 200 languages**, which allows us to perform text-to-text and speech-to-text machine translation, including for zero-shot language and modality combinations. 
+
+Our text-to-text results are competitive compared to the state-of-the-art NLLB~1B model, despite the fixed-size bottleneck representation. Our zero-shot speech-to-text translation results compare favorably with strong supervised baselines such as Whisper.
+
+
+Model inference support thanks [Fairseq2](https://github.com/facebookresearch/fairseq2)
+
+
+## Installing
+
+See our github [repo](https://reimagined-broccoli-941276ee.pages.github.io/nightly/installation/from_source_conda)
+
+## Usage
+Compute text sentence embeddings:
+```python
+from sonar.inference_pipelines.text import TextToEmbeddingModelPipeline
+t2vec_model = TextToEmbeddingModelPipeline(encoder="text_sonar_basic_encoder",
+                                           tokenizer="text_sonar_basic_encoder")
+sentences = ['My name is SONAR.', 'I can embed the sentences into vectorial space.']
+t2vec_model.predict(sentences, source_lang="eng_Latn").shape
+# torch.Size([2, 1024])
+```
+
+Translate with SONAR
+```python
+from sonar.inference_pipelines.text import TextToTextModelPipeline
+t2t_model = TextToTextModelPipeline(encoder="text_sonar_basic_encoder",
+                                    decoder="text_sonar_basic_decoder",
+                                    tokenizer="text_sonar_basic_encoder")  # tokenizer is attached to both encoder and decoder cards
+
+sentences = ['My name is SONAR.', 'I can embed the sentences into vectorial space.']
+t2t_model.predict(sentences, source_lang="eng_Latn", target_lang="fra_Latn")
+# ['Mon nom est SONAR.', "Je peux intégrer les phrases dans l'espace vectoriel."]
+```
+
+Compute speech sentence embeddings:
+```python
+import torch
+from sonar.inference_pipelines.speech import SpeechToEmbeddingPipeline, SpeechInferenceParams
+
+speech_embedding_dp_builder = SpeechToEmbeddingPipeline.load_from_name("sonar_speech_encoder_eng")
+
+speech_ctx = SpeechInferenceParams(
+    data_file="..../test_fleurs_fra-eng.tsv",
+    audio_root_dir=".../audio_zips",
+    audio_path_index=2,
+    batch_size=4,
+)
+
+speech_embedding_dp = speech_embedding_dp_builder.build_pipeline(speech_ctx)
+with torch.inference_mode():
+    speech_emb = next(iter(speech_embedding_dp))
+speech_emb["audio"]["data"].sentence_embeddings
+```
+
+
+Speech-to-text with SONAR
+```python
+import torch
+from sonar.inference_pipelines import SpeechToTextPipeline, SpeechInferenceParams
+
+speech_to_text_dp_builder = SpeechToTextPipeline.load_from_name(encoder_name="sonar_speech_encoder_eng", 
+                                                                decoder_name="text_sonar_basic_decoder")
+
+speech_ctx = SpeechInferenceParams(
+    data_file=".../test_fleurs_fra-eng.tsv",
+    audio_root_dir=".../audio_zips",
+    audio_path_index=2,
+    target_lang='fra_Latn',
+    batch_size=4,
+)
+speech_to_text_dp = speech_to_text_dp_builder.build_pipeline(speech_ctx)
+with torch.inference_mode():
+    speech_text_translation = next(iter(speech_to_text_dp))
+speech_text_translation
+```
+
+Predicting [cross-lingual semantic similarity](https://github.com/facebookresearch/fairseq/tree/nllb/examples/nllb/human_XSTS_eval) 
+with BLASER-2 models
+```Python
+import torch
+from sonar.models.blaser.loader import load_blaser_model
+
+blaser_ref = load_blaser_model("blaser_st2st_ref_v2_0").eval()
+blaser_qe = load_blaser_model("blaser_st2st_qe_v2_0").eval()
+# BLASER-2 is supposed to work with SONAR speech and text embeddings,
+# but we didn't include their extraction in this snippet, to keep it simple.
+emb = torch.ones([1, 1024])
+print(blaser_ref(src=emb, ref=emb, mt=emb).item())  # 5.2552
+print(blaser_qe(src=emb, mt=emb).item())  # 4.9819
+```
+
+See more complete demo notebooks :
+* [sonar text2text similarity and translation](examples/sonar_text_demo.ipynb)
+* [sonar speech2text and other data pipeline examples](examples/inference_pipelines.ipynb)
+
+
+## Model details
+
+- **Developed by:** Paul-Ambroise Duquenne et al.
+- **License:** CC-BY-NC 4.0 license
+- **Cite as:**
+
+  @article{Duquenne:2023:sonar_arxiv,
+    author = {Paul-Ambroise Duquenne and Holger Schwenk and Benoit Sagot},
+    title = {{SONAR:} Sentence-Level Multimodal and Language-Agnostic Representations},
+    publisher = {arXiv},
+    year = {2023},
+    url = {https://arxiv.org/abs/unk},
+  }