Initial upload of CSM-1B

.gitattributes

@@ -0,0 +1,41 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+prompts/conversational_a.wav filter=lfs diff=lfs merge=lfs -text
+prompts/conversational_b.wav filter=lfs diff=lfs merge=lfs -text
+prompts/read_speech_a.wav filter=lfs diff=lfs merge=lfs -text
+prompts/read_speech_b.wav filter=lfs diff=lfs merge=lfs -text
+prompts/read_speech_c.wav filter=lfs diff=lfs merge=lfs -text
+prompts/read_speech_d.wav filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,120 @@
+---
+license: apache-2.0
+language:
+  - en
+pipeline_tag: text-to-speech
+tags:
+  - model_hub_mixin
+  - pytorch_model_hub_mixin
+  - text-to-speech
+---
+
+## CSM 1B
+
+**2025/03/13** - We are releasing the 1B CSM variant. Code is available on GitHub: [SesameAILabs/csm](https://github.com/SesameAILabs/csm).
+
+---
+
+CSM (Conversational Speech Model) is a speech generation model from [Sesame](sesame.com) that generates RVQ audio codes from text and audio inputs. The model architecture employs a [Llama](https://www.llama.com/) backbone and a smaller audio decoder that produces [Mimi](https://huggingface.co/kyutai/mimi) audio codes.
+
+A fine-tuned variant of CSM powers the [interactive voice demo](https://www.sesame.com/voicedemo) shown in our [blog post](https://www.sesame.com/research/crossing_the_uncanny_valley_of_voice).
+
+A hosted [HuggingFace space](https://huggingface.co/spaces/sesame/csm-1b) is also available for testing audio generation.
+
+## Usage
+
+Setup the repo
+
+```bash
+git clone git@github.com:SesameAILabs/csm.git
+cd csm
+python3.10 -m venv .venv
+source .venv/bin/activate
+pip install -r requirements.txt
+
+# You will need access to sesame/csm-1b and meta-llama/Llama-3.2-1B
+huggingface-cli login
+```
+
+Generate a sentence
+
+```python
+from generator import load_csm_1b
+import torchaudio
+
+generator = load_csm_1b(device="cuda")
+
+audio = generator.generate(
+    text="Hello from Sesame.",
+    speaker=0,
+    context=[],
+    max_audio_length_ms=10_000,
+)
+
+torchaudio.save("audio.wav", audio.unsqueeze(0).cpu(), generator.sample_rate)
+```
+
+CSM sounds best when provided with context. You can prompt or provide context to the model using a `Segment` for each speaker utterance.
+
+```python
+speakers = [0, 1, 0, 0]
+transcripts = [
+    "Hey how are you doing.",
+    "Pretty good, pretty good.",
+    "I'm great.",
+    "So happy to be speaking to you.",
+]
+audio_paths = [
+    "utterance_0.wav",
+    "utterance_1.wav",
+    "utterance_2.wav",
+    "utterance_3.wav",
+]
+
+def load_audio(audio_path):
+    audio_tensor, sample_rate = torchaudio.load(audio_path)
+    audio_tensor = torchaudio.functional.resample(
+        audio_tensor.squeeze(0), orig_freq=sample_rate, new_freq=generator.sample_rate
+    )
+    return audio_tensor
+
+segments = [
+    Segment(text=transcript, speaker=speaker, audio=load_audio(audio_path))
+    for transcript, speaker, audio_path in zip(transcripts, speakers, audio_paths)
+]
+audio = generator.generate(
+    text="Me too, this is some cool stuff huh?",
+    speaker=1,
+    context=segments,
+    max_audio_length_ms=10_000,
+)
+
+torchaudio.save("audio.wav", audio.unsqueeze(0).cpu(), generator.sample_rate)
+```
+
+## FAQ
+
+**Does this model come with any voices?**
+
+The model open sourced here is a base generation model. It is capable of producing a variety of voices, but it has not been fine-tuned on any specific voice.
+
+**Can I converse with the model?**
+
+CSM is trained to be an audio generation model and not a general purpose multimodal LLM. It cannot generate text. We suggest using a separate LLM for text generation.
+
+**Does it support other languages?**
+
+The model has some capacity for non-English languages due to data contamination in the training data, but it likely won't do well.
+
+## Misuse and abuse ⚠️
+
+This project provides a high-quality speech generation model for research and educational purposes. While we encourage responsible and ethical use, we **explicitly prohibit** the following:
+
+- **Impersonation or Fraud**: Do not use this model to generate speech that mimics real individuals without their explicit consent.
+- **Misinformation or Deception**: Do not use this model to create deceptive or misleading content, such as fake news or fraudulent calls.
+- **Illegal or Harmful Activities**: Do not use this model for any illegal, harmful, or malicious purposes.
+
+By using this model, you agree to comply with all applicable laws and ethical guidelines. We are **not responsible** for any misuse, and we strongly condemn unethical applications of this technology.
+
+**Authors**
+Johan Schalkwyk, Ankit Kumar, Dan Lyth, Sefik Emre Eskimez, Zack Hodari, Cinjon Resnick, Ramon Sanabria, Raven Jiang, and the Sesame team.

config.json

@@ -0,0 +1,125 @@
+{
+    "architectures": [
+      "CsmForConditionalGeneration"
+    ],
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "audio_eos_token_id": 128003,
+    "audio_token_id": 128002,
+    "bos_token_id": 128000,
+    "codebook_eos_token_id": 0,
+    "codebook_pad_token_id": 2050,
+    "codec_config": {
+      "_name_or_path": "kyutai/mimi",
+      "architectures": [
+        "MimiModel"
+      ],
+      "attention_bias": false,
+      "attention_dropout": 0.0,
+      "audio_channels": 1,
+      "codebook_dim": 256,
+      "codebook_size": 2048,
+      "compress": 2,
+      "dilation_growth_rate": 2,
+      "frame_rate": 12.5,
+      "head_dim": 64,
+      "hidden_act": "gelu",
+      "hidden_size": 512,
+      "initializer_range": 0.02,
+      "intermediate_size": 2048,
+      "kernel_size": 7,
+      "last_kernel_size": 3,
+      "layer_scale_initial_scale": 0.01,
+      "max_position_embeddings": 8000,
+      "model_type": "mimi",
+      "norm_eps": 1e-05,
+      "normalize": false,
+      "num_attention_heads": 8,
+      "num_filters": 64,
+      "num_hidden_layers": 8,
+      "num_key_value_heads": 8,
+      "num_quantizers": 32,
+      "num_residual_layers": 1,
+      "num_semantic_quantizers": 1,
+      "pad_mode": "constant",
+      "residual_kernel_size": 3,
+      "rope_theta": 10000.0,
+      "sampling_rate": 24000,
+      "sliding_window": 250,
+      "torch_dtype": "float32",
+      "trim_right_ratio": 1.0,
+      "upsample_groups": 512,
+      "upsampling_ratios": [
+        8,
+        6,
+        5,
+        4
+      ],
+      "use_cache": false,
+      "use_causal_conv": true,
+      "use_conv_shortcut": false,
+      "vector_quantization_hidden_dimension": 256
+    },
+    "depth_decoder_config": {
+      "attention_bias": false,
+      "attention_dropout": 0.0,
+      "backbone_hidden_size": 2048,
+      "head_dim": 128,
+      "hidden_act": "silu",
+      "hidden_size": 1024,
+      "initializer_range": 0.02,
+      "intermediate_size": 8192,
+      "max_position_embeddings": 33,
+      "mlp_bias": false,
+      "model_type": "csm_depth_decoder_model",
+      "num_attention_heads": 8,
+      "num_codebooks": 32,
+      "num_hidden_layers": 4,
+      "num_key_value_heads": 2,
+      "rms_norm_eps": 1e-05,
+      "rope_scaling": {
+        "factor": 32.0,
+        "high_freq_factor": 0.0078125,
+        "low_freq_factor": 0.001953125,
+        "original_max_position_embeddings": 16,
+        "rope_type": "llama3"
+      },
+      "rope_theta": 500000,
+      "use_cache": true,
+      "vocab_size": 2051
+    },
+    "head_dim": 64,
+    "hidden_act": "silu",
+    "hidden_size": 2048,
+    "initializer_range": 0.02,
+    "intermediate_size": 8192,
+    "max_position_embeddings": 2048,
+    "mlp_bias": false,
+    "model_type": "csm",
+    "num_attention_heads": 32,
+    "num_codebooks": 32,
+    "num_hidden_layers": 16,
+    "num_key_value_heads": 8,
+    "pad_token_id": 128002,
+    "rms_norm_eps": 1e-05,
+    "rope_scaling": {
+      "factor": 32.0,
+      "high_freq_factor": 0.5,
+      "low_freq_factor": 0.125,
+      "original_max_position_embeddings": 1024,
+      "rope_type": "llama3"
+    },
+    "rope_theta": 500000,
+    "text_vocab_size": 128256,
+    "tie_codebooks_embeddings": true,
+    "tie_word_embeddings": false,
+    "torch_dtype": "float32",
+    "transformers_version": "4.52.0.dev0",
+    "use_cache": true,
+    "vocab_size": 2051,
+    "audio_num_codebooks": 32,
+    "audio_vocab_size": 2051,
+    "backbone_flavor": "llama-1B",
+    "decoder_flavor": "llama-100M",
+    "transformers_weights": "transformers.safetensors.index.json"
+  }

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2e7721144afe38b906d4f1048671da639fe142423f4a26283606ecebe894f4bf
+size 6211186784

prompts/conversational_a.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:356648c1bc6c1da7883004557e9b21a2ef7d01682d8b9d02d6dcb950b348b04f
+size 2646044

prompts/conversational_b.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c247153011385d33aaeed193adfec562c32182e2facd30cc8cd0b3e820e94afb
+size 2646044

prompts/read_speech_a.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:59480708f84c77ab2967d14d821c2ccade9d7761685d060575121f49a149005b
+size 831412

prompts/read_speech_b.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f582640265864499cbe6a8c687ea0f9e08e7fa41eeb2caa923d0a3bada55fcef
+size 576052

prompts/read_speech_c.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7da15ab3ee7f8bbc8abfce73ce65936a80a535ae4a86db2d9c4756caba69e9c3
+size 385964

prompts/read_speech_d.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:09cad0494f9d0038b0f0eb039f47d752c45e56d92679f96587e20f67b2c1b7d8
+size 435884