Update README.md

README.md

@@ -1,7 +1,129 @@
----
-license: mit
-tags:
-- unsloth
-- trl
-- sft
----
+---
+license: mit
+tags:
+- unsloth
+- trl
+- sft
+datasets:
+- olafdil/French_MultiSpeaker_Diarization
+language:
+- fr
+base_model:
+- unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit
+pipeline_tag: text-generation
+---
+# Fine-Tuned Model: Meta-Llama-3.1-8B-Instruct-bnb-4bit
+
+This is a fine-tuned version of the [Meta-Llama-3.1-8B-Instruct-bnb-4bit](https://huggingface.co/unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit) model, adapted for French multi-speaker diarization tasks. Below, you'll find details about the fine-tuning process, dataset, and how to use this model.
+
+---
+
+## Model Details
+
+- **Base Model**: Meta-Llama-3.1-8B-Instruct-bnb-4bit
+- **Quantization**: 4-bit quantization for reduced memory usage
+- **Purpose**: Fine-tuned for multi-speaker diarization in French.
+- **Techniques**:
+  - LoRA (Low-Rank Adaptation) for efficient fine-tuning.
+  - Target modules: `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`.
+  - Rank: `16`
+  - LoRA alpha: `16`
+  - Gradient checkpointing: Enabled.
+
+---
+
+## Dataset
+
+The model was fine-tuned on the `French_MultiSpeaker_Diarization` dataset, hosted on the Hugging Face Hub:
+
+- **Dataset Name**: [French_MultiSpeaker_Diarization](https://huggingface.co/datasets/olafdil/French_MultiSpeaker_Diarization)
+- **Split Used**: Train
+- **Dataset Content**:
+  - Multispeaker conversational data in French.
+  - Includes labeled diarization information to improve diarization capabilities.
+
+---
+
+## Training Configuration
+
+### Hyperparameters
+
+- **Max Sequence Length**: `120,000`
+- **LoRA Dropout**: `0`
+- **Bias**: `none`
+- **Use Gradient Checkpointing**: Enabled for efficiency.
+- **Custom Prompting**: Chat templates applied for formatting prompts (e.g., `llama-3.1` template).
+
+### Training Workflow
+
+1. **Model Loading**:
+   - Loaded the base model using `FastLanguageModel.from_pretrained()`.
+   - Applied 4-bit quantization for memory efficiency.
+
+2. **Dataset Preparation**:
+   - The dataset was tokenized using a custom chat template from the `unsloth.chat_templates` library.
+   - Prompts formatted with `apply_chat_template()` to suit the diarization task.
+
+3. **Fine-Tuning**:
+   - LoRA applied to specific layers for adaptation.
+   - Gradient checkpointing enabled to reduce memory overhead during training.
+
+---
+
+## Usage
+
+### Load the Model
+You can load this model directly from Hugging Face:
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_name = "olafdil/FrDiarization-Llama-3.1-8B-4bit"
+model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto")
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+```
+### Inference Example
+```python
+template = """
+I have an audio transcription where multiple speakers are involved in a conversation.
+Your task is to distinguish the different speakers and diarize the text accordingly.
+Each speaker's dialogue should be clearly labeled, such as 'Speaker 1:', 'Speaker 2:', etc.
+Ensure that the labels remain consistent throughout the transcription and that the text is formatted neatly.
+Here's the transcription:
+"""
+transciption = "Your input transcription here"
+prompt = template + transcription
+
+inputs = tokenizer(prompt, return_tensors="pt")
+outputs = model.generate(**inputs)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+
+## Dependencies
+
+The following libraries were used:
+
+- `transformers`
+- `datasets`
+- `unsloth`
+- `torch`
+
+To install the dependencies, you can use:
+
+```bash
+pip install transformers datasets torch unsloth
+```
+
+## Limitations
+
+- The model has been fine-tuned specifically for French multi-speaker diarization tasks and may not generalize well to other tasks or languages.
+- 4-bit quantization reduces memory usage but may slightly affect precision.
+
+---
+
+## Citation
+
+If you use this model, please consider citing the base model and the dataset:
+
+- **Base Model**: Meta-Llama-3.1-8B-Instruct-bnb-4bit
+- **Dataset**: French MultiSpeaker Diarization