Update README.md

README.md

@@ -3,6 +3,7 @@ library_name: transformers
 license: apache-2.0
 datasets:
 - HuggingFaceTB/smoltalk
+- HuggingFaceH4/ultrafeedback_binarized
 base_model:
 - JingzeShi/Doge-20M
 language:
@@ -13,9 +14,11 @@ pipeline_tag: question-answering
 
 # **Doge 20M Instruct**
 
+![architecture](https://cdn-uploads.huggingface.co/production/uploads/673ab3647afcea17eb4378fd/BBPbIRiw1KxdgYp_5CABf.png)
+
 Doge is an ongoing research project where we aim to train a series of small language models to further explore whether the Transformer framework allows for more complex feedforward network structures, enabling the model to have fewer cache states and larger knowledge capacity.
 
-In addition, Doge uses Dynamic Mask Attention as sequence transformation and can use Multi-Layer Perceptron or Cross Domain Mixture of Experts as state transformation. Dynamic Mask Attention allows the Transformer to use self-attention during training and state space during inference, and Cross Domain Mixture of Experts can directly inherit the weights of Multi-Layer Perceptron for further training. This model is trained by Jingze Shi, it only allows text input and text generation, for detailed algorithm and model architecture, please refer to [Wonderful Matrices](https://arxiv.org/abs/2412.11834), the ongoing research repository is [Wonderful Matrices](https://github.com/LoserCheems/WonderfulMatrices).
+In addition, Doge uses Dynamic Mask Attention as sequence transformation and can use Multi-Layer Perceptron or Cross Domain Mixture of Experts as state transformation. Dynamic Mask Attention allows the Transformer to use self-attention during training and state space during inference, and Cross Domain Mixture of Experts can directly inherit the weights of Multi-Layer Perceptron for further training. This model is trained by Jingze Shi, it only allows text input and text generation, for detailed algorithm and model architecture, please refer to [Wonderful Matrices](https://arxiv.org/abs/2412.11834), all training details are in [here](https://github.com/LoserCheems/WonderfulMatrices).
 
 
 ## Uses
@@ -31,12 +34,15 @@ generation_config = GenerationConfig(
       use_cache=True, 
       do_sample=True, 
       temperature=0.8, 
+      top_p=0.9,
       repetition_penalty=1.0
 )
 steamer = TextStreamer(
       tokenizer=tokenizer, 
       skip_prompt=True
 )
+
+prompt = "Hi, how are you doing today?"
 conversation = [
       {"role": "user", "content": prompt}
 ]
@@ -57,16 +63,34 @@ outputs = model.generate(
 
 ## Model Details
 
+We build the Doge-Instruct by first SFT on [SmolTalk](https://huggingface.co/datasets/HuggingFaceTB/smoltalk) and then DPO on [UltraFeedback Binarized](https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized).
+
 > TODO: The larger model is under training and will be uploaded soon.
 
-**Training**:
+**SFT**:
+| Model | Training Data | Epochs | Content Length | LR | Batch Size | Precision |
+|---|---|---|---|---|---|---|
+| [Doge-20M-Instruct-SFT](https://huggingface.co/JingzeShi/Doge-20M-Instruct-SFT) | [HuggingFaceTB/smoltalk](https://huggingface.co/datasets/HuggingFaceTB/smoltalk) | 2 | 8192 | 8e-4 | 1M | bfloat16 |
+| [Doge-60M-Instruct](https://huggingface.co/JingzeShi/Doge-60M-Instruct) | [HuggingFaceTB/smoltalk](https://huggingface.co/datasets/HuggingFaceTB/smoltalk) | 2 | 8192 | 6e-4 | 1M | bfloat16 |
+
+**DPO**:
 | Model | Training Data | Epochs | Content Length | LR | Batch Size | Precision |
 |---|---|---|---|---|---|---|
-| [Doge-20M-Instruct](https://huggingface.co/JingzeShi/Doge-20M-Instruct) | [HuggingFaceTB/smoltalk](https://huggingface.co/datasets/HuggingFaceTB/smoltalk) | 2 | 8192 | 8e-5 | 1M | bfloat16 |
-| [Doge-60M-Instruct](https://huggingface.co/JingzeShi/Doge-60M-Instruct) | [HuggingFaceTB/smoltalk](https://huggingface.co/datasets/HuggingFaceTB/smoltalk) | 2 | 8192 | 6e-5 | 1M | bfloat16 |
+| [Doge-20M-Instruct](https://huggingface.co/JingzeShi/Doge-20M-Instruct) | [HuggingFaceH4/ultrafeedback_binarized](https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized) | 2 | 1024 | 8e-5 | 0.125M | bfloat16 |
+| [Doge-60M-Instruct](https://huggingface.co/JingzeShi/Doge-60M-Instruct) | [HuggingFaceH4/ultrafeedback_binarized](https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized) | 2 | 1024 | 6e-5 | 0.125M | bfloat16 |
+
+
+**Procedure**:
+
+**SFT**:
+[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="150" height="24"/>](https://wandb.ai/loser_cheems/huggingface/runs/xl21ytg8) 
+
+**DPO**:
+[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="150" height="24"/>](https://wandb.ai/loser_cheems/huggingface/runs/874wshj2)
+
 
 **Environment**:
-- Image: nvcr.io/nvidia/pytorch:24.10-py3
+- Image: nvcr.io/nvidia/pytorch:24.12-py3
 - Hardware: 1x NVIDIA RTX 4090
 - Software: Transformers, TRL