Update README.md

README.md

@@ -9,7 +9,7 @@ pipeline_tag: image-text-to-text
 
 # Model Card for Model ID
 
-
+![TaivisionLM]()
 ## Model Details
 
 ## English
@@ -22,7 +22,7 @@ pipeline_tag: image-text-to-text
 Ready to experience the Traditional Chinese visual language model? Let's go! 🖼️🤖
 
 
-## Traditional Chinese
+## 繁體中文
 # 臺視: 首創獨一無二的視覺語言模型!! 🚀
 
 🌟 TaiVisionLM 是一個小型的視覺語言模型（僅有 12 億參數），可以根據圖像輸入來回覆繁體中文指令！🌟
@@ -57,7 +57,7 @@ Here's the summary of the development process:
 - **Model type:** [Image-Text-to-Text](https://huggingface.co/tasks/image-text-to-text)
 - **Language(s) (NLP):** *Traditional Chinese*
 
-## 中文
+## 繁體中文
 這個模型是一個多模態的語言模型，結合了 [SigLIP](https://huggingface.co/docs/transformers/en/model_doc/siglip) 作為其視覺編碼器，並使用 [Tinyllama](https://huggingface.co/benchang1110/Taiwan-tinyllama-v1.0-chat) 作為語言模型。視覺投影器將這兩種模態結合在一起。  
 其架構與 [PaliGemma](https://huggingface.co/docs/transformers/v4.44.0/model_doc/paligemma) 非常相似。
 
@@ -79,6 +79,8 @@ Here's the summary of the development process:
 
 ## How to Get Started with the Model
 
+## English
+
 In Transformers, you can load the model and do inference as follows:
 
 **IMPORTANT NOTE:** TaiVisionLM model is not yet integrated natively into the Transformers library. So you need to set ```trust_remote_code=True``` when loading the model. It will download the ```configuration_taivisionlm.py```, ```modeling_taivisionlm.py``` and ```processing_taivisionlm.py``` files from the repo. You can check out the content of these files under the *Files and Versions* tab and pin the specific versions if you have any concerns regarding malicious code.
@@ -101,15 +103,42 @@ outputs = processor.tokenizer.decode(model.generate(**inputs,max_length=512)[0])
 print(outputs)
 ```
 
+## 中文
+利用 transformers，可以用下面程式碼進行推論:
+
+**重要通知:** 台視 (TaiVisionLM)還沒被整合進transformers，因此在下載模型時要使用 ```trust_remote_code=True```，下載模型將會使用``configuration_taivisionlm.py```、 ```modeling_taivisionlm.py``` 和 ```processing_taivisionlm.py``` 這三個檔案，若擔心有惡意程式碼，請先點選右方 *Files and Versions* 來查看程式碼內容。
+
+```python
+from transformers import AutoProcessor, AutoModelForCausalLM, AutoConfig
+from PIL import Image
+import requests
+import torch
+
+config = AutoConfig.from_pretrained("benchang1110/TaiVision-base",trust_remote_code=True)
+processor = AutoProcessor.from_pretrained("benchang1110/TaiVision-base",trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained("benchang1110/TaiVision-base",trust_remote_code=True,torch_dtype=torch.float16,attn_implementation="sdpa").to('cuda')
+model.eval()
+url = "https://media.wired.com/photos/598e35fb99d76447c4eb1f28/master/pass/phonepicutres-TA.jpg"
+image = Image.open(requests.get(url, stream=True).raw).convert("RGB")
+text = "描述圖片"
+inputs = processor(text=text,images=image, return_tensors="pt",padding=False).to('cuda')
+outputs = processor.tokenizer.decode(model.generate(**inputs,max_length=512)[0])
+print(outputs)
+```
+
 ### Training Procedure
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
 The following training hyperparameters are used in feature alignment and task specific training stages respectively:
 
 - **Feature Alignment**
 
 | Data size    | Global Batch Size | Learning Rate | Epochs | Max Length | Weight Decay |
 |--------------|-------------------|---------------|--------|------------|--------------|
-| 1B        | 16               | 5e-4          | 1      | 2048       | 1e-5            |
+| 1B        | 16               | 5e-5          | 1      | 2048       | 1e-5            |
 
+We use full-parameter finetuning for the projector and apply LoRA to the language model.
 
-We use full-parameter finetuning for the projector and apply LoRA to the language model.
+### Compute Infrastructure
+- **Feature Alignment**
+- 1xV100(32GB), took approximately 16 GPU hours.
+