Upload togethercomputer/RedPajama-INCITE-Instruct-3B-v1 ctranslate fp16 weights

README.md

@@ -1,7 +1,9 @@
 ---
 tags:
-- ctransate2
+- ctranslate2
 - int8
+- float16
+
 license: apache-2.0
 language:
 - en
@@ -32,18 +34,18 @@ inference:
     max_new_tokens: 128
 ---
 # # Fast-Inference with Ctranslate2
-Speedup inference by 2x-8x using int8 inference in C++
+Speedup inference while reducing memory by 2x-4x using int8 inference in C++ on CPU or GPU.
 
 quantized version of [togethercomputer/RedPajama-INCITE-Instruct-3B-v1](https://huggingface.co/togethercomputer/RedPajama-INCITE-Instruct-3B-v1)
 ```bash
-pip install hf-hub-ctranslate2>=2.0.6 ctranslate2>=3.13.0
+pip install hf-hub-ctranslate2>=2.0.6 
 ```
 Converted on 2023-05-19 using
 ```
 ct2-transformers-converter --model togethercomputer/RedPajama-INCITE-Instruct-3B-v1 --output_dir /home/michael/tmp-ct2fast-RedPajama-INCITE-Instruct-3B-v1 --force --copy_files tokenizer.json README.md tokenizer_config.json generation_config.json special_tokens_map.json .gitattributes --quantization float16
 ```
 
-Checkpoint compatible to [ctranslate2](https://github.com/OpenNMT/CTranslate2) and [hf-hub-ctranslate2](https://github.com/michaelfeil/hf-hub-ctranslate2)
+Checkpoint compatible to [ctranslate2>=3.13.0](https://github.com/OpenNMT/CTranslate2) and [hf-hub-ctranslate2>=2.0.6](https://github.com/michaelfeil/hf-hub-ctranslate2)
 - `compute_type=int8_float16` for `device="cuda"` 
 - `compute_type=int8`  for `device="cpu"`
 
@@ -61,7 +63,7 @@ model = GeneratorCT2fromHfHub(
         tokenizer=AutoTokenizer.from_pretrained("togethercomputer/RedPajama-INCITE-Instruct-3B-v1")
 )
 outputs = model.generate(
-    text=["How do you call a fast Flan-ingo?", "User: How are you doing?"],
+    text=["How do you call a fast Flan-ingo?", "User: How are you doing? Bot:"],
 )
 print(outputs)
 ```
@@ -71,6 +73,11 @@ This is just a quantized version. Licence conditions are intended to be idential
 
 # Original description
     
+tags:
+- ctranslate2
+- int8
+- float16
+
 
 # RedPajama-INCITE-Instruct-3B-v1