Update README.md

README.md

@@ -16,27 +16,29 @@ The model is doing predictions on the *sentence-level*. It takes as input a docu
 whether the sentence is supported by the document: **MiniCheck-Model(document, claim) -> {0, 1}**
 
 
-MiniCheck-Flan-T5-Large is fine tuned from `google/flan-t5-large` ([Chung et al., 2022](https://arxiv.org/pdf/2210.11416.pdf)) 
+**MiniCheck-Flan-T5-Large is the best fack-checking model with size < 1B** and reaches GPT-4 performance. It is fine tuned from `google/flan-t5-large` ([Chung et al., 2022](https://arxiv.org/pdf/2210.11416.pdf)) 
 on the combination of 35K data:
 - 21K ANLI data ([Nie et al., 2020](https://aclanthology.org/2020.acl-main.441.pdf))
 - 14K synthetic data generated from scratch in a structed way (more details in the paper).
 
 
 ### Model Variants
-We also have other two MiniCheck model variants:
-- [lytang/MiniCheck-RoBERTa-Large](https://huggingface.co/lytang/MiniCheck-RoBERTa-Large)
-- [lytang/MiniCheck-DeBERTa-v3-Large](https://huggingface.co/lytang/MiniCheck-DeBERTa-v3-Large)
+We also have other three MiniCheck model variants:
+- [bespokelabs/Bespoke-Minicheck-7B](https://huggingface.co/bespokelabs/Bespoke-MiniCheck-7B) (Model Size: 7B)
+- [lytang/MiniCheck-RoBERTa-Large](https://huggingface.co/lytang/MiniCheck-RoBERTa-Large) (Model Size: 0.4B)
+- [lytang/MiniCheck-DeBERTa-v3-Large](https://huggingface.co/lytang/MiniCheck-DeBERTa-v3-Large) (Model Size: 0.4B)
 
 
 ### Model Performance
 
 <p align="center">
-    <img src="./cost-vs-bacc.png" width="360">
+    <img src="./performance.png" width="550">
 </p>
 
 
+
 The performance of these models is evaluated on our new collected benchmark (unseen by our models during training), [LLM-AggreFact](https://huggingface.co/datasets/lytang/LLM-AggreFact), 
-from 10 recent human annotated datasets on fact-checking and grounding LLM generations. Our most capable model MiniCheck-Flan-T5-Large outperform all
+from 11 recent human annotated datasets on fact-checking and grounding LLM generations. MiniCheck-Flan-T5-Large outperform all
 exisiting specialized fact-checkers with a similar scale by a large margin (4-10% absolute increase) and is on par with GPT-4, but 400x cheaper. See full results in our work.
 
 Note: We only evaluated the performance of our models on real claims -- without any human intervention in 
@@ -53,13 +55,15 @@ Please first clone our [GitHub Repo](https://github.com/Liyan06/MiniCheck) and i
 
 ```python
 from minicheck.minicheck import MiniCheck
+import os
+os.environ["CUDA_VISIBLE_DEVICES"] = "0"
 
 doc = "A group of students gather in the school library to study for their upcoming final exams."
 claim_1 = "The students are preparing for an examination."
 claim_2 = "The students are on vacation."
 
-# model_name can be one of ['roberta-large', 'deberta-v3-large', 'flan-t5-large']
-scorer = MiniCheck(model_name='flan-t5-large', device=f'cuda:0', cache_dir='./ckpts')
+# model_name can be one of ['roberta-large', 'deberta-v3-large', 'flan-t5-large', 'Bespoke-MiniCheck-7B']
+scorer = MiniCheck(model_name='flan-t5-large', cache_dir='./ckpts')
 pred_label, raw_prob, _, _ = scorer.score(docs=[doc, doc], claims=[claim_1, claim_2])
 
 print(pred_label) # [1, 0]
@@ -72,14 +76,16 @@ print(raw_prob)   # [0.9805923700332642, 0.007121307775378227]
 import pandas as pd
 from datasets import load_dataset
 from minicheck.minicheck import MiniCheck
+import os
+os.environ["CUDA_VISIBLE_DEVICES"] = "0"
 
-# load 13K test data
+# load 29K test data
 df = pd.DataFrame(load_dataset("lytang/LLM-AggreFact")['test'])
 docs = df.doc.values
 claims = df.claim.values
 
-scorer = MiniCheck(model_name='flan-t5-large', device=f'cuda:0', cache_dir='./ckpts')
-pred_label, raw_prob, _, _ = scorer.score(docs=docs, claims=claims)  # ~ 20 mins, depending on hardware
+scorer = MiniCheck(model_name='flan-t5-large', cache_dir='./ckpts')
+pred_label, raw_prob, _, _ = scorer.score(docs=docs, claims=claims)  # ~ 500 docs/min, depending on hardware
 ```
 
 To evalaute the result on the benchmark