Create README.md

README.md

@@ -0,0 +1,66 @@
+---
+license: apache-2.0
+datasets:
+- squad_v2
+language:
+- en
+library_name: transformers
+pipeline_tag: text-classification
+inference: false
+---
+# longformer-large-4096 fine-tuned to SQuAD2.0 for answerability score
+This model determines whether the question is answerable (or unanswerable) given the context. 
+The output is a probability where values close to 0.0 indicate that the question is unanswerable and values close to 1.0 means answerable.
+
+- Input: `question` and `context`
+- Output: `probability` (i.e. logit -> sigmoid)
+
+## Model Details
+
+longformer-large-4096 model is fine-tuned to the SQuAD2.0 dataset where the input is a concatenation of ```question + context```. 
+Due to class imbalance in SQuAD2.0, we resample such that the model is trained on a 50/50 split between answerable and unanswerable samples in SQuAD2.0.
+
+## How to Use the Model
+
+Use the code below to get started with the model. 
+
+```python
+>>> import torch
+>>> from transformers import LongformerTokenizer, LongformerForSequenceClassification
+
+>>> tokenizer = LongformerTokenizer.from_pretrained("potsawee/longformer-large-4096-answerable-squad2")
+>>> model = LongformerForSequenceClassification.from_pretrained("potsawee/longformer-large-4096-answerable-squad2")
+
+>>> context = """
+British government ministers have been banned from using Chinese-owned social media app TikTok on their work phones and devices on security grounds.
+The government fears sensitive data held on official phones could be accessed by the Chinese government.
+Cabinet Minister Oliver Dowden said the ban was a "precautionary" move but would come into effect immediately.
+""".replace("\n", " ").strip()
+
+>>> question1   = "Which application have been banned by the British government?"
+>>> input_text1 = question1 + ' ' + tokenizer.sep_token + ' ' + context
+>>> inputs1     = tokenizer(input_text1, max_length=4096, truncation=True, return_tensors="pt")
+>>> prob1 = torch.sigmoid(model(**inputs1).logits.squeeze(-1))
+>>> print("P(answerable|question1, context) = {:.2f}%".format(prob1.item()*100))
+P(answerable|question1, context) = 99.21% # highly answerable
+
+>>> question2   = "Is Facebook popular among young students in America?"
+>>> input_text2 = question2 + ' ' + tokenizer.sep_token + ' ' + context
+>>> inputs2     = tokenizer(input_text2, max_length=4096, truncation=True, return_tensors="pt")
+>>> prob2 = torch.sigmoid(model(**inputs2).logits.squeeze(-1))
+>>> print("P(answerable|question2, context) = {:.2f}%".format(prob2.item()*100))
+P(answerable|question2, context) = 2.53% # highly unanswerable
+```
+
+## Citation
+
+```bibtex
+@misc{manakul2023selfcheckgpt,
+      title={SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models}, 
+      author={Potsawee Manakul and Adian Liusie and Mark J. F. Gales},
+      year={2023},
+      eprint={2303.08896},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL}
+}
+```