Update README.md

README.md

@@ -1,3 +1,66 @@
----
-license: mit
----
+---
+license: mit
+language:
+- en
+base_model:
+- mistralai/Mistral-7B-Instruct-v0.3
+---
+# Model Card for `first_mistral`
+`first_mistral` is a language model trained to act as a listwise reranker, decoding from the first-token logits only to improve efficiency while maintaining effectiveness.
+`first_mistral` is built on Mistral-7B-Instruct-v0.3, following [FIRST](https://arxiv.org/abs/2406.15657)'s strategy, trained using 40K GPT-4 labeled rerank instances from RankZephyr. 
+More details can be found in the paper.
+
+## Model description
+
+- **Model type:** A 7B parameter listwise reranker fine-tuned from Mistral-7B-Instruct-v0.3
+- **Language(s) (NLP):** Primarily English
+- **License:** MIT
+- **Fine-tuned from model:** [mistralai/Mistral-7B-Instruct-v0.3](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3)
+
+### Model Sources
+
+- **Repository:** https://github.com/castorini/rank_llm
+- **Paper:** https://arxiv.org/abs/2411.05508
+
+## Evaluation
+
+At the time of release, `first_mistral` outperforms the original FIRST implementation on most subsets of the BEIR benchmark.
+More details that compare other LLM rerankers on more datasets can be found in the paper.
+
+| Dataset        | FIRST (original) | first_mistral |
+|----------------|-------------|--------------|
+| climate-fever  | **0.2672**      | 0.2417   |
+| dbpedia-entity | **0.5092**  | 0.5033       |
+| fever          | 0.8164      | **0.8413**   |
+| fiqa           | 0.4223      | **0.4778**   |
+| hotpotqa       | 0.7424      | **0.7705**   |
+| msmarco        | 0.4425      | **0.4512**   |
+| nfcorpus       | 0.3725  | **0.3816**       |
+| nq             | 0.6638  | **0.6985**       |
+| scidocs        | 0.2047  | **0.2110**       |
+| scifact        | 0.7459      | **0.7769**   |
+| trec-covid     | 0.7913  | **0.7666**       |
+| Average        | 0.5435      | **0.5564**   |
+
+## Citation
+If you find `first_mistral` useful for your work, please consider citing:
+
+```
+@ARTICLE{chen2024firstrepro,
+  title   = title={An Early FIRST Reproduction and Improvements to Single-Token Decoding for Fast Listwise Reranking},
+  author  = {Zijian Chen and Ronak Pradeep and Jimmy Lin},
+  year    = {2024},
+  journal = {arXiv:2411.05508}
+}
+```
+
+If you would like to cite the FIRST methodology, please consider citing:
+
+```
+@ARTICLE{reddy2024first,
+  title   = {FIRST: Faster Improved Listwise Reranking with Single Token Decoding},
+  author  = {Reddy, Revanth Gangi and Doo, JaeHyeok and Xu, Yifei and Sultan, Md Arafat and Swain, Deevya and Sil, Avirup and Ji, Heng},
+  year    = {2024}
+  journal = {arXiv:2406.15657},
+}
+```