Update README.md
Browse files
README.md
CHANGED
|
@@ -4,6 +4,10 @@ This repo contains the model for our paper [Adapting Large Language Models via R
|
|
| 4 |
|
| 5 |
We explore **continued pre-training on domain-specific corpora** for large language models. While this approach enriches LLMs with domain knowledge, it significantly hurts their prompting ability for question answering. Inspired by human learning via reading comprehension, we propose a simple method to **transform large-scale pre-training corpora into reading comprehension texts**, consistently improving prompting performance across tasks in **biomedicine, finance, and law domains**. Our 7B model competes with much larger domain-specific models like BloombergGPT-50B. Moreover, our domain-specific reading comprehension texts enhance model performance even on general benchmarks, indicating potential for developing a general LLM across more domains.
|
| 6 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 7 |
## GitHub repo:
|
| 8 |
https://github.com/microsoft/LMOps
|
| 9 |
|
|
@@ -11,7 +15,7 @@ https://github.com/microsoft/LMOps
|
|
| 11 |
Our models of different domains are now available in Huggingface: [Biomedicine-LLM](https://huggingface.co/AdaptLLM/medicine-LLM), [Finance-LLM](https://huggingface.co/AdaptLLM/finance-LLM) and [Law-LLM](https://huggingface.co/AdaptLLM/law-LLM), the performances of our AdaptLLM compared to other domain-specific LLMs are:
|
| 12 |
|
| 13 |
<p align='center'>
|
| 14 |
-
<img src="./
|
| 15 |
</p>
|
| 16 |
|
| 17 |
## Domain-specific Tasks:
|
|
|
|
| 4 |
|
| 5 |
We explore **continued pre-training on domain-specific corpora** for large language models. While this approach enriches LLMs with domain knowledge, it significantly hurts their prompting ability for question answering. Inspired by human learning via reading comprehension, we propose a simple method to **transform large-scale pre-training corpora into reading comprehension texts**, consistently improving prompting performance across tasks in **biomedicine, finance, and law domains**. Our 7B model competes with much larger domain-specific models like BloombergGPT-50B. Moreover, our domain-specific reading comprehension texts enhance model performance even on general benchmarks, indicating potential for developing a general LLM across more domains.
|
| 6 |
|
| 7 |
+
<p align='center'>
|
| 8 |
+
<img src="https://cdn-uploads.huggingface.co/production/uploads/650801ced5578ef7e20b33d4/Qi89o8zcwqbLtdhtNbMvV.png" width="700">
|
| 9 |
+
</p>
|
| 10 |
+
|
| 11 |
## GitHub repo:
|
| 12 |
https://github.com/microsoft/LMOps
|
| 13 |
|
|
|
|
| 15 |
Our models of different domains are now available in Huggingface: [Biomedicine-LLM](https://huggingface.co/AdaptLLM/medicine-LLM), [Finance-LLM](https://huggingface.co/AdaptLLM/finance-LLM) and [Law-LLM](https://huggingface.co/AdaptLLM/law-LLM), the performances of our AdaptLLM compared to other domain-specific LLMs are:
|
| 16 |
|
| 17 |
<p align='center'>
|
| 18 |
+
<img src="https://cdn-uploads.huggingface.co/production/uploads/650801ced5578ef7e20b33d4/6efPwitFgy-pLTzvccdcP.png" width="700">
|
| 19 |
</p>
|
| 20 |
|
| 21 |
## Domain-specific Tasks:
|