Update README.md
Browse files
README.md
CHANGED
|
@@ -1,3 +1,23 @@
|
|
| 1 |
---
|
| 2 |
license: cc-by-nc-4.0
|
| 3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
license: cc-by-nc-4.0
|
| 3 |
---
|
| 4 |
+
This model is a generation model trained via [semiparametric token-sequence co-supervision](https://github.com/kaistAI/Semiparametric_Token-Sequence_Co-Supervision) on top of Llama2-7B.
|
| 5 |
+
The embedding model which constructs the nonparametric sequence embedding spaces is in [here](https://huggingface.co/kaist-ai/cosupervision-emb_seq-Llama2_7b).
|
| 6 |
+
The models are trained on information-seeking datasets provided by [self-rag](https://selfrag.github.io/) with co-supervision from next token prediction (NTP) and next sequence prediction (NSP).
|
| 7 |
+
In the inference step, the model generates a response by retrieving relevant sequences.
|
| 8 |
+
See full descriptions in our paper.
|
| 9 |
+
|
| 10 |
+
### Usage
|
| 11 |
+
Here, we show an easy way to quickly download our model from HuggingFace.
|
| 12 |
+
Make sure to install dependencies listed at requirements.txt.
|
| 13 |
+
To run our full inference pipeline with embedding model, please use our [code](https://github.com/kaistAI/Semiparametric_Token-Sequence_Co-Supervision).
|
| 14 |
+
|
| 15 |
+
```
|
| 16 |
+
from transformers import AutoTokenizer, LlamaForCausalLM
|
| 17 |
+
|
| 18 |
+
model = LlamaForCausalLM.from_pretrained(
|
| 19 |
+
"kaist-ai/cosupervision-emb_seq-Llama2_7b",
|
| 20 |
+
load_in_8bit=True if train_config.quantization else None,
|
| 21 |
+
device_map="auto" if train_config.quantization else None,
|
| 22 |
+
)
|
| 23 |
+
```
|