Clover-Hill
/

MemoryDecoder-gpt2-small

Model card Files Files and versions

Clover-Hill commited on Sep 29

Commit

276bc2b

·

verified ·

1 Parent(s): 5211579

Update README.md

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -107,7 +107,7 @@ logger.info(f"Base Model output: {tokenizer.decode(out_ids[0], skip_special_toke
 - **Training Data:** WikiText-103
 - **Training Objective:** Hybrid KL divergence and language modeling loss
-- **Supervision Signal:** kNN distributions from GPT2-xl
 - **Hyperparameters:**
   - Learning rate: 1e-3
   - Beta (loss balance): 0.5

 - **Training Data:** WikiText-103
 - **Training Objective:** Hybrid KL divergence and language modeling loss
+- **Supervision Signal:** kNN distributions from GPT2-xl, it is suggested to use the finetuned version of GPT2-xl [here](https://huggingface.co/Clover-Hill/gpt2-xl-finetuned-wikitext103).
 - **Hyperparameters:**
   - Learning rate: 1e-3
   - Beta (loss balance): 0.5