Update README.md

README.md

@@ -1,3 +1,51 @@
 ---
 license: apache-2.0
+datasets: billsum
+tags:
+- summarization
 ---
+
+# Longformer Encoder-Decoder (LED) fine-tuned on Billsum
+
+
+
+This model is a fine-tuned version of [led-base-16384](https://huggingface.co/allenai/led-base-16384) on the [billsum](https://huggingface.co/datasets/billsum) dataset.
+
+As described in [Longformer: The Long-Document Transformer](https://arxiv.org/pdf/2004.05150.pdf) by Iz Beltagy, Matthew E. Peters, Arman Cohan, *led-base-16384* was initialized from [*bart-base*](https://huggingface.co/facebook/bart-base) since both models share the exact same architecture. To be able to process 16K tokens, *bart-base*'s position embedding matrix was simply copied 16 times.
+
+
+## How to use
+
+```Python
+from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
+
+device = "cuda" if torch.cuda.is_available() else "cpu"
+
+tokenizer = AutoTokenizer.from_pretrained("d0r1h/LEDBill")
+model = AutoModelForSeq2SeqLM.from_pretrained("d0r1h/LEDBill", return_dict_in_generate=True).to(device)
+
+case = "......."
+
+input_ids = tokenizer(case, return_tensors="pt").input_ids.to(device)
+global_attention_mask = torch.zeros_like(input_ids)
+global_attention_mask[:, 0] = 1
+
+sequences = model.generate(input_ids, 
+                           global_attention_mask=global_attention_mask).sequences
+summary = tokenizer.batch_decode(sequences, 
+                                 skip_special_tokens=True)
+                                 
+```
+
+
+## Evaluation results
+
+When the model is used for summarizing Billsum documents(10 sample), it achieves the following results:
+
+| Model |            rouge1-f  |  rouge1-p  |  rouge2-f   |  rouge2-p  | rougeL-f   |  rougeL-p  |
+|:-----------:|:-----:|:-----:|:------:|:-----:|:------:|:-----:|
+| LEDBill           | **34** |  **37** | **15**  | **16** | **30**  | **32** | 
+| led-base          | 2     |  15     | 0      | 0     | 2      | 15 |
+
+[This notebook](https://colab.research.google.com/drive/1iEEFbWeTGUSDesmxHIU2QDsPQM85Ka1K?usp=sharing) shows how *led* can effectively be used for downstream task such summarization.
+