Update README.md

README.md

@@ -1,3 +1,84 @@
 ---
+language: en
+tags:
+- exbert
 license: mit
 ---
+
+# ColD Fusion model
+
+Finetuned model that aims to be a great base model. It improves over RoBERTa base, trained on 35 datasets. 
+Full details at [this paper](https://arxiv.org/abs/2212.01378).
+
+## Paper Abstract:
+
+Pretraining has been shown to scale well with compute, data size and data diversity. Multitask learning trains on a 
+mixture of supervised datasets and produces improved performance compared to self-supervised pretraining. Until now, 
+massively multitask learning required simultaneous access to all datasets in the mixture and heavy compute resources 
+that are only available to well-resourced teams.
+
+In this paper, we propose ColD Fusion, a method that provides the benefits of multitask learning but leverages distributed 
+computation and requires limited communication and no sharing of data. Consequentially, ColD Fusion can create a synergistic 
+loop, where finetuned models can be recycled to continually improve the pretrained model they are based on. We show that 
+ColD Fusion yields comparable benefits to multitask pretraining by producing a model that (a) attains strong performance on 
+all of the datasets it was multitask trained on and (b) is a better starting point for finetuning on unseen datasets. We find 
+ColD Fusion outperforms RoBERTa and even previous multitask models. Specifically, when training and testing on 35 diverse datasets, 
+ColD Fusion-based model outperforms RoBERTa by 2.45 points in average without any changes to the architecture.
+
+
+### How to use
+
+You can use this model directly with a pipeline for masked language modeling:
+
+```python
+>>> from transformers import pipeline
+>>> unmasker = pipeline('fill-mask', model='ibm/ColD-Fusion')
+>>> unmasker("Hello I'm a <mask> model.")
+```
+
+Here is how to use this model to get the features of a given text in PyTorch:
+
+```python
+from transformers import RobertaTokenizer, RobertaModel
+tokenizer = RobertaTokenizer.from_pretrained('ibm/ColD-Fusion')
+model = RobertaModel.from_pretrained('ibm/ColD-Fusion')
+text = "Replace me by any text you'd like."
+encoded_input = tokenizer(text, return_tensors='pt')
+output = model(**encoded_input)
+```
+
+and in TensorFlow:
+
+```python
+from transformers import RobertaTokenizer, TFRobertaModel
+tokenizer = RobertaTokenizer.from_pretrained('ibm/ColD-Fusion')
+model = TFRobertaModel.from_pretrained('ibm/ColD-Fusion')
+text = "Replace me by any text you'd like."
+encoded_input = tokenizer(text, return_tensors='tf')
+output = model(encoded_input)
+```
+
+## Evaluation results
+See full evaluation results of this model and many more [here](https://ibm.github.io/model-recycling/roberta-base_table.html)
+When fine-tuned on downstream tasks, this model achieves the following results:
+
+
+
+### BibTeX entry and citation info
+
+```bibtex
+@article{ColDFusion,
+  author    = {Shachar Don-Yehiya, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, Leshem ChoshenYinhan Liu and},
+  title     = {ColD Fusion: Collaborative Descent for Distributed Multitask Finetuning},
+  journal   = {CoRR},
+  volume    = {abs/2212.01378},
+  year      = {2022},
+  url       = {https://arxiv.org/abs/2212.01378},
+  archivePrefix = {arXiv},
+  eprint    = {2212.01378},
+}
+```
+
+<a href="https://huggingface.co/exbert/?model=ibm/ColD-Fusion">
+	<img width="300px" src="https://cdn-media.huggingface.co/exbert/button.png">
+</a>