| | --- |
| | license: apache-2.0 |
| | language: en |
| | tags: |
| | - deberta-v3-large |
| | - text-classification |
| | - nli |
| | - natural-language-inference |
| | - multitask |
| | - multi-task |
| | - pipeline |
| | - extreme-multi-task |
| | - extreme-mtl |
| | - tasksource |
| | - zero-shot |
| | - rlhf |
| | pipeline_tag: zero-shot-classification |
| | datasets: |
| | - glue |
| | - super_glue |
| | - anli |
| | - metaeval/babi_nli |
| | - sick |
| | - snli |
| | - scitail |
| | - hans |
| | - alisawuffles/WANLI |
| | - metaeval/recast |
| | - sileod/probability_words_nli |
| | - joey234/nan-nli |
| | - pietrolesci/nli_fever |
| | - pietrolesci/breaking_nli |
| | - pietrolesci/conj_nli |
| | - pietrolesci/fracas |
| | - pietrolesci/dialogue_nli |
| | - pietrolesci/mpe |
| | - pietrolesci/dnc |
| | - pietrolesci/gpt3_nli |
| | - pietrolesci/recast_white |
| | - pietrolesci/joci |
| | - martn-nguyen/contrast_nli |
| | - pietrolesci/robust_nli |
| | - pietrolesci/robust_nli_is_sd |
| | - pietrolesci/robust_nli_li_ts |
| | - pietrolesci/gen_debiased_nli |
| | - pietrolesci/add_one_rte |
| | - metaeval/imppres |
| | - pietrolesci/glue_diagnostics |
| | - hlgd |
| | - paws |
| | - quora |
| | - medical_questions_pairs |
| | - conll2003 |
| | - Anthropic/hh-rlhf |
| | - Anthropic/model-written-evals |
| | - truthful_qa |
| | - nightingal3/fig-qa |
| | - tasksource/bigbench |
| | - bigbench |
| | - blimp |
| | - cos_e |
| | - cosmos_qa |
| | - dream |
| | - openbookqa |
| | - qasc |
| | - quartz |
| | - quail |
| | - head_qa |
| | - sciq |
| | - social_i_qa |
| | - wiki_hop |
| | - wiqa |
| | - piqa |
| | - hellaswag |
| | - pkavumba/balanced-copa |
| | - 12ml/e-CARE |
| | - art |
| | - tasksource/mmlu |
| | - winogrande |
| | - codah |
| | - ai2_arc |
| | - definite_pronoun_resolution |
| | - swag |
| | - math_qa |
| | - metaeval/utilitarianism |
| | - mteb/amazon_counterfactual |
| | - SetFit/insincere-questions |
| | - SetFit/toxic_conversations |
| | - turingbench/TuringBench |
| | - trec |
| | - tals/vitaminc |
| | - hope_edi |
| | - strombergnlp/rumoureval_2019 |
| | - ethos |
| | - tweet_eval |
| | - discovery |
| | - pragmeval |
| | - silicone |
| | - lex_glue |
| | - papluca/language-identification |
| | - imdb |
| | - rotten_tomatoes |
| | - ag_news |
| | - yelp_review_full |
| | - financial_phrasebank |
| | - poem_sentiment |
| | - dbpedia_14 |
| | - amazon_polarity |
| | - app_reviews |
| | - hate_speech18 |
| | - sms_spam |
| | - humicroedit |
| | - snips_built_in_intents |
| | - banking77 |
| | - hate_speech_offensive |
| | - yahoo_answers_topics |
| | - pacovaldez/stackoverflow-questions |
| | - zapsdcn/hyperpartisan_news |
| | - zapsdcn/sciie |
| | - zapsdcn/citation_intent |
| | - go_emotions |
| | - scicite |
| | - liar |
| | - relbert/lexical_relation_classification |
| | - metaeval/linguisticprobing |
| | - metaeval/crowdflower |
| | - metaeval/ethics |
| | - emo |
| | - google_wellformed_query |
| | - tweets_hate_speech_detection |
| | - has_part |
| | - wnut_17 |
| | - ncbi_disease |
| | - acronym_identification |
| | - jnlpba |
| | - species_800 |
| | - SpeedOfMagic/ontonotes_english |
| | - blog_authorship_corpus |
| | - launch/open_question_type |
| | - health_fact |
| | - commonsense_qa |
| | - mc_taco |
| | - ade_corpus_v2 |
| | - prajjwal1/discosense |
| | - circa |
| | - YaHi/EffectiveFeedbackStudentWriting |
| | - Ericwang/promptSentiment |
| | - Ericwang/promptNLI |
| | - Ericwang/promptSpoke |
| | - Ericwang/promptProficiency |
| | - Ericwang/promptGrammar |
| | - Ericwang/promptCoherence |
| | - PiC/phrase_similarity |
| | - copenlu/scientific-exaggeration-detection |
| | - quarel |
| | - mwong/fever-evidence-related |
| | - numer_sense |
| | - dynabench/dynasent |
| | - raquiba/Sarcasm_News_Headline |
| | - sem_eval_2010_task_8 |
| | - demo-org/auditor_review |
| | - medmcqa |
| | - aqua_rat |
| | - RuyuanWan/Dynasent_Disagreement |
| | - RuyuanWan/Politeness_Disagreement |
| | - RuyuanWan/SBIC_Disagreement |
| | - RuyuanWan/SChem_Disagreement |
| | - RuyuanWan/Dilemmas_Disagreement |
| | - lucasmccabe/logiqa |
| | - wiki_qa |
| | - metaeval/cycic_classification |
| | - metaeval/cycic_multiplechoice |
| | - metaeval/sts-companion |
| | - metaeval/commonsense_qa_2.0 |
| | - metaeval/lingnli |
| | - metaeval/monotonicity-entailment |
| | - metaeval/arct |
| | - metaeval/scinli |
| | - metaeval/naturallogic |
| | - onestop_qa |
| | - demelin/moral_stories |
| | - corypaik/prost |
| | - aps/dynahate |
| | - metaeval/syntactic-augmentation-nli |
| | - metaeval/autotnli |
| | - lasha-nlp/CONDAQA |
| | - openai/webgpt_comparisons |
| | - Dahoas/synthetic-instruct-gptj-pairwise |
| | - metaeval/scruples |
| | - metaeval/wouldyourather |
| | - sileod/attempto-nli |
| | - metaeval/defeasible-nli |
| | - metaeval/help-nli |
| | - metaeval/nli-veridicality-transitivity |
| | - metaeval/natural-language-satisfiability |
| | - metaeval/lonli |
| | - metaeval/dadc-limit-nli |
| | - ColumbiaNLP/FLUTE |
| | - metaeval/strategy-qa |
| | - openai/summarize_from_feedback |
| | - metaeval/folio |
| | - metaeval/tomi-nli |
| | - metaeval/avicenna |
| | - stanfordnlp/SHP |
| | - GBaker/MedQA-USMLE-4-options-hf |
| | - sileod/wikimedqa |
| | - declare-lab/cicero |
| | - amydeng2000/CREAK |
| | - metaeval/mutual |
| | - inverse-scaling/NeQA |
| | - inverse-scaling/quote-repetition |
| | - inverse-scaling/redefine-math |
| | - metaeval/puzzte |
| | - metaeval/implicatures |
| | - race |
| | - metaeval/spartqa-yn |
| | - metaeval/spartqa-mchoice |
| | - metaeval/temporal-nli |
| | metrics: |
| | - accuracy |
| | library_name: transformers |
| | --- |
| | |
| | # Model Card for DeBERTa-v3-large-tasksource-nli |
| |
|
| | DeBERTa-v3-large fine-tuned with multi-task learning on 600 tasks of the [tasksource collection](https://github.com/sileod/tasksource/) |
| | You can further fine-tune this model to use it for any classification or multiple-choice task. |
| | This checkpoint has strong zero-shot validation performance on many tasks (e.g. 77% on WNLI). |
| | The untuned model CLS embedding also has strong linear probing performance (90% on MNLI), due to the multitask training. |
| |
|
| | This is the shared model with the MNLI classifier on top. Its encoder was trained on many datasets including bigbench, Anthropic rlhf, anli... alongside many NLI and classification tasks with a SequenceClassification heads while using only one shared encoder. |
| | Each task had a specific CLS embedding, which is dropped 10% of the time to facilitate model use without it. All multiple-choice model used the same classification layers. For classification tasks, models shared weights if their labels matched. |
| | The number of examples per task was capped to 64k. The model was trained for 80k steps with a batch size of 384, and a peak learning rate of 2e-5. |
| |
|
| |
|
| | tasksource training code: https://colab.research.google.com/drive/1iB4Oxl9_B5W3ZDzXoWJN-olUbqLBxgQS?usp=sharing |
| | |
| | ### Software |
| | https://github.com/sileod/tasksource/ \ |
| | https://github.com/sileod/tasknet/ \ |
| | Training took 6 days on Nvidia A100 40GB GPU. |
| | |
| | |
| | |
| | # Citation |
| | |
| | More details on this [article:](https://arxiv.org/abs/2301.05948) |
| | ```bib |
| | @article{sileo2023tasksource, |
| | title={tasksource: Structured Dataset Preprocessing Annotations for Frictionless Extreme Multi-Task Learning and Evaluation}, |
| | author={Sileo, Damien}, |
| | url= {https://arxiv.org/abs/2301.05948}, |
| | journal={arXiv preprint arXiv:2301.05948}, |
| | year={2023} |
| | } |
| | ``` |
| | |
| | # Loading a specific classifier |
| | Classifiers for all tasks available. See https://huggingface.co/sileod/deberta-v3-large-tasksource-adapters |
| | |
| | <img src="https://www.dropbox.com/s/eyfw8i1ekzxj3fa/task_embeddings.png?dl=1" width="1000" height=""> |
| | |
| | |
| | # Model Card Contact |
| | |
| | damien.sileo@inria.fr |
| | |
| | |
| | </details> |
