Update README.md

README.md

@@ -1,33 +1,39 @@
 ---
-# For reference on model card metadata, see the spec: https://github.com/huggingface/hub-docs/blob/main/modelcard.md?plain=1
-# Doc / guide: https://huggingface.co/docs/hub/model-cards
-{}
+datasets:
+- semaj83/ctmatch
+language:
+- en
+metrics:
+- f1
+pipeline_tag: text-classification
+tags:
+- medical
 ---
 
-# Model Card for Model ID
+# Model Card for semaj83/scibert_finetuned_ctmatch 
 
-<!-- Provide a quick summary of what the model is/does. -->
 
-This modelcard aims to be a base template for new models. It has been generated using [this raw template](https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/templates/modelcard_template.md?plain=1).
+This model can be used for classifying "<topic> [SEP] <clinical trial documents>" pairs into 3 classes, 0, 1, 2, or not relevant, partially relevant, and relevant.
 
 ## Model Details
 
+Fine-tuned from 'allenai/scibert_scivocab_uncased' on triples of labelled topic, documents, relevance labels.
+These triples were processed using ctproc, collated from the openly available TREC22 Precision Medicine and CSIRO datasets here: 
+https://huggingface.co/datasets/semaj83/ctmatch
+
 ### Model Description
 
-<!-- Provide a longer summary of what this model is. -->
+Transformer model with linear sequence classification head, trained with cross-entropy on ~30k triples and evaluated using f1.
 
 
 
-- **Developed by:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
+- **Developed by:** James Kelly
+- **Model type:** SequenceClassification
+- **Language(s) (NLP):** English
 - **License:** [MIT]
-- **Finetuned from model [optional]:** [allenai/scibert_scivocab_uncased]
-
-### Model Sources [optional]
+- **Finetuned from model:** [allenai/scibert_scivocab_uncased]
 
-<!-- Provide the basic links for the model. -->
+### Model Sources 
 
 - **Repository:** [https://github.com/semajyllek/ctmatch]
 - **Paper [optional]:** [More Information Needed]
@@ -39,163 +45,98 @@ This modelcard aims to be a base template for new models. It has been generated
 
 ### Direct Use
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
-
-### Downstream Use [optional]
 
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
 
 [More Information Needed]
 
-### Out-of-Scope Use
+### Downstream Use 
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+ctmatch IR pipeline for matching large set of clinical trials documents to text topic.
 
-[More Information Needed]
-
-## Bias, Risks, and Limitations
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
 
-[More Information Needed]
+## Bias, Risks, and Limitations
 
-### Recommendations
+Please see dataset sources for information on patient descriptions (topics), constructed by medical professionals for these datasets. 
+No personal health information about real individuals is contained in the related dataset. 
+Links in dataset location on hub.
 
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+The claissifier model performs much better on deciding if a pair is 0 - not relevant, than differentiating between 1, partially relevant, and 2, relevant,
+though this is still an important clinical task.
 
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
 
 ## How to Get Started with the Model
 
 Use the code below to get started with the model.
 
-[More Information Needed]
-
-## Training Details
-
-### Training Data
-
-<!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
-
-### Training Procedure 
+'''
+from transformers import AutoTokenizer, AutoModelForSequenceClassification
 
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+tokenizer = AutoTokenizer.from_pretrained("semaj83/scibert_finetuned_ctmatch")
 
-#### Preprocessing [optional]
-
-[More Information Needed]
-
-
-#### Training Hyperparameters
-
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-
-#### Speeds, Sizes, Times [optional]
-
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-
-[More Information Needed]
-
-## Evaluation
-
-<!-- This section describes the evaluation protocols and provides the results. -->
-
-### Testing Data, Factors & Metrics
-
-#### Testing Data
-
-<!-- This should link to a Data Card if possible. -->
-
-[More Information Needed]
+model = AutoModelForSequenceClassification.from_pretrained("semaj83/scibert_finetuned_ctmatch")
 
-#### Factors
+'''
 
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
-
-#### Metrics
-
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
-
-### Results
-
-[More Information Needed]
-
-#### Summary
-
-
-
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
-
-## Environmental Impact
-
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+## Training Details
 
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+see notebook in ctmatch repo.
 
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
 
-## Technical Specifications [optional]
 
-### Model Architecture and Objective
+### Training Data
 
-[More Information Needed]
+https://huggingface.co/datasets/semaj83/ctmatch
 
-### Compute Infrastructure
 
-[More Information Needed]
+#### Preprocessing 
 
-#### Hardware
+If using ctmatch labelled dataset, using the tokenizer alone is sufficient. If using raw topic and/or clinical trial documents, 
+you may need to use ctproc or another method to extract relevant fields and preprocess text.
 
-[More Information Needed]
 
-#### Software
-
-[More Information Needed]
-
-## Citation [optional]
+#### Training Hyperparameters
 
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+`
+max_sequence_length=512
+batch_size=8
+padding='max_length'
+truncation=True
+learning_rate=2e-5
+train_epochs=5
+weight_decay=0.01
+warmup_steps=500
+seed=42
+splits={"train":0.8, "val":0.1}
+use_trainer=True
+fp16=True
+early_stopping=True
 
-**BibTeX:**
+`
 
-[More Information Needed]
 
-**APA:**
+## Evaluation
 
-[More Information Needed]
+sklearn classifier table on random test split:
 
-## Glossary [optional]
+`
+                precision    recall  f1-score   support
 
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+           0       0.88      0.93      0.90      5430
+           1       0.56      0.56      0.56      1331
+           2       0.65      0.49      0.56      1178
 
-[More Information Needed]
+    accuracy                           0.80      7939
+   macro avg       0.70      0.66      0.67      7939
+weighted avg       0.79      0.80      0.79      7939
 
-## More Information [optional]
+`
 
-[More Information Needed]
 
-## Model Card Authors [optional]
+## Model Card Authors 
 
-[More Information Needed]
+James Kelly
 
 ## Model Card Contact
 
-[More Information Needed]
-
-
+mrkellyjam@gmail.com