preliminary clean up of model card from old example

README.md

@@ -1,37 +1,51 @@
 ---
+base_model:
+- google/flan-t5-large
+datasets:
+- deepmind/math_dataset
+language:
+- en
 library_name: transformers
-tags: []
+metrics:
+- exact_match
 ---
 
 # Model Card for Model ID
 
-<!-- Provide a quick summary of what the model is/does. -->
+Welcome to the 🤖🧮CyberSolve LinAlg 1.2🧠📐 model card! 
 
+We introduce **CyberSolve LinAlg 1.2**, a sequence-to-sequence large language model trained to solve linear equations. Specifically, *CyberSolve LingAlg 1.2* is a downstream version of the *FLAN-T5 large* 
+model, [Google/FLAN-T5-large](https://huggingface.co/google/flan-t5-large), fine-tuned on the one-dimensional linear algebra split of the Google DeepMind mathematics dataset.
+The model weights of *CyberSolve LinAlg 1.2* are a further downstream checkpoint from the original *CyberSolve LinAlg 1.1* checkpoint, trained for additional epochs to improve model capability.
+
+**Note**: This is currently the most capable version of CyberSolve LinAlg. See this model demoed in the [CyberSolve LinAlg 1.2 🤖 Space](https://huggingface.co/spaces/MarioBarbeque/CyberSolveLingAlg1.2).
 
 
 ## Model Details
 
-### Model Description
+### Model Description and Overview
+
+
+To construct **CyberSolve LinAlg 1.2**, the *FLAN-T5 large* model is fined-tuned using a custom PyTorch training loop optimized for multiple GPUs. We supervise a training of *FLAN-T5 large* on the *algebra__linear_1d* split of the [deepmind/math_dataset](https://huggingface.co/datasets/deepmind/math_dataset), an open source 
+dateset from Google DeepMind available through the 🤗 hub [deepmind/math_dataset](https://huggingface.co/datasets/deepmind/math_dataset). This large dataset consists of code generating mathematical problems and their solutions to a variety of tasks across unique mathematical disciplines. 
 
-<!-- Provide a longer summary of what this model is. -->
+In this preliminary family of CyberSolve models, we are specifically interested in understanding the ability of neural models to solve non-trivial mathematical tasks. As such, the CyberSolve **LinAlg 1.x** family of models are trained on a set of 2M simpler, one-dimension linear equations. We preprocessed the data and simulated the training process on a smaller, 
+downsampled set of the dataset before training for multiple epochs over the dataset's entirety. This model in particular has been trained for 2 additional epochs, limited only by funds, beyond the original *CyberSolve LinAlg 1.1* checkpoint.
 
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+Version 1.2 is the most capable version of CyberSolve LinAlg, scoring a **90.75** exact match score on the evaluation set of 10k linear equations from the DeepMind *algebra__linear_1d* split. This is a non-trivial improvement from the exact match score of **86.56** attained by *CyberSolve LinAlg 1.1*.
 
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
 
-### Model Sources [optional]
+- **Developed by:** John Graham Reynolds
+- **Funded by:** Vanderbilt University
+- **Model type:** Text-to-Text Generation
+- **Language(s) (NLP):** English
+- **Finetuned from model:** "Google/FLAN-T5-large"
+
+### Model Source
 
 <!-- Provide the basic links for the model. -->
 
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+- **Repository:** TODO https://github.com/johngrahamreynolds/DistilBERT-DeNiro TODO
 
 ## Uses
 
@@ -39,161 +53,128 @@ This is the model card of a 🤗 transformers model that has been pushed on the
 
 ### Direct Use
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
+In order to effectively query the model's ability to solve linear equations, a string of the format `Solve <any one-dimensional linear equation>.` should be tokenized and passed to the model's `generate` attribute. An example input string is `input_text = "Solve 24 = 1601*c - 1605*c for c."`.
+The model will attempt to solve the equation, outputting its prediction in a simple numeric format. See the example below.
 
-### Downstream Use [optional]
+## How to Use and Query the Model
 
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+Use the code below to get started with the model. Reference the Nvidia `apex` package for optimized inference. Users pass a `text` string detailing a sentence with a `[MASK]` token. The model will provide options 
+to fill the mask based on the sentence context and its background of knowledge. Note - the DistilBERT base model was trained on a very large general corpus of text.
+In our training, we have fine-tuned the model on the large IMDB movie review dataset. That is, the model is now accustomed to filling `[MASK]` tokens with words related to
+the domain of movies/tv/films. To see the model's afinity for cinematic lingo, it is best to be considerate in one's prompt engineering. Specifically, to most likely generate movie related text, 
+one should ideally pass a masked `text` string that could reasonably be found in someone's review of a movie. See the example below:
 
-[More Information Needed]
+``` python
 
-### Out-of-Scope Use
+# import apex
+import torch
+from transformers import T5ForConditionalGeneration, T5Tokenizer
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+model = T5ForConditionalGeneration.from_pretrained("MarioBarbeque/CyberSolve-LinAlg-1.2").to("cuda")
+tokenizer = AutoTokenizer.from_pretrained("google/flan-t5-large") # CyberSolve uses the same tokenizer as the base FLAN-T5 model
 
-[More Information Needed]
+# Pass the model instruction to solve a linear equation in the following simple format
+input_text = "Solve 24 = 1601*c - 1605*c for c."
+input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to("cuda")
 
-## Bias, Risks, and Limitations
+outputs = model.generate(input_ids)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+```
 
-[More Information Needed]
+This code outputs the following:
 
-### Recommendations
+``` python
+-6
+```
 
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-
-## How to Get Started with the Model
-
-Use the code below to get started with the model.
-
-[More Information Needed]
 
 ## Training Details
 
-### Training Data
-
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+### Training Data / Preprocessing
 
-[More Information Needed]
+The data used comes from the Stanford NLP 🤗 hub. The model card can be found [here](https://huggingface.co/datasets/stanfordnlp/imdb). This dataset is preprocessed in the 
+following way: The train and test splits are tokenized, concatenated, and chunked into chunks of 256 tokens. We subsequently load the training data into a `DataCollator` that 
+applies a custom random masking function when batching. We mask of 15% of tokens in each chunk. The evaluation data is masked in its entirety, to remove randomness when evaluating,
+and passed to a `DataCollator` with the default collating function.
 
 ### Training Procedure
 
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
+The model was trained locally on a single-node with one 16GB Nvidia T4 using 🤗 Transformers, 🤗 Tokenizers, and a custom PyTorch training loop that made use of 🤗 Accelerate. 
 
 
 #### Training Hyperparameters
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+- **Precision:** We use FP32 precision, as follows immediately from the precision inhereted for the original "DistilBERT/distilbert-base-uncased" model.
+- **Optimizer:** `apex.optimizers.FusedAdam`, a fused kernel version of the AdamW optimizer from Nvidia `apex`
+- **Learning Rate:** We use a linear learing rate scheduler with an initial learning rate of 5e-5
+- **Batch Size:** 32
+- **Number of Training Steps**: 2877 steps over the course of 3 epochs
+
 
-#### Speeds, Sizes, Times [optional]
+## Evaluation / Metrics
 
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+We evaluate our masked language model's performance using the `perplexity` metric, which has a few mathematical defitions. We define the perplexity as the exponential of the cross-entropy. 
+To remove randomness in our metrics, we premask our evaluation dataset with a single masking function. This ensures we are evaluating with respect to the same set of labels each epoch.
+See the wikipedia links for perplexity and cross-entropy below for more a detailed discussion and various other definitions.
 
-[More Information Needed]
+Cross-entropy: [https://en.wikipedia.org/wiki/Cross-entropy](https://en.wikipedia.org/wiki/Cross-entropy)
 
-## Evaluation
+Perplexity: [https://en.wikipedia.org/wiki/Perplexity](https://en.wikipedia.org/wiki/Perplexity)
 
-<!-- This section describes the evaluation protocols and provides the results. -->
 
 ### Testing Data, Factors & Metrics
 
 #### Testing Data
 
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
-
-#### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
-
-#### Metrics
-
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
+The IMDB dataset from Stanford NLP comes pre-split into training and testing data of 25k reviews each. Our preprocessing, which included the chunking of concatenated, tokenized inputs
+into chunks of 256 tokens, increased these respective splits by approximately ~5k records each. We apply a single masking function to the evaluation dataset before testing as mentioned above.
 
 ### Results
 
-[More Information Needed]
-
-#### Summary
-
+We find the following perplexity metrics over 3 training epochs:
 
+ | epoch | perplexity |
+ |-------|------------|
+ |0      |  17.38 |
+ |1      |  16.28 |
+ |2      |  15.78 |
 
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
+#### Summary
 
-[More Information Needed]
+We train this model for the purpose of attempting a local training of a masked language model using both the 🤗 ecosystem and a custom PyTorch training and evaluation loop. 
+We look forward to further fine-tuning this model on more film/actor/cinema related data in order to further improve the model's knowledge and ability in this domain - 
+indeed cinema is one of the author's favorite things.
 
 ## Environmental Impact
 
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+- **Hardware Type:** Nvidia Tesla T4 16GB
+- **Hours used:** 1.2
+- **Cloud Provider:** Microsoft Azure
+- **Compute Region:** EastUS
+- **Carbon Emitted:** 0.03 kgCO2
 
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
 
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
+Experiments were conducted using Azure in region eastus, which has a carbon efficiency of 0.37 kgCO$_2$eq/kWh. A cumulative of 1.2 hours of computation was performed on hardware of type T4 (TDP of 70W).
 
-## Technical Specifications [optional]
+Total emissions are estimated to be 0.03 kgCO$_2$eq of which 100 percents were directly offset by the cloud provider.
 
-### Model Architecture and Objective
-
-[More Information Needed]
-
-### Compute Infrastructure
-
-[More Information Needed]
+Estimations were conducted using the MachineLearning Impact calculator presented in  [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
 
 #### Hardware
 
-[More Information Needed]
+The model was trained locally in an Azure Databricks workspace using a single node with 1 16GB Nvidia T4 GPU for 1.2 GPU Hours.
 
 #### Software
 
-[More Information Needed]
-
-## Citation [optional]
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
-**BibTeX:**
-
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
-
-## Glossary [optional]
-
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
-
-## More Information [optional]
-
-[More Information Needed]
-
-## Model Card Authors [optional]
+Training utilized PyTorch, 🤗 Transformers, 🤗 Tokenizers, 🤗 Datasets, 🤗 Accelerate, and more in an Azure Databricks execution environment.
 
-[More Information Needed]
+#### Citations
 
-## Model Card Contact
 
-[More Information Needed]
+@article{lacoste2019quantifying,
+  title={Quantifying the Carbon Emissions of Machine Learning},
+  author={Lacoste, Alexandre and Luccioni, Alexandra and Schmidt, Victor and Dandres, Thomas},
+  journal={arXiv preprint arXiv:1910.09700},
+  year={2019}
+}