Update README.md

README.md

@@ -1,4 +1,27 @@
-- Pretraining
+---
+task_categories:
+- text-generation
+---
+# Description
+This language model is the version 0.0 of a Gradio Coding Assistant. It is an instruction fine-tuned version of [StarCoder](https://huggingface.co/bigcode/starcoder) that is
+design to provide assistance to developers who are using [gradio](https://www.gradio.app).
+
+# Dataset
+The dataset is multi-source. Its content comes from the following sources
+- The stack
+More precisely, we looked into [the-stack-dedup](https://huggingface.co/datasets/bigcode/the-stack-dedup) which contain codes permissive licenses. We shortlisted the files whose
+content incorporated the keyword `gradio`.
+- GitHub Issues
+We scrapped all the issues of the official repository [the-gradio-app/gradio](https://github.com/gradio-app/gradio) and added them to our training dataset.
+- Spaces on Hugging Face Hub
+We used the [huggingface_hub api](https://huggingface.co/docs/huggingface_hub/package_reference/hf_api) to scrape the data from the spaces which are designed with gradio. We kept track of those
+with permissive licenses, namely MIT and Apache 2.0. This set of code was further deduplicated.
+
+# Training setting and hyperparameters
+For our fine-tuning, we decided to follow a 2-step strategy.
+- Pretraining (Fine-tuning) with next token prediction on the previously built gradio dataset (this step should familiarize the model with the gradio syntax.)
+- Instruction fine-tuning on an instruction fine-tuning (this step should make the model conversational)
+ ## Pretraining
   Gradio - ready 50 steps
 - Fine-tuning
   Oasst Guanaco 100 steps