Update README.md

README.md

@@ -1,3 +1,48 @@
 ---
 license: llama2
 ---
+
+
+This is Phind v2 QLoRa finetune using my PythonTutor LIMA dataset:
+https://huggingface.co/datasets/KrisPi/PythonTutor-LIMA-Finetune
+
+My shy attempt to democratize task-specific, cheap fine-tuning focused around LIMA-like datasets -everybody can afford to generate them (less than 20$) and everybody can finetune them (7 hours in total using 2x3090 GPU ~3$+5$ on vast.ai) 
+
+At the moment of publishing this adapter, there are already production-ready solutions for serving several LorA adapters. I honestly believe that the route of a reproducible, vast collection of adapters on the top of current SOTA models, will enable the open-source community to access GPT-4 level LLMs in the next 12 months.
+
+My main inspirations for this were blazing fast implementation of multi-LORA in Exllamav2 backend, Jon's LMoE and Airoboros dataset, r/LocalLLaMA opinions around models based on LIMA finetunes, and of course the LIMA paper itself.   
+To prove the point I'm planning to create a few more finetunes like this, starting with the Airoboros "contextual" category for RAG solutions, adapters for React and DevOps YAML scripting. 
+
+5 epochs, LR=1e-05, batch=2, gradient accumulation 32 (i.e. trying to simulate batch 64), max_len=1024. Rank and Alpha both 128 targeting all modules. trained in bfloat16. Constant schedule, no warm-up.
+Flash-Attention 2 turned off due to an issue with batching
+
+Evals:
+HumanEval score (2.4 p.p improvement to best Phind v2 score!) for the new prompt:
+**{'pass@1': 0.7621951219512195}**
+**Base + Extra**
+**{'pass@1': 0.7073170731707317}**
+Base prompt (0.51 p.p improvement)
+{'pass@1': 0.725609756097561}
+Base + Extra
+{'pass@1': 0.6585365853658537}
+Phind v2 with Python Tutor custom prompt is only getting:
+{'pass@1': 0.7073170731707317}
+Base + Extra
+{'pass@1': 0.6463414634146342}
+After several HumanEval tests and prompts Phind v2 was maximum able to score: 73.78%
+**All evals using Transformers 8bit**
+
+In the long term, I'm planning on experimenting with LIMA + DPO Fine-Tuning, but so far I noticed that LIMA datasets need to be both general and task-specific. The best result I got with around 30% of samples that were task specific. 
+https://huggingface.co/datasets/KrisPi/PythonTutor-Evol-1k-DPO-GPT4_vs_35
+
+    r=128,
+    lora_alpha=128,
+    target_modules=['q_proj','k_proj','v_proj','o_proj','gate_proj','down_proj','up_proj'],
+    lora_dropout=0.03,
+    
+bnb_config = BitsAndBytesConfig(
+    load_in_4bit=True,
+    bnb_4bit_quant_type="nf4",
+    bnb_4bit_compute_dtype=torch.bfloat16,
+    bnb_4bit_use_double_quant=True,
+)