Update README.md

README.md

@@ -15,9 +15,11 @@ tags:
 widget:
 - text: "### Task\nGenerate a SQL query to answer the following question:\n`How many heads of the departments are older than 56?`\n\n### Database Schema\nThe query will run on a database with the following schema:\nCREATE TABLE head (age INTEGER)\n\n### Answer\nGiven the database schema, here is the SQL query that answers `How many heads of the departments are older than 56?`:\n```sql"
   example_title: "One Table"
+- text: "### Task\nGenerate a SQL query to answer the following question:\n`Show the name and number of employees for the departments managed by heads whose temporary acting value is 'Yes'?`\n\n### Database Schema\nThe query will run on a database with the following schema:\nCREATE TABLE management (department_id VARCHAR, temporary_acting VARCHAR); CREATE TABLE department (name VARCHAR, num_employees VARCHAR, department_id VARCHAR)\n\n### Answer\nGiven the database schema, here is the SQL query that answers `Show the name and number of employees for the departments managed by heads whose temporary acting value is 'Yes'?`:\n```sql"
+  example_title: "Two Tables" 
 ---
 
-# Update: 14-03-2024 - The model card is still updating. Thanks for being patient! 💜💜
+# Thanks for being patient! 💜💜
 
 # Model Card for Model ID
 
@@ -175,26 +177,50 @@ Users (both direct and downstream) should be made aware of the risks, biases and
 
 <!-- This should link to a Dataset Card if possible. -->
 
-[More Information Needed]
+Used b-mc2/sql-create-context and split the data into training and testing datasets. The holdout dataset is used for testing the model. 
 
 #### Factors
 
 <!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+The complexity of the questions are calculated using the number of tables per question, number of joins, group by, and sub queries per answer. This complexity is used to prepare the test data by stratifying the split around the complexity.
 
-[More Information Needed]
 
 #### Metrics
 
 <!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
-
+* __Execution Success:__ This metric is used to find out if the generated query is executable without arising any errors. For this, a sqllite3 connection is made to the memory, and using context the dummy tables are created. Then the predicted SQL is executed. This checks out if the generated query is in proper syntax, and if the model is hallucinating any new columns.
+* __Inference Time:__ This metric is used to find out which model is providing results in less amount of time. This combined with the execution success, gives the efficiency of the model.
+- 
 ### Results
 
-[More Information Needed]
+* __Execution Success:__ Finetuned Phi-2 has 29% more success rate than the SQLCoder-7b-2
+* __Inference Time:__ Finetuned Phi-2 has 41% increased inference speed than SQLCoder-7b-2  
 
 #### Summary
-
+* __Reduced Inference Time and Memory Footprint:__ The fine-tuned Phi-2 model
+demonstrated a reduction in inference time and memory usage compared to the DeFog
+SQLCoder. This is attributed to Phi-2's smaller size and the efficiency of quantization
+techniques employed during fine-tuning. This finding implies that NL2SQL models can
+be deployed on lower-powered devices like laptops or even mobile phones, potentially
+democratizing access to this technology for a wider range of users.
+
+* __Competitive Performance on Easy and Medium Queries:__ The fine-tuned Phi-2
+achieved comparable performance to the DeFog SQLCoder in terms of accuracy on easy,
+medium, and hard difficulty queries. This indicates that Phi-2, despite its smaller size,
+can effectively handle a significant portion of real-world NL2SQL tasks, especially for
+simpler queries.
+
+* __Challenges with Complex Queries:__ While Phi-2 performed well on easier queries, it
+encountered challenges with complex queries, exhibiting a drop in execution success
+compared to the DeFog SQLCoder. This highlights the trade-off between model size and
+complexity, suggesting that larger models might still be necessary for tackling highly
+intricate tasks.
+
+* __Potential for Further Improvement:__ The fine-tuning process employed in this study
+can be further optimized by exploring different hyperparameter configurations and
+potentially investigating alternative fine-tuning techniques like adapter-based methods.
+This optimization has the potential to improve the model's performance on complex
+queries while maintaining its efficiency.
 
 ## Environmental Impact
 
@@ -202,11 +228,11 @@ Users (both direct and downstream) should be made aware of the risks, biases and
 
 Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
 
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
+- **Hardware Type:** A100 PCIE 40GB X1
+- **Hours used:** 18 Hours
+- **Cloud Provider:** Google Cloud
+- **Compute Region:** Asia-East-1
+- **Carbon Emitted:** 2.52 kg eq. CO2
 
 
 ## Citation [optional]