Update README.md

README.md

@@ -1,199 +1,225 @@
 ---
 library_name: transformers
-tags: []
+license: mit
+datasets:
+- gretelai/synthetic_text_to_sql
+pipeline_tag: text-generation
 ---
 
-# Model Card for Model ID
-
-<!-- Provide a quick summary of what the model is/does. -->
-
-
+# Model Card for LLaMA 3.2 3B Instruct Text2SQL
 
 ## Model Details
 
 ### Model Description
 
-<!-- Provide a longer summary of what this model is. -->
-
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
+This is a fine-tuned version of LLaMA 3.2 3B Instruct model, specifically optimized for Text-to-SQL generation tasks. The model has been trained to convert natural language queries into structured SQL commands.
 
-### Model Sources [optional]
+- **Developed by:** Zhafran Ramadhan
+- **Model type:** Decoder-only Language Model
+- **Language(s):** English - MultiLingual
+- **License:** MIT
+- **Finetuned from model:** LLaMA 3.2 3B Instruct
 
-<!-- Provide the basic links for the model. -->
+### Model Sources
+- **Repository:** https://wandb.ai/zhafranr/LLaMA_3-2_3B_Instruct_FineTune_Text2SQL
+- **Dataset:** https://huggingface.co/datasets/gretelai/synthetic_text_to_sql
 
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+## How to Get Started with the Model
 
+### Installation
+```python
+pip install transformers torch
+```
+
+### Input Format and Usage
+The model expects input in a specific format following this template:
+```text
+<|begin_of_text|><|start_header_id|>system<|end_header_id|>
+
+[System context and database schema]
+
+<|eot_id|><|start_header_id|>user<|end_header_id|>
+
+[User query]
+
+<|eot_id|><|start_header_id|>assistant<|end_header_id|>
+```
+
+### Basic Usage
+```python
+from transformers import pipeline
+
+# Initialize the pipeline
+generator = pipeline(
+    "text-generation",
+    model="[YOUR_HUGGINGFACE_MODEL_ID]",  # Replace with your model ID
+    torch_dtype=torch.float16,
+    device_map="auto"
+)
+
+def generate_sql_query(context, question):
+    # Format the prompt according to the training template
+    prompt = f"""<|begin_of_text|><|start_header_id|>system<|end_header_id|>
+
+Cutting Knowledge Date: December 2023
+Today Date: 07 Nov 2024
+
+You are a specialized SQL query generator focused solely on the provided RAG database. Your tasks are:
+1. Generate SQL queries based on user requests that are related to querying the RAG database.
+2. Only output the SQL query itself, without any additional explanation or commentary.
+3. Use the context provided from the RAG database to craft accurate queries.
+
+Context: {context}
+<|eot_id|><|start_header_id|>user<|end_header_id|>
+
+{question}<|eot_id|><|start_header_id|>assistant<|end_header_id|>"""
+
+    response = generator(
+        prompt,
+        max_length=500,
+        num_return_sequences=1,
+        temperature=0.1,
+        do_sample=True,
+        pad_token_id=generator.tokenizer.eos_token_id
+    )
+    
+    return response[0]['generated_text']
+
+# Example usage
+context = """CREATE TABLE upgrades (id INT, cost FLOAT, type TEXT);
+INSERT INTO upgrades (id, cost, type) VALUES 
+(1, 500, 'Insulation'), 
+(2, 1000, 'HVAC'), 
+(3, 1500, 'Lighting');"""
+
+questions = [
+    "Find the energy efficiency upgrades with the highest cost and their types.",
+    "Show me all upgrades costing less than 1000 dollars.",
+    "Calculate the average cost of all upgrades."
+]
+
+for question in questions:
+    sql = generate_sql_query(context, question)
+    print(f"\nQuestion: {question}")
+    print(f"Generated SQL: {sql}\n")
+```
+
+### Advanced Usage with Custom System Prompt
+```python
+def generate_sql_with_custom_prompt(context, question, custom_system_prompt=""):
+    base_prompt = """<|begin_of_text|><|start_header_id|>system<|end_header_id|>
+
+Cutting Knowledge Date: December 2023
+Today Date: 07 Nov 2024
+
+You are a specialized SQL query generator focused solely on the provided RAG database."""
+
+    full_prompt = f"""{base_prompt}
+{custom_system_prompt}
+
+Context: {context}
+<|eot_id|><|start_header_id|>user<|end_header_id|>
+
+{question}<|eot_id|><|start_header_id|>assistant<|end_header_id|>"""
+
+    response = generator(
+        full_prompt,
+        max_length=500,
+        num_return_sequences=1,
+        temperature=0.1,
+        do_sample=True,
+        pad_token_id=generator.tokenizer.eos_token_id
+    )
+    
+    return response[0]['generated_text']
+```
+
+### Best Practices
+1. **Input Formatting**:
+   - Always include the special tokens (<|begin_of_text|>, <|eot_id|>, etc.)
+   - Provide complete database schema in context
+   - Keep questions clear and focused on data retrieval
+
+2. **Parameter Configuration**:
+   - Use temperature=0.1 for consistent SQL generation
+   - Adjust max_length based on expected query complexity
+   - Enable do_sample for more natural completions
+
+3. **Context Management**:
+   - Include relevant table schemas
+   - Provide sample data when needed
+   - Keep context concise but complete
+     
 ## Uses
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
 ### Direct Use
-
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
-
-### Downstream Use [optional]
-
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-
-[More Information Needed]
+The model is designed for converting natural language questions into SQL queries. It can be used for:
+- Database query generation from natural language
+- SQL query assistance
+- Data analysis automation
 
 ### Out-of-Scope Use
-
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-
-[More Information Needed]
-
-## Bias, Risks, and Limitations
-
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-
-[More Information Needed]
-
-### Recommendations
-
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-
-## How to Get Started with the Model
-
-Use the code below to get started with the model.
-
-[More Information Needed]
+- Production deployment without human validation
+- Critical decision-making without human oversight
+- Direct database execution without query validation
 
 ## Training Details
 
 ### Training Data
-
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
+- Dataset: gretelai/synthetic_text_to_sql
+- Data preprocessing: Standard text-to-SQL formatting
 
 ### Training Procedure
 
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
-
-
 #### Training Hyperparameters
-
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-
-#### Speeds, Sizes, Times [optional]
-
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-
-[More Information Needed]
+- **Total Steps:** 4,149
+- **Final Training Loss:** 0.1168
+- **Evaluation Loss:** 0.2125
+- **Learning Rate:** Dynamic with final LR = 0
+- **Epochs:** 2.99
+- **Gradient Norm:** 1.3121
+
+#### Performance Metrics
+- **Training Samples/Second:** 6.291
+- **Evaluation Samples/Second:** 19.325
+- **Steps/Second:** 3.868
+- **Total FLOPS:** 1.92e18
+
+#### Training Infrastructure
+- **Hardware:** Single NVIDIA H100 GPU
+- **Training Duration:** 5-6 hours
+- **Total Runtime:** 16,491.75 seconds
+- **Model Preparation Time:** 0.0051 seconds
 
 ## Evaluation
 
-<!-- This section describes the evaluation protocols and provides the results. -->
-
-### Testing Data, Factors & Metrics
-
-#### Testing Data
-
-<!-- This should link to a Dataset Card if possible. -->
+### Metrics
+The model's performance was tracked using several key metrics:
+- **Training Loss:** Started at ~1.2, converged to 0.1168
+- **Evaluation Loss:** 0.2125
+- **Processing Efficiency:** 19.325 samples per second during evaluation
 
-[More Information Needed]
-
-#### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
-
-#### Metrics
-
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
-
-### Results
-
-[More Information Needed]
-
-#### Summary
-
-
-
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
+### Results Summary
+- Achieved stable convergence after ~4000 steps
+- Maintained consistent performance metrics throughout training
+- Shows good balance between training and evaluation loss
 
 ## Environmental Impact
 
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+- **Hardware Type:** NVIDIA H100 GPU
+- **Hours used:** ~6 hours
+- **Training Location:** [User to specify]
 
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-
-## Technical Specifications [optional]
-
-### Model Architecture and Objective
-
-[More Information Needed]
+## Technical Specifications
 
 ### Compute Infrastructure
-
-[More Information Needed]
-
-#### Hardware
-
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
-
-## Citation [optional]
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
-**BibTeX:**
-
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
-
-## Glossary [optional]
-
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
-
-## More Information [optional]
-
-[More Information Needed]
-
-## Model Card Authors [optional]
-
-[More Information Needed]
+- **GPU:** NVIDIA H100
+- **Training Duration:** 5-6 hours
+- **Total Steps:** 4,149
+- **FLOPs Utilized:** 1.92e18
 
 ## Model Card Contact
+[Contact information to be added by Zhafran Ramadhan]
 
-[More Information Needed]
+---
+*Note: This model card follows the guidelines set by the ML community for responsible AI development and deployment.*