semantixai
/

Lloro-SQL

@@ -1,26 +1,28 @@
 ---
 language:
 - pt
-license: llama3
-library_name: transformers
 tags:
 - code
 - sql
 - finetuned
 - portugues-BR
-base_model: meta-llama/Meta-Llama-3-8B-Instruct
 ---
-**Lloro SQL**
 <img src="https://cdn-uploads.huggingface.co/production/uploads/653176dc69fffcfe1543860a/h0kNd9OTEu1QdGNjHKXoq.png" width="300" alt="Lloro-7b Logo"/>
 Lloro SQL, developed by Semantix Research Labs, is a language Model that was trained to effectively transform Portuguese queries into SQL Code. It is a fine-tuned version of meta-llama/Meta-Llama-3-8B-Instruct,  that was trained on GretelAI public datasets.  The fine-tuning process was performed using the QLORA metodology on a GPU A100 with 40 GB of RAM.
-**Model description**
 Model type: A 7B parameter  fine-tuned on GretelAI public datasets.
@@ -28,10 +30,7 @@ Language(s) (NLP): Primarily Portuguese, but the model is capable to understand
 Finetuned from model: meta-llama/Meta-Llama-3-8B-Instruct
-**What is Lloro's intended use(s)?**
 Lloro is built for Text2SQL in Portuguese contexts .
@@ -39,9 +38,7 @@ Input : Text
 Output : Text (Code)
-**Usage**
 Using an OpenAI compatible inference server (like [vLLM](https://docs.vllm.ai/en/latest/index.html))
@@ -66,55 +63,69 @@ def generate_responses(instruction, client=client):
 output = generate_responses(user_prompt)
 ```
-**Params**
 Training Parameters
-| Params                           | Training Data                   | Examples                        | Tokens     | LR     |
-|----------------------------------|---------------------------------|---------------------------------|------------|--------|
-| 8B                               | GretelAI public datasets        | 65000                           | 18.000.000 | 9e-5   |
-**Model Sources**
-GretelAI: https://huggingface.co/datasets/gretelai/synthetic_text_to_sql
-**Performance**
-| Modelo         | LLM as Judge | Code Bleu Score | Rouge-L |  CodeBert- Precision | CodeBert-Recall | CodeBert-F1 | CodeBert-F3 |
 |----------------|--------------|-----------------|---------|----------------------|-----------------|-------------|-------------|
-| Llama 3 - Base | 65.48%       | 0.4583          | 0.6361  |  0.8815              | 0.8871          | 0.8835      | 0.8862      |
-| Llama 3 - FT   | 62.57%       | 0.6512          | 0.7965  |  0.9458              | 0.9469          | 0.9459      | 0.9466      |
-**Training Infos:**
 The following hyperparameters were used during training:
 | Parameter                 | Value                |
 |---------------------------|----------------------|
-| learning_rate             | 1e-4                 |
-| weight_decay              | 0.001               |
-| train_batch_size          | 16                    |
-| eval_batch_size           | 8                   |
 | seed                      | 42                   |
-| optimizer                 | Adam - adamw_8bit |
 | lr_scheduler_type         | cosine               |
-| num_epochs                | 3.0                  |
-**QLoRA hyperparameters**
 The following parameters related with the Quantized Low-Rank Adaptation and Quantization were used during training:
 | Parameter       | Value   |
 |-----------------|---------|
-| lora_r          | 16      |
-| lora_alpha      | 64      |
 | lora_dropout    | 0       |
-**Framework versions**
 | Library       | Version   |
 |---------------|-----------|
 | accelerate    | 0.21.0    |
@@ -127,4 +138,4 @@ The following parameters related with the Quantized Low-Rank Adaptation and Quan
 | scikit-learn  | 1.3.2     |
 | Tokenizers    | 0.14.1    |
 | Transformers  | 4.37.2    |
-| trl           | 0.4.7     |

 ---
+library_name: transformers
+base_model: meta-llama/Meta-Llama-3-8B-Instruct
+license: llama3
 language:
 - pt
 tags:
 - code
 - sql
 - finetuned
 - portugues-BR
+co2_eq_emissions:
+  emissions: 1450
+  source: "Lacoste, Alexandre, et al. “Quantifying the Carbon Emissions of Machine Learning.” ArXiv (Cornell University), 21 Oct. 2019, https://doi.org/10.48550/arxiv.1910.09700."
+  training_type: "fine-tuning"
+  geographical_location: "Council Bluffs, Iowa, USA."
+  hardware_used: "1 A100 40GB GPU"
 ---
+# Lloro SQL
 <img src="https://cdn-uploads.huggingface.co/production/uploads/653176dc69fffcfe1543860a/h0kNd9OTEu1QdGNjHKXoq.png" width="300" alt="Lloro-7b Logo"/>
 Lloro SQL, developed by Semantix Research Labs, is a language Model that was trained to effectively transform Portuguese queries into SQL Code. It is a fine-tuned version of meta-llama/Meta-Llama-3-8B-Instruct,  that was trained on GretelAI public datasets.  The fine-tuning process was performed using the QLORA metodology on a GPU A100 with 40 GB of RAM.
+## Model description
 Model type: A 7B parameter  fine-tuned on GretelAI public datasets.
 Finetuned from model: meta-llama/Meta-Llama-3-8B-Instruct
+## What is Lloro's intended use(s)?
 Lloro is built for Text2SQL in Portuguese contexts .
 Output : Text (Code)
+## Usage
 Using an OpenAI compatible inference server (like [vLLM](https://docs.vllm.ai/en/latest/index.html))
 output = generate_responses(user_prompt)
 ```
+## Params
 Training Parameters
+| Params                           | Training Data                             | Examples                        | Tokens     | LR     |
+|----------------------------------|-------------------------------------------|---------------------------------|------------|--------|
+| 8B                               | GretelAI public datasets + Synthetic Data | 102970                          | 18.654.222 | 2e-4   |
+## Model Sources
+GretelAI: <https://huggingface.co/datasets/gretelai/synthetic_text_to_sql>
+## Performance
+### Test Dataset
+| Model          | LLM as Judge | Code Bleu Score | Rouge-L |  CodeBert- Precision | CodeBert-Recall | CodeBert-F1 | CodeBert-F3 |
 |----------------|--------------|-----------------|---------|----------------------|-----------------|-------------|-------------|
+| Llama 3 8B     | 65.48%       | 0.4583          | 0.6361  |  0.8815              | 0.8871          | 0.8835      | 0.8862      |
+| Lloro - SQL    | 71.33%       | 0.6512          | 0.7965  |  0.9458              | 0.9469          | 0.9459      | 0.9466      |
+| GPT - 3.5 Turbo| 67.52%       | 0.6232          | 0.9967  |  0.9151              | 0.9152          | 0.9142      | 0.9175      |
+### Database Benchmark
+| Model          | Score        |
+|----------------|--------------|
+| Llama 3 - Base | 35.55%       |
+| Lloro - SQL    | 49.48%       |
+| GPT - 3.5 Turbo| 46.15%       |
+## Training Infos
 The following hyperparameters were used during training:
 | Parameter                 | Value                |
 |---------------------------|----------------------|
+| learning_rate             | 2e-4                 |
+| weight_decay              | 0.001                |
+| train_batch_size          | 16                   |
+| eval_batch_size           | 8                    |
 | seed                      | 42                   |
+| optimizer                 | Adam - adamw_8bit    |
 | lr_scheduler_type         | cosine               |
+| num_epochs                | 4.0                  |
+## QLoRA hyperparameters
 The following parameters related with the Quantized Low-Rank Adaptation and Quantization were used during training:
 | Parameter       | Value   |
 |-----------------|---------|
+| lora_r          | 64      |
+| lora_alpha      | 128     |
 | lora_dropout    | 0       |
+## Experiments
+| Model                 | Epochs | Overfitting | Final Epochs | Training Hours  | CO2 Emission (Kg) |
+|-----------------------|--------|-------------|--------------|-----------------|-------------------|
+| Llama 3 8B Instruct   | 5      | Yes         | 4            | 10.16           | 1.45              |
+## Framework versions
 | Library       | Version   |
 |---------------|-----------|
 | accelerate    | 0.21.0    |
 | scikit-learn  | 1.3.2     |
 | Tokenizers    | 0.14.1    |
 | Transformers  | 4.37.2    |
+| trl           | 0.4.7     |