Update README.md

README.md

@@ -1,199 +1,186 @@
 ---
 library_name: transformers
-tags: []
+tags:
+- LLMs-Aviation
+- AI-Regulatory-Compliance
+- RAC-AI-Colombia
+license: apache-2.0
+datasets:
+- somosnlp/Reglamento_aeronautico_Colombiano_FULL_ChatML_format_gemma
+language:
+- es
 ---
 
-# Model Card for Model ID
 
-<!-- Provide a quick summary of what the model is/does. -->
+# Model Card for Gemma-FULL-RAC-Colombia 🛫
 
+<p align="center">
+  <img src="https://cdn-uploads.huggingface.co/production/uploads/6419c2f6b4adb0e101b17b6c/JlVz94BKdUkRN8xU00EgT.png" alt="Model Illustration" width="500">
+</p>
 
+🌟 This model card proudly presents Gemma-FULL-RAC-Colombia, a groundbreaking language model fine-tuned with precision on the Colombian Aeronautical Regulations (Reglamento Aeronáutico Colombiano, RAC) dataset. Harnessing the advanced capabilities of Google's original Gemma model, this adaptation focuses keenly on the aviation regulatory sphere within Colombia.
 
-## Model Details
+## Model Details 📈
 
-### Model Description
+### Model Description 📘
 
-<!-- Provide a longer summary of what this model is. -->
+🚀 The Gemma-FULL-RAC-Colombia model marks an innovative leap in deploying language models for niche domains like aviation law and regulation. Developed by Edison Bejarano and Nicolai Potes, it's finely tuned using LoRA (Low-Rank Adaptation), ensuring minimal adjustment to the model's weights while preserving the extensive knowledge and prowess of Google's Gemma. This meticulous approach guarantees not just the retention of broad language comprehension and generation skills but also a profound grasp of the terminologies, concepts, and regulatory frameworks particular to Colombian aviation.
 
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+- **Developed by:** [Edison Bejarano](https://huggingface.co/ejbejaranos) and [Nicolai Potes](https://huggingface.co/NickyNicky) ✨
+- **Funded by:** Fundación Universitaria Los Libertadores 🏛️
+- **Shared by [optional]:** This project is a collective endeavor of passionate NLP and aviation regulation specialists, united by a vision to make aviation laws more accessible and understandable. 👥
+- **Model type:** Causal Language Model, specially fine-tuned with LoRA for the distinct area of Colombian Aeronautical Regulations. 📊
+- **Language(s) (NLP):** Spanish, tailored for the Colombian legal and regulatory context. 🇨🇴
+- **License:** Apache License. This open-source license ensures that the model can be freely used, modified, and distributed. Please check the Hugging Face model page for specific licensing details. 🔓
+- **Finetuned from model [optional]:** Derived from Google's Gemma model, utilizing versions such as `gemma-2b-it` for comprehensive training. 🧠
 
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
+### Loss Function through Epochs 📉
+<p align="center">
+  <img src="https://cdn-uploads.huggingface.co/production/uploads/6419c2f6b4adb0e101b17b6c/vufJ_u_5ZrdzNcSrqwjBx.png" alt="Loss Function Graph" width="500">
+</p>
 
-### Model Sources [optional]
 
-<!-- Provide the basic links for the model. -->
+### Model Sources 📚
 
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+- **Repository:** To be added
+- **Paper [optional]:** To be added
+- **Demo [optional]:** To be added
 
-## Uses
+## Uses 🛠️
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+The Gemma-FULL-RAC-Colombia model is designed to enhance the understanding and application of the Colombian Aeronautical Regulations (RAC) through natural language processing. It's tailored for professionals and enthusiasts in the aviation industry, regulatory agencies, legal experts, and AI researchers with an interest in domain-specific language model applications.
 
-### Direct Use
+### Direct Use 🎯
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+The model can be directly used to generate text, answer questions, and provide insights related to the Colombian Aeronautical Regulations without further fine-tuning. It's ideal for creating educational content, simplifying legal language, and assisting in regulatory compliance efforts.
 
-[More Information Needed]
+### Downstream Use [optional] 🔄
 
-### Downstream Use [optional]
+When fine-tuned, Gemma-FULL-RAC-Colombia can be integrated into larger systems for automated compliance checks, document summarization, and even training simulators for pilots and air traffic controllers, offering a deeper, contextual understanding of regulations.
 
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+### Out-of-Scope Use 🚫
 
-[More Information Needed]
+Misuse includes any application that promotes unethical practices, misinterprets aviation law, or uses the model for malicious purposes. The model is not designed for navigational purposes or to replace professional legal advice.
 
-### Out-of-Scope Use
+## Bias, Risks, and Limitations ⚠️
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+The model, while powerful, has limitations inherent to AI, including biases present in the training data. It may not cover all nuances of aviation regulations outside of Colombia or adapt to changes in law without updates.
 
-[More Information Needed]
+### Recommendations 📝
 
-## Bias, Risks, and Limitations
+Users should verify model outputs against current regulations and consult with professionals for critical applications. Awareness of the model's scope and limitations is crucial for effective use.
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+## How to Get Started with the Model 🚀
 
-[More Information Needed]
+```python
+# Assuming Hugging Face's Transformers and Datasets are installed
+from transformers import AutoModelForCausalLM, AutoTokenizer
 
-### Recommendations
+model_name = "somosnlp/gemma-FULL-RAC-Colombia"
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForCausalLM.from_pretrained(model_name)
 
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+# Example usage
+input_text = "Explain the regulations for drone flights in Colombia."
+input_ids = tokenizer.encode(input_text, return_tensors="pt")
+output = model.generate(input_ids, max_length=100)
+print(tokenizer.decode(output[0], skip_special_tokens=True))
+```
 
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+## Training Details 🧠
 
-## How to Get Started with the Model
+The Gemma-FULL-RAC-Colombia model was specifically fine-tuned on the "Reglamento Aeronáutico Colombiano FULL ChatML format gemma" dataset available through Hugging Face Datasets. This dataset encompasses a comprehensive collection of the Colombian Aeronautical Regulations (RAC), tailored to train the model for understanding and generating responses related to aviation regulations in Colombia.
 
-Use the code below to get started with the model.
+### Environment and Libraries 🖥️
 
-[More Information Needed]
+The training was executed in a Python environment leveraging the following main libraries:
 
-## Training Details
+- `transformers` for model loading and fine-tuning facilities.
+- `datasets` for handling the dataset efficiently.
+- `torch` for the underlying deep learning framework.
+- `peft` for applying the Low-Rank Adaptation (LoRA) adjustments to the model.
 
-### Training Data
+### Model Architecture 🏗️
 
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+The base model is Google's Gemma, an advanced language model with billions of parameters, capable of understanding context and generating human-like text. For this project, specific versions of Gemma (e.g., `gemma-2b-it`) were used as starting points.
 
-[More Information Needed]
+### Fine-tuning with LoRA 🔧
 
-### Training Procedure
+LoRA (Low-Rank Adaptation) was employed to adapt the Gemma model to the specific domain of Colombian Aeronautical Regulations without extensively retraining the entire model. This approach allows for the efficient update of the model's weights, focusing on the adaptation layers to integrate the domain-specific knowledge effectively.
 
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+- **LoRA Configurations:** r=64, lora_alpha=32, among other settings tailored to balance performance and efficiency.
 
-#### Preprocessing [optional]
+### Training Process 🚀
 
-[More Information Needed]
+Training was carried out considering the following key parameters:
 
+- Batch size: Adapted based on the available computational resources.
+- Learning rate: Fine-tuned for optimal convergence without overfitting.
+- Epochs: The model was trained for a sufficient number of epochs to ensure that it adequately learned from the dataset, with early stopping criteria to prevent overfitting.
 
-#### Training Hyperparameters
+### Evaluation and Performance 📊
+'''Pending'''
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+### Model Merging and Saving 💾
 
-#### Speeds, Sizes, Times [optional]
+After fine-tuning, the LoRA-adjusted weights were merged back with the base Gemma model to create the final version of Gemma-FULL-RAC-Colombia. The model was then saved and made available through Hugging Face, ensuring easy access for further use and development.
 
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+The training process was meticulously documented to ensure reproducibility and facilitate further research and development in the domain of specialized language models for regulatory compliance and legal tech applications.
 
-[More Information Needed]
 
-## Evaluation
+### Training Data 📚
 
-<!-- This section describes the evaluation protocols and provides the results. -->
+The training data utilized for fine-tuning the Gemma-FULL-RAC-Colombia model is available on Hugging Face Datasets under the dataset identifier:
 
-### Testing Data, Factors & Metrics
+[somosnlp/Reglamento_aeronautico_Colombiano_FULL_ChatML_format_gemma](https://huggingface.co/datasets/somosnlp/Reglamento_aeronautico_Colombiano_FULL_ChatML_format_gemma)
 
-#### Testing Data
+This dataset comprises a targeted collection of the Colombian Aeronautical Regulations (RAC), specifically including regulations from RAC 1 to RAC 5. The selection is optimized for chat and language modeling tasks, aiming to train the model to understand and generate accurate responses related to the initial segments of Colombian aviation regulations.
 
-<!-- This should link to a Dataset Card if possible. -->
 
-[More Information Needed]
+### Training Procedure
 
-#### Factors
+The Gemma-FULL-RAC-Colombia model underwent a meticulous training process, leveraging the NVIDIA V100 GPU for its computational needs. The procedure was designed to fine-tune the model on a dataset comprising sections RAC 1 to RAC 5 of the Colombian Aeronautical Regulations.
 
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+#### Preprocessing
 
-[More Information Needed]
+The dataset was preprocessed to convert the regulatory texts into a format suitable for the model, utilizing the tokenizer associated with the `google/gemma-2b-it` model. This step ensured that the input data was optimized for training efficiency and model comprehension.
 
-#### Metrics
+#### Training Hyperparameters
 
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+The training utilized a mixed precision regime to balance computational efficiency and model accuracy. Specific hyperparameters included:
 
-[More Information Needed]
+- **Training regime:** Mixed precision (FP16)
 
-### Results
+#### Speeds, Sizes, Times
 
-[More Information Needed]
+The model was trained over a span of approximately 4.67 hours (runtime: 16,020 seconds), with checkpoints indicating progress at various intervals. The training demonstrated efficient utilization of the V100 GPU, maintaining a steady throughput.
 
-#### Summary
+#### Factors
 
+Training focused on the model's ability to understand and generate text related to the specific domain of Colombian Aeronautical Regulations.
 
+#### Metrics
 
-## Model Examination [optional]
+The primary metric for evaluating the model's performance was the training loss, which was observed to decrease steadily over time, indicating effective learning and adaptation by the model.
 
-<!-- Relevant interpretability work for the model goes here -->
+### Results
 
-[More Information Needed]
+The training process resulted in a model capable of accurately interpreting and responding to queries related to the Colombian Aeronautical Regulations, with a notable final training loss of 0.6204 at the last checkpoint.
 
 ## Environmental Impact
 
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-
-## Technical Specifications [optional]
-
-### Model Architecture and Objective
-
-[More Information Needed]
-
-### Compute Infrastructure
-
-[More Information Needed]
-
-#### Hardware
-
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
-
-## Citation [optional]
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
-**BibTeX:**
-
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
-
-## Glossary [optional]
-
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
-
-## More Information [optional]
-
-[More Information Needed]
+Given the use of an NVIDIA V100 GPU for approximately 4.67 hours, the carbon emissions can be estimated using the Machine Learning Impact calculator. This tool accounts for the hardware type, runtime, and other factors to provide a comprehensive view of the environmental impact of training large AI models.
 
-## Model Card Authors [optional]
+- **Hardware Type:** NVIDIA V100 GPU
+- **Hours used:** ~4.67
+- **Carbon Emitted:** 554.5625
 
-[More Information Needed]
+# Constants
+power_consumption_kW = 0.25  # 250 watts in kW
+runtime_hours = 4.67
+carbon_intensity_gCO2eq_per_kWh = 475  # Global average carbon intensity
 
-## Model Card Contact
+# Calculate carbon emissions
+carbon_emitted_gCO2eq = power_consumption_kW * runtime_hours * carbon_intensity_gCO2eq_per_kWh
 
-[More Information Needed]
+carbon_emitted_gCO2eq = 554.5625