|
|
--- |
|
|
base_model: lapa-llm/lapa-v0.1.2-instruct |
|
|
library_name: transformers |
|
|
model_name: Lapa-function-calling |
|
|
tags: |
|
|
- generated_from_trainer |
|
|
- trl |
|
|
- sft |
|
|
licence: license |
|
|
license: gemma |
|
|
datasets: |
|
|
- lmassaron/hermes-function-calling-v1 |
|
|
language: |
|
|
- uk |
|
|
- en |
|
|
--- |
|
|
# Model Card for Lapa-function-calling |
|
|
|
|
|
This model is a fine-tuned version of [lapa-llm/lapa-v0.1.2-instruct](https://huggingface.co/lapa-llm/lapa-v0.1.2-instruct). |
|
|
It has been trained using [TRL](https://github.com/huggingface/trl). |
|
|
|
|
|
# Special Thanks |
|
|
The support of vast.ai team allowed this fine tune to happen. Many thanks! |
|
|
|
|
|
# Evaluation |
|
|
This is the first iteration of fine-tuning Lapa for function calling. In the future, we plan to add metrics and improve training. <br> |
|
|
During this phase new tokens (including tool_call) were introduced to the model and we evaluated how well it uses and understands the purpose of tool_call.<br> |
|
|
|
|
|
# Metrics |
|
|
Accuracy in function calling (if response contains tool_call token) - find_longest_common_sequence_length(ground_truth_tokens, generated_tokens) / len(ground_truth_tokens)<br> |
|
|
Match in helpful exchange (if response does not contain tool_call token) - Computes the percentage of matching elements between generated tokens and ground truth tokens<br> |
|
|
## Performance before fine-tuning: |
|
|
Accuracy in function calling: 0.48022<br> |
|
|
Match in helpful exchange: 0.09064<br> |
|
|
## Performance after fine-tuning: |
|
|
Accuracy in function calling: 0.94833<br> |
|
|
Match in helpful exchange: 0.09829 |
|
|
# Quick start |
|
|
``` |
|
|
from peft import PeftModel, PeftConfig |
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
peft_model_id = "TymofiiNasobko/Lapa-function-calling" |
|
|
peftconfig = PeftConfig.from_pretrained(peft_model_id) |
|
|
model = AutoModelForCausalLM.from_pretrained( |
|
|
peftconfig.base_model_name_or_path, |
|
|
attn_implementation="eager", |
|
|
device_map=device, |
|
|
) |
|
|
tokenizer = AutoTokenizer.from_pretrained(peft_model_id) |
|
|
model.resize_token_embeddings(len(tokenizer)) |
|
|
model = PeftModel.from_pretrained(model, peft_model_id) |
|
|
model = model.to(compute_dtype) |
|
|
model = model.eval() |
|
|
``` |
|
|
## Training procedure |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This model was trained with SFT. |
|
|
|
|
|
### Framework versions |
|
|
|
|
|
- TRL: 0.25.1 |
|
|
- Transformers: 4.57.1 |
|
|
- Pytorch: 2.9.1 |
|
|
- Datasets: 4.4.1 |
|
|
- Tokenizers: 0.22.1 |
|
|
|
|
|
## Citations |
|
|
|
|
|
|
|
|
|
|
|
Cite TRL as: |
|
|
|
|
|
```bibtex |
|
|
@misc{vonwerra2022trl, |
|
|
title = {{TRL: Transformer Reinforcement Learning}}, |
|
|
author = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallou{\'e}dec}, |
|
|
year = 2020, |
|
|
journal = {GitHub repository}, |
|
|
publisher = {GitHub}, |
|
|
howpublished = {\url{https://github.com/huggingface/trl}} |
|
|
} |
|
|
``` |
