Tiny Aya Global Tool-Calling Adapter

This repository contains a PEFT LoRA adapter fine-tuned from CohereLabs/tiny-aya-global. It is an adapter, not a standalone copy of the base model. The base model must be loaded when using this adapter.

Training

The adapter was trained for 500 optimizer steps using TRL SFTTrainer and 4-bit NF4 QLoRA. Configuration: learning rate 1e-4, LoRA rank 16, alpha 32, dropout 0.05, per-device batch size 2, gradient accumulation 4, BF16 compute, gradient checkpointing, and maximum sequence length 8192.

The training pool was prepared from Salesforce xLAM Function Calling 60K, Team-ACE ToolACE, and Salesforce APIGen-MT-5K. The prepared training split contains 5,000 xLAM, 2,500 ToolACE, and 1,000 APIGen-MT examples. The run stopped after 500 steps, so it did not necessarily consume every training example. Dataset rows and test predictions are not included in this repository.

Inference

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

base_id = "CohereLabs/tiny-aya-global"
adapter_id = "skshreyas714/tiny-aya-global-tool-calling-500steps"
tokenizer = AutoTokenizer.from_pretrained(adapter_id)
compute_dtype = torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float16
quantization = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=compute_dtype,
)
base = AutoModelForCausalLM.from_pretrained(
    base_id,
    device_map={"": 0},
    dtype=compute_dtype,
    quantization_config=quantization,
    attn_implementation="sdpa",
)
model = PeftModel.from_pretrained(base, adapter_id).eval()
messages = [{"role": "user", "content": "Find movie theaters near Los Angeles."}]
tools = [{"type": "function", "function": {"name": "search_movie_theaters", "description": "Find nearby movie theaters", "parameters": {"type": "object", "properties": {"latitude": {"type": "number"}, "longitude": {"type": "number"}, "distance": {"type": "integer"}}, "required": ["latitude", "longitude", "distance"]}}}]
inputs = tokenizer.apply_chat_template(
    messages, tools=tools, add_generation_prompt=True,
    tokenize=True, return_dict=True, return_tensors="pt"
).to(model.device)
with torch.inference_mode():
    output = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=False))

License and use

The base Tiny Aya model is under Cohere Labs' CC-BY-NC license and acceptable-use policy. The training mixture includes APIGen-MT, whose dataset card specifies CC BY-NC 4.0 and additional use restrictions concerning GPT-generated data. Review and comply with the base-model and every source-dataset license before use. This adapter is intended for non-commercial research; do not assume commercial-use rights.

The generated tool call is a model output. Validate it against the application's tool schema and authorization rules before execution.

Downloads last month
47
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for skshreyas714/tiny-aya-global-tool-calling-500steps

Adapter
(395)
this model