noFFENSE/gpt2-355M-function-calling

GPT-2 355M fine-tuned for function calling

GPT-2 medium (355M) implemented from scratch in PyTorch — the architecture, weight loading from OpenAI's original TensorFlow checkpoints, training loop, and evaluation harness are all hand-written (no transformers, no PEFT) — and fine-tuned for one epoch on the full Glaive Function Calling v2 dataset (112,960 dialogs, Kaggle T4, ~9.4 h).

Given a JSON function schema and a user request, the model decides whether to call the function and emits a <functioncall> JSON payload:

###SYSTEM: You are a helpful assistant with access to the following functions. Use them if required -
{ "name": "get_current_weather", ... }
###USER: What's the weather like in Almaty right now?
###ASSISTANT: <functioncall> {"name": "get_current_weather", "arguments": '{"location": "Almaty"}'}

Try it live: Gradio Space — before vs after · Code & pipeline: GitHub · Write-up: project page

Files

File	What it is
gpt2-355M-function-calling.pth	the fine-tuned model (state_dict)
gpt2-355M-base.pth	pretrained GPT-2 355M converted to the same format — for before/after comparison without TensorFlow

The state_dicts target the hand-written GPTModel from the gpt2fc package, not transformers.GPT2LMHeadModel.

Usage

pip install "gpt2fc @ git+https://github.com/mron03/gpt2-function-calling" huggingface_hub

from huggingface_hub import hf_hub_download
from gpt2fc.config import get_device
from gpt2fc.inference.generate import get_tokenizer, load_finetuned_model, run_inference
from gpt2fc.inference.parser import extract_functioncall

path = hf_hub_download("noFFENSE/gpt2-355M-function-calling", "gpt2-355M-function-calling.pth")
device = get_device("auto")
model = load_finetuned_model(path, "355M", device)

prompt = (
    "###SYSTEM: You are a helpful assistant with access to the following functions. "
    'Use them if required -\n{"name": "get_current_weather", "parameters": '
    '{"type": "object", "properties": {"location": {"type": "string"}}}}\n'
    "###USER: What's the weather like in Almaty right now?"
)
reply = run_inference(model, get_tokenizer(), prompt, 64, device)
print(extract_functioncall(reply))
# {'name': 'get_current_weather', 'arguments': {'location': 'Almaty'}}

Evaluation

Full held-out test split: 3,929 well-formed dialogs, 1,856 of which have a function call in the ground truth — the denominator for all metrics. Strict scoring: an unparseable prediction counts as an error everywhere. (Run on a Kaggle T4 via cloud/kaggle_eval.ipynb from the repo.)

Metric	Value
Function-call parse rate	88.4%
Function name accuracy	88.0%
Argument keys accuracy	81.4%
Exact match (name + all argument values)	77.5%

Main failure mode: replying conversationally ("Sure, let me look that up") instead of emitting the call. Of the 1,641 parsed calls, all but ~8 named the correct function.

Quirks worth knowing

• The model reproduces Glaive's format faithfully — including its single-quoted arguments blob ("arguments": '{"k": "v"}'), which is not valid JSON. The gpt2fc parser handles the unwrapping.
• Single-turn only: SYSTEM + USER → ASSISTANT. It has not been trained on FUNCTION RESPONSE turns.
• It inherits dataset habits, e.g. politely refusing flight-booking requests, because Glaive's assistants always did.

Training

AdamW, lr 5e-5, weight decay 0.1, batch size 8, sequences capped at 512 tokens, cross-entropy on response tokens only (prompt and padding masked with ignore_index=-100). Architecture learned from Sebastian Raschka's Build a Large Language Model From Scratch; the function-calling extension, training runs, and evaluation harness are my own.