# How We Fine-Tuned a 0.5B Model to Handle Customer Support Tickets Using GRPO **OpenEnv × Scalar Hackathon** --- ## The Goal Fine-tune `Qwen2.5-0.5B-Instruct` — a tiny 0.5 billion parameter model — to act as a customer support agent using reinforcement learning. Actual RL: the model takes actions, gets scored, and learns purely from the reward signal. The environment: a live API that presents support tickets and grades the agent's responses across three progressively harder tasks. The challenge was making a model this small learn multi-step reasoning well enough to handle real customer scenarios. --- ## The Environment The [Support Ticket Environment](https://algocore-support-ticket-env.hf.space) presents tickets from five categories — billing, account, technical, refund, and general — and asks the agent to respond with a structured JSON action: ```json {"action_type": "classify", "category": "billing"} {"action_type": "reply", "reply_text": "We've processed your refund..."} {"action_type": "escalate", "reply_text": "Escalating to engineering."} {"action_type": "close", "reply_text": "Closing ticket."} ``` Three tasks of increasing difficulty, each returning a reward between 0.0 and 1.0 with no gradient signal from human labels: | Task | What it tests | |------|--------------| | **Task 1 — Classify** | Classify the ticket into the correct category | | **Task 2 — Classify + Act** | Classify first, then take the correct action (reply / escalate / close) | | **Task 3 — Full Resolution** | Classify, take the correct action, and write a quality reply addressing the specific issue | --- ## The Algorithm: GRPO We used **GRPO** (Group Relative Policy Optimization) via `trl.GRPOTrainer` — a modern RL algorithm designed specifically for language models: - Generates a **group** of completions for each prompt - Scores all completions with the reward function - Normalises advantages **within the group** — no separate value network needed - Applies PPO-style clipped ratio updates against a frozen reference model with KL penalty This makes GRPO significantly more memory-efficient than standard PPO for LLMs while delivering stable, consistent training. A 0.5B model + LoRA + GRPO fits comfortably on a single T4 GPU. **Key hyperparameters:** ``` Group size G = 4 KL beta = 0.04 LoRA rank = 16 (applied across all attention and MLP projection layers) Framework = HuggingFace transformers + peft (no Unsloth) ``` --- ## Reward Engineering: Where the Real Work Happened Before training, we did a thorough audit of the reward functions and built a precise, well-calibrated scoring system. This is what made the training signal reliable. ### Diverse ticket bank — 50 tickets across all categories We built a rich training set of 50 realistic support tickets spanning all five categories with varied scenarios, specific resolution hints, and a range of correct actions. This gave the model genuine breadth to learn from rather than a narrow set of patterns. ### Perfectly calibrated reward weights The Task 3 reward is a weighted sum across four components, tuned to sum to exactly 1.00: ``` 0.20 (classify) + 0.40 (action) + 0.25 (reply quality) + 0.15 (efficiency) = 1.00 ``` Every reward signal is clean, bounded, and meaningful. ### Two-tier reply quality scoring Reply quality uses a two-tier keyword system: ``` Category keywords (broad relevance) → +0.03 each Resolution hint keywords (ticket-specific) → +0.05 each ``` A generic reply scores lower than a reply that directly addresses the customer's specific issue — exactly the behaviour we want the model to learn. ### Classification credit carries forward In Task 2, a correct classification at step 0 earns 0.3 credit that is preserved into step 1. The model is rewarded for the full quality of its decision-making across both steps, not just the final action. ### Real correctness tracking in Task 3 Classification correctness is tracked precisely — the 0.20 bonus in Task 3 is only awarded when the model actually classifies correctly, creating a genuine learning signal for accurate categorisation. ### Format reward shaping A second reward function runs in parallel, reinforcing correct JSON structure: ``` Valid action_type → +0.15 bonus Valid action_type + valid category → +0.20 bonus Invalid action_type → −0.20 penalty ``` This kept the model outputting parseable, structured responses throughout training. --- ## Training Results Trained on 1,000 prompts, 3 epochs, Kaggle T4 ×2. Final loss: **0.0008**. | Task | Before GRPO | After GRPO | Delta | |------|-------------|------------|-------| | Task 1 — Classify | 0.667 | **1.000** | +0.333 | | Task 2 — Action | 0.117 | **0.450** | +0.333 | | Task 3 — Full Resolve | 0.083 | **0.258** | +0.175 | | **Overall** | **0.289** | **0.569** | **+0.280** | **Task 1 is fully solved** — the model classifies support tickets with perfect accuracy after training. Task 2 nearly quadrupled(four-fold) its score, showing the model learned both the classification and action selection steps reliably. Task 3 more than tripled its score, demonstrating that even a 0.5B model can learn to write contextually relevant replies when the reward signal is precise. Overall, the agent nearly doubled its score from 0.289 to 0.569 — a **+0.280 improvement purely from GRPO with no supervised labels**. - **Training Logs:** [WandB Run](https://wandb.ai/vighneshdev1990-/support-ticket-grpo/runs/33q716zb) --- ## What Made It Work Careful reward engineering mattered more than any hyperparameter choice. The key decisions: - **Resolution hints in scoring** — giving the reward function ticket-specific targets meant the model got a stronger signal for replies that actually resolved the customer's issue, not just topically relevant ones. - **Accumulated classification credit** — rewarding the full decision chain across steps encouraged coherent multi-step behaviour rather than optimising each step in isolation. - **Diverse ticket bank** — 50 varied tickets pushed the model to generalise across real support scenarios rather than overfit to a narrow distribution. - **Parallel reward functions** — running both the task reward and the format reward simultaneously gave the model signal on output quality and structural correctness at every step. --- ## Reproduce It - **Model:** [AlgoCore/support-ticket-grpo-model](https://huggingface.co/AlgoCore/support-ticket-grpo-model) - **Environment:** [algocore-support-ticket-env.hf.space](https://algocore-support-ticket-env.hf.space) - **Notebook:** Included in the HF repo — runs end-to-end on Kaggle T4 with `HF_TOKEN` and `WANDB_API_KEY` in Secrets.