README.md

---
license: other
license_name: llama3.2
license_link: https://www.llama.com/llama3_2/license/
base_model: meta-llama/Llama-3.2-1B
library_name: peft
tags:
- lora
- peft
- text-classification
- customer-support
language:
- en
pipeline_tag: text-classification
---

# Model Card: Ticket Classifier (Llama-3.2-1B + LoRA)

## Model Description

A LoRA adapter fine-tuned on top of `meta-llama/Llama-3.2-1B` for support ticket classification.

## Intended Use

Classify customer support tickets into one of five categories:
- `billing` — payment, invoice, charge, refund queries
- `technical` — bugs, errors, product not working
- `account` — login, password, profile, access issues
- `shipping` — delivery, tracking, lost package queries
- `general` — feedback, feature requests, and inquiries that don't fit a specific support category

**Out-of-scope use:** This model is not intended for legal- or compliance-sensitive ticket routing without human review, and has not been evaluated on real customer data or non-English tickets.

## Training Data

600 synthetic support tickets generated by `claude-sonnet-4-6` — 120 per category. Tickets vary in length (1–5 sentences) and tone (frustrated, polite, confused, urgent). No real customer data was used.

Generation used explicit per-category instructions (rather than a single generic prompt) so that each category — including `general` — has a clear positive definition. An earlier version of this dataset used an undefined `general` category, which caused the generation model to fill it with tickets indistinguishable from the other four categories; this was corrected before the final training run below.

**Known limitation:** Class balance (120/category) is artificial. Real-world ticket distributions are rarely this even, so reported metrics may not reflect performance under real class imbalance.

## Training Details

| Parameter | Value |
|---|---|
| Base model | meta-llama/Llama-3.2-1B |
| Method | QLoRA (4-bit) |
| LoRA rank | 16 |
| LoRA alpha | 32 |
| Target modules | q_proj, v_proj |
| LoRA dropout | 0.05 |
| Training hardware | Google Colab T4 (free tier) |
| Epochs | 2 |
| Learning rate | 2e-4 |
| Batch size | 8 (train and eval) |
| Experiment tracker | W&B |

## Evaluation

Evaluated on a held-out 20% split (120 examples) of the synthetic dataset.

| Epoch | Training Loss | Validation Loss | Accuracy | F1 (macro) |
|---|---|---|---|---|
| 1 | 0.249 | 0.376 | 0.943 | 0.945 |
| 2 | 0.015 | 0.248 | 0.951 | 0.953 |

**Final validation metrics:** accuracy 0.951, F1 (macro) 0.953, loss 0.248.

**Confusion matrix summary:** `billing`, `shipping`, and `general` were classified with zero errors. The remaining errors (5 of 120 tickets) were concentrated between `technical` and `account`, likely reflecting genuine overlap (e.g. login issues that are also technical errors) rather than a labeling artifact.

An earlier training run on an unrefined dataset (undefined `general` category) scored 71.7% accuracy / 0.704 F1 (macro) on the same model architecture and hyperparameters, with most errors concentrated in `general` misclassifications. This gap was traced to a data generation issue rather than a modeling issue — see Training Data above.

## Limitations

- Trained and evaluated entirely on synthetic data — has not been validated against real customer support tickets, which may differ in style, length, or ambiguity
- Five fixed categories — tickets spanning multiple categories default to the closest match
- Small residual confusion between `technical` and `account` categories
- English only
- Not evaluated for adversarial inputs or prompt injection
- Artificially balanced training classes (see Training Data)

## License

This adapter is released under Apache 2.0. The base model, `meta-llama/Llama-3.2-1B`, is subject to Meta's Llama 3.2 Community License, which includes usage restrictions (e.g. on very large-scale commercial deployments and certain use cases). Review the [Llama 3.2 license](https://www.llama.com/llama3_2/license/) before deploying this adapter.

## How to Use

```python
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
from peft import PeftModel

BASE_MODEL = "meta-llama/Llama-3.2-1B"
ADAPTER = "sheethal00/ticket-classifier-lora"

id2label = {0: "billing", 1: "technical", 2: "account", 3: "shipping", 4: "general"}
label2id = {v: k for k, v in id2label.items()}

tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

base_model = AutoModelForSequenceClassification.from_pretrained(
    BASE_MODEL,
    num_labels=5,
    id2label=id2label,
    label2id=label2id,
    torch_dtype=torch.float16,  # use torch.float32 if running on CPU
)
base_model.config.pad_token_id = tokenizer.pad_token_id

model = PeftModel.from_pretrained(base_model, ADAPTER)
model.eval()

# Inference
text = "I was charged twice for my subscription this month, can you refund the extra charge?"
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=128)

with torch.no_grad():
    logits = model(**inputs).logits

predicted_id = logits.argmax(dim=-1).item()
print(model.config.id2label[predicted_id])  # -> "billing"
```

**Note:** This adapter was trained with 4-bit quantization (QLoRA). For inference, full precision (fp16/fp32) works fine and is simpler to set up; if you want to match the training setup exactly, load `base_model` with a `BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.float16)` (requires a CUDA GPU — 4-bit quantization is not supported on CPU).