appout/houston-ticket-summarizer

Houston ticket summarizer

A fine-tuned Qwen3-0.6B that summarizes support tickets as three lines. Small enough to run entirely in the browser (llama.cpp / WASM), so ticket text never leaves the agent's machine.

Problem: what was actually wrong
Action:  what the support team did
Outcome: where it ended up

Results

Measured on 150 held-out tickets the model never saw during training, graded against their source text:

	base Qwen3-0.6B	this model
Usable summary	24%	75%
Problem correct	83%	87%
Action correct	57%	89%
Outcome correct	33%	90%
Invents nothing	28%	76%

Files

File	Size	Notes
houston-0.6b-q8_0.gguf	610 MB	the shipped build, 75% usable

Q4_K_M was also built (379 MB, 71% usable) and is not faster in WASM, where dequantization compute rather than memory bandwidth is the bottleneck.

Prompt contract

Quality drops silently if this is not followed exactly.

System prompt, verbatim:

Summarize this support ticket as Problem / Action / Outcome.

User message, this exact layout:

Ticket T-2811 | Unique number: 386515 | Tenant: Example Co | Client: Assem
Title: Delete user
Description: Assem asked to delete the user below.
Comments:
- Three users deleted, attached screenshot before and after
- mail scheduled to be sent to the customer 9:00 am tomorrow

• Header parts are joined by |; omit any that are empty.
• No author names on comments. Training data had none, and adding them makes the model attribute actions to whichever name it sees.
• No workflow events (Ticket opened, Ticket acquired, bulk-close, System auto assigned...). Short human notes like Updated the client or done must be kept: that is where the outcome lives.
• Emails and phone numbers masked as <email> / <phone>.
• Optional trailing Status: <status> line when a status is known.

Thinking must be disabled. The assistant turn is pre-filled with an empty think block:

<|im_start|>system
Summarize this support ticket as Problem / Action / Outcome.<|im_end|>
<|im_start|>user
{ticket}<|im_end|>
<|im_start|>assistant
<think>

</think>

With Qwen3's default template the model reasons out loud and exhausts its token budget before answering.

Decode: greedy (temperature 0), max_tokens 200, stop on <|im_end|>.

Training

Base	Qwen/Qwen3-0.6B
Method	full fine-tune, not LoRA (at 0.6B the optimizer state fits one GPU, so LoRA only adds a constraint)
Data	1,057 ticket/summary pairs distilled from a larger teacher model, 60 val, 150 frozen holdout
LR	1e-5 (LoRA-scale rates destroy a full fine-tune)
Epochs	3, effective batch 16, max_len 2048, seed 1337
Loss	assistant tokens only, prompt masked
Hardware	one Colab L4, ~15 min

Training data is internal support tickets with emails and phone numbers redacted. It is not released.

Limitations

• English only. The corpus was English support tickets. Arabic is not covered.
• ~25% of summaries assert something the ticket does not say, almost always reading a request as a completed action ("the team removed X" when the ticket only asks for X to be removed). The teacher model gets these right, so this is 0.6B capacity, not bad data. A larger student is the fix.
• Stale outcomes. On long threads the model sometimes reports an intermediate state instead of the final one.
• Summary length is not controlled; it produces three lines, not a length budget.
• Not a general summarizer. It expects the layout above and degrades on anything else.

Intended use

Showing a support agent, at a glance, what a ticket is about. Not for automated decisions, ticket routing, or anything where an unsupported assertion in one summary in four would matter.