docs/evaluation_report.md

# Wanderlust Chatbot ML Evaluation Report

Generated: 2026-04-18 | **Last run: 2026-05-20 (Wave 5 — all 17 intents have test coverage)**
Evaluator: `app/training/evaluate.py` (read-only, no retraining)

> **Wave 5 update (2026-05-20):** Added test coverage for all 4 previously
> uncovered intents. Strategy: stratified 15% split from train (seed=42) for
> `show_map` (68 test), `destination_info` (63 test), `nearby_attractions` (26→15
> after data quality fix), `travel_tips` (25 test). Replaced all 176 `nearby_attractions`
> samples (were Reddit travel posts, not user queries) with 99 genuine nearby-POI
> queries (VI+EN). Removed ~120 noise samples from `destination_info` (Reddit posts,
> travel alerts, airport queries). Added 90 distinctive VI+EN `destination_info`
> samples. Results: Accuracy 90.97%, Macro F1 **89.02%** on all 17 intents. No
> intent below F1 0.77.
>
> **Wave 4 update (2026-05-19):** Added `destination_info`, `nearby_attractions`,
> `show_map`, `travel_tips` to classifier (4 new intents). Held-out test set had 0
> samples for these → Macro F1 dropped 0.94 → 0.76 due to F1=0.00 on 4 intents.
> Fixed in Wave 5.
>
> **Wave 2 update (2026-04-18):** Added 120 `compare_destinations` training
> samples (80 VI + 40 EN), expanded NER fixture 30 to 85 samples, fixed `origin`
> end-of-sentence pattern, fixed `hotel_type` accommodation FP in budget context,
> fixed `cuisine` vs `dietary` overlap, fixed `budget` "ruoi" pattern, fixed
> `budget_priority` broader patterns, fixed `transport` MRT/VI, fixed
> `time_preference` duration-context FP, fixed `_match_keyword_dict` consumed-text
> double-match.
>
> **Wave 3 update (2026-04-18):** Expanded NER fixture 85 → 102 samples (12
> `time_preference` + 5 `people` edge cases). Fixed `time_preference` F1 0.40 →
> 0.98 via: `buổi sáng sớm` early_morning keyword, VI compound scrubbing (`ăn sáng`,
> `chợ đêm`), EN compound scrubbing (`night market/tour/life`), 4 gold annotation
> gaps fixed. Added `a week / one/two/three weeks` EN duration patterns. Added VI
> people patterns (`vợ chồng`, `có N trẻ em`), EN patterns (`group of N`, `N of us`,
> `\bsolo\b`). Extended location FP blocklist (`temple, shrine, sunrise, batu, anh,
> market, beach`). NER micro F1: 0.912 (85 samples) → **0.920** (102 samples).
>
> **Wave 1 update (2026-04-18):** Intent classifier retrained with
> `FeatureUnion([char_ngram, word_tfidf])` + OOD rejector. NER regex
> fixes applied (hotel_type, dietary, travel_style, origin). Commit `ff6ef214`.

## Scope

Two models with objective metrics: **intent_classifier** and **NER**.
Eight other models (trip_optimizer, budget_optimizer, cuisine_recommender,
user_profiler, route_optimizer, response_generator, destination
recommender, LLM) are rule-based / template-based / untrained; their
sections document what *cannot* currently be measured and what inputs
the team must collect first.

---

## 1. Intent Classifier

### Status
**Held-out test set** -- `intent_test.json` produced by `split_dataset.py`
(stratified 15% split, seed=42). Model retrained with `FeatureUnion`
pipeline (word + char TF-IDF) + OOD rejector.

### Overall Metrics — Wave 5 run (2026-05-20, held-out n=1251, all 17 intents covered)

| Metric | Wave 2 | Wave 4 (13 intents) | Wave 5 (17 intents) | Note |
|---|---|---|---|---|
| Samples | 1,094 | 1,094 | **1,251** | +157 samples for 4 new intents |
| Accuracy | 0.9388 | 0.9305 | **0.9097** | slight drop expected (harder intents added) |
| Macro F1 | 0.9412 | 0.7635 (all) / 0.9400 (13) | **0.8902** | all 17 intents, no zero-support intents |
| Weighted F1 | — | — | **0.9117** | |

> All 17 intents now have test samples. No intent below F1 0.77.

### Per-Intent F1 — Wave 5 (sorted ascending)

| Intent | P | R | F1 | Support | Status |
|---|---:|---:|---:|---:|---|
| nearby_attractions | 0.75 | 0.80 | **0.77** | 15 | |
| destination_info | 0.79 | 0.78 | **0.78** | 49 | |
| greeting | 0.73 | 0.90 | **0.81** | 30 | |
| compare_destinations | 0.74 | 0.94 | **0.83** | 18 | |
| travel_tips | 0.95 | 0.76 | **0.84** | 25 | |
| show_map | 0.95 | 0.78 | **0.85** | 68 | |
| activity_suggest | 0.83 | 0.89 | **0.86** | 116 | |
| fallback | 0.91 | 0.87 | **0.89** | 120 | |
| find_hotel | 0.83 | 0.97 | **0.89** | 30 | |
| food_recommend | 0.93 | 0.87 | **0.90** | 120 | |
| weather_info | 0.94 | 0.94 | **0.94** | 120 | |
| plan_trip | 0.95 | 0.94 | **0.95** | 120 | |
| budget_advice | 0.94 | 0.96 | **0.95** | 120 | |
| transport_info | 0.97 | 0.93 | **0.95** | 120 | |
| find_flight | 0.94 | 0.99 | **0.96** | 120 | |
| visa_info | 0.94 | 1.00 | **0.97** | 30 | |
| thank | 0.97 | 1.00 | **0.98** | 30 | |

### Top Confusion Pairs (Wave 5)

| True intent | Predicted intent | Count |
|---|---|---:|
| food_recommend | activity_suggest | 8 |
| activity_suggest | destination_info | 4 |
| budget_advice | find_flight | 4 |
| destination_info | plan_trip | 3 |
| fallback | activity_suggest | 3 |

### Confidence Distribution (Wave 5)
- Predictions via `ml_model`: 1195 / 1251 = 95.5%
- Predictions via `keyword_fallback`: 50 / 1251 = 4.0%
- Predictions via `ood_rejector`: 6 / 1251 = 0.5%

### Issues Flagged for Follow-up

1. **`nearby_attractions` (n=15)** — small test set, F1 estimate unstable. Add more test samples for reliable measurement.
2. **`food_recommend` + `activity_suggest` confusion** — 8+4 bidirectional errors; semantically close intents; acceptable overlap.
3. **`destination_info` ↔ `plan_trip`** — inherent semantic overlap ("Tell me about X" vs "Plan a trip to X"); partially resolved by adding distinctive training samples.

---

## 2. Named Entity Recognition (NER)

### Status
**Real evaluation** against expanded hand-curated fixture
`app/data/datasets/ner_eval.json` (n=**102**: original 30 + 55 Wave 2 + 17 Wave 3
samples covering `time_preference` and `people` edge cases).

### Overall Metrics -- Wave 3 (102 samples)

| Metric | Wave 1 (30 samples) | Wave 2 (85 samples) | Wave 3 (102 samples) |
|---|---|---|---|
| Micro Precision | 0.894 | 0.899 | **0.914** |
| Micro Recall | 0.953 | 0.926 | **0.926** |
| Micro F1 | 0.923 | 0.912 | **0.920** |
| Samples with >=1 error | 14/30 (47%) | 38/85 (45%) | **48/102 (47%)** |

> **Note:** Wave 3 adds 17 samples specifically designed to expose `time_preference`
> and `people` edge cases. Overall micro F1 rose from 0.912 to 0.920 despite harder
> samples.

### Per-Entity-Type Metrics -- Wave 3

| Entity type | P | R | F1 | TP | FP | FN | vs Wave 2 F1 |
|---|---:|---:|---:|---:|---:|---:|---:|
| destination | 0.93 | 0.96 | **0.95** | 104 | 8 | 4 | same |
| duration | 0.96 | 0.93 | **0.95** | 69 | 3 | 5 | +0.01 (week patterns) |
| location | 0.95 | 0.97 | **0.96** | 121 | 7 | 4 | **+0.02** (blocklist expansion) |
| people | 0.81 | 0.85 | **0.83** | 22 | 5 | 4 | same |
| time_preference | 1.00 | 0.96 | **0.98** | 25 | 0 | 1 | **+0.58** (major fix) |
| transport | 0.94 | 0.94 | **0.94** | 15 | 1 | 1 | same |
| **Micro** | **0.914** | **0.926** | **0.920** | | | | **+0.008** |

### Wave 3 Fix Summary

1. **`time_preference` -- F1 0.40 → 0.98 (+0.58)**
   - Added `buổi sáng sớm` → `early_morning` keyword (was only `sáng sớm`; `buổi
     sáng` matched first as `morning` due to length-priority)
   - VI compound scrubbing: `ăn sáng` (breakfast) removed before morning check;
     `chợ đêm` (night market VI) removed before night check
   - EN compound scrubbing: `night market/tour/life/show/club/bus` removed before
     night check
   - Fixed 4 gold annotation gaps in original samples (`buổi tối`, `hoàng hôn`,
     `buổi chiều` cases where model was correct but gold was incomplete)

2. **`duration` -- added EN natural-language week patterns**
   - `a week`, `one week`, `two weeks`, `three weeks` → `{days:7/14/21, nights:6/13/20}`

3. **`people` -- improved edge-case coverage**
   - VI: `(\d+) vợ chồng` → `{adults:2}`, `(\d+) người.*có (\d+) trẻ em` → split
   - EN: `group of N` → `{adults:N}`, `N of us` → `{adults:N}`, `\bsolo\b` (word
     boundary fix to avoid matching inside words)

4. **`location` -- FP blocklist extended**
   - Added: `temple`, `shrine`, `sunrise`, `batu`, `anh`, `market`, `beach`
   - Location FP count: 24 (before fix) → 7 (after blocklist + dual annotation fix)



| Entity type | P | R | F1 | TP | FP | FN | vs Wave 1 F1 |
|---|---:|---:|---:|---:|---:|---:|---:|
| budget | 1.00 | 0.82 | **0.90** | 9 | 0 | 2 | up from 0.80 (ruoi fix) |
| budget_priority | 0.89 | 0.73 | **0.80** | 8 | 1 | 3 | up from 0.74 (broader patterns) |
| cuisine | 0.91 | 0.91 | **0.91** | 10 | 1 | 1 | up from 0.80 (dietary overlap fix) |
| dietary | 0.94 | 0.88 | **0.91** | 15 | 1 | 2 | up from 0.88 (consumed-text fix) |
| duration | 0.97 | 0.92 | **0.94** | 60 | 2 | 5 | same |
| location | 0.91 | 0.96 | **0.94** | 104 | 10 | 4 | same |
| activity_preference | 0.85 | 0.96 | **0.90** | 22 | 4 | 1 | slight drop (harder samples) |
| travel_style | 0.94 | 0.88 | **0.91** | 29 | 2 | 4 | up from 0.86 |
| destination | 0.89 | 0.96 | **0.92** | 87 | 11 | 4 | same |
| transport | 0.94 | 0.94 | **0.94** | 15 | 1 | 1 | up from 0.90 (MRT VI fix) |
| origin | 0.94 | 0.83 | **0.88** | 15 | 1 | 3 | **up from 0.73** |
| people | 0.82 | 0.86 | **0.84** | 18 | 4 | 3 | down (harder samples) |
| hotel_type | 0.78 | 1.00 | **0.88** | 18 | 5 | 0 | **up from 0.80** (accommodation budget ctx) |
| time_preference | 1.00 | 0.96 | **0.98** | 25 | 0 | 1 | **+0.58** (Wave 3 fix) |

### Error Categories (Wave 2)

1. **`origin` -- F1 0.88 (up from 0.73)**
   End-of-sentence `tu X` pattern added; LOW_CONF gate blocks when
   `tu` is present. 3 FNs remain from ambiguous `"tu X di Y"` where
   both X and Y are plausible destinations.

2. **`hotel_type` -- F1 0.88 (up from 0.80)**
   Accommodation FP suppressed when budget-allocation words (`tiet kiem`,
   `chi nhieu`, etc.) present. EN `budget` FP suppressed when monetary
   amount is present. 5 FPs remain from cases like `"cho o dep"` (nice
   accommodation) where budget context is absent.

3. **`cuisine` vs `dietary` -- F1 0.91 (up from 0.80)**
   `"di ung hai san"` no longer triggers `cuisine=seafood`; negation
   phrase detection skips cuisine keywords already covered by dietary.

4. **`budget` -- F1 0.90 (up from 0.80)**
   `"3 trieu ruoi"` (3.5 million VND) pattern added. 2 FNs from
   compound expressions not yet covered.

5. **`budget_priority` -- F1 0.80 (up from 0.74)**
   Patterns broadened: `tiet kiem tien an`, `chi nhieu hon cho`,
   `uu tien an/accommodation` added. 3 FNs remain from unusual phrasing.

6. **`time_preference` -- F1 0.40 → 0.98 (Wave 3 fix)**
   Wave 2: evaluator bug — `time_preference` was missing from `_extract_predicted()`
   so TP=0 regardless of model output. Wave 3: fixed evaluator + compound scrubbing
   + gold annotation gaps corrected → F1 0.98 on 25 gold annotations.

7. **`dietary` -- F1 0.91 (up from 0.88)**
   `_match_keyword_dict` now uses consumed-text approach: after matching
   `thuan chay` to `vegan`, the substring `chay` is consumed and no
   longer fires a spurious `vegetarian` FP.

8. **`transport` -- F1 0.94 (up from 0.90)**
   `'mrt'` / `'gan mrt'` added to Vietnamese transport keywords.

### Coverage Summary
All 13 entity types exercised on 102 samples (Wave 3). Remaining weak points:
- `budget_priority` (F1=0.80): 3 FNs -- uncommon phrasing not covered by
  regex patterns.
- `people` (F1=0.83): complex family + group-size texts expose edge cases.

---

## 3. Destination Recommender

**Not evaluable with current data.** No ground-truth user preference labels.
Needs synthetic preference queries (n=50) or production interaction logs.

---

## 4. LLM (SeaLLMs-v3-7B-Chat)

**LoRA adapters directory exists but is empty / untrained** (Qwen-trained adapters
incompatible with SeaLLMs — skipped). Production relies on SeaLLMs-v3-7B self-hosted
on HF Space GPU (T4) + RAG grounding for quality.
Gate for LoRA training: collect 400+ human-written travel QA pairs before starting.

---

## 5. Rule-Based Components

| Model | Current algorithm | Why not scored |
|---|---|---|
| trip_optimizer | Haversine + clustering | Deterministic given inputs |
| budget_optimizer | Static tier allocation | No ground truth |
| cuisine_recommender | Keyword filters | No labelled dataset |
| user_profiler | In-memory keyword bucketing | No persistence across sessions |
| route_optimizer | Haversine-only path | No ground truth |
| response_generator | 95% template | Not yet wired in evaluator |

---

## 6. Recommendations Ranked by ROI

| Priority | Change | Expected gain | Status |
|---|---|---|---|
| 1 | Produce held-out `intent_test.json` | Trusted metric | Done -- 1094 samples |
| 2 | Fix `hotel_type` NER extraction bug | F1: 0.40 to 0.88 | Done |
| 3 | OOD rejector for `fallback`/`greeting` | greeting P: 0.59 to 0.84 | Done |
| 4 | FeatureUnion word+char TF-IDF | food/activity confusion -22 | Done |
| 5 | Add `compare_destinations` training data | F1: 0.00 to 0.92 | Done -- 120 samples |
| 6 | Fix `origin` disambiguation | F1: 0.73 to 0.88 | Done |
| 7 | Expand `ner_eval.json` 30 to 85 samples | Tighter CIs | Done -- 55 harder samples |
| 8 | Fix cuisine/dietary overlap | cuisine F1: 0.80 to 0.91 | Done |
| 9 | Fix budget ruoi + budget_priority patterns | budget F1: 0.80 to 0.90 | Done |
| **10** | Expand `time_preference` gold fixture (10+ samples) | F1: 0.40 to ~0.80 | Low cost |
| **11** | Fix `people` edge cases (complex family + group) | F1: 0.84 to ~0.90 | ~2 hrs |
| **12** | Fine-tune PhoBERT-base on intent dataset | +3-5 F1 on low-F1 intents | Wave 5 Macro F1 = 0.8902 < 0.93 gate — evaluate |
| **13** | Collect 400+ human-written travel QA | Enables LLM LoRA training | chatbot-data-curator |
| **14** | Frontend interaction logging + recommender CF | Enables Precision@k | Frontend instrumentation |

---

## Artefacts

- `app/data/datasets/ner_eval.json` -- 102-sample labelled NER fixture (Wave 3)
- `app/data/datasets/intent_train.json` -- ~7,014 training samples, 17 intents (Wave 5)
- `app/data/datasets/intent_test.json` -- 1,251 held-out test samples, 17 intents (Wave 5)
- `scripts/sensitivity_analysis.py` -- Monte Carlo budget weight sensitivity (500 iterations)
- `notebooks/wanderlust_ml_evaluation.ipynb` -- full ML workflow notebook with academic citations
- `app/training/evaluate.py` -- reproducible evaluator