docs/evaluation_report.md

# Wanderlust Chatbot ML Evaluation Report

Generated: 2026-04-18 | Last run: 2026-05-20 (Wave 5 — all 17 intents have test coverage) Evaluator: app/training/evaluate.py (read-only, no retraining)

Wave 5 update (2026-05-20): Added test coverage for all 4 previously uncovered intents. Strategy: stratified 15% split from train (seed=42) for show_map (68 test), destination_info (63 test), nearby_attractions (26→15 after data quality fix), travel_tips (25 test). Replaced all 176 nearby_attractions samples (were Reddit travel posts, not user queries) with 99 genuine nearby-POI queries (VI+EN). Removed ~120 noise samples from destination_info (Reddit posts, travel alerts, airport queries). Added 90 distinctive VI+EN destination_info samples. Results: Accuracy 90.97%, Macro F1 89.02% on all 17 intents. No intent below F1 0.77.

Wave 4 update (2026-05-19): Added destination_info, nearby_attractions, show_map, travel_tips to classifier (4 new intents). Held-out test set had 0 samples for these → Macro F1 dropped 0.94 → 0.76 due to F1=0.00 on 4 intents. Fixed in Wave 5.

Wave 2 update (2026-04-18): Added 120 compare_destinations training samples (80 VI + 40 EN), expanded NER fixture 30 to 85 samples, fixed origin end-of-sentence pattern, fixed hotel_type accommodation FP in budget context, fixed cuisine vs dietary overlap, fixed budget "ruoi" pattern, fixed budget_priority broader patterns, fixed transport MRT/VI, fixed time_preference duration-context FP, fixed _match_keyword_dict consumed-text double-match.

Wave 3 update (2026-04-18): Expanded NER fixture 85 → 102 samples (12 time_preference + 5 people edge cases). Fixed time_preference F1 0.40 → 0.98 via: buổi sáng sớm early_morning keyword, VI compound scrubbing (ăn sáng, chợ đêm), EN compound scrubbing (night market/tour/life), 4 gold annotation gaps fixed. Added a week / one/two/three weeks EN duration patterns. Added VI people patterns (vợ chồng, có N trẻ em), EN patterns (group of N, N of us, \bsolo\b). Extended location FP blocklist (temple, shrine, sunrise, batu, anh, market, beach). NER micro F1: 0.912 (85 samples) → 0.920 (102 samples).

Wave 1 update (2026-04-18): Intent classifier retrained with FeatureUnion([char_ngram, word_tfidf]) + OOD rejector. NER regex fixes applied (hotel_type, dietary, travel_style, origin). Commit ff6ef214.

Scope

Two models with objective metrics: intent_classifier and NER. Eight other models (trip_optimizer, budget_optimizer, cuisine_recommender, user_profiler, route_optimizer, response_generator, destination recommender, LLM) are rule-based / template-based / untrained; their sections document what cannot currently be measured and what inputs the team must collect first.

---

1. Intent Classifier

Status

Held-out test set -- intent_test.json produced by split_dataset.py (stratified 15% split, seed=42). Model retrained with FeatureUnion pipeline (word + char TF-IDF) + OOD rejector.

Overall Metrics — Wave 5 run (2026-05-20, held-out n=1251, all 17 intents covered)

Metric	Wave 2	Wave 4 (13 intents)	Wave 5 (17 intents)	Note
Samples	1,094	1,094	1,251	+157 samples for 4 new intents
Accuracy	0.9388	0.9305	0.9097	slight drop expected (harder intents added)
Macro F1	0.9412	0.7635 (all) / 0.9400 (13)	0.8902	all 17 intents, no zero-support intents
Weighted F1	—	—	0.9117

All 17 intents now have test samples. No intent below F1 0.77.

Per-Intent F1 — Wave 5 (sorted ascending)

Intent	P	R	F1	Support	Status
nearby_attractions	0.75	0.80	0.77	15
destination_info	0.79	0.78	0.78	49
greeting	0.73	0.90	0.81	30
compare_destinations	0.74	0.94	0.83	18
travel_tips	0.95	0.76	0.84	25
show_map	0.95	0.78	0.85	68
activity_suggest	0.83	0.89	0.86	116
fallback	0.91	0.87	0.89	120
find_hotel	0.83	0.97	0.89	30
food_recommend	0.93	0.87	0.90	120
weather_info	0.94	0.94	0.94	120
plan_trip	0.95	0.94	0.95	120
budget_advice	0.94	0.96	0.95	120
transport_info	0.97	0.93	0.95	120
find_flight	0.94	0.99	0.96	120
visa_info	0.94	1.00	0.97	30
thank	0.97	1.00	0.98	30

Top Confusion Pairs (Wave 5)

True intent	Predicted intent	Count
food_recommend	activity_suggest	8
activity_suggest	destination_info	4
budget_advice	find_flight	4
destination_info	plan_trip	3
fallback	activity_suggest	3

Confidence Distribution (Wave 5)

• Predictions via ml_model: 1195 / 1251 = 95.5%
• Predictions via keyword_fallback: 50 / 1251 = 4.0%
• Predictions via ood_rejector: 6 / 1251 = 0.5%

Issues Flagged for Follow-up

1. nearby_attractions (n=15) — small test set, F1 estimate unstable. Add more test samples for reliable measurement.
2. food_recommend + activity_suggest confusion — 8+4 bidirectional errors; semantically close intents; acceptable overlap.
3. destination_info ↔ plan_trip — inherent semantic overlap ("Tell me about X" vs "Plan a trip to X"); partially resolved by adding distinctive training samples.

---

2. Named Entity Recognition (NER)

Status

Real evaluation against expanded hand-curated fixture app/data/datasets/ner_eval.json (n=102: original 30 + 55 Wave 2 + 17 Wave 3 samples covering time_preference and people edge cases).

Overall Metrics -- Wave 3 (102 samples)

Metric	Wave 1 (30 samples)	Wave 2 (85 samples)	Wave 3 (102 samples)
Micro Precision	0.894	0.899	0.914
Micro Recall	0.953	0.926	0.926
Micro F1	0.923	0.912	0.920
Samples with >=1 error	14/30 (47%)	38/85 (45%)	48/102 (47%)

Note: Wave 3 adds 17 samples specifically designed to expose time_preference and people edge cases. Overall micro F1 rose from 0.912 to 0.920 despite harder samples.

Per-Entity-Type Metrics -- Wave 3

Entity type	P	R	F1	TP	FP	FN	vs Wave 2 F1
destination	0.93	0.96	0.95	104	8	4	same
duration	0.96	0.93	0.95	69	3	5	+0.01 (week patterns)
location	0.95	0.97	0.96	121	7	4	+0.02 (blocklist expansion)
people	0.81	0.85	0.83	22	5	4	same
time_preference	1.00	0.96	0.98	25	0	1	+0.58 (major fix)
transport	0.94	0.94	0.94	15	1	1	same
Micro	0.914	0.926	0.920				+0.008

Wave 3 Fix Summary

1. time_preference -- F1 0.40 → 0.98 (+0.58)

   • Added buổi sáng sớm → early_morning keyword (was only sáng sớm; buổi sáng matched first as morning due to length-priority)
   • VI compound scrubbing: ăn sáng (breakfast) removed before morning check; chợ đêm (night market VI) removed before night check
   • EN compound scrubbing: night market/tour/life/show/club/bus removed before night check
   • Fixed 4 gold annotation gaps in original samples (buổi tối, hoàng hôn, buổi chiều cases where model was correct but gold was incomplete)

2. duration -- added EN natural-language week patterns

   • a week, one week, two weeks, three weeks → {days:7/14/21, nights:6/13/20}

3. people -- improved edge-case coverage

   • VI: (\d+) vợ chồng → {adults:2}, (\d+) người.*có (\d+) trẻ em → split
   • EN: group of N → {adults:N}, N of us → {adults:N}, \bsolo\b (word boundary fix to avoid matching inside words)

4. location -- FP blocklist extended

   • Added: temple, shrine, sunrise, batu, anh, market, beach
   • Location FP count: 24 (before fix) → 7 (after blocklist + dual annotation fix)

Entity type	P	R	F1	TP	FP	FN	vs Wave 1 F1
budget	1.00	0.82	0.90	9	0	2	up from 0.80 (ruoi fix)
budget_priority	0.89	0.73	0.80	8	1	3	up from 0.74 (broader patterns)
cuisine	0.91	0.91	0.91	10	1	1	up from 0.80 (dietary overlap fix)
dietary	0.94	0.88	0.91	15	1	2	up from 0.88 (consumed-text fix)
duration	0.97	0.92	0.94	60	2	5	same
location	0.91	0.96	0.94	104	10	4	same
activity_preference	0.85	0.96	0.90	22	4	1	slight drop (harder samples)
travel_style	0.94	0.88	0.91	29	2	4	up from 0.86
destination	0.89	0.96	0.92	87	11	4	same
transport	0.94	0.94	0.94	15	1	1	up from 0.90 (MRT VI fix)
origin	0.94	0.83	0.88	15	1	3	up from 0.73
people	0.82	0.86	0.84	18	4	3	down (harder samples)
hotel_type	0.78	1.00	0.88	18	5	0	up from 0.80 (accommodation budget ctx)
time_preference	1.00	0.96	0.98	25	0	1	+0.58 (Wave 3 fix)

Error Categories (Wave 2)

1. origin -- F1 0.88 (up from 0.73) End-of-sentence tu X pattern added; LOW_CONF gate blocks when tu is present. 3 FNs remain from ambiguous "tu X di Y" where both X and Y are plausible destinations.

2. hotel_type -- F1 0.88 (up from 0.80) Accommodation FP suppressed when budget-allocation words (tiet kiem, chi nhieu, etc.) present. EN budget FP suppressed when monetary amount is present. 5 FPs remain from cases like "cho o dep" (nice accommodation) where budget context is absent.

3. cuisine vs dietary -- F1 0.91 (up from 0.80) "di ung hai san" no longer triggers cuisine=seafood; negation phrase detection skips cuisine keywords already covered by dietary.

4. budget -- F1 0.90 (up from 0.80) "3 trieu ruoi" (3.5 million VND) pattern added. 2 FNs from compound expressions not yet covered.

5. budget_priority -- F1 0.80 (up from 0.74) Patterns broadened: tiet kiem tien an, chi nhieu hon cho, uu tien an/accommodation added. 3 FNs remain from unusual phrasing.

6. time_preference -- F1 0.40 → 0.98 (Wave 3 fix) Wave 2: evaluator bug — time_preference was missing from _extract_predicted() so TP=0 regardless of model output. Wave 3: fixed evaluator + compound scrubbing

   • gold annotation gaps corrected → F1 0.98 on 25 gold annotations.

7. dietary -- F1 0.91 (up from 0.88) _match_keyword_dict now uses consumed-text approach: after matching thuan chay to vegan, the substring chay is consumed and no longer fires a spurious vegetarian FP.

8. transport -- F1 0.94 (up from 0.90) 'mrt' / 'gan mrt' added to Vietnamese transport keywords.

Coverage Summary

All 13 entity types exercised on 102 samples (Wave 3). Remaining weak points:

• budget_priority (F1=0.80): 3 FNs -- uncommon phrasing not covered by regex patterns.
• people (F1=0.83): complex family + group-size texts expose edge cases.

---

3. Destination Recommender

Not evaluable with current data. No ground-truth user preference labels. Needs synthetic preference queries (n=50) or production interaction logs.

---

4. LLM (SeaLLMs-v3-7B-Chat)

LoRA adapters directory exists but is empty / untrained (Qwen-trained adapters incompatible with SeaLLMs — skipped). Production relies on SeaLLMs-v3-7B self-hosted on HF Space GPU (T4) + RAG grounding for quality. Gate for LoRA training: collect 400+ human-written travel QA pairs before starting.

---

5. Rule-Based Components

Model	Current algorithm	Why not scored
trip_optimizer	Haversine + clustering	Deterministic given inputs
budget_optimizer	Static tier allocation	No ground truth
cuisine_recommender	Keyword filters	No labelled dataset
user_profiler	In-memory keyword bucketing	No persistence across sessions
route_optimizer	Haversine-only path	No ground truth
response_generator	95% template	Not yet wired in evaluator

---

6. Recommendations Ranked by ROI

Priority	Change	Expected gain	Status
1	Produce held-out intent_test.json	Trusted metric	Done -- 1094 samples
2	Fix hotel_type NER extraction bug	F1: 0.40 to 0.88	Done
3	OOD rejector for fallback/greeting	greeting P: 0.59 to 0.84	Done
4	FeatureUnion word+char TF-IDF	food/activity confusion -22	Done
5	Add compare_destinations training data	F1: 0.00 to 0.92	Done -- 120 samples
6	Fix origin disambiguation	F1: 0.73 to 0.88	Done
7	Expand ner_eval.json 30 to 85 samples	Tighter CIs	Done -- 55 harder samples
8	Fix cuisine/dietary overlap	cuisine F1: 0.80 to 0.91	Done
9	Fix budget ruoi + budget_priority patterns	budget F1: 0.80 to 0.90	Done
10	Expand time_preference gold fixture (10+ samples)	F1: 0.40 to ~0.80	Low cost
11	Fix people edge cases (complex family + group)	F1: 0.84 to ~0.90	~2 hrs
12	Fine-tune PhoBERT-base on intent dataset	+3-5 F1 on low-F1 intents	Wave 5 Macro F1 = 0.8902 < 0.93 gate — evaluate
13	Collect 400+ human-written travel QA	Enables LLM LoRA training	chatbot-data-curator
14	Frontend interaction logging + recommender CF	Enables Precision@k	Frontend instrumentation

---

Artefacts

• app/data/datasets/ner_eval.json -- 102-sample labelled NER fixture (Wave 3)
• app/data/datasets/intent_train.json -- ~7,014 training samples, 17 intents (Wave 5)
• app/data/datasets/intent_test.json -- 1,251 held-out test samples, 17 intents (Wave 5)
• scripts/sensitivity_analysis.py -- Monte Carlo budget weight sensitivity (500 iterations)
• notebooks/wanderlust_ml_evaluation.ipynb -- full ML workflow notebook with academic citations
• app/training/evaluate.py -- reproducible evaluator