AIDC-CTCM/content-tagging-competency-mapping

Content Tagging & Competency Mapping – Qwen3.5-4B

The production model for the Content Tagging & Competency Mapping capstone (AI Data Center Bootcamp · SDA × WeCloudData × BeamData).

This is the base Qwen3.5-4B model, unmodified. It was selected after benchmarking five candidate models and is served with vLLM on Kubernetes to automatically tag BeamData's course content.

Task

Each chunk of course content (slides, notebooks, markdown lectures, quiz spreadsheets) is tagged against a fixed taxonomy:

Field · Learning type · Level · Competency

The same tagging prompt and decoding settings were used for every candidate model.

Example outputs

File	Field	Competency	Level
Decision Trees Revised.pptx	Machine Learning	Decision Tree Modeling	Intermediate
Introduction to Pandas_.pptx	Data Science	Pandas & Data Wrangling	Beginner
HandsOn1_Window_Functions.md	SQL	Window Functions	Intermediate
Prompt_Engineering.ipynb	Generative AI	Prompt Engineering	Intermediate
Retrieval_Augmented_Generation.ipynb	Generative AI	RAG Systems	Intermediate

Aggregated per file from a full run over 231 chunks; Level shows the majority tag.

Benchmark & model selection

All models were scored on the same 85-row human-labeled ground truth. SLO: TTFT p95 < 0.3 s, end-to-end p95 < 4.0 s.

Model	Exact-match accuracy	Latency SLO	Cost / 1M requests
Qwen3.5-4B (selected)	58.8%	Both met to concurrency 4; at 8, E2E met, TTFT ~7% over	4.98 SAR
Qwen2.5-VL-7B Instruct	57.6%	Both met at every concurrency (1–8)	5.05 SAR
Qwen3.5-9B	57.6%	Same pattern as 4B	8.06 SAR
Llama-3.2-11B Vision Instruct	56.5%	Saturates after concurrency 2	25.02 SAR
GPT-4o (API baseline)	52.9%	E2E met; TTFT never met	not computed

Why Qwen3.5-4B: highest exact-match accuracy, 100% Field accuracy (tied for best), and the lowest cost per request. The larger Qwen3.5-9B was 38% slower and 62% more expensive with no accuracy gain.

Deployment

• Served with vLLM on Kubernetes (namespace contentmap)
• 1 × NVIDIA RTX A6000 (~49 GB VRAM), shared across candidate models, one active at a time

vllm serve AIDC-CTCM/content-tagging-competency-mapping --served-model-name Qwen3.5-4B

Limitations

• 85 labeled rows is small relative to 231 chunks, so accuracy figures are directional.
• Level is the weakest field for every model (61–67%), traced to an ambiguous taxonomy definition.
• A LoRA fine-tuned variant matched the base model (50/85 vs. 50/85), so the base model was kept. That benchmark shares source documents with some training data, so it is not fully blind.

Project code

Full pipeline, benchmarks, Kubernetes manifests, and ops runbook: https://github.com/HatunMQ/Content-Tagging-Competency-Mapping

License

A copy of Qwen3.5-4B, distributed under its original license. See LICENSE.