data/k8s_docs/QUESTION_PLAN.md

K8s Golden Dataset — Question Plan

Status: Structural guide for Week 1 step 5 authoring (v1.1 plan). This document defines the 25-question target distribution, per-type source-page mapping, and authoring constraints. It does NOT contain the 25 specific question texts — those are authored during step 5 in a fresh session, per cross-cutting #8 pilot-first discipline.

Upstream contracts:

• Taxonomy: CRAG 8-type (Yang et al., NeurIPS 2024) — see DECISIONS.md "K8s golden dataset uses CRAG's 8-type taxonomy as the schema".
• Source pages: see SOURCES.md (28 pages, category-locked; 8 already pulled, 20 to pull at step 4).
• Schema: see agent_bench/evaluation/harness.py GoldenQuestion plus the v1.1 plan's methodology #3 source-attribution fields.
• Flavor A/B for false_premise: see DECISIONS.md "False-premise questions come in two flavors".

---

Target distribution (25 questions total)

CRAG type	Count	Schema field	Notes
simple	5–6	question_type: "simple"	Baseline retrieval: direct lookup in 1 page, 1–2 sentence answer.
simple_w_condition	3–4	question_type: "simple_w_condition"	Answer depends on a condition stated in the question (enforcement level, volume type, Pod phase).
comparison	3–4	question_type: "comparison"	Answer compares two concepts across 2 pages; reranker stress.
multi_hop	5–6	question_type: "multi_hop"	Answer synthesizes 2–4 pages; reranker-stressing by construction.
false_premise	3–4	question_type: "false_premise"	Grounded refusal stress. Flavor A (pure refusal) + flavor B (documented negative).
set / aggregation / post_processing_heavy	0–3	respective values	Optional. Include only if natural from corpus content.
Total	25

Orthogonal flag: time_sensitive: bool on 2–3 questions. Does NOT replace question_type — it's an independent property for version-bounded content (feature state, API version migration, deprecations).

---

Per-type source-page mapping

Each row identifies the K8s concept pages a question of that type should draw from. Multi-hop and comparison questions list multiple pages intentionally.

simple (5–6 slots)

Pool questions where a 1–2 sentence answer lives inside a single page.

Candidate source	CRAG slot justification
k8s_pods.md	Pod IP semantics, container sharing, ephemeral containers
k8s_deployment.md	What a Deployment is, declarative update mechanic
k8s_configmap.md	What a ConfigMap is, immutable field
k8s_secret.md	What a Secret is, volume mount modes
RBAC Authorization (step 4 page)	RBAC primitive definitions (Role, RoleBinding, ClusterRole)
StatefulSet (step 4 page)	StatefulSet identity guarantees
DaemonSet (step 4 page)	One-per-node scheduling contract
Namespaces (step 4 page)	Namespace scoping for resources

Authoring rule: Each simple question must have exactly one expected source page and 1–2 source snippets. KHR target ≥ 0.60 on the authored keywords.

simple_w_condition (3–4 slots)

Pool questions where the answer explicitly depends on a condition named in the question.

Candidate source	Condition that shapes the answer
k8s_pod_security_admission.md	enforcement level: enforce / audit / warn
k8s_secret.md	mount mode: environment variable vs file in volume
Liveness/Readiness/Startup Probes (step 4)	probe type: liveness vs readiness vs startup
Volumes (step 4)	volume type: emptyDir vs configMap vs persistentVolumeClaim
Node-pressure Eviction (k8s_node_pressure_eviction.md)	resource under pressure: memory vs disk vs inodes

Authoring rule: The condition must be named in the question stem, not implied. The expected answer must change materially if the condition flips. Example: "How is a Secret mounted as a volume versus consumed as an environment variable?" is a valid simple_w_condition; "How is a Secret mounted?" is simple.

comparison (3–4 slots)

Pool questions where the answer explicitly compares two K8s concepts that span 2 pages.

Page pair	Concept compared
Deployment vs StatefulSet (step 4)	stateless vs stateful workload semantics
Deployment vs DaemonSet (step 4)	replica-count vs one-per-node scheduling
ConfigMap vs Secret	non-confidential vs confidential data, mount parity
Service vs Ingress (step 4)	L4 vs L7 exposure
Taints/Tolerations vs Node Affinity (step 4)	opt-out vs opt-in placement
Liveness vs Readiness probes (step 4)	restart vs traffic-routing semantics

Authoring rule: The question must force retrieval from both pages. Reranker stress is intentional — questions where BM25 would find one side but miss the other are the target. Expected sources: 2 pages minimum.

multi_hop (5–6 slots)

Pool questions where the answer synthesizes 2–4 pages. These are the primary reranker stressors.

Page set (example)	Hop path
Pod + Service + Ingress (step 4)	How external traffic reaches a Pod through Service → Ingress
Deployment + ReplicaSet + Pod	How a Deployment rollout changes the underlying ReplicaSet and Pod set
ConfigMap + Deployment	How a ConfigMap update propagates to Pods via env vars or mounted volume
HPA + Deployment + Metrics Server (partial step 4)	How HPA reads metrics and scales a Deployment
NetworkPolicy + Pod + Namespace (partial step 4)	How NetworkPolicy selectors resolve across namespaces
Job + Pod + Container lifecycle (partial step 4)	How a Job's completions and parallelism interact with Pod restart policy

Authoring rule: Expected sources ≥ 2 pages. The question must not be answerable from any single page alone. source_chunk_ids must list at least one chunk from each expected page; partial credit is granted in the evaluator if at least one expected chunk is cited (see agent_bench/evaluation/harness.py).

false_premise (3–4 slots)

Pool questions whose premise is wrong. Split across two flavors:

Flavor A — pure refusal (at least 1 slot):

• Premise targets a capability that does not exist in the K8s corpus (not in any pulled page).
• Example seed: "How do I configure Claude API rate limits in a Kubernetes Deployment?" (wrong domain — Claude API is not a K8s concept)
• Schema: category: "out_of_scope", expected_sources: [], source_snippets: [].
• Evaluator expectation: answer contains refusal phrasing AND cites zero sources.

Flavor B — documented negative (at least 1 slot, ideally 2):

• Corpus contains an explicit negative statement (e.g. NetworkPolicy "Anything TLS related" limitation at chunk 63 of k8s_network_policies.md).
• Example already in pilot: k8s_pilot_005 (NetworkPolicy mTLS).
• Schema: category: "retrieval", question_type: "false_premise", expected_sources: [<negative-answer page>], source_snippets: [<verbatim negative statement>].
• Evaluator expectation: answer reports the documented negative with citation, does NOT open with "the documentation does not provide instructions" phrasing (per pilot_005 Fix 1 + Fix 2 revert analysis).

Other flavor-B candidate pages for authoring:

• Pod Security Standards — explicit statements about what each profile does NOT permit
• RBAC Authorization — explicit statements about what RBAC does NOT provide (e.g. no deny rules)
• NetworkPolicy — additional negative clauses beyond the pilot_005 mTLS one

set / aggregation / post_processing_heavy (0–3 slots)

Include only if a K8s page naturally supports the pattern:

• set: "Which Kubernetes resources can expose a Service?" (answer is a set drawn from the Service page). Include 0–1 of this type if a clean example emerges; otherwise leave slot empty.
• aggregation: Unlikely to fit K8s docs (docs describe concepts, not tabular data). Likely leave empty.
• post_processing_heavy: Unlikely to fit K8s docs. Likely leave empty.

Default: Leave 0–3 as 0. Only author these if a question emerges organically during step 5. Do not force-author to hit a target count; the plan explicitly says "0–3, included only where corpus content naturally supports".

---

time_sensitive flag placement (2–3 questions)

Flag questions whose correct answer depends on K8s version state:

Candidate	Why time-sensitive
HPA API version	autoscaling/v1 vs autoscaling/v2 — v2 stable since 1.23
Pod Security Admission stability	"stable as of v1.25" — feature state in the page
PodSecurityPolicy removal	PSP removed in 1.25; migration path to PSA

Authoring rule: Set time_sensitive: true on exactly 2–3 questions. Distribute across ≥2 different CRAG types (e.g. one simple, one simple_w_condition) so the flag is not concentrated in a single type. Each time_sensitive question must cite a specific K8s version or feature state in the source snippet, otherwise the flag is not load-bearing.

---

Difficulty distribution

Loose guidance, not a hard constraint:

• easy: 8–10 questions — mostly simple and single-page simple_w_condition
• medium: 10–12 questions — comparison, most multi_hop, straightforward false_premise
• hard: 4–6 questions — deep multi_hop, flavor-B false_premise, time_sensitive + multi_hop combinations

The pilot's 6-question set is all easy/medium. Step 5 should add the hard tier.

---

Authoring checklist (per question)

For each of the 25 questions, the step 5 author must fill:

Field	Required	Notes
id	yes	k8s_<NNN> zero-padded (e.g. k8s_001)
question	yes	Natural-language question in the voice of a recruiter or developer
expected_answer_keywords	yes	3–6 keywords that MUST appear in a correct answer; drives keyword_hit_rate
expected_sources	yes	List of .md filenames from SOURCES.md; ≥1 for scoped questions, [] for flavor-A false-premise
category	yes	retrieval / calculation / out_of_scope
difficulty	yes	easy / medium / hard
requires_calculator	yes	false for all K8s questions (no calc tool use expected)
reference_answer	yes	1–3 sentence answer used by the optional LLM judge
question_type	yes	CRAG taxonomy value (exactly one of the 8 canonical strings)
time_sensitive	yes	bool; true on exactly 2–3 questions
source_chunk_ids	yes	Content-hashed chunk IDs (stable across reindex); must be [] for flavor-A false-premise
source_snippets	yes	~20 words verbatim per chunk; drift-detection field
source_pages	yes	Human-readable page anchor (e.g. "concepts/workloads/pods")
source_sections	yes	Deepest heading containing the snippet

Deprecation note: The pilot schema has is_multi_hop: bool. Step 5 may retire this field in favor of question_type == "multi_hop", but only after confirming the evaluator's partial-credit logic (agent_bench/evaluation/harness.py:38) is updated to read from question_type. Do NOT remove is_multi_hop without the corresponding harness update, or existing pilot questions will break partial-credit scoring.

---

Pilot-first validation before step 5 authoring

Before writing the 25 questions, step 5 author must:

1. Confirm the 20 new pages from step 4 are ingested and reachable via the pipeline (smoke-query test per SOURCES.md's post-ingest validation).
2. Re-run make evaluate on the existing 6-question pilot dataset against the newly-expanded corpus. The pilot's existing questions must still pass their per-question gates — if adding 20 new pages drops pilot P@5 materially, investigate before adding more questions on top.
3. Hand-draft 2–3 questions first, run them through the pipeline, and confirm retrieval surfaces the expected chunks. This is the final pilot-first checkpoint before bulk authoring.

Only after these three checks pass does the step 5 author proceed to the full 25-question authoring session.

Post-authoring observations (step 5 shipped 2026-04-14)

Pilot→full generalization numbers: pilot (6Q) P@5=0.80, R@5=1.00, KHR=0.81 → full (25Q post-fix) P@5=0.83, R@5=0.96, KHR=0.90. R@5 movement is within expected variance when corpus breadth expands from 8 → 28 pages; KHR jump from 0.81→0.90 is an open question — the 25Q distribution may skew toward questions where the golden keyword set is more readily satisfied (simple + simple_w_condition

• set together = 11/25 questions with short, high-precision expected answers), vs the pilot's retrieval-heavy mix. Worth revisiting if KHR drifts on future corpora — if consistent across datasets, it's authoring signal that the keyword set should be tightened for CRAG type parity.

Flavor-B reproducibility finding: k8s_022 (RBAC deny rules) and pilot_005 (NetworkPolicy mTLS) both produce refusal-phrased answers when the documented negative is in retrieved context. Two independent reproductions confirm the LLM-hedges-on-documented-negative pattern is a class of failure mode, not a one-off — strengthens the case for the deferred Fix 2 + targeted prompt guidance stacked experiment. Authoring itself is clean on both: retrieval surfaces the expected chunks, citation accuracy 1.00, snippets verify against chunk IDs.