| { |
| "adapter": "artifacts/askai-v1/adapter", |
| "category_counts": { |
| "customer_support": 1, |
| "data_analysis": 1, |
| "documentation": 1, |
| "financial_planning": 1, |
| "legal_compliance": 1, |
| "marketing_collateral": 1, |
| "marketing_strategy": 1, |
| "product_design": 1, |
| "sales_enablement": 1, |
| "software_engineering": 1 |
| }, |
| "data": "artifacts/askai-v1/data/test.jsonl", |
| "metrics": { |
| "complete_contract": 1.0, |
| "routing_accuracy": 1.0, |
| "valid_json": 1.0, |
| "valid_schema": 1.0 |
| }, |
| "model": "mlx-community/Qwen3-Coder-Next-4bit", |
| "results": [ |
| { |
| "category": "software_engineering", |
| "complete_contract": true, |
| "correct_agent": true, |
| "error": null, |
| "expected_agent": "software_engineer", |
| "generated": "{\"action\":\"call_agent\",\"agent\":\"software_engineer\",\"budget\":{\"max_tokens\":4096},\"context_refs\":[\"user_request\"],\"expected_output\":\"completed_task_with_evidence\",\"instruction\":\"You are a senior embedded/connected-vehicle software engineer at a Tier-1 automotive supplier. Design, implement, and test a new high-priority user-facing feature for a connected-car companion app and dealer-services platform serving vehicle owners, dealership staff, and fleet managers.\\n\\n## Context\\nThe platform connects vehicles to mobile apps, dealer backends, and fleet management systems. Every change touches vehicle systems and must respect functional-safety standards (ISO 26262 ASIL-B or ASIL-A scope, depending on the feature), OTA update integrity (signed, verified, rollback-capable), and dealer-franchise agreements (no unauthorized access to dealer systems or customer data). The feature must ship behind a feature flag with a staged rollout plan.\\n\\n## Deliverables\\n1. **Design document** — user flows for vehicle owners (remote diagnostics, service scheduling, OTA notifications), dealership staff (service-appointment management, diagnostic data review), and fleet managers (vehicle health monitoring, maintenance scheduling); safety classification and threat model; OTA update integrity requirements; data-access controls per franchise agreements.\\n2. **Implementation** — backend services, mobile app screens, and fleet-dashboard views; unit and integration tests (mock vehicle CAN bus and OTA update paths); CI/CD pipeline with safety-impact review gates.\\n3. **Feature flag and rollout plan** — per-vehicle, per-dealer, and per-fleet rollout; rollback triggers; instrumentation for adoption, OTA update success/failure, and dealer-system access.\\n4. **End-to-end staging validation** — test report covering owner, dealer, and fleet flows, OTA integrity, and safety-classified components.\\n5. **Code review, staging sign-off, and adoption instrumentation** — metrics for adoption, OTA success, and dealer-system access.\\n\\n## Constraints\\n- ISO 26262 functional-safety requirements must be addressed for any safety-relevant component.\\n- OTA updates must be signed, verified, and rollback-capable; no OTA path may be bypassed.\\n- Dealer-franchise agreements must be respected: no unauthorized access to dealer systems or customer data.\\n- All changes must be behind a feature flag with a staged rollout plan.\\n\\n## Done When\\n- Design document approved, safety classification assigned, and threat model reviewed.\\n- Implementation complete with unit and integration tests; CI/CD gates in place.\\n- Feature flag and rollout plan documented and executed in staging.\\n- End-to-end validation passes for owner, dealer, and fleet flows.\\n- Code review complete, staging sign-off obtained, and adoption instrumentation in place.\",\"model\":\"auto\"}", |
| "source_id": "task-0012", |
| "valid_json": true, |
| "valid_schema": true |
| }, |
| { |
| "category": "marketing_strategy", |
| "complete_contract": true, |
| "correct_agent": true, |
| "error": null, |
| "expected_agent": "marketing_strategist", |
| "generated": "{\"action\":\"call_agent\",\"agent\":\"marketing_strategist\",\"budget\":{\"max_tokens\":4096},\"context_refs\":[\"user_request\"],\"expected_output\":\"completed_task_with_evidence\",\"instruction\":\"You are a senior automotive-tech GTM lead. Build the complete go-to-market plan for a connected-car companion app and dealer-services platform, sold to vehicle owners, dealership staff, and fleet managers.\\n\\n## Positioning\\nState the core promise: one platform that keeps vehicles running longer, service costs lower, and ownership simpler — for drivers, dealers, and fleets. Anchor it in the platform's real-time diagnostics, scheduled maintenance, and dealer-booking integration.\\n\\n## ICP & Segmentation\\n- **Vehicle owners:** new-car buyers, EV adopters, and owners of vehicles past warranty (where service costs rise). Segment by vehicle age, EV vs. ICE, and subscription status.\\n- **Dealership staff:** service advisors, sales reps, and service managers. Segment by store size and service-bay capacity.\\n- **Fleet managers:** corporate fleets, rental companies, and last-mile delivery fleets. Segment by fleet size and vehicle mix.\\n\\n## Messaging Pillars\\n1. **Own your vehicle:** proactive maintenance, no surprise repairs.\\n2. **Own your dealer relationship:** transparent pricing, one-click service booking.\\n3. **Own your fleet costs:** real-time health data, route-optimized service scheduling.\\n\\n## Channel Mix\\n- **Dealer networks:** dealer-partner onboarding, co-branded service reminders, service-advisor training.\\n- **Auto review sites:** targeted reviews, service-appointment conversion tracking, reputation management.\\n- **In-vehicle touchpoints:** OTA app updates, service-intent push notifications, connected-service reminders.\\n\\n## 90-Day Timeline\\n- **Days 1–20:** dealer-partner onboarding, app store and review-site presence, fleet pilot selection.\\n- **Days 21–50:** owner acquisition push (EV and post-warranty segments), service-advisor training, fleet pilot launch.\\n- **Days 51–90:** review-site reputation push, fleet expansion, 30-day retention and NPS review.\\n\\n## Budget Allocation\\nSplit across dealer-partner enablement, review-site and in-vehicle acquisition, and fleet pilot expansion — specify percentages.\\n\\n## KPIs\\n- Owner: app installs, service-appointment conversion, retention at 30/90 days, NPS.\\n- Dealer: service-appointment volume, advisor adoption, review-site sentiment.\\n- Fleet: vehicle uptime, service-cost variance, platform adoption.\\n\\n## Constraints\\nNo dealer-partner incentives without a signed agreement; no in-vehicle messaging without OEM partnership or explicit owner opt-in.\\n\\n## Done When\\nA single document covering all sections above, approved by an executive team and executable by marketing starting Monday.\",\"model\":\"auto\"}", |
| "source_id": "task-0132", |
| "valid_json": true, |
| "valid_schema": true |
| }, |
| { |
| "category": "marketing_collateral", |
| "complete_contract": true, |
| "correct_agent": true, |
| "error": null, |
| "expected_agent": "marketing_copywriter", |
| "generated": "{\"action\":\"call_agent\",\"agent\":\"marketing_copywriter\",\"budget\":{\"max_tokens\":4096},\"context_refs\":[\"user_request\"],\"expected_output\":\"completed_task_with_evidence\",\"instruction\":\"You are a senior healthcare marketing copywriter who has shipped patient-facing digital health products through HIPAA-compliant review. Write the complete, conversion-optimized landing page copy for a patient-facing telehealth and care-coordination platform serving patients, clinicians, and care coordinators.\\n\\n## Deliverables\\n\\n1. **Hero headline variants** (3–5 options) — patient-first framing, benefit-driven, under 12 words. A/B-testable.\\n2. **Benefit sections** (4–5) with proof points: appointment scheduling, secure video visits, care-team coordination, medication management, and care-plan sharing. Every claim must be traceable to a clinical-safety review point.\\n3. **Objection-handling blocks** — privacy/PHI concerns (\\\"Is my data safe?\\\"), appointment reliability, and clinician availability.\\n4. **Social-proof placement** — patient testimonials, clinician quotes, and care-coordinator stats; each tagged with review status (e.g., \\\"PHI-safe for public display\\\") and source.\\n5. **CTA copy** — distinct CTAs for patients (\\\"Book your first visit\\\"), clinicians (\\\"Join our network\\\"), and care coordinators (\\\"Connect your patients\\\").\\n\\n## Constraints\\n\\n- **HIPAA/PHI compliance**: no patient names, PHI, or specific clinical outcomes in testimonials; no unreviewed claims about treatment efficacy. Flag every claim with its review status.\\n- **Clinical-safety review**: any claim about outcomes, safety, or care quality must cite the review process.\\n- **Tone**: warm, trustworthy, and empowering — never clinical jargon for patients.\\n\\n## Format\\n\\n- Wireframe-annotated: section → headline → body → CTA → review-status tag.\\n- Deliverables: complete, ready for design handoff; no placeholders.\\n\\n## Done when\\n\\nAll sections are annotated with placement rationale, every claim is review-tracked, and the copy is conversion-optimized for all three audiences without violating compliance boundaries.\",\"model\":\"auto\"}", |
| "source_id": "task-0241", |
| "valid_json": true, |
| "valid_schema": true |
| }, |
| { |
| "category": "documentation", |
| "complete_contract": true, |
| "correct_agent": true, |
| "error": null, |
| "expected_agent": "technical_writer", |
| "generated": "{\"action\":\"call_agent\",\"agent\":\"technical_writer\",\"budget\":{\"max_tokens\":4096},\"context_refs\":[\"user_request\"],\"expected_output\":\"completed_task_with_evidence\",\"instruction\":\"You are the API documentation lead for a connected-car companion app and dealer-services platform. Produce the complete public API reference for the platform.\\n\\n## Context\\nThe API powers vehicle telemetry, remote diagnostics, service scheduling, and dealer-partner integrations. It must respect functional-safety standards (ISO 21434/14229-1), OTA update integrity (signed, verified payloads), and dealer-franchise agreements (no unauthorized access to dealer data or pricing). The sandbox environment is the target for the quickstart.\\n\\n## Deliverables\\n1. **Quickstart** — under five minutes, runnable as-is: register a developer account, obtain an API key, call a vehicle-telemetry endpoint, and interpret the response. Include a curl example and a minimal Python snippet.\\n2. **Endpoint catalog** — every endpoint: path, method, purpose, parameters (path/query/body), response schema, and example. Group by domain: vehicle, diagnostics, service, dealer, OTA, and admin.\\n3. **Auth model** — API key, OAuth 2.0 for dealer-partner integrations, and vehicle identity tokens; token scopes and rotation; refresh semantics.\\n4. **Error codes** — standard HTTP codes plus domain-specific ones (e.g., `409.100` for conflicting OTA update state, `403.200` for unauthorized dealer-data access).\\n5. **Rate limits** — per-key limits, per-vehicle limits, and a burst policy.\\n6. **Versioning policy** — semantic versioning, deprecation schedule, and sandbox vs. production parity.\\n\\n## Constraints\\n- No endpoint may expose dealer pricing, franchise territory data, or vehicle-control commands without explicit safety-approval flags.\\n- OTA update payloads must be signed; the reference must document the signature format and verification requirement.\\n- All examples must be runnable against the sandbox environment as written.\\n\\n## Done when\\n- Every endpoint is documented with parameters, examples, and error codes.\\n- The quickstart succeeds in under five minutes.\\n- Auth, rate limits, and versioning are explicit.\\n- Safety- and franchise-compliance constraints are enforced in examples.\",\"model\":\"auto\"}", |
| "source_id": "task-0372", |
| "valid_json": true, |
| "valid_schema": true |
| }, |
| { |
| "category": "sales_enablement", |
| "complete_contract": true, |
| "correct_agent": true, |
| "error": null, |
| "expected_agent": "sales_specialist", |
| "generated": "{\"action\":\"call_agent\",\"agent\":\"sales_specialist\",\"budget\":{\"max_tokens\":4096},\"context_refs\":[\"user_request\"],\"expected_output\":\"completed_task_with_evidence\",\"instruction\":\"You are a senior health-tech sales leader who has sold EHR-integrated care coordination platforms to health systems and payers. Build a 12-slide sales pitch deck for a patient-facing telehealth and care-coordination platform serving patients, clinicians, and care coordinators, ready for a new sales hire to deliver credibly after one read-through.\\n\\n## Slide structure (12 slides)\\n1. **Cover** — platform name, tagline, contact.\\n2. **Problem framing** — care fragmentation: patients fall through the cracks between appointments, EHRs don't talk to each other, care coordinators drown in pagers and spreadsheets. One stat: 30% of patients miss follow-up appointments within 30 days of discharge.\\n3. **Cost of inaction** — avoidable ED visits, readmissions, and malpractice risk from coordination failures; quantify per-member-per-month cost to a health system or payer.\\n4. **Solution narrative** — the platform: secure patient portal, automated appointment reminders, clinician alerting, and care-coordinator workflow. One-sentence value prop per stakeholder.\\n5. **Proof — EHR/claims data** — real-world metrics: 25% reduction in avoidable ED visits, 18% fewer 30-day readmissions, 90% appointment adherence. Source: [de-identified EHR/claims dataset].\\n6. **Proof — patient-reported outcomes** — satisfaction scores, engagement rates, and NPS from live deployments.\\n7. **Workflow demo** — 1–2 annotated screenshots: patient scheduling, clinician alert, coordinator caseload view.\\n8. **Pricing architecture** — per-member-per-month tiered pricing, per-facility licensing, optional implementation fee. No hidden costs.\\n9. **Integration & security** — HIPAA-compliant, FHIR-ready, SOC 2 attestation, and typical integration timeline.\\n10. **Referenceable customers** — 1–2 named health systems or payer partners with permission.\\n11. **ROI case study** — one concrete example: cost savings vs. implementation spend over 12 months.\\n12. **Next steps** — pilot proposal, integration scoping, contract timeline.\\n\\n## Deliverables\\n- Slide deck (PDF or PPTX outline) with speaker notes.\\n- Talk track per slide: 60–90 seconds, aligned with the data and narrative.\\n- Objection notes: 5–7 common objections (\\\"too expensive,\\\" \\\"integration risk,\\\" \\\"clinician adoption\\\") with rebuttals and data.\\n- One-page executive summary.\\n\\n## Constraints\\n- No marketing fluff; every claim must be backed by the cited data.\\n- Language must be neutral and clinician-respectful — this is a B2B2C pitch.\\n- No invented metrics; flag any placeholder data that needs real sourcing.\\n\\n## Done when\\n- 12 slides, complete talk track, and objection notes.\\n- New hire can deliver the pitch credibly after one read-through.\\n- Every claim is traceable to a cited data source or referenceable customer.\",\"model\":\"auto\"}", |
| "source_id": "task-0461", |
| "valid_json": true, |
| "valid_schema": true |
| }, |
| { |
| "category": "data_analysis", |
| "complete_contract": true, |
| "correct_agent": true, |
| "error": null, |
| "expected_agent": "data_analyst", |
| "generated": "{\"action\":\"call_agent\",\"agent\":\"data_analyst\",\"budget\":{\"max_tokens\":4096},\"context_refs\":[\"user_request\"],\"expected_output\":\"completed_task_with_evidence\",\"instruction\":\"You are a senior analytics engineer at a connected-car companion app and dealer-services platform. Design and specify the executive KPI dashboard that the CEO, CMO, and COO review weekly.\\n\\n## Context\\nThe platform connects vehicle owners to dealer service networks, selling extended warranties and service contracts. Data comes from vehicle telemetry (telematics, OBD, connected infotainment), dealer service histories (DMS), and warranty claims (WMS). The dashboard must reflect real business health, not vanity metrics.\\n\\n## Deliverable\\n8–10 metrics, each with:\\n- **Name** — precise, unambiguous (e.g., \\\"Service-Driven Retention Rate\\\")\\n- **Calculation** — exact formula, denominator, numerator, and any windowing (e.g., trailing 90 days, rolling 12-month cohort)\\n- **Data source** — telemetry platform, dealer DMS, or warranty WMS, with field names\\n- **Refresh cadence** — daily, weekly, or monthly; justify the choice\\n- **Target/threshold** — a single number or range; a warning and critical tier\\n- **Decision** — the specific action the metric should trigger (e.g., \\\"escalate to dealer ops if below threshold for 2 consecutive weeks\\\")\\n- **Owner** — named role (e.g., \\\"Head of Dealer Network\\\")\\n\\n## Constraints\\n- No metrics about app downloads, DAU, or NPS — these are engagement metrics, not business-health metrics.\\n- Every metric must be traceable to a revenue or cost variable: warranty sales, service revenue, dealer-partner retention, or claims cost.\\n- State any assumptions explicitly (e.g., \\\"claims cost includes only paid claims, not open reserves\\\").\\n\\n## Format\\nTable with columns: Name | Calculation | Data Source | Refresh Cadence | Target/Threshold | Decision | Owner\\n\\n## Done When\\nImplementable by an analytics engineer without clarifying questions; every metric has an owner and a decision trigger; no engagement vanity metrics appear.\",\"model\":\"auto\"}", |
| "source_id": "task-0572", |
| "valid_json": true, |
| "valid_schema": true |
| }, |
| { |
| "category": "product_design", |
| "complete_contract": true, |
| "correct_agent": true, |
| "error": null, |
| "expected_agent": "product_designer", |
| "generated": "{\"action\":\"call_agent\",\"agent\":\"product_designer\",\"budget\":{\"max_tokens\":4096},\"context_refs\":[\"user_request\"],\"expected_output\":\"completed_task_with_evidence\",\"instruction\":\"You are a senior product manager for a connected-car companion app and dealer-services platform serving vehicle owners, dealership staff, and fleet managers. Write a complete PRD for the next major feature, grounded in vehicle telemetry, service histories, and warranty claims data.\\n\\n## Deliverables\\n\\n1. **Problem Statement** — quantify the gap: missed service appointments, warranty-claim processing delays, and technician dispatch inefficiencies, each with a source (telemetry, service history, or warranty-claims data). State the business impact on warranty costs, service revenue, and customer satisfaction.\\n2. **Goals and Non-Goals** — 3–5 measurable goals; 3–4 explicit non-goals (e.g., direct warranty-claim adjudication, replacing dealer-internal DMS, or handling PII beyond VIN and mileage).\\n3. **User Stories with Acceptance Criteria** — separate sets for vehicle owners (reminders, service-appointment booking, repair-status tracking), dealership staff (technician dispatch, warranty-claim submission, service-history review), and fleet managers (multi-vehicle scheduling, maintenance-cost reporting, recall management). Every acceptance criterion must be testable.\\n4. **Edge Cases** — at least 6: vehicle offline for extended periods, warranty-claim rejection by the OEM, recall-action conflicts with local regulations, dealer-appointment no-shows, fleet-manager permission boundaries across brands, and OTA-update failure during a scheduled service.\\n5. **Rollout Plan** — phased: pilot fleet, dealer-partner rollout, full launch; each phase with a success gate and rollback trigger.\\n6. **Success Metrics** — 3–5 per stakeholder group, with targets and measurement method.\\n7. **Regulatory and Contractual Implications** — a dedicated section covering:\\n - Functional-safety standards: how the feature's logic (e.g., automated service recommendations) must be designed to avoid safety-critical misjudgments, with a recommendation on ISO 26262 classification.\\n - OTA update integrity: signature verification, rollback protection, and update-window restrictions during active vehicle operation.\\n - Dealer-franchise agreements: territory restrictions, OEM-approval requirements for any feature that affects warranty-claim submission or service pricing.\\n\\n## Constraints\\n\\n- No feature may override or replace dealer-internal DMS or OEM warranty-claim adjudication systems.\\n- Any recommendation logic must be explainable to the vehicle owner.\\n- All PII must be handled per applicable privacy regulations.\\n\\n## Done When\\n\\nEngineering can estimate and begin building without a kickoff meeting: every story has acceptance criteria, edge cases are documented, rollout gates are explicit, and safety/OTA/franchise implications are addressed.\\n\\n## Done When Not\\n\\nA kickoff meeting is required to resolve ambiguity about warranty-claim workflow, safety classification, or franchise-approval requirements.\",\"model\":\"auto\"}", |
| "source_id": "task-0672", |
| "valid_json": true, |
| "valid_schema": true |
| }, |
| { |
| "category": "financial_planning", |
| "complete_contract": true, |
| "correct_agent": true, |
| "error": null, |
| "expected_agent": "financial_analyst", |
| "generated": "{\"action\":\"call_agent\",\"agent\":\"financial_analyst\",\"budget\":{\"max_tokens\":4096},\"context_refs\":[\"user_request\"],\"expected_output\":\"completed_task_with_evidence\",\"instruction\":\"You are a senior FP&A analyst at an automotive tech company. Build the driver-based revenue model for the connected-car companion app and dealer-services platform, grounded in vehicle telemetry, service histories, and warranty claims data.\\n\\n## Required structure\\n\\n1. **Driver tree** — acquisition (new vehicle installs, app download rate, dealer-partner integrations), conversion (first-service booking, subscription upgrade, connected-service activation), retention (telemetry-driven service reminders, warranty-claim-triggered upsell, dealer-visit frequency), and expansion (additional connected services, premium diagnostics, dealer-partner revenue share). Each driver must trace to a specific data source (telemetry, service history, warranty claims) and include its formula.\\n2. **24-month monthly projections** — revenue by stream (subscription, service bookings, warranty-claim upsell, dealer-partner revenue share), CAC, LTV, and margin. Include a base case and bull/bear scenarios toggled by vehicle install base growth, app engagement rate, and service-booking conversion.\\n3. **Assumption documentation** — every assumption tagged with source (e.g., \\\"telemetry: OTA update opt-in rate from vehicle platform logs\\\") and sensitivity (±10% change → revenue impact).\\n4. **Skeptic-proofing** — a section addressing the board's likely questions: why install base assumptions are realistic given vehicle production and OTA update rollout timing, why service-booking conversion rates are achievable, and why warranty-claim upsell attach rates are sustainable.\\n\\n## Constraints\\n\\n- No revenue from data sales or ad inventory — model must be subscription and service-booking based.\\n- All drivers must flow through to revenue in every scenario; test one driver at a time.\\n- State explicitly where warranty-claim data is shared with OEMs and the privacy/compliance implications.\\n\\n## Done when\\n\\n- Every driver change flows correctly to revenue across all scenarios.\\n- Every assumption is sourced and sensitized.\\n- The model withstands questions about install-base timing, service-booking conversion, and warranty-claim attach rates.\",\"model\":\"auto\"}", |
| "source_id": "task-0772", |
| "valid_json": true, |
| "valid_schema": true |
| }, |
| { |
| "category": "legal_compliance", |
| "complete_contract": true, |
| "correct_agent": true, |
| "error": null, |
| "expected_agent": "compliance_specialist", |
| "generated": "{\"action\":\"call_agent\",\"agent\":\"compliance_specialist\",\"budget\":{\"max_tokens\":4096},\"context_refs\":[\"user_request\"],\"expected_output\":\"completed_task_with_evidence\",\"instruction\":\"You are a senior automotive-data privacy counsel drafting the privacy policy for a connected-car companion app and dealer-services platform. Vehicle telemetry (driving behavior, location, vehicle health), service histories, and warranty claims data flow through this platform to owners, dealerships, and fleet managers.\\n\\n## What to produce\\n\\n1. **Privacy policy** — plain language, one version covering vehicle owners, dealership staff, and fleet managers, with a \\\"What data we collect\\\" table and a \\\"How we use your data\\\" table. Cover:\\n - Telemetry: location, driving behavior, vehicle health (engine, battery, brakes), OTA update logs, and diagnostic trouble codes\\n - Service histories: repair orders, parts used, labor, technician notes\\n - Warranty claims: claim status, manufacturer approval, repair costs\\n - Dealer-franchise agreements: data access rights tied to franchise territory and service-authorization status\\n - Functional-safety standards: how OTA update integrity (signed, verified, rollback-capable) is protected and audited\\n - GDPR/CCPA: consent mechanisms, right to access/delete, and jurisdictional carve-outs for safety-critical data that cannot be deleted (e.g., airbag deployment logs tied to incident investigations)\\n - Data retention: warranty-claim data retained per manufacturer warranty periods, OTA logs retained per functional-safety audit requirements, service histories per dealer record-retention rules\\n\\n2. **Data-inventory worksheet** — columns: Data field, Source (vehicle, dealer, owner), Purpose, Recipients (dealer, manufacturer, insurer, owner), Retention period, Legal basis (consent, contract, legal obligation), Safety-critical flag, OTA-update-related flag, GDPR/CCPA-sensitive flag.\\n\\n## Constraints\\n\\n- Do not invent data fields or retention periods — use placeholders with `[bracketed rationale]` so the company must fill them in.\\n- Flag every open question: franchise-territory data-access conflicts, warranty-claim data sharing with manufacturers, OTA-update integrity audit frequency, and jurisdictional conflicts where safety-critical data cannot be deleted.\\n- Use a consistent tone: owner-friendly, dealer-accurate, fleet-manager precise.\\n\\n## Done when\\n\\n- Policy covers all three audiences and all data types.\\n- Inventory worksheet is complete and fillable.\\n- Every open question is explicitly flagged; nothing is hidden.\\n- Counsel can review with no gaps in coverage.\",\"model\":\"auto\"}", |
| "source_id": "task-0872", |
| "valid_json": true, |
| "valid_schema": true |
| }, |
| { |
| "category": "customer_support", |
| "complete_contract": true, |
| "correct_agent": true, |
| "error": null, |
| "expected_agent": "customer_support_specialist", |
| "generated": "{\"action\":\"call_agent\",\"agent\":\"customer_support_specialist\",\"budget\":{\"max_tokens\":4096},\"context_refs\":[\"user_request\"],\"expected_output\":\"completed_task_with_evidence\",\"instruction\":\"You are a senior support-content writer for a patient-facing telehealth and care-coordination platform. Your job is to build the help center that keeps patients, clinicians, and care coordinators self-served.\\n\\n## Context\\nPatients book appointments, join video visits, review care plans, and message care teams. Clinicians conduct visits, document notes, and assign care tasks. Care coordinators manage caseloads, schedule appointments, and coordinate with providers. Every article must stand alone — no article should require reading another to complete the task.\\n\\n## Deliverable\\nTen articles, each fully self-contained and capable of resolving its ticket type without agent contact. Structure each article identically:\\n\\n1. **Task-oriented title** — how users would search it (\\\"join telehealth visit on iPhone,\\\" \\\"reset password for care portal,\\\" \\\"schedule a follow-up appointment\\\")\\n2. **Numbered steps** — exact, platform-specific, with device/browser notes where relevant\\n3. **Expected outcome** — what the user should see or be able to do next\\n4. **Troubleshooting branch** — 3–4 common failure modes (\\\"visit not loading,\\\" \\\"appointment not found,\\\" \\\"password reset email not received\\\") with specific fixes, including HIPAA-compliant messaging notes (never suggest sharing PHI outside the platform)\\n\\n## Ticket themes to cover\\n- Joining or troubleshooting a telehealth visit (video, audio, or phone-only)\\n- Scheduling, rescheduling, or canceling appointments\\n- Accessing and updating the care plan or care-team directory\\n- Resetting passwords or recovering account access\\n- Messaging the care team through the portal\\n- Understanding insurance and billing questions (copay, coverage, billing codes)\\n- Managing appointment reminders and notifications\\n- Accessing visit notes or clinical documentation\\n- Coordinating care-team changes or adding a family caregiver\\n- Technical issues (camera/mic not working, app crashes, browser compatibility)\\n\\n## Constraints\\n- No PHI or protected health information may be discussed outside the secure portal; never suggest sharing PHI via email, text, or phone.\\n- All articles must pass a patient-readability test (plain language, no medical jargon).\\n- Every troubleshooting branch must include a platform-specific fix, not generic tech support.\\n\\n## Done when\\nTen articles exist, each fully resolvable without agent contact, and every ticket theme is covered at least once.\",\"model\":\"auto\"}", |
| "source_id": "task-0961", |
| "valid_json": true, |
| "valid_schema": true |
| } |
| ], |
| "sample_count": 10, |
| "sampling": "first held-out example per category" |
| } |
|
|
