Update app.py

app.py

@@ -672,14 +672,37 @@ def execute_in_sandbox(script: str, dataframes: List[Any]) -> str:
 def _create_python_script(user_scenario: str, schema_context: str) -> str:
     EXPERT_ANALYTICAL_GUIDELINES = """
 --- EXPERT ANALYTICAL GUIDELINES ---
-When writing your script, you MUST follow these expert business rules:
-1.  **Linking Datasets Rule:** If you need to connect facilities to health zones when the 'zone' column is not in the facility list,
-    you must first identify the high-priority zone from the beds data, then find the major city (by facility count) in the facility list,
-    and *then* assess that city's capacity. Do not try to filter the facility list by a 'zone' column if it does not exist in the schema.
-2.  **Prioritization Rule:** To prioritize locations, you MUST combine the most recent population data with specific high-risk health indicators
-    to create a multi-factor risk score.
-3.  **Capacity Calculation Rule:** For capacity over a 3-month window, assume **60 working days**.
-4.  **Cost Calculation Rule:** Sum 'Startup cost' and 'Ongoing cost' per person before multiplying.
+When writing your script, you MUST follow these expert analytical principles:
+
+**DATA INTEGRATION & LINKING:**
+1.  When linking datasets, identify the correct join keys by examining column names and values. Never assume column names match across datasets.
+2.  If a required column doesn't exist in a dataset, derive it from related data or clearly note its absence in the output.
+3.  Use the most recent/relevant time period data when multiple periods exist (e.g., prefer 2021 over 2013 census data if both available).
+
+**AGGREGATION & GROUPING:**
+4.  When asked about "specialties," "categories," or "types," group by the broadest categorical column first (e.g., 'Specialty' not 'Procedure').
+5.  When asked about specific items, use the most granular level (e.g., specific procedures, individual facilities).
+6.  Always verify the appropriate level of aggregation matches the user's question.
+
+**PRIORITIZATION & RANKING:**
+7.  To prioritize locations/facilities, create a composite risk score combining: (a) population/volume, (b) relevant health indicators, and (c) recency of data.
+8.  When ranking, consider both absolute values AND relative performance against benchmarks (provincial/national averages).
+9.  Include sample sizes/record counts alongside rankings to indicate statistical reliability.
+
+**CALCULATIONS & ESTIMATES:**
+10. For time-based capacity calculations, use standard assumptions: 60 working days per 3-month period, 5 days/week, unless data specifies otherwise.
+11. For cost calculations, always separate and sum component costs (startup + ongoing + variable) before multiplying by volume.
+12. When extracting numeric values from text fields, use robust parsing: strip currency symbols, handle ranges (take midpoint), convert percentages.
+
+**UNITS & VALIDATION:**
+13. Preserve and label units correctly: percentages (%), currency (CAD/USD), time (days/weeks), clinical measures (mmHg for BP, % for A1c, kg/m² for BMI).
+14. Validate calculated values against reasonable ranges (e.g., A1c typically 4-14%, BP typically 60-200 mmHg).
+15. Flag outliers or unexpected values in the output for human review.
+
+**OUTPUT COMPLETENESS:**
+16. For each evaluation question, ensure the JSON output contains all data needed to answer it fully.
+17. Include both raw values AND calculated metrics (averages, percentages, rankings).
+18. When comparing to benchmarks, include both the benchmark value and the comparison result.
 """
     prompt_for_coder = f"""\
 You are an expert Python data scientist. Your job is to write a script to extract the data needed to answer the user's request.
@@ -732,7 +755,7 @@ def _generate_long_report(prompt: str) -> str:
 
 def _generate_final_report(user_scenario: str, validated_json_str: str) -> str:
     prompt_for_writer = f"""\
-You are an expert management consultant and data analyst.
+You are an expert healthcare management consultant and data analyst.
 A data science script has run to extract key findings. You have the user's original request and the validated JSON data.
 
 Your task is to synthesize these validated findings into a single, comprehensive, and professional report that directly answers all of the user's questions with detailed justifications.
@@ -745,11 +768,41 @@ Your task is to synthesize these validated findings into a single, comprehensive
 {validated_json_str}
 --- END VALIDATED DATA ---
 
+--- ANALYTICAL INTERPRETATION GUIDELINES ---
+When writing your report, follow these principles:
+
+**ACCURACY & UNITS:**
+- Report numerical values with appropriate precision (1-2 decimal places for percentages, whole numbers for counts).
+- Always include correct units: % for percentages, days for wait times, $ for costs, mmHg for blood pressure, % for A1c, kg/m² for BMI.
+- Verify that values make clinical/operational sense before reporting (e.g., A1c should be 4-14%, not measured in mmHg).
+
+**CONTEXT & BENCHMARKS:**
+- Compare findings against relevant benchmarks (provincial averages, national standards, historical baselines).
+- Explain what "good" vs "poor" performance means in context.
+- Quantify differences (e.g., "50 days above average" not just "higher than average").
+
+**CAUSATION & INTERPRETATION:**
+- Distinguish correlation from causation; avoid overstating causal claims.
+- Consider confounding factors (case complexity, patient demographics, resource constraints).
+- Acknowledge data limitations and uncertainty.
+
+**RECOMMENDATIONS:**
+- Make recommendations specific, actionable, and tied directly to the data findings.
+- Prioritize recommendations by impact and feasibility.
+- Include implementation considerations (resources needed, timeline, risks).
+- Suggest metrics for monitoring success.
+
+**COMPLETENESS:**
+- Address EVERY evaluation question explicitly.
+- If data is insufficient to answer a question fully, state what's missing and provide the best available answer.
+- Cross-reference related findings to provide a coherent narrative.
+
 Now, write the final, polished report. The report MUST:
 1.  Follow the "Expected Output Format" requested by the user.
 2.  Use tables, bullet points, and DETAILED narrative justifications for each recommendation.
 3.  Synthesize the validated data into actionable insights. Do not just copy the raw numbers; interpret them.
 4.  Ensure you fully address ALL evaluation questions, especially the final recommendations.
+5.  Verify all units and values are clinically/operationally plausible before including them.
 """
     return _generate_long_report(prompt_for_writer)