| 1. Negation + Cloud/Platform Combos (tests polarity + bridge) | |
| Show me the worst 5 performing queries for targets not on cloud | |
| Average anomaly score for all targets except those running on Linux or AIX | |
| Give me the top 3 worst SQL queries from targets that are not cloud-hosted and not demo | |
| Worst anomalies excluding any target on Linux or Windows | |
| 2. Vague / Multi-Condition Traps (tests holistic metadata detection) | |
| Show the worst performing queries on all cloud targets that are not Oracle | |
| Average anomaly score for targets that are on cloud but stopped | |
| *******Top 3 slowest queries from non-Linux targets that are still active and not demo | |
| ***********Performance issues for targets that are either on cloud or in us-east-1 but not demo | |
| 3. Referential / Follow-up Style (tests memory + bridge persistence) | |
| Show me the worst queries from the ones that are on cloud (after asking about all targets first) | |
| *********What about the expensive ones that are not on Linux? (follow-up style) | |
| Give me the average anomaly score for those targets we talked about that are not running on cloud | |
| 4. Aggregation + Complex Bridge Cases | |
| Give me the average anomaly score grouped by target for all non-cloud databases that use Oracle or PostgreSQL | |
| Show the total number of severe anomalies across targets that are on cloud but not in us-east-1 | |
| *******Minimum and maximum anomaly scores for targets not running on Linux and not demo | |
| 5. Evil Router-Trap + Digest/Plan Combos (tests intent override) | |
| Show me the worst 3 performing queries for all targets on cloud and also show their execution plans | |
| What do the actual SQL queries look like for the worst anomalies on targets not running on cloud? | |
| Dig up the execution plans for the top 3 bottlenecks on cloud-hosted targets that are not demo | |
| 18. Bonus Nuclear Test (maximum brittleness) | |
| Give me the worst performing SQL queries for targets that are cloud but not Oracle and not stopped and not in us-east-1 | |
| derby_target_cases = [ | |
| ("List all targets running on Linux.", "SELECT * FROM target WHERE hlc_server_platform = 'Linux'"), | |
| ("How many MySQL databases are there?", "SELECT count(*) FROM target WHERE hlc_database_type = 'MySQL'"), | |
| ("Show all cloud-hosted targets.", "SELECT * FROM target WHERE is_on_cloud = 'true'"), | |
| ("List all targets with status Running.", "SELECT name FROM target WHERE db_target_status = 'Running'"), | |
| ("Show all Oracle targets.", "SELECT * FROM target WHERE hlc_database_type = 'Oracle'"), | |
| ("How many targets are deployed on cloud?", "SELECT count(*) FROM target WHERE is_on_cloud = 'true'"), | |
| ("List demo targets.", "SELECT * FROM target WHERE is_demo_target = 'true'"), | |
| ] | |
| derby_digest_cases = [ | |
| ("Show the SQL query text for digest 'NULL--abc123'.", "SELECT digest_text FROM performance_schema WHERE digest = 'NULL--abc123'"), | |
| ("Show the plain text execution plan for the digest 'myschema--def456'.", "SELECT T2.sql_plain_text_plan FROM performance_schema AS T1 JOIN sql_plan AS T2 ON T1.sql_plan_id = T2.id WHERE T1.digest = 'myschema--def456'"), | |
| ("Get the execution plan for digest 'NULL--999abc'.", "SELECT T2.sql_plan FROM performance_schema AS T1 JOIN sql_plan AS T2 ON T1.sql_plan_id = T2.id WHERE T1.digest = 'NULL--999abc'"), | |
| ] | |
| derby_template_cases = [ | |
| ("How many templates are system defaults?", "SELECT count(*) FROM template WHERE is_system_default = 'true'"), | |
| ("List all dynamic templates.", "SELECT * FROM template WHERE is_dynamic = 'true'"), | |
| ] | |
| derby_report_cases = [ | |
| ("List all scheduled reports.", "SELECT * FROM report WHERE is_scheduled = 'true'"), | |
| ("Show all reports that send PDF emails.", "SELECT * FROM report WHERE is_email_pdf_report_enabled = 'true'"), | |
| ("How many reports run daily?", "SELECT count(*) FROM report WHERE scheduler_params_period = 'DAILY'"), | |
| ("List reports scheduled on Monday.", "SELECT * FROM report WHERE scheduler_params_day_of_week = 'MONDAY'"), | |
| ] | |
| derby_integration_cases = [ | |
| ("List all active integrations.", "SELECT * FROM integration WHERE enabled = 'true'"), | |
| ("Show all disabled integrations.", "SELECT * FROM integration WHERE enabled = 'false'"), | |
| ("How many integrations are enabled?", "SELECT count(*) FROM integration WHERE enabled = 'true'"), | |
| ] | |
| derby_collector_cases = [ | |
| ("List all active collectors.", "SELECT * FROM dbactdc_instance WHERE status = 'Active'"), | |
| ("Show all remote collector instances.", "SELECT * FROM dbactdc_instance WHERE is_remote = 'true'"), | |
| ("How many collectors are currently inactive?", "SELECT count(*) FROM dbactdc_instance WHERE status = 'Inactive'"), | |
| ] | |
| derby_pricing_cases = [ | |
| ("Show all AWS cloud pricing entries.", "SELECT * FROM cloud_database_pricing WHERE cloud_database_provider = 'AWS'"), | |
| (******************"What is the average credit cost across all cloud providers?", "SELECT avg(credit_cost) FROM cloud_database_pricing"), | |
| ] | |
| influx_cases = [ | |
| ("Show all performance bottlenecks.", "SELECT target, metric, data_value_double, time FROM sys_target_alerts WHERE is_demo_alert != 'true'"), | |
| ("What are the anomalies detected for each target?", "SELECT target, metric, data_value_double, time FROM sys_target_alerts WHERE is_demo_alert != 'true'"), | |
| ("Show all spikes excluding demo data.", "SELECT target, metric, data_value_double, time FROM sys_target_alerts WHERE is_demo_alert != 'true'"), | |
| ("What is the average anomaly score for each target?", "SELECT target, avg(data_value_double) FROM sys_target_alerts WHERE is_demo_alert != 'true' GROUP BY target"), | |
| ("****************What is the minimum anomaly value for each target?", "SELECT target, min(data_value_double) FROM sys_target_alerts WHERE is_demo_alert != 'true' GROUP BY target"), | |
| ("Show the total anomaly count per target.", "SELECT target, count(*) FROM sys_target_alerts WHERE is_demo_alert != 'true' GROUP BY target"), | |
| ("Find the top 3 worst performing queries.", "SELECT target, metric, data_value_double, time FROM sys_target_alerts WHERE is_demo_alert != 'true' ORDER BY data_value_double DESC LIMIT 3"), | |
| ("Show the top 10 highest anomaly scores.", "SELECT target, metric, data_value_double, time FROM sys_target_alerts WHERE is_demo_alert != 'true' ORDER BY data_value_double DESC LIMIT 10"), | |
| ("Find the bottom 5 least severe alerts.", "SELECT target, metric, data_value_double, time FROM sys_target_alerts WHERE is_demo_alert != 'true' ORDER BY data_value_double ASC LIMIT 5"), | |
| ("What capture sources are used for alerts?", "SELECT target, capture, data_value_double FROM sys_target_alerts WHERE is_demo_alert != 'true'"), | |
| ("Show capture methods for all performance issues.", "SELECT target, capture, data_value_double FROM sys_target_alerts WHERE is_demo_alert != 'true'"), | |
| ("Which SQL digests have the worst performance bottlenecks?", "SELECT target, metric, data_value_double, time FROM sys_target_alerts WHERE is_demo_alert != 'true' ORDER BY data_value_double DESC LIMIT 5"), | |
| ("Show top sql issues.", "SELECT target, metric, data_value_double, time FROM sys_target_alerts WHERE is_demo_alert != 'true' ORDER BY data_value_double DESC LIMIT 5"), | |
| ] | |
| coworker_cases = [ | |
| ("What is the current status of all my targets?", "SELECT name, db_target_status FROM target"), | |
| ("Give me 'MySQL_QUICK_1711_1' target information", "SELECT * FROM target WHERE name = 'MySQL_QUICK_1711_1'"), | |
| ("Give me the complete profile for the all target", "SELECT * FROM target"), | |
| ("Give me anomalies for all targets", "SELECT target, metric, data_value_double, time FROM sys_target_alerts WHERE is_demo_alert != 'true'"), | |
| (***********"How many targets we have?", "SELECT count(*) FROM target"), | |
| ("What are the top sql issue on 'DB_PROD_015' target", "SELECT target, metric, data_value_double, time FROM sys_target_alerts WHERE target = 'MySQL_QUICK_1711_1' AND is_demo_alert != 'true' ORDER BY data_value_double DESC LIMIT 5") | |
| ] | |
| multi_step_cases = [ | |
| "What is the actual SQL query text for the top 2 worst performing queries?", | |
| "Identify the single worst spike", | |
| "Get the SQL query texts causing anomalies.", | |
| "Show me the execution plans for the top 5 performance bottlenecks.", | |
| "What SQL is running on the most anomalous target?", | |
| "Show me the actual queries behind the top 3 spikes.", | |
| "********Which queries are causing the worst bottlenecks and what do they look like?", | |
| "Get execution plans for the top 2 anomalies.", | |
| "********I need to see the query text for the biggest bottleneck we have.", | |
| "Dig up the execution plans for the 4 most severe issues.", | |
| "********What does the actual SQL look like for the worst spike?" | |
| ] | |
| complex_conversational_cases = [ | |
| # Conversational Status & Platform | |
| ("Hey, can you just show me all the databases that are currently down?", "SELECT name FROM target WHERE db_target_status = 'Stopped'"), | |
| ("*******Which of my Linux targets are running Oracle?", "SELECT * FROM target WHERE hlc_server_platform = 'Linux' AND hlc_database_type = 'Oracle'"), | |
| # Implicit Math & Aggregation | |
| ("I need to know how much we are paying for AWS, just give me the average.", "SELECT avg(credit_cost) FROM cloud_database_pricing WHERE cloud_database_provider = 'AWS'"), | |
| ("Just count how many remote agents we have active right now.", "SELECT count(*) FROM dbactdc_instance WHERE is_remote = 'true' AND status = 'Active'"), | |
| # Complex Cross-Filtering | |
| ("Get me the scheduled reports that send out PDF emails.", "SELECT * FROM report WHERE is_scheduled = 'true' AND is_email_pdf_report_enabled = 'true'"), | |
| ("Show me the dynamic templates used for FinOps.", "SELECT * FROM template WHERE is_dynamic = 'true' AND template_type = 'FinOps'"), | |
| # Conversational Influx & Multi-Step Triggers | |
| ("Are there any severe issues on my systems right now?", "SELECT target, metric, data_value_double, time FROM sys_target_alerts WHERE is_demo_alert != 'true' ORDER BY data_value_double DESC LIMIT 5"), | |
| ("*********Give me the exact SQL queries that are responsible for the top 3 highest spikes.", "SELECT target, metric, data_value_double, time FROM sys_target_alerts WHERE is_demo_alert != 'true' ORDER BY data_value_double DESC LIMIT 3") | |
| ] | |
| grounded_lookup_cases = [ | |
| ("I need the full configuration profile for the DB_DEV_007 database.", "SELECT * FROM target WHERE name = 'DB_DEV_007'"), | |
| ("Is Cassandra_Cluster currently running or stopped?", "SELECT name, db_target_status FROM target WHERE name = 'Cassandra_Cluster'"), | |
| ("Show me all the PostgreSQL databases we are hosting on AIX machines.", "SELECT * FROM target WHERE hlc_database_type = 'PostgreSQL' AND hlc_server_platform = 'AIX'"), | |
| ("List all targets running on Windows that use SQLServer.", "SELECT * FROM target WHERE hlc_server_platform = 'Windows' AND hlc_database_type = 'SQLServer'"), | |
| ("List all targets deployed in the eu-central-1 region.", "SELECT * FROM target WHERE cloud_region = 'eu-central-1'"), | |
| ("Which of our databases are hosted on-prem?", "SELECT * FROM target WHERE cloud_region = 'on-prem'"), | |
| ("Give me the credit costs for Oracle databases located in us-east-1.", "SELECT name, hlc_credit_cost FROM target WHERE hlc_database_type = 'Oracle' AND cloud_region = 'us-east-1'"), | |
| ("Are there any performance bottlenecks for DB_DEV_013?", "SELECT target, metric, data_value_double, time FROM sys_target_alerts WHERE target = 'DB_DEV_013' AND is_demo_alert != 'true'") | |
| ] | |
| # ============================================================================= | |
| # NEW TRICKY / STRESS-TEST CASES (added here for fragility testing) | |
| # ============================================================================= | |
| tricky_stress_cases = [ | |
| # 1. Vague listing (rescue logic stress) | |
| ("Give me an overview of everything we monitor", "SELECT name, hlc_database_type, db_target_status FROM target"), | |
| ("Show me the current picture of all reports", "SELECT name, title, scheduler_params_period FROM report"), | |
| ("What does the collector landscape look like right now?", "SELECT name, status, platform FROM dbactdc_instance"), | |
| # 2. Multi-condition + lexicon conflicts | |
| ("Show me all Linux Oracle targets that are currently stopped", "SELECT * FROM target WHERE hlc_server_platform = 'Linux' AND hlc_database_type = 'Oracle' AND db_target_status = 'Stopped'"), | |
| ("List collectors that are monitoring Oracle databases on AWS", "SELECT * FROM dbactdc_instance WHERE is_remote = 'true'"), # Note: indirect, may need target join in real system | |
| ("How many active integrations do we have for targets running on-prem?", "SELECT count(*) FROM integration WHERE enabled = 'true'"), # Indirect — should not crash | |
| # 3. Negation + tricky phrasing | |
| ("Show me all performance issues for stopped targets", "SELECT target, metric, data_value_double, time FROM sys_target_alerts WHERE is_demo_alert != 'true' AND db_target_status != 'Stopped'"), | |
| ("Give me anomalies that are NOT coming from Cassandra_Cluster", "SELECT target, metric, data_value_double, time FROM sys_target_alerts WHERE target != 'Cassandra_Cluster' AND is_demo_alert != 'true'"), | |
| ("List every target that is NOT running on Linux or AIX", "SELECT * FROM target WHERE hlc_server_platform NOT IN ('Linux', 'AIX')"), | |
| # # 4. Ambiguous / conversational / follow-up style | |
| # ("Tell me about the ones that are down right now", "SELECT name, db_target_status FROM target WHERE db_target_status = 'Stopped'"), | |
| # ("What about the expensive ones?", "SELECT name, hlc_credit_cost FROM target WHERE hlc_credit_cost > 0 ORDER BY hlc_credit_cost DESC LIMIT 5"), | |
| # ("Show me the worst ones, but only the real ones, not the demo stuff", "SELECT target, metric, data_value_double, time FROM sys_target_alerts WHERE is_demo_alert != 'true' ORDER BY data_value_double DESC LIMIT 5"), | |
| # Bonus evil ones | |
| ("**************Give me the credit costs for all Oracle databases hosted on Linux in us-east-1", "SELECT name, hlc_credit_cost FROM target WHERE hlc_database_type = 'Oracle' AND hlc_server_platform = 'Linux' AND cloud_region = 'us-east-1'"), | |
| ("List all dynamic FinOps templates used by active collectors", "SELECT * FROM template WHERE is_dynamic = 'true' AND template_type = 'FinOps'"), # No direct link — should not hallucinate join | |
| ] | |
| 1. is_global_query — will fail to wipe memory: | |
| "Show me everything you're monitoring" | |
| "Give me the full picture of all databases" | |
| "What does the entire system look like?" | |
| "Across every target, what's the average anomaly score?" | |
| "Pull data from all of them" | |
| 2. is_anomaly_like bridge trigger — will fire incorrectly on non-Influx queries: | |
| "What is the worst credit cost across all providers?" — "worst" triggers bridge unnecessarily | |
| "Show me the top performing templates" — "performing" triggers bridge | |
| "Which targets have the worst configuration?" — "worst" triggers bridge, no Influx intent | |
| "Performance of my report scheduler" — "performance" triggers bridge | |
| "Give me the most severe credit overspend" — "severe" triggers bridge on Derby pricing query | |
| And will fail to trigger bridge when it should: | |
| "Dig up issues from targets not on Linux" — "issues" not in trigger list? Actually it is. But: | |
| "Find slow databases not in us-east-1" — "slow" not in list | |
| "Which targets are struggling?" — no trigger words at all | |
| 3. col_first_op — will corrupt mixed polarity queries: | |
| "Worst queries for targets on Linux or Windows but not Oracle or MySQL" — Linux/Windows are positive (=), Oracle/MySQL are negative (!=). col_first_op for hlc_database_type would be set by whichever comes first | |
| "Anomalies for MySQL targets not in us-east-1 or eu-central-1" — MySQL is positive, regions are negative, different columns so safe here — but: | |
| "Performance issues for targets on Linux but not Linux servers in eu-central-1" — same column, mixed intent | |
| 4. asked_for_cost — "average" too broad, fires on wrong queries: | |
| "Average anomaly score for cloud targets" — "average" matches, pulls cloud_database_pricing into an Influx query | |
| "What is the average credit score of my targets?" — "average" + "credit" both match, but this is a target query not a pricing query | |
| "Show me the average number of running targets" — "average" matches inappropriately | |
| 5. follow_up_pattern — will miss: | |
| "And what about those on Windows?" — "And what about" not matched | |
| "From that set, which ones are stopped?" — "from that set" not matched | |
| "Among them, show me the worst performers" — "among them" not matched | |
| "Now filter by Oracle only" — no follow-up words at all but clearly a refinement | |
| "Narrow it down to just cloud targets" — completely missed | |
| Will fire incorrectly: | |
| "What about the pricing table?" — "what about" triggers memory injection from previous query even though it's a fresh topic | |
| "How about showing me all collectors?" — "how about" triggers memory injection | |
| 1. Target <-> Pricing (target + cloud_database_pricing) | |
| Prompt: "Show me the target names and their cloud pricing credit costs for the AWS provider." | |
| What to watch for: It must generate JOIN cloud_database_pricing ON target.cloud_region = cloud_database_pricing.region. | |
| 2. Target <-> Collector (target + dbactdc_instance) | |
| Prompt: "List the target databases that are managed by remote collector agents." | |
| What to watch for: It must detect "remote" as dbactdc_instance.is_remote = 'true' and generate JOIN dbactdc_instance ON target.hlc_server_name = dbactdc_instance.name. | |
| 3. Reports <-> Templates (report + report_template) | |
| Prompt: "Which scheduled reports are currently using FinOps templates?" | |
| What to watch for: It must map "scheduled" to report.is_scheduled and "FinOps" to report_template.template_type, generating a clean join on report_id. | |
| 4. Templates <-> Charts (template + charts_template_element) | |
| Prompt: "Show me the dynamic dashboard templates and the metrics used in their chart elements." | |
| What to watch for: It should join template to charts_template_element on template_id and select template.name alongside charts_template_element.metric. | |
| 5. The 3-Table Deep Dive (report -> report_template -> report_template_element_alert) | |
| Prompt: "List the report titles that have element alerts with an alert value greater than 90." | |
| What to watch for: This is the ultimate stress test. It requires a 3-table chain. The model must traverse from report through report_template to reach report_template_element_alert to filter the alert_value. | |