SOA Dependency Analysis

Prompt:
## Prompt
I need you to generate a PySpark script that implements strong dependency relationship analysis for high-impact modules of use cases, based on the Payment Platform W SOA architecture (SOA Dependency Analysis).

**Business Background and Objective**: In the Payment Platform W SOA architecture, it is necessary to analyze the call relationships of high-quality network use cases (`qualitynetwork_case=1` or `importance=1`). Through upstream and downstream traversal of strong dependency (`dependent_intensity="strong"`) call chains, identify modules directly and indirectly affected by use cases, record impact chain evidence, and output a strong dependency module relationship table for high-impact use cases, for use in premium network evaluation and architecture governance.

**Input Tables**:
- `internal_platform_db.t_dm_callrelation_extended_info_hour_pyspark_209`
- `internal_platform_db.t_dm_usecase_and_asset_callrelation_hour_pyspark_209`
- `internal_platform_db.t_dwd_module_info_hour_pyspark_209`
- `internal_platform_db.t_app_callrelation_for_quality_analyse_hour_pyspark_209`

**Data Range and Filter Conditions**:
- Run date `ds` is fixed as `'p_2026051300'` (hourly partition)
- From `t_dm_usecase_and_asset_callrelation_hour`, filter use cases where `qualitynetwork_case=1` or `importance=1`
- Exclude modules not entering the premium network where `no_entry_premium_network_flag=1`
- From `t_dm_callrelation_extended_info_hour`, filter strong dependency call relationships where `is_involved_to_high_usecase=1` and `dependent_intensity="strong"`
- Filter out invalid call relationships `ossid:0` and `0:0`

**Processing Logic**:
1. From the use case and call relationship table, obtain directly associated modules (mapping of `enter_module_name` to `case_id`), mark `is_dir=1`, with evidence "direct association"
2. Build `case_id_relation_dict`: using `"callee_module_name***callee_interface_name"` as key, and the associated `case_id` set as value
3. Build `relation_dict` from the call relationship table: using `"callee_module_name***callee_interface_name"` as key, and the `"called_module_name***called_interface_name"` set as value
4. Use a DFS algorithm to traverse downstream call chains, add traversed modules to the results, mark `is_dir=0`, with evidence as the downstream analysis chain
5. For use case entries where `analyse_type="cgi"`, build a reverse call relationship dictionary `reverse_relation_dict`
6. Use a DFS algorithm to traverse upstream call chains, add traversed modules to the results, mark `is_dir=0`, with evidence as the upstream analysis chain
7. Merge the results of direct association, downstream analysis, and upstream analysis, and write to the target table

**Output Requirements**:
- Target table: `internal_platform_db.t_dwd_architecture_module_high_case_relation_strong_hour_pyspark_209`
- Write strategy: `INSERT OVERWRITE TABLE`
- Output columns: `databus_imp_date` (STRING, date), `case_id` (STRING, use case ID), `module_name` (STRING, module name), `is_dir` (BIGINT, 1=direct association, 0=indirect association), `evidence` (STRING, impact evidence chain)

#!/usr/bin/env python3
"""pyspark_018 ground truth: SOA dependency analysis"""
import json
from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName('dataclaw_eval_gt_pyspark_018') \
    .enableHiveSupport() \
    .config('spark.sql.warehouse.dir', '/tmp/hive_warehouse') \
    .getOrCreate()

ds = 'p_2026051300'
today = '2026051300'

# Output table DDL
spark.sql('''
DROP TABLE IF EXISTS internal_platform_db.t_dwd_architecture_module_high_case_relation_strong_hour_pyspark_209
''')
spark.sql('''
CREATE TABLE IF NOT EXISTS internal_platform_db.t_dwd_architecture_module_high_case_relation_strong_hour_pyspark_209 (
    databus_imp_date STRING,
    case_id STRING,
    module_name STRING,
    is_dir BIGINT,
    evidence STRING
)
STORED AS ORC
''')

# Create temp views from input tables
spark.sql(f'''
CREATE OR REPLACE TEMP VIEW t_dm_callrelation_extended_info_hour AS
SELECT * FROM internal_platform_db.t_dm_callrelation_extended_info_hour_pyspark_209
WHERE ds = '{ds}'
''')

spark.sql(f'''
CREATE OR REPLACE TEMP VIEW t_dm_usecase_and_asset_callrelation_hour AS
SELECT * FROM internal_platform_db.t_dm_usecase_and_asset_callrelation_hour_pyspark_209
WHERE ds = '{ds}'
''')

spark.sql(f'''
CREATE OR REPLACE TEMP VIEW t_dwd_module_info_hour AS
SELECT * FROM internal_platform_db.t_dwd_module_info_hour_pyspark_209
WHERE ds = '{ds}'
''')

spark.sql(f'''
CREATE OR REPLACE TEMP VIEW t_app_callrelation_for_quality_analyse_hour AS
SELECT * FROM internal_platform_db.t_app_callrelation_for_quality_analyse_hour_pyspark_209
WHERE ds = '{ds}'
''')

print("step 1")
print("获取直接相关的模块")

T_SCHEMA = ('databus_imp_date', 'case_id', 'module_name', 'is_dir', 'evidence')

# Get direct relations - modules directly associated with cases
dir_sql = """
select distinct
    relation.enter_module_name,
    relation.case_id
from t_dm_usecase_and_asset_callrelation_hour relation
left join (
    select module_name
    from t_dwd_module_info_hour
    where no_entry_premium_network_flag = 1
) modules_info on relation.enter_module_name = modules_info.module_name
where (qualitynetwork_case = 1 or importance = 1)
    and (enter_module_name is not null and enter_module_name <> '')
    and modules_info.module_name is null
"""

service_datas = spark.sql(dir_sql).toPandas()

module_evidence = dict()

class Evidence:
    def __init__(self, module_name):
        self.module_name = module_name

    def set_case(self, case_id):
        self.case_id = case_id
        return self

    def set_evidence(self, evidence):
        self.evidence = evidence
        return self

    def to_string(self, comment):
        return "[用例ID:{}--{}--证据链路:{}]".format(self.case_id, comment, self.evidence)

result_list = []
for data in service_datas.values:
    enter_module_name = data[0]
    case_id = data[1]
    is_dir = 1

    if enter_module_name not in module_evidence:
        module_evidence[enter_module_name] = Evidence(enter_module_name).set_case(case_id).set_evidence("直接关联")
    result_list.append((today, case_id, enter_module_name, is_dir, module_evidence[enter_module_name].to_string("直接关联")))

print("直接影响的个数")
origin_length = len(result_list)
print(origin_length)

print("step 2")
print("获取用例和调用关系入口模块-接口映射")

case_id_relation_dict_sql = """
select
    enter_module_name, case_id, callee_module_name, callee_interface_name
from t_dm_usecase_and_asset_callrelation_hour
where (qualitynetwork_case = 1 or importance = 1)
    and (enter_module_name is not null and enter_module_name <> '')
    and (callee_module_name is not null and callee_module_name <> '')
    and (callee_interface_name is not null and callee_interface_name <> '')
    and (called_module_name is not null and called_module_name <> '')
    and (called_interface_name is not null and called_interface_name <> '')
    and enter_module_name = callee_module_name
"""

case_id_relation_dict = dict()
service_datas = spark.sql(case_id_relation_dict_sql).toPandas()
for data in service_datas.values:
    enter_module_name = data[0]
    case_id = data[1]
    callee_module_name = data[2]
    callee_interface_name = data[3]

    relation = "{}***{}".format(callee_module_name, callee_interface_name)
    if relation in case_id_relation_dict:
        case_id_relation_dict[relation].add(case_id)
    else:
        case_id_relation_dict[relation] = set()
        case_id_relation_dict[relation].add(case_id)

print("step 3")
print("获取剔除了不进入精品网模块的强依赖调用关系")

relation_dict_sql = """
select
    relation.callee_module_name,
    relation.callee_interface_name,
    relation.called_module_name,
    relation.called_interface_name
from
    (select
        callee_module_name, callee_interface_name, called_module_name, called_interface_name
    from t_dm_callrelation_extended_info_hour
    where is_involved_to_high_usecase = 1
        and dependent_intensity = 'strong'
    ) relation
left join (
    select module_name
    from t_dwd_module_info_hour
    where no_entry_premium_network_flag = 1
) modules_callee on relation.callee_module_name = modules_callee.module_name
left join (
    select module_name
    from t_dwd_module_info_hour
    where no_entry_premium_network_flag = 1
) modules_called on relation.called_module_name = modules_called.module_name
where modules_callee.module_name is null
    and modules_called.module_name is null
"""

relation_dict = dict()
service_datas = spark.sql(relation_dict_sql).toPandas()
for data in service_datas.values:
    callee_module_name = data[0]
    callee_interface_name = data[1]
    called_module_name = data[2]
    called_interface_name = data[3]

    relation_callee = "{}***{}".format(callee_module_name, callee_interface_name)
    relation_called = "{}***{}".format(called_module_name, called_interface_name)
    if relation_callee in relation_dict:
        relation_dict[relation_callee].add(relation_called)
    else:
        relation_dict[relation_callee] = set()
        relation_dict[relation_callee].add(relation_called)

def dfs(relation_dict, called_info, record_list, chain_list, space, case_id_now):
    if called_info not in relation_dict:
        return
    called_infos = relation_dict[called_info]
    for called_info_item in called_infos:
        if called_info_item in record_list:
            continue
        record_list.add(called_info_item)

        chain_list.append(called_info_item)

        module_name_now = called_info_item.split("***")[0]
        if module_name_now != "" and module_name_now not in module_evidence:
            module_evidence[module_name_now] = Evidence(module_name_now).set_evidence(space.join(chain_list)).set_case(case_id_now)

        dfs(relation_dict, called_info_item, record_list, chain_list, space, case_id_now)

        chain_list.pop()

def dfs_fist(relation_dict, key, reverse, case_id_now):
    space = "<-" if reverse else "->"
    record_list = set()
    chain_list = list()

    if key not in relation_dict:
        return None

    chain_list.append(key)

    module_name_now = key.split("***")[0]
    if module_name_now != "" and module_name_now not in module_evidence:
        module_evidence[module_name_now] = Evidence(module_name_now).set_evidence(space.join(chain_list)).set_case(case_id_now)

    called_infos = relation_dict[key]
    for called_info in called_infos:
        if called_info in record_list:
            continue
        record_list.add(called_info)

        module_name_now = called_info.split("***")[0]
        chain_list.append(called_info)
        if module_name_now != "" and module_name_now not in module_evidence:
            module_evidence[module_name_now] = Evidence(module_name_now).set_evidence(space.join(chain_list)).set_case(case_id_now)

        dfs(relation_dict, called_info, record_list, chain_list, space, case_id_now)
        chain_list.pop()
    return record_list

# Downstream analysis
print("step 4")
print("开始下游分析")

is_dir = 0
module_evidence = dict()
for key in case_id_relation_dict:
    case_id_now = list(case_id_relation_dict[key])[0]
    modules_list = dfs_fist(relation_dict, key, 0, case_id_now)
    if modules_list is None:
        continue
    cases = case_id_relation_dict[key]
    for case_id in cases:
        for module in modules_list:
            module_name = module.split("***")[0]
            if module_name == "":
                continue
            result_list.append((today, case_id, module_name, is_dir, module_evidence[module_name].to_string("下游分析")))

print("间接影响的个数")
print(len(result_list) - origin_length)

# Upstream analysis (CGI)
print("step 5")
print("获取cgi用例和调用关系入口模块-接口映射")

case_id_relation_dict_sql = """
select
    enter_module_name, case_id, callee_module_name, callee_interface_name
from t_dm_usecase_and_asset_callrelation_hour
where (qualitynetwork_case = 1 or importance = 1)
    and (enter_module_name is not null and enter_module_name <> '')
    and (callee_module_name is not null and callee_module_name <> '')
    and (callee_interface_name is not null and callee_interface_name <> '')
    and (called_module_name is not null and called_module_name <> '')
    and (called_interface_name is not null and called_interface_name <> '')
    and enter_module_name = callee_module_name
    and analyse_type = 'cgi'
"""

case_id_relation_dict = dict()
service_datas = spark.sql(case_id_relation_dict_sql).toPandas()
for data in service_datas.values:
    enter_module_name = data[0]
    case_id = data[1]
    callee_module_name = data[2]
    callee_interface_name = data[3]

    relation = "{}***{}".format(callee_module_name, callee_interface_name)
    if relation in case_id_relation_dict:
        case_id_relation_dict[relation].add(case_id)
    else:
        case_id_relation_dict[relation] = set()
        case_id_relation_dict[relation].add(case_id)

# Build reverse relation dict
all_relation_dict_sql = """
select distinct
    callee_module_name,
    callee_interface_name,
    called_module_name,
    called_interface_name
from t_app_callrelation_for_quality_analyse_hour
where callee_module_name <> 'ossid:0'
    and called_module_name <> 'ossid:0'
    and callee_module_name <> '0:0'
    and called_module_name <> '0:0'
"""

reverse_relation_dict = dict()
service_datas = spark.sql(all_relation_dict_sql).toPandas()
for data in service_datas.values:
    callee_module_name = data[0]
    callee_interface_name = data[1]
    called_module_name = data[2]
    called_interface_name = data[3]

    relation_callee = "{}***{}".format(callee_module_name, callee_interface_name)
    relation_called = "{}***{}".format(called_module_name, called_interface_name)
    if relation_called in reverse_relation_dict:
        reverse_relation_dict[relation_called].add(relation_callee)
    else:
        reverse_relation_dict[relation_called] = set()
        reverse_relation_dict[relation_called].add(relation_callee)

module_evidence = dict()
is_dir = 0
reverse_result_list = list()
for key in case_id_relation_dict:
    case_id_now = list(case_id_relation_dict[key])[0]
    modules_list = dfs_fist(reverse_relation_dict, key, 1, case_id_now)
    if modules_list is None:
        continue
    cases = case_id_relation_dict[key]
    for case_id in cases:
        for module in modules_list:
            module_name = module.split("***")[0]
            if module_name == "":
                continue
            reverse_result_list.append((today, case_id, module_name, is_dir, module_evidence[module_name].to_string("上游分析")))

print("反向影响的个数")
print(len(reverse_result_list))
result_list = result_list + reverse_result_list

# Write output
t_df = spark.createDataFrame(result_list, T_SCHEMA)
t_df.write.mode('overwrite').format('orc').saveAsTable(
    'internal_platform_db.t_dwd_architecture_module_high_case_relation_strong_hour_pyspark_209'
)

print('Ground truth computation complete')
spark.stop()


Campaign User Label Computation

Prompt:
## Prompt
I need you to generate a PySpark script that implements campaign user targeting label computation based on brand campaign activities (Campaign User Label Computation).

**Business Background and Objective**: In the Payment Platform W brand management platform, the targeting labels for campaign configurations are stored in JSON format (containing `label_groups` and `exclude_label_groups`). These need to be parsed into a structured label detail table, and transaction frequency labels (`label_type=7`) need to be automatically backfilled, for downstream user filtering and campaign delivery.

**Input Tables**:
- `internal_platform_db.t_dwd_brand_mgmt_activity_all_hour_pyspark_206`
- `internal_platform_db.t_dwd_gift_mgmt_targeted_act_hour_pyspark_206`

**Data Range and Filter Conditions**:
- Run date `ds` is fixed as `'p_2026051300'` (hourly partition)
- From `t_dwd_brand_mgmt_activity_all_hour`, filter campaigns where `factid>10 AND factstatus IN (2,6,7)`
- Union with the `factid` and `factivitylabel` from `t_dwd_gift_mgmt_targeted_act_hour`

**Processing Logic**:
1. Obtain the campaign `factivitylabel` JSON data by unioning the brand management campaign table and the pay-gift table
2. Use `flatMap` to parse the `label_groups.target_labels` and `exclude_label_groups.target_labels` from each row's JSON into independent rows
3. Extract `label_type`, `label_value`, `effective_range_type`, `effective_range_value`, and `is_exclude` fields from `raw_json`
4. Filter out invalid labels (records where `label_type`, `label_value`, or `effective_range_type` is empty or 0), and deduplicate
5. For all brands (`fbrandid`) with valid campaigns, backfill the four values (1, 2, 3, 4) for `label_type=7` (transaction frequency), with `effective_range_type=2` and `is_exclude=0`
6. Use a left_anti join to exclude already existing records, avoiding duplication

**Output Requirements**:
- Target table: `internal_platform_db.brand_mgmt_act_label_pyspark_206`
- Write strategy: `INSERT OVERWRITE TABLE`
- Output columns: `ds` (STRING, date), `label_type` (STRING, label type), `label_value` (STRING, label value), `effective_range_type` (STRING, effective range type), `effective_range_value` (STRING, effective range value), `raw_json` (STRING, original JSON), `is_exclude` (INT, 0=include, 1=exclude)

#!/usr/bin/env python3
"""pyspark_019 ground truth: campaign user label computation"""
import json
from pyspark.sql import SparkSession
from pyspark.sql.functions import lit, get_json_object, when, to_json, struct

spark = SparkSession.builder \
    .appName('dataclaw_eval_gt_pyspark_019') \
    .enableHiveSupport() \
    .config('spark.sql.warehouse.dir', '/tmp/hive_warehouse') \
    .getOrCreate()

hour = '2026051300'
HOUR_PARTITION = 'p_' + str(hour)

# Output table DDL
spark.sql('''
DROP TABLE IF EXISTS internal_platform_db.brand_mgmt_act_label_pyspark_206
''')
spark.sql('''
CREATE TABLE IF NOT EXISTS internal_platform_db.brand_mgmt_act_label_pyspark_206 (
    ds STRING,
    label_type STRING,
    label_value STRING,
    effective_range_type STRING,
    effective_range_value STRING,
    raw_json STRING,
    is_exclude INT
)
STORED AS ORC
''')

# Step 1: Read activity data from brand manager + paygift, filter and union
activity_label = (
    spark.sql(f'''
        SELECT factid, factivitylabel
        FROM internal_platform_db.t_dwd_brand_mgmt_activity_all_hour_pyspark_206
        WHERE ds = '{HOUR_PARTITION}'
            AND factid > 10
            AND factstatus IN (2, 6, 7)
    ''')
).union(
    spark.sql(f'''
        SELECT factid, factivitylabel
        FROM internal_platform_db.t_dwd_gift_mgmt_targeted_act_hour_pyspark_206
        WHERE ds = '{HOUR_PARTITION}'
    ''')
)

activity_label.show(truncate=False)

def parse_labels(row):
    result = []
    if row["factivitylabel"] is None:
        return result
    labels = json.loads(row["factivitylabel"])
    if labels is None:
        return result
    if "label_groups" in labels.keys():
        for group in labels["label_groups"]:
            if "target_labels" in group.keys():
                for l in group["target_labels"]:
                    json_str = json.dumps(l)
                    result.append((json_str,))
    if "exclude_label_groups" in labels.keys():
        for group in labels["exclude_label_groups"]:
            if "target_labels" in group.keys():
                for l in group["target_labels"]:
                    json_str = json.dumps(l)
                    result.append((json_str,))
    return result

# Step 2: flatMap to parse JSON labels into individual rows
single_labels = activity_label.rdd.flatMap(lambda row: parse_labels(row))
print(1)
if single_labels.isEmpty():
    single_labels_df = spark.createDataFrame([], "raw_json: string")
else:
    single_labels_df = spark.createDataFrame(single_labels, ["raw_json"])

single_labels_df = (
    single_labels_df.withColumn("ds", lit(hour))
    .withColumn("label_type", get_json_object(single_labels_df.raw_json, "$.label_type"))
    .withColumn("label_value", get_json_object(single_labels_df.raw_json, "$.label_value"))
    .withColumn("effective_range_type", get_json_object(single_labels_df.raw_json, "$.effective_range_type"))
    .withColumn("effective_range_value", get_json_object(single_labels_df.raw_json, "$.effective_range_value"))
    .withColumn("is_exclude", when(get_json_object(single_labels_df.raw_json, "$.is_exclude") == "true", 1).otherwise(0))
)

single_labels_df.show(truncate=False)

# Step 3: Dedup and filter valid labels
filtered_labels = (
    single_labels_df.select(
        "ds", "label_type", "label_value",
        "effective_range_type", "effective_range_value", "raw_json", "is_exclude",
    )
    .filter("nvl(label_type, 0) != 0 AND nvl(label_value, '0') != '0' AND nvl(effective_range_type, 0) != 0")
    .distinct()
)

# Step 4: Supplement label_type=7 (transaction frequency labels)
all_act_brands = (
    spark.sql(f'''
        SELECT DISTINCT fbrandid
        FROM internal_platform_db.t_dwd_brand_mgmt_activity_all_hour_pyspark_206
        WHERE ds = '{HOUR_PARTITION}'
            AND factid > 10
            AND factstatus IN (2, 6, 7)
    ''')
    .withColumnRenamed("fbrandid", "effective_range_value")
)

freq_label_values = spark.createDataFrame(
    [("1",), ("2",), ("3",), ("4",)], ["label_value"]
)

extra_freq_labels = (
    all_act_brands.crossJoin(freq_label_values)
    .withColumn("ds", lit(hour))
    .withColumn("label_type", lit("7"))
    .withColumn("effective_range_type", lit("2"))
    .withColumn("is_exclude", lit(0))
)

extra_freq_labels = extra_freq_labels.withColumn(
    "raw_json",
    to_json(
        struct(
            extra_freq_labels.label_type,
            extra_freq_labels.label_value,
            extra_freq_labels.effective_range_type,
            extra_freq_labels.effective_range_value,
        )
    ),
).select(
    "ds", "label_type", "label_value",
    "effective_range_type", "effective_range_value", "raw_json", "is_exclude",
)

# Use left_anti join to keep only records not already in filtered_labels
new_freq_labels = extra_freq_labels.join(
    filtered_labels,
    on=["label_type", "label_value", "effective_range_type", "effective_range_value", "is_exclude"],
    how="left_anti",
)

result_labels = filtered_labels.unionByName(new_freq_labels)

# Write output
result_labels.write.mode('overwrite').format('orc').saveAsTable(
    'internal_platform_db.brand_mgmt_act_label_pyspark_206'
)

print('Ground truth computation complete')
spark.stop()


小程序H5监控

Prompt:
## Prompt
我需要你生成一段 PySpark 代码，实现小程序打开第三方 H5 页面的大盘监控指标计算（Mini-Program H5 Monitoring）。

**业务背景与目标**：小程序开放平台需要监控小程序内打开第三方 H5 页面的整体情况，包括 PV、UV、域名数、链接数等核心指标，按整体大盘和 Top100 小程序两个维度分别统计，同时按域名归属关系（同主体/关联主体/第三方）进行细分，产出结构化监控指标表。

**输入表**：
- `internal_platform_db.ods_daily_miniapp_h5_expand_raw_pyspark_212`
- `internal_platform_db.ods_daily_miniapp_h5_top100_pv_raw_pyspark_212`

**数据范围与过滤条件**：
- 运行日期 ds 固定为 '20260513'（日分区）
- ods_daily_miniapp_h5_expand_raw 提供全量 H5 访问明细
- ods_daily_miniapp_h5_top100_pv_raw 提供 Top100 PV 小程序的 H5 访问明细
- url 列重命名为 url_wop（无参数 URL）

**处理逻辑**：
1. 从 expand_raw 表计算整体大盘指标：total PV=总行数, total UV=去重 useruin, total domain=去重 domain, total url_wop=去重 url_wop，stat_type='total'
2. 从 top100_raw 表按 appid 分组计算：PV=count, UV=去重 useruin, domain=去重 domain, url_wop=去重 url_wop，stat_type='total_top'
3. 从 expand_raw 表按 related（域名归属关系）分组计算：PV=count, UV=去重 useruin, domain=去重 domain, url_wop=去重 url_wop，stat_type='contactor_{related}'
4. 从 top100_raw 表按 appid+related 分组计算同样指标，stat_type='contactor_{related}_top'
5. 使用 pandas merge 将各维度指标合并为统一格式
6. 添加 ds 列（BIGINT 类型），生成最终结果

**输出要求**：
- 目标表：`internal_platform_db.ods_daily_miniapp_webview_total_pyspark_212`
- 写入策略：INSERT OVERWRITE TABLE
- 输出列：ds（BIGINT，日期）, appid（STRING，小程序appid，大盘级为空字符串）, pv（BIGINT，页面浏览量）, uv（BIGINT，独立访客数）, domain（BIGINT，域名数）, url_wop（BIGINT，去参链接数）, stat_type（STRING，统计类型：total/total_top/contactor_{related}/contactor_{related}_top）

#!/usr/bin/env python3
"""pyspark_020 ground truth: mini-program H5 monitoring metrics"""
import pandas as pd
from pyspark.sql import SparkSession
import pyspark.sql.functions as F

spark = SparkSession.builder \
    .appName('dataclaw_eval_gt_pyspark_020') \
    .enableHiveSupport() \
    .config('spark.sql.warehouse.dir', '/tmp/hive_warehouse') \
    .getOrCreate()

ds = '20260513'

# Output table DDL
spark.sql('''
DROP TABLE IF EXISTS internal_platform_db.ods_daily_miniapp_webview_total_pyspark_212
''')
spark.sql('''
CREATE TABLE IF NOT EXISTS internal_platform_db.ods_daily_miniapp_webview_total_pyspark_212 (
    ds BIGINT COMMENT 'date',
    appid STRING COMMENT 'appid',
    pv BIGINT COMMENT 'page views',
    uv BIGINT COMMENT 'unique visitors',
    domain BIGINT COMMENT 'domain count',
    url_wop BIGINT COMMENT 'url without param count',
    stat_type STRING COMMENT 'statistic type'
)
STORED AS ORC
''')

cols = ['appid', 'pv', 'uv', 'domain', 'url_wop', 'stat_type']

# Read input data
web_stats_raw = spark.sql(f'''
    SELECT useruin, appid, url_, domain, url, contactor_domain, related
    FROM internal_platform_db.ods_daily_miniapp_h5_expand_raw_pyspark_212
    WHERE ds = '{ds}'
''')
web_stats = web_stats_raw.withColumnRenamed('url', 'url_wop')
web_stats.persist()
web_stats_cnt = web_stats.count()
print(f"Fetch total web_stats info: {web_stats_cnt}")

web_stats_topuser_id_raw = spark.sql(f'''
    SELECT useruin, appid, url_, domain, url, contactor_domain, related
    FROM internal_platform_db.ods_daily_miniapp_h5_top100_pv_raw_pyspark_212
    WHERE ds = '{ds}'
''')
web_stats_topuin = web_stats_topuser_id_raw.withColumnRenamed('url', 'url_wop')
web_stats_topuin.persist()
web_stats_topuser_id_cnt = web_stats_topuin.count()
print(f"Fetch total web_stats_topuin info: {web_stats_topuser_id_cnt}")

# 1. Overall metrics: total PV/UV/domain/URL counts
cate = 'total'
total_pv = web_stats_cnt
total_uv = web_stats[['useruin']].drop_duplicates().count()
total_domain_cnt = web_stats[['domain']].drop_duplicates().count()
total_url_cnt = web_stats[['url_wop']].drop_duplicates().count()
df_total = pd.DataFrame(data=[('', total_pv, total_uv, total_domain_cnt, total_url_cnt, cate)], columns=cols)

# Per-app metrics for top100
cate = 'total_top'
total_top_pv = web_stats_topuin.groupby('appid').count().withColumnRenamed('count', 'pv').toPandas()
total_top_uv = web_stats_topuin[['appid', 'useruin']].drop_duplicates().groupby('appid').count().withColumnRenamed('count', 'uv').toPandas()
total_top_domain_cnt = web_stats_topuin[['appid', 'domain']].drop_duplicates().groupby('appid').count().withColumnRenamed('count', 'domain').toPandas()
total_top_url_cnt = web_stats_topuin[['appid', 'url_wop']].drop_duplicates().groupby('appid').count().withColumnRenamed('count', 'url_wop').toPandas()
total_top_metrics = total_top_pv.merge(total_top_uv, how='left', on='appid') \
    .merge(total_top_domain_cnt, how='left', on='appid') \
    .merge(total_top_url_cnt, how='left', on='appid')
total_top_metrics['stat_type'] = cate

pdf_total = pd.concat([df_total[cols], total_top_metrics[cols]], ignore_index=True, axis=0)

# 2. Contactor distribution metrics
cate = 'contactor'
contactor_pv_dist = web_stats.groupby('related').count().withColumnRenamed('count', 'pv').toPandas()
contactor_pv_dist['stat_type'] = cate + '_' + contactor_pv_dist.related
contactor_uv_dist = web_stats[['related', 'useruin']].drop_duplicates().groupby('related').count().withColumnRenamed('count', 'uv').toPandas()
contactor_uv_dist['stat_type'] = cate + '_' + contactor_uv_dist.related
contactor_domain_dist = web_stats[['related', 'domain']].drop_duplicates().groupby('related').count().withColumnRenamed('count', 'domain').toPandas()
contactor_domain_dist['stat_type'] = cate + '_' + contactor_domain_dist.related
contactor_url_dist = web_stats[['related', 'url_wop']].drop_duplicates().groupby('related').count().withColumnRenamed('count', 'url_wop').toPandas()
contactor_url_dist['stat_type'] = cate + '_' + contactor_url_dist.related
df_contactor = contactor_pv_dist[['stat_type', 'pv']].merge(contactor_uv_dist[['stat_type', 'uv']]) \
    .merge(contactor_domain_dist[['stat_type', 'domain']]) \
    .merge(contactor_url_dist[['stat_type', 'url_wop']])
df_contactor['appid'] = ''

# Per-app contactor metrics for top100
contactor_top_pv = web_stats_topuin.groupby(['appid', 'related']).count().withColumnRenamed('count', 'pv').toPandas()
contactor_top_uv = web_stats_topuin[['appid', 'related', 'useruin']].drop_duplicates().groupby(['appid', 'related']).count().withColumnRenamed('count', 'uv').toPandas()
contactor_top_domain = web_stats_topuin[['appid', 'related', 'domain']].drop_duplicates().groupby(['appid', 'related']).count().withColumnRenamed('count', 'domain').toPandas()
contactor_top_url = web_stats_topuin[['appid', 'related', 'url_wop']].drop_duplicates().groupby(['appid', 'related']).count().withColumnRenamed('count', 'url_wop').toPandas()
contactor_top_metrics = contactor_top_pv.merge(contactor_top_uv, how='left', on=['appid', 'related']) \
    .merge(contactor_top_domain, how='left', on=['appid', 'related']) \
    .merge(contactor_top_url, how='left', on=['appid', 'related'])
contactor_top_metrics['stat_type'] = cate + '_' + contactor_top_metrics.related + '_top'

pdf_contactor = pd.concat([df_contactor[cols], contactor_top_metrics[cols]], ignore_index=True, axis=0)

# Combine all metrics
pdf_metrics = pd.concat([pdf_total, pdf_contactor], ignore_index=True, axis=0)
sdf_metrics = spark.createDataFrame(pdf_metrics)
res = sdf_metrics.withColumn('ds', F.lit(int(ds))).select('ds', 'appid', 'pv', 'uv', 'domain', 'url_wop', 'stat_type').repartition(1)
res.persist()
res_cnt = res.count()
print(f"Fetch statistic metrics: {res_cnt}")

# Write output
res.write.mode('overwrite').format('orc').saveAsTable(
    'internal_platform_db.ods_daily_miniapp_webview_total_pyspark_212'
)

print('Ground truth computation complete')
spark.stop()


Order Stream and Payment Stream Interval Join

Prompt:
## Prompt
I need you to write a Flink SQL that uses the built-in datagen connector to simulate an order stream and a payment stream, performing an Interval Join on `order_id`, and writing matched records directly to the console.

**Business Background and Objective**: Simulate an order stream and a payment stream using two datagen built-in tables, each generating 100 million records. The `order_id` and `user_id` in the order stream range from 1 to 1,000,000, and the `order_id` in the payment stream also ranges from 1 to 1,000,000. Both tables use `LOCALTIMESTAMP` as the event time and set a 5-second watermark delay.

Perform an Interval Join on the two tables using `order_id`, matching only records where the payment time falls within 5 minutes after the order time. Upon a successful match, output the original fields directly to the console table, including `order_id`, `user_id`, `pay_amount`, the order time, and the payment time.

**Source Table Definitions**:
- `orders_source` (order stream, datagen connector):
  - `order_id INT`: Order ID, randomly generated in the range 1–1,000,000
  - `user_id INT`: User ID, randomly generated in the range 1–1,000,000
  - `order_time`: Uses `LOCALTIMESTAMP` to generate event time, with a WATERMARK delay of 5 seconds
  - Generation rate: `rows-per-second = 1000`

- `payments_source` (payment stream, datagen connector):
  - `order_id INT`: Order ID, randomly generated in the range 1–1,000,000
  - `pay_amount DOUBLE`: Payment amount, randomly generated in the range 1–100,000
  - `pay_time`: Uses `LOCALTIMESTAMP` to generate event time, with a WATERMARK delay of 5 seconds
  - Generation rate: `rows-per-second = 1000`

**Output Table Definition**:
- `console_output` (print connector):
  - `order_id INT`
  - `user_id INT`
  - `pay_amount DOUBLE`
  - `order_time TIMESTAMP(3)`
  - `pay_time TIMESTAMP(3)`

**Join Logic**:
- Join condition: `orders_source.order_id = payments_source.order_id`
- Time window: `payments_source.pay_time BETWEEN orders_source.order_time AND orders_source.order_time + INTERVAL '5' MINUTE`
- Note: When writing to the sink, you must handle the dual rowtime attribute conflict by applying `CAST(pay_time AS TIMESTAMP(3))` to downgrade `pay_time` to a plain timestamp.

**Output Requirements**:
- Use `INSERT INTO console_output` to output the matched results.
- Output field order: `order_id`, `user_id`, `pay_amount`, `order_time`, `pay_time`

-- online_001: 订单流 + 支付流 Interval Join
-- 每个源生成 1 亿条数据 @ 1000 rows/sec

-- 1. 订单流 datagen 源表
CREATE TABLE orders_source (
    order_id INT,
    user_id INT,
    order_time AS LOCALTIMESTAMP,
    WATERMARK FOR order_time AS order_time - INTERVAL '5' SECOND
) WITH (
    'connector' = 'datagen',
    'rows-per-second' = '1000',
    'fields.order_id.kind' = 'random',
    'fields.order_id.min' = '1',
    'fields.order_id.max' = '1000000',
    'fields.user_id.kind' = 'random',
    'fields.user_id.min' = '1',
    'fields.user_id.max' = '1000000'
);

-- 2. 支付流 datagen 源表
CREATE TABLE payments_source (
    order_id INT,
    pay_amount DOUBLE,
    pay_time AS LOCALTIMESTAMP,
    WATERMARK FOR pay_time AS pay_time - INTERVAL '5' SECOND
) WITH (
    'connector' = 'datagen',
    'rows-per-second' = '1000',
    'fields.order_id.kind' = 'random',
    'fields.order_id.min' = '1',
    'fields.order_id.max' = '1000000',
    'fields.pay_amount.kind' = 'random',
    'fields.pay_amount.min' = '1',
    'fields.pay_amount.max' = '100000'
);

-- 3. Console 输出表
CREATE TABLE console_output (
    order_id INT,
    user_id INT,
    pay_amount DOUBLE,
    order_time TIMESTAMP(3),
    pay_time TIMESTAMP(3)
) WITH (
    'connector' = 'print'
);

-- 4. Interval Join: 匹配支付时间在订单时间后 5 分钟内的记录
INSERT INTO console_output
SELECT
    o.order_id,
    o.user_id,
    p.pay_amount,
    o.order_time,
    CAST(p.pay_time AS TIMESTAMP(3)) AS pay_time
FROM orders_source o
JOIN payments_source p
    ON o.order_id = p.order_id
WHERE
    p.pay_time BETWEEN o.order_time AND o.order_time + INTERVAL '5' MINUTE;


Order-Payment Interval Join + Windowed Aggregation

Prompt:
## Prompt
I need you to write a Flink SQL that uses two datagen built-in tables to simulate an order stream and a payment stream, performs an Interval Join on `order_id`, and then aggregates the order count and total payment amount over a 10-minute tumbling window, outputting the results to the console.

**Business Background and Objective**: Simulate an order stream and a payment stream using two datagen built-in tables, each generating 100 million records. The `order_id` and `user_id` in the order stream range from 1 to 1,000,000, and the `order_id` in the payment stream also ranges from 1 to 1,000,000. Both tables use `LOCALTIMESTAMP` as the event time and set a 5-second watermark delay.

Perform an Interval Join on the two tables using `order_id`, matching only records where the payment time falls within 10 minutes before or after the order time. Then, aggregate the successfully matched records over a 10-minute tumbling window to compute the order count and total payment amount per window, and finally output the results to the console table.

**Source Table Definitions**:
- `orders_source` (order stream, datagen connector):
  - `order_id INT`: Order ID, randomly generated in the range 1–1,000,000
  - `user_id INT`: User ID, randomly generated in the range 1–1,000,000
  - `event_time`: Uses `LOCALTIMESTAMP` to generate event time, with a WATERMARK delay of 5 seconds
  - Generation rate: `rows-per-second = 1000`

- `payments_source` (payment stream, datagen connector):
  - `order_id INT`: Order ID, randomly generated in the range 1–1,000,000
  - `pay_amount DOUBLE`: Payment amount, randomly generated in the range 1–100,000
  - `event_time`: Uses `LOCALTIMESTAMP` to generate event time, with a WATERMARK delay of 5 seconds
  - Generation rate: `rows-per-second = 1000`

**Output Table Definition**:
- `console_output` (print connector):
  - `window_start VARCHAR`: Window start time
  - `window_end VARCHAR`: Window end time
  - `order_count BIGINT`: Order count within the window
  - `total_pay_amount DOUBLE`: Total payment amount within the window

**Join + Aggregation Logic**:
- Interval Join condition: `orders_source.order_id = payments_source.order_id`
- Time window: `payments_source.event_time BETWEEN orders_source.event_time - INTERVAL '10' MINUTE AND orders_source.event_time + INTERVAL '10' MINUTE`
- Tumbling window: `TUMBLE` 10 minutes
- Aggregation metrics: `COUNT(*) AS order_count`, `SUM(pay_amount) AS total_pay_amount`
- Convert window times to `VARCHAR` for output

**Output Requirements**:
- Use `INSERT INTO console_output` to output the results.
- Output field order: `window_start`, `window_end`, `order_count`, `total_pay_amount`

-- online_002: 订单支付 Interval Join + Tumble 窗口聚合
-- 每个源生成 1 亿条数据 @ 1000 rows/sec

-- 1. 订单流 datagen 源表
CREATE TABLE orders_source (
    order_id INT,
    user_id INT,
    event_time AS LOCALTIMESTAMP,
    WATERMARK FOR event_time AS event_time - INTERVAL '5' SECOND
) WITH (
    'connector' = 'datagen',
    'rows-per-second' = '1000',
    'fields.order_id.kind' = 'random',
    'fields.order_id.min' = '1',
    'fields.order_id.max' = '1000000',
    'fields.user_id.kind' = 'random',
    'fields.user_id.min' = '1',
    'fields.user_id.max' = '1000000'
);

-- 2. 支付流 datagen 源表
CREATE TABLE payments_source (
    order_id INT,
    pay_amount DOUBLE,
    event_time AS LOCALTIMESTAMP,
    WATERMARK FOR event_time AS event_time - INTERVAL '5' SECOND
) WITH (
    'connector' = 'datagen',
    'rows-per-second' = '1000',
    'fields.order_id.kind' = 'random',
    'fields.order_id.min' = '1',
    'fields.order_id.max' = '1000000',
    'fields.pay_amount.kind' = 'random',
    'fields.pay_amount.min' = '1',
    'fields.pay_amount.max' = '100000'
);

-- 3. Console 输出表
CREATE TABLE console_output (
    window_start VARCHAR,
    window_end VARCHAR,
    order_count BIGINT,
    total_pay_amount DOUBLE
) WITH (
    'connector' = 'print'
);

-- 4. Interval Join + Tumble 10min 窗口聚合
INSERT INTO console_output
SELECT
    CAST(TUMBLE_START(o.event_time, INTERVAL '10' MINUTE) AS VARCHAR) AS window_start,
    CAST(TUMBLE_END(o.event_time, INTERVAL '10' MINUTE) AS VARCHAR) AS window_end,
    COUNT(*) AS order_count,
    SUM(p.pay_amount) AS total_pay_amount
FROM orders_source o
JOIN payments_source p
    ON o.order_id = p.order_id
    AND p.event_time BETWEEN o.event_time - INTERVAL '10' MINUTE
                        AND o.event_time + INTERVAL '10' MINUTE
GROUP BY TUMBLE(o.event_time, INTERVAL '10' MINUTE);


Word Frequency Tumbling Window Statistics

Prompt:
## Prompt
I need you to write a Flink SQL that uses a datagen built-in table to simulate a word frequency data stream, groups by `word` and opens a 1-minute tumbling window to count the occurrences of each word, and outputs the results to the console.

**Business Background and Objective**: Simulate a word frequency data stream using a datagen built-in table, generating 1 billion records in the `word` field. Use `LOCALTIMESTAMP` as the event time and set a 5-second watermark delay. Group by `word` and count the occurrences of each word over a 1-minute tumbling window, then output `word` and `cnt` to the console table.

**Source Table Definition**:
- `word_source` (word frequency stream, datagen connector):
  - `word VARCHAR`: Word, randomly generated (value range 1–1000 to simulate distinct words)
  - `event_time`: Uses `LOCALTIMESTAMP` to generate event time, with a WATERMARK delay of 5 seconds
  - Generation rate: `rows-per-second = 1000`

**Output Table Definition**:
- `console_output` (print connector):
  - `word VARCHAR`: Word
  - `cnt BIGINT`: Number of occurrences of the word within the current window

**Aggregation Logic**:
- Group by `word`, using a 1-minute tumbling window
- Aggregation metric: `COUNT(*) AS cnt`
- The window is defined by the `TUMBLE` function

**Output Requirements**:
- Use `INSERT INTO console_output` to output the results.
- Output field order: `word`, `cnt`

-- online_003: 词频滚动窗口统计
-- 单源生成 10 亿条数据 @ 1000 rows/sec

-- 1. 词频流 datagen 源表
CREATE TABLE word_source (
    word VARCHAR,
    event_time AS LOCALTIMESTAMP,
    WATERMARK FOR event_time AS event_time - INTERVAL '5' SECOND
) WITH (
    'connector' = 'datagen',
    'rows-per-second' = '1000',
    'fields.word.kind' = 'random',
    'fields.word.length' = '5'
);

-- 2. Console 输出表
CREATE TABLE console_output (
    word VARCHAR,
    cnt BIGINT
) WITH (
    'connector' = 'print'
);

-- 3. Tumble 1min 窗口词频统计
INSERT INTO console_output
SELECT
    word,
    COUNT(*) AS cnt
FROM word_source
GROUP BY TUMBLE(event_time, INTERVAL '1' MINUTE), word;


Dual Window Aggregation (Tumble + Hop)

Prompt:
## Prompt
I need you to write a Flink SQL that uses a datagen built-in table to generate order events, and computes the sales volume and order count per product using both a tumbling window and a sliding window, outputting the results to two separate console tables.

**Business Background and Objective**: Generate order events (`order_id BIGINT`, `product_id INT`, `quantity INT`) using a datagen built-in table. Use `LOCALTIMESTAMP` as the event time and set a 5-second watermark delay.

Compute statistics using two window types:
1. **1-minute tumbling window**: Compute the sales volume `SUM(quantity) AS sales` and order count `COUNT(*) AS order_cnt` per product.
2. **Sliding window** (slides every 30 seconds, covering the past 1 minute): Compute the sales volume `SUM(quantity) AS sales` and order count `COUNT(*) AS order_cnt` per product.

Print the results of both window types to two separate console tables, with each record carrying a window type identifier `window_type`.

**Source Table Definition**:
- `orders_source` (order event stream, datagen connector):
  - `order_id BIGINT`: Order ID, randomly generated in the range 1–1,000,000
  - `product_id INT`: Product ID, randomly generated in the range 1–10,000
  - `quantity INT`: Quantity, randomly generated in the range 1–10
  - `event_time`: Uses `LOCALTIMESTAMP` to generate event time, with a WATERMARK delay of 5 seconds
  - Generation rate: `rows-per-second = 50`

**Output Table Definitions**:
- `console_tumble` (print connector, tumbling window results):
  - `window_type STRING`: Window type identifier, fixed as `'TUMBLE'`
  - `window_start TIMESTAMP(3)`: Window start time
  - `window_end TIMESTAMP(3)`: Window end time
  - `product_id INT`: Product ID
  - `sales INT`: Sales volume of the product within the window
  - `order_cnt BIGINT`: Order count of the product within the window

- `console_hop` (print connector, sliding window results):
  - `window_type STRING`: Window type identifier, fixed as `'HOP'`
  - `window_start TIMESTAMP(3)`: Window start time
  - `window_end TIMESTAMP(3)`: Window end time
  - `product_id INT`: Product ID
  - `sales INT`: Sales volume of the product within the window
  - `order_cnt BIGINT`: Order count of the product within the window

**Aggregation Logic**:
- Tumbling window: `GROUP BY TUMBLE(event_time, INTERVAL '1' MINUTE), product_id`
- Sliding window: `GROUP BY HOP(event_time, INTERVAL '30' SECOND, INTERVAL '1' MINUTE), product_id`
- Aggregation metrics: `SUM(quantity) AS sales`, `COUNT(*) AS order_cnt`
- Both window times and the window type must be explicitly output

**Output Requirements**:
- Use two `INSERT INTO` statements to output to `console_tumble` and `console_hop` respectively.
- Output field order: `window_type`, `window_start`, `window_end`, `product_id`, `sales`, `order_cnt`

-- online_004: 双窗口聚合 (Tumble + Hop)
-- 单源生成 @ 50 rows/sec

-- 1. 订单事件 datagen 源表
CREATE TABLE orders_source (
    order_id BIGINT,
    product_id INT,
    quantity INT,
    event_time AS LOCALTIMESTAMP,
    WATERMARK FOR event_time AS event_time - INTERVAL '5' SECOND
) WITH (
    'connector' = 'datagen',
    'rows-per-second' = '50',
    'fields.order_id.kind' = 'random',
    'fields.order_id.min' = '1',
    'fields.order_id.max' = '1000000',
    'fields.product_id.kind' = 'random',
    'fields.product_id.min' = '1',
    'fields.product_id.max' = '10000',
    'fields.quantity.kind' = 'random',
    'fields.quantity.min' = '1',
    'fields.quantity.max' = '10'
);

-- 2. Console 输出表 (Tumble 滚动窗口)
CREATE TABLE console_tumble (
    window_type STRING,
    window_start TIMESTAMP(3),
    window_end TIMESTAMP(3),
    product_id INT,
    sales INT,
    order_cnt BIGINT
) WITH (
    'connector' = 'print'
);

-- 3. Console 输出表 (Hop 滑动窗口)
CREATE TABLE console_hop (
    window_type STRING,
    window_start TIMESTAMP(3),
    window_end TIMESTAMP(3),
    product_id INT,
    sales INT,
    order_cnt BIGINT
) WITH (
    'connector' = 'print'
);

-- 4a. Tumble 1min 滚动窗口：SUM 销量 + COUNT 订单数
INSERT INTO console_tumble
SELECT
    'TUMBLE' AS window_type,
    TUMBLE_START(event_time, INTERVAL '1' MINUTE) AS window_start,
    TUMBLE_END(event_time, INTERVAL '1' MINUTE) AS window_end,
    product_id,
    CAST(SUM(quantity) AS INT) AS sales,
    COUNT(*) AS order_cnt
FROM orders_source
GROUP BY TUMBLE(event_time, INTERVAL '1' MINUTE), product_id;

-- 4b. Hop 滑动窗口 (30s slide, 1min size)：SUM 销量 + COUNT 订单数
INSERT INTO console_hop
SELECT
    'HOP' AS window_type,
    HOP_START(event_time, INTERVAL '30' SECOND, INTERVAL '1' MINUTE) AS window_start,
    HOP_END(event_time, INTERVAL '30' SECOND, INTERVAL '1' MINUTE) AS window_end,
    product_id,
    CAST(SUM(quantity) AS INT) AS sales,
    COUNT(*) AS order_cnt
FROM orders_source
GROUP BY HOP(event_time, INTERVAL '30' SECOND, INTERVAL '1' MINUTE), product_id;


滚动窗口 TopN 商品排行

Prompt:
## Prompt
我需要你写一段 Flink SQL，用 datagen 内置表模拟电商订单，按 10 秒滚动窗口做 Top-N 商品排行，输出销售额最高的前 3 个商品到 console。

**业务背景与目标**：用 datagen 内置表生成电商订单（字段 order_id / product_id / amount），每秒生成约 10 条。以 LOCALTIMESTAMP 作为事件时间，设置 5 秒的 Watermark 延迟。

按 10 秒滚动窗口做 Top-N 排行：每个窗口内统计每个商品的销售额 SUM(amount) 和订单数 COUNT(*)，按销售额降序取前三名，打印到 console。

**源表定义**：
- `orders_source`（订单流，datagen 连接器）：
  - order_id BIGINT：订单 ID，随机取值 1~1000000
  - product_id INT：商品 ID，随机取值 1~1000
  - amount DOUBLE：订单金额，随机取值 1~10000
  - event_time：使用 LOCALTIMESTAMP 生成事件时间，设置 WATERMARK 延迟 5 秒
  - 生成速率：rows-per-second = 10

**输出表定义**：
- `console_output`（print 连接器）：
  - window_start TIMESTAMP(3)：窗口起始时间
  - product_id INT：商品 ID
  - sales DOUBLE：该窗口内销售额
  - order_cnt BIGINT：该窗口内订单数
  - rn BIGINT：排名序号

**TopN 逻辑**：
- 使用 10 秒滚动窗口聚合：SUM(amount) AS sales, COUNT(*) AS order_cnt
- 在窗口内使用 ROW_NUMBER() 按销售额降序排名
- 只输出排名前 3 的记录（rn <= 3）

**输出要求**：
- 使用 INSERT INTO console_output 将 Top 3 结果输出
- 输出字段顺序：window_start, product_id, sales, order_cnt, rn

-- online_005: 滚动窗口 TopN 商品排行
-- 单源生成 @ 10 rows/sec, Tumble 10s + ROW_NUMBER Top3

-- 1. 订单流 datagen 源表
CREATE TABLE orders_source (
    order_id BIGINT,
    product_id INT,
    amount DOUBLE,
    event_time AS LOCALTIMESTAMP,
    WATERMARK FOR event_time AS event_time - INTERVAL '5' SECOND
) WITH (
    'connector' = 'datagen',
    'rows-per-second' = '10',
    'fields.order_id.kind' = 'random',
    'fields.order_id.min' = '1',
    'fields.order_id.max' = '1000000',
    'fields.product_id.kind' = 'random',
    'fields.product_id.min' = '1',
    'fields.product_id.max' = '1000',
    'fields.amount.kind' = 'random',
    'fields.amount.min' = '1',
    'fields.amount.max' = '10000'
);

-- 2. Console 输出表
CREATE TABLE console_output (
    window_start TIMESTAMP(3),
    product_id INT,
    sales DOUBLE,
    order_cnt BIGINT,
    rn BIGINT
) WITH (
    'connector' = 'print'
);

-- 3. Tumble 10s 窗口聚合 + ROW_NUMBER Top 3
INSERT INTO console_output
SELECT
    window_start,
    product_id,
    sales,
    order_cnt,
    rn
FROM (
    SELECT
        TUMBLE_START(event_time, INTERVAL '10' SECOND) AS window_start,
        product_id,
        SUM(amount) AS sales,
        COUNT(*) AS order_cnt,
        ROW_NUMBER() OVER (
            PARTITION BY TUMBLE_START(event_time, INTERVAL '10' SECOND)
            ORDER BY SUM(amount) DESC
        ) AS rn
    FROM orders_source
    GROUP BY TUMBLE(event_time, INTERVAL '10' SECOND), product_id
)
WHERE rn <= 3;


异常事件过滤 + 窗口 TopN

Prompt:
## Prompt
我需要你写一段 Flink SQL，用 datagen 内置表造设备事件流，过滤出异常事件后按省份做窗口 TopN 排名，输出到 console。

**业务背景与目标**：用内置 datagen 表生成设备事件数据，字段包括 device_id（int）、event_type（int）、province（string）、ts（timestamp），每秒 80 条，event_type 取值 1-10，province 长度 4 字符，用 ts 字段作为事件时间开 1 分钟滚动窗口（watermark 延迟 5 秒），先过滤 event_type=3 的异常事件，然后按省份分组统计每分钟的异常事件数，按异常数降序排名取 Top-5，输出字段包括窗口开始时间、省份、异常事件数、排名，打印到 console。

**源表定义**：
- `device_source`（设备事件流，datagen 连接器）：
  - device_id INT：设备 ID，随机生成
  - event_type INT：事件类型，随机取值 1~10
  - province VARCHAR：省份，随机生成，长度 4 字符
  - event_time：使用 LOCALTIMESTAMP 生成事件时间，设置 WATERMARK 延迟 5 秒
  - 生成速率：rows-per-second = 80

**输出表定义**：
- `console_output`（print 连接器）：
  - window_start TIMESTAMP：窗口开始时间
  - province VARCHAR：省份
  - abnormal_cnt BIGINT：异常事件数
  - `rank` BIGINT：排名（注意 rank 是关键字，需要用反引号包裹）

**查询逻辑**：
- 过滤：WHERE event_type = 3（异常事件）
- 窗口：TUMBLE(event_time, INTERVAL '1' MINUTE) 滚动窗口
- 聚合：按 province 和窗口分组，COUNT(*) 统计异常事件数
- 排名：ROW_NUMBER() OVER (PARTITION BY window_start ORDER BY abnormal_cnt DESC) AS `rank`
- Top-N：WHERE `rank` <= 5

**输出要求**：
- 使用 INSERT INTO console_output 输出结果
- 输出字段顺序：window_start, province, abnormal_cnt, `rank`

-- flinksql_006: 异常事件过滤 + 窗口 TopN
-- datagen 生成设备事件流 @ 80 rows/sec, event_type=3 过滤后 Tumble 1min Top-5

-- 1. 设备事件流 datagen 源表
CREATE TABLE device_source (
    device_id INT,
    event_type INT,
    province VARCHAR,
    event_time AS LOCALTIMESTAMP,
    WATERMARK FOR event_time AS event_time - INTERVAL '5' SECOND
) WITH (
    'connector' = 'datagen',
    'rows-per-second' = '80',
    'fields.event_type.kind' = 'random',
    'fields.event_type.min' = '1',
    'fields.event_type.max' = '10',
    'fields.province.kind' = 'random',
    'fields.province.length' = '4'
);

-- 2. Console 输出表
CREATE TABLE console_output (
    window_start TIMESTAMP,
    province VARCHAR,
    abnormal_cnt BIGINT,
    `rank` BIGINT
) WITH (
    'connector' = 'print'
);

-- 3. 过滤异常事件(event_type=3) → Tumble 窗口 → Top-5
INSERT INTO console_output
SELECT
    window_start,
    province,
    abnormal_cnt,
    `rank`
FROM (
    SELECT
        TUMBLE_START(event_time, INTERVAL '1' MINUTE) AS window_start,
        province,
        COUNT(*) AS abnormal_cnt,
        ROW_NUMBER() OVER (PARTITION BY TUMBLE_START(event_time, INTERVAL '1' MINUTE) ORDER BY COUNT(*) DESC) AS `rank`
    FROM device_source
    WHERE event_type = 3
    GROUP BY TUMBLE(event_time, INTERVAL '1' MINUTE), province
)
WHERE `rank` <= 5;


用户行为滚动窗口统计

Prompt:
## Prompt
我需要你写一段 Flink SQL，用 datagen 内置表造用户行为流，开滚动窗口统计每个用户的 action 数，结果写到 console。

**业务背景与目标**：用 datagen 内置数据源表模拟用户行为数据流，每秒生成 50 条记录，包含三个字段：userId（用户ID，字符串，长度6）、action（动作类型，字符串，长度8）、ftime（事件时间，timestamp 类型）。请基于这个 datagen 表，按事件时间做 1 分钟滚动窗口聚合，统计每个用户在每个 1 分钟窗口内的 action 总次数，并把结果（用户ID、动作次数、窗口起始时间）写入到内置控制台表中输出。事件时间字段 ftime 使用 LOCALTIMESTAMP 作为 computed column 并设置 watermark 延迟 5 秒。

**源表定义**：
- `user_source`（用户行为流，datagen 连接器）：
  - userId VARCHAR：用户 ID，随机生成，长度 6 字符
  - action VARCHAR：动作类型，随机生成，长度 8 字符
  - ftime：使用 LOCALTIMESTAMP 生成事件时间，设置 WATERMARK 延迟 5 秒
  - 生成速率：rows-per-second = 50

**输出表定义**：
- `console_output`（print 连接器）：
  - user_id VARCHAR：用户 ID
  - action_count BIGINT：动作次数
  - window_start TIMESTAMP：窗口起始时间

**查询逻辑**：
- 窗口：TUMBLE(ftime, INTERVAL '1' MINUTE) 滚动窗口
- 聚合：按 userId 分组，COUNT(*) 统计 action 总次数
- 输出：窗口起始时间 + 用户 ID + 动作次数

**输出要求**：
- 使用 INSERT INTO console_output 输出结果
- 输出字段顺序：user_id, action_count, window_start

-- flinksql_007: 用户行为滚动窗口统计
-- datagen 生成用户行为流 @ 50 rows/sec, Tumble 1min 窗口聚合

-- 1. 用户行为流 datagen 源表
CREATE TABLE user_source (
    userId VARCHAR,
    action VARCHAR,
    ftime AS LOCALTIMESTAMP,
    WATERMARK FOR ftime AS ftime - INTERVAL '5' SECOND
) WITH (
    'connector' = 'datagen',
    'rows-per-second' = '50',
    'fields.userId.length' = '6',
    'fields.action.length' = '8'
);

-- 2. Console 输出表
CREATE TABLE console_output (
    user_id VARCHAR,
    action_count BIGINT,
    window_start TIMESTAMP
) WITH (
    'connector' = 'print'
);

-- 3. Tumble 1min 窗口聚合: 统计每个用户每分钟的 action 次数
INSERT INTO console_output
SELECT
    userId AS user_id,
    COUNT(*) AS action_count,
    TUMBLE_START(ftime, INTERVAL '1' MINUTE) AS window_start
FROM user_source
GROUP BY TUMBLE(ftime, INTERVAL '1' MINUTE), userId;


Session Window User Behavior Aggregation

Prompt:
## Prompt
I need you to write a Flink SQL that uses a datagen built-in table to simulate a user behavior stream, performs session window aggregation to compute behavioral metrics per user within each session, and outputs the results to the console.

**Business Background and Objective**: Simulate a user behavior data stream using a datagen built-in table that generates 50 records per second, containing four fields: `userId` (user ID, string of length 6), `action` (action type, string of length 8), `action_time` (action occurrence time, timestamp), and `ftime` (event time, timestamp). Based on this datagen table, using `ftime` as the event time (with a 5-second watermark delay), perform a 30-minute session window (`SESSION`) aggregation grouped by `userId`. Compute the following metrics for each user within each session window: session start time, session end time, total action count, number of distinct action types, and the last action time (the maximum value of `action_time`). Output the results to a built-in console table.

**Source Table Definition**:
- `user_source` (user behavior stream, datagen connector):
  - `userId VARCHAR`: User ID, randomly generated, 6 characters in length
  - `action VARCHAR`: Action type, randomly generated, 8 characters in length
  - `action_time TIMESTAMP(3)`: Action occurrence time
  - `ftime`: Uses `LOCALTIMESTAMP` to generate event time, with a WATERMARK delay of 5 seconds
  - Generation rate: `rows-per-second = 50`

**Output Table Definition**:
- `console_output` (print connector):
  - `user_id VARCHAR`: User ID
  - `session_start TIMESTAMP`: Session start time
  - `session_end TIMESTAMP`: Session end time
  - `action_count BIGINT`: Total action count
  - `unique_actions BIGINT`: Number of distinct action types
  - `last_action_time TIMESTAMP`: Last action time

**Query Logic**:
- Window: `SESSION(ftime, INTERVAL '30' MINUTE)` session window
- Grouping: Group by `userId` and the session window
- Aggregation: `COUNT(*)` for total action count, `COUNT(DISTINCT action)` for distinct action type count, `MAX(action_time)` for last action time
- Window functions: Use `SESSION_START` / `SESSION_END` to obtain the window start and end times

**Output Requirements**:
- Use `INSERT INTO console_output` to output the results.
- Output field order: `user_id`, `session_start`, `session_end`, `action_count`, `unique_actions`, `last_action_time`

-- flinksql_008: 会话窗口用户行为聚合
-- datagen 生成用户行为流 @ 50 rows/sec, SESSION 30min 窗口聚合

-- 1. 用户行为流 datagen 源表
CREATE TABLE user_source (
    userId VARCHAR,
    action VARCHAR,
    action_time TIMESTAMP(3),
    ftime AS LOCALTIMESTAMP,
    WATERMARK FOR ftime AS ftime - INTERVAL '5' SECOND
) WITH (
    'connector' = 'datagen',
    'rows-per-second' = '50',
    'fields.userId.length' = '6',
    'fields.action.length' = '8'
);

-- 2. Console 输出表
CREATE TABLE console_output (
    user_id VARCHAR,
    session_start TIMESTAMP,
    session_end TIMESTAMP,
    action_count BIGINT,
    unique_actions BIGINT,
    last_action_time TIMESTAMP
) WITH (
    'connector' = 'print'
);

-- 3. SESSION 30min 窗口聚合: 统计每个用户每段会话的行为指标
INSERT INTO console_output
SELECT
    userId AS user_id,
    SESSION_START(ftime, INTERVAL '30' MINUTE) AS session_start,
    SESSION_END(ftime, INTERVAL '30' MINUTE) AS session_end,
    COUNT(*) AS action_count,
    COUNT(DISTINCT action) AS unique_actions,
    MAX(action_time) AS last_action_time
FROM user_source
GROUP BY SESSION(ftime, INTERVAL '30' MINUTE), userId;


Tumbling Window PV/UV Statistics

Prompt:
## Prompt
I need you to write a Flink SQL that uses a datagen built-in table to simulate a user click stream, computes PV and UV per page over a window, and writes the results to the console.

**Business Background and Objective**: Simulate a user click stream using a datagen built-in table (`user_id` ranging from 1–1000, `page_id` ranging from 1–50, 10,000 records per second). Compute the PV (`COUNT(*)`) and UV (`COUNT(DISTINCT user_id)`) per page over a 1-minute tumbling window. Output the results (window start, window end, `page_id`, `pv`, `uv`) to the console.

**Source Table Definition**:
- `click_source` (user click stream, datagen connector):
  - `user_id INT`: User ID, randomly generated in the range 1–1000
  - `page_id INT`: Page ID, randomly generated in the range 1–50
  - `event_time`: Uses `LOCALTIMESTAMP` to generate event time, with a WATERMARK delay of 5 seconds
  - Generation rate: `rows-per-second = 10000`

**Output Table Definition**:
- `console_output` (print connector):
  - `window_start TIMESTAMP`: Window start time
  - `window_end TIMESTAMP`: Window end time
  - `page_id INT`: Page ID
  - `pv BIGINT`: Page views (PV)
  - `uv BIGINT`: Unique visitors (UV)

**Query Logic**:
- Window: `TUMBLE(event_time, INTERVAL '1' MINUTE)` tumbling window
- Grouping: Group by `page_id` and the window
- PV: `COUNT(*)` to compute page views
- UV: `COUNT(DISTINCT user_id)` to compute unique visitors
- Window functions: Use `TUMBLE_START` / `TUMBLE_END` to obtain the window start and end times

**Output Requirements**:
- Use `INSERT INTO console_output` to output the results.
- Output field order: `window_start`, `window_end`, `page_id`, `pv`, `uv`

-- flinksql_009: 滚动窗口 PV/UV 统计
-- datagen 生成用户点击流 @ 10000 rows/sec, Tumble 1min PV/UV

-- 1. 用户点击流 datagen 源表
CREATE TABLE click_source (
    user_id INT,
    page_id INT,
    event_time AS LOCALTIMESTAMP,
    WATERMARK FOR event_time AS event_time - INTERVAL '5' SECOND
) WITH (
    'connector' = 'datagen',
    'rows-per-second' = '10000',
    'fields.user_id.kind' = 'random',
    'fields.user_id.min' = '1',
    'fields.user_id.max' = '1000',
    'fields.page_id.kind' = 'random',
    'fields.page_id.min' = '1',
    'fields.page_id.max' = '50'
);

-- 2. Console 输出表
CREATE TABLE console_output (
    window_start TIMESTAMP,
    window_end TIMESTAMP,
    page_id INT,
    pv BIGINT,
    uv BIGINT
) WITH (
    'connector' = 'print'
);

-- 3. Tumble 1min 窗口聚合: 按页面统计 PV 和 UV
INSERT INTO console_output
SELECT
    TUMBLE_START(event_time, INTERVAL '1' MINUTE) AS window_start,
    TUMBLE_END(event_time, INTERVAL '1' MINUTE) AS window_end,
    page_id,
    COUNT(*) AS pv,
    COUNT(DISTINCT user_id) AS uv
FROM click_source
GROUP BY TUMBLE(event_time, INTERVAL '1' MINUTE), page_id;


滑动窗口(HOP)店铺统计

Prompt:
## Prompt
我需要你写一段 Flink SQL，用 datagen 模拟订单流，按店铺做滑动窗口统计订单金额和数量，写入 console。

**业务背景与目标**：用 datagen 模拟订单流（order_id 取值 1~1000000、shop_id 取值 1~100、amount 取值 0~1000，每秒 5000 条），按 1 分钟步长、5 分钟窗口大小做 HOP 滑动窗口统计每个店铺的订单总额（SUM）、订单数（COUNT）和平均金额（AVG），写入 console。

**源表定义**：
- `order_source`（订单流，datagen 连接器）：
  - order_id BIGINT：订单 ID，随机取值 1~1000000
  - shop_id INT：店铺 ID，随机取值 1~100
  - amount DOUBLE：订单金额，随机取值 0~1000
  - event_time：使用 LOCALTIMESTAMP 生成事件时间，设置 WATERMARK 延迟 10 秒
  - 生成速率：rows-per-second = 5000

**输出表定义**：
- `console_output`（print 连接器）：
  - window_start TIMESTAMP：窗口开始时间
  - window_end TIMESTAMP：窗口结束时间
  - shop_id INT：店铺 ID
  - total_amount DOUBLE：订单总额
  - order_cnt BIGINT：订单数量
  - avg_amount DOUBLE：平均订单金额

**查询逻辑**：
- 窗口：HOP(event_time, INTERVAL '1' MINUTE, INTERVAL '5' MINUTE) 滑动窗口
  - 步长（slide）：1 分钟
  - 窗口大小（size）：5 分钟
- 分组：按 shop_id 和窗口分组
- 聚合：SUM(amount) 订单总额，COUNT(*) 订单数，AVG(amount) 平均金额
- 窗口函数：HOP_START / HOP_END 获取窗口起止时间

**输出要求**：
- 使用 INSERT INTO console_output 输出结果
- 输出字段顺序：window_start, window_end, shop_id, total_amount, order_cnt, avg_amount

-- flinksql_010: 滑动窗口(HOP)店铺统计
-- datagen 生成订单流 @ 5000 rows/sec, HOP 1min slide 5min window

-- 1. 订单流 datagen 源表
CREATE TABLE order_source (
    order_id BIGINT,
    shop_id INT,
    amount DOUBLE,
    event_time AS LOCALTIMESTAMP,
    WATERMARK FOR event_time AS event_time - INTERVAL '10' SECOND
) WITH (
    'connector' = 'datagen',
    'rows-per-second' = '5000',
    'fields.order_id.kind' = 'random',
    'fields.order_id.min' = '1',
    'fields.order_id.max' = '1000000',
    'fields.shop_id.kind' = 'random',
    'fields.shop_id.min' = '1',
    'fields.shop_id.max' = '100',
    'fields.amount.kind' = 'random',
    'fields.amount.min' = '0',
    'fields.amount.max' = '1000'
);

-- 2. Console 输出表
CREATE TABLE console_output (
    window_start TIMESTAMP,
    window_end TIMESTAMP,
    shop_id INT,
    total_amount DOUBLE,
    order_cnt BIGINT,
    avg_amount DOUBLE
) WITH (
    'connector' = 'print'
);

-- 3. HOP 滑动窗口聚合: 按店铺统计订单总额、数量和平均金额
INSERT INTO console_output
SELECT
    HOP_START(event_time, INTERVAL '1' MINUTE, INTERVAL '5' MINUTE) AS window_start,
    HOP_END(event_time, INTERVAL '1' MINUTE, INTERVAL '5' MINUTE) AS window_end,
    shop_id,
    SUM(amount) AS total_amount,
    COUNT(*) AS order_cnt,
    AVG(amount) AS avg_amount
FROM order_source
GROUP BY HOP(event_time, INTERVAL '1' MINUTE, INTERVAL '5' MINUTE), shop_id;


Tumbling Window Product TopN Ranking

Prompt:
## Prompt
I need you to write a Flink SQL that uses the built-in datagen connector to simulate an order data source, aggregates product sales over a 10-minute tumbling window, and then uses the `ROW_NUMBER` window function to retrieve the top 5 products by sales within each window, outputting the results to the console.

**Business Background and Objective**: Simulate an order stream using a datagen built-in table, generating massive order data. Group by a 10-minute tumbling window to compute the total sales amount and order count per product within each window, and finally use `ROW_NUMBER` ranking to retrieve the top 5 products by sales amount within each window.

**Source Table Definition**:
- `order_source` (order stream, datagen connector):
  - `order_id BIGINT`: Order ID, randomly generated in the range 1–10,000,000
  - `product_id INT`: Product ID, randomly generated in the range 1–200
  - `amount DOUBLE`: Order amount, randomly generated in the range 0–500
  - `event_time`: Uses `LOCALTIMESTAMP` to generate event time, with a WATERMARK delay of 5 seconds
  - Generation rate: `rows-per-second = 8000`

**Output Table Definition**:
- `console_output` (print connector):
  - `window_start TIMESTAMP(3)`: Tumbling window start time
  - `product_id INT`: Product ID
  - `sales DOUBLE`: Total sales amount of the product within the window
  - `order_cnt BIGINT`: Number of orders for the product within the window
  - `rn BIGINT`: Sales ranking within the window

**SQL Logic**:
- Apply `TUMBLE(event_time, INTERVAL '10' MINUTE)` tumbling window on `order_source`
- Group by `window_start`, `window_end`, `product_id`
- Aggregate `SUM(amount) AS sales` and `COUNT(*) AS order_cnt`
- Use `ROW_NUMBER() OVER (PARTITION BY window_start, window_end ORDER BY sales DESC) AS rn`
- Outer query filter: `WHERE rn <= 5`
- Final SELECT: `window_start`, `product_id`, `sales`, `order_cnt`, `rn`

**Output Requirements**:
- Use `INSERT INTO console_output` to output the results.
- Output field order: `window_start`, `product_id`, `sales`, `order_cnt`, `rn`

-- flinksql_011: 滚动窗口商品 TopN 排行
-- 10-min TUMBLE + SUM(amount) + COUNT(*) + ROW_NUMBER Top-5

-- 1. 订单源 datagen 表
CREATE TABLE order_source (
    order_id BIGINT,
    product_id INT,
    amount DOUBLE,
    event_time AS LOCALTIMESTAMP,
    WATERMARK FOR event_time AS event_time - INTERVAL '5' SECOND
) WITH (
    'connector' = 'datagen',
    'rows-per-second' = '8000',
    'fields.order_id.kind' = 'random',
    'fields.order_id.min' = '1',
    'fields.order_id.max' = '10000000',
    'fields.product_id.kind' = 'random',
    'fields.product_id.min' = '1',
    'fields.product_id.max' = '200',
    'fields.amount.kind' = 'random',
    'fields.amount.min' = '0',
    'fields.amount.max' = '500'
);

-- 2. Console 输出表
CREATE TABLE console_output (
    window_start TIMESTAMP(3),
    product_id INT,
    sales DOUBLE,
    order_cnt BIGINT,
    rn BIGINT
) WITH (
    'connector' = 'print'
);

-- 3. TUMBLE 窗口聚合 + ROW_NUMBER Top-5
INSERT INTO console_output
SELECT
    window_start,
    product_id,
    sales,
    order_cnt,
    rn
FROM (
    SELECT
        window_start,
        window_end,
        product_id,
        sales,
        order_cnt,
        ROW_NUMBER() OVER (
            PARTITION BY window_start, window_end
            ORDER BY sales DESC
        ) AS rn
    FROM (
        SELECT
            TUMBLE_START(event_time, INTERVAL '10' MINUTE) AS window_start,
            TUMBLE_END(event_time, INTERVAL '10' MINUTE) AS window_end,
            product_id,
            SUM(amount) AS sales,
            COUNT(*) AS order_cnt
        FROM order_source
        GROUP BY
            TUMBLE(event_time, INTERVAL '10' MINUTE),
            product_id
    )
)
WHERE rn <= 5;


Session Window User Behavior Statistics

Prompt:
## Prompt
I need you to write a Flink SQL that uses the built-in datagen connector to simulate a user behavior data source, computes per-user behavioral statistics over a 30-minute session window (`SESSION`), and outputs the results to the console.

**Business Background and Objective**: Simulate a user behavior stream using a datagen built-in table. Group by a 30-minute session window to compute the total action count (`COUNT`) and the number of distinct action types (`COUNT DISTINCT`) per user within each session window. Output the results to the console.

**Source Table Definition**:
- `user_actions` (user behavior stream, datagen connector):
  - `user_id INT`: User ID, randomly generated in the range 1–5000
  - `action_id INT`: Action type ID, randomly generated in the range 1–10
  - `event_time`: Uses `LOCALTIMESTAMP` to generate event time, with a WATERMARK delay of 5 seconds
  - Generation rate: `rows-per-second = 2000`

**Output Table Definition**:
- `console_output` (print connector):
  - `user_id INT`: User ID
  - `session_start TIMESTAMP(3)`: Session window start time
  - `session_end TIMESTAMP(3)`: Session window end time
  - `action_count BIGINT`: Total action count
  - `unique_actions BIGINT`: Number of distinct action types

**SQL Logic**:
- Apply `SESSION(event_time, INTERVAL '30' MINUTE)` session window on `user_actions`
- Group by `SESSION(event_time, INTERVAL '30' MINUTE)` and `user_id`
- Aggregate `COUNT(*) AS action_count` and `COUNT(DISTINCT action_id) AS unique_actions`
- Use `SESSION_START` and `SESSION_END` functions to obtain the window start and end times

**Output Requirements**:
- Use `INSERT INTO console_output` to output the results.
- Output field order: `user_id`, `session_start`, `session_end`, `action_count`, `unique_actions`

-- flinksql_012: 会话窗口用户行为统计
-- 30-min SESSION + COUNT + COUNT DISTINCT

-- 1. 用户行为 datagen 源表
CREATE TABLE user_actions (
    user_id INT,
    action_id INT,
    event_time AS LOCALTIMESTAMP,
    WATERMARK FOR event_time AS event_time - INTERVAL '5' SECOND
) WITH (
    'connector' = 'datagen',
    'rows-per-second' = '2000',
    'fields.user_id.kind' = 'random',
    'fields.user_id.min' = '1',
    'fields.user_id.max' = '5000',
    'fields.action_id.kind' = 'random',
    'fields.action_id.min' = '1',
    'fields.action_id.max' = '10'
);

-- 2. Console 输出表
CREATE TABLE console_output (
    user_id INT,
    session_start TIMESTAMP(3),
    session_end TIMESTAMP(3),
    action_count BIGINT,
    unique_actions BIGINT
) WITH (
    'connector' = 'print'
);

-- 3. SESSION 窗口聚合
INSERT INTO console_output
SELECT
    user_id,
    SESSION_START(event_time, INTERVAL '30' MINUTE) AS session_start,
    SESSION_END(event_time, INTERVAL '30' MINUTE) AS session_end,
    COUNT(*) AS action_count,
    COUNT(DISTINCT action_id) AS unique_actions
FROM user_actions
GROUP BY
    SESSION(event_time, INTERVAL '30' MINUTE),
    user_id;


累积窗口日活统计 (Cumulate UV)

Prompt:
## Prompt
我需要你写一段 Flink SQL，用 datagen 内置连接器模拟用户登录数据源，通过累积窗口（CUMULATE）统计日活 UV 和登录次数，输出到 console。

**业务背景与目标**：用 datagen 内置表模拟用户登录流水。使用 CUMULATE TVF 累积窗口，窗口步长 1 分钟、最大窗口 1 天，统计每个累积窗口内的独立用户数（UV）和登录总次数，输出到 console。

**源表定义**：
- `user_source`（用户登录流，datagen 连接器）：
  - user_id INT：用户 ID，随机取值 1~100000
  - device_id INT：设备 ID，随机取值 1~5
  - event_time：使用 LOCALTIMESTAMP 生成事件时间，设置 WATERMARK 延迟 5 秒
  - 生成速率：rows-per-second = 3000

**输出表定义**：
- `console_output`（print 连接器）：
  - window_start TIMESTAMP(3)：累积窗口起始时间
  - window_end TIMESTAMP(3)：累积窗口结束时间
  - uv BIGINT：独立用户数（去重 user_id）
  - login_cnt BIGINT：登录总次数

**SQL 逻辑**：
- 使用 CUMULATE TVF：TABLE(CUMULATE(TABLE user_source, DESCRIPTOR(event_time), INTERVAL '1' MINUTE, INTERVAL '1' DAY))
- 按 window_start 和 window_end 分组
- 聚合 COUNT(DISTINCT user_id) AS uv 和 COUNT(*) AS login_cnt

**输出要求**：
- 使用 INSERT INTO console_output 将结果输出
- 输出字段顺序：window_start, window_end, uv, login_cnt

-- flinksql_013: 累积窗口日活统计 (Cumulate UV)
-- CUMULATE(1min step, 1day max) + COUNT DISTINCT + COUNT

-- 1. 用户登录 datagen 源表
CREATE TABLE user_source (
    user_id INT,
    device_id INT,
    event_time AS LOCALTIMESTAMP,
    WATERMARK FOR event_time AS event_time - INTERVAL '5' SECOND
) WITH (
    'connector' = 'datagen',
    'rows-per-second' = '3000',
    'fields.user_id.kind' = 'random',
    'fields.user_id.min' = '1',
    'fields.user_id.max' = '100000',
    'fields.device_id.kind' = 'random',
    'fields.device_id.min' = '1',
    'fields.device_id.max' = '5'
);

-- 2. Console 输出表
CREATE TABLE console_output (
    window_start TIMESTAMP(3),
    window_end TIMESTAMP(3),
    uv BIGINT,
    login_cnt BIGINT
) WITH (
    'connector' = 'print'
);

-- 3. CUMULATE TVF 窗口聚合
INSERT INTO console_output
SELECT
    window_start,
    window_end,
    COUNT(DISTINCT user_id) AS uv,
    COUNT(*) AS login_cnt
FROM TABLE(
    CUMULATE(
        TABLE user_source,
        DESCRIPTOR(event_time),
        INTERVAL '1' MINUTE,
        INTERVAL '1' DAY
    )
)
GROUP BY window_start, window_end;


Interval Join + 窗口聚合 (订单支付)

Prompt:
## Prompt
我需要你写一段 Flink SQL，用两个 datagen 内置连接器模拟订单流和支付流，先按 order_id 做 Interval Join 匹配订单与支付，再对匹配结果做 10 分钟滚动窗口聚合统计，输出到 console。

**业务背景与目标**：用两张 datagen 内置表分别模拟订单流和支付流。先按 order_id 做 Interval Join（支付时间在订单时间前后 10 分钟内），对匹配上的记录按 10 分钟滚动窗口聚合，统计每个窗口内的订单数和支付总额，输出到 console。

**源表定义**：
- `orders_source`（订单流，datagen 连接器）：
  - order_id INT：订单 ID，随机取值 1~1000000
  - user_id INT：用户 ID，随机取值 1~100000
  - order_time：使用 LOCALTIMESTAMP 生成事件时间，设置 WATERMARK 延迟 60 秒
  - 生成速率：rows-per-second = 10000

- `payments_source`（支付流，datagen 连接器）：
  - order_id INT：订单 ID，随机取值 1~1000000
  - pay_amount DOUBLE：支付金额，随机取值 0~5000
  - pay_time：使用 LOCALTIMESTAMP 生成事件时间，设置 WATERMARK 延迟 60 秒
  - 生成速率：rows-per-second = 10000

**输出表定义**：
- `console_output`（print 连接器）：
  - window_start VARCHAR：滚动窗口起始时间
  - window_end VARCHAR：滚动窗口结束时间
  - order_count BIGINT：窗口内匹配订单数
  - total_pay DOUBLE：窗口内支付总额

**SQL 逻辑**：
- 第一步：orders_source 与 payments_source 做 Interval Join
  - 关联条件：o.order_id = p.order_id
  - 时间窗口：p.pay_time BETWEEN o.order_time - INTERVAL '10' MINUTE AND o.order_time + INTERVAL '10' MINUTE
  - 注意：对 pay_time 做 CAST(pay_time AS TIMESTAMP(3)) 降为普通时间戳
- 第二步：对 Join 结果做 10 分钟 TUMBLE 滚动窗口聚合
  - 使用 TUMBLE 按 o.order_time 作为事件时间
  - 聚合 COUNT(*) AS order_count 和 SUM(p.pay_amount) AS total_pay
  - 使用 TUMBLE_START 和 TUMBLE_END（CAST 为 VARCHAR）

**输出要求**：
- 使用 INSERT INTO console_output 将结果输出
- 输出字段顺序：window_start, window_end, order_count, total_pay

-- flinksql_014: Interval Join + 10-min TUMBLE 窗口聚合 (订单支付)
-- 双源 Interval Join → TUMBLE 聚合 → COUNT + SUM

-- 1. 订单流 datagen 源表
CREATE TABLE orders_source (
    order_id INT,
    user_id INT,
    order_time AS LOCALTIMESTAMP,
    WATERMARK FOR order_time AS order_time - INTERVAL '60' SECOND
) WITH (
    'connector' = 'datagen',
    'rows-per-second' = '10000',
    'fields.order_id.kind' = 'random',
    'fields.order_id.min' = '1',
    'fields.order_id.max' = '1000000',
    'fields.user_id.kind' = 'random',
    'fields.user_id.min' = '1',
    'fields.user_id.max' = '100000'
);

-- 2. 支付流 datagen 源表
CREATE TABLE payments_source (
    order_id INT,
    pay_amount DOUBLE,
    pay_time AS LOCALTIMESTAMP,
    WATERMARK FOR pay_time AS pay_time - INTERVAL '60' SECOND
) WITH (
    'connector' = 'datagen',
    'rows-per-second' = '10000',
    'fields.order_id.kind' = 'random',
    'fields.order_id.min' = '1',
    'fields.order_id.max' = '1000000',
    'fields.pay_amount.kind' = 'random',
    'fields.pay_amount.min' = '0',
    'fields.pay_amount.max' = '5000'
);

-- 3. Console 输出表
CREATE TABLE console_output (
    window_start VARCHAR,
    window_end VARCHAR,
    order_count BIGINT,
    total_pay DOUBLE
) WITH (
    'connector' = 'print'
);

-- 4. Interval Join + TUMBLE 窗口聚合
INSERT INTO console_output
SELECT
    CAST(TUMBLE_START(o.order_time, INTERVAL '10' MINUTE) AS VARCHAR) AS window_start,
    CAST(TUMBLE_END(o.order_time, INTERVAL '10' MINUTE) AS VARCHAR) AS window_end,
    COUNT(*) AS order_count,
    SUM(p.pay_amount) AS total_pay
FROM orders_source o
JOIN payments_source p
    ON o.order_id = p.order_id
WHERE
    p.pay_time BETWEEN o.order_time - INTERVAL '10' MINUTE
                   AND o.order_time + INTERVAL '10' MINUTE
GROUP BY
    TUMBLE(o.order_time, INTERVAL '10' MINUTE);


Regular Join 曝光点击关联

Prompt:
## Prompt
我需要你写一段 Flink SQL，用两个 datagen 内置连接器分别模拟广告曝光流和广告点击流，通过 Regular Join（常规内连接）关联曝光与点击，输出到 console。

**业务背景与目标**：用两张 datagen 内置表分别模拟广告曝光和广告点击数据。Regular Join 不需要事件时间或 Watermark，直接按 user_id 和 ad_id 做内连接，将同一用户对同一广告的曝光和点击关联起来输出。

**源表定义**：
- `impression_source`（曝光流，datagen 连接器）：
  - imp_id BIGINT：曝光 ID，随机取值 1~10000000
  - user_id INT：用户 ID，随机取值 1~100000
  - ad_id INT：广告 ID，随机取值 1~500
  - 生成速率：rows-per-second = 5000
  - 注意：Regular Join 不需要定义 event_time 或 WATERMARK

- `click_source`（点击流，datagen 连接器）：
  - click_id BIGINT：点击 ID，随机取值 1~10000000
  - user_id INT：用户 ID，随机取值 1~100000
  - ad_id INT：广告 ID，随机取值 1~500
  - 生成速率：rows-per-second = 1000
  - 注意：Regular Join 不需要定义 event_time 或 WATERMARK

**输出表定义**：
- `console_output`（print 连接器）：
  - user_id INT：用户 ID
  - ad_id INT：广告 ID
  - imp_id BIGINT：曝光 ID
  - click_id BIGINT：点击 ID

**Join 逻辑**：
- 使用 INNER JOIN 关联曝光流和点击流
- 关联条件：i.user_id = c.user_id AND i.ad_id = c.ad_id
- 这是一个 Regular Join，不需要 Interval Join 的时间窗口条件
- 注意：Regular Join 的 state 会持续增长，在生产环境需注意 TTL 配置

**输出要求**：
- 使用 INSERT INTO console_output 将匹配结果输出
- 输出字段顺序：user_id, ad_id, imp_id, click_id

-- flinksql_015: Regular Join (广告曝光 + 点击)
-- 双源无 WATERMARK + INNER JOIN on user_id AND ad_id

-- 1. 曝光流 datagen 源表 (无 event_time, 无 WATERMARK)
CREATE TABLE impression_source (
    imp_id BIGINT,
    user_id INT,
    ad_id INT
) WITH (
    'connector' = 'datagen',
    'rows-per-second' = '5000',
    'fields.imp_id.kind' = 'random',
    'fields.imp_id.min' = '1',
    'fields.imp_id.max' = '10000000',
    'fields.user_id.kind' = 'random',
    'fields.user_id.min' = '1',
    'fields.user_id.max' = '100000',
    'fields.ad_id.kind' = 'random',
    'fields.ad_id.min' = '1',
    'fields.ad_id.max' = '500'
);

-- 2. 点击流 datagen 源表 (无 event_time, 无 WATERMARK)
CREATE TABLE click_source (
    click_id BIGINT,
    user_id INT,
    ad_id INT
) WITH (
    'connector' = 'datagen',
    'rows-per-second' = '1000',
    'fields.click_id.kind' = 'random',
    'fields.click_id.min' = '1',
    'fields.click_id.max' = '10000000',
    'fields.user_id.kind' = 'random',
    'fields.user_id.min' = '1',
    'fields.user_id.max' = '100000',
    'fields.ad_id.kind' = 'random',
    'fields.ad_id.min' = '1',
    'fields.ad_id.max' = '500'
);

-- 3. Console 输出表
CREATE TABLE console_output (
    user_id INT,
    ad_id INT,
    imp_id BIGINT,
    click_id BIGINT
) WITH (
    'connector' = 'print'
);

-- 4. Regular Join: 按 user_id + ad_id 关联曝光和点击
INSERT INTO console_output
SELECT
    i.user_id,
    i.ad_id,
    i.imp_id,
    c.click_id
FROM impression_source i
INNER JOIN click_source c
    ON i.user_id = c.user_id
    AND i.ad_id = c.ad_id;


滚动窗口多维聚合 (GROUPING SETS)

Prompt:
## Prompt
我需要你写一段 Flink SQL，用 datagen 内置连接器模拟事件流，按 1 分钟滚动窗口做 GROUPING SETS 多维聚合，结果打到 console。

**业务背景与目标**：用 datagen 内置表生成设备事件流，共 5000 行/秒。字段包括 os_id（操作系统 ID，1~3）、device_id（设备 ID，1~20）、city_id（城市 ID，1~100）。使用 LOCALTIMESTAMP 作为事件时间，设置 5 秒 Watermark 延迟。

按 1 分钟 TUMBLE 滚动窗口，使用 GROUPING SETS 进行多维分组聚合，分组组合包括：(os_id, device_id)、(city_id)、以及全局汇总()。统计每个维度组合的事件数 COUNT(*)，输出到 console 表。

**源表定义**：
- `event_source`（datagen 连接器）：
  - os_id INT：操作系统 ID，随机取值 1~3
  - device_id INT：设备 ID，随机取值 1~20
  - city_id INT：城市 ID，随机取值 1~100
  - event_time：使用 LOCALTIMESTAMP 生成事件时间，设置 WATERMARK 延迟 5 秒
  - 生成速率：rows-per-second = 5000

**输出表定义**：
- `console_output`（print 连接器）：
  - window_start TIMESTAMP(3)：窗口起始时间
  - os_id INT：操作系统 ID（聚合维度可为 NULL）
  - device_id INT：设备 ID（聚合维度可为 NULL）
  - city_id INT：城市 ID（聚合维度可为 NULL）
  - event_cnt BIGINT：事件计数

**聚合逻辑**：
- 滚动窗口：TUMBLE 1 分钟
- 分组方式：GROUPING SETS ((os_id, device_id), (city_id), ())
- 聚合指标：COUNT(*) AS event_cnt
- 窗口起始时间用 TUMBLE_START 计算

**输出要求**：
- 使用 INSERT INTO console_output 将结果输出
- 输出字段顺序：window_start, os_id, device_id, city_id, event_cnt

-- flinksql_016: 滚动窗口多维聚合 (GROUPING SETS) — TVF syntax
-- 生成 5000 行/秒的设备事件流

-- 1. 设备事件流 datagen 源表
CREATE TABLE event_source (
    os_id INT,
    device_id INT,
    city_id INT,
    event_time AS LOCALTIMESTAMP,
    WATERMARK FOR event_time AS event_time - INTERVAL '5' SECOND
) WITH (
    'connector' = 'datagen',
    'rows-per-second' = '5000',
    'fields.os_id.kind' = 'random',
    'fields.os_id.min' = '1',
    'fields.os_id.max' = '3',
    'fields.device_id.kind' = 'random',
    'fields.device_id.min' = '1',
    'fields.device_id.max' = '20',
    'fields.city_id.kind' = 'random',
    'fields.city_id.min' = '1',
    'fields.city_id.max' = '100'
);

-- 2. Console 输出表
CREATE TABLE console_output (
    window_start TIMESTAMP(3),
    os_id INT,
    device_id INT,
    city_id INT,
    event_cnt BIGINT
) WITH (
    'connector' = 'print'
);

-- 3. TUMBLE TVF 1min + GROUPING SETS 多维聚合
INSERT INTO console_output
SELECT
    window_start,
    os_id,
    device_id,
    city_id,
    COUNT(*) AS event_cnt
FROM TABLE(
    TUMBLE(TABLE event_source, DESCRIPTOR(event_time), INTERVAL '1' MINUTE)
)
GROUP BY window_start, window_end,
    GROUPING SETS ((os_id, device_id), (city_id), ());


Row-Level TopN (Windowless)

Prompt:
## Prompt
I need you to write a Flink SQL that uses the built-in datagen connector to simulate a spending stream, retrieves the top 3 records by spending amount grouped by `country_id` without any window, and outputs the results directly to the console.

**Business Background and Objective**: Generate user spending records using a datagen built-in table at a rate of 1000 rows per second. Fields include `country_id` (country ID, 1–30), `user_id` (user ID, 1–100,000), and `cost_money` (spending amount, 0–10,000). No event time or watermark is required, since no window operations are involved.

Use the `ROW_NUMBER()` window function, partitioned by `country_id` and sorted by `cost_money` in descending order, keeping only the top 3 records per partition (`row_num <= 3`), and output the results directly to the console table. This is a continuous TopN query where each country's top 3 dynamically updates as new data flows in.

**Source Table Definition**:
- `spend_source` (datagen connector):
  - `country_id INT`: Country ID, randomly generated in the range 1–30
  - `user_id INT`: User ID, randomly generated in the range 1–100,000
  - `cost_money DOUBLE`: Spending amount, randomly generated in the range 0–10,000
  - Generation rate: `rows-per-second = 1000`

**Output Table Definition**:
- `console_output` (print connector):
  - `country_id INT`: Country ID
  - `user_id INT`: User ID
  - `cost_money DOUBLE`: Spending amount
  - `row_num BIGINT`: Row number

**TopN Logic**:
- Use `ROW_NUMBER() OVER (PARTITION BY country_id ORDER BY cost_money DESC)` to generate row numbers
- Apply `WHERE row_num <= 3` to keep the top 3 per country
- No window operation is required; no event time or watermark needs to be defined

**Output Requirements**:
- Use `INSERT INTO console_output` to output the results.
- Output field order: `country_id`, `user_id`, `cost_money`, `row_num`

-- flinksql_017: 行级 TopN (无窗口)
-- 生成 1000 行/秒的用户消费流

-- 1. 用户消费流 datagen 源表（无事件时间/Watermark）
CREATE TABLE spend_source (
    country_id INT,
    user_id INT,
    cost_money DOUBLE
) WITH (
    'connector' = 'datagen',
    'rows-per-second' = '1000',
    'fields.country_id.kind' = 'random',
    'fields.country_id.min' = '1',
    'fields.country_id.max' = '30',
    'fields.user_id.kind' = 'random',
    'fields.user_id.min' = '1',
    'fields.user_id.max' = '100000',
    'fields.cost_money.kind' = 'random',
    'fields.cost_money.min' = '0',
    'fields.cost_money.max' = '10000'
);

-- 2. Console 输出表
CREATE TABLE console_output (
    country_id INT,
    user_id INT,
    cost_money DOUBLE,
    row_num BIGINT
) WITH (
    'connector' = 'print'
);

-- 3. 行级 TopN: 每个 country_id 取 cost_money 最高的前 3 条
INSERT INTO console_output
SELECT
    country_id,
    user_id,
    cost_money,
    row_num
FROM (
    SELECT
        country_id,
        user_id,
        cost_money,
        ROW_NUMBER() OVER (
            PARTITION BY country_id
            ORDER BY cost_money DESC
        ) AS row_num
    FROM spend_source
) t
WHERE row_num <= 3;


Dual Output (Detail + Windowed Aggregation)

Prompt:
## Prompt
I need you to write a Flink SQL that uses the built-in datagen connector to simulate an order stream, simultaneously outputting detail records to `console1` and windowed aggregation results over a 1-minute tumbling window to `console2`.

**Business Background and Objective**: Generate an order event stream using a datagen built-in table at a rate of 2000 rows per second. Fields include `order_id` (order ID), `user_id` (user ID, 1–10,000), and `amount` (order amount, 0–1000). Use `LOCALTIMESTAMP` as the event time and set a 5-second watermark delay.

The same source data must be simultaneously output to two sinks:
- `console1`: Detail passthrough, directly outputting the original fields
- `console2`: Aggregation over a 1-minute `TUMBLE` tumbling window, grouped by `user_id` to compute the order count and total amount

**Source Table Definition**:
- `order_source` (datagen connector):
  - `order_id BIGINT`: Order ID, randomly generated
  - `user_id INT`: User ID, randomly generated in the range 1–10,000
  - `amount DOUBLE`: Order amount, randomly generated in the range 0–1000
  - `event_time`: Uses `LOCALTIMESTAMP` to generate event time, with a WATERMARK delay of 5 seconds
  - Generation rate: `rows-per-second = 2000`

**Output Table 1 Definition**:
- `console1` (print connector, detail output):
  - `order_id BIGINT`
  - `user_id INT`
  - `amount DOUBLE`

**Output Table 2 Definition**:
- `console2` (print connector, aggregation output):
  - `window_start TIMESTAMP(3)`: Window start time
  - `user_id INT`: User ID
  - `order_cnt BIGINT`: Order count within the window
  - `total_amount DOUBLE`: Total order amount within the window

**Aggregation Logic**:
- Detail output: Direct passthrough from `order_source`
- Aggregation output: `TUMBLE` 1-minute tumbling window, `GROUP BY user_id`
- Aggregation metrics: `COUNT(*) AS order_cnt`, `SUM(amount) AS total_amount`

**Output Requirements**:
- Use two `INSERT INTO` statements to write to `console1` and `console2` respectively.
- `console1` output field order: `order_id`, `user_id`, `amount`
- `console2` output field order: `window_start`, `user_id`, `order_cnt`, `total_amount`

-- flinksql_018: 双输出 (明细 + 窗口聚合)
-- 生成 2000 行/秒的订单事件流

-- 1. 订单事件流 datagen 源表
CREATE TABLE order_source (
    order_id BIGINT,
    user_id INT,
    amount DOUBLE,
    event_time AS LOCALTIMESTAMP,
    WATERMARK FOR event_time AS event_time - INTERVAL '5' SECOND
) WITH (
    'connector' = 'datagen',
    'rows-per-second' = '2000',
    'fields.order_id.kind' = 'random',
    'fields.order_id.min' = '1',
    'fields.order_id.max' = '1000000000',
    'fields.user_id.kind' = 'random',
    'fields.user_id.min' = '1',
    'fields.user_id.max' = '10000',
    'fields.amount.kind' = 'random',
    'fields.amount.min' = '0',
    'fields.amount.max' = '1000'
);

-- 2. Console1: 明细透传输出表
CREATE TABLE console1 (
    order_id BIGINT,
    user_id INT,
    amount DOUBLE
) WITH (
    'connector' = 'print'
);

-- 3. Console2: 窗口聚合输出表
CREATE TABLE console2 (
    window_start TIMESTAMP(3),
    user_id INT,
    order_cnt BIGINT,
    total_amount DOUBLE
) WITH (
    'connector' = 'print'
);

-- 4. 明细透传：直接输出原始字段
INSERT INTO console1
SELECT
    order_id,
    user_id,
    amount
FROM order_source;

-- 5. 窗口聚合：TUMBLE 1min 按 user_id 分组统计
INSERT INTO console2
SELECT
    TUMBLE_START(event_time, INTERVAL '1' MINUTE) AS window_start,
    user_id,
    COUNT(*) AS order_cnt,
    SUM(amount) AS total_amount
FROM order_source
GROUP BY TUMBLE(event_time, INTERVAL '1' MINUTE), user_id;


全局分组 + 过滤统计

Prompt:
## Prompt
我需要你写一段 Flink SQL，用 datagen 内置连接器模拟交易流，按状态过滤后全局分组统计，结果打到 console。

**业务背景与目标**：用 datagen 内置表生成交易事件流，共 3000 行/秒。字段包括 trade_id（交易 ID）、type（交易类型，1~5）、status（状态，0~2）、amount（交易金额，0~5000）。

注意：`type` 和 `status` 是 SQL 保留字，需要使用反引号包裹（例如 `` `type` ``、`` `status` ``）。

先按 status = 1 过滤，再按 type 分组统计交易笔数和交易金额总和，最后输出到 console 表。因为不涉及窗口操作，无需定义事件时间和 Watermark。

**源表定义**：
- `trade_source`（datagen 连接器）：
  - trade_id BIGINT：交易 ID，随机取值
  - `` `type` `` INT：交易类型，随机取值 1~5
  - `` `status` `` INT：状态，随机取值 0~2
  - amount DOUBLE：交易金额，随机取值 0~5000
  - 生成速率：rows-per-second = 3000

**输出表定义**：
- `console_output`（print 连接器）：
  - `` `type` `` INT：交易类型
  - trade_cnt BIGINT：交易笔数
  - total_amount DOUBLE：交易金额总和

**过滤与聚合逻辑**：
- 过滤条件：`` `status` `` = 1（仅统计状态为 1 的交易）
- 分组：按 `` `type` `` 分组
- 聚合指标：COUNT(*) AS trade_cnt, SUM(amount) AS total_amount

**输出要求**：
- 使用 INSERT INTO console_output 将结果输出
- 输出字段顺序：`` `type` ``, trade_cnt, total_amount

-- flinksql_019: 全局分组 + 过滤统计
-- 生成 3000 行/秒的交易事件流，使用反引号包裹保留字

-- 1. 交易事件流 datagen 源表（无事件时间/Watermark）
CREATE TABLE trade_source (
    trade_id BIGINT,
    `type` INT,
    `status` INT,
    amount DOUBLE
) WITH (
    'connector' = 'datagen',
    'rows-per-second' = '3000',
    'fields.trade_id.kind' = 'random',
    'fields.trade_id.min' = '1',
    'fields.trade_id.max' = '1000000000',
    'fields.type.kind' = 'random',
    'fields.type.min' = '1',
    'fields.type.max' = '5',
    'fields.status.kind' = 'random',
    'fields.status.min' = '0',
    'fields.status.max' = '2',
    'fields.amount.kind' = 'random',
    'fields.amount.min' = '0',
    'fields.amount.max' = '5000'
);

-- 2. Console 输出表
CREATE TABLE console_output (
    `type` INT,
    trade_cnt BIGINT,
    total_amount DOUBLE
) WITH (
    'connector' = 'print'
);

-- 3. 过滤 status=1 + 按 type 分组统计
INSERT INTO console_output
SELECT
    `type`,
    COUNT(*) AS trade_cnt,
    SUM(amount) AS total_amount
FROM trade_source
WHERE `status` = 1
GROUP BY `type`;


窗口去重 (TVF Dedup)

Prompt:
## Prompt
我需要你写一段 Flink SQL，用 datagen 内置连接器模拟事件流，使用 TUMBLE TVF 窗口表值函数 + ROW_NUMBER 做窗口去重，每个窗口内每个用户只保留最早的一条事件。

**业务背景与目标**：用 datagen 内置表生成用户事件流，共 5000 行/秒。字段包括 user_id（用户 ID，1~10000）、event_id（事件 ID）、event_type（事件类型，1~8）。使用 LOCALTIMESTAMP 作为事件时间，设置 10 秒 Watermark 延迟。

使用 TUMBLE 表值函数（TABLE(TUMBLE(...))）将事件分配到 5 分钟滚动窗口，然后在每个（window_start, user_id）分区内，按事件时间升序使用 ROW_NUMBER() 编号，只保留每个窗口每个用户的第一条事件（rn = 1），最后输出到 console。

**源表定义**：
- `event_source`（datagen 连接器）：
  - user_id INT：用户 ID，随机取值 1~10000
  - event_id BIGINT：事件 ID，随机取值
  - event_type INT：事件类型，随机取值 1~8
  - event_time：使用 LOCALTIMESTAMP 生成事件时间，设置 WATERMARK 延迟 10 秒
  - 生成速率：rows-per-second = 5000

**输出表定义**：
- `console_output`（print 连接器）：
  - window_start TIMESTAMP(3)：窗口起始时间
  - window_end TIMESTAMP(3)：窗口结束时间
  - user_id INT：用户 ID
  - event_id BIGINT：事件 ID
  - event_type INT：事件类型

**去重逻辑**：
- 窗口：TUMBLE TVF 5 分钟滚动窗口
- 排序：ROW_NUMBER() OVER (PARTITION BY window_start, user_id ORDER BY event_time ASC)
- 过滤：WHERE rn = 1 保留每个窗口每个用户的第一条事件
- 窗口时间字段可从 TVF 的 window_start 和 window_end 获得

**输出要求**：
- 使用 INSERT INTO console_output 将结果输出
- 输出字段顺序：window_start, window_end, user_id, event_id, event_type

-- flinksql_020: 窗口去重 (TVF Dedup)
-- TUMBLE TVF + ROW_NUMBER 按窗口去重，保留每个窗口每个用户的第一条

-- 1. 用户事件流 datagen 源表
CREATE TABLE event_source (
    user_id INT,
    event_id BIGINT,
    event_type INT,
    event_time AS LOCALTIMESTAMP,
    WATERMARK FOR event_time AS event_time - INTERVAL '10' SECOND
) WITH (
    'connector' = 'datagen',
    'rows-per-second' = '5000',
    'fields.user_id.kind' = 'random',
    'fields.user_id.min' = '1',
    'fields.user_id.max' = '10000',
    'fields.event_id.kind' = 'random',
    'fields.event_id.min' = '1',
    'fields.event_id.max' = '1000000000',
    'fields.event_type.kind' = 'random',
    'fields.event_type.min' = '1',
    'fields.event_type.max' = '8'
);

-- 2. Console 输出表
CREATE TABLE console_output (
    window_start TIMESTAMP(3),
    window_end TIMESTAMP(3),
    user_id INT,
    event_id BIGINT,
    event_type INT
) WITH (
    'connector' = 'print'
);

-- 3. TUMBLE TVF + ROW_NUMBER 窗口去重
INSERT INTO console_output
SELECT
    window_start,
    window_end,
    user_id,
    event_id,
    event_type
FROM (
    SELECT
        window_start,
        window_end,
        user_id,
        event_id,
        event_type,
        ROW_NUMBER() OVER (
            PARTITION BY window_start, user_id
            ORDER BY event_time ASC
        ) AS rn
    FROM TABLE(
        TUMBLE(TABLE event_source, DESCRIPTOR(event_time), INTERVAL '5' MINUTE)
    )
) t
WHERE rn = 1;