代码评审组织排行

Prompt:
## Prompt
我需要你生成一段 PySpark 代码，产出代码协作平台代码评审（CR）按组织维度的排名表。

**业务背景与目标**：代码协作平台每天都会产生代码评审数据，包括评审注释（line_note）和评审耗时（used_time）。运营希望按公司/部门/中心/小组四级组织维度，对成员进行两个维度的 Top10 排名：(1) 按评审注释数降序排名（notes_create_desc）；(2) 按评审投入时间降序排名（cr_used_time_desc）。通过 rank_space 交叉连接生成完整的排名空间，左连接实际数据，产出组织排名明细表。

**输入表**：
- `internal_platform_db.dim_org_framework_org_d_f_pyspark_220`
- `internal_platform_db.dim_code_employee_user_org_relation_bare_d_f_pyspark_220`
- `internal_platform_db.dual_pyspark_220`
- `internal_platform_db.dwm_code_review_note_user_stat_d_f_pyspark_220`
- `internal_platform_db.dwd_code_review_used_time_d_i_pyspark_220`

**数据范围与过滤条件**：
- 运行日期 ds 固定为 '20260513'
- 月份范围：month_start='2026-05-01', month_end='2026-06-01'
- org 数据过滤 CONCAT(year,month,day) = ds
- note 数据过滤 concat(year,month,day) = ds，且 note_crt_date 在 [month_start, month_end) 区间
- used_time 数据过滤 dt 在 [month_start, month_end) 区间，且 (is_reviewer=1 OR is_fileowner=1)

**处理逻辑**：
1. 从 dim_org_framework_org 构建四级组织 CTE（cp 公司, dp 部门, ct 中心, gp 小组），每级提取对应的层级 ID
2. 将员工关联到各层级组织（cp_user, dp_user, ct_user, gp_user）
3. 计算 line_note_list（按 author_id 汇总 line_human_note_cnt）和 used_time_list（按 reviewer_id 汇总 duration_today）
4. 对每级组织的用户按两个 ranktype 分别做 rank() 窗口排序
5. 生成 rank_space（ranktype × rank 1-10 的笛卡尔积）
6. 各组织级别 cross join rank_space，left join 对应的排名结果
7. 四级结果 union all 后写入目标表

**输出要求**：
- 目标表：`internal_platform_db.ads_code_code_platform_mon_cr_org_rank_d_i_pyspark_220`
- 写入策略：INSERT OVERWRITE TABLE
- 输出列：month_name, month_start, month_end, org_id, org_type, ranktype, rank, user_id, user_name, value_string, value_int, value_float, value_datetime, created_at

#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
pyspark_220: 代码协作平台CR组织排名 - Code Review Org Ranking (L3)
- 沙箱化改写：硬编码日期 '20260513'，移除 pytoolkit/dateutil 依赖
- 使用 spark.sql() 执行全部 CTE 逻辑
"""

import datetime
from pyspark.sql import SparkSession

spark = (
    SparkSession.builder
    .appName('pyspark_016_cr_org_rank')
    .enableHiveSupport()
    .getOrCreate()
)

# Hardcoded dates
ds = '20260513'
year = '2026'
month = '05'
day = '13'
month_start = '2026-05-01'
month_end = '2026-06-01'

db = 'internal_platform_db'
tbl = 'ads_code_code_platform_mon_cr_org_rank_d_i_pyspark_220'

# Drop and create output table
spark.sql('''
DROP TABLE IF EXISTS {db}.{tbl}
'''.format(db=db, tbl=tbl))

spark.sql('''
CREATE TABLE {db}.{tbl} (
    `month_name` STRING COMMENT 'cr创建时间转7位月份，yyyy-mm',
    `month_start` STRING COMMENT '月份开始时点',
    `month_end` STRING COMMENT '月份结束时点',
    `org_id` BIGINT COMMENT '组织id',
    `org_type` STRING COMMENT '组织类型',
    `ranktype` STRING COMMENT '排名类型',
    `rank` BIGINT COMMENT '排名',
    `user_id` BIGINT COMMENT '代码协作平台user_id',
    `user_name` STRING COMMENT 'user_name',
    `value_string` STRING COMMENT '排名所用文本',
    `value_int` BIGINT COMMENT '排名所用整数',
    `value_float` DOUBLE COMMENT '排名所用浮点数',
    `value_datetime` STRING COMMENT '排名所用时间',
    `created_at` STRING COMMENT '数据生成时间'
)
PARTITIONED BY (
    `year` STRING,
    `month` STRING,
    `day` STRING
)
STORED AS ORC
'''.format(db=db, tbl=tbl))

# Main ETL query
query_sql = """
    WITH org_list AS (
        SELECT org_id AS id, org_name AS name, company_id, company_name,
               org_bg_01_id, org_bg_02_id,
               business_line_01_id, business_line_02_id,
               department_01_id, department_02_id, department_03_id, department_04_id,
               center_01_id, center_02_id, center_03_id,
               group_01_id, group_02_id, group_03_id, group_04_id,
               group_05_id, group_06_id, group_07_id, group_08_id
        FROM internal_platform_db.dim_org_framework_org_d_f_pyspark_220
        WHERE CONCAT(year,month,day) = '{ds}'
    ),
    cp AS (
        SELECT o.id org_id,'company' org_type
            ,company_id
            ,-10000 org_bg_id
            ,-10000 business_line_id
            ,-10000 department_id
            ,-10000 center_id
            ,-10000 group_id
        FROM org_list AS o
        WHERE id = company_id
    ),
    dp AS (
        SELECT o.id org_id,'department' org_type
            ,company_id
            ,org_bg_01_id org_bg_id
            ,business_line_01_id business_line_id
            ,department_01_id department_id
            ,-10000 center_id
            ,-10000 group_id
        FROM org_list as o
        WHERE id = department_01_id
    ),
    ct AS (
        SELECT o.id org_id,'center' org_type
            ,company_id
            ,org_bg_01_id org_bg_id
            ,business_line_01_id business_line_id
            ,department_01_id department_id
            ,center_01_id center_id
            ,-10000 group_id
        FROM org_list as o
        WHERE id = center_01_id
    ),
    gp AS (
        SELECT o.id org_id,'group' org_type
            ,company_id
            ,org_bg_01_id org_bg_id
            ,business_line_01_id business_line_id
            ,department_01_id department_id
            ,center_01_id center_id
            ,group_01_id group_id
        FROM org_list as o
        WHERE id = group_01_id
    ),
    user_org as (
        select user_id,
            user_name,
            org_id,
            company_id,
            org_bg_id,
            business_line_id,
            department_id,
            center_id,
            group_id
        from internal_platform_db.dim_code_employee_user_org_relation_bare_d_f_pyspark_220
        where concat(year, month, day) = '{ds}'
    ),
    rank_space as (
        select t1.ranktype,t2.rank
        from (
            select 'notes_create_desc' as ranktype from internal_platform_db.dual_pyspark_220 union all
            select 'cr_used_time_desc' as ranktype from internal_platform_db.dual_pyspark_220
        ) as t1
        cross join (
            select 1  as rank from internal_platform_db.dual_pyspark_220 union all
            select 2  as rank from internal_platform_db.dual_pyspark_220 union all
            select 3  as rank from internal_platform_db.dual_pyspark_220 union all
            select 4  as rank from internal_platform_db.dual_pyspark_220 union all
            select 5  as rank from internal_platform_db.dual_pyspark_220 union all
            select 6  as rank from internal_platform_db.dual_pyspark_220 union all
            select 7  as rank from internal_platform_db.dual_pyspark_220 union all
            select 8  as rank from internal_platform_db.dual_pyspark_220 union all
            select 9  as rank from internal_platform_db.dual_pyspark_220 union all
            select 10 as rank from internal_platform_db.dual_pyspark_220
        ) as t2 on 1 = 1
    ),
    line_note_list as (
        select author_id user_id
            ,sum(line_human_note_cnt) line_human_note_cnt
        from internal_platform_db.dwm_code_review_note_user_stat_d_f_pyspark_220
        where concat(year,month,day) = '{ds}'
            and note_crt_date >= '{month_start}'
            and note_crt_date < '{month_end}'
        group by author_id
    ),
    used_time_list as (
        select reviewer_id user_id
            , sum(duration_today) duration_today
        from internal_platform_db.dwd_code_review_used_time_d_i_pyspark_220
        where concat(year,'-', month,'-', day) >= '{month_start}'
            and concat(year,'-', month,'-', day) < '{month_end}'
            and (is_reviewer = 1 or is_fileowner = 1)
        group by reviewer_id
    ),
    cp_user as (
        select t1.org_id,t1.org_type,t1.company_id,t1.org_bg_id,t1.business_line_id,t1.department_id,t1.center_id,t1.group_id,t2.user_id,t2.user_name
        from cp as t1 left join user_org as t2 on t1.org_id = t2.company_id
        group by t1.org_id,t1.org_type,t1.company_id,t1.org_bg_id,t1.business_line_id,t1.department_id,t1.center_id,t1.group_id,t2.user_id,t2.user_name
    ),
    dp_user as (
        select t1.org_id,t1.org_type,t1.company_id,t1.org_bg_id,t1.business_line_id,t1.department_id,t1.center_id,t1.group_id,t2.user_id,t2.user_name
        from dp as t1 left join user_org as t2 on t1.org_id = t2.department_id
        group by t1.org_id,t1.org_type,t1.company_id,t1.org_bg_id,t1.business_line_id,t1.department_id,t1.center_id,t1.group_id,t2.user_id,t2.user_name
    ),
    ct_user as (
        select t1.org_id,t1.org_type,t1.company_id,t1.org_bg_id,t1.business_line_id,t1.department_id,t1.center_id,t1.group_id,t2.user_id,t2.user_name
        from ct as t1 left join user_org as t2 on t1.org_id = t2.center_id
        group by t1.org_id,t1.org_type,t1.company_id,t1.org_bg_id,t1.business_line_id,t1.department_id,t1.center_id,t1.group_id,t2.user_id,t2.user_name
    ),
    gp_user as (
        select t1.org_id,t1.org_type,t1.company_id,t1.org_bg_id,t1.business_line_id,t1.department_id,t1.center_id,t1.group_id,t2.user_id,t2.user_name
        from gp as t1 left join user_org as t2 on t1.org_id = t2.group_id
        group by t1.org_id,t1.org_type,t1.company_id,t1.org_bg_id,t1.business_line_id,t1.department_id,t1.center_id,t1.group_id,t2.user_id,t2.user_name
    ),
    cp_rank as (
        select t1.*
            ,'notes_create_desc' ranktype
            ,rank() over(partition by t1.org_id order by t2.line_human_note_cnt  desc) rank
            ,null value_string
            ,t2.line_human_note_cnt value_int
            ,null value_float
            ,null value_datetime
        from cp_user as t1
        inner join line_note_list as t2 on t1.user_id = t2.user_id and t2.line_human_note_cnt > 0

        union all
        select t1.*
            ,'cr_used_time_desc' ranktype
            ,rank() over(partition by t1.org_id order by t2.duration_today  desc) rank
            ,null value_string
            ,t2.duration_today value_int
            ,null value_float
            ,null value_datetime
        from cp_user as t1
        inner join used_time_list as t2 on t1.user_id = t2.user_id and t2.duration_today > 0
    ),
    dp_rank as (
        select t1.*
            ,'notes_create_desc' ranktype
            ,rank() over(partition by t1.org_id order by t2.line_human_note_cnt  desc) rank
            ,null value_string
            ,t2.line_human_note_cnt value_int
            ,null value_float
            ,null value_datetime
        from dp_user as t1
        inner join line_note_list as t2 on t1.user_id = t2.user_id and t2.line_human_note_cnt > 0

        union all
        select t1.*
            ,'cr_used_time_desc' ranktype
            ,rank() over(partition by t1.org_id order by t2.duration_today  desc) rank
            ,null value_string
            ,t2.duration_today value_int
            ,null value_float
            ,null value_datetime
        from dp_user as t1
        inner join used_time_list as t2 on t1.user_id = t2.user_id and t2.duration_today > 0
    ),
    ct_rank as (
        select t1.*
            ,'notes_create_desc' ranktype
            ,rank() over(partition by t1.org_id order by t2.line_human_note_cnt  desc) rank
            ,null value_string
            ,t2.line_human_note_cnt value_int
            ,null value_float
            ,null value_datetime
        from ct_user as t1
        inner join line_note_list as t2 on t1.user_id = t2.user_id and t2.line_human_note_cnt > 0

        union all
        select t1.*
            ,'cr_used_time_desc' ranktype
            ,rank() over(partition by t1.org_id order by t2.duration_today  desc) rank
            ,null value_string
            ,t2.duration_today value_int
            ,null value_float
            ,null value_datetime
        from ct_user as t1
        inner join used_time_list as t2 on t1.user_id = t2.user_id and t2.duration_today > 0
    ),
    gp_rank as (
        select t1.*
            ,'notes_create_desc' ranktype
            ,rank() over(partition by t1.org_id order by t2.line_human_note_cnt  desc) rank
            ,null value_string
            ,t2.line_human_note_cnt value_int
            ,null value_float
            ,null value_datetime
        from gp_user as t1
        inner join line_note_list as t2 on t1.user_id = t2.user_id and t2.line_human_note_cnt > 0

        union all
        select t1.*
            ,'cr_used_time_desc' ranktype
            ,rank() over(partition by t1.org_id order by t2.duration_today  desc) rank
            ,null value_string
            ,t2.duration_today value_int
            ,null value_float
            ,null value_datetime
        from gp_user as t1
        inner join used_time_list as t2 on t1.user_id = t2.user_id and t2.duration_today > 0
    )
    INSERT overwrite TABLE {db}.{tbl} partition(YEAR='{year}',MONTH='{month}',DAY='{day}')
    SELECT /*+ REPARTITION(64) */
        *
    from (
        select '{year}-{month}' month_name
            ,'{month_start} 00:00:00' month_start
            ,'{month_end} 00:00:00'   month_end
            ,t1.org_id
            ,t1.org_type
            ,t2.ranktype
            ,t2.rank
            ,t3.user_id
            ,t3.user_name
            ,t3.value_string
            ,t3.value_int
            ,t3.value_float
            ,t3.value_datetime
            ,CAST('2026-05-13 00:00:00' AS STRING) AS created_at
        from cp as t1
        cross join rank_space as t2 on 1 = 1
        left join cp_rank as t3 on t1.org_id = t3.org_id and t2.rank = t3.rank and t2.ranktype = t3.ranktype

        union all
        select '{year}-{month}' month_name
            ,'{month_start} 00:00:00' month_start
            ,'{month_end} 00:00:00'   month_end
            ,t1.org_id
            ,t1.org_type
            ,t2.ranktype
            ,t2.rank
            ,t3.user_id
            ,t3.user_name
            ,t3.value_string
            ,t3.value_int
            ,t3.value_float
            ,t3.value_datetime
            ,CAST('2026-05-13 00:00:00' AS STRING) AS created_at
        from dp as t1
        cross join rank_space as t2 on 1 = 1
        left join dp_rank as t3 on t1.org_id = t3.org_id and t2.rank = t3.rank and t2.ranktype = t3.ranktype

        union all
        select '{year}-{month}' month_name
            ,'{month_start} 00:00:00' month_start
            ,'{month_end} 00:00:00'   month_end
            ,t1.org_id
            ,t1.org_type
            ,t2.ranktype
            ,t2.rank
            ,t3.user_id
            ,t3.user_name
            ,t3.value_string
            ,t3.value_int
            ,t3.value_float
            ,t3.value_datetime
            ,CAST('2026-05-13 00:00:00' AS STRING) AS created_at
        from ct as t1
        cross join rank_space as t2 on 1 = 1
        left join ct_rank as t3 on t1.org_id = t3.org_id and t2.rank = t3.rank and t2.ranktype = t3.ranktype

        union all
            select '{year}-{month}' month_name
            ,'{month_start} 00:00:00' month_start
            ,'{month_end} 00:00:00'   month_end
            ,t1.org_id
            ,t1.org_type
            ,t2.ranktype
            ,t2.rank
            ,t3.user_id
            ,t3.user_name
            ,t3.value_string
            ,t3.value_int
            ,t3.value_float
            ,t3.value_datetime
            ,CAST('2026-05-13 00:00:00' AS STRING) AS created_at
        from gp as t1
        cross join rank_space as t2 on 1 = 1
        left join gp_rank as t3 on t1.org_id = t3.org_id and t2.rank = t3.rank and t2.ranktype = t3.ranktype
    )
""".format(
    db=db, tbl=tbl,
    year=year, month=month, day=day, ds=ds,
    month_start=month_start, month_end=month_end,
)

spark.sql(query_sql)

print("pyspark_016 done")

spark.stop()


Anomaly Detection Analysis

Prompt:
## Prompt
I need you to generate a PySpark script that implements anomaly detection analysis based on business data.

**Business Background and Objective**: In the advertising delivery business, operations needs to monitor changes in key metrics (such as `real_cost`) across different industry (`industry_for_business`) dimensions. By comparing and analyzing metric data between the pre-period and post-period, the metrics are summarized by industry dimension, labeling whether the data belongs to the pre-analysis period (`is_post=0`) or the post-analysis period (`is_post=1`). The anomaly detection analysis result table is produced for downstream analysis modules to use.

**Input Table**:
- `internal_platform_db.f_account_manage_analysis_d_pyspark_208`

**Data Range and Filter Conditions**:
- Run date `ds` is fixed as `'20260513'`
- Pre-period: `process_time BETWEEN '20260510' AND '20260512'`
- Post-period: `process_time BETWEEN '20260513' AND '20260513'`
- Filter `partition_time = '20260513'`

**Processing Logic**:
1. Read data from the input table where `partition_time='20260513'`
2. Divide into pre-period (pre, `is_post=0`) and post-period (post, `is_post=1`) by `process_time`
3. Aggregate by the `industry_for_business` dimension, computing `SUM(real_cost)` as the metric value
4. Union the pre and post results and write to the target table

**Output Requirements**:
- Target table: `internal_platform_db.abnormal_analysis_result_pyspark_208`
- Write strategy: `INSERT OVERWRITE TABLE`
- Output columns: `is_post` (INT, 0=pre-period, 1=post-period), `industry_for_business` (STRING, industry), `real_cost` (DOUBLE, sum of `real_cost`)

#!/usr/bin/env python3
"""pyspark_017 ground truth: anomaly detection analysis (simplified)"""
from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName('dataclaw_eval_gt_pyspark_017') \
    .enableHiveSupport() \
    .config('spark.sql.warehouse.dir', '/tmp/hive_warehouse') \
    .getOrCreate()

# Hardcoded task configuration (originally from MySQL t_abnormal_detection_and_analysis_task_conf)
# task_type=8 means use f_account_manage_analysis_d as source table
task_conf = {
    'id': 1,
    'pre_start_date': '20260510',
    'pre_end_date': '20260512',
    'post_start_date': '20260513',
    'post_end_date': '20260513',
    'metrics': 'real_cost',
    'dimensions': 'industry_for_business',
    'conditions': '[]',
    'partition_time': '20260513',
    'task_type': 8
}

partition = task_conf['partition_time']
pre_start = task_conf['pre_start_date']
pre_end = task_conf['pre_end_date']
post_start = task_conf['post_start_date']
post_end = task_conf['post_end_date']
metrics = task_conf['metrics']
dimension_col = task_conf['dimensions']

# Create output table DDL
spark.sql('''
DROP TABLE IF EXISTS internal_platform_db.abnormal_analysis_result_pyspark_208
''')
spark.sql('''
CREATE TABLE IF NOT EXISTS internal_platform_db.abnormal_analysis_result_pyspark_208 (
    is_post INT COMMENT '0=pre period, 1=post period',
    industry_for_business STRING COMMENT '行业',
    real_cost DOUBLE COMMENT 'real_cost日均'
)
STORED AS ORC
''')

# Read data from input table filtered by partition_time
input_df = spark.sql(f'''
SELECT
    process_time,
    industry_for_business,
    real_cost
FROM internal_platform_db.f_account_manage_analysis_d_pyspark_208
WHERE partition_time = '{partition}'
    AND process_time BETWEEN '{pre_start}' AND '{post_end}'
''')

# Compute pre-period metrics (simple aggregation: sum real_cost per industry_for_business)
pre_df = spark.sql(f'''
SELECT
    0 AS is_post,
    industry_for_business,
    SUM(real_cost) AS real_cost
FROM internal_platform_db.f_account_manage_analysis_d_pyspark_208
WHERE partition_time = '{partition}'
    AND process_time BETWEEN '{pre_start}' AND '{pre_end}'
GROUP BY industry_for_business
''')

# Compute post-period metrics
post_df = spark.sql(f'''
SELECT
    1 AS is_post,
    industry_for_business,
    SUM(real_cost) AS real_cost
FROM internal_platform_db.f_account_manage_analysis_d_pyspark_208
WHERE partition_time = '{partition}'
    AND process_time BETWEEN '{post_start}' AND '{post_end}'
GROUP BY industry_for_business
''')

# Union pre and post results
result_df = pre_df.union(post_df)
result_df.show()

# Write result to output table
result_df.write.format("orc").mode('overwrite').saveAsTable(
    'internal_platform_db.abnormal_analysis_result_pyspark_208'
)

print('Ground truth computation complete')
spark.stop()


SOA Dependency Analysis

Prompt:
## Prompt
I need you to generate a PySpark script that implements strong dependency relationship analysis for high-impact modules of use cases, based on the Payment Platform W SOA architecture (SOA Dependency Analysis).

**Business Background and Objective**: In the Payment Platform W SOA architecture, it is necessary to analyze the call relationships of high-quality network use cases (`qualitynetwork_case=1` or `importance=1`). Through upstream and downstream traversal of strong dependency (`dependent_intensity="strong"`) call chains, identify modules directly and indirectly affected by use cases, record impact chain evidence, and output a strong dependency module relationship table for high-impact use cases, for use in premium network evaluation and architecture governance.

**Input Tables**:
- `internal_platform_db.t_dm_callrelation_extended_info_hour_pyspark_209`
- `internal_platform_db.t_dm_usecase_and_asset_callrelation_hour_pyspark_209`
- `internal_platform_db.t_dwd_module_info_hour_pyspark_209`
- `internal_platform_db.t_app_callrelation_for_quality_analyse_hour_pyspark_209`

**Data Range and Filter Conditions**:
- Run date `ds` is fixed as `'p_2026051300'` (hourly partition)
- From `t_dm_usecase_and_asset_callrelation_hour`, filter use cases where `qualitynetwork_case=1` or `importance=1`
- Exclude modules not entering the premium network where `no_entry_premium_network_flag=1`
- From `t_dm_callrelation_extended_info_hour`, filter strong dependency call relationships where `is_involved_to_high_usecase=1` and `dependent_intensity="strong"`
- Filter out invalid call relationships `ossid:0` and `0:0`

**Processing Logic**:
1. From the use case and call relationship table, obtain directly associated modules (mapping of `enter_module_name` to `case_id`), mark `is_dir=1`, with evidence "direct association"
2. Build `case_id_relation_dict`: using `"callee_module_name***callee_interface_name"` as key, and the associated `case_id` set as value
3. Build `relation_dict` from the call relationship table: using `"callee_module_name***callee_interface_name"` as key, and the `"called_module_name***called_interface_name"` set as value
4. Use a DFS algorithm to traverse downstream call chains, add traversed modules to the results, mark `is_dir=0`, with evidence as the downstream analysis chain
5. For use case entries where `analyse_type="cgi"`, build a reverse call relationship dictionary `reverse_relation_dict`
6. Use a DFS algorithm to traverse upstream call chains, add traversed modules to the results, mark `is_dir=0`, with evidence as the upstream analysis chain
7. Merge the results of direct association, downstream analysis, and upstream analysis, and write to the target table

**Output Requirements**:
- Target table: `internal_platform_db.t_dwd_architecture_module_high_case_relation_strong_hour_pyspark_209`
- Write strategy: `INSERT OVERWRITE TABLE`
- Output columns: `databus_imp_date` (STRING, date), `case_id` (STRING, use case ID), `module_name` (STRING, module name), `is_dir` (BIGINT, 1=direct association, 0=indirect association), `evidence` (STRING, impact evidence chain)

#!/usr/bin/env python3
"""pyspark_018 ground truth: SOA dependency analysis"""
import json
from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName('dataclaw_eval_gt_pyspark_018') \
    .enableHiveSupport() \
    .config('spark.sql.warehouse.dir', '/tmp/hive_warehouse') \
    .getOrCreate()

ds = 'p_2026051300'
today = '2026051300'

# Output table DDL
spark.sql('''
DROP TABLE IF EXISTS internal_platform_db.t_dwd_architecture_module_high_case_relation_strong_hour_pyspark_209
''')
spark.sql('''
CREATE TABLE IF NOT EXISTS internal_platform_db.t_dwd_architecture_module_high_case_relation_strong_hour_pyspark_209 (
    databus_imp_date STRING,
    case_id STRING,
    module_name STRING,
    is_dir BIGINT,
    evidence STRING
)
STORED AS ORC
''')

# Create temp views from input tables
spark.sql(f'''
CREATE OR REPLACE TEMP VIEW t_dm_callrelation_extended_info_hour AS
SELECT * FROM internal_platform_db.t_dm_callrelation_extended_info_hour_pyspark_209
WHERE ds = '{ds}'
''')

spark.sql(f'''
CREATE OR REPLACE TEMP VIEW t_dm_usecase_and_asset_callrelation_hour AS
SELECT * FROM internal_platform_db.t_dm_usecase_and_asset_callrelation_hour_pyspark_209
WHERE ds = '{ds}'
''')

spark.sql(f'''
CREATE OR REPLACE TEMP VIEW t_dwd_module_info_hour AS
SELECT * FROM internal_platform_db.t_dwd_module_info_hour_pyspark_209
WHERE ds = '{ds}'
''')

spark.sql(f'''
CREATE OR REPLACE TEMP VIEW t_app_callrelation_for_quality_analyse_hour AS
SELECT * FROM internal_platform_db.t_app_callrelation_for_quality_analyse_hour_pyspark_209
WHERE ds = '{ds}'
''')

print("step 1")
print("获取直接相关的模块")

T_SCHEMA = ('databus_imp_date', 'case_id', 'module_name', 'is_dir', 'evidence')

# Get direct relations - modules directly associated with cases
dir_sql = """
select distinct
    relation.enter_module_name,
    relation.case_id
from t_dm_usecase_and_asset_callrelation_hour relation
left join (
    select module_name
    from t_dwd_module_info_hour
    where no_entry_premium_network_flag = 1
) modules_info on relation.enter_module_name = modules_info.module_name
where (qualitynetwork_case = 1 or importance = 1)
    and (enter_module_name is not null and enter_module_name <> '')
    and modules_info.module_name is null
"""

service_datas = spark.sql(dir_sql).toPandas()

module_evidence = dict()

class Evidence:
    def __init__(self, module_name):
        self.module_name = module_name

    def set_case(self, case_id):
        self.case_id = case_id
        return self

    def set_evidence(self, evidence):
        self.evidence = evidence
        return self

    def to_string(self, comment):
        return "[用例ID:{}--{}--证据链路:{}]".format(self.case_id, comment, self.evidence)

result_list = []
for data in service_datas.values:
    enter_module_name = data[0]
    case_id = data[1]
    is_dir = 1

    if enter_module_name not in module_evidence:
        module_evidence[enter_module_name] = Evidence(enter_module_name).set_case(case_id).set_evidence("直接关联")
    result_list.append((today, case_id, enter_module_name, is_dir, module_evidence[enter_module_name].to_string("直接关联")))

print("直接影响的个数")
origin_length = len(result_list)
print(origin_length)

print("step 2")
print("获取用例和调用关系入口模块-接口映射")

case_id_relation_dict_sql = """
select
    enter_module_name, case_id, callee_module_name, callee_interface_name
from t_dm_usecase_and_asset_callrelation_hour
where (qualitynetwork_case = 1 or importance = 1)
    and (enter_module_name is not null and enter_module_name <> '')
    and (callee_module_name is not null and callee_module_name <> '')
    and (callee_interface_name is not null and callee_interface_name <> '')
    and (called_module_name is not null and called_module_name <> '')
    and (called_interface_name is not null and called_interface_name <> '')
    and enter_module_name = callee_module_name
"""

case_id_relation_dict = dict()
service_datas = spark.sql(case_id_relation_dict_sql).toPandas()
for data in service_datas.values:
    enter_module_name = data[0]
    case_id = data[1]
    callee_module_name = data[2]
    callee_interface_name = data[3]

    relation = "{}***{}".format(callee_module_name, callee_interface_name)
    if relation in case_id_relation_dict:
        case_id_relation_dict[relation].add(case_id)
    else:
        case_id_relation_dict[relation] = set()
        case_id_relation_dict[relation].add(case_id)

print("step 3")
print("获取剔除了不进入精品网模块的强依赖调用关系")

relation_dict_sql = """
select
    relation.callee_module_name,
    relation.callee_interface_name,
    relation.called_module_name,
    relation.called_interface_name
from
    (select
        callee_module_name, callee_interface_name, called_module_name, called_interface_name
    from t_dm_callrelation_extended_info_hour
    where is_involved_to_high_usecase = 1
        and dependent_intensity = 'strong'
    ) relation
left join (
    select module_name
    from t_dwd_module_info_hour
    where no_entry_premium_network_flag = 1
) modules_callee on relation.callee_module_name = modules_callee.module_name
left join (
    select module_name
    from t_dwd_module_info_hour
    where no_entry_premium_network_flag = 1
) modules_called on relation.called_module_name = modules_called.module_name
where modules_callee.module_name is null
    and modules_called.module_name is null
"""

relation_dict = dict()
service_datas = spark.sql(relation_dict_sql).toPandas()
for data in service_datas.values:
    callee_module_name = data[0]
    callee_interface_name = data[1]
    called_module_name = data[2]
    called_interface_name = data[3]

    relation_callee = "{}***{}".format(callee_module_name, callee_interface_name)
    relation_called = "{}***{}".format(called_module_name, called_interface_name)
    if relation_callee in relation_dict:
        relation_dict[relation_callee].add(relation_called)
    else:
        relation_dict[relation_callee] = set()
        relation_dict[relation_callee].add(relation_called)

def dfs(relation_dict, called_info, record_list, chain_list, space, case_id_now):
    if called_info not in relation_dict:
        return
    called_infos = relation_dict[called_info]
    for called_info_item in called_infos:
        if called_info_item in record_list:
            continue
        record_list.add(called_info_item)

        chain_list.append(called_info_item)

        module_name_now = called_info_item.split("***")[0]
        if module_name_now != "" and module_name_now not in module_evidence:
            module_evidence[module_name_now] = Evidence(module_name_now).set_evidence(space.join(chain_list)).set_case(case_id_now)

        dfs(relation_dict, called_info_item, record_list, chain_list, space, case_id_now)

        chain_list.pop()

def dfs_fist(relation_dict, key, reverse, case_id_now):
    space = "<-" if reverse else "->"
    record_list = set()
    chain_list = list()

    if key not in relation_dict:
        return None

    chain_list.append(key)

    module_name_now = key.split("***")[0]
    if module_name_now != "" and module_name_now not in module_evidence:
        module_evidence[module_name_now] = Evidence(module_name_now).set_evidence(space.join(chain_list)).set_case(case_id_now)

    called_infos = relation_dict[key]
    for called_info in called_infos:
        if called_info in record_list:
            continue
        record_list.add(called_info)

        module_name_now = called_info.split("***")[0]
        chain_list.append(called_info)
        if module_name_now != "" and module_name_now not in module_evidence:
            module_evidence[module_name_now] = Evidence(module_name_now).set_evidence(space.join(chain_list)).set_case(case_id_now)

        dfs(relation_dict, called_info, record_list, chain_list, space, case_id_now)
        chain_list.pop()
    return record_list

# Downstream analysis
print("step 4")
print("开始下游分析")

is_dir = 0
module_evidence = dict()
for key in case_id_relation_dict:
    case_id_now = list(case_id_relation_dict[key])[0]
    modules_list = dfs_fist(relation_dict, key, 0, case_id_now)
    if modules_list is None:
        continue
    cases = case_id_relation_dict[key]
    for case_id in cases:
        for module in modules_list:
            module_name = module.split("***")[0]
            if module_name == "":
                continue
            result_list.append((today, case_id, module_name, is_dir, module_evidence[module_name].to_string("下游分析")))

print("间接影响的个数")
print(len(result_list) - origin_length)

# Upstream analysis (CGI)
print("step 5")
print("获取cgi用例和调用关系入口模块-接口映射")

case_id_relation_dict_sql = """
select
    enter_module_name, case_id, callee_module_name, callee_interface_name
from t_dm_usecase_and_asset_callrelation_hour
where (qualitynetwork_case = 1 or importance = 1)
    and (enter_module_name is not null and enter_module_name <> '')
    and (callee_module_name is not null and callee_module_name <> '')
    and (callee_interface_name is not null and callee_interface_name <> '')
    and (called_module_name is not null and called_module_name <> '')
    and (called_interface_name is not null and called_interface_name <> '')
    and enter_module_name = callee_module_name
    and analyse_type = 'cgi'
"""

case_id_relation_dict = dict()
service_datas = spark.sql(case_id_relation_dict_sql).toPandas()
for data in service_datas.values:
    enter_module_name = data[0]
    case_id = data[1]
    callee_module_name = data[2]
    callee_interface_name = data[3]

    relation = "{}***{}".format(callee_module_name, callee_interface_name)
    if relation in case_id_relation_dict:
        case_id_relation_dict[relation].add(case_id)
    else:
        case_id_relation_dict[relation] = set()
        case_id_relation_dict[relation].add(case_id)

# Build reverse relation dict
all_relation_dict_sql = """
select distinct
    callee_module_name,
    callee_interface_name,
    called_module_name,
    called_interface_name
from t_app_callrelation_for_quality_analyse_hour
where callee_module_name <> 'ossid:0'
    and called_module_name <> 'ossid:0'
    and callee_module_name <> '0:0'
    and called_module_name <> '0:0'
"""

reverse_relation_dict = dict()
service_datas = spark.sql(all_relation_dict_sql).toPandas()
for data in service_datas.values:
    callee_module_name = data[0]
    callee_interface_name = data[1]
    called_module_name = data[2]
    called_interface_name = data[3]

    relation_callee = "{}***{}".format(callee_module_name, callee_interface_name)
    relation_called = "{}***{}".format(called_module_name, called_interface_name)
    if relation_called in reverse_relation_dict:
        reverse_relation_dict[relation_called].add(relation_callee)
    else:
        reverse_relation_dict[relation_called] = set()
        reverse_relation_dict[relation_called].add(relation_callee)

module_evidence = dict()
is_dir = 0
reverse_result_list = list()
for key in case_id_relation_dict:
    case_id_now = list(case_id_relation_dict[key])[0]
    modules_list = dfs_fist(reverse_relation_dict, key, 1, case_id_now)
    if modules_list is None:
        continue
    cases = case_id_relation_dict[key]
    for case_id in cases:
        for module in modules_list:
            module_name = module.split("***")[0]
            if module_name == "":
                continue
            reverse_result_list.append((today, case_id, module_name, is_dir, module_evidence[module_name].to_string("上游分析")))

print("反向影响的个数")
print(len(reverse_result_list))
result_list = result_list + reverse_result_list

# Write output
t_df = spark.createDataFrame(result_list, T_SCHEMA)
t_df.write.mode('overwrite').format('orc').saveAsTable(
    'internal_platform_db.t_dwd_architecture_module_high_case_relation_strong_hour_pyspark_209'
)

print('Ground truth computation complete')
spark.stop()


Campaign User Label Computation

Prompt:
## Prompt
I need you to generate a PySpark script that implements campaign user targeting label computation based on brand campaign activities (Campaign User Label Computation).

**Business Background and Objective**: In the Payment Platform W brand management platform, the targeting labels for campaign configurations are stored in JSON format (containing `label_groups` and `exclude_label_groups`). These need to be parsed into a structured label detail table, and transaction frequency labels (`label_type=7`) need to be automatically backfilled, for downstream user filtering and campaign delivery.

**Input Tables**:
- `internal_platform_db.t_dwd_brand_mgmt_activity_all_hour_pyspark_206`
- `internal_platform_db.t_dwd_gift_mgmt_targeted_act_hour_pyspark_206`

**Data Range and Filter Conditions**:
- Run date `ds` is fixed as `'p_2026051300'` (hourly partition)
- From `t_dwd_brand_mgmt_activity_all_hour`, filter campaigns where `factid>10 AND factstatus IN (2,6,7)`
- Union with the `factid` and `factivitylabel` from `t_dwd_gift_mgmt_targeted_act_hour`

**Processing Logic**:
1. Obtain the campaign `factivitylabel` JSON data by unioning the brand management campaign table and the pay-gift table
2. Use `flatMap` to parse the `label_groups.target_labels` and `exclude_label_groups.target_labels` from each row's JSON into independent rows
3. Extract `label_type`, `label_value`, `effective_range_type`, `effective_range_value`, and `is_exclude` fields from `raw_json`
4. Filter out invalid labels (records where `label_type`, `label_value`, or `effective_range_type` is empty or 0), and deduplicate
5. For all brands (`fbrandid`) with valid campaigns, backfill the four values (1, 2, 3, 4) for `label_type=7` (transaction frequency), with `effective_range_type=2` and `is_exclude=0`
6. Use a left_anti join to exclude already existing records, avoiding duplication

**Output Requirements**:
- Target table: `internal_platform_db.brand_mgmt_act_label_pyspark_206`
- Write strategy: `INSERT OVERWRITE TABLE`
- Output columns: `ds` (STRING, date), `label_type` (STRING, label type), `label_value` (STRING, label value), `effective_range_type` (STRING, effective range type), `effective_range_value` (STRING, effective range value), `raw_json` (STRING, original JSON), `is_exclude` (INT, 0=include, 1=exclude)

#!/usr/bin/env python3
"""pyspark_019 ground truth: campaign user label computation"""
import json
from pyspark.sql import SparkSession
from pyspark.sql.functions import lit, get_json_object, when, to_json, struct

spark = SparkSession.builder \
    .appName('dataclaw_eval_gt_pyspark_019') \
    .enableHiveSupport() \
    .config('spark.sql.warehouse.dir', '/tmp/hive_warehouse') \
    .getOrCreate()

hour = '2026051300'
HOUR_PARTITION = 'p_' + str(hour)

# Output table DDL
spark.sql('''
DROP TABLE IF EXISTS internal_platform_db.brand_mgmt_act_label_pyspark_206
''')
spark.sql('''
CREATE TABLE IF NOT EXISTS internal_platform_db.brand_mgmt_act_label_pyspark_206 (
    ds STRING,
    label_type STRING,
    label_value STRING,
    effective_range_type STRING,
    effective_range_value STRING,
    raw_json STRING,
    is_exclude INT
)
STORED AS ORC
''')

# Step 1: Read activity data from brand manager + paygift, filter and union
activity_label = (
    spark.sql(f'''
        SELECT factid, factivitylabel
        FROM internal_platform_db.t_dwd_brand_mgmt_activity_all_hour_pyspark_206
        WHERE ds = '{HOUR_PARTITION}'
            AND factid > 10
            AND factstatus IN (2, 6, 7)
    ''')
).union(
    spark.sql(f'''
        SELECT factid, factivitylabel
        FROM internal_platform_db.t_dwd_gift_mgmt_targeted_act_hour_pyspark_206
        WHERE ds = '{HOUR_PARTITION}'
    ''')
)

activity_label.show(truncate=False)

def parse_labels(row):
    result = []
    if row["factivitylabel"] is None:
        return result
    labels = json.loads(row["factivitylabel"])
    if labels is None:
        return result
    if "label_groups" in labels.keys():
        for group in labels["label_groups"]:
            if "target_labels" in group.keys():
                for l in group["target_labels"]:
                    json_str = json.dumps(l)
                    result.append((json_str,))
    if "exclude_label_groups" in labels.keys():
        for group in labels["exclude_label_groups"]:
            if "target_labels" in group.keys():
                for l in group["target_labels"]:
                    json_str = json.dumps(l)
                    result.append((json_str,))
    return result

# Step 2: flatMap to parse JSON labels into individual rows
single_labels = activity_label.rdd.flatMap(lambda row: parse_labels(row))
print(1)
if single_labels.isEmpty():
    single_labels_df = spark.createDataFrame([], "raw_json: string")
else:
    single_labels_df = spark.createDataFrame(single_labels, ["raw_json"])

single_labels_df = (
    single_labels_df.withColumn("ds", lit(hour))
    .withColumn("label_type", get_json_object(single_labels_df.raw_json, "$.label_type"))
    .withColumn("label_value", get_json_object(single_labels_df.raw_json, "$.label_value"))
    .withColumn("effective_range_type", get_json_object(single_labels_df.raw_json, "$.effective_range_type"))
    .withColumn("effective_range_value", get_json_object(single_labels_df.raw_json, "$.effective_range_value"))
    .withColumn("is_exclude", when(get_json_object(single_labels_df.raw_json, "$.is_exclude") == "true", 1).otherwise(0))
)

single_labels_df.show(truncate=False)

# Step 3: Dedup and filter valid labels
filtered_labels = (
    single_labels_df.select(
        "ds", "label_type", "label_value",
        "effective_range_type", "effective_range_value", "raw_json", "is_exclude",
    )
    .filter("nvl(label_type, 0) != 0 AND nvl(label_value, '0') != '0' AND nvl(effective_range_type, 0) != 0")
    .distinct()
)

# Step 4: Supplement label_type=7 (transaction frequency labels)
all_act_brands = (
    spark.sql(f'''
        SELECT DISTINCT fbrandid
        FROM internal_platform_db.t_dwd_brand_mgmt_activity_all_hour_pyspark_206
        WHERE ds = '{HOUR_PARTITION}'
            AND factid > 10
            AND factstatus IN (2, 6, 7)
    ''')
    .withColumnRenamed("fbrandid", "effective_range_value")
)

freq_label_values = spark.createDataFrame(
    [("1",), ("2",), ("3",), ("4",)], ["label_value"]
)

extra_freq_labels = (
    all_act_brands.crossJoin(freq_label_values)
    .withColumn("ds", lit(hour))
    .withColumn("label_type", lit("7"))
    .withColumn("effective_range_type", lit("2"))
    .withColumn("is_exclude", lit(0))
)

extra_freq_labels = extra_freq_labels.withColumn(
    "raw_json",
    to_json(
        struct(
            extra_freq_labels.label_type,
            extra_freq_labels.label_value,
            extra_freq_labels.effective_range_type,
            extra_freq_labels.effective_range_value,
        )
    ),
).select(
    "ds", "label_type", "label_value",
    "effective_range_type", "effective_range_value", "raw_json", "is_exclude",
)

# Use left_anti join to keep only records not already in filtered_labels
new_freq_labels = extra_freq_labels.join(
    filtered_labels,
    on=["label_type", "label_value", "effective_range_type", "effective_range_value", "is_exclude"],
    how="left_anti",
)

result_labels = filtered_labels.unionByName(new_freq_labels)

# Write output
result_labels.write.mode('overwrite').format('orc').saveAsTable(
    'internal_platform_db.brand_mgmt_act_label_pyspark_206'
)

print('Ground truth computation complete')
spark.stop()


小程序H5监控

Prompt:
## Prompt
我需要你生成一段 PySpark 代码，实现小程序打开第三方 H5 页面的大盘监控指标计算（Mini-Program H5 Monitoring）。

**业务背景与目标**：小程序开放平台需要监控小程序内打开第三方 H5 页面的整体情况，包括 PV、UV、域名数、链接数等核心指标，按整体大盘和 Top100 小程序两个维度分别统计，同时按域名归属关系（同主体/关联主体/第三方）进行细分，产出结构化监控指标表。

**输入表**：
- `internal_platform_db.ods_daily_miniapp_h5_expand_raw_pyspark_212`
- `internal_platform_db.ods_daily_miniapp_h5_top100_pv_raw_pyspark_212`

**数据范围与过滤条件**：
- 运行日期 ds 固定为 '20260513'（日分区）
- ods_daily_miniapp_h5_expand_raw 提供全量 H5 访问明细
- ods_daily_miniapp_h5_top100_pv_raw 提供 Top100 PV 小程序的 H5 访问明细
- url 列重命名为 url_wop（无参数 URL）

**处理逻辑**：
1. 从 expand_raw 表计算整体大盘指标：total PV=总行数, total UV=去重 useruin, total domain=去重 domain, total url_wop=去重 url_wop，stat_type='total'
2. 从 top100_raw 表按 appid 分组计算：PV=count, UV=去重 useruin, domain=去重 domain, url_wop=去重 url_wop，stat_type='total_top'
3. 从 expand_raw 表按 related（域名归属关系）分组计算：PV=count, UV=去重 useruin, domain=去重 domain, url_wop=去重 url_wop，stat_type='contactor_{related}'
4. 从 top100_raw 表按 appid+related 分组计算同样指标，stat_type='contactor_{related}_top'
5. 使用 pandas merge 将各维度指标合并为统一格式
6. 添加 ds 列（BIGINT 类型），生成最终结果

**输出要求**：
- 目标表：`internal_platform_db.ods_daily_miniapp_webview_total_pyspark_212`
- 写入策略：INSERT OVERWRITE TABLE
- 输出列：ds（BIGINT，日期）, appid（STRING，小程序appid，大盘级为空字符串）, pv（BIGINT，页面浏览量）, uv（BIGINT，独立访客数）, domain（BIGINT，域名数）, url_wop（BIGINT，去参链接数）, stat_type（STRING，统计类型：total/total_top/contactor_{related}/contactor_{related}_top）

#!/usr/bin/env python3
"""pyspark_020 ground truth: mini-program H5 monitoring metrics"""
import pandas as pd
from pyspark.sql import SparkSession
import pyspark.sql.functions as F

spark = SparkSession.builder \
    .appName('dataclaw_eval_gt_pyspark_020') \
    .enableHiveSupport() \
    .config('spark.sql.warehouse.dir', '/tmp/hive_warehouse') \
    .getOrCreate()

ds = '20260513'

# Output table DDL
spark.sql('''
DROP TABLE IF EXISTS internal_platform_db.ods_daily_miniapp_webview_total_pyspark_212
''')
spark.sql('''
CREATE TABLE IF NOT EXISTS internal_platform_db.ods_daily_miniapp_webview_total_pyspark_212 (
    ds BIGINT COMMENT 'date',
    appid STRING COMMENT 'appid',
    pv BIGINT COMMENT 'page views',
    uv BIGINT COMMENT 'unique visitors',
    domain BIGINT COMMENT 'domain count',
    url_wop BIGINT COMMENT 'url without param count',
    stat_type STRING COMMENT 'statistic type'
)
STORED AS ORC
''')

cols = ['appid', 'pv', 'uv', 'domain', 'url_wop', 'stat_type']

# Read input data
web_stats_raw = spark.sql(f'''
    SELECT useruin, appid, url_, domain, url, contactor_domain, related
    FROM internal_platform_db.ods_daily_miniapp_h5_expand_raw_pyspark_212
    WHERE ds = '{ds}'
''')
web_stats = web_stats_raw.withColumnRenamed('url', 'url_wop')
web_stats.persist()
web_stats_cnt = web_stats.count()
print(f"Fetch total web_stats info: {web_stats_cnt}")

web_stats_topuser_id_raw = spark.sql(f'''
    SELECT useruin, appid, url_, domain, url, contactor_domain, related
    FROM internal_platform_db.ods_daily_miniapp_h5_top100_pv_raw_pyspark_212
    WHERE ds = '{ds}'
''')
web_stats_topuin = web_stats_topuser_id_raw.withColumnRenamed('url', 'url_wop')
web_stats_topuin.persist()
web_stats_topuser_id_cnt = web_stats_topuin.count()
print(f"Fetch total web_stats_topuin info: {web_stats_topuser_id_cnt}")

# 1. Overall metrics: total PV/UV/domain/URL counts
cate = 'total'
total_pv = web_stats_cnt
total_uv = web_stats[['useruin']].drop_duplicates().count()
total_domain_cnt = web_stats[['domain']].drop_duplicates().count()
total_url_cnt = web_stats[['url_wop']].drop_duplicates().count()
df_total = pd.DataFrame(data=[('', total_pv, total_uv, total_domain_cnt, total_url_cnt, cate)], columns=cols)

# Per-app metrics for top100
cate = 'total_top'
total_top_pv = web_stats_topuin.groupby('appid').count().withColumnRenamed('count', 'pv').toPandas()
total_top_uv = web_stats_topuin[['appid', 'useruin']].drop_duplicates().groupby('appid').count().withColumnRenamed('count', 'uv').toPandas()
total_top_domain_cnt = web_stats_topuin[['appid', 'domain']].drop_duplicates().groupby('appid').count().withColumnRenamed('count', 'domain').toPandas()
total_top_url_cnt = web_stats_topuin[['appid', 'url_wop']].drop_duplicates().groupby('appid').count().withColumnRenamed('count', 'url_wop').toPandas()
total_top_metrics = total_top_pv.merge(total_top_uv, how='left', on='appid') \
    .merge(total_top_domain_cnt, how='left', on='appid') \
    .merge(total_top_url_cnt, how='left', on='appid')
total_top_metrics['stat_type'] = cate

pdf_total = pd.concat([df_total[cols], total_top_metrics[cols]], ignore_index=True, axis=0)

# 2. Contactor distribution metrics
cate = 'contactor'
contactor_pv_dist = web_stats.groupby('related').count().withColumnRenamed('count', 'pv').toPandas()
contactor_pv_dist['stat_type'] = cate + '_' + contactor_pv_dist.related
contactor_uv_dist = web_stats[['related', 'useruin']].drop_duplicates().groupby('related').count().withColumnRenamed('count', 'uv').toPandas()
contactor_uv_dist['stat_type'] = cate + '_' + contactor_uv_dist.related
contactor_domain_dist = web_stats[['related', 'domain']].drop_duplicates().groupby('related').count().withColumnRenamed('count', 'domain').toPandas()
contactor_domain_dist['stat_type'] = cate + '_' + contactor_domain_dist.related
contactor_url_dist = web_stats[['related', 'url_wop']].drop_duplicates().groupby('related').count().withColumnRenamed('count', 'url_wop').toPandas()
contactor_url_dist['stat_type'] = cate + '_' + contactor_url_dist.related
df_contactor = contactor_pv_dist[['stat_type', 'pv']].merge(contactor_uv_dist[['stat_type', 'uv']]) \
    .merge(contactor_domain_dist[['stat_type', 'domain']]) \
    .merge(contactor_url_dist[['stat_type', 'url_wop']])
df_contactor['appid'] = ''

# Per-app contactor metrics for top100
contactor_top_pv = web_stats_topuin.groupby(['appid', 'related']).count().withColumnRenamed('count', 'pv').toPandas()
contactor_top_uv = web_stats_topuin[['appid', 'related', 'useruin']].drop_duplicates().groupby(['appid', 'related']).count().withColumnRenamed('count', 'uv').toPandas()
contactor_top_domain = web_stats_topuin[['appid', 'related', 'domain']].drop_duplicates().groupby(['appid', 'related']).count().withColumnRenamed('count', 'domain').toPandas()
contactor_top_url = web_stats_topuin[['appid', 'related', 'url_wop']].drop_duplicates().groupby(['appid', 'related']).count().withColumnRenamed('count', 'url_wop').toPandas()
contactor_top_metrics = contactor_top_pv.merge(contactor_top_uv, how='left', on=['appid', 'related']) \
    .merge(contactor_top_domain, how='left', on=['appid', 'related']) \
    .merge(contactor_top_url, how='left', on=['appid', 'related'])
contactor_top_metrics['stat_type'] = cate + '_' + contactor_top_metrics.related + '_top'

pdf_contactor = pd.concat([df_contactor[cols], contactor_top_metrics[cols]], ignore_index=True, axis=0)

# Combine all metrics
pdf_metrics = pd.concat([pdf_total, pdf_contactor], ignore_index=True, axis=0)
sdf_metrics = spark.createDataFrame(pdf_metrics)
res = sdf_metrics.withColumn('ds', F.lit(int(ds))).select('ds', 'appid', 'pv', 'uv', 'domain', 'url_wop', 'stat_type').repartition(1)
res.persist()
res_cnt = res.count()
print(f"Fetch statistic metrics: {res_cnt}")

# Write output
res.write.mode('overwrite').format('orc').saveAsTable(
    'internal_platform_db.ods_daily_miniapp_webview_total_pyspark_212'
)

print('Ground truth computation complete')
spark.stop()