被动客服满意度归因环比对比

Prompt:
## Prompt
我需要你生成一段 PySpark 代码，产出"被动客服会话满意度归因因素"的环比对比表。

**业务背景与目标**：被动客服每天都会按 reason / reason_detail / session_type 维度产出一份满意度归因明细（dwd_ww_kf_session_satify_attrib）。运营希望对每一条归因因素同时做两个维度的对比：(1) 与 14 天前同一 reason+reason_detail+session_type 的指标做差值；(2) 与上月相同类型日期（工作日 vs 工作日、周末 vs 周末）的平均指标做差值。本任务把当天、14 天前、上月这三组数据拉齐后，逐条计算满意度差值、占比差值、贡献率，输出到下游对比表里供运营复盘。

**输入表**：
- `internal_platform_db.caseR1_dwd_ww_kf_session_satify_attrib_v3`

**数据范围与过滤条件（业务说法）**：
- 运行日期 statdate 固定为 '20260513'（沙箱化日期，原任务由调度框架传入）。
- 14 天前日期 = statdate - 14 天 = '20260429'。
- 上月相同类型日期：根据 statdate 的工作日/周末属性，从上一个自然月里取出所有同类型日期组成日期列表（例如 statdate 是周三，则取上月所有工作日）；用该列表 IN 过滤主表得到"上月同类型样本"。
- 三组切片都从同一张明细表过滤得到，分别注册成临时视图 today / 14d / month_avg。

**表关联关系**：
- left join：当天视图 t0 ⋈ 14 天前视图 t1 on (t0.reason=t1.reason AND t0.reason_detail=t1.reason_detail AND t0.session_type=t1.session_type) → df_compare_14d。
- left join：当天视图 t0 ⋈ 上月平均视图 t1 on 同上 → df_compare_month。
- df_compare_14d 与 df_compare_month 通过 union all 拼接成 df_all 一次性写入。

**聚合与计算规则（需体现在 SQL/PySpark 中）**：
1. **上月平均聚合**：对上月切片，先按 session_type 计算月级总量 `all_month_session_cnt = sum(all_session_cnt)` 和加权满意度 `all_month_session_score = sum(all_session_satify_score*all_session_cnt) / sum(all_session_cnt)`；再按 (session_type, reason, reason_detail) 聚合，得到月度 satify_score = sum(session_cnt*satify_score)/sum(session_cnt)、session_rate = sum(session_cnt)/all_month_session_cnt、satify_attrib = (sum(session_cnt*satify_score*session_rate/100)/sum(session_cnt))/all_month_session_score - 1。
2. **派生差值字段**（每条 reason+reason_detail+session_type 行同时计算 14 天对比和上月对比两组）：
   - all_session_satify_diff = t0.all_session_satify_score - t1.all_session_satify_score
   - satify_score_diff = t0.satify_score - t1.satify_score
   - session_rate_diff = t0.session_rate - t1.session_rate
   - satify_attrib_diff = (t0.satify_score*t0.session_rate - t1.satify_score*t1.session_rate) / 100
   - satify_attrib_rate = satify_attrib_diff / all_session_satify_diff（贡献率）
   - self_satify_attrib_rate = (t0.satify_score*t0.session_rate) / (t1.satify_score*t1.session_rate) - 1（自身环比）
   - compare_satify_attrib_diff = t0.satify_attrib - t1.satify_attrib
3. **compare_interval 字段**：14 天对比那批固定写 '14天前'；上月对比那批固定写 '上月'。
4. **输出 imp_date** 全部填当天 statdate '20260513'。

**输出要求**：
- 目标表：`internal_platform_db.caseR1_dwd_ww_kf_session_satify_attrib_compare_v3`
- 写入策略：INSERT OVERWRITE TABLE，使用显式列名清单（22 列）避免列序错位。
- 如果目标表不存在，请先按 Hive 标准建表（ORC 存储），再写入数据。

#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
caseR1: 被动客服满意度归因 - 14天/上月对比 (SparkBase -> SparkSession 改写)
- 沙箱无分区版：原 part_name=last14day / part_name=part_names 改成 WHERE imp_date='...' / WHERE imp_date IN (...)
- 原 self.statdate 硬编码为 '20260513'（沙箱化日期）
- 原 self.save_dw_data 改为 spark.sql INSERT OVERWRITE TABLE
"""

import datetime
from pyspark.sql import SparkSession

spark = (
    SparkSession.builder
    .appName('caseR1_satify_attrib_compare')
    .enableHiveSupport()
    .getOrCreate()
)

# 沙箱化运行日期（fixture 模板硬编码）
statdate = '20260513'

# ---------------------------- 14 天前 ----------------------------
last14day = (datetime.datetime.strptime(statdate, "%Y%m%d") - datetime.timedelta(days=14)).strftime("%Y%m%d")
print(statdate, last14day)

# 当天视图
spark.sql("""
SELECT * FROM internal_platform_db.caseR1_dwd_ww_kf_session_satify_attrib_v3
WHERE imp_date = '{0}'
""".format(statdate)).createOrReplaceTempView("dwd_ww_kf_session_satify_attrib_today")

# 14 天前视图
spark.sql("""
SELECT * FROM internal_platform_db.caseR1_dwd_ww_kf_session_satify_attrib_v3
WHERE imp_date = '{0}'
""".format(last14day)).createOrReplaceTempView("dwd_ww_kf_session_satify_attrib_14d")

sql_14d = """
SELECT
    "{0}" AS imp_date
    ,'14天前' AS compare_interval
    ,t0.reason
    ,t0.reason_detail
    ,CAST(t0.session_cnt AS DOUBLE) AS session_cnt
    ,t0.satify_score
    ,t0.session_rate
    ,t0.all_session_satify_score
    ,t0.satify_attrib
    ,t1.all_session_satify_score AS compare_all_session_satify_score
    ,t1.satify_score AS compare_satify_score
    ,t1.session_rate AS compare_session_rate
    ,t0.all_session_satify_score - t1.all_session_satify_score AS all_session_satify_diff
    ,t0.satify_score - t1.satify_score AS satify_score_diff
    ,t0.session_rate - t1.session_rate AS session_rate_diff
    ,(t0.satify_score*t0.session_rate - t1.satify_score*t1.session_rate)/100 AS satify_attrib_diff
    ,((t0.satify_score*t0.session_rate - t1.satify_score*t1.session_rate)/100)/(t0.all_session_satify_score - t1.all_session_satify_score) AS satify_attrib_rate
    ,(t0.satify_score*t0.session_rate)/(t1.satify_score*t1.session_rate) - 1 AS self_satify_attrib_rate
    ,CAST(t1.session_cnt AS DOUBLE) AS compare_session_cnt
    ,t1.satify_attrib AS compare_satify_attrib
    ,t0.satify_attrib - t1.satify_attrib AS compare_satify_attrib_diff
    ,t0.session_type
FROM dwd_ww_kf_session_satify_attrib_today t0
LEFT JOIN dwd_ww_kf_session_satify_attrib_14d t1
    ON t0.reason = t1.reason AND t0.reason_detail = t1.reason_detail AND t0.session_type = t1.session_type
""".format(statdate)
df_compare_14d = spark.sql(sql_14d).cache()

# ---------------------------- 上月对比 ----------------------------
# 纯 datetime 实现 lastmonth_firstday，避免依赖 dateutil
_today = datetime.datetime.strptime(statdate, "%Y%m%d")
_thismonth_first = _today.replace(day=1)
# 上月第 1 天 = 本月第 1 天 - 1 天 → 取该日期所在月的第 1 天
_lastmonth_anchor = _thismonth_first - datetime.timedelta(days=1)
lastmonth_firstday = _lastmonth_anchor.replace(day=1).strftime("%Y%m%d")
thismonth_firstday = statdate[0:6] + "01"
interval = (datetime.datetime.strptime(thismonth_firstday, "%Y%m%d")
            - datetime.datetime.strptime(lastmonth_firstday, "%Y%m%d"))

weekday_list = []
holiday_list = []
for i in range(interval.days):
    date = (datetime.datetime.strptime(lastmonth_firstday, "%Y%m%d")
            + datetime.timedelta(days=i)).strftime("%Y%m%d")
    week_flag = datetime.datetime.strptime(date, "%Y%m%d").weekday()
    if week_flag in (0, 1, 2, 3, 4):
        weekday_list.append(date)
    else:
        holiday_list.append(date)

today_week = datetime.datetime.strptime(statdate, "%Y%m%d").weekday()
part_names = weekday_list if today_week in (0, 1, 2, 3, 4) else holiday_list

# 上月对应类型日期视图（多分区联合）
in_clause = ",".join(["'%s'" % d for d in part_names])
spark.sql("""
SELECT * FROM internal_platform_db.caseR1_dwd_ww_kf_session_satify_attrib_v3
WHERE imp_date IN ({0})
""".format(in_clause)).createOrReplaceTempView("dwd_ww_kf_session_satify_attrib_lastmonth")

# 上月统计
spark.sql("""
SELECT
    session_type
    ,count(distinct imp_date) AS date_cnt
    ,sum(all_session_cnt) AS all_month_session_cnt
    ,sum(all_session_satify_score*all_session_cnt)/sum(all_session_cnt) AS all_month_session_score
FROM (
    SELECT DISTINCT imp_date,session_type,all_session_satify_score,all_session_cnt
    FROM dwd_ww_kf_session_satify_attrib_lastmonth
)
GROUP BY session_type
""").createOrReplaceTempView("lastmonth_stat")

# 每类因素的上月平均
spark.sql("""
SELECT
    t0.session_type
    ,reason
    ,reason_detail
    ,all_month_session_cnt
    ,all_month_session_score AS all_session_satify_score
    ,sum_session_cnt
    ,sum_session_cnt/date_cnt session_cnt
    ,satify_score
    ,sum_session_cnt*100/all_month_session_cnt AS session_rate
    ,(satify_score - all_month_session_score)*(sum_session_cnt/all_month_session_cnt) AS satify_attrib
FROM (
    SELECT
        session_type
        ,reason
        ,reason_detail
        ,sum(session_cnt) sum_session_cnt
        ,sum(session_cnt*satify_score)/sum(session_cnt) AS satify_score
    FROM dwd_ww_kf_session_satify_attrib_lastmonth
    GROUP BY session_type,reason,reason_detail
) t0
LEFT JOIN lastmonth_stat t1 ON t0.session_type = t1.session_type
""").createOrReplaceTempView("month_avg_stat")

sql_month = """
SELECT
    "{0}" AS imp_date
    ,'上月' AS compare_interval
    ,t0.reason
    ,t0.reason_detail
    ,CAST(t0.session_cnt AS DOUBLE) AS session_cnt
    ,t0.satify_score
    ,t0.session_rate
    ,t0.all_session_satify_score
    ,t0.satify_attrib
    ,t1.all_session_satify_score AS compare_all_session_satify_score
    ,t1.satify_score AS compare_satify_score
    ,t1.session_rate AS compare_session_rate
    ,t0.all_session_satify_score - t1.all_session_satify_score AS all_session_satify_diff
    ,t0.satify_score - t1.satify_score AS satify_score_diff
    ,t0.session_rate - t1.session_rate AS session_rate_diff
    ,(t0.satify_score*t0.session_rate - t1.satify_score*t1.session_rate)/100 AS satify_attrib_diff
    ,((t0.satify_score*t0.session_rate - t1.satify_score*t1.session_rate)/100)/(t0.all_session_satify_score - t1.all_session_satify_score) AS satify_attrib_rate
    ,(t0.satify_score*t0.session_rate)/(t1.satify_score*t1.session_rate) - 1 AS self_satify_attrib_rate
    ,CAST(t1.session_cnt AS DOUBLE) AS compare_session_cnt
    ,t1.satify_attrib AS compare_satify_attrib
    ,t0.satify_attrib - t1.satify_attrib AS compare_satify_attrib_diff
    ,t0.session_type
FROM dwd_ww_kf_session_satify_attrib_today t0
LEFT JOIN month_avg_stat t1
    ON t0.reason = t1.reason AND t0.reason_detail = t1.reason_detail AND t0.session_type = t1.session_type
""".format(statdate)
df_compare_month = spark.sql(sql_month).cache()

df_all = df_compare_month.union(df_compare_14d)
df_all.createOrReplaceTempView("df_all_view")

# 确保输出表以正确 schema 存在（agent 可能未建表或建了错误 schema）
spark.sql("DROP TABLE IF EXISTS internal_platform_db.caseR1_dwd_ww_kf_session_satify_attrib_compare_v3")
spark.sql('''
CREATE TABLE internal_platform_db.caseR1_dwd_ww_kf_session_satify_attrib_compare_v3 (
    `compare_interval` STRING,
    `reason` STRING,
    `reason_detail` STRING,
    `session_cnt` DOUBLE,
    `satify_score` DOUBLE,
    `session_rate` DOUBLE,
    `all_session_satify_score` DOUBLE,
    `satify_attrib` DOUBLE,
    `compare_all_session_satify_score` DOUBLE,
    `compare_satify_score` DOUBLE,
    `compare_session_rate` DOUBLE,
    `all_session_satify_diff` DOUBLE,
    `satify_score_diff` DOUBLE,
    `session_rate_diff` DOUBLE,
    `satify_attrib_diff` DOUBLE,
    `satify_attrib_rate` DOUBLE,
    `self_satify_attrib_rate` DOUBLE,
    `compare_session_cnt` DOUBLE,
    `compare_satify_attrib` DOUBLE,
    `compare_satify_attrib_diff` DOUBLE,
    `session_type` BIGINT,
    `imp_date` STRING
) STORED AS ORC
''')

# 落地：DDL 列序 (compare_interval, reason, ..., session_type, imp_date)
spark.sql("""
INSERT OVERWRITE TABLE internal_platform_db.caseR1_dwd_ww_kf_session_satify_attrib_compare_v3
(
    compare_interval, reason, reason_detail, session_cnt, satify_score, session_rate,
    all_session_satify_score, satify_attrib, compare_all_session_satify_score,
    compare_satify_score, compare_session_rate, all_session_satify_diff, satify_score_diff,
    session_rate_diff, satify_attrib_diff, satify_attrib_rate, self_satify_attrib_rate,
    compare_session_cnt, compare_satify_attrib, compare_satify_attrib_diff, session_type, imp_date
)
SELECT
    compare_interval, reason, reason_detail, session_cnt, satify_score, session_rate,
    all_session_satify_score, satify_attrib, compare_all_session_satify_score,
    compare_satify_score, compare_session_rate, all_session_satify_diff, satify_score_diff,
    session_rate_diff, satify_attrib_diff, satify_attrib_rate, self_satify_attrib_rate,
    compare_session_cnt, compare_satify_attrib, compare_satify_attrib_diff, session_type, imp_date
FROM df_all_view
""")

print("caseR1 done")


代码review的note diff按天同步到DWV层

Prompt:
## Prompt
我需要你生成一段 PySpark 代码，按天把代码评审场景下"代码 note 的 diff 文件明细"从 ODS 层同步到 DWV 层。

**业务背景与目标**：上游 databus 每天会落一份当天产生的 note diff 明细到 ods 表（每行 = 一条 note 评论 + 它附带的 diff 文本）。下游 DWV 层希望按 year/month/day 三级日期分区组织，方便日期回溯，且要对一天内出现的重复行（同一个 note id 配同一段 diff 文本）做去重。

**输入表**：
- `internal_platform_db.case5_ods_code_v2_note_diff_files_diff_v3`

**数据范围与过滤条件（业务说法）**：
- 只取业务日期等于本次运行日期 ds 的数据。
- ds 固定为 '20260513'（沙箱化日期，原任务通过 sys.argv[1] 传入；TaskType=63 不传命令行参数，所以硬编码避免 IndexError）。

**表关联关系**：
（无 Join，单表抽取。）

**聚合与计算规则（需体现在 SQL 中）**：
- 直接 SELECT id, diff 两列，并做 DISTINCT 去重（同一天内同 id 同 diff 只保留一条）。
- 在 SELECT 头部追加三列分区字段，由 ds 拆分得到：year='2026'、month='05'、day='13'。
- 使用 INSERT OVERWRITE TABLE 全表覆盖写入。

**输出要求**：
- 目标表：`internal_platform_db.case5_dwv_code_v2_note_diff_files_diff_v3`
- 输出表 schema（按此顺序写入，类型与业务含义如下）：
  1. year STRING：年，原为分区列
  2. month STRING：月，原为分区列
  3. day STRING：日，原为分区列
  4. id BIGINT：note id
  5. diff STRING：diff 文本内容
- 写入策略：INSERT OVERWRITE 全表覆盖，本次写入 year='2026', month='05', day='13'。
- 沙箱化说明：原任务在生产环境写 PARTITION(year, month, day) 分区表；沙箱版三列分区列已沙箱化为普通列。
- 如果目标表不存在，请先按 Hive 标准建表（ORC 存储），再写入数据。

#!/usr/bin/env bash
from __future__ import print_function

import re
import sys
import os
import json

import datetime

from pyspark.sql import SparkSession

if __name__ == "__main__":

    print('PythonSQL start')
    #yyyyMMdd
    ds = '20260513'  # was: sys.argv[1]
    match = re.match(r"([\d]{4})([\d]{2})([\d]{2})", ds)
    daySub = match.group(1, 2, 3)

    ds_year = daySub[0]
    ds_month = daySub[1]
    ds_day = daySub[2]

    last_day = datetime.datetime(
        int(daySub[0]), int(daySub[1]), int(daySub[2]))
    last_day_str = last_day.strftime('%Y%m%d')

    print('last_day_str=%s' % last_day_str)
    print('ds_year=%s' % ds_year)
    print('ds_month=%s' % ds_month)
    print('ds_day=%s' % ds_day)

    spark = SparkSession.builder.appName('case5_dwv_code_v2_note_diff_files_diff_d_f').enableHiveSupport().getOrCreate()

    # 确保输出表以正确 schema 存在（agent 可能未建表或建了错误 schema）
    spark.sql("DROP TABLE IF EXISTS internal_platform_db.case5_dwv_code_v2_note_diff_files_diff_v3")
    spark.sql('''
CREATE TABLE internal_platform_db.case5_dwv_code_v2_note_diff_files_diff_v3 (
    `year` STRING COMMENT '年，原为分区列',
    `month` STRING COMMENT '月，原为分区列',
    `day` STRING COMMENT '日，原为分区列',
    `id` BIGINT COMMENT '-',
    `diff` STRING COMMENT '-'
) STORED AS ORC
''')

    sql = '''
            insert overwrite table internal_platform_db.case5_dwv_code_v2_note_diff_files_diff_v3
            SELECT
                    DISTINCT
                    '$year' AS year, '$month' AS month, '$day' AS day, id ,diff
            FROM internal_platform_db.case5_ods_code_v2_note_diff_files_diff_v3 where databus_imp_date='$ds'
        '''

    sql = sql.replace('$ds', last_day_str)
    sql = sql.replace('$year', ds_year)
    sql = sql.replace('$month', ds_month)
    sql = sql.replace('$day', ds_day)

    print(sql)
    result = spark.sql(sql)
    print(result.show())
    spark.stop()


Sync Code Review review-diff from ODS to DWV Layer by Day

Prompt:
## Prompt
I need you to generate a PySpark script that synchronizes the "code review diff file details" from the ODS layer to the DWV layer by day for the code review scenario.

**Business Background and Objective**: The upstream databus produces a daily review diff detail (one row = one review comment + corresponding diff text) that lands in the ODS table. The downstream DWV layer wants to organize the data with a three-level date partition (year/month/day) for convenient date-based retrieval, and to deduplicate rows with the same id and diff within the same day. This task has the same structure as case5 (note diff), except the source/target tables are the review series.

**Input Table**:
- `internal_platform_db.case6_ods_code_v2_review_diff_files_diff_v3`

**Data Range and Filter Conditions (business description)**:
  - Only take data where the business date equals the current run date `ds`.
  - `ds` is fixed as `'20260513'` (sandboxed date; the original task passes this via `sys.argv[1]`; TaskType=63 does not pass command-line arguments, so it is hardcoded to avoid IndexError).

**Table Join Relationships**: (No joins, single-table extraction.)

**Aggregation and Computation Rules (must be reflected in the SQL)**:
  - Directly SELECT `id`, `diff` columns, and apply `DISTINCT` deduplication (only one row is retained for the same id and diff within the same day).
  - Prepend three partition columns at the beginning of the SELECT, derived by splitting `ds`: `year='2026'`, `month='05'`, `day='13'`.
  - Use `INSERT OVERWRITE TABLE` to overwrite the entire table.

**Output Requirements**:
  - Target table: `internal_platform_db.case6_dwv_code_v2_review_diff_files_diff_v3`
  - Output table schema (write in this order, types and business semantics as follows):
    1. `year` STRING: year, originally a partition column
    2. `month` STRING: month, originally a partition column
    3. `day` STRING: day, originally a partition column
    4. `id` BIGINT: review id
    5. `diff` STRING: diff text content
  - Write strategy: `INSERT OVERWRITE` full table overwrite, this write targets `year='2026'`, `month='05'`, `day='13'`.
  - Sandbox note: The original task writes to a `PARTITION(year, month, day)` partitioned table in production; in the sandbox version, the three partition columns have been sandboxed as regular columns.
  - If the target table does not exist, first create it using standard Hive format (ORC storage), then write the data.

# data-mocker rewrite-gt
# source: pyspark_0428_desensitized.jsonl idx=43924
# task_name: dwv_code_v2_review_diff_files_diff_d_f
# rewrite rules:
#   1) dept_t_yg_code.ods_code_v2_review_diff_files_diff_d_f
#       -> internal_platform_db.case6_ods_code_v2_review_diff_files_diff_v3
#   2) code.dwv_code_v2_review_diff_files_diff_d_f
#       -> internal_platform_db.case6_dwv_code_v2_review_diff_files_diff_v3
#   3) INSERT OVERWRITE TABLE ... PARTITION(year=...,month=...,day=...) SELECT id,diff
#       -> INSERT OVERWRITE TABLE ... SELECT '$year' AS year, '$month' AS month, '$day' AS day, id, diff
#      (沙箱用无分区表，分区列改为普通列)
#   4) sys.argv[1] -> 硬编码 '20260513'
#      (datawd PySpark TaskType=63 不直传命令行参数，硬编码避免 IndexError)
# ---
#!/usr/bin/env bash
from __future__ import print_function

import re
import sys
import os
import json

import datetime

from pyspark.sql import SparkSession

if __name__ == "__main__":

    print('PythonSQL start')
    #yyyyMMdd
    ds = '20260513'  # was: sys.argv[1]
    match = re.match(r"([\d]{4})([\d]{2})([\d]{2})", ds)
    daySub = match.group(1, 2, 3)

    ds_year = daySub[0]
    ds_month = daySub[1]
    ds_day = daySub[2]

    last_day = datetime.datetime(
        int(daySub[0]), int(daySub[1]), int(daySub[2]))
    last_day_str = last_day.strftime('%Y%m%d')

    print('last_day_str=%s' % last_day_str)
    print('ds_year=%s' % ds_year)
    print('ds_month=%s' % ds_month)
    print('ds_day=%s' % ds_day)

    spark = SparkSession.builder.appName('case6_dwv_code_v2_review_diff_files_diff_d_f').enableHiveSupport().getOrCreate()

    # 确保输出表以正确 schema 存在（agent 可能未建表或建了错误 schema）
    spark.sql("DROP TABLE IF EXISTS internal_platform_db.case6_dwv_code_v2_review_diff_files_diff_v3")
    spark.sql('''
CREATE TABLE internal_platform_db.case6_dwv_code_v2_review_diff_files_diff_v3 (
    `year` STRING COMMENT '年，原为分区列',
    `month` STRING COMMENT '月，原为分区列',
    `day` STRING COMMENT '日，原为分区列',
    `id` BIGINT COMMENT '-',
    `diff` STRING COMMENT '-'
) STORED AS ORC
''')

    sql = '''
            insert overwrite table internal_platform_db.case6_dwv_code_v2_review_diff_files_diff_v3
            SELECT
                    DISTINCT
                    '$year' AS year, '$month' AS month, '$day' AS day, id ,diff
            FROM internal_platform_db.case6_ods_code_v2_review_diff_files_diff_v3 where databus_imp_date='$ds'
        '''

    sql = sql.replace('$ds', last_day_str)
    sql = sql.replace('$year', ds_year)
    sql = sql.replace('$month', ds_month)
    sql = sql.replace('$day', ds_day)

    print(sql)
    result = spark.sql(sql)
    print(result.show())
    spark.stop()


销售线索企业邮品牌明细

Prompt:
## Prompt
我需要你生成一段 PySpark 代码，产出"销售线索企业邮品牌明细"。

**业务背景与目标**：线下销售团队有一份三非企业的销售线索表（含 corpid 唯一键），数据仓库侧有一份每月统计的邮箱平台画像表（每个 corpid 多行，按 top_domain / domain_isp 维度统计当月被唤醒的不同别名邮箱数量）。运营想要的是：对每一个三非线索企业，把它的活跃邮箱（domain_alias_monthwakeup_uniq >= 1）提取出来，并且把底层细粒度的 domain_isp 字段（如 'cloud_a'、'mail_provider_n'、'biz' 等）映射成业务可识别的标准品牌（'电商平台A'、'邮件平台N'、'企业邮品牌C' 等），最终落到 DWD 层供下游使用。

**输入表**：
- `internal_platform_db.project_alpha_ods_tele_sale_clue_info_mysql_v3`
- `internal_platform_db.project_alpha_dws_ww_corpstat_usermail_v3`

**数据范围与过滤条件（业务说法）**：
  - 只保留有活跃邮箱的企业，即 b.domain_alias_monthwakeup_uniq >= 1。
  - 服务商不能为空，即 b.domain_isp IS NOT NULL。
  - 运行日期 ds 固定为 '20260513'，写到输出 imp_date 列。

**表关联关系**：
  - 子查询 a：`SELECT DISTINCT corpid FROM project_alpha_ods_tele_sale_clue_info_mysql_v3`（拿到所有三非线索企业的 corpid）。
  - inner join：a ⋈ project_alpha_dws_ww_corpstat_usermail_v3 b on a.corpid = b.corpid。

**聚合与计算规则（需体现在 SQL 中）**：
  1. 派生列 `domain_mx`（邮箱品牌），用 CASE WHEN 把 domain_isp 映射到标准品牌：      - 'other_zijian' → '其他自建'      - 'coremail_local' → 'coremail本地版'      - 'mail_provider_n' → '邮件平台N'      - IN ('outlook','outlook_cn') → 'outlook'      - IN ('cloud_a','ali','ecommerce_a','ecommerce_a') → '电商平台A'      - 'eyou_local' → '亿邮本地版'      - 'exchange_local' → 'exchange'      - 'coremail' → 'coremail saas版'      - IN ('263','263mail') → '263'      - 'postfix_local' → 'postfix本地版'      - 'collabf' → '协作平台F'      - 'biz' → '企业邮品牌C'      - ELSE → '其他saas'
  2. imp_date 字段固定填 '20260513'。

**输出要求**：
  - 目标表：`internal_platform_db.project_alpha_dwd_ww_tele_sale_corp_domain_mx_v3`
  - 输出表 schema（按此顺序、含义如下，5 列显式写出）：     1. imp_date STRING：运行日期 YYYYMMDD = '20260513'     2. corpid BIGINT：企业 ID     3. top_domain STRING：最多人使用的邮箱域名     4. domain_isp STRING：邮箱服务商底层标识（保留原值）     5. domain_mx STRING：CASE WHEN 派生的标准品牌
  - 写入策略：INSERT OVERWRITE TABLE，使用显式列名清单。
  - 沙箱化说明：原任务由 SparkBase 框架封装（job.load_dw_data + selectExpr 链路 + job.save_dw_data）；沙箱版改写为顶层 SparkSession + 一段 INNER JOIN + WHERE + CASE WHEN 的 SQL。
  - 如果目标表不存在，请先按 Hive 标准建表（ORC 存储），再写入数据。

# project_alpha rewrite-gt
# source: data/v7_gt_codes/case_row08_case_0028.py (v7 row 7, manifest_row 8)
# task_name: t_dwd_ww_tele_sale_corp_domain_mx_copilot
# rewrite rules:
#   1) internal_platform_db.ods_tele_sale_clue_info_mysql_copilot
#       -> internal_platform_db.project_alpha_ods_tele_sale_clue_info_mysql_v3
#   2) internal_platform_db.dws_ww_corpstat_usermail_copilot
#       -> internal_platform_db.project_alpha_dws_ww_corpstat_usermail_v3
#   3) internal_platform_db.dwd_ww_tele_sale_corp_domain_mx_copilot
#       -> internal_platform_db.project_alpha_dwd_ww_tele_sale_corp_domain_mx_v3
#   4) 删除 dw_spark_base_python3.SparkBase（沙箱无）：
#      - job = SparkBase() -> spark = SparkSession.builder...getOrCreate()
#      - job.load_dw_data(db, tbl).select/join/where/selectExpr 链路
#        -> 直接 spark.sql 写 INNER JOIN + WHERE + CASE WHEN 的完整 SELECT
#      - job.save_dw_data(df, db, tbl) -> spark.sql('INSERT OVERWRITE TABLE ...')
#      - job.statdate -> 硬编码 '20260513'
#   5) INSERT 用显式列名清单（5 列）避免列序错位
#   6) 沙箱表无分区，原 imp_date 分区列已沙箱化为普通列；本 GT 不读它
#      （原 GT 也不过滤 imp_date，是从全表读 → 沙箱表只灌目标 ds 即可）
# ---
from __future__ import print_function

import sys
from pyspark.sql import SparkSession

if __name__ == "__main__":
    ds = '20260513'

    spark = (
        SparkSession.builder
        .appName("project_alpha_dwd_ww_tele_sale_corp_domain_mx")
        .enableHiveSupport()
        .getOrCreate()
    )

    # 确保输出表以正确 schema 存在
    spark.sql("DROP TABLE IF EXISTS internal_platform_db.project_alpha_dwd_ww_tele_sale_corp_domain_mx_v3")
    spark.sql('''
CREATE TABLE internal_platform_db.project_alpha_dwd_ww_tele_sale_corp_domain_mx_v3 (
    `imp_date` STRING COMMENT '运行日期(YYYYMMDD)',
    `corpid` BIGINT COMMENT 'corpid',
    `top_domain` STRING COMMENT '最多人邮箱域名',
    `domain_isp` STRING COMMENT '邮箱服务商',
    `domain_mx` STRING COMMENT 'CASE WHEN 派生品牌'
) STORED AS ORCFILE
''')

    # 销售线索企业邮品牌明细
    sql = """
    INSERT OVERWRITE TABLE internal_platform_db.project_alpha_dwd_ww_tele_sale_corp_domain_mx_v3 (
        imp_date, corpid, top_domain, domain_isp, domain_mx
    )
    SELECT
        '{0}' as imp_date,
        b.corpid,
        b.top_domain,
        b.domain_isp,
        (CASE
            WHEN b.domain_isp = 'other_zijian' THEN '其他自建'
            WHEN b.domain_isp = 'coremail_local' THEN 'coremail本地版'
            WHEN b.domain_isp = 'mail_provider_n' THEN '邮件平台N'
            WHEN b.domain_isp IN ('outlook','outlook_cn') THEN 'outlook'
            WHEN b.domain_isp IN ('cloud_a','ali','ecommerce_a','ecommerce_a') THEN '电商平台A'
            WHEN b.domain_isp = 'eyou_local' THEN '亿邮本地版'
            WHEN b.domain_isp = 'exchange_local' THEN 'exchange'
            WHEN b.domain_isp = 'coremail' THEN 'coremail saas版'
            WHEN b.domain_isp IN ('263','263mail') THEN '263'
            WHEN b.domain_isp = 'postfix_local' THEN 'postfix本地版'
            WHEN b.domain_isp = 'collabf' THEN '协作平台F'
            WHEN b.domain_isp = 'biz' THEN '企业邮品牌C'
            ELSE '其他saas'
        END) AS domain_mx
    FROM (
        SELECT DISTINCT corpid
        FROM internal_platform_db.project_alpha_ods_tele_sale_clue_info_mysql_v3
    ) a
    INNER JOIN internal_platform_db.project_alpha_dws_ww_corpstat_usermail_v3 b
        ON a.corpid = b.corpid
    WHERE b.domain_alias_monthwakeup_uniq >= 1
      AND b.domain_isp IS NOT NULL
    """.format(ds)
    print("sql:")
    print(sql)
    spark.sql(sql)
    print("saved")

    spark.stop()
    sys.exit(0)


AI Search Parse Quality Daily Detail (case9 version)

Prompt:
## Prompt
Business Background: Compute daily details of AI search web page parse quality by site + directory dimensions. Take data from the past few days, aggregate success rate, index page ratio, short content ratio, and failure step counts by the (date_key, host, fld) dimension, and join with the Top500 site priority labels.

**Input Tables**:
- `internal_platform_db.ai_engine_classify_parse_result_daily_copilot_v3`
- `internal_platform_db.sec_app_hy_top_500_sites_tag_v1_copilot_v3`

  Data Range:
  - Fact table filter: `date_key >= '20260508' AND date_key <= '20260513'` (hardcoded, today='20260513', looking back 5 days)
  - Dimension table participates in full

  Processing Logic:

  1. Field preparation: Select the following columns from the fact table:

  - `date_key` retained as-is

  - `host` → renamed to `source_host`

  - `fld` → renamed to `source_fld`

  - `when(col("is_error") == 0, 1).otherwise(0)` → `success` (note: `is_error` is STRING; comparing with `== 0` triggers implicit type conversion in Spark)

  - `when(col("is_error") == 1, 1).otherwise(0)` → `failure`

  - `web_type` retained as-is

  - `from_json(col("parse_html"), schema_of_json('{"content": "string"}')).getField("content")` → `content`

  - `error_step` retained as-is

  - Then `withColumn("content_length", length(col("content")))`
  2. Four aggregations (all grouped by `date_key`, `source_host`, `source_fld`):

  - summary: `sum(success)` → `successful_parses`, `sum(failure)` → `failed_parses`, `count(*)` → `total_parses`

  - index_page_ratio: `sum(when(web_type=='索引页', 1).otherwise(0))` → `index_page_count`

  - content_length_ratio: first `filter(success==1)`, then `sum(when(content_length<50, 1).otherwise(0))` → `short_content_count`

  - error_reasons: first `filter(failure==1)`, then separately `sum(when(error_step==1/2/3, 1).otherwise(0))` → `error_step_1/2/3_count`; additionally `sum(when(error_step.isNotNull(), 1).otherwise(0))` → `total_error_steps` (this field is not used subsequently)
  3. Merge: Left join the four aggregation results on `[date_key, source_host, source_fld]` to form `final_result`
  4. Compute ratios:

  - `index_page_count_rate = index_page_count / total_parses`

  - `short_content_count_rate = short_content_count / successful_parses` (no divide-by-zero protection; produces null when `successful_parses=0`)
  5. Hardcoded empty fields:

  - `withColumn("error_info_list", lit(""))` — no `collect_list`, directly fixed as empty string

  - `withColumn("is_host", lit(""))` — not taken from the dimension table, directly fixed as empty string

  6. Join dimension table (key: OR condition):
  `final_result.join(`

`priority_info`, — contains only `host`, `fld`, `priority` columns

`(final_result.source_fld == priority_info.fld) OR`

`(final_result.source_host == priority_info.host),`

`"left"`

  `)`
  6. Note: The join condition is OR (matching on either fld or host), not AND. This may cause one main table record to match multiple rows in the dimension table.
  7. Final select (16 columns, in this order):

  - `date_key`, `source_host`→`host`, `source_fld`→`fld`, `is_host` (from step 5's `lit("")`), `priority` (from dimension table join), `successful_parses`, `failed_parses`, `total_parses`, `index_page_count`, `index_page_count_rate`, `short_content_count`,
  `short_content_count_rate`, `error_step_1_count`, `error_step_2_count`, `error_step_3_count`, `error_info_list` (from step 5's `lit("")`)
  8. Deduplication: `dropDuplicates(["date_key", "host", "fld", "priority", "successful_parses", "total_parses", "index_page_count"])`

  - Note: Because the OR join may produce multiple rows (one main table row matching multiple dimension table rows), deduplication is used to converge the inflated rows
  9. Write: `final_result.write.mode("overwrite").insertInto(target table)`

  Output table: `internal_platform_db.ai_engine_classify_parse_result_daily_detail_copilot_v3`

# data-mocker rewrite-gt
# source: 离线计算pyspark例子_v7.xlsx 工作表1 row=10 col=3
# row_idx: 9
# intent_short: 帮我做个AI搜索引擎解析质量的每日明细统计，按站点和目录维度算一下成功率、索引页占比、短内容占比，还有各环节的失败分布。另外需要把Top500站点的优先级标签关联上
# v3 tables: internal_platform_db.ai_engine_classify_parse_result_daily_detail_copilot_v3, internal_platform_db.ai_engine_classify_parse_result_daily_copilot_v3, internal_platform_db.sec_app_hy_top_500_sites_tag_v1_copilot_v3
# extra fqns scanned from GT (not in v7_rows.tables): internal_platform_db.ai_engine_classify_parse_result_daily_detail_copilot
# rewrite rule: <fqn> -> <fqn>_v3 (covers bare / `fqn` / db.`tbl`)
# ---
#!/usr/bin/env python
# coding: utf-8

import os
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, count, when, length, sum as _sum, from_json, schema_of_json
from pyspark.sql.types import StringType
from datetime import datetime, timedelta
from pyspark.sql.functions import lit
from pyspark.sql import functions as F
# 创建SparkSession
spark = (

   SparkSession.builder.enableHiveSupport()

   .config("spark.driver.memory", "6g")
        .config("spark.executor.cores", 6)

   .config("spark.executor.memory", "12g")
        .getOrCreate()
)

# 确保输出表以正确 schema 存在
spark.sql("DROP TABLE IF EXISTS internal_platform_db.ai_engine_classify_parse_result_daily_detail_copilot_v3")
spark.sql('''
CREATE TABLE internal_platform_db.ai_engine_classify_parse_result_daily_detail_copilot_v3 (
`date_key` STRING COMMENT '-',
`host` STRING COMMENT '-',
`fld` STRING COMMENT '-',
`is_host` STRING COMMENT '-',
`priority` STRING COMMENT '-',
`successful_parses` BIGINT COMMENT '-',
`failed_parses` BIGINT COMMENT '-',
`total_parses` BIGINT COMMENT '-',
`index_page_count` BIGINT COMMENT '-',
`index_page_count_rate` DOUBLE COMMENT '-',
`short_content_count` BIGINT COMMENT '-',
`short_content_count_rate` DOUBLE COMMENT '-',
`error_step_1_count` BIGINT COMMENT '-',
`error_step_2_count` BIGINT COMMENT '-',
`error_step_3_count` BIGINT COMMENT '-',
`error_info_list` STRING COMMENT '-'
) STORED AS ORCFILE
''')

# # 获取今天的日期，并计算10天前的日期
# today = datetime.now()
# ten_days_ago = today - timedelta(days=5)

# # 将日期格式化为 YYYYMMDD
# today_str = today.strftime('%Y%m%d')
# ten_days_ago_str = ten_days_ago.strftime('%Y%m%d')
today_str = '20260513'
ten_days_ago_str = '20260508'
print(today_str, ten_days_ago_str)

# 读取 priority 信息
priority_info = spark.table("internal_platform_db.sec_app_hy_top_500_sites_tag_v1_copilot_v3").select(
    "host", "fld",
   "priority"
)

# 读取数据，从今天往前10天的数据
df = spark.table("internal_platform_db.ai_engine_classify_parse_result_daily_copilot_v3").filter(
    (col("date_key") >= ten_days_ago_str) & (col("date_key") <= today_str)
)

# 解析数据
parsed_data = df.select(
    "date_key",

   col("host").alias("source_host") ,
    when(col("is_error") == 0, 1).otherwise(0).alias("success"),
    when(col("is_error") == 1, 1).otherwise(0).alias("failure"),
    "web_type",
    # 从 parse_html 中提取 content 字段

   from_json(col("parse_html"), schema_of_json('{"content": "string"}')).getField("content").alias("content"),
    "error_step",
    col("fld").alias("source_fld")  # 使用别名避免歧义
)

# 计算内容长度
parsed_data = parsed_data.withColumn("content_length", length(col("content")))

# 在 summary_copilot 中保留 source_fld 字段
summary = parsed_data.groupBy("date_key", "source_host", "source_fld").agg(  # 使用别名
    _sum("success").alias("successful_parses"),

   _sum("failure").alias("failed_parses"),
    count("*").alias("total_parses")
)

# 计算索引页比例
index_page_ratio = parsed_data.groupBy("date_key","source_host", "source_fld").agg(  # 使用别名

   _sum(when(col("web_type") == "索引页", 1).otherwise(0)).alias("index_page_count")
)

# 计算内容长度比例
content_length_ratio = parsed_data.filter(col("success") == 1).groupBy("date_key","source_host", "source_fld").agg(  # 使用别名

   _sum(when(col("content_length") < 50, 1).otherwise(0)).alias("short_content_count")
)

# 计算错误原因
error_reasons = parsed_data.filter(col("failure") == 1).groupBy("date_key", "source_host", "source_fld").agg(  # 使用别名

   _sum(when(col("error_step") == 1, 1).otherwise(0)).alias("error_step_1_count"),
    _sum(when(col("error_step") == 2, 1).otherwise(0)).alias("error_step_2_count"),

   _sum(when(col("error_step") == 3, 1).otherwise(0)).alias("error_step_3_count"),

   _sum(when(col("error_step").isNotNull(), 1).otherwise(0)).alias("total_error_steps")
)

# 最终合并
final_result = summary.join(index_page_ratio, ["date_key", "source_host", "source_fld"], "left") \
    .join(content_length_ratio, ["date_key","source_host", "source_fld"], "left") \
    .join(error_reasons, ["date_key","source_host", "source_fld"], "left")

# 计算比例
final_result = final_result.withColumn(

   "index_page_count_rate",
    col("index_page_count") / col("total_parses")
).withColumn(

   "short_content_count_rate",

   col("short_content_count") / col("successful_parses")
)

# 添加 error_info_list 字段
final_result = final_result.withColumn("error_info_list", lit(""))
final_result = final_result.withColumn("is_host", lit(""))

####fld存在500tag则扩散对应的ccdata的host组合---host存在则直接匹配host

final_result = final_result.join(
    priority_info,
    (final_result.source_fld == priority_info.fld) |
    (final_result.source_host == priority_info.host),
    "left"
)
# 选择所需的列
final_result = final_result.select(
    "date_key",

   F.col("source_host").alias("host"),

   F.col("source_fld").alias("fld"),  # 使用 source_fld
    "is_host",
    "priority",
    "successful_parses",
    "failed_parses",
    "total_parses",
    "index_page_count",

   "index_page_count_rate",

   "short_content_count",

   "short_content_count_rate",
    "error_step_1_count",

   "error_step_2_count",

   "error_step_3_count",
    "error_info_list"
)

final_result = final_result.dropDuplicates(["date_key", "host", "fld","priority","successful_parses","total_parses","index_page_count"])
# 将结果写入目标表
final_result.write.mode("overwrite").insertInto("internal_platform_db.ai_engine_classify_parse_result_daily_detail_copilot_v3")


AI Search Parse Quality Daily Detail

Prompt:
## Prompt
I need you to generate a PySpark script that produces the "AI Search Parse Quality Daily Detail".

Business Background: Compute daily details of AI search web page parse quality by site + directory dimensions. Take data from the past few days, aggregate success rate, index page ratio, short content ratio, and failure step counts by the (date_key, host, fld) dimension, and join with the Top500 site priority labels.

**Input Tables**:
- `internal_platform_db.case10_ai_engine_classify_parse_result_daily_v3`
- `internal_platform_db.case10_sec_app_hy_top_500_sites_tag_v1_v3`

  Data Range:

  - Fact table filter: `date_key >= '20260508' AND date_key <= '20260513'` (hardcoded, today='20260513', looking back 5 days)
  - Dimension table participates in full, no pre-filtering

  Processing Logic:

  1. Field preparation: Select the following columns from the fact table:

  - `date_key` as-is

  - `host` → renamed `source_host`

  - `when(col("is_error") == 0, 1).otherwise(0)` → `success` (`is_error` is STRING; relies on Spark implicit conversion to compare with integer 0)

  - `when(col("is_error") == 1, 1).otherwise(0)` → `failure`

  - `web_type` as-is

  - `from_json(col("parse_html"), schema_of_json('{"content": "string"}')).getField("content")` → `content`

  - `error_step` as-is

  - `fld` → renamed `source_fld`

  - Then `withColumn`: `content_length = length(content)`
  2. Four independent aggregations (all grouped by `date_key`, `source_host`, `source_fld`):

  - summary: `sum(success)` → `successful_parses`, `sum(failure)` → `failed_parses`, `count(*)` → `total_parses`

  - index_page_ratio: `sum(when(web_type=='索引页', 1).otherwise(0))` → `index_page_count`

  - content_length_ratio: first `filter(success==1)`, then `sum(when(content_length<50, 1).otherwise(0))` → `short_content_count`

  - error_reasons: first `filter(failure==1)`, then:

- `sum(when(error_step==1, 1).otherwise(0))` → `error_step_1_count`

- `sum(when(error_step==2, 1).otherwise(0))` → `error_step_2_count`

- `sum(when(error_step==3, 1).otherwise(0))` → `error_step_3_count`

- `sum(when(error_step.isNotNull(), 1).otherwise(0))` → `total_error_steps` (this field is not used or output subsequently)
  3. Merge: Left join the four aggregation results sequentially on `[date_key, source_host, source_fld]`

  4. Compute ratios:

  - `index_page_count_rate = index_page_count / total_parses`

  - `short_content_count_rate = short_content_count / successful_parses` (no divide-by-zero protection; produces null when `successful_parses=0`)
  5. Hardcoded empty fields:

  - `withColumn("error_info_list", lit(""))` — fixed empty string, no `collect_list`

  - `withColumn("is_host", lit(""))` — fixed empty string, not obtained from dimension table
  6. Join dimension table (OR condition):
  `final_result.join(`

`priority_info`, — contains only `host`, `fld`, `priority`

`(source_fld == priority_info.fld) OR (source_host == priority_info.host),`

`"left"`

  `)`
  6. Key: The condition is OR (matching on either fld or host), not AND. One main table record may match multiple dimension table rows, causing row inflation.
  7. Final select (16 columns, in this order):
  `date_key`, `source_host`→`host`, `source_fld`→`fld`, `is_host` (step5's `lit("")`), `priority` (from dimension table join), `successful_parses`, `failed_parses`, `total_parses`, `index_page_count`, `index_page_count_rate`, `short_content_count`, `short_content_count_rate`,
  `error_step_1_count`, `error_step_2_count`, `error_step_3_count`, `error_info_list` (step5's `lit("")`)
  8. Deduplication: `dropDuplicates(["date_key", "host", "fld", "priority", "successful_parses", "total_parses", "index_page_count"])` (to converge inflated rows from the OR join)
  9. Write: Write via `createOrReplaceTempView` + `INSERT OVERWRITE TABLE`, explicitly specifying column name order (the output table DDL column order differs from the DataFrame column order):

  - Output table column order: `host`, `fld`, `is_host`, `priority`, `successful_parses`, `failed_parses`, `total_parses`, `index_page_count`, `index_page_count_rate`, `short_content_count`, `short_content_count_rate`, `error_step_1_count`, `error_step_2_count`,
  `error_step_3_count`, `error_info_list`, `date_key`

  - All fields are written with `CAST AS STRING`

  Output table: `internal_platform_db.case10_ai_engine_classify_parse_result_daily_detail_v3`
  - Output table DDL column order (note that `date_key` is last): `host`, `fld`, `is_host`, `priority`, `successful_parses`, `failed_parses`, `total_parses`, `index_page_count`, `index_page_count_rate`, `short_content_count`, `short_content_count_rate`, `error_step_1_count`,
  `error_step_2_count`, `error_step_3_count`, `error_info_list`, `date_key`
  - All column types are STRING

# case10 rewrite-gt
# source: data/v7_gt_codes/case_row10_case_0043.py (v7 row 9, manifest_row 10)
# task_name: ai_engine_classify_parse_result_daily_detail
# rewrite rules:
#   1) internal_platform_db.ai_engine_classify_parse_result_daily_copilot
#       -> internal_platform_db.case10_ai_engine_classify_parse_result_daily_v3
#   2) internal_platform_db.sec_app_hy_top_500_sites_tag_v1_copilot
#       -> internal_platform_db.case10_sec_app_hy_top_500_sites_tag_v1_v3
#   3) internal_platform_db.ai_engine_classify_parse_result_daily_detail_copilot
#       -> internal_platform_db.case10_ai_engine_classify_parse_result_daily_detail_v3
#   4) today = datetime.now() -> 硬编码 today_str = '20260513'
#      ten_days_ago_str = '20260508'（today - 5 天，与原 timedelta(days=5) 一致）
#   5) 沙箱主表去 date_key 分区（thive UI 建分区表 INSERT VALUES 全 silent no-op）
#      -> 改为普通 STRING 列；GT 仍按 date_key 范围过滤
#   6) 末尾 final_result.write.mode("overwrite").insertInto(...) -> 改为
#      createOrReplaceTempView + spark.sql('INSERT OVERWRITE TABLE ... SELECT ...')
#      显式列名清单（16 列）避免 DataFrame select 列序与 output DDL 列序错位
# ---
import os
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, count, when, length, sum as _sum, from_json, schema_of_json
from pyspark.sql.types import StringType
from datetime import datetime, timedelta
from pyspark.sql.functions import lit
from pyspark.sql import functions as F

spark = (
    SparkSession.builder
        .appName("case10_ai_engine_classify_parse_result_daily_detail")
        .enableHiveSupport()
        .config("spark.driver.memory", "6g")
        .config("spark.executor.cores", 6)
        .config("spark.executor.memory", "12g")
        .getOrCreate()
)

# 确保输出表以正确 schema 存在
spark.sql("DROP TABLE IF EXISTS internal_platform_db.case10_ai_engine_classify_parse_result_daily_detail_v3")
spark.sql('''
CREATE TABLE internal_platform_db.case10_ai_engine_classify_parse_result_daily_detail_v3 (
    `host` STRING,
    `fld` STRING,
    `is_host` STRING,
    `priority` STRING,
    `successful_parses` STRING,
    `failed_parses` STRING,
    `total_parses` STRING,
    `index_page_count` STRING,
    `index_page_count_rate` STRING,
    `short_content_count` STRING,
    `short_content_count_rate` STRING,
    `error_step_1_count` STRING,
    `error_step_2_count` STRING,
    `error_step_3_count` STRING,
    `error_info_list` STRING,
    `date_key` STRING
) STORED AS ORCFILE
''')

# 沙箱化运行日期（原 today=datetime.now()，ten_days_ago=today-timedelta(days=5)）
today_str = '20260513'
ten_days_ago_str = '20260508'
print(today_str, ten_days_ago_str)

# 读取 priority 信息
priority_info = spark.table("internal_platform_db.case10_sec_app_hy_top_500_sites_tag_v1_v3").select(
    "host", "fld", "priority"
)

# 读取数据，从今天往前 5 天的数据
df = spark.table("internal_platform_db.case10_ai_engine_classify_parse_result_daily_v3").filter(
    (col("date_key") >= ten_days_ago_str) & (col("date_key") <= today_str)
)

# 解析数据
parsed_data = df.select(
    "date_key",
    col("host").alias("source_host"),
    when(col("is_error") == 0, 1).otherwise(0).alias("success"),
    when(col("is_error") == 1, 1).otherwise(0).alias("failure"),
    "web_type",
    from_json(col("parse_html"), schema_of_json('{"content": "string"}')).getField("content").alias("content"),
    "error_step",
    col("fld").alias("source_fld"),
)

# 计算内容长度
parsed_data = parsed_data.withColumn("content_length", length(col("content")))

summary = parsed_data.groupBy("date_key", "source_host", "source_fld").agg(
    _sum("success").alias("successful_parses"),
    _sum("failure").alias("failed_parses"),
    count("*").alias("total_parses"),
)

index_page_ratio = parsed_data.groupBy("date_key", "source_host", "source_fld").agg(
    _sum(when(col("web_type") == "索引页", 1).otherwise(0)).alias("index_page_count")
)

content_length_ratio = parsed_data.filter(col("success") == 1).groupBy("date_key", "source_host", "source_fld").agg(
    _sum(when(col("content_length") < 50, 1).otherwise(0)).alias("short_content_count")
)

error_reasons = parsed_data.filter(col("failure") == 1).groupBy("date_key", "source_host", "source_fld").agg(
    _sum(when(col("error_step") == 1, 1).otherwise(0)).alias("error_step_1_count"),
    _sum(when(col("error_step") == 2, 1).otherwise(0)).alias("error_step_2_count"),
    _sum(when(col("error_step") == 3, 1).otherwise(0)).alias("error_step_3_count"),
    _sum(when(col("error_step").isNotNull(), 1).otherwise(0)).alias("total_error_steps"),
)

final_result = summary.join(index_page_ratio, ["date_key", "source_host", "source_fld"], "left") \
    .join(content_length_ratio, ["date_key", "source_host", "source_fld"], "left") \
    .join(error_reasons, ["date_key", "source_host", "source_fld"], "left")

final_result = final_result.withColumn(
    "index_page_count_rate",
    col("index_page_count") / col("total_parses"),
).withColumn(
    "short_content_count_rate",
    col("short_content_count") / col("successful_parses"),
)

final_result = final_result.withColumn("error_info_list", lit(""))
final_result = final_result.withColumn("is_host", lit(""))

# fld 存在 500tag 则扩散 host 组合 / host 存在则直接匹配 host
final_result = final_result.join(
    priority_info,
    (final_result.source_fld == priority_info.fld) |
    (final_result.source_host == priority_info.host),
    "left",
)

final_result = final_result.select(
    "date_key",
    F.col("source_host").alias("host"),
    F.col("source_fld").alias("fld"),
    "is_host",
    "priority",
    "successful_parses",
    "failed_parses",
    "total_parses",
    "index_page_count",
    "index_page_count_rate",
    "short_content_count",
    "short_content_count_rate",
    "error_step_1_count",
    "error_step_2_count",
    "error_step_3_count",
    "error_info_list",
)

final_result = final_result.dropDuplicates([
    "date_key", "host", "fld", "priority",
    "successful_parses", "total_parses", "index_page_count",
])

# INSERT OVERWRITE 显式列名（16 列）；output DDL 列序：host, fld, is_host, priority, ..., date_key
final_result.createOrReplaceTempView("case10_final_result_view")
insert_sql = """
INSERT OVERWRITE TABLE internal_platform_db.case10_ai_engine_classify_parse_result_daily_detail_v3 (
    host, fld, is_host, priority,
    successful_parses, failed_parses, total_parses,
    index_page_count, index_page_count_rate,
    short_content_count, short_content_count_rate,
    error_step_1_count, error_step_2_count, error_step_3_count,
    error_info_list, date_key
)
SELECT
    CAST(host AS STRING),
    CAST(fld AS STRING),
    CAST(is_host AS STRING),
    CAST(priority AS STRING),
    CAST(successful_parses AS STRING),
    CAST(failed_parses AS STRING),
    CAST(total_parses AS STRING),
    CAST(index_page_count AS STRING),
    CAST(index_page_count_rate AS STRING),
    CAST(short_content_count AS STRING),
    CAST(short_content_count_rate AS STRING),
    CAST(error_step_1_count AS STRING),
    CAST(error_step_2_count AS STRING),
    CAST(error_step_3_count AS STRING),
    CAST(error_info_list AS STRING),
    CAST(date_key AS STRING)
FROM case10_final_result_view
"""
print("insert_sql:")
print(insert_sql)
spark.sql(insert_sql)
print("saved")

spark.stop()


Sync Cloud-Check User Active Status to Application Layer

Prompt:
## Prompt
I need you to generate a PySpark script that synchronizes the "user cloud-check active status" detail from ODS to the application layer DWV, and applies compliance encryption to the phone number to generate a unionid.

**Business Background and Objective**: In the upstream ODS, the `phone_enc` field is the bottom-layer encrypted phone number. The downstream application layer wants to further convert it into a `unionid` column (referred to as `phone` in business terms) using a deterministic one-way encryption. In addition to encryption, the active status fields (inactive days, sleep flag, first/recent/last cloud-check dates) must be passed through as-is, written out by `ds`.

**Input Table**:
- `internal_platform_db.case15_ods_phone_active_df_v3`

**Data Range and Filter Conditions (business description)**:
  - Only take the `ds` = current day partition; for this run, `ds` is fixed as `'20260513'` (the original task passes this via `sys.argv[1]`; TaskType=63 does not pass command-line arguments, so it is hardcoded to avoid IndexError).

**Table Join Relationships**: (No joins, single-table extraction.)

**Aggregation and Computation Rules (must be reflected in the SQL)**:
  1. Derived column `phone`: The original task encrypts `phone_enc` via the `OpenidCrypt` UDF; this UDF's jar (`hdfs://cluster-alpha/...`) is unavailable in the sandbox (tested exit code 1), so `sha2(phone_enc, 256)` is used as a replacement — also a deterministic one-way encryption, with equivalent business semantics.
  2. Column renaming (alias during passthrough):
     - `his_call_date_cloud` → `his_call_date_total`
     - `first_call_date_cloud` → `first_call_date_total`
     - `last_call_date_cloud` → `last_call_date_total`
  3. Other fields (`phone_type`, `not_active_days`, `is_sleep_flag`, `ds`) are passed through as-is.

**Output Requirements**:
  - Target table: `internal_platform_db.case15_dwv_phone_regul_unionid_df_v3`
  - Output table schema (in this order, with the following semantics):
    1. `phone` STRING: encrypted unionid
    2. `phone_type` STRING: phone number type
    3. `not_active_days` INT: recent consecutive inactive days
    4. `is_sleep_flag` STRING: whether currently in sleep state
    5. `his_call_date_total` STRING: historical first query date
    6. `first_call_date_total` STRING: recent first query date
    7. `last_call_date_total` STRING: last query date
    8. `ds` BIGINT: business date (originally a partition column, sandboxed as a regular column)
  - Write strategy: `INSERT OVERWRITE TABLE`, using an explicit column name list (8 columns) to avoid column order misalignment; config `spark.sql.storeAssignmentPolicy=LEGACY` to accommodate implicit type conversion.
  - Sandbox note: The original `PARTITION(ds={ds})` write is changed to a regular column + `WHERE ds = {ds}` regular column filter; the `OpenidCrypt` UDF is replaced with `sha2(phone_enc, 256)`; `jar_addr` addJar and `registerJavaFunction` are removed.
  - If the target table does not exist, first create it using standard Hive format (ORC storage), then write the data.

# data-mocker rewrite-gt
# source: v7 xlsx row 16 (case15)
# task_name: t_app_regul_user_comm_security_cloud_active_status_phone_uid_df_copilot
# rewrite rules:
#   1) internal_platform_db.t_app_comm_security_user_cloud_active_status_phone_df_copilot
#       -> internal_platform_db.case15_ods_phone_active_df_v3
#   2) internal_platform_db.t_app_regul_user_comm_security_cloud_active_status_phone_uid_df_copilot
#       -> internal_platform_db.case15_dwv_phone_regul_unionid_df_v3
#   3) PARTITION(ds=...) 删除；output 表 ds 改为普通列，SELECT 头部补 ds 列
#      WHERE ds = {ds} 保留（普通列过滤）
#      INSERT 用显式列名避免列序错位
#   4) sys.argv[1] -> 硬编码 '20260513'
#      (datawd PySpark TaskType=63 不直传命令行参数)
#   5) OpenidCrypt UDF -> sha2(phone_enc, 256)
#      (原 jar hdfs://cluster-alpha/... 在沙箱不可用，实测 exit code 1；
#       sha2 同样是确定性单向加密，业务语义等价)
#   6) 移除 jar_addr addJar + registerJavaFunction（不再需要）
# ---
import sys
from pyspark.sql import SparkSession

"""

脚本说明：API-SDK云查（sha2 加密版）

"""

# 生成和配置spark实例
def get_spark_cli():
    spark = SparkSession.builder \
        .enableHiveSupport() \
        .config("spark.sql.storeAssignmentPolicy", "LEGACY") \
        .getOrCreate()
    return spark

# 生成spark sql语句
def get_sql_str():
    ds = '20260513'  # was: sys.argv[1]
    sql = f"""
INSERT OVERWRITE TABLE internal_platform_db.case15_dwv_phone_regul_unionid_df_v3
(phone, phone_type, not_active_days, is_sleep_flag, his_call_date_total, first_call_date_total, last_call_date_total, ds)
SELECT
sha2(phone_enc, 256) as phone,
phone_type,
not_active_days,
is_sleep_flag,
his_call_date_cloud as his_call_date_total,
first_call_date_cloud as first_call_date_total,
last_call_date_cloud as last_call_date_total,
ds
FROM internal_platform_db.case15_ods_phone_active_df_v3
where ds = {ds}
"""
    print(f"sql: {sql}")
    return sql

# 执行
if __name__ == "__main__":
    spark = get_spark_cli()
    # 确保输出表以正确 schema 存在
    spark.sql("DROP TABLE IF EXISTS internal_platform_db.case15_dwv_phone_regul_unionid_df_v3")
    spark.sql('''
CREATE TABLE internal_platform_db.case15_dwv_phone_regul_unionid_df_v3 (
    `phone` STRING COMMENT '加密后的 unionid',
    `phone_type` STRING COMMENT '号码类型',
    `not_active_days` INT COMMENT '最近连续不活跃天数',
    `is_sleep_flag` STRING COMMENT '当前是否处于休眠状态',
    `his_call_date_total` STRING COMMENT '历史首次查询日期',
    `first_call_date_total` STRING COMMENT '最近首次查询日期',
    `last_call_date_total` STRING COMMENT '最后查询日期',
    `ds` BIGINT COMMENT 'partition fields（原为分区列）'
) STORED AS ORCFILE
''')
    SQL = get_sql_str()
    spark.sql(SQL)


Cleanse API Cloud-Check Logs

Prompt:
## Prompt
I need you to generate a PySpark script that cleanses the "API cloud-check log" hourly table from ODS and synchronizes it to the DWD layer.

**Business Background and Objective**: API-SDK cloud-check writes a raw log to ODS every hour (containing the caller's phone number `phonenum`/`phonenum_reg`, device IMEI/GUID/IMSI, calling IP (integer), query type, etc.). The downstream DWD layer wants: (1) filter out dirty data (empty phone number or IP, out-of-range IP); (2) use an IP library UDF to reverse-lookup country/province/city/district; (3) rename several fields and CAST them to the downstream schema.

**Input Table**:
- `internal_platform_db.case16_ods_comm_security_user_api_query_mgr_hi_v3`

**Data Range and Filter Conditions (business description)**:
  - `ds` = current day partition; for this run, `ds` is fixed as `'20260513'` (original `sys.argv[1]` sandboxed and hardcoded).
  - `phonenum_reg IS NOT NULL`
  - `phonenum IS NOT NULL`
  - `ip IS NOT NULL`
  - `ip BETWEEN 0 AND 4294967295` (valid uint32 range, to avoid dirty data)

**Table Join Relationships**: (No joins, single-table cleansing.)

**Aggregation and Computation Rules (must be reflected in the SQL)**:
  1. Column renaming / CAST:
     - `phonenum_reg` → `phone_num`
     - `phonenum` → `phone_num_original`
     - `imei` → `imei_reg` (retain the original `imei` as well)
     - `time` → `log_time`
     - `query_type` / `calltype` / `phonetype` / `src` all CAST AS STRING
  2. Derived IP reverse-lookup columns `ip_country` / `ip_province` / `ip_city` / `ip_district`:
     - The original task reverse-looks up via the `get_ip_info(ip)[0..3]` UDF; this UDF's jar (`hdfs://cluster-alpha/...`) is unavailable in the sandbox, so the sandbox version replaces it with constants `lit('CN')` / `lit('default_province')` / `lit('default_city')` / `lit('default_district')`.
     - The original task also has a pure Python `int_to_ip_string` UDF (using only socket+struct, available in the sandbox), but since the output columns have been replaced with `lit`, it is no longer referenced and is removed to keep the GT clean.

**Output Requirements**:
  - Target table: `internal_platform_db.case16_dwd_comm_security_user_api_query_mgr_hi_v3`
  - Output table schema (17 columns, in this order, with the following semantics):
    1. `phone_num` STRING: normalized phone number
    2. `phone_num_original` STRING: original phone number
    3. `imei_reg` STRING: normalized IMEI (i.e., `imei`)
    4. `imei` STRING: original IMEI
    5. `guid` STRING, 6. `imsi` STRING: device identifiers
    7. `log_time` STRING: log time (i.e., `time`)
    8. `ip` BIGINT: query IP (integer)
    9. `query_type` STRING, 10. `calltype` STRING, 11. `phonetype` STRING, 12. `src` STRING: query categories
    13. `ip_country` STRING: country (sandbox constant `'CN'`)
    14. `ip_province` STRING: province (sandbox constant `'default_province'`)
    15. `ip_city` STRING: city (sandbox constant `'default_city'`)
    16. `ip_district` STRING: district (sandbox constant `'default_district'`)
    17. `ds` BIGINT: business date (originally a partition column, sandboxed as a regular column)
  - Write strategy: `INSERT OVERWRITE TABLE`, using an explicit column name list (17 columns).
  - Sandbox note: `PARTITION(ds={ds})` is changed to a regular column + `WHERE` filter; the `get_ip_info` UDF is sandboxed as `lit` constants; `addJar` + `registerJavaFunction("get_ip_info", ...)` are removed; `ArrayType` / `StringType` imports are removed (no longer needed).
  - If the target table does not exist, first create it using standard Hive format (ORC storage), then write the data.

# data-mocker rewrite-gt
# source: v7 xlsx row 17 (case16)
# task_name: t_dwd_comm_security_user_api_query_mgr_hi_copilot
# rewrite rules:
#   1) internal_platform_db.t_ods_comm_security_user_api_query_mgr_hi_copilot
#       -> internal_platform_db.case16_ods_comm_security_user_api_query_mgr_hi_v3
#   2) internal_platform_db.t_dwd_comm_security_user_api_query_mgr_hi_copilot
#       -> internal_platform_db.case16_dwd_comm_security_user_api_query_mgr_hi_v3
#   3) PARTITION(ds={ds}) 删除；output 表 ds 改为普通列，SELECT 末尾补 ds
#      WHERE ds = {ds} 保留（普通列过滤）
#      INSERT 用显式列名避免列序错位
#   4) sys.argv[1] -> 硬编码 '20260513'
#   5) get_ip_info(...)[0..3] UDF -> lit('CN'/'default'/'default'/'default')
#      (原 jar hdfs://cluster-alpha/... 在沙箱不可用，同 case15 OpenidCrypt 处理)
#   6) 移除 addJar + registerJavaFunction("get_ip_info", ...)
#   7) int_to_ip_string Python UDF 保留（纯 socket+struct 沙箱可用），
#      但因为输出列已 lit 替换，实际不再被引用 → 一并删除以保持 GT 干净
#   8) 删除 ArrayType/StringType import（已不再需要）
# ---
import sys
from pyspark.sql import SparkSession

"""

脚本说明：API-SDK云查（lit 替换 IP 反查 UDF 版）

"""

# 生成和配置spark实例
def get_spark_cli():
    spark = SparkSession.builder \
        .enableHiveSupport() \
        .config("spark.sql.storeAssignmentPolicy", "LEGACY") \
        .getOrCreate()
    return spark

# 生成spark sql语句
def get_sql_str():
    ds = '20260513'  # was: sys.argv[1]
    sql = f"""
INSERT OVERWRITE TABLE internal_platform_db.case16_dwd_comm_security_user_api_query_mgr_hi_v3
(phone_num, phone_num_original, imei_reg, imei, guid, imsi, log_time, ip, query_type, calltype, phonetype, src, ip_country, ip_province, ip_city, ip_district, ds)
SELECT
    phonenum_reg AS phone_num,
    phonenum AS phone_num_original,
    imei AS imei_reg,
    imei,
    guid,
    imsi,
    time AS log_time,
    ip,
    CAST(query_type AS STRING) AS query_type,
    CAST(calltype AS STRING) AS calltype,
    CAST(phonetype AS STRING) AS phonetype,
    CAST(src AS STRING) AS src,
    'CN' AS ip_country,
    'default_province' AS ip_province,
    'default_city' AS ip_city,
    'default_district' AS ip_district,
    ds
FROM internal_platform_db.case16_ods_comm_security_user_api_query_mgr_hi_v3
WHERE ds = {ds}
  AND phonenum_reg IS NOT NULL
  AND phonenum IS NOT NULL
  AND ip IS NOT NULL
  AND ip BETWEEN 0 AND 4294967295
"""
    print(f"sql: {sql}")
    return sql

# 执行
if __name__ == "__main__":
    spark = get_spark_cli()

    # 确保输出表以正确 schema 存在
    spark.sql("DROP TABLE IF EXISTS internal_platform_db.case16_dwd_comm_security_user_api_query_mgr_hi_v3")
    spark.sql('''
CREATE TABLE internal_platform_db.case16_dwd_comm_security_user_api_query_mgr_hi_v3 (
    `phone_num` STRING,
    `phone_num_original` STRING,
    `imei_reg` STRING,
    `imei` STRING,
    `guid` STRING,
    `imsi` STRING,
    `log_time` STRING,
    `ip` BIGINT,
    `query_type` STRING,
    `calltype` STRING,
    `phonetype` STRING,
    `src` STRING,
    `ip_country` STRING,
    `ip_province` STRING,
    `ip_city` STRING,
    `ip_district` STRING,
    `ds` BIGINT
) STORED AS ORCFILE
''')

    SQL = get_sql_str()
    spark.sql(SQL)


Anti-Spam H5 Work Order Log Daily Detail Sync

Prompt:
## Prompt
I need you to generate a PySpark script that synchronizes the "Anti-Spam H5 Work Order Log" from ODS to the DWD detail table with full-field passthrough, and adds a run date column.

**Business Background and Objective**: The anti-spam H5 work order service writes each appeal/report operation log to the raw `log_80001099` table (48 business fields + `databus_imp_date` partition). The downstream DWD wants to transport the full-field detail of the current day's partition every day, and additionally add a column `imp_date` (run date, YYYYMMDD) for convenient subsequent filtering by run date. The task has no aggregation and no joins — pure detail transport + one derived column.

**Input Table**:
- `internal_platform_db.caseR18_ods_log_80001099_v3`

**Data Range and Filter Conditions (business description)**:
  - The original task in production reads from the full table (without `imp_date` filter); the sandbox v3 table is loaded by `ds` for the corresponding date, so the SELECT has no WHERE clause and reads the full table.
  - The run date `ds` is fixed as `'20260513'` and will be written as the new column `imp_date`.

**Table Join Relationships**: (No joins, single-table passthrough.)

**Aggregation and Computation Rules (must be reflected in the SQL)**:
  - Derived column `imp_date = '20260513'` (run date constant), placed as the first column in the SELECT.
  - The following 49 columns are passed through as-is: `databus_imp_date` + 48 business fields, in the same order as the DDL.
  - Use `INSERT OVERWRITE TABLE` + explicit column name list (50 columns) to avoid column order misalignment.

**Output Requirements**:
  - Target table: `internal_platform_db.caseR18_dwd_log_80001099_daily_v3`
  - Output table schema (50 columns, in this order, with the following semantics; the first 2 columns are run/original partition date, the remaining 48 are business fields):
    1. `imp_date` STRING: run date YYYYMMDD = `'20260513'`
    2. `databus_imp_date` STRING: original partition date (passed through from input)
    3. `logid` BIGINT, 4. `svrtime` STRING, 5. `svrip` STRING, 6. `module` STRING
    7. `form_id` STRING, 8. `form_type` BIGINT, 9. `action` BIGINT, 10. `retcode` BIGINT
    11. `vid` BIGINT, 12. `corpid` BIGINT, 13. `gid` BIGINT
    14. `appeal_kind` BIGINT (appeal problem type), 15. `report_kind` BIGINT (report problem type), 16. `fraud_kind` BIGINT (fraud type), 17. `loss_amount` BIGINT (loss amount)
    18. `suspect_vid` BIGINT, 19. `suspect_corpid` BIGINT
    20. `order_id` STRING (work order number), 21. `order_result` BIGINT, 22. `order_source` BIGINT, 23. `order_action` BIGINT
    24. `match_rule` STRING, 25. `roomid` BIGINT (problematic group number), 26. `create_vid` BIGINT, 27. `openid` STRING
    28. `isfromwx` BIGINT (interception party ww/wx), 29. `spamtype` STRING, 30. `auto_finish_order_rule` STRING
    31. `action_time` BIGINT (action occurrence time), 32. `industry_name` STRING, 33. `second_industry_name` STRING, 34. `is_ka` BIGINT
    35. `create_time` STRING, 36. `urgent_time` STRING, 37. `result_time` STRING, 38. `reopen_time` BIGINT
    39. `block_uin` BIGINT, 40. `block_user_id_type` BIGINT, 41. `version` STRING (v1/v2 differentiation)
    42. `h5_source` BIGINT (0: frontline customer service, 1: user self-service), 43. `h5_platform` BIGINT (1: BizComm, 2: PlatformW), 44. `template_type` BIGINT
    45. `current_owner` STRING, 46. `oid` BIGINT (appeal work order numeric ID), 47. `reason` STRING (QA closing evidence reason, base64)
    48. `order_finish_type` STRING, 49. `reply_content` STRING (scripted reply, base64), 50. `match_rule_scene_id` BIGINT
  - Write strategy: `INSERT OVERWRITE TABLE` + explicit 50-column name list.
  - Sandbox note: The original task is encapsulated in SparkBase (SparkJob class), using the `self.load_dw_data().createOrReplaceTempView()` + `self.save_dw_data()` chain; the sandbox version is rewritten as a top-level SparkSession + a single `INSERT OVERWRITE TABLE ... SELECT`.
  - If the target table does not exist, first create it using standard Hive format (ORC storage), then write the data.

# caseR18 rewrite-gt
# source: data/v7_gt_codes/case_row19_case_0011.py (v7 row 18, manifest_row 19)
# task_name: t_dwd_ww_log_80001099_daily_copilot
# rewrite rules:
#   1) internal_platform_db.log_80001099_copilot
#       -> internal_platform_db.caseR18_ods_log_80001099_v3
#   2) internal_platform_db.dwd_ww_log_80001099_daily_copilot
#       -> internal_platform_db.caseR18_dwd_log_80001099_daily_v3
#   3) 删除 dw_spark_base_python3.SparkBase 框架封装：
#      - SparkJob(SparkBase) class -> 直接顶层 SparkSession
#      - self.load_dw_data(db, tbl).createOrReplaceTempView() -> 直接在 sql 里写全名
#      - self.save_dw_data(df, db_name, tbl_name) -> spark.sql('INSERT OVERWRITE TABLE ...')
#      - self.statdate -> 硬编码 '20260513'
#      - self.spark -> 顶层 spark 变量
#   4) INSERT 用显式列名避免列序错位（50 列：imp_date + databus_imp_date + 48 业务列）
#   5) 沙箱 v3 表无分区，databus_imp_date 在输入表中已沙箱化为普通列
# ---
from __future__ import print_function

import sys
from pyspark.sql import SparkSession

if __name__ == "__main__":
    ds = '20260513'

    spark = (
        SparkSession.builder
        .appName("caseR18_dwd_log_80001099_daily")
        .enableHiveSupport()
        .getOrCreate()
    )

    # 确保输出表以正确 schema 存在
    spark.sql("DROP TABLE IF EXISTS internal_platform_db.caseR18_dwd_log_80001099_daily_v3")
    spark.sql('''
CREATE TABLE internal_platform_db.caseR18_dwd_log_80001099_daily_v3 (
    `imp_date` STRING COMMENT '运行日期(YYYYMMDD)',
    `databus_imp_date` STRING COMMENT '原分区日期(从输入透传)',
    `logid` BIGINT COMMENT 'logid',
    `svrtime` STRING COMMENT 'svrtime',
    `svrip` STRING COMMENT 'svrip',
    `module` STRING COMMENT 'module',
    `form_id` STRING COMMENT '表单唯一码',
    `form_type` BIGINT COMMENT '表单类型',
    `action` BIGINT COMMENT '动作类型',
    `retcode` BIGINT COMMENT '返回值',
    `vid` BIGINT COMMENT '用户帐号',
    `corpid` BIGINT COMMENT '用户公司账号',
    `gid` BIGINT COMMENT '用户gid',
    `appeal_kind` BIGINT COMMENT '申诉问题类型',
    `report_kind` BIGINT COMMENT '举报问题类型',
    `fraud_kind` BIGINT COMMENT '诈骗类型',
    `loss_amount` BIGINT COMMENT '损失金额',
    `suspect_vid` BIGINT COMMENT '举报嫌疑vid',
    `suspect_corpid` BIGINT COMMENT '举报嫌疑corpid',
    `order_id` STRING COMMENT '工单编号',
    `order_result` BIGINT COMMENT '工单结果',
    `order_source` BIGINT COMMENT '工单来源类型',
    `order_action` BIGINT COMMENT '工单动作',
    `match_rule` STRING COMMENT '工单相关的拦截规则',
    `roomid` BIGINT COMMENT '出问题的群号',
    `create_vid` BIGINT COMMENT '工单创建者vid',
    `openid` STRING COMMENT 'openid',
    `isfromwx` BIGINT COMMENT '拦截方:ww or wx',
    `spamtype` STRING COMMENT '审核定性',
    `auto_finish_order_rule` STRING COMMENT '自动结单策略',
    `action_time` BIGINT COMMENT '行为发生时间',
    `industry_name` STRING COMMENT '一级行业名称',
    `second_industry_name` STRING COMMENT '二级行业名称',
    `is_ka` BIGINT COMMENT 'corpid 是否为 ka',
    `create_time` STRING COMMENT '建单时间',
    `urgent_time` STRING COMMENT '催单时间',
    `result_time` STRING COMMENT '结单时间',
    `reopen_time` BIGINT COMMENT '重开时间(时间戳秒)',
    `block_uin` BIGINT COMMENT '实际拦截UIN',
    `block_user_id_type` BIGINT COMMENT '拦截UIN类型',
    `version` STRING COMMENT 'v1/v2 区分',
    `h5_source` BIGINT COMMENT '0:一线客服 1:用户自助',
    `h5_platform` BIGINT COMMENT '1:企业通讯平台B 2:平台W',
    `template_type` BIGINT COMMENT '在线申诉模板类型',
    `current_owner` STRING COMMENT '当前工单操作人',
    `oid` BIGINT COMMENT '申诉工单数字ID',
    `reason` STRING COMMENT 'QA结单证据原因(base64)',
    `order_finish_type` STRING COMMENT '工单结单类型',
    `reply_content` STRING COMMENT '话术回复(base64)',
    `match_rule_scene_id` BIGINT COMMENT '策略打击场景ID'
) STORED AS ORCFILE
''')

    # 反垃圾H5工单日志的每日明细同步
    sql = """
    INSERT OVERWRITE TABLE internal_platform_db.caseR18_dwd_log_80001099_daily_v3 (
        imp_date, databus_imp_date, logid, svrtime, svrip, module, form_id, form_type, action, retcode,
        vid, corpid, gid, appeal_kind, report_kind, fraud_kind, loss_amount, suspect_vid, suspect_corpid, order_id,
        order_result, order_source, order_action, match_rule, roomid, create_vid, openid, isfromwx, spamtype, auto_finish_order_rule,
        action_time, industry_name, second_industry_name, is_ka, create_time, urgent_time, result_time, reopen_time, block_uin, block_user_id_type,
        version, h5_source, h5_platform, template_type, current_owner, oid, reason, order_finish_type, reply_content, match_rule_scene_id
    )
    select '{0}' as imp_date,
        databus_imp_date,
        logid,
        svrtime,
        svrip,
        module,
        form_id,
        form_type,
        action,
        retcode,
        vid,
        corpid,
        gid,
        appeal_kind,
        report_kind,
        fraud_kind,
        loss_amount,
        suspect_vid,
        suspect_corpid,
        order_id,
        order_result,
        order_source,
        order_action,
        match_rule,
        roomid,
        create_vid,
        openid,
        isfromwx,
        spamtype,
        auto_finish_order_rule,
        action_time,
        industry_name,
        second_industry_name,
        is_ka,
        create_time,
        urgent_time,
        result_time,
        reopen_time,
        block_uin,
        block_user_id_type,
        version,
        h5_source,
        h5_platform,
        template_type,
        current_owner,
        oid,
        reason,
        order_finish_type,
        reply_content,
        match_rule_scene_id
    from internal_platform_db.caseR18_ods_log_80001099_v3
    """.format(ds)
    print("sql:")
    print(sql)
    spark.sql(sql)
    print("saved")

    spark.stop()
    sys.exit(0)


项目管理工具PM工单变更去重

Prompt:
## Prompt
我需要你生成一段 PySpark 代码，将 项目管理工具PM 工单变更明细表做去重处理后写入 DWV 层。

**业务背景**：上游 ODS 表 ods_pmtool_v_workitem_changes_df 按 databus_imp_date 分区存储 项目管理工具PM 工单的每次变更记录。同一个工单变更 ID 可能存在多条变更记录（如同一条 id 在不同时间点被多次修改），需要按 id 分组去重，保留最新的 workspace_id/workitem_id/workitem_type_id 等数值字段，以及最早的 creator/created/change_summary 等字符串字段，最终写入 DWV 汇总层供下游使用。

**输入表**：
- `internal_platform_db.ods_pmtool_v_workitem_changes_df_pyspark_245`

**数据范围**：只取 databus_imp_date = '20260513' 分区的数据。

**计算规则**：
1. 按 id 分组 GROUP BY
2. workspace_id、workitem_id、workitem_type_id 取 max（最新值）
3. creator、created、change_summary、comment、changes、entity_type、change_type 取 first（最早值）
4. _srcinstanceid、_srcdatabasename 输出空字符串

**输出要求**：
- 目标表：`internal_platform_db.dwv_pmtool_workitem_changes_df_pyspark_245`
- 分区：year='2026', month='05', day='13'
- 写入策略：INSERT OVERWRITE TABLE ... PARTITION(year='2026', month='05', day='13')
- 如果目标表不存在，请先按 Hive 标准建表（ORC 存储 + 分区），再写入数据。

#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""pyspark_010: 项目管理工具PM workitem changes DWV - GROUP BY dedup + partition write"""

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName('dwv_pmtool_workitem_changes_df') \
    .enableHiveSupport() \
    .getOrCreate()

ds = '20260513'
ds_year = '2026'
ds_month = '05'
ds_day = '13'

# 输出表 DDL
spark.sql("DROP TABLE IF EXISTS internal_platform_db.dwv_pmtool_workitem_changes_df_pyspark_245")
spark.sql('''
CREATE TABLE internal_platform_db.dwv_pmtool_workitem_changes_df_pyspark_245 (
  `id` STRING,
  `workspace_id` BIGINT,
  `workitem_id` BIGINT,
  `workitem_type_id` BIGINT,
  `creator` STRING,
  `created` STRING,
  `change_summary` STRING,
  `comment` STRING,
  `changes` STRING,
  `entity_type` STRING,
  `change_type` STRING,
  `_srcinstanceid` STRING,
  `_srcdatabasename` STRING
)
PARTITIONED BY (`year` STRING, `month` STRING, `day` STRING)
STORED AS ORC
''')

sql = '''
    WITH workitem_changes AS (
        SELECT * FROM internal_platform_db.ods_pmtool_v_workitem_changes_df_pyspark_245
        WHERE databus_imp_date = '20260513'
    )
    INSERT OVERWRITE TABLE internal_platform_db.dwv_pmtool_workitem_changes_df_pyspark_245
    PARTITION(year='2026', month='05', day='13')
    SELECT
        `id`, max(`workspace_id`) AS workspace_id,
        max(`workitem_id`) AS workitem_id,
        max(`workitem_type_id`) AS workitem_type_id,
        first(`creator`) AS creator,
        first(`created`) AS created,
        first(`change_summary`) AS change_summary,
        first(`comment`) AS comment,
        first(`changes`) AS changes,
        first(`entity_type`) AS entity_type,
        first(`change_type`) AS change_type,
        '' AS _srcinstanceid,
        '' AS _srcdatabasename
    FROM workitem_changes
    GROUP BY id
'''

print(sql)
result = spark.sql(sql)
print(result.show())
spark.stop()


Phone Number Encrypted Prefix Quality Score Statistics

Prompt:
## Prompt
I need you to generate a PySpark script that computes a phone number encrypted prefix quality score statistics table.

**Business Background and Objective**: The phone number quality score table stores the quality score (`maochi` field) for each phone number encrypted string. Operations needs to perform aggregation analysis on the first 7 characters of the encrypted prefix for each phone number, computing the average quality score, malicious ratio, and whitelist ratio under that prefix. The aggregation results are then backfilled onto each detail record to produce a wide table with prefix-level statistical dimensions.

**Input Table**:
- `internal_platform_db.t_dwd_phone_enc_quality_score_df_pyspark_216`

**Data Range and Filter Conditions**:
- Run date `ds` is fixed as `'20260513'`
- Filter out invalid records where `maochi = -9`

**Computation Rules**:
1. Use `substr(phone_enc, 1, 7)` to extract the phone number encrypted prefix `phone_enc_p7`
2. Group by `phone_enc_p7` and aggregate:
   - `prefix_1_7_score = floor(avg(maochi) * 10000)`: prefix average quality score (scaled by 10000)
   - `prefix_1_7_evil_score = floor(count(if(maochi >= -4 and maochi <= -2, 1, NULL)) * 10000 / count(1))`: prefix malicious ratio (in parts per ten thousand)
   - `prefix_1_7_white_score = floor(count(if(maochi >= 2 and maochi <= 4, 1, NULL)) * 10000 / count(1))`: prefix whitelist ratio (in parts per ten thousand)
3. LEFT JOIN to associate the prefix aggregation results back to the original detail records

**Output Requirements**:
- Target table: `internal_platform_db.t_dwd_phone_enc_prefix_quality_score_df_pyspark_216`
- Partition field: `p_date='20260513'`
- Output columns: `ds`, `phone_enc`, `quality_score`, `prefix_1_7_score`, `prefix_1_7_evil_score`, `prefix_1_7_white_score`
- Write strategy: First create the table (ORC storage), then use `INSERT OVERWRITE TABLE PARTITION(p_date=...)` to write

#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
pyspark_011: Phone enc prefix quality score
- Read t_dwd_phone_enc_quality_score_df, filter maochi != -9
- Use substr() to extract phone_enc prefix (first 7 chars) instead of Java UDF IdSubstr
- CTE aggregation per prefix: avg quality_score, evil ratio, white ratio
- LEFT JOIN back to original, output with ds='20260513'
"""

from pyspark.sql import SparkSession

spark = (
    SparkSession.builder
    .appName('pyspark_011_phone_enc_prefix_score')
    .enableHiveSupport()
    .getOrCreate()
)

TODAY = '20260513'

query = """
    with t_phone_enc_score as (
        select
            phone_enc,
            maochi,
            substr(phone_enc, 1, 7) as phone_enc_p7
        from internal_platform_db.t_dwd_phone_enc_quality_score_df_pyspark_216
        where ds = {today} and maochi != -9
    ), t_phone_enc_p7_score as (
        select
            phone_enc_p7,
            floor(avg(maochi) * 10000) as prefix_1_7_score,
            floor(count(if(maochi >= -4 and maochi <= -2, 1, NULL)) * 10000 / count(1)) as prefix_1_7_evil_score,
            floor(count(if(maochi >= 2 and maochi <= 4, 1, NULL)) * 10000 / count(1)) as prefix_1_7_white_score
        from
            t_phone_enc_score
        group by
            phone_enc_p7
    ), t_merge as (
        select
            phone_enc,
            maochi as quality_score,
            prefix_1_7_score,
            prefix_1_7_evil_score,
            prefix_1_7_white_score
        from
            t_phone_enc_score t1
        left join
            t_phone_enc_p7_score t2
        on
            t1.phone_enc_p7 = t2.phone_enc_p7
    )
    select {today} as ds, phone_enc, quality_score, prefix_1_7_score, prefix_1_7_evil_score, prefix_1_7_white_score from t_merge
""".format(today=TODAY)
print(f"query_sql: {query}")
df = spark.sql(query)
df.createOrReplaceTempView("tmp_result")

# Output table DDL
spark.sql('DROP TABLE IF EXISTS internal_platform_db.t_dwd_phone_enc_prefix_quality_score_df_pyspark_216')
spark.sql('''
CREATE TABLE internal_platform_db.t_dwd_phone_enc_prefix_quality_score_df_pyspark_216 (
    `ds` STRING,
    `phone_enc` STRING,
    `quality_score` INT,
    `prefix_1_7_score` BIGINT,
    `prefix_1_7_evil_score` BIGINT,
    `prefix_1_7_white_score` BIGINT
) PARTITIONED BY (`p_date` STRING)
STORED AS ORC
''')

# Write output via INSERT OVERWRITE with explicit partition
spark.sql("""
INSERT OVERWRITE TABLE internal_platform_db.t_dwd_phone_enc_prefix_quality_score_df_pyspark_216
PARTITION (p_date='{0}')
SELECT ds, phone_enc, quality_score, prefix_1_7_score, prefix_1_7_evil_score, prefix_1_7_white_score
FROM tmp_result
""".format(TODAY))

print("pyspark_011 done")

spark.stop()


网页解析结果Top500详情统计

Prompt:
## Prompt
我需要你生成一段 PySpark 代码，对网页解析结果进行质量统计，产出 Top500 站点的详情日报表。

**业务背景与目标**：每天都会对大量网页进行解析，解析结果记录在 `ai_engine_classify_parse_result_daily` 表中。安全部门需要按站点维度统计解析质量，包括成功/失败解析数、索引页占比、短内容占比、各错误步骤分布等，并与站点优先级信息关联，生成一张按日期和站点粒度的解析质量详情表。

**输入表**：
- `internal_platform_db.sec_app_hy_top_500_sites_tag_v1_pyspark_100`
- `internal_platform_db.ai_engine_classify_parse_result_daily_pyspark_100`

**数据范围与过滤条件**：
- 运行日期 today_str 固定为 '20260513'
- 数据范围：date_key 从 '20260508' 到 '20260513'（最近 5 天）

**计算规则**：
1. 从 parse_html JSON 字段中提取 content 字段（使用 from_json + schema_of_json）
2. 计算 content_length = length(content)
3. 按 (date_key, host, fld) 分组聚合：
   - 成功解析数 successful_parses、失败解析数 failed_parses、总解析数 total_parses
   - 索引页数 index_page_count（web_type='索引页'）、索引页比例 index_page_count_rate = index_page_count/total_parses
   - 短内容数 short_content_count（content_length < 50）、短内容比例 short_content_count_rate = short_content_count/successful_parses
   - 各错误步骤计数：error_step_1_count、error_step_2_count、error_step_3_count
4. LEFT JOIN 关联 priority_info：匹配条件为 source_fld = priority_info.fld OR source_host = priority_info.host
5. 去重：dropDuplicates(["date_key", "host", "fld", "priority", "successful_parses", "total_parses", "index_page_count"])

**输出要求**：
- 目标表：`internal_platform_db.ai_engine_classify_parse_result_daily_detail_pyspark_100`
- 写入策略：先建表（ORC 存储，16 列），再使用 df.write.mode("overwrite").insertInto() 写入
- 输出列：date_key, host, fld, is_host, priority, successful_parses, failed_parses, total_parses, index_page_count, index_page_count_rate, short_content_count, short_content_count_rate, error_step_1_count, error_step_2_count, error_step_3_count, error_info_list

#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
pyspark_012: AI search parse results top500 detail
- 3 input tables: priority_info, daily parse results
- Date range: hardcode today_str='20260513', five_days_ago='20260508'
- Parse content JSON from parse_html field using from_json
- Compute summary stats, index_page_ratio, content_length_ratio, error_reasons
- Join with priority_info on fld or host match
- Output via df.write.mode("overwrite").insertInto()
"""

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, count, when, length, sum as _sum, from_json, schema_of_json, lit
from pyspark.sql import functions as F

spark = (
    SparkSession.builder
    .appName('pyspark_012_ai_engine_parse_detail')
    .enableHiveSupport()
    .getOrCreate()
)

today_str = '20260513'
five_days_ago = '20260508'
print(today_str, five_days_ago)

# Read priority info
priority_info = spark.table("internal_platform_db.sec_app_hy_top_500_sites_tag_v1_pyspark_100").select(
    "host", "fld", "priority"
)

# Read data from past 5 days
df = spark.table("internal_platform_db.ai_engine_classify_parse_result_daily_pyspark_100").filter(
    (col("date_key") >= five_days_ago) & (col("date_key") <= today_str)
)

# Parse data
parsed_data = df.select(
    "date_key",
    col("host").alias("source_host"),
    when(col("is_error") == 0, 1).otherwise(0).alias("success"),
    when(col("is_error") == 1, 1).otherwise(0).alias("failure"),
    "web_type",
    from_json(col("parse_html"), schema_of_json('{"content": "string"}')).getField("content").alias("content"),
    "error_step",
    col("fld").alias("source_fld")
)

# Compute content length
parsed_data = parsed_data.withColumn("content_length", length(col("content")))

# Summary stats
summary = parsed_data.groupBy("date_key", "source_host", "source_fld").agg(
    _sum("success").alias("successful_parses"),
    _sum("failure").alias("failed_parses"),
    count("*").alias("total_parses")
)

# Index page ratio
index_page_ratio = parsed_data.groupBy("date_key", "source_host", "source_fld").agg(
    _sum(when(col("web_type") == "索引页", 1).otherwise(0)).alias("index_page_count")
)

# Content length ratio
content_length_ratio = parsed_data.filter(col("success") == 1).groupBy("date_key", "source_host", "source_fld").agg(
    _sum(when(col("content_length") < 50, 1).otherwise(0)).alias("short_content_count")
)

# Error reasons
error_reasons = parsed_data.filter(col("failure") == 1).groupBy("date_key", "source_host", "source_fld").agg(
    _sum(when(col("error_step") == 1, 1).otherwise(0)).alias("error_step_1_count"),
    _sum(when(col("error_step") == 2, 1).otherwise(0)).alias("error_step_2_count"),
    _sum(when(col("error_step") == 3, 1).otherwise(0)).alias("error_step_3_count"),
    _sum(when(col("error_step").isNotNull(), 1).otherwise(0)).alias("total_error_steps")
)

# Merge all aggregations
final_result = summary.join(index_page_ratio, ["date_key", "source_host", "source_fld"], "left") \
    .join(content_length_ratio, ["date_key", "source_host", "source_fld"], "left") \
    .join(error_reasons, ["date_key", "source_host", "source_fld"], "left")

# Calculate ratios
final_result = final_result.withColumn(
    "index_page_count_rate",
    col("index_page_count") / col("total_parses")
).withColumn(
    "short_content_count_rate",
    col("short_content_count") / col("successful_parses")
)

# Add placeholder columns
final_result = final_result.withColumn("error_info_list", lit("default_value"))
final_result = final_result.withColumn("is_host", lit("default_value"))

# Join with priority info
final_result = final_result.join(
    priority_info,
    (final_result.source_fld == priority_info.fld) |
    (final_result.source_host == priority_info.host),
    "left"
)

# Select final columns
final_result = final_result.select(
    "date_key",
    F.col("source_host").alias("host"),
    F.col("source_fld").alias("fld"),
    "is_host",
    "priority",
    "successful_parses",
    "failed_parses",
    "total_parses",
    "index_page_count",
    "index_page_count_rate",
    "short_content_count",
    "short_content_count_rate",
    "error_step_1_count",
    "error_step_2_count",
    "error_step_3_count",
    "error_info_list"
)

final_result = final_result.dropDuplicates(["date_key", "host", "fld", "priority", "successful_parses", "total_parses", "index_page_count"])

# Output table DDL
spark.sql('DROP TABLE IF EXISTS internal_platform_db.ai_engine_classify_parse_result_daily_detail_pyspark_100')
spark.sql('''
CREATE TABLE internal_platform_db.ai_engine_classify_parse_result_daily_detail_pyspark_100 (
    `date_key` STRING,
    `host` STRING,
    `fld` STRING,
    `is_host` STRING,
    `priority` STRING,
    `successful_parses` STRING,
    `failed_parses` STRING,
    `total_parses` STRING,
    `index_page_count` STRING,
    `index_page_count_rate` STRING,
    `short_content_count` STRING,
    `short_content_count_rate` STRING,
    `error_step_1_count` STRING,
    `error_step_2_count` STRING,
    `error_step_3_count` STRING,
    `error_info_list` STRING
)
STORED AS ORC
''')

# Write output
final_result.write.mode("overwrite").insertInto("internal_platform_db.ai_engine_classify_parse_result_daily_detail_pyspark_100")

print("pyspark_012 done")

spark.stop()


银行XURL安全关系关联

Prompt:
## Prompt
我需要你生成一段 PySpark 代码，从平台 URL 安全关系中提取银行X相关的贷中和意愿用户样本。

**业务背景与目标**：安全部门维护了一份平台 guid 与 URL 的关系表，记录了用户访问过的 URL 及其 cgi 信息。风控业务需要从这份关系表中筛选出访问过银行X（bankx.com.cn）特定页面的用户，按贷款阶段分为"贷中用户"和"意愿用户"两类，输出到贷款用户样本表中，供下游风控模型使用。

**输入表**：
- `internal_platform_db.t_dwd_urlsafe_rela_guid_url_di_pyspark_101`

**数据范围与过滤条件**：
- 运行日期 ds 固定为 20260513（BIGINT 类型）
- 域名过滤：domain = 'bankx.com.cn'
- 站点过滤：site = 'm1.bankx.com.cn'

**计算规则**：
1. **贷中用户识别**：cgi 包含 `http://m1.bankx.com.cn/BankX_MBSERVER/NEW/APP/MOBILE-BANK-H5/LOANS/WXMAIN/LOAN-REPAY-PLAN`，loan_stage 标记为 'in-loan'
2. **意愿用户识别**：cgi 包含以下任一 URL 模式（18 个 will_cgis），loan_stage 标记为 'will'：
   - LOANS/MAIN/XWMINI, LOANS/AUTHORIZATION, LOANS/DATA-SELF-ATTESTATION-CHECK, LOANS/DATA-SELF-ATTESTATION-EDIT, LOANS/MAIN/XWMINI-RESULT
   - FEIMA 系列路径
3. 两类用户分别选取字段：uid（来自 guid）, loan_type='unknown', city='unknown', industry='unknown', entity_name='unknown', legal_person='unknown', loan_stage, details（来自 url）
4. 各用户组内按 uid 去重
5. Union 合并后，整体按 uid 去重
6. 写入时同时指定分区值：ds=20260513, data_source='social', id_type='guid'

**输出要求**：
- 目标表：`internal_platform_db.t_rta_loan_url_samples_pyspark_101`
- 分区字段：ds BIGINT, data_source STRING, id_type STRING
- 写入策略：先建表（ORC 存储，8 个非分区列 + 3 个分区列），再 INSERT OVERWRITE TABLE PARTITION(ds=..., data_source=..., id_type=...) SELECT * FROM tmp_view

#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
pyspark_013: URL safety relation join (银行X)
- 2 input tables: t_dwd_urlsafe_rela_guid_url_di, t_rta_loan_url_samples
- Hardcode: ds='20260513', target_domain='bankx.com.cn', data_source='social'
- Define in_loan_cgi and will_cgis URL patterns for BankX bank
- Build in-loan users: match on in_loan_cgi, select uid + unknown placeholders
- Build will users: match on will_cgis, select uid + unknown placeholders
- Union, dropDuplicates, insert overwrite into output table partitioned by (ds, data_source, id_type)
"""

from pyspark.sql import SparkSession
from pyspark.sql import functions as F

spark = SparkSession.builder.appName("pyspark_013_bankx").enableHiveSupport().getOrCreate()

ds = '20260513'
target_domain = 'bankx.com.cn'
data_source = 'social'
dst_tb = "internal_platform_db.t_rta_loan_url_samples_pyspark_101"
target_dfs = []

df = spark.sql(
    f"select guid as uid, url, site, cgi from internal_platform_db.t_dwd_urlsafe_rela_guid_url_di_pyspark_101 where ds={ds} and domain='{target_domain}'"
).cache()

will_cgis = [
    "http://m1.bankx.com.cn/BankX_MBSERVER/NEW/APP/MOBILE-BANK-H5/LOANS/MAIN/XWMINI",
    "http://m1.bankx.com.cn/BankX_MBSERVER/NEW/APP/MOBILE-BANK-H5/FEIMA/4/786_3ZC6V00SCTS/KVP64MGHE6",
    "http://m1.bankx.com.cn/BankX_MBSERVER/NEW/APP/MOBILE-BANK-NG/LOAN/OPEN-MINI-PROGRAM",
    "http://m1.bankx.com.cn/BankX_MBSERVER/NEW/APP/MOBILE-BANK-H5/FEIMA/4/786_3ZC6V00SCTS/EH29NY7VX6O",
    "http://m1.bankx.com.cn/BankX_MBSERVER/NEW/APP/MOBILE-BANK-H5/LOANS/AUTHORIZATION",
    "http://m1.bankx.com.cn/BankX_MBSERVER/NEW/APP/MOBILE-BANK-H5/FEIMA/404/5297_ID8OJSK0CA/GO6WAHCH1Q8",
    "http://m1.bankx.com.cn/BankX_MBSERVER/NEW/APP/MOBILE-BANK-H5/FEIMA/4/2030_X896J30LP0M/32Z34ISSCYM",
    "http://m1.bankx.com.cn/BankX_MBSERVER/NEW/APP/MOBILE-BANK-H5/FEIMA/404/4995_PQD15LFBWHK/J4AUCBG4T9Q",
    "http://m1.bankx.com.cn/BankX_MBSERVER/NEW/APP/MOBILE-BANK-H5/LOANS/DATA-SELF-ATTESTATION-CHECK",
    "http://m1.bankx.com.cn/BankX_MBSERVER/NEW/APP/MOBILE-BANK-H5/FEIMA/4/782_GCZRA2IPYHI/M5L9X3HI1YD",
    "http://m1.bankx.com.cn/BankX_MBSERVER/NEW/APP/MOBILE-BANK-H5/FEIMA/4/1488_AIX7S10A5D/YXES5G44DDI",
    "http://m1.bankx.com.cn/BankX_MBSERVER/NEW/APP/MOBILE-BANK-H5/LOANS/DATA-SELF-ATTESTATION-EDIT",
    "http://m1.bankx.com.cn/BankX_MBSERVER/NEW/APP/MOBILE-BANK-H5/FEIMA/4/672_C1HLFOASYSA/3RVSG4RH23C",
    "http://m1.bankx.com.cn/BankX_MBSERVER/NEW/APP/MOBILE-BANK-H5/LOANS/MAIN/XWMINI-RESULT",
    "http://m1.bankx.com.cn/BankX_MBSERVER/NEW/APP/MOBILE-BANK-H5/FEIMA/4/1488_AIX7S10A5D/2NS3G56ZR4P",
    "http://m1.bankx.com.cn/BankX_MBSERVER/NEW/APP/MOBILE-BANK-H5/FEIMA/4/2030_X896J30LP0M/2NS3G56ZR4P",
    "http://m1.bankx.com.cn/BankX_MBSERVER/NEW/APP/MOBILE-BANK-H5/FEIMA/404/4995_PQD15LFBWHK/2NS3G56ZR4P",
    "http://m1.bankx.com.cn/BankX_MBSERVER/NEW/APP/MOBILE-BANK-H5/FEIMA/4/1792_ZHAWK9C9K1/LXTCEQ1UIO"
]

in_loan_cgi = ["http://m1.bankx.com.cn/BankX_MBSERVER/NEW/APP/MOBILE-BANK-H5/LOANS/WXMAIN/LOAN-REPAY-PLAN"]

# Build in-loan users
inloan_condition = None
for url in in_loan_cgi:
    if inloan_condition is None:
        inloan_condition = F.col("cgi").contains(url)
    else:
        inloan_condition = inloan_condition | F.col("cgi").contains(url)

target_dfs.append(
    df.filter(F.col("site") == "m1.bankx.com.cn")
    .filter(inloan_condition)
    .withColumn("loan_stage", F.lit('in-loan'))
    .selectExpr(
        "uid",
        "'unknown' as loan_type",
        "'unknown' as city",
        "'unknown' as industry",
        "'unknown' as entity_name",
        "'unknown' as legal_person",
        "loan_stage",
        "url as details",
    ).dropDuplicates(["uid"])
)

# Build will users
will_condition = None
for url in will_cgis:
    if will_condition is None:
        will_condition = F.col("cgi").contains(url)
    else:
        will_condition = will_condition | F.col("cgi").contains(url)

target_dfs.append(
    df.filter(F.col("site") == "m1.bankx.com.cn")
    .filter(will_condition)
    .withColumn("loan_stage", F.lit('will'))
    .selectExpr(
        "uid",
        "'unknown' as loan_type",
        "'unknown' as city",
        "'unknown' as industry",
        "'unknown' as entity_name",
        "'unknown' as legal_person",
        "loan_stage",
        "url as details",
    ).dropDuplicates(["uid"])
)

# Union all
result_df = target_dfs[0]
for df_item in target_dfs[1:]:
    result_df = result_df.union(df_item)

result_df = result_df.cache()
result_df = result_df.dropDuplicates(["uid"])
print(f"len of result_df: {result_df.count()}")

result_df.createOrReplaceTempView("tmp_df")

# Output table DDL
spark.sql('DROP TABLE IF EXISTS internal_platform_db.t_rta_loan_url_samples_pyspark_101')
spark.sql('''
CREATE TABLE internal_platform_db.t_rta_loan_url_samples_pyspark_101 (
    `uid` STRING,
    `loan_type` STRING,
    `city` STRING,
    `industry` STRING,
    `entity_name` STRING,
    `legal_person` STRING,
    `loan_stage` STRING,
    `details` STRING
) PARTITIONED BY (`ds` BIGINT, `data_source` STRING, `id_type` STRING)
STORED AS ORC
''')

sql = f"insert overwrite table {dst_tb} partition(ds={ds}, data_source='{data_source}', id_type='guid') select * from tmp_df"
print(sql)
spark.sql(sql)
print("pyspark_013 done")

spark.stop()


过去半年在线过的实验计划信息查询

Prompt:
## Prompt
我需要你生成一段 PySpark 代码，查询过去半年内上线过的实验计划信息，取每个实验分组的最新一条记录。

**业务背景与目标**：实验平台按小时维度存储了实验计划的上线记录（fdate, exp_group_id, trigger_list）。运营需要从过去 180 天（半年）的历史数据中，按实验分组 exp_group_id 维度取最新的一条记录，输出到结果表中，用于后续的实验分析。

**输入表**：
- `internal_platform_db.exp_hours_input_pyspark_102`

**数据范围与过滤条件**：
- 运行日期 cur_date 固定为 '20260513'
- 历史天数 history_days 固定为 180
- 读取的分区范围：从 p_2026051300 往前推 180 * 24 = 4320 个小时的所有分区
- 分区格式：p_YYYYMMDDHH

**计算规则**：
1. 生成历史分区列表：使用 Python time 模块，从 cur_date 当天 00:00:00 开始，依次减去 1 小时，共生成 history_days * 24 个分区名（如 p_2026051300, p_2026051223, p_2026051222, ...）
2. 使用 WHERE p_hour IN (...) 过滤读取所有历史分区数据
3. 使用窗口函数 row_number() OVER (PARTITION BY exp_group_id ORDER BY fdate DESC) 计算 date_rank_no
4. 过滤 date_rank_no = 1，取每个实验分组的最新记录
5. 添加 fdate = lit(cur_date) 列

**输出要求**：
- 目标表：`internal_platform_db.exp_hours_output_pyspark_102`
- 写入策略：先建表（ORC 存储，5 列），再 INSERT OVERWRITE TABLE 写入
- 输出列：exp_date, exp_group_id, trigger_list, date_rank_no, fdate

#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
pyspark_014: Experiment plans past half year query
- 1 input table: exp_hours_input_pyspark_102 (fdate, exp_group_id, trigger_list, partitioned by p_hour)
- Read all partitions from p_2026051300 back to p_2026051300 - 24*180 hours (half year)
- Hardcode: cur_date='20260513', history_days=180
- Generate list of hour partitions to read
- Window function: rank() over (partition by exp_group_id order by fdate desc), keep rank=1
- Add fdate = lit(cur_date)
- Output table: exp_hours_output_pyspark_102 via INSERT OVERWRITE TABLE
"""

import time
from pyspark.sql import SparkSession
from pyspark.sql import functions as F
from pyspark.sql.window import Window

cur_date = '20260513'
history_days = 180

def get_history_hours(cur_date, history_days):
    cur_hour = cur_date + " 00:00:00"
    cur_ts = int(time.mktime(time.strptime(cur_hour, "%Y%m%d %H:%M:%S")))
    history_hours = []
    history_hour_num = history_days * 24
    for i in range(history_hour_num):
        history_hour = time.strftime("%Y%m%d%H", time.localtime(cur_ts - i * 3600))
        history_hours.append("p_" + history_hour)
    return history_hours

partition_list = get_history_hours(cur_date, history_days)
print("cur_date: " + cur_date)
print("history_days: " + str(history_days))
print("partition_list length: " + str(len(partition_list)))

spark = SparkSession.builder \
    .appName('pyspark_014_exp_plans') \
    .enableHiveSupport() \
    .getOrCreate()

# Read all history partitions using a single SQL with OR conditions on partition values
# Build partition filter string
partition_filters = ", ".join(["'%s'" % p for p in partition_list])
spark.sql("""
    SELECT fdate, exp_group_id, trigger_list
    FROM internal_platform_db.exp_hours_input_pyspark_102
    WHERE p_hour IN ({0})
""".format(partition_filters)).createOrReplaceTempView("t_exp_hours")

sql = '''
    select
        fdate as exp_date,
        exp_group_id,
        trigger_list
    from
        t_exp_hours
'''
print("sql: " + sql)
exp_hours_df = spark.sql(sql)
exp_hours_df.printSchema()

# Window function: rank by exp_date desc per exp_group_id
exp_partition_window = Window.partitionBy(["exp_group_id"]).orderBy(exp_hours_df["exp_date"].desc())
exp_hours_df = exp_hours_df.withColumn("date_rank_no", F.row_number().over(exp_partition_window))
exp_hours_df = exp_hours_df.filter(exp_hours_df.date_rank_no == 1)
exp_hours_df = exp_hours_df.withColumn("fdate", F.lit(cur_date))

# Output table DDL
spark.sql('DROP TABLE IF EXISTS internal_platform_db.exp_hours_output_pyspark_102')
spark.sql('''
CREATE TABLE internal_platform_db.exp_hours_output_pyspark_102 (
    `exp_date` STRING,
    `exp_group_id` STRING,
    `trigger_list` STRING,
    `date_rank_no` INT,
    `fdate` STRING
)
STORED AS ORC
''')

# Write output via INSERT OVERWRITE TABLE
exp_hours_df.createOrReplaceTempView("tmp_result")
spark.sql("""
INSERT OVERWRITE TABLE internal_platform_db.exp_hours_output_pyspark_102
SELECT exp_date, exp_group_id, trigger_list, date_rank_no, fdate
FROM tmp_result
""")

print("pyspark_014 done")
spark.stop()


用户标签同步ETL

Prompt:
## Prompt
我需要你生成一段 PySpark 代码，完成"用户标签同步ETL"的数据加工任务。

**业务背景与目标**：安全平台每天从 event_log 日志中采集用户设备与标签数据（t_sh_event_log_v2_xab00014231），数据中包含手机号、IMEI、IMSI、AndroidID 等敏感信息，这些字段在原始日志中已经过在线加密（OnlineCrypt）存储为加密字段。本任务需要将加密字段解密后，使用加密算法重新加密，同时对数据类型字段做 INT 转换，最终写入下游用户标签同步表（t_ods_comm_security_user_tag_sync_hi）供安全分析使用。

**输入表**：
- `internal_platform_db.t_sh_event_log_v2_xab00014231_pyspark_254`

**数据范围与过滤条件（业务说法）**：
- 运行日期 ds 固定为 '20260513'（沙箱化日期，原任务由调度框架传入）。
- 从输入表中读取 ds = '20260513' 分区的全量数据，不做额外过滤。

**计算规则（需体现在 SQL/PySpark 中）**：
1. **字段映射与类型转换**：
   - server_ip、tag_name、ip、ip_md5、ip_c_segment、manufacture、model、language、region、sdk_version、ip_country、ip_province、ip_city、ip_district → 原样透传
   - `time` → log_time（字段重命名）
   - src → CAST(src AS INT)
   - phonetype → CAST(phonetype AS INT)
   - apn → CAST(apn AS INT)
   - auth_type → CAST(auth_type AS INT)
   - call_type → CAST(call_type AS INT)
   - duration → CAST(duration AS INT)

2. **敏感字段加密**（原逻辑：OnlineCrypt 解密 → IdCrypt 重新加密；沙箱改写为 sha2 哈希）：
   - phonenum → sha2(phonenum, 256) AS phonenum_enc
   - self_phone → sha2(self_phone, 256) AS self_phone_enc
   - imei → sha2(imei, 256) AS imei_enc
   - imsi → sha2(imsi, 256) AS imsi_enc
   - phonenum_reg → sha2(phonenum_reg, 256) AS phonenum_reg_enc
   - android_id → sha2(android_id, 256) AS android_id_enc

**输出要求**：
- 目标表：`internal_platform_db.t_ods_comm_security_user_tag_sync_hi_pyspark_254`
- 写入策略：INSERT OVERWRITE TABLE，使用显式分区写入 partition(ds='20260513')
- 如果目标表不存在，请先按 Hive 标准建表（ORC 存储，分区列 ds），再写入数据。

#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
pyspark_015: 用户标签同步ETL (Java UDF -> sha2 改写)
- 原 pytoolkit/IdCrypt/OnlineCrypt 替换为内置 sha2() 函数
- 原 self.statdate 硬编码为 '20260513'（沙箱化日期）
- 原 self.save_dw_data 改为 spark.sql INSERT OVERWRITE TABLE
"""

from pyspark.sql import SparkSession

spark = (
    SparkSession.builder
    .appName('pyspark_015_user_tag_sync')
    .enableHiveSupport()
    .getOrCreate()
)

ds = '20260513'

# Create output table DDL
spark.sql('DROP TABLE IF EXISTS internal_platform_db.t_ods_comm_security_user_tag_sync_hi_pyspark_254')
spark.sql('''
CREATE TABLE internal_platform_db.t_ods_comm_security_user_tag_sync_hi_pyspark_254 (
  server_ip STRING,
  log_time STRING,
  src INT,
  phonenum_enc STRING,
  tag_name STRING,
  self_phone_enc STRING,
  ip STRING,
  imei_enc STRING,
  imsi_enc STRING,
  phonenum_reg_enc STRING,
  phonetype INT,
  apn INT,
  ip_md5 STRING,
  ip_c_segment STRING,
  auth_type INT,
  android_id_enc STRING,
  manufacture STRING,
  model STRING,
  language STRING,
  region STRING,
  sdk_version STRING,
  call_type INT,
  duration INT,
  ip_country STRING,
  ip_province STRING,
  ip_city STRING,
  ip_district STRING
) PARTITIONED BY (ds STRING)
STORED AS ORC
''')

# Business logic: read from source, apply transformations, write to target
# Original: OnlineCrypt decrypt encrypted fields -> IdCrypt re-encrypt
# Rewrite: sha2() on plaintext fields to produce encrypted output
spark.sql("""
INSERT OVERWRITE TABLE internal_platform_db.t_ods_comm_security_user_tag_sync_hi_pyspark_254 partition(ds='{ds}')
SELECT  server_ip
        ,`time` AS log_time
        ,CAST(src AS INT) AS src
        ,sha2(phonenum, 256) AS phonenum_enc
        ,tag_name
        ,sha2(self_phone, 256) AS self_phone_enc
        ,ip
        ,sha2(imei, 256) AS imei_enc
        ,sha2(imsi, 256) AS imsi_enc
        ,sha2(phonenum_reg, 256) AS phonenum_reg_enc
        ,CAST(phonetype AS INT) AS phonetype
        ,CAST(apn AS INT) AS apn
        ,ip_md5
        ,ip_c_segment
        ,CAST(auth_type AS INT) AS auth_type
        ,sha2(android_id, 256) AS android_id_enc
        ,manufacture
        ,model
        ,language
        ,region
        ,sdk_version
        ,CAST(call_type AS INT) AS call_type
        ,CAST(duration AS INT) AS duration
        ,ip_country
        ,ip_province
        ,ip_city
        ,ip_district
FROM    internal_platform_db.t_sh_event_log_v2_xab00014231_pyspark_254
WHERE   ds = '{ds}'
""".format(ds=ds))

print("pyspark_015 done")

spark.stop()