Upload traffic classifier with ONNX exports

DATA_SOURCES.md

@@ -0,0 +1,116 @@
+# 数据源说明
+
+本项目当前使用三分类标签：
+
+| label | 含义 |
+| --- | --- |
+| 0 | 非研发相关 |
+| 1 | 研发相关 |
+| 2 | 中性 |
+
+## 当前推荐数据集
+
+推荐使用精简后的数据集：
+
+- 原始精简 CSV：`data/refined_compact_labeled.csv`
+- 训练集：`data_refined/train.csv`
+- 验证集：`data_refined/val.csv`
+- 测试集：`data_refined/test.csv`
+
+当前精简 CSV 的类别分布：
+
+| label | 含义 | 数量 |
+| --- | --- | ---: |
+| 0 | 非研发相关 | 22337 |
+| 1 | 研发相关 | 68779 |
+| 2 | 中性 | 2312 |
+
+切分比例由 `data_prep.py` 生成，约为 `8:1:1`。
+
+## 数据来源
+
+### THUOCL
+
+- 仓库：https://github.com/thunlp/THUOCL
+- 用途：提供中文领域词库。
+- 中间产物：`data/thuocl_labeled.csv`
+
+初始映射中，`THUOCL_IT.txt` 标为研发相关，其他领域曾用于非研发或中性候选。当前精简数据集中仅保留部分高权重词：
+
+- `IT`：研发相关，全量保留去重后的词条
+- 其他 THUOCL 领域：非研发相关，每个领域最多保留 300 条
+
+### DomainWordsDict
+
+- 仓库：https://github.com/liuhuanyong/DomainWordsDict
+- 用途：提供大规模中文领域词库。
+- 中间产物：`data/domain_words_labeled.csv`
+
+当前精简映射：
+
+- `计算机业`：研发相关，全量保留去重后的词条
+- `电子工程`：研发相关，全量保留去重后的词条
+- `通信工程`：研发相关，全量保留去重后的词条
+- 其他领域：非研发相关，每个领域最多保留 300 条
+
+旧版曾把 `网络用语`、`诗词歌赋`、`文学名著`、`汉语言学` 标为中性；当前已取消这个策略，这些领域在精简数据集中都按非研发相关处理。
+
+### computerese-cross-references
+
+- 仓库：https://github.com/EarsEyesMouth/computerese-cross-references
+- 用途：补充计算机专业术语中英文对照。
+- 中间产物：`data/computerese_labeled.csv`
+
+该仓库主要内容在 `README.md` 的 Markdown 表格中。当前抽取策略：
+
+- 英文术语全部标为研发相关
+- 中文释义只保留明显具有技术语义的短语
+- 过滤掉过于泛化的中文词，例如“获取”这类容易导致误判的词
+
+当前精简数据集中，`computerese` 全量保留去重后的词条，约 1650 条。
+
+### stopwords
+
+- 仓库：https://github.com/goto456/stopwords
+- 用途：提供中性类样本。
+- 使用文件：
+  - `baidu_stopwords.txt`
+  - `cn_stopwords.txt`
+  - `hit_stopwords.txt`
+  - `scu_stopwords.txt`
+
+当前策略：
+
+- 所有去重后的停用词标为 `2=中性`
+- 中性类只来自该停用词仓库
+- 例如“继续”会被标为中性
+
+## 生成策略
+
+精简数据集的目标是避免中性和非研发样本过多，降低类别不均衡和噪声。
+
+当前规则：
+
+| 类别 | 来源 | 保留策略 |
+| --- | --- | --- |
+| 研发相关 | `IT`、`计算机业`、`电子工程`、`通信工程`、`computerese` | 全量保留去重后的词条，并优先于非研发来源 |
+| 中性 | stopwords | 全量去重保留 |
+| 非研发相关 | 其他领域词库 | 每个领域最多 300 条 |
+
+生成后的 CSV 字段：
+
+| 字段 | 含义 |
+| --- | --- |
+| `text` | 训练文本 |
+| `label` | 标签 ID |
+| `label_name` | 标签名称 |
+| `category` | 来源领域或类别 |
+| `df` | 原词库权重或词频，部分来源为空 |
+| `source_file` | 来源文件 |
+
+## 注意事项
+
+- 当前数据主要是词库，不是完整用户 query。训练出的模型更偏向领域词识别，后续应补充真实用户话术。
+- 中性类目前使用停用词近似表达，如“继续”“好的”“然后”等，适合覆盖短指令和弱语义输入。
+- 非研发相关按领域抽样压缩后，与研发相关数量大致均衡，适合先训练一个轻量 baseline。
+- 若后续新增真实标注样本，建议优先加入 `data/refined_compact_labeled.csv` 或单独维护一个人工标注 CSV，再合并后重新切分。

README.md

@@ -0,0 +1,70 @@
+---
+language:
+- zh
+pipeline_tag: text-classification
+tags:
+- chinese
+- text-classification
+- onnx
+- intent-classification
+library_name: transformers
+---
+
+# traffic-classify
+
+Chinese text classifier for traffic / intent classification into three labels:
+
+- `0`: 非研发相关
+- `1`: 研发相关
+- `2`: 中性
+
+## Files
+
+- Root directory: Transformers model, tokenizer, and config.
+- `onnx/model_fp32.onnx`: FP32 ONNX export.
+- `onnx/model_int8.onnx`: INT8 quantized ONNX model for CPU inference.
+
+## Evaluation
+On the held-out evaluation set, the classifier achieves an overall accuracy of
+approximately `94%`, with a weighted F1-score of approximately `94%`,
+demonstrating strong classification capability across different traffic types.
+
+The per-class F1-scores are:
+
+- 研发相关: `0.96`
+- 中性: `0.93`
+- 非研发相关: `0.88`
+
+The results indicate that the model performs particularly well in identifying
+研发-related traffic, while non-研发 traffic remains relatively more
+challenging due to semantic overlap with technical discussions.
+
+## Dataset Policy
+
+The dataset uses a compact three-class policy:
+
+- 研发相关: full software / IT / electronics / communication terminology sources.
+- 中性: stopwords only.
+- 非研发相关: remaining dictionary categories, compactly sampled.
+
+See the project `DATA_SOURCES.md` for source details.
+
+## Usage
+
+```python
+from transformers import AutoModelForSequenceClassification, AutoTokenizer
+import torch
+
+repo_id = "pawizard/traffic-classify"
+tokenizer = AutoTokenizer.from_pretrained(repo_id)
+model = AutoModelForSequenceClassification.from_pretrained(repo_id)
+
+text = "这个 NullPointerException 报错怎么修复"
+inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=128)
+
+with torch.no_grad():
+    logits = model(**inputs).logits
+    pred = int(logits.argmax(dim=-1).item())
+
+print(model.config.id2label[pred])
+```

config.json

@@ -0,0 +1,41 @@
+{
+  "architectures": [
+    "BertForSequenceClassification"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "directionality": "bidi",
+  "dtype": "float32",
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "id2label": {
+    "0": "\u975e\u7814\u53d1\u76f8\u5173",
+    "1": "\u7814\u53d1\u76f8\u5173",
+    "2": "\u4e2d\u6027"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "label2id": {
+    "\u4e2d\u6027": 2,
+    "\u7814\u53d1\u76f8\u5173": 1,
+    "\u975e\u7814\u53d1\u76f8\u5173": 0
+  },
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "pooler_fc_size": 768,
+  "pooler_num_attention_heads": 12,
+  "pooler_num_fc_layers": 3,
+  "pooler_size_per_head": 128,
+  "pooler_type": "first_token_transform",
+  "position_embedding_type": "absolute",
+  "transformers_version": "4.57.6",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 21128
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1a6069ff6eb49177c3cdb6af9346ae7ff0c758948f43c4cda5e330e44b858992
+size 409103316

onnx/model_fp32.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:61f5f979e68c905893f42ed7c6d193a0aa529e0e6fc682ee2796bcb5037b2d6c
+size 409343847

onnx/model_int8.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:34faa12a375093cd90d0aa26b369f2e655f975df855aa669f34d1fdf38fd19ba
+size 103101880

onnx/special_tokens_map.json

@@ -0,0 +1,37 @@
+{
+  "cls_token": {
+    "content": "[CLS]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "[MASK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "[PAD]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "[SEP]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "[UNK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

onnx/tokenizer_config.json

@@ -0,0 +1,63 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "101": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "102": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "103": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": false,
+  "cls_token": "[CLS]",
+  "do_lower_case": true,
+  "extra_special_tokens": {},
+  "mask_token": "[MASK]",
+  "max_length": 128,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_to_multiple_of": null,
+  "pad_token": "[PAD]",
+  "pad_token_type_id": 0,
+  "padding_side": "right",
+  "sep_token": "[SEP]",
+  "stride": 0,
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "truncation_side": "right",
+  "truncation_strategy": "longest_first",
+  "unk_token": "[UNK]"
+}

special_tokens_map.json

@@ -0,0 +1,37 @@
+{
+  "cls_token": {
+    "content": "[CLS]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "[MASK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "[PAD]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "[SEP]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "[UNK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,56 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "101": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "102": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "103": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": false,
+  "cls_token": "[CLS]",
+  "do_lower_case": true,
+  "extra_special_tokens": {},
+  "mask_token": "[MASK]",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "unk_token": "[UNK]"
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:772ab76968477d5a4289da87bbdd0f254aef4a28339fe192732b6f0c3de8a847
+size 5841