Safetensors
qwen3_5
wbhVince829 commited on
Commit
8173abd
·
1 Parent(s): d13b01d

update quickstart and teletable

Browse files
Files changed (1) hide show
  1. README.md +81 -66
README.md CHANGED
@@ -14,7 +14,7 @@ license: apache-2.0
14
  - **Among open-source models**, TelecomGPT-R1 leads DeepSeek-V3-0324 (685B) by **+30.3**, LLaMA-3.3-70B by **+34.9**, and Qwen2.5-72B by **+35.6**, while operating at roughly **25× fewer active parameters than the next-best open entrant**.
15
  - **Among closed-source models**, TelecomGPT-R1 reaches SOTA performance across both the general-purpose frontier tier and the telecom-specialized tier, as detailed in the two bullets below.
16
  - **Among general-purpose frontier models**, TelecomGPT-R1 leads Gemini-3.1-Pro by **+14.0**, Claude-Opus-4.6 by **+16.3**, and GPT-5 by **+17.7**. These systems sit at the **trillion-parameter-class frontier** (active-parameter counts are not publicly disclosed but are widely reported as orders of magnitude larger than 27B), making the margin a parameter-efficiency result as much as an accuracy result.
17
- - **Among telecom-specialized models**, TelecomGPT-R1 is **on par with the leading closed operator-internal telecom model AT&T's OTel-LLM-8.3B-QnA**, and leads SoftBank LTM by **+16.0**, demonstrating that an open telecom reasoning model can reach SOTA performance alongside top operator-internal baselines on the GSMA Open Telco Leaderboard. *(On TeleTables, we follow the original paper's evaluation protocol by attaching the table content directly to the prompt — a table-grounded reasoning setup rather than retrieval without table id or content.)*
18
 
19
  **In one line: TelecomGPT-R1 demonstrates that an open 27B telecom reasoning model can reach SOTA performance across the full breadth of the GSMA Open Telco Leaderboard.**
20
 
@@ -23,6 +23,82 @@ license: apache-2.0
23
  **Figure 1 | TelecomGPT-R1 vs frontier closed-source models on the GSMA Open Telco Leaderboard.** *Each spoke is one benchmark (plus the overall average), normalized by its per-axis leaderboard best so that `1.0` = best score on that benchmark. Our 27B open-source policy reaches `1.0` on **five of eight axes** (3GPP-TSG, srsRANBench, TeleLogs, TeleTables, Average) and stays at or above `0.95` on every other axis, visibly tracing the outer edge of the radar where no other model, open or closed, matches it on all axes simultaneously.*
24
 
25
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
26
  ---
27
 
28
 
@@ -130,71 +206,6 @@ KU/DFI's role is to build that open commons. The program now spans the key layer
130
  - **Model weights.** [KU-DFI/TelecomGPT-R1](https://huggingface.co/KU-DFI/TelecomGPT-R1/tree/main)
131
  - **Unified benchmark.** [GSMA Open Telco Leaderboard](https://huggingface.co/spaces/GSMA/open-telco-leaderboard)
132
 
133
- ### Quickstart
134
-
135
- Here is a code snippet demonstrating how to load TelecomGPT-R1 with `transformers` and generate a telecom-grounded response:
136
-
137
- ```python
138
- from transformers import AutoModelForCausalLM, AutoTokenizer
139
-
140
- model_name = "KU-DFI/TelecomGPT-R1"
141
-
142
- model = AutoModelForCausalLM.from_pretrained(
143
- model_name,
144
- torch_dtype="auto",
145
- device_map="auto",
146
- )
147
- tokenizer = AutoTokenizer.from_pretrained(model_name)
148
-
149
- prompt = (
150
- "A 5G NR cell is observing repeated random-access failures from cell-edge UEs. "
151
- "Drive-test capture shows: average RSRP = -108 dBm, average RSRQ = -16 dB, "
152
- "PRACH preamble attempts averaging 8 with no Msg2 (RAR) received within "
153
- "ra-ResponseWindow, UE timing-advance range 4-7 km, and PRACH configuration "
154
- "uses preamble format A1 with zeroCorrelationZoneConfig = 8. "
155
- "Diagnose the most likely root cause and propose a configuration change."
156
- )
157
- messages = [
158
- {
159
- "role": "system",
160
- "content": (
161
- "You are TelecomGPT-R1, an open 27B telecom reasoning model from "
162
- "KU/DFI. Reason step-by-step over 3GPP standards, RAN logs, RF and "
163
- "network derivations, and telecom code."
164
- ),
165
- },
166
- {"role": "user", "content": prompt},
167
- ]
168
- text = tokenizer.apply_chat_template(
169
- messages,
170
- tokenize=False,
171
- add_generation_prompt=True,
172
- )
173
- model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
174
-
175
- generated_ids = model.generate(
176
- **model_inputs,
177
- max_new_tokens=2048,
178
- )
179
- generated_ids = [
180
- output_ids[len(input_ids):]
181
- for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
182
- ]
183
-
184
- response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
185
- print(response)
186
- ```
187
-
188
- For production / batch serving on operator-confidential data, host with [vLLM](https://github.com/vllm-project/vllm):
189
-
190
- ```bash
191
- vllm serve KU-DFI/TelecomGPT-R1 \
192
- --tensor-parallel-size 4 \
193
- --max-model-len 32768 \
194
- --gpu-memory-utilization 0.90
195
- ```
196
-
197
- **Hardware**: TelecomGPT-R1 (27B, bf16) fits on a single H100 80GB or MI300X; for high-throughput inference behind an operator firewall a single H100/MI300 node serves the model end-to-end.
198
 
199
  ### Citation
200
 
@@ -220,3 +231,7 @@ vllm serve KU-DFI/TelecomGPT-R1 \
220
  ### Acknowledgements
221
 
222
  This work was supported by the Digital Future Institute of Khalifa University; the College of Information Science and Electronic Engineering, Zhejiang University; the College of Computer Science and Technology, Zhejiang University; and the Research Computing team of Khalifa University.
 
 
 
 
 
14
  - **Among open-source models**, TelecomGPT-R1 leads DeepSeek-V3-0324 (685B) by **+30.3**, LLaMA-3.3-70B by **+34.9**, and Qwen2.5-72B by **+35.6**, while operating at roughly **25× fewer active parameters than the next-best open entrant**.
15
  - **Among closed-source models**, TelecomGPT-R1 reaches SOTA performance across both the general-purpose frontier tier and the telecom-specialized tier, as detailed in the two bullets below.
16
  - **Among general-purpose frontier models**, TelecomGPT-R1 leads Gemini-3.1-Pro by **+14.0**, Claude-Opus-4.6 by **+16.3**, and GPT-5 by **+17.7**. These systems sit at the **trillion-parameter-class frontier** (active-parameter counts are not publicly disclosed but are widely reported as orders of magnitude larger than 27B), making the margin a parameter-efficiency result as much as an accuracy result.
17
+ - **Among telecom-specialized models**, TelecomGPT-R1 is **on par with the leading closed operator-internal telecom model AT&T's OTel-LLM-8.3B-QnA**, and leads SoftBank LTM by **+16.0**, demonstrating that an open telecom reasoning model can reach SOTA performance alongside top operator-internal baselines on the GSMA Open Telco Leaderboard.
18
 
19
  **In one line: TelecomGPT-R1 demonstrates that an open 27B telecom reasoning model can reach SOTA performance across the full breadth of the GSMA Open Telco Leaderboard.**
20
 
 
23
  **Figure 1 | TelecomGPT-R1 vs frontier closed-source models on the GSMA Open Telco Leaderboard.** *Each spoke is one benchmark (plus the overall average), normalized by its per-axis leaderboard best so that `1.0` = best score on that benchmark. Our 27B open-source policy reaches `1.0` on **five of eight axes** (3GPP-TSG, srsRANBench, TeleLogs, TeleTables, Average) and stays at or above `0.95` on every other axis, visibly tracing the outer edge of the radar where no other model, open or closed, matches it on all axes simultaneously.*
24
 
25
 
26
+
27
+
28
+ ---
29
+
30
+
31
+ ### Quickstart
32
+
33
+ **Requirements:** `transformers >= 4.51.0`, `torch >= 2.1`.
34
+
35
+ Here is a code snippet demonstrating how to load TelecomGPT-R1 with `transformers` and generate a telecom-grounded response:
36
+
37
+ ```python
38
+ from transformers import AutoModelForCausalLM, AutoTokenizer
39
+
40
+ model_name = "KU-DFI/TelecomGPT-R1"
41
+
42
+ model = AutoModelForCausalLM.from_pretrained(
43
+ model_name,
44
+ torch_dtype="auto",
45
+ device_map="auto",
46
+ )
47
+ tokenizer = AutoTokenizer.from_pretrained(model_name)
48
+
49
+ prompt = (
50
+ "A 5G NR cell is observing repeated random-access failures from cell-edge UEs. "
51
+ "Drive-test capture shows: average RSRP = -108 dBm, average RSRQ = -16 dB, "
52
+ "PRACH preamble attempts averaging 8 with no Msg2 (RAR) received within "
53
+ "ra-ResponseWindow, UE timing-advance range 4-7 km, and PRACH configuration "
54
+ "uses preamble format A1 with zeroCorrelationZoneConfig = 8. "
55
+ "Diagnose the most likely root cause and propose a configuration change."
56
+ )
57
+ messages = [
58
+ {
59
+ "role": "system",
60
+ "content": (
61
+ "You are TelecomGPT-R1, an open 27B telecom reasoning model from "
62
+ "KU/DFI. Reason step-by-step over 3GPP standards, RAN logs, RF and "
63
+ "network derivations, and telecom code."
64
+ ),
65
+ },
66
+ {"role": "user", "content": prompt},
67
+ ]
68
+ text = tokenizer.apply_chat_template(
69
+ messages,
70
+ tokenize=False,
71
+ add_generation_prompt=True,
72
+ )
73
+ model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
74
+
75
+ generated_ids = model.generate(
76
+ **model_inputs,
77
+ max_new_tokens=2048,
78
+ )
79
+ generated_ids = [
80
+ output_ids[len(input_ids):]
81
+ for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
82
+ ]
83
+
84
+ response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
85
+ print(response)
86
+ ```
87
+
88
+ For production / batch serving on operator-confidential data, host with [vLLM](https://github.com/vllm-project/vllm):
89
+
90
+ ```bash
91
+ vllm serve KU-DFI/TelecomGPT-R1 \
92
+ --tensor-parallel-size 1 \
93
+ --max-model-len 8192 \
94
+ --gpu-memory-utilization 0.85
95
+ ```
96
+
97
+ (Scale `--tensor-parallel-size`, `--max-model-len`, and `--gpu-memory-utilization` up as needed for multi-GPU nodes or higher-throughput serving.)
98
+
99
+ **Hardware**: TelecomGPT-R1 (27B, bf16) fits on a single H100 80GB or MI300X with the default settings above; multi-GPU nodes allow longer contexts and larger batches behind an operator firewall.
100
+
101
+
102
  ---
103
 
104
 
 
206
  - **Model weights.** [KU-DFI/TelecomGPT-R1](https://huggingface.co/KU-DFI/TelecomGPT-R1/tree/main)
207
  - **Unified benchmark.** [GSMA Open Telco Leaderboard](https://huggingface.co/spaces/GSMA/open-telco-leaderboard)
208
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
209
 
210
  ### Citation
211
 
 
231
  ### Acknowledgements
232
 
233
  This work was supported by the Digital Future Institute of Khalifa University; the College of Information Science and Electronic Engineering, Zhejiang University; the College of Computer Science and Technology, Zhejiang University; and the Research Computing team of Khalifa University.
234
+
235
+ ---
236
+
237
+ [^teletables]: On TeleTables, we follow the original paper's evaluation protocol by attaching the table content directly to the prompt — a table-grounded reasoning setup rather than retrieval without table id or content.