Copy of farbodtavakkoli/OTel-LLM-8.3B-Classification at ee00910

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,154 @@
+---
+license: apache-2.0
+language:
+  - en
+base_model: EssentialAI/rnj-1
+tags:
+  - classification
+  - telecom
+  - 5g
+  - root-cause-analysis
+  - telelogs
+datasets:
+  - netop/TeleLogs
+pipeline_tag: text-classification
+---
+
+# OTel-LLM-8.3B-Classification
+
+**OTel-LLM-8.3B-Classification** is a telecom-specialized classification model fine-tuned for **5G network root cause analysis (RCA)**, achieving **~99% accuracy** on the [TeleLogs benchmark](https://huggingface.co/datasets/netop/TeleLogs). It is part of the [OTel Family of Models](https://huggingface.co/collections/farbodtavakkoli/otel-llm), an open-source initiative to build industry-standard AI models for the global telecommunications sector.
+
+## Model Details
+
+| Attribute | Value |
+|---|---|
+| **Base Model** | [EssentialAI/rnj-1](https://huggingface.co/EssentialAI/rnj-1) |
+| **Parameters** | 8.3B |
+| **Training Method** | Full parameter fine-tuning with classification head |
+| **Task** | Multi-class sequence classification (8 root cause classes) |
+| **Language** | English |
+| **License** | Apache 2.0 |
+
+## Benchmark Results
+
+This model achieves **~99% accuracy** on the TeleLogs test set, higher than the published baselines in the TeleLogs paper shown below, including the reasoning-augmented models. It is reported as an auxiliary experiment in the OTel paper (NeurIPS 2026) and is not part of the paper's 30-model baseline table.
+
+| Model | Reasoning | Test pass@1 | Test maj@4 |
+|---|---|---|---|
+| Qwen2.5-32B-Instruct | ❌ | 18.85% | 19.60% |
+| DeepSeek-R1-Distill-Llama-70B | ✅ | 29.42% | 34.84% |
+| QwQ-32B | ✅ | 33.62% | 39.00% |
+| Qwen3-32B | ✅ | 33.77% | 37.04% |
+| Qwen2.5-RCA-1.5B | ✅ | 87.56% | 87.73% |
+| Qwen2.5-RCA-7B | ✅ | 87.01% | 88.89% |
+| Qwen2.5-RCA-32B | ✅ | 95.86% | 96.18% |
+| **OTel-LLM-8.3B-Classification** | ❌ | **~99%** | — |
+
+## TeleLogs Dataset
+
+[TeleLogs](https://huggingface.co/datasets/netop/TeleLogs) is a synthetic dataset designed to advance research on root cause analysis in 5G networks. It simulates drive-test scenarios involving a user equipment (UE) moving through a region covered by multiple 5G base stations (gNodeBs). Each instance includes a symptom (throughput degradation below 600 Mbps) and one or more root causes from 8 predefined classes:
+
+1. Test vehicle speed exceeds 40 km/h, impacting user throughput
+2. Downtilt angle of the serving cell is too large, causing weak coverage at the far end
+3. Serving cell coverage distance exceeds 1 km, resulting in poor RSRP
+4. Non-colocated co-frequency neighboring cells cause severe interference
+5. Neighbor cell and serving cell have the same PCI mod 30, causing reference signal overlap
+6. Frequent handovers degrading user performance
+7. Misconfigured handover thresholds degrading user performance
+8. Average scheduled resource blocks (RBs) of the serving cell are below 160
+
+For more details, see the [TeleLogs paper](https://arxiv.org/abs/2507.21974).
+
+## Training Details
+
+### Approach
+
+The model was trained as a **sequence classification** model. The classification head consists of a **Dropout(0.1) layer followed by a Linear projection** from the model's hidden dimension to the 8 root cause classes.
+
+### Hyperparameters
+
+| Parameter | Value |
+|---|---|
+| **Learning Rate** | 7e-6 |
+| **LR Scheduler** | Cosine |
+| **Warmup Steps** | 150 |
+| **Batch Size (per device)** | 4 |
+| **Gradient Accumulation Steps** | 2 |
+| **Effective Batch Size** | 64 (4 × 2 × 8 GPUs) |
+| **Epochs** | 25 |
+| **Weight Decay** | 0.01 |
+| **Label Smoothing** | 0.1 |
+| **Max Gradient Norm** | 0.5 |
+| **Max Sequence Length** | 5000 tokens |
+| **Precision** | BF16 |
+| **Attention** | Flash Attention 2 |
+| **Distributed Training** | FSDP |
+| **Gradient Checkpointing** | Enabled |
+
+### Classification Head
+
+```
+Dropout(0.1) → Linear(hidden_size, 8, bias=False)
+```
+
+Weight initialization: Normal distribution with mean=0.0, std=0.01.
+
+## Intended Use
+
+This model is optimized for:
+
+- **5G network root cause analysis** — classifying network log data into specific fault categories
+- **Telecom troubleshooting** — automated diagnosis of throughput degradation causes from drive-test measurements
+
+## Related Models
+
+### Language Models
+- [OTel LLM Collection](https://huggingface.co/collections/farbodtavakkoli/otel-llm)
+
+### Embedding Models
+- [OTel Embedding Collection](https://huggingface.co/collections/farbodtavakkoli/otel-embedding)
+
+### Reranker Models
+- [OTel Reranker Collection](https://huggingface.co/collections/farbodtavakkoli/otel-reranker)
+
+## Related Datasets 
+
+- [TeleLogs](https://huggingface.co/datasets/netop/TeleLogs) — 5G RCA benchmark
+- [OTel-Embedding](https://huggingface.co/datasets/farbodtavakkoli/OTel-Embedding)
+- [OTel-Safety](https://huggingface.co/datasets/farbodtavakkoli/OTel-Safety)
+- [OTel-LLM](https://huggingface.co/datasets/farbodtavakkoli/OTel-LLM)
+- [OTel-Reranker](https://huggingface.co/datasets/farbodtavakkoli/OTel-Reranker)
+
+## Training Infrastructure
+
+- **Framework**: ScalarLM (GPU-agnostic)
+- **Compute**: TensorWave with AMD GPUs and Azure with NVIDIA GPUs
+- **Training Code**: [github.com/farbodtavakkoli/OTel](https://github.com/farbodtavakkoli/OTel)
+
+## Citation
+
+```bibtex
+@inproceedings{tavakkoli2026otel,
+  title     = {OTel: Open Telco AI Datasets, Benchmarks, and Models},
+  author    = {Tavakkoli, Farbod and Diamos, Gregory and Church, Kenneth and Kanter, David and Austin, Mark and Karim, Imtiaz and Rahman, Mirza Masfiqur and Debbah, Merouane Abdelkader and Nezami, Zeinab and Maatouk, Ali and Tassiulas, Leandros and Ying, Rex and Sorros, Nick and Powell, Louis and Vasiloglou, Nikolaos and Vaswani, Ashish and Singla, Somanshu and Chaluvaraju, Adarsh},
+  booktitle = {Advances in Neural Information Processing Systems (NeurIPS), Evaluations and Datasets Track},
+  year      = {2026},
+  url       = {https://github.com/farbodtavakkoli/OTel}
+}
+```
+
+If you use the TeleLogs dataset, please also cite:
+
+```bibtex
+@article{sana2025reasoning,
+  title={{Reasoning Language Models for Root Cause Analysis in 5G Wireless Networks}},
+  author={Mohamed Sana and Nicola Piovesan and Antonio De Domenico and Yibin Kang and Haozhe Zhang and Merouane Debbah and Fadhel Ayed},
+  year={2025},
+  eprint={arXiv:2507.21974},
+  url={https://arxiv.org/abs/2507.21974}
+}
+```
+
+## Contact
+
+If you have any technical questions, please feel free to reach out to [farbod.tavakkoli@att.com](mailto:farbod.tavakkoli@att.com) or [farbodtavakoli@gmail.com](mailto:farbodtavakoli@gmail.com)

chat_template.jinja

@@ -0,0 +1,128 @@
+{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) -%}
+{%- set emit = namespace(started=false) -%}
+
+{# ---------- Build base system message (always emitted) ---------- #}
+{%- set base_system = 'You are rnj-1, a foundation model trained by Essential AI.\n' -%}
+
+{# ---------- Default system prompt if user system is absent ---------- #}
+{%- set default_system = 'You are a helpful assistant.' -%}
+
+{# Detect whether the first message is a user-provided system message #}
+{%- set has_user_system = (messages
+    and messages[0].role == 'system'
+    and (messages[0].content is string)) -%}
+
+{# The system instruction that should apply (user system wins; else default) #}
+{%- set effective_system = (has_user_system and messages[0].content) or default_system -%}
+
+
+{# ---------- Optional tools preface as a synthetic system message ---------- #}
+{%- if tools %}
+  {%- set sys_preamble -%}
+# Tools
+
+You may call one or more functions to assist with the user query.
+
+You are provided with function signatures within <tools></tools> XML tags:
+<tools>
+{%- for tool in tools %}
+{{ "\n" ~ (tool | tojson) }}
+{% endfor %}
+</tools>
+
+For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
+<tool_call>
+{"name": <function-name>, "arguments": <args-json-object>}
+</tool_call>
+  {%- endset -%}
+
+  {# Always include effective_system; user system prevails over default #}
+  {%- set sys_content = effective_system ~ "\n\n" ~ sys_preamble -%}
+
+  {%- set content = '<|start_header_id|>system<|end_header_id|>\n'
+      ~ base_system ~ '\n' ~ sys_content ~ '<|eot_id|>' -%}
+  {%- if not emit.started -%}{%- set content = bos_token ~ content -%}{%- set emit.started = true -%}{%- endif -%}
+  {{- content -}}
+{%- else %}
+  {# No tools: always emit base_system + effective_system #}
+  {%- set content = '<|start_header_id|>system<|end_header_id|>\n'
+      ~ base_system ~ '\n' ~ effective_system ~ '<|eot_id|>' -%}
+  {%- if not emit.started -%}{%- set content = bos_token ~ content -%}{%- set emit.started = true -%}{%- endif -%}
+  {{- content -}}
+{%- endif -%}
+
+{# ---------- Locate last user query for multi-step tool behavior ---------- #}
+{%- for message in messages[::-1] %}
+  {%- set index = (messages|length - 1) - loop.index0 -%}
+  {%- if ns.multi_step_tool
+        and message.role == "user"
+        and message.content is string
+        and not (message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) -%}
+    {%- set ns.multi_step_tool = false -%}
+    {%- set ns.last_query_index = index -%}
+  {%- endif -%}
+{%- endfor -%}
+
+{# ---------- Walk all messages and emit in Llama-3 format ---------- #}
+{%- for message in messages %}
+  {%- if message.content is string -%}
+    {%- set content = message.content -%}
+  {%- else -%}
+    {%- set content = '' -%}
+  {%- endif -%}
+
+  {# Skip the FIRST system message if it existed, since we already embedded it in effective_system #}
+  {%- if loop.first and message.role == "system" -%}
+    {# no-op #}
+
+  {%- elif (message.role == "user") or (message.role == "system") -%}
+    {%- set block = '<|start_header_id|>' ~ message.role ~ '<|end_header_id|>\n' ~ content ~ '<|eot_id|>' -%}
+    {%- if not emit.started -%}{%- set block = bos_token ~ block -%}{%- set emit.started = true -%}{%- endif -%}
+    {{- block -}}
+
+  {%- elif message.role == "assistant" -%}
+    {%- set body = content -%}
+    {%- set header = '<|start_header_id|>assistant<|end_header_id|>\n' -%}
+    {%- if not emit.started -%}{{ bos_token }}{%- set emit.started = true -%}{%- endif -%}
+    {{- header -}}
+    {% generation %}
+    {{- body -}}
+    {%- if message.tool_calls -%}
+      {%- for tool_call in message.tool_calls -%}
+        {%- if tool_call.function -%}{%- set tc = tool_call.function -%}{%- else -%}{%- set tc = tool_call -%}{%- endif -%}
+        {%- set args_json = (tc.arguments if (tc.arguments is string) else (tc.arguments | tojson)) -%}
+        {%- if loop.first -%}
+          {{- '<tool_call>\n{"name": "' ~ tc.name ~ '", "arguments": ' ~ args_json ~ '}\n</tool_call>' -}}
+        {%- else -%}
+          {{- '\n<tool_call>\n{"name": "' ~ tc.name ~ '", "arguments": ' ~ args_json ~ '}\n</tool_call>' -}}
+        {%- endif -%}
+      {%- endfor -%}
+    {%- endif -%}
+    {{- '<|eot_id|>' -}}{%- endgeneration -%}
+
+  {%- elif message.role == "tool" -%}
+    {%- set open_user = (loop.first or (loop.index0 > 0 and messages[loop.index0 - 1].role != "tool")) -%}
+    {%- set close_user = (loop.last or (loop.index0 < messages|length - 1 and messages[loop.index0 + 1].role != "tool")) -%}
+
+    {%- if open_user -%}
+      {%- set header = '<|start_header_id|>user<|end_header_id|>\n' -%}
+      {%- if not emit.started -%}{%- set header = bos_token ~ header -%}{%- set emit.started = true -%}{%- endif -%}
+      {{- header -}}
+    {%- endif -%}
+    {%- if open_user -%}
+      {{- '<tool_response>\n' -}}
+    {%- else -%}
+      {{- '\n<tool_response>\n' -}}
+    {%- endif -%}
+    {{- content -}}
+    {{- '\n</tool_response>' -}}
+
+    {%- if close_user -%}
+      {{- '<|eot_id|>' -}}
+    {%- endif -%}
+  {%- endif -%}
+{%- endfor -%}
+
+{%- if add_generation_prompt -%}
+  {{- '<|start_header_id|>assistant<|end_header_id|>\n' -}}
+{%- endif -%}

config.json

@@ -0,0 +1,141 @@
+{
+  "_sliding_window_pattern": 1,
+  "architectures": [
+    "Gemma3TextForSequenceClassification"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "attn_logit_softcapping": null,
+  "bos_token_id": 2,
+  "cache_implementation": "hybrid",
+  "dtype": "bfloat16",
+  "eos_token_id": 1,
+  "final_logit_softcapping": 30.0,
+  "head_dim": 128,
+  "hidden_act": "gelu_pytorch_tanh",
+  "hidden_activation": "gelu_pytorch_tanh",
+  "hidden_size": 4096,
+  "id2label": {
+    "0": "C1",
+    "1": "C2",
+    "2": "C3",
+    "3": "C4",
+    "4": "C5",
+    "5": "C6",
+    "6": "C7",
+    "7": "C8"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 16384,
+  "label2id": {
+    "C1": 0,
+    "C2": 1,
+    "C3": 2,
+    "C4": 3,
+    "C5": 4,
+    "C6": 5,
+    "C7": 6,
+    "C8": 7
+  },
+  "layer_type": [
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention"
+  ],
+  "layer_types": [
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention"
+  ],
+  "max_position_embeddings": 32768,
+  "model_type": "gemma3_text",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 8,
+  "pad_token_id": 128001,
+  "query_pre_attn_scalar": 128,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "full_attention": {
+      "attn_factor": 1.0,
+      "beta_fast": 64.0,
+      "beta_slow": 1.0,
+      "extrapolation_factor": 1.0,
+      "factor": 4.0,
+      "original_max_position_embeddings": 8192,
+      "rope_theta": 10000,
+      "rope_type": "yarn"
+    },
+    "rope_theta": null,
+    "rope_type": "default",
+    "sliding_attention": {
+      "rope_theta": 10000,
+      "rope_type": "default"
+    }
+  },
+  "sliding_window": 32768,
+  "sliding_window_pattern": 1,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.2.0",
+  "use_bidirectional_attention": false,
+  "use_cache": false,
+  "vocab_size": 128256
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:61272c9f48ee08baa3f1761bf3403b2535eafbda150ca41dfb0619ab7b4be3ed
+size 16621117448

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:96a2db531b75969ce3aeaaf126eb7957f7e07a28c7e770705787ef035f19d9da
+size 17209795

tokenizer_config.json

@@ -0,0 +1,14 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<|begin_of_text|>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|end_of_text|>",
+  "is_local": false,
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<|end_of_text|>",
+  "tokenizer_class": "TokenizersBackend"
+}