skatzR commited on
Commit
8b23def
Β·
verified Β·
1 Parent(s): 99f92f8

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +263 -3
README.md CHANGED
@@ -1,3 +1,263 @@
1
- ---
2
- license: mit
3
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ base_model:
4
+ - FacebookAI/xlm-roberta-large
5
+ language:
6
+ - ru
7
+ tags:
8
+ - Reasoning
9
+ - Logical-Analysis
10
+ - Text-Classification
11
+ - AI-Safety
12
+ - Evaluation
13
+ - Judge-model
14
+
15
+ ---
16
+
17
+ [![Hugging Face](https://img.shields.io/badge/Hugging%20Face-model-blue)](https://huggingface.co/skatzR/RQA-X1)
18
+
19
+ # 🧠 RQA β€” Reasoning Quality Analyzer (v1)
20
+
21
+ **RQA** is a **judge model** designed to evaluate the *quality of reasoning in text*.
22
+ It does **not** generate, rewrite, or explain content β€” instead, it **assesses whether a text contains logical problems**, and if so, **what kind**.
23
+
24
+ > **RQA is a judge, not a teacher and not a generator.**
25
+
26
+ ---
27
+
28
+ ## πŸ” What Problem Does RQA Solve?
29
+
30
+ Modern LLM-generated and human-written texts often:
31
+
32
+ - sound coherent,
33
+ - use correct vocabulary,
34
+ - follow a plausible narrative,
35
+
36
+ …but still contain **logical problems** that are:
37
+
38
+ - subtle,
39
+ - hidden in structure,
40
+ - difficult to detect with standard classifiers.
41
+
42
+ **RQA focuses specifically on reasoning quality**, not style or factual correctness.
43
+
44
+ ---
45
+
46
+ ## 🧩 Model Overview
47
+
48
+ | Property | Value |
49
+ |--------|------|
50
+ | **Model Type** | Judge / Evaluator |
51
+ | **Base Encoder** | XLM-RoBERTa Large |
52
+ | **Pooling** | Mean pooling |
53
+ | **Heads** | 2 (binary + multi-label) |
54
+ | **Language** | Russian πŸ‡·πŸ‡Ί |
55
+ | **License** | Mit |
56
+
57
+ ---
58
+
59
+ ## 🧠 What the Model Predicts
60
+
61
+ RQA produces **two independent outputs**:
62
+
63
+ ### 1️⃣ Logical Issue Detection
64
+
65
+ - **Binary decision**
66
+ `has_logical_issue ∈ {0, 1}`
67
+ - Calibrated probability is provided
68
+
69
+ ### 2️⃣ Error Type Classification (Multi-label)
70
+
71
+ If a logical issue exists, the model can identify one or more of the following error types:
72
+
73
+ - `false_causality`
74
+ - `unsupported_claim`
75
+ - `overgeneralization`
76
+ - `missing_premise`
77
+ - `contradiction`
78
+ - `circular_reasoning`
79
+
80
+ > Error classification is applied **only if a logical issue is detected**.
81
+
82
+ ---
83
+
84
+ ## 🧠 Hidden Logical Problems (Key Concept)
85
+
86
+ RQA explicitly distinguishes between:
87
+
88
+ - **Explicit logical errors**
89
+ (clearly identifiable fallacies)
90
+
91
+ - **Hidden logical problems**
92
+ (structural issues such as:
93
+ - implicit assumptions,
94
+ - shifts of criteria,
95
+ - persuasive but unsupported reasoning)
96
+
97
+ Hidden problems are **not labeling mistakes** β€” they are a **separate, intentional difficulty class**.
98
+
99
+ ---
100
+
101
+ ## πŸ—οΈ Architecture Details
102
+
103
+ - **Encoder**: XLM-RoBERTa Large (pretrained weights preserved)
104
+ - **Pooling**: Mean pooling (more stable than CLS for long texts)
105
+ - **Two independent heads**:
106
+ - Binary head: `has_logical_issue`
107
+ - Multi-label head: `error_types`
108
+ - **Separate projections and dropout** to reduce negative transfer
109
+
110
+ ---
111
+
112
+ ## πŸŽ“ Training Philosophy
113
+
114
+ ### πŸ”’ Strict Data Contract
115
+
116
+ - Logical texts **cannot** contain errors
117
+ - Hidden problems **cannot** contain explicit error labels
118
+ - Invalid samples are **removed**, never auto-fixed
119
+
120
+ ### βš–οΈ Balanced Difficulty
121
+
122
+ - Hidden problems ≀ **30%** of all problematic texts
123
+ (`hidden / (explicit + hidden) ≀ 0.3`)
124
+
125
+ ### 🎯 Loss Design
126
+
127
+ - Binary cross-entropy for issue detection
128
+ - Masked multi-label loss for error types
129
+ - **Uncertainty-weighted loss** for stable multi-task training
130
+
131
+ ---
132
+
133
+ ## 🌑️ Confidence Calibration
134
+
135
+ RQA uses **post-hoc Temperature Scaling**:
136
+
137
+ - Separate calibration for:
138
+ - `has_logical_issue`
139
+ - each error type
140
+ - Ensures predicted probabilities reflect real confidence
141
+ - Enables safe thresholding in production
142
+
143
+ ---
144
+
145
+ ## πŸš€ Intended Use
146
+
147
+ ### βœ… Recommended for:
148
+
149
+ - Reasoning quality evaluation
150
+ - LLM output auditing
151
+ - AI safety pipelines
152
+ - Educational or analytical tooling
153
+ - Pre-filtering or routing in generation systems
154
+
155
+ ### ❌ Not intended for:
156
+
157
+ - Text generation
158
+ - Explanation or correction of errors
159
+ - Style or grammar analysis
160
+ - Factual verification
161
+
162
+ ---
163
+
164
+ ## πŸ§ͺ Model Behavior
165
+
166
+ - Conservative by design
167
+ - Optimized for **low false positives**
168
+ - Explicitly robust to:
169
+ - topic changes,
170
+ - writing style,
171
+ - emotional tone
172
+
173
+ The model judges **logic**, not rhetoric.
174
+
175
+ ---
176
+
177
+ ## πŸ“¦ Output Example
178
+
179
+ ```json
180
+ {
181
+ "has_logical_issue": true,
182
+ "has_issue_probability": 0.87,
183
+ "errors": [
184
+ { "type": "missing_premise", "probability": 0.72 },
185
+ { "type": "overgeneralization", "probability": 0.61 }
186
+ ]
187
+ }
188
+ ```
189
+ ---
190
+
191
+ ## πŸ“š Training Data (High-level)
192
+
193
+ - **Custom-generated dataset**
194
+ - **Thousands of long-form argumentative texts**
195
+ - **Multiple domains and reasoning modes**
196
+ - **Carefully controlled balance of:**
197
+ - logical texts
198
+ - explicit errors
199
+ - hidden problems
200
+
201
+ > The dataset was designed specifically for **judge behavior**, not for text generation.
202
+
203
+ ---
204
+
205
+ ## ⚠️ Limitations
206
+
207
+ - RQA evaluates **reasoning structure**, not factual truth
208
+ - A logically valid argument may still be **factually incorrect**
209
+ - Subtle philosophical disagreements are **not always logical errors**
210
+ - The model may over-detect issues in highly rhetorical or persuasive texts.
211
+
212
+ ---
213
+
214
+ ## 🧩 Philosophy
215
+
216
+ > **Good reasoning is not about sounding convincing β€”
217
+ > it is about what actually follows from what.**
218
+
219
+ RQA is built to reflect this principle.
220
+
221
+ ---
222
+
223
+ ## πŸ”§ Implementation Details
224
+
225
+ This model uses a custom Hugging Face architecture (`modeling_rqa.py`)
226
+ and is loaded with:
227
+
228
+ - `trust_remote_code=True`
229
+ - `safetensors` weights (no `.bin` file)
230
+
231
+ This is expected and fully supported by Hugging Face.
232
+
233
+ ---
234
+
235
+ ## πŸš€ Quick Start
236
+
237
+ ```python
238
+ from transformers import AutoTokenizer, AutoModel
239
+
240
+ tokenizer = AutoTokenizer.from_pretrained(
241
+ "USERNAME/RQA-v1",
242
+ trust_remote_code=True
243
+ )
244
+
245
+ model = AutoModel.from_pretrained(
246
+ "USERNAME/RQA-v1",
247
+ trust_remote_code=True
248
+ )
249
+
250
+ inputs = tokenizer("Your text here", return_tensors="pt")
251
+ outputs = model(**inputs)
252
+
253
+ has_issue_logits = outputs["has_issue_logits"]
254
+ errors_logits = outputs["errors_logits"]
255
+ ```
256
+
257
+ ---
258
+
259
+ ## πŸ“œ License
260
+
261
+ MIT
262
+
263
+ ---