Hugging Face's logo

Spaces:
slamos
/
bc-test
Paused

App Files Community
lamossta commited on
Commit
71d7b3a
·
1 Parent(s): ffcf8df

ntbs

Browse files
Files changed (6) hide show
  1. src/notebooks/inspect_gutenberg_data.ipynb +0 -0
  2. src/notebooks/inspect_pairs.ipynb +0 -0
  3. src/notebooks/inspect_paragraphs.ipynb +0 -0
  4. src/notebooks/inspect_sentences.ipynb +0 -0
  5. src/notebooks/inspect_splits.ipynb +712 -0
  6. src/notebooks/inspect_ted_data.ipynb +0 -0
src/notebooks/inspect_gutenberg_data.ipynb ADDED
The diff for this file is too large to render. See raw diff
 
src/notebooks/inspect_pairs.ipynb ADDED
The diff for this file is too large to render. See raw diff
 
src/notebooks/inspect_paragraphs.ipynb ADDED
The diff for this file is too large to render. See raw diff
 
src/notebooks/inspect_sentences.ipynb ADDED
The diff for this file is too large to render. See raw diff
 
src/notebooks/inspect_splits.ipynb ADDED
@@ -0,0 +1,712 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "cells": [
3
+ {
4
+ "cell_type": "markdown",
5
+ "id": "s7bupuhhjhn",
6
+ "source": "# Inspect Label Distributions per Dataset Configuration\n\nShow pair counts and label distributions (SAME\\_PARAGRAPH, NEW\\_PARAGRAPH, NEWLINE) for each individual dataset and each combined configuration used in training.",
7
+ "metadata": {}
8
+ },
9
+ {
10
+ "cell_type": "code",
11
+ "id": "9yin74u3upm",
12
+ "source": "import os, sys\nos.chdir(os.path.join(os.path.dirname(os.getcwd()), \"..\"))\nprint(\"Working dir:\", os.getcwd())",
13
+ "metadata": {
14
+ "ExecuteTime": {
15
+ "end_time": "2026-04-02T21:49:50.940454Z",
16
+ "start_time": "2026-04-02T21:49:50.933243Z"
17
+ }
18
+ },
19
+ "outputs": [
20
+ {
21
+ "name": "stdout",
22
+ "output_type": "stream",
23
+ "text": [
24
+ "Working dir: /mnt/c/Lamosst/pohovory/bottlecap/bottlecap-ml-dev-test\n"
25
+ ]
26
+ }
27
+ ],
28
+ "execution_count": 1
29
+ },
30
+ {
31
+ "cell_type": "code",
32
+ "id": "1ilqpxsr8n",
33
+ "source": "from collections import Counter\nfrom pathlib import Path\n\nimport pandas as pd\n\nfrom src.datasets.combined_pairs_dataset import (\n CombinedPairsDataset,\n CombinedPairsConfig,\n ID2LABEL,\n)\n\nLABEL_NAMES = {0: \"SAME_PARAGRAPH\", 1: \"NEW_PARAGRAPH\", 2: \"NEWLINE\"}\nALL_DOMAINS = {\"pubmed\", \"wikipedia\", \"gutenberg\", \"recipes\"}\n\nCONFIGS = {\n \"PubMed\": ALL_DOMAINS - {\"pubmed\"},\n \"Wikipedia\": ALL_DOMAINS - {\"wikipedia\"},\n \"Gutenberg\": ALL_DOMAINS - {\"gutenberg\"},\n \"Recipes\": ALL_DOMAINS - {\"recipes\"},\n \"Wikipedia + PubMed\": ALL_DOMAINS - {\"wikipedia\", \"pubmed\"},\n \"Wikipedia + PubMed + Recipes\": ALL_DOMAINS - {\"wikipedia\", \"pubmed\", \"recipes\"},\n \"Wikipedia + PubMed + Gutenberg\": ALL_DOMAINS - {\"wikipedia\", \"pubmed\", \"gutenberg\"},\n}\n\nall_splits = {}\nfor name, exclude in CONFIGS.items():\n cfg = CombinedPairsConfig(exclude_domains=exclude)\n builder = CombinedPairsDataset(cfg)\n all_splits[name] = builder.build_splits()\n total = sum(len(v) for v in all_splits[name].values())\n print(f\"{name}: {total:,} total pairs\")",
34
+ "metadata": {
35
+ "ExecuteTime": {
36
+ "end_time": "2026-04-02T21:50:06.232209Z",
37
+ "start_time": "2026-04-02T21:49:50.964204Z"
38
+ }
39
+ },
40
+ "outputs": [
41
+ {
42
+ "name": "stderr",
43
+ "output_type": "stream",
44
+ "text": [
45
+ "/home/lamossta/.local/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
46
+ " from .autonotebook import tqdm as notebook_tqdm\n"
47
+ ]
48
+ },
49
+ {
50
+ "name": "stdout",
51
+ "output_type": "stream",
52
+ "text": [
53
+ "PubMed: 46,345 total pairs\n",
54
+ "Wikipedia: 17,895 total pairs\n",
55
+ "Gutenberg: 79,639 total pairs\n",
56
+ "Recipes: 1,946 total pairs\n",
57
+ "Wikipedia + PubMed: 64,240 total pairs\n",
58
+ "Wikipedia + PubMed + Recipes: 66,186 total pairs\n",
59
+ "Wikipedia + PubMed + Gutenberg: 146,979 total pairs\n"
60
+ ]
61
+ }
62
+ ],
63
+ "execution_count": 2
64
+ },
65
+ {
66
+ "cell_type": "markdown",
67
+ "id": "2bfuz0r29kz",
68
+ "source": "## 1. Individual datasets — label distribution per split",
69
+ "metadata": {}
70
+ },
71
+ {
72
+ "cell_type": "code",
73
+ "id": "kep6ox8m8b",
74
+ "source": "def build_summary_table(splits_dict: dict[str, list[dict]]) -> pd.DataFrame:\n rows = []\n for split_name, pairs in splits_dict.items():\n counts = Counter(p[\"label\"] for p in pairs)\n total = len(pairs)\n rows.append({\n \"Split\": split_name.upper(),\n \"Total\": f\"{total:,}\",\n \"SAME_PARA\": f\"{counts.get(0,0):,} ({counts.get(0,0)/max(total,1)*100:.1f}%)\",\n \"NEW_PARA\": f\"{counts.get(1,0):,} ({counts.get(1,0)/max(total,1)*100:.1f}%)\",\n \"NEWLINE\": f\"{counts.get(2,0):,} ({counts.get(2,0)/max(total,1)*100:.1f}%)\",\n })\n return pd.DataFrame(rows).set_index(\"Split\")\n\n\nindividual = [\"PubMed\", \"Wikipedia\", \"Gutenberg\", \"Recipes\"]\nfor name in individual:\n print(f\"\\n{'='*60}\")\n print(f\" {name}\")\n print(f\"{'='*60}\")\n display(build_summary_table(all_splits[name]))",
75
+ "metadata": {
76
+ "ExecuteTime": {
77
+ "end_time": "2026-04-02T21:50:06.462905Z",
78
+ "start_time": "2026-04-02T21:50:06.336414Z"
79
+ }
80
+ },
81
+ "outputs": [
82
+ {
83
+ "name": "stdout",
84
+ "output_type": "stream",
85
+ "text": [
86
+ "\n",
87
+ "============================================================\n",
88
+ " PubMed\n",
89
+ "============================================================\n"
90
+ ]
91
+ },
92
+ {
93
+ "data": {
94
+ "text/plain": [
95
+ " Total SAME_PARA NEW_PARA NEWLINE\n",
96
+ "Split \n",
97
+ "TRAIN 38,682 29,397 (76.0%) 8,732 (22.6%) 553 (1.4%)\n",
98
+ "VAL 3,774 2,721 (72.1%) 974 (25.8%) 79 (2.1%)\n",
99
+ "TEST 3,889 2,813 (72.3%) 972 (25.0%) 104 (2.7%)"
100
+ ],
101
+ "text/html": [
102
+ "<div>\n",
103
+ "<style scoped>\n",
104
+ " .dataframe tbody tr th:only-of-type {\n",
105
+ " vertical-align: middle;\n",
106
+ " }\n",
107
+ "\n",
108
+ " .dataframe tbody tr th {\n",
109
+ " vertical-align: top;\n",
110
+ " }\n",
111
+ "\n",
112
+ " .dataframe thead th {\n",
113
+ " text-align: right;\n",
114
+ " }\n",
115
+ "</style>\n",
116
+ "<table border=\"1\" class=\"dataframe\">\n",
117
+ " <thead>\n",
118
+ " <tr style=\"text-align: right;\">\n",
119
+ " <th></th>\n",
120
+ " <th>Total</th>\n",
121
+ " <th>SAME_PARA</th>\n",
122
+ " <th>NEW_PARA</th>\n",
123
+ " <th>NEWLINE</th>\n",
124
+ " </tr>\n",
125
+ " <tr>\n",
126
+ " <th>Split</th>\n",
127
+ " <th></th>\n",
128
+ " <th></th>\n",
129
+ " <th></th>\n",
130
+ " <th></th>\n",
131
+ " </tr>\n",
132
+ " </thead>\n",
133
+ " <tbody>\n",
134
+ " <tr>\n",
135
+ " <th>TRAIN</th>\n",
136
+ " <td>38,682</td>\n",
137
+ " <td>29,397 (76.0%)</td>\n",
138
+ " <td>8,732 (22.6%)</td>\n",
139
+ " <td>553 (1.4%)</td>\n",
140
+ " </tr>\n",
141
+ " <tr>\n",
142
+ " <th>VAL</th>\n",
143
+ " <td>3,774</td>\n",
144
+ " <td>2,721 (72.1%)</td>\n",
145
+ " <td>974 (25.8%)</td>\n",
146
+ " <td>79 (2.1%)</td>\n",
147
+ " </tr>\n",
148
+ " <tr>\n",
149
+ " <th>TEST</th>\n",
150
+ " <td>3,889</td>\n",
151
+ " <td>2,813 (72.3%)</td>\n",
152
+ " <td>972 (25.0%)</td>\n",
153
+ " <td>104 (2.7%)</td>\n",
154
+ " </tr>\n",
155
+ " </tbody>\n",
156
+ "</table>\n",
157
+ "</div>"
158
+ ]
159
+ },
160
+ "metadata": {},
161
+ "output_type": "display_data",
162
+ "jetTransient": {
163
+ "display_id": null
164
+ }
165
+ },
166
+ {
167
+ "name": "stdout",
168
+ "output_type": "stream",
169
+ "text": [
170
+ "\n",
171
+ "============================================================\n",
172
+ " Wikipedia\n",
173
+ "============================================================\n"
174
+ ]
175
+ },
176
+ {
177
+ "data": {
178
+ "text/plain": [
179
+ " Total SAME_PARA NEW_PARA NEWLINE\n",
180
+ "Split \n",
181
+ "TRAIN 16,384 11,459 (69.9%) 2,477 (15.1%) 2,448 (14.9%)\n",
182
+ "VAL 615 408 (66.3%) 194 (31.5%) 13 (2.1%)\n",
183
+ "TEST 896 625 (69.8%) 234 (26.1%) 37 (4.1%)"
184
+ ],
185
+ "text/html": [
186
+ "<div>\n",
187
+ "<style scoped>\n",
188
+ " .dataframe tbody tr th:only-of-type {\n",
189
+ " vertical-align: middle;\n",
190
+ " }\n",
191
+ "\n",
192
+ " .dataframe tbody tr th {\n",
193
+ " vertical-align: top;\n",
194
+ " }\n",
195
+ "\n",
196
+ " .dataframe thead th {\n",
197
+ " text-align: right;\n",
198
+ " }\n",
199
+ "</style>\n",
200
+ "<table border=\"1\" class=\"dataframe\">\n",
201
+ " <thead>\n",
202
+ " <tr style=\"text-align: right;\">\n",
203
+ " <th></th>\n",
204
+ " <th>Total</th>\n",
205
+ " <th>SAME_PARA</th>\n",
206
+ " <th>NEW_PARA</th>\n",
207
+ " <th>NEWLINE</th>\n",
208
+ " </tr>\n",
209
+ " <tr>\n",
210
+ " <th>Split</th>\n",
211
+ " <th></th>\n",
212
+ " <th></th>\n",
213
+ " <th></th>\n",
214
+ " <th></th>\n",
215
+ " </tr>\n",
216
+ " </thead>\n",
217
+ " <tbody>\n",
218
+ " <tr>\n",
219
+ " <th>TRAIN</th>\n",
220
+ " <td>16,384</td>\n",
221
+ " <td>11,459 (69.9%)</td>\n",
222
+ " <td>2,477 (15.1%)</td>\n",
223
+ " <td>2,448 (14.9%)</td>\n",
224
+ " </tr>\n",
225
+ " <tr>\n",
226
+ " <th>VAL</th>\n",
227
+ " <td>615</td>\n",
228
+ " <td>408 (66.3%)</td>\n",
229
+ " <td>194 (31.5%)</td>\n",
230
+ " <td>13 (2.1%)</td>\n",
231
+ " </tr>\n",
232
+ " <tr>\n",
233
+ " <th>TEST</th>\n",
234
+ " <td>896</td>\n",
235
+ " <td>625 (69.8%)</td>\n",
236
+ " <td>234 (26.1%)</td>\n",
237
+ " <td>37 (4.1%)</td>\n",
238
+ " </tr>\n",
239
+ " </tbody>\n",
240
+ "</table>\n",
241
+ "</div>"
242
+ ]
243
+ },
244
+ "metadata": {},
245
+ "output_type": "display_data",
246
+ "jetTransient": {
247
+ "display_id": null
248
+ }
249
+ },
250
+ {
251
+ "name": "stdout",
252
+ "output_type": "stream",
253
+ "text": [
254
+ "\n",
255
+ "============================================================\n",
256
+ " Gutenberg\n",
257
+ "============================================================\n"
258
+ ]
259
+ },
260
+ {
261
+ "data": {
262
+ "text/plain": [
263
+ " Total SAME_PARA NEW_PARA NEWLINE\n",
264
+ "Split \n",
265
+ "TRAIN 45,000 44,134 (98.1%) 742 (1.6%) 124 (0.3%)\n",
266
+ "VAL 16,895 16,321 (96.6%) 252 (1.5%) 322 (1.9%)\n",
267
+ "TEST 17,744 17,383 (98.0%) 337 (1.9%) 24 (0.1%)"
268
+ ],
269
+ "text/html": [
270
+ "<div>\n",
271
+ "<style scoped>\n",
272
+ " .dataframe tbody tr th:only-of-type {\n",
273
+ " vertical-align: middle;\n",
274
+ " }\n",
275
+ "\n",
276
+ " .dataframe tbody tr th {\n",
277
+ " vertical-align: top;\n",
278
+ " }\n",
279
+ "\n",
280
+ " .dataframe thead th {\n",
281
+ " text-align: right;\n",
282
+ " }\n",
283
+ "</style>\n",
284
+ "<table border=\"1\" class=\"dataframe\">\n",
285
+ " <thead>\n",
286
+ " <tr style=\"text-align: right;\">\n",
287
+ " <th></th>\n",
288
+ " <th>Total</th>\n",
289
+ " <th>SAME_PARA</th>\n",
290
+ " <th>NEW_PARA</th>\n",
291
+ " <th>NEWLINE</th>\n",
292
+ " </tr>\n",
293
+ " <tr>\n",
294
+ " <th>Split</th>\n",
295
+ " <th></th>\n",
296
+ " <th></th>\n",
297
+ " <th></th>\n",
298
+ " <th></th>\n",
299
+ " </tr>\n",
300
+ " </thead>\n",
301
+ " <tbody>\n",
302
+ " <tr>\n",
303
+ " <th>TRAIN</th>\n",
304
+ " <td>45,000</td>\n",
305
+ " <td>44,134 (98.1%)</td>\n",
306
+ " <td>742 (1.6%)</td>\n",
307
+ " <td>124 (0.3%)</td>\n",
308
+ " </tr>\n",
309
+ " <tr>\n",
310
+ " <th>VAL</th>\n",
311
+ " <td>16,895</td>\n",
312
+ " <td>16,321 (96.6%)</td>\n",
313
+ " <td>252 (1.5%)</td>\n",
314
+ " <td>322 (1.9%)</td>\n",
315
+ " </tr>\n",
316
+ " <tr>\n",
317
+ " <th>TEST</th>\n",
318
+ " <td>17,744</td>\n",
319
+ " <td>17,383 (98.0%)</td>\n",
320
+ " <td>337 (1.9%)</td>\n",
321
+ " <td>24 (0.1%)</td>\n",
322
+ " </tr>\n",
323
+ " </tbody>\n",
324
+ "</table>\n",
325
+ "</div>"
326
+ ]
327
+ },
328
+ "metadata": {},
329
+ "output_type": "display_data",
330
+ "jetTransient": {
331
+ "display_id": null
332
+ }
333
+ },
334
+ {
335
+ "name": "stdout",
336
+ "output_type": "stream",
337
+ "text": [
338
+ "\n",
339
+ "============================================================\n",
340
+ " Recipes\n",
341
+ "============================================================\n"
342
+ ]
343
+ },
344
+ {
345
+ "data": {
346
+ "text/plain": [
347
+ " Total SAME_PARA NEW_PARA NEWLINE\n",
348
+ "Split \n",
349
+ "TRAIN 1,548 229 (14.8%) 161 (10.4%) 1,158 (74.8%)\n",
350
+ "VAL 228 29 (12.7%) 20 (8.8%) 179 (78.5%)\n",
351
+ "TEST 170 9 (5.3%) 20 (11.8%) 141 (82.9%)"
352
+ ],
353
+ "text/html": [
354
+ "<div>\n",
355
+ "<style scoped>\n",
356
+ " .dataframe tbody tr th:only-of-type {\n",
357
+ " vertical-align: middle;\n",
358
+ " }\n",
359
+ "\n",
360
+ " .dataframe tbody tr th {\n",
361
+ " vertical-align: top;\n",
362
+ " }\n",
363
+ "\n",
364
+ " .dataframe thead th {\n",
365
+ " text-align: right;\n",
366
+ " }\n",
367
+ "</style>\n",
368
+ "<table border=\"1\" class=\"dataframe\">\n",
369
+ " <thead>\n",
370
+ " <tr style=\"text-align: right;\">\n",
371
+ " <th></th>\n",
372
+ " <th>Total</th>\n",
373
+ " <th>SAME_PARA</th>\n",
374
+ " <th>NEW_PARA</th>\n",
375
+ " <th>NEWLINE</th>\n",
376
+ " </tr>\n",
377
+ " <tr>\n",
378
+ " <th>Split</th>\n",
379
+ " <th></th>\n",
380
+ " <th></th>\n",
381
+ " <th></th>\n",
382
+ " <th></th>\n",
383
+ " </tr>\n",
384
+ " </thead>\n",
385
+ " <tbody>\n",
386
+ " <tr>\n",
387
+ " <th>TRAIN</th>\n",
388
+ " <td>1,548</td>\n",
389
+ " <td>229 (14.8%)</td>\n",
390
+ " <td>161 (10.4%)</td>\n",
391
+ " <td>1,158 (74.8%)</td>\n",
392
+ " </tr>\n",
393
+ " <tr>\n",
394
+ " <th>VAL</th>\n",
395
+ " <td>228</td>\n",
396
+ " <td>29 (12.7%)</td>\n",
397
+ " <td>20 (8.8%)</td>\n",
398
+ " <td>179 (78.5%)</td>\n",
399
+ " </tr>\n",
400
+ " <tr>\n",
401
+ " <th>TEST</th>\n",
402
+ " <td>170</td>\n",
403
+ " <td>9 (5.3%)</td>\n",
404
+ " <td>20 (11.8%)</td>\n",
405
+ " <td>141 (82.9%)</td>\n",
406
+ " </tr>\n",
407
+ " </tbody>\n",
408
+ "</table>\n",
409
+ "</div>"
410
+ ]
411
+ },
412
+ "metadata": {},
413
+ "output_type": "display_data",
414
+ "jetTransient": {
415
+ "display_id": null
416
+ }
417
+ }
418
+ ],
419
+ "execution_count": 3
420
+ },
421
+ {
422
+ "cell_type": "markdown",
423
+ "id": "gg9dk3u18lc",
424
+ "source": "## 2. Combined configurations — label distribution per split",
425
+ "metadata": {
426
+ "ExecuteTime": {
427
+ "end_time": "2026-04-01T00:06:03.089421Z",
428
+ "start_time": "2026-04-01T00:06:02.576126Z"
429
+ }
430
+ }
431
+ },
432
+ {
433
+ "cell_type": "code",
434
+ "id": "5tlsab97e6r",
435
+ "source": "combined = [\"Wikipedia + PubMed\", \"Wikipedia + PubMed + Recipes\", \"Wikipedia + PubMed + Gutenberg\"]\nfor name in combined:\n print(f\"\\n{'='*60}\")\n print(f\" {name}\")\n print(f\"{'='*60}\")\n display(build_summary_table(all_splits[name]))",
436
+ "metadata": {
437
+ "ExecuteTime": {
438
+ "end_time": "2026-04-02T21:50:07.017542Z",
439
+ "start_time": "2026-04-02T21:50:06.801419Z"
440
+ }
441
+ },
442
+ "outputs": [
443
+ {
444
+ "name": "stdout",
445
+ "output_type": "stream",
446
+ "text": [
447
+ "\n",
448
+ "============================================================\n",
449
+ " Wikipedia + PubMed\n",
450
+ "============================================================\n"
451
+ ]
452
+ },
453
+ {
454
+ "data": {
455
+ "text/plain": [
456
+ " Total SAME_PARA NEW_PARA NEWLINE\n",
457
+ "Split \n",
458
+ "TRAIN 52,596 39,001 (74.2%) 10,936 (20.8%) 2,659 (5.1%)\n",
459
+ "VAL 5,264 3,736 (71.0%) 1,418 (26.9%) 110 (2.1%)\n",
460
+ "TEST 6,380 4,686 (73.4%) 1,229 (19.3%) 465 (7.3%)"
461
+ ],
462
+ "text/html": [
463
+ "<div>\n",
464
+ "<style scoped>\n",
465
+ " .dataframe tbody tr th:only-of-type {\n",
466
+ " vertical-align: middle;\n",
467
+ " }\n",
468
+ "\n",
469
+ " .dataframe tbody tr th {\n",
470
+ " vertical-align: top;\n",
471
+ " }\n",
472
+ "\n",
473
+ " .dataframe thead th {\n",
474
+ " text-align: right;\n",
475
+ " }\n",
476
+ "</style>\n",
477
+ "<table border=\"1\" class=\"dataframe\">\n",
478
+ " <thead>\n",
479
+ " <tr style=\"text-align: right;\">\n",
480
+ " <th></th>\n",
481
+ " <th>Total</th>\n",
482
+ " <th>SAME_PARA</th>\n",
483
+ " <th>NEW_PARA</th>\n",
484
+ " <th>NEWLINE</th>\n",
485
+ " </tr>\n",
486
+ " <tr>\n",
487
+ " <th>Split</th>\n",
488
+ " <th></th>\n",
489
+ " <th></th>\n",
490
+ " <th></th>\n",
491
+ " <th></th>\n",
492
+ " </tr>\n",
493
+ " </thead>\n",
494
+ " <tbody>\n",
495
+ " <tr>\n",
496
+ " <th>TRAIN</th>\n",
497
+ " <td>52,596</td>\n",
498
+ " <td>39,001 (74.2%)</td>\n",
499
+ " <td>10,936 (20.8%)</td>\n",
500
+ " <td>2,659 (5.1%)</td>\n",
501
+ " </tr>\n",
502
+ " <tr>\n",
503
+ " <th>VAL</th>\n",
504
+ " <td>5,264</td>\n",
505
+ " <td>3,736 (71.0%)</td>\n",
506
+ " <td>1,418 (26.9%)</td>\n",
507
+ " <td>110 (2.1%)</td>\n",
508
+ " </tr>\n",
509
+ " <tr>\n",
510
+ " <th>TEST</th>\n",
511
+ " <td>6,380</td>\n",
512
+ " <td>4,686 (73.4%)</td>\n",
513
+ " <td>1,229 (19.3%)</td>\n",
514
+ " <td>465 (7.3%)</td>\n",
515
+ " </tr>\n",
516
+ " </tbody>\n",
517
+ "</table>\n",
518
+ "</div>"
519
+ ]
520
+ },
521
+ "metadata": {},
522
+ "output_type": "display_data",
523
+ "jetTransient": {
524
+ "display_id": null
525
+ }
526
+ },
527
+ {
528
+ "name": "stdout",
529
+ "output_type": "stream",
530
+ "text": [
531
+ "\n",
532
+ "============================================================\n",
533
+ " Wikipedia + PubMed + Recipes\n",
534
+ "============================================================\n"
535
+ ]
536
+ },
537
+ {
538
+ "data": {
539
+ "text/plain": [
540
+ " Total SAME_PARA NEW_PARA NEWLINE\n",
541
+ "Split \n",
542
+ "TRAIN 54,158 39,190 (72.4%) 11,096 (20.5%) 3,872 (7.1%)\n",
543
+ "VAL 5,501 3,796 (69.0%) 1,439 (26.2%) 266 (4.8%)\n",
544
+ "TEST 6,527 4,704 (72.1%) 1,249 (19.1%) 574 (8.8%)"
545
+ ],
546
+ "text/html": [
547
+ "<div>\n",
548
+ "<style scoped>\n",
549
+ " .dataframe tbody tr th:only-of-type {\n",
550
+ " vertical-align: middle;\n",
551
+ " }\n",
552
+ "\n",
553
+ " .dataframe tbody tr th {\n",
554
+ " vertical-align: top;\n",
555
+ " }\n",
556
+ "\n",
557
+ " .dataframe thead th {\n",
558
+ " text-align: right;\n",
559
+ " }\n",
560
+ "</style>\n",
561
+ "<table border=\"1\" class=\"dataframe\">\n",
562
+ " <thead>\n",
563
+ " <tr style=\"text-align: right;\">\n",
564
+ " <th></th>\n",
565
+ " <th>Total</th>\n",
566
+ " <th>SAME_PARA</th>\n",
567
+ " <th>NEW_PARA</th>\n",
568
+ " <th>NEWLINE</th>\n",
569
+ " </tr>\n",
570
+ " <tr>\n",
571
+ " <th>Split</th>\n",
572
+ " <th></th>\n",
573
+ " <th></th>\n",
574
+ " <th></th>\n",
575
+ " <th></th>\n",
576
+ " </tr>\n",
577
+ " </thead>\n",
578
+ " <tbody>\n",
579
+ " <tr>\n",
580
+ " <th>TRAIN</th>\n",
581
+ " <td>54,158</td>\n",
582
+ " <td>39,190 (72.4%)</td>\n",
583
+ " <td>11,096 (20.5%)</td>\n",
584
+ " <td>3,872 (7.1%)</td>\n",
585
+ " </tr>\n",
586
+ " <tr>\n",
587
+ " <th>VAL</th>\n",
588
+ " <td>5,501</td>\n",
589
+ " <td>3,796 (69.0%)</td>\n",
590
+ " <td>1,439 (26.2%)</td>\n",
591
+ " <td>266 (4.8%)</td>\n",
592
+ " </tr>\n",
593
+ " <tr>\n",
594
+ " <th>TEST</th>\n",
595
+ " <td>6,527</td>\n",
596
+ " <td>4,704 (72.1%)</td>\n",
597
+ " <td>1,249 (19.1%)</td>\n",
598
+ " <td>574 (8.8%)</td>\n",
599
+ " </tr>\n",
600
+ " </tbody>\n",
601
+ "</table>\n",
602
+ "</div>"
603
+ ]
604
+ },
605
+ "metadata": {},
606
+ "output_type": "display_data",
607
+ "jetTransient": {
608
+ "display_id": null
609
+ }
610
+ },
611
+ {
612
+ "name": "stdout",
613
+ "output_type": "stream",
614
+ "text": [
615
+ "\n",
616
+ "============================================================\n",
617
+ " Wikipedia + PubMed + Gutenberg\n",
618
+ "============================================================\n"
619
+ ]
620
+ },
621
+ {
622
+ "data": {
623
+ "text/plain": [
624
+ " Total SAME_PARA NEW_PARA NEWLINE\n",
625
+ "Split \n",
626
+ "TRAIN 97,596 82,973 (85.0%) 11,731 (12.0%) 2,892 (3.0%)\n",
627
+ "VAL 25,431 23,592 (92.8%) 1,692 (6.7%) 147 (0.6%)\n",
628
+ "TEST 23,952 21,966 (91.7%) 1,467 (6.1%) 519 (2.2%)"
629
+ ],
630
+ "text/html": [
631
+ "<div>\n",
632
+ "<style scoped>\n",
633
+ " .dataframe tbody tr th:only-of-type {\n",
634
+ " vertical-align: middle;\n",
635
+ " }\n",
636
+ "\n",
637
+ " .dataframe tbody tr th {\n",
638
+ " vertical-align: top;\n",
639
+ " }\n",
640
+ "\n",
641
+ " .dataframe thead th {\n",
642
+ " text-align: right;\n",
643
+ " }\n",
644
+ "</style>\n",
645
+ "<table border=\"1\" class=\"dataframe\">\n",
646
+ " <thead>\n",
647
+ " <tr style=\"text-align: right;\">\n",
648
+ " <th></th>\n",
649
+ " <th>Total</th>\n",
650
+ " <th>SAME_PARA</th>\n",
651
+ " <th>NEW_PARA</th>\n",
652
+ " <th>NEWLINE</th>\n",
653
+ " </tr>\n",
654
+ " <tr>\n",
655
+ " <th>Split</th>\n",
656
+ " <th></th>\n",
657
+ " <th></th>\n",
658
+ " <th></th>\n",
659
+ " <th></th>\n",
660
+ " </tr>\n",
661
+ " </thead>\n",
662
+ " <tbody>\n",
663
+ " <tr>\n",
664
+ " <th>TRAIN</th>\n",
665
+ " <td>97,596</td>\n",
666
+ " <td>82,973 (85.0%)</td>\n",
667
+ " <td>11,731 (12.0%)</td>\n",
668
+ " <td>2,892 (3.0%)</td>\n",
669
+ " </tr>\n",
670
+ " <tr>\n",
671
+ " <th>VAL</th>\n",
672
+ " <td>25,431</td>\n",
673
+ " <td>23,592 (92.8%)</td>\n",
674
+ " <td>1,692 (6.7%)</td>\n",
675
+ " <td>147 (0.6%)</td>\n",
676
+ " </tr>\n",
677
+ " <tr>\n",
678
+ " <th>TEST</th>\n",
679
+ " <td>23,952</td>\n",
680
+ " <td>21,966 (91.7%)</td>\n",
681
+ " <td>1,467 (6.1%)</td>\n",
682
+ " <td>519 (2.2%)</td>\n",
683
+ " </tr>\n",
684
+ " </tbody>\n",
685
+ "</table>\n",
686
+ "</div>"
687
+ ]
688
+ },
689
+ "metadata": {},
690
+ "output_type": "display_data",
691
+ "jetTransient": {
692
+ "display_id": null
693
+ }
694
+ }
695
+ ],
696
+ "execution_count": 4
697
+ }
698
+ ],
699
+ "metadata": {
700
+ "kernelspec": {
701
+ "display_name": "Python 3",
702
+ "language": "python",
703
+ "name": "python3"
704
+ },
705
+ "language_info": {
706
+ "name": "python",
707
+ "version": "3.11.0"
708
+ }
709
+ },
710
+ "nbformat": 4,
711
+ "nbformat_minor": 5
712
+ }
src/notebooks/inspect_ted_data.ipynb ADDED
The diff for this file is too large to render. See raw diff