oberbics/llama-3.1-newspaper-arguments-agent-optimized_V3

Llama 3.1 8B Instruct Argument Mining Adapter

This is a 4-bit QLoRA adapter fine-tuned on top of meta-llama/Llama-3.1-8B-Instruct for argument extraction and mining from historical German newspaper texts.

It is trained to extract argument structures verbatim from source texts and output them in a structured XML format.

Model Details

• Base Model: meta-llama/Llama-3.1-8B-Instruct
• Method: QLoRA (4-bit Double Quantization, nf4)
• Language: German, English (mostly German historical newspaper text)
• Target Task: Structured argument extraction

Training Parameters & Hyperparameters

Parameter	Value
Epochs	3
LoRA Rank (r)	32
LoRA Alpha	32
LoRA Dropout	0.1
Target Modules	All linear layers (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj)
Per Device Train Batch Size	1
Gradient Accumulation Steps	8 (Effective batch size of 8)
Learning Rate	3e-5
Learning Rate Schedule	Cosine
Warmup Ratio	0.05
Optimizer	paged_adamw_8bit
Max Sequence Length	2048
Gradient Checkpointing	Enabled

Dataset & Split Details

• Dataset: argument_mining_final.xlsx (German historical newspaper articles)
• Train split: 364 rows (216 NA-only, 148 containing arguments)
• Validation split: 92 rows (54 NA-only, 38 containing arguments)
• Ratio: 80% Train / 20% Validation (Stratified split preserving NA ratios)

Evaluation & Loss Progress Logs

Step	Epoch	Training Loss	Validation Loss
5	0.11	2.6920	-
10	0.22	2.6030	-
15	0.33	2.4700	-
20	0.44	2.0750	-
25	0.55	2.0100	-
30	0.66	1.5800	-
35	0.77	1.3870	-
40	0.88	1.2340	-
45	0.99	1.3520	-
50	1.09	1.1270	1.1360
55	1.20	1.0640	-
60	1.31	1.0960	-
65	1.42	1.2160	-
70	1.53	1.0590	-
75	1.64	1.2030	-
80	1.75	1.4240	-
85	1.86	1.0400	-
90	1.97	0.9421	-
95	2.07	1.1070	-
100	2.18	1.1210	1.1010
115	2.51	0.9683	-
120	2.62	1.1400	-
125	2.73	1.1520	-
130	2.84	1.0990	-
135	2.95	1.2520	-
138	3.00	-	1.0980

• Total Training Runtime: 1,237 seconds (~20.6 minutes)
• Average Train Loss: 1.3820
• Final Validation Loss: 1.0980 (best model loaded at end)

XML Schema & Prompt Format

The model is trained to extract argumentative units matching the following XML tags:

<argument>Verbatim extracted argument text from the source text</argument>
<claim>Core claim / implication in one sentence</claim>
<explanation>Why this represents an argument</explanation>
<confidence>Confidence rating (0-1)</confidence>

If no arguments exist in the text, it outputs NA for all fields.

Sample Inference Format:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>
Cutting Knowledge Date: December 2023
Today Date: 26 Jul 2024

You are an expert at analyzing historical texts and you hate to summarize

OUTPUT FORMAT - EXACTLY these 4 XML tags and NOTHING else:
<argument>Original argument text OR "NA"</argument>
<claim>Core claim (implication) in one sentence OR "NA"</claim>
<explanation>Why this is an argument OR "NA"</explanation>
<confidence>0-1</confidence>
...
<|eot_id|><|start_header_id|>user<|end_header_id|>
Extract argumentative units in its original form...
Text to analyze: [ARTICLE TEXT]<|eot_id|><|start_header_id|>assistant<|end_header_id|>